Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure

Orbiter Check History

Showing 50 of 335296 reports (Page 116 of 6706)
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip: 
2 out of 60 is failed (threshold=3) [.....................................XX.....................]
duration: 
9.90599663s
trace: 
https://jaeger.dev.storj.io/trace/75ef8b5e6bc2819a
FAIL (2025-12-27 14:15:28 UTC) 
orbiter warning: context deadline exceeded
Duration: 9.906024984s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
20 out of 60 is failed (threshold=3) [.XX.......X....X..XX.X.XXXX..X..X.....X.XXX..X...XX.........]
duration: 
5.703656171s
trace: 
https://jaeger.dev.storj.io/trace/7a3b11897bee958b
FAIL (2025-12-27 14:14:54 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 27 14:09:21.611393571 (526h40m21.482476616s) [upload=461959]: Starting upload
Dec 27 14:09:21.611546842 (526h40m21.482629877s) [upload=461959]: Done starting upload
Dec 27 14:09:21.611609071 (526h40m21.482692116s) [upload=461959]: Uploading segments...
Dec 27 14:09:21.612269158 (526h40m21.483352213s) [upload=461959]: Got next segment. Inline: false
Dec 27 14:09:21.612349201 (526h40m21.483432256s) [upload=461959]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 14:09:21.612356555 (526h40m21.4834396s) [upload=461959]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 14:09:21.856299604 (526h40m21.727382649s) [seg_pos={0 0}, upload=461959]: Begin upload segment...
Dec 27 14:09:21.856370038 (526h40m21.727453073s) [seg_pos={0 0}, upload=461959]: Done begin upload segment.
Dec 27 14:09:21.856420784 (526h40m21.727503819s) [noise=true , address=145.79.227.12:4765   , piece_id=5YKAMK34JOHYLZJT2D5WRZBY6N43346GSRFYFXHWZ6Z3AWWAQJ6A, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.856479476 (526h40m21.727562521s) [noise=true , address=145.79.227.29:4942   , piece_id=R6L233G7TZUNMLBODN4UEJH66YDWXL66B2WGAVQPHCIH5JXC74FA, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.857625017 (526h40m21.728708052s) [noise=true , address=45.91.116.13:4155    , piece_id=LBAG6T3GGWKHCOHA7T53VKSOX2O55KKQ4EGMRKDCK6PGFCTUIJVQ, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.858100663 (526h40m21.729183698s) [noise=true , address=145.79.227.10:4402   , piece_id=XUDYA3V3NLMFVKRX5ERUSILGUIOLX6NJNXWIL2PZDT6GFJMA3NDA, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.858170176 (526h40m21.729253231s) [noise=true , address=45.91.116.7:4009     , piece_id=US7DNVKAFLNLTFQB45RKVON242QU3RF3T5IELTT2WCNMUN6ZAYBQ, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.858402087 (526h40m21.729485122s) [noise=true , address=45.91.116.11:4120    , piece_id=UXEDL7GFL64HCRQ67THFOC2MKAVGYREYNQS7WEXJYGBOY3H7EYWQ, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.858794865 (526h40m21.7298779s) [noise=true , address=77.90.31.17:4773     , piece_id=5JIB2CV4VJSBBFINGJ4WCWLQN4H2T4WSEL6XDB3A755TJQKQOKYQ, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.859040964 (526h40m21.730124009s) [noise=true , address=145.79.227.19:4078   , piece_id=UAMIORNB3U6JKMHUIQRNRUNYW4T7S6YHERJG7ZVIHC5KLEXXCF6Q, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.861213120 (526h40m21.732296205s) [noise=true , address=145.79.227.45:4230   , piece_id=CPFKFQEL7BVP3V6I2LCDPVCGQACW3RDR7QT2V7XIJS77H6ZU7RIA, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.862069922 (526h40m21.733152957s) [noise=true , address=145.79.227.50:4375   , piece_id=XUAHWMEE2ZKSLFO3Z6CC7CZJZOSDSMNMDXYVGVK5ZLO2TDYZCH4Q, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.863841424 (526h40m21.734924469s) [noise=true , address=145.79.227.20:4438   , piece_id=42J67NFOF35HJMHKU2FLTASKR2TGKIY6NRBISTOIDIWBY64ACWSQ, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.867587077 (526h40m21.738670122s) [noise=true , address=145.79.227.42:4123   , piece_id=K6WUIK2EKFXSGSUXIOCSV4R3MOB2S4HPLEXOJRAKMCUC2GWYTEYA, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.869214605 (526h40m21.74029765s) [noise=true , address=45.91.116.6:4974     , piece_id=QXEEQHFYAH67J6OSDKWDVJCJV2S7ENNTQUWGV6TO5MMLTQ7JSOBQ, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.870424339 (526h40m21.741507384s) [noise=true , address=145.79.227.23:4546   , piece_id=YAGKC7A2NH6QNUCYJH3TIGQXTZZPDQ4FZTBR6LLAXSYMJTFAMI7A, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.871897163 (526h40m21.742980208s) [noise=true , address=145.79.227.17:4970   , piece_id=2SEU5CWQM23OWEPWAGUO34UBDJKT7TES2N3TZEIDIWDCCUBW5NEQ, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.872851330 (526h40m21.743934376s) [noise=true , address=145.79.227.36:4519   , piece_id=TVVXVEG5UT3CDL5AWEU4M2TQTOUTIV2RPCGBOGDCD2SRJIE4KBVA, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:23.532718349 (526h40m23.403801384s) [upload=461959]: Next returned nil segment
Dec 27 14:09:23.532735001 (526h40m23.403818047s) [upload=461959]: Waiting for error group managing segments...
Dec 27 14:09:23.862990534 (526h40m23.734073569s) [noise=true , address=145.79.227.12:4765   , piece_id=5YKAMK34JOHYLZJT2D5WRZBY6N43346GSRFYFXHWZ6Z3AWWAQJ6A, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:23.871865954 (526h40m23.742948989s) [noise=true , address=145.79.227.50:4375   , piece_id=XUAHWMEE2ZKSLFO3Z6CC7CZJZOSDSMNMDXYVGVK5ZLO2TDYZCH4Q, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:23.874552059 (526h40m23.745635104s) [noise=true , address=145.79.227.36:4519   , piece_id=TVVXVEG5UT3CDL5AWEU4M2TQTOUTIV2RPCGBOGDCD2SRJIE4KBVA, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:23.875081458 (526h40m23.746164493s) [noise=true , address=145.79.227.29:4942   , piece_id=R6L233G7TZUNMLBODN4UEJH66YDWXL66B2WGAVQPHCIH5JXC74FA, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:23.880889777 (526h40m23.751972812s) [noise=true , address=145.79.227.17:4970   , piece_id=2SEU5CWQM23OWEPWAGUO34UBDJKT7TES2N3TZEIDIWDCCUBW5NEQ, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:23.881779612 (526h40m23.752862657s) [noise=true , address=145.79.227.23:4546   , piece_id=YAGKC7A2NH6QNUCYJH3TIGQXTZZPDQ4FZTBR6LLAXSYMJTFAMI7A, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:23.888373497 (526h40m23.759456542s) [noise=true , address=145.79.227.10:4402   , piece_id=XUDYA3V3NLMFVKRX5ERUSILGUIOLX6NJNXWIL2PZDT6GFJMA3NDA, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:23.896892469 (526h40m23.767975504s) [noise=true , address=145.79.227.42:4123   , piece_id=K6WUIK2EKFXSGSUXIOCSV4R3MOB2S4HPLEXOJRAKMCUC2GWYTEYA, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:23.899060166 (526h40m23.770143202s) [noise=true , address=145.79.227.20:4438   , piece_id=42J67NFOF35HJMHKU2FLTASKR2TGKIY6NRBISTOIDIWBY64ACWSQ, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:23.899525874 (526h40m23.770608919s) [noise=true , address=145.79.227.45:4230   , piece_id=CPFKFQEL7BVP3V6I2LCDPVCGQACW3RDR7QT2V7XIJS77H6ZU7RIA, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:23.907289918 (526h40m23.778372953s) [noise=true , address=145.79.227.19:4078   , piece_id=UAMIORNB3U6JKMHUIQRNRUNYW4T7S6YHERJG7ZVIHC5KLEXXCF6Q, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:24.122094452 (526h40m23.993177497s) [noise=true , address=45.91.116.11:4120    , piece_id=UXEDL7GFL64HCRQ67THFOC2MKAVGYREYNQS7WEXJYGBOY3H7EYWQ, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:24.166811121 (526h40m24.037894166s) [noise=true , address=77.90.31.17:4773     , piece_id=5JIB2CV4VJSBBFINGJ4WCWLQN4H2T4WSEL6XDB3A755TJQKQOKYQ, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:24.166864091 (526h40m24.037947126s) [seg_pos={0 0}, upload=461959]: Segment reached optimal threshold of 13 pieces.
Dec 27 14:09:24.167067258 (526h40m24.038150303s) [noise=true , address=45.91.116.7:4009     , piece_id=US7DNVKAFLNLTFQB45RKVON242QU3RF3T5IELTT2WCNMUN6ZAYBQ, seg_pos={0 0}, upload=461959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF
Dec 27 14:09:24.167077768 (526h40m24.038160823s) [noise=true , address=45.91.116.7:4009     , piece_id=US7DNVKAFLNLTFQB45RKVON242QU3RF3T5IELTT2WCNMUN6ZAYBQ, seg_pos={0 0}, upload=461959]: Long tail cancellation due to optimal threshold
Dec 27 14:09:24.167395192 (526h40m24.038478237s) [noise=true , address=45.91.116.13:4155    , piece_id=LBAG6T3GGWKHCOHA7T53VKSOX2O55KKQ4EGMRKDCK6PGFCTUIJVQ, seg_pos={0 0}, upload=461959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF
Dec 27 14:09:24.167402437 (526h40m24.038485472s) [noise=true , address=45.91.116.13:4155    , piece_id=LBAG6T3GGWKHCOHA7T53VKSOX2O55KKQ4EGMRKDCK6PGFCTUIJVQ, seg_pos={0 0}, upload=461959]: Long tail cancellation due to optimal threshold
Dec 27 14:09:24.167513648 (526h40m24.038596694s) [noise=true , address=45.91.116.6:4974     , piece_id=QXEEQHFYAH67J6OSDKWDVJCJV2S7ENNTQUWGV6TO5MMLTQ7JSOBQ, seg_pos={0 0}, upload=461959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF
Dec 27 14:09:24.167519349 (526h40m24.038602384s) [noise=true , address=45.91.116.6:4974     , piece_id=QXEEQHFYAH67J6OSDKWDVJCJV2S7ENNTQUWGV6TO5MMLTQ7JSOBQ, seg_pos={0 0}, upload=461959]: Long tail cancellation due to optimal threshold
Dec 27 14:09:24.167532774 (526h40m24.038615809s) [seg_pos={0 0}, upload=461959]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 14:09:24.167612757 (526h40m24.038695792s) [upload=461959]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 14:09:24.167618839 (526h40m24.038701884s) [upload=461959]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 14:09:24.455792179 (526h40m24.326875214s) [upload=461959]: Done uploading segments...
Dec 27 14:09:24.455820133 (526h40m24.326903168s) [upload=461959]: Upload finished. err: <nil>
Duration: 5.703675629s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
4 out of 60 is failed (threshold=3) [.........X.............XX.................................X.]
duration: 
47.348227988s
trace: 
https://jaeger.dev.storj.io/trace/30098651d72a60c0
FAIL (2025-12-27 14:14:44 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 122trJbDHuqeBMQHZT97HP2PgkeuLyUiDiuRm4nayu6ZTH7tat6, Piece ID: 5JTU5Y24LBUWCCJE2ALXFYMH6DQ45UCIPM6X5FDESKSUJM7CLI6Q) context deadline exceeded; piecestore: (Node ID: 12TaXapd6q5Vh1CQ5tQP98iQqVY99HFdGGv8CebGq49D9aKR9n2, Piece ID: 5DLPLIRQM3M7SQZPTVVU3CJ75FP3AYLGRUADRA7I22433UKRXNSQ) context deadline exceeded; piecestore: (Node ID: 1mmKSzdU8tViT7WLrppbMMsbAzB2MetMSoLj6ToyLHyQ5RaNjA, Piece ID: JUSY6UY7I3L43KXO3MB7O2KWDSJ7LTFRCBHIQEFR2L4BW5E5VWRQ) context deadline exceeded
======LOGS======
AVTIQOFIMTIU45SJPEBLBABQ, seg_pos={0 0}, upload=462013]: Uploading piece...
Dec 27 14:12:57.923596308 (526h43m57.794679343s) [noise=true , address=77.90.31.12:4599     , piece_id=LZ3YVYONYMFTVXOJEX43SD23SQEATWFNNMRWV2CE6YHI7OSI27JA, seg_pos={0 0}, upload=462013]: Uploading piece...
Dec 27 14:12:57.923971432 (526h43m57.795054467s) [noise=true , address=45.91.116.6:4983     , piece_id=5JTU5Y24LBUWCCJE2ALXFYMH6DQ45UCIPM6X5FDESKSUJM7CLI6Q, seg_pos={0 0}, upload=462013]: Uploading piece...
Dec 27 14:12:57.924200558 (526h43m57.795283593s) [noise=true , address=77.90.31.18:4837     , piece_id=IVSHYSEVNYF5ARFGT6KKPLU7I2TPUEU2TRPU6NHW5GFPDVPF3P7Q, seg_pos={0 0}, upload=462013]: Uploading piece...
Dec 27 14:12:57.924293756 (526h43m57.795376801s) [noise=true , address=77.90.31.10:4546     , piece_id=V3S7W7NUG5REGC6PRIMD4LL5ZEJMHT3ORLV5NATAMPVJ54RKYZVA, seg_pos={0 0}, upload=462013]: Uploading piece...
Dec 27 14:12:57.924668840 (526h43m57.795751875s) [noise=true , address=145.79.227.34:4918   , piece_id=7KAA2LG2TGLRDOQHTJVRHJIDSDUSHBNIICMT5WCYHN5UIHIKUWRQ, seg_pos={0 0}, upload=462013]: Uploading piece...
Dec 27 14:12:57.924819297 (526h43m57.795902342s) [noise=true , address=45.91.116.12:4234    , piece_id=JUSY6UY7I3L43KXO3MB7O2KWDSJ7LTFRCBHIQEFR2L4BW5E5VWRQ, seg_pos={0 0}, upload=462013]: Uploading piece...
Dec 27 14:12:57.925054945 (526h43m57.79613799s) [noise=true , address=145.79.227.46:4633   , piece_id=PYNPAJYIFPVDGYJQVHUXRTDSUBXEDYKEVKI4KPSNBWHZFLJNTSBA, seg_pos={0 0}, upload=462013]: Uploading piece...
Dec 27 14:12:57.925257120 (526h43m57.796340155s) [noise=true , address=64.49.13.169:4705    , piece_id=5A5NYCMNFLEONT2SJGTVADCLSW5IBUBPRHNDCYVB3SMO23VV4KXQ, seg_pos={0 0}, upload=462013]: Uploading piece...
Dec 27 14:12:57.925641512 (526h43m57.796724557s) [noise=true , address=64.49.13.26:4947     , piece_id=J6V7JSP6PLJY3LRZNN5SGWTC3F2LPNVVXWNKQBLFQIKLD5SRTNKA, seg_pos={0 0}, upload=462013]: Uploading piece...
Dec 27 14:12:57.926182492 (526h43m57.797265527s) [noise=true , address=145.79.227.30:4774   , piece_id=HDJ47XSDHU4U4L6D6AQGAGIOXR3V3AP4FMSSZJRIPQGMYLXUJPNQ, seg_pos={0 0}, upload=462013]: Uploading piece...
Dec 27 14:12:57.927422182 (526h43m57.798505217s) [noise=true , address=145.79.227.16:4921   , piece_id=Z46CVIJMCE267S7GFRZSZWCD4E74Z24CKOVKPDHJX2SQ2E6UEBHQ, seg_pos={0 0}, upload=462013]: Uploading piece...
Dec 27 14:12:57.928841094 (526h43m57.799924139s) [noise=true , address=45.91.116.13:4161    , piece_id=5DLPLIRQM3M7SQZPTVVU3CJ75FP3AYLGRUADRA7I22433UKRXNSQ, seg_pos={0 0}, upload=462013]: Uploading piece...
Dec 27 14:12:57.930256310 (526h43m57.801339345s) [noise=true , address=145.79.227.26:4676   , piece_id=MJYYF6NF6VZ5HAE365CGE2J2MDTBXQUYLFDQMLL3N5EBWSZ43NXQ, seg_pos={0 0}, upload=462013]: Uploading piece...
Dec 27 14:12:57.930779987 (526h43m57.801863022s) [noise=true , address=77.90.31.19:4872     , piece_id=6DL6JJPLSJEQZBULJAW2HRSBYYQJRIMLUWTFHOMRILM7MHO46TTQ, seg_pos={0 0}, upload=462013]: Uploading piece...
Dec 27 14:12:57.931331567 (526h43m57.802414612s) [noise=true , address=77.90.31.14:4673     , piece_id=TYJQ363AEINSWTPKNPS62ORSN45K6LDY24BSHKHCZXB5EIHQJD4A, seg_pos={0 0}, upload=462013]: Uploading piece...
Dec 27 14:12:59.232117106 (526h43m59.103200151s) [upload=462013]: Next returned nil segment
Dec 27 14:12:59.232136594 (526h43m59.103219629s) [upload=462013]: Waiting for error group managing segments...
Dec 27 14:12:59.583294709 (526h43m59.454377744s) [noise=true , address=64.49.13.26:4947     , piece_id=J6V7JSP6PLJY3LRZNN5SGWTC3F2LPNVVXWNKQBLFQIKLD5SRTNKA, seg_pos={0 0}, upload=462013]: Done uploading piece. err: <nil>
Dec 27 14:12:59.799392195 (526h43m59.67047523s) [noise=true , address=145.79.227.26:4676   , piece_id=MJYYF6NF6VZ5HAE365CGE2J2MDTBXQUYLFDQMLL3N5EBWSZ43NXQ, seg_pos={0 0}, upload=462013]: Done uploading piece. err: <nil>
Dec 27 14:12:59.805471561 (526h43m59.676554596s) [noise=true , address=145.79.227.16:4921   , piece_id=Z46CVIJMCE267S7GFRZSZWCD4E74Z24CKOVKPDHJX2SQ2E6UEBHQ, seg_pos={0 0}, upload=462013]: Done uploading piece. err: <nil>
Dec 27 14:12:59.825493146 (526h43m59.696576181s) [noise=true , address=145.79.227.46:4633   , piece_id=PYNPAJYIFPVDGYJQVHUXRTDSUBXEDYKEVKI4KPSNBWHZFLJNTSBA, seg_pos={0 0}, upload=462013]: Done uploading piece. err: <nil>
Dec 27 14:12:59.986644555 (526h43m59.85772759s) [noise=true , address=77.90.31.12:4599     , piece_id=LZ3YVYONYMFTVXOJEX43SD23SQEATWFNNMRWV2CE6YHI7OSI27JA, seg_pos={0 0}, upload=462013]: Done uploading piece. err: <nil>
Dec 27 14:13:00.089544566 (526h43m59.960627611s) [noise=true , address=45.91.116.12:4234    , piece_id=JUSY6UY7I3L43KXO3MB7O2KWDSJ7LTFRCBHIQEFR2L4BW5E5VWRQ, seg_pos={0 0}, upload=462013]: Done uploading piece. err: <nil>
Dec 27 14:13:00.094637113 (526h43m59.965720168s) [noise=true , address=77.90.31.18:4837     , piece_id=IVSHYSEVNYF5ARFGT6KKPLU7I2TPUEU2TRPU6NHW5GFPDVPF3P7Q, seg_pos={0 0}, upload=462013]: Done uploading piece. err: <nil>
Dec 27 14:13:00.309980844 (526h44m0.181063889s) [noise=true , address=77.90.31.15:4706     , piece_id=AXE72GOW3RDZQTTUCVY5H6DI6MIQAVTIQOFIMTIU45SJPEBLBABQ, seg_pos={0 0}, upload=462013]: Done uploading piece. err: <nil>
Dec 27 14:13:00.420822636 (526h44m0.291905671s) [noise=true , address=45.91.116.13:4161    , piece_id=5DLPLIRQM3M7SQZPTVVU3CJ75FP3AYLGRUADRA7I22433UKRXNSQ, seg_pos={0 0}, upload=462013]: Done uploading piece. err: <nil>
Dec 27 14:13:00.658521415 (526h44m0.52960447s) [noise=true , address=77.90.31.19:4872     , piece_id=6DL6JJPLSJEQZBULJAW2HRSBYYQJRIMLUWTFHOMRILM7MHO46TTQ, seg_pos={0 0}, upload=462013]: Done uploading piece. err: <nil>
Dec 27 14:13:00.692059858 (526h44m0.563142893s) [noise=true , address=145.79.227.44:4222   , piece_id=TO6N63KTTH567YTJNVRSRT2RTZPA4JZROYNEMRJXJC2XFRLXTM3Q, seg_pos={0 0}, upload=462013]: Done uploading piece. err: <nil>
Dec 27 14:13:00.891647316 (526h44m0.762730361s) [noise=true , address=145.79.227.34:4918   , piece_id=7KAA2LG2TGLRDOQHTJVRHJIDSDUSHBNIICMT5WCYHN5UIHIKUWRQ, seg_pos={0 0}, upload=462013]: Done uploading piece. err: <nil>
Dec 27 14:13:01.175999412 (526h44m1.047082457s) [noise=true , address=145.79.227.30:4774   , piece_id=HDJ47XSDHU4U4L6D6AQGAGIOXR3V3AP4FMSSZJRIPQGMYLXUJPNQ, seg_pos={0 0}, upload=462013]: Done uploading piece. err: <nil>
Dec 27 14:13:01.706330393 (526h44m1.577413438s) [noise=true , address=45.91.116.6:4983     , piece_id=5JTU5Y24LBUWCCJE2ALXFYMH6DQ45UCIPM6X5FDESKSUJM7CLI6Q, seg_pos={0 0}, upload=462013]: Done uploading piece. err: <nil>
Dec 27 14:13:01.706358246 (526h44m1.577441291s) [seg_pos={0 0}, upload=462013]: Segment reached optimal threshold of 14 pieces.
Dec 27 14:13:01.706673707 (526h44m1.577756752s) [noise=true , address=64.49.13.169:4705    , piece_id=5A5NYCMNFLEONT2SJGTVADCLSW5IBUBPRHNDCYVB3SMO23VV4KXQ, seg_pos={0 0}, upload=462013]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:33542->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:33542->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33542->64.49.13.169:4705: use of closed network connection
Dec 27 14:13:01.706687494 (526h44m1.577770539s) [noise=true , address=64.49.13.169:4705    , piece_id=5A5NYCMNFLEONT2SJGTVADCLSW5IBUBPRHNDCYVB3SMO23VV4KXQ, seg_pos={0 0}, upload=462013]: Long tail cancellation due to optimal threshold
Dec 27 14:13:01.706702292 (526h44m1.577785337s) [noise=true , address=77.90.31.10:4546     , piece_id=V3S7W7NUG5REGC6PRIMD4LL5ZEJMHT3ORLV5NATAMPVJ54RKYZVA, seg_pos={0 0}, upload=462013]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: read tcp 188.166.225.170:57032->77.90.31.10:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 27 14:13:01.706709184 (526h44m1.57779223s) [noise=true , address=77.90.31.10:4546     , piece_id=V3S7W7NUG5REGC6PRIMD4LL5ZEJMHT3ORLV5NATAMPVJ54RKYZVA, seg_pos={0 0}, upload=462013]: Long tail cancellation due to optimal threshold
Dec 27 14:13:01.706954321 (526h44m1.578037386s) [noise=true , address=77.90.31.14:4673     , piece_id=TYJQ363AEINSWTPKNPS62ORSN45K6LDY24BSHKHCZXB5EIHQJD4A, seg_pos={0 0}, upload=462013]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh): protocol: expected piece hash; context canceled; EOF
Dec 27 14:13:01.706965673 (526h44m1.578048729s) [noise=true , address=77.90.31.14:4673     , piece_id=TYJQ363AEINSWTPKNPS62ORSN45K6LDY24BSHKHCZXB5EIHQJD4A, seg_pos={0 0}, upload=462013]: Long tail cancellation due to optimal threshold
Dec 27 14:13:01.707148722 (526h44m1.578231777s) [noise=true , address=145.79.227.33:4901   , piece_id=WPNDZ6NIZ77VRWHQJRCZYXRLD3WJEBGUPKNSLIRQA4VJPKBQLIPQ, seg_pos={0 0}, upload=462013]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1): protocol: expected piece hash; context canceled; EOF
Dec 27 14:13:01.707158450 (526h44m1.578241495s) [noise=true , address=145.79.227.33:4901   , piece_id=WPNDZ6NIZ77VRWHQJRCZYXRLD3WJEBGUPKNSLIRQA4VJPKBQLIPQ, seg_pos={0 0}, upload=462013]: Long tail cancellation due to optimal threshold
Dec 27 14:13:01.707187896 (526h44m1.578270941s) [seg_pos={0 0}, upload=462013]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 14:13:01.707288167 (526h44m1.578371212s) [upload=462013]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 14:13:01.707299669 (526h44m1.578382754s) [upload=462013]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 14:13:02.002128984 (526h44m1.873212049s) [upload=462013]: Done uploading segments...
Dec 27 14:13:02.002158611 (526h44m1.873241666s) [upload=462013]: Upload finished. err: <nil>
Duration: 47.348246354s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
42 out of 60 is failed (threshold=6) [.X.X.X.X.X..XXX.XX.XX.X.XX.X.XXXXXXXXXXX.X..XXXX.XXXXXXX.XXX]
duration: 
5.001867568s
trace: 
https://jaeger.dev.storj.io/trace/16eee1b96b088c04
FAIL (2025-12-27 14:14:37 UTC) 
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001895411s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip: 
2 out of 60 is failed (threshold=3) [......................................XX....................]
duration: 
12.350715955s
trace: 
https://jaeger.dev.storj.io/trace/75ef8b5e6bc2819a
FAIL (2025-12-27 14:14:31 UTC) 
orbiter warning: context deadline exceeded
Duration: 12.350740242s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
20 out of 60 is failed (threshold=3) [..XX.......X....X..XX.X.XXXX..X..X.....X.XXX..X...XX........]
duration: 
6.540756246s
trace: 
https://jaeger.dev.storj.io/trace/7a3b11897bee958b
FAIL (2025-12-27 14:14:11 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 27 14:09:21.611393571 (526h40m21.482476616s) [upload=461959]: Starting upload
Dec 27 14:09:21.611546842 (526h40m21.482629877s) [upload=461959]: Done starting upload
Dec 27 14:09:21.611609071 (526h40m21.482692116s) [upload=461959]: Uploading segments...
Dec 27 14:09:21.612269158 (526h40m21.483352213s) [upload=461959]: Got next segment. Inline: false
Dec 27 14:09:21.612349201 (526h40m21.483432256s) [upload=461959]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 14:09:21.612356555 (526h40m21.4834396s) [upload=461959]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 14:09:21.856299604 (526h40m21.727382649s) [seg_pos={0 0}, upload=461959]: Begin upload segment...
Dec 27 14:09:21.856370038 (526h40m21.727453073s) [seg_pos={0 0}, upload=461959]: Done begin upload segment.
Dec 27 14:09:21.856420784 (526h40m21.727503819s) [noise=true , address=145.79.227.12:4765   , piece_id=5YKAMK34JOHYLZJT2D5WRZBY6N43346GSRFYFXHWZ6Z3AWWAQJ6A, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.856479476 (526h40m21.727562521s) [noise=true , address=145.79.227.29:4942   , piece_id=R6L233G7TZUNMLBODN4UEJH66YDWXL66B2WGAVQPHCIH5JXC74FA, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.857625017 (526h40m21.728708052s) [noise=true , address=45.91.116.13:4155    , piece_id=LBAG6T3GGWKHCOHA7T53VKSOX2O55KKQ4EGMRKDCK6PGFCTUIJVQ, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.858100663 (526h40m21.729183698s) [noise=true , address=145.79.227.10:4402   , piece_id=XUDYA3V3NLMFVKRX5ERUSILGUIOLX6NJNXWIL2PZDT6GFJMA3NDA, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.858170176 (526h40m21.729253231s) [noise=true , address=45.91.116.7:4009     , piece_id=US7DNVKAFLNLTFQB45RKVON242QU3RF3T5IELTT2WCNMUN6ZAYBQ, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.858402087 (526h40m21.729485122s) [noise=true , address=45.91.116.11:4120    , piece_id=UXEDL7GFL64HCRQ67THFOC2MKAVGYREYNQS7WEXJYGBOY3H7EYWQ, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.858794865 (526h40m21.7298779s) [noise=true , address=77.90.31.17:4773     , piece_id=5JIB2CV4VJSBBFINGJ4WCWLQN4H2T4WSEL6XDB3A755TJQKQOKYQ, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.859040964 (526h40m21.730124009s) [noise=true , address=145.79.227.19:4078   , piece_id=UAMIORNB3U6JKMHUIQRNRUNYW4T7S6YHERJG7ZVIHC5KLEXXCF6Q, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.861213120 (526h40m21.732296205s) [noise=true , address=145.79.227.45:4230   , piece_id=CPFKFQEL7BVP3V6I2LCDPVCGQACW3RDR7QT2V7XIJS77H6ZU7RIA, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.862069922 (526h40m21.733152957s) [noise=true , address=145.79.227.50:4375   , piece_id=XUAHWMEE2ZKSLFO3Z6CC7CZJZOSDSMNMDXYVGVK5ZLO2TDYZCH4Q, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.863841424 (526h40m21.734924469s) [noise=true , address=145.79.227.20:4438   , piece_id=42J67NFOF35HJMHKU2FLTASKR2TGKIY6NRBISTOIDIWBY64ACWSQ, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.867587077 (526h40m21.738670122s) [noise=true , address=145.79.227.42:4123   , piece_id=K6WUIK2EKFXSGSUXIOCSV4R3MOB2S4HPLEXOJRAKMCUC2GWYTEYA, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.869214605 (526h40m21.74029765s) [noise=true , address=45.91.116.6:4974     , piece_id=QXEEQHFYAH67J6OSDKWDVJCJV2S7ENNTQUWGV6TO5MMLTQ7JSOBQ, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.870424339 (526h40m21.741507384s) [noise=true , address=145.79.227.23:4546   , piece_id=YAGKC7A2NH6QNUCYJH3TIGQXTZZPDQ4FZTBR6LLAXSYMJTFAMI7A, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.871897163 (526h40m21.742980208s) [noise=true , address=145.79.227.17:4970   , piece_id=2SEU5CWQM23OWEPWAGUO34UBDJKT7TES2N3TZEIDIWDCCUBW5NEQ, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.872851330 (526h40m21.743934376s) [noise=true , address=145.79.227.36:4519   , piece_id=TVVXVEG5UT3CDL5AWEU4M2TQTOUTIV2RPCGBOGDCD2SRJIE4KBVA, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:23.532718349 (526h40m23.403801384s) [upload=461959]: Next returned nil segment
Dec 27 14:09:23.532735001 (526h40m23.403818047s) [upload=461959]: Waiting for error group managing segments...
Dec 27 14:09:23.862990534 (526h40m23.734073569s) [noise=true , address=145.79.227.12:4765   , piece_id=5YKAMK34JOHYLZJT2D5WRZBY6N43346GSRFYFXHWZ6Z3AWWAQJ6A, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:23.871865954 (526h40m23.742948989s) [noise=true , address=145.79.227.50:4375   , piece_id=XUAHWMEE2ZKSLFO3Z6CC7CZJZOSDSMNMDXYVGVK5ZLO2TDYZCH4Q, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:23.874552059 (526h40m23.745635104s) [noise=true , address=145.79.227.36:4519   , piece_id=TVVXVEG5UT3CDL5AWEU4M2TQTOUTIV2RPCGBOGDCD2SRJIE4KBVA, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:23.875081458 (526h40m23.746164493s) [noise=true , address=145.79.227.29:4942   , piece_id=R6L233G7TZUNMLBODN4UEJH66YDWXL66B2WGAVQPHCIH5JXC74FA, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:23.880889777 (526h40m23.751972812s) [noise=true , address=145.79.227.17:4970   , piece_id=2SEU5CWQM23OWEPWAGUO34UBDJKT7TES2N3TZEIDIWDCCUBW5NEQ, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:23.881779612 (526h40m23.752862657s) [noise=true , address=145.79.227.23:4546   , piece_id=YAGKC7A2NH6QNUCYJH3TIGQXTZZPDQ4FZTBR6LLAXSYMJTFAMI7A, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:23.888373497 (526h40m23.759456542s) [noise=true , address=145.79.227.10:4402   , piece_id=XUDYA3V3NLMFVKRX5ERUSILGUIOLX6NJNXWIL2PZDT6GFJMA3NDA, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:23.896892469 (526h40m23.767975504s) [noise=true , address=145.79.227.42:4123   , piece_id=K6WUIK2EKFXSGSUXIOCSV4R3MOB2S4HPLEXOJRAKMCUC2GWYTEYA, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:23.899060166 (526h40m23.770143202s) [noise=true , address=145.79.227.20:4438   , piece_id=42J67NFOF35HJMHKU2FLTASKR2TGKIY6NRBISTOIDIWBY64ACWSQ, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:23.899525874 (526h40m23.770608919s) [noise=true , address=145.79.227.45:4230   , piece_id=CPFKFQEL7BVP3V6I2LCDPVCGQACW3RDR7QT2V7XIJS77H6ZU7RIA, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:23.907289918 (526h40m23.778372953s) [noise=true , address=145.79.227.19:4078   , piece_id=UAMIORNB3U6JKMHUIQRNRUNYW4T7S6YHERJG7ZVIHC5KLEXXCF6Q, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:24.122094452 (526h40m23.993177497s) [noise=true , address=45.91.116.11:4120    , piece_id=UXEDL7GFL64HCRQ67THFOC2MKAVGYREYNQS7WEXJYGBOY3H7EYWQ, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:24.166811121 (526h40m24.037894166s) [noise=true , address=77.90.31.17:4773     , piece_id=5JIB2CV4VJSBBFINGJ4WCWLQN4H2T4WSEL6XDB3A755TJQKQOKYQ, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:24.166864091 (526h40m24.037947126s) [seg_pos={0 0}, upload=461959]: Segment reached optimal threshold of 13 pieces.
Dec 27 14:09:24.167067258 (526h40m24.038150303s) [noise=true , address=45.91.116.7:4009     , piece_id=US7DNVKAFLNLTFQB45RKVON242QU3RF3T5IELTT2WCNMUN6ZAYBQ, seg_pos={0 0}, upload=461959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF
Dec 27 14:09:24.167077768 (526h40m24.038160823s) [noise=true , address=45.91.116.7:4009     , piece_id=US7DNVKAFLNLTFQB45RKVON242QU3RF3T5IELTT2WCNMUN6ZAYBQ, seg_pos={0 0}, upload=461959]: Long tail cancellation due to optimal threshold
Dec 27 14:09:24.167395192 (526h40m24.038478237s) [noise=true , address=45.91.116.13:4155    , piece_id=LBAG6T3GGWKHCOHA7T53VKSOX2O55KKQ4EGMRKDCK6PGFCTUIJVQ, seg_pos={0 0}, upload=461959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF
Dec 27 14:09:24.167402437 (526h40m24.038485472s) [noise=true , address=45.91.116.13:4155    , piece_id=LBAG6T3GGWKHCOHA7T53VKSOX2O55KKQ4EGMRKDCK6PGFCTUIJVQ, seg_pos={0 0}, upload=461959]: Long tail cancellation due to optimal threshold
Dec 27 14:09:24.167513648 (526h40m24.038596694s) [noise=true , address=45.91.116.6:4974     , piece_id=QXEEQHFYAH67J6OSDKWDVJCJV2S7ENNTQUWGV6TO5MMLTQ7JSOBQ, seg_pos={0 0}, upload=461959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF
Dec 27 14:09:24.167519349 (526h40m24.038602384s) [noise=true , address=45.91.116.6:4974     , piece_id=QXEEQHFYAH67J6OSDKWDVJCJV2S7ENNTQUWGV6TO5MMLTQ7JSOBQ, seg_pos={0 0}, upload=461959]: Long tail cancellation due to optimal threshold
Dec 27 14:09:24.167532774 (526h40m24.038615809s) [seg_pos={0 0}, upload=461959]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 14:09:24.167612757 (526h40m24.038695792s) [upload=461959]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 14:09:24.167618839 (526h40m24.038701884s) [upload=461959]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 14:09:24.455792179 (526h40m24.326875214s) [upload=461959]: Done uploading segments...
Dec 27 14:09:24.455820133 (526h40m24.326903168s) [upload=461959]: Upload finished. err: <nil>
Duration: 6.540786452s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
20 out of 60 is failed (threshold=3) [...XX.......X....X..XX.X.XXXX..X..X.....X.XXX..X...XX.......]
duration: 
18.107773501s
trace: 
https://jaeger.dev.storj.io/trace/7a3b11897bee958b
FAIL (2025-12-27 14:14:03 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 27 14:09:21.611393571 (526h40m21.482476616s) [upload=461959]: Starting upload
Dec 27 14:09:21.611546842 (526h40m21.482629877s) [upload=461959]: Done starting upload
Dec 27 14:09:21.611609071 (526h40m21.482692116s) [upload=461959]: Uploading segments...
Dec 27 14:09:21.612269158 (526h40m21.483352213s) [upload=461959]: Got next segment. Inline: false
Dec 27 14:09:21.612349201 (526h40m21.483432256s) [upload=461959]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 14:09:21.612356555 (526h40m21.4834396s) [upload=461959]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 14:09:21.856299604 (526h40m21.727382649s) [seg_pos={0 0}, upload=461959]: Begin upload segment...
Dec 27 14:09:21.856370038 (526h40m21.727453073s) [seg_pos={0 0}, upload=461959]: Done begin upload segment.
Dec 27 14:09:21.856420784 (526h40m21.727503819s) [noise=true , address=145.79.227.12:4765   , piece_id=5YKAMK34JOHYLZJT2D5WRZBY6N43346GSRFYFXHWZ6Z3AWWAQJ6A, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.856479476 (526h40m21.727562521s) [noise=true , address=145.79.227.29:4942   , piece_id=R6L233G7TZUNMLBODN4UEJH66YDWXL66B2WGAVQPHCIH5JXC74FA, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.857625017 (526h40m21.728708052s) [noise=true , address=45.91.116.13:4155    , piece_id=LBAG6T3GGWKHCOHA7T53VKSOX2O55KKQ4EGMRKDCK6PGFCTUIJVQ, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.858100663 (526h40m21.729183698s) [noise=true , address=145.79.227.10:4402   , piece_id=XUDYA3V3NLMFVKRX5ERUSILGUIOLX6NJNXWIL2PZDT6GFJMA3NDA, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.858170176 (526h40m21.729253231s) [noise=true , address=45.91.116.7:4009     , piece_id=US7DNVKAFLNLTFQB45RKVON242QU3RF3T5IELTT2WCNMUN6ZAYBQ, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.858402087 (526h40m21.729485122s) [noise=true , address=45.91.116.11:4120    , piece_id=UXEDL7GFL64HCRQ67THFOC2MKAVGYREYNQS7WEXJYGBOY3H7EYWQ, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.858794865 (526h40m21.7298779s) [noise=true , address=77.90.31.17:4773     , piece_id=5JIB2CV4VJSBBFINGJ4WCWLQN4H2T4WSEL6XDB3A755TJQKQOKYQ, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.859040964 (526h40m21.730124009s) [noise=true , address=145.79.227.19:4078   , piece_id=UAMIORNB3U6JKMHUIQRNRUNYW4T7S6YHERJG7ZVIHC5KLEXXCF6Q, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.861213120 (526h40m21.732296205s) [noise=true , address=145.79.227.45:4230   , piece_id=CPFKFQEL7BVP3V6I2LCDPVCGQACW3RDR7QT2V7XIJS77H6ZU7RIA, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.862069922 (526h40m21.733152957s) [noise=true , address=145.79.227.50:4375   , piece_id=XUAHWMEE2ZKSLFO3Z6CC7CZJZOSDSMNMDXYVGVK5ZLO2TDYZCH4Q, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.863841424 (526h40m21.734924469s) [noise=true , address=145.79.227.20:4438   , piece_id=42J67NFOF35HJMHKU2FLTASKR2TGKIY6NRBISTOIDIWBY64ACWSQ, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.867587077 (526h40m21.738670122s) [noise=true , address=145.79.227.42:4123   , piece_id=K6WUIK2EKFXSGSUXIOCSV4R3MOB2S4HPLEXOJRAKMCUC2GWYTEYA, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.869214605 (526h40m21.74029765s) [noise=true , address=45.91.116.6:4974     , piece_id=QXEEQHFYAH67J6OSDKWDVJCJV2S7ENNTQUWGV6TO5MMLTQ7JSOBQ, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.870424339 (526h40m21.741507384s) [noise=true , address=145.79.227.23:4546   , piece_id=YAGKC7A2NH6QNUCYJH3TIGQXTZZPDQ4FZTBR6LLAXSYMJTFAMI7A, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.871897163 (526h40m21.742980208s) [noise=true , address=145.79.227.17:4970   , piece_id=2SEU5CWQM23OWEPWAGUO34UBDJKT7TES2N3TZEIDIWDCCUBW5NEQ, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.872851330 (526h40m21.743934376s) [noise=true , address=145.79.227.36:4519   , piece_id=TVVXVEG5UT3CDL5AWEU4M2TQTOUTIV2RPCGBOGDCD2SRJIE4KBVA, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:23.532718349 (526h40m23.403801384s) [upload=461959]: Next returned nil segment
Dec 27 14:09:23.532735001 (526h40m23.403818047s) [upload=461959]: Waiting for error group managing segments...
Dec 27 14:09:23.862990534 (526h40m23.734073569s) [noise=true , address=145.79.227.12:4765   , piece_id=5YKAMK34JOHYLZJT2D5WRZBY6N43346GSRFYFXHWZ6Z3AWWAQJ6A, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:23.871865954 (526h40m23.742948989s) [noise=true , address=145.79.227.50:4375   , piece_id=XUAHWMEE2ZKSLFO3Z6CC7CZJZOSDSMNMDXYVGVK5ZLO2TDYZCH4Q, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:23.874552059 (526h40m23.745635104s) [noise=true , address=145.79.227.36:4519   , piece_id=TVVXVEG5UT3CDL5AWEU4M2TQTOUTIV2RPCGBOGDCD2SRJIE4KBVA, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:23.875081458 (526h40m23.746164493s) [noise=true , address=145.79.227.29:4942   , piece_id=R6L233G7TZUNMLBODN4UEJH66YDWXL66B2WGAVQPHCIH5JXC74FA, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:23.880889777 (526h40m23.751972812s) [noise=true , address=145.79.227.17:4970   , piece_id=2SEU5CWQM23OWEPWAGUO34UBDJKT7TES2N3TZEIDIWDCCUBW5NEQ, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:23.881779612 (526h40m23.752862657s) [noise=true , address=145.79.227.23:4546   , piece_id=YAGKC7A2NH6QNUCYJH3TIGQXTZZPDQ4FZTBR6LLAXSYMJTFAMI7A, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:23.888373497 (526h40m23.759456542s) [noise=true , address=145.79.227.10:4402   , piece_id=XUDYA3V3NLMFVKRX5ERUSILGUIOLX6NJNXWIL2PZDT6GFJMA3NDA, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:23.896892469 (526h40m23.767975504s) [noise=true , address=145.79.227.42:4123   , piece_id=K6WUIK2EKFXSGSUXIOCSV4R3MOB2S4HPLEXOJRAKMCUC2GWYTEYA, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:23.899060166 (526h40m23.770143202s) [noise=true , address=145.79.227.20:4438   , piece_id=42J67NFOF35HJMHKU2FLTASKR2TGKIY6NRBISTOIDIWBY64ACWSQ, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:23.899525874 (526h40m23.770608919s) [noise=true , address=145.79.227.45:4230   , piece_id=CPFKFQEL7BVP3V6I2LCDPVCGQACW3RDR7QT2V7XIJS77H6ZU7RIA, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:23.907289918 (526h40m23.778372953s) [noise=true , address=145.79.227.19:4078   , piece_id=UAMIORNB3U6JKMHUIQRNRUNYW4T7S6YHERJG7ZVIHC5KLEXXCF6Q, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:24.122094452 (526h40m23.993177497s) [noise=true , address=45.91.116.11:4120    , piece_id=UXEDL7GFL64HCRQ67THFOC2MKAVGYREYNQS7WEXJYGBOY3H7EYWQ, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:24.166811121 (526h40m24.037894166s) [noise=true , address=77.90.31.17:4773     , piece_id=5JIB2CV4VJSBBFINGJ4WCWLQN4H2T4WSEL6XDB3A755TJQKQOKYQ, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:24.166864091 (526h40m24.037947126s) [seg_pos={0 0}, upload=461959]: Segment reached optimal threshold of 13 pieces.
Dec 27 14:09:24.167067258 (526h40m24.038150303s) [noise=true , address=45.91.116.7:4009     , piece_id=US7DNVKAFLNLTFQB45RKVON242QU3RF3T5IELTT2WCNMUN6ZAYBQ, seg_pos={0 0}, upload=461959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF
Dec 27 14:09:24.167077768 (526h40m24.038160823s) [noise=true , address=45.91.116.7:4009     , piece_id=US7DNVKAFLNLTFQB45RKVON242QU3RF3T5IELTT2WCNMUN6ZAYBQ, seg_pos={0 0}, upload=461959]: Long tail cancellation due to optimal threshold
Dec 27 14:09:24.167395192 (526h40m24.038478237s) [noise=true , address=45.91.116.13:4155    , piece_id=LBAG6T3GGWKHCOHA7T53VKSOX2O55KKQ4EGMRKDCK6PGFCTUIJVQ, seg_pos={0 0}, upload=461959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF
Dec 27 14:09:24.167402437 (526h40m24.038485472s) [noise=true , address=45.91.116.13:4155    , piece_id=LBAG6T3GGWKHCOHA7T53VKSOX2O55KKQ4EGMRKDCK6PGFCTUIJVQ, seg_pos={0 0}, upload=461959]: Long tail cancellation due to optimal threshold
Dec 27 14:09:24.167513648 (526h40m24.038596694s) [noise=true , address=45.91.116.6:4974     , piece_id=QXEEQHFYAH67J6OSDKWDVJCJV2S7ENNTQUWGV6TO5MMLTQ7JSOBQ, seg_pos={0 0}, upload=461959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF
Dec 27 14:09:24.167519349 (526h40m24.038602384s) [noise=true , address=45.91.116.6:4974     , piece_id=QXEEQHFYAH67J6OSDKWDVJCJV2S7ENNTQUWGV6TO5MMLTQ7JSOBQ, seg_pos={0 0}, upload=461959]: Long tail cancellation due to optimal threshold
Dec 27 14:09:24.167532774 (526h40m24.038615809s) [seg_pos={0 0}, upload=461959]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 14:09:24.167612757 (526h40m24.038695792s) [upload=461959]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 14:09:24.167618839 (526h40m24.038701884s) [upload=461959]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 14:09:24.455792179 (526h40m24.326875214s) [upload=461959]: Done uploading segments...
Dec 27 14:09:24.455820133 (526h40m24.326903168s) [upload=461959]: Upload finished. err: <nil>
Duration: 18.107801223s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
4 out of 60 is failed (threshold=3) [..........X.............XX.................................X]
duration: 
1m0.002449852s
trace: 
https://jaeger.dev.storj.io/trace/30098651d72a60c0
FAIL (2025-12-27 14:13:56 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 122trJbDHuqeBMQHZT97HP2PgkeuLyUiDiuRm4nayu6ZTH7tat6, Piece ID: 5JTU5Y24LBUWCCJE2ALXFYMH6DQ45UCIPM6X5FDESKSUJM7CLI6Q) context deadline exceeded; piecestore: (Node ID: 12TaXapd6q5Vh1CQ5tQP98iQqVY99HFdGGv8CebGq49D9aKR9n2, Piece ID: 5DLPLIRQM3M7SQZPTVVU3CJ75FP3AYLGRUADRA7I22433UKRXNSQ) context deadline exceeded; piecestore: (Node ID: 1mmKSzdU8tViT7WLrppbMMsbAzB2MetMSoLj6ToyLHyQ5RaNjA, Piece ID: JUSY6UY7I3L43KXO3MB7O2KWDSJ7LTFRCBHIQEFR2L4BW5E5VWRQ) context deadline exceeded
======LOGS======
AVTIQOFIMTIU45SJPEBLBABQ, seg_pos={0 0}, upload=462013]: Uploading piece...
Dec 27 14:12:57.923596308 (526h43m57.794679343s) [noise=true , address=77.90.31.12:4599     , piece_id=LZ3YVYONYMFTVXOJEX43SD23SQEATWFNNMRWV2CE6YHI7OSI27JA, seg_pos={0 0}, upload=462013]: Uploading piece...
Dec 27 14:12:57.923971432 (526h43m57.795054467s) [noise=true , address=45.91.116.6:4983     , piece_id=5JTU5Y24LBUWCCJE2ALXFYMH6DQ45UCIPM6X5FDESKSUJM7CLI6Q, seg_pos={0 0}, upload=462013]: Uploading piece...
Dec 27 14:12:57.924200558 (526h43m57.795283593s) [noise=true , address=77.90.31.18:4837     , piece_id=IVSHYSEVNYF5ARFGT6KKPLU7I2TPUEU2TRPU6NHW5GFPDVPF3P7Q, seg_pos={0 0}, upload=462013]: Uploading piece...
Dec 27 14:12:57.924293756 (526h43m57.795376801s) [noise=true , address=77.90.31.10:4546     , piece_id=V3S7W7NUG5REGC6PRIMD4LL5ZEJMHT3ORLV5NATAMPVJ54RKYZVA, seg_pos={0 0}, upload=462013]: Uploading piece...
Dec 27 14:12:57.924668840 (526h43m57.795751875s) [noise=true , address=145.79.227.34:4918   , piece_id=7KAA2LG2TGLRDOQHTJVRHJIDSDUSHBNIICMT5WCYHN5UIHIKUWRQ, seg_pos={0 0}, upload=462013]: Uploading piece...
Dec 27 14:12:57.924819297 (526h43m57.795902342s) [noise=true , address=45.91.116.12:4234    , piece_id=JUSY6UY7I3L43KXO3MB7O2KWDSJ7LTFRCBHIQEFR2L4BW5E5VWRQ, seg_pos={0 0}, upload=462013]: Uploading piece...
Dec 27 14:12:57.925054945 (526h43m57.79613799s) [noise=true , address=145.79.227.46:4633   , piece_id=PYNPAJYIFPVDGYJQVHUXRTDSUBXEDYKEVKI4KPSNBWHZFLJNTSBA, seg_pos={0 0}, upload=462013]: Uploading piece...
Dec 27 14:12:57.925257120 (526h43m57.796340155s) [noise=true , address=64.49.13.169:4705    , piece_id=5A5NYCMNFLEONT2SJGTVADCLSW5IBUBPRHNDCYVB3SMO23VV4KXQ, seg_pos={0 0}, upload=462013]: Uploading piece...
Dec 27 14:12:57.925641512 (526h43m57.796724557s) [noise=true , address=64.49.13.26:4947     , piece_id=J6V7JSP6PLJY3LRZNN5SGWTC3F2LPNVVXWNKQBLFQIKLD5SRTNKA, seg_pos={0 0}, upload=462013]: Uploading piece...
Dec 27 14:12:57.926182492 (526h43m57.797265527s) [noise=true , address=145.79.227.30:4774   , piece_id=HDJ47XSDHU4U4L6D6AQGAGIOXR3V3AP4FMSSZJRIPQGMYLXUJPNQ, seg_pos={0 0}, upload=462013]: Uploading piece...
Dec 27 14:12:57.927422182 (526h43m57.798505217s) [noise=true , address=145.79.227.16:4921   , piece_id=Z46CVIJMCE267S7GFRZSZWCD4E74Z24CKOVKPDHJX2SQ2E6UEBHQ, seg_pos={0 0}, upload=462013]: Uploading piece...
Dec 27 14:12:57.928841094 (526h43m57.799924139s) [noise=true , address=45.91.116.13:4161    , piece_id=5DLPLIRQM3M7SQZPTVVU3CJ75FP3AYLGRUADRA7I22433UKRXNSQ, seg_pos={0 0}, upload=462013]: Uploading piece...
Dec 27 14:12:57.930256310 (526h43m57.801339345s) [noise=true , address=145.79.227.26:4676   , piece_id=MJYYF6NF6VZ5HAE365CGE2J2MDTBXQUYLFDQMLL3N5EBWSZ43NXQ, seg_pos={0 0}, upload=462013]: Uploading piece...
Dec 27 14:12:57.930779987 (526h43m57.801863022s) [noise=true , address=77.90.31.19:4872     , piece_id=6DL6JJPLSJEQZBULJAW2HRSBYYQJRIMLUWTFHOMRILM7MHO46TTQ, seg_pos={0 0}, upload=462013]: Uploading piece...
Dec 27 14:12:57.931331567 (526h43m57.802414612s) [noise=true , address=77.90.31.14:4673     , piece_id=TYJQ363AEINSWTPKNPS62ORSN45K6LDY24BSHKHCZXB5EIHQJD4A, seg_pos={0 0}, upload=462013]: Uploading piece...
Dec 27 14:12:59.232117106 (526h43m59.103200151s) [upload=462013]: Next returned nil segment
Dec 27 14:12:59.232136594 (526h43m59.103219629s) [upload=462013]: Waiting for error group managing segments...
Dec 27 14:12:59.583294709 (526h43m59.454377744s) [noise=true , address=64.49.13.26:4947     , piece_id=J6V7JSP6PLJY3LRZNN5SGWTC3F2LPNVVXWNKQBLFQIKLD5SRTNKA, seg_pos={0 0}, upload=462013]: Done uploading piece. err: <nil>
Dec 27 14:12:59.799392195 (526h43m59.67047523s) [noise=true , address=145.79.227.26:4676   , piece_id=MJYYF6NF6VZ5HAE365CGE2J2MDTBXQUYLFDQMLL3N5EBWSZ43NXQ, seg_pos={0 0}, upload=462013]: Done uploading piece. err: <nil>
Dec 27 14:12:59.805471561 (526h43m59.676554596s) [noise=true , address=145.79.227.16:4921   , piece_id=Z46CVIJMCE267S7GFRZSZWCD4E74Z24CKOVKPDHJX2SQ2E6UEBHQ, seg_pos={0 0}, upload=462013]: Done uploading piece. err: <nil>
Dec 27 14:12:59.825493146 (526h43m59.696576181s) [noise=true , address=145.79.227.46:4633   , piece_id=PYNPAJYIFPVDGYJQVHUXRTDSUBXEDYKEVKI4KPSNBWHZFLJNTSBA, seg_pos={0 0}, upload=462013]: Done uploading piece. err: <nil>
Dec 27 14:12:59.986644555 (526h43m59.85772759s) [noise=true , address=77.90.31.12:4599     , piece_id=LZ3YVYONYMFTVXOJEX43SD23SQEATWFNNMRWV2CE6YHI7OSI27JA, seg_pos={0 0}, upload=462013]: Done uploading piece. err: <nil>
Dec 27 14:13:00.089544566 (526h43m59.960627611s) [noise=true , address=45.91.116.12:4234    , piece_id=JUSY6UY7I3L43KXO3MB7O2KWDSJ7LTFRCBHIQEFR2L4BW5E5VWRQ, seg_pos={0 0}, upload=462013]: Done uploading piece. err: <nil>
Dec 27 14:13:00.094637113 (526h43m59.965720168s) [noise=true , address=77.90.31.18:4837     , piece_id=IVSHYSEVNYF5ARFGT6KKPLU7I2TPUEU2TRPU6NHW5GFPDVPF3P7Q, seg_pos={0 0}, upload=462013]: Done uploading piece. err: <nil>
Dec 27 14:13:00.309980844 (526h44m0.181063889s) [noise=true , address=77.90.31.15:4706     , piece_id=AXE72GOW3RDZQTTUCVY5H6DI6MIQAVTIQOFIMTIU45SJPEBLBABQ, seg_pos={0 0}, upload=462013]: Done uploading piece. err: <nil>
Dec 27 14:13:00.420822636 (526h44m0.291905671s) [noise=true , address=45.91.116.13:4161    , piece_id=5DLPLIRQM3M7SQZPTVVU3CJ75FP3AYLGRUADRA7I22433UKRXNSQ, seg_pos={0 0}, upload=462013]: Done uploading piece. err: <nil>
Dec 27 14:13:00.658521415 (526h44m0.52960447s) [noise=true , address=77.90.31.19:4872     , piece_id=6DL6JJPLSJEQZBULJAW2HRSBYYQJRIMLUWTFHOMRILM7MHO46TTQ, seg_pos={0 0}, upload=462013]: Done uploading piece. err: <nil>
Dec 27 14:13:00.692059858 (526h44m0.563142893s) [noise=true , address=145.79.227.44:4222   , piece_id=TO6N63KTTH567YTJNVRSRT2RTZPA4JZROYNEMRJXJC2XFRLXTM3Q, seg_pos={0 0}, upload=462013]: Done uploading piece. err: <nil>
Dec 27 14:13:00.891647316 (526h44m0.762730361s) [noise=true , address=145.79.227.34:4918   , piece_id=7KAA2LG2TGLRDOQHTJVRHJIDSDUSHBNIICMT5WCYHN5UIHIKUWRQ, seg_pos={0 0}, upload=462013]: Done uploading piece. err: <nil>
Dec 27 14:13:01.175999412 (526h44m1.047082457s) [noise=true , address=145.79.227.30:4774   , piece_id=HDJ47XSDHU4U4L6D6AQGAGIOXR3V3AP4FMSSZJRIPQGMYLXUJPNQ, seg_pos={0 0}, upload=462013]: Done uploading piece. err: <nil>
Dec 27 14:13:01.706330393 (526h44m1.577413438s) [noise=true , address=45.91.116.6:4983     , piece_id=5JTU5Y24LBUWCCJE2ALXFYMH6DQ45UCIPM6X5FDESKSUJM7CLI6Q, seg_pos={0 0}, upload=462013]: Done uploading piece. err: <nil>
Dec 27 14:13:01.706358246 (526h44m1.577441291s) [seg_pos={0 0}, upload=462013]: Segment reached optimal threshold of 14 pieces.
Dec 27 14:13:01.706673707 (526h44m1.577756752s) [noise=true , address=64.49.13.169:4705    , piece_id=5A5NYCMNFLEONT2SJGTVADCLSW5IBUBPRHNDCYVB3SMO23VV4KXQ, seg_pos={0 0}, upload=462013]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:33542->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:33542->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33542->64.49.13.169:4705: use of closed network connection
Dec 27 14:13:01.706687494 (526h44m1.577770539s) [noise=true , address=64.49.13.169:4705    , piece_id=5A5NYCMNFLEONT2SJGTVADCLSW5IBUBPRHNDCYVB3SMO23VV4KXQ, seg_pos={0 0}, upload=462013]: Long tail cancellation due to optimal threshold
Dec 27 14:13:01.706702292 (526h44m1.577785337s) [noise=true , address=77.90.31.10:4546     , piece_id=V3S7W7NUG5REGC6PRIMD4LL5ZEJMHT3ORLV5NATAMPVJ54RKYZVA, seg_pos={0 0}, upload=462013]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: read tcp 188.166.225.170:57032->77.90.31.10:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 27 14:13:01.706709184 (526h44m1.57779223s) [noise=true , address=77.90.31.10:4546     , piece_id=V3S7W7NUG5REGC6PRIMD4LL5ZEJMHT3ORLV5NATAMPVJ54RKYZVA, seg_pos={0 0}, upload=462013]: Long tail cancellation due to optimal threshold
Dec 27 14:13:01.706954321 (526h44m1.578037386s) [noise=true , address=77.90.31.14:4673     , piece_id=TYJQ363AEINSWTPKNPS62ORSN45K6LDY24BSHKHCZXB5EIHQJD4A, seg_pos={0 0}, upload=462013]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh): protocol: expected piece hash; context canceled; EOF
Dec 27 14:13:01.706965673 (526h44m1.578048729s) [noise=true , address=77.90.31.14:4673     , piece_id=TYJQ363AEINSWTPKNPS62ORSN45K6LDY24BSHKHCZXB5EIHQJD4A, seg_pos={0 0}, upload=462013]: Long tail cancellation due to optimal threshold
Dec 27 14:13:01.707148722 (526h44m1.578231777s) [noise=true , address=145.79.227.33:4901   , piece_id=WPNDZ6NIZ77VRWHQJRCZYXRLD3WJEBGUPKNSLIRQA4VJPKBQLIPQ, seg_pos={0 0}, upload=462013]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1): protocol: expected piece hash; context canceled; EOF
Dec 27 14:13:01.707158450 (526h44m1.578241495s) [noise=true , address=145.79.227.33:4901   , piece_id=WPNDZ6NIZ77VRWHQJRCZYXRLD3WJEBGUPKNSLIRQA4VJPKBQLIPQ, seg_pos={0 0}, upload=462013]: Long tail cancellation due to optimal threshold
Dec 27 14:13:01.707187896 (526h44m1.578270941s) [seg_pos={0 0}, upload=462013]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 14:13:01.707288167 (526h44m1.578371212s) [upload=462013]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 14:13:01.707299669 (526h44m1.578382754s) [upload=462013]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 14:13:02.002128984 (526h44m1.873212049s) [upload=462013]: Done uploading segments...
Dec 27 14:13:02.002158611 (526h44m1.873241666s) [upload=462013]: Upload finished. err: <nil>
Duration: 1m0.002464449s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip: 
2 out of 60 is failed (threshold=3) [.......................................XX...................]
duration: 
14.455635364s
trace: 
https://jaeger.dev.storj.io/trace/75ef8b5e6bc2819a
FAIL (2025-12-27 14:13:56 UTC) 
orbiter warning: context deadline exceeded
Duration: 14.455658146s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
41 out of 60 is failed (threshold=6) [..X.X.X.X.X..XXX.XX.XX.X.XX.X.XXXXXXXXXXX.X..XXXX.XXXXXXX.XX]
duration: 
5.001698733s
trace: 
https://jaeger.dev.storj.io/trace/3933cdd66aab1fad
FAIL (2025-12-27 14:13:32 UTC) 
orbiter warning: context deadline exceeded
Duration: 5.001725846s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
40 out of 60 is failed (threshold=6) [...X.X.X.X.X..XXX.XX.XX.X.XX.X.XXXXXXXXXXX.X..XXXX.XXXXXXX.X]
duration: 
5.001534151s
trace: 
https://jaeger.dev.storj.io/trace/64da31e3d1052f13
FAIL (2025-12-27 14:13:03 UTC) 
orbiter warning: context deadline exceeded
Duration: 5.001557405s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
4 out of 60 is failed (threshold=3) [X..........X.............XX.................................]
duration: 
56.006254701s
trace: 
https://jaeger.dev.storj.io/trace/3ed1e657d506c4ee
FAIL (2025-12-27 14:12:56 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, Piece ID: HEMLEGWLVMK4LG5DM3PUWU5HSUEHKBECYOMHUXIQRGZGNXPXFLBQ) context deadline exceeded; piecestore: (Node ID: 12wXqePK1kK7iBo6wHcbqkRmkED8LGJ3rzQNJAWa14cHYhx1oyB, Piece ID: M5NU2MR5LQWOKNLVKKEJXMJNVEIZVDDNIZHWNN7XXHVTJRS2OQNA) context deadline exceeded; piecestore: (Node ID: 12ATtVD3XyFxzEqwQpSF4t1MWygqEa4Y1CkNj1s43eud8UtuSmx, Piece ID: ET7O467BME3OED32PFLJKYVIWZRDF6WMPXUT34OFWSPXLSMKKOWQ) context deadline exceeded; piecestore: (Node ID: 12UjFkFn2UEvr57LDorcijDCvkBkkKitgZEFx6Yqfk5gMStQvJG, Piece ID: SF3IKAIYXQR4YPWNVXT66SEHVM54R3K3VJHA6L5RBOU3DITOR5LQ) context deadline exceeded
======LOGS======
) [noise=true , address=77.90.31.16:4739     , piece_id=H6RNS5VOYTMWX2N6GLV2KOQDPFPCBWPDNLINB7SXDJWHMJL3GYQQ, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.319646322 (526h14m34.190729357s) [noise=true , address=77.90.31.21:4899     , piece_id=SF3IKAIYXQR4YPWNVXT66SEHVM54R3K3VJHA6L5RBOU3DITOR5LQ, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.319933248 (526h14m34.191016283s) [noise=true , address=45.91.116.12:4234    , piece_id=Y2FIVYLZBQAYBO2TDGIZ4RDQHQJ3WOQH565A7ESIKANQQTQDFQZQ, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.320400588 (526h14m34.191483633s) [noise=true , address=145.79.227.29:4952   , piece_id=VXZLF4OGRIR3MYQMMLKLTD235A55F4YTHB3D4YCVCQUL4FZTSFJQ, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.320407341 (526h14m34.191490386s) [noise=true , address=145.79.227.46:4621   , piece_id=XZ4MKAZAGQKVEJJGSZ5QGT4AT3MFXXNMXICVFCI6SP3E4EVEHZGQ, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.320675030 (526h14m34.191758075s) [noise=true , address=45.91.116.10:4113    , piece_id=OFHVWJWEHSVOOONDLXBZY72VB5G6FFXH2NXL45KY7TJ6CPVZ5UTQ, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.320802022 (526h14m34.191885067s) [noise=true , address=77.90.31.11:4569     , piece_id=YC2Q3HPJDVY3NEDZNC2Z4KVN6S6DDDGRARASF7KALFAOQ335A7HA, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.321169643 (526h14m34.192252678s) [noise=true , address=77.90.31.19:4869     , piece_id=HEMLEGWLVMK4LG5DM3PUWU5HSUEHKBECYOMHUXIQRGZGNXPXFLBQ, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.321443483 (526h14m34.192526518s) [noise=true , address=45.91.116.14:4202    , piece_id=M5NU2MR5LQWOKNLVKKEJXMJNVEIZVDDNIZHWNN7XXHVTJRS2OQNA, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.321731813 (526h14m34.192814848s) [noise=true , address=77.90.31.18:4819     , piece_id=YS4DXTHLGCPFOKKLBA2DK2BVOAPTQEGVOPBSRCLFV26SVP6YFHGA, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.322221776 (526h14m34.193304811s) [noise=true , address=77.90.31.15:4708     , piece_id=3NX6E54XMZXQAQQE27LTWUSEF5GAIF2XOPQFFAP7LOA2PL7KDL3Q, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.322547776 (526h14m34.193630811s) [noise=true , address=145.79.227.40:4079   , piece_id=237LYFTAG7G4HMIUKFZDDUTYJC7AAHHXTO5EIO46UEHY24BOVB3A, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.322908093 (526h14m34.193991138s) [noise=true , address=145.79.227.29:4975   , piece_id=DAMNBVP7DRSW5OAGYRRP4K5SNJ2WEUT7IAXO77KWRHLQEMBNERZQ, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.323267968 (526h14m34.194351003s) [noise=true , address=145.79.227.30:4787   , piece_id=OB2OTY3ZEHIUFYM6FY7HIS7HQ353Y2HL6XU4476HXOD26QVA4DPQ, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.323618165 (526h14m34.19470121s) [noise=true , address=64.49.13.169:4710    , piece_id=XIZKYK2E5FQJ37LYN535M7GXBBKXUQXFQWVMBWIBFDOAIIFGUAJA, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.324084893 (526h14m34.195167928s) [noise=true , address=193.149.164.143:4694 , piece_id=ET7O467BME3OED32PFLJKYVIWZRDF6WMPXUT34OFWSPXLSMKKOWQ, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:35.783540136 (526h14m35.654623181s) [upload=461595]: Next returned nil segment
Dec 27 13:43:35.783555856 (526h14m35.654638891s) [upload=461595]: Waiting for error group managing segments...
Dec 27 13:43:36.192287250 (526h14m36.063370296s) [noise=true , address=145.79.227.29:4952   , piece_id=VXZLF4OGRIR3MYQMMLKLTD235A55F4YTHB3D4YCVCQUL4FZTSFJQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:36.200200289 (526h14m36.071283334s) [noise=true , address=145.79.227.46:4621   , piece_id=XZ4MKAZAGQKVEJJGSZ5QGT4AT3MFXXNMXICVFCI6SP3E4EVEHZGQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:36.238087532 (526h14m36.109170577s) [noise=true , address=145.79.227.29:4975   , piece_id=DAMNBVP7DRSW5OAGYRRP4K5SNJ2WEUT7IAXO77KWRHLQEMBNERZQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:36.379631346 (526h14m36.250714391s) [noise=true , address=77.90.31.19:4869     , piece_id=HEMLEGWLVMK4LG5DM3PUWU5HSUEHKBECYOMHUXIQRGZGNXPXFLBQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:36.382945286 (526h14m36.254028321s) [noise=true , address=145.79.227.34:4909   , piece_id=Z6RF5PSOGOZMOXPQYXHMXGFXIELVXXVU4NXXRTAQCIELQXLMVXBQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:36.400290103 (526h14m36.271373138s) [noise=true , address=77.90.31.18:4819     , piece_id=YS4DXTHLGCPFOKKLBA2DK2BVOAPTQEGVOPBSRCLFV26SVP6YFHGA, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:36.436707208 (526h14m36.307790243s) [noise=true , address=45.91.116.14:4202    , piece_id=M5NU2MR5LQWOKNLVKKEJXMJNVEIZVDDNIZHWNN7XXHVTJRS2OQNA, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:36.487651475 (526h14m36.35873451s) [noise=true , address=45.91.116.12:4234    , piece_id=Y2FIVYLZBQAYBO2TDGIZ4RDQHQJ3WOQH565A7ESIKANQQTQDFQZQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:36.500850769 (526h14m36.371933814s) [noise=true , address=77.90.31.11:4569     , piece_id=YC2Q3HPJDVY3NEDZNC2Z4KVN6S6DDDGRARASF7KALFAOQ335A7HA, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:36.518828551 (526h14m36.389911586s) [noise=true , address=77.90.31.15:4708     , piece_id=3NX6E54XMZXQAQQE27LTWUSEF5GAIF2XOPQFFAP7LOA2PL7KDL3Q, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:37.051607243 (526h14m36.922690289s) [noise=true , address=145.79.227.44:4228   , piece_id=6SAYEAOTTCOUFPU75C7YRIULWJGXL4PK2YKCKFB47EUBWMRQQQOQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:37.096826909 (526h14m36.967909954s) [noise=true , address=145.79.227.30:4787   , piece_id=OB2OTY3ZEHIUFYM6FY7HIS7HQ353Y2HL6XU4476HXOD26QVA4DPQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:37.140738995 (526h14m37.01182203s) [noise=true , address=77.90.31.21:4899     , piece_id=SF3IKAIYXQR4YPWNVXT66SEHVM54R3K3VJHA6L5RBOU3DITOR5LQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:37.323613990 (526h14m37.194697035s) [noise=true , address=193.149.164.143:4694 , piece_id=ET7O467BME3OED32PFLJKYVIWZRDF6WMPXUT34OFWSPXLSMKKOWQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:37.323649618 (526h14m37.194732663s) [seg_pos={0 0}, upload=461595]: Segment reached optimal threshold of 14 pieces.
Dec 27 13:43:37.323852865 (526h14m37.1949359s) [noise=true , address=45.91.116.10:4113    , piece_id=OFHVWJWEHSVOOONDLXBZY72VB5G6FFXH2NXL45KY7TJ6CPVZ5UTQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12puRMny4nbgv2by4tAbXcBBSgG2d6BfwcDDRDcFzEdjSVUJxtd): protocol: expected piece hash; context canceled; EOF
Dec 27 13:43:37.323860349 (526h14m37.194943384s) [noise=true , address=45.91.116.10:4113    , piece_id=OFHVWJWEHSVOOONDLXBZY72VB5G6FFXH2NXL45KY7TJ6CPVZ5UTQ, seg_pos={0 0}, upload=461595]: Long tail cancellation due to optimal threshold
Dec 27 13:43:37.324005135 (526h14m37.1950882s) [noise=true , address=145.79.227.40:4079   , piece_id=237LYFTAG7G4HMIUKFZDDUTYJC7AAHHXTO5EIO46UEHY24BOVB3A, seg_pos={0 0}, upload=461595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RYKzn5zVSmxVbWwnBdX8oqtG1WE2MhPeEiWLf5dZeFAut6s2Y): protocol: expected piece hash; context canceled; EOF
Dec 27 13:43:37.324022659 (526h14m37.195105704s) [noise=true , address=145.79.227.40:4079   , piece_id=237LYFTAG7G4HMIUKFZDDUTYJC7AAHHXTO5EIO46UEHY24BOVB3A, seg_pos={0 0}, upload=461595]: Long tail cancellation due to optimal threshold
Dec 27 13:43:37.324143378 (526h14m37.195226413s) [noise=true , address=77.90.31.16:4739     , piece_id=H6RNS5VOYTMWX2N6GLV2KOQDPFPCBWPDNLINB7SXDJWHMJL3GYQQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129B3N7yjJvwCzoVLTqWsPbcLQ5TQUyAzg1RCkbSAVKPWYTG8eb): protocol: expected piece hash; context canceled; EOF
Dec 27 13:43:37.324154520 (526h14m37.195237555s) [noise=true , address=77.90.31.16:4739     , piece_id=H6RNS5VOYTMWX2N6GLV2KOQDPFPCBWPDNLINB7SXDJWHMJL3GYQQ, seg_pos={0 0}, upload=461595]: Long tail cancellation due to optimal threshold
Dec 27 13:43:37.324213442 (526h14m37.195296487s) [noise=true , address=64.49.13.169:4710    , piece_id=XIZKYK2E5FQJ37LYN535M7GXBBKXUQXFQWVMBWIBFDOAIIFGUAJA, seg_pos={0 0}, upload=461595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:57438->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:57438->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57438->64.49.13.169:4710: use of closed network connection
Dec 27 13:43:37.324249360 (526h14m37.195332405s) [noise=true , address=64.49.13.169:4710    , piece_id=XIZKYK2E5FQJ37LYN535M7GXBBKXUQXFQWVMBWIBFDOAIIFGUAJA, seg_pos={0 0}, upload=461595]: Long tail cancellation due to optimal threshold
Dec 27 13:43:37.324275360 (526h14m37.195358405s) [seg_pos={0 0}, upload=461595]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 13:43:37.324374018 (526h14m37.195457053s) [upload=461595]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 13:43:37.324381291 (526h14m37.195464337s) [upload=461595]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 13:43:37.679036406 (526h14m37.550119451s) [upload=461595]: Done uploading segments...
Dec 27 13:43:37.679056985 (526h14m37.55014003s) [upload=461595]: Upload finished. err: <nil>
Duration: 56.006284758s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
21 out of 60 is failed (threshold=3) [X...XX.......X....X..XX.X.XXXX..X..X.....X.XXX..X...XX......]
duration: 
5.660576781s
trace: 
https://jaeger.dev.storj.io/trace/7a3b11897bee958b
FAIL (2025-12-27 14:12:47 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 27 14:09:21.611393571 (526h40m21.482476616s) [upload=461959]: Starting upload
Dec 27 14:09:21.611546842 (526h40m21.482629877s) [upload=461959]: Done starting upload
Dec 27 14:09:21.611609071 (526h40m21.482692116s) [upload=461959]: Uploading segments...
Dec 27 14:09:21.612269158 (526h40m21.483352213s) [upload=461959]: Got next segment. Inline: false
Dec 27 14:09:21.612349201 (526h40m21.483432256s) [upload=461959]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 14:09:21.612356555 (526h40m21.4834396s) [upload=461959]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 14:09:21.856299604 (526h40m21.727382649s) [seg_pos={0 0}, upload=461959]: Begin upload segment...
Dec 27 14:09:21.856370038 (526h40m21.727453073s) [seg_pos={0 0}, upload=461959]: Done begin upload segment.
Dec 27 14:09:21.856420784 (526h40m21.727503819s) [noise=true , address=145.79.227.12:4765   , piece_id=5YKAMK34JOHYLZJT2D5WRZBY6N43346GSRFYFXHWZ6Z3AWWAQJ6A, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.856479476 (526h40m21.727562521s) [noise=true , address=145.79.227.29:4942   , piece_id=R6L233G7TZUNMLBODN4UEJH66YDWXL66B2WGAVQPHCIH5JXC74FA, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.857625017 (526h40m21.728708052s) [noise=true , address=45.91.116.13:4155    , piece_id=LBAG6T3GGWKHCOHA7T53VKSOX2O55KKQ4EGMRKDCK6PGFCTUIJVQ, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.858100663 (526h40m21.729183698s) [noise=true , address=145.79.227.10:4402   , piece_id=XUDYA3V3NLMFVKRX5ERUSILGUIOLX6NJNXWIL2PZDT6GFJMA3NDA, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.858170176 (526h40m21.729253231s) [noise=true , address=45.91.116.7:4009     , piece_id=US7DNVKAFLNLTFQB45RKVON242QU3RF3T5IELTT2WCNMUN6ZAYBQ, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.858402087 (526h40m21.729485122s) [noise=true , address=45.91.116.11:4120    , piece_id=UXEDL7GFL64HCRQ67THFOC2MKAVGYREYNQS7WEXJYGBOY3H7EYWQ, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.858794865 (526h40m21.7298779s) [noise=true , address=77.90.31.17:4773     , piece_id=5JIB2CV4VJSBBFINGJ4WCWLQN4H2T4WSEL6XDB3A755TJQKQOKYQ, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.859040964 (526h40m21.730124009s) [noise=true , address=145.79.227.19:4078   , piece_id=UAMIORNB3U6JKMHUIQRNRUNYW4T7S6YHERJG7ZVIHC5KLEXXCF6Q, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.861213120 (526h40m21.732296205s) [noise=true , address=145.79.227.45:4230   , piece_id=CPFKFQEL7BVP3V6I2LCDPVCGQACW3RDR7QT2V7XIJS77H6ZU7RIA, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.862069922 (526h40m21.733152957s) [noise=true , address=145.79.227.50:4375   , piece_id=XUAHWMEE2ZKSLFO3Z6CC7CZJZOSDSMNMDXYVGVK5ZLO2TDYZCH4Q, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.863841424 (526h40m21.734924469s) [noise=true , address=145.79.227.20:4438   , piece_id=42J67NFOF35HJMHKU2FLTASKR2TGKIY6NRBISTOIDIWBY64ACWSQ, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.867587077 (526h40m21.738670122s) [noise=true , address=145.79.227.42:4123   , piece_id=K6WUIK2EKFXSGSUXIOCSV4R3MOB2S4HPLEXOJRAKMCUC2GWYTEYA, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.869214605 (526h40m21.74029765s) [noise=true , address=45.91.116.6:4974     , piece_id=QXEEQHFYAH67J6OSDKWDVJCJV2S7ENNTQUWGV6TO5MMLTQ7JSOBQ, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.870424339 (526h40m21.741507384s) [noise=true , address=145.79.227.23:4546   , piece_id=YAGKC7A2NH6QNUCYJH3TIGQXTZZPDQ4FZTBR6LLAXSYMJTFAMI7A, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.871897163 (526h40m21.742980208s) [noise=true , address=145.79.227.17:4970   , piece_id=2SEU5CWQM23OWEPWAGUO34UBDJKT7TES2N3TZEIDIWDCCUBW5NEQ, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.872851330 (526h40m21.743934376s) [noise=true , address=145.79.227.36:4519   , piece_id=TVVXVEG5UT3CDL5AWEU4M2TQTOUTIV2RPCGBOGDCD2SRJIE4KBVA, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:23.532718349 (526h40m23.403801384s) [upload=461959]: Next returned nil segment
Dec 27 14:09:23.532735001 (526h40m23.403818047s) [upload=461959]: Waiting for error group managing segments...
Dec 27 14:09:23.862990534 (526h40m23.734073569s) [noise=true , address=145.79.227.12:4765   , piece_id=5YKAMK34JOHYLZJT2D5WRZBY6N43346GSRFYFXHWZ6Z3AWWAQJ6A, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:23.871865954 (526h40m23.742948989s) [noise=true , address=145.79.227.50:4375   , piece_id=XUAHWMEE2ZKSLFO3Z6CC7CZJZOSDSMNMDXYVGVK5ZLO2TDYZCH4Q, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:23.874552059 (526h40m23.745635104s) [noise=true , address=145.79.227.36:4519   , piece_id=TVVXVEG5UT3CDL5AWEU4M2TQTOUTIV2RPCGBOGDCD2SRJIE4KBVA, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:23.875081458 (526h40m23.746164493s) [noise=true , address=145.79.227.29:4942   , piece_id=R6L233G7TZUNMLBODN4UEJH66YDWXL66B2WGAVQPHCIH5JXC74FA, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:23.880889777 (526h40m23.751972812s) [noise=true , address=145.79.227.17:4970   , piece_id=2SEU5CWQM23OWEPWAGUO34UBDJKT7TES2N3TZEIDIWDCCUBW5NEQ, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:23.881779612 (526h40m23.752862657s) [noise=true , address=145.79.227.23:4546   , piece_id=YAGKC7A2NH6QNUCYJH3TIGQXTZZPDQ4FZTBR6LLAXSYMJTFAMI7A, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:23.888373497 (526h40m23.759456542s) [noise=true , address=145.79.227.10:4402   , piece_id=XUDYA3V3NLMFVKRX5ERUSILGUIOLX6NJNXWIL2PZDT6GFJMA3NDA, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:23.896892469 (526h40m23.767975504s) [noise=true , address=145.79.227.42:4123   , piece_id=K6WUIK2EKFXSGSUXIOCSV4R3MOB2S4HPLEXOJRAKMCUC2GWYTEYA, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:23.899060166 (526h40m23.770143202s) [noise=true , address=145.79.227.20:4438   , piece_id=42J67NFOF35HJMHKU2FLTASKR2TGKIY6NRBISTOIDIWBY64ACWSQ, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:23.899525874 (526h40m23.770608919s) [noise=true , address=145.79.227.45:4230   , piece_id=CPFKFQEL7BVP3V6I2LCDPVCGQACW3RDR7QT2V7XIJS77H6ZU7RIA, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:23.907289918 (526h40m23.778372953s) [noise=true , address=145.79.227.19:4078   , piece_id=UAMIORNB3U6JKMHUIQRNRUNYW4T7S6YHERJG7ZVIHC5KLEXXCF6Q, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:24.122094452 (526h40m23.993177497s) [noise=true , address=45.91.116.11:4120    , piece_id=UXEDL7GFL64HCRQ67THFOC2MKAVGYREYNQS7WEXJYGBOY3H7EYWQ, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:24.166811121 (526h40m24.037894166s) [noise=true , address=77.90.31.17:4773     , piece_id=5JIB2CV4VJSBBFINGJ4WCWLQN4H2T4WSEL6XDB3A755TJQKQOKYQ, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:24.166864091 (526h40m24.037947126s) [seg_pos={0 0}, upload=461959]: Segment reached optimal threshold of 13 pieces.
Dec 27 14:09:24.167067258 (526h40m24.038150303s) [noise=true , address=45.91.116.7:4009     , piece_id=US7DNVKAFLNLTFQB45RKVON242QU3RF3T5IELTT2WCNMUN6ZAYBQ, seg_pos={0 0}, upload=461959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF
Dec 27 14:09:24.167077768 (526h40m24.038160823s) [noise=true , address=45.91.116.7:4009     , piece_id=US7DNVKAFLNLTFQB45RKVON242QU3RF3T5IELTT2WCNMUN6ZAYBQ, seg_pos={0 0}, upload=461959]: Long tail cancellation due to optimal threshold
Dec 27 14:09:24.167395192 (526h40m24.038478237s) [noise=true , address=45.91.116.13:4155    , piece_id=LBAG6T3GGWKHCOHA7T53VKSOX2O55KKQ4EGMRKDCK6PGFCTUIJVQ, seg_pos={0 0}, upload=461959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF
Dec 27 14:09:24.167402437 (526h40m24.038485472s) [noise=true , address=45.91.116.13:4155    , piece_id=LBAG6T3GGWKHCOHA7T53VKSOX2O55KKQ4EGMRKDCK6PGFCTUIJVQ, seg_pos={0 0}, upload=461959]: Long tail cancellation due to optimal threshold
Dec 27 14:09:24.167513648 (526h40m24.038596694s) [noise=true , address=45.91.116.6:4974     , piece_id=QXEEQHFYAH67J6OSDKWDVJCJV2S7ENNTQUWGV6TO5MMLTQ7JSOBQ, seg_pos={0 0}, upload=461959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF
Dec 27 14:09:24.167519349 (526h40m24.038602384s) [noise=true , address=45.91.116.6:4974     , piece_id=QXEEQHFYAH67J6OSDKWDVJCJV2S7ENNTQUWGV6TO5MMLTQ7JSOBQ, seg_pos={0 0}, upload=461959]: Long tail cancellation due to optimal threshold
Dec 27 14:09:24.167532774 (526h40m24.038615809s) [seg_pos={0 0}, upload=461959]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 14:09:24.167612757 (526h40m24.038695792s) [upload=461959]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 14:09:24.167618839 (526h40m24.038701884s) [upload=461959]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 14:09:24.455792179 (526h40m24.326875214s) [upload=461959]: Done uploading segments...
Dec 27 14:09:24.455820133 (526h40m24.326903168s) [upload=461959]: Upload finished. err: <nil>
Duration: 5.660606678s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip: 
2 out of 60 is failed (threshold=3) [........................................XX..................]
duration: 
11.932675486s
trace: 
https://jaeger.dev.storj.io/trace/75ef8b5e6bc2819a
FAIL (2025-12-27 14:12:38 UTC) 
orbiter warning: context deadline exceeded
Duration: 11.932702306s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
21 out of 60 is failed (threshold=3) [.X...XX.......X....X..XX.X.XXXX..X..X.....X.XXX..X...XX.....]
duration: 
6.093110579s
trace: 
https://jaeger.dev.storj.io/trace/7a3b11897bee958b
FAIL (2025-12-27 14:12:29 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 27 14:09:21.611393571 (526h40m21.482476616s) [upload=461959]: Starting upload
Dec 27 14:09:21.611546842 (526h40m21.482629877s) [upload=461959]: Done starting upload
Dec 27 14:09:21.611609071 (526h40m21.482692116s) [upload=461959]: Uploading segments...
Dec 27 14:09:21.612269158 (526h40m21.483352213s) [upload=461959]: Got next segment. Inline: false
Dec 27 14:09:21.612349201 (526h40m21.483432256s) [upload=461959]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 14:09:21.612356555 (526h40m21.4834396s) [upload=461959]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 14:09:21.856299604 (526h40m21.727382649s) [seg_pos={0 0}, upload=461959]: Begin upload segment...
Dec 27 14:09:21.856370038 (526h40m21.727453073s) [seg_pos={0 0}, upload=461959]: Done begin upload segment.
Dec 27 14:09:21.856420784 (526h40m21.727503819s) [noise=true , address=145.79.227.12:4765   , piece_id=5YKAMK34JOHYLZJT2D5WRZBY6N43346GSRFYFXHWZ6Z3AWWAQJ6A, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.856479476 (526h40m21.727562521s) [noise=true , address=145.79.227.29:4942   , piece_id=R6L233G7TZUNMLBODN4UEJH66YDWXL66B2WGAVQPHCIH5JXC74FA, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.857625017 (526h40m21.728708052s) [noise=true , address=45.91.116.13:4155    , piece_id=LBAG6T3GGWKHCOHA7T53VKSOX2O55KKQ4EGMRKDCK6PGFCTUIJVQ, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.858100663 (526h40m21.729183698s) [noise=true , address=145.79.227.10:4402   , piece_id=XUDYA3V3NLMFVKRX5ERUSILGUIOLX6NJNXWIL2PZDT6GFJMA3NDA, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.858170176 (526h40m21.729253231s) [noise=true , address=45.91.116.7:4009     , piece_id=US7DNVKAFLNLTFQB45RKVON242QU3RF3T5IELTT2WCNMUN6ZAYBQ, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.858402087 (526h40m21.729485122s) [noise=true , address=45.91.116.11:4120    , piece_id=UXEDL7GFL64HCRQ67THFOC2MKAVGYREYNQS7WEXJYGBOY3H7EYWQ, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.858794865 (526h40m21.7298779s) [noise=true , address=77.90.31.17:4773     , piece_id=5JIB2CV4VJSBBFINGJ4WCWLQN4H2T4WSEL6XDB3A755TJQKQOKYQ, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.859040964 (526h40m21.730124009s) [noise=true , address=145.79.227.19:4078   , piece_id=UAMIORNB3U6JKMHUIQRNRUNYW4T7S6YHERJG7ZVIHC5KLEXXCF6Q, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.861213120 (526h40m21.732296205s) [noise=true , address=145.79.227.45:4230   , piece_id=CPFKFQEL7BVP3V6I2LCDPVCGQACW3RDR7QT2V7XIJS77H6ZU7RIA, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.862069922 (526h40m21.733152957s) [noise=true , address=145.79.227.50:4375   , piece_id=XUAHWMEE2ZKSLFO3Z6CC7CZJZOSDSMNMDXYVGVK5ZLO2TDYZCH4Q, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.863841424 (526h40m21.734924469s) [noise=true , address=145.79.227.20:4438   , piece_id=42J67NFOF35HJMHKU2FLTASKR2TGKIY6NRBISTOIDIWBY64ACWSQ, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.867587077 (526h40m21.738670122s) [noise=true , address=145.79.227.42:4123   , piece_id=K6WUIK2EKFXSGSUXIOCSV4R3MOB2S4HPLEXOJRAKMCUC2GWYTEYA, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.869214605 (526h40m21.74029765s) [noise=true , address=45.91.116.6:4974     , piece_id=QXEEQHFYAH67J6OSDKWDVJCJV2S7ENNTQUWGV6TO5MMLTQ7JSOBQ, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.870424339 (526h40m21.741507384s) [noise=true , address=145.79.227.23:4546   , piece_id=YAGKC7A2NH6QNUCYJH3TIGQXTZZPDQ4FZTBR6LLAXSYMJTFAMI7A, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.871897163 (526h40m21.742980208s) [noise=true , address=145.79.227.17:4970   , piece_id=2SEU5CWQM23OWEPWAGUO34UBDJKT7TES2N3TZEIDIWDCCUBW5NEQ, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.872851330 (526h40m21.743934376s) [noise=true , address=145.79.227.36:4519   , piece_id=TVVXVEG5UT3CDL5AWEU4M2TQTOUTIV2RPCGBOGDCD2SRJIE4KBVA, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:23.532718349 (526h40m23.403801384s) [upload=461959]: Next returned nil segment
Dec 27 14:09:23.532735001 (526h40m23.403818047s) [upload=461959]: Waiting for error group managing segments...
Dec 27 14:09:23.862990534 (526h40m23.734073569s) [noise=true , address=145.79.227.12:4765   , piece_id=5YKAMK34JOHYLZJT2D5WRZBY6N43346GSRFYFXHWZ6Z3AWWAQJ6A, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:23.871865954 (526h40m23.742948989s) [noise=true , address=145.79.227.50:4375   , piece_id=XUAHWMEE2ZKSLFO3Z6CC7CZJZOSDSMNMDXYVGVK5ZLO2TDYZCH4Q, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:23.874552059 (526h40m23.745635104s) [noise=true , address=145.79.227.36:4519   , piece_id=TVVXVEG5UT3CDL5AWEU4M2TQTOUTIV2RPCGBOGDCD2SRJIE4KBVA, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:23.875081458 (526h40m23.746164493s) [noise=true , address=145.79.227.29:4942   , piece_id=R6L233G7TZUNMLBODN4UEJH66YDWXL66B2WGAVQPHCIH5JXC74FA, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:23.880889777 (526h40m23.751972812s) [noise=true , address=145.79.227.17:4970   , piece_id=2SEU5CWQM23OWEPWAGUO34UBDJKT7TES2N3TZEIDIWDCCUBW5NEQ, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:23.881779612 (526h40m23.752862657s) [noise=true , address=145.79.227.23:4546   , piece_id=YAGKC7A2NH6QNUCYJH3TIGQXTZZPDQ4FZTBR6LLAXSYMJTFAMI7A, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:23.888373497 (526h40m23.759456542s) [noise=true , address=145.79.227.10:4402   , piece_id=XUDYA3V3NLMFVKRX5ERUSILGUIOLX6NJNXWIL2PZDT6GFJMA3NDA, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:23.896892469 (526h40m23.767975504s) [noise=true , address=145.79.227.42:4123   , piece_id=K6WUIK2EKFXSGSUXIOCSV4R3MOB2S4HPLEXOJRAKMCUC2GWYTEYA, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:23.899060166 (526h40m23.770143202s) [noise=true , address=145.79.227.20:4438   , piece_id=42J67NFOF35HJMHKU2FLTASKR2TGKIY6NRBISTOIDIWBY64ACWSQ, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:23.899525874 (526h40m23.770608919s) [noise=true , address=145.79.227.45:4230   , piece_id=CPFKFQEL7BVP3V6I2LCDPVCGQACW3RDR7QT2V7XIJS77H6ZU7RIA, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:23.907289918 (526h40m23.778372953s) [noise=true , address=145.79.227.19:4078   , piece_id=UAMIORNB3U6JKMHUIQRNRUNYW4T7S6YHERJG7ZVIHC5KLEXXCF6Q, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:24.122094452 (526h40m23.993177497s) [noise=true , address=45.91.116.11:4120    , piece_id=UXEDL7GFL64HCRQ67THFOC2MKAVGYREYNQS7WEXJYGBOY3H7EYWQ, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:24.166811121 (526h40m24.037894166s) [noise=true , address=77.90.31.17:4773     , piece_id=5JIB2CV4VJSBBFINGJ4WCWLQN4H2T4WSEL6XDB3A755TJQKQOKYQ, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:24.166864091 (526h40m24.037947126s) [seg_pos={0 0}, upload=461959]: Segment reached optimal threshold of 13 pieces.
Dec 27 14:09:24.167067258 (526h40m24.038150303s) [noise=true , address=45.91.116.7:4009     , piece_id=US7DNVKAFLNLTFQB45RKVON242QU3RF3T5IELTT2WCNMUN6ZAYBQ, seg_pos={0 0}, upload=461959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF
Dec 27 14:09:24.167077768 (526h40m24.038160823s) [noise=true , address=45.91.116.7:4009     , piece_id=US7DNVKAFLNLTFQB45RKVON242QU3RF3T5IELTT2WCNMUN6ZAYBQ, seg_pos={0 0}, upload=461959]: Long tail cancellation due to optimal threshold
Dec 27 14:09:24.167395192 (526h40m24.038478237s) [noise=true , address=45.91.116.13:4155    , piece_id=LBAG6T3GGWKHCOHA7T53VKSOX2O55KKQ4EGMRKDCK6PGFCTUIJVQ, seg_pos={0 0}, upload=461959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF
Dec 27 14:09:24.167402437 (526h40m24.038485472s) [noise=true , address=45.91.116.13:4155    , piece_id=LBAG6T3GGWKHCOHA7T53VKSOX2O55KKQ4EGMRKDCK6PGFCTUIJVQ, seg_pos={0 0}, upload=461959]: Long tail cancellation due to optimal threshold
Dec 27 14:09:24.167513648 (526h40m24.038596694s) [noise=true , address=45.91.116.6:4974     , piece_id=QXEEQHFYAH67J6OSDKWDVJCJV2S7ENNTQUWGV6TO5MMLTQ7JSOBQ, seg_pos={0 0}, upload=461959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF
Dec 27 14:09:24.167519349 (526h40m24.038602384s) [noise=true , address=45.91.116.6:4974     , piece_id=QXEEQHFYAH67J6OSDKWDVJCJV2S7ENNTQUWGV6TO5MMLTQ7JSOBQ, seg_pos={0 0}, upload=461959]: Long tail cancellation due to optimal threshold
Dec 27 14:09:24.167532774 (526h40m24.038615809s) [seg_pos={0 0}, upload=461959]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 14:09:24.167612757 (526h40m24.038695792s) [upload=461959]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 14:09:24.167618839 (526h40m24.038701884s) [upload=461959]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 14:09:24.455792179 (526h40m24.326875214s) [upload=461959]: Done uploading segments...
Dec 27 14:09:24.455820133 (526h40m24.326903168s) [upload=461959]: Upload finished. err: <nil>
Duration: 6.093135285s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip: 
2 out of 60 is failed (threshold=3) [.........................................XX.................]
duration: 
15.651306174s
trace: 
https://jaeger.dev.storj.io/trace/75ef8b5e6bc2819a
FAIL (2025-12-27 14:12:19 UTC) 
orbiter warning: context deadline exceeded
Duration: 15.651328919s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
21 out of 60 is failed (threshold=3) [..X...XX.......X....X..XX.X.XXXX..X..X.....X.XXX..X...XX....]
duration: 
5.671884411s
trace: 
https://jaeger.dev.storj.io/trace/7a3b11897bee958b
FAIL (2025-12-27 14:12:10 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 27 14:09:21.611393571 (526h40m21.482476616s) [upload=461959]: Starting upload
Dec 27 14:09:21.611546842 (526h40m21.482629877s) [upload=461959]: Done starting upload
Dec 27 14:09:21.611609071 (526h40m21.482692116s) [upload=461959]: Uploading segments...
Dec 27 14:09:21.612269158 (526h40m21.483352213s) [upload=461959]: Got next segment. Inline: false
Dec 27 14:09:21.612349201 (526h40m21.483432256s) [upload=461959]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 14:09:21.612356555 (526h40m21.4834396s) [upload=461959]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 14:09:21.856299604 (526h40m21.727382649s) [seg_pos={0 0}, upload=461959]: Begin upload segment...
Dec 27 14:09:21.856370038 (526h40m21.727453073s) [seg_pos={0 0}, upload=461959]: Done begin upload segment.
Dec 27 14:09:21.856420784 (526h40m21.727503819s) [noise=true , address=145.79.227.12:4765   , piece_id=5YKAMK34JOHYLZJT2D5WRZBY6N43346GSRFYFXHWZ6Z3AWWAQJ6A, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.856479476 (526h40m21.727562521s) [noise=true , address=145.79.227.29:4942   , piece_id=R6L233G7TZUNMLBODN4UEJH66YDWXL66B2WGAVQPHCIH5JXC74FA, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.857625017 (526h40m21.728708052s) [noise=true , address=45.91.116.13:4155    , piece_id=LBAG6T3GGWKHCOHA7T53VKSOX2O55KKQ4EGMRKDCK6PGFCTUIJVQ, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.858100663 (526h40m21.729183698s) [noise=true , address=145.79.227.10:4402   , piece_id=XUDYA3V3NLMFVKRX5ERUSILGUIOLX6NJNXWIL2PZDT6GFJMA3NDA, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.858170176 (526h40m21.729253231s) [noise=true , address=45.91.116.7:4009     , piece_id=US7DNVKAFLNLTFQB45RKVON242QU3RF3T5IELTT2WCNMUN6ZAYBQ, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.858402087 (526h40m21.729485122s) [noise=true , address=45.91.116.11:4120    , piece_id=UXEDL7GFL64HCRQ67THFOC2MKAVGYREYNQS7WEXJYGBOY3H7EYWQ, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.858794865 (526h40m21.7298779s) [noise=true , address=77.90.31.17:4773     , piece_id=5JIB2CV4VJSBBFINGJ4WCWLQN4H2T4WSEL6XDB3A755TJQKQOKYQ, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.859040964 (526h40m21.730124009s) [noise=true , address=145.79.227.19:4078   , piece_id=UAMIORNB3U6JKMHUIQRNRUNYW4T7S6YHERJG7ZVIHC5KLEXXCF6Q, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.861213120 (526h40m21.732296205s) [noise=true , address=145.79.227.45:4230   , piece_id=CPFKFQEL7BVP3V6I2LCDPVCGQACW3RDR7QT2V7XIJS77H6ZU7RIA, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.862069922 (526h40m21.733152957s) [noise=true , address=145.79.227.50:4375   , piece_id=XUAHWMEE2ZKSLFO3Z6CC7CZJZOSDSMNMDXYVGVK5ZLO2TDYZCH4Q, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.863841424 (526h40m21.734924469s) [noise=true , address=145.79.227.20:4438   , piece_id=42J67NFOF35HJMHKU2FLTASKR2TGKIY6NRBISTOIDIWBY64ACWSQ, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.867587077 (526h40m21.738670122s) [noise=true , address=145.79.227.42:4123   , piece_id=K6WUIK2EKFXSGSUXIOCSV4R3MOB2S4HPLEXOJRAKMCUC2GWYTEYA, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.869214605 (526h40m21.74029765s) [noise=true , address=45.91.116.6:4974     , piece_id=QXEEQHFYAH67J6OSDKWDVJCJV2S7ENNTQUWGV6TO5MMLTQ7JSOBQ, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.870424339 (526h40m21.741507384s) [noise=true , address=145.79.227.23:4546   , piece_id=YAGKC7A2NH6QNUCYJH3TIGQXTZZPDQ4FZTBR6LLAXSYMJTFAMI7A, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.871897163 (526h40m21.742980208s) [noise=true , address=145.79.227.17:4970   , piece_id=2SEU5CWQM23OWEPWAGUO34UBDJKT7TES2N3TZEIDIWDCCUBW5NEQ, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.872851330 (526h40m21.743934376s) [noise=true , address=145.79.227.36:4519   , piece_id=TVVXVEG5UT3CDL5AWEU4M2TQTOUTIV2RPCGBOGDCD2SRJIE4KBVA, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:23.532718349 (526h40m23.403801384s) [upload=461959]: Next returned nil segment
Dec 27 14:09:23.532735001 (526h40m23.403818047s) [upload=461959]: Waiting for error group managing segments...
Dec 27 14:09:23.862990534 (526h40m23.734073569s) [noise=true , address=145.79.227.12:4765   , piece_id=5YKAMK34JOHYLZJT2D5WRZBY6N43346GSRFYFXHWZ6Z3AWWAQJ6A, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:23.871865954 (526h40m23.742948989s) [noise=true , address=145.79.227.50:4375   , piece_id=XUAHWMEE2ZKSLFO3Z6CC7CZJZOSDSMNMDXYVGVK5ZLO2TDYZCH4Q, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:23.874552059 (526h40m23.745635104s) [noise=true , address=145.79.227.36:4519   , piece_id=TVVXVEG5UT3CDL5AWEU4M2TQTOUTIV2RPCGBOGDCD2SRJIE4KBVA, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:23.875081458 (526h40m23.746164493s) [noise=true , address=145.79.227.29:4942   , piece_id=R6L233G7TZUNMLBODN4UEJH66YDWXL66B2WGAVQPHCIH5JXC74FA, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:23.880889777 (526h40m23.751972812s) [noise=true , address=145.79.227.17:4970   , piece_id=2SEU5CWQM23OWEPWAGUO34UBDJKT7TES2N3TZEIDIWDCCUBW5NEQ, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:23.881779612 (526h40m23.752862657s) [noise=true , address=145.79.227.23:4546   , piece_id=YAGKC7A2NH6QNUCYJH3TIGQXTZZPDQ4FZTBR6LLAXSYMJTFAMI7A, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:23.888373497 (526h40m23.759456542s) [noise=true , address=145.79.227.10:4402   , piece_id=XUDYA3V3NLMFVKRX5ERUSILGUIOLX6NJNXWIL2PZDT6GFJMA3NDA, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:23.896892469 (526h40m23.767975504s) [noise=true , address=145.79.227.42:4123   , piece_id=K6WUIK2EKFXSGSUXIOCSV4R3MOB2S4HPLEXOJRAKMCUC2GWYTEYA, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:23.899060166 (526h40m23.770143202s) [noise=true , address=145.79.227.20:4438   , piece_id=42J67NFOF35HJMHKU2FLTASKR2TGKIY6NRBISTOIDIWBY64ACWSQ, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:23.899525874 (526h40m23.770608919s) [noise=true , address=145.79.227.45:4230   , piece_id=CPFKFQEL7BVP3V6I2LCDPVCGQACW3RDR7QT2V7XIJS77H6ZU7RIA, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:23.907289918 (526h40m23.778372953s) [noise=true , address=145.79.227.19:4078   , piece_id=UAMIORNB3U6JKMHUIQRNRUNYW4T7S6YHERJG7ZVIHC5KLEXXCF6Q, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:24.122094452 (526h40m23.993177497s) [noise=true , address=45.91.116.11:4120    , piece_id=UXEDL7GFL64HCRQ67THFOC2MKAVGYREYNQS7WEXJYGBOY3H7EYWQ, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:24.166811121 (526h40m24.037894166s) [noise=true , address=77.90.31.17:4773     , piece_id=5JIB2CV4VJSBBFINGJ4WCWLQN4H2T4WSEL6XDB3A755TJQKQOKYQ, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:24.166864091 (526h40m24.037947126s) [seg_pos={0 0}, upload=461959]: Segment reached optimal threshold of 13 pieces.
Dec 27 14:09:24.167067258 (526h40m24.038150303s) [noise=true , address=45.91.116.7:4009     , piece_id=US7DNVKAFLNLTFQB45RKVON242QU3RF3T5IELTT2WCNMUN6ZAYBQ, seg_pos={0 0}, upload=461959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF
Dec 27 14:09:24.167077768 (526h40m24.038160823s) [noise=true , address=45.91.116.7:4009     , piece_id=US7DNVKAFLNLTFQB45RKVON242QU3RF3T5IELTT2WCNMUN6ZAYBQ, seg_pos={0 0}, upload=461959]: Long tail cancellation due to optimal threshold
Dec 27 14:09:24.167395192 (526h40m24.038478237s) [noise=true , address=45.91.116.13:4155    , piece_id=LBAG6T3GGWKHCOHA7T53VKSOX2O55KKQ4EGMRKDCK6PGFCTUIJVQ, seg_pos={0 0}, upload=461959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF
Dec 27 14:09:24.167402437 (526h40m24.038485472s) [noise=true , address=45.91.116.13:4155    , piece_id=LBAG6T3GGWKHCOHA7T53VKSOX2O55KKQ4EGMRKDCK6PGFCTUIJVQ, seg_pos={0 0}, upload=461959]: Long tail cancellation due to optimal threshold
Dec 27 14:09:24.167513648 (526h40m24.038596694s) [noise=true , address=45.91.116.6:4974     , piece_id=QXEEQHFYAH67J6OSDKWDVJCJV2S7ENNTQUWGV6TO5MMLTQ7JSOBQ, seg_pos={0 0}, upload=461959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF
Dec 27 14:09:24.167519349 (526h40m24.038602384s) [noise=true , address=45.91.116.6:4974     , piece_id=QXEEQHFYAH67J6OSDKWDVJCJV2S7ENNTQUWGV6TO5MMLTQ7JSOBQ, seg_pos={0 0}, upload=461959]: Long tail cancellation due to optimal threshold
Dec 27 14:09:24.167532774 (526h40m24.038615809s) [seg_pos={0 0}, upload=461959]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 14:09:24.167612757 (526h40m24.038695792s) [upload=461959]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 14:09:24.167618839 (526h40m24.038701884s) [upload=461959]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 14:09:24.455792179 (526h40m24.326875214s) [upload=461959]: Done uploading segments...
Dec 27 14:09:24.455820133 (526h40m24.326903168s) [upload=461959]: Upload finished. err: <nil>
Duration: 5.671938022s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
39 out of 60 is failed (threshold=6) [....X.X.X.X.X..XXX.XX.XX.X.XX.X.XXXXXXXXXXX.X..XXXX.XXXXXXX.]
duration: 
5.001420464s
trace: 
https://jaeger.dev.storj.io/trace/4488be0ae555676a
FAIL (2025-12-27 14:11:59 UTC) 
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001478365s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip: 
2 out of 60 is failed (threshold=3) [..........................................XX................]
duration: 
7.836421102s
trace: 
https://jaeger.dev.storj.io/trace/75ef8b5e6bc2819a
FAIL (2025-12-27 14:11:53 UTC) 
orbiter warning: context deadline exceeded
Duration: 7.836455957s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
4 out of 60 is failed (threshold=3) [.X..........X.............XX................................]
duration: 
58.390448932s
trace: 
https://jaeger.dev.storj.io/trace/3ed1e657d506c4ee
FAIL (2025-12-27 14:11:48 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, Piece ID: HEMLEGWLVMK4LG5DM3PUWU5HSUEHKBECYOMHUXIQRGZGNXPXFLBQ) context deadline exceeded; piecestore: (Node ID: 12wXqePK1kK7iBo6wHcbqkRmkED8LGJ3rzQNJAWa14cHYhx1oyB, Piece ID: M5NU2MR5LQWOKNLVKKEJXMJNVEIZVDDNIZHWNN7XXHVTJRS2OQNA) context deadline exceeded; piecestore: (Node ID: 12ATtVD3XyFxzEqwQpSF4t1MWygqEa4Y1CkNj1s43eud8UtuSmx, Piece ID: ET7O467BME3OED32PFLJKYVIWZRDF6WMPXUT34OFWSPXLSMKKOWQ) context deadline exceeded; piecestore: (Node ID: 12UjFkFn2UEvr57LDorcijDCvkBkkKitgZEFx6Yqfk5gMStQvJG, Piece ID: SF3IKAIYXQR4YPWNVXT66SEHVM54R3K3VJHA6L5RBOU3DITOR5LQ) context deadline exceeded
======LOGS======
) [noise=true , address=77.90.31.16:4739     , piece_id=H6RNS5VOYTMWX2N6GLV2KOQDPFPCBWPDNLINB7SXDJWHMJL3GYQQ, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.319646322 (526h14m34.190729357s) [noise=true , address=77.90.31.21:4899     , piece_id=SF3IKAIYXQR4YPWNVXT66SEHVM54R3K3VJHA6L5RBOU3DITOR5LQ, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.319933248 (526h14m34.191016283s) [noise=true , address=45.91.116.12:4234    , piece_id=Y2FIVYLZBQAYBO2TDGIZ4RDQHQJ3WOQH565A7ESIKANQQTQDFQZQ, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.320400588 (526h14m34.191483633s) [noise=true , address=145.79.227.29:4952   , piece_id=VXZLF4OGRIR3MYQMMLKLTD235A55F4YTHB3D4YCVCQUL4FZTSFJQ, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.320407341 (526h14m34.191490386s) [noise=true , address=145.79.227.46:4621   , piece_id=XZ4MKAZAGQKVEJJGSZ5QGT4AT3MFXXNMXICVFCI6SP3E4EVEHZGQ, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.320675030 (526h14m34.191758075s) [noise=true , address=45.91.116.10:4113    , piece_id=OFHVWJWEHSVOOONDLXBZY72VB5G6FFXH2NXL45KY7TJ6CPVZ5UTQ, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.320802022 (526h14m34.191885067s) [noise=true , address=77.90.31.11:4569     , piece_id=YC2Q3HPJDVY3NEDZNC2Z4KVN6S6DDDGRARASF7KALFAOQ335A7HA, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.321169643 (526h14m34.192252678s) [noise=true , address=77.90.31.19:4869     , piece_id=HEMLEGWLVMK4LG5DM3PUWU5HSUEHKBECYOMHUXIQRGZGNXPXFLBQ, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.321443483 (526h14m34.192526518s) [noise=true , address=45.91.116.14:4202    , piece_id=M5NU2MR5LQWOKNLVKKEJXMJNVEIZVDDNIZHWNN7XXHVTJRS2OQNA, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.321731813 (526h14m34.192814848s) [noise=true , address=77.90.31.18:4819     , piece_id=YS4DXTHLGCPFOKKLBA2DK2BVOAPTQEGVOPBSRCLFV26SVP6YFHGA, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.322221776 (526h14m34.193304811s) [noise=true , address=77.90.31.15:4708     , piece_id=3NX6E54XMZXQAQQE27LTWUSEF5GAIF2XOPQFFAP7LOA2PL7KDL3Q, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.322547776 (526h14m34.193630811s) [noise=true , address=145.79.227.40:4079   , piece_id=237LYFTAG7G4HMIUKFZDDUTYJC7AAHHXTO5EIO46UEHY24BOVB3A, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.322908093 (526h14m34.193991138s) [noise=true , address=145.79.227.29:4975   , piece_id=DAMNBVP7DRSW5OAGYRRP4K5SNJ2WEUT7IAXO77KWRHLQEMBNERZQ, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.323267968 (526h14m34.194351003s) [noise=true , address=145.79.227.30:4787   , piece_id=OB2OTY3ZEHIUFYM6FY7HIS7HQ353Y2HL6XU4476HXOD26QVA4DPQ, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.323618165 (526h14m34.19470121s) [noise=true , address=64.49.13.169:4710    , piece_id=XIZKYK2E5FQJ37LYN535M7GXBBKXUQXFQWVMBWIBFDOAIIFGUAJA, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.324084893 (526h14m34.195167928s) [noise=true , address=193.149.164.143:4694 , piece_id=ET7O467BME3OED32PFLJKYVIWZRDF6WMPXUT34OFWSPXLSMKKOWQ, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:35.783540136 (526h14m35.654623181s) [upload=461595]: Next returned nil segment
Dec 27 13:43:35.783555856 (526h14m35.654638891s) [upload=461595]: Waiting for error group managing segments...
Dec 27 13:43:36.192287250 (526h14m36.063370296s) [noise=true , address=145.79.227.29:4952   , piece_id=VXZLF4OGRIR3MYQMMLKLTD235A55F4YTHB3D4YCVCQUL4FZTSFJQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:36.200200289 (526h14m36.071283334s) [noise=true , address=145.79.227.46:4621   , piece_id=XZ4MKAZAGQKVEJJGSZ5QGT4AT3MFXXNMXICVFCI6SP3E4EVEHZGQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:36.238087532 (526h14m36.109170577s) [noise=true , address=145.79.227.29:4975   , piece_id=DAMNBVP7DRSW5OAGYRRP4K5SNJ2WEUT7IAXO77KWRHLQEMBNERZQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:36.379631346 (526h14m36.250714391s) [noise=true , address=77.90.31.19:4869     , piece_id=HEMLEGWLVMK4LG5DM3PUWU5HSUEHKBECYOMHUXIQRGZGNXPXFLBQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:36.382945286 (526h14m36.254028321s) [noise=true , address=145.79.227.34:4909   , piece_id=Z6RF5PSOGOZMOXPQYXHMXGFXIELVXXVU4NXXRTAQCIELQXLMVXBQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:36.400290103 (526h14m36.271373138s) [noise=true , address=77.90.31.18:4819     , piece_id=YS4DXTHLGCPFOKKLBA2DK2BVOAPTQEGVOPBSRCLFV26SVP6YFHGA, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:36.436707208 (526h14m36.307790243s) [noise=true , address=45.91.116.14:4202    , piece_id=M5NU2MR5LQWOKNLVKKEJXMJNVEIZVDDNIZHWNN7XXHVTJRS2OQNA, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:36.487651475 (526h14m36.35873451s) [noise=true , address=45.91.116.12:4234    , piece_id=Y2FIVYLZBQAYBO2TDGIZ4RDQHQJ3WOQH565A7ESIKANQQTQDFQZQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:36.500850769 (526h14m36.371933814s) [noise=true , address=77.90.31.11:4569     , piece_id=YC2Q3HPJDVY3NEDZNC2Z4KVN6S6DDDGRARASF7KALFAOQ335A7HA, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:36.518828551 (526h14m36.389911586s) [noise=true , address=77.90.31.15:4708     , piece_id=3NX6E54XMZXQAQQE27LTWUSEF5GAIF2XOPQFFAP7LOA2PL7KDL3Q, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:37.051607243 (526h14m36.922690289s) [noise=true , address=145.79.227.44:4228   , piece_id=6SAYEAOTTCOUFPU75C7YRIULWJGXL4PK2YKCKFB47EUBWMRQQQOQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:37.096826909 (526h14m36.967909954s) [noise=true , address=145.79.227.30:4787   , piece_id=OB2OTY3ZEHIUFYM6FY7HIS7HQ353Y2HL6XU4476HXOD26QVA4DPQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:37.140738995 (526h14m37.01182203s) [noise=true , address=77.90.31.21:4899     , piece_id=SF3IKAIYXQR4YPWNVXT66SEHVM54R3K3VJHA6L5RBOU3DITOR5LQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:37.323613990 (526h14m37.194697035s) [noise=true , address=193.149.164.143:4694 , piece_id=ET7O467BME3OED32PFLJKYVIWZRDF6WMPXUT34OFWSPXLSMKKOWQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:37.323649618 (526h14m37.194732663s) [seg_pos={0 0}, upload=461595]: Segment reached optimal threshold of 14 pieces.
Dec 27 13:43:37.323852865 (526h14m37.1949359s) [noise=true , address=45.91.116.10:4113    , piece_id=OFHVWJWEHSVOOONDLXBZY72VB5G6FFXH2NXL45KY7TJ6CPVZ5UTQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12puRMny4nbgv2by4tAbXcBBSgG2d6BfwcDDRDcFzEdjSVUJxtd): protocol: expected piece hash; context canceled; EOF
Dec 27 13:43:37.323860349 (526h14m37.194943384s) [noise=true , address=45.91.116.10:4113    , piece_id=OFHVWJWEHSVOOONDLXBZY72VB5G6FFXH2NXL45KY7TJ6CPVZ5UTQ, seg_pos={0 0}, upload=461595]: Long tail cancellation due to optimal threshold
Dec 27 13:43:37.324005135 (526h14m37.1950882s) [noise=true , address=145.79.227.40:4079   , piece_id=237LYFTAG7G4HMIUKFZDDUTYJC7AAHHXTO5EIO46UEHY24BOVB3A, seg_pos={0 0}, upload=461595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RYKzn5zVSmxVbWwnBdX8oqtG1WE2MhPeEiWLf5dZeFAut6s2Y): protocol: expected piece hash; context canceled; EOF
Dec 27 13:43:37.324022659 (526h14m37.195105704s) [noise=true , address=145.79.227.40:4079   , piece_id=237LYFTAG7G4HMIUKFZDDUTYJC7AAHHXTO5EIO46UEHY24BOVB3A, seg_pos={0 0}, upload=461595]: Long tail cancellation due to optimal threshold
Dec 27 13:43:37.324143378 (526h14m37.195226413s) [noise=true , address=77.90.31.16:4739     , piece_id=H6RNS5VOYTMWX2N6GLV2KOQDPFPCBWPDNLINB7SXDJWHMJL3GYQQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129B3N7yjJvwCzoVLTqWsPbcLQ5TQUyAzg1RCkbSAVKPWYTG8eb): protocol: expected piece hash; context canceled; EOF
Dec 27 13:43:37.324154520 (526h14m37.195237555s) [noise=true , address=77.90.31.16:4739     , piece_id=H6RNS5VOYTMWX2N6GLV2KOQDPFPCBWPDNLINB7SXDJWHMJL3GYQQ, seg_pos={0 0}, upload=461595]: Long tail cancellation due to optimal threshold
Dec 27 13:43:37.324213442 (526h14m37.195296487s) [noise=true , address=64.49.13.169:4710    , piece_id=XIZKYK2E5FQJ37LYN535M7GXBBKXUQXFQWVMBWIBFDOAIIFGUAJA, seg_pos={0 0}, upload=461595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:57438->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:57438->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57438->64.49.13.169:4710: use of closed network connection
Dec 27 13:43:37.324249360 (526h14m37.195332405s) [noise=true , address=64.49.13.169:4710    , piece_id=XIZKYK2E5FQJ37LYN535M7GXBBKXUQXFQWVMBWIBFDOAIIFGUAJA, seg_pos={0 0}, upload=461595]: Long tail cancellation due to optimal threshold
Dec 27 13:43:37.324275360 (526h14m37.195358405s) [seg_pos={0 0}, upload=461595]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 13:43:37.324374018 (526h14m37.195457053s) [upload=461595]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 13:43:37.324381291 (526h14m37.195464337s) [upload=461595]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 13:43:37.679036406 (526h14m37.550119451s) [upload=461595]: Done uploading segments...
Dec 27 13:43:37.679056985 (526h14m37.55014003s) [upload=461595]: Upload finished. err: <nil>
Duration: 58.390470995s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
21 out of 60 is failed (threshold=3) [...X...XX.......X....X..XX.X.XXXX..X..X.....X.XXX..X...XX...]
duration: 
5.655791588s
trace: 
https://jaeger.dev.storj.io/trace/7a3b11897bee958b
FAIL (2025-12-27 14:11:47 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 27 14:09:21.611393571 (526h40m21.482476616s) [upload=461959]: Starting upload
Dec 27 14:09:21.611546842 (526h40m21.482629877s) [upload=461959]: Done starting upload
Dec 27 14:09:21.611609071 (526h40m21.482692116s) [upload=461959]: Uploading segments...
Dec 27 14:09:21.612269158 (526h40m21.483352213s) [upload=461959]: Got next segment. Inline: false
Dec 27 14:09:21.612349201 (526h40m21.483432256s) [upload=461959]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 14:09:21.612356555 (526h40m21.4834396s) [upload=461959]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 14:09:21.856299604 (526h40m21.727382649s) [seg_pos={0 0}, upload=461959]: Begin upload segment...
Dec 27 14:09:21.856370038 (526h40m21.727453073s) [seg_pos={0 0}, upload=461959]: Done begin upload segment.
Dec 27 14:09:21.856420784 (526h40m21.727503819s) [noise=true , address=145.79.227.12:4765   , piece_id=5YKAMK34JOHYLZJT2D5WRZBY6N43346GSRFYFXHWZ6Z3AWWAQJ6A, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.856479476 (526h40m21.727562521s) [noise=true , address=145.79.227.29:4942   , piece_id=R6L233G7TZUNMLBODN4UEJH66YDWXL66B2WGAVQPHCIH5JXC74FA, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.857625017 (526h40m21.728708052s) [noise=true , address=45.91.116.13:4155    , piece_id=LBAG6T3GGWKHCOHA7T53VKSOX2O55KKQ4EGMRKDCK6PGFCTUIJVQ, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.858100663 (526h40m21.729183698s) [noise=true , address=145.79.227.10:4402   , piece_id=XUDYA3V3NLMFVKRX5ERUSILGUIOLX6NJNXWIL2PZDT6GFJMA3NDA, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.858170176 (526h40m21.729253231s) [noise=true , address=45.91.116.7:4009     , piece_id=US7DNVKAFLNLTFQB45RKVON242QU3RF3T5IELTT2WCNMUN6ZAYBQ, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.858402087 (526h40m21.729485122s) [noise=true , address=45.91.116.11:4120    , piece_id=UXEDL7GFL64HCRQ67THFOC2MKAVGYREYNQS7WEXJYGBOY3H7EYWQ, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.858794865 (526h40m21.7298779s) [noise=true , address=77.90.31.17:4773     , piece_id=5JIB2CV4VJSBBFINGJ4WCWLQN4H2T4WSEL6XDB3A755TJQKQOKYQ, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.859040964 (526h40m21.730124009s) [noise=true , address=145.79.227.19:4078   , piece_id=UAMIORNB3U6JKMHUIQRNRUNYW4T7S6YHERJG7ZVIHC5KLEXXCF6Q, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.861213120 (526h40m21.732296205s) [noise=true , address=145.79.227.45:4230   , piece_id=CPFKFQEL7BVP3V6I2LCDPVCGQACW3RDR7QT2V7XIJS77H6ZU7RIA, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.862069922 (526h40m21.733152957s) [noise=true , address=145.79.227.50:4375   , piece_id=XUAHWMEE2ZKSLFO3Z6CC7CZJZOSDSMNMDXYVGVK5ZLO2TDYZCH4Q, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.863841424 (526h40m21.734924469s) [noise=true , address=145.79.227.20:4438   , piece_id=42J67NFOF35HJMHKU2FLTASKR2TGKIY6NRBISTOIDIWBY64ACWSQ, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.867587077 (526h40m21.738670122s) [noise=true , address=145.79.227.42:4123   , piece_id=K6WUIK2EKFXSGSUXIOCSV4R3MOB2S4HPLEXOJRAKMCUC2GWYTEYA, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.869214605 (526h40m21.74029765s) [noise=true , address=45.91.116.6:4974     , piece_id=QXEEQHFYAH67J6OSDKWDVJCJV2S7ENNTQUWGV6TO5MMLTQ7JSOBQ, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.870424339 (526h40m21.741507384s) [noise=true , address=145.79.227.23:4546   , piece_id=YAGKC7A2NH6QNUCYJH3TIGQXTZZPDQ4FZTBR6LLAXSYMJTFAMI7A, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.871897163 (526h40m21.742980208s) [noise=true , address=145.79.227.17:4970   , piece_id=2SEU5CWQM23OWEPWAGUO34UBDJKT7TES2N3TZEIDIWDCCUBW5NEQ, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.872851330 (526h40m21.743934376s) [noise=true , address=145.79.227.36:4519   , piece_id=TVVXVEG5UT3CDL5AWEU4M2TQTOUTIV2RPCGBOGDCD2SRJIE4KBVA, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:23.532718349 (526h40m23.403801384s) [upload=461959]: Next returned nil segment
Dec 27 14:09:23.532735001 (526h40m23.403818047s) [upload=461959]: Waiting for error group managing segments...
Dec 27 14:09:23.862990534 (526h40m23.734073569s) [noise=true , address=145.79.227.12:4765   , piece_id=5YKAMK34JOHYLZJT2D5WRZBY6N43346GSRFYFXHWZ6Z3AWWAQJ6A, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:23.871865954 (526h40m23.742948989s) [noise=true , address=145.79.227.50:4375   , piece_id=XUAHWMEE2ZKSLFO3Z6CC7CZJZOSDSMNMDXYVGVK5ZLO2TDYZCH4Q, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:23.874552059 (526h40m23.745635104s) [noise=true , address=145.79.227.36:4519   , piece_id=TVVXVEG5UT3CDL5AWEU4M2TQTOUTIV2RPCGBOGDCD2SRJIE4KBVA, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:23.875081458 (526h40m23.746164493s) [noise=true , address=145.79.227.29:4942   , piece_id=R6L233G7TZUNMLBODN4UEJH66YDWXL66B2WGAVQPHCIH5JXC74FA, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:23.880889777 (526h40m23.751972812s) [noise=true , address=145.79.227.17:4970   , piece_id=2SEU5CWQM23OWEPWAGUO34UBDJKT7TES2N3TZEIDIWDCCUBW5NEQ, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:23.881779612 (526h40m23.752862657s) [noise=true , address=145.79.227.23:4546   , piece_id=YAGKC7A2NH6QNUCYJH3TIGQXTZZPDQ4FZTBR6LLAXSYMJTFAMI7A, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:23.888373497 (526h40m23.759456542s) [noise=true , address=145.79.227.10:4402   , piece_id=XUDYA3V3NLMFVKRX5ERUSILGUIOLX6NJNXWIL2PZDT6GFJMA3NDA, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:23.896892469 (526h40m23.767975504s) [noise=true , address=145.79.227.42:4123   , piece_id=K6WUIK2EKFXSGSUXIOCSV4R3MOB2S4HPLEXOJRAKMCUC2GWYTEYA, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:23.899060166 (526h40m23.770143202s) [noise=true , address=145.79.227.20:4438   , piece_id=42J67NFOF35HJMHKU2FLTASKR2TGKIY6NRBISTOIDIWBY64ACWSQ, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:23.899525874 (526h40m23.770608919s) [noise=true , address=145.79.227.45:4230   , piece_id=CPFKFQEL7BVP3V6I2LCDPVCGQACW3RDR7QT2V7XIJS77H6ZU7RIA, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:23.907289918 (526h40m23.778372953s) [noise=true , address=145.79.227.19:4078   , piece_id=UAMIORNB3U6JKMHUIQRNRUNYW4T7S6YHERJG7ZVIHC5KLEXXCF6Q, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:24.122094452 (526h40m23.993177497s) [noise=true , address=45.91.116.11:4120    , piece_id=UXEDL7GFL64HCRQ67THFOC2MKAVGYREYNQS7WEXJYGBOY3H7EYWQ, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:24.166811121 (526h40m24.037894166s) [noise=true , address=77.90.31.17:4773     , piece_id=5JIB2CV4VJSBBFINGJ4WCWLQN4H2T4WSEL6XDB3A755TJQKQOKYQ, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:24.166864091 (526h40m24.037947126s) [seg_pos={0 0}, upload=461959]: Segment reached optimal threshold of 13 pieces.
Dec 27 14:09:24.167067258 (526h40m24.038150303s) [noise=true , address=45.91.116.7:4009     , piece_id=US7DNVKAFLNLTFQB45RKVON242QU3RF3T5IELTT2WCNMUN6ZAYBQ, seg_pos={0 0}, upload=461959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF
Dec 27 14:09:24.167077768 (526h40m24.038160823s) [noise=true , address=45.91.116.7:4009     , piece_id=US7DNVKAFLNLTFQB45RKVON242QU3RF3T5IELTT2WCNMUN6ZAYBQ, seg_pos={0 0}, upload=461959]: Long tail cancellation due to optimal threshold
Dec 27 14:09:24.167395192 (526h40m24.038478237s) [noise=true , address=45.91.116.13:4155    , piece_id=LBAG6T3GGWKHCOHA7T53VKSOX2O55KKQ4EGMRKDCK6PGFCTUIJVQ, seg_pos={0 0}, upload=461959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF
Dec 27 14:09:24.167402437 (526h40m24.038485472s) [noise=true , address=45.91.116.13:4155    , piece_id=LBAG6T3GGWKHCOHA7T53VKSOX2O55KKQ4EGMRKDCK6PGFCTUIJVQ, seg_pos={0 0}, upload=461959]: Long tail cancellation due to optimal threshold
Dec 27 14:09:24.167513648 (526h40m24.038596694s) [noise=true , address=45.91.116.6:4974     , piece_id=QXEEQHFYAH67J6OSDKWDVJCJV2S7ENNTQUWGV6TO5MMLTQ7JSOBQ, seg_pos={0 0}, upload=461959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF
Dec 27 14:09:24.167519349 (526h40m24.038602384s) [noise=true , address=45.91.116.6:4974     , piece_id=QXEEQHFYAH67J6OSDKWDVJCJV2S7ENNTQUWGV6TO5MMLTQ7JSOBQ, seg_pos={0 0}, upload=461959]: Long tail cancellation due to optimal threshold
Dec 27 14:09:24.167532774 (526h40m24.038615809s) [seg_pos={0 0}, upload=461959]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 14:09:24.167612757 (526h40m24.038695792s) [upload=461959]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 14:09:24.167618839 (526h40m24.038701884s) [upload=461959]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 14:09:24.455792179 (526h40m24.326875214s) [upload=461959]: Done uploading segments...
Dec 27 14:09:24.455820133 (526h40m24.326903168s) [upload=461959]: Upload finished. err: <nil>
Duration: 5.655815944s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
40 out of 60 is failed (threshold=6) [X....X.X.X.X.X..XXX.XX.XX.X.XX.X.XXXXXXXXXXX.X..XXXX.XXXXXXX]
duration: 
5.001947267s
trace: 
https://jaeger.dev.storj.io/trace/4488be0ae555676a
FAIL (2025-12-27 14:11:34 UTC) 
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001971482s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
22 out of 60 is failed (threshold=3) [X...X...XX.......X....X..XX.X.XXXX..X..X.....X.XXX..X...XX..]
duration: 
5.619602279s
trace: 
https://jaeger.dev.storj.io/trace/7a3b11897bee958b
FAIL (2025-12-27 14:11:18 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 27 14:09:21.611393571 (526h40m21.482476616s) [upload=461959]: Starting upload
Dec 27 14:09:21.611546842 (526h40m21.482629877s) [upload=461959]: Done starting upload
Dec 27 14:09:21.611609071 (526h40m21.482692116s) [upload=461959]: Uploading segments...
Dec 27 14:09:21.612269158 (526h40m21.483352213s) [upload=461959]: Got next segment. Inline: false
Dec 27 14:09:21.612349201 (526h40m21.483432256s) [upload=461959]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 14:09:21.612356555 (526h40m21.4834396s) [upload=461959]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 14:09:21.856299604 (526h40m21.727382649s) [seg_pos={0 0}, upload=461959]: Begin upload segment...
Dec 27 14:09:21.856370038 (526h40m21.727453073s) [seg_pos={0 0}, upload=461959]: Done begin upload segment.
Dec 27 14:09:21.856420784 (526h40m21.727503819s) [noise=true , address=145.79.227.12:4765   , piece_id=5YKAMK34JOHYLZJT2D5WRZBY6N43346GSRFYFXHWZ6Z3AWWAQJ6A, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.856479476 (526h40m21.727562521s) [noise=true , address=145.79.227.29:4942   , piece_id=R6L233G7TZUNMLBODN4UEJH66YDWXL66B2WGAVQPHCIH5JXC74FA, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.857625017 (526h40m21.728708052s) [noise=true , address=45.91.116.13:4155    , piece_id=LBAG6T3GGWKHCOHA7T53VKSOX2O55KKQ4EGMRKDCK6PGFCTUIJVQ, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.858100663 (526h40m21.729183698s) [noise=true , address=145.79.227.10:4402   , piece_id=XUDYA3V3NLMFVKRX5ERUSILGUIOLX6NJNXWIL2PZDT6GFJMA3NDA, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.858170176 (526h40m21.729253231s) [noise=true , address=45.91.116.7:4009     , piece_id=US7DNVKAFLNLTFQB45RKVON242QU3RF3T5IELTT2WCNMUN6ZAYBQ, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.858402087 (526h40m21.729485122s) [noise=true , address=45.91.116.11:4120    , piece_id=UXEDL7GFL64HCRQ67THFOC2MKAVGYREYNQS7WEXJYGBOY3H7EYWQ, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.858794865 (526h40m21.7298779s) [noise=true , address=77.90.31.17:4773     , piece_id=5JIB2CV4VJSBBFINGJ4WCWLQN4H2T4WSEL6XDB3A755TJQKQOKYQ, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.859040964 (526h40m21.730124009s) [noise=true , address=145.79.227.19:4078   , piece_id=UAMIORNB3U6JKMHUIQRNRUNYW4T7S6YHERJG7ZVIHC5KLEXXCF6Q, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.861213120 (526h40m21.732296205s) [noise=true , address=145.79.227.45:4230   , piece_id=CPFKFQEL7BVP3V6I2LCDPVCGQACW3RDR7QT2V7XIJS77H6ZU7RIA, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.862069922 (526h40m21.733152957s) [noise=true , address=145.79.227.50:4375   , piece_id=XUAHWMEE2ZKSLFO3Z6CC7CZJZOSDSMNMDXYVGVK5ZLO2TDYZCH4Q, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.863841424 (526h40m21.734924469s) [noise=true , address=145.79.227.20:4438   , piece_id=42J67NFOF35HJMHKU2FLTASKR2TGKIY6NRBISTOIDIWBY64ACWSQ, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.867587077 (526h40m21.738670122s) [noise=true , address=145.79.227.42:4123   , piece_id=K6WUIK2EKFXSGSUXIOCSV4R3MOB2S4HPLEXOJRAKMCUC2GWYTEYA, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.869214605 (526h40m21.74029765s) [noise=true , address=45.91.116.6:4974     , piece_id=QXEEQHFYAH67J6OSDKWDVJCJV2S7ENNTQUWGV6TO5MMLTQ7JSOBQ, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.870424339 (526h40m21.741507384s) [noise=true , address=145.79.227.23:4546   , piece_id=YAGKC7A2NH6QNUCYJH3TIGQXTZZPDQ4FZTBR6LLAXSYMJTFAMI7A, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.871897163 (526h40m21.742980208s) [noise=true , address=145.79.227.17:4970   , piece_id=2SEU5CWQM23OWEPWAGUO34UBDJKT7TES2N3TZEIDIWDCCUBW5NEQ, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.872851330 (526h40m21.743934376s) [noise=true , address=145.79.227.36:4519   , piece_id=TVVXVEG5UT3CDL5AWEU4M2TQTOUTIV2RPCGBOGDCD2SRJIE4KBVA, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:23.532718349 (526h40m23.403801384s) [upload=461959]: Next returned nil segment
Dec 27 14:09:23.532735001 (526h40m23.403818047s) [upload=461959]: Waiting for error group managing segments...
Dec 27 14:09:23.862990534 (526h40m23.734073569s) [noise=true , address=145.79.227.12:4765   , piece_id=5YKAMK34JOHYLZJT2D5WRZBY6N43346GSRFYFXHWZ6Z3AWWAQJ6A, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:23.871865954 (526h40m23.742948989s) [noise=true , address=145.79.227.50:4375   , piece_id=XUAHWMEE2ZKSLFO3Z6CC7CZJZOSDSMNMDXYVGVK5ZLO2TDYZCH4Q, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:23.874552059 (526h40m23.745635104s) [noise=true , address=145.79.227.36:4519   , piece_id=TVVXVEG5UT3CDL5AWEU4M2TQTOUTIV2RPCGBOGDCD2SRJIE4KBVA, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:23.875081458 (526h40m23.746164493s) [noise=true , address=145.79.227.29:4942   , piece_id=R6L233G7TZUNMLBODN4UEJH66YDWXL66B2WGAVQPHCIH5JXC74FA, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:23.880889777 (526h40m23.751972812s) [noise=true , address=145.79.227.17:4970   , piece_id=2SEU5CWQM23OWEPWAGUO34UBDJKT7TES2N3TZEIDIWDCCUBW5NEQ, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:23.881779612 (526h40m23.752862657s) [noise=true , address=145.79.227.23:4546   , piece_id=YAGKC7A2NH6QNUCYJH3TIGQXTZZPDQ4FZTBR6LLAXSYMJTFAMI7A, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:23.888373497 (526h40m23.759456542s) [noise=true , address=145.79.227.10:4402   , piece_id=XUDYA3V3NLMFVKRX5ERUSILGUIOLX6NJNXWIL2PZDT6GFJMA3NDA, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:23.896892469 (526h40m23.767975504s) [noise=true , address=145.79.227.42:4123   , piece_id=K6WUIK2EKFXSGSUXIOCSV4R3MOB2S4HPLEXOJRAKMCUC2GWYTEYA, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:23.899060166 (526h40m23.770143202s) [noise=true , address=145.79.227.20:4438   , piece_id=42J67NFOF35HJMHKU2FLTASKR2TGKIY6NRBISTOIDIWBY64ACWSQ, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:23.899525874 (526h40m23.770608919s) [noise=true , address=145.79.227.45:4230   , piece_id=CPFKFQEL7BVP3V6I2LCDPVCGQACW3RDR7QT2V7XIJS77H6ZU7RIA, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:23.907289918 (526h40m23.778372953s) [noise=true , address=145.79.227.19:4078   , piece_id=UAMIORNB3U6JKMHUIQRNRUNYW4T7S6YHERJG7ZVIHC5KLEXXCF6Q, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:24.122094452 (526h40m23.993177497s) [noise=true , address=45.91.116.11:4120    , piece_id=UXEDL7GFL64HCRQ67THFOC2MKAVGYREYNQS7WEXJYGBOY3H7EYWQ, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:24.166811121 (526h40m24.037894166s) [noise=true , address=77.90.31.17:4773     , piece_id=5JIB2CV4VJSBBFINGJ4WCWLQN4H2T4WSEL6XDB3A755TJQKQOKYQ, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:24.166864091 (526h40m24.037947126s) [seg_pos={0 0}, upload=461959]: Segment reached optimal threshold of 13 pieces.
Dec 27 14:09:24.167067258 (526h40m24.038150303s) [noise=true , address=45.91.116.7:4009     , piece_id=US7DNVKAFLNLTFQB45RKVON242QU3RF3T5IELTT2WCNMUN6ZAYBQ, seg_pos={0 0}, upload=461959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF
Dec 27 14:09:24.167077768 (526h40m24.038160823s) [noise=true , address=45.91.116.7:4009     , piece_id=US7DNVKAFLNLTFQB45RKVON242QU3RF3T5IELTT2WCNMUN6ZAYBQ, seg_pos={0 0}, upload=461959]: Long tail cancellation due to optimal threshold
Dec 27 14:09:24.167395192 (526h40m24.038478237s) [noise=true , address=45.91.116.13:4155    , piece_id=LBAG6T3GGWKHCOHA7T53VKSOX2O55KKQ4EGMRKDCK6PGFCTUIJVQ, seg_pos={0 0}, upload=461959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF
Dec 27 14:09:24.167402437 (526h40m24.038485472s) [noise=true , address=45.91.116.13:4155    , piece_id=LBAG6T3GGWKHCOHA7T53VKSOX2O55KKQ4EGMRKDCK6PGFCTUIJVQ, seg_pos={0 0}, upload=461959]: Long tail cancellation due to optimal threshold
Dec 27 14:09:24.167513648 (526h40m24.038596694s) [noise=true , address=45.91.116.6:4974     , piece_id=QXEEQHFYAH67J6OSDKWDVJCJV2S7ENNTQUWGV6TO5MMLTQ7JSOBQ, seg_pos={0 0}, upload=461959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF
Dec 27 14:09:24.167519349 (526h40m24.038602384s) [noise=true , address=45.91.116.6:4974     , piece_id=QXEEQHFYAH67J6OSDKWDVJCJV2S7ENNTQUWGV6TO5MMLTQ7JSOBQ, seg_pos={0 0}, upload=461959]: Long tail cancellation due to optimal threshold
Dec 27 14:09:24.167532774 (526h40m24.038615809s) [seg_pos={0 0}, upload=461959]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 14:09:24.167612757 (526h40m24.038695792s) [upload=461959]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 14:09:24.167618839 (526h40m24.038701884s) [upload=461959]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 14:09:24.455792179 (526h40m24.326875214s) [upload=461959]: Done uploading segments...
Dec 27 14:09:24.455820133 (526h40m24.326903168s) [upload=461959]: Upload finished. err: <nil>
Duration: 5.619634349s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip: 
2 out of 60 is failed (threshold=3) [...........................................XX...............]
duration: 
19.110660624s
trace: 
https://jaeger.dev.storj.io/trace/75ef8b5e6bc2819a
FAIL (2025-12-27 14:11:00 UTC) 
orbiter warning: context deadline exceeded
Duration: 19.110676795s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
40 out of 60 is failed (threshold=6) [XX....X.X.X.X.X..XXX.XX.XX.X.XX.X.XXXXXXXXXXX.X..XXXX.XXXXXX]
duration: 
5.00177058s
trace: 
https://jaeger.dev.storj.io/trace/78c6df6a2a69a6d
FAIL (2025-12-27 14:11:00 UTC) 
orbiter warning: context deadline exceeded
Duration: 5.001793253s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip: 
2 out of 60 is failed (threshold=3) [............................................XX..............]
duration: 
17.858607927s
trace: 
https://jaeger.dev.storj.io/trace/75ef8b5e6bc2819a
FAIL (2025-12-27 14:10:41 UTC) 
orbiter warning: context deadline exceeded
Duration: 17.858852463s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
4 out of 60 is failed (threshold=3) [..X..........X.............XX...............................]
duration: 
35.350669181s
trace: 
https://jaeger.dev.storj.io/trace/3ed1e657d506c4ee
FAIL (2025-12-27 14:10:28 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, Piece ID: HEMLEGWLVMK4LG5DM3PUWU5HSUEHKBECYOMHUXIQRGZGNXPXFLBQ) context deadline exceeded; piecestore: (Node ID: 12wXqePK1kK7iBo6wHcbqkRmkED8LGJ3rzQNJAWa14cHYhx1oyB, Piece ID: M5NU2MR5LQWOKNLVKKEJXMJNVEIZVDDNIZHWNN7XXHVTJRS2OQNA) context deadline exceeded; piecestore: (Node ID: 12ATtVD3XyFxzEqwQpSF4t1MWygqEa4Y1CkNj1s43eud8UtuSmx, Piece ID: ET7O467BME3OED32PFLJKYVIWZRDF6WMPXUT34OFWSPXLSMKKOWQ) context deadline exceeded; piecestore: (Node ID: 12UjFkFn2UEvr57LDorcijDCvkBkkKitgZEFx6Yqfk5gMStQvJG, Piece ID: SF3IKAIYXQR4YPWNVXT66SEHVM54R3K3VJHA6L5RBOU3DITOR5LQ) context deadline exceeded
======LOGS======
) [noise=true , address=77.90.31.16:4739     , piece_id=H6RNS5VOYTMWX2N6GLV2KOQDPFPCBWPDNLINB7SXDJWHMJL3GYQQ, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.319646322 (526h14m34.190729357s) [noise=true , address=77.90.31.21:4899     , piece_id=SF3IKAIYXQR4YPWNVXT66SEHVM54R3K3VJHA6L5RBOU3DITOR5LQ, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.319933248 (526h14m34.191016283s) [noise=true , address=45.91.116.12:4234    , piece_id=Y2FIVYLZBQAYBO2TDGIZ4RDQHQJ3WOQH565A7ESIKANQQTQDFQZQ, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.320400588 (526h14m34.191483633s) [noise=true , address=145.79.227.29:4952   , piece_id=VXZLF4OGRIR3MYQMMLKLTD235A55F4YTHB3D4YCVCQUL4FZTSFJQ, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.320407341 (526h14m34.191490386s) [noise=true , address=145.79.227.46:4621   , piece_id=XZ4MKAZAGQKVEJJGSZ5QGT4AT3MFXXNMXICVFCI6SP3E4EVEHZGQ, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.320675030 (526h14m34.191758075s) [noise=true , address=45.91.116.10:4113    , piece_id=OFHVWJWEHSVOOONDLXBZY72VB5G6FFXH2NXL45KY7TJ6CPVZ5UTQ, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.320802022 (526h14m34.191885067s) [noise=true , address=77.90.31.11:4569     , piece_id=YC2Q3HPJDVY3NEDZNC2Z4KVN6S6DDDGRARASF7KALFAOQ335A7HA, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.321169643 (526h14m34.192252678s) [noise=true , address=77.90.31.19:4869     , piece_id=HEMLEGWLVMK4LG5DM3PUWU5HSUEHKBECYOMHUXIQRGZGNXPXFLBQ, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.321443483 (526h14m34.192526518s) [noise=true , address=45.91.116.14:4202    , piece_id=M5NU2MR5LQWOKNLVKKEJXMJNVEIZVDDNIZHWNN7XXHVTJRS2OQNA, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.321731813 (526h14m34.192814848s) [noise=true , address=77.90.31.18:4819     , piece_id=YS4DXTHLGCPFOKKLBA2DK2BVOAPTQEGVOPBSRCLFV26SVP6YFHGA, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.322221776 (526h14m34.193304811s) [noise=true , address=77.90.31.15:4708     , piece_id=3NX6E54XMZXQAQQE27LTWUSEF5GAIF2XOPQFFAP7LOA2PL7KDL3Q, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.322547776 (526h14m34.193630811s) [noise=true , address=145.79.227.40:4079   , piece_id=237LYFTAG7G4HMIUKFZDDUTYJC7AAHHXTO5EIO46UEHY24BOVB3A, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.322908093 (526h14m34.193991138s) [noise=true , address=145.79.227.29:4975   , piece_id=DAMNBVP7DRSW5OAGYRRP4K5SNJ2WEUT7IAXO77KWRHLQEMBNERZQ, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.323267968 (526h14m34.194351003s) [noise=true , address=145.79.227.30:4787   , piece_id=OB2OTY3ZEHIUFYM6FY7HIS7HQ353Y2HL6XU4476HXOD26QVA4DPQ, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.323618165 (526h14m34.19470121s) [noise=true , address=64.49.13.169:4710    , piece_id=XIZKYK2E5FQJ37LYN535M7GXBBKXUQXFQWVMBWIBFDOAIIFGUAJA, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.324084893 (526h14m34.195167928s) [noise=true , address=193.149.164.143:4694 , piece_id=ET7O467BME3OED32PFLJKYVIWZRDF6WMPXUT34OFWSPXLSMKKOWQ, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:35.783540136 (526h14m35.654623181s) [upload=461595]: Next returned nil segment
Dec 27 13:43:35.783555856 (526h14m35.654638891s) [upload=461595]: Waiting for error group managing segments...
Dec 27 13:43:36.192287250 (526h14m36.063370296s) [noise=true , address=145.79.227.29:4952   , piece_id=VXZLF4OGRIR3MYQMMLKLTD235A55F4YTHB3D4YCVCQUL4FZTSFJQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:36.200200289 (526h14m36.071283334s) [noise=true , address=145.79.227.46:4621   , piece_id=XZ4MKAZAGQKVEJJGSZ5QGT4AT3MFXXNMXICVFCI6SP3E4EVEHZGQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:36.238087532 (526h14m36.109170577s) [noise=true , address=145.79.227.29:4975   , piece_id=DAMNBVP7DRSW5OAGYRRP4K5SNJ2WEUT7IAXO77KWRHLQEMBNERZQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:36.379631346 (526h14m36.250714391s) [noise=true , address=77.90.31.19:4869     , piece_id=HEMLEGWLVMK4LG5DM3PUWU5HSUEHKBECYOMHUXIQRGZGNXPXFLBQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:36.382945286 (526h14m36.254028321s) [noise=true , address=145.79.227.34:4909   , piece_id=Z6RF5PSOGOZMOXPQYXHMXGFXIELVXXVU4NXXRTAQCIELQXLMVXBQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:36.400290103 (526h14m36.271373138s) [noise=true , address=77.90.31.18:4819     , piece_id=YS4DXTHLGCPFOKKLBA2DK2BVOAPTQEGVOPBSRCLFV26SVP6YFHGA, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:36.436707208 (526h14m36.307790243s) [noise=true , address=45.91.116.14:4202    , piece_id=M5NU2MR5LQWOKNLVKKEJXMJNVEIZVDDNIZHWNN7XXHVTJRS2OQNA, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:36.487651475 (526h14m36.35873451s) [noise=true , address=45.91.116.12:4234    , piece_id=Y2FIVYLZBQAYBO2TDGIZ4RDQHQJ3WOQH565A7ESIKANQQTQDFQZQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:36.500850769 (526h14m36.371933814s) [noise=true , address=77.90.31.11:4569     , piece_id=YC2Q3HPJDVY3NEDZNC2Z4KVN6S6DDDGRARASF7KALFAOQ335A7HA, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:36.518828551 (526h14m36.389911586s) [noise=true , address=77.90.31.15:4708     , piece_id=3NX6E54XMZXQAQQE27LTWUSEF5GAIF2XOPQFFAP7LOA2PL7KDL3Q, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:37.051607243 (526h14m36.922690289s) [noise=true , address=145.79.227.44:4228   , piece_id=6SAYEAOTTCOUFPU75C7YRIULWJGXL4PK2YKCKFB47EUBWMRQQQOQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:37.096826909 (526h14m36.967909954s) [noise=true , address=145.79.227.30:4787   , piece_id=OB2OTY3ZEHIUFYM6FY7HIS7HQ353Y2HL6XU4476HXOD26QVA4DPQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:37.140738995 (526h14m37.01182203s) [noise=true , address=77.90.31.21:4899     , piece_id=SF3IKAIYXQR4YPWNVXT66SEHVM54R3K3VJHA6L5RBOU3DITOR5LQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:37.323613990 (526h14m37.194697035s) [noise=true , address=193.149.164.143:4694 , piece_id=ET7O467BME3OED32PFLJKYVIWZRDF6WMPXUT34OFWSPXLSMKKOWQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:37.323649618 (526h14m37.194732663s) [seg_pos={0 0}, upload=461595]: Segment reached optimal threshold of 14 pieces.
Dec 27 13:43:37.323852865 (526h14m37.1949359s) [noise=true , address=45.91.116.10:4113    , piece_id=OFHVWJWEHSVOOONDLXBZY72VB5G6FFXH2NXL45KY7TJ6CPVZ5UTQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12puRMny4nbgv2by4tAbXcBBSgG2d6BfwcDDRDcFzEdjSVUJxtd): protocol: expected piece hash; context canceled; EOF
Dec 27 13:43:37.323860349 (526h14m37.194943384s) [noise=true , address=45.91.116.10:4113    , piece_id=OFHVWJWEHSVOOONDLXBZY72VB5G6FFXH2NXL45KY7TJ6CPVZ5UTQ, seg_pos={0 0}, upload=461595]: Long tail cancellation due to optimal threshold
Dec 27 13:43:37.324005135 (526h14m37.1950882s) [noise=true , address=145.79.227.40:4079   , piece_id=237LYFTAG7G4HMIUKFZDDUTYJC7AAHHXTO5EIO46UEHY24BOVB3A, seg_pos={0 0}, upload=461595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RYKzn5zVSmxVbWwnBdX8oqtG1WE2MhPeEiWLf5dZeFAut6s2Y): protocol: expected piece hash; context canceled; EOF
Dec 27 13:43:37.324022659 (526h14m37.195105704s) [noise=true , address=145.79.227.40:4079   , piece_id=237LYFTAG7G4HMIUKFZDDUTYJC7AAHHXTO5EIO46UEHY24BOVB3A, seg_pos={0 0}, upload=461595]: Long tail cancellation due to optimal threshold
Dec 27 13:43:37.324143378 (526h14m37.195226413s) [noise=true , address=77.90.31.16:4739     , piece_id=H6RNS5VOYTMWX2N6GLV2KOQDPFPCBWPDNLINB7SXDJWHMJL3GYQQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129B3N7yjJvwCzoVLTqWsPbcLQ5TQUyAzg1RCkbSAVKPWYTG8eb): protocol: expected piece hash; context canceled; EOF
Dec 27 13:43:37.324154520 (526h14m37.195237555s) [noise=true , address=77.90.31.16:4739     , piece_id=H6RNS5VOYTMWX2N6GLV2KOQDPFPCBWPDNLINB7SXDJWHMJL3GYQQ, seg_pos={0 0}, upload=461595]: Long tail cancellation due to optimal threshold
Dec 27 13:43:37.324213442 (526h14m37.195296487s) [noise=true , address=64.49.13.169:4710    , piece_id=XIZKYK2E5FQJ37LYN535M7GXBBKXUQXFQWVMBWIBFDOAIIFGUAJA, seg_pos={0 0}, upload=461595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:57438->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:57438->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57438->64.49.13.169:4710: use of closed network connection
Dec 27 13:43:37.324249360 (526h14m37.195332405s) [noise=true , address=64.49.13.169:4710    , piece_id=XIZKYK2E5FQJ37LYN535M7GXBBKXUQXFQWVMBWIBFDOAIIFGUAJA, seg_pos={0 0}, upload=461595]: Long tail cancellation due to optimal threshold
Dec 27 13:43:37.324275360 (526h14m37.195358405s) [seg_pos={0 0}, upload=461595]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 13:43:37.324374018 (526h14m37.195457053s) [upload=461595]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 13:43:37.324381291 (526h14m37.195464337s) [upload=461595]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 13:43:37.679036406 (526h14m37.550119451s) [upload=461595]: Done uploading segments...
Dec 27 13:43:37.679056985 (526h14m37.55014003s) [upload=461595]: Upload finished. err: <nil>
Duration: 35.350691674s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
22 out of 60 is failed (threshold=3) [.X...X...XX.......X....X..XX.X.XXXX..X..X.....X.XXX..X...XX.]
duration: 
5.667101965s
trace: 
https://jaeger.dev.storj.io/trace/7a3b11897bee958b
FAIL (2025-12-27 14:10:26 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 27 14:09:21.611393571 (526h40m21.482476616s) [upload=461959]: Starting upload
Dec 27 14:09:21.611546842 (526h40m21.482629877s) [upload=461959]: Done starting upload
Dec 27 14:09:21.611609071 (526h40m21.482692116s) [upload=461959]: Uploading segments...
Dec 27 14:09:21.612269158 (526h40m21.483352213s) [upload=461959]: Got next segment. Inline: false
Dec 27 14:09:21.612349201 (526h40m21.483432256s) [upload=461959]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 14:09:21.612356555 (526h40m21.4834396s) [upload=461959]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 14:09:21.856299604 (526h40m21.727382649s) [seg_pos={0 0}, upload=461959]: Begin upload segment...
Dec 27 14:09:21.856370038 (526h40m21.727453073s) [seg_pos={0 0}, upload=461959]: Done begin upload segment.
Dec 27 14:09:21.856420784 (526h40m21.727503819s) [noise=true , address=145.79.227.12:4765   , piece_id=5YKAMK34JOHYLZJT2D5WRZBY6N43346GSRFYFXHWZ6Z3AWWAQJ6A, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.856479476 (526h40m21.727562521s) [noise=true , address=145.79.227.29:4942   , piece_id=R6L233G7TZUNMLBODN4UEJH66YDWXL66B2WGAVQPHCIH5JXC74FA, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.857625017 (526h40m21.728708052s) [noise=true , address=45.91.116.13:4155    , piece_id=LBAG6T3GGWKHCOHA7T53VKSOX2O55KKQ4EGMRKDCK6PGFCTUIJVQ, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.858100663 (526h40m21.729183698s) [noise=true , address=145.79.227.10:4402   , piece_id=XUDYA3V3NLMFVKRX5ERUSILGUIOLX6NJNXWIL2PZDT6GFJMA3NDA, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.858170176 (526h40m21.729253231s) [noise=true , address=45.91.116.7:4009     , piece_id=US7DNVKAFLNLTFQB45RKVON242QU3RF3T5IELTT2WCNMUN6ZAYBQ, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.858402087 (526h40m21.729485122s) [noise=true , address=45.91.116.11:4120    , piece_id=UXEDL7GFL64HCRQ67THFOC2MKAVGYREYNQS7WEXJYGBOY3H7EYWQ, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.858794865 (526h40m21.7298779s) [noise=true , address=77.90.31.17:4773     , piece_id=5JIB2CV4VJSBBFINGJ4WCWLQN4H2T4WSEL6XDB3A755TJQKQOKYQ, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.859040964 (526h40m21.730124009s) [noise=true , address=145.79.227.19:4078   , piece_id=UAMIORNB3U6JKMHUIQRNRUNYW4T7S6YHERJG7ZVIHC5KLEXXCF6Q, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.861213120 (526h40m21.732296205s) [noise=true , address=145.79.227.45:4230   , piece_id=CPFKFQEL7BVP3V6I2LCDPVCGQACW3RDR7QT2V7XIJS77H6ZU7RIA, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.862069922 (526h40m21.733152957s) [noise=true , address=145.79.227.50:4375   , piece_id=XUAHWMEE2ZKSLFO3Z6CC7CZJZOSDSMNMDXYVGVK5ZLO2TDYZCH4Q, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.863841424 (526h40m21.734924469s) [noise=true , address=145.79.227.20:4438   , piece_id=42J67NFOF35HJMHKU2FLTASKR2TGKIY6NRBISTOIDIWBY64ACWSQ, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.867587077 (526h40m21.738670122s) [noise=true , address=145.79.227.42:4123   , piece_id=K6WUIK2EKFXSGSUXIOCSV4R3MOB2S4HPLEXOJRAKMCUC2GWYTEYA, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.869214605 (526h40m21.74029765s) [noise=true , address=45.91.116.6:4974     , piece_id=QXEEQHFYAH67J6OSDKWDVJCJV2S7ENNTQUWGV6TO5MMLTQ7JSOBQ, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.870424339 (526h40m21.741507384s) [noise=true , address=145.79.227.23:4546   , piece_id=YAGKC7A2NH6QNUCYJH3TIGQXTZZPDQ4FZTBR6LLAXSYMJTFAMI7A, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.871897163 (526h40m21.742980208s) [noise=true , address=145.79.227.17:4970   , piece_id=2SEU5CWQM23OWEPWAGUO34UBDJKT7TES2N3TZEIDIWDCCUBW5NEQ, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.872851330 (526h40m21.743934376s) [noise=true , address=145.79.227.36:4519   , piece_id=TVVXVEG5UT3CDL5AWEU4M2TQTOUTIV2RPCGBOGDCD2SRJIE4KBVA, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:23.532718349 (526h40m23.403801384s) [upload=461959]: Next returned nil segment
Dec 27 14:09:23.532735001 (526h40m23.403818047s) [upload=461959]: Waiting for error group managing segments...
Dec 27 14:09:23.862990534 (526h40m23.734073569s) [noise=true , address=145.79.227.12:4765   , piece_id=5YKAMK34JOHYLZJT2D5WRZBY6N43346GSRFYFXHWZ6Z3AWWAQJ6A, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:23.871865954 (526h40m23.742948989s) [noise=true , address=145.79.227.50:4375   , piece_id=XUAHWMEE2ZKSLFO3Z6CC7CZJZOSDSMNMDXYVGVK5ZLO2TDYZCH4Q, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:23.874552059 (526h40m23.745635104s) [noise=true , address=145.79.227.36:4519   , piece_id=TVVXVEG5UT3CDL5AWEU4M2TQTOUTIV2RPCGBOGDCD2SRJIE4KBVA, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:23.875081458 (526h40m23.746164493s) [noise=true , address=145.79.227.29:4942   , piece_id=R6L233G7TZUNMLBODN4UEJH66YDWXL66B2WGAVQPHCIH5JXC74FA, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:23.880889777 (526h40m23.751972812s) [noise=true , address=145.79.227.17:4970   , piece_id=2SEU5CWQM23OWEPWAGUO34UBDJKT7TES2N3TZEIDIWDCCUBW5NEQ, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:23.881779612 (526h40m23.752862657s) [noise=true , address=145.79.227.23:4546   , piece_id=YAGKC7A2NH6QNUCYJH3TIGQXTZZPDQ4FZTBR6LLAXSYMJTFAMI7A, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:23.888373497 (526h40m23.759456542s) [noise=true , address=145.79.227.10:4402   , piece_id=XUDYA3V3NLMFVKRX5ERUSILGUIOLX6NJNXWIL2PZDT6GFJMA3NDA, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:23.896892469 (526h40m23.767975504s) [noise=true , address=145.79.227.42:4123   , piece_id=K6WUIK2EKFXSGSUXIOCSV4R3MOB2S4HPLEXOJRAKMCUC2GWYTEYA, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:23.899060166 (526h40m23.770143202s) [noise=true , address=145.79.227.20:4438   , piece_id=42J67NFOF35HJMHKU2FLTASKR2TGKIY6NRBISTOIDIWBY64ACWSQ, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:23.899525874 (526h40m23.770608919s) [noise=true , address=145.79.227.45:4230   , piece_id=CPFKFQEL7BVP3V6I2LCDPVCGQACW3RDR7QT2V7XIJS77H6ZU7RIA, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:23.907289918 (526h40m23.778372953s) [noise=true , address=145.79.227.19:4078   , piece_id=UAMIORNB3U6JKMHUIQRNRUNYW4T7S6YHERJG7ZVIHC5KLEXXCF6Q, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:24.122094452 (526h40m23.993177497s) [noise=true , address=45.91.116.11:4120    , piece_id=UXEDL7GFL64HCRQ67THFOC2MKAVGYREYNQS7WEXJYGBOY3H7EYWQ, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:24.166811121 (526h40m24.037894166s) [noise=true , address=77.90.31.17:4773     , piece_id=5JIB2CV4VJSBBFINGJ4WCWLQN4H2T4WSEL6XDB3A755TJQKQOKYQ, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:24.166864091 (526h40m24.037947126s) [seg_pos={0 0}, upload=461959]: Segment reached optimal threshold of 13 pieces.
Dec 27 14:09:24.167067258 (526h40m24.038150303s) [noise=true , address=45.91.116.7:4009     , piece_id=US7DNVKAFLNLTFQB45RKVON242QU3RF3T5IELTT2WCNMUN6ZAYBQ, seg_pos={0 0}, upload=461959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF
Dec 27 14:09:24.167077768 (526h40m24.038160823s) [noise=true , address=45.91.116.7:4009     , piece_id=US7DNVKAFLNLTFQB45RKVON242QU3RF3T5IELTT2WCNMUN6ZAYBQ, seg_pos={0 0}, upload=461959]: Long tail cancellation due to optimal threshold
Dec 27 14:09:24.167395192 (526h40m24.038478237s) [noise=true , address=45.91.116.13:4155    , piece_id=LBAG6T3GGWKHCOHA7T53VKSOX2O55KKQ4EGMRKDCK6PGFCTUIJVQ, seg_pos={0 0}, upload=461959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF
Dec 27 14:09:24.167402437 (526h40m24.038485472s) [noise=true , address=45.91.116.13:4155    , piece_id=LBAG6T3GGWKHCOHA7T53VKSOX2O55KKQ4EGMRKDCK6PGFCTUIJVQ, seg_pos={0 0}, upload=461959]: Long tail cancellation due to optimal threshold
Dec 27 14:09:24.167513648 (526h40m24.038596694s) [noise=true , address=45.91.116.6:4974     , piece_id=QXEEQHFYAH67J6OSDKWDVJCJV2S7ENNTQUWGV6TO5MMLTQ7JSOBQ, seg_pos={0 0}, upload=461959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF
Dec 27 14:09:24.167519349 (526h40m24.038602384s) [noise=true , address=45.91.116.6:4974     , piece_id=QXEEQHFYAH67J6OSDKWDVJCJV2S7ENNTQUWGV6TO5MMLTQ7JSOBQ, seg_pos={0 0}, upload=461959]: Long tail cancellation due to optimal threshold
Dec 27 14:09:24.167532774 (526h40m24.038615809s) [seg_pos={0 0}, upload=461959]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 14:09:24.167612757 (526h40m24.038695792s) [upload=461959]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 14:09:24.167618839 (526h40m24.038701884s) [upload=461959]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 14:09:24.455792179 (526h40m24.326875214s) [upload=461959]: Done uploading segments...
Dec 27 14:09:24.455820133 (526h40m24.326903168s) [upload=461959]: Upload finished. err: <nil>
Duration: 5.667131912s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
39 out of 60 is failed (threshold=6) [.XX....X.X.X.X.X..XXX.XX.XX.X.XX.X.XXXXXXXXXXX.X..XXXX.XXXXX]
duration: 
5.002164311s
trace: 
https://jaeger.dev.storj.io/trace/3906425aedf15c2
FAIL (2025-12-27 14:10:23 UTC) 
orbiter warning: context deadline exceeded
Duration: 5.002198986s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip: 
2 out of 60 is failed (threshold=3) [.............................................XX.............]
duration: 
19.023844778s
trace: 
https://jaeger.dev.storj.io/trace/75ef8b5e6bc2819a
FAIL (2025-12-27 14:09:53 UTC) 
orbiter warning: context deadline exceeded
Duration: 19.023865306s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
4 out of 60 is failed (threshold=3) [...X..........X.............XX..............................]
duration: 
26.469509991s
trace: 
https://jaeger.dev.storj.io/trace/3ed1e657d506c4ee
FAIL (2025-12-27 14:09:52 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, Piece ID: HEMLEGWLVMK4LG5DM3PUWU5HSUEHKBECYOMHUXIQRGZGNXPXFLBQ) context deadline exceeded; piecestore: (Node ID: 12wXqePK1kK7iBo6wHcbqkRmkED8LGJ3rzQNJAWa14cHYhx1oyB, Piece ID: M5NU2MR5LQWOKNLVKKEJXMJNVEIZVDDNIZHWNN7XXHVTJRS2OQNA) context deadline exceeded; piecestore: (Node ID: 12ATtVD3XyFxzEqwQpSF4t1MWygqEa4Y1CkNj1s43eud8UtuSmx, Piece ID: ET7O467BME3OED32PFLJKYVIWZRDF6WMPXUT34OFWSPXLSMKKOWQ) context deadline exceeded; piecestore: (Node ID: 12UjFkFn2UEvr57LDorcijDCvkBkkKitgZEFx6Yqfk5gMStQvJG, Piece ID: SF3IKAIYXQR4YPWNVXT66SEHVM54R3K3VJHA6L5RBOU3DITOR5LQ) context deadline exceeded
======LOGS======
) [noise=true , address=77.90.31.16:4739     , piece_id=H6RNS5VOYTMWX2N6GLV2KOQDPFPCBWPDNLINB7SXDJWHMJL3GYQQ, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.319646322 (526h14m34.190729357s) [noise=true , address=77.90.31.21:4899     , piece_id=SF3IKAIYXQR4YPWNVXT66SEHVM54R3K3VJHA6L5RBOU3DITOR5LQ, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.319933248 (526h14m34.191016283s) [noise=true , address=45.91.116.12:4234    , piece_id=Y2FIVYLZBQAYBO2TDGIZ4RDQHQJ3WOQH565A7ESIKANQQTQDFQZQ, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.320400588 (526h14m34.191483633s) [noise=true , address=145.79.227.29:4952   , piece_id=VXZLF4OGRIR3MYQMMLKLTD235A55F4YTHB3D4YCVCQUL4FZTSFJQ, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.320407341 (526h14m34.191490386s) [noise=true , address=145.79.227.46:4621   , piece_id=XZ4MKAZAGQKVEJJGSZ5QGT4AT3MFXXNMXICVFCI6SP3E4EVEHZGQ, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.320675030 (526h14m34.191758075s) [noise=true , address=45.91.116.10:4113    , piece_id=OFHVWJWEHSVOOONDLXBZY72VB5G6FFXH2NXL45KY7TJ6CPVZ5UTQ, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.320802022 (526h14m34.191885067s) [noise=true , address=77.90.31.11:4569     , piece_id=YC2Q3HPJDVY3NEDZNC2Z4KVN6S6DDDGRARASF7KALFAOQ335A7HA, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.321169643 (526h14m34.192252678s) [noise=true , address=77.90.31.19:4869     , piece_id=HEMLEGWLVMK4LG5DM3PUWU5HSUEHKBECYOMHUXIQRGZGNXPXFLBQ, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.321443483 (526h14m34.192526518s) [noise=true , address=45.91.116.14:4202    , piece_id=M5NU2MR5LQWOKNLVKKEJXMJNVEIZVDDNIZHWNN7XXHVTJRS2OQNA, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.321731813 (526h14m34.192814848s) [noise=true , address=77.90.31.18:4819     , piece_id=YS4DXTHLGCPFOKKLBA2DK2BVOAPTQEGVOPBSRCLFV26SVP6YFHGA, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.322221776 (526h14m34.193304811s) [noise=true , address=77.90.31.15:4708     , piece_id=3NX6E54XMZXQAQQE27LTWUSEF5GAIF2XOPQFFAP7LOA2PL7KDL3Q, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.322547776 (526h14m34.193630811s) [noise=true , address=145.79.227.40:4079   , piece_id=237LYFTAG7G4HMIUKFZDDUTYJC7AAHHXTO5EIO46UEHY24BOVB3A, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.322908093 (526h14m34.193991138s) [noise=true , address=145.79.227.29:4975   , piece_id=DAMNBVP7DRSW5OAGYRRP4K5SNJ2WEUT7IAXO77KWRHLQEMBNERZQ, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.323267968 (526h14m34.194351003s) [noise=true , address=145.79.227.30:4787   , piece_id=OB2OTY3ZEHIUFYM6FY7HIS7HQ353Y2HL6XU4476HXOD26QVA4DPQ, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.323618165 (526h14m34.19470121s) [noise=true , address=64.49.13.169:4710    , piece_id=XIZKYK2E5FQJ37LYN535M7GXBBKXUQXFQWVMBWIBFDOAIIFGUAJA, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.324084893 (526h14m34.195167928s) [noise=true , address=193.149.164.143:4694 , piece_id=ET7O467BME3OED32PFLJKYVIWZRDF6WMPXUT34OFWSPXLSMKKOWQ, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:35.783540136 (526h14m35.654623181s) [upload=461595]: Next returned nil segment
Dec 27 13:43:35.783555856 (526h14m35.654638891s) [upload=461595]: Waiting for error group managing segments...
Dec 27 13:43:36.192287250 (526h14m36.063370296s) [noise=true , address=145.79.227.29:4952   , piece_id=VXZLF4OGRIR3MYQMMLKLTD235A55F4YTHB3D4YCVCQUL4FZTSFJQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:36.200200289 (526h14m36.071283334s) [noise=true , address=145.79.227.46:4621   , piece_id=XZ4MKAZAGQKVEJJGSZ5QGT4AT3MFXXNMXICVFCI6SP3E4EVEHZGQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:36.238087532 (526h14m36.109170577s) [noise=true , address=145.79.227.29:4975   , piece_id=DAMNBVP7DRSW5OAGYRRP4K5SNJ2WEUT7IAXO77KWRHLQEMBNERZQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:36.379631346 (526h14m36.250714391s) [noise=true , address=77.90.31.19:4869     , piece_id=HEMLEGWLVMK4LG5DM3PUWU5HSUEHKBECYOMHUXIQRGZGNXPXFLBQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:36.382945286 (526h14m36.254028321s) [noise=true , address=145.79.227.34:4909   , piece_id=Z6RF5PSOGOZMOXPQYXHMXGFXIELVXXVU4NXXRTAQCIELQXLMVXBQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:36.400290103 (526h14m36.271373138s) [noise=true , address=77.90.31.18:4819     , piece_id=YS4DXTHLGCPFOKKLBA2DK2BVOAPTQEGVOPBSRCLFV26SVP6YFHGA, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:36.436707208 (526h14m36.307790243s) [noise=true , address=45.91.116.14:4202    , piece_id=M5NU2MR5LQWOKNLVKKEJXMJNVEIZVDDNIZHWNN7XXHVTJRS2OQNA, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:36.487651475 (526h14m36.35873451s) [noise=true , address=45.91.116.12:4234    , piece_id=Y2FIVYLZBQAYBO2TDGIZ4RDQHQJ3WOQH565A7ESIKANQQTQDFQZQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:36.500850769 (526h14m36.371933814s) [noise=true , address=77.90.31.11:4569     , piece_id=YC2Q3HPJDVY3NEDZNC2Z4KVN6S6DDDGRARASF7KALFAOQ335A7HA, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:36.518828551 (526h14m36.389911586s) [noise=true , address=77.90.31.15:4708     , piece_id=3NX6E54XMZXQAQQE27LTWUSEF5GAIF2XOPQFFAP7LOA2PL7KDL3Q, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:37.051607243 (526h14m36.922690289s) [noise=true , address=145.79.227.44:4228   , piece_id=6SAYEAOTTCOUFPU75C7YRIULWJGXL4PK2YKCKFB47EUBWMRQQQOQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:37.096826909 (526h14m36.967909954s) [noise=true , address=145.79.227.30:4787   , piece_id=OB2OTY3ZEHIUFYM6FY7HIS7HQ353Y2HL6XU4476HXOD26QVA4DPQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:37.140738995 (526h14m37.01182203s) [noise=true , address=77.90.31.21:4899     , piece_id=SF3IKAIYXQR4YPWNVXT66SEHVM54R3K3VJHA6L5RBOU3DITOR5LQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:37.323613990 (526h14m37.194697035s) [noise=true , address=193.149.164.143:4694 , piece_id=ET7O467BME3OED32PFLJKYVIWZRDF6WMPXUT34OFWSPXLSMKKOWQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:37.323649618 (526h14m37.194732663s) [seg_pos={0 0}, upload=461595]: Segment reached optimal threshold of 14 pieces.
Dec 27 13:43:37.323852865 (526h14m37.1949359s) [noise=true , address=45.91.116.10:4113    , piece_id=OFHVWJWEHSVOOONDLXBZY72VB5G6FFXH2NXL45KY7TJ6CPVZ5UTQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12puRMny4nbgv2by4tAbXcBBSgG2d6BfwcDDRDcFzEdjSVUJxtd): protocol: expected piece hash; context canceled; EOF
Dec 27 13:43:37.323860349 (526h14m37.194943384s) [noise=true , address=45.91.116.10:4113    , piece_id=OFHVWJWEHSVOOONDLXBZY72VB5G6FFXH2NXL45KY7TJ6CPVZ5UTQ, seg_pos={0 0}, upload=461595]: Long tail cancellation due to optimal threshold
Dec 27 13:43:37.324005135 (526h14m37.1950882s) [noise=true , address=145.79.227.40:4079   , piece_id=237LYFTAG7G4HMIUKFZDDUTYJC7AAHHXTO5EIO46UEHY24BOVB3A, seg_pos={0 0}, upload=461595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RYKzn5zVSmxVbWwnBdX8oqtG1WE2MhPeEiWLf5dZeFAut6s2Y): protocol: expected piece hash; context canceled; EOF
Dec 27 13:43:37.324022659 (526h14m37.195105704s) [noise=true , address=145.79.227.40:4079   , piece_id=237LYFTAG7G4HMIUKFZDDUTYJC7AAHHXTO5EIO46UEHY24BOVB3A, seg_pos={0 0}, upload=461595]: Long tail cancellation due to optimal threshold
Dec 27 13:43:37.324143378 (526h14m37.195226413s) [noise=true , address=77.90.31.16:4739     , piece_id=H6RNS5VOYTMWX2N6GLV2KOQDPFPCBWPDNLINB7SXDJWHMJL3GYQQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129B3N7yjJvwCzoVLTqWsPbcLQ5TQUyAzg1RCkbSAVKPWYTG8eb): protocol: expected piece hash; context canceled; EOF
Dec 27 13:43:37.324154520 (526h14m37.195237555s) [noise=true , address=77.90.31.16:4739     , piece_id=H6RNS5VOYTMWX2N6GLV2KOQDPFPCBWPDNLINB7SXDJWHMJL3GYQQ, seg_pos={0 0}, upload=461595]: Long tail cancellation due to optimal threshold
Dec 27 13:43:37.324213442 (526h14m37.195296487s) [noise=true , address=64.49.13.169:4710    , piece_id=XIZKYK2E5FQJ37LYN535M7GXBBKXUQXFQWVMBWIBFDOAIIFGUAJA, seg_pos={0 0}, upload=461595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:57438->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:57438->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57438->64.49.13.169:4710: use of closed network connection
Dec 27 13:43:37.324249360 (526h14m37.195332405s) [noise=true , address=64.49.13.169:4710    , piece_id=XIZKYK2E5FQJ37LYN535M7GXBBKXUQXFQWVMBWIBFDOAIIFGUAJA, seg_pos={0 0}, upload=461595]: Long tail cancellation due to optimal threshold
Dec 27 13:43:37.324275360 (526h14m37.195358405s) [seg_pos={0 0}, upload=461595]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 13:43:37.324374018 (526h14m37.195457053s) [upload=461595]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 13:43:37.324381291 (526h14m37.195464337s) [upload=461595]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 13:43:37.679036406 (526h14m37.550119451s) [upload=461595]: Done uploading segments...
Dec 27 13:43:37.679056985 (526h14m37.55014003s) [upload=461595]: Upload finished. err: <nil>
Duration: 26.469531772s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
22 out of 60 is failed (threshold=3) [..X...X...XX.......X....X..XX.X.XXXX..X..X.....X.XXX..X...XX]
duration: 
30.004017323s
trace: 
https://jaeger.dev.storj.io/trace/7a3b11897bee958b
FAIL (2025-12-27 14:09:50 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 27 14:09:21.611393571 (526h40m21.482476616s) [upload=461959]: Starting upload
Dec 27 14:09:21.611546842 (526h40m21.482629877s) [upload=461959]: Done starting upload
Dec 27 14:09:21.611609071 (526h40m21.482692116s) [upload=461959]: Uploading segments...
Dec 27 14:09:21.612269158 (526h40m21.483352213s) [upload=461959]: Got next segment. Inline: false
Dec 27 14:09:21.612349201 (526h40m21.483432256s) [upload=461959]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 14:09:21.612356555 (526h40m21.4834396s) [upload=461959]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 14:09:21.856299604 (526h40m21.727382649s) [seg_pos={0 0}, upload=461959]: Begin upload segment...
Dec 27 14:09:21.856370038 (526h40m21.727453073s) [seg_pos={0 0}, upload=461959]: Done begin upload segment.
Dec 27 14:09:21.856420784 (526h40m21.727503819s) [noise=true , address=145.79.227.12:4765   , piece_id=5YKAMK34JOHYLZJT2D5WRZBY6N43346GSRFYFXHWZ6Z3AWWAQJ6A, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.856479476 (526h40m21.727562521s) [noise=true , address=145.79.227.29:4942   , piece_id=R6L233G7TZUNMLBODN4UEJH66YDWXL66B2WGAVQPHCIH5JXC74FA, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.857625017 (526h40m21.728708052s) [noise=true , address=45.91.116.13:4155    , piece_id=LBAG6T3GGWKHCOHA7T53VKSOX2O55KKQ4EGMRKDCK6PGFCTUIJVQ, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.858100663 (526h40m21.729183698s) [noise=true , address=145.79.227.10:4402   , piece_id=XUDYA3V3NLMFVKRX5ERUSILGUIOLX6NJNXWIL2PZDT6GFJMA3NDA, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.858170176 (526h40m21.729253231s) [noise=true , address=45.91.116.7:4009     , piece_id=US7DNVKAFLNLTFQB45RKVON242QU3RF3T5IELTT2WCNMUN6ZAYBQ, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.858402087 (526h40m21.729485122s) [noise=true , address=45.91.116.11:4120    , piece_id=UXEDL7GFL64HCRQ67THFOC2MKAVGYREYNQS7WEXJYGBOY3H7EYWQ, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.858794865 (526h40m21.7298779s) [noise=true , address=77.90.31.17:4773     , piece_id=5JIB2CV4VJSBBFINGJ4WCWLQN4H2T4WSEL6XDB3A755TJQKQOKYQ, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.859040964 (526h40m21.730124009s) [noise=true , address=145.79.227.19:4078   , piece_id=UAMIORNB3U6JKMHUIQRNRUNYW4T7S6YHERJG7ZVIHC5KLEXXCF6Q, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.861213120 (526h40m21.732296205s) [noise=true , address=145.79.227.45:4230   , piece_id=CPFKFQEL7BVP3V6I2LCDPVCGQACW3RDR7QT2V7XIJS77H6ZU7RIA, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.862069922 (526h40m21.733152957s) [noise=true , address=145.79.227.50:4375   , piece_id=XUAHWMEE2ZKSLFO3Z6CC7CZJZOSDSMNMDXYVGVK5ZLO2TDYZCH4Q, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.863841424 (526h40m21.734924469s) [noise=true , address=145.79.227.20:4438   , piece_id=42J67NFOF35HJMHKU2FLTASKR2TGKIY6NRBISTOIDIWBY64ACWSQ, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.867587077 (526h40m21.738670122s) [noise=true , address=145.79.227.42:4123   , piece_id=K6WUIK2EKFXSGSUXIOCSV4R3MOB2S4HPLEXOJRAKMCUC2GWYTEYA, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.869214605 (526h40m21.74029765s) [noise=true , address=45.91.116.6:4974     , piece_id=QXEEQHFYAH67J6OSDKWDVJCJV2S7ENNTQUWGV6TO5MMLTQ7JSOBQ, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.870424339 (526h40m21.741507384s) [noise=true , address=145.79.227.23:4546   , piece_id=YAGKC7A2NH6QNUCYJH3TIGQXTZZPDQ4FZTBR6LLAXSYMJTFAMI7A, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.871897163 (526h40m21.742980208s) [noise=true , address=145.79.227.17:4970   , piece_id=2SEU5CWQM23OWEPWAGUO34UBDJKT7TES2N3TZEIDIWDCCUBW5NEQ, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:21.872851330 (526h40m21.743934376s) [noise=true , address=145.79.227.36:4519   , piece_id=TVVXVEG5UT3CDL5AWEU4M2TQTOUTIV2RPCGBOGDCD2SRJIE4KBVA, seg_pos={0 0}, upload=461959]: Uploading piece...
Dec 27 14:09:23.532718349 (526h40m23.403801384s) [upload=461959]: Next returned nil segment
Dec 27 14:09:23.532735001 (526h40m23.403818047s) [upload=461959]: Waiting for error group managing segments...
Dec 27 14:09:23.862990534 (526h40m23.734073569s) [noise=true , address=145.79.227.12:4765   , piece_id=5YKAMK34JOHYLZJT2D5WRZBY6N43346GSRFYFXHWZ6Z3AWWAQJ6A, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:23.871865954 (526h40m23.742948989s) [noise=true , address=145.79.227.50:4375   , piece_id=XUAHWMEE2ZKSLFO3Z6CC7CZJZOSDSMNMDXYVGVK5ZLO2TDYZCH4Q, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:23.874552059 (526h40m23.745635104s) [noise=true , address=145.79.227.36:4519   , piece_id=TVVXVEG5UT3CDL5AWEU4M2TQTOUTIV2RPCGBOGDCD2SRJIE4KBVA, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:23.875081458 (526h40m23.746164493s) [noise=true , address=145.79.227.29:4942   , piece_id=R6L233G7TZUNMLBODN4UEJH66YDWXL66B2WGAVQPHCIH5JXC74FA, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:23.880889777 (526h40m23.751972812s) [noise=true , address=145.79.227.17:4970   , piece_id=2SEU5CWQM23OWEPWAGUO34UBDJKT7TES2N3TZEIDIWDCCUBW5NEQ, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:23.881779612 (526h40m23.752862657s) [noise=true , address=145.79.227.23:4546   , piece_id=YAGKC7A2NH6QNUCYJH3TIGQXTZZPDQ4FZTBR6LLAXSYMJTFAMI7A, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:23.888373497 (526h40m23.759456542s) [noise=true , address=145.79.227.10:4402   , piece_id=XUDYA3V3NLMFVKRX5ERUSILGUIOLX6NJNXWIL2PZDT6GFJMA3NDA, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:23.896892469 (526h40m23.767975504s) [noise=true , address=145.79.227.42:4123   , piece_id=K6WUIK2EKFXSGSUXIOCSV4R3MOB2S4HPLEXOJRAKMCUC2GWYTEYA, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:23.899060166 (526h40m23.770143202s) [noise=true , address=145.79.227.20:4438   , piece_id=42J67NFOF35HJMHKU2FLTASKR2TGKIY6NRBISTOIDIWBY64ACWSQ, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:23.899525874 (526h40m23.770608919s) [noise=true , address=145.79.227.45:4230   , piece_id=CPFKFQEL7BVP3V6I2LCDPVCGQACW3RDR7QT2V7XIJS77H6ZU7RIA, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:23.907289918 (526h40m23.778372953s) [noise=true , address=145.79.227.19:4078   , piece_id=UAMIORNB3U6JKMHUIQRNRUNYW4T7S6YHERJG7ZVIHC5KLEXXCF6Q, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:24.122094452 (526h40m23.993177497s) [noise=true , address=45.91.116.11:4120    , piece_id=UXEDL7GFL64HCRQ67THFOC2MKAVGYREYNQS7WEXJYGBOY3H7EYWQ, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:24.166811121 (526h40m24.037894166s) [noise=true , address=77.90.31.17:4773     , piece_id=5JIB2CV4VJSBBFINGJ4WCWLQN4H2T4WSEL6XDB3A755TJQKQOKYQ, seg_pos={0 0}, upload=461959]: Done uploading piece. err: <nil>
Dec 27 14:09:24.166864091 (526h40m24.037947126s) [seg_pos={0 0}, upload=461959]: Segment reached optimal threshold of 13 pieces.
Dec 27 14:09:24.167067258 (526h40m24.038150303s) [noise=true , address=45.91.116.7:4009     , piece_id=US7DNVKAFLNLTFQB45RKVON242QU3RF3T5IELTT2WCNMUN6ZAYBQ, seg_pos={0 0}, upload=461959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF
Dec 27 14:09:24.167077768 (526h40m24.038160823s) [noise=true , address=45.91.116.7:4009     , piece_id=US7DNVKAFLNLTFQB45RKVON242QU3RF3T5IELTT2WCNMUN6ZAYBQ, seg_pos={0 0}, upload=461959]: Long tail cancellation due to optimal threshold
Dec 27 14:09:24.167395192 (526h40m24.038478237s) [noise=true , address=45.91.116.13:4155    , piece_id=LBAG6T3GGWKHCOHA7T53VKSOX2O55KKQ4EGMRKDCK6PGFCTUIJVQ, seg_pos={0 0}, upload=461959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF
Dec 27 14:09:24.167402437 (526h40m24.038485472s) [noise=true , address=45.91.116.13:4155    , piece_id=LBAG6T3GGWKHCOHA7T53VKSOX2O55KKQ4EGMRKDCK6PGFCTUIJVQ, seg_pos={0 0}, upload=461959]: Long tail cancellation due to optimal threshold
Dec 27 14:09:24.167513648 (526h40m24.038596694s) [noise=true , address=45.91.116.6:4974     , piece_id=QXEEQHFYAH67J6OSDKWDVJCJV2S7ENNTQUWGV6TO5MMLTQ7JSOBQ, seg_pos={0 0}, upload=461959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF
Dec 27 14:09:24.167519349 (526h40m24.038602384s) [noise=true , address=45.91.116.6:4974     , piece_id=QXEEQHFYAH67J6OSDKWDVJCJV2S7ENNTQUWGV6TO5MMLTQ7JSOBQ, seg_pos={0 0}, upload=461959]: Long tail cancellation due to optimal threshold
Dec 27 14:09:24.167532774 (526h40m24.038615809s) [seg_pos={0 0}, upload=461959]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 14:09:24.167612757 (526h40m24.038695792s) [upload=461959]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 14:09:24.167618839 (526h40m24.038701884s) [upload=461959]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 14:09:24.455792179 (526h40m24.326875214s) [upload=461959]: Done uploading segments...
Dec 27 14:09:24.455820133 (526h40m24.326903168s) [upload=461959]: Upload finished. err: <nil>
Duration: 30.004031822s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
38 out of 60 is failed (threshold=6) [..XX....X.X.X.X.X..XXX.XX.XX.X.XX.X.XXXXXXXXXXX.X..XXXX.XXXX]
duration: 
5.001137127s
trace: 
https://jaeger.dev.storj.io/trace/1d470356fe614af6
FAIL (2025-12-27 14:09:38 UTC) 
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001154189s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
5 out of 60 is failed (threshold=3) [X...X..........X.............XX.............................]
duration: 
39.585648684s
trace: 
https://jaeger.dev.storj.io/trace/3ed1e657d506c4ee
FAIL (2025-12-27 14:09:25 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, Piece ID: HEMLEGWLVMK4LG5DM3PUWU5HSUEHKBECYOMHUXIQRGZGNXPXFLBQ) context deadline exceeded; piecestore: (Node ID: 12wXqePK1kK7iBo6wHcbqkRmkED8LGJ3rzQNJAWa14cHYhx1oyB, Piece ID: M5NU2MR5LQWOKNLVKKEJXMJNVEIZVDDNIZHWNN7XXHVTJRS2OQNA) context deadline exceeded; piecestore: (Node ID: 12ATtVD3XyFxzEqwQpSF4t1MWygqEa4Y1CkNj1s43eud8UtuSmx, Piece ID: ET7O467BME3OED32PFLJKYVIWZRDF6WMPXUT34OFWSPXLSMKKOWQ) context deadline exceeded; piecestore: (Node ID: 12UjFkFn2UEvr57LDorcijDCvkBkkKitgZEFx6Yqfk5gMStQvJG, Piece ID: SF3IKAIYXQR4YPWNVXT66SEHVM54R3K3VJHA6L5RBOU3DITOR5LQ) context deadline exceeded
======LOGS======
) [noise=true , address=77.90.31.16:4739     , piece_id=H6RNS5VOYTMWX2N6GLV2KOQDPFPCBWPDNLINB7SXDJWHMJL3GYQQ, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.319646322 (526h14m34.190729357s) [noise=true , address=77.90.31.21:4899     , piece_id=SF3IKAIYXQR4YPWNVXT66SEHVM54R3K3VJHA6L5RBOU3DITOR5LQ, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.319933248 (526h14m34.191016283s) [noise=true , address=45.91.116.12:4234    , piece_id=Y2FIVYLZBQAYBO2TDGIZ4RDQHQJ3WOQH565A7ESIKANQQTQDFQZQ, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.320400588 (526h14m34.191483633s) [noise=true , address=145.79.227.29:4952   , piece_id=VXZLF4OGRIR3MYQMMLKLTD235A55F4YTHB3D4YCVCQUL4FZTSFJQ, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.320407341 (526h14m34.191490386s) [noise=true , address=145.79.227.46:4621   , piece_id=XZ4MKAZAGQKVEJJGSZ5QGT4AT3MFXXNMXICVFCI6SP3E4EVEHZGQ, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.320675030 (526h14m34.191758075s) [noise=true , address=45.91.116.10:4113    , piece_id=OFHVWJWEHSVOOONDLXBZY72VB5G6FFXH2NXL45KY7TJ6CPVZ5UTQ, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.320802022 (526h14m34.191885067s) [noise=true , address=77.90.31.11:4569     , piece_id=YC2Q3HPJDVY3NEDZNC2Z4KVN6S6DDDGRARASF7KALFAOQ335A7HA, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.321169643 (526h14m34.192252678s) [noise=true , address=77.90.31.19:4869     , piece_id=HEMLEGWLVMK4LG5DM3PUWU5HSUEHKBECYOMHUXIQRGZGNXPXFLBQ, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.321443483 (526h14m34.192526518s) [noise=true , address=45.91.116.14:4202    , piece_id=M5NU2MR5LQWOKNLVKKEJXMJNVEIZVDDNIZHWNN7XXHVTJRS2OQNA, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.321731813 (526h14m34.192814848s) [noise=true , address=77.90.31.18:4819     , piece_id=YS4DXTHLGCPFOKKLBA2DK2BVOAPTQEGVOPBSRCLFV26SVP6YFHGA, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.322221776 (526h14m34.193304811s) [noise=true , address=77.90.31.15:4708     , piece_id=3NX6E54XMZXQAQQE27LTWUSEF5GAIF2XOPQFFAP7LOA2PL7KDL3Q, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.322547776 (526h14m34.193630811s) [noise=true , address=145.79.227.40:4079   , piece_id=237LYFTAG7G4HMIUKFZDDUTYJC7AAHHXTO5EIO46UEHY24BOVB3A, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.322908093 (526h14m34.193991138s) [noise=true , address=145.79.227.29:4975   , piece_id=DAMNBVP7DRSW5OAGYRRP4K5SNJ2WEUT7IAXO77KWRHLQEMBNERZQ, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.323267968 (526h14m34.194351003s) [noise=true , address=145.79.227.30:4787   , piece_id=OB2OTY3ZEHIUFYM6FY7HIS7HQ353Y2HL6XU4476HXOD26QVA4DPQ, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.323618165 (526h14m34.19470121s) [noise=true , address=64.49.13.169:4710    , piece_id=XIZKYK2E5FQJ37LYN535M7GXBBKXUQXFQWVMBWIBFDOAIIFGUAJA, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.324084893 (526h14m34.195167928s) [noise=true , address=193.149.164.143:4694 , piece_id=ET7O467BME3OED32PFLJKYVIWZRDF6WMPXUT34OFWSPXLSMKKOWQ, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:35.783540136 (526h14m35.654623181s) [upload=461595]: Next returned nil segment
Dec 27 13:43:35.783555856 (526h14m35.654638891s) [upload=461595]: Waiting for error group managing segments...
Dec 27 13:43:36.192287250 (526h14m36.063370296s) [noise=true , address=145.79.227.29:4952   , piece_id=VXZLF4OGRIR3MYQMMLKLTD235A55F4YTHB3D4YCVCQUL4FZTSFJQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:36.200200289 (526h14m36.071283334s) [noise=true , address=145.79.227.46:4621   , piece_id=XZ4MKAZAGQKVEJJGSZ5QGT4AT3MFXXNMXICVFCI6SP3E4EVEHZGQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:36.238087532 (526h14m36.109170577s) [noise=true , address=145.79.227.29:4975   , piece_id=DAMNBVP7DRSW5OAGYRRP4K5SNJ2WEUT7IAXO77KWRHLQEMBNERZQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:36.379631346 (526h14m36.250714391s) [noise=true , address=77.90.31.19:4869     , piece_id=HEMLEGWLVMK4LG5DM3PUWU5HSUEHKBECYOMHUXIQRGZGNXPXFLBQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:36.382945286 (526h14m36.254028321s) [noise=true , address=145.79.227.34:4909   , piece_id=Z6RF5PSOGOZMOXPQYXHMXGFXIELVXXVU4NXXRTAQCIELQXLMVXBQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:36.400290103 (526h14m36.271373138s) [noise=true , address=77.90.31.18:4819     , piece_id=YS4DXTHLGCPFOKKLBA2DK2BVOAPTQEGVOPBSRCLFV26SVP6YFHGA, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:36.436707208 (526h14m36.307790243s) [noise=true , address=45.91.116.14:4202    , piece_id=M5NU2MR5LQWOKNLVKKEJXMJNVEIZVDDNIZHWNN7XXHVTJRS2OQNA, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:36.487651475 (526h14m36.35873451s) [noise=true , address=45.91.116.12:4234    , piece_id=Y2FIVYLZBQAYBO2TDGIZ4RDQHQJ3WOQH565A7ESIKANQQTQDFQZQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:36.500850769 (526h14m36.371933814s) [noise=true , address=77.90.31.11:4569     , piece_id=YC2Q3HPJDVY3NEDZNC2Z4KVN6S6DDDGRARASF7KALFAOQ335A7HA, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:36.518828551 (526h14m36.389911586s) [noise=true , address=77.90.31.15:4708     , piece_id=3NX6E54XMZXQAQQE27LTWUSEF5GAIF2XOPQFFAP7LOA2PL7KDL3Q, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:37.051607243 (526h14m36.922690289s) [noise=true , address=145.79.227.44:4228   , piece_id=6SAYEAOTTCOUFPU75C7YRIULWJGXL4PK2YKCKFB47EUBWMRQQQOQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:37.096826909 (526h14m36.967909954s) [noise=true , address=145.79.227.30:4787   , piece_id=OB2OTY3ZEHIUFYM6FY7HIS7HQ353Y2HL6XU4476HXOD26QVA4DPQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:37.140738995 (526h14m37.01182203s) [noise=true , address=77.90.31.21:4899     , piece_id=SF3IKAIYXQR4YPWNVXT66SEHVM54R3K3VJHA6L5RBOU3DITOR5LQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:37.323613990 (526h14m37.194697035s) [noise=true , address=193.149.164.143:4694 , piece_id=ET7O467BME3OED32PFLJKYVIWZRDF6WMPXUT34OFWSPXLSMKKOWQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:37.323649618 (526h14m37.194732663s) [seg_pos={0 0}, upload=461595]: Segment reached optimal threshold of 14 pieces.
Dec 27 13:43:37.323852865 (526h14m37.1949359s) [noise=true , address=45.91.116.10:4113    , piece_id=OFHVWJWEHSVOOONDLXBZY72VB5G6FFXH2NXL45KY7TJ6CPVZ5UTQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12puRMny4nbgv2by4tAbXcBBSgG2d6BfwcDDRDcFzEdjSVUJxtd): protocol: expected piece hash; context canceled; EOF
Dec 27 13:43:37.323860349 (526h14m37.194943384s) [noise=true , address=45.91.116.10:4113    , piece_id=OFHVWJWEHSVOOONDLXBZY72VB5G6FFXH2NXL45KY7TJ6CPVZ5UTQ, seg_pos={0 0}, upload=461595]: Long tail cancellation due to optimal threshold
Dec 27 13:43:37.324005135 (526h14m37.1950882s) [noise=true , address=145.79.227.40:4079   , piece_id=237LYFTAG7G4HMIUKFZDDUTYJC7AAHHXTO5EIO46UEHY24BOVB3A, seg_pos={0 0}, upload=461595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RYKzn5zVSmxVbWwnBdX8oqtG1WE2MhPeEiWLf5dZeFAut6s2Y): protocol: expected piece hash; context canceled; EOF
Dec 27 13:43:37.324022659 (526h14m37.195105704s) [noise=true , address=145.79.227.40:4079   , piece_id=237LYFTAG7G4HMIUKFZDDUTYJC7AAHHXTO5EIO46UEHY24BOVB3A, seg_pos={0 0}, upload=461595]: Long tail cancellation due to optimal threshold
Dec 27 13:43:37.324143378 (526h14m37.195226413s) [noise=true , address=77.90.31.16:4739     , piece_id=H6RNS5VOYTMWX2N6GLV2KOQDPFPCBWPDNLINB7SXDJWHMJL3GYQQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129B3N7yjJvwCzoVLTqWsPbcLQ5TQUyAzg1RCkbSAVKPWYTG8eb): protocol: expected piece hash; context canceled; EOF
Dec 27 13:43:37.324154520 (526h14m37.195237555s) [noise=true , address=77.90.31.16:4739     , piece_id=H6RNS5VOYTMWX2N6GLV2KOQDPFPCBWPDNLINB7SXDJWHMJL3GYQQ, seg_pos={0 0}, upload=461595]: Long tail cancellation due to optimal threshold
Dec 27 13:43:37.324213442 (526h14m37.195296487s) [noise=true , address=64.49.13.169:4710    , piece_id=XIZKYK2E5FQJ37LYN535M7GXBBKXUQXFQWVMBWIBFDOAIIFGUAJA, seg_pos={0 0}, upload=461595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:57438->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:57438->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57438->64.49.13.169:4710: use of closed network connection
Dec 27 13:43:37.324249360 (526h14m37.195332405s) [noise=true , address=64.49.13.169:4710    , piece_id=XIZKYK2E5FQJ37LYN535M7GXBBKXUQXFQWVMBWIBFDOAIIFGUAJA, seg_pos={0 0}, upload=461595]: Long tail cancellation due to optimal threshold
Dec 27 13:43:37.324275360 (526h14m37.195358405s) [seg_pos={0 0}, upload=461595]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 13:43:37.324374018 (526h14m37.195457053s) [upload=461595]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 13:43:37.324381291 (526h14m37.195464337s) [upload=461595]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 13:43:37.679036406 (526h14m37.550119451s) [upload=461595]: Done uploading segments...
Dec 27 13:43:37.679056985 (526h14m37.55014003s) [upload=461595]: Upload finished. err: <nil>
Duration: 39.585664996s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
21 out of 60 is failed (threshold=3) [...X...X...XX.......X....X..XX.X.XXXX..X..X.....X.XXX..X...X]
duration: 
30.00407606s
trace: 
https://jaeger.dev.storj.io/trace/4c98e53f494f893e
FAIL (2025-12-27 14:09:20 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: YY6WYIWHZX2VI66DAPKFU746SE5U7AFHW4RC2V2TWR2GU76AD3IQ) context deadline exceeded; piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: 3CWJAPICY7KRWSYREIC6YIHNQ76OQKET24DR6BGUNRVXOHYUCNKA) context deadline exceeded
======LOGS======
7 14:08:51.368593599 (526h39m51.239676644s) [upload=461951]: Starting upload
Dec 27 14:08:51.368682167 (526h39m51.239765202s) [upload=461951]: Done starting upload
Dec 27 14:08:51.368732454 (526h39m51.239815489s) [upload=461951]: Uploading segments...
Dec 27 14:08:51.368939839 (526h39m51.240022884s) [upload=461951]: Got next segment. Inline: false
Dec 27 14:08:51.368987639 (526h39m51.240070694s) [upload=461951]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 14:08:51.368993460 (526h39m51.240076505s) [upload=461951]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 14:08:51.621069327 (526h39m51.492152362s) [seg_pos={0 0}, upload=461951]: Begin upload segment...
Dec 27 14:08:51.621121836 (526h39m51.492204881s) [seg_pos={0 0}, upload=461951]: Done begin upload segment.
Dec 27 14:08:51.621147926 (526h39m51.492230971s) [noise=true , address=145.79.227.14:4837   , piece_id=OLXUB2XCJP23266ITPVSAZVGZEGNSAUGAQ4I6NSOBNJBPBVSZKVA, seg_pos={0 0}, upload=461951]: Uploading piece...
Dec 27 14:08:51.621181320 (526h39m51.492264365s) [noise=true , address=145.79.227.16:4911   , piece_id=LH3PRD2MKM3Y7R7RSBFIWUETMVBWI6NBZMWITH4VMGM4JKBSOW7Q, seg_pos={0 0}, upload=461951]: Uploading piece...
Dec 27 14:08:51.622119296 (526h39m51.493202341s) [noise=true , address=145.79.227.37:4567   , piece_id=DISEWKG22IQGBJSFW3HWICKZCWUAFUKXW32Z7Y4ESWLRVD2VXRLA, seg_pos={0 0}, upload=461951]: Uploading piece...
Dec 27 14:08:51.622618817 (526h39m51.493701852s) [noise=true , address=145.79.227.41:4087   , piece_id=ZRVHK6XZMTUXQOGHRSPJBDGZ5QVJEM5SLIPBZUOYW6NIKUKGEIMQ, seg_pos={0 0}, upload=461951]: Uploading piece...
Dec 27 14:08:51.622965186 (526h39m51.494048221s) [noise=true , address=145.79.227.48:4301   , piece_id=OIWFVRRB47BEL4RNEMYQWWBEJP62J5Y6YMNSKIANC6UBBHQ2TIEQ, seg_pos={0 0}, upload=461951]: Uploading piece...
Dec 27 14:08:51.624883519 (526h39m51.495966564s) [noise=true , address=45.91.116.9:4262     , piece_id=JFQMGVNEUUWFBK3N2B526IQX457HTOFYZXS7PS2VH7MMK6W2N5BA, seg_pos={0 0}, upload=461951]: Uploading piece...
Dec 27 14:08:51.626805971 (526h39m51.497889006s) [noise=true , address=45.91.116.5:4300     , piece_id=EEMZW742DNC4TMM6FDUWJN635SCA34ZKF5H64X4JBJZWSSUUU6NQ, seg_pos={0 0}, upload=461951]: Uploading piece...
Dec 27 14:08:51.628288904 (526h39m51.499371939s) [noise=true , address=145.79.227.21:4475   , piece_id=RPUKVNPV7WOMZJWFS5IDBLTXBIOGUJXDCSE3T4RPAHOJNV5XXFFQ, seg_pos={0 0}, upload=461951]: Uploading piece...
Dec 27 14:08:51.629619116 (526h39m51.500702182s) [noise=true , address=145.79.227.2:4114    , piece_id=D4JOIDE6DHK3PYPIQQO736Z6N3RNQA4H35EPU7OQWLCVG6VQJMPQ, seg_pos={0 0}, upload=461951]: Uploading piece...
Dec 27 14:08:51.631061574 (526h39m51.502144619s) [noise=true , address=145.79.227.45:4231   , piece_id=5U5WZBP5AJ665AURCAWJBDQMGVDZAPUNNPLM465MZN3K5PNWULFQ, seg_pos={0 0}, upload=461951]: Uploading piece...
Dec 27 14:08:51.631172004 (526h39m51.502255049s) [noise=true , address=45.91.116.13:4156    , piece_id=RJT5XL3YAIEDGV4PH7V5IAE5FNZ42ODHNKOPMEOG235RGXMQIFGA, seg_pos={0 0}, upload=461951]: Uploading piece...
Dec 27 14:08:51.632779505 (526h39m51.50386255s) [noise=true , address=45.91.116.7:4010     , piece_id=YY6WYIWHZX2VI66DAPKFU746SE5U7AFHW4RC2V2TWR2GU76AD3IQ, seg_pos={0 0}, upload=461951]: Uploading piece...
Dec 27 14:08:51.633014202 (526h39m51.504097237s) [noise=true , address=145.79.227.50:4374   , piece_id=VUVC7MD5SHPVZTDQ4SFSCVQ2DFNMNEUEJZVQSBNM4UDI6NIRM4PQ, seg_pos={0 0}, upload=461951]: Uploading piece...
Dec 27 14:08:51.634955108 (526h39m51.506038143s) [noise=true , address=45.91.116.11:4118    , piece_id=MRA5LWUSUNCKFBL4T4DXWFXOIHOVRDXNPITO3QIKS4Z4LIFDDPEQ, seg_pos={0 0}, upload=461951]: Uploading piece...
Dec 27 14:08:51.637744609 (526h39m51.508827654s) [noise=true , address=45.91.116.6:4976     , piece_id=YQLRXANMMBPFDO7ZM22Z5BLYV4DJLCBF6NQM52EDGPLWF3MB7V7A, seg_pos={0 0}, upload=461951]: Uploading piece...
Dec 27 14:08:51.639585946 (526h39m51.510668991s) [noise=true , address=77.90.31.19:4842     , piece_id=3CWJAPICY7KRWSYREIC6YIHNQ76OQKET24DR6BGUNRVXOHYUCNKA, seg_pos={0 0}, upload=461951]: Uploading piece...
Dec 27 14:08:53.341676916 (526h39m53.212759961s) [upload=461951]: Next returned nil segment
Dec 27 14:08:53.341691293 (526h39m53.212774328s) [upload=461951]: Waiting for error group managing segments...
Dec 27 14:08:53.626524975 (526h39m53.49760801s) [noise=true , address=145.79.227.16:4911   , piece_id=LH3PRD2MKM3Y7R7RSBFIWUETMVBWI6NBZMWITH4VMGM4JKBSOW7Q, seg_pos={0 0}, upload=461951]: Done uploading piece. err: <nil>
Dec 27 14:08:53.630443217 (526h39m53.501526262s) [noise=true , address=145.79.227.37:4567   , piece_id=DISEWKG22IQGBJSFW3HWICKZCWUAFUKXW32Z7Y4ESWLRVD2VXRLA, seg_pos={0 0}, upload=461951]: Done uploading piece. err: <nil>
Dec 27 14:08:53.633359289 (526h39m53.504442324s) [noise=true , address=145.79.227.14:4837   , piece_id=OLXUB2XCJP23266ITPVSAZVGZEGNSAUGAQ4I6NSOBNJBPBVSZKVA, seg_pos={0 0}, upload=461951]: Done uploading piece. err: <nil>
Dec 27 14:08:53.635558566 (526h39m53.506641611s) [noise=true , address=145.79.227.2:4114    , piece_id=D4JOIDE6DHK3PYPIQQO736Z6N3RNQA4H35EPU7OQWLCVG6VQJMPQ, seg_pos={0 0}, upload=461951]: Done uploading piece. err: <nil>
Dec 27 14:08:53.648736630 (526h39m53.519819665s) [noise=true , address=145.79.227.50:4374   , piece_id=VUVC7MD5SHPVZTDQ4SFSCVQ2DFNMNEUEJZVQSBNM4UDI6NIRM4PQ, seg_pos={0 0}, upload=461951]: Done uploading piece. err: <nil>
Dec 27 14:08:53.648866707 (526h39m53.519949752s) [noise=true , address=145.79.227.48:4301   , piece_id=OIWFVRRB47BEL4RNEMYQWWBEJP62J5Y6YMNSKIANC6UBBHQ2TIEQ, seg_pos={0 0}, upload=461951]: Done uploading piece. err: <nil>
Dec 27 14:08:53.652038027 (526h39m53.523121062s) [noise=true , address=145.79.227.41:4087   , piece_id=ZRVHK6XZMTUXQOGHRSPJBDGZ5QVJEM5SLIPBZUOYW6NIKUKGEIMQ, seg_pos={0 0}, upload=461951]: Done uploading piece. err: <nil>
Dec 27 14:08:53.656621063 (526h39m53.527704108s) [noise=true , address=145.79.227.45:4231   , piece_id=5U5WZBP5AJ665AURCAWJBDQMGVDZAPUNNPLM465MZN3K5PNWULFQ, seg_pos={0 0}, upload=461951]: Done uploading piece. err: <nil>
Dec 27 14:08:53.657295878 (526h39m53.528378924s) [noise=true , address=145.79.227.21:4475   , piece_id=RPUKVNPV7WOMZJWFS5IDBLTXBIOGUJXDCSE3T4RPAHOJNV5XXFFQ, seg_pos={0 0}, upload=461951]: Done uploading piece. err: <nil>
Dec 27 14:08:53.834356432 (526h39m53.705439477s) [noise=true , address=77.90.31.19:4842     , piece_id=3CWJAPICY7KRWSYREIC6YIHNQ76OQKET24DR6BGUNRVXOHYUCNKA, seg_pos={0 0}, upload=461951]: Done uploading piece. err: <nil>
Dec 27 14:08:53.855516864 (526h39m53.7265999s) [noise=true , address=45.91.116.11:4118    , piece_id=MRA5LWUSUNCKFBL4T4DXWFXOIHOVRDXNPITO3QIKS4Z4LIFDDPEQ, seg_pos={0 0}, upload=461951]: Done uploading piece. err: <nil>
Dec 27 14:08:53.855804823 (526h39m53.726887858s) [noise=true , address=45.91.116.7:4010     , piece_id=YY6WYIWHZX2VI66DAPKFU746SE5U7AFHW4RC2V2TWR2GU76AD3IQ, seg_pos={0 0}, upload=461951]: Done uploading piece. err: <nil>
Dec 27 14:08:53.869702967 (526h39m53.740786013s) [noise=true , address=45.91.116.13:4156    , piece_id=RJT5XL3YAIEDGV4PH7V5IAE5FNZ42ODHNKOPMEOG235RGXMQIFGA, seg_pos={0 0}, upload=461951]: Done uploading piece. err: <nil>
Dec 27 14:08:53.869768262 (526h39m53.740851297s) [seg_pos={0 0}, upload=461951]: Segment reached optimal threshold of 13 pieces.
Dec 27 14:08:53.870087029 (526h39m53.741170074s) [noise=true , address=45.91.116.6:4976     , piece_id=YQLRXANMMBPFDO7ZM22Z5BLYV4DJLCBF6NQM52EDGPLWF3MB7V7A, seg_pos={0 0}, upload=461951]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF
Dec 27 14:08:53.870100564 (526h39m53.741183599s) [noise=true , address=45.91.116.6:4976     , piece_id=YQLRXANMMBPFDO7ZM22Z5BLYV4DJLCBF6NQM52EDGPLWF3MB7V7A, seg_pos={0 0}, upload=461951]: Long tail cancellation due to optimal threshold
Dec 27 14:08:53.870167752 (526h39m53.741250807s) [noise=true , address=45.91.116.9:4262     , piece_id=JFQMGVNEUUWFBK3N2B526IQX457HTOFYZXS7PS2VH7MMK6W2N5BA, seg_pos={0 0}, upload=461951]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: write tcp 188.166.225.170:53050->45.91.116.9:4262: use of closed network connection; write tcp 188.166.225.170:53050->45.91.116.9:4262: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53050->45.91.116.9:4262: use of closed network connection
Dec 27 14:08:53.870177901 (526h39m53.741260946s) [noise=true , address=45.91.116.9:4262     , piece_id=JFQMGVNEUUWFBK3N2B526IQX457HTOFYZXS7PS2VH7MMK6W2N5BA, seg_pos={0 0}, upload=461951]: Long tail cancellation due to optimal threshold
Dec 27 14:08:53.870432617 (526h39m53.741515672s) [noise=true , address=45.91.116.5:4300     , piece_id=EEMZW742DNC4TMM6FDUWJN635SCA34ZKF5H64X4JBJZWSSUUU6NQ, seg_pos={0 0}, upload=461951]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF
Dec 27 14:08:53.870442796 (526h39m53.741525841s) [noise=true , address=45.91.116.5:4300     , piece_id=EEMZW742DNC4TMM6FDUWJN635SCA34ZKF5H64X4JBJZWSSUUU6NQ, seg_pos={0 0}, upload=461951]: Long tail cancellation due to optimal threshold
Dec 27 14:08:53.870471671 (526h39m53.741554706s) [seg_pos={0 0}, upload=461951]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 14:08:53.870571361 (526h39m53.741654406s) [upload=461951]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 14:08:53.870579496 (526h39m53.741662541s) [upload=461951]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 14:08:54.169520220 (526h39m54.040603256s) [upload=461951]: Done uploading segments...
Dec 27 14:08:54.169540279 (526h39m54.040623314s) [upload=461951]: Upload finished. err: <nil>
Duration: 30.004100967s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
37 out of 60 is failed (threshold=6) [...XX....X.X.X.X.X..XXX.XX.XX.X.XX.X.XXXXXXXXXXX.X..XXXX.XXX]
duration: 
5.002033834s
trace: 
https://jaeger.dev.storj.io/trace/71e461ef93aad596
FAIL (2025-12-27 14:09:04 UTC) 
orbiter warning: context deadline exceeded
Duration: 5.00205831s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip: 
2 out of 60 is failed (threshold=3) [..............................................XX............]
duration: 
14.217984041s
trace: 
https://jaeger.dev.storj.io/trace/75ef8b5e6bc2819a
FAIL (2025-12-27 14:08:51 UTC) 
orbiter warning: context deadline exceeded
Duration: 14.218013356s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
5 out of 60 is failed (threshold=3) [.X...X..........X.............XX............................]
duration: 
32.91024647s
trace: 
https://jaeger.dev.storj.io/trace/3ed1e657d506c4ee
FAIL (2025-12-27 14:08:46 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, Piece ID: HEMLEGWLVMK4LG5DM3PUWU5HSUEHKBECYOMHUXIQRGZGNXPXFLBQ) context deadline exceeded; piecestore: (Node ID: 12wXqePK1kK7iBo6wHcbqkRmkED8LGJ3rzQNJAWa14cHYhx1oyB, Piece ID: M5NU2MR5LQWOKNLVKKEJXMJNVEIZVDDNIZHWNN7XXHVTJRS2OQNA) context deadline exceeded; piecestore: (Node ID: 12ATtVD3XyFxzEqwQpSF4t1MWygqEa4Y1CkNj1s43eud8UtuSmx, Piece ID: ET7O467BME3OED32PFLJKYVIWZRDF6WMPXUT34OFWSPXLSMKKOWQ) context deadline exceeded; piecestore: (Node ID: 12UjFkFn2UEvr57LDorcijDCvkBkkKitgZEFx6Yqfk5gMStQvJG, Piece ID: SF3IKAIYXQR4YPWNVXT66SEHVM54R3K3VJHA6L5RBOU3DITOR5LQ) context deadline exceeded
======LOGS======
) [noise=true , address=77.90.31.16:4739     , piece_id=H6RNS5VOYTMWX2N6GLV2KOQDPFPCBWPDNLINB7SXDJWHMJL3GYQQ, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.319646322 (526h14m34.190729357s) [noise=true , address=77.90.31.21:4899     , piece_id=SF3IKAIYXQR4YPWNVXT66SEHVM54R3K3VJHA6L5RBOU3DITOR5LQ, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.319933248 (526h14m34.191016283s) [noise=true , address=45.91.116.12:4234    , piece_id=Y2FIVYLZBQAYBO2TDGIZ4RDQHQJ3WOQH565A7ESIKANQQTQDFQZQ, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.320400588 (526h14m34.191483633s) [noise=true , address=145.79.227.29:4952   , piece_id=VXZLF4OGRIR3MYQMMLKLTD235A55F4YTHB3D4YCVCQUL4FZTSFJQ, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.320407341 (526h14m34.191490386s) [noise=true , address=145.79.227.46:4621   , piece_id=XZ4MKAZAGQKVEJJGSZ5QGT4AT3MFXXNMXICVFCI6SP3E4EVEHZGQ, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.320675030 (526h14m34.191758075s) [noise=true , address=45.91.116.10:4113    , piece_id=OFHVWJWEHSVOOONDLXBZY72VB5G6FFXH2NXL45KY7TJ6CPVZ5UTQ, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.320802022 (526h14m34.191885067s) [noise=true , address=77.90.31.11:4569     , piece_id=YC2Q3HPJDVY3NEDZNC2Z4KVN6S6DDDGRARASF7KALFAOQ335A7HA, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.321169643 (526h14m34.192252678s) [noise=true , address=77.90.31.19:4869     , piece_id=HEMLEGWLVMK4LG5DM3PUWU5HSUEHKBECYOMHUXIQRGZGNXPXFLBQ, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.321443483 (526h14m34.192526518s) [noise=true , address=45.91.116.14:4202    , piece_id=M5NU2MR5LQWOKNLVKKEJXMJNVEIZVDDNIZHWNN7XXHVTJRS2OQNA, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.321731813 (526h14m34.192814848s) [noise=true , address=77.90.31.18:4819     , piece_id=YS4DXTHLGCPFOKKLBA2DK2BVOAPTQEGVOPBSRCLFV26SVP6YFHGA, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.322221776 (526h14m34.193304811s) [noise=true , address=77.90.31.15:4708     , piece_id=3NX6E54XMZXQAQQE27LTWUSEF5GAIF2XOPQFFAP7LOA2PL7KDL3Q, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.322547776 (526h14m34.193630811s) [noise=true , address=145.79.227.40:4079   , piece_id=237LYFTAG7G4HMIUKFZDDUTYJC7AAHHXTO5EIO46UEHY24BOVB3A, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.322908093 (526h14m34.193991138s) [noise=true , address=145.79.227.29:4975   , piece_id=DAMNBVP7DRSW5OAGYRRP4K5SNJ2WEUT7IAXO77KWRHLQEMBNERZQ, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.323267968 (526h14m34.194351003s) [noise=true , address=145.79.227.30:4787   , piece_id=OB2OTY3ZEHIUFYM6FY7HIS7HQ353Y2HL6XU4476HXOD26QVA4DPQ, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.323618165 (526h14m34.19470121s) [noise=true , address=64.49.13.169:4710    , piece_id=XIZKYK2E5FQJ37LYN535M7GXBBKXUQXFQWVMBWIBFDOAIIFGUAJA, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.324084893 (526h14m34.195167928s) [noise=true , address=193.149.164.143:4694 , piece_id=ET7O467BME3OED32PFLJKYVIWZRDF6WMPXUT34OFWSPXLSMKKOWQ, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:35.783540136 (526h14m35.654623181s) [upload=461595]: Next returned nil segment
Dec 27 13:43:35.783555856 (526h14m35.654638891s) [upload=461595]: Waiting for error group managing segments...
Dec 27 13:43:36.192287250 (526h14m36.063370296s) [noise=true , address=145.79.227.29:4952   , piece_id=VXZLF4OGRIR3MYQMMLKLTD235A55F4YTHB3D4YCVCQUL4FZTSFJQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:36.200200289 (526h14m36.071283334s) [noise=true , address=145.79.227.46:4621   , piece_id=XZ4MKAZAGQKVEJJGSZ5QGT4AT3MFXXNMXICVFCI6SP3E4EVEHZGQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:36.238087532 (526h14m36.109170577s) [noise=true , address=145.79.227.29:4975   , piece_id=DAMNBVP7DRSW5OAGYRRP4K5SNJ2WEUT7IAXO77KWRHLQEMBNERZQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:36.379631346 (526h14m36.250714391s) [noise=true , address=77.90.31.19:4869     , piece_id=HEMLEGWLVMK4LG5DM3PUWU5HSUEHKBECYOMHUXIQRGZGNXPXFLBQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:36.382945286 (526h14m36.254028321s) [noise=true , address=145.79.227.34:4909   , piece_id=Z6RF5PSOGOZMOXPQYXHMXGFXIELVXXVU4NXXRTAQCIELQXLMVXBQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:36.400290103 (526h14m36.271373138s) [noise=true , address=77.90.31.18:4819     , piece_id=YS4DXTHLGCPFOKKLBA2DK2BVOAPTQEGVOPBSRCLFV26SVP6YFHGA, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:36.436707208 (526h14m36.307790243s) [noise=true , address=45.91.116.14:4202    , piece_id=M5NU2MR5LQWOKNLVKKEJXMJNVEIZVDDNIZHWNN7XXHVTJRS2OQNA, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:36.487651475 (526h14m36.35873451s) [noise=true , address=45.91.116.12:4234    , piece_id=Y2FIVYLZBQAYBO2TDGIZ4RDQHQJ3WOQH565A7ESIKANQQTQDFQZQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:36.500850769 (526h14m36.371933814s) [noise=true , address=77.90.31.11:4569     , piece_id=YC2Q3HPJDVY3NEDZNC2Z4KVN6S6DDDGRARASF7KALFAOQ335A7HA, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:36.518828551 (526h14m36.389911586s) [noise=true , address=77.90.31.15:4708     , piece_id=3NX6E54XMZXQAQQE27LTWUSEF5GAIF2XOPQFFAP7LOA2PL7KDL3Q, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:37.051607243 (526h14m36.922690289s) [noise=true , address=145.79.227.44:4228   , piece_id=6SAYEAOTTCOUFPU75C7YRIULWJGXL4PK2YKCKFB47EUBWMRQQQOQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:37.096826909 (526h14m36.967909954s) [noise=true , address=145.79.227.30:4787   , piece_id=OB2OTY3ZEHIUFYM6FY7HIS7HQ353Y2HL6XU4476HXOD26QVA4DPQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:37.140738995 (526h14m37.01182203s) [noise=true , address=77.90.31.21:4899     , piece_id=SF3IKAIYXQR4YPWNVXT66SEHVM54R3K3VJHA6L5RBOU3DITOR5LQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:37.323613990 (526h14m37.194697035s) [noise=true , address=193.149.164.143:4694 , piece_id=ET7O467BME3OED32PFLJKYVIWZRDF6WMPXUT34OFWSPXLSMKKOWQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:37.323649618 (526h14m37.194732663s) [seg_pos={0 0}, upload=461595]: Segment reached optimal threshold of 14 pieces.
Dec 27 13:43:37.323852865 (526h14m37.1949359s) [noise=true , address=45.91.116.10:4113    , piece_id=OFHVWJWEHSVOOONDLXBZY72VB5G6FFXH2NXL45KY7TJ6CPVZ5UTQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12puRMny4nbgv2by4tAbXcBBSgG2d6BfwcDDRDcFzEdjSVUJxtd): protocol: expected piece hash; context canceled; EOF
Dec 27 13:43:37.323860349 (526h14m37.194943384s) [noise=true , address=45.91.116.10:4113    , piece_id=OFHVWJWEHSVOOONDLXBZY72VB5G6FFXH2NXL45KY7TJ6CPVZ5UTQ, seg_pos={0 0}, upload=461595]: Long tail cancellation due to optimal threshold
Dec 27 13:43:37.324005135 (526h14m37.1950882s) [noise=true , address=145.79.227.40:4079   , piece_id=237LYFTAG7G4HMIUKFZDDUTYJC7AAHHXTO5EIO46UEHY24BOVB3A, seg_pos={0 0}, upload=461595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RYKzn5zVSmxVbWwnBdX8oqtG1WE2MhPeEiWLf5dZeFAut6s2Y): protocol: expected piece hash; context canceled; EOF
Dec 27 13:43:37.324022659 (526h14m37.195105704s) [noise=true , address=145.79.227.40:4079   , piece_id=237LYFTAG7G4HMIUKFZDDUTYJC7AAHHXTO5EIO46UEHY24BOVB3A, seg_pos={0 0}, upload=461595]: Long tail cancellation due to optimal threshold
Dec 27 13:43:37.324143378 (526h14m37.195226413s) [noise=true , address=77.90.31.16:4739     , piece_id=H6RNS5VOYTMWX2N6GLV2KOQDPFPCBWPDNLINB7SXDJWHMJL3GYQQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129B3N7yjJvwCzoVLTqWsPbcLQ5TQUyAzg1RCkbSAVKPWYTG8eb): protocol: expected piece hash; context canceled; EOF
Dec 27 13:43:37.324154520 (526h14m37.195237555s) [noise=true , address=77.90.31.16:4739     , piece_id=H6RNS5VOYTMWX2N6GLV2KOQDPFPCBWPDNLINB7SXDJWHMJL3GYQQ, seg_pos={0 0}, upload=461595]: Long tail cancellation due to optimal threshold
Dec 27 13:43:37.324213442 (526h14m37.195296487s) [noise=true , address=64.49.13.169:4710    , piece_id=XIZKYK2E5FQJ37LYN535M7GXBBKXUQXFQWVMBWIBFDOAIIFGUAJA, seg_pos={0 0}, upload=461595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:57438->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:57438->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57438->64.49.13.169:4710: use of closed network connection
Dec 27 13:43:37.324249360 (526h14m37.195332405s) [noise=true , address=64.49.13.169:4710    , piece_id=XIZKYK2E5FQJ37LYN535M7GXBBKXUQXFQWVMBWIBFDOAIIFGUAJA, seg_pos={0 0}, upload=461595]: Long tail cancellation due to optimal threshold
Dec 27 13:43:37.324275360 (526h14m37.195358405s) [seg_pos={0 0}, upload=461595]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 13:43:37.324374018 (526h14m37.195457053s) [upload=461595]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 13:43:37.324381291 (526h14m37.195464337s) [upload=461595]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 13:43:37.679036406 (526h14m37.550119451s) [upload=461595]: Done uploading segments...
Dec 27 13:43:37.679056985 (526h14m37.55014003s) [upload=461595]: Upload finished. err: <nil>
Duration: 32.910273531s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
36 out of 60 is failed (threshold=6) [....XX....X.X.X.X.X..XXX.XX.XX.X.XX.X.XXXXXXXXXXX.X..XXXX.XX]
duration: 
5.001343558s
trace: 
https://jaeger.dev.storj.io/trace/262a559018f28a12
FAIL (2025-12-27 14:08:39 UTC) 
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001363897s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
21 out of 60 is failed (threshold=3) [X...X...X...XX.......X....X..XX.X.XXXX..X..X.....X.XXX..X...]
duration: 
5.682955584s
trace: 
https://jaeger.dev.storj.io/trace/6e0c241f5b6f833e
FAIL (2025-12-27 14:08:33 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: 7H7F4E3NFIUZLSLZDTICFEVR3ZDS2E2XV5FEFIU5NM4TBSFQ4H2A) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: DDUX6F7V3GETJCN6FU6DI4HFDJOVRVDYJI7RXR6A3H2BIJQCMENQ) context deadline exceeded
======LOGS======
Dec 27 14:05:42.174622851 (526h36m42.045705886s) [upload=461903]: Starting upload
Dec 27 14:05:42.174711871 (526h36m42.045794926s) [upload=461903]: Done starting upload
Dec 27 14:05:42.174810228 (526h36m42.045893273s) [upload=461903]: Uploading segments...
Dec 27 14:05:42.175091664 (526h36m42.046174719s) [upload=461903]: Got next segment. Inline: false
Dec 27 14:05:42.175140587 (526h36m42.046223622s) [upload=461903]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 14:05:42.175147209 (526h36m42.046230254s) [upload=461903]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 14:05:42.425678453 (526h36m42.296761498s) [seg_pos={0 0}, upload=461903]: Begin upload segment...
Dec 27 14:05:42.425738487 (526h36m42.296821523s) [seg_pos={0 0}, upload=461903]: Done begin upload segment.
Dec 27 14:05:42.425769326 (526h36m42.296852371s) [noise=true , address=77.90.31.18:4808     , piece_id=55D2V4ECKRXWDLCBX2VYHGYK2GZVDWIMK4GYMHJFPZZUESPAZCWQ, seg_pos={0 0}, upload=461903]: Uploading piece...
Dec 27 14:05:42.425861192 (526h36m42.296944227s) [noise=true , address=145.79.227.2:4113    , piece_id=LSOBOIUBMGT4HWCDJMDJUKGJQT4AO6IIHC3KQZXHJ4TQCCOHO44Q, seg_pos={0 0}, upload=461903]: Uploading piece...
Dec 27 14:05:42.426381272 (526h36m42.297464317s) [noise=true , address=145.79.227.12:4767   , piece_id=7JK6ZY4LQPSY2HVJYHPZG3LFES3UXZA3U5RBL6GMCU3F6AJMAUFQ, seg_pos={0 0}, upload=461903]: Uploading piece...
Dec 27 14:05:42.426727581 (526h36m42.297810616s) [noise=true , address=45.91.116.11:4119    , piece_id=GL4JYGWFBKEWVSIDMIYDENMN2XAGMQVBDGDS2XT443UDODBRYXVQ, seg_pos={0 0}, upload=461903]: Uploading piece...
Dec 27 14:05:42.427005120 (526h36m42.298088155s) [noise=true , address=145.79.227.50:4373   , piece_id=2PBWUVG46JZ3F22EUXZYKWRXLBY2YX6EIHVSUQI2EWANV4DFEGIA, seg_pos={0 0}, upload=461903]: Uploading piece...
Dec 27 14:05:42.427282517 (526h36m42.298365562s) [noise=true , address=77.90.31.16:4736     , piece_id=7H7F4E3NFIUZLSLZDTICFEVR3ZDS2E2XV5FEFIU5NM4TBSFQ4H2A, seg_pos={0 0}, upload=461903]: Uploading piece...
Dec 27 14:05:42.428321215 (526h36m42.29940425s) [noise=true , address=145.79.227.45:4231   , piece_id=4GYK4OYBLZZ6HQ5DITMFWKHFRG427NJRX22FOFVKQOQYJ2TORNFQ, seg_pos={0 0}, upload=461903]: Uploading piece...
Dec 27 14:05:42.428681642 (526h36m42.299764677s) [noise=true , address=77.90.31.11:4556     , piece_id=UOJV3RKCWOXMVOLX44NVNJUFYJVWRPFXMAAY2AYTKIFKMO7KXQ6Q, seg_pos={0 0}, upload=461903]: Uploading piece...
Dec 27 14:05:42.430016554 (526h36m42.301099589s) [noise=true , address=145.79.227.16:4911   , piece_id=L3SNXPPJMTOL6PO6QGUPRJGNNLHIDCB6VNJPGUCDD4ZNZ27J2ZWA, seg_pos={0 0}, upload=461903]: Uploading piece...
Dec 27 14:05:42.430172351 (526h36m42.301255396s) [noise=true , address=45.91.116.5:4297     , piece_id=DDUX6F7V3GETJCN6FU6DI4HFDJOVRVDYJI7RXR6A3H2BIJQCMENQ, seg_pos={0 0}, upload=461903]: Uploading piece...
Dec 27 14:05:42.431452768 (526h36m42.302535803s) [noise=true , address=145.79.227.41:4085   , piece_id=AAO5GFX4SYKTYUHB6WBPCIZ3XX7KXR4EB4T7VEZPNQSZODECJZNA, seg_pos={0 0}, upload=461903]: Uploading piece...
Dec 27 14:05:42.431468078 (526h36m42.302551123s) [noise=true , address=145.79.227.14:4837   , piece_id=O5XJ7MCAZVEZ4TUC2HLRSOP2X3FNSNTAXALP4C2YUGTESWEG23QQ, seg_pos={0 0}, upload=461903]: Uploading piece...
Dec 27 14:05:42.432756331 (526h36m42.303839366s) [noise=true , address=77.90.31.13:4627     , piece_id=TCO2SHFH4EC5KNB4LUO5T4J43C5DANDBZEYH2PZK4O422OOMPP2A, seg_pos={0 0}, upload=461903]: Uploading piece...
Dec 27 14:05:42.434033873 (526h36m42.305116938s) [noise=true , address=145.79.227.46:4601   , piece_id=JRHHU56YAVKME5WDKHS22NIXFVW5VFWLIFRI2TRIGYAHVAMXI7FQ, seg_pos={0 0}, upload=461903]: Uploading piece...
Dec 27 14:05:42.435025542 (526h36m42.306108577s) [noise=true , address=145.79.227.36:4518   , piece_id=5KENHRDZ53AGQGGOA43LWCQZUYFJPLUQMVLANGZUU7JFKZQWGZ3Q, seg_pos={0 0}, upload=461903]: Uploading piece...
Dec 27 14:05:42.437075566 (526h36m42.308158601s) [noise=true , address=145.79.227.10:4402   , piece_id=QUITOKPDBY74NRBDJUGMX76WRAIETACPQHJI2O4SFVYUCUIV7EOQ, seg_pos={0 0}, upload=461903]: Uploading piece...
Dec 27 14:05:44.135402769 (526h36m44.006485804s) [upload=461903]: Next returned nil segment
Dec 27 14:05:44.135423068 (526h36m44.006506103s) [upload=461903]: Waiting for error group managing segments...
Dec 27 14:05:44.417681756 (526h36m44.288764791s) [noise=true , address=145.79.227.46:4601   , piece_id=JRHHU56YAVKME5WDKHS22NIXFVW5VFWLIFRI2TRIGYAHVAMXI7FQ, seg_pos={0 0}, upload=461903]: Done uploading piece. err: <nil>
Dec 27 14:05:44.434422193 (526h36m44.305505238s) [noise=true , address=145.79.227.12:4767   , piece_id=7JK6ZY4LQPSY2HVJYHPZG3LFES3UXZA3U5RBL6GMCU3F6AJMAUFQ, seg_pos={0 0}, upload=461903]: Done uploading piece. err: <nil>
Dec 27 14:05:44.436344174 (526h36m44.307427209s) [noise=true , address=145.79.227.14:4837   , piece_id=O5XJ7MCAZVEZ4TUC2HLRSOP2X3FNSNTAXALP4C2YUGTESWEG23QQ, seg_pos={0 0}, upload=461903]: Done uploading piece. err: <nil>
Dec 27 14:05:44.442475128 (526h36m44.313558173s) [noise=true , address=145.79.227.45:4231   , piece_id=4GYK4OYBLZZ6HQ5DITMFWKHFRG427NJRX22FOFVKQOQYJ2TORNFQ, seg_pos={0 0}, upload=461903]: Done uploading piece. err: <nil>
Dec 27 14:05:44.445306850 (526h36m44.316389885s) [noise=true , address=145.79.227.36:4518   , piece_id=5KENHRDZ53AGQGGOA43LWCQZUYFJPLUQMVLANGZUU7JFKZQWGZ3Q, seg_pos={0 0}, upload=461903]: Done uploading piece. err: <nil>
Dec 27 14:05:44.453544626 (526h36m44.324627661s) [noise=true , address=145.79.227.41:4085   , piece_id=AAO5GFX4SYKTYUHB6WBPCIZ3XX7KXR4EB4T7VEZPNQSZODECJZNA, seg_pos={0 0}, upload=461903]: Done uploading piece. err: <nil>
Dec 27 14:05:44.456117115 (526h36m44.32720016s) [noise=true , address=145.79.227.16:4911   , piece_id=L3SNXPPJMTOL6PO6QGUPRJGNNLHIDCB6VNJPGUCDD4ZNZ27J2ZWA, seg_pos={0 0}, upload=461903]: Done uploading piece. err: <nil>
Dec 27 14:05:44.456558094 (526h36m44.327641129s) [noise=true , address=145.79.227.50:4373   , piece_id=2PBWUVG46JZ3F22EUXZYKWRXLBY2YX6EIHVSUQI2EWANV4DFEGIA, seg_pos={0 0}, upload=461903]: Done uploading piece. err: <nil>
Dec 27 14:05:44.456865349 (526h36m44.327948384s) [noise=true , address=145.79.227.2:4113    , piece_id=LSOBOIUBMGT4HWCDJMDJUKGJQT4AO6IIHC3KQZXHJ4TQCCOHO44Q, seg_pos={0 0}, upload=461903]: Done uploading piece. err: <nil>
Dec 27 14:05:44.464205105 (526h36m44.33528814s) [noise=true , address=145.79.227.10:4402   , piece_id=QUITOKPDBY74NRBDJUGMX76WRAIETACPQHJI2O4SFVYUCUIV7EOQ, seg_pos={0 0}, upload=461903]: Done uploading piece. err: <nil>
Dec 27 14:05:44.603684487 (526h36m44.474767522s) [noise=true , address=77.90.31.16:4736     , piece_id=7H7F4E3NFIUZLSLZDTICFEVR3ZDS2E2XV5FEFIU5NM4TBSFQ4H2A, seg_pos={0 0}, upload=461903]: Done uploading piece. err: <nil>
Dec 27 14:05:44.634489305 (526h36m44.50557235s) [noise=true , address=77.90.31.18:4808     , piece_id=55D2V4ECKRXWDLCBX2VYHGYK2GZVDWIMK4GYMHJFPZZUESPAZCWQ, seg_pos={0 0}, upload=461903]: Done uploading piece. err: <nil>
Dec 27 14:05:44.649453970 (526h36m44.520537006s) [noise=true , address=45.91.116.5:4297     , piece_id=DDUX6F7V3GETJCN6FU6DI4HFDJOVRVDYJI7RXR6A3H2BIJQCMENQ, seg_pos={0 0}, upload=461903]: Done uploading piece. err: <nil>
Dec 27 14:05:44.649476163 (526h36m44.520559228s) [seg_pos={0 0}, upload=461903]: Segment reached optimal threshold of 13 pieces.
Dec 27 14:05:44.649708966 (526h36m44.520792001s) [noise=true , address=77.90.31.11:4556     , piece_id=UOJV3RKCWOXMVOLX44NVNJUFYJVWRPFXMAAY2AYTKIFKMO7KXQ6Q, seg_pos={0 0}, upload=461903]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF
Dec 27 14:05:44.649715268 (526h36m44.520798303s) [noise=true , address=77.90.31.11:4556     , piece_id=UOJV3RKCWOXMVOLX44NVNJUFYJVWRPFXMAAY2AYTKIFKMO7KXQ6Q, seg_pos={0 0}, upload=461903]: Long tail cancellation due to optimal threshold
Dec 27 14:05:44.650062809 (526h36m44.521145864s) [noise=true , address=77.90.31.13:4627     , piece_id=TCO2SHFH4EC5KNB4LUO5T4J43C5DANDBZEYH2PZK4O422OOMPP2A, seg_pos={0 0}, upload=461903]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: expected piece hash; context canceled; EOF
Dec 27 14:05:44.650071978 (526h36m44.521155023s) [noise=true , address=77.90.31.13:4627     , piece_id=TCO2SHFH4EC5KNB4LUO5T4J43C5DANDBZEYH2PZK4O422OOMPP2A, seg_pos={0 0}, upload=461903]: Long tail cancellation due to optimal threshold
Dec 27 14:05:44.650217314 (526h36m44.521300349s) [noise=true , address=45.91.116.11:4119    , piece_id=GL4JYGWFBKEWVSIDMIYDENMN2XAGMQVBDGDS2XT443UDODBRYXVQ, seg_pos={0 0}, upload=461903]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF
Dec 27 14:05:44.650223436 (526h36m44.521306471s) [noise=true , address=45.91.116.11:4119    , piece_id=GL4JYGWFBKEWVSIDMIYDENMN2XAGMQVBDGDS2XT443UDODBRYXVQ, seg_pos={0 0}, upload=461903]: Long tail cancellation due to optimal threshold
Dec 27 14:05:44.650238524 (526h36m44.521321559s) [seg_pos={0 0}, upload=461903]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 14:05:44.650305211 (526h36m44.521388246s) [upload=461903]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 14:05:44.650311043 (526h36m44.521394078s) [upload=461903]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 14:05:44.941554529 (526h36m44.812637564s) [upload=461903]: Done uploading segments...
Dec 27 14:05:44.941574837 (526h36m44.812657872s) [upload=461903]: Upload finished. err: <nil>
Duration: 5.682981844s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip: 
2 out of 60 is failed (threshold=3) [...............................................XX...........]
duration: 
22.399498004s
trace: 
https://jaeger.dev.storj.io/trace/75ef8b5e6bc2819a
FAIL (2025-12-27 14:07:57 UTC) 
orbiter warning: context deadline exceeded
Duration: 22.399523282s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
5 out of 60 is failed (threshold=3) [..X...X..........X.............XX...........................]
duration: 
42.39238249s
trace: 
https://jaeger.dev.storj.io/trace/3ed1e657d506c4ee
FAIL (2025-12-27 14:07:52 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, Piece ID: HEMLEGWLVMK4LG5DM3PUWU5HSUEHKBECYOMHUXIQRGZGNXPXFLBQ) context deadline exceeded; piecestore: (Node ID: 12wXqePK1kK7iBo6wHcbqkRmkED8LGJ3rzQNJAWa14cHYhx1oyB, Piece ID: M5NU2MR5LQWOKNLVKKEJXMJNVEIZVDDNIZHWNN7XXHVTJRS2OQNA) context deadline exceeded; piecestore: (Node ID: 12ATtVD3XyFxzEqwQpSF4t1MWygqEa4Y1CkNj1s43eud8UtuSmx, Piece ID: ET7O467BME3OED32PFLJKYVIWZRDF6WMPXUT34OFWSPXLSMKKOWQ) context deadline exceeded; piecestore: (Node ID: 12UjFkFn2UEvr57LDorcijDCvkBkkKitgZEFx6Yqfk5gMStQvJG, Piece ID: SF3IKAIYXQR4YPWNVXT66SEHVM54R3K3VJHA6L5RBOU3DITOR5LQ) context deadline exceeded
======LOGS======
) [noise=true , address=77.90.31.16:4739     , piece_id=H6RNS5VOYTMWX2N6GLV2KOQDPFPCBWPDNLINB7SXDJWHMJL3GYQQ, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.319646322 (526h14m34.190729357s) [noise=true , address=77.90.31.21:4899     , piece_id=SF3IKAIYXQR4YPWNVXT66SEHVM54R3K3VJHA6L5RBOU3DITOR5LQ, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.319933248 (526h14m34.191016283s) [noise=true , address=45.91.116.12:4234    , piece_id=Y2FIVYLZBQAYBO2TDGIZ4RDQHQJ3WOQH565A7ESIKANQQTQDFQZQ, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.320400588 (526h14m34.191483633s) [noise=true , address=145.79.227.29:4952   , piece_id=VXZLF4OGRIR3MYQMMLKLTD235A55F4YTHB3D4YCVCQUL4FZTSFJQ, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.320407341 (526h14m34.191490386s) [noise=true , address=145.79.227.46:4621   , piece_id=XZ4MKAZAGQKVEJJGSZ5QGT4AT3MFXXNMXICVFCI6SP3E4EVEHZGQ, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.320675030 (526h14m34.191758075s) [noise=true , address=45.91.116.10:4113    , piece_id=OFHVWJWEHSVOOONDLXBZY72VB5G6FFXH2NXL45KY7TJ6CPVZ5UTQ, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.320802022 (526h14m34.191885067s) [noise=true , address=77.90.31.11:4569     , piece_id=YC2Q3HPJDVY3NEDZNC2Z4KVN6S6DDDGRARASF7KALFAOQ335A7HA, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.321169643 (526h14m34.192252678s) [noise=true , address=77.90.31.19:4869     , piece_id=HEMLEGWLVMK4LG5DM3PUWU5HSUEHKBECYOMHUXIQRGZGNXPXFLBQ, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.321443483 (526h14m34.192526518s) [noise=true , address=45.91.116.14:4202    , piece_id=M5NU2MR5LQWOKNLVKKEJXMJNVEIZVDDNIZHWNN7XXHVTJRS2OQNA, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.321731813 (526h14m34.192814848s) [noise=true , address=77.90.31.18:4819     , piece_id=YS4DXTHLGCPFOKKLBA2DK2BVOAPTQEGVOPBSRCLFV26SVP6YFHGA, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.322221776 (526h14m34.193304811s) [noise=true , address=77.90.31.15:4708     , piece_id=3NX6E54XMZXQAQQE27LTWUSEF5GAIF2XOPQFFAP7LOA2PL7KDL3Q, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.322547776 (526h14m34.193630811s) [noise=true , address=145.79.227.40:4079   , piece_id=237LYFTAG7G4HMIUKFZDDUTYJC7AAHHXTO5EIO46UEHY24BOVB3A, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.322908093 (526h14m34.193991138s) [noise=true , address=145.79.227.29:4975   , piece_id=DAMNBVP7DRSW5OAGYRRP4K5SNJ2WEUT7IAXO77KWRHLQEMBNERZQ, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.323267968 (526h14m34.194351003s) [noise=true , address=145.79.227.30:4787   , piece_id=OB2OTY3ZEHIUFYM6FY7HIS7HQ353Y2HL6XU4476HXOD26QVA4DPQ, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.323618165 (526h14m34.19470121s) [noise=true , address=64.49.13.169:4710    , piece_id=XIZKYK2E5FQJ37LYN535M7GXBBKXUQXFQWVMBWIBFDOAIIFGUAJA, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.324084893 (526h14m34.195167928s) [noise=true , address=193.149.164.143:4694 , piece_id=ET7O467BME3OED32PFLJKYVIWZRDF6WMPXUT34OFWSPXLSMKKOWQ, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:35.783540136 (526h14m35.654623181s) [upload=461595]: Next returned nil segment
Dec 27 13:43:35.783555856 (526h14m35.654638891s) [upload=461595]: Waiting for error group managing segments...
Dec 27 13:43:36.192287250 (526h14m36.063370296s) [noise=true , address=145.79.227.29:4952   , piece_id=VXZLF4OGRIR3MYQMMLKLTD235A55F4YTHB3D4YCVCQUL4FZTSFJQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:36.200200289 (526h14m36.071283334s) [noise=true , address=145.79.227.46:4621   , piece_id=XZ4MKAZAGQKVEJJGSZ5QGT4AT3MFXXNMXICVFCI6SP3E4EVEHZGQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:36.238087532 (526h14m36.109170577s) [noise=true , address=145.79.227.29:4975   , piece_id=DAMNBVP7DRSW5OAGYRRP4K5SNJ2WEUT7IAXO77KWRHLQEMBNERZQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:36.379631346 (526h14m36.250714391s) [noise=true , address=77.90.31.19:4869     , piece_id=HEMLEGWLVMK4LG5DM3PUWU5HSUEHKBECYOMHUXIQRGZGNXPXFLBQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:36.382945286 (526h14m36.254028321s) [noise=true , address=145.79.227.34:4909   , piece_id=Z6RF5PSOGOZMOXPQYXHMXGFXIELVXXVU4NXXRTAQCIELQXLMVXBQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:36.400290103 (526h14m36.271373138s) [noise=true , address=77.90.31.18:4819     , piece_id=YS4DXTHLGCPFOKKLBA2DK2BVOAPTQEGVOPBSRCLFV26SVP6YFHGA, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:36.436707208 (526h14m36.307790243s) [noise=true , address=45.91.116.14:4202    , piece_id=M5NU2MR5LQWOKNLVKKEJXMJNVEIZVDDNIZHWNN7XXHVTJRS2OQNA, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:36.487651475 (526h14m36.35873451s) [noise=true , address=45.91.116.12:4234    , piece_id=Y2FIVYLZBQAYBO2TDGIZ4RDQHQJ3WOQH565A7ESIKANQQTQDFQZQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:36.500850769 (526h14m36.371933814s) [noise=true , address=77.90.31.11:4569     , piece_id=YC2Q3HPJDVY3NEDZNC2Z4KVN6S6DDDGRARASF7KALFAOQ335A7HA, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:36.518828551 (526h14m36.389911586s) [noise=true , address=77.90.31.15:4708     , piece_id=3NX6E54XMZXQAQQE27LTWUSEF5GAIF2XOPQFFAP7LOA2PL7KDL3Q, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:37.051607243 (526h14m36.922690289s) [noise=true , address=145.79.227.44:4228   , piece_id=6SAYEAOTTCOUFPU75C7YRIULWJGXL4PK2YKCKFB47EUBWMRQQQOQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:37.096826909 (526h14m36.967909954s) [noise=true , address=145.79.227.30:4787   , piece_id=OB2OTY3ZEHIUFYM6FY7HIS7HQ353Y2HL6XU4476HXOD26QVA4DPQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:37.140738995 (526h14m37.01182203s) [noise=true , address=77.90.31.21:4899     , piece_id=SF3IKAIYXQR4YPWNVXT66SEHVM54R3K3VJHA6L5RBOU3DITOR5LQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:37.323613990 (526h14m37.194697035s) [noise=true , address=193.149.164.143:4694 , piece_id=ET7O467BME3OED32PFLJKYVIWZRDF6WMPXUT34OFWSPXLSMKKOWQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:37.323649618 (526h14m37.194732663s) [seg_pos={0 0}, upload=461595]: Segment reached optimal threshold of 14 pieces.
Dec 27 13:43:37.323852865 (526h14m37.1949359s) [noise=true , address=45.91.116.10:4113    , piece_id=OFHVWJWEHSVOOONDLXBZY72VB5G6FFXH2NXL45KY7TJ6CPVZ5UTQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12puRMny4nbgv2by4tAbXcBBSgG2d6BfwcDDRDcFzEdjSVUJxtd): protocol: expected piece hash; context canceled; EOF
Dec 27 13:43:37.323860349 (526h14m37.194943384s) [noise=true , address=45.91.116.10:4113    , piece_id=OFHVWJWEHSVOOONDLXBZY72VB5G6FFXH2NXL45KY7TJ6CPVZ5UTQ, seg_pos={0 0}, upload=461595]: Long tail cancellation due to optimal threshold
Dec 27 13:43:37.324005135 (526h14m37.1950882s) [noise=true , address=145.79.227.40:4079   , piece_id=237LYFTAG7G4HMIUKFZDDUTYJC7AAHHXTO5EIO46UEHY24BOVB3A, seg_pos={0 0}, upload=461595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RYKzn5zVSmxVbWwnBdX8oqtG1WE2MhPeEiWLf5dZeFAut6s2Y): protocol: expected piece hash; context canceled; EOF
Dec 27 13:43:37.324022659 (526h14m37.195105704s) [noise=true , address=145.79.227.40:4079   , piece_id=237LYFTAG7G4HMIUKFZDDUTYJC7AAHHXTO5EIO46UEHY24BOVB3A, seg_pos={0 0}, upload=461595]: Long tail cancellation due to optimal threshold
Dec 27 13:43:37.324143378 (526h14m37.195226413s) [noise=true , address=77.90.31.16:4739     , piece_id=H6RNS5VOYTMWX2N6GLV2KOQDPFPCBWPDNLINB7SXDJWHMJL3GYQQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129B3N7yjJvwCzoVLTqWsPbcLQ5TQUyAzg1RCkbSAVKPWYTG8eb): protocol: expected piece hash; context canceled; EOF
Dec 27 13:43:37.324154520 (526h14m37.195237555s) [noise=true , address=77.90.31.16:4739     , piece_id=H6RNS5VOYTMWX2N6GLV2KOQDPFPCBWPDNLINB7SXDJWHMJL3GYQQ, seg_pos={0 0}, upload=461595]: Long tail cancellation due to optimal threshold
Dec 27 13:43:37.324213442 (526h14m37.195296487s) [noise=true , address=64.49.13.169:4710    , piece_id=XIZKYK2E5FQJ37LYN535M7GXBBKXUQXFQWVMBWIBFDOAIIFGUAJA, seg_pos={0 0}, upload=461595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:57438->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:57438->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57438->64.49.13.169:4710: use of closed network connection
Dec 27 13:43:37.324249360 (526h14m37.195332405s) [noise=true , address=64.49.13.169:4710    , piece_id=XIZKYK2E5FQJ37LYN535M7GXBBKXUQXFQWVMBWIBFDOAIIFGUAJA, seg_pos={0 0}, upload=461595]: Long tail cancellation due to optimal threshold
Dec 27 13:43:37.324275360 (526h14m37.195358405s) [seg_pos={0 0}, upload=461595]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 13:43:37.324374018 (526h14m37.195457053s) [upload=461595]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 13:43:37.324381291 (526h14m37.195464337s) [upload=461595]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 13:43:37.679036406 (526h14m37.550119451s) [upload=461595]: Done uploading segments...
Dec 27 13:43:37.679056985 (526h14m37.55014003s) [upload=461595]: Upload finished. err: <nil>
Duration: 42.39239765s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
36 out of 60 is failed (threshold=6) [X....XX....X.X.X.X.X..XXX.XX.XX.X.XX.X.XXXXXXXXXXX.X..XXXX.X]
duration: 
5.001174475s
trace: 
https://jaeger.dev.storj.io/trace/324fe21facb0c98a
FAIL (2025-12-27 14:07:44 UTC) 
orbiter warning: context deadline exceeded
Duration: 5.001200495s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
21 out of 60 is failed (threshold=3) [.X...X...X...XX.......X....X..XX.X.XXXX..X..X.....X.XXX..X..]
duration: 
5.66279289s
trace: 
https://jaeger.dev.storj.io/trace/6e0c241f5b6f833e
FAIL (2025-12-27 14:07:42 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: 7H7F4E3NFIUZLSLZDTICFEVR3ZDS2E2XV5FEFIU5NM4TBSFQ4H2A) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: DDUX6F7V3GETJCN6FU6DI4HFDJOVRVDYJI7RXR6A3H2BIJQCMENQ) context deadline exceeded
======LOGS======
Dec 27 14:05:42.174622851 (526h36m42.045705886s) [upload=461903]: Starting upload
Dec 27 14:05:42.174711871 (526h36m42.045794926s) [upload=461903]: Done starting upload
Dec 27 14:05:42.174810228 (526h36m42.045893273s) [upload=461903]: Uploading segments...
Dec 27 14:05:42.175091664 (526h36m42.046174719s) [upload=461903]: Got next segment. Inline: false
Dec 27 14:05:42.175140587 (526h36m42.046223622s) [upload=461903]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 14:05:42.175147209 (526h36m42.046230254s) [upload=461903]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 14:05:42.425678453 (526h36m42.296761498s) [seg_pos={0 0}, upload=461903]: Begin upload segment...
Dec 27 14:05:42.425738487 (526h36m42.296821523s) [seg_pos={0 0}, upload=461903]: Done begin upload segment.
Dec 27 14:05:42.425769326 (526h36m42.296852371s) [noise=true , address=77.90.31.18:4808     , piece_id=55D2V4ECKRXWDLCBX2VYHGYK2GZVDWIMK4GYMHJFPZZUESPAZCWQ, seg_pos={0 0}, upload=461903]: Uploading piece...
Dec 27 14:05:42.425861192 (526h36m42.296944227s) [noise=true , address=145.79.227.2:4113    , piece_id=LSOBOIUBMGT4HWCDJMDJUKGJQT4AO6IIHC3KQZXHJ4TQCCOHO44Q, seg_pos={0 0}, upload=461903]: Uploading piece...
Dec 27 14:05:42.426381272 (526h36m42.297464317s) [noise=true , address=145.79.227.12:4767   , piece_id=7JK6ZY4LQPSY2HVJYHPZG3LFES3UXZA3U5RBL6GMCU3F6AJMAUFQ, seg_pos={0 0}, upload=461903]: Uploading piece...
Dec 27 14:05:42.426727581 (526h36m42.297810616s) [noise=true , address=45.91.116.11:4119    , piece_id=GL4JYGWFBKEWVSIDMIYDENMN2XAGMQVBDGDS2XT443UDODBRYXVQ, seg_pos={0 0}, upload=461903]: Uploading piece...
Dec 27 14:05:42.427005120 (526h36m42.298088155s) [noise=true , address=145.79.227.50:4373   , piece_id=2PBWUVG46JZ3F22EUXZYKWRXLBY2YX6EIHVSUQI2EWANV4DFEGIA, seg_pos={0 0}, upload=461903]: Uploading piece...
Dec 27 14:05:42.427282517 (526h36m42.298365562s) [noise=true , address=77.90.31.16:4736     , piece_id=7H7F4E3NFIUZLSLZDTICFEVR3ZDS2E2XV5FEFIU5NM4TBSFQ4H2A, seg_pos={0 0}, upload=461903]: Uploading piece...
Dec 27 14:05:42.428321215 (526h36m42.29940425s) [noise=true , address=145.79.227.45:4231   , piece_id=4GYK4OYBLZZ6HQ5DITMFWKHFRG427NJRX22FOFVKQOQYJ2TORNFQ, seg_pos={0 0}, upload=461903]: Uploading piece...
Dec 27 14:05:42.428681642 (526h36m42.299764677s) [noise=true , address=77.90.31.11:4556     , piece_id=UOJV3RKCWOXMVOLX44NVNJUFYJVWRPFXMAAY2AYTKIFKMO7KXQ6Q, seg_pos={0 0}, upload=461903]: Uploading piece...
Dec 27 14:05:42.430016554 (526h36m42.301099589s) [noise=true , address=145.79.227.16:4911   , piece_id=L3SNXPPJMTOL6PO6QGUPRJGNNLHIDCB6VNJPGUCDD4ZNZ27J2ZWA, seg_pos={0 0}, upload=461903]: Uploading piece...
Dec 27 14:05:42.430172351 (526h36m42.301255396s) [noise=true , address=45.91.116.5:4297     , piece_id=DDUX6F7V3GETJCN6FU6DI4HFDJOVRVDYJI7RXR6A3H2BIJQCMENQ, seg_pos={0 0}, upload=461903]: Uploading piece...
Dec 27 14:05:42.431452768 (526h36m42.302535803s) [noise=true , address=145.79.227.41:4085   , piece_id=AAO5GFX4SYKTYUHB6WBPCIZ3XX7KXR4EB4T7VEZPNQSZODECJZNA, seg_pos={0 0}, upload=461903]: Uploading piece...
Dec 27 14:05:42.431468078 (526h36m42.302551123s) [noise=true , address=145.79.227.14:4837   , piece_id=O5XJ7MCAZVEZ4TUC2HLRSOP2X3FNSNTAXALP4C2YUGTESWEG23QQ, seg_pos={0 0}, upload=461903]: Uploading piece...
Dec 27 14:05:42.432756331 (526h36m42.303839366s) [noise=true , address=77.90.31.13:4627     , piece_id=TCO2SHFH4EC5KNB4LUO5T4J43C5DANDBZEYH2PZK4O422OOMPP2A, seg_pos={0 0}, upload=461903]: Uploading piece...
Dec 27 14:05:42.434033873 (526h36m42.305116938s) [noise=true , address=145.79.227.46:4601   , piece_id=JRHHU56YAVKME5WDKHS22NIXFVW5VFWLIFRI2TRIGYAHVAMXI7FQ, seg_pos={0 0}, upload=461903]: Uploading piece...
Dec 27 14:05:42.435025542 (526h36m42.306108577s) [noise=true , address=145.79.227.36:4518   , piece_id=5KENHRDZ53AGQGGOA43LWCQZUYFJPLUQMVLANGZUU7JFKZQWGZ3Q, seg_pos={0 0}, upload=461903]: Uploading piece...
Dec 27 14:05:42.437075566 (526h36m42.308158601s) [noise=true , address=145.79.227.10:4402   , piece_id=QUITOKPDBY74NRBDJUGMX76WRAIETACPQHJI2O4SFVYUCUIV7EOQ, seg_pos={0 0}, upload=461903]: Uploading piece...
Dec 27 14:05:44.135402769 (526h36m44.006485804s) [upload=461903]: Next returned nil segment
Dec 27 14:05:44.135423068 (526h36m44.006506103s) [upload=461903]: Waiting for error group managing segments...
Dec 27 14:05:44.417681756 (526h36m44.288764791s) [noise=true , address=145.79.227.46:4601   , piece_id=JRHHU56YAVKME5WDKHS22NIXFVW5VFWLIFRI2TRIGYAHVAMXI7FQ, seg_pos={0 0}, upload=461903]: Done uploading piece. err: <nil>
Dec 27 14:05:44.434422193 (526h36m44.305505238s) [noise=true , address=145.79.227.12:4767   , piece_id=7JK6ZY4LQPSY2HVJYHPZG3LFES3UXZA3U5RBL6GMCU3F6AJMAUFQ, seg_pos={0 0}, upload=461903]: Done uploading piece. err: <nil>
Dec 27 14:05:44.436344174 (526h36m44.307427209s) [noise=true , address=145.79.227.14:4837   , piece_id=O5XJ7MCAZVEZ4TUC2HLRSOP2X3FNSNTAXALP4C2YUGTESWEG23QQ, seg_pos={0 0}, upload=461903]: Done uploading piece. err: <nil>
Dec 27 14:05:44.442475128 (526h36m44.313558173s) [noise=true , address=145.79.227.45:4231   , piece_id=4GYK4OYBLZZ6HQ5DITMFWKHFRG427NJRX22FOFVKQOQYJ2TORNFQ, seg_pos={0 0}, upload=461903]: Done uploading piece. err: <nil>
Dec 27 14:05:44.445306850 (526h36m44.316389885s) [noise=true , address=145.79.227.36:4518   , piece_id=5KENHRDZ53AGQGGOA43LWCQZUYFJPLUQMVLANGZUU7JFKZQWGZ3Q, seg_pos={0 0}, upload=461903]: Done uploading piece. err: <nil>
Dec 27 14:05:44.453544626 (526h36m44.324627661s) [noise=true , address=145.79.227.41:4085   , piece_id=AAO5GFX4SYKTYUHB6WBPCIZ3XX7KXR4EB4T7VEZPNQSZODECJZNA, seg_pos={0 0}, upload=461903]: Done uploading piece. err: <nil>
Dec 27 14:05:44.456117115 (526h36m44.32720016s) [noise=true , address=145.79.227.16:4911   , piece_id=L3SNXPPJMTOL6PO6QGUPRJGNNLHIDCB6VNJPGUCDD4ZNZ27J2ZWA, seg_pos={0 0}, upload=461903]: Done uploading piece. err: <nil>
Dec 27 14:05:44.456558094 (526h36m44.327641129s) [noise=true , address=145.79.227.50:4373   , piece_id=2PBWUVG46JZ3F22EUXZYKWRXLBY2YX6EIHVSUQI2EWANV4DFEGIA, seg_pos={0 0}, upload=461903]: Done uploading piece. err: <nil>
Dec 27 14:05:44.456865349 (526h36m44.327948384s) [noise=true , address=145.79.227.2:4113    , piece_id=LSOBOIUBMGT4HWCDJMDJUKGJQT4AO6IIHC3KQZXHJ4TQCCOHO44Q, seg_pos={0 0}, upload=461903]: Done uploading piece. err: <nil>
Dec 27 14:05:44.464205105 (526h36m44.33528814s) [noise=true , address=145.79.227.10:4402   , piece_id=QUITOKPDBY74NRBDJUGMX76WRAIETACPQHJI2O4SFVYUCUIV7EOQ, seg_pos={0 0}, upload=461903]: Done uploading piece. err: <nil>
Dec 27 14:05:44.603684487 (526h36m44.474767522s) [noise=true , address=77.90.31.16:4736     , piece_id=7H7F4E3NFIUZLSLZDTICFEVR3ZDS2E2XV5FEFIU5NM4TBSFQ4H2A, seg_pos={0 0}, upload=461903]: Done uploading piece. err: <nil>
Dec 27 14:05:44.634489305 (526h36m44.50557235s) [noise=true , address=77.90.31.18:4808     , piece_id=55D2V4ECKRXWDLCBX2VYHGYK2GZVDWIMK4GYMHJFPZZUESPAZCWQ, seg_pos={0 0}, upload=461903]: Done uploading piece. err: <nil>
Dec 27 14:05:44.649453970 (526h36m44.520537006s) [noise=true , address=45.91.116.5:4297     , piece_id=DDUX6F7V3GETJCN6FU6DI4HFDJOVRVDYJI7RXR6A3H2BIJQCMENQ, seg_pos={0 0}, upload=461903]: Done uploading piece. err: <nil>
Dec 27 14:05:44.649476163 (526h36m44.520559228s) [seg_pos={0 0}, upload=461903]: Segment reached optimal threshold of 13 pieces.
Dec 27 14:05:44.649708966 (526h36m44.520792001s) [noise=true , address=77.90.31.11:4556     , piece_id=UOJV3RKCWOXMVOLX44NVNJUFYJVWRPFXMAAY2AYTKIFKMO7KXQ6Q, seg_pos={0 0}, upload=461903]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF
Dec 27 14:05:44.649715268 (526h36m44.520798303s) [noise=true , address=77.90.31.11:4556     , piece_id=UOJV3RKCWOXMVOLX44NVNJUFYJVWRPFXMAAY2AYTKIFKMO7KXQ6Q, seg_pos={0 0}, upload=461903]: Long tail cancellation due to optimal threshold
Dec 27 14:05:44.650062809 (526h36m44.521145864s) [noise=true , address=77.90.31.13:4627     , piece_id=TCO2SHFH4EC5KNB4LUO5T4J43C5DANDBZEYH2PZK4O422OOMPP2A, seg_pos={0 0}, upload=461903]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: expected piece hash; context canceled; EOF
Dec 27 14:05:44.650071978 (526h36m44.521155023s) [noise=true , address=77.90.31.13:4627     , piece_id=TCO2SHFH4EC5KNB4LUO5T4J43C5DANDBZEYH2PZK4O422OOMPP2A, seg_pos={0 0}, upload=461903]: Long tail cancellation due to optimal threshold
Dec 27 14:05:44.650217314 (526h36m44.521300349s) [noise=true , address=45.91.116.11:4119    , piece_id=GL4JYGWFBKEWVSIDMIYDENMN2XAGMQVBDGDS2XT443UDODBRYXVQ, seg_pos={0 0}, upload=461903]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF
Dec 27 14:05:44.650223436 (526h36m44.521306471s) [noise=true , address=45.91.116.11:4119    , piece_id=GL4JYGWFBKEWVSIDMIYDENMN2XAGMQVBDGDS2XT443UDODBRYXVQ, seg_pos={0 0}, upload=461903]: Long tail cancellation due to optimal threshold
Dec 27 14:05:44.650238524 (526h36m44.521321559s) [seg_pos={0 0}, upload=461903]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 14:05:44.650305211 (526h36m44.521388246s) [upload=461903]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 14:05:44.650311043 (526h36m44.521394078s) [upload=461903]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 14:05:44.941554529 (526h36m44.812637564s) [upload=461903]: Done uploading segments...
Dec 27 14:05:44.941574837 (526h36m44.812657872s) [upload=461903]: Upload finished. err: <nil>
Duration: 5.662818267s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
5 out of 60 is failed (threshold=3) [...X...X..........X.............XX..........................]
duration: 
44.358277227s
trace: 
https://jaeger.dev.storj.io/trace/3ed1e657d506c4ee
FAIL (2025-12-27 14:07:10 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, Piece ID: HEMLEGWLVMK4LG5DM3PUWU5HSUEHKBECYOMHUXIQRGZGNXPXFLBQ) context deadline exceeded; piecestore: (Node ID: 12wXqePK1kK7iBo6wHcbqkRmkED8LGJ3rzQNJAWa14cHYhx1oyB, Piece ID: M5NU2MR5LQWOKNLVKKEJXMJNVEIZVDDNIZHWNN7XXHVTJRS2OQNA) context deadline exceeded; piecestore: (Node ID: 12ATtVD3XyFxzEqwQpSF4t1MWygqEa4Y1CkNj1s43eud8UtuSmx, Piece ID: ET7O467BME3OED32PFLJKYVIWZRDF6WMPXUT34OFWSPXLSMKKOWQ) context deadline exceeded; piecestore: (Node ID: 12UjFkFn2UEvr57LDorcijDCvkBkkKitgZEFx6Yqfk5gMStQvJG, Piece ID: SF3IKAIYXQR4YPWNVXT66SEHVM54R3K3VJHA6L5RBOU3DITOR5LQ) context deadline exceeded
======LOGS======
) [noise=true , address=77.90.31.16:4739     , piece_id=H6RNS5VOYTMWX2N6GLV2KOQDPFPCBWPDNLINB7SXDJWHMJL3GYQQ, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.319646322 (526h14m34.190729357s) [noise=true , address=77.90.31.21:4899     , piece_id=SF3IKAIYXQR4YPWNVXT66SEHVM54R3K3VJHA6L5RBOU3DITOR5LQ, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.319933248 (526h14m34.191016283s) [noise=true , address=45.91.116.12:4234    , piece_id=Y2FIVYLZBQAYBO2TDGIZ4RDQHQJ3WOQH565A7ESIKANQQTQDFQZQ, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.320400588 (526h14m34.191483633s) [noise=true , address=145.79.227.29:4952   , piece_id=VXZLF4OGRIR3MYQMMLKLTD235A55F4YTHB3D4YCVCQUL4FZTSFJQ, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.320407341 (526h14m34.191490386s) [noise=true , address=145.79.227.46:4621   , piece_id=XZ4MKAZAGQKVEJJGSZ5QGT4AT3MFXXNMXICVFCI6SP3E4EVEHZGQ, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.320675030 (526h14m34.191758075s) [noise=true , address=45.91.116.10:4113    , piece_id=OFHVWJWEHSVOOONDLXBZY72VB5G6FFXH2NXL45KY7TJ6CPVZ5UTQ, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.320802022 (526h14m34.191885067s) [noise=true , address=77.90.31.11:4569     , piece_id=YC2Q3HPJDVY3NEDZNC2Z4KVN6S6DDDGRARASF7KALFAOQ335A7HA, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.321169643 (526h14m34.192252678s) [noise=true , address=77.90.31.19:4869     , piece_id=HEMLEGWLVMK4LG5DM3PUWU5HSUEHKBECYOMHUXIQRGZGNXPXFLBQ, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.321443483 (526h14m34.192526518s) [noise=true , address=45.91.116.14:4202    , piece_id=M5NU2MR5LQWOKNLVKKEJXMJNVEIZVDDNIZHWNN7XXHVTJRS2OQNA, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.321731813 (526h14m34.192814848s) [noise=true , address=77.90.31.18:4819     , piece_id=YS4DXTHLGCPFOKKLBA2DK2BVOAPTQEGVOPBSRCLFV26SVP6YFHGA, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.322221776 (526h14m34.193304811s) [noise=true , address=77.90.31.15:4708     , piece_id=3NX6E54XMZXQAQQE27LTWUSEF5GAIF2XOPQFFAP7LOA2PL7KDL3Q, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.322547776 (526h14m34.193630811s) [noise=true , address=145.79.227.40:4079   , piece_id=237LYFTAG7G4HMIUKFZDDUTYJC7AAHHXTO5EIO46UEHY24BOVB3A, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.322908093 (526h14m34.193991138s) [noise=true , address=145.79.227.29:4975   , piece_id=DAMNBVP7DRSW5OAGYRRP4K5SNJ2WEUT7IAXO77KWRHLQEMBNERZQ, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.323267968 (526h14m34.194351003s) [noise=true , address=145.79.227.30:4787   , piece_id=OB2OTY3ZEHIUFYM6FY7HIS7HQ353Y2HL6XU4476HXOD26QVA4DPQ, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.323618165 (526h14m34.19470121s) [noise=true , address=64.49.13.169:4710    , piece_id=XIZKYK2E5FQJ37LYN535M7GXBBKXUQXFQWVMBWIBFDOAIIFGUAJA, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.324084893 (526h14m34.195167928s) [noise=true , address=193.149.164.143:4694 , piece_id=ET7O467BME3OED32PFLJKYVIWZRDF6WMPXUT34OFWSPXLSMKKOWQ, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:35.783540136 (526h14m35.654623181s) [upload=461595]: Next returned nil segment
Dec 27 13:43:35.783555856 (526h14m35.654638891s) [upload=461595]: Waiting for error group managing segments...
Dec 27 13:43:36.192287250 (526h14m36.063370296s) [noise=true , address=145.79.227.29:4952   , piece_id=VXZLF4OGRIR3MYQMMLKLTD235A55F4YTHB3D4YCVCQUL4FZTSFJQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:36.200200289 (526h14m36.071283334s) [noise=true , address=145.79.227.46:4621   , piece_id=XZ4MKAZAGQKVEJJGSZ5QGT4AT3MFXXNMXICVFCI6SP3E4EVEHZGQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:36.238087532 (526h14m36.109170577s) [noise=true , address=145.79.227.29:4975   , piece_id=DAMNBVP7DRSW5OAGYRRP4K5SNJ2WEUT7IAXO77KWRHLQEMBNERZQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:36.379631346 (526h14m36.250714391s) [noise=true , address=77.90.31.19:4869     , piece_id=HEMLEGWLVMK4LG5DM3PUWU5HSUEHKBECYOMHUXIQRGZGNXPXFLBQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:36.382945286 (526h14m36.254028321s) [noise=true , address=145.79.227.34:4909   , piece_id=Z6RF5PSOGOZMOXPQYXHMXGFXIELVXXVU4NXXRTAQCIELQXLMVXBQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:36.400290103 (526h14m36.271373138s) [noise=true , address=77.90.31.18:4819     , piece_id=YS4DXTHLGCPFOKKLBA2DK2BVOAPTQEGVOPBSRCLFV26SVP6YFHGA, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:36.436707208 (526h14m36.307790243s) [noise=true , address=45.91.116.14:4202    , piece_id=M5NU2MR5LQWOKNLVKKEJXMJNVEIZVDDNIZHWNN7XXHVTJRS2OQNA, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:36.487651475 (526h14m36.35873451s) [noise=true , address=45.91.116.12:4234    , piece_id=Y2FIVYLZBQAYBO2TDGIZ4RDQHQJ3WOQH565A7ESIKANQQTQDFQZQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:36.500850769 (526h14m36.371933814s) [noise=true , address=77.90.31.11:4569     , piece_id=YC2Q3HPJDVY3NEDZNC2Z4KVN6S6DDDGRARASF7KALFAOQ335A7HA, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:36.518828551 (526h14m36.389911586s) [noise=true , address=77.90.31.15:4708     , piece_id=3NX6E54XMZXQAQQE27LTWUSEF5GAIF2XOPQFFAP7LOA2PL7KDL3Q, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:37.051607243 (526h14m36.922690289s) [noise=true , address=145.79.227.44:4228   , piece_id=6SAYEAOTTCOUFPU75C7YRIULWJGXL4PK2YKCKFB47EUBWMRQQQOQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:37.096826909 (526h14m36.967909954s) [noise=true , address=145.79.227.30:4787   , piece_id=OB2OTY3ZEHIUFYM6FY7HIS7HQ353Y2HL6XU4476HXOD26QVA4DPQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:37.140738995 (526h14m37.01182203s) [noise=true , address=77.90.31.21:4899     , piece_id=SF3IKAIYXQR4YPWNVXT66SEHVM54R3K3VJHA6L5RBOU3DITOR5LQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:37.323613990 (526h14m37.194697035s) [noise=true , address=193.149.164.143:4694 , piece_id=ET7O467BME3OED32PFLJKYVIWZRDF6WMPXUT34OFWSPXLSMKKOWQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:37.323649618 (526h14m37.194732663s) [seg_pos={0 0}, upload=461595]: Segment reached optimal threshold of 14 pieces.
Dec 27 13:43:37.323852865 (526h14m37.1949359s) [noise=true , address=45.91.116.10:4113    , piece_id=OFHVWJWEHSVOOONDLXBZY72VB5G6FFXH2NXL45KY7TJ6CPVZ5UTQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12puRMny4nbgv2by4tAbXcBBSgG2d6BfwcDDRDcFzEdjSVUJxtd): protocol: expected piece hash; context canceled; EOF
Dec 27 13:43:37.323860349 (526h14m37.194943384s) [noise=true , address=45.91.116.10:4113    , piece_id=OFHVWJWEHSVOOONDLXBZY72VB5G6FFXH2NXL45KY7TJ6CPVZ5UTQ, seg_pos={0 0}, upload=461595]: Long tail cancellation due to optimal threshold
Dec 27 13:43:37.324005135 (526h14m37.1950882s) [noise=true , address=145.79.227.40:4079   , piece_id=237LYFTAG7G4HMIUKFZDDUTYJC7AAHHXTO5EIO46UEHY24BOVB3A, seg_pos={0 0}, upload=461595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RYKzn5zVSmxVbWwnBdX8oqtG1WE2MhPeEiWLf5dZeFAut6s2Y): protocol: expected piece hash; context canceled; EOF
Dec 27 13:43:37.324022659 (526h14m37.195105704s) [noise=true , address=145.79.227.40:4079   , piece_id=237LYFTAG7G4HMIUKFZDDUTYJC7AAHHXTO5EIO46UEHY24BOVB3A, seg_pos={0 0}, upload=461595]: Long tail cancellation due to optimal threshold
Dec 27 13:43:37.324143378 (526h14m37.195226413s) [noise=true , address=77.90.31.16:4739     , piece_id=H6RNS5VOYTMWX2N6GLV2KOQDPFPCBWPDNLINB7SXDJWHMJL3GYQQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129B3N7yjJvwCzoVLTqWsPbcLQ5TQUyAzg1RCkbSAVKPWYTG8eb): protocol: expected piece hash; context canceled; EOF
Dec 27 13:43:37.324154520 (526h14m37.195237555s) [noise=true , address=77.90.31.16:4739     , piece_id=H6RNS5VOYTMWX2N6GLV2KOQDPFPCBWPDNLINB7SXDJWHMJL3GYQQ, seg_pos={0 0}, upload=461595]: Long tail cancellation due to optimal threshold
Dec 27 13:43:37.324213442 (526h14m37.195296487s) [noise=true , address=64.49.13.169:4710    , piece_id=XIZKYK2E5FQJ37LYN535M7GXBBKXUQXFQWVMBWIBFDOAIIFGUAJA, seg_pos={0 0}, upload=461595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:57438->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:57438->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57438->64.49.13.169:4710: use of closed network connection
Dec 27 13:43:37.324249360 (526h14m37.195332405s) [noise=true , address=64.49.13.169:4710    , piece_id=XIZKYK2E5FQJ37LYN535M7GXBBKXUQXFQWVMBWIBFDOAIIFGUAJA, seg_pos={0 0}, upload=461595]: Long tail cancellation due to optimal threshold
Dec 27 13:43:37.324275360 (526h14m37.195358405s) [seg_pos={0 0}, upload=461595]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 13:43:37.324374018 (526h14m37.195457053s) [upload=461595]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 13:43:37.324381291 (526h14m37.195464337s) [upload=461595]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 13:43:37.679036406 (526h14m37.550119451s) [upload=461595]: Done uploading segments...
Dec 27 13:43:37.679056985 (526h14m37.55014003s) [upload=461595]: Upload finished. err: <nil>
Duration: 44.35830562s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
36 out of 60 is failed (threshold=6) [XX....XX....X.X.X.X.X..XXX.XX.XX.X.XX.X.XXXXXXXXXXX.X..XXXX.]
duration: 
5.001307032s
trace: 
https://jaeger.dev.storj.io/trace/5a5b66fdbd7b3071
FAIL (2025-12-27 14:07:07 UTC) 
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001331209s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip: 
2 out of 60 is failed (threshold=3) [................................................XX..........]
duration: 
7.058420243s
trace: 
https://jaeger.dev.storj.io/trace/75ef8b5e6bc2819a
FAIL (2025-12-27 14:06:32 UTC) 
orbiter warning: context deadline exceeded
Duration: 7.058449739s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
21 out of 60 is failed (threshold=3) [..X...X...X...XX.......X....X..XX.X.XXXX..X..X.....X.XXX..X.]
duration: 
5.888169517s
trace: 
https://jaeger.dev.storj.io/trace/6e0c241f5b6f833e
FAIL (2025-12-27 14:06:29 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: 7H7F4E3NFIUZLSLZDTICFEVR3ZDS2E2XV5FEFIU5NM4TBSFQ4H2A) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: DDUX6F7V3GETJCN6FU6DI4HFDJOVRVDYJI7RXR6A3H2BIJQCMENQ) context deadline exceeded
======LOGS======
Dec 27 14:05:42.174622851 (526h36m42.045705886s) [upload=461903]: Starting upload
Dec 27 14:05:42.174711871 (526h36m42.045794926s) [upload=461903]: Done starting upload
Dec 27 14:05:42.174810228 (526h36m42.045893273s) [upload=461903]: Uploading segments...
Dec 27 14:05:42.175091664 (526h36m42.046174719s) [upload=461903]: Got next segment. Inline: false
Dec 27 14:05:42.175140587 (526h36m42.046223622s) [upload=461903]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 14:05:42.175147209 (526h36m42.046230254s) [upload=461903]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 14:05:42.425678453 (526h36m42.296761498s) [seg_pos={0 0}, upload=461903]: Begin upload segment...
Dec 27 14:05:42.425738487 (526h36m42.296821523s) [seg_pos={0 0}, upload=461903]: Done begin upload segment.
Dec 27 14:05:42.425769326 (526h36m42.296852371s) [noise=true , address=77.90.31.18:4808     , piece_id=55D2V4ECKRXWDLCBX2VYHGYK2GZVDWIMK4GYMHJFPZZUESPAZCWQ, seg_pos={0 0}, upload=461903]: Uploading piece...
Dec 27 14:05:42.425861192 (526h36m42.296944227s) [noise=true , address=145.79.227.2:4113    , piece_id=LSOBOIUBMGT4HWCDJMDJUKGJQT4AO6IIHC3KQZXHJ4TQCCOHO44Q, seg_pos={0 0}, upload=461903]: Uploading piece...
Dec 27 14:05:42.426381272 (526h36m42.297464317s) [noise=true , address=145.79.227.12:4767   , piece_id=7JK6ZY4LQPSY2HVJYHPZG3LFES3UXZA3U5RBL6GMCU3F6AJMAUFQ, seg_pos={0 0}, upload=461903]: Uploading piece...
Dec 27 14:05:42.426727581 (526h36m42.297810616s) [noise=true , address=45.91.116.11:4119    , piece_id=GL4JYGWFBKEWVSIDMIYDENMN2XAGMQVBDGDS2XT443UDODBRYXVQ, seg_pos={0 0}, upload=461903]: Uploading piece...
Dec 27 14:05:42.427005120 (526h36m42.298088155s) [noise=true , address=145.79.227.50:4373   , piece_id=2PBWUVG46JZ3F22EUXZYKWRXLBY2YX6EIHVSUQI2EWANV4DFEGIA, seg_pos={0 0}, upload=461903]: Uploading piece...
Dec 27 14:05:42.427282517 (526h36m42.298365562s) [noise=true , address=77.90.31.16:4736     , piece_id=7H7F4E3NFIUZLSLZDTICFEVR3ZDS2E2XV5FEFIU5NM4TBSFQ4H2A, seg_pos={0 0}, upload=461903]: Uploading piece...
Dec 27 14:05:42.428321215 (526h36m42.29940425s) [noise=true , address=145.79.227.45:4231   , piece_id=4GYK4OYBLZZ6HQ5DITMFWKHFRG427NJRX22FOFVKQOQYJ2TORNFQ, seg_pos={0 0}, upload=461903]: Uploading piece...
Dec 27 14:05:42.428681642 (526h36m42.299764677s) [noise=true , address=77.90.31.11:4556     , piece_id=UOJV3RKCWOXMVOLX44NVNJUFYJVWRPFXMAAY2AYTKIFKMO7KXQ6Q, seg_pos={0 0}, upload=461903]: Uploading piece...
Dec 27 14:05:42.430016554 (526h36m42.301099589s) [noise=true , address=145.79.227.16:4911   , piece_id=L3SNXPPJMTOL6PO6QGUPRJGNNLHIDCB6VNJPGUCDD4ZNZ27J2ZWA, seg_pos={0 0}, upload=461903]: Uploading piece...
Dec 27 14:05:42.430172351 (526h36m42.301255396s) [noise=true , address=45.91.116.5:4297     , piece_id=DDUX6F7V3GETJCN6FU6DI4HFDJOVRVDYJI7RXR6A3H2BIJQCMENQ, seg_pos={0 0}, upload=461903]: Uploading piece...
Dec 27 14:05:42.431452768 (526h36m42.302535803s) [noise=true , address=145.79.227.41:4085   , piece_id=AAO5GFX4SYKTYUHB6WBPCIZ3XX7KXR4EB4T7VEZPNQSZODECJZNA, seg_pos={0 0}, upload=461903]: Uploading piece...
Dec 27 14:05:42.431468078 (526h36m42.302551123s) [noise=true , address=145.79.227.14:4837   , piece_id=O5XJ7MCAZVEZ4TUC2HLRSOP2X3FNSNTAXALP4C2YUGTESWEG23QQ, seg_pos={0 0}, upload=461903]: Uploading piece...
Dec 27 14:05:42.432756331 (526h36m42.303839366s) [noise=true , address=77.90.31.13:4627     , piece_id=TCO2SHFH4EC5KNB4LUO5T4J43C5DANDBZEYH2PZK4O422OOMPP2A, seg_pos={0 0}, upload=461903]: Uploading piece...
Dec 27 14:05:42.434033873 (526h36m42.305116938s) [noise=true , address=145.79.227.46:4601   , piece_id=JRHHU56YAVKME5WDKHS22NIXFVW5VFWLIFRI2TRIGYAHVAMXI7FQ, seg_pos={0 0}, upload=461903]: Uploading piece...
Dec 27 14:05:42.435025542 (526h36m42.306108577s) [noise=true , address=145.79.227.36:4518   , piece_id=5KENHRDZ53AGQGGOA43LWCQZUYFJPLUQMVLANGZUU7JFKZQWGZ3Q, seg_pos={0 0}, upload=461903]: Uploading piece...
Dec 27 14:05:42.437075566 (526h36m42.308158601s) [noise=true , address=145.79.227.10:4402   , piece_id=QUITOKPDBY74NRBDJUGMX76WRAIETACPQHJI2O4SFVYUCUIV7EOQ, seg_pos={0 0}, upload=461903]: Uploading piece...
Dec 27 14:05:44.135402769 (526h36m44.006485804s) [upload=461903]: Next returned nil segment
Dec 27 14:05:44.135423068 (526h36m44.006506103s) [upload=461903]: Waiting for error group managing segments...
Dec 27 14:05:44.417681756 (526h36m44.288764791s) [noise=true , address=145.79.227.46:4601   , piece_id=JRHHU56YAVKME5WDKHS22NIXFVW5VFWLIFRI2TRIGYAHVAMXI7FQ, seg_pos={0 0}, upload=461903]: Done uploading piece. err: <nil>
Dec 27 14:05:44.434422193 (526h36m44.305505238s) [noise=true , address=145.79.227.12:4767   , piece_id=7JK6ZY4LQPSY2HVJYHPZG3LFES3UXZA3U5RBL6GMCU3F6AJMAUFQ, seg_pos={0 0}, upload=461903]: Done uploading piece. err: <nil>
Dec 27 14:05:44.436344174 (526h36m44.307427209s) [noise=true , address=145.79.227.14:4837   , piece_id=O5XJ7MCAZVEZ4TUC2HLRSOP2X3FNSNTAXALP4C2YUGTESWEG23QQ, seg_pos={0 0}, upload=461903]: Done uploading piece. err: <nil>
Dec 27 14:05:44.442475128 (526h36m44.313558173s) [noise=true , address=145.79.227.45:4231   , piece_id=4GYK4OYBLZZ6HQ5DITMFWKHFRG427NJRX22FOFVKQOQYJ2TORNFQ, seg_pos={0 0}, upload=461903]: Done uploading piece. err: <nil>
Dec 27 14:05:44.445306850 (526h36m44.316389885s) [noise=true , address=145.79.227.36:4518   , piece_id=5KENHRDZ53AGQGGOA43LWCQZUYFJPLUQMVLANGZUU7JFKZQWGZ3Q, seg_pos={0 0}, upload=461903]: Done uploading piece. err: <nil>
Dec 27 14:05:44.453544626 (526h36m44.324627661s) [noise=true , address=145.79.227.41:4085   , piece_id=AAO5GFX4SYKTYUHB6WBPCIZ3XX7KXR4EB4T7VEZPNQSZODECJZNA, seg_pos={0 0}, upload=461903]: Done uploading piece. err: <nil>
Dec 27 14:05:44.456117115 (526h36m44.32720016s) [noise=true , address=145.79.227.16:4911   , piece_id=L3SNXPPJMTOL6PO6QGUPRJGNNLHIDCB6VNJPGUCDD4ZNZ27J2ZWA, seg_pos={0 0}, upload=461903]: Done uploading piece. err: <nil>
Dec 27 14:05:44.456558094 (526h36m44.327641129s) [noise=true , address=145.79.227.50:4373   , piece_id=2PBWUVG46JZ3F22EUXZYKWRXLBY2YX6EIHVSUQI2EWANV4DFEGIA, seg_pos={0 0}, upload=461903]: Done uploading piece. err: <nil>
Dec 27 14:05:44.456865349 (526h36m44.327948384s) [noise=true , address=145.79.227.2:4113    , piece_id=LSOBOIUBMGT4HWCDJMDJUKGJQT4AO6IIHC3KQZXHJ4TQCCOHO44Q, seg_pos={0 0}, upload=461903]: Done uploading piece. err: <nil>
Dec 27 14:05:44.464205105 (526h36m44.33528814s) [noise=true , address=145.79.227.10:4402   , piece_id=QUITOKPDBY74NRBDJUGMX76WRAIETACPQHJI2O4SFVYUCUIV7EOQ, seg_pos={0 0}, upload=461903]: Done uploading piece. err: <nil>
Dec 27 14:05:44.603684487 (526h36m44.474767522s) [noise=true , address=77.90.31.16:4736     , piece_id=7H7F4E3NFIUZLSLZDTICFEVR3ZDS2E2XV5FEFIU5NM4TBSFQ4H2A, seg_pos={0 0}, upload=461903]: Done uploading piece. err: <nil>
Dec 27 14:05:44.634489305 (526h36m44.50557235s) [noise=true , address=77.90.31.18:4808     , piece_id=55D2V4ECKRXWDLCBX2VYHGYK2GZVDWIMK4GYMHJFPZZUESPAZCWQ, seg_pos={0 0}, upload=461903]: Done uploading piece. err: <nil>
Dec 27 14:05:44.649453970 (526h36m44.520537006s) [noise=true , address=45.91.116.5:4297     , piece_id=DDUX6F7V3GETJCN6FU6DI4HFDJOVRVDYJI7RXR6A3H2BIJQCMENQ, seg_pos={0 0}, upload=461903]: Done uploading piece. err: <nil>
Dec 27 14:05:44.649476163 (526h36m44.520559228s) [seg_pos={0 0}, upload=461903]: Segment reached optimal threshold of 13 pieces.
Dec 27 14:05:44.649708966 (526h36m44.520792001s) [noise=true , address=77.90.31.11:4556     , piece_id=UOJV3RKCWOXMVOLX44NVNJUFYJVWRPFXMAAY2AYTKIFKMO7KXQ6Q, seg_pos={0 0}, upload=461903]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF
Dec 27 14:05:44.649715268 (526h36m44.520798303s) [noise=true , address=77.90.31.11:4556     , piece_id=UOJV3RKCWOXMVOLX44NVNJUFYJVWRPFXMAAY2AYTKIFKMO7KXQ6Q, seg_pos={0 0}, upload=461903]: Long tail cancellation due to optimal threshold
Dec 27 14:05:44.650062809 (526h36m44.521145864s) [noise=true , address=77.90.31.13:4627     , piece_id=TCO2SHFH4EC5KNB4LUO5T4J43C5DANDBZEYH2PZK4O422OOMPP2A, seg_pos={0 0}, upload=461903]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: expected piece hash; context canceled; EOF
Dec 27 14:05:44.650071978 (526h36m44.521155023s) [noise=true , address=77.90.31.13:4627     , piece_id=TCO2SHFH4EC5KNB4LUO5T4J43C5DANDBZEYH2PZK4O422OOMPP2A, seg_pos={0 0}, upload=461903]: Long tail cancellation due to optimal threshold
Dec 27 14:05:44.650217314 (526h36m44.521300349s) [noise=true , address=45.91.116.11:4119    , piece_id=GL4JYGWFBKEWVSIDMIYDENMN2XAGMQVBDGDS2XT443UDODBRYXVQ, seg_pos={0 0}, upload=461903]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF
Dec 27 14:05:44.650223436 (526h36m44.521306471s) [noise=true , address=45.91.116.11:4119    , piece_id=GL4JYGWFBKEWVSIDMIYDENMN2XAGMQVBDGDS2XT443UDODBRYXVQ, seg_pos={0 0}, upload=461903]: Long tail cancellation due to optimal threshold
Dec 27 14:05:44.650238524 (526h36m44.521321559s) [seg_pos={0 0}, upload=461903]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 14:05:44.650305211 (526h36m44.521388246s) [upload=461903]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 14:05:44.650311043 (526h36m44.521394078s) [upload=461903]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 14:05:44.941554529 (526h36m44.812637564s) [upload=461903]: Done uploading segments...
Dec 27 14:05:44.941574837 (526h36m44.812657872s) [upload=461903]: Upload finished. err: <nil>
Duration: 5.888195315s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
6 out of 60 is failed (threshold=3) [X...X...X..........X.............XX.........................]
duration: 
25.03658569s
trace: 
https://jaeger.dev.storj.io/trace/3ed1e657d506c4ee
FAIL (2025-12-27 14:06:24 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, Piece ID: HEMLEGWLVMK4LG5DM3PUWU5HSUEHKBECYOMHUXIQRGZGNXPXFLBQ) context deadline exceeded; piecestore: (Node ID: 12wXqePK1kK7iBo6wHcbqkRmkED8LGJ3rzQNJAWa14cHYhx1oyB, Piece ID: M5NU2MR5LQWOKNLVKKEJXMJNVEIZVDDNIZHWNN7XXHVTJRS2OQNA) context deadline exceeded; piecestore: (Node ID: 12ATtVD3XyFxzEqwQpSF4t1MWygqEa4Y1CkNj1s43eud8UtuSmx, Piece ID: ET7O467BME3OED32PFLJKYVIWZRDF6WMPXUT34OFWSPXLSMKKOWQ) context deadline exceeded; piecestore: (Node ID: 12UjFkFn2UEvr57LDorcijDCvkBkkKitgZEFx6Yqfk5gMStQvJG, Piece ID: SF3IKAIYXQR4YPWNVXT66SEHVM54R3K3VJHA6L5RBOU3DITOR5LQ) context deadline exceeded
======LOGS======
) [noise=true , address=77.90.31.16:4739     , piece_id=H6RNS5VOYTMWX2N6GLV2KOQDPFPCBWPDNLINB7SXDJWHMJL3GYQQ, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.319646322 (526h14m34.190729357s) [noise=true , address=77.90.31.21:4899     , piece_id=SF3IKAIYXQR4YPWNVXT66SEHVM54R3K3VJHA6L5RBOU3DITOR5LQ, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.319933248 (526h14m34.191016283s) [noise=true , address=45.91.116.12:4234    , piece_id=Y2FIVYLZBQAYBO2TDGIZ4RDQHQJ3WOQH565A7ESIKANQQTQDFQZQ, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.320400588 (526h14m34.191483633s) [noise=true , address=145.79.227.29:4952   , piece_id=VXZLF4OGRIR3MYQMMLKLTD235A55F4YTHB3D4YCVCQUL4FZTSFJQ, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.320407341 (526h14m34.191490386s) [noise=true , address=145.79.227.46:4621   , piece_id=XZ4MKAZAGQKVEJJGSZ5QGT4AT3MFXXNMXICVFCI6SP3E4EVEHZGQ, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.320675030 (526h14m34.191758075s) [noise=true , address=45.91.116.10:4113    , piece_id=OFHVWJWEHSVOOONDLXBZY72VB5G6FFXH2NXL45KY7TJ6CPVZ5UTQ, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.320802022 (526h14m34.191885067s) [noise=true , address=77.90.31.11:4569     , piece_id=YC2Q3HPJDVY3NEDZNC2Z4KVN6S6DDDGRARASF7KALFAOQ335A7HA, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.321169643 (526h14m34.192252678s) [noise=true , address=77.90.31.19:4869     , piece_id=HEMLEGWLVMK4LG5DM3PUWU5HSUEHKBECYOMHUXIQRGZGNXPXFLBQ, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.321443483 (526h14m34.192526518s) [noise=true , address=45.91.116.14:4202    , piece_id=M5NU2MR5LQWOKNLVKKEJXMJNVEIZVDDNIZHWNN7XXHVTJRS2OQNA, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.321731813 (526h14m34.192814848s) [noise=true , address=77.90.31.18:4819     , piece_id=YS4DXTHLGCPFOKKLBA2DK2BVOAPTQEGVOPBSRCLFV26SVP6YFHGA, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.322221776 (526h14m34.193304811s) [noise=true , address=77.90.31.15:4708     , piece_id=3NX6E54XMZXQAQQE27LTWUSEF5GAIF2XOPQFFAP7LOA2PL7KDL3Q, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.322547776 (526h14m34.193630811s) [noise=true , address=145.79.227.40:4079   , piece_id=237LYFTAG7G4HMIUKFZDDUTYJC7AAHHXTO5EIO46UEHY24BOVB3A, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.322908093 (526h14m34.193991138s) [noise=true , address=145.79.227.29:4975   , piece_id=DAMNBVP7DRSW5OAGYRRP4K5SNJ2WEUT7IAXO77KWRHLQEMBNERZQ, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.323267968 (526h14m34.194351003s) [noise=true , address=145.79.227.30:4787   , piece_id=OB2OTY3ZEHIUFYM6FY7HIS7HQ353Y2HL6XU4476HXOD26QVA4DPQ, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.323618165 (526h14m34.19470121s) [noise=true , address=64.49.13.169:4710    , piece_id=XIZKYK2E5FQJ37LYN535M7GXBBKXUQXFQWVMBWIBFDOAIIFGUAJA, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:34.324084893 (526h14m34.195167928s) [noise=true , address=193.149.164.143:4694 , piece_id=ET7O467BME3OED32PFLJKYVIWZRDF6WMPXUT34OFWSPXLSMKKOWQ, seg_pos={0 0}, upload=461595]: Uploading piece...
Dec 27 13:43:35.783540136 (526h14m35.654623181s) [upload=461595]: Next returned nil segment
Dec 27 13:43:35.783555856 (526h14m35.654638891s) [upload=461595]: Waiting for error group managing segments...
Dec 27 13:43:36.192287250 (526h14m36.063370296s) [noise=true , address=145.79.227.29:4952   , piece_id=VXZLF4OGRIR3MYQMMLKLTD235A55F4YTHB3D4YCVCQUL4FZTSFJQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:36.200200289 (526h14m36.071283334s) [noise=true , address=145.79.227.46:4621   , piece_id=XZ4MKAZAGQKVEJJGSZ5QGT4AT3MFXXNMXICVFCI6SP3E4EVEHZGQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:36.238087532 (526h14m36.109170577s) [noise=true , address=145.79.227.29:4975   , piece_id=DAMNBVP7DRSW5OAGYRRP4K5SNJ2WEUT7IAXO77KWRHLQEMBNERZQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:36.379631346 (526h14m36.250714391s) [noise=true , address=77.90.31.19:4869     , piece_id=HEMLEGWLVMK4LG5DM3PUWU5HSUEHKBECYOMHUXIQRGZGNXPXFLBQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:36.382945286 (526h14m36.254028321s) [noise=true , address=145.79.227.34:4909   , piece_id=Z6RF5PSOGOZMOXPQYXHMXGFXIELVXXVU4NXXRTAQCIELQXLMVXBQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:36.400290103 (526h14m36.271373138s) [noise=true , address=77.90.31.18:4819     , piece_id=YS4DXTHLGCPFOKKLBA2DK2BVOAPTQEGVOPBSRCLFV26SVP6YFHGA, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:36.436707208 (526h14m36.307790243s) [noise=true , address=45.91.116.14:4202    , piece_id=M5NU2MR5LQWOKNLVKKEJXMJNVEIZVDDNIZHWNN7XXHVTJRS2OQNA, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:36.487651475 (526h14m36.35873451s) [noise=true , address=45.91.116.12:4234    , piece_id=Y2FIVYLZBQAYBO2TDGIZ4RDQHQJ3WOQH565A7ESIKANQQTQDFQZQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:36.500850769 (526h14m36.371933814s) [noise=true , address=77.90.31.11:4569     , piece_id=YC2Q3HPJDVY3NEDZNC2Z4KVN6S6DDDGRARASF7KALFAOQ335A7HA, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:36.518828551 (526h14m36.389911586s) [noise=true , address=77.90.31.15:4708     , piece_id=3NX6E54XMZXQAQQE27LTWUSEF5GAIF2XOPQFFAP7LOA2PL7KDL3Q, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:37.051607243 (526h14m36.922690289s) [noise=true , address=145.79.227.44:4228   , piece_id=6SAYEAOTTCOUFPU75C7YRIULWJGXL4PK2YKCKFB47EUBWMRQQQOQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:37.096826909 (526h14m36.967909954s) [noise=true , address=145.79.227.30:4787   , piece_id=OB2OTY3ZEHIUFYM6FY7HIS7HQ353Y2HL6XU4476HXOD26QVA4DPQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:37.140738995 (526h14m37.01182203s) [noise=true , address=77.90.31.21:4899     , piece_id=SF3IKAIYXQR4YPWNVXT66SEHVM54R3K3VJHA6L5RBOU3DITOR5LQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:37.323613990 (526h14m37.194697035s) [noise=true , address=193.149.164.143:4694 , piece_id=ET7O467BME3OED32PFLJKYVIWZRDF6WMPXUT34OFWSPXLSMKKOWQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: <nil>
Dec 27 13:43:37.323649618 (526h14m37.194732663s) [seg_pos={0 0}, upload=461595]: Segment reached optimal threshold of 14 pieces.
Dec 27 13:43:37.323852865 (526h14m37.1949359s) [noise=true , address=45.91.116.10:4113    , piece_id=OFHVWJWEHSVOOONDLXBZY72VB5G6FFXH2NXL45KY7TJ6CPVZ5UTQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12puRMny4nbgv2by4tAbXcBBSgG2d6BfwcDDRDcFzEdjSVUJxtd): protocol: expected piece hash; context canceled; EOF
Dec 27 13:43:37.323860349 (526h14m37.194943384s) [noise=true , address=45.91.116.10:4113    , piece_id=OFHVWJWEHSVOOONDLXBZY72VB5G6FFXH2NXL45KY7TJ6CPVZ5UTQ, seg_pos={0 0}, upload=461595]: Long tail cancellation due to optimal threshold
Dec 27 13:43:37.324005135 (526h14m37.1950882s) [noise=true , address=145.79.227.40:4079   , piece_id=237LYFTAG7G4HMIUKFZDDUTYJC7AAHHXTO5EIO46UEHY24BOVB3A, seg_pos={0 0}, upload=461595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RYKzn5zVSmxVbWwnBdX8oqtG1WE2MhPeEiWLf5dZeFAut6s2Y): protocol: expected piece hash; context canceled; EOF
Dec 27 13:43:37.324022659 (526h14m37.195105704s) [noise=true , address=145.79.227.40:4079   , piece_id=237LYFTAG7G4HMIUKFZDDUTYJC7AAHHXTO5EIO46UEHY24BOVB3A, seg_pos={0 0}, upload=461595]: Long tail cancellation due to optimal threshold
Dec 27 13:43:37.324143378 (526h14m37.195226413s) [noise=true , address=77.90.31.16:4739     , piece_id=H6RNS5VOYTMWX2N6GLV2KOQDPFPCBWPDNLINB7SXDJWHMJL3GYQQ, seg_pos={0 0}, upload=461595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129B3N7yjJvwCzoVLTqWsPbcLQ5TQUyAzg1RCkbSAVKPWYTG8eb): protocol: expected piece hash; context canceled; EOF
Dec 27 13:43:37.324154520 (526h14m37.195237555s) [noise=true , address=77.90.31.16:4739     , piece_id=H6RNS5VOYTMWX2N6GLV2KOQDPFPCBWPDNLINB7SXDJWHMJL3GYQQ, seg_pos={0 0}, upload=461595]: Long tail cancellation due to optimal threshold
Dec 27 13:43:37.324213442 (526h14m37.195296487s) [noise=true , address=64.49.13.169:4710    , piece_id=XIZKYK2E5FQJ37LYN535M7GXBBKXUQXFQWVMBWIBFDOAIIFGUAJA, seg_pos={0 0}, upload=461595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:57438->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:57438->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57438->64.49.13.169:4710: use of closed network connection
Dec 27 13:43:37.324249360 (526h14m37.195332405s) [noise=true , address=64.49.13.169:4710    , piece_id=XIZKYK2E5FQJ37LYN535M7GXBBKXUQXFQWVMBWIBFDOAIIFGUAJA, seg_pos={0 0}, upload=461595]: Long tail cancellation due to optimal threshold
Dec 27 13:43:37.324275360 (526h14m37.195358405s) [seg_pos={0 0}, upload=461595]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 13:43:37.324374018 (526h14m37.195457053s) [upload=461595]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 13:43:37.324381291 (526h14m37.195464337s) [upload=461595]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 13:43:37.679036406 (526h14m37.550119451s) [upload=461595]: Done uploading segments...
Dec 27 13:43:37.679056985 (526h14m37.55014003s) [upload=461595]: Upload finished. err: <nil>
Duration: 25.036605778s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
21 out of 60 is failed (threshold=3) [...X...X...X...XX.......X....X..XX.X.XXXX..X..X.....X.XXX..X]
duration: 
30.004030958s
trace: 
https://jaeger.dev.storj.io/trace/6e0c241f5b6f833e
FAIL (2025-12-27 14:06:11 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: 7H7F4E3NFIUZLSLZDTICFEVR3ZDS2E2XV5FEFIU5NM4TBSFQ4H2A) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: DDUX6F7V3GETJCN6FU6DI4HFDJOVRVDYJI7RXR6A3H2BIJQCMENQ) context deadline exceeded
======LOGS======
Dec 27 14:05:42.174622851 (526h36m42.045705886s) [upload=461903]: Starting upload
Dec 27 14:05:42.174711871 (526h36m42.045794926s) [upload=461903]: Done starting upload
Dec 27 14:05:42.174810228 (526h36m42.045893273s) [upload=461903]: Uploading segments...
Dec 27 14:05:42.175091664 (526h36m42.046174719s) [upload=461903]: Got next segment. Inline: false
Dec 27 14:05:42.175140587 (526h36m42.046223622s) [upload=461903]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 14:05:42.175147209 (526h36m42.046230254s) [upload=461903]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 14:05:42.425678453 (526h36m42.296761498s) [seg_pos={0 0}, upload=461903]: Begin upload segment...
Dec 27 14:05:42.425738487 (526h36m42.296821523s) [seg_pos={0 0}, upload=461903]: Done begin upload segment.
Dec 27 14:05:42.425769326 (526h36m42.296852371s) [noise=true , address=77.90.31.18:4808     , piece_id=55D2V4ECKRXWDLCBX2VYHGYK2GZVDWIMK4GYMHJFPZZUESPAZCWQ, seg_pos={0 0}, upload=461903]: Uploading piece...
Dec 27 14:05:42.425861192 (526h36m42.296944227s) [noise=true , address=145.79.227.2:4113    , piece_id=LSOBOIUBMGT4HWCDJMDJUKGJQT4AO6IIHC3KQZXHJ4TQCCOHO44Q, seg_pos={0 0}, upload=461903]: Uploading piece...
Dec 27 14:05:42.426381272 (526h36m42.297464317s) [noise=true , address=145.79.227.12:4767   , piece_id=7JK6ZY4LQPSY2HVJYHPZG3LFES3UXZA3U5RBL6GMCU3F6AJMAUFQ, seg_pos={0 0}, upload=461903]: Uploading piece...
Dec 27 14:05:42.426727581 (526h36m42.297810616s) [noise=true , address=45.91.116.11:4119    , piece_id=GL4JYGWFBKEWVSIDMIYDENMN2XAGMQVBDGDS2XT443UDODBRYXVQ, seg_pos={0 0}, upload=461903]: Uploading piece...
Dec 27 14:05:42.427005120 (526h36m42.298088155s) [noise=true , address=145.79.227.50:4373   , piece_id=2PBWUVG46JZ3F22EUXZYKWRXLBY2YX6EIHVSUQI2EWANV4DFEGIA, seg_pos={0 0}, upload=461903]: Uploading piece...
Dec 27 14:05:42.427282517 (526h36m42.298365562s) [noise=true , address=77.90.31.16:4736     , piece_id=7H7F4E3NFIUZLSLZDTICFEVR3ZDS2E2XV5FEFIU5NM4TBSFQ4H2A, seg_pos={0 0}, upload=461903]: Uploading piece...
Dec 27 14:05:42.428321215 (526h36m42.29940425s) [noise=true , address=145.79.227.45:4231   , piece_id=4GYK4OYBLZZ6HQ5DITMFWKHFRG427NJRX22FOFVKQOQYJ2TORNFQ, seg_pos={0 0}, upload=461903]: Uploading piece...
Dec 27 14:05:42.428681642 (526h36m42.299764677s) [noise=true , address=77.90.31.11:4556     , piece_id=UOJV3RKCWOXMVOLX44NVNJUFYJVWRPFXMAAY2AYTKIFKMO7KXQ6Q, seg_pos={0 0}, upload=461903]: Uploading piece...
Dec 27 14:05:42.430016554 (526h36m42.301099589s) [noise=true , address=145.79.227.16:4911   , piece_id=L3SNXPPJMTOL6PO6QGUPRJGNNLHIDCB6VNJPGUCDD4ZNZ27J2ZWA, seg_pos={0 0}, upload=461903]: Uploading piece...
Dec 27 14:05:42.430172351 (526h36m42.301255396s) [noise=true , address=45.91.116.5:4297     , piece_id=DDUX6F7V3GETJCN6FU6DI4HFDJOVRVDYJI7RXR6A3H2BIJQCMENQ, seg_pos={0 0}, upload=461903]: Uploading piece...
Dec 27 14:05:42.431452768 (526h36m42.302535803s) [noise=true , address=145.79.227.41:4085   , piece_id=AAO5GFX4SYKTYUHB6WBPCIZ3XX7KXR4EB4T7VEZPNQSZODECJZNA, seg_pos={0 0}, upload=461903]: Uploading piece...
Dec 27 14:05:42.431468078 (526h36m42.302551123s) [noise=true , address=145.79.227.14:4837   , piece_id=O5XJ7MCAZVEZ4TUC2HLRSOP2X3FNSNTAXALP4C2YUGTESWEG23QQ, seg_pos={0 0}, upload=461903]: Uploading piece...
Dec 27 14:05:42.432756331 (526h36m42.303839366s) [noise=true , address=77.90.31.13:4627     , piece_id=TCO2SHFH4EC5KNB4LUO5T4J43C5DANDBZEYH2PZK4O422OOMPP2A, seg_pos={0 0}, upload=461903]: Uploading piece...
Dec 27 14:05:42.434033873 (526h36m42.305116938s) [noise=true , address=145.79.227.46:4601   , piece_id=JRHHU56YAVKME5WDKHS22NIXFVW5VFWLIFRI2TRIGYAHVAMXI7FQ, seg_pos={0 0}, upload=461903]: Uploading piece...
Dec 27 14:05:42.435025542 (526h36m42.306108577s) [noise=true , address=145.79.227.36:4518   , piece_id=5KENHRDZ53AGQGGOA43LWCQZUYFJPLUQMVLANGZUU7JFKZQWGZ3Q, seg_pos={0 0}, upload=461903]: Uploading piece...
Dec 27 14:05:42.437075566 (526h36m42.308158601s) [noise=true , address=145.79.227.10:4402   , piece_id=QUITOKPDBY74NRBDJUGMX76WRAIETACPQHJI2O4SFVYUCUIV7EOQ, seg_pos={0 0}, upload=461903]: Uploading piece...
Dec 27 14:05:44.135402769 (526h36m44.006485804s) [upload=461903]: Next returned nil segment
Dec 27 14:05:44.135423068 (526h36m44.006506103s) [upload=461903]: Waiting for error group managing segments...
Dec 27 14:05:44.417681756 (526h36m44.288764791s) [noise=true , address=145.79.227.46:4601   , piece_id=JRHHU56YAVKME5WDKHS22NIXFVW5VFWLIFRI2TRIGYAHVAMXI7FQ, seg_pos={0 0}, upload=461903]: Done uploading piece. err: <nil>
Dec 27 14:05:44.434422193 (526h36m44.305505238s) [noise=true , address=145.79.227.12:4767   , piece_id=7JK6ZY4LQPSY2HVJYHPZG3LFES3UXZA3U5RBL6GMCU3F6AJMAUFQ, seg_pos={0 0}, upload=461903]: Done uploading piece. err: <nil>
Dec 27 14:05:44.436344174 (526h36m44.307427209s) [noise=true , address=145.79.227.14:4837   , piece_id=O5XJ7MCAZVEZ4TUC2HLRSOP2X3FNSNTAXALP4C2YUGTESWEG23QQ, seg_pos={0 0}, upload=461903]: Done uploading piece. err: <nil>
Dec 27 14:05:44.442475128 (526h36m44.313558173s) [noise=true , address=145.79.227.45:4231   , piece_id=4GYK4OYBLZZ6HQ5DITMFWKHFRG427NJRX22FOFVKQOQYJ2TORNFQ, seg_pos={0 0}, upload=461903]: Done uploading piece. err: <nil>
Dec 27 14:05:44.445306850 (526h36m44.316389885s) [noise=true , address=145.79.227.36:4518   , piece_id=5KENHRDZ53AGQGGOA43LWCQZUYFJPLUQMVLANGZUU7JFKZQWGZ3Q, seg_pos={0 0}, upload=461903]: Done uploading piece. err: <nil>
Dec 27 14:05:44.453544626 (526h36m44.324627661s) [noise=true , address=145.79.227.41:4085   , piece_id=AAO5GFX4SYKTYUHB6WBPCIZ3XX7KXR4EB4T7VEZPNQSZODECJZNA, seg_pos={0 0}, upload=461903]: Done uploading piece. err: <nil>
Dec 27 14:05:44.456117115 (526h36m44.32720016s) [noise=true , address=145.79.227.16:4911   , piece_id=L3SNXPPJMTOL6PO6QGUPRJGNNLHIDCB6VNJPGUCDD4ZNZ27J2ZWA, seg_pos={0 0}, upload=461903]: Done uploading piece. err: <nil>
Dec 27 14:05:44.456558094 (526h36m44.327641129s) [noise=true , address=145.79.227.50:4373   , piece_id=2PBWUVG46JZ3F22EUXZYKWRXLBY2YX6EIHVSUQI2EWANV4DFEGIA, seg_pos={0 0}, upload=461903]: Done uploading piece. err: <nil>
Dec 27 14:05:44.456865349 (526h36m44.327948384s) [noise=true , address=145.79.227.2:4113    , piece_id=LSOBOIUBMGT4HWCDJMDJUKGJQT4AO6IIHC3KQZXHJ4TQCCOHO44Q, seg_pos={0 0}, upload=461903]: Done uploading piece. err: <nil>
Dec 27 14:05:44.464205105 (526h36m44.33528814s) [noise=true , address=145.79.227.10:4402   , piece_id=QUITOKPDBY74NRBDJUGMX76WRAIETACPQHJI2O4SFVYUCUIV7EOQ, seg_pos={0 0}, upload=461903]: Done uploading piece. err: <nil>
Dec 27 14:05:44.603684487 (526h36m44.474767522s) [noise=true , address=77.90.31.16:4736     , piece_id=7H7F4E3NFIUZLSLZDTICFEVR3ZDS2E2XV5FEFIU5NM4TBSFQ4H2A, seg_pos={0 0}, upload=461903]: Done uploading piece. err: <nil>
Dec 27 14:05:44.634489305 (526h36m44.50557235s) [noise=true , address=77.90.31.18:4808     , piece_id=55D2V4ECKRXWDLCBX2VYHGYK2GZVDWIMK4GYMHJFPZZUESPAZCWQ, seg_pos={0 0}, upload=461903]: Done uploading piece. err: <nil>
Dec 27 14:05:44.649453970 (526h36m44.520537006s) [noise=true , address=45.91.116.5:4297     , piece_id=DDUX6F7V3GETJCN6FU6DI4HFDJOVRVDYJI7RXR6A3H2BIJQCMENQ, seg_pos={0 0}, upload=461903]: Done uploading piece. err: <nil>
Dec 27 14:05:44.649476163 (526h36m44.520559228s) [seg_pos={0 0}, upload=461903]: Segment reached optimal threshold of 13 pieces.
Dec 27 14:05:44.649708966 (526h36m44.520792001s) [noise=true , address=77.90.31.11:4556     , piece_id=UOJV3RKCWOXMVOLX44NVNJUFYJVWRPFXMAAY2AYTKIFKMO7KXQ6Q, seg_pos={0 0}, upload=461903]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF
Dec 27 14:05:44.649715268 (526h36m44.520798303s) [noise=true , address=77.90.31.11:4556     , piece_id=UOJV3RKCWOXMVOLX44NVNJUFYJVWRPFXMAAY2AYTKIFKMO7KXQ6Q, seg_pos={0 0}, upload=461903]: Long tail cancellation due to optimal threshold
Dec 27 14:05:44.650062809 (526h36m44.521145864s) [noise=true , address=77.90.31.13:4627     , piece_id=TCO2SHFH4EC5KNB4LUO5T4J43C5DANDBZEYH2PZK4O422OOMPP2A, seg_pos={0 0}, upload=461903]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: expected piece hash; context canceled; EOF
Dec 27 14:05:44.650071978 (526h36m44.521155023s) [noise=true , address=77.90.31.13:4627     , piece_id=TCO2SHFH4EC5KNB4LUO5T4J43C5DANDBZEYH2PZK4O422OOMPP2A, seg_pos={0 0}, upload=461903]: Long tail cancellation due to optimal threshold
Dec 27 14:05:44.650217314 (526h36m44.521300349s) [noise=true , address=45.91.116.11:4119    , piece_id=GL4JYGWFBKEWVSIDMIYDENMN2XAGMQVBDGDS2XT443UDODBRYXVQ, seg_pos={0 0}, upload=461903]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF
Dec 27 14:05:44.650223436 (526h36m44.521306471s) [noise=true , address=45.91.116.11:4119    , piece_id=GL4JYGWFBKEWVSIDMIYDENMN2XAGMQVBDGDS2XT443UDODBRYXVQ, seg_pos={0 0}, upload=461903]: Long tail cancellation due to optimal threshold
Dec 27 14:05:44.650238524 (526h36m44.521321559s) [seg_pos={0 0}, upload=461903]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 14:05:44.650305211 (526h36m44.521388246s) [upload=461903]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 14:05:44.650311043 (526h36m44.521394078s) [upload=461903]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 14:05:44.941554529 (526h36m44.812637564s) [upload=461903]: Done uploading segments...
Dec 27 14:05:44.941574837 (526h36m44.812657872s) [upload=461903]: Upload finished. err: <nil>
Duration: 30.004056897s core uplink us-select-1