Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................................X......................] duration: 3.528041251s trace: https://jaeger.dev.storj.io/trace/22a47ec27f9a101c FAIL (2025-12-30 22:10:17 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.528064886s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..X...X.........X.........................X.................] duration: 5.881078347s trace: https://jaeger.dev.storj.io/trace/57a7ab98e7eb0d3f FAIL (2025-12-30 22:10:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 21:58:42.298454527 (606h29m42.169537592s) [upload=531550]: Starting upload Dec 30 21:58:42.298523438 (606h29m42.169606473s) [upload=531550]: Done starting upload Dec 30 21:58:42.298568774 (606h29m42.169651809s) [upload=531550]: Uploading segments... Dec 30 21:58:42.298697850 (606h29m42.169780885s) [upload=531550]: Got next segment. Inline: false Dec 30 21:58:42.298728649 (606h29m42.169811694s) [upload=531550]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 21:58:42.298733227 (606h29m42.169816262s) [upload=531550]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 21:58:42.545734213 (606h29m42.416817288s) [seg_pos={0 0}, upload=531550]: Begin upload segment... Dec 30 21:58:42.545786032 (606h29m42.416869078s) [seg_pos={0 0}, upload=531550]: Done begin upload segment. Dec 30 21:58:42.545855104 (606h29m42.416938149s) [noise=true , address=145.79.227.23:4547 , piece_id=LYGUMTAP7OPJWGRVFY65GUZFE4BBTV73JTG657DR5OM26LA7UFZA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.545897384 (606h29m42.416980449s) [noise=true , address=145.79.227.49:4339 , piece_id=WBBPZK2VY2QVRTQNXEKQ7TRYPWBDYS2TGTKEDAAXQ2AT4TUQXTQQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.546757752 (606h29m42.417840817s) [noise=true , address=45.91.116.12:4225 , piece_id=LLYEKE4TJ54KECLAC7KFUV44IVHYCYLF45I43G5V3DPV5QDBUOIQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.546816995 (606h29m42.41790004s) [noise=true , address=145.79.227.40:4050 , piece_id=LA7KLHHWRNHULLPAZWQ6MLS76SLN3BONYXZJXCUHGFC5JZ4SQRIA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.547216927 (606h29m42.418299962s) [noise=true , address=145.79.227.15:4876 , piece_id=L2OIJAB6OOANT2EPFNVNLTMYCH334JGFBFASD44EHOAWTAHKUV5Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.547615996 (606h29m42.418699042s) [noise=true , address=145.79.227.17:4971 , piece_id=DYZUALBL3V75TKGK57LOJABWVCOD5UVS3WH74I55OVUDGRQUA3QQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.547841055 (606h29m42.4189241s) [noise=true , address=77.90.31.21:4877 , piece_id=BD2FEFNAGYWZYDA2IMOVHA7JE66EXSF3A3PL54Q2DUOJFG5ALG4Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548023322 (606h29m42.419106357s) [noise=true , address=45.91.116.14:4190 , piece_id=HHU5YZBTJEEWYVST6XQCKGWKRLRQIIR2UQTQ27RLK53BNAVOSHFQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548380923 (606h29m42.419463968s) [noise=true , address=45.91.116.11:4120 , piece_id=FZ5L6FM7IHX65EVFGVXXHEF43XQ4EBUH4C7VZUSTH2MVWLNR5M2Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548560475 (606h29m42.41964351s) [noise=true , address=145.79.227.39:4013 , piece_id=I4XP47SFJICIYIPEN4E7UC2HXRXH4IXVJYGHXTZBCUESAHNFQXJA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548719268 (606h29m42.419802313s) [noise=true , address=145.79.227.31:4797 , piece_id=S5YBLITONGU6RHPQENP2TCVVWOHEMXPMYXLNYWBZNXWMFI26PVUA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549024107 (606h29m42.420107152s) [noise=true , address=145.79.227.7:4294 , piece_id=C6N5OIRLV3LLSCEIOGQJKWPKENGZYNAPY6V2AYZNKUMACBP5SLWQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549197708 (606h29m42.420280753s) [noise=true , address=145.79.227.16:4910 , piece_id=O4YO367LEVZSRI5E75YEZ7W7YI2KGKKQDXCOHQ3LALE4WUAPQVHQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549506406 (606h29m42.420589441s) [noise=true , address=145.79.227.12:4765 , piece_id=4ZF5YQP4Z5HBTX5YWUDHZV55Q47WASRLMV6ZYJXJRL2QBAVQQM4Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549971591 (606h29m42.421054626s) [noise=true , address=145.79.227.32:4835 , piece_id=CN4NMCFNYKOCG2JCIEXJBL2PLEYWD5THCZB4WQDURIECFVHLLX6Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.551571588 (606h29m42.422654623s) [noise=true , address=145.79.227.24:4581 , piece_id=Y6GBEOR7DQX2H6LBHXYFOXRBX3GP7AKPYVGTQNMUEXKPXLONUUGQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:44.247717212 (606h29m44.118800248s) [upload=531550]: Next returned nil segment Dec 30 21:58:44.247730348 (606h29m44.118813383s) [upload=531550]: Waiting for error group managing segments... Dec 30 21:58:44.532148720 (606h29m44.403231755s) [noise=true , address=145.79.227.31:4797 , piece_id=S5YBLITONGU6RHPQENP2TCVVWOHEMXPMYXLNYWBZNXWMFI26PVUA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.534655293 (606h29m44.405738328s) [noise=true , address=145.79.227.39:4013 , piece_id=I4XP47SFJICIYIPEN4E7UC2HXRXH4IXVJYGHXTZBCUESAHNFQXJA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.544709427 (606h29m44.415792462s) [noise=true , address=145.79.227.12:4765 , piece_id=4ZF5YQP4Z5HBTX5YWUDHZV55Q47WASRLMV6ZYJXJRL2QBAVQQM4Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.545822306 (606h29m44.416905342s) [noise=true , address=145.79.227.7:4294 , piece_id=C6N5OIRLV3LLSCEIOGQJKWPKENGZYNAPY6V2AYZNKUMACBP5SLWQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.549465162 (606h29m44.420548198s) [noise=true , address=145.79.227.24:4581 , piece_id=Y6GBEOR7DQX2H6LBHXYFOXRBX3GP7AKPYVGTQNMUEXKPXLONUUGQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.556935928 (606h29m44.428018964s) [noise=true , address=145.79.227.15:4876 , piece_id=L2OIJAB6OOANT2EPFNVNLTMYCH334JGFBFASD44EHOAWTAHKUV5Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.561283296 (606h29m44.432366331s) [noise=true , address=145.79.227.16:4910 , piece_id=O4YO367LEVZSRI5E75YEZ7W7YI2KGKKQDXCOHQ3LALE4WUAPQVHQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.614194264 (606h29m44.485277309s) [noise=true , address=145.79.227.17:4971 , piece_id=DYZUALBL3V75TKGK57LOJABWVCOD5UVS3WH74I55OVUDGRQUA3QQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.615630689 (606h29m44.486713724s) [noise=true , address=145.79.227.32:4835 , piece_id=CN4NMCFNYKOCG2JCIEXJBL2PLEYWD5THCZB4WQDURIECFVHLLX6Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.639672077 (606h29m44.510755122s) [noise=true , address=145.79.227.23:4547 , piece_id=LYGUMTAP7OPJWGRVFY65GUZFE4BBTV73JTG657DR5OM26LA7UFZA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.699618771 (606h29m44.570701817s) [noise=true , address=77.90.31.21:4877 , piece_id=BD2FEFNAGYWZYDA2IMOVHA7JE66EXSF3A3PL54Q2DUOJFG5ALG4Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.841535416 (606h29m44.712618462s) [noise=true , address=45.91.116.14:4190 , piece_id=HHU5YZBTJEEWYVST6XQCKGWKRLRQIIR2UQTQ27RLK53BNAVOSHFQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.974660841 (606h29m44.845743886s) [noise=true , address=45.91.116.12:4225 , piece_id=LLYEKE4TJ54KECLAC7KFUV44IVHYCYLF45I43G5V3DPV5QDBUOIQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.974681401 (606h29m44.845764436s) [seg_pos={0 0}, upload=531550]: Segment reached optimal threshold of 13 pieces. Dec 30 21:58:44.974855342 (606h29m44.845938377s) [noise=true , address=145.79.227.49:4339 , piece_id=WBBPZK2VY2QVRTQNXEKQ7TRYPWBDYS2TGTKEDAAXQ2AT4TUQXTQQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 30 21:58:44.974861423 (606h29m44.845944458s) [noise=true , address=145.79.227.49:4339 , piece_id=WBBPZK2VY2QVRTQNXEKQ7TRYPWBDYS2TGTKEDAAXQ2AT4TUQXTQQ, seg_pos={0 0}, upload=531550]: Long tail cancellation due to optimal threshold Dec 30 21:58:44.974968166 (606h29m44.846051201s) [noise=true , address=145.79.227.40:4050 , piece_id=LA7KLHHWRNHULLPAZWQ6MLS76SLN3BONYXZJXCUHGFC5JZ4SQRIA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 30 21:58:44.974973627 (606h29m44.846056662s) [noise=true , address=145.79.227.40:4050 , piece_id=LA7KLHHWRNHULLPAZWQ6MLS76SLN3BONYXZJXCUHGFC5JZ4SQRIA, seg_pos={0 0}, upload=531550]: Long tail cancellation due to optimal threshold Dec 30 21:58:44.975076152 (606h29m44.846159187s) [noise=true , address=45.91.116.11:4120 , piece_id=FZ5L6FM7IHX65EVFGVXXHEF43XQ4EBUH4C7VZUSTH2MVWLNR5M2Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 30 21:58:44.975083497 (606h29m44.846166532s) [noise=true , address=45.91.116.11:4120 , piece_id=FZ5L6FM7IHX65EVFGVXXHEF43XQ4EBUH4C7VZUSTH2MVWLNR5M2Q, seg_pos={0 0}, upload=531550]: Long tail cancellation due to optimal threshold Dec 30 21:58:44.975096711 (606h29m44.846179747s) [seg_pos={0 0}, upload=531550]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 21:58:44.975160703 (606h29m44.846243739s) [upload=531550]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 21:58:44.975164691 (606h29m44.846247726s) [upload=531550]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 21:58:45.272782726 (606h29m45.143865772s) [upload=531550]: Done uploading segments... Dec 30 21:58:45.272806352 (606h29m45.143889387s) [upload=531550]: Upload finished. err: <nil> Duration: 5.881100911s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.XX......X.........X........................................] duration: 15.395592812s trace: https://jaeger.dev.storj.io/trace/5cdd5ba7b1cb22c7 FAIL (2025-12-30 22:10:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 21:37:03.355296401 (606h8m3.226379446s) [noise=true , address=145.79.227.22:4543 , piece_id=WWCZ6SEEXQ2KUJ32AGTIJNMSJROLYSJ4XQGBQJC4Q5DACZVA65RQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.355839374 (606h8m3.22692241s) [noise=true , address=145.79.227.35:4492 , piece_id=HCKFSQMJYEP6B5H4OX5AZ5TWP6Z4H4Z6WA2T7Z2H5KNZ4SDJRMUQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.356356930 (606h8m3.227439965s) [noise=true , address=45.91.116.13:4182 , piece_id=BYAYNP4JAAN3AW2LYSK74WJE4BRWUR5Z6K67MTLK6A73KEJ35WEQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.356675316 (606h8m3.227758361s) [noise=true , address=77.90.31.12:4612 , piece_id=63TS3SUEFUE6PORZ4W76OXRWFL7SBN773UGGY3DU7JPBAW66N7UA, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.356992732 (606h8m3.228075767s) [noise=true , address=64.49.13.140:4507 , piece_id=E7T6WL45F6GHECCRANX6AIKWFRZ77TURRWORYEYI43MET2WG7SRQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.357330644 (606h8m3.228413679s) [noise=true , address=77.90.31.21:4886 , piece_id=ZQRCPTVC3WFVQPZ36AU6KJW7PD3T557CXYQIICLDGOKK6JCPVZ5Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.357663649 (606h8m3.228746684s) [noise=true , address=145.79.227.49:4367 , piece_id=R2T2ABLXBPDVP3ZVA2TAYMCLQQ7JQUVQYCTHCKGMFBRVWCDAO4PQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.359827449 (606h8m3.230910484s) [noise=true , address=145.79.227.34:4918 , piece_id=HUMOHOT3HOYCBJ23WXQXWU3X6C6JQTRLLHTD4RQLCQTCB3HXI7EQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.360086692 (606h8m3.231169727s) [noise=true , address=145.79.227.7:4320 , piece_id=MAICNTHGLZ6W27HXDOOZXOBRGQGEXNSA4BJT2T5D5PHRVV6ZMX2Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.361622167 (606h8m3.232705212s) [noise=true , address=145.79.227.33:4888 , piece_id=I4426P3VETHPOL5WHWJEGVYRXJULTCNDX3WSUTS3XZVY36Q2N44Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.362965514 (606h8m3.234048559s) [noise=true , address=45.91.116.11:4133 , piece_id=NEW7ZOVP5E7HYGYD4LG5UG2MDLAQW2OBRTNZDFIYO4CQYMVAWRPQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.364308883 (606h8m3.235391928s) [noise=true , address=145.79.227.26:4663 , piece_id=DYEFQZL4X74QR2LAYURT5ZTR2IWIDB55VLBQOGZM36MA54P2SWJA, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.365734507 (606h8m3.236817542s) [noise=true , address=145.79.227.40:4059 , piece_id=7WSUBR3C3PWVBCWD6KRGSD6MPX4IH2YQOAVO4J4SWZORNHGS2IPQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.366982764 (606h8m3.238065799s) [noise=true , address=145.79.227.29:4973 , piece_id=5OROXELR55A47BFPSJJEWVQU4YW6CAOK6LMEHCINM4FN3OVI2Y6Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.368390294 (606h8m3.239473339s) [noise=true , address=77.90.31.19:4852 , piece_id=2NYO63GCBGCYBFYTMUGNF2NG5FSJOD3AOUOD2FZZIWDINK4X6DJQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:04.736295138 (606h8m4.607378183s) [upload=531240]: Next returned nil segment Dec 30 21:37:04.736311850 (606h8m4.607394885s) [upload=531240]: Waiting for error group managing segments... Dec 30 21:37:05.104989591 (606h8m4.976072636s) [noise=true , address=64.49.13.140:4507 , piece_id=E7T6WL45F6GHECCRANX6AIKWFRZ77TURRWORYEYI43MET2WG7SRQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.208809748 (606h8m5.079892793s) [noise=true , address=145.79.227.22:4543 , piece_id=WWCZ6SEEXQ2KUJ32AGTIJNMSJROLYSJ4XQGBQJC4Q5DACZVA65RQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.441125389 (606h8m5.312208444s) [noise=true , address=77.90.31.19:4852 , piece_id=2NYO63GCBGCYBFYTMUGNF2NG5FSJOD3AOUOD2FZZIWDINK4X6DJQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.483630904 (606h8m5.35471394s) [noise=true , address=77.90.31.12:4612 , piece_id=63TS3SUEFUE6PORZ4W76OXRWFL7SBN773UGGY3DU7JPBAW66N7UA, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.494478811 (606h8m5.365561846s) [noise=true , address=77.90.31.11:4559 , piece_id=3DYD4CNPJQN3S4EYZJGREQTTZJDSL2QTC3OT6G2VFUGMLSVCZUFA, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.497452683 (606h8m5.368535729s) [noise=true , address=77.90.31.17:4769 , piece_id=QXDYESBKGXRY7BCFYYF2O7OXS7Y4Z4KUMJB6AXZAWX735SYU56GQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.661529377 (606h8m5.532612412s) [noise=true , address=45.91.116.13:4182 , piece_id=BYAYNP4JAAN3AW2LYSK74WJE4BRWUR5Z6K67MTLK6A73KEJ35WEQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.729773112 (606h8m5.600856157s) [noise=true , address=45.91.116.11:4133 , piece_id=NEW7ZOVP5E7HYGYD4LG5UG2MDLAQW2OBRTNZDFIYO4CQYMVAWRPQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.940743308 (606h8m5.811826353s) [noise=true , address=77.90.31.21:4886 , piece_id=ZQRCPTVC3WFVQPZ36AU6KJW7PD3T557CXYQIICLDGOKK6JCPVZ5Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.084240044 (606h8m5.95532309s) [noise=true , address=145.79.227.30:4784 , piece_id=DSXUDNYCWO5F6IYXMIVJNEASYXIZ7EO7RKAS7QRUYK5OU7GOOB5Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.121095032 (606h8m5.992178067s) [noise=true , address=145.79.227.40:4059 , piece_id=7WSUBR3C3PWVBCWD6KRGSD6MPX4IH2YQOAVO4J4SWZORNHGS2IPQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.169195983 (606h8m6.040279028s) [noise=true , address=145.79.227.49:4367 , piece_id=R2T2ABLXBPDVP3ZVA2TAYMCLQQ7JQUVQYCTHCKGMFBRVWCDAO4PQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.169380634 (606h8m6.040463679s) [noise=true , address=145.79.227.33:4888 , piece_id=I4426P3VETHPOL5WHWJEGVYRXJULTCNDX3WSUTS3XZVY36Q2N44Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.179060175 (606h8m6.050143211s) [noise=true , address=145.79.227.29:4973 , piece_id=5OROXELR55A47BFPSJJEWVQU4YW6CAOK6LMEHCINM4FN3OVI2Y6Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.179080424 (606h8m6.050163459s) [seg_pos={0 0}, upload=531240]: Segment reached optimal threshold of 14 pieces. Dec 30 21:37:06.179262801 (606h8m6.050345837s) [noise=true , address=145.79.227.26:4663 , piece_id=DYEFQZL4X74QR2LAYURT5ZTR2IWIDB55VLBQOGZM36MA54P2SWJA, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SwpAaXTA3BqUvwubzmJkvwokdz6SSpT9t8HPtQcXXJxmcgfm2): protocol: write tcp 188.166.225.170:43184->145.79.227.26:4663: use of closed network connection; write tcp 188.166.225.170:43184->145.79.227.26:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43184->145.79.227.26:4663: use of closed network connection Dec 30 21:37:06.179268703 (606h8m6.050351748s) [noise=true , address=145.79.227.26:4663 , piece_id=DYEFQZL4X74QR2LAYURT5ZTR2IWIDB55VLBQOGZM36MA54P2SWJA, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179363724 (606h8m6.050446769s) [noise=true , address=145.79.227.7:4320 , piece_id=MAICNTHGLZ6W27HXDOOZXOBRGQGEXNSA4BJT2T5D5PHRVV6ZMX2Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126rEgnQw1w7hsFpRqQP9nLaumdBfdtkgYjdw6WE6nXrxk5kj4B): protocol: write tcp 188.166.225.170:45228->145.79.227.7:4320: use of closed network connection; write tcp 188.166.225.170:45228->145.79.227.7:4320: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45228->145.79.227.7:4320: use of closed network connection Dec 30 21:37:06.179369525 (606h8m6.05045257s) [noise=true , address=145.79.227.7:4320 , piece_id=MAICNTHGLZ6W27HXDOOZXOBRGQGEXNSA4BJT2T5D5PHRVV6ZMX2Q, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179532996 (606h8m6.050616061s) [noise=true , address=145.79.227.34:4918 , piece_id=HUMOHOT3HOYCBJ23WXQXWU3X6C6JQTRLLHTD4RQLCQTCB3HXI7EQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF Dec 30 21:37:06.179572371 (606h8m6.050655406s) [noise=true , address=145.79.227.34:4918 , piece_id=HUMOHOT3HOYCBJ23WXQXWU3X6C6JQTRLLHTD4RQLCQTCB3HXI7EQ, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179638587 (606h8m6.050721622s) [noise=true , address=145.79.227.35:4492 , piece_id=HCKFSQMJYEP6B5H4OX5AZ5TWP6Z4H4Z6WA2T7Z2H5KNZ4SDJRMUQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1roa5GGYmqCX2AxSrqFgqyV2CtBEYZavuGTseiwDy4pjkT6bip): protocol: expected piece hash; context canceled; EOF Dec 30 21:37:06.179647855 (606h8m6.05073089s) [noise=true , address=145.79.227.35:4492 , piece_id=HCKFSQMJYEP6B5H4OX5AZ5TWP6Z4H4Z6WA2T7Z2H5KNZ4SDJRMUQ, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179666641 (606h8m6.050749686s) [seg_pos={0 0}, upload=531240]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 21:37:06.179742766 (606h8m6.050825811s) [upload=531240]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 21:37:06.179747545 (606h8m6.05083059s) [upload=531240]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 21:37:06.490986034 (606h8m6.362069079s) [upload=531240]: Done uploading segments... Dec 30 21:37:06.491006261 (606h8m6.362089307s) [upload=531240]: Upload finished. err: <nil> Duration: 15.395621657s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [X..X...X.........X.........................X................] duration: 6.258157668s trace: https://jaeger.dev.storj.io/trace/57a7ab98e7eb0d3f FAIL (2025-12-30 22:09:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 21:58:42.298454527 (606h29m42.169537592s) [upload=531550]: Starting upload Dec 30 21:58:42.298523438 (606h29m42.169606473s) [upload=531550]: Done starting upload Dec 30 21:58:42.298568774 (606h29m42.169651809s) [upload=531550]: Uploading segments... Dec 30 21:58:42.298697850 (606h29m42.169780885s) [upload=531550]: Got next segment. Inline: false Dec 30 21:58:42.298728649 (606h29m42.169811694s) [upload=531550]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 21:58:42.298733227 (606h29m42.169816262s) [upload=531550]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 21:58:42.545734213 (606h29m42.416817288s) [seg_pos={0 0}, upload=531550]: Begin upload segment... Dec 30 21:58:42.545786032 (606h29m42.416869078s) [seg_pos={0 0}, upload=531550]: Done begin upload segment. Dec 30 21:58:42.545855104 (606h29m42.416938149s) [noise=true , address=145.79.227.23:4547 , piece_id=LYGUMTAP7OPJWGRVFY65GUZFE4BBTV73JTG657DR5OM26LA7UFZA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.545897384 (606h29m42.416980449s) [noise=true , address=145.79.227.49:4339 , piece_id=WBBPZK2VY2QVRTQNXEKQ7TRYPWBDYS2TGTKEDAAXQ2AT4TUQXTQQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.546757752 (606h29m42.417840817s) [noise=true , address=45.91.116.12:4225 , piece_id=LLYEKE4TJ54KECLAC7KFUV44IVHYCYLF45I43G5V3DPV5QDBUOIQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.546816995 (606h29m42.41790004s) [noise=true , address=145.79.227.40:4050 , piece_id=LA7KLHHWRNHULLPAZWQ6MLS76SLN3BONYXZJXCUHGFC5JZ4SQRIA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.547216927 (606h29m42.418299962s) [noise=true , address=145.79.227.15:4876 , piece_id=L2OIJAB6OOANT2EPFNVNLTMYCH334JGFBFASD44EHOAWTAHKUV5Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.547615996 (606h29m42.418699042s) [noise=true , address=145.79.227.17:4971 , piece_id=DYZUALBL3V75TKGK57LOJABWVCOD5UVS3WH74I55OVUDGRQUA3QQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.547841055 (606h29m42.4189241s) [noise=true , address=77.90.31.21:4877 , piece_id=BD2FEFNAGYWZYDA2IMOVHA7JE66EXSF3A3PL54Q2DUOJFG5ALG4Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548023322 (606h29m42.419106357s) [noise=true , address=45.91.116.14:4190 , piece_id=HHU5YZBTJEEWYVST6XQCKGWKRLRQIIR2UQTQ27RLK53BNAVOSHFQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548380923 (606h29m42.419463968s) [noise=true , address=45.91.116.11:4120 , piece_id=FZ5L6FM7IHX65EVFGVXXHEF43XQ4EBUH4C7VZUSTH2MVWLNR5M2Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548560475 (606h29m42.41964351s) [noise=true , address=145.79.227.39:4013 , piece_id=I4XP47SFJICIYIPEN4E7UC2HXRXH4IXVJYGHXTZBCUESAHNFQXJA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548719268 (606h29m42.419802313s) [noise=true , address=145.79.227.31:4797 , piece_id=S5YBLITONGU6RHPQENP2TCVVWOHEMXPMYXLNYWBZNXWMFI26PVUA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549024107 (606h29m42.420107152s) [noise=true , address=145.79.227.7:4294 , piece_id=C6N5OIRLV3LLSCEIOGQJKWPKENGZYNAPY6V2AYZNKUMACBP5SLWQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549197708 (606h29m42.420280753s) [noise=true , address=145.79.227.16:4910 , piece_id=O4YO367LEVZSRI5E75YEZ7W7YI2KGKKQDXCOHQ3LALE4WUAPQVHQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549506406 (606h29m42.420589441s) [noise=true , address=145.79.227.12:4765 , piece_id=4ZF5YQP4Z5HBTX5YWUDHZV55Q47WASRLMV6ZYJXJRL2QBAVQQM4Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549971591 (606h29m42.421054626s) [noise=true , address=145.79.227.32:4835 , piece_id=CN4NMCFNYKOCG2JCIEXJBL2PLEYWD5THCZB4WQDURIECFVHLLX6Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.551571588 (606h29m42.422654623s) [noise=true , address=145.79.227.24:4581 , piece_id=Y6GBEOR7DQX2H6LBHXYFOXRBX3GP7AKPYVGTQNMUEXKPXLONUUGQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:44.247717212 (606h29m44.118800248s) [upload=531550]: Next returned nil segment Dec 30 21:58:44.247730348 (606h29m44.118813383s) [upload=531550]: Waiting for error group managing segments... Dec 30 21:58:44.532148720 (606h29m44.403231755s) [noise=true , address=145.79.227.31:4797 , piece_id=S5YBLITONGU6RHPQENP2TCVVWOHEMXPMYXLNYWBZNXWMFI26PVUA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.534655293 (606h29m44.405738328s) [noise=true , address=145.79.227.39:4013 , piece_id=I4XP47SFJICIYIPEN4E7UC2HXRXH4IXVJYGHXTZBCUESAHNFQXJA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.544709427 (606h29m44.415792462s) [noise=true , address=145.79.227.12:4765 , piece_id=4ZF5YQP4Z5HBTX5YWUDHZV55Q47WASRLMV6ZYJXJRL2QBAVQQM4Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.545822306 (606h29m44.416905342s) [noise=true , address=145.79.227.7:4294 , piece_id=C6N5OIRLV3LLSCEIOGQJKWPKENGZYNAPY6V2AYZNKUMACBP5SLWQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.549465162 (606h29m44.420548198s) [noise=true , address=145.79.227.24:4581 , piece_id=Y6GBEOR7DQX2H6LBHXYFOXRBX3GP7AKPYVGTQNMUEXKPXLONUUGQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.556935928 (606h29m44.428018964s) [noise=true , address=145.79.227.15:4876 , piece_id=L2OIJAB6OOANT2EPFNVNLTMYCH334JGFBFASD44EHOAWTAHKUV5Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.561283296 (606h29m44.432366331s) [noise=true , address=145.79.227.16:4910 , piece_id=O4YO367LEVZSRI5E75YEZ7W7YI2KGKKQDXCOHQ3LALE4WUAPQVHQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.614194264 (606h29m44.485277309s) [noise=true , address=145.79.227.17:4971 , piece_id=DYZUALBL3V75TKGK57LOJABWVCOD5UVS3WH74I55OVUDGRQUA3QQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.615630689 (606h29m44.486713724s) [noise=true , address=145.79.227.32:4835 , piece_id=CN4NMCFNYKOCG2JCIEXJBL2PLEYWD5THCZB4WQDURIECFVHLLX6Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.639672077 (606h29m44.510755122s) [noise=true , address=145.79.227.23:4547 , piece_id=LYGUMTAP7OPJWGRVFY65GUZFE4BBTV73JTG657DR5OM26LA7UFZA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.699618771 (606h29m44.570701817s) [noise=true , address=77.90.31.21:4877 , piece_id=BD2FEFNAGYWZYDA2IMOVHA7JE66EXSF3A3PL54Q2DUOJFG5ALG4Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.841535416 (606h29m44.712618462s) [noise=true , address=45.91.116.14:4190 , piece_id=HHU5YZBTJEEWYVST6XQCKGWKRLRQIIR2UQTQ27RLK53BNAVOSHFQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.974660841 (606h29m44.845743886s) [noise=true , address=45.91.116.12:4225 , piece_id=LLYEKE4TJ54KECLAC7KFUV44IVHYCYLF45I43G5V3DPV5QDBUOIQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.974681401 (606h29m44.845764436s) [seg_pos={0 0}, upload=531550]: Segment reached optimal threshold of 13 pieces. Dec 30 21:58:44.974855342 (606h29m44.845938377s) [noise=true , address=145.79.227.49:4339 , piece_id=WBBPZK2VY2QVRTQNXEKQ7TRYPWBDYS2TGTKEDAAXQ2AT4TUQXTQQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 30 21:58:44.974861423 (606h29m44.845944458s) [noise=true , address=145.79.227.49:4339 , piece_id=WBBPZK2VY2QVRTQNXEKQ7TRYPWBDYS2TGTKEDAAXQ2AT4TUQXTQQ, seg_pos={0 0}, upload=531550]: Long tail cancellation due to optimal threshold Dec 30 21:58:44.974968166 (606h29m44.846051201s) [noise=true , address=145.79.227.40:4050 , piece_id=LA7KLHHWRNHULLPAZWQ6MLS76SLN3BONYXZJXCUHGFC5JZ4SQRIA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 30 21:58:44.974973627 (606h29m44.846056662s) [noise=true , address=145.79.227.40:4050 , piece_id=LA7KLHHWRNHULLPAZWQ6MLS76SLN3BONYXZJXCUHGFC5JZ4SQRIA, seg_pos={0 0}, upload=531550]: Long tail cancellation due to optimal threshold Dec 30 21:58:44.975076152 (606h29m44.846159187s) [noise=true , address=45.91.116.11:4120 , piece_id=FZ5L6FM7IHX65EVFGVXXHEF43XQ4EBUH4C7VZUSTH2MVWLNR5M2Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 30 21:58:44.975083497 (606h29m44.846166532s) [noise=true , address=45.91.116.11:4120 , piece_id=FZ5L6FM7IHX65EVFGVXXHEF43XQ4EBUH4C7VZUSTH2MVWLNR5M2Q, seg_pos={0 0}, upload=531550]: Long tail cancellation due to optimal threshold Dec 30 21:58:44.975096711 (606h29m44.846179747s) [seg_pos={0 0}, upload=531550]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 21:58:44.975160703 (606h29m44.846243739s) [upload=531550]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 21:58:44.975164691 (606h29m44.846247726s) [upload=531550]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 21:58:45.272782726 (606h29m45.143865772s) [upload=531550]: Done uploading segments... Dec 30 21:58:45.272806352 (606h29m45.143889387s) [upload=531550]: Upload finished. err: <nil> Duration: 6.258186061s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................................X.....................] duration: 3.76415994s trace: https://jaeger.dev.storj.io/trace/22a47ec27f9a101c FAIL (2025-12-30 22:09:39 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.764188034s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [..XX......X.........X.......................................] duration: 15.112797661s trace: https://jaeger.dev.storj.io/trace/5cdd5ba7b1cb22c7 FAIL (2025-12-30 22:09:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 21:37:03.355296401 (606h8m3.226379446s) [noise=true , address=145.79.227.22:4543 , piece_id=WWCZ6SEEXQ2KUJ32AGTIJNMSJROLYSJ4XQGBQJC4Q5DACZVA65RQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.355839374 (606h8m3.22692241s) [noise=true , address=145.79.227.35:4492 , piece_id=HCKFSQMJYEP6B5H4OX5AZ5TWP6Z4H4Z6WA2T7Z2H5KNZ4SDJRMUQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.356356930 (606h8m3.227439965s) [noise=true , address=45.91.116.13:4182 , piece_id=BYAYNP4JAAN3AW2LYSK74WJE4BRWUR5Z6K67MTLK6A73KEJ35WEQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.356675316 (606h8m3.227758361s) [noise=true , address=77.90.31.12:4612 , piece_id=63TS3SUEFUE6PORZ4W76OXRWFL7SBN773UGGY3DU7JPBAW66N7UA, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.356992732 (606h8m3.228075767s) [noise=true , address=64.49.13.140:4507 , piece_id=E7T6WL45F6GHECCRANX6AIKWFRZ77TURRWORYEYI43MET2WG7SRQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.357330644 (606h8m3.228413679s) [noise=true , address=77.90.31.21:4886 , piece_id=ZQRCPTVC3WFVQPZ36AU6KJW7PD3T557CXYQIICLDGOKK6JCPVZ5Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.357663649 (606h8m3.228746684s) [noise=true , address=145.79.227.49:4367 , piece_id=R2T2ABLXBPDVP3ZVA2TAYMCLQQ7JQUVQYCTHCKGMFBRVWCDAO4PQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.359827449 (606h8m3.230910484s) [noise=true , address=145.79.227.34:4918 , piece_id=HUMOHOT3HOYCBJ23WXQXWU3X6C6JQTRLLHTD4RQLCQTCB3HXI7EQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.360086692 (606h8m3.231169727s) [noise=true , address=145.79.227.7:4320 , piece_id=MAICNTHGLZ6W27HXDOOZXOBRGQGEXNSA4BJT2T5D5PHRVV6ZMX2Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.361622167 (606h8m3.232705212s) [noise=true , address=145.79.227.33:4888 , piece_id=I4426P3VETHPOL5WHWJEGVYRXJULTCNDX3WSUTS3XZVY36Q2N44Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.362965514 (606h8m3.234048559s) [noise=true , address=45.91.116.11:4133 , piece_id=NEW7ZOVP5E7HYGYD4LG5UG2MDLAQW2OBRTNZDFIYO4CQYMVAWRPQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.364308883 (606h8m3.235391928s) [noise=true , address=145.79.227.26:4663 , piece_id=DYEFQZL4X74QR2LAYURT5ZTR2IWIDB55VLBQOGZM36MA54P2SWJA, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.365734507 (606h8m3.236817542s) [noise=true , address=145.79.227.40:4059 , piece_id=7WSUBR3C3PWVBCWD6KRGSD6MPX4IH2YQOAVO4J4SWZORNHGS2IPQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.366982764 (606h8m3.238065799s) [noise=true , address=145.79.227.29:4973 , piece_id=5OROXELR55A47BFPSJJEWVQU4YW6CAOK6LMEHCINM4FN3OVI2Y6Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.368390294 (606h8m3.239473339s) [noise=true , address=77.90.31.19:4852 , piece_id=2NYO63GCBGCYBFYTMUGNF2NG5FSJOD3AOUOD2FZZIWDINK4X6DJQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:04.736295138 (606h8m4.607378183s) [upload=531240]: Next returned nil segment Dec 30 21:37:04.736311850 (606h8m4.607394885s) [upload=531240]: Waiting for error group managing segments... Dec 30 21:37:05.104989591 (606h8m4.976072636s) [noise=true , address=64.49.13.140:4507 , piece_id=E7T6WL45F6GHECCRANX6AIKWFRZ77TURRWORYEYI43MET2WG7SRQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.208809748 (606h8m5.079892793s) [noise=true , address=145.79.227.22:4543 , piece_id=WWCZ6SEEXQ2KUJ32AGTIJNMSJROLYSJ4XQGBQJC4Q5DACZVA65RQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.441125389 (606h8m5.312208444s) [noise=true , address=77.90.31.19:4852 , piece_id=2NYO63GCBGCYBFYTMUGNF2NG5FSJOD3AOUOD2FZZIWDINK4X6DJQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.483630904 (606h8m5.35471394s) [noise=true , address=77.90.31.12:4612 , piece_id=63TS3SUEFUE6PORZ4W76OXRWFL7SBN773UGGY3DU7JPBAW66N7UA, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.494478811 (606h8m5.365561846s) [noise=true , address=77.90.31.11:4559 , piece_id=3DYD4CNPJQN3S4EYZJGREQTTZJDSL2QTC3OT6G2VFUGMLSVCZUFA, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.497452683 (606h8m5.368535729s) [noise=true , address=77.90.31.17:4769 , piece_id=QXDYESBKGXRY7BCFYYF2O7OXS7Y4Z4KUMJB6AXZAWX735SYU56GQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.661529377 (606h8m5.532612412s) [noise=true , address=45.91.116.13:4182 , piece_id=BYAYNP4JAAN3AW2LYSK74WJE4BRWUR5Z6K67MTLK6A73KEJ35WEQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.729773112 (606h8m5.600856157s) [noise=true , address=45.91.116.11:4133 , piece_id=NEW7ZOVP5E7HYGYD4LG5UG2MDLAQW2OBRTNZDFIYO4CQYMVAWRPQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.940743308 (606h8m5.811826353s) [noise=true , address=77.90.31.21:4886 , piece_id=ZQRCPTVC3WFVQPZ36AU6KJW7PD3T557CXYQIICLDGOKK6JCPVZ5Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.084240044 (606h8m5.95532309s) [noise=true , address=145.79.227.30:4784 , piece_id=DSXUDNYCWO5F6IYXMIVJNEASYXIZ7EO7RKAS7QRUYK5OU7GOOB5Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.121095032 (606h8m5.992178067s) [noise=true , address=145.79.227.40:4059 , piece_id=7WSUBR3C3PWVBCWD6KRGSD6MPX4IH2YQOAVO4J4SWZORNHGS2IPQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.169195983 (606h8m6.040279028s) [noise=true , address=145.79.227.49:4367 , piece_id=R2T2ABLXBPDVP3ZVA2TAYMCLQQ7JQUVQYCTHCKGMFBRVWCDAO4PQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.169380634 (606h8m6.040463679s) [noise=true , address=145.79.227.33:4888 , piece_id=I4426P3VETHPOL5WHWJEGVYRXJULTCNDX3WSUTS3XZVY36Q2N44Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.179060175 (606h8m6.050143211s) [noise=true , address=145.79.227.29:4973 , piece_id=5OROXELR55A47BFPSJJEWVQU4YW6CAOK6LMEHCINM4FN3OVI2Y6Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.179080424 (606h8m6.050163459s) [seg_pos={0 0}, upload=531240]: Segment reached optimal threshold of 14 pieces. Dec 30 21:37:06.179262801 (606h8m6.050345837s) [noise=true , address=145.79.227.26:4663 , piece_id=DYEFQZL4X74QR2LAYURT5ZTR2IWIDB55VLBQOGZM36MA54P2SWJA, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SwpAaXTA3BqUvwubzmJkvwokdz6SSpT9t8HPtQcXXJxmcgfm2): protocol: write tcp 188.166.225.170:43184->145.79.227.26:4663: use of closed network connection; write tcp 188.166.225.170:43184->145.79.227.26:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43184->145.79.227.26:4663: use of closed network connection Dec 30 21:37:06.179268703 (606h8m6.050351748s) [noise=true , address=145.79.227.26:4663 , piece_id=DYEFQZL4X74QR2LAYURT5ZTR2IWIDB55VLBQOGZM36MA54P2SWJA, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179363724 (606h8m6.050446769s) [noise=true , address=145.79.227.7:4320 , piece_id=MAICNTHGLZ6W27HXDOOZXOBRGQGEXNSA4BJT2T5D5PHRVV6ZMX2Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126rEgnQw1w7hsFpRqQP9nLaumdBfdtkgYjdw6WE6nXrxk5kj4B): protocol: write tcp 188.166.225.170:45228->145.79.227.7:4320: use of closed network connection; write tcp 188.166.225.170:45228->145.79.227.7:4320: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45228->145.79.227.7:4320: use of closed network connection Dec 30 21:37:06.179369525 (606h8m6.05045257s) [noise=true , address=145.79.227.7:4320 , piece_id=MAICNTHGLZ6W27HXDOOZXOBRGQGEXNSA4BJT2T5D5PHRVV6ZMX2Q, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179532996 (606h8m6.050616061s) [noise=true , address=145.79.227.34:4918 , piece_id=HUMOHOT3HOYCBJ23WXQXWU3X6C6JQTRLLHTD4RQLCQTCB3HXI7EQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF Dec 30 21:37:06.179572371 (606h8m6.050655406s) [noise=true , address=145.79.227.34:4918 , piece_id=HUMOHOT3HOYCBJ23WXQXWU3X6C6JQTRLLHTD4RQLCQTCB3HXI7EQ, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179638587 (606h8m6.050721622s) [noise=true , address=145.79.227.35:4492 , piece_id=HCKFSQMJYEP6B5H4OX5AZ5TWP6Z4H4Z6WA2T7Z2H5KNZ4SDJRMUQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1roa5GGYmqCX2AxSrqFgqyV2CtBEYZavuGTseiwDy4pjkT6bip): protocol: expected piece hash; context canceled; EOF Dec 30 21:37:06.179647855 (606h8m6.05073089s) [noise=true , address=145.79.227.35:4492 , piece_id=HCKFSQMJYEP6B5H4OX5AZ5TWP6Z4H4Z6WA2T7Z2H5KNZ4SDJRMUQ, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179666641 (606h8m6.050749686s) [seg_pos={0 0}, upload=531240]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 21:37:06.179742766 (606h8m6.050825811s) [upload=531240]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 21:37:06.179747545 (606h8m6.05083059s) [upload=531240]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 21:37:06.490986034 (606h8m6.362069079s) [upload=531240]: Done uploading segments... Dec 30 21:37:06.491006261 (606h8m6.362089307s) [upload=531240]: Upload finished. err: <nil> Duration: 15.112839171s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.X..X...X.........X.........................X...............] duration: 7.374843246s trace: https://jaeger.dev.storj.io/trace/57a7ab98e7eb0d3f FAIL (2025-12-30 22:09:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 21:58:42.298454527 (606h29m42.169537592s) [upload=531550]: Starting upload Dec 30 21:58:42.298523438 (606h29m42.169606473s) [upload=531550]: Done starting upload Dec 30 21:58:42.298568774 (606h29m42.169651809s) [upload=531550]: Uploading segments... Dec 30 21:58:42.298697850 (606h29m42.169780885s) [upload=531550]: Got next segment. Inline: false Dec 30 21:58:42.298728649 (606h29m42.169811694s) [upload=531550]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 21:58:42.298733227 (606h29m42.169816262s) [upload=531550]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 21:58:42.545734213 (606h29m42.416817288s) [seg_pos={0 0}, upload=531550]: Begin upload segment... Dec 30 21:58:42.545786032 (606h29m42.416869078s) [seg_pos={0 0}, upload=531550]: Done begin upload segment. Dec 30 21:58:42.545855104 (606h29m42.416938149s) [noise=true , address=145.79.227.23:4547 , piece_id=LYGUMTAP7OPJWGRVFY65GUZFE4BBTV73JTG657DR5OM26LA7UFZA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.545897384 (606h29m42.416980449s) [noise=true , address=145.79.227.49:4339 , piece_id=WBBPZK2VY2QVRTQNXEKQ7TRYPWBDYS2TGTKEDAAXQ2AT4TUQXTQQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.546757752 (606h29m42.417840817s) [noise=true , address=45.91.116.12:4225 , piece_id=LLYEKE4TJ54KECLAC7KFUV44IVHYCYLF45I43G5V3DPV5QDBUOIQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.546816995 (606h29m42.41790004s) [noise=true , address=145.79.227.40:4050 , piece_id=LA7KLHHWRNHULLPAZWQ6MLS76SLN3BONYXZJXCUHGFC5JZ4SQRIA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.547216927 (606h29m42.418299962s) [noise=true , address=145.79.227.15:4876 , piece_id=L2OIJAB6OOANT2EPFNVNLTMYCH334JGFBFASD44EHOAWTAHKUV5Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.547615996 (606h29m42.418699042s) [noise=true , address=145.79.227.17:4971 , piece_id=DYZUALBL3V75TKGK57LOJABWVCOD5UVS3WH74I55OVUDGRQUA3QQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.547841055 (606h29m42.4189241s) [noise=true , address=77.90.31.21:4877 , piece_id=BD2FEFNAGYWZYDA2IMOVHA7JE66EXSF3A3PL54Q2DUOJFG5ALG4Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548023322 (606h29m42.419106357s) [noise=true , address=45.91.116.14:4190 , piece_id=HHU5YZBTJEEWYVST6XQCKGWKRLRQIIR2UQTQ27RLK53BNAVOSHFQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548380923 (606h29m42.419463968s) [noise=true , address=45.91.116.11:4120 , piece_id=FZ5L6FM7IHX65EVFGVXXHEF43XQ4EBUH4C7VZUSTH2MVWLNR5M2Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548560475 (606h29m42.41964351s) [noise=true , address=145.79.227.39:4013 , piece_id=I4XP47SFJICIYIPEN4E7UC2HXRXH4IXVJYGHXTZBCUESAHNFQXJA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548719268 (606h29m42.419802313s) [noise=true , address=145.79.227.31:4797 , piece_id=S5YBLITONGU6RHPQENP2TCVVWOHEMXPMYXLNYWBZNXWMFI26PVUA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549024107 (606h29m42.420107152s) [noise=true , address=145.79.227.7:4294 , piece_id=C6N5OIRLV3LLSCEIOGQJKWPKENGZYNAPY6V2AYZNKUMACBP5SLWQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549197708 (606h29m42.420280753s) [noise=true , address=145.79.227.16:4910 , piece_id=O4YO367LEVZSRI5E75YEZ7W7YI2KGKKQDXCOHQ3LALE4WUAPQVHQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549506406 (606h29m42.420589441s) [noise=true , address=145.79.227.12:4765 , piece_id=4ZF5YQP4Z5HBTX5YWUDHZV55Q47WASRLMV6ZYJXJRL2QBAVQQM4Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549971591 (606h29m42.421054626s) [noise=true , address=145.79.227.32:4835 , piece_id=CN4NMCFNYKOCG2JCIEXJBL2PLEYWD5THCZB4WQDURIECFVHLLX6Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.551571588 (606h29m42.422654623s) [noise=true , address=145.79.227.24:4581 , piece_id=Y6GBEOR7DQX2H6LBHXYFOXRBX3GP7AKPYVGTQNMUEXKPXLONUUGQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:44.247717212 (606h29m44.118800248s) [upload=531550]: Next returned nil segment Dec 30 21:58:44.247730348 (606h29m44.118813383s) [upload=531550]: Waiting for error group managing segments... Dec 30 21:58:44.532148720 (606h29m44.403231755s) [noise=true , address=145.79.227.31:4797 , piece_id=S5YBLITONGU6RHPQENP2TCVVWOHEMXPMYXLNYWBZNXWMFI26PVUA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.534655293 (606h29m44.405738328s) [noise=true , address=145.79.227.39:4013 , piece_id=I4XP47SFJICIYIPEN4E7UC2HXRXH4IXVJYGHXTZBCUESAHNFQXJA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.544709427 (606h29m44.415792462s) [noise=true , address=145.79.227.12:4765 , piece_id=4ZF5YQP4Z5HBTX5YWUDHZV55Q47WASRLMV6ZYJXJRL2QBAVQQM4Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.545822306 (606h29m44.416905342s) [noise=true , address=145.79.227.7:4294 , piece_id=C6N5OIRLV3LLSCEIOGQJKWPKENGZYNAPY6V2AYZNKUMACBP5SLWQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.549465162 (606h29m44.420548198s) [noise=true , address=145.79.227.24:4581 , piece_id=Y6GBEOR7DQX2H6LBHXYFOXRBX3GP7AKPYVGTQNMUEXKPXLONUUGQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.556935928 (606h29m44.428018964s) [noise=true , address=145.79.227.15:4876 , piece_id=L2OIJAB6OOANT2EPFNVNLTMYCH334JGFBFASD44EHOAWTAHKUV5Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.561283296 (606h29m44.432366331s) [noise=true , address=145.79.227.16:4910 , piece_id=O4YO367LEVZSRI5E75YEZ7W7YI2KGKKQDXCOHQ3LALE4WUAPQVHQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.614194264 (606h29m44.485277309s) [noise=true , address=145.79.227.17:4971 , piece_id=DYZUALBL3V75TKGK57LOJABWVCOD5UVS3WH74I55OVUDGRQUA3QQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.615630689 (606h29m44.486713724s) [noise=true , address=145.79.227.32:4835 , piece_id=CN4NMCFNYKOCG2JCIEXJBL2PLEYWD5THCZB4WQDURIECFVHLLX6Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.639672077 (606h29m44.510755122s) [noise=true , address=145.79.227.23:4547 , piece_id=LYGUMTAP7OPJWGRVFY65GUZFE4BBTV73JTG657DR5OM26LA7UFZA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.699618771 (606h29m44.570701817s) [noise=true , address=77.90.31.21:4877 , piece_id=BD2FEFNAGYWZYDA2IMOVHA7JE66EXSF3A3PL54Q2DUOJFG5ALG4Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.841535416 (606h29m44.712618462s) [noise=true , address=45.91.116.14:4190 , piece_id=HHU5YZBTJEEWYVST6XQCKGWKRLRQIIR2UQTQ27RLK53BNAVOSHFQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.974660841 (606h29m44.845743886s) [noise=true , address=45.91.116.12:4225 , piece_id=LLYEKE4TJ54KECLAC7KFUV44IVHYCYLF45I43G5V3DPV5QDBUOIQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.974681401 (606h29m44.845764436s) [seg_pos={0 0}, upload=531550]: Segment reached optimal threshold of 13 pieces. Dec 30 21:58:44.974855342 (606h29m44.845938377s) [noise=true , address=145.79.227.49:4339 , piece_id=WBBPZK2VY2QVRTQNXEKQ7TRYPWBDYS2TGTKEDAAXQ2AT4TUQXTQQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 30 21:58:44.974861423 (606h29m44.845944458s) [noise=true , address=145.79.227.49:4339 , piece_id=WBBPZK2VY2QVRTQNXEKQ7TRYPWBDYS2TGTKEDAAXQ2AT4TUQXTQQ, seg_pos={0 0}, upload=531550]: Long tail cancellation due to optimal threshold Dec 30 21:58:44.974968166 (606h29m44.846051201s) [noise=true , address=145.79.227.40:4050 , piece_id=LA7KLHHWRNHULLPAZWQ6MLS76SLN3BONYXZJXCUHGFC5JZ4SQRIA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 30 21:58:44.974973627 (606h29m44.846056662s) [noise=true , address=145.79.227.40:4050 , piece_id=LA7KLHHWRNHULLPAZWQ6MLS76SLN3BONYXZJXCUHGFC5JZ4SQRIA, seg_pos={0 0}, upload=531550]: Long tail cancellation due to optimal threshold Dec 30 21:58:44.975076152 (606h29m44.846159187s) [noise=true , address=45.91.116.11:4120 , piece_id=FZ5L6FM7IHX65EVFGVXXHEF43XQ4EBUH4C7VZUSTH2MVWLNR5M2Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 30 21:58:44.975083497 (606h29m44.846166532s) [noise=true , address=45.91.116.11:4120 , piece_id=FZ5L6FM7IHX65EVFGVXXHEF43XQ4EBUH4C7VZUSTH2MVWLNR5M2Q, seg_pos={0 0}, upload=531550]: Long tail cancellation due to optimal threshold Dec 30 21:58:44.975096711 (606h29m44.846179747s) [seg_pos={0 0}, upload=531550]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 21:58:44.975160703 (606h29m44.846243739s) [upload=531550]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 21:58:44.975164691 (606h29m44.846247726s) [upload=531550]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 21:58:45.272782726 (606h29m45.143865772s) [upload=531550]: Done uploading segments... Dec 30 21:58:45.272806352 (606h29m45.143889387s) [upload=531550]: Upload finished. err: <nil> Duration: 7.374886229s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [...XX......X.........X......................................] duration: 15.446325983s trace: https://jaeger.dev.storj.io/trace/5cdd5ba7b1cb22c7 FAIL (2025-12-30 22:08:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 21:37:03.355296401 (606h8m3.226379446s) [noise=true , address=145.79.227.22:4543 , piece_id=WWCZ6SEEXQ2KUJ32AGTIJNMSJROLYSJ4XQGBQJC4Q5DACZVA65RQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.355839374 (606h8m3.22692241s) [noise=true , address=145.79.227.35:4492 , piece_id=HCKFSQMJYEP6B5H4OX5AZ5TWP6Z4H4Z6WA2T7Z2H5KNZ4SDJRMUQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.356356930 (606h8m3.227439965s) [noise=true , address=45.91.116.13:4182 , piece_id=BYAYNP4JAAN3AW2LYSK74WJE4BRWUR5Z6K67MTLK6A73KEJ35WEQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.356675316 (606h8m3.227758361s) [noise=true , address=77.90.31.12:4612 , piece_id=63TS3SUEFUE6PORZ4W76OXRWFL7SBN773UGGY3DU7JPBAW66N7UA, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.356992732 (606h8m3.228075767s) [noise=true , address=64.49.13.140:4507 , piece_id=E7T6WL45F6GHECCRANX6AIKWFRZ77TURRWORYEYI43MET2WG7SRQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.357330644 (606h8m3.228413679s) [noise=true , address=77.90.31.21:4886 , piece_id=ZQRCPTVC3WFVQPZ36AU6KJW7PD3T557CXYQIICLDGOKK6JCPVZ5Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.357663649 (606h8m3.228746684s) [noise=true , address=145.79.227.49:4367 , piece_id=R2T2ABLXBPDVP3ZVA2TAYMCLQQ7JQUVQYCTHCKGMFBRVWCDAO4PQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.359827449 (606h8m3.230910484s) [noise=true , address=145.79.227.34:4918 , piece_id=HUMOHOT3HOYCBJ23WXQXWU3X6C6JQTRLLHTD4RQLCQTCB3HXI7EQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.360086692 (606h8m3.231169727s) [noise=true , address=145.79.227.7:4320 , piece_id=MAICNTHGLZ6W27HXDOOZXOBRGQGEXNSA4BJT2T5D5PHRVV6ZMX2Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.361622167 (606h8m3.232705212s) [noise=true , address=145.79.227.33:4888 , piece_id=I4426P3VETHPOL5WHWJEGVYRXJULTCNDX3WSUTS3XZVY36Q2N44Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.362965514 (606h8m3.234048559s) [noise=true , address=45.91.116.11:4133 , piece_id=NEW7ZOVP5E7HYGYD4LG5UG2MDLAQW2OBRTNZDFIYO4CQYMVAWRPQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.364308883 (606h8m3.235391928s) [noise=true , address=145.79.227.26:4663 , piece_id=DYEFQZL4X74QR2LAYURT5ZTR2IWIDB55VLBQOGZM36MA54P2SWJA, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.365734507 (606h8m3.236817542s) [noise=true , address=145.79.227.40:4059 , piece_id=7WSUBR3C3PWVBCWD6KRGSD6MPX4IH2YQOAVO4J4SWZORNHGS2IPQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.366982764 (606h8m3.238065799s) [noise=true , address=145.79.227.29:4973 , piece_id=5OROXELR55A47BFPSJJEWVQU4YW6CAOK6LMEHCINM4FN3OVI2Y6Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.368390294 (606h8m3.239473339s) [noise=true , address=77.90.31.19:4852 , piece_id=2NYO63GCBGCYBFYTMUGNF2NG5FSJOD3AOUOD2FZZIWDINK4X6DJQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:04.736295138 (606h8m4.607378183s) [upload=531240]: Next returned nil segment Dec 30 21:37:04.736311850 (606h8m4.607394885s) [upload=531240]: Waiting for error group managing segments... Dec 30 21:37:05.104989591 (606h8m4.976072636s) [noise=true , address=64.49.13.140:4507 , piece_id=E7T6WL45F6GHECCRANX6AIKWFRZ77TURRWORYEYI43MET2WG7SRQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.208809748 (606h8m5.079892793s) [noise=true , address=145.79.227.22:4543 , piece_id=WWCZ6SEEXQ2KUJ32AGTIJNMSJROLYSJ4XQGBQJC4Q5DACZVA65RQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.441125389 (606h8m5.312208444s) [noise=true , address=77.90.31.19:4852 , piece_id=2NYO63GCBGCYBFYTMUGNF2NG5FSJOD3AOUOD2FZZIWDINK4X6DJQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.483630904 (606h8m5.35471394s) [noise=true , address=77.90.31.12:4612 , piece_id=63TS3SUEFUE6PORZ4W76OXRWFL7SBN773UGGY3DU7JPBAW66N7UA, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.494478811 (606h8m5.365561846s) [noise=true , address=77.90.31.11:4559 , piece_id=3DYD4CNPJQN3S4EYZJGREQTTZJDSL2QTC3OT6G2VFUGMLSVCZUFA, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.497452683 (606h8m5.368535729s) [noise=true , address=77.90.31.17:4769 , piece_id=QXDYESBKGXRY7BCFYYF2O7OXS7Y4Z4KUMJB6AXZAWX735SYU56GQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.661529377 (606h8m5.532612412s) [noise=true , address=45.91.116.13:4182 , piece_id=BYAYNP4JAAN3AW2LYSK74WJE4BRWUR5Z6K67MTLK6A73KEJ35WEQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.729773112 (606h8m5.600856157s) [noise=true , address=45.91.116.11:4133 , piece_id=NEW7ZOVP5E7HYGYD4LG5UG2MDLAQW2OBRTNZDFIYO4CQYMVAWRPQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.940743308 (606h8m5.811826353s) [noise=true , address=77.90.31.21:4886 , piece_id=ZQRCPTVC3WFVQPZ36AU6KJW7PD3T557CXYQIICLDGOKK6JCPVZ5Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.084240044 (606h8m5.95532309s) [noise=true , address=145.79.227.30:4784 , piece_id=DSXUDNYCWO5F6IYXMIVJNEASYXIZ7EO7RKAS7QRUYK5OU7GOOB5Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.121095032 (606h8m5.992178067s) [noise=true , address=145.79.227.40:4059 , piece_id=7WSUBR3C3PWVBCWD6KRGSD6MPX4IH2YQOAVO4J4SWZORNHGS2IPQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.169195983 (606h8m6.040279028s) [noise=true , address=145.79.227.49:4367 , piece_id=R2T2ABLXBPDVP3ZVA2TAYMCLQQ7JQUVQYCTHCKGMFBRVWCDAO4PQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.169380634 (606h8m6.040463679s) [noise=true , address=145.79.227.33:4888 , piece_id=I4426P3VETHPOL5WHWJEGVYRXJULTCNDX3WSUTS3XZVY36Q2N44Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.179060175 (606h8m6.050143211s) [noise=true , address=145.79.227.29:4973 , piece_id=5OROXELR55A47BFPSJJEWVQU4YW6CAOK6LMEHCINM4FN3OVI2Y6Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.179080424 (606h8m6.050163459s) [seg_pos={0 0}, upload=531240]: Segment reached optimal threshold of 14 pieces. Dec 30 21:37:06.179262801 (606h8m6.050345837s) [noise=true , address=145.79.227.26:4663 , piece_id=DYEFQZL4X74QR2LAYURT5ZTR2IWIDB55VLBQOGZM36MA54P2SWJA, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SwpAaXTA3BqUvwubzmJkvwokdz6SSpT9t8HPtQcXXJxmcgfm2): protocol: write tcp 188.166.225.170:43184->145.79.227.26:4663: use of closed network connection; write tcp 188.166.225.170:43184->145.79.227.26:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43184->145.79.227.26:4663: use of closed network connection Dec 30 21:37:06.179268703 (606h8m6.050351748s) [noise=true , address=145.79.227.26:4663 , piece_id=DYEFQZL4X74QR2LAYURT5ZTR2IWIDB55VLBQOGZM36MA54P2SWJA, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179363724 (606h8m6.050446769s) [noise=true , address=145.79.227.7:4320 , piece_id=MAICNTHGLZ6W27HXDOOZXOBRGQGEXNSA4BJT2T5D5PHRVV6ZMX2Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126rEgnQw1w7hsFpRqQP9nLaumdBfdtkgYjdw6WE6nXrxk5kj4B): protocol: write tcp 188.166.225.170:45228->145.79.227.7:4320: use of closed network connection; write tcp 188.166.225.170:45228->145.79.227.7:4320: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45228->145.79.227.7:4320: use of closed network connection Dec 30 21:37:06.179369525 (606h8m6.05045257s) [noise=true , address=145.79.227.7:4320 , piece_id=MAICNTHGLZ6W27HXDOOZXOBRGQGEXNSA4BJT2T5D5PHRVV6ZMX2Q, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179532996 (606h8m6.050616061s) [noise=true , address=145.79.227.34:4918 , piece_id=HUMOHOT3HOYCBJ23WXQXWU3X6C6JQTRLLHTD4RQLCQTCB3HXI7EQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF Dec 30 21:37:06.179572371 (606h8m6.050655406s) [noise=true , address=145.79.227.34:4918 , piece_id=HUMOHOT3HOYCBJ23WXQXWU3X6C6JQTRLLHTD4RQLCQTCB3HXI7EQ, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179638587 (606h8m6.050721622s) [noise=true , address=145.79.227.35:4492 , piece_id=HCKFSQMJYEP6B5H4OX5AZ5TWP6Z4H4Z6WA2T7Z2H5KNZ4SDJRMUQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1roa5GGYmqCX2AxSrqFgqyV2CtBEYZavuGTseiwDy4pjkT6bip): protocol: expected piece hash; context canceled; EOF Dec 30 21:37:06.179647855 (606h8m6.05073089s) [noise=true , address=145.79.227.35:4492 , piece_id=HCKFSQMJYEP6B5H4OX5AZ5TWP6Z4H4Z6WA2T7Z2H5KNZ4SDJRMUQ, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179666641 (606h8m6.050749686s) [seg_pos={0 0}, upload=531240]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 21:37:06.179742766 (606h8m6.050825811s) [upload=531240]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 21:37:06.179747545 (606h8m6.05083059s) [upload=531240]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 21:37:06.490986034 (606h8m6.362069079s) [upload=531240]: Done uploading segments... Dec 30 21:37:06.491006261 (606h8m6.362089307s) [upload=531240]: Upload finished. err: <nil> Duration: 15.446342685s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..X..X...X.........X.........................X..............] duration: 6.126315858s trace: https://jaeger.dev.storj.io/trace/57a7ab98e7eb0d3f FAIL (2025-12-30 22:08:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 21:58:42.298454527 (606h29m42.169537592s) [upload=531550]: Starting upload Dec 30 21:58:42.298523438 (606h29m42.169606473s) [upload=531550]: Done starting upload Dec 30 21:58:42.298568774 (606h29m42.169651809s) [upload=531550]: Uploading segments... Dec 30 21:58:42.298697850 (606h29m42.169780885s) [upload=531550]: Got next segment. Inline: false Dec 30 21:58:42.298728649 (606h29m42.169811694s) [upload=531550]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 21:58:42.298733227 (606h29m42.169816262s) [upload=531550]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 21:58:42.545734213 (606h29m42.416817288s) [seg_pos={0 0}, upload=531550]: Begin upload segment... Dec 30 21:58:42.545786032 (606h29m42.416869078s) [seg_pos={0 0}, upload=531550]: Done begin upload segment. Dec 30 21:58:42.545855104 (606h29m42.416938149s) [noise=true , address=145.79.227.23:4547 , piece_id=LYGUMTAP7OPJWGRVFY65GUZFE4BBTV73JTG657DR5OM26LA7UFZA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.545897384 (606h29m42.416980449s) [noise=true , address=145.79.227.49:4339 , piece_id=WBBPZK2VY2QVRTQNXEKQ7TRYPWBDYS2TGTKEDAAXQ2AT4TUQXTQQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.546757752 (606h29m42.417840817s) [noise=true , address=45.91.116.12:4225 , piece_id=LLYEKE4TJ54KECLAC7KFUV44IVHYCYLF45I43G5V3DPV5QDBUOIQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.546816995 (606h29m42.41790004s) [noise=true , address=145.79.227.40:4050 , piece_id=LA7KLHHWRNHULLPAZWQ6MLS76SLN3BONYXZJXCUHGFC5JZ4SQRIA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.547216927 (606h29m42.418299962s) [noise=true , address=145.79.227.15:4876 , piece_id=L2OIJAB6OOANT2EPFNVNLTMYCH334JGFBFASD44EHOAWTAHKUV5Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.547615996 (606h29m42.418699042s) [noise=true , address=145.79.227.17:4971 , piece_id=DYZUALBL3V75TKGK57LOJABWVCOD5UVS3WH74I55OVUDGRQUA3QQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.547841055 (606h29m42.4189241s) [noise=true , address=77.90.31.21:4877 , piece_id=BD2FEFNAGYWZYDA2IMOVHA7JE66EXSF3A3PL54Q2DUOJFG5ALG4Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548023322 (606h29m42.419106357s) [noise=true , address=45.91.116.14:4190 , piece_id=HHU5YZBTJEEWYVST6XQCKGWKRLRQIIR2UQTQ27RLK53BNAVOSHFQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548380923 (606h29m42.419463968s) [noise=true , address=45.91.116.11:4120 , piece_id=FZ5L6FM7IHX65EVFGVXXHEF43XQ4EBUH4C7VZUSTH2MVWLNR5M2Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548560475 (606h29m42.41964351s) [noise=true , address=145.79.227.39:4013 , piece_id=I4XP47SFJICIYIPEN4E7UC2HXRXH4IXVJYGHXTZBCUESAHNFQXJA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548719268 (606h29m42.419802313s) [noise=true , address=145.79.227.31:4797 , piece_id=S5YBLITONGU6RHPQENP2TCVVWOHEMXPMYXLNYWBZNXWMFI26PVUA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549024107 (606h29m42.420107152s) [noise=true , address=145.79.227.7:4294 , piece_id=C6N5OIRLV3LLSCEIOGQJKWPKENGZYNAPY6V2AYZNKUMACBP5SLWQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549197708 (606h29m42.420280753s) [noise=true , address=145.79.227.16:4910 , piece_id=O4YO367LEVZSRI5E75YEZ7W7YI2KGKKQDXCOHQ3LALE4WUAPQVHQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549506406 (606h29m42.420589441s) [noise=true , address=145.79.227.12:4765 , piece_id=4ZF5YQP4Z5HBTX5YWUDHZV55Q47WASRLMV6ZYJXJRL2QBAVQQM4Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549971591 (606h29m42.421054626s) [noise=true , address=145.79.227.32:4835 , piece_id=CN4NMCFNYKOCG2JCIEXJBL2PLEYWD5THCZB4WQDURIECFVHLLX6Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.551571588 (606h29m42.422654623s) [noise=true , address=145.79.227.24:4581 , piece_id=Y6GBEOR7DQX2H6LBHXYFOXRBX3GP7AKPYVGTQNMUEXKPXLONUUGQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:44.247717212 (606h29m44.118800248s) [upload=531550]: Next returned nil segment Dec 30 21:58:44.247730348 (606h29m44.118813383s) [upload=531550]: Waiting for error group managing segments... Dec 30 21:58:44.532148720 (606h29m44.403231755s) [noise=true , address=145.79.227.31:4797 , piece_id=S5YBLITONGU6RHPQENP2TCVVWOHEMXPMYXLNYWBZNXWMFI26PVUA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.534655293 (606h29m44.405738328s) [noise=true , address=145.79.227.39:4013 , piece_id=I4XP47SFJICIYIPEN4E7UC2HXRXH4IXVJYGHXTZBCUESAHNFQXJA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.544709427 (606h29m44.415792462s) [noise=true , address=145.79.227.12:4765 , piece_id=4ZF5YQP4Z5HBTX5YWUDHZV55Q47WASRLMV6ZYJXJRL2QBAVQQM4Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.545822306 (606h29m44.416905342s) [noise=true , address=145.79.227.7:4294 , piece_id=C6N5OIRLV3LLSCEIOGQJKWPKENGZYNAPY6V2AYZNKUMACBP5SLWQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.549465162 (606h29m44.420548198s) [noise=true , address=145.79.227.24:4581 , piece_id=Y6GBEOR7DQX2H6LBHXYFOXRBX3GP7AKPYVGTQNMUEXKPXLONUUGQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.556935928 (606h29m44.428018964s) [noise=true , address=145.79.227.15:4876 , piece_id=L2OIJAB6OOANT2EPFNVNLTMYCH334JGFBFASD44EHOAWTAHKUV5Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.561283296 (606h29m44.432366331s) [noise=true , address=145.79.227.16:4910 , piece_id=O4YO367LEVZSRI5E75YEZ7W7YI2KGKKQDXCOHQ3LALE4WUAPQVHQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.614194264 (606h29m44.485277309s) [noise=true , address=145.79.227.17:4971 , piece_id=DYZUALBL3V75TKGK57LOJABWVCOD5UVS3WH74I55OVUDGRQUA3QQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.615630689 (606h29m44.486713724s) [noise=true , address=145.79.227.32:4835 , piece_id=CN4NMCFNYKOCG2JCIEXJBL2PLEYWD5THCZB4WQDURIECFVHLLX6Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.639672077 (606h29m44.510755122s) [noise=true , address=145.79.227.23:4547 , piece_id=LYGUMTAP7OPJWGRVFY65GUZFE4BBTV73JTG657DR5OM26LA7UFZA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.699618771 (606h29m44.570701817s) [noise=true , address=77.90.31.21:4877 , piece_id=BD2FEFNAGYWZYDA2IMOVHA7JE66EXSF3A3PL54Q2DUOJFG5ALG4Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.841535416 (606h29m44.712618462s) [noise=true , address=45.91.116.14:4190 , piece_id=HHU5YZBTJEEWYVST6XQCKGWKRLRQIIR2UQTQ27RLK53BNAVOSHFQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.974660841 (606h29m44.845743886s) [noise=true , address=45.91.116.12:4225 , piece_id=LLYEKE4TJ54KECLAC7KFUV44IVHYCYLF45I43G5V3DPV5QDBUOIQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.974681401 (606h29m44.845764436s) [seg_pos={0 0}, upload=531550]: Segment reached optimal threshold of 13 pieces. Dec 30 21:58:44.974855342 (606h29m44.845938377s) [noise=true , address=145.79.227.49:4339 , piece_id=WBBPZK2VY2QVRTQNXEKQ7TRYPWBDYS2TGTKEDAAXQ2AT4TUQXTQQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 30 21:58:44.974861423 (606h29m44.845944458s) [noise=true , address=145.79.227.49:4339 , piece_id=WBBPZK2VY2QVRTQNXEKQ7TRYPWBDYS2TGTKEDAAXQ2AT4TUQXTQQ, seg_pos={0 0}, upload=531550]: Long tail cancellation due to optimal threshold Dec 30 21:58:44.974968166 (606h29m44.846051201s) [noise=true , address=145.79.227.40:4050 , piece_id=LA7KLHHWRNHULLPAZWQ6MLS76SLN3BONYXZJXCUHGFC5JZ4SQRIA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 30 21:58:44.974973627 (606h29m44.846056662s) [noise=true , address=145.79.227.40:4050 , piece_id=LA7KLHHWRNHULLPAZWQ6MLS76SLN3BONYXZJXCUHGFC5JZ4SQRIA, seg_pos={0 0}, upload=531550]: Long tail cancellation due to optimal threshold Dec 30 21:58:44.975076152 (606h29m44.846159187s) [noise=true , address=45.91.116.11:4120 , piece_id=FZ5L6FM7IHX65EVFGVXXHEF43XQ4EBUH4C7VZUSTH2MVWLNR5M2Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 30 21:58:44.975083497 (606h29m44.846166532s) [noise=true , address=45.91.116.11:4120 , piece_id=FZ5L6FM7IHX65EVFGVXXHEF43XQ4EBUH4C7VZUSTH2MVWLNR5M2Q, seg_pos={0 0}, upload=531550]: Long tail cancellation due to optimal threshold Dec 30 21:58:44.975096711 (606h29m44.846179747s) [seg_pos={0 0}, upload=531550]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 21:58:44.975160703 (606h29m44.846243739s) [upload=531550]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 21:58:44.975164691 (606h29m44.846247726s) [upload=531550]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 21:58:45.272782726 (606h29m45.143865772s) [upload=531550]: Done uploading segments... Dec 30 21:58:45.272806352 (606h29m45.143889387s) [upload=531550]: Upload finished. err: <nil> Duration: 6.126347669s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................................X....................] duration: 3.70091405s trace: https://jaeger.dev.storj.io/trace/22a47ec27f9a101c FAIL (2025-12-30 22:08:36 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.700941253s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [....XX......X.........X.....................................] duration: 50.507564775s trace: https://jaeger.dev.storj.io/trace/5cdd5ba7b1cb22c7 FAIL (2025-12-30 22:08:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 21:37:03.355296401 (606h8m3.226379446s) [noise=true , address=145.79.227.22:4543 , piece_id=WWCZ6SEEXQ2KUJ32AGTIJNMSJROLYSJ4XQGBQJC4Q5DACZVA65RQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.355839374 (606h8m3.22692241s) [noise=true , address=145.79.227.35:4492 , piece_id=HCKFSQMJYEP6B5H4OX5AZ5TWP6Z4H4Z6WA2T7Z2H5KNZ4SDJRMUQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.356356930 (606h8m3.227439965s) [noise=true , address=45.91.116.13:4182 , piece_id=BYAYNP4JAAN3AW2LYSK74WJE4BRWUR5Z6K67MTLK6A73KEJ35WEQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.356675316 (606h8m3.227758361s) [noise=true , address=77.90.31.12:4612 , piece_id=63TS3SUEFUE6PORZ4W76OXRWFL7SBN773UGGY3DU7JPBAW66N7UA, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.356992732 (606h8m3.228075767s) [noise=true , address=64.49.13.140:4507 , piece_id=E7T6WL45F6GHECCRANX6AIKWFRZ77TURRWORYEYI43MET2WG7SRQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.357330644 (606h8m3.228413679s) [noise=true , address=77.90.31.21:4886 , piece_id=ZQRCPTVC3WFVQPZ36AU6KJW7PD3T557CXYQIICLDGOKK6JCPVZ5Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.357663649 (606h8m3.228746684s) [noise=true , address=145.79.227.49:4367 , piece_id=R2T2ABLXBPDVP3ZVA2TAYMCLQQ7JQUVQYCTHCKGMFBRVWCDAO4PQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.359827449 (606h8m3.230910484s) [noise=true , address=145.79.227.34:4918 , piece_id=HUMOHOT3HOYCBJ23WXQXWU3X6C6JQTRLLHTD4RQLCQTCB3HXI7EQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.360086692 (606h8m3.231169727s) [noise=true , address=145.79.227.7:4320 , piece_id=MAICNTHGLZ6W27HXDOOZXOBRGQGEXNSA4BJT2T5D5PHRVV6ZMX2Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.361622167 (606h8m3.232705212s) [noise=true , address=145.79.227.33:4888 , piece_id=I4426P3VETHPOL5WHWJEGVYRXJULTCNDX3WSUTS3XZVY36Q2N44Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.362965514 (606h8m3.234048559s) [noise=true , address=45.91.116.11:4133 , piece_id=NEW7ZOVP5E7HYGYD4LG5UG2MDLAQW2OBRTNZDFIYO4CQYMVAWRPQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.364308883 (606h8m3.235391928s) [noise=true , address=145.79.227.26:4663 , piece_id=DYEFQZL4X74QR2LAYURT5ZTR2IWIDB55VLBQOGZM36MA54P2SWJA, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.365734507 (606h8m3.236817542s) [noise=true , address=145.79.227.40:4059 , piece_id=7WSUBR3C3PWVBCWD6KRGSD6MPX4IH2YQOAVO4J4SWZORNHGS2IPQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.366982764 (606h8m3.238065799s) [noise=true , address=145.79.227.29:4973 , piece_id=5OROXELR55A47BFPSJJEWVQU4YW6CAOK6LMEHCINM4FN3OVI2Y6Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.368390294 (606h8m3.239473339s) [noise=true , address=77.90.31.19:4852 , piece_id=2NYO63GCBGCYBFYTMUGNF2NG5FSJOD3AOUOD2FZZIWDINK4X6DJQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:04.736295138 (606h8m4.607378183s) [upload=531240]: Next returned nil segment Dec 30 21:37:04.736311850 (606h8m4.607394885s) [upload=531240]: Waiting for error group managing segments... Dec 30 21:37:05.104989591 (606h8m4.976072636s) [noise=true , address=64.49.13.140:4507 , piece_id=E7T6WL45F6GHECCRANX6AIKWFRZ77TURRWORYEYI43MET2WG7SRQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.208809748 (606h8m5.079892793s) [noise=true , address=145.79.227.22:4543 , piece_id=WWCZ6SEEXQ2KUJ32AGTIJNMSJROLYSJ4XQGBQJC4Q5DACZVA65RQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.441125389 (606h8m5.312208444s) [noise=true , address=77.90.31.19:4852 , piece_id=2NYO63GCBGCYBFYTMUGNF2NG5FSJOD3AOUOD2FZZIWDINK4X6DJQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.483630904 (606h8m5.35471394s) [noise=true , address=77.90.31.12:4612 , piece_id=63TS3SUEFUE6PORZ4W76OXRWFL7SBN773UGGY3DU7JPBAW66N7UA, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.494478811 (606h8m5.365561846s) [noise=true , address=77.90.31.11:4559 , piece_id=3DYD4CNPJQN3S4EYZJGREQTTZJDSL2QTC3OT6G2VFUGMLSVCZUFA, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.497452683 (606h8m5.368535729s) [noise=true , address=77.90.31.17:4769 , piece_id=QXDYESBKGXRY7BCFYYF2O7OXS7Y4Z4KUMJB6AXZAWX735SYU56GQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.661529377 (606h8m5.532612412s) [noise=true , address=45.91.116.13:4182 , piece_id=BYAYNP4JAAN3AW2LYSK74WJE4BRWUR5Z6K67MTLK6A73KEJ35WEQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.729773112 (606h8m5.600856157s) [noise=true , address=45.91.116.11:4133 , piece_id=NEW7ZOVP5E7HYGYD4LG5UG2MDLAQW2OBRTNZDFIYO4CQYMVAWRPQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.940743308 (606h8m5.811826353s) [noise=true , address=77.90.31.21:4886 , piece_id=ZQRCPTVC3WFVQPZ36AU6KJW7PD3T557CXYQIICLDGOKK6JCPVZ5Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.084240044 (606h8m5.95532309s) [noise=true , address=145.79.227.30:4784 , piece_id=DSXUDNYCWO5F6IYXMIVJNEASYXIZ7EO7RKAS7QRUYK5OU7GOOB5Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.121095032 (606h8m5.992178067s) [noise=true , address=145.79.227.40:4059 , piece_id=7WSUBR3C3PWVBCWD6KRGSD6MPX4IH2YQOAVO4J4SWZORNHGS2IPQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.169195983 (606h8m6.040279028s) [noise=true , address=145.79.227.49:4367 , piece_id=R2T2ABLXBPDVP3ZVA2TAYMCLQQ7JQUVQYCTHCKGMFBRVWCDAO4PQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.169380634 (606h8m6.040463679s) [noise=true , address=145.79.227.33:4888 , piece_id=I4426P3VETHPOL5WHWJEGVYRXJULTCNDX3WSUTS3XZVY36Q2N44Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.179060175 (606h8m6.050143211s) [noise=true , address=145.79.227.29:4973 , piece_id=5OROXELR55A47BFPSJJEWVQU4YW6CAOK6LMEHCINM4FN3OVI2Y6Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.179080424 (606h8m6.050163459s) [seg_pos={0 0}, upload=531240]: Segment reached optimal threshold of 14 pieces. Dec 30 21:37:06.179262801 (606h8m6.050345837s) [noise=true , address=145.79.227.26:4663 , piece_id=DYEFQZL4X74QR2LAYURT5ZTR2IWIDB55VLBQOGZM36MA54P2SWJA, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SwpAaXTA3BqUvwubzmJkvwokdz6SSpT9t8HPtQcXXJxmcgfm2): protocol: write tcp 188.166.225.170:43184->145.79.227.26:4663: use of closed network connection; write tcp 188.166.225.170:43184->145.79.227.26:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43184->145.79.227.26:4663: use of closed network connection Dec 30 21:37:06.179268703 (606h8m6.050351748s) [noise=true , address=145.79.227.26:4663 , piece_id=DYEFQZL4X74QR2LAYURT5ZTR2IWIDB55VLBQOGZM36MA54P2SWJA, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179363724 (606h8m6.050446769s) [noise=true , address=145.79.227.7:4320 , piece_id=MAICNTHGLZ6W27HXDOOZXOBRGQGEXNSA4BJT2T5D5PHRVV6ZMX2Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126rEgnQw1w7hsFpRqQP9nLaumdBfdtkgYjdw6WE6nXrxk5kj4B): protocol: write tcp 188.166.225.170:45228->145.79.227.7:4320: use of closed network connection; write tcp 188.166.225.170:45228->145.79.227.7:4320: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45228->145.79.227.7:4320: use of closed network connection Dec 30 21:37:06.179369525 (606h8m6.05045257s) [noise=true , address=145.79.227.7:4320 , piece_id=MAICNTHGLZ6W27HXDOOZXOBRGQGEXNSA4BJT2T5D5PHRVV6ZMX2Q, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179532996 (606h8m6.050616061s) [noise=true , address=145.79.227.34:4918 , piece_id=HUMOHOT3HOYCBJ23WXQXWU3X6C6JQTRLLHTD4RQLCQTCB3HXI7EQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF Dec 30 21:37:06.179572371 (606h8m6.050655406s) [noise=true , address=145.79.227.34:4918 , piece_id=HUMOHOT3HOYCBJ23WXQXWU3X6C6JQTRLLHTD4RQLCQTCB3HXI7EQ, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179638587 (606h8m6.050721622s) [noise=true , address=145.79.227.35:4492 , piece_id=HCKFSQMJYEP6B5H4OX5AZ5TWP6Z4H4Z6WA2T7Z2H5KNZ4SDJRMUQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1roa5GGYmqCX2AxSrqFgqyV2CtBEYZavuGTseiwDy4pjkT6bip): protocol: expected piece hash; context canceled; EOF Dec 30 21:37:06.179647855 (606h8m6.05073089s) [noise=true , address=145.79.227.35:4492 , piece_id=HCKFSQMJYEP6B5H4OX5AZ5TWP6Z4H4Z6WA2T7Z2H5KNZ4SDJRMUQ, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179666641 (606h8m6.050749686s) [seg_pos={0 0}, upload=531240]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 21:37:06.179742766 (606h8m6.050825811s) [upload=531240]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 21:37:06.179747545 (606h8m6.05083059s) [upload=531240]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 21:37:06.490986034 (606h8m6.362069079s) [upload=531240]: Done uploading segments... Dec 30 21:37:06.491006261 (606h8m6.362089307s) [upload=531240]: Upload finished. err: <nil> Duration: 50.507593209s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...X..X...X.........X.........................X.............] duration: 18.839917617s trace: https://jaeger.dev.storj.io/trace/57a7ab98e7eb0d3f FAIL (2025-12-30 22:08:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 21:58:42.298454527 (606h29m42.169537592s) [upload=531550]: Starting upload Dec 30 21:58:42.298523438 (606h29m42.169606473s) [upload=531550]: Done starting upload Dec 30 21:58:42.298568774 (606h29m42.169651809s) [upload=531550]: Uploading segments... Dec 30 21:58:42.298697850 (606h29m42.169780885s) [upload=531550]: Got next segment. Inline: false Dec 30 21:58:42.298728649 (606h29m42.169811694s) [upload=531550]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 21:58:42.298733227 (606h29m42.169816262s) [upload=531550]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 21:58:42.545734213 (606h29m42.416817288s) [seg_pos={0 0}, upload=531550]: Begin upload segment... Dec 30 21:58:42.545786032 (606h29m42.416869078s) [seg_pos={0 0}, upload=531550]: Done begin upload segment. Dec 30 21:58:42.545855104 (606h29m42.416938149s) [noise=true , address=145.79.227.23:4547 , piece_id=LYGUMTAP7OPJWGRVFY65GUZFE4BBTV73JTG657DR5OM26LA7UFZA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.545897384 (606h29m42.416980449s) [noise=true , address=145.79.227.49:4339 , piece_id=WBBPZK2VY2QVRTQNXEKQ7TRYPWBDYS2TGTKEDAAXQ2AT4TUQXTQQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.546757752 (606h29m42.417840817s) [noise=true , address=45.91.116.12:4225 , piece_id=LLYEKE4TJ54KECLAC7KFUV44IVHYCYLF45I43G5V3DPV5QDBUOIQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.546816995 (606h29m42.41790004s) [noise=true , address=145.79.227.40:4050 , piece_id=LA7KLHHWRNHULLPAZWQ6MLS76SLN3BONYXZJXCUHGFC5JZ4SQRIA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.547216927 (606h29m42.418299962s) [noise=true , address=145.79.227.15:4876 , piece_id=L2OIJAB6OOANT2EPFNVNLTMYCH334JGFBFASD44EHOAWTAHKUV5Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.547615996 (606h29m42.418699042s) [noise=true , address=145.79.227.17:4971 , piece_id=DYZUALBL3V75TKGK57LOJABWVCOD5UVS3WH74I55OVUDGRQUA3QQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.547841055 (606h29m42.4189241s) [noise=true , address=77.90.31.21:4877 , piece_id=BD2FEFNAGYWZYDA2IMOVHA7JE66EXSF3A3PL54Q2DUOJFG5ALG4Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548023322 (606h29m42.419106357s) [noise=true , address=45.91.116.14:4190 , piece_id=HHU5YZBTJEEWYVST6XQCKGWKRLRQIIR2UQTQ27RLK53BNAVOSHFQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548380923 (606h29m42.419463968s) [noise=true , address=45.91.116.11:4120 , piece_id=FZ5L6FM7IHX65EVFGVXXHEF43XQ4EBUH4C7VZUSTH2MVWLNR5M2Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548560475 (606h29m42.41964351s) [noise=true , address=145.79.227.39:4013 , piece_id=I4XP47SFJICIYIPEN4E7UC2HXRXH4IXVJYGHXTZBCUESAHNFQXJA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548719268 (606h29m42.419802313s) [noise=true , address=145.79.227.31:4797 , piece_id=S5YBLITONGU6RHPQENP2TCVVWOHEMXPMYXLNYWBZNXWMFI26PVUA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549024107 (606h29m42.420107152s) [noise=true , address=145.79.227.7:4294 , piece_id=C6N5OIRLV3LLSCEIOGQJKWPKENGZYNAPY6V2AYZNKUMACBP5SLWQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549197708 (606h29m42.420280753s) [noise=true , address=145.79.227.16:4910 , piece_id=O4YO367LEVZSRI5E75YEZ7W7YI2KGKKQDXCOHQ3LALE4WUAPQVHQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549506406 (606h29m42.420589441s) [noise=true , address=145.79.227.12:4765 , piece_id=4ZF5YQP4Z5HBTX5YWUDHZV55Q47WASRLMV6ZYJXJRL2QBAVQQM4Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549971591 (606h29m42.421054626s) [noise=true , address=145.79.227.32:4835 , piece_id=CN4NMCFNYKOCG2JCIEXJBL2PLEYWD5THCZB4WQDURIECFVHLLX6Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.551571588 (606h29m42.422654623s) [noise=true , address=145.79.227.24:4581 , piece_id=Y6GBEOR7DQX2H6LBHXYFOXRBX3GP7AKPYVGTQNMUEXKPXLONUUGQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:44.247717212 (606h29m44.118800248s) [upload=531550]: Next returned nil segment Dec 30 21:58:44.247730348 (606h29m44.118813383s) [upload=531550]: Waiting for error group managing segments... Dec 30 21:58:44.532148720 (606h29m44.403231755s) [noise=true , address=145.79.227.31:4797 , piece_id=S5YBLITONGU6RHPQENP2TCVVWOHEMXPMYXLNYWBZNXWMFI26PVUA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.534655293 (606h29m44.405738328s) [noise=true , address=145.79.227.39:4013 , piece_id=I4XP47SFJICIYIPEN4E7UC2HXRXH4IXVJYGHXTZBCUESAHNFQXJA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.544709427 (606h29m44.415792462s) [noise=true , address=145.79.227.12:4765 , piece_id=4ZF5YQP4Z5HBTX5YWUDHZV55Q47WASRLMV6ZYJXJRL2QBAVQQM4Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.545822306 (606h29m44.416905342s) [noise=true , address=145.79.227.7:4294 , piece_id=C6N5OIRLV3LLSCEIOGQJKWPKENGZYNAPY6V2AYZNKUMACBP5SLWQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.549465162 (606h29m44.420548198s) [noise=true , address=145.79.227.24:4581 , piece_id=Y6GBEOR7DQX2H6LBHXYFOXRBX3GP7AKPYVGTQNMUEXKPXLONUUGQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.556935928 (606h29m44.428018964s) [noise=true , address=145.79.227.15:4876 , piece_id=L2OIJAB6OOANT2EPFNVNLTMYCH334JGFBFASD44EHOAWTAHKUV5Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.561283296 (606h29m44.432366331s) [noise=true , address=145.79.227.16:4910 , piece_id=O4YO367LEVZSRI5E75YEZ7W7YI2KGKKQDXCOHQ3LALE4WUAPQVHQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.614194264 (606h29m44.485277309s) [noise=true , address=145.79.227.17:4971 , piece_id=DYZUALBL3V75TKGK57LOJABWVCOD5UVS3WH74I55OVUDGRQUA3QQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.615630689 (606h29m44.486713724s) [noise=true , address=145.79.227.32:4835 , piece_id=CN4NMCFNYKOCG2JCIEXJBL2PLEYWD5THCZB4WQDURIECFVHLLX6Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.639672077 (606h29m44.510755122s) [noise=true , address=145.79.227.23:4547 , piece_id=LYGUMTAP7OPJWGRVFY65GUZFE4BBTV73JTG657DR5OM26LA7UFZA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.699618771 (606h29m44.570701817s) [noise=true , address=77.90.31.21:4877 , piece_id=BD2FEFNAGYWZYDA2IMOVHA7JE66EXSF3A3PL54Q2DUOJFG5ALG4Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.841535416 (606h29m44.712618462s) [noise=true , address=45.91.116.14:4190 , piece_id=HHU5YZBTJEEWYVST6XQCKGWKRLRQIIR2UQTQ27RLK53BNAVOSHFQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.974660841 (606h29m44.845743886s) [noise=true , address=45.91.116.12:4225 , piece_id=LLYEKE4TJ54KECLAC7KFUV44IVHYCYLF45I43G5V3DPV5QDBUOIQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.974681401 (606h29m44.845764436s) [seg_pos={0 0}, upload=531550]: Segment reached optimal threshold of 13 pieces. Dec 30 21:58:44.974855342 (606h29m44.845938377s) [noise=true , address=145.79.227.49:4339 , piece_id=WBBPZK2VY2QVRTQNXEKQ7TRYPWBDYS2TGTKEDAAXQ2AT4TUQXTQQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 30 21:58:44.974861423 (606h29m44.845944458s) [noise=true , address=145.79.227.49:4339 , piece_id=WBBPZK2VY2QVRTQNXEKQ7TRYPWBDYS2TGTKEDAAXQ2AT4TUQXTQQ, seg_pos={0 0}, upload=531550]: Long tail cancellation due to optimal threshold Dec 30 21:58:44.974968166 (606h29m44.846051201s) [noise=true , address=145.79.227.40:4050 , piece_id=LA7KLHHWRNHULLPAZWQ6MLS76SLN3BONYXZJXCUHGFC5JZ4SQRIA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 30 21:58:44.974973627 (606h29m44.846056662s) [noise=true , address=145.79.227.40:4050 , piece_id=LA7KLHHWRNHULLPAZWQ6MLS76SLN3BONYXZJXCUHGFC5JZ4SQRIA, seg_pos={0 0}, upload=531550]: Long tail cancellation due to optimal threshold Dec 30 21:58:44.975076152 (606h29m44.846159187s) [noise=true , address=45.91.116.11:4120 , piece_id=FZ5L6FM7IHX65EVFGVXXHEF43XQ4EBUH4C7VZUSTH2MVWLNR5M2Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 30 21:58:44.975083497 (606h29m44.846166532s) [noise=true , address=45.91.116.11:4120 , piece_id=FZ5L6FM7IHX65EVFGVXXHEF43XQ4EBUH4C7VZUSTH2MVWLNR5M2Q, seg_pos={0 0}, upload=531550]: Long tail cancellation due to optimal threshold Dec 30 21:58:44.975096711 (606h29m44.846179747s) [seg_pos={0 0}, upload=531550]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 21:58:44.975160703 (606h29m44.846243739s) [upload=531550]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 21:58:44.975164691 (606h29m44.846247726s) [upload=531550]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 21:58:45.272782726 (606h29m45.143865772s) [upload=531550]: Done uploading segments... Dec 30 21:58:45.272806352 (606h29m45.143889387s) [upload=531550]: Upload finished. err: <nil> Duration: 18.839947663s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................................X...................] duration: 3.646519932s trace: https://jaeger.dev.storj.io/trace/22a47ec27f9a101c FAIL (2025-12-30 22:08:02 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.646549638s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................................X..................] duration: 3.515842988s trace: https://jaeger.dev.storj.io/trace/22a47ec27f9a101c FAIL (2025-12-30 22:07:44 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.515864459s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................................X.................] duration: 3.535667656s trace: https://jaeger.dev.storj.io/trace/22a47ec27f9a101c FAIL (2025-12-30 22:07:06 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.535696572s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....XX......X.........X....................................] duration: 15.440559262s trace: https://jaeger.dev.storj.io/trace/5cdd5ba7b1cb22c7 FAIL (2025-12-30 22:06:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 21:37:03.355296401 (606h8m3.226379446s) [noise=true , address=145.79.227.22:4543 , piece_id=WWCZ6SEEXQ2KUJ32AGTIJNMSJROLYSJ4XQGBQJC4Q5DACZVA65RQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.355839374 (606h8m3.22692241s) [noise=true , address=145.79.227.35:4492 , piece_id=HCKFSQMJYEP6B5H4OX5AZ5TWP6Z4H4Z6WA2T7Z2H5KNZ4SDJRMUQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.356356930 (606h8m3.227439965s) [noise=true , address=45.91.116.13:4182 , piece_id=BYAYNP4JAAN3AW2LYSK74WJE4BRWUR5Z6K67MTLK6A73KEJ35WEQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.356675316 (606h8m3.227758361s) [noise=true , address=77.90.31.12:4612 , piece_id=63TS3SUEFUE6PORZ4W76OXRWFL7SBN773UGGY3DU7JPBAW66N7UA, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.356992732 (606h8m3.228075767s) [noise=true , address=64.49.13.140:4507 , piece_id=E7T6WL45F6GHECCRANX6AIKWFRZ77TURRWORYEYI43MET2WG7SRQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.357330644 (606h8m3.228413679s) [noise=true , address=77.90.31.21:4886 , piece_id=ZQRCPTVC3WFVQPZ36AU6KJW7PD3T557CXYQIICLDGOKK6JCPVZ5Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.357663649 (606h8m3.228746684s) [noise=true , address=145.79.227.49:4367 , piece_id=R2T2ABLXBPDVP3ZVA2TAYMCLQQ7JQUVQYCTHCKGMFBRVWCDAO4PQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.359827449 (606h8m3.230910484s) [noise=true , address=145.79.227.34:4918 , piece_id=HUMOHOT3HOYCBJ23WXQXWU3X6C6JQTRLLHTD4RQLCQTCB3HXI7EQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.360086692 (606h8m3.231169727s) [noise=true , address=145.79.227.7:4320 , piece_id=MAICNTHGLZ6W27HXDOOZXOBRGQGEXNSA4BJT2T5D5PHRVV6ZMX2Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.361622167 (606h8m3.232705212s) [noise=true , address=145.79.227.33:4888 , piece_id=I4426P3VETHPOL5WHWJEGVYRXJULTCNDX3WSUTS3XZVY36Q2N44Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.362965514 (606h8m3.234048559s) [noise=true , address=45.91.116.11:4133 , piece_id=NEW7ZOVP5E7HYGYD4LG5UG2MDLAQW2OBRTNZDFIYO4CQYMVAWRPQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.364308883 (606h8m3.235391928s) [noise=true , address=145.79.227.26:4663 , piece_id=DYEFQZL4X74QR2LAYURT5ZTR2IWIDB55VLBQOGZM36MA54P2SWJA, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.365734507 (606h8m3.236817542s) [noise=true , address=145.79.227.40:4059 , piece_id=7WSUBR3C3PWVBCWD6KRGSD6MPX4IH2YQOAVO4J4SWZORNHGS2IPQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.366982764 (606h8m3.238065799s) [noise=true , address=145.79.227.29:4973 , piece_id=5OROXELR55A47BFPSJJEWVQU4YW6CAOK6LMEHCINM4FN3OVI2Y6Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.368390294 (606h8m3.239473339s) [noise=true , address=77.90.31.19:4852 , piece_id=2NYO63GCBGCYBFYTMUGNF2NG5FSJOD3AOUOD2FZZIWDINK4X6DJQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:04.736295138 (606h8m4.607378183s) [upload=531240]: Next returned nil segment Dec 30 21:37:04.736311850 (606h8m4.607394885s) [upload=531240]: Waiting for error group managing segments... Dec 30 21:37:05.104989591 (606h8m4.976072636s) [noise=true , address=64.49.13.140:4507 , piece_id=E7T6WL45F6GHECCRANX6AIKWFRZ77TURRWORYEYI43MET2WG7SRQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.208809748 (606h8m5.079892793s) [noise=true , address=145.79.227.22:4543 , piece_id=WWCZ6SEEXQ2KUJ32AGTIJNMSJROLYSJ4XQGBQJC4Q5DACZVA65RQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.441125389 (606h8m5.312208444s) [noise=true , address=77.90.31.19:4852 , piece_id=2NYO63GCBGCYBFYTMUGNF2NG5FSJOD3AOUOD2FZZIWDINK4X6DJQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.483630904 (606h8m5.35471394s) [noise=true , address=77.90.31.12:4612 , piece_id=63TS3SUEFUE6PORZ4W76OXRWFL7SBN773UGGY3DU7JPBAW66N7UA, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.494478811 (606h8m5.365561846s) [noise=true , address=77.90.31.11:4559 , piece_id=3DYD4CNPJQN3S4EYZJGREQTTZJDSL2QTC3OT6G2VFUGMLSVCZUFA, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.497452683 (606h8m5.368535729s) [noise=true , address=77.90.31.17:4769 , piece_id=QXDYESBKGXRY7BCFYYF2O7OXS7Y4Z4KUMJB6AXZAWX735SYU56GQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.661529377 (606h8m5.532612412s) [noise=true , address=45.91.116.13:4182 , piece_id=BYAYNP4JAAN3AW2LYSK74WJE4BRWUR5Z6K67MTLK6A73KEJ35WEQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.729773112 (606h8m5.600856157s) [noise=true , address=45.91.116.11:4133 , piece_id=NEW7ZOVP5E7HYGYD4LG5UG2MDLAQW2OBRTNZDFIYO4CQYMVAWRPQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.940743308 (606h8m5.811826353s) [noise=true , address=77.90.31.21:4886 , piece_id=ZQRCPTVC3WFVQPZ36AU6KJW7PD3T557CXYQIICLDGOKK6JCPVZ5Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.084240044 (606h8m5.95532309s) [noise=true , address=145.79.227.30:4784 , piece_id=DSXUDNYCWO5F6IYXMIVJNEASYXIZ7EO7RKAS7QRUYK5OU7GOOB5Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.121095032 (606h8m5.992178067s) [noise=true , address=145.79.227.40:4059 , piece_id=7WSUBR3C3PWVBCWD6KRGSD6MPX4IH2YQOAVO4J4SWZORNHGS2IPQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.169195983 (606h8m6.040279028s) [noise=true , address=145.79.227.49:4367 , piece_id=R2T2ABLXBPDVP3ZVA2TAYMCLQQ7JQUVQYCTHCKGMFBRVWCDAO4PQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.169380634 (606h8m6.040463679s) [noise=true , address=145.79.227.33:4888 , piece_id=I4426P3VETHPOL5WHWJEGVYRXJULTCNDX3WSUTS3XZVY36Q2N44Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.179060175 (606h8m6.050143211s) [noise=true , address=145.79.227.29:4973 , piece_id=5OROXELR55A47BFPSJJEWVQU4YW6CAOK6LMEHCINM4FN3OVI2Y6Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.179080424 (606h8m6.050163459s) [seg_pos={0 0}, upload=531240]: Segment reached optimal threshold of 14 pieces. Dec 30 21:37:06.179262801 (606h8m6.050345837s) [noise=true , address=145.79.227.26:4663 , piece_id=DYEFQZL4X74QR2LAYURT5ZTR2IWIDB55VLBQOGZM36MA54P2SWJA, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SwpAaXTA3BqUvwubzmJkvwokdz6SSpT9t8HPtQcXXJxmcgfm2): protocol: write tcp 188.166.225.170:43184->145.79.227.26:4663: use of closed network connection; write tcp 188.166.225.170:43184->145.79.227.26:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43184->145.79.227.26:4663: use of closed network connection Dec 30 21:37:06.179268703 (606h8m6.050351748s) [noise=true , address=145.79.227.26:4663 , piece_id=DYEFQZL4X74QR2LAYURT5ZTR2IWIDB55VLBQOGZM36MA54P2SWJA, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179363724 (606h8m6.050446769s) [noise=true , address=145.79.227.7:4320 , piece_id=MAICNTHGLZ6W27HXDOOZXOBRGQGEXNSA4BJT2T5D5PHRVV6ZMX2Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126rEgnQw1w7hsFpRqQP9nLaumdBfdtkgYjdw6WE6nXrxk5kj4B): protocol: write tcp 188.166.225.170:45228->145.79.227.7:4320: use of closed network connection; write tcp 188.166.225.170:45228->145.79.227.7:4320: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45228->145.79.227.7:4320: use of closed network connection Dec 30 21:37:06.179369525 (606h8m6.05045257s) [noise=true , address=145.79.227.7:4320 , piece_id=MAICNTHGLZ6W27HXDOOZXOBRGQGEXNSA4BJT2T5D5PHRVV6ZMX2Q, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179532996 (606h8m6.050616061s) [noise=true , address=145.79.227.34:4918 , piece_id=HUMOHOT3HOYCBJ23WXQXWU3X6C6JQTRLLHTD4RQLCQTCB3HXI7EQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF Dec 30 21:37:06.179572371 (606h8m6.050655406s) [noise=true , address=145.79.227.34:4918 , piece_id=HUMOHOT3HOYCBJ23WXQXWU3X6C6JQTRLLHTD4RQLCQTCB3HXI7EQ, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179638587 (606h8m6.050721622s) [noise=true , address=145.79.227.35:4492 , piece_id=HCKFSQMJYEP6B5H4OX5AZ5TWP6Z4H4Z6WA2T7Z2H5KNZ4SDJRMUQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1roa5GGYmqCX2AxSrqFgqyV2CtBEYZavuGTseiwDy4pjkT6bip): protocol: expected piece hash; context canceled; EOF Dec 30 21:37:06.179647855 (606h8m6.05073089s) [noise=true , address=145.79.227.35:4492 , piece_id=HCKFSQMJYEP6B5H4OX5AZ5TWP6Z4H4Z6WA2T7Z2H5KNZ4SDJRMUQ, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179666641 (606h8m6.050749686s) [seg_pos={0 0}, upload=531240]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 21:37:06.179742766 (606h8m6.050825811s) [upload=531240]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 21:37:06.179747545 (606h8m6.05083059s) [upload=531240]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 21:37:06.490986034 (606h8m6.362069079s) [upload=531240]: Done uploading segments... Dec 30 21:37:06.491006261 (606h8m6.362089307s) [upload=531240]: Upload finished. err: <nil> Duration: 15.440585972s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [....X..X...X.........X.........................X............] duration: 5.966049498s trace: https://jaeger.dev.storj.io/trace/57a7ab98e7eb0d3f FAIL (2025-12-30 22:06:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 21:58:42.298454527 (606h29m42.169537592s) [upload=531550]: Starting upload Dec 30 21:58:42.298523438 (606h29m42.169606473s) [upload=531550]: Done starting upload Dec 30 21:58:42.298568774 (606h29m42.169651809s) [upload=531550]: Uploading segments... Dec 30 21:58:42.298697850 (606h29m42.169780885s) [upload=531550]: Got next segment. Inline: false Dec 30 21:58:42.298728649 (606h29m42.169811694s) [upload=531550]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 21:58:42.298733227 (606h29m42.169816262s) [upload=531550]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 21:58:42.545734213 (606h29m42.416817288s) [seg_pos={0 0}, upload=531550]: Begin upload segment... Dec 30 21:58:42.545786032 (606h29m42.416869078s) [seg_pos={0 0}, upload=531550]: Done begin upload segment. Dec 30 21:58:42.545855104 (606h29m42.416938149s) [noise=true , address=145.79.227.23:4547 , piece_id=LYGUMTAP7OPJWGRVFY65GUZFE4BBTV73JTG657DR5OM26LA7UFZA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.545897384 (606h29m42.416980449s) [noise=true , address=145.79.227.49:4339 , piece_id=WBBPZK2VY2QVRTQNXEKQ7TRYPWBDYS2TGTKEDAAXQ2AT4TUQXTQQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.546757752 (606h29m42.417840817s) [noise=true , address=45.91.116.12:4225 , piece_id=LLYEKE4TJ54KECLAC7KFUV44IVHYCYLF45I43G5V3DPV5QDBUOIQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.546816995 (606h29m42.41790004s) [noise=true , address=145.79.227.40:4050 , piece_id=LA7KLHHWRNHULLPAZWQ6MLS76SLN3BONYXZJXCUHGFC5JZ4SQRIA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.547216927 (606h29m42.418299962s) [noise=true , address=145.79.227.15:4876 , piece_id=L2OIJAB6OOANT2EPFNVNLTMYCH334JGFBFASD44EHOAWTAHKUV5Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.547615996 (606h29m42.418699042s) [noise=true , address=145.79.227.17:4971 , piece_id=DYZUALBL3V75TKGK57LOJABWVCOD5UVS3WH74I55OVUDGRQUA3QQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.547841055 (606h29m42.4189241s) [noise=true , address=77.90.31.21:4877 , piece_id=BD2FEFNAGYWZYDA2IMOVHA7JE66EXSF3A3PL54Q2DUOJFG5ALG4Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548023322 (606h29m42.419106357s) [noise=true , address=45.91.116.14:4190 , piece_id=HHU5YZBTJEEWYVST6XQCKGWKRLRQIIR2UQTQ27RLK53BNAVOSHFQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548380923 (606h29m42.419463968s) [noise=true , address=45.91.116.11:4120 , piece_id=FZ5L6FM7IHX65EVFGVXXHEF43XQ4EBUH4C7VZUSTH2MVWLNR5M2Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548560475 (606h29m42.41964351s) [noise=true , address=145.79.227.39:4013 , piece_id=I4XP47SFJICIYIPEN4E7UC2HXRXH4IXVJYGHXTZBCUESAHNFQXJA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548719268 (606h29m42.419802313s) [noise=true , address=145.79.227.31:4797 , piece_id=S5YBLITONGU6RHPQENP2TCVVWOHEMXPMYXLNYWBZNXWMFI26PVUA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549024107 (606h29m42.420107152s) [noise=true , address=145.79.227.7:4294 , piece_id=C6N5OIRLV3LLSCEIOGQJKWPKENGZYNAPY6V2AYZNKUMACBP5SLWQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549197708 (606h29m42.420280753s) [noise=true , address=145.79.227.16:4910 , piece_id=O4YO367LEVZSRI5E75YEZ7W7YI2KGKKQDXCOHQ3LALE4WUAPQVHQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549506406 (606h29m42.420589441s) [noise=true , address=145.79.227.12:4765 , piece_id=4ZF5YQP4Z5HBTX5YWUDHZV55Q47WASRLMV6ZYJXJRL2QBAVQQM4Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549971591 (606h29m42.421054626s) [noise=true , address=145.79.227.32:4835 , piece_id=CN4NMCFNYKOCG2JCIEXJBL2PLEYWD5THCZB4WQDURIECFVHLLX6Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.551571588 (606h29m42.422654623s) [noise=true , address=145.79.227.24:4581 , piece_id=Y6GBEOR7DQX2H6LBHXYFOXRBX3GP7AKPYVGTQNMUEXKPXLONUUGQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:44.247717212 (606h29m44.118800248s) [upload=531550]: Next returned nil segment Dec 30 21:58:44.247730348 (606h29m44.118813383s) [upload=531550]: Waiting for error group managing segments... Dec 30 21:58:44.532148720 (606h29m44.403231755s) [noise=true , address=145.79.227.31:4797 , piece_id=S5YBLITONGU6RHPQENP2TCVVWOHEMXPMYXLNYWBZNXWMFI26PVUA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.534655293 (606h29m44.405738328s) [noise=true , address=145.79.227.39:4013 , piece_id=I4XP47SFJICIYIPEN4E7UC2HXRXH4IXVJYGHXTZBCUESAHNFQXJA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.544709427 (606h29m44.415792462s) [noise=true , address=145.79.227.12:4765 , piece_id=4ZF5YQP4Z5HBTX5YWUDHZV55Q47WASRLMV6ZYJXJRL2QBAVQQM4Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.545822306 (606h29m44.416905342s) [noise=true , address=145.79.227.7:4294 , piece_id=C6N5OIRLV3LLSCEIOGQJKWPKENGZYNAPY6V2AYZNKUMACBP5SLWQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.549465162 (606h29m44.420548198s) [noise=true , address=145.79.227.24:4581 , piece_id=Y6GBEOR7DQX2H6LBHXYFOXRBX3GP7AKPYVGTQNMUEXKPXLONUUGQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.556935928 (606h29m44.428018964s) [noise=true , address=145.79.227.15:4876 , piece_id=L2OIJAB6OOANT2EPFNVNLTMYCH334JGFBFASD44EHOAWTAHKUV5Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.561283296 (606h29m44.432366331s) [noise=true , address=145.79.227.16:4910 , piece_id=O4YO367LEVZSRI5E75YEZ7W7YI2KGKKQDXCOHQ3LALE4WUAPQVHQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.614194264 (606h29m44.485277309s) [noise=true , address=145.79.227.17:4971 , piece_id=DYZUALBL3V75TKGK57LOJABWVCOD5UVS3WH74I55OVUDGRQUA3QQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.615630689 (606h29m44.486713724s) [noise=true , address=145.79.227.32:4835 , piece_id=CN4NMCFNYKOCG2JCIEXJBL2PLEYWD5THCZB4WQDURIECFVHLLX6Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.639672077 (606h29m44.510755122s) [noise=true , address=145.79.227.23:4547 , piece_id=LYGUMTAP7OPJWGRVFY65GUZFE4BBTV73JTG657DR5OM26LA7UFZA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.699618771 (606h29m44.570701817s) [noise=true , address=77.90.31.21:4877 , piece_id=BD2FEFNAGYWZYDA2IMOVHA7JE66EXSF3A3PL54Q2DUOJFG5ALG4Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.841535416 (606h29m44.712618462s) [noise=true , address=45.91.116.14:4190 , piece_id=HHU5YZBTJEEWYVST6XQCKGWKRLRQIIR2UQTQ27RLK53BNAVOSHFQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.974660841 (606h29m44.845743886s) [noise=true , address=45.91.116.12:4225 , piece_id=LLYEKE4TJ54KECLAC7KFUV44IVHYCYLF45I43G5V3DPV5QDBUOIQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.974681401 (606h29m44.845764436s) [seg_pos={0 0}, upload=531550]: Segment reached optimal threshold of 13 pieces. Dec 30 21:58:44.974855342 (606h29m44.845938377s) [noise=true , address=145.79.227.49:4339 , piece_id=WBBPZK2VY2QVRTQNXEKQ7TRYPWBDYS2TGTKEDAAXQ2AT4TUQXTQQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 30 21:58:44.974861423 (606h29m44.845944458s) [noise=true , address=145.79.227.49:4339 , piece_id=WBBPZK2VY2QVRTQNXEKQ7TRYPWBDYS2TGTKEDAAXQ2AT4TUQXTQQ, seg_pos={0 0}, upload=531550]: Long tail cancellation due to optimal threshold Dec 30 21:58:44.974968166 (606h29m44.846051201s) [noise=true , address=145.79.227.40:4050 , piece_id=LA7KLHHWRNHULLPAZWQ6MLS76SLN3BONYXZJXCUHGFC5JZ4SQRIA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 30 21:58:44.974973627 (606h29m44.846056662s) [noise=true , address=145.79.227.40:4050 , piece_id=LA7KLHHWRNHULLPAZWQ6MLS76SLN3BONYXZJXCUHGFC5JZ4SQRIA, seg_pos={0 0}, upload=531550]: Long tail cancellation due to optimal threshold Dec 30 21:58:44.975076152 (606h29m44.846159187s) [noise=true , address=45.91.116.11:4120 , piece_id=FZ5L6FM7IHX65EVFGVXXHEF43XQ4EBUH4C7VZUSTH2MVWLNR5M2Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 30 21:58:44.975083497 (606h29m44.846166532s) [noise=true , address=45.91.116.11:4120 , piece_id=FZ5L6FM7IHX65EVFGVXXHEF43XQ4EBUH4C7VZUSTH2MVWLNR5M2Q, seg_pos={0 0}, upload=531550]: Long tail cancellation due to optimal threshold Dec 30 21:58:44.975096711 (606h29m44.846179747s) [seg_pos={0 0}, upload=531550]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 21:58:44.975160703 (606h29m44.846243739s) [upload=531550]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 21:58:44.975164691 (606h29m44.846247726s) [upload=531550]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 21:58:45.272782726 (606h29m45.143865772s) [upload=531550]: Done uploading segments... Dec 30 21:58:45.272806352 (606h29m45.143889387s) [upload=531550]: Upload finished. err: <nil> Duration: 5.966076599s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....X..X...X.........X.........................X...........] duration: 5.927595483s trace: https://jaeger.dev.storj.io/trace/57a7ab98e7eb0d3f FAIL (2025-12-30 22:06:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 21:58:42.298454527 (606h29m42.169537592s) [upload=531550]: Starting upload Dec 30 21:58:42.298523438 (606h29m42.169606473s) [upload=531550]: Done starting upload Dec 30 21:58:42.298568774 (606h29m42.169651809s) [upload=531550]: Uploading segments... Dec 30 21:58:42.298697850 (606h29m42.169780885s) [upload=531550]: Got next segment. Inline: false Dec 30 21:58:42.298728649 (606h29m42.169811694s) [upload=531550]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 21:58:42.298733227 (606h29m42.169816262s) [upload=531550]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 21:58:42.545734213 (606h29m42.416817288s) [seg_pos={0 0}, upload=531550]: Begin upload segment... Dec 30 21:58:42.545786032 (606h29m42.416869078s) [seg_pos={0 0}, upload=531550]: Done begin upload segment. Dec 30 21:58:42.545855104 (606h29m42.416938149s) [noise=true , address=145.79.227.23:4547 , piece_id=LYGUMTAP7OPJWGRVFY65GUZFE4BBTV73JTG657DR5OM26LA7UFZA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.545897384 (606h29m42.416980449s) [noise=true , address=145.79.227.49:4339 , piece_id=WBBPZK2VY2QVRTQNXEKQ7TRYPWBDYS2TGTKEDAAXQ2AT4TUQXTQQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.546757752 (606h29m42.417840817s) [noise=true , address=45.91.116.12:4225 , piece_id=LLYEKE4TJ54KECLAC7KFUV44IVHYCYLF45I43G5V3DPV5QDBUOIQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.546816995 (606h29m42.41790004s) [noise=true , address=145.79.227.40:4050 , piece_id=LA7KLHHWRNHULLPAZWQ6MLS76SLN3BONYXZJXCUHGFC5JZ4SQRIA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.547216927 (606h29m42.418299962s) [noise=true , address=145.79.227.15:4876 , piece_id=L2OIJAB6OOANT2EPFNVNLTMYCH334JGFBFASD44EHOAWTAHKUV5Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.547615996 (606h29m42.418699042s) [noise=true , address=145.79.227.17:4971 , piece_id=DYZUALBL3V75TKGK57LOJABWVCOD5UVS3WH74I55OVUDGRQUA3QQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.547841055 (606h29m42.4189241s) [noise=true , address=77.90.31.21:4877 , piece_id=BD2FEFNAGYWZYDA2IMOVHA7JE66EXSF3A3PL54Q2DUOJFG5ALG4Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548023322 (606h29m42.419106357s) [noise=true , address=45.91.116.14:4190 , piece_id=HHU5YZBTJEEWYVST6XQCKGWKRLRQIIR2UQTQ27RLK53BNAVOSHFQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548380923 (606h29m42.419463968s) [noise=true , address=45.91.116.11:4120 , piece_id=FZ5L6FM7IHX65EVFGVXXHEF43XQ4EBUH4C7VZUSTH2MVWLNR5M2Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548560475 (606h29m42.41964351s) [noise=true , address=145.79.227.39:4013 , piece_id=I4XP47SFJICIYIPEN4E7UC2HXRXH4IXVJYGHXTZBCUESAHNFQXJA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548719268 (606h29m42.419802313s) [noise=true , address=145.79.227.31:4797 , piece_id=S5YBLITONGU6RHPQENP2TCVVWOHEMXPMYXLNYWBZNXWMFI26PVUA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549024107 (606h29m42.420107152s) [noise=true , address=145.79.227.7:4294 , piece_id=C6N5OIRLV3LLSCEIOGQJKWPKENGZYNAPY6V2AYZNKUMACBP5SLWQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549197708 (606h29m42.420280753s) [noise=true , address=145.79.227.16:4910 , piece_id=O4YO367LEVZSRI5E75YEZ7W7YI2KGKKQDXCOHQ3LALE4WUAPQVHQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549506406 (606h29m42.420589441s) [noise=true , address=145.79.227.12:4765 , piece_id=4ZF5YQP4Z5HBTX5YWUDHZV55Q47WASRLMV6ZYJXJRL2QBAVQQM4Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549971591 (606h29m42.421054626s) [noise=true , address=145.79.227.32:4835 , piece_id=CN4NMCFNYKOCG2JCIEXJBL2PLEYWD5THCZB4WQDURIECFVHLLX6Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.551571588 (606h29m42.422654623s) [noise=true , address=145.79.227.24:4581 , piece_id=Y6GBEOR7DQX2H6LBHXYFOXRBX3GP7AKPYVGTQNMUEXKPXLONUUGQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:44.247717212 (606h29m44.118800248s) [upload=531550]: Next returned nil segment Dec 30 21:58:44.247730348 (606h29m44.118813383s) [upload=531550]: Waiting for error group managing segments... Dec 30 21:58:44.532148720 (606h29m44.403231755s) [noise=true , address=145.79.227.31:4797 , piece_id=S5YBLITONGU6RHPQENP2TCVVWOHEMXPMYXLNYWBZNXWMFI26PVUA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.534655293 (606h29m44.405738328s) [noise=true , address=145.79.227.39:4013 , piece_id=I4XP47SFJICIYIPEN4E7UC2HXRXH4IXVJYGHXTZBCUESAHNFQXJA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.544709427 (606h29m44.415792462s) [noise=true , address=145.79.227.12:4765 , piece_id=4ZF5YQP4Z5HBTX5YWUDHZV55Q47WASRLMV6ZYJXJRL2QBAVQQM4Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.545822306 (606h29m44.416905342s) [noise=true , address=145.79.227.7:4294 , piece_id=C6N5OIRLV3LLSCEIOGQJKWPKENGZYNAPY6V2AYZNKUMACBP5SLWQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.549465162 (606h29m44.420548198s) [noise=true , address=145.79.227.24:4581 , piece_id=Y6GBEOR7DQX2H6LBHXYFOXRBX3GP7AKPYVGTQNMUEXKPXLONUUGQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.556935928 (606h29m44.428018964s) [noise=true , address=145.79.227.15:4876 , piece_id=L2OIJAB6OOANT2EPFNVNLTMYCH334JGFBFASD44EHOAWTAHKUV5Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.561283296 (606h29m44.432366331s) [noise=true , address=145.79.227.16:4910 , piece_id=O4YO367LEVZSRI5E75YEZ7W7YI2KGKKQDXCOHQ3LALE4WUAPQVHQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.614194264 (606h29m44.485277309s) [noise=true , address=145.79.227.17:4971 , piece_id=DYZUALBL3V75TKGK57LOJABWVCOD5UVS3WH74I55OVUDGRQUA3QQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.615630689 (606h29m44.486713724s) [noise=true , address=145.79.227.32:4835 , piece_id=CN4NMCFNYKOCG2JCIEXJBL2PLEYWD5THCZB4WQDURIECFVHLLX6Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.639672077 (606h29m44.510755122s) [noise=true , address=145.79.227.23:4547 , piece_id=LYGUMTAP7OPJWGRVFY65GUZFE4BBTV73JTG657DR5OM26LA7UFZA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.699618771 (606h29m44.570701817s) [noise=true , address=77.90.31.21:4877 , piece_id=BD2FEFNAGYWZYDA2IMOVHA7JE66EXSF3A3PL54Q2DUOJFG5ALG4Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.841535416 (606h29m44.712618462s) [noise=true , address=45.91.116.14:4190 , piece_id=HHU5YZBTJEEWYVST6XQCKGWKRLRQIIR2UQTQ27RLK53BNAVOSHFQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.974660841 (606h29m44.845743886s) [noise=true , address=45.91.116.12:4225 , piece_id=LLYEKE4TJ54KECLAC7KFUV44IVHYCYLF45I43G5V3DPV5QDBUOIQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.974681401 (606h29m44.845764436s) [seg_pos={0 0}, upload=531550]: Segment reached optimal threshold of 13 pieces. Dec 30 21:58:44.974855342 (606h29m44.845938377s) [noise=true , address=145.79.227.49:4339 , piece_id=WBBPZK2VY2QVRTQNXEKQ7TRYPWBDYS2TGTKEDAAXQ2AT4TUQXTQQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 30 21:58:44.974861423 (606h29m44.845944458s) [noise=true , address=145.79.227.49:4339 , piece_id=WBBPZK2VY2QVRTQNXEKQ7TRYPWBDYS2TGTKEDAAXQ2AT4TUQXTQQ, seg_pos={0 0}, upload=531550]: Long tail cancellation due to optimal threshold Dec 30 21:58:44.974968166 (606h29m44.846051201s) [noise=true , address=145.79.227.40:4050 , piece_id=LA7KLHHWRNHULLPAZWQ6MLS76SLN3BONYXZJXCUHGFC5JZ4SQRIA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 30 21:58:44.974973627 (606h29m44.846056662s) [noise=true , address=145.79.227.40:4050 , piece_id=LA7KLHHWRNHULLPAZWQ6MLS76SLN3BONYXZJXCUHGFC5JZ4SQRIA, seg_pos={0 0}, upload=531550]: Long tail cancellation due to optimal threshold Dec 30 21:58:44.975076152 (606h29m44.846159187s) [noise=true , address=45.91.116.11:4120 , piece_id=FZ5L6FM7IHX65EVFGVXXHEF43XQ4EBUH4C7VZUSTH2MVWLNR5M2Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 30 21:58:44.975083497 (606h29m44.846166532s) [noise=true , address=45.91.116.11:4120 , piece_id=FZ5L6FM7IHX65EVFGVXXHEF43XQ4EBUH4C7VZUSTH2MVWLNR5M2Q, seg_pos={0 0}, upload=531550]: Long tail cancellation due to optimal threshold Dec 30 21:58:44.975096711 (606h29m44.846179747s) [seg_pos={0 0}, upload=531550]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 21:58:44.975160703 (606h29m44.846243739s) [upload=531550]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 21:58:44.975164691 (606h29m44.846247726s) [upload=531550]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 21:58:45.272782726 (606h29m45.143865772s) [upload=531550]: Done uploading segments... Dec 30 21:58:45.272806352 (606h29m45.143889387s) [upload=531550]: Upload finished. err: <nil> Duration: 5.92761999s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [......XX......X.........X...................................] duration: 7.66115422s trace: https://jaeger.dev.storj.io/trace/5cdd5ba7b1cb22c7 FAIL (2025-12-30 22:06:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 21:37:03.355296401 (606h8m3.226379446s) [noise=true , address=145.79.227.22:4543 , piece_id=WWCZ6SEEXQ2KUJ32AGTIJNMSJROLYSJ4XQGBQJC4Q5DACZVA65RQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.355839374 (606h8m3.22692241s) [noise=true , address=145.79.227.35:4492 , piece_id=HCKFSQMJYEP6B5H4OX5AZ5TWP6Z4H4Z6WA2T7Z2H5KNZ4SDJRMUQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.356356930 (606h8m3.227439965s) [noise=true , address=45.91.116.13:4182 , piece_id=BYAYNP4JAAN3AW2LYSK74WJE4BRWUR5Z6K67MTLK6A73KEJ35WEQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.356675316 (606h8m3.227758361s) [noise=true , address=77.90.31.12:4612 , piece_id=63TS3SUEFUE6PORZ4W76OXRWFL7SBN773UGGY3DU7JPBAW66N7UA, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.356992732 (606h8m3.228075767s) [noise=true , address=64.49.13.140:4507 , piece_id=E7T6WL45F6GHECCRANX6AIKWFRZ77TURRWORYEYI43MET2WG7SRQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.357330644 (606h8m3.228413679s) [noise=true , address=77.90.31.21:4886 , piece_id=ZQRCPTVC3WFVQPZ36AU6KJW7PD3T557CXYQIICLDGOKK6JCPVZ5Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.357663649 (606h8m3.228746684s) [noise=true , address=145.79.227.49:4367 , piece_id=R2T2ABLXBPDVP3ZVA2TAYMCLQQ7JQUVQYCTHCKGMFBRVWCDAO4PQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.359827449 (606h8m3.230910484s) [noise=true , address=145.79.227.34:4918 , piece_id=HUMOHOT3HOYCBJ23WXQXWU3X6C6JQTRLLHTD4RQLCQTCB3HXI7EQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.360086692 (606h8m3.231169727s) [noise=true , address=145.79.227.7:4320 , piece_id=MAICNTHGLZ6W27HXDOOZXOBRGQGEXNSA4BJT2T5D5PHRVV6ZMX2Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.361622167 (606h8m3.232705212s) [noise=true , address=145.79.227.33:4888 , piece_id=I4426P3VETHPOL5WHWJEGVYRXJULTCNDX3WSUTS3XZVY36Q2N44Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.362965514 (606h8m3.234048559s) [noise=true , address=45.91.116.11:4133 , piece_id=NEW7ZOVP5E7HYGYD4LG5UG2MDLAQW2OBRTNZDFIYO4CQYMVAWRPQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.364308883 (606h8m3.235391928s) [noise=true , address=145.79.227.26:4663 , piece_id=DYEFQZL4X74QR2LAYURT5ZTR2IWIDB55VLBQOGZM36MA54P2SWJA, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.365734507 (606h8m3.236817542s) [noise=true , address=145.79.227.40:4059 , piece_id=7WSUBR3C3PWVBCWD6KRGSD6MPX4IH2YQOAVO4J4SWZORNHGS2IPQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.366982764 (606h8m3.238065799s) [noise=true , address=145.79.227.29:4973 , piece_id=5OROXELR55A47BFPSJJEWVQU4YW6CAOK6LMEHCINM4FN3OVI2Y6Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.368390294 (606h8m3.239473339s) [noise=true , address=77.90.31.19:4852 , piece_id=2NYO63GCBGCYBFYTMUGNF2NG5FSJOD3AOUOD2FZZIWDINK4X6DJQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:04.736295138 (606h8m4.607378183s) [upload=531240]: Next returned nil segment Dec 30 21:37:04.736311850 (606h8m4.607394885s) [upload=531240]: Waiting for error group managing segments... Dec 30 21:37:05.104989591 (606h8m4.976072636s) [noise=true , address=64.49.13.140:4507 , piece_id=E7T6WL45F6GHECCRANX6AIKWFRZ77TURRWORYEYI43MET2WG7SRQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.208809748 (606h8m5.079892793s) [noise=true , address=145.79.227.22:4543 , piece_id=WWCZ6SEEXQ2KUJ32AGTIJNMSJROLYSJ4XQGBQJC4Q5DACZVA65RQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.441125389 (606h8m5.312208444s) [noise=true , address=77.90.31.19:4852 , piece_id=2NYO63GCBGCYBFYTMUGNF2NG5FSJOD3AOUOD2FZZIWDINK4X6DJQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.483630904 (606h8m5.35471394s) [noise=true , address=77.90.31.12:4612 , piece_id=63TS3SUEFUE6PORZ4W76OXRWFL7SBN773UGGY3DU7JPBAW66N7UA, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.494478811 (606h8m5.365561846s) [noise=true , address=77.90.31.11:4559 , piece_id=3DYD4CNPJQN3S4EYZJGREQTTZJDSL2QTC3OT6G2VFUGMLSVCZUFA, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.497452683 (606h8m5.368535729s) [noise=true , address=77.90.31.17:4769 , piece_id=QXDYESBKGXRY7BCFYYF2O7OXS7Y4Z4KUMJB6AXZAWX735SYU56GQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.661529377 (606h8m5.532612412s) [noise=true , address=45.91.116.13:4182 , piece_id=BYAYNP4JAAN3AW2LYSK74WJE4BRWUR5Z6K67MTLK6A73KEJ35WEQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.729773112 (606h8m5.600856157s) [noise=true , address=45.91.116.11:4133 , piece_id=NEW7ZOVP5E7HYGYD4LG5UG2MDLAQW2OBRTNZDFIYO4CQYMVAWRPQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.940743308 (606h8m5.811826353s) [noise=true , address=77.90.31.21:4886 , piece_id=ZQRCPTVC3WFVQPZ36AU6KJW7PD3T557CXYQIICLDGOKK6JCPVZ5Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.084240044 (606h8m5.95532309s) [noise=true , address=145.79.227.30:4784 , piece_id=DSXUDNYCWO5F6IYXMIVJNEASYXIZ7EO7RKAS7QRUYK5OU7GOOB5Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.121095032 (606h8m5.992178067s) [noise=true , address=145.79.227.40:4059 , piece_id=7WSUBR3C3PWVBCWD6KRGSD6MPX4IH2YQOAVO4J4SWZORNHGS2IPQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.169195983 (606h8m6.040279028s) [noise=true , address=145.79.227.49:4367 , piece_id=R2T2ABLXBPDVP3ZVA2TAYMCLQQ7JQUVQYCTHCKGMFBRVWCDAO4PQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.169380634 (606h8m6.040463679s) [noise=true , address=145.79.227.33:4888 , piece_id=I4426P3VETHPOL5WHWJEGVYRXJULTCNDX3WSUTS3XZVY36Q2N44Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.179060175 (606h8m6.050143211s) [noise=true , address=145.79.227.29:4973 , piece_id=5OROXELR55A47BFPSJJEWVQU4YW6CAOK6LMEHCINM4FN3OVI2Y6Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.179080424 (606h8m6.050163459s) [seg_pos={0 0}, upload=531240]: Segment reached optimal threshold of 14 pieces. Dec 30 21:37:06.179262801 (606h8m6.050345837s) [noise=true , address=145.79.227.26:4663 , piece_id=DYEFQZL4X74QR2LAYURT5ZTR2IWIDB55VLBQOGZM36MA54P2SWJA, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SwpAaXTA3BqUvwubzmJkvwokdz6SSpT9t8HPtQcXXJxmcgfm2): protocol: write tcp 188.166.225.170:43184->145.79.227.26:4663: use of closed network connection; write tcp 188.166.225.170:43184->145.79.227.26:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43184->145.79.227.26:4663: use of closed network connection Dec 30 21:37:06.179268703 (606h8m6.050351748s) [noise=true , address=145.79.227.26:4663 , piece_id=DYEFQZL4X74QR2LAYURT5ZTR2IWIDB55VLBQOGZM36MA54P2SWJA, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179363724 (606h8m6.050446769s) [noise=true , address=145.79.227.7:4320 , piece_id=MAICNTHGLZ6W27HXDOOZXOBRGQGEXNSA4BJT2T5D5PHRVV6ZMX2Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126rEgnQw1w7hsFpRqQP9nLaumdBfdtkgYjdw6WE6nXrxk5kj4B): protocol: write tcp 188.166.225.170:45228->145.79.227.7:4320: use of closed network connection; write tcp 188.166.225.170:45228->145.79.227.7:4320: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45228->145.79.227.7:4320: use of closed network connection Dec 30 21:37:06.179369525 (606h8m6.05045257s) [noise=true , address=145.79.227.7:4320 , piece_id=MAICNTHGLZ6W27HXDOOZXOBRGQGEXNSA4BJT2T5D5PHRVV6ZMX2Q, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179532996 (606h8m6.050616061s) [noise=true , address=145.79.227.34:4918 , piece_id=HUMOHOT3HOYCBJ23WXQXWU3X6C6JQTRLLHTD4RQLCQTCB3HXI7EQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF Dec 30 21:37:06.179572371 (606h8m6.050655406s) [noise=true , address=145.79.227.34:4918 , piece_id=HUMOHOT3HOYCBJ23WXQXWU3X6C6JQTRLLHTD4RQLCQTCB3HXI7EQ, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179638587 (606h8m6.050721622s) [noise=true , address=145.79.227.35:4492 , piece_id=HCKFSQMJYEP6B5H4OX5AZ5TWP6Z4H4Z6WA2T7Z2H5KNZ4SDJRMUQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1roa5GGYmqCX2AxSrqFgqyV2CtBEYZavuGTseiwDy4pjkT6bip): protocol: expected piece hash; context canceled; EOF Dec 30 21:37:06.179647855 (606h8m6.05073089s) [noise=true , address=145.79.227.35:4492 , piece_id=HCKFSQMJYEP6B5H4OX5AZ5TWP6Z4H4Z6WA2T7Z2H5KNZ4SDJRMUQ, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179666641 (606h8m6.050749686s) [seg_pos={0 0}, upload=531240]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 21:37:06.179742766 (606h8m6.050825811s) [upload=531240]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 21:37:06.179747545 (606h8m6.05083059s) [upload=531240]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 21:37:06.490986034 (606h8m6.362069079s) [upload=531240]: Done uploading segments... Dec 30 21:37:06.491006261 (606h8m6.362089307s) [upload=531240]: Upload finished. err: <nil> Duration: 7.661176964s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................................X................] duration: 3.607773504s trace: https://jaeger.dev.storj.io/trace/22a47ec27f9a101c FAIL (2025-12-30 22:06:03 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.607803071s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......XX......X.........X..................................] duration: 15.391326153s trace: https://jaeger.dev.storj.io/trace/5cdd5ba7b1cb22c7 FAIL (2025-12-30 22:05:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 21:37:03.355296401 (606h8m3.226379446s) [noise=true , address=145.79.227.22:4543 , piece_id=WWCZ6SEEXQ2KUJ32AGTIJNMSJROLYSJ4XQGBQJC4Q5DACZVA65RQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.355839374 (606h8m3.22692241s) [noise=true , address=145.79.227.35:4492 , piece_id=HCKFSQMJYEP6B5H4OX5AZ5TWP6Z4H4Z6WA2T7Z2H5KNZ4SDJRMUQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.356356930 (606h8m3.227439965s) [noise=true , address=45.91.116.13:4182 , piece_id=BYAYNP4JAAN3AW2LYSK74WJE4BRWUR5Z6K67MTLK6A73KEJ35WEQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.356675316 (606h8m3.227758361s) [noise=true , address=77.90.31.12:4612 , piece_id=63TS3SUEFUE6PORZ4W76OXRWFL7SBN773UGGY3DU7JPBAW66N7UA, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.356992732 (606h8m3.228075767s) [noise=true , address=64.49.13.140:4507 , piece_id=E7T6WL45F6GHECCRANX6AIKWFRZ77TURRWORYEYI43MET2WG7SRQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.357330644 (606h8m3.228413679s) [noise=true , address=77.90.31.21:4886 , piece_id=ZQRCPTVC3WFVQPZ36AU6KJW7PD3T557CXYQIICLDGOKK6JCPVZ5Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.357663649 (606h8m3.228746684s) [noise=true , address=145.79.227.49:4367 , piece_id=R2T2ABLXBPDVP3ZVA2TAYMCLQQ7JQUVQYCTHCKGMFBRVWCDAO4PQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.359827449 (606h8m3.230910484s) [noise=true , address=145.79.227.34:4918 , piece_id=HUMOHOT3HOYCBJ23WXQXWU3X6C6JQTRLLHTD4RQLCQTCB3HXI7EQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.360086692 (606h8m3.231169727s) [noise=true , address=145.79.227.7:4320 , piece_id=MAICNTHGLZ6W27HXDOOZXOBRGQGEXNSA4BJT2T5D5PHRVV6ZMX2Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.361622167 (606h8m3.232705212s) [noise=true , address=145.79.227.33:4888 , piece_id=I4426P3VETHPOL5WHWJEGVYRXJULTCNDX3WSUTS3XZVY36Q2N44Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.362965514 (606h8m3.234048559s) [noise=true , address=45.91.116.11:4133 , piece_id=NEW7ZOVP5E7HYGYD4LG5UG2MDLAQW2OBRTNZDFIYO4CQYMVAWRPQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.364308883 (606h8m3.235391928s) [noise=true , address=145.79.227.26:4663 , piece_id=DYEFQZL4X74QR2LAYURT5ZTR2IWIDB55VLBQOGZM36MA54P2SWJA, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.365734507 (606h8m3.236817542s) [noise=true , address=145.79.227.40:4059 , piece_id=7WSUBR3C3PWVBCWD6KRGSD6MPX4IH2YQOAVO4J4SWZORNHGS2IPQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.366982764 (606h8m3.238065799s) [noise=true , address=145.79.227.29:4973 , piece_id=5OROXELR55A47BFPSJJEWVQU4YW6CAOK6LMEHCINM4FN3OVI2Y6Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.368390294 (606h8m3.239473339s) [noise=true , address=77.90.31.19:4852 , piece_id=2NYO63GCBGCYBFYTMUGNF2NG5FSJOD3AOUOD2FZZIWDINK4X6DJQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:04.736295138 (606h8m4.607378183s) [upload=531240]: Next returned nil segment Dec 30 21:37:04.736311850 (606h8m4.607394885s) [upload=531240]: Waiting for error group managing segments... Dec 30 21:37:05.104989591 (606h8m4.976072636s) [noise=true , address=64.49.13.140:4507 , piece_id=E7T6WL45F6GHECCRANX6AIKWFRZ77TURRWORYEYI43MET2WG7SRQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.208809748 (606h8m5.079892793s) [noise=true , address=145.79.227.22:4543 , piece_id=WWCZ6SEEXQ2KUJ32AGTIJNMSJROLYSJ4XQGBQJC4Q5DACZVA65RQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.441125389 (606h8m5.312208444s) [noise=true , address=77.90.31.19:4852 , piece_id=2NYO63GCBGCYBFYTMUGNF2NG5FSJOD3AOUOD2FZZIWDINK4X6DJQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.483630904 (606h8m5.35471394s) [noise=true , address=77.90.31.12:4612 , piece_id=63TS3SUEFUE6PORZ4W76OXRWFL7SBN773UGGY3DU7JPBAW66N7UA, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.494478811 (606h8m5.365561846s) [noise=true , address=77.90.31.11:4559 , piece_id=3DYD4CNPJQN3S4EYZJGREQTTZJDSL2QTC3OT6G2VFUGMLSVCZUFA, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.497452683 (606h8m5.368535729s) [noise=true , address=77.90.31.17:4769 , piece_id=QXDYESBKGXRY7BCFYYF2O7OXS7Y4Z4KUMJB6AXZAWX735SYU56GQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.661529377 (606h8m5.532612412s) [noise=true , address=45.91.116.13:4182 , piece_id=BYAYNP4JAAN3AW2LYSK74WJE4BRWUR5Z6K67MTLK6A73KEJ35WEQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.729773112 (606h8m5.600856157s) [noise=true , address=45.91.116.11:4133 , piece_id=NEW7ZOVP5E7HYGYD4LG5UG2MDLAQW2OBRTNZDFIYO4CQYMVAWRPQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.940743308 (606h8m5.811826353s) [noise=true , address=77.90.31.21:4886 , piece_id=ZQRCPTVC3WFVQPZ36AU6KJW7PD3T557CXYQIICLDGOKK6JCPVZ5Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.084240044 (606h8m5.95532309s) [noise=true , address=145.79.227.30:4784 , piece_id=DSXUDNYCWO5F6IYXMIVJNEASYXIZ7EO7RKAS7QRUYK5OU7GOOB5Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.121095032 (606h8m5.992178067s) [noise=true , address=145.79.227.40:4059 , piece_id=7WSUBR3C3PWVBCWD6KRGSD6MPX4IH2YQOAVO4J4SWZORNHGS2IPQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.169195983 (606h8m6.040279028s) [noise=true , address=145.79.227.49:4367 , piece_id=R2T2ABLXBPDVP3ZVA2TAYMCLQQ7JQUVQYCTHCKGMFBRVWCDAO4PQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.169380634 (606h8m6.040463679s) [noise=true , address=145.79.227.33:4888 , piece_id=I4426P3VETHPOL5WHWJEGVYRXJULTCNDX3WSUTS3XZVY36Q2N44Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.179060175 (606h8m6.050143211s) [noise=true , address=145.79.227.29:4973 , piece_id=5OROXELR55A47BFPSJJEWVQU4YW6CAOK6LMEHCINM4FN3OVI2Y6Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.179080424 (606h8m6.050163459s) [seg_pos={0 0}, upload=531240]: Segment reached optimal threshold of 14 pieces. Dec 30 21:37:06.179262801 (606h8m6.050345837s) [noise=true , address=145.79.227.26:4663 , piece_id=DYEFQZL4X74QR2LAYURT5ZTR2IWIDB55VLBQOGZM36MA54P2SWJA, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SwpAaXTA3BqUvwubzmJkvwokdz6SSpT9t8HPtQcXXJxmcgfm2): protocol: write tcp 188.166.225.170:43184->145.79.227.26:4663: use of closed network connection; write tcp 188.166.225.170:43184->145.79.227.26:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43184->145.79.227.26:4663: use of closed network connection Dec 30 21:37:06.179268703 (606h8m6.050351748s) [noise=true , address=145.79.227.26:4663 , piece_id=DYEFQZL4X74QR2LAYURT5ZTR2IWIDB55VLBQOGZM36MA54P2SWJA, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179363724 (606h8m6.050446769s) [noise=true , address=145.79.227.7:4320 , piece_id=MAICNTHGLZ6W27HXDOOZXOBRGQGEXNSA4BJT2T5D5PHRVV6ZMX2Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126rEgnQw1w7hsFpRqQP9nLaumdBfdtkgYjdw6WE6nXrxk5kj4B): protocol: write tcp 188.166.225.170:45228->145.79.227.7:4320: use of closed network connection; write tcp 188.166.225.170:45228->145.79.227.7:4320: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45228->145.79.227.7:4320: use of closed network connection Dec 30 21:37:06.179369525 (606h8m6.05045257s) [noise=true , address=145.79.227.7:4320 , piece_id=MAICNTHGLZ6W27HXDOOZXOBRGQGEXNSA4BJT2T5D5PHRVV6ZMX2Q, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179532996 (606h8m6.050616061s) [noise=true , address=145.79.227.34:4918 , piece_id=HUMOHOT3HOYCBJ23WXQXWU3X6C6JQTRLLHTD4RQLCQTCB3HXI7EQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF Dec 30 21:37:06.179572371 (606h8m6.050655406s) [noise=true , address=145.79.227.34:4918 , piece_id=HUMOHOT3HOYCBJ23WXQXWU3X6C6JQTRLLHTD4RQLCQTCB3HXI7EQ, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179638587 (606h8m6.050721622s) [noise=true , address=145.79.227.35:4492 , piece_id=HCKFSQMJYEP6B5H4OX5AZ5TWP6Z4H4Z6WA2T7Z2H5KNZ4SDJRMUQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1roa5GGYmqCX2AxSrqFgqyV2CtBEYZavuGTseiwDy4pjkT6bip): protocol: expected piece hash; context canceled; EOF Dec 30 21:37:06.179647855 (606h8m6.05073089s) [noise=true , address=145.79.227.35:4492 , piece_id=HCKFSQMJYEP6B5H4OX5AZ5TWP6Z4H4Z6WA2T7Z2H5KNZ4SDJRMUQ, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179666641 (606h8m6.050749686s) [seg_pos={0 0}, upload=531240]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 21:37:06.179742766 (606h8m6.050825811s) [upload=531240]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 21:37:06.179747545 (606h8m6.05083059s) [upload=531240]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 21:37:06.490986034 (606h8m6.362069079s) [upload=531240]: Done uploading segments... Dec 30 21:37:06.491006261 (606h8m6.362089307s) [upload=531240]: Upload finished. err: <nil> Duration: 15.391352294s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [......X..X...X.........X.........................X..........] duration: 5.738212108s trace: https://jaeger.dev.storj.io/trace/57a7ab98e7eb0d3f FAIL (2025-12-30 22:05:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 21:58:42.298454527 (606h29m42.169537592s) [upload=531550]: Starting upload Dec 30 21:58:42.298523438 (606h29m42.169606473s) [upload=531550]: Done starting upload Dec 30 21:58:42.298568774 (606h29m42.169651809s) [upload=531550]: Uploading segments... Dec 30 21:58:42.298697850 (606h29m42.169780885s) [upload=531550]: Got next segment. Inline: false Dec 30 21:58:42.298728649 (606h29m42.169811694s) [upload=531550]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 21:58:42.298733227 (606h29m42.169816262s) [upload=531550]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 21:58:42.545734213 (606h29m42.416817288s) [seg_pos={0 0}, upload=531550]: Begin upload segment... Dec 30 21:58:42.545786032 (606h29m42.416869078s) [seg_pos={0 0}, upload=531550]: Done begin upload segment. Dec 30 21:58:42.545855104 (606h29m42.416938149s) [noise=true , address=145.79.227.23:4547 , piece_id=LYGUMTAP7OPJWGRVFY65GUZFE4BBTV73JTG657DR5OM26LA7UFZA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.545897384 (606h29m42.416980449s) [noise=true , address=145.79.227.49:4339 , piece_id=WBBPZK2VY2QVRTQNXEKQ7TRYPWBDYS2TGTKEDAAXQ2AT4TUQXTQQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.546757752 (606h29m42.417840817s) [noise=true , address=45.91.116.12:4225 , piece_id=LLYEKE4TJ54KECLAC7KFUV44IVHYCYLF45I43G5V3DPV5QDBUOIQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.546816995 (606h29m42.41790004s) [noise=true , address=145.79.227.40:4050 , piece_id=LA7KLHHWRNHULLPAZWQ6MLS76SLN3BONYXZJXCUHGFC5JZ4SQRIA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.547216927 (606h29m42.418299962s) [noise=true , address=145.79.227.15:4876 , piece_id=L2OIJAB6OOANT2EPFNVNLTMYCH334JGFBFASD44EHOAWTAHKUV5Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.547615996 (606h29m42.418699042s) [noise=true , address=145.79.227.17:4971 , piece_id=DYZUALBL3V75TKGK57LOJABWVCOD5UVS3WH74I55OVUDGRQUA3QQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.547841055 (606h29m42.4189241s) [noise=true , address=77.90.31.21:4877 , piece_id=BD2FEFNAGYWZYDA2IMOVHA7JE66EXSF3A3PL54Q2DUOJFG5ALG4Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548023322 (606h29m42.419106357s) [noise=true , address=45.91.116.14:4190 , piece_id=HHU5YZBTJEEWYVST6XQCKGWKRLRQIIR2UQTQ27RLK53BNAVOSHFQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548380923 (606h29m42.419463968s) [noise=true , address=45.91.116.11:4120 , piece_id=FZ5L6FM7IHX65EVFGVXXHEF43XQ4EBUH4C7VZUSTH2MVWLNR5M2Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548560475 (606h29m42.41964351s) [noise=true , address=145.79.227.39:4013 , piece_id=I4XP47SFJICIYIPEN4E7UC2HXRXH4IXVJYGHXTZBCUESAHNFQXJA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548719268 (606h29m42.419802313s) [noise=true , address=145.79.227.31:4797 , piece_id=S5YBLITONGU6RHPQENP2TCVVWOHEMXPMYXLNYWBZNXWMFI26PVUA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549024107 (606h29m42.420107152s) [noise=true , address=145.79.227.7:4294 , piece_id=C6N5OIRLV3LLSCEIOGQJKWPKENGZYNAPY6V2AYZNKUMACBP5SLWQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549197708 (606h29m42.420280753s) [noise=true , address=145.79.227.16:4910 , piece_id=O4YO367LEVZSRI5E75YEZ7W7YI2KGKKQDXCOHQ3LALE4WUAPQVHQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549506406 (606h29m42.420589441s) [noise=true , address=145.79.227.12:4765 , piece_id=4ZF5YQP4Z5HBTX5YWUDHZV55Q47WASRLMV6ZYJXJRL2QBAVQQM4Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549971591 (606h29m42.421054626s) [noise=true , address=145.79.227.32:4835 , piece_id=CN4NMCFNYKOCG2JCIEXJBL2PLEYWD5THCZB4WQDURIECFVHLLX6Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.551571588 (606h29m42.422654623s) [noise=true , address=145.79.227.24:4581 , piece_id=Y6GBEOR7DQX2H6LBHXYFOXRBX3GP7AKPYVGTQNMUEXKPXLONUUGQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:44.247717212 (606h29m44.118800248s) [upload=531550]: Next returned nil segment Dec 30 21:58:44.247730348 (606h29m44.118813383s) [upload=531550]: Waiting for error group managing segments... Dec 30 21:58:44.532148720 (606h29m44.403231755s) [noise=true , address=145.79.227.31:4797 , piece_id=S5YBLITONGU6RHPQENP2TCVVWOHEMXPMYXLNYWBZNXWMFI26PVUA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.534655293 (606h29m44.405738328s) [noise=true , address=145.79.227.39:4013 , piece_id=I4XP47SFJICIYIPEN4E7UC2HXRXH4IXVJYGHXTZBCUESAHNFQXJA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.544709427 (606h29m44.415792462s) [noise=true , address=145.79.227.12:4765 , piece_id=4ZF5YQP4Z5HBTX5YWUDHZV55Q47WASRLMV6ZYJXJRL2QBAVQQM4Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.545822306 (606h29m44.416905342s) [noise=true , address=145.79.227.7:4294 , piece_id=C6N5OIRLV3LLSCEIOGQJKWPKENGZYNAPY6V2AYZNKUMACBP5SLWQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.549465162 (606h29m44.420548198s) [noise=true , address=145.79.227.24:4581 , piece_id=Y6GBEOR7DQX2H6LBHXYFOXRBX3GP7AKPYVGTQNMUEXKPXLONUUGQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.556935928 (606h29m44.428018964s) [noise=true , address=145.79.227.15:4876 , piece_id=L2OIJAB6OOANT2EPFNVNLTMYCH334JGFBFASD44EHOAWTAHKUV5Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.561283296 (606h29m44.432366331s) [noise=true , address=145.79.227.16:4910 , piece_id=O4YO367LEVZSRI5E75YEZ7W7YI2KGKKQDXCOHQ3LALE4WUAPQVHQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.614194264 (606h29m44.485277309s) [noise=true , address=145.79.227.17:4971 , piece_id=DYZUALBL3V75TKGK57LOJABWVCOD5UVS3WH74I55OVUDGRQUA3QQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.615630689 (606h29m44.486713724s) [noise=true , address=145.79.227.32:4835 , piece_id=CN4NMCFNYKOCG2JCIEXJBL2PLEYWD5THCZB4WQDURIECFVHLLX6Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.639672077 (606h29m44.510755122s) [noise=true , address=145.79.227.23:4547 , piece_id=LYGUMTAP7OPJWGRVFY65GUZFE4BBTV73JTG657DR5OM26LA7UFZA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.699618771 (606h29m44.570701817s) [noise=true , address=77.90.31.21:4877 , piece_id=BD2FEFNAGYWZYDA2IMOVHA7JE66EXSF3A3PL54Q2DUOJFG5ALG4Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.841535416 (606h29m44.712618462s) [noise=true , address=45.91.116.14:4190 , piece_id=HHU5YZBTJEEWYVST6XQCKGWKRLRQIIR2UQTQ27RLK53BNAVOSHFQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.974660841 (606h29m44.845743886s) [noise=true , address=45.91.116.12:4225 , piece_id=LLYEKE4TJ54KECLAC7KFUV44IVHYCYLF45I43G5V3DPV5QDBUOIQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.974681401 (606h29m44.845764436s) [seg_pos={0 0}, upload=531550]: Segment reached optimal threshold of 13 pieces. Dec 30 21:58:44.974855342 (606h29m44.845938377s) [noise=true , address=145.79.227.49:4339 , piece_id=WBBPZK2VY2QVRTQNXEKQ7TRYPWBDYS2TGTKEDAAXQ2AT4TUQXTQQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 30 21:58:44.974861423 (606h29m44.845944458s) [noise=true , address=145.79.227.49:4339 , piece_id=WBBPZK2VY2QVRTQNXEKQ7TRYPWBDYS2TGTKEDAAXQ2AT4TUQXTQQ, seg_pos={0 0}, upload=531550]: Long tail cancellation due to optimal threshold Dec 30 21:58:44.974968166 (606h29m44.846051201s) [noise=true , address=145.79.227.40:4050 , piece_id=LA7KLHHWRNHULLPAZWQ6MLS76SLN3BONYXZJXCUHGFC5JZ4SQRIA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 30 21:58:44.974973627 (606h29m44.846056662s) [noise=true , address=145.79.227.40:4050 , piece_id=LA7KLHHWRNHULLPAZWQ6MLS76SLN3BONYXZJXCUHGFC5JZ4SQRIA, seg_pos={0 0}, upload=531550]: Long tail cancellation due to optimal threshold Dec 30 21:58:44.975076152 (606h29m44.846159187s) [noise=true , address=45.91.116.11:4120 , piece_id=FZ5L6FM7IHX65EVFGVXXHEF43XQ4EBUH4C7VZUSTH2MVWLNR5M2Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 30 21:58:44.975083497 (606h29m44.846166532s) [noise=true , address=45.91.116.11:4120 , piece_id=FZ5L6FM7IHX65EVFGVXXHEF43XQ4EBUH4C7VZUSTH2MVWLNR5M2Q, seg_pos={0 0}, upload=531550]: Long tail cancellation due to optimal threshold Dec 30 21:58:44.975096711 (606h29m44.846179747s) [seg_pos={0 0}, upload=531550]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 21:58:44.975160703 (606h29m44.846243739s) [upload=531550]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 21:58:44.975164691 (606h29m44.846247726s) [upload=531550]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 21:58:45.272782726 (606h29m45.143865772s) [upload=531550]: Done uploading segments... Dec 30 21:58:45.272806352 (606h29m45.143889387s) [upload=531550]: Upload finished. err: <nil> Duration: 5.73822849s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......X..X...X.........X.........................X.........] duration: 27.376662441s trace: https://jaeger.dev.storj.io/trace/57a7ab98e7eb0d3f FAIL (2025-12-30 22:05:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 21:58:42.298454527 (606h29m42.169537592s) [upload=531550]: Starting upload Dec 30 21:58:42.298523438 (606h29m42.169606473s) [upload=531550]: Done starting upload Dec 30 21:58:42.298568774 (606h29m42.169651809s) [upload=531550]: Uploading segments... Dec 30 21:58:42.298697850 (606h29m42.169780885s) [upload=531550]: Got next segment. Inline: false Dec 30 21:58:42.298728649 (606h29m42.169811694s) [upload=531550]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 21:58:42.298733227 (606h29m42.169816262s) [upload=531550]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 21:58:42.545734213 (606h29m42.416817288s) [seg_pos={0 0}, upload=531550]: Begin upload segment... Dec 30 21:58:42.545786032 (606h29m42.416869078s) [seg_pos={0 0}, upload=531550]: Done begin upload segment. Dec 30 21:58:42.545855104 (606h29m42.416938149s) [noise=true , address=145.79.227.23:4547 , piece_id=LYGUMTAP7OPJWGRVFY65GUZFE4BBTV73JTG657DR5OM26LA7UFZA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.545897384 (606h29m42.416980449s) [noise=true , address=145.79.227.49:4339 , piece_id=WBBPZK2VY2QVRTQNXEKQ7TRYPWBDYS2TGTKEDAAXQ2AT4TUQXTQQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.546757752 (606h29m42.417840817s) [noise=true , address=45.91.116.12:4225 , piece_id=LLYEKE4TJ54KECLAC7KFUV44IVHYCYLF45I43G5V3DPV5QDBUOIQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.546816995 (606h29m42.41790004s) [noise=true , address=145.79.227.40:4050 , piece_id=LA7KLHHWRNHULLPAZWQ6MLS76SLN3BONYXZJXCUHGFC5JZ4SQRIA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.547216927 (606h29m42.418299962s) [noise=true , address=145.79.227.15:4876 , piece_id=L2OIJAB6OOANT2EPFNVNLTMYCH334JGFBFASD44EHOAWTAHKUV5Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.547615996 (606h29m42.418699042s) [noise=true , address=145.79.227.17:4971 , piece_id=DYZUALBL3V75TKGK57LOJABWVCOD5UVS3WH74I55OVUDGRQUA3QQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.547841055 (606h29m42.4189241s) [noise=true , address=77.90.31.21:4877 , piece_id=BD2FEFNAGYWZYDA2IMOVHA7JE66EXSF3A3PL54Q2DUOJFG5ALG4Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548023322 (606h29m42.419106357s) [noise=true , address=45.91.116.14:4190 , piece_id=HHU5YZBTJEEWYVST6XQCKGWKRLRQIIR2UQTQ27RLK53BNAVOSHFQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548380923 (606h29m42.419463968s) [noise=true , address=45.91.116.11:4120 , piece_id=FZ5L6FM7IHX65EVFGVXXHEF43XQ4EBUH4C7VZUSTH2MVWLNR5M2Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548560475 (606h29m42.41964351s) [noise=true , address=145.79.227.39:4013 , piece_id=I4XP47SFJICIYIPEN4E7UC2HXRXH4IXVJYGHXTZBCUESAHNFQXJA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548719268 (606h29m42.419802313s) [noise=true , address=145.79.227.31:4797 , piece_id=S5YBLITONGU6RHPQENP2TCVVWOHEMXPMYXLNYWBZNXWMFI26PVUA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549024107 (606h29m42.420107152s) [noise=true , address=145.79.227.7:4294 , piece_id=C6N5OIRLV3LLSCEIOGQJKWPKENGZYNAPY6V2AYZNKUMACBP5SLWQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549197708 (606h29m42.420280753s) [noise=true , address=145.79.227.16:4910 , piece_id=O4YO367LEVZSRI5E75YEZ7W7YI2KGKKQDXCOHQ3LALE4WUAPQVHQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549506406 (606h29m42.420589441s) [noise=true , address=145.79.227.12:4765 , piece_id=4ZF5YQP4Z5HBTX5YWUDHZV55Q47WASRLMV6ZYJXJRL2QBAVQQM4Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549971591 (606h29m42.421054626s) [noise=true , address=145.79.227.32:4835 , piece_id=CN4NMCFNYKOCG2JCIEXJBL2PLEYWD5THCZB4WQDURIECFVHLLX6Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.551571588 (606h29m42.422654623s) [noise=true , address=145.79.227.24:4581 , piece_id=Y6GBEOR7DQX2H6LBHXYFOXRBX3GP7AKPYVGTQNMUEXKPXLONUUGQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:44.247717212 (606h29m44.118800248s) [upload=531550]: Next returned nil segment Dec 30 21:58:44.247730348 (606h29m44.118813383s) [upload=531550]: Waiting for error group managing segments... Dec 30 21:58:44.532148720 (606h29m44.403231755s) [noise=true , address=145.79.227.31:4797 , piece_id=S5YBLITONGU6RHPQENP2TCVVWOHEMXPMYXLNYWBZNXWMFI26PVUA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.534655293 (606h29m44.405738328s) [noise=true , address=145.79.227.39:4013 , piece_id=I4XP47SFJICIYIPEN4E7UC2HXRXH4IXVJYGHXTZBCUESAHNFQXJA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.544709427 (606h29m44.415792462s) [noise=true , address=145.79.227.12:4765 , piece_id=4ZF5YQP4Z5HBTX5YWUDHZV55Q47WASRLMV6ZYJXJRL2QBAVQQM4Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.545822306 (606h29m44.416905342s) [noise=true , address=145.79.227.7:4294 , piece_id=C6N5OIRLV3LLSCEIOGQJKWPKENGZYNAPY6V2AYZNKUMACBP5SLWQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.549465162 (606h29m44.420548198s) [noise=true , address=145.79.227.24:4581 , piece_id=Y6GBEOR7DQX2H6LBHXYFOXRBX3GP7AKPYVGTQNMUEXKPXLONUUGQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.556935928 (606h29m44.428018964s) [noise=true , address=145.79.227.15:4876 , piece_id=L2OIJAB6OOANT2EPFNVNLTMYCH334JGFBFASD44EHOAWTAHKUV5Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.561283296 (606h29m44.432366331s) [noise=true , address=145.79.227.16:4910 , piece_id=O4YO367LEVZSRI5E75YEZ7W7YI2KGKKQDXCOHQ3LALE4WUAPQVHQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.614194264 (606h29m44.485277309s) [noise=true , address=145.79.227.17:4971 , piece_id=DYZUALBL3V75TKGK57LOJABWVCOD5UVS3WH74I55OVUDGRQUA3QQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.615630689 (606h29m44.486713724s) [noise=true , address=145.79.227.32:4835 , piece_id=CN4NMCFNYKOCG2JCIEXJBL2PLEYWD5THCZB4WQDURIECFVHLLX6Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.639672077 (606h29m44.510755122s) [noise=true , address=145.79.227.23:4547 , piece_id=LYGUMTAP7OPJWGRVFY65GUZFE4BBTV73JTG657DR5OM26LA7UFZA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.699618771 (606h29m44.570701817s) [noise=true , address=77.90.31.21:4877 , piece_id=BD2FEFNAGYWZYDA2IMOVHA7JE66EXSF3A3PL54Q2DUOJFG5ALG4Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.841535416 (606h29m44.712618462s) [noise=true , address=45.91.116.14:4190 , piece_id=HHU5YZBTJEEWYVST6XQCKGWKRLRQIIR2UQTQ27RLK53BNAVOSHFQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.974660841 (606h29m44.845743886s) [noise=true , address=45.91.116.12:4225 , piece_id=LLYEKE4TJ54KECLAC7KFUV44IVHYCYLF45I43G5V3DPV5QDBUOIQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.974681401 (606h29m44.845764436s) [seg_pos={0 0}, upload=531550]: Segment reached optimal threshold of 13 pieces. Dec 30 21:58:44.974855342 (606h29m44.845938377s) [noise=true , address=145.79.227.49:4339 , piece_id=WBBPZK2VY2QVRTQNXEKQ7TRYPWBDYS2TGTKEDAAXQ2AT4TUQXTQQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 30 21:58:44.974861423 (606h29m44.845944458s) [noise=true , address=145.79.227.49:4339 , piece_id=WBBPZK2VY2QVRTQNXEKQ7TRYPWBDYS2TGTKEDAAXQ2AT4TUQXTQQ, seg_pos={0 0}, upload=531550]: Long tail cancellation due to optimal threshold Dec 30 21:58:44.974968166 (606h29m44.846051201s) [noise=true , address=145.79.227.40:4050 , piece_id=LA7KLHHWRNHULLPAZWQ6MLS76SLN3BONYXZJXCUHGFC5JZ4SQRIA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 30 21:58:44.974973627 (606h29m44.846056662s) [noise=true , address=145.79.227.40:4050 , piece_id=LA7KLHHWRNHULLPAZWQ6MLS76SLN3BONYXZJXCUHGFC5JZ4SQRIA, seg_pos={0 0}, upload=531550]: Long tail cancellation due to optimal threshold Dec 30 21:58:44.975076152 (606h29m44.846159187s) [noise=true , address=45.91.116.11:4120 , piece_id=FZ5L6FM7IHX65EVFGVXXHEF43XQ4EBUH4C7VZUSTH2MVWLNR5M2Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 30 21:58:44.975083497 (606h29m44.846166532s) [noise=true , address=45.91.116.11:4120 , piece_id=FZ5L6FM7IHX65EVFGVXXHEF43XQ4EBUH4C7VZUSTH2MVWLNR5M2Q, seg_pos={0 0}, upload=531550]: Long tail cancellation due to optimal threshold Dec 30 21:58:44.975096711 (606h29m44.846179747s) [seg_pos={0 0}, upload=531550]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 21:58:44.975160703 (606h29m44.846243739s) [upload=531550]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 21:58:44.975164691 (606h29m44.846247726s) [upload=531550]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 21:58:45.272782726 (606h29m45.143865772s) [upload=531550]: Done uploading segments... Dec 30 21:58:45.272806352 (606h29m45.143889387s) [upload=531550]: Upload finished. err: <nil> Duration: 27.376688792s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............................................X...............] duration: 3.873861169s trace: https://jaeger.dev.storj.io/trace/22a47ec27f9a101c FAIL (2025-12-30 22:04:56 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.873889693s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [........XX......X.........X.................................] duration: 15.317844753s trace: https://jaeger.dev.storj.io/trace/5cdd5ba7b1cb22c7 FAIL (2025-12-30 22:04:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 21:37:03.355296401 (606h8m3.226379446s) [noise=true , address=145.79.227.22:4543 , piece_id=WWCZ6SEEXQ2KUJ32AGTIJNMSJROLYSJ4XQGBQJC4Q5DACZVA65RQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.355839374 (606h8m3.22692241s) [noise=true , address=145.79.227.35:4492 , piece_id=HCKFSQMJYEP6B5H4OX5AZ5TWP6Z4H4Z6WA2T7Z2H5KNZ4SDJRMUQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.356356930 (606h8m3.227439965s) [noise=true , address=45.91.116.13:4182 , piece_id=BYAYNP4JAAN3AW2LYSK74WJE4BRWUR5Z6K67MTLK6A73KEJ35WEQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.356675316 (606h8m3.227758361s) [noise=true , address=77.90.31.12:4612 , piece_id=63TS3SUEFUE6PORZ4W76OXRWFL7SBN773UGGY3DU7JPBAW66N7UA, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.356992732 (606h8m3.228075767s) [noise=true , address=64.49.13.140:4507 , piece_id=E7T6WL45F6GHECCRANX6AIKWFRZ77TURRWORYEYI43MET2WG7SRQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.357330644 (606h8m3.228413679s) [noise=true , address=77.90.31.21:4886 , piece_id=ZQRCPTVC3WFVQPZ36AU6KJW7PD3T557CXYQIICLDGOKK6JCPVZ5Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.357663649 (606h8m3.228746684s) [noise=true , address=145.79.227.49:4367 , piece_id=R2T2ABLXBPDVP3ZVA2TAYMCLQQ7JQUVQYCTHCKGMFBRVWCDAO4PQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.359827449 (606h8m3.230910484s) [noise=true , address=145.79.227.34:4918 , piece_id=HUMOHOT3HOYCBJ23WXQXWU3X6C6JQTRLLHTD4RQLCQTCB3HXI7EQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.360086692 (606h8m3.231169727s) [noise=true , address=145.79.227.7:4320 , piece_id=MAICNTHGLZ6W27HXDOOZXOBRGQGEXNSA4BJT2T5D5PHRVV6ZMX2Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.361622167 (606h8m3.232705212s) [noise=true , address=145.79.227.33:4888 , piece_id=I4426P3VETHPOL5WHWJEGVYRXJULTCNDX3WSUTS3XZVY36Q2N44Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.362965514 (606h8m3.234048559s) [noise=true , address=45.91.116.11:4133 , piece_id=NEW7ZOVP5E7HYGYD4LG5UG2MDLAQW2OBRTNZDFIYO4CQYMVAWRPQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.364308883 (606h8m3.235391928s) [noise=true , address=145.79.227.26:4663 , piece_id=DYEFQZL4X74QR2LAYURT5ZTR2IWIDB55VLBQOGZM36MA54P2SWJA, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.365734507 (606h8m3.236817542s) [noise=true , address=145.79.227.40:4059 , piece_id=7WSUBR3C3PWVBCWD6KRGSD6MPX4IH2YQOAVO4J4SWZORNHGS2IPQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.366982764 (606h8m3.238065799s) [noise=true , address=145.79.227.29:4973 , piece_id=5OROXELR55A47BFPSJJEWVQU4YW6CAOK6LMEHCINM4FN3OVI2Y6Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.368390294 (606h8m3.239473339s) [noise=true , address=77.90.31.19:4852 , piece_id=2NYO63GCBGCYBFYTMUGNF2NG5FSJOD3AOUOD2FZZIWDINK4X6DJQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:04.736295138 (606h8m4.607378183s) [upload=531240]: Next returned nil segment Dec 30 21:37:04.736311850 (606h8m4.607394885s) [upload=531240]: Waiting for error group managing segments... Dec 30 21:37:05.104989591 (606h8m4.976072636s) [noise=true , address=64.49.13.140:4507 , piece_id=E7T6WL45F6GHECCRANX6AIKWFRZ77TURRWORYEYI43MET2WG7SRQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.208809748 (606h8m5.079892793s) [noise=true , address=145.79.227.22:4543 , piece_id=WWCZ6SEEXQ2KUJ32AGTIJNMSJROLYSJ4XQGBQJC4Q5DACZVA65RQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.441125389 (606h8m5.312208444s) [noise=true , address=77.90.31.19:4852 , piece_id=2NYO63GCBGCYBFYTMUGNF2NG5FSJOD3AOUOD2FZZIWDINK4X6DJQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.483630904 (606h8m5.35471394s) [noise=true , address=77.90.31.12:4612 , piece_id=63TS3SUEFUE6PORZ4W76OXRWFL7SBN773UGGY3DU7JPBAW66N7UA, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.494478811 (606h8m5.365561846s) [noise=true , address=77.90.31.11:4559 , piece_id=3DYD4CNPJQN3S4EYZJGREQTTZJDSL2QTC3OT6G2VFUGMLSVCZUFA, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.497452683 (606h8m5.368535729s) [noise=true , address=77.90.31.17:4769 , piece_id=QXDYESBKGXRY7BCFYYF2O7OXS7Y4Z4KUMJB6AXZAWX735SYU56GQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.661529377 (606h8m5.532612412s) [noise=true , address=45.91.116.13:4182 , piece_id=BYAYNP4JAAN3AW2LYSK74WJE4BRWUR5Z6K67MTLK6A73KEJ35WEQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.729773112 (606h8m5.600856157s) [noise=true , address=45.91.116.11:4133 , piece_id=NEW7ZOVP5E7HYGYD4LG5UG2MDLAQW2OBRTNZDFIYO4CQYMVAWRPQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.940743308 (606h8m5.811826353s) [noise=true , address=77.90.31.21:4886 , piece_id=ZQRCPTVC3WFVQPZ36AU6KJW7PD3T557CXYQIICLDGOKK6JCPVZ5Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.084240044 (606h8m5.95532309s) [noise=true , address=145.79.227.30:4784 , piece_id=DSXUDNYCWO5F6IYXMIVJNEASYXIZ7EO7RKAS7QRUYK5OU7GOOB5Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.121095032 (606h8m5.992178067s) [noise=true , address=145.79.227.40:4059 , piece_id=7WSUBR3C3PWVBCWD6KRGSD6MPX4IH2YQOAVO4J4SWZORNHGS2IPQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.169195983 (606h8m6.040279028s) [noise=true , address=145.79.227.49:4367 , piece_id=R2T2ABLXBPDVP3ZVA2TAYMCLQQ7JQUVQYCTHCKGMFBRVWCDAO4PQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.169380634 (606h8m6.040463679s) [noise=true , address=145.79.227.33:4888 , piece_id=I4426P3VETHPOL5WHWJEGVYRXJULTCNDX3WSUTS3XZVY36Q2N44Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.179060175 (606h8m6.050143211s) [noise=true , address=145.79.227.29:4973 , piece_id=5OROXELR55A47BFPSJJEWVQU4YW6CAOK6LMEHCINM4FN3OVI2Y6Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.179080424 (606h8m6.050163459s) [seg_pos={0 0}, upload=531240]: Segment reached optimal threshold of 14 pieces. Dec 30 21:37:06.179262801 (606h8m6.050345837s) [noise=true , address=145.79.227.26:4663 , piece_id=DYEFQZL4X74QR2LAYURT5ZTR2IWIDB55VLBQOGZM36MA54P2SWJA, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SwpAaXTA3BqUvwubzmJkvwokdz6SSpT9t8HPtQcXXJxmcgfm2): protocol: write tcp 188.166.225.170:43184->145.79.227.26:4663: use of closed network connection; write tcp 188.166.225.170:43184->145.79.227.26:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43184->145.79.227.26:4663: use of closed network connection Dec 30 21:37:06.179268703 (606h8m6.050351748s) [noise=true , address=145.79.227.26:4663 , piece_id=DYEFQZL4X74QR2LAYURT5ZTR2IWIDB55VLBQOGZM36MA54P2SWJA, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179363724 (606h8m6.050446769s) [noise=true , address=145.79.227.7:4320 , piece_id=MAICNTHGLZ6W27HXDOOZXOBRGQGEXNSA4BJT2T5D5PHRVV6ZMX2Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126rEgnQw1w7hsFpRqQP9nLaumdBfdtkgYjdw6WE6nXrxk5kj4B): protocol: write tcp 188.166.225.170:45228->145.79.227.7:4320: use of closed network connection; write tcp 188.166.225.170:45228->145.79.227.7:4320: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45228->145.79.227.7:4320: use of closed network connection Dec 30 21:37:06.179369525 (606h8m6.05045257s) [noise=true , address=145.79.227.7:4320 , piece_id=MAICNTHGLZ6W27HXDOOZXOBRGQGEXNSA4BJT2T5D5PHRVV6ZMX2Q, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179532996 (606h8m6.050616061s) [noise=true , address=145.79.227.34:4918 , piece_id=HUMOHOT3HOYCBJ23WXQXWU3X6C6JQTRLLHTD4RQLCQTCB3HXI7EQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF Dec 30 21:37:06.179572371 (606h8m6.050655406s) [noise=true , address=145.79.227.34:4918 , piece_id=HUMOHOT3HOYCBJ23WXQXWU3X6C6JQTRLLHTD4RQLCQTCB3HXI7EQ, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179638587 (606h8m6.050721622s) [noise=true , address=145.79.227.35:4492 , piece_id=HCKFSQMJYEP6B5H4OX5AZ5TWP6Z4H4Z6WA2T7Z2H5KNZ4SDJRMUQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1roa5GGYmqCX2AxSrqFgqyV2CtBEYZavuGTseiwDy4pjkT6bip): protocol: expected piece hash; context canceled; EOF Dec 30 21:37:06.179647855 (606h8m6.05073089s) [noise=true , address=145.79.227.35:4492 , piece_id=HCKFSQMJYEP6B5H4OX5AZ5TWP6Z4H4Z6WA2T7Z2H5KNZ4SDJRMUQ, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179666641 (606h8m6.050749686s) [seg_pos={0 0}, upload=531240]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 21:37:06.179742766 (606h8m6.050825811s) [upload=531240]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 21:37:06.179747545 (606h8m6.05083059s) [upload=531240]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 21:37:06.490986034 (606h8m6.362069079s) [upload=531240]: Done uploading segments... Dec 30 21:37:06.491006261 (606h8m6.362089307s) [upload=531240]: Upload finished. err: <nil> Duration: 15.317888076s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [........X..X...X.........X.........................X........] duration: 5.913179574s trace: https://jaeger.dev.storj.io/trace/57a7ab98e7eb0d3f FAIL (2025-12-30 22:04:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 21:58:42.298454527 (606h29m42.169537592s) [upload=531550]: Starting upload Dec 30 21:58:42.298523438 (606h29m42.169606473s) [upload=531550]: Done starting upload Dec 30 21:58:42.298568774 (606h29m42.169651809s) [upload=531550]: Uploading segments... Dec 30 21:58:42.298697850 (606h29m42.169780885s) [upload=531550]: Got next segment. Inline: false Dec 30 21:58:42.298728649 (606h29m42.169811694s) [upload=531550]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 21:58:42.298733227 (606h29m42.169816262s) [upload=531550]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 21:58:42.545734213 (606h29m42.416817288s) [seg_pos={0 0}, upload=531550]: Begin upload segment... Dec 30 21:58:42.545786032 (606h29m42.416869078s) [seg_pos={0 0}, upload=531550]: Done begin upload segment. Dec 30 21:58:42.545855104 (606h29m42.416938149s) [noise=true , address=145.79.227.23:4547 , piece_id=LYGUMTAP7OPJWGRVFY65GUZFE4BBTV73JTG657DR5OM26LA7UFZA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.545897384 (606h29m42.416980449s) [noise=true , address=145.79.227.49:4339 , piece_id=WBBPZK2VY2QVRTQNXEKQ7TRYPWBDYS2TGTKEDAAXQ2AT4TUQXTQQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.546757752 (606h29m42.417840817s) [noise=true , address=45.91.116.12:4225 , piece_id=LLYEKE4TJ54KECLAC7KFUV44IVHYCYLF45I43G5V3DPV5QDBUOIQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.546816995 (606h29m42.41790004s) [noise=true , address=145.79.227.40:4050 , piece_id=LA7KLHHWRNHULLPAZWQ6MLS76SLN3BONYXZJXCUHGFC5JZ4SQRIA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.547216927 (606h29m42.418299962s) [noise=true , address=145.79.227.15:4876 , piece_id=L2OIJAB6OOANT2EPFNVNLTMYCH334JGFBFASD44EHOAWTAHKUV5Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.547615996 (606h29m42.418699042s) [noise=true , address=145.79.227.17:4971 , piece_id=DYZUALBL3V75TKGK57LOJABWVCOD5UVS3WH74I55OVUDGRQUA3QQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.547841055 (606h29m42.4189241s) [noise=true , address=77.90.31.21:4877 , piece_id=BD2FEFNAGYWZYDA2IMOVHA7JE66EXSF3A3PL54Q2DUOJFG5ALG4Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548023322 (606h29m42.419106357s) [noise=true , address=45.91.116.14:4190 , piece_id=HHU5YZBTJEEWYVST6XQCKGWKRLRQIIR2UQTQ27RLK53BNAVOSHFQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548380923 (606h29m42.419463968s) [noise=true , address=45.91.116.11:4120 , piece_id=FZ5L6FM7IHX65EVFGVXXHEF43XQ4EBUH4C7VZUSTH2MVWLNR5M2Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548560475 (606h29m42.41964351s) [noise=true , address=145.79.227.39:4013 , piece_id=I4XP47SFJICIYIPEN4E7UC2HXRXH4IXVJYGHXTZBCUESAHNFQXJA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548719268 (606h29m42.419802313s) [noise=true , address=145.79.227.31:4797 , piece_id=S5YBLITONGU6RHPQENP2TCVVWOHEMXPMYXLNYWBZNXWMFI26PVUA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549024107 (606h29m42.420107152s) [noise=true , address=145.79.227.7:4294 , piece_id=C6N5OIRLV3LLSCEIOGQJKWPKENGZYNAPY6V2AYZNKUMACBP5SLWQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549197708 (606h29m42.420280753s) [noise=true , address=145.79.227.16:4910 , piece_id=O4YO367LEVZSRI5E75YEZ7W7YI2KGKKQDXCOHQ3LALE4WUAPQVHQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549506406 (606h29m42.420589441s) [noise=true , address=145.79.227.12:4765 , piece_id=4ZF5YQP4Z5HBTX5YWUDHZV55Q47WASRLMV6ZYJXJRL2QBAVQQM4Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549971591 (606h29m42.421054626s) [noise=true , address=145.79.227.32:4835 , piece_id=CN4NMCFNYKOCG2JCIEXJBL2PLEYWD5THCZB4WQDURIECFVHLLX6Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.551571588 (606h29m42.422654623s) [noise=true , address=145.79.227.24:4581 , piece_id=Y6GBEOR7DQX2H6LBHXYFOXRBX3GP7AKPYVGTQNMUEXKPXLONUUGQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:44.247717212 (606h29m44.118800248s) [upload=531550]: Next returned nil segment Dec 30 21:58:44.247730348 (606h29m44.118813383s) [upload=531550]: Waiting for error group managing segments... Dec 30 21:58:44.532148720 (606h29m44.403231755s) [noise=true , address=145.79.227.31:4797 , piece_id=S5YBLITONGU6RHPQENP2TCVVWOHEMXPMYXLNYWBZNXWMFI26PVUA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.534655293 (606h29m44.405738328s) [noise=true , address=145.79.227.39:4013 , piece_id=I4XP47SFJICIYIPEN4E7UC2HXRXH4IXVJYGHXTZBCUESAHNFQXJA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.544709427 (606h29m44.415792462s) [noise=true , address=145.79.227.12:4765 , piece_id=4ZF5YQP4Z5HBTX5YWUDHZV55Q47WASRLMV6ZYJXJRL2QBAVQQM4Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.545822306 (606h29m44.416905342s) [noise=true , address=145.79.227.7:4294 , piece_id=C6N5OIRLV3LLSCEIOGQJKWPKENGZYNAPY6V2AYZNKUMACBP5SLWQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.549465162 (606h29m44.420548198s) [noise=true , address=145.79.227.24:4581 , piece_id=Y6GBEOR7DQX2H6LBHXYFOXRBX3GP7AKPYVGTQNMUEXKPXLONUUGQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.556935928 (606h29m44.428018964s) [noise=true , address=145.79.227.15:4876 , piece_id=L2OIJAB6OOANT2EPFNVNLTMYCH334JGFBFASD44EHOAWTAHKUV5Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.561283296 (606h29m44.432366331s) [noise=true , address=145.79.227.16:4910 , piece_id=O4YO367LEVZSRI5E75YEZ7W7YI2KGKKQDXCOHQ3LALE4WUAPQVHQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.614194264 (606h29m44.485277309s) [noise=true , address=145.79.227.17:4971 , piece_id=DYZUALBL3V75TKGK57LOJABWVCOD5UVS3WH74I55OVUDGRQUA3QQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.615630689 (606h29m44.486713724s) [noise=true , address=145.79.227.32:4835 , piece_id=CN4NMCFNYKOCG2JCIEXJBL2PLEYWD5THCZB4WQDURIECFVHLLX6Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.639672077 (606h29m44.510755122s) [noise=true , address=145.79.227.23:4547 , piece_id=LYGUMTAP7OPJWGRVFY65GUZFE4BBTV73JTG657DR5OM26LA7UFZA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.699618771 (606h29m44.570701817s) [noise=true , address=77.90.31.21:4877 , piece_id=BD2FEFNAGYWZYDA2IMOVHA7JE66EXSF3A3PL54Q2DUOJFG5ALG4Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.841535416 (606h29m44.712618462s) [noise=true , address=45.91.116.14:4190 , piece_id=HHU5YZBTJEEWYVST6XQCKGWKRLRQIIR2UQTQ27RLK53BNAVOSHFQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.974660841 (606h29m44.845743886s) [noise=true , address=45.91.116.12:4225 , piece_id=LLYEKE4TJ54KECLAC7KFUV44IVHYCYLF45I43G5V3DPV5QDBUOIQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.974681401 (606h29m44.845764436s) [seg_pos={0 0}, upload=531550]: Segment reached optimal threshold of 13 pieces. Dec 30 21:58:44.974855342 (606h29m44.845938377s) [noise=true , address=145.79.227.49:4339 , piece_id=WBBPZK2VY2QVRTQNXEKQ7TRYPWBDYS2TGTKEDAAXQ2AT4TUQXTQQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 30 21:58:44.974861423 (606h29m44.845944458s) [noise=true , address=145.79.227.49:4339 , piece_id=WBBPZK2VY2QVRTQNXEKQ7TRYPWBDYS2TGTKEDAAXQ2AT4TUQXTQQ, seg_pos={0 0}, upload=531550]: Long tail cancellation due to optimal threshold Dec 30 21:58:44.974968166 (606h29m44.846051201s) [noise=true , address=145.79.227.40:4050 , piece_id=LA7KLHHWRNHULLPAZWQ6MLS76SLN3BONYXZJXCUHGFC5JZ4SQRIA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 30 21:58:44.974973627 (606h29m44.846056662s) [noise=true , address=145.79.227.40:4050 , piece_id=LA7KLHHWRNHULLPAZWQ6MLS76SLN3BONYXZJXCUHGFC5JZ4SQRIA, seg_pos={0 0}, upload=531550]: Long tail cancellation due to optimal threshold Dec 30 21:58:44.975076152 (606h29m44.846159187s) [noise=true , address=45.91.116.11:4120 , piece_id=FZ5L6FM7IHX65EVFGVXXHEF43XQ4EBUH4C7VZUSTH2MVWLNR5M2Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 30 21:58:44.975083497 (606h29m44.846166532s) [noise=true , address=45.91.116.11:4120 , piece_id=FZ5L6FM7IHX65EVFGVXXHEF43XQ4EBUH4C7VZUSTH2MVWLNR5M2Q, seg_pos={0 0}, upload=531550]: Long tail cancellation due to optimal threshold Dec 30 21:58:44.975096711 (606h29m44.846179747s) [seg_pos={0 0}, upload=531550]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 21:58:44.975160703 (606h29m44.846243739s) [upload=531550]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 21:58:44.975164691 (606h29m44.846247726s) [upload=531550]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 21:58:45.272782726 (606h29m45.143865772s) [upload=531550]: Done uploading segments... Dec 30 21:58:45.272806352 (606h29m45.143889387s) [upload=531550]: Upload finished. err: <nil> Duration: 5.913205493s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.........XX......X.........X................................] duration: 53.077425128s trace: https://jaeger.dev.storj.io/trace/5cdd5ba7b1cb22c7 FAIL (2025-12-30 22:04:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 21:37:03.355296401 (606h8m3.226379446s) [noise=true , address=145.79.227.22:4543 , piece_id=WWCZ6SEEXQ2KUJ32AGTIJNMSJROLYSJ4XQGBQJC4Q5DACZVA65RQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.355839374 (606h8m3.22692241s) [noise=true , address=145.79.227.35:4492 , piece_id=HCKFSQMJYEP6B5H4OX5AZ5TWP6Z4H4Z6WA2T7Z2H5KNZ4SDJRMUQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.356356930 (606h8m3.227439965s) [noise=true , address=45.91.116.13:4182 , piece_id=BYAYNP4JAAN3AW2LYSK74WJE4BRWUR5Z6K67MTLK6A73KEJ35WEQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.356675316 (606h8m3.227758361s) [noise=true , address=77.90.31.12:4612 , piece_id=63TS3SUEFUE6PORZ4W76OXRWFL7SBN773UGGY3DU7JPBAW66N7UA, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.356992732 (606h8m3.228075767s) [noise=true , address=64.49.13.140:4507 , piece_id=E7T6WL45F6GHECCRANX6AIKWFRZ77TURRWORYEYI43MET2WG7SRQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.357330644 (606h8m3.228413679s) [noise=true , address=77.90.31.21:4886 , piece_id=ZQRCPTVC3WFVQPZ36AU6KJW7PD3T557CXYQIICLDGOKK6JCPVZ5Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.357663649 (606h8m3.228746684s) [noise=true , address=145.79.227.49:4367 , piece_id=R2T2ABLXBPDVP3ZVA2TAYMCLQQ7JQUVQYCTHCKGMFBRVWCDAO4PQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.359827449 (606h8m3.230910484s) [noise=true , address=145.79.227.34:4918 , piece_id=HUMOHOT3HOYCBJ23WXQXWU3X6C6JQTRLLHTD4RQLCQTCB3HXI7EQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.360086692 (606h8m3.231169727s) [noise=true , address=145.79.227.7:4320 , piece_id=MAICNTHGLZ6W27HXDOOZXOBRGQGEXNSA4BJT2T5D5PHRVV6ZMX2Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.361622167 (606h8m3.232705212s) [noise=true , address=145.79.227.33:4888 , piece_id=I4426P3VETHPOL5WHWJEGVYRXJULTCNDX3WSUTS3XZVY36Q2N44Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.362965514 (606h8m3.234048559s) [noise=true , address=45.91.116.11:4133 , piece_id=NEW7ZOVP5E7HYGYD4LG5UG2MDLAQW2OBRTNZDFIYO4CQYMVAWRPQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.364308883 (606h8m3.235391928s) [noise=true , address=145.79.227.26:4663 , piece_id=DYEFQZL4X74QR2LAYURT5ZTR2IWIDB55VLBQOGZM36MA54P2SWJA, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.365734507 (606h8m3.236817542s) [noise=true , address=145.79.227.40:4059 , piece_id=7WSUBR3C3PWVBCWD6KRGSD6MPX4IH2YQOAVO4J4SWZORNHGS2IPQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.366982764 (606h8m3.238065799s) [noise=true , address=145.79.227.29:4973 , piece_id=5OROXELR55A47BFPSJJEWVQU4YW6CAOK6LMEHCINM4FN3OVI2Y6Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.368390294 (606h8m3.239473339s) [noise=true , address=77.90.31.19:4852 , piece_id=2NYO63GCBGCYBFYTMUGNF2NG5FSJOD3AOUOD2FZZIWDINK4X6DJQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:04.736295138 (606h8m4.607378183s) [upload=531240]: Next returned nil segment Dec 30 21:37:04.736311850 (606h8m4.607394885s) [upload=531240]: Waiting for error group managing segments... Dec 30 21:37:05.104989591 (606h8m4.976072636s) [noise=true , address=64.49.13.140:4507 , piece_id=E7T6WL45F6GHECCRANX6AIKWFRZ77TURRWORYEYI43MET2WG7SRQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.208809748 (606h8m5.079892793s) [noise=true , address=145.79.227.22:4543 , piece_id=WWCZ6SEEXQ2KUJ32AGTIJNMSJROLYSJ4XQGBQJC4Q5DACZVA65RQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.441125389 (606h8m5.312208444s) [noise=true , address=77.90.31.19:4852 , piece_id=2NYO63GCBGCYBFYTMUGNF2NG5FSJOD3AOUOD2FZZIWDINK4X6DJQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.483630904 (606h8m5.35471394s) [noise=true , address=77.90.31.12:4612 , piece_id=63TS3SUEFUE6PORZ4W76OXRWFL7SBN773UGGY3DU7JPBAW66N7UA, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.494478811 (606h8m5.365561846s) [noise=true , address=77.90.31.11:4559 , piece_id=3DYD4CNPJQN3S4EYZJGREQTTZJDSL2QTC3OT6G2VFUGMLSVCZUFA, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.497452683 (606h8m5.368535729s) [noise=true , address=77.90.31.17:4769 , piece_id=QXDYESBKGXRY7BCFYYF2O7OXS7Y4Z4KUMJB6AXZAWX735SYU56GQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.661529377 (606h8m5.532612412s) [noise=true , address=45.91.116.13:4182 , piece_id=BYAYNP4JAAN3AW2LYSK74WJE4BRWUR5Z6K67MTLK6A73KEJ35WEQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.729773112 (606h8m5.600856157s) [noise=true , address=45.91.116.11:4133 , piece_id=NEW7ZOVP5E7HYGYD4LG5UG2MDLAQW2OBRTNZDFIYO4CQYMVAWRPQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.940743308 (606h8m5.811826353s) [noise=true , address=77.90.31.21:4886 , piece_id=ZQRCPTVC3WFVQPZ36AU6KJW7PD3T557CXYQIICLDGOKK6JCPVZ5Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.084240044 (606h8m5.95532309s) [noise=true , address=145.79.227.30:4784 , piece_id=DSXUDNYCWO5F6IYXMIVJNEASYXIZ7EO7RKAS7QRUYK5OU7GOOB5Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.121095032 (606h8m5.992178067s) [noise=true , address=145.79.227.40:4059 , piece_id=7WSUBR3C3PWVBCWD6KRGSD6MPX4IH2YQOAVO4J4SWZORNHGS2IPQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.169195983 (606h8m6.040279028s) [noise=true , address=145.79.227.49:4367 , piece_id=R2T2ABLXBPDVP3ZVA2TAYMCLQQ7JQUVQYCTHCKGMFBRVWCDAO4PQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.169380634 (606h8m6.040463679s) [noise=true , address=145.79.227.33:4888 , piece_id=I4426P3VETHPOL5WHWJEGVYRXJULTCNDX3WSUTS3XZVY36Q2N44Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.179060175 (606h8m6.050143211s) [noise=true , address=145.79.227.29:4973 , piece_id=5OROXELR55A47BFPSJJEWVQU4YW6CAOK6LMEHCINM4FN3OVI2Y6Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.179080424 (606h8m6.050163459s) [seg_pos={0 0}, upload=531240]: Segment reached optimal threshold of 14 pieces. Dec 30 21:37:06.179262801 (606h8m6.050345837s) [noise=true , address=145.79.227.26:4663 , piece_id=DYEFQZL4X74QR2LAYURT5ZTR2IWIDB55VLBQOGZM36MA54P2SWJA, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SwpAaXTA3BqUvwubzmJkvwokdz6SSpT9t8HPtQcXXJxmcgfm2): protocol: write tcp 188.166.225.170:43184->145.79.227.26:4663: use of closed network connection; write tcp 188.166.225.170:43184->145.79.227.26:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43184->145.79.227.26:4663: use of closed network connection Dec 30 21:37:06.179268703 (606h8m6.050351748s) [noise=true , address=145.79.227.26:4663 , piece_id=DYEFQZL4X74QR2LAYURT5ZTR2IWIDB55VLBQOGZM36MA54P2SWJA, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179363724 (606h8m6.050446769s) [noise=true , address=145.79.227.7:4320 , piece_id=MAICNTHGLZ6W27HXDOOZXOBRGQGEXNSA4BJT2T5D5PHRVV6ZMX2Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126rEgnQw1w7hsFpRqQP9nLaumdBfdtkgYjdw6WE6nXrxk5kj4B): protocol: write tcp 188.166.225.170:45228->145.79.227.7:4320: use of closed network connection; write tcp 188.166.225.170:45228->145.79.227.7:4320: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45228->145.79.227.7:4320: use of closed network connection Dec 30 21:37:06.179369525 (606h8m6.05045257s) [noise=true , address=145.79.227.7:4320 , piece_id=MAICNTHGLZ6W27HXDOOZXOBRGQGEXNSA4BJT2T5D5PHRVV6ZMX2Q, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179532996 (606h8m6.050616061s) [noise=true , address=145.79.227.34:4918 , piece_id=HUMOHOT3HOYCBJ23WXQXWU3X6C6JQTRLLHTD4RQLCQTCB3HXI7EQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF Dec 30 21:37:06.179572371 (606h8m6.050655406s) [noise=true , address=145.79.227.34:4918 , piece_id=HUMOHOT3HOYCBJ23WXQXWU3X6C6JQTRLLHTD4RQLCQTCB3HXI7EQ, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179638587 (606h8m6.050721622s) [noise=true , address=145.79.227.35:4492 , piece_id=HCKFSQMJYEP6B5H4OX5AZ5TWP6Z4H4Z6WA2T7Z2H5KNZ4SDJRMUQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1roa5GGYmqCX2AxSrqFgqyV2CtBEYZavuGTseiwDy4pjkT6bip): protocol: expected piece hash; context canceled; EOF Dec 30 21:37:06.179647855 (606h8m6.05073089s) [noise=true , address=145.79.227.35:4492 , piece_id=HCKFSQMJYEP6B5H4OX5AZ5TWP6Z4H4Z6WA2T7Z2H5KNZ4SDJRMUQ, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179666641 (606h8m6.050749686s) [seg_pos={0 0}, upload=531240]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 21:37:06.179742766 (606h8m6.050825811s) [upload=531240]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 21:37:06.179747545 (606h8m6.05083059s) [upload=531240]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 21:37:06.490986034 (606h8m6.362069079s) [upload=531240]: Done uploading segments... Dec 30 21:37:06.491006261 (606h8m6.362089307s) [upload=531240]: Upload finished. err: <nil> Duration: 53.077450334s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............................................X..............] duration: 3.590158453s trace: https://jaeger.dev.storj.io/trace/22a47ec27f9a101c FAIL (2025-12-30 22:03:42 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.590175826s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.........X..X...X.........X.........................X.......] duration: 5.851205063s trace: https://jaeger.dev.storj.io/trace/57a7ab98e7eb0d3f FAIL (2025-12-30 22:03:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 21:58:42.298454527 (606h29m42.169537592s) [upload=531550]: Starting upload Dec 30 21:58:42.298523438 (606h29m42.169606473s) [upload=531550]: Done starting upload Dec 30 21:58:42.298568774 (606h29m42.169651809s) [upload=531550]: Uploading segments... Dec 30 21:58:42.298697850 (606h29m42.169780885s) [upload=531550]: Got next segment. Inline: false Dec 30 21:58:42.298728649 (606h29m42.169811694s) [upload=531550]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 21:58:42.298733227 (606h29m42.169816262s) [upload=531550]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 21:58:42.545734213 (606h29m42.416817288s) [seg_pos={0 0}, upload=531550]: Begin upload segment... Dec 30 21:58:42.545786032 (606h29m42.416869078s) [seg_pos={0 0}, upload=531550]: Done begin upload segment. Dec 30 21:58:42.545855104 (606h29m42.416938149s) [noise=true , address=145.79.227.23:4547 , piece_id=LYGUMTAP7OPJWGRVFY65GUZFE4BBTV73JTG657DR5OM26LA7UFZA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.545897384 (606h29m42.416980449s) [noise=true , address=145.79.227.49:4339 , piece_id=WBBPZK2VY2QVRTQNXEKQ7TRYPWBDYS2TGTKEDAAXQ2AT4TUQXTQQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.546757752 (606h29m42.417840817s) [noise=true , address=45.91.116.12:4225 , piece_id=LLYEKE4TJ54KECLAC7KFUV44IVHYCYLF45I43G5V3DPV5QDBUOIQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.546816995 (606h29m42.41790004s) [noise=true , address=145.79.227.40:4050 , piece_id=LA7KLHHWRNHULLPAZWQ6MLS76SLN3BONYXZJXCUHGFC5JZ4SQRIA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.547216927 (606h29m42.418299962s) [noise=true , address=145.79.227.15:4876 , piece_id=L2OIJAB6OOANT2EPFNVNLTMYCH334JGFBFASD44EHOAWTAHKUV5Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.547615996 (606h29m42.418699042s) [noise=true , address=145.79.227.17:4971 , piece_id=DYZUALBL3V75TKGK57LOJABWVCOD5UVS3WH74I55OVUDGRQUA3QQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.547841055 (606h29m42.4189241s) [noise=true , address=77.90.31.21:4877 , piece_id=BD2FEFNAGYWZYDA2IMOVHA7JE66EXSF3A3PL54Q2DUOJFG5ALG4Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548023322 (606h29m42.419106357s) [noise=true , address=45.91.116.14:4190 , piece_id=HHU5YZBTJEEWYVST6XQCKGWKRLRQIIR2UQTQ27RLK53BNAVOSHFQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548380923 (606h29m42.419463968s) [noise=true , address=45.91.116.11:4120 , piece_id=FZ5L6FM7IHX65EVFGVXXHEF43XQ4EBUH4C7VZUSTH2MVWLNR5M2Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548560475 (606h29m42.41964351s) [noise=true , address=145.79.227.39:4013 , piece_id=I4XP47SFJICIYIPEN4E7UC2HXRXH4IXVJYGHXTZBCUESAHNFQXJA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548719268 (606h29m42.419802313s) [noise=true , address=145.79.227.31:4797 , piece_id=S5YBLITONGU6RHPQENP2TCVVWOHEMXPMYXLNYWBZNXWMFI26PVUA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549024107 (606h29m42.420107152s) [noise=true , address=145.79.227.7:4294 , piece_id=C6N5OIRLV3LLSCEIOGQJKWPKENGZYNAPY6V2AYZNKUMACBP5SLWQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549197708 (606h29m42.420280753s) [noise=true , address=145.79.227.16:4910 , piece_id=O4YO367LEVZSRI5E75YEZ7W7YI2KGKKQDXCOHQ3LALE4WUAPQVHQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549506406 (606h29m42.420589441s) [noise=true , address=145.79.227.12:4765 , piece_id=4ZF5YQP4Z5HBTX5YWUDHZV55Q47WASRLMV6ZYJXJRL2QBAVQQM4Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549971591 (606h29m42.421054626s) [noise=true , address=145.79.227.32:4835 , piece_id=CN4NMCFNYKOCG2JCIEXJBL2PLEYWD5THCZB4WQDURIECFVHLLX6Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.551571588 (606h29m42.422654623s) [noise=true , address=145.79.227.24:4581 , piece_id=Y6GBEOR7DQX2H6LBHXYFOXRBX3GP7AKPYVGTQNMUEXKPXLONUUGQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:44.247717212 (606h29m44.118800248s) [upload=531550]: Next returned nil segment Dec 30 21:58:44.247730348 (606h29m44.118813383s) [upload=531550]: Waiting for error group managing segments... Dec 30 21:58:44.532148720 (606h29m44.403231755s) [noise=true , address=145.79.227.31:4797 , piece_id=S5YBLITONGU6RHPQENP2TCVVWOHEMXPMYXLNYWBZNXWMFI26PVUA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.534655293 (606h29m44.405738328s) [noise=true , address=145.79.227.39:4013 , piece_id=I4XP47SFJICIYIPEN4E7UC2HXRXH4IXVJYGHXTZBCUESAHNFQXJA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.544709427 (606h29m44.415792462s) [noise=true , address=145.79.227.12:4765 , piece_id=4ZF5YQP4Z5HBTX5YWUDHZV55Q47WASRLMV6ZYJXJRL2QBAVQQM4Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.545822306 (606h29m44.416905342s) [noise=true , address=145.79.227.7:4294 , piece_id=C6N5OIRLV3LLSCEIOGQJKWPKENGZYNAPY6V2AYZNKUMACBP5SLWQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.549465162 (606h29m44.420548198s) [noise=true , address=145.79.227.24:4581 , piece_id=Y6GBEOR7DQX2H6LBHXYFOXRBX3GP7AKPYVGTQNMUEXKPXLONUUGQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.556935928 (606h29m44.428018964s) [noise=true , address=145.79.227.15:4876 , piece_id=L2OIJAB6OOANT2EPFNVNLTMYCH334JGFBFASD44EHOAWTAHKUV5Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.561283296 (606h29m44.432366331s) [noise=true , address=145.79.227.16:4910 , piece_id=O4YO367LEVZSRI5E75YEZ7W7YI2KGKKQDXCOHQ3LALE4WUAPQVHQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.614194264 (606h29m44.485277309s) [noise=true , address=145.79.227.17:4971 , piece_id=DYZUALBL3V75TKGK57LOJABWVCOD5UVS3WH74I55OVUDGRQUA3QQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.615630689 (606h29m44.486713724s) [noise=true , address=145.79.227.32:4835 , piece_id=CN4NMCFNYKOCG2JCIEXJBL2PLEYWD5THCZB4WQDURIECFVHLLX6Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.639672077 (606h29m44.510755122s) [noise=true , address=145.79.227.23:4547 , piece_id=LYGUMTAP7OPJWGRVFY65GUZFE4BBTV73JTG657DR5OM26LA7UFZA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.699618771 (606h29m44.570701817s) [noise=true , address=77.90.31.21:4877 , piece_id=BD2FEFNAGYWZYDA2IMOVHA7JE66EXSF3A3PL54Q2DUOJFG5ALG4Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.841535416 (606h29m44.712618462s) [noise=true , address=45.91.116.14:4190 , piece_id=HHU5YZBTJEEWYVST6XQCKGWKRLRQIIR2UQTQ27RLK53BNAVOSHFQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.974660841 (606h29m44.845743886s) [noise=true , address=45.91.116.12:4225 , piece_id=LLYEKE4TJ54KECLAC7KFUV44IVHYCYLF45I43G5V3DPV5QDBUOIQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.974681401 (606h29m44.845764436s) [seg_pos={0 0}, upload=531550]: Segment reached optimal threshold of 13 pieces. Dec 30 21:58:44.974855342 (606h29m44.845938377s) [noise=true , address=145.79.227.49:4339 , piece_id=WBBPZK2VY2QVRTQNXEKQ7TRYPWBDYS2TGTKEDAAXQ2AT4TUQXTQQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 30 21:58:44.974861423 (606h29m44.845944458s) [noise=true , address=145.79.227.49:4339 , piece_id=WBBPZK2VY2QVRTQNXEKQ7TRYPWBDYS2TGTKEDAAXQ2AT4TUQXTQQ, seg_pos={0 0}, upload=531550]: Long tail cancellation due to optimal threshold Dec 30 21:58:44.974968166 (606h29m44.846051201s) [noise=true , address=145.79.227.40:4050 , piece_id=LA7KLHHWRNHULLPAZWQ6MLS76SLN3BONYXZJXCUHGFC5JZ4SQRIA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 30 21:58:44.974973627 (606h29m44.846056662s) [noise=true , address=145.79.227.40:4050 , piece_id=LA7KLHHWRNHULLPAZWQ6MLS76SLN3BONYXZJXCUHGFC5JZ4SQRIA, seg_pos={0 0}, upload=531550]: Long tail cancellation due to optimal threshold Dec 30 21:58:44.975076152 (606h29m44.846159187s) [noise=true , address=45.91.116.11:4120 , piece_id=FZ5L6FM7IHX65EVFGVXXHEF43XQ4EBUH4C7VZUSTH2MVWLNR5M2Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 30 21:58:44.975083497 (606h29m44.846166532s) [noise=true , address=45.91.116.11:4120 , piece_id=FZ5L6FM7IHX65EVFGVXXHEF43XQ4EBUH4C7VZUSTH2MVWLNR5M2Q, seg_pos={0 0}, upload=531550]: Long tail cancellation due to optimal threshold Dec 30 21:58:44.975096711 (606h29m44.846179747s) [seg_pos={0 0}, upload=531550]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 21:58:44.975160703 (606h29m44.846243739s) [upload=531550]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 21:58:44.975164691 (606h29m44.846247726s) [upload=531550]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 21:58:45.272782726 (606h29m45.143865772s) [upload=531550]: Done uploading segments... Dec 30 21:58:45.272806352 (606h29m45.143889387s) [upload=531550]: Upload finished. err: <nil> Duration: 5.851231033s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [..........XX......X.........X...............................] duration: 15.446521166s trace: https://jaeger.dev.storj.io/trace/5cdd5ba7b1cb22c7 FAIL (2025-12-30 22:02:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 21:37:03.355296401 (606h8m3.226379446s) [noise=true , address=145.79.227.22:4543 , piece_id=WWCZ6SEEXQ2KUJ32AGTIJNMSJROLYSJ4XQGBQJC4Q5DACZVA65RQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.355839374 (606h8m3.22692241s) [noise=true , address=145.79.227.35:4492 , piece_id=HCKFSQMJYEP6B5H4OX5AZ5TWP6Z4H4Z6WA2T7Z2H5KNZ4SDJRMUQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.356356930 (606h8m3.227439965s) [noise=true , address=45.91.116.13:4182 , piece_id=BYAYNP4JAAN3AW2LYSK74WJE4BRWUR5Z6K67MTLK6A73KEJ35WEQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.356675316 (606h8m3.227758361s) [noise=true , address=77.90.31.12:4612 , piece_id=63TS3SUEFUE6PORZ4W76OXRWFL7SBN773UGGY3DU7JPBAW66N7UA, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.356992732 (606h8m3.228075767s) [noise=true , address=64.49.13.140:4507 , piece_id=E7T6WL45F6GHECCRANX6AIKWFRZ77TURRWORYEYI43MET2WG7SRQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.357330644 (606h8m3.228413679s) [noise=true , address=77.90.31.21:4886 , piece_id=ZQRCPTVC3WFVQPZ36AU6KJW7PD3T557CXYQIICLDGOKK6JCPVZ5Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.357663649 (606h8m3.228746684s) [noise=true , address=145.79.227.49:4367 , piece_id=R2T2ABLXBPDVP3ZVA2TAYMCLQQ7JQUVQYCTHCKGMFBRVWCDAO4PQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.359827449 (606h8m3.230910484s) [noise=true , address=145.79.227.34:4918 , piece_id=HUMOHOT3HOYCBJ23WXQXWU3X6C6JQTRLLHTD4RQLCQTCB3HXI7EQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.360086692 (606h8m3.231169727s) [noise=true , address=145.79.227.7:4320 , piece_id=MAICNTHGLZ6W27HXDOOZXOBRGQGEXNSA4BJT2T5D5PHRVV6ZMX2Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.361622167 (606h8m3.232705212s) [noise=true , address=145.79.227.33:4888 , piece_id=I4426P3VETHPOL5WHWJEGVYRXJULTCNDX3WSUTS3XZVY36Q2N44Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.362965514 (606h8m3.234048559s) [noise=true , address=45.91.116.11:4133 , piece_id=NEW7ZOVP5E7HYGYD4LG5UG2MDLAQW2OBRTNZDFIYO4CQYMVAWRPQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.364308883 (606h8m3.235391928s) [noise=true , address=145.79.227.26:4663 , piece_id=DYEFQZL4X74QR2LAYURT5ZTR2IWIDB55VLBQOGZM36MA54P2SWJA, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.365734507 (606h8m3.236817542s) [noise=true , address=145.79.227.40:4059 , piece_id=7WSUBR3C3PWVBCWD6KRGSD6MPX4IH2YQOAVO4J4SWZORNHGS2IPQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.366982764 (606h8m3.238065799s) [noise=true , address=145.79.227.29:4973 , piece_id=5OROXELR55A47BFPSJJEWVQU4YW6CAOK6LMEHCINM4FN3OVI2Y6Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.368390294 (606h8m3.239473339s) [noise=true , address=77.90.31.19:4852 , piece_id=2NYO63GCBGCYBFYTMUGNF2NG5FSJOD3AOUOD2FZZIWDINK4X6DJQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:04.736295138 (606h8m4.607378183s) [upload=531240]: Next returned nil segment Dec 30 21:37:04.736311850 (606h8m4.607394885s) [upload=531240]: Waiting for error group managing segments... Dec 30 21:37:05.104989591 (606h8m4.976072636s) [noise=true , address=64.49.13.140:4507 , piece_id=E7T6WL45F6GHECCRANX6AIKWFRZ77TURRWORYEYI43MET2WG7SRQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.208809748 (606h8m5.079892793s) [noise=true , address=145.79.227.22:4543 , piece_id=WWCZ6SEEXQ2KUJ32AGTIJNMSJROLYSJ4XQGBQJC4Q5DACZVA65RQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.441125389 (606h8m5.312208444s) [noise=true , address=77.90.31.19:4852 , piece_id=2NYO63GCBGCYBFYTMUGNF2NG5FSJOD3AOUOD2FZZIWDINK4X6DJQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.483630904 (606h8m5.35471394s) [noise=true , address=77.90.31.12:4612 , piece_id=63TS3SUEFUE6PORZ4W76OXRWFL7SBN773UGGY3DU7JPBAW66N7UA, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.494478811 (606h8m5.365561846s) [noise=true , address=77.90.31.11:4559 , piece_id=3DYD4CNPJQN3S4EYZJGREQTTZJDSL2QTC3OT6G2VFUGMLSVCZUFA, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.497452683 (606h8m5.368535729s) [noise=true , address=77.90.31.17:4769 , piece_id=QXDYESBKGXRY7BCFYYF2O7OXS7Y4Z4KUMJB6AXZAWX735SYU56GQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.661529377 (606h8m5.532612412s) [noise=true , address=45.91.116.13:4182 , piece_id=BYAYNP4JAAN3AW2LYSK74WJE4BRWUR5Z6K67MTLK6A73KEJ35WEQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.729773112 (606h8m5.600856157s) [noise=true , address=45.91.116.11:4133 , piece_id=NEW7ZOVP5E7HYGYD4LG5UG2MDLAQW2OBRTNZDFIYO4CQYMVAWRPQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.940743308 (606h8m5.811826353s) [noise=true , address=77.90.31.21:4886 , piece_id=ZQRCPTVC3WFVQPZ36AU6KJW7PD3T557CXYQIICLDGOKK6JCPVZ5Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.084240044 (606h8m5.95532309s) [noise=true , address=145.79.227.30:4784 , piece_id=DSXUDNYCWO5F6IYXMIVJNEASYXIZ7EO7RKAS7QRUYK5OU7GOOB5Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.121095032 (606h8m5.992178067s) [noise=true , address=145.79.227.40:4059 , piece_id=7WSUBR3C3PWVBCWD6KRGSD6MPX4IH2YQOAVO4J4SWZORNHGS2IPQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.169195983 (606h8m6.040279028s) [noise=true , address=145.79.227.49:4367 , piece_id=R2T2ABLXBPDVP3ZVA2TAYMCLQQ7JQUVQYCTHCKGMFBRVWCDAO4PQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.169380634 (606h8m6.040463679s) [noise=true , address=145.79.227.33:4888 , piece_id=I4426P3VETHPOL5WHWJEGVYRXJULTCNDX3WSUTS3XZVY36Q2N44Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.179060175 (606h8m6.050143211s) [noise=true , address=145.79.227.29:4973 , piece_id=5OROXELR55A47BFPSJJEWVQU4YW6CAOK6LMEHCINM4FN3OVI2Y6Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.179080424 (606h8m6.050163459s) [seg_pos={0 0}, upload=531240]: Segment reached optimal threshold of 14 pieces. Dec 30 21:37:06.179262801 (606h8m6.050345837s) [noise=true , address=145.79.227.26:4663 , piece_id=DYEFQZL4X74QR2LAYURT5ZTR2IWIDB55VLBQOGZM36MA54P2SWJA, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SwpAaXTA3BqUvwubzmJkvwokdz6SSpT9t8HPtQcXXJxmcgfm2): protocol: write tcp 188.166.225.170:43184->145.79.227.26:4663: use of closed network connection; write tcp 188.166.225.170:43184->145.79.227.26:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43184->145.79.227.26:4663: use of closed network connection Dec 30 21:37:06.179268703 (606h8m6.050351748s) [noise=true , address=145.79.227.26:4663 , piece_id=DYEFQZL4X74QR2LAYURT5ZTR2IWIDB55VLBQOGZM36MA54P2SWJA, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179363724 (606h8m6.050446769s) [noise=true , address=145.79.227.7:4320 , piece_id=MAICNTHGLZ6W27HXDOOZXOBRGQGEXNSA4BJT2T5D5PHRVV6ZMX2Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126rEgnQw1w7hsFpRqQP9nLaumdBfdtkgYjdw6WE6nXrxk5kj4B): protocol: write tcp 188.166.225.170:45228->145.79.227.7:4320: use of closed network connection; write tcp 188.166.225.170:45228->145.79.227.7:4320: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45228->145.79.227.7:4320: use of closed network connection Dec 30 21:37:06.179369525 (606h8m6.05045257s) [noise=true , address=145.79.227.7:4320 , piece_id=MAICNTHGLZ6W27HXDOOZXOBRGQGEXNSA4BJT2T5D5PHRVV6ZMX2Q, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179532996 (606h8m6.050616061s) [noise=true , address=145.79.227.34:4918 , piece_id=HUMOHOT3HOYCBJ23WXQXWU3X6C6JQTRLLHTD4RQLCQTCB3HXI7EQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF Dec 30 21:37:06.179572371 (606h8m6.050655406s) [noise=true , address=145.79.227.34:4918 , piece_id=HUMOHOT3HOYCBJ23WXQXWU3X6C6JQTRLLHTD4RQLCQTCB3HXI7EQ, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179638587 (606h8m6.050721622s) [noise=true , address=145.79.227.35:4492 , piece_id=HCKFSQMJYEP6B5H4OX5AZ5TWP6Z4H4Z6WA2T7Z2H5KNZ4SDJRMUQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1roa5GGYmqCX2AxSrqFgqyV2CtBEYZavuGTseiwDy4pjkT6bip): protocol: expected piece hash; context canceled; EOF Dec 30 21:37:06.179647855 (606h8m6.05073089s) [noise=true , address=145.79.227.35:4492 , piece_id=HCKFSQMJYEP6B5H4OX5AZ5TWP6Z4H4Z6WA2T7Z2H5KNZ4SDJRMUQ, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179666641 (606h8m6.050749686s) [seg_pos={0 0}, upload=531240]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 21:37:06.179742766 (606h8m6.050825811s) [upload=531240]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 21:37:06.179747545 (606h8m6.05083059s) [upload=531240]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 21:37:06.490986034 (606h8m6.362069079s) [upload=531240]: Done uploading segments... Dec 30 21:37:06.491006261 (606h8m6.362089307s) [upload=531240]: Upload finished. err: <nil> Duration: 15.446548609s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............................................X.............] duration: 3.751598431s trace: https://jaeger.dev.storj.io/trace/22a47ec27f9a101c FAIL (2025-12-30 22:02:45 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.751648546s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [X.........X..X...X.........X.........................X......] duration: 6.073854703s trace: https://jaeger.dev.storj.io/trace/57a7ab98e7eb0d3f FAIL (2025-12-30 22:02:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 21:58:42.298454527 (606h29m42.169537592s) [upload=531550]: Starting upload Dec 30 21:58:42.298523438 (606h29m42.169606473s) [upload=531550]: Done starting upload Dec 30 21:58:42.298568774 (606h29m42.169651809s) [upload=531550]: Uploading segments... Dec 30 21:58:42.298697850 (606h29m42.169780885s) [upload=531550]: Got next segment. Inline: false Dec 30 21:58:42.298728649 (606h29m42.169811694s) [upload=531550]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 21:58:42.298733227 (606h29m42.169816262s) [upload=531550]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 21:58:42.545734213 (606h29m42.416817288s) [seg_pos={0 0}, upload=531550]: Begin upload segment... Dec 30 21:58:42.545786032 (606h29m42.416869078s) [seg_pos={0 0}, upload=531550]: Done begin upload segment. Dec 30 21:58:42.545855104 (606h29m42.416938149s) [noise=true , address=145.79.227.23:4547 , piece_id=LYGUMTAP7OPJWGRVFY65GUZFE4BBTV73JTG657DR5OM26LA7UFZA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.545897384 (606h29m42.416980449s) [noise=true , address=145.79.227.49:4339 , piece_id=WBBPZK2VY2QVRTQNXEKQ7TRYPWBDYS2TGTKEDAAXQ2AT4TUQXTQQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.546757752 (606h29m42.417840817s) [noise=true , address=45.91.116.12:4225 , piece_id=LLYEKE4TJ54KECLAC7KFUV44IVHYCYLF45I43G5V3DPV5QDBUOIQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.546816995 (606h29m42.41790004s) [noise=true , address=145.79.227.40:4050 , piece_id=LA7KLHHWRNHULLPAZWQ6MLS76SLN3BONYXZJXCUHGFC5JZ4SQRIA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.547216927 (606h29m42.418299962s) [noise=true , address=145.79.227.15:4876 , piece_id=L2OIJAB6OOANT2EPFNVNLTMYCH334JGFBFASD44EHOAWTAHKUV5Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.547615996 (606h29m42.418699042s) [noise=true , address=145.79.227.17:4971 , piece_id=DYZUALBL3V75TKGK57LOJABWVCOD5UVS3WH74I55OVUDGRQUA3QQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.547841055 (606h29m42.4189241s) [noise=true , address=77.90.31.21:4877 , piece_id=BD2FEFNAGYWZYDA2IMOVHA7JE66EXSF3A3PL54Q2DUOJFG5ALG4Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548023322 (606h29m42.419106357s) [noise=true , address=45.91.116.14:4190 , piece_id=HHU5YZBTJEEWYVST6XQCKGWKRLRQIIR2UQTQ27RLK53BNAVOSHFQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548380923 (606h29m42.419463968s) [noise=true , address=45.91.116.11:4120 , piece_id=FZ5L6FM7IHX65EVFGVXXHEF43XQ4EBUH4C7VZUSTH2MVWLNR5M2Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548560475 (606h29m42.41964351s) [noise=true , address=145.79.227.39:4013 , piece_id=I4XP47SFJICIYIPEN4E7UC2HXRXH4IXVJYGHXTZBCUESAHNFQXJA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548719268 (606h29m42.419802313s) [noise=true , address=145.79.227.31:4797 , piece_id=S5YBLITONGU6RHPQENP2TCVVWOHEMXPMYXLNYWBZNXWMFI26PVUA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549024107 (606h29m42.420107152s) [noise=true , address=145.79.227.7:4294 , piece_id=C6N5OIRLV3LLSCEIOGQJKWPKENGZYNAPY6V2AYZNKUMACBP5SLWQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549197708 (606h29m42.420280753s) [noise=true , address=145.79.227.16:4910 , piece_id=O4YO367LEVZSRI5E75YEZ7W7YI2KGKKQDXCOHQ3LALE4WUAPQVHQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549506406 (606h29m42.420589441s) [noise=true , address=145.79.227.12:4765 , piece_id=4ZF5YQP4Z5HBTX5YWUDHZV55Q47WASRLMV6ZYJXJRL2QBAVQQM4Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549971591 (606h29m42.421054626s) [noise=true , address=145.79.227.32:4835 , piece_id=CN4NMCFNYKOCG2JCIEXJBL2PLEYWD5THCZB4WQDURIECFVHLLX6Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.551571588 (606h29m42.422654623s) [noise=true , address=145.79.227.24:4581 , piece_id=Y6GBEOR7DQX2H6LBHXYFOXRBX3GP7AKPYVGTQNMUEXKPXLONUUGQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:44.247717212 (606h29m44.118800248s) [upload=531550]: Next returned nil segment Dec 30 21:58:44.247730348 (606h29m44.118813383s) [upload=531550]: Waiting for error group managing segments... Dec 30 21:58:44.532148720 (606h29m44.403231755s) [noise=true , address=145.79.227.31:4797 , piece_id=S5YBLITONGU6RHPQENP2TCVVWOHEMXPMYXLNYWBZNXWMFI26PVUA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.534655293 (606h29m44.405738328s) [noise=true , address=145.79.227.39:4013 , piece_id=I4XP47SFJICIYIPEN4E7UC2HXRXH4IXVJYGHXTZBCUESAHNFQXJA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.544709427 (606h29m44.415792462s) [noise=true , address=145.79.227.12:4765 , piece_id=4ZF5YQP4Z5HBTX5YWUDHZV55Q47WASRLMV6ZYJXJRL2QBAVQQM4Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.545822306 (606h29m44.416905342s) [noise=true , address=145.79.227.7:4294 , piece_id=C6N5OIRLV3LLSCEIOGQJKWPKENGZYNAPY6V2AYZNKUMACBP5SLWQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.549465162 (606h29m44.420548198s) [noise=true , address=145.79.227.24:4581 , piece_id=Y6GBEOR7DQX2H6LBHXYFOXRBX3GP7AKPYVGTQNMUEXKPXLONUUGQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.556935928 (606h29m44.428018964s) [noise=true , address=145.79.227.15:4876 , piece_id=L2OIJAB6OOANT2EPFNVNLTMYCH334JGFBFASD44EHOAWTAHKUV5Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.561283296 (606h29m44.432366331s) [noise=true , address=145.79.227.16:4910 , piece_id=O4YO367LEVZSRI5E75YEZ7W7YI2KGKKQDXCOHQ3LALE4WUAPQVHQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.614194264 (606h29m44.485277309s) [noise=true , address=145.79.227.17:4971 , piece_id=DYZUALBL3V75TKGK57LOJABWVCOD5UVS3WH74I55OVUDGRQUA3QQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.615630689 (606h29m44.486713724s) [noise=true , address=145.79.227.32:4835 , piece_id=CN4NMCFNYKOCG2JCIEXJBL2PLEYWD5THCZB4WQDURIECFVHLLX6Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.639672077 (606h29m44.510755122s) [noise=true , address=145.79.227.23:4547 , piece_id=LYGUMTAP7OPJWGRVFY65GUZFE4BBTV73JTG657DR5OM26LA7UFZA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.699618771 (606h29m44.570701817s) [noise=true , address=77.90.31.21:4877 , piece_id=BD2FEFNAGYWZYDA2IMOVHA7JE66EXSF3A3PL54Q2DUOJFG5ALG4Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.841535416 (606h29m44.712618462s) [noise=true , address=45.91.116.14:4190 , piece_id=HHU5YZBTJEEWYVST6XQCKGWKRLRQIIR2UQTQ27RLK53BNAVOSHFQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.974660841 (606h29m44.845743886s) [noise=true , address=45.91.116.12:4225 , piece_id=LLYEKE4TJ54KECLAC7KFUV44IVHYCYLF45I43G5V3DPV5QDBUOIQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.974681401 (606h29m44.845764436s) [seg_pos={0 0}, upload=531550]: Segment reached optimal threshold of 13 pieces. Dec 30 21:58:44.974855342 (606h29m44.845938377s) [noise=true , address=145.79.227.49:4339 , piece_id=WBBPZK2VY2QVRTQNXEKQ7TRYPWBDYS2TGTKEDAAXQ2AT4TUQXTQQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 30 21:58:44.974861423 (606h29m44.845944458s) [noise=true , address=145.79.227.49:4339 , piece_id=WBBPZK2VY2QVRTQNXEKQ7TRYPWBDYS2TGTKEDAAXQ2AT4TUQXTQQ, seg_pos={0 0}, upload=531550]: Long tail cancellation due to optimal threshold Dec 30 21:58:44.974968166 (606h29m44.846051201s) [noise=true , address=145.79.227.40:4050 , piece_id=LA7KLHHWRNHULLPAZWQ6MLS76SLN3BONYXZJXCUHGFC5JZ4SQRIA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 30 21:58:44.974973627 (606h29m44.846056662s) [noise=true , address=145.79.227.40:4050 , piece_id=LA7KLHHWRNHULLPAZWQ6MLS76SLN3BONYXZJXCUHGFC5JZ4SQRIA, seg_pos={0 0}, upload=531550]: Long tail cancellation due to optimal threshold Dec 30 21:58:44.975076152 (606h29m44.846159187s) [noise=true , address=45.91.116.11:4120 , piece_id=FZ5L6FM7IHX65EVFGVXXHEF43XQ4EBUH4C7VZUSTH2MVWLNR5M2Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 30 21:58:44.975083497 (606h29m44.846166532s) [noise=true , address=45.91.116.11:4120 , piece_id=FZ5L6FM7IHX65EVFGVXXHEF43XQ4EBUH4C7VZUSTH2MVWLNR5M2Q, seg_pos={0 0}, upload=531550]: Long tail cancellation due to optimal threshold Dec 30 21:58:44.975096711 (606h29m44.846179747s) [seg_pos={0 0}, upload=531550]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 21:58:44.975160703 (606h29m44.846243739s) [upload=531550]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 21:58:44.975164691 (606h29m44.846247726s) [upload=531550]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 21:58:45.272782726 (606h29m45.143865772s) [upload=531550]: Done uploading segments... Dec 30 21:58:45.272806352 (606h29m45.143889387s) [upload=531550]: Upload finished. err: <nil> Duration: 6.073888578s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............................................X............] duration: 4.506058338s trace: https://jaeger.dev.storj.io/trace/22a47ec27f9a101c FAIL (2025-12-30 22:02:11 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.506082863s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [...........XX......X.........X..............................] duration: 13.942975564s trace: https://jaeger.dev.storj.io/trace/5cdd5ba7b1cb22c7 FAIL (2025-12-30 22:01:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 21:37:03.355296401 (606h8m3.226379446s) [noise=true , address=145.79.227.22:4543 , piece_id=WWCZ6SEEXQ2KUJ32AGTIJNMSJROLYSJ4XQGBQJC4Q5DACZVA65RQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.355839374 (606h8m3.22692241s) [noise=true , address=145.79.227.35:4492 , piece_id=HCKFSQMJYEP6B5H4OX5AZ5TWP6Z4H4Z6WA2T7Z2H5KNZ4SDJRMUQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.356356930 (606h8m3.227439965s) [noise=true , address=45.91.116.13:4182 , piece_id=BYAYNP4JAAN3AW2LYSK74WJE4BRWUR5Z6K67MTLK6A73KEJ35WEQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.356675316 (606h8m3.227758361s) [noise=true , address=77.90.31.12:4612 , piece_id=63TS3SUEFUE6PORZ4W76OXRWFL7SBN773UGGY3DU7JPBAW66N7UA, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.356992732 (606h8m3.228075767s) [noise=true , address=64.49.13.140:4507 , piece_id=E7T6WL45F6GHECCRANX6AIKWFRZ77TURRWORYEYI43MET2WG7SRQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.357330644 (606h8m3.228413679s) [noise=true , address=77.90.31.21:4886 , piece_id=ZQRCPTVC3WFVQPZ36AU6KJW7PD3T557CXYQIICLDGOKK6JCPVZ5Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.357663649 (606h8m3.228746684s) [noise=true , address=145.79.227.49:4367 , piece_id=R2T2ABLXBPDVP3ZVA2TAYMCLQQ7JQUVQYCTHCKGMFBRVWCDAO4PQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.359827449 (606h8m3.230910484s) [noise=true , address=145.79.227.34:4918 , piece_id=HUMOHOT3HOYCBJ23WXQXWU3X6C6JQTRLLHTD4RQLCQTCB3HXI7EQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.360086692 (606h8m3.231169727s) [noise=true , address=145.79.227.7:4320 , piece_id=MAICNTHGLZ6W27HXDOOZXOBRGQGEXNSA4BJT2T5D5PHRVV6ZMX2Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.361622167 (606h8m3.232705212s) [noise=true , address=145.79.227.33:4888 , piece_id=I4426P3VETHPOL5WHWJEGVYRXJULTCNDX3WSUTS3XZVY36Q2N44Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.362965514 (606h8m3.234048559s) [noise=true , address=45.91.116.11:4133 , piece_id=NEW7ZOVP5E7HYGYD4LG5UG2MDLAQW2OBRTNZDFIYO4CQYMVAWRPQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.364308883 (606h8m3.235391928s) [noise=true , address=145.79.227.26:4663 , piece_id=DYEFQZL4X74QR2LAYURT5ZTR2IWIDB55VLBQOGZM36MA54P2SWJA, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.365734507 (606h8m3.236817542s) [noise=true , address=145.79.227.40:4059 , piece_id=7WSUBR3C3PWVBCWD6KRGSD6MPX4IH2YQOAVO4J4SWZORNHGS2IPQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.366982764 (606h8m3.238065799s) [noise=true , address=145.79.227.29:4973 , piece_id=5OROXELR55A47BFPSJJEWVQU4YW6CAOK6LMEHCINM4FN3OVI2Y6Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.368390294 (606h8m3.239473339s) [noise=true , address=77.90.31.19:4852 , piece_id=2NYO63GCBGCYBFYTMUGNF2NG5FSJOD3AOUOD2FZZIWDINK4X6DJQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:04.736295138 (606h8m4.607378183s) [upload=531240]: Next returned nil segment Dec 30 21:37:04.736311850 (606h8m4.607394885s) [upload=531240]: Waiting for error group managing segments... Dec 30 21:37:05.104989591 (606h8m4.976072636s) [noise=true , address=64.49.13.140:4507 , piece_id=E7T6WL45F6GHECCRANX6AIKWFRZ77TURRWORYEYI43MET2WG7SRQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.208809748 (606h8m5.079892793s) [noise=true , address=145.79.227.22:4543 , piece_id=WWCZ6SEEXQ2KUJ32AGTIJNMSJROLYSJ4XQGBQJC4Q5DACZVA65RQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.441125389 (606h8m5.312208444s) [noise=true , address=77.90.31.19:4852 , piece_id=2NYO63GCBGCYBFYTMUGNF2NG5FSJOD3AOUOD2FZZIWDINK4X6DJQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.483630904 (606h8m5.35471394s) [noise=true , address=77.90.31.12:4612 , piece_id=63TS3SUEFUE6PORZ4W76OXRWFL7SBN773UGGY3DU7JPBAW66N7UA, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.494478811 (606h8m5.365561846s) [noise=true , address=77.90.31.11:4559 , piece_id=3DYD4CNPJQN3S4EYZJGREQTTZJDSL2QTC3OT6G2VFUGMLSVCZUFA, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.497452683 (606h8m5.368535729s) [noise=true , address=77.90.31.17:4769 , piece_id=QXDYESBKGXRY7BCFYYF2O7OXS7Y4Z4KUMJB6AXZAWX735SYU56GQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.661529377 (606h8m5.532612412s) [noise=true , address=45.91.116.13:4182 , piece_id=BYAYNP4JAAN3AW2LYSK74WJE4BRWUR5Z6K67MTLK6A73KEJ35WEQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.729773112 (606h8m5.600856157s) [noise=true , address=45.91.116.11:4133 , piece_id=NEW7ZOVP5E7HYGYD4LG5UG2MDLAQW2OBRTNZDFIYO4CQYMVAWRPQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.940743308 (606h8m5.811826353s) [noise=true , address=77.90.31.21:4886 , piece_id=ZQRCPTVC3WFVQPZ36AU6KJW7PD3T557CXYQIICLDGOKK6JCPVZ5Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.084240044 (606h8m5.95532309s) [noise=true , address=145.79.227.30:4784 , piece_id=DSXUDNYCWO5F6IYXMIVJNEASYXIZ7EO7RKAS7QRUYK5OU7GOOB5Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.121095032 (606h8m5.992178067s) [noise=true , address=145.79.227.40:4059 , piece_id=7WSUBR3C3PWVBCWD6KRGSD6MPX4IH2YQOAVO4J4SWZORNHGS2IPQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.169195983 (606h8m6.040279028s) [noise=true , address=145.79.227.49:4367 , piece_id=R2T2ABLXBPDVP3ZVA2TAYMCLQQ7JQUVQYCTHCKGMFBRVWCDAO4PQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.169380634 (606h8m6.040463679s) [noise=true , address=145.79.227.33:4888 , piece_id=I4426P3VETHPOL5WHWJEGVYRXJULTCNDX3WSUTS3XZVY36Q2N44Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.179060175 (606h8m6.050143211s) [noise=true , address=145.79.227.29:4973 , piece_id=5OROXELR55A47BFPSJJEWVQU4YW6CAOK6LMEHCINM4FN3OVI2Y6Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.179080424 (606h8m6.050163459s) [seg_pos={0 0}, upload=531240]: Segment reached optimal threshold of 14 pieces. Dec 30 21:37:06.179262801 (606h8m6.050345837s) [noise=true , address=145.79.227.26:4663 , piece_id=DYEFQZL4X74QR2LAYURT5ZTR2IWIDB55VLBQOGZM36MA54P2SWJA, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SwpAaXTA3BqUvwubzmJkvwokdz6SSpT9t8HPtQcXXJxmcgfm2): protocol: write tcp 188.166.225.170:43184->145.79.227.26:4663: use of closed network connection; write tcp 188.166.225.170:43184->145.79.227.26:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43184->145.79.227.26:4663: use of closed network connection Dec 30 21:37:06.179268703 (606h8m6.050351748s) [noise=true , address=145.79.227.26:4663 , piece_id=DYEFQZL4X74QR2LAYURT5ZTR2IWIDB55VLBQOGZM36MA54P2SWJA, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179363724 (606h8m6.050446769s) [noise=true , address=145.79.227.7:4320 , piece_id=MAICNTHGLZ6W27HXDOOZXOBRGQGEXNSA4BJT2T5D5PHRVV6ZMX2Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126rEgnQw1w7hsFpRqQP9nLaumdBfdtkgYjdw6WE6nXrxk5kj4B): protocol: write tcp 188.166.225.170:45228->145.79.227.7:4320: use of closed network connection; write tcp 188.166.225.170:45228->145.79.227.7:4320: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45228->145.79.227.7:4320: use of closed network connection Dec 30 21:37:06.179369525 (606h8m6.05045257s) [noise=true , address=145.79.227.7:4320 , piece_id=MAICNTHGLZ6W27HXDOOZXOBRGQGEXNSA4BJT2T5D5PHRVV6ZMX2Q, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179532996 (606h8m6.050616061s) [noise=true , address=145.79.227.34:4918 , piece_id=HUMOHOT3HOYCBJ23WXQXWU3X6C6JQTRLLHTD4RQLCQTCB3HXI7EQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF Dec 30 21:37:06.179572371 (606h8m6.050655406s) [noise=true , address=145.79.227.34:4918 , piece_id=HUMOHOT3HOYCBJ23WXQXWU3X6C6JQTRLLHTD4RQLCQTCB3HXI7EQ, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179638587 (606h8m6.050721622s) [noise=true , address=145.79.227.35:4492 , piece_id=HCKFSQMJYEP6B5H4OX5AZ5TWP6Z4H4Z6WA2T7Z2H5KNZ4SDJRMUQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1roa5GGYmqCX2AxSrqFgqyV2CtBEYZavuGTseiwDy4pjkT6bip): protocol: expected piece hash; context canceled; EOF Dec 30 21:37:06.179647855 (606h8m6.05073089s) [noise=true , address=145.79.227.35:4492 , piece_id=HCKFSQMJYEP6B5H4OX5AZ5TWP6Z4H4Z6WA2T7Z2H5KNZ4SDJRMUQ, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179666641 (606h8m6.050749686s) [seg_pos={0 0}, upload=531240]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 21:37:06.179742766 (606h8m6.050825811s) [upload=531240]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 21:37:06.179747545 (606h8m6.05083059s) [upload=531240]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 21:37:06.490986034 (606h8m6.362069079s) [upload=531240]: Done uploading segments... Dec 30 21:37:06.491006261 (606h8m6.362089307s) [upload=531240]: Upload finished. err: <nil> Duration: 13.943001524s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.X.........X..X...X.........X.........................X.....] duration: 6.151527894s trace: https://jaeger.dev.storj.io/trace/57a7ab98e7eb0d3f FAIL (2025-12-30 22:01:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 21:58:42.298454527 (606h29m42.169537592s) [upload=531550]: Starting upload Dec 30 21:58:42.298523438 (606h29m42.169606473s) [upload=531550]: Done starting upload Dec 30 21:58:42.298568774 (606h29m42.169651809s) [upload=531550]: Uploading segments... Dec 30 21:58:42.298697850 (606h29m42.169780885s) [upload=531550]: Got next segment. Inline: false Dec 30 21:58:42.298728649 (606h29m42.169811694s) [upload=531550]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 21:58:42.298733227 (606h29m42.169816262s) [upload=531550]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 21:58:42.545734213 (606h29m42.416817288s) [seg_pos={0 0}, upload=531550]: Begin upload segment... Dec 30 21:58:42.545786032 (606h29m42.416869078s) [seg_pos={0 0}, upload=531550]: Done begin upload segment. Dec 30 21:58:42.545855104 (606h29m42.416938149s) [noise=true , address=145.79.227.23:4547 , piece_id=LYGUMTAP7OPJWGRVFY65GUZFE4BBTV73JTG657DR5OM26LA7UFZA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.545897384 (606h29m42.416980449s) [noise=true , address=145.79.227.49:4339 , piece_id=WBBPZK2VY2QVRTQNXEKQ7TRYPWBDYS2TGTKEDAAXQ2AT4TUQXTQQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.546757752 (606h29m42.417840817s) [noise=true , address=45.91.116.12:4225 , piece_id=LLYEKE4TJ54KECLAC7KFUV44IVHYCYLF45I43G5V3DPV5QDBUOIQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.546816995 (606h29m42.41790004s) [noise=true , address=145.79.227.40:4050 , piece_id=LA7KLHHWRNHULLPAZWQ6MLS76SLN3BONYXZJXCUHGFC5JZ4SQRIA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.547216927 (606h29m42.418299962s) [noise=true , address=145.79.227.15:4876 , piece_id=L2OIJAB6OOANT2EPFNVNLTMYCH334JGFBFASD44EHOAWTAHKUV5Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.547615996 (606h29m42.418699042s) [noise=true , address=145.79.227.17:4971 , piece_id=DYZUALBL3V75TKGK57LOJABWVCOD5UVS3WH74I55OVUDGRQUA3QQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.547841055 (606h29m42.4189241s) [noise=true , address=77.90.31.21:4877 , piece_id=BD2FEFNAGYWZYDA2IMOVHA7JE66EXSF3A3PL54Q2DUOJFG5ALG4Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548023322 (606h29m42.419106357s) [noise=true , address=45.91.116.14:4190 , piece_id=HHU5YZBTJEEWYVST6XQCKGWKRLRQIIR2UQTQ27RLK53BNAVOSHFQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548380923 (606h29m42.419463968s) [noise=true , address=45.91.116.11:4120 , piece_id=FZ5L6FM7IHX65EVFGVXXHEF43XQ4EBUH4C7VZUSTH2MVWLNR5M2Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548560475 (606h29m42.41964351s) [noise=true , address=145.79.227.39:4013 , piece_id=I4XP47SFJICIYIPEN4E7UC2HXRXH4IXVJYGHXTZBCUESAHNFQXJA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548719268 (606h29m42.419802313s) [noise=true , address=145.79.227.31:4797 , piece_id=S5YBLITONGU6RHPQENP2TCVVWOHEMXPMYXLNYWBZNXWMFI26PVUA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549024107 (606h29m42.420107152s) [noise=true , address=145.79.227.7:4294 , piece_id=C6N5OIRLV3LLSCEIOGQJKWPKENGZYNAPY6V2AYZNKUMACBP5SLWQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549197708 (606h29m42.420280753s) [noise=true , address=145.79.227.16:4910 , piece_id=O4YO367LEVZSRI5E75YEZ7W7YI2KGKKQDXCOHQ3LALE4WUAPQVHQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549506406 (606h29m42.420589441s) [noise=true , address=145.79.227.12:4765 , piece_id=4ZF5YQP4Z5HBTX5YWUDHZV55Q47WASRLMV6ZYJXJRL2QBAVQQM4Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549971591 (606h29m42.421054626s) [noise=true , address=145.79.227.32:4835 , piece_id=CN4NMCFNYKOCG2JCIEXJBL2PLEYWD5THCZB4WQDURIECFVHLLX6Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.551571588 (606h29m42.422654623s) [noise=true , address=145.79.227.24:4581 , piece_id=Y6GBEOR7DQX2H6LBHXYFOXRBX3GP7AKPYVGTQNMUEXKPXLONUUGQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:44.247717212 (606h29m44.118800248s) [upload=531550]: Next returned nil segment Dec 30 21:58:44.247730348 (606h29m44.118813383s) [upload=531550]: Waiting for error group managing segments... Dec 30 21:58:44.532148720 (606h29m44.403231755s) [noise=true , address=145.79.227.31:4797 , piece_id=S5YBLITONGU6RHPQENP2TCVVWOHEMXPMYXLNYWBZNXWMFI26PVUA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.534655293 (606h29m44.405738328s) [noise=true , address=145.79.227.39:4013 , piece_id=I4XP47SFJICIYIPEN4E7UC2HXRXH4IXVJYGHXTZBCUESAHNFQXJA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.544709427 (606h29m44.415792462s) [noise=true , address=145.79.227.12:4765 , piece_id=4ZF5YQP4Z5HBTX5YWUDHZV55Q47WASRLMV6ZYJXJRL2QBAVQQM4Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.545822306 (606h29m44.416905342s) [noise=true , address=145.79.227.7:4294 , piece_id=C6N5OIRLV3LLSCEIOGQJKWPKENGZYNAPY6V2AYZNKUMACBP5SLWQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.549465162 (606h29m44.420548198s) [noise=true , address=145.79.227.24:4581 , piece_id=Y6GBEOR7DQX2H6LBHXYFOXRBX3GP7AKPYVGTQNMUEXKPXLONUUGQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.556935928 (606h29m44.428018964s) [noise=true , address=145.79.227.15:4876 , piece_id=L2OIJAB6OOANT2EPFNVNLTMYCH334JGFBFASD44EHOAWTAHKUV5Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.561283296 (606h29m44.432366331s) [noise=true , address=145.79.227.16:4910 , piece_id=O4YO367LEVZSRI5E75YEZ7W7YI2KGKKQDXCOHQ3LALE4WUAPQVHQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.614194264 (606h29m44.485277309s) [noise=true , address=145.79.227.17:4971 , piece_id=DYZUALBL3V75TKGK57LOJABWVCOD5UVS3WH74I55OVUDGRQUA3QQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.615630689 (606h29m44.486713724s) [noise=true , address=145.79.227.32:4835 , piece_id=CN4NMCFNYKOCG2JCIEXJBL2PLEYWD5THCZB4WQDURIECFVHLLX6Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.639672077 (606h29m44.510755122s) [noise=true , address=145.79.227.23:4547 , piece_id=LYGUMTAP7OPJWGRVFY65GUZFE4BBTV73JTG657DR5OM26LA7UFZA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.699618771 (606h29m44.570701817s) [noise=true , address=77.90.31.21:4877 , piece_id=BD2FEFNAGYWZYDA2IMOVHA7JE66EXSF3A3PL54Q2DUOJFG5ALG4Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.841535416 (606h29m44.712618462s) [noise=true , address=45.91.116.14:4190 , piece_id=HHU5YZBTJEEWYVST6XQCKGWKRLRQIIR2UQTQ27RLK53BNAVOSHFQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.974660841 (606h29m44.845743886s) [noise=true , address=45.91.116.12:4225 , piece_id=LLYEKE4TJ54KECLAC7KFUV44IVHYCYLF45I43G5V3DPV5QDBUOIQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.974681401 (606h29m44.845764436s) [seg_pos={0 0}, upload=531550]: Segment reached optimal threshold of 13 pieces. Dec 30 21:58:44.974855342 (606h29m44.845938377s) [noise=true , address=145.79.227.49:4339 , piece_id=WBBPZK2VY2QVRTQNXEKQ7TRYPWBDYS2TGTKEDAAXQ2AT4TUQXTQQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 30 21:58:44.974861423 (606h29m44.845944458s) [noise=true , address=145.79.227.49:4339 , piece_id=WBBPZK2VY2QVRTQNXEKQ7TRYPWBDYS2TGTKEDAAXQ2AT4TUQXTQQ, seg_pos={0 0}, upload=531550]: Long tail cancellation due to optimal threshold Dec 30 21:58:44.974968166 (606h29m44.846051201s) [noise=true , address=145.79.227.40:4050 , piece_id=LA7KLHHWRNHULLPAZWQ6MLS76SLN3BONYXZJXCUHGFC5JZ4SQRIA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 30 21:58:44.974973627 (606h29m44.846056662s) [noise=true , address=145.79.227.40:4050 , piece_id=LA7KLHHWRNHULLPAZWQ6MLS76SLN3BONYXZJXCUHGFC5JZ4SQRIA, seg_pos={0 0}, upload=531550]: Long tail cancellation due to optimal threshold Dec 30 21:58:44.975076152 (606h29m44.846159187s) [noise=true , address=45.91.116.11:4120 , piece_id=FZ5L6FM7IHX65EVFGVXXHEF43XQ4EBUH4C7VZUSTH2MVWLNR5M2Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 30 21:58:44.975083497 (606h29m44.846166532s) [noise=true , address=45.91.116.11:4120 , piece_id=FZ5L6FM7IHX65EVFGVXXHEF43XQ4EBUH4C7VZUSTH2MVWLNR5M2Q, seg_pos={0 0}, upload=531550]: Long tail cancellation due to optimal threshold Dec 30 21:58:44.975096711 (606h29m44.846179747s) [seg_pos={0 0}, upload=531550]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 21:58:44.975160703 (606h29m44.846243739s) [upload=531550]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 21:58:44.975164691 (606h29m44.846247726s) [upload=531550]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 21:58:45.272782726 (606h29m45.143865772s) [upload=531550]: Done uploading segments... Dec 30 21:58:45.272806352 (606h29m45.143889387s) [upload=531550]: Upload finished. err: <nil> Duration: 6.151552069s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [............XX......X.........X.............................] duration: 15.402477594s trace: https://jaeger.dev.storj.io/trace/5cdd5ba7b1cb22c7 FAIL (2025-12-30 22:01:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 21:37:03.355296401 (606h8m3.226379446s) [noise=true , address=145.79.227.22:4543 , piece_id=WWCZ6SEEXQ2KUJ32AGTIJNMSJROLYSJ4XQGBQJC4Q5DACZVA65RQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.355839374 (606h8m3.22692241s) [noise=true , address=145.79.227.35:4492 , piece_id=HCKFSQMJYEP6B5H4OX5AZ5TWP6Z4H4Z6WA2T7Z2H5KNZ4SDJRMUQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.356356930 (606h8m3.227439965s) [noise=true , address=45.91.116.13:4182 , piece_id=BYAYNP4JAAN3AW2LYSK74WJE4BRWUR5Z6K67MTLK6A73KEJ35WEQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.356675316 (606h8m3.227758361s) [noise=true , address=77.90.31.12:4612 , piece_id=63TS3SUEFUE6PORZ4W76OXRWFL7SBN773UGGY3DU7JPBAW66N7UA, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.356992732 (606h8m3.228075767s) [noise=true , address=64.49.13.140:4507 , piece_id=E7T6WL45F6GHECCRANX6AIKWFRZ77TURRWORYEYI43MET2WG7SRQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.357330644 (606h8m3.228413679s) [noise=true , address=77.90.31.21:4886 , piece_id=ZQRCPTVC3WFVQPZ36AU6KJW7PD3T557CXYQIICLDGOKK6JCPVZ5Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.357663649 (606h8m3.228746684s) [noise=true , address=145.79.227.49:4367 , piece_id=R2T2ABLXBPDVP3ZVA2TAYMCLQQ7JQUVQYCTHCKGMFBRVWCDAO4PQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.359827449 (606h8m3.230910484s) [noise=true , address=145.79.227.34:4918 , piece_id=HUMOHOT3HOYCBJ23WXQXWU3X6C6JQTRLLHTD4RQLCQTCB3HXI7EQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.360086692 (606h8m3.231169727s) [noise=true , address=145.79.227.7:4320 , piece_id=MAICNTHGLZ6W27HXDOOZXOBRGQGEXNSA4BJT2T5D5PHRVV6ZMX2Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.361622167 (606h8m3.232705212s) [noise=true , address=145.79.227.33:4888 , piece_id=I4426P3VETHPOL5WHWJEGVYRXJULTCNDX3WSUTS3XZVY36Q2N44Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.362965514 (606h8m3.234048559s) [noise=true , address=45.91.116.11:4133 , piece_id=NEW7ZOVP5E7HYGYD4LG5UG2MDLAQW2OBRTNZDFIYO4CQYMVAWRPQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.364308883 (606h8m3.235391928s) [noise=true , address=145.79.227.26:4663 , piece_id=DYEFQZL4X74QR2LAYURT5ZTR2IWIDB55VLBQOGZM36MA54P2SWJA, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.365734507 (606h8m3.236817542s) [noise=true , address=145.79.227.40:4059 , piece_id=7WSUBR3C3PWVBCWD6KRGSD6MPX4IH2YQOAVO4J4SWZORNHGS2IPQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.366982764 (606h8m3.238065799s) [noise=true , address=145.79.227.29:4973 , piece_id=5OROXELR55A47BFPSJJEWVQU4YW6CAOK6LMEHCINM4FN3OVI2Y6Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.368390294 (606h8m3.239473339s) [noise=true , address=77.90.31.19:4852 , piece_id=2NYO63GCBGCYBFYTMUGNF2NG5FSJOD3AOUOD2FZZIWDINK4X6DJQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:04.736295138 (606h8m4.607378183s) [upload=531240]: Next returned nil segment Dec 30 21:37:04.736311850 (606h8m4.607394885s) [upload=531240]: Waiting for error group managing segments... Dec 30 21:37:05.104989591 (606h8m4.976072636s) [noise=true , address=64.49.13.140:4507 , piece_id=E7T6WL45F6GHECCRANX6AIKWFRZ77TURRWORYEYI43MET2WG7SRQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.208809748 (606h8m5.079892793s) [noise=true , address=145.79.227.22:4543 , piece_id=WWCZ6SEEXQ2KUJ32AGTIJNMSJROLYSJ4XQGBQJC4Q5DACZVA65RQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.441125389 (606h8m5.312208444s) [noise=true , address=77.90.31.19:4852 , piece_id=2NYO63GCBGCYBFYTMUGNF2NG5FSJOD3AOUOD2FZZIWDINK4X6DJQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.483630904 (606h8m5.35471394s) [noise=true , address=77.90.31.12:4612 , piece_id=63TS3SUEFUE6PORZ4W76OXRWFL7SBN773UGGY3DU7JPBAW66N7UA, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.494478811 (606h8m5.365561846s) [noise=true , address=77.90.31.11:4559 , piece_id=3DYD4CNPJQN3S4EYZJGREQTTZJDSL2QTC3OT6G2VFUGMLSVCZUFA, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.497452683 (606h8m5.368535729s) [noise=true , address=77.90.31.17:4769 , piece_id=QXDYESBKGXRY7BCFYYF2O7OXS7Y4Z4KUMJB6AXZAWX735SYU56GQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.661529377 (606h8m5.532612412s) [noise=true , address=45.91.116.13:4182 , piece_id=BYAYNP4JAAN3AW2LYSK74WJE4BRWUR5Z6K67MTLK6A73KEJ35WEQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.729773112 (606h8m5.600856157s) [noise=true , address=45.91.116.11:4133 , piece_id=NEW7ZOVP5E7HYGYD4LG5UG2MDLAQW2OBRTNZDFIYO4CQYMVAWRPQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.940743308 (606h8m5.811826353s) [noise=true , address=77.90.31.21:4886 , piece_id=ZQRCPTVC3WFVQPZ36AU6KJW7PD3T557CXYQIICLDGOKK6JCPVZ5Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.084240044 (606h8m5.95532309s) [noise=true , address=145.79.227.30:4784 , piece_id=DSXUDNYCWO5F6IYXMIVJNEASYXIZ7EO7RKAS7QRUYK5OU7GOOB5Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.121095032 (606h8m5.992178067s) [noise=true , address=145.79.227.40:4059 , piece_id=7WSUBR3C3PWVBCWD6KRGSD6MPX4IH2YQOAVO4J4SWZORNHGS2IPQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.169195983 (606h8m6.040279028s) [noise=true , address=145.79.227.49:4367 , piece_id=R2T2ABLXBPDVP3ZVA2TAYMCLQQ7JQUVQYCTHCKGMFBRVWCDAO4PQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.169380634 (606h8m6.040463679s) [noise=true , address=145.79.227.33:4888 , piece_id=I4426P3VETHPOL5WHWJEGVYRXJULTCNDX3WSUTS3XZVY36Q2N44Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.179060175 (606h8m6.050143211s) [noise=true , address=145.79.227.29:4973 , piece_id=5OROXELR55A47BFPSJJEWVQU4YW6CAOK6LMEHCINM4FN3OVI2Y6Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.179080424 (606h8m6.050163459s) [seg_pos={0 0}, upload=531240]: Segment reached optimal threshold of 14 pieces. Dec 30 21:37:06.179262801 (606h8m6.050345837s) [noise=true , address=145.79.227.26:4663 , piece_id=DYEFQZL4X74QR2LAYURT5ZTR2IWIDB55VLBQOGZM36MA54P2SWJA, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SwpAaXTA3BqUvwubzmJkvwokdz6SSpT9t8HPtQcXXJxmcgfm2): protocol: write tcp 188.166.225.170:43184->145.79.227.26:4663: use of closed network connection; write tcp 188.166.225.170:43184->145.79.227.26:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43184->145.79.227.26:4663: use of closed network connection Dec 30 21:37:06.179268703 (606h8m6.050351748s) [noise=true , address=145.79.227.26:4663 , piece_id=DYEFQZL4X74QR2LAYURT5ZTR2IWIDB55VLBQOGZM36MA54P2SWJA, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179363724 (606h8m6.050446769s) [noise=true , address=145.79.227.7:4320 , piece_id=MAICNTHGLZ6W27HXDOOZXOBRGQGEXNSA4BJT2T5D5PHRVV6ZMX2Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126rEgnQw1w7hsFpRqQP9nLaumdBfdtkgYjdw6WE6nXrxk5kj4B): protocol: write tcp 188.166.225.170:45228->145.79.227.7:4320: use of closed network connection; write tcp 188.166.225.170:45228->145.79.227.7:4320: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45228->145.79.227.7:4320: use of closed network connection Dec 30 21:37:06.179369525 (606h8m6.05045257s) [noise=true , address=145.79.227.7:4320 , piece_id=MAICNTHGLZ6W27HXDOOZXOBRGQGEXNSA4BJT2T5D5PHRVV6ZMX2Q, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179532996 (606h8m6.050616061s) [noise=true , address=145.79.227.34:4918 , piece_id=HUMOHOT3HOYCBJ23WXQXWU3X6C6JQTRLLHTD4RQLCQTCB3HXI7EQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF Dec 30 21:37:06.179572371 (606h8m6.050655406s) [noise=true , address=145.79.227.34:4918 , piece_id=HUMOHOT3HOYCBJ23WXQXWU3X6C6JQTRLLHTD4RQLCQTCB3HXI7EQ, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179638587 (606h8m6.050721622s) [noise=true , address=145.79.227.35:4492 , piece_id=HCKFSQMJYEP6B5H4OX5AZ5TWP6Z4H4Z6WA2T7Z2H5KNZ4SDJRMUQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1roa5GGYmqCX2AxSrqFgqyV2CtBEYZavuGTseiwDy4pjkT6bip): protocol: expected piece hash; context canceled; EOF Dec 30 21:37:06.179647855 (606h8m6.05073089s) [noise=true , address=145.79.227.35:4492 , piece_id=HCKFSQMJYEP6B5H4OX5AZ5TWP6Z4H4Z6WA2T7Z2H5KNZ4SDJRMUQ, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179666641 (606h8m6.050749686s) [seg_pos={0 0}, upload=531240]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 21:37:06.179742766 (606h8m6.050825811s) [upload=531240]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 21:37:06.179747545 (606h8m6.05083059s) [upload=531240]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 21:37:06.490986034 (606h8m6.362069079s) [upload=531240]: Done uploading segments... Dec 30 21:37:06.491006261 (606h8m6.362089307s) [upload=531240]: Upload finished. err: <nil> Duration: 15.402512031s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [X.X.........X..X...X.........X.........................X....] duration: 6.065314152s trace: https://jaeger.dev.storj.io/trace/57a7ab98e7eb0d3f FAIL (2025-12-30 22:01:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 21:58:42.298454527 (606h29m42.169537592s) [upload=531550]: Starting upload Dec 30 21:58:42.298523438 (606h29m42.169606473s) [upload=531550]: Done starting upload Dec 30 21:58:42.298568774 (606h29m42.169651809s) [upload=531550]: Uploading segments... Dec 30 21:58:42.298697850 (606h29m42.169780885s) [upload=531550]: Got next segment. Inline: false Dec 30 21:58:42.298728649 (606h29m42.169811694s) [upload=531550]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 21:58:42.298733227 (606h29m42.169816262s) [upload=531550]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 21:58:42.545734213 (606h29m42.416817288s) [seg_pos={0 0}, upload=531550]: Begin upload segment... Dec 30 21:58:42.545786032 (606h29m42.416869078s) [seg_pos={0 0}, upload=531550]: Done begin upload segment. Dec 30 21:58:42.545855104 (606h29m42.416938149s) [noise=true , address=145.79.227.23:4547 , piece_id=LYGUMTAP7OPJWGRVFY65GUZFE4BBTV73JTG657DR5OM26LA7UFZA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.545897384 (606h29m42.416980449s) [noise=true , address=145.79.227.49:4339 , piece_id=WBBPZK2VY2QVRTQNXEKQ7TRYPWBDYS2TGTKEDAAXQ2AT4TUQXTQQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.546757752 (606h29m42.417840817s) [noise=true , address=45.91.116.12:4225 , piece_id=LLYEKE4TJ54KECLAC7KFUV44IVHYCYLF45I43G5V3DPV5QDBUOIQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.546816995 (606h29m42.41790004s) [noise=true , address=145.79.227.40:4050 , piece_id=LA7KLHHWRNHULLPAZWQ6MLS76SLN3BONYXZJXCUHGFC5JZ4SQRIA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.547216927 (606h29m42.418299962s) [noise=true , address=145.79.227.15:4876 , piece_id=L2OIJAB6OOANT2EPFNVNLTMYCH334JGFBFASD44EHOAWTAHKUV5Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.547615996 (606h29m42.418699042s) [noise=true , address=145.79.227.17:4971 , piece_id=DYZUALBL3V75TKGK57LOJABWVCOD5UVS3WH74I55OVUDGRQUA3QQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.547841055 (606h29m42.4189241s) [noise=true , address=77.90.31.21:4877 , piece_id=BD2FEFNAGYWZYDA2IMOVHA7JE66EXSF3A3PL54Q2DUOJFG5ALG4Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548023322 (606h29m42.419106357s) [noise=true , address=45.91.116.14:4190 , piece_id=HHU5YZBTJEEWYVST6XQCKGWKRLRQIIR2UQTQ27RLK53BNAVOSHFQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548380923 (606h29m42.419463968s) [noise=true , address=45.91.116.11:4120 , piece_id=FZ5L6FM7IHX65EVFGVXXHEF43XQ4EBUH4C7VZUSTH2MVWLNR5M2Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548560475 (606h29m42.41964351s) [noise=true , address=145.79.227.39:4013 , piece_id=I4XP47SFJICIYIPEN4E7UC2HXRXH4IXVJYGHXTZBCUESAHNFQXJA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548719268 (606h29m42.419802313s) [noise=true , address=145.79.227.31:4797 , piece_id=S5YBLITONGU6RHPQENP2TCVVWOHEMXPMYXLNYWBZNXWMFI26PVUA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549024107 (606h29m42.420107152s) [noise=true , address=145.79.227.7:4294 , piece_id=C6N5OIRLV3LLSCEIOGQJKWPKENGZYNAPY6V2AYZNKUMACBP5SLWQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549197708 (606h29m42.420280753s) [noise=true , address=145.79.227.16:4910 , piece_id=O4YO367LEVZSRI5E75YEZ7W7YI2KGKKQDXCOHQ3LALE4WUAPQVHQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549506406 (606h29m42.420589441s) [noise=true , address=145.79.227.12:4765 , piece_id=4ZF5YQP4Z5HBTX5YWUDHZV55Q47WASRLMV6ZYJXJRL2QBAVQQM4Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549971591 (606h29m42.421054626s) [noise=true , address=145.79.227.32:4835 , piece_id=CN4NMCFNYKOCG2JCIEXJBL2PLEYWD5THCZB4WQDURIECFVHLLX6Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.551571588 (606h29m42.422654623s) [noise=true , address=145.79.227.24:4581 , piece_id=Y6GBEOR7DQX2H6LBHXYFOXRBX3GP7AKPYVGTQNMUEXKPXLONUUGQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:44.247717212 (606h29m44.118800248s) [upload=531550]: Next returned nil segment Dec 30 21:58:44.247730348 (606h29m44.118813383s) [upload=531550]: Waiting for error group managing segments... Dec 30 21:58:44.532148720 (606h29m44.403231755s) [noise=true , address=145.79.227.31:4797 , piece_id=S5YBLITONGU6RHPQENP2TCVVWOHEMXPMYXLNYWBZNXWMFI26PVUA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.534655293 (606h29m44.405738328s) [noise=true , address=145.79.227.39:4013 , piece_id=I4XP47SFJICIYIPEN4E7UC2HXRXH4IXVJYGHXTZBCUESAHNFQXJA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.544709427 (606h29m44.415792462s) [noise=true , address=145.79.227.12:4765 , piece_id=4ZF5YQP4Z5HBTX5YWUDHZV55Q47WASRLMV6ZYJXJRL2QBAVQQM4Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.545822306 (606h29m44.416905342s) [noise=true , address=145.79.227.7:4294 , piece_id=C6N5OIRLV3LLSCEIOGQJKWPKENGZYNAPY6V2AYZNKUMACBP5SLWQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.549465162 (606h29m44.420548198s) [noise=true , address=145.79.227.24:4581 , piece_id=Y6GBEOR7DQX2H6LBHXYFOXRBX3GP7AKPYVGTQNMUEXKPXLONUUGQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.556935928 (606h29m44.428018964s) [noise=true , address=145.79.227.15:4876 , piece_id=L2OIJAB6OOANT2EPFNVNLTMYCH334JGFBFASD44EHOAWTAHKUV5Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.561283296 (606h29m44.432366331s) [noise=true , address=145.79.227.16:4910 , piece_id=O4YO367LEVZSRI5E75YEZ7W7YI2KGKKQDXCOHQ3LALE4WUAPQVHQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.614194264 (606h29m44.485277309s) [noise=true , address=145.79.227.17:4971 , piece_id=DYZUALBL3V75TKGK57LOJABWVCOD5UVS3WH74I55OVUDGRQUA3QQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.615630689 (606h29m44.486713724s) [noise=true , address=145.79.227.32:4835 , piece_id=CN4NMCFNYKOCG2JCIEXJBL2PLEYWD5THCZB4WQDURIECFVHLLX6Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.639672077 (606h29m44.510755122s) [noise=true , address=145.79.227.23:4547 , piece_id=LYGUMTAP7OPJWGRVFY65GUZFE4BBTV73JTG657DR5OM26LA7UFZA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.699618771 (606h29m44.570701817s) [noise=true , address=77.90.31.21:4877 , piece_id=BD2FEFNAGYWZYDA2IMOVHA7JE66EXSF3A3PL54Q2DUOJFG5ALG4Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.841535416 (606h29m44.712618462s) [noise=true , address=45.91.116.14:4190 , piece_id=HHU5YZBTJEEWYVST6XQCKGWKRLRQIIR2UQTQ27RLK53BNAVOSHFQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.974660841 (606h29m44.845743886s) [noise=true , address=45.91.116.12:4225 , piece_id=LLYEKE4TJ54KECLAC7KFUV44IVHYCYLF45I43G5V3DPV5QDBUOIQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.974681401 (606h29m44.845764436s) [seg_pos={0 0}, upload=531550]: Segment reached optimal threshold of 13 pieces. Dec 30 21:58:44.974855342 (606h29m44.845938377s) [noise=true , address=145.79.227.49:4339 , piece_id=WBBPZK2VY2QVRTQNXEKQ7TRYPWBDYS2TGTKEDAAXQ2AT4TUQXTQQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 30 21:58:44.974861423 (606h29m44.845944458s) [noise=true , address=145.79.227.49:4339 , piece_id=WBBPZK2VY2QVRTQNXEKQ7TRYPWBDYS2TGTKEDAAXQ2AT4TUQXTQQ, seg_pos={0 0}, upload=531550]: Long tail cancellation due to optimal threshold Dec 30 21:58:44.974968166 (606h29m44.846051201s) [noise=true , address=145.79.227.40:4050 , piece_id=LA7KLHHWRNHULLPAZWQ6MLS76SLN3BONYXZJXCUHGFC5JZ4SQRIA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 30 21:58:44.974973627 (606h29m44.846056662s) [noise=true , address=145.79.227.40:4050 , piece_id=LA7KLHHWRNHULLPAZWQ6MLS76SLN3BONYXZJXCUHGFC5JZ4SQRIA, seg_pos={0 0}, upload=531550]: Long tail cancellation due to optimal threshold Dec 30 21:58:44.975076152 (606h29m44.846159187s) [noise=true , address=45.91.116.11:4120 , piece_id=FZ5L6FM7IHX65EVFGVXXHEF43XQ4EBUH4C7VZUSTH2MVWLNR5M2Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 30 21:58:44.975083497 (606h29m44.846166532s) [noise=true , address=45.91.116.11:4120 , piece_id=FZ5L6FM7IHX65EVFGVXXHEF43XQ4EBUH4C7VZUSTH2MVWLNR5M2Q, seg_pos={0 0}, upload=531550]: Long tail cancellation due to optimal threshold Dec 30 21:58:44.975096711 (606h29m44.846179747s) [seg_pos={0 0}, upload=531550]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 21:58:44.975160703 (606h29m44.846243739s) [upload=531550]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 21:58:44.975164691 (606h29m44.846247726s) [upload=531550]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 21:58:45.272782726 (606h29m45.143865772s) [upload=531550]: Done uploading segments... Dec 30 21:58:45.272806352 (606h29m45.143889387s) [upload=531550]: Upload finished. err: <nil> Duration: 6.065336926s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................................................X...........] duration: 3.567166252s trace: https://jaeger.dev.storj.io/trace/22a47ec27f9a101c FAIL (2025-12-30 22:01:00 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.567193884s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.X.X.........X..X...X.........X.........................X...] duration: 5.835457523s trace: https://jaeger.dev.storj.io/trace/57a7ab98e7eb0d3f FAIL (2025-12-30 22:00:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 21:58:42.298454527 (606h29m42.169537592s) [upload=531550]: Starting upload Dec 30 21:58:42.298523438 (606h29m42.169606473s) [upload=531550]: Done starting upload Dec 30 21:58:42.298568774 (606h29m42.169651809s) [upload=531550]: Uploading segments... Dec 30 21:58:42.298697850 (606h29m42.169780885s) [upload=531550]: Got next segment. Inline: false Dec 30 21:58:42.298728649 (606h29m42.169811694s) [upload=531550]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 21:58:42.298733227 (606h29m42.169816262s) [upload=531550]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 21:58:42.545734213 (606h29m42.416817288s) [seg_pos={0 0}, upload=531550]: Begin upload segment... Dec 30 21:58:42.545786032 (606h29m42.416869078s) [seg_pos={0 0}, upload=531550]: Done begin upload segment. Dec 30 21:58:42.545855104 (606h29m42.416938149s) [noise=true , address=145.79.227.23:4547 , piece_id=LYGUMTAP7OPJWGRVFY65GUZFE4BBTV73JTG657DR5OM26LA7UFZA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.545897384 (606h29m42.416980449s) [noise=true , address=145.79.227.49:4339 , piece_id=WBBPZK2VY2QVRTQNXEKQ7TRYPWBDYS2TGTKEDAAXQ2AT4TUQXTQQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.546757752 (606h29m42.417840817s) [noise=true , address=45.91.116.12:4225 , piece_id=LLYEKE4TJ54KECLAC7KFUV44IVHYCYLF45I43G5V3DPV5QDBUOIQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.546816995 (606h29m42.41790004s) [noise=true , address=145.79.227.40:4050 , piece_id=LA7KLHHWRNHULLPAZWQ6MLS76SLN3BONYXZJXCUHGFC5JZ4SQRIA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.547216927 (606h29m42.418299962s) [noise=true , address=145.79.227.15:4876 , piece_id=L2OIJAB6OOANT2EPFNVNLTMYCH334JGFBFASD44EHOAWTAHKUV5Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.547615996 (606h29m42.418699042s) [noise=true , address=145.79.227.17:4971 , piece_id=DYZUALBL3V75TKGK57LOJABWVCOD5UVS3WH74I55OVUDGRQUA3QQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.547841055 (606h29m42.4189241s) [noise=true , address=77.90.31.21:4877 , piece_id=BD2FEFNAGYWZYDA2IMOVHA7JE66EXSF3A3PL54Q2DUOJFG5ALG4Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548023322 (606h29m42.419106357s) [noise=true , address=45.91.116.14:4190 , piece_id=HHU5YZBTJEEWYVST6XQCKGWKRLRQIIR2UQTQ27RLK53BNAVOSHFQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548380923 (606h29m42.419463968s) [noise=true , address=45.91.116.11:4120 , piece_id=FZ5L6FM7IHX65EVFGVXXHEF43XQ4EBUH4C7VZUSTH2MVWLNR5M2Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548560475 (606h29m42.41964351s) [noise=true , address=145.79.227.39:4013 , piece_id=I4XP47SFJICIYIPEN4E7UC2HXRXH4IXVJYGHXTZBCUESAHNFQXJA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548719268 (606h29m42.419802313s) [noise=true , address=145.79.227.31:4797 , piece_id=S5YBLITONGU6RHPQENP2TCVVWOHEMXPMYXLNYWBZNXWMFI26PVUA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549024107 (606h29m42.420107152s) [noise=true , address=145.79.227.7:4294 , piece_id=C6N5OIRLV3LLSCEIOGQJKWPKENGZYNAPY6V2AYZNKUMACBP5SLWQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549197708 (606h29m42.420280753s) [noise=true , address=145.79.227.16:4910 , piece_id=O4YO367LEVZSRI5E75YEZ7W7YI2KGKKQDXCOHQ3LALE4WUAPQVHQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549506406 (606h29m42.420589441s) [noise=true , address=145.79.227.12:4765 , piece_id=4ZF5YQP4Z5HBTX5YWUDHZV55Q47WASRLMV6ZYJXJRL2QBAVQQM4Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549971591 (606h29m42.421054626s) [noise=true , address=145.79.227.32:4835 , piece_id=CN4NMCFNYKOCG2JCIEXJBL2PLEYWD5THCZB4WQDURIECFVHLLX6Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.551571588 (606h29m42.422654623s) [noise=true , address=145.79.227.24:4581 , piece_id=Y6GBEOR7DQX2H6LBHXYFOXRBX3GP7AKPYVGTQNMUEXKPXLONUUGQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:44.247717212 (606h29m44.118800248s) [upload=531550]: Next returned nil segment Dec 30 21:58:44.247730348 (606h29m44.118813383s) [upload=531550]: Waiting for error group managing segments... Dec 30 21:58:44.532148720 (606h29m44.403231755s) [noise=true , address=145.79.227.31:4797 , piece_id=S5YBLITONGU6RHPQENP2TCVVWOHEMXPMYXLNYWBZNXWMFI26PVUA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.534655293 (606h29m44.405738328s) [noise=true , address=145.79.227.39:4013 , piece_id=I4XP47SFJICIYIPEN4E7UC2HXRXH4IXVJYGHXTZBCUESAHNFQXJA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.544709427 (606h29m44.415792462s) [noise=true , address=145.79.227.12:4765 , piece_id=4ZF5YQP4Z5HBTX5YWUDHZV55Q47WASRLMV6ZYJXJRL2QBAVQQM4Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.545822306 (606h29m44.416905342s) [noise=true , address=145.79.227.7:4294 , piece_id=C6N5OIRLV3LLSCEIOGQJKWPKENGZYNAPY6V2AYZNKUMACBP5SLWQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.549465162 (606h29m44.420548198s) [noise=true , address=145.79.227.24:4581 , piece_id=Y6GBEOR7DQX2H6LBHXYFOXRBX3GP7AKPYVGTQNMUEXKPXLONUUGQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.556935928 (606h29m44.428018964s) [noise=true , address=145.79.227.15:4876 , piece_id=L2OIJAB6OOANT2EPFNVNLTMYCH334JGFBFASD44EHOAWTAHKUV5Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.561283296 (606h29m44.432366331s) [noise=true , address=145.79.227.16:4910 , piece_id=O4YO367LEVZSRI5E75YEZ7W7YI2KGKKQDXCOHQ3LALE4WUAPQVHQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.614194264 (606h29m44.485277309s) [noise=true , address=145.79.227.17:4971 , piece_id=DYZUALBL3V75TKGK57LOJABWVCOD5UVS3WH74I55OVUDGRQUA3QQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.615630689 (606h29m44.486713724s) [noise=true , address=145.79.227.32:4835 , piece_id=CN4NMCFNYKOCG2JCIEXJBL2PLEYWD5THCZB4WQDURIECFVHLLX6Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.639672077 (606h29m44.510755122s) [noise=true , address=145.79.227.23:4547 , piece_id=LYGUMTAP7OPJWGRVFY65GUZFE4BBTV73JTG657DR5OM26LA7UFZA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.699618771 (606h29m44.570701817s) [noise=true , address=77.90.31.21:4877 , piece_id=BD2FEFNAGYWZYDA2IMOVHA7JE66EXSF3A3PL54Q2DUOJFG5ALG4Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.841535416 (606h29m44.712618462s) [noise=true , address=45.91.116.14:4190 , piece_id=HHU5YZBTJEEWYVST6XQCKGWKRLRQIIR2UQTQ27RLK53BNAVOSHFQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.974660841 (606h29m44.845743886s) [noise=true , address=45.91.116.12:4225 , piece_id=LLYEKE4TJ54KECLAC7KFUV44IVHYCYLF45I43G5V3DPV5QDBUOIQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.974681401 (606h29m44.845764436s) [seg_pos={0 0}, upload=531550]: Segment reached optimal threshold of 13 pieces. Dec 30 21:58:44.974855342 (606h29m44.845938377s) [noise=true , address=145.79.227.49:4339 , piece_id=WBBPZK2VY2QVRTQNXEKQ7TRYPWBDYS2TGTKEDAAXQ2AT4TUQXTQQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 30 21:58:44.974861423 (606h29m44.845944458s) [noise=true , address=145.79.227.49:4339 , piece_id=WBBPZK2VY2QVRTQNXEKQ7TRYPWBDYS2TGTKEDAAXQ2AT4TUQXTQQ, seg_pos={0 0}, upload=531550]: Long tail cancellation due to optimal threshold Dec 30 21:58:44.974968166 (606h29m44.846051201s) [noise=true , address=145.79.227.40:4050 , piece_id=LA7KLHHWRNHULLPAZWQ6MLS76SLN3BONYXZJXCUHGFC5JZ4SQRIA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 30 21:58:44.974973627 (606h29m44.846056662s) [noise=true , address=145.79.227.40:4050 , piece_id=LA7KLHHWRNHULLPAZWQ6MLS76SLN3BONYXZJXCUHGFC5JZ4SQRIA, seg_pos={0 0}, upload=531550]: Long tail cancellation due to optimal threshold Dec 30 21:58:44.975076152 (606h29m44.846159187s) [noise=true , address=45.91.116.11:4120 , piece_id=FZ5L6FM7IHX65EVFGVXXHEF43XQ4EBUH4C7VZUSTH2MVWLNR5M2Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 30 21:58:44.975083497 (606h29m44.846166532s) [noise=true , address=45.91.116.11:4120 , piece_id=FZ5L6FM7IHX65EVFGVXXHEF43XQ4EBUH4C7VZUSTH2MVWLNR5M2Q, seg_pos={0 0}, upload=531550]: Long tail cancellation due to optimal threshold Dec 30 21:58:44.975096711 (606h29m44.846179747s) [seg_pos={0 0}, upload=531550]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 21:58:44.975160703 (606h29m44.846243739s) [upload=531550]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 21:58:44.975164691 (606h29m44.846247726s) [upload=531550]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 21:58:45.272782726 (606h29m45.143865772s) [upload=531550]: Done uploading segments... Dec 30 21:58:45.272806352 (606h29m45.143889387s) [upload=531550]: Upload finished. err: <nil> Duration: 5.835491316s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.............XX......X.........X............................] duration: 15.540037452s trace: https://jaeger.dev.storj.io/trace/5cdd5ba7b1cb22c7 FAIL (2025-12-30 22:00:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 21:37:03.355296401 (606h8m3.226379446s) [noise=true , address=145.79.227.22:4543 , piece_id=WWCZ6SEEXQ2KUJ32AGTIJNMSJROLYSJ4XQGBQJC4Q5DACZVA65RQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.355839374 (606h8m3.22692241s) [noise=true , address=145.79.227.35:4492 , piece_id=HCKFSQMJYEP6B5H4OX5AZ5TWP6Z4H4Z6WA2T7Z2H5KNZ4SDJRMUQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.356356930 (606h8m3.227439965s) [noise=true , address=45.91.116.13:4182 , piece_id=BYAYNP4JAAN3AW2LYSK74WJE4BRWUR5Z6K67MTLK6A73KEJ35WEQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.356675316 (606h8m3.227758361s) [noise=true , address=77.90.31.12:4612 , piece_id=63TS3SUEFUE6PORZ4W76OXRWFL7SBN773UGGY3DU7JPBAW66N7UA, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.356992732 (606h8m3.228075767s) [noise=true , address=64.49.13.140:4507 , piece_id=E7T6WL45F6GHECCRANX6AIKWFRZ77TURRWORYEYI43MET2WG7SRQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.357330644 (606h8m3.228413679s) [noise=true , address=77.90.31.21:4886 , piece_id=ZQRCPTVC3WFVQPZ36AU6KJW7PD3T557CXYQIICLDGOKK6JCPVZ5Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.357663649 (606h8m3.228746684s) [noise=true , address=145.79.227.49:4367 , piece_id=R2T2ABLXBPDVP3ZVA2TAYMCLQQ7JQUVQYCTHCKGMFBRVWCDAO4PQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.359827449 (606h8m3.230910484s) [noise=true , address=145.79.227.34:4918 , piece_id=HUMOHOT3HOYCBJ23WXQXWU3X6C6JQTRLLHTD4RQLCQTCB3HXI7EQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.360086692 (606h8m3.231169727s) [noise=true , address=145.79.227.7:4320 , piece_id=MAICNTHGLZ6W27HXDOOZXOBRGQGEXNSA4BJT2T5D5PHRVV6ZMX2Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.361622167 (606h8m3.232705212s) [noise=true , address=145.79.227.33:4888 , piece_id=I4426P3VETHPOL5WHWJEGVYRXJULTCNDX3WSUTS3XZVY36Q2N44Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.362965514 (606h8m3.234048559s) [noise=true , address=45.91.116.11:4133 , piece_id=NEW7ZOVP5E7HYGYD4LG5UG2MDLAQW2OBRTNZDFIYO4CQYMVAWRPQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.364308883 (606h8m3.235391928s) [noise=true , address=145.79.227.26:4663 , piece_id=DYEFQZL4X74QR2LAYURT5ZTR2IWIDB55VLBQOGZM36MA54P2SWJA, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.365734507 (606h8m3.236817542s) [noise=true , address=145.79.227.40:4059 , piece_id=7WSUBR3C3PWVBCWD6KRGSD6MPX4IH2YQOAVO4J4SWZORNHGS2IPQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.366982764 (606h8m3.238065799s) [noise=true , address=145.79.227.29:4973 , piece_id=5OROXELR55A47BFPSJJEWVQU4YW6CAOK6LMEHCINM4FN3OVI2Y6Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.368390294 (606h8m3.239473339s) [noise=true , address=77.90.31.19:4852 , piece_id=2NYO63GCBGCYBFYTMUGNF2NG5FSJOD3AOUOD2FZZIWDINK4X6DJQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:04.736295138 (606h8m4.607378183s) [upload=531240]: Next returned nil segment Dec 30 21:37:04.736311850 (606h8m4.607394885s) [upload=531240]: Waiting for error group managing segments... Dec 30 21:37:05.104989591 (606h8m4.976072636s) [noise=true , address=64.49.13.140:4507 , piece_id=E7T6WL45F6GHECCRANX6AIKWFRZ77TURRWORYEYI43MET2WG7SRQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.208809748 (606h8m5.079892793s) [noise=true , address=145.79.227.22:4543 , piece_id=WWCZ6SEEXQ2KUJ32AGTIJNMSJROLYSJ4XQGBQJC4Q5DACZVA65RQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.441125389 (606h8m5.312208444s) [noise=true , address=77.90.31.19:4852 , piece_id=2NYO63GCBGCYBFYTMUGNF2NG5FSJOD3AOUOD2FZZIWDINK4X6DJQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.483630904 (606h8m5.35471394s) [noise=true , address=77.90.31.12:4612 , piece_id=63TS3SUEFUE6PORZ4W76OXRWFL7SBN773UGGY3DU7JPBAW66N7UA, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.494478811 (606h8m5.365561846s) [noise=true , address=77.90.31.11:4559 , piece_id=3DYD4CNPJQN3S4EYZJGREQTTZJDSL2QTC3OT6G2VFUGMLSVCZUFA, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.497452683 (606h8m5.368535729s) [noise=true , address=77.90.31.17:4769 , piece_id=QXDYESBKGXRY7BCFYYF2O7OXS7Y4Z4KUMJB6AXZAWX735SYU56GQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.661529377 (606h8m5.532612412s) [noise=true , address=45.91.116.13:4182 , piece_id=BYAYNP4JAAN3AW2LYSK74WJE4BRWUR5Z6K67MTLK6A73KEJ35WEQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.729773112 (606h8m5.600856157s) [noise=true , address=45.91.116.11:4133 , piece_id=NEW7ZOVP5E7HYGYD4LG5UG2MDLAQW2OBRTNZDFIYO4CQYMVAWRPQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.940743308 (606h8m5.811826353s) [noise=true , address=77.90.31.21:4886 , piece_id=ZQRCPTVC3WFVQPZ36AU6KJW7PD3T557CXYQIICLDGOKK6JCPVZ5Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.084240044 (606h8m5.95532309s) [noise=true , address=145.79.227.30:4784 , piece_id=DSXUDNYCWO5F6IYXMIVJNEASYXIZ7EO7RKAS7QRUYK5OU7GOOB5Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.121095032 (606h8m5.992178067s) [noise=true , address=145.79.227.40:4059 , piece_id=7WSUBR3C3PWVBCWD6KRGSD6MPX4IH2YQOAVO4J4SWZORNHGS2IPQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.169195983 (606h8m6.040279028s) [noise=true , address=145.79.227.49:4367 , piece_id=R2T2ABLXBPDVP3ZVA2TAYMCLQQ7JQUVQYCTHCKGMFBRVWCDAO4PQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.169380634 (606h8m6.040463679s) [noise=true , address=145.79.227.33:4888 , piece_id=I4426P3VETHPOL5WHWJEGVYRXJULTCNDX3WSUTS3XZVY36Q2N44Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.179060175 (606h8m6.050143211s) [noise=true , address=145.79.227.29:4973 , piece_id=5OROXELR55A47BFPSJJEWVQU4YW6CAOK6LMEHCINM4FN3OVI2Y6Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.179080424 (606h8m6.050163459s) [seg_pos={0 0}, upload=531240]: Segment reached optimal threshold of 14 pieces. Dec 30 21:37:06.179262801 (606h8m6.050345837s) [noise=true , address=145.79.227.26:4663 , piece_id=DYEFQZL4X74QR2LAYURT5ZTR2IWIDB55VLBQOGZM36MA54P2SWJA, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SwpAaXTA3BqUvwubzmJkvwokdz6SSpT9t8HPtQcXXJxmcgfm2): protocol: write tcp 188.166.225.170:43184->145.79.227.26:4663: use of closed network connection; write tcp 188.166.225.170:43184->145.79.227.26:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43184->145.79.227.26:4663: use of closed network connection Dec 30 21:37:06.179268703 (606h8m6.050351748s) [noise=true , address=145.79.227.26:4663 , piece_id=DYEFQZL4X74QR2LAYURT5ZTR2IWIDB55VLBQOGZM36MA54P2SWJA, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179363724 (606h8m6.050446769s) [noise=true , address=145.79.227.7:4320 , piece_id=MAICNTHGLZ6W27HXDOOZXOBRGQGEXNSA4BJT2T5D5PHRVV6ZMX2Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126rEgnQw1w7hsFpRqQP9nLaumdBfdtkgYjdw6WE6nXrxk5kj4B): protocol: write tcp 188.166.225.170:45228->145.79.227.7:4320: use of closed network connection; write tcp 188.166.225.170:45228->145.79.227.7:4320: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45228->145.79.227.7:4320: use of closed network connection Dec 30 21:37:06.179369525 (606h8m6.05045257s) [noise=true , address=145.79.227.7:4320 , piece_id=MAICNTHGLZ6W27HXDOOZXOBRGQGEXNSA4BJT2T5D5PHRVV6ZMX2Q, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179532996 (606h8m6.050616061s) [noise=true , address=145.79.227.34:4918 , piece_id=HUMOHOT3HOYCBJ23WXQXWU3X6C6JQTRLLHTD4RQLCQTCB3HXI7EQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF Dec 30 21:37:06.179572371 (606h8m6.050655406s) [noise=true , address=145.79.227.34:4918 , piece_id=HUMOHOT3HOYCBJ23WXQXWU3X6C6JQTRLLHTD4RQLCQTCB3HXI7EQ, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179638587 (606h8m6.050721622s) [noise=true , address=145.79.227.35:4492 , piece_id=HCKFSQMJYEP6B5H4OX5AZ5TWP6Z4H4Z6WA2T7Z2H5KNZ4SDJRMUQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1roa5GGYmqCX2AxSrqFgqyV2CtBEYZavuGTseiwDy4pjkT6bip): protocol: expected piece hash; context canceled; EOF Dec 30 21:37:06.179647855 (606h8m6.05073089s) [noise=true , address=145.79.227.35:4492 , piece_id=HCKFSQMJYEP6B5H4OX5AZ5TWP6Z4H4Z6WA2T7Z2H5KNZ4SDJRMUQ, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179666641 (606h8m6.050749686s) [seg_pos={0 0}, upload=531240]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 21:37:06.179742766 (606h8m6.050825811s) [upload=531240]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 21:37:06.179747545 (606h8m6.05083059s) [upload=531240]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 21:37:06.490986034 (606h8m6.362069079s) [upload=531240]: Done uploading segments... Dec 30 21:37:06.491006261 (606h8m6.362089307s) [upload=531240]: Upload finished. err: <nil> Duration: 15.540069814s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [..X.X.........X..X...X.........X.........................X..] duration: 16.767061584s trace: https://jaeger.dev.storj.io/trace/57a7ab98e7eb0d3f FAIL (2025-12-30 22:00:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 21:58:42.298454527 (606h29m42.169537592s) [upload=531550]: Starting upload Dec 30 21:58:42.298523438 (606h29m42.169606473s) [upload=531550]: Done starting upload Dec 30 21:58:42.298568774 (606h29m42.169651809s) [upload=531550]: Uploading segments... Dec 30 21:58:42.298697850 (606h29m42.169780885s) [upload=531550]: Got next segment. Inline: false Dec 30 21:58:42.298728649 (606h29m42.169811694s) [upload=531550]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 21:58:42.298733227 (606h29m42.169816262s) [upload=531550]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 21:58:42.545734213 (606h29m42.416817288s) [seg_pos={0 0}, upload=531550]: Begin upload segment... Dec 30 21:58:42.545786032 (606h29m42.416869078s) [seg_pos={0 0}, upload=531550]: Done begin upload segment. Dec 30 21:58:42.545855104 (606h29m42.416938149s) [noise=true , address=145.79.227.23:4547 , piece_id=LYGUMTAP7OPJWGRVFY65GUZFE4BBTV73JTG657DR5OM26LA7UFZA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.545897384 (606h29m42.416980449s) [noise=true , address=145.79.227.49:4339 , piece_id=WBBPZK2VY2QVRTQNXEKQ7TRYPWBDYS2TGTKEDAAXQ2AT4TUQXTQQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.546757752 (606h29m42.417840817s) [noise=true , address=45.91.116.12:4225 , piece_id=LLYEKE4TJ54KECLAC7KFUV44IVHYCYLF45I43G5V3DPV5QDBUOIQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.546816995 (606h29m42.41790004s) [noise=true , address=145.79.227.40:4050 , piece_id=LA7KLHHWRNHULLPAZWQ6MLS76SLN3BONYXZJXCUHGFC5JZ4SQRIA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.547216927 (606h29m42.418299962s) [noise=true , address=145.79.227.15:4876 , piece_id=L2OIJAB6OOANT2EPFNVNLTMYCH334JGFBFASD44EHOAWTAHKUV5Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.547615996 (606h29m42.418699042s) [noise=true , address=145.79.227.17:4971 , piece_id=DYZUALBL3V75TKGK57LOJABWVCOD5UVS3WH74I55OVUDGRQUA3QQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.547841055 (606h29m42.4189241s) [noise=true , address=77.90.31.21:4877 , piece_id=BD2FEFNAGYWZYDA2IMOVHA7JE66EXSF3A3PL54Q2DUOJFG5ALG4Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548023322 (606h29m42.419106357s) [noise=true , address=45.91.116.14:4190 , piece_id=HHU5YZBTJEEWYVST6XQCKGWKRLRQIIR2UQTQ27RLK53BNAVOSHFQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548380923 (606h29m42.419463968s) [noise=true , address=45.91.116.11:4120 , piece_id=FZ5L6FM7IHX65EVFGVXXHEF43XQ4EBUH4C7VZUSTH2MVWLNR5M2Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548560475 (606h29m42.41964351s) [noise=true , address=145.79.227.39:4013 , piece_id=I4XP47SFJICIYIPEN4E7UC2HXRXH4IXVJYGHXTZBCUESAHNFQXJA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548719268 (606h29m42.419802313s) [noise=true , address=145.79.227.31:4797 , piece_id=S5YBLITONGU6RHPQENP2TCVVWOHEMXPMYXLNYWBZNXWMFI26PVUA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549024107 (606h29m42.420107152s) [noise=true , address=145.79.227.7:4294 , piece_id=C6N5OIRLV3LLSCEIOGQJKWPKENGZYNAPY6V2AYZNKUMACBP5SLWQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549197708 (606h29m42.420280753s) [noise=true , address=145.79.227.16:4910 , piece_id=O4YO367LEVZSRI5E75YEZ7W7YI2KGKKQDXCOHQ3LALE4WUAPQVHQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549506406 (606h29m42.420589441s) [noise=true , address=145.79.227.12:4765 , piece_id=4ZF5YQP4Z5HBTX5YWUDHZV55Q47WASRLMV6ZYJXJRL2QBAVQQM4Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549971591 (606h29m42.421054626s) [noise=true , address=145.79.227.32:4835 , piece_id=CN4NMCFNYKOCG2JCIEXJBL2PLEYWD5THCZB4WQDURIECFVHLLX6Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.551571588 (606h29m42.422654623s) [noise=true , address=145.79.227.24:4581 , piece_id=Y6GBEOR7DQX2H6LBHXYFOXRBX3GP7AKPYVGTQNMUEXKPXLONUUGQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:44.247717212 (606h29m44.118800248s) [upload=531550]: Next returned nil segment Dec 30 21:58:44.247730348 (606h29m44.118813383s) [upload=531550]: Waiting for error group managing segments... Dec 30 21:58:44.532148720 (606h29m44.403231755s) [noise=true , address=145.79.227.31:4797 , piece_id=S5YBLITONGU6RHPQENP2TCVVWOHEMXPMYXLNYWBZNXWMFI26PVUA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.534655293 (606h29m44.405738328s) [noise=true , address=145.79.227.39:4013 , piece_id=I4XP47SFJICIYIPEN4E7UC2HXRXH4IXVJYGHXTZBCUESAHNFQXJA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.544709427 (606h29m44.415792462s) [noise=true , address=145.79.227.12:4765 , piece_id=4ZF5YQP4Z5HBTX5YWUDHZV55Q47WASRLMV6ZYJXJRL2QBAVQQM4Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.545822306 (606h29m44.416905342s) [noise=true , address=145.79.227.7:4294 , piece_id=C6N5OIRLV3LLSCEIOGQJKWPKENGZYNAPY6V2AYZNKUMACBP5SLWQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.549465162 (606h29m44.420548198s) [noise=true , address=145.79.227.24:4581 , piece_id=Y6GBEOR7DQX2H6LBHXYFOXRBX3GP7AKPYVGTQNMUEXKPXLONUUGQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.556935928 (606h29m44.428018964s) [noise=true , address=145.79.227.15:4876 , piece_id=L2OIJAB6OOANT2EPFNVNLTMYCH334JGFBFASD44EHOAWTAHKUV5Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.561283296 (606h29m44.432366331s) [noise=true , address=145.79.227.16:4910 , piece_id=O4YO367LEVZSRI5E75YEZ7W7YI2KGKKQDXCOHQ3LALE4WUAPQVHQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.614194264 (606h29m44.485277309s) [noise=true , address=145.79.227.17:4971 , piece_id=DYZUALBL3V75TKGK57LOJABWVCOD5UVS3WH74I55OVUDGRQUA3QQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.615630689 (606h29m44.486713724s) [noise=true , address=145.79.227.32:4835 , piece_id=CN4NMCFNYKOCG2JCIEXJBL2PLEYWD5THCZB4WQDURIECFVHLLX6Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.639672077 (606h29m44.510755122s) [noise=true , address=145.79.227.23:4547 , piece_id=LYGUMTAP7OPJWGRVFY65GUZFE4BBTV73JTG657DR5OM26LA7UFZA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.699618771 (606h29m44.570701817s) [noise=true , address=77.90.31.21:4877 , piece_id=BD2FEFNAGYWZYDA2IMOVHA7JE66EXSF3A3PL54Q2DUOJFG5ALG4Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.841535416 (606h29m44.712618462s) [noise=true , address=45.91.116.14:4190 , piece_id=HHU5YZBTJEEWYVST6XQCKGWKRLRQIIR2UQTQ27RLK53BNAVOSHFQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.974660841 (606h29m44.845743886s) [noise=true , address=45.91.116.12:4225 , piece_id=LLYEKE4TJ54KECLAC7KFUV44IVHYCYLF45I43G5V3DPV5QDBUOIQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.974681401 (606h29m44.845764436s) [seg_pos={0 0}, upload=531550]: Segment reached optimal threshold of 13 pieces. Dec 30 21:58:44.974855342 (606h29m44.845938377s) [noise=true , address=145.79.227.49:4339 , piece_id=WBBPZK2VY2QVRTQNXEKQ7TRYPWBDYS2TGTKEDAAXQ2AT4TUQXTQQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 30 21:58:44.974861423 (606h29m44.845944458s) [noise=true , address=145.79.227.49:4339 , piece_id=WBBPZK2VY2QVRTQNXEKQ7TRYPWBDYS2TGTKEDAAXQ2AT4TUQXTQQ, seg_pos={0 0}, upload=531550]: Long tail cancellation due to optimal threshold Dec 30 21:58:44.974968166 (606h29m44.846051201s) [noise=true , address=145.79.227.40:4050 , piece_id=LA7KLHHWRNHULLPAZWQ6MLS76SLN3BONYXZJXCUHGFC5JZ4SQRIA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 30 21:58:44.974973627 (606h29m44.846056662s) [noise=true , address=145.79.227.40:4050 , piece_id=LA7KLHHWRNHULLPAZWQ6MLS76SLN3BONYXZJXCUHGFC5JZ4SQRIA, seg_pos={0 0}, upload=531550]: Long tail cancellation due to optimal threshold Dec 30 21:58:44.975076152 (606h29m44.846159187s) [noise=true , address=45.91.116.11:4120 , piece_id=FZ5L6FM7IHX65EVFGVXXHEF43XQ4EBUH4C7VZUSTH2MVWLNR5M2Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 30 21:58:44.975083497 (606h29m44.846166532s) [noise=true , address=45.91.116.11:4120 , piece_id=FZ5L6FM7IHX65EVFGVXXHEF43XQ4EBUH4C7VZUSTH2MVWLNR5M2Q, seg_pos={0 0}, upload=531550]: Long tail cancellation due to optimal threshold Dec 30 21:58:44.975096711 (606h29m44.846179747s) [seg_pos={0 0}, upload=531550]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 21:58:44.975160703 (606h29m44.846243739s) [upload=531550]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 21:58:44.975164691 (606h29m44.846247726s) [upload=531550]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 21:58:45.272782726 (606h29m45.143865772s) [upload=531550]: Done uploading segments... Dec 30 21:58:45.272806352 (606h29m45.143889387s) [upload=531550]: Upload finished. err: <nil> Duration: 16.767089868s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................................................X..........] duration: 3.631097655s trace: https://jaeger.dev.storj.io/trace/22a47ec27f9a101c FAIL (2025-12-30 22:00:11 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.631137411s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................................................X.........] duration: 3.637103018s trace: https://jaeger.dev.storj.io/trace/22a47ec27f9a101c FAIL (2025-12-30 21:59:50 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.637131652s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [..............XX......X.........X...........................] duration: 15.301812629s trace: https://jaeger.dev.storj.io/trace/5cdd5ba7b1cb22c7 FAIL (2025-12-30 21:59:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 21:37:03.355296401 (606h8m3.226379446s) [noise=true , address=145.79.227.22:4543 , piece_id=WWCZ6SEEXQ2KUJ32AGTIJNMSJROLYSJ4XQGBQJC4Q5DACZVA65RQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.355839374 (606h8m3.22692241s) [noise=true , address=145.79.227.35:4492 , piece_id=HCKFSQMJYEP6B5H4OX5AZ5TWP6Z4H4Z6WA2T7Z2H5KNZ4SDJRMUQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.356356930 (606h8m3.227439965s) [noise=true , address=45.91.116.13:4182 , piece_id=BYAYNP4JAAN3AW2LYSK74WJE4BRWUR5Z6K67MTLK6A73KEJ35WEQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.356675316 (606h8m3.227758361s) [noise=true , address=77.90.31.12:4612 , piece_id=63TS3SUEFUE6PORZ4W76OXRWFL7SBN773UGGY3DU7JPBAW66N7UA, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.356992732 (606h8m3.228075767s) [noise=true , address=64.49.13.140:4507 , piece_id=E7T6WL45F6GHECCRANX6AIKWFRZ77TURRWORYEYI43MET2WG7SRQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.357330644 (606h8m3.228413679s) [noise=true , address=77.90.31.21:4886 , piece_id=ZQRCPTVC3WFVQPZ36AU6KJW7PD3T557CXYQIICLDGOKK6JCPVZ5Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.357663649 (606h8m3.228746684s) [noise=true , address=145.79.227.49:4367 , piece_id=R2T2ABLXBPDVP3ZVA2TAYMCLQQ7JQUVQYCTHCKGMFBRVWCDAO4PQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.359827449 (606h8m3.230910484s) [noise=true , address=145.79.227.34:4918 , piece_id=HUMOHOT3HOYCBJ23WXQXWU3X6C6JQTRLLHTD4RQLCQTCB3HXI7EQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.360086692 (606h8m3.231169727s) [noise=true , address=145.79.227.7:4320 , piece_id=MAICNTHGLZ6W27HXDOOZXOBRGQGEXNSA4BJT2T5D5PHRVV6ZMX2Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.361622167 (606h8m3.232705212s) [noise=true , address=145.79.227.33:4888 , piece_id=I4426P3VETHPOL5WHWJEGVYRXJULTCNDX3WSUTS3XZVY36Q2N44Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.362965514 (606h8m3.234048559s) [noise=true , address=45.91.116.11:4133 , piece_id=NEW7ZOVP5E7HYGYD4LG5UG2MDLAQW2OBRTNZDFIYO4CQYMVAWRPQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.364308883 (606h8m3.235391928s) [noise=true , address=145.79.227.26:4663 , piece_id=DYEFQZL4X74QR2LAYURT5ZTR2IWIDB55VLBQOGZM36MA54P2SWJA, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.365734507 (606h8m3.236817542s) [noise=true , address=145.79.227.40:4059 , piece_id=7WSUBR3C3PWVBCWD6KRGSD6MPX4IH2YQOAVO4J4SWZORNHGS2IPQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.366982764 (606h8m3.238065799s) [noise=true , address=145.79.227.29:4973 , piece_id=5OROXELR55A47BFPSJJEWVQU4YW6CAOK6LMEHCINM4FN3OVI2Y6Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.368390294 (606h8m3.239473339s) [noise=true , address=77.90.31.19:4852 , piece_id=2NYO63GCBGCYBFYTMUGNF2NG5FSJOD3AOUOD2FZZIWDINK4X6DJQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:04.736295138 (606h8m4.607378183s) [upload=531240]: Next returned nil segment Dec 30 21:37:04.736311850 (606h8m4.607394885s) [upload=531240]: Waiting for error group managing segments... Dec 30 21:37:05.104989591 (606h8m4.976072636s) [noise=true , address=64.49.13.140:4507 , piece_id=E7T6WL45F6GHECCRANX6AIKWFRZ77TURRWORYEYI43MET2WG7SRQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.208809748 (606h8m5.079892793s) [noise=true , address=145.79.227.22:4543 , piece_id=WWCZ6SEEXQ2KUJ32AGTIJNMSJROLYSJ4XQGBQJC4Q5DACZVA65RQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.441125389 (606h8m5.312208444s) [noise=true , address=77.90.31.19:4852 , piece_id=2NYO63GCBGCYBFYTMUGNF2NG5FSJOD3AOUOD2FZZIWDINK4X6DJQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.483630904 (606h8m5.35471394s) [noise=true , address=77.90.31.12:4612 , piece_id=63TS3SUEFUE6PORZ4W76OXRWFL7SBN773UGGY3DU7JPBAW66N7UA, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.494478811 (606h8m5.365561846s) [noise=true , address=77.90.31.11:4559 , piece_id=3DYD4CNPJQN3S4EYZJGREQTTZJDSL2QTC3OT6G2VFUGMLSVCZUFA, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.497452683 (606h8m5.368535729s) [noise=true , address=77.90.31.17:4769 , piece_id=QXDYESBKGXRY7BCFYYF2O7OXS7Y4Z4KUMJB6AXZAWX735SYU56GQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.661529377 (606h8m5.532612412s) [noise=true , address=45.91.116.13:4182 , piece_id=BYAYNP4JAAN3AW2LYSK74WJE4BRWUR5Z6K67MTLK6A73KEJ35WEQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.729773112 (606h8m5.600856157s) [noise=true , address=45.91.116.11:4133 , piece_id=NEW7ZOVP5E7HYGYD4LG5UG2MDLAQW2OBRTNZDFIYO4CQYMVAWRPQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.940743308 (606h8m5.811826353s) [noise=true , address=77.90.31.21:4886 , piece_id=ZQRCPTVC3WFVQPZ36AU6KJW7PD3T557CXYQIICLDGOKK6JCPVZ5Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.084240044 (606h8m5.95532309s) [noise=true , address=145.79.227.30:4784 , piece_id=DSXUDNYCWO5F6IYXMIVJNEASYXIZ7EO7RKAS7QRUYK5OU7GOOB5Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.121095032 (606h8m5.992178067s) [noise=true , address=145.79.227.40:4059 , piece_id=7WSUBR3C3PWVBCWD6KRGSD6MPX4IH2YQOAVO4J4SWZORNHGS2IPQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.169195983 (606h8m6.040279028s) [noise=true , address=145.79.227.49:4367 , piece_id=R2T2ABLXBPDVP3ZVA2TAYMCLQQ7JQUVQYCTHCKGMFBRVWCDAO4PQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.169380634 (606h8m6.040463679s) [noise=true , address=145.79.227.33:4888 , piece_id=I4426P3VETHPOL5WHWJEGVYRXJULTCNDX3WSUTS3XZVY36Q2N44Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.179060175 (606h8m6.050143211s) [noise=true , address=145.79.227.29:4973 , piece_id=5OROXELR55A47BFPSJJEWVQU4YW6CAOK6LMEHCINM4FN3OVI2Y6Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.179080424 (606h8m6.050163459s) [seg_pos={0 0}, upload=531240]: Segment reached optimal threshold of 14 pieces. Dec 30 21:37:06.179262801 (606h8m6.050345837s) [noise=true , address=145.79.227.26:4663 , piece_id=DYEFQZL4X74QR2LAYURT5ZTR2IWIDB55VLBQOGZM36MA54P2SWJA, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SwpAaXTA3BqUvwubzmJkvwokdz6SSpT9t8HPtQcXXJxmcgfm2): protocol: write tcp 188.166.225.170:43184->145.79.227.26:4663: use of closed network connection; write tcp 188.166.225.170:43184->145.79.227.26:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43184->145.79.227.26:4663: use of closed network connection Dec 30 21:37:06.179268703 (606h8m6.050351748s) [noise=true , address=145.79.227.26:4663 , piece_id=DYEFQZL4X74QR2LAYURT5ZTR2IWIDB55VLBQOGZM36MA54P2SWJA, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179363724 (606h8m6.050446769s) [noise=true , address=145.79.227.7:4320 , piece_id=MAICNTHGLZ6W27HXDOOZXOBRGQGEXNSA4BJT2T5D5PHRVV6ZMX2Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126rEgnQw1w7hsFpRqQP9nLaumdBfdtkgYjdw6WE6nXrxk5kj4B): protocol: write tcp 188.166.225.170:45228->145.79.227.7:4320: use of closed network connection; write tcp 188.166.225.170:45228->145.79.227.7:4320: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45228->145.79.227.7:4320: use of closed network connection Dec 30 21:37:06.179369525 (606h8m6.05045257s) [noise=true , address=145.79.227.7:4320 , piece_id=MAICNTHGLZ6W27HXDOOZXOBRGQGEXNSA4BJT2T5D5PHRVV6ZMX2Q, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179532996 (606h8m6.050616061s) [noise=true , address=145.79.227.34:4918 , piece_id=HUMOHOT3HOYCBJ23WXQXWU3X6C6JQTRLLHTD4RQLCQTCB3HXI7EQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF Dec 30 21:37:06.179572371 (606h8m6.050655406s) [noise=true , address=145.79.227.34:4918 , piece_id=HUMOHOT3HOYCBJ23WXQXWU3X6C6JQTRLLHTD4RQLCQTCB3HXI7EQ, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179638587 (606h8m6.050721622s) [noise=true , address=145.79.227.35:4492 , piece_id=HCKFSQMJYEP6B5H4OX5AZ5TWP6Z4H4Z6WA2T7Z2H5KNZ4SDJRMUQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1roa5GGYmqCX2AxSrqFgqyV2CtBEYZavuGTseiwDy4pjkT6bip): protocol: expected piece hash; context canceled; EOF Dec 30 21:37:06.179647855 (606h8m6.05073089s) [noise=true , address=145.79.227.35:4492 , piece_id=HCKFSQMJYEP6B5H4OX5AZ5TWP6Z4H4Z6WA2T7Z2H5KNZ4SDJRMUQ, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179666641 (606h8m6.050749686s) [seg_pos={0 0}, upload=531240]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 21:37:06.179742766 (606h8m6.050825811s) [upload=531240]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 21:37:06.179747545 (606h8m6.05083059s) [upload=531240]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 21:37:06.490986034 (606h8m6.362069079s) [upload=531240]: Done uploading segments... Dec 30 21:37:06.491006261 (606h8m6.362089307s) [upload=531240]: Upload finished. err: <nil> Duration: 15.301834139s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [...X.X.........X..X...X.........X.........................X.] duration: 6.052608401s trace: https://jaeger.dev.storj.io/trace/57a7ab98e7eb0d3f FAIL (2025-12-30 21:59:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 21:58:42.298454527 (606h29m42.169537592s) [upload=531550]: Starting upload Dec 30 21:58:42.298523438 (606h29m42.169606473s) [upload=531550]: Done starting upload Dec 30 21:58:42.298568774 (606h29m42.169651809s) [upload=531550]: Uploading segments... Dec 30 21:58:42.298697850 (606h29m42.169780885s) [upload=531550]: Got next segment. Inline: false Dec 30 21:58:42.298728649 (606h29m42.169811694s) [upload=531550]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 21:58:42.298733227 (606h29m42.169816262s) [upload=531550]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 21:58:42.545734213 (606h29m42.416817288s) [seg_pos={0 0}, upload=531550]: Begin upload segment... Dec 30 21:58:42.545786032 (606h29m42.416869078s) [seg_pos={0 0}, upload=531550]: Done begin upload segment. Dec 30 21:58:42.545855104 (606h29m42.416938149s) [noise=true , address=145.79.227.23:4547 , piece_id=LYGUMTAP7OPJWGRVFY65GUZFE4BBTV73JTG657DR5OM26LA7UFZA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.545897384 (606h29m42.416980449s) [noise=true , address=145.79.227.49:4339 , piece_id=WBBPZK2VY2QVRTQNXEKQ7TRYPWBDYS2TGTKEDAAXQ2AT4TUQXTQQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.546757752 (606h29m42.417840817s) [noise=true , address=45.91.116.12:4225 , piece_id=LLYEKE4TJ54KECLAC7KFUV44IVHYCYLF45I43G5V3DPV5QDBUOIQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.546816995 (606h29m42.41790004s) [noise=true , address=145.79.227.40:4050 , piece_id=LA7KLHHWRNHULLPAZWQ6MLS76SLN3BONYXZJXCUHGFC5JZ4SQRIA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.547216927 (606h29m42.418299962s) [noise=true , address=145.79.227.15:4876 , piece_id=L2OIJAB6OOANT2EPFNVNLTMYCH334JGFBFASD44EHOAWTAHKUV5Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.547615996 (606h29m42.418699042s) [noise=true , address=145.79.227.17:4971 , piece_id=DYZUALBL3V75TKGK57LOJABWVCOD5UVS3WH74I55OVUDGRQUA3QQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.547841055 (606h29m42.4189241s) [noise=true , address=77.90.31.21:4877 , piece_id=BD2FEFNAGYWZYDA2IMOVHA7JE66EXSF3A3PL54Q2DUOJFG5ALG4Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548023322 (606h29m42.419106357s) [noise=true , address=45.91.116.14:4190 , piece_id=HHU5YZBTJEEWYVST6XQCKGWKRLRQIIR2UQTQ27RLK53BNAVOSHFQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548380923 (606h29m42.419463968s) [noise=true , address=45.91.116.11:4120 , piece_id=FZ5L6FM7IHX65EVFGVXXHEF43XQ4EBUH4C7VZUSTH2MVWLNR5M2Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548560475 (606h29m42.41964351s) [noise=true , address=145.79.227.39:4013 , piece_id=I4XP47SFJICIYIPEN4E7UC2HXRXH4IXVJYGHXTZBCUESAHNFQXJA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548719268 (606h29m42.419802313s) [noise=true , address=145.79.227.31:4797 , piece_id=S5YBLITONGU6RHPQENP2TCVVWOHEMXPMYXLNYWBZNXWMFI26PVUA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549024107 (606h29m42.420107152s) [noise=true , address=145.79.227.7:4294 , piece_id=C6N5OIRLV3LLSCEIOGQJKWPKENGZYNAPY6V2AYZNKUMACBP5SLWQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549197708 (606h29m42.420280753s) [noise=true , address=145.79.227.16:4910 , piece_id=O4YO367LEVZSRI5E75YEZ7W7YI2KGKKQDXCOHQ3LALE4WUAPQVHQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549506406 (606h29m42.420589441s) [noise=true , address=145.79.227.12:4765 , piece_id=4ZF5YQP4Z5HBTX5YWUDHZV55Q47WASRLMV6ZYJXJRL2QBAVQQM4Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549971591 (606h29m42.421054626s) [noise=true , address=145.79.227.32:4835 , piece_id=CN4NMCFNYKOCG2JCIEXJBL2PLEYWD5THCZB4WQDURIECFVHLLX6Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.551571588 (606h29m42.422654623s) [noise=true , address=145.79.227.24:4581 , piece_id=Y6GBEOR7DQX2H6LBHXYFOXRBX3GP7AKPYVGTQNMUEXKPXLONUUGQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:44.247717212 (606h29m44.118800248s) [upload=531550]: Next returned nil segment Dec 30 21:58:44.247730348 (606h29m44.118813383s) [upload=531550]: Waiting for error group managing segments... Dec 30 21:58:44.532148720 (606h29m44.403231755s) [noise=true , address=145.79.227.31:4797 , piece_id=S5YBLITONGU6RHPQENP2TCVVWOHEMXPMYXLNYWBZNXWMFI26PVUA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.534655293 (606h29m44.405738328s) [noise=true , address=145.79.227.39:4013 , piece_id=I4XP47SFJICIYIPEN4E7UC2HXRXH4IXVJYGHXTZBCUESAHNFQXJA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.544709427 (606h29m44.415792462s) [noise=true , address=145.79.227.12:4765 , piece_id=4ZF5YQP4Z5HBTX5YWUDHZV55Q47WASRLMV6ZYJXJRL2QBAVQQM4Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.545822306 (606h29m44.416905342s) [noise=true , address=145.79.227.7:4294 , piece_id=C6N5OIRLV3LLSCEIOGQJKWPKENGZYNAPY6V2AYZNKUMACBP5SLWQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.549465162 (606h29m44.420548198s) [noise=true , address=145.79.227.24:4581 , piece_id=Y6GBEOR7DQX2H6LBHXYFOXRBX3GP7AKPYVGTQNMUEXKPXLONUUGQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.556935928 (606h29m44.428018964s) [noise=true , address=145.79.227.15:4876 , piece_id=L2OIJAB6OOANT2EPFNVNLTMYCH334JGFBFASD44EHOAWTAHKUV5Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.561283296 (606h29m44.432366331s) [noise=true , address=145.79.227.16:4910 , piece_id=O4YO367LEVZSRI5E75YEZ7W7YI2KGKKQDXCOHQ3LALE4WUAPQVHQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.614194264 (606h29m44.485277309s) [noise=true , address=145.79.227.17:4971 , piece_id=DYZUALBL3V75TKGK57LOJABWVCOD5UVS3WH74I55OVUDGRQUA3QQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.615630689 (606h29m44.486713724s) [noise=true , address=145.79.227.32:4835 , piece_id=CN4NMCFNYKOCG2JCIEXJBL2PLEYWD5THCZB4WQDURIECFVHLLX6Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.639672077 (606h29m44.510755122s) [noise=true , address=145.79.227.23:4547 , piece_id=LYGUMTAP7OPJWGRVFY65GUZFE4BBTV73JTG657DR5OM26LA7UFZA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.699618771 (606h29m44.570701817s) [noise=true , address=77.90.31.21:4877 , piece_id=BD2FEFNAGYWZYDA2IMOVHA7JE66EXSF3A3PL54Q2DUOJFG5ALG4Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.841535416 (606h29m44.712618462s) [noise=true , address=45.91.116.14:4190 , piece_id=HHU5YZBTJEEWYVST6XQCKGWKRLRQIIR2UQTQ27RLK53BNAVOSHFQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.974660841 (606h29m44.845743886s) [noise=true , address=45.91.116.12:4225 , piece_id=LLYEKE4TJ54KECLAC7KFUV44IVHYCYLF45I43G5V3DPV5QDBUOIQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.974681401 (606h29m44.845764436s) [seg_pos={0 0}, upload=531550]: Segment reached optimal threshold of 13 pieces. Dec 30 21:58:44.974855342 (606h29m44.845938377s) [noise=true , address=145.79.227.49:4339 , piece_id=WBBPZK2VY2QVRTQNXEKQ7TRYPWBDYS2TGTKEDAAXQ2AT4TUQXTQQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 30 21:58:44.974861423 (606h29m44.845944458s) [noise=true , address=145.79.227.49:4339 , piece_id=WBBPZK2VY2QVRTQNXEKQ7TRYPWBDYS2TGTKEDAAXQ2AT4TUQXTQQ, seg_pos={0 0}, upload=531550]: Long tail cancellation due to optimal threshold Dec 30 21:58:44.974968166 (606h29m44.846051201s) [noise=true , address=145.79.227.40:4050 , piece_id=LA7KLHHWRNHULLPAZWQ6MLS76SLN3BONYXZJXCUHGFC5JZ4SQRIA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 30 21:58:44.974973627 (606h29m44.846056662s) [noise=true , address=145.79.227.40:4050 , piece_id=LA7KLHHWRNHULLPAZWQ6MLS76SLN3BONYXZJXCUHGFC5JZ4SQRIA, seg_pos={0 0}, upload=531550]: Long tail cancellation due to optimal threshold Dec 30 21:58:44.975076152 (606h29m44.846159187s) [noise=true , address=45.91.116.11:4120 , piece_id=FZ5L6FM7IHX65EVFGVXXHEF43XQ4EBUH4C7VZUSTH2MVWLNR5M2Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 30 21:58:44.975083497 (606h29m44.846166532s) [noise=true , address=45.91.116.11:4120 , piece_id=FZ5L6FM7IHX65EVFGVXXHEF43XQ4EBUH4C7VZUSTH2MVWLNR5M2Q, seg_pos={0 0}, upload=531550]: Long tail cancellation due to optimal threshold Dec 30 21:58:44.975096711 (606h29m44.846179747s) [seg_pos={0 0}, upload=531550]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 21:58:44.975160703 (606h29m44.846243739s) [upload=531550]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 21:58:44.975164691 (606h29m44.846247726s) [upload=531550]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 21:58:45.272782726 (606h29m45.143865772s) [upload=531550]: Done uploading segments... Dec 30 21:58:45.272806352 (606h29m45.143889387s) [upload=531550]: Upload finished. err: <nil> Duration: 6.052637386s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................................................X........] duration: 3.575681027s trace: https://jaeger.dev.storj.io/trace/22a47ec27f9a101c FAIL (2025-12-30 21:59:18 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.575705884s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [....X.X.........X..X...X.........X.........................X] duration: 30.002141368s trace: https://jaeger.dev.storj.io/trace/57a7ab98e7eb0d3f FAIL (2025-12-30 21:59:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 21:58:42.298454527 (606h29m42.169537592s) [upload=531550]: Starting upload Dec 30 21:58:42.298523438 (606h29m42.169606473s) [upload=531550]: Done starting upload Dec 30 21:58:42.298568774 (606h29m42.169651809s) [upload=531550]: Uploading segments... Dec 30 21:58:42.298697850 (606h29m42.169780885s) [upload=531550]: Got next segment. Inline: false Dec 30 21:58:42.298728649 (606h29m42.169811694s) [upload=531550]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 21:58:42.298733227 (606h29m42.169816262s) [upload=531550]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 21:58:42.545734213 (606h29m42.416817288s) [seg_pos={0 0}, upload=531550]: Begin upload segment... Dec 30 21:58:42.545786032 (606h29m42.416869078s) [seg_pos={0 0}, upload=531550]: Done begin upload segment. Dec 30 21:58:42.545855104 (606h29m42.416938149s) [noise=true , address=145.79.227.23:4547 , piece_id=LYGUMTAP7OPJWGRVFY65GUZFE4BBTV73JTG657DR5OM26LA7UFZA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.545897384 (606h29m42.416980449s) [noise=true , address=145.79.227.49:4339 , piece_id=WBBPZK2VY2QVRTQNXEKQ7TRYPWBDYS2TGTKEDAAXQ2AT4TUQXTQQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.546757752 (606h29m42.417840817s) [noise=true , address=45.91.116.12:4225 , piece_id=LLYEKE4TJ54KECLAC7KFUV44IVHYCYLF45I43G5V3DPV5QDBUOIQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.546816995 (606h29m42.41790004s) [noise=true , address=145.79.227.40:4050 , piece_id=LA7KLHHWRNHULLPAZWQ6MLS76SLN3BONYXZJXCUHGFC5JZ4SQRIA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.547216927 (606h29m42.418299962s) [noise=true , address=145.79.227.15:4876 , piece_id=L2OIJAB6OOANT2EPFNVNLTMYCH334JGFBFASD44EHOAWTAHKUV5Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.547615996 (606h29m42.418699042s) [noise=true , address=145.79.227.17:4971 , piece_id=DYZUALBL3V75TKGK57LOJABWVCOD5UVS3WH74I55OVUDGRQUA3QQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.547841055 (606h29m42.4189241s) [noise=true , address=77.90.31.21:4877 , piece_id=BD2FEFNAGYWZYDA2IMOVHA7JE66EXSF3A3PL54Q2DUOJFG5ALG4Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548023322 (606h29m42.419106357s) [noise=true , address=45.91.116.14:4190 , piece_id=HHU5YZBTJEEWYVST6XQCKGWKRLRQIIR2UQTQ27RLK53BNAVOSHFQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548380923 (606h29m42.419463968s) [noise=true , address=45.91.116.11:4120 , piece_id=FZ5L6FM7IHX65EVFGVXXHEF43XQ4EBUH4C7VZUSTH2MVWLNR5M2Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548560475 (606h29m42.41964351s) [noise=true , address=145.79.227.39:4013 , piece_id=I4XP47SFJICIYIPEN4E7UC2HXRXH4IXVJYGHXTZBCUESAHNFQXJA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548719268 (606h29m42.419802313s) [noise=true , address=145.79.227.31:4797 , piece_id=S5YBLITONGU6RHPQENP2TCVVWOHEMXPMYXLNYWBZNXWMFI26PVUA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549024107 (606h29m42.420107152s) [noise=true , address=145.79.227.7:4294 , piece_id=C6N5OIRLV3LLSCEIOGQJKWPKENGZYNAPY6V2AYZNKUMACBP5SLWQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549197708 (606h29m42.420280753s) [noise=true , address=145.79.227.16:4910 , piece_id=O4YO367LEVZSRI5E75YEZ7W7YI2KGKKQDXCOHQ3LALE4WUAPQVHQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549506406 (606h29m42.420589441s) [noise=true , address=145.79.227.12:4765 , piece_id=4ZF5YQP4Z5HBTX5YWUDHZV55Q47WASRLMV6ZYJXJRL2QBAVQQM4Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549971591 (606h29m42.421054626s) [noise=true , address=145.79.227.32:4835 , piece_id=CN4NMCFNYKOCG2JCIEXJBL2PLEYWD5THCZB4WQDURIECFVHLLX6Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.551571588 (606h29m42.422654623s) [noise=true , address=145.79.227.24:4581 , piece_id=Y6GBEOR7DQX2H6LBHXYFOXRBX3GP7AKPYVGTQNMUEXKPXLONUUGQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:44.247717212 (606h29m44.118800248s) [upload=531550]: Next returned nil segment Dec 30 21:58:44.247730348 (606h29m44.118813383s) [upload=531550]: Waiting for error group managing segments... Dec 30 21:58:44.532148720 (606h29m44.403231755s) [noise=true , address=145.79.227.31:4797 , piece_id=S5YBLITONGU6RHPQENP2TCVVWOHEMXPMYXLNYWBZNXWMFI26PVUA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.534655293 (606h29m44.405738328s) [noise=true , address=145.79.227.39:4013 , piece_id=I4XP47SFJICIYIPEN4E7UC2HXRXH4IXVJYGHXTZBCUESAHNFQXJA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.544709427 (606h29m44.415792462s) [noise=true , address=145.79.227.12:4765 , piece_id=4ZF5YQP4Z5HBTX5YWUDHZV55Q47WASRLMV6ZYJXJRL2QBAVQQM4Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.545822306 (606h29m44.416905342s) [noise=true , address=145.79.227.7:4294 , piece_id=C6N5OIRLV3LLSCEIOGQJKWPKENGZYNAPY6V2AYZNKUMACBP5SLWQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.549465162 (606h29m44.420548198s) [noise=true , address=145.79.227.24:4581 , piece_id=Y6GBEOR7DQX2H6LBHXYFOXRBX3GP7AKPYVGTQNMUEXKPXLONUUGQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.556935928 (606h29m44.428018964s) [noise=true , address=145.79.227.15:4876 , piece_id=L2OIJAB6OOANT2EPFNVNLTMYCH334JGFBFASD44EHOAWTAHKUV5Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.561283296 (606h29m44.432366331s) [noise=true , address=145.79.227.16:4910 , piece_id=O4YO367LEVZSRI5E75YEZ7W7YI2KGKKQDXCOHQ3LALE4WUAPQVHQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.614194264 (606h29m44.485277309s) [noise=true , address=145.79.227.17:4971 , piece_id=DYZUALBL3V75TKGK57LOJABWVCOD5UVS3WH74I55OVUDGRQUA3QQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.615630689 (606h29m44.486713724s) [noise=true , address=145.79.227.32:4835 , piece_id=CN4NMCFNYKOCG2JCIEXJBL2PLEYWD5THCZB4WQDURIECFVHLLX6Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.639672077 (606h29m44.510755122s) [noise=true , address=145.79.227.23:4547 , piece_id=LYGUMTAP7OPJWGRVFY65GUZFE4BBTV73JTG657DR5OM26LA7UFZA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.699618771 (606h29m44.570701817s) [noise=true , address=77.90.31.21:4877 , piece_id=BD2FEFNAGYWZYDA2IMOVHA7JE66EXSF3A3PL54Q2DUOJFG5ALG4Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.841535416 (606h29m44.712618462s) [noise=true , address=45.91.116.14:4190 , piece_id=HHU5YZBTJEEWYVST6XQCKGWKRLRQIIR2UQTQ27RLK53BNAVOSHFQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.974660841 (606h29m44.845743886s) [noise=true , address=45.91.116.12:4225 , piece_id=LLYEKE4TJ54KECLAC7KFUV44IVHYCYLF45I43G5V3DPV5QDBUOIQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.974681401 (606h29m44.845764436s) [seg_pos={0 0}, upload=531550]: Segment reached optimal threshold of 13 pieces. Dec 30 21:58:44.974855342 (606h29m44.845938377s) [noise=true , address=145.79.227.49:4339 , piece_id=WBBPZK2VY2QVRTQNXEKQ7TRYPWBDYS2TGTKEDAAXQ2AT4TUQXTQQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 30 21:58:44.974861423 (606h29m44.845944458s) [noise=true , address=145.79.227.49:4339 , piece_id=WBBPZK2VY2QVRTQNXEKQ7TRYPWBDYS2TGTKEDAAXQ2AT4TUQXTQQ, seg_pos={0 0}, upload=531550]: Long tail cancellation due to optimal threshold Dec 30 21:58:44.974968166 (606h29m44.846051201s) [noise=true , address=145.79.227.40:4050 , piece_id=LA7KLHHWRNHULLPAZWQ6MLS76SLN3BONYXZJXCUHGFC5JZ4SQRIA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 30 21:58:44.974973627 (606h29m44.846056662s) [noise=true , address=145.79.227.40:4050 , piece_id=LA7KLHHWRNHULLPAZWQ6MLS76SLN3BONYXZJXCUHGFC5JZ4SQRIA, seg_pos={0 0}, upload=531550]: Long tail cancellation due to optimal threshold Dec 30 21:58:44.975076152 (606h29m44.846159187s) [noise=true , address=45.91.116.11:4120 , piece_id=FZ5L6FM7IHX65EVFGVXXHEF43XQ4EBUH4C7VZUSTH2MVWLNR5M2Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 30 21:58:44.975083497 (606h29m44.846166532s) [noise=true , address=45.91.116.11:4120 , piece_id=FZ5L6FM7IHX65EVFGVXXHEF43XQ4EBUH4C7VZUSTH2MVWLNR5M2Q, seg_pos={0 0}, upload=531550]: Long tail cancellation due to optimal threshold Dec 30 21:58:44.975096711 (606h29m44.846179747s) [seg_pos={0 0}, upload=531550]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 21:58:44.975160703 (606h29m44.846243739s) [upload=531550]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 21:58:44.975164691 (606h29m44.846247726s) [upload=531550]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 21:58:45.272782726 (606h29m45.143865772s) [upload=531550]: Done uploading segments... Dec 30 21:58:45.272806352 (606h29m45.143889387s) [upload=531550]: Upload finished. err: <nil> Duration: 30.002162007s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [...............XX......X.........X..........................] duration: 18.35331722s trace: https://jaeger.dev.storj.io/trace/5cdd5ba7b1cb22c7 FAIL (2025-12-30 21:58:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 21:37:03.355296401 (606h8m3.226379446s) [noise=true , address=145.79.227.22:4543 , piece_id=WWCZ6SEEXQ2KUJ32AGTIJNMSJROLYSJ4XQGBQJC4Q5DACZVA65RQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.355839374 (606h8m3.22692241s) [noise=true , address=145.79.227.35:4492 , piece_id=HCKFSQMJYEP6B5H4OX5AZ5TWP6Z4H4Z6WA2T7Z2H5KNZ4SDJRMUQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.356356930 (606h8m3.227439965s) [noise=true , address=45.91.116.13:4182 , piece_id=BYAYNP4JAAN3AW2LYSK74WJE4BRWUR5Z6K67MTLK6A73KEJ35WEQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.356675316 (606h8m3.227758361s) [noise=true , address=77.90.31.12:4612 , piece_id=63TS3SUEFUE6PORZ4W76OXRWFL7SBN773UGGY3DU7JPBAW66N7UA, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.356992732 (606h8m3.228075767s) [noise=true , address=64.49.13.140:4507 , piece_id=E7T6WL45F6GHECCRANX6AIKWFRZ77TURRWORYEYI43MET2WG7SRQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.357330644 (606h8m3.228413679s) [noise=true , address=77.90.31.21:4886 , piece_id=ZQRCPTVC3WFVQPZ36AU6KJW7PD3T557CXYQIICLDGOKK6JCPVZ5Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.357663649 (606h8m3.228746684s) [noise=true , address=145.79.227.49:4367 , piece_id=R2T2ABLXBPDVP3ZVA2TAYMCLQQ7JQUVQYCTHCKGMFBRVWCDAO4PQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.359827449 (606h8m3.230910484s) [noise=true , address=145.79.227.34:4918 , piece_id=HUMOHOT3HOYCBJ23WXQXWU3X6C6JQTRLLHTD4RQLCQTCB3HXI7EQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.360086692 (606h8m3.231169727s) [noise=true , address=145.79.227.7:4320 , piece_id=MAICNTHGLZ6W27HXDOOZXOBRGQGEXNSA4BJT2T5D5PHRVV6ZMX2Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.361622167 (606h8m3.232705212s) [noise=true , address=145.79.227.33:4888 , piece_id=I4426P3VETHPOL5WHWJEGVYRXJULTCNDX3WSUTS3XZVY36Q2N44Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.362965514 (606h8m3.234048559s) [noise=true , address=45.91.116.11:4133 , piece_id=NEW7ZOVP5E7HYGYD4LG5UG2MDLAQW2OBRTNZDFIYO4CQYMVAWRPQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.364308883 (606h8m3.235391928s) [noise=true , address=145.79.227.26:4663 , piece_id=DYEFQZL4X74QR2LAYURT5ZTR2IWIDB55VLBQOGZM36MA54P2SWJA, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.365734507 (606h8m3.236817542s) [noise=true , address=145.79.227.40:4059 , piece_id=7WSUBR3C3PWVBCWD6KRGSD6MPX4IH2YQOAVO4J4SWZORNHGS2IPQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.366982764 (606h8m3.238065799s) [noise=true , address=145.79.227.29:4973 , piece_id=5OROXELR55A47BFPSJJEWVQU4YW6CAOK6LMEHCINM4FN3OVI2Y6Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.368390294 (606h8m3.239473339s) [noise=true , address=77.90.31.19:4852 , piece_id=2NYO63GCBGCYBFYTMUGNF2NG5FSJOD3AOUOD2FZZIWDINK4X6DJQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:04.736295138 (606h8m4.607378183s) [upload=531240]: Next returned nil segment Dec 30 21:37:04.736311850 (606h8m4.607394885s) [upload=531240]: Waiting for error group managing segments... Dec 30 21:37:05.104989591 (606h8m4.976072636s) [noise=true , address=64.49.13.140:4507 , piece_id=E7T6WL45F6GHECCRANX6AIKWFRZ77TURRWORYEYI43MET2WG7SRQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.208809748 (606h8m5.079892793s) [noise=true , address=145.79.227.22:4543 , piece_id=WWCZ6SEEXQ2KUJ32AGTIJNMSJROLYSJ4XQGBQJC4Q5DACZVA65RQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.441125389 (606h8m5.312208444s) [noise=true , address=77.90.31.19:4852 , piece_id=2NYO63GCBGCYBFYTMUGNF2NG5FSJOD3AOUOD2FZZIWDINK4X6DJQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.483630904 (606h8m5.35471394s) [noise=true , address=77.90.31.12:4612 , piece_id=63TS3SUEFUE6PORZ4W76OXRWFL7SBN773UGGY3DU7JPBAW66N7UA, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.494478811 (606h8m5.365561846s) [noise=true , address=77.90.31.11:4559 , piece_id=3DYD4CNPJQN3S4EYZJGREQTTZJDSL2QTC3OT6G2VFUGMLSVCZUFA, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.497452683 (606h8m5.368535729s) [noise=true , address=77.90.31.17:4769 , piece_id=QXDYESBKGXRY7BCFYYF2O7OXS7Y4Z4KUMJB6AXZAWX735SYU56GQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.661529377 (606h8m5.532612412s) [noise=true , address=45.91.116.13:4182 , piece_id=BYAYNP4JAAN3AW2LYSK74WJE4BRWUR5Z6K67MTLK6A73KEJ35WEQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.729773112 (606h8m5.600856157s) [noise=true , address=45.91.116.11:4133 , piece_id=NEW7ZOVP5E7HYGYD4LG5UG2MDLAQW2OBRTNZDFIYO4CQYMVAWRPQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.940743308 (606h8m5.811826353s) [noise=true , address=77.90.31.21:4886 , piece_id=ZQRCPTVC3WFVQPZ36AU6KJW7PD3T557CXYQIICLDGOKK6JCPVZ5Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.084240044 (606h8m5.95532309s) [noise=true , address=145.79.227.30:4784 , piece_id=DSXUDNYCWO5F6IYXMIVJNEASYXIZ7EO7RKAS7QRUYK5OU7GOOB5Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.121095032 (606h8m5.992178067s) [noise=true , address=145.79.227.40:4059 , piece_id=7WSUBR3C3PWVBCWD6KRGSD6MPX4IH2YQOAVO4J4SWZORNHGS2IPQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.169195983 (606h8m6.040279028s) [noise=true , address=145.79.227.49:4367 , piece_id=R2T2ABLXBPDVP3ZVA2TAYMCLQQ7JQUVQYCTHCKGMFBRVWCDAO4PQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.169380634 (606h8m6.040463679s) [noise=true , address=145.79.227.33:4888 , piece_id=I4426P3VETHPOL5WHWJEGVYRXJULTCNDX3WSUTS3XZVY36Q2N44Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.179060175 (606h8m6.050143211s) [noise=true , address=145.79.227.29:4973 , piece_id=5OROXELR55A47BFPSJJEWVQU4YW6CAOK6LMEHCINM4FN3OVI2Y6Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.179080424 (606h8m6.050163459s) [seg_pos={0 0}, upload=531240]: Segment reached optimal threshold of 14 pieces. Dec 30 21:37:06.179262801 (606h8m6.050345837s) [noise=true , address=145.79.227.26:4663 , piece_id=DYEFQZL4X74QR2LAYURT5ZTR2IWIDB55VLBQOGZM36MA54P2SWJA, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SwpAaXTA3BqUvwubzmJkvwokdz6SSpT9t8HPtQcXXJxmcgfm2): protocol: write tcp 188.166.225.170:43184->145.79.227.26:4663: use of closed network connection; write tcp 188.166.225.170:43184->145.79.227.26:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43184->145.79.227.26:4663: use of closed network connection Dec 30 21:37:06.179268703 (606h8m6.050351748s) [noise=true , address=145.79.227.26:4663 , piece_id=DYEFQZL4X74QR2LAYURT5ZTR2IWIDB55VLBQOGZM36MA54P2SWJA, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179363724 (606h8m6.050446769s) [noise=true , address=145.79.227.7:4320 , piece_id=MAICNTHGLZ6W27HXDOOZXOBRGQGEXNSA4BJT2T5D5PHRVV6ZMX2Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126rEgnQw1w7hsFpRqQP9nLaumdBfdtkgYjdw6WE6nXrxk5kj4B): protocol: write tcp 188.166.225.170:45228->145.79.227.7:4320: use of closed network connection; write tcp 188.166.225.170:45228->145.79.227.7:4320: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45228->145.79.227.7:4320: use of closed network connection Dec 30 21:37:06.179369525 (606h8m6.05045257s) [noise=true , address=145.79.227.7:4320 , piece_id=MAICNTHGLZ6W27HXDOOZXOBRGQGEXNSA4BJT2T5D5PHRVV6ZMX2Q, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179532996 (606h8m6.050616061s) [noise=true , address=145.79.227.34:4918 , piece_id=HUMOHOT3HOYCBJ23WXQXWU3X6C6JQTRLLHTD4RQLCQTCB3HXI7EQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF Dec 30 21:37:06.179572371 (606h8m6.050655406s) [noise=true , address=145.79.227.34:4918 , piece_id=HUMOHOT3HOYCBJ23WXQXWU3X6C6JQTRLLHTD4RQLCQTCB3HXI7EQ, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179638587 (606h8m6.050721622s) [noise=true , address=145.79.227.35:4492 , piece_id=HCKFSQMJYEP6B5H4OX5AZ5TWP6Z4H4Z6WA2T7Z2H5KNZ4SDJRMUQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1roa5GGYmqCX2AxSrqFgqyV2CtBEYZavuGTseiwDy4pjkT6bip): protocol: expected piece hash; context canceled; EOF Dec 30 21:37:06.179647855 (606h8m6.05073089s) [noise=true , address=145.79.227.35:4492 , piece_id=HCKFSQMJYEP6B5H4OX5AZ5TWP6Z4H4Z6WA2T7Z2H5KNZ4SDJRMUQ, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179666641 (606h8m6.050749686s) [seg_pos={0 0}, upload=531240]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 21:37:06.179742766 (606h8m6.050825811s) [upload=531240]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 21:37:06.179747545 (606h8m6.05083059s) [upload=531240]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 21:37:06.490986034 (606h8m6.362069079s) [upload=531240]: Done uploading segments... Dec 30 21:37:06.491006261 (606h8m6.362089307s) [upload=531240]: Upload finished. err: <nil> Duration: 18.35334269s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................................................X.......] duration: 3.559298794s trace: https://jaeger.dev.storj.io/trace/22a47ec27f9a101c FAIL (2025-12-30 21:58:33 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.559323521s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [................XX......X.........X.........................] duration: 17.437928194s trace: https://jaeger.dev.storj.io/trace/5cdd5ba7b1cb22c7 FAIL (2025-12-30 21:58:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 21:37:03.355296401 (606h8m3.226379446s) [noise=true , address=145.79.227.22:4543 , piece_id=WWCZ6SEEXQ2KUJ32AGTIJNMSJROLYSJ4XQGBQJC4Q5DACZVA65RQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.355839374 (606h8m3.22692241s) [noise=true , address=145.79.227.35:4492 , piece_id=HCKFSQMJYEP6B5H4OX5AZ5TWP6Z4H4Z6WA2T7Z2H5KNZ4SDJRMUQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.356356930 (606h8m3.227439965s) [noise=true , address=45.91.116.13:4182 , piece_id=BYAYNP4JAAN3AW2LYSK74WJE4BRWUR5Z6K67MTLK6A73KEJ35WEQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.356675316 (606h8m3.227758361s) [noise=true , address=77.90.31.12:4612 , piece_id=63TS3SUEFUE6PORZ4W76OXRWFL7SBN773UGGY3DU7JPBAW66N7UA, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.356992732 (606h8m3.228075767s) [noise=true , address=64.49.13.140:4507 , piece_id=E7T6WL45F6GHECCRANX6AIKWFRZ77TURRWORYEYI43MET2WG7SRQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.357330644 (606h8m3.228413679s) [noise=true , address=77.90.31.21:4886 , piece_id=ZQRCPTVC3WFVQPZ36AU6KJW7PD3T557CXYQIICLDGOKK6JCPVZ5Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.357663649 (606h8m3.228746684s) [noise=true , address=145.79.227.49:4367 , piece_id=R2T2ABLXBPDVP3ZVA2TAYMCLQQ7JQUVQYCTHCKGMFBRVWCDAO4PQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.359827449 (606h8m3.230910484s) [noise=true , address=145.79.227.34:4918 , piece_id=HUMOHOT3HOYCBJ23WXQXWU3X6C6JQTRLLHTD4RQLCQTCB3HXI7EQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.360086692 (606h8m3.231169727s) [noise=true , address=145.79.227.7:4320 , piece_id=MAICNTHGLZ6W27HXDOOZXOBRGQGEXNSA4BJT2T5D5PHRVV6ZMX2Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.361622167 (606h8m3.232705212s) [noise=true , address=145.79.227.33:4888 , piece_id=I4426P3VETHPOL5WHWJEGVYRXJULTCNDX3WSUTS3XZVY36Q2N44Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.362965514 (606h8m3.234048559s) [noise=true , address=45.91.116.11:4133 , piece_id=NEW7ZOVP5E7HYGYD4LG5UG2MDLAQW2OBRTNZDFIYO4CQYMVAWRPQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.364308883 (606h8m3.235391928s) [noise=true , address=145.79.227.26:4663 , piece_id=DYEFQZL4X74QR2LAYURT5ZTR2IWIDB55VLBQOGZM36MA54P2SWJA, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.365734507 (606h8m3.236817542s) [noise=true , address=145.79.227.40:4059 , piece_id=7WSUBR3C3PWVBCWD6KRGSD6MPX4IH2YQOAVO4J4SWZORNHGS2IPQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.366982764 (606h8m3.238065799s) [noise=true , address=145.79.227.29:4973 , piece_id=5OROXELR55A47BFPSJJEWVQU4YW6CAOK6LMEHCINM4FN3OVI2Y6Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.368390294 (606h8m3.239473339s) [noise=true , address=77.90.31.19:4852 , piece_id=2NYO63GCBGCYBFYTMUGNF2NG5FSJOD3AOUOD2FZZIWDINK4X6DJQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:04.736295138 (606h8m4.607378183s) [upload=531240]: Next returned nil segment Dec 30 21:37:04.736311850 (606h8m4.607394885s) [upload=531240]: Waiting for error group managing segments... Dec 30 21:37:05.104989591 (606h8m4.976072636s) [noise=true , address=64.49.13.140:4507 , piece_id=E7T6WL45F6GHECCRANX6AIKWFRZ77TURRWORYEYI43MET2WG7SRQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.208809748 (606h8m5.079892793s) [noise=true , address=145.79.227.22:4543 , piece_id=WWCZ6SEEXQ2KUJ32AGTIJNMSJROLYSJ4XQGBQJC4Q5DACZVA65RQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.441125389 (606h8m5.312208444s) [noise=true , address=77.90.31.19:4852 , piece_id=2NYO63GCBGCYBFYTMUGNF2NG5FSJOD3AOUOD2FZZIWDINK4X6DJQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.483630904 (606h8m5.35471394s) [noise=true , address=77.90.31.12:4612 , piece_id=63TS3SUEFUE6PORZ4W76OXRWFL7SBN773UGGY3DU7JPBAW66N7UA, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.494478811 (606h8m5.365561846s) [noise=true , address=77.90.31.11:4559 , piece_id=3DYD4CNPJQN3S4EYZJGREQTTZJDSL2QTC3OT6G2VFUGMLSVCZUFA, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.497452683 (606h8m5.368535729s) [noise=true , address=77.90.31.17:4769 , piece_id=QXDYESBKGXRY7BCFYYF2O7OXS7Y4Z4KUMJB6AXZAWX735SYU56GQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.661529377 (606h8m5.532612412s) [noise=true , address=45.91.116.13:4182 , piece_id=BYAYNP4JAAN3AW2LYSK74WJE4BRWUR5Z6K67MTLK6A73KEJ35WEQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.729773112 (606h8m5.600856157s) [noise=true , address=45.91.116.11:4133 , piece_id=NEW7ZOVP5E7HYGYD4LG5UG2MDLAQW2OBRTNZDFIYO4CQYMVAWRPQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.940743308 (606h8m5.811826353s) [noise=true , address=77.90.31.21:4886 , piece_id=ZQRCPTVC3WFVQPZ36AU6KJW7PD3T557CXYQIICLDGOKK6JCPVZ5Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.084240044 (606h8m5.95532309s) [noise=true , address=145.79.227.30:4784 , piece_id=DSXUDNYCWO5F6IYXMIVJNEASYXIZ7EO7RKAS7QRUYK5OU7GOOB5Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.121095032 (606h8m5.992178067s) [noise=true , address=145.79.227.40:4059 , piece_id=7WSUBR3C3PWVBCWD6KRGSD6MPX4IH2YQOAVO4J4SWZORNHGS2IPQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.169195983 (606h8m6.040279028s) [noise=true , address=145.79.227.49:4367 , piece_id=R2T2ABLXBPDVP3ZVA2TAYMCLQQ7JQUVQYCTHCKGMFBRVWCDAO4PQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.169380634 (606h8m6.040463679s) [noise=true , address=145.79.227.33:4888 , piece_id=I4426P3VETHPOL5WHWJEGVYRXJULTCNDX3WSUTS3XZVY36Q2N44Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.179060175 (606h8m6.050143211s) [noise=true , address=145.79.227.29:4973 , piece_id=5OROXELR55A47BFPSJJEWVQU4YW6CAOK6LMEHCINM4FN3OVI2Y6Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.179080424 (606h8m6.050163459s) [seg_pos={0 0}, upload=531240]: Segment reached optimal threshold of 14 pieces. Dec 30 21:37:06.179262801 (606h8m6.050345837s) [noise=true , address=145.79.227.26:4663 , piece_id=DYEFQZL4X74QR2LAYURT5ZTR2IWIDB55VLBQOGZM36MA54P2SWJA, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SwpAaXTA3BqUvwubzmJkvwokdz6SSpT9t8HPtQcXXJxmcgfm2): protocol: write tcp 188.166.225.170:43184->145.79.227.26:4663: use of closed network connection; write tcp 188.166.225.170:43184->145.79.227.26:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43184->145.79.227.26:4663: use of closed network connection Dec 30 21:37:06.179268703 (606h8m6.050351748s) [noise=true , address=145.79.227.26:4663 , piece_id=DYEFQZL4X74QR2LAYURT5ZTR2IWIDB55VLBQOGZM36MA54P2SWJA, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179363724 (606h8m6.050446769s) [noise=true , address=145.79.227.7:4320 , piece_id=MAICNTHGLZ6W27HXDOOZXOBRGQGEXNSA4BJT2T5D5PHRVV6ZMX2Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126rEgnQw1w7hsFpRqQP9nLaumdBfdtkgYjdw6WE6nXrxk5kj4B): protocol: write tcp 188.166.225.170:45228->145.79.227.7:4320: use of closed network connection; write tcp 188.166.225.170:45228->145.79.227.7:4320: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45228->145.79.227.7:4320: use of closed network connection Dec 30 21:37:06.179369525 (606h8m6.05045257s) [noise=true , address=145.79.227.7:4320 , piece_id=MAICNTHGLZ6W27HXDOOZXOBRGQGEXNSA4BJT2T5D5PHRVV6ZMX2Q, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179532996 (606h8m6.050616061s) [noise=true , address=145.79.227.34:4918 , piece_id=HUMOHOT3HOYCBJ23WXQXWU3X6C6JQTRLLHTD4RQLCQTCB3HXI7EQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF Dec 30 21:37:06.179572371 (606h8m6.050655406s) [noise=true , address=145.79.227.34:4918 , piece_id=HUMOHOT3HOYCBJ23WXQXWU3X6C6JQTRLLHTD4RQLCQTCB3HXI7EQ, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179638587 (606h8m6.050721622s) [noise=true , address=145.79.227.35:4492 , piece_id=HCKFSQMJYEP6B5H4OX5AZ5TWP6Z4H4Z6WA2T7Z2H5KNZ4SDJRMUQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1roa5GGYmqCX2AxSrqFgqyV2CtBEYZavuGTseiwDy4pjkT6bip): protocol: expected piece hash; context canceled; EOF Dec 30 21:37:06.179647855 (606h8m6.05073089s) [noise=true , address=145.79.227.35:4492 , piece_id=HCKFSQMJYEP6B5H4OX5AZ5TWP6Z4H4Z6WA2T7Z2H5KNZ4SDJRMUQ, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179666641 (606h8m6.050749686s) [seg_pos={0 0}, upload=531240]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 21:37:06.179742766 (606h8m6.050825811s) [upload=531240]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 21:37:06.179747545 (606h8m6.05083059s) [upload=531240]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 21:37:06.490986034 (606h8m6.362069079s) [upload=531240]: Done uploading segments... Dec 30 21:37:06.491006261 (606h8m6.362089307s) [upload=531240]: Upload finished. err: <nil> Duration: 17.437958672s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)