Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......X..X...X.........X.........................X.........] duration: 27.376662441s trace: https://jaeger.dev.storj.io/trace/57a7ab98e7eb0d3f FAIL (2025-12-30 22:05:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 21:58:42.298454527 (606h29m42.169537592s) [upload=531550]: Starting upload Dec 30 21:58:42.298523438 (606h29m42.169606473s) [upload=531550]: Done starting upload Dec 30 21:58:42.298568774 (606h29m42.169651809s) [upload=531550]: Uploading segments... Dec 30 21:58:42.298697850 (606h29m42.169780885s) [upload=531550]: Got next segment. Inline: false Dec 30 21:58:42.298728649 (606h29m42.169811694s) [upload=531550]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 21:58:42.298733227 (606h29m42.169816262s) [upload=531550]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 21:58:42.545734213 (606h29m42.416817288s) [seg_pos={0 0}, upload=531550]: Begin upload segment... Dec 30 21:58:42.545786032 (606h29m42.416869078s) [seg_pos={0 0}, upload=531550]: Done begin upload segment. Dec 30 21:58:42.545855104 (606h29m42.416938149s) [noise=true , address=145.79.227.23:4547 , piece_id=LYGUMTAP7OPJWGRVFY65GUZFE4BBTV73JTG657DR5OM26LA7UFZA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.545897384 (606h29m42.416980449s) [noise=true , address=145.79.227.49:4339 , piece_id=WBBPZK2VY2QVRTQNXEKQ7TRYPWBDYS2TGTKEDAAXQ2AT4TUQXTQQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.546757752 (606h29m42.417840817s) [noise=true , address=45.91.116.12:4225 , piece_id=LLYEKE4TJ54KECLAC7KFUV44IVHYCYLF45I43G5V3DPV5QDBUOIQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.546816995 (606h29m42.41790004s) [noise=true , address=145.79.227.40:4050 , piece_id=LA7KLHHWRNHULLPAZWQ6MLS76SLN3BONYXZJXCUHGFC5JZ4SQRIA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.547216927 (606h29m42.418299962s) [noise=true , address=145.79.227.15:4876 , piece_id=L2OIJAB6OOANT2EPFNVNLTMYCH334JGFBFASD44EHOAWTAHKUV5Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.547615996 (606h29m42.418699042s) [noise=true , address=145.79.227.17:4971 , piece_id=DYZUALBL3V75TKGK57LOJABWVCOD5UVS3WH74I55OVUDGRQUA3QQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.547841055 (606h29m42.4189241s) [noise=true , address=77.90.31.21:4877 , piece_id=BD2FEFNAGYWZYDA2IMOVHA7JE66EXSF3A3PL54Q2DUOJFG5ALG4Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548023322 (606h29m42.419106357s) [noise=true , address=45.91.116.14:4190 , piece_id=HHU5YZBTJEEWYVST6XQCKGWKRLRQIIR2UQTQ27RLK53BNAVOSHFQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548380923 (606h29m42.419463968s) [noise=true , address=45.91.116.11:4120 , piece_id=FZ5L6FM7IHX65EVFGVXXHEF43XQ4EBUH4C7VZUSTH2MVWLNR5M2Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548560475 (606h29m42.41964351s) [noise=true , address=145.79.227.39:4013 , piece_id=I4XP47SFJICIYIPEN4E7UC2HXRXH4IXVJYGHXTZBCUESAHNFQXJA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548719268 (606h29m42.419802313s) [noise=true , address=145.79.227.31:4797 , piece_id=S5YBLITONGU6RHPQENP2TCVVWOHEMXPMYXLNYWBZNXWMFI26PVUA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549024107 (606h29m42.420107152s) [noise=true , address=145.79.227.7:4294 , piece_id=C6N5OIRLV3LLSCEIOGQJKWPKENGZYNAPY6V2AYZNKUMACBP5SLWQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549197708 (606h29m42.420280753s) [noise=true , address=145.79.227.16:4910 , piece_id=O4YO367LEVZSRI5E75YEZ7W7YI2KGKKQDXCOHQ3LALE4WUAPQVHQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549506406 (606h29m42.420589441s) [noise=true , address=145.79.227.12:4765 , piece_id=4ZF5YQP4Z5HBTX5YWUDHZV55Q47WASRLMV6ZYJXJRL2QBAVQQM4Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549971591 (606h29m42.421054626s) [noise=true , address=145.79.227.32:4835 , piece_id=CN4NMCFNYKOCG2JCIEXJBL2PLEYWD5THCZB4WQDURIECFVHLLX6Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.551571588 (606h29m42.422654623s) [noise=true , address=145.79.227.24:4581 , piece_id=Y6GBEOR7DQX2H6LBHXYFOXRBX3GP7AKPYVGTQNMUEXKPXLONUUGQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:44.247717212 (606h29m44.118800248s) [upload=531550]: Next returned nil segment Dec 30 21:58:44.247730348 (606h29m44.118813383s) [upload=531550]: Waiting for error group managing segments... Dec 30 21:58:44.532148720 (606h29m44.403231755s) [noise=true , address=145.79.227.31:4797 , piece_id=S5YBLITONGU6RHPQENP2TCVVWOHEMXPMYXLNYWBZNXWMFI26PVUA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.534655293 (606h29m44.405738328s) [noise=true , address=145.79.227.39:4013 , piece_id=I4XP47SFJICIYIPEN4E7UC2HXRXH4IXVJYGHXTZBCUESAHNFQXJA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.544709427 (606h29m44.415792462s) [noise=true , address=145.79.227.12:4765 , piece_id=4ZF5YQP4Z5HBTX5YWUDHZV55Q47WASRLMV6ZYJXJRL2QBAVQQM4Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.545822306 (606h29m44.416905342s) [noise=true , address=145.79.227.7:4294 , piece_id=C6N5OIRLV3LLSCEIOGQJKWPKENGZYNAPY6V2AYZNKUMACBP5SLWQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.549465162 (606h29m44.420548198s) [noise=true , address=145.79.227.24:4581 , piece_id=Y6GBEOR7DQX2H6LBHXYFOXRBX3GP7AKPYVGTQNMUEXKPXLONUUGQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.556935928 (606h29m44.428018964s) [noise=true , address=145.79.227.15:4876 , piece_id=L2OIJAB6OOANT2EPFNVNLTMYCH334JGFBFASD44EHOAWTAHKUV5Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.561283296 (606h29m44.432366331s) [noise=true , address=145.79.227.16:4910 , piece_id=O4YO367LEVZSRI5E75YEZ7W7YI2KGKKQDXCOHQ3LALE4WUAPQVHQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.614194264 (606h29m44.485277309s) [noise=true , address=145.79.227.17:4971 , piece_id=DYZUALBL3V75TKGK57LOJABWVCOD5UVS3WH74I55OVUDGRQUA3QQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.615630689 (606h29m44.486713724s) [noise=true , address=145.79.227.32:4835 , piece_id=CN4NMCFNYKOCG2JCIEXJBL2PLEYWD5THCZB4WQDURIECFVHLLX6Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.639672077 (606h29m44.510755122s) [noise=true , address=145.79.227.23:4547 , piece_id=LYGUMTAP7OPJWGRVFY65GUZFE4BBTV73JTG657DR5OM26LA7UFZA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.699618771 (606h29m44.570701817s) [noise=true , address=77.90.31.21:4877 , piece_id=BD2FEFNAGYWZYDA2IMOVHA7JE66EXSF3A3PL54Q2DUOJFG5ALG4Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.841535416 (606h29m44.712618462s) [noise=true , address=45.91.116.14:4190 , piece_id=HHU5YZBTJEEWYVST6XQCKGWKRLRQIIR2UQTQ27RLK53BNAVOSHFQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.974660841 (606h29m44.845743886s) [noise=true , address=45.91.116.12:4225 , piece_id=LLYEKE4TJ54KECLAC7KFUV44IVHYCYLF45I43G5V3DPV5QDBUOIQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.974681401 (606h29m44.845764436s) [seg_pos={0 0}, upload=531550]: Segment reached optimal threshold of 13 pieces. Dec 30 21:58:44.974855342 (606h29m44.845938377s) [noise=true , address=145.79.227.49:4339 , piece_id=WBBPZK2VY2QVRTQNXEKQ7TRYPWBDYS2TGTKEDAAXQ2AT4TUQXTQQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 30 21:58:44.974861423 (606h29m44.845944458s) [noise=true , address=145.79.227.49:4339 , piece_id=WBBPZK2VY2QVRTQNXEKQ7TRYPWBDYS2TGTKEDAAXQ2AT4TUQXTQQ, seg_pos={0 0}, upload=531550]: Long tail cancellation due to optimal threshold Dec 30 21:58:44.974968166 (606h29m44.846051201s) [noise=true , address=145.79.227.40:4050 , piece_id=LA7KLHHWRNHULLPAZWQ6MLS76SLN3BONYXZJXCUHGFC5JZ4SQRIA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 30 21:58:44.974973627 (606h29m44.846056662s) [noise=true , address=145.79.227.40:4050 , piece_id=LA7KLHHWRNHULLPAZWQ6MLS76SLN3BONYXZJXCUHGFC5JZ4SQRIA, seg_pos={0 0}, upload=531550]: Long tail cancellation due to optimal threshold Dec 30 21:58:44.975076152 (606h29m44.846159187s) [noise=true , address=45.91.116.11:4120 , piece_id=FZ5L6FM7IHX65EVFGVXXHEF43XQ4EBUH4C7VZUSTH2MVWLNR5M2Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 30 21:58:44.975083497 (606h29m44.846166532s) [noise=true , address=45.91.116.11:4120 , piece_id=FZ5L6FM7IHX65EVFGVXXHEF43XQ4EBUH4C7VZUSTH2MVWLNR5M2Q, seg_pos={0 0}, upload=531550]: Long tail cancellation due to optimal threshold Dec 30 21:58:44.975096711 (606h29m44.846179747s) [seg_pos={0 0}, upload=531550]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 21:58:44.975160703 (606h29m44.846243739s) [upload=531550]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 21:58:44.975164691 (606h29m44.846247726s) [upload=531550]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 21:58:45.272782726 (606h29m45.143865772s) [upload=531550]: Done uploading segments... Dec 30 21:58:45.272806352 (606h29m45.143889387s) [upload=531550]: Upload finished. err: <nil> Duration: 27.376688792s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............................................X...............] duration: 3.873861169s trace: https://jaeger.dev.storj.io/trace/22a47ec27f9a101c FAIL (2025-12-30 22:04:56 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.873889693s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [........XX......X.........X.................................] duration: 15.317844753s trace: https://jaeger.dev.storj.io/trace/5cdd5ba7b1cb22c7 FAIL (2025-12-30 22:04:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 21:37:03.355296401 (606h8m3.226379446s) [noise=true , address=145.79.227.22:4543 , piece_id=WWCZ6SEEXQ2KUJ32AGTIJNMSJROLYSJ4XQGBQJC4Q5DACZVA65RQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.355839374 (606h8m3.22692241s) [noise=true , address=145.79.227.35:4492 , piece_id=HCKFSQMJYEP6B5H4OX5AZ5TWP6Z4H4Z6WA2T7Z2H5KNZ4SDJRMUQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.356356930 (606h8m3.227439965s) [noise=true , address=45.91.116.13:4182 , piece_id=BYAYNP4JAAN3AW2LYSK74WJE4BRWUR5Z6K67MTLK6A73KEJ35WEQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.356675316 (606h8m3.227758361s) [noise=true , address=77.90.31.12:4612 , piece_id=63TS3SUEFUE6PORZ4W76OXRWFL7SBN773UGGY3DU7JPBAW66N7UA, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.356992732 (606h8m3.228075767s) [noise=true , address=64.49.13.140:4507 , piece_id=E7T6WL45F6GHECCRANX6AIKWFRZ77TURRWORYEYI43MET2WG7SRQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.357330644 (606h8m3.228413679s) [noise=true , address=77.90.31.21:4886 , piece_id=ZQRCPTVC3WFVQPZ36AU6KJW7PD3T557CXYQIICLDGOKK6JCPVZ5Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.357663649 (606h8m3.228746684s) [noise=true , address=145.79.227.49:4367 , piece_id=R2T2ABLXBPDVP3ZVA2TAYMCLQQ7JQUVQYCTHCKGMFBRVWCDAO4PQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.359827449 (606h8m3.230910484s) [noise=true , address=145.79.227.34:4918 , piece_id=HUMOHOT3HOYCBJ23WXQXWU3X6C6JQTRLLHTD4RQLCQTCB3HXI7EQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.360086692 (606h8m3.231169727s) [noise=true , address=145.79.227.7:4320 , piece_id=MAICNTHGLZ6W27HXDOOZXOBRGQGEXNSA4BJT2T5D5PHRVV6ZMX2Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.361622167 (606h8m3.232705212s) [noise=true , address=145.79.227.33:4888 , piece_id=I4426P3VETHPOL5WHWJEGVYRXJULTCNDX3WSUTS3XZVY36Q2N44Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.362965514 (606h8m3.234048559s) [noise=true , address=45.91.116.11:4133 , piece_id=NEW7ZOVP5E7HYGYD4LG5UG2MDLAQW2OBRTNZDFIYO4CQYMVAWRPQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.364308883 (606h8m3.235391928s) [noise=true , address=145.79.227.26:4663 , piece_id=DYEFQZL4X74QR2LAYURT5ZTR2IWIDB55VLBQOGZM36MA54P2SWJA, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.365734507 (606h8m3.236817542s) [noise=true , address=145.79.227.40:4059 , piece_id=7WSUBR3C3PWVBCWD6KRGSD6MPX4IH2YQOAVO4J4SWZORNHGS2IPQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.366982764 (606h8m3.238065799s) [noise=true , address=145.79.227.29:4973 , piece_id=5OROXELR55A47BFPSJJEWVQU4YW6CAOK6LMEHCINM4FN3OVI2Y6Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.368390294 (606h8m3.239473339s) [noise=true , address=77.90.31.19:4852 , piece_id=2NYO63GCBGCYBFYTMUGNF2NG5FSJOD3AOUOD2FZZIWDINK4X6DJQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:04.736295138 (606h8m4.607378183s) [upload=531240]: Next returned nil segment Dec 30 21:37:04.736311850 (606h8m4.607394885s) [upload=531240]: Waiting for error group managing segments... Dec 30 21:37:05.104989591 (606h8m4.976072636s) [noise=true , address=64.49.13.140:4507 , piece_id=E7T6WL45F6GHECCRANX6AIKWFRZ77TURRWORYEYI43MET2WG7SRQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.208809748 (606h8m5.079892793s) [noise=true , address=145.79.227.22:4543 , piece_id=WWCZ6SEEXQ2KUJ32AGTIJNMSJROLYSJ4XQGBQJC4Q5DACZVA65RQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.441125389 (606h8m5.312208444s) [noise=true , address=77.90.31.19:4852 , piece_id=2NYO63GCBGCYBFYTMUGNF2NG5FSJOD3AOUOD2FZZIWDINK4X6DJQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.483630904 (606h8m5.35471394s) [noise=true , address=77.90.31.12:4612 , piece_id=63TS3SUEFUE6PORZ4W76OXRWFL7SBN773UGGY3DU7JPBAW66N7UA, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.494478811 (606h8m5.365561846s) [noise=true , address=77.90.31.11:4559 , piece_id=3DYD4CNPJQN3S4EYZJGREQTTZJDSL2QTC3OT6G2VFUGMLSVCZUFA, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.497452683 (606h8m5.368535729s) [noise=true , address=77.90.31.17:4769 , piece_id=QXDYESBKGXRY7BCFYYF2O7OXS7Y4Z4KUMJB6AXZAWX735SYU56GQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.661529377 (606h8m5.532612412s) [noise=true , address=45.91.116.13:4182 , piece_id=BYAYNP4JAAN3AW2LYSK74WJE4BRWUR5Z6K67MTLK6A73KEJ35WEQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.729773112 (606h8m5.600856157s) [noise=true , address=45.91.116.11:4133 , piece_id=NEW7ZOVP5E7HYGYD4LG5UG2MDLAQW2OBRTNZDFIYO4CQYMVAWRPQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.940743308 (606h8m5.811826353s) [noise=true , address=77.90.31.21:4886 , piece_id=ZQRCPTVC3WFVQPZ36AU6KJW7PD3T557CXYQIICLDGOKK6JCPVZ5Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.084240044 (606h8m5.95532309s) [noise=true , address=145.79.227.30:4784 , piece_id=DSXUDNYCWO5F6IYXMIVJNEASYXIZ7EO7RKAS7QRUYK5OU7GOOB5Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.121095032 (606h8m5.992178067s) [noise=true , address=145.79.227.40:4059 , piece_id=7WSUBR3C3PWVBCWD6KRGSD6MPX4IH2YQOAVO4J4SWZORNHGS2IPQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.169195983 (606h8m6.040279028s) [noise=true , address=145.79.227.49:4367 , piece_id=R2T2ABLXBPDVP3ZVA2TAYMCLQQ7JQUVQYCTHCKGMFBRVWCDAO4PQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.169380634 (606h8m6.040463679s) [noise=true , address=145.79.227.33:4888 , piece_id=I4426P3VETHPOL5WHWJEGVYRXJULTCNDX3WSUTS3XZVY36Q2N44Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.179060175 (606h8m6.050143211s) [noise=true , address=145.79.227.29:4973 , piece_id=5OROXELR55A47BFPSJJEWVQU4YW6CAOK6LMEHCINM4FN3OVI2Y6Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.179080424 (606h8m6.050163459s) [seg_pos={0 0}, upload=531240]: Segment reached optimal threshold of 14 pieces. Dec 30 21:37:06.179262801 (606h8m6.050345837s) [noise=true , address=145.79.227.26:4663 , piece_id=DYEFQZL4X74QR2LAYURT5ZTR2IWIDB55VLBQOGZM36MA54P2SWJA, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SwpAaXTA3BqUvwubzmJkvwokdz6SSpT9t8HPtQcXXJxmcgfm2): protocol: write tcp 188.166.225.170:43184->145.79.227.26:4663: use of closed network connection; write tcp 188.166.225.170:43184->145.79.227.26:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43184->145.79.227.26:4663: use of closed network connection Dec 30 21:37:06.179268703 (606h8m6.050351748s) [noise=true , address=145.79.227.26:4663 , piece_id=DYEFQZL4X74QR2LAYURT5ZTR2IWIDB55VLBQOGZM36MA54P2SWJA, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179363724 (606h8m6.050446769s) [noise=true , address=145.79.227.7:4320 , piece_id=MAICNTHGLZ6W27HXDOOZXOBRGQGEXNSA4BJT2T5D5PHRVV6ZMX2Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126rEgnQw1w7hsFpRqQP9nLaumdBfdtkgYjdw6WE6nXrxk5kj4B): protocol: write tcp 188.166.225.170:45228->145.79.227.7:4320: use of closed network connection; write tcp 188.166.225.170:45228->145.79.227.7:4320: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45228->145.79.227.7:4320: use of closed network connection Dec 30 21:37:06.179369525 (606h8m6.05045257s) [noise=true , address=145.79.227.7:4320 , piece_id=MAICNTHGLZ6W27HXDOOZXOBRGQGEXNSA4BJT2T5D5PHRVV6ZMX2Q, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179532996 (606h8m6.050616061s) [noise=true , address=145.79.227.34:4918 , piece_id=HUMOHOT3HOYCBJ23WXQXWU3X6C6JQTRLLHTD4RQLCQTCB3HXI7EQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF Dec 30 21:37:06.179572371 (606h8m6.050655406s) [noise=true , address=145.79.227.34:4918 , piece_id=HUMOHOT3HOYCBJ23WXQXWU3X6C6JQTRLLHTD4RQLCQTCB3HXI7EQ, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179638587 (606h8m6.050721622s) [noise=true , address=145.79.227.35:4492 , piece_id=HCKFSQMJYEP6B5H4OX5AZ5TWP6Z4H4Z6WA2T7Z2H5KNZ4SDJRMUQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1roa5GGYmqCX2AxSrqFgqyV2CtBEYZavuGTseiwDy4pjkT6bip): protocol: expected piece hash; context canceled; EOF Dec 30 21:37:06.179647855 (606h8m6.05073089s) [noise=true , address=145.79.227.35:4492 , piece_id=HCKFSQMJYEP6B5H4OX5AZ5TWP6Z4H4Z6WA2T7Z2H5KNZ4SDJRMUQ, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179666641 (606h8m6.050749686s) [seg_pos={0 0}, upload=531240]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 21:37:06.179742766 (606h8m6.050825811s) [upload=531240]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 21:37:06.179747545 (606h8m6.05083059s) [upload=531240]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 21:37:06.490986034 (606h8m6.362069079s) [upload=531240]: Done uploading segments... Dec 30 21:37:06.491006261 (606h8m6.362089307s) [upload=531240]: Upload finished. err: <nil> Duration: 15.317888076s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [........X..X...X.........X.........................X........] duration: 5.913179574s trace: https://jaeger.dev.storj.io/trace/57a7ab98e7eb0d3f FAIL (2025-12-30 22:04:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 21:58:42.298454527 (606h29m42.169537592s) [upload=531550]: Starting upload Dec 30 21:58:42.298523438 (606h29m42.169606473s) [upload=531550]: Done starting upload Dec 30 21:58:42.298568774 (606h29m42.169651809s) [upload=531550]: Uploading segments... Dec 30 21:58:42.298697850 (606h29m42.169780885s) [upload=531550]: Got next segment. Inline: false Dec 30 21:58:42.298728649 (606h29m42.169811694s) [upload=531550]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 21:58:42.298733227 (606h29m42.169816262s) [upload=531550]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 21:58:42.545734213 (606h29m42.416817288s) [seg_pos={0 0}, upload=531550]: Begin upload segment... Dec 30 21:58:42.545786032 (606h29m42.416869078s) [seg_pos={0 0}, upload=531550]: Done begin upload segment. Dec 30 21:58:42.545855104 (606h29m42.416938149s) [noise=true , address=145.79.227.23:4547 , piece_id=LYGUMTAP7OPJWGRVFY65GUZFE4BBTV73JTG657DR5OM26LA7UFZA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.545897384 (606h29m42.416980449s) [noise=true , address=145.79.227.49:4339 , piece_id=WBBPZK2VY2QVRTQNXEKQ7TRYPWBDYS2TGTKEDAAXQ2AT4TUQXTQQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.546757752 (606h29m42.417840817s) [noise=true , address=45.91.116.12:4225 , piece_id=LLYEKE4TJ54KECLAC7KFUV44IVHYCYLF45I43G5V3DPV5QDBUOIQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.546816995 (606h29m42.41790004s) [noise=true , address=145.79.227.40:4050 , piece_id=LA7KLHHWRNHULLPAZWQ6MLS76SLN3BONYXZJXCUHGFC5JZ4SQRIA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.547216927 (606h29m42.418299962s) [noise=true , address=145.79.227.15:4876 , piece_id=L2OIJAB6OOANT2EPFNVNLTMYCH334JGFBFASD44EHOAWTAHKUV5Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.547615996 (606h29m42.418699042s) [noise=true , address=145.79.227.17:4971 , piece_id=DYZUALBL3V75TKGK57LOJABWVCOD5UVS3WH74I55OVUDGRQUA3QQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.547841055 (606h29m42.4189241s) [noise=true , address=77.90.31.21:4877 , piece_id=BD2FEFNAGYWZYDA2IMOVHA7JE66EXSF3A3PL54Q2DUOJFG5ALG4Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548023322 (606h29m42.419106357s) [noise=true , address=45.91.116.14:4190 , piece_id=HHU5YZBTJEEWYVST6XQCKGWKRLRQIIR2UQTQ27RLK53BNAVOSHFQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548380923 (606h29m42.419463968s) [noise=true , address=45.91.116.11:4120 , piece_id=FZ5L6FM7IHX65EVFGVXXHEF43XQ4EBUH4C7VZUSTH2MVWLNR5M2Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548560475 (606h29m42.41964351s) [noise=true , address=145.79.227.39:4013 , piece_id=I4XP47SFJICIYIPEN4E7UC2HXRXH4IXVJYGHXTZBCUESAHNFQXJA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548719268 (606h29m42.419802313s) [noise=true , address=145.79.227.31:4797 , piece_id=S5YBLITONGU6RHPQENP2TCVVWOHEMXPMYXLNYWBZNXWMFI26PVUA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549024107 (606h29m42.420107152s) [noise=true , address=145.79.227.7:4294 , piece_id=C6N5OIRLV3LLSCEIOGQJKWPKENGZYNAPY6V2AYZNKUMACBP5SLWQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549197708 (606h29m42.420280753s) [noise=true , address=145.79.227.16:4910 , piece_id=O4YO367LEVZSRI5E75YEZ7W7YI2KGKKQDXCOHQ3LALE4WUAPQVHQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549506406 (606h29m42.420589441s) [noise=true , address=145.79.227.12:4765 , piece_id=4ZF5YQP4Z5HBTX5YWUDHZV55Q47WASRLMV6ZYJXJRL2QBAVQQM4Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549971591 (606h29m42.421054626s) [noise=true , address=145.79.227.32:4835 , piece_id=CN4NMCFNYKOCG2JCIEXJBL2PLEYWD5THCZB4WQDURIECFVHLLX6Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.551571588 (606h29m42.422654623s) [noise=true , address=145.79.227.24:4581 , piece_id=Y6GBEOR7DQX2H6LBHXYFOXRBX3GP7AKPYVGTQNMUEXKPXLONUUGQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:44.247717212 (606h29m44.118800248s) [upload=531550]: Next returned nil segment Dec 30 21:58:44.247730348 (606h29m44.118813383s) [upload=531550]: Waiting for error group managing segments... Dec 30 21:58:44.532148720 (606h29m44.403231755s) [noise=true , address=145.79.227.31:4797 , piece_id=S5YBLITONGU6RHPQENP2TCVVWOHEMXPMYXLNYWBZNXWMFI26PVUA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.534655293 (606h29m44.405738328s) [noise=true , address=145.79.227.39:4013 , piece_id=I4XP47SFJICIYIPEN4E7UC2HXRXH4IXVJYGHXTZBCUESAHNFQXJA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.544709427 (606h29m44.415792462s) [noise=true , address=145.79.227.12:4765 , piece_id=4ZF5YQP4Z5HBTX5YWUDHZV55Q47WASRLMV6ZYJXJRL2QBAVQQM4Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.545822306 (606h29m44.416905342s) [noise=true , address=145.79.227.7:4294 , piece_id=C6N5OIRLV3LLSCEIOGQJKWPKENGZYNAPY6V2AYZNKUMACBP5SLWQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.549465162 (606h29m44.420548198s) [noise=true , address=145.79.227.24:4581 , piece_id=Y6GBEOR7DQX2H6LBHXYFOXRBX3GP7AKPYVGTQNMUEXKPXLONUUGQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.556935928 (606h29m44.428018964s) [noise=true , address=145.79.227.15:4876 , piece_id=L2OIJAB6OOANT2EPFNVNLTMYCH334JGFBFASD44EHOAWTAHKUV5Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.561283296 (606h29m44.432366331s) [noise=true , address=145.79.227.16:4910 , piece_id=O4YO367LEVZSRI5E75YEZ7W7YI2KGKKQDXCOHQ3LALE4WUAPQVHQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.614194264 (606h29m44.485277309s) [noise=true , address=145.79.227.17:4971 , piece_id=DYZUALBL3V75TKGK57LOJABWVCOD5UVS3WH74I55OVUDGRQUA3QQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.615630689 (606h29m44.486713724s) [noise=true , address=145.79.227.32:4835 , piece_id=CN4NMCFNYKOCG2JCIEXJBL2PLEYWD5THCZB4WQDURIECFVHLLX6Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.639672077 (606h29m44.510755122s) [noise=true , address=145.79.227.23:4547 , piece_id=LYGUMTAP7OPJWGRVFY65GUZFE4BBTV73JTG657DR5OM26LA7UFZA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.699618771 (606h29m44.570701817s) [noise=true , address=77.90.31.21:4877 , piece_id=BD2FEFNAGYWZYDA2IMOVHA7JE66EXSF3A3PL54Q2DUOJFG5ALG4Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.841535416 (606h29m44.712618462s) [noise=true , address=45.91.116.14:4190 , piece_id=HHU5YZBTJEEWYVST6XQCKGWKRLRQIIR2UQTQ27RLK53BNAVOSHFQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.974660841 (606h29m44.845743886s) [noise=true , address=45.91.116.12:4225 , piece_id=LLYEKE4TJ54KECLAC7KFUV44IVHYCYLF45I43G5V3DPV5QDBUOIQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.974681401 (606h29m44.845764436s) [seg_pos={0 0}, upload=531550]: Segment reached optimal threshold of 13 pieces. Dec 30 21:58:44.974855342 (606h29m44.845938377s) [noise=true , address=145.79.227.49:4339 , piece_id=WBBPZK2VY2QVRTQNXEKQ7TRYPWBDYS2TGTKEDAAXQ2AT4TUQXTQQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 30 21:58:44.974861423 (606h29m44.845944458s) [noise=true , address=145.79.227.49:4339 , piece_id=WBBPZK2VY2QVRTQNXEKQ7TRYPWBDYS2TGTKEDAAXQ2AT4TUQXTQQ, seg_pos={0 0}, upload=531550]: Long tail cancellation due to optimal threshold Dec 30 21:58:44.974968166 (606h29m44.846051201s) [noise=true , address=145.79.227.40:4050 , piece_id=LA7KLHHWRNHULLPAZWQ6MLS76SLN3BONYXZJXCUHGFC5JZ4SQRIA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 30 21:58:44.974973627 (606h29m44.846056662s) [noise=true , address=145.79.227.40:4050 , piece_id=LA7KLHHWRNHULLPAZWQ6MLS76SLN3BONYXZJXCUHGFC5JZ4SQRIA, seg_pos={0 0}, upload=531550]: Long tail cancellation due to optimal threshold Dec 30 21:58:44.975076152 (606h29m44.846159187s) [noise=true , address=45.91.116.11:4120 , piece_id=FZ5L6FM7IHX65EVFGVXXHEF43XQ4EBUH4C7VZUSTH2MVWLNR5M2Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 30 21:58:44.975083497 (606h29m44.846166532s) [noise=true , address=45.91.116.11:4120 , piece_id=FZ5L6FM7IHX65EVFGVXXHEF43XQ4EBUH4C7VZUSTH2MVWLNR5M2Q, seg_pos={0 0}, upload=531550]: Long tail cancellation due to optimal threshold Dec 30 21:58:44.975096711 (606h29m44.846179747s) [seg_pos={0 0}, upload=531550]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 21:58:44.975160703 (606h29m44.846243739s) [upload=531550]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 21:58:44.975164691 (606h29m44.846247726s) [upload=531550]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 21:58:45.272782726 (606h29m45.143865772s) [upload=531550]: Done uploading segments... Dec 30 21:58:45.272806352 (606h29m45.143889387s) [upload=531550]: Upload finished. err: <nil> Duration: 5.913205493s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.........XX......X.........X................................] duration: 53.077425128s trace: https://jaeger.dev.storj.io/trace/5cdd5ba7b1cb22c7 FAIL (2025-12-30 22:04:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 21:37:03.355296401 (606h8m3.226379446s) [noise=true , address=145.79.227.22:4543 , piece_id=WWCZ6SEEXQ2KUJ32AGTIJNMSJROLYSJ4XQGBQJC4Q5DACZVA65RQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.355839374 (606h8m3.22692241s) [noise=true , address=145.79.227.35:4492 , piece_id=HCKFSQMJYEP6B5H4OX5AZ5TWP6Z4H4Z6WA2T7Z2H5KNZ4SDJRMUQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.356356930 (606h8m3.227439965s) [noise=true , address=45.91.116.13:4182 , piece_id=BYAYNP4JAAN3AW2LYSK74WJE4BRWUR5Z6K67MTLK6A73KEJ35WEQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.356675316 (606h8m3.227758361s) [noise=true , address=77.90.31.12:4612 , piece_id=63TS3SUEFUE6PORZ4W76OXRWFL7SBN773UGGY3DU7JPBAW66N7UA, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.356992732 (606h8m3.228075767s) [noise=true , address=64.49.13.140:4507 , piece_id=E7T6WL45F6GHECCRANX6AIKWFRZ77TURRWORYEYI43MET2WG7SRQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.357330644 (606h8m3.228413679s) [noise=true , address=77.90.31.21:4886 , piece_id=ZQRCPTVC3WFVQPZ36AU6KJW7PD3T557CXYQIICLDGOKK6JCPVZ5Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.357663649 (606h8m3.228746684s) [noise=true , address=145.79.227.49:4367 , piece_id=R2T2ABLXBPDVP3ZVA2TAYMCLQQ7JQUVQYCTHCKGMFBRVWCDAO4PQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.359827449 (606h8m3.230910484s) [noise=true , address=145.79.227.34:4918 , piece_id=HUMOHOT3HOYCBJ23WXQXWU3X6C6JQTRLLHTD4RQLCQTCB3HXI7EQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.360086692 (606h8m3.231169727s) [noise=true , address=145.79.227.7:4320 , piece_id=MAICNTHGLZ6W27HXDOOZXOBRGQGEXNSA4BJT2T5D5PHRVV6ZMX2Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.361622167 (606h8m3.232705212s) [noise=true , address=145.79.227.33:4888 , piece_id=I4426P3VETHPOL5WHWJEGVYRXJULTCNDX3WSUTS3XZVY36Q2N44Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.362965514 (606h8m3.234048559s) [noise=true , address=45.91.116.11:4133 , piece_id=NEW7ZOVP5E7HYGYD4LG5UG2MDLAQW2OBRTNZDFIYO4CQYMVAWRPQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.364308883 (606h8m3.235391928s) [noise=true , address=145.79.227.26:4663 , piece_id=DYEFQZL4X74QR2LAYURT5ZTR2IWIDB55VLBQOGZM36MA54P2SWJA, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.365734507 (606h8m3.236817542s) [noise=true , address=145.79.227.40:4059 , piece_id=7WSUBR3C3PWVBCWD6KRGSD6MPX4IH2YQOAVO4J4SWZORNHGS2IPQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.366982764 (606h8m3.238065799s) [noise=true , address=145.79.227.29:4973 , piece_id=5OROXELR55A47BFPSJJEWVQU4YW6CAOK6LMEHCINM4FN3OVI2Y6Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.368390294 (606h8m3.239473339s) [noise=true , address=77.90.31.19:4852 , piece_id=2NYO63GCBGCYBFYTMUGNF2NG5FSJOD3AOUOD2FZZIWDINK4X6DJQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:04.736295138 (606h8m4.607378183s) [upload=531240]: Next returned nil segment Dec 30 21:37:04.736311850 (606h8m4.607394885s) [upload=531240]: Waiting for error group managing segments... Dec 30 21:37:05.104989591 (606h8m4.976072636s) [noise=true , address=64.49.13.140:4507 , piece_id=E7T6WL45F6GHECCRANX6AIKWFRZ77TURRWORYEYI43MET2WG7SRQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.208809748 (606h8m5.079892793s) [noise=true , address=145.79.227.22:4543 , piece_id=WWCZ6SEEXQ2KUJ32AGTIJNMSJROLYSJ4XQGBQJC4Q5DACZVA65RQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.441125389 (606h8m5.312208444s) [noise=true , address=77.90.31.19:4852 , piece_id=2NYO63GCBGCYBFYTMUGNF2NG5FSJOD3AOUOD2FZZIWDINK4X6DJQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.483630904 (606h8m5.35471394s) [noise=true , address=77.90.31.12:4612 , piece_id=63TS3SUEFUE6PORZ4W76OXRWFL7SBN773UGGY3DU7JPBAW66N7UA, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.494478811 (606h8m5.365561846s) [noise=true , address=77.90.31.11:4559 , piece_id=3DYD4CNPJQN3S4EYZJGREQTTZJDSL2QTC3OT6G2VFUGMLSVCZUFA, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.497452683 (606h8m5.368535729s) [noise=true , address=77.90.31.17:4769 , piece_id=QXDYESBKGXRY7BCFYYF2O7OXS7Y4Z4KUMJB6AXZAWX735SYU56GQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.661529377 (606h8m5.532612412s) [noise=true , address=45.91.116.13:4182 , piece_id=BYAYNP4JAAN3AW2LYSK74WJE4BRWUR5Z6K67MTLK6A73KEJ35WEQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.729773112 (606h8m5.600856157s) [noise=true , address=45.91.116.11:4133 , piece_id=NEW7ZOVP5E7HYGYD4LG5UG2MDLAQW2OBRTNZDFIYO4CQYMVAWRPQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.940743308 (606h8m5.811826353s) [noise=true , address=77.90.31.21:4886 , piece_id=ZQRCPTVC3WFVQPZ36AU6KJW7PD3T557CXYQIICLDGOKK6JCPVZ5Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.084240044 (606h8m5.95532309s) [noise=true , address=145.79.227.30:4784 , piece_id=DSXUDNYCWO5F6IYXMIVJNEASYXIZ7EO7RKAS7QRUYK5OU7GOOB5Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.121095032 (606h8m5.992178067s) [noise=true , address=145.79.227.40:4059 , piece_id=7WSUBR3C3PWVBCWD6KRGSD6MPX4IH2YQOAVO4J4SWZORNHGS2IPQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.169195983 (606h8m6.040279028s) [noise=true , address=145.79.227.49:4367 , piece_id=R2T2ABLXBPDVP3ZVA2TAYMCLQQ7JQUVQYCTHCKGMFBRVWCDAO4PQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.169380634 (606h8m6.040463679s) [noise=true , address=145.79.227.33:4888 , piece_id=I4426P3VETHPOL5WHWJEGVYRXJULTCNDX3WSUTS3XZVY36Q2N44Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.179060175 (606h8m6.050143211s) [noise=true , address=145.79.227.29:4973 , piece_id=5OROXELR55A47BFPSJJEWVQU4YW6CAOK6LMEHCINM4FN3OVI2Y6Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.179080424 (606h8m6.050163459s) [seg_pos={0 0}, upload=531240]: Segment reached optimal threshold of 14 pieces. Dec 30 21:37:06.179262801 (606h8m6.050345837s) [noise=true , address=145.79.227.26:4663 , piece_id=DYEFQZL4X74QR2LAYURT5ZTR2IWIDB55VLBQOGZM36MA54P2SWJA, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SwpAaXTA3BqUvwubzmJkvwokdz6SSpT9t8HPtQcXXJxmcgfm2): protocol: write tcp 188.166.225.170:43184->145.79.227.26:4663: use of closed network connection; write tcp 188.166.225.170:43184->145.79.227.26:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43184->145.79.227.26:4663: use of closed network connection Dec 30 21:37:06.179268703 (606h8m6.050351748s) [noise=true , address=145.79.227.26:4663 , piece_id=DYEFQZL4X74QR2LAYURT5ZTR2IWIDB55VLBQOGZM36MA54P2SWJA, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179363724 (606h8m6.050446769s) [noise=true , address=145.79.227.7:4320 , piece_id=MAICNTHGLZ6W27HXDOOZXOBRGQGEXNSA4BJT2T5D5PHRVV6ZMX2Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126rEgnQw1w7hsFpRqQP9nLaumdBfdtkgYjdw6WE6nXrxk5kj4B): protocol: write tcp 188.166.225.170:45228->145.79.227.7:4320: use of closed network connection; write tcp 188.166.225.170:45228->145.79.227.7:4320: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45228->145.79.227.7:4320: use of closed network connection Dec 30 21:37:06.179369525 (606h8m6.05045257s) [noise=true , address=145.79.227.7:4320 , piece_id=MAICNTHGLZ6W27HXDOOZXOBRGQGEXNSA4BJT2T5D5PHRVV6ZMX2Q, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179532996 (606h8m6.050616061s) [noise=true , address=145.79.227.34:4918 , piece_id=HUMOHOT3HOYCBJ23WXQXWU3X6C6JQTRLLHTD4RQLCQTCB3HXI7EQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF Dec 30 21:37:06.179572371 (606h8m6.050655406s) [noise=true , address=145.79.227.34:4918 , piece_id=HUMOHOT3HOYCBJ23WXQXWU3X6C6JQTRLLHTD4RQLCQTCB3HXI7EQ, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179638587 (606h8m6.050721622s) [noise=true , address=145.79.227.35:4492 , piece_id=HCKFSQMJYEP6B5H4OX5AZ5TWP6Z4H4Z6WA2T7Z2H5KNZ4SDJRMUQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1roa5GGYmqCX2AxSrqFgqyV2CtBEYZavuGTseiwDy4pjkT6bip): protocol: expected piece hash; context canceled; EOF Dec 30 21:37:06.179647855 (606h8m6.05073089s) [noise=true , address=145.79.227.35:4492 , piece_id=HCKFSQMJYEP6B5H4OX5AZ5TWP6Z4H4Z6WA2T7Z2H5KNZ4SDJRMUQ, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179666641 (606h8m6.050749686s) [seg_pos={0 0}, upload=531240]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 21:37:06.179742766 (606h8m6.050825811s) [upload=531240]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 21:37:06.179747545 (606h8m6.05083059s) [upload=531240]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 21:37:06.490986034 (606h8m6.362069079s) [upload=531240]: Done uploading segments... Dec 30 21:37:06.491006261 (606h8m6.362089307s) [upload=531240]: Upload finished. err: <nil> Duration: 53.077450334s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............................................X..............] duration: 3.590158453s trace: https://jaeger.dev.storj.io/trace/22a47ec27f9a101c FAIL (2025-12-30 22:03:42 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.590175826s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.........X..X...X.........X.........................X.......] duration: 5.851205063s trace: https://jaeger.dev.storj.io/trace/57a7ab98e7eb0d3f FAIL (2025-12-30 22:03:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 21:58:42.298454527 (606h29m42.169537592s) [upload=531550]: Starting upload Dec 30 21:58:42.298523438 (606h29m42.169606473s) [upload=531550]: Done starting upload Dec 30 21:58:42.298568774 (606h29m42.169651809s) [upload=531550]: Uploading segments... Dec 30 21:58:42.298697850 (606h29m42.169780885s) [upload=531550]: Got next segment. Inline: false Dec 30 21:58:42.298728649 (606h29m42.169811694s) [upload=531550]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 21:58:42.298733227 (606h29m42.169816262s) [upload=531550]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 21:58:42.545734213 (606h29m42.416817288s) [seg_pos={0 0}, upload=531550]: Begin upload segment... Dec 30 21:58:42.545786032 (606h29m42.416869078s) [seg_pos={0 0}, upload=531550]: Done begin upload segment. Dec 30 21:58:42.545855104 (606h29m42.416938149s) [noise=true , address=145.79.227.23:4547 , piece_id=LYGUMTAP7OPJWGRVFY65GUZFE4BBTV73JTG657DR5OM26LA7UFZA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.545897384 (606h29m42.416980449s) [noise=true , address=145.79.227.49:4339 , piece_id=WBBPZK2VY2QVRTQNXEKQ7TRYPWBDYS2TGTKEDAAXQ2AT4TUQXTQQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.546757752 (606h29m42.417840817s) [noise=true , address=45.91.116.12:4225 , piece_id=LLYEKE4TJ54KECLAC7KFUV44IVHYCYLF45I43G5V3DPV5QDBUOIQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.546816995 (606h29m42.41790004s) [noise=true , address=145.79.227.40:4050 , piece_id=LA7KLHHWRNHULLPAZWQ6MLS76SLN3BONYXZJXCUHGFC5JZ4SQRIA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.547216927 (606h29m42.418299962s) [noise=true , address=145.79.227.15:4876 , piece_id=L2OIJAB6OOANT2EPFNVNLTMYCH334JGFBFASD44EHOAWTAHKUV5Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.547615996 (606h29m42.418699042s) [noise=true , address=145.79.227.17:4971 , piece_id=DYZUALBL3V75TKGK57LOJABWVCOD5UVS3WH74I55OVUDGRQUA3QQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.547841055 (606h29m42.4189241s) [noise=true , address=77.90.31.21:4877 , piece_id=BD2FEFNAGYWZYDA2IMOVHA7JE66EXSF3A3PL54Q2DUOJFG5ALG4Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548023322 (606h29m42.419106357s) [noise=true , address=45.91.116.14:4190 , piece_id=HHU5YZBTJEEWYVST6XQCKGWKRLRQIIR2UQTQ27RLK53BNAVOSHFQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548380923 (606h29m42.419463968s) [noise=true , address=45.91.116.11:4120 , piece_id=FZ5L6FM7IHX65EVFGVXXHEF43XQ4EBUH4C7VZUSTH2MVWLNR5M2Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548560475 (606h29m42.41964351s) [noise=true , address=145.79.227.39:4013 , piece_id=I4XP47SFJICIYIPEN4E7UC2HXRXH4IXVJYGHXTZBCUESAHNFQXJA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548719268 (606h29m42.419802313s) [noise=true , address=145.79.227.31:4797 , piece_id=S5YBLITONGU6RHPQENP2TCVVWOHEMXPMYXLNYWBZNXWMFI26PVUA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549024107 (606h29m42.420107152s) [noise=true , address=145.79.227.7:4294 , piece_id=C6N5OIRLV3LLSCEIOGQJKWPKENGZYNAPY6V2AYZNKUMACBP5SLWQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549197708 (606h29m42.420280753s) [noise=true , address=145.79.227.16:4910 , piece_id=O4YO367LEVZSRI5E75YEZ7W7YI2KGKKQDXCOHQ3LALE4WUAPQVHQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549506406 (606h29m42.420589441s) [noise=true , address=145.79.227.12:4765 , piece_id=4ZF5YQP4Z5HBTX5YWUDHZV55Q47WASRLMV6ZYJXJRL2QBAVQQM4Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549971591 (606h29m42.421054626s) [noise=true , address=145.79.227.32:4835 , piece_id=CN4NMCFNYKOCG2JCIEXJBL2PLEYWD5THCZB4WQDURIECFVHLLX6Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.551571588 (606h29m42.422654623s) [noise=true , address=145.79.227.24:4581 , piece_id=Y6GBEOR7DQX2H6LBHXYFOXRBX3GP7AKPYVGTQNMUEXKPXLONUUGQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:44.247717212 (606h29m44.118800248s) [upload=531550]: Next returned nil segment Dec 30 21:58:44.247730348 (606h29m44.118813383s) [upload=531550]: Waiting for error group managing segments... Dec 30 21:58:44.532148720 (606h29m44.403231755s) [noise=true , address=145.79.227.31:4797 , piece_id=S5YBLITONGU6RHPQENP2TCVVWOHEMXPMYXLNYWBZNXWMFI26PVUA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.534655293 (606h29m44.405738328s) [noise=true , address=145.79.227.39:4013 , piece_id=I4XP47SFJICIYIPEN4E7UC2HXRXH4IXVJYGHXTZBCUESAHNFQXJA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.544709427 (606h29m44.415792462s) [noise=true , address=145.79.227.12:4765 , piece_id=4ZF5YQP4Z5HBTX5YWUDHZV55Q47WASRLMV6ZYJXJRL2QBAVQQM4Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.545822306 (606h29m44.416905342s) [noise=true , address=145.79.227.7:4294 , piece_id=C6N5OIRLV3LLSCEIOGQJKWPKENGZYNAPY6V2AYZNKUMACBP5SLWQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.549465162 (606h29m44.420548198s) [noise=true , address=145.79.227.24:4581 , piece_id=Y6GBEOR7DQX2H6LBHXYFOXRBX3GP7AKPYVGTQNMUEXKPXLONUUGQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.556935928 (606h29m44.428018964s) [noise=true , address=145.79.227.15:4876 , piece_id=L2OIJAB6OOANT2EPFNVNLTMYCH334JGFBFASD44EHOAWTAHKUV5Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.561283296 (606h29m44.432366331s) [noise=true , address=145.79.227.16:4910 , piece_id=O4YO367LEVZSRI5E75YEZ7W7YI2KGKKQDXCOHQ3LALE4WUAPQVHQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.614194264 (606h29m44.485277309s) [noise=true , address=145.79.227.17:4971 , piece_id=DYZUALBL3V75TKGK57LOJABWVCOD5UVS3WH74I55OVUDGRQUA3QQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.615630689 (606h29m44.486713724s) [noise=true , address=145.79.227.32:4835 , piece_id=CN4NMCFNYKOCG2JCIEXJBL2PLEYWD5THCZB4WQDURIECFVHLLX6Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.639672077 (606h29m44.510755122s) [noise=true , address=145.79.227.23:4547 , piece_id=LYGUMTAP7OPJWGRVFY65GUZFE4BBTV73JTG657DR5OM26LA7UFZA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.699618771 (606h29m44.570701817s) [noise=true , address=77.90.31.21:4877 , piece_id=BD2FEFNAGYWZYDA2IMOVHA7JE66EXSF3A3PL54Q2DUOJFG5ALG4Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.841535416 (606h29m44.712618462s) [noise=true , address=45.91.116.14:4190 , piece_id=HHU5YZBTJEEWYVST6XQCKGWKRLRQIIR2UQTQ27RLK53BNAVOSHFQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.974660841 (606h29m44.845743886s) [noise=true , address=45.91.116.12:4225 , piece_id=LLYEKE4TJ54KECLAC7KFUV44IVHYCYLF45I43G5V3DPV5QDBUOIQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.974681401 (606h29m44.845764436s) [seg_pos={0 0}, upload=531550]: Segment reached optimal threshold of 13 pieces. Dec 30 21:58:44.974855342 (606h29m44.845938377s) [noise=true , address=145.79.227.49:4339 , piece_id=WBBPZK2VY2QVRTQNXEKQ7TRYPWBDYS2TGTKEDAAXQ2AT4TUQXTQQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 30 21:58:44.974861423 (606h29m44.845944458s) [noise=true , address=145.79.227.49:4339 , piece_id=WBBPZK2VY2QVRTQNXEKQ7TRYPWBDYS2TGTKEDAAXQ2AT4TUQXTQQ, seg_pos={0 0}, upload=531550]: Long tail cancellation due to optimal threshold Dec 30 21:58:44.974968166 (606h29m44.846051201s) [noise=true , address=145.79.227.40:4050 , piece_id=LA7KLHHWRNHULLPAZWQ6MLS76SLN3BONYXZJXCUHGFC5JZ4SQRIA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 30 21:58:44.974973627 (606h29m44.846056662s) [noise=true , address=145.79.227.40:4050 , piece_id=LA7KLHHWRNHULLPAZWQ6MLS76SLN3BONYXZJXCUHGFC5JZ4SQRIA, seg_pos={0 0}, upload=531550]: Long tail cancellation due to optimal threshold Dec 30 21:58:44.975076152 (606h29m44.846159187s) [noise=true , address=45.91.116.11:4120 , piece_id=FZ5L6FM7IHX65EVFGVXXHEF43XQ4EBUH4C7VZUSTH2MVWLNR5M2Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 30 21:58:44.975083497 (606h29m44.846166532s) [noise=true , address=45.91.116.11:4120 , piece_id=FZ5L6FM7IHX65EVFGVXXHEF43XQ4EBUH4C7VZUSTH2MVWLNR5M2Q, seg_pos={0 0}, upload=531550]: Long tail cancellation due to optimal threshold Dec 30 21:58:44.975096711 (606h29m44.846179747s) [seg_pos={0 0}, upload=531550]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 21:58:44.975160703 (606h29m44.846243739s) [upload=531550]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 21:58:44.975164691 (606h29m44.846247726s) [upload=531550]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 21:58:45.272782726 (606h29m45.143865772s) [upload=531550]: Done uploading segments... Dec 30 21:58:45.272806352 (606h29m45.143889387s) [upload=531550]: Upload finished. err: <nil> Duration: 5.851231033s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [..........XX......X.........X...............................] duration: 15.446521166s trace: https://jaeger.dev.storj.io/trace/5cdd5ba7b1cb22c7 FAIL (2025-12-30 22:02:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 21:37:03.355296401 (606h8m3.226379446s) [noise=true , address=145.79.227.22:4543 , piece_id=WWCZ6SEEXQ2KUJ32AGTIJNMSJROLYSJ4XQGBQJC4Q5DACZVA65RQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.355839374 (606h8m3.22692241s) [noise=true , address=145.79.227.35:4492 , piece_id=HCKFSQMJYEP6B5H4OX5AZ5TWP6Z4H4Z6WA2T7Z2H5KNZ4SDJRMUQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.356356930 (606h8m3.227439965s) [noise=true , address=45.91.116.13:4182 , piece_id=BYAYNP4JAAN3AW2LYSK74WJE4BRWUR5Z6K67MTLK6A73KEJ35WEQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.356675316 (606h8m3.227758361s) [noise=true , address=77.90.31.12:4612 , piece_id=63TS3SUEFUE6PORZ4W76OXRWFL7SBN773UGGY3DU7JPBAW66N7UA, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.356992732 (606h8m3.228075767s) [noise=true , address=64.49.13.140:4507 , piece_id=E7T6WL45F6GHECCRANX6AIKWFRZ77TURRWORYEYI43MET2WG7SRQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.357330644 (606h8m3.228413679s) [noise=true , address=77.90.31.21:4886 , piece_id=ZQRCPTVC3WFVQPZ36AU6KJW7PD3T557CXYQIICLDGOKK6JCPVZ5Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.357663649 (606h8m3.228746684s) [noise=true , address=145.79.227.49:4367 , piece_id=R2T2ABLXBPDVP3ZVA2TAYMCLQQ7JQUVQYCTHCKGMFBRVWCDAO4PQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.359827449 (606h8m3.230910484s) [noise=true , address=145.79.227.34:4918 , piece_id=HUMOHOT3HOYCBJ23WXQXWU3X6C6JQTRLLHTD4RQLCQTCB3HXI7EQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.360086692 (606h8m3.231169727s) [noise=true , address=145.79.227.7:4320 , piece_id=MAICNTHGLZ6W27HXDOOZXOBRGQGEXNSA4BJT2T5D5PHRVV6ZMX2Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.361622167 (606h8m3.232705212s) [noise=true , address=145.79.227.33:4888 , piece_id=I4426P3VETHPOL5WHWJEGVYRXJULTCNDX3WSUTS3XZVY36Q2N44Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.362965514 (606h8m3.234048559s) [noise=true , address=45.91.116.11:4133 , piece_id=NEW7ZOVP5E7HYGYD4LG5UG2MDLAQW2OBRTNZDFIYO4CQYMVAWRPQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.364308883 (606h8m3.235391928s) [noise=true , address=145.79.227.26:4663 , piece_id=DYEFQZL4X74QR2LAYURT5ZTR2IWIDB55VLBQOGZM36MA54P2SWJA, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.365734507 (606h8m3.236817542s) [noise=true , address=145.79.227.40:4059 , piece_id=7WSUBR3C3PWVBCWD6KRGSD6MPX4IH2YQOAVO4J4SWZORNHGS2IPQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.366982764 (606h8m3.238065799s) [noise=true , address=145.79.227.29:4973 , piece_id=5OROXELR55A47BFPSJJEWVQU4YW6CAOK6LMEHCINM4FN3OVI2Y6Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.368390294 (606h8m3.239473339s) [noise=true , address=77.90.31.19:4852 , piece_id=2NYO63GCBGCYBFYTMUGNF2NG5FSJOD3AOUOD2FZZIWDINK4X6DJQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:04.736295138 (606h8m4.607378183s) [upload=531240]: Next returned nil segment Dec 30 21:37:04.736311850 (606h8m4.607394885s) [upload=531240]: Waiting for error group managing segments... Dec 30 21:37:05.104989591 (606h8m4.976072636s) [noise=true , address=64.49.13.140:4507 , piece_id=E7T6WL45F6GHECCRANX6AIKWFRZ77TURRWORYEYI43MET2WG7SRQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.208809748 (606h8m5.079892793s) [noise=true , address=145.79.227.22:4543 , piece_id=WWCZ6SEEXQ2KUJ32AGTIJNMSJROLYSJ4XQGBQJC4Q5DACZVA65RQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.441125389 (606h8m5.312208444s) [noise=true , address=77.90.31.19:4852 , piece_id=2NYO63GCBGCYBFYTMUGNF2NG5FSJOD3AOUOD2FZZIWDINK4X6DJQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.483630904 (606h8m5.35471394s) [noise=true , address=77.90.31.12:4612 , piece_id=63TS3SUEFUE6PORZ4W76OXRWFL7SBN773UGGY3DU7JPBAW66N7UA, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.494478811 (606h8m5.365561846s) [noise=true , address=77.90.31.11:4559 , piece_id=3DYD4CNPJQN3S4EYZJGREQTTZJDSL2QTC3OT6G2VFUGMLSVCZUFA, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.497452683 (606h8m5.368535729s) [noise=true , address=77.90.31.17:4769 , piece_id=QXDYESBKGXRY7BCFYYF2O7OXS7Y4Z4KUMJB6AXZAWX735SYU56GQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.661529377 (606h8m5.532612412s) [noise=true , address=45.91.116.13:4182 , piece_id=BYAYNP4JAAN3AW2LYSK74WJE4BRWUR5Z6K67MTLK6A73KEJ35WEQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.729773112 (606h8m5.600856157s) [noise=true , address=45.91.116.11:4133 , piece_id=NEW7ZOVP5E7HYGYD4LG5UG2MDLAQW2OBRTNZDFIYO4CQYMVAWRPQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.940743308 (606h8m5.811826353s) [noise=true , address=77.90.31.21:4886 , piece_id=ZQRCPTVC3WFVQPZ36AU6KJW7PD3T557CXYQIICLDGOKK6JCPVZ5Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.084240044 (606h8m5.95532309s) [noise=true , address=145.79.227.30:4784 , piece_id=DSXUDNYCWO5F6IYXMIVJNEASYXIZ7EO7RKAS7QRUYK5OU7GOOB5Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.121095032 (606h8m5.992178067s) [noise=true , address=145.79.227.40:4059 , piece_id=7WSUBR3C3PWVBCWD6KRGSD6MPX4IH2YQOAVO4J4SWZORNHGS2IPQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.169195983 (606h8m6.040279028s) [noise=true , address=145.79.227.49:4367 , piece_id=R2T2ABLXBPDVP3ZVA2TAYMCLQQ7JQUVQYCTHCKGMFBRVWCDAO4PQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.169380634 (606h8m6.040463679s) [noise=true , address=145.79.227.33:4888 , piece_id=I4426P3VETHPOL5WHWJEGVYRXJULTCNDX3WSUTS3XZVY36Q2N44Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.179060175 (606h8m6.050143211s) [noise=true , address=145.79.227.29:4973 , piece_id=5OROXELR55A47BFPSJJEWVQU4YW6CAOK6LMEHCINM4FN3OVI2Y6Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.179080424 (606h8m6.050163459s) [seg_pos={0 0}, upload=531240]: Segment reached optimal threshold of 14 pieces. Dec 30 21:37:06.179262801 (606h8m6.050345837s) [noise=true , address=145.79.227.26:4663 , piece_id=DYEFQZL4X74QR2LAYURT5ZTR2IWIDB55VLBQOGZM36MA54P2SWJA, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SwpAaXTA3BqUvwubzmJkvwokdz6SSpT9t8HPtQcXXJxmcgfm2): protocol: write tcp 188.166.225.170:43184->145.79.227.26:4663: use of closed network connection; write tcp 188.166.225.170:43184->145.79.227.26:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43184->145.79.227.26:4663: use of closed network connection Dec 30 21:37:06.179268703 (606h8m6.050351748s) [noise=true , address=145.79.227.26:4663 , piece_id=DYEFQZL4X74QR2LAYURT5ZTR2IWIDB55VLBQOGZM36MA54P2SWJA, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179363724 (606h8m6.050446769s) [noise=true , address=145.79.227.7:4320 , piece_id=MAICNTHGLZ6W27HXDOOZXOBRGQGEXNSA4BJT2T5D5PHRVV6ZMX2Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126rEgnQw1w7hsFpRqQP9nLaumdBfdtkgYjdw6WE6nXrxk5kj4B): protocol: write tcp 188.166.225.170:45228->145.79.227.7:4320: use of closed network connection; write tcp 188.166.225.170:45228->145.79.227.7:4320: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45228->145.79.227.7:4320: use of closed network connection Dec 30 21:37:06.179369525 (606h8m6.05045257s) [noise=true , address=145.79.227.7:4320 , piece_id=MAICNTHGLZ6W27HXDOOZXOBRGQGEXNSA4BJT2T5D5PHRVV6ZMX2Q, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179532996 (606h8m6.050616061s) [noise=true , address=145.79.227.34:4918 , piece_id=HUMOHOT3HOYCBJ23WXQXWU3X6C6JQTRLLHTD4RQLCQTCB3HXI7EQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF Dec 30 21:37:06.179572371 (606h8m6.050655406s) [noise=true , address=145.79.227.34:4918 , piece_id=HUMOHOT3HOYCBJ23WXQXWU3X6C6JQTRLLHTD4RQLCQTCB3HXI7EQ, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179638587 (606h8m6.050721622s) [noise=true , address=145.79.227.35:4492 , piece_id=HCKFSQMJYEP6B5H4OX5AZ5TWP6Z4H4Z6WA2T7Z2H5KNZ4SDJRMUQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1roa5GGYmqCX2AxSrqFgqyV2CtBEYZavuGTseiwDy4pjkT6bip): protocol: expected piece hash; context canceled; EOF Dec 30 21:37:06.179647855 (606h8m6.05073089s) [noise=true , address=145.79.227.35:4492 , piece_id=HCKFSQMJYEP6B5H4OX5AZ5TWP6Z4H4Z6WA2T7Z2H5KNZ4SDJRMUQ, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179666641 (606h8m6.050749686s) [seg_pos={0 0}, upload=531240]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 21:37:06.179742766 (606h8m6.050825811s) [upload=531240]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 21:37:06.179747545 (606h8m6.05083059s) [upload=531240]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 21:37:06.490986034 (606h8m6.362069079s) [upload=531240]: Done uploading segments... Dec 30 21:37:06.491006261 (606h8m6.362089307s) [upload=531240]: Upload finished. err: <nil> Duration: 15.446548609s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............................................X.............] duration: 3.751598431s trace: https://jaeger.dev.storj.io/trace/22a47ec27f9a101c FAIL (2025-12-30 22:02:45 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.751648546s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [X.........X..X...X.........X.........................X......] duration: 6.073854703s trace: https://jaeger.dev.storj.io/trace/57a7ab98e7eb0d3f FAIL (2025-12-30 22:02:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 21:58:42.298454527 (606h29m42.169537592s) [upload=531550]: Starting upload Dec 30 21:58:42.298523438 (606h29m42.169606473s) [upload=531550]: Done starting upload Dec 30 21:58:42.298568774 (606h29m42.169651809s) [upload=531550]: Uploading segments... Dec 30 21:58:42.298697850 (606h29m42.169780885s) [upload=531550]: Got next segment. Inline: false Dec 30 21:58:42.298728649 (606h29m42.169811694s) [upload=531550]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 21:58:42.298733227 (606h29m42.169816262s) [upload=531550]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 21:58:42.545734213 (606h29m42.416817288s) [seg_pos={0 0}, upload=531550]: Begin upload segment... Dec 30 21:58:42.545786032 (606h29m42.416869078s) [seg_pos={0 0}, upload=531550]: Done begin upload segment. Dec 30 21:58:42.545855104 (606h29m42.416938149s) [noise=true , address=145.79.227.23:4547 , piece_id=LYGUMTAP7OPJWGRVFY65GUZFE4BBTV73JTG657DR5OM26LA7UFZA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.545897384 (606h29m42.416980449s) [noise=true , address=145.79.227.49:4339 , piece_id=WBBPZK2VY2QVRTQNXEKQ7TRYPWBDYS2TGTKEDAAXQ2AT4TUQXTQQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.546757752 (606h29m42.417840817s) [noise=true , address=45.91.116.12:4225 , piece_id=LLYEKE4TJ54KECLAC7KFUV44IVHYCYLF45I43G5V3DPV5QDBUOIQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.546816995 (606h29m42.41790004s) [noise=true , address=145.79.227.40:4050 , piece_id=LA7KLHHWRNHULLPAZWQ6MLS76SLN3BONYXZJXCUHGFC5JZ4SQRIA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.547216927 (606h29m42.418299962s) [noise=true , address=145.79.227.15:4876 , piece_id=L2OIJAB6OOANT2EPFNVNLTMYCH334JGFBFASD44EHOAWTAHKUV5Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.547615996 (606h29m42.418699042s) [noise=true , address=145.79.227.17:4971 , piece_id=DYZUALBL3V75TKGK57LOJABWVCOD5UVS3WH74I55OVUDGRQUA3QQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.547841055 (606h29m42.4189241s) [noise=true , address=77.90.31.21:4877 , piece_id=BD2FEFNAGYWZYDA2IMOVHA7JE66EXSF3A3PL54Q2DUOJFG5ALG4Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548023322 (606h29m42.419106357s) [noise=true , address=45.91.116.14:4190 , piece_id=HHU5YZBTJEEWYVST6XQCKGWKRLRQIIR2UQTQ27RLK53BNAVOSHFQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548380923 (606h29m42.419463968s) [noise=true , address=45.91.116.11:4120 , piece_id=FZ5L6FM7IHX65EVFGVXXHEF43XQ4EBUH4C7VZUSTH2MVWLNR5M2Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548560475 (606h29m42.41964351s) [noise=true , address=145.79.227.39:4013 , piece_id=I4XP47SFJICIYIPEN4E7UC2HXRXH4IXVJYGHXTZBCUESAHNFQXJA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548719268 (606h29m42.419802313s) [noise=true , address=145.79.227.31:4797 , piece_id=S5YBLITONGU6RHPQENP2TCVVWOHEMXPMYXLNYWBZNXWMFI26PVUA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549024107 (606h29m42.420107152s) [noise=true , address=145.79.227.7:4294 , piece_id=C6N5OIRLV3LLSCEIOGQJKWPKENGZYNAPY6V2AYZNKUMACBP5SLWQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549197708 (606h29m42.420280753s) [noise=true , address=145.79.227.16:4910 , piece_id=O4YO367LEVZSRI5E75YEZ7W7YI2KGKKQDXCOHQ3LALE4WUAPQVHQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549506406 (606h29m42.420589441s) [noise=true , address=145.79.227.12:4765 , piece_id=4ZF5YQP4Z5HBTX5YWUDHZV55Q47WASRLMV6ZYJXJRL2QBAVQQM4Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549971591 (606h29m42.421054626s) [noise=true , address=145.79.227.32:4835 , piece_id=CN4NMCFNYKOCG2JCIEXJBL2PLEYWD5THCZB4WQDURIECFVHLLX6Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.551571588 (606h29m42.422654623s) [noise=true , address=145.79.227.24:4581 , piece_id=Y6GBEOR7DQX2H6LBHXYFOXRBX3GP7AKPYVGTQNMUEXKPXLONUUGQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:44.247717212 (606h29m44.118800248s) [upload=531550]: Next returned nil segment Dec 30 21:58:44.247730348 (606h29m44.118813383s) [upload=531550]: Waiting for error group managing segments... Dec 30 21:58:44.532148720 (606h29m44.403231755s) [noise=true , address=145.79.227.31:4797 , piece_id=S5YBLITONGU6RHPQENP2TCVVWOHEMXPMYXLNYWBZNXWMFI26PVUA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.534655293 (606h29m44.405738328s) [noise=true , address=145.79.227.39:4013 , piece_id=I4XP47SFJICIYIPEN4E7UC2HXRXH4IXVJYGHXTZBCUESAHNFQXJA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.544709427 (606h29m44.415792462s) [noise=true , address=145.79.227.12:4765 , piece_id=4ZF5YQP4Z5HBTX5YWUDHZV55Q47WASRLMV6ZYJXJRL2QBAVQQM4Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.545822306 (606h29m44.416905342s) [noise=true , address=145.79.227.7:4294 , piece_id=C6N5OIRLV3LLSCEIOGQJKWPKENGZYNAPY6V2AYZNKUMACBP5SLWQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.549465162 (606h29m44.420548198s) [noise=true , address=145.79.227.24:4581 , piece_id=Y6GBEOR7DQX2H6LBHXYFOXRBX3GP7AKPYVGTQNMUEXKPXLONUUGQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.556935928 (606h29m44.428018964s) [noise=true , address=145.79.227.15:4876 , piece_id=L2OIJAB6OOANT2EPFNVNLTMYCH334JGFBFASD44EHOAWTAHKUV5Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.561283296 (606h29m44.432366331s) [noise=true , address=145.79.227.16:4910 , piece_id=O4YO367LEVZSRI5E75YEZ7W7YI2KGKKQDXCOHQ3LALE4WUAPQVHQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.614194264 (606h29m44.485277309s) [noise=true , address=145.79.227.17:4971 , piece_id=DYZUALBL3V75TKGK57LOJABWVCOD5UVS3WH74I55OVUDGRQUA3QQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.615630689 (606h29m44.486713724s) [noise=true , address=145.79.227.32:4835 , piece_id=CN4NMCFNYKOCG2JCIEXJBL2PLEYWD5THCZB4WQDURIECFVHLLX6Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.639672077 (606h29m44.510755122s) [noise=true , address=145.79.227.23:4547 , piece_id=LYGUMTAP7OPJWGRVFY65GUZFE4BBTV73JTG657DR5OM26LA7UFZA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.699618771 (606h29m44.570701817s) [noise=true , address=77.90.31.21:4877 , piece_id=BD2FEFNAGYWZYDA2IMOVHA7JE66EXSF3A3PL54Q2DUOJFG5ALG4Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.841535416 (606h29m44.712618462s) [noise=true , address=45.91.116.14:4190 , piece_id=HHU5YZBTJEEWYVST6XQCKGWKRLRQIIR2UQTQ27RLK53BNAVOSHFQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.974660841 (606h29m44.845743886s) [noise=true , address=45.91.116.12:4225 , piece_id=LLYEKE4TJ54KECLAC7KFUV44IVHYCYLF45I43G5V3DPV5QDBUOIQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.974681401 (606h29m44.845764436s) [seg_pos={0 0}, upload=531550]: Segment reached optimal threshold of 13 pieces. Dec 30 21:58:44.974855342 (606h29m44.845938377s) [noise=true , address=145.79.227.49:4339 , piece_id=WBBPZK2VY2QVRTQNXEKQ7TRYPWBDYS2TGTKEDAAXQ2AT4TUQXTQQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 30 21:58:44.974861423 (606h29m44.845944458s) [noise=true , address=145.79.227.49:4339 , piece_id=WBBPZK2VY2QVRTQNXEKQ7TRYPWBDYS2TGTKEDAAXQ2AT4TUQXTQQ, seg_pos={0 0}, upload=531550]: Long tail cancellation due to optimal threshold Dec 30 21:58:44.974968166 (606h29m44.846051201s) [noise=true , address=145.79.227.40:4050 , piece_id=LA7KLHHWRNHULLPAZWQ6MLS76SLN3BONYXZJXCUHGFC5JZ4SQRIA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 30 21:58:44.974973627 (606h29m44.846056662s) [noise=true , address=145.79.227.40:4050 , piece_id=LA7KLHHWRNHULLPAZWQ6MLS76SLN3BONYXZJXCUHGFC5JZ4SQRIA, seg_pos={0 0}, upload=531550]: Long tail cancellation due to optimal threshold Dec 30 21:58:44.975076152 (606h29m44.846159187s) [noise=true , address=45.91.116.11:4120 , piece_id=FZ5L6FM7IHX65EVFGVXXHEF43XQ4EBUH4C7VZUSTH2MVWLNR5M2Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 30 21:58:44.975083497 (606h29m44.846166532s) [noise=true , address=45.91.116.11:4120 , piece_id=FZ5L6FM7IHX65EVFGVXXHEF43XQ4EBUH4C7VZUSTH2MVWLNR5M2Q, seg_pos={0 0}, upload=531550]: Long tail cancellation due to optimal threshold Dec 30 21:58:44.975096711 (606h29m44.846179747s) [seg_pos={0 0}, upload=531550]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 21:58:44.975160703 (606h29m44.846243739s) [upload=531550]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 21:58:44.975164691 (606h29m44.846247726s) [upload=531550]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 21:58:45.272782726 (606h29m45.143865772s) [upload=531550]: Done uploading segments... Dec 30 21:58:45.272806352 (606h29m45.143889387s) [upload=531550]: Upload finished. err: <nil> Duration: 6.073888578s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............................................X............] duration: 4.506058338s trace: https://jaeger.dev.storj.io/trace/22a47ec27f9a101c FAIL (2025-12-30 22:02:11 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.506082863s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [...........XX......X.........X..............................] duration: 13.942975564s trace: https://jaeger.dev.storj.io/trace/5cdd5ba7b1cb22c7 FAIL (2025-12-30 22:01:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 21:37:03.355296401 (606h8m3.226379446s) [noise=true , address=145.79.227.22:4543 , piece_id=WWCZ6SEEXQ2KUJ32AGTIJNMSJROLYSJ4XQGBQJC4Q5DACZVA65RQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.355839374 (606h8m3.22692241s) [noise=true , address=145.79.227.35:4492 , piece_id=HCKFSQMJYEP6B5H4OX5AZ5TWP6Z4H4Z6WA2T7Z2H5KNZ4SDJRMUQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.356356930 (606h8m3.227439965s) [noise=true , address=45.91.116.13:4182 , piece_id=BYAYNP4JAAN3AW2LYSK74WJE4BRWUR5Z6K67MTLK6A73KEJ35WEQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.356675316 (606h8m3.227758361s) [noise=true , address=77.90.31.12:4612 , piece_id=63TS3SUEFUE6PORZ4W76OXRWFL7SBN773UGGY3DU7JPBAW66N7UA, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.356992732 (606h8m3.228075767s) [noise=true , address=64.49.13.140:4507 , piece_id=E7T6WL45F6GHECCRANX6AIKWFRZ77TURRWORYEYI43MET2WG7SRQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.357330644 (606h8m3.228413679s) [noise=true , address=77.90.31.21:4886 , piece_id=ZQRCPTVC3WFVQPZ36AU6KJW7PD3T557CXYQIICLDGOKK6JCPVZ5Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.357663649 (606h8m3.228746684s) [noise=true , address=145.79.227.49:4367 , piece_id=R2T2ABLXBPDVP3ZVA2TAYMCLQQ7JQUVQYCTHCKGMFBRVWCDAO4PQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.359827449 (606h8m3.230910484s) [noise=true , address=145.79.227.34:4918 , piece_id=HUMOHOT3HOYCBJ23WXQXWU3X6C6JQTRLLHTD4RQLCQTCB3HXI7EQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.360086692 (606h8m3.231169727s) [noise=true , address=145.79.227.7:4320 , piece_id=MAICNTHGLZ6W27HXDOOZXOBRGQGEXNSA4BJT2T5D5PHRVV6ZMX2Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.361622167 (606h8m3.232705212s) [noise=true , address=145.79.227.33:4888 , piece_id=I4426P3VETHPOL5WHWJEGVYRXJULTCNDX3WSUTS3XZVY36Q2N44Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.362965514 (606h8m3.234048559s) [noise=true , address=45.91.116.11:4133 , piece_id=NEW7ZOVP5E7HYGYD4LG5UG2MDLAQW2OBRTNZDFIYO4CQYMVAWRPQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.364308883 (606h8m3.235391928s) [noise=true , address=145.79.227.26:4663 , piece_id=DYEFQZL4X74QR2LAYURT5ZTR2IWIDB55VLBQOGZM36MA54P2SWJA, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.365734507 (606h8m3.236817542s) [noise=true , address=145.79.227.40:4059 , piece_id=7WSUBR3C3PWVBCWD6KRGSD6MPX4IH2YQOAVO4J4SWZORNHGS2IPQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.366982764 (606h8m3.238065799s) [noise=true , address=145.79.227.29:4973 , piece_id=5OROXELR55A47BFPSJJEWVQU4YW6CAOK6LMEHCINM4FN3OVI2Y6Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.368390294 (606h8m3.239473339s) [noise=true , address=77.90.31.19:4852 , piece_id=2NYO63GCBGCYBFYTMUGNF2NG5FSJOD3AOUOD2FZZIWDINK4X6DJQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:04.736295138 (606h8m4.607378183s) [upload=531240]: Next returned nil segment Dec 30 21:37:04.736311850 (606h8m4.607394885s) [upload=531240]: Waiting for error group managing segments... Dec 30 21:37:05.104989591 (606h8m4.976072636s) [noise=true , address=64.49.13.140:4507 , piece_id=E7T6WL45F6GHECCRANX6AIKWFRZ77TURRWORYEYI43MET2WG7SRQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.208809748 (606h8m5.079892793s) [noise=true , address=145.79.227.22:4543 , piece_id=WWCZ6SEEXQ2KUJ32AGTIJNMSJROLYSJ4XQGBQJC4Q5DACZVA65RQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.441125389 (606h8m5.312208444s) [noise=true , address=77.90.31.19:4852 , piece_id=2NYO63GCBGCYBFYTMUGNF2NG5FSJOD3AOUOD2FZZIWDINK4X6DJQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.483630904 (606h8m5.35471394s) [noise=true , address=77.90.31.12:4612 , piece_id=63TS3SUEFUE6PORZ4W76OXRWFL7SBN773UGGY3DU7JPBAW66N7UA, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.494478811 (606h8m5.365561846s) [noise=true , address=77.90.31.11:4559 , piece_id=3DYD4CNPJQN3S4EYZJGREQTTZJDSL2QTC3OT6G2VFUGMLSVCZUFA, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.497452683 (606h8m5.368535729s) [noise=true , address=77.90.31.17:4769 , piece_id=QXDYESBKGXRY7BCFYYF2O7OXS7Y4Z4KUMJB6AXZAWX735SYU56GQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.661529377 (606h8m5.532612412s) [noise=true , address=45.91.116.13:4182 , piece_id=BYAYNP4JAAN3AW2LYSK74WJE4BRWUR5Z6K67MTLK6A73KEJ35WEQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.729773112 (606h8m5.600856157s) [noise=true , address=45.91.116.11:4133 , piece_id=NEW7ZOVP5E7HYGYD4LG5UG2MDLAQW2OBRTNZDFIYO4CQYMVAWRPQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.940743308 (606h8m5.811826353s) [noise=true , address=77.90.31.21:4886 , piece_id=ZQRCPTVC3WFVQPZ36AU6KJW7PD3T557CXYQIICLDGOKK6JCPVZ5Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.084240044 (606h8m5.95532309s) [noise=true , address=145.79.227.30:4784 , piece_id=DSXUDNYCWO5F6IYXMIVJNEASYXIZ7EO7RKAS7QRUYK5OU7GOOB5Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.121095032 (606h8m5.992178067s) [noise=true , address=145.79.227.40:4059 , piece_id=7WSUBR3C3PWVBCWD6KRGSD6MPX4IH2YQOAVO4J4SWZORNHGS2IPQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.169195983 (606h8m6.040279028s) [noise=true , address=145.79.227.49:4367 , piece_id=R2T2ABLXBPDVP3ZVA2TAYMCLQQ7JQUVQYCTHCKGMFBRVWCDAO4PQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.169380634 (606h8m6.040463679s) [noise=true , address=145.79.227.33:4888 , piece_id=I4426P3VETHPOL5WHWJEGVYRXJULTCNDX3WSUTS3XZVY36Q2N44Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.179060175 (606h8m6.050143211s) [noise=true , address=145.79.227.29:4973 , piece_id=5OROXELR55A47BFPSJJEWVQU4YW6CAOK6LMEHCINM4FN3OVI2Y6Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.179080424 (606h8m6.050163459s) [seg_pos={0 0}, upload=531240]: Segment reached optimal threshold of 14 pieces. Dec 30 21:37:06.179262801 (606h8m6.050345837s) [noise=true , address=145.79.227.26:4663 , piece_id=DYEFQZL4X74QR2LAYURT5ZTR2IWIDB55VLBQOGZM36MA54P2SWJA, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SwpAaXTA3BqUvwubzmJkvwokdz6SSpT9t8HPtQcXXJxmcgfm2): protocol: write tcp 188.166.225.170:43184->145.79.227.26:4663: use of closed network connection; write tcp 188.166.225.170:43184->145.79.227.26:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43184->145.79.227.26:4663: use of closed network connection Dec 30 21:37:06.179268703 (606h8m6.050351748s) [noise=true , address=145.79.227.26:4663 , piece_id=DYEFQZL4X74QR2LAYURT5ZTR2IWIDB55VLBQOGZM36MA54P2SWJA, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179363724 (606h8m6.050446769s) [noise=true , address=145.79.227.7:4320 , piece_id=MAICNTHGLZ6W27HXDOOZXOBRGQGEXNSA4BJT2T5D5PHRVV6ZMX2Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126rEgnQw1w7hsFpRqQP9nLaumdBfdtkgYjdw6WE6nXrxk5kj4B): protocol: write tcp 188.166.225.170:45228->145.79.227.7:4320: use of closed network connection; write tcp 188.166.225.170:45228->145.79.227.7:4320: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45228->145.79.227.7:4320: use of closed network connection Dec 30 21:37:06.179369525 (606h8m6.05045257s) [noise=true , address=145.79.227.7:4320 , piece_id=MAICNTHGLZ6W27HXDOOZXOBRGQGEXNSA4BJT2T5D5PHRVV6ZMX2Q, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179532996 (606h8m6.050616061s) [noise=true , address=145.79.227.34:4918 , piece_id=HUMOHOT3HOYCBJ23WXQXWU3X6C6JQTRLLHTD4RQLCQTCB3HXI7EQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF Dec 30 21:37:06.179572371 (606h8m6.050655406s) [noise=true , address=145.79.227.34:4918 , piece_id=HUMOHOT3HOYCBJ23WXQXWU3X6C6JQTRLLHTD4RQLCQTCB3HXI7EQ, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179638587 (606h8m6.050721622s) [noise=true , address=145.79.227.35:4492 , piece_id=HCKFSQMJYEP6B5H4OX5AZ5TWP6Z4H4Z6WA2T7Z2H5KNZ4SDJRMUQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1roa5GGYmqCX2AxSrqFgqyV2CtBEYZavuGTseiwDy4pjkT6bip): protocol: expected piece hash; context canceled; EOF Dec 30 21:37:06.179647855 (606h8m6.05073089s) [noise=true , address=145.79.227.35:4492 , piece_id=HCKFSQMJYEP6B5H4OX5AZ5TWP6Z4H4Z6WA2T7Z2H5KNZ4SDJRMUQ, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179666641 (606h8m6.050749686s) [seg_pos={0 0}, upload=531240]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 21:37:06.179742766 (606h8m6.050825811s) [upload=531240]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 21:37:06.179747545 (606h8m6.05083059s) [upload=531240]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 21:37:06.490986034 (606h8m6.362069079s) [upload=531240]: Done uploading segments... Dec 30 21:37:06.491006261 (606h8m6.362089307s) [upload=531240]: Upload finished. err: <nil> Duration: 13.943001524s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.X.........X..X...X.........X.........................X.....] duration: 6.151527894s trace: https://jaeger.dev.storj.io/trace/57a7ab98e7eb0d3f FAIL (2025-12-30 22:01:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 21:58:42.298454527 (606h29m42.169537592s) [upload=531550]: Starting upload Dec 30 21:58:42.298523438 (606h29m42.169606473s) [upload=531550]: Done starting upload Dec 30 21:58:42.298568774 (606h29m42.169651809s) [upload=531550]: Uploading segments... Dec 30 21:58:42.298697850 (606h29m42.169780885s) [upload=531550]: Got next segment. Inline: false Dec 30 21:58:42.298728649 (606h29m42.169811694s) [upload=531550]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 21:58:42.298733227 (606h29m42.169816262s) [upload=531550]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 21:58:42.545734213 (606h29m42.416817288s) [seg_pos={0 0}, upload=531550]: Begin upload segment... Dec 30 21:58:42.545786032 (606h29m42.416869078s) [seg_pos={0 0}, upload=531550]: Done begin upload segment. Dec 30 21:58:42.545855104 (606h29m42.416938149s) [noise=true , address=145.79.227.23:4547 , piece_id=LYGUMTAP7OPJWGRVFY65GUZFE4BBTV73JTG657DR5OM26LA7UFZA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.545897384 (606h29m42.416980449s) [noise=true , address=145.79.227.49:4339 , piece_id=WBBPZK2VY2QVRTQNXEKQ7TRYPWBDYS2TGTKEDAAXQ2AT4TUQXTQQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.546757752 (606h29m42.417840817s) [noise=true , address=45.91.116.12:4225 , piece_id=LLYEKE4TJ54KECLAC7KFUV44IVHYCYLF45I43G5V3DPV5QDBUOIQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.546816995 (606h29m42.41790004s) [noise=true , address=145.79.227.40:4050 , piece_id=LA7KLHHWRNHULLPAZWQ6MLS76SLN3BONYXZJXCUHGFC5JZ4SQRIA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.547216927 (606h29m42.418299962s) [noise=true , address=145.79.227.15:4876 , piece_id=L2OIJAB6OOANT2EPFNVNLTMYCH334JGFBFASD44EHOAWTAHKUV5Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.547615996 (606h29m42.418699042s) [noise=true , address=145.79.227.17:4971 , piece_id=DYZUALBL3V75TKGK57LOJABWVCOD5UVS3WH74I55OVUDGRQUA3QQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.547841055 (606h29m42.4189241s) [noise=true , address=77.90.31.21:4877 , piece_id=BD2FEFNAGYWZYDA2IMOVHA7JE66EXSF3A3PL54Q2DUOJFG5ALG4Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548023322 (606h29m42.419106357s) [noise=true , address=45.91.116.14:4190 , piece_id=HHU5YZBTJEEWYVST6XQCKGWKRLRQIIR2UQTQ27RLK53BNAVOSHFQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548380923 (606h29m42.419463968s) [noise=true , address=45.91.116.11:4120 , piece_id=FZ5L6FM7IHX65EVFGVXXHEF43XQ4EBUH4C7VZUSTH2MVWLNR5M2Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548560475 (606h29m42.41964351s) [noise=true , address=145.79.227.39:4013 , piece_id=I4XP47SFJICIYIPEN4E7UC2HXRXH4IXVJYGHXTZBCUESAHNFQXJA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548719268 (606h29m42.419802313s) [noise=true , address=145.79.227.31:4797 , piece_id=S5YBLITONGU6RHPQENP2TCVVWOHEMXPMYXLNYWBZNXWMFI26PVUA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549024107 (606h29m42.420107152s) [noise=true , address=145.79.227.7:4294 , piece_id=C6N5OIRLV3LLSCEIOGQJKWPKENGZYNAPY6V2AYZNKUMACBP5SLWQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549197708 (606h29m42.420280753s) [noise=true , address=145.79.227.16:4910 , piece_id=O4YO367LEVZSRI5E75YEZ7W7YI2KGKKQDXCOHQ3LALE4WUAPQVHQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549506406 (606h29m42.420589441s) [noise=true , address=145.79.227.12:4765 , piece_id=4ZF5YQP4Z5HBTX5YWUDHZV55Q47WASRLMV6ZYJXJRL2QBAVQQM4Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549971591 (606h29m42.421054626s) [noise=true , address=145.79.227.32:4835 , piece_id=CN4NMCFNYKOCG2JCIEXJBL2PLEYWD5THCZB4WQDURIECFVHLLX6Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.551571588 (606h29m42.422654623s) [noise=true , address=145.79.227.24:4581 , piece_id=Y6GBEOR7DQX2H6LBHXYFOXRBX3GP7AKPYVGTQNMUEXKPXLONUUGQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:44.247717212 (606h29m44.118800248s) [upload=531550]: Next returned nil segment Dec 30 21:58:44.247730348 (606h29m44.118813383s) [upload=531550]: Waiting for error group managing segments... Dec 30 21:58:44.532148720 (606h29m44.403231755s) [noise=true , address=145.79.227.31:4797 , piece_id=S5YBLITONGU6RHPQENP2TCVVWOHEMXPMYXLNYWBZNXWMFI26PVUA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.534655293 (606h29m44.405738328s) [noise=true , address=145.79.227.39:4013 , piece_id=I4XP47SFJICIYIPEN4E7UC2HXRXH4IXVJYGHXTZBCUESAHNFQXJA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.544709427 (606h29m44.415792462s) [noise=true , address=145.79.227.12:4765 , piece_id=4ZF5YQP4Z5HBTX5YWUDHZV55Q47WASRLMV6ZYJXJRL2QBAVQQM4Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.545822306 (606h29m44.416905342s) [noise=true , address=145.79.227.7:4294 , piece_id=C6N5OIRLV3LLSCEIOGQJKWPKENGZYNAPY6V2AYZNKUMACBP5SLWQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.549465162 (606h29m44.420548198s) [noise=true , address=145.79.227.24:4581 , piece_id=Y6GBEOR7DQX2H6LBHXYFOXRBX3GP7AKPYVGTQNMUEXKPXLONUUGQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.556935928 (606h29m44.428018964s) [noise=true , address=145.79.227.15:4876 , piece_id=L2OIJAB6OOANT2EPFNVNLTMYCH334JGFBFASD44EHOAWTAHKUV5Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.561283296 (606h29m44.432366331s) [noise=true , address=145.79.227.16:4910 , piece_id=O4YO367LEVZSRI5E75YEZ7W7YI2KGKKQDXCOHQ3LALE4WUAPQVHQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.614194264 (606h29m44.485277309s) [noise=true , address=145.79.227.17:4971 , piece_id=DYZUALBL3V75TKGK57LOJABWVCOD5UVS3WH74I55OVUDGRQUA3QQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.615630689 (606h29m44.486713724s) [noise=true , address=145.79.227.32:4835 , piece_id=CN4NMCFNYKOCG2JCIEXJBL2PLEYWD5THCZB4WQDURIECFVHLLX6Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.639672077 (606h29m44.510755122s) [noise=true , address=145.79.227.23:4547 , piece_id=LYGUMTAP7OPJWGRVFY65GUZFE4BBTV73JTG657DR5OM26LA7UFZA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.699618771 (606h29m44.570701817s) [noise=true , address=77.90.31.21:4877 , piece_id=BD2FEFNAGYWZYDA2IMOVHA7JE66EXSF3A3PL54Q2DUOJFG5ALG4Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.841535416 (606h29m44.712618462s) [noise=true , address=45.91.116.14:4190 , piece_id=HHU5YZBTJEEWYVST6XQCKGWKRLRQIIR2UQTQ27RLK53BNAVOSHFQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.974660841 (606h29m44.845743886s) [noise=true , address=45.91.116.12:4225 , piece_id=LLYEKE4TJ54KECLAC7KFUV44IVHYCYLF45I43G5V3DPV5QDBUOIQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.974681401 (606h29m44.845764436s) [seg_pos={0 0}, upload=531550]: Segment reached optimal threshold of 13 pieces. Dec 30 21:58:44.974855342 (606h29m44.845938377s) [noise=true , address=145.79.227.49:4339 , piece_id=WBBPZK2VY2QVRTQNXEKQ7TRYPWBDYS2TGTKEDAAXQ2AT4TUQXTQQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 30 21:58:44.974861423 (606h29m44.845944458s) [noise=true , address=145.79.227.49:4339 , piece_id=WBBPZK2VY2QVRTQNXEKQ7TRYPWBDYS2TGTKEDAAXQ2AT4TUQXTQQ, seg_pos={0 0}, upload=531550]: Long tail cancellation due to optimal threshold Dec 30 21:58:44.974968166 (606h29m44.846051201s) [noise=true , address=145.79.227.40:4050 , piece_id=LA7KLHHWRNHULLPAZWQ6MLS76SLN3BONYXZJXCUHGFC5JZ4SQRIA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 30 21:58:44.974973627 (606h29m44.846056662s) [noise=true , address=145.79.227.40:4050 , piece_id=LA7KLHHWRNHULLPAZWQ6MLS76SLN3BONYXZJXCUHGFC5JZ4SQRIA, seg_pos={0 0}, upload=531550]: Long tail cancellation due to optimal threshold Dec 30 21:58:44.975076152 (606h29m44.846159187s) [noise=true , address=45.91.116.11:4120 , piece_id=FZ5L6FM7IHX65EVFGVXXHEF43XQ4EBUH4C7VZUSTH2MVWLNR5M2Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 30 21:58:44.975083497 (606h29m44.846166532s) [noise=true , address=45.91.116.11:4120 , piece_id=FZ5L6FM7IHX65EVFGVXXHEF43XQ4EBUH4C7VZUSTH2MVWLNR5M2Q, seg_pos={0 0}, upload=531550]: Long tail cancellation due to optimal threshold Dec 30 21:58:44.975096711 (606h29m44.846179747s) [seg_pos={0 0}, upload=531550]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 21:58:44.975160703 (606h29m44.846243739s) [upload=531550]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 21:58:44.975164691 (606h29m44.846247726s) [upload=531550]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 21:58:45.272782726 (606h29m45.143865772s) [upload=531550]: Done uploading segments... Dec 30 21:58:45.272806352 (606h29m45.143889387s) [upload=531550]: Upload finished. err: <nil> Duration: 6.151552069s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [............XX......X.........X.............................] duration: 15.402477594s trace: https://jaeger.dev.storj.io/trace/5cdd5ba7b1cb22c7 FAIL (2025-12-30 22:01:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 21:37:03.355296401 (606h8m3.226379446s) [noise=true , address=145.79.227.22:4543 , piece_id=WWCZ6SEEXQ2KUJ32AGTIJNMSJROLYSJ4XQGBQJC4Q5DACZVA65RQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.355839374 (606h8m3.22692241s) [noise=true , address=145.79.227.35:4492 , piece_id=HCKFSQMJYEP6B5H4OX5AZ5TWP6Z4H4Z6WA2T7Z2H5KNZ4SDJRMUQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.356356930 (606h8m3.227439965s) [noise=true , address=45.91.116.13:4182 , piece_id=BYAYNP4JAAN3AW2LYSK74WJE4BRWUR5Z6K67MTLK6A73KEJ35WEQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.356675316 (606h8m3.227758361s) [noise=true , address=77.90.31.12:4612 , piece_id=63TS3SUEFUE6PORZ4W76OXRWFL7SBN773UGGY3DU7JPBAW66N7UA, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.356992732 (606h8m3.228075767s) [noise=true , address=64.49.13.140:4507 , piece_id=E7T6WL45F6GHECCRANX6AIKWFRZ77TURRWORYEYI43MET2WG7SRQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.357330644 (606h8m3.228413679s) [noise=true , address=77.90.31.21:4886 , piece_id=ZQRCPTVC3WFVQPZ36AU6KJW7PD3T557CXYQIICLDGOKK6JCPVZ5Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.357663649 (606h8m3.228746684s) [noise=true , address=145.79.227.49:4367 , piece_id=R2T2ABLXBPDVP3ZVA2TAYMCLQQ7JQUVQYCTHCKGMFBRVWCDAO4PQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.359827449 (606h8m3.230910484s) [noise=true , address=145.79.227.34:4918 , piece_id=HUMOHOT3HOYCBJ23WXQXWU3X6C6JQTRLLHTD4RQLCQTCB3HXI7EQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.360086692 (606h8m3.231169727s) [noise=true , address=145.79.227.7:4320 , piece_id=MAICNTHGLZ6W27HXDOOZXOBRGQGEXNSA4BJT2T5D5PHRVV6ZMX2Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.361622167 (606h8m3.232705212s) [noise=true , address=145.79.227.33:4888 , piece_id=I4426P3VETHPOL5WHWJEGVYRXJULTCNDX3WSUTS3XZVY36Q2N44Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.362965514 (606h8m3.234048559s) [noise=true , address=45.91.116.11:4133 , piece_id=NEW7ZOVP5E7HYGYD4LG5UG2MDLAQW2OBRTNZDFIYO4CQYMVAWRPQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.364308883 (606h8m3.235391928s) [noise=true , address=145.79.227.26:4663 , piece_id=DYEFQZL4X74QR2LAYURT5ZTR2IWIDB55VLBQOGZM36MA54P2SWJA, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.365734507 (606h8m3.236817542s) [noise=true , address=145.79.227.40:4059 , piece_id=7WSUBR3C3PWVBCWD6KRGSD6MPX4IH2YQOAVO4J4SWZORNHGS2IPQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.366982764 (606h8m3.238065799s) [noise=true , address=145.79.227.29:4973 , piece_id=5OROXELR55A47BFPSJJEWVQU4YW6CAOK6LMEHCINM4FN3OVI2Y6Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.368390294 (606h8m3.239473339s) [noise=true , address=77.90.31.19:4852 , piece_id=2NYO63GCBGCYBFYTMUGNF2NG5FSJOD3AOUOD2FZZIWDINK4X6DJQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:04.736295138 (606h8m4.607378183s) [upload=531240]: Next returned nil segment Dec 30 21:37:04.736311850 (606h8m4.607394885s) [upload=531240]: Waiting for error group managing segments... Dec 30 21:37:05.104989591 (606h8m4.976072636s) [noise=true , address=64.49.13.140:4507 , piece_id=E7T6WL45F6GHECCRANX6AIKWFRZ77TURRWORYEYI43MET2WG7SRQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.208809748 (606h8m5.079892793s) [noise=true , address=145.79.227.22:4543 , piece_id=WWCZ6SEEXQ2KUJ32AGTIJNMSJROLYSJ4XQGBQJC4Q5DACZVA65RQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.441125389 (606h8m5.312208444s) [noise=true , address=77.90.31.19:4852 , piece_id=2NYO63GCBGCYBFYTMUGNF2NG5FSJOD3AOUOD2FZZIWDINK4X6DJQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.483630904 (606h8m5.35471394s) [noise=true , address=77.90.31.12:4612 , piece_id=63TS3SUEFUE6PORZ4W76OXRWFL7SBN773UGGY3DU7JPBAW66N7UA, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.494478811 (606h8m5.365561846s) [noise=true , address=77.90.31.11:4559 , piece_id=3DYD4CNPJQN3S4EYZJGREQTTZJDSL2QTC3OT6G2VFUGMLSVCZUFA, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.497452683 (606h8m5.368535729s) [noise=true , address=77.90.31.17:4769 , piece_id=QXDYESBKGXRY7BCFYYF2O7OXS7Y4Z4KUMJB6AXZAWX735SYU56GQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.661529377 (606h8m5.532612412s) [noise=true , address=45.91.116.13:4182 , piece_id=BYAYNP4JAAN3AW2LYSK74WJE4BRWUR5Z6K67MTLK6A73KEJ35WEQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.729773112 (606h8m5.600856157s) [noise=true , address=45.91.116.11:4133 , piece_id=NEW7ZOVP5E7HYGYD4LG5UG2MDLAQW2OBRTNZDFIYO4CQYMVAWRPQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.940743308 (606h8m5.811826353s) [noise=true , address=77.90.31.21:4886 , piece_id=ZQRCPTVC3WFVQPZ36AU6KJW7PD3T557CXYQIICLDGOKK6JCPVZ5Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.084240044 (606h8m5.95532309s) [noise=true , address=145.79.227.30:4784 , piece_id=DSXUDNYCWO5F6IYXMIVJNEASYXIZ7EO7RKAS7QRUYK5OU7GOOB5Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.121095032 (606h8m5.992178067s) [noise=true , address=145.79.227.40:4059 , piece_id=7WSUBR3C3PWVBCWD6KRGSD6MPX4IH2YQOAVO4J4SWZORNHGS2IPQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.169195983 (606h8m6.040279028s) [noise=true , address=145.79.227.49:4367 , piece_id=R2T2ABLXBPDVP3ZVA2TAYMCLQQ7JQUVQYCTHCKGMFBRVWCDAO4PQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.169380634 (606h8m6.040463679s) [noise=true , address=145.79.227.33:4888 , piece_id=I4426P3VETHPOL5WHWJEGVYRXJULTCNDX3WSUTS3XZVY36Q2N44Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.179060175 (606h8m6.050143211s) [noise=true , address=145.79.227.29:4973 , piece_id=5OROXELR55A47BFPSJJEWVQU4YW6CAOK6LMEHCINM4FN3OVI2Y6Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.179080424 (606h8m6.050163459s) [seg_pos={0 0}, upload=531240]: Segment reached optimal threshold of 14 pieces. Dec 30 21:37:06.179262801 (606h8m6.050345837s) [noise=true , address=145.79.227.26:4663 , piece_id=DYEFQZL4X74QR2LAYURT5ZTR2IWIDB55VLBQOGZM36MA54P2SWJA, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SwpAaXTA3BqUvwubzmJkvwokdz6SSpT9t8HPtQcXXJxmcgfm2): protocol: write tcp 188.166.225.170:43184->145.79.227.26:4663: use of closed network connection; write tcp 188.166.225.170:43184->145.79.227.26:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43184->145.79.227.26:4663: use of closed network connection Dec 30 21:37:06.179268703 (606h8m6.050351748s) [noise=true , address=145.79.227.26:4663 , piece_id=DYEFQZL4X74QR2LAYURT5ZTR2IWIDB55VLBQOGZM36MA54P2SWJA, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179363724 (606h8m6.050446769s) [noise=true , address=145.79.227.7:4320 , piece_id=MAICNTHGLZ6W27HXDOOZXOBRGQGEXNSA4BJT2T5D5PHRVV6ZMX2Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126rEgnQw1w7hsFpRqQP9nLaumdBfdtkgYjdw6WE6nXrxk5kj4B): protocol: write tcp 188.166.225.170:45228->145.79.227.7:4320: use of closed network connection; write tcp 188.166.225.170:45228->145.79.227.7:4320: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45228->145.79.227.7:4320: use of closed network connection Dec 30 21:37:06.179369525 (606h8m6.05045257s) [noise=true , address=145.79.227.7:4320 , piece_id=MAICNTHGLZ6W27HXDOOZXOBRGQGEXNSA4BJT2T5D5PHRVV6ZMX2Q, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179532996 (606h8m6.050616061s) [noise=true , address=145.79.227.34:4918 , piece_id=HUMOHOT3HOYCBJ23WXQXWU3X6C6JQTRLLHTD4RQLCQTCB3HXI7EQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF Dec 30 21:37:06.179572371 (606h8m6.050655406s) [noise=true , address=145.79.227.34:4918 , piece_id=HUMOHOT3HOYCBJ23WXQXWU3X6C6JQTRLLHTD4RQLCQTCB3HXI7EQ, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179638587 (606h8m6.050721622s) [noise=true , address=145.79.227.35:4492 , piece_id=HCKFSQMJYEP6B5H4OX5AZ5TWP6Z4H4Z6WA2T7Z2H5KNZ4SDJRMUQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1roa5GGYmqCX2AxSrqFgqyV2CtBEYZavuGTseiwDy4pjkT6bip): protocol: expected piece hash; context canceled; EOF Dec 30 21:37:06.179647855 (606h8m6.05073089s) [noise=true , address=145.79.227.35:4492 , piece_id=HCKFSQMJYEP6B5H4OX5AZ5TWP6Z4H4Z6WA2T7Z2H5KNZ4SDJRMUQ, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179666641 (606h8m6.050749686s) [seg_pos={0 0}, upload=531240]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 21:37:06.179742766 (606h8m6.050825811s) [upload=531240]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 21:37:06.179747545 (606h8m6.05083059s) [upload=531240]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 21:37:06.490986034 (606h8m6.362069079s) [upload=531240]: Done uploading segments... Dec 30 21:37:06.491006261 (606h8m6.362089307s) [upload=531240]: Upload finished. err: <nil> Duration: 15.402512031s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [X.X.........X..X...X.........X.........................X....] duration: 6.065314152s trace: https://jaeger.dev.storj.io/trace/57a7ab98e7eb0d3f FAIL (2025-12-30 22:01:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 21:58:42.298454527 (606h29m42.169537592s) [upload=531550]: Starting upload Dec 30 21:58:42.298523438 (606h29m42.169606473s) [upload=531550]: Done starting upload Dec 30 21:58:42.298568774 (606h29m42.169651809s) [upload=531550]: Uploading segments... Dec 30 21:58:42.298697850 (606h29m42.169780885s) [upload=531550]: Got next segment. Inline: false Dec 30 21:58:42.298728649 (606h29m42.169811694s) [upload=531550]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 21:58:42.298733227 (606h29m42.169816262s) [upload=531550]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 21:58:42.545734213 (606h29m42.416817288s) [seg_pos={0 0}, upload=531550]: Begin upload segment... Dec 30 21:58:42.545786032 (606h29m42.416869078s) [seg_pos={0 0}, upload=531550]: Done begin upload segment. Dec 30 21:58:42.545855104 (606h29m42.416938149s) [noise=true , address=145.79.227.23:4547 , piece_id=LYGUMTAP7OPJWGRVFY65GUZFE4BBTV73JTG657DR5OM26LA7UFZA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.545897384 (606h29m42.416980449s) [noise=true , address=145.79.227.49:4339 , piece_id=WBBPZK2VY2QVRTQNXEKQ7TRYPWBDYS2TGTKEDAAXQ2AT4TUQXTQQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.546757752 (606h29m42.417840817s) [noise=true , address=45.91.116.12:4225 , piece_id=LLYEKE4TJ54KECLAC7KFUV44IVHYCYLF45I43G5V3DPV5QDBUOIQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.546816995 (606h29m42.41790004s) [noise=true , address=145.79.227.40:4050 , piece_id=LA7KLHHWRNHULLPAZWQ6MLS76SLN3BONYXZJXCUHGFC5JZ4SQRIA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.547216927 (606h29m42.418299962s) [noise=true , address=145.79.227.15:4876 , piece_id=L2OIJAB6OOANT2EPFNVNLTMYCH334JGFBFASD44EHOAWTAHKUV5Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.547615996 (606h29m42.418699042s) [noise=true , address=145.79.227.17:4971 , piece_id=DYZUALBL3V75TKGK57LOJABWVCOD5UVS3WH74I55OVUDGRQUA3QQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.547841055 (606h29m42.4189241s) [noise=true , address=77.90.31.21:4877 , piece_id=BD2FEFNAGYWZYDA2IMOVHA7JE66EXSF3A3PL54Q2DUOJFG5ALG4Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548023322 (606h29m42.419106357s) [noise=true , address=45.91.116.14:4190 , piece_id=HHU5YZBTJEEWYVST6XQCKGWKRLRQIIR2UQTQ27RLK53BNAVOSHFQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548380923 (606h29m42.419463968s) [noise=true , address=45.91.116.11:4120 , piece_id=FZ5L6FM7IHX65EVFGVXXHEF43XQ4EBUH4C7VZUSTH2MVWLNR5M2Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548560475 (606h29m42.41964351s) [noise=true , address=145.79.227.39:4013 , piece_id=I4XP47SFJICIYIPEN4E7UC2HXRXH4IXVJYGHXTZBCUESAHNFQXJA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548719268 (606h29m42.419802313s) [noise=true , address=145.79.227.31:4797 , piece_id=S5YBLITONGU6RHPQENP2TCVVWOHEMXPMYXLNYWBZNXWMFI26PVUA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549024107 (606h29m42.420107152s) [noise=true , address=145.79.227.7:4294 , piece_id=C6N5OIRLV3LLSCEIOGQJKWPKENGZYNAPY6V2AYZNKUMACBP5SLWQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549197708 (606h29m42.420280753s) [noise=true , address=145.79.227.16:4910 , piece_id=O4YO367LEVZSRI5E75YEZ7W7YI2KGKKQDXCOHQ3LALE4WUAPQVHQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549506406 (606h29m42.420589441s) [noise=true , address=145.79.227.12:4765 , piece_id=4ZF5YQP4Z5HBTX5YWUDHZV55Q47WASRLMV6ZYJXJRL2QBAVQQM4Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549971591 (606h29m42.421054626s) [noise=true , address=145.79.227.32:4835 , piece_id=CN4NMCFNYKOCG2JCIEXJBL2PLEYWD5THCZB4WQDURIECFVHLLX6Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.551571588 (606h29m42.422654623s) [noise=true , address=145.79.227.24:4581 , piece_id=Y6GBEOR7DQX2H6LBHXYFOXRBX3GP7AKPYVGTQNMUEXKPXLONUUGQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:44.247717212 (606h29m44.118800248s) [upload=531550]: Next returned nil segment Dec 30 21:58:44.247730348 (606h29m44.118813383s) [upload=531550]: Waiting for error group managing segments... Dec 30 21:58:44.532148720 (606h29m44.403231755s) [noise=true , address=145.79.227.31:4797 , piece_id=S5YBLITONGU6RHPQENP2TCVVWOHEMXPMYXLNYWBZNXWMFI26PVUA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.534655293 (606h29m44.405738328s) [noise=true , address=145.79.227.39:4013 , piece_id=I4XP47SFJICIYIPEN4E7UC2HXRXH4IXVJYGHXTZBCUESAHNFQXJA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.544709427 (606h29m44.415792462s) [noise=true , address=145.79.227.12:4765 , piece_id=4ZF5YQP4Z5HBTX5YWUDHZV55Q47WASRLMV6ZYJXJRL2QBAVQQM4Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.545822306 (606h29m44.416905342s) [noise=true , address=145.79.227.7:4294 , piece_id=C6N5OIRLV3LLSCEIOGQJKWPKENGZYNAPY6V2AYZNKUMACBP5SLWQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.549465162 (606h29m44.420548198s) [noise=true , address=145.79.227.24:4581 , piece_id=Y6GBEOR7DQX2H6LBHXYFOXRBX3GP7AKPYVGTQNMUEXKPXLONUUGQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.556935928 (606h29m44.428018964s) [noise=true , address=145.79.227.15:4876 , piece_id=L2OIJAB6OOANT2EPFNVNLTMYCH334JGFBFASD44EHOAWTAHKUV5Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.561283296 (606h29m44.432366331s) [noise=true , address=145.79.227.16:4910 , piece_id=O4YO367LEVZSRI5E75YEZ7W7YI2KGKKQDXCOHQ3LALE4WUAPQVHQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.614194264 (606h29m44.485277309s) [noise=true , address=145.79.227.17:4971 , piece_id=DYZUALBL3V75TKGK57LOJABWVCOD5UVS3WH74I55OVUDGRQUA3QQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.615630689 (606h29m44.486713724s) [noise=true , address=145.79.227.32:4835 , piece_id=CN4NMCFNYKOCG2JCIEXJBL2PLEYWD5THCZB4WQDURIECFVHLLX6Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.639672077 (606h29m44.510755122s) [noise=true , address=145.79.227.23:4547 , piece_id=LYGUMTAP7OPJWGRVFY65GUZFE4BBTV73JTG657DR5OM26LA7UFZA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.699618771 (606h29m44.570701817s) [noise=true , address=77.90.31.21:4877 , piece_id=BD2FEFNAGYWZYDA2IMOVHA7JE66EXSF3A3PL54Q2DUOJFG5ALG4Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.841535416 (606h29m44.712618462s) [noise=true , address=45.91.116.14:4190 , piece_id=HHU5YZBTJEEWYVST6XQCKGWKRLRQIIR2UQTQ27RLK53BNAVOSHFQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.974660841 (606h29m44.845743886s) [noise=true , address=45.91.116.12:4225 , piece_id=LLYEKE4TJ54KECLAC7KFUV44IVHYCYLF45I43G5V3DPV5QDBUOIQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.974681401 (606h29m44.845764436s) [seg_pos={0 0}, upload=531550]: Segment reached optimal threshold of 13 pieces. Dec 30 21:58:44.974855342 (606h29m44.845938377s) [noise=true , address=145.79.227.49:4339 , piece_id=WBBPZK2VY2QVRTQNXEKQ7TRYPWBDYS2TGTKEDAAXQ2AT4TUQXTQQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 30 21:58:44.974861423 (606h29m44.845944458s) [noise=true , address=145.79.227.49:4339 , piece_id=WBBPZK2VY2QVRTQNXEKQ7TRYPWBDYS2TGTKEDAAXQ2AT4TUQXTQQ, seg_pos={0 0}, upload=531550]: Long tail cancellation due to optimal threshold Dec 30 21:58:44.974968166 (606h29m44.846051201s) [noise=true , address=145.79.227.40:4050 , piece_id=LA7KLHHWRNHULLPAZWQ6MLS76SLN3BONYXZJXCUHGFC5JZ4SQRIA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 30 21:58:44.974973627 (606h29m44.846056662s) [noise=true , address=145.79.227.40:4050 , piece_id=LA7KLHHWRNHULLPAZWQ6MLS76SLN3BONYXZJXCUHGFC5JZ4SQRIA, seg_pos={0 0}, upload=531550]: Long tail cancellation due to optimal threshold Dec 30 21:58:44.975076152 (606h29m44.846159187s) [noise=true , address=45.91.116.11:4120 , piece_id=FZ5L6FM7IHX65EVFGVXXHEF43XQ4EBUH4C7VZUSTH2MVWLNR5M2Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 30 21:58:44.975083497 (606h29m44.846166532s) [noise=true , address=45.91.116.11:4120 , piece_id=FZ5L6FM7IHX65EVFGVXXHEF43XQ4EBUH4C7VZUSTH2MVWLNR5M2Q, seg_pos={0 0}, upload=531550]: Long tail cancellation due to optimal threshold Dec 30 21:58:44.975096711 (606h29m44.846179747s) [seg_pos={0 0}, upload=531550]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 21:58:44.975160703 (606h29m44.846243739s) [upload=531550]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 21:58:44.975164691 (606h29m44.846247726s) [upload=531550]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 21:58:45.272782726 (606h29m45.143865772s) [upload=531550]: Done uploading segments... Dec 30 21:58:45.272806352 (606h29m45.143889387s) [upload=531550]: Upload finished. err: <nil> Duration: 6.065336926s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................................................X...........] duration: 3.567166252s trace: https://jaeger.dev.storj.io/trace/22a47ec27f9a101c FAIL (2025-12-30 22:01:00 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.567193884s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.X.X.........X..X...X.........X.........................X...] duration: 5.835457523s trace: https://jaeger.dev.storj.io/trace/57a7ab98e7eb0d3f FAIL (2025-12-30 22:00:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 21:58:42.298454527 (606h29m42.169537592s) [upload=531550]: Starting upload Dec 30 21:58:42.298523438 (606h29m42.169606473s) [upload=531550]: Done starting upload Dec 30 21:58:42.298568774 (606h29m42.169651809s) [upload=531550]: Uploading segments... Dec 30 21:58:42.298697850 (606h29m42.169780885s) [upload=531550]: Got next segment. Inline: false Dec 30 21:58:42.298728649 (606h29m42.169811694s) [upload=531550]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 21:58:42.298733227 (606h29m42.169816262s) [upload=531550]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 21:58:42.545734213 (606h29m42.416817288s) [seg_pos={0 0}, upload=531550]: Begin upload segment... Dec 30 21:58:42.545786032 (606h29m42.416869078s) [seg_pos={0 0}, upload=531550]: Done begin upload segment. Dec 30 21:58:42.545855104 (606h29m42.416938149s) [noise=true , address=145.79.227.23:4547 , piece_id=LYGUMTAP7OPJWGRVFY65GUZFE4BBTV73JTG657DR5OM26LA7UFZA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.545897384 (606h29m42.416980449s) [noise=true , address=145.79.227.49:4339 , piece_id=WBBPZK2VY2QVRTQNXEKQ7TRYPWBDYS2TGTKEDAAXQ2AT4TUQXTQQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.546757752 (606h29m42.417840817s) [noise=true , address=45.91.116.12:4225 , piece_id=LLYEKE4TJ54KECLAC7KFUV44IVHYCYLF45I43G5V3DPV5QDBUOIQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.546816995 (606h29m42.41790004s) [noise=true , address=145.79.227.40:4050 , piece_id=LA7KLHHWRNHULLPAZWQ6MLS76SLN3BONYXZJXCUHGFC5JZ4SQRIA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.547216927 (606h29m42.418299962s) [noise=true , address=145.79.227.15:4876 , piece_id=L2OIJAB6OOANT2EPFNVNLTMYCH334JGFBFASD44EHOAWTAHKUV5Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.547615996 (606h29m42.418699042s) [noise=true , address=145.79.227.17:4971 , piece_id=DYZUALBL3V75TKGK57LOJABWVCOD5UVS3WH74I55OVUDGRQUA3QQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.547841055 (606h29m42.4189241s) [noise=true , address=77.90.31.21:4877 , piece_id=BD2FEFNAGYWZYDA2IMOVHA7JE66EXSF3A3PL54Q2DUOJFG5ALG4Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548023322 (606h29m42.419106357s) [noise=true , address=45.91.116.14:4190 , piece_id=HHU5YZBTJEEWYVST6XQCKGWKRLRQIIR2UQTQ27RLK53BNAVOSHFQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548380923 (606h29m42.419463968s) [noise=true , address=45.91.116.11:4120 , piece_id=FZ5L6FM7IHX65EVFGVXXHEF43XQ4EBUH4C7VZUSTH2MVWLNR5M2Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548560475 (606h29m42.41964351s) [noise=true , address=145.79.227.39:4013 , piece_id=I4XP47SFJICIYIPEN4E7UC2HXRXH4IXVJYGHXTZBCUESAHNFQXJA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548719268 (606h29m42.419802313s) [noise=true , address=145.79.227.31:4797 , piece_id=S5YBLITONGU6RHPQENP2TCVVWOHEMXPMYXLNYWBZNXWMFI26PVUA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549024107 (606h29m42.420107152s) [noise=true , address=145.79.227.7:4294 , piece_id=C6N5OIRLV3LLSCEIOGQJKWPKENGZYNAPY6V2AYZNKUMACBP5SLWQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549197708 (606h29m42.420280753s) [noise=true , address=145.79.227.16:4910 , piece_id=O4YO367LEVZSRI5E75YEZ7W7YI2KGKKQDXCOHQ3LALE4WUAPQVHQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549506406 (606h29m42.420589441s) [noise=true , address=145.79.227.12:4765 , piece_id=4ZF5YQP4Z5HBTX5YWUDHZV55Q47WASRLMV6ZYJXJRL2QBAVQQM4Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549971591 (606h29m42.421054626s) [noise=true , address=145.79.227.32:4835 , piece_id=CN4NMCFNYKOCG2JCIEXJBL2PLEYWD5THCZB4WQDURIECFVHLLX6Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.551571588 (606h29m42.422654623s) [noise=true , address=145.79.227.24:4581 , piece_id=Y6GBEOR7DQX2H6LBHXYFOXRBX3GP7AKPYVGTQNMUEXKPXLONUUGQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:44.247717212 (606h29m44.118800248s) [upload=531550]: Next returned nil segment Dec 30 21:58:44.247730348 (606h29m44.118813383s) [upload=531550]: Waiting for error group managing segments... Dec 30 21:58:44.532148720 (606h29m44.403231755s) [noise=true , address=145.79.227.31:4797 , piece_id=S5YBLITONGU6RHPQENP2TCVVWOHEMXPMYXLNYWBZNXWMFI26PVUA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.534655293 (606h29m44.405738328s) [noise=true , address=145.79.227.39:4013 , piece_id=I4XP47SFJICIYIPEN4E7UC2HXRXH4IXVJYGHXTZBCUESAHNFQXJA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.544709427 (606h29m44.415792462s) [noise=true , address=145.79.227.12:4765 , piece_id=4ZF5YQP4Z5HBTX5YWUDHZV55Q47WASRLMV6ZYJXJRL2QBAVQQM4Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.545822306 (606h29m44.416905342s) [noise=true , address=145.79.227.7:4294 , piece_id=C6N5OIRLV3LLSCEIOGQJKWPKENGZYNAPY6V2AYZNKUMACBP5SLWQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.549465162 (606h29m44.420548198s) [noise=true , address=145.79.227.24:4581 , piece_id=Y6GBEOR7DQX2H6LBHXYFOXRBX3GP7AKPYVGTQNMUEXKPXLONUUGQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.556935928 (606h29m44.428018964s) [noise=true , address=145.79.227.15:4876 , piece_id=L2OIJAB6OOANT2EPFNVNLTMYCH334JGFBFASD44EHOAWTAHKUV5Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.561283296 (606h29m44.432366331s) [noise=true , address=145.79.227.16:4910 , piece_id=O4YO367LEVZSRI5E75YEZ7W7YI2KGKKQDXCOHQ3LALE4WUAPQVHQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.614194264 (606h29m44.485277309s) [noise=true , address=145.79.227.17:4971 , piece_id=DYZUALBL3V75TKGK57LOJABWVCOD5UVS3WH74I55OVUDGRQUA3QQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.615630689 (606h29m44.486713724s) [noise=true , address=145.79.227.32:4835 , piece_id=CN4NMCFNYKOCG2JCIEXJBL2PLEYWD5THCZB4WQDURIECFVHLLX6Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.639672077 (606h29m44.510755122s) [noise=true , address=145.79.227.23:4547 , piece_id=LYGUMTAP7OPJWGRVFY65GUZFE4BBTV73JTG657DR5OM26LA7UFZA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.699618771 (606h29m44.570701817s) [noise=true , address=77.90.31.21:4877 , piece_id=BD2FEFNAGYWZYDA2IMOVHA7JE66EXSF3A3PL54Q2DUOJFG5ALG4Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.841535416 (606h29m44.712618462s) [noise=true , address=45.91.116.14:4190 , piece_id=HHU5YZBTJEEWYVST6XQCKGWKRLRQIIR2UQTQ27RLK53BNAVOSHFQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.974660841 (606h29m44.845743886s) [noise=true , address=45.91.116.12:4225 , piece_id=LLYEKE4TJ54KECLAC7KFUV44IVHYCYLF45I43G5V3DPV5QDBUOIQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.974681401 (606h29m44.845764436s) [seg_pos={0 0}, upload=531550]: Segment reached optimal threshold of 13 pieces. Dec 30 21:58:44.974855342 (606h29m44.845938377s) [noise=true , address=145.79.227.49:4339 , piece_id=WBBPZK2VY2QVRTQNXEKQ7TRYPWBDYS2TGTKEDAAXQ2AT4TUQXTQQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 30 21:58:44.974861423 (606h29m44.845944458s) [noise=true , address=145.79.227.49:4339 , piece_id=WBBPZK2VY2QVRTQNXEKQ7TRYPWBDYS2TGTKEDAAXQ2AT4TUQXTQQ, seg_pos={0 0}, upload=531550]: Long tail cancellation due to optimal threshold Dec 30 21:58:44.974968166 (606h29m44.846051201s) [noise=true , address=145.79.227.40:4050 , piece_id=LA7KLHHWRNHULLPAZWQ6MLS76SLN3BONYXZJXCUHGFC5JZ4SQRIA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 30 21:58:44.974973627 (606h29m44.846056662s) [noise=true , address=145.79.227.40:4050 , piece_id=LA7KLHHWRNHULLPAZWQ6MLS76SLN3BONYXZJXCUHGFC5JZ4SQRIA, seg_pos={0 0}, upload=531550]: Long tail cancellation due to optimal threshold Dec 30 21:58:44.975076152 (606h29m44.846159187s) [noise=true , address=45.91.116.11:4120 , piece_id=FZ5L6FM7IHX65EVFGVXXHEF43XQ4EBUH4C7VZUSTH2MVWLNR5M2Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 30 21:58:44.975083497 (606h29m44.846166532s) [noise=true , address=45.91.116.11:4120 , piece_id=FZ5L6FM7IHX65EVFGVXXHEF43XQ4EBUH4C7VZUSTH2MVWLNR5M2Q, seg_pos={0 0}, upload=531550]: Long tail cancellation due to optimal threshold Dec 30 21:58:44.975096711 (606h29m44.846179747s) [seg_pos={0 0}, upload=531550]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 21:58:44.975160703 (606h29m44.846243739s) [upload=531550]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 21:58:44.975164691 (606h29m44.846247726s) [upload=531550]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 21:58:45.272782726 (606h29m45.143865772s) [upload=531550]: Done uploading segments... Dec 30 21:58:45.272806352 (606h29m45.143889387s) [upload=531550]: Upload finished. err: <nil> Duration: 5.835491316s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.............XX......X.........X............................] duration: 15.540037452s trace: https://jaeger.dev.storj.io/trace/5cdd5ba7b1cb22c7 FAIL (2025-12-30 22:00:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 21:37:03.355296401 (606h8m3.226379446s) [noise=true , address=145.79.227.22:4543 , piece_id=WWCZ6SEEXQ2KUJ32AGTIJNMSJROLYSJ4XQGBQJC4Q5DACZVA65RQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.355839374 (606h8m3.22692241s) [noise=true , address=145.79.227.35:4492 , piece_id=HCKFSQMJYEP6B5H4OX5AZ5TWP6Z4H4Z6WA2T7Z2H5KNZ4SDJRMUQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.356356930 (606h8m3.227439965s) [noise=true , address=45.91.116.13:4182 , piece_id=BYAYNP4JAAN3AW2LYSK74WJE4BRWUR5Z6K67MTLK6A73KEJ35WEQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.356675316 (606h8m3.227758361s) [noise=true , address=77.90.31.12:4612 , piece_id=63TS3SUEFUE6PORZ4W76OXRWFL7SBN773UGGY3DU7JPBAW66N7UA, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.356992732 (606h8m3.228075767s) [noise=true , address=64.49.13.140:4507 , piece_id=E7T6WL45F6GHECCRANX6AIKWFRZ77TURRWORYEYI43MET2WG7SRQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.357330644 (606h8m3.228413679s) [noise=true , address=77.90.31.21:4886 , piece_id=ZQRCPTVC3WFVQPZ36AU6KJW7PD3T557CXYQIICLDGOKK6JCPVZ5Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.357663649 (606h8m3.228746684s) [noise=true , address=145.79.227.49:4367 , piece_id=R2T2ABLXBPDVP3ZVA2TAYMCLQQ7JQUVQYCTHCKGMFBRVWCDAO4PQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.359827449 (606h8m3.230910484s) [noise=true , address=145.79.227.34:4918 , piece_id=HUMOHOT3HOYCBJ23WXQXWU3X6C6JQTRLLHTD4RQLCQTCB3HXI7EQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.360086692 (606h8m3.231169727s) [noise=true , address=145.79.227.7:4320 , piece_id=MAICNTHGLZ6W27HXDOOZXOBRGQGEXNSA4BJT2T5D5PHRVV6ZMX2Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.361622167 (606h8m3.232705212s) [noise=true , address=145.79.227.33:4888 , piece_id=I4426P3VETHPOL5WHWJEGVYRXJULTCNDX3WSUTS3XZVY36Q2N44Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.362965514 (606h8m3.234048559s) [noise=true , address=45.91.116.11:4133 , piece_id=NEW7ZOVP5E7HYGYD4LG5UG2MDLAQW2OBRTNZDFIYO4CQYMVAWRPQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.364308883 (606h8m3.235391928s) [noise=true , address=145.79.227.26:4663 , piece_id=DYEFQZL4X74QR2LAYURT5ZTR2IWIDB55VLBQOGZM36MA54P2SWJA, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.365734507 (606h8m3.236817542s) [noise=true , address=145.79.227.40:4059 , piece_id=7WSUBR3C3PWVBCWD6KRGSD6MPX4IH2YQOAVO4J4SWZORNHGS2IPQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.366982764 (606h8m3.238065799s) [noise=true , address=145.79.227.29:4973 , piece_id=5OROXELR55A47BFPSJJEWVQU4YW6CAOK6LMEHCINM4FN3OVI2Y6Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.368390294 (606h8m3.239473339s) [noise=true , address=77.90.31.19:4852 , piece_id=2NYO63GCBGCYBFYTMUGNF2NG5FSJOD3AOUOD2FZZIWDINK4X6DJQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:04.736295138 (606h8m4.607378183s) [upload=531240]: Next returned nil segment Dec 30 21:37:04.736311850 (606h8m4.607394885s) [upload=531240]: Waiting for error group managing segments... Dec 30 21:37:05.104989591 (606h8m4.976072636s) [noise=true , address=64.49.13.140:4507 , piece_id=E7T6WL45F6GHECCRANX6AIKWFRZ77TURRWORYEYI43MET2WG7SRQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.208809748 (606h8m5.079892793s) [noise=true , address=145.79.227.22:4543 , piece_id=WWCZ6SEEXQ2KUJ32AGTIJNMSJROLYSJ4XQGBQJC4Q5DACZVA65RQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.441125389 (606h8m5.312208444s) [noise=true , address=77.90.31.19:4852 , piece_id=2NYO63GCBGCYBFYTMUGNF2NG5FSJOD3AOUOD2FZZIWDINK4X6DJQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.483630904 (606h8m5.35471394s) [noise=true , address=77.90.31.12:4612 , piece_id=63TS3SUEFUE6PORZ4W76OXRWFL7SBN773UGGY3DU7JPBAW66N7UA, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.494478811 (606h8m5.365561846s) [noise=true , address=77.90.31.11:4559 , piece_id=3DYD4CNPJQN3S4EYZJGREQTTZJDSL2QTC3OT6G2VFUGMLSVCZUFA, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.497452683 (606h8m5.368535729s) [noise=true , address=77.90.31.17:4769 , piece_id=QXDYESBKGXRY7BCFYYF2O7OXS7Y4Z4KUMJB6AXZAWX735SYU56GQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.661529377 (606h8m5.532612412s) [noise=true , address=45.91.116.13:4182 , piece_id=BYAYNP4JAAN3AW2LYSK74WJE4BRWUR5Z6K67MTLK6A73KEJ35WEQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.729773112 (606h8m5.600856157s) [noise=true , address=45.91.116.11:4133 , piece_id=NEW7ZOVP5E7HYGYD4LG5UG2MDLAQW2OBRTNZDFIYO4CQYMVAWRPQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.940743308 (606h8m5.811826353s) [noise=true , address=77.90.31.21:4886 , piece_id=ZQRCPTVC3WFVQPZ36AU6KJW7PD3T557CXYQIICLDGOKK6JCPVZ5Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.084240044 (606h8m5.95532309s) [noise=true , address=145.79.227.30:4784 , piece_id=DSXUDNYCWO5F6IYXMIVJNEASYXIZ7EO7RKAS7QRUYK5OU7GOOB5Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.121095032 (606h8m5.992178067s) [noise=true , address=145.79.227.40:4059 , piece_id=7WSUBR3C3PWVBCWD6KRGSD6MPX4IH2YQOAVO4J4SWZORNHGS2IPQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.169195983 (606h8m6.040279028s) [noise=true , address=145.79.227.49:4367 , piece_id=R2T2ABLXBPDVP3ZVA2TAYMCLQQ7JQUVQYCTHCKGMFBRVWCDAO4PQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.169380634 (606h8m6.040463679s) [noise=true , address=145.79.227.33:4888 , piece_id=I4426P3VETHPOL5WHWJEGVYRXJULTCNDX3WSUTS3XZVY36Q2N44Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.179060175 (606h8m6.050143211s) [noise=true , address=145.79.227.29:4973 , piece_id=5OROXELR55A47BFPSJJEWVQU4YW6CAOK6LMEHCINM4FN3OVI2Y6Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.179080424 (606h8m6.050163459s) [seg_pos={0 0}, upload=531240]: Segment reached optimal threshold of 14 pieces. Dec 30 21:37:06.179262801 (606h8m6.050345837s) [noise=true , address=145.79.227.26:4663 , piece_id=DYEFQZL4X74QR2LAYURT5ZTR2IWIDB55VLBQOGZM36MA54P2SWJA, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SwpAaXTA3BqUvwubzmJkvwokdz6SSpT9t8HPtQcXXJxmcgfm2): protocol: write tcp 188.166.225.170:43184->145.79.227.26:4663: use of closed network connection; write tcp 188.166.225.170:43184->145.79.227.26:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43184->145.79.227.26:4663: use of closed network connection Dec 30 21:37:06.179268703 (606h8m6.050351748s) [noise=true , address=145.79.227.26:4663 , piece_id=DYEFQZL4X74QR2LAYURT5ZTR2IWIDB55VLBQOGZM36MA54P2SWJA, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179363724 (606h8m6.050446769s) [noise=true , address=145.79.227.7:4320 , piece_id=MAICNTHGLZ6W27HXDOOZXOBRGQGEXNSA4BJT2T5D5PHRVV6ZMX2Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126rEgnQw1w7hsFpRqQP9nLaumdBfdtkgYjdw6WE6nXrxk5kj4B): protocol: write tcp 188.166.225.170:45228->145.79.227.7:4320: use of closed network connection; write tcp 188.166.225.170:45228->145.79.227.7:4320: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45228->145.79.227.7:4320: use of closed network connection Dec 30 21:37:06.179369525 (606h8m6.05045257s) [noise=true , address=145.79.227.7:4320 , piece_id=MAICNTHGLZ6W27HXDOOZXOBRGQGEXNSA4BJT2T5D5PHRVV6ZMX2Q, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179532996 (606h8m6.050616061s) [noise=true , address=145.79.227.34:4918 , piece_id=HUMOHOT3HOYCBJ23WXQXWU3X6C6JQTRLLHTD4RQLCQTCB3HXI7EQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF Dec 30 21:37:06.179572371 (606h8m6.050655406s) [noise=true , address=145.79.227.34:4918 , piece_id=HUMOHOT3HOYCBJ23WXQXWU3X6C6JQTRLLHTD4RQLCQTCB3HXI7EQ, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179638587 (606h8m6.050721622s) [noise=true , address=145.79.227.35:4492 , piece_id=HCKFSQMJYEP6B5H4OX5AZ5TWP6Z4H4Z6WA2T7Z2H5KNZ4SDJRMUQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1roa5GGYmqCX2AxSrqFgqyV2CtBEYZavuGTseiwDy4pjkT6bip): protocol: expected piece hash; context canceled; EOF Dec 30 21:37:06.179647855 (606h8m6.05073089s) [noise=true , address=145.79.227.35:4492 , piece_id=HCKFSQMJYEP6B5H4OX5AZ5TWP6Z4H4Z6WA2T7Z2H5KNZ4SDJRMUQ, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179666641 (606h8m6.050749686s) [seg_pos={0 0}, upload=531240]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 21:37:06.179742766 (606h8m6.050825811s) [upload=531240]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 21:37:06.179747545 (606h8m6.05083059s) [upload=531240]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 21:37:06.490986034 (606h8m6.362069079s) [upload=531240]: Done uploading segments... Dec 30 21:37:06.491006261 (606h8m6.362089307s) [upload=531240]: Upload finished. err: <nil> Duration: 15.540069814s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [..X.X.........X..X...X.........X.........................X..] duration: 16.767061584s trace: https://jaeger.dev.storj.io/trace/57a7ab98e7eb0d3f FAIL (2025-12-30 22:00:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 21:58:42.298454527 (606h29m42.169537592s) [upload=531550]: Starting upload Dec 30 21:58:42.298523438 (606h29m42.169606473s) [upload=531550]: Done starting upload Dec 30 21:58:42.298568774 (606h29m42.169651809s) [upload=531550]: Uploading segments... Dec 30 21:58:42.298697850 (606h29m42.169780885s) [upload=531550]: Got next segment. Inline: false Dec 30 21:58:42.298728649 (606h29m42.169811694s) [upload=531550]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 21:58:42.298733227 (606h29m42.169816262s) [upload=531550]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 21:58:42.545734213 (606h29m42.416817288s) [seg_pos={0 0}, upload=531550]: Begin upload segment... Dec 30 21:58:42.545786032 (606h29m42.416869078s) [seg_pos={0 0}, upload=531550]: Done begin upload segment. Dec 30 21:58:42.545855104 (606h29m42.416938149s) [noise=true , address=145.79.227.23:4547 , piece_id=LYGUMTAP7OPJWGRVFY65GUZFE4BBTV73JTG657DR5OM26LA7UFZA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.545897384 (606h29m42.416980449s) [noise=true , address=145.79.227.49:4339 , piece_id=WBBPZK2VY2QVRTQNXEKQ7TRYPWBDYS2TGTKEDAAXQ2AT4TUQXTQQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.546757752 (606h29m42.417840817s) [noise=true , address=45.91.116.12:4225 , piece_id=LLYEKE4TJ54KECLAC7KFUV44IVHYCYLF45I43G5V3DPV5QDBUOIQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.546816995 (606h29m42.41790004s) [noise=true , address=145.79.227.40:4050 , piece_id=LA7KLHHWRNHULLPAZWQ6MLS76SLN3BONYXZJXCUHGFC5JZ4SQRIA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.547216927 (606h29m42.418299962s) [noise=true , address=145.79.227.15:4876 , piece_id=L2OIJAB6OOANT2EPFNVNLTMYCH334JGFBFASD44EHOAWTAHKUV5Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.547615996 (606h29m42.418699042s) [noise=true , address=145.79.227.17:4971 , piece_id=DYZUALBL3V75TKGK57LOJABWVCOD5UVS3WH74I55OVUDGRQUA3QQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.547841055 (606h29m42.4189241s) [noise=true , address=77.90.31.21:4877 , piece_id=BD2FEFNAGYWZYDA2IMOVHA7JE66EXSF3A3PL54Q2DUOJFG5ALG4Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548023322 (606h29m42.419106357s) [noise=true , address=45.91.116.14:4190 , piece_id=HHU5YZBTJEEWYVST6XQCKGWKRLRQIIR2UQTQ27RLK53BNAVOSHFQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548380923 (606h29m42.419463968s) [noise=true , address=45.91.116.11:4120 , piece_id=FZ5L6FM7IHX65EVFGVXXHEF43XQ4EBUH4C7VZUSTH2MVWLNR5M2Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548560475 (606h29m42.41964351s) [noise=true , address=145.79.227.39:4013 , piece_id=I4XP47SFJICIYIPEN4E7UC2HXRXH4IXVJYGHXTZBCUESAHNFQXJA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548719268 (606h29m42.419802313s) [noise=true , address=145.79.227.31:4797 , piece_id=S5YBLITONGU6RHPQENP2TCVVWOHEMXPMYXLNYWBZNXWMFI26PVUA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549024107 (606h29m42.420107152s) [noise=true , address=145.79.227.7:4294 , piece_id=C6N5OIRLV3LLSCEIOGQJKWPKENGZYNAPY6V2AYZNKUMACBP5SLWQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549197708 (606h29m42.420280753s) [noise=true , address=145.79.227.16:4910 , piece_id=O4YO367LEVZSRI5E75YEZ7W7YI2KGKKQDXCOHQ3LALE4WUAPQVHQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549506406 (606h29m42.420589441s) [noise=true , address=145.79.227.12:4765 , piece_id=4ZF5YQP4Z5HBTX5YWUDHZV55Q47WASRLMV6ZYJXJRL2QBAVQQM4Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549971591 (606h29m42.421054626s) [noise=true , address=145.79.227.32:4835 , piece_id=CN4NMCFNYKOCG2JCIEXJBL2PLEYWD5THCZB4WQDURIECFVHLLX6Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.551571588 (606h29m42.422654623s) [noise=true , address=145.79.227.24:4581 , piece_id=Y6GBEOR7DQX2H6LBHXYFOXRBX3GP7AKPYVGTQNMUEXKPXLONUUGQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:44.247717212 (606h29m44.118800248s) [upload=531550]: Next returned nil segment Dec 30 21:58:44.247730348 (606h29m44.118813383s) [upload=531550]: Waiting for error group managing segments... Dec 30 21:58:44.532148720 (606h29m44.403231755s) [noise=true , address=145.79.227.31:4797 , piece_id=S5YBLITONGU6RHPQENP2TCVVWOHEMXPMYXLNYWBZNXWMFI26PVUA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.534655293 (606h29m44.405738328s) [noise=true , address=145.79.227.39:4013 , piece_id=I4XP47SFJICIYIPEN4E7UC2HXRXH4IXVJYGHXTZBCUESAHNFQXJA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.544709427 (606h29m44.415792462s) [noise=true , address=145.79.227.12:4765 , piece_id=4ZF5YQP4Z5HBTX5YWUDHZV55Q47WASRLMV6ZYJXJRL2QBAVQQM4Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.545822306 (606h29m44.416905342s) [noise=true , address=145.79.227.7:4294 , piece_id=C6N5OIRLV3LLSCEIOGQJKWPKENGZYNAPY6V2AYZNKUMACBP5SLWQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.549465162 (606h29m44.420548198s) [noise=true , address=145.79.227.24:4581 , piece_id=Y6GBEOR7DQX2H6LBHXYFOXRBX3GP7AKPYVGTQNMUEXKPXLONUUGQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.556935928 (606h29m44.428018964s) [noise=true , address=145.79.227.15:4876 , piece_id=L2OIJAB6OOANT2EPFNVNLTMYCH334JGFBFASD44EHOAWTAHKUV5Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.561283296 (606h29m44.432366331s) [noise=true , address=145.79.227.16:4910 , piece_id=O4YO367LEVZSRI5E75YEZ7W7YI2KGKKQDXCOHQ3LALE4WUAPQVHQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.614194264 (606h29m44.485277309s) [noise=true , address=145.79.227.17:4971 , piece_id=DYZUALBL3V75TKGK57LOJABWVCOD5UVS3WH74I55OVUDGRQUA3QQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.615630689 (606h29m44.486713724s) [noise=true , address=145.79.227.32:4835 , piece_id=CN4NMCFNYKOCG2JCIEXJBL2PLEYWD5THCZB4WQDURIECFVHLLX6Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.639672077 (606h29m44.510755122s) [noise=true , address=145.79.227.23:4547 , piece_id=LYGUMTAP7OPJWGRVFY65GUZFE4BBTV73JTG657DR5OM26LA7UFZA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.699618771 (606h29m44.570701817s) [noise=true , address=77.90.31.21:4877 , piece_id=BD2FEFNAGYWZYDA2IMOVHA7JE66EXSF3A3PL54Q2DUOJFG5ALG4Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.841535416 (606h29m44.712618462s) [noise=true , address=45.91.116.14:4190 , piece_id=HHU5YZBTJEEWYVST6XQCKGWKRLRQIIR2UQTQ27RLK53BNAVOSHFQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.974660841 (606h29m44.845743886s) [noise=true , address=45.91.116.12:4225 , piece_id=LLYEKE4TJ54KECLAC7KFUV44IVHYCYLF45I43G5V3DPV5QDBUOIQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.974681401 (606h29m44.845764436s) [seg_pos={0 0}, upload=531550]: Segment reached optimal threshold of 13 pieces. Dec 30 21:58:44.974855342 (606h29m44.845938377s) [noise=true , address=145.79.227.49:4339 , piece_id=WBBPZK2VY2QVRTQNXEKQ7TRYPWBDYS2TGTKEDAAXQ2AT4TUQXTQQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 30 21:58:44.974861423 (606h29m44.845944458s) [noise=true , address=145.79.227.49:4339 , piece_id=WBBPZK2VY2QVRTQNXEKQ7TRYPWBDYS2TGTKEDAAXQ2AT4TUQXTQQ, seg_pos={0 0}, upload=531550]: Long tail cancellation due to optimal threshold Dec 30 21:58:44.974968166 (606h29m44.846051201s) [noise=true , address=145.79.227.40:4050 , piece_id=LA7KLHHWRNHULLPAZWQ6MLS76SLN3BONYXZJXCUHGFC5JZ4SQRIA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 30 21:58:44.974973627 (606h29m44.846056662s) [noise=true , address=145.79.227.40:4050 , piece_id=LA7KLHHWRNHULLPAZWQ6MLS76SLN3BONYXZJXCUHGFC5JZ4SQRIA, seg_pos={0 0}, upload=531550]: Long tail cancellation due to optimal threshold Dec 30 21:58:44.975076152 (606h29m44.846159187s) [noise=true , address=45.91.116.11:4120 , piece_id=FZ5L6FM7IHX65EVFGVXXHEF43XQ4EBUH4C7VZUSTH2MVWLNR5M2Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 30 21:58:44.975083497 (606h29m44.846166532s) [noise=true , address=45.91.116.11:4120 , piece_id=FZ5L6FM7IHX65EVFGVXXHEF43XQ4EBUH4C7VZUSTH2MVWLNR5M2Q, seg_pos={0 0}, upload=531550]: Long tail cancellation due to optimal threshold Dec 30 21:58:44.975096711 (606h29m44.846179747s) [seg_pos={0 0}, upload=531550]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 21:58:44.975160703 (606h29m44.846243739s) [upload=531550]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 21:58:44.975164691 (606h29m44.846247726s) [upload=531550]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 21:58:45.272782726 (606h29m45.143865772s) [upload=531550]: Done uploading segments... Dec 30 21:58:45.272806352 (606h29m45.143889387s) [upload=531550]: Upload finished. err: <nil> Duration: 16.767089868s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................................................X..........] duration: 3.631097655s trace: https://jaeger.dev.storj.io/trace/22a47ec27f9a101c FAIL (2025-12-30 22:00:11 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.631137411s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................................................X.........] duration: 3.637103018s trace: https://jaeger.dev.storj.io/trace/22a47ec27f9a101c FAIL (2025-12-30 21:59:50 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.637131652s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [..............XX......X.........X...........................] duration: 15.301812629s trace: https://jaeger.dev.storj.io/trace/5cdd5ba7b1cb22c7 FAIL (2025-12-30 21:59:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 21:37:03.355296401 (606h8m3.226379446s) [noise=true , address=145.79.227.22:4543 , piece_id=WWCZ6SEEXQ2KUJ32AGTIJNMSJROLYSJ4XQGBQJC4Q5DACZVA65RQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.355839374 (606h8m3.22692241s) [noise=true , address=145.79.227.35:4492 , piece_id=HCKFSQMJYEP6B5H4OX5AZ5TWP6Z4H4Z6WA2T7Z2H5KNZ4SDJRMUQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.356356930 (606h8m3.227439965s) [noise=true , address=45.91.116.13:4182 , piece_id=BYAYNP4JAAN3AW2LYSK74WJE4BRWUR5Z6K67MTLK6A73KEJ35WEQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.356675316 (606h8m3.227758361s) [noise=true , address=77.90.31.12:4612 , piece_id=63TS3SUEFUE6PORZ4W76OXRWFL7SBN773UGGY3DU7JPBAW66N7UA, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.356992732 (606h8m3.228075767s) [noise=true , address=64.49.13.140:4507 , piece_id=E7T6WL45F6GHECCRANX6AIKWFRZ77TURRWORYEYI43MET2WG7SRQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.357330644 (606h8m3.228413679s) [noise=true , address=77.90.31.21:4886 , piece_id=ZQRCPTVC3WFVQPZ36AU6KJW7PD3T557CXYQIICLDGOKK6JCPVZ5Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.357663649 (606h8m3.228746684s) [noise=true , address=145.79.227.49:4367 , piece_id=R2T2ABLXBPDVP3ZVA2TAYMCLQQ7JQUVQYCTHCKGMFBRVWCDAO4PQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.359827449 (606h8m3.230910484s) [noise=true , address=145.79.227.34:4918 , piece_id=HUMOHOT3HOYCBJ23WXQXWU3X6C6JQTRLLHTD4RQLCQTCB3HXI7EQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.360086692 (606h8m3.231169727s) [noise=true , address=145.79.227.7:4320 , piece_id=MAICNTHGLZ6W27HXDOOZXOBRGQGEXNSA4BJT2T5D5PHRVV6ZMX2Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.361622167 (606h8m3.232705212s) [noise=true , address=145.79.227.33:4888 , piece_id=I4426P3VETHPOL5WHWJEGVYRXJULTCNDX3WSUTS3XZVY36Q2N44Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.362965514 (606h8m3.234048559s) [noise=true , address=45.91.116.11:4133 , piece_id=NEW7ZOVP5E7HYGYD4LG5UG2MDLAQW2OBRTNZDFIYO4CQYMVAWRPQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.364308883 (606h8m3.235391928s) [noise=true , address=145.79.227.26:4663 , piece_id=DYEFQZL4X74QR2LAYURT5ZTR2IWIDB55VLBQOGZM36MA54P2SWJA, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.365734507 (606h8m3.236817542s) [noise=true , address=145.79.227.40:4059 , piece_id=7WSUBR3C3PWVBCWD6KRGSD6MPX4IH2YQOAVO4J4SWZORNHGS2IPQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.366982764 (606h8m3.238065799s) [noise=true , address=145.79.227.29:4973 , piece_id=5OROXELR55A47BFPSJJEWVQU4YW6CAOK6LMEHCINM4FN3OVI2Y6Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.368390294 (606h8m3.239473339s) [noise=true , address=77.90.31.19:4852 , piece_id=2NYO63GCBGCYBFYTMUGNF2NG5FSJOD3AOUOD2FZZIWDINK4X6DJQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:04.736295138 (606h8m4.607378183s) [upload=531240]: Next returned nil segment Dec 30 21:37:04.736311850 (606h8m4.607394885s) [upload=531240]: Waiting for error group managing segments... Dec 30 21:37:05.104989591 (606h8m4.976072636s) [noise=true , address=64.49.13.140:4507 , piece_id=E7T6WL45F6GHECCRANX6AIKWFRZ77TURRWORYEYI43MET2WG7SRQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.208809748 (606h8m5.079892793s) [noise=true , address=145.79.227.22:4543 , piece_id=WWCZ6SEEXQ2KUJ32AGTIJNMSJROLYSJ4XQGBQJC4Q5DACZVA65RQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.441125389 (606h8m5.312208444s) [noise=true , address=77.90.31.19:4852 , piece_id=2NYO63GCBGCYBFYTMUGNF2NG5FSJOD3AOUOD2FZZIWDINK4X6DJQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.483630904 (606h8m5.35471394s) [noise=true , address=77.90.31.12:4612 , piece_id=63TS3SUEFUE6PORZ4W76OXRWFL7SBN773UGGY3DU7JPBAW66N7UA, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.494478811 (606h8m5.365561846s) [noise=true , address=77.90.31.11:4559 , piece_id=3DYD4CNPJQN3S4EYZJGREQTTZJDSL2QTC3OT6G2VFUGMLSVCZUFA, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.497452683 (606h8m5.368535729s) [noise=true , address=77.90.31.17:4769 , piece_id=QXDYESBKGXRY7BCFYYF2O7OXS7Y4Z4KUMJB6AXZAWX735SYU56GQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.661529377 (606h8m5.532612412s) [noise=true , address=45.91.116.13:4182 , piece_id=BYAYNP4JAAN3AW2LYSK74WJE4BRWUR5Z6K67MTLK6A73KEJ35WEQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.729773112 (606h8m5.600856157s) [noise=true , address=45.91.116.11:4133 , piece_id=NEW7ZOVP5E7HYGYD4LG5UG2MDLAQW2OBRTNZDFIYO4CQYMVAWRPQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.940743308 (606h8m5.811826353s) [noise=true , address=77.90.31.21:4886 , piece_id=ZQRCPTVC3WFVQPZ36AU6KJW7PD3T557CXYQIICLDGOKK6JCPVZ5Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.084240044 (606h8m5.95532309s) [noise=true , address=145.79.227.30:4784 , piece_id=DSXUDNYCWO5F6IYXMIVJNEASYXIZ7EO7RKAS7QRUYK5OU7GOOB5Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.121095032 (606h8m5.992178067s) [noise=true , address=145.79.227.40:4059 , piece_id=7WSUBR3C3PWVBCWD6KRGSD6MPX4IH2YQOAVO4J4SWZORNHGS2IPQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.169195983 (606h8m6.040279028s) [noise=true , address=145.79.227.49:4367 , piece_id=R2T2ABLXBPDVP3ZVA2TAYMCLQQ7JQUVQYCTHCKGMFBRVWCDAO4PQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.169380634 (606h8m6.040463679s) [noise=true , address=145.79.227.33:4888 , piece_id=I4426P3VETHPOL5WHWJEGVYRXJULTCNDX3WSUTS3XZVY36Q2N44Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.179060175 (606h8m6.050143211s) [noise=true , address=145.79.227.29:4973 , piece_id=5OROXELR55A47BFPSJJEWVQU4YW6CAOK6LMEHCINM4FN3OVI2Y6Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.179080424 (606h8m6.050163459s) [seg_pos={0 0}, upload=531240]: Segment reached optimal threshold of 14 pieces. Dec 30 21:37:06.179262801 (606h8m6.050345837s) [noise=true , address=145.79.227.26:4663 , piece_id=DYEFQZL4X74QR2LAYURT5ZTR2IWIDB55VLBQOGZM36MA54P2SWJA, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SwpAaXTA3BqUvwubzmJkvwokdz6SSpT9t8HPtQcXXJxmcgfm2): protocol: write tcp 188.166.225.170:43184->145.79.227.26:4663: use of closed network connection; write tcp 188.166.225.170:43184->145.79.227.26:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43184->145.79.227.26:4663: use of closed network connection Dec 30 21:37:06.179268703 (606h8m6.050351748s) [noise=true , address=145.79.227.26:4663 , piece_id=DYEFQZL4X74QR2LAYURT5ZTR2IWIDB55VLBQOGZM36MA54P2SWJA, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179363724 (606h8m6.050446769s) [noise=true , address=145.79.227.7:4320 , piece_id=MAICNTHGLZ6W27HXDOOZXOBRGQGEXNSA4BJT2T5D5PHRVV6ZMX2Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126rEgnQw1w7hsFpRqQP9nLaumdBfdtkgYjdw6WE6nXrxk5kj4B): protocol: write tcp 188.166.225.170:45228->145.79.227.7:4320: use of closed network connection; write tcp 188.166.225.170:45228->145.79.227.7:4320: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45228->145.79.227.7:4320: use of closed network connection Dec 30 21:37:06.179369525 (606h8m6.05045257s) [noise=true , address=145.79.227.7:4320 , piece_id=MAICNTHGLZ6W27HXDOOZXOBRGQGEXNSA4BJT2T5D5PHRVV6ZMX2Q, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179532996 (606h8m6.050616061s) [noise=true , address=145.79.227.34:4918 , piece_id=HUMOHOT3HOYCBJ23WXQXWU3X6C6JQTRLLHTD4RQLCQTCB3HXI7EQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF Dec 30 21:37:06.179572371 (606h8m6.050655406s) [noise=true , address=145.79.227.34:4918 , piece_id=HUMOHOT3HOYCBJ23WXQXWU3X6C6JQTRLLHTD4RQLCQTCB3HXI7EQ, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179638587 (606h8m6.050721622s) [noise=true , address=145.79.227.35:4492 , piece_id=HCKFSQMJYEP6B5H4OX5AZ5TWP6Z4H4Z6WA2T7Z2H5KNZ4SDJRMUQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1roa5GGYmqCX2AxSrqFgqyV2CtBEYZavuGTseiwDy4pjkT6bip): protocol: expected piece hash; context canceled; EOF Dec 30 21:37:06.179647855 (606h8m6.05073089s) [noise=true , address=145.79.227.35:4492 , piece_id=HCKFSQMJYEP6B5H4OX5AZ5TWP6Z4H4Z6WA2T7Z2H5KNZ4SDJRMUQ, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179666641 (606h8m6.050749686s) [seg_pos={0 0}, upload=531240]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 21:37:06.179742766 (606h8m6.050825811s) [upload=531240]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 21:37:06.179747545 (606h8m6.05083059s) [upload=531240]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 21:37:06.490986034 (606h8m6.362069079s) [upload=531240]: Done uploading segments... Dec 30 21:37:06.491006261 (606h8m6.362089307s) [upload=531240]: Upload finished. err: <nil> Duration: 15.301834139s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [...X.X.........X..X...X.........X.........................X.] duration: 6.052608401s trace: https://jaeger.dev.storj.io/trace/57a7ab98e7eb0d3f FAIL (2025-12-30 21:59:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 21:58:42.298454527 (606h29m42.169537592s) [upload=531550]: Starting upload Dec 30 21:58:42.298523438 (606h29m42.169606473s) [upload=531550]: Done starting upload Dec 30 21:58:42.298568774 (606h29m42.169651809s) [upload=531550]: Uploading segments... Dec 30 21:58:42.298697850 (606h29m42.169780885s) [upload=531550]: Got next segment. Inline: false Dec 30 21:58:42.298728649 (606h29m42.169811694s) [upload=531550]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 21:58:42.298733227 (606h29m42.169816262s) [upload=531550]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 21:58:42.545734213 (606h29m42.416817288s) [seg_pos={0 0}, upload=531550]: Begin upload segment... Dec 30 21:58:42.545786032 (606h29m42.416869078s) [seg_pos={0 0}, upload=531550]: Done begin upload segment. Dec 30 21:58:42.545855104 (606h29m42.416938149s) [noise=true , address=145.79.227.23:4547 , piece_id=LYGUMTAP7OPJWGRVFY65GUZFE4BBTV73JTG657DR5OM26LA7UFZA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.545897384 (606h29m42.416980449s) [noise=true , address=145.79.227.49:4339 , piece_id=WBBPZK2VY2QVRTQNXEKQ7TRYPWBDYS2TGTKEDAAXQ2AT4TUQXTQQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.546757752 (606h29m42.417840817s) [noise=true , address=45.91.116.12:4225 , piece_id=LLYEKE4TJ54KECLAC7KFUV44IVHYCYLF45I43G5V3DPV5QDBUOIQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.546816995 (606h29m42.41790004s) [noise=true , address=145.79.227.40:4050 , piece_id=LA7KLHHWRNHULLPAZWQ6MLS76SLN3BONYXZJXCUHGFC5JZ4SQRIA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.547216927 (606h29m42.418299962s) [noise=true , address=145.79.227.15:4876 , piece_id=L2OIJAB6OOANT2EPFNVNLTMYCH334JGFBFASD44EHOAWTAHKUV5Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.547615996 (606h29m42.418699042s) [noise=true , address=145.79.227.17:4971 , piece_id=DYZUALBL3V75TKGK57LOJABWVCOD5UVS3WH74I55OVUDGRQUA3QQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.547841055 (606h29m42.4189241s) [noise=true , address=77.90.31.21:4877 , piece_id=BD2FEFNAGYWZYDA2IMOVHA7JE66EXSF3A3PL54Q2DUOJFG5ALG4Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548023322 (606h29m42.419106357s) [noise=true , address=45.91.116.14:4190 , piece_id=HHU5YZBTJEEWYVST6XQCKGWKRLRQIIR2UQTQ27RLK53BNAVOSHFQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548380923 (606h29m42.419463968s) [noise=true , address=45.91.116.11:4120 , piece_id=FZ5L6FM7IHX65EVFGVXXHEF43XQ4EBUH4C7VZUSTH2MVWLNR5M2Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548560475 (606h29m42.41964351s) [noise=true , address=145.79.227.39:4013 , piece_id=I4XP47SFJICIYIPEN4E7UC2HXRXH4IXVJYGHXTZBCUESAHNFQXJA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548719268 (606h29m42.419802313s) [noise=true , address=145.79.227.31:4797 , piece_id=S5YBLITONGU6RHPQENP2TCVVWOHEMXPMYXLNYWBZNXWMFI26PVUA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549024107 (606h29m42.420107152s) [noise=true , address=145.79.227.7:4294 , piece_id=C6N5OIRLV3LLSCEIOGQJKWPKENGZYNAPY6V2AYZNKUMACBP5SLWQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549197708 (606h29m42.420280753s) [noise=true , address=145.79.227.16:4910 , piece_id=O4YO367LEVZSRI5E75YEZ7W7YI2KGKKQDXCOHQ3LALE4WUAPQVHQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549506406 (606h29m42.420589441s) [noise=true , address=145.79.227.12:4765 , piece_id=4ZF5YQP4Z5HBTX5YWUDHZV55Q47WASRLMV6ZYJXJRL2QBAVQQM4Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549971591 (606h29m42.421054626s) [noise=true , address=145.79.227.32:4835 , piece_id=CN4NMCFNYKOCG2JCIEXJBL2PLEYWD5THCZB4WQDURIECFVHLLX6Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.551571588 (606h29m42.422654623s) [noise=true , address=145.79.227.24:4581 , piece_id=Y6GBEOR7DQX2H6LBHXYFOXRBX3GP7AKPYVGTQNMUEXKPXLONUUGQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:44.247717212 (606h29m44.118800248s) [upload=531550]: Next returned nil segment Dec 30 21:58:44.247730348 (606h29m44.118813383s) [upload=531550]: Waiting for error group managing segments... Dec 30 21:58:44.532148720 (606h29m44.403231755s) [noise=true , address=145.79.227.31:4797 , piece_id=S5YBLITONGU6RHPQENP2TCVVWOHEMXPMYXLNYWBZNXWMFI26PVUA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.534655293 (606h29m44.405738328s) [noise=true , address=145.79.227.39:4013 , piece_id=I4XP47SFJICIYIPEN4E7UC2HXRXH4IXVJYGHXTZBCUESAHNFQXJA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.544709427 (606h29m44.415792462s) [noise=true , address=145.79.227.12:4765 , piece_id=4ZF5YQP4Z5HBTX5YWUDHZV55Q47WASRLMV6ZYJXJRL2QBAVQQM4Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.545822306 (606h29m44.416905342s) [noise=true , address=145.79.227.7:4294 , piece_id=C6N5OIRLV3LLSCEIOGQJKWPKENGZYNAPY6V2AYZNKUMACBP5SLWQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.549465162 (606h29m44.420548198s) [noise=true , address=145.79.227.24:4581 , piece_id=Y6GBEOR7DQX2H6LBHXYFOXRBX3GP7AKPYVGTQNMUEXKPXLONUUGQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.556935928 (606h29m44.428018964s) [noise=true , address=145.79.227.15:4876 , piece_id=L2OIJAB6OOANT2EPFNVNLTMYCH334JGFBFASD44EHOAWTAHKUV5Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.561283296 (606h29m44.432366331s) [noise=true , address=145.79.227.16:4910 , piece_id=O4YO367LEVZSRI5E75YEZ7W7YI2KGKKQDXCOHQ3LALE4WUAPQVHQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.614194264 (606h29m44.485277309s) [noise=true , address=145.79.227.17:4971 , piece_id=DYZUALBL3V75TKGK57LOJABWVCOD5UVS3WH74I55OVUDGRQUA3QQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.615630689 (606h29m44.486713724s) [noise=true , address=145.79.227.32:4835 , piece_id=CN4NMCFNYKOCG2JCIEXJBL2PLEYWD5THCZB4WQDURIECFVHLLX6Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.639672077 (606h29m44.510755122s) [noise=true , address=145.79.227.23:4547 , piece_id=LYGUMTAP7OPJWGRVFY65GUZFE4BBTV73JTG657DR5OM26LA7UFZA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.699618771 (606h29m44.570701817s) [noise=true , address=77.90.31.21:4877 , piece_id=BD2FEFNAGYWZYDA2IMOVHA7JE66EXSF3A3PL54Q2DUOJFG5ALG4Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.841535416 (606h29m44.712618462s) [noise=true , address=45.91.116.14:4190 , piece_id=HHU5YZBTJEEWYVST6XQCKGWKRLRQIIR2UQTQ27RLK53BNAVOSHFQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.974660841 (606h29m44.845743886s) [noise=true , address=45.91.116.12:4225 , piece_id=LLYEKE4TJ54KECLAC7KFUV44IVHYCYLF45I43G5V3DPV5QDBUOIQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.974681401 (606h29m44.845764436s) [seg_pos={0 0}, upload=531550]: Segment reached optimal threshold of 13 pieces. Dec 30 21:58:44.974855342 (606h29m44.845938377s) [noise=true , address=145.79.227.49:4339 , piece_id=WBBPZK2VY2QVRTQNXEKQ7TRYPWBDYS2TGTKEDAAXQ2AT4TUQXTQQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 30 21:58:44.974861423 (606h29m44.845944458s) [noise=true , address=145.79.227.49:4339 , piece_id=WBBPZK2VY2QVRTQNXEKQ7TRYPWBDYS2TGTKEDAAXQ2AT4TUQXTQQ, seg_pos={0 0}, upload=531550]: Long tail cancellation due to optimal threshold Dec 30 21:58:44.974968166 (606h29m44.846051201s) [noise=true , address=145.79.227.40:4050 , piece_id=LA7KLHHWRNHULLPAZWQ6MLS76SLN3BONYXZJXCUHGFC5JZ4SQRIA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 30 21:58:44.974973627 (606h29m44.846056662s) [noise=true , address=145.79.227.40:4050 , piece_id=LA7KLHHWRNHULLPAZWQ6MLS76SLN3BONYXZJXCUHGFC5JZ4SQRIA, seg_pos={0 0}, upload=531550]: Long tail cancellation due to optimal threshold Dec 30 21:58:44.975076152 (606h29m44.846159187s) [noise=true , address=45.91.116.11:4120 , piece_id=FZ5L6FM7IHX65EVFGVXXHEF43XQ4EBUH4C7VZUSTH2MVWLNR5M2Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 30 21:58:44.975083497 (606h29m44.846166532s) [noise=true , address=45.91.116.11:4120 , piece_id=FZ5L6FM7IHX65EVFGVXXHEF43XQ4EBUH4C7VZUSTH2MVWLNR5M2Q, seg_pos={0 0}, upload=531550]: Long tail cancellation due to optimal threshold Dec 30 21:58:44.975096711 (606h29m44.846179747s) [seg_pos={0 0}, upload=531550]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 21:58:44.975160703 (606h29m44.846243739s) [upload=531550]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 21:58:44.975164691 (606h29m44.846247726s) [upload=531550]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 21:58:45.272782726 (606h29m45.143865772s) [upload=531550]: Done uploading segments... Dec 30 21:58:45.272806352 (606h29m45.143889387s) [upload=531550]: Upload finished. err: <nil> Duration: 6.052637386s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................................................X........] duration: 3.575681027s trace: https://jaeger.dev.storj.io/trace/22a47ec27f9a101c FAIL (2025-12-30 21:59:18 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.575705884s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [....X.X.........X..X...X.........X.........................X] duration: 30.002141368s trace: https://jaeger.dev.storj.io/trace/57a7ab98e7eb0d3f FAIL (2025-12-30 21:59:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 21:58:42.298454527 (606h29m42.169537592s) [upload=531550]: Starting upload Dec 30 21:58:42.298523438 (606h29m42.169606473s) [upload=531550]: Done starting upload Dec 30 21:58:42.298568774 (606h29m42.169651809s) [upload=531550]: Uploading segments... Dec 30 21:58:42.298697850 (606h29m42.169780885s) [upload=531550]: Got next segment. Inline: false Dec 30 21:58:42.298728649 (606h29m42.169811694s) [upload=531550]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 21:58:42.298733227 (606h29m42.169816262s) [upload=531550]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 21:58:42.545734213 (606h29m42.416817288s) [seg_pos={0 0}, upload=531550]: Begin upload segment... Dec 30 21:58:42.545786032 (606h29m42.416869078s) [seg_pos={0 0}, upload=531550]: Done begin upload segment. Dec 30 21:58:42.545855104 (606h29m42.416938149s) [noise=true , address=145.79.227.23:4547 , piece_id=LYGUMTAP7OPJWGRVFY65GUZFE4BBTV73JTG657DR5OM26LA7UFZA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.545897384 (606h29m42.416980449s) [noise=true , address=145.79.227.49:4339 , piece_id=WBBPZK2VY2QVRTQNXEKQ7TRYPWBDYS2TGTKEDAAXQ2AT4TUQXTQQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.546757752 (606h29m42.417840817s) [noise=true , address=45.91.116.12:4225 , piece_id=LLYEKE4TJ54KECLAC7KFUV44IVHYCYLF45I43G5V3DPV5QDBUOIQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.546816995 (606h29m42.41790004s) [noise=true , address=145.79.227.40:4050 , piece_id=LA7KLHHWRNHULLPAZWQ6MLS76SLN3BONYXZJXCUHGFC5JZ4SQRIA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.547216927 (606h29m42.418299962s) [noise=true , address=145.79.227.15:4876 , piece_id=L2OIJAB6OOANT2EPFNVNLTMYCH334JGFBFASD44EHOAWTAHKUV5Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.547615996 (606h29m42.418699042s) [noise=true , address=145.79.227.17:4971 , piece_id=DYZUALBL3V75TKGK57LOJABWVCOD5UVS3WH74I55OVUDGRQUA3QQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.547841055 (606h29m42.4189241s) [noise=true , address=77.90.31.21:4877 , piece_id=BD2FEFNAGYWZYDA2IMOVHA7JE66EXSF3A3PL54Q2DUOJFG5ALG4Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548023322 (606h29m42.419106357s) [noise=true , address=45.91.116.14:4190 , piece_id=HHU5YZBTJEEWYVST6XQCKGWKRLRQIIR2UQTQ27RLK53BNAVOSHFQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548380923 (606h29m42.419463968s) [noise=true , address=45.91.116.11:4120 , piece_id=FZ5L6FM7IHX65EVFGVXXHEF43XQ4EBUH4C7VZUSTH2MVWLNR5M2Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548560475 (606h29m42.41964351s) [noise=true , address=145.79.227.39:4013 , piece_id=I4XP47SFJICIYIPEN4E7UC2HXRXH4IXVJYGHXTZBCUESAHNFQXJA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.548719268 (606h29m42.419802313s) [noise=true , address=145.79.227.31:4797 , piece_id=S5YBLITONGU6RHPQENP2TCVVWOHEMXPMYXLNYWBZNXWMFI26PVUA, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549024107 (606h29m42.420107152s) [noise=true , address=145.79.227.7:4294 , piece_id=C6N5OIRLV3LLSCEIOGQJKWPKENGZYNAPY6V2AYZNKUMACBP5SLWQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549197708 (606h29m42.420280753s) [noise=true , address=145.79.227.16:4910 , piece_id=O4YO367LEVZSRI5E75YEZ7W7YI2KGKKQDXCOHQ3LALE4WUAPQVHQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549506406 (606h29m42.420589441s) [noise=true , address=145.79.227.12:4765 , piece_id=4ZF5YQP4Z5HBTX5YWUDHZV55Q47WASRLMV6ZYJXJRL2QBAVQQM4Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.549971591 (606h29m42.421054626s) [noise=true , address=145.79.227.32:4835 , piece_id=CN4NMCFNYKOCG2JCIEXJBL2PLEYWD5THCZB4WQDURIECFVHLLX6Q, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:42.551571588 (606h29m42.422654623s) [noise=true , address=145.79.227.24:4581 , piece_id=Y6GBEOR7DQX2H6LBHXYFOXRBX3GP7AKPYVGTQNMUEXKPXLONUUGQ, seg_pos={0 0}, upload=531550]: Uploading piece... Dec 30 21:58:44.247717212 (606h29m44.118800248s) [upload=531550]: Next returned nil segment Dec 30 21:58:44.247730348 (606h29m44.118813383s) [upload=531550]: Waiting for error group managing segments... Dec 30 21:58:44.532148720 (606h29m44.403231755s) [noise=true , address=145.79.227.31:4797 , piece_id=S5YBLITONGU6RHPQENP2TCVVWOHEMXPMYXLNYWBZNXWMFI26PVUA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.534655293 (606h29m44.405738328s) [noise=true , address=145.79.227.39:4013 , piece_id=I4XP47SFJICIYIPEN4E7UC2HXRXH4IXVJYGHXTZBCUESAHNFQXJA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.544709427 (606h29m44.415792462s) [noise=true , address=145.79.227.12:4765 , piece_id=4ZF5YQP4Z5HBTX5YWUDHZV55Q47WASRLMV6ZYJXJRL2QBAVQQM4Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.545822306 (606h29m44.416905342s) [noise=true , address=145.79.227.7:4294 , piece_id=C6N5OIRLV3LLSCEIOGQJKWPKENGZYNAPY6V2AYZNKUMACBP5SLWQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.549465162 (606h29m44.420548198s) [noise=true , address=145.79.227.24:4581 , piece_id=Y6GBEOR7DQX2H6LBHXYFOXRBX3GP7AKPYVGTQNMUEXKPXLONUUGQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.556935928 (606h29m44.428018964s) [noise=true , address=145.79.227.15:4876 , piece_id=L2OIJAB6OOANT2EPFNVNLTMYCH334JGFBFASD44EHOAWTAHKUV5Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.561283296 (606h29m44.432366331s) [noise=true , address=145.79.227.16:4910 , piece_id=O4YO367LEVZSRI5E75YEZ7W7YI2KGKKQDXCOHQ3LALE4WUAPQVHQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.614194264 (606h29m44.485277309s) [noise=true , address=145.79.227.17:4971 , piece_id=DYZUALBL3V75TKGK57LOJABWVCOD5UVS3WH74I55OVUDGRQUA3QQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.615630689 (606h29m44.486713724s) [noise=true , address=145.79.227.32:4835 , piece_id=CN4NMCFNYKOCG2JCIEXJBL2PLEYWD5THCZB4WQDURIECFVHLLX6Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.639672077 (606h29m44.510755122s) [noise=true , address=145.79.227.23:4547 , piece_id=LYGUMTAP7OPJWGRVFY65GUZFE4BBTV73JTG657DR5OM26LA7UFZA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.699618771 (606h29m44.570701817s) [noise=true , address=77.90.31.21:4877 , piece_id=BD2FEFNAGYWZYDA2IMOVHA7JE66EXSF3A3PL54Q2DUOJFG5ALG4Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.841535416 (606h29m44.712618462s) [noise=true , address=45.91.116.14:4190 , piece_id=HHU5YZBTJEEWYVST6XQCKGWKRLRQIIR2UQTQ27RLK53BNAVOSHFQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.974660841 (606h29m44.845743886s) [noise=true , address=45.91.116.12:4225 , piece_id=LLYEKE4TJ54KECLAC7KFUV44IVHYCYLF45I43G5V3DPV5QDBUOIQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: <nil> Dec 30 21:58:44.974681401 (606h29m44.845764436s) [seg_pos={0 0}, upload=531550]: Segment reached optimal threshold of 13 pieces. Dec 30 21:58:44.974855342 (606h29m44.845938377s) [noise=true , address=145.79.227.49:4339 , piece_id=WBBPZK2VY2QVRTQNXEKQ7TRYPWBDYS2TGTKEDAAXQ2AT4TUQXTQQ, seg_pos={0 0}, upload=531550]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 30 21:58:44.974861423 (606h29m44.845944458s) [noise=true , address=145.79.227.49:4339 , piece_id=WBBPZK2VY2QVRTQNXEKQ7TRYPWBDYS2TGTKEDAAXQ2AT4TUQXTQQ, seg_pos={0 0}, upload=531550]: Long tail cancellation due to optimal threshold Dec 30 21:58:44.974968166 (606h29m44.846051201s) [noise=true , address=145.79.227.40:4050 , piece_id=LA7KLHHWRNHULLPAZWQ6MLS76SLN3BONYXZJXCUHGFC5JZ4SQRIA, seg_pos={0 0}, upload=531550]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 30 21:58:44.974973627 (606h29m44.846056662s) [noise=true , address=145.79.227.40:4050 , piece_id=LA7KLHHWRNHULLPAZWQ6MLS76SLN3BONYXZJXCUHGFC5JZ4SQRIA, seg_pos={0 0}, upload=531550]: Long tail cancellation due to optimal threshold Dec 30 21:58:44.975076152 (606h29m44.846159187s) [noise=true , address=45.91.116.11:4120 , piece_id=FZ5L6FM7IHX65EVFGVXXHEF43XQ4EBUH4C7VZUSTH2MVWLNR5M2Q, seg_pos={0 0}, upload=531550]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 30 21:58:44.975083497 (606h29m44.846166532s) [noise=true , address=45.91.116.11:4120 , piece_id=FZ5L6FM7IHX65EVFGVXXHEF43XQ4EBUH4C7VZUSTH2MVWLNR5M2Q, seg_pos={0 0}, upload=531550]: Long tail cancellation due to optimal threshold Dec 30 21:58:44.975096711 (606h29m44.846179747s) [seg_pos={0 0}, upload=531550]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 21:58:44.975160703 (606h29m44.846243739s) [upload=531550]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 21:58:44.975164691 (606h29m44.846247726s) [upload=531550]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 21:58:45.272782726 (606h29m45.143865772s) [upload=531550]: Done uploading segments... Dec 30 21:58:45.272806352 (606h29m45.143889387s) [upload=531550]: Upload finished. err: <nil> Duration: 30.002162007s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [...............XX......X.........X..........................] duration: 18.35331722s trace: https://jaeger.dev.storj.io/trace/5cdd5ba7b1cb22c7 FAIL (2025-12-30 21:58:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 21:37:03.355296401 (606h8m3.226379446s) [noise=true , address=145.79.227.22:4543 , piece_id=WWCZ6SEEXQ2KUJ32AGTIJNMSJROLYSJ4XQGBQJC4Q5DACZVA65RQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.355839374 (606h8m3.22692241s) [noise=true , address=145.79.227.35:4492 , piece_id=HCKFSQMJYEP6B5H4OX5AZ5TWP6Z4H4Z6WA2T7Z2H5KNZ4SDJRMUQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.356356930 (606h8m3.227439965s) [noise=true , address=45.91.116.13:4182 , piece_id=BYAYNP4JAAN3AW2LYSK74WJE4BRWUR5Z6K67MTLK6A73KEJ35WEQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.356675316 (606h8m3.227758361s) [noise=true , address=77.90.31.12:4612 , piece_id=63TS3SUEFUE6PORZ4W76OXRWFL7SBN773UGGY3DU7JPBAW66N7UA, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.356992732 (606h8m3.228075767s) [noise=true , address=64.49.13.140:4507 , piece_id=E7T6WL45F6GHECCRANX6AIKWFRZ77TURRWORYEYI43MET2WG7SRQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.357330644 (606h8m3.228413679s) [noise=true , address=77.90.31.21:4886 , piece_id=ZQRCPTVC3WFVQPZ36AU6KJW7PD3T557CXYQIICLDGOKK6JCPVZ5Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.357663649 (606h8m3.228746684s) [noise=true , address=145.79.227.49:4367 , piece_id=R2T2ABLXBPDVP3ZVA2TAYMCLQQ7JQUVQYCTHCKGMFBRVWCDAO4PQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.359827449 (606h8m3.230910484s) [noise=true , address=145.79.227.34:4918 , piece_id=HUMOHOT3HOYCBJ23WXQXWU3X6C6JQTRLLHTD4RQLCQTCB3HXI7EQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.360086692 (606h8m3.231169727s) [noise=true , address=145.79.227.7:4320 , piece_id=MAICNTHGLZ6W27HXDOOZXOBRGQGEXNSA4BJT2T5D5PHRVV6ZMX2Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.361622167 (606h8m3.232705212s) [noise=true , address=145.79.227.33:4888 , piece_id=I4426P3VETHPOL5WHWJEGVYRXJULTCNDX3WSUTS3XZVY36Q2N44Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.362965514 (606h8m3.234048559s) [noise=true , address=45.91.116.11:4133 , piece_id=NEW7ZOVP5E7HYGYD4LG5UG2MDLAQW2OBRTNZDFIYO4CQYMVAWRPQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.364308883 (606h8m3.235391928s) [noise=true , address=145.79.227.26:4663 , piece_id=DYEFQZL4X74QR2LAYURT5ZTR2IWIDB55VLBQOGZM36MA54P2SWJA, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.365734507 (606h8m3.236817542s) [noise=true , address=145.79.227.40:4059 , piece_id=7WSUBR3C3PWVBCWD6KRGSD6MPX4IH2YQOAVO4J4SWZORNHGS2IPQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.366982764 (606h8m3.238065799s) [noise=true , address=145.79.227.29:4973 , piece_id=5OROXELR55A47BFPSJJEWVQU4YW6CAOK6LMEHCINM4FN3OVI2Y6Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.368390294 (606h8m3.239473339s) [noise=true , address=77.90.31.19:4852 , piece_id=2NYO63GCBGCYBFYTMUGNF2NG5FSJOD3AOUOD2FZZIWDINK4X6DJQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:04.736295138 (606h8m4.607378183s) [upload=531240]: Next returned nil segment Dec 30 21:37:04.736311850 (606h8m4.607394885s) [upload=531240]: Waiting for error group managing segments... Dec 30 21:37:05.104989591 (606h8m4.976072636s) [noise=true , address=64.49.13.140:4507 , piece_id=E7T6WL45F6GHECCRANX6AIKWFRZ77TURRWORYEYI43MET2WG7SRQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.208809748 (606h8m5.079892793s) [noise=true , address=145.79.227.22:4543 , piece_id=WWCZ6SEEXQ2KUJ32AGTIJNMSJROLYSJ4XQGBQJC4Q5DACZVA65RQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.441125389 (606h8m5.312208444s) [noise=true , address=77.90.31.19:4852 , piece_id=2NYO63GCBGCYBFYTMUGNF2NG5FSJOD3AOUOD2FZZIWDINK4X6DJQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.483630904 (606h8m5.35471394s) [noise=true , address=77.90.31.12:4612 , piece_id=63TS3SUEFUE6PORZ4W76OXRWFL7SBN773UGGY3DU7JPBAW66N7UA, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.494478811 (606h8m5.365561846s) [noise=true , address=77.90.31.11:4559 , piece_id=3DYD4CNPJQN3S4EYZJGREQTTZJDSL2QTC3OT6G2VFUGMLSVCZUFA, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.497452683 (606h8m5.368535729s) [noise=true , address=77.90.31.17:4769 , piece_id=QXDYESBKGXRY7BCFYYF2O7OXS7Y4Z4KUMJB6AXZAWX735SYU56GQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.661529377 (606h8m5.532612412s) [noise=true , address=45.91.116.13:4182 , piece_id=BYAYNP4JAAN3AW2LYSK74WJE4BRWUR5Z6K67MTLK6A73KEJ35WEQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.729773112 (606h8m5.600856157s) [noise=true , address=45.91.116.11:4133 , piece_id=NEW7ZOVP5E7HYGYD4LG5UG2MDLAQW2OBRTNZDFIYO4CQYMVAWRPQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.940743308 (606h8m5.811826353s) [noise=true , address=77.90.31.21:4886 , piece_id=ZQRCPTVC3WFVQPZ36AU6KJW7PD3T557CXYQIICLDGOKK6JCPVZ5Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.084240044 (606h8m5.95532309s) [noise=true , address=145.79.227.30:4784 , piece_id=DSXUDNYCWO5F6IYXMIVJNEASYXIZ7EO7RKAS7QRUYK5OU7GOOB5Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.121095032 (606h8m5.992178067s) [noise=true , address=145.79.227.40:4059 , piece_id=7WSUBR3C3PWVBCWD6KRGSD6MPX4IH2YQOAVO4J4SWZORNHGS2IPQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.169195983 (606h8m6.040279028s) [noise=true , address=145.79.227.49:4367 , piece_id=R2T2ABLXBPDVP3ZVA2TAYMCLQQ7JQUVQYCTHCKGMFBRVWCDAO4PQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.169380634 (606h8m6.040463679s) [noise=true , address=145.79.227.33:4888 , piece_id=I4426P3VETHPOL5WHWJEGVYRXJULTCNDX3WSUTS3XZVY36Q2N44Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.179060175 (606h8m6.050143211s) [noise=true , address=145.79.227.29:4973 , piece_id=5OROXELR55A47BFPSJJEWVQU4YW6CAOK6LMEHCINM4FN3OVI2Y6Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.179080424 (606h8m6.050163459s) [seg_pos={0 0}, upload=531240]: Segment reached optimal threshold of 14 pieces. Dec 30 21:37:06.179262801 (606h8m6.050345837s) [noise=true , address=145.79.227.26:4663 , piece_id=DYEFQZL4X74QR2LAYURT5ZTR2IWIDB55VLBQOGZM36MA54P2SWJA, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SwpAaXTA3BqUvwubzmJkvwokdz6SSpT9t8HPtQcXXJxmcgfm2): protocol: write tcp 188.166.225.170:43184->145.79.227.26:4663: use of closed network connection; write tcp 188.166.225.170:43184->145.79.227.26:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43184->145.79.227.26:4663: use of closed network connection Dec 30 21:37:06.179268703 (606h8m6.050351748s) [noise=true , address=145.79.227.26:4663 , piece_id=DYEFQZL4X74QR2LAYURT5ZTR2IWIDB55VLBQOGZM36MA54P2SWJA, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179363724 (606h8m6.050446769s) [noise=true , address=145.79.227.7:4320 , piece_id=MAICNTHGLZ6W27HXDOOZXOBRGQGEXNSA4BJT2T5D5PHRVV6ZMX2Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126rEgnQw1w7hsFpRqQP9nLaumdBfdtkgYjdw6WE6nXrxk5kj4B): protocol: write tcp 188.166.225.170:45228->145.79.227.7:4320: use of closed network connection; write tcp 188.166.225.170:45228->145.79.227.7:4320: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45228->145.79.227.7:4320: use of closed network connection Dec 30 21:37:06.179369525 (606h8m6.05045257s) [noise=true , address=145.79.227.7:4320 , piece_id=MAICNTHGLZ6W27HXDOOZXOBRGQGEXNSA4BJT2T5D5PHRVV6ZMX2Q, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179532996 (606h8m6.050616061s) [noise=true , address=145.79.227.34:4918 , piece_id=HUMOHOT3HOYCBJ23WXQXWU3X6C6JQTRLLHTD4RQLCQTCB3HXI7EQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF Dec 30 21:37:06.179572371 (606h8m6.050655406s) [noise=true , address=145.79.227.34:4918 , piece_id=HUMOHOT3HOYCBJ23WXQXWU3X6C6JQTRLLHTD4RQLCQTCB3HXI7EQ, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179638587 (606h8m6.050721622s) [noise=true , address=145.79.227.35:4492 , piece_id=HCKFSQMJYEP6B5H4OX5AZ5TWP6Z4H4Z6WA2T7Z2H5KNZ4SDJRMUQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1roa5GGYmqCX2AxSrqFgqyV2CtBEYZavuGTseiwDy4pjkT6bip): protocol: expected piece hash; context canceled; EOF Dec 30 21:37:06.179647855 (606h8m6.05073089s) [noise=true , address=145.79.227.35:4492 , piece_id=HCKFSQMJYEP6B5H4OX5AZ5TWP6Z4H4Z6WA2T7Z2H5KNZ4SDJRMUQ, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179666641 (606h8m6.050749686s) [seg_pos={0 0}, upload=531240]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 21:37:06.179742766 (606h8m6.050825811s) [upload=531240]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 21:37:06.179747545 (606h8m6.05083059s) [upload=531240]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 21:37:06.490986034 (606h8m6.362069079s) [upload=531240]: Done uploading segments... Dec 30 21:37:06.491006261 (606h8m6.362089307s) [upload=531240]: Upload finished. err: <nil> Duration: 18.35334269s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................................................X.......] duration: 3.559298794s trace: https://jaeger.dev.storj.io/trace/22a47ec27f9a101c FAIL (2025-12-30 21:58:33 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.559323521s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [................XX......X.........X.........................] duration: 17.437928194s trace: https://jaeger.dev.storj.io/trace/5cdd5ba7b1cb22c7 FAIL (2025-12-30 21:58:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 21:37:03.355296401 (606h8m3.226379446s) [noise=true , address=145.79.227.22:4543 , piece_id=WWCZ6SEEXQ2KUJ32AGTIJNMSJROLYSJ4XQGBQJC4Q5DACZVA65RQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.355839374 (606h8m3.22692241s) [noise=true , address=145.79.227.35:4492 , piece_id=HCKFSQMJYEP6B5H4OX5AZ5TWP6Z4H4Z6WA2T7Z2H5KNZ4SDJRMUQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.356356930 (606h8m3.227439965s) [noise=true , address=45.91.116.13:4182 , piece_id=BYAYNP4JAAN3AW2LYSK74WJE4BRWUR5Z6K67MTLK6A73KEJ35WEQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.356675316 (606h8m3.227758361s) [noise=true , address=77.90.31.12:4612 , piece_id=63TS3SUEFUE6PORZ4W76OXRWFL7SBN773UGGY3DU7JPBAW66N7UA, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.356992732 (606h8m3.228075767s) [noise=true , address=64.49.13.140:4507 , piece_id=E7T6WL45F6GHECCRANX6AIKWFRZ77TURRWORYEYI43MET2WG7SRQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.357330644 (606h8m3.228413679s) [noise=true , address=77.90.31.21:4886 , piece_id=ZQRCPTVC3WFVQPZ36AU6KJW7PD3T557CXYQIICLDGOKK6JCPVZ5Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.357663649 (606h8m3.228746684s) [noise=true , address=145.79.227.49:4367 , piece_id=R2T2ABLXBPDVP3ZVA2TAYMCLQQ7JQUVQYCTHCKGMFBRVWCDAO4PQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.359827449 (606h8m3.230910484s) [noise=true , address=145.79.227.34:4918 , piece_id=HUMOHOT3HOYCBJ23WXQXWU3X6C6JQTRLLHTD4RQLCQTCB3HXI7EQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.360086692 (606h8m3.231169727s) [noise=true , address=145.79.227.7:4320 , piece_id=MAICNTHGLZ6W27HXDOOZXOBRGQGEXNSA4BJT2T5D5PHRVV6ZMX2Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.361622167 (606h8m3.232705212s) [noise=true , address=145.79.227.33:4888 , piece_id=I4426P3VETHPOL5WHWJEGVYRXJULTCNDX3WSUTS3XZVY36Q2N44Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.362965514 (606h8m3.234048559s) [noise=true , address=45.91.116.11:4133 , piece_id=NEW7ZOVP5E7HYGYD4LG5UG2MDLAQW2OBRTNZDFIYO4CQYMVAWRPQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.364308883 (606h8m3.235391928s) [noise=true , address=145.79.227.26:4663 , piece_id=DYEFQZL4X74QR2LAYURT5ZTR2IWIDB55VLBQOGZM36MA54P2SWJA, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.365734507 (606h8m3.236817542s) [noise=true , address=145.79.227.40:4059 , piece_id=7WSUBR3C3PWVBCWD6KRGSD6MPX4IH2YQOAVO4J4SWZORNHGS2IPQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.366982764 (606h8m3.238065799s) [noise=true , address=145.79.227.29:4973 , piece_id=5OROXELR55A47BFPSJJEWVQU4YW6CAOK6LMEHCINM4FN3OVI2Y6Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.368390294 (606h8m3.239473339s) [noise=true , address=77.90.31.19:4852 , piece_id=2NYO63GCBGCYBFYTMUGNF2NG5FSJOD3AOUOD2FZZIWDINK4X6DJQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:04.736295138 (606h8m4.607378183s) [upload=531240]: Next returned nil segment Dec 30 21:37:04.736311850 (606h8m4.607394885s) [upload=531240]: Waiting for error group managing segments... Dec 30 21:37:05.104989591 (606h8m4.976072636s) [noise=true , address=64.49.13.140:4507 , piece_id=E7T6WL45F6GHECCRANX6AIKWFRZ77TURRWORYEYI43MET2WG7SRQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.208809748 (606h8m5.079892793s) [noise=true , address=145.79.227.22:4543 , piece_id=WWCZ6SEEXQ2KUJ32AGTIJNMSJROLYSJ4XQGBQJC4Q5DACZVA65RQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.441125389 (606h8m5.312208444s) [noise=true , address=77.90.31.19:4852 , piece_id=2NYO63GCBGCYBFYTMUGNF2NG5FSJOD3AOUOD2FZZIWDINK4X6DJQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.483630904 (606h8m5.35471394s) [noise=true , address=77.90.31.12:4612 , piece_id=63TS3SUEFUE6PORZ4W76OXRWFL7SBN773UGGY3DU7JPBAW66N7UA, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.494478811 (606h8m5.365561846s) [noise=true , address=77.90.31.11:4559 , piece_id=3DYD4CNPJQN3S4EYZJGREQTTZJDSL2QTC3OT6G2VFUGMLSVCZUFA, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.497452683 (606h8m5.368535729s) [noise=true , address=77.90.31.17:4769 , piece_id=QXDYESBKGXRY7BCFYYF2O7OXS7Y4Z4KUMJB6AXZAWX735SYU56GQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.661529377 (606h8m5.532612412s) [noise=true , address=45.91.116.13:4182 , piece_id=BYAYNP4JAAN3AW2LYSK74WJE4BRWUR5Z6K67MTLK6A73KEJ35WEQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.729773112 (606h8m5.600856157s) [noise=true , address=45.91.116.11:4133 , piece_id=NEW7ZOVP5E7HYGYD4LG5UG2MDLAQW2OBRTNZDFIYO4CQYMVAWRPQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.940743308 (606h8m5.811826353s) [noise=true , address=77.90.31.21:4886 , piece_id=ZQRCPTVC3WFVQPZ36AU6KJW7PD3T557CXYQIICLDGOKK6JCPVZ5Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.084240044 (606h8m5.95532309s) [noise=true , address=145.79.227.30:4784 , piece_id=DSXUDNYCWO5F6IYXMIVJNEASYXIZ7EO7RKAS7QRUYK5OU7GOOB5Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.121095032 (606h8m5.992178067s) [noise=true , address=145.79.227.40:4059 , piece_id=7WSUBR3C3PWVBCWD6KRGSD6MPX4IH2YQOAVO4J4SWZORNHGS2IPQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.169195983 (606h8m6.040279028s) [noise=true , address=145.79.227.49:4367 , piece_id=R2T2ABLXBPDVP3ZVA2TAYMCLQQ7JQUVQYCTHCKGMFBRVWCDAO4PQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.169380634 (606h8m6.040463679s) [noise=true , address=145.79.227.33:4888 , piece_id=I4426P3VETHPOL5WHWJEGVYRXJULTCNDX3WSUTS3XZVY36Q2N44Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.179060175 (606h8m6.050143211s) [noise=true , address=145.79.227.29:4973 , piece_id=5OROXELR55A47BFPSJJEWVQU4YW6CAOK6LMEHCINM4FN3OVI2Y6Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.179080424 (606h8m6.050163459s) [seg_pos={0 0}, upload=531240]: Segment reached optimal threshold of 14 pieces. Dec 30 21:37:06.179262801 (606h8m6.050345837s) [noise=true , address=145.79.227.26:4663 , piece_id=DYEFQZL4X74QR2LAYURT5ZTR2IWIDB55VLBQOGZM36MA54P2SWJA, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SwpAaXTA3BqUvwubzmJkvwokdz6SSpT9t8HPtQcXXJxmcgfm2): protocol: write tcp 188.166.225.170:43184->145.79.227.26:4663: use of closed network connection; write tcp 188.166.225.170:43184->145.79.227.26:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43184->145.79.227.26:4663: use of closed network connection Dec 30 21:37:06.179268703 (606h8m6.050351748s) [noise=true , address=145.79.227.26:4663 , piece_id=DYEFQZL4X74QR2LAYURT5ZTR2IWIDB55VLBQOGZM36MA54P2SWJA, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179363724 (606h8m6.050446769s) [noise=true , address=145.79.227.7:4320 , piece_id=MAICNTHGLZ6W27HXDOOZXOBRGQGEXNSA4BJT2T5D5PHRVV6ZMX2Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126rEgnQw1w7hsFpRqQP9nLaumdBfdtkgYjdw6WE6nXrxk5kj4B): protocol: write tcp 188.166.225.170:45228->145.79.227.7:4320: use of closed network connection; write tcp 188.166.225.170:45228->145.79.227.7:4320: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45228->145.79.227.7:4320: use of closed network connection Dec 30 21:37:06.179369525 (606h8m6.05045257s) [noise=true , address=145.79.227.7:4320 , piece_id=MAICNTHGLZ6W27HXDOOZXOBRGQGEXNSA4BJT2T5D5PHRVV6ZMX2Q, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179532996 (606h8m6.050616061s) [noise=true , address=145.79.227.34:4918 , piece_id=HUMOHOT3HOYCBJ23WXQXWU3X6C6JQTRLLHTD4RQLCQTCB3HXI7EQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF Dec 30 21:37:06.179572371 (606h8m6.050655406s) [noise=true , address=145.79.227.34:4918 , piece_id=HUMOHOT3HOYCBJ23WXQXWU3X6C6JQTRLLHTD4RQLCQTCB3HXI7EQ, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179638587 (606h8m6.050721622s) [noise=true , address=145.79.227.35:4492 , piece_id=HCKFSQMJYEP6B5H4OX5AZ5TWP6Z4H4Z6WA2T7Z2H5KNZ4SDJRMUQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1roa5GGYmqCX2AxSrqFgqyV2CtBEYZavuGTseiwDy4pjkT6bip): protocol: expected piece hash; context canceled; EOF Dec 30 21:37:06.179647855 (606h8m6.05073089s) [noise=true , address=145.79.227.35:4492 , piece_id=HCKFSQMJYEP6B5H4OX5AZ5TWP6Z4H4Z6WA2T7Z2H5KNZ4SDJRMUQ, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179666641 (606h8m6.050749686s) [seg_pos={0 0}, upload=531240]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 21:37:06.179742766 (606h8m6.050825811s) [upload=531240]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 21:37:06.179747545 (606h8m6.05083059s) [upload=531240]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 21:37:06.490986034 (606h8m6.362069079s) [upload=531240]: Done uploading segments... Dec 30 21:37:06.491006261 (606h8m6.362089307s) [upload=531240]: Upload finished. err: <nil> Duration: 17.437958672s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [X....X.X.........X..X...X.........X.........................] duration: 6.68144201s trace: https://jaeger.dev.storj.io/trace/7b3713b537cd96b5 FAIL (2025-12-30 21:58:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: 3Y4O3DHH6VTTSFEDFLTGODSDJ2SINYTOVILJVKGHLEJRJAQGQRGQ) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: WPYVE54HEHASLRFMAHRNAGN4SBRLDDNNCLTUL6Q7IXCUTPKEQPJQ) context deadline exceeded ======LOGS====== Dec 30 21:37:24.517587467 (606h8m24.388670502s) [upload=531247]: Starting upload Dec 30 21:37:24.517683741 (606h8m24.388766776s) [upload=531247]: Done starting upload Dec 30 21:37:24.517750909 (606h8m24.388833964s) [upload=531247]: Uploading segments... Dec 30 21:37:24.517962091 (606h8m24.389045136s) [upload=531247]: Got next segment. Inline: false Dec 30 21:37:24.518006275 (606h8m24.38908932s) [upload=531247]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 21:37:24.518012496 (606h8m24.389095541s) [upload=531247]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 21:37:24.762005721 (606h8m24.633088796s) [seg_pos={0 0}, upload=531247]: Begin upload segment... Dec 30 21:37:24.762059043 (606h8m24.633142088s) [seg_pos={0 0}, upload=531247]: Done begin upload segment. Dec 30 21:37:24.762093669 (606h8m24.633176714s) [noise=true , address=145.79.227.13:4804 , piece_id=3T5EGEMHWDKHX3YMO46UAJLQLTXF72YYQROWVRGIJQMTXWWZTZUA, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.762105101 (606h8m24.633188136s) [noise=true , address=145.79.227.26:4654 , piece_id=YWMN73I2AGL5LMTBTDBR4REV35EPJ5FOC4FMQUMLHNYQBSM2R7YA, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.762162530 (606h8m24.633245575s) [noise=true , address=145.79.227.10:4403 , piece_id=NBFU5OK5RM4B5P53RMR6FH5XHHLYV3IMCAEEBF4DK7RQKF2YV56A, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.762806516 (606h8m24.633889581s) [noise=true , address=145.79.227.50:4374 , piece_id=4LK62BGBA6ZCAWL2K7JVJQN7VVZYVW7KHMFU6TOIAAXASEJQVP7Q, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.763247446 (606h8m24.634330491s) [noise=true , address=77.90.31.14:4662 , piece_id=3Y4O3DHH6VTTSFEDFLTGODSDJ2SINYTOVILJVKGHLEJRJAQGQRGQ, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.763647648 (606h8m24.634730683s) [noise=true , address=45.91.116.13:4155 , piece_id=WPYVE54HEHASLRFMAHRNAGN4SBRLDDNNCLTUL6Q7IXCUTPKEQPJQ, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.764127802 (606h8m24.635210847s) [noise=true , address=145.79.227.42:4121 , piece_id=M54EV7DTR4DH26KTXOQLG2IMQSRUKGOPVSXX2UEAHQGGGBLYCXWA, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.764542943 (606h8m24.635625978s) [noise=true , address=77.90.31.21:4880 , piece_id=MR3WNPHOPGXHD4GVIU4ONLKSPVYFPPWIINBKWIZWFENFVTZTSM3A, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.764959005 (606h8m24.63604205s) [noise=true , address=145.79.227.19:4079 , piece_id=W7EQPOU26CG3BPCWJGMTVV7KKDPOYA3HLYFT2HW2S4XJQZ6XUS5Q, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.766131597 (606h8m24.637214632s) [noise=true , address=45.91.116.10:4082 , piece_id=MYWKXEGVEDCGTIF26YFQTCLPLKXQUTFRE54AKLQ44PXV4EVR7ZHA, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.766690141 (606h8m24.637773176s) [noise=true , address=145.79.227.16:4911 , piece_id=ITFOWXPVEJU5SJX6A5EFQUMYFBCBV26EN4L3LA33AACJLJNQIPZQ, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.767199661 (606h8m24.638282696s) [noise=true , address=145.79.227.6:4258 , piece_id=LZPMJ6EZND6MF2MOMHUBNZ5SSJFYPJJYNW7UVQQEPYFEPLE5ZQCQ, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.767507377 (606h8m24.638590422s) [noise=true , address=145.79.227.37:4565 , piece_id=CNOZYEA5XVIFDAYZML66XRA3CVSUJJNWS3O75TIN4EZ7HTMCSNIQ, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.768056854 (606h8m24.639139899s) [noise=true , address=145.79.227.38:4979 , piece_id=DPTVSWBFBN2J3JO4QLRX4NJSDJE26AOW3QOSNXEDR6AFSVBDXCDA, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.768417160 (606h8m24.639500195s) [noise=true , address=145.79.227.40:4049 , piece_id=B54BGYWT6VSUUIGBVTAS6LHPYPEAK4LEX3HIGYLLBOPWONDBBYTA, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.768769200 (606h8m24.639852245s) [noise=true , address=145.79.227.24:4581 , piece_id=BYFXRVA7D4VCPXC3IGLN3T6GVHIIPB3MFICSXGDRKL2E5AUUW3WQ, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:26.469279868 (606h8m26.340362913s) [upload=531247]: Next returned nil segment Dec 30 21:37:26.469303293 (606h8m26.340386338s) [upload=531247]: Waiting for error group managing segments... Dec 30 21:37:26.756737396 (606h8m26.627820441s) [noise=true , address=145.79.227.10:4403 , piece_id=NBFU5OK5RM4B5P53RMR6FH5XHHLYV3IMCAEEBF4DK7RQKF2YV56A, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:26.768723167 (606h8m26.639806212s) [noise=true , address=145.79.227.42:4121 , piece_id=M54EV7DTR4DH26KTXOQLG2IMQSRUKGOPVSXX2UEAHQGGGBLYCXWA, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:26.768815023 (606h8m26.639898058s) [noise=true , address=145.79.227.38:4979 , piece_id=DPTVSWBFBN2J3JO4QLRX4NJSDJE26AOW3QOSNXEDR6AFSVBDXCDA, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:26.771265408 (606h8m26.642348453s) [noise=true , address=145.79.227.26:4654 , piece_id=YWMN73I2AGL5LMTBTDBR4REV35EPJ5FOC4FMQUMLHNYQBSM2R7YA, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:26.772261125 (606h8m26.64334416s) [noise=true , address=145.79.227.24:4581 , piece_id=BYFXRVA7D4VCPXC3IGLN3T6GVHIIPB3MFICSXGDRKL2E5AUUW3WQ, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:26.780818909 (606h8m26.651901954s) [noise=true , address=145.79.227.37:4565 , piece_id=CNOZYEA5XVIFDAYZML66XRA3CVSUJJNWS3O75TIN4EZ7HTMCSNIQ, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:26.781024501 (606h8m26.652107546s) [noise=true , address=145.79.227.50:4374 , piece_id=4LK62BGBA6ZCAWL2K7JVJQN7VVZYVW7KHMFU6TOIAAXASEJQVP7Q, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:26.781280018 (606h8m26.652363053s) [noise=true , address=145.79.227.13:4804 , piece_id=3T5EGEMHWDKHX3YMO46UAJLQLTXF72YYQROWVRGIJQMTXWWZTZUA, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:26.840740007 (606h8m26.711823042s) [noise=true , address=145.79.227.16:4911 , piece_id=ITFOWXPVEJU5SJX6A5EFQUMYFBCBV26EN4L3LA33AACJLJNQIPZQ, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:26.946038900 (606h8m26.817121936s) [noise=true , address=145.79.227.19:4079 , piece_id=W7EQPOU26CG3BPCWJGMTVV7KKDPOYA3HLYFT2HW2S4XJQZ6XUS5Q, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:27.027197388 (606h8m26.898280423s) [noise=true , address=77.90.31.14:4662 , piece_id=3Y4O3DHH6VTTSFEDFLTGODSDJ2SINYTOVILJVKGHLEJRJAQGQRGQ, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:27.065603629 (606h8m26.936686664s) [noise=true , address=45.91.116.10:4082 , piece_id=MYWKXEGVEDCGTIF26YFQTCLPLKXQUTFRE54AKLQ44PXV4EVR7ZHA, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:27.784121213 (606h8m27.655204248s) [noise=true , address=45.91.116.13:4155 , piece_id=WPYVE54HEHASLRFMAHRNAGN4SBRLDDNNCLTUL6Q7IXCUTPKEQPJQ, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:27.784142512 (606h8m27.655225558s) [seg_pos={0 0}, upload=531247]: Segment reached optimal threshold of 13 pieces. Dec 30 21:37:27.784300243 (606h8m27.655383278s) [noise=true , address=77.90.31.21:4880 , piece_id=MR3WNPHOPGXHD4GVIU4ONLKSPVYFPPWIINBKWIZWFENFVTZTSM3A, seg_pos={0 0}, upload=531247]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 30 21:37:27.784306134 (606h8m27.655389179s) [noise=true , address=77.90.31.21:4880 , piece_id=MR3WNPHOPGXHD4GVIU4ONLKSPVYFPPWIINBKWIZWFENFVTZTSM3A, seg_pos={0 0}, upload=531247]: Long tail cancellation due to optimal threshold Dec 30 21:37:27.784359966 (606h8m27.655443011s) [noise=true , address=145.79.227.40:4049 , piece_id=B54BGYWT6VSUUIGBVTAS6LHPYPEAK4LEX3HIGYLLBOPWONDBBYTA, seg_pos={0 0}, upload=531247]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 30 21:37:27.784370467 (606h8m27.655453522s) [noise=true , address=145.79.227.40:4049 , piece_id=B54BGYWT6VSUUIGBVTAS6LHPYPEAK4LEX3HIGYLLBOPWONDBBYTA, seg_pos={0 0}, upload=531247]: Long tail cancellation due to optimal threshold Dec 30 21:37:27.784429228 (606h8m27.655512263s) [noise=true , address=145.79.227.6:4258 , piece_id=LZPMJ6EZND6MF2MOMHUBNZ5SSJFYPJJYNW7UVQQEPYFEPLE5ZQCQ, seg_pos={0 0}, upload=531247]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF): protocol: write tcp 188.166.225.170:56840->145.79.227.6:4258: use of closed network connection; write tcp 188.166.225.170:56840->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56840->145.79.227.6:4258: use of closed network connection Dec 30 21:37:27.784436353 (606h8m27.655519398s) [noise=true , address=145.79.227.6:4258 , piece_id=LZPMJ6EZND6MF2MOMHUBNZ5SSJFYPJJYNW7UVQQEPYFEPLE5ZQCQ, seg_pos={0 0}, upload=531247]: Long tail cancellation due to optimal threshold Dec 30 21:37:27.784452513 (606h8m27.655535548s) [seg_pos={0 0}, upload=531247]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 21:37:27.784519210 (606h8m27.655602245s) [upload=531247]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 21:37:27.784523348 (606h8m27.655606393s) [upload=531247]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 21:37:28.083941079 (606h8m27.955024115s) [upload=531247]: Done uploading segments... Dec 30 21:37:28.083959936 (606h8m27.955042981s) [upload=531247]: Upload finished. err: <nil> Duration: 6.681473471s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.X....X.X.........X..X...X.........X........................] duration: 6.047020501s trace: https://jaeger.dev.storj.io/trace/7b3713b537cd96b5 FAIL (2025-12-30 21:57:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: 3Y4O3DHH6VTTSFEDFLTGODSDJ2SINYTOVILJVKGHLEJRJAQGQRGQ) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: WPYVE54HEHASLRFMAHRNAGN4SBRLDDNNCLTUL6Q7IXCUTPKEQPJQ) context deadline exceeded ======LOGS====== Dec 30 21:37:24.517587467 (606h8m24.388670502s) [upload=531247]: Starting upload Dec 30 21:37:24.517683741 (606h8m24.388766776s) [upload=531247]: Done starting upload Dec 30 21:37:24.517750909 (606h8m24.388833964s) [upload=531247]: Uploading segments... Dec 30 21:37:24.517962091 (606h8m24.389045136s) [upload=531247]: Got next segment. Inline: false Dec 30 21:37:24.518006275 (606h8m24.38908932s) [upload=531247]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 21:37:24.518012496 (606h8m24.389095541s) [upload=531247]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 21:37:24.762005721 (606h8m24.633088796s) [seg_pos={0 0}, upload=531247]: Begin upload segment... Dec 30 21:37:24.762059043 (606h8m24.633142088s) [seg_pos={0 0}, upload=531247]: Done begin upload segment. Dec 30 21:37:24.762093669 (606h8m24.633176714s) [noise=true , address=145.79.227.13:4804 , piece_id=3T5EGEMHWDKHX3YMO46UAJLQLTXF72YYQROWVRGIJQMTXWWZTZUA, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.762105101 (606h8m24.633188136s) [noise=true , address=145.79.227.26:4654 , piece_id=YWMN73I2AGL5LMTBTDBR4REV35EPJ5FOC4FMQUMLHNYQBSM2R7YA, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.762162530 (606h8m24.633245575s) [noise=true , address=145.79.227.10:4403 , piece_id=NBFU5OK5RM4B5P53RMR6FH5XHHLYV3IMCAEEBF4DK7RQKF2YV56A, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.762806516 (606h8m24.633889581s) [noise=true , address=145.79.227.50:4374 , piece_id=4LK62BGBA6ZCAWL2K7JVJQN7VVZYVW7KHMFU6TOIAAXASEJQVP7Q, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.763247446 (606h8m24.634330491s) [noise=true , address=77.90.31.14:4662 , piece_id=3Y4O3DHH6VTTSFEDFLTGODSDJ2SINYTOVILJVKGHLEJRJAQGQRGQ, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.763647648 (606h8m24.634730683s) [noise=true , address=45.91.116.13:4155 , piece_id=WPYVE54HEHASLRFMAHRNAGN4SBRLDDNNCLTUL6Q7IXCUTPKEQPJQ, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.764127802 (606h8m24.635210847s) [noise=true , address=145.79.227.42:4121 , piece_id=M54EV7DTR4DH26KTXOQLG2IMQSRUKGOPVSXX2UEAHQGGGBLYCXWA, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.764542943 (606h8m24.635625978s) [noise=true , address=77.90.31.21:4880 , piece_id=MR3WNPHOPGXHD4GVIU4ONLKSPVYFPPWIINBKWIZWFENFVTZTSM3A, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.764959005 (606h8m24.63604205s) [noise=true , address=145.79.227.19:4079 , piece_id=W7EQPOU26CG3BPCWJGMTVV7KKDPOYA3HLYFT2HW2S4XJQZ6XUS5Q, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.766131597 (606h8m24.637214632s) [noise=true , address=45.91.116.10:4082 , piece_id=MYWKXEGVEDCGTIF26YFQTCLPLKXQUTFRE54AKLQ44PXV4EVR7ZHA, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.766690141 (606h8m24.637773176s) [noise=true , address=145.79.227.16:4911 , piece_id=ITFOWXPVEJU5SJX6A5EFQUMYFBCBV26EN4L3LA33AACJLJNQIPZQ, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.767199661 (606h8m24.638282696s) [noise=true , address=145.79.227.6:4258 , piece_id=LZPMJ6EZND6MF2MOMHUBNZ5SSJFYPJJYNW7UVQQEPYFEPLE5ZQCQ, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.767507377 (606h8m24.638590422s) [noise=true , address=145.79.227.37:4565 , piece_id=CNOZYEA5XVIFDAYZML66XRA3CVSUJJNWS3O75TIN4EZ7HTMCSNIQ, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.768056854 (606h8m24.639139899s) [noise=true , address=145.79.227.38:4979 , piece_id=DPTVSWBFBN2J3JO4QLRX4NJSDJE26AOW3QOSNXEDR6AFSVBDXCDA, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.768417160 (606h8m24.639500195s) [noise=true , address=145.79.227.40:4049 , piece_id=B54BGYWT6VSUUIGBVTAS6LHPYPEAK4LEX3HIGYLLBOPWONDBBYTA, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.768769200 (606h8m24.639852245s) [noise=true , address=145.79.227.24:4581 , piece_id=BYFXRVA7D4VCPXC3IGLN3T6GVHIIPB3MFICSXGDRKL2E5AUUW3WQ, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:26.469279868 (606h8m26.340362913s) [upload=531247]: Next returned nil segment Dec 30 21:37:26.469303293 (606h8m26.340386338s) [upload=531247]: Waiting for error group managing segments... Dec 30 21:37:26.756737396 (606h8m26.627820441s) [noise=true , address=145.79.227.10:4403 , piece_id=NBFU5OK5RM4B5P53RMR6FH5XHHLYV3IMCAEEBF4DK7RQKF2YV56A, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:26.768723167 (606h8m26.639806212s) [noise=true , address=145.79.227.42:4121 , piece_id=M54EV7DTR4DH26KTXOQLG2IMQSRUKGOPVSXX2UEAHQGGGBLYCXWA, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:26.768815023 (606h8m26.639898058s) [noise=true , address=145.79.227.38:4979 , piece_id=DPTVSWBFBN2J3JO4QLRX4NJSDJE26AOW3QOSNXEDR6AFSVBDXCDA, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:26.771265408 (606h8m26.642348453s) [noise=true , address=145.79.227.26:4654 , piece_id=YWMN73I2AGL5LMTBTDBR4REV35EPJ5FOC4FMQUMLHNYQBSM2R7YA, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:26.772261125 (606h8m26.64334416s) [noise=true , address=145.79.227.24:4581 , piece_id=BYFXRVA7D4VCPXC3IGLN3T6GVHIIPB3MFICSXGDRKL2E5AUUW3WQ, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:26.780818909 (606h8m26.651901954s) [noise=true , address=145.79.227.37:4565 , piece_id=CNOZYEA5XVIFDAYZML66XRA3CVSUJJNWS3O75TIN4EZ7HTMCSNIQ, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:26.781024501 (606h8m26.652107546s) [noise=true , address=145.79.227.50:4374 , piece_id=4LK62BGBA6ZCAWL2K7JVJQN7VVZYVW7KHMFU6TOIAAXASEJQVP7Q, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:26.781280018 (606h8m26.652363053s) [noise=true , address=145.79.227.13:4804 , piece_id=3T5EGEMHWDKHX3YMO46UAJLQLTXF72YYQROWVRGIJQMTXWWZTZUA, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:26.840740007 (606h8m26.711823042s) [noise=true , address=145.79.227.16:4911 , piece_id=ITFOWXPVEJU5SJX6A5EFQUMYFBCBV26EN4L3LA33AACJLJNQIPZQ, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:26.946038900 (606h8m26.817121936s) [noise=true , address=145.79.227.19:4079 , piece_id=W7EQPOU26CG3BPCWJGMTVV7KKDPOYA3HLYFT2HW2S4XJQZ6XUS5Q, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:27.027197388 (606h8m26.898280423s) [noise=true , address=77.90.31.14:4662 , piece_id=3Y4O3DHH6VTTSFEDFLTGODSDJ2SINYTOVILJVKGHLEJRJAQGQRGQ, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:27.065603629 (606h8m26.936686664s) [noise=true , address=45.91.116.10:4082 , piece_id=MYWKXEGVEDCGTIF26YFQTCLPLKXQUTFRE54AKLQ44PXV4EVR7ZHA, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:27.784121213 (606h8m27.655204248s) [noise=true , address=45.91.116.13:4155 , piece_id=WPYVE54HEHASLRFMAHRNAGN4SBRLDDNNCLTUL6Q7IXCUTPKEQPJQ, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:27.784142512 (606h8m27.655225558s) [seg_pos={0 0}, upload=531247]: Segment reached optimal threshold of 13 pieces. Dec 30 21:37:27.784300243 (606h8m27.655383278s) [noise=true , address=77.90.31.21:4880 , piece_id=MR3WNPHOPGXHD4GVIU4ONLKSPVYFPPWIINBKWIZWFENFVTZTSM3A, seg_pos={0 0}, upload=531247]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 30 21:37:27.784306134 (606h8m27.655389179s) [noise=true , address=77.90.31.21:4880 , piece_id=MR3WNPHOPGXHD4GVIU4ONLKSPVYFPPWIINBKWIZWFENFVTZTSM3A, seg_pos={0 0}, upload=531247]: Long tail cancellation due to optimal threshold Dec 30 21:37:27.784359966 (606h8m27.655443011s) [noise=true , address=145.79.227.40:4049 , piece_id=B54BGYWT6VSUUIGBVTAS6LHPYPEAK4LEX3HIGYLLBOPWONDBBYTA, seg_pos={0 0}, upload=531247]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 30 21:37:27.784370467 (606h8m27.655453522s) [noise=true , address=145.79.227.40:4049 , piece_id=B54BGYWT6VSUUIGBVTAS6LHPYPEAK4LEX3HIGYLLBOPWONDBBYTA, seg_pos={0 0}, upload=531247]: Long tail cancellation due to optimal threshold Dec 30 21:37:27.784429228 (606h8m27.655512263s) [noise=true , address=145.79.227.6:4258 , piece_id=LZPMJ6EZND6MF2MOMHUBNZ5SSJFYPJJYNW7UVQQEPYFEPLE5ZQCQ, seg_pos={0 0}, upload=531247]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF): protocol: write tcp 188.166.225.170:56840->145.79.227.6:4258: use of closed network connection; write tcp 188.166.225.170:56840->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56840->145.79.227.6:4258: use of closed network connection Dec 30 21:37:27.784436353 (606h8m27.655519398s) [noise=true , address=145.79.227.6:4258 , piece_id=LZPMJ6EZND6MF2MOMHUBNZ5SSJFYPJJYNW7UVQQEPYFEPLE5ZQCQ, seg_pos={0 0}, upload=531247]: Long tail cancellation due to optimal threshold Dec 30 21:37:27.784452513 (606h8m27.655535548s) [seg_pos={0 0}, upload=531247]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 21:37:27.784519210 (606h8m27.655602245s) [upload=531247]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 21:37:27.784523348 (606h8m27.655606393s) [upload=531247]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 21:37:28.083941079 (606h8m27.955024115s) [upload=531247]: Done uploading segments... Dec 30 21:37:28.083959936 (606h8m27.955042981s) [upload=531247]: Upload finished. err: <nil> Duration: 6.047048675s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.................XX......X.........X........................] duration: 15.216279696s trace: https://jaeger.dev.storj.io/trace/5cdd5ba7b1cb22c7 FAIL (2025-12-30 21:57:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 21:37:03.355296401 (606h8m3.226379446s) [noise=true , address=145.79.227.22:4543 , piece_id=WWCZ6SEEXQ2KUJ32AGTIJNMSJROLYSJ4XQGBQJC4Q5DACZVA65RQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.355839374 (606h8m3.22692241s) [noise=true , address=145.79.227.35:4492 , piece_id=HCKFSQMJYEP6B5H4OX5AZ5TWP6Z4H4Z6WA2T7Z2H5KNZ4SDJRMUQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.356356930 (606h8m3.227439965s) [noise=true , address=45.91.116.13:4182 , piece_id=BYAYNP4JAAN3AW2LYSK74WJE4BRWUR5Z6K67MTLK6A73KEJ35WEQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.356675316 (606h8m3.227758361s) [noise=true , address=77.90.31.12:4612 , piece_id=63TS3SUEFUE6PORZ4W76OXRWFL7SBN773UGGY3DU7JPBAW66N7UA, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.356992732 (606h8m3.228075767s) [noise=true , address=64.49.13.140:4507 , piece_id=E7T6WL45F6GHECCRANX6AIKWFRZ77TURRWORYEYI43MET2WG7SRQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.357330644 (606h8m3.228413679s) [noise=true , address=77.90.31.21:4886 , piece_id=ZQRCPTVC3WFVQPZ36AU6KJW7PD3T557CXYQIICLDGOKK6JCPVZ5Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.357663649 (606h8m3.228746684s) [noise=true , address=145.79.227.49:4367 , piece_id=R2T2ABLXBPDVP3ZVA2TAYMCLQQ7JQUVQYCTHCKGMFBRVWCDAO4PQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.359827449 (606h8m3.230910484s) [noise=true , address=145.79.227.34:4918 , piece_id=HUMOHOT3HOYCBJ23WXQXWU3X6C6JQTRLLHTD4RQLCQTCB3HXI7EQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.360086692 (606h8m3.231169727s) [noise=true , address=145.79.227.7:4320 , piece_id=MAICNTHGLZ6W27HXDOOZXOBRGQGEXNSA4BJT2T5D5PHRVV6ZMX2Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.361622167 (606h8m3.232705212s) [noise=true , address=145.79.227.33:4888 , piece_id=I4426P3VETHPOL5WHWJEGVYRXJULTCNDX3WSUTS3XZVY36Q2N44Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.362965514 (606h8m3.234048559s) [noise=true , address=45.91.116.11:4133 , piece_id=NEW7ZOVP5E7HYGYD4LG5UG2MDLAQW2OBRTNZDFIYO4CQYMVAWRPQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.364308883 (606h8m3.235391928s) [noise=true , address=145.79.227.26:4663 , piece_id=DYEFQZL4X74QR2LAYURT5ZTR2IWIDB55VLBQOGZM36MA54P2SWJA, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.365734507 (606h8m3.236817542s) [noise=true , address=145.79.227.40:4059 , piece_id=7WSUBR3C3PWVBCWD6KRGSD6MPX4IH2YQOAVO4J4SWZORNHGS2IPQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.366982764 (606h8m3.238065799s) [noise=true , address=145.79.227.29:4973 , piece_id=5OROXELR55A47BFPSJJEWVQU4YW6CAOK6LMEHCINM4FN3OVI2Y6Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.368390294 (606h8m3.239473339s) [noise=true , address=77.90.31.19:4852 , piece_id=2NYO63GCBGCYBFYTMUGNF2NG5FSJOD3AOUOD2FZZIWDINK4X6DJQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:04.736295138 (606h8m4.607378183s) [upload=531240]: Next returned nil segment Dec 30 21:37:04.736311850 (606h8m4.607394885s) [upload=531240]: Waiting for error group managing segments... Dec 30 21:37:05.104989591 (606h8m4.976072636s) [noise=true , address=64.49.13.140:4507 , piece_id=E7T6WL45F6GHECCRANX6AIKWFRZ77TURRWORYEYI43MET2WG7SRQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.208809748 (606h8m5.079892793s) [noise=true , address=145.79.227.22:4543 , piece_id=WWCZ6SEEXQ2KUJ32AGTIJNMSJROLYSJ4XQGBQJC4Q5DACZVA65RQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.441125389 (606h8m5.312208444s) [noise=true , address=77.90.31.19:4852 , piece_id=2NYO63GCBGCYBFYTMUGNF2NG5FSJOD3AOUOD2FZZIWDINK4X6DJQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.483630904 (606h8m5.35471394s) [noise=true , address=77.90.31.12:4612 , piece_id=63TS3SUEFUE6PORZ4W76OXRWFL7SBN773UGGY3DU7JPBAW66N7UA, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.494478811 (606h8m5.365561846s) [noise=true , address=77.90.31.11:4559 , piece_id=3DYD4CNPJQN3S4EYZJGREQTTZJDSL2QTC3OT6G2VFUGMLSVCZUFA, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.497452683 (606h8m5.368535729s) [noise=true , address=77.90.31.17:4769 , piece_id=QXDYESBKGXRY7BCFYYF2O7OXS7Y4Z4KUMJB6AXZAWX735SYU56GQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.661529377 (606h8m5.532612412s) [noise=true , address=45.91.116.13:4182 , piece_id=BYAYNP4JAAN3AW2LYSK74WJE4BRWUR5Z6K67MTLK6A73KEJ35WEQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.729773112 (606h8m5.600856157s) [noise=true , address=45.91.116.11:4133 , piece_id=NEW7ZOVP5E7HYGYD4LG5UG2MDLAQW2OBRTNZDFIYO4CQYMVAWRPQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.940743308 (606h8m5.811826353s) [noise=true , address=77.90.31.21:4886 , piece_id=ZQRCPTVC3WFVQPZ36AU6KJW7PD3T557CXYQIICLDGOKK6JCPVZ5Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.084240044 (606h8m5.95532309s) [noise=true , address=145.79.227.30:4784 , piece_id=DSXUDNYCWO5F6IYXMIVJNEASYXIZ7EO7RKAS7QRUYK5OU7GOOB5Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.121095032 (606h8m5.992178067s) [noise=true , address=145.79.227.40:4059 , piece_id=7WSUBR3C3PWVBCWD6KRGSD6MPX4IH2YQOAVO4J4SWZORNHGS2IPQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.169195983 (606h8m6.040279028s) [noise=true , address=145.79.227.49:4367 , piece_id=R2T2ABLXBPDVP3ZVA2TAYMCLQQ7JQUVQYCTHCKGMFBRVWCDAO4PQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.169380634 (606h8m6.040463679s) [noise=true , address=145.79.227.33:4888 , piece_id=I4426P3VETHPOL5WHWJEGVYRXJULTCNDX3WSUTS3XZVY36Q2N44Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.179060175 (606h8m6.050143211s) [noise=true , address=145.79.227.29:4973 , piece_id=5OROXELR55A47BFPSJJEWVQU4YW6CAOK6LMEHCINM4FN3OVI2Y6Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.179080424 (606h8m6.050163459s) [seg_pos={0 0}, upload=531240]: Segment reached optimal threshold of 14 pieces. Dec 30 21:37:06.179262801 (606h8m6.050345837s) [noise=true , address=145.79.227.26:4663 , piece_id=DYEFQZL4X74QR2LAYURT5ZTR2IWIDB55VLBQOGZM36MA54P2SWJA, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SwpAaXTA3BqUvwubzmJkvwokdz6SSpT9t8HPtQcXXJxmcgfm2): protocol: write tcp 188.166.225.170:43184->145.79.227.26:4663: use of closed network connection; write tcp 188.166.225.170:43184->145.79.227.26:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43184->145.79.227.26:4663: use of closed network connection Dec 30 21:37:06.179268703 (606h8m6.050351748s) [noise=true , address=145.79.227.26:4663 , piece_id=DYEFQZL4X74QR2LAYURT5ZTR2IWIDB55VLBQOGZM36MA54P2SWJA, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179363724 (606h8m6.050446769s) [noise=true , address=145.79.227.7:4320 , piece_id=MAICNTHGLZ6W27HXDOOZXOBRGQGEXNSA4BJT2T5D5PHRVV6ZMX2Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126rEgnQw1w7hsFpRqQP9nLaumdBfdtkgYjdw6WE6nXrxk5kj4B): protocol: write tcp 188.166.225.170:45228->145.79.227.7:4320: use of closed network connection; write tcp 188.166.225.170:45228->145.79.227.7:4320: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45228->145.79.227.7:4320: use of closed network connection Dec 30 21:37:06.179369525 (606h8m6.05045257s) [noise=true , address=145.79.227.7:4320 , piece_id=MAICNTHGLZ6W27HXDOOZXOBRGQGEXNSA4BJT2T5D5PHRVV6ZMX2Q, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179532996 (606h8m6.050616061s) [noise=true , address=145.79.227.34:4918 , piece_id=HUMOHOT3HOYCBJ23WXQXWU3X6C6JQTRLLHTD4RQLCQTCB3HXI7EQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF Dec 30 21:37:06.179572371 (606h8m6.050655406s) [noise=true , address=145.79.227.34:4918 , piece_id=HUMOHOT3HOYCBJ23WXQXWU3X6C6JQTRLLHTD4RQLCQTCB3HXI7EQ, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179638587 (606h8m6.050721622s) [noise=true , address=145.79.227.35:4492 , piece_id=HCKFSQMJYEP6B5H4OX5AZ5TWP6Z4H4Z6WA2T7Z2H5KNZ4SDJRMUQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1roa5GGYmqCX2AxSrqFgqyV2CtBEYZavuGTseiwDy4pjkT6bip): protocol: expected piece hash; context canceled; EOF Dec 30 21:37:06.179647855 (606h8m6.05073089s) [noise=true , address=145.79.227.35:4492 , piece_id=HCKFSQMJYEP6B5H4OX5AZ5TWP6Z4H4Z6WA2T7Z2H5KNZ4SDJRMUQ, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179666641 (606h8m6.050749686s) [seg_pos={0 0}, upload=531240]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 21:37:06.179742766 (606h8m6.050825811s) [upload=531240]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 21:37:06.179747545 (606h8m6.05083059s) [upload=531240]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 21:37:06.490986034 (606h8m6.362069079s) [upload=531240]: Done uploading segments... Dec 30 21:37:06.491006261 (606h8m6.362089307s) [upload=531240]: Upload finished. err: <nil> Duration: 15.216315453s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................................................X......] duration: 3.596230285s trace: https://jaeger.dev.storj.io/trace/22a47ec27f9a101c FAIL (2025-12-30 21:57:20 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.596251724s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [..X....X.X.........X..X...X.........X.......................] duration: 6.01047471s trace: https://jaeger.dev.storj.io/trace/7b3713b537cd96b5 FAIL (2025-12-30 21:56:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: 3Y4O3DHH6VTTSFEDFLTGODSDJ2SINYTOVILJVKGHLEJRJAQGQRGQ) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: WPYVE54HEHASLRFMAHRNAGN4SBRLDDNNCLTUL6Q7IXCUTPKEQPJQ) context deadline exceeded ======LOGS====== Dec 30 21:37:24.517587467 (606h8m24.388670502s) [upload=531247]: Starting upload Dec 30 21:37:24.517683741 (606h8m24.388766776s) [upload=531247]: Done starting upload Dec 30 21:37:24.517750909 (606h8m24.388833964s) [upload=531247]: Uploading segments... Dec 30 21:37:24.517962091 (606h8m24.389045136s) [upload=531247]: Got next segment. Inline: false Dec 30 21:37:24.518006275 (606h8m24.38908932s) [upload=531247]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 21:37:24.518012496 (606h8m24.389095541s) [upload=531247]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 21:37:24.762005721 (606h8m24.633088796s) [seg_pos={0 0}, upload=531247]: Begin upload segment... Dec 30 21:37:24.762059043 (606h8m24.633142088s) [seg_pos={0 0}, upload=531247]: Done begin upload segment. Dec 30 21:37:24.762093669 (606h8m24.633176714s) [noise=true , address=145.79.227.13:4804 , piece_id=3T5EGEMHWDKHX3YMO46UAJLQLTXF72YYQROWVRGIJQMTXWWZTZUA, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.762105101 (606h8m24.633188136s) [noise=true , address=145.79.227.26:4654 , piece_id=YWMN73I2AGL5LMTBTDBR4REV35EPJ5FOC4FMQUMLHNYQBSM2R7YA, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.762162530 (606h8m24.633245575s) [noise=true , address=145.79.227.10:4403 , piece_id=NBFU5OK5RM4B5P53RMR6FH5XHHLYV3IMCAEEBF4DK7RQKF2YV56A, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.762806516 (606h8m24.633889581s) [noise=true , address=145.79.227.50:4374 , piece_id=4LK62BGBA6ZCAWL2K7JVJQN7VVZYVW7KHMFU6TOIAAXASEJQVP7Q, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.763247446 (606h8m24.634330491s) [noise=true , address=77.90.31.14:4662 , piece_id=3Y4O3DHH6VTTSFEDFLTGODSDJ2SINYTOVILJVKGHLEJRJAQGQRGQ, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.763647648 (606h8m24.634730683s) [noise=true , address=45.91.116.13:4155 , piece_id=WPYVE54HEHASLRFMAHRNAGN4SBRLDDNNCLTUL6Q7IXCUTPKEQPJQ, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.764127802 (606h8m24.635210847s) [noise=true , address=145.79.227.42:4121 , piece_id=M54EV7DTR4DH26KTXOQLG2IMQSRUKGOPVSXX2UEAHQGGGBLYCXWA, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.764542943 (606h8m24.635625978s) [noise=true , address=77.90.31.21:4880 , piece_id=MR3WNPHOPGXHD4GVIU4ONLKSPVYFPPWIINBKWIZWFENFVTZTSM3A, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.764959005 (606h8m24.63604205s) [noise=true , address=145.79.227.19:4079 , piece_id=W7EQPOU26CG3BPCWJGMTVV7KKDPOYA3HLYFT2HW2S4XJQZ6XUS5Q, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.766131597 (606h8m24.637214632s) [noise=true , address=45.91.116.10:4082 , piece_id=MYWKXEGVEDCGTIF26YFQTCLPLKXQUTFRE54AKLQ44PXV4EVR7ZHA, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.766690141 (606h8m24.637773176s) [noise=true , address=145.79.227.16:4911 , piece_id=ITFOWXPVEJU5SJX6A5EFQUMYFBCBV26EN4L3LA33AACJLJNQIPZQ, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.767199661 (606h8m24.638282696s) [noise=true , address=145.79.227.6:4258 , piece_id=LZPMJ6EZND6MF2MOMHUBNZ5SSJFYPJJYNW7UVQQEPYFEPLE5ZQCQ, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.767507377 (606h8m24.638590422s) [noise=true , address=145.79.227.37:4565 , piece_id=CNOZYEA5XVIFDAYZML66XRA3CVSUJJNWS3O75TIN4EZ7HTMCSNIQ, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.768056854 (606h8m24.639139899s) [noise=true , address=145.79.227.38:4979 , piece_id=DPTVSWBFBN2J3JO4QLRX4NJSDJE26AOW3QOSNXEDR6AFSVBDXCDA, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.768417160 (606h8m24.639500195s) [noise=true , address=145.79.227.40:4049 , piece_id=B54BGYWT6VSUUIGBVTAS6LHPYPEAK4LEX3HIGYLLBOPWONDBBYTA, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.768769200 (606h8m24.639852245s) [noise=true , address=145.79.227.24:4581 , piece_id=BYFXRVA7D4VCPXC3IGLN3T6GVHIIPB3MFICSXGDRKL2E5AUUW3WQ, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:26.469279868 (606h8m26.340362913s) [upload=531247]: Next returned nil segment Dec 30 21:37:26.469303293 (606h8m26.340386338s) [upload=531247]: Waiting for error group managing segments... Dec 30 21:37:26.756737396 (606h8m26.627820441s) [noise=true , address=145.79.227.10:4403 , piece_id=NBFU5OK5RM4B5P53RMR6FH5XHHLYV3IMCAEEBF4DK7RQKF2YV56A, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:26.768723167 (606h8m26.639806212s) [noise=true , address=145.79.227.42:4121 , piece_id=M54EV7DTR4DH26KTXOQLG2IMQSRUKGOPVSXX2UEAHQGGGBLYCXWA, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:26.768815023 (606h8m26.639898058s) [noise=true , address=145.79.227.38:4979 , piece_id=DPTVSWBFBN2J3JO4QLRX4NJSDJE26AOW3QOSNXEDR6AFSVBDXCDA, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:26.771265408 (606h8m26.642348453s) [noise=true , address=145.79.227.26:4654 , piece_id=YWMN73I2AGL5LMTBTDBR4REV35EPJ5FOC4FMQUMLHNYQBSM2R7YA, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:26.772261125 (606h8m26.64334416s) [noise=true , address=145.79.227.24:4581 , piece_id=BYFXRVA7D4VCPXC3IGLN3T6GVHIIPB3MFICSXGDRKL2E5AUUW3WQ, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:26.780818909 (606h8m26.651901954s) [noise=true , address=145.79.227.37:4565 , piece_id=CNOZYEA5XVIFDAYZML66XRA3CVSUJJNWS3O75TIN4EZ7HTMCSNIQ, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:26.781024501 (606h8m26.652107546s) [noise=true , address=145.79.227.50:4374 , piece_id=4LK62BGBA6ZCAWL2K7JVJQN7VVZYVW7KHMFU6TOIAAXASEJQVP7Q, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:26.781280018 (606h8m26.652363053s) [noise=true , address=145.79.227.13:4804 , piece_id=3T5EGEMHWDKHX3YMO46UAJLQLTXF72YYQROWVRGIJQMTXWWZTZUA, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:26.840740007 (606h8m26.711823042s) [noise=true , address=145.79.227.16:4911 , piece_id=ITFOWXPVEJU5SJX6A5EFQUMYFBCBV26EN4L3LA33AACJLJNQIPZQ, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:26.946038900 (606h8m26.817121936s) [noise=true , address=145.79.227.19:4079 , piece_id=W7EQPOU26CG3BPCWJGMTVV7KKDPOYA3HLYFT2HW2S4XJQZ6XUS5Q, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:27.027197388 (606h8m26.898280423s) [noise=true , address=77.90.31.14:4662 , piece_id=3Y4O3DHH6VTTSFEDFLTGODSDJ2SINYTOVILJVKGHLEJRJAQGQRGQ, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:27.065603629 (606h8m26.936686664s) [noise=true , address=45.91.116.10:4082 , piece_id=MYWKXEGVEDCGTIF26YFQTCLPLKXQUTFRE54AKLQ44PXV4EVR7ZHA, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:27.784121213 (606h8m27.655204248s) [noise=true , address=45.91.116.13:4155 , piece_id=WPYVE54HEHASLRFMAHRNAGN4SBRLDDNNCLTUL6Q7IXCUTPKEQPJQ, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:27.784142512 (606h8m27.655225558s) [seg_pos={0 0}, upload=531247]: Segment reached optimal threshold of 13 pieces. Dec 30 21:37:27.784300243 (606h8m27.655383278s) [noise=true , address=77.90.31.21:4880 , piece_id=MR3WNPHOPGXHD4GVIU4ONLKSPVYFPPWIINBKWIZWFENFVTZTSM3A, seg_pos={0 0}, upload=531247]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 30 21:37:27.784306134 (606h8m27.655389179s) [noise=true , address=77.90.31.21:4880 , piece_id=MR3WNPHOPGXHD4GVIU4ONLKSPVYFPPWIINBKWIZWFENFVTZTSM3A, seg_pos={0 0}, upload=531247]: Long tail cancellation due to optimal threshold Dec 30 21:37:27.784359966 (606h8m27.655443011s) [noise=true , address=145.79.227.40:4049 , piece_id=B54BGYWT6VSUUIGBVTAS6LHPYPEAK4LEX3HIGYLLBOPWONDBBYTA, seg_pos={0 0}, upload=531247]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 30 21:37:27.784370467 (606h8m27.655453522s) [noise=true , address=145.79.227.40:4049 , piece_id=B54BGYWT6VSUUIGBVTAS6LHPYPEAK4LEX3HIGYLLBOPWONDBBYTA, seg_pos={0 0}, upload=531247]: Long tail cancellation due to optimal threshold Dec 30 21:37:27.784429228 (606h8m27.655512263s) [noise=true , address=145.79.227.6:4258 , piece_id=LZPMJ6EZND6MF2MOMHUBNZ5SSJFYPJJYNW7UVQQEPYFEPLE5ZQCQ, seg_pos={0 0}, upload=531247]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF): protocol: write tcp 188.166.225.170:56840->145.79.227.6:4258: use of closed network connection; write tcp 188.166.225.170:56840->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56840->145.79.227.6:4258: use of closed network connection Dec 30 21:37:27.784436353 (606h8m27.655519398s) [noise=true , address=145.79.227.6:4258 , piece_id=LZPMJ6EZND6MF2MOMHUBNZ5SSJFYPJJYNW7UVQQEPYFEPLE5ZQCQ, seg_pos={0 0}, upload=531247]: Long tail cancellation due to optimal threshold Dec 30 21:37:27.784452513 (606h8m27.655535548s) [seg_pos={0 0}, upload=531247]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 21:37:27.784519210 (606h8m27.655602245s) [upload=531247]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 21:37:27.784523348 (606h8m27.655606393s) [upload=531247]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 21:37:28.083941079 (606h8m27.955024115s) [upload=531247]: Done uploading segments... Dec 30 21:37:28.083959936 (606h8m27.955042981s) [upload=531247]: Upload finished. err: <nil> Duration: 6.010503104s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [..................XX......X.........X.......................] duration: 15.363001777s trace: https://jaeger.dev.storj.io/trace/5cdd5ba7b1cb22c7 FAIL (2025-12-30 21:56:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 21:37:03.355296401 (606h8m3.226379446s) [noise=true , address=145.79.227.22:4543 , piece_id=WWCZ6SEEXQ2KUJ32AGTIJNMSJROLYSJ4XQGBQJC4Q5DACZVA65RQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.355839374 (606h8m3.22692241s) [noise=true , address=145.79.227.35:4492 , piece_id=HCKFSQMJYEP6B5H4OX5AZ5TWP6Z4H4Z6WA2T7Z2H5KNZ4SDJRMUQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.356356930 (606h8m3.227439965s) [noise=true , address=45.91.116.13:4182 , piece_id=BYAYNP4JAAN3AW2LYSK74WJE4BRWUR5Z6K67MTLK6A73KEJ35WEQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.356675316 (606h8m3.227758361s) [noise=true , address=77.90.31.12:4612 , piece_id=63TS3SUEFUE6PORZ4W76OXRWFL7SBN773UGGY3DU7JPBAW66N7UA, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.356992732 (606h8m3.228075767s) [noise=true , address=64.49.13.140:4507 , piece_id=E7T6WL45F6GHECCRANX6AIKWFRZ77TURRWORYEYI43MET2WG7SRQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.357330644 (606h8m3.228413679s) [noise=true , address=77.90.31.21:4886 , piece_id=ZQRCPTVC3WFVQPZ36AU6KJW7PD3T557CXYQIICLDGOKK6JCPVZ5Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.357663649 (606h8m3.228746684s) [noise=true , address=145.79.227.49:4367 , piece_id=R2T2ABLXBPDVP3ZVA2TAYMCLQQ7JQUVQYCTHCKGMFBRVWCDAO4PQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.359827449 (606h8m3.230910484s) [noise=true , address=145.79.227.34:4918 , piece_id=HUMOHOT3HOYCBJ23WXQXWU3X6C6JQTRLLHTD4RQLCQTCB3HXI7EQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.360086692 (606h8m3.231169727s) [noise=true , address=145.79.227.7:4320 , piece_id=MAICNTHGLZ6W27HXDOOZXOBRGQGEXNSA4BJT2T5D5PHRVV6ZMX2Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.361622167 (606h8m3.232705212s) [noise=true , address=145.79.227.33:4888 , piece_id=I4426P3VETHPOL5WHWJEGVYRXJULTCNDX3WSUTS3XZVY36Q2N44Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.362965514 (606h8m3.234048559s) [noise=true , address=45.91.116.11:4133 , piece_id=NEW7ZOVP5E7HYGYD4LG5UG2MDLAQW2OBRTNZDFIYO4CQYMVAWRPQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.364308883 (606h8m3.235391928s) [noise=true , address=145.79.227.26:4663 , piece_id=DYEFQZL4X74QR2LAYURT5ZTR2IWIDB55VLBQOGZM36MA54P2SWJA, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.365734507 (606h8m3.236817542s) [noise=true , address=145.79.227.40:4059 , piece_id=7WSUBR3C3PWVBCWD6KRGSD6MPX4IH2YQOAVO4J4SWZORNHGS2IPQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.366982764 (606h8m3.238065799s) [noise=true , address=145.79.227.29:4973 , piece_id=5OROXELR55A47BFPSJJEWVQU4YW6CAOK6LMEHCINM4FN3OVI2Y6Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.368390294 (606h8m3.239473339s) [noise=true , address=77.90.31.19:4852 , piece_id=2NYO63GCBGCYBFYTMUGNF2NG5FSJOD3AOUOD2FZZIWDINK4X6DJQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:04.736295138 (606h8m4.607378183s) [upload=531240]: Next returned nil segment Dec 30 21:37:04.736311850 (606h8m4.607394885s) [upload=531240]: Waiting for error group managing segments... Dec 30 21:37:05.104989591 (606h8m4.976072636s) [noise=true , address=64.49.13.140:4507 , piece_id=E7T6WL45F6GHECCRANX6AIKWFRZ77TURRWORYEYI43MET2WG7SRQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.208809748 (606h8m5.079892793s) [noise=true , address=145.79.227.22:4543 , piece_id=WWCZ6SEEXQ2KUJ32AGTIJNMSJROLYSJ4XQGBQJC4Q5DACZVA65RQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.441125389 (606h8m5.312208444s) [noise=true , address=77.90.31.19:4852 , piece_id=2NYO63GCBGCYBFYTMUGNF2NG5FSJOD3AOUOD2FZZIWDINK4X6DJQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.483630904 (606h8m5.35471394s) [noise=true , address=77.90.31.12:4612 , piece_id=63TS3SUEFUE6PORZ4W76OXRWFL7SBN773UGGY3DU7JPBAW66N7UA, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.494478811 (606h8m5.365561846s) [noise=true , address=77.90.31.11:4559 , piece_id=3DYD4CNPJQN3S4EYZJGREQTTZJDSL2QTC3OT6G2VFUGMLSVCZUFA, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.497452683 (606h8m5.368535729s) [noise=true , address=77.90.31.17:4769 , piece_id=QXDYESBKGXRY7BCFYYF2O7OXS7Y4Z4KUMJB6AXZAWX735SYU56GQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.661529377 (606h8m5.532612412s) [noise=true , address=45.91.116.13:4182 , piece_id=BYAYNP4JAAN3AW2LYSK74WJE4BRWUR5Z6K67MTLK6A73KEJ35WEQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.729773112 (606h8m5.600856157s) [noise=true , address=45.91.116.11:4133 , piece_id=NEW7ZOVP5E7HYGYD4LG5UG2MDLAQW2OBRTNZDFIYO4CQYMVAWRPQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.940743308 (606h8m5.811826353s) [noise=true , address=77.90.31.21:4886 , piece_id=ZQRCPTVC3WFVQPZ36AU6KJW7PD3T557CXYQIICLDGOKK6JCPVZ5Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.084240044 (606h8m5.95532309s) [noise=true , address=145.79.227.30:4784 , piece_id=DSXUDNYCWO5F6IYXMIVJNEASYXIZ7EO7RKAS7QRUYK5OU7GOOB5Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.121095032 (606h8m5.992178067s) [noise=true , address=145.79.227.40:4059 , piece_id=7WSUBR3C3PWVBCWD6KRGSD6MPX4IH2YQOAVO4J4SWZORNHGS2IPQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.169195983 (606h8m6.040279028s) [noise=true , address=145.79.227.49:4367 , piece_id=R2T2ABLXBPDVP3ZVA2TAYMCLQQ7JQUVQYCTHCKGMFBRVWCDAO4PQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.169380634 (606h8m6.040463679s) [noise=true , address=145.79.227.33:4888 , piece_id=I4426P3VETHPOL5WHWJEGVYRXJULTCNDX3WSUTS3XZVY36Q2N44Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.179060175 (606h8m6.050143211s) [noise=true , address=145.79.227.29:4973 , piece_id=5OROXELR55A47BFPSJJEWVQU4YW6CAOK6LMEHCINM4FN3OVI2Y6Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.179080424 (606h8m6.050163459s) [seg_pos={0 0}, upload=531240]: Segment reached optimal threshold of 14 pieces. Dec 30 21:37:06.179262801 (606h8m6.050345837s) [noise=true , address=145.79.227.26:4663 , piece_id=DYEFQZL4X74QR2LAYURT5ZTR2IWIDB55VLBQOGZM36MA54P2SWJA, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SwpAaXTA3BqUvwubzmJkvwokdz6SSpT9t8HPtQcXXJxmcgfm2): protocol: write tcp 188.166.225.170:43184->145.79.227.26:4663: use of closed network connection; write tcp 188.166.225.170:43184->145.79.227.26:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43184->145.79.227.26:4663: use of closed network connection Dec 30 21:37:06.179268703 (606h8m6.050351748s) [noise=true , address=145.79.227.26:4663 , piece_id=DYEFQZL4X74QR2LAYURT5ZTR2IWIDB55VLBQOGZM36MA54P2SWJA, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179363724 (606h8m6.050446769s) [noise=true , address=145.79.227.7:4320 , piece_id=MAICNTHGLZ6W27HXDOOZXOBRGQGEXNSA4BJT2T5D5PHRVV6ZMX2Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126rEgnQw1w7hsFpRqQP9nLaumdBfdtkgYjdw6WE6nXrxk5kj4B): protocol: write tcp 188.166.225.170:45228->145.79.227.7:4320: use of closed network connection; write tcp 188.166.225.170:45228->145.79.227.7:4320: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45228->145.79.227.7:4320: use of closed network connection Dec 30 21:37:06.179369525 (606h8m6.05045257s) [noise=true , address=145.79.227.7:4320 , piece_id=MAICNTHGLZ6W27HXDOOZXOBRGQGEXNSA4BJT2T5D5PHRVV6ZMX2Q, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179532996 (606h8m6.050616061s) [noise=true , address=145.79.227.34:4918 , piece_id=HUMOHOT3HOYCBJ23WXQXWU3X6C6JQTRLLHTD4RQLCQTCB3HXI7EQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF Dec 30 21:37:06.179572371 (606h8m6.050655406s) [noise=true , address=145.79.227.34:4918 , piece_id=HUMOHOT3HOYCBJ23WXQXWU3X6C6JQTRLLHTD4RQLCQTCB3HXI7EQ, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179638587 (606h8m6.050721622s) [noise=true , address=145.79.227.35:4492 , piece_id=HCKFSQMJYEP6B5H4OX5AZ5TWP6Z4H4Z6WA2T7Z2H5KNZ4SDJRMUQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1roa5GGYmqCX2AxSrqFgqyV2CtBEYZavuGTseiwDy4pjkT6bip): protocol: expected piece hash; context canceled; EOF Dec 30 21:37:06.179647855 (606h8m6.05073089s) [noise=true , address=145.79.227.35:4492 , piece_id=HCKFSQMJYEP6B5H4OX5AZ5TWP6Z4H4Z6WA2T7Z2H5KNZ4SDJRMUQ, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179666641 (606h8m6.050749686s) [seg_pos={0 0}, upload=531240]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 21:37:06.179742766 (606h8m6.050825811s) [upload=531240]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 21:37:06.179747545 (606h8m6.05083059s) [upload=531240]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 21:37:06.490986034 (606h8m6.362069079s) [upload=531240]: Done uploading segments... Dec 30 21:37:06.491006261 (606h8m6.362089307s) [upload=531240]: Upload finished. err: <nil> Duration: 15.36303012s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [...X....X.X.........X..X...X.........X......................] duration: 5.725038893s trace: https://jaeger.dev.storj.io/trace/7b3713b537cd96b5 FAIL (2025-12-30 21:56:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: 3Y4O3DHH6VTTSFEDFLTGODSDJ2SINYTOVILJVKGHLEJRJAQGQRGQ) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: WPYVE54HEHASLRFMAHRNAGN4SBRLDDNNCLTUL6Q7IXCUTPKEQPJQ) context deadline exceeded ======LOGS====== Dec 30 21:37:24.517587467 (606h8m24.388670502s) [upload=531247]: Starting upload Dec 30 21:37:24.517683741 (606h8m24.388766776s) [upload=531247]: Done starting upload Dec 30 21:37:24.517750909 (606h8m24.388833964s) [upload=531247]: Uploading segments... Dec 30 21:37:24.517962091 (606h8m24.389045136s) [upload=531247]: Got next segment. Inline: false Dec 30 21:37:24.518006275 (606h8m24.38908932s) [upload=531247]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 21:37:24.518012496 (606h8m24.389095541s) [upload=531247]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 21:37:24.762005721 (606h8m24.633088796s) [seg_pos={0 0}, upload=531247]: Begin upload segment... Dec 30 21:37:24.762059043 (606h8m24.633142088s) [seg_pos={0 0}, upload=531247]: Done begin upload segment. Dec 30 21:37:24.762093669 (606h8m24.633176714s) [noise=true , address=145.79.227.13:4804 , piece_id=3T5EGEMHWDKHX3YMO46UAJLQLTXF72YYQROWVRGIJQMTXWWZTZUA, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.762105101 (606h8m24.633188136s) [noise=true , address=145.79.227.26:4654 , piece_id=YWMN73I2AGL5LMTBTDBR4REV35EPJ5FOC4FMQUMLHNYQBSM2R7YA, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.762162530 (606h8m24.633245575s) [noise=true , address=145.79.227.10:4403 , piece_id=NBFU5OK5RM4B5P53RMR6FH5XHHLYV3IMCAEEBF4DK7RQKF2YV56A, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.762806516 (606h8m24.633889581s) [noise=true , address=145.79.227.50:4374 , piece_id=4LK62BGBA6ZCAWL2K7JVJQN7VVZYVW7KHMFU6TOIAAXASEJQVP7Q, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.763247446 (606h8m24.634330491s) [noise=true , address=77.90.31.14:4662 , piece_id=3Y4O3DHH6VTTSFEDFLTGODSDJ2SINYTOVILJVKGHLEJRJAQGQRGQ, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.763647648 (606h8m24.634730683s) [noise=true , address=45.91.116.13:4155 , piece_id=WPYVE54HEHASLRFMAHRNAGN4SBRLDDNNCLTUL6Q7IXCUTPKEQPJQ, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.764127802 (606h8m24.635210847s) [noise=true , address=145.79.227.42:4121 , piece_id=M54EV7DTR4DH26KTXOQLG2IMQSRUKGOPVSXX2UEAHQGGGBLYCXWA, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.764542943 (606h8m24.635625978s) [noise=true , address=77.90.31.21:4880 , piece_id=MR3WNPHOPGXHD4GVIU4ONLKSPVYFPPWIINBKWIZWFENFVTZTSM3A, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.764959005 (606h8m24.63604205s) [noise=true , address=145.79.227.19:4079 , piece_id=W7EQPOU26CG3BPCWJGMTVV7KKDPOYA3HLYFT2HW2S4XJQZ6XUS5Q, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.766131597 (606h8m24.637214632s) [noise=true , address=45.91.116.10:4082 , piece_id=MYWKXEGVEDCGTIF26YFQTCLPLKXQUTFRE54AKLQ44PXV4EVR7ZHA, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.766690141 (606h8m24.637773176s) [noise=true , address=145.79.227.16:4911 , piece_id=ITFOWXPVEJU5SJX6A5EFQUMYFBCBV26EN4L3LA33AACJLJNQIPZQ, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.767199661 (606h8m24.638282696s) [noise=true , address=145.79.227.6:4258 , piece_id=LZPMJ6EZND6MF2MOMHUBNZ5SSJFYPJJYNW7UVQQEPYFEPLE5ZQCQ, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.767507377 (606h8m24.638590422s) [noise=true , address=145.79.227.37:4565 , piece_id=CNOZYEA5XVIFDAYZML66XRA3CVSUJJNWS3O75TIN4EZ7HTMCSNIQ, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.768056854 (606h8m24.639139899s) [noise=true , address=145.79.227.38:4979 , piece_id=DPTVSWBFBN2J3JO4QLRX4NJSDJE26AOW3QOSNXEDR6AFSVBDXCDA, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.768417160 (606h8m24.639500195s) [noise=true , address=145.79.227.40:4049 , piece_id=B54BGYWT6VSUUIGBVTAS6LHPYPEAK4LEX3HIGYLLBOPWONDBBYTA, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.768769200 (606h8m24.639852245s) [noise=true , address=145.79.227.24:4581 , piece_id=BYFXRVA7D4VCPXC3IGLN3T6GVHIIPB3MFICSXGDRKL2E5AUUW3WQ, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:26.469279868 (606h8m26.340362913s) [upload=531247]: Next returned nil segment Dec 30 21:37:26.469303293 (606h8m26.340386338s) [upload=531247]: Waiting for error group managing segments... Dec 30 21:37:26.756737396 (606h8m26.627820441s) [noise=true , address=145.79.227.10:4403 , piece_id=NBFU5OK5RM4B5P53RMR6FH5XHHLYV3IMCAEEBF4DK7RQKF2YV56A, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:26.768723167 (606h8m26.639806212s) [noise=true , address=145.79.227.42:4121 , piece_id=M54EV7DTR4DH26KTXOQLG2IMQSRUKGOPVSXX2UEAHQGGGBLYCXWA, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:26.768815023 (606h8m26.639898058s) [noise=true , address=145.79.227.38:4979 , piece_id=DPTVSWBFBN2J3JO4QLRX4NJSDJE26AOW3QOSNXEDR6AFSVBDXCDA, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:26.771265408 (606h8m26.642348453s) [noise=true , address=145.79.227.26:4654 , piece_id=YWMN73I2AGL5LMTBTDBR4REV35EPJ5FOC4FMQUMLHNYQBSM2R7YA, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:26.772261125 (606h8m26.64334416s) [noise=true , address=145.79.227.24:4581 , piece_id=BYFXRVA7D4VCPXC3IGLN3T6GVHIIPB3MFICSXGDRKL2E5AUUW3WQ, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:26.780818909 (606h8m26.651901954s) [noise=true , address=145.79.227.37:4565 , piece_id=CNOZYEA5XVIFDAYZML66XRA3CVSUJJNWS3O75TIN4EZ7HTMCSNIQ, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:26.781024501 (606h8m26.652107546s) [noise=true , address=145.79.227.50:4374 , piece_id=4LK62BGBA6ZCAWL2K7JVJQN7VVZYVW7KHMFU6TOIAAXASEJQVP7Q, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:26.781280018 (606h8m26.652363053s) [noise=true , address=145.79.227.13:4804 , piece_id=3T5EGEMHWDKHX3YMO46UAJLQLTXF72YYQROWVRGIJQMTXWWZTZUA, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:26.840740007 (606h8m26.711823042s) [noise=true , address=145.79.227.16:4911 , piece_id=ITFOWXPVEJU5SJX6A5EFQUMYFBCBV26EN4L3LA33AACJLJNQIPZQ, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:26.946038900 (606h8m26.817121936s) [noise=true , address=145.79.227.19:4079 , piece_id=W7EQPOU26CG3BPCWJGMTVV7KKDPOYA3HLYFT2HW2S4XJQZ6XUS5Q, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:27.027197388 (606h8m26.898280423s) [noise=true , address=77.90.31.14:4662 , piece_id=3Y4O3DHH6VTTSFEDFLTGODSDJ2SINYTOVILJVKGHLEJRJAQGQRGQ, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:27.065603629 (606h8m26.936686664s) [noise=true , address=45.91.116.10:4082 , piece_id=MYWKXEGVEDCGTIF26YFQTCLPLKXQUTFRE54AKLQ44PXV4EVR7ZHA, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:27.784121213 (606h8m27.655204248s) [noise=true , address=45.91.116.13:4155 , piece_id=WPYVE54HEHASLRFMAHRNAGN4SBRLDDNNCLTUL6Q7IXCUTPKEQPJQ, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:27.784142512 (606h8m27.655225558s) [seg_pos={0 0}, upload=531247]: Segment reached optimal threshold of 13 pieces. Dec 30 21:37:27.784300243 (606h8m27.655383278s) [noise=true , address=77.90.31.21:4880 , piece_id=MR3WNPHOPGXHD4GVIU4ONLKSPVYFPPWIINBKWIZWFENFVTZTSM3A, seg_pos={0 0}, upload=531247]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 30 21:37:27.784306134 (606h8m27.655389179s) [noise=true , address=77.90.31.21:4880 , piece_id=MR3WNPHOPGXHD4GVIU4ONLKSPVYFPPWIINBKWIZWFENFVTZTSM3A, seg_pos={0 0}, upload=531247]: Long tail cancellation due to optimal threshold Dec 30 21:37:27.784359966 (606h8m27.655443011s) [noise=true , address=145.79.227.40:4049 , piece_id=B54BGYWT6VSUUIGBVTAS6LHPYPEAK4LEX3HIGYLLBOPWONDBBYTA, seg_pos={0 0}, upload=531247]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 30 21:37:27.784370467 (606h8m27.655453522s) [noise=true , address=145.79.227.40:4049 , piece_id=B54BGYWT6VSUUIGBVTAS6LHPYPEAK4LEX3HIGYLLBOPWONDBBYTA, seg_pos={0 0}, upload=531247]: Long tail cancellation due to optimal threshold Dec 30 21:37:27.784429228 (606h8m27.655512263s) [noise=true , address=145.79.227.6:4258 , piece_id=LZPMJ6EZND6MF2MOMHUBNZ5SSJFYPJJYNW7UVQQEPYFEPLE5ZQCQ, seg_pos={0 0}, upload=531247]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF): protocol: write tcp 188.166.225.170:56840->145.79.227.6:4258: use of closed network connection; write tcp 188.166.225.170:56840->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56840->145.79.227.6:4258: use of closed network connection Dec 30 21:37:27.784436353 (606h8m27.655519398s) [noise=true , address=145.79.227.6:4258 , piece_id=LZPMJ6EZND6MF2MOMHUBNZ5SSJFYPJJYNW7UVQQEPYFEPLE5ZQCQ, seg_pos={0 0}, upload=531247]: Long tail cancellation due to optimal threshold Dec 30 21:37:27.784452513 (606h8m27.655535548s) [seg_pos={0 0}, upload=531247]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 21:37:27.784519210 (606h8m27.655602245s) [upload=531247]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 21:37:27.784523348 (606h8m27.655606393s) [upload=531247]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 21:37:28.083941079 (606h8m27.955024115s) [upload=531247]: Done uploading segments... Dec 30 21:37:28.083959936 (606h8m27.955042981s) [upload=531247]: Upload finished. err: <nil> Duration: 5.725082574s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................................................X.....] duration: 3.524667327s trace: https://jaeger.dev.storj.io/trace/22a47ec27f9a101c FAIL (2025-12-30 21:56:15 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.524695371s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [....X....X.X.........X..X...X.........X.....................] duration: 8.445895951s trace: https://jaeger.dev.storj.io/trace/7b3713b537cd96b5 FAIL (2025-12-30 21:55:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: 3Y4O3DHH6VTTSFEDFLTGODSDJ2SINYTOVILJVKGHLEJRJAQGQRGQ) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: WPYVE54HEHASLRFMAHRNAGN4SBRLDDNNCLTUL6Q7IXCUTPKEQPJQ) context deadline exceeded ======LOGS====== Dec 30 21:37:24.517587467 (606h8m24.388670502s) [upload=531247]: Starting upload Dec 30 21:37:24.517683741 (606h8m24.388766776s) [upload=531247]: Done starting upload Dec 30 21:37:24.517750909 (606h8m24.388833964s) [upload=531247]: Uploading segments... Dec 30 21:37:24.517962091 (606h8m24.389045136s) [upload=531247]: Got next segment. Inline: false Dec 30 21:37:24.518006275 (606h8m24.38908932s) [upload=531247]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 21:37:24.518012496 (606h8m24.389095541s) [upload=531247]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 21:37:24.762005721 (606h8m24.633088796s) [seg_pos={0 0}, upload=531247]: Begin upload segment... Dec 30 21:37:24.762059043 (606h8m24.633142088s) [seg_pos={0 0}, upload=531247]: Done begin upload segment. Dec 30 21:37:24.762093669 (606h8m24.633176714s) [noise=true , address=145.79.227.13:4804 , piece_id=3T5EGEMHWDKHX3YMO46UAJLQLTXF72YYQROWVRGIJQMTXWWZTZUA, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.762105101 (606h8m24.633188136s) [noise=true , address=145.79.227.26:4654 , piece_id=YWMN73I2AGL5LMTBTDBR4REV35EPJ5FOC4FMQUMLHNYQBSM2R7YA, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.762162530 (606h8m24.633245575s) [noise=true , address=145.79.227.10:4403 , piece_id=NBFU5OK5RM4B5P53RMR6FH5XHHLYV3IMCAEEBF4DK7RQKF2YV56A, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.762806516 (606h8m24.633889581s) [noise=true , address=145.79.227.50:4374 , piece_id=4LK62BGBA6ZCAWL2K7JVJQN7VVZYVW7KHMFU6TOIAAXASEJQVP7Q, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.763247446 (606h8m24.634330491s) [noise=true , address=77.90.31.14:4662 , piece_id=3Y4O3DHH6VTTSFEDFLTGODSDJ2SINYTOVILJVKGHLEJRJAQGQRGQ, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.763647648 (606h8m24.634730683s) [noise=true , address=45.91.116.13:4155 , piece_id=WPYVE54HEHASLRFMAHRNAGN4SBRLDDNNCLTUL6Q7IXCUTPKEQPJQ, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.764127802 (606h8m24.635210847s) [noise=true , address=145.79.227.42:4121 , piece_id=M54EV7DTR4DH26KTXOQLG2IMQSRUKGOPVSXX2UEAHQGGGBLYCXWA, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.764542943 (606h8m24.635625978s) [noise=true , address=77.90.31.21:4880 , piece_id=MR3WNPHOPGXHD4GVIU4ONLKSPVYFPPWIINBKWIZWFENFVTZTSM3A, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.764959005 (606h8m24.63604205s) [noise=true , address=145.79.227.19:4079 , piece_id=W7EQPOU26CG3BPCWJGMTVV7KKDPOYA3HLYFT2HW2S4XJQZ6XUS5Q, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.766131597 (606h8m24.637214632s) [noise=true , address=45.91.116.10:4082 , piece_id=MYWKXEGVEDCGTIF26YFQTCLPLKXQUTFRE54AKLQ44PXV4EVR7ZHA, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.766690141 (606h8m24.637773176s) [noise=true , address=145.79.227.16:4911 , piece_id=ITFOWXPVEJU5SJX6A5EFQUMYFBCBV26EN4L3LA33AACJLJNQIPZQ, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.767199661 (606h8m24.638282696s) [noise=true , address=145.79.227.6:4258 , piece_id=LZPMJ6EZND6MF2MOMHUBNZ5SSJFYPJJYNW7UVQQEPYFEPLE5ZQCQ, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.767507377 (606h8m24.638590422s) [noise=true , address=145.79.227.37:4565 , piece_id=CNOZYEA5XVIFDAYZML66XRA3CVSUJJNWS3O75TIN4EZ7HTMCSNIQ, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.768056854 (606h8m24.639139899s) [noise=true , address=145.79.227.38:4979 , piece_id=DPTVSWBFBN2J3JO4QLRX4NJSDJE26AOW3QOSNXEDR6AFSVBDXCDA, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.768417160 (606h8m24.639500195s) [noise=true , address=145.79.227.40:4049 , piece_id=B54BGYWT6VSUUIGBVTAS6LHPYPEAK4LEX3HIGYLLBOPWONDBBYTA, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.768769200 (606h8m24.639852245s) [noise=true , address=145.79.227.24:4581 , piece_id=BYFXRVA7D4VCPXC3IGLN3T6GVHIIPB3MFICSXGDRKL2E5AUUW3WQ, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:26.469279868 (606h8m26.340362913s) [upload=531247]: Next returned nil segment Dec 30 21:37:26.469303293 (606h8m26.340386338s) [upload=531247]: Waiting for error group managing segments... Dec 30 21:37:26.756737396 (606h8m26.627820441s) [noise=true , address=145.79.227.10:4403 , piece_id=NBFU5OK5RM4B5P53RMR6FH5XHHLYV3IMCAEEBF4DK7RQKF2YV56A, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:26.768723167 (606h8m26.639806212s) [noise=true , address=145.79.227.42:4121 , piece_id=M54EV7DTR4DH26KTXOQLG2IMQSRUKGOPVSXX2UEAHQGGGBLYCXWA, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:26.768815023 (606h8m26.639898058s) [noise=true , address=145.79.227.38:4979 , piece_id=DPTVSWBFBN2J3JO4QLRX4NJSDJE26AOW3QOSNXEDR6AFSVBDXCDA, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:26.771265408 (606h8m26.642348453s) [noise=true , address=145.79.227.26:4654 , piece_id=YWMN73I2AGL5LMTBTDBR4REV35EPJ5FOC4FMQUMLHNYQBSM2R7YA, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:26.772261125 (606h8m26.64334416s) [noise=true , address=145.79.227.24:4581 , piece_id=BYFXRVA7D4VCPXC3IGLN3T6GVHIIPB3MFICSXGDRKL2E5AUUW3WQ, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:26.780818909 (606h8m26.651901954s) [noise=true , address=145.79.227.37:4565 , piece_id=CNOZYEA5XVIFDAYZML66XRA3CVSUJJNWS3O75TIN4EZ7HTMCSNIQ, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:26.781024501 (606h8m26.652107546s) [noise=true , address=145.79.227.50:4374 , piece_id=4LK62BGBA6ZCAWL2K7JVJQN7VVZYVW7KHMFU6TOIAAXASEJQVP7Q, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:26.781280018 (606h8m26.652363053s) [noise=true , address=145.79.227.13:4804 , piece_id=3T5EGEMHWDKHX3YMO46UAJLQLTXF72YYQROWVRGIJQMTXWWZTZUA, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:26.840740007 (606h8m26.711823042s) [noise=true , address=145.79.227.16:4911 , piece_id=ITFOWXPVEJU5SJX6A5EFQUMYFBCBV26EN4L3LA33AACJLJNQIPZQ, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:26.946038900 (606h8m26.817121936s) [noise=true , address=145.79.227.19:4079 , piece_id=W7EQPOU26CG3BPCWJGMTVV7KKDPOYA3HLYFT2HW2S4XJQZ6XUS5Q, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:27.027197388 (606h8m26.898280423s) [noise=true , address=77.90.31.14:4662 , piece_id=3Y4O3DHH6VTTSFEDFLTGODSDJ2SINYTOVILJVKGHLEJRJAQGQRGQ, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:27.065603629 (606h8m26.936686664s) [noise=true , address=45.91.116.10:4082 , piece_id=MYWKXEGVEDCGTIF26YFQTCLPLKXQUTFRE54AKLQ44PXV4EVR7ZHA, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:27.784121213 (606h8m27.655204248s) [noise=true , address=45.91.116.13:4155 , piece_id=WPYVE54HEHASLRFMAHRNAGN4SBRLDDNNCLTUL6Q7IXCUTPKEQPJQ, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:27.784142512 (606h8m27.655225558s) [seg_pos={0 0}, upload=531247]: Segment reached optimal threshold of 13 pieces. Dec 30 21:37:27.784300243 (606h8m27.655383278s) [noise=true , address=77.90.31.21:4880 , piece_id=MR3WNPHOPGXHD4GVIU4ONLKSPVYFPPWIINBKWIZWFENFVTZTSM3A, seg_pos={0 0}, upload=531247]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 30 21:37:27.784306134 (606h8m27.655389179s) [noise=true , address=77.90.31.21:4880 , piece_id=MR3WNPHOPGXHD4GVIU4ONLKSPVYFPPWIINBKWIZWFENFVTZTSM3A, seg_pos={0 0}, upload=531247]: Long tail cancellation due to optimal threshold Dec 30 21:37:27.784359966 (606h8m27.655443011s) [noise=true , address=145.79.227.40:4049 , piece_id=B54BGYWT6VSUUIGBVTAS6LHPYPEAK4LEX3HIGYLLBOPWONDBBYTA, seg_pos={0 0}, upload=531247]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 30 21:37:27.784370467 (606h8m27.655453522s) [noise=true , address=145.79.227.40:4049 , piece_id=B54BGYWT6VSUUIGBVTAS6LHPYPEAK4LEX3HIGYLLBOPWONDBBYTA, seg_pos={0 0}, upload=531247]: Long tail cancellation due to optimal threshold Dec 30 21:37:27.784429228 (606h8m27.655512263s) [noise=true , address=145.79.227.6:4258 , piece_id=LZPMJ6EZND6MF2MOMHUBNZ5SSJFYPJJYNW7UVQQEPYFEPLE5ZQCQ, seg_pos={0 0}, upload=531247]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF): protocol: write tcp 188.166.225.170:56840->145.79.227.6:4258: use of closed network connection; write tcp 188.166.225.170:56840->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56840->145.79.227.6:4258: use of closed network connection Dec 30 21:37:27.784436353 (606h8m27.655519398s) [noise=true , address=145.79.227.6:4258 , piece_id=LZPMJ6EZND6MF2MOMHUBNZ5SSJFYPJJYNW7UVQQEPYFEPLE5ZQCQ, seg_pos={0 0}, upload=531247]: Long tail cancellation due to optimal threshold Dec 30 21:37:27.784452513 (606h8m27.655535548s) [seg_pos={0 0}, upload=531247]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 21:37:27.784519210 (606h8m27.655602245s) [upload=531247]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 21:37:27.784523348 (606h8m27.655606393s) [upload=531247]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 21:37:28.083941079 (606h8m27.955024115s) [upload=531247]: Done uploading segments... Dec 30 21:37:28.083959936 (606h8m27.955042981s) [upload=531247]: Upload finished. err: <nil> Duration: 8.445925346s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [...................XX......X.........X......................] duration: 45.096216489s trace: https://jaeger.dev.storj.io/trace/5cdd5ba7b1cb22c7 FAIL (2025-12-30 21:55:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 21:37:03.355296401 (606h8m3.226379446s) [noise=true , address=145.79.227.22:4543 , piece_id=WWCZ6SEEXQ2KUJ32AGTIJNMSJROLYSJ4XQGBQJC4Q5DACZVA65RQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.355839374 (606h8m3.22692241s) [noise=true , address=145.79.227.35:4492 , piece_id=HCKFSQMJYEP6B5H4OX5AZ5TWP6Z4H4Z6WA2T7Z2H5KNZ4SDJRMUQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.356356930 (606h8m3.227439965s) [noise=true , address=45.91.116.13:4182 , piece_id=BYAYNP4JAAN3AW2LYSK74WJE4BRWUR5Z6K67MTLK6A73KEJ35WEQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.356675316 (606h8m3.227758361s) [noise=true , address=77.90.31.12:4612 , piece_id=63TS3SUEFUE6PORZ4W76OXRWFL7SBN773UGGY3DU7JPBAW66N7UA, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.356992732 (606h8m3.228075767s) [noise=true , address=64.49.13.140:4507 , piece_id=E7T6WL45F6GHECCRANX6AIKWFRZ77TURRWORYEYI43MET2WG7SRQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.357330644 (606h8m3.228413679s) [noise=true , address=77.90.31.21:4886 , piece_id=ZQRCPTVC3WFVQPZ36AU6KJW7PD3T557CXYQIICLDGOKK6JCPVZ5Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.357663649 (606h8m3.228746684s) [noise=true , address=145.79.227.49:4367 , piece_id=R2T2ABLXBPDVP3ZVA2TAYMCLQQ7JQUVQYCTHCKGMFBRVWCDAO4PQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.359827449 (606h8m3.230910484s) [noise=true , address=145.79.227.34:4918 , piece_id=HUMOHOT3HOYCBJ23WXQXWU3X6C6JQTRLLHTD4RQLCQTCB3HXI7EQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.360086692 (606h8m3.231169727s) [noise=true , address=145.79.227.7:4320 , piece_id=MAICNTHGLZ6W27HXDOOZXOBRGQGEXNSA4BJT2T5D5PHRVV6ZMX2Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.361622167 (606h8m3.232705212s) [noise=true , address=145.79.227.33:4888 , piece_id=I4426P3VETHPOL5WHWJEGVYRXJULTCNDX3WSUTS3XZVY36Q2N44Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.362965514 (606h8m3.234048559s) [noise=true , address=45.91.116.11:4133 , piece_id=NEW7ZOVP5E7HYGYD4LG5UG2MDLAQW2OBRTNZDFIYO4CQYMVAWRPQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.364308883 (606h8m3.235391928s) [noise=true , address=145.79.227.26:4663 , piece_id=DYEFQZL4X74QR2LAYURT5ZTR2IWIDB55VLBQOGZM36MA54P2SWJA, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.365734507 (606h8m3.236817542s) [noise=true , address=145.79.227.40:4059 , piece_id=7WSUBR3C3PWVBCWD6KRGSD6MPX4IH2YQOAVO4J4SWZORNHGS2IPQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.366982764 (606h8m3.238065799s) [noise=true , address=145.79.227.29:4973 , piece_id=5OROXELR55A47BFPSJJEWVQU4YW6CAOK6LMEHCINM4FN3OVI2Y6Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.368390294 (606h8m3.239473339s) [noise=true , address=77.90.31.19:4852 , piece_id=2NYO63GCBGCYBFYTMUGNF2NG5FSJOD3AOUOD2FZZIWDINK4X6DJQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:04.736295138 (606h8m4.607378183s) [upload=531240]: Next returned nil segment Dec 30 21:37:04.736311850 (606h8m4.607394885s) [upload=531240]: Waiting for error group managing segments... Dec 30 21:37:05.104989591 (606h8m4.976072636s) [noise=true , address=64.49.13.140:4507 , piece_id=E7T6WL45F6GHECCRANX6AIKWFRZ77TURRWORYEYI43MET2WG7SRQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.208809748 (606h8m5.079892793s) [noise=true , address=145.79.227.22:4543 , piece_id=WWCZ6SEEXQ2KUJ32AGTIJNMSJROLYSJ4XQGBQJC4Q5DACZVA65RQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.441125389 (606h8m5.312208444s) [noise=true , address=77.90.31.19:4852 , piece_id=2NYO63GCBGCYBFYTMUGNF2NG5FSJOD3AOUOD2FZZIWDINK4X6DJQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.483630904 (606h8m5.35471394s) [noise=true , address=77.90.31.12:4612 , piece_id=63TS3SUEFUE6PORZ4W76OXRWFL7SBN773UGGY3DU7JPBAW66N7UA, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.494478811 (606h8m5.365561846s) [noise=true , address=77.90.31.11:4559 , piece_id=3DYD4CNPJQN3S4EYZJGREQTTZJDSL2QTC3OT6G2VFUGMLSVCZUFA, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.497452683 (606h8m5.368535729s) [noise=true , address=77.90.31.17:4769 , piece_id=QXDYESBKGXRY7BCFYYF2O7OXS7Y4Z4KUMJB6AXZAWX735SYU56GQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.661529377 (606h8m5.532612412s) [noise=true , address=45.91.116.13:4182 , piece_id=BYAYNP4JAAN3AW2LYSK74WJE4BRWUR5Z6K67MTLK6A73KEJ35WEQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.729773112 (606h8m5.600856157s) [noise=true , address=45.91.116.11:4133 , piece_id=NEW7ZOVP5E7HYGYD4LG5UG2MDLAQW2OBRTNZDFIYO4CQYMVAWRPQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.940743308 (606h8m5.811826353s) [noise=true , address=77.90.31.21:4886 , piece_id=ZQRCPTVC3WFVQPZ36AU6KJW7PD3T557CXYQIICLDGOKK6JCPVZ5Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.084240044 (606h8m5.95532309s) [noise=true , address=145.79.227.30:4784 , piece_id=DSXUDNYCWO5F6IYXMIVJNEASYXIZ7EO7RKAS7QRUYK5OU7GOOB5Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.121095032 (606h8m5.992178067s) [noise=true , address=145.79.227.40:4059 , piece_id=7WSUBR3C3PWVBCWD6KRGSD6MPX4IH2YQOAVO4J4SWZORNHGS2IPQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.169195983 (606h8m6.040279028s) [noise=true , address=145.79.227.49:4367 , piece_id=R2T2ABLXBPDVP3ZVA2TAYMCLQQ7JQUVQYCTHCKGMFBRVWCDAO4PQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.169380634 (606h8m6.040463679s) [noise=true , address=145.79.227.33:4888 , piece_id=I4426P3VETHPOL5WHWJEGVYRXJULTCNDX3WSUTS3XZVY36Q2N44Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.179060175 (606h8m6.050143211s) [noise=true , address=145.79.227.29:4973 , piece_id=5OROXELR55A47BFPSJJEWVQU4YW6CAOK6LMEHCINM4FN3OVI2Y6Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.179080424 (606h8m6.050163459s) [seg_pos={0 0}, upload=531240]: Segment reached optimal threshold of 14 pieces. Dec 30 21:37:06.179262801 (606h8m6.050345837s) [noise=true , address=145.79.227.26:4663 , piece_id=DYEFQZL4X74QR2LAYURT5ZTR2IWIDB55VLBQOGZM36MA54P2SWJA, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SwpAaXTA3BqUvwubzmJkvwokdz6SSpT9t8HPtQcXXJxmcgfm2): protocol: write tcp 188.166.225.170:43184->145.79.227.26:4663: use of closed network connection; write tcp 188.166.225.170:43184->145.79.227.26:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43184->145.79.227.26:4663: use of closed network connection Dec 30 21:37:06.179268703 (606h8m6.050351748s) [noise=true , address=145.79.227.26:4663 , piece_id=DYEFQZL4X74QR2LAYURT5ZTR2IWIDB55VLBQOGZM36MA54P2SWJA, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179363724 (606h8m6.050446769s) [noise=true , address=145.79.227.7:4320 , piece_id=MAICNTHGLZ6W27HXDOOZXOBRGQGEXNSA4BJT2T5D5PHRVV6ZMX2Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126rEgnQw1w7hsFpRqQP9nLaumdBfdtkgYjdw6WE6nXrxk5kj4B): protocol: write tcp 188.166.225.170:45228->145.79.227.7:4320: use of closed network connection; write tcp 188.166.225.170:45228->145.79.227.7:4320: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45228->145.79.227.7:4320: use of closed network connection Dec 30 21:37:06.179369525 (606h8m6.05045257s) [noise=true , address=145.79.227.7:4320 , piece_id=MAICNTHGLZ6W27HXDOOZXOBRGQGEXNSA4BJT2T5D5PHRVV6ZMX2Q, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179532996 (606h8m6.050616061s) [noise=true , address=145.79.227.34:4918 , piece_id=HUMOHOT3HOYCBJ23WXQXWU3X6C6JQTRLLHTD4RQLCQTCB3HXI7EQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF Dec 30 21:37:06.179572371 (606h8m6.050655406s) [noise=true , address=145.79.227.34:4918 , piece_id=HUMOHOT3HOYCBJ23WXQXWU3X6C6JQTRLLHTD4RQLCQTCB3HXI7EQ, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179638587 (606h8m6.050721622s) [noise=true , address=145.79.227.35:4492 , piece_id=HCKFSQMJYEP6B5H4OX5AZ5TWP6Z4H4Z6WA2T7Z2H5KNZ4SDJRMUQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1roa5GGYmqCX2AxSrqFgqyV2CtBEYZavuGTseiwDy4pjkT6bip): protocol: expected piece hash; context canceled; EOF Dec 30 21:37:06.179647855 (606h8m6.05073089s) [noise=true , address=145.79.227.35:4492 , piece_id=HCKFSQMJYEP6B5H4OX5AZ5TWP6Z4H4Z6WA2T7Z2H5KNZ4SDJRMUQ, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179666641 (606h8m6.050749686s) [seg_pos={0 0}, upload=531240]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 21:37:06.179742766 (606h8m6.050825811s) [upload=531240]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 21:37:06.179747545 (606h8m6.05083059s) [upload=531240]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 21:37:06.490986034 (606h8m6.362069079s) [upload=531240]: Done uploading segments... Dec 30 21:37:06.491006261 (606h8m6.362089307s) [upload=531240]: Upload finished. err: <nil> Duration: 45.096240514s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................................................X....] duration: 3.605099644s trace: https://jaeger.dev.storj.io/trace/22a47ec27f9a101c FAIL (2025-12-30 21:55:12 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.605130962s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.....X....X.X.........X..X...X.........X....................] duration: 6.611982868s trace: https://jaeger.dev.storj.io/trace/7b3713b537cd96b5 FAIL (2025-12-30 21:55:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: 3Y4O3DHH6VTTSFEDFLTGODSDJ2SINYTOVILJVKGHLEJRJAQGQRGQ) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: WPYVE54HEHASLRFMAHRNAGN4SBRLDDNNCLTUL6Q7IXCUTPKEQPJQ) context deadline exceeded ======LOGS====== Dec 30 21:37:24.517587467 (606h8m24.388670502s) [upload=531247]: Starting upload Dec 30 21:37:24.517683741 (606h8m24.388766776s) [upload=531247]: Done starting upload Dec 30 21:37:24.517750909 (606h8m24.388833964s) [upload=531247]: Uploading segments... Dec 30 21:37:24.517962091 (606h8m24.389045136s) [upload=531247]: Got next segment. Inline: false Dec 30 21:37:24.518006275 (606h8m24.38908932s) [upload=531247]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 21:37:24.518012496 (606h8m24.389095541s) [upload=531247]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 21:37:24.762005721 (606h8m24.633088796s) [seg_pos={0 0}, upload=531247]: Begin upload segment... Dec 30 21:37:24.762059043 (606h8m24.633142088s) [seg_pos={0 0}, upload=531247]: Done begin upload segment. Dec 30 21:37:24.762093669 (606h8m24.633176714s) [noise=true , address=145.79.227.13:4804 , piece_id=3T5EGEMHWDKHX3YMO46UAJLQLTXF72YYQROWVRGIJQMTXWWZTZUA, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.762105101 (606h8m24.633188136s) [noise=true , address=145.79.227.26:4654 , piece_id=YWMN73I2AGL5LMTBTDBR4REV35EPJ5FOC4FMQUMLHNYQBSM2R7YA, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.762162530 (606h8m24.633245575s) [noise=true , address=145.79.227.10:4403 , piece_id=NBFU5OK5RM4B5P53RMR6FH5XHHLYV3IMCAEEBF4DK7RQKF2YV56A, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.762806516 (606h8m24.633889581s) [noise=true , address=145.79.227.50:4374 , piece_id=4LK62BGBA6ZCAWL2K7JVJQN7VVZYVW7KHMFU6TOIAAXASEJQVP7Q, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.763247446 (606h8m24.634330491s) [noise=true , address=77.90.31.14:4662 , piece_id=3Y4O3DHH6VTTSFEDFLTGODSDJ2SINYTOVILJVKGHLEJRJAQGQRGQ, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.763647648 (606h8m24.634730683s) [noise=true , address=45.91.116.13:4155 , piece_id=WPYVE54HEHASLRFMAHRNAGN4SBRLDDNNCLTUL6Q7IXCUTPKEQPJQ, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.764127802 (606h8m24.635210847s) [noise=true , address=145.79.227.42:4121 , piece_id=M54EV7DTR4DH26KTXOQLG2IMQSRUKGOPVSXX2UEAHQGGGBLYCXWA, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.764542943 (606h8m24.635625978s) [noise=true , address=77.90.31.21:4880 , piece_id=MR3WNPHOPGXHD4GVIU4ONLKSPVYFPPWIINBKWIZWFENFVTZTSM3A, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.764959005 (606h8m24.63604205s) [noise=true , address=145.79.227.19:4079 , piece_id=W7EQPOU26CG3BPCWJGMTVV7KKDPOYA3HLYFT2HW2S4XJQZ6XUS5Q, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.766131597 (606h8m24.637214632s) [noise=true , address=45.91.116.10:4082 , piece_id=MYWKXEGVEDCGTIF26YFQTCLPLKXQUTFRE54AKLQ44PXV4EVR7ZHA, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.766690141 (606h8m24.637773176s) [noise=true , address=145.79.227.16:4911 , piece_id=ITFOWXPVEJU5SJX6A5EFQUMYFBCBV26EN4L3LA33AACJLJNQIPZQ, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.767199661 (606h8m24.638282696s) [noise=true , address=145.79.227.6:4258 , piece_id=LZPMJ6EZND6MF2MOMHUBNZ5SSJFYPJJYNW7UVQQEPYFEPLE5ZQCQ, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.767507377 (606h8m24.638590422s) [noise=true , address=145.79.227.37:4565 , piece_id=CNOZYEA5XVIFDAYZML66XRA3CVSUJJNWS3O75TIN4EZ7HTMCSNIQ, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.768056854 (606h8m24.639139899s) [noise=true , address=145.79.227.38:4979 , piece_id=DPTVSWBFBN2J3JO4QLRX4NJSDJE26AOW3QOSNXEDR6AFSVBDXCDA, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.768417160 (606h8m24.639500195s) [noise=true , address=145.79.227.40:4049 , piece_id=B54BGYWT6VSUUIGBVTAS6LHPYPEAK4LEX3HIGYLLBOPWONDBBYTA, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.768769200 (606h8m24.639852245s) [noise=true , address=145.79.227.24:4581 , piece_id=BYFXRVA7D4VCPXC3IGLN3T6GVHIIPB3MFICSXGDRKL2E5AUUW3WQ, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:26.469279868 (606h8m26.340362913s) [upload=531247]: Next returned nil segment Dec 30 21:37:26.469303293 (606h8m26.340386338s) [upload=531247]: Waiting for error group managing segments... Dec 30 21:37:26.756737396 (606h8m26.627820441s) [noise=true , address=145.79.227.10:4403 , piece_id=NBFU5OK5RM4B5P53RMR6FH5XHHLYV3IMCAEEBF4DK7RQKF2YV56A, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:26.768723167 (606h8m26.639806212s) [noise=true , address=145.79.227.42:4121 , piece_id=M54EV7DTR4DH26KTXOQLG2IMQSRUKGOPVSXX2UEAHQGGGBLYCXWA, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:26.768815023 (606h8m26.639898058s) [noise=true , address=145.79.227.38:4979 , piece_id=DPTVSWBFBN2J3JO4QLRX4NJSDJE26AOW3QOSNXEDR6AFSVBDXCDA, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:26.771265408 (606h8m26.642348453s) [noise=true , address=145.79.227.26:4654 , piece_id=YWMN73I2AGL5LMTBTDBR4REV35EPJ5FOC4FMQUMLHNYQBSM2R7YA, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:26.772261125 (606h8m26.64334416s) [noise=true , address=145.79.227.24:4581 , piece_id=BYFXRVA7D4VCPXC3IGLN3T6GVHIIPB3MFICSXGDRKL2E5AUUW3WQ, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:26.780818909 (606h8m26.651901954s) [noise=true , address=145.79.227.37:4565 , piece_id=CNOZYEA5XVIFDAYZML66XRA3CVSUJJNWS3O75TIN4EZ7HTMCSNIQ, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:26.781024501 (606h8m26.652107546s) [noise=true , address=145.79.227.50:4374 , piece_id=4LK62BGBA6ZCAWL2K7JVJQN7VVZYVW7KHMFU6TOIAAXASEJQVP7Q, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:26.781280018 (606h8m26.652363053s) [noise=true , address=145.79.227.13:4804 , piece_id=3T5EGEMHWDKHX3YMO46UAJLQLTXF72YYQROWVRGIJQMTXWWZTZUA, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:26.840740007 (606h8m26.711823042s) [noise=true , address=145.79.227.16:4911 , piece_id=ITFOWXPVEJU5SJX6A5EFQUMYFBCBV26EN4L3LA33AACJLJNQIPZQ, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:26.946038900 (606h8m26.817121936s) [noise=true , address=145.79.227.19:4079 , piece_id=W7EQPOU26CG3BPCWJGMTVV7KKDPOYA3HLYFT2HW2S4XJQZ6XUS5Q, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:27.027197388 (606h8m26.898280423s) [noise=true , address=77.90.31.14:4662 , piece_id=3Y4O3DHH6VTTSFEDFLTGODSDJ2SINYTOVILJVKGHLEJRJAQGQRGQ, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:27.065603629 (606h8m26.936686664s) [noise=true , address=45.91.116.10:4082 , piece_id=MYWKXEGVEDCGTIF26YFQTCLPLKXQUTFRE54AKLQ44PXV4EVR7ZHA, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:27.784121213 (606h8m27.655204248s) [noise=true , address=45.91.116.13:4155 , piece_id=WPYVE54HEHASLRFMAHRNAGN4SBRLDDNNCLTUL6Q7IXCUTPKEQPJQ, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:27.784142512 (606h8m27.655225558s) [seg_pos={0 0}, upload=531247]: Segment reached optimal threshold of 13 pieces. Dec 30 21:37:27.784300243 (606h8m27.655383278s) [noise=true , address=77.90.31.21:4880 , piece_id=MR3WNPHOPGXHD4GVIU4ONLKSPVYFPPWIINBKWIZWFENFVTZTSM3A, seg_pos={0 0}, upload=531247]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 30 21:37:27.784306134 (606h8m27.655389179s) [noise=true , address=77.90.31.21:4880 , piece_id=MR3WNPHOPGXHD4GVIU4ONLKSPVYFPPWIINBKWIZWFENFVTZTSM3A, seg_pos={0 0}, upload=531247]: Long tail cancellation due to optimal threshold Dec 30 21:37:27.784359966 (606h8m27.655443011s) [noise=true , address=145.79.227.40:4049 , piece_id=B54BGYWT6VSUUIGBVTAS6LHPYPEAK4LEX3HIGYLLBOPWONDBBYTA, seg_pos={0 0}, upload=531247]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 30 21:37:27.784370467 (606h8m27.655453522s) [noise=true , address=145.79.227.40:4049 , piece_id=B54BGYWT6VSUUIGBVTAS6LHPYPEAK4LEX3HIGYLLBOPWONDBBYTA, seg_pos={0 0}, upload=531247]: Long tail cancellation due to optimal threshold Dec 30 21:37:27.784429228 (606h8m27.655512263s) [noise=true , address=145.79.227.6:4258 , piece_id=LZPMJ6EZND6MF2MOMHUBNZ5SSJFYPJJYNW7UVQQEPYFEPLE5ZQCQ, seg_pos={0 0}, upload=531247]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF): protocol: write tcp 188.166.225.170:56840->145.79.227.6:4258: use of closed network connection; write tcp 188.166.225.170:56840->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56840->145.79.227.6:4258: use of closed network connection Dec 30 21:37:27.784436353 (606h8m27.655519398s) [noise=true , address=145.79.227.6:4258 , piece_id=LZPMJ6EZND6MF2MOMHUBNZ5SSJFYPJJYNW7UVQQEPYFEPLE5ZQCQ, seg_pos={0 0}, upload=531247]: Long tail cancellation due to optimal threshold Dec 30 21:37:27.784452513 (606h8m27.655535548s) [seg_pos={0 0}, upload=531247]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 21:37:27.784519210 (606h8m27.655602245s) [upload=531247]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 21:37:27.784523348 (606h8m27.655606393s) [upload=531247]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 21:37:28.083941079 (606h8m27.955024115s) [upload=531247]: Done uploading segments... Dec 30 21:37:28.083959936 (606h8m27.955042981s) [upload=531247]: Upload finished. err: <nil> Duration: 6.612014068s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [......X....X.X.........X..X...X.........X...................] duration: 7.702714379s trace: https://jaeger.dev.storj.io/trace/7b3713b537cd96b5 FAIL (2025-12-30 21:54:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: 3Y4O3DHH6VTTSFEDFLTGODSDJ2SINYTOVILJVKGHLEJRJAQGQRGQ) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: WPYVE54HEHASLRFMAHRNAGN4SBRLDDNNCLTUL6Q7IXCUTPKEQPJQ) context deadline exceeded ======LOGS====== Dec 30 21:37:24.517587467 (606h8m24.388670502s) [upload=531247]: Starting upload Dec 30 21:37:24.517683741 (606h8m24.388766776s) [upload=531247]: Done starting upload Dec 30 21:37:24.517750909 (606h8m24.388833964s) [upload=531247]: Uploading segments... Dec 30 21:37:24.517962091 (606h8m24.389045136s) [upload=531247]: Got next segment. Inline: false Dec 30 21:37:24.518006275 (606h8m24.38908932s) [upload=531247]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 21:37:24.518012496 (606h8m24.389095541s) [upload=531247]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 21:37:24.762005721 (606h8m24.633088796s) [seg_pos={0 0}, upload=531247]: Begin upload segment... Dec 30 21:37:24.762059043 (606h8m24.633142088s) [seg_pos={0 0}, upload=531247]: Done begin upload segment. Dec 30 21:37:24.762093669 (606h8m24.633176714s) [noise=true , address=145.79.227.13:4804 , piece_id=3T5EGEMHWDKHX3YMO46UAJLQLTXF72YYQROWVRGIJQMTXWWZTZUA, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.762105101 (606h8m24.633188136s) [noise=true , address=145.79.227.26:4654 , piece_id=YWMN73I2AGL5LMTBTDBR4REV35EPJ5FOC4FMQUMLHNYQBSM2R7YA, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.762162530 (606h8m24.633245575s) [noise=true , address=145.79.227.10:4403 , piece_id=NBFU5OK5RM4B5P53RMR6FH5XHHLYV3IMCAEEBF4DK7RQKF2YV56A, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.762806516 (606h8m24.633889581s) [noise=true , address=145.79.227.50:4374 , piece_id=4LK62BGBA6ZCAWL2K7JVJQN7VVZYVW7KHMFU6TOIAAXASEJQVP7Q, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.763247446 (606h8m24.634330491s) [noise=true , address=77.90.31.14:4662 , piece_id=3Y4O3DHH6VTTSFEDFLTGODSDJ2SINYTOVILJVKGHLEJRJAQGQRGQ, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.763647648 (606h8m24.634730683s) [noise=true , address=45.91.116.13:4155 , piece_id=WPYVE54HEHASLRFMAHRNAGN4SBRLDDNNCLTUL6Q7IXCUTPKEQPJQ, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.764127802 (606h8m24.635210847s) [noise=true , address=145.79.227.42:4121 , piece_id=M54EV7DTR4DH26KTXOQLG2IMQSRUKGOPVSXX2UEAHQGGGBLYCXWA, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.764542943 (606h8m24.635625978s) [noise=true , address=77.90.31.21:4880 , piece_id=MR3WNPHOPGXHD4GVIU4ONLKSPVYFPPWIINBKWIZWFENFVTZTSM3A, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.764959005 (606h8m24.63604205s) [noise=true , address=145.79.227.19:4079 , piece_id=W7EQPOU26CG3BPCWJGMTVV7KKDPOYA3HLYFT2HW2S4XJQZ6XUS5Q, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.766131597 (606h8m24.637214632s) [noise=true , address=45.91.116.10:4082 , piece_id=MYWKXEGVEDCGTIF26YFQTCLPLKXQUTFRE54AKLQ44PXV4EVR7ZHA, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.766690141 (606h8m24.637773176s) [noise=true , address=145.79.227.16:4911 , piece_id=ITFOWXPVEJU5SJX6A5EFQUMYFBCBV26EN4L3LA33AACJLJNQIPZQ, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.767199661 (606h8m24.638282696s) [noise=true , address=145.79.227.6:4258 , piece_id=LZPMJ6EZND6MF2MOMHUBNZ5SSJFYPJJYNW7UVQQEPYFEPLE5ZQCQ, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.767507377 (606h8m24.638590422s) [noise=true , address=145.79.227.37:4565 , piece_id=CNOZYEA5XVIFDAYZML66XRA3CVSUJJNWS3O75TIN4EZ7HTMCSNIQ, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.768056854 (606h8m24.639139899s) [noise=true , address=145.79.227.38:4979 , piece_id=DPTVSWBFBN2J3JO4QLRX4NJSDJE26AOW3QOSNXEDR6AFSVBDXCDA, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.768417160 (606h8m24.639500195s) [noise=true , address=145.79.227.40:4049 , piece_id=B54BGYWT6VSUUIGBVTAS6LHPYPEAK4LEX3HIGYLLBOPWONDBBYTA, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.768769200 (606h8m24.639852245s) [noise=true , address=145.79.227.24:4581 , piece_id=BYFXRVA7D4VCPXC3IGLN3T6GVHIIPB3MFICSXGDRKL2E5AUUW3WQ, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:26.469279868 (606h8m26.340362913s) [upload=531247]: Next returned nil segment Dec 30 21:37:26.469303293 (606h8m26.340386338s) [upload=531247]: Waiting for error group managing segments... Dec 30 21:37:26.756737396 (606h8m26.627820441s) [noise=true , address=145.79.227.10:4403 , piece_id=NBFU5OK5RM4B5P53RMR6FH5XHHLYV3IMCAEEBF4DK7RQKF2YV56A, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:26.768723167 (606h8m26.639806212s) [noise=true , address=145.79.227.42:4121 , piece_id=M54EV7DTR4DH26KTXOQLG2IMQSRUKGOPVSXX2UEAHQGGGBLYCXWA, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:26.768815023 (606h8m26.639898058s) [noise=true , address=145.79.227.38:4979 , piece_id=DPTVSWBFBN2J3JO4QLRX4NJSDJE26AOW3QOSNXEDR6AFSVBDXCDA, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:26.771265408 (606h8m26.642348453s) [noise=true , address=145.79.227.26:4654 , piece_id=YWMN73I2AGL5LMTBTDBR4REV35EPJ5FOC4FMQUMLHNYQBSM2R7YA, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:26.772261125 (606h8m26.64334416s) [noise=true , address=145.79.227.24:4581 , piece_id=BYFXRVA7D4VCPXC3IGLN3T6GVHIIPB3MFICSXGDRKL2E5AUUW3WQ, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:26.780818909 (606h8m26.651901954s) [noise=true , address=145.79.227.37:4565 , piece_id=CNOZYEA5XVIFDAYZML66XRA3CVSUJJNWS3O75TIN4EZ7HTMCSNIQ, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:26.781024501 (606h8m26.652107546s) [noise=true , address=145.79.227.50:4374 , piece_id=4LK62BGBA6ZCAWL2K7JVJQN7VVZYVW7KHMFU6TOIAAXASEJQVP7Q, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:26.781280018 (606h8m26.652363053s) [noise=true , address=145.79.227.13:4804 , piece_id=3T5EGEMHWDKHX3YMO46UAJLQLTXF72YYQROWVRGIJQMTXWWZTZUA, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:26.840740007 (606h8m26.711823042s) [noise=true , address=145.79.227.16:4911 , piece_id=ITFOWXPVEJU5SJX6A5EFQUMYFBCBV26EN4L3LA33AACJLJNQIPZQ, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:26.946038900 (606h8m26.817121936s) [noise=true , address=145.79.227.19:4079 , piece_id=W7EQPOU26CG3BPCWJGMTVV7KKDPOYA3HLYFT2HW2S4XJQZ6XUS5Q, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:27.027197388 (606h8m26.898280423s) [noise=true , address=77.90.31.14:4662 , piece_id=3Y4O3DHH6VTTSFEDFLTGODSDJ2SINYTOVILJVKGHLEJRJAQGQRGQ, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:27.065603629 (606h8m26.936686664s) [noise=true , address=45.91.116.10:4082 , piece_id=MYWKXEGVEDCGTIF26YFQTCLPLKXQUTFRE54AKLQ44PXV4EVR7ZHA, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:27.784121213 (606h8m27.655204248s) [noise=true , address=45.91.116.13:4155 , piece_id=WPYVE54HEHASLRFMAHRNAGN4SBRLDDNNCLTUL6Q7IXCUTPKEQPJQ, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:27.784142512 (606h8m27.655225558s) [seg_pos={0 0}, upload=531247]: Segment reached optimal threshold of 13 pieces. Dec 30 21:37:27.784300243 (606h8m27.655383278s) [noise=true , address=77.90.31.21:4880 , piece_id=MR3WNPHOPGXHD4GVIU4ONLKSPVYFPPWIINBKWIZWFENFVTZTSM3A, seg_pos={0 0}, upload=531247]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 30 21:37:27.784306134 (606h8m27.655389179s) [noise=true , address=77.90.31.21:4880 , piece_id=MR3WNPHOPGXHD4GVIU4ONLKSPVYFPPWIINBKWIZWFENFVTZTSM3A, seg_pos={0 0}, upload=531247]: Long tail cancellation due to optimal threshold Dec 30 21:37:27.784359966 (606h8m27.655443011s) [noise=true , address=145.79.227.40:4049 , piece_id=B54BGYWT6VSUUIGBVTAS6LHPYPEAK4LEX3HIGYLLBOPWONDBBYTA, seg_pos={0 0}, upload=531247]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 30 21:37:27.784370467 (606h8m27.655453522s) [noise=true , address=145.79.227.40:4049 , piece_id=B54BGYWT6VSUUIGBVTAS6LHPYPEAK4LEX3HIGYLLBOPWONDBBYTA, seg_pos={0 0}, upload=531247]: Long tail cancellation due to optimal threshold Dec 30 21:37:27.784429228 (606h8m27.655512263s) [noise=true , address=145.79.227.6:4258 , piece_id=LZPMJ6EZND6MF2MOMHUBNZ5SSJFYPJJYNW7UVQQEPYFEPLE5ZQCQ, seg_pos={0 0}, upload=531247]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF): protocol: write tcp 188.166.225.170:56840->145.79.227.6:4258: use of closed network connection; write tcp 188.166.225.170:56840->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56840->145.79.227.6:4258: use of closed network connection Dec 30 21:37:27.784436353 (606h8m27.655519398s) [noise=true , address=145.79.227.6:4258 , piece_id=LZPMJ6EZND6MF2MOMHUBNZ5SSJFYPJJYNW7UVQQEPYFEPLE5ZQCQ, seg_pos={0 0}, upload=531247]: Long tail cancellation due to optimal threshold Dec 30 21:37:27.784452513 (606h8m27.655535548s) [seg_pos={0 0}, upload=531247]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 21:37:27.784519210 (606h8m27.655602245s) [upload=531247]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 21:37:27.784523348 (606h8m27.655606393s) [upload=531247]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 21:37:28.083941079 (606h8m27.955024115s) [upload=531247]: Done uploading segments... Dec 30 21:37:28.083959936 (606h8m27.955042981s) [upload=531247]: Upload finished. err: <nil> Duration: 7.702770395s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [....................XX......X.........X.....................] duration: 15.315003242s trace: https://jaeger.dev.storj.io/trace/5cdd5ba7b1cb22c7 FAIL (2025-12-30 21:54:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 21:37:03.355296401 (606h8m3.226379446s) [noise=true , address=145.79.227.22:4543 , piece_id=WWCZ6SEEXQ2KUJ32AGTIJNMSJROLYSJ4XQGBQJC4Q5DACZVA65RQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.355839374 (606h8m3.22692241s) [noise=true , address=145.79.227.35:4492 , piece_id=HCKFSQMJYEP6B5H4OX5AZ5TWP6Z4H4Z6WA2T7Z2H5KNZ4SDJRMUQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.356356930 (606h8m3.227439965s) [noise=true , address=45.91.116.13:4182 , piece_id=BYAYNP4JAAN3AW2LYSK74WJE4BRWUR5Z6K67MTLK6A73KEJ35WEQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.356675316 (606h8m3.227758361s) [noise=true , address=77.90.31.12:4612 , piece_id=63TS3SUEFUE6PORZ4W76OXRWFL7SBN773UGGY3DU7JPBAW66N7UA, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.356992732 (606h8m3.228075767s) [noise=true , address=64.49.13.140:4507 , piece_id=E7T6WL45F6GHECCRANX6AIKWFRZ77TURRWORYEYI43MET2WG7SRQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.357330644 (606h8m3.228413679s) [noise=true , address=77.90.31.21:4886 , piece_id=ZQRCPTVC3WFVQPZ36AU6KJW7PD3T557CXYQIICLDGOKK6JCPVZ5Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.357663649 (606h8m3.228746684s) [noise=true , address=145.79.227.49:4367 , piece_id=R2T2ABLXBPDVP3ZVA2TAYMCLQQ7JQUVQYCTHCKGMFBRVWCDAO4PQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.359827449 (606h8m3.230910484s) [noise=true , address=145.79.227.34:4918 , piece_id=HUMOHOT3HOYCBJ23WXQXWU3X6C6JQTRLLHTD4RQLCQTCB3HXI7EQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.360086692 (606h8m3.231169727s) [noise=true , address=145.79.227.7:4320 , piece_id=MAICNTHGLZ6W27HXDOOZXOBRGQGEXNSA4BJT2T5D5PHRVV6ZMX2Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.361622167 (606h8m3.232705212s) [noise=true , address=145.79.227.33:4888 , piece_id=I4426P3VETHPOL5WHWJEGVYRXJULTCNDX3WSUTS3XZVY36Q2N44Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.362965514 (606h8m3.234048559s) [noise=true , address=45.91.116.11:4133 , piece_id=NEW7ZOVP5E7HYGYD4LG5UG2MDLAQW2OBRTNZDFIYO4CQYMVAWRPQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.364308883 (606h8m3.235391928s) [noise=true , address=145.79.227.26:4663 , piece_id=DYEFQZL4X74QR2LAYURT5ZTR2IWIDB55VLBQOGZM36MA54P2SWJA, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.365734507 (606h8m3.236817542s) [noise=true , address=145.79.227.40:4059 , piece_id=7WSUBR3C3PWVBCWD6KRGSD6MPX4IH2YQOAVO4J4SWZORNHGS2IPQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.366982764 (606h8m3.238065799s) [noise=true , address=145.79.227.29:4973 , piece_id=5OROXELR55A47BFPSJJEWVQU4YW6CAOK6LMEHCINM4FN3OVI2Y6Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.368390294 (606h8m3.239473339s) [noise=true , address=77.90.31.19:4852 , piece_id=2NYO63GCBGCYBFYTMUGNF2NG5FSJOD3AOUOD2FZZIWDINK4X6DJQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:04.736295138 (606h8m4.607378183s) [upload=531240]: Next returned nil segment Dec 30 21:37:04.736311850 (606h8m4.607394885s) [upload=531240]: Waiting for error group managing segments... Dec 30 21:37:05.104989591 (606h8m4.976072636s) [noise=true , address=64.49.13.140:4507 , piece_id=E7T6WL45F6GHECCRANX6AIKWFRZ77TURRWORYEYI43MET2WG7SRQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.208809748 (606h8m5.079892793s) [noise=true , address=145.79.227.22:4543 , piece_id=WWCZ6SEEXQ2KUJ32AGTIJNMSJROLYSJ4XQGBQJC4Q5DACZVA65RQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.441125389 (606h8m5.312208444s) [noise=true , address=77.90.31.19:4852 , piece_id=2NYO63GCBGCYBFYTMUGNF2NG5FSJOD3AOUOD2FZZIWDINK4X6DJQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.483630904 (606h8m5.35471394s) [noise=true , address=77.90.31.12:4612 , piece_id=63TS3SUEFUE6PORZ4W76OXRWFL7SBN773UGGY3DU7JPBAW66N7UA, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.494478811 (606h8m5.365561846s) [noise=true , address=77.90.31.11:4559 , piece_id=3DYD4CNPJQN3S4EYZJGREQTTZJDSL2QTC3OT6G2VFUGMLSVCZUFA, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.497452683 (606h8m5.368535729s) [noise=true , address=77.90.31.17:4769 , piece_id=QXDYESBKGXRY7BCFYYF2O7OXS7Y4Z4KUMJB6AXZAWX735SYU56GQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.661529377 (606h8m5.532612412s) [noise=true , address=45.91.116.13:4182 , piece_id=BYAYNP4JAAN3AW2LYSK74WJE4BRWUR5Z6K67MTLK6A73KEJ35WEQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.729773112 (606h8m5.600856157s) [noise=true , address=45.91.116.11:4133 , piece_id=NEW7ZOVP5E7HYGYD4LG5UG2MDLAQW2OBRTNZDFIYO4CQYMVAWRPQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.940743308 (606h8m5.811826353s) [noise=true , address=77.90.31.21:4886 , piece_id=ZQRCPTVC3WFVQPZ36AU6KJW7PD3T557CXYQIICLDGOKK6JCPVZ5Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.084240044 (606h8m5.95532309s) [noise=true , address=145.79.227.30:4784 , piece_id=DSXUDNYCWO5F6IYXMIVJNEASYXIZ7EO7RKAS7QRUYK5OU7GOOB5Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.121095032 (606h8m5.992178067s) [noise=true , address=145.79.227.40:4059 , piece_id=7WSUBR3C3PWVBCWD6KRGSD6MPX4IH2YQOAVO4J4SWZORNHGS2IPQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.169195983 (606h8m6.040279028s) [noise=true , address=145.79.227.49:4367 , piece_id=R2T2ABLXBPDVP3ZVA2TAYMCLQQ7JQUVQYCTHCKGMFBRVWCDAO4PQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.169380634 (606h8m6.040463679s) [noise=true , address=145.79.227.33:4888 , piece_id=I4426P3VETHPOL5WHWJEGVYRXJULTCNDX3WSUTS3XZVY36Q2N44Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.179060175 (606h8m6.050143211s) [noise=true , address=145.79.227.29:4973 , piece_id=5OROXELR55A47BFPSJJEWVQU4YW6CAOK6LMEHCINM4FN3OVI2Y6Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.179080424 (606h8m6.050163459s) [seg_pos={0 0}, upload=531240]: Segment reached optimal threshold of 14 pieces. Dec 30 21:37:06.179262801 (606h8m6.050345837s) [noise=true , address=145.79.227.26:4663 , piece_id=DYEFQZL4X74QR2LAYURT5ZTR2IWIDB55VLBQOGZM36MA54P2SWJA, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SwpAaXTA3BqUvwubzmJkvwokdz6SSpT9t8HPtQcXXJxmcgfm2): protocol: write tcp 188.166.225.170:43184->145.79.227.26:4663: use of closed network connection; write tcp 188.166.225.170:43184->145.79.227.26:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43184->145.79.227.26:4663: use of closed network connection Dec 30 21:37:06.179268703 (606h8m6.050351748s) [noise=true , address=145.79.227.26:4663 , piece_id=DYEFQZL4X74QR2LAYURT5ZTR2IWIDB55VLBQOGZM36MA54P2SWJA, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179363724 (606h8m6.050446769s) [noise=true , address=145.79.227.7:4320 , piece_id=MAICNTHGLZ6W27HXDOOZXOBRGQGEXNSA4BJT2T5D5PHRVV6ZMX2Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126rEgnQw1w7hsFpRqQP9nLaumdBfdtkgYjdw6WE6nXrxk5kj4B): protocol: write tcp 188.166.225.170:45228->145.79.227.7:4320: use of closed network connection; write tcp 188.166.225.170:45228->145.79.227.7:4320: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45228->145.79.227.7:4320: use of closed network connection Dec 30 21:37:06.179369525 (606h8m6.05045257s) [noise=true , address=145.79.227.7:4320 , piece_id=MAICNTHGLZ6W27HXDOOZXOBRGQGEXNSA4BJT2T5D5PHRVV6ZMX2Q, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179532996 (606h8m6.050616061s) [noise=true , address=145.79.227.34:4918 , piece_id=HUMOHOT3HOYCBJ23WXQXWU3X6C6JQTRLLHTD4RQLCQTCB3HXI7EQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF Dec 30 21:37:06.179572371 (606h8m6.050655406s) [noise=true , address=145.79.227.34:4918 , piece_id=HUMOHOT3HOYCBJ23WXQXWU3X6C6JQTRLLHTD4RQLCQTCB3HXI7EQ, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179638587 (606h8m6.050721622s) [noise=true , address=145.79.227.35:4492 , piece_id=HCKFSQMJYEP6B5H4OX5AZ5TWP6Z4H4Z6WA2T7Z2H5KNZ4SDJRMUQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1roa5GGYmqCX2AxSrqFgqyV2CtBEYZavuGTseiwDy4pjkT6bip): protocol: expected piece hash; context canceled; EOF Dec 30 21:37:06.179647855 (606h8m6.05073089s) [noise=true , address=145.79.227.35:4492 , piece_id=HCKFSQMJYEP6B5H4OX5AZ5TWP6Z4H4Z6WA2T7Z2H5KNZ4SDJRMUQ, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179666641 (606h8m6.050749686s) [seg_pos={0 0}, upload=531240]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 21:37:06.179742766 (606h8m6.050825811s) [upload=531240]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 21:37:06.179747545 (606h8m6.05083059s) [upload=531240]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 21:37:06.490986034 (606h8m6.362069079s) [upload=531240]: Done uploading segments... Dec 30 21:37:06.491006261 (606h8m6.362089307s) [upload=531240]: Upload finished. err: <nil> Duration: 15.315019224s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.......X....X.X.........X..X...X.........X..................] duration: 28.32928653s trace: https://jaeger.dev.storj.io/trace/7b3713b537cd96b5 FAIL (2025-12-30 21:54:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: 3Y4O3DHH6VTTSFEDFLTGODSDJ2SINYTOVILJVKGHLEJRJAQGQRGQ) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: WPYVE54HEHASLRFMAHRNAGN4SBRLDDNNCLTUL6Q7IXCUTPKEQPJQ) context deadline exceeded ======LOGS====== Dec 30 21:37:24.517587467 (606h8m24.388670502s) [upload=531247]: Starting upload Dec 30 21:37:24.517683741 (606h8m24.388766776s) [upload=531247]: Done starting upload Dec 30 21:37:24.517750909 (606h8m24.388833964s) [upload=531247]: Uploading segments... Dec 30 21:37:24.517962091 (606h8m24.389045136s) [upload=531247]: Got next segment. Inline: false Dec 30 21:37:24.518006275 (606h8m24.38908932s) [upload=531247]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 21:37:24.518012496 (606h8m24.389095541s) [upload=531247]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 21:37:24.762005721 (606h8m24.633088796s) [seg_pos={0 0}, upload=531247]: Begin upload segment... Dec 30 21:37:24.762059043 (606h8m24.633142088s) [seg_pos={0 0}, upload=531247]: Done begin upload segment. Dec 30 21:37:24.762093669 (606h8m24.633176714s) [noise=true , address=145.79.227.13:4804 , piece_id=3T5EGEMHWDKHX3YMO46UAJLQLTXF72YYQROWVRGIJQMTXWWZTZUA, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.762105101 (606h8m24.633188136s) [noise=true , address=145.79.227.26:4654 , piece_id=YWMN73I2AGL5LMTBTDBR4REV35EPJ5FOC4FMQUMLHNYQBSM2R7YA, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.762162530 (606h8m24.633245575s) [noise=true , address=145.79.227.10:4403 , piece_id=NBFU5OK5RM4B5P53RMR6FH5XHHLYV3IMCAEEBF4DK7RQKF2YV56A, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.762806516 (606h8m24.633889581s) [noise=true , address=145.79.227.50:4374 , piece_id=4LK62BGBA6ZCAWL2K7JVJQN7VVZYVW7KHMFU6TOIAAXASEJQVP7Q, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.763247446 (606h8m24.634330491s) [noise=true , address=77.90.31.14:4662 , piece_id=3Y4O3DHH6VTTSFEDFLTGODSDJ2SINYTOVILJVKGHLEJRJAQGQRGQ, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.763647648 (606h8m24.634730683s) [noise=true , address=45.91.116.13:4155 , piece_id=WPYVE54HEHASLRFMAHRNAGN4SBRLDDNNCLTUL6Q7IXCUTPKEQPJQ, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.764127802 (606h8m24.635210847s) [noise=true , address=145.79.227.42:4121 , piece_id=M54EV7DTR4DH26KTXOQLG2IMQSRUKGOPVSXX2UEAHQGGGBLYCXWA, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.764542943 (606h8m24.635625978s) [noise=true , address=77.90.31.21:4880 , piece_id=MR3WNPHOPGXHD4GVIU4ONLKSPVYFPPWIINBKWIZWFENFVTZTSM3A, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.764959005 (606h8m24.63604205s) [noise=true , address=145.79.227.19:4079 , piece_id=W7EQPOU26CG3BPCWJGMTVV7KKDPOYA3HLYFT2HW2S4XJQZ6XUS5Q, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.766131597 (606h8m24.637214632s) [noise=true , address=45.91.116.10:4082 , piece_id=MYWKXEGVEDCGTIF26YFQTCLPLKXQUTFRE54AKLQ44PXV4EVR7ZHA, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.766690141 (606h8m24.637773176s) [noise=true , address=145.79.227.16:4911 , piece_id=ITFOWXPVEJU5SJX6A5EFQUMYFBCBV26EN4L3LA33AACJLJNQIPZQ, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.767199661 (606h8m24.638282696s) [noise=true , address=145.79.227.6:4258 , piece_id=LZPMJ6EZND6MF2MOMHUBNZ5SSJFYPJJYNW7UVQQEPYFEPLE5ZQCQ, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.767507377 (606h8m24.638590422s) [noise=true , address=145.79.227.37:4565 , piece_id=CNOZYEA5XVIFDAYZML66XRA3CVSUJJNWS3O75TIN4EZ7HTMCSNIQ, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.768056854 (606h8m24.639139899s) [noise=true , address=145.79.227.38:4979 , piece_id=DPTVSWBFBN2J3JO4QLRX4NJSDJE26AOW3QOSNXEDR6AFSVBDXCDA, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.768417160 (606h8m24.639500195s) [noise=true , address=145.79.227.40:4049 , piece_id=B54BGYWT6VSUUIGBVTAS6LHPYPEAK4LEX3HIGYLLBOPWONDBBYTA, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.768769200 (606h8m24.639852245s) [noise=true , address=145.79.227.24:4581 , piece_id=BYFXRVA7D4VCPXC3IGLN3T6GVHIIPB3MFICSXGDRKL2E5AUUW3WQ, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:26.469279868 (606h8m26.340362913s) [upload=531247]: Next returned nil segment Dec 30 21:37:26.469303293 (606h8m26.340386338s) [upload=531247]: Waiting for error group managing segments... Dec 30 21:37:26.756737396 (606h8m26.627820441s) [noise=true , address=145.79.227.10:4403 , piece_id=NBFU5OK5RM4B5P53RMR6FH5XHHLYV3IMCAEEBF4DK7RQKF2YV56A, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:26.768723167 (606h8m26.639806212s) [noise=true , address=145.79.227.42:4121 , piece_id=M54EV7DTR4DH26KTXOQLG2IMQSRUKGOPVSXX2UEAHQGGGBLYCXWA, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:26.768815023 (606h8m26.639898058s) [noise=true , address=145.79.227.38:4979 , piece_id=DPTVSWBFBN2J3JO4QLRX4NJSDJE26AOW3QOSNXEDR6AFSVBDXCDA, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:26.771265408 (606h8m26.642348453s) [noise=true , address=145.79.227.26:4654 , piece_id=YWMN73I2AGL5LMTBTDBR4REV35EPJ5FOC4FMQUMLHNYQBSM2R7YA, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:26.772261125 (606h8m26.64334416s) [noise=true , address=145.79.227.24:4581 , piece_id=BYFXRVA7D4VCPXC3IGLN3T6GVHIIPB3MFICSXGDRKL2E5AUUW3WQ, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:26.780818909 (606h8m26.651901954s) [noise=true , address=145.79.227.37:4565 , piece_id=CNOZYEA5XVIFDAYZML66XRA3CVSUJJNWS3O75TIN4EZ7HTMCSNIQ, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:26.781024501 (606h8m26.652107546s) [noise=true , address=145.79.227.50:4374 , piece_id=4LK62BGBA6ZCAWL2K7JVJQN7VVZYVW7KHMFU6TOIAAXASEJQVP7Q, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:26.781280018 (606h8m26.652363053s) [noise=true , address=145.79.227.13:4804 , piece_id=3T5EGEMHWDKHX3YMO46UAJLQLTXF72YYQROWVRGIJQMTXWWZTZUA, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:26.840740007 (606h8m26.711823042s) [noise=true , address=145.79.227.16:4911 , piece_id=ITFOWXPVEJU5SJX6A5EFQUMYFBCBV26EN4L3LA33AACJLJNQIPZQ, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:26.946038900 (606h8m26.817121936s) [noise=true , address=145.79.227.19:4079 , piece_id=W7EQPOU26CG3BPCWJGMTVV7KKDPOYA3HLYFT2HW2S4XJQZ6XUS5Q, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:27.027197388 (606h8m26.898280423s) [noise=true , address=77.90.31.14:4662 , piece_id=3Y4O3DHH6VTTSFEDFLTGODSDJ2SINYTOVILJVKGHLEJRJAQGQRGQ, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:27.065603629 (606h8m26.936686664s) [noise=true , address=45.91.116.10:4082 , piece_id=MYWKXEGVEDCGTIF26YFQTCLPLKXQUTFRE54AKLQ44PXV4EVR7ZHA, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:27.784121213 (606h8m27.655204248s) [noise=true , address=45.91.116.13:4155 , piece_id=WPYVE54HEHASLRFMAHRNAGN4SBRLDDNNCLTUL6Q7IXCUTPKEQPJQ, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:27.784142512 (606h8m27.655225558s) [seg_pos={0 0}, upload=531247]: Segment reached optimal threshold of 13 pieces. Dec 30 21:37:27.784300243 (606h8m27.655383278s) [noise=true , address=77.90.31.21:4880 , piece_id=MR3WNPHOPGXHD4GVIU4ONLKSPVYFPPWIINBKWIZWFENFVTZTSM3A, seg_pos={0 0}, upload=531247]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 30 21:37:27.784306134 (606h8m27.655389179s) [noise=true , address=77.90.31.21:4880 , piece_id=MR3WNPHOPGXHD4GVIU4ONLKSPVYFPPWIINBKWIZWFENFVTZTSM3A, seg_pos={0 0}, upload=531247]: Long tail cancellation due to optimal threshold Dec 30 21:37:27.784359966 (606h8m27.655443011s) [noise=true , address=145.79.227.40:4049 , piece_id=B54BGYWT6VSUUIGBVTAS6LHPYPEAK4LEX3HIGYLLBOPWONDBBYTA, seg_pos={0 0}, upload=531247]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 30 21:37:27.784370467 (606h8m27.655453522s) [noise=true , address=145.79.227.40:4049 , piece_id=B54BGYWT6VSUUIGBVTAS6LHPYPEAK4LEX3HIGYLLBOPWONDBBYTA, seg_pos={0 0}, upload=531247]: Long tail cancellation due to optimal threshold Dec 30 21:37:27.784429228 (606h8m27.655512263s) [noise=true , address=145.79.227.6:4258 , piece_id=LZPMJ6EZND6MF2MOMHUBNZ5SSJFYPJJYNW7UVQQEPYFEPLE5ZQCQ, seg_pos={0 0}, upload=531247]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF): protocol: write tcp 188.166.225.170:56840->145.79.227.6:4258: use of closed network connection; write tcp 188.166.225.170:56840->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56840->145.79.227.6:4258: use of closed network connection Dec 30 21:37:27.784436353 (606h8m27.655519398s) [noise=true , address=145.79.227.6:4258 , piece_id=LZPMJ6EZND6MF2MOMHUBNZ5SSJFYPJJYNW7UVQQEPYFEPLE5ZQCQ, seg_pos={0 0}, upload=531247]: Long tail cancellation due to optimal threshold Dec 30 21:37:27.784452513 (606h8m27.655535548s) [seg_pos={0 0}, upload=531247]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 21:37:27.784519210 (606h8m27.655602245s) [upload=531247]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 21:37:27.784523348 (606h8m27.655606393s) [upload=531247]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 21:37:28.083941079 (606h8m27.955024115s) [upload=531247]: Done uploading segments... Dec 30 21:37:28.083959936 (606h8m27.955042981s) [upload=531247]: Upload finished. err: <nil> Duration: 28.329317238s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....................XX......X.........X....................] duration: 52.707098791s trace: https://jaeger.dev.storj.io/trace/5cdd5ba7b1cb22c7 FAIL (2025-12-30 21:54:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 21:37:03.355296401 (606h8m3.226379446s) [noise=true , address=145.79.227.22:4543 , piece_id=WWCZ6SEEXQ2KUJ32AGTIJNMSJROLYSJ4XQGBQJC4Q5DACZVA65RQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.355839374 (606h8m3.22692241s) [noise=true , address=145.79.227.35:4492 , piece_id=HCKFSQMJYEP6B5H4OX5AZ5TWP6Z4H4Z6WA2T7Z2H5KNZ4SDJRMUQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.356356930 (606h8m3.227439965s) [noise=true , address=45.91.116.13:4182 , piece_id=BYAYNP4JAAN3AW2LYSK74WJE4BRWUR5Z6K67MTLK6A73KEJ35WEQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.356675316 (606h8m3.227758361s) [noise=true , address=77.90.31.12:4612 , piece_id=63TS3SUEFUE6PORZ4W76OXRWFL7SBN773UGGY3DU7JPBAW66N7UA, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.356992732 (606h8m3.228075767s) [noise=true , address=64.49.13.140:4507 , piece_id=E7T6WL45F6GHECCRANX6AIKWFRZ77TURRWORYEYI43MET2WG7SRQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.357330644 (606h8m3.228413679s) [noise=true , address=77.90.31.21:4886 , piece_id=ZQRCPTVC3WFVQPZ36AU6KJW7PD3T557CXYQIICLDGOKK6JCPVZ5Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.357663649 (606h8m3.228746684s) [noise=true , address=145.79.227.49:4367 , piece_id=R2T2ABLXBPDVP3ZVA2TAYMCLQQ7JQUVQYCTHCKGMFBRVWCDAO4PQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.359827449 (606h8m3.230910484s) [noise=true , address=145.79.227.34:4918 , piece_id=HUMOHOT3HOYCBJ23WXQXWU3X6C6JQTRLLHTD4RQLCQTCB3HXI7EQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.360086692 (606h8m3.231169727s) [noise=true , address=145.79.227.7:4320 , piece_id=MAICNTHGLZ6W27HXDOOZXOBRGQGEXNSA4BJT2T5D5PHRVV6ZMX2Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.361622167 (606h8m3.232705212s) [noise=true , address=145.79.227.33:4888 , piece_id=I4426P3VETHPOL5WHWJEGVYRXJULTCNDX3WSUTS3XZVY36Q2N44Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.362965514 (606h8m3.234048559s) [noise=true , address=45.91.116.11:4133 , piece_id=NEW7ZOVP5E7HYGYD4LG5UG2MDLAQW2OBRTNZDFIYO4CQYMVAWRPQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.364308883 (606h8m3.235391928s) [noise=true , address=145.79.227.26:4663 , piece_id=DYEFQZL4X74QR2LAYURT5ZTR2IWIDB55VLBQOGZM36MA54P2SWJA, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.365734507 (606h8m3.236817542s) [noise=true , address=145.79.227.40:4059 , piece_id=7WSUBR3C3PWVBCWD6KRGSD6MPX4IH2YQOAVO4J4SWZORNHGS2IPQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.366982764 (606h8m3.238065799s) [noise=true , address=145.79.227.29:4973 , piece_id=5OROXELR55A47BFPSJJEWVQU4YW6CAOK6LMEHCINM4FN3OVI2Y6Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.368390294 (606h8m3.239473339s) [noise=true , address=77.90.31.19:4852 , piece_id=2NYO63GCBGCYBFYTMUGNF2NG5FSJOD3AOUOD2FZZIWDINK4X6DJQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:04.736295138 (606h8m4.607378183s) [upload=531240]: Next returned nil segment Dec 30 21:37:04.736311850 (606h8m4.607394885s) [upload=531240]: Waiting for error group managing segments... Dec 30 21:37:05.104989591 (606h8m4.976072636s) [noise=true , address=64.49.13.140:4507 , piece_id=E7T6WL45F6GHECCRANX6AIKWFRZ77TURRWORYEYI43MET2WG7SRQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.208809748 (606h8m5.079892793s) [noise=true , address=145.79.227.22:4543 , piece_id=WWCZ6SEEXQ2KUJ32AGTIJNMSJROLYSJ4XQGBQJC4Q5DACZVA65RQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.441125389 (606h8m5.312208444s) [noise=true , address=77.90.31.19:4852 , piece_id=2NYO63GCBGCYBFYTMUGNF2NG5FSJOD3AOUOD2FZZIWDINK4X6DJQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.483630904 (606h8m5.35471394s) [noise=true , address=77.90.31.12:4612 , piece_id=63TS3SUEFUE6PORZ4W76OXRWFL7SBN773UGGY3DU7JPBAW66N7UA, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.494478811 (606h8m5.365561846s) [noise=true , address=77.90.31.11:4559 , piece_id=3DYD4CNPJQN3S4EYZJGREQTTZJDSL2QTC3OT6G2VFUGMLSVCZUFA, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.497452683 (606h8m5.368535729s) [noise=true , address=77.90.31.17:4769 , piece_id=QXDYESBKGXRY7BCFYYF2O7OXS7Y4Z4KUMJB6AXZAWX735SYU56GQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.661529377 (606h8m5.532612412s) [noise=true , address=45.91.116.13:4182 , piece_id=BYAYNP4JAAN3AW2LYSK74WJE4BRWUR5Z6K67MTLK6A73KEJ35WEQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.729773112 (606h8m5.600856157s) [noise=true , address=45.91.116.11:4133 , piece_id=NEW7ZOVP5E7HYGYD4LG5UG2MDLAQW2OBRTNZDFIYO4CQYMVAWRPQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.940743308 (606h8m5.811826353s) [noise=true , address=77.90.31.21:4886 , piece_id=ZQRCPTVC3WFVQPZ36AU6KJW7PD3T557CXYQIICLDGOKK6JCPVZ5Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.084240044 (606h8m5.95532309s) [noise=true , address=145.79.227.30:4784 , piece_id=DSXUDNYCWO5F6IYXMIVJNEASYXIZ7EO7RKAS7QRUYK5OU7GOOB5Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.121095032 (606h8m5.992178067s) [noise=true , address=145.79.227.40:4059 , piece_id=7WSUBR3C3PWVBCWD6KRGSD6MPX4IH2YQOAVO4J4SWZORNHGS2IPQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.169195983 (606h8m6.040279028s) [noise=true , address=145.79.227.49:4367 , piece_id=R2T2ABLXBPDVP3ZVA2TAYMCLQQ7JQUVQYCTHCKGMFBRVWCDAO4PQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.169380634 (606h8m6.040463679s) [noise=true , address=145.79.227.33:4888 , piece_id=I4426P3VETHPOL5WHWJEGVYRXJULTCNDX3WSUTS3XZVY36Q2N44Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.179060175 (606h8m6.050143211s) [noise=true , address=145.79.227.29:4973 , piece_id=5OROXELR55A47BFPSJJEWVQU4YW6CAOK6LMEHCINM4FN3OVI2Y6Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.179080424 (606h8m6.050163459s) [seg_pos={0 0}, upload=531240]: Segment reached optimal threshold of 14 pieces. Dec 30 21:37:06.179262801 (606h8m6.050345837s) [noise=true , address=145.79.227.26:4663 , piece_id=DYEFQZL4X74QR2LAYURT5ZTR2IWIDB55VLBQOGZM36MA54P2SWJA, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SwpAaXTA3BqUvwubzmJkvwokdz6SSpT9t8HPtQcXXJxmcgfm2): protocol: write tcp 188.166.225.170:43184->145.79.227.26:4663: use of closed network connection; write tcp 188.166.225.170:43184->145.79.227.26:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43184->145.79.227.26:4663: use of closed network connection Dec 30 21:37:06.179268703 (606h8m6.050351748s) [noise=true , address=145.79.227.26:4663 , piece_id=DYEFQZL4X74QR2LAYURT5ZTR2IWIDB55VLBQOGZM36MA54P2SWJA, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179363724 (606h8m6.050446769s) [noise=true , address=145.79.227.7:4320 , piece_id=MAICNTHGLZ6W27HXDOOZXOBRGQGEXNSA4BJT2T5D5PHRVV6ZMX2Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126rEgnQw1w7hsFpRqQP9nLaumdBfdtkgYjdw6WE6nXrxk5kj4B): protocol: write tcp 188.166.225.170:45228->145.79.227.7:4320: use of closed network connection; write tcp 188.166.225.170:45228->145.79.227.7:4320: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45228->145.79.227.7:4320: use of closed network connection Dec 30 21:37:06.179369525 (606h8m6.05045257s) [noise=true , address=145.79.227.7:4320 , piece_id=MAICNTHGLZ6W27HXDOOZXOBRGQGEXNSA4BJT2T5D5PHRVV6ZMX2Q, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179532996 (606h8m6.050616061s) [noise=true , address=145.79.227.34:4918 , piece_id=HUMOHOT3HOYCBJ23WXQXWU3X6C6JQTRLLHTD4RQLCQTCB3HXI7EQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF Dec 30 21:37:06.179572371 (606h8m6.050655406s) [noise=true , address=145.79.227.34:4918 , piece_id=HUMOHOT3HOYCBJ23WXQXWU3X6C6JQTRLLHTD4RQLCQTCB3HXI7EQ, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179638587 (606h8m6.050721622s) [noise=true , address=145.79.227.35:4492 , piece_id=HCKFSQMJYEP6B5H4OX5AZ5TWP6Z4H4Z6WA2T7Z2H5KNZ4SDJRMUQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1roa5GGYmqCX2AxSrqFgqyV2CtBEYZavuGTseiwDy4pjkT6bip): protocol: expected piece hash; context canceled; EOF Dec 30 21:37:06.179647855 (606h8m6.05073089s) [noise=true , address=145.79.227.35:4492 , piece_id=HCKFSQMJYEP6B5H4OX5AZ5TWP6Z4H4Z6WA2T7Z2H5KNZ4SDJRMUQ, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179666641 (606h8m6.050749686s) [seg_pos={0 0}, upload=531240]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 21:37:06.179742766 (606h8m6.050825811s) [upload=531240]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 21:37:06.179747545 (606h8m6.05083059s) [upload=531240]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 21:37:06.490986034 (606h8m6.362069079s) [upload=531240]: Done uploading segments... Dec 30 21:37:06.491006261 (606h8m6.362089307s) [upload=531240]: Upload finished. err: <nil> Duration: 52.707119171s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................................................X...] duration: 3.543360524s trace: https://jaeger.dev.storj.io/trace/22a47ec27f9a101c FAIL (2025-12-30 21:53:58 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.543383417s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [........X....X.X.........X..X...X.........X.................] duration: 18.555353543s trace: https://jaeger.dev.storj.io/trace/7b3713b537cd96b5 FAIL (2025-12-30 21:53:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: 3Y4O3DHH6VTTSFEDFLTGODSDJ2SINYTOVILJVKGHLEJRJAQGQRGQ) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: WPYVE54HEHASLRFMAHRNAGN4SBRLDDNNCLTUL6Q7IXCUTPKEQPJQ) context deadline exceeded ======LOGS====== Dec 30 21:37:24.517587467 (606h8m24.388670502s) [upload=531247]: Starting upload Dec 30 21:37:24.517683741 (606h8m24.388766776s) [upload=531247]: Done starting upload Dec 30 21:37:24.517750909 (606h8m24.388833964s) [upload=531247]: Uploading segments... Dec 30 21:37:24.517962091 (606h8m24.389045136s) [upload=531247]: Got next segment. Inline: false Dec 30 21:37:24.518006275 (606h8m24.38908932s) [upload=531247]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 21:37:24.518012496 (606h8m24.389095541s) [upload=531247]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 21:37:24.762005721 (606h8m24.633088796s) [seg_pos={0 0}, upload=531247]: Begin upload segment... Dec 30 21:37:24.762059043 (606h8m24.633142088s) [seg_pos={0 0}, upload=531247]: Done begin upload segment. Dec 30 21:37:24.762093669 (606h8m24.633176714s) [noise=true , address=145.79.227.13:4804 , piece_id=3T5EGEMHWDKHX3YMO46UAJLQLTXF72YYQROWVRGIJQMTXWWZTZUA, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.762105101 (606h8m24.633188136s) [noise=true , address=145.79.227.26:4654 , piece_id=YWMN73I2AGL5LMTBTDBR4REV35EPJ5FOC4FMQUMLHNYQBSM2R7YA, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.762162530 (606h8m24.633245575s) [noise=true , address=145.79.227.10:4403 , piece_id=NBFU5OK5RM4B5P53RMR6FH5XHHLYV3IMCAEEBF4DK7RQKF2YV56A, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.762806516 (606h8m24.633889581s) [noise=true , address=145.79.227.50:4374 , piece_id=4LK62BGBA6ZCAWL2K7JVJQN7VVZYVW7KHMFU6TOIAAXASEJQVP7Q, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.763247446 (606h8m24.634330491s) [noise=true , address=77.90.31.14:4662 , piece_id=3Y4O3DHH6VTTSFEDFLTGODSDJ2SINYTOVILJVKGHLEJRJAQGQRGQ, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.763647648 (606h8m24.634730683s) [noise=true , address=45.91.116.13:4155 , piece_id=WPYVE54HEHASLRFMAHRNAGN4SBRLDDNNCLTUL6Q7IXCUTPKEQPJQ, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.764127802 (606h8m24.635210847s) [noise=true , address=145.79.227.42:4121 , piece_id=M54EV7DTR4DH26KTXOQLG2IMQSRUKGOPVSXX2UEAHQGGGBLYCXWA, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.764542943 (606h8m24.635625978s) [noise=true , address=77.90.31.21:4880 , piece_id=MR3WNPHOPGXHD4GVIU4ONLKSPVYFPPWIINBKWIZWFENFVTZTSM3A, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.764959005 (606h8m24.63604205s) [noise=true , address=145.79.227.19:4079 , piece_id=W7EQPOU26CG3BPCWJGMTVV7KKDPOYA3HLYFT2HW2S4XJQZ6XUS5Q, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.766131597 (606h8m24.637214632s) [noise=true , address=45.91.116.10:4082 , piece_id=MYWKXEGVEDCGTIF26YFQTCLPLKXQUTFRE54AKLQ44PXV4EVR7ZHA, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.766690141 (606h8m24.637773176s) [noise=true , address=145.79.227.16:4911 , piece_id=ITFOWXPVEJU5SJX6A5EFQUMYFBCBV26EN4L3LA33AACJLJNQIPZQ, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.767199661 (606h8m24.638282696s) [noise=true , address=145.79.227.6:4258 , piece_id=LZPMJ6EZND6MF2MOMHUBNZ5SSJFYPJJYNW7UVQQEPYFEPLE5ZQCQ, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.767507377 (606h8m24.638590422s) [noise=true , address=145.79.227.37:4565 , piece_id=CNOZYEA5XVIFDAYZML66XRA3CVSUJJNWS3O75TIN4EZ7HTMCSNIQ, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.768056854 (606h8m24.639139899s) [noise=true , address=145.79.227.38:4979 , piece_id=DPTVSWBFBN2J3JO4QLRX4NJSDJE26AOW3QOSNXEDR6AFSVBDXCDA, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.768417160 (606h8m24.639500195s) [noise=true , address=145.79.227.40:4049 , piece_id=B54BGYWT6VSUUIGBVTAS6LHPYPEAK4LEX3HIGYLLBOPWONDBBYTA, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.768769200 (606h8m24.639852245s) [noise=true , address=145.79.227.24:4581 , piece_id=BYFXRVA7D4VCPXC3IGLN3T6GVHIIPB3MFICSXGDRKL2E5AUUW3WQ, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:26.469279868 (606h8m26.340362913s) [upload=531247]: Next returned nil segment Dec 30 21:37:26.469303293 (606h8m26.340386338s) [upload=531247]: Waiting for error group managing segments... Dec 30 21:37:26.756737396 (606h8m26.627820441s) [noise=true , address=145.79.227.10:4403 , piece_id=NBFU5OK5RM4B5P53RMR6FH5XHHLYV3IMCAEEBF4DK7RQKF2YV56A, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:26.768723167 (606h8m26.639806212s) [noise=true , address=145.79.227.42:4121 , piece_id=M54EV7DTR4DH26KTXOQLG2IMQSRUKGOPVSXX2UEAHQGGGBLYCXWA, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:26.768815023 (606h8m26.639898058s) [noise=true , address=145.79.227.38:4979 , piece_id=DPTVSWBFBN2J3JO4QLRX4NJSDJE26AOW3QOSNXEDR6AFSVBDXCDA, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:26.771265408 (606h8m26.642348453s) [noise=true , address=145.79.227.26:4654 , piece_id=YWMN73I2AGL5LMTBTDBR4REV35EPJ5FOC4FMQUMLHNYQBSM2R7YA, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:26.772261125 (606h8m26.64334416s) [noise=true , address=145.79.227.24:4581 , piece_id=BYFXRVA7D4VCPXC3IGLN3T6GVHIIPB3MFICSXGDRKL2E5AUUW3WQ, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:26.780818909 (606h8m26.651901954s) [noise=true , address=145.79.227.37:4565 , piece_id=CNOZYEA5XVIFDAYZML66XRA3CVSUJJNWS3O75TIN4EZ7HTMCSNIQ, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:26.781024501 (606h8m26.652107546s) [noise=true , address=145.79.227.50:4374 , piece_id=4LK62BGBA6ZCAWL2K7JVJQN7VVZYVW7KHMFU6TOIAAXASEJQVP7Q, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:26.781280018 (606h8m26.652363053s) [noise=true , address=145.79.227.13:4804 , piece_id=3T5EGEMHWDKHX3YMO46UAJLQLTXF72YYQROWVRGIJQMTXWWZTZUA, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:26.840740007 (606h8m26.711823042s) [noise=true , address=145.79.227.16:4911 , piece_id=ITFOWXPVEJU5SJX6A5EFQUMYFBCBV26EN4L3LA33AACJLJNQIPZQ, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:26.946038900 (606h8m26.817121936s) [noise=true , address=145.79.227.19:4079 , piece_id=W7EQPOU26CG3BPCWJGMTVV7KKDPOYA3HLYFT2HW2S4XJQZ6XUS5Q, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:27.027197388 (606h8m26.898280423s) [noise=true , address=77.90.31.14:4662 , piece_id=3Y4O3DHH6VTTSFEDFLTGODSDJ2SINYTOVILJVKGHLEJRJAQGQRGQ, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:27.065603629 (606h8m26.936686664s) [noise=true , address=45.91.116.10:4082 , piece_id=MYWKXEGVEDCGTIF26YFQTCLPLKXQUTFRE54AKLQ44PXV4EVR7ZHA, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:27.784121213 (606h8m27.655204248s) [noise=true , address=45.91.116.13:4155 , piece_id=WPYVE54HEHASLRFMAHRNAGN4SBRLDDNNCLTUL6Q7IXCUTPKEQPJQ, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:27.784142512 (606h8m27.655225558s) [seg_pos={0 0}, upload=531247]: Segment reached optimal threshold of 13 pieces. Dec 30 21:37:27.784300243 (606h8m27.655383278s) [noise=true , address=77.90.31.21:4880 , piece_id=MR3WNPHOPGXHD4GVIU4ONLKSPVYFPPWIINBKWIZWFENFVTZTSM3A, seg_pos={0 0}, upload=531247]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 30 21:37:27.784306134 (606h8m27.655389179s) [noise=true , address=77.90.31.21:4880 , piece_id=MR3WNPHOPGXHD4GVIU4ONLKSPVYFPPWIINBKWIZWFENFVTZTSM3A, seg_pos={0 0}, upload=531247]: Long tail cancellation due to optimal threshold Dec 30 21:37:27.784359966 (606h8m27.655443011s) [noise=true , address=145.79.227.40:4049 , piece_id=B54BGYWT6VSUUIGBVTAS6LHPYPEAK4LEX3HIGYLLBOPWONDBBYTA, seg_pos={0 0}, upload=531247]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 30 21:37:27.784370467 (606h8m27.655453522s) [noise=true , address=145.79.227.40:4049 , piece_id=B54BGYWT6VSUUIGBVTAS6LHPYPEAK4LEX3HIGYLLBOPWONDBBYTA, seg_pos={0 0}, upload=531247]: Long tail cancellation due to optimal threshold Dec 30 21:37:27.784429228 (606h8m27.655512263s) [noise=true , address=145.79.227.6:4258 , piece_id=LZPMJ6EZND6MF2MOMHUBNZ5SSJFYPJJYNW7UVQQEPYFEPLE5ZQCQ, seg_pos={0 0}, upload=531247]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF): protocol: write tcp 188.166.225.170:56840->145.79.227.6:4258: use of closed network connection; write tcp 188.166.225.170:56840->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56840->145.79.227.6:4258: use of closed network connection Dec 30 21:37:27.784436353 (606h8m27.655519398s) [noise=true , address=145.79.227.6:4258 , piece_id=LZPMJ6EZND6MF2MOMHUBNZ5SSJFYPJJYNW7UVQQEPYFEPLE5ZQCQ, seg_pos={0 0}, upload=531247]: Long tail cancellation due to optimal threshold Dec 30 21:37:27.784452513 (606h8m27.655535548s) [seg_pos={0 0}, upload=531247]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 21:37:27.784519210 (606h8m27.655602245s) [upload=531247]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 21:37:27.784523348 (606h8m27.655606393s) [upload=531247]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 21:37:28.083941079 (606h8m27.955024115s) [upload=531247]: Done uploading segments... Dec 30 21:37:28.083959936 (606h8m27.955042981s) [upload=531247]: Upload finished. err: <nil> Duration: 18.555379783s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................................................X..] duration: 3.509551317s trace: https://jaeger.dev.storj.io/trace/22a47ec27f9a101c FAIL (2025-12-30 21:52:52 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.509565754s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [X........X....X.X.........X..X...X.........X................] duration: 5.704199704s trace: https://jaeger.dev.storj.io/trace/7b3713b537cd96b5 FAIL (2025-12-30 21:52:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: 3Y4O3DHH6VTTSFEDFLTGODSDJ2SINYTOVILJVKGHLEJRJAQGQRGQ) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: WPYVE54HEHASLRFMAHRNAGN4SBRLDDNNCLTUL6Q7IXCUTPKEQPJQ) context deadline exceeded ======LOGS====== Dec 30 21:37:24.517587467 (606h8m24.388670502s) [upload=531247]: Starting upload Dec 30 21:37:24.517683741 (606h8m24.388766776s) [upload=531247]: Done starting upload Dec 30 21:37:24.517750909 (606h8m24.388833964s) [upload=531247]: Uploading segments... Dec 30 21:37:24.517962091 (606h8m24.389045136s) [upload=531247]: Got next segment. Inline: false Dec 30 21:37:24.518006275 (606h8m24.38908932s) [upload=531247]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 21:37:24.518012496 (606h8m24.389095541s) [upload=531247]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 21:37:24.762005721 (606h8m24.633088796s) [seg_pos={0 0}, upload=531247]: Begin upload segment... Dec 30 21:37:24.762059043 (606h8m24.633142088s) [seg_pos={0 0}, upload=531247]: Done begin upload segment. Dec 30 21:37:24.762093669 (606h8m24.633176714s) [noise=true , address=145.79.227.13:4804 , piece_id=3T5EGEMHWDKHX3YMO46UAJLQLTXF72YYQROWVRGIJQMTXWWZTZUA, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.762105101 (606h8m24.633188136s) [noise=true , address=145.79.227.26:4654 , piece_id=YWMN73I2AGL5LMTBTDBR4REV35EPJ5FOC4FMQUMLHNYQBSM2R7YA, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.762162530 (606h8m24.633245575s) [noise=true , address=145.79.227.10:4403 , piece_id=NBFU5OK5RM4B5P53RMR6FH5XHHLYV3IMCAEEBF4DK7RQKF2YV56A, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.762806516 (606h8m24.633889581s) [noise=true , address=145.79.227.50:4374 , piece_id=4LK62BGBA6ZCAWL2K7JVJQN7VVZYVW7KHMFU6TOIAAXASEJQVP7Q, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.763247446 (606h8m24.634330491s) [noise=true , address=77.90.31.14:4662 , piece_id=3Y4O3DHH6VTTSFEDFLTGODSDJ2SINYTOVILJVKGHLEJRJAQGQRGQ, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.763647648 (606h8m24.634730683s) [noise=true , address=45.91.116.13:4155 , piece_id=WPYVE54HEHASLRFMAHRNAGN4SBRLDDNNCLTUL6Q7IXCUTPKEQPJQ, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.764127802 (606h8m24.635210847s) [noise=true , address=145.79.227.42:4121 , piece_id=M54EV7DTR4DH26KTXOQLG2IMQSRUKGOPVSXX2UEAHQGGGBLYCXWA, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.764542943 (606h8m24.635625978s) [noise=true , address=77.90.31.21:4880 , piece_id=MR3WNPHOPGXHD4GVIU4ONLKSPVYFPPWIINBKWIZWFENFVTZTSM3A, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.764959005 (606h8m24.63604205s) [noise=true , address=145.79.227.19:4079 , piece_id=W7EQPOU26CG3BPCWJGMTVV7KKDPOYA3HLYFT2HW2S4XJQZ6XUS5Q, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.766131597 (606h8m24.637214632s) [noise=true , address=45.91.116.10:4082 , piece_id=MYWKXEGVEDCGTIF26YFQTCLPLKXQUTFRE54AKLQ44PXV4EVR7ZHA, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.766690141 (606h8m24.637773176s) [noise=true , address=145.79.227.16:4911 , piece_id=ITFOWXPVEJU5SJX6A5EFQUMYFBCBV26EN4L3LA33AACJLJNQIPZQ, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.767199661 (606h8m24.638282696s) [noise=true , address=145.79.227.6:4258 , piece_id=LZPMJ6EZND6MF2MOMHUBNZ5SSJFYPJJYNW7UVQQEPYFEPLE5ZQCQ, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.767507377 (606h8m24.638590422s) [noise=true , address=145.79.227.37:4565 , piece_id=CNOZYEA5XVIFDAYZML66XRA3CVSUJJNWS3O75TIN4EZ7HTMCSNIQ, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.768056854 (606h8m24.639139899s) [noise=true , address=145.79.227.38:4979 , piece_id=DPTVSWBFBN2J3JO4QLRX4NJSDJE26AOW3QOSNXEDR6AFSVBDXCDA, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.768417160 (606h8m24.639500195s) [noise=true , address=145.79.227.40:4049 , piece_id=B54BGYWT6VSUUIGBVTAS6LHPYPEAK4LEX3HIGYLLBOPWONDBBYTA, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:24.768769200 (606h8m24.639852245s) [noise=true , address=145.79.227.24:4581 , piece_id=BYFXRVA7D4VCPXC3IGLN3T6GVHIIPB3MFICSXGDRKL2E5AUUW3WQ, seg_pos={0 0}, upload=531247]: Uploading piece... Dec 30 21:37:26.469279868 (606h8m26.340362913s) [upload=531247]: Next returned nil segment Dec 30 21:37:26.469303293 (606h8m26.340386338s) [upload=531247]: Waiting for error group managing segments... Dec 30 21:37:26.756737396 (606h8m26.627820441s) [noise=true , address=145.79.227.10:4403 , piece_id=NBFU5OK5RM4B5P53RMR6FH5XHHLYV3IMCAEEBF4DK7RQKF2YV56A, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:26.768723167 (606h8m26.639806212s) [noise=true , address=145.79.227.42:4121 , piece_id=M54EV7DTR4DH26KTXOQLG2IMQSRUKGOPVSXX2UEAHQGGGBLYCXWA, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:26.768815023 (606h8m26.639898058s) [noise=true , address=145.79.227.38:4979 , piece_id=DPTVSWBFBN2J3JO4QLRX4NJSDJE26AOW3QOSNXEDR6AFSVBDXCDA, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:26.771265408 (606h8m26.642348453s) [noise=true , address=145.79.227.26:4654 , piece_id=YWMN73I2AGL5LMTBTDBR4REV35EPJ5FOC4FMQUMLHNYQBSM2R7YA, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:26.772261125 (606h8m26.64334416s) [noise=true , address=145.79.227.24:4581 , piece_id=BYFXRVA7D4VCPXC3IGLN3T6GVHIIPB3MFICSXGDRKL2E5AUUW3WQ, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:26.780818909 (606h8m26.651901954s) [noise=true , address=145.79.227.37:4565 , piece_id=CNOZYEA5XVIFDAYZML66XRA3CVSUJJNWS3O75TIN4EZ7HTMCSNIQ, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:26.781024501 (606h8m26.652107546s) [noise=true , address=145.79.227.50:4374 , piece_id=4LK62BGBA6ZCAWL2K7JVJQN7VVZYVW7KHMFU6TOIAAXASEJQVP7Q, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:26.781280018 (606h8m26.652363053s) [noise=true , address=145.79.227.13:4804 , piece_id=3T5EGEMHWDKHX3YMO46UAJLQLTXF72YYQROWVRGIJQMTXWWZTZUA, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:26.840740007 (606h8m26.711823042s) [noise=true , address=145.79.227.16:4911 , piece_id=ITFOWXPVEJU5SJX6A5EFQUMYFBCBV26EN4L3LA33AACJLJNQIPZQ, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:26.946038900 (606h8m26.817121936s) [noise=true , address=145.79.227.19:4079 , piece_id=W7EQPOU26CG3BPCWJGMTVV7KKDPOYA3HLYFT2HW2S4XJQZ6XUS5Q, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:27.027197388 (606h8m26.898280423s) [noise=true , address=77.90.31.14:4662 , piece_id=3Y4O3DHH6VTTSFEDFLTGODSDJ2SINYTOVILJVKGHLEJRJAQGQRGQ, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:27.065603629 (606h8m26.936686664s) [noise=true , address=45.91.116.10:4082 , piece_id=MYWKXEGVEDCGTIF26YFQTCLPLKXQUTFRE54AKLQ44PXV4EVR7ZHA, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:27.784121213 (606h8m27.655204248s) [noise=true , address=45.91.116.13:4155 , piece_id=WPYVE54HEHASLRFMAHRNAGN4SBRLDDNNCLTUL6Q7IXCUTPKEQPJQ, seg_pos={0 0}, upload=531247]: Done uploading piece. err: <nil> Dec 30 21:37:27.784142512 (606h8m27.655225558s) [seg_pos={0 0}, upload=531247]: Segment reached optimal threshold of 13 pieces. Dec 30 21:37:27.784300243 (606h8m27.655383278s) [noise=true , address=77.90.31.21:4880 , piece_id=MR3WNPHOPGXHD4GVIU4ONLKSPVYFPPWIINBKWIZWFENFVTZTSM3A, seg_pos={0 0}, upload=531247]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 30 21:37:27.784306134 (606h8m27.655389179s) [noise=true , address=77.90.31.21:4880 , piece_id=MR3WNPHOPGXHD4GVIU4ONLKSPVYFPPWIINBKWIZWFENFVTZTSM3A, seg_pos={0 0}, upload=531247]: Long tail cancellation due to optimal threshold Dec 30 21:37:27.784359966 (606h8m27.655443011s) [noise=true , address=145.79.227.40:4049 , piece_id=B54BGYWT6VSUUIGBVTAS6LHPYPEAK4LEX3HIGYLLBOPWONDBBYTA, seg_pos={0 0}, upload=531247]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 30 21:37:27.784370467 (606h8m27.655453522s) [noise=true , address=145.79.227.40:4049 , piece_id=B54BGYWT6VSUUIGBVTAS6LHPYPEAK4LEX3HIGYLLBOPWONDBBYTA, seg_pos={0 0}, upload=531247]: Long tail cancellation due to optimal threshold Dec 30 21:37:27.784429228 (606h8m27.655512263s) [noise=true , address=145.79.227.6:4258 , piece_id=LZPMJ6EZND6MF2MOMHUBNZ5SSJFYPJJYNW7UVQQEPYFEPLE5ZQCQ, seg_pos={0 0}, upload=531247]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF): protocol: write tcp 188.166.225.170:56840->145.79.227.6:4258: use of closed network connection; write tcp 188.166.225.170:56840->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56840->145.79.227.6:4258: use of closed network connection Dec 30 21:37:27.784436353 (606h8m27.655519398s) [noise=true , address=145.79.227.6:4258 , piece_id=LZPMJ6EZND6MF2MOMHUBNZ5SSJFYPJJYNW7UVQQEPYFEPLE5ZQCQ, seg_pos={0 0}, upload=531247]: Long tail cancellation due to optimal threshold Dec 30 21:37:27.784452513 (606h8m27.655535548s) [seg_pos={0 0}, upload=531247]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 21:37:27.784519210 (606h8m27.655602245s) [upload=531247]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 21:37:27.784523348 (606h8m27.655606393s) [upload=531247]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 21:37:28.083941079 (606h8m27.955024115s) [upload=531247]: Done uploading segments... Dec 30 21:37:28.083959936 (606h8m27.955042981s) [upload=531247]: Upload finished. err: <nil> Duration: 5.704290337s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [......................XX......X.........X...................] duration: 15.458350441s trace: https://jaeger.dev.storj.io/trace/5cdd5ba7b1cb22c7 FAIL (2025-12-30 21:52:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 21:37:03.355296401 (606h8m3.226379446s) [noise=true , address=145.79.227.22:4543 , piece_id=WWCZ6SEEXQ2KUJ32AGTIJNMSJROLYSJ4XQGBQJC4Q5DACZVA65RQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.355839374 (606h8m3.22692241s) [noise=true , address=145.79.227.35:4492 , piece_id=HCKFSQMJYEP6B5H4OX5AZ5TWP6Z4H4Z6WA2T7Z2H5KNZ4SDJRMUQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.356356930 (606h8m3.227439965s) [noise=true , address=45.91.116.13:4182 , piece_id=BYAYNP4JAAN3AW2LYSK74WJE4BRWUR5Z6K67MTLK6A73KEJ35WEQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.356675316 (606h8m3.227758361s) [noise=true , address=77.90.31.12:4612 , piece_id=63TS3SUEFUE6PORZ4W76OXRWFL7SBN773UGGY3DU7JPBAW66N7UA, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.356992732 (606h8m3.228075767s) [noise=true , address=64.49.13.140:4507 , piece_id=E7T6WL45F6GHECCRANX6AIKWFRZ77TURRWORYEYI43MET2WG7SRQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.357330644 (606h8m3.228413679s) [noise=true , address=77.90.31.21:4886 , piece_id=ZQRCPTVC3WFVQPZ36AU6KJW7PD3T557CXYQIICLDGOKK6JCPVZ5Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.357663649 (606h8m3.228746684s) [noise=true , address=145.79.227.49:4367 , piece_id=R2T2ABLXBPDVP3ZVA2TAYMCLQQ7JQUVQYCTHCKGMFBRVWCDAO4PQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.359827449 (606h8m3.230910484s) [noise=true , address=145.79.227.34:4918 , piece_id=HUMOHOT3HOYCBJ23WXQXWU3X6C6JQTRLLHTD4RQLCQTCB3HXI7EQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.360086692 (606h8m3.231169727s) [noise=true , address=145.79.227.7:4320 , piece_id=MAICNTHGLZ6W27HXDOOZXOBRGQGEXNSA4BJT2T5D5PHRVV6ZMX2Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.361622167 (606h8m3.232705212s) [noise=true , address=145.79.227.33:4888 , piece_id=I4426P3VETHPOL5WHWJEGVYRXJULTCNDX3WSUTS3XZVY36Q2N44Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.362965514 (606h8m3.234048559s) [noise=true , address=45.91.116.11:4133 , piece_id=NEW7ZOVP5E7HYGYD4LG5UG2MDLAQW2OBRTNZDFIYO4CQYMVAWRPQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.364308883 (606h8m3.235391928s) [noise=true , address=145.79.227.26:4663 , piece_id=DYEFQZL4X74QR2LAYURT5ZTR2IWIDB55VLBQOGZM36MA54P2SWJA, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.365734507 (606h8m3.236817542s) [noise=true , address=145.79.227.40:4059 , piece_id=7WSUBR3C3PWVBCWD6KRGSD6MPX4IH2YQOAVO4J4SWZORNHGS2IPQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.366982764 (606h8m3.238065799s) [noise=true , address=145.79.227.29:4973 , piece_id=5OROXELR55A47BFPSJJEWVQU4YW6CAOK6LMEHCINM4FN3OVI2Y6Q, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:03.368390294 (606h8m3.239473339s) [noise=true , address=77.90.31.19:4852 , piece_id=2NYO63GCBGCYBFYTMUGNF2NG5FSJOD3AOUOD2FZZIWDINK4X6DJQ, seg_pos={0 0}, upload=531240]: Uploading piece... Dec 30 21:37:04.736295138 (606h8m4.607378183s) [upload=531240]: Next returned nil segment Dec 30 21:37:04.736311850 (606h8m4.607394885s) [upload=531240]: Waiting for error group managing segments... Dec 30 21:37:05.104989591 (606h8m4.976072636s) [noise=true , address=64.49.13.140:4507 , piece_id=E7T6WL45F6GHECCRANX6AIKWFRZ77TURRWORYEYI43MET2WG7SRQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.208809748 (606h8m5.079892793s) [noise=true , address=145.79.227.22:4543 , piece_id=WWCZ6SEEXQ2KUJ32AGTIJNMSJROLYSJ4XQGBQJC4Q5DACZVA65RQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.441125389 (606h8m5.312208444s) [noise=true , address=77.90.31.19:4852 , piece_id=2NYO63GCBGCYBFYTMUGNF2NG5FSJOD3AOUOD2FZZIWDINK4X6DJQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.483630904 (606h8m5.35471394s) [noise=true , address=77.90.31.12:4612 , piece_id=63TS3SUEFUE6PORZ4W76OXRWFL7SBN773UGGY3DU7JPBAW66N7UA, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.494478811 (606h8m5.365561846s) [noise=true , address=77.90.31.11:4559 , piece_id=3DYD4CNPJQN3S4EYZJGREQTTZJDSL2QTC3OT6G2VFUGMLSVCZUFA, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.497452683 (606h8m5.368535729s) [noise=true , address=77.90.31.17:4769 , piece_id=QXDYESBKGXRY7BCFYYF2O7OXS7Y4Z4KUMJB6AXZAWX735SYU56GQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.661529377 (606h8m5.532612412s) [noise=true , address=45.91.116.13:4182 , piece_id=BYAYNP4JAAN3AW2LYSK74WJE4BRWUR5Z6K67MTLK6A73KEJ35WEQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.729773112 (606h8m5.600856157s) [noise=true , address=45.91.116.11:4133 , piece_id=NEW7ZOVP5E7HYGYD4LG5UG2MDLAQW2OBRTNZDFIYO4CQYMVAWRPQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:05.940743308 (606h8m5.811826353s) [noise=true , address=77.90.31.21:4886 , piece_id=ZQRCPTVC3WFVQPZ36AU6KJW7PD3T557CXYQIICLDGOKK6JCPVZ5Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.084240044 (606h8m5.95532309s) [noise=true , address=145.79.227.30:4784 , piece_id=DSXUDNYCWO5F6IYXMIVJNEASYXIZ7EO7RKAS7QRUYK5OU7GOOB5Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.121095032 (606h8m5.992178067s) [noise=true , address=145.79.227.40:4059 , piece_id=7WSUBR3C3PWVBCWD6KRGSD6MPX4IH2YQOAVO4J4SWZORNHGS2IPQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.169195983 (606h8m6.040279028s) [noise=true , address=145.79.227.49:4367 , piece_id=R2T2ABLXBPDVP3ZVA2TAYMCLQQ7JQUVQYCTHCKGMFBRVWCDAO4PQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.169380634 (606h8m6.040463679s) [noise=true , address=145.79.227.33:4888 , piece_id=I4426P3VETHPOL5WHWJEGVYRXJULTCNDX3WSUTS3XZVY36Q2N44Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.179060175 (606h8m6.050143211s) [noise=true , address=145.79.227.29:4973 , piece_id=5OROXELR55A47BFPSJJEWVQU4YW6CAOK6LMEHCINM4FN3OVI2Y6Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: <nil> Dec 30 21:37:06.179080424 (606h8m6.050163459s) [seg_pos={0 0}, upload=531240]: Segment reached optimal threshold of 14 pieces. Dec 30 21:37:06.179262801 (606h8m6.050345837s) [noise=true , address=145.79.227.26:4663 , piece_id=DYEFQZL4X74QR2LAYURT5ZTR2IWIDB55VLBQOGZM36MA54P2SWJA, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SwpAaXTA3BqUvwubzmJkvwokdz6SSpT9t8HPtQcXXJxmcgfm2): protocol: write tcp 188.166.225.170:43184->145.79.227.26:4663: use of closed network connection; write tcp 188.166.225.170:43184->145.79.227.26:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43184->145.79.227.26:4663: use of closed network connection Dec 30 21:37:06.179268703 (606h8m6.050351748s) [noise=true , address=145.79.227.26:4663 , piece_id=DYEFQZL4X74QR2LAYURT5ZTR2IWIDB55VLBQOGZM36MA54P2SWJA, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179363724 (606h8m6.050446769s) [noise=true , address=145.79.227.7:4320 , piece_id=MAICNTHGLZ6W27HXDOOZXOBRGQGEXNSA4BJT2T5D5PHRVV6ZMX2Q, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126rEgnQw1w7hsFpRqQP9nLaumdBfdtkgYjdw6WE6nXrxk5kj4B): protocol: write tcp 188.166.225.170:45228->145.79.227.7:4320: use of closed network connection; write tcp 188.166.225.170:45228->145.79.227.7:4320: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45228->145.79.227.7:4320: use of closed network connection Dec 30 21:37:06.179369525 (606h8m6.05045257s) [noise=true , address=145.79.227.7:4320 , piece_id=MAICNTHGLZ6W27HXDOOZXOBRGQGEXNSA4BJT2T5D5PHRVV6ZMX2Q, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179532996 (606h8m6.050616061s) [noise=true , address=145.79.227.34:4918 , piece_id=HUMOHOT3HOYCBJ23WXQXWU3X6C6JQTRLLHTD4RQLCQTCB3HXI7EQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF Dec 30 21:37:06.179572371 (606h8m6.050655406s) [noise=true , address=145.79.227.34:4918 , piece_id=HUMOHOT3HOYCBJ23WXQXWU3X6C6JQTRLLHTD4RQLCQTCB3HXI7EQ, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179638587 (606h8m6.050721622s) [noise=true , address=145.79.227.35:4492 , piece_id=HCKFSQMJYEP6B5H4OX5AZ5TWP6Z4H4Z6WA2T7Z2H5KNZ4SDJRMUQ, seg_pos={0 0}, upload=531240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1roa5GGYmqCX2AxSrqFgqyV2CtBEYZavuGTseiwDy4pjkT6bip): protocol: expected piece hash; context canceled; EOF Dec 30 21:37:06.179647855 (606h8m6.05073089s) [noise=true , address=145.79.227.35:4492 , piece_id=HCKFSQMJYEP6B5H4OX5AZ5TWP6Z4H4Z6WA2T7Z2H5KNZ4SDJRMUQ, seg_pos={0 0}, upload=531240]: Long tail cancellation due to optimal threshold Dec 30 21:37:06.179666641 (606h8m6.050749686s) [seg_pos={0 0}, upload=531240]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 21:37:06.179742766 (606h8m6.050825811s) [upload=531240]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 21:37:06.179747545 (606h8m6.05083059s) [upload=531240]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 21:37:06.490986034 (606h8m6.362069079s) [upload=531240]: Done uploading segments... Dec 30 21:37:06.491006261 (606h8m6.362089307s) [upload=531240]: Upload finished. err: <nil> Duration: 15.458368906s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................................................X.] duration: 3.587457369s trace: https://jaeger.dev.storj.io/trace/22a47ec27f9a101c FAIL (2025-12-30 21:51:50 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.587484711s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)