Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure

Orbiter Check History

Showing 50 of 335937 reports (Page 121 of 6719)
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
34 out of 60 is failed (threshold=6) [XXX..X..XX.XXXX.....XXX.X.XXXX.X.XXXX...XXXX.XXXXX...X.....X]
duration: 
5.001521547s
trace: 
https://jaeger.dev.storj.io/trace/57ec6320e984041b
FAIL (2025-12-27 15:32:43 UTC) 
orbiter warning: context deadline exceeded
Duration: 5.001575428s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
20 out of 60 is failed (threshold=3) [.......X..X.X..X...XXXXXXX..X.X..X.X.....X...X......X.X..X..]
duration: 
5.706425235s
trace: 
https://jaeger.dev.storj.io/trace/522a43b73399bbba
FAIL (2025-12-27 15:32:39 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 52W7HESUL4YA6OMET3H64EJXHDBESCUC5VAJEEEXEPPM6HBNW7BQ) context deadline exceeded; piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: AQFGSNIIUH5ZAA5DSUNN7GZTV3JJ3YRQTVY5XLDLPHPCIYQ4U6QQ) context deadline exceeded
======LOGS======
2m34.086203997s) [upload=463153]: Uploading segments...
Dec 27 15:31:34.215346692 (528h2m34.086429737s) [upload=463153]: Got next segment. Inline: false
Dec 27 15:31:34.215374916 (528h2m34.086457951s) [upload=463153]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 15:31:34.215379064 (528h2m34.086462109s) [upload=463153]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 15:31:34.462608585 (528h2m34.33369162s) [seg_pos={0 0}, upload=463153]: Begin upload segment...
Dec 27 15:31:34.462659352 (528h2m34.333742398s) [seg_pos={0 0}, upload=463153]: Done begin upload segment.
Dec 27 15:31:34.462748422 (528h2m34.333831467s) [noise=true , address=145.79.227.52:4445   , piece_id=X2ON65EDXEJVR2NXKMUYGJYJMA62KZ3QXG4VBOUGFDFSG56VWF3A, seg_pos={0 0}, upload=463153]: Uploading piece...
Dec 27 15:31:34.462796123 (528h2m34.333879168s) [noise=true , address=77.90.31.14:4661     , piece_id=AQFGSNIIUH5ZAA5DSUNN7GZTV3JJ3YRQTVY5XLDLPHPCIYQ4U6QQ, seg_pos={0 0}, upload=463153]: Uploading piece...
Dec 27 15:31:34.463437474 (528h2m34.334520519s) [noise=true , address=145.79.227.42:4123   , piece_id=YFKS7437G3M3SMTEHSTUYXMZO2CS7SCZ4YHZODSH4FSL5D64OENA, seg_pos={0 0}, upload=463153]: Uploading piece...
Dec 27 15:31:34.463844980 (528h2m34.334928025s) [noise=true , address=145.79.227.51:4409   , piece_id=CHN6PACDXRAKJECHHWIQWYMXNIOMZCSX6X6YBPJTFBMR5UXC35JA, seg_pos={0 0}, upload=463153]: Uploading piece...
Dec 27 15:31:34.464152144 (528h2m34.335235179s) [noise=true , address=145.79.227.38:4979   , piece_id=UN7DNPRLE2HIO2B3KUXX5Z5554JFMB6XR7ELGUGVGVBZFTXEDODQ, seg_pos={0 0}, upload=463153]: Uploading piece...
Dec 27 15:31:34.464449932 (528h2m34.335532967s) [noise=true , address=145.79.227.11:4729   , piece_id=JWNDP4YNP5FHTRASKVF2LJHZASNOK7T5KRYLEMEIDUGNC2TZ4WCQ, seg_pos={0 0}, upload=463153]: Uploading piece...
Dec 27 15:31:34.464789368 (528h2m34.335872413s) [noise=true , address=145.79.227.3:4149    , piece_id=32LS4UUSAWIDNE6UE4GP2YIXPILHS5IBN6OAPAZ5VA6SQVK42WOQ, seg_pos={0 0}, upload=463153]: Uploading piece...
Dec 27 15:31:34.465139475 (528h2m34.33622251s) [noise=true , address=45.91.116.4:4937     , piece_id=B2U3DALRGH5U2FJLQTTVD2C345ZKIWXP3F57PFDBQNBJAPQY4ARQ, seg_pos={0 0}, upload=463153]: Uploading piece...
Dec 27 15:31:34.465596745 (528h2m34.33667979s) [noise=true , address=45.91.116.10:4084    , piece_id=JUKG7B5WVJ7PWWINVH6VLQH4HVCYHG6IDO2ZD5Y5LBCIJRUCEVMQ, seg_pos={0 0}, upload=463153]: Uploading piece...
Dec 27 15:31:34.467395972 (528h2m34.338479017s) [noise=true , address=45.91.116.12:4225    , piece_id=X2CMM65PCOTDV6SIR6ZNN7F72OICXTQ44CHCKBH62CE45K4EXKJA, seg_pos={0 0}, upload=463153]: Uploading piece...
Dec 27 15:31:34.468960661 (528h2m34.340043706s) [noise=true , address=145.79.227.33:4869   , piece_id=CNIVSRNCUSZJFORFT4SSJLEYACBKLZSW6YPTZPAVC5SHDLJKLPOA, seg_pos={0 0}, upload=463153]: Uploading piece...
Dec 27 15:31:34.470669756 (528h2m34.341752801s) [noise=true , address=145.79.227.50:4373   , piece_id=3QOJVY4KLUIVWT5CYJ3PHS4ZYW4DFYMIB375UR2HBNB4FJEYDMBA, seg_pos={0 0}, upload=463153]: Uploading piece...
Dec 27 15:31:34.472588489 (528h2m34.343671524s) [noise=true , address=145.79.227.36:4519   , piece_id=UEZWKVXIDLC2JIMRPYDFBBES25JWBFEO3JVQ4ESQDOPPXWASPLFQ, seg_pos={0 0}, upload=463153]: Uploading piece...
Dec 27 15:31:34.475052040 (528h2m34.346135075s) [noise=true , address=145.79.227.37:4565   , piece_id=5ZCRHMG2CQPEM4A5R5ZWLUB4KZO2VHD2TDT4AP7Q6IKBYR7YABZQ, seg_pos={0 0}, upload=463153]: Uploading piece...
Dec 27 15:31:34.478883847 (528h2m34.349966892s) [noise=true , address=45.91.116.8:4048     , piece_id=52W7HESUL4YA6OMET3H64EJXHDBESCUC5VAJEEEXEPPM6HBNW7BQ, seg_pos={0 0}, upload=463153]: Uploading piece...
Dec 27 15:31:34.482498600 (528h2m34.353581645s) [noise=true , address=145.79.227.48:4301   , piece_id=2W7VO7654QKTKK5YFWBOTK4H57I3PJ4UIUEISTLOIR2F5WY6BSTQ, seg_pos={0 0}, upload=463153]: Uploading piece...
Dec 27 15:31:36.079768051 (528h2m35.950851096s) [upload=463153]: Next returned nil segment
Dec 27 15:31:36.079782369 (528h2m35.950865414s) [upload=463153]: Waiting for error group managing segments...
Dec 27 15:31:36.456664874 (528h2m36.327747919s) [noise=true , address=145.79.227.51:4409   , piece_id=CHN6PACDXRAKJECHHWIQWYMXNIOMZCSX6X6YBPJTFBMR5UXC35JA, seg_pos={0 0}, upload=463153]: Done uploading piece. err: <nil>
Dec 27 15:31:36.465589868 (528h2m36.336672913s) [noise=true , address=145.79.227.3:4149    , piece_id=32LS4UUSAWIDNE6UE4GP2YIXPILHS5IBN6OAPAZ5VA6SQVK42WOQ, seg_pos={0 0}, upload=463153]: Done uploading piece. err: <nil>
Dec 27 15:31:36.470502723 (528h2m36.341585778s) [noise=true , address=145.79.227.11:4729   , piece_id=JWNDP4YNP5FHTRASKVF2LJHZASNOK7T5KRYLEMEIDUGNC2TZ4WCQ, seg_pos={0 0}, upload=463153]: Done uploading piece. err: <nil>
Dec 27 15:31:36.478656260 (528h2m36.349739305s) [noise=true , address=145.79.227.42:4123   , piece_id=YFKS7437G3M3SMTEHSTUYXMZO2CS7SCZ4YHZODSH4FSL5D64OENA, seg_pos={0 0}, upload=463153]: Done uploading piece. err: <nil>
Dec 27 15:31:36.480936943 (528h2m36.352019978s) [noise=true , address=145.79.227.50:4373   , piece_id=3QOJVY4KLUIVWT5CYJ3PHS4ZYW4DFYMIB375UR2HBNB4FJEYDMBA, seg_pos={0 0}, upload=463153]: Done uploading piece. err: <nil>
Dec 27 15:31:36.492065783 (528h2m36.363148818s) [noise=true , address=145.79.227.38:4979   , piece_id=UN7DNPRLE2HIO2B3KUXX5Z5554JFMB6XR7ELGUGVGVBZFTXEDODQ, seg_pos={0 0}, upload=463153]: Done uploading piece. err: <nil>
Dec 27 15:31:36.494214175 (528h2m36.36529722s) [noise=true , address=145.79.227.52:4445   , piece_id=X2ON65EDXEJVR2NXKMUYGJYJMA62KZ3QXG4VBOUGFDFSG56VWF3A, seg_pos={0 0}, upload=463153]: Done uploading piece. err: <nil>
Dec 27 15:31:36.496700709 (528h2m36.367783744s) [noise=true , address=145.79.227.48:4301   , piece_id=2W7VO7654QKTKK5YFWBOTK4H57I3PJ4UIUEISTLOIR2F5WY6BSTQ, seg_pos={0 0}, upload=463153]: Done uploading piece. err: <nil>
Dec 27 15:31:36.508879120 (528h2m36.379962165s) [noise=true , address=145.79.227.37:4565   , piece_id=5ZCRHMG2CQPEM4A5R5ZWLUB4KZO2VHD2TDT4AP7Q6IKBYR7YABZQ, seg_pos={0 0}, upload=463153]: Done uploading piece. err: <nil>
Dec 27 15:31:36.511295511 (528h2m36.382378556s) [noise=true , address=145.79.227.36:4519   , piece_id=UEZWKVXIDLC2JIMRPYDFBBES25JWBFEO3JVQ4ESQDOPPXWASPLFQ, seg_pos={0 0}, upload=463153]: Done uploading piece. err: <nil>
Dec 27 15:31:36.673785822 (528h2m36.544868857s) [noise=true , address=77.90.31.14:4661     , piece_id=AQFGSNIIUH5ZAA5DSUNN7GZTV3JJ3YRQTVY5XLDLPHPCIYQ4U6QQ, seg_pos={0 0}, upload=463153]: Done uploading piece. err: <nil>
Dec 27 15:31:37.005236026 (528h2m36.876319081s) [noise=true , address=45.91.116.4:4937     , piece_id=B2U3DALRGH5U2FJLQTTVD2C345ZKIWXP3F57PFDBQNBJAPQY4ARQ, seg_pos={0 0}, upload=463153]: Done uploading piece. err: <nil>
Dec 27 15:31:37.081575021 (528h2m36.952658056s) [noise=true , address=45.91.116.8:4048     , piece_id=52W7HESUL4YA6OMET3H64EJXHDBESCUC5VAJEEEXEPPM6HBNW7BQ, seg_pos={0 0}, upload=463153]: Done uploading piece. err: <nil>
Dec 27 15:31:37.081596342 (528h2m36.952679377s) [seg_pos={0 0}, upload=463153]: Segment reached optimal threshold of 13 pieces.
Dec 27 15:31:37.081961918 (528h2m36.953044963s) [noise=true , address=145.79.227.33:4869   , piece_id=CNIVSRNCUSZJFORFT4SSJLEYACBKLZSW6YPTZPAVC5SHDLJKLPOA, seg_pos={0 0}, upload=463153]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF
Dec 27 15:31:37.081972548 (528h2m36.953055593s) [noise=true , address=145.79.227.33:4869   , piece_id=CNIVSRNCUSZJFORFT4SSJLEYACBKLZSW6YPTZPAVC5SHDLJKLPOA, seg_pos={0 0}, upload=463153]: Long tail cancellation due to optimal threshold
Dec 27 15:31:37.082138894 (528h2m36.95322194s) [noise=true , address=45.91.116.10:4084    , piece_id=JUKG7B5WVJ7PWWINVH6VLQH4HVCYHG6IDO2ZD5Y5LBCIJRUCEVMQ, seg_pos={0 0}, upload=463153]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: write tcp 188.166.225.170:36398->45.91.116.10:4084: use of closed network connection; write tcp 188.166.225.170:36398->45.91.116.10:4084: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36398->45.91.116.10:4084: use of closed network connection
Dec 27 15:31:37.082155006 (528h2m36.953238041s) [noise=true , address=45.91.116.10:4084    , piece_id=JUKG7B5WVJ7PWWINVH6VLQH4HVCYHG6IDO2ZD5Y5LBCIJRUCEVMQ, seg_pos={0 0}, upload=463153]: Long tail cancellation due to optimal threshold
Dec 27 15:31:37.082289120 (528h2m36.953372155s) [noise=true , address=45.91.116.12:4225    , piece_id=X2CMM65PCOTDV6SIR6ZNN7F72OICXTQ44CHCKBH62CE45K4EXKJA, seg_pos={0 0}, upload=463153]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:58418->45.91.116.12:4225: use of closed network connection; write tcp 188.166.225.170:58418->45.91.116.12:4225: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58418->45.91.116.12:4225: use of closed network connection
Dec 27 15:31:37.082299571 (528h2m36.953382626s) [noise=true , address=45.91.116.12:4225    , piece_id=X2CMM65PCOTDV6SIR6ZNN7F72OICXTQ44CHCKBH62CE45K4EXKJA, seg_pos={0 0}, upload=463153]: Long tail cancellation due to optimal threshold
Dec 27 15:31:37.082325521 (528h2m36.953408566s) [seg_pos={0 0}, upload=463153]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 15:31:37.082441892 (528h2m36.953524937s) [upload=463153]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 15:31:37.082447492 (528h2m36.953530527s) [upload=463153]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 15:31:37.380252539 (528h2m37.251335575s) [upload=463153]: Done uploading segments...
Dec 27 15:31:37.380273720 (528h2m37.251356765s) [upload=463153]: Upload finished. err: <nil>
Duration: 5.706454341s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
21 out of 60 is failed (threshold=3) [X.......X..X.X..X...XXXXXXX..X.X..X.X.....X...X......X.X..X.]
duration: 
6.102248738s
trace: 
https://jaeger.dev.storj.io/trace/522a43b73399bbba
FAIL (2025-12-27 15:32:09 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 52W7HESUL4YA6OMET3H64EJXHDBESCUC5VAJEEEXEPPM6HBNW7BQ) context deadline exceeded; piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: AQFGSNIIUH5ZAA5DSUNN7GZTV3JJ3YRQTVY5XLDLPHPCIYQ4U6QQ) context deadline exceeded
======LOGS======
2m34.086203997s) [upload=463153]: Uploading segments...
Dec 27 15:31:34.215346692 (528h2m34.086429737s) [upload=463153]: Got next segment. Inline: false
Dec 27 15:31:34.215374916 (528h2m34.086457951s) [upload=463153]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 15:31:34.215379064 (528h2m34.086462109s) [upload=463153]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 15:31:34.462608585 (528h2m34.33369162s) [seg_pos={0 0}, upload=463153]: Begin upload segment...
Dec 27 15:31:34.462659352 (528h2m34.333742398s) [seg_pos={0 0}, upload=463153]: Done begin upload segment.
Dec 27 15:31:34.462748422 (528h2m34.333831467s) [noise=true , address=145.79.227.52:4445   , piece_id=X2ON65EDXEJVR2NXKMUYGJYJMA62KZ3QXG4VBOUGFDFSG56VWF3A, seg_pos={0 0}, upload=463153]: Uploading piece...
Dec 27 15:31:34.462796123 (528h2m34.333879168s) [noise=true , address=77.90.31.14:4661     , piece_id=AQFGSNIIUH5ZAA5DSUNN7GZTV3JJ3YRQTVY5XLDLPHPCIYQ4U6QQ, seg_pos={0 0}, upload=463153]: Uploading piece...
Dec 27 15:31:34.463437474 (528h2m34.334520519s) [noise=true , address=145.79.227.42:4123   , piece_id=YFKS7437G3M3SMTEHSTUYXMZO2CS7SCZ4YHZODSH4FSL5D64OENA, seg_pos={0 0}, upload=463153]: Uploading piece...
Dec 27 15:31:34.463844980 (528h2m34.334928025s) [noise=true , address=145.79.227.51:4409   , piece_id=CHN6PACDXRAKJECHHWIQWYMXNIOMZCSX6X6YBPJTFBMR5UXC35JA, seg_pos={0 0}, upload=463153]: Uploading piece...
Dec 27 15:31:34.464152144 (528h2m34.335235179s) [noise=true , address=145.79.227.38:4979   , piece_id=UN7DNPRLE2HIO2B3KUXX5Z5554JFMB6XR7ELGUGVGVBZFTXEDODQ, seg_pos={0 0}, upload=463153]: Uploading piece...
Dec 27 15:31:34.464449932 (528h2m34.335532967s) [noise=true , address=145.79.227.11:4729   , piece_id=JWNDP4YNP5FHTRASKVF2LJHZASNOK7T5KRYLEMEIDUGNC2TZ4WCQ, seg_pos={0 0}, upload=463153]: Uploading piece...
Dec 27 15:31:34.464789368 (528h2m34.335872413s) [noise=true , address=145.79.227.3:4149    , piece_id=32LS4UUSAWIDNE6UE4GP2YIXPILHS5IBN6OAPAZ5VA6SQVK42WOQ, seg_pos={0 0}, upload=463153]: Uploading piece...
Dec 27 15:31:34.465139475 (528h2m34.33622251s) [noise=true , address=45.91.116.4:4937     , piece_id=B2U3DALRGH5U2FJLQTTVD2C345ZKIWXP3F57PFDBQNBJAPQY4ARQ, seg_pos={0 0}, upload=463153]: Uploading piece...
Dec 27 15:31:34.465596745 (528h2m34.33667979s) [noise=true , address=45.91.116.10:4084    , piece_id=JUKG7B5WVJ7PWWINVH6VLQH4HVCYHG6IDO2ZD5Y5LBCIJRUCEVMQ, seg_pos={0 0}, upload=463153]: Uploading piece...
Dec 27 15:31:34.467395972 (528h2m34.338479017s) [noise=true , address=45.91.116.12:4225    , piece_id=X2CMM65PCOTDV6SIR6ZNN7F72OICXTQ44CHCKBH62CE45K4EXKJA, seg_pos={0 0}, upload=463153]: Uploading piece...
Dec 27 15:31:34.468960661 (528h2m34.340043706s) [noise=true , address=145.79.227.33:4869   , piece_id=CNIVSRNCUSZJFORFT4SSJLEYACBKLZSW6YPTZPAVC5SHDLJKLPOA, seg_pos={0 0}, upload=463153]: Uploading piece...
Dec 27 15:31:34.470669756 (528h2m34.341752801s) [noise=true , address=145.79.227.50:4373   , piece_id=3QOJVY4KLUIVWT5CYJ3PHS4ZYW4DFYMIB375UR2HBNB4FJEYDMBA, seg_pos={0 0}, upload=463153]: Uploading piece...
Dec 27 15:31:34.472588489 (528h2m34.343671524s) [noise=true , address=145.79.227.36:4519   , piece_id=UEZWKVXIDLC2JIMRPYDFBBES25JWBFEO3JVQ4ESQDOPPXWASPLFQ, seg_pos={0 0}, upload=463153]: Uploading piece...
Dec 27 15:31:34.475052040 (528h2m34.346135075s) [noise=true , address=145.79.227.37:4565   , piece_id=5ZCRHMG2CQPEM4A5R5ZWLUB4KZO2VHD2TDT4AP7Q6IKBYR7YABZQ, seg_pos={0 0}, upload=463153]: Uploading piece...
Dec 27 15:31:34.478883847 (528h2m34.349966892s) [noise=true , address=45.91.116.8:4048     , piece_id=52W7HESUL4YA6OMET3H64EJXHDBESCUC5VAJEEEXEPPM6HBNW7BQ, seg_pos={0 0}, upload=463153]: Uploading piece...
Dec 27 15:31:34.482498600 (528h2m34.353581645s) [noise=true , address=145.79.227.48:4301   , piece_id=2W7VO7654QKTKK5YFWBOTK4H57I3PJ4UIUEISTLOIR2F5WY6BSTQ, seg_pos={0 0}, upload=463153]: Uploading piece...
Dec 27 15:31:36.079768051 (528h2m35.950851096s) [upload=463153]: Next returned nil segment
Dec 27 15:31:36.079782369 (528h2m35.950865414s) [upload=463153]: Waiting for error group managing segments...
Dec 27 15:31:36.456664874 (528h2m36.327747919s) [noise=true , address=145.79.227.51:4409   , piece_id=CHN6PACDXRAKJECHHWIQWYMXNIOMZCSX6X6YBPJTFBMR5UXC35JA, seg_pos={0 0}, upload=463153]: Done uploading piece. err: <nil>
Dec 27 15:31:36.465589868 (528h2m36.336672913s) [noise=true , address=145.79.227.3:4149    , piece_id=32LS4UUSAWIDNE6UE4GP2YIXPILHS5IBN6OAPAZ5VA6SQVK42WOQ, seg_pos={0 0}, upload=463153]: Done uploading piece. err: <nil>
Dec 27 15:31:36.470502723 (528h2m36.341585778s) [noise=true , address=145.79.227.11:4729   , piece_id=JWNDP4YNP5FHTRASKVF2LJHZASNOK7T5KRYLEMEIDUGNC2TZ4WCQ, seg_pos={0 0}, upload=463153]: Done uploading piece. err: <nil>
Dec 27 15:31:36.478656260 (528h2m36.349739305s) [noise=true , address=145.79.227.42:4123   , piece_id=YFKS7437G3M3SMTEHSTUYXMZO2CS7SCZ4YHZODSH4FSL5D64OENA, seg_pos={0 0}, upload=463153]: Done uploading piece. err: <nil>
Dec 27 15:31:36.480936943 (528h2m36.352019978s) [noise=true , address=145.79.227.50:4373   , piece_id=3QOJVY4KLUIVWT5CYJ3PHS4ZYW4DFYMIB375UR2HBNB4FJEYDMBA, seg_pos={0 0}, upload=463153]: Done uploading piece. err: <nil>
Dec 27 15:31:36.492065783 (528h2m36.363148818s) [noise=true , address=145.79.227.38:4979   , piece_id=UN7DNPRLE2HIO2B3KUXX5Z5554JFMB6XR7ELGUGVGVBZFTXEDODQ, seg_pos={0 0}, upload=463153]: Done uploading piece. err: <nil>
Dec 27 15:31:36.494214175 (528h2m36.36529722s) [noise=true , address=145.79.227.52:4445   , piece_id=X2ON65EDXEJVR2NXKMUYGJYJMA62KZ3QXG4VBOUGFDFSG56VWF3A, seg_pos={0 0}, upload=463153]: Done uploading piece. err: <nil>
Dec 27 15:31:36.496700709 (528h2m36.367783744s) [noise=true , address=145.79.227.48:4301   , piece_id=2W7VO7654QKTKK5YFWBOTK4H57I3PJ4UIUEISTLOIR2F5WY6BSTQ, seg_pos={0 0}, upload=463153]: Done uploading piece. err: <nil>
Dec 27 15:31:36.508879120 (528h2m36.379962165s) [noise=true , address=145.79.227.37:4565   , piece_id=5ZCRHMG2CQPEM4A5R5ZWLUB4KZO2VHD2TDT4AP7Q6IKBYR7YABZQ, seg_pos={0 0}, upload=463153]: Done uploading piece. err: <nil>
Dec 27 15:31:36.511295511 (528h2m36.382378556s) [noise=true , address=145.79.227.36:4519   , piece_id=UEZWKVXIDLC2JIMRPYDFBBES25JWBFEO3JVQ4ESQDOPPXWASPLFQ, seg_pos={0 0}, upload=463153]: Done uploading piece. err: <nil>
Dec 27 15:31:36.673785822 (528h2m36.544868857s) [noise=true , address=77.90.31.14:4661     , piece_id=AQFGSNIIUH5ZAA5DSUNN7GZTV3JJ3YRQTVY5XLDLPHPCIYQ4U6QQ, seg_pos={0 0}, upload=463153]: Done uploading piece. err: <nil>
Dec 27 15:31:37.005236026 (528h2m36.876319081s) [noise=true , address=45.91.116.4:4937     , piece_id=B2U3DALRGH5U2FJLQTTVD2C345ZKIWXP3F57PFDBQNBJAPQY4ARQ, seg_pos={0 0}, upload=463153]: Done uploading piece. err: <nil>
Dec 27 15:31:37.081575021 (528h2m36.952658056s) [noise=true , address=45.91.116.8:4048     , piece_id=52W7HESUL4YA6OMET3H64EJXHDBESCUC5VAJEEEXEPPM6HBNW7BQ, seg_pos={0 0}, upload=463153]: Done uploading piece. err: <nil>
Dec 27 15:31:37.081596342 (528h2m36.952679377s) [seg_pos={0 0}, upload=463153]: Segment reached optimal threshold of 13 pieces.
Dec 27 15:31:37.081961918 (528h2m36.953044963s) [noise=true , address=145.79.227.33:4869   , piece_id=CNIVSRNCUSZJFORFT4SSJLEYACBKLZSW6YPTZPAVC5SHDLJKLPOA, seg_pos={0 0}, upload=463153]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF
Dec 27 15:31:37.081972548 (528h2m36.953055593s) [noise=true , address=145.79.227.33:4869   , piece_id=CNIVSRNCUSZJFORFT4SSJLEYACBKLZSW6YPTZPAVC5SHDLJKLPOA, seg_pos={0 0}, upload=463153]: Long tail cancellation due to optimal threshold
Dec 27 15:31:37.082138894 (528h2m36.95322194s) [noise=true , address=45.91.116.10:4084    , piece_id=JUKG7B5WVJ7PWWINVH6VLQH4HVCYHG6IDO2ZD5Y5LBCIJRUCEVMQ, seg_pos={0 0}, upload=463153]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: write tcp 188.166.225.170:36398->45.91.116.10:4084: use of closed network connection; write tcp 188.166.225.170:36398->45.91.116.10:4084: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36398->45.91.116.10:4084: use of closed network connection
Dec 27 15:31:37.082155006 (528h2m36.953238041s) [noise=true , address=45.91.116.10:4084    , piece_id=JUKG7B5WVJ7PWWINVH6VLQH4HVCYHG6IDO2ZD5Y5LBCIJRUCEVMQ, seg_pos={0 0}, upload=463153]: Long tail cancellation due to optimal threshold
Dec 27 15:31:37.082289120 (528h2m36.953372155s) [noise=true , address=45.91.116.12:4225    , piece_id=X2CMM65PCOTDV6SIR6ZNN7F72OICXTQ44CHCKBH62CE45K4EXKJA, seg_pos={0 0}, upload=463153]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:58418->45.91.116.12:4225: use of closed network connection; write tcp 188.166.225.170:58418->45.91.116.12:4225: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58418->45.91.116.12:4225: use of closed network connection
Dec 27 15:31:37.082299571 (528h2m36.953382626s) [noise=true , address=45.91.116.12:4225    , piece_id=X2CMM65PCOTDV6SIR6ZNN7F72OICXTQ44CHCKBH62CE45K4EXKJA, seg_pos={0 0}, upload=463153]: Long tail cancellation due to optimal threshold
Dec 27 15:31:37.082325521 (528h2m36.953408566s) [seg_pos={0 0}, upload=463153]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 15:31:37.082441892 (528h2m36.953524937s) [upload=463153]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 15:31:37.082447492 (528h2m36.953530527s) [upload=463153]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 15:31:37.380252539 (528h2m37.251335575s) [upload=463153]: Done uploading segments...
Dec 27 15:31:37.380273720 (528h2m37.251356765s) [upload=463153]: Upload finished. err: <nil>
Duration: 6.102272132s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
22 out of 60 is failed (threshold=3) [XX.......X..X.X..X...XXXXXXX..X.X..X.X.....X...X......X.X..X]
duration: 
30.003773982s
trace: 
https://jaeger.dev.storj.io/trace/522a43b73399bbba
FAIL (2025-12-27 15:32:03 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 52W7HESUL4YA6OMET3H64EJXHDBESCUC5VAJEEEXEPPM6HBNW7BQ) context deadline exceeded; piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: AQFGSNIIUH5ZAA5DSUNN7GZTV3JJ3YRQTVY5XLDLPHPCIYQ4U6QQ) context deadline exceeded
======LOGS======
2m34.086203997s) [upload=463153]: Uploading segments...
Dec 27 15:31:34.215346692 (528h2m34.086429737s) [upload=463153]: Got next segment. Inline: false
Dec 27 15:31:34.215374916 (528h2m34.086457951s) [upload=463153]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 15:31:34.215379064 (528h2m34.086462109s) [upload=463153]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 15:31:34.462608585 (528h2m34.33369162s) [seg_pos={0 0}, upload=463153]: Begin upload segment...
Dec 27 15:31:34.462659352 (528h2m34.333742398s) [seg_pos={0 0}, upload=463153]: Done begin upload segment.
Dec 27 15:31:34.462748422 (528h2m34.333831467s) [noise=true , address=145.79.227.52:4445   , piece_id=X2ON65EDXEJVR2NXKMUYGJYJMA62KZ3QXG4VBOUGFDFSG56VWF3A, seg_pos={0 0}, upload=463153]: Uploading piece...
Dec 27 15:31:34.462796123 (528h2m34.333879168s) [noise=true , address=77.90.31.14:4661     , piece_id=AQFGSNIIUH5ZAA5DSUNN7GZTV3JJ3YRQTVY5XLDLPHPCIYQ4U6QQ, seg_pos={0 0}, upload=463153]: Uploading piece...
Dec 27 15:31:34.463437474 (528h2m34.334520519s) [noise=true , address=145.79.227.42:4123   , piece_id=YFKS7437G3M3SMTEHSTUYXMZO2CS7SCZ4YHZODSH4FSL5D64OENA, seg_pos={0 0}, upload=463153]: Uploading piece...
Dec 27 15:31:34.463844980 (528h2m34.334928025s) [noise=true , address=145.79.227.51:4409   , piece_id=CHN6PACDXRAKJECHHWIQWYMXNIOMZCSX6X6YBPJTFBMR5UXC35JA, seg_pos={0 0}, upload=463153]: Uploading piece...
Dec 27 15:31:34.464152144 (528h2m34.335235179s) [noise=true , address=145.79.227.38:4979   , piece_id=UN7DNPRLE2HIO2B3KUXX5Z5554JFMB6XR7ELGUGVGVBZFTXEDODQ, seg_pos={0 0}, upload=463153]: Uploading piece...
Dec 27 15:31:34.464449932 (528h2m34.335532967s) [noise=true , address=145.79.227.11:4729   , piece_id=JWNDP4YNP5FHTRASKVF2LJHZASNOK7T5KRYLEMEIDUGNC2TZ4WCQ, seg_pos={0 0}, upload=463153]: Uploading piece...
Dec 27 15:31:34.464789368 (528h2m34.335872413s) [noise=true , address=145.79.227.3:4149    , piece_id=32LS4UUSAWIDNE6UE4GP2YIXPILHS5IBN6OAPAZ5VA6SQVK42WOQ, seg_pos={0 0}, upload=463153]: Uploading piece...
Dec 27 15:31:34.465139475 (528h2m34.33622251s) [noise=true , address=45.91.116.4:4937     , piece_id=B2U3DALRGH5U2FJLQTTVD2C345ZKIWXP3F57PFDBQNBJAPQY4ARQ, seg_pos={0 0}, upload=463153]: Uploading piece...
Dec 27 15:31:34.465596745 (528h2m34.33667979s) [noise=true , address=45.91.116.10:4084    , piece_id=JUKG7B5WVJ7PWWINVH6VLQH4HVCYHG6IDO2ZD5Y5LBCIJRUCEVMQ, seg_pos={0 0}, upload=463153]: Uploading piece...
Dec 27 15:31:34.467395972 (528h2m34.338479017s) [noise=true , address=45.91.116.12:4225    , piece_id=X2CMM65PCOTDV6SIR6ZNN7F72OICXTQ44CHCKBH62CE45K4EXKJA, seg_pos={0 0}, upload=463153]: Uploading piece...
Dec 27 15:31:34.468960661 (528h2m34.340043706s) [noise=true , address=145.79.227.33:4869   , piece_id=CNIVSRNCUSZJFORFT4SSJLEYACBKLZSW6YPTZPAVC5SHDLJKLPOA, seg_pos={0 0}, upload=463153]: Uploading piece...
Dec 27 15:31:34.470669756 (528h2m34.341752801s) [noise=true , address=145.79.227.50:4373   , piece_id=3QOJVY4KLUIVWT5CYJ3PHS4ZYW4DFYMIB375UR2HBNB4FJEYDMBA, seg_pos={0 0}, upload=463153]: Uploading piece...
Dec 27 15:31:34.472588489 (528h2m34.343671524s) [noise=true , address=145.79.227.36:4519   , piece_id=UEZWKVXIDLC2JIMRPYDFBBES25JWBFEO3JVQ4ESQDOPPXWASPLFQ, seg_pos={0 0}, upload=463153]: Uploading piece...
Dec 27 15:31:34.475052040 (528h2m34.346135075s) [noise=true , address=145.79.227.37:4565   , piece_id=5ZCRHMG2CQPEM4A5R5ZWLUB4KZO2VHD2TDT4AP7Q6IKBYR7YABZQ, seg_pos={0 0}, upload=463153]: Uploading piece...
Dec 27 15:31:34.478883847 (528h2m34.349966892s) [noise=true , address=45.91.116.8:4048     , piece_id=52W7HESUL4YA6OMET3H64EJXHDBESCUC5VAJEEEXEPPM6HBNW7BQ, seg_pos={0 0}, upload=463153]: Uploading piece...
Dec 27 15:31:34.482498600 (528h2m34.353581645s) [noise=true , address=145.79.227.48:4301   , piece_id=2W7VO7654QKTKK5YFWBOTK4H57I3PJ4UIUEISTLOIR2F5WY6BSTQ, seg_pos={0 0}, upload=463153]: Uploading piece...
Dec 27 15:31:36.079768051 (528h2m35.950851096s) [upload=463153]: Next returned nil segment
Dec 27 15:31:36.079782369 (528h2m35.950865414s) [upload=463153]: Waiting for error group managing segments...
Dec 27 15:31:36.456664874 (528h2m36.327747919s) [noise=true , address=145.79.227.51:4409   , piece_id=CHN6PACDXRAKJECHHWIQWYMXNIOMZCSX6X6YBPJTFBMR5UXC35JA, seg_pos={0 0}, upload=463153]: Done uploading piece. err: <nil>
Dec 27 15:31:36.465589868 (528h2m36.336672913s) [noise=true , address=145.79.227.3:4149    , piece_id=32LS4UUSAWIDNE6UE4GP2YIXPILHS5IBN6OAPAZ5VA6SQVK42WOQ, seg_pos={0 0}, upload=463153]: Done uploading piece. err: <nil>
Dec 27 15:31:36.470502723 (528h2m36.341585778s) [noise=true , address=145.79.227.11:4729   , piece_id=JWNDP4YNP5FHTRASKVF2LJHZASNOK7T5KRYLEMEIDUGNC2TZ4WCQ, seg_pos={0 0}, upload=463153]: Done uploading piece. err: <nil>
Dec 27 15:31:36.478656260 (528h2m36.349739305s) [noise=true , address=145.79.227.42:4123   , piece_id=YFKS7437G3M3SMTEHSTUYXMZO2CS7SCZ4YHZODSH4FSL5D64OENA, seg_pos={0 0}, upload=463153]: Done uploading piece. err: <nil>
Dec 27 15:31:36.480936943 (528h2m36.352019978s) [noise=true , address=145.79.227.50:4373   , piece_id=3QOJVY4KLUIVWT5CYJ3PHS4ZYW4DFYMIB375UR2HBNB4FJEYDMBA, seg_pos={0 0}, upload=463153]: Done uploading piece. err: <nil>
Dec 27 15:31:36.492065783 (528h2m36.363148818s) [noise=true , address=145.79.227.38:4979   , piece_id=UN7DNPRLE2HIO2B3KUXX5Z5554JFMB6XR7ELGUGVGVBZFTXEDODQ, seg_pos={0 0}, upload=463153]: Done uploading piece. err: <nil>
Dec 27 15:31:36.494214175 (528h2m36.36529722s) [noise=true , address=145.79.227.52:4445   , piece_id=X2ON65EDXEJVR2NXKMUYGJYJMA62KZ3QXG4VBOUGFDFSG56VWF3A, seg_pos={0 0}, upload=463153]: Done uploading piece. err: <nil>
Dec 27 15:31:36.496700709 (528h2m36.367783744s) [noise=true , address=145.79.227.48:4301   , piece_id=2W7VO7654QKTKK5YFWBOTK4H57I3PJ4UIUEISTLOIR2F5WY6BSTQ, seg_pos={0 0}, upload=463153]: Done uploading piece. err: <nil>
Dec 27 15:31:36.508879120 (528h2m36.379962165s) [noise=true , address=145.79.227.37:4565   , piece_id=5ZCRHMG2CQPEM4A5R5ZWLUB4KZO2VHD2TDT4AP7Q6IKBYR7YABZQ, seg_pos={0 0}, upload=463153]: Done uploading piece. err: <nil>
Dec 27 15:31:36.511295511 (528h2m36.382378556s) [noise=true , address=145.79.227.36:4519   , piece_id=UEZWKVXIDLC2JIMRPYDFBBES25JWBFEO3JVQ4ESQDOPPXWASPLFQ, seg_pos={0 0}, upload=463153]: Done uploading piece. err: <nil>
Dec 27 15:31:36.673785822 (528h2m36.544868857s) [noise=true , address=77.90.31.14:4661     , piece_id=AQFGSNIIUH5ZAA5DSUNN7GZTV3JJ3YRQTVY5XLDLPHPCIYQ4U6QQ, seg_pos={0 0}, upload=463153]: Done uploading piece. err: <nil>
Dec 27 15:31:37.005236026 (528h2m36.876319081s) [noise=true , address=45.91.116.4:4937     , piece_id=B2U3DALRGH5U2FJLQTTVD2C345ZKIWXP3F57PFDBQNBJAPQY4ARQ, seg_pos={0 0}, upload=463153]: Done uploading piece. err: <nil>
Dec 27 15:31:37.081575021 (528h2m36.952658056s) [noise=true , address=45.91.116.8:4048     , piece_id=52W7HESUL4YA6OMET3H64EJXHDBESCUC5VAJEEEXEPPM6HBNW7BQ, seg_pos={0 0}, upload=463153]: Done uploading piece. err: <nil>
Dec 27 15:31:37.081596342 (528h2m36.952679377s) [seg_pos={0 0}, upload=463153]: Segment reached optimal threshold of 13 pieces.
Dec 27 15:31:37.081961918 (528h2m36.953044963s) [noise=true , address=145.79.227.33:4869   , piece_id=CNIVSRNCUSZJFORFT4SSJLEYACBKLZSW6YPTZPAVC5SHDLJKLPOA, seg_pos={0 0}, upload=463153]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF
Dec 27 15:31:37.081972548 (528h2m36.953055593s) [noise=true , address=145.79.227.33:4869   , piece_id=CNIVSRNCUSZJFORFT4SSJLEYACBKLZSW6YPTZPAVC5SHDLJKLPOA, seg_pos={0 0}, upload=463153]: Long tail cancellation due to optimal threshold
Dec 27 15:31:37.082138894 (528h2m36.95322194s) [noise=true , address=45.91.116.10:4084    , piece_id=JUKG7B5WVJ7PWWINVH6VLQH4HVCYHG6IDO2ZD5Y5LBCIJRUCEVMQ, seg_pos={0 0}, upload=463153]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: write tcp 188.166.225.170:36398->45.91.116.10:4084: use of closed network connection; write tcp 188.166.225.170:36398->45.91.116.10:4084: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36398->45.91.116.10:4084: use of closed network connection
Dec 27 15:31:37.082155006 (528h2m36.953238041s) [noise=true , address=45.91.116.10:4084    , piece_id=JUKG7B5WVJ7PWWINVH6VLQH4HVCYHG6IDO2ZD5Y5LBCIJRUCEVMQ, seg_pos={0 0}, upload=463153]: Long tail cancellation due to optimal threshold
Dec 27 15:31:37.082289120 (528h2m36.953372155s) [noise=true , address=45.91.116.12:4225    , piece_id=X2CMM65PCOTDV6SIR6ZNN7F72OICXTQ44CHCKBH62CE45K4EXKJA, seg_pos={0 0}, upload=463153]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:58418->45.91.116.12:4225: use of closed network connection; write tcp 188.166.225.170:58418->45.91.116.12:4225: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58418->45.91.116.12:4225: use of closed network connection
Dec 27 15:31:37.082299571 (528h2m36.953382626s) [noise=true , address=45.91.116.12:4225    , piece_id=X2CMM65PCOTDV6SIR6ZNN7F72OICXTQ44CHCKBH62CE45K4EXKJA, seg_pos={0 0}, upload=463153]: Long tail cancellation due to optimal threshold
Dec 27 15:31:37.082325521 (528h2m36.953408566s) [seg_pos={0 0}, upload=463153]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 15:31:37.082441892 (528h2m36.953524937s) [upload=463153]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 15:31:37.082447492 (528h2m36.953530527s) [upload=463153]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 15:31:37.380252539 (528h2m37.251335575s) [upload=463153]: Done uploading segments...
Dec 27 15:31:37.380273720 (528h2m37.251356765s) [upload=463153]: Upload finished. err: <nil>
Duration: 30.003800031s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
7 out of 60 is failed (threshold=3) [...X..............X....X...................X.....XX......X..]
duration: 
51.107385063s
trace: 
https://jaeger.dev.storj.io/trace/24120a988203c524
FAIL (2025-12-27 15:31:59 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 191MKpEnoarS7sK4Gs63ZM8zGc5jevFX8VeqViP82duW33nJJz, Piece ID: 6GXHQB47DWIXBK4ZTC5QIA4362JRMBOQKXXGBZ3M6FVCLFSZIE2A) context deadline exceeded; piecestore: (Node ID: 1b2aaYQ4o7X5yLcXG3moJ3d4oxFoNSCwsqacwg9mshJ4MPoGdg, Piece ID: JNNC647LJKVXZIIWY4GXSKOJ3OY3WWGPHWKFWGO5TI6M2SWIXHIA) context deadline exceeded; piecestore: (Node ID: 12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, Piece ID: L2BJ6DADQGARNEFIF7M4AKQ4RDDDER64NZUKOYKAT7QBMMOWVEIA) context deadline exceeded
======LOGS======
GEJCQQVP2NVYJ6LZWA3HUA, seg_pos={0 0}, upload=463114]: Uploading piece...
Dec 27 15:29:09.918056404 (528h0m9.789139439s) [noise=true , address=77.90.31.13:4645     , piece_id=JNNC647LJKVXZIIWY4GXSKOJ3OY3WWGPHWKFWGO5TI6M2SWIXHIA, seg_pos={0 0}, upload=463114]: Uploading piece...
Dec 27 15:29:09.918708286 (528h0m9.789791331s) [noise=true , address=77.90.31.12:4593     , piece_id=RMUWIRJ2XRORPV4RJKRL2JUZZOMMOGO3TDTWM22B7LTCERERYHWQ, seg_pos={0 0}, upload=463114]: Uploading piece...
Dec 27 15:29:09.919456690 (528h0m9.790539736s) [noise=true , address=77.90.31.14:4695     , piece_id=H265WN7TXOO3LG4QTFAB5S3TE2MUON7OCCQ7H5PA5NIBCEVGOM4A, seg_pos={0 0}, upload=463114]: Uploading piece...
Dec 27 15:29:09.920039571 (528h0m9.791122616s) [noise=true , address=145.79.227.35:4512   , piece_id=DX4PHB5M4NDOFTWWEGJM43D4VBIRIGADX4HILEMRO2ZU4OV2S47Q, seg_pos={0 0}, upload=463114]: Uploading piece...
Dec 27 15:29:09.920276141 (528h0m9.791359186s) [noise=true , address=145.79.227.34:4926   , piece_id=LE5TDKNEJ2QPKW2ZIMMPBO56TAQRGFTSCAR5IHD66X4E6FIYXT5Q, seg_pos={0 0}, upload=463114]: Uploading piece...
Dec 27 15:29:09.920474529 (528h0m9.791557564s) [noise=true , address=145.79.227.13:4816   , piece_id=2OCASX7RNCMKBIVDVELFO7WIHKZMY27RII5C2BWIZXIG33BBAV5Q, seg_pos={0 0}, upload=463114]: Uploading piece...
Dec 27 15:29:09.921046528 (528h0m9.792129563s) [noise=true , address=145.79.227.40:4069   , piece_id=Z3ZYDWOIMGFRIRYP4WKYSL4YIJF7TZT73EOVUYQOOMNM42PGWHBQ, seg_pos={0 0}, upload=463114]: Uploading piece...
Dec 27 15:29:09.921603759 (528h0m9.792686794s) [noise=true , address=77.90.31.21:4883     , piece_id=6GXHQB47DWIXBK4ZTC5QIA4362JRMBOQKXXGBZ3M6FVCLFSZIE2A, seg_pos={0 0}, upload=463114]: Uploading piece...
Dec 27 15:29:09.922032846 (528h0m9.793115881s) [noise=true , address=145.79.227.44:4210   , piece_id=JMSSYMTYXUCJRFCOUCVRK4W6KX7LTGSCKKAO7XC7UGWKNGHNEMQQ, seg_pos={0 0}, upload=463114]: Uploading piece...
Dec 27 15:29:09.923593568 (528h0m9.794676613s) [noise=true , address=45.91.116.10:4103    , piece_id=QU2UHH6G7QETR2DF3A6U3NZLYXY3WD5JO4GKU3HRUO35CJJP5LJA, seg_pos={0 0}, upload=463114]: Uploading piece...
Dec 27 15:29:09.924048133 (528h0m9.795131169s) [noise=true , address=77.90.31.18:4840     , piece_id=L2BJ6DADQGARNEFIF7M4AKQ4RDDDER64NZUKOYKAT7QBMMOWVEIA, seg_pos={0 0}, upload=463114]: Uploading piece...
Dec 27 15:29:09.924539719 (528h0m9.795622754s) [noise=true , address=64.49.13.139:4470    , piece_id=3MAQ4R3XTRQT72O3DEIQSAFZAGT2CJTMPT2IKLCDLIVLPESQRVGA, seg_pos={0 0}, upload=463114]: Uploading piece...
Dec 27 15:29:09.925024673 (528h0m9.796107708s) [noise=true , address=145.79.227.49:4368   , piece_id=3LXJYQGC74WHBFCISTNZMBVLSUGSMU7XOZIFP5ZRU57TXJ3YTM4Q, seg_pos={0 0}, upload=463114]: Uploading piece...
Dec 27 15:29:09.925500990 (528h0m9.796584035s) [noise=true , address=77.90.31.17:4795     , piece_id=FTJLV46VS5BACPHHVVOCU7EEOEU3Q66P3B3WTPUT3ACG6GLYTDUA, seg_pos={0 0}, upload=463114]: Uploading piece...
Dec 27 15:29:11.205949065 (528h0m11.0770321s) [upload=463114]: Next returned nil segment
Dec 27 15:29:11.205963343 (528h0m11.077046378s) [upload=463114]: Waiting for error group managing segments...
Dec 27 15:29:11.610155861 (528h0m11.481238906s) [noise=true , address=64.49.13.139:4470    , piece_id=3MAQ4R3XTRQT72O3DEIQSAFZAGT2CJTMPT2IKLCDLIVLPESQRVGA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil>
Dec 27 15:29:11.812270605 (528h0m11.683353641s) [noise=true , address=145.79.227.35:4512   , piece_id=DX4PHB5M4NDOFTWWEGJM43D4VBIRIGADX4HILEMRO2ZU4OV2S47Q, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil>
Dec 27 15:29:11.818242477 (528h0m11.689325522s) [noise=true , address=145.79.227.13:4816   , piece_id=2OCASX7RNCMKBIVDVELFO7WIHKZMY27RII5C2BWIZXIG33BBAV5Q, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil>
Dec 27 15:29:11.938211621 (528h0m11.809294666s) [noise=true , address=145.79.227.49:4368   , piece_id=3LXJYQGC74WHBFCISTNZMBVLSUGSMU7XOZIFP5ZRU57TXJ3YTM4Q, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil>
Dec 27 15:29:12.064896693 (528h0m11.935979738s) [noise=true , address=77.90.31.17:4795     , piece_id=FTJLV46VS5BACPHHVVOCU7EEOEU3Q66P3B3WTPUT3ACG6GLYTDUA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil>
Dec 27 15:29:12.070731273 (528h0m11.941814308s) [noise=true , address=77.90.31.21:4883     , piece_id=6GXHQB47DWIXBK4ZTC5QIA4362JRMBOQKXXGBZ3M6FVCLFSZIE2A, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil>
Dec 27 15:29:12.107949174 (528h0m11.979032239s) [noise=true , address=77.90.31.11:4573     , piece_id=SRXHTEXDEKKJ2WPGY66BOAP7WD7Z75HK3B5DL2XIGFWDYQGI76YA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil>
Dec 27 15:29:12.151478532 (528h0m12.022561567s) [noise=true , address=77.90.31.13:4645     , piece_id=JNNC647LJKVXZIIWY4GXSKOJ3OY3WWGPHWKFWGO5TI6M2SWIXHIA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil>
Dec 27 15:29:12.335397247 (528h0m12.206480292s) [noise=true , address=45.91.116.10:4103    , piece_id=QU2UHH6G7QETR2DF3A6U3NZLYXY3WD5JO4GKU3HRUO35CJJP5LJA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil>
Dec 27 15:29:12.410397992 (528h0m12.281481027s) [noise=true , address=77.90.31.18:4840     , piece_id=L2BJ6DADQGARNEFIF7M4AKQ4RDDDER64NZUKOYKAT7QBMMOWVEIA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil>
Dec 27 15:29:12.667101894 (528h0m12.538184929s) [noise=true , address=77.90.31.12:4593     , piece_id=RMUWIRJ2XRORPV4RJKRL2JUZZOMMOGO3TDTWM22B7LTCERERYHWQ, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil>
Dec 27 15:29:12.785380610 (528h0m12.656463655s) [noise=true , address=145.79.227.44:4210   , piece_id=JMSSYMTYXUCJRFCOUCVRK4W6KX7LTGSCKKAO7XC7UGWKNGHNEMQQ, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil>
Dec 27 15:29:12.828722731 (528h0m12.699805766s) [noise=true , address=193.149.164.143:4697 , piece_id=S4CQ3366KVFS5RMZR67MPGSLXKDWRAGEJCQQVP2NVYJ6LZWA3HUA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil>
Dec 27 15:29:12.947757466 (528h0m12.818840501s) [noise=true , address=145.79.227.34:4926   , piece_id=LE5TDKNEJ2QPKW2ZIMMPBO56TAQRGFTSCAR5IHD66X4E6FIYXT5Q, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil>
Dec 27 15:29:12.947777434 (528h0m12.818860469s) [seg_pos={0 0}, upload=463114]: Segment reached optimal threshold of 14 pieces.
Dec 27 15:29:12.947947177 (528h0m12.819030212s) [noise=true , address=77.90.31.14:4695     , piece_id=H265WN7TXOO3LG4QTFAB5S3TE2MUON7OCCQ7H5PA5NIBCEVGOM4A, seg_pos={0 0}, upload=463114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12q2kob5oYxuxwUDh4vjNDBpW4c6LndrjHbsJ4VRDiBsLxjgLdy): protocol: write tcp 188.166.225.170:36046->77.90.31.14:4695: use of closed network connection; write tcp 188.166.225.170:36046->77.90.31.14:4695: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36046->77.90.31.14:4695: use of closed network connection
Dec 27 15:29:12.947953169 (528h0m12.819036204s) [noise=true , address=77.90.31.14:4695     , piece_id=H265WN7TXOO3LG4QTFAB5S3TE2MUON7OCCQ7H5PA5NIBCEVGOM4A, seg_pos={0 0}, upload=463114]: Long tail cancellation due to optimal threshold
Dec 27 15:29:12.948200800 (528h0m12.819283845s) [noise=true , address=77.90.31.19:4869     , piece_id=KENCENCTEBHPVIPCECDQYLIWHYLT6G7ZETYMN4DA6LPQRLVQETNA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8): protocol: read tcp 188.166.225.170:32932->77.90.31.19:4869: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 27 15:29:12.948216079 (528h0m12.819299124s) [noise=true , address=77.90.31.19:4869     , piece_id=KENCENCTEBHPVIPCECDQYLIWHYLT6G7ZETYMN4DA6LPQRLVQETNA, seg_pos={0 0}, upload=463114]: Long tail cancellation due to optimal threshold
Dec 27 15:29:12.948310158 (528h0m12.819393203s) [noise=true , address=64.49.13.169:4711    , piece_id=IMYFGTTZJUS5CEYCF75SUFUIGA3QFQYTFVCKYKFBM2DDQIKQHS7A, seg_pos={0 0}, upload=463114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:51826->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:51826->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51826->64.49.13.169:4711: use of closed network connection
Dec 27 15:29:12.948316139 (528h0m12.819399174s) [noise=true , address=64.49.13.169:4711    , piece_id=IMYFGTTZJUS5CEYCF75SUFUIGA3QFQYTFVCKYKFBM2DDQIKQHS7A, seg_pos={0 0}, upload=463114]: Long tail cancellation due to optimal threshold
Dec 27 15:29:12.948360164 (528h0m12.819443219s) [noise=true , address=145.79.227.40:4069   , piece_id=Z3ZYDWOIMGFRIRYP4WKYSL4YIJF7TZT73EOVUYQOOMNM42PGWHBQ, seg_pos={0 0}, upload=463114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF
Dec 27 15:29:12.948372617 (528h0m12.819455652s) [noise=true , address=145.79.227.40:4069   , piece_id=Z3ZYDWOIMGFRIRYP4WKYSL4YIJF7TZT73EOVUYQOOMNM42PGWHBQ, seg_pos={0 0}, upload=463114]: Long tail cancellation due to optimal threshold
Dec 27 15:29:12.948398968 (528h0m12.819482013s) [seg_pos={0 0}, upload=463114]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 15:29:12.948477878 (528h0m12.819560913s) [upload=463114]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 15:29:12.948483889 (528h0m12.819566934s) [upload=463114]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 15:29:13.237238452 (528h0m13.108321498s) [upload=463114]: Done uploading segments...
Dec 27 15:29:13.237262939 (528h0m13.108345984s) [upload=463114]: Upload finished. err: <nil>
Duration: 51.107405182s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
33 out of 60 is failed (threshold=6) [.XXX..X..XX.XXXX.....XXX.X.XXXX.X.XXXX...XXXX.XXXXX...X.....]
duration: 
3.434797318s
trace: 
https://jaeger.dev.storj.io/trace/5973653b14c3bbcb
FAIL (2025-12-27 15:31:29 UTC) 
orbiter warning: context deadline exceeded
Duration: 3.434819521s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
7 out of 60 is failed (threshold=3) [....X..............X....X...................X.....XX......X.]
duration: 
58.584714521s
trace: 
https://jaeger.dev.storj.io/trace/24120a988203c524
FAIL (2025-12-27 15:31:07 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 191MKpEnoarS7sK4Gs63ZM8zGc5jevFX8VeqViP82duW33nJJz, Piece ID: 6GXHQB47DWIXBK4ZTC5QIA4362JRMBOQKXXGBZ3M6FVCLFSZIE2A) context deadline exceeded; piecestore: (Node ID: 1b2aaYQ4o7X5yLcXG3moJ3d4oxFoNSCwsqacwg9mshJ4MPoGdg, Piece ID: JNNC647LJKVXZIIWY4GXSKOJ3OY3WWGPHWKFWGO5TI6M2SWIXHIA) context deadline exceeded; piecestore: (Node ID: 12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, Piece ID: L2BJ6DADQGARNEFIF7M4AKQ4RDDDER64NZUKOYKAT7QBMMOWVEIA) context deadline exceeded
======LOGS======
GEJCQQVP2NVYJ6LZWA3HUA, seg_pos={0 0}, upload=463114]: Uploading piece...
Dec 27 15:29:09.918056404 (528h0m9.789139439s) [noise=true , address=77.90.31.13:4645     , piece_id=JNNC647LJKVXZIIWY4GXSKOJ3OY3WWGPHWKFWGO5TI6M2SWIXHIA, seg_pos={0 0}, upload=463114]: Uploading piece...
Dec 27 15:29:09.918708286 (528h0m9.789791331s) [noise=true , address=77.90.31.12:4593     , piece_id=RMUWIRJ2XRORPV4RJKRL2JUZZOMMOGO3TDTWM22B7LTCERERYHWQ, seg_pos={0 0}, upload=463114]: Uploading piece...
Dec 27 15:29:09.919456690 (528h0m9.790539736s) [noise=true , address=77.90.31.14:4695     , piece_id=H265WN7TXOO3LG4QTFAB5S3TE2MUON7OCCQ7H5PA5NIBCEVGOM4A, seg_pos={0 0}, upload=463114]: Uploading piece...
Dec 27 15:29:09.920039571 (528h0m9.791122616s) [noise=true , address=145.79.227.35:4512   , piece_id=DX4PHB5M4NDOFTWWEGJM43D4VBIRIGADX4HILEMRO2ZU4OV2S47Q, seg_pos={0 0}, upload=463114]: Uploading piece...
Dec 27 15:29:09.920276141 (528h0m9.791359186s) [noise=true , address=145.79.227.34:4926   , piece_id=LE5TDKNEJ2QPKW2ZIMMPBO56TAQRGFTSCAR5IHD66X4E6FIYXT5Q, seg_pos={0 0}, upload=463114]: Uploading piece...
Dec 27 15:29:09.920474529 (528h0m9.791557564s) [noise=true , address=145.79.227.13:4816   , piece_id=2OCASX7RNCMKBIVDVELFO7WIHKZMY27RII5C2BWIZXIG33BBAV5Q, seg_pos={0 0}, upload=463114]: Uploading piece...
Dec 27 15:29:09.921046528 (528h0m9.792129563s) [noise=true , address=145.79.227.40:4069   , piece_id=Z3ZYDWOIMGFRIRYP4WKYSL4YIJF7TZT73EOVUYQOOMNM42PGWHBQ, seg_pos={0 0}, upload=463114]: Uploading piece...
Dec 27 15:29:09.921603759 (528h0m9.792686794s) [noise=true , address=77.90.31.21:4883     , piece_id=6GXHQB47DWIXBK4ZTC5QIA4362JRMBOQKXXGBZ3M6FVCLFSZIE2A, seg_pos={0 0}, upload=463114]: Uploading piece...
Dec 27 15:29:09.922032846 (528h0m9.793115881s) [noise=true , address=145.79.227.44:4210   , piece_id=JMSSYMTYXUCJRFCOUCVRK4W6KX7LTGSCKKAO7XC7UGWKNGHNEMQQ, seg_pos={0 0}, upload=463114]: Uploading piece...
Dec 27 15:29:09.923593568 (528h0m9.794676613s) [noise=true , address=45.91.116.10:4103    , piece_id=QU2UHH6G7QETR2DF3A6U3NZLYXY3WD5JO4GKU3HRUO35CJJP5LJA, seg_pos={0 0}, upload=463114]: Uploading piece...
Dec 27 15:29:09.924048133 (528h0m9.795131169s) [noise=true , address=77.90.31.18:4840     , piece_id=L2BJ6DADQGARNEFIF7M4AKQ4RDDDER64NZUKOYKAT7QBMMOWVEIA, seg_pos={0 0}, upload=463114]: Uploading piece...
Dec 27 15:29:09.924539719 (528h0m9.795622754s) [noise=true , address=64.49.13.139:4470    , piece_id=3MAQ4R3XTRQT72O3DEIQSAFZAGT2CJTMPT2IKLCDLIVLPESQRVGA, seg_pos={0 0}, upload=463114]: Uploading piece...
Dec 27 15:29:09.925024673 (528h0m9.796107708s) [noise=true , address=145.79.227.49:4368   , piece_id=3LXJYQGC74WHBFCISTNZMBVLSUGSMU7XOZIFP5ZRU57TXJ3YTM4Q, seg_pos={0 0}, upload=463114]: Uploading piece...
Dec 27 15:29:09.925500990 (528h0m9.796584035s) [noise=true , address=77.90.31.17:4795     , piece_id=FTJLV46VS5BACPHHVVOCU7EEOEU3Q66P3B3WTPUT3ACG6GLYTDUA, seg_pos={0 0}, upload=463114]: Uploading piece...
Dec 27 15:29:11.205949065 (528h0m11.0770321s) [upload=463114]: Next returned nil segment
Dec 27 15:29:11.205963343 (528h0m11.077046378s) [upload=463114]: Waiting for error group managing segments...
Dec 27 15:29:11.610155861 (528h0m11.481238906s) [noise=true , address=64.49.13.139:4470    , piece_id=3MAQ4R3XTRQT72O3DEIQSAFZAGT2CJTMPT2IKLCDLIVLPESQRVGA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil>
Dec 27 15:29:11.812270605 (528h0m11.683353641s) [noise=true , address=145.79.227.35:4512   , piece_id=DX4PHB5M4NDOFTWWEGJM43D4VBIRIGADX4HILEMRO2ZU4OV2S47Q, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil>
Dec 27 15:29:11.818242477 (528h0m11.689325522s) [noise=true , address=145.79.227.13:4816   , piece_id=2OCASX7RNCMKBIVDVELFO7WIHKZMY27RII5C2BWIZXIG33BBAV5Q, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil>
Dec 27 15:29:11.938211621 (528h0m11.809294666s) [noise=true , address=145.79.227.49:4368   , piece_id=3LXJYQGC74WHBFCISTNZMBVLSUGSMU7XOZIFP5ZRU57TXJ3YTM4Q, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil>
Dec 27 15:29:12.064896693 (528h0m11.935979738s) [noise=true , address=77.90.31.17:4795     , piece_id=FTJLV46VS5BACPHHVVOCU7EEOEU3Q66P3B3WTPUT3ACG6GLYTDUA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil>
Dec 27 15:29:12.070731273 (528h0m11.941814308s) [noise=true , address=77.90.31.21:4883     , piece_id=6GXHQB47DWIXBK4ZTC5QIA4362JRMBOQKXXGBZ3M6FVCLFSZIE2A, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil>
Dec 27 15:29:12.107949174 (528h0m11.979032239s) [noise=true , address=77.90.31.11:4573     , piece_id=SRXHTEXDEKKJ2WPGY66BOAP7WD7Z75HK3B5DL2XIGFWDYQGI76YA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil>
Dec 27 15:29:12.151478532 (528h0m12.022561567s) [noise=true , address=77.90.31.13:4645     , piece_id=JNNC647LJKVXZIIWY4GXSKOJ3OY3WWGPHWKFWGO5TI6M2SWIXHIA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil>
Dec 27 15:29:12.335397247 (528h0m12.206480292s) [noise=true , address=45.91.116.10:4103    , piece_id=QU2UHH6G7QETR2DF3A6U3NZLYXY3WD5JO4GKU3HRUO35CJJP5LJA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil>
Dec 27 15:29:12.410397992 (528h0m12.281481027s) [noise=true , address=77.90.31.18:4840     , piece_id=L2BJ6DADQGARNEFIF7M4AKQ4RDDDER64NZUKOYKAT7QBMMOWVEIA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil>
Dec 27 15:29:12.667101894 (528h0m12.538184929s) [noise=true , address=77.90.31.12:4593     , piece_id=RMUWIRJ2XRORPV4RJKRL2JUZZOMMOGO3TDTWM22B7LTCERERYHWQ, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil>
Dec 27 15:29:12.785380610 (528h0m12.656463655s) [noise=true , address=145.79.227.44:4210   , piece_id=JMSSYMTYXUCJRFCOUCVRK4W6KX7LTGSCKKAO7XC7UGWKNGHNEMQQ, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil>
Dec 27 15:29:12.828722731 (528h0m12.699805766s) [noise=true , address=193.149.164.143:4697 , piece_id=S4CQ3366KVFS5RMZR67MPGSLXKDWRAGEJCQQVP2NVYJ6LZWA3HUA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil>
Dec 27 15:29:12.947757466 (528h0m12.818840501s) [noise=true , address=145.79.227.34:4926   , piece_id=LE5TDKNEJ2QPKW2ZIMMPBO56TAQRGFTSCAR5IHD66X4E6FIYXT5Q, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil>
Dec 27 15:29:12.947777434 (528h0m12.818860469s) [seg_pos={0 0}, upload=463114]: Segment reached optimal threshold of 14 pieces.
Dec 27 15:29:12.947947177 (528h0m12.819030212s) [noise=true , address=77.90.31.14:4695     , piece_id=H265WN7TXOO3LG4QTFAB5S3TE2MUON7OCCQ7H5PA5NIBCEVGOM4A, seg_pos={0 0}, upload=463114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12q2kob5oYxuxwUDh4vjNDBpW4c6LndrjHbsJ4VRDiBsLxjgLdy): protocol: write tcp 188.166.225.170:36046->77.90.31.14:4695: use of closed network connection; write tcp 188.166.225.170:36046->77.90.31.14:4695: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36046->77.90.31.14:4695: use of closed network connection
Dec 27 15:29:12.947953169 (528h0m12.819036204s) [noise=true , address=77.90.31.14:4695     , piece_id=H265WN7TXOO3LG4QTFAB5S3TE2MUON7OCCQ7H5PA5NIBCEVGOM4A, seg_pos={0 0}, upload=463114]: Long tail cancellation due to optimal threshold
Dec 27 15:29:12.948200800 (528h0m12.819283845s) [noise=true , address=77.90.31.19:4869     , piece_id=KENCENCTEBHPVIPCECDQYLIWHYLT6G7ZETYMN4DA6LPQRLVQETNA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8): protocol: read tcp 188.166.225.170:32932->77.90.31.19:4869: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 27 15:29:12.948216079 (528h0m12.819299124s) [noise=true , address=77.90.31.19:4869     , piece_id=KENCENCTEBHPVIPCECDQYLIWHYLT6G7ZETYMN4DA6LPQRLVQETNA, seg_pos={0 0}, upload=463114]: Long tail cancellation due to optimal threshold
Dec 27 15:29:12.948310158 (528h0m12.819393203s) [noise=true , address=64.49.13.169:4711    , piece_id=IMYFGTTZJUS5CEYCF75SUFUIGA3QFQYTFVCKYKFBM2DDQIKQHS7A, seg_pos={0 0}, upload=463114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:51826->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:51826->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51826->64.49.13.169:4711: use of closed network connection
Dec 27 15:29:12.948316139 (528h0m12.819399174s) [noise=true , address=64.49.13.169:4711    , piece_id=IMYFGTTZJUS5CEYCF75SUFUIGA3QFQYTFVCKYKFBM2DDQIKQHS7A, seg_pos={0 0}, upload=463114]: Long tail cancellation due to optimal threshold
Dec 27 15:29:12.948360164 (528h0m12.819443219s) [noise=true , address=145.79.227.40:4069   , piece_id=Z3ZYDWOIMGFRIRYP4WKYSL4YIJF7TZT73EOVUYQOOMNM42PGWHBQ, seg_pos={0 0}, upload=463114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF
Dec 27 15:29:12.948372617 (528h0m12.819455652s) [noise=true , address=145.79.227.40:4069   , piece_id=Z3ZYDWOIMGFRIRYP4WKYSL4YIJF7TZT73EOVUYQOOMNM42PGWHBQ, seg_pos={0 0}, upload=463114]: Long tail cancellation due to optimal threshold
Dec 27 15:29:12.948398968 (528h0m12.819482013s) [seg_pos={0 0}, upload=463114]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 15:29:12.948477878 (528h0m12.819560913s) [upload=463114]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 15:29:12.948483889 (528h0m12.819566934s) [upload=463114]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 15:29:13.237238452 (528h0m13.108321498s) [upload=463114]: Done uploading segments...
Dec 27 15:29:13.237262939 (528h0m13.108345984s) [upload=463114]: Upload finished. err: <nil>
Duration: 58.584738598s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
21 out of 60 is failed (threshold=3) [.XX.......X..X.X..X...XXXXXXX..X.X..X.X.....X...X......X.X..]
duration: 
5.855355947s
trace: 
https://jaeger.dev.storj.io/trace/232738e743a7710f
FAIL (2025-12-27 15:30:58 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: SSHEERUL26LLWTIRKLGHOH5Y3XWQ3VNGYOUFURIE3QX67NA7QK4A) context deadline exceeded; piecestore: (Node ID: 1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, Piece ID: I4NMMYGC4PYBVLQBDAVDQP4EVB4NOKFA2QAYWYE7R4YRJZUQS5MQ) context deadline exceeded
======LOGS======
Dec 27 15:30:03.202013435 (528h1m3.07309648s) [upload=463129]: Starting upload
Dec 27 15:30:03.202162749 (528h1m3.073245794s) [upload=463129]: Done starting upload
Dec 27 15:30:03.202279562 (528h1m3.073362607s) [upload=463129]: Uploading segments...
Dec 27 15:30:03.202458873 (528h1m3.073541918s) [upload=463129]: Got next segment. Inline: false
Dec 27 15:30:03.202541731 (528h1m3.073624776s) [upload=463129]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 15:30:03.202550759 (528h1m3.073633824s) [upload=463129]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 15:30:03.447228601 (528h1m3.318311656s) [seg_pos={0 0}, upload=463129]: Begin upload segment...
Dec 27 15:30:03.447292532 (528h1m3.318375578s) [seg_pos={0 0}, upload=463129]: Done begin upload segment.
Dec 27 15:30:03.447345413 (528h1m3.318428488s) [noise=true , address=145.79.227.43:4159   , piece_id=OZQE36KJGMUCMTF3XWTJTMSC7SD7QNVSJYQEFGBEK6BCJET2P4CQ, seg_pos={0 0}, upload=463129]: Uploading piece...
Dec 27 15:30:03.447475240 (528h1m3.318558295s) [noise=true , address=45.91.116.4:4937     , piece_id=LFI5SJVUI5CQ6XB2ZIPJA5METXQTLTG3FEFSE5HLLTJJPH5XBBRA, seg_pos={0 0}, upload=463129]: Uploading piece...
Dec 27 15:30:03.448000881 (528h1m3.319083926s) [noise=true , address=145.79.227.3:4150    , piece_id=XMPRVXIHPZ6G3YY5LRWVTCCOVAR4RHIHYSO2FC6I5XPU74GJJW3A, seg_pos={0 0}, upload=463129]: Uploading piece...
Dec 27 15:30:03.448524428 (528h1m3.319607473s) [noise=true , address=145.79.227.46:4603   , piece_id=SC6S4AJVSRWBUYWLVPSAUXNTAZ47HVH5QNEAX5LUS2WOGNJS6OMA, seg_pos={0 0}, upload=463129]: Uploading piece...
Dec 27 15:30:03.450312654 (528h1m3.321395699s) [noise=true , address=145.79.227.40:4049   , piece_id=LMVUBWD4I3Q3GNZC6V6E6ZBGOLR6Z4K2QE634MTW2BLL73JY35YA, seg_pos={0 0}, upload=463129]: Uploading piece...
Dec 27 15:30:03.450833946 (528h1m3.321916991s) [noise=true , address=145.79.227.51:4411   , piece_id=B46UJTZWQ5I5AVHTP3NIZ5T7R2XEDTJ7MDVH2R32S34IHQ25QIUA, seg_pos={0 0}, upload=463129]: Uploading piece...
Dec 27 15:30:03.451380025 (528h1m3.32246308s) [noise=true , address=145.79.227.13:4804   , piece_id=D2PVRI5AXLIS7V5RRDY5ZIJUWUB6JQ5CW6STLJSO26MLA2EO2RTQ, seg_pos={0 0}, upload=463129]: Uploading piece...
Dec 27 15:30:03.451886691 (528h1m3.322969736s) [noise=true , address=145.79.227.45:4231   , piece_id=FIXT3Q56D6F2JGFFFK3EIQPJOGL4QII2DVYFC6OJ7E3L6FRSAJOA, seg_pos={0 0}, upload=463129]: Uploading piece...
Dec 27 15:30:03.452628703 (528h1m3.323711748s) [noise=true , address=145.79.227.15:4876   , piece_id=6VYNR3VMOGAUGCPQETKAF244ZOQUPRECLBO6NWV6JPTGOOFAY3AA, seg_pos={0 0}, upload=463129]: Uploading piece...
Dec 27 15:30:03.453370095 (528h1m3.32445314s) [noise=true , address=45.91.116.6:4975     , piece_id=3O43RYN3EOZ4I37XK3GK54BWJF4XRGWN5ETKUNZYZONWT4WX72XA, seg_pos={0 0}, upload=463129]: Uploading piece...
Dec 27 15:30:03.456040610 (528h1m3.327123665s) [noise=true , address=145.79.227.10:4403   , piece_id=DLAPXIGKGR4CTFMUWJQKC6IO6AV3IOSO2JQFVB7PVMFKPSYNFFEQ, seg_pos={0 0}, upload=463129]: Uploading piece...
Dec 27 15:30:03.456859440 (528h1m3.327942515s) [noise=true , address=145.79.227.50:4373   , piece_id=Z5SMO4PCHSCFMDCDY7XSOLYA6W4I5ZWZZJKLZXSDGONKLQFL7PCA, seg_pos={0 0}, upload=463129]: Uploading piece...
Dec 27 15:30:03.459937971 (528h1m3.331021016s) [noise=true , address=145.79.227.37:4566   , piece_id=XYLUOY3VMVSXMXQB4F2EBQWIA424NJJYDJCSMWIZP4UEG7SY55IA, seg_pos={0 0}, upload=463129]: Uploading piece...
Dec 27 15:30:03.460531822 (528h1m3.331614867s) [noise=true , address=77.90.31.10:4517     , piece_id=SSHEERUL26LLWTIRKLGHOH5Y3XWQ3VNGYOUFURIE3QX67NA7QK4A, seg_pos={0 0}, upload=463129]: Uploading piece...
Dec 27 15:30:03.465021191 (528h1m3.336104236s) [noise=true , address=77.90.31.11:4553     , piece_id=I4NMMYGC4PYBVLQBDAVDQP4EVB4NOKFA2QAYWYE7R4YRJZUQS5MQ, seg_pos={0 0}, upload=463129]: Uploading piece...
Dec 27 15:30:03.468659359 (528h1m3.339742404s) [noise=true , address=145.79.227.28:4727   , piece_id=U5IUWVWOR6QLUXMGVPYLHSLH3FLUYMJOFPIBB66CJC2VL2NLXZLA, seg_pos={0 0}, upload=463129]: Uploading piece...
Dec 27 15:30:05.126254340 (528h1m4.997337385s) [upload=463129]: Next returned nil segment
Dec 27 15:30:05.126274680 (528h1m4.997357735s) [upload=463129]: Waiting for error group managing segments...
Dec 27 15:30:05.530162829 (528h1m5.401245884s) [noise=true , address=145.79.227.15:4876   , piece_id=6VYNR3VMOGAUGCPQETKAF244ZOQUPRECLBO6NWV6JPTGOOFAY3AA, seg_pos={0 0}, upload=463129]: Done uploading piece. err: <nil>
Dec 27 15:30:05.540036951 (528h1m5.411119996s) [noise=true , address=145.79.227.46:4603   , piece_id=SC6S4AJVSRWBUYWLVPSAUXNTAZ47HVH5QNEAX5LUS2WOGNJS6OMA, seg_pos={0 0}, upload=463129]: Done uploading piece. err: <nil>
Dec 27 15:30:05.562110754 (528h1m5.433193799s) [noise=true , address=145.79.227.13:4804   , piece_id=D2PVRI5AXLIS7V5RRDY5ZIJUWUB6JQ5CW6STLJSO26MLA2EO2RTQ, seg_pos={0 0}, upload=463129]: Done uploading piece. err: <nil>
Dec 27 15:30:05.562660631 (528h1m5.433743676s) [noise=true , address=145.79.227.51:4411   , piece_id=B46UJTZWQ5I5AVHTP3NIZ5T7R2XEDTJ7MDVH2R32S34IHQ25QIUA, seg_pos={0 0}, upload=463129]: Done uploading piece. err: <nil>
Dec 27 15:30:05.573199559 (528h1m5.444282664s) [noise=true , address=145.79.227.28:4727   , piece_id=U5IUWVWOR6QLUXMGVPYLHSLH3FLUYMJOFPIBB66CJC2VL2NLXZLA, seg_pos={0 0}, upload=463129]: Done uploading piece. err: <nil>
Dec 27 15:30:05.575287635 (528h1m5.44637068s) [noise=true , address=145.79.227.50:4373   , piece_id=Z5SMO4PCHSCFMDCDY7XSOLYA6W4I5ZWZZJKLZXSDGONKLQFL7PCA, seg_pos={0 0}, upload=463129]: Done uploading piece. err: <nil>
Dec 27 15:30:05.577932291 (528h1m5.449015336s) [noise=true , address=145.79.227.10:4403   , piece_id=DLAPXIGKGR4CTFMUWJQKC6IO6AV3IOSO2JQFVB7PVMFKPSYNFFEQ, seg_pos={0 0}, upload=463129]: Done uploading piece. err: <nil>
Dec 27 15:30:05.583145548 (528h1m5.454228593s) [noise=true , address=145.79.227.45:4231   , piece_id=FIXT3Q56D6F2JGFFFK3EIQPJOGL4QII2DVYFC6OJ7E3L6FRSAJOA, seg_pos={0 0}, upload=463129]: Done uploading piece. err: <nil>
Dec 27 15:30:05.588678233 (528h1m5.459761278s) [noise=true , address=145.79.227.3:4150    , piece_id=XMPRVXIHPZ6G3YY5LRWVTCCOVAR4RHIHYSO2FC6I5XPU74GJJW3A, seg_pos={0 0}, upload=463129]: Done uploading piece. err: <nil>
Dec 27 15:30:05.598997924 (528h1m5.470080969s) [noise=true , address=145.79.227.37:4566   , piece_id=XYLUOY3VMVSXMXQB4F2EBQWIA424NJJYDJCSMWIZP4UEG7SY55IA, seg_pos={0 0}, upload=463129]: Done uploading piece. err: <nil>
Dec 27 15:30:05.824274082 (528h1m5.695357127s) [noise=true , address=77.90.31.10:4517     , piece_id=SSHEERUL26LLWTIRKLGHOH5Y3XWQ3VNGYOUFURIE3QX67NA7QK4A, seg_pos={0 0}, upload=463129]: Done uploading piece. err: <nil>
Dec 27 15:30:06.398684147 (528h1m6.269767202s) [noise=true , address=77.90.31.11:4553     , piece_id=I4NMMYGC4PYBVLQBDAVDQP4EVB4NOKFA2QAYWYE7R4YRJZUQS5MQ, seg_pos={0 0}, upload=463129]: Done uploading piece. err: <nil>
Dec 27 15:30:06.459209859 (528h1m6.330292904s) [noise=true , address=45.91.116.6:4975     , piece_id=3O43RYN3EOZ4I37XK3GK54BWJF4XRGWN5ETKUNZYZONWT4WX72XA, seg_pos={0 0}, upload=463129]: Done uploading piece. err: <nil>
Dec 27 15:30:06.459240017 (528h1m6.330323052s) [seg_pos={0 0}, upload=463129]: Segment reached optimal threshold of 13 pieces.
Dec 27 15:30:06.459463141 (528h1m6.330546176s) [noise=true , address=145.79.227.40:4049   , piece_id=LMVUBWD4I3Q3GNZC6V6E6ZBGOLR6Z4K2QE634MTW2BLL73JY35YA, seg_pos={0 0}, upload=463129]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF
Dec 27 15:30:06.459470576 (528h1m6.330553611s) [noise=true , address=145.79.227.40:4049   , piece_id=LMVUBWD4I3Q3GNZC6V6E6ZBGOLR6Z4K2QE634MTW2BLL73JY35YA, seg_pos={0 0}, upload=463129]: Long tail cancellation due to optimal threshold
Dec 27 15:30:06.459697808 (528h1m6.330780853s) [noise=true , address=145.79.227.43:4159   , piece_id=OZQE36KJGMUCMTF3XWTJTMSC7SD7QNVSJYQEFGBEK6BCJET2P4CQ, seg_pos={0 0}, upload=463129]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF
Dec 27 15:30:06.459717607 (528h1m6.330800652s) [noise=true , address=145.79.227.43:4159   , piece_id=OZQE36KJGMUCMTF3XWTJTMSC7SD7QNVSJYQEFGBEK6BCJET2P4CQ, seg_pos={0 0}, upload=463129]: Long tail cancellation due to optimal threshold
Dec 27 15:30:06.459773963 (528h1m6.330857008s) [noise=true , address=45.91.116.4:4937     , piece_id=LFI5SJVUI5CQ6XB2ZIPJA5METXQTLTG3FEFSE5HLLTJJPH5XBBRA, seg_pos={0 0}, upload=463129]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF
Dec 27 15:30:06.459782389 (528h1m6.330865434s) [noise=true , address=45.91.116.4:4937     , piece_id=LFI5SJVUI5CQ6XB2ZIPJA5METXQTLTG3FEFSE5HLLTJJPH5XBBRA, seg_pos={0 0}, upload=463129]: Long tail cancellation due to optimal threshold
Dec 27 15:30:06.459831834 (528h1m6.330914879s) [seg_pos={0 0}, upload=463129]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 15:30:06.459916374 (528h1m6.330999409s) [upload=463129]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 15:30:06.459922235 (528h1m6.33100528s) [upload=463129]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 15:30:06.753073741 (528h1m6.624156786s) [upload=463129]: Done uploading segments...
Dec 27 15:30:06.753098749 (528h1m6.624181794s) [upload=463129]: Upload finished. err: <nil>
Duration: 5.855389902s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
21 out of 60 is failed (threshold=3) [..XX.......X..X.X..X...XXXXXXX..X.X..X.X.....X...X......X.X.]
duration: 
6.112236988s
trace: 
https://jaeger.dev.storj.io/trace/232738e743a7710f
FAIL (2025-12-27 15:30:41 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: SSHEERUL26LLWTIRKLGHOH5Y3XWQ3VNGYOUFURIE3QX67NA7QK4A) context deadline exceeded; piecestore: (Node ID: 1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, Piece ID: I4NMMYGC4PYBVLQBDAVDQP4EVB4NOKFA2QAYWYE7R4YRJZUQS5MQ) context deadline exceeded
======LOGS======
Dec 27 15:30:03.202013435 (528h1m3.07309648s) [upload=463129]: Starting upload
Dec 27 15:30:03.202162749 (528h1m3.073245794s) [upload=463129]: Done starting upload
Dec 27 15:30:03.202279562 (528h1m3.073362607s) [upload=463129]: Uploading segments...
Dec 27 15:30:03.202458873 (528h1m3.073541918s) [upload=463129]: Got next segment. Inline: false
Dec 27 15:30:03.202541731 (528h1m3.073624776s) [upload=463129]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 15:30:03.202550759 (528h1m3.073633824s) [upload=463129]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 15:30:03.447228601 (528h1m3.318311656s) [seg_pos={0 0}, upload=463129]: Begin upload segment...
Dec 27 15:30:03.447292532 (528h1m3.318375578s) [seg_pos={0 0}, upload=463129]: Done begin upload segment.
Dec 27 15:30:03.447345413 (528h1m3.318428488s) [noise=true , address=145.79.227.43:4159   , piece_id=OZQE36KJGMUCMTF3XWTJTMSC7SD7QNVSJYQEFGBEK6BCJET2P4CQ, seg_pos={0 0}, upload=463129]: Uploading piece...
Dec 27 15:30:03.447475240 (528h1m3.318558295s) [noise=true , address=45.91.116.4:4937     , piece_id=LFI5SJVUI5CQ6XB2ZIPJA5METXQTLTG3FEFSE5HLLTJJPH5XBBRA, seg_pos={0 0}, upload=463129]: Uploading piece...
Dec 27 15:30:03.448000881 (528h1m3.319083926s) [noise=true , address=145.79.227.3:4150    , piece_id=XMPRVXIHPZ6G3YY5LRWVTCCOVAR4RHIHYSO2FC6I5XPU74GJJW3A, seg_pos={0 0}, upload=463129]: Uploading piece...
Dec 27 15:30:03.448524428 (528h1m3.319607473s) [noise=true , address=145.79.227.46:4603   , piece_id=SC6S4AJVSRWBUYWLVPSAUXNTAZ47HVH5QNEAX5LUS2WOGNJS6OMA, seg_pos={0 0}, upload=463129]: Uploading piece...
Dec 27 15:30:03.450312654 (528h1m3.321395699s) [noise=true , address=145.79.227.40:4049   , piece_id=LMVUBWD4I3Q3GNZC6V6E6ZBGOLR6Z4K2QE634MTW2BLL73JY35YA, seg_pos={0 0}, upload=463129]: Uploading piece...
Dec 27 15:30:03.450833946 (528h1m3.321916991s) [noise=true , address=145.79.227.51:4411   , piece_id=B46UJTZWQ5I5AVHTP3NIZ5T7R2XEDTJ7MDVH2R32S34IHQ25QIUA, seg_pos={0 0}, upload=463129]: Uploading piece...
Dec 27 15:30:03.451380025 (528h1m3.32246308s) [noise=true , address=145.79.227.13:4804   , piece_id=D2PVRI5AXLIS7V5RRDY5ZIJUWUB6JQ5CW6STLJSO26MLA2EO2RTQ, seg_pos={0 0}, upload=463129]: Uploading piece...
Dec 27 15:30:03.451886691 (528h1m3.322969736s) [noise=true , address=145.79.227.45:4231   , piece_id=FIXT3Q56D6F2JGFFFK3EIQPJOGL4QII2DVYFC6OJ7E3L6FRSAJOA, seg_pos={0 0}, upload=463129]: Uploading piece...
Dec 27 15:30:03.452628703 (528h1m3.323711748s) [noise=true , address=145.79.227.15:4876   , piece_id=6VYNR3VMOGAUGCPQETKAF244ZOQUPRECLBO6NWV6JPTGOOFAY3AA, seg_pos={0 0}, upload=463129]: Uploading piece...
Dec 27 15:30:03.453370095 (528h1m3.32445314s) [noise=true , address=45.91.116.6:4975     , piece_id=3O43RYN3EOZ4I37XK3GK54BWJF4XRGWN5ETKUNZYZONWT4WX72XA, seg_pos={0 0}, upload=463129]: Uploading piece...
Dec 27 15:30:03.456040610 (528h1m3.327123665s) [noise=true , address=145.79.227.10:4403   , piece_id=DLAPXIGKGR4CTFMUWJQKC6IO6AV3IOSO2JQFVB7PVMFKPSYNFFEQ, seg_pos={0 0}, upload=463129]: Uploading piece...
Dec 27 15:30:03.456859440 (528h1m3.327942515s) [noise=true , address=145.79.227.50:4373   , piece_id=Z5SMO4PCHSCFMDCDY7XSOLYA6W4I5ZWZZJKLZXSDGONKLQFL7PCA, seg_pos={0 0}, upload=463129]: Uploading piece...
Dec 27 15:30:03.459937971 (528h1m3.331021016s) [noise=true , address=145.79.227.37:4566   , piece_id=XYLUOY3VMVSXMXQB4F2EBQWIA424NJJYDJCSMWIZP4UEG7SY55IA, seg_pos={0 0}, upload=463129]: Uploading piece...
Dec 27 15:30:03.460531822 (528h1m3.331614867s) [noise=true , address=77.90.31.10:4517     , piece_id=SSHEERUL26LLWTIRKLGHOH5Y3XWQ3VNGYOUFURIE3QX67NA7QK4A, seg_pos={0 0}, upload=463129]: Uploading piece...
Dec 27 15:30:03.465021191 (528h1m3.336104236s) [noise=true , address=77.90.31.11:4553     , piece_id=I4NMMYGC4PYBVLQBDAVDQP4EVB4NOKFA2QAYWYE7R4YRJZUQS5MQ, seg_pos={0 0}, upload=463129]: Uploading piece...
Dec 27 15:30:03.468659359 (528h1m3.339742404s) [noise=true , address=145.79.227.28:4727   , piece_id=U5IUWVWOR6QLUXMGVPYLHSLH3FLUYMJOFPIBB66CJC2VL2NLXZLA, seg_pos={0 0}, upload=463129]: Uploading piece...
Dec 27 15:30:05.126254340 (528h1m4.997337385s) [upload=463129]: Next returned nil segment
Dec 27 15:30:05.126274680 (528h1m4.997357735s) [upload=463129]: Waiting for error group managing segments...
Dec 27 15:30:05.530162829 (528h1m5.401245884s) [noise=true , address=145.79.227.15:4876   , piece_id=6VYNR3VMOGAUGCPQETKAF244ZOQUPRECLBO6NWV6JPTGOOFAY3AA, seg_pos={0 0}, upload=463129]: Done uploading piece. err: <nil>
Dec 27 15:30:05.540036951 (528h1m5.411119996s) [noise=true , address=145.79.227.46:4603   , piece_id=SC6S4AJVSRWBUYWLVPSAUXNTAZ47HVH5QNEAX5LUS2WOGNJS6OMA, seg_pos={0 0}, upload=463129]: Done uploading piece. err: <nil>
Dec 27 15:30:05.562110754 (528h1m5.433193799s) [noise=true , address=145.79.227.13:4804   , piece_id=D2PVRI5AXLIS7V5RRDY5ZIJUWUB6JQ5CW6STLJSO26MLA2EO2RTQ, seg_pos={0 0}, upload=463129]: Done uploading piece. err: <nil>
Dec 27 15:30:05.562660631 (528h1m5.433743676s) [noise=true , address=145.79.227.51:4411   , piece_id=B46UJTZWQ5I5AVHTP3NIZ5T7R2XEDTJ7MDVH2R32S34IHQ25QIUA, seg_pos={0 0}, upload=463129]: Done uploading piece. err: <nil>
Dec 27 15:30:05.573199559 (528h1m5.444282664s) [noise=true , address=145.79.227.28:4727   , piece_id=U5IUWVWOR6QLUXMGVPYLHSLH3FLUYMJOFPIBB66CJC2VL2NLXZLA, seg_pos={0 0}, upload=463129]: Done uploading piece. err: <nil>
Dec 27 15:30:05.575287635 (528h1m5.44637068s) [noise=true , address=145.79.227.50:4373   , piece_id=Z5SMO4PCHSCFMDCDY7XSOLYA6W4I5ZWZZJKLZXSDGONKLQFL7PCA, seg_pos={0 0}, upload=463129]: Done uploading piece. err: <nil>
Dec 27 15:30:05.577932291 (528h1m5.449015336s) [noise=true , address=145.79.227.10:4403   , piece_id=DLAPXIGKGR4CTFMUWJQKC6IO6AV3IOSO2JQFVB7PVMFKPSYNFFEQ, seg_pos={0 0}, upload=463129]: Done uploading piece. err: <nil>
Dec 27 15:30:05.583145548 (528h1m5.454228593s) [noise=true , address=145.79.227.45:4231   , piece_id=FIXT3Q56D6F2JGFFFK3EIQPJOGL4QII2DVYFC6OJ7E3L6FRSAJOA, seg_pos={0 0}, upload=463129]: Done uploading piece. err: <nil>
Dec 27 15:30:05.588678233 (528h1m5.459761278s) [noise=true , address=145.79.227.3:4150    , piece_id=XMPRVXIHPZ6G3YY5LRWVTCCOVAR4RHIHYSO2FC6I5XPU74GJJW3A, seg_pos={0 0}, upload=463129]: Done uploading piece. err: <nil>
Dec 27 15:30:05.598997924 (528h1m5.470080969s) [noise=true , address=145.79.227.37:4566   , piece_id=XYLUOY3VMVSXMXQB4F2EBQWIA424NJJYDJCSMWIZP4UEG7SY55IA, seg_pos={0 0}, upload=463129]: Done uploading piece. err: <nil>
Dec 27 15:30:05.824274082 (528h1m5.695357127s) [noise=true , address=77.90.31.10:4517     , piece_id=SSHEERUL26LLWTIRKLGHOH5Y3XWQ3VNGYOUFURIE3QX67NA7QK4A, seg_pos={0 0}, upload=463129]: Done uploading piece. err: <nil>
Dec 27 15:30:06.398684147 (528h1m6.269767202s) [noise=true , address=77.90.31.11:4553     , piece_id=I4NMMYGC4PYBVLQBDAVDQP4EVB4NOKFA2QAYWYE7R4YRJZUQS5MQ, seg_pos={0 0}, upload=463129]: Done uploading piece. err: <nil>
Dec 27 15:30:06.459209859 (528h1m6.330292904s) [noise=true , address=45.91.116.6:4975     , piece_id=3O43RYN3EOZ4I37XK3GK54BWJF4XRGWN5ETKUNZYZONWT4WX72XA, seg_pos={0 0}, upload=463129]: Done uploading piece. err: <nil>
Dec 27 15:30:06.459240017 (528h1m6.330323052s) [seg_pos={0 0}, upload=463129]: Segment reached optimal threshold of 13 pieces.
Dec 27 15:30:06.459463141 (528h1m6.330546176s) [noise=true , address=145.79.227.40:4049   , piece_id=LMVUBWD4I3Q3GNZC6V6E6ZBGOLR6Z4K2QE634MTW2BLL73JY35YA, seg_pos={0 0}, upload=463129]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF
Dec 27 15:30:06.459470576 (528h1m6.330553611s) [noise=true , address=145.79.227.40:4049   , piece_id=LMVUBWD4I3Q3GNZC6V6E6ZBGOLR6Z4K2QE634MTW2BLL73JY35YA, seg_pos={0 0}, upload=463129]: Long tail cancellation due to optimal threshold
Dec 27 15:30:06.459697808 (528h1m6.330780853s) [noise=true , address=145.79.227.43:4159   , piece_id=OZQE36KJGMUCMTF3XWTJTMSC7SD7QNVSJYQEFGBEK6BCJET2P4CQ, seg_pos={0 0}, upload=463129]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF
Dec 27 15:30:06.459717607 (528h1m6.330800652s) [noise=true , address=145.79.227.43:4159   , piece_id=OZQE36KJGMUCMTF3XWTJTMSC7SD7QNVSJYQEFGBEK6BCJET2P4CQ, seg_pos={0 0}, upload=463129]: Long tail cancellation due to optimal threshold
Dec 27 15:30:06.459773963 (528h1m6.330857008s) [noise=true , address=45.91.116.4:4937     , piece_id=LFI5SJVUI5CQ6XB2ZIPJA5METXQTLTG3FEFSE5HLLTJJPH5XBBRA, seg_pos={0 0}, upload=463129]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF
Dec 27 15:30:06.459782389 (528h1m6.330865434s) [noise=true , address=45.91.116.4:4937     , piece_id=LFI5SJVUI5CQ6XB2ZIPJA5METXQTLTG3FEFSE5HLLTJJPH5XBBRA, seg_pos={0 0}, upload=463129]: Long tail cancellation due to optimal threshold
Dec 27 15:30:06.459831834 (528h1m6.330914879s) [seg_pos={0 0}, upload=463129]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 15:30:06.459916374 (528h1m6.330999409s) [upload=463129]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 15:30:06.459922235 (528h1m6.33100528s) [upload=463129]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 15:30:06.753073741 (528h1m6.624156786s) [upload=463129]: Done uploading segments...
Dec 27 15:30:06.753098749 (528h1m6.624181794s) [upload=463129]: Upload finished. err: <nil>
Duration: 6.112262976s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
33 out of 60 is failed (threshold=6) [..XXX..X..XX.XXXX.....XXX.X.XXXX.X.XXXX...XXXX.XXXXX...X....]
duration: 
5.000991788s
trace: 
https://jaeger.dev.storj.io/trace/5973653b14c3bbcb
FAIL (2025-12-27 15:30:41 UTC) 
orbiter warning: context deadline exceeded
Duration: 5.001076729s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
22 out of 60 is failed (threshold=3) [X..XX.......X..X.X..X...XXXXXXX..X.X..X.X.....X...X......X.X]
duration: 
30.003048976s
trace: 
https://jaeger.dev.storj.io/trace/232738e743a7710f
FAIL (2025-12-27 15:30:32 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: SSHEERUL26LLWTIRKLGHOH5Y3XWQ3VNGYOUFURIE3QX67NA7QK4A) context deadline exceeded; piecestore: (Node ID: 1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, Piece ID: I4NMMYGC4PYBVLQBDAVDQP4EVB4NOKFA2QAYWYE7R4YRJZUQS5MQ) context deadline exceeded
======LOGS======
Dec 27 15:30:03.202013435 (528h1m3.07309648s) [upload=463129]: Starting upload
Dec 27 15:30:03.202162749 (528h1m3.073245794s) [upload=463129]: Done starting upload
Dec 27 15:30:03.202279562 (528h1m3.073362607s) [upload=463129]: Uploading segments...
Dec 27 15:30:03.202458873 (528h1m3.073541918s) [upload=463129]: Got next segment. Inline: false
Dec 27 15:30:03.202541731 (528h1m3.073624776s) [upload=463129]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 15:30:03.202550759 (528h1m3.073633824s) [upload=463129]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 15:30:03.447228601 (528h1m3.318311656s) [seg_pos={0 0}, upload=463129]: Begin upload segment...
Dec 27 15:30:03.447292532 (528h1m3.318375578s) [seg_pos={0 0}, upload=463129]: Done begin upload segment.
Dec 27 15:30:03.447345413 (528h1m3.318428488s) [noise=true , address=145.79.227.43:4159   , piece_id=OZQE36KJGMUCMTF3XWTJTMSC7SD7QNVSJYQEFGBEK6BCJET2P4CQ, seg_pos={0 0}, upload=463129]: Uploading piece...
Dec 27 15:30:03.447475240 (528h1m3.318558295s) [noise=true , address=45.91.116.4:4937     , piece_id=LFI5SJVUI5CQ6XB2ZIPJA5METXQTLTG3FEFSE5HLLTJJPH5XBBRA, seg_pos={0 0}, upload=463129]: Uploading piece...
Dec 27 15:30:03.448000881 (528h1m3.319083926s) [noise=true , address=145.79.227.3:4150    , piece_id=XMPRVXIHPZ6G3YY5LRWVTCCOVAR4RHIHYSO2FC6I5XPU74GJJW3A, seg_pos={0 0}, upload=463129]: Uploading piece...
Dec 27 15:30:03.448524428 (528h1m3.319607473s) [noise=true , address=145.79.227.46:4603   , piece_id=SC6S4AJVSRWBUYWLVPSAUXNTAZ47HVH5QNEAX5LUS2WOGNJS6OMA, seg_pos={0 0}, upload=463129]: Uploading piece...
Dec 27 15:30:03.450312654 (528h1m3.321395699s) [noise=true , address=145.79.227.40:4049   , piece_id=LMVUBWD4I3Q3GNZC6V6E6ZBGOLR6Z4K2QE634MTW2BLL73JY35YA, seg_pos={0 0}, upload=463129]: Uploading piece...
Dec 27 15:30:03.450833946 (528h1m3.321916991s) [noise=true , address=145.79.227.51:4411   , piece_id=B46UJTZWQ5I5AVHTP3NIZ5T7R2XEDTJ7MDVH2R32S34IHQ25QIUA, seg_pos={0 0}, upload=463129]: Uploading piece...
Dec 27 15:30:03.451380025 (528h1m3.32246308s) [noise=true , address=145.79.227.13:4804   , piece_id=D2PVRI5AXLIS7V5RRDY5ZIJUWUB6JQ5CW6STLJSO26MLA2EO2RTQ, seg_pos={0 0}, upload=463129]: Uploading piece...
Dec 27 15:30:03.451886691 (528h1m3.322969736s) [noise=true , address=145.79.227.45:4231   , piece_id=FIXT3Q56D6F2JGFFFK3EIQPJOGL4QII2DVYFC6OJ7E3L6FRSAJOA, seg_pos={0 0}, upload=463129]: Uploading piece...
Dec 27 15:30:03.452628703 (528h1m3.323711748s) [noise=true , address=145.79.227.15:4876   , piece_id=6VYNR3VMOGAUGCPQETKAF244ZOQUPRECLBO6NWV6JPTGOOFAY3AA, seg_pos={0 0}, upload=463129]: Uploading piece...
Dec 27 15:30:03.453370095 (528h1m3.32445314s) [noise=true , address=45.91.116.6:4975     , piece_id=3O43RYN3EOZ4I37XK3GK54BWJF4XRGWN5ETKUNZYZONWT4WX72XA, seg_pos={0 0}, upload=463129]: Uploading piece...
Dec 27 15:30:03.456040610 (528h1m3.327123665s) [noise=true , address=145.79.227.10:4403   , piece_id=DLAPXIGKGR4CTFMUWJQKC6IO6AV3IOSO2JQFVB7PVMFKPSYNFFEQ, seg_pos={0 0}, upload=463129]: Uploading piece...
Dec 27 15:30:03.456859440 (528h1m3.327942515s) [noise=true , address=145.79.227.50:4373   , piece_id=Z5SMO4PCHSCFMDCDY7XSOLYA6W4I5ZWZZJKLZXSDGONKLQFL7PCA, seg_pos={0 0}, upload=463129]: Uploading piece...
Dec 27 15:30:03.459937971 (528h1m3.331021016s) [noise=true , address=145.79.227.37:4566   , piece_id=XYLUOY3VMVSXMXQB4F2EBQWIA424NJJYDJCSMWIZP4UEG7SY55IA, seg_pos={0 0}, upload=463129]: Uploading piece...
Dec 27 15:30:03.460531822 (528h1m3.331614867s) [noise=true , address=77.90.31.10:4517     , piece_id=SSHEERUL26LLWTIRKLGHOH5Y3XWQ3VNGYOUFURIE3QX67NA7QK4A, seg_pos={0 0}, upload=463129]: Uploading piece...
Dec 27 15:30:03.465021191 (528h1m3.336104236s) [noise=true , address=77.90.31.11:4553     , piece_id=I4NMMYGC4PYBVLQBDAVDQP4EVB4NOKFA2QAYWYE7R4YRJZUQS5MQ, seg_pos={0 0}, upload=463129]: Uploading piece...
Dec 27 15:30:03.468659359 (528h1m3.339742404s) [noise=true , address=145.79.227.28:4727   , piece_id=U5IUWVWOR6QLUXMGVPYLHSLH3FLUYMJOFPIBB66CJC2VL2NLXZLA, seg_pos={0 0}, upload=463129]: Uploading piece...
Dec 27 15:30:05.126254340 (528h1m4.997337385s) [upload=463129]: Next returned nil segment
Dec 27 15:30:05.126274680 (528h1m4.997357735s) [upload=463129]: Waiting for error group managing segments...
Dec 27 15:30:05.530162829 (528h1m5.401245884s) [noise=true , address=145.79.227.15:4876   , piece_id=6VYNR3VMOGAUGCPQETKAF244ZOQUPRECLBO6NWV6JPTGOOFAY3AA, seg_pos={0 0}, upload=463129]: Done uploading piece. err: <nil>
Dec 27 15:30:05.540036951 (528h1m5.411119996s) [noise=true , address=145.79.227.46:4603   , piece_id=SC6S4AJVSRWBUYWLVPSAUXNTAZ47HVH5QNEAX5LUS2WOGNJS6OMA, seg_pos={0 0}, upload=463129]: Done uploading piece. err: <nil>
Dec 27 15:30:05.562110754 (528h1m5.433193799s) [noise=true , address=145.79.227.13:4804   , piece_id=D2PVRI5AXLIS7V5RRDY5ZIJUWUB6JQ5CW6STLJSO26MLA2EO2RTQ, seg_pos={0 0}, upload=463129]: Done uploading piece. err: <nil>
Dec 27 15:30:05.562660631 (528h1m5.433743676s) [noise=true , address=145.79.227.51:4411   , piece_id=B46UJTZWQ5I5AVHTP3NIZ5T7R2XEDTJ7MDVH2R32S34IHQ25QIUA, seg_pos={0 0}, upload=463129]: Done uploading piece. err: <nil>
Dec 27 15:30:05.573199559 (528h1m5.444282664s) [noise=true , address=145.79.227.28:4727   , piece_id=U5IUWVWOR6QLUXMGVPYLHSLH3FLUYMJOFPIBB66CJC2VL2NLXZLA, seg_pos={0 0}, upload=463129]: Done uploading piece. err: <nil>
Dec 27 15:30:05.575287635 (528h1m5.44637068s) [noise=true , address=145.79.227.50:4373   , piece_id=Z5SMO4PCHSCFMDCDY7XSOLYA6W4I5ZWZZJKLZXSDGONKLQFL7PCA, seg_pos={0 0}, upload=463129]: Done uploading piece. err: <nil>
Dec 27 15:30:05.577932291 (528h1m5.449015336s) [noise=true , address=145.79.227.10:4403   , piece_id=DLAPXIGKGR4CTFMUWJQKC6IO6AV3IOSO2JQFVB7PVMFKPSYNFFEQ, seg_pos={0 0}, upload=463129]: Done uploading piece. err: <nil>
Dec 27 15:30:05.583145548 (528h1m5.454228593s) [noise=true , address=145.79.227.45:4231   , piece_id=FIXT3Q56D6F2JGFFFK3EIQPJOGL4QII2DVYFC6OJ7E3L6FRSAJOA, seg_pos={0 0}, upload=463129]: Done uploading piece. err: <nil>
Dec 27 15:30:05.588678233 (528h1m5.459761278s) [noise=true , address=145.79.227.3:4150    , piece_id=XMPRVXIHPZ6G3YY5LRWVTCCOVAR4RHIHYSO2FC6I5XPU74GJJW3A, seg_pos={0 0}, upload=463129]: Done uploading piece. err: <nil>
Dec 27 15:30:05.598997924 (528h1m5.470080969s) [noise=true , address=145.79.227.37:4566   , piece_id=XYLUOY3VMVSXMXQB4F2EBQWIA424NJJYDJCSMWIZP4UEG7SY55IA, seg_pos={0 0}, upload=463129]: Done uploading piece. err: <nil>
Dec 27 15:30:05.824274082 (528h1m5.695357127s) [noise=true , address=77.90.31.10:4517     , piece_id=SSHEERUL26LLWTIRKLGHOH5Y3XWQ3VNGYOUFURIE3QX67NA7QK4A, seg_pos={0 0}, upload=463129]: Done uploading piece. err: <nil>
Dec 27 15:30:06.398684147 (528h1m6.269767202s) [noise=true , address=77.90.31.11:4553     , piece_id=I4NMMYGC4PYBVLQBDAVDQP4EVB4NOKFA2QAYWYE7R4YRJZUQS5MQ, seg_pos={0 0}, upload=463129]: Done uploading piece. err: <nil>
Dec 27 15:30:06.459209859 (528h1m6.330292904s) [noise=true , address=45.91.116.6:4975     , piece_id=3O43RYN3EOZ4I37XK3GK54BWJF4XRGWN5ETKUNZYZONWT4WX72XA, seg_pos={0 0}, upload=463129]: Done uploading piece. err: <nil>
Dec 27 15:30:06.459240017 (528h1m6.330323052s) [seg_pos={0 0}, upload=463129]: Segment reached optimal threshold of 13 pieces.
Dec 27 15:30:06.459463141 (528h1m6.330546176s) [noise=true , address=145.79.227.40:4049   , piece_id=LMVUBWD4I3Q3GNZC6V6E6ZBGOLR6Z4K2QE634MTW2BLL73JY35YA, seg_pos={0 0}, upload=463129]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF
Dec 27 15:30:06.459470576 (528h1m6.330553611s) [noise=true , address=145.79.227.40:4049   , piece_id=LMVUBWD4I3Q3GNZC6V6E6ZBGOLR6Z4K2QE634MTW2BLL73JY35YA, seg_pos={0 0}, upload=463129]: Long tail cancellation due to optimal threshold
Dec 27 15:30:06.459697808 (528h1m6.330780853s) [noise=true , address=145.79.227.43:4159   , piece_id=OZQE36KJGMUCMTF3XWTJTMSC7SD7QNVSJYQEFGBEK6BCJET2P4CQ, seg_pos={0 0}, upload=463129]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF
Dec 27 15:30:06.459717607 (528h1m6.330800652s) [noise=true , address=145.79.227.43:4159   , piece_id=OZQE36KJGMUCMTF3XWTJTMSC7SD7QNVSJYQEFGBEK6BCJET2P4CQ, seg_pos={0 0}, upload=463129]: Long tail cancellation due to optimal threshold
Dec 27 15:30:06.459773963 (528h1m6.330857008s) [noise=true , address=45.91.116.4:4937     , piece_id=LFI5SJVUI5CQ6XB2ZIPJA5METXQTLTG3FEFSE5HLLTJJPH5XBBRA, seg_pos={0 0}, upload=463129]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF
Dec 27 15:30:06.459782389 (528h1m6.330865434s) [noise=true , address=45.91.116.4:4937     , piece_id=LFI5SJVUI5CQ6XB2ZIPJA5METXQTLTG3FEFSE5HLLTJJPH5XBBRA, seg_pos={0 0}, upload=463129]: Long tail cancellation due to optimal threshold
Dec 27 15:30:06.459831834 (528h1m6.330914879s) [seg_pos={0 0}, upload=463129]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 15:30:06.459916374 (528h1m6.330999409s) [upload=463129]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 15:30:06.459922235 (528h1m6.33100528s) [upload=463129]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 15:30:06.753073741 (528h1m6.624156786s) [upload=463129]: Done uploading segments...
Dec 27 15:30:06.753098749 (528h1m6.624181794s) [upload=463129]: Upload finished. err: <nil>
Duration: 30.003077991s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
33 out of 60 is failed (threshold=6) [...XXX..X..XX.XXXX.....XXX.X.XXXX.X.XXXX...XXXX.XXXXX...X...]
duration: 
5.001325296s
trace: 
https://jaeger.dev.storj.io/trace/5973653b14c3bbcb
FAIL (2025-12-27 15:30:18 UTC) 
orbiter warning: context deadline exceeded
Duration: 5.001355895s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
7 out of 60 is failed (threshold=3) [.....X..............X....X...................X.....XX......X]
duration: 
1m0.005181433s
trace: 
https://jaeger.dev.storj.io/trace/24120a988203c524
FAIL (2025-12-27 15:30:08 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 191MKpEnoarS7sK4Gs63ZM8zGc5jevFX8VeqViP82duW33nJJz, Piece ID: 6GXHQB47DWIXBK4ZTC5QIA4362JRMBOQKXXGBZ3M6FVCLFSZIE2A) context deadline exceeded; piecestore: (Node ID: 1b2aaYQ4o7X5yLcXG3moJ3d4oxFoNSCwsqacwg9mshJ4MPoGdg, Piece ID: JNNC647LJKVXZIIWY4GXSKOJ3OY3WWGPHWKFWGO5TI6M2SWIXHIA) context deadline exceeded; piecestore: (Node ID: 12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, Piece ID: L2BJ6DADQGARNEFIF7M4AKQ4RDDDER64NZUKOYKAT7QBMMOWVEIA) context deadline exceeded
======LOGS======
GEJCQQVP2NVYJ6LZWA3HUA, seg_pos={0 0}, upload=463114]: Uploading piece...
Dec 27 15:29:09.918056404 (528h0m9.789139439s) [noise=true , address=77.90.31.13:4645     , piece_id=JNNC647LJKVXZIIWY4GXSKOJ3OY3WWGPHWKFWGO5TI6M2SWIXHIA, seg_pos={0 0}, upload=463114]: Uploading piece...
Dec 27 15:29:09.918708286 (528h0m9.789791331s) [noise=true , address=77.90.31.12:4593     , piece_id=RMUWIRJ2XRORPV4RJKRL2JUZZOMMOGO3TDTWM22B7LTCERERYHWQ, seg_pos={0 0}, upload=463114]: Uploading piece...
Dec 27 15:29:09.919456690 (528h0m9.790539736s) [noise=true , address=77.90.31.14:4695     , piece_id=H265WN7TXOO3LG4QTFAB5S3TE2MUON7OCCQ7H5PA5NIBCEVGOM4A, seg_pos={0 0}, upload=463114]: Uploading piece...
Dec 27 15:29:09.920039571 (528h0m9.791122616s) [noise=true , address=145.79.227.35:4512   , piece_id=DX4PHB5M4NDOFTWWEGJM43D4VBIRIGADX4HILEMRO2ZU4OV2S47Q, seg_pos={0 0}, upload=463114]: Uploading piece...
Dec 27 15:29:09.920276141 (528h0m9.791359186s) [noise=true , address=145.79.227.34:4926   , piece_id=LE5TDKNEJ2QPKW2ZIMMPBO56TAQRGFTSCAR5IHD66X4E6FIYXT5Q, seg_pos={0 0}, upload=463114]: Uploading piece...
Dec 27 15:29:09.920474529 (528h0m9.791557564s) [noise=true , address=145.79.227.13:4816   , piece_id=2OCASX7RNCMKBIVDVELFO7WIHKZMY27RII5C2BWIZXIG33BBAV5Q, seg_pos={0 0}, upload=463114]: Uploading piece...
Dec 27 15:29:09.921046528 (528h0m9.792129563s) [noise=true , address=145.79.227.40:4069   , piece_id=Z3ZYDWOIMGFRIRYP4WKYSL4YIJF7TZT73EOVUYQOOMNM42PGWHBQ, seg_pos={0 0}, upload=463114]: Uploading piece...
Dec 27 15:29:09.921603759 (528h0m9.792686794s) [noise=true , address=77.90.31.21:4883     , piece_id=6GXHQB47DWIXBK4ZTC5QIA4362JRMBOQKXXGBZ3M6FVCLFSZIE2A, seg_pos={0 0}, upload=463114]: Uploading piece...
Dec 27 15:29:09.922032846 (528h0m9.793115881s) [noise=true , address=145.79.227.44:4210   , piece_id=JMSSYMTYXUCJRFCOUCVRK4W6KX7LTGSCKKAO7XC7UGWKNGHNEMQQ, seg_pos={0 0}, upload=463114]: Uploading piece...
Dec 27 15:29:09.923593568 (528h0m9.794676613s) [noise=true , address=45.91.116.10:4103    , piece_id=QU2UHH6G7QETR2DF3A6U3NZLYXY3WD5JO4GKU3HRUO35CJJP5LJA, seg_pos={0 0}, upload=463114]: Uploading piece...
Dec 27 15:29:09.924048133 (528h0m9.795131169s) [noise=true , address=77.90.31.18:4840     , piece_id=L2BJ6DADQGARNEFIF7M4AKQ4RDDDER64NZUKOYKAT7QBMMOWVEIA, seg_pos={0 0}, upload=463114]: Uploading piece...
Dec 27 15:29:09.924539719 (528h0m9.795622754s) [noise=true , address=64.49.13.139:4470    , piece_id=3MAQ4R3XTRQT72O3DEIQSAFZAGT2CJTMPT2IKLCDLIVLPESQRVGA, seg_pos={0 0}, upload=463114]: Uploading piece...
Dec 27 15:29:09.925024673 (528h0m9.796107708s) [noise=true , address=145.79.227.49:4368   , piece_id=3LXJYQGC74WHBFCISTNZMBVLSUGSMU7XOZIFP5ZRU57TXJ3YTM4Q, seg_pos={0 0}, upload=463114]: Uploading piece...
Dec 27 15:29:09.925500990 (528h0m9.796584035s) [noise=true , address=77.90.31.17:4795     , piece_id=FTJLV46VS5BACPHHVVOCU7EEOEU3Q66P3B3WTPUT3ACG6GLYTDUA, seg_pos={0 0}, upload=463114]: Uploading piece...
Dec 27 15:29:11.205949065 (528h0m11.0770321s) [upload=463114]: Next returned nil segment
Dec 27 15:29:11.205963343 (528h0m11.077046378s) [upload=463114]: Waiting for error group managing segments...
Dec 27 15:29:11.610155861 (528h0m11.481238906s) [noise=true , address=64.49.13.139:4470    , piece_id=3MAQ4R3XTRQT72O3DEIQSAFZAGT2CJTMPT2IKLCDLIVLPESQRVGA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil>
Dec 27 15:29:11.812270605 (528h0m11.683353641s) [noise=true , address=145.79.227.35:4512   , piece_id=DX4PHB5M4NDOFTWWEGJM43D4VBIRIGADX4HILEMRO2ZU4OV2S47Q, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil>
Dec 27 15:29:11.818242477 (528h0m11.689325522s) [noise=true , address=145.79.227.13:4816   , piece_id=2OCASX7RNCMKBIVDVELFO7WIHKZMY27RII5C2BWIZXIG33BBAV5Q, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil>
Dec 27 15:29:11.938211621 (528h0m11.809294666s) [noise=true , address=145.79.227.49:4368   , piece_id=3LXJYQGC74WHBFCISTNZMBVLSUGSMU7XOZIFP5ZRU57TXJ3YTM4Q, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil>
Dec 27 15:29:12.064896693 (528h0m11.935979738s) [noise=true , address=77.90.31.17:4795     , piece_id=FTJLV46VS5BACPHHVVOCU7EEOEU3Q66P3B3WTPUT3ACG6GLYTDUA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil>
Dec 27 15:29:12.070731273 (528h0m11.941814308s) [noise=true , address=77.90.31.21:4883     , piece_id=6GXHQB47DWIXBK4ZTC5QIA4362JRMBOQKXXGBZ3M6FVCLFSZIE2A, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil>
Dec 27 15:29:12.107949174 (528h0m11.979032239s) [noise=true , address=77.90.31.11:4573     , piece_id=SRXHTEXDEKKJ2WPGY66BOAP7WD7Z75HK3B5DL2XIGFWDYQGI76YA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil>
Dec 27 15:29:12.151478532 (528h0m12.022561567s) [noise=true , address=77.90.31.13:4645     , piece_id=JNNC647LJKVXZIIWY4GXSKOJ3OY3WWGPHWKFWGO5TI6M2SWIXHIA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil>
Dec 27 15:29:12.335397247 (528h0m12.206480292s) [noise=true , address=45.91.116.10:4103    , piece_id=QU2UHH6G7QETR2DF3A6U3NZLYXY3WD5JO4GKU3HRUO35CJJP5LJA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil>
Dec 27 15:29:12.410397992 (528h0m12.281481027s) [noise=true , address=77.90.31.18:4840     , piece_id=L2BJ6DADQGARNEFIF7M4AKQ4RDDDER64NZUKOYKAT7QBMMOWVEIA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil>
Dec 27 15:29:12.667101894 (528h0m12.538184929s) [noise=true , address=77.90.31.12:4593     , piece_id=RMUWIRJ2XRORPV4RJKRL2JUZZOMMOGO3TDTWM22B7LTCERERYHWQ, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil>
Dec 27 15:29:12.785380610 (528h0m12.656463655s) [noise=true , address=145.79.227.44:4210   , piece_id=JMSSYMTYXUCJRFCOUCVRK4W6KX7LTGSCKKAO7XC7UGWKNGHNEMQQ, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil>
Dec 27 15:29:12.828722731 (528h0m12.699805766s) [noise=true , address=193.149.164.143:4697 , piece_id=S4CQ3366KVFS5RMZR67MPGSLXKDWRAGEJCQQVP2NVYJ6LZWA3HUA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil>
Dec 27 15:29:12.947757466 (528h0m12.818840501s) [noise=true , address=145.79.227.34:4926   , piece_id=LE5TDKNEJ2QPKW2ZIMMPBO56TAQRGFTSCAR5IHD66X4E6FIYXT5Q, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil>
Dec 27 15:29:12.947777434 (528h0m12.818860469s) [seg_pos={0 0}, upload=463114]: Segment reached optimal threshold of 14 pieces.
Dec 27 15:29:12.947947177 (528h0m12.819030212s) [noise=true , address=77.90.31.14:4695     , piece_id=H265WN7TXOO3LG4QTFAB5S3TE2MUON7OCCQ7H5PA5NIBCEVGOM4A, seg_pos={0 0}, upload=463114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12q2kob5oYxuxwUDh4vjNDBpW4c6LndrjHbsJ4VRDiBsLxjgLdy): protocol: write tcp 188.166.225.170:36046->77.90.31.14:4695: use of closed network connection; write tcp 188.166.225.170:36046->77.90.31.14:4695: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36046->77.90.31.14:4695: use of closed network connection
Dec 27 15:29:12.947953169 (528h0m12.819036204s) [noise=true , address=77.90.31.14:4695     , piece_id=H265WN7TXOO3LG4QTFAB5S3TE2MUON7OCCQ7H5PA5NIBCEVGOM4A, seg_pos={0 0}, upload=463114]: Long tail cancellation due to optimal threshold
Dec 27 15:29:12.948200800 (528h0m12.819283845s) [noise=true , address=77.90.31.19:4869     , piece_id=KENCENCTEBHPVIPCECDQYLIWHYLT6G7ZETYMN4DA6LPQRLVQETNA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8): protocol: read tcp 188.166.225.170:32932->77.90.31.19:4869: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 27 15:29:12.948216079 (528h0m12.819299124s) [noise=true , address=77.90.31.19:4869     , piece_id=KENCENCTEBHPVIPCECDQYLIWHYLT6G7ZETYMN4DA6LPQRLVQETNA, seg_pos={0 0}, upload=463114]: Long tail cancellation due to optimal threshold
Dec 27 15:29:12.948310158 (528h0m12.819393203s) [noise=true , address=64.49.13.169:4711    , piece_id=IMYFGTTZJUS5CEYCF75SUFUIGA3QFQYTFVCKYKFBM2DDQIKQHS7A, seg_pos={0 0}, upload=463114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:51826->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:51826->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51826->64.49.13.169:4711: use of closed network connection
Dec 27 15:29:12.948316139 (528h0m12.819399174s) [noise=true , address=64.49.13.169:4711    , piece_id=IMYFGTTZJUS5CEYCF75SUFUIGA3QFQYTFVCKYKFBM2DDQIKQHS7A, seg_pos={0 0}, upload=463114]: Long tail cancellation due to optimal threshold
Dec 27 15:29:12.948360164 (528h0m12.819443219s) [noise=true , address=145.79.227.40:4069   , piece_id=Z3ZYDWOIMGFRIRYP4WKYSL4YIJF7TZT73EOVUYQOOMNM42PGWHBQ, seg_pos={0 0}, upload=463114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF
Dec 27 15:29:12.948372617 (528h0m12.819455652s) [noise=true , address=145.79.227.40:4069   , piece_id=Z3ZYDWOIMGFRIRYP4WKYSL4YIJF7TZT73EOVUYQOOMNM42PGWHBQ, seg_pos={0 0}, upload=463114]: Long tail cancellation due to optimal threshold
Dec 27 15:29:12.948398968 (528h0m12.819482013s) [seg_pos={0 0}, upload=463114]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 15:29:12.948477878 (528h0m12.819560913s) [upload=463114]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 15:29:12.948483889 (528h0m12.819566934s) [upload=463114]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 15:29:13.237238452 (528h0m13.108321498s) [upload=463114]: Done uploading segments...
Dec 27 15:29:13.237262939 (528h0m13.108345984s) [upload=463114]: Upload finished. err: <nil>
Duration: 1m0.00519533s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
33 out of 60 is failed (threshold=6) [....XXX..X..XX.XXXX.....XXX.X.XXXX.X.XXXX...XXXX.XXXXX...X..]
duration: 
5.000879034s
trace: 
https://jaeger.dev.storj.io/trace/5973653b14c3bbcb
FAIL (2025-12-27 15:29:12 UTC) 
orbiter warning: context deadline exceeded
Duration: 5.00090359s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
21 out of 60 is failed (threshold=3) [.X..XX.......X..X.X..X...XXXXXXX..X.X..X.X.....X...X......X.]
duration: 
5.621801337s
trace: 
https://jaeger.dev.storj.io/trace/7b79ac0553c90089
FAIL (2025-12-27 15:29:08 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: TQYSZ7MGYYPBE5JMSWV5R6BQMLQ4HMWFFQ4W54O4E75OAUWRPCGQ) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: GTZH5HZ3H24H4ROS6ELSUKJBGKEUTKD5HJTVNJ5BRETRYRTXGCNQ) context deadline exceeded
======LOGS======
Dec 27 15:28:33.465851383 (527h59m33.336934418s) [upload=463106]: Starting upload
Dec 27 15:28:33.465907279 (527h59m33.336990324s) [upload=463106]: Done starting upload
Dec 27 15:28:33.466096420 (527h59m33.337179465s) [upload=463106]: Uploading segments...
Dec 27 15:28:33.466134482 (527h59m33.337217527s) [upload=463106]: Got next segment. Inline: false
Dec 27 15:28:33.466163618 (527h59m33.337246653s) [upload=463106]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 15:28:33.466168387 (527h59m33.337251432s) [upload=463106]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 15:28:33.712366784 (527h59m33.583449819s) [seg_pos={0 0}, upload=463106]: Begin upload segment...
Dec 27 15:28:33.712410658 (527h59m33.583493703s) [seg_pos={0 0}, upload=463106]: Done begin upload segment.
Dec 27 15:28:33.712436357 (527h59m33.583519402s) [noise=true , address=145.79.227.3:4151    , piece_id=IWDWU4M6NF4GF4SN3JDZ3GMALCLFFHYXFIZYOYJZEJ7MSRHG3YOA, seg_pos={0 0}, upload=463106]: Uploading piece...
Dec 27 15:28:33.712481533 (527h59m33.583564578s) [noise=true , address=145.79.227.48:4302   , piece_id=AEMMBCPNN74CGSXA2PID3PMWRPXYEBHXGTS52DC6W7KT667WBR2Q, seg_pos={0 0}, upload=463106]: Uploading piece...
Dec 27 15:28:33.712939194 (527h59m33.584022229s) [noise=true , address=145.79.227.47:4265   , piece_id=43C3U3Y4CC5RW47RXAKPCNKL22SKN35QLUB3PLTWLKQ6F7QE5EJQ, seg_pos={0 0}, upload=463106]: Uploading piece...
Dec 27 15:28:33.713315811 (527h59m33.584398856s) [noise=true , address=45.91.116.6:4975     , piece_id=TQYSZ7MGYYPBE5JMSWV5R6BQMLQ4HMWFFQ4W54O4E75OAUWRPCGQ, seg_pos={0 0}, upload=463106]: Uploading piece...
Dec 27 15:28:33.713776077 (527h59m33.584859123s) [noise=true , address=45.91.116.8:4046     , piece_id=GTZH5HZ3H24H4ROS6ELSUKJBGKEUTKD5HJTVNJ5BRETRYRTXGCNQ, seg_pos={0 0}, upload=463106]: Uploading piece...
Dec 27 15:28:33.714231715 (527h59m33.58531476s) [noise=true , address=45.91.116.9:4263     , piece_id=RJRCKSNQTSSJP7EDTRLLDBLEOGRV7D4Y3FS5LJUMAAAZRWF4QALQ, seg_pos={0 0}, upload=463106]: Uploading piece...
Dec 27 15:28:33.714685930 (527h59m33.585768965s) [noise=true , address=145.79.227.8:4329    , piece_id=FSACQITU5R4HCBHWWS4CIJLHPP2HTTRK5BYEZUFVOZIVZLTHXJOQ, seg_pos={0 0}, upload=463106]: Uploading piece...
Dec 27 15:28:33.715251548 (527h59m33.586334593s) [noise=true , address=145.79.227.50:4373   , piece_id=GIJMUCTZUIZRVUMLHLKETBCTCVUEZH4GDJGWQOARTXG4FDMRO25A, seg_pos={0 0}, upload=463106]: Uploading piece...
Dec 27 15:28:33.715263099 (527h59m33.586346134s) [noise=true , address=145.79.227.36:4517   , piece_id=L4D5QTY7CGBB2YTAFCWKLVG6QGDKPD6ECGULUWLZXD335YBG62MA, seg_pos={0 0}, upload=463106]: Uploading piece...
Dec 27 15:28:33.715685383 (527h59m33.586768418s) [noise=true , address=145.79.227.5:4223    , piece_id=O3PWYVZCOUZ42KSNEHTRORPA2EC62NJQBRUKCEG2O4OEIMH2BBEQ, seg_pos={0 0}, upload=463106]: Uploading piece...
Dec 27 15:28:33.716198070 (527h59m33.587281115s) [noise=true , address=77.90.31.17:4773     , piece_id=OFT7VY5RZ3GZKGSYAP7WC5MC2PB6WHJ6O73MJEM6GEIZIKA6DKNQ, seg_pos={0 0}, upload=463106]: Uploading piece...
Dec 27 15:28:33.716545381 (527h59m33.587628416s) [noise=true , address=145.79.227.19:4078   , piece_id=PUOZVVJQQZGCTVHZQDWUXDGKRZYHYF4KJHKYCHWEDQ7F3OWACSRA, seg_pos={0 0}, upload=463106]: Uploading piece...
Dec 27 15:28:33.716990449 (527h59m33.588073484s) [noise=true , address=145.79.227.17:4971   , piece_id=55JHID46Y3DXBWHCW2CNDARKU3DEREWXMSRDOOGKQ5WA2SNFTAIQ, seg_pos={0 0}, upload=463106]: Uploading piece...
Dec 27 15:28:33.717360553 (527h59m33.588443598s) [noise=true , address=145.79.227.10:4402   , piece_id=GZPVSHJYSS4E7SWV5KR4KKRTL77OFT2HGSJES3BNOB3BVDARA3WA, seg_pos={0 0}, upload=463106]: Uploading piece...
Dec 27 15:28:33.718646041 (527h59m33.589729116s) [noise=true , address=45.91.116.14:4191    , piece_id=JCI7ONZOKM7K5LAMJCNFEKRUYV35UUWGGQV2SKLQXQI477SWWINQ, seg_pos={0 0}, upload=463106]: Uploading piece...
Dec 27 15:28:33.724840997 (527h59m33.595924032s) [noise=true , address=77.90.31.19:4844     , piece_id=F3UA7P2K4SQJ2QSXMHA4YDJ7SUTK3CRXO72HIYOB7ELQFY7TRCKQ, seg_pos={0 0}, upload=463106]: Uploading piece...
Dec 27 15:28:35.385015360 (527h59m35.256098395s) [upload=463106]: Next returned nil segment
Dec 27 15:28:35.385028075 (527h59m35.25611111s) [upload=463106]: Waiting for error group managing segments...
Dec 27 15:28:35.715583074 (527h59m35.586666119s) [noise=true , address=145.79.227.19:4078   , piece_id=PUOZVVJQQZGCTVHZQDWUXDGKRZYHYF4KJHKYCHWEDQ7F3OWACSRA, seg_pos={0 0}, upload=463106]: Done uploading piece. err: <nil>
Dec 27 15:28:35.722558366 (527h59m35.593641401s) [noise=true , address=145.79.227.8:4329    , piece_id=FSACQITU5R4HCBHWWS4CIJLHPP2HTTRK5BYEZUFVOZIVZLTHXJOQ, seg_pos={0 0}, upload=463106]: Done uploading piece. err: <nil>
Dec 27 15:28:35.730441788 (527h59m35.601524833s) [noise=true , address=145.79.227.10:4402   , piece_id=GZPVSHJYSS4E7SWV5KR4KKRTL77OFT2HGSJES3BNOB3BVDARA3WA, seg_pos={0 0}, upload=463106]: Done uploading piece. err: <nil>
Dec 27 15:28:35.732697143 (527h59m35.603780178s) [noise=true , address=145.79.227.5:4223    , piece_id=O3PWYVZCOUZ42KSNEHTRORPA2EC62NJQBRUKCEG2O4OEIMH2BBEQ, seg_pos={0 0}, upload=463106]: Done uploading piece. err: <nil>
Dec 27 15:28:35.734724362 (527h59m35.605807398s) [noise=true , address=145.79.227.36:4517   , piece_id=L4D5QTY7CGBB2YTAFCWKLVG6QGDKPD6ECGULUWLZXD335YBG62MA, seg_pos={0 0}, upload=463106]: Done uploading piece. err: <nil>
Dec 27 15:28:35.738382358 (527h59m35.609465403s) [noise=true , address=145.79.227.47:4265   , piece_id=43C3U3Y4CC5RW47RXAKPCNKL22SKN35QLUB3PLTWLKQ6F7QE5EJQ, seg_pos={0 0}, upload=463106]: Done uploading piece. err: <nil>
Dec 27 15:28:35.742888348 (527h59m35.613971393s) [noise=true , address=145.79.227.48:4302   , piece_id=AEMMBCPNN74CGSXA2PID3PMWRPXYEBHXGTS52DC6W7KT667WBR2Q, seg_pos={0 0}, upload=463106]: Done uploading piece. err: <nil>
Dec 27 15:28:35.744009463 (527h59m35.615092508s) [noise=true , address=145.79.227.3:4151    , piece_id=IWDWU4M6NF4GF4SN3JDZ3GMALCLFFHYXFIZYOYJZEJ7MSRHG3YOA, seg_pos={0 0}, upload=463106]: Done uploading piece. err: <nil>
Dec 27 15:28:35.744961095 (527h59m35.61604413s) [noise=true , address=145.79.227.17:4971   , piece_id=55JHID46Y3DXBWHCW2CNDARKU3DEREWXMSRDOOGKQ5WA2SNFTAIQ, seg_pos={0 0}, upload=463106]: Done uploading piece. err: <nil>
Dec 27 15:28:35.745196794 (527h59m35.616279829s) [noise=true , address=145.79.227.50:4373   , piece_id=GIJMUCTZUIZRVUMLHLKETBCTCVUEZH4GDJGWQOARTXG4FDMRO25A, seg_pos={0 0}, upload=463106]: Done uploading piece. err: <nil>
Dec 27 15:28:36.031177646 (527h59m35.902260681s) [noise=true , address=45.91.116.8:4046     , piece_id=GTZH5HZ3H24H4ROS6ELSUKJBGKEUTKD5HJTVNJ5BRETRYRTXGCNQ, seg_pos={0 0}, upload=463106]: Done uploading piece. err: <nil>
Dec 27 15:28:36.049296227 (527h59m35.920379272s) [noise=true , address=45.91.116.6:4975     , piece_id=TQYSZ7MGYYPBE5JMSWV5R6BQMLQ4HMWFFQ4W54O4E75OAUWRPCGQ, seg_pos={0 0}, upload=463106]: Done uploading piece. err: <nil>
Dec 27 15:28:36.278862745 (527h59m36.14994579s) [noise=true , address=45.91.116.9:4263     , piece_id=RJRCKSNQTSSJP7EDTRLLDBLEOGRV7D4Y3FS5LJUMAAAZRWF4QALQ, seg_pos={0 0}, upload=463106]: Done uploading piece. err: <nil>
Dec 27 15:28:36.278887922 (527h59m36.149970977s) [seg_pos={0 0}, upload=463106]: Segment reached optimal threshold of 13 pieces.
Dec 27 15:28:36.279174007 (527h59m36.150257052s) [noise=true , address=45.91.116.14:4191    , piece_id=JCI7ONZOKM7K5LAMJCNFEKRUYV35UUWGGQV2SKLQXQI477SWWINQ, seg_pos={0 0}, upload=463106]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: write tcp 188.166.225.170:34796->45.91.116.14:4191: use of closed network connection; write tcp 188.166.225.170:34796->45.91.116.14:4191: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34796->45.91.116.14:4191: use of closed network connection
Dec 27 15:28:36.279183275 (527h59m36.15026632s) [noise=true , address=45.91.116.14:4191    , piece_id=JCI7ONZOKM7K5LAMJCNFEKRUYV35UUWGGQV2SKLQXQI477SWWINQ, seg_pos={0 0}, upload=463106]: Long tail cancellation due to optimal threshold
Dec 27 15:28:36.279193824 (527h59m36.15027689s) [noise=true , address=77.90.31.19:4844     , piece_id=F3UA7P2K4SQJ2QSXMHA4YDJ7SUTK3CRXO72HIYOB7ELQFY7TRCKQ, seg_pos={0 0}, upload=463106]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF
Dec 27 15:28:36.279203544 (527h59m36.150286609s) [noise=true , address=77.90.31.19:4844     , piece_id=F3UA7P2K4SQJ2QSXMHA4YDJ7SUTK3CRXO72HIYOB7ELQFY7TRCKQ, seg_pos={0 0}, upload=463106]: Long tail cancellation due to optimal threshold
Dec 27 15:28:36.279429393 (527h59m36.150512438s) [noise=true , address=77.90.31.17:4773     , piece_id=OFT7VY5RZ3GZKGSYAP7WC5MC2PB6WHJ6O73MJEM6GEIZIKA6DKNQ, seg_pos={0 0}, upload=463106]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: context canceled; context canceled
Dec 27 15:28:36.279436827 (527h59m36.150519862s) [noise=true , address=77.90.31.17:4773     , piece_id=OFT7VY5RZ3GZKGSYAP7WC5MC2PB6WHJ6O73MJEM6GEIZIKA6DKNQ, seg_pos={0 0}, upload=463106]: Long tail cancellation due to optimal threshold
Dec 27 15:28:36.279454361 (527h59m36.150537406s) [seg_pos={0 0}, upload=463106]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 15:28:36.279532229 (527h59m36.150615264s) [upload=463106]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 15:28:36.279544372 (527h59m36.150627417s) [upload=463106]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 15:28:36.567793082 (527h59m36.438876117s) [upload=463106]: Done uploading segments...
Dec 27 15:28:36.567811988 (527h59m36.438895023s) [upload=463106]: Upload finished. err: <nil>
Duration: 5.621820553s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
6 out of 60 is failed (threshold=3) [......X..............X....X...................X.....XX......]
duration: 
59.516374168s
trace: 
https://jaeger.dev.storj.io/trace/1cf86c3db9e735d0
FAIL (2025-12-27 15:29:08 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
PBKNYT65NEYNQADUXQG2J3WTCLD343XLSQ, seg_pos={0 0}, upload=463022]: Uploading piece...
Dec 27 15:22:34.632070090 (527h53m34.503153125s) [noise=true , address=77.90.31.11:4565     , piece_id=B4VPI2I53PUUOLNS6C6GFKY3ZYDQ5RRQZCKC4QVLSVJI777RR6KQ, seg_pos={0 0}, upload=463022]: Uploading piece...
Dec 27 15:22:34.632523263 (527h53m34.503606298s) [noise=true , address=77.90.31.10:4540     , piece_id=MVAPVVOZIC5C6BL5S74PXSKTLTCGLIOSN37M33OUYVPD53NG3GLQ, seg_pos={0 0}, upload=463022]: Uploading piece...
Dec 27 15:22:34.632676374 (527h53m34.503759419s) [noise=true , address=77.90.31.13:4659     , piece_id=STKDI6VSN67XV43CB4ASCJTPDU6NA75REYKEE6D5K3EFM5FCDMYA, seg_pos={0 0}, upload=463022]: Uploading piece...
Dec 27 15:22:34.632931610 (527h53m34.504014655s) [noise=true , address=145.79.227.52:4466   , piece_id=CZYMGB53RT6FROGYMOYOPOQJ5CSZDW4YYAVDQWYWVLIDINN3LJVA, seg_pos={0 0}, upload=463022]: Uploading piece...
Dec 27 15:22:34.633357982 (527h53m34.504441017s) [noise=true , address=145.79.227.13:4816   , piece_id=LTSC26UNXCIHK4PDGYKE7GZCQ7MXCWWRGBLIHI6FPO6ISE4BPH7A, seg_pos={0 0}, upload=463022]: Uploading piece...
Dec 27 15:22:34.633490544 (527h53m34.504573579s) [noise=true , address=145.79.227.44:4218   , piece_id=2KHGZL76RA6CMI33PTF5WZN5TRIK3LRL6XT5IJP6LNJP5ZRVMHXA, seg_pos={0 0}, upload=463022]: Uploading piece...
Dec 27 15:22:34.633891278 (527h53m34.504974313s) [noise=true , address=77.90.31.18:4820     , piece_id=TSKDIH6HQUUQRTQTJDXGBX3OALVLYIUKR5QCPJFPFXTCASNN5JMQ, seg_pos={0 0}, upload=463022]: Uploading piece...
Dec 27 15:22:34.634043096 (527h53m34.505126141s) [noise=true , address=145.79.227.17:4997   , piece_id=VX4S5TXLHFNBRE7OYPDZAGYDGN2AD34ZL5JQ2YUSBV656RO6A5UA, seg_pos={0 0}, upload=463022]: Uploading piece...
Dec 27 15:22:34.634325093 (527h53m34.505408128s) [noise=true , address=77.90.31.16:4739     , piece_id=M7IJW7WX2FNPFWNJPBUAHQQOKPFXDSWAWMC2IHUYQZ4JPXF3YZXQ, seg_pos={0 0}, upload=463022]: Uploading piece...
Dec 27 15:22:34.634722230 (527h53m34.505805265s) [noise=true , address=77.90.31.15:4716     , piece_id=MLOLX2R56NCS4KT3FLJJNNPDAMUA6S5NSFCX57W7V6GMIZ4457DA, seg_pos={0 0}, upload=463022]: Uploading piece...
Dec 27 15:22:34.635114116 (527h53m34.506197151s) [noise=true , address=45.91.116.13:4171    , piece_id=YPPWJWZ6H4XMNLGVDHNRQPMZJDXYQRO2EYLNUVCC6WSHIRRJSEWQ, seg_pos={0 0}, upload=463022]: Uploading piece...
Dec 27 15:22:34.635719949 (527h53m34.506802984s) [noise=true , address=77.90.31.12:4618     , piece_id=ZII3XYSUJ7WTNNZA77GO54GFVGVHOGUUJOFZG2OJDTDDP6CAPC3A, seg_pos={0 0}, upload=463022]: Uploading piece...
Dec 27 15:22:34.636184344 (527h53m34.507267379s) [noise=true , address=145.79.227.30:4775   , piece_id=RG6IZXLWJSMKNHY2VR7VHDATQ2LAHIEGYF6WDQIZXUTGETQKS3DQ, seg_pos={0 0}, upload=463022]: Uploading piece...
Dec 27 15:22:36.104663675 (527h53m35.97574671s) [upload=463022]: Next returned nil segment
Dec 27 15:22:36.104675988 (527h53m35.975759023s) [upload=463022]: Waiting for error group managing segments...
Dec 27 15:22:36.505615923 (527h53m36.376698958s) [noise=true , address=145.79.227.17:4997   , piece_id=VX4S5TXLHFNBRE7OYPDZAGYDGN2AD34ZL5JQ2YUSBV656RO6A5UA, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil>
Dec 27 15:22:36.509752460 (527h53m36.380835505s) [noise=true , address=145.79.227.13:4816   , piece_id=LTSC26UNXCIHK4PDGYKE7GZCQ7MXCWWRGBLIHI6FPO6ISE4BPH7A, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil>
Dec 27 15:22:36.517493871 (527h53m36.388576906s) [noise=true , address=145.79.227.14:4865   , piece_id=XOXM2D6IS4SG6DN24DEGIPV7NK5FWDI7QVVJK7L3QB25AUPRNZIQ, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil>
Dec 27 15:22:36.623194766 (527h53m36.494277801s) [noise=true , address=145.79.227.52:4466   , piece_id=CZYMGB53RT6FROGYMOYOPOQJ5CSZDW4YYAVDQWYWVLIDINN3LJVA, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil>
Dec 27 15:22:36.749453706 (527h53m36.620536741s) [noise=true , address=45.91.116.11:4145    , piece_id=77YGIECBFASFKBHIQO333RHATPHZCA7JQW2WPNSGEGM24TTWDVGA, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil>
Dec 27 15:22:36.780705355 (527h53m36.6517884s) [noise=true , address=77.90.31.18:4820     , piece_id=TSKDIH6HQUUQRTQTJDXGBX3OALVLYIUKR5QCPJFPFXTCASNN5JMQ, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil>
Dec 27 15:22:36.803640327 (527h53m36.674723362s) [noise=true , address=77.90.31.13:4659     , piece_id=STKDI6VSN67XV43CB4ASCJTPDU6NA75REYKEE6D5K3EFM5FCDMYA, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil>
Dec 27 15:22:36.841101831 (527h53m36.712184876s) [noise=true , address=77.90.31.16:4739     , piece_id=M7IJW7WX2FNPFWNJPBUAHQQOKPFXDSWAWMC2IHUYQZ4JPXF3YZXQ, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil>
Dec 27 15:22:37.481224175 (527h53m37.35230722s) [noise=true , address=77.90.31.15:4716     , piece_id=MLOLX2R56NCS4KT3FLJJNNPDAMUA6S5NSFCX57W7V6GMIZ4457DA, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil>
Dec 27 15:22:37.504901952 (527h53m37.375984997s) [noise=true , address=145.79.227.30:4775   , piece_id=RG6IZXLWJSMKNHY2VR7VHDATQ2LAHIEGYF6WDQIZXUTGETQKS3DQ, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil>
Dec 27 15:22:37.553631302 (527h53m37.424714337s) [noise=true , address=145.79.227.44:4218   , piece_id=2KHGZL76RA6CMI33PTF5WZN5TRIK3LRL6XT5IJP6LNJP5ZRVMHXA, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil>
Dec 27 15:22:37.596982441 (527h53m37.468065476s) [noise=true , address=145.79.227.40:4063   , piece_id=SRV3KMN3XXUOJA2GLEJ27UKWRANPEKCT3NC7BBDAVMTCI6FZCNPQ, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil>
Dec 27 15:22:37.792107735 (527h53m37.663190771s) [noise=true , address=77.90.31.17:4802     , piece_id=TFQZEPPQN5RYDUNTGLILJUHZ3QLTU7BT3QP67SS5Y4ZRUKPG6A5A, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil>
Dec 27 15:22:37.802639390 (527h53m37.673722425s) [noise=true , address=77.90.31.12:4618     , piece_id=ZII3XYSUJ7WTNNZA77GO54GFVGVHOGUUJOFZG2OJDTDDP6CAPC3A, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil>
Dec 27 15:22:37.802661622 (527h53m37.673744667s) [seg_pos={0 0}, upload=463022]: Segment reached optimal threshold of 14 pieces.
Dec 27 15:22:37.802830423 (527h53m37.673913468s) [noise=true , address=77.90.31.11:4565     , piece_id=B4VPI2I53PUUOLNS6C6GFKY3ZYDQ5RRQZCKC4QVLSVJI777RR6KQ, seg_pos={0 0}, upload=463022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xnwFcUEdmpLjmx8NXZZcpiVQwTySXDRYbchuz4oREBq1yP6wh): protocol: write tcp 188.166.225.170:60558->77.90.31.11:4565: use of closed network connection; write tcp 188.166.225.170:60558->77.90.31.11:4565: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60558->77.90.31.11:4565: use of closed network connection
Dec 27 15:22:37.802837557 (527h53m37.673920592s) [noise=true , address=77.90.31.11:4565     , piece_id=B4VPI2I53PUUOLNS6C6GFKY3ZYDQ5RRQZCKC4QVLSVJI777RR6KQ, seg_pos={0 0}, upload=463022]: Long tail cancellation due to optimal threshold
Dec 27 15:22:37.803060181 (527h53m37.674143216s) [noise=true , address=45.91.116.13:4171    , piece_id=YPPWJWZ6H4XMNLGVDHNRQPMZJDXYQRO2EYLNUVCC6WSHIRRJSEWQ, seg_pos={0 0}, upload=463022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122dmyMaZDYbCvoYvCc9m8Vf5dA8aJiw9qPh1G5AQWKnBig4mVg): protocol: write tcp 188.166.225.170:35780->45.91.116.13:4171: use of closed network connection; write tcp 188.166.225.170:35780->45.91.116.13:4171: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35780->45.91.116.13:4171: use of closed network connection
Dec 27 15:22:37.803068377 (527h53m37.674151422s) [noise=true , address=45.91.116.13:4171    , piece_id=YPPWJWZ6H4XMNLGVDHNRQPMZJDXYQRO2EYLNUVCC6WSHIRRJSEWQ, seg_pos={0 0}, upload=463022]: Long tail cancellation due to optimal threshold
Dec 27 15:22:37.803144622 (527h53m37.674227667s) [noise=true , address=64.49.13.169:4708    , piece_id=J7UIJKZWYB2HZ2TCTMPBKNYT65NEYNQADUXQG2J3WTCLD343XLSQ, seg_pos={0 0}, upload=463022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:51138->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:51138->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51138->64.49.13.169:4708: use of closed network connection
Dec 27 15:22:37.803149080 (527h53m37.674232115s) [noise=true , address=64.49.13.169:4708    , piece_id=J7UIJKZWYB2HZ2TCTMPBKNYT65NEYNQADUXQG2J3WTCLD343XLSQ, seg_pos={0 0}, upload=463022]: Long tail cancellation due to optimal threshold
Dec 27 15:22:37.803258279 (527h53m37.674341314s) [noise=true , address=77.90.31.10:4540     , piece_id=MVAPVVOZIC5C6BL5S74PXSKTLTCGLIOSN37M33OUYVPD53NG3GLQ, seg_pos={0 0}, upload=463022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA): protocol: expected piece hash; context canceled; EOF
Dec 27 15:22:37.803263358 (527h53m37.674346393s) [noise=true , address=77.90.31.10:4540     , piece_id=MVAPVVOZIC5C6BL5S74PXSKTLTCGLIOSN37M33OUYVPD53NG3GLQ, seg_pos={0 0}, upload=463022]: Long tail cancellation due to optimal threshold
Dec 27 15:22:37.803277855 (527h53m37.674360901s) [seg_pos={0 0}, upload=463022]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 15:22:37.803341717 (527h53m37.674424762s) [upload=463022]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 15:22:37.803346236 (527h53m37.674429271s) [upload=463022]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 15:22:38.115241263 (527h53m37.986324298s) [upload=463022]: Done uploading segments...
Dec 27 15:22:38.115265930 (527h53m37.986348975s) [upload=463022]: Upload finished. err: <nil>
Duration: 59.516389066s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
21 out of 60 is failed (threshold=3) [..X..XX.......X..X.X..X...XXXXXXX..X.X..X.X.....X...X......X]
duration: 
30.003581329s
trace: 
https://jaeger.dev.storj.io/trace/7b79ac0553c90089
FAIL (2025-12-27 15:29:02 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: TQYSZ7MGYYPBE5JMSWV5R6BQMLQ4HMWFFQ4W54O4E75OAUWRPCGQ) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: GTZH5HZ3H24H4ROS6ELSUKJBGKEUTKD5HJTVNJ5BRETRYRTXGCNQ) context deadline exceeded
======LOGS======
Dec 27 15:28:33.465851383 (527h59m33.336934418s) [upload=463106]: Starting upload
Dec 27 15:28:33.465907279 (527h59m33.336990324s) [upload=463106]: Done starting upload
Dec 27 15:28:33.466096420 (527h59m33.337179465s) [upload=463106]: Uploading segments...
Dec 27 15:28:33.466134482 (527h59m33.337217527s) [upload=463106]: Got next segment. Inline: false
Dec 27 15:28:33.466163618 (527h59m33.337246653s) [upload=463106]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 15:28:33.466168387 (527h59m33.337251432s) [upload=463106]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 15:28:33.712366784 (527h59m33.583449819s) [seg_pos={0 0}, upload=463106]: Begin upload segment...
Dec 27 15:28:33.712410658 (527h59m33.583493703s) [seg_pos={0 0}, upload=463106]: Done begin upload segment.
Dec 27 15:28:33.712436357 (527h59m33.583519402s) [noise=true , address=145.79.227.3:4151    , piece_id=IWDWU4M6NF4GF4SN3JDZ3GMALCLFFHYXFIZYOYJZEJ7MSRHG3YOA, seg_pos={0 0}, upload=463106]: Uploading piece...
Dec 27 15:28:33.712481533 (527h59m33.583564578s) [noise=true , address=145.79.227.48:4302   , piece_id=AEMMBCPNN74CGSXA2PID3PMWRPXYEBHXGTS52DC6W7KT667WBR2Q, seg_pos={0 0}, upload=463106]: Uploading piece...
Dec 27 15:28:33.712939194 (527h59m33.584022229s) [noise=true , address=145.79.227.47:4265   , piece_id=43C3U3Y4CC5RW47RXAKPCNKL22SKN35QLUB3PLTWLKQ6F7QE5EJQ, seg_pos={0 0}, upload=463106]: Uploading piece...
Dec 27 15:28:33.713315811 (527h59m33.584398856s) [noise=true , address=45.91.116.6:4975     , piece_id=TQYSZ7MGYYPBE5JMSWV5R6BQMLQ4HMWFFQ4W54O4E75OAUWRPCGQ, seg_pos={0 0}, upload=463106]: Uploading piece...
Dec 27 15:28:33.713776077 (527h59m33.584859123s) [noise=true , address=45.91.116.8:4046     , piece_id=GTZH5HZ3H24H4ROS6ELSUKJBGKEUTKD5HJTVNJ5BRETRYRTXGCNQ, seg_pos={0 0}, upload=463106]: Uploading piece...
Dec 27 15:28:33.714231715 (527h59m33.58531476s) [noise=true , address=45.91.116.9:4263     , piece_id=RJRCKSNQTSSJP7EDTRLLDBLEOGRV7D4Y3FS5LJUMAAAZRWF4QALQ, seg_pos={0 0}, upload=463106]: Uploading piece...
Dec 27 15:28:33.714685930 (527h59m33.585768965s) [noise=true , address=145.79.227.8:4329    , piece_id=FSACQITU5R4HCBHWWS4CIJLHPP2HTTRK5BYEZUFVOZIVZLTHXJOQ, seg_pos={0 0}, upload=463106]: Uploading piece...
Dec 27 15:28:33.715251548 (527h59m33.586334593s) [noise=true , address=145.79.227.50:4373   , piece_id=GIJMUCTZUIZRVUMLHLKETBCTCVUEZH4GDJGWQOARTXG4FDMRO25A, seg_pos={0 0}, upload=463106]: Uploading piece...
Dec 27 15:28:33.715263099 (527h59m33.586346134s) [noise=true , address=145.79.227.36:4517   , piece_id=L4D5QTY7CGBB2YTAFCWKLVG6QGDKPD6ECGULUWLZXD335YBG62MA, seg_pos={0 0}, upload=463106]: Uploading piece...
Dec 27 15:28:33.715685383 (527h59m33.586768418s) [noise=true , address=145.79.227.5:4223    , piece_id=O3PWYVZCOUZ42KSNEHTRORPA2EC62NJQBRUKCEG2O4OEIMH2BBEQ, seg_pos={0 0}, upload=463106]: Uploading piece...
Dec 27 15:28:33.716198070 (527h59m33.587281115s) [noise=true , address=77.90.31.17:4773     , piece_id=OFT7VY5RZ3GZKGSYAP7WC5MC2PB6WHJ6O73MJEM6GEIZIKA6DKNQ, seg_pos={0 0}, upload=463106]: Uploading piece...
Dec 27 15:28:33.716545381 (527h59m33.587628416s) [noise=true , address=145.79.227.19:4078   , piece_id=PUOZVVJQQZGCTVHZQDWUXDGKRZYHYF4KJHKYCHWEDQ7F3OWACSRA, seg_pos={0 0}, upload=463106]: Uploading piece...
Dec 27 15:28:33.716990449 (527h59m33.588073484s) [noise=true , address=145.79.227.17:4971   , piece_id=55JHID46Y3DXBWHCW2CNDARKU3DEREWXMSRDOOGKQ5WA2SNFTAIQ, seg_pos={0 0}, upload=463106]: Uploading piece...
Dec 27 15:28:33.717360553 (527h59m33.588443598s) [noise=true , address=145.79.227.10:4402   , piece_id=GZPVSHJYSS4E7SWV5KR4KKRTL77OFT2HGSJES3BNOB3BVDARA3WA, seg_pos={0 0}, upload=463106]: Uploading piece...
Dec 27 15:28:33.718646041 (527h59m33.589729116s) [noise=true , address=45.91.116.14:4191    , piece_id=JCI7ONZOKM7K5LAMJCNFEKRUYV35UUWGGQV2SKLQXQI477SWWINQ, seg_pos={0 0}, upload=463106]: Uploading piece...
Dec 27 15:28:33.724840997 (527h59m33.595924032s) [noise=true , address=77.90.31.19:4844     , piece_id=F3UA7P2K4SQJ2QSXMHA4YDJ7SUTK3CRXO72HIYOB7ELQFY7TRCKQ, seg_pos={0 0}, upload=463106]: Uploading piece...
Dec 27 15:28:35.385015360 (527h59m35.256098395s) [upload=463106]: Next returned nil segment
Dec 27 15:28:35.385028075 (527h59m35.25611111s) [upload=463106]: Waiting for error group managing segments...
Dec 27 15:28:35.715583074 (527h59m35.586666119s) [noise=true , address=145.79.227.19:4078   , piece_id=PUOZVVJQQZGCTVHZQDWUXDGKRZYHYF4KJHKYCHWEDQ7F3OWACSRA, seg_pos={0 0}, upload=463106]: Done uploading piece. err: <nil>
Dec 27 15:28:35.722558366 (527h59m35.593641401s) [noise=true , address=145.79.227.8:4329    , piece_id=FSACQITU5R4HCBHWWS4CIJLHPP2HTTRK5BYEZUFVOZIVZLTHXJOQ, seg_pos={0 0}, upload=463106]: Done uploading piece. err: <nil>
Dec 27 15:28:35.730441788 (527h59m35.601524833s) [noise=true , address=145.79.227.10:4402   , piece_id=GZPVSHJYSS4E7SWV5KR4KKRTL77OFT2HGSJES3BNOB3BVDARA3WA, seg_pos={0 0}, upload=463106]: Done uploading piece. err: <nil>
Dec 27 15:28:35.732697143 (527h59m35.603780178s) [noise=true , address=145.79.227.5:4223    , piece_id=O3PWYVZCOUZ42KSNEHTRORPA2EC62NJQBRUKCEG2O4OEIMH2BBEQ, seg_pos={0 0}, upload=463106]: Done uploading piece. err: <nil>
Dec 27 15:28:35.734724362 (527h59m35.605807398s) [noise=true , address=145.79.227.36:4517   , piece_id=L4D5QTY7CGBB2YTAFCWKLVG6QGDKPD6ECGULUWLZXD335YBG62MA, seg_pos={0 0}, upload=463106]: Done uploading piece. err: <nil>
Dec 27 15:28:35.738382358 (527h59m35.609465403s) [noise=true , address=145.79.227.47:4265   , piece_id=43C3U3Y4CC5RW47RXAKPCNKL22SKN35QLUB3PLTWLKQ6F7QE5EJQ, seg_pos={0 0}, upload=463106]: Done uploading piece. err: <nil>
Dec 27 15:28:35.742888348 (527h59m35.613971393s) [noise=true , address=145.79.227.48:4302   , piece_id=AEMMBCPNN74CGSXA2PID3PMWRPXYEBHXGTS52DC6W7KT667WBR2Q, seg_pos={0 0}, upload=463106]: Done uploading piece. err: <nil>
Dec 27 15:28:35.744009463 (527h59m35.615092508s) [noise=true , address=145.79.227.3:4151    , piece_id=IWDWU4M6NF4GF4SN3JDZ3GMALCLFFHYXFIZYOYJZEJ7MSRHG3YOA, seg_pos={0 0}, upload=463106]: Done uploading piece. err: <nil>
Dec 27 15:28:35.744961095 (527h59m35.61604413s) [noise=true , address=145.79.227.17:4971   , piece_id=55JHID46Y3DXBWHCW2CNDARKU3DEREWXMSRDOOGKQ5WA2SNFTAIQ, seg_pos={0 0}, upload=463106]: Done uploading piece. err: <nil>
Dec 27 15:28:35.745196794 (527h59m35.616279829s) [noise=true , address=145.79.227.50:4373   , piece_id=GIJMUCTZUIZRVUMLHLKETBCTCVUEZH4GDJGWQOARTXG4FDMRO25A, seg_pos={0 0}, upload=463106]: Done uploading piece. err: <nil>
Dec 27 15:28:36.031177646 (527h59m35.902260681s) [noise=true , address=45.91.116.8:4046     , piece_id=GTZH5HZ3H24H4ROS6ELSUKJBGKEUTKD5HJTVNJ5BRETRYRTXGCNQ, seg_pos={0 0}, upload=463106]: Done uploading piece. err: <nil>
Dec 27 15:28:36.049296227 (527h59m35.920379272s) [noise=true , address=45.91.116.6:4975     , piece_id=TQYSZ7MGYYPBE5JMSWV5R6BQMLQ4HMWFFQ4W54O4E75OAUWRPCGQ, seg_pos={0 0}, upload=463106]: Done uploading piece. err: <nil>
Dec 27 15:28:36.278862745 (527h59m36.14994579s) [noise=true , address=45.91.116.9:4263     , piece_id=RJRCKSNQTSSJP7EDTRLLDBLEOGRV7D4Y3FS5LJUMAAAZRWF4QALQ, seg_pos={0 0}, upload=463106]: Done uploading piece. err: <nil>
Dec 27 15:28:36.278887922 (527h59m36.149970977s) [seg_pos={0 0}, upload=463106]: Segment reached optimal threshold of 13 pieces.
Dec 27 15:28:36.279174007 (527h59m36.150257052s) [noise=true , address=45.91.116.14:4191    , piece_id=JCI7ONZOKM7K5LAMJCNFEKRUYV35UUWGGQV2SKLQXQI477SWWINQ, seg_pos={0 0}, upload=463106]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: write tcp 188.166.225.170:34796->45.91.116.14:4191: use of closed network connection; write tcp 188.166.225.170:34796->45.91.116.14:4191: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34796->45.91.116.14:4191: use of closed network connection
Dec 27 15:28:36.279183275 (527h59m36.15026632s) [noise=true , address=45.91.116.14:4191    , piece_id=JCI7ONZOKM7K5LAMJCNFEKRUYV35UUWGGQV2SKLQXQI477SWWINQ, seg_pos={0 0}, upload=463106]: Long tail cancellation due to optimal threshold
Dec 27 15:28:36.279193824 (527h59m36.15027689s) [noise=true , address=77.90.31.19:4844     , piece_id=F3UA7P2K4SQJ2QSXMHA4YDJ7SUTK3CRXO72HIYOB7ELQFY7TRCKQ, seg_pos={0 0}, upload=463106]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF
Dec 27 15:28:36.279203544 (527h59m36.150286609s) [noise=true , address=77.90.31.19:4844     , piece_id=F3UA7P2K4SQJ2QSXMHA4YDJ7SUTK3CRXO72HIYOB7ELQFY7TRCKQ, seg_pos={0 0}, upload=463106]: Long tail cancellation due to optimal threshold
Dec 27 15:28:36.279429393 (527h59m36.150512438s) [noise=true , address=77.90.31.17:4773     , piece_id=OFT7VY5RZ3GZKGSYAP7WC5MC2PB6WHJ6O73MJEM6GEIZIKA6DKNQ, seg_pos={0 0}, upload=463106]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: context canceled; context canceled
Dec 27 15:28:36.279436827 (527h59m36.150519862s) [noise=true , address=77.90.31.17:4773     , piece_id=OFT7VY5RZ3GZKGSYAP7WC5MC2PB6WHJ6O73MJEM6GEIZIKA6DKNQ, seg_pos={0 0}, upload=463106]: Long tail cancellation due to optimal threshold
Dec 27 15:28:36.279454361 (527h59m36.150537406s) [seg_pos={0 0}, upload=463106]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 15:28:36.279532229 (527h59m36.150615264s) [upload=463106]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 15:28:36.279544372 (527h59m36.150627417s) [upload=463106]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 15:28:36.567793082 (527h59m36.438876117s) [upload=463106]: Done uploading segments...
Dec 27 15:28:36.567811988 (527h59m36.438895023s) [upload=463106]: Upload finished. err: <nil>
Duration: 30.003608111s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
6 out of 60 is failed (threshold=3) [.......X..............X....X...................X.....XX.....]
duration: 
52.466869968s
trace: 
https://jaeger.dev.storj.io/trace/1cf86c3db9e735d0
FAIL (2025-12-27 15:28:08 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
PBKNYT65NEYNQADUXQG2J3WTCLD343XLSQ, seg_pos={0 0}, upload=463022]: Uploading piece...
Dec 27 15:22:34.632070090 (527h53m34.503153125s) [noise=true , address=77.90.31.11:4565     , piece_id=B4VPI2I53PUUOLNS6C6GFKY3ZYDQ5RRQZCKC4QVLSVJI777RR6KQ, seg_pos={0 0}, upload=463022]: Uploading piece...
Dec 27 15:22:34.632523263 (527h53m34.503606298s) [noise=true , address=77.90.31.10:4540     , piece_id=MVAPVVOZIC5C6BL5S74PXSKTLTCGLIOSN37M33OUYVPD53NG3GLQ, seg_pos={0 0}, upload=463022]: Uploading piece...
Dec 27 15:22:34.632676374 (527h53m34.503759419s) [noise=true , address=77.90.31.13:4659     , piece_id=STKDI6VSN67XV43CB4ASCJTPDU6NA75REYKEE6D5K3EFM5FCDMYA, seg_pos={0 0}, upload=463022]: Uploading piece...
Dec 27 15:22:34.632931610 (527h53m34.504014655s) [noise=true , address=145.79.227.52:4466   , piece_id=CZYMGB53RT6FROGYMOYOPOQJ5CSZDW4YYAVDQWYWVLIDINN3LJVA, seg_pos={0 0}, upload=463022]: Uploading piece...
Dec 27 15:22:34.633357982 (527h53m34.504441017s) [noise=true , address=145.79.227.13:4816   , piece_id=LTSC26UNXCIHK4PDGYKE7GZCQ7MXCWWRGBLIHI6FPO6ISE4BPH7A, seg_pos={0 0}, upload=463022]: Uploading piece...
Dec 27 15:22:34.633490544 (527h53m34.504573579s) [noise=true , address=145.79.227.44:4218   , piece_id=2KHGZL76RA6CMI33PTF5WZN5TRIK3LRL6XT5IJP6LNJP5ZRVMHXA, seg_pos={0 0}, upload=463022]: Uploading piece...
Dec 27 15:22:34.633891278 (527h53m34.504974313s) [noise=true , address=77.90.31.18:4820     , piece_id=TSKDIH6HQUUQRTQTJDXGBX3OALVLYIUKR5QCPJFPFXTCASNN5JMQ, seg_pos={0 0}, upload=463022]: Uploading piece...
Dec 27 15:22:34.634043096 (527h53m34.505126141s) [noise=true , address=145.79.227.17:4997   , piece_id=VX4S5TXLHFNBRE7OYPDZAGYDGN2AD34ZL5JQ2YUSBV656RO6A5UA, seg_pos={0 0}, upload=463022]: Uploading piece...
Dec 27 15:22:34.634325093 (527h53m34.505408128s) [noise=true , address=77.90.31.16:4739     , piece_id=M7IJW7WX2FNPFWNJPBUAHQQOKPFXDSWAWMC2IHUYQZ4JPXF3YZXQ, seg_pos={0 0}, upload=463022]: Uploading piece...
Dec 27 15:22:34.634722230 (527h53m34.505805265s) [noise=true , address=77.90.31.15:4716     , piece_id=MLOLX2R56NCS4KT3FLJJNNPDAMUA6S5NSFCX57W7V6GMIZ4457DA, seg_pos={0 0}, upload=463022]: Uploading piece...
Dec 27 15:22:34.635114116 (527h53m34.506197151s) [noise=true , address=45.91.116.13:4171    , piece_id=YPPWJWZ6H4XMNLGVDHNRQPMZJDXYQRO2EYLNUVCC6WSHIRRJSEWQ, seg_pos={0 0}, upload=463022]: Uploading piece...
Dec 27 15:22:34.635719949 (527h53m34.506802984s) [noise=true , address=77.90.31.12:4618     , piece_id=ZII3XYSUJ7WTNNZA77GO54GFVGVHOGUUJOFZG2OJDTDDP6CAPC3A, seg_pos={0 0}, upload=463022]: Uploading piece...
Dec 27 15:22:34.636184344 (527h53m34.507267379s) [noise=true , address=145.79.227.30:4775   , piece_id=RG6IZXLWJSMKNHY2VR7VHDATQ2LAHIEGYF6WDQIZXUTGETQKS3DQ, seg_pos={0 0}, upload=463022]: Uploading piece...
Dec 27 15:22:36.104663675 (527h53m35.97574671s) [upload=463022]: Next returned nil segment
Dec 27 15:22:36.104675988 (527h53m35.975759023s) [upload=463022]: Waiting for error group managing segments...
Dec 27 15:22:36.505615923 (527h53m36.376698958s) [noise=true , address=145.79.227.17:4997   , piece_id=VX4S5TXLHFNBRE7OYPDZAGYDGN2AD34ZL5JQ2YUSBV656RO6A5UA, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil>
Dec 27 15:22:36.509752460 (527h53m36.380835505s) [noise=true , address=145.79.227.13:4816   , piece_id=LTSC26UNXCIHK4PDGYKE7GZCQ7MXCWWRGBLIHI6FPO6ISE4BPH7A, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil>
Dec 27 15:22:36.517493871 (527h53m36.388576906s) [noise=true , address=145.79.227.14:4865   , piece_id=XOXM2D6IS4SG6DN24DEGIPV7NK5FWDI7QVVJK7L3QB25AUPRNZIQ, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil>
Dec 27 15:22:36.623194766 (527h53m36.494277801s) [noise=true , address=145.79.227.52:4466   , piece_id=CZYMGB53RT6FROGYMOYOPOQJ5CSZDW4YYAVDQWYWVLIDINN3LJVA, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil>
Dec 27 15:22:36.749453706 (527h53m36.620536741s) [noise=true , address=45.91.116.11:4145    , piece_id=77YGIECBFASFKBHIQO333RHATPHZCA7JQW2WPNSGEGM24TTWDVGA, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil>
Dec 27 15:22:36.780705355 (527h53m36.6517884s) [noise=true , address=77.90.31.18:4820     , piece_id=TSKDIH6HQUUQRTQTJDXGBX3OALVLYIUKR5QCPJFPFXTCASNN5JMQ, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil>
Dec 27 15:22:36.803640327 (527h53m36.674723362s) [noise=true , address=77.90.31.13:4659     , piece_id=STKDI6VSN67XV43CB4ASCJTPDU6NA75REYKEE6D5K3EFM5FCDMYA, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil>
Dec 27 15:22:36.841101831 (527h53m36.712184876s) [noise=true , address=77.90.31.16:4739     , piece_id=M7IJW7WX2FNPFWNJPBUAHQQOKPFXDSWAWMC2IHUYQZ4JPXF3YZXQ, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil>
Dec 27 15:22:37.481224175 (527h53m37.35230722s) [noise=true , address=77.90.31.15:4716     , piece_id=MLOLX2R56NCS4KT3FLJJNNPDAMUA6S5NSFCX57W7V6GMIZ4457DA, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil>
Dec 27 15:22:37.504901952 (527h53m37.375984997s) [noise=true , address=145.79.227.30:4775   , piece_id=RG6IZXLWJSMKNHY2VR7VHDATQ2LAHIEGYF6WDQIZXUTGETQKS3DQ, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil>
Dec 27 15:22:37.553631302 (527h53m37.424714337s) [noise=true , address=145.79.227.44:4218   , piece_id=2KHGZL76RA6CMI33PTF5WZN5TRIK3LRL6XT5IJP6LNJP5ZRVMHXA, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil>
Dec 27 15:22:37.596982441 (527h53m37.468065476s) [noise=true , address=145.79.227.40:4063   , piece_id=SRV3KMN3XXUOJA2GLEJ27UKWRANPEKCT3NC7BBDAVMTCI6FZCNPQ, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil>
Dec 27 15:22:37.792107735 (527h53m37.663190771s) [noise=true , address=77.90.31.17:4802     , piece_id=TFQZEPPQN5RYDUNTGLILJUHZ3QLTU7BT3QP67SS5Y4ZRUKPG6A5A, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil>
Dec 27 15:22:37.802639390 (527h53m37.673722425s) [noise=true , address=77.90.31.12:4618     , piece_id=ZII3XYSUJ7WTNNZA77GO54GFVGVHOGUUJOFZG2OJDTDDP6CAPC3A, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil>
Dec 27 15:22:37.802661622 (527h53m37.673744667s) [seg_pos={0 0}, upload=463022]: Segment reached optimal threshold of 14 pieces.
Dec 27 15:22:37.802830423 (527h53m37.673913468s) [noise=true , address=77.90.31.11:4565     , piece_id=B4VPI2I53PUUOLNS6C6GFKY3ZYDQ5RRQZCKC4QVLSVJI777RR6KQ, seg_pos={0 0}, upload=463022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xnwFcUEdmpLjmx8NXZZcpiVQwTySXDRYbchuz4oREBq1yP6wh): protocol: write tcp 188.166.225.170:60558->77.90.31.11:4565: use of closed network connection; write tcp 188.166.225.170:60558->77.90.31.11:4565: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60558->77.90.31.11:4565: use of closed network connection
Dec 27 15:22:37.802837557 (527h53m37.673920592s) [noise=true , address=77.90.31.11:4565     , piece_id=B4VPI2I53PUUOLNS6C6GFKY3ZYDQ5RRQZCKC4QVLSVJI777RR6KQ, seg_pos={0 0}, upload=463022]: Long tail cancellation due to optimal threshold
Dec 27 15:22:37.803060181 (527h53m37.674143216s) [noise=true , address=45.91.116.13:4171    , piece_id=YPPWJWZ6H4XMNLGVDHNRQPMZJDXYQRO2EYLNUVCC6WSHIRRJSEWQ, seg_pos={0 0}, upload=463022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122dmyMaZDYbCvoYvCc9m8Vf5dA8aJiw9qPh1G5AQWKnBig4mVg): protocol: write tcp 188.166.225.170:35780->45.91.116.13:4171: use of closed network connection; write tcp 188.166.225.170:35780->45.91.116.13:4171: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35780->45.91.116.13:4171: use of closed network connection
Dec 27 15:22:37.803068377 (527h53m37.674151422s) [noise=true , address=45.91.116.13:4171    , piece_id=YPPWJWZ6H4XMNLGVDHNRQPMZJDXYQRO2EYLNUVCC6WSHIRRJSEWQ, seg_pos={0 0}, upload=463022]: Long tail cancellation due to optimal threshold
Dec 27 15:22:37.803144622 (527h53m37.674227667s) [noise=true , address=64.49.13.169:4708    , piece_id=J7UIJKZWYB2HZ2TCTMPBKNYT65NEYNQADUXQG2J3WTCLD343XLSQ, seg_pos={0 0}, upload=463022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:51138->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:51138->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51138->64.49.13.169:4708: use of closed network connection
Dec 27 15:22:37.803149080 (527h53m37.674232115s) [noise=true , address=64.49.13.169:4708    , piece_id=J7UIJKZWYB2HZ2TCTMPBKNYT65NEYNQADUXQG2J3WTCLD343XLSQ, seg_pos={0 0}, upload=463022]: Long tail cancellation due to optimal threshold
Dec 27 15:22:37.803258279 (527h53m37.674341314s) [noise=true , address=77.90.31.10:4540     , piece_id=MVAPVVOZIC5C6BL5S74PXSKTLTCGLIOSN37M33OUYVPD53NG3GLQ, seg_pos={0 0}, upload=463022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA): protocol: expected piece hash; context canceled; EOF
Dec 27 15:22:37.803263358 (527h53m37.674346393s) [noise=true , address=77.90.31.10:4540     , piece_id=MVAPVVOZIC5C6BL5S74PXSKTLTCGLIOSN37M33OUYVPD53NG3GLQ, seg_pos={0 0}, upload=463022]: Long tail cancellation due to optimal threshold
Dec 27 15:22:37.803277855 (527h53m37.674360901s) [seg_pos={0 0}, upload=463022]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 15:22:37.803341717 (527h53m37.674424762s) [upload=463022]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 15:22:37.803346236 (527h53m37.674429271s) [upload=463022]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 15:22:38.115241263 (527h53m37.986324298s) [upload=463022]: Done uploading segments...
Dec 27 15:22:38.115265930 (527h53m37.986348975s) [upload=463022]: Upload finished. err: <nil>
Duration: 52.466892493s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
33 out of 60 is failed (threshold=6) [.....XXX..X..XX.XXXX.....XXX.X.XXXX.X.XXXX...XXXX.XXXXX...X.]
duration: 
3.475193782s
trace: 
https://jaeger.dev.storj.io/trace/5973653b14c3bbcb
FAIL (2025-12-27 15:27:58 UTC) 
orbiter warning: context deadline exceeded
Duration: 3.475220453s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
34 out of 60 is failed (threshold=6) [X.....XXX..X..XX.XXXX.....XXX.X.XXXX.X.XXXX...XXXX.XXXXX...X]
duration: 
5.001012527s
trace: 
https://jaeger.dev.storj.io/trace/5973653b14c3bbcb
FAIL (2025-12-27 15:27:39 UTC) 
orbiter warning: context deadline exceeded
Duration: 5.001036303s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
20 out of 60 is failed (threshold=3) [...X..XX.......X..X.X..X...XXXXXXX..X.X..X.X.....X...X......]
duration: 
5.963379424s
trace: 
https://jaeger.dev.storj.io/trace/3382203165a15a75
FAIL (2025-12-27 15:27:25 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: EFNF7USD6GGEH3V4T42NKM2JPEA3OCYTQOXTNIVRDA6COXPG5JTA) context deadline exceeded; piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: J3OER7CS2XJ6NVGM4X5XM6DWCWP2LZBOSVWD2CQNXABR4TEJXTIA) context deadline exceeded; piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: DBGSITGU4SZAAYFK5KDZ2MXONZ27XTNAV3OPQCQAUHKYBQNPNXSA) context deadline exceeded
======LOGS======
27 15:22:47.299628936 (527h53m47.170711971s) [upload=463024]: Starting upload
Dec 27 15:22:47.299696604 (527h53m47.17077964s) [upload=463024]: Done starting upload
Dec 27 15:22:47.299787547 (527h53m47.170870622s) [upload=463024]: Uploading segments...
Dec 27 15:22:47.299913407 (527h53m47.170996442s) [upload=463024]: Got next segment. Inline: false
Dec 27 15:22:47.299953483 (527h53m47.171036518s) [upload=463024]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 15:22:47.299958212 (527h53m47.171041257s) [upload=463024]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 15:22:47.545973752 (527h53m47.417056787s) [seg_pos={0 0}, upload=463024]: Begin upload segment...
Dec 27 15:22:47.546024249 (527h53m47.417107284s) [seg_pos={0 0}, upload=463024]: Done begin upload segment.
Dec 27 15:22:47.546057682 (527h53m47.417140717s) [noise=true , address=145.79.227.3:4151    , piece_id=627M5Y34ADXGUIAOOGMTG5PAEEZWBT3PAUOZHPI5M23TFTAV2OWQ, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:47.546065778 (527h53m47.417148813s) [noise=true , address=145.79.227.2:4113    , piece_id=YPFORZR5QCXBPQEQZKUVOIUVM4NYKOLU4KDYA7HRJDVMM3KXLKUQ, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:47.546207497 (527h53m47.417290542s) [noise=true , address=145.79.227.24:4581   , piece_id=7HS6Q5Q3ILUXV6BML37VQY2JS4P77CBO6ZTSENMY56MYOYJW6ANQ, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:47.546489023 (527h53m47.417572068s) [noise=true , address=145.79.227.29:4941   , piece_id=VHMADVXOXZOTYAVGE52WNY2IJWQMSQ7LNRSNXOJEZTFBYWLKIRIA, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:47.546754940 (527h53m47.417837975s) [noise=true , address=145.79.227.42:4121   , piece_id=6UQBDGBP6MIDTPV7RYC5NE7JOW6DP74P5QK3BBHVK4MV5UCKGZNA, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:47.547012460 (527h53m47.418095495s) [noise=true , address=145.79.227.47:4265   , piece_id=44SEXUFB2LZHONSA7GDQ63JOYEYETU5ZUMWF45SKXAE5AP7KVWOA, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:47.547340776 (527h53m47.418423811s) [noise=true , address=45.91.116.14:4192    , piece_id=JRQVETWQDYFMV3OR3BOK54FNOPFAZFQJH73WAXLM4KUDVKMKNF2Q, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:47.547610368 (527h53m47.418693413s) [noise=true , address=145.79.227.7:4295    , piece_id=4YVVZDHQNWCCA6XFV7A7ACJL7TUKPFPKKESO5WBIC7XEIDNULCWA, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:47.547903517 (527h53m47.418986552s) [noise=true , address=45.91.116.13:4154    , piece_id=EFNF7USD6GGEH3V4T42NKM2JPEA3OCYTQOXTNIVRDA6COXPG5JTA, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:47.548994664 (527h53m47.4200777s) [noise=true , address=145.79.227.50:4374   , piece_id=PTUKPJS4VWMT7CWQZIFOMI66JNJS6D2J3OMDYUBF4IL6LXMP465Q, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:47.550097284 (527h53m47.421180319s) [noise=true , address=145.79.227.30:4761   , piece_id=AGS3KQJG5YMSHYQA2C4UQRCYYW62QARBO4I2QWCOEBN445HMYQGQ, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:47.551125933 (527h53m47.422208968s) [noise=true , address=45.91.116.11:4119    , piece_id=J3OER7CS2XJ6NVGM4X5XM6DWCWP2LZBOSVWD2CQNXABR4TEJXTIA, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:47.552145084 (527h53m47.423228119s) [noise=true , address=145.79.227.19:4078   , piece_id=YZCJB4ZJN6BNWDDPOTGHEWQNRD5BCGZZACXVDF6XOX4FOLGZGDBQ, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:47.553167881 (527h53m47.424250916s) [noise=true , address=45.91.116.6:4976     , piece_id=ZHQFB66SB4X4AVSYKPFNDZMWURFQRVZT6QG7QC23JCRW22443M7A, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:47.554213733 (527h53m47.425296768s) [noise=true , address=45.91.116.12:4228    , piece_id=DBGSITGU4SZAAYFK5KDZ2MXONZ27XTNAV3OPQCQAUHKYBQNPNXSA, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:47.555339717 (527h53m47.426422752s) [noise=true , address=145.79.227.38:4977   , piece_id=UVSIVE7ZZECB74JPH7RFRAI74SX5HKKQK2ZHDRQ4K5X7JUZOWKWA, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:49.174094752 (527h53m49.045177787s) [upload=463024]: Next returned nil segment
Dec 27 15:22:49.174108467 (527h53m49.045191512s) [upload=463024]: Waiting for error group managing segments...
Dec 27 15:22:49.550005476 (527h53m49.421088511s) [noise=true , address=145.79.227.29:4941   , piece_id=VHMADVXOXZOTYAVGE52WNY2IJWQMSQ7LNRSNXOJEZTFBYWLKIRIA, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil>
Dec 27 15:22:49.551387929 (527h53m49.422470964s) [noise=true , address=145.79.227.7:4295    , piece_id=4YVVZDHQNWCCA6XFV7A7ACJL7TUKPFPKKESO5WBIC7XEIDNULCWA, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil>
Dec 27 15:22:49.554614222 (527h53m49.425697267s) [noise=true , address=145.79.227.19:4078   , piece_id=YZCJB4ZJN6BNWDDPOTGHEWQNRD5BCGZZACXVDF6XOX4FOLGZGDBQ, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil>
Dec 27 15:22:49.556863425 (527h53m49.42794646s) [noise=true , address=145.79.227.42:4121   , piece_id=6UQBDGBP6MIDTPV7RYC5NE7JOW6DP74P5QK3BBHVK4MV5UCKGZNA, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil>
Dec 27 15:22:49.557246234 (527h53m49.428329279s) [noise=true , address=145.79.227.38:4977   , piece_id=UVSIVE7ZZECB74JPH7RFRAI74SX5HKKQK2ZHDRQ4K5X7JUZOWKWA, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil>
Dec 27 15:22:49.558682599 (527h53m49.429765644s) [noise=true , address=145.79.227.2:4113    , piece_id=YPFORZR5QCXBPQEQZKUVOIUVM4NYKOLU4KDYA7HRJDVMM3KXLKUQ, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil>
Dec 27 15:22:49.564448258 (527h53m49.435531303s) [noise=true , address=145.79.227.3:4151    , piece_id=627M5Y34ADXGUIAOOGMTG5PAEEZWBT3PAUOZHPI5M23TFTAV2OWQ, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil>
Dec 27 15:22:49.566142284 (527h53m49.437225329s) [noise=true , address=145.79.227.50:4374   , piece_id=PTUKPJS4VWMT7CWQZIFOMI66JNJS6D2J3OMDYUBF4IL6LXMP465Q, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil>
Dec 27 15:22:49.574698126 (527h53m49.445781171s) [noise=true , address=145.79.227.24:4581   , piece_id=7HS6Q5Q3ILUXV6BML37VQY2JS4P77CBO6ZTSENMY56MYOYJW6ANQ, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil>
Dec 27 15:22:49.586061825 (527h53m49.45714487s) [noise=true , address=145.79.227.47:4265   , piece_id=44SEXUFB2LZHONSA7GDQ63JOYEYETU5ZUMWF45SKXAE5AP7KVWOA, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil>
Dec 27 15:22:49.793614724 (527h53m49.664697759s) [noise=true , address=45.91.116.11:4119    , piece_id=J3OER7CS2XJ6NVGM4X5XM6DWCWP2LZBOSVWD2CQNXABR4TEJXTIA, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil>
Dec 27 15:22:49.874495066 (527h53m49.745578101s) [noise=true , address=45.91.116.12:4228    , piece_id=DBGSITGU4SZAAYFK5KDZ2MXONZ27XTNAV3OPQCQAUHKYBQNPNXSA, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil>
Dec 27 15:22:49.950055185 (527h53m49.821138231s) [noise=true , address=45.91.116.13:4154    , piece_id=EFNF7USD6GGEH3V4T42NKM2JPEA3OCYTQOXTNIVRDA6COXPG5JTA, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil>
Dec 27 15:22:49.950079642 (527h53m49.821162677s) [seg_pos={0 0}, upload=463024]: Segment reached optimal threshold of 13 pieces.
Dec 27 15:22:49.950443906 (527h53m49.821526952s) [noise=true , address=145.79.227.30:4761   , piece_id=AGS3KQJG5YMSHYQA2C4UQRCYYW62QARBO4I2QWCOEBN445HMYQGQ, seg_pos={0 0}, upload=463024]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF
Dec 27 15:22:49.950463724 (527h53m49.821546759s) [noise=true , address=145.79.227.30:4761   , piece_id=AGS3KQJG5YMSHYQA2C4UQRCYYW62QARBO4I2QWCOEBN445HMYQGQ, seg_pos={0 0}, upload=463024]: Long tail cancellation due to optimal threshold
Dec 27 15:22:49.950484183 (527h53m49.821567228s) [noise=true , address=45.91.116.14:4192    , piece_id=JRQVETWQDYFMV3OR3BOK54FNOPFAZFQJH73WAXLM4KUDVKMKNF2Q, seg_pos={0 0}, upload=463024]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: write tcp 188.166.225.170:49708->45.91.116.14:4192: use of closed network connection; write tcp 188.166.225.170:49708->45.91.116.14:4192: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49708->45.91.116.14:4192: use of closed network connection
Dec 27 15:22:49.950523267 (527h53m49.821606302s) [noise=true , address=45.91.116.14:4192    , piece_id=JRQVETWQDYFMV3OR3BOK54FNOPFAZFQJH73WAXLM4KUDVKMKNF2Q, seg_pos={0 0}, upload=463024]: Long tail cancellation due to optimal threshold
Dec 27 15:22:49.950900406 (527h53m49.821983451s) [noise=true , address=45.91.116.6:4976     , piece_id=ZHQFB66SB4X4AVSYKPFNDZMWURFQRVZT6QG7QC23JCRW22443M7A, seg_pos={0 0}, upload=463024]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF
Dec 27 15:22:49.950911727 (527h53m49.821994772s) [noise=true , address=45.91.116.6:4976     , piece_id=ZHQFB66SB4X4AVSYKPFNDZMWURFQRVZT6QG7QC23JCRW22443M7A, seg_pos={0 0}, upload=463024]: Long tail cancellation due to optimal threshold
Dec 27 15:22:49.950943928 (527h53m49.822026993s) [seg_pos={0 0}, upload=463024]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 15:22:49.951104575 (527h53m49.82218762s) [upload=463024]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 15:22:49.951113601 (527h53m49.822196657s) [upload=463024]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 15:22:50.253003756 (527h53m50.124086801s) [upload=463024]: Done uploading segments...
Dec 27 15:22:50.253023765 (527h53m50.1241068s) [upload=463024]: Upload finished. err: <nil>
Duration: 5.96340804s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
34 out of 60 is failed (threshold=6) [XX.....XXX..X..XX.XXXX.....XXX.X.XXXX.X.XXXX...XXXX.XXXXX...]
duration: 
3.530026555s
trace: 
https://jaeger.dev.storj.io/trace/2209e5f196d4ee2a
FAIL (2025-12-27 15:27:17 UTC) 
orbiter warning: context deadline exceeded
Duration: 3.530052103s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
6 out of 60 is failed (threshold=3) [........X..............X....X...................X.....XX....]
duration: 
46.621618701s
trace: 
https://jaeger.dev.storj.io/trace/1cf86c3db9e735d0
FAIL (2025-12-27 15:27:04 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
PBKNYT65NEYNQADUXQG2J3WTCLD343XLSQ, seg_pos={0 0}, upload=463022]: Uploading piece...
Dec 27 15:22:34.632070090 (527h53m34.503153125s) [noise=true , address=77.90.31.11:4565     , piece_id=B4VPI2I53PUUOLNS6C6GFKY3ZYDQ5RRQZCKC4QVLSVJI777RR6KQ, seg_pos={0 0}, upload=463022]: Uploading piece...
Dec 27 15:22:34.632523263 (527h53m34.503606298s) [noise=true , address=77.90.31.10:4540     , piece_id=MVAPVVOZIC5C6BL5S74PXSKTLTCGLIOSN37M33OUYVPD53NG3GLQ, seg_pos={0 0}, upload=463022]: Uploading piece...
Dec 27 15:22:34.632676374 (527h53m34.503759419s) [noise=true , address=77.90.31.13:4659     , piece_id=STKDI6VSN67XV43CB4ASCJTPDU6NA75REYKEE6D5K3EFM5FCDMYA, seg_pos={0 0}, upload=463022]: Uploading piece...
Dec 27 15:22:34.632931610 (527h53m34.504014655s) [noise=true , address=145.79.227.52:4466   , piece_id=CZYMGB53RT6FROGYMOYOPOQJ5CSZDW4YYAVDQWYWVLIDINN3LJVA, seg_pos={0 0}, upload=463022]: Uploading piece...
Dec 27 15:22:34.633357982 (527h53m34.504441017s) [noise=true , address=145.79.227.13:4816   , piece_id=LTSC26UNXCIHK4PDGYKE7GZCQ7MXCWWRGBLIHI6FPO6ISE4BPH7A, seg_pos={0 0}, upload=463022]: Uploading piece...
Dec 27 15:22:34.633490544 (527h53m34.504573579s) [noise=true , address=145.79.227.44:4218   , piece_id=2KHGZL76RA6CMI33PTF5WZN5TRIK3LRL6XT5IJP6LNJP5ZRVMHXA, seg_pos={0 0}, upload=463022]: Uploading piece...
Dec 27 15:22:34.633891278 (527h53m34.504974313s) [noise=true , address=77.90.31.18:4820     , piece_id=TSKDIH6HQUUQRTQTJDXGBX3OALVLYIUKR5QCPJFPFXTCASNN5JMQ, seg_pos={0 0}, upload=463022]: Uploading piece...
Dec 27 15:22:34.634043096 (527h53m34.505126141s) [noise=true , address=145.79.227.17:4997   , piece_id=VX4S5TXLHFNBRE7OYPDZAGYDGN2AD34ZL5JQ2YUSBV656RO6A5UA, seg_pos={0 0}, upload=463022]: Uploading piece...
Dec 27 15:22:34.634325093 (527h53m34.505408128s) [noise=true , address=77.90.31.16:4739     , piece_id=M7IJW7WX2FNPFWNJPBUAHQQOKPFXDSWAWMC2IHUYQZ4JPXF3YZXQ, seg_pos={0 0}, upload=463022]: Uploading piece...
Dec 27 15:22:34.634722230 (527h53m34.505805265s) [noise=true , address=77.90.31.15:4716     , piece_id=MLOLX2R56NCS4KT3FLJJNNPDAMUA6S5NSFCX57W7V6GMIZ4457DA, seg_pos={0 0}, upload=463022]: Uploading piece...
Dec 27 15:22:34.635114116 (527h53m34.506197151s) [noise=true , address=45.91.116.13:4171    , piece_id=YPPWJWZ6H4XMNLGVDHNRQPMZJDXYQRO2EYLNUVCC6WSHIRRJSEWQ, seg_pos={0 0}, upload=463022]: Uploading piece...
Dec 27 15:22:34.635719949 (527h53m34.506802984s) [noise=true , address=77.90.31.12:4618     , piece_id=ZII3XYSUJ7WTNNZA77GO54GFVGVHOGUUJOFZG2OJDTDDP6CAPC3A, seg_pos={0 0}, upload=463022]: Uploading piece...
Dec 27 15:22:34.636184344 (527h53m34.507267379s) [noise=true , address=145.79.227.30:4775   , piece_id=RG6IZXLWJSMKNHY2VR7VHDATQ2LAHIEGYF6WDQIZXUTGETQKS3DQ, seg_pos={0 0}, upload=463022]: Uploading piece...
Dec 27 15:22:36.104663675 (527h53m35.97574671s) [upload=463022]: Next returned nil segment
Dec 27 15:22:36.104675988 (527h53m35.975759023s) [upload=463022]: Waiting for error group managing segments...
Dec 27 15:22:36.505615923 (527h53m36.376698958s) [noise=true , address=145.79.227.17:4997   , piece_id=VX4S5TXLHFNBRE7OYPDZAGYDGN2AD34ZL5JQ2YUSBV656RO6A5UA, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil>
Dec 27 15:22:36.509752460 (527h53m36.380835505s) [noise=true , address=145.79.227.13:4816   , piece_id=LTSC26UNXCIHK4PDGYKE7GZCQ7MXCWWRGBLIHI6FPO6ISE4BPH7A, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil>
Dec 27 15:22:36.517493871 (527h53m36.388576906s) [noise=true , address=145.79.227.14:4865   , piece_id=XOXM2D6IS4SG6DN24DEGIPV7NK5FWDI7QVVJK7L3QB25AUPRNZIQ, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil>
Dec 27 15:22:36.623194766 (527h53m36.494277801s) [noise=true , address=145.79.227.52:4466   , piece_id=CZYMGB53RT6FROGYMOYOPOQJ5CSZDW4YYAVDQWYWVLIDINN3LJVA, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil>
Dec 27 15:22:36.749453706 (527h53m36.620536741s) [noise=true , address=45.91.116.11:4145    , piece_id=77YGIECBFASFKBHIQO333RHATPHZCA7JQW2WPNSGEGM24TTWDVGA, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil>
Dec 27 15:22:36.780705355 (527h53m36.6517884s) [noise=true , address=77.90.31.18:4820     , piece_id=TSKDIH6HQUUQRTQTJDXGBX3OALVLYIUKR5QCPJFPFXTCASNN5JMQ, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil>
Dec 27 15:22:36.803640327 (527h53m36.674723362s) [noise=true , address=77.90.31.13:4659     , piece_id=STKDI6VSN67XV43CB4ASCJTPDU6NA75REYKEE6D5K3EFM5FCDMYA, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil>
Dec 27 15:22:36.841101831 (527h53m36.712184876s) [noise=true , address=77.90.31.16:4739     , piece_id=M7IJW7WX2FNPFWNJPBUAHQQOKPFXDSWAWMC2IHUYQZ4JPXF3YZXQ, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil>
Dec 27 15:22:37.481224175 (527h53m37.35230722s) [noise=true , address=77.90.31.15:4716     , piece_id=MLOLX2R56NCS4KT3FLJJNNPDAMUA6S5NSFCX57W7V6GMIZ4457DA, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil>
Dec 27 15:22:37.504901952 (527h53m37.375984997s) [noise=true , address=145.79.227.30:4775   , piece_id=RG6IZXLWJSMKNHY2VR7VHDATQ2LAHIEGYF6WDQIZXUTGETQKS3DQ, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil>
Dec 27 15:22:37.553631302 (527h53m37.424714337s) [noise=true , address=145.79.227.44:4218   , piece_id=2KHGZL76RA6CMI33PTF5WZN5TRIK3LRL6XT5IJP6LNJP5ZRVMHXA, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil>
Dec 27 15:22:37.596982441 (527h53m37.468065476s) [noise=true , address=145.79.227.40:4063   , piece_id=SRV3KMN3XXUOJA2GLEJ27UKWRANPEKCT3NC7BBDAVMTCI6FZCNPQ, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil>
Dec 27 15:22:37.792107735 (527h53m37.663190771s) [noise=true , address=77.90.31.17:4802     , piece_id=TFQZEPPQN5RYDUNTGLILJUHZ3QLTU7BT3QP67SS5Y4ZRUKPG6A5A, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil>
Dec 27 15:22:37.802639390 (527h53m37.673722425s) [noise=true , address=77.90.31.12:4618     , piece_id=ZII3XYSUJ7WTNNZA77GO54GFVGVHOGUUJOFZG2OJDTDDP6CAPC3A, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil>
Dec 27 15:22:37.802661622 (527h53m37.673744667s) [seg_pos={0 0}, upload=463022]: Segment reached optimal threshold of 14 pieces.
Dec 27 15:22:37.802830423 (527h53m37.673913468s) [noise=true , address=77.90.31.11:4565     , piece_id=B4VPI2I53PUUOLNS6C6GFKY3ZYDQ5RRQZCKC4QVLSVJI777RR6KQ, seg_pos={0 0}, upload=463022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xnwFcUEdmpLjmx8NXZZcpiVQwTySXDRYbchuz4oREBq1yP6wh): protocol: write tcp 188.166.225.170:60558->77.90.31.11:4565: use of closed network connection; write tcp 188.166.225.170:60558->77.90.31.11:4565: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60558->77.90.31.11:4565: use of closed network connection
Dec 27 15:22:37.802837557 (527h53m37.673920592s) [noise=true , address=77.90.31.11:4565     , piece_id=B4VPI2I53PUUOLNS6C6GFKY3ZYDQ5RRQZCKC4QVLSVJI777RR6KQ, seg_pos={0 0}, upload=463022]: Long tail cancellation due to optimal threshold
Dec 27 15:22:37.803060181 (527h53m37.674143216s) [noise=true , address=45.91.116.13:4171    , piece_id=YPPWJWZ6H4XMNLGVDHNRQPMZJDXYQRO2EYLNUVCC6WSHIRRJSEWQ, seg_pos={0 0}, upload=463022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122dmyMaZDYbCvoYvCc9m8Vf5dA8aJiw9qPh1G5AQWKnBig4mVg): protocol: write tcp 188.166.225.170:35780->45.91.116.13:4171: use of closed network connection; write tcp 188.166.225.170:35780->45.91.116.13:4171: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35780->45.91.116.13:4171: use of closed network connection
Dec 27 15:22:37.803068377 (527h53m37.674151422s) [noise=true , address=45.91.116.13:4171    , piece_id=YPPWJWZ6H4XMNLGVDHNRQPMZJDXYQRO2EYLNUVCC6WSHIRRJSEWQ, seg_pos={0 0}, upload=463022]: Long tail cancellation due to optimal threshold
Dec 27 15:22:37.803144622 (527h53m37.674227667s) [noise=true , address=64.49.13.169:4708    , piece_id=J7UIJKZWYB2HZ2TCTMPBKNYT65NEYNQADUXQG2J3WTCLD343XLSQ, seg_pos={0 0}, upload=463022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:51138->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:51138->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51138->64.49.13.169:4708: use of closed network connection
Dec 27 15:22:37.803149080 (527h53m37.674232115s) [noise=true , address=64.49.13.169:4708    , piece_id=J7UIJKZWYB2HZ2TCTMPBKNYT65NEYNQADUXQG2J3WTCLD343XLSQ, seg_pos={0 0}, upload=463022]: Long tail cancellation due to optimal threshold
Dec 27 15:22:37.803258279 (527h53m37.674341314s) [noise=true , address=77.90.31.10:4540     , piece_id=MVAPVVOZIC5C6BL5S74PXSKTLTCGLIOSN37M33OUYVPD53NG3GLQ, seg_pos={0 0}, upload=463022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA): protocol: expected piece hash; context canceled; EOF
Dec 27 15:22:37.803263358 (527h53m37.674346393s) [noise=true , address=77.90.31.10:4540     , piece_id=MVAPVVOZIC5C6BL5S74PXSKTLTCGLIOSN37M33OUYVPD53NG3GLQ, seg_pos={0 0}, upload=463022]: Long tail cancellation due to optimal threshold
Dec 27 15:22:37.803277855 (527h53m37.674360901s) [seg_pos={0 0}, upload=463022]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 15:22:37.803341717 (527h53m37.674424762s) [upload=463022]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 15:22:37.803346236 (527h53m37.674429271s) [upload=463022]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 15:22:38.115241263 (527h53m37.986324298s) [upload=463022]: Done uploading segments...
Dec 27 15:22:38.115265930 (527h53m37.986348975s) [upload=463022]: Upload finished. err: <nil>
Duration: 46.62174402s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
20 out of 60 is failed (threshold=3) [....X..XX.......X..X.X..X...XXXXXXX..X.X..X.X.....X...X.....]
duration: 
5.672726488s
trace: 
https://jaeger.dev.storj.io/trace/3382203165a15a75
FAIL (2025-12-27 15:27:00 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: EFNF7USD6GGEH3V4T42NKM2JPEA3OCYTQOXTNIVRDA6COXPG5JTA) context deadline exceeded; piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: J3OER7CS2XJ6NVGM4X5XM6DWCWP2LZBOSVWD2CQNXABR4TEJXTIA) context deadline exceeded; piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: DBGSITGU4SZAAYFK5KDZ2MXONZ27XTNAV3OPQCQAUHKYBQNPNXSA) context deadline exceeded
======LOGS======
27 15:22:47.299628936 (527h53m47.170711971s) [upload=463024]: Starting upload
Dec 27 15:22:47.299696604 (527h53m47.17077964s) [upload=463024]: Done starting upload
Dec 27 15:22:47.299787547 (527h53m47.170870622s) [upload=463024]: Uploading segments...
Dec 27 15:22:47.299913407 (527h53m47.170996442s) [upload=463024]: Got next segment. Inline: false
Dec 27 15:22:47.299953483 (527h53m47.171036518s) [upload=463024]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 15:22:47.299958212 (527h53m47.171041257s) [upload=463024]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 15:22:47.545973752 (527h53m47.417056787s) [seg_pos={0 0}, upload=463024]: Begin upload segment...
Dec 27 15:22:47.546024249 (527h53m47.417107284s) [seg_pos={0 0}, upload=463024]: Done begin upload segment.
Dec 27 15:22:47.546057682 (527h53m47.417140717s) [noise=true , address=145.79.227.3:4151    , piece_id=627M5Y34ADXGUIAOOGMTG5PAEEZWBT3PAUOZHPI5M23TFTAV2OWQ, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:47.546065778 (527h53m47.417148813s) [noise=true , address=145.79.227.2:4113    , piece_id=YPFORZR5QCXBPQEQZKUVOIUVM4NYKOLU4KDYA7HRJDVMM3KXLKUQ, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:47.546207497 (527h53m47.417290542s) [noise=true , address=145.79.227.24:4581   , piece_id=7HS6Q5Q3ILUXV6BML37VQY2JS4P77CBO6ZTSENMY56MYOYJW6ANQ, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:47.546489023 (527h53m47.417572068s) [noise=true , address=145.79.227.29:4941   , piece_id=VHMADVXOXZOTYAVGE52WNY2IJWQMSQ7LNRSNXOJEZTFBYWLKIRIA, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:47.546754940 (527h53m47.417837975s) [noise=true , address=145.79.227.42:4121   , piece_id=6UQBDGBP6MIDTPV7RYC5NE7JOW6DP74P5QK3BBHVK4MV5UCKGZNA, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:47.547012460 (527h53m47.418095495s) [noise=true , address=145.79.227.47:4265   , piece_id=44SEXUFB2LZHONSA7GDQ63JOYEYETU5ZUMWF45SKXAE5AP7KVWOA, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:47.547340776 (527h53m47.418423811s) [noise=true , address=45.91.116.14:4192    , piece_id=JRQVETWQDYFMV3OR3BOK54FNOPFAZFQJH73WAXLM4KUDVKMKNF2Q, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:47.547610368 (527h53m47.418693413s) [noise=true , address=145.79.227.7:4295    , piece_id=4YVVZDHQNWCCA6XFV7A7ACJL7TUKPFPKKESO5WBIC7XEIDNULCWA, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:47.547903517 (527h53m47.418986552s) [noise=true , address=45.91.116.13:4154    , piece_id=EFNF7USD6GGEH3V4T42NKM2JPEA3OCYTQOXTNIVRDA6COXPG5JTA, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:47.548994664 (527h53m47.4200777s) [noise=true , address=145.79.227.50:4374   , piece_id=PTUKPJS4VWMT7CWQZIFOMI66JNJS6D2J3OMDYUBF4IL6LXMP465Q, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:47.550097284 (527h53m47.421180319s) [noise=true , address=145.79.227.30:4761   , piece_id=AGS3KQJG5YMSHYQA2C4UQRCYYW62QARBO4I2QWCOEBN445HMYQGQ, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:47.551125933 (527h53m47.422208968s) [noise=true , address=45.91.116.11:4119    , piece_id=J3OER7CS2XJ6NVGM4X5XM6DWCWP2LZBOSVWD2CQNXABR4TEJXTIA, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:47.552145084 (527h53m47.423228119s) [noise=true , address=145.79.227.19:4078   , piece_id=YZCJB4ZJN6BNWDDPOTGHEWQNRD5BCGZZACXVDF6XOX4FOLGZGDBQ, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:47.553167881 (527h53m47.424250916s) [noise=true , address=45.91.116.6:4976     , piece_id=ZHQFB66SB4X4AVSYKPFNDZMWURFQRVZT6QG7QC23JCRW22443M7A, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:47.554213733 (527h53m47.425296768s) [noise=true , address=45.91.116.12:4228    , piece_id=DBGSITGU4SZAAYFK5KDZ2MXONZ27XTNAV3OPQCQAUHKYBQNPNXSA, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:47.555339717 (527h53m47.426422752s) [noise=true , address=145.79.227.38:4977   , piece_id=UVSIVE7ZZECB74JPH7RFRAI74SX5HKKQK2ZHDRQ4K5X7JUZOWKWA, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:49.174094752 (527h53m49.045177787s) [upload=463024]: Next returned nil segment
Dec 27 15:22:49.174108467 (527h53m49.045191512s) [upload=463024]: Waiting for error group managing segments...
Dec 27 15:22:49.550005476 (527h53m49.421088511s) [noise=true , address=145.79.227.29:4941   , piece_id=VHMADVXOXZOTYAVGE52WNY2IJWQMSQ7LNRSNXOJEZTFBYWLKIRIA, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil>
Dec 27 15:22:49.551387929 (527h53m49.422470964s) [noise=true , address=145.79.227.7:4295    , piece_id=4YVVZDHQNWCCA6XFV7A7ACJL7TUKPFPKKESO5WBIC7XEIDNULCWA, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil>
Dec 27 15:22:49.554614222 (527h53m49.425697267s) [noise=true , address=145.79.227.19:4078   , piece_id=YZCJB4ZJN6BNWDDPOTGHEWQNRD5BCGZZACXVDF6XOX4FOLGZGDBQ, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil>
Dec 27 15:22:49.556863425 (527h53m49.42794646s) [noise=true , address=145.79.227.42:4121   , piece_id=6UQBDGBP6MIDTPV7RYC5NE7JOW6DP74P5QK3BBHVK4MV5UCKGZNA, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil>
Dec 27 15:22:49.557246234 (527h53m49.428329279s) [noise=true , address=145.79.227.38:4977   , piece_id=UVSIVE7ZZECB74JPH7RFRAI74SX5HKKQK2ZHDRQ4K5X7JUZOWKWA, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil>
Dec 27 15:22:49.558682599 (527h53m49.429765644s) [noise=true , address=145.79.227.2:4113    , piece_id=YPFORZR5QCXBPQEQZKUVOIUVM4NYKOLU4KDYA7HRJDVMM3KXLKUQ, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil>
Dec 27 15:22:49.564448258 (527h53m49.435531303s) [noise=true , address=145.79.227.3:4151    , piece_id=627M5Y34ADXGUIAOOGMTG5PAEEZWBT3PAUOZHPI5M23TFTAV2OWQ, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil>
Dec 27 15:22:49.566142284 (527h53m49.437225329s) [noise=true , address=145.79.227.50:4374   , piece_id=PTUKPJS4VWMT7CWQZIFOMI66JNJS6D2J3OMDYUBF4IL6LXMP465Q, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil>
Dec 27 15:22:49.574698126 (527h53m49.445781171s) [noise=true , address=145.79.227.24:4581   , piece_id=7HS6Q5Q3ILUXV6BML37VQY2JS4P77CBO6ZTSENMY56MYOYJW6ANQ, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil>
Dec 27 15:22:49.586061825 (527h53m49.45714487s) [noise=true , address=145.79.227.47:4265   , piece_id=44SEXUFB2LZHONSA7GDQ63JOYEYETU5ZUMWF45SKXAE5AP7KVWOA, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil>
Dec 27 15:22:49.793614724 (527h53m49.664697759s) [noise=true , address=45.91.116.11:4119    , piece_id=J3OER7CS2XJ6NVGM4X5XM6DWCWP2LZBOSVWD2CQNXABR4TEJXTIA, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil>
Dec 27 15:22:49.874495066 (527h53m49.745578101s) [noise=true , address=45.91.116.12:4228    , piece_id=DBGSITGU4SZAAYFK5KDZ2MXONZ27XTNAV3OPQCQAUHKYBQNPNXSA, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil>
Dec 27 15:22:49.950055185 (527h53m49.821138231s) [noise=true , address=45.91.116.13:4154    , piece_id=EFNF7USD6GGEH3V4T42NKM2JPEA3OCYTQOXTNIVRDA6COXPG5JTA, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil>
Dec 27 15:22:49.950079642 (527h53m49.821162677s) [seg_pos={0 0}, upload=463024]: Segment reached optimal threshold of 13 pieces.
Dec 27 15:22:49.950443906 (527h53m49.821526952s) [noise=true , address=145.79.227.30:4761   , piece_id=AGS3KQJG5YMSHYQA2C4UQRCYYW62QARBO4I2QWCOEBN445HMYQGQ, seg_pos={0 0}, upload=463024]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF
Dec 27 15:22:49.950463724 (527h53m49.821546759s) [noise=true , address=145.79.227.30:4761   , piece_id=AGS3KQJG5YMSHYQA2C4UQRCYYW62QARBO4I2QWCOEBN445HMYQGQ, seg_pos={0 0}, upload=463024]: Long tail cancellation due to optimal threshold
Dec 27 15:22:49.950484183 (527h53m49.821567228s) [noise=true , address=45.91.116.14:4192    , piece_id=JRQVETWQDYFMV3OR3BOK54FNOPFAZFQJH73WAXLM4KUDVKMKNF2Q, seg_pos={0 0}, upload=463024]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: write tcp 188.166.225.170:49708->45.91.116.14:4192: use of closed network connection; write tcp 188.166.225.170:49708->45.91.116.14:4192: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49708->45.91.116.14:4192: use of closed network connection
Dec 27 15:22:49.950523267 (527h53m49.821606302s) [noise=true , address=45.91.116.14:4192    , piece_id=JRQVETWQDYFMV3OR3BOK54FNOPFAZFQJH73WAXLM4KUDVKMKNF2Q, seg_pos={0 0}, upload=463024]: Long tail cancellation due to optimal threshold
Dec 27 15:22:49.950900406 (527h53m49.821983451s) [noise=true , address=45.91.116.6:4976     , piece_id=ZHQFB66SB4X4AVSYKPFNDZMWURFQRVZT6QG7QC23JCRW22443M7A, seg_pos={0 0}, upload=463024]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF
Dec 27 15:22:49.950911727 (527h53m49.821994772s) [noise=true , address=45.91.116.6:4976     , piece_id=ZHQFB66SB4X4AVSYKPFNDZMWURFQRVZT6QG7QC23JCRW22443M7A, seg_pos={0 0}, upload=463024]: Long tail cancellation due to optimal threshold
Dec 27 15:22:49.950943928 (527h53m49.822026993s) [seg_pos={0 0}, upload=463024]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 15:22:49.951104575 (527h53m49.82218762s) [upload=463024]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 15:22:49.951113601 (527h53m49.822196657s) [upload=463024]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 15:22:50.253003756 (527h53m50.124086801s) [upload=463024]: Done uploading segments...
Dec 27 15:22:50.253023765 (527h53m50.1241068s) [upload=463024]: Upload finished. err: <nil>
Duration: 5.672754892s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
20 out of 60 is failed (threshold=3) [.....X..XX.......X..X.X..X...XXXXXXX..X.X..X.X.....X...X....]
duration: 
5.882976636s
trace: 
https://jaeger.dev.storj.io/trace/3382203165a15a75
FAIL (2025-12-27 15:26:32 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: EFNF7USD6GGEH3V4T42NKM2JPEA3OCYTQOXTNIVRDA6COXPG5JTA) context deadline exceeded; piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: J3OER7CS2XJ6NVGM4X5XM6DWCWP2LZBOSVWD2CQNXABR4TEJXTIA) context deadline exceeded; piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: DBGSITGU4SZAAYFK5KDZ2MXONZ27XTNAV3OPQCQAUHKYBQNPNXSA) context deadline exceeded
======LOGS======
27 15:22:47.299628936 (527h53m47.170711971s) [upload=463024]: Starting upload
Dec 27 15:22:47.299696604 (527h53m47.17077964s) [upload=463024]: Done starting upload
Dec 27 15:22:47.299787547 (527h53m47.170870622s) [upload=463024]: Uploading segments...
Dec 27 15:22:47.299913407 (527h53m47.170996442s) [upload=463024]: Got next segment. Inline: false
Dec 27 15:22:47.299953483 (527h53m47.171036518s) [upload=463024]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 15:22:47.299958212 (527h53m47.171041257s) [upload=463024]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 15:22:47.545973752 (527h53m47.417056787s) [seg_pos={0 0}, upload=463024]: Begin upload segment...
Dec 27 15:22:47.546024249 (527h53m47.417107284s) [seg_pos={0 0}, upload=463024]: Done begin upload segment.
Dec 27 15:22:47.546057682 (527h53m47.417140717s) [noise=true , address=145.79.227.3:4151    , piece_id=627M5Y34ADXGUIAOOGMTG5PAEEZWBT3PAUOZHPI5M23TFTAV2OWQ, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:47.546065778 (527h53m47.417148813s) [noise=true , address=145.79.227.2:4113    , piece_id=YPFORZR5QCXBPQEQZKUVOIUVM4NYKOLU4KDYA7HRJDVMM3KXLKUQ, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:47.546207497 (527h53m47.417290542s) [noise=true , address=145.79.227.24:4581   , piece_id=7HS6Q5Q3ILUXV6BML37VQY2JS4P77CBO6ZTSENMY56MYOYJW6ANQ, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:47.546489023 (527h53m47.417572068s) [noise=true , address=145.79.227.29:4941   , piece_id=VHMADVXOXZOTYAVGE52WNY2IJWQMSQ7LNRSNXOJEZTFBYWLKIRIA, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:47.546754940 (527h53m47.417837975s) [noise=true , address=145.79.227.42:4121   , piece_id=6UQBDGBP6MIDTPV7RYC5NE7JOW6DP74P5QK3BBHVK4MV5UCKGZNA, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:47.547012460 (527h53m47.418095495s) [noise=true , address=145.79.227.47:4265   , piece_id=44SEXUFB2LZHONSA7GDQ63JOYEYETU5ZUMWF45SKXAE5AP7KVWOA, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:47.547340776 (527h53m47.418423811s) [noise=true , address=45.91.116.14:4192    , piece_id=JRQVETWQDYFMV3OR3BOK54FNOPFAZFQJH73WAXLM4KUDVKMKNF2Q, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:47.547610368 (527h53m47.418693413s) [noise=true , address=145.79.227.7:4295    , piece_id=4YVVZDHQNWCCA6XFV7A7ACJL7TUKPFPKKESO5WBIC7XEIDNULCWA, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:47.547903517 (527h53m47.418986552s) [noise=true , address=45.91.116.13:4154    , piece_id=EFNF7USD6GGEH3V4T42NKM2JPEA3OCYTQOXTNIVRDA6COXPG5JTA, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:47.548994664 (527h53m47.4200777s) [noise=true , address=145.79.227.50:4374   , piece_id=PTUKPJS4VWMT7CWQZIFOMI66JNJS6D2J3OMDYUBF4IL6LXMP465Q, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:47.550097284 (527h53m47.421180319s) [noise=true , address=145.79.227.30:4761   , piece_id=AGS3KQJG5YMSHYQA2C4UQRCYYW62QARBO4I2QWCOEBN445HMYQGQ, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:47.551125933 (527h53m47.422208968s) [noise=true , address=45.91.116.11:4119    , piece_id=J3OER7CS2XJ6NVGM4X5XM6DWCWP2LZBOSVWD2CQNXABR4TEJXTIA, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:47.552145084 (527h53m47.423228119s) [noise=true , address=145.79.227.19:4078   , piece_id=YZCJB4ZJN6BNWDDPOTGHEWQNRD5BCGZZACXVDF6XOX4FOLGZGDBQ, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:47.553167881 (527h53m47.424250916s) [noise=true , address=45.91.116.6:4976     , piece_id=ZHQFB66SB4X4AVSYKPFNDZMWURFQRVZT6QG7QC23JCRW22443M7A, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:47.554213733 (527h53m47.425296768s) [noise=true , address=45.91.116.12:4228    , piece_id=DBGSITGU4SZAAYFK5KDZ2MXONZ27XTNAV3OPQCQAUHKYBQNPNXSA, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:47.555339717 (527h53m47.426422752s) [noise=true , address=145.79.227.38:4977   , piece_id=UVSIVE7ZZECB74JPH7RFRAI74SX5HKKQK2ZHDRQ4K5X7JUZOWKWA, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:49.174094752 (527h53m49.045177787s) [upload=463024]: Next returned nil segment
Dec 27 15:22:49.174108467 (527h53m49.045191512s) [upload=463024]: Waiting for error group managing segments...
Dec 27 15:22:49.550005476 (527h53m49.421088511s) [noise=true , address=145.79.227.29:4941   , piece_id=VHMADVXOXZOTYAVGE52WNY2IJWQMSQ7LNRSNXOJEZTFBYWLKIRIA, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil>
Dec 27 15:22:49.551387929 (527h53m49.422470964s) [noise=true , address=145.79.227.7:4295    , piece_id=4YVVZDHQNWCCA6XFV7A7ACJL7TUKPFPKKESO5WBIC7XEIDNULCWA, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil>
Dec 27 15:22:49.554614222 (527h53m49.425697267s) [noise=true , address=145.79.227.19:4078   , piece_id=YZCJB4ZJN6BNWDDPOTGHEWQNRD5BCGZZACXVDF6XOX4FOLGZGDBQ, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil>
Dec 27 15:22:49.556863425 (527h53m49.42794646s) [noise=true , address=145.79.227.42:4121   , piece_id=6UQBDGBP6MIDTPV7RYC5NE7JOW6DP74P5QK3BBHVK4MV5UCKGZNA, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil>
Dec 27 15:22:49.557246234 (527h53m49.428329279s) [noise=true , address=145.79.227.38:4977   , piece_id=UVSIVE7ZZECB74JPH7RFRAI74SX5HKKQK2ZHDRQ4K5X7JUZOWKWA, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil>
Dec 27 15:22:49.558682599 (527h53m49.429765644s) [noise=true , address=145.79.227.2:4113    , piece_id=YPFORZR5QCXBPQEQZKUVOIUVM4NYKOLU4KDYA7HRJDVMM3KXLKUQ, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil>
Dec 27 15:22:49.564448258 (527h53m49.435531303s) [noise=true , address=145.79.227.3:4151    , piece_id=627M5Y34ADXGUIAOOGMTG5PAEEZWBT3PAUOZHPI5M23TFTAV2OWQ, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil>
Dec 27 15:22:49.566142284 (527h53m49.437225329s) [noise=true , address=145.79.227.50:4374   , piece_id=PTUKPJS4VWMT7CWQZIFOMI66JNJS6D2J3OMDYUBF4IL6LXMP465Q, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil>
Dec 27 15:22:49.574698126 (527h53m49.445781171s) [noise=true , address=145.79.227.24:4581   , piece_id=7HS6Q5Q3ILUXV6BML37VQY2JS4P77CBO6ZTSENMY56MYOYJW6ANQ, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil>
Dec 27 15:22:49.586061825 (527h53m49.45714487s) [noise=true , address=145.79.227.47:4265   , piece_id=44SEXUFB2LZHONSA7GDQ63JOYEYETU5ZUMWF45SKXAE5AP7KVWOA, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil>
Dec 27 15:22:49.793614724 (527h53m49.664697759s) [noise=true , address=45.91.116.11:4119    , piece_id=J3OER7CS2XJ6NVGM4X5XM6DWCWP2LZBOSVWD2CQNXABR4TEJXTIA, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil>
Dec 27 15:22:49.874495066 (527h53m49.745578101s) [noise=true , address=45.91.116.12:4228    , piece_id=DBGSITGU4SZAAYFK5KDZ2MXONZ27XTNAV3OPQCQAUHKYBQNPNXSA, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil>
Dec 27 15:22:49.950055185 (527h53m49.821138231s) [noise=true , address=45.91.116.13:4154    , piece_id=EFNF7USD6GGEH3V4T42NKM2JPEA3OCYTQOXTNIVRDA6COXPG5JTA, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil>
Dec 27 15:22:49.950079642 (527h53m49.821162677s) [seg_pos={0 0}, upload=463024]: Segment reached optimal threshold of 13 pieces.
Dec 27 15:22:49.950443906 (527h53m49.821526952s) [noise=true , address=145.79.227.30:4761   , piece_id=AGS3KQJG5YMSHYQA2C4UQRCYYW62QARBO4I2QWCOEBN445HMYQGQ, seg_pos={0 0}, upload=463024]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF
Dec 27 15:22:49.950463724 (527h53m49.821546759s) [noise=true , address=145.79.227.30:4761   , piece_id=AGS3KQJG5YMSHYQA2C4UQRCYYW62QARBO4I2QWCOEBN445HMYQGQ, seg_pos={0 0}, upload=463024]: Long tail cancellation due to optimal threshold
Dec 27 15:22:49.950484183 (527h53m49.821567228s) [noise=true , address=45.91.116.14:4192    , piece_id=JRQVETWQDYFMV3OR3BOK54FNOPFAZFQJH73WAXLM4KUDVKMKNF2Q, seg_pos={0 0}, upload=463024]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: write tcp 188.166.225.170:49708->45.91.116.14:4192: use of closed network connection; write tcp 188.166.225.170:49708->45.91.116.14:4192: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49708->45.91.116.14:4192: use of closed network connection
Dec 27 15:22:49.950523267 (527h53m49.821606302s) [noise=true , address=45.91.116.14:4192    , piece_id=JRQVETWQDYFMV3OR3BOK54FNOPFAZFQJH73WAXLM4KUDVKMKNF2Q, seg_pos={0 0}, upload=463024]: Long tail cancellation due to optimal threshold
Dec 27 15:22:49.950900406 (527h53m49.821983451s) [noise=true , address=45.91.116.6:4976     , piece_id=ZHQFB66SB4X4AVSYKPFNDZMWURFQRVZT6QG7QC23JCRW22443M7A, seg_pos={0 0}, upload=463024]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF
Dec 27 15:22:49.950911727 (527h53m49.821994772s) [noise=true , address=45.91.116.6:4976     , piece_id=ZHQFB66SB4X4AVSYKPFNDZMWURFQRVZT6QG7QC23JCRW22443M7A, seg_pos={0 0}, upload=463024]: Long tail cancellation due to optimal threshold
Dec 27 15:22:49.950943928 (527h53m49.822026993s) [seg_pos={0 0}, upload=463024]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 15:22:49.951104575 (527h53m49.82218762s) [upload=463024]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 15:22:49.951113601 (527h53m49.822196657s) [upload=463024]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 15:22:50.253003756 (527h53m50.124086801s) [upload=463024]: Done uploading segments...
Dec 27 15:22:50.253023765 (527h53m50.1241068s) [upload=463024]: Upload finished. err: <nil>
Duration: 5.883001585s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
6 out of 60 is failed (threshold=3) [.........X..............X....X...................X.....XX...]
duration: 
36.521123239s
trace: 
https://jaeger.dev.storj.io/trace/1cf86c3db9e735d0
FAIL (2025-12-27 15:26:18 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
PBKNYT65NEYNQADUXQG2J3WTCLD343XLSQ, seg_pos={0 0}, upload=463022]: Uploading piece...
Dec 27 15:22:34.632070090 (527h53m34.503153125s) [noise=true , address=77.90.31.11:4565     , piece_id=B4VPI2I53PUUOLNS6C6GFKY3ZYDQ5RRQZCKC4QVLSVJI777RR6KQ, seg_pos={0 0}, upload=463022]: Uploading piece...
Dec 27 15:22:34.632523263 (527h53m34.503606298s) [noise=true , address=77.90.31.10:4540     , piece_id=MVAPVVOZIC5C6BL5S74PXSKTLTCGLIOSN37M33OUYVPD53NG3GLQ, seg_pos={0 0}, upload=463022]: Uploading piece...
Dec 27 15:22:34.632676374 (527h53m34.503759419s) [noise=true , address=77.90.31.13:4659     , piece_id=STKDI6VSN67XV43CB4ASCJTPDU6NA75REYKEE6D5K3EFM5FCDMYA, seg_pos={0 0}, upload=463022]: Uploading piece...
Dec 27 15:22:34.632931610 (527h53m34.504014655s) [noise=true , address=145.79.227.52:4466   , piece_id=CZYMGB53RT6FROGYMOYOPOQJ5CSZDW4YYAVDQWYWVLIDINN3LJVA, seg_pos={0 0}, upload=463022]: Uploading piece...
Dec 27 15:22:34.633357982 (527h53m34.504441017s) [noise=true , address=145.79.227.13:4816   , piece_id=LTSC26UNXCIHK4PDGYKE7GZCQ7MXCWWRGBLIHI6FPO6ISE4BPH7A, seg_pos={0 0}, upload=463022]: Uploading piece...
Dec 27 15:22:34.633490544 (527h53m34.504573579s) [noise=true , address=145.79.227.44:4218   , piece_id=2KHGZL76RA6CMI33PTF5WZN5TRIK3LRL6XT5IJP6LNJP5ZRVMHXA, seg_pos={0 0}, upload=463022]: Uploading piece...
Dec 27 15:22:34.633891278 (527h53m34.504974313s) [noise=true , address=77.90.31.18:4820     , piece_id=TSKDIH6HQUUQRTQTJDXGBX3OALVLYIUKR5QCPJFPFXTCASNN5JMQ, seg_pos={0 0}, upload=463022]: Uploading piece...
Dec 27 15:22:34.634043096 (527h53m34.505126141s) [noise=true , address=145.79.227.17:4997   , piece_id=VX4S5TXLHFNBRE7OYPDZAGYDGN2AD34ZL5JQ2YUSBV656RO6A5UA, seg_pos={0 0}, upload=463022]: Uploading piece...
Dec 27 15:22:34.634325093 (527h53m34.505408128s) [noise=true , address=77.90.31.16:4739     , piece_id=M7IJW7WX2FNPFWNJPBUAHQQOKPFXDSWAWMC2IHUYQZ4JPXF3YZXQ, seg_pos={0 0}, upload=463022]: Uploading piece...
Dec 27 15:22:34.634722230 (527h53m34.505805265s) [noise=true , address=77.90.31.15:4716     , piece_id=MLOLX2R56NCS4KT3FLJJNNPDAMUA6S5NSFCX57W7V6GMIZ4457DA, seg_pos={0 0}, upload=463022]: Uploading piece...
Dec 27 15:22:34.635114116 (527h53m34.506197151s) [noise=true , address=45.91.116.13:4171    , piece_id=YPPWJWZ6H4XMNLGVDHNRQPMZJDXYQRO2EYLNUVCC6WSHIRRJSEWQ, seg_pos={0 0}, upload=463022]: Uploading piece...
Dec 27 15:22:34.635719949 (527h53m34.506802984s) [noise=true , address=77.90.31.12:4618     , piece_id=ZII3XYSUJ7WTNNZA77GO54GFVGVHOGUUJOFZG2OJDTDDP6CAPC3A, seg_pos={0 0}, upload=463022]: Uploading piece...
Dec 27 15:22:34.636184344 (527h53m34.507267379s) [noise=true , address=145.79.227.30:4775   , piece_id=RG6IZXLWJSMKNHY2VR7VHDATQ2LAHIEGYF6WDQIZXUTGETQKS3DQ, seg_pos={0 0}, upload=463022]: Uploading piece...
Dec 27 15:22:36.104663675 (527h53m35.97574671s) [upload=463022]: Next returned nil segment
Dec 27 15:22:36.104675988 (527h53m35.975759023s) [upload=463022]: Waiting for error group managing segments...
Dec 27 15:22:36.505615923 (527h53m36.376698958s) [noise=true , address=145.79.227.17:4997   , piece_id=VX4S5TXLHFNBRE7OYPDZAGYDGN2AD34ZL5JQ2YUSBV656RO6A5UA, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil>
Dec 27 15:22:36.509752460 (527h53m36.380835505s) [noise=true , address=145.79.227.13:4816   , piece_id=LTSC26UNXCIHK4PDGYKE7GZCQ7MXCWWRGBLIHI6FPO6ISE4BPH7A, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil>
Dec 27 15:22:36.517493871 (527h53m36.388576906s) [noise=true , address=145.79.227.14:4865   , piece_id=XOXM2D6IS4SG6DN24DEGIPV7NK5FWDI7QVVJK7L3QB25AUPRNZIQ, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil>
Dec 27 15:22:36.623194766 (527h53m36.494277801s) [noise=true , address=145.79.227.52:4466   , piece_id=CZYMGB53RT6FROGYMOYOPOQJ5CSZDW4YYAVDQWYWVLIDINN3LJVA, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil>
Dec 27 15:22:36.749453706 (527h53m36.620536741s) [noise=true , address=45.91.116.11:4145    , piece_id=77YGIECBFASFKBHIQO333RHATPHZCA7JQW2WPNSGEGM24TTWDVGA, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil>
Dec 27 15:22:36.780705355 (527h53m36.6517884s) [noise=true , address=77.90.31.18:4820     , piece_id=TSKDIH6HQUUQRTQTJDXGBX3OALVLYIUKR5QCPJFPFXTCASNN5JMQ, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil>
Dec 27 15:22:36.803640327 (527h53m36.674723362s) [noise=true , address=77.90.31.13:4659     , piece_id=STKDI6VSN67XV43CB4ASCJTPDU6NA75REYKEE6D5K3EFM5FCDMYA, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil>
Dec 27 15:22:36.841101831 (527h53m36.712184876s) [noise=true , address=77.90.31.16:4739     , piece_id=M7IJW7WX2FNPFWNJPBUAHQQOKPFXDSWAWMC2IHUYQZ4JPXF3YZXQ, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil>
Dec 27 15:22:37.481224175 (527h53m37.35230722s) [noise=true , address=77.90.31.15:4716     , piece_id=MLOLX2R56NCS4KT3FLJJNNPDAMUA6S5NSFCX57W7V6GMIZ4457DA, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil>
Dec 27 15:22:37.504901952 (527h53m37.375984997s) [noise=true , address=145.79.227.30:4775   , piece_id=RG6IZXLWJSMKNHY2VR7VHDATQ2LAHIEGYF6WDQIZXUTGETQKS3DQ, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil>
Dec 27 15:22:37.553631302 (527h53m37.424714337s) [noise=true , address=145.79.227.44:4218   , piece_id=2KHGZL76RA6CMI33PTF5WZN5TRIK3LRL6XT5IJP6LNJP5ZRVMHXA, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil>
Dec 27 15:22:37.596982441 (527h53m37.468065476s) [noise=true , address=145.79.227.40:4063   , piece_id=SRV3KMN3XXUOJA2GLEJ27UKWRANPEKCT3NC7BBDAVMTCI6FZCNPQ, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil>
Dec 27 15:22:37.792107735 (527h53m37.663190771s) [noise=true , address=77.90.31.17:4802     , piece_id=TFQZEPPQN5RYDUNTGLILJUHZ3QLTU7BT3QP67SS5Y4ZRUKPG6A5A, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil>
Dec 27 15:22:37.802639390 (527h53m37.673722425s) [noise=true , address=77.90.31.12:4618     , piece_id=ZII3XYSUJ7WTNNZA77GO54GFVGVHOGUUJOFZG2OJDTDDP6CAPC3A, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil>
Dec 27 15:22:37.802661622 (527h53m37.673744667s) [seg_pos={0 0}, upload=463022]: Segment reached optimal threshold of 14 pieces.
Dec 27 15:22:37.802830423 (527h53m37.673913468s) [noise=true , address=77.90.31.11:4565     , piece_id=B4VPI2I53PUUOLNS6C6GFKY3ZYDQ5RRQZCKC4QVLSVJI777RR6KQ, seg_pos={0 0}, upload=463022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xnwFcUEdmpLjmx8NXZZcpiVQwTySXDRYbchuz4oREBq1yP6wh): protocol: write tcp 188.166.225.170:60558->77.90.31.11:4565: use of closed network connection; write tcp 188.166.225.170:60558->77.90.31.11:4565: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60558->77.90.31.11:4565: use of closed network connection
Dec 27 15:22:37.802837557 (527h53m37.673920592s) [noise=true , address=77.90.31.11:4565     , piece_id=B4VPI2I53PUUOLNS6C6GFKY3ZYDQ5RRQZCKC4QVLSVJI777RR6KQ, seg_pos={0 0}, upload=463022]: Long tail cancellation due to optimal threshold
Dec 27 15:22:37.803060181 (527h53m37.674143216s) [noise=true , address=45.91.116.13:4171    , piece_id=YPPWJWZ6H4XMNLGVDHNRQPMZJDXYQRO2EYLNUVCC6WSHIRRJSEWQ, seg_pos={0 0}, upload=463022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122dmyMaZDYbCvoYvCc9m8Vf5dA8aJiw9qPh1G5AQWKnBig4mVg): protocol: write tcp 188.166.225.170:35780->45.91.116.13:4171: use of closed network connection; write tcp 188.166.225.170:35780->45.91.116.13:4171: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35780->45.91.116.13:4171: use of closed network connection
Dec 27 15:22:37.803068377 (527h53m37.674151422s) [noise=true , address=45.91.116.13:4171    , piece_id=YPPWJWZ6H4XMNLGVDHNRQPMZJDXYQRO2EYLNUVCC6WSHIRRJSEWQ, seg_pos={0 0}, upload=463022]: Long tail cancellation due to optimal threshold
Dec 27 15:22:37.803144622 (527h53m37.674227667s) [noise=true , address=64.49.13.169:4708    , piece_id=J7UIJKZWYB2HZ2TCTMPBKNYT65NEYNQADUXQG2J3WTCLD343XLSQ, seg_pos={0 0}, upload=463022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:51138->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:51138->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51138->64.49.13.169:4708: use of closed network connection
Dec 27 15:22:37.803149080 (527h53m37.674232115s) [noise=true , address=64.49.13.169:4708    , piece_id=J7UIJKZWYB2HZ2TCTMPBKNYT65NEYNQADUXQG2J3WTCLD343XLSQ, seg_pos={0 0}, upload=463022]: Long tail cancellation due to optimal threshold
Dec 27 15:22:37.803258279 (527h53m37.674341314s) [noise=true , address=77.90.31.10:4540     , piece_id=MVAPVVOZIC5C6BL5S74PXSKTLTCGLIOSN37M33OUYVPD53NG3GLQ, seg_pos={0 0}, upload=463022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA): protocol: expected piece hash; context canceled; EOF
Dec 27 15:22:37.803263358 (527h53m37.674346393s) [noise=true , address=77.90.31.10:4540     , piece_id=MVAPVVOZIC5C6BL5S74PXSKTLTCGLIOSN37M33OUYVPD53NG3GLQ, seg_pos={0 0}, upload=463022]: Long tail cancellation due to optimal threshold
Dec 27 15:22:37.803277855 (527h53m37.674360901s) [seg_pos={0 0}, upload=463022]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 15:22:37.803341717 (527h53m37.674424762s) [upload=463022]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 15:22:37.803346236 (527h53m37.674429271s) [upload=463022]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 15:22:38.115241263 (527h53m37.986324298s) [upload=463022]: Done uploading segments...
Dec 27 15:22:38.115265930 (527h53m37.986348975s) [upload=463022]: Upload finished. err: <nil>
Duration: 36.521140271s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
35 out of 60 is failed (threshold=6) [XXX.....XXX..X..XX.XXXX.....XXX.X.XXXX.X.XXXX...XXXX.XXXXX..]
duration: 
3.896831259s
trace: 
https://jaeger.dev.storj.io/trace/2209e5f196d4ee2a
FAIL (2025-12-27 15:26:03 UTC) 
orbiter warning: context deadline exceeded
Duration: 3.896861657s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
6 out of 60 is failed (threshold=3) [..........X..............X....X...................X.....XX..]
duration: 
58.657156113s
trace: 
https://jaeger.dev.storj.io/trace/1cf86c3db9e735d0
FAIL (2025-12-27 15:25:41 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
PBKNYT65NEYNQADUXQG2J3WTCLD343XLSQ, seg_pos={0 0}, upload=463022]: Uploading piece...
Dec 27 15:22:34.632070090 (527h53m34.503153125s) [noise=true , address=77.90.31.11:4565     , piece_id=B4VPI2I53PUUOLNS6C6GFKY3ZYDQ5RRQZCKC4QVLSVJI777RR6KQ, seg_pos={0 0}, upload=463022]: Uploading piece...
Dec 27 15:22:34.632523263 (527h53m34.503606298s) [noise=true , address=77.90.31.10:4540     , piece_id=MVAPVVOZIC5C6BL5S74PXSKTLTCGLIOSN37M33OUYVPD53NG3GLQ, seg_pos={0 0}, upload=463022]: Uploading piece...
Dec 27 15:22:34.632676374 (527h53m34.503759419s) [noise=true , address=77.90.31.13:4659     , piece_id=STKDI6VSN67XV43CB4ASCJTPDU6NA75REYKEE6D5K3EFM5FCDMYA, seg_pos={0 0}, upload=463022]: Uploading piece...
Dec 27 15:22:34.632931610 (527h53m34.504014655s) [noise=true , address=145.79.227.52:4466   , piece_id=CZYMGB53RT6FROGYMOYOPOQJ5CSZDW4YYAVDQWYWVLIDINN3LJVA, seg_pos={0 0}, upload=463022]: Uploading piece...
Dec 27 15:22:34.633357982 (527h53m34.504441017s) [noise=true , address=145.79.227.13:4816   , piece_id=LTSC26UNXCIHK4PDGYKE7GZCQ7MXCWWRGBLIHI6FPO6ISE4BPH7A, seg_pos={0 0}, upload=463022]: Uploading piece...
Dec 27 15:22:34.633490544 (527h53m34.504573579s) [noise=true , address=145.79.227.44:4218   , piece_id=2KHGZL76RA6CMI33PTF5WZN5TRIK3LRL6XT5IJP6LNJP5ZRVMHXA, seg_pos={0 0}, upload=463022]: Uploading piece...
Dec 27 15:22:34.633891278 (527h53m34.504974313s) [noise=true , address=77.90.31.18:4820     , piece_id=TSKDIH6HQUUQRTQTJDXGBX3OALVLYIUKR5QCPJFPFXTCASNN5JMQ, seg_pos={0 0}, upload=463022]: Uploading piece...
Dec 27 15:22:34.634043096 (527h53m34.505126141s) [noise=true , address=145.79.227.17:4997   , piece_id=VX4S5TXLHFNBRE7OYPDZAGYDGN2AD34ZL5JQ2YUSBV656RO6A5UA, seg_pos={0 0}, upload=463022]: Uploading piece...
Dec 27 15:22:34.634325093 (527h53m34.505408128s) [noise=true , address=77.90.31.16:4739     , piece_id=M7IJW7WX2FNPFWNJPBUAHQQOKPFXDSWAWMC2IHUYQZ4JPXF3YZXQ, seg_pos={0 0}, upload=463022]: Uploading piece...
Dec 27 15:22:34.634722230 (527h53m34.505805265s) [noise=true , address=77.90.31.15:4716     , piece_id=MLOLX2R56NCS4KT3FLJJNNPDAMUA6S5NSFCX57W7V6GMIZ4457DA, seg_pos={0 0}, upload=463022]: Uploading piece...
Dec 27 15:22:34.635114116 (527h53m34.506197151s) [noise=true , address=45.91.116.13:4171    , piece_id=YPPWJWZ6H4XMNLGVDHNRQPMZJDXYQRO2EYLNUVCC6WSHIRRJSEWQ, seg_pos={0 0}, upload=463022]: Uploading piece...
Dec 27 15:22:34.635719949 (527h53m34.506802984s) [noise=true , address=77.90.31.12:4618     , piece_id=ZII3XYSUJ7WTNNZA77GO54GFVGVHOGUUJOFZG2OJDTDDP6CAPC3A, seg_pos={0 0}, upload=463022]: Uploading piece...
Dec 27 15:22:34.636184344 (527h53m34.507267379s) [noise=true , address=145.79.227.30:4775   , piece_id=RG6IZXLWJSMKNHY2VR7VHDATQ2LAHIEGYF6WDQIZXUTGETQKS3DQ, seg_pos={0 0}, upload=463022]: Uploading piece...
Dec 27 15:22:36.104663675 (527h53m35.97574671s) [upload=463022]: Next returned nil segment
Dec 27 15:22:36.104675988 (527h53m35.975759023s) [upload=463022]: Waiting for error group managing segments...
Dec 27 15:22:36.505615923 (527h53m36.376698958s) [noise=true , address=145.79.227.17:4997   , piece_id=VX4S5TXLHFNBRE7OYPDZAGYDGN2AD34ZL5JQ2YUSBV656RO6A5UA, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil>
Dec 27 15:22:36.509752460 (527h53m36.380835505s) [noise=true , address=145.79.227.13:4816   , piece_id=LTSC26UNXCIHK4PDGYKE7GZCQ7MXCWWRGBLIHI6FPO6ISE4BPH7A, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil>
Dec 27 15:22:36.517493871 (527h53m36.388576906s) [noise=true , address=145.79.227.14:4865   , piece_id=XOXM2D6IS4SG6DN24DEGIPV7NK5FWDI7QVVJK7L3QB25AUPRNZIQ, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil>
Dec 27 15:22:36.623194766 (527h53m36.494277801s) [noise=true , address=145.79.227.52:4466   , piece_id=CZYMGB53RT6FROGYMOYOPOQJ5CSZDW4YYAVDQWYWVLIDINN3LJVA, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil>
Dec 27 15:22:36.749453706 (527h53m36.620536741s) [noise=true , address=45.91.116.11:4145    , piece_id=77YGIECBFASFKBHIQO333RHATPHZCA7JQW2WPNSGEGM24TTWDVGA, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil>
Dec 27 15:22:36.780705355 (527h53m36.6517884s) [noise=true , address=77.90.31.18:4820     , piece_id=TSKDIH6HQUUQRTQTJDXGBX3OALVLYIUKR5QCPJFPFXTCASNN5JMQ, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil>
Dec 27 15:22:36.803640327 (527h53m36.674723362s) [noise=true , address=77.90.31.13:4659     , piece_id=STKDI6VSN67XV43CB4ASCJTPDU6NA75REYKEE6D5K3EFM5FCDMYA, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil>
Dec 27 15:22:36.841101831 (527h53m36.712184876s) [noise=true , address=77.90.31.16:4739     , piece_id=M7IJW7WX2FNPFWNJPBUAHQQOKPFXDSWAWMC2IHUYQZ4JPXF3YZXQ, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil>
Dec 27 15:22:37.481224175 (527h53m37.35230722s) [noise=true , address=77.90.31.15:4716     , piece_id=MLOLX2R56NCS4KT3FLJJNNPDAMUA6S5NSFCX57W7V6GMIZ4457DA, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil>
Dec 27 15:22:37.504901952 (527h53m37.375984997s) [noise=true , address=145.79.227.30:4775   , piece_id=RG6IZXLWJSMKNHY2VR7VHDATQ2LAHIEGYF6WDQIZXUTGETQKS3DQ, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil>
Dec 27 15:22:37.553631302 (527h53m37.424714337s) [noise=true , address=145.79.227.44:4218   , piece_id=2KHGZL76RA6CMI33PTF5WZN5TRIK3LRL6XT5IJP6LNJP5ZRVMHXA, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil>
Dec 27 15:22:37.596982441 (527h53m37.468065476s) [noise=true , address=145.79.227.40:4063   , piece_id=SRV3KMN3XXUOJA2GLEJ27UKWRANPEKCT3NC7BBDAVMTCI6FZCNPQ, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil>
Dec 27 15:22:37.792107735 (527h53m37.663190771s) [noise=true , address=77.90.31.17:4802     , piece_id=TFQZEPPQN5RYDUNTGLILJUHZ3QLTU7BT3QP67SS5Y4ZRUKPG6A5A, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil>
Dec 27 15:22:37.802639390 (527h53m37.673722425s) [noise=true , address=77.90.31.12:4618     , piece_id=ZII3XYSUJ7WTNNZA77GO54GFVGVHOGUUJOFZG2OJDTDDP6CAPC3A, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil>
Dec 27 15:22:37.802661622 (527h53m37.673744667s) [seg_pos={0 0}, upload=463022]: Segment reached optimal threshold of 14 pieces.
Dec 27 15:22:37.802830423 (527h53m37.673913468s) [noise=true , address=77.90.31.11:4565     , piece_id=B4VPI2I53PUUOLNS6C6GFKY3ZYDQ5RRQZCKC4QVLSVJI777RR6KQ, seg_pos={0 0}, upload=463022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xnwFcUEdmpLjmx8NXZZcpiVQwTySXDRYbchuz4oREBq1yP6wh): protocol: write tcp 188.166.225.170:60558->77.90.31.11:4565: use of closed network connection; write tcp 188.166.225.170:60558->77.90.31.11:4565: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60558->77.90.31.11:4565: use of closed network connection
Dec 27 15:22:37.802837557 (527h53m37.673920592s) [noise=true , address=77.90.31.11:4565     , piece_id=B4VPI2I53PUUOLNS6C6GFKY3ZYDQ5RRQZCKC4QVLSVJI777RR6KQ, seg_pos={0 0}, upload=463022]: Long tail cancellation due to optimal threshold
Dec 27 15:22:37.803060181 (527h53m37.674143216s) [noise=true , address=45.91.116.13:4171    , piece_id=YPPWJWZ6H4XMNLGVDHNRQPMZJDXYQRO2EYLNUVCC6WSHIRRJSEWQ, seg_pos={0 0}, upload=463022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122dmyMaZDYbCvoYvCc9m8Vf5dA8aJiw9qPh1G5AQWKnBig4mVg): protocol: write tcp 188.166.225.170:35780->45.91.116.13:4171: use of closed network connection; write tcp 188.166.225.170:35780->45.91.116.13:4171: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35780->45.91.116.13:4171: use of closed network connection
Dec 27 15:22:37.803068377 (527h53m37.674151422s) [noise=true , address=45.91.116.13:4171    , piece_id=YPPWJWZ6H4XMNLGVDHNRQPMZJDXYQRO2EYLNUVCC6WSHIRRJSEWQ, seg_pos={0 0}, upload=463022]: Long tail cancellation due to optimal threshold
Dec 27 15:22:37.803144622 (527h53m37.674227667s) [noise=true , address=64.49.13.169:4708    , piece_id=J7UIJKZWYB2HZ2TCTMPBKNYT65NEYNQADUXQG2J3WTCLD343XLSQ, seg_pos={0 0}, upload=463022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:51138->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:51138->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51138->64.49.13.169:4708: use of closed network connection
Dec 27 15:22:37.803149080 (527h53m37.674232115s) [noise=true , address=64.49.13.169:4708    , piece_id=J7UIJKZWYB2HZ2TCTMPBKNYT65NEYNQADUXQG2J3WTCLD343XLSQ, seg_pos={0 0}, upload=463022]: Long tail cancellation due to optimal threshold
Dec 27 15:22:37.803258279 (527h53m37.674341314s) [noise=true , address=77.90.31.10:4540     , piece_id=MVAPVVOZIC5C6BL5S74PXSKTLTCGLIOSN37M33OUYVPD53NG3GLQ, seg_pos={0 0}, upload=463022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA): protocol: expected piece hash; context canceled; EOF
Dec 27 15:22:37.803263358 (527h53m37.674346393s) [noise=true , address=77.90.31.10:4540     , piece_id=MVAPVVOZIC5C6BL5S74PXSKTLTCGLIOSN37M33OUYVPD53NG3GLQ, seg_pos={0 0}, upload=463022]: Long tail cancellation due to optimal threshold
Dec 27 15:22:37.803277855 (527h53m37.674360901s) [seg_pos={0 0}, upload=463022]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 15:22:37.803341717 (527h53m37.674424762s) [upload=463022]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 15:22:37.803346236 (527h53m37.674429271s) [upload=463022]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 15:22:38.115241263 (527h53m37.986324298s) [upload=463022]: Done uploading segments...
Dec 27 15:22:38.115265930 (527h53m37.986348975s) [upload=463022]: Upload finished. err: <nil>
Duration: 58.657223322s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
21 out of 60 is failed (threshold=3) [X.....X..XX.......X..X.X..X...XXXXXXX..X.X..X.X.....X...X...]
duration: 
5.816987971s
trace: 
https://jaeger.dev.storj.io/trace/3382203165a15a75
FAIL (2025-12-27 15:25:30 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: EFNF7USD6GGEH3V4T42NKM2JPEA3OCYTQOXTNIVRDA6COXPG5JTA) context deadline exceeded; piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: J3OER7CS2XJ6NVGM4X5XM6DWCWP2LZBOSVWD2CQNXABR4TEJXTIA) context deadline exceeded; piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: DBGSITGU4SZAAYFK5KDZ2MXONZ27XTNAV3OPQCQAUHKYBQNPNXSA) context deadline exceeded
======LOGS======
27 15:22:47.299628936 (527h53m47.170711971s) [upload=463024]: Starting upload
Dec 27 15:22:47.299696604 (527h53m47.17077964s) [upload=463024]: Done starting upload
Dec 27 15:22:47.299787547 (527h53m47.170870622s) [upload=463024]: Uploading segments...
Dec 27 15:22:47.299913407 (527h53m47.170996442s) [upload=463024]: Got next segment. Inline: false
Dec 27 15:22:47.299953483 (527h53m47.171036518s) [upload=463024]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 15:22:47.299958212 (527h53m47.171041257s) [upload=463024]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 15:22:47.545973752 (527h53m47.417056787s) [seg_pos={0 0}, upload=463024]: Begin upload segment...
Dec 27 15:22:47.546024249 (527h53m47.417107284s) [seg_pos={0 0}, upload=463024]: Done begin upload segment.
Dec 27 15:22:47.546057682 (527h53m47.417140717s) [noise=true , address=145.79.227.3:4151    , piece_id=627M5Y34ADXGUIAOOGMTG5PAEEZWBT3PAUOZHPI5M23TFTAV2OWQ, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:47.546065778 (527h53m47.417148813s) [noise=true , address=145.79.227.2:4113    , piece_id=YPFORZR5QCXBPQEQZKUVOIUVM4NYKOLU4KDYA7HRJDVMM3KXLKUQ, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:47.546207497 (527h53m47.417290542s) [noise=true , address=145.79.227.24:4581   , piece_id=7HS6Q5Q3ILUXV6BML37VQY2JS4P77CBO6ZTSENMY56MYOYJW6ANQ, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:47.546489023 (527h53m47.417572068s) [noise=true , address=145.79.227.29:4941   , piece_id=VHMADVXOXZOTYAVGE52WNY2IJWQMSQ7LNRSNXOJEZTFBYWLKIRIA, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:47.546754940 (527h53m47.417837975s) [noise=true , address=145.79.227.42:4121   , piece_id=6UQBDGBP6MIDTPV7RYC5NE7JOW6DP74P5QK3BBHVK4MV5UCKGZNA, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:47.547012460 (527h53m47.418095495s) [noise=true , address=145.79.227.47:4265   , piece_id=44SEXUFB2LZHONSA7GDQ63JOYEYETU5ZUMWF45SKXAE5AP7KVWOA, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:47.547340776 (527h53m47.418423811s) [noise=true , address=45.91.116.14:4192    , piece_id=JRQVETWQDYFMV3OR3BOK54FNOPFAZFQJH73WAXLM4KUDVKMKNF2Q, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:47.547610368 (527h53m47.418693413s) [noise=true , address=145.79.227.7:4295    , piece_id=4YVVZDHQNWCCA6XFV7A7ACJL7TUKPFPKKESO5WBIC7XEIDNULCWA, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:47.547903517 (527h53m47.418986552s) [noise=true , address=45.91.116.13:4154    , piece_id=EFNF7USD6GGEH3V4T42NKM2JPEA3OCYTQOXTNIVRDA6COXPG5JTA, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:47.548994664 (527h53m47.4200777s) [noise=true , address=145.79.227.50:4374   , piece_id=PTUKPJS4VWMT7CWQZIFOMI66JNJS6D2J3OMDYUBF4IL6LXMP465Q, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:47.550097284 (527h53m47.421180319s) [noise=true , address=145.79.227.30:4761   , piece_id=AGS3KQJG5YMSHYQA2C4UQRCYYW62QARBO4I2QWCOEBN445HMYQGQ, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:47.551125933 (527h53m47.422208968s) [noise=true , address=45.91.116.11:4119    , piece_id=J3OER7CS2XJ6NVGM4X5XM6DWCWP2LZBOSVWD2CQNXABR4TEJXTIA, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:47.552145084 (527h53m47.423228119s) [noise=true , address=145.79.227.19:4078   , piece_id=YZCJB4ZJN6BNWDDPOTGHEWQNRD5BCGZZACXVDF6XOX4FOLGZGDBQ, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:47.553167881 (527h53m47.424250916s) [noise=true , address=45.91.116.6:4976     , piece_id=ZHQFB66SB4X4AVSYKPFNDZMWURFQRVZT6QG7QC23JCRW22443M7A, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:47.554213733 (527h53m47.425296768s) [noise=true , address=45.91.116.12:4228    , piece_id=DBGSITGU4SZAAYFK5KDZ2MXONZ27XTNAV3OPQCQAUHKYBQNPNXSA, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:47.555339717 (527h53m47.426422752s) [noise=true , address=145.79.227.38:4977   , piece_id=UVSIVE7ZZECB74JPH7RFRAI74SX5HKKQK2ZHDRQ4K5X7JUZOWKWA, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:49.174094752 (527h53m49.045177787s) [upload=463024]: Next returned nil segment
Dec 27 15:22:49.174108467 (527h53m49.045191512s) [upload=463024]: Waiting for error group managing segments...
Dec 27 15:22:49.550005476 (527h53m49.421088511s) [noise=true , address=145.79.227.29:4941   , piece_id=VHMADVXOXZOTYAVGE52WNY2IJWQMSQ7LNRSNXOJEZTFBYWLKIRIA, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil>
Dec 27 15:22:49.551387929 (527h53m49.422470964s) [noise=true , address=145.79.227.7:4295    , piece_id=4YVVZDHQNWCCA6XFV7A7ACJL7TUKPFPKKESO5WBIC7XEIDNULCWA, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil>
Dec 27 15:22:49.554614222 (527h53m49.425697267s) [noise=true , address=145.79.227.19:4078   , piece_id=YZCJB4ZJN6BNWDDPOTGHEWQNRD5BCGZZACXVDF6XOX4FOLGZGDBQ, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil>
Dec 27 15:22:49.556863425 (527h53m49.42794646s) [noise=true , address=145.79.227.42:4121   , piece_id=6UQBDGBP6MIDTPV7RYC5NE7JOW6DP74P5QK3BBHVK4MV5UCKGZNA, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil>
Dec 27 15:22:49.557246234 (527h53m49.428329279s) [noise=true , address=145.79.227.38:4977   , piece_id=UVSIVE7ZZECB74JPH7RFRAI74SX5HKKQK2ZHDRQ4K5X7JUZOWKWA, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil>
Dec 27 15:22:49.558682599 (527h53m49.429765644s) [noise=true , address=145.79.227.2:4113    , piece_id=YPFORZR5QCXBPQEQZKUVOIUVM4NYKOLU4KDYA7HRJDVMM3KXLKUQ, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil>
Dec 27 15:22:49.564448258 (527h53m49.435531303s) [noise=true , address=145.79.227.3:4151    , piece_id=627M5Y34ADXGUIAOOGMTG5PAEEZWBT3PAUOZHPI5M23TFTAV2OWQ, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil>
Dec 27 15:22:49.566142284 (527h53m49.437225329s) [noise=true , address=145.79.227.50:4374   , piece_id=PTUKPJS4VWMT7CWQZIFOMI66JNJS6D2J3OMDYUBF4IL6LXMP465Q, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil>
Dec 27 15:22:49.574698126 (527h53m49.445781171s) [noise=true , address=145.79.227.24:4581   , piece_id=7HS6Q5Q3ILUXV6BML37VQY2JS4P77CBO6ZTSENMY56MYOYJW6ANQ, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil>
Dec 27 15:22:49.586061825 (527h53m49.45714487s) [noise=true , address=145.79.227.47:4265   , piece_id=44SEXUFB2LZHONSA7GDQ63JOYEYETU5ZUMWF45SKXAE5AP7KVWOA, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil>
Dec 27 15:22:49.793614724 (527h53m49.664697759s) [noise=true , address=45.91.116.11:4119    , piece_id=J3OER7CS2XJ6NVGM4X5XM6DWCWP2LZBOSVWD2CQNXABR4TEJXTIA, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil>
Dec 27 15:22:49.874495066 (527h53m49.745578101s) [noise=true , address=45.91.116.12:4228    , piece_id=DBGSITGU4SZAAYFK5KDZ2MXONZ27XTNAV3OPQCQAUHKYBQNPNXSA, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil>
Dec 27 15:22:49.950055185 (527h53m49.821138231s) [noise=true , address=45.91.116.13:4154    , piece_id=EFNF7USD6GGEH3V4T42NKM2JPEA3OCYTQOXTNIVRDA6COXPG5JTA, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil>
Dec 27 15:22:49.950079642 (527h53m49.821162677s) [seg_pos={0 0}, upload=463024]: Segment reached optimal threshold of 13 pieces.
Dec 27 15:22:49.950443906 (527h53m49.821526952s) [noise=true , address=145.79.227.30:4761   , piece_id=AGS3KQJG5YMSHYQA2C4UQRCYYW62QARBO4I2QWCOEBN445HMYQGQ, seg_pos={0 0}, upload=463024]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF
Dec 27 15:22:49.950463724 (527h53m49.821546759s) [noise=true , address=145.79.227.30:4761   , piece_id=AGS3KQJG5YMSHYQA2C4UQRCYYW62QARBO4I2QWCOEBN445HMYQGQ, seg_pos={0 0}, upload=463024]: Long tail cancellation due to optimal threshold
Dec 27 15:22:49.950484183 (527h53m49.821567228s) [noise=true , address=45.91.116.14:4192    , piece_id=JRQVETWQDYFMV3OR3BOK54FNOPFAZFQJH73WAXLM4KUDVKMKNF2Q, seg_pos={0 0}, upload=463024]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: write tcp 188.166.225.170:49708->45.91.116.14:4192: use of closed network connection; write tcp 188.166.225.170:49708->45.91.116.14:4192: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49708->45.91.116.14:4192: use of closed network connection
Dec 27 15:22:49.950523267 (527h53m49.821606302s) [noise=true , address=45.91.116.14:4192    , piece_id=JRQVETWQDYFMV3OR3BOK54FNOPFAZFQJH73WAXLM4KUDVKMKNF2Q, seg_pos={0 0}, upload=463024]: Long tail cancellation due to optimal threshold
Dec 27 15:22:49.950900406 (527h53m49.821983451s) [noise=true , address=45.91.116.6:4976     , piece_id=ZHQFB66SB4X4AVSYKPFNDZMWURFQRVZT6QG7QC23JCRW22443M7A, seg_pos={0 0}, upload=463024]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF
Dec 27 15:22:49.950911727 (527h53m49.821994772s) [noise=true , address=45.91.116.6:4976     , piece_id=ZHQFB66SB4X4AVSYKPFNDZMWURFQRVZT6QG7QC23JCRW22443M7A, seg_pos={0 0}, upload=463024]: Long tail cancellation due to optimal threshold
Dec 27 15:22:49.950943928 (527h53m49.822026993s) [seg_pos={0 0}, upload=463024]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 15:22:49.951104575 (527h53m49.82218762s) [upload=463024]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 15:22:49.951113601 (527h53m49.822196657s) [upload=463024]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 15:22:50.253003756 (527h53m50.124086801s) [upload=463024]: Done uploading segments...
Dec 27 15:22:50.253023765 (527h53m50.1241068s) [upload=463024]: Upload finished. err: <nil>
Duration: 5.81701393s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
36 out of 60 is failed (threshold=6) [XXXX.....XXX..X..XX.XXXX.....XXX.X.XXXX.X.XXXX...XXXX.XXXXX.]
duration: 
5.001639602s
trace: 
https://jaeger.dev.storj.io/trace/2209e5f196d4ee2a
FAIL (2025-12-27 15:24:55 UTC) 
orbiter warning: context deadline exceeded
Duration: 5.001666694s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
21 out of 60 is failed (threshold=3) [.X.....X..XX.......X..X.X..X...XXXXXXX..X.X..X.X.....X...X..]
duration: 
5.993960568s
trace: 
https://jaeger.dev.storj.io/trace/3382203165a15a75
FAIL (2025-12-27 15:24:21 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: EFNF7USD6GGEH3V4T42NKM2JPEA3OCYTQOXTNIVRDA6COXPG5JTA) context deadline exceeded; piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: J3OER7CS2XJ6NVGM4X5XM6DWCWP2LZBOSVWD2CQNXABR4TEJXTIA) context deadline exceeded; piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: DBGSITGU4SZAAYFK5KDZ2MXONZ27XTNAV3OPQCQAUHKYBQNPNXSA) context deadline exceeded
======LOGS======
27 15:22:47.299628936 (527h53m47.170711971s) [upload=463024]: Starting upload
Dec 27 15:22:47.299696604 (527h53m47.17077964s) [upload=463024]: Done starting upload
Dec 27 15:22:47.299787547 (527h53m47.170870622s) [upload=463024]: Uploading segments...
Dec 27 15:22:47.299913407 (527h53m47.170996442s) [upload=463024]: Got next segment. Inline: false
Dec 27 15:22:47.299953483 (527h53m47.171036518s) [upload=463024]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 15:22:47.299958212 (527h53m47.171041257s) [upload=463024]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 15:22:47.545973752 (527h53m47.417056787s) [seg_pos={0 0}, upload=463024]: Begin upload segment...
Dec 27 15:22:47.546024249 (527h53m47.417107284s) [seg_pos={0 0}, upload=463024]: Done begin upload segment.
Dec 27 15:22:47.546057682 (527h53m47.417140717s) [noise=true , address=145.79.227.3:4151    , piece_id=627M5Y34ADXGUIAOOGMTG5PAEEZWBT3PAUOZHPI5M23TFTAV2OWQ, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:47.546065778 (527h53m47.417148813s) [noise=true , address=145.79.227.2:4113    , piece_id=YPFORZR5QCXBPQEQZKUVOIUVM4NYKOLU4KDYA7HRJDVMM3KXLKUQ, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:47.546207497 (527h53m47.417290542s) [noise=true , address=145.79.227.24:4581   , piece_id=7HS6Q5Q3ILUXV6BML37VQY2JS4P77CBO6ZTSENMY56MYOYJW6ANQ, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:47.546489023 (527h53m47.417572068s) [noise=true , address=145.79.227.29:4941   , piece_id=VHMADVXOXZOTYAVGE52WNY2IJWQMSQ7LNRSNXOJEZTFBYWLKIRIA, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:47.546754940 (527h53m47.417837975s) [noise=true , address=145.79.227.42:4121   , piece_id=6UQBDGBP6MIDTPV7RYC5NE7JOW6DP74P5QK3BBHVK4MV5UCKGZNA, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:47.547012460 (527h53m47.418095495s) [noise=true , address=145.79.227.47:4265   , piece_id=44SEXUFB2LZHONSA7GDQ63JOYEYETU5ZUMWF45SKXAE5AP7KVWOA, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:47.547340776 (527h53m47.418423811s) [noise=true , address=45.91.116.14:4192    , piece_id=JRQVETWQDYFMV3OR3BOK54FNOPFAZFQJH73WAXLM4KUDVKMKNF2Q, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:47.547610368 (527h53m47.418693413s) [noise=true , address=145.79.227.7:4295    , piece_id=4YVVZDHQNWCCA6XFV7A7ACJL7TUKPFPKKESO5WBIC7XEIDNULCWA, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:47.547903517 (527h53m47.418986552s) [noise=true , address=45.91.116.13:4154    , piece_id=EFNF7USD6GGEH3V4T42NKM2JPEA3OCYTQOXTNIVRDA6COXPG5JTA, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:47.548994664 (527h53m47.4200777s) [noise=true , address=145.79.227.50:4374   , piece_id=PTUKPJS4VWMT7CWQZIFOMI66JNJS6D2J3OMDYUBF4IL6LXMP465Q, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:47.550097284 (527h53m47.421180319s) [noise=true , address=145.79.227.30:4761   , piece_id=AGS3KQJG5YMSHYQA2C4UQRCYYW62QARBO4I2QWCOEBN445HMYQGQ, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:47.551125933 (527h53m47.422208968s) [noise=true , address=45.91.116.11:4119    , piece_id=J3OER7CS2XJ6NVGM4X5XM6DWCWP2LZBOSVWD2CQNXABR4TEJXTIA, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:47.552145084 (527h53m47.423228119s) [noise=true , address=145.79.227.19:4078   , piece_id=YZCJB4ZJN6BNWDDPOTGHEWQNRD5BCGZZACXVDF6XOX4FOLGZGDBQ, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:47.553167881 (527h53m47.424250916s) [noise=true , address=45.91.116.6:4976     , piece_id=ZHQFB66SB4X4AVSYKPFNDZMWURFQRVZT6QG7QC23JCRW22443M7A, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:47.554213733 (527h53m47.425296768s) [noise=true , address=45.91.116.12:4228    , piece_id=DBGSITGU4SZAAYFK5KDZ2MXONZ27XTNAV3OPQCQAUHKYBQNPNXSA, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:47.555339717 (527h53m47.426422752s) [noise=true , address=145.79.227.38:4977   , piece_id=UVSIVE7ZZECB74JPH7RFRAI74SX5HKKQK2ZHDRQ4K5X7JUZOWKWA, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:49.174094752 (527h53m49.045177787s) [upload=463024]: Next returned nil segment
Dec 27 15:22:49.174108467 (527h53m49.045191512s) [upload=463024]: Waiting for error group managing segments...
Dec 27 15:22:49.550005476 (527h53m49.421088511s) [noise=true , address=145.79.227.29:4941   , piece_id=VHMADVXOXZOTYAVGE52WNY2IJWQMSQ7LNRSNXOJEZTFBYWLKIRIA, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil>
Dec 27 15:22:49.551387929 (527h53m49.422470964s) [noise=true , address=145.79.227.7:4295    , piece_id=4YVVZDHQNWCCA6XFV7A7ACJL7TUKPFPKKESO5WBIC7XEIDNULCWA, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil>
Dec 27 15:22:49.554614222 (527h53m49.425697267s) [noise=true , address=145.79.227.19:4078   , piece_id=YZCJB4ZJN6BNWDDPOTGHEWQNRD5BCGZZACXVDF6XOX4FOLGZGDBQ, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil>
Dec 27 15:22:49.556863425 (527h53m49.42794646s) [noise=true , address=145.79.227.42:4121   , piece_id=6UQBDGBP6MIDTPV7RYC5NE7JOW6DP74P5QK3BBHVK4MV5UCKGZNA, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil>
Dec 27 15:22:49.557246234 (527h53m49.428329279s) [noise=true , address=145.79.227.38:4977   , piece_id=UVSIVE7ZZECB74JPH7RFRAI74SX5HKKQK2ZHDRQ4K5X7JUZOWKWA, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil>
Dec 27 15:22:49.558682599 (527h53m49.429765644s) [noise=true , address=145.79.227.2:4113    , piece_id=YPFORZR5QCXBPQEQZKUVOIUVM4NYKOLU4KDYA7HRJDVMM3KXLKUQ, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil>
Dec 27 15:22:49.564448258 (527h53m49.435531303s) [noise=true , address=145.79.227.3:4151    , piece_id=627M5Y34ADXGUIAOOGMTG5PAEEZWBT3PAUOZHPI5M23TFTAV2OWQ, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil>
Dec 27 15:22:49.566142284 (527h53m49.437225329s) [noise=true , address=145.79.227.50:4374   , piece_id=PTUKPJS4VWMT7CWQZIFOMI66JNJS6D2J3OMDYUBF4IL6LXMP465Q, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil>
Dec 27 15:22:49.574698126 (527h53m49.445781171s) [noise=true , address=145.79.227.24:4581   , piece_id=7HS6Q5Q3ILUXV6BML37VQY2JS4P77CBO6ZTSENMY56MYOYJW6ANQ, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil>
Dec 27 15:22:49.586061825 (527h53m49.45714487s) [noise=true , address=145.79.227.47:4265   , piece_id=44SEXUFB2LZHONSA7GDQ63JOYEYETU5ZUMWF45SKXAE5AP7KVWOA, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil>
Dec 27 15:22:49.793614724 (527h53m49.664697759s) [noise=true , address=45.91.116.11:4119    , piece_id=J3OER7CS2XJ6NVGM4X5XM6DWCWP2LZBOSVWD2CQNXABR4TEJXTIA, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil>
Dec 27 15:22:49.874495066 (527h53m49.745578101s) [noise=true , address=45.91.116.12:4228    , piece_id=DBGSITGU4SZAAYFK5KDZ2MXONZ27XTNAV3OPQCQAUHKYBQNPNXSA, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil>
Dec 27 15:22:49.950055185 (527h53m49.821138231s) [noise=true , address=45.91.116.13:4154    , piece_id=EFNF7USD6GGEH3V4T42NKM2JPEA3OCYTQOXTNIVRDA6COXPG5JTA, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil>
Dec 27 15:22:49.950079642 (527h53m49.821162677s) [seg_pos={0 0}, upload=463024]: Segment reached optimal threshold of 13 pieces.
Dec 27 15:22:49.950443906 (527h53m49.821526952s) [noise=true , address=145.79.227.30:4761   , piece_id=AGS3KQJG5YMSHYQA2C4UQRCYYW62QARBO4I2QWCOEBN445HMYQGQ, seg_pos={0 0}, upload=463024]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF
Dec 27 15:22:49.950463724 (527h53m49.821546759s) [noise=true , address=145.79.227.30:4761   , piece_id=AGS3KQJG5YMSHYQA2C4UQRCYYW62QARBO4I2QWCOEBN445HMYQGQ, seg_pos={0 0}, upload=463024]: Long tail cancellation due to optimal threshold
Dec 27 15:22:49.950484183 (527h53m49.821567228s) [noise=true , address=45.91.116.14:4192    , piece_id=JRQVETWQDYFMV3OR3BOK54FNOPFAZFQJH73WAXLM4KUDVKMKNF2Q, seg_pos={0 0}, upload=463024]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: write tcp 188.166.225.170:49708->45.91.116.14:4192: use of closed network connection; write tcp 188.166.225.170:49708->45.91.116.14:4192: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49708->45.91.116.14:4192: use of closed network connection
Dec 27 15:22:49.950523267 (527h53m49.821606302s) [noise=true , address=45.91.116.14:4192    , piece_id=JRQVETWQDYFMV3OR3BOK54FNOPFAZFQJH73WAXLM4KUDVKMKNF2Q, seg_pos={0 0}, upload=463024]: Long tail cancellation due to optimal threshold
Dec 27 15:22:49.950900406 (527h53m49.821983451s) [noise=true , address=45.91.116.6:4976     , piece_id=ZHQFB66SB4X4AVSYKPFNDZMWURFQRVZT6QG7QC23JCRW22443M7A, seg_pos={0 0}, upload=463024]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF
Dec 27 15:22:49.950911727 (527h53m49.821994772s) [noise=true , address=45.91.116.6:4976     , piece_id=ZHQFB66SB4X4AVSYKPFNDZMWURFQRVZT6QG7QC23JCRW22443M7A, seg_pos={0 0}, upload=463024]: Long tail cancellation due to optimal threshold
Dec 27 15:22:49.950943928 (527h53m49.822026993s) [seg_pos={0 0}, upload=463024]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 15:22:49.951104575 (527h53m49.82218762s) [upload=463024]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 15:22:49.951113601 (527h53m49.822196657s) [upload=463024]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 15:22:50.253003756 (527h53m50.124086801s) [upload=463024]: Done uploading segments...
Dec 27 15:22:50.253023765 (527h53m50.1241068s) [upload=463024]: Upload finished. err: <nil>
Duration: 5.993991897s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
6 out of 60 is failed (threshold=3) [...........X..............X....X...................X.....XX.]
duration: 
39.592708847s
trace: 
https://jaeger.dev.storj.io/trace/1cf86c3db9e735d0
FAIL (2025-12-27 15:24:13 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
PBKNYT65NEYNQADUXQG2J3WTCLD343XLSQ, seg_pos={0 0}, upload=463022]: Uploading piece...
Dec 27 15:22:34.632070090 (527h53m34.503153125s) [noise=true , address=77.90.31.11:4565     , piece_id=B4VPI2I53PUUOLNS6C6GFKY3ZYDQ5RRQZCKC4QVLSVJI777RR6KQ, seg_pos={0 0}, upload=463022]: Uploading piece...
Dec 27 15:22:34.632523263 (527h53m34.503606298s) [noise=true , address=77.90.31.10:4540     , piece_id=MVAPVVOZIC5C6BL5S74PXSKTLTCGLIOSN37M33OUYVPD53NG3GLQ, seg_pos={0 0}, upload=463022]: Uploading piece...
Dec 27 15:22:34.632676374 (527h53m34.503759419s) [noise=true , address=77.90.31.13:4659     , piece_id=STKDI6VSN67XV43CB4ASCJTPDU6NA75REYKEE6D5K3EFM5FCDMYA, seg_pos={0 0}, upload=463022]: Uploading piece...
Dec 27 15:22:34.632931610 (527h53m34.504014655s) [noise=true , address=145.79.227.52:4466   , piece_id=CZYMGB53RT6FROGYMOYOPOQJ5CSZDW4YYAVDQWYWVLIDINN3LJVA, seg_pos={0 0}, upload=463022]: Uploading piece...
Dec 27 15:22:34.633357982 (527h53m34.504441017s) [noise=true , address=145.79.227.13:4816   , piece_id=LTSC26UNXCIHK4PDGYKE7GZCQ7MXCWWRGBLIHI6FPO6ISE4BPH7A, seg_pos={0 0}, upload=463022]: Uploading piece...
Dec 27 15:22:34.633490544 (527h53m34.504573579s) [noise=true , address=145.79.227.44:4218   , piece_id=2KHGZL76RA6CMI33PTF5WZN5TRIK3LRL6XT5IJP6LNJP5ZRVMHXA, seg_pos={0 0}, upload=463022]: Uploading piece...
Dec 27 15:22:34.633891278 (527h53m34.504974313s) [noise=true , address=77.90.31.18:4820     , piece_id=TSKDIH6HQUUQRTQTJDXGBX3OALVLYIUKR5QCPJFPFXTCASNN5JMQ, seg_pos={0 0}, upload=463022]: Uploading piece...
Dec 27 15:22:34.634043096 (527h53m34.505126141s) [noise=true , address=145.79.227.17:4997   , piece_id=VX4S5TXLHFNBRE7OYPDZAGYDGN2AD34ZL5JQ2YUSBV656RO6A5UA, seg_pos={0 0}, upload=463022]: Uploading piece...
Dec 27 15:22:34.634325093 (527h53m34.505408128s) [noise=true , address=77.90.31.16:4739     , piece_id=M7IJW7WX2FNPFWNJPBUAHQQOKPFXDSWAWMC2IHUYQZ4JPXF3YZXQ, seg_pos={0 0}, upload=463022]: Uploading piece...
Dec 27 15:22:34.634722230 (527h53m34.505805265s) [noise=true , address=77.90.31.15:4716     , piece_id=MLOLX2R56NCS4KT3FLJJNNPDAMUA6S5NSFCX57W7V6GMIZ4457DA, seg_pos={0 0}, upload=463022]: Uploading piece...
Dec 27 15:22:34.635114116 (527h53m34.506197151s) [noise=true , address=45.91.116.13:4171    , piece_id=YPPWJWZ6H4XMNLGVDHNRQPMZJDXYQRO2EYLNUVCC6WSHIRRJSEWQ, seg_pos={0 0}, upload=463022]: Uploading piece...
Dec 27 15:22:34.635719949 (527h53m34.506802984s) [noise=true , address=77.90.31.12:4618     , piece_id=ZII3XYSUJ7WTNNZA77GO54GFVGVHOGUUJOFZG2OJDTDDP6CAPC3A, seg_pos={0 0}, upload=463022]: Uploading piece...
Dec 27 15:22:34.636184344 (527h53m34.507267379s) [noise=true , address=145.79.227.30:4775   , piece_id=RG6IZXLWJSMKNHY2VR7VHDATQ2LAHIEGYF6WDQIZXUTGETQKS3DQ, seg_pos={0 0}, upload=463022]: Uploading piece...
Dec 27 15:22:36.104663675 (527h53m35.97574671s) [upload=463022]: Next returned nil segment
Dec 27 15:22:36.104675988 (527h53m35.975759023s) [upload=463022]: Waiting for error group managing segments...
Dec 27 15:22:36.505615923 (527h53m36.376698958s) [noise=true , address=145.79.227.17:4997   , piece_id=VX4S5TXLHFNBRE7OYPDZAGYDGN2AD34ZL5JQ2YUSBV656RO6A5UA, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil>
Dec 27 15:22:36.509752460 (527h53m36.380835505s) [noise=true , address=145.79.227.13:4816   , piece_id=LTSC26UNXCIHK4PDGYKE7GZCQ7MXCWWRGBLIHI6FPO6ISE4BPH7A, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil>
Dec 27 15:22:36.517493871 (527h53m36.388576906s) [noise=true , address=145.79.227.14:4865   , piece_id=XOXM2D6IS4SG6DN24DEGIPV7NK5FWDI7QVVJK7L3QB25AUPRNZIQ, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil>
Dec 27 15:22:36.623194766 (527h53m36.494277801s) [noise=true , address=145.79.227.52:4466   , piece_id=CZYMGB53RT6FROGYMOYOPOQJ5CSZDW4YYAVDQWYWVLIDINN3LJVA, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil>
Dec 27 15:22:36.749453706 (527h53m36.620536741s) [noise=true , address=45.91.116.11:4145    , piece_id=77YGIECBFASFKBHIQO333RHATPHZCA7JQW2WPNSGEGM24TTWDVGA, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil>
Dec 27 15:22:36.780705355 (527h53m36.6517884s) [noise=true , address=77.90.31.18:4820     , piece_id=TSKDIH6HQUUQRTQTJDXGBX3OALVLYIUKR5QCPJFPFXTCASNN5JMQ, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil>
Dec 27 15:22:36.803640327 (527h53m36.674723362s) [noise=true , address=77.90.31.13:4659     , piece_id=STKDI6VSN67XV43CB4ASCJTPDU6NA75REYKEE6D5K3EFM5FCDMYA, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil>
Dec 27 15:22:36.841101831 (527h53m36.712184876s) [noise=true , address=77.90.31.16:4739     , piece_id=M7IJW7WX2FNPFWNJPBUAHQQOKPFXDSWAWMC2IHUYQZ4JPXF3YZXQ, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil>
Dec 27 15:22:37.481224175 (527h53m37.35230722s) [noise=true , address=77.90.31.15:4716     , piece_id=MLOLX2R56NCS4KT3FLJJNNPDAMUA6S5NSFCX57W7V6GMIZ4457DA, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil>
Dec 27 15:22:37.504901952 (527h53m37.375984997s) [noise=true , address=145.79.227.30:4775   , piece_id=RG6IZXLWJSMKNHY2VR7VHDATQ2LAHIEGYF6WDQIZXUTGETQKS3DQ, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil>
Dec 27 15:22:37.553631302 (527h53m37.424714337s) [noise=true , address=145.79.227.44:4218   , piece_id=2KHGZL76RA6CMI33PTF5WZN5TRIK3LRL6XT5IJP6LNJP5ZRVMHXA, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil>
Dec 27 15:22:37.596982441 (527h53m37.468065476s) [noise=true , address=145.79.227.40:4063   , piece_id=SRV3KMN3XXUOJA2GLEJ27UKWRANPEKCT3NC7BBDAVMTCI6FZCNPQ, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil>
Dec 27 15:22:37.792107735 (527h53m37.663190771s) [noise=true , address=77.90.31.17:4802     , piece_id=TFQZEPPQN5RYDUNTGLILJUHZ3QLTU7BT3QP67SS5Y4ZRUKPG6A5A, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil>
Dec 27 15:22:37.802639390 (527h53m37.673722425s) [noise=true , address=77.90.31.12:4618     , piece_id=ZII3XYSUJ7WTNNZA77GO54GFVGVHOGUUJOFZG2OJDTDDP6CAPC3A, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil>
Dec 27 15:22:37.802661622 (527h53m37.673744667s) [seg_pos={0 0}, upload=463022]: Segment reached optimal threshold of 14 pieces.
Dec 27 15:22:37.802830423 (527h53m37.673913468s) [noise=true , address=77.90.31.11:4565     , piece_id=B4VPI2I53PUUOLNS6C6GFKY3ZYDQ5RRQZCKC4QVLSVJI777RR6KQ, seg_pos={0 0}, upload=463022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xnwFcUEdmpLjmx8NXZZcpiVQwTySXDRYbchuz4oREBq1yP6wh): protocol: write tcp 188.166.225.170:60558->77.90.31.11:4565: use of closed network connection; write tcp 188.166.225.170:60558->77.90.31.11:4565: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60558->77.90.31.11:4565: use of closed network connection
Dec 27 15:22:37.802837557 (527h53m37.673920592s) [noise=true , address=77.90.31.11:4565     , piece_id=B4VPI2I53PUUOLNS6C6GFKY3ZYDQ5RRQZCKC4QVLSVJI777RR6KQ, seg_pos={0 0}, upload=463022]: Long tail cancellation due to optimal threshold
Dec 27 15:22:37.803060181 (527h53m37.674143216s) [noise=true , address=45.91.116.13:4171    , piece_id=YPPWJWZ6H4XMNLGVDHNRQPMZJDXYQRO2EYLNUVCC6WSHIRRJSEWQ, seg_pos={0 0}, upload=463022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122dmyMaZDYbCvoYvCc9m8Vf5dA8aJiw9qPh1G5AQWKnBig4mVg): protocol: write tcp 188.166.225.170:35780->45.91.116.13:4171: use of closed network connection; write tcp 188.166.225.170:35780->45.91.116.13:4171: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35780->45.91.116.13:4171: use of closed network connection
Dec 27 15:22:37.803068377 (527h53m37.674151422s) [noise=true , address=45.91.116.13:4171    , piece_id=YPPWJWZ6H4XMNLGVDHNRQPMZJDXYQRO2EYLNUVCC6WSHIRRJSEWQ, seg_pos={0 0}, upload=463022]: Long tail cancellation due to optimal threshold
Dec 27 15:22:37.803144622 (527h53m37.674227667s) [noise=true , address=64.49.13.169:4708    , piece_id=J7UIJKZWYB2HZ2TCTMPBKNYT65NEYNQADUXQG2J3WTCLD343XLSQ, seg_pos={0 0}, upload=463022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:51138->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:51138->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51138->64.49.13.169:4708: use of closed network connection
Dec 27 15:22:37.803149080 (527h53m37.674232115s) [noise=true , address=64.49.13.169:4708    , piece_id=J7UIJKZWYB2HZ2TCTMPBKNYT65NEYNQADUXQG2J3WTCLD343XLSQ, seg_pos={0 0}, upload=463022]: Long tail cancellation due to optimal threshold
Dec 27 15:22:37.803258279 (527h53m37.674341314s) [noise=true , address=77.90.31.10:4540     , piece_id=MVAPVVOZIC5C6BL5S74PXSKTLTCGLIOSN37M33OUYVPD53NG3GLQ, seg_pos={0 0}, upload=463022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA): protocol: expected piece hash; context canceled; EOF
Dec 27 15:22:37.803263358 (527h53m37.674346393s) [noise=true , address=77.90.31.10:4540     , piece_id=MVAPVVOZIC5C6BL5S74PXSKTLTCGLIOSN37M33OUYVPD53NG3GLQ, seg_pos={0 0}, upload=463022]: Long tail cancellation due to optimal threshold
Dec 27 15:22:37.803277855 (527h53m37.674360901s) [seg_pos={0 0}, upload=463022]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 15:22:37.803341717 (527h53m37.674424762s) [upload=463022]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 15:22:37.803346236 (527h53m37.674429271s) [upload=463022]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 15:22:38.115241263 (527h53m37.986324298s) [upload=463022]: Done uploading segments...
Dec 27 15:22:38.115265930 (527h53m37.986348975s) [upload=463022]: Upload finished. err: <nil>
Duration: 39.592726571s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
37 out of 60 is failed (threshold=6) [XXXXX.....XXX..X..XX.XXXX.....XXX.X.XXXX.X.XXXX...XXXX.XXXXX]
duration: 
5.00101679s
trace: 
https://jaeger.dev.storj.io/trace/2209e5f196d4ee2a
FAIL (2025-12-27 15:24:12 UTC) 
orbiter warning: context deadline exceeded
Duration: 5.001039853s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
21 out of 60 is failed (threshold=3) [..X.....X..XX.......X..X.X..X...XXXXXXX..X.X..X.X.....X...X.]
duration: 
6.404026519s
trace: 
https://jaeger.dev.storj.io/trace/3382203165a15a75
FAIL (2025-12-27 15:23:36 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: EFNF7USD6GGEH3V4T42NKM2JPEA3OCYTQOXTNIVRDA6COXPG5JTA) context deadline exceeded; piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: J3OER7CS2XJ6NVGM4X5XM6DWCWP2LZBOSVWD2CQNXABR4TEJXTIA) context deadline exceeded; piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: DBGSITGU4SZAAYFK5KDZ2MXONZ27XTNAV3OPQCQAUHKYBQNPNXSA) context deadline exceeded
======LOGS======
27 15:22:47.299628936 (527h53m47.170711971s) [upload=463024]: Starting upload
Dec 27 15:22:47.299696604 (527h53m47.17077964s) [upload=463024]: Done starting upload
Dec 27 15:22:47.299787547 (527h53m47.170870622s) [upload=463024]: Uploading segments...
Dec 27 15:22:47.299913407 (527h53m47.170996442s) [upload=463024]: Got next segment. Inline: false
Dec 27 15:22:47.299953483 (527h53m47.171036518s) [upload=463024]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 15:22:47.299958212 (527h53m47.171041257s) [upload=463024]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 15:22:47.545973752 (527h53m47.417056787s) [seg_pos={0 0}, upload=463024]: Begin upload segment...
Dec 27 15:22:47.546024249 (527h53m47.417107284s) [seg_pos={0 0}, upload=463024]: Done begin upload segment.
Dec 27 15:22:47.546057682 (527h53m47.417140717s) [noise=true , address=145.79.227.3:4151    , piece_id=627M5Y34ADXGUIAOOGMTG5PAEEZWBT3PAUOZHPI5M23TFTAV2OWQ, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:47.546065778 (527h53m47.417148813s) [noise=true , address=145.79.227.2:4113    , piece_id=YPFORZR5QCXBPQEQZKUVOIUVM4NYKOLU4KDYA7HRJDVMM3KXLKUQ, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:47.546207497 (527h53m47.417290542s) [noise=true , address=145.79.227.24:4581   , piece_id=7HS6Q5Q3ILUXV6BML37VQY2JS4P77CBO6ZTSENMY56MYOYJW6ANQ, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:47.546489023 (527h53m47.417572068s) [noise=true , address=145.79.227.29:4941   , piece_id=VHMADVXOXZOTYAVGE52WNY2IJWQMSQ7LNRSNXOJEZTFBYWLKIRIA, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:47.546754940 (527h53m47.417837975s) [noise=true , address=145.79.227.42:4121   , piece_id=6UQBDGBP6MIDTPV7RYC5NE7JOW6DP74P5QK3BBHVK4MV5UCKGZNA, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:47.547012460 (527h53m47.418095495s) [noise=true , address=145.79.227.47:4265   , piece_id=44SEXUFB2LZHONSA7GDQ63JOYEYETU5ZUMWF45SKXAE5AP7KVWOA, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:47.547340776 (527h53m47.418423811s) [noise=true , address=45.91.116.14:4192    , piece_id=JRQVETWQDYFMV3OR3BOK54FNOPFAZFQJH73WAXLM4KUDVKMKNF2Q, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:47.547610368 (527h53m47.418693413s) [noise=true , address=145.79.227.7:4295    , piece_id=4YVVZDHQNWCCA6XFV7A7ACJL7TUKPFPKKESO5WBIC7XEIDNULCWA, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:47.547903517 (527h53m47.418986552s) [noise=true , address=45.91.116.13:4154    , piece_id=EFNF7USD6GGEH3V4T42NKM2JPEA3OCYTQOXTNIVRDA6COXPG5JTA, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:47.548994664 (527h53m47.4200777s) [noise=true , address=145.79.227.50:4374   , piece_id=PTUKPJS4VWMT7CWQZIFOMI66JNJS6D2J3OMDYUBF4IL6LXMP465Q, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:47.550097284 (527h53m47.421180319s) [noise=true , address=145.79.227.30:4761   , piece_id=AGS3KQJG5YMSHYQA2C4UQRCYYW62QARBO4I2QWCOEBN445HMYQGQ, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:47.551125933 (527h53m47.422208968s) [noise=true , address=45.91.116.11:4119    , piece_id=J3OER7CS2XJ6NVGM4X5XM6DWCWP2LZBOSVWD2CQNXABR4TEJXTIA, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:47.552145084 (527h53m47.423228119s) [noise=true , address=145.79.227.19:4078   , piece_id=YZCJB4ZJN6BNWDDPOTGHEWQNRD5BCGZZACXVDF6XOX4FOLGZGDBQ, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:47.553167881 (527h53m47.424250916s) [noise=true , address=45.91.116.6:4976     , piece_id=ZHQFB66SB4X4AVSYKPFNDZMWURFQRVZT6QG7QC23JCRW22443M7A, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:47.554213733 (527h53m47.425296768s) [noise=true , address=45.91.116.12:4228    , piece_id=DBGSITGU4SZAAYFK5KDZ2MXONZ27XTNAV3OPQCQAUHKYBQNPNXSA, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:47.555339717 (527h53m47.426422752s) [noise=true , address=145.79.227.38:4977   , piece_id=UVSIVE7ZZECB74JPH7RFRAI74SX5HKKQK2ZHDRQ4K5X7JUZOWKWA, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:49.174094752 (527h53m49.045177787s) [upload=463024]: Next returned nil segment
Dec 27 15:22:49.174108467 (527h53m49.045191512s) [upload=463024]: Waiting for error group managing segments...
Dec 27 15:22:49.550005476 (527h53m49.421088511s) [noise=true , address=145.79.227.29:4941   , piece_id=VHMADVXOXZOTYAVGE52WNY2IJWQMSQ7LNRSNXOJEZTFBYWLKIRIA, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil>
Dec 27 15:22:49.551387929 (527h53m49.422470964s) [noise=true , address=145.79.227.7:4295    , piece_id=4YVVZDHQNWCCA6XFV7A7ACJL7TUKPFPKKESO5WBIC7XEIDNULCWA, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil>
Dec 27 15:22:49.554614222 (527h53m49.425697267s) [noise=true , address=145.79.227.19:4078   , piece_id=YZCJB4ZJN6BNWDDPOTGHEWQNRD5BCGZZACXVDF6XOX4FOLGZGDBQ, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil>
Dec 27 15:22:49.556863425 (527h53m49.42794646s) [noise=true , address=145.79.227.42:4121   , piece_id=6UQBDGBP6MIDTPV7RYC5NE7JOW6DP74P5QK3BBHVK4MV5UCKGZNA, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil>
Dec 27 15:22:49.557246234 (527h53m49.428329279s) [noise=true , address=145.79.227.38:4977   , piece_id=UVSIVE7ZZECB74JPH7RFRAI74SX5HKKQK2ZHDRQ4K5X7JUZOWKWA, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil>
Dec 27 15:22:49.558682599 (527h53m49.429765644s) [noise=true , address=145.79.227.2:4113    , piece_id=YPFORZR5QCXBPQEQZKUVOIUVM4NYKOLU4KDYA7HRJDVMM3KXLKUQ, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil>
Dec 27 15:22:49.564448258 (527h53m49.435531303s) [noise=true , address=145.79.227.3:4151    , piece_id=627M5Y34ADXGUIAOOGMTG5PAEEZWBT3PAUOZHPI5M23TFTAV2OWQ, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil>
Dec 27 15:22:49.566142284 (527h53m49.437225329s) [noise=true , address=145.79.227.50:4374   , piece_id=PTUKPJS4VWMT7CWQZIFOMI66JNJS6D2J3OMDYUBF4IL6LXMP465Q, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil>
Dec 27 15:22:49.574698126 (527h53m49.445781171s) [noise=true , address=145.79.227.24:4581   , piece_id=7HS6Q5Q3ILUXV6BML37VQY2JS4P77CBO6ZTSENMY56MYOYJW6ANQ, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil>
Dec 27 15:22:49.586061825 (527h53m49.45714487s) [noise=true , address=145.79.227.47:4265   , piece_id=44SEXUFB2LZHONSA7GDQ63JOYEYETU5ZUMWF45SKXAE5AP7KVWOA, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil>
Dec 27 15:22:49.793614724 (527h53m49.664697759s) [noise=true , address=45.91.116.11:4119    , piece_id=J3OER7CS2XJ6NVGM4X5XM6DWCWP2LZBOSVWD2CQNXABR4TEJXTIA, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil>
Dec 27 15:22:49.874495066 (527h53m49.745578101s) [noise=true , address=45.91.116.12:4228    , piece_id=DBGSITGU4SZAAYFK5KDZ2MXONZ27XTNAV3OPQCQAUHKYBQNPNXSA, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil>
Dec 27 15:22:49.950055185 (527h53m49.821138231s) [noise=true , address=45.91.116.13:4154    , piece_id=EFNF7USD6GGEH3V4T42NKM2JPEA3OCYTQOXTNIVRDA6COXPG5JTA, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil>
Dec 27 15:22:49.950079642 (527h53m49.821162677s) [seg_pos={0 0}, upload=463024]: Segment reached optimal threshold of 13 pieces.
Dec 27 15:22:49.950443906 (527h53m49.821526952s) [noise=true , address=145.79.227.30:4761   , piece_id=AGS3KQJG5YMSHYQA2C4UQRCYYW62QARBO4I2QWCOEBN445HMYQGQ, seg_pos={0 0}, upload=463024]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF
Dec 27 15:22:49.950463724 (527h53m49.821546759s) [noise=true , address=145.79.227.30:4761   , piece_id=AGS3KQJG5YMSHYQA2C4UQRCYYW62QARBO4I2QWCOEBN445HMYQGQ, seg_pos={0 0}, upload=463024]: Long tail cancellation due to optimal threshold
Dec 27 15:22:49.950484183 (527h53m49.821567228s) [noise=true , address=45.91.116.14:4192    , piece_id=JRQVETWQDYFMV3OR3BOK54FNOPFAZFQJH73WAXLM4KUDVKMKNF2Q, seg_pos={0 0}, upload=463024]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: write tcp 188.166.225.170:49708->45.91.116.14:4192: use of closed network connection; write tcp 188.166.225.170:49708->45.91.116.14:4192: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49708->45.91.116.14:4192: use of closed network connection
Dec 27 15:22:49.950523267 (527h53m49.821606302s) [noise=true , address=45.91.116.14:4192    , piece_id=JRQVETWQDYFMV3OR3BOK54FNOPFAZFQJH73WAXLM4KUDVKMKNF2Q, seg_pos={0 0}, upload=463024]: Long tail cancellation due to optimal threshold
Dec 27 15:22:49.950900406 (527h53m49.821983451s) [noise=true , address=45.91.116.6:4976     , piece_id=ZHQFB66SB4X4AVSYKPFNDZMWURFQRVZT6QG7QC23JCRW22443M7A, seg_pos={0 0}, upload=463024]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF
Dec 27 15:22:49.950911727 (527h53m49.821994772s) [noise=true , address=45.91.116.6:4976     , piece_id=ZHQFB66SB4X4AVSYKPFNDZMWURFQRVZT6QG7QC23JCRW22443M7A, seg_pos={0 0}, upload=463024]: Long tail cancellation due to optimal threshold
Dec 27 15:22:49.950943928 (527h53m49.822026993s) [seg_pos={0 0}, upload=463024]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 15:22:49.951104575 (527h53m49.82218762s) [upload=463024]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 15:22:49.951113601 (527h53m49.822196657s) [upload=463024]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 15:22:50.253003756 (527h53m50.124086801s) [upload=463024]: Done uploading segments...
Dec 27 15:22:50.253023765 (527h53m50.1241068s) [upload=463024]: Upload finished. err: <nil>
Duration: 6.404063189s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
7 out of 60 is failed (threshold=3) [X...........X..............X....X...................X.....XX]
duration: 
1m0.002028377s
trace: 
https://jaeger.dev.storj.io/trace/1cf86c3db9e735d0
FAIL (2025-12-27 15:23:33 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
PBKNYT65NEYNQADUXQG2J3WTCLD343XLSQ, seg_pos={0 0}, upload=463022]: Uploading piece...
Dec 27 15:22:34.632070090 (527h53m34.503153125s) [noise=true , address=77.90.31.11:4565     , piece_id=B4VPI2I53PUUOLNS6C6GFKY3ZYDQ5RRQZCKC4QVLSVJI777RR6KQ, seg_pos={0 0}, upload=463022]: Uploading piece...
Dec 27 15:22:34.632523263 (527h53m34.503606298s) [noise=true , address=77.90.31.10:4540     , piece_id=MVAPVVOZIC5C6BL5S74PXSKTLTCGLIOSN37M33OUYVPD53NG3GLQ, seg_pos={0 0}, upload=463022]: Uploading piece...
Dec 27 15:22:34.632676374 (527h53m34.503759419s) [noise=true , address=77.90.31.13:4659     , piece_id=STKDI6VSN67XV43CB4ASCJTPDU6NA75REYKEE6D5K3EFM5FCDMYA, seg_pos={0 0}, upload=463022]: Uploading piece...
Dec 27 15:22:34.632931610 (527h53m34.504014655s) [noise=true , address=145.79.227.52:4466   , piece_id=CZYMGB53RT6FROGYMOYOPOQJ5CSZDW4YYAVDQWYWVLIDINN3LJVA, seg_pos={0 0}, upload=463022]: Uploading piece...
Dec 27 15:22:34.633357982 (527h53m34.504441017s) [noise=true , address=145.79.227.13:4816   , piece_id=LTSC26UNXCIHK4PDGYKE7GZCQ7MXCWWRGBLIHI6FPO6ISE4BPH7A, seg_pos={0 0}, upload=463022]: Uploading piece...
Dec 27 15:22:34.633490544 (527h53m34.504573579s) [noise=true , address=145.79.227.44:4218   , piece_id=2KHGZL76RA6CMI33PTF5WZN5TRIK3LRL6XT5IJP6LNJP5ZRVMHXA, seg_pos={0 0}, upload=463022]: Uploading piece...
Dec 27 15:22:34.633891278 (527h53m34.504974313s) [noise=true , address=77.90.31.18:4820     , piece_id=TSKDIH6HQUUQRTQTJDXGBX3OALVLYIUKR5QCPJFPFXTCASNN5JMQ, seg_pos={0 0}, upload=463022]: Uploading piece...
Dec 27 15:22:34.634043096 (527h53m34.505126141s) [noise=true , address=145.79.227.17:4997   , piece_id=VX4S5TXLHFNBRE7OYPDZAGYDGN2AD34ZL5JQ2YUSBV656RO6A5UA, seg_pos={0 0}, upload=463022]: Uploading piece...
Dec 27 15:22:34.634325093 (527h53m34.505408128s) [noise=true , address=77.90.31.16:4739     , piece_id=M7IJW7WX2FNPFWNJPBUAHQQOKPFXDSWAWMC2IHUYQZ4JPXF3YZXQ, seg_pos={0 0}, upload=463022]: Uploading piece...
Dec 27 15:22:34.634722230 (527h53m34.505805265s) [noise=true , address=77.90.31.15:4716     , piece_id=MLOLX2R56NCS4KT3FLJJNNPDAMUA6S5NSFCX57W7V6GMIZ4457DA, seg_pos={0 0}, upload=463022]: Uploading piece...
Dec 27 15:22:34.635114116 (527h53m34.506197151s) [noise=true , address=45.91.116.13:4171    , piece_id=YPPWJWZ6H4XMNLGVDHNRQPMZJDXYQRO2EYLNUVCC6WSHIRRJSEWQ, seg_pos={0 0}, upload=463022]: Uploading piece...
Dec 27 15:22:34.635719949 (527h53m34.506802984s) [noise=true , address=77.90.31.12:4618     , piece_id=ZII3XYSUJ7WTNNZA77GO54GFVGVHOGUUJOFZG2OJDTDDP6CAPC3A, seg_pos={0 0}, upload=463022]: Uploading piece...
Dec 27 15:22:34.636184344 (527h53m34.507267379s) [noise=true , address=145.79.227.30:4775   , piece_id=RG6IZXLWJSMKNHY2VR7VHDATQ2LAHIEGYF6WDQIZXUTGETQKS3DQ, seg_pos={0 0}, upload=463022]: Uploading piece...
Dec 27 15:22:36.104663675 (527h53m35.97574671s) [upload=463022]: Next returned nil segment
Dec 27 15:22:36.104675988 (527h53m35.975759023s) [upload=463022]: Waiting for error group managing segments...
Dec 27 15:22:36.505615923 (527h53m36.376698958s) [noise=true , address=145.79.227.17:4997   , piece_id=VX4S5TXLHFNBRE7OYPDZAGYDGN2AD34ZL5JQ2YUSBV656RO6A5UA, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil>
Dec 27 15:22:36.509752460 (527h53m36.380835505s) [noise=true , address=145.79.227.13:4816   , piece_id=LTSC26UNXCIHK4PDGYKE7GZCQ7MXCWWRGBLIHI6FPO6ISE4BPH7A, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil>
Dec 27 15:22:36.517493871 (527h53m36.388576906s) [noise=true , address=145.79.227.14:4865   , piece_id=XOXM2D6IS4SG6DN24DEGIPV7NK5FWDI7QVVJK7L3QB25AUPRNZIQ, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil>
Dec 27 15:22:36.623194766 (527h53m36.494277801s) [noise=true , address=145.79.227.52:4466   , piece_id=CZYMGB53RT6FROGYMOYOPOQJ5CSZDW4YYAVDQWYWVLIDINN3LJVA, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil>
Dec 27 15:22:36.749453706 (527h53m36.620536741s) [noise=true , address=45.91.116.11:4145    , piece_id=77YGIECBFASFKBHIQO333RHATPHZCA7JQW2WPNSGEGM24TTWDVGA, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil>
Dec 27 15:22:36.780705355 (527h53m36.6517884s) [noise=true , address=77.90.31.18:4820     , piece_id=TSKDIH6HQUUQRTQTJDXGBX3OALVLYIUKR5QCPJFPFXTCASNN5JMQ, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil>
Dec 27 15:22:36.803640327 (527h53m36.674723362s) [noise=true , address=77.90.31.13:4659     , piece_id=STKDI6VSN67XV43CB4ASCJTPDU6NA75REYKEE6D5K3EFM5FCDMYA, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil>
Dec 27 15:22:36.841101831 (527h53m36.712184876s) [noise=true , address=77.90.31.16:4739     , piece_id=M7IJW7WX2FNPFWNJPBUAHQQOKPFXDSWAWMC2IHUYQZ4JPXF3YZXQ, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil>
Dec 27 15:22:37.481224175 (527h53m37.35230722s) [noise=true , address=77.90.31.15:4716     , piece_id=MLOLX2R56NCS4KT3FLJJNNPDAMUA6S5NSFCX57W7V6GMIZ4457DA, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil>
Dec 27 15:22:37.504901952 (527h53m37.375984997s) [noise=true , address=145.79.227.30:4775   , piece_id=RG6IZXLWJSMKNHY2VR7VHDATQ2LAHIEGYF6WDQIZXUTGETQKS3DQ, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil>
Dec 27 15:22:37.553631302 (527h53m37.424714337s) [noise=true , address=145.79.227.44:4218   , piece_id=2KHGZL76RA6CMI33PTF5WZN5TRIK3LRL6XT5IJP6LNJP5ZRVMHXA, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil>
Dec 27 15:22:37.596982441 (527h53m37.468065476s) [noise=true , address=145.79.227.40:4063   , piece_id=SRV3KMN3XXUOJA2GLEJ27UKWRANPEKCT3NC7BBDAVMTCI6FZCNPQ, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil>
Dec 27 15:22:37.792107735 (527h53m37.663190771s) [noise=true , address=77.90.31.17:4802     , piece_id=TFQZEPPQN5RYDUNTGLILJUHZ3QLTU7BT3QP67SS5Y4ZRUKPG6A5A, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil>
Dec 27 15:22:37.802639390 (527h53m37.673722425s) [noise=true , address=77.90.31.12:4618     , piece_id=ZII3XYSUJ7WTNNZA77GO54GFVGVHOGUUJOFZG2OJDTDDP6CAPC3A, seg_pos={0 0}, upload=463022]: Done uploading piece. err: <nil>
Dec 27 15:22:37.802661622 (527h53m37.673744667s) [seg_pos={0 0}, upload=463022]: Segment reached optimal threshold of 14 pieces.
Dec 27 15:22:37.802830423 (527h53m37.673913468s) [noise=true , address=77.90.31.11:4565     , piece_id=B4VPI2I53PUUOLNS6C6GFKY3ZYDQ5RRQZCKC4QVLSVJI777RR6KQ, seg_pos={0 0}, upload=463022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xnwFcUEdmpLjmx8NXZZcpiVQwTySXDRYbchuz4oREBq1yP6wh): protocol: write tcp 188.166.225.170:60558->77.90.31.11:4565: use of closed network connection; write tcp 188.166.225.170:60558->77.90.31.11:4565: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60558->77.90.31.11:4565: use of closed network connection
Dec 27 15:22:37.802837557 (527h53m37.673920592s) [noise=true , address=77.90.31.11:4565     , piece_id=B4VPI2I53PUUOLNS6C6GFKY3ZYDQ5RRQZCKC4QVLSVJI777RR6KQ, seg_pos={0 0}, upload=463022]: Long tail cancellation due to optimal threshold
Dec 27 15:22:37.803060181 (527h53m37.674143216s) [noise=true , address=45.91.116.13:4171    , piece_id=YPPWJWZ6H4XMNLGVDHNRQPMZJDXYQRO2EYLNUVCC6WSHIRRJSEWQ, seg_pos={0 0}, upload=463022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122dmyMaZDYbCvoYvCc9m8Vf5dA8aJiw9qPh1G5AQWKnBig4mVg): protocol: write tcp 188.166.225.170:35780->45.91.116.13:4171: use of closed network connection; write tcp 188.166.225.170:35780->45.91.116.13:4171: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35780->45.91.116.13:4171: use of closed network connection
Dec 27 15:22:37.803068377 (527h53m37.674151422s) [noise=true , address=45.91.116.13:4171    , piece_id=YPPWJWZ6H4XMNLGVDHNRQPMZJDXYQRO2EYLNUVCC6WSHIRRJSEWQ, seg_pos={0 0}, upload=463022]: Long tail cancellation due to optimal threshold
Dec 27 15:22:37.803144622 (527h53m37.674227667s) [noise=true , address=64.49.13.169:4708    , piece_id=J7UIJKZWYB2HZ2TCTMPBKNYT65NEYNQADUXQG2J3WTCLD343XLSQ, seg_pos={0 0}, upload=463022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:51138->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:51138->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51138->64.49.13.169:4708: use of closed network connection
Dec 27 15:22:37.803149080 (527h53m37.674232115s) [noise=true , address=64.49.13.169:4708    , piece_id=J7UIJKZWYB2HZ2TCTMPBKNYT65NEYNQADUXQG2J3WTCLD343XLSQ, seg_pos={0 0}, upload=463022]: Long tail cancellation due to optimal threshold
Dec 27 15:22:37.803258279 (527h53m37.674341314s) [noise=true , address=77.90.31.10:4540     , piece_id=MVAPVVOZIC5C6BL5S74PXSKTLTCGLIOSN37M33OUYVPD53NG3GLQ, seg_pos={0 0}, upload=463022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA): protocol: expected piece hash; context canceled; EOF
Dec 27 15:22:37.803263358 (527h53m37.674346393s) [noise=true , address=77.90.31.10:4540     , piece_id=MVAPVVOZIC5C6BL5S74PXSKTLTCGLIOSN37M33OUYVPD53NG3GLQ, seg_pos={0 0}, upload=463022]: Long tail cancellation due to optimal threshold
Dec 27 15:22:37.803277855 (527h53m37.674360901s) [seg_pos={0 0}, upload=463022]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 15:22:37.803341717 (527h53m37.674424762s) [upload=463022]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 15:22:37.803346236 (527h53m37.674429271s) [upload=463022]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 15:22:38.115241263 (527h53m37.986324298s) [upload=463022]: Done uploading segments...
Dec 27 15:22:38.115265930 (527h53m37.986348975s) [upload=463022]: Upload finished. err: <nil>
Duration: 1m0.002038956s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
21 out of 60 is failed (threshold=3) [...X.....X..XX.......X..X.X..X...XXXXXXX..X.X..X.X.....X...X]
duration: 
30.003105037s
trace: 
https://jaeger.dev.storj.io/trace/3382203165a15a75
FAIL (2025-12-27 15:23:16 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: EFNF7USD6GGEH3V4T42NKM2JPEA3OCYTQOXTNIVRDA6COXPG5JTA) context deadline exceeded; piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: J3OER7CS2XJ6NVGM4X5XM6DWCWP2LZBOSVWD2CQNXABR4TEJXTIA) context deadline exceeded; piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: DBGSITGU4SZAAYFK5KDZ2MXONZ27XTNAV3OPQCQAUHKYBQNPNXSA) context deadline exceeded
======LOGS======
27 15:22:47.299628936 (527h53m47.170711971s) [upload=463024]: Starting upload
Dec 27 15:22:47.299696604 (527h53m47.17077964s) [upload=463024]: Done starting upload
Dec 27 15:22:47.299787547 (527h53m47.170870622s) [upload=463024]: Uploading segments...
Dec 27 15:22:47.299913407 (527h53m47.170996442s) [upload=463024]: Got next segment. Inline: false
Dec 27 15:22:47.299953483 (527h53m47.171036518s) [upload=463024]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 15:22:47.299958212 (527h53m47.171041257s) [upload=463024]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 15:22:47.545973752 (527h53m47.417056787s) [seg_pos={0 0}, upload=463024]: Begin upload segment...
Dec 27 15:22:47.546024249 (527h53m47.417107284s) [seg_pos={0 0}, upload=463024]: Done begin upload segment.
Dec 27 15:22:47.546057682 (527h53m47.417140717s) [noise=true , address=145.79.227.3:4151    , piece_id=627M5Y34ADXGUIAOOGMTG5PAEEZWBT3PAUOZHPI5M23TFTAV2OWQ, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:47.546065778 (527h53m47.417148813s) [noise=true , address=145.79.227.2:4113    , piece_id=YPFORZR5QCXBPQEQZKUVOIUVM4NYKOLU4KDYA7HRJDVMM3KXLKUQ, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:47.546207497 (527h53m47.417290542s) [noise=true , address=145.79.227.24:4581   , piece_id=7HS6Q5Q3ILUXV6BML37VQY2JS4P77CBO6ZTSENMY56MYOYJW6ANQ, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:47.546489023 (527h53m47.417572068s) [noise=true , address=145.79.227.29:4941   , piece_id=VHMADVXOXZOTYAVGE52WNY2IJWQMSQ7LNRSNXOJEZTFBYWLKIRIA, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:47.546754940 (527h53m47.417837975s) [noise=true , address=145.79.227.42:4121   , piece_id=6UQBDGBP6MIDTPV7RYC5NE7JOW6DP74P5QK3BBHVK4MV5UCKGZNA, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:47.547012460 (527h53m47.418095495s) [noise=true , address=145.79.227.47:4265   , piece_id=44SEXUFB2LZHONSA7GDQ63JOYEYETU5ZUMWF45SKXAE5AP7KVWOA, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:47.547340776 (527h53m47.418423811s) [noise=true , address=45.91.116.14:4192    , piece_id=JRQVETWQDYFMV3OR3BOK54FNOPFAZFQJH73WAXLM4KUDVKMKNF2Q, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:47.547610368 (527h53m47.418693413s) [noise=true , address=145.79.227.7:4295    , piece_id=4YVVZDHQNWCCA6XFV7A7ACJL7TUKPFPKKESO5WBIC7XEIDNULCWA, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:47.547903517 (527h53m47.418986552s) [noise=true , address=45.91.116.13:4154    , piece_id=EFNF7USD6GGEH3V4T42NKM2JPEA3OCYTQOXTNIVRDA6COXPG5JTA, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:47.548994664 (527h53m47.4200777s) [noise=true , address=145.79.227.50:4374   , piece_id=PTUKPJS4VWMT7CWQZIFOMI66JNJS6D2J3OMDYUBF4IL6LXMP465Q, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:47.550097284 (527h53m47.421180319s) [noise=true , address=145.79.227.30:4761   , piece_id=AGS3KQJG5YMSHYQA2C4UQRCYYW62QARBO4I2QWCOEBN445HMYQGQ, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:47.551125933 (527h53m47.422208968s) [noise=true , address=45.91.116.11:4119    , piece_id=J3OER7CS2XJ6NVGM4X5XM6DWCWP2LZBOSVWD2CQNXABR4TEJXTIA, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:47.552145084 (527h53m47.423228119s) [noise=true , address=145.79.227.19:4078   , piece_id=YZCJB4ZJN6BNWDDPOTGHEWQNRD5BCGZZACXVDF6XOX4FOLGZGDBQ, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:47.553167881 (527h53m47.424250916s) [noise=true , address=45.91.116.6:4976     , piece_id=ZHQFB66SB4X4AVSYKPFNDZMWURFQRVZT6QG7QC23JCRW22443M7A, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:47.554213733 (527h53m47.425296768s) [noise=true , address=45.91.116.12:4228    , piece_id=DBGSITGU4SZAAYFK5KDZ2MXONZ27XTNAV3OPQCQAUHKYBQNPNXSA, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:47.555339717 (527h53m47.426422752s) [noise=true , address=145.79.227.38:4977   , piece_id=UVSIVE7ZZECB74JPH7RFRAI74SX5HKKQK2ZHDRQ4K5X7JUZOWKWA, seg_pos={0 0}, upload=463024]: Uploading piece...
Dec 27 15:22:49.174094752 (527h53m49.045177787s) [upload=463024]: Next returned nil segment
Dec 27 15:22:49.174108467 (527h53m49.045191512s) [upload=463024]: Waiting for error group managing segments...
Dec 27 15:22:49.550005476 (527h53m49.421088511s) [noise=true , address=145.79.227.29:4941   , piece_id=VHMADVXOXZOTYAVGE52WNY2IJWQMSQ7LNRSNXOJEZTFBYWLKIRIA, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil>
Dec 27 15:22:49.551387929 (527h53m49.422470964s) [noise=true , address=145.79.227.7:4295    , piece_id=4YVVZDHQNWCCA6XFV7A7ACJL7TUKPFPKKESO5WBIC7XEIDNULCWA, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil>
Dec 27 15:22:49.554614222 (527h53m49.425697267s) [noise=true , address=145.79.227.19:4078   , piece_id=YZCJB4ZJN6BNWDDPOTGHEWQNRD5BCGZZACXVDF6XOX4FOLGZGDBQ, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil>
Dec 27 15:22:49.556863425 (527h53m49.42794646s) [noise=true , address=145.79.227.42:4121   , piece_id=6UQBDGBP6MIDTPV7RYC5NE7JOW6DP74P5QK3BBHVK4MV5UCKGZNA, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil>
Dec 27 15:22:49.557246234 (527h53m49.428329279s) [noise=true , address=145.79.227.38:4977   , piece_id=UVSIVE7ZZECB74JPH7RFRAI74SX5HKKQK2ZHDRQ4K5X7JUZOWKWA, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil>
Dec 27 15:22:49.558682599 (527h53m49.429765644s) [noise=true , address=145.79.227.2:4113    , piece_id=YPFORZR5QCXBPQEQZKUVOIUVM4NYKOLU4KDYA7HRJDVMM3KXLKUQ, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil>
Dec 27 15:22:49.564448258 (527h53m49.435531303s) [noise=true , address=145.79.227.3:4151    , piece_id=627M5Y34ADXGUIAOOGMTG5PAEEZWBT3PAUOZHPI5M23TFTAV2OWQ, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil>
Dec 27 15:22:49.566142284 (527h53m49.437225329s) [noise=true , address=145.79.227.50:4374   , piece_id=PTUKPJS4VWMT7CWQZIFOMI66JNJS6D2J3OMDYUBF4IL6LXMP465Q, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil>
Dec 27 15:22:49.574698126 (527h53m49.445781171s) [noise=true , address=145.79.227.24:4581   , piece_id=7HS6Q5Q3ILUXV6BML37VQY2JS4P77CBO6ZTSENMY56MYOYJW6ANQ, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil>
Dec 27 15:22:49.586061825 (527h53m49.45714487s) [noise=true , address=145.79.227.47:4265   , piece_id=44SEXUFB2LZHONSA7GDQ63JOYEYETU5ZUMWF45SKXAE5AP7KVWOA, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil>
Dec 27 15:22:49.793614724 (527h53m49.664697759s) [noise=true , address=45.91.116.11:4119    , piece_id=J3OER7CS2XJ6NVGM4X5XM6DWCWP2LZBOSVWD2CQNXABR4TEJXTIA, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil>
Dec 27 15:22:49.874495066 (527h53m49.745578101s) [noise=true , address=45.91.116.12:4228    , piece_id=DBGSITGU4SZAAYFK5KDZ2MXONZ27XTNAV3OPQCQAUHKYBQNPNXSA, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil>
Dec 27 15:22:49.950055185 (527h53m49.821138231s) [noise=true , address=45.91.116.13:4154    , piece_id=EFNF7USD6GGEH3V4T42NKM2JPEA3OCYTQOXTNIVRDA6COXPG5JTA, seg_pos={0 0}, upload=463024]: Done uploading piece. err: <nil>
Dec 27 15:22:49.950079642 (527h53m49.821162677s) [seg_pos={0 0}, upload=463024]: Segment reached optimal threshold of 13 pieces.
Dec 27 15:22:49.950443906 (527h53m49.821526952s) [noise=true , address=145.79.227.30:4761   , piece_id=AGS3KQJG5YMSHYQA2C4UQRCYYW62QARBO4I2QWCOEBN445HMYQGQ, seg_pos={0 0}, upload=463024]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF
Dec 27 15:22:49.950463724 (527h53m49.821546759s) [noise=true , address=145.79.227.30:4761   , piece_id=AGS3KQJG5YMSHYQA2C4UQRCYYW62QARBO4I2QWCOEBN445HMYQGQ, seg_pos={0 0}, upload=463024]: Long tail cancellation due to optimal threshold
Dec 27 15:22:49.950484183 (527h53m49.821567228s) [noise=true , address=45.91.116.14:4192    , piece_id=JRQVETWQDYFMV3OR3BOK54FNOPFAZFQJH73WAXLM4KUDVKMKNF2Q, seg_pos={0 0}, upload=463024]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: write tcp 188.166.225.170:49708->45.91.116.14:4192: use of closed network connection; write tcp 188.166.225.170:49708->45.91.116.14:4192: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49708->45.91.116.14:4192: use of closed network connection
Dec 27 15:22:49.950523267 (527h53m49.821606302s) [noise=true , address=45.91.116.14:4192    , piece_id=JRQVETWQDYFMV3OR3BOK54FNOPFAZFQJH73WAXLM4KUDVKMKNF2Q, seg_pos={0 0}, upload=463024]: Long tail cancellation due to optimal threshold
Dec 27 15:22:49.950900406 (527h53m49.821983451s) [noise=true , address=45.91.116.6:4976     , piece_id=ZHQFB66SB4X4AVSYKPFNDZMWURFQRVZT6QG7QC23JCRW22443M7A, seg_pos={0 0}, upload=463024]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF
Dec 27 15:22:49.950911727 (527h53m49.821994772s) [noise=true , address=45.91.116.6:4976     , piece_id=ZHQFB66SB4X4AVSYKPFNDZMWURFQRVZT6QG7QC23JCRW22443M7A, seg_pos={0 0}, upload=463024]: Long tail cancellation due to optimal threshold
Dec 27 15:22:49.950943928 (527h53m49.822026993s) [seg_pos={0 0}, upload=463024]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 15:22:49.951104575 (527h53m49.82218762s) [upload=463024]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 15:22:49.951113601 (527h53m49.822196657s) [upload=463024]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 15:22:50.253003756 (527h53m50.124086801s) [upload=463024]: Done uploading segments...
Dec 27 15:22:50.253023765 (527h53m50.1241068s) [upload=463024]: Upload finished. err: <nil>
Duration: 30.003128943s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
37 out of 60 is failed (threshold=6) [XXXXXX.....XXX..X..XX.XXXX.....XXX.X.XXXX.X.XXXX...XXXX.XXXX]
duration: 
5.001638028s
trace: 
https://jaeger.dev.storj.io/trace/4d56275e57b7d1cf
FAIL (2025-12-27 15:23:08 UTC) 
orbiter warning: context deadline exceeded
Duration: 5.001661423s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
6 out of 60 is failed (threshold=3) [.X...........X..............X....X...................X.....X]
duration: 
1m0.003716905s
trace: 
https://jaeger.dev.storj.io/trace/3fabb9325ff74319
FAIL (2025-12-27 15:22:33 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
4s) [noise=true , address=77.90.31.19:4860     , piece_id=ANNIF26M6O2O2QYF5BN5QEBR7SZPN52POARGWI7YTUQMCZVKDM5Q, seg_pos={0 0}, upload=463007]: Uploading piece...
Dec 27 15:21:34.369010277 (527h52m34.240093322s) [noise=true , address=77.90.31.12:4612     , piece_id=RNA453I2LXJ3H65AQ27TJDHMXPNBWGVY5YUPCBBBHBIQQVJDRNFQ, seg_pos={0 0}, upload=463007]: Uploading piece...
Dec 27 15:21:34.369404528 (527h52m34.240487563s) [noise=true , address=145.79.227.33:4882   , piece_id=ZQFLNXAGJJ2MQZSZVURDEX5OQAW3FUOMDMMLGF43UGZAVM2KAC6A, seg_pos={0 0}, upload=463007]: Uploading piece...
Dec 27 15:21:34.369911163 (527h52m34.240994208s) [noise=true , address=145.79.227.30:4768   , piece_id=QCA27RWVM2CZSQPX7YVHHTN5O2E7EFUY3GKXRASNDSHL6JFICPSA, seg_pos={0 0}, upload=463007]: Uploading piece...
Dec 27 15:21:34.370341542 (527h52m34.241424587s) [noise=true , address=45.91.116.14:4223    , piece_id=72TLZL42QCE6J465MGT4NK7Z3EKF5SNFEAPXC3DEOTC4ZJ3G75TQ, seg_pos={0 0}, upload=463007]: Uploading piece...
Dec 27 15:21:34.370759267 (527h52m34.241842302s) [noise=true , address=45.91.116.10:4113    , piece_id=FFWWG53B3H4LLFO6524OTXA3CQ5VFHMKUAL74IE4CYKGBC3X222A, seg_pos={0 0}, upload=463007]: Uploading piece...
Dec 27 15:21:34.371245233 (527h52m34.242328278s) [noise=true , address=77.90.31.13:4650     , piece_id=X5AECLLFHIQR2JASCHVBFN267DAWW3QTME2DWDZSL4GAHUXQ4TKQ, seg_pos={0 0}, upload=463007]: Uploading piece...
Dec 27 15:21:34.371737310 (527h52m34.242820355s) [noise=true , address=77.90.31.11:4566     , piece_id=5F32QOJPDB3UOIQDGLSG3YQYJEWMRGSMIZVXR2YJUTLGNMHUQ4MQ, seg_pos={0 0}, upload=463007]: Uploading piece...
Dec 27 15:21:34.372165416 (527h52m34.243248451s) [noise=true , address=145.79.227.34:4938   , piece_id=UTPXQZBMDY5SNGYWPH2I63LXFXQMM3SS5FC3LE6V4R5HZSH5QWXQ, seg_pos={0 0}, upload=463007]: Uploading piece...
Dec 27 15:21:34.372918970 (527h52m34.244002005s) [noise=true , address=77.90.31.21:4882     , piece_id=DMRHPTRACFA2A6E4JI2DNOOVLUZM7K43PQI4NH3QZUJ2QNGGW7SA, seg_pos={0 0}, upload=463007]: Uploading piece...
Dec 27 15:21:34.373404675 (527h52m34.24448771s) [noise=true , address=77.90.31.18:4825     , piece_id=SYRXD477QV743OHZC5PKQR3B5T5UEHB33BPAMWG6NPTGDUBTUGMQ, seg_pos={0 0}, upload=463007]: Uploading piece...
Dec 27 15:21:34.374011060 (527h52m34.245094105s) [noise=true , address=145.79.227.40:4078   , piece_id=TTY2JM753TXH3IEM57QHMAETKLQ7RK2B42TYUISVD62QUSAE3W2A, seg_pos={0 0}, upload=463007]: Uploading piece...
Dec 27 15:21:34.374503377 (527h52m34.245586422s) [noise=true , address=64.49.13.169:4706    , piece_id=BPFMA4R5AIHZNHZFG5O3WRPSH6MFIRZ2IKVSY6DONCIAMDUYHQRQ, seg_pos={0 0}, upload=463007]: Uploading piece...
Dec 27 15:21:34.374964655 (527h52m34.2460477s) [noise=true , address=77.90.31.10:4552     , piece_id=OON4JXAO6IRXWPKMGGWFDCHPTMVKOIZGAOM367MPJSJUAFD3OU4A, seg_pos={0 0}, upload=463007]: Uploading piece...
Dec 27 15:21:34.377921746 (527h52m34.249004791s) [noise=true , address=45.91.116.5:4329     , piece_id=JUPEFJMOGXJCGJGORVKODIAXEVEKC3CTRSFW4MJK7AEZYJX44A7Q, seg_pos={0 0}, upload=463007]: Uploading piece...
Dec 27 15:21:34.378561905 (527h52m34.24964495s) [noise=true , address=145.79.227.43:4192   , piece_id=FF4IWBETZLI6NKOXXFN5CTOJ3EXQC235HRU2XLKK7Q42WRE25J3A, seg_pos={0 0}, upload=463007]: Uploading piece...
Dec 27 15:21:35.855010390 (527h52m35.726093425s) [upload=463007]: Next returned nil segment
Dec 27 15:21:35.855021472 (527h52m35.726104517s) [upload=463007]: Waiting for error group managing segments...
Dec 27 15:21:36.291926761 (527h52m36.163009806s) [noise=true , address=145.79.227.34:4938   , piece_id=UTPXQZBMDY5SNGYWPH2I63LXFXQMM3SS5FC3LE6V4R5HZSH5QWXQ, seg_pos={0 0}, upload=463007]: Done uploading piece. err: <nil>
Dec 27 15:21:36.418807306 (527h52m36.289890341s) [noise=true , address=77.90.31.14:4686     , piece_id=ZVRJPH4ALTNQFI6OG3VB65LMBH4FNVMLB5N2RNVIREBFJ2ZOW6OQ, seg_pos={0 0}, upload=463007]: Done uploading piece. err: <nil>
Dec 27 15:21:36.426031451 (527h52m36.297114496s) [noise=true , address=77.90.31.11:4566     , piece_id=5F32QOJPDB3UOIQDGLSG3YQYJEWMRGSMIZVXR2YJUTLGNMHUQ4MQ, seg_pos={0 0}, upload=463007]: Done uploading piece. err: <nil>
Dec 27 15:21:36.706741176 (527h52m36.577824211s) [noise=true , address=45.91.116.10:4113    , piece_id=FFWWG53B3H4LLFO6524OTXA3CQ5VFHMKUAL74IE4CYKGBC3X222A, seg_pos={0 0}, upload=463007]: Done uploading piece. err: <nil>
Dec 27 15:21:36.707216270 (527h52m36.578299315s) [noise=true , address=77.90.31.18:4825     , piece_id=SYRXD477QV743OHZC5PKQR3B5T5UEHB33BPAMWG6NPTGDUBTUGMQ, seg_pos={0 0}, upload=463007]: Done uploading piece. err: <nil>
Dec 27 15:21:36.751195264 (527h52m36.622278299s) [noise=true , address=45.91.116.14:4223    , piece_id=72TLZL42QCE6J465MGT4NK7Z3EKF5SNFEAPXC3DEOTC4ZJ3G75TQ, seg_pos={0 0}, upload=463007]: Done uploading piece. err: <nil>
Dec 27 15:21:36.892739010 (527h52m36.763822045s) [noise=true , address=77.90.31.19:4860     , piece_id=ANNIF26M6O2O2QYF5BN5QEBR7SZPN52POARGWI7YTUQMCZVKDM5Q, seg_pos={0 0}, upload=463007]: Done uploading piece. err: <nil>
Dec 27 15:21:36.905165182 (527h52m36.776248227s) [noise=true , address=77.90.31.13:4650     , piece_id=X5AECLLFHIQR2JASCHVBFN267DAWW3QTME2DWDZSL4GAHUXQ4TKQ, seg_pos={0 0}, upload=463007]: Done uploading piece. err: <nil>
Dec 27 15:21:37.145642736 (527h52m37.016725771s) [noise=true , address=145.79.227.44:4208   , piece_id=FNRCXLIPXZV3T6CDNZUAY2BR3D6CAXQC4J2T5OKWVSQBXLTAAFAA, seg_pos={0 0}, upload=463007]: Done uploading piece. err: <nil>
Dec 27 15:21:37.154877892 (527h52m37.025960927s) [noise=true , address=145.79.227.43:4192   , piece_id=FF4IWBETZLI6NKOXXFN5CTOJ3EXQC235HRU2XLKK7Q42WRE25J3A, seg_pos={0 0}, upload=463007]: Done uploading piece. err: <nil>
Dec 27 15:21:37.199396764 (527h52m37.070479799s) [noise=true , address=77.90.31.10:4552     , piece_id=OON4JXAO6IRXWPKMGGWFDCHPTMVKOIZGAOM367MPJSJUAFD3OU4A, seg_pos={0 0}, upload=463007]: Done uploading piece. err: <nil>
Dec 27 15:21:37.214836033 (527h52m37.085919078s) [noise=true , address=45.91.116.5:4329     , piece_id=JUPEFJMOGXJCGJGORVKODIAXEVEKC3CTRSFW4MJK7AEZYJX44A7Q, seg_pos={0 0}, upload=463007]: Done uploading piece. err: <nil>
Dec 27 15:21:37.218432663 (527h52m37.089515718s) [noise=true , address=77.90.31.12:4612     , piece_id=RNA453I2LXJ3H65AQ27TJDHMXPNBWGVY5YUPCBBBHBIQQVJDRNFQ, seg_pos={0 0}, upload=463007]: Done uploading piece. err: <nil>
Dec 27 15:21:37.265045552 (527h52m37.136128597s) [noise=true , address=145.79.227.30:4768   , piece_id=QCA27RWVM2CZSQPX7YVHHTN5O2E7EFUY3GKXRASNDSHL6JFICPSA, seg_pos={0 0}, upload=463007]: Done uploading piece. err: <nil>
Dec 27 15:21:37.265064668 (527h52m37.136147703s) [seg_pos={0 0}, upload=463007]: Segment reached optimal threshold of 14 pieces.
Dec 27 15:21:37.265271973 (527h52m37.136355018s) [noise=true , address=145.79.227.40:4078   , piece_id=TTY2JM753TXH3IEM57QHMAETKLQ7RK2B42TYUISVD62QUSAE3W2A, seg_pos={0 0}, upload=463007]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tzgtenV6us5ssD7xWGpZmVUoWEfbFhBNewbK9irHtZPkyDtRd): protocol: expected piece hash; context canceled; EOF
Dec 27 15:21:37.265280269 (527h52m37.136363314s) [noise=true , address=145.79.227.40:4078   , piece_id=TTY2JM753TXH3IEM57QHMAETKLQ7RK2B42TYUISVD62QUSAE3W2A, seg_pos={0 0}, upload=463007]: Long tail cancellation due to optimal threshold
Dec 27 15:21:37.265551095 (527h52m37.13663414s) [noise=true , address=64.49.13.169:4706    , piece_id=BPFMA4R5AIHZNHZFG5O3WRPSH6MFIRZ2IKVSY6DONCIAMDUYHQRQ, seg_pos={0 0}, upload=463007]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:55168->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:55168->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55168->64.49.13.169:4706: use of closed network connection
Dec 27 15:21:37.265558649 (527h52m37.136641684s) [noise=true , address=64.49.13.169:4706    , piece_id=BPFMA4R5AIHZNHZFG5O3WRPSH6MFIRZ2IKVSY6DONCIAMDUYHQRQ, seg_pos={0 0}, upload=463007]: Long tail cancellation due to optimal threshold
Dec 27 15:21:37.265788817 (527h52m37.136871862s) [noise=true , address=145.79.227.33:4882   , piece_id=ZQFLNXAGJJ2MQZSZVURDEX5OQAW3FUOMDMMLGF43UGZAVM2KAC6A, seg_pos={0 0}, upload=463007]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TU7roqQsKNGvNeqmWdVxMregk9Dqn3YZpfdvK6xp7c3zYvccc): protocol: expected piece hash; context canceled; EOF
Dec 27 15:21:37.265848420 (527h52m37.136931485s) [noise=true , address=145.79.227.33:4882   , piece_id=ZQFLNXAGJJ2MQZSZVURDEX5OQAW3FUOMDMMLGF43UGZAVM2KAC6A, seg_pos={0 0}, upload=463007]: Long tail cancellation due to optimal threshold
Dec 27 15:21:37.266011692 (527h52m37.137094727s) [noise=true , address=77.90.31.21:4882     , piece_id=DMRHPTRACFA2A6E4JI2DNOOVLUZM7K43PQI4NH3QZUJ2QNGGW7SA, seg_pos={0 0}, upload=463007]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1): protocol: expected piece hash; context canceled; EOF
Dec 27 15:21:37.266019476 (527h52m37.137102511s) [noise=true , address=77.90.31.21:4882     , piece_id=DMRHPTRACFA2A6E4JI2DNOOVLUZM7K43PQI4NH3QZUJ2QNGGW7SA, seg_pos={0 0}, upload=463007]: Long tail cancellation due to optimal threshold
Dec 27 15:21:37.266040246 (527h52m37.137123291s) [seg_pos={0 0}, upload=463007]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 15:21:37.266109889 (527h52m37.137192924s) [upload=463007]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 15:21:37.266115389 (527h52m37.137198424s) [upload=463007]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 15:21:37.554391692 (527h52m37.425474747s) [upload=463007]: Done uploading segments...
Dec 27 15:21:37.554419855 (527h52m37.4255029s) [upload=463007]: Upload finished. err: <nil>
Duration: 1m0.003730651s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
21 out of 60 is failed (threshold=3) [X...X.....X..XX.......X..X.X..X...XXXXXXX..X.X..X.X.....X...]
duration: 
6.09157214s
trace: 
https://jaeger.dev.storj.io/trace/4340d7129ab7e03f
FAIL (2025-12-27 15:22:06 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: THLECT7WT7WVNNGDX2WK7IF4B47DAGAOYZV37NCUJYRIGQ3BB3PQ) context deadline exceeded; piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: FSSSGIHYQDHIPTB7DGP5IOFCDKEKBBIRMZKUSNGROYLBEHBVNQHA) context deadline exceeded; piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: QNWOWBKGGHKTF72OW3LUXZE2BCFL6FXVGEJGCVBL3XYBKRHCDA2A) context deadline exceeded; piecestore: (Node ID: 1tGpuYdPLiy4ncDjxrZbQfp7Bq79NVBHDmr4vjaL4qMpudrvgR, Piece ID: O4I7P5LAMRLSIOJOV2XGRVSZ24IJFMRVK3YEFSPAYO3ZXGHT4TGQ) context deadline exceeded
======LOGS======
Dec 27 15:19:35.326624967 (527h50m35.197708002s) [upload=462977]: Starting upload
Dec 27 15:19:35.326698597 (527h50m35.197781642s) [upload=462977]: Done starting upload
Dec 27 15:19:35.326775264 (527h50m35.197858299s) [upload=462977]: Uploading segments...
Dec 27 15:19:35.326900632 (527h50m35.197983667s) [upload=462977]: Got next segment. Inline: false
Dec 27 15:19:35.327029468 (527h50m35.198112513s) [upload=462977]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 15:19:35.327035970 (527h50m35.198119015s) [upload=462977]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 15:19:35.575151438 (527h50m35.446234493s) [seg_pos={0 0}, upload=462977]: Begin upload segment...
Dec 27 15:19:35.575277387 (527h50m35.446360432s) [seg_pos={0 0}, upload=462977]: Done begin upload segment.
Dec 27 15:19:35.575344425 (527h50m35.44642747s) [noise=true , address=145.79.227.35:4481   , piece_id=YNSBY6Q4UBO2O46Q3EW3NH6YDFW4BQAHJ7TESZ7O34LCS7KT6OIA, seg_pos={0 0}, upload=462977]: Uploading piece...
Dec 27 15:19:35.575358512 (527h50m35.446441557s) [noise=true , address=77.90.31.16:4736     , piece_id=6TO6C34RUNZ454OIKSMLO2ZAC72SPFS3QGMUFKTH6UOZSBMQATCA, seg_pos={0 0}, upload=462977]: Uploading piece...
Dec 27 15:19:35.575431381 (527h50m35.446514436s) [noise=true , address=145.79.227.52:4447   , piece_id=ZVUSMFX4YQSTZ36TSJI5NRQ3BE5JUTCMFY54ZIH2TAKZCMFDF37Q, seg_pos={0 0}, upload=462977]: Uploading piece...
Dec 27 15:19:35.576020322 (527h50m35.447103368s) [noise=true , address=45.91.116.13:4155    , piece_id=MOLWGIRUFYBMUPHOQJ42SOWGPP6UW5XPKW4FBBVP56RZCATFV67A, seg_pos={0 0}, upload=462977]: Uploading piece...
Dec 27 15:19:35.576457054 (527h50m35.447540089s) [noise=true , address=145.79.227.39:4015   , piece_id=SF7UPUO5NBSSOZ7NYSIP5DERX24YGU4VUGJ3X2NZGRBIWXRG2VEQ, seg_pos={0 0}, upload=462977]: Uploading piece...
Dec 27 15:19:35.576903584 (527h50m35.447986629s) [noise=true , address=45.91.116.7:4009     , piece_id=P7N3J663OUERIIYZAAQD3TRIIHCNPJRPEJVJZCR7POTV6SLSTRRA, seg_pos={0 0}, upload=462977]: Uploading piece...
Dec 27 15:19:35.577265433 (527h50m35.448348478s) [noise=true , address=45.91.116.10:4084    , piece_id=QNWOWBKGGHKTF72OW3LUXZE2BCFL6FXVGEJGCVBL3XYBKRHCDA2A, seg_pos={0 0}, upload=462977]: Uploading piece...
Dec 27 15:19:35.577781526 (527h50m35.448864561s) [noise=true , address=145.79.227.48:4302   , piece_id=O4I7P5LAMRLSIOJOV2XGRVSZ24IJFMRVK3YEFSPAYO3ZXGHT4TGQ, seg_pos={0 0}, upload=462977]: Uploading piece...
Dec 27 15:19:35.578261690 (527h50m35.449344735s) [noise=true , address=145.79.227.16:4911   , piece_id=GHH4FKVNY2ZSLHCZ5BS7H5HVV7XVA4QZMWIBYNOZE63UXJRKPMUQ, seg_pos={0 0}, upload=462977]: Uploading piece...
Dec 27 15:19:35.578982232 (527h50m35.450065267s) [noise=true , address=45.91.116.9:4261     , piece_id=6QG7D53YB6KPKLD2MYSTW7XZPBXGQKTC5LF4OIC2WX57HA7QFQ3A, seg_pos={0 0}, upload=462977]: Uploading piece...
Dec 27 15:19:35.580552592 (527h50m35.451635638s) [noise=true , address=145.79.227.30:4762   , piece_id=Q4UOW45D4MREICS4RQI3LJWW74VHJBPNJVE5FGNJPSRFCYWSR2NQ, seg_pos={0 0}, upload=462977]: Uploading piece...
Dec 27 15:19:35.580608849 (527h50m35.451691894s) [noise=true , address=77.90.31.11:4556     , piece_id=FSSSGIHYQDHIPTB7DGP5IOFCDKEKBBIRMZKUSNGROYLBEHBVNQHA, seg_pos={0 0}, upload=462977]: Uploading piece...
Dec 27 15:19:35.582562860 (527h50m35.453645905s) [noise=true , address=77.90.31.10:4518     , piece_id=THLECT7WT7WVNNGDX2WK7IF4B47DAGAOYZV37NCUJYRIGQ3BB3PQ, seg_pos={0 0}, upload=462977]: Uploading piece...
Dec 27 15:19:35.584806623 (527h50m35.455889668s) [noise=true , address=145.79.227.42:4121   , piece_id=ZL34UCFOPFF3QGIHHUVBZGXVGWMNC2AKCMFXZOWOBEHOFU4BJJKQ, seg_pos={0 0}, upload=462977]: Uploading piece...
Dec 27 15:19:35.587466858 (527h50m35.458549913s) [noise=true , address=145.79.227.47:4265   , piece_id=GXLBVVMRFYGXCGUEHWMCV4XM3QRZS54ITGKOC6NSXZQZ7WK6KOFQ, seg_pos={0 0}, upload=462977]: Uploading piece...
Dec 27 15:19:35.589399328 (527h50m35.460482373s) [noise=true , address=145.79.227.10:4403   , piece_id=6UINIY4GZXC2IKPPFZTQD3EVJGIDIY5RY2SJZVW3POHCNT33LZZQ, seg_pos={0 0}, upload=462977]: Uploading piece...
Dec 27 15:19:37.294264078 (527h50m37.165347113s) [upload=462977]: Next returned nil segment
Dec 27 15:19:37.294277052 (527h50m37.165360087s) [upload=462977]: Waiting for error group managing segments...
Dec 27 15:19:37.584769364 (527h50m37.455852399s) [noise=true , address=145.79.227.42:4121   , piece_id=ZL34UCFOPFF3QGIHHUVBZGXVGWMNC2AKCMFXZOWOBEHOFU4BJJKQ, seg_pos={0 0}, upload=462977]: Done uploading piece. err: <nil>
Dec 27 15:19:37.587919692 (527h50m37.459002737s) [noise=true , address=145.79.227.48:4302   , piece_id=O4I7P5LAMRLSIOJOV2XGRVSZ24IJFMRVK3YEFSPAYO3ZXGHT4TGQ, seg_pos={0 0}, upload=462977]: Done uploading piece. err: <nil>
Dec 27 15:19:37.595538479 (527h50m37.466621524s) [noise=true , address=145.79.227.10:4403   , piece_id=6UINIY4GZXC2IKPPFZTQD3EVJGIDIY5RY2SJZVW3POHCNT33LZZQ, seg_pos={0 0}, upload=462977]: Done uploading piece. err: <nil>
Dec 27 15:19:37.597846745 (527h50m37.46892979s) [noise=true , address=145.79.227.39:4015   , piece_id=SF7UPUO5NBSSOZ7NYSIP5DERX24YGU4VUGJ3X2NZGRBIWXRG2VEQ, seg_pos={0 0}, upload=462977]: Done uploading piece. err: <nil>
Dec 27 15:19:37.601914871 (527h50m37.472997906s) [noise=true , address=145.79.227.35:4481   , piece_id=YNSBY6Q4UBO2O46Q3EW3NH6YDFW4BQAHJ7TESZ7O34LCS7KT6OIA, seg_pos={0 0}, upload=462977]: Done uploading piece. err: <nil>
Dec 27 15:19:37.610231108 (527h50m37.481314163s) [noise=true , address=145.79.227.16:4911   , piece_id=GHH4FKVNY2ZSLHCZ5BS7H5HVV7XVA4QZMWIBYNOZE63UXJRKPMUQ, seg_pos={0 0}, upload=462977]: Done uploading piece. err: <nil>
Dec 27 15:19:37.610409387 (527h50m37.481492432s) [noise=true , address=145.79.227.47:4265   , piece_id=GXLBVVMRFYGXCGUEHWMCV4XM3QRZS54ITGKOC6NSXZQZ7WK6KOFQ, seg_pos={0 0}, upload=462977]: Done uploading piece. err: <nil>
Dec 27 15:19:37.611494423 (527h50m37.482577468s) [noise=true , address=145.79.227.52:4447   , piece_id=ZVUSMFX4YQSTZ36TSJI5NRQ3BE5JUTCMFY54ZIH2TAKZCMFDF37Q, seg_pos={0 0}, upload=462977]: Done uploading piece. err: <nil>
Dec 27 15:19:37.802153223 (527h50m37.673236258s) [noise=true , address=45.91.116.7:4009     , piece_id=P7N3J663OUERIIYZAAQD3TRIIHCNPJRPEJVJZCR7POTV6SLSTRRA, seg_pos={0 0}, upload=462977]: Done uploading piece. err: <nil>
Dec 27 15:19:37.847871228 (527h50m37.718954283s) [noise=true , address=77.90.31.11:4556     , piece_id=FSSSGIHYQDHIPTB7DGP5IOFCDKEKBBIRMZKUSNGROYLBEHBVNQHA, seg_pos={0 0}, upload=462977]: Done uploading piece. err: <nil>
Dec 27 15:19:37.919243645 (527h50m37.79032668s) [noise=true , address=45.91.116.10:4084    , piece_id=QNWOWBKGGHKTF72OW3LUXZE2BCFL6FXVGEJGCVBL3XYBKRHCDA2A, seg_pos={0 0}, upload=462977]: Done uploading piece. err: <nil>
Dec 27 15:19:37.986813229 (527h50m37.857896264s) [noise=true , address=45.91.116.13:4155    , piece_id=MOLWGIRUFYBMUPHOQJ42SOWGPP6UW5XPKW4FBBVP56RZCATFV67A, seg_pos={0 0}, upload=462977]: Done uploading piece. err: <nil>
Dec 27 15:19:38.458986455 (527h50m38.33006949s) [noise=true , address=77.90.31.10:4518     , piece_id=THLECT7WT7WVNNGDX2WK7IF4B47DAGAOYZV37NCUJYRIGQ3BB3PQ, seg_pos={0 0}, upload=462977]: Done uploading piece. err: <nil>
Dec 27 15:19:38.459004680 (527h50m38.330087716s) [seg_pos={0 0}, upload=462977]: Segment reached optimal threshold of 13 pieces.
Dec 27 15:19:38.459183651 (527h50m38.330266696s) [noise=true , address=145.79.227.30:4762   , piece_id=Q4UOW45D4MREICS4RQI3LJWW74VHJBPNJVE5FGNJPSRFCYWSR2NQ, seg_pos={0 0}, upload=462977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF
Dec 27 15:19:38.459189492 (527h50m38.330272527s) [noise=true , address=145.79.227.30:4762   , piece_id=Q4UOW45D4MREICS4RQI3LJWW74VHJBPNJVE5FGNJPSRFCYWSR2NQ, seg_pos={0 0}, upload=462977]: Long tail cancellation due to optimal threshold
Dec 27 15:19:38.459294222 (527h50m38.330377257s) [noise=true , address=45.91.116.9:4261     , piece_id=6QG7D53YB6KPKLD2MYSTW7XZPBXGQKTC5LF4OIC2WX57HA7QFQ3A, seg_pos={0 0}, upload=462977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF
Dec 27 15:19:38.459299892 (527h50m38.330382937s) [noise=true , address=45.91.116.9:4261     , piece_id=6QG7D53YB6KPKLD2MYSTW7XZPBXGQKTC5LF4OIC2WX57HA7QFQ3A, seg_pos={0 0}, upload=462977]: Long tail cancellation due to optimal threshold
Dec 27 15:19:38.459463304 (527h50m38.330546339s) [noise=true , address=77.90.31.16:4736     , piece_id=6TO6C34RUNZ454OIKSMLO2ZAC72SPFS3QGMUFKTH6UOZSBMQATCA, seg_pos={0 0}, upload=462977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF
Dec 27 15:19:38.459470407 (527h50m38.330553442s) [noise=true , address=77.90.31.16:4736     , piece_id=6TO6C34RUNZ454OIKSMLO2ZAC72SPFS3QGMUFKTH6UOZSBMQATCA, seg_pos={0 0}, upload=462977]: Long tail cancellation due to optimal threshold
Dec 27 15:19:38.459491307 (527h50m38.330574342s) [seg_pos={0 0}, upload=462977]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 15:19:38.459555860 (527h50m38.330638895s) [upload=462977]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 15:19:38.459561571 (527h50m38.330644606s) [upload=462977]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 15:19:38.757600261 (527h50m38.628683316s) [upload=462977]: Done uploading segments...
Dec 27 15:19:38.757625068 (527h50m38.628708123s) [upload=462977]: Upload finished. err: <nil>
Duration: 6.09160436s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
37 out of 60 is failed (threshold=6) [XXXXXXX.....XXX..X..XX.XXXX.....XXX.X.XXXX.X.XXXX...XXXX.XXX]
duration: 
5.001677465s
trace: 
https://jaeger.dev.storj.io/trace/758320a971ae2408
FAIL (2025-12-27 15:22:06 UTC) 
orbiter warning: context deadline exceeded
Duration: 5.001707421s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
36 out of 60 is failed (threshold=6) [.XXXXXXX.....XXX..X..XX.XXXX.....XXX.X.XXXX.X.XXXX...XXXX.XX]
duration: 
5.001134057s
trace: 
https://jaeger.dev.storj.io/trace/5ae7a6e60821a93b
FAIL (2025-12-27 15:21:43 UTC) 
orbiter warning: context deadline exceeded
Duration: 5.001160007s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
5 out of 60 is failed (threshold=3) [..X...........X..............X....X...................X.....]
duration: 
43.172883275s
trace: 
https://jaeger.dev.storj.io/trace/51d2e44a5388dd3e
FAIL (2025-12-27 15:21:33 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1geUSD7qjW2yuMayTJ1Zy1KTYVC19adZ9nUWMRtmg2TktJSsm4, Piece ID: FFDCFG5I7KTPNADJREI2ZTCZS2OGHMDKCXALKV3VS4L3QPYNEGFQ) context deadline exceeded; piecestore: (Node ID: 12S3ShBLc5k6bJJVNB6p68zF4kWAyVfDbF2stUyZngzTNtaEGz8, Piece ID: DE6GCEP2XJMVWXVKBY6UVOIC4RTVNJMPGLFL3QSSEW6BKY5S2GUQ) context deadline exceeded; piecestore: (Node ID: 12Vno3GkFXR7KdTUEAhvesqif7KBF6DZoeMTv8n9MpRbYZNktaT, Piece ID: OTO6TPBALIYRN7BHAO2NWWJA6MS5I25TLVA5C3MJ4VSQ7JDG5CXQ) context deadline exceeded
======LOGS======
  , piece_id=Z45MLEVRUQWQ2BVDUUBXBBEOZZWYYTHFNDE4LNRI2JLBUNHWD5GA, seg_pos={0 0}, upload=462935]: Uploading piece...
Dec 27 15:16:33.887438882 (527h47m33.758521917s) [noise=true , address=145.79.227.40:4052   , piece_id=PYSSKZ4VCZNBC7AWLUHAXQ22HU5GJXY26KPLQJN33PJBECO3FLMA, seg_pos={0 0}, upload=462935]: Uploading piece...
Dec 27 15:16:33.887737339 (527h47m33.758820374s) [noise=true , address=77.90.31.16:4761     , piece_id=2LXURVZ2IGTJERJFBNCRIWMDBLGHZM7XTBWIQAK3XTZIWGXRTOAQ, seg_pos={0 0}, upload=462935]: Uploading piece...
Dec 27 15:16:33.888853915 (527h47m33.75993696s) [noise=true , address=64.49.13.169:4706    , piece_id=I7G6HRKRTFN5DBWC7B7XIXV5RT3I5MSRXIDEIEIHHW7DRAHC5IAA, seg_pos={0 0}, upload=462935]: Uploading piece...
Dec 27 15:16:33.889312548 (527h47m33.760395594s) [noise=true , address=145.79.227.38:4984   , piece_id=R2DBGTZL6XK4MONKDYNU3AUMA7YXTGIUUZXM5CYDGP2TTFZNTYNQ, seg_pos={0 0}, upload=462935]: Uploading piece...
Dec 27 15:16:33.889894367 (527h47m33.760977412s) [noise=true , address=45.91.116.9:4275     , piece_id=DE6GCEP2XJMVWXVKBY6UVOIC4RTVNJMPGLFL3QSSEW6BKY5S2GUQ, seg_pos={0 0}, upload=462935]: Uploading piece...
Dec 27 15:16:33.890793499 (527h47m33.761876544s) [noise=true , address=45.91.116.13:4162    , piece_id=FFDCFG5I7KTPNADJREI2ZTCZS2OGHMDKCXALKV3VS4L3QPYNEGFQ, seg_pos={0 0}, upload=462935]: Uploading piece...
Dec 27 15:16:33.890936371 (527h47m33.762019416s) [noise=true , address=77.90.31.10:4541     , piece_id=ESQWS4AAYNHVNZCP2JXOCYKHLOALVDUNK5XUS5XXUNX5ASFZR25Q, seg_pos={0 0}, upload=462935]: Uploading piece...
Dec 27 15:16:33.892310177 (527h47m33.763393222s) [noise=true , address=145.79.227.9:4373    , piece_id=HPWEICNYP5UEF34YGDETNXGZEENVZP7JH4ORK7SN5PJE3A762GQQ, seg_pos={0 0}, upload=462935]: Uploading piece...
Dec 27 15:16:33.893865538 (527h47m33.764948583s) [noise=true , address=145.79.227.33:4889   , piece_id=ZVZ27B3XMIBI5MRYQTNAOH75NWSSJYDGBQISESP6U3YLO3H5OQSQ, seg_pos={0 0}, upload=462935]: Uploading piece...
Dec 27 15:16:33.897117241 (527h47m33.768200286s) [noise=true , address=77.90.31.15:4711     , piece_id=WF6BG55JI62B3MLWCRVA2OPQ6Y7DJG35NRAGOB4MYQEVNZZUPCXQ, seg_pos={0 0}, upload=462935]: Uploading piece...
Dec 27 15:16:33.901733621 (527h47m33.772816666s) [noise=true , address=77.90.31.21:4907     , piece_id=M6QS3DRLWECSW7DNJB742GLVARS74TXUEOCXL73NYH7A6AKZZLBQ, seg_pos={0 0}, upload=462935]: Uploading piece...
Dec 27 15:16:33.902601734 (527h47m33.773684769s) [noise=true , address=145.79.227.34:4919   , piece_id=LFCL2BTCUJ5HHTYUOUD4MBARGBWNM3LNVXGYALYZJYDT3L3QFSXQ, seg_pos={0 0}, upload=462935]: Uploading piece...
Dec 27 15:16:33.904727542 (527h47m33.775810578s) [noise=true , address=77.90.31.18:4816     , piece_id=OTO6TPBALIYRN7BHAO2NWWJA6MS5I25TLVA5C3MJ4VSQ7JDG5CXQ, seg_pos={0 0}, upload=462935]: Uploading piece...
Dec 27 15:16:33.905748506 (527h47m33.776831551s) [noise=true , address=77.90.31.13:4642     , piece_id=5E2LXQYSFJABRHXDB6XHG3AC44TN4ZYATKYIJAWGKIGIK4AHQJKA, seg_pos={0 0}, upload=462935]: Uploading piece...
Dec 27 15:16:35.355522705 (527h47m35.22660574s) [upload=462935]: Next returned nil segment
Dec 27 15:16:35.355540480 (527h47m35.226623515s) [upload=462935]: Waiting for error group managing segments...
Dec 27 15:16:35.802401195 (527h47m35.67348423s) [noise=true , address=145.79.227.9:4373    , piece_id=HPWEICNYP5UEF34YGDETNXGZEENVZP7JH4ORK7SN5PJE3A762GQQ, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil>
Dec 27 15:16:35.963221887 (527h47m35.834304922s) [noise=true , address=77.90.31.15:4711     , piece_id=WF6BG55JI62B3MLWCRVA2OPQ6Y7DJG35NRAGOB4MYQEVNZZUPCXQ, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil>
Dec 27 15:16:35.964808539 (527h47m35.835891584s) [noise=true , address=77.90.31.17:4798     , piece_id=Y7YSMCX3ZJUSIAXCWZ5UT6WULY7NDLEBT3DMVHAOZWQLLPUFQH3Q, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil>
Dec 27 15:16:35.976052790 (527h47m35.847135825s) [noise=true , address=77.90.31.16:4761     , piece_id=2LXURVZ2IGTJERJFBNCRIWMDBLGHZM7XTBWIQAK3XTZIWGXRTOAQ, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil>
Dec 27 15:16:36.055389572 (527h47m35.926472617s) [noise=true , address=77.90.31.11:4583     , piece_id=KPBSYNDRBV7TMMDDGBLWPS53XNRTPVGCSVGV2EC4FY24XGWKRILQ, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil>
Dec 27 15:16:36.056167062 (527h47m35.927250097s) [noise=true , address=77.90.31.10:4541     , piece_id=ESQWS4AAYNHVNZCP2JXOCYKHLOALVDUNK5XUS5XXUNX5ASFZR25Q, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil>
Dec 27 15:16:36.088861078 (527h47m35.959944113s) [noise=true , address=45.91.116.9:4275     , piece_id=DE6GCEP2XJMVWXVKBY6UVOIC4RTVNJMPGLFL3QSSEW6BKY5S2GUQ, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil>
Dec 27 15:16:36.193327481 (527h47m36.064410516s) [noise=true , address=45.91.116.13:4162    , piece_id=FFDCFG5I7KTPNADJREI2ZTCZS2OGHMDKCXALKV3VS4L3QPYNEGFQ, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil>
Dec 27 15:16:36.650250267 (527h47m36.521333302s) [noise=true , address=145.79.227.44:4204   , piece_id=KAJQZK26B2SDCRPBZTSR2B3DWNGLNFCWAGXEWVX57CUKQYBDSQAQ, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil>
Dec 27 15:16:36.692781133 (527h47m36.563864168s) [noise=true , address=145.79.227.40:4052   , piece_id=PYSSKZ4VCZNBC7AWLUHAXQ22HU5GJXY26KPLQJN33PJBECO3FLMA, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil>
Dec 27 15:16:36.715571250 (527h47m36.586654285s) [noise=true , address=145.79.227.33:4889   , piece_id=ZVZ27B3XMIBI5MRYQTNAOH75NWSSJYDGBQISESP6U3YLO3H5OQSQ, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil>
Dec 27 15:16:36.889959848 (527h47m36.761042883s) [noise=true , address=77.90.31.18:4816     , piece_id=OTO6TPBALIYRN7BHAO2NWWJA6MS5I25TLVA5C3MJ4VSQ7JDG5CXQ, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil>
Dec 27 15:16:36.973000061 (527h47m36.844083096s) [noise=true , address=145.79.227.30:4769   , piece_id=Z45MLEVRUQWQ2BVDUUBXBBEOZZWYYTHFNDE4LNRI2JLBUNHWD5GA, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil>
Dec 27 15:16:37.295879234 (527h47m37.166962269s) [noise=true , address=145.79.227.34:4919   , piece_id=LFCL2BTCUJ5HHTYUOUD4MBARGBWNM3LNVXGYALYZJYDT3L3QFSXQ, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil>
Dec 27 15:16:37.295899252 (527h47m37.166982297s) [seg_pos={0 0}, upload=462935]: Segment reached optimal threshold of 14 pieces.
Dec 27 15:16:37.296092550 (527h47m37.167175585s) [noise=true , address=77.90.31.13:4642     , piece_id=5E2LXQYSFJABRHXDB6XHG3AC44TN4ZYATKYIJAWGKIGIK4AHQJKA, seg_pos={0 0}, upload=462935]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xdfVQUEMLaGgw7hsBtHZVTKL9BTTxPeYve53efw8GPdSQDL9D): protocol: expected piece hash; context canceled; EOF
Dec 27 15:16:37.296098371 (527h47m37.167181406s) [noise=true , address=77.90.31.13:4642     , piece_id=5E2LXQYSFJABRHXDB6XHG3AC44TN4ZYATKYIJAWGKIGIK4AHQJKA, seg_pos={0 0}, upload=462935]: Long tail cancellation due to optimal threshold
Dec 27 15:16:37.296288694 (527h47m37.167371729s) [noise=true , address=77.90.31.21:4907     , piece_id=M6QS3DRLWECSW7DNJB742GLVARS74TXUEOCXL73NYH7A6AKZZLBQ, seg_pos={0 0}, upload=462935]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RCGnjZC1UT9SUmFirNhS9pRBtxjcy91ss7c3Mk2kKnnHn2F6s): protocol: expected piece hash; context canceled; EOF
Dec 27 15:16:37.296295597 (527h47m37.167378632s) [noise=true , address=77.90.31.21:4907     , piece_id=M6QS3DRLWECSW7DNJB742GLVARS74TXUEOCXL73NYH7A6AKZZLBQ, seg_pos={0 0}, upload=462935]: Long tail cancellation due to optimal threshold
Dec 27 15:16:37.296354058 (527h47m37.167437093s) [noise=true , address=64.49.13.169:4706    , piece_id=I7G6HRKRTFN5DBWC7B7XIXV5RT3I5MSRXIDEIEIHHW7DRAHC5IAA, seg_pos={0 0}, upload=462935]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:54418->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:54418->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54418->64.49.13.169:4706: use of closed network connection
Dec 27 15:16:37.296371711 (527h47m37.167454746s) [noise=true , address=64.49.13.169:4706    , piece_id=I7G6HRKRTFN5DBWC7B7XIXV5RT3I5MSRXIDEIEIHHW7DRAHC5IAA, seg_pos={0 0}, upload=462935]: Long tail cancellation due to optimal threshold
Dec 27 15:16:37.296451423 (527h47m37.167534458s) [noise=true , address=145.79.227.38:4984   , piece_id=R2DBGTZL6XK4MONKDYNU3AUMA7YXTGIUUZXM5CYDGP2TTFZNTYNQ, seg_pos={0 0}, upload=462935]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gozVE5KTcqGagY5veHpMqMQa3QHaNumynNSPvSb6LsKBHALuv): protocol: write tcp 188.166.225.170:38938->145.79.227.38:4984: use of closed network connection; write tcp 188.166.225.170:38938->145.79.227.38:4984: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38938->145.79.227.38:4984: use of closed network connection
Dec 27 15:16:37.296457736 (527h47m37.167540781s) [noise=true , address=145.79.227.38:4984   , piece_id=R2DBGTZL6XK4MONKDYNU3AUMA7YXTGIUUZXM5CYDGP2TTFZNTYNQ, seg_pos={0 0}, upload=462935]: Long tail cancellation due to optimal threshold
Dec 27 15:16:37.296482552 (527h47m37.167565587s) [seg_pos={0 0}, upload=462935]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 15:16:37.296569268 (527h47m37.167652303s) [upload=462935]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 15:16:37.296573766 (527h47m37.167656811s) [upload=462935]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 15:16:37.598452278 (527h47m37.469535313s) [upload=462935]: Done uploading segments...
Dec 27 15:16:37.598470573 (527h47m37.469553609s) [upload=462935]: Upload finished. err: <nil>
Duration: 43.172906429s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
21 out of 60 is failed (threshold=3) [.X...X.....X..XX.......X..X.X..X...XXXXXXX..X.X..X.X.....X..]
duration: 
5.975241392s
trace: 
https://jaeger.dev.storj.io/trace/4340d7129ab7e03f
FAIL (2025-12-27 15:21:31 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: THLECT7WT7WVNNGDX2WK7IF4B47DAGAOYZV37NCUJYRIGQ3BB3PQ) context deadline exceeded; piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: FSSSGIHYQDHIPTB7DGP5IOFCDKEKBBIRMZKUSNGROYLBEHBVNQHA) context deadline exceeded; piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: QNWOWBKGGHKTF72OW3LUXZE2BCFL6FXVGEJGCVBL3XYBKRHCDA2A) context deadline exceeded; piecestore: (Node ID: 1tGpuYdPLiy4ncDjxrZbQfp7Bq79NVBHDmr4vjaL4qMpudrvgR, Piece ID: O4I7P5LAMRLSIOJOV2XGRVSZ24IJFMRVK3YEFSPAYO3ZXGHT4TGQ) context deadline exceeded
======LOGS======
Dec 27 15:19:35.326624967 (527h50m35.197708002s) [upload=462977]: Starting upload
Dec 27 15:19:35.326698597 (527h50m35.197781642s) [upload=462977]: Done starting upload
Dec 27 15:19:35.326775264 (527h50m35.197858299s) [upload=462977]: Uploading segments...
Dec 27 15:19:35.326900632 (527h50m35.197983667s) [upload=462977]: Got next segment. Inline: false
Dec 27 15:19:35.327029468 (527h50m35.198112513s) [upload=462977]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 15:19:35.327035970 (527h50m35.198119015s) [upload=462977]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 15:19:35.575151438 (527h50m35.446234493s) [seg_pos={0 0}, upload=462977]: Begin upload segment...
Dec 27 15:19:35.575277387 (527h50m35.446360432s) [seg_pos={0 0}, upload=462977]: Done begin upload segment.
Dec 27 15:19:35.575344425 (527h50m35.44642747s) [noise=true , address=145.79.227.35:4481   , piece_id=YNSBY6Q4UBO2O46Q3EW3NH6YDFW4BQAHJ7TESZ7O34LCS7KT6OIA, seg_pos={0 0}, upload=462977]: Uploading piece...
Dec 27 15:19:35.575358512 (527h50m35.446441557s) [noise=true , address=77.90.31.16:4736     , piece_id=6TO6C34RUNZ454OIKSMLO2ZAC72SPFS3QGMUFKTH6UOZSBMQATCA, seg_pos={0 0}, upload=462977]: Uploading piece...
Dec 27 15:19:35.575431381 (527h50m35.446514436s) [noise=true , address=145.79.227.52:4447   , piece_id=ZVUSMFX4YQSTZ36TSJI5NRQ3BE5JUTCMFY54ZIH2TAKZCMFDF37Q, seg_pos={0 0}, upload=462977]: Uploading piece...
Dec 27 15:19:35.576020322 (527h50m35.447103368s) [noise=true , address=45.91.116.13:4155    , piece_id=MOLWGIRUFYBMUPHOQJ42SOWGPP6UW5XPKW4FBBVP56RZCATFV67A, seg_pos={0 0}, upload=462977]: Uploading piece...
Dec 27 15:19:35.576457054 (527h50m35.447540089s) [noise=true , address=145.79.227.39:4015   , piece_id=SF7UPUO5NBSSOZ7NYSIP5DERX24YGU4VUGJ3X2NZGRBIWXRG2VEQ, seg_pos={0 0}, upload=462977]: Uploading piece...
Dec 27 15:19:35.576903584 (527h50m35.447986629s) [noise=true , address=45.91.116.7:4009     , piece_id=P7N3J663OUERIIYZAAQD3TRIIHCNPJRPEJVJZCR7POTV6SLSTRRA, seg_pos={0 0}, upload=462977]: Uploading piece...
Dec 27 15:19:35.577265433 (527h50m35.448348478s) [noise=true , address=45.91.116.10:4084    , piece_id=QNWOWBKGGHKTF72OW3LUXZE2BCFL6FXVGEJGCVBL3XYBKRHCDA2A, seg_pos={0 0}, upload=462977]: Uploading piece...
Dec 27 15:19:35.577781526 (527h50m35.448864561s) [noise=true , address=145.79.227.48:4302   , piece_id=O4I7P5LAMRLSIOJOV2XGRVSZ24IJFMRVK3YEFSPAYO3ZXGHT4TGQ, seg_pos={0 0}, upload=462977]: Uploading piece...
Dec 27 15:19:35.578261690 (527h50m35.449344735s) [noise=true , address=145.79.227.16:4911   , piece_id=GHH4FKVNY2ZSLHCZ5BS7H5HVV7XVA4QZMWIBYNOZE63UXJRKPMUQ, seg_pos={0 0}, upload=462977]: Uploading piece...
Dec 27 15:19:35.578982232 (527h50m35.450065267s) [noise=true , address=45.91.116.9:4261     , piece_id=6QG7D53YB6KPKLD2MYSTW7XZPBXGQKTC5LF4OIC2WX57HA7QFQ3A, seg_pos={0 0}, upload=462977]: Uploading piece...
Dec 27 15:19:35.580552592 (527h50m35.451635638s) [noise=true , address=145.79.227.30:4762   , piece_id=Q4UOW45D4MREICS4RQI3LJWW74VHJBPNJVE5FGNJPSRFCYWSR2NQ, seg_pos={0 0}, upload=462977]: Uploading piece...
Dec 27 15:19:35.580608849 (527h50m35.451691894s) [noise=true , address=77.90.31.11:4556     , piece_id=FSSSGIHYQDHIPTB7DGP5IOFCDKEKBBIRMZKUSNGROYLBEHBVNQHA, seg_pos={0 0}, upload=462977]: Uploading piece...
Dec 27 15:19:35.582562860 (527h50m35.453645905s) [noise=true , address=77.90.31.10:4518     , piece_id=THLECT7WT7WVNNGDX2WK7IF4B47DAGAOYZV37NCUJYRIGQ3BB3PQ, seg_pos={0 0}, upload=462977]: Uploading piece...
Dec 27 15:19:35.584806623 (527h50m35.455889668s) [noise=true , address=145.79.227.42:4121   , piece_id=ZL34UCFOPFF3QGIHHUVBZGXVGWMNC2AKCMFXZOWOBEHOFU4BJJKQ, seg_pos={0 0}, upload=462977]: Uploading piece...
Dec 27 15:19:35.587466858 (527h50m35.458549913s) [noise=true , address=145.79.227.47:4265   , piece_id=GXLBVVMRFYGXCGUEHWMCV4XM3QRZS54ITGKOC6NSXZQZ7WK6KOFQ, seg_pos={0 0}, upload=462977]: Uploading piece...
Dec 27 15:19:35.589399328 (527h50m35.460482373s) [noise=true , address=145.79.227.10:4403   , piece_id=6UINIY4GZXC2IKPPFZTQD3EVJGIDIY5RY2SJZVW3POHCNT33LZZQ, seg_pos={0 0}, upload=462977]: Uploading piece...
Dec 27 15:19:37.294264078 (527h50m37.165347113s) [upload=462977]: Next returned nil segment
Dec 27 15:19:37.294277052 (527h50m37.165360087s) [upload=462977]: Waiting for error group managing segments...
Dec 27 15:19:37.584769364 (527h50m37.455852399s) [noise=true , address=145.79.227.42:4121   , piece_id=ZL34UCFOPFF3QGIHHUVBZGXVGWMNC2AKCMFXZOWOBEHOFU4BJJKQ, seg_pos={0 0}, upload=462977]: Done uploading piece. err: <nil>
Dec 27 15:19:37.587919692 (527h50m37.459002737s) [noise=true , address=145.79.227.48:4302   , piece_id=O4I7P5LAMRLSIOJOV2XGRVSZ24IJFMRVK3YEFSPAYO3ZXGHT4TGQ, seg_pos={0 0}, upload=462977]: Done uploading piece. err: <nil>
Dec 27 15:19:37.595538479 (527h50m37.466621524s) [noise=true , address=145.79.227.10:4403   , piece_id=6UINIY4GZXC2IKPPFZTQD3EVJGIDIY5RY2SJZVW3POHCNT33LZZQ, seg_pos={0 0}, upload=462977]: Done uploading piece. err: <nil>
Dec 27 15:19:37.597846745 (527h50m37.46892979s) [noise=true , address=145.79.227.39:4015   , piece_id=SF7UPUO5NBSSOZ7NYSIP5DERX24YGU4VUGJ3X2NZGRBIWXRG2VEQ, seg_pos={0 0}, upload=462977]: Done uploading piece. err: <nil>
Dec 27 15:19:37.601914871 (527h50m37.472997906s) [noise=true , address=145.79.227.35:4481   , piece_id=YNSBY6Q4UBO2O46Q3EW3NH6YDFW4BQAHJ7TESZ7O34LCS7KT6OIA, seg_pos={0 0}, upload=462977]: Done uploading piece. err: <nil>
Dec 27 15:19:37.610231108 (527h50m37.481314163s) [noise=true , address=145.79.227.16:4911   , piece_id=GHH4FKVNY2ZSLHCZ5BS7H5HVV7XVA4QZMWIBYNOZE63UXJRKPMUQ, seg_pos={0 0}, upload=462977]: Done uploading piece. err: <nil>
Dec 27 15:19:37.610409387 (527h50m37.481492432s) [noise=true , address=145.79.227.47:4265   , piece_id=GXLBVVMRFYGXCGUEHWMCV4XM3QRZS54ITGKOC6NSXZQZ7WK6KOFQ, seg_pos={0 0}, upload=462977]: Done uploading piece. err: <nil>
Dec 27 15:19:37.611494423 (527h50m37.482577468s) [noise=true , address=145.79.227.52:4447   , piece_id=ZVUSMFX4YQSTZ36TSJI5NRQ3BE5JUTCMFY54ZIH2TAKZCMFDF37Q, seg_pos={0 0}, upload=462977]: Done uploading piece. err: <nil>
Dec 27 15:19:37.802153223 (527h50m37.673236258s) [noise=true , address=45.91.116.7:4009     , piece_id=P7N3J663OUERIIYZAAQD3TRIIHCNPJRPEJVJZCR7POTV6SLSTRRA, seg_pos={0 0}, upload=462977]: Done uploading piece. err: <nil>
Dec 27 15:19:37.847871228 (527h50m37.718954283s) [noise=true , address=77.90.31.11:4556     , piece_id=FSSSGIHYQDHIPTB7DGP5IOFCDKEKBBIRMZKUSNGROYLBEHBVNQHA, seg_pos={0 0}, upload=462977]: Done uploading piece. err: <nil>
Dec 27 15:19:37.919243645 (527h50m37.79032668s) [noise=true , address=45.91.116.10:4084    , piece_id=QNWOWBKGGHKTF72OW3LUXZE2BCFL6FXVGEJGCVBL3XYBKRHCDA2A, seg_pos={0 0}, upload=462977]: Done uploading piece. err: <nil>
Dec 27 15:19:37.986813229 (527h50m37.857896264s) [noise=true , address=45.91.116.13:4155    , piece_id=MOLWGIRUFYBMUPHOQJ42SOWGPP6UW5XPKW4FBBVP56RZCATFV67A, seg_pos={0 0}, upload=462977]: Done uploading piece. err: <nil>
Dec 27 15:19:38.458986455 (527h50m38.33006949s) [noise=true , address=77.90.31.10:4518     , piece_id=THLECT7WT7WVNNGDX2WK7IF4B47DAGAOYZV37NCUJYRIGQ3BB3PQ, seg_pos={0 0}, upload=462977]: Done uploading piece. err: <nil>
Dec 27 15:19:38.459004680 (527h50m38.330087716s) [seg_pos={0 0}, upload=462977]: Segment reached optimal threshold of 13 pieces.
Dec 27 15:19:38.459183651 (527h50m38.330266696s) [noise=true , address=145.79.227.30:4762   , piece_id=Q4UOW45D4MREICS4RQI3LJWW74VHJBPNJVE5FGNJPSRFCYWSR2NQ, seg_pos={0 0}, upload=462977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF
Dec 27 15:19:38.459189492 (527h50m38.330272527s) [noise=true , address=145.79.227.30:4762   , piece_id=Q4UOW45D4MREICS4RQI3LJWW74VHJBPNJVE5FGNJPSRFCYWSR2NQ, seg_pos={0 0}, upload=462977]: Long tail cancellation due to optimal threshold
Dec 27 15:19:38.459294222 (527h50m38.330377257s) [noise=true , address=45.91.116.9:4261     , piece_id=6QG7D53YB6KPKLD2MYSTW7XZPBXGQKTC5LF4OIC2WX57HA7QFQ3A, seg_pos={0 0}, upload=462977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF
Dec 27 15:19:38.459299892 (527h50m38.330382937s) [noise=true , address=45.91.116.9:4261     , piece_id=6QG7D53YB6KPKLD2MYSTW7XZPBXGQKTC5LF4OIC2WX57HA7QFQ3A, seg_pos={0 0}, upload=462977]: Long tail cancellation due to optimal threshold
Dec 27 15:19:38.459463304 (527h50m38.330546339s) [noise=true , address=77.90.31.16:4736     , piece_id=6TO6C34RUNZ454OIKSMLO2ZAC72SPFS3QGMUFKTH6UOZSBMQATCA, seg_pos={0 0}, upload=462977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF
Dec 27 15:19:38.459470407 (527h50m38.330553442s) [noise=true , address=77.90.31.16:4736     , piece_id=6TO6C34RUNZ454OIKSMLO2ZAC72SPFS3QGMUFKTH6UOZSBMQATCA, seg_pos={0 0}, upload=462977]: Long tail cancellation due to optimal threshold
Dec 27 15:19:38.459491307 (527h50m38.330574342s) [seg_pos={0 0}, upload=462977]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 15:19:38.459555860 (527h50m38.330638895s) [upload=462977]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 15:19:38.459561571 (527h50m38.330644606s) [upload=462977]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 15:19:38.757600261 (527h50m38.628683316s) [upload=462977]: Done uploading segments...
Dec 27 15:19:38.757625068 (527h50m38.628708123s) [upload=462977]: Upload finished. err: <nil>
Duration: 5.975271118s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
35 out of 60 is failed (threshold=6) [..XXXXXXX.....XXX..X..XX.XXXX.....XXX.X.XXXX.X.XXXX...XXXX.X]
duration: 
5.000871009s
trace: 
https://jaeger.dev.storj.io/trace/29e2dcba2798532e
FAIL (2025-12-27 15:21:01 UTC) 
orbiter warning: context deadline exceeded
Duration: 5.000896316s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
21 out of 60 is failed (threshold=3) [..X...X.....X..XX.......X..X.X..X...XXXXXXX..X.X..X.X.....X.]
duration: 
6.045625277s
trace: 
https://jaeger.dev.storj.io/trace/4340d7129ab7e03f
FAIL (2025-12-27 15:20:52 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: THLECT7WT7WVNNGDX2WK7IF4B47DAGAOYZV37NCUJYRIGQ3BB3PQ) context deadline exceeded; piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: FSSSGIHYQDHIPTB7DGP5IOFCDKEKBBIRMZKUSNGROYLBEHBVNQHA) context deadline exceeded; piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: QNWOWBKGGHKTF72OW3LUXZE2BCFL6FXVGEJGCVBL3XYBKRHCDA2A) context deadline exceeded; piecestore: (Node ID: 1tGpuYdPLiy4ncDjxrZbQfp7Bq79NVBHDmr4vjaL4qMpudrvgR, Piece ID: O4I7P5LAMRLSIOJOV2XGRVSZ24IJFMRVK3YEFSPAYO3ZXGHT4TGQ) context deadline exceeded
======LOGS======
Dec 27 15:19:35.326624967 (527h50m35.197708002s) [upload=462977]: Starting upload
Dec 27 15:19:35.326698597 (527h50m35.197781642s) [upload=462977]: Done starting upload
Dec 27 15:19:35.326775264 (527h50m35.197858299s) [upload=462977]: Uploading segments...
Dec 27 15:19:35.326900632 (527h50m35.197983667s) [upload=462977]: Got next segment. Inline: false
Dec 27 15:19:35.327029468 (527h50m35.198112513s) [upload=462977]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 15:19:35.327035970 (527h50m35.198119015s) [upload=462977]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 15:19:35.575151438 (527h50m35.446234493s) [seg_pos={0 0}, upload=462977]: Begin upload segment...
Dec 27 15:19:35.575277387 (527h50m35.446360432s) [seg_pos={0 0}, upload=462977]: Done begin upload segment.
Dec 27 15:19:35.575344425 (527h50m35.44642747s) [noise=true , address=145.79.227.35:4481   , piece_id=YNSBY6Q4UBO2O46Q3EW3NH6YDFW4BQAHJ7TESZ7O34LCS7KT6OIA, seg_pos={0 0}, upload=462977]: Uploading piece...
Dec 27 15:19:35.575358512 (527h50m35.446441557s) [noise=true , address=77.90.31.16:4736     , piece_id=6TO6C34RUNZ454OIKSMLO2ZAC72SPFS3QGMUFKTH6UOZSBMQATCA, seg_pos={0 0}, upload=462977]: Uploading piece...
Dec 27 15:19:35.575431381 (527h50m35.446514436s) [noise=true , address=145.79.227.52:4447   , piece_id=ZVUSMFX4YQSTZ36TSJI5NRQ3BE5JUTCMFY54ZIH2TAKZCMFDF37Q, seg_pos={0 0}, upload=462977]: Uploading piece...
Dec 27 15:19:35.576020322 (527h50m35.447103368s) [noise=true , address=45.91.116.13:4155    , piece_id=MOLWGIRUFYBMUPHOQJ42SOWGPP6UW5XPKW4FBBVP56RZCATFV67A, seg_pos={0 0}, upload=462977]: Uploading piece...
Dec 27 15:19:35.576457054 (527h50m35.447540089s) [noise=true , address=145.79.227.39:4015   , piece_id=SF7UPUO5NBSSOZ7NYSIP5DERX24YGU4VUGJ3X2NZGRBIWXRG2VEQ, seg_pos={0 0}, upload=462977]: Uploading piece...
Dec 27 15:19:35.576903584 (527h50m35.447986629s) [noise=true , address=45.91.116.7:4009     , piece_id=P7N3J663OUERIIYZAAQD3TRIIHCNPJRPEJVJZCR7POTV6SLSTRRA, seg_pos={0 0}, upload=462977]: Uploading piece...
Dec 27 15:19:35.577265433 (527h50m35.448348478s) [noise=true , address=45.91.116.10:4084    , piece_id=QNWOWBKGGHKTF72OW3LUXZE2BCFL6FXVGEJGCVBL3XYBKRHCDA2A, seg_pos={0 0}, upload=462977]: Uploading piece...
Dec 27 15:19:35.577781526 (527h50m35.448864561s) [noise=true , address=145.79.227.48:4302   , piece_id=O4I7P5LAMRLSIOJOV2XGRVSZ24IJFMRVK3YEFSPAYO3ZXGHT4TGQ, seg_pos={0 0}, upload=462977]: Uploading piece...
Dec 27 15:19:35.578261690 (527h50m35.449344735s) [noise=true , address=145.79.227.16:4911   , piece_id=GHH4FKVNY2ZSLHCZ5BS7H5HVV7XVA4QZMWIBYNOZE63UXJRKPMUQ, seg_pos={0 0}, upload=462977]: Uploading piece...
Dec 27 15:19:35.578982232 (527h50m35.450065267s) [noise=true , address=45.91.116.9:4261     , piece_id=6QG7D53YB6KPKLD2MYSTW7XZPBXGQKTC5LF4OIC2WX57HA7QFQ3A, seg_pos={0 0}, upload=462977]: Uploading piece...
Dec 27 15:19:35.580552592 (527h50m35.451635638s) [noise=true , address=145.79.227.30:4762   , piece_id=Q4UOW45D4MREICS4RQI3LJWW74VHJBPNJVE5FGNJPSRFCYWSR2NQ, seg_pos={0 0}, upload=462977]: Uploading piece...
Dec 27 15:19:35.580608849 (527h50m35.451691894s) [noise=true , address=77.90.31.11:4556     , piece_id=FSSSGIHYQDHIPTB7DGP5IOFCDKEKBBIRMZKUSNGROYLBEHBVNQHA, seg_pos={0 0}, upload=462977]: Uploading piece...
Dec 27 15:19:35.582562860 (527h50m35.453645905s) [noise=true , address=77.90.31.10:4518     , piece_id=THLECT7WT7WVNNGDX2WK7IF4B47DAGAOYZV37NCUJYRIGQ3BB3PQ, seg_pos={0 0}, upload=462977]: Uploading piece...
Dec 27 15:19:35.584806623 (527h50m35.455889668s) [noise=true , address=145.79.227.42:4121   , piece_id=ZL34UCFOPFF3QGIHHUVBZGXVGWMNC2AKCMFXZOWOBEHOFU4BJJKQ, seg_pos={0 0}, upload=462977]: Uploading piece...
Dec 27 15:19:35.587466858 (527h50m35.458549913s) [noise=true , address=145.79.227.47:4265   , piece_id=GXLBVVMRFYGXCGUEHWMCV4XM3QRZS54ITGKOC6NSXZQZ7WK6KOFQ, seg_pos={0 0}, upload=462977]: Uploading piece...
Dec 27 15:19:35.589399328 (527h50m35.460482373s) [noise=true , address=145.79.227.10:4403   , piece_id=6UINIY4GZXC2IKPPFZTQD3EVJGIDIY5RY2SJZVW3POHCNT33LZZQ, seg_pos={0 0}, upload=462977]: Uploading piece...
Dec 27 15:19:37.294264078 (527h50m37.165347113s) [upload=462977]: Next returned nil segment
Dec 27 15:19:37.294277052 (527h50m37.165360087s) [upload=462977]: Waiting for error group managing segments...
Dec 27 15:19:37.584769364 (527h50m37.455852399s) [noise=true , address=145.79.227.42:4121   , piece_id=ZL34UCFOPFF3QGIHHUVBZGXVGWMNC2AKCMFXZOWOBEHOFU4BJJKQ, seg_pos={0 0}, upload=462977]: Done uploading piece. err: <nil>
Dec 27 15:19:37.587919692 (527h50m37.459002737s) [noise=true , address=145.79.227.48:4302   , piece_id=O4I7P5LAMRLSIOJOV2XGRVSZ24IJFMRVK3YEFSPAYO3ZXGHT4TGQ, seg_pos={0 0}, upload=462977]: Done uploading piece. err: <nil>
Dec 27 15:19:37.595538479 (527h50m37.466621524s) [noise=true , address=145.79.227.10:4403   , piece_id=6UINIY4GZXC2IKPPFZTQD3EVJGIDIY5RY2SJZVW3POHCNT33LZZQ, seg_pos={0 0}, upload=462977]: Done uploading piece. err: <nil>
Dec 27 15:19:37.597846745 (527h50m37.46892979s) [noise=true , address=145.79.227.39:4015   , piece_id=SF7UPUO5NBSSOZ7NYSIP5DERX24YGU4VUGJ3X2NZGRBIWXRG2VEQ, seg_pos={0 0}, upload=462977]: Done uploading piece. err: <nil>
Dec 27 15:19:37.601914871 (527h50m37.472997906s) [noise=true , address=145.79.227.35:4481   , piece_id=YNSBY6Q4UBO2O46Q3EW3NH6YDFW4BQAHJ7TESZ7O34LCS7KT6OIA, seg_pos={0 0}, upload=462977]: Done uploading piece. err: <nil>
Dec 27 15:19:37.610231108 (527h50m37.481314163s) [noise=true , address=145.79.227.16:4911   , piece_id=GHH4FKVNY2ZSLHCZ5BS7H5HVV7XVA4QZMWIBYNOZE63UXJRKPMUQ, seg_pos={0 0}, upload=462977]: Done uploading piece. err: <nil>
Dec 27 15:19:37.610409387 (527h50m37.481492432s) [noise=true , address=145.79.227.47:4265   , piece_id=GXLBVVMRFYGXCGUEHWMCV4XM3QRZS54ITGKOC6NSXZQZ7WK6KOFQ, seg_pos={0 0}, upload=462977]: Done uploading piece. err: <nil>
Dec 27 15:19:37.611494423 (527h50m37.482577468s) [noise=true , address=145.79.227.52:4447   , piece_id=ZVUSMFX4YQSTZ36TSJI5NRQ3BE5JUTCMFY54ZIH2TAKZCMFDF37Q, seg_pos={0 0}, upload=462977]: Done uploading piece. err: <nil>
Dec 27 15:19:37.802153223 (527h50m37.673236258s) [noise=true , address=45.91.116.7:4009     , piece_id=P7N3J663OUERIIYZAAQD3TRIIHCNPJRPEJVJZCR7POTV6SLSTRRA, seg_pos={0 0}, upload=462977]: Done uploading piece. err: <nil>
Dec 27 15:19:37.847871228 (527h50m37.718954283s) [noise=true , address=77.90.31.11:4556     , piece_id=FSSSGIHYQDHIPTB7DGP5IOFCDKEKBBIRMZKUSNGROYLBEHBVNQHA, seg_pos={0 0}, upload=462977]: Done uploading piece. err: <nil>
Dec 27 15:19:37.919243645 (527h50m37.79032668s) [noise=true , address=45.91.116.10:4084    , piece_id=QNWOWBKGGHKTF72OW3LUXZE2BCFL6FXVGEJGCVBL3XYBKRHCDA2A, seg_pos={0 0}, upload=462977]: Done uploading piece. err: <nil>
Dec 27 15:19:37.986813229 (527h50m37.857896264s) [noise=true , address=45.91.116.13:4155    , piece_id=MOLWGIRUFYBMUPHOQJ42SOWGPP6UW5XPKW4FBBVP56RZCATFV67A, seg_pos={0 0}, upload=462977]: Done uploading piece. err: <nil>
Dec 27 15:19:38.458986455 (527h50m38.33006949s) [noise=true , address=77.90.31.10:4518     , piece_id=THLECT7WT7WVNNGDX2WK7IF4B47DAGAOYZV37NCUJYRIGQ3BB3PQ, seg_pos={0 0}, upload=462977]: Done uploading piece. err: <nil>
Dec 27 15:19:38.459004680 (527h50m38.330087716s) [seg_pos={0 0}, upload=462977]: Segment reached optimal threshold of 13 pieces.
Dec 27 15:19:38.459183651 (527h50m38.330266696s) [noise=true , address=145.79.227.30:4762   , piece_id=Q4UOW45D4MREICS4RQI3LJWW74VHJBPNJVE5FGNJPSRFCYWSR2NQ, seg_pos={0 0}, upload=462977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF
Dec 27 15:19:38.459189492 (527h50m38.330272527s) [noise=true , address=145.79.227.30:4762   , piece_id=Q4UOW45D4MREICS4RQI3LJWW74VHJBPNJVE5FGNJPSRFCYWSR2NQ, seg_pos={0 0}, upload=462977]: Long tail cancellation due to optimal threshold
Dec 27 15:19:38.459294222 (527h50m38.330377257s) [noise=true , address=45.91.116.9:4261     , piece_id=6QG7D53YB6KPKLD2MYSTW7XZPBXGQKTC5LF4OIC2WX57HA7QFQ3A, seg_pos={0 0}, upload=462977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF
Dec 27 15:19:38.459299892 (527h50m38.330382937s) [noise=true , address=45.91.116.9:4261     , piece_id=6QG7D53YB6KPKLD2MYSTW7XZPBXGQKTC5LF4OIC2WX57HA7QFQ3A, seg_pos={0 0}, upload=462977]: Long tail cancellation due to optimal threshold
Dec 27 15:19:38.459463304 (527h50m38.330546339s) [noise=true , address=77.90.31.16:4736     , piece_id=6TO6C34RUNZ454OIKSMLO2ZAC72SPFS3QGMUFKTH6UOZSBMQATCA, seg_pos={0 0}, upload=462977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF
Dec 27 15:19:38.459470407 (527h50m38.330553442s) [noise=true , address=77.90.31.16:4736     , piece_id=6TO6C34RUNZ454OIKSMLO2ZAC72SPFS3QGMUFKTH6UOZSBMQATCA, seg_pos={0 0}, upload=462977]: Long tail cancellation due to optimal threshold
Dec 27 15:19:38.459491307 (527h50m38.330574342s) [seg_pos={0 0}, upload=462977]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 15:19:38.459555860 (527h50m38.330638895s) [upload=462977]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 15:19:38.459561571 (527h50m38.330644606s) [upload=462977]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 15:19:38.757600261 (527h50m38.628683316s) [upload=462977]: Done uploading segments...
Dec 27 15:19:38.757625068 (527h50m38.628708123s) [upload=462977]: Upload finished. err: <nil>
Duration: 6.045647007s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
6 out of 60 is failed (threshold=3) [X..X...........X..............X....X...................X....]
duration: 
47.93692753s
trace: 
https://jaeger.dev.storj.io/trace/51d2e44a5388dd3e
FAIL (2025-12-27 15:20:49 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1geUSD7qjW2yuMayTJ1Zy1KTYVC19adZ9nUWMRtmg2TktJSsm4, Piece ID: FFDCFG5I7KTPNADJREI2ZTCZS2OGHMDKCXALKV3VS4L3QPYNEGFQ) context deadline exceeded; piecestore: (Node ID: 12S3ShBLc5k6bJJVNB6p68zF4kWAyVfDbF2stUyZngzTNtaEGz8, Piece ID: DE6GCEP2XJMVWXVKBY6UVOIC4RTVNJMPGLFL3QSSEW6BKY5S2GUQ) context deadline exceeded; piecestore: (Node ID: 12Vno3GkFXR7KdTUEAhvesqif7KBF6DZoeMTv8n9MpRbYZNktaT, Piece ID: OTO6TPBALIYRN7BHAO2NWWJA6MS5I25TLVA5C3MJ4VSQ7JDG5CXQ) context deadline exceeded
======LOGS======
  , piece_id=Z45MLEVRUQWQ2BVDUUBXBBEOZZWYYTHFNDE4LNRI2JLBUNHWD5GA, seg_pos={0 0}, upload=462935]: Uploading piece...
Dec 27 15:16:33.887438882 (527h47m33.758521917s) [noise=true , address=145.79.227.40:4052   , piece_id=PYSSKZ4VCZNBC7AWLUHAXQ22HU5GJXY26KPLQJN33PJBECO3FLMA, seg_pos={0 0}, upload=462935]: Uploading piece...
Dec 27 15:16:33.887737339 (527h47m33.758820374s) [noise=true , address=77.90.31.16:4761     , piece_id=2LXURVZ2IGTJERJFBNCRIWMDBLGHZM7XTBWIQAK3XTZIWGXRTOAQ, seg_pos={0 0}, upload=462935]: Uploading piece...
Dec 27 15:16:33.888853915 (527h47m33.75993696s) [noise=true , address=64.49.13.169:4706    , piece_id=I7G6HRKRTFN5DBWC7B7XIXV5RT3I5MSRXIDEIEIHHW7DRAHC5IAA, seg_pos={0 0}, upload=462935]: Uploading piece...
Dec 27 15:16:33.889312548 (527h47m33.760395594s) [noise=true , address=145.79.227.38:4984   , piece_id=R2DBGTZL6XK4MONKDYNU3AUMA7YXTGIUUZXM5CYDGP2TTFZNTYNQ, seg_pos={0 0}, upload=462935]: Uploading piece...
Dec 27 15:16:33.889894367 (527h47m33.760977412s) [noise=true , address=45.91.116.9:4275     , piece_id=DE6GCEP2XJMVWXVKBY6UVOIC4RTVNJMPGLFL3QSSEW6BKY5S2GUQ, seg_pos={0 0}, upload=462935]: Uploading piece...
Dec 27 15:16:33.890793499 (527h47m33.761876544s) [noise=true , address=45.91.116.13:4162    , piece_id=FFDCFG5I7KTPNADJREI2ZTCZS2OGHMDKCXALKV3VS4L3QPYNEGFQ, seg_pos={0 0}, upload=462935]: Uploading piece...
Dec 27 15:16:33.890936371 (527h47m33.762019416s) [noise=true , address=77.90.31.10:4541     , piece_id=ESQWS4AAYNHVNZCP2JXOCYKHLOALVDUNK5XUS5XXUNX5ASFZR25Q, seg_pos={0 0}, upload=462935]: Uploading piece...
Dec 27 15:16:33.892310177 (527h47m33.763393222s) [noise=true , address=145.79.227.9:4373    , piece_id=HPWEICNYP5UEF34YGDETNXGZEENVZP7JH4ORK7SN5PJE3A762GQQ, seg_pos={0 0}, upload=462935]: Uploading piece...
Dec 27 15:16:33.893865538 (527h47m33.764948583s) [noise=true , address=145.79.227.33:4889   , piece_id=ZVZ27B3XMIBI5MRYQTNAOH75NWSSJYDGBQISESP6U3YLO3H5OQSQ, seg_pos={0 0}, upload=462935]: Uploading piece...
Dec 27 15:16:33.897117241 (527h47m33.768200286s) [noise=true , address=77.90.31.15:4711     , piece_id=WF6BG55JI62B3MLWCRVA2OPQ6Y7DJG35NRAGOB4MYQEVNZZUPCXQ, seg_pos={0 0}, upload=462935]: Uploading piece...
Dec 27 15:16:33.901733621 (527h47m33.772816666s) [noise=true , address=77.90.31.21:4907     , piece_id=M6QS3DRLWECSW7DNJB742GLVARS74TXUEOCXL73NYH7A6AKZZLBQ, seg_pos={0 0}, upload=462935]: Uploading piece...
Dec 27 15:16:33.902601734 (527h47m33.773684769s) [noise=true , address=145.79.227.34:4919   , piece_id=LFCL2BTCUJ5HHTYUOUD4MBARGBWNM3LNVXGYALYZJYDT3L3QFSXQ, seg_pos={0 0}, upload=462935]: Uploading piece...
Dec 27 15:16:33.904727542 (527h47m33.775810578s) [noise=true , address=77.90.31.18:4816     , piece_id=OTO6TPBALIYRN7BHAO2NWWJA6MS5I25TLVA5C3MJ4VSQ7JDG5CXQ, seg_pos={0 0}, upload=462935]: Uploading piece...
Dec 27 15:16:33.905748506 (527h47m33.776831551s) [noise=true , address=77.90.31.13:4642     , piece_id=5E2LXQYSFJABRHXDB6XHG3AC44TN4ZYATKYIJAWGKIGIK4AHQJKA, seg_pos={0 0}, upload=462935]: Uploading piece...
Dec 27 15:16:35.355522705 (527h47m35.22660574s) [upload=462935]: Next returned nil segment
Dec 27 15:16:35.355540480 (527h47m35.226623515s) [upload=462935]: Waiting for error group managing segments...
Dec 27 15:16:35.802401195 (527h47m35.67348423s) [noise=true , address=145.79.227.9:4373    , piece_id=HPWEICNYP5UEF34YGDETNXGZEENVZP7JH4ORK7SN5PJE3A762GQQ, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil>
Dec 27 15:16:35.963221887 (527h47m35.834304922s) [noise=true , address=77.90.31.15:4711     , piece_id=WF6BG55JI62B3MLWCRVA2OPQ6Y7DJG35NRAGOB4MYQEVNZZUPCXQ, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil>
Dec 27 15:16:35.964808539 (527h47m35.835891584s) [noise=true , address=77.90.31.17:4798     , piece_id=Y7YSMCX3ZJUSIAXCWZ5UT6WULY7NDLEBT3DMVHAOZWQLLPUFQH3Q, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil>
Dec 27 15:16:35.976052790 (527h47m35.847135825s) [noise=true , address=77.90.31.16:4761     , piece_id=2LXURVZ2IGTJERJFBNCRIWMDBLGHZM7XTBWIQAK3XTZIWGXRTOAQ, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil>
Dec 27 15:16:36.055389572 (527h47m35.926472617s) [noise=true , address=77.90.31.11:4583     , piece_id=KPBSYNDRBV7TMMDDGBLWPS53XNRTPVGCSVGV2EC4FY24XGWKRILQ, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil>
Dec 27 15:16:36.056167062 (527h47m35.927250097s) [noise=true , address=77.90.31.10:4541     , piece_id=ESQWS4AAYNHVNZCP2JXOCYKHLOALVDUNK5XUS5XXUNX5ASFZR25Q, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil>
Dec 27 15:16:36.088861078 (527h47m35.959944113s) [noise=true , address=45.91.116.9:4275     , piece_id=DE6GCEP2XJMVWXVKBY6UVOIC4RTVNJMPGLFL3QSSEW6BKY5S2GUQ, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil>
Dec 27 15:16:36.193327481 (527h47m36.064410516s) [noise=true , address=45.91.116.13:4162    , piece_id=FFDCFG5I7KTPNADJREI2ZTCZS2OGHMDKCXALKV3VS4L3QPYNEGFQ, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil>
Dec 27 15:16:36.650250267 (527h47m36.521333302s) [noise=true , address=145.79.227.44:4204   , piece_id=KAJQZK26B2SDCRPBZTSR2B3DWNGLNFCWAGXEWVX57CUKQYBDSQAQ, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil>
Dec 27 15:16:36.692781133 (527h47m36.563864168s) [noise=true , address=145.79.227.40:4052   , piece_id=PYSSKZ4VCZNBC7AWLUHAXQ22HU5GJXY26KPLQJN33PJBECO3FLMA, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil>
Dec 27 15:16:36.715571250 (527h47m36.586654285s) [noise=true , address=145.79.227.33:4889   , piece_id=ZVZ27B3XMIBI5MRYQTNAOH75NWSSJYDGBQISESP6U3YLO3H5OQSQ, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil>
Dec 27 15:16:36.889959848 (527h47m36.761042883s) [noise=true , address=77.90.31.18:4816     , piece_id=OTO6TPBALIYRN7BHAO2NWWJA6MS5I25TLVA5C3MJ4VSQ7JDG5CXQ, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil>
Dec 27 15:16:36.973000061 (527h47m36.844083096s) [noise=true , address=145.79.227.30:4769   , piece_id=Z45MLEVRUQWQ2BVDUUBXBBEOZZWYYTHFNDE4LNRI2JLBUNHWD5GA, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil>
Dec 27 15:16:37.295879234 (527h47m37.166962269s) [noise=true , address=145.79.227.34:4919   , piece_id=LFCL2BTCUJ5HHTYUOUD4MBARGBWNM3LNVXGYALYZJYDT3L3QFSXQ, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil>
Dec 27 15:16:37.295899252 (527h47m37.166982297s) [seg_pos={0 0}, upload=462935]: Segment reached optimal threshold of 14 pieces.
Dec 27 15:16:37.296092550 (527h47m37.167175585s) [noise=true , address=77.90.31.13:4642     , piece_id=5E2LXQYSFJABRHXDB6XHG3AC44TN4ZYATKYIJAWGKIGIK4AHQJKA, seg_pos={0 0}, upload=462935]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xdfVQUEMLaGgw7hsBtHZVTKL9BTTxPeYve53efw8GPdSQDL9D): protocol: expected piece hash; context canceled; EOF
Dec 27 15:16:37.296098371 (527h47m37.167181406s) [noise=true , address=77.90.31.13:4642     , piece_id=5E2LXQYSFJABRHXDB6XHG3AC44TN4ZYATKYIJAWGKIGIK4AHQJKA, seg_pos={0 0}, upload=462935]: Long tail cancellation due to optimal threshold
Dec 27 15:16:37.296288694 (527h47m37.167371729s) [noise=true , address=77.90.31.21:4907     , piece_id=M6QS3DRLWECSW7DNJB742GLVARS74TXUEOCXL73NYH7A6AKZZLBQ, seg_pos={0 0}, upload=462935]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RCGnjZC1UT9SUmFirNhS9pRBtxjcy91ss7c3Mk2kKnnHn2F6s): protocol: expected piece hash; context canceled; EOF
Dec 27 15:16:37.296295597 (527h47m37.167378632s) [noise=true , address=77.90.31.21:4907     , piece_id=M6QS3DRLWECSW7DNJB742GLVARS74TXUEOCXL73NYH7A6AKZZLBQ, seg_pos={0 0}, upload=462935]: Long tail cancellation due to optimal threshold
Dec 27 15:16:37.296354058 (527h47m37.167437093s) [noise=true , address=64.49.13.169:4706    , piece_id=I7G6HRKRTFN5DBWC7B7XIXV5RT3I5MSRXIDEIEIHHW7DRAHC5IAA, seg_pos={0 0}, upload=462935]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:54418->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:54418->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54418->64.49.13.169:4706: use of closed network connection
Dec 27 15:16:37.296371711 (527h47m37.167454746s) [noise=true , address=64.49.13.169:4706    , piece_id=I7G6HRKRTFN5DBWC7B7XIXV5RT3I5MSRXIDEIEIHHW7DRAHC5IAA, seg_pos={0 0}, upload=462935]: Long tail cancellation due to optimal threshold
Dec 27 15:16:37.296451423 (527h47m37.167534458s) [noise=true , address=145.79.227.38:4984   , piece_id=R2DBGTZL6XK4MONKDYNU3AUMA7YXTGIUUZXM5CYDGP2TTFZNTYNQ, seg_pos={0 0}, upload=462935]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gozVE5KTcqGagY5veHpMqMQa3QHaNumynNSPvSb6LsKBHALuv): protocol: write tcp 188.166.225.170:38938->145.79.227.38:4984: use of closed network connection; write tcp 188.166.225.170:38938->145.79.227.38:4984: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38938->145.79.227.38:4984: use of closed network connection
Dec 27 15:16:37.296457736 (527h47m37.167540781s) [noise=true , address=145.79.227.38:4984   , piece_id=R2DBGTZL6XK4MONKDYNU3AUMA7YXTGIUUZXM5CYDGP2TTFZNTYNQ, seg_pos={0 0}, upload=462935]: Long tail cancellation due to optimal threshold
Dec 27 15:16:37.296482552 (527h47m37.167565587s) [seg_pos={0 0}, upload=462935]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 15:16:37.296569268 (527h47m37.167652303s) [upload=462935]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 15:16:37.296573766 (527h47m37.167656811s) [upload=462935]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 15:16:37.598452278 (527h47m37.469535313s) [upload=462935]: Done uploading segments...
Dec 27 15:16:37.598470573 (527h47m37.469553609s) [upload=462935]: Upload finished. err: <nil>
Duration: 47.936942227s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
35 out of 60 is failed (threshold=6) [X..XXXXXXX.....XXX..X..XX.XXXX.....XXX.X.XXXX.X.XXXX...XXXX.]
duration: 
5.002042237s
trace: 
https://jaeger.dev.storj.io/trace/2fa9e2612d66081c
FAIL (2025-12-27 15:20:37 UTC) 
orbiter warning: context deadline exceeded
Duration: 5.002066553s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
21 out of 60 is failed (threshold=3) [...X...X.....X..XX.......X..X.X..X...XXXXXXX..X.X..X.X.....X]
duration: 
30.005936528s
trace: 
https://jaeger.dev.storj.io/trace/4340d7129ab7e03f
FAIL (2025-12-27 15:20:04 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: THLECT7WT7WVNNGDX2WK7IF4B47DAGAOYZV37NCUJYRIGQ3BB3PQ) context deadline exceeded; piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: FSSSGIHYQDHIPTB7DGP5IOFCDKEKBBIRMZKUSNGROYLBEHBVNQHA) context deadline exceeded; piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: QNWOWBKGGHKTF72OW3LUXZE2BCFL6FXVGEJGCVBL3XYBKRHCDA2A) context deadline exceeded; piecestore: (Node ID: 1tGpuYdPLiy4ncDjxrZbQfp7Bq79NVBHDmr4vjaL4qMpudrvgR, Piece ID: O4I7P5LAMRLSIOJOV2XGRVSZ24IJFMRVK3YEFSPAYO3ZXGHT4TGQ) context deadline exceeded
======LOGS======
Dec 27 15:19:35.326624967 (527h50m35.197708002s) [upload=462977]: Starting upload
Dec 27 15:19:35.326698597 (527h50m35.197781642s) [upload=462977]: Done starting upload
Dec 27 15:19:35.326775264 (527h50m35.197858299s) [upload=462977]: Uploading segments...
Dec 27 15:19:35.326900632 (527h50m35.197983667s) [upload=462977]: Got next segment. Inline: false
Dec 27 15:19:35.327029468 (527h50m35.198112513s) [upload=462977]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 15:19:35.327035970 (527h50m35.198119015s) [upload=462977]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 15:19:35.575151438 (527h50m35.446234493s) [seg_pos={0 0}, upload=462977]: Begin upload segment...
Dec 27 15:19:35.575277387 (527h50m35.446360432s) [seg_pos={0 0}, upload=462977]: Done begin upload segment.
Dec 27 15:19:35.575344425 (527h50m35.44642747s) [noise=true , address=145.79.227.35:4481   , piece_id=YNSBY6Q4UBO2O46Q3EW3NH6YDFW4BQAHJ7TESZ7O34LCS7KT6OIA, seg_pos={0 0}, upload=462977]: Uploading piece...
Dec 27 15:19:35.575358512 (527h50m35.446441557s) [noise=true , address=77.90.31.16:4736     , piece_id=6TO6C34RUNZ454OIKSMLO2ZAC72SPFS3QGMUFKTH6UOZSBMQATCA, seg_pos={0 0}, upload=462977]: Uploading piece...
Dec 27 15:19:35.575431381 (527h50m35.446514436s) [noise=true , address=145.79.227.52:4447   , piece_id=ZVUSMFX4YQSTZ36TSJI5NRQ3BE5JUTCMFY54ZIH2TAKZCMFDF37Q, seg_pos={0 0}, upload=462977]: Uploading piece...
Dec 27 15:19:35.576020322 (527h50m35.447103368s) [noise=true , address=45.91.116.13:4155    , piece_id=MOLWGIRUFYBMUPHOQJ42SOWGPP6UW5XPKW4FBBVP56RZCATFV67A, seg_pos={0 0}, upload=462977]: Uploading piece...
Dec 27 15:19:35.576457054 (527h50m35.447540089s) [noise=true , address=145.79.227.39:4015   , piece_id=SF7UPUO5NBSSOZ7NYSIP5DERX24YGU4VUGJ3X2NZGRBIWXRG2VEQ, seg_pos={0 0}, upload=462977]: Uploading piece...
Dec 27 15:19:35.576903584 (527h50m35.447986629s) [noise=true , address=45.91.116.7:4009     , piece_id=P7N3J663OUERIIYZAAQD3TRIIHCNPJRPEJVJZCR7POTV6SLSTRRA, seg_pos={0 0}, upload=462977]: Uploading piece...
Dec 27 15:19:35.577265433 (527h50m35.448348478s) [noise=true , address=45.91.116.10:4084    , piece_id=QNWOWBKGGHKTF72OW3LUXZE2BCFL6FXVGEJGCVBL3XYBKRHCDA2A, seg_pos={0 0}, upload=462977]: Uploading piece...
Dec 27 15:19:35.577781526 (527h50m35.448864561s) [noise=true , address=145.79.227.48:4302   , piece_id=O4I7P5LAMRLSIOJOV2XGRVSZ24IJFMRVK3YEFSPAYO3ZXGHT4TGQ, seg_pos={0 0}, upload=462977]: Uploading piece...
Dec 27 15:19:35.578261690 (527h50m35.449344735s) [noise=true , address=145.79.227.16:4911   , piece_id=GHH4FKVNY2ZSLHCZ5BS7H5HVV7XVA4QZMWIBYNOZE63UXJRKPMUQ, seg_pos={0 0}, upload=462977]: Uploading piece...
Dec 27 15:19:35.578982232 (527h50m35.450065267s) [noise=true , address=45.91.116.9:4261     , piece_id=6QG7D53YB6KPKLD2MYSTW7XZPBXGQKTC5LF4OIC2WX57HA7QFQ3A, seg_pos={0 0}, upload=462977]: Uploading piece...
Dec 27 15:19:35.580552592 (527h50m35.451635638s) [noise=true , address=145.79.227.30:4762   , piece_id=Q4UOW45D4MREICS4RQI3LJWW74VHJBPNJVE5FGNJPSRFCYWSR2NQ, seg_pos={0 0}, upload=462977]: Uploading piece...
Dec 27 15:19:35.580608849 (527h50m35.451691894s) [noise=true , address=77.90.31.11:4556     , piece_id=FSSSGIHYQDHIPTB7DGP5IOFCDKEKBBIRMZKUSNGROYLBEHBVNQHA, seg_pos={0 0}, upload=462977]: Uploading piece...
Dec 27 15:19:35.582562860 (527h50m35.453645905s) [noise=true , address=77.90.31.10:4518     , piece_id=THLECT7WT7WVNNGDX2WK7IF4B47DAGAOYZV37NCUJYRIGQ3BB3PQ, seg_pos={0 0}, upload=462977]: Uploading piece...
Dec 27 15:19:35.584806623 (527h50m35.455889668s) [noise=true , address=145.79.227.42:4121   , piece_id=ZL34UCFOPFF3QGIHHUVBZGXVGWMNC2AKCMFXZOWOBEHOFU4BJJKQ, seg_pos={0 0}, upload=462977]: Uploading piece...
Dec 27 15:19:35.587466858 (527h50m35.458549913s) [noise=true , address=145.79.227.47:4265   , piece_id=GXLBVVMRFYGXCGUEHWMCV4XM3QRZS54ITGKOC6NSXZQZ7WK6KOFQ, seg_pos={0 0}, upload=462977]: Uploading piece...
Dec 27 15:19:35.589399328 (527h50m35.460482373s) [noise=true , address=145.79.227.10:4403   , piece_id=6UINIY4GZXC2IKPPFZTQD3EVJGIDIY5RY2SJZVW3POHCNT33LZZQ, seg_pos={0 0}, upload=462977]: Uploading piece...
Dec 27 15:19:37.294264078 (527h50m37.165347113s) [upload=462977]: Next returned nil segment
Dec 27 15:19:37.294277052 (527h50m37.165360087s) [upload=462977]: Waiting for error group managing segments...
Dec 27 15:19:37.584769364 (527h50m37.455852399s) [noise=true , address=145.79.227.42:4121   , piece_id=ZL34UCFOPFF3QGIHHUVBZGXVGWMNC2AKCMFXZOWOBEHOFU4BJJKQ, seg_pos={0 0}, upload=462977]: Done uploading piece. err: <nil>
Dec 27 15:19:37.587919692 (527h50m37.459002737s) [noise=true , address=145.79.227.48:4302   , piece_id=O4I7P5LAMRLSIOJOV2XGRVSZ24IJFMRVK3YEFSPAYO3ZXGHT4TGQ, seg_pos={0 0}, upload=462977]: Done uploading piece. err: <nil>
Dec 27 15:19:37.595538479 (527h50m37.466621524s) [noise=true , address=145.79.227.10:4403   , piece_id=6UINIY4GZXC2IKPPFZTQD3EVJGIDIY5RY2SJZVW3POHCNT33LZZQ, seg_pos={0 0}, upload=462977]: Done uploading piece. err: <nil>
Dec 27 15:19:37.597846745 (527h50m37.46892979s) [noise=true , address=145.79.227.39:4015   , piece_id=SF7UPUO5NBSSOZ7NYSIP5DERX24YGU4VUGJ3X2NZGRBIWXRG2VEQ, seg_pos={0 0}, upload=462977]: Done uploading piece. err: <nil>
Dec 27 15:19:37.601914871 (527h50m37.472997906s) [noise=true , address=145.79.227.35:4481   , piece_id=YNSBY6Q4UBO2O46Q3EW3NH6YDFW4BQAHJ7TESZ7O34LCS7KT6OIA, seg_pos={0 0}, upload=462977]: Done uploading piece. err: <nil>
Dec 27 15:19:37.610231108 (527h50m37.481314163s) [noise=true , address=145.79.227.16:4911   , piece_id=GHH4FKVNY2ZSLHCZ5BS7H5HVV7XVA4QZMWIBYNOZE63UXJRKPMUQ, seg_pos={0 0}, upload=462977]: Done uploading piece. err: <nil>
Dec 27 15:19:37.610409387 (527h50m37.481492432s) [noise=true , address=145.79.227.47:4265   , piece_id=GXLBVVMRFYGXCGUEHWMCV4XM3QRZS54ITGKOC6NSXZQZ7WK6KOFQ, seg_pos={0 0}, upload=462977]: Done uploading piece. err: <nil>
Dec 27 15:19:37.611494423 (527h50m37.482577468s) [noise=true , address=145.79.227.52:4447   , piece_id=ZVUSMFX4YQSTZ36TSJI5NRQ3BE5JUTCMFY54ZIH2TAKZCMFDF37Q, seg_pos={0 0}, upload=462977]: Done uploading piece. err: <nil>
Dec 27 15:19:37.802153223 (527h50m37.673236258s) [noise=true , address=45.91.116.7:4009     , piece_id=P7N3J663OUERIIYZAAQD3TRIIHCNPJRPEJVJZCR7POTV6SLSTRRA, seg_pos={0 0}, upload=462977]: Done uploading piece. err: <nil>
Dec 27 15:19:37.847871228 (527h50m37.718954283s) [noise=true , address=77.90.31.11:4556     , piece_id=FSSSGIHYQDHIPTB7DGP5IOFCDKEKBBIRMZKUSNGROYLBEHBVNQHA, seg_pos={0 0}, upload=462977]: Done uploading piece. err: <nil>
Dec 27 15:19:37.919243645 (527h50m37.79032668s) [noise=true , address=45.91.116.10:4084    , piece_id=QNWOWBKGGHKTF72OW3LUXZE2BCFL6FXVGEJGCVBL3XYBKRHCDA2A, seg_pos={0 0}, upload=462977]: Done uploading piece. err: <nil>
Dec 27 15:19:37.986813229 (527h50m37.857896264s) [noise=true , address=45.91.116.13:4155    , piece_id=MOLWGIRUFYBMUPHOQJ42SOWGPP6UW5XPKW4FBBVP56RZCATFV67A, seg_pos={0 0}, upload=462977]: Done uploading piece. err: <nil>
Dec 27 15:19:38.458986455 (527h50m38.33006949s) [noise=true , address=77.90.31.10:4518     , piece_id=THLECT7WT7WVNNGDX2WK7IF4B47DAGAOYZV37NCUJYRIGQ3BB3PQ, seg_pos={0 0}, upload=462977]: Done uploading piece. err: <nil>
Dec 27 15:19:38.459004680 (527h50m38.330087716s) [seg_pos={0 0}, upload=462977]: Segment reached optimal threshold of 13 pieces.
Dec 27 15:19:38.459183651 (527h50m38.330266696s) [noise=true , address=145.79.227.30:4762   , piece_id=Q4UOW45D4MREICS4RQI3LJWW74VHJBPNJVE5FGNJPSRFCYWSR2NQ, seg_pos={0 0}, upload=462977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF
Dec 27 15:19:38.459189492 (527h50m38.330272527s) [noise=true , address=145.79.227.30:4762   , piece_id=Q4UOW45D4MREICS4RQI3LJWW74VHJBPNJVE5FGNJPSRFCYWSR2NQ, seg_pos={0 0}, upload=462977]: Long tail cancellation due to optimal threshold
Dec 27 15:19:38.459294222 (527h50m38.330377257s) [noise=true , address=45.91.116.9:4261     , piece_id=6QG7D53YB6KPKLD2MYSTW7XZPBXGQKTC5LF4OIC2WX57HA7QFQ3A, seg_pos={0 0}, upload=462977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF
Dec 27 15:19:38.459299892 (527h50m38.330382937s) [noise=true , address=45.91.116.9:4261     , piece_id=6QG7D53YB6KPKLD2MYSTW7XZPBXGQKTC5LF4OIC2WX57HA7QFQ3A, seg_pos={0 0}, upload=462977]: Long tail cancellation due to optimal threshold
Dec 27 15:19:38.459463304 (527h50m38.330546339s) [noise=true , address=77.90.31.16:4736     , piece_id=6TO6C34RUNZ454OIKSMLO2ZAC72SPFS3QGMUFKTH6UOZSBMQATCA, seg_pos={0 0}, upload=462977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF
Dec 27 15:19:38.459470407 (527h50m38.330553442s) [noise=true , address=77.90.31.16:4736     , piece_id=6TO6C34RUNZ454OIKSMLO2ZAC72SPFS3QGMUFKTH6UOZSBMQATCA, seg_pos={0 0}, upload=462977]: Long tail cancellation due to optimal threshold
Dec 27 15:19:38.459491307 (527h50m38.330574342s) [seg_pos={0 0}, upload=462977]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 15:19:38.459555860 (527h50m38.330638895s) [upload=462977]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 15:19:38.459561571 (527h50m38.330644606s) [upload=462977]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 15:19:38.757600261 (527h50m38.628683316s) [upload=462977]: Done uploading segments...
Dec 27 15:19:38.757625068 (527h50m38.628708123s) [upload=462977]: Upload finished. err: <nil>
Duration: 30.005956004s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
6 out of 60 is failed (threshold=3) [.X..X...........X..............X....X...................X...]
duration: 
50.93121874s
trace: 
https://jaeger.dev.storj.io/trace/51d2e44a5388dd3e
FAIL (2025-12-27 15:20:01 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1geUSD7qjW2yuMayTJ1Zy1KTYVC19adZ9nUWMRtmg2TktJSsm4, Piece ID: FFDCFG5I7KTPNADJREI2ZTCZS2OGHMDKCXALKV3VS4L3QPYNEGFQ) context deadline exceeded; piecestore: (Node ID: 12S3ShBLc5k6bJJVNB6p68zF4kWAyVfDbF2stUyZngzTNtaEGz8, Piece ID: DE6GCEP2XJMVWXVKBY6UVOIC4RTVNJMPGLFL3QSSEW6BKY5S2GUQ) context deadline exceeded; piecestore: (Node ID: 12Vno3GkFXR7KdTUEAhvesqif7KBF6DZoeMTv8n9MpRbYZNktaT, Piece ID: OTO6TPBALIYRN7BHAO2NWWJA6MS5I25TLVA5C3MJ4VSQ7JDG5CXQ) context deadline exceeded
======LOGS======
  , piece_id=Z45MLEVRUQWQ2BVDUUBXBBEOZZWYYTHFNDE4LNRI2JLBUNHWD5GA, seg_pos={0 0}, upload=462935]: Uploading piece...
Dec 27 15:16:33.887438882 (527h47m33.758521917s) [noise=true , address=145.79.227.40:4052   , piece_id=PYSSKZ4VCZNBC7AWLUHAXQ22HU5GJXY26KPLQJN33PJBECO3FLMA, seg_pos={0 0}, upload=462935]: Uploading piece...
Dec 27 15:16:33.887737339 (527h47m33.758820374s) [noise=true , address=77.90.31.16:4761     , piece_id=2LXURVZ2IGTJERJFBNCRIWMDBLGHZM7XTBWIQAK3XTZIWGXRTOAQ, seg_pos={0 0}, upload=462935]: Uploading piece...
Dec 27 15:16:33.888853915 (527h47m33.75993696s) [noise=true , address=64.49.13.169:4706    , piece_id=I7G6HRKRTFN5DBWC7B7XIXV5RT3I5MSRXIDEIEIHHW7DRAHC5IAA, seg_pos={0 0}, upload=462935]: Uploading piece...
Dec 27 15:16:33.889312548 (527h47m33.760395594s) [noise=true , address=145.79.227.38:4984   , piece_id=R2DBGTZL6XK4MONKDYNU3AUMA7YXTGIUUZXM5CYDGP2TTFZNTYNQ, seg_pos={0 0}, upload=462935]: Uploading piece...
Dec 27 15:16:33.889894367 (527h47m33.760977412s) [noise=true , address=45.91.116.9:4275     , piece_id=DE6GCEP2XJMVWXVKBY6UVOIC4RTVNJMPGLFL3QSSEW6BKY5S2GUQ, seg_pos={0 0}, upload=462935]: Uploading piece...
Dec 27 15:16:33.890793499 (527h47m33.761876544s) [noise=true , address=45.91.116.13:4162    , piece_id=FFDCFG5I7KTPNADJREI2ZTCZS2OGHMDKCXALKV3VS4L3QPYNEGFQ, seg_pos={0 0}, upload=462935]: Uploading piece...
Dec 27 15:16:33.890936371 (527h47m33.762019416s) [noise=true , address=77.90.31.10:4541     , piece_id=ESQWS4AAYNHVNZCP2JXOCYKHLOALVDUNK5XUS5XXUNX5ASFZR25Q, seg_pos={0 0}, upload=462935]: Uploading piece...
Dec 27 15:16:33.892310177 (527h47m33.763393222s) [noise=true , address=145.79.227.9:4373    , piece_id=HPWEICNYP5UEF34YGDETNXGZEENVZP7JH4ORK7SN5PJE3A762GQQ, seg_pos={0 0}, upload=462935]: Uploading piece...
Dec 27 15:16:33.893865538 (527h47m33.764948583s) [noise=true , address=145.79.227.33:4889   , piece_id=ZVZ27B3XMIBI5MRYQTNAOH75NWSSJYDGBQISESP6U3YLO3H5OQSQ, seg_pos={0 0}, upload=462935]: Uploading piece...
Dec 27 15:16:33.897117241 (527h47m33.768200286s) [noise=true , address=77.90.31.15:4711     , piece_id=WF6BG55JI62B3MLWCRVA2OPQ6Y7DJG35NRAGOB4MYQEVNZZUPCXQ, seg_pos={0 0}, upload=462935]: Uploading piece...
Dec 27 15:16:33.901733621 (527h47m33.772816666s) [noise=true , address=77.90.31.21:4907     , piece_id=M6QS3DRLWECSW7DNJB742GLVARS74TXUEOCXL73NYH7A6AKZZLBQ, seg_pos={0 0}, upload=462935]: Uploading piece...
Dec 27 15:16:33.902601734 (527h47m33.773684769s) [noise=true , address=145.79.227.34:4919   , piece_id=LFCL2BTCUJ5HHTYUOUD4MBARGBWNM3LNVXGYALYZJYDT3L3QFSXQ, seg_pos={0 0}, upload=462935]: Uploading piece...
Dec 27 15:16:33.904727542 (527h47m33.775810578s) [noise=true , address=77.90.31.18:4816     , piece_id=OTO6TPBALIYRN7BHAO2NWWJA6MS5I25TLVA5C3MJ4VSQ7JDG5CXQ, seg_pos={0 0}, upload=462935]: Uploading piece...
Dec 27 15:16:33.905748506 (527h47m33.776831551s) [noise=true , address=77.90.31.13:4642     , piece_id=5E2LXQYSFJABRHXDB6XHG3AC44TN4ZYATKYIJAWGKIGIK4AHQJKA, seg_pos={0 0}, upload=462935]: Uploading piece...
Dec 27 15:16:35.355522705 (527h47m35.22660574s) [upload=462935]: Next returned nil segment
Dec 27 15:16:35.355540480 (527h47m35.226623515s) [upload=462935]: Waiting for error group managing segments...
Dec 27 15:16:35.802401195 (527h47m35.67348423s) [noise=true , address=145.79.227.9:4373    , piece_id=HPWEICNYP5UEF34YGDETNXGZEENVZP7JH4ORK7SN5PJE3A762GQQ, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil>
Dec 27 15:16:35.963221887 (527h47m35.834304922s) [noise=true , address=77.90.31.15:4711     , piece_id=WF6BG55JI62B3MLWCRVA2OPQ6Y7DJG35NRAGOB4MYQEVNZZUPCXQ, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil>
Dec 27 15:16:35.964808539 (527h47m35.835891584s) [noise=true , address=77.90.31.17:4798     , piece_id=Y7YSMCX3ZJUSIAXCWZ5UT6WULY7NDLEBT3DMVHAOZWQLLPUFQH3Q, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil>
Dec 27 15:16:35.976052790 (527h47m35.847135825s) [noise=true , address=77.90.31.16:4761     , piece_id=2LXURVZ2IGTJERJFBNCRIWMDBLGHZM7XTBWIQAK3XTZIWGXRTOAQ, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil>
Dec 27 15:16:36.055389572 (527h47m35.926472617s) [noise=true , address=77.90.31.11:4583     , piece_id=KPBSYNDRBV7TMMDDGBLWPS53XNRTPVGCSVGV2EC4FY24XGWKRILQ, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil>
Dec 27 15:16:36.056167062 (527h47m35.927250097s) [noise=true , address=77.90.31.10:4541     , piece_id=ESQWS4AAYNHVNZCP2JXOCYKHLOALVDUNK5XUS5XXUNX5ASFZR25Q, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil>
Dec 27 15:16:36.088861078 (527h47m35.959944113s) [noise=true , address=45.91.116.9:4275     , piece_id=DE6GCEP2XJMVWXVKBY6UVOIC4RTVNJMPGLFL3QSSEW6BKY5S2GUQ, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil>
Dec 27 15:16:36.193327481 (527h47m36.064410516s) [noise=true , address=45.91.116.13:4162    , piece_id=FFDCFG5I7KTPNADJREI2ZTCZS2OGHMDKCXALKV3VS4L3QPYNEGFQ, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil>
Dec 27 15:16:36.650250267 (527h47m36.521333302s) [noise=true , address=145.79.227.44:4204   , piece_id=KAJQZK26B2SDCRPBZTSR2B3DWNGLNFCWAGXEWVX57CUKQYBDSQAQ, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil>
Dec 27 15:16:36.692781133 (527h47m36.563864168s) [noise=true , address=145.79.227.40:4052   , piece_id=PYSSKZ4VCZNBC7AWLUHAXQ22HU5GJXY26KPLQJN33PJBECO3FLMA, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil>
Dec 27 15:16:36.715571250 (527h47m36.586654285s) [noise=true , address=145.79.227.33:4889   , piece_id=ZVZ27B3XMIBI5MRYQTNAOH75NWSSJYDGBQISESP6U3YLO3H5OQSQ, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil>
Dec 27 15:16:36.889959848 (527h47m36.761042883s) [noise=true , address=77.90.31.18:4816     , piece_id=OTO6TPBALIYRN7BHAO2NWWJA6MS5I25TLVA5C3MJ4VSQ7JDG5CXQ, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil>
Dec 27 15:16:36.973000061 (527h47m36.844083096s) [noise=true , address=145.79.227.30:4769   , piece_id=Z45MLEVRUQWQ2BVDUUBXBBEOZZWYYTHFNDE4LNRI2JLBUNHWD5GA, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil>
Dec 27 15:16:37.295879234 (527h47m37.166962269s) [noise=true , address=145.79.227.34:4919   , piece_id=LFCL2BTCUJ5HHTYUOUD4MBARGBWNM3LNVXGYALYZJYDT3L3QFSXQ, seg_pos={0 0}, upload=462935]: Done uploading piece. err: <nil>
Dec 27 15:16:37.295899252 (527h47m37.166982297s) [seg_pos={0 0}, upload=462935]: Segment reached optimal threshold of 14 pieces.
Dec 27 15:16:37.296092550 (527h47m37.167175585s) [noise=true , address=77.90.31.13:4642     , piece_id=5E2LXQYSFJABRHXDB6XHG3AC44TN4ZYATKYIJAWGKIGIK4AHQJKA, seg_pos={0 0}, upload=462935]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xdfVQUEMLaGgw7hsBtHZVTKL9BTTxPeYve53efw8GPdSQDL9D): protocol: expected piece hash; context canceled; EOF
Dec 27 15:16:37.296098371 (527h47m37.167181406s) [noise=true , address=77.90.31.13:4642     , piece_id=5E2LXQYSFJABRHXDB6XHG3AC44TN4ZYATKYIJAWGKIGIK4AHQJKA, seg_pos={0 0}, upload=462935]: Long tail cancellation due to optimal threshold
Dec 27 15:16:37.296288694 (527h47m37.167371729s) [noise=true , address=77.90.31.21:4907     , piece_id=M6QS3DRLWECSW7DNJB742GLVARS74TXUEOCXL73NYH7A6AKZZLBQ, seg_pos={0 0}, upload=462935]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RCGnjZC1UT9SUmFirNhS9pRBtxjcy91ss7c3Mk2kKnnHn2F6s): protocol: expected piece hash; context canceled; EOF
Dec 27 15:16:37.296295597 (527h47m37.167378632s) [noise=true , address=77.90.31.21:4907     , piece_id=M6QS3DRLWECSW7DNJB742GLVARS74TXUEOCXL73NYH7A6AKZZLBQ, seg_pos={0 0}, upload=462935]: Long tail cancellation due to optimal threshold
Dec 27 15:16:37.296354058 (527h47m37.167437093s) [noise=true , address=64.49.13.169:4706    , piece_id=I7G6HRKRTFN5DBWC7B7XIXV5RT3I5MSRXIDEIEIHHW7DRAHC5IAA, seg_pos={0 0}, upload=462935]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:54418->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:54418->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54418->64.49.13.169:4706: use of closed network connection
Dec 27 15:16:37.296371711 (527h47m37.167454746s) [noise=true , address=64.49.13.169:4706    , piece_id=I7G6HRKRTFN5DBWC7B7XIXV5RT3I5MSRXIDEIEIHHW7DRAHC5IAA, seg_pos={0 0}, upload=462935]: Long tail cancellation due to optimal threshold
Dec 27 15:16:37.296451423 (527h47m37.167534458s) [noise=true , address=145.79.227.38:4984   , piece_id=R2DBGTZL6XK4MONKDYNU3AUMA7YXTGIUUZXM5CYDGP2TTFZNTYNQ, seg_pos={0 0}, upload=462935]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gozVE5KTcqGagY5veHpMqMQa3QHaNumynNSPvSb6LsKBHALuv): protocol: write tcp 188.166.225.170:38938->145.79.227.38:4984: use of closed network connection; write tcp 188.166.225.170:38938->145.79.227.38:4984: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38938->145.79.227.38:4984: use of closed network connection
Dec 27 15:16:37.296457736 (527h47m37.167540781s) [noise=true , address=145.79.227.38:4984   , piece_id=R2DBGTZL6XK4MONKDYNU3AUMA7YXTGIUUZXM5CYDGP2TTFZNTYNQ, seg_pos={0 0}, upload=462935]: Long tail cancellation due to optimal threshold
Dec 27 15:16:37.296482552 (527h47m37.167565587s) [seg_pos={0 0}, upload=462935]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 27 15:16:37.296569268 (527h47m37.167652303s) [upload=462935]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 15:16:37.296573766 (527h47m37.167656811s) [upload=462935]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 15:16:37.598452278 (527h47m37.469535313s) [upload=462935]: Done uploading segments...
Dec 27 15:16:37.598470573 (527h47m37.469553609s) [upload=462935]: Upload finished. err: <nil>
Duration: 50.93123986s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
36 out of 60 is failed (threshold=6) [XX..XXXXXXX.....XXX..X..XX.XXXX.....XXX.X.XXXX.X.XXXX...XXXX]
duration: 
5.001217325s
trace: 
https://jaeger.dev.storj.io/trace/2fa9e2612d66081c
FAIL (2025-12-27 15:19:27 UTC) 
orbiter warning: context deadline exceeded
Duration: 5.001239276s core uplink us-select-1 us1-placement-20