Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure

Orbiter Check History

Showing 50 of 362092 reports (Page 246 of 7242)
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [...........................................X................]
duration: 
6.808919399s
trace: 
https://jaeger.dev.storj.io/trace/5dd0609e3a2ac547
FAIL (2025-12-31 07:55:25 UTC) 
orbiter warning: context deadline exceeded
Duration: 6.808941914s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
19 out of 60 is failed (threshold=3) [..X.X.X.X.X......XXX.X....XXX..X...........X......X.X.X.XX..]
duration: 
5.890387932s
trace: 
https://jaeger.dev.storj.io/trace/6c9f60d6c2a165da
FAIL (2025-12-31 07:55:19 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: PW5BS4DU62NA7RUSDX3A2EYBWMQDU6CLVKOQ3LRDSUZSF4AUP64Q) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: ZQB3NTHBABFQ7FEQJNHEVPRTRF4K6C3FJV4SZ4KEIYM6ZCW7CCAA) context deadline exceeded
======LOGS======
Dec 31 07:53:38.027484676 (616h24m37.898567721s) [upload=540182]: Starting upload
Dec 31 07:53:38.027534512 (616h24m37.898617547s) [upload=540182]: Done starting upload
Dec 31 07:53:38.027801700 (616h24m37.898884735s) [upload=540182]: Uploading segments...
Dec 31 07:53:38.027979689 (616h24m37.899062734s) [upload=540182]: Got next segment. Inline: false
Dec 31 07:53:38.028073868 (616h24m37.899156903s) [upload=540182]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 07:53:38.028082775 (616h24m37.89916581s) [upload=540182]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 07:53:38.277891170 (616h24m38.148974205s) [seg_pos={0 0}, upload=540182]: Begin upload segment...
Dec 31 07:53:38.277968046 (616h24m38.149051081s) [seg_pos={0 0}, upload=540182]: Done begin upload segment.
Dec 31 07:53:38.278029704 (616h24m38.149112749s) [noise=true , address=45.91.116.6:4975     , piece_id=7H4553CB6VLOISS34HUKZLLQMIZFCF3CUH5L2GP2VU5M6TEIEHMA, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.278042758 (616h24m38.149125794s) [noise=true , address=77.90.31.12:4589     , piece_id=5E5XAACC5DCA6MOP4TFJ23AW3RAG3Q36MVHGSLZ5O2RMCIZNOG2A, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.278144813 (616h24m38.149227848s) [noise=true , address=145.79.227.50:4375   , piece_id=CXKJCYVE6SJOOVQWHH34MCITFOFEV27SP4OKAPXF4DJT3EXIHWGQ, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.278830739 (616h24m38.149913784s) [noise=true , address=77.90.31.19:4842     , piece_id=PW5BS4DU62NA7RUSDX3A2EYBWMQDU6CLVKOQ3LRDSUZSF4AUP64Q, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.279143605 (616h24m38.15022664s) [noise=true , address=77.90.31.15:4700     , piece_id=ZQB3NTHBABFQ7FEQJNHEVPRTRF4K6C3FJV4SZ4KEIYM6ZCW7CCAA, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.279489213 (616h24m38.150572258s) [noise=true , address=45.91.116.13:4154    , piece_id=LJFCLUXCGK56OQR3EFB73WBAHG7VFQ27Y7X2LYS4BPP7GYBBONUQ, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.279878865 (616h24m38.15096191s) [noise=true , address=145.79.227.26:4655   , piece_id=YHNBMQCH6QZRCTHVFDZRVHMXB2BP3MJ7B3RG3QIPBQ4PQW5JYB6A, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.280269218 (616h24m38.151352253s) [noise=true , address=145.79.227.37:4566   , piece_id=6E73Y5FUFAGQLU4IYUQNJ2J7XXXRJQGPM7KJRHMYQ3ENEBN76QKQ, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.280653881 (616h24m38.151736926s) [noise=true , address=145.79.227.24:4581   , piece_id=IKQBNNGA2QEVTMZZMGPMHLLIKLEDKZCL34X2DDIFZFHW2GBKLCXA, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.281852182 (616h24m38.152935227s) [noise=true , address=145.79.227.16:4910   , piece_id=6L6QC3KSIG6CVDPLG55O6Q24L3QZ72GL3OTPAETHJG7I4Z4WMYKA, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.281889793 (616h24m38.152972828s) [noise=true , address=145.79.227.45:4229   , piece_id=3EKCB2QETOCEMR2WCFYDZKVYT2R3CBOOKMUHXBMANKMOJD7H7IZA, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.283093776 (616h24m38.154176821s) [noise=true , address=145.79.227.12:4767   , piece_id=GGEX2BKWC4URKRI34IAR7Z2KAP62FVRDM3ELUTNIH6WB4QN3JLRQ, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.283497055 (616h24m38.15458009s) [noise=true , address=145.79.227.23:4545   , piece_id=F5KYOLWB2IZRRCKVYFGHFZ23UZV7FUR2PCF2HO7GSHEMKIY5IL3Q, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.284318098 (616h24m38.155401133s) [noise=true , address=145.79.227.51:4410   , piece_id=VYULD4SFKPQRYZSOPM2PUQC4LUQAQ4WXRYVSOSRYCA5N5LFV4FAA, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.285603535 (616h24m38.15668657s) [noise=true , address=145.79.227.52:4446   , piece_id=6FI2W5DULY5FFDFIFXLNSDUX7EEVDGB3WLWIEEZXFKNOR3AIOSIQ, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.286960870 (616h24m38.158043915s) [noise=true , address=145.79.227.18:4006   , piece_id=LFDSYTOQR42MSDEIRDUXGRNBUF3ONMCL6BJTMRNLNDHHNY5V7ZDQ, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:39.976475363 (616h24m39.847558408s) [upload=540182]: Next returned nil segment
Dec 31 07:53:39.976503577 (616h24m39.847586622s) [upload=540182]: Waiting for error group managing segments...
Dec 31 07:53:40.263089662 (616h24m40.134172697s) [noise=true , address=145.79.227.52:4446   , piece_id=6FI2W5DULY5FFDFIFXLNSDUX7EEVDGB3WLWIEEZXFKNOR3AIOSIQ, seg_pos={0 0}, upload=540182]: Done uploading piece. err: <nil>
Dec 31 07:53:40.263514351 (616h24m40.134597386s) [noise=true , address=145.79.227.16:4910   , piece_id=6L6QC3KSIG6CVDPLG55O6Q24L3QZ72GL3OTPAETHJG7I4Z4WMYKA, seg_pos={0 0}, upload=540182]: Done uploading piece. err: <nil>
Dec 31 07:53:40.266215304 (616h24m40.137298339s) [noise=true , address=145.79.227.12:4767   , piece_id=GGEX2BKWC4URKRI34IAR7Z2KAP62FVRDM3ELUTNIH6WB4QN3JLRQ, seg_pos={0 0}, upload=540182]: Done uploading piece. err: <nil>
Dec 31 07:53:40.274366505 (616h24m40.14544955s) [noise=true , address=145.79.227.51:4410   , piece_id=VYULD4SFKPQRYZSOPM2PUQC4LUQAQ4WXRYVSOSRYCA5N5LFV4FAA, seg_pos={0 0}, upload=540182]: Done uploading piece. err: <nil>
Dec 31 07:53:40.277241870 (616h24m40.148324905s) [noise=true , address=145.79.227.26:4655   , piece_id=YHNBMQCH6QZRCTHVFDZRVHMXB2BP3MJ7B3RG3QIPBQ4PQW5JYB6A, seg_pos={0 0}, upload=540182]: Done uploading piece. err: <nil>
Dec 31 07:53:40.277655779 (616h24m40.148738814s) [noise=true , address=145.79.227.37:4566   , piece_id=6E73Y5FUFAGQLU4IYUQNJ2J7XXXRJQGPM7KJRHMYQ3ENEBN76QKQ, seg_pos={0 0}, upload=540182]: Done uploading piece. err: <nil>
Dec 31 07:53:40.288485420 (616h24m40.159568465s) [noise=true , address=145.79.227.50:4375   , piece_id=CXKJCYVE6SJOOVQWHH34MCITFOFEV27SP4OKAPXF4DJT3EXIHWGQ, seg_pos={0 0}, upload=540182]: Done uploading piece. err: <nil>
Dec 31 07:53:40.290211387 (616h24m40.161294432s) [noise=true , address=145.79.227.18:4006   , piece_id=LFDSYTOQR42MSDEIRDUXGRNBUF3ONMCL6BJTMRNLNDHHNY5V7ZDQ, seg_pos={0 0}, upload=540182]: Done uploading piece. err: <nil>
Dec 31 07:53:40.293396742 (616h24m40.164479787s) [noise=true , address=145.79.227.24:4581   , piece_id=IKQBNNGA2QEVTMZZMGPMHLLIKLEDKZCL34X2DDIFZFHW2GBKLCXA, seg_pos={0 0}, upload=540182]: Done uploading piece. err: <nil>
Dec 31 07:53:40.302471963 (616h24m40.173554998s) [noise=true , address=145.79.227.45:4229   , piece_id=3EKCB2QETOCEMR2WCFYDZKVYT2R3CBOOKMUHXBMANKMOJD7H7IZA, seg_pos={0 0}, upload=540182]: Done uploading piece. err: <nil>
Dec 31 07:53:40.354700213 (616h24m40.225783248s) [noise=true , address=145.79.227.23:4545   , piece_id=F5KYOLWB2IZRRCKVYFGHFZ23UZV7FUR2PCF2HO7GSHEMKIY5IL3Q, seg_pos={0 0}, upload=540182]: Done uploading piece. err: <nil>
Dec 31 07:53:40.453140008 (616h24m40.324223043s) [noise=true , address=77.90.31.19:4842     , piece_id=PW5BS4DU62NA7RUSDX3A2EYBWMQDU6CLVKOQ3LRDSUZSF4AUP64Q, seg_pos={0 0}, upload=540182]: Done uploading piece. err: <nil>
Dec 31 07:53:40.527310551 (616h24m40.398393596s) [noise=true , address=77.90.31.15:4700     , piece_id=ZQB3NTHBABFQ7FEQJNHEVPRTRF4K6C3FJV4SZ4KEIYM6ZCW7CCAA, seg_pos={0 0}, upload=540182]: Done uploading piece. err: <nil>
Dec 31 07:53:40.527327724 (616h24m40.398410759s) [seg_pos={0 0}, upload=540182]: Segment reached optimal threshold of 13 pieces.
Dec 31 07:53:40.527495442 (616h24m40.398578477s) [noise=true , address=45.91.116.13:4154    , piece_id=LJFCLUXCGK56OQR3EFB73WBAHG7VFQ27Y7X2LYS4BPP7GYBBONUQ, seg_pos={0 0}, upload=540182]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF
Dec 31 07:53:40.527501184 (616h24m40.398584219s) [noise=true , address=45.91.116.13:4154    , piece_id=LJFCLUXCGK56OQR3EFB73WBAHG7VFQ27Y7X2LYS4BPP7GYBBONUQ, seg_pos={0 0}, upload=540182]: Long tail cancellation due to optimal threshold
Dec 31 07:53:40.527703939 (616h24m40.398786974s) [noise=true , address=77.90.31.12:4589     , piece_id=5E5XAACC5DCA6MOP4TFJ23AW3RAG3Q36MVHGSLZ5O2RMCIZNOG2A, seg_pos={0 0}, upload=540182]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: write tcp 188.166.225.170:59116->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:59116->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59116->77.90.31.12:4589: use of closed network connection
Dec 31 07:53:40.527709891 (616h24m40.398792927s) [noise=true , address=77.90.31.12:4589     , piece_id=5E5XAACC5DCA6MOP4TFJ23AW3RAG3Q36MVHGSLZ5O2RMCIZNOG2A, seg_pos={0 0}, upload=540182]: Long tail cancellation due to optimal threshold
Dec 31 07:53:40.528022527 (616h24m40.399105562s) [noise=true , address=45.91.116.6:4975     , piece_id=7H4553CB6VLOISS34HUKZLLQMIZFCF3CUH5L2GP2VU5M6TEIEHMA, seg_pos={0 0}, upload=540182]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF
Dec 31 07:53:40.528036814 (616h24m40.399119849s) [noise=true , address=45.91.116.6:4975     , piece_id=7H4553CB6VLOISS34HUKZLLQMIZFCF3CUH5L2GP2VU5M6TEIEHMA, seg_pos={0 0}, upload=540182]: Long tail cancellation due to optimal threshold
Dec 31 07:53:40.528063705 (616h24m40.39914674s) [seg_pos={0 0}, upload=540182]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 07:53:40.528171600 (616h24m40.399254635s) [upload=540182]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 07:53:40.528177852 (616h24m40.399260887s) [upload=540182]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 07:53:40.824344511 (616h24m40.695427566s) [upload=540182]: Done uploading segments...
Dec 31 07:53:40.824370379 (616h24m40.695453454s) [upload=540182]: Upload finished. err: <nil>
Duration: 5.890416497s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
4 out of 60 is failed (threshold=6) [................................X.........X............X..X.]
duration: 
3.536982714s
trace: 
https://jaeger.dev.storj.io/trace/e2308925a42e319
FAIL (2025-12-31 07:55:14 UTC) 
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.537018552s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
2 out of 60 is failed (threshold=3) [......X............X........................................]
duration: 
24.597363932s
trace: 
https://jaeger.dev.storj.io/trace/6ce9bac61d803c5a
FAIL (2025-12-31 07:55:03 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
56     , piece_id=3PKC4JZPZVBQOBUHPQAG4HZ67DRTOY7GR35QVWLV3LN5JHXPMENA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.554293132 (615h55m26.425376177s) [noise=true , address=145.79.227.43:4186   , piece_id=ENVLK652WUVPKR3VPGXXMJ4IAEFHK67JOODS26ASTM4I2V7ABUWA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.554505907 (615h55m26.425588942s) [noise=true , address=77.90.31.14:4673     , piece_id=XYDCDFNKMWJK4HDELS4NLBVLUBR76YHT4RP3M5JO5QMCSI6MOGYA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.554648990 (615h55m26.425732025s) [noise=true , address=64.49.13.140:4505    , piece_id=EBEDU3F3QMDLNIDSRRXUHGSZ3VH5ITKRDDI2QS5XKBOGKOCN3LNA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.554994317 (615h55m26.426077352s) [noise=true , address=64.49.13.169:4707    , piece_id=RB7LFR5DM7PUHCJPB5ABWN63Q2XQOGNOG2W2FCKXFWU5MGVPRPVA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.555494150 (615h55m26.426577185s) [noise=true , address=64.49.13.137:4500    , piece_id=DD4JKFRHB2FKWAOTS5RZU5M3ASDGJTRX2XRB5ZROVOZNPN6BER2Q, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.555978812 (615h55m26.427061847s) [noise=true , address=45.91.116.10:4087    , piece_id=MJUIJFY6KEXEMMQKWA2NC3RE5PG55QCDZ7ECWRZG7CBROSA7LL3Q, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.556325282 (615h55m26.427408317s) [noise=true , address=145.79.227.29:4965   , piece_id=L6L2H7KMBTYKC24ZMACWRYK6TGOK5KHDO2YCFP5TJS66NXXP3KNQ, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.556728720 (615h55m26.427811765s) [noise=true , address=45.91.116.6:4977     , piece_id=G7JKD7INUS3D3RUIHSSVYCSOVTHSRG5HLVKERSCAD5NUWSFRZ2EA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.557164801 (615h55m26.428247836s) [noise=true , address=145.79.227.49:4370   , piece_id=TA4D4VCQXWYCB6FTZDRU6ZVZ4V6PZCFKQMA3EJYL5UEHHMI66JDQ, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.557530657 (615h55m26.428613692s) [noise=true , address=145.79.227.33:4896   , piece_id=TCDN3ER54RKZQ73BEI6DCGQW3IUNJXNNC3DUXI7NH7SFIPLHTF4A, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.557963953 (615h55m26.429046998s) [noise=true , address=77.90.31.12:4620     , piece_id=AQOFDODGJ7TMWA3ZQGWJW7XZT3FSKK3GR4R4RSIFIMHYP3HVJQDA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.558357422 (615h55m26.429440457s) [noise=true , address=145.79.227.40:4067   , piece_id=ITZ4SYUISESGVG72CBXHMAHZAD3XOOWYO5UUSFIT22R765N3W25A, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.558648285 (615h55m26.42973133s) [noise=true , address=145.79.227.2:4147    , piece_id=SHUFD6UKDFM54RGOZGJEUOUYCMKLAN2IMLWKJXZIZ5PAZVNUE4JQ, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.559228450 (615h55m26.430311485s) [noise=true , address=145.79.227.44:4228   , piece_id=YAQ7I3TOSLCYCHI2SJEEGCYGTEFWHX2CEYVZASRFL3BODTEONFTA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:27.974040599 (615h55m27.845123634s) [upload=539756]: Next returned nil segment
Dec 31 07:24:27.974074934 (615h55m27.845157969s) [upload=539756]: Waiting for error group managing segments...
Dec 31 07:24:28.334073958 (615h55m28.205156993s) [noise=true , address=64.49.13.140:4505    , piece_id=EBEDU3F3QMDLNIDSRRXUHGSZ3VH5ITKRDDI2QS5XKBOGKOCN3LNA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.427413642 (615h55m28.298496677s) [noise=true , address=145.79.227.2:4147    , piece_id=SHUFD6UKDFM54RGOZGJEUOUYCMKLAN2IMLWKJXZIZ5PAZVNUE4JQ, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.522823096 (615h55m28.393906131s) [noise=true , address=145.79.227.29:4965   , piece_id=L6L2H7KMBTYKC24ZMACWRYK6TGOK5KHDO2YCFP5TJS66NXXP3KNQ, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.585931513 (615h55m28.457014558s) [noise=true , address=77.90.31.13:4656     , piece_id=3PKC4JZPZVBQOBUHPQAG4HZ67DRTOY7GR35QVWLV3LN5JHXPMENA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.591579187 (615h55m28.462662262s) [noise=true , address=64.49.13.137:4500    , piece_id=DD4JKFRHB2FKWAOTS5RZU5M3ASDGJTRX2XRB5ZROVOZNPN6BER2Q, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.615175368 (615h55m28.486258413s) [noise=true , address=77.90.31.14:4673     , piece_id=XYDCDFNKMWJK4HDELS4NLBVLUBR76YHT4RP3M5JO5QMCSI6MOGYA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.699191016 (615h55m28.570274051s) [noise=true , address=45.91.116.6:4977     , piece_id=G7JKD7INUS3D3RUIHSSVYCSOVTHSRG5HLVKERSCAD5NUWSFRZ2EA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.713295994 (615h55m28.58437908s) [noise=true , address=45.91.116.10:4087    , piece_id=MJUIJFY6KEXEMMQKWA2NC3RE5PG55QCDZ7ECWRZG7CBROSA7LL3Q, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.794645326 (615h55m28.665728361s) [noise=true , address=77.90.31.12:4620     , piece_id=AQOFDODGJ7TMWA3ZQGWJW7XZT3FSKK3GR4R4RSIFIMHYP3HVJQDA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.341891306 (615h55m29.212974351s) [noise=true , address=145.79.227.33:4896   , piece_id=TCDN3ER54RKZQ73BEI6DCGQW3IUNJXNNC3DUXI7NH7SFIPLHTF4A, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.345487925 (615h55m29.21657096s) [noise=true , address=145.79.227.49:4370   , piece_id=TA4D4VCQXWYCB6FTZDRU6ZVZ4V6PZCFKQMA3EJYL5UEHHMI66JDQ, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.406150395 (615h55m29.27723343s) [noise=true , address=145.79.227.44:4228   , piece_id=YAQ7I3TOSLCYCHI2SJEEGCYGTEFWHX2CEYVZASRFL3BODTEONFTA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.414891660 (615h55m29.285974695s) [noise=true , address=193.149.164.143:4698 , piece_id=IYUHVRK3ZH3JADUJI627OUTHO7CRPNNKZVQTOCALMDYKVDGSHIJQ, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.431785668 (615h55m29.302868703s) [noise=true , address=145.79.227.40:4067   , piece_id=ITZ4SYUISESGVG72CBXHMAHZAD3XOOWYO5UUSFIT22R765N3W25A, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.431807049 (615h55m29.302890104s) [seg_pos={0 0}, upload=539756]: Segment reached optimal threshold of 14 pieces.
Dec 31 07:24:29.432061032 (615h55m29.303144087s) [noise=true , address=145.79.227.30:4784   , piece_id=M5NJQBX6JI5QI2PPA52B6A2CNLOXVWKYYYNX6CJDAA4WJNT4SHWA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pfXWex2WsuhmLb2smRpE46cHT9aipSTHW3e5oyELS1GhF1cFr): protocol: expected piece hash; context canceled; EOF
Dec 31 07:24:29.432072876 (615h55m29.303155931s) [noise=true , address=145.79.227.30:4784   , piece_id=M5NJQBX6JI5QI2PPA52B6A2CNLOXVWKYYYNX6CJDAA4WJNT4SHWA, seg_pos={0 0}, upload=539756]: Long tail cancellation due to optimal threshold
Dec 31 07:24:29.432090819 (615h55m29.303173884s) [noise=true , address=64.49.13.169:4709    , piece_id=CBAB3KCCPOW3P3LYM4T73CVDOD7FPJCPXYIXKDV7XTC2FKESTDRQ, seg_pos={0 0}, upload=539756]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:33942->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:33942->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33942->64.49.13.169:4709: use of closed network connection
Dec 31 07:24:29.432098063 (615h55m29.303181118s) [noise=true , address=64.49.13.169:4709    , piece_id=CBAB3KCCPOW3P3LYM4T73CVDOD7FPJCPXYIXKDV7XTC2FKESTDRQ, seg_pos={0 0}, upload=539756]: Long tail cancellation due to optimal threshold
Dec 31 07:24:29.432212572 (615h55m29.303295617s) [noise=true , address=64.49.13.169:4707    , piece_id=RB7LFR5DM7PUHCJPB5ABWN63Q2XQOGNOG2W2FCKXFWU5MGVPRPVA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:41264->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:41264->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41264->64.49.13.169:4707: use of closed network connection
Dec 31 07:24:29.432220747 (615h55m29.303303782s) [noise=true , address=64.49.13.169:4707    , piece_id=RB7LFR5DM7PUHCJPB5ABWN63Q2XQOGNOG2W2FCKXFWU5MGVPRPVA, seg_pos={0 0}, upload=539756]: Long tail cancellation due to optimal threshold
Dec 31 07:24:29.432465934 (615h55m29.303548979s) [noise=true , address=145.79.227.43:4186   , piece_id=ENVLK652WUVPKR3VPGXXMJ4IAEFHK67JOODS26ASTM4I2V7ABUWA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Gu3vXy2JUkKDaF4MkMCgVmYspW4Ez8EAfv6e4GeCkrwYTe5T): protocol: expected piece hash; context canceled; EOF
Dec 31 07:24:29.432474971 (615h55m29.303558006s) [noise=true , address=145.79.227.43:4186   , piece_id=ENVLK652WUVPKR3VPGXXMJ4IAEFHK67JOODS26ASTM4I2V7ABUWA, seg_pos={0 0}, upload=539756]: Long tail cancellation due to optimal threshold
Dec 31 07:24:29.432580261 (615h55m29.303663296s) [seg_pos={0 0}, upload=539756]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 07:24:29.432664502 (615h55m29.303747547s) [upload=539756]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 07:24:29.432671305 (615h55m29.30375435s) [upload=539756]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 07:24:29.717299118 (615h55m29.588382153s) [upload=539756]: Done uploading segments...
Dec 31 07:24:29.717320589 (615h55m29.588403624s) [upload=539756]: Upload finished. err: <nil>
Duration: 24.597397576s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [............................................X...............]
duration: 
6.137251226s
trace: 
https://jaeger.dev.storj.io/trace/5dd0609e3a2ac547
FAIL (2025-12-31 07:54:44 UTC) 
orbiter warning: context deadline exceeded
Duration: 6.137276975s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)
antiblip: 
1 out of 60 is failed (threshold=3) [............................................X...............]
duration: 
2.606314865s
trace: 
https://jaeger.dev.storj.io/trace/59c3b821b75eba3a
FAIL (2025-12-31 07:54:38 UTC) 
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded
Duration: 2.60634875s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
2 out of 60 is failed (threshold=3) [.......X............X.......................................]
duration: 
29.383483901s
trace: 
https://jaeger.dev.storj.io/trace/6ce9bac61d803c5a
FAIL (2025-12-31 07:54:17 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
56     , piece_id=3PKC4JZPZVBQOBUHPQAG4HZ67DRTOY7GR35QVWLV3LN5JHXPMENA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.554293132 (615h55m26.425376177s) [noise=true , address=145.79.227.43:4186   , piece_id=ENVLK652WUVPKR3VPGXXMJ4IAEFHK67JOODS26ASTM4I2V7ABUWA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.554505907 (615h55m26.425588942s) [noise=true , address=77.90.31.14:4673     , piece_id=XYDCDFNKMWJK4HDELS4NLBVLUBR76YHT4RP3M5JO5QMCSI6MOGYA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.554648990 (615h55m26.425732025s) [noise=true , address=64.49.13.140:4505    , piece_id=EBEDU3F3QMDLNIDSRRXUHGSZ3VH5ITKRDDI2QS5XKBOGKOCN3LNA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.554994317 (615h55m26.426077352s) [noise=true , address=64.49.13.169:4707    , piece_id=RB7LFR5DM7PUHCJPB5ABWN63Q2XQOGNOG2W2FCKXFWU5MGVPRPVA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.555494150 (615h55m26.426577185s) [noise=true , address=64.49.13.137:4500    , piece_id=DD4JKFRHB2FKWAOTS5RZU5M3ASDGJTRX2XRB5ZROVOZNPN6BER2Q, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.555978812 (615h55m26.427061847s) [noise=true , address=45.91.116.10:4087    , piece_id=MJUIJFY6KEXEMMQKWA2NC3RE5PG55QCDZ7ECWRZG7CBROSA7LL3Q, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.556325282 (615h55m26.427408317s) [noise=true , address=145.79.227.29:4965   , piece_id=L6L2H7KMBTYKC24ZMACWRYK6TGOK5KHDO2YCFP5TJS66NXXP3KNQ, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.556728720 (615h55m26.427811765s) [noise=true , address=45.91.116.6:4977     , piece_id=G7JKD7INUS3D3RUIHSSVYCSOVTHSRG5HLVKERSCAD5NUWSFRZ2EA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.557164801 (615h55m26.428247836s) [noise=true , address=145.79.227.49:4370   , piece_id=TA4D4VCQXWYCB6FTZDRU6ZVZ4V6PZCFKQMA3EJYL5UEHHMI66JDQ, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.557530657 (615h55m26.428613692s) [noise=true , address=145.79.227.33:4896   , piece_id=TCDN3ER54RKZQ73BEI6DCGQW3IUNJXNNC3DUXI7NH7SFIPLHTF4A, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.557963953 (615h55m26.429046998s) [noise=true , address=77.90.31.12:4620     , piece_id=AQOFDODGJ7TMWA3ZQGWJW7XZT3FSKK3GR4R4RSIFIMHYP3HVJQDA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.558357422 (615h55m26.429440457s) [noise=true , address=145.79.227.40:4067   , piece_id=ITZ4SYUISESGVG72CBXHMAHZAD3XOOWYO5UUSFIT22R765N3W25A, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.558648285 (615h55m26.42973133s) [noise=true , address=145.79.227.2:4147    , piece_id=SHUFD6UKDFM54RGOZGJEUOUYCMKLAN2IMLWKJXZIZ5PAZVNUE4JQ, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.559228450 (615h55m26.430311485s) [noise=true , address=145.79.227.44:4228   , piece_id=YAQ7I3TOSLCYCHI2SJEEGCYGTEFWHX2CEYVZASRFL3BODTEONFTA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:27.974040599 (615h55m27.845123634s) [upload=539756]: Next returned nil segment
Dec 31 07:24:27.974074934 (615h55m27.845157969s) [upload=539756]: Waiting for error group managing segments...
Dec 31 07:24:28.334073958 (615h55m28.205156993s) [noise=true , address=64.49.13.140:4505    , piece_id=EBEDU3F3QMDLNIDSRRXUHGSZ3VH5ITKRDDI2QS5XKBOGKOCN3LNA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.427413642 (615h55m28.298496677s) [noise=true , address=145.79.227.2:4147    , piece_id=SHUFD6UKDFM54RGOZGJEUOUYCMKLAN2IMLWKJXZIZ5PAZVNUE4JQ, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.522823096 (615h55m28.393906131s) [noise=true , address=145.79.227.29:4965   , piece_id=L6L2H7KMBTYKC24ZMACWRYK6TGOK5KHDO2YCFP5TJS66NXXP3KNQ, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.585931513 (615h55m28.457014558s) [noise=true , address=77.90.31.13:4656     , piece_id=3PKC4JZPZVBQOBUHPQAG4HZ67DRTOY7GR35QVWLV3LN5JHXPMENA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.591579187 (615h55m28.462662262s) [noise=true , address=64.49.13.137:4500    , piece_id=DD4JKFRHB2FKWAOTS5RZU5M3ASDGJTRX2XRB5ZROVOZNPN6BER2Q, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.615175368 (615h55m28.486258413s) [noise=true , address=77.90.31.14:4673     , piece_id=XYDCDFNKMWJK4HDELS4NLBVLUBR76YHT4RP3M5JO5QMCSI6MOGYA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.699191016 (615h55m28.570274051s) [noise=true , address=45.91.116.6:4977     , piece_id=G7JKD7INUS3D3RUIHSSVYCSOVTHSRG5HLVKERSCAD5NUWSFRZ2EA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.713295994 (615h55m28.58437908s) [noise=true , address=45.91.116.10:4087    , piece_id=MJUIJFY6KEXEMMQKWA2NC3RE5PG55QCDZ7ECWRZG7CBROSA7LL3Q, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.794645326 (615h55m28.665728361s) [noise=true , address=77.90.31.12:4620     , piece_id=AQOFDODGJ7TMWA3ZQGWJW7XZT3FSKK3GR4R4RSIFIMHYP3HVJQDA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.341891306 (615h55m29.212974351s) [noise=true , address=145.79.227.33:4896   , piece_id=TCDN3ER54RKZQ73BEI6DCGQW3IUNJXNNC3DUXI7NH7SFIPLHTF4A, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.345487925 (615h55m29.21657096s) [noise=true , address=145.79.227.49:4370   , piece_id=TA4D4VCQXWYCB6FTZDRU6ZVZ4V6PZCFKQMA3EJYL5UEHHMI66JDQ, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.406150395 (615h55m29.27723343s) [noise=true , address=145.79.227.44:4228   , piece_id=YAQ7I3TOSLCYCHI2SJEEGCYGTEFWHX2CEYVZASRFL3BODTEONFTA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.414891660 (615h55m29.285974695s) [noise=true , address=193.149.164.143:4698 , piece_id=IYUHVRK3ZH3JADUJI627OUTHO7CRPNNKZVQTOCALMDYKVDGSHIJQ, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.431785668 (615h55m29.302868703s) [noise=true , address=145.79.227.40:4067   , piece_id=ITZ4SYUISESGVG72CBXHMAHZAD3XOOWYO5UUSFIT22R765N3W25A, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.431807049 (615h55m29.302890104s) [seg_pos={0 0}, upload=539756]: Segment reached optimal threshold of 14 pieces.
Dec 31 07:24:29.432061032 (615h55m29.303144087s) [noise=true , address=145.79.227.30:4784   , piece_id=M5NJQBX6JI5QI2PPA52B6A2CNLOXVWKYYYNX6CJDAA4WJNT4SHWA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pfXWex2WsuhmLb2smRpE46cHT9aipSTHW3e5oyELS1GhF1cFr): protocol: expected piece hash; context canceled; EOF
Dec 31 07:24:29.432072876 (615h55m29.303155931s) [noise=true , address=145.79.227.30:4784   , piece_id=M5NJQBX6JI5QI2PPA52B6A2CNLOXVWKYYYNX6CJDAA4WJNT4SHWA, seg_pos={0 0}, upload=539756]: Long tail cancellation due to optimal threshold
Dec 31 07:24:29.432090819 (615h55m29.303173884s) [noise=true , address=64.49.13.169:4709    , piece_id=CBAB3KCCPOW3P3LYM4T73CVDOD7FPJCPXYIXKDV7XTC2FKESTDRQ, seg_pos={0 0}, upload=539756]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:33942->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:33942->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33942->64.49.13.169:4709: use of closed network connection
Dec 31 07:24:29.432098063 (615h55m29.303181118s) [noise=true , address=64.49.13.169:4709    , piece_id=CBAB3KCCPOW3P3LYM4T73CVDOD7FPJCPXYIXKDV7XTC2FKESTDRQ, seg_pos={0 0}, upload=539756]: Long tail cancellation due to optimal threshold
Dec 31 07:24:29.432212572 (615h55m29.303295617s) [noise=true , address=64.49.13.169:4707    , piece_id=RB7LFR5DM7PUHCJPB5ABWN63Q2XQOGNOG2W2FCKXFWU5MGVPRPVA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:41264->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:41264->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41264->64.49.13.169:4707: use of closed network connection
Dec 31 07:24:29.432220747 (615h55m29.303303782s) [noise=true , address=64.49.13.169:4707    , piece_id=RB7LFR5DM7PUHCJPB5ABWN63Q2XQOGNOG2W2FCKXFWU5MGVPRPVA, seg_pos={0 0}, upload=539756]: Long tail cancellation due to optimal threshold
Dec 31 07:24:29.432465934 (615h55m29.303548979s) [noise=true , address=145.79.227.43:4186   , piece_id=ENVLK652WUVPKR3VPGXXMJ4IAEFHK67JOODS26ASTM4I2V7ABUWA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Gu3vXy2JUkKDaF4MkMCgVmYspW4Ez8EAfv6e4GeCkrwYTe5T): protocol: expected piece hash; context canceled; EOF
Dec 31 07:24:29.432474971 (615h55m29.303558006s) [noise=true , address=145.79.227.43:4186   , piece_id=ENVLK652WUVPKR3VPGXXMJ4IAEFHK67JOODS26ASTM4I2V7ABUWA, seg_pos={0 0}, upload=539756]: Long tail cancellation due to optimal threshold
Dec 31 07:24:29.432580261 (615h55m29.303663296s) [seg_pos={0 0}, upload=539756]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 07:24:29.432664502 (615h55m29.303747547s) [upload=539756]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 07:24:29.432671305 (615h55m29.30375435s) [upload=539756]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 07:24:29.717299118 (615h55m29.588382153s) [upload=539756]: Done uploading segments...
Dec 31 07:24:29.717320589 (615h55m29.588403624s) [upload=539756]: Upload finished. err: <nil>
Duration: 29.383500623s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
19 out of 60 is failed (threshold=3) [...X.X.X.X.X......XXX.X....XXX..X...........X......X.X.X.XX.]
duration: 
6.115880242s
trace: 
https://jaeger.dev.storj.io/trace/6c9f60d6c2a165da
FAIL (2025-12-31 07:54:13 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: PW5BS4DU62NA7RUSDX3A2EYBWMQDU6CLVKOQ3LRDSUZSF4AUP64Q) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: ZQB3NTHBABFQ7FEQJNHEVPRTRF4K6C3FJV4SZ4KEIYM6ZCW7CCAA) context deadline exceeded
======LOGS======
Dec 31 07:53:38.027484676 (616h24m37.898567721s) [upload=540182]: Starting upload
Dec 31 07:53:38.027534512 (616h24m37.898617547s) [upload=540182]: Done starting upload
Dec 31 07:53:38.027801700 (616h24m37.898884735s) [upload=540182]: Uploading segments...
Dec 31 07:53:38.027979689 (616h24m37.899062734s) [upload=540182]: Got next segment. Inline: false
Dec 31 07:53:38.028073868 (616h24m37.899156903s) [upload=540182]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 07:53:38.028082775 (616h24m37.89916581s) [upload=540182]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 07:53:38.277891170 (616h24m38.148974205s) [seg_pos={0 0}, upload=540182]: Begin upload segment...
Dec 31 07:53:38.277968046 (616h24m38.149051081s) [seg_pos={0 0}, upload=540182]: Done begin upload segment.
Dec 31 07:53:38.278029704 (616h24m38.149112749s) [noise=true , address=45.91.116.6:4975     , piece_id=7H4553CB6VLOISS34HUKZLLQMIZFCF3CUH5L2GP2VU5M6TEIEHMA, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.278042758 (616h24m38.149125794s) [noise=true , address=77.90.31.12:4589     , piece_id=5E5XAACC5DCA6MOP4TFJ23AW3RAG3Q36MVHGSLZ5O2RMCIZNOG2A, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.278144813 (616h24m38.149227848s) [noise=true , address=145.79.227.50:4375   , piece_id=CXKJCYVE6SJOOVQWHH34MCITFOFEV27SP4OKAPXF4DJT3EXIHWGQ, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.278830739 (616h24m38.149913784s) [noise=true , address=77.90.31.19:4842     , piece_id=PW5BS4DU62NA7RUSDX3A2EYBWMQDU6CLVKOQ3LRDSUZSF4AUP64Q, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.279143605 (616h24m38.15022664s) [noise=true , address=77.90.31.15:4700     , piece_id=ZQB3NTHBABFQ7FEQJNHEVPRTRF4K6C3FJV4SZ4KEIYM6ZCW7CCAA, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.279489213 (616h24m38.150572258s) [noise=true , address=45.91.116.13:4154    , piece_id=LJFCLUXCGK56OQR3EFB73WBAHG7VFQ27Y7X2LYS4BPP7GYBBONUQ, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.279878865 (616h24m38.15096191s) [noise=true , address=145.79.227.26:4655   , piece_id=YHNBMQCH6QZRCTHVFDZRVHMXB2BP3MJ7B3RG3QIPBQ4PQW5JYB6A, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.280269218 (616h24m38.151352253s) [noise=true , address=145.79.227.37:4566   , piece_id=6E73Y5FUFAGQLU4IYUQNJ2J7XXXRJQGPM7KJRHMYQ3ENEBN76QKQ, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.280653881 (616h24m38.151736926s) [noise=true , address=145.79.227.24:4581   , piece_id=IKQBNNGA2QEVTMZZMGPMHLLIKLEDKZCL34X2DDIFZFHW2GBKLCXA, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.281852182 (616h24m38.152935227s) [noise=true , address=145.79.227.16:4910   , piece_id=6L6QC3KSIG6CVDPLG55O6Q24L3QZ72GL3OTPAETHJG7I4Z4WMYKA, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.281889793 (616h24m38.152972828s) [noise=true , address=145.79.227.45:4229   , piece_id=3EKCB2QETOCEMR2WCFYDZKVYT2R3CBOOKMUHXBMANKMOJD7H7IZA, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.283093776 (616h24m38.154176821s) [noise=true , address=145.79.227.12:4767   , piece_id=GGEX2BKWC4URKRI34IAR7Z2KAP62FVRDM3ELUTNIH6WB4QN3JLRQ, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.283497055 (616h24m38.15458009s) [noise=true , address=145.79.227.23:4545   , piece_id=F5KYOLWB2IZRRCKVYFGHFZ23UZV7FUR2PCF2HO7GSHEMKIY5IL3Q, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.284318098 (616h24m38.155401133s) [noise=true , address=145.79.227.51:4410   , piece_id=VYULD4SFKPQRYZSOPM2PUQC4LUQAQ4WXRYVSOSRYCA5N5LFV4FAA, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.285603535 (616h24m38.15668657s) [noise=true , address=145.79.227.52:4446   , piece_id=6FI2W5DULY5FFDFIFXLNSDUX7EEVDGB3WLWIEEZXFKNOR3AIOSIQ, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.286960870 (616h24m38.158043915s) [noise=true , address=145.79.227.18:4006   , piece_id=LFDSYTOQR42MSDEIRDUXGRNBUF3ONMCL6BJTMRNLNDHHNY5V7ZDQ, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:39.976475363 (616h24m39.847558408s) [upload=540182]: Next returned nil segment
Dec 31 07:53:39.976503577 (616h24m39.847586622s) [upload=540182]: Waiting for error group managing segments...
Dec 31 07:53:40.263089662 (616h24m40.134172697s) [noise=true , address=145.79.227.52:4446   , piece_id=6FI2W5DULY5FFDFIFXLNSDUX7EEVDGB3WLWIEEZXFKNOR3AIOSIQ, seg_pos={0 0}, upload=540182]: Done uploading piece. err: <nil>
Dec 31 07:53:40.263514351 (616h24m40.134597386s) [noise=true , address=145.79.227.16:4910   , piece_id=6L6QC3KSIG6CVDPLG55O6Q24L3QZ72GL3OTPAETHJG7I4Z4WMYKA, seg_pos={0 0}, upload=540182]: Done uploading piece. err: <nil>
Dec 31 07:53:40.266215304 (616h24m40.137298339s) [noise=true , address=145.79.227.12:4767   , piece_id=GGEX2BKWC4URKRI34IAR7Z2KAP62FVRDM3ELUTNIH6WB4QN3JLRQ, seg_pos={0 0}, upload=540182]: Done uploading piece. err: <nil>
Dec 31 07:53:40.274366505 (616h24m40.14544955s) [noise=true , address=145.79.227.51:4410   , piece_id=VYULD4SFKPQRYZSOPM2PUQC4LUQAQ4WXRYVSOSRYCA5N5LFV4FAA, seg_pos={0 0}, upload=540182]: Done uploading piece. err: <nil>
Dec 31 07:53:40.277241870 (616h24m40.148324905s) [noise=true , address=145.79.227.26:4655   , piece_id=YHNBMQCH6QZRCTHVFDZRVHMXB2BP3MJ7B3RG3QIPBQ4PQW5JYB6A, seg_pos={0 0}, upload=540182]: Done uploading piece. err: <nil>
Dec 31 07:53:40.277655779 (616h24m40.148738814s) [noise=true , address=145.79.227.37:4566   , piece_id=6E73Y5FUFAGQLU4IYUQNJ2J7XXXRJQGPM7KJRHMYQ3ENEBN76QKQ, seg_pos={0 0}, upload=540182]: Done uploading piece. err: <nil>
Dec 31 07:53:40.288485420 (616h24m40.159568465s) [noise=true , address=145.79.227.50:4375   , piece_id=CXKJCYVE6SJOOVQWHH34MCITFOFEV27SP4OKAPXF4DJT3EXIHWGQ, seg_pos={0 0}, upload=540182]: Done uploading piece. err: <nil>
Dec 31 07:53:40.290211387 (616h24m40.161294432s) [noise=true , address=145.79.227.18:4006   , piece_id=LFDSYTOQR42MSDEIRDUXGRNBUF3ONMCL6BJTMRNLNDHHNY5V7ZDQ, seg_pos={0 0}, upload=540182]: Done uploading piece. err: <nil>
Dec 31 07:53:40.293396742 (616h24m40.164479787s) [noise=true , address=145.79.227.24:4581   , piece_id=IKQBNNGA2QEVTMZZMGPMHLLIKLEDKZCL34X2DDIFZFHW2GBKLCXA, seg_pos={0 0}, upload=540182]: Done uploading piece. err: <nil>
Dec 31 07:53:40.302471963 (616h24m40.173554998s) [noise=true , address=145.79.227.45:4229   , piece_id=3EKCB2QETOCEMR2WCFYDZKVYT2R3CBOOKMUHXBMANKMOJD7H7IZA, seg_pos={0 0}, upload=540182]: Done uploading piece. err: <nil>
Dec 31 07:53:40.354700213 (616h24m40.225783248s) [noise=true , address=145.79.227.23:4545   , piece_id=F5KYOLWB2IZRRCKVYFGHFZ23UZV7FUR2PCF2HO7GSHEMKIY5IL3Q, seg_pos={0 0}, upload=540182]: Done uploading piece. err: <nil>
Dec 31 07:53:40.453140008 (616h24m40.324223043s) [noise=true , address=77.90.31.19:4842     , piece_id=PW5BS4DU62NA7RUSDX3A2EYBWMQDU6CLVKOQ3LRDSUZSF4AUP64Q, seg_pos={0 0}, upload=540182]: Done uploading piece. err: <nil>
Dec 31 07:53:40.527310551 (616h24m40.398393596s) [noise=true , address=77.90.31.15:4700     , piece_id=ZQB3NTHBABFQ7FEQJNHEVPRTRF4K6C3FJV4SZ4KEIYM6ZCW7CCAA, seg_pos={0 0}, upload=540182]: Done uploading piece. err: <nil>
Dec 31 07:53:40.527327724 (616h24m40.398410759s) [seg_pos={0 0}, upload=540182]: Segment reached optimal threshold of 13 pieces.
Dec 31 07:53:40.527495442 (616h24m40.398578477s) [noise=true , address=45.91.116.13:4154    , piece_id=LJFCLUXCGK56OQR3EFB73WBAHG7VFQ27Y7X2LYS4BPP7GYBBONUQ, seg_pos={0 0}, upload=540182]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF
Dec 31 07:53:40.527501184 (616h24m40.398584219s) [noise=true , address=45.91.116.13:4154    , piece_id=LJFCLUXCGK56OQR3EFB73WBAHG7VFQ27Y7X2LYS4BPP7GYBBONUQ, seg_pos={0 0}, upload=540182]: Long tail cancellation due to optimal threshold
Dec 31 07:53:40.527703939 (616h24m40.398786974s) [noise=true , address=77.90.31.12:4589     , piece_id=5E5XAACC5DCA6MOP4TFJ23AW3RAG3Q36MVHGSLZ5O2RMCIZNOG2A, seg_pos={0 0}, upload=540182]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: write tcp 188.166.225.170:59116->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:59116->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59116->77.90.31.12:4589: use of closed network connection
Dec 31 07:53:40.527709891 (616h24m40.398792927s) [noise=true , address=77.90.31.12:4589     , piece_id=5E5XAACC5DCA6MOP4TFJ23AW3RAG3Q36MVHGSLZ5O2RMCIZNOG2A, seg_pos={0 0}, upload=540182]: Long tail cancellation due to optimal threshold
Dec 31 07:53:40.528022527 (616h24m40.399105562s) [noise=true , address=45.91.116.6:4975     , piece_id=7H4553CB6VLOISS34HUKZLLQMIZFCF3CUH5L2GP2VU5M6TEIEHMA, seg_pos={0 0}, upload=540182]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF
Dec 31 07:53:40.528036814 (616h24m40.399119849s) [noise=true , address=45.91.116.6:4975     , piece_id=7H4553CB6VLOISS34HUKZLLQMIZFCF3CUH5L2GP2VU5M6TEIEHMA, seg_pos={0 0}, upload=540182]: Long tail cancellation due to optimal threshold
Dec 31 07:53:40.528063705 (616h24m40.39914674s) [seg_pos={0 0}, upload=540182]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 07:53:40.528171600 (616h24m40.399254635s) [upload=540182]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 07:53:40.528177852 (616h24m40.399260887s) [upload=540182]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 07:53:40.824344511 (616h24m40.695427566s) [upload=540182]: Done uploading segments...
Dec 31 07:53:40.824370379 (616h24m40.695453454s) [upload=540182]: Upload finished. err: <nil>
Duration: 6.115895953s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
5 out of 60 is failed (threshold=6) [X................................X.........X............X..X]
duration: 
5.001432114s
trace: 
https://jaeger.dev.storj.io/trace/e2308925a42e319
FAIL (2025-12-31 07:54:08 UTC) 
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001453675s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
19 out of 60 is failed (threshold=3) [....X.X.X.X.X......XXX.X....XXX..X...........X......X.X.X.XX]
duration: 
30.003931798s
trace: 
https://jaeger.dev.storj.io/trace/6c9f60d6c2a165da
FAIL (2025-12-31 07:54:07 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: PW5BS4DU62NA7RUSDX3A2EYBWMQDU6CLVKOQ3LRDSUZSF4AUP64Q) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: ZQB3NTHBABFQ7FEQJNHEVPRTRF4K6C3FJV4SZ4KEIYM6ZCW7CCAA) context deadline exceeded
======LOGS======
Dec 31 07:53:38.027484676 (616h24m37.898567721s) [upload=540182]: Starting upload
Dec 31 07:53:38.027534512 (616h24m37.898617547s) [upload=540182]: Done starting upload
Dec 31 07:53:38.027801700 (616h24m37.898884735s) [upload=540182]: Uploading segments...
Dec 31 07:53:38.027979689 (616h24m37.899062734s) [upload=540182]: Got next segment. Inline: false
Dec 31 07:53:38.028073868 (616h24m37.899156903s) [upload=540182]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 07:53:38.028082775 (616h24m37.89916581s) [upload=540182]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 07:53:38.277891170 (616h24m38.148974205s) [seg_pos={0 0}, upload=540182]: Begin upload segment...
Dec 31 07:53:38.277968046 (616h24m38.149051081s) [seg_pos={0 0}, upload=540182]: Done begin upload segment.
Dec 31 07:53:38.278029704 (616h24m38.149112749s) [noise=true , address=45.91.116.6:4975     , piece_id=7H4553CB6VLOISS34HUKZLLQMIZFCF3CUH5L2GP2VU5M6TEIEHMA, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.278042758 (616h24m38.149125794s) [noise=true , address=77.90.31.12:4589     , piece_id=5E5XAACC5DCA6MOP4TFJ23AW3RAG3Q36MVHGSLZ5O2RMCIZNOG2A, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.278144813 (616h24m38.149227848s) [noise=true , address=145.79.227.50:4375   , piece_id=CXKJCYVE6SJOOVQWHH34MCITFOFEV27SP4OKAPXF4DJT3EXIHWGQ, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.278830739 (616h24m38.149913784s) [noise=true , address=77.90.31.19:4842     , piece_id=PW5BS4DU62NA7RUSDX3A2EYBWMQDU6CLVKOQ3LRDSUZSF4AUP64Q, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.279143605 (616h24m38.15022664s) [noise=true , address=77.90.31.15:4700     , piece_id=ZQB3NTHBABFQ7FEQJNHEVPRTRF4K6C3FJV4SZ4KEIYM6ZCW7CCAA, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.279489213 (616h24m38.150572258s) [noise=true , address=45.91.116.13:4154    , piece_id=LJFCLUXCGK56OQR3EFB73WBAHG7VFQ27Y7X2LYS4BPP7GYBBONUQ, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.279878865 (616h24m38.15096191s) [noise=true , address=145.79.227.26:4655   , piece_id=YHNBMQCH6QZRCTHVFDZRVHMXB2BP3MJ7B3RG3QIPBQ4PQW5JYB6A, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.280269218 (616h24m38.151352253s) [noise=true , address=145.79.227.37:4566   , piece_id=6E73Y5FUFAGQLU4IYUQNJ2J7XXXRJQGPM7KJRHMYQ3ENEBN76QKQ, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.280653881 (616h24m38.151736926s) [noise=true , address=145.79.227.24:4581   , piece_id=IKQBNNGA2QEVTMZZMGPMHLLIKLEDKZCL34X2DDIFZFHW2GBKLCXA, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.281852182 (616h24m38.152935227s) [noise=true , address=145.79.227.16:4910   , piece_id=6L6QC3KSIG6CVDPLG55O6Q24L3QZ72GL3OTPAETHJG7I4Z4WMYKA, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.281889793 (616h24m38.152972828s) [noise=true , address=145.79.227.45:4229   , piece_id=3EKCB2QETOCEMR2WCFYDZKVYT2R3CBOOKMUHXBMANKMOJD7H7IZA, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.283093776 (616h24m38.154176821s) [noise=true , address=145.79.227.12:4767   , piece_id=GGEX2BKWC4URKRI34IAR7Z2KAP62FVRDM3ELUTNIH6WB4QN3JLRQ, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.283497055 (616h24m38.15458009s) [noise=true , address=145.79.227.23:4545   , piece_id=F5KYOLWB2IZRRCKVYFGHFZ23UZV7FUR2PCF2HO7GSHEMKIY5IL3Q, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.284318098 (616h24m38.155401133s) [noise=true , address=145.79.227.51:4410   , piece_id=VYULD4SFKPQRYZSOPM2PUQC4LUQAQ4WXRYVSOSRYCA5N5LFV4FAA, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.285603535 (616h24m38.15668657s) [noise=true , address=145.79.227.52:4446   , piece_id=6FI2W5DULY5FFDFIFXLNSDUX7EEVDGB3WLWIEEZXFKNOR3AIOSIQ, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:38.286960870 (616h24m38.158043915s) [noise=true , address=145.79.227.18:4006   , piece_id=LFDSYTOQR42MSDEIRDUXGRNBUF3ONMCL6BJTMRNLNDHHNY5V7ZDQ, seg_pos={0 0}, upload=540182]: Uploading piece...
Dec 31 07:53:39.976475363 (616h24m39.847558408s) [upload=540182]: Next returned nil segment
Dec 31 07:53:39.976503577 (616h24m39.847586622s) [upload=540182]: Waiting for error group managing segments...
Dec 31 07:53:40.263089662 (616h24m40.134172697s) [noise=true , address=145.79.227.52:4446   , piece_id=6FI2W5DULY5FFDFIFXLNSDUX7EEVDGB3WLWIEEZXFKNOR3AIOSIQ, seg_pos={0 0}, upload=540182]: Done uploading piece. err: <nil>
Dec 31 07:53:40.263514351 (616h24m40.134597386s) [noise=true , address=145.79.227.16:4910   , piece_id=6L6QC3KSIG6CVDPLG55O6Q24L3QZ72GL3OTPAETHJG7I4Z4WMYKA, seg_pos={0 0}, upload=540182]: Done uploading piece. err: <nil>
Dec 31 07:53:40.266215304 (616h24m40.137298339s) [noise=true , address=145.79.227.12:4767   , piece_id=GGEX2BKWC4URKRI34IAR7Z2KAP62FVRDM3ELUTNIH6WB4QN3JLRQ, seg_pos={0 0}, upload=540182]: Done uploading piece. err: <nil>
Dec 31 07:53:40.274366505 (616h24m40.14544955s) [noise=true , address=145.79.227.51:4410   , piece_id=VYULD4SFKPQRYZSOPM2PUQC4LUQAQ4WXRYVSOSRYCA5N5LFV4FAA, seg_pos={0 0}, upload=540182]: Done uploading piece. err: <nil>
Dec 31 07:53:40.277241870 (616h24m40.148324905s) [noise=true , address=145.79.227.26:4655   , piece_id=YHNBMQCH6QZRCTHVFDZRVHMXB2BP3MJ7B3RG3QIPBQ4PQW5JYB6A, seg_pos={0 0}, upload=540182]: Done uploading piece. err: <nil>
Dec 31 07:53:40.277655779 (616h24m40.148738814s) [noise=true , address=145.79.227.37:4566   , piece_id=6E73Y5FUFAGQLU4IYUQNJ2J7XXXRJQGPM7KJRHMYQ3ENEBN76QKQ, seg_pos={0 0}, upload=540182]: Done uploading piece. err: <nil>
Dec 31 07:53:40.288485420 (616h24m40.159568465s) [noise=true , address=145.79.227.50:4375   , piece_id=CXKJCYVE6SJOOVQWHH34MCITFOFEV27SP4OKAPXF4DJT3EXIHWGQ, seg_pos={0 0}, upload=540182]: Done uploading piece. err: <nil>
Dec 31 07:53:40.290211387 (616h24m40.161294432s) [noise=true , address=145.79.227.18:4006   , piece_id=LFDSYTOQR42MSDEIRDUXGRNBUF3ONMCL6BJTMRNLNDHHNY5V7ZDQ, seg_pos={0 0}, upload=540182]: Done uploading piece. err: <nil>
Dec 31 07:53:40.293396742 (616h24m40.164479787s) [noise=true , address=145.79.227.24:4581   , piece_id=IKQBNNGA2QEVTMZZMGPMHLLIKLEDKZCL34X2DDIFZFHW2GBKLCXA, seg_pos={0 0}, upload=540182]: Done uploading piece. err: <nil>
Dec 31 07:53:40.302471963 (616h24m40.173554998s) [noise=true , address=145.79.227.45:4229   , piece_id=3EKCB2QETOCEMR2WCFYDZKVYT2R3CBOOKMUHXBMANKMOJD7H7IZA, seg_pos={0 0}, upload=540182]: Done uploading piece. err: <nil>
Dec 31 07:53:40.354700213 (616h24m40.225783248s) [noise=true , address=145.79.227.23:4545   , piece_id=F5KYOLWB2IZRRCKVYFGHFZ23UZV7FUR2PCF2HO7GSHEMKIY5IL3Q, seg_pos={0 0}, upload=540182]: Done uploading piece. err: <nil>
Dec 31 07:53:40.453140008 (616h24m40.324223043s) [noise=true , address=77.90.31.19:4842     , piece_id=PW5BS4DU62NA7RUSDX3A2EYBWMQDU6CLVKOQ3LRDSUZSF4AUP64Q, seg_pos={0 0}, upload=540182]: Done uploading piece. err: <nil>
Dec 31 07:53:40.527310551 (616h24m40.398393596s) [noise=true , address=77.90.31.15:4700     , piece_id=ZQB3NTHBABFQ7FEQJNHEVPRTRF4K6C3FJV4SZ4KEIYM6ZCW7CCAA, seg_pos={0 0}, upload=540182]: Done uploading piece. err: <nil>
Dec 31 07:53:40.527327724 (616h24m40.398410759s) [seg_pos={0 0}, upload=540182]: Segment reached optimal threshold of 13 pieces.
Dec 31 07:53:40.527495442 (616h24m40.398578477s) [noise=true , address=45.91.116.13:4154    , piece_id=LJFCLUXCGK56OQR3EFB73WBAHG7VFQ27Y7X2LYS4BPP7GYBBONUQ, seg_pos={0 0}, upload=540182]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF
Dec 31 07:53:40.527501184 (616h24m40.398584219s) [noise=true , address=45.91.116.13:4154    , piece_id=LJFCLUXCGK56OQR3EFB73WBAHG7VFQ27Y7X2LYS4BPP7GYBBONUQ, seg_pos={0 0}, upload=540182]: Long tail cancellation due to optimal threshold
Dec 31 07:53:40.527703939 (616h24m40.398786974s) [noise=true , address=77.90.31.12:4589     , piece_id=5E5XAACC5DCA6MOP4TFJ23AW3RAG3Q36MVHGSLZ5O2RMCIZNOG2A, seg_pos={0 0}, upload=540182]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: write tcp 188.166.225.170:59116->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:59116->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59116->77.90.31.12:4589: use of closed network connection
Dec 31 07:53:40.527709891 (616h24m40.398792927s) [noise=true , address=77.90.31.12:4589     , piece_id=5E5XAACC5DCA6MOP4TFJ23AW3RAG3Q36MVHGSLZ5O2RMCIZNOG2A, seg_pos={0 0}, upload=540182]: Long tail cancellation due to optimal threshold
Dec 31 07:53:40.528022527 (616h24m40.399105562s) [noise=true , address=45.91.116.6:4975     , piece_id=7H4553CB6VLOISS34HUKZLLQMIZFCF3CUH5L2GP2VU5M6TEIEHMA, seg_pos={0 0}, upload=540182]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF
Dec 31 07:53:40.528036814 (616h24m40.399119849s) [noise=true , address=45.91.116.6:4975     , piece_id=7H4553CB6VLOISS34HUKZLLQMIZFCF3CUH5L2GP2VU5M6TEIEHMA, seg_pos={0 0}, upload=540182]: Long tail cancellation due to optimal threshold
Dec 31 07:53:40.528063705 (616h24m40.39914674s) [seg_pos={0 0}, upload=540182]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 07:53:40.528171600 (616h24m40.399254635s) [upload=540182]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 07:53:40.528177852 (616h24m40.399260887s) [upload=540182]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 07:53:40.824344511 (616h24m40.695427566s) [upload=540182]: Done uploading segments...
Dec 31 07:53:40.824370379 (616h24m40.695453454s) [upload=540182]: Upload finished. err: <nil>
Duration: 30.003945965s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)
antiblip: 
1 out of 60 is failed (threshold=3) [.............................................X..............]
duration: 
2.470161477s
trace: 
https://jaeger.dev.storj.io/trace/59c3b821b75eba3a
FAIL (2025-12-31 07:53:55 UTC) 
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded
Duration: 2.47018871s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [.............................................X..............]
duration: 
16.516682423s
trace: 
https://jaeger.dev.storj.io/trace/5dd0609e3a2ac547
FAIL (2025-12-31 07:53:50 UTC) 
orbiter warning: context deadline exceeded
Duration: 16.516702902s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
2 out of 60 is failed (threshold=3) [........X............X......................................]
duration: 
41.890720569s
trace: 
https://jaeger.dev.storj.io/trace/6ce9bac61d803c5a
FAIL (2025-12-31 07:53:48 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
56     , piece_id=3PKC4JZPZVBQOBUHPQAG4HZ67DRTOY7GR35QVWLV3LN5JHXPMENA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.554293132 (615h55m26.425376177s) [noise=true , address=145.79.227.43:4186   , piece_id=ENVLK652WUVPKR3VPGXXMJ4IAEFHK67JOODS26ASTM4I2V7ABUWA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.554505907 (615h55m26.425588942s) [noise=true , address=77.90.31.14:4673     , piece_id=XYDCDFNKMWJK4HDELS4NLBVLUBR76YHT4RP3M5JO5QMCSI6MOGYA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.554648990 (615h55m26.425732025s) [noise=true , address=64.49.13.140:4505    , piece_id=EBEDU3F3QMDLNIDSRRXUHGSZ3VH5ITKRDDI2QS5XKBOGKOCN3LNA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.554994317 (615h55m26.426077352s) [noise=true , address=64.49.13.169:4707    , piece_id=RB7LFR5DM7PUHCJPB5ABWN63Q2XQOGNOG2W2FCKXFWU5MGVPRPVA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.555494150 (615h55m26.426577185s) [noise=true , address=64.49.13.137:4500    , piece_id=DD4JKFRHB2FKWAOTS5RZU5M3ASDGJTRX2XRB5ZROVOZNPN6BER2Q, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.555978812 (615h55m26.427061847s) [noise=true , address=45.91.116.10:4087    , piece_id=MJUIJFY6KEXEMMQKWA2NC3RE5PG55QCDZ7ECWRZG7CBROSA7LL3Q, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.556325282 (615h55m26.427408317s) [noise=true , address=145.79.227.29:4965   , piece_id=L6L2H7KMBTYKC24ZMACWRYK6TGOK5KHDO2YCFP5TJS66NXXP3KNQ, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.556728720 (615h55m26.427811765s) [noise=true , address=45.91.116.6:4977     , piece_id=G7JKD7INUS3D3RUIHSSVYCSOVTHSRG5HLVKERSCAD5NUWSFRZ2EA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.557164801 (615h55m26.428247836s) [noise=true , address=145.79.227.49:4370   , piece_id=TA4D4VCQXWYCB6FTZDRU6ZVZ4V6PZCFKQMA3EJYL5UEHHMI66JDQ, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.557530657 (615h55m26.428613692s) [noise=true , address=145.79.227.33:4896   , piece_id=TCDN3ER54RKZQ73BEI6DCGQW3IUNJXNNC3DUXI7NH7SFIPLHTF4A, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.557963953 (615h55m26.429046998s) [noise=true , address=77.90.31.12:4620     , piece_id=AQOFDODGJ7TMWA3ZQGWJW7XZT3FSKK3GR4R4RSIFIMHYP3HVJQDA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.558357422 (615h55m26.429440457s) [noise=true , address=145.79.227.40:4067   , piece_id=ITZ4SYUISESGVG72CBXHMAHZAD3XOOWYO5UUSFIT22R765N3W25A, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.558648285 (615h55m26.42973133s) [noise=true , address=145.79.227.2:4147    , piece_id=SHUFD6UKDFM54RGOZGJEUOUYCMKLAN2IMLWKJXZIZ5PAZVNUE4JQ, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.559228450 (615h55m26.430311485s) [noise=true , address=145.79.227.44:4228   , piece_id=YAQ7I3TOSLCYCHI2SJEEGCYGTEFWHX2CEYVZASRFL3BODTEONFTA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:27.974040599 (615h55m27.845123634s) [upload=539756]: Next returned nil segment
Dec 31 07:24:27.974074934 (615h55m27.845157969s) [upload=539756]: Waiting for error group managing segments...
Dec 31 07:24:28.334073958 (615h55m28.205156993s) [noise=true , address=64.49.13.140:4505    , piece_id=EBEDU3F3QMDLNIDSRRXUHGSZ3VH5ITKRDDI2QS5XKBOGKOCN3LNA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.427413642 (615h55m28.298496677s) [noise=true , address=145.79.227.2:4147    , piece_id=SHUFD6UKDFM54RGOZGJEUOUYCMKLAN2IMLWKJXZIZ5PAZVNUE4JQ, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.522823096 (615h55m28.393906131s) [noise=true , address=145.79.227.29:4965   , piece_id=L6L2H7KMBTYKC24ZMACWRYK6TGOK5KHDO2YCFP5TJS66NXXP3KNQ, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.585931513 (615h55m28.457014558s) [noise=true , address=77.90.31.13:4656     , piece_id=3PKC4JZPZVBQOBUHPQAG4HZ67DRTOY7GR35QVWLV3LN5JHXPMENA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.591579187 (615h55m28.462662262s) [noise=true , address=64.49.13.137:4500    , piece_id=DD4JKFRHB2FKWAOTS5RZU5M3ASDGJTRX2XRB5ZROVOZNPN6BER2Q, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.615175368 (615h55m28.486258413s) [noise=true , address=77.90.31.14:4673     , piece_id=XYDCDFNKMWJK4HDELS4NLBVLUBR76YHT4RP3M5JO5QMCSI6MOGYA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.699191016 (615h55m28.570274051s) [noise=true , address=45.91.116.6:4977     , piece_id=G7JKD7INUS3D3RUIHSSVYCSOVTHSRG5HLVKERSCAD5NUWSFRZ2EA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.713295994 (615h55m28.58437908s) [noise=true , address=45.91.116.10:4087    , piece_id=MJUIJFY6KEXEMMQKWA2NC3RE5PG55QCDZ7ECWRZG7CBROSA7LL3Q, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.794645326 (615h55m28.665728361s) [noise=true , address=77.90.31.12:4620     , piece_id=AQOFDODGJ7TMWA3ZQGWJW7XZT3FSKK3GR4R4RSIFIMHYP3HVJQDA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.341891306 (615h55m29.212974351s) [noise=true , address=145.79.227.33:4896   , piece_id=TCDN3ER54RKZQ73BEI6DCGQW3IUNJXNNC3DUXI7NH7SFIPLHTF4A, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.345487925 (615h55m29.21657096s) [noise=true , address=145.79.227.49:4370   , piece_id=TA4D4VCQXWYCB6FTZDRU6ZVZ4V6PZCFKQMA3EJYL5UEHHMI66JDQ, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.406150395 (615h55m29.27723343s) [noise=true , address=145.79.227.44:4228   , piece_id=YAQ7I3TOSLCYCHI2SJEEGCYGTEFWHX2CEYVZASRFL3BODTEONFTA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.414891660 (615h55m29.285974695s) [noise=true , address=193.149.164.143:4698 , piece_id=IYUHVRK3ZH3JADUJI627OUTHO7CRPNNKZVQTOCALMDYKVDGSHIJQ, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.431785668 (615h55m29.302868703s) [noise=true , address=145.79.227.40:4067   , piece_id=ITZ4SYUISESGVG72CBXHMAHZAD3XOOWYO5UUSFIT22R765N3W25A, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.431807049 (615h55m29.302890104s) [seg_pos={0 0}, upload=539756]: Segment reached optimal threshold of 14 pieces.
Dec 31 07:24:29.432061032 (615h55m29.303144087s) [noise=true , address=145.79.227.30:4784   , piece_id=M5NJQBX6JI5QI2PPA52B6A2CNLOXVWKYYYNX6CJDAA4WJNT4SHWA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pfXWex2WsuhmLb2smRpE46cHT9aipSTHW3e5oyELS1GhF1cFr): protocol: expected piece hash; context canceled; EOF
Dec 31 07:24:29.432072876 (615h55m29.303155931s) [noise=true , address=145.79.227.30:4784   , piece_id=M5NJQBX6JI5QI2PPA52B6A2CNLOXVWKYYYNX6CJDAA4WJNT4SHWA, seg_pos={0 0}, upload=539756]: Long tail cancellation due to optimal threshold
Dec 31 07:24:29.432090819 (615h55m29.303173884s) [noise=true , address=64.49.13.169:4709    , piece_id=CBAB3KCCPOW3P3LYM4T73CVDOD7FPJCPXYIXKDV7XTC2FKESTDRQ, seg_pos={0 0}, upload=539756]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:33942->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:33942->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33942->64.49.13.169:4709: use of closed network connection
Dec 31 07:24:29.432098063 (615h55m29.303181118s) [noise=true , address=64.49.13.169:4709    , piece_id=CBAB3KCCPOW3P3LYM4T73CVDOD7FPJCPXYIXKDV7XTC2FKESTDRQ, seg_pos={0 0}, upload=539756]: Long tail cancellation due to optimal threshold
Dec 31 07:24:29.432212572 (615h55m29.303295617s) [noise=true , address=64.49.13.169:4707    , piece_id=RB7LFR5DM7PUHCJPB5ABWN63Q2XQOGNOG2W2FCKXFWU5MGVPRPVA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:41264->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:41264->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41264->64.49.13.169:4707: use of closed network connection
Dec 31 07:24:29.432220747 (615h55m29.303303782s) [noise=true , address=64.49.13.169:4707    , piece_id=RB7LFR5DM7PUHCJPB5ABWN63Q2XQOGNOG2W2FCKXFWU5MGVPRPVA, seg_pos={0 0}, upload=539756]: Long tail cancellation due to optimal threshold
Dec 31 07:24:29.432465934 (615h55m29.303548979s) [noise=true , address=145.79.227.43:4186   , piece_id=ENVLK652WUVPKR3VPGXXMJ4IAEFHK67JOODS26ASTM4I2V7ABUWA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Gu3vXy2JUkKDaF4MkMCgVmYspW4Ez8EAfv6e4GeCkrwYTe5T): protocol: expected piece hash; context canceled; EOF
Dec 31 07:24:29.432474971 (615h55m29.303558006s) [noise=true , address=145.79.227.43:4186   , piece_id=ENVLK652WUVPKR3VPGXXMJ4IAEFHK67JOODS26ASTM4I2V7ABUWA, seg_pos={0 0}, upload=539756]: Long tail cancellation due to optimal threshold
Dec 31 07:24:29.432580261 (615h55m29.303663296s) [seg_pos={0 0}, upload=539756]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 07:24:29.432664502 (615h55m29.303747547s) [upload=539756]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 07:24:29.432671305 (615h55m29.30375435s) [upload=539756]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 07:24:29.717299118 (615h55m29.588382153s) [upload=539756]: Done uploading segments...
Dec 31 07:24:29.717320589 (615h55m29.588403624s) [upload=539756]: Upload finished. err: <nil>
Duration: 41.89075255s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)
antiblip: 
1 out of 60 is failed (threshold=3) [..............................................X.............]
duration: 
3.219385651s
trace: 
https://jaeger.dev.storj.io/trace/59c3b821b75eba3a
FAIL (2025-12-31 07:53:38 UTC) 
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded
Duration: 3.219435176s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)
antiblip: 
1 out of 60 is failed (threshold=3) [...............................................X............]
duration: 
3.586716175s
trace: 
https://jaeger.dev.storj.io/trace/59c3b821b75eba3a
FAIL (2025-12-31 07:53:07 UTC) 
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded
Duration: 3.586733018s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
19 out of 60 is failed (threshold=3) [X....X.X.X.X.X......XXX.X....XXX..X...........X......X.X.X.X]
duration: 
30.003479253s
trace: 
https://jaeger.dev.storj.io/trace/780f2a5d425cca30
FAIL (2025-12-31 07:52:53 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: 2UJBHMT3INMNVGMJRLAE5GRJZX2Z7QU4VANEYZZUFBVW5MQJHU7Q) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: SY3FY6XNHIYZNGAIIH4MFZTYWAJ7742Y7DTPVQ7YGFU5OBI6AXIQ) context deadline exceeded
======LOGS======
t segment. Inline: false
Dec 31 07:52:23.825346530 (616h23m23.696429575s) [upload=540161]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 07:52:23.825353353 (616h23m23.696436408s) [upload=540161]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 07:52:24.074620447 (616h23m23.945703482s) [seg_pos={0 0}, upload=540161]: Begin upload segment...
Dec 31 07:52:24.074671584 (616h23m23.945754629s) [seg_pos={0 0}, upload=540161]: Done begin upload segment.
Dec 31 07:52:24.074706801 (616h23m23.945789846s) [noise=true , address=145.79.227.23:4546   , piece_id=7ZDKTSRFXS3MA3LASWFHJAH4PKBQR6KCXPOCF67PE5L67HAHR22A, seg_pos={0 0}, upload=540161]: Uploading piece...
Dec 31 07:52:24.074842459 (616h23m23.945925504s) [noise=true , address=145.79.227.16:4910   , piece_id=B7SVVOHQGMLHW2MRYXPVGM7FRJVA24VA36EFOMRZ5455NUUQO2MQ, seg_pos={0 0}, upload=540161]: Uploading piece...
Dec 31 07:52:24.075343253 (616h23m23.946426298s) [noise=true , address=145.79.227.41:4086   , piece_id=Z4EACFPQMDUKTECKPNZJWSPY2TWOPA34762C5A444LOQSAEETUKA, seg_pos={0 0}, upload=540161]: Uploading piece...
Dec 31 07:52:24.076077171 (616h23m23.947160216s) [noise=true , address=145.79.227.19:4077   , piece_id=6OPSEG3UJICOOO3FB7C5PUDJABWHJ6A675P3LOBKKLXXQSBQ4K5Q, seg_pos={0 0}, upload=540161]: Uploading piece...
Dec 31 07:52:24.076233398 (616h23m23.947316443s) [noise=true , address=45.91.116.6:4975     , piece_id=SY3FY6XNHIYZNGAIIH4MFZTYWAJ7742Y7DTPVQ7YGFU5OBI6AXIQ, seg_pos={0 0}, upload=540161]: Uploading piece...
Dec 31 07:52:24.077034253 (616h23m23.948117288s) [noise=true , address=77.90.31.11:4554     , piece_id=KQ2MICSQHJWDJIJV3A34AG5LQBQ72NGXHDH5MTHGFNXMB2E53AYA, seg_pos={0 0}, upload=540161]: Uploading piece...
Dec 31 07:52:24.077068127 (616h23m23.948151162s) [noise=true , address=145.79.227.46:4603   , piece_id=WQXPW3ZMGWNSMCKKCUQCTPZEIVZODH2D2PT6BQEUTHLKEJL5LQVA, seg_pos={0 0}, upload=540161]: Uploading piece...
Dec 31 07:52:24.078082028 (616h23m23.949165073s) [noise=true , address=45.91.116.5:4300     , piece_id=ZLMSRRIIWQJPE6O7W7VE537CSSFXTCSAUAIQLYXMFP7LVUDC3AXQ, seg_pos={0 0}, upload=540161]: Uploading piece...
Dec 31 07:52:24.078723069 (616h23m23.949806114s) [noise=true , address=145.79.227.39:4014   , piece_id=MRV5LXPT44YDINXDSXB2YLP34VVQTXGXOMSB6E7VXVWHLCGNHHKQ, seg_pos={0 0}, upload=540161]: Uploading piece...
Dec 31 07:52:24.079269931 (616h23m23.950352966s) [noise=true , address=145.79.227.8:4329    , piece_id=B6OJP2STGAZ2MMZZ6Q6XS4LETPSIQFCHW32Q33DVRCSS3EILXAWA, seg_pos={0 0}, upload=540161]: Uploading piece...
Dec 31 07:52:24.079733483 (616h23m23.950816528s) [noise=true , address=145.79.227.42:4122   , piece_id=O6PSEQ7QUWP5MZXC4T3IEOUKXMZK6SME6DJBDW5IYIAIV2D7STAQ, seg_pos={0 0}, upload=540161]: Uploading piece...
Dec 31 07:52:24.080171987 (616h23m23.951255032s) [noise=true , address=145.79.227.24:4583   , piece_id=LGKDSJP424LWHOFRTWJBVPEY5XURZOBIQJGKISABZLRNACMHOP2A, seg_pos={0 0}, upload=540161]: Uploading piece...
Dec 31 07:52:24.081184636 (616h23m23.952267672s) [noise=true , address=45.91.116.14:4192    , piece_id=PA5XI2SEPTAM6NHLSZQU7WG5YJCTSONTMT5GGHNBIKBPTTXUVDQQ, seg_pos={0 0}, upload=540161]: Uploading piece...
Dec 31 07:52:24.081721027 (616h23m23.952804063s) [noise=true , address=45.91.116.10:4081    , piece_id=2UJBHMT3INMNVGMJRLAE5GRJZX2Z7QU4VANEYZZUFBVW5MQJHU7Q, seg_pos={0 0}, upload=540161]: Uploading piece...
Dec 31 07:52:24.082054523 (616h23m23.953137568s) [noise=true , address=45.91.116.7:4012     , piece_id=ZEHO5WPKHJCC3PDYJ24CQ6IRWV2MZHHBZGEGIPXDUN2THXI3XPTQ, seg_pos={0 0}, upload=540161]: Uploading piece...
Dec 31 07:52:24.082268981 (616h23m23.953352026s) [noise=true , address=145.79.227.50:4375   , piece_id=GQS7PTQTWLEKXJCOKXN6HZCL4DDUV2R6M662WLFLV64YMLB3OLNA, seg_pos={0 0}, upload=540161]: Uploading piece...
Dec 31 07:52:25.682650871 (616h23m25.553733906s) [upload=540161]: Next returned nil segment
Dec 31 07:52:25.682662573 (616h23m25.553745618s) [upload=540161]: Waiting for error group managing segments...
Dec 31 07:52:26.064740170 (616h23m25.935823215s) [noise=true , address=145.79.227.42:4122   , piece_id=O6PSEQ7QUWP5MZXC4T3IEOUKXMZK6SME6DJBDW5IYIAIV2D7STAQ, seg_pos={0 0}, upload=540161]: Done uploading piece. err: <nil>
Dec 31 07:52:26.065650974 (616h23m25.936734009s) [noise=true , address=145.79.227.23:4546   , piece_id=7ZDKTSRFXS3MA3LASWFHJAH4PKBQR6KCXPOCF67PE5L67HAHR22A, seg_pos={0 0}, upload=540161]: Done uploading piece. err: <nil>
Dec 31 07:52:26.068003122 (616h23m25.939086157s) [noise=true , address=145.79.227.19:4077   , piece_id=6OPSEG3UJICOOO3FB7C5PUDJABWHJ6A675P3LOBKKLXXQSBQ4K5Q, seg_pos={0 0}, upload=540161]: Done uploading piece. err: <nil>
Dec 31 07:52:26.068771295 (616h23m25.93985433s) [noise=true , address=145.79.227.39:4014   , piece_id=MRV5LXPT44YDINXDSXB2YLP34VVQTXGXOMSB6E7VXVWHLCGNHHKQ, seg_pos={0 0}, upload=540161]: Done uploading piece. err: <nil>
Dec 31 07:52:26.082898245 (616h23m25.95398129s) [noise=true , address=145.79.227.41:4086   , piece_id=Z4EACFPQMDUKTECKPNZJWSPY2TWOPA34762C5A444LOQSAEETUKA, seg_pos={0 0}, upload=540161]: Done uploading piece. err: <nil>
Dec 31 07:52:26.084216175 (616h23m25.95529922s) [noise=true , address=145.79.227.8:4329    , piece_id=B6OJP2STGAZ2MMZZ6Q6XS4LETPSIQFCHW32Q33DVRCSS3EILXAWA, seg_pos={0 0}, upload=540161]: Done uploading piece. err: <nil>
Dec 31 07:52:26.091993925 (616h23m25.96307697s) [noise=true , address=145.79.227.46:4603   , piece_id=WQXPW3ZMGWNSMCKKCUQCTPZEIVZODH2D2PT6BQEUTHLKEJL5LQVA, seg_pos={0 0}, upload=540161]: Done uploading piece. err: <nil>
Dec 31 07:52:26.145871666 (616h23m26.016954701s) [noise=true , address=145.79.227.16:4910   , piece_id=B7SVVOHQGMLHW2MRYXPVGM7FRJVA24VA36EFOMRZ5455NUUQO2MQ, seg_pos={0 0}, upload=540161]: Done uploading piece. err: <nil>
Dec 31 07:52:26.154342947 (616h23m26.025425982s) [noise=true , address=145.79.227.50:4375   , piece_id=GQS7PTQTWLEKXJCOKXN6HZCL4DDUV2R6M662WLFLV64YMLB3OLNA, seg_pos={0 0}, upload=540161]: Done uploading piece. err: <nil>
Dec 31 07:52:26.156875870 (616h23m26.027958915s) [noise=true , address=145.79.227.24:4583   , piece_id=LGKDSJP424LWHOFRTWJBVPEY5XURZOBIQJGKISABZLRNACMHOP2A, seg_pos={0 0}, upload=540161]: Done uploading piece. err: <nil>
Dec 31 07:52:26.404838770 (616h23m26.275921805s) [noise=true , address=45.91.116.6:4975     , piece_id=SY3FY6XNHIYZNGAIIH4MFZTYWAJ7742Y7DTPVQ7YGFU5OBI6AXIQ, seg_pos={0 0}, upload=540161]: Done uploading piece. err: <nil>
Dec 31 07:52:26.487485923 (616h23m26.358568968s) [noise=true , address=45.91.116.10:4081    , piece_id=2UJBHMT3INMNVGMJRLAE5GRJZX2Z7QU4VANEYZZUFBVW5MQJHU7Q, seg_pos={0 0}, upload=540161]: Done uploading piece. err: <nil>
Dec 31 07:52:26.558106627 (616h23m26.429189662s) [noise=true , address=45.91.116.14:4192    , piece_id=PA5XI2SEPTAM6NHLSZQU7WG5YJCTSONTMT5GGHNBIKBPTTXUVDQQ, seg_pos={0 0}, upload=540161]: Done uploading piece. err: <nil>
Dec 31 07:52:26.558129981 (616h23m26.429213027s) [seg_pos={0 0}, upload=540161]: Segment reached optimal threshold of 13 pieces.
Dec 31 07:52:26.558482913 (616h23m26.429565948s) [noise=true , address=45.91.116.7:4012     , piece_id=ZEHO5WPKHJCC3PDYJ24CQ6IRWV2MZHHBZGEGIPXDUN2THXI3XPTQ, seg_pos={0 0}, upload=540161]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: write tcp 188.166.225.170:46362->45.91.116.7:4012: use of closed network connection; write tcp 188.166.225.170:46362->45.91.116.7:4012: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46362->45.91.116.7:4012: use of closed network connection
Dec 31 07:52:26.558491019 (616h23m26.429574064s) [noise=true , address=45.91.116.7:4012     , piece_id=ZEHO5WPKHJCC3PDYJ24CQ6IRWV2MZHHBZGEGIPXDUN2THXI3XPTQ, seg_pos={0 0}, upload=540161]: Long tail cancellation due to optimal threshold
Dec 31 07:52:26.558809525 (616h23m26.42989256s) [noise=true , address=77.90.31.11:4554     , piece_id=KQ2MICSQHJWDJIJV3A34AG5LQBQ72NGXHDH5MTHGFNXMB2E53AYA, seg_pos={0 0}, upload=540161]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: write tcp 188.166.225.170:44202->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:44202->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44202->77.90.31.11:4554: use of closed network connection
Dec 31 07:52:26.558818873 (616h23m26.429901919s) [noise=true , address=77.90.31.11:4554     , piece_id=KQ2MICSQHJWDJIJV3A34AG5LQBQ72NGXHDH5MTHGFNXMB2E53AYA, seg_pos={0 0}, upload=540161]: Long tail cancellation due to optimal threshold
Dec 31 07:52:26.559143732 (616h23m26.430226767s) [noise=true , address=45.91.116.5:4300     , piece_id=ZLMSRRIIWQJPE6O7W7VE537CSSFXTCSAUAIQLYXMFP7LVUDC3AXQ, seg_pos={0 0}, upload=540161]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): internal: context canceled; piecestore: piecestore close: write tcp 188.166.225.170:60356->45.91.116.5:4300: use of closed network connection
Dec 31 07:52:26.559151727 (616h23m26.430234762s) [noise=true , address=45.91.116.5:4300     , piece_id=ZLMSRRIIWQJPE6O7W7VE537CSSFXTCSAUAIQLYXMFP7LVUDC3AXQ, seg_pos={0 0}, upload=540161]: Long tail cancellation due to optimal threshold
Dec 31 07:52:26.559170673 (616h23m26.430253718s) [seg_pos={0 0}, upload=540161]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 07:52:26.559276565 (616h23m26.4303596s) [upload=540161]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 07:52:26.559283297 (616h23m26.430366343s) [upload=540161]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 07:52:26.851054996 (616h23m26.722138032s) [upload=540161]: Done uploading segments...
Dec 31 07:52:26.851068904 (616h23m26.722151939s) [upload=540161]: Upload finished. err: <nil>
Duration: 30.003498259s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
2 out of 60 is failed (threshold=3) [.........X............X.....................................]
duration: 
15.175449843s
trace: 
https://jaeger.dev.storj.io/trace/6ce9bac61d803c5a
FAIL (2025-12-31 07:52:52 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
56     , piece_id=3PKC4JZPZVBQOBUHPQAG4HZ67DRTOY7GR35QVWLV3LN5JHXPMENA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.554293132 (615h55m26.425376177s) [noise=true , address=145.79.227.43:4186   , piece_id=ENVLK652WUVPKR3VPGXXMJ4IAEFHK67JOODS26ASTM4I2V7ABUWA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.554505907 (615h55m26.425588942s) [noise=true , address=77.90.31.14:4673     , piece_id=XYDCDFNKMWJK4HDELS4NLBVLUBR76YHT4RP3M5JO5QMCSI6MOGYA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.554648990 (615h55m26.425732025s) [noise=true , address=64.49.13.140:4505    , piece_id=EBEDU3F3QMDLNIDSRRXUHGSZ3VH5ITKRDDI2QS5XKBOGKOCN3LNA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.554994317 (615h55m26.426077352s) [noise=true , address=64.49.13.169:4707    , piece_id=RB7LFR5DM7PUHCJPB5ABWN63Q2XQOGNOG2W2FCKXFWU5MGVPRPVA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.555494150 (615h55m26.426577185s) [noise=true , address=64.49.13.137:4500    , piece_id=DD4JKFRHB2FKWAOTS5RZU5M3ASDGJTRX2XRB5ZROVOZNPN6BER2Q, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.555978812 (615h55m26.427061847s) [noise=true , address=45.91.116.10:4087    , piece_id=MJUIJFY6KEXEMMQKWA2NC3RE5PG55QCDZ7ECWRZG7CBROSA7LL3Q, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.556325282 (615h55m26.427408317s) [noise=true , address=145.79.227.29:4965   , piece_id=L6L2H7KMBTYKC24ZMACWRYK6TGOK5KHDO2YCFP5TJS66NXXP3KNQ, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.556728720 (615h55m26.427811765s) [noise=true , address=45.91.116.6:4977     , piece_id=G7JKD7INUS3D3RUIHSSVYCSOVTHSRG5HLVKERSCAD5NUWSFRZ2EA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.557164801 (615h55m26.428247836s) [noise=true , address=145.79.227.49:4370   , piece_id=TA4D4VCQXWYCB6FTZDRU6ZVZ4V6PZCFKQMA3EJYL5UEHHMI66JDQ, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.557530657 (615h55m26.428613692s) [noise=true , address=145.79.227.33:4896   , piece_id=TCDN3ER54RKZQ73BEI6DCGQW3IUNJXNNC3DUXI7NH7SFIPLHTF4A, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.557963953 (615h55m26.429046998s) [noise=true , address=77.90.31.12:4620     , piece_id=AQOFDODGJ7TMWA3ZQGWJW7XZT3FSKK3GR4R4RSIFIMHYP3HVJQDA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.558357422 (615h55m26.429440457s) [noise=true , address=145.79.227.40:4067   , piece_id=ITZ4SYUISESGVG72CBXHMAHZAD3XOOWYO5UUSFIT22R765N3W25A, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.558648285 (615h55m26.42973133s) [noise=true , address=145.79.227.2:4147    , piece_id=SHUFD6UKDFM54RGOZGJEUOUYCMKLAN2IMLWKJXZIZ5PAZVNUE4JQ, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.559228450 (615h55m26.430311485s) [noise=true , address=145.79.227.44:4228   , piece_id=YAQ7I3TOSLCYCHI2SJEEGCYGTEFWHX2CEYVZASRFL3BODTEONFTA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:27.974040599 (615h55m27.845123634s) [upload=539756]: Next returned nil segment
Dec 31 07:24:27.974074934 (615h55m27.845157969s) [upload=539756]: Waiting for error group managing segments...
Dec 31 07:24:28.334073958 (615h55m28.205156993s) [noise=true , address=64.49.13.140:4505    , piece_id=EBEDU3F3QMDLNIDSRRXUHGSZ3VH5ITKRDDI2QS5XKBOGKOCN3LNA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.427413642 (615h55m28.298496677s) [noise=true , address=145.79.227.2:4147    , piece_id=SHUFD6UKDFM54RGOZGJEUOUYCMKLAN2IMLWKJXZIZ5PAZVNUE4JQ, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.522823096 (615h55m28.393906131s) [noise=true , address=145.79.227.29:4965   , piece_id=L6L2H7KMBTYKC24ZMACWRYK6TGOK5KHDO2YCFP5TJS66NXXP3KNQ, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.585931513 (615h55m28.457014558s) [noise=true , address=77.90.31.13:4656     , piece_id=3PKC4JZPZVBQOBUHPQAG4HZ67DRTOY7GR35QVWLV3LN5JHXPMENA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.591579187 (615h55m28.462662262s) [noise=true , address=64.49.13.137:4500    , piece_id=DD4JKFRHB2FKWAOTS5RZU5M3ASDGJTRX2XRB5ZROVOZNPN6BER2Q, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.615175368 (615h55m28.486258413s) [noise=true , address=77.90.31.14:4673     , piece_id=XYDCDFNKMWJK4HDELS4NLBVLUBR76YHT4RP3M5JO5QMCSI6MOGYA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.699191016 (615h55m28.570274051s) [noise=true , address=45.91.116.6:4977     , piece_id=G7JKD7INUS3D3RUIHSSVYCSOVTHSRG5HLVKERSCAD5NUWSFRZ2EA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.713295994 (615h55m28.58437908s) [noise=true , address=45.91.116.10:4087    , piece_id=MJUIJFY6KEXEMMQKWA2NC3RE5PG55QCDZ7ECWRZG7CBROSA7LL3Q, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.794645326 (615h55m28.665728361s) [noise=true , address=77.90.31.12:4620     , piece_id=AQOFDODGJ7TMWA3ZQGWJW7XZT3FSKK3GR4R4RSIFIMHYP3HVJQDA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.341891306 (615h55m29.212974351s) [noise=true , address=145.79.227.33:4896   , piece_id=TCDN3ER54RKZQ73BEI6DCGQW3IUNJXNNC3DUXI7NH7SFIPLHTF4A, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.345487925 (615h55m29.21657096s) [noise=true , address=145.79.227.49:4370   , piece_id=TA4D4VCQXWYCB6FTZDRU6ZVZ4V6PZCFKQMA3EJYL5UEHHMI66JDQ, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.406150395 (615h55m29.27723343s) [noise=true , address=145.79.227.44:4228   , piece_id=YAQ7I3TOSLCYCHI2SJEEGCYGTEFWHX2CEYVZASRFL3BODTEONFTA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.414891660 (615h55m29.285974695s) [noise=true , address=193.149.164.143:4698 , piece_id=IYUHVRK3ZH3JADUJI627OUTHO7CRPNNKZVQTOCALMDYKVDGSHIJQ, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.431785668 (615h55m29.302868703s) [noise=true , address=145.79.227.40:4067   , piece_id=ITZ4SYUISESGVG72CBXHMAHZAD3XOOWYO5UUSFIT22R765N3W25A, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.431807049 (615h55m29.302890104s) [seg_pos={0 0}, upload=539756]: Segment reached optimal threshold of 14 pieces.
Dec 31 07:24:29.432061032 (615h55m29.303144087s) [noise=true , address=145.79.227.30:4784   , piece_id=M5NJQBX6JI5QI2PPA52B6A2CNLOXVWKYYYNX6CJDAA4WJNT4SHWA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pfXWex2WsuhmLb2smRpE46cHT9aipSTHW3e5oyELS1GhF1cFr): protocol: expected piece hash; context canceled; EOF
Dec 31 07:24:29.432072876 (615h55m29.303155931s) [noise=true , address=145.79.227.30:4784   , piece_id=M5NJQBX6JI5QI2PPA52B6A2CNLOXVWKYYYNX6CJDAA4WJNT4SHWA, seg_pos={0 0}, upload=539756]: Long tail cancellation due to optimal threshold
Dec 31 07:24:29.432090819 (615h55m29.303173884s) [noise=true , address=64.49.13.169:4709    , piece_id=CBAB3KCCPOW3P3LYM4T73CVDOD7FPJCPXYIXKDV7XTC2FKESTDRQ, seg_pos={0 0}, upload=539756]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:33942->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:33942->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33942->64.49.13.169:4709: use of closed network connection
Dec 31 07:24:29.432098063 (615h55m29.303181118s) [noise=true , address=64.49.13.169:4709    , piece_id=CBAB3KCCPOW3P3LYM4T73CVDOD7FPJCPXYIXKDV7XTC2FKESTDRQ, seg_pos={0 0}, upload=539756]: Long tail cancellation due to optimal threshold
Dec 31 07:24:29.432212572 (615h55m29.303295617s) [noise=true , address=64.49.13.169:4707    , piece_id=RB7LFR5DM7PUHCJPB5ABWN63Q2XQOGNOG2W2FCKXFWU5MGVPRPVA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:41264->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:41264->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41264->64.49.13.169:4707: use of closed network connection
Dec 31 07:24:29.432220747 (615h55m29.303303782s) [noise=true , address=64.49.13.169:4707    , piece_id=RB7LFR5DM7PUHCJPB5ABWN63Q2XQOGNOG2W2FCKXFWU5MGVPRPVA, seg_pos={0 0}, upload=539756]: Long tail cancellation due to optimal threshold
Dec 31 07:24:29.432465934 (615h55m29.303548979s) [noise=true , address=145.79.227.43:4186   , piece_id=ENVLK652WUVPKR3VPGXXMJ4IAEFHK67JOODS26ASTM4I2V7ABUWA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Gu3vXy2JUkKDaF4MkMCgVmYspW4Ez8EAfv6e4GeCkrwYTe5T): protocol: expected piece hash; context canceled; EOF
Dec 31 07:24:29.432474971 (615h55m29.303558006s) [noise=true , address=145.79.227.43:4186   , piece_id=ENVLK652WUVPKR3VPGXXMJ4IAEFHK67JOODS26ASTM4I2V7ABUWA, seg_pos={0 0}, upload=539756]: Long tail cancellation due to optimal threshold
Dec 31 07:24:29.432580261 (615h55m29.303663296s) [seg_pos={0 0}, upload=539756]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 07:24:29.432664502 (615h55m29.303747547s) [upload=539756]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 07:24:29.432671305 (615h55m29.30375435s) [upload=539756]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 07:24:29.717299118 (615h55m29.588382153s) [upload=539756]: Done uploading segments...
Dec 31 07:24:29.717320589 (615h55m29.588403624s) [upload=539756]: Upload finished. err: <nil>
Duration: 15.175467177s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
4 out of 60 is failed (threshold=6) [.X................................X.........X............X..]
duration: 
3.504088449s
trace: 
https://jaeger.dev.storj.io/trace/4dbc5f818d3c3b7
FAIL (2025-12-31 07:52:52 UTC) 
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.50410956s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
2 out of 60 is failed (threshold=3) [..........X............X....................................]
duration: 
47.965407746s
trace: 
https://jaeger.dev.storj.io/trace/6ce9bac61d803c5a
FAIL (2025-12-31 07:52:37 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
56     , piece_id=3PKC4JZPZVBQOBUHPQAG4HZ67DRTOY7GR35QVWLV3LN5JHXPMENA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.554293132 (615h55m26.425376177s) [noise=true , address=145.79.227.43:4186   , piece_id=ENVLK652WUVPKR3VPGXXMJ4IAEFHK67JOODS26ASTM4I2V7ABUWA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.554505907 (615h55m26.425588942s) [noise=true , address=77.90.31.14:4673     , piece_id=XYDCDFNKMWJK4HDELS4NLBVLUBR76YHT4RP3M5JO5QMCSI6MOGYA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.554648990 (615h55m26.425732025s) [noise=true , address=64.49.13.140:4505    , piece_id=EBEDU3F3QMDLNIDSRRXUHGSZ3VH5ITKRDDI2QS5XKBOGKOCN3LNA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.554994317 (615h55m26.426077352s) [noise=true , address=64.49.13.169:4707    , piece_id=RB7LFR5DM7PUHCJPB5ABWN63Q2XQOGNOG2W2FCKXFWU5MGVPRPVA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.555494150 (615h55m26.426577185s) [noise=true , address=64.49.13.137:4500    , piece_id=DD4JKFRHB2FKWAOTS5RZU5M3ASDGJTRX2XRB5ZROVOZNPN6BER2Q, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.555978812 (615h55m26.427061847s) [noise=true , address=45.91.116.10:4087    , piece_id=MJUIJFY6KEXEMMQKWA2NC3RE5PG55QCDZ7ECWRZG7CBROSA7LL3Q, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.556325282 (615h55m26.427408317s) [noise=true , address=145.79.227.29:4965   , piece_id=L6L2H7KMBTYKC24ZMACWRYK6TGOK5KHDO2YCFP5TJS66NXXP3KNQ, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.556728720 (615h55m26.427811765s) [noise=true , address=45.91.116.6:4977     , piece_id=G7JKD7INUS3D3RUIHSSVYCSOVTHSRG5HLVKERSCAD5NUWSFRZ2EA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.557164801 (615h55m26.428247836s) [noise=true , address=145.79.227.49:4370   , piece_id=TA4D4VCQXWYCB6FTZDRU6ZVZ4V6PZCFKQMA3EJYL5UEHHMI66JDQ, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.557530657 (615h55m26.428613692s) [noise=true , address=145.79.227.33:4896   , piece_id=TCDN3ER54RKZQ73BEI6DCGQW3IUNJXNNC3DUXI7NH7SFIPLHTF4A, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.557963953 (615h55m26.429046998s) [noise=true , address=77.90.31.12:4620     , piece_id=AQOFDODGJ7TMWA3ZQGWJW7XZT3FSKK3GR4R4RSIFIMHYP3HVJQDA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.558357422 (615h55m26.429440457s) [noise=true , address=145.79.227.40:4067   , piece_id=ITZ4SYUISESGVG72CBXHMAHZAD3XOOWYO5UUSFIT22R765N3W25A, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.558648285 (615h55m26.42973133s) [noise=true , address=145.79.227.2:4147    , piece_id=SHUFD6UKDFM54RGOZGJEUOUYCMKLAN2IMLWKJXZIZ5PAZVNUE4JQ, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.559228450 (615h55m26.430311485s) [noise=true , address=145.79.227.44:4228   , piece_id=YAQ7I3TOSLCYCHI2SJEEGCYGTEFWHX2CEYVZASRFL3BODTEONFTA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:27.974040599 (615h55m27.845123634s) [upload=539756]: Next returned nil segment
Dec 31 07:24:27.974074934 (615h55m27.845157969s) [upload=539756]: Waiting for error group managing segments...
Dec 31 07:24:28.334073958 (615h55m28.205156993s) [noise=true , address=64.49.13.140:4505    , piece_id=EBEDU3F3QMDLNIDSRRXUHGSZ3VH5ITKRDDI2QS5XKBOGKOCN3LNA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.427413642 (615h55m28.298496677s) [noise=true , address=145.79.227.2:4147    , piece_id=SHUFD6UKDFM54RGOZGJEUOUYCMKLAN2IMLWKJXZIZ5PAZVNUE4JQ, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.522823096 (615h55m28.393906131s) [noise=true , address=145.79.227.29:4965   , piece_id=L6L2H7KMBTYKC24ZMACWRYK6TGOK5KHDO2YCFP5TJS66NXXP3KNQ, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.585931513 (615h55m28.457014558s) [noise=true , address=77.90.31.13:4656     , piece_id=3PKC4JZPZVBQOBUHPQAG4HZ67DRTOY7GR35QVWLV3LN5JHXPMENA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.591579187 (615h55m28.462662262s) [noise=true , address=64.49.13.137:4500    , piece_id=DD4JKFRHB2FKWAOTS5RZU5M3ASDGJTRX2XRB5ZROVOZNPN6BER2Q, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.615175368 (615h55m28.486258413s) [noise=true , address=77.90.31.14:4673     , piece_id=XYDCDFNKMWJK4HDELS4NLBVLUBR76YHT4RP3M5JO5QMCSI6MOGYA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.699191016 (615h55m28.570274051s) [noise=true , address=45.91.116.6:4977     , piece_id=G7JKD7INUS3D3RUIHSSVYCSOVTHSRG5HLVKERSCAD5NUWSFRZ2EA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.713295994 (615h55m28.58437908s) [noise=true , address=45.91.116.10:4087    , piece_id=MJUIJFY6KEXEMMQKWA2NC3RE5PG55QCDZ7ECWRZG7CBROSA7LL3Q, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.794645326 (615h55m28.665728361s) [noise=true , address=77.90.31.12:4620     , piece_id=AQOFDODGJ7TMWA3ZQGWJW7XZT3FSKK3GR4R4RSIFIMHYP3HVJQDA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.341891306 (615h55m29.212974351s) [noise=true , address=145.79.227.33:4896   , piece_id=TCDN3ER54RKZQ73BEI6DCGQW3IUNJXNNC3DUXI7NH7SFIPLHTF4A, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.345487925 (615h55m29.21657096s) [noise=true , address=145.79.227.49:4370   , piece_id=TA4D4VCQXWYCB6FTZDRU6ZVZ4V6PZCFKQMA3EJYL5UEHHMI66JDQ, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.406150395 (615h55m29.27723343s) [noise=true , address=145.79.227.44:4228   , piece_id=YAQ7I3TOSLCYCHI2SJEEGCYGTEFWHX2CEYVZASRFL3BODTEONFTA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.414891660 (615h55m29.285974695s) [noise=true , address=193.149.164.143:4698 , piece_id=IYUHVRK3ZH3JADUJI627OUTHO7CRPNNKZVQTOCALMDYKVDGSHIJQ, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.431785668 (615h55m29.302868703s) [noise=true , address=145.79.227.40:4067   , piece_id=ITZ4SYUISESGVG72CBXHMAHZAD3XOOWYO5UUSFIT22R765N3W25A, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.431807049 (615h55m29.302890104s) [seg_pos={0 0}, upload=539756]: Segment reached optimal threshold of 14 pieces.
Dec 31 07:24:29.432061032 (615h55m29.303144087s) [noise=true , address=145.79.227.30:4784   , piece_id=M5NJQBX6JI5QI2PPA52B6A2CNLOXVWKYYYNX6CJDAA4WJNT4SHWA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pfXWex2WsuhmLb2smRpE46cHT9aipSTHW3e5oyELS1GhF1cFr): protocol: expected piece hash; context canceled; EOF
Dec 31 07:24:29.432072876 (615h55m29.303155931s) [noise=true , address=145.79.227.30:4784   , piece_id=M5NJQBX6JI5QI2PPA52B6A2CNLOXVWKYYYNX6CJDAA4WJNT4SHWA, seg_pos={0 0}, upload=539756]: Long tail cancellation due to optimal threshold
Dec 31 07:24:29.432090819 (615h55m29.303173884s) [noise=true , address=64.49.13.169:4709    , piece_id=CBAB3KCCPOW3P3LYM4T73CVDOD7FPJCPXYIXKDV7XTC2FKESTDRQ, seg_pos={0 0}, upload=539756]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:33942->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:33942->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33942->64.49.13.169:4709: use of closed network connection
Dec 31 07:24:29.432098063 (615h55m29.303181118s) [noise=true , address=64.49.13.169:4709    , piece_id=CBAB3KCCPOW3P3LYM4T73CVDOD7FPJCPXYIXKDV7XTC2FKESTDRQ, seg_pos={0 0}, upload=539756]: Long tail cancellation due to optimal threshold
Dec 31 07:24:29.432212572 (615h55m29.303295617s) [noise=true , address=64.49.13.169:4707    , piece_id=RB7LFR5DM7PUHCJPB5ABWN63Q2XQOGNOG2W2FCKXFWU5MGVPRPVA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:41264->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:41264->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41264->64.49.13.169:4707: use of closed network connection
Dec 31 07:24:29.432220747 (615h55m29.303303782s) [noise=true , address=64.49.13.169:4707    , piece_id=RB7LFR5DM7PUHCJPB5ABWN63Q2XQOGNOG2W2FCKXFWU5MGVPRPVA, seg_pos={0 0}, upload=539756]: Long tail cancellation due to optimal threshold
Dec 31 07:24:29.432465934 (615h55m29.303548979s) [noise=true , address=145.79.227.43:4186   , piece_id=ENVLK652WUVPKR3VPGXXMJ4IAEFHK67JOODS26ASTM4I2V7ABUWA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Gu3vXy2JUkKDaF4MkMCgVmYspW4Ez8EAfv6e4GeCkrwYTe5T): protocol: expected piece hash; context canceled; EOF
Dec 31 07:24:29.432474971 (615h55m29.303558006s) [noise=true , address=145.79.227.43:4186   , piece_id=ENVLK652WUVPKR3VPGXXMJ4IAEFHK67JOODS26ASTM4I2V7ABUWA, seg_pos={0 0}, upload=539756]: Long tail cancellation due to optimal threshold
Dec 31 07:24:29.432580261 (615h55m29.303663296s) [seg_pos={0 0}, upload=539756]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 07:24:29.432664502 (615h55m29.303747547s) [upload=539756]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 07:24:29.432671305 (615h55m29.30375435s) [upload=539756]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 07:24:29.717299118 (615h55m29.588382153s) [upload=539756]: Done uploading segments...
Dec 31 07:24:29.717320589 (615h55m29.588403624s) [upload=539756]: Upload finished. err: <nil>
Duration: 47.965428094s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [..............................................X.............]
duration: 
6.850578263s
trace: 
https://jaeger.dev.storj.io/trace/5dd0609e3a2ac547
FAIL (2025-12-31 07:52:36 UTC) 
orbiter warning: context deadline exceeded
Duration: 6.850600445s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)
antiblip: 
1 out of 60 is failed (threshold=3) [................................................X...........]
duration: 
3.82334768s
trace: 
https://jaeger.dev.storj.io/trace/59c3b821b75eba3a
FAIL (2025-12-31 07:52:16 UTC) 
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded
Duration: 3.823385851s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
4 out of 60 is failed (threshold=6) [..X................................X.........X............X.]
duration: 
4.689519597s
trace: 
https://jaeger.dev.storj.io/trace/4dbc5f818d3c3b7
FAIL (2025-12-31 07:52:02 UTC) 
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.689545786s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
18 out of 60 is failed (threshold=3) [.X....X.X.X.X.X......XXX.X....XXX..X...........X......X.X.X.]
duration: 
5.828839369s
trace: 
https://jaeger.dev.storj.io/trace/b9a66a0708d887e
FAIL (2025-12-31 07:52:00 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 31 07:51:02.386794824 (616h22m2.257877859s) [upload=540141]: Starting upload
Dec 31 07:51:02.386883673 (616h22m2.257966748s) [upload=540141]: Done starting upload
Dec 31 07:51:02.387061752 (616h22m2.258144807s) [upload=540141]: Uploading segments...
Dec 31 07:51:02.387169487 (616h22m2.258252532s) [upload=540141]: Got next segment. Inline: false
Dec 31 07:51:02.387235102 (616h22m2.258318157s) [upload=540141]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 07:51:02.387242135 (616h22m2.25832518s) [upload=540141]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 07:51:02.631477047 (616h22m2.502560102s) [seg_pos={0 0}, upload=540141]: Begin upload segment...
Dec 31 07:51:02.631536671 (616h22m2.502619716s) [seg_pos={0 0}, upload=540141]: Done begin upload segment.
Dec 31 07:51:02.631572469 (616h22m2.502655524s) [noise=true , address=145.79.227.5:4223    , piece_id=U7A2ZGU3CUSID72JOCL23CVC3BSYOYHRW6MTDHGFMQB2V34MY7XA, seg_pos={0 0}, upload=540141]: Uploading piece...
Dec 31 07:51:02.633123201 (616h22m2.504206246s) [noise=true , address=145.79.227.44:4193   , piece_id=AIE6JX7EQMVED7XDIWNQ65TZJTJVCY5CR7IWQCE37I3M6RYWVSKQ, seg_pos={0 0}, upload=540141]: Uploading piece...
Dec 31 07:51:02.634110732 (616h22m2.505193777s) [noise=true , address=145.79.227.36:4517   , piece_id=4GCERGAIUOSKTH533BBCFGSZUZLTOIJJWAMTFEKUMBZRQARYIK6Q, seg_pos={0 0}, upload=540141]: Uploading piece...
Dec 31 07:51:02.634670077 (616h22m2.505753122s) [noise=true , address=145.79.227.29:4943   , piece_id=7ZCXKYQET76RXQ4HVQJFLZBUSXCVVYYQO7B7VH5EFQX5RBFLD4QA, seg_pos={0 0}, upload=540141]: Uploading piece...
Dec 31 07:51:02.635257716 (616h22m2.506340761s) [noise=true , address=77.90.31.16:4735     , piece_id=GPHX2PRAK34RGV4QSKDA2EP377FG26VY4RQAEYILTXVNXKYQ4IRA, seg_pos={0 0}, upload=540141]: Uploading piece...
Dec 31 07:51:02.636355436 (616h22m2.507438471s) [noise=true , address=145.79.227.24:4582   , piece_id=QSMFJVHVRFVFDL4DT2UHLWMVF66HTQQ3X3HETXKVL4JERZFCP4QA, seg_pos={0 0}, upload=540141]: Uploading piece...
Dec 31 07:51:02.637031744 (616h22m2.508114789s) [noise=true , address=145.79.227.13:4804   , piece_id=QUGILQI63G7CTO6FEKLQYPH3GTGYBFVIHFEGTQZ4FPNKVHIJ6O7A, seg_pos={0 0}, upload=540141]: Uploading piece...
Dec 31 07:51:02.637900207 (616h22m2.508983252s) [noise=true , address=77.90.31.11:4554     , piece_id=UN3ILRE7M3GJOWR6MZ5TKFIMFKJ64VHJI54ALMRVBBG5GM7ZVCGQ, seg_pos={0 0}, upload=540141]: Uploading piece...
Dec 31 07:51:02.639562833 (616h22m2.510645878s) [noise=true , address=145.79.227.50:4375   , piece_id=JYNH7FCK7S5J33MUIKIKBQL4LEXE2AUIGP27F7ILGDF6BCLYRVNA, seg_pos={0 0}, upload=540141]: Uploading piece...
Dec 31 07:51:02.641683061 (616h22m2.512766116s) [noise=true , address=45.91.116.5:4297     , piece_id=2FPB4D776UYFTR4AL4NAWLKRC2NCAOECKIVAISSFV3MQ5EFIHFXA, seg_pos={0 0}, upload=540141]: Uploading piece...
Dec 31 07:51:02.644611627 (616h22m2.515694672s) [noise=true , address=145.79.227.17:4970   , piece_id=IEZ3XCQ4ILSHKG3FMPUMRX3ED6K375LQGZDOIPO3RDXH3HPFEMGQ, seg_pos={0 0}, upload=540141]: Uploading piece...
Dec 31 07:51:02.646169363 (616h22m2.517252408s) [noise=true , address=145.79.227.45:4229   , piece_id=QNNEZTZLYLXB4CJHIQZ7XWFFNQTJHHGB4G6A6OWJPM5CULAT2MMA, seg_pos={0 0}, upload=540141]: Uploading piece...
Dec 31 07:51:02.647613794 (616h22m2.518696849s) [noise=true , address=145.79.227.41:4085   , piece_id=RVNDIJAJBAPI2HUIH7INGP7743IZKZ6FJE4S46E5XVWMQF4XKXBA, seg_pos={0 0}, upload=540141]: Uploading piece...
Dec 31 07:51:02.649442155 (616h22m2.52052521s) [noise=true , address=145.79.227.2:4113    , piece_id=4UE7ZTZQZ4W5HILOQWKBG5PZLOLKEPNTAVCQWB4HEIPHY3MBANBQ, seg_pos={0 0}, upload=540141]: Uploading piece...
Dec 31 07:51:02.650856880 (616h22m2.521939935s) [noise=true , address=145.79.227.12:4765   , piece_id=WD57BZBPHBVRZ4BJLY6JIRLQEESFXMAHZEHNKPQBIESPCZTAUBXQ, seg_pos={0 0}, upload=540141]: Uploading piece...
Dec 31 07:51:02.651427886 (616h22m2.522510921s) [noise=true , address=145.79.227.31:4798   , piece_id=UQJP3U5QFT5JHJ5DTP7QRZNVY2535FPVODKFOABGW3WP74NOPYDQ, seg_pos={0 0}, upload=540141]: Uploading piece...
Dec 31 07:51:04.306991452 (616h22m4.178074497s) [upload=540141]: Next returned nil segment
Dec 31 07:51:04.307028723 (616h22m4.178111768s) [upload=540141]: Waiting for error group managing segments...
Dec 31 07:51:04.637093559 (616h22m4.508176594s) [noise=true , address=145.79.227.5:4223    , piece_id=U7A2ZGU3CUSID72JOCL23CVC3BSYOYHRW6MTDHGFMQB2V34MY7XA, seg_pos={0 0}, upload=540141]: Done uploading piece. err: <nil>
Dec 31 07:51:04.639966499 (616h22m4.511049544s) [noise=true , address=145.79.227.2:4113    , piece_id=4UE7ZTZQZ4W5HILOQWKBG5PZLOLKEPNTAVCQWB4HEIPHY3MBANBQ, seg_pos={0 0}, upload=540141]: Done uploading piece. err: <nil>
Dec 31 07:51:04.641266174 (616h22m4.51234922s) [noise=true , address=145.79.227.13:4804   , piece_id=QUGILQI63G7CTO6FEKLQYPH3GTGYBFVIHFEGTQZ4FPNKVHIJ6O7A, seg_pos={0 0}, upload=540141]: Done uploading piece. err: <nil>
Dec 31 07:51:04.648310517 (616h22m4.519393562s) [noise=true , address=145.79.227.45:4229   , piece_id=QNNEZTZLYLXB4CJHIQZ7XWFFNQTJHHGB4G6A6OWJPM5CULAT2MMA, seg_pos={0 0}, upload=540141]: Done uploading piece. err: <nil>
Dec 31 07:51:04.656511744 (616h22m4.527594789s) [noise=true , address=145.79.227.41:4085   , piece_id=RVNDIJAJBAPI2HUIH7INGP7743IZKZ6FJE4S46E5XVWMQF4XKXBA, seg_pos={0 0}, upload=540141]: Done uploading piece. err: <nil>
Dec 31 07:51:04.663003926 (616h22m4.534086972s) [noise=true , address=145.79.227.36:4517   , piece_id=4GCERGAIUOSKTH533BBCFGSZUZLTOIJJWAMTFEKUMBZRQARYIK6Q, seg_pos={0 0}, upload=540141]: Done uploading piece. err: <nil>
Dec 31 07:51:04.663531511 (616h22m4.534614556s) [noise=true , address=145.79.227.31:4798   , piece_id=UQJP3U5QFT5JHJ5DTP7QRZNVY2535FPVODKFOABGW3WP74NOPYDQ, seg_pos={0 0}, upload=540141]: Done uploading piece. err: <nil>
Dec 31 07:51:04.669874068 (616h22m4.540957103s) [noise=true , address=145.79.227.12:4765   , piece_id=WD57BZBPHBVRZ4BJLY6JIRLQEESFXMAHZEHNKPQBIESPCZTAUBXQ, seg_pos={0 0}, upload=540141]: Done uploading piece. err: <nil>
Dec 31 07:51:04.673966300 (616h22m4.545049335s) [noise=true , address=145.79.227.24:4582   , piece_id=QSMFJVHVRFVFDL4DT2UHLWMVF66HTQQ3X3HETXKVL4JERZFCP4QA, seg_pos={0 0}, upload=540141]: Done uploading piece. err: <nil>
Dec 31 07:51:04.728762481 (616h22m4.599845536s) [noise=true , address=145.79.227.50:4375   , piece_id=JYNH7FCK7S5J33MUIKIKBQL4LEXE2AUIGP27F7ILGDF6BCLYRVNA, seg_pos={0 0}, upload=540141]: Done uploading piece. err: <nil>
Dec 31 07:51:04.731376869 (616h22m4.602459904s) [noise=true , address=145.79.227.17:4970   , piece_id=IEZ3XCQ4ILSHKG3FMPUMRX3ED6K375LQGZDOIPO3RDXH3HPFEMGQ, seg_pos={0 0}, upload=540141]: Done uploading piece. err: <nil>
Dec 31 07:51:05.026421641 (616h22m4.897504676s) [noise=true , address=45.91.116.5:4297     , piece_id=2FPB4D776UYFTR4AL4NAWLKRC2NCAOECKIVAISSFV3MQ5EFIHFXA, seg_pos={0 0}, upload=540141]: Done uploading piece. err: <nil>
Dec 31 07:51:05.071647599 (616h22m4.942730644s) [noise=true , address=77.90.31.11:4554     , piece_id=UN3ILRE7M3GJOWR6MZ5TKFIMFKJ64VHJI54ALMRVBBG5GM7ZVCGQ, seg_pos={0 0}, upload=540141]: Done uploading piece. err: <nil>
Dec 31 07:51:05.071675381 (616h22m4.942758426s) [seg_pos={0 0}, upload=540141]: Segment reached optimal threshold of 13 pieces.
Dec 31 07:51:05.072120689 (616h22m4.943203754s) [noise=true , address=145.79.227.29:4943   , piece_id=7ZCXKYQET76RXQ4HVQJFLZBUSXCVVYYQO7B7VH5EFQX5RBFLD4QA, seg_pos={0 0}, upload=540141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz): protocol: expected piece hash; context canceled; EOF
Dec 31 07:51:05.072132842 (616h22m4.943215887s) [noise=true , address=145.79.227.29:4943   , piece_id=7ZCXKYQET76RXQ4HVQJFLZBUSXCVVYYQO7B7VH5EFQX5RBFLD4QA, seg_pos={0 0}, upload=540141]: Long tail cancellation due to optimal threshold
Dec 31 07:51:05.072587348 (616h22m4.943670383s) [noise=true , address=145.79.227.44:4193   , piece_id=AIE6JX7EQMVED7XDIWNQ65TZJTJVCY5CR7IWQCE37I3M6RYWVSKQ, seg_pos={0 0}, upload=540141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF
Dec 31 07:51:05.072596675 (616h22m4.943679721s) [noise=true , address=145.79.227.44:4193   , piece_id=AIE6JX7EQMVED7XDIWNQ65TZJTJVCY5CR7IWQCE37I3M6RYWVSKQ, seg_pos={0 0}, upload=540141]: Long tail cancellation due to optimal threshold
Dec 31 07:51:05.072766098 (616h22m4.943849153s) [noise=true , address=77.90.31.16:4735     , piece_id=GPHX2PRAK34RGV4QSKDA2EP377FG26VY4RQAEYILTXVNXKYQ4IRA, seg_pos={0 0}, upload=540141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF
Dec 31 07:51:05.072775756 (616h22m4.943858801s) [noise=true , address=77.90.31.16:4735     , piece_id=GPHX2PRAK34RGV4QSKDA2EP377FG26VY4RQAEYILTXVNXKYQ4IRA, seg_pos={0 0}, upload=540141]: Long tail cancellation due to optimal threshold
Dec 31 07:51:05.072798109 (616h22m4.943881154s) [seg_pos={0 0}, upload=540141]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 07:51:05.072935842 (616h22m4.944018887s) [upload=540141]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 07:51:05.072944167 (616h22m4.944027212s) [upload=540141]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 07:51:05.366342304 (616h22m5.237425349s) [upload=540141]: Done uploading segments...
Dec 31 07:51:05.366368162 (616h22m5.237451207s) [upload=540141]: Upload finished. err: <nil>
Duration: 5.828862434s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)
antiblip: 
1 out of 60 is failed (threshold=3) [.................................................X..........]
duration: 
2.257563526s
trace: 
https://jaeger.dev.storj.io/trace/59c3b821b75eba3a
FAIL (2025-12-31 07:51:45 UTC) 
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded
Duration: 2.257588124s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
4 out of 60 is failed (threshold=6) [...X................................X.........X............X]
duration: 
5.002128685s
trace: 
https://jaeger.dev.storj.io/trace/4dbc5f818d3c3b7
FAIL (2025-12-31 07:51:40 UTC) 
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.002151337s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [...............................................X............]
duration: 
6.512753934s
trace: 
https://jaeger.dev.storj.io/trace/5dd0609e3a2ac547
FAIL (2025-12-31 07:51:32 UTC) 
orbiter warning: context deadline exceeded
Duration: 6.512780924s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
18 out of 60 is failed (threshold=3) [..X....X.X.X.X.X......XXX.X....XXX..X...........X......X.X.X]
duration: 
30.003135802s
trace: 
https://jaeger.dev.storj.io/trace/b9a66a0708d887e
FAIL (2025-12-31 07:51:31 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 31 07:51:02.386794824 (616h22m2.257877859s) [upload=540141]: Starting upload
Dec 31 07:51:02.386883673 (616h22m2.257966748s) [upload=540141]: Done starting upload
Dec 31 07:51:02.387061752 (616h22m2.258144807s) [upload=540141]: Uploading segments...
Dec 31 07:51:02.387169487 (616h22m2.258252532s) [upload=540141]: Got next segment. Inline: false
Dec 31 07:51:02.387235102 (616h22m2.258318157s) [upload=540141]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 07:51:02.387242135 (616h22m2.25832518s) [upload=540141]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 07:51:02.631477047 (616h22m2.502560102s) [seg_pos={0 0}, upload=540141]: Begin upload segment...
Dec 31 07:51:02.631536671 (616h22m2.502619716s) [seg_pos={0 0}, upload=540141]: Done begin upload segment.
Dec 31 07:51:02.631572469 (616h22m2.502655524s) [noise=true , address=145.79.227.5:4223    , piece_id=U7A2ZGU3CUSID72JOCL23CVC3BSYOYHRW6MTDHGFMQB2V34MY7XA, seg_pos={0 0}, upload=540141]: Uploading piece...
Dec 31 07:51:02.633123201 (616h22m2.504206246s) [noise=true , address=145.79.227.44:4193   , piece_id=AIE6JX7EQMVED7XDIWNQ65TZJTJVCY5CR7IWQCE37I3M6RYWVSKQ, seg_pos={0 0}, upload=540141]: Uploading piece...
Dec 31 07:51:02.634110732 (616h22m2.505193777s) [noise=true , address=145.79.227.36:4517   , piece_id=4GCERGAIUOSKTH533BBCFGSZUZLTOIJJWAMTFEKUMBZRQARYIK6Q, seg_pos={0 0}, upload=540141]: Uploading piece...
Dec 31 07:51:02.634670077 (616h22m2.505753122s) [noise=true , address=145.79.227.29:4943   , piece_id=7ZCXKYQET76RXQ4HVQJFLZBUSXCVVYYQO7B7VH5EFQX5RBFLD4QA, seg_pos={0 0}, upload=540141]: Uploading piece...
Dec 31 07:51:02.635257716 (616h22m2.506340761s) [noise=true , address=77.90.31.16:4735     , piece_id=GPHX2PRAK34RGV4QSKDA2EP377FG26VY4RQAEYILTXVNXKYQ4IRA, seg_pos={0 0}, upload=540141]: Uploading piece...
Dec 31 07:51:02.636355436 (616h22m2.507438471s) [noise=true , address=145.79.227.24:4582   , piece_id=QSMFJVHVRFVFDL4DT2UHLWMVF66HTQQ3X3HETXKVL4JERZFCP4QA, seg_pos={0 0}, upload=540141]: Uploading piece...
Dec 31 07:51:02.637031744 (616h22m2.508114789s) [noise=true , address=145.79.227.13:4804   , piece_id=QUGILQI63G7CTO6FEKLQYPH3GTGYBFVIHFEGTQZ4FPNKVHIJ6O7A, seg_pos={0 0}, upload=540141]: Uploading piece...
Dec 31 07:51:02.637900207 (616h22m2.508983252s) [noise=true , address=77.90.31.11:4554     , piece_id=UN3ILRE7M3GJOWR6MZ5TKFIMFKJ64VHJI54ALMRVBBG5GM7ZVCGQ, seg_pos={0 0}, upload=540141]: Uploading piece...
Dec 31 07:51:02.639562833 (616h22m2.510645878s) [noise=true , address=145.79.227.50:4375   , piece_id=JYNH7FCK7S5J33MUIKIKBQL4LEXE2AUIGP27F7ILGDF6BCLYRVNA, seg_pos={0 0}, upload=540141]: Uploading piece...
Dec 31 07:51:02.641683061 (616h22m2.512766116s) [noise=true , address=45.91.116.5:4297     , piece_id=2FPB4D776UYFTR4AL4NAWLKRC2NCAOECKIVAISSFV3MQ5EFIHFXA, seg_pos={0 0}, upload=540141]: Uploading piece...
Dec 31 07:51:02.644611627 (616h22m2.515694672s) [noise=true , address=145.79.227.17:4970   , piece_id=IEZ3XCQ4ILSHKG3FMPUMRX3ED6K375LQGZDOIPO3RDXH3HPFEMGQ, seg_pos={0 0}, upload=540141]: Uploading piece...
Dec 31 07:51:02.646169363 (616h22m2.517252408s) [noise=true , address=145.79.227.45:4229   , piece_id=QNNEZTZLYLXB4CJHIQZ7XWFFNQTJHHGB4G6A6OWJPM5CULAT2MMA, seg_pos={0 0}, upload=540141]: Uploading piece...
Dec 31 07:51:02.647613794 (616h22m2.518696849s) [noise=true , address=145.79.227.41:4085   , piece_id=RVNDIJAJBAPI2HUIH7INGP7743IZKZ6FJE4S46E5XVWMQF4XKXBA, seg_pos={0 0}, upload=540141]: Uploading piece...
Dec 31 07:51:02.649442155 (616h22m2.52052521s) [noise=true , address=145.79.227.2:4113    , piece_id=4UE7ZTZQZ4W5HILOQWKBG5PZLOLKEPNTAVCQWB4HEIPHY3MBANBQ, seg_pos={0 0}, upload=540141]: Uploading piece...
Dec 31 07:51:02.650856880 (616h22m2.521939935s) [noise=true , address=145.79.227.12:4765   , piece_id=WD57BZBPHBVRZ4BJLY6JIRLQEESFXMAHZEHNKPQBIESPCZTAUBXQ, seg_pos={0 0}, upload=540141]: Uploading piece...
Dec 31 07:51:02.651427886 (616h22m2.522510921s) [noise=true , address=145.79.227.31:4798   , piece_id=UQJP3U5QFT5JHJ5DTP7QRZNVY2535FPVODKFOABGW3WP74NOPYDQ, seg_pos={0 0}, upload=540141]: Uploading piece...
Dec 31 07:51:04.306991452 (616h22m4.178074497s) [upload=540141]: Next returned nil segment
Dec 31 07:51:04.307028723 (616h22m4.178111768s) [upload=540141]: Waiting for error group managing segments...
Dec 31 07:51:04.637093559 (616h22m4.508176594s) [noise=true , address=145.79.227.5:4223    , piece_id=U7A2ZGU3CUSID72JOCL23CVC3BSYOYHRW6MTDHGFMQB2V34MY7XA, seg_pos={0 0}, upload=540141]: Done uploading piece. err: <nil>
Dec 31 07:51:04.639966499 (616h22m4.511049544s) [noise=true , address=145.79.227.2:4113    , piece_id=4UE7ZTZQZ4W5HILOQWKBG5PZLOLKEPNTAVCQWB4HEIPHY3MBANBQ, seg_pos={0 0}, upload=540141]: Done uploading piece. err: <nil>
Dec 31 07:51:04.641266174 (616h22m4.51234922s) [noise=true , address=145.79.227.13:4804   , piece_id=QUGILQI63G7CTO6FEKLQYPH3GTGYBFVIHFEGTQZ4FPNKVHIJ6O7A, seg_pos={0 0}, upload=540141]: Done uploading piece. err: <nil>
Dec 31 07:51:04.648310517 (616h22m4.519393562s) [noise=true , address=145.79.227.45:4229   , piece_id=QNNEZTZLYLXB4CJHIQZ7XWFFNQTJHHGB4G6A6OWJPM5CULAT2MMA, seg_pos={0 0}, upload=540141]: Done uploading piece. err: <nil>
Dec 31 07:51:04.656511744 (616h22m4.527594789s) [noise=true , address=145.79.227.41:4085   , piece_id=RVNDIJAJBAPI2HUIH7INGP7743IZKZ6FJE4S46E5XVWMQF4XKXBA, seg_pos={0 0}, upload=540141]: Done uploading piece. err: <nil>
Dec 31 07:51:04.663003926 (616h22m4.534086972s) [noise=true , address=145.79.227.36:4517   , piece_id=4GCERGAIUOSKTH533BBCFGSZUZLTOIJJWAMTFEKUMBZRQARYIK6Q, seg_pos={0 0}, upload=540141]: Done uploading piece. err: <nil>
Dec 31 07:51:04.663531511 (616h22m4.534614556s) [noise=true , address=145.79.227.31:4798   , piece_id=UQJP3U5QFT5JHJ5DTP7QRZNVY2535FPVODKFOABGW3WP74NOPYDQ, seg_pos={0 0}, upload=540141]: Done uploading piece. err: <nil>
Dec 31 07:51:04.669874068 (616h22m4.540957103s) [noise=true , address=145.79.227.12:4765   , piece_id=WD57BZBPHBVRZ4BJLY6JIRLQEESFXMAHZEHNKPQBIESPCZTAUBXQ, seg_pos={0 0}, upload=540141]: Done uploading piece. err: <nil>
Dec 31 07:51:04.673966300 (616h22m4.545049335s) [noise=true , address=145.79.227.24:4582   , piece_id=QSMFJVHVRFVFDL4DT2UHLWMVF66HTQQ3X3HETXKVL4JERZFCP4QA, seg_pos={0 0}, upload=540141]: Done uploading piece. err: <nil>
Dec 31 07:51:04.728762481 (616h22m4.599845536s) [noise=true , address=145.79.227.50:4375   , piece_id=JYNH7FCK7S5J33MUIKIKBQL4LEXE2AUIGP27F7ILGDF6BCLYRVNA, seg_pos={0 0}, upload=540141]: Done uploading piece. err: <nil>
Dec 31 07:51:04.731376869 (616h22m4.602459904s) [noise=true , address=145.79.227.17:4970   , piece_id=IEZ3XCQ4ILSHKG3FMPUMRX3ED6K375LQGZDOIPO3RDXH3HPFEMGQ, seg_pos={0 0}, upload=540141]: Done uploading piece. err: <nil>
Dec 31 07:51:05.026421641 (616h22m4.897504676s) [noise=true , address=45.91.116.5:4297     , piece_id=2FPB4D776UYFTR4AL4NAWLKRC2NCAOECKIVAISSFV3MQ5EFIHFXA, seg_pos={0 0}, upload=540141]: Done uploading piece. err: <nil>
Dec 31 07:51:05.071647599 (616h22m4.942730644s) [noise=true , address=77.90.31.11:4554     , piece_id=UN3ILRE7M3GJOWR6MZ5TKFIMFKJ64VHJI54ALMRVBBG5GM7ZVCGQ, seg_pos={0 0}, upload=540141]: Done uploading piece. err: <nil>
Dec 31 07:51:05.071675381 (616h22m4.942758426s) [seg_pos={0 0}, upload=540141]: Segment reached optimal threshold of 13 pieces.
Dec 31 07:51:05.072120689 (616h22m4.943203754s) [noise=true , address=145.79.227.29:4943   , piece_id=7ZCXKYQET76RXQ4HVQJFLZBUSXCVVYYQO7B7VH5EFQX5RBFLD4QA, seg_pos={0 0}, upload=540141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz): protocol: expected piece hash; context canceled; EOF
Dec 31 07:51:05.072132842 (616h22m4.943215887s) [noise=true , address=145.79.227.29:4943   , piece_id=7ZCXKYQET76RXQ4HVQJFLZBUSXCVVYYQO7B7VH5EFQX5RBFLD4QA, seg_pos={0 0}, upload=540141]: Long tail cancellation due to optimal threshold
Dec 31 07:51:05.072587348 (616h22m4.943670383s) [noise=true , address=145.79.227.44:4193   , piece_id=AIE6JX7EQMVED7XDIWNQ65TZJTJVCY5CR7IWQCE37I3M6RYWVSKQ, seg_pos={0 0}, upload=540141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF
Dec 31 07:51:05.072596675 (616h22m4.943679721s) [noise=true , address=145.79.227.44:4193   , piece_id=AIE6JX7EQMVED7XDIWNQ65TZJTJVCY5CR7IWQCE37I3M6RYWVSKQ, seg_pos={0 0}, upload=540141]: Long tail cancellation due to optimal threshold
Dec 31 07:51:05.072766098 (616h22m4.943849153s) [noise=true , address=77.90.31.16:4735     , piece_id=GPHX2PRAK34RGV4QSKDA2EP377FG26VY4RQAEYILTXVNXKYQ4IRA, seg_pos={0 0}, upload=540141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF
Dec 31 07:51:05.072775756 (616h22m4.943858801s) [noise=true , address=77.90.31.16:4735     , piece_id=GPHX2PRAK34RGV4QSKDA2EP377FG26VY4RQAEYILTXVNXKYQ4IRA, seg_pos={0 0}, upload=540141]: Long tail cancellation due to optimal threshold
Dec 31 07:51:05.072798109 (616h22m4.943881154s) [seg_pos={0 0}, upload=540141]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 07:51:05.072935842 (616h22m4.944018887s) [upload=540141]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 07:51:05.072944167 (616h22m4.944027212s) [upload=540141]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 07:51:05.366342304 (616h22m5.237425349s) [upload=540141]: Done uploading segments...
Dec 31 07:51:05.366368162 (616h22m5.237451207s) [upload=540141]: Upload finished. err: <nil>
Duration: 30.003175628s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
2 out of 60 is failed (threshold=3) [...........X............X...................................]
duration: 
24.113122358s
trace: 
https://jaeger.dev.storj.io/trace/6ce9bac61d803c5a
FAIL (2025-12-31 07:51:14 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
56     , piece_id=3PKC4JZPZVBQOBUHPQAG4HZ67DRTOY7GR35QVWLV3LN5JHXPMENA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.554293132 (615h55m26.425376177s) [noise=true , address=145.79.227.43:4186   , piece_id=ENVLK652WUVPKR3VPGXXMJ4IAEFHK67JOODS26ASTM4I2V7ABUWA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.554505907 (615h55m26.425588942s) [noise=true , address=77.90.31.14:4673     , piece_id=XYDCDFNKMWJK4HDELS4NLBVLUBR76YHT4RP3M5JO5QMCSI6MOGYA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.554648990 (615h55m26.425732025s) [noise=true , address=64.49.13.140:4505    , piece_id=EBEDU3F3QMDLNIDSRRXUHGSZ3VH5ITKRDDI2QS5XKBOGKOCN3LNA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.554994317 (615h55m26.426077352s) [noise=true , address=64.49.13.169:4707    , piece_id=RB7LFR5DM7PUHCJPB5ABWN63Q2XQOGNOG2W2FCKXFWU5MGVPRPVA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.555494150 (615h55m26.426577185s) [noise=true , address=64.49.13.137:4500    , piece_id=DD4JKFRHB2FKWAOTS5RZU5M3ASDGJTRX2XRB5ZROVOZNPN6BER2Q, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.555978812 (615h55m26.427061847s) [noise=true , address=45.91.116.10:4087    , piece_id=MJUIJFY6KEXEMMQKWA2NC3RE5PG55QCDZ7ECWRZG7CBROSA7LL3Q, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.556325282 (615h55m26.427408317s) [noise=true , address=145.79.227.29:4965   , piece_id=L6L2H7KMBTYKC24ZMACWRYK6TGOK5KHDO2YCFP5TJS66NXXP3KNQ, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.556728720 (615h55m26.427811765s) [noise=true , address=45.91.116.6:4977     , piece_id=G7JKD7INUS3D3RUIHSSVYCSOVTHSRG5HLVKERSCAD5NUWSFRZ2EA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.557164801 (615h55m26.428247836s) [noise=true , address=145.79.227.49:4370   , piece_id=TA4D4VCQXWYCB6FTZDRU6ZVZ4V6PZCFKQMA3EJYL5UEHHMI66JDQ, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.557530657 (615h55m26.428613692s) [noise=true , address=145.79.227.33:4896   , piece_id=TCDN3ER54RKZQ73BEI6DCGQW3IUNJXNNC3DUXI7NH7SFIPLHTF4A, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.557963953 (615h55m26.429046998s) [noise=true , address=77.90.31.12:4620     , piece_id=AQOFDODGJ7TMWA3ZQGWJW7XZT3FSKK3GR4R4RSIFIMHYP3HVJQDA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.558357422 (615h55m26.429440457s) [noise=true , address=145.79.227.40:4067   , piece_id=ITZ4SYUISESGVG72CBXHMAHZAD3XOOWYO5UUSFIT22R765N3W25A, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.558648285 (615h55m26.42973133s) [noise=true , address=145.79.227.2:4147    , piece_id=SHUFD6UKDFM54RGOZGJEUOUYCMKLAN2IMLWKJXZIZ5PAZVNUE4JQ, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.559228450 (615h55m26.430311485s) [noise=true , address=145.79.227.44:4228   , piece_id=YAQ7I3TOSLCYCHI2SJEEGCYGTEFWHX2CEYVZASRFL3BODTEONFTA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:27.974040599 (615h55m27.845123634s) [upload=539756]: Next returned nil segment
Dec 31 07:24:27.974074934 (615h55m27.845157969s) [upload=539756]: Waiting for error group managing segments...
Dec 31 07:24:28.334073958 (615h55m28.205156993s) [noise=true , address=64.49.13.140:4505    , piece_id=EBEDU3F3QMDLNIDSRRXUHGSZ3VH5ITKRDDI2QS5XKBOGKOCN3LNA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.427413642 (615h55m28.298496677s) [noise=true , address=145.79.227.2:4147    , piece_id=SHUFD6UKDFM54RGOZGJEUOUYCMKLAN2IMLWKJXZIZ5PAZVNUE4JQ, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.522823096 (615h55m28.393906131s) [noise=true , address=145.79.227.29:4965   , piece_id=L6L2H7KMBTYKC24ZMACWRYK6TGOK5KHDO2YCFP5TJS66NXXP3KNQ, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.585931513 (615h55m28.457014558s) [noise=true , address=77.90.31.13:4656     , piece_id=3PKC4JZPZVBQOBUHPQAG4HZ67DRTOY7GR35QVWLV3LN5JHXPMENA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.591579187 (615h55m28.462662262s) [noise=true , address=64.49.13.137:4500    , piece_id=DD4JKFRHB2FKWAOTS5RZU5M3ASDGJTRX2XRB5ZROVOZNPN6BER2Q, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.615175368 (615h55m28.486258413s) [noise=true , address=77.90.31.14:4673     , piece_id=XYDCDFNKMWJK4HDELS4NLBVLUBR76YHT4RP3M5JO5QMCSI6MOGYA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.699191016 (615h55m28.570274051s) [noise=true , address=45.91.116.6:4977     , piece_id=G7JKD7INUS3D3RUIHSSVYCSOVTHSRG5HLVKERSCAD5NUWSFRZ2EA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.713295994 (615h55m28.58437908s) [noise=true , address=45.91.116.10:4087    , piece_id=MJUIJFY6KEXEMMQKWA2NC3RE5PG55QCDZ7ECWRZG7CBROSA7LL3Q, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.794645326 (615h55m28.665728361s) [noise=true , address=77.90.31.12:4620     , piece_id=AQOFDODGJ7TMWA3ZQGWJW7XZT3FSKK3GR4R4RSIFIMHYP3HVJQDA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.341891306 (615h55m29.212974351s) [noise=true , address=145.79.227.33:4896   , piece_id=TCDN3ER54RKZQ73BEI6DCGQW3IUNJXNNC3DUXI7NH7SFIPLHTF4A, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.345487925 (615h55m29.21657096s) [noise=true , address=145.79.227.49:4370   , piece_id=TA4D4VCQXWYCB6FTZDRU6ZVZ4V6PZCFKQMA3EJYL5UEHHMI66JDQ, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.406150395 (615h55m29.27723343s) [noise=true , address=145.79.227.44:4228   , piece_id=YAQ7I3TOSLCYCHI2SJEEGCYGTEFWHX2CEYVZASRFL3BODTEONFTA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.414891660 (615h55m29.285974695s) [noise=true , address=193.149.164.143:4698 , piece_id=IYUHVRK3ZH3JADUJI627OUTHO7CRPNNKZVQTOCALMDYKVDGSHIJQ, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.431785668 (615h55m29.302868703s) [noise=true , address=145.79.227.40:4067   , piece_id=ITZ4SYUISESGVG72CBXHMAHZAD3XOOWYO5UUSFIT22R765N3W25A, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.431807049 (615h55m29.302890104s) [seg_pos={0 0}, upload=539756]: Segment reached optimal threshold of 14 pieces.
Dec 31 07:24:29.432061032 (615h55m29.303144087s) [noise=true , address=145.79.227.30:4784   , piece_id=M5NJQBX6JI5QI2PPA52B6A2CNLOXVWKYYYNX6CJDAA4WJNT4SHWA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pfXWex2WsuhmLb2smRpE46cHT9aipSTHW3e5oyELS1GhF1cFr): protocol: expected piece hash; context canceled; EOF
Dec 31 07:24:29.432072876 (615h55m29.303155931s) [noise=true , address=145.79.227.30:4784   , piece_id=M5NJQBX6JI5QI2PPA52B6A2CNLOXVWKYYYNX6CJDAA4WJNT4SHWA, seg_pos={0 0}, upload=539756]: Long tail cancellation due to optimal threshold
Dec 31 07:24:29.432090819 (615h55m29.303173884s) [noise=true , address=64.49.13.169:4709    , piece_id=CBAB3KCCPOW3P3LYM4T73CVDOD7FPJCPXYIXKDV7XTC2FKESTDRQ, seg_pos={0 0}, upload=539756]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:33942->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:33942->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33942->64.49.13.169:4709: use of closed network connection
Dec 31 07:24:29.432098063 (615h55m29.303181118s) [noise=true , address=64.49.13.169:4709    , piece_id=CBAB3KCCPOW3P3LYM4T73CVDOD7FPJCPXYIXKDV7XTC2FKESTDRQ, seg_pos={0 0}, upload=539756]: Long tail cancellation due to optimal threshold
Dec 31 07:24:29.432212572 (615h55m29.303295617s) [noise=true , address=64.49.13.169:4707    , piece_id=RB7LFR5DM7PUHCJPB5ABWN63Q2XQOGNOG2W2FCKXFWU5MGVPRPVA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:41264->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:41264->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41264->64.49.13.169:4707: use of closed network connection
Dec 31 07:24:29.432220747 (615h55m29.303303782s) [noise=true , address=64.49.13.169:4707    , piece_id=RB7LFR5DM7PUHCJPB5ABWN63Q2XQOGNOG2W2FCKXFWU5MGVPRPVA, seg_pos={0 0}, upload=539756]: Long tail cancellation due to optimal threshold
Dec 31 07:24:29.432465934 (615h55m29.303548979s) [noise=true , address=145.79.227.43:4186   , piece_id=ENVLK652WUVPKR3VPGXXMJ4IAEFHK67JOODS26ASTM4I2V7ABUWA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Gu3vXy2JUkKDaF4MkMCgVmYspW4Ez8EAfv6e4GeCkrwYTe5T): protocol: expected piece hash; context canceled; EOF
Dec 31 07:24:29.432474971 (615h55m29.303558006s) [noise=true , address=145.79.227.43:4186   , piece_id=ENVLK652WUVPKR3VPGXXMJ4IAEFHK67JOODS26ASTM4I2V7ABUWA, seg_pos={0 0}, upload=539756]: Long tail cancellation due to optimal threshold
Dec 31 07:24:29.432580261 (615h55m29.303663296s) [seg_pos={0 0}, upload=539756]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 07:24:29.432664502 (615h55m29.303747547s) [upload=539756]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 07:24:29.432671305 (615h55m29.30375435s) [upload=539756]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 07:24:29.717299118 (615h55m29.588382153s) [upload=539756]: Done uploading segments...
Dec 31 07:24:29.717320589 (615h55m29.588403624s) [upload=539756]: Upload finished. err: <nil>
Duration: 24.113148528s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [................................................X...........]
duration: 
6.236366316s
trace: 
https://jaeger.dev.storj.io/trace/5dd0609e3a2ac547
FAIL (2025-12-31 07:51:13 UTC) 
orbiter warning: context deadline exceeded
Duration: 6.236392166s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
4 out of 60 is failed (threshold=6) [X...X................................X.........X............]
duration: 
5.001009161s
trace: 
https://jaeger.dev.storj.io/trace/6fc9bc24e7467571
FAIL (2025-12-31 07:51:12 UTC) 
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001169268s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)
antiblip: 
1 out of 60 is failed (threshold=3) [..................................................X.........]
duration: 
6.304743451s
trace: 
https://jaeger.dev.storj.io/trace/59c3b821b75eba3a
FAIL (2025-12-31 07:51:01 UTC) 
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded
Duration: 6.30477986s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
17 out of 60 is failed (threshold=3) [...X....X.X.X.X.X......XXX.X....XXX..X...........X......X.X.]
duration: 
6.018284636s
trace: 
https://jaeger.dev.storj.io/trace/1a71eb5bd93981af
FAIL (2025-12-31 07:50:49 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
07:50:13.835961273 (616h21m13.707044338s) [upload=540130]: Starting upload
Dec 31 07:50:13.836062425 (616h21m13.70714549s) [upload=540130]: Done starting upload
Dec 31 07:50:13.836266012 (616h21m13.707349048s) [upload=540130]: Uploading segments...
Dec 31 07:50:13.836382685 (616h21m13.70746573s) [upload=540130]: Got next segment. Inline: false
Dec 31 07:50:13.836441146 (616h21m13.707524191s) [upload=540130]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 07:50:13.836447468 (616h21m13.707530503s) [upload=540130]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 07:50:14.080654579 (616h21m13.951737624s) [seg_pos={0 0}, upload=540130]: Begin upload segment...
Dec 31 07:50:14.080721567 (616h21m13.951804602s) [seg_pos={0 0}, upload=540130]: Done begin upload segment.
Dec 31 07:50:14.080734872 (616h21m13.951817917s) [noise=true , address=145.79.227.42:4123   , piece_id=X52CDJL36PBKBBERX2RCRERUHFTLXUJQWID2IRQRDVYBRQMMAVNA, seg_pos={0 0}, upload=540130]: Uploading piece...
Dec 31 07:50:14.080744570 (616h21m13.951827605s) [noise=true , address=145.79.227.45:4231   , piece_id=PP5BISNE5PGIFYI22EFU2R66LAQYOROMWX46BMPBXWRSLVHJV2YQ, seg_pos={0 0}, upload=540130]: Uploading piece...
Dec 31 07:50:14.081581754 (616h21m13.952664809s) [noise=true , address=145.79.227.39:4014   , piece_id=QTQ62XAYS2FYOHMUKZOU2GV3UWHPZILNF2K2V3WI7VU7ZFETTYFA, seg_pos={0 0}, upload=540130]: Uploading piece...
Dec 31 07:50:14.081656456 (616h21m13.952739501s) [noise=true , address=145.79.227.31:4798   , piece_id=WJA2TTPEMLKWWUX6S62GIKMG6OUFZO5OJGINBBEIFU5OOINOT5ZQ, seg_pos={0 0}, upload=540130]: Uploading piece...
Dec 31 07:50:14.082048724 (616h21m13.953131769s) [noise=true , address=45.91.116.9:4263     , piece_id=JYKGWRF3IA7FWWJY7QZFNOM6WOWOTCCKKZYLGEAQKQBTYWBLJFBA, seg_pos={0 0}, upload=540130]: Uploading piece...
Dec 31 07:50:14.082281146 (616h21m13.953364191s) [noise=true , address=145.79.227.16:4911   , piece_id=ICFDSO3IMTWB3ALB6NCE3ZZADCPPDNAW3BPWM7FP256S7FHF6VNQ, seg_pos={0 0}, upload=540130]: Uploading piece...
Dec 31 07:50:14.082453033 (616h21m13.953536088s) [noise=true , address=45.91.116.6:4975     , piece_id=NRO4DTGVS7V7HAY3DFYMPL52DSRH3LS6E2EAXA3DSJVLHD7JBRZQ, seg_pos={0 0}, upload=540130]: Uploading piece...
Dec 31 07:50:14.082918490 (616h21m13.954001535s) [noise=true , address=145.79.227.22:4511   , piece_id=BNBMLNARFWOHBJ625BQ6SHSALCJ6DK3GDWHQZUAZZMXCFB3373EA, seg_pos={0 0}, upload=540130]: Uploading piece...
Dec 31 07:50:14.082978373 (616h21m13.954061458s) [noise=true , address=145.79.227.13:4804   , piece_id=H7MIO4L43XKMOS6Q4YF5DYUN2ML7WTZLDC5ZARDLRC2J6RL66F2A, seg_pos={0 0}, upload=540130]: Uploading piece...
Dec 31 07:50:14.083499846 (616h21m13.954582881s) [noise=true , address=77.90.31.17:4773     , piece_id=R5WYXJQU6TROJPZRTS4FTPRXJ4LSDUVW2XYZUMCOWRPBCY54RFEA, seg_pos={0 0}, upload=540130]: Uploading piece...
Dec 31 07:50:14.083540675 (616h21m13.95462372s) [noise=true , address=45.91.116.12:4227    , piece_id=BHDYBZYDEIGGM43X3TFX5K7EWL56QRSS5NQVRC5M4AEE5XD6OOJA, seg_pos={0 0}, upload=540130]: Uploading piece...
Dec 31 07:50:14.084117923 (616h21m13.955200998s) [noise=true , address=145.79.227.12:4767   , piece_id=QPIZLPAB72S7R4TBARYJLZC5PKBI5A76TX47V55KIUD7V4JMJPTQ, seg_pos={0 0}, upload=540130]: Uploading piece...
Dec 31 07:50:14.084272729 (616h21m13.955355784s) [noise=true , address=145.79.227.21:4474   , piece_id=GXR3RV363AOQ244TMOOJWC34S2FVXQAMZQAOGUGMC3UDEBFB7LGA, seg_pos={0 0}, upload=540130]: Uploading piece...
Dec 31 07:50:14.084551239 (616h21m13.955634274s) [noise=true , address=145.79.227.17:4970   , piece_id=5UWCS6FGHJC6BVIBFY4KDUBDFQJW7XNAIFGNHBQBKYCWKM3BBIAA, seg_pos={0 0}, upload=540130]: Uploading piece...
Dec 31 07:50:14.084933497 (616h21m13.956016532s) [noise=true , address=145.79.227.11:4730   , piece_id=3PYOSLOBQMF4MYAXEXW7UGX7GJSAMV3EKSMFRYZRWUFWGQ6LTWJA, seg_pos={0 0}, upload=540130]: Uploading piece...
Dec 31 07:50:14.085394875 (616h21m13.95647792s) [noise=true , address=145.79.227.5:4223    , piece_id=S5QFIS3G3O6LLPPUYOMJJ3XQ65UGMCG4YGXTSKTK2VBVY3PYIZLA, seg_pos={0 0}, upload=540130]: Uploading piece...
Dec 31 07:50:15.773911727 (616h21m15.644994762s) [upload=540130]: Next returned nil segment
Dec 31 07:50:15.773924272 (616h21m15.645007317s) [upload=540130]: Waiting for error group managing segments...
Dec 31 07:50:16.060935968 (616h21m15.932019003s) [noise=true , address=145.79.227.16:4911   , piece_id=ICFDSO3IMTWB3ALB6NCE3ZZADCPPDNAW3BPWM7FP256S7FHF6VNQ, seg_pos={0 0}, upload=540130]: Done uploading piece. err: <nil>
Dec 31 07:50:16.066680917 (616h21m15.937763952s) [noise=true , address=145.79.227.22:4511   , piece_id=BNBMLNARFWOHBJ625BQ6SHSALCJ6DK3GDWHQZUAZZMXCFB3373EA, seg_pos={0 0}, upload=540130]: Done uploading piece. err: <nil>
Dec 31 07:50:16.067285969 (616h21m15.938369004s) [noise=true , address=145.79.227.13:4804   , piece_id=H7MIO4L43XKMOS6Q4YF5DYUN2ML7WTZLDC5ZARDLRC2J6RL66F2A, seg_pos={0 0}, upload=540130]: Done uploading piece. err: <nil>
Dec 31 07:50:16.074304904 (616h21m15.945387939s) [noise=true , address=145.79.227.31:4798   , piece_id=WJA2TTPEMLKWWUX6S62GIKMG6OUFZO5OJGINBBEIFU5OOINOT5ZQ, seg_pos={0 0}, upload=540130]: Done uploading piece. err: <nil>
Dec 31 07:50:16.091671072 (616h21m15.962754107s) [noise=true , address=145.79.227.42:4123   , piece_id=X52CDJL36PBKBBERX2RCRERUHFTLXUJQWID2IRQRDVYBRQMMAVNA, seg_pos={0 0}, upload=540130]: Done uploading piece. err: <nil>
Dec 31 07:50:16.091940876 (616h21m15.963023911s) [noise=true , address=145.79.227.5:4223    , piece_id=S5QFIS3G3O6LLPPUYOMJJ3XQ65UGMCG4YGXTSKTK2VBVY3PYIZLA, seg_pos={0 0}, upload=540130]: Done uploading piece. err: <nil>
Dec 31 07:50:16.092107543 (616h21m15.963190588s) [noise=true , address=145.79.227.45:4231   , piece_id=PP5BISNE5PGIFYI22EFU2R66LAQYOROMWX46BMPBXWRSLVHJV2YQ, seg_pos={0 0}, upload=540130]: Done uploading piece. err: <nil>
Dec 31 07:50:16.094320908 (616h21m15.965403953s) [noise=true , address=145.79.227.12:4767   , piece_id=QPIZLPAB72S7R4TBARYJLZC5PKBI5A76TX47V55KIUD7V4JMJPTQ, seg_pos={0 0}, upload=540130]: Done uploading piece. err: <nil>
Dec 31 07:50:16.101864181 (616h21m15.972947226s) [noise=true , address=145.79.227.17:4970   , piece_id=5UWCS6FGHJC6BVIBFY4KDUBDFQJW7XNAIFGNHBQBKYCWKM3BBIAA, seg_pos={0 0}, upload=540130]: Done uploading piece. err: <nil>
Dec 31 07:50:16.160938448 (616h21m16.032021493s) [noise=true , address=145.79.227.11:4730   , piece_id=3PYOSLOBQMF4MYAXEXW7UGX7GJSAMV3EKSMFRYZRWUFWGQ6LTWJA, seg_pos={0 0}, upload=540130]: Done uploading piece. err: <nil>
Dec 31 07:50:16.164001220 (616h21m16.035084255s) [noise=true , address=145.79.227.21:4474   , piece_id=GXR3RV363AOQ244TMOOJWC34S2FVXQAMZQAOGUGMC3UDEBFB7LGA, seg_pos={0 0}, upload=540130]: Done uploading piece. err: <nil>
Dec 31 07:50:16.312808293 (616h21m16.183891338s) [noise=true , address=77.90.31.17:4773     , piece_id=R5WYXJQU6TROJPZRTS4FTPRXJ4LSDUVW2XYZUMCOWRPBCY54RFEA, seg_pos={0 0}, upload=540130]: Done uploading piece. err: <nil>
Dec 31 07:50:16.385309145 (616h21m16.25639218s) [noise=true , address=45.91.116.9:4263     , piece_id=JYKGWRF3IA7FWWJY7QZFNOM6WOWOTCCKKZYLGEAQKQBTYWBLJFBA, seg_pos={0 0}, upload=540130]: Done uploading piece. err: <nil>
Dec 31 07:50:16.385335956 (616h21m16.256419001s) [seg_pos={0 0}, upload=540130]: Segment reached optimal threshold of 13 pieces.
Dec 31 07:50:16.385549212 (616h21m16.256632257s) [noise=true , address=145.79.227.39:4014   , piece_id=QTQ62XAYS2FYOHMUKZOU2GV3UWHPZILNF2K2V3WI7VU7ZFETTYFA, seg_pos={0 0}, upload=540130]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: expected piece hash; context canceled; EOF
Dec 31 07:50:16.385558891 (616h21m16.256641936s) [noise=true , address=145.79.227.39:4014   , piece_id=QTQ62XAYS2FYOHMUKZOU2GV3UWHPZILNF2K2V3WI7VU7ZFETTYFA, seg_pos={0 0}, upload=540130]: Long tail cancellation due to optimal threshold
Dec 31 07:50:16.385749364 (616h21m16.256832399s) [noise=true , address=45.91.116.6:4975     , piece_id=NRO4DTGVS7V7HAY3DFYMPL52DSRH3LS6E2EAXA3DSJVLHD7JBRZQ, seg_pos={0 0}, upload=540130]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF
Dec 31 07:50:16.385764483 (616h21m16.256847518s) [noise=true , address=45.91.116.6:4975     , piece_id=NRO4DTGVS7V7HAY3DFYMPL52DSRH3LS6E2EAXA3DSJVLHD7JBRZQ, seg_pos={0 0}, upload=540130]: Long tail cancellation due to optimal threshold
Dec 31 07:50:16.385962630 (616h21m16.257045685s) [noise=true , address=45.91.116.12:4227    , piece_id=BHDYBZYDEIGGM43X3TFX5K7EWL56QRSS5NQVRC5M4AEE5XD6OOJA, seg_pos={0 0}, upload=540130]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: write tcp 188.166.225.170:33022->45.91.116.12:4227: use of closed network connection; write tcp 188.166.225.170:33022->45.91.116.12:4227: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33022->45.91.116.12:4227: use of closed network connection
Dec 31 07:50:16.385973842 (616h21m16.257056877s) [noise=true , address=45.91.116.12:4227    , piece_id=BHDYBZYDEIGGM43X3TFX5K7EWL56QRSS5NQVRC5M4AEE5XD6OOJA, seg_pos={0 0}, upload=540130]: Long tail cancellation due to optimal threshold
Dec 31 07:50:16.385997626 (616h21m16.257080661s) [seg_pos={0 0}, upload=540130]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 07:50:16.386084672 (616h21m16.257167707s) [upload=540130]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 07:50:16.386090984 (616h21m16.257174039s) [upload=540130]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 07:50:16.669784681 (616h21m16.540867737s) [upload=540130]: Done uploading segments...
Dec 31 07:50:16.669914799 (616h21m16.540997845s) [upload=540130]: Upload finished. err: <nil>
Duration: 6.018301889s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
4 out of 60 is failed (threshold=6) [.X...X................................X.........X...........]
duration: 
3.41782505s
trace: 
https://jaeger.dev.storj.io/trace/6fc9bc24e7467571
FAIL (2025-12-31 07:50:46 UTC) 
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.417883271s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
2 out of 60 is failed (threshold=3) [............X............X..................................]
duration: 
30.366457671s
trace: 
https://jaeger.dev.storj.io/trace/6ce9bac61d803c5a
FAIL (2025-12-31 07:50:44 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
56     , piece_id=3PKC4JZPZVBQOBUHPQAG4HZ67DRTOY7GR35QVWLV3LN5JHXPMENA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.554293132 (615h55m26.425376177s) [noise=true , address=145.79.227.43:4186   , piece_id=ENVLK652WUVPKR3VPGXXMJ4IAEFHK67JOODS26ASTM4I2V7ABUWA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.554505907 (615h55m26.425588942s) [noise=true , address=77.90.31.14:4673     , piece_id=XYDCDFNKMWJK4HDELS4NLBVLUBR76YHT4RP3M5JO5QMCSI6MOGYA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.554648990 (615h55m26.425732025s) [noise=true , address=64.49.13.140:4505    , piece_id=EBEDU3F3QMDLNIDSRRXUHGSZ3VH5ITKRDDI2QS5XKBOGKOCN3LNA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.554994317 (615h55m26.426077352s) [noise=true , address=64.49.13.169:4707    , piece_id=RB7LFR5DM7PUHCJPB5ABWN63Q2XQOGNOG2W2FCKXFWU5MGVPRPVA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.555494150 (615h55m26.426577185s) [noise=true , address=64.49.13.137:4500    , piece_id=DD4JKFRHB2FKWAOTS5RZU5M3ASDGJTRX2XRB5ZROVOZNPN6BER2Q, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.555978812 (615h55m26.427061847s) [noise=true , address=45.91.116.10:4087    , piece_id=MJUIJFY6KEXEMMQKWA2NC3RE5PG55QCDZ7ECWRZG7CBROSA7LL3Q, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.556325282 (615h55m26.427408317s) [noise=true , address=145.79.227.29:4965   , piece_id=L6L2H7KMBTYKC24ZMACWRYK6TGOK5KHDO2YCFP5TJS66NXXP3KNQ, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.556728720 (615h55m26.427811765s) [noise=true , address=45.91.116.6:4977     , piece_id=G7JKD7INUS3D3RUIHSSVYCSOVTHSRG5HLVKERSCAD5NUWSFRZ2EA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.557164801 (615h55m26.428247836s) [noise=true , address=145.79.227.49:4370   , piece_id=TA4D4VCQXWYCB6FTZDRU6ZVZ4V6PZCFKQMA3EJYL5UEHHMI66JDQ, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.557530657 (615h55m26.428613692s) [noise=true , address=145.79.227.33:4896   , piece_id=TCDN3ER54RKZQ73BEI6DCGQW3IUNJXNNC3DUXI7NH7SFIPLHTF4A, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.557963953 (615h55m26.429046998s) [noise=true , address=77.90.31.12:4620     , piece_id=AQOFDODGJ7TMWA3ZQGWJW7XZT3FSKK3GR4R4RSIFIMHYP3HVJQDA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.558357422 (615h55m26.429440457s) [noise=true , address=145.79.227.40:4067   , piece_id=ITZ4SYUISESGVG72CBXHMAHZAD3XOOWYO5UUSFIT22R765N3W25A, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.558648285 (615h55m26.42973133s) [noise=true , address=145.79.227.2:4147    , piece_id=SHUFD6UKDFM54RGOZGJEUOUYCMKLAN2IMLWKJXZIZ5PAZVNUE4JQ, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.559228450 (615h55m26.430311485s) [noise=true , address=145.79.227.44:4228   , piece_id=YAQ7I3TOSLCYCHI2SJEEGCYGTEFWHX2CEYVZASRFL3BODTEONFTA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:27.974040599 (615h55m27.845123634s) [upload=539756]: Next returned nil segment
Dec 31 07:24:27.974074934 (615h55m27.845157969s) [upload=539756]: Waiting for error group managing segments...
Dec 31 07:24:28.334073958 (615h55m28.205156993s) [noise=true , address=64.49.13.140:4505    , piece_id=EBEDU3F3QMDLNIDSRRXUHGSZ3VH5ITKRDDI2QS5XKBOGKOCN3LNA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.427413642 (615h55m28.298496677s) [noise=true , address=145.79.227.2:4147    , piece_id=SHUFD6UKDFM54RGOZGJEUOUYCMKLAN2IMLWKJXZIZ5PAZVNUE4JQ, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.522823096 (615h55m28.393906131s) [noise=true , address=145.79.227.29:4965   , piece_id=L6L2H7KMBTYKC24ZMACWRYK6TGOK5KHDO2YCFP5TJS66NXXP3KNQ, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.585931513 (615h55m28.457014558s) [noise=true , address=77.90.31.13:4656     , piece_id=3PKC4JZPZVBQOBUHPQAG4HZ67DRTOY7GR35QVWLV3LN5JHXPMENA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.591579187 (615h55m28.462662262s) [noise=true , address=64.49.13.137:4500    , piece_id=DD4JKFRHB2FKWAOTS5RZU5M3ASDGJTRX2XRB5ZROVOZNPN6BER2Q, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.615175368 (615h55m28.486258413s) [noise=true , address=77.90.31.14:4673     , piece_id=XYDCDFNKMWJK4HDELS4NLBVLUBR76YHT4RP3M5JO5QMCSI6MOGYA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.699191016 (615h55m28.570274051s) [noise=true , address=45.91.116.6:4977     , piece_id=G7JKD7INUS3D3RUIHSSVYCSOVTHSRG5HLVKERSCAD5NUWSFRZ2EA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.713295994 (615h55m28.58437908s) [noise=true , address=45.91.116.10:4087    , piece_id=MJUIJFY6KEXEMMQKWA2NC3RE5PG55QCDZ7ECWRZG7CBROSA7LL3Q, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.794645326 (615h55m28.665728361s) [noise=true , address=77.90.31.12:4620     , piece_id=AQOFDODGJ7TMWA3ZQGWJW7XZT3FSKK3GR4R4RSIFIMHYP3HVJQDA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.341891306 (615h55m29.212974351s) [noise=true , address=145.79.227.33:4896   , piece_id=TCDN3ER54RKZQ73BEI6DCGQW3IUNJXNNC3DUXI7NH7SFIPLHTF4A, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.345487925 (615h55m29.21657096s) [noise=true , address=145.79.227.49:4370   , piece_id=TA4D4VCQXWYCB6FTZDRU6ZVZ4V6PZCFKQMA3EJYL5UEHHMI66JDQ, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.406150395 (615h55m29.27723343s) [noise=true , address=145.79.227.44:4228   , piece_id=YAQ7I3TOSLCYCHI2SJEEGCYGTEFWHX2CEYVZASRFL3BODTEONFTA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.414891660 (615h55m29.285974695s) [noise=true , address=193.149.164.143:4698 , piece_id=IYUHVRK3ZH3JADUJI627OUTHO7CRPNNKZVQTOCALMDYKVDGSHIJQ, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.431785668 (615h55m29.302868703s) [noise=true , address=145.79.227.40:4067   , piece_id=ITZ4SYUISESGVG72CBXHMAHZAD3XOOWYO5UUSFIT22R765N3W25A, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.431807049 (615h55m29.302890104s) [seg_pos={0 0}, upload=539756]: Segment reached optimal threshold of 14 pieces.
Dec 31 07:24:29.432061032 (615h55m29.303144087s) [noise=true , address=145.79.227.30:4784   , piece_id=M5NJQBX6JI5QI2PPA52B6A2CNLOXVWKYYYNX6CJDAA4WJNT4SHWA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pfXWex2WsuhmLb2smRpE46cHT9aipSTHW3e5oyELS1GhF1cFr): protocol: expected piece hash; context canceled; EOF
Dec 31 07:24:29.432072876 (615h55m29.303155931s) [noise=true , address=145.79.227.30:4784   , piece_id=M5NJQBX6JI5QI2PPA52B6A2CNLOXVWKYYYNX6CJDAA4WJNT4SHWA, seg_pos={0 0}, upload=539756]: Long tail cancellation due to optimal threshold
Dec 31 07:24:29.432090819 (615h55m29.303173884s) [noise=true , address=64.49.13.169:4709    , piece_id=CBAB3KCCPOW3P3LYM4T73CVDOD7FPJCPXYIXKDV7XTC2FKESTDRQ, seg_pos={0 0}, upload=539756]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:33942->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:33942->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33942->64.49.13.169:4709: use of closed network connection
Dec 31 07:24:29.432098063 (615h55m29.303181118s) [noise=true , address=64.49.13.169:4709    , piece_id=CBAB3KCCPOW3P3LYM4T73CVDOD7FPJCPXYIXKDV7XTC2FKESTDRQ, seg_pos={0 0}, upload=539756]: Long tail cancellation due to optimal threshold
Dec 31 07:24:29.432212572 (615h55m29.303295617s) [noise=true , address=64.49.13.169:4707    , piece_id=RB7LFR5DM7PUHCJPB5ABWN63Q2XQOGNOG2W2FCKXFWU5MGVPRPVA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:41264->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:41264->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41264->64.49.13.169:4707: use of closed network connection
Dec 31 07:24:29.432220747 (615h55m29.303303782s) [noise=true , address=64.49.13.169:4707    , piece_id=RB7LFR5DM7PUHCJPB5ABWN63Q2XQOGNOG2W2FCKXFWU5MGVPRPVA, seg_pos={0 0}, upload=539756]: Long tail cancellation due to optimal threshold
Dec 31 07:24:29.432465934 (615h55m29.303548979s) [noise=true , address=145.79.227.43:4186   , piece_id=ENVLK652WUVPKR3VPGXXMJ4IAEFHK67JOODS26ASTM4I2V7ABUWA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Gu3vXy2JUkKDaF4MkMCgVmYspW4Ez8EAfv6e4GeCkrwYTe5T): protocol: expected piece hash; context canceled; EOF
Dec 31 07:24:29.432474971 (615h55m29.303558006s) [noise=true , address=145.79.227.43:4186   , piece_id=ENVLK652WUVPKR3VPGXXMJ4IAEFHK67JOODS26ASTM4I2V7ABUWA, seg_pos={0 0}, upload=539756]: Long tail cancellation due to optimal threshold
Dec 31 07:24:29.432580261 (615h55m29.303663296s) [seg_pos={0 0}, upload=539756]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 07:24:29.432664502 (615h55m29.303747547s) [upload=539756]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 07:24:29.432671305 (615h55m29.30375435s) [upload=539756]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 07:24:29.717299118 (615h55m29.588382153s) [upload=539756]: Done uploading segments...
Dec 31 07:24:29.717320589 (615h55m29.588403624s) [upload=539756]: Upload finished. err: <nil>
Duration: 30.366475494s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
17 out of 60 is failed (threshold=3) [....X....X.X.X.X.X......XXX.X....XXX..X...........X......X.X]
duration: 
30.00342522s
trace: 
https://jaeger.dev.storj.io/trace/1a71eb5bd93981af
FAIL (2025-12-31 07:50:43 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
07:50:13.835961273 (616h21m13.707044338s) [upload=540130]: Starting upload
Dec 31 07:50:13.836062425 (616h21m13.70714549s) [upload=540130]: Done starting upload
Dec 31 07:50:13.836266012 (616h21m13.707349048s) [upload=540130]: Uploading segments...
Dec 31 07:50:13.836382685 (616h21m13.70746573s) [upload=540130]: Got next segment. Inline: false
Dec 31 07:50:13.836441146 (616h21m13.707524191s) [upload=540130]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 07:50:13.836447468 (616h21m13.707530503s) [upload=540130]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 07:50:14.080654579 (616h21m13.951737624s) [seg_pos={0 0}, upload=540130]: Begin upload segment...
Dec 31 07:50:14.080721567 (616h21m13.951804602s) [seg_pos={0 0}, upload=540130]: Done begin upload segment.
Dec 31 07:50:14.080734872 (616h21m13.951817917s) [noise=true , address=145.79.227.42:4123   , piece_id=X52CDJL36PBKBBERX2RCRERUHFTLXUJQWID2IRQRDVYBRQMMAVNA, seg_pos={0 0}, upload=540130]: Uploading piece...
Dec 31 07:50:14.080744570 (616h21m13.951827605s) [noise=true , address=145.79.227.45:4231   , piece_id=PP5BISNE5PGIFYI22EFU2R66LAQYOROMWX46BMPBXWRSLVHJV2YQ, seg_pos={0 0}, upload=540130]: Uploading piece...
Dec 31 07:50:14.081581754 (616h21m13.952664809s) [noise=true , address=145.79.227.39:4014   , piece_id=QTQ62XAYS2FYOHMUKZOU2GV3UWHPZILNF2K2V3WI7VU7ZFETTYFA, seg_pos={0 0}, upload=540130]: Uploading piece...
Dec 31 07:50:14.081656456 (616h21m13.952739501s) [noise=true , address=145.79.227.31:4798   , piece_id=WJA2TTPEMLKWWUX6S62GIKMG6OUFZO5OJGINBBEIFU5OOINOT5ZQ, seg_pos={0 0}, upload=540130]: Uploading piece...
Dec 31 07:50:14.082048724 (616h21m13.953131769s) [noise=true , address=45.91.116.9:4263     , piece_id=JYKGWRF3IA7FWWJY7QZFNOM6WOWOTCCKKZYLGEAQKQBTYWBLJFBA, seg_pos={0 0}, upload=540130]: Uploading piece...
Dec 31 07:50:14.082281146 (616h21m13.953364191s) [noise=true , address=145.79.227.16:4911   , piece_id=ICFDSO3IMTWB3ALB6NCE3ZZADCPPDNAW3BPWM7FP256S7FHF6VNQ, seg_pos={0 0}, upload=540130]: Uploading piece...
Dec 31 07:50:14.082453033 (616h21m13.953536088s) [noise=true , address=45.91.116.6:4975     , piece_id=NRO4DTGVS7V7HAY3DFYMPL52DSRH3LS6E2EAXA3DSJVLHD7JBRZQ, seg_pos={0 0}, upload=540130]: Uploading piece...
Dec 31 07:50:14.082918490 (616h21m13.954001535s) [noise=true , address=145.79.227.22:4511   , piece_id=BNBMLNARFWOHBJ625BQ6SHSALCJ6DK3GDWHQZUAZZMXCFB3373EA, seg_pos={0 0}, upload=540130]: Uploading piece...
Dec 31 07:50:14.082978373 (616h21m13.954061458s) [noise=true , address=145.79.227.13:4804   , piece_id=H7MIO4L43XKMOS6Q4YF5DYUN2ML7WTZLDC5ZARDLRC2J6RL66F2A, seg_pos={0 0}, upload=540130]: Uploading piece...
Dec 31 07:50:14.083499846 (616h21m13.954582881s) [noise=true , address=77.90.31.17:4773     , piece_id=R5WYXJQU6TROJPZRTS4FTPRXJ4LSDUVW2XYZUMCOWRPBCY54RFEA, seg_pos={0 0}, upload=540130]: Uploading piece...
Dec 31 07:50:14.083540675 (616h21m13.95462372s) [noise=true , address=45.91.116.12:4227    , piece_id=BHDYBZYDEIGGM43X3TFX5K7EWL56QRSS5NQVRC5M4AEE5XD6OOJA, seg_pos={0 0}, upload=540130]: Uploading piece...
Dec 31 07:50:14.084117923 (616h21m13.955200998s) [noise=true , address=145.79.227.12:4767   , piece_id=QPIZLPAB72S7R4TBARYJLZC5PKBI5A76TX47V55KIUD7V4JMJPTQ, seg_pos={0 0}, upload=540130]: Uploading piece...
Dec 31 07:50:14.084272729 (616h21m13.955355784s) [noise=true , address=145.79.227.21:4474   , piece_id=GXR3RV363AOQ244TMOOJWC34S2FVXQAMZQAOGUGMC3UDEBFB7LGA, seg_pos={0 0}, upload=540130]: Uploading piece...
Dec 31 07:50:14.084551239 (616h21m13.955634274s) [noise=true , address=145.79.227.17:4970   , piece_id=5UWCS6FGHJC6BVIBFY4KDUBDFQJW7XNAIFGNHBQBKYCWKM3BBIAA, seg_pos={0 0}, upload=540130]: Uploading piece...
Dec 31 07:50:14.084933497 (616h21m13.956016532s) [noise=true , address=145.79.227.11:4730   , piece_id=3PYOSLOBQMF4MYAXEXW7UGX7GJSAMV3EKSMFRYZRWUFWGQ6LTWJA, seg_pos={0 0}, upload=540130]: Uploading piece...
Dec 31 07:50:14.085394875 (616h21m13.95647792s) [noise=true , address=145.79.227.5:4223    , piece_id=S5QFIS3G3O6LLPPUYOMJJ3XQ65UGMCG4YGXTSKTK2VBVY3PYIZLA, seg_pos={0 0}, upload=540130]: Uploading piece...
Dec 31 07:50:15.773911727 (616h21m15.644994762s) [upload=540130]: Next returned nil segment
Dec 31 07:50:15.773924272 (616h21m15.645007317s) [upload=540130]: Waiting for error group managing segments...
Dec 31 07:50:16.060935968 (616h21m15.932019003s) [noise=true , address=145.79.227.16:4911   , piece_id=ICFDSO3IMTWB3ALB6NCE3ZZADCPPDNAW3BPWM7FP256S7FHF6VNQ, seg_pos={0 0}, upload=540130]: Done uploading piece. err: <nil>
Dec 31 07:50:16.066680917 (616h21m15.937763952s) [noise=true , address=145.79.227.22:4511   , piece_id=BNBMLNARFWOHBJ625BQ6SHSALCJ6DK3GDWHQZUAZZMXCFB3373EA, seg_pos={0 0}, upload=540130]: Done uploading piece. err: <nil>
Dec 31 07:50:16.067285969 (616h21m15.938369004s) [noise=true , address=145.79.227.13:4804   , piece_id=H7MIO4L43XKMOS6Q4YF5DYUN2ML7WTZLDC5ZARDLRC2J6RL66F2A, seg_pos={0 0}, upload=540130]: Done uploading piece. err: <nil>
Dec 31 07:50:16.074304904 (616h21m15.945387939s) [noise=true , address=145.79.227.31:4798   , piece_id=WJA2TTPEMLKWWUX6S62GIKMG6OUFZO5OJGINBBEIFU5OOINOT5ZQ, seg_pos={0 0}, upload=540130]: Done uploading piece. err: <nil>
Dec 31 07:50:16.091671072 (616h21m15.962754107s) [noise=true , address=145.79.227.42:4123   , piece_id=X52CDJL36PBKBBERX2RCRERUHFTLXUJQWID2IRQRDVYBRQMMAVNA, seg_pos={0 0}, upload=540130]: Done uploading piece. err: <nil>
Dec 31 07:50:16.091940876 (616h21m15.963023911s) [noise=true , address=145.79.227.5:4223    , piece_id=S5QFIS3G3O6LLPPUYOMJJ3XQ65UGMCG4YGXTSKTK2VBVY3PYIZLA, seg_pos={0 0}, upload=540130]: Done uploading piece. err: <nil>
Dec 31 07:50:16.092107543 (616h21m15.963190588s) [noise=true , address=145.79.227.45:4231   , piece_id=PP5BISNE5PGIFYI22EFU2R66LAQYOROMWX46BMPBXWRSLVHJV2YQ, seg_pos={0 0}, upload=540130]: Done uploading piece. err: <nil>
Dec 31 07:50:16.094320908 (616h21m15.965403953s) [noise=true , address=145.79.227.12:4767   , piece_id=QPIZLPAB72S7R4TBARYJLZC5PKBI5A76TX47V55KIUD7V4JMJPTQ, seg_pos={0 0}, upload=540130]: Done uploading piece. err: <nil>
Dec 31 07:50:16.101864181 (616h21m15.972947226s) [noise=true , address=145.79.227.17:4970   , piece_id=5UWCS6FGHJC6BVIBFY4KDUBDFQJW7XNAIFGNHBQBKYCWKM3BBIAA, seg_pos={0 0}, upload=540130]: Done uploading piece. err: <nil>
Dec 31 07:50:16.160938448 (616h21m16.032021493s) [noise=true , address=145.79.227.11:4730   , piece_id=3PYOSLOBQMF4MYAXEXW7UGX7GJSAMV3EKSMFRYZRWUFWGQ6LTWJA, seg_pos={0 0}, upload=540130]: Done uploading piece. err: <nil>
Dec 31 07:50:16.164001220 (616h21m16.035084255s) [noise=true , address=145.79.227.21:4474   , piece_id=GXR3RV363AOQ244TMOOJWC34S2FVXQAMZQAOGUGMC3UDEBFB7LGA, seg_pos={0 0}, upload=540130]: Done uploading piece. err: <nil>
Dec 31 07:50:16.312808293 (616h21m16.183891338s) [noise=true , address=77.90.31.17:4773     , piece_id=R5WYXJQU6TROJPZRTS4FTPRXJ4LSDUVW2XYZUMCOWRPBCY54RFEA, seg_pos={0 0}, upload=540130]: Done uploading piece. err: <nil>
Dec 31 07:50:16.385309145 (616h21m16.25639218s) [noise=true , address=45.91.116.9:4263     , piece_id=JYKGWRF3IA7FWWJY7QZFNOM6WOWOTCCKKZYLGEAQKQBTYWBLJFBA, seg_pos={0 0}, upload=540130]: Done uploading piece. err: <nil>
Dec 31 07:50:16.385335956 (616h21m16.256419001s) [seg_pos={0 0}, upload=540130]: Segment reached optimal threshold of 13 pieces.
Dec 31 07:50:16.385549212 (616h21m16.256632257s) [noise=true , address=145.79.227.39:4014   , piece_id=QTQ62XAYS2FYOHMUKZOU2GV3UWHPZILNF2K2V3WI7VU7ZFETTYFA, seg_pos={0 0}, upload=540130]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: expected piece hash; context canceled; EOF
Dec 31 07:50:16.385558891 (616h21m16.256641936s) [noise=true , address=145.79.227.39:4014   , piece_id=QTQ62XAYS2FYOHMUKZOU2GV3UWHPZILNF2K2V3WI7VU7ZFETTYFA, seg_pos={0 0}, upload=540130]: Long tail cancellation due to optimal threshold
Dec 31 07:50:16.385749364 (616h21m16.256832399s) [noise=true , address=45.91.116.6:4975     , piece_id=NRO4DTGVS7V7HAY3DFYMPL52DSRH3LS6E2EAXA3DSJVLHD7JBRZQ, seg_pos={0 0}, upload=540130]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF
Dec 31 07:50:16.385764483 (616h21m16.256847518s) [noise=true , address=45.91.116.6:4975     , piece_id=NRO4DTGVS7V7HAY3DFYMPL52DSRH3LS6E2EAXA3DSJVLHD7JBRZQ, seg_pos={0 0}, upload=540130]: Long tail cancellation due to optimal threshold
Dec 31 07:50:16.385962630 (616h21m16.257045685s) [noise=true , address=45.91.116.12:4227    , piece_id=BHDYBZYDEIGGM43X3TFX5K7EWL56QRSS5NQVRC5M4AEE5XD6OOJA, seg_pos={0 0}, upload=540130]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: write tcp 188.166.225.170:33022->45.91.116.12:4227: use of closed network connection; write tcp 188.166.225.170:33022->45.91.116.12:4227: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33022->45.91.116.12:4227: use of closed network connection
Dec 31 07:50:16.385973842 (616h21m16.257056877s) [noise=true , address=45.91.116.12:4227    , piece_id=BHDYBZYDEIGGM43X3TFX5K7EWL56QRSS5NQVRC5M4AEE5XD6OOJA, seg_pos={0 0}, upload=540130]: Long tail cancellation due to optimal threshold
Dec 31 07:50:16.385997626 (616h21m16.257080661s) [seg_pos={0 0}, upload=540130]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 07:50:16.386084672 (616h21m16.257167707s) [upload=540130]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 07:50:16.386090984 (616h21m16.257174039s) [upload=540130]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 07:50:16.669784681 (616h21m16.540867737s) [upload=540130]: Done uploading segments...
Dec 31 07:50:16.669914799 (616h21m16.540997845s) [upload=540130]: Upload finished. err: <nil>
Duration: 30.003453523s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
2 out of 60 is failed (threshold=3) [.............X............X.................................]
duration: 
42.577668188s
trace: 
https://jaeger.dev.storj.io/trace/6ce9bac61d803c5a
FAIL (2025-12-31 07:50:13 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
56     , piece_id=3PKC4JZPZVBQOBUHPQAG4HZ67DRTOY7GR35QVWLV3LN5JHXPMENA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.554293132 (615h55m26.425376177s) [noise=true , address=145.79.227.43:4186   , piece_id=ENVLK652WUVPKR3VPGXXMJ4IAEFHK67JOODS26ASTM4I2V7ABUWA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.554505907 (615h55m26.425588942s) [noise=true , address=77.90.31.14:4673     , piece_id=XYDCDFNKMWJK4HDELS4NLBVLUBR76YHT4RP3M5JO5QMCSI6MOGYA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.554648990 (615h55m26.425732025s) [noise=true , address=64.49.13.140:4505    , piece_id=EBEDU3F3QMDLNIDSRRXUHGSZ3VH5ITKRDDI2QS5XKBOGKOCN3LNA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.554994317 (615h55m26.426077352s) [noise=true , address=64.49.13.169:4707    , piece_id=RB7LFR5DM7PUHCJPB5ABWN63Q2XQOGNOG2W2FCKXFWU5MGVPRPVA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.555494150 (615h55m26.426577185s) [noise=true , address=64.49.13.137:4500    , piece_id=DD4JKFRHB2FKWAOTS5RZU5M3ASDGJTRX2XRB5ZROVOZNPN6BER2Q, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.555978812 (615h55m26.427061847s) [noise=true , address=45.91.116.10:4087    , piece_id=MJUIJFY6KEXEMMQKWA2NC3RE5PG55QCDZ7ECWRZG7CBROSA7LL3Q, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.556325282 (615h55m26.427408317s) [noise=true , address=145.79.227.29:4965   , piece_id=L6L2H7KMBTYKC24ZMACWRYK6TGOK5KHDO2YCFP5TJS66NXXP3KNQ, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.556728720 (615h55m26.427811765s) [noise=true , address=45.91.116.6:4977     , piece_id=G7JKD7INUS3D3RUIHSSVYCSOVTHSRG5HLVKERSCAD5NUWSFRZ2EA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.557164801 (615h55m26.428247836s) [noise=true , address=145.79.227.49:4370   , piece_id=TA4D4VCQXWYCB6FTZDRU6ZVZ4V6PZCFKQMA3EJYL5UEHHMI66JDQ, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.557530657 (615h55m26.428613692s) [noise=true , address=145.79.227.33:4896   , piece_id=TCDN3ER54RKZQ73BEI6DCGQW3IUNJXNNC3DUXI7NH7SFIPLHTF4A, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.557963953 (615h55m26.429046998s) [noise=true , address=77.90.31.12:4620     , piece_id=AQOFDODGJ7TMWA3ZQGWJW7XZT3FSKK3GR4R4RSIFIMHYP3HVJQDA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.558357422 (615h55m26.429440457s) [noise=true , address=145.79.227.40:4067   , piece_id=ITZ4SYUISESGVG72CBXHMAHZAD3XOOWYO5UUSFIT22R765N3W25A, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.558648285 (615h55m26.42973133s) [noise=true , address=145.79.227.2:4147    , piece_id=SHUFD6UKDFM54RGOZGJEUOUYCMKLAN2IMLWKJXZIZ5PAZVNUE4JQ, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.559228450 (615h55m26.430311485s) [noise=true , address=145.79.227.44:4228   , piece_id=YAQ7I3TOSLCYCHI2SJEEGCYGTEFWHX2CEYVZASRFL3BODTEONFTA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:27.974040599 (615h55m27.845123634s) [upload=539756]: Next returned nil segment
Dec 31 07:24:27.974074934 (615h55m27.845157969s) [upload=539756]: Waiting for error group managing segments...
Dec 31 07:24:28.334073958 (615h55m28.205156993s) [noise=true , address=64.49.13.140:4505    , piece_id=EBEDU3F3QMDLNIDSRRXUHGSZ3VH5ITKRDDI2QS5XKBOGKOCN3LNA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.427413642 (615h55m28.298496677s) [noise=true , address=145.79.227.2:4147    , piece_id=SHUFD6UKDFM54RGOZGJEUOUYCMKLAN2IMLWKJXZIZ5PAZVNUE4JQ, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.522823096 (615h55m28.393906131s) [noise=true , address=145.79.227.29:4965   , piece_id=L6L2H7KMBTYKC24ZMACWRYK6TGOK5KHDO2YCFP5TJS66NXXP3KNQ, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.585931513 (615h55m28.457014558s) [noise=true , address=77.90.31.13:4656     , piece_id=3PKC4JZPZVBQOBUHPQAG4HZ67DRTOY7GR35QVWLV3LN5JHXPMENA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.591579187 (615h55m28.462662262s) [noise=true , address=64.49.13.137:4500    , piece_id=DD4JKFRHB2FKWAOTS5RZU5M3ASDGJTRX2XRB5ZROVOZNPN6BER2Q, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.615175368 (615h55m28.486258413s) [noise=true , address=77.90.31.14:4673     , piece_id=XYDCDFNKMWJK4HDELS4NLBVLUBR76YHT4RP3M5JO5QMCSI6MOGYA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.699191016 (615h55m28.570274051s) [noise=true , address=45.91.116.6:4977     , piece_id=G7JKD7INUS3D3RUIHSSVYCSOVTHSRG5HLVKERSCAD5NUWSFRZ2EA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.713295994 (615h55m28.58437908s) [noise=true , address=45.91.116.10:4087    , piece_id=MJUIJFY6KEXEMMQKWA2NC3RE5PG55QCDZ7ECWRZG7CBROSA7LL3Q, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.794645326 (615h55m28.665728361s) [noise=true , address=77.90.31.12:4620     , piece_id=AQOFDODGJ7TMWA3ZQGWJW7XZT3FSKK3GR4R4RSIFIMHYP3HVJQDA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.341891306 (615h55m29.212974351s) [noise=true , address=145.79.227.33:4896   , piece_id=TCDN3ER54RKZQ73BEI6DCGQW3IUNJXNNC3DUXI7NH7SFIPLHTF4A, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.345487925 (615h55m29.21657096s) [noise=true , address=145.79.227.49:4370   , piece_id=TA4D4VCQXWYCB6FTZDRU6ZVZ4V6PZCFKQMA3EJYL5UEHHMI66JDQ, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.406150395 (615h55m29.27723343s) [noise=true , address=145.79.227.44:4228   , piece_id=YAQ7I3TOSLCYCHI2SJEEGCYGTEFWHX2CEYVZASRFL3BODTEONFTA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.414891660 (615h55m29.285974695s) [noise=true , address=193.149.164.143:4698 , piece_id=IYUHVRK3ZH3JADUJI627OUTHO7CRPNNKZVQTOCALMDYKVDGSHIJQ, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.431785668 (615h55m29.302868703s) [noise=true , address=145.79.227.40:4067   , piece_id=ITZ4SYUISESGVG72CBXHMAHZAD3XOOWYO5UUSFIT22R765N3W25A, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.431807049 (615h55m29.302890104s) [seg_pos={0 0}, upload=539756]: Segment reached optimal threshold of 14 pieces.
Dec 31 07:24:29.432061032 (615h55m29.303144087s) [noise=true , address=145.79.227.30:4784   , piece_id=M5NJQBX6JI5QI2PPA52B6A2CNLOXVWKYYYNX6CJDAA4WJNT4SHWA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pfXWex2WsuhmLb2smRpE46cHT9aipSTHW3e5oyELS1GhF1cFr): protocol: expected piece hash; context canceled; EOF
Dec 31 07:24:29.432072876 (615h55m29.303155931s) [noise=true , address=145.79.227.30:4784   , piece_id=M5NJQBX6JI5QI2PPA52B6A2CNLOXVWKYYYNX6CJDAA4WJNT4SHWA, seg_pos={0 0}, upload=539756]: Long tail cancellation due to optimal threshold
Dec 31 07:24:29.432090819 (615h55m29.303173884s) [noise=true , address=64.49.13.169:4709    , piece_id=CBAB3KCCPOW3P3LYM4T73CVDOD7FPJCPXYIXKDV7XTC2FKESTDRQ, seg_pos={0 0}, upload=539756]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:33942->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:33942->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33942->64.49.13.169:4709: use of closed network connection
Dec 31 07:24:29.432098063 (615h55m29.303181118s) [noise=true , address=64.49.13.169:4709    , piece_id=CBAB3KCCPOW3P3LYM4T73CVDOD7FPJCPXYIXKDV7XTC2FKESTDRQ, seg_pos={0 0}, upload=539756]: Long tail cancellation due to optimal threshold
Dec 31 07:24:29.432212572 (615h55m29.303295617s) [noise=true , address=64.49.13.169:4707    , piece_id=RB7LFR5DM7PUHCJPB5ABWN63Q2XQOGNOG2W2FCKXFWU5MGVPRPVA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:41264->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:41264->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41264->64.49.13.169:4707: use of closed network connection
Dec 31 07:24:29.432220747 (615h55m29.303303782s) [noise=true , address=64.49.13.169:4707    , piece_id=RB7LFR5DM7PUHCJPB5ABWN63Q2XQOGNOG2W2FCKXFWU5MGVPRPVA, seg_pos={0 0}, upload=539756]: Long tail cancellation due to optimal threshold
Dec 31 07:24:29.432465934 (615h55m29.303548979s) [noise=true , address=145.79.227.43:4186   , piece_id=ENVLK652WUVPKR3VPGXXMJ4IAEFHK67JOODS26ASTM4I2V7ABUWA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Gu3vXy2JUkKDaF4MkMCgVmYspW4Ez8EAfv6e4GeCkrwYTe5T): protocol: expected piece hash; context canceled; EOF
Dec 31 07:24:29.432474971 (615h55m29.303558006s) [noise=true , address=145.79.227.43:4186   , piece_id=ENVLK652WUVPKR3VPGXXMJ4IAEFHK67JOODS26ASTM4I2V7ABUWA, seg_pos={0 0}, upload=539756]: Long tail cancellation due to optimal threshold
Dec 31 07:24:29.432580261 (615h55m29.303663296s) [seg_pos={0 0}, upload=539756]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 07:24:29.432664502 (615h55m29.303747547s) [upload=539756]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 07:24:29.432671305 (615h55m29.30375435s) [upload=539756]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 07:24:29.717299118 (615h55m29.588382153s) [upload=539756]: Done uploading segments...
Dec 31 07:24:29.717320589 (615h55m29.588403624s) [upload=539756]: Upload finished. err: <nil>
Duration: 42.577690231s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [.................................................X..........]
duration: 
6.041075211s
trace: 
https://jaeger.dev.storj.io/trace/5dd0609e3a2ac547
FAIL (2025-12-31 07:50:11 UTC) 
orbiter warning: context deadline exceeded
Duration: 6.041104728s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
16 out of 60 is failed (threshold=3) [.....X....X.X.X.X.X......XXX.X....XXX..X...........X......X.]
duration: 
6.629887117s
trace: 
https://jaeger.dev.storj.io/trace/7b896eeb780c4abd
FAIL (2025-12-31 07:49:57 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
1 07:49:21.128152979 (616h20m20.999236024s) [upload=540118]: Starting upload
Dec 31 07:49:21.128246488 (616h20m20.999329543s) [upload=540118]: Done starting upload
Dec 31 07:49:21.128345235 (616h20m20.99942827s) [upload=540118]: Uploading segments...
Dec 31 07:49:21.128548452 (616h20m20.999631497s) [upload=540118]: Got next segment. Inline: false
Dec 31 07:49:21.128605952 (616h20m20.999689007s) [upload=540118]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 07:49:21.128612183 (616h20m20.999695229s) [upload=540118]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 07:49:21.373684801 (616h20m21.244767846s) [seg_pos={0 0}, upload=540118]: Begin upload segment...
Dec 31 07:49:21.373744274 (616h20m21.24482732s) [seg_pos={0 0}, upload=540118]: Done begin upload segment.
Dec 31 07:49:21.373787747 (616h20m21.244870792s) [noise=true , address=145.79.227.44:4195   , piece_id=S2KBRZDYTHW7VWSOTIVPO2E2QZWXJ5IVNJKCNMQLY4AQG2Q4FJUA, seg_pos={0 0}, upload=540118]: Uploading piece...
Dec 31 07:49:21.373793518 (616h20m21.244876553s) [noise=true , address=145.79.227.37:4565   , piece_id=6I74AJHXFKRINGJCLJWTCXZ2ZRH4NAUAW5JQR7XIG2UT6VUFKUQA, seg_pos={0 0}, upload=540118]: Uploading piece...
Dec 31 07:49:21.374469746 (616h20m21.245552791s) [noise=true , address=77.90.31.12:4589     , piece_id=XNOPOY4QIZBMES7XBECJA2X3EEV7JMGDZ3TUNDUAJCBTB4KITGUQ, seg_pos={0 0}, upload=540118]: Uploading piece...
Dec 31 07:49:21.374502298 (616h20m21.245585343s) [noise=true , address=145.79.227.38:4978   , piece_id=4YTUAM7ZQFCFDRCLU6VA5CSZGQ7QC6F7KVUVZUKDAMKXS3L7YPQQ, seg_pos={0 0}, upload=540118]: Uploading piece...
Dec 31 07:49:21.375001819 (616h20m21.246084854s) [noise=true , address=145.79.227.15:4874   , piece_id=7ASBKXQHTF27FD6BUXO4KBCG2EMTCJBZASES5AUJ5ADD6E4HXYPQ, seg_pos={0 0}, upload=540118]: Uploading piece...
Dec 31 07:49:21.375086159 (616h20m21.246169204s) [noise=true , address=77.90.31.21:4877     , piece_id=XZK2KSZ2UD6BLPEITKERV5X6NVYZO65RSJVFDNRVJ7PPJEHOCEZA, seg_pos={0 0}, upload=540118]: Uploading piece...
Dec 31 07:49:21.375446085 (616h20m21.24652912s) [noise=true , address=45.91.116.14:4192    , piece_id=L4E343B3HDSQS5UNNXKPSTLNMRFY2TPS5NQJYPENHCNGWZAN3BZQ, seg_pos={0 0}, upload=540118]: Uploading piece...
Dec 31 07:49:21.375548780 (616h20m21.246631825s) [noise=true , address=45.91.116.12:4227    , piece_id=IVM52J54V6KYNRL6XCDDM6VHS4YREVM43WH4MIEAURYXTDBU3CAA, seg_pos={0 0}, upload=540118]: Uploading piece...
Dec 31 07:49:21.376076135 (616h20m21.24715917s) [noise=true , address=145.79.227.17:4971   , piece_id=XEHTHVXQZ3ISAVJXHV7GFB7PZ2264SPN53Y3U3KZN6UA74NGPKFQ, seg_pos={0 0}, upload=540118]: Uploading piece...
Dec 31 07:49:21.376193408 (616h20m21.247276453s) [noise=true , address=145.79.227.51:4411   , piece_id=DFWPMYIBXYDN3MWGFCB4PXAWJDX2XE36WZUESRWFXLU3Z3K7Q4BA, seg_pos={0 0}, upload=540118]: Uploading piece...
Dec 31 07:49:21.376790004 (616h20m21.247873039s) [noise=true , address=145.79.227.52:4445   , piece_id=G5KFO7W32PJCTAKJHLETQ2OTW2P7M2L7WPKDXYJMMLBOGYXBUFFA, seg_pos={0 0}, upload=540118]: Uploading piece...
Dec 31 07:49:21.377378003 (616h20m21.248461048s) [noise=true , address=145.79.227.45:4230   , piece_id=FBMERVJZDLJSST7LYETFMFHMLHHGDSU5KKDDU2ASQNL3F6TLCI4A, seg_pos={0 0}, upload=540118]: Uploading piece...
Dec 31 07:49:21.377779708 (616h20m21.248862753s) [noise=true , address=45.91.116.8:4046     , piece_id=K2NBRJQORMED4PZQTGB5ZJBCUC2BVUJSK5YAQ6I44QBLPBM4VC7Q, seg_pos={0 0}, upload=540118]: Uploading piece...
Dec 31 07:49:21.378142058 (616h20m21.249225093s) [noise=true , address=77.90.31.15:4700     , piece_id=FSKJ5GF6YC3A3AKTOQDXFB6VOT3KZVFK42757WCZUYDPQNEWKL6Q, seg_pos={0 0}, upload=540118]: Uploading piece...
Dec 31 07:49:21.378457028 (616h20m21.249540073s) [noise=true , address=145.79.227.8:4330    , piece_id=YFM4ONEAE2QSVO3D3RWKPPFVTSSKVAQUXM74EM2ENL2WQYSWFIOA, seg_pos={0 0}, upload=540118]: Uploading piece...
Dec 31 07:49:21.378927503 (616h20m21.250010538s) [noise=true , address=145.79.227.19:4077   , piece_id=HXLP3FOFAMNKK2ZWWFQIK36HSBOB4MBMTFDUVMX2XRDHHJ3M6ULQ, seg_pos={0 0}, upload=540118]: Uploading piece...
Dec 31 07:49:23.085140182 (616h20m22.956223227s) [upload=540118]: Next returned nil segment
Dec 31 07:49:23.085170600 (616h20m22.956253635s) [upload=540118]: Waiting for error group managing segments...
Dec 31 07:49:23.362502985 (616h20m23.23358603s) [noise=true , address=145.79.227.38:4978   , piece_id=4YTUAM7ZQFCFDRCLU6VA5CSZGQ7QC6F7KVUVZUKDAMKXS3L7YPQQ, seg_pos={0 0}, upload=540118]: Done uploading piece. err: <nil>
Dec 31 07:49:23.363035519 (616h20m23.234118564s) [noise=true , address=145.79.227.19:4077   , piece_id=HXLP3FOFAMNKK2ZWWFQIK36HSBOB4MBMTFDUVMX2XRDHHJ3M6ULQ, seg_pos={0 0}, upload=540118]: Done uploading piece. err: <nil>
Dec 31 07:49:23.364140613 (616h20m23.235223658s) [noise=true , address=145.79.227.15:4874   , piece_id=7ASBKXQHTF27FD6BUXO4KBCG2EMTCJBZASES5AUJ5ADD6E4HXYPQ, seg_pos={0 0}, upload=540118]: Done uploading piece. err: <nil>
Dec 31 07:49:23.366746986 (616h20m23.237830021s) [noise=true , address=145.79.227.8:4330    , piece_id=YFM4ONEAE2QSVO3D3RWKPPFVTSSKVAQUXM74EM2ENL2WQYSWFIOA, seg_pos={0 0}, upload=540118]: Done uploading piece. err: <nil>
Dec 31 07:49:23.379595953 (616h20m23.250678988s) [noise=true , address=145.79.227.17:4971   , piece_id=XEHTHVXQZ3ISAVJXHV7GFB7PZ2264SPN53Y3U3KZN6UA74NGPKFQ, seg_pos={0 0}, upload=540118]: Done uploading piece. err: <nil>
Dec 31 07:49:23.379967481 (616h20m23.251050516s) [noise=true , address=145.79.227.37:4565   , piece_id=6I74AJHXFKRINGJCLJWTCXZ2ZRH4NAUAW5JQR7XIG2UT6VUFKUQA, seg_pos={0 0}, upload=540118]: Done uploading piece. err: <nil>
Dec 31 07:49:23.381535176 (616h20m23.252618211s) [noise=true , address=145.79.227.51:4411   , piece_id=DFWPMYIBXYDN3MWGFCB4PXAWJDX2XE36WZUESRWFXLU3Z3K7Q4BA, seg_pos={0 0}, upload=540118]: Done uploading piece. err: <nil>
Dec 31 07:49:23.385748509 (616h20m23.256831554s) [noise=true , address=145.79.227.52:4445   , piece_id=G5KFO7W32PJCTAKJHLETQ2OTW2P7M2L7WPKDXYJMMLBOGYXBUFFA, seg_pos={0 0}, upload=540118]: Done uploading piece. err: <nil>
Dec 31 07:49:23.443973718 (616h20m23.315056763s) [noise=true , address=145.79.227.45:4230   , piece_id=FBMERVJZDLJSST7LYETFMFHMLHHGDSU5KKDDU2ASQNL3F6TLCI4A, seg_pos={0 0}, upload=540118]: Done uploading piece. err: <nil>
Dec 31 07:49:23.568685269 (616h20m23.439768304s) [noise=true , address=77.90.31.12:4589     , piece_id=XNOPOY4QIZBMES7XBECJA2X3EEV7JMGDZ3TUNDUAJCBTB4KITGUQ, seg_pos={0 0}, upload=540118]: Done uploading piece. err: <nil>
Dec 31 07:49:23.659736483 (616h20m23.530819518s) [noise=true , address=77.90.31.21:4877     , piece_id=XZK2KSZ2UD6BLPEITKERV5X6NVYZO65RSJVFDNRVJ7PPJEHOCEZA, seg_pos={0 0}, upload=540118]: Done uploading piece. err: <nil>
Dec 31 07:49:23.743894915 (616h20m23.61497796s) [noise=true , address=45.91.116.8:4046     , piece_id=K2NBRJQORMED4PZQTGB5ZJBCUC2BVUJSK5YAQ6I44QBLPBM4VC7Q, seg_pos={0 0}, upload=540118]: Done uploading piece. err: <nil>
Dec 31 07:49:23.845283845 (616h20m23.71636688s) [noise=true , address=77.90.31.15:4700     , piece_id=FSKJ5GF6YC3A3AKTOQDXFB6VOT3KZVFK42757WCZUYDPQNEWKL6Q, seg_pos={0 0}, upload=540118]: Done uploading piece. err: <nil>
Dec 31 07:49:23.845305636 (616h20m23.716388671s) [seg_pos={0 0}, upload=540118]: Segment reached optimal threshold of 13 pieces.
Dec 31 07:49:23.845563417 (616h20m23.716646463s) [noise=true , address=45.91.116.12:4227    , piece_id=IVM52J54V6KYNRL6XCDDM6VHS4YREVM43WH4MIEAURYXTDBU3CAA, seg_pos={0 0}, upload=540118]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF
Dec 31 07:49:23.845571032 (616h20m23.716654087s) [noise=true , address=45.91.116.12:4227    , piece_id=IVM52J54V6KYNRL6XCDDM6VHS4YREVM43WH4MIEAURYXTDBU3CAA, seg_pos={0 0}, upload=540118]: Long tail cancellation due to optimal threshold
Dec 31 07:49:23.845733822 (616h20m23.716816857s) [noise=true , address=145.79.227.44:4195   , piece_id=S2KBRZDYTHW7VWSOTIVPO2E2QZWXJ5IVNJKCNMQLY4AQG2Q4FJUA, seg_pos={0 0}, upload=540118]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF
Dec 31 07:49:23.845749181 (616h20m23.716832216s) [noise=true , address=145.79.227.44:4195   , piece_id=S2KBRZDYTHW7VWSOTIVPO2E2QZWXJ5IVNJKCNMQLY4AQG2Q4FJUA, seg_pos={0 0}, upload=540118]: Long tail cancellation due to optimal threshold
Dec 31 07:49:23.845915177 (616h20m23.716998222s) [noise=true , address=45.91.116.14:4192    , piece_id=L4E343B3HDSQS5UNNXKPSTLNMRFY2TPS5NQJYPENHCNGWZAN3BZQ, seg_pos={0 0}, upload=540118]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: write tcp 188.166.225.170:37366->45.91.116.14:4192: use of closed network connection; write tcp 188.166.225.170:37366->45.91.116.14:4192: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37366->45.91.116.14:4192: use of closed network connection
Dec 31 07:49:23.845925426 (616h20m23.717008472s) [noise=true , address=45.91.116.14:4192    , piece_id=L4E343B3HDSQS5UNNXKPSTLNMRFY2TPS5NQJYPENHCNGWZAN3BZQ, seg_pos={0 0}, upload=540118]: Long tail cancellation due to optimal threshold
Dec 31 07:49:23.845945756 (616h20m23.717028801s) [seg_pos={0 0}, upload=540118]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 07:49:23.846028783 (616h20m23.717111838s) [upload=540118]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 07:49:23.846034594 (616h20m23.717117639s) [upload=540118]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 07:49:24.134752079 (616h20m24.005835114s) [upload=540118]: Done uploading segments...
Dec 31 07:49:24.134776184 (616h20m24.005859229s) [upload=540118]: Upload finished. err: <nil>
Duration: 6.629914028s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
17 out of 60 is failed (threshold=3) [X.....X....X.X.X.X.X......XXX.X....XXX..X...........X......X]
duration: 
30.003093454s
trace: 
https://jaeger.dev.storj.io/trace/7b896eeb780c4abd
FAIL (2025-12-31 07:49:50 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
1 07:49:21.128152979 (616h20m20.999236024s) [upload=540118]: Starting upload
Dec 31 07:49:21.128246488 (616h20m20.999329543s) [upload=540118]: Done starting upload
Dec 31 07:49:21.128345235 (616h20m20.99942827s) [upload=540118]: Uploading segments...
Dec 31 07:49:21.128548452 (616h20m20.999631497s) [upload=540118]: Got next segment. Inline: false
Dec 31 07:49:21.128605952 (616h20m20.999689007s) [upload=540118]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 07:49:21.128612183 (616h20m20.999695229s) [upload=540118]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 07:49:21.373684801 (616h20m21.244767846s) [seg_pos={0 0}, upload=540118]: Begin upload segment...
Dec 31 07:49:21.373744274 (616h20m21.24482732s) [seg_pos={0 0}, upload=540118]: Done begin upload segment.
Dec 31 07:49:21.373787747 (616h20m21.244870792s) [noise=true , address=145.79.227.44:4195   , piece_id=S2KBRZDYTHW7VWSOTIVPO2E2QZWXJ5IVNJKCNMQLY4AQG2Q4FJUA, seg_pos={0 0}, upload=540118]: Uploading piece...
Dec 31 07:49:21.373793518 (616h20m21.244876553s) [noise=true , address=145.79.227.37:4565   , piece_id=6I74AJHXFKRINGJCLJWTCXZ2ZRH4NAUAW5JQR7XIG2UT6VUFKUQA, seg_pos={0 0}, upload=540118]: Uploading piece...
Dec 31 07:49:21.374469746 (616h20m21.245552791s) [noise=true , address=77.90.31.12:4589     , piece_id=XNOPOY4QIZBMES7XBECJA2X3EEV7JMGDZ3TUNDUAJCBTB4KITGUQ, seg_pos={0 0}, upload=540118]: Uploading piece...
Dec 31 07:49:21.374502298 (616h20m21.245585343s) [noise=true , address=145.79.227.38:4978   , piece_id=4YTUAM7ZQFCFDRCLU6VA5CSZGQ7QC6F7KVUVZUKDAMKXS3L7YPQQ, seg_pos={0 0}, upload=540118]: Uploading piece...
Dec 31 07:49:21.375001819 (616h20m21.246084854s) [noise=true , address=145.79.227.15:4874   , piece_id=7ASBKXQHTF27FD6BUXO4KBCG2EMTCJBZASES5AUJ5ADD6E4HXYPQ, seg_pos={0 0}, upload=540118]: Uploading piece...
Dec 31 07:49:21.375086159 (616h20m21.246169204s) [noise=true , address=77.90.31.21:4877     , piece_id=XZK2KSZ2UD6BLPEITKERV5X6NVYZO65RSJVFDNRVJ7PPJEHOCEZA, seg_pos={0 0}, upload=540118]: Uploading piece...
Dec 31 07:49:21.375446085 (616h20m21.24652912s) [noise=true , address=45.91.116.14:4192    , piece_id=L4E343B3HDSQS5UNNXKPSTLNMRFY2TPS5NQJYPENHCNGWZAN3BZQ, seg_pos={0 0}, upload=540118]: Uploading piece...
Dec 31 07:49:21.375548780 (616h20m21.246631825s) [noise=true , address=45.91.116.12:4227    , piece_id=IVM52J54V6KYNRL6XCDDM6VHS4YREVM43WH4MIEAURYXTDBU3CAA, seg_pos={0 0}, upload=540118]: Uploading piece...
Dec 31 07:49:21.376076135 (616h20m21.24715917s) [noise=true , address=145.79.227.17:4971   , piece_id=XEHTHVXQZ3ISAVJXHV7GFB7PZ2264SPN53Y3U3KZN6UA74NGPKFQ, seg_pos={0 0}, upload=540118]: Uploading piece...
Dec 31 07:49:21.376193408 (616h20m21.247276453s) [noise=true , address=145.79.227.51:4411   , piece_id=DFWPMYIBXYDN3MWGFCB4PXAWJDX2XE36WZUESRWFXLU3Z3K7Q4BA, seg_pos={0 0}, upload=540118]: Uploading piece...
Dec 31 07:49:21.376790004 (616h20m21.247873039s) [noise=true , address=145.79.227.52:4445   , piece_id=G5KFO7W32PJCTAKJHLETQ2OTW2P7M2L7WPKDXYJMMLBOGYXBUFFA, seg_pos={0 0}, upload=540118]: Uploading piece...
Dec 31 07:49:21.377378003 (616h20m21.248461048s) [noise=true , address=145.79.227.45:4230   , piece_id=FBMERVJZDLJSST7LYETFMFHMLHHGDSU5KKDDU2ASQNL3F6TLCI4A, seg_pos={0 0}, upload=540118]: Uploading piece...
Dec 31 07:49:21.377779708 (616h20m21.248862753s) [noise=true , address=45.91.116.8:4046     , piece_id=K2NBRJQORMED4PZQTGB5ZJBCUC2BVUJSK5YAQ6I44QBLPBM4VC7Q, seg_pos={0 0}, upload=540118]: Uploading piece...
Dec 31 07:49:21.378142058 (616h20m21.249225093s) [noise=true , address=77.90.31.15:4700     , piece_id=FSKJ5GF6YC3A3AKTOQDXFB6VOT3KZVFK42757WCZUYDPQNEWKL6Q, seg_pos={0 0}, upload=540118]: Uploading piece...
Dec 31 07:49:21.378457028 (616h20m21.249540073s) [noise=true , address=145.79.227.8:4330    , piece_id=YFM4ONEAE2QSVO3D3RWKPPFVTSSKVAQUXM74EM2ENL2WQYSWFIOA, seg_pos={0 0}, upload=540118]: Uploading piece...
Dec 31 07:49:21.378927503 (616h20m21.250010538s) [noise=true , address=145.79.227.19:4077   , piece_id=HXLP3FOFAMNKK2ZWWFQIK36HSBOB4MBMTFDUVMX2XRDHHJ3M6ULQ, seg_pos={0 0}, upload=540118]: Uploading piece...
Dec 31 07:49:23.085140182 (616h20m22.956223227s) [upload=540118]: Next returned nil segment
Dec 31 07:49:23.085170600 (616h20m22.956253635s) [upload=540118]: Waiting for error group managing segments...
Dec 31 07:49:23.362502985 (616h20m23.23358603s) [noise=true , address=145.79.227.38:4978   , piece_id=4YTUAM7ZQFCFDRCLU6VA5CSZGQ7QC6F7KVUVZUKDAMKXS3L7YPQQ, seg_pos={0 0}, upload=540118]: Done uploading piece. err: <nil>
Dec 31 07:49:23.363035519 (616h20m23.234118564s) [noise=true , address=145.79.227.19:4077   , piece_id=HXLP3FOFAMNKK2ZWWFQIK36HSBOB4MBMTFDUVMX2XRDHHJ3M6ULQ, seg_pos={0 0}, upload=540118]: Done uploading piece. err: <nil>
Dec 31 07:49:23.364140613 (616h20m23.235223658s) [noise=true , address=145.79.227.15:4874   , piece_id=7ASBKXQHTF27FD6BUXO4KBCG2EMTCJBZASES5AUJ5ADD6E4HXYPQ, seg_pos={0 0}, upload=540118]: Done uploading piece. err: <nil>
Dec 31 07:49:23.366746986 (616h20m23.237830021s) [noise=true , address=145.79.227.8:4330    , piece_id=YFM4ONEAE2QSVO3D3RWKPPFVTSSKVAQUXM74EM2ENL2WQYSWFIOA, seg_pos={0 0}, upload=540118]: Done uploading piece. err: <nil>
Dec 31 07:49:23.379595953 (616h20m23.250678988s) [noise=true , address=145.79.227.17:4971   , piece_id=XEHTHVXQZ3ISAVJXHV7GFB7PZ2264SPN53Y3U3KZN6UA74NGPKFQ, seg_pos={0 0}, upload=540118]: Done uploading piece. err: <nil>
Dec 31 07:49:23.379967481 (616h20m23.251050516s) [noise=true , address=145.79.227.37:4565   , piece_id=6I74AJHXFKRINGJCLJWTCXZ2ZRH4NAUAW5JQR7XIG2UT6VUFKUQA, seg_pos={0 0}, upload=540118]: Done uploading piece. err: <nil>
Dec 31 07:49:23.381535176 (616h20m23.252618211s) [noise=true , address=145.79.227.51:4411   , piece_id=DFWPMYIBXYDN3MWGFCB4PXAWJDX2XE36WZUESRWFXLU3Z3K7Q4BA, seg_pos={0 0}, upload=540118]: Done uploading piece. err: <nil>
Dec 31 07:49:23.385748509 (616h20m23.256831554s) [noise=true , address=145.79.227.52:4445   , piece_id=G5KFO7W32PJCTAKJHLETQ2OTW2P7M2L7WPKDXYJMMLBOGYXBUFFA, seg_pos={0 0}, upload=540118]: Done uploading piece. err: <nil>
Dec 31 07:49:23.443973718 (616h20m23.315056763s) [noise=true , address=145.79.227.45:4230   , piece_id=FBMERVJZDLJSST7LYETFMFHMLHHGDSU5KKDDU2ASQNL3F6TLCI4A, seg_pos={0 0}, upload=540118]: Done uploading piece. err: <nil>
Dec 31 07:49:23.568685269 (616h20m23.439768304s) [noise=true , address=77.90.31.12:4589     , piece_id=XNOPOY4QIZBMES7XBECJA2X3EEV7JMGDZ3TUNDUAJCBTB4KITGUQ, seg_pos={0 0}, upload=540118]: Done uploading piece. err: <nil>
Dec 31 07:49:23.659736483 (616h20m23.530819518s) [noise=true , address=77.90.31.21:4877     , piece_id=XZK2KSZ2UD6BLPEITKERV5X6NVYZO65RSJVFDNRVJ7PPJEHOCEZA, seg_pos={0 0}, upload=540118]: Done uploading piece. err: <nil>
Dec 31 07:49:23.743894915 (616h20m23.61497796s) [noise=true , address=45.91.116.8:4046     , piece_id=K2NBRJQORMED4PZQTGB5ZJBCUC2BVUJSK5YAQ6I44QBLPBM4VC7Q, seg_pos={0 0}, upload=540118]: Done uploading piece. err: <nil>
Dec 31 07:49:23.845283845 (616h20m23.71636688s) [noise=true , address=77.90.31.15:4700     , piece_id=FSKJ5GF6YC3A3AKTOQDXFB6VOT3KZVFK42757WCZUYDPQNEWKL6Q, seg_pos={0 0}, upload=540118]: Done uploading piece. err: <nil>
Dec 31 07:49:23.845305636 (616h20m23.716388671s) [seg_pos={0 0}, upload=540118]: Segment reached optimal threshold of 13 pieces.
Dec 31 07:49:23.845563417 (616h20m23.716646463s) [noise=true , address=45.91.116.12:4227    , piece_id=IVM52J54V6KYNRL6XCDDM6VHS4YREVM43WH4MIEAURYXTDBU3CAA, seg_pos={0 0}, upload=540118]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF
Dec 31 07:49:23.845571032 (616h20m23.716654087s) [noise=true , address=45.91.116.12:4227    , piece_id=IVM52J54V6KYNRL6XCDDM6VHS4YREVM43WH4MIEAURYXTDBU3CAA, seg_pos={0 0}, upload=540118]: Long tail cancellation due to optimal threshold
Dec 31 07:49:23.845733822 (616h20m23.716816857s) [noise=true , address=145.79.227.44:4195   , piece_id=S2KBRZDYTHW7VWSOTIVPO2E2QZWXJ5IVNJKCNMQLY4AQG2Q4FJUA, seg_pos={0 0}, upload=540118]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF
Dec 31 07:49:23.845749181 (616h20m23.716832216s) [noise=true , address=145.79.227.44:4195   , piece_id=S2KBRZDYTHW7VWSOTIVPO2E2QZWXJ5IVNJKCNMQLY4AQG2Q4FJUA, seg_pos={0 0}, upload=540118]: Long tail cancellation due to optimal threshold
Dec 31 07:49:23.845915177 (616h20m23.716998222s) [noise=true , address=45.91.116.14:4192    , piece_id=L4E343B3HDSQS5UNNXKPSTLNMRFY2TPS5NQJYPENHCNGWZAN3BZQ, seg_pos={0 0}, upload=540118]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: write tcp 188.166.225.170:37366->45.91.116.14:4192: use of closed network connection; write tcp 188.166.225.170:37366->45.91.116.14:4192: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37366->45.91.116.14:4192: use of closed network connection
Dec 31 07:49:23.845925426 (616h20m23.717008472s) [noise=true , address=45.91.116.14:4192    , piece_id=L4E343B3HDSQS5UNNXKPSTLNMRFY2TPS5NQJYPENHCNGWZAN3BZQ, seg_pos={0 0}, upload=540118]: Long tail cancellation due to optimal threshold
Dec 31 07:49:23.845945756 (616h20m23.717028801s) [seg_pos={0 0}, upload=540118]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 07:49:23.846028783 (616h20m23.717111838s) [upload=540118]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 07:49:23.846034594 (616h20m23.717117639s) [upload=540118]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 07:49:24.134752079 (616h20m24.005835114s) [upload=540118]: Done uploading segments...
Dec 31 07:49:24.134776184 (616h20m24.005859229s) [upload=540118]: Upload finished. err: <nil>
Duration: 30.003124955s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
4 out of 60 is failed (threshold=6) [..X...X................................X.........X..........]
duration: 
5.00055162s
trace: 
https://jaeger.dev.storj.io/trace/6fc9bc24e7467571
FAIL (2025-12-31 07:49:49 UTC) 
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.000578341s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)
antiblip: 
1 out of 60 is failed (threshold=3) [...................................................X........]
duration: 
3.678639273s
trace: 
https://jaeger.dev.storj.io/trace/59c3b821b75eba3a
FAIL (2025-12-31 07:49:48 UTC) 
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded
Duration: 3.6786692s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [..................................................X.........]
duration: 
12.503494707s
trace: 
https://jaeger.dev.storj.io/trace/5dd0609e3a2ac547
FAIL (2025-12-31 07:49:36 UTC) 
orbiter warning: context deadline exceeded
Duration: 12.503523251s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
2 out of 60 is failed (threshold=3) [..............X............X................................]
duration: 
35.234667845s
trace: 
https://jaeger.dev.storj.io/trace/6ce9bac61d803c5a
FAIL (2025-12-31 07:49:31 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
56     , piece_id=3PKC4JZPZVBQOBUHPQAG4HZ67DRTOY7GR35QVWLV3LN5JHXPMENA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.554293132 (615h55m26.425376177s) [noise=true , address=145.79.227.43:4186   , piece_id=ENVLK652WUVPKR3VPGXXMJ4IAEFHK67JOODS26ASTM4I2V7ABUWA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.554505907 (615h55m26.425588942s) [noise=true , address=77.90.31.14:4673     , piece_id=XYDCDFNKMWJK4HDELS4NLBVLUBR76YHT4RP3M5JO5QMCSI6MOGYA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.554648990 (615h55m26.425732025s) [noise=true , address=64.49.13.140:4505    , piece_id=EBEDU3F3QMDLNIDSRRXUHGSZ3VH5ITKRDDI2QS5XKBOGKOCN3LNA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.554994317 (615h55m26.426077352s) [noise=true , address=64.49.13.169:4707    , piece_id=RB7LFR5DM7PUHCJPB5ABWN63Q2XQOGNOG2W2FCKXFWU5MGVPRPVA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.555494150 (615h55m26.426577185s) [noise=true , address=64.49.13.137:4500    , piece_id=DD4JKFRHB2FKWAOTS5RZU5M3ASDGJTRX2XRB5ZROVOZNPN6BER2Q, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.555978812 (615h55m26.427061847s) [noise=true , address=45.91.116.10:4087    , piece_id=MJUIJFY6KEXEMMQKWA2NC3RE5PG55QCDZ7ECWRZG7CBROSA7LL3Q, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.556325282 (615h55m26.427408317s) [noise=true , address=145.79.227.29:4965   , piece_id=L6L2H7KMBTYKC24ZMACWRYK6TGOK5KHDO2YCFP5TJS66NXXP3KNQ, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.556728720 (615h55m26.427811765s) [noise=true , address=45.91.116.6:4977     , piece_id=G7JKD7INUS3D3RUIHSSVYCSOVTHSRG5HLVKERSCAD5NUWSFRZ2EA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.557164801 (615h55m26.428247836s) [noise=true , address=145.79.227.49:4370   , piece_id=TA4D4VCQXWYCB6FTZDRU6ZVZ4V6PZCFKQMA3EJYL5UEHHMI66JDQ, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.557530657 (615h55m26.428613692s) [noise=true , address=145.79.227.33:4896   , piece_id=TCDN3ER54RKZQ73BEI6DCGQW3IUNJXNNC3DUXI7NH7SFIPLHTF4A, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.557963953 (615h55m26.429046998s) [noise=true , address=77.90.31.12:4620     , piece_id=AQOFDODGJ7TMWA3ZQGWJW7XZT3FSKK3GR4R4RSIFIMHYP3HVJQDA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.558357422 (615h55m26.429440457s) [noise=true , address=145.79.227.40:4067   , piece_id=ITZ4SYUISESGVG72CBXHMAHZAD3XOOWYO5UUSFIT22R765N3W25A, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.558648285 (615h55m26.42973133s) [noise=true , address=145.79.227.2:4147    , piece_id=SHUFD6UKDFM54RGOZGJEUOUYCMKLAN2IMLWKJXZIZ5PAZVNUE4JQ, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.559228450 (615h55m26.430311485s) [noise=true , address=145.79.227.44:4228   , piece_id=YAQ7I3TOSLCYCHI2SJEEGCYGTEFWHX2CEYVZASRFL3BODTEONFTA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:27.974040599 (615h55m27.845123634s) [upload=539756]: Next returned nil segment
Dec 31 07:24:27.974074934 (615h55m27.845157969s) [upload=539756]: Waiting for error group managing segments...
Dec 31 07:24:28.334073958 (615h55m28.205156993s) [noise=true , address=64.49.13.140:4505    , piece_id=EBEDU3F3QMDLNIDSRRXUHGSZ3VH5ITKRDDI2QS5XKBOGKOCN3LNA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.427413642 (615h55m28.298496677s) [noise=true , address=145.79.227.2:4147    , piece_id=SHUFD6UKDFM54RGOZGJEUOUYCMKLAN2IMLWKJXZIZ5PAZVNUE4JQ, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.522823096 (615h55m28.393906131s) [noise=true , address=145.79.227.29:4965   , piece_id=L6L2H7KMBTYKC24ZMACWRYK6TGOK5KHDO2YCFP5TJS66NXXP3KNQ, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.585931513 (615h55m28.457014558s) [noise=true , address=77.90.31.13:4656     , piece_id=3PKC4JZPZVBQOBUHPQAG4HZ67DRTOY7GR35QVWLV3LN5JHXPMENA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.591579187 (615h55m28.462662262s) [noise=true , address=64.49.13.137:4500    , piece_id=DD4JKFRHB2FKWAOTS5RZU5M3ASDGJTRX2XRB5ZROVOZNPN6BER2Q, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.615175368 (615h55m28.486258413s) [noise=true , address=77.90.31.14:4673     , piece_id=XYDCDFNKMWJK4HDELS4NLBVLUBR76YHT4RP3M5JO5QMCSI6MOGYA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.699191016 (615h55m28.570274051s) [noise=true , address=45.91.116.6:4977     , piece_id=G7JKD7INUS3D3RUIHSSVYCSOVTHSRG5HLVKERSCAD5NUWSFRZ2EA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.713295994 (615h55m28.58437908s) [noise=true , address=45.91.116.10:4087    , piece_id=MJUIJFY6KEXEMMQKWA2NC3RE5PG55QCDZ7ECWRZG7CBROSA7LL3Q, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.794645326 (615h55m28.665728361s) [noise=true , address=77.90.31.12:4620     , piece_id=AQOFDODGJ7TMWA3ZQGWJW7XZT3FSKK3GR4R4RSIFIMHYP3HVJQDA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.341891306 (615h55m29.212974351s) [noise=true , address=145.79.227.33:4896   , piece_id=TCDN3ER54RKZQ73BEI6DCGQW3IUNJXNNC3DUXI7NH7SFIPLHTF4A, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.345487925 (615h55m29.21657096s) [noise=true , address=145.79.227.49:4370   , piece_id=TA4D4VCQXWYCB6FTZDRU6ZVZ4V6PZCFKQMA3EJYL5UEHHMI66JDQ, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.406150395 (615h55m29.27723343s) [noise=true , address=145.79.227.44:4228   , piece_id=YAQ7I3TOSLCYCHI2SJEEGCYGTEFWHX2CEYVZASRFL3BODTEONFTA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.414891660 (615h55m29.285974695s) [noise=true , address=193.149.164.143:4698 , piece_id=IYUHVRK3ZH3JADUJI627OUTHO7CRPNNKZVQTOCALMDYKVDGSHIJQ, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.431785668 (615h55m29.302868703s) [noise=true , address=145.79.227.40:4067   , piece_id=ITZ4SYUISESGVG72CBXHMAHZAD3XOOWYO5UUSFIT22R765N3W25A, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.431807049 (615h55m29.302890104s) [seg_pos={0 0}, upload=539756]: Segment reached optimal threshold of 14 pieces.
Dec 31 07:24:29.432061032 (615h55m29.303144087s) [noise=true , address=145.79.227.30:4784   , piece_id=M5NJQBX6JI5QI2PPA52B6A2CNLOXVWKYYYNX6CJDAA4WJNT4SHWA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pfXWex2WsuhmLb2smRpE46cHT9aipSTHW3e5oyELS1GhF1cFr): protocol: expected piece hash; context canceled; EOF
Dec 31 07:24:29.432072876 (615h55m29.303155931s) [noise=true , address=145.79.227.30:4784   , piece_id=M5NJQBX6JI5QI2PPA52B6A2CNLOXVWKYYYNX6CJDAA4WJNT4SHWA, seg_pos={0 0}, upload=539756]: Long tail cancellation due to optimal threshold
Dec 31 07:24:29.432090819 (615h55m29.303173884s) [noise=true , address=64.49.13.169:4709    , piece_id=CBAB3KCCPOW3P3LYM4T73CVDOD7FPJCPXYIXKDV7XTC2FKESTDRQ, seg_pos={0 0}, upload=539756]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:33942->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:33942->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33942->64.49.13.169:4709: use of closed network connection
Dec 31 07:24:29.432098063 (615h55m29.303181118s) [noise=true , address=64.49.13.169:4709    , piece_id=CBAB3KCCPOW3P3LYM4T73CVDOD7FPJCPXYIXKDV7XTC2FKESTDRQ, seg_pos={0 0}, upload=539756]: Long tail cancellation due to optimal threshold
Dec 31 07:24:29.432212572 (615h55m29.303295617s) [noise=true , address=64.49.13.169:4707    , piece_id=RB7LFR5DM7PUHCJPB5ABWN63Q2XQOGNOG2W2FCKXFWU5MGVPRPVA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:41264->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:41264->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41264->64.49.13.169:4707: use of closed network connection
Dec 31 07:24:29.432220747 (615h55m29.303303782s) [noise=true , address=64.49.13.169:4707    , piece_id=RB7LFR5DM7PUHCJPB5ABWN63Q2XQOGNOG2W2FCKXFWU5MGVPRPVA, seg_pos={0 0}, upload=539756]: Long tail cancellation due to optimal threshold
Dec 31 07:24:29.432465934 (615h55m29.303548979s) [noise=true , address=145.79.227.43:4186   , piece_id=ENVLK652WUVPKR3VPGXXMJ4IAEFHK67JOODS26ASTM4I2V7ABUWA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Gu3vXy2JUkKDaF4MkMCgVmYspW4Ez8EAfv6e4GeCkrwYTe5T): protocol: expected piece hash; context canceled; EOF
Dec 31 07:24:29.432474971 (615h55m29.303558006s) [noise=true , address=145.79.227.43:4186   , piece_id=ENVLK652WUVPKR3VPGXXMJ4IAEFHK67JOODS26ASTM4I2V7ABUWA, seg_pos={0 0}, upload=539756]: Long tail cancellation due to optimal threshold
Dec 31 07:24:29.432580261 (615h55m29.303663296s) [seg_pos={0 0}, upload=539756]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 07:24:29.432664502 (615h55m29.303747547s) [upload=539756]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 07:24:29.432671305 (615h55m29.30375435s) [upload=539756]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 07:24:29.717299118 (615h55m29.588382153s) [upload=539756]: Done uploading segments...
Dec 31 07:24:29.717320589 (615h55m29.588403624s) [upload=539756]: Upload finished. err: <nil>
Duration: 35.234683775s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
4 out of 60 is failed (threshold=6) [...X...X................................X.........X.........]
duration: 
5.001337779s
trace: 
https://jaeger.dev.storj.io/trace/6fc9bc24e7467571
FAIL (2025-12-31 07:49:13 UTC) 
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001367585s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
16 out of 60 is failed (threshold=3) [.X.....X....X.X.X.X.X......XXX.X....XXX..X...........X......]
duration: 
5.707365143s
trace: 
https://jaeger.dev.storj.io/trace/42e2703703b08f61
FAIL (2025-12-31 07:49:10 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: MI7XNXWQS44JIBIQGUUY7GKPSWTYZB24UBYBU2LG2FBDPHNJNXCA) context deadline exceeded; piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: DHJ6KXLSQJ7FFNCSBZ47YVFDAG6NGAQBQ6JUEHBJHRR2BHCEMJXQ) context deadline exceeded
======LOGS======
Dec 31 07:44:01.047566524 (616h15m0.918649569s) [upload=540041]: Starting upload
Dec 31 07:44:01.047641978 (616h15m0.918725023s) [upload=540041]: Done starting upload
Dec 31 07:44:01.047768658 (616h15m0.918851694s) [upload=540041]: Uploading segments...
Dec 31 07:44:01.047869851 (616h15m0.918952886s) [upload=540041]: Got next segment. Inline: false
Dec 31 07:44:01.047916280 (616h15m0.918999325s) [upload=540041]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 07:44:01.047921680 (616h15m0.919004716s) [upload=540041]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 07:44:01.295677183 (616h15m1.166760218s) [seg_pos={0 0}, upload=540041]: Begin upload segment...
Dec 31 07:44:01.295738318 (616h15m1.166821363s) [seg_pos={0 0}, upload=540041]: Done begin upload segment.
Dec 31 07:44:01.295797722 (616h15m1.166880767s) [noise=true , address=145.79.227.8:4330    , piece_id=LJYYIPAAZWWSTBR7ZVVS6OAJRBFGQ5EWMO2Q6CJXKSVGNU2ICI7A, seg_pos={0 0}, upload=540041]: Uploading piece...
Dec 31 07:44:01.295836014 (616h15m1.166919069s) [noise=true , address=145.79.227.39:4013   , piece_id=DP5ACORNEZQVBWUC457NI7Q2EP2QFD3DIREO42BPRYIOKNRHXFQA, seg_pos={0 0}, upload=540041]: Uploading piece...
Dec 31 07:44:01.296516851 (616h15m1.167599896s) [noise=true , address=77.90.31.15:4697     , piece_id=MI7XNXWQS44JIBIQGUUY7GKPSWTYZB24UBYBU2LG2FBDPHNJNXCA, seg_pos={0 0}, upload=540041]: Uploading piece...
Dec 31 07:44:01.296915800 (616h15m1.167998846s) [noise=true , address=145.79.227.45:4229   , piece_id=A7LCYJD4GAVKQQNXD5VCNZSJ3NKY5DLOX32TRZJ262GT764RGY7Q, seg_pos={0 0}, upload=540041]: Uploading piece...
Dec 31 07:44:01.297238384 (616h15m1.16832143s) [noise=true , address=145.79.227.11:4731   , piece_id=R3JT6ZZSKLVMHFK2YIKFBGIYIAIYK42OVG5ZFVI2EXWTDEDUFVSQ, seg_pos={0 0}, upload=540041]: Uploading piece...
Dec 31 07:44:01.297726454 (616h15m1.168809499s) [noise=true , address=145.79.227.16:4909   , piece_id=K4FG7MSNS624MN5Z6DQM32SFE6JEJ3GQR4IJRFIASB2ZBO2CTYXA, seg_pos={0 0}, upload=540041]: Uploading piece...
Dec 31 07:44:01.298552417 (616h15m1.169635452s) [noise=true , address=45.91.116.9:4261     , piece_id=OK5E4KMYBUCSSQJ556XELE52EZXVQUQNYFEC2DHPJHDTQY7T7E4Q, seg_pos={0 0}, upload=540041]: Uploading piece...
Dec 31 07:44:01.298873288 (616h15m1.169956333s) [noise=true , address=145.79.227.19:4077   , piece_id=P36DAY2YPYV5KLROS45JTCCSGURC45DYWDZPUARETHTIPWAPUYUA, seg_pos={0 0}, upload=540041]: Uploading piece...
Dec 31 07:44:01.299368290 (616h15m1.170451335s) [noise=true , address=45.91.116.5:4300     , piece_id=YFMQRE4CCU3SGUFPVWHR7LHAVCRK4JHLSJOTUUPT6HAX7I2P7P5Q, seg_pos={0 0}, upload=540041]: Uploading piece...
Dec 31 07:44:01.300865331 (616h15m1.171948376s) [noise=true , address=77.90.31.10:4517     , piece_id=DHJ6KXLSQJ7FFNCSBZ47YVFDAG6NGAQBQ6JUEHBJHRR2BHCEMJXQ, seg_pos={0 0}, upload=540041]: Uploading piece...
Dec 31 07:44:01.302083751 (616h15m1.173166786s) [noise=true , address=145.79.227.33:4871   , piece_id=YDYUCYFXL5O3AAGWSF2Q6NINXWN5TOQPWOM6IA5FEFZF4EHPQ2SQ, seg_pos={0 0}, upload=540041]: Uploading piece...
Dec 31 07:44:01.303232819 (616h15m1.174315854s) [noise=true , address=145.79.227.21:4474   , piece_id=BGH6HWHAO2PW2OCWV7ZAR2SRYWQKNRDC6EV77I76LEKX3FJA4ULQ, seg_pos={0 0}, upload=540041]: Uploading piece...
Dec 31 07:44:01.304941242 (616h15m1.176024287s) [noise=true , address=145.79.227.47:4266   , piece_id=GABATZO42ANXGKAWYNOWKFF7E5RM2JFH2EJRMBYLLCIRI7MIH6GA, seg_pos={0 0}, upload=540041]: Uploading piece...
Dec 31 07:44:01.306837263 (616h15m1.177920308s) [noise=true , address=145.79.227.6:4259    , piece_id=BQHFT2HP5NCKFJNSWFDH326SY23WOHH4KZYRNYLMP7NTN3STMYDQ, seg_pos={0 0}, upload=540041]: Uploading piece...
Dec 31 07:44:01.308130966 (616h15m1.179214051s) [noise=true , address=145.79.227.7:4295    , piece_id=HGW6AL3EVQUO6ZPYSGOUQIWKTSC66CRRL7NE2ZVHGQLVKQHJ6NJA, seg_pos={0 0}, upload=540041]: Uploading piece...
Dec 31 07:44:01.309279433 (616h15m1.180362478s) [noise=true , address=45.91.116.13:4156    , piece_id=J33TQGOIZUJ5L3EP2DRKL4VTGYMZRGX4U6OQVUMCMX3MVG64DODA, seg_pos={0 0}, upload=540041]: Uploading piece...
Dec 31 07:44:02.989139526 (616h15m2.860222581s) [upload=540041]: Next returned nil segment
Dec 31 07:44:02.989156668 (616h15m2.860239703s) [upload=540041]: Waiting for error group managing segments...
Dec 31 07:44:03.273227008 (616h15m3.144310043s) [noise=true , address=145.79.227.8:4330    , piece_id=LJYYIPAAZWWSTBR7ZVVS6OAJRBFGQ5EWMO2Q6CJXKSVGNU2ICI7A, seg_pos={0 0}, upload=540041]: Done uploading piece. err: <nil>
Dec 31 07:44:03.289928921 (616h15m3.161011996s) [noise=true , address=145.79.227.47:4266   , piece_id=GABATZO42ANXGKAWYNOWKFF7E5RM2JFH2EJRMBYLLCIRI7MIH6GA, seg_pos={0 0}, upload=540041]: Done uploading piece. err: <nil>
Dec 31 07:44:03.295170211 (616h15m3.166253256s) [noise=true , address=145.79.227.45:4229   , piece_id=A7LCYJD4GAVKQQNXD5VCNZSJ3NKY5DLOX32TRZJ262GT764RGY7Q, seg_pos={0 0}, upload=540041]: Done uploading piece. err: <nil>
Dec 31 07:44:03.299334931 (616h15m3.170417976s) [noise=true , address=145.79.227.11:4731   , piece_id=R3JT6ZZSKLVMHFK2YIKFBGIYIAIYK42OVG5ZFVI2EXWTDEDUFVSQ, seg_pos={0 0}, upload=540041]: Done uploading piece. err: <nil>
Dec 31 07:44:03.300607674 (616h15m3.171690719s) [noise=true , address=145.79.227.21:4474   , piece_id=BGH6HWHAO2PW2OCWV7ZAR2SRYWQKNRDC6EV77I76LEKX3FJA4ULQ, seg_pos={0 0}, upload=540041]: Done uploading piece. err: <nil>
Dec 31 07:44:03.307557488 (616h15m3.178640533s) [noise=true , address=145.79.227.19:4077   , piece_id=P36DAY2YPYV5KLROS45JTCCSGURC45DYWDZPUARETHTIPWAPUYUA, seg_pos={0 0}, upload=540041]: Done uploading piece. err: <nil>
Dec 31 07:44:03.310867621 (616h15m3.181950656s) [noise=true , address=145.79.227.39:4013   , piece_id=DP5ACORNEZQVBWUC457NI7Q2EP2QFD3DIREO42BPRYIOKNRHXFQA, seg_pos={0 0}, upload=540041]: Done uploading piece. err: <nil>
Dec 31 07:44:03.320003918 (616h15m3.191086963s) [noise=true , address=145.79.227.6:4259    , piece_id=BQHFT2HP5NCKFJNSWFDH326SY23WOHH4KZYRNYLMP7NTN3STMYDQ, seg_pos={0 0}, upload=540041]: Done uploading piece. err: <nil>
Dec 31 07:44:03.368788982 (616h15m3.239872027s) [noise=true , address=145.79.227.16:4909   , piece_id=K4FG7MSNS624MN5Z6DQM32SFE6JEJ3GQR4IJRFIASB2ZBO2CTYXA, seg_pos={0 0}, upload=540041]: Done uploading piece. err: <nil>
Dec 31 07:44:03.384248708 (616h15m3.255331754s) [noise=true , address=145.79.227.7:4295    , piece_id=HGW6AL3EVQUO6ZPYSGOUQIWKTSC66CRRL7NE2ZVHGQLVKQHJ6NJA, seg_pos={0 0}, upload=540041]: Done uploading piece. err: <nil>
Dec 31 07:44:03.570073366 (616h15m3.441156411s) [noise=true , address=77.90.31.10:4517     , piece_id=DHJ6KXLSQJ7FFNCSBZ47YVFDAG6NGAQBQ6JUEHBJHRR2BHCEMJXQ, seg_pos={0 0}, upload=540041]: Done uploading piece. err: <nil>
Dec 31 07:44:03.670704060 (616h15m3.541787105s) [noise=true , address=77.90.31.15:4697     , piece_id=MI7XNXWQS44JIBIQGUUY7GKPSWTYZB24UBYBU2LG2FBDPHNJNXCA, seg_pos={0 0}, upload=540041]: Done uploading piece. err: <nil>
Dec 31 07:44:03.670841321 (616h15m3.541924367s) [noise=true , address=45.91.116.5:4300     , piece_id=YFMQRE4CCU3SGUFPVWHR7LHAVCRK4JHLSJOTUUPT6HAX7I2P7P5Q, seg_pos={0 0}, upload=540041]: Done uploading piece. err: <nil>
Dec 31 07:44:03.670866620 (616h15m3.541949665s) [seg_pos={0 0}, upload=540041]: Segment reached optimal threshold of 13 pieces.
Dec 31 07:44:03.671093271 (616h15m3.542176316s) [noise=true , address=45.91.116.13:4156    , piece_id=J33TQGOIZUJ5L3EP2DRKL4VTGYMZRGX4U6OQVUMCMX3MVG64DODA, seg_pos={0 0}, upload=540041]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF
Dec 31 07:44:03.671102129 (616h15m3.542185174s) [noise=true , address=45.91.116.13:4156    , piece_id=J33TQGOIZUJ5L3EP2DRKL4VTGYMZRGX4U6OQVUMCMX3MVG64DODA, seg_pos={0 0}, upload=540041]: Long tail cancellation due to optimal threshold
Dec 31 07:44:03.671165218 (616h15m3.542248283s) [noise=true , address=145.79.227.33:4871   , piece_id=YDYUCYFXL5O3AAGWSF2Q6NINXWN5TOQPWOM6IA5FEFZF4EHPQ2SQ, seg_pos={0 0}, upload=540041]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF
Dec 31 07:44:03.671177732 (616h15m3.542260777s) [noise=true , address=145.79.227.33:4871   , piece_id=YDYUCYFXL5O3AAGWSF2Q6NINXWN5TOQPWOM6IA5FEFZF4EHPQ2SQ, seg_pos={0 0}, upload=540041]: Long tail cancellation due to optimal threshold
Dec 31 07:44:03.671375198 (616h15m3.542458243s) [noise=true , address=45.91.116.9:4261     , piece_id=OK5E4KMYBUCSSQJ556XELE52EZXVQUQNYFEC2DHPJHDTQY7T7E4Q, seg_pos={0 0}, upload=540041]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF
Dec 31 07:44:03.671384326 (616h15m3.542467371s) [noise=true , address=45.91.116.9:4261     , piece_id=OK5E4KMYBUCSSQJ556XELE52EZXVQUQNYFEC2DHPJHDTQY7T7E4Q, seg_pos={0 0}, upload=540041]: Long tail cancellation due to optimal threshold
Dec 31 07:44:03.671411457 (616h15m3.542494502s) [seg_pos={0 0}, upload=540041]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 07:44:03.671502591 (616h15m3.542585636s) [upload=540041]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 07:44:03.671509173 (616h15m3.542592218s) [upload=540041]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 07:44:03.975595416 (616h15m3.846678461s) [upload=540041]: Done uploading segments...
Dec 31 07:44:03.975618651 (616h15m3.846701686s) [upload=540041]: Upload finished. err: <nil>
Duration: 5.707393186s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)
antiblip: 
1 out of 60 is failed (threshold=3) [....................................................X.......]
duration: 
3.4112501s
trace: 
https://jaeger.dev.storj.io/trace/59c3b821b75eba3a
FAIL (2025-12-31 07:49:00 UTC) 
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded
Duration: 3.411274356s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
2 out of 60 is failed (threshold=3) [...............X............X...............................]
duration: 
33.940787682s
trace: 
https://jaeger.dev.storj.io/trace/6ce9bac61d803c5a
FAIL (2025-12-31 07:48:55 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
56     , piece_id=3PKC4JZPZVBQOBUHPQAG4HZ67DRTOY7GR35QVWLV3LN5JHXPMENA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.554293132 (615h55m26.425376177s) [noise=true , address=145.79.227.43:4186   , piece_id=ENVLK652WUVPKR3VPGXXMJ4IAEFHK67JOODS26ASTM4I2V7ABUWA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.554505907 (615h55m26.425588942s) [noise=true , address=77.90.31.14:4673     , piece_id=XYDCDFNKMWJK4HDELS4NLBVLUBR76YHT4RP3M5JO5QMCSI6MOGYA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.554648990 (615h55m26.425732025s) [noise=true , address=64.49.13.140:4505    , piece_id=EBEDU3F3QMDLNIDSRRXUHGSZ3VH5ITKRDDI2QS5XKBOGKOCN3LNA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.554994317 (615h55m26.426077352s) [noise=true , address=64.49.13.169:4707    , piece_id=RB7LFR5DM7PUHCJPB5ABWN63Q2XQOGNOG2W2FCKXFWU5MGVPRPVA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.555494150 (615h55m26.426577185s) [noise=true , address=64.49.13.137:4500    , piece_id=DD4JKFRHB2FKWAOTS5RZU5M3ASDGJTRX2XRB5ZROVOZNPN6BER2Q, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.555978812 (615h55m26.427061847s) [noise=true , address=45.91.116.10:4087    , piece_id=MJUIJFY6KEXEMMQKWA2NC3RE5PG55QCDZ7ECWRZG7CBROSA7LL3Q, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.556325282 (615h55m26.427408317s) [noise=true , address=145.79.227.29:4965   , piece_id=L6L2H7KMBTYKC24ZMACWRYK6TGOK5KHDO2YCFP5TJS66NXXP3KNQ, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.556728720 (615h55m26.427811765s) [noise=true , address=45.91.116.6:4977     , piece_id=G7JKD7INUS3D3RUIHSSVYCSOVTHSRG5HLVKERSCAD5NUWSFRZ2EA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.557164801 (615h55m26.428247836s) [noise=true , address=145.79.227.49:4370   , piece_id=TA4D4VCQXWYCB6FTZDRU6ZVZ4V6PZCFKQMA3EJYL5UEHHMI66JDQ, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.557530657 (615h55m26.428613692s) [noise=true , address=145.79.227.33:4896   , piece_id=TCDN3ER54RKZQ73BEI6DCGQW3IUNJXNNC3DUXI7NH7SFIPLHTF4A, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.557963953 (615h55m26.429046998s) [noise=true , address=77.90.31.12:4620     , piece_id=AQOFDODGJ7TMWA3ZQGWJW7XZT3FSKK3GR4R4RSIFIMHYP3HVJQDA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.558357422 (615h55m26.429440457s) [noise=true , address=145.79.227.40:4067   , piece_id=ITZ4SYUISESGVG72CBXHMAHZAD3XOOWYO5UUSFIT22R765N3W25A, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.558648285 (615h55m26.42973133s) [noise=true , address=145.79.227.2:4147    , piece_id=SHUFD6UKDFM54RGOZGJEUOUYCMKLAN2IMLWKJXZIZ5PAZVNUE4JQ, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:26.559228450 (615h55m26.430311485s) [noise=true , address=145.79.227.44:4228   , piece_id=YAQ7I3TOSLCYCHI2SJEEGCYGTEFWHX2CEYVZASRFL3BODTEONFTA, seg_pos={0 0}, upload=539756]: Uploading piece...
Dec 31 07:24:27.974040599 (615h55m27.845123634s) [upload=539756]: Next returned nil segment
Dec 31 07:24:27.974074934 (615h55m27.845157969s) [upload=539756]: Waiting for error group managing segments...
Dec 31 07:24:28.334073958 (615h55m28.205156993s) [noise=true , address=64.49.13.140:4505    , piece_id=EBEDU3F3QMDLNIDSRRXUHGSZ3VH5ITKRDDI2QS5XKBOGKOCN3LNA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.427413642 (615h55m28.298496677s) [noise=true , address=145.79.227.2:4147    , piece_id=SHUFD6UKDFM54RGOZGJEUOUYCMKLAN2IMLWKJXZIZ5PAZVNUE4JQ, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.522823096 (615h55m28.393906131s) [noise=true , address=145.79.227.29:4965   , piece_id=L6L2H7KMBTYKC24ZMACWRYK6TGOK5KHDO2YCFP5TJS66NXXP3KNQ, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.585931513 (615h55m28.457014558s) [noise=true , address=77.90.31.13:4656     , piece_id=3PKC4JZPZVBQOBUHPQAG4HZ67DRTOY7GR35QVWLV3LN5JHXPMENA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.591579187 (615h55m28.462662262s) [noise=true , address=64.49.13.137:4500    , piece_id=DD4JKFRHB2FKWAOTS5RZU5M3ASDGJTRX2XRB5ZROVOZNPN6BER2Q, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.615175368 (615h55m28.486258413s) [noise=true , address=77.90.31.14:4673     , piece_id=XYDCDFNKMWJK4HDELS4NLBVLUBR76YHT4RP3M5JO5QMCSI6MOGYA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.699191016 (615h55m28.570274051s) [noise=true , address=45.91.116.6:4977     , piece_id=G7JKD7INUS3D3RUIHSSVYCSOVTHSRG5HLVKERSCAD5NUWSFRZ2EA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.713295994 (615h55m28.58437908s) [noise=true , address=45.91.116.10:4087    , piece_id=MJUIJFY6KEXEMMQKWA2NC3RE5PG55QCDZ7ECWRZG7CBROSA7LL3Q, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:28.794645326 (615h55m28.665728361s) [noise=true , address=77.90.31.12:4620     , piece_id=AQOFDODGJ7TMWA3ZQGWJW7XZT3FSKK3GR4R4RSIFIMHYP3HVJQDA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.341891306 (615h55m29.212974351s) [noise=true , address=145.79.227.33:4896   , piece_id=TCDN3ER54RKZQ73BEI6DCGQW3IUNJXNNC3DUXI7NH7SFIPLHTF4A, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.345487925 (615h55m29.21657096s) [noise=true , address=145.79.227.49:4370   , piece_id=TA4D4VCQXWYCB6FTZDRU6ZVZ4V6PZCFKQMA3EJYL5UEHHMI66JDQ, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.406150395 (615h55m29.27723343s) [noise=true , address=145.79.227.44:4228   , piece_id=YAQ7I3TOSLCYCHI2SJEEGCYGTEFWHX2CEYVZASRFL3BODTEONFTA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.414891660 (615h55m29.285974695s) [noise=true , address=193.149.164.143:4698 , piece_id=IYUHVRK3ZH3JADUJI627OUTHO7CRPNNKZVQTOCALMDYKVDGSHIJQ, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.431785668 (615h55m29.302868703s) [noise=true , address=145.79.227.40:4067   , piece_id=ITZ4SYUISESGVG72CBXHMAHZAD3XOOWYO5UUSFIT22R765N3W25A, seg_pos={0 0}, upload=539756]: Done uploading piece. err: <nil>
Dec 31 07:24:29.431807049 (615h55m29.302890104s) [seg_pos={0 0}, upload=539756]: Segment reached optimal threshold of 14 pieces.
Dec 31 07:24:29.432061032 (615h55m29.303144087s) [noise=true , address=145.79.227.30:4784   , piece_id=M5NJQBX6JI5QI2PPA52B6A2CNLOXVWKYYYNX6CJDAA4WJNT4SHWA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pfXWex2WsuhmLb2smRpE46cHT9aipSTHW3e5oyELS1GhF1cFr): protocol: expected piece hash; context canceled; EOF
Dec 31 07:24:29.432072876 (615h55m29.303155931s) [noise=true , address=145.79.227.30:4784   , piece_id=M5NJQBX6JI5QI2PPA52B6A2CNLOXVWKYYYNX6CJDAA4WJNT4SHWA, seg_pos={0 0}, upload=539756]: Long tail cancellation due to optimal threshold
Dec 31 07:24:29.432090819 (615h55m29.303173884s) [noise=true , address=64.49.13.169:4709    , piece_id=CBAB3KCCPOW3P3LYM4T73CVDOD7FPJCPXYIXKDV7XTC2FKESTDRQ, seg_pos={0 0}, upload=539756]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:33942->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:33942->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33942->64.49.13.169:4709: use of closed network connection
Dec 31 07:24:29.432098063 (615h55m29.303181118s) [noise=true , address=64.49.13.169:4709    , piece_id=CBAB3KCCPOW3P3LYM4T73CVDOD7FPJCPXYIXKDV7XTC2FKESTDRQ, seg_pos={0 0}, upload=539756]: Long tail cancellation due to optimal threshold
Dec 31 07:24:29.432212572 (615h55m29.303295617s) [noise=true , address=64.49.13.169:4707    , piece_id=RB7LFR5DM7PUHCJPB5ABWN63Q2XQOGNOG2W2FCKXFWU5MGVPRPVA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:41264->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:41264->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41264->64.49.13.169:4707: use of closed network connection
Dec 31 07:24:29.432220747 (615h55m29.303303782s) [noise=true , address=64.49.13.169:4707    , piece_id=RB7LFR5DM7PUHCJPB5ABWN63Q2XQOGNOG2W2FCKXFWU5MGVPRPVA, seg_pos={0 0}, upload=539756]: Long tail cancellation due to optimal threshold
Dec 31 07:24:29.432465934 (615h55m29.303548979s) [noise=true , address=145.79.227.43:4186   , piece_id=ENVLK652WUVPKR3VPGXXMJ4IAEFHK67JOODS26ASTM4I2V7ABUWA, seg_pos={0 0}, upload=539756]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Gu3vXy2JUkKDaF4MkMCgVmYspW4Ez8EAfv6e4GeCkrwYTe5T): protocol: expected piece hash; context canceled; EOF
Dec 31 07:24:29.432474971 (615h55m29.303558006s) [noise=true , address=145.79.227.43:4186   , piece_id=ENVLK652WUVPKR3VPGXXMJ4IAEFHK67JOODS26ASTM4I2V7ABUWA, seg_pos={0 0}, upload=539756]: Long tail cancellation due to optimal threshold
Dec 31 07:24:29.432580261 (615h55m29.303663296s) [seg_pos={0 0}, upload=539756]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 07:24:29.432664502 (615h55m29.303747547s) [upload=539756]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 07:24:29.432671305 (615h55m29.30375435s) [upload=539756]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 07:24:29.717299118 (615h55m29.588382153s) [upload=539756]: Done uploading segments...
Dec 31 07:24:29.717320589 (615h55m29.588403624s) [upload=539756]: Upload finished. err: <nil>
Duration: 33.940813881s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [...................................................X........]
duration: 
6.382469478s
trace: 
https://jaeger.dev.storj.io/trace/5dd0609e3a2ac547
FAIL (2025-12-31 07:48:39 UTC) 
orbiter warning: context deadline exceeded
Duration: 6.382496459s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)
antiblip: 
1 out of 60 is failed (threshold=3) [.....................................................X......]
duration: 
3.920860423s
trace: 
https://jaeger.dev.storj.io/trace/59c3b821b75eba3a
FAIL (2025-12-31 07:48:31 UTC) 
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded
Duration: 3.920889239s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
4 out of 60 is failed (threshold=6) [....X...X................................X.........X........]
duration: 
3.530681275s
trace: 
https://jaeger.dev.storj.io/trace/6fc9bc24e7467571
FAIL (2025-12-31 07:48:29 UTC) 
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.530735099s core uplink us-select-1 us1-placement-20