Orbiter History

Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure

Orbiter Check History

Showing 50 of 899292 reports (Page 11349 of 17986)

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 19 out of 60 is failed (threshold=6) [XX....X.X.X...X..X.X.X..X...XX.....X......XX...X.X....X.X...] duration: 5.00060139s trace: https://jaeger.dev.storj.io/trace/31d935b4c9de8559 FAIL (2025-12-28 04:16:08 UTC) orbiter warning: context deadline exceeded Duration: 5.000625826s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................................X......X.....] duration: 31.147082247s trace: https://jaeger.dev.storj.io/trace/26c246d73008cc49 FAIL (2025-12-28 04:15:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== oise=true , address=145.79.227.43:4191 , piece_id=CLYBACZJDTAFTBEMEMSOXP4I22PDYSVIBX3AMNN4DISQUJWYFDMQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.179395486 (540h42m11.050478531s) [noise=true , address=145.79.227.9:4371 , piece_id=2ON3P6ZFEUEBFFLCHYY5BWAAJEFX43KKU5MU5TLBQHSZ52RCAYLA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.179823281 (540h42m11.050906336s) [noise=true , address=77.90.31.18:4819 , piece_id=FVZE5P55HHT3RGOI4M2OFV6NER4BNVGFSZOMDRADO5JYI54DRHLQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.180232490 (540h42m11.051315535s) [noise=true , address=77.90.31.17:4778 , piece_id=WHTKIVKEZNFREOTNDTLLBB24JGWKIXWBJKZ2EUM3QL2SM7Z4YXDA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.180945087 (540h42m11.052028132s) [noise=true , address=45.91.116.6:4003 , piece_id=H4VWZIMYVM2M3SX76LSS64ARSENBQL6OOTL5EY5XE4IAORO7OWGQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181189232 (540h42m11.052272277s) [noise=true , address=77.90.31.21:4906 , piece_id=SGYCZ2E6IMEJA332PTI7DQ4FWRJR6NG4OYRUP6H2KHXS2TKC5S2A, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181508139 (540h42m11.052591174s) [noise=true , address=77.90.31.15:4714 , piece_id=YVXSR2B74BDK4BV5QGLZLWOKVXV3TWKSW5M3KG323U7VUUBIQU3Q, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181597209 (540h42m11.052680254s) [noise=true , address=145.79.227.11:4755 , piece_id=F37J7GC6RB2PL3B4B6MFVAPB6IOGSJSKVQ5UNFHFSBJ7C3SABURQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181882131 (540h42m11.052965166s) [noise=true , address=145.79.227.33:4876 , piece_id=UEOY3HYI3WWJTXUSBALBXIGCMBC5M6XGR42SIOBXQNDCZ3UFMWVA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182132017 (540h42m11.053215052s) [noise=true , address=145.79.227.40:4053 , piece_id=V7R5MTCCKCGHV7KD4LAXFBUJI3HRGVAJUCVYL4WPL3M32CVQXOBA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182274007 (540h42m11.053357052s) [noise=true , address=145.79.227.51:4432 , piece_id=C5P63KNNZLSOBH772MRMYOJOQGLE7KXGELQN437HTVPAM72RXUCA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182451545 (540h42m11.05353458s) [noise=true , address=145.79.227.30:4784 , piece_id=UTH47JZWG3SGX3KXBFJJALALTPP6INYORUHDYSS6CNDEUQ6BDL7A, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182777516 (540h42m11.053860551s) [noise=true , address=64.49.13.169:4708 , piece_id=6OBO4M26AXDJXUTOBI66S2JJARDEM2CTTFHUIAOI4CX2PZITRWFQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182919286 (540h42m11.054002331s) [noise=true , address=145.79.227.6:4267 , piece_id=SDG3WH6TPR47OHG57NMVFAHOYVAL45XMJ7S22FZL6IFR6DS45HQQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.183101804 (540h42m11.054184839s) [noise=true , address=145.79.227.34:4924 , piece_id=R7MT4YS5IPGHOCNZX77ITOPXL57DOYD2HJXPJYYBGNVUWOT33JPA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.183514740 (540h42m11.054597785s) [noise=true , address=193.149.164.143:4696 , piece_id=OJQFP4YJ3EF52S2XQN5QPKMQJDS57QEOQQ4PUPARPGPAVSLGKMFA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:12.562530634 (540h42m12.433613679s) [upload=474194]: Next returned nil segment Dec 28 04:11:12.562544100 (540h42m12.433627135s) [upload=474194]: Waiting for error group managing segments... Dec 28 04:11:13.045899872 (540h42m12.916982917s) [noise=true , address=145.79.227.9:4371 , piece_id=2ON3P6ZFEUEBFFLCHYY5BWAAJEFX43KKU5MU5TLBQHSZ52RCAYLA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.068300556 (540h42m12.939383591s) [noise=true , address=145.79.227.11:4755 , piece_id=F37J7GC6RB2PL3B4B6MFVAPB6IOGSJSKVQ5UNFHFSBJ7C3SABURQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.073683536 (540h42m12.944766571s) [noise=true , address=145.79.227.40:4053 , piece_id=V7R5MTCCKCGHV7KD4LAXFBUJI3HRGVAJUCVYL4WPL3M32CVQXOBA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.125067800 (540h42m12.996150835s) [noise=true , address=145.79.227.6:4267 , piece_id=SDG3WH6TPR47OHG57NMVFAHOYVAL45XMJ7S22FZL6IFR6DS45HQQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.156395944 (540h42m13.027478989s) [noise=true , address=145.79.227.51:4432 , piece_id=C5P63KNNZLSOBH772MRMYOJOQGLE7KXGELQN437HTVPAM72RXUCA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.221620070 (540h42m13.092703105s) [noise=true , address=77.90.31.15:4714 , piece_id=YVXSR2B74BDK4BV5QGLZLWOKVXV3TWKSW5M3KG323U7VUUBIQU3Q, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.266169559 (540h42m13.137252594s) [noise=true , address=145.79.227.33:4876 , piece_id=UEOY3HYI3WWJTXUSBALBXIGCMBC5M6XGR42SIOBXQNDCZ3UFMWVA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.273253176 (540h42m13.144336221s) [noise=true , address=77.90.31.12:4604 , piece_id=DVQ53JEZL2GDTXFDDIWN4BLSJJHRZORII7UNF5HS3GIM5VJX3Y2Q, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.324930308 (540h42m13.196013343s) [noise=true , address=77.90.31.17:4778 , piece_id=WHTKIVKEZNFREOTNDTLLBB24JGWKIXWBJKZ2EUM3QL2SM7Z4YXDA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.333001758 (540h42m13.204084793s) [noise=true , address=77.90.31.18:4819 , piece_id=FVZE5P55HHT3RGOI4M2OFV6NER4BNVGFSZOMDRADO5JYI54DRHLQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.418239402 (540h42m13.289322437s) [noise=true , address=45.91.116.6:4003 , piece_id=H4VWZIMYVM2M3SX76LSS64ARSENBQL6OOTL5EY5XE4IAORO7OWGQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.460099681 (540h42m13.331182726s) [noise=true , address=77.90.31.21:4906 , piece_id=SGYCZ2E6IMEJA332PTI7DQ4FWRJR6NG4OYRUP6H2KHXS2TKC5S2A, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.860538777 (540h42m13.731621812s) [noise=true , address=193.149.164.143:4696 , piece_id=OJQFP4YJ3EF52S2XQN5QPKMQJDS57QEOQQ4PUPARPGPAVSLGKMFA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:14.127606583 (540h42m13.998689618s) [noise=true , address=145.79.227.34:4924 , piece_id=R7MT4YS5IPGHOCNZX77ITOPXL57DOYD2HJXPJYYBGNVUWOT33JPA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:14.127625258 (540h42m13.998708303s) [seg_pos={0 0}, upload=474194]: Segment reached optimal threshold of 14 pieces. Dec 28 04:11:14.127786726 (540h42m13.998869761s) [noise=true , address=45.91.116.4:4966 , piece_id=ZTHC25VIBEHSKRULTNAS25DK2DBVI7ATO46SDMMTON7PBBPRMWBQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DSHTtpwcx1h1utGXNou11K7Tcg1ZhkkMdMgY6BDD2ifG6LQ7S): protocol: expected piece hash; context canceled; EOF Dec 28 04:11:14.127792547 (540h42m13.998875582s) [noise=true , address=45.91.116.4:4966 , piece_id=ZTHC25VIBEHSKRULTNAS25DK2DBVI7ATO46SDMMTON7PBBPRMWBQ, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128017335 (540h42m13.99910037s) [noise=true , address=145.79.227.43:4191 , piece_id=CLYBACZJDTAFTBEMEMSOXP4I22PDYSVIBX3AMNN4DISQUJWYFDMQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw): protocol: expected piece hash; context canceled; EOF Dec 28 04:11:14.128023747 (540h42m13.999106782s) [noise=true , address=145.79.227.43:4191 , piece_id=CLYBACZJDTAFTBEMEMSOXP4I22PDYSVIBX3AMNN4DISQUJWYFDMQ, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128139027 (540h42m13.999222102s) [noise=true , address=64.49.13.169:4708 , piece_id=6OBO4M26AXDJXUTOBI66S2JJARDEM2CTTFHUIAOI4CX2PZITRWFQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:40936->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:40936->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40936->64.49.13.169:4708: use of closed network connection Dec 28 04:11:14.128150668 (540h42m13.999233733s) [noise=true , address=64.49.13.169:4708 , piece_id=6OBO4M26AXDJXUTOBI66S2JJARDEM2CTTFHUIAOI4CX2PZITRWFQ, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128292198 (540h42m13.999375243s) [noise=true , address=145.79.227.30:4784 , piece_id=UTH47JZWG3SGX3KXBFJJALALTPP6INYORUHDYSS6CNDEUQ6BDL7A, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pfXWex2WsuhmLb2smRpE46cHT9aipSTHW3e5oyELS1GhF1cFr): protocol: expected piece hash; context canceled; EOF Dec 28 04:11:14.128299212 (540h42m13.999382257s) [noise=true , address=145.79.227.30:4784 , piece_id=UTH47JZWG3SGX3KXBFJJALALTPP6INYORUHDYSS6CNDEUQ6BDL7A, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128395815 (540h42m13.99947885s) [seg_pos={0 0}, upload=474194]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 04:11:14.128488171 (540h42m13.999571206s) [upload=474194]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:11:14.128493561 (540h42m13.999576597s) [upload=474194]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:11:14.572773387 (540h42m14.443856452s) [upload=474194]: Done uploading segments... Dec 28 04:11:14.572800389 (540h42m14.443883454s) [upload=474194]: Upload finished. err: <nil> Duration: 31.147110471s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 50 out of 60 is failed (threshold=3) [.XXXXX.XXXXXXXXXXXXXXXXX..XXXX.XXXX.XXXXX.XXXXXXX.XXXXXX..XX] duration: 30.002240145s trace: https://jaeger.dev.storj.io/trace/5664ebcb754d3ee7 FAIL (2025-12-28 04:15:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: RIRQKW7KTWM4IFI4KCAW4LPGW5QMX625TPBUF5BCQJPFKVNXEBRQ) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: IDWNBC6CCR4MHQP2ZDLAVTPXOFPGHSNZJSEQSLRG6P3RWEU5HUMA) context deadline exceeded ======LOGS====== Dec 28 04:15:22.159368766 (540h46m22.030451811s) [upload=474254]: Starting upload Dec 28 04:15:22.159447656 (540h46m22.030530701s) [upload=474254]: Done starting upload Dec 28 04:15:22.159484607 (540h46m22.030567652s) [upload=474254]: Uploading segments... Dec 28 04:15:22.159642086 (540h46m22.030725131s) [upload=474254]: Got next segment. Inline: false Dec 28 04:15:22.159689145 (540h46m22.03077218s) [upload=474254]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 04:15:22.159693895 (540h46m22.03077693s) [upload=474254]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 04:15:22.399656400 (540h46m22.270739455s) [seg_pos={0 0}, upload=474254]: Begin upload segment... Dec 28 04:15:22.399726483 (540h46m22.270809529s) [seg_pos={0 0}, upload=474254]: Done begin upload segment. Dec 28 04:15:22.399745751 (540h46m22.270828786s) [noise=true , address=145.79.227.44:4195 , piece_id=ZFJF4A5OOEBZ3TBDPJUR73CKGRUDDVBL3CYDDSSLJNVXUQXIAUOA, seg_pos={0 0}, upload=474254]: Uploading piece... Dec 28 04:15:22.399776369 (540h46m22.270859404s) [noise=true , address=145.79.227.45:4229 , piece_id=FE7GMMC2AS6EWYNO3IJ2TXFZMTCAZWB2CTTGXUPLSIJGIMVBVJ2Q, seg_pos={0 0}, upload=474254]: Uploading piece... Dec 28 04:15:22.400425284 (540h46m22.271508329s) [noise=true , address=45.91.116.7:4009 , piece_id=Q4YTMPS3CQAIGIB3N7XRUXWZX6GTS7V6VOMV5N26QCKKLZTQSIJQ, seg_pos={0 0}, upload=474254]: Uploading piece... Dec 28 04:15:22.400849151 (540h46m22.271932196s) [noise=true , address=145.79.227.14:4840 , piece_id=N7OVKA6VVN7ABMF2JUDTHOADPMHHCHZWRV6GZJ2AAGV54KFCIAGQ, seg_pos={0 0}, upload=474254]: Uploading piece... Dec 28 04:15:22.401251768 (540h46m22.272334813s) [noise=true , address=45.91.116.14:4190 , piece_id=T3XUE7GYF2TBOCRHBLBNKRJMLINB3Q3YHRMUBNTVOSBJEPQNTFTA, seg_pos={0 0}, upload=474254]: Uploading piece... Dec 28 04:15:22.401779232 (540h46m22.272862277s) [noise=true , address=145.79.227.36:4517 , piece_id=BIJFT4L35MBRVYL6A2MJIDIZ3GLLK3IHR6BDLCOYOU465TKF25MQ, seg_pos={0 0}, upload=474254]: Uploading piece... Dec 28 04:15:22.402306045 (540h46m22.27338909s) [noise=true , address=145.79.227.51:4410 , piece_id=WTIJBCOEKWB5J3Q6J2ODCXGDURWN5H4HOOLXWXMHNCVNRNMDGTSA, seg_pos={0 0}, upload=474254]: Uploading piece... Dec 28 04:15:22.402935674 (540h46m22.274018719s) [noise=true , address=77.90.31.18:4805 , piece_id=7KLWALFRG3IIOC5ZXDKELYBSCQPMK2A7MDB6QTT3PZJGGAVDM2EQ, seg_pos={0 0}, upload=474254]: Uploading piece... Dec 28 04:15:22.404587440 (540h46m22.275670485s) [noise=true , address=145.79.227.31:4798 , piece_id=IAJAHXHUGQ6E77XXYHQ475FBD2LBZ6R73LDZIOBAX5D6KS3MYF7Q, seg_pos={0 0}, upload=474254]: Uploading piece... Dec 28 04:15:22.406259113 (540h46m22.277342188s) [noise=true , address=145.79.227.16:4909 , piece_id=R7R5LTAVZP2YTLJMFKP3R3GAHEFHCVYCCHEBN3KYR2F6YBM4BQQQ, seg_pos={0 0}, upload=474254]: Uploading piece... Dec 28 04:15:22.408022049 (540h46m22.279105094s) [noise=true , address=45.91.116.5:4298 , piece_id=GDR7XRSOXSLH6CRF4TW6SIRGQVZ6IF2K22WGKPLL7DUHN6OHZV5A, seg_pos={0 0}, upload=474254]: Uploading piece... Dec 28 04:15:22.411944469 (540h46m22.283027514s) [noise=true , address=77.90.31.11:4556 , piece_id=622EXS2QCP45LJCTXZRCQNUW7A5GVSFMWOAX7VVTON5CPK6WYU4Q, seg_pos={0 0}, upload=474254]: Uploading piece... Dec 28 04:15:22.413402886 (540h46m22.284485921s) [noise=true , address=145.79.227.35:4481 , piece_id=PLIUZBASFOGT33757LA2JAK3QSHLI6HZFY7LIBMZ2AXZRACVRCXA, seg_pos={0 0}, upload=474254]: Uploading piece... Dec 28 04:15:22.415970906 (540h46m22.287053951s) [noise=true , address=45.91.116.11:4117 , piece_id=RIRQKW7KTWM4IFI4KCAW4LPGW5QMX625TPBUF5BCQJPFKVNXEBRQ, seg_pos={0 0}, upload=474254]: Uploading piece... Dec 28 04:15:22.418097985 (540h46m22.28918102s) [noise=true , address=45.91.116.6:4974 , piece_id=IDWNBC6CCR4MHQP2ZDLAVTPXOFPGHSNZJSEQSLRG6P3RWEU5HUMA, seg_pos={0 0}, upload=474254]: Uploading piece... Dec 28 04:15:22.419710286 (540h46m22.290793331s) [noise=true , address=145.79.227.17:4971 , piece_id=C7LEHPQEPTBLKUBLEFH7G2EN4XZ45IZ5U5HQSZD35H4Q43DQDKUQ, seg_pos={0 0}, upload=474254]: Uploading piece... Dec 28 04:15:24.129574366 (540h46m24.000657411s) [upload=474254]: Next returned nil segment Dec 28 04:15:24.129592159 (540h46m24.000675204s) [upload=474254]: Waiting for error group managing segments... Dec 28 04:15:24.422818516 (540h46m24.293901561s) [noise=true , address=145.79.227.35:4481 , piece_id=PLIUZBASFOGT33757LA2JAK3QSHLI6HZFY7LIBMZ2AXZRACVRCXA, seg_pos={0 0}, upload=474254]: Done uploading piece. err: <nil> Dec 28 04:15:24.422859875 (540h46m24.29394291s) [noise=true , address=145.79.227.17:4971 , piece_id=C7LEHPQEPTBLKUBLEFH7G2EN4XZ45IZ5U5HQSZD35H4Q43DQDKUQ, seg_pos={0 0}, upload=474254]: Done uploading piece. err: <nil> Dec 28 04:15:24.430133435 (540h46m24.30121648s) [noise=true , address=145.79.227.36:4517 , piece_id=BIJFT4L35MBRVYL6A2MJIDIZ3GLLK3IHR6BDLCOYOU465TKF25MQ, seg_pos={0 0}, upload=474254]: Done uploading piece. err: <nil> Dec 28 04:15:24.442861801 (540h46m24.313944836s) [noise=true , address=145.79.227.51:4410 , piece_id=WTIJBCOEKWB5J3Q6J2ODCXGDURWN5H4HOOLXWXMHNCVNRNMDGTSA, seg_pos={0 0}, upload=474254]: Done uploading piece. err: <nil> Dec 28 04:15:24.446893889 (540h46m24.317976934s) [noise=true , address=145.79.227.16:4909 , piece_id=R7R5LTAVZP2YTLJMFKP3R3GAHEFHCVYCCHEBN3KYR2F6YBM4BQQQ, seg_pos={0 0}, upload=474254]: Done uploading piece. err: <nil> Dec 28 04:15:24.448795891 (540h46m24.319878936s) [noise=true , address=145.79.227.45:4229 , piece_id=FE7GMMC2AS6EWYNO3IJ2TXFZMTCAZWB2CTTGXUPLSIJGIMVBVJ2Q, seg_pos={0 0}, upload=474254]: Done uploading piece. err: <nil> Dec 28 04:15:24.464542014 (540h46m24.335625059s) [noise=true , address=145.79.227.14:4840 , piece_id=N7OVKA6VVN7ABMF2JUDTHOADPMHHCHZWRV6GZJ2AAGV54KFCIAGQ, seg_pos={0 0}, upload=474254]: Done uploading piece. err: <nil> Dec 28 04:15:24.470184829 (540h46m24.341267864s) [noise=true , address=145.79.227.31:4798 , piece_id=IAJAHXHUGQ6E77XXYHQ475FBD2LBZ6R73LDZIOBAX5D6KS3MYF7Q, seg_pos={0 0}, upload=474254]: Done uploading piece. err: <nil> Dec 28 04:15:24.582423960 (540h46m24.453507045s) [noise=true , address=77.90.31.11:4556 , piece_id=622EXS2QCP45LJCTXZRCQNUW7A5GVSFMWOAX7VVTON5CPK6WYU4Q, seg_pos={0 0}, upload=474254]: Done uploading piece. err: <nil> Dec 28 04:15:24.644540368 (540h46m24.515623414s) [noise=true , address=45.91.116.14:4190 , piece_id=T3XUE7GYF2TBOCRHBLBNKRJMLINB3Q3YHRMUBNTVOSBJEPQNTFTA, seg_pos={0 0}, upload=474254]: Done uploading piece. err: <nil> Dec 28 04:15:24.648302873 (540h46m24.519385918s) [noise=true , address=45.91.116.11:4117 , piece_id=RIRQKW7KTWM4IFI4KCAW4LPGW5QMX625TPBUF5BCQJPFKVNXEBRQ, seg_pos={0 0}, upload=474254]: Done uploading piece. err: <nil> Dec 28 04:15:24.660419203 (540h46m24.531502248s) [noise=true , address=45.91.116.6:4974 , piece_id=IDWNBC6CCR4MHQP2ZDLAVTPXOFPGHSNZJSEQSLRG6P3RWEU5HUMA, seg_pos={0 0}, upload=474254]: Done uploading piece. err: <nil> Dec 28 04:15:24.696428561 (540h46m24.567511596s) [noise=true , address=77.90.31.18:4805 , piece_id=7KLWALFRG3IIOC5ZXDKELYBSCQPMK2A7MDB6QTT3PZJGGAVDM2EQ, seg_pos={0 0}, upload=474254]: Done uploading piece. err: <nil> Dec 28 04:15:24.696451575 (540h46m24.56753461s) [seg_pos={0 0}, upload=474254]: Segment reached optimal threshold of 13 pieces. Dec 28 04:15:24.696652267 (540h46m24.567735312s) [noise=true , address=45.91.116.5:4298 , piece_id=GDR7XRSOXSLH6CRF4TW6SIRGQVZ6IF2K22WGKPLL7DUHN6OHZV5A, seg_pos={0 0}, upload=474254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:42464->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:42464->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42464->45.91.116.5:4298: use of closed network connection Dec 28 04:15:24.696658729 (540h46m24.567741774s) [noise=true , address=45.91.116.5:4298 , piece_id=GDR7XRSOXSLH6CRF4TW6SIRGQVZ6IF2K22WGKPLL7DUHN6OHZV5A, seg_pos={0 0}, upload=474254]: Long tail cancellation due to optimal threshold Dec 28 04:15:24.696808625 (540h46m24.56789167s) [noise=true , address=145.79.227.44:4195 , piece_id=ZFJF4A5OOEBZ3TBDPJUR73CKGRUDDVBL3CYDDSSLJNVXUQXIAUOA, seg_pos={0 0}, upload=474254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 28 04:15:24.696828212 (540h46m24.567911247s) [noise=true , address=145.79.227.44:4195 , piece_id=ZFJF4A5OOEBZ3TBDPJUR73CKGRUDDVBL3CYDDSSLJNVXUQXIAUOA, seg_pos={0 0}, upload=474254]: Long tail cancellation due to optimal threshold Dec 28 04:15:24.696845345 (540h46m24.56792839s) [noise=true , address=45.91.116.7:4009 , piece_id=Q4YTMPS3CQAIGIB3N7XRUXWZX6GTS7V6VOMV5N26QCKKLZTQSIJQ, seg_pos={0 0}, upload=474254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 28 04:15:24.696853700 (540h46m24.567936745s) [noise=true , address=45.91.116.7:4009 , piece_id=Q4YTMPS3CQAIGIB3N7XRUXWZX6GTS7V6VOMV5N26QCKKLZTQSIJQ, seg_pos={0 0}, upload=474254]: Long tail cancellation due to optimal threshold Dec 28 04:15:24.696906551 (540h46m24.567989586s) [seg_pos={0 0}, upload=474254]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 04:15:24.696978358 (540h46m24.568061393s) [upload=474254]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:15:24.696984480 (540h46m24.568067525s) [upload=474254]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:15:24.978934344 (540h46m24.850017389s) [upload=474254]: Done uploading segments... Dec 28 04:15:24.978956447 (540h46m24.850039492s) [upload=474254]: Upload finished. err: <nil> Duration: 30.002268047s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 12.175385143s trace: https://jaeger.dev.storj.io/trace/3b8130f32e60318d FAIL (2025-12-28 04:15:24 UTC) orbiter warning: context deadline exceeded Duration: 12.175407776s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 50 out of 60 is failed (threshold=3) [X.XXXXX.XXXXXXXXXXXXXXXXX..XXXX.XXXX.XXXXX.XXXXXXX.XXXXXX..X] duration: 30.003444s trace: https://jaeger.dev.storj.io/trace/84838eb9ea99c0 FAIL (2025-12-28 04:15:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: 65AZ73LQV65XBDBSSDCFCO2WPHEURHBZXD7VEA7SGVYN4ZLD4P2Q) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: FSAUWTJL2SIHXV3DWPGAUIGMCZQ3WPNZ4CGEITPWDRGKURQP4NOA) context deadline exceeded ======LOGS====== Dec 28 04:14:42.879119398 (540h45m42.750202433s) [upload=474245]: Starting upload Dec 28 04:14:42.879203257 (540h45m42.750286292s) [upload=474245]: Done starting upload Dec 28 04:14:42.879337733 (540h45m42.750420778s) [upload=474245]: Uploading segments... Dec 28 04:14:42.879430659 (540h45m42.750513704s) [upload=474245]: Got next segment. Inline: false Dec 28 04:14:42.879471538 (540h45m42.750554583s) [upload=474245]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 04:14:42.879498639 (540h45m42.750581674s) [upload=474245]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 04:14:43.131020655 (540h45m43.00210369s) [seg_pos={0 0}, upload=474245]: Begin upload segment... Dec 28 04:14:43.131065010 (540h45m43.002148045s) [seg_pos={0 0}, upload=474245]: Done begin upload segment. Dec 28 04:14:43.131092633 (540h45m43.002175668s) [noise=true , address=77.90.31.16:4736 , piece_id=3BHPB7YSETP3PPUWL3CPJIZ55KXDC6BUOQSS7NNHG6NXAMHGJQEA, seg_pos={0 0}, upload=474245]: Uploading piece... Dec 28 04:14:43.131243109 (540h45m43.002326164s) [noise=true , address=45.91.116.14:4189 , piece_id=65AZ73LQV65XBDBSSDCFCO2WPHEURHBZXD7VEA7SGVYN4ZLD4P2Q, seg_pos={0 0}, upload=474245]: Uploading piece... Dec 28 04:14:43.132216212 (540h45m43.003299267s) [noise=true , address=145.79.227.13:4803 , piece_id=NJGMDQUBQ6JCZ6KSGUITNF3IAC6EFIWOXL2DFYPJEWP233E33WBA, seg_pos={0 0}, upload=474245]: Uploading piece... Dec 28 04:14:43.132449516 (540h45m43.003532561s) [noise=true , address=145.79.227.19:4077 , piece_id=S6NM7VYJMEVVLQUUITTFLRQ4SNXUHWBTHFXPAMK4L5QI3RKJFO3A, seg_pos={0 0}, upload=474245]: Uploading piece... Dec 28 04:14:43.132992510 (540h45m43.004075555s) [noise=true , address=145.79.227.50:4375 , piece_id=IGXNZMV54QHXDAXLX3G2JIALJD7II3H5BIATUGSAQSRJLLYKRGKQ, seg_pos={0 0}, upload=474245]: Uploading piece... Dec 28 04:14:43.133531978 (540h45m43.004615023s) [noise=true , address=45.91.116.11:4120 , piece_id=Q3JBOH5YXX7BA46CICSVV53RORPMXWRBP7TB6MDTJPBSTPDS2JNQ, seg_pos={0 0}, upload=474245]: Uploading piece... Dec 28 04:14:43.134167097 (540h45m43.005250142s) [noise=true , address=45.91.116.9:4263 , piece_id=NKNHZ5SJWS7YLGDPDD6L4JYGT3JFAVMKDGHC6MGTFDN2BALFXLYA, seg_pos={0 0}, upload=474245]: Uploading piece... Dec 28 04:14:43.134266536 (540h45m43.005349581s) [noise=true , address=145.79.227.38:4977 , piece_id=6CO56H4KZYE7UFM53TAJFZKBQDQ6QYH3GZOLBC7GSYRI2LFHF24A, seg_pos={0 0}, upload=474245]: Uploading piece... Dec 28 04:14:43.134741029 (540h45m43.005824074s) [noise=true , address=45.91.116.12:4226 , piece_id=LKR4H5GL2X2JSQSKBJALMUUQM75ZPBNE2KVUFBPG2PBMYUQD6TVA, seg_pos={0 0}, upload=474245]: Uploading piece... Dec 28 04:14:43.134848554 (540h45m43.005931589s) [noise=true , address=77.90.31.15:4700 , piece_id=FSAUWTJL2SIHXV3DWPGAUIGMCZQ3WPNZ4CGEITPWDRGKURQP4NOA, seg_pos={0 0}, upload=474245]: Uploading piece... Dec 28 04:14:43.135347875 (540h45m43.00643092s) [noise=true , address=145.79.227.17:4970 , piece_id=N6EA7U5BNDSJAJQF7BEXBY6LNZAQSLJ4EK43TBNPJ7KVLRQAL3NA, seg_pos={0 0}, upload=474245]: Uploading piece... Dec 28 04:14:43.135420614 (540h45m43.006503649s) [noise=true , address=145.79.227.12:4765 , piece_id=J5WNBBSAFGBH4CF5PSXJT7SHMMJ7DKJI37K3GIMITLK3S44MZXBA, seg_pos={0 0}, upload=474245]: Uploading piece... Dec 28 04:14:43.135938300 (540h45m43.007021345s) [noise=true , address=45.91.116.8:4048 , piece_id=G7OIVJYC5T2YSOCSRSEX3QMY7A72DPY3YGVUTVCMCZFYK5E2H3BA, seg_pos={0 0}, upload=474245]: Uploading piece... Dec 28 04:14:43.136104115 (540h45m43.00718719s) [noise=true , address=145.79.227.6:4257 , piece_id=NZSRGAI6YLIWY3WFNX2M2ATNY7MRC5MULBY4P5JZBMM2YX4TD6SA, seg_pos={0 0}, upload=474245]: Uploading piece... Dec 28 04:14:43.136428032 (540h45m43.007511067s) [noise=true , address=145.79.227.37:4567 , piece_id=3S657K5NFEMGOGAQYE5PQPH2KWNQFFFQ4N7BZSTBAWTOIN7AKC5Q, seg_pos={0 0}, upload=474245]: Uploading piece... Dec 28 04:14:43.137044255 (540h45m43.00812729s) [noise=true , address=145.79.227.36:4519 , piece_id=ENEBH5IJGIEOR7FJIF3RI5XKS642EYXALUU2G7LXWGFJPWM2OWKQ, seg_pos={0 0}, upload=474245]: Uploading piece... Dec 28 04:14:44.851594095 (540h45m44.72267713s) [upload=474245]: Next returned nil segment Dec 28 04:14:44.851624643 (540h45m44.722707679s) [upload=474245]: Waiting for error group managing segments... Dec 28 04:14:45.135847977 (540h45m45.006931012s) [noise=true , address=145.79.227.13:4803 , piece_id=NJGMDQUBQ6JCZ6KSGUITNF3IAC6EFIWOXL2DFYPJEWP233E33WBA, seg_pos={0 0}, upload=474245]: Done uploading piece. err: <nil> Dec 28 04:14:45.143318573 (540h45m45.014401618s) [noise=true , address=145.79.227.36:4519 , piece_id=ENEBH5IJGIEOR7FJIF3RI5XKS642EYXALUU2G7LXWGFJPWM2OWKQ, seg_pos={0 0}, upload=474245]: Done uploading piece. err: <nil> Dec 28 04:14:45.147070276 (540h45m45.018153321s) [noise=true , address=145.79.227.38:4977 , piece_id=6CO56H4KZYE7UFM53TAJFZKBQDQ6QYH3GZOLBC7GSYRI2LFHF24A, seg_pos={0 0}, upload=474245]: Done uploading piece. err: <nil> Dec 28 04:14:45.156623126 (540h45m45.027706161s) [noise=true , address=145.79.227.19:4077 , piece_id=S6NM7VYJMEVVLQUUITTFLRQ4SNXUHWBTHFXPAMK4L5QI3RKJFO3A, seg_pos={0 0}, upload=474245]: Done uploading piece. err: <nil> Dec 28 04:14:45.163329887 (540h45m45.034412932s) [noise=true , address=145.79.227.50:4375 , piece_id=IGXNZMV54QHXDAXLX3G2JIALJD7II3H5BIATUGSAQSRJLLYKRGKQ, seg_pos={0 0}, upload=474245]: Done uploading piece. err: <nil> Dec 28 04:14:45.170536819 (540h45m45.041619864s) [noise=true , address=145.79.227.6:4257 , piece_id=NZSRGAI6YLIWY3WFNX2M2ATNY7MRC5MULBY4P5JZBMM2YX4TD6SA, seg_pos={0 0}, upload=474245]: Done uploading piece. err: <nil> Dec 28 04:14:45.174557505 (540h45m45.04564055s) [noise=true , address=145.79.227.17:4970 , piece_id=N6EA7U5BNDSJAJQF7BEXBY6LNZAQSLJ4EK43TBNPJ7KVLRQAL3NA, seg_pos={0 0}, upload=474245]: Done uploading piece. err: <nil> Dec 28 04:14:45.178914050 (540h45m45.049997085s) [noise=true , address=145.79.227.12:4765 , piece_id=J5WNBBSAFGBH4CF5PSXJT7SHMMJ7DKJI37K3GIMITLK3S44MZXBA, seg_pos={0 0}, upload=474245]: Done uploading piece. err: <nil> Dec 28 04:14:45.336037247 (540h45m45.207120282s) [noise=true , address=77.90.31.15:4700 , piece_id=FSAUWTJL2SIHXV3DWPGAUIGMCZQ3WPNZ4CGEITPWDRGKURQP4NOA, seg_pos={0 0}, upload=474245]: Done uploading piece. err: <nil> Dec 28 04:14:45.391069246 (540h45m45.262152281s) [noise=true , address=45.91.116.14:4189 , piece_id=65AZ73LQV65XBDBSSDCFCO2WPHEURHBZXD7VEA7SGVYN4ZLD4P2Q, seg_pos={0 0}, upload=474245]: Done uploading piece. err: <nil> Dec 28 04:14:45.394177154 (540h45m45.265260209s) [noise=true , address=45.91.116.12:4226 , piece_id=LKR4H5GL2X2JSQSKBJALMUUQM75ZPBNE2KVUFBPG2PBMYUQD6TVA, seg_pos={0 0}, upload=474245]: Done uploading piece. err: <nil> Dec 28 04:14:45.407790836 (540h45m45.278873881s) [noise=true , address=77.90.31.16:4736 , piece_id=3BHPB7YSETP3PPUWL3CPJIZ55KXDC6BUOQSS7NNHG6NXAMHGJQEA, seg_pos={0 0}, upload=474245]: Done uploading piece. err: <nil> Dec 28 04:14:45.463141542 (540h45m45.334224587s) [noise=true , address=45.91.116.11:4120 , piece_id=Q3JBOH5YXX7BA46CICSVV53RORPMXWRBP7TB6MDTJPBSTPDS2JNQ, seg_pos={0 0}, upload=474245]: Done uploading piece. err: <nil> Dec 28 04:14:45.463166139 (540h45m45.334249175s) [seg_pos={0 0}, upload=474245]: Segment reached optimal threshold of 13 pieces. Dec 28 04:14:45.463413400 (540h45m45.334496445s) [noise=true , address=45.91.116.9:4263 , piece_id=NKNHZ5SJWS7YLGDPDD6L4JYGT3JFAVMKDGHC6MGTFDN2BALFXLYA, seg_pos={0 0}, upload=474245]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 28 04:14:45.463422758 (540h45m45.334505793s) [noise=true , address=45.91.116.9:4263 , piece_id=NKNHZ5SJWS7YLGDPDD6L4JYGT3JFAVMKDGHC6MGTFDN2BALFXLYA, seg_pos={0 0}, upload=474245]: Long tail cancellation due to optimal threshold Dec 28 04:14:45.463602700 (540h45m45.334685755s) [noise=true , address=145.79.227.37:4567 , piece_id=3S657K5NFEMGOGAQYE5PQPH2KWNQFFFQ4N7BZSTBAWTOIN7AKC5Q, seg_pos={0 0}, upload=474245]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: expected piece hash; context canceled; EOF Dec 28 04:14:45.463632317 (540h45m45.334715362s) [noise=true , address=145.79.227.37:4567 , piece_id=3S657K5NFEMGOGAQYE5PQPH2KWNQFFFQ4N7BZSTBAWTOIN7AKC5Q, seg_pos={0 0}, upload=474245]: Long tail cancellation due to optimal threshold Dec 28 04:14:45.463852756 (540h45m45.334935801s) [noise=true , address=45.91.116.8:4048 , piece_id=G7OIVJYC5T2YSOCSRSEX3QMY7A72DPY3YGVUTVCMCZFYK5E2H3BA, seg_pos={0 0}, upload=474245]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 28 04:14:45.463866513 (540h45m45.334949558s) [noise=true , address=45.91.116.8:4048 , piece_id=G7OIVJYC5T2YSOCSRSEX3QMY7A72DPY3YGVUTVCMCZFYK5E2H3BA, seg_pos={0 0}, upload=474245]: Long tail cancellation due to optimal threshold Dec 28 04:14:45.463890027 (540h45m45.334973062s) [seg_pos={0 0}, upload=474245]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 04:14:45.463983786 (540h45m45.335066821s) [upload=474245]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:14:45.463990358 (540h45m45.335073393s) [upload=474245]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:14:45.752719131 (540h45m45.623802187s) [upload=474245]: Done uploading segments... Dec 28 04:14:45.752746013 (540h45m45.623829058s) [upload=474245]: Upload finished. err: <nil> Duration: 30.003468104s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 20 out of 60 is failed (threshold=6) [XXX....X.X.X...X..X.X.X..X...XX.....X......XX...X.X....X.X..] duration: 3.532703821s trace: https://jaeger.dev.storj.io/trace/31d935b4c9de8559 FAIL (2025-12-28 04:15:06 UTC) orbiter warning: context deadline exceeded Duration: 3.53274547s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................................X......X....] duration: 32.955642721s trace: https://jaeger.dev.storj.io/trace/26c246d73008cc49 FAIL (2025-12-28 04:14:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== oise=true , address=145.79.227.43:4191 , piece_id=CLYBACZJDTAFTBEMEMSOXP4I22PDYSVIBX3AMNN4DISQUJWYFDMQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.179395486 (540h42m11.050478531s) [noise=true , address=145.79.227.9:4371 , piece_id=2ON3P6ZFEUEBFFLCHYY5BWAAJEFX43KKU5MU5TLBQHSZ52RCAYLA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.179823281 (540h42m11.050906336s) [noise=true , address=77.90.31.18:4819 , piece_id=FVZE5P55HHT3RGOI4M2OFV6NER4BNVGFSZOMDRADO5JYI54DRHLQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.180232490 (540h42m11.051315535s) [noise=true , address=77.90.31.17:4778 , piece_id=WHTKIVKEZNFREOTNDTLLBB24JGWKIXWBJKZ2EUM3QL2SM7Z4YXDA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.180945087 (540h42m11.052028132s) [noise=true , address=45.91.116.6:4003 , piece_id=H4VWZIMYVM2M3SX76LSS64ARSENBQL6OOTL5EY5XE4IAORO7OWGQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181189232 (540h42m11.052272277s) [noise=true , address=77.90.31.21:4906 , piece_id=SGYCZ2E6IMEJA332PTI7DQ4FWRJR6NG4OYRUP6H2KHXS2TKC5S2A, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181508139 (540h42m11.052591174s) [noise=true , address=77.90.31.15:4714 , piece_id=YVXSR2B74BDK4BV5QGLZLWOKVXV3TWKSW5M3KG323U7VUUBIQU3Q, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181597209 (540h42m11.052680254s) [noise=true , address=145.79.227.11:4755 , piece_id=F37J7GC6RB2PL3B4B6MFVAPB6IOGSJSKVQ5UNFHFSBJ7C3SABURQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181882131 (540h42m11.052965166s) [noise=true , address=145.79.227.33:4876 , piece_id=UEOY3HYI3WWJTXUSBALBXIGCMBC5M6XGR42SIOBXQNDCZ3UFMWVA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182132017 (540h42m11.053215052s) [noise=true , address=145.79.227.40:4053 , piece_id=V7R5MTCCKCGHV7KD4LAXFBUJI3HRGVAJUCVYL4WPL3M32CVQXOBA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182274007 (540h42m11.053357052s) [noise=true , address=145.79.227.51:4432 , piece_id=C5P63KNNZLSOBH772MRMYOJOQGLE7KXGELQN437HTVPAM72RXUCA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182451545 (540h42m11.05353458s) [noise=true , address=145.79.227.30:4784 , piece_id=UTH47JZWG3SGX3KXBFJJALALTPP6INYORUHDYSS6CNDEUQ6BDL7A, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182777516 (540h42m11.053860551s) [noise=true , address=64.49.13.169:4708 , piece_id=6OBO4M26AXDJXUTOBI66S2JJARDEM2CTTFHUIAOI4CX2PZITRWFQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182919286 (540h42m11.054002331s) [noise=true , address=145.79.227.6:4267 , piece_id=SDG3WH6TPR47OHG57NMVFAHOYVAL45XMJ7S22FZL6IFR6DS45HQQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.183101804 (540h42m11.054184839s) [noise=true , address=145.79.227.34:4924 , piece_id=R7MT4YS5IPGHOCNZX77ITOPXL57DOYD2HJXPJYYBGNVUWOT33JPA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.183514740 (540h42m11.054597785s) [noise=true , address=193.149.164.143:4696 , piece_id=OJQFP4YJ3EF52S2XQN5QPKMQJDS57QEOQQ4PUPARPGPAVSLGKMFA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:12.562530634 (540h42m12.433613679s) [upload=474194]: Next returned nil segment Dec 28 04:11:12.562544100 (540h42m12.433627135s) [upload=474194]: Waiting for error group managing segments... Dec 28 04:11:13.045899872 (540h42m12.916982917s) [noise=true , address=145.79.227.9:4371 , piece_id=2ON3P6ZFEUEBFFLCHYY5BWAAJEFX43KKU5MU5TLBQHSZ52RCAYLA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.068300556 (540h42m12.939383591s) [noise=true , address=145.79.227.11:4755 , piece_id=F37J7GC6RB2PL3B4B6MFVAPB6IOGSJSKVQ5UNFHFSBJ7C3SABURQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.073683536 (540h42m12.944766571s) [noise=true , address=145.79.227.40:4053 , piece_id=V7R5MTCCKCGHV7KD4LAXFBUJI3HRGVAJUCVYL4WPL3M32CVQXOBA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.125067800 (540h42m12.996150835s) [noise=true , address=145.79.227.6:4267 , piece_id=SDG3WH6TPR47OHG57NMVFAHOYVAL45XMJ7S22FZL6IFR6DS45HQQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.156395944 (540h42m13.027478989s) [noise=true , address=145.79.227.51:4432 , piece_id=C5P63KNNZLSOBH772MRMYOJOQGLE7KXGELQN437HTVPAM72RXUCA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.221620070 (540h42m13.092703105s) [noise=true , address=77.90.31.15:4714 , piece_id=YVXSR2B74BDK4BV5QGLZLWOKVXV3TWKSW5M3KG323U7VUUBIQU3Q, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.266169559 (540h42m13.137252594s) [noise=true , address=145.79.227.33:4876 , piece_id=UEOY3HYI3WWJTXUSBALBXIGCMBC5M6XGR42SIOBXQNDCZ3UFMWVA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.273253176 (540h42m13.144336221s) [noise=true , address=77.90.31.12:4604 , piece_id=DVQ53JEZL2GDTXFDDIWN4BLSJJHRZORII7UNF5HS3GIM5VJX3Y2Q, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.324930308 (540h42m13.196013343s) [noise=true , address=77.90.31.17:4778 , piece_id=WHTKIVKEZNFREOTNDTLLBB24JGWKIXWBJKZ2EUM3QL2SM7Z4YXDA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.333001758 (540h42m13.204084793s) [noise=true , address=77.90.31.18:4819 , piece_id=FVZE5P55HHT3RGOI4M2OFV6NER4BNVGFSZOMDRADO5JYI54DRHLQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.418239402 (540h42m13.289322437s) [noise=true , address=45.91.116.6:4003 , piece_id=H4VWZIMYVM2M3SX76LSS64ARSENBQL6OOTL5EY5XE4IAORO7OWGQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.460099681 (540h42m13.331182726s) [noise=true , address=77.90.31.21:4906 , piece_id=SGYCZ2E6IMEJA332PTI7DQ4FWRJR6NG4OYRUP6H2KHXS2TKC5S2A, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.860538777 (540h42m13.731621812s) [noise=true , address=193.149.164.143:4696 , piece_id=OJQFP4YJ3EF52S2XQN5QPKMQJDS57QEOQQ4PUPARPGPAVSLGKMFA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:14.127606583 (540h42m13.998689618s) [noise=true , address=145.79.227.34:4924 , piece_id=R7MT4YS5IPGHOCNZX77ITOPXL57DOYD2HJXPJYYBGNVUWOT33JPA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:14.127625258 (540h42m13.998708303s) [seg_pos={0 0}, upload=474194]: Segment reached optimal threshold of 14 pieces. Dec 28 04:11:14.127786726 (540h42m13.998869761s) [noise=true , address=45.91.116.4:4966 , piece_id=ZTHC25VIBEHSKRULTNAS25DK2DBVI7ATO46SDMMTON7PBBPRMWBQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DSHTtpwcx1h1utGXNou11K7Tcg1ZhkkMdMgY6BDD2ifG6LQ7S): protocol: expected piece hash; context canceled; EOF Dec 28 04:11:14.127792547 (540h42m13.998875582s) [noise=true , address=45.91.116.4:4966 , piece_id=ZTHC25VIBEHSKRULTNAS25DK2DBVI7ATO46SDMMTON7PBBPRMWBQ, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128017335 (540h42m13.99910037s) [noise=true , address=145.79.227.43:4191 , piece_id=CLYBACZJDTAFTBEMEMSOXP4I22PDYSVIBX3AMNN4DISQUJWYFDMQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw): protocol: expected piece hash; context canceled; EOF Dec 28 04:11:14.128023747 (540h42m13.999106782s) [noise=true , address=145.79.227.43:4191 , piece_id=CLYBACZJDTAFTBEMEMSOXP4I22PDYSVIBX3AMNN4DISQUJWYFDMQ, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128139027 (540h42m13.999222102s) [noise=true , address=64.49.13.169:4708 , piece_id=6OBO4M26AXDJXUTOBI66S2JJARDEM2CTTFHUIAOI4CX2PZITRWFQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:40936->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:40936->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40936->64.49.13.169:4708: use of closed network connection Dec 28 04:11:14.128150668 (540h42m13.999233733s) [noise=true , address=64.49.13.169:4708 , piece_id=6OBO4M26AXDJXUTOBI66S2JJARDEM2CTTFHUIAOI4CX2PZITRWFQ, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128292198 (540h42m13.999375243s) [noise=true , address=145.79.227.30:4784 , piece_id=UTH47JZWG3SGX3KXBFJJALALTPP6INYORUHDYSS6CNDEUQ6BDL7A, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pfXWex2WsuhmLb2smRpE46cHT9aipSTHW3e5oyELS1GhF1cFr): protocol: expected piece hash; context canceled; EOF Dec 28 04:11:14.128299212 (540h42m13.999382257s) [noise=true , address=145.79.227.30:4784 , piece_id=UTH47JZWG3SGX3KXBFJJALALTPP6INYORUHDYSS6CNDEUQ6BDL7A, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128395815 (540h42m13.99947885s) [seg_pos={0 0}, upload=474194]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 04:11:14.128488171 (540h42m13.999571206s) [upload=474194]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:11:14.128493561 (540h42m13.999576597s) [upload=474194]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:11:14.572773387 (540h42m14.443856452s) [upload=474194]: Done uploading segments... Dec 28 04:11:14.572800389 (540h42m14.443883454s) [upload=474194]: Upload finished. err: <nil> Duration: 32.955670456s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 50 out of 60 is failed (threshold=3) [XX.XXXXX.XXXXXXXXXXXXXXXXX..XXXX.XXXX.XXXXX.XXXXXXX.XXXXXX..] duration: 14.491729696s trace: https://jaeger.dev.storj.io/trace/319ce7f5bba17d6e FAIL (2025-12-28 04:14:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ec 28 04:12:38.726334110 (540h43m38.597417155s) [upload=474215]: Starting upload Dec 28 04:12:38.726426036 (540h43m38.597509101s) [upload=474215]: Done starting upload Dec 28 04:12:38.726626287 (540h43m38.597709322s) [upload=474215]: Uploading segments... Dec 28 04:12:38.727559725 (540h43m38.59864277s) [upload=474215]: Got next segment. Inline: false Dec 28 04:12:38.727608728 (540h43m38.598691763s) [upload=474215]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 04:12:38.727613487 (540h43m38.598696522s) [upload=474215]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 04:12:38.970997057 (540h43m38.842080092s) [seg_pos={0 0}, upload=474215]: Begin upload segment... Dec 28 04:12:38.971044887 (540h43m38.842127932s) [seg_pos={0 0}, upload=474215]: Done begin upload segment. Dec 28 04:12:38.971082419 (540h43m38.842165454s) [noise=true , address=45.91.116.4:4937 , piece_id=QNPOBYDWPVDD3KBB7ER5O2R3CIKGT4SYK3B4GI2HMA4KIDD76UNA, seg_pos={0 0}, upload=474215]: Uploading piece... Dec 28 04:12:38.971098439 (540h43m38.842181474s) [noise=true , address=145.79.227.3:4150 , piece_id=GUEYHIOJ7CMOEQKFNGQS7HYPSW7X6ZD7USBS26VFH7NSDL7H7NAQ, seg_pos={0 0}, upload=474215]: Uploading piece... Dec 28 04:12:38.971231382 (540h43m38.842314427s) [noise=true , address=45.91.116.14:4190 , piece_id=S6EEFINMLCOEGAZ6PUBLMJ642CHGRPHBS3JUNHXJW3YEOVT5V46A, seg_pos={0 0}, upload=474215]: Uploading piece... Dec 28 04:12:38.971587631 (540h43m38.842670666s) [noise=true , address=45.91.116.13:4154 , piece_id=7XCD3R4BMWVXCFNKN3ULETEL4LFVL7P2CDWPEKEDQQV6QUTEZ5SQ, seg_pos={0 0}, upload=474215]: Uploading piece... Dec 28 04:12:38.971622718 (540h43m38.842705753s) [noise=true , address=145.79.227.12:4767 , piece_id=VW2GZ3QGK67R3JCXTA3E76ZRBHXRMPIJU2PSGEBEZM6CI2BSTAFQ, seg_pos={0 0}, upload=474215]: Uploading piece... Dec 28 04:12:38.971902561 (540h43m38.842985606s) [noise=true , address=77.90.31.14:4661 , piece_id=ZWNZGRAKYMNKWIHFJTZKFAXWQXY373UCFNXR3LBGTYUNYC64UUJQ, seg_pos={0 0}, upload=474215]: Uploading piece... Dec 28 04:12:38.972232859 (540h43m38.843315894s) [noise=true , address=145.79.227.47:4266 , piece_id=JL5JIICISIMMEKGD43AMGAFQH5RTSWVGPOOSZB5GFFXGFNHFMFYQ, seg_pos={0 0}, upload=474215]: Uploading piece... Dec 28 04:12:38.972534594 (540h43m38.843617629s) [noise=true , address=45.91.116.11:4120 , piece_id=RVKDXZAJP2T5DUQCHONPAHHFQBTYCERSRBFG2SB6BVXVHDHT47WQ, seg_pos={0 0}, upload=474215]: Uploading piece... Dec 28 04:12:38.972787115 (540h43m38.84387015s) [noise=true , address=45.91.116.6:4973 , piece_id=PQYA2MXXRTEZK55ABEGSACOBWK25XTUXH3W7II56OB6KV7FTAEPQ, seg_pos={0 0}, upload=474215]: Uploading piece... Dec 28 04:12:38.973050807 (540h43m38.844133852s) [noise=true , address=45.91.116.5:4299 , piece_id=O5XMGGHYJZCIO7U7H7GBRKPME77YBJGO6QCBYLSWK443D6CD6WFQ, seg_pos={0 0}, upload=474215]: Uploading piece... Dec 28 04:12:38.973313447 (540h43m38.844396482s) [noise=true , address=45.91.116.8:4045 , piece_id=K7FRQITAZJMJLK557DWGRI4YI2YVI5MHDEZC6YOTMQIETDD4WRMQ, seg_pos={0 0}, upload=474215]: Uploading piece... Dec 28 04:12:38.974132226 (540h43m38.845215271s) [noise=true , address=145.79.227.45:4230 , piece_id=63LRU4PIETFBWVDOU2TAR3FSJYBZGY4Q2CFQXVXW7ZEROXZ226GQ, seg_pos={0 0}, upload=474215]: Uploading piece... Dec 28 04:12:38.974473065 (540h43m38.8455561s) [noise=true , address=145.79.227.11:4732 , piece_id=CH4XXVYC7SUBJJVWCPIKAXDTMXX6JQ5OBQV7MWUP5VRTFRZVGIJA, seg_pos={0 0}, upload=474215]: Uploading piece... Dec 28 04:12:38.974825517 (540h43m38.845908552s) [noise=true , address=145.79.227.31:4798 , piece_id=XDGSPIDVQFPOSA5M4RJPKZP5TJBIDF2U6OJDGD5QW5UKCNT2IPAA, seg_pos={0 0}, upload=474215]: Uploading piece... Dec 28 04:12:38.975180542 (540h43m38.846263587s) [noise=true , address=77.90.31.17:4770 , piece_id=3VQAPUKV3TRP2WC4WMBYI37N7ZCPKU5YHPBCFTKGOYUVNMJKMAGA, seg_pos={0 0}, upload=474215]: Uploading piece... Dec 28 04:12:38.975495292 (540h43m38.846578337s) [noise=true , address=77.90.31.16:4736 , piece_id=EQKZQCL7PYGHGG33TCRTKQTJL2M4ZUP2H3AFT5V7XSQQB2KYGCWA, seg_pos={0 0}, upload=474215]: Uploading piece... Dec 28 04:12:40.773622607 (540h43m40.644705642s) [upload=474215]: Next returned nil segment Dec 28 04:12:40.773633357 (540h43m40.644716392s) [upload=474215]: Waiting for error group managing segments... Dec 28 04:12:41.071171164 (540h43m40.9422542s) [noise=true , address=145.79.227.45:4230 , piece_id=63LRU4PIETFBWVDOU2TAR3FSJYBZGY4Q2CFQXVXW7ZEROXZ226GQ, seg_pos={0 0}, upload=474215]: Done uploading piece. err: <nil> Dec 28 04:12:41.088540888 (540h43m40.959623923s) [noise=true , address=145.79.227.12:4767 , piece_id=VW2GZ3QGK67R3JCXTA3E76ZRBHXRMPIJU2PSGEBEZM6CI2BSTAFQ, seg_pos={0 0}, upload=474215]: Done uploading piece. err: <nil> Dec 28 04:12:41.095541709 (540h43m40.966624754s) [noise=true , address=145.79.227.31:4798 , piece_id=XDGSPIDVQFPOSA5M4RJPKZP5TJBIDF2U6OJDGD5QW5UKCNT2IPAA, seg_pos={0 0}, upload=474215]: Done uploading piece. err: <nil> Dec 28 04:12:41.095607585 (540h43m40.96669062s) [noise=true , address=145.79.227.11:4732 , piece_id=CH4XXVYC7SUBJJVWCPIKAXDTMXX6JQ5OBQV7MWUP5VRTFRZVGIJA, seg_pos={0 0}, upload=474215]: Done uploading piece. err: <nil> Dec 28 04:12:41.202004841 (540h43m41.073087876s) [noise=true , address=145.79.227.47:4266 , piece_id=JL5JIICISIMMEKGD43AMGAFQH5RTSWVGPOOSZB5GFFXGFNHFMFYQ, seg_pos={0 0}, upload=474215]: Done uploading piece. err: <nil> Dec 28 04:12:41.265377403 (540h43m41.136460438s) [noise=true , address=45.91.116.11:4120 , piece_id=RVKDXZAJP2T5DUQCHONPAHHFQBTYCERSRBFG2SB6BVXVHDHT47WQ, seg_pos={0 0}, upload=474215]: Done uploading piece. err: <nil> Dec 28 04:12:41.268392994 (540h43m41.139476029s) [noise=true , address=45.91.116.8:4045 , piece_id=K7FRQITAZJMJLK557DWGRI4YI2YVI5MHDEZC6YOTMQIETDD4WRMQ, seg_pos={0 0}, upload=474215]: Done uploading piece. err: <nil> Dec 28 04:12:41.276154464 (540h43m41.147237509s) [noise=true , address=77.90.31.14:4661 , piece_id=ZWNZGRAKYMNKWIHFJTZKFAXWQXY373UCFNXR3LBGTYUNYC64UUJQ, seg_pos={0 0}, upload=474215]: Done uploading piece. err: <nil> Dec 28 04:12:41.322814822 (540h43m41.193897867s) [noise=true , address=45.91.116.14:4190 , piece_id=S6EEFINMLCOEGAZ6PUBLMJ642CHGRPHBS3JUNHXJW3YEOVT5V46A, seg_pos={0 0}, upload=474215]: Done uploading piece. err: <nil> Dec 28 04:12:41.325496728 (540h43m41.196579763s) [noise=true , address=45.91.116.5:4299 , piece_id=O5XMGGHYJZCIO7U7H7GBRKPME77YBJGO6QCBYLSWK443D6CD6WFQ, seg_pos={0 0}, upload=474215]: Done uploading piece. err: <nil> Dec 28 04:12:41.329321932 (540h43m41.200404967s) [noise=true , address=45.91.116.6:4973 , piece_id=PQYA2MXXRTEZK55ABEGSACOBWK25XTUXH3W7II56OB6KV7FTAEPQ, seg_pos={0 0}, upload=474215]: Done uploading piece. err: <nil> Dec 28 04:12:41.340599787 (540h43m41.211682822s) [noise=true , address=45.91.116.4:4937 , piece_id=QNPOBYDWPVDD3KBB7ER5O2R3CIKGT4SYK3B4GI2HMA4KIDD76UNA, seg_pos={0 0}, upload=474215]: Done uploading piece. err: <nil> Dec 28 04:12:41.364954202 (540h43m41.236037237s) [noise=true , address=77.90.31.17:4770 , piece_id=3VQAPUKV3TRP2WC4WMBYI37N7ZCPKU5YHPBCFTKGOYUVNMJKMAGA, seg_pos={0 0}, upload=474215]: Done uploading piece. err: <nil> Dec 28 04:12:41.364983147 (540h43m41.236066182s) [seg_pos={0 0}, upload=474215]: Segment reached optimal threshold of 13 pieces. Dec 28 04:12:41.365269041 (540h43m41.236352076s) [noise=true , address=45.91.116.13:4154 , piece_id=7XCD3R4BMWVXCFNKN3ULETEL4LFVL7P2CDWPEKEDQQV6QUTEZ5SQ, seg_pos={0 0}, upload=474215]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 28 04:12:41.365276335 (540h43m41.23635938s) [noise=true , address=45.91.116.13:4154 , piece_id=7XCD3R4BMWVXCFNKN3ULETEL4LFVL7P2CDWPEKEDQQV6QUTEZ5SQ, seg_pos={0 0}, upload=474215]: Long tail cancellation due to optimal threshold Dec 28 04:12:41.365317864 (540h43m41.236400899s) [noise=true , address=145.79.227.3:4150 , piece_id=GUEYHIOJ7CMOEQKFNGQS7HYPSW7X6ZD7USBS26VFH7NSDL7H7NAQ, seg_pos={0 0}, upload=474215]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: write tcp 188.166.225.170:47130->145.79.227.3:4150: use of closed network connection; write tcp 188.166.225.170:47130->145.79.227.3:4150: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47130->145.79.227.3:4150: use of closed network connection Dec 28 04:12:41.365333905 (540h43m41.23641694s) [noise=true , address=145.79.227.3:4150 , piece_id=GUEYHIOJ7CMOEQKFNGQS7HYPSW7X6ZD7USBS26VFH7NSDL7H7NAQ, seg_pos={0 0}, upload=474215]: Long tail cancellation due to optimal threshold Dec 28 04:12:41.365419728 (540h43m41.236502763s) [noise=true , address=77.90.31.16:4736 , piece_id=EQKZQCL7PYGHGG33TCRTKQTJL2M4ZUP2H3AFT5V7XSQQB2KYGCWA, seg_pos={0 0}, upload=474215]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 28 04:12:41.365426520 (540h43m41.236509566s) [noise=true , address=77.90.31.16:4736 , piece_id=EQKZQCL7PYGHGG33TCRTKQTJL2M4ZUP2H3AFT5V7XSQQB2KYGCWA, seg_pos={0 0}, upload=474215]: Long tail cancellation due to optimal threshold Dec 28 04:12:41.365443583 (540h43m41.236526629s) [seg_pos={0 0}, upload=474215]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 04:12:41.365539597 (540h43m41.236622632s) [upload=474215]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:12:41.365544826 (540h43m41.236627861s) [upload=474215]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:12:41.651265873 (540h43m41.522348908s) [upload=474215]: Done uploading segments... Dec 28 04:12:41.651287303 (540h43m41.522370348s) [upload=474215]: Upload finished. err: <nil> Duration: 14.491761757s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 21 out of 60 is failed (threshold=6) [XXXX....X.X.X...X..X.X.X..X...XX.....X......XX...X.X....X.X.] duration: 4.946554619s trace: https://jaeger.dev.storj.io/trace/31d935b4c9de8559 FAIL (2025-12-28 04:14:26 UTC) orbiter warning: context deadline exceeded Duration: 4.946582372s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 6.222354342s trace: https://jaeger.dev.storj.io/trace/3b8130f32e60318d FAIL (2025-12-28 04:14:09 UTC) orbiter warning: context deadline exceeded Duration: 6.222382085s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................................X......X...] duration: 33.106314212s trace: https://jaeger.dev.storj.io/trace/26c246d73008cc49 FAIL (2025-12-28 04:14:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== oise=true , address=145.79.227.43:4191 , piece_id=CLYBACZJDTAFTBEMEMSOXP4I22PDYSVIBX3AMNN4DISQUJWYFDMQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.179395486 (540h42m11.050478531s) [noise=true , address=145.79.227.9:4371 , piece_id=2ON3P6ZFEUEBFFLCHYY5BWAAJEFX43KKU5MU5TLBQHSZ52RCAYLA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.179823281 (540h42m11.050906336s) [noise=true , address=77.90.31.18:4819 , piece_id=FVZE5P55HHT3RGOI4M2OFV6NER4BNVGFSZOMDRADO5JYI54DRHLQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.180232490 (540h42m11.051315535s) [noise=true , address=77.90.31.17:4778 , piece_id=WHTKIVKEZNFREOTNDTLLBB24JGWKIXWBJKZ2EUM3QL2SM7Z4YXDA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.180945087 (540h42m11.052028132s) [noise=true , address=45.91.116.6:4003 , piece_id=H4VWZIMYVM2M3SX76LSS64ARSENBQL6OOTL5EY5XE4IAORO7OWGQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181189232 (540h42m11.052272277s) [noise=true , address=77.90.31.21:4906 , piece_id=SGYCZ2E6IMEJA332PTI7DQ4FWRJR6NG4OYRUP6H2KHXS2TKC5S2A, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181508139 (540h42m11.052591174s) [noise=true , address=77.90.31.15:4714 , piece_id=YVXSR2B74BDK4BV5QGLZLWOKVXV3TWKSW5M3KG323U7VUUBIQU3Q, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181597209 (540h42m11.052680254s) [noise=true , address=145.79.227.11:4755 , piece_id=F37J7GC6RB2PL3B4B6MFVAPB6IOGSJSKVQ5UNFHFSBJ7C3SABURQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181882131 (540h42m11.052965166s) [noise=true , address=145.79.227.33:4876 , piece_id=UEOY3HYI3WWJTXUSBALBXIGCMBC5M6XGR42SIOBXQNDCZ3UFMWVA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182132017 (540h42m11.053215052s) [noise=true , address=145.79.227.40:4053 , piece_id=V7R5MTCCKCGHV7KD4LAXFBUJI3HRGVAJUCVYL4WPL3M32CVQXOBA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182274007 (540h42m11.053357052s) [noise=true , address=145.79.227.51:4432 , piece_id=C5P63KNNZLSOBH772MRMYOJOQGLE7KXGELQN437HTVPAM72RXUCA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182451545 (540h42m11.05353458s) [noise=true , address=145.79.227.30:4784 , piece_id=UTH47JZWG3SGX3KXBFJJALALTPP6INYORUHDYSS6CNDEUQ6BDL7A, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182777516 (540h42m11.053860551s) [noise=true , address=64.49.13.169:4708 , piece_id=6OBO4M26AXDJXUTOBI66S2JJARDEM2CTTFHUIAOI4CX2PZITRWFQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182919286 (540h42m11.054002331s) [noise=true , address=145.79.227.6:4267 , piece_id=SDG3WH6TPR47OHG57NMVFAHOYVAL45XMJ7S22FZL6IFR6DS45HQQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.183101804 (540h42m11.054184839s) [noise=true , address=145.79.227.34:4924 , piece_id=R7MT4YS5IPGHOCNZX77ITOPXL57DOYD2HJXPJYYBGNVUWOT33JPA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.183514740 (540h42m11.054597785s) [noise=true , address=193.149.164.143:4696 , piece_id=OJQFP4YJ3EF52S2XQN5QPKMQJDS57QEOQQ4PUPARPGPAVSLGKMFA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:12.562530634 (540h42m12.433613679s) [upload=474194]: Next returned nil segment Dec 28 04:11:12.562544100 (540h42m12.433627135s) [upload=474194]: Waiting for error group managing segments... Dec 28 04:11:13.045899872 (540h42m12.916982917s) [noise=true , address=145.79.227.9:4371 , piece_id=2ON3P6ZFEUEBFFLCHYY5BWAAJEFX43KKU5MU5TLBQHSZ52RCAYLA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.068300556 (540h42m12.939383591s) [noise=true , address=145.79.227.11:4755 , piece_id=F37J7GC6RB2PL3B4B6MFVAPB6IOGSJSKVQ5UNFHFSBJ7C3SABURQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.073683536 (540h42m12.944766571s) [noise=true , address=145.79.227.40:4053 , piece_id=V7R5MTCCKCGHV7KD4LAXFBUJI3HRGVAJUCVYL4WPL3M32CVQXOBA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.125067800 (540h42m12.996150835s) [noise=true , address=145.79.227.6:4267 , piece_id=SDG3WH6TPR47OHG57NMVFAHOYVAL45XMJ7S22FZL6IFR6DS45HQQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.156395944 (540h42m13.027478989s) [noise=true , address=145.79.227.51:4432 , piece_id=C5P63KNNZLSOBH772MRMYOJOQGLE7KXGELQN437HTVPAM72RXUCA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.221620070 (540h42m13.092703105s) [noise=true , address=77.90.31.15:4714 , piece_id=YVXSR2B74BDK4BV5QGLZLWOKVXV3TWKSW5M3KG323U7VUUBIQU3Q, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.266169559 (540h42m13.137252594s) [noise=true , address=145.79.227.33:4876 , piece_id=UEOY3HYI3WWJTXUSBALBXIGCMBC5M6XGR42SIOBXQNDCZ3UFMWVA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.273253176 (540h42m13.144336221s) [noise=true , address=77.90.31.12:4604 , piece_id=DVQ53JEZL2GDTXFDDIWN4BLSJJHRZORII7UNF5HS3GIM5VJX3Y2Q, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.324930308 (540h42m13.196013343s) [noise=true , address=77.90.31.17:4778 , piece_id=WHTKIVKEZNFREOTNDTLLBB24JGWKIXWBJKZ2EUM3QL2SM7Z4YXDA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.333001758 (540h42m13.204084793s) [noise=true , address=77.90.31.18:4819 , piece_id=FVZE5P55HHT3RGOI4M2OFV6NER4BNVGFSZOMDRADO5JYI54DRHLQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.418239402 (540h42m13.289322437s) [noise=true , address=45.91.116.6:4003 , piece_id=H4VWZIMYVM2M3SX76LSS64ARSENBQL6OOTL5EY5XE4IAORO7OWGQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.460099681 (540h42m13.331182726s) [noise=true , address=77.90.31.21:4906 , piece_id=SGYCZ2E6IMEJA332PTI7DQ4FWRJR6NG4OYRUP6H2KHXS2TKC5S2A, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.860538777 (540h42m13.731621812s) [noise=true , address=193.149.164.143:4696 , piece_id=OJQFP4YJ3EF52S2XQN5QPKMQJDS57QEOQQ4PUPARPGPAVSLGKMFA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:14.127606583 (540h42m13.998689618s) [noise=true , address=145.79.227.34:4924 , piece_id=R7MT4YS5IPGHOCNZX77ITOPXL57DOYD2HJXPJYYBGNVUWOT33JPA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:14.127625258 (540h42m13.998708303s) [seg_pos={0 0}, upload=474194]: Segment reached optimal threshold of 14 pieces. Dec 28 04:11:14.127786726 (540h42m13.998869761s) [noise=true , address=45.91.116.4:4966 , piece_id=ZTHC25VIBEHSKRULTNAS25DK2DBVI7ATO46SDMMTON7PBBPRMWBQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DSHTtpwcx1h1utGXNou11K7Tcg1ZhkkMdMgY6BDD2ifG6LQ7S): protocol: expected piece hash; context canceled; EOF Dec 28 04:11:14.127792547 (540h42m13.998875582s) [noise=true , address=45.91.116.4:4966 , piece_id=ZTHC25VIBEHSKRULTNAS25DK2DBVI7ATO46SDMMTON7PBBPRMWBQ, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128017335 (540h42m13.99910037s) [noise=true , address=145.79.227.43:4191 , piece_id=CLYBACZJDTAFTBEMEMSOXP4I22PDYSVIBX3AMNN4DISQUJWYFDMQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw): protocol: expected piece hash; context canceled; EOF Dec 28 04:11:14.128023747 (540h42m13.999106782s) [noise=true , address=145.79.227.43:4191 , piece_id=CLYBACZJDTAFTBEMEMSOXP4I22PDYSVIBX3AMNN4DISQUJWYFDMQ, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128139027 (540h42m13.999222102s) [noise=true , address=64.49.13.169:4708 , piece_id=6OBO4M26AXDJXUTOBI66S2JJARDEM2CTTFHUIAOI4CX2PZITRWFQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:40936->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:40936->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40936->64.49.13.169:4708: use of closed network connection Dec 28 04:11:14.128150668 (540h42m13.999233733s) [noise=true , address=64.49.13.169:4708 , piece_id=6OBO4M26AXDJXUTOBI66S2JJARDEM2CTTFHUIAOI4CX2PZITRWFQ, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128292198 (540h42m13.999375243s) [noise=true , address=145.79.227.30:4784 , piece_id=UTH47JZWG3SGX3KXBFJJALALTPP6INYORUHDYSS6CNDEUQ6BDL7A, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pfXWex2WsuhmLb2smRpE46cHT9aipSTHW3e5oyELS1GhF1cFr): protocol: expected piece hash; context canceled; EOF Dec 28 04:11:14.128299212 (540h42m13.999382257s) [noise=true , address=145.79.227.30:4784 , piece_id=UTH47JZWG3SGX3KXBFJJALALTPP6INYORUHDYSS6CNDEUQ6BDL7A, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128395815 (540h42m13.99947885s) [seg_pos={0 0}, upload=474194]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 04:11:14.128488171 (540h42m13.999571206s) [upload=474194]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:11:14.128493561 (540h42m13.999576597s) [upload=474194]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:11:14.572773387 (540h42m14.443856452s) [upload=474194]: Done uploading segments... Dec 28 04:11:14.572800389 (540h42m14.443883454s) [upload=474194]: Upload finished. err: <nil> Duration: 33.106342978s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 21 out of 60 is failed (threshold=6) [.XXXX....X.X.X...X..X.X.X..X...XX.....X......XX...X.X....X.X] duration: 5.001304479s trace: https://jaeger.dev.storj.io/trace/31d935b4c9de8559 FAIL (2025-12-28 04:13:53 UTC) orbiter warning: context deadline exceeded Duration: 5.001320088s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 7.801700033s trace: https://jaeger.dev.storj.io/trace/3b8130f32e60318d FAIL (2025-12-28 04:13:39 UTC) orbiter warning: context deadline exceeded Duration: 7.801818439s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 51 out of 60 is failed (threshold=3) [XXX.XXXXX.XXXXXXXXXXXXXXXXX..XXXX.XXXX.XXXXX.XXXXXXX.XXXXXX.] duration: 20.268088212s trace: https://jaeger.dev.storj.io/trace/319ce7f5bba17d6e FAIL (2025-12-28 04:13:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ec 28 04:12:38.726334110 (540h43m38.597417155s) [upload=474215]: Starting upload Dec 28 04:12:38.726426036 (540h43m38.597509101s) [upload=474215]: Done starting upload Dec 28 04:12:38.726626287 (540h43m38.597709322s) [upload=474215]: Uploading segments... Dec 28 04:12:38.727559725 (540h43m38.59864277s) [upload=474215]: Got next segment. Inline: false Dec 28 04:12:38.727608728 (540h43m38.598691763s) [upload=474215]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 04:12:38.727613487 (540h43m38.598696522s) [upload=474215]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 04:12:38.970997057 (540h43m38.842080092s) [seg_pos={0 0}, upload=474215]: Begin upload segment... Dec 28 04:12:38.971044887 (540h43m38.842127932s) [seg_pos={0 0}, upload=474215]: Done begin upload segment. Dec 28 04:12:38.971082419 (540h43m38.842165454s) [noise=true , address=45.91.116.4:4937 , piece_id=QNPOBYDWPVDD3KBB7ER5O2R3CIKGT4SYK3B4GI2HMA4KIDD76UNA, seg_pos={0 0}, upload=474215]: Uploading piece... Dec 28 04:12:38.971098439 (540h43m38.842181474s) [noise=true , address=145.79.227.3:4150 , piece_id=GUEYHIOJ7CMOEQKFNGQS7HYPSW7X6ZD7USBS26VFH7NSDL7H7NAQ, seg_pos={0 0}, upload=474215]: Uploading piece... Dec 28 04:12:38.971231382 (540h43m38.842314427s) [noise=true , address=45.91.116.14:4190 , piece_id=S6EEFINMLCOEGAZ6PUBLMJ642CHGRPHBS3JUNHXJW3YEOVT5V46A, seg_pos={0 0}, upload=474215]: Uploading piece... Dec 28 04:12:38.971587631 (540h43m38.842670666s) [noise=true , address=45.91.116.13:4154 , piece_id=7XCD3R4BMWVXCFNKN3ULETEL4LFVL7P2CDWPEKEDQQV6QUTEZ5SQ, seg_pos={0 0}, upload=474215]: Uploading piece... Dec 28 04:12:38.971622718 (540h43m38.842705753s) [noise=true , address=145.79.227.12:4767 , piece_id=VW2GZ3QGK67R3JCXTA3E76ZRBHXRMPIJU2PSGEBEZM6CI2BSTAFQ, seg_pos={0 0}, upload=474215]: Uploading piece... Dec 28 04:12:38.971902561 (540h43m38.842985606s) [noise=true , address=77.90.31.14:4661 , piece_id=ZWNZGRAKYMNKWIHFJTZKFAXWQXY373UCFNXR3LBGTYUNYC64UUJQ, seg_pos={0 0}, upload=474215]: Uploading piece... Dec 28 04:12:38.972232859 (540h43m38.843315894s) [noise=true , address=145.79.227.47:4266 , piece_id=JL5JIICISIMMEKGD43AMGAFQH5RTSWVGPOOSZB5GFFXGFNHFMFYQ, seg_pos={0 0}, upload=474215]: Uploading piece... Dec 28 04:12:38.972534594 (540h43m38.843617629s) [noise=true , address=45.91.116.11:4120 , piece_id=RVKDXZAJP2T5DUQCHONPAHHFQBTYCERSRBFG2SB6BVXVHDHT47WQ, seg_pos={0 0}, upload=474215]: Uploading piece... Dec 28 04:12:38.972787115 (540h43m38.84387015s) [noise=true , address=45.91.116.6:4973 , piece_id=PQYA2MXXRTEZK55ABEGSACOBWK25XTUXH3W7II56OB6KV7FTAEPQ, seg_pos={0 0}, upload=474215]: Uploading piece... Dec 28 04:12:38.973050807 (540h43m38.844133852s) [noise=true , address=45.91.116.5:4299 , piece_id=O5XMGGHYJZCIO7U7H7GBRKPME77YBJGO6QCBYLSWK443D6CD6WFQ, seg_pos={0 0}, upload=474215]: Uploading piece... Dec 28 04:12:38.973313447 (540h43m38.844396482s) [noise=true , address=45.91.116.8:4045 , piece_id=K7FRQITAZJMJLK557DWGRI4YI2YVI5MHDEZC6YOTMQIETDD4WRMQ, seg_pos={0 0}, upload=474215]: Uploading piece... Dec 28 04:12:38.974132226 (540h43m38.845215271s) [noise=true , address=145.79.227.45:4230 , piece_id=63LRU4PIETFBWVDOU2TAR3FSJYBZGY4Q2CFQXVXW7ZEROXZ226GQ, seg_pos={0 0}, upload=474215]: Uploading piece... Dec 28 04:12:38.974473065 (540h43m38.8455561s) [noise=true , address=145.79.227.11:4732 , piece_id=CH4XXVYC7SUBJJVWCPIKAXDTMXX6JQ5OBQV7MWUP5VRTFRZVGIJA, seg_pos={0 0}, upload=474215]: Uploading piece... Dec 28 04:12:38.974825517 (540h43m38.845908552s) [noise=true , address=145.79.227.31:4798 , piece_id=XDGSPIDVQFPOSA5M4RJPKZP5TJBIDF2U6OJDGD5QW5UKCNT2IPAA, seg_pos={0 0}, upload=474215]: Uploading piece... Dec 28 04:12:38.975180542 (540h43m38.846263587s) [noise=true , address=77.90.31.17:4770 , piece_id=3VQAPUKV3TRP2WC4WMBYI37N7ZCPKU5YHPBCFTKGOYUVNMJKMAGA, seg_pos={0 0}, upload=474215]: Uploading piece... Dec 28 04:12:38.975495292 (540h43m38.846578337s) [noise=true , address=77.90.31.16:4736 , piece_id=EQKZQCL7PYGHGG33TCRTKQTJL2M4ZUP2H3AFT5V7XSQQB2KYGCWA, seg_pos={0 0}, upload=474215]: Uploading piece... Dec 28 04:12:40.773622607 (540h43m40.644705642s) [upload=474215]: Next returned nil segment Dec 28 04:12:40.773633357 (540h43m40.644716392s) [upload=474215]: Waiting for error group managing segments... Dec 28 04:12:41.071171164 (540h43m40.9422542s) [noise=true , address=145.79.227.45:4230 , piece_id=63LRU4PIETFBWVDOU2TAR3FSJYBZGY4Q2CFQXVXW7ZEROXZ226GQ, seg_pos={0 0}, upload=474215]: Done uploading piece. err: <nil> Dec 28 04:12:41.088540888 (540h43m40.959623923s) [noise=true , address=145.79.227.12:4767 , piece_id=VW2GZ3QGK67R3JCXTA3E76ZRBHXRMPIJU2PSGEBEZM6CI2BSTAFQ, seg_pos={0 0}, upload=474215]: Done uploading piece. err: <nil> Dec 28 04:12:41.095541709 (540h43m40.966624754s) [noise=true , address=145.79.227.31:4798 , piece_id=XDGSPIDVQFPOSA5M4RJPKZP5TJBIDF2U6OJDGD5QW5UKCNT2IPAA, seg_pos={0 0}, upload=474215]: Done uploading piece. err: <nil> Dec 28 04:12:41.095607585 (540h43m40.96669062s) [noise=true , address=145.79.227.11:4732 , piece_id=CH4XXVYC7SUBJJVWCPIKAXDTMXX6JQ5OBQV7MWUP5VRTFRZVGIJA, seg_pos={0 0}, upload=474215]: Done uploading piece. err: <nil> Dec 28 04:12:41.202004841 (540h43m41.073087876s) [noise=true , address=145.79.227.47:4266 , piece_id=JL5JIICISIMMEKGD43AMGAFQH5RTSWVGPOOSZB5GFFXGFNHFMFYQ, seg_pos={0 0}, upload=474215]: Done uploading piece. err: <nil> Dec 28 04:12:41.265377403 (540h43m41.136460438s) [noise=true , address=45.91.116.11:4120 , piece_id=RVKDXZAJP2T5DUQCHONPAHHFQBTYCERSRBFG2SB6BVXVHDHT47WQ, seg_pos={0 0}, upload=474215]: Done uploading piece. err: <nil> Dec 28 04:12:41.268392994 (540h43m41.139476029s) [noise=true , address=45.91.116.8:4045 , piece_id=K7FRQITAZJMJLK557DWGRI4YI2YVI5MHDEZC6YOTMQIETDD4WRMQ, seg_pos={0 0}, upload=474215]: Done uploading piece. err: <nil> Dec 28 04:12:41.276154464 (540h43m41.147237509s) [noise=true , address=77.90.31.14:4661 , piece_id=ZWNZGRAKYMNKWIHFJTZKFAXWQXY373UCFNXR3LBGTYUNYC64UUJQ, seg_pos={0 0}, upload=474215]: Done uploading piece. err: <nil> Dec 28 04:12:41.322814822 (540h43m41.193897867s) [noise=true , address=45.91.116.14:4190 , piece_id=S6EEFINMLCOEGAZ6PUBLMJ642CHGRPHBS3JUNHXJW3YEOVT5V46A, seg_pos={0 0}, upload=474215]: Done uploading piece. err: <nil> Dec 28 04:12:41.325496728 (540h43m41.196579763s) [noise=true , address=45.91.116.5:4299 , piece_id=O5XMGGHYJZCIO7U7H7GBRKPME77YBJGO6QCBYLSWK443D6CD6WFQ, seg_pos={0 0}, upload=474215]: Done uploading piece. err: <nil> Dec 28 04:12:41.329321932 (540h43m41.200404967s) [noise=true , address=45.91.116.6:4973 , piece_id=PQYA2MXXRTEZK55ABEGSACOBWK25XTUXH3W7II56OB6KV7FTAEPQ, seg_pos={0 0}, upload=474215]: Done uploading piece. err: <nil> Dec 28 04:12:41.340599787 (540h43m41.211682822s) [noise=true , address=45.91.116.4:4937 , piece_id=QNPOBYDWPVDD3KBB7ER5O2R3CIKGT4SYK3B4GI2HMA4KIDD76UNA, seg_pos={0 0}, upload=474215]: Done uploading piece. err: <nil> Dec 28 04:12:41.364954202 (540h43m41.236037237s) [noise=true , address=77.90.31.17:4770 , piece_id=3VQAPUKV3TRP2WC4WMBYI37N7ZCPKU5YHPBCFTKGOYUVNMJKMAGA, seg_pos={0 0}, upload=474215]: Done uploading piece. err: <nil> Dec 28 04:12:41.364983147 (540h43m41.236066182s) [seg_pos={0 0}, upload=474215]: Segment reached optimal threshold of 13 pieces. Dec 28 04:12:41.365269041 (540h43m41.236352076s) [noise=true , address=45.91.116.13:4154 , piece_id=7XCD3R4BMWVXCFNKN3ULETEL4LFVL7P2CDWPEKEDQQV6QUTEZ5SQ, seg_pos={0 0}, upload=474215]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 28 04:12:41.365276335 (540h43m41.23635938s) [noise=true , address=45.91.116.13:4154 , piece_id=7XCD3R4BMWVXCFNKN3ULETEL4LFVL7P2CDWPEKEDQQV6QUTEZ5SQ, seg_pos={0 0}, upload=474215]: Long tail cancellation due to optimal threshold Dec 28 04:12:41.365317864 (540h43m41.236400899s) [noise=true , address=145.79.227.3:4150 , piece_id=GUEYHIOJ7CMOEQKFNGQS7HYPSW7X6ZD7USBS26VFH7NSDL7H7NAQ, seg_pos={0 0}, upload=474215]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: write tcp 188.166.225.170:47130->145.79.227.3:4150: use of closed network connection; write tcp 188.166.225.170:47130->145.79.227.3:4150: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47130->145.79.227.3:4150: use of closed network connection Dec 28 04:12:41.365333905 (540h43m41.23641694s) [noise=true , address=145.79.227.3:4150 , piece_id=GUEYHIOJ7CMOEQKFNGQS7HYPSW7X6ZD7USBS26VFH7NSDL7H7NAQ, seg_pos={0 0}, upload=474215]: Long tail cancellation due to optimal threshold Dec 28 04:12:41.365419728 (540h43m41.236502763s) [noise=true , address=77.90.31.16:4736 , piece_id=EQKZQCL7PYGHGG33TCRTKQTJL2M4ZUP2H3AFT5V7XSQQB2KYGCWA, seg_pos={0 0}, upload=474215]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 28 04:12:41.365426520 (540h43m41.236509566s) [noise=true , address=77.90.31.16:4736 , piece_id=EQKZQCL7PYGHGG33TCRTKQTJL2M4ZUP2H3AFT5V7XSQQB2KYGCWA, seg_pos={0 0}, upload=474215]: Long tail cancellation due to optimal threshold Dec 28 04:12:41.365443583 (540h43m41.236526629s) [seg_pos={0 0}, upload=474215]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 04:12:41.365539597 (540h43m41.236622632s) [upload=474215]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:12:41.365544826 (540h43m41.236627861s) [upload=474215]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:12:41.651265873 (540h43m41.522348908s) [upload=474215]: Done uploading segments... Dec 28 04:12:41.651287303 (540h43m41.522370348s) [upload=474215]: Upload finished. err: <nil> Duration: 20.268114862s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................................X......X..] duration: 21.258228177s trace: https://jaeger.dev.storj.io/trace/26c246d73008cc49 FAIL (2025-12-28 04:13:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== oise=true , address=145.79.227.43:4191 , piece_id=CLYBACZJDTAFTBEMEMSOXP4I22PDYSVIBX3AMNN4DISQUJWYFDMQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.179395486 (540h42m11.050478531s) [noise=true , address=145.79.227.9:4371 , piece_id=2ON3P6ZFEUEBFFLCHYY5BWAAJEFX43KKU5MU5TLBQHSZ52RCAYLA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.179823281 (540h42m11.050906336s) [noise=true , address=77.90.31.18:4819 , piece_id=FVZE5P55HHT3RGOI4M2OFV6NER4BNVGFSZOMDRADO5JYI54DRHLQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.180232490 (540h42m11.051315535s) [noise=true , address=77.90.31.17:4778 , piece_id=WHTKIVKEZNFREOTNDTLLBB24JGWKIXWBJKZ2EUM3QL2SM7Z4YXDA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.180945087 (540h42m11.052028132s) [noise=true , address=45.91.116.6:4003 , piece_id=H4VWZIMYVM2M3SX76LSS64ARSENBQL6OOTL5EY5XE4IAORO7OWGQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181189232 (540h42m11.052272277s) [noise=true , address=77.90.31.21:4906 , piece_id=SGYCZ2E6IMEJA332PTI7DQ4FWRJR6NG4OYRUP6H2KHXS2TKC5S2A, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181508139 (540h42m11.052591174s) [noise=true , address=77.90.31.15:4714 , piece_id=YVXSR2B74BDK4BV5QGLZLWOKVXV3TWKSW5M3KG323U7VUUBIQU3Q, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181597209 (540h42m11.052680254s) [noise=true , address=145.79.227.11:4755 , piece_id=F37J7GC6RB2PL3B4B6MFVAPB6IOGSJSKVQ5UNFHFSBJ7C3SABURQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181882131 (540h42m11.052965166s) [noise=true , address=145.79.227.33:4876 , piece_id=UEOY3HYI3WWJTXUSBALBXIGCMBC5M6XGR42SIOBXQNDCZ3UFMWVA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182132017 (540h42m11.053215052s) [noise=true , address=145.79.227.40:4053 , piece_id=V7R5MTCCKCGHV7KD4LAXFBUJI3HRGVAJUCVYL4WPL3M32CVQXOBA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182274007 (540h42m11.053357052s) [noise=true , address=145.79.227.51:4432 , piece_id=C5P63KNNZLSOBH772MRMYOJOQGLE7KXGELQN437HTVPAM72RXUCA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182451545 (540h42m11.05353458s) [noise=true , address=145.79.227.30:4784 , piece_id=UTH47JZWG3SGX3KXBFJJALALTPP6INYORUHDYSS6CNDEUQ6BDL7A, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182777516 (540h42m11.053860551s) [noise=true , address=64.49.13.169:4708 , piece_id=6OBO4M26AXDJXUTOBI66S2JJARDEM2CTTFHUIAOI4CX2PZITRWFQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182919286 (540h42m11.054002331s) [noise=true , address=145.79.227.6:4267 , piece_id=SDG3WH6TPR47OHG57NMVFAHOYVAL45XMJ7S22FZL6IFR6DS45HQQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.183101804 (540h42m11.054184839s) [noise=true , address=145.79.227.34:4924 , piece_id=R7MT4YS5IPGHOCNZX77ITOPXL57DOYD2HJXPJYYBGNVUWOT33JPA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.183514740 (540h42m11.054597785s) [noise=true , address=193.149.164.143:4696 , piece_id=OJQFP4YJ3EF52S2XQN5QPKMQJDS57QEOQQ4PUPARPGPAVSLGKMFA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:12.562530634 (540h42m12.433613679s) [upload=474194]: Next returned nil segment Dec 28 04:11:12.562544100 (540h42m12.433627135s) [upload=474194]: Waiting for error group managing segments... Dec 28 04:11:13.045899872 (540h42m12.916982917s) [noise=true , address=145.79.227.9:4371 , piece_id=2ON3P6ZFEUEBFFLCHYY5BWAAJEFX43KKU5MU5TLBQHSZ52RCAYLA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.068300556 (540h42m12.939383591s) [noise=true , address=145.79.227.11:4755 , piece_id=F37J7GC6RB2PL3B4B6MFVAPB6IOGSJSKVQ5UNFHFSBJ7C3SABURQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.073683536 (540h42m12.944766571s) [noise=true , address=145.79.227.40:4053 , piece_id=V7R5MTCCKCGHV7KD4LAXFBUJI3HRGVAJUCVYL4WPL3M32CVQXOBA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.125067800 (540h42m12.996150835s) [noise=true , address=145.79.227.6:4267 , piece_id=SDG3WH6TPR47OHG57NMVFAHOYVAL45XMJ7S22FZL6IFR6DS45HQQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.156395944 (540h42m13.027478989s) [noise=true , address=145.79.227.51:4432 , piece_id=C5P63KNNZLSOBH772MRMYOJOQGLE7KXGELQN437HTVPAM72RXUCA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.221620070 (540h42m13.092703105s) [noise=true , address=77.90.31.15:4714 , piece_id=YVXSR2B74BDK4BV5QGLZLWOKVXV3TWKSW5M3KG323U7VUUBIQU3Q, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.266169559 (540h42m13.137252594s) [noise=true , address=145.79.227.33:4876 , piece_id=UEOY3HYI3WWJTXUSBALBXIGCMBC5M6XGR42SIOBXQNDCZ3UFMWVA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.273253176 (540h42m13.144336221s) [noise=true , address=77.90.31.12:4604 , piece_id=DVQ53JEZL2GDTXFDDIWN4BLSJJHRZORII7UNF5HS3GIM5VJX3Y2Q, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.324930308 (540h42m13.196013343s) [noise=true , address=77.90.31.17:4778 , piece_id=WHTKIVKEZNFREOTNDTLLBB24JGWKIXWBJKZ2EUM3QL2SM7Z4YXDA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.333001758 (540h42m13.204084793s) [noise=true , address=77.90.31.18:4819 , piece_id=FVZE5P55HHT3RGOI4M2OFV6NER4BNVGFSZOMDRADO5JYI54DRHLQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.418239402 (540h42m13.289322437s) [noise=true , address=45.91.116.6:4003 , piece_id=H4VWZIMYVM2M3SX76LSS64ARSENBQL6OOTL5EY5XE4IAORO7OWGQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.460099681 (540h42m13.331182726s) [noise=true , address=77.90.31.21:4906 , piece_id=SGYCZ2E6IMEJA332PTI7DQ4FWRJR6NG4OYRUP6H2KHXS2TKC5S2A, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.860538777 (540h42m13.731621812s) [noise=true , address=193.149.164.143:4696 , piece_id=OJQFP4YJ3EF52S2XQN5QPKMQJDS57QEOQQ4PUPARPGPAVSLGKMFA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:14.127606583 (540h42m13.998689618s) [noise=true , address=145.79.227.34:4924 , piece_id=R7MT4YS5IPGHOCNZX77ITOPXL57DOYD2HJXPJYYBGNVUWOT33JPA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:14.127625258 (540h42m13.998708303s) [seg_pos={0 0}, upload=474194]: Segment reached optimal threshold of 14 pieces. Dec 28 04:11:14.127786726 (540h42m13.998869761s) [noise=true , address=45.91.116.4:4966 , piece_id=ZTHC25VIBEHSKRULTNAS25DK2DBVI7ATO46SDMMTON7PBBPRMWBQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DSHTtpwcx1h1utGXNou11K7Tcg1ZhkkMdMgY6BDD2ifG6LQ7S): protocol: expected piece hash; context canceled; EOF Dec 28 04:11:14.127792547 (540h42m13.998875582s) [noise=true , address=45.91.116.4:4966 , piece_id=ZTHC25VIBEHSKRULTNAS25DK2DBVI7ATO46SDMMTON7PBBPRMWBQ, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128017335 (540h42m13.99910037s) [noise=true , address=145.79.227.43:4191 , piece_id=CLYBACZJDTAFTBEMEMSOXP4I22PDYSVIBX3AMNN4DISQUJWYFDMQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw): protocol: expected piece hash; context canceled; EOF Dec 28 04:11:14.128023747 (540h42m13.999106782s) [noise=true , address=145.79.227.43:4191 , piece_id=CLYBACZJDTAFTBEMEMSOXP4I22PDYSVIBX3AMNN4DISQUJWYFDMQ, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128139027 (540h42m13.999222102s) [noise=true , address=64.49.13.169:4708 , piece_id=6OBO4M26AXDJXUTOBI66S2JJARDEM2CTTFHUIAOI4CX2PZITRWFQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:40936->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:40936->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40936->64.49.13.169:4708: use of closed network connection Dec 28 04:11:14.128150668 (540h42m13.999233733s) [noise=true , address=64.49.13.169:4708 , piece_id=6OBO4M26AXDJXUTOBI66S2JJARDEM2CTTFHUIAOI4CX2PZITRWFQ, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128292198 (540h42m13.999375243s) [noise=true , address=145.79.227.30:4784 , piece_id=UTH47JZWG3SGX3KXBFJJALALTPP6INYORUHDYSS6CNDEUQ6BDL7A, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pfXWex2WsuhmLb2smRpE46cHT9aipSTHW3e5oyELS1GhF1cFr): protocol: expected piece hash; context canceled; EOF Dec 28 04:11:14.128299212 (540h42m13.999382257s) [noise=true , address=145.79.227.30:4784 , piece_id=UTH47JZWG3SGX3KXBFJJALALTPP6INYORUHDYSS6CNDEUQ6BDL7A, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128395815 (540h42m13.99947885s) [seg_pos={0 0}, upload=474194]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 04:11:14.128488171 (540h42m13.999571206s) [upload=474194]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:11:14.128493561 (540h42m13.999576597s) [upload=474194]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:11:14.572773387 (540h42m14.443856452s) [upload=474194]: Done uploading segments... Dec 28 04:11:14.572800389 (540h42m14.443883454s) [upload=474194]: Upload finished. err: <nil> Duration: 21.258257022s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 21 out of 60 is failed (threshold=6) [X.XXXX....X.X.X...X..X.X.X..X...XX.....X......XX...X.X....X.] duration: 5.000850685s trace: https://jaeger.dev.storj.io/trace/f61ff6946e8a16e FAIL (2025-12-28 04:13:14 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000877956s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 52 out of 60 is failed (threshold=3) [XXXX.XXXXX.XXXXXXXXXXXXXXXXX..XXXX.XXXX.XXXXX.XXXXXXX.XXXXXX] duration: 30.003317978s trace: https://jaeger.dev.storj.io/trace/319ce7f5bba17d6e FAIL (2025-12-28 04:13:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ec 28 04:12:38.726334110 (540h43m38.597417155s) [upload=474215]: Starting upload Dec 28 04:12:38.726426036 (540h43m38.597509101s) [upload=474215]: Done starting upload Dec 28 04:12:38.726626287 (540h43m38.597709322s) [upload=474215]: Uploading segments... Dec 28 04:12:38.727559725 (540h43m38.59864277s) [upload=474215]: Got next segment. Inline: false Dec 28 04:12:38.727608728 (540h43m38.598691763s) [upload=474215]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 04:12:38.727613487 (540h43m38.598696522s) [upload=474215]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 04:12:38.970997057 (540h43m38.842080092s) [seg_pos={0 0}, upload=474215]: Begin upload segment... Dec 28 04:12:38.971044887 (540h43m38.842127932s) [seg_pos={0 0}, upload=474215]: Done begin upload segment. Dec 28 04:12:38.971082419 (540h43m38.842165454s) [noise=true , address=45.91.116.4:4937 , piece_id=QNPOBYDWPVDD3KBB7ER5O2R3CIKGT4SYK3B4GI2HMA4KIDD76UNA, seg_pos={0 0}, upload=474215]: Uploading piece... Dec 28 04:12:38.971098439 (540h43m38.842181474s) [noise=true , address=145.79.227.3:4150 , piece_id=GUEYHIOJ7CMOEQKFNGQS7HYPSW7X6ZD7USBS26VFH7NSDL7H7NAQ, seg_pos={0 0}, upload=474215]: Uploading piece... Dec 28 04:12:38.971231382 (540h43m38.842314427s) [noise=true , address=45.91.116.14:4190 , piece_id=S6EEFINMLCOEGAZ6PUBLMJ642CHGRPHBS3JUNHXJW3YEOVT5V46A, seg_pos={0 0}, upload=474215]: Uploading piece... Dec 28 04:12:38.971587631 (540h43m38.842670666s) [noise=true , address=45.91.116.13:4154 , piece_id=7XCD3R4BMWVXCFNKN3ULETEL4LFVL7P2CDWPEKEDQQV6QUTEZ5SQ, seg_pos={0 0}, upload=474215]: Uploading piece... Dec 28 04:12:38.971622718 (540h43m38.842705753s) [noise=true , address=145.79.227.12:4767 , piece_id=VW2GZ3QGK67R3JCXTA3E76ZRBHXRMPIJU2PSGEBEZM6CI2BSTAFQ, seg_pos={0 0}, upload=474215]: Uploading piece... Dec 28 04:12:38.971902561 (540h43m38.842985606s) [noise=true , address=77.90.31.14:4661 , piece_id=ZWNZGRAKYMNKWIHFJTZKFAXWQXY373UCFNXR3LBGTYUNYC64UUJQ, seg_pos={0 0}, upload=474215]: Uploading piece... Dec 28 04:12:38.972232859 (540h43m38.843315894s) [noise=true , address=145.79.227.47:4266 , piece_id=JL5JIICISIMMEKGD43AMGAFQH5RTSWVGPOOSZB5GFFXGFNHFMFYQ, seg_pos={0 0}, upload=474215]: Uploading piece... Dec 28 04:12:38.972534594 (540h43m38.843617629s) [noise=true , address=45.91.116.11:4120 , piece_id=RVKDXZAJP2T5DUQCHONPAHHFQBTYCERSRBFG2SB6BVXVHDHT47WQ, seg_pos={0 0}, upload=474215]: Uploading piece... Dec 28 04:12:38.972787115 (540h43m38.84387015s) [noise=true , address=45.91.116.6:4973 , piece_id=PQYA2MXXRTEZK55ABEGSACOBWK25XTUXH3W7II56OB6KV7FTAEPQ, seg_pos={0 0}, upload=474215]: Uploading piece... Dec 28 04:12:38.973050807 (540h43m38.844133852s) [noise=true , address=45.91.116.5:4299 , piece_id=O5XMGGHYJZCIO7U7H7GBRKPME77YBJGO6QCBYLSWK443D6CD6WFQ, seg_pos={0 0}, upload=474215]: Uploading piece... Dec 28 04:12:38.973313447 (540h43m38.844396482s) [noise=true , address=45.91.116.8:4045 , piece_id=K7FRQITAZJMJLK557DWGRI4YI2YVI5MHDEZC6YOTMQIETDD4WRMQ, seg_pos={0 0}, upload=474215]: Uploading piece... Dec 28 04:12:38.974132226 (540h43m38.845215271s) [noise=true , address=145.79.227.45:4230 , piece_id=63LRU4PIETFBWVDOU2TAR3FSJYBZGY4Q2CFQXVXW7ZEROXZ226GQ, seg_pos={0 0}, upload=474215]: Uploading piece... Dec 28 04:12:38.974473065 (540h43m38.8455561s) [noise=true , address=145.79.227.11:4732 , piece_id=CH4XXVYC7SUBJJVWCPIKAXDTMXX6JQ5OBQV7MWUP5VRTFRZVGIJA, seg_pos={0 0}, upload=474215]: Uploading piece... Dec 28 04:12:38.974825517 (540h43m38.845908552s) [noise=true , address=145.79.227.31:4798 , piece_id=XDGSPIDVQFPOSA5M4RJPKZP5TJBIDF2U6OJDGD5QW5UKCNT2IPAA, seg_pos={0 0}, upload=474215]: Uploading piece... Dec 28 04:12:38.975180542 (540h43m38.846263587s) [noise=true , address=77.90.31.17:4770 , piece_id=3VQAPUKV3TRP2WC4WMBYI37N7ZCPKU5YHPBCFTKGOYUVNMJKMAGA, seg_pos={0 0}, upload=474215]: Uploading piece... Dec 28 04:12:38.975495292 (540h43m38.846578337s) [noise=true , address=77.90.31.16:4736 , piece_id=EQKZQCL7PYGHGG33TCRTKQTJL2M4ZUP2H3AFT5V7XSQQB2KYGCWA, seg_pos={0 0}, upload=474215]: Uploading piece... Dec 28 04:12:40.773622607 (540h43m40.644705642s) [upload=474215]: Next returned nil segment Dec 28 04:12:40.773633357 (540h43m40.644716392s) [upload=474215]: Waiting for error group managing segments... Dec 28 04:12:41.071171164 (540h43m40.9422542s) [noise=true , address=145.79.227.45:4230 , piece_id=63LRU4PIETFBWVDOU2TAR3FSJYBZGY4Q2CFQXVXW7ZEROXZ226GQ, seg_pos={0 0}, upload=474215]: Done uploading piece. err: <nil> Dec 28 04:12:41.088540888 (540h43m40.959623923s) [noise=true , address=145.79.227.12:4767 , piece_id=VW2GZ3QGK67R3JCXTA3E76ZRBHXRMPIJU2PSGEBEZM6CI2BSTAFQ, seg_pos={0 0}, upload=474215]: Done uploading piece. err: <nil> Dec 28 04:12:41.095541709 (540h43m40.966624754s) [noise=true , address=145.79.227.31:4798 , piece_id=XDGSPIDVQFPOSA5M4RJPKZP5TJBIDF2U6OJDGD5QW5UKCNT2IPAA, seg_pos={0 0}, upload=474215]: Done uploading piece. err: <nil> Dec 28 04:12:41.095607585 (540h43m40.96669062s) [noise=true , address=145.79.227.11:4732 , piece_id=CH4XXVYC7SUBJJVWCPIKAXDTMXX6JQ5OBQV7MWUP5VRTFRZVGIJA, seg_pos={0 0}, upload=474215]: Done uploading piece. err: <nil> Dec 28 04:12:41.202004841 (540h43m41.073087876s) [noise=true , address=145.79.227.47:4266 , piece_id=JL5JIICISIMMEKGD43AMGAFQH5RTSWVGPOOSZB5GFFXGFNHFMFYQ, seg_pos={0 0}, upload=474215]: Done uploading piece. err: <nil> Dec 28 04:12:41.265377403 (540h43m41.136460438s) [noise=true , address=45.91.116.11:4120 , piece_id=RVKDXZAJP2T5DUQCHONPAHHFQBTYCERSRBFG2SB6BVXVHDHT47WQ, seg_pos={0 0}, upload=474215]: Done uploading piece. err: <nil> Dec 28 04:12:41.268392994 (540h43m41.139476029s) [noise=true , address=45.91.116.8:4045 , piece_id=K7FRQITAZJMJLK557DWGRI4YI2YVI5MHDEZC6YOTMQIETDD4WRMQ, seg_pos={0 0}, upload=474215]: Done uploading piece. err: <nil> Dec 28 04:12:41.276154464 (540h43m41.147237509s) [noise=true , address=77.90.31.14:4661 , piece_id=ZWNZGRAKYMNKWIHFJTZKFAXWQXY373UCFNXR3LBGTYUNYC64UUJQ, seg_pos={0 0}, upload=474215]: Done uploading piece. err: <nil> Dec 28 04:12:41.322814822 (540h43m41.193897867s) [noise=true , address=45.91.116.14:4190 , piece_id=S6EEFINMLCOEGAZ6PUBLMJ642CHGRPHBS3JUNHXJW3YEOVT5V46A, seg_pos={0 0}, upload=474215]: Done uploading piece. err: <nil> Dec 28 04:12:41.325496728 (540h43m41.196579763s) [noise=true , address=45.91.116.5:4299 , piece_id=O5XMGGHYJZCIO7U7H7GBRKPME77YBJGO6QCBYLSWK443D6CD6WFQ, seg_pos={0 0}, upload=474215]: Done uploading piece. err: <nil> Dec 28 04:12:41.329321932 (540h43m41.200404967s) [noise=true , address=45.91.116.6:4973 , piece_id=PQYA2MXXRTEZK55ABEGSACOBWK25XTUXH3W7II56OB6KV7FTAEPQ, seg_pos={0 0}, upload=474215]: Done uploading piece. err: <nil> Dec 28 04:12:41.340599787 (540h43m41.211682822s) [noise=true , address=45.91.116.4:4937 , piece_id=QNPOBYDWPVDD3KBB7ER5O2R3CIKGT4SYK3B4GI2HMA4KIDD76UNA, seg_pos={0 0}, upload=474215]: Done uploading piece. err: <nil> Dec 28 04:12:41.364954202 (540h43m41.236037237s) [noise=true , address=77.90.31.17:4770 , piece_id=3VQAPUKV3TRP2WC4WMBYI37N7ZCPKU5YHPBCFTKGOYUVNMJKMAGA, seg_pos={0 0}, upload=474215]: Done uploading piece. err: <nil> Dec 28 04:12:41.364983147 (540h43m41.236066182s) [seg_pos={0 0}, upload=474215]: Segment reached optimal threshold of 13 pieces. Dec 28 04:12:41.365269041 (540h43m41.236352076s) [noise=true , address=45.91.116.13:4154 , piece_id=7XCD3R4BMWVXCFNKN3ULETEL4LFVL7P2CDWPEKEDQQV6QUTEZ5SQ, seg_pos={0 0}, upload=474215]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 28 04:12:41.365276335 (540h43m41.23635938s) [noise=true , address=45.91.116.13:4154 , piece_id=7XCD3R4BMWVXCFNKN3ULETEL4LFVL7P2CDWPEKEDQQV6QUTEZ5SQ, seg_pos={0 0}, upload=474215]: Long tail cancellation due to optimal threshold Dec 28 04:12:41.365317864 (540h43m41.236400899s) [noise=true , address=145.79.227.3:4150 , piece_id=GUEYHIOJ7CMOEQKFNGQS7HYPSW7X6ZD7USBS26VFH7NSDL7H7NAQ, seg_pos={0 0}, upload=474215]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: write tcp 188.166.225.170:47130->145.79.227.3:4150: use of closed network connection; write tcp 188.166.225.170:47130->145.79.227.3:4150: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47130->145.79.227.3:4150: use of closed network connection Dec 28 04:12:41.365333905 (540h43m41.23641694s) [noise=true , address=145.79.227.3:4150 , piece_id=GUEYHIOJ7CMOEQKFNGQS7HYPSW7X6ZD7USBS26VFH7NSDL7H7NAQ, seg_pos={0 0}, upload=474215]: Long tail cancellation due to optimal threshold Dec 28 04:12:41.365419728 (540h43m41.236502763s) [noise=true , address=77.90.31.16:4736 , piece_id=EQKZQCL7PYGHGG33TCRTKQTJL2M4ZUP2H3AFT5V7XSQQB2KYGCWA, seg_pos={0 0}, upload=474215]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 28 04:12:41.365426520 (540h43m41.236509566s) [noise=true , address=77.90.31.16:4736 , piece_id=EQKZQCL7PYGHGG33TCRTKQTJL2M4ZUP2H3AFT5V7XSQQB2KYGCWA, seg_pos={0 0}, upload=474215]: Long tail cancellation due to optimal threshold Dec 28 04:12:41.365443583 (540h43m41.236526629s) [seg_pos={0 0}, upload=474215]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 04:12:41.365539597 (540h43m41.236622632s) [upload=474215]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:12:41.365544826 (540h43m41.236627861s) [upload=474215]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:12:41.651265873 (540h43m41.522348908s) [upload=474215]: Done uploading segments... Dec 28 04:12:41.651287303 (540h43m41.522370348s) [upload=474215]: Upload finished. err: <nil> Duration: 30.003336062s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................................X......X.] duration: 31.622339776s trace: https://jaeger.dev.storj.io/trace/26c246d73008cc49 FAIL (2025-12-28 04:12:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== oise=true , address=145.79.227.43:4191 , piece_id=CLYBACZJDTAFTBEMEMSOXP4I22PDYSVIBX3AMNN4DISQUJWYFDMQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.179395486 (540h42m11.050478531s) [noise=true , address=145.79.227.9:4371 , piece_id=2ON3P6ZFEUEBFFLCHYY5BWAAJEFX43KKU5MU5TLBQHSZ52RCAYLA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.179823281 (540h42m11.050906336s) [noise=true , address=77.90.31.18:4819 , piece_id=FVZE5P55HHT3RGOI4M2OFV6NER4BNVGFSZOMDRADO5JYI54DRHLQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.180232490 (540h42m11.051315535s) [noise=true , address=77.90.31.17:4778 , piece_id=WHTKIVKEZNFREOTNDTLLBB24JGWKIXWBJKZ2EUM3QL2SM7Z4YXDA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.180945087 (540h42m11.052028132s) [noise=true , address=45.91.116.6:4003 , piece_id=H4VWZIMYVM2M3SX76LSS64ARSENBQL6OOTL5EY5XE4IAORO7OWGQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181189232 (540h42m11.052272277s) [noise=true , address=77.90.31.21:4906 , piece_id=SGYCZ2E6IMEJA332PTI7DQ4FWRJR6NG4OYRUP6H2KHXS2TKC5S2A, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181508139 (540h42m11.052591174s) [noise=true , address=77.90.31.15:4714 , piece_id=YVXSR2B74BDK4BV5QGLZLWOKVXV3TWKSW5M3KG323U7VUUBIQU3Q, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181597209 (540h42m11.052680254s) [noise=true , address=145.79.227.11:4755 , piece_id=F37J7GC6RB2PL3B4B6MFVAPB6IOGSJSKVQ5UNFHFSBJ7C3SABURQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181882131 (540h42m11.052965166s) [noise=true , address=145.79.227.33:4876 , piece_id=UEOY3HYI3WWJTXUSBALBXIGCMBC5M6XGR42SIOBXQNDCZ3UFMWVA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182132017 (540h42m11.053215052s) [noise=true , address=145.79.227.40:4053 , piece_id=V7R5MTCCKCGHV7KD4LAXFBUJI3HRGVAJUCVYL4WPL3M32CVQXOBA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182274007 (540h42m11.053357052s) [noise=true , address=145.79.227.51:4432 , piece_id=C5P63KNNZLSOBH772MRMYOJOQGLE7KXGELQN437HTVPAM72RXUCA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182451545 (540h42m11.05353458s) [noise=true , address=145.79.227.30:4784 , piece_id=UTH47JZWG3SGX3KXBFJJALALTPP6INYORUHDYSS6CNDEUQ6BDL7A, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182777516 (540h42m11.053860551s) [noise=true , address=64.49.13.169:4708 , piece_id=6OBO4M26AXDJXUTOBI66S2JJARDEM2CTTFHUIAOI4CX2PZITRWFQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182919286 (540h42m11.054002331s) [noise=true , address=145.79.227.6:4267 , piece_id=SDG3WH6TPR47OHG57NMVFAHOYVAL45XMJ7S22FZL6IFR6DS45HQQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.183101804 (540h42m11.054184839s) [noise=true , address=145.79.227.34:4924 , piece_id=R7MT4YS5IPGHOCNZX77ITOPXL57DOYD2HJXPJYYBGNVUWOT33JPA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.183514740 (540h42m11.054597785s) [noise=true , address=193.149.164.143:4696 , piece_id=OJQFP4YJ3EF52S2XQN5QPKMQJDS57QEOQQ4PUPARPGPAVSLGKMFA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:12.562530634 (540h42m12.433613679s) [upload=474194]: Next returned nil segment Dec 28 04:11:12.562544100 (540h42m12.433627135s) [upload=474194]: Waiting for error group managing segments... Dec 28 04:11:13.045899872 (540h42m12.916982917s) [noise=true , address=145.79.227.9:4371 , piece_id=2ON3P6ZFEUEBFFLCHYY5BWAAJEFX43KKU5MU5TLBQHSZ52RCAYLA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.068300556 (540h42m12.939383591s) [noise=true , address=145.79.227.11:4755 , piece_id=F37J7GC6RB2PL3B4B6MFVAPB6IOGSJSKVQ5UNFHFSBJ7C3SABURQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.073683536 (540h42m12.944766571s) [noise=true , address=145.79.227.40:4053 , piece_id=V7R5MTCCKCGHV7KD4LAXFBUJI3HRGVAJUCVYL4WPL3M32CVQXOBA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.125067800 (540h42m12.996150835s) [noise=true , address=145.79.227.6:4267 , piece_id=SDG3WH6TPR47OHG57NMVFAHOYVAL45XMJ7S22FZL6IFR6DS45HQQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.156395944 (540h42m13.027478989s) [noise=true , address=145.79.227.51:4432 , piece_id=C5P63KNNZLSOBH772MRMYOJOQGLE7KXGELQN437HTVPAM72RXUCA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.221620070 (540h42m13.092703105s) [noise=true , address=77.90.31.15:4714 , piece_id=YVXSR2B74BDK4BV5QGLZLWOKVXV3TWKSW5M3KG323U7VUUBIQU3Q, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.266169559 (540h42m13.137252594s) [noise=true , address=145.79.227.33:4876 , piece_id=UEOY3HYI3WWJTXUSBALBXIGCMBC5M6XGR42SIOBXQNDCZ3UFMWVA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.273253176 (540h42m13.144336221s) [noise=true , address=77.90.31.12:4604 , piece_id=DVQ53JEZL2GDTXFDDIWN4BLSJJHRZORII7UNF5HS3GIM5VJX3Y2Q, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.324930308 (540h42m13.196013343s) [noise=true , address=77.90.31.17:4778 , piece_id=WHTKIVKEZNFREOTNDTLLBB24JGWKIXWBJKZ2EUM3QL2SM7Z4YXDA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.333001758 (540h42m13.204084793s) [noise=true , address=77.90.31.18:4819 , piece_id=FVZE5P55HHT3RGOI4M2OFV6NER4BNVGFSZOMDRADO5JYI54DRHLQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.418239402 (540h42m13.289322437s) [noise=true , address=45.91.116.6:4003 , piece_id=H4VWZIMYVM2M3SX76LSS64ARSENBQL6OOTL5EY5XE4IAORO7OWGQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.460099681 (540h42m13.331182726s) [noise=true , address=77.90.31.21:4906 , piece_id=SGYCZ2E6IMEJA332PTI7DQ4FWRJR6NG4OYRUP6H2KHXS2TKC5S2A, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.860538777 (540h42m13.731621812s) [noise=true , address=193.149.164.143:4696 , piece_id=OJQFP4YJ3EF52S2XQN5QPKMQJDS57QEOQQ4PUPARPGPAVSLGKMFA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:14.127606583 (540h42m13.998689618s) [noise=true , address=145.79.227.34:4924 , piece_id=R7MT4YS5IPGHOCNZX77ITOPXL57DOYD2HJXPJYYBGNVUWOT33JPA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:14.127625258 (540h42m13.998708303s) [seg_pos={0 0}, upload=474194]: Segment reached optimal threshold of 14 pieces. Dec 28 04:11:14.127786726 (540h42m13.998869761s) [noise=true , address=45.91.116.4:4966 , piece_id=ZTHC25VIBEHSKRULTNAS25DK2DBVI7ATO46SDMMTON7PBBPRMWBQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DSHTtpwcx1h1utGXNou11K7Tcg1ZhkkMdMgY6BDD2ifG6LQ7S): protocol: expected piece hash; context canceled; EOF Dec 28 04:11:14.127792547 (540h42m13.998875582s) [noise=true , address=45.91.116.4:4966 , piece_id=ZTHC25VIBEHSKRULTNAS25DK2DBVI7ATO46SDMMTON7PBBPRMWBQ, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128017335 (540h42m13.99910037s) [noise=true , address=145.79.227.43:4191 , piece_id=CLYBACZJDTAFTBEMEMSOXP4I22PDYSVIBX3AMNN4DISQUJWYFDMQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw): protocol: expected piece hash; context canceled; EOF Dec 28 04:11:14.128023747 (540h42m13.999106782s) [noise=true , address=145.79.227.43:4191 , piece_id=CLYBACZJDTAFTBEMEMSOXP4I22PDYSVIBX3AMNN4DISQUJWYFDMQ, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128139027 (540h42m13.999222102s) [noise=true , address=64.49.13.169:4708 , piece_id=6OBO4M26AXDJXUTOBI66S2JJARDEM2CTTFHUIAOI4CX2PZITRWFQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:40936->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:40936->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40936->64.49.13.169:4708: use of closed network connection Dec 28 04:11:14.128150668 (540h42m13.999233733s) [noise=true , address=64.49.13.169:4708 , piece_id=6OBO4M26AXDJXUTOBI66S2JJARDEM2CTTFHUIAOI4CX2PZITRWFQ, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128292198 (540h42m13.999375243s) [noise=true , address=145.79.227.30:4784 , piece_id=UTH47JZWG3SGX3KXBFJJALALTPP6INYORUHDYSS6CNDEUQ6BDL7A, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pfXWex2WsuhmLb2smRpE46cHT9aipSTHW3e5oyELS1GhF1cFr): protocol: expected piece hash; context canceled; EOF Dec 28 04:11:14.128299212 (540h42m13.999382257s) [noise=true , address=145.79.227.30:4784 , piece_id=UTH47JZWG3SGX3KXBFJJALALTPP6INYORUHDYSS6CNDEUQ6BDL7A, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128395815 (540h42m13.99947885s) [seg_pos={0 0}, upload=474194]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 04:11:14.128488171 (540h42m13.999571206s) [upload=474194]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:11:14.128493561 (540h42m13.999576597s) [upload=474194]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:11:14.572773387 (540h42m14.443856452s) [upload=474194]: Done uploading segments... Dec 28 04:11:14.572800389 (540h42m14.443883454s) [upload=474194]: Upload finished. err: <nil> Duration: 31.622366288s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 6.932729289s trace: https://jaeger.dev.storj.io/trace/3b8130f32e60318d FAIL (2025-12-28 04:12:43 UTC) orbiter warning: context deadline exceeded Duration: 6.932757091s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 52 out of 60 is failed (threshold=3) [XXXXX.XXXXX.XXXXXXXXXXXXXXXXX..XXXX.XXXX.XXXXX.XXXXXXX.XXXXX] duration: 30.001983394s trace: https://jaeger.dev.storj.io/trace/70d3a5998b32963d FAIL (2025-12-28 04:12:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: OW2BXQRPMP5TZUOMXEPVVTXEVH3VS5OKW5UHXTWLG7NKR5JQJFTQ) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: JTTLQMBWYIXQK2EPRQWRCHGIKGLSYICHIYE2VSTL25B3Y6GW3NDA) context deadline exceeded; piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: XAYMISD76VBGMAPBNN4AM2TEMAB3U32MW7SNVJYS2XYRWOFGQEVQ) context deadline exceeded; piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: EERCMK6XMHOXLJ4LVR4UMI4UBQG3JTVMZMDW6VDQOSWMIDWCX5PA) context deadline exceeded ======LOGS====== Dec 28 04:12:06.839127752 (540h43m6.710210797s) [upload=474203]: Starting upload Dec 28 04:12:06.839182195 (540h43m6.71026524s) [upload=474203]: Done starting upload Dec 28 04:12:06.839363571 (540h43m6.710446606s) [upload=474203]: Uploading segments... Dec 28 04:12:06.840224470 (540h43m6.711307515s) [upload=474203]: Got next segment. Inline: false Dec 28 04:12:06.840308280 (540h43m6.711391335s) [upload=474203]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 04:12:06.840315052 (540h43m6.711398107s) [upload=474203]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 04:12:07.237677349 (540h43m7.108760394s) [seg_pos={0 0}, upload=474203]: Begin upload segment... Dec 28 04:12:07.237726824 (540h43m7.108809859s) [seg_pos={0 0}, upload=474203]: Done begin upload segment. Dec 28 04:12:07.237771719 (540h43m7.108854754s) [noise=true , address=145.79.227.36:4519 , piece_id=LSFIF3S7YLLCUXA4OR6F7ZMHXYA64YPQWWKPIJDYVLPP7VVBZ2XQ, seg_pos={0 0}, upload=474203]: Uploading piece... Dec 28 04:12:07.237837725 (540h43m7.10892076s) [noise=true , address=77.90.31.15:4700 , piece_id=JTTLQMBWYIXQK2EPRQWRCHGIKGLSYICHIYE2VSTL25B3Y6GW3NDA, seg_pos={0 0}, upload=474203]: Uploading piece... Dec 28 04:12:07.238117588 (540h43m7.109200633s) [noise=true , address=145.79.227.39:4013 , piece_id=VZGLE7MA7FHPZYQQW2HBURKUXCEGC4633QCRPAVINWM6744ATZYQ, seg_pos={0 0}, upload=474203]: Uploading piece... Dec 28 04:12:07.238471652 (540h43m7.109554697s) [noise=true , address=145.79.227.38:4977 , piece_id=6OAYRG46FGHWEJI3LD2QOWGXJ4XRF3N67VG5HUCYBTDUJ35DWCPA, seg_pos={0 0}, upload=474203]: Uploading piece... Dec 28 04:12:07.238899247 (540h43m7.109982282s) [noise=true , address=45.91.116.8:4047 , piece_id=XAYMISD76VBGMAPBNN4AM2TEMAB3U32MW7SNVJYS2XYRWOFGQEVQ, seg_pos={0 0}, upload=474203]: Uploading piece... Dec 28 04:12:07.239272567 (540h43m7.110355602s) [noise=true , address=77.90.31.19:4842 , piece_id=EERCMK6XMHOXLJ4LVR4UMI4UBQG3JTVMZMDW6VDQOSWMIDWCX5PA, seg_pos={0 0}, upload=474203]: Uploading piece... Dec 28 04:12:07.239651839 (540h43m7.110734874s) [noise=true , address=145.79.227.52:4445 , piece_id=T3FB5I3JRKF7PRXRJBK7WYLUBOU77LL63QC5CUEHQS7WHJHPT6ZA, seg_pos={0 0}, upload=474203]: Uploading piece... Dec 28 04:12:07.239984924 (540h43m7.111067959s) [noise=true , address=77.90.31.10:4517 , piece_id=LPFAESILJPVY5YBQOUFKRAZ5XV7RVWET3SKLAZMWQRZ56F4RAWIQ, seg_pos={0 0}, upload=474203]: Uploading piece... Dec 28 04:12:07.240323608 (540h43m7.111406653s) [noise=true , address=145.79.227.4:4187 , piece_id=LQPEUIGKPYEBXJZPH2ORV4JDB3CHBS3IXGWXOK67IPDF5E7IUCHA, seg_pos={0 0}, upload=474203]: Uploading piece... Dec 28 04:12:07.241703456 (540h43m7.112786491s) [noise=true , address=145.79.227.15:4876 , piece_id=S3MUSOD4WP2W6YDPG4J6YKAXJVNWFNNKBGAZFPYYFVZB4B7QASWQ, seg_pos={0 0}, upload=474203]: Uploading piece... Dec 28 04:12:07.243125834 (540h43m7.114208879s) [noise=true , address=145.79.227.47:4266 , piece_id=VULRI6HJXUBLH7FOZVX7GT2LNQLJ37XODHFUFSNVJ6BEADMWD5JA, seg_pos={0 0}, upload=474203]: Uploading piece... Dec 28 04:12:07.244540819 (540h43m7.115623864s) [noise=true , address=145.79.227.16:4912 , piece_id=CVL2NLPXQHFLAQ4XRUU5OP6HKHLWE6LK3SZ4FFL37IAQKEU2MCVA, seg_pos={0 0}, upload=474203]: Uploading piece... Dec 28 04:12:07.246125766 (540h43m7.117208802s) [noise=true , address=145.79.227.13:4802 , piece_id=QXHDKNSALZY5QNIYPSQI7CZ6PWUER7HMSI7TUHSNIKIV72GVBDVQ, seg_pos={0 0}, upload=474203]: Uploading piece... Dec 28 04:12:07.247785597 (540h43m7.118868642s) [noise=true , address=77.90.31.18:4805 , piece_id=T3DJODCW3T2VDYMC5VOHJHEM6U53GTZSU4HXX564EH2SCYMVYE5A, seg_pos={0 0}, upload=474203]: Uploading piece... Dec 28 04:12:07.249092696 (540h43m7.120175741s) [noise=true , address=145.79.227.35:4481 , piece_id=YRMPL7E2X37LC7R2GX2GF53E72NKEDTUX7TFKKGUARXTD7CWMFIA, seg_pos={0 0}, upload=474203]: Uploading piece... Dec 28 04:12:07.250513602 (540h43m7.121596637s) [noise=true , address=45.91.116.6:4975 , piece_id=OW2BXQRPMP5TZUOMXEPVVTXEVH3VS5OKW5UHXTWLG7NKR5JQJFTQ, seg_pos={0 0}, upload=474203]: Uploading piece... Dec 28 04:12:09.022905995 (540h43m8.89398904s) [upload=474203]: Next returned nil segment Dec 28 04:12:09.022930922 (540h43m8.894013978s) [upload=474203]: Waiting for error group managing segments... Dec 28 04:12:09.385312842 (540h43m9.256395877s) [noise=true , address=145.79.227.13:4802 , piece_id=QXHDKNSALZY5QNIYPSQI7CZ6PWUER7HMSI7TUHSNIKIV72GVBDVQ, seg_pos={0 0}, upload=474203]: Done uploading piece. err: <nil> Dec 28 04:12:09.419578959 (540h43m9.290662004s) [noise=true , address=145.79.227.47:4266 , piece_id=VULRI6HJXUBLH7FOZVX7GT2LNQLJ37XODHFUFSNVJ6BEADMWD5JA, seg_pos={0 0}, upload=474203]: Done uploading piece. err: <nil> Dec 28 04:12:09.478568786 (540h43m9.349651831s) [noise=true , address=77.90.31.15:4700 , piece_id=JTTLQMBWYIXQK2EPRQWRCHGIKGLSYICHIYE2VSTL25B3Y6GW3NDA, seg_pos={0 0}, upload=474203]: Done uploading piece. err: <nil> Dec 28 04:12:09.514765750 (540h43m9.385848795s) [noise=true , address=45.91.116.6:4975 , piece_id=OW2BXQRPMP5TZUOMXEPVVTXEVH3VS5OKW5UHXTWLG7NKR5JQJFTQ, seg_pos={0 0}, upload=474203]: Done uploading piece. err: <nil> Dec 28 04:12:09.520779641 (540h43m9.391862686s) [noise=true , address=77.90.31.18:4805 , piece_id=T3DJODCW3T2VDYMC5VOHJHEM6U53GTZSU4HXX564EH2SCYMVYE5A, seg_pos={0 0}, upload=474203]: Done uploading piece. err: <nil> Dec 28 04:12:09.524447516 (540h43m9.395530561s) [noise=true , address=45.91.116.8:4047 , piece_id=XAYMISD76VBGMAPBNN4AM2TEMAB3U32MW7SNVJYS2XYRWOFGQEVQ, seg_pos={0 0}, upload=474203]: Done uploading piece. err: <nil> Dec 28 04:12:09.543496227 (540h43m9.414579273s) [noise=true , address=77.90.31.10:4517 , piece_id=LPFAESILJPVY5YBQOUFKRAZ5XV7RVWET3SKLAZMWQRZ56F4RAWIQ, seg_pos={0 0}, upload=474203]: Done uploading piece. err: <nil> Dec 28 04:12:09.639983283 (540h43m9.511066328s) [noise=true , address=145.79.227.39:4013 , piece_id=VZGLE7MA7FHPZYQQW2HBURKUXCEGC4633QCRPAVINWM6744ATZYQ, seg_pos={0 0}, upload=474203]: Done uploading piece. err: <nil> Dec 28 04:12:09.653104267 (540h43m9.524187302s) [noise=true , address=145.79.227.38:4977 , piece_id=6OAYRG46FGHWEJI3LD2QOWGXJ4XRF3N67VG5HUCYBTDUJ35DWCPA, seg_pos={0 0}, upload=474203]: Done uploading piece. err: <nil> Dec 28 04:12:09.669115205 (540h43m9.54019824s) [noise=true , address=145.79.227.52:4445 , piece_id=T3FB5I3JRKF7PRXRJBK7WYLUBOU77LL63QC5CUEHQS7WHJHPT6ZA, seg_pos={0 0}, upload=474203]: Done uploading piece. err: <nil> Dec 28 04:12:09.797433232 (540h43m9.668516277s) [noise=true , address=77.90.31.19:4842 , piece_id=EERCMK6XMHOXLJ4LVR4UMI4UBQG3JTVMZMDW6VDQOSWMIDWCX5PA, seg_pos={0 0}, upload=474203]: Done uploading piece. err: <nil> Dec 28 04:12:10.463413007 (540h43m10.334496042s) [noise=true , address=145.79.227.36:4519 , piece_id=LSFIF3S7YLLCUXA4OR6F7ZMHXYA64YPQWWKPIJDYVLPP7VVBZ2XQ, seg_pos={0 0}, upload=474203]: Done uploading piece. err: <nil> Dec 28 04:12:10.581072759 (540h43m10.452155804s) [noise=true , address=145.79.227.15:4876 , piece_id=S3MUSOD4WP2W6YDPG4J6YKAXJVNWFNNKBGAZFPYYFVZB4B7QASWQ, seg_pos={0 0}, upload=474203]: Done uploading piece. err: <nil> Dec 28 04:12:10.581104909 (540h43m10.452187955s) [seg_pos={0 0}, upload=474203]: Segment reached optimal threshold of 13 pieces. Dec 28 04:12:10.581349465 (540h43m10.45243251s) [noise=true , address=145.79.227.35:4481 , piece_id=YRMPL7E2X37LC7R2GX2GF53E72NKEDTUX7TFKKGUARXTD7CWMFIA, seg_pos={0 0}, upload=474203]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 28 04:12:10.581359184 (540h43m10.452442239s) [noise=true , address=145.79.227.35:4481 , piece_id=YRMPL7E2X37LC7R2GX2GF53E72NKEDTUX7TFKKGUARXTD7CWMFIA, seg_pos={0 0}, upload=474203]: Long tail cancellation due to optimal threshold Dec 28 04:12:10.581539738 (540h43m10.452622783s) [noise=true , address=145.79.227.4:4187 , piece_id=LQPEUIGKPYEBXJZPH2ORV4JDB3CHBS3IXGWXOK67IPDF5E7IUCHA, seg_pos={0 0}, upload=474203]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: expected piece hash; context canceled; EOF Dec 28 04:12:10.581557611 (540h43m10.452640657s) [noise=true , address=145.79.227.4:4187 , piece_id=LQPEUIGKPYEBXJZPH2ORV4JDB3CHBS3IXGWXOK67IPDF5E7IUCHA, seg_pos={0 0}, upload=474203]: Long tail cancellation due to optimal threshold Dec 28 04:12:10.581689393 (540h43m10.452772428s) [noise=true , address=145.79.227.16:4912 , piece_id=CVL2NLPXQHFLAQ4XRUU5OP6HKHLWE6LK3SZ4FFL37IAQKEU2MCVA, seg_pos={0 0}, upload=474203]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR): protocol: expected piece hash; context canceled; EOF Dec 28 04:12:10.581699983 (540h43m10.452783019s) [noise=true , address=145.79.227.16:4912 , piece_id=CVL2NLPXQHFLAQ4XRUU5OP6HKHLWE6LK3SZ4FFL37IAQKEU2MCVA, seg_pos={0 0}, upload=474203]: Long tail cancellation due to optimal threshold Dec 28 04:12:10.581757933 (540h43m10.452840968s) [seg_pos={0 0}, upload=474203]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 04:12:10.581965689 (540h43m10.453048734s) [upload=474203]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:12:10.581972622 (540h43m10.453055657s) [upload=474203]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:12:10.890767484 (540h43m10.761850529s) [upload=474203]: Done uploading segments... Dec 28 04:12:10.890788685 (540h43m10.76187172s) [upload=474203]: Upload finished. err: <nil> Duration: 30.002014512s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 7.529959875s trace: https://jaeger.dev.storj.io/trace/3b8130f32e60318d FAIL (2025-12-28 04:12:29 UTC) orbiter warning: context deadline exceeded Duration: 7.5299778s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 21 out of 60 is failed (threshold=6) [.X.XXXX....X.X.X...X..X.X.X..X...XX.....X......XX...X.X....X] duration: 5.001108367s trace: https://jaeger.dev.storj.io/trace/f61ff6946e8a16e FAIL (2025-12-28 04:12:21 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001133475s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [X...................................................X......X] duration: 1m0.005074501s trace: https://jaeger.dev.storj.io/trace/26c246d73008cc49 FAIL (2025-12-28 04:12:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== oise=true , address=145.79.227.43:4191 , piece_id=CLYBACZJDTAFTBEMEMSOXP4I22PDYSVIBX3AMNN4DISQUJWYFDMQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.179395486 (540h42m11.050478531s) [noise=true , address=145.79.227.9:4371 , piece_id=2ON3P6ZFEUEBFFLCHYY5BWAAJEFX43KKU5MU5TLBQHSZ52RCAYLA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.179823281 (540h42m11.050906336s) [noise=true , address=77.90.31.18:4819 , piece_id=FVZE5P55HHT3RGOI4M2OFV6NER4BNVGFSZOMDRADO5JYI54DRHLQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.180232490 (540h42m11.051315535s) [noise=true , address=77.90.31.17:4778 , piece_id=WHTKIVKEZNFREOTNDTLLBB24JGWKIXWBJKZ2EUM3QL2SM7Z4YXDA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.180945087 (540h42m11.052028132s) [noise=true , address=45.91.116.6:4003 , piece_id=H4VWZIMYVM2M3SX76LSS64ARSENBQL6OOTL5EY5XE4IAORO7OWGQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181189232 (540h42m11.052272277s) [noise=true , address=77.90.31.21:4906 , piece_id=SGYCZ2E6IMEJA332PTI7DQ4FWRJR6NG4OYRUP6H2KHXS2TKC5S2A, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181508139 (540h42m11.052591174s) [noise=true , address=77.90.31.15:4714 , piece_id=YVXSR2B74BDK4BV5QGLZLWOKVXV3TWKSW5M3KG323U7VUUBIQU3Q, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181597209 (540h42m11.052680254s) [noise=true , address=145.79.227.11:4755 , piece_id=F37J7GC6RB2PL3B4B6MFVAPB6IOGSJSKVQ5UNFHFSBJ7C3SABURQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181882131 (540h42m11.052965166s) [noise=true , address=145.79.227.33:4876 , piece_id=UEOY3HYI3WWJTXUSBALBXIGCMBC5M6XGR42SIOBXQNDCZ3UFMWVA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182132017 (540h42m11.053215052s) [noise=true , address=145.79.227.40:4053 , piece_id=V7R5MTCCKCGHV7KD4LAXFBUJI3HRGVAJUCVYL4WPL3M32CVQXOBA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182274007 (540h42m11.053357052s) [noise=true , address=145.79.227.51:4432 , piece_id=C5P63KNNZLSOBH772MRMYOJOQGLE7KXGELQN437HTVPAM72RXUCA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182451545 (540h42m11.05353458s) [noise=true , address=145.79.227.30:4784 , piece_id=UTH47JZWG3SGX3KXBFJJALALTPP6INYORUHDYSS6CNDEUQ6BDL7A, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182777516 (540h42m11.053860551s) [noise=true , address=64.49.13.169:4708 , piece_id=6OBO4M26AXDJXUTOBI66S2JJARDEM2CTTFHUIAOI4CX2PZITRWFQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182919286 (540h42m11.054002331s) [noise=true , address=145.79.227.6:4267 , piece_id=SDG3WH6TPR47OHG57NMVFAHOYVAL45XMJ7S22FZL6IFR6DS45HQQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.183101804 (540h42m11.054184839s) [noise=true , address=145.79.227.34:4924 , piece_id=R7MT4YS5IPGHOCNZX77ITOPXL57DOYD2HJXPJYYBGNVUWOT33JPA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.183514740 (540h42m11.054597785s) [noise=true , address=193.149.164.143:4696 , piece_id=OJQFP4YJ3EF52S2XQN5QPKMQJDS57QEOQQ4PUPARPGPAVSLGKMFA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:12.562530634 (540h42m12.433613679s) [upload=474194]: Next returned nil segment Dec 28 04:11:12.562544100 (540h42m12.433627135s) [upload=474194]: Waiting for error group managing segments... Dec 28 04:11:13.045899872 (540h42m12.916982917s) [noise=true , address=145.79.227.9:4371 , piece_id=2ON3P6ZFEUEBFFLCHYY5BWAAJEFX43KKU5MU5TLBQHSZ52RCAYLA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.068300556 (540h42m12.939383591s) [noise=true , address=145.79.227.11:4755 , piece_id=F37J7GC6RB2PL3B4B6MFVAPB6IOGSJSKVQ5UNFHFSBJ7C3SABURQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.073683536 (540h42m12.944766571s) [noise=true , address=145.79.227.40:4053 , piece_id=V7R5MTCCKCGHV7KD4LAXFBUJI3HRGVAJUCVYL4WPL3M32CVQXOBA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.125067800 (540h42m12.996150835s) [noise=true , address=145.79.227.6:4267 , piece_id=SDG3WH6TPR47OHG57NMVFAHOYVAL45XMJ7S22FZL6IFR6DS45HQQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.156395944 (540h42m13.027478989s) [noise=true , address=145.79.227.51:4432 , piece_id=C5P63KNNZLSOBH772MRMYOJOQGLE7KXGELQN437HTVPAM72RXUCA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.221620070 (540h42m13.092703105s) [noise=true , address=77.90.31.15:4714 , piece_id=YVXSR2B74BDK4BV5QGLZLWOKVXV3TWKSW5M3KG323U7VUUBIQU3Q, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.266169559 (540h42m13.137252594s) [noise=true , address=145.79.227.33:4876 , piece_id=UEOY3HYI3WWJTXUSBALBXIGCMBC5M6XGR42SIOBXQNDCZ3UFMWVA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.273253176 (540h42m13.144336221s) [noise=true , address=77.90.31.12:4604 , piece_id=DVQ53JEZL2GDTXFDDIWN4BLSJJHRZORII7UNF5HS3GIM5VJX3Y2Q, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.324930308 (540h42m13.196013343s) [noise=true , address=77.90.31.17:4778 , piece_id=WHTKIVKEZNFREOTNDTLLBB24JGWKIXWBJKZ2EUM3QL2SM7Z4YXDA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.333001758 (540h42m13.204084793s) [noise=true , address=77.90.31.18:4819 , piece_id=FVZE5P55HHT3RGOI4M2OFV6NER4BNVGFSZOMDRADO5JYI54DRHLQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.418239402 (540h42m13.289322437s) [noise=true , address=45.91.116.6:4003 , piece_id=H4VWZIMYVM2M3SX76LSS64ARSENBQL6OOTL5EY5XE4IAORO7OWGQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.460099681 (540h42m13.331182726s) [noise=true , address=77.90.31.21:4906 , piece_id=SGYCZ2E6IMEJA332PTI7DQ4FWRJR6NG4OYRUP6H2KHXS2TKC5S2A, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.860538777 (540h42m13.731621812s) [noise=true , address=193.149.164.143:4696 , piece_id=OJQFP4YJ3EF52S2XQN5QPKMQJDS57QEOQQ4PUPARPGPAVSLGKMFA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:14.127606583 (540h42m13.998689618s) [noise=true , address=145.79.227.34:4924 , piece_id=R7MT4YS5IPGHOCNZX77ITOPXL57DOYD2HJXPJYYBGNVUWOT33JPA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:14.127625258 (540h42m13.998708303s) [seg_pos={0 0}, upload=474194]: Segment reached optimal threshold of 14 pieces. Dec 28 04:11:14.127786726 (540h42m13.998869761s) [noise=true , address=45.91.116.4:4966 , piece_id=ZTHC25VIBEHSKRULTNAS25DK2DBVI7ATO46SDMMTON7PBBPRMWBQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DSHTtpwcx1h1utGXNou11K7Tcg1ZhkkMdMgY6BDD2ifG6LQ7S): protocol: expected piece hash; context canceled; EOF Dec 28 04:11:14.127792547 (540h42m13.998875582s) [noise=true , address=45.91.116.4:4966 , piece_id=ZTHC25VIBEHSKRULTNAS25DK2DBVI7ATO46SDMMTON7PBBPRMWBQ, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128017335 (540h42m13.99910037s) [noise=true , address=145.79.227.43:4191 , piece_id=CLYBACZJDTAFTBEMEMSOXP4I22PDYSVIBX3AMNN4DISQUJWYFDMQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw): protocol: expected piece hash; context canceled; EOF Dec 28 04:11:14.128023747 (540h42m13.999106782s) [noise=true , address=145.79.227.43:4191 , piece_id=CLYBACZJDTAFTBEMEMSOXP4I22PDYSVIBX3AMNN4DISQUJWYFDMQ, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128139027 (540h42m13.999222102s) [noise=true , address=64.49.13.169:4708 , piece_id=6OBO4M26AXDJXUTOBI66S2JJARDEM2CTTFHUIAOI4CX2PZITRWFQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:40936->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:40936->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40936->64.49.13.169:4708: use of closed network connection Dec 28 04:11:14.128150668 (540h42m13.999233733s) [noise=true , address=64.49.13.169:4708 , piece_id=6OBO4M26AXDJXUTOBI66S2JJARDEM2CTTFHUIAOI4CX2PZITRWFQ, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128292198 (540h42m13.999375243s) [noise=true , address=145.79.227.30:4784 , piece_id=UTH47JZWG3SGX3KXBFJJALALTPP6INYORUHDYSS6CNDEUQ6BDL7A, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pfXWex2WsuhmLb2smRpE46cHT9aipSTHW3e5oyELS1GhF1cFr): protocol: expected piece hash; context canceled; EOF Dec 28 04:11:14.128299212 (540h42m13.999382257s) [noise=true , address=145.79.227.30:4784 , piece_id=UTH47JZWG3SGX3KXBFJJALALTPP6INYORUHDYSS6CNDEUQ6BDL7A, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128395815 (540h42m13.99947885s) [seg_pos={0 0}, upload=474194]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 04:11:14.128488171 (540h42m13.999571206s) [upload=474194]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:11:14.128493561 (540h42m13.999576597s) [upload=474194]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:11:14.572773387 (540h42m14.443856452s) [upload=474194]: Done uploading segments... Dec 28 04:11:14.572800389 (540h42m14.443883454s) [upload=474194]: Upload finished. err: <nil> Duration: 1m0.005198567s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 6.818231991s trace: https://jaeger.dev.storj.io/trace/3b8130f32e60318d FAIL (2025-12-28 04:11:51 UTC) orbiter warning: context deadline exceeded Duration: 6.818256457s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 52 out of 60 is failed (threshold=3) [XXXXXX.XXXXX.XXXXXXXXXXXXXXXXX..XXXX.XXXX.XXXXX.XXXXXXX.XXXX] duration: 30.002614376s trace: https://jaeger.dev.storj.io/trace/813897c3f9577b4 FAIL (2025-12-28 04:11:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, Piece ID: AJFX73O4QN2EZTYGUEQY3SJGDWCM6PCC6VLLL2FSSOCJ4EZP2TLQ) context deadline exceeded; piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: 5G4OVO6ZYTJDRPTUD65LWQHEXBSBHCZ2L4NXC2MUMYILL55QCM4Q) context deadline exceeded ======LOGS====== 28 04:11:19.606702918 (540h42m19.477785963s) [upload=474195]: Starting upload Dec 28 04:11:19.606831693 (540h42m19.477914738s) [upload=474195]: Done starting upload Dec 28 04:11:19.606978643 (540h42m19.478061688s) [upload=474195]: Uploading segments... Dec 28 04:11:19.607201338 (540h42m19.478284383s) [upload=474195]: Got next segment. Inline: false Dec 28 04:11:19.607245321 (540h42m19.478328366s) [upload=474195]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 04:11:19.607484116 (540h42m19.478567161s) [upload=474195]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 04:11:19.857787801 (540h42m19.728870846s) [seg_pos={0 0}, upload=474195]: Begin upload segment... Dec 28 04:11:19.857906988 (540h42m19.728990034s) [seg_pos={0 0}, upload=474195]: Done begin upload segment. Dec 28 04:11:19.857965710 (540h42m19.729048755s) [noise=true , address=45.91.116.10:4084 , piece_id=5G4OVO6ZYTJDRPTUD65LWQHEXBSBHCZ2L4NXC2MUMYILL55QCM4Q, seg_pos={0 0}, upload=474195]: Uploading piece... Dec 28 04:11:19.858065149 (540h42m19.729148194s) [noise=true , address=145.79.227.14:4837 , piece_id=R5JS6BJSRAIF4S5A6NBNEGVVXBDAURANZXEYUBLZNXSXQFGMJAJQ, seg_pos={0 0}, upload=474195]: Uploading piece... Dec 28 04:11:19.858766104 (540h42m19.729849139s) [noise=true , address=145.79.227.45:4230 , piece_id=5TXGDOUJPULUIZHVBCX77JCBWBEZUTKVGS75S2KIUWXEOKFSNF3Q, seg_pos={0 0}, upload=474195]: Uploading piece... Dec 28 04:11:19.859088779 (540h42m19.730171814s) [noise=true , address=145.79.227.24:4583 , piece_id=3376PWW5HBIB53BGT5ZWVOULALVYIQYNPIAY5AL2LBG5EAPR3RKA, seg_pos={0 0}, upload=474195]: Uploading piece... Dec 28 04:11:19.859356278 (540h42m19.730439313s) [noise=true , address=77.90.31.17:4773 , piece_id=AJFX73O4QN2EZTYGUEQY3SJGDWCM6PCC6VLLL2FSSOCJ4EZP2TLQ, seg_pos={0 0}, upload=474195]: Uploading piece... Dec 28 04:11:19.859698259 (540h42m19.730781294s) [noise=true , address=145.79.227.46:4601 , piece_id=54KCBQMDO7IXHAX6WS2DQZP5PRNHMH26I2XKWDZH4SC2MKETWE2A, seg_pos={0 0}, upload=474195]: Uploading piece... Dec 28 04:11:19.859962462 (540h42m19.731045507s) [noise=true , address=145.79.227.10:4402 , piece_id=GFWUZSD5ZL3BD2HP7PIUM5YTDXLMH6KQEE53S4FFHAMJUPSGXLAQ, seg_pos={0 0}, upload=474195]: Uploading piece... Dec 28 04:11:19.860300776 (540h42m19.731383821s) [noise=true , address=145.79.227.15:4873 , piece_id=GL5AJ5QSVPLPRLYVP7HHMAHWQLTVPP5ET5Z6VMCKN6ZH56DPJSMA, seg_pos={0 0}, upload=474195]: Uploading piece... Dec 28 04:11:19.860568366 (540h42m19.731651411s) [noise=true , address=145.79.227.31:4798 , piece_id=332LK6VT7NOAFZK4RENW6R7S5XJ5O2NICD2U7DVOSC5ZSVBBN74A, seg_pos={0 0}, upload=474195]: Uploading piece... Dec 28 04:11:19.861952401 (540h42m19.733035446s) [noise=true , address=77.90.31.16:4736 , piece_id=J7OLOEYIVQBWDZ422DBUPB3Y6DIBL7ZJKRZ5SXOFNI7KDTEPIBBQ, seg_pos={0 0}, upload=474195]: Uploading piece... Dec 28 04:11:19.862528037 (540h42m19.733611072s) [noise=true , address=77.90.31.15:4697 , piece_id=USMVP6UDU5WJKO4XNBJLI7EJCBF7PITVFTYTNUHKMI3D54DQEIXA, seg_pos={0 0}, upload=474195]: Uploading piece... Dec 28 04:11:19.863080349 (540h42m19.734163394s) [noise=true , address=45.91.116.9:4262 , piece_id=RKXRMDAVTXGWRH7ST6ZF3CC3M36HT7COQBVA6TIJ5CWSKG3KVPZQ, seg_pos={0 0}, upload=474195]: Uploading piece... Dec 28 04:11:19.863472615 (540h42m19.734555661s) [noise=true , address=145.79.227.9:4365 , piece_id=THLKTSFXTS3BLHU4UWKGDO5UR4YXKSDDGP7BD5ZDJNXCUDAMTF5Q, seg_pos={0 0}, upload=474195]: Uploading piece... Dec 28 04:11:19.863782446 (540h42m19.734865481s) [noise=true , address=45.91.116.14:4190 , piece_id=ZXCLC2DJTIS5MDF3D6XDTZOQWZH6OU5XLDCMO5JWPZSONYBYQBPA, seg_pos={0 0}, upload=474195]: Uploading piece... Dec 28 04:11:19.864082507 (540h42m19.735165542s) [noise=true , address=77.90.31.10:4519 , piece_id=2ABCIVK2UQUKX3NVK44IA4UASQYDBIGMXBIUDOWDACAMPGDVNWHA, seg_pos={0 0}, upload=474195]: Uploading piece... Dec 28 04:11:19.864485705 (540h42m19.735568751s) [noise=true , address=145.79.227.38:4979 , piece_id=M2YLWT2ZBKCLARHODZC3BBQOKWDYFX3M3SO2KMJUSHATPXOKNZTQ, seg_pos={0 0}, upload=474195]: Uploading piece... Dec 28 04:11:21.673637083 (540h42m21.544720128s) [upload=474195]: Next returned nil segment Dec 28 04:11:21.673654947 (540h42m21.544738002s) [upload=474195]: Waiting for error group managing segments... Dec 28 04:11:21.965321094 (540h42m21.836404129s) [noise=true , address=145.79.227.38:4979 , piece_id=M2YLWT2ZBKCLARHODZC3BBQOKWDYFX3M3SO2KMJUSHATPXOKNZTQ, seg_pos={0 0}, upload=474195]: Done uploading piece. err: <nil> Dec 28 04:11:21.966200818 (540h42m21.837283864s) [noise=true , address=145.79.227.31:4798 , piece_id=332LK6VT7NOAFZK4RENW6R7S5XJ5O2NICD2U7DVOSC5ZSVBBN74A, seg_pos={0 0}, upload=474195]: Done uploading piece. err: <nil> Dec 28 04:11:21.966810619 (540h42m21.837893654s) [noise=true , address=145.79.227.14:4837 , piece_id=R5JS6BJSRAIF4S5A6NBNEGVVXBDAURANZXEYUBLZNXSXQFGMJAJQ, seg_pos={0 0}, upload=474195]: Done uploading piece. err: <nil> Dec 28 04:11:21.967649628 (540h42m21.838732663s) [noise=true , address=145.79.227.45:4230 , piece_id=5TXGDOUJPULUIZHVBCX77JCBWBEZUTKVGS75S2KIUWXEOKFSNF3Q, seg_pos={0 0}, upload=474195]: Done uploading piece. err: <nil> Dec 28 04:11:21.985700078 (540h42m21.856783123s) [noise=true , address=145.79.227.46:4601 , piece_id=54KCBQMDO7IXHAX6WS2DQZP5PRNHMH26I2XKWDZH4SC2MKETWE2A, seg_pos={0 0}, upload=474195]: Done uploading piece. err: <nil> Dec 28 04:11:22.006386007 (540h42m21.877469042s) [noise=true , address=145.79.227.9:4365 , piece_id=THLKTSFXTS3BLHU4UWKGDO5UR4YXKSDDGP7BD5ZDJNXCUDAMTF5Q, seg_pos={0 0}, upload=474195]: Done uploading piece. err: <nil> Dec 28 04:11:22.024123733 (540h42m21.895206768s) [noise=true , address=145.79.227.15:4873 , piece_id=GL5AJ5QSVPLPRLYVP7HHMAHWQLTVPP5ET5Z6VMCKN6ZH56DPJSMA, seg_pos={0 0}, upload=474195]: Done uploading piece. err: <nil> Dec 28 04:11:22.110461703 (540h42m21.981544738s) [noise=true , address=45.91.116.10:4084 , piece_id=5G4OVO6ZYTJDRPTUD65LWQHEXBSBHCZ2L4NXC2MUMYILL55QCM4Q, seg_pos={0 0}, upload=474195]: Done uploading piece. err: <nil> Dec 28 04:11:22.137883919 (540h42m22.008966954s) [noise=true , address=45.91.116.9:4262 , piece_id=RKXRMDAVTXGWRH7ST6ZF3CC3M36HT7COQBVA6TIJ5CWSKG3KVPZQ, seg_pos={0 0}, upload=474195]: Done uploading piece. err: <nil> Dec 28 04:11:22.139199093 (540h42m22.010282128s) [noise=true , address=77.90.31.16:4736 , piece_id=J7OLOEYIVQBWDZ422DBUPB3Y6DIBL7ZJKRZ5SXOFNI7KDTEPIBBQ, seg_pos={0 0}, upload=474195]: Done uploading piece. err: <nil> Dec 28 04:11:22.158216084 (540h42m22.029299129s) [noise=true , address=77.90.31.15:4697 , piece_id=USMVP6UDU5WJKO4XNBJLI7EJCBF7PITVFTYTNUHKMI3D54DQEIXA, seg_pos={0 0}, upload=474195]: Done uploading piece. err: <nil> Dec 28 04:11:22.230243816 (540h42m22.101326861s) [noise=true , address=45.91.116.14:4190 , piece_id=ZXCLC2DJTIS5MDF3D6XDTZOQWZH6OU5XLDCMO5JWPZSONYBYQBPA, seg_pos={0 0}, upload=474195]: Done uploading piece. err: <nil> Dec 28 04:11:22.729789246 (540h42m22.600872281s) [noise=true , address=77.90.31.17:4773 , piece_id=AJFX73O4QN2EZTYGUEQY3SJGDWCM6PCC6VLLL2FSSOCJ4EZP2TLQ, seg_pos={0 0}, upload=474195]: Done uploading piece. err: <nil> Dec 28 04:11:22.729829723 (540h42m22.600912768s) [seg_pos={0 0}, upload=474195]: Segment reached optimal threshold of 13 pieces. Dec 28 04:11:22.730077495 (540h42m22.60116053s) [noise=true , address=145.79.227.10:4402 , piece_id=GFWUZSD5ZL3BD2HP7PIUM5YTDXLMH6KQEE53S4FFHAMJUPSGXLAQ, seg_pos={0 0}, upload=474195]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u): protocol: expected piece hash; context canceled; EOF Dec 28 04:11:22.730084548 (540h42m22.601167583s) [noise=true , address=145.79.227.10:4402 , piece_id=GFWUZSD5ZL3BD2HP7PIUM5YTDXLMH6KQEE53S4FFHAMJUPSGXLAQ, seg_pos={0 0}, upload=474195]: Long tail cancellation due to optimal threshold Dec 28 04:11:22.730335977 (540h42m22.601419012s) [noise=true , address=77.90.31.10:4519 , piece_id=2ABCIVK2UQUKX3NVK44IA4UASQYDBIGMXBIUDOWDACAMPGDVNWHA, seg_pos={0 0}, upload=474195]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: write tcp 188.166.225.170:53690->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:53690->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53690->77.90.31.10:4519: use of closed network connection Dec 28 04:11:22.730343181 (540h42m22.601426216s) [noise=true , address=77.90.31.10:4519 , piece_id=2ABCIVK2UQUKX3NVK44IA4UASQYDBIGMXBIUDOWDACAMPGDVNWHA, seg_pos={0 0}, upload=474195]: Long tail cancellation due to optimal threshold Dec 28 04:11:22.730491263 (540h42m22.601574298s) [noise=true , address=145.79.227.24:4583 , piece_id=3376PWW5HBIB53BGT5ZWVOULALVYIQYNPIAY5AL2LBG5EAPR3RKA, seg_pos={0 0}, upload=474195]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19): protocol: expected piece hash; context canceled; EOF Dec 28 04:11:22.730497855 (540h42m22.60158089s) [noise=true , address=145.79.227.24:4583 , piece_id=3376PWW5HBIB53BGT5ZWVOULALVYIQYNPIAY5AL2LBG5EAPR3RKA, seg_pos={0 0}, upload=474195]: Long tail cancellation due to optimal threshold Dec 28 04:11:22.730515599 (540h42m22.601598634s) [seg_pos={0 0}, upload=474195]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 04:11:22.730580462 (540h42m22.601663497s) [upload=474195]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:11:22.730586915 (540h42m22.60166995s) [upload=474195]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:11:23.028695489 (540h42m22.899778554s) [upload=474195]: Done uploading segments... Dec 28 04:11:23.028725145 (540h42m22.89980818s) [upload=474195]: Upload finished. err: <nil> Duration: 30.002639725s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 11.035492342s trace: https://jaeger.dev.storj.io/trace/3b8130f32e60318d FAIL (2025-12-28 04:11:28 UTC) orbiter warning: context deadline exceeded Duration: 11.035521347s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 20 out of 60 is failed (threshold=6) [..X.XXXX....X.X.X...X..X.X.X..X...XX.....X......XX...X.X....] duration: 5.001141788s trace: https://jaeger.dev.storj.io/trace/693a9ad598fe2536 FAIL (2025-12-28 04:11:08 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001172276s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X...................................................X......] duration: 37.079451842s trace: https://jaeger.dev.storj.io/trace/591b5c08c3d8a813 FAIL (2025-12-28 04:11:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== A, seg_pos={0 0}, upload=474110]: Uploading piece... Dec 28 04:05:28.680332318 (540h36m28.551415353s) [noise=true , address=145.79.227.43:4182 , piece_id=VZEVVQ4T5SS2PLTVY4FQDGZVB2DRPG2VMDUCP5ECZPIQBKG2RCSA, seg_pos={0 0}, upload=474110]: Uploading piece... Dec 28 04:05:28.680692264 (540h36m28.551775309s) [noise=true , address=64.49.13.169:4709 , piece_id=A7RHDJ7GK5OZGUVBJW7X7FSYMVY6OIY3SHW66NCRU6SSISGCKYDA, seg_pos={0 0}, upload=474110]: Uploading piece... Dec 28 04:05:28.681012203 (540h36m28.552095238s) [noise=true , address=77.90.31.18:4819 , piece_id=NXICQKVF2XGMA7RTEL4DOWZ4Q5EKY2RIWL24LMJY47TS32L73F7Q, seg_pos={0 0}, upload=474110]: Uploading piece... Dec 28 04:05:28.681332643 (540h36m28.552415678s) [noise=true , address=64.49.13.140:4507 , piece_id=PT6VHNLKRVNYJKIEYXRSA57LIQJMA3TKG3FCKJI5DDSDOXZVZCLQ, seg_pos={0 0}, upload=474110]: Uploading piece... Dec 28 04:05:28.681662891 (540h36m28.552745936s) [noise=true , address=77.90.31.12:4597 , piece_id=V4ZPTTPBTO6OEZM6Z66P4WIQ3OEOL2EOV2C6TWDS6Q27MOC6M3CQ, seg_pos={0 0}, upload=474110]: Uploading piece... Dec 28 04:05:28.682067222 (540h36m28.553150257s) [noise=true , address=145.79.227.40:4069 , piece_id=7YQ5FZTREX3LU3OYUHTXEMUISHIWPLXB74JUHF6O7CFCQBWN4QGQ, seg_pos={0 0}, upload=474110]: Uploading piece... Dec 28 04:05:28.682393643 (540h36m28.553476678s) [noise=true , address=145.79.227.11:4744 , piece_id=JTTI237QP4VHMZFDIX6MOUGYPFODL5GGUW3IDKZXQLVXCRHGN3YQ, seg_pos={0 0}, upload=474110]: Uploading piece... Dec 28 04:05:28.682756655 (540h36m28.55383969s) [noise=true , address=77.90.31.17:4779 , piece_id=GMVQKCFKGZU7AGQ7AT3ULHJCJM2B2VC2Z572ET3LZQJBD7X4M6NQ, seg_pos={0 0}, upload=474110]: Uploading piece... Dec 28 04:05:28.683195209 (540h36m28.554278254s) [noise=true , address=145.79.227.34:4909 , piece_id=NDEZQCRKSSPYWQOQIDY7QMSKAUIL3O52MGDUQ3KVHTP35R75KWDQ, seg_pos={0 0}, upload=474110]: Uploading piece... Dec 28 04:05:28.683539605 (540h36m28.55462265s) [noise=true , address=77.90.31.13:4658 , piece_id=253DTNCSSNTDZR2Q56TT34OYAJPELSHIPO4PAOSNZ2KDQ3SUZXQA, seg_pos={0 0}, upload=474110]: Uploading piece... Dec 28 04:05:28.689846314 (540h36m28.560929359s) [noise=true , address=77.90.31.11:4577 , piece_id=NMWLIKEFIMQUHQVA4342PSF3RQ5WAYSNHJPZJ5VU5PEJIISQMEGA, seg_pos={0 0}, upload=474110]: Uploading piece... Dec 28 04:05:28.692671473 (540h36m28.563754528s) [noise=true , address=193.149.164.143:4695 , piece_id=HD5JUFDHKF5G4HA3CRIVTSAUH4MXW7HITIZLBD2J7FGGFMPQXOOQ, seg_pos={0 0}, upload=474110]: Uploading piece... Dec 28 04:05:28.695438943 (540h36m28.566521989s) [noise=true , address=45.91.116.11:4139 , piece_id=35R4AZATBN22P2KLD6VNWHJD5TAS4HA5GSV7OU5CCBD4YUYUWJTA, seg_pos={0 0}, upload=474110]: Uploading piece... Dec 28 04:05:28.698799452 (540h36m28.569882487s) [noise=true , address=45.91.116.12:4249 , piece_id=JM5XOZSORND2I3RGG2NVHN26CQDOK3VKNLJS6K4E2CKR62467RHQ, seg_pos={0 0}, upload=474110]: Uploading piece... Dec 28 04:05:28.706563095 (540h36m28.57764613s) [noise=true , address=145.79.227.44:4221 , piece_id=JG45XU4WRCR6H343JZXFJ6JWNOJVNKBDREIHI6GXEENEAPI3LXUA, seg_pos={0 0}, upload=474110]: Uploading piece... Dec 28 04:05:30.092893206 (540h36m29.963976251s) [upload=474110]: Next returned nil segment Dec 28 04:05:30.092906743 (540h36m29.963989778s) [upload=474110]: Waiting for error group managing segments... Dec 28 04:05:30.603591324 (540h36m30.474674369s) [noise=true , address=145.79.227.43:4182 , piece_id=VZEVVQ4T5SS2PLTVY4FQDGZVB2DRPG2VMDUCP5ECZPIQBKG2RCSA, seg_pos={0 0}, upload=474110]: Done uploading piece. err: <nil> Dec 28 04:05:30.605517772 (540h36m30.476600807s) [noise=true , address=145.79.227.51:4432 , piece_id=YN66BQJBHIRZW2CCPL55RLISG5LCM53MUUVNJNN3L5UYE2CHJPDQ, seg_pos={0 0}, upload=474110]: Done uploading piece. err: <nil> Dec 28 04:05:30.649134745 (540h36m30.52021778s) [noise=true , address=64.49.13.140:4507 , piece_id=PT6VHNLKRVNYJKIEYXRSA57LIQJMA3TKG3FCKJI5DDSDOXZVZCLQ, seg_pos={0 0}, upload=474110]: Done uploading piece. err: <nil> Dec 28 04:05:30.721256115 (540h36m30.59233915s) [noise=true , address=77.90.31.21:4891 , piece_id=VW4VGEYAY5JNK6F3P6G4B4JZ2AW4O5CWRYXFBAKJ2HXQJ7UW6IJA, seg_pos={0 0}, upload=474110]: Done uploading piece. err: <nil> Dec 28 04:05:30.751313910 (540h36m30.622396955s) [noise=true , address=77.90.31.11:4577 , piece_id=NMWLIKEFIMQUHQVA4342PSF3RQ5WAYSNHJPZJ5VU5PEJIISQMEGA, seg_pos={0 0}, upload=474110]: Done uploading piece. err: <nil> Dec 28 04:05:30.800719756 (540h36m30.671802831s) [noise=true , address=45.91.116.11:4139 , piece_id=35R4AZATBN22P2KLD6VNWHJD5TAS4HA5GSV7OU5CCBD4YUYUWJTA, seg_pos={0 0}, upload=474110]: Done uploading piece. err: <nil> Dec 28 04:05:30.823740571 (540h36m30.694823606s) [noise=true , address=77.90.31.12:4597 , piece_id=V4ZPTTPBTO6OEZM6Z66P4WIQ3OEOL2EOV2C6TWDS6Q27MOC6M3CQ, seg_pos={0 0}, upload=474110]: Done uploading piece. err: <nil> Dec 28 04:05:30.847552863 (540h36m30.718635898s) [noise=true , address=145.79.227.12:4795 , piece_id=AQNYUYTFXYD6BXPHQ7UXWNY732L56PJ4KOZT5XZGLZYFZPND2QYA, seg_pos={0 0}, upload=474110]: Done uploading piece. err: <nil> Dec 28 04:05:30.850526996 (540h36m30.721610031s) [noise=true , address=77.90.31.17:4779 , piece_id=GMVQKCFKGZU7AGQ7AT3ULHJCJM2B2VC2Z572ET3LZQJBD7X4M6NQ, seg_pos={0 0}, upload=474110]: Done uploading piece. err: <nil> Dec 28 04:05:30.896096206 (540h36m30.767179251s) [noise=true , address=77.90.31.13:4658 , piece_id=253DTNCSSNTDZR2Q56TT34OYAJPELSHIPO4PAOSNZ2KDQ3SUZXQA, seg_pos={0 0}, upload=474110]: Done uploading piece. err: <nil> Dec 28 04:05:31.410821180 (540h36m31.281904225s) [noise=true , address=145.79.227.11:4744 , piece_id=JTTI237QP4VHMZFDIX6MOUGYPFODL5GGUW3IDKZXQLVXCRHGN3YQ, seg_pos={0 0}, upload=474110]: Done uploading piece. err: <nil> Dec 28 04:05:31.494949853 (540h36m31.366032898s) [noise=true , address=145.79.227.40:4069 , piece_id=7YQ5FZTREX3LU3OYUHTXEMUISHIWPLXB74JUHF6O7CFCQBWN4QGQ, seg_pos={0 0}, upload=474110]: Done uploading piece. err: <nil> Dec 28 04:05:31.502794130 (540h36m31.373877175s) [noise=true , address=145.79.227.44:4221 , piece_id=JG45XU4WRCR6H343JZXFJ6JWNOJVNKBDREIHI6GXEENEAPI3LXUA, seg_pos={0 0}, upload=474110]: Done uploading piece. err: <nil> Dec 28 04:05:31.508696539 (540h36m31.379779574s) [noise=true , address=193.149.164.143:4695 , piece_id=HD5JUFDHKF5G4HA3CRIVTSAUH4MXW7HITIZLBD2J7FGGFMPQXOOQ, seg_pos={0 0}, upload=474110]: Done uploading piece. err: <nil> Dec 28 04:05:31.508716888 (540h36m31.379799923s) [seg_pos={0 0}, upload=474110]: Segment reached optimal threshold of 14 pieces. Dec 28 04:05:31.508997072 (540h36m31.380080117s) [noise=true , address=64.49.13.169:4709 , piece_id=A7RHDJ7GK5OZGUVBJW7X7FSYMVY6OIY3SHW66NCRU6SSISGCKYDA, seg_pos={0 0}, upload=474110]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:49772->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:49772->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49772->64.49.13.169:4709: use of closed network connection Dec 28 04:05:31.509004666 (540h36m31.380087701s) [noise=true , address=64.49.13.169:4709 , piece_id=A7RHDJ7GK5OZGUVBJW7X7FSYMVY6OIY3SHW66NCRU6SSISGCKYDA, seg_pos={0 0}, upload=474110]: Long tail cancellation due to optimal threshold Dec 28 04:05:31.509120086 (540h36m31.380203141s) [noise=true , address=45.91.116.12:4249 , piece_id=JM5XOZSORND2I3RGG2NVHN26CQDOK3VKNLJS6K4E2CKR62467RHQ, seg_pos={0 0}, upload=474110]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ghkFFJVqKzG9GbJjYVtcadUCr64bCY8dkEjbYGSShMG9e7WvA): protocol: read tcp 188.166.225.170:53796->45.91.116.12:4249: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 04:05:31.509136768 (540h36m31.380219823s) [noise=true , address=45.91.116.12:4249 , piece_id=JM5XOZSORND2I3RGG2NVHN26CQDOK3VKNLJS6K4E2CKR62467RHQ, seg_pos={0 0}, upload=474110]: Long tail cancellation due to optimal threshold Dec 28 04:05:31.509377987 (540h36m31.380461042s) [noise=true , address=145.79.227.34:4909 , piece_id=NDEZQCRKSSPYWQOQIDY7QMSKAUIL3O52MGDUQ3KVHTP35R75KWDQ, seg_pos={0 0}, upload=474110]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14Pqp5cbhmd44QnikAuD5TgsKFDR3gB2qtLuVSroaWVv4U2cDh): protocol: expected piece hash; context canceled; EOF Dec 28 04:05:31.509388978 (540h36m31.380472013s) [noise=true , address=145.79.227.34:4909 , piece_id=NDEZQCRKSSPYWQOQIDY7QMSKAUIL3O52MGDUQ3KVHTP35R75KWDQ, seg_pos={0 0}, upload=474110]: Long tail cancellation due to optimal threshold Dec 28 04:05:31.509451777 (540h36m31.380534832s) [noise=true , address=77.90.31.18:4819 , piece_id=NXICQKVF2XGMA7RTEL4DOWZ4Q5EKY2RIWL24LMJY47TS32L73F7Q, seg_pos={0 0}, upload=474110]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12nKVfrEuBKwyvtfYLJyfJ1Fo5gVVJU2Z6DNNvDEK8sKTmCW1Da): protocol: expected piece hash; context canceled; EOF Dec 28 04:05:31.509462578 (540h36m31.380545633s) [noise=true , address=77.90.31.18:4819 , piece_id=NXICQKVF2XGMA7RTEL4DOWZ4Q5EKY2RIWL24LMJY47TS32L73F7Q, seg_pos={0 0}, upload=474110]: Long tail cancellation due to optimal threshold Dec 28 04:05:31.509494629 (540h36m31.380577704s) [seg_pos={0 0}, upload=474110]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 04:05:31.509591002 (540h36m31.380674058s) [upload=474110]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:05:31.509597485 (540h36m31.38068053s) [upload=474110]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:05:31.807491599 (540h36m31.678574634s) [upload=474110]: Done uploading segments... Dec 28 04:05:31.807512519 (540h36m31.678595564s) [upload=474110]: Upload finished. err: <nil> Duration: 37.079467481s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 20 out of 60 is failed (threshold=6) [...X.XXXX....X.X.X...X..X.X.X..X...XX.....X......XX...X.X...] duration: 4.89681689s trace: https://jaeger.dev.storj.io/trace/693a9ad598fe2536 FAIL (2025-12-28 04:10:51 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.89686432s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 17.401544922s trace: https://jaeger.dev.storj.io/trace/3b8130f32e60318d FAIL (2025-12-28 04:10:50 UTC) orbiter warning: context deadline exceeded Duration: 17.401570991s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 52 out of 60 is failed (threshold=3) [XXXXXXX.XXXXX.XXXXXXXXXXXXXXXXX..XXXX.XXXX.XXXXX.XXXXXXX.XXX] duration: 30.004842743s trace: https://jaeger.dev.storj.io/trace/667d215170123035 FAIL (2025-12-28 04:10:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: APPTXR7TZISLPMXVFQ25FJIV5ZH4GVNM5L2S4BB3DDHVQ3OZQBZA) context deadline exceeded; piecestore: (Node ID: 1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, Piece ID: CYK57GK6R2WWQGOFPEWERO6FZ4KWQKTIAS2V2443QV7GPWY5XXZQ) context deadline exceeded ======LOGS====== Dec 28 04:10:15.798957108 (540h41m15.670040153s) [upload=474182]: Starting upload Dec 28 04:10:15.799042941 (540h41m15.670126006s) [upload=474182]: Done starting upload Dec 28 04:10:15.799130288 (540h41m15.670213323s) [upload=474182]: Uploading segments... Dec 28 04:10:15.799293598 (540h41m15.670376643s) [upload=474182]: Got next segment. Inline: false Dec 28 04:10:15.799332061 (540h41m15.670415096s) [upload=474182]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 04:10:15.799338093 (540h41m15.670421139s) [upload=474182]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 04:10:16.168454327 (540h41m16.039537372s) [seg_pos={0 0}, upload=474182]: Begin upload segment... Dec 28 04:10:16.168508080 (540h41m16.039591125s) [seg_pos={0 0}, upload=474182]: Done begin upload segment. Dec 28 04:10:16.168543026 (540h41m16.039626071s) [noise=true , address=145.79.227.45:4229 , piece_id=FZB2I5EHCBBUVEC6QVGHEWXEHNGMSQDOUK6PP2MMENENHD6CQEKA, seg_pos={0 0}, upload=474182]: Uploading piece... Dec 28 04:10:16.168606247 (540h41m16.039689292s) [noise=true , address=145.79.227.18:4007 , piece_id=MVJ6F2PCOPJP2ABHQYGOXD4E3QTEWX3IZ4DNNPR4UWNY2KH2WCJQ, seg_pos={0 0}, upload=474182]: Uploading piece... Dec 28 04:10:16.168760290 (540h41m16.039843325s) [noise=true , address=45.91.116.8:4046 , piece_id=SITTQ4YULCR5HIRRVJZ4UVJLBYOMAXSRXVPMUPU7UP2LRKYDQYEA, seg_pos={0 0}, upload=474182]: Uploading piece... Dec 28 04:10:16.169512462 (540h41m16.040595508s) [noise=true , address=77.90.31.13:4627 , piece_id=HA6ISUMSUZLO3MHNQOL7MCGPVJUG2LPGWSIV6NUYF2DJ6QZVJM6Q, seg_pos={0 0}, upload=474182]: Uploading piece... Dec 28 04:10:16.169773068 (540h41m16.040856113s) [noise=true , address=45.91.116.13:4155 , piece_id=RUZQ2WMTDYNWRPU64DSCJ5YPHDJNQ6DXEGHBVU5VUWZ5ESAJ7N7Q, seg_pos={0 0}, upload=474182]: Uploading piece... Dec 28 04:10:16.170245508 (540h41m16.041328553s) [noise=true , address=145.79.227.39:4015 , piece_id=C6S3Y2XTPOPWEQL3TBL6HYF7J2C4OMYGSLHIO6VWN4NW7TXTAXFQ, seg_pos={0 0}, upload=474182]: Uploading piece... Dec 28 04:10:16.170904714 (540h41m16.041987749s) [noise=true , address=45.91.116.11:4118 , piece_id=APPTXR7TZISLPMXVFQ25FJIV5ZH4GVNM5L2S4BB3DDHVQ3OZQBZA, seg_pos={0 0}, upload=474182]: Uploading piece... Dec 28 04:10:16.171308151 (540h41m16.042391186s) [noise=true , address=145.79.227.47:4265 , piece_id=2FUTRVN3S34QS2CEYEDW5RLKMXH52OMWXRCJW2PDXDBCE4ELZBYQ, seg_pos={0 0}, upload=474182]: Uploading piece... Dec 28 04:10:16.171673167 (540h41m16.042756202s) [noise=true , address=145.79.227.48:4303 , piece_id=A2O2HOD2ULCRF7PXDV5LXNQJZTKRE5R7H2IXLA4AH5U7DCHANDTQ, seg_pos={0 0}, upload=474182]: Uploading piece... Dec 28 04:10:16.172069561 (540h41m16.043152606s) [noise=true , address=145.79.227.31:4797 , piece_id=G6MRPBBHDNCIPU3VAASDCCQS4RV3DEWAPPUHPVHCFTGNKQD2Z5EA, seg_pos={0 0}, upload=474182]: Uploading piece... Dec 28 04:10:16.172511753 (540h41m16.043594799s) [noise=true , address=145.79.227.19:4079 , piece_id=G2CDTBMDCCZAHSL53BQN5F3VOEE6FJ3UHZIQYCRIFCSTTH5CKC4Q, seg_pos={0 0}, upload=474182]: Uploading piece... Dec 28 04:10:16.174530688 (540h41m16.045613733s) [noise=true , address=145.79.227.24:4583 , piece_id=3V5JRE62QRI3GGVAAVC46Z6R4YJRC4YW6IFFFDCIRLW2GDLCHDAQ, seg_pos={0 0}, upload=474182]: Uploading piece... Dec 28 04:10:16.176834995 (540h41m16.047918041s) [noise=true , address=77.90.31.19:4841 , piece_id=CYK57GK6R2WWQGOFPEWERO6FZ4KWQKTIAS2V2443QV7GPWY5XXZQ, seg_pos={0 0}, upload=474182]: Uploading piece... Dec 28 04:10:16.177390624 (540h41m16.048473669s) [noise=true , address=145.79.227.4:4186 , piece_id=IMRWJZC3ILUCDWAKTMP7QEDM4P4PNGXC2VPI4372PZ4PBJJAQ5OA, seg_pos={0 0}, upload=474182]: Uploading piece... Dec 28 04:10:16.177837855 (540h41m16.04892089s) [noise=true , address=145.79.227.13:4804 , piece_id=GBOLMCQS3GONSYUDAKTDZ3NQOZJZ2H2KOOXWUXCMTGRJACK74JZQ, seg_pos={0 0}, upload=474182]: Uploading piece... Dec 28 04:10:16.178331475 (540h41m16.04941452s) [noise=true , address=145.79.227.15:4874 , piece_id=JHRFOCFOMOM33RE3QHWDRKMY4QBFPWN56CASGFDS6HISS7CVTIFQ, seg_pos={0 0}, upload=474182]: Uploading piece... Dec 28 04:10:17.894285530 (540h41m17.765368565s) [upload=474182]: Next returned nil segment Dec 28 04:10:17.894332569 (540h41m17.765415604s) [upload=474182]: Waiting for error group managing segments... Dec 28 04:10:18.181855236 (540h41m18.052938282s) [noise=true , address=145.79.227.31:4797 , piece_id=G6MRPBBHDNCIPU3VAASDCCQS4RV3DEWAPPUHPVHCFTGNKQD2Z5EA, seg_pos={0 0}, upload=474182]: Done uploading piece. err: <nil> Dec 28 04:10:18.209042685 (540h41m18.08012572s) [noise=true , address=145.79.227.48:4303 , piece_id=A2O2HOD2ULCRF7PXDV5LXNQJZTKRE5R7H2IXLA4AH5U7DCHANDTQ, seg_pos={0 0}, upload=474182]: Done uploading piece. err: <nil> Dec 28 04:10:18.214514053 (540h41m18.085597098s) [noise=true , address=145.79.227.45:4229 , piece_id=FZB2I5EHCBBUVEC6QVGHEWXEHNGMSQDOUK6PP2MMENENHD6CQEKA, seg_pos={0 0}, upload=474182]: Done uploading piece. err: <nil> Dec 28 04:10:18.221950775 (540h41m18.09303382s) [noise=true , address=145.79.227.19:4079 , piece_id=G2CDTBMDCCZAHSL53BQN5F3VOEE6FJ3UHZIQYCRIFCSTTH5CKC4Q, seg_pos={0 0}, upload=474182]: Done uploading piece. err: <nil> Dec 28 04:10:18.234109016 (540h41m18.105192051s) [noise=true , address=145.79.227.18:4007 , piece_id=MVJ6F2PCOPJP2ABHQYGOXD4E3QTEWX3IZ4DNNPR4UWNY2KH2WCJQ, seg_pos={0 0}, upload=474182]: Done uploading piece. err: <nil> Dec 28 04:10:18.239465075 (540h41m18.11054812s) [noise=true , address=145.79.227.15:4874 , piece_id=JHRFOCFOMOM33RE3QHWDRKMY4QBFPWN56CASGFDS6HISS7CVTIFQ, seg_pos={0 0}, upload=474182]: Done uploading piece. err: <nil> Dec 28 04:10:18.241147117 (540h41m18.112230152s) [noise=true , address=145.79.227.47:4265 , piece_id=2FUTRVN3S34QS2CEYEDW5RLKMXH52OMWXRCJW2PDXDBCE4ELZBYQ, seg_pos={0 0}, upload=474182]: Done uploading piece. err: <nil> Dec 28 04:10:18.435912260 (540h41m18.306995305s) [noise=true , address=77.90.31.19:4841 , piece_id=CYK57GK6R2WWQGOFPEWERO6FZ4KWQKTIAS2V2443QV7GPWY5XXZQ, seg_pos={0 0}, upload=474182]: Done uploading piece. err: <nil> Dec 28 04:10:18.509530681 (540h41m18.380613716s) [noise=true , address=77.90.31.13:4627 , piece_id=HA6ISUMSUZLO3MHNQOL7MCGPVJUG2LPGWSIV6NUYF2DJ6QZVJM6Q, seg_pos={0 0}, upload=474182]: Done uploading piece. err: <nil> Dec 28 04:10:18.539621468 (540h41m18.410704513s) [noise=true , address=45.91.116.11:4118 , piece_id=APPTXR7TZISLPMXVFQ25FJIV5ZH4GVNM5L2S4BB3DDHVQ3OZQBZA, seg_pos={0 0}, upload=474182]: Done uploading piece. err: <nil> Dec 28 04:10:18.548350950 (540h41m18.419433995s) [noise=true , address=45.91.116.13:4155 , piece_id=RUZQ2WMTDYNWRPU64DSCJ5YPHDJNQ6DXEGHBVU5VUWZ5ESAJ7N7Q, seg_pos={0 0}, upload=474182]: Done uploading piece. err: <nil> Dec 28 04:10:19.241395730 (540h41m19.112478775s) [noise=true , address=45.91.116.8:4046 , piece_id=SITTQ4YULCR5HIRRVJZ4UVJLBYOMAXSRXVPMUPU7UP2LRKYDQYEA, seg_pos={0 0}, upload=474182]: Done uploading piece. err: <nil> Dec 28 04:10:19.404545702 (540h41m19.275628737s) [noise=true , address=145.79.227.24:4583 , piece_id=3V5JRE62QRI3GGVAAVC46Z6R4YJRC4YW6IFFFDCIRLW2GDLCHDAQ, seg_pos={0 0}, upload=474182]: Done uploading piece. err: <nil> Dec 28 04:10:19.404586460 (540h41m19.275669495s) [seg_pos={0 0}, upload=474182]: Segment reached optimal threshold of 13 pieces. Dec 28 04:10:19.404806990 (540h41m19.275890045s) [noise=true , address=145.79.227.39:4015 , piece_id=C6S3Y2XTPOPWEQL3TBL6HYF7J2C4OMYGSLHIO6VWN4NW7TXTAXFQ, seg_pos={0 0}, upload=474182]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE): protocol: expected piece hash; context canceled; EOF Dec 28 04:10:19.404815396 (540h41m19.275898441s) [noise=true , address=145.79.227.39:4015 , piece_id=C6S3Y2XTPOPWEQL3TBL6HYF7J2C4OMYGSLHIO6VWN4NW7TXTAXFQ, seg_pos={0 0}, upload=474182]: Long tail cancellation due to optimal threshold Dec 28 04:10:19.405026899 (540h41m19.276109944s) [noise=true , address=145.79.227.13:4804 , piece_id=GBOLMCQS3GONSYUDAKTDZ3NQOZJZ2H2KOOXWUXCMTGRJACK74JZQ, seg_pos={0 0}, upload=474182]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g): protocol: expected piece hash; context canceled; EOF Dec 28 04:10:19.405046155 (540h41m19.27612922s) [noise=true , address=145.79.227.13:4804 , piece_id=GBOLMCQS3GONSYUDAKTDZ3NQOZJZ2H2KOOXWUXCMTGRJACK74JZQ, seg_pos={0 0}, upload=474182]: Long tail cancellation due to optimal threshold Dec 28 04:10:19.405090520 (540h41m19.276173555s) [noise=true , address=145.79.227.4:4186 , piece_id=IMRWJZC3ILUCDWAKTMP7QEDM4P4PNGXC2VPI4372PZ4PBJJAQ5OA, seg_pos={0 0}, upload=474182]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht): protocol: expected piece hash; context canceled; EOF Dec 28 04:10:19.405099467 (540h41m19.276182502s) [noise=true , address=145.79.227.4:4186 , piece_id=IMRWJZC3ILUCDWAKTMP7QEDM4P4PNGXC2VPI4372PZ4PBJJAQ5OA, seg_pos={0 0}, upload=474182]: Long tail cancellation due to optimal threshold Dec 28 04:10:19.405121038 (540h41m19.276204073s) [seg_pos={0 0}, upload=474182]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 04:10:19.405200780 (540h41m19.276283825s) [upload=474182]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:10:19.405206811 (540h41m19.276289866s) [upload=474182]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:10:19.854739078 (540h41m19.725822113s) [upload=474182]: Done uploading segments... Dec 28 04:10:19.854760358 (540h41m19.725843393s) [upload=474182]: Upload finished. err: <nil> Duration: 30.004868943s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X...................................................X.....] duration: 52.851349295s trace: https://jaeger.dev.storj.io/trace/591b5c08c3d8a813 FAIL (2025-12-28 04:10:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== A, seg_pos={0 0}, upload=474110]: Uploading piece... Dec 28 04:05:28.680332318 (540h36m28.551415353s) [noise=true , address=145.79.227.43:4182 , piece_id=VZEVVQ4T5SS2PLTVY4FQDGZVB2DRPG2VMDUCP5ECZPIQBKG2RCSA, seg_pos={0 0}, upload=474110]: Uploading piece... Dec 28 04:05:28.680692264 (540h36m28.551775309s) [noise=true , address=64.49.13.169:4709 , piece_id=A7RHDJ7GK5OZGUVBJW7X7FSYMVY6OIY3SHW66NCRU6SSISGCKYDA, seg_pos={0 0}, upload=474110]: Uploading piece... Dec 28 04:05:28.681012203 (540h36m28.552095238s) [noise=true , address=77.90.31.18:4819 , piece_id=NXICQKVF2XGMA7RTEL4DOWZ4Q5EKY2RIWL24LMJY47TS32L73F7Q, seg_pos={0 0}, upload=474110]: Uploading piece... Dec 28 04:05:28.681332643 (540h36m28.552415678s) [noise=true , address=64.49.13.140:4507 , piece_id=PT6VHNLKRVNYJKIEYXRSA57LIQJMA3TKG3FCKJI5DDSDOXZVZCLQ, seg_pos={0 0}, upload=474110]: Uploading piece... Dec 28 04:05:28.681662891 (540h36m28.552745936s) [noise=true , address=77.90.31.12:4597 , piece_id=V4ZPTTPBTO6OEZM6Z66P4WIQ3OEOL2EOV2C6TWDS6Q27MOC6M3CQ, seg_pos={0 0}, upload=474110]: Uploading piece... Dec 28 04:05:28.682067222 (540h36m28.553150257s) [noise=true , address=145.79.227.40:4069 , piece_id=7YQ5FZTREX3LU3OYUHTXEMUISHIWPLXB74JUHF6O7CFCQBWN4QGQ, seg_pos={0 0}, upload=474110]: Uploading piece... Dec 28 04:05:28.682393643 (540h36m28.553476678s) [noise=true , address=145.79.227.11:4744 , piece_id=JTTI237QP4VHMZFDIX6MOUGYPFODL5GGUW3IDKZXQLVXCRHGN3YQ, seg_pos={0 0}, upload=474110]: Uploading piece... Dec 28 04:05:28.682756655 (540h36m28.55383969s) [noise=true , address=77.90.31.17:4779 , piece_id=GMVQKCFKGZU7AGQ7AT3ULHJCJM2B2VC2Z572ET3LZQJBD7X4M6NQ, seg_pos={0 0}, upload=474110]: Uploading piece... Dec 28 04:05:28.683195209 (540h36m28.554278254s) [noise=true , address=145.79.227.34:4909 , piece_id=NDEZQCRKSSPYWQOQIDY7QMSKAUIL3O52MGDUQ3KVHTP35R75KWDQ, seg_pos={0 0}, upload=474110]: Uploading piece... Dec 28 04:05:28.683539605 (540h36m28.55462265s) [noise=true , address=77.90.31.13:4658 , piece_id=253DTNCSSNTDZR2Q56TT34OYAJPELSHIPO4PAOSNZ2KDQ3SUZXQA, seg_pos={0 0}, upload=474110]: Uploading piece... Dec 28 04:05:28.689846314 (540h36m28.560929359s) [noise=true , address=77.90.31.11:4577 , piece_id=NMWLIKEFIMQUHQVA4342PSF3RQ5WAYSNHJPZJ5VU5PEJIISQMEGA, seg_pos={0 0}, upload=474110]: Uploading piece... Dec 28 04:05:28.692671473 (540h36m28.563754528s) [noise=true , address=193.149.164.143:4695 , piece_id=HD5JUFDHKF5G4HA3CRIVTSAUH4MXW7HITIZLBD2J7FGGFMPQXOOQ, seg_pos={0 0}, upload=474110]: Uploading piece... Dec 28 04:05:28.695438943 (540h36m28.566521989s) [noise=true , address=45.91.116.11:4139 , piece_id=35R4AZATBN22P2KLD6VNWHJD5TAS4HA5GSV7OU5CCBD4YUYUWJTA, seg_pos={0 0}, upload=474110]: Uploading piece... Dec 28 04:05:28.698799452 (540h36m28.569882487s) [noise=true , address=45.91.116.12:4249 , piece_id=JM5XOZSORND2I3RGG2NVHN26CQDOK3VKNLJS6K4E2CKR62467RHQ, seg_pos={0 0}, upload=474110]: Uploading piece... Dec 28 04:05:28.706563095 (540h36m28.57764613s) [noise=true , address=145.79.227.44:4221 , piece_id=JG45XU4WRCR6H343JZXFJ6JWNOJVNKBDREIHI6GXEENEAPI3LXUA, seg_pos={0 0}, upload=474110]: Uploading piece... Dec 28 04:05:30.092893206 (540h36m29.963976251s) [upload=474110]: Next returned nil segment Dec 28 04:05:30.092906743 (540h36m29.963989778s) [upload=474110]: Waiting for error group managing segments... Dec 28 04:05:30.603591324 (540h36m30.474674369s) [noise=true , address=145.79.227.43:4182 , piece_id=VZEVVQ4T5SS2PLTVY4FQDGZVB2DRPG2VMDUCP5ECZPIQBKG2RCSA, seg_pos={0 0}, upload=474110]: Done uploading piece. err: <nil> Dec 28 04:05:30.605517772 (540h36m30.476600807s) [noise=true , address=145.79.227.51:4432 , piece_id=YN66BQJBHIRZW2CCPL55RLISG5LCM53MUUVNJNN3L5UYE2CHJPDQ, seg_pos={0 0}, upload=474110]: Done uploading piece. err: <nil> Dec 28 04:05:30.649134745 (540h36m30.52021778s) [noise=true , address=64.49.13.140:4507 , piece_id=PT6VHNLKRVNYJKIEYXRSA57LIQJMA3TKG3FCKJI5DDSDOXZVZCLQ, seg_pos={0 0}, upload=474110]: Done uploading piece. err: <nil> Dec 28 04:05:30.721256115 (540h36m30.59233915s) [noise=true , address=77.90.31.21:4891 , piece_id=VW4VGEYAY5JNK6F3P6G4B4JZ2AW4O5CWRYXFBAKJ2HXQJ7UW6IJA, seg_pos={0 0}, upload=474110]: Done uploading piece. err: <nil> Dec 28 04:05:30.751313910 (540h36m30.622396955s) [noise=true , address=77.90.31.11:4577 , piece_id=NMWLIKEFIMQUHQVA4342PSF3RQ5WAYSNHJPZJ5VU5PEJIISQMEGA, seg_pos={0 0}, upload=474110]: Done uploading piece. err: <nil> Dec 28 04:05:30.800719756 (540h36m30.671802831s) [noise=true , address=45.91.116.11:4139 , piece_id=35R4AZATBN22P2KLD6VNWHJD5TAS4HA5GSV7OU5CCBD4YUYUWJTA, seg_pos={0 0}, upload=474110]: Done uploading piece. err: <nil> Dec 28 04:05:30.823740571 (540h36m30.694823606s) [noise=true , address=77.90.31.12:4597 , piece_id=V4ZPTTPBTO6OEZM6Z66P4WIQ3OEOL2EOV2C6TWDS6Q27MOC6M3CQ, seg_pos={0 0}, upload=474110]: Done uploading piece. err: <nil> Dec 28 04:05:30.847552863 (540h36m30.718635898s) [noise=true , address=145.79.227.12:4795 , piece_id=AQNYUYTFXYD6BXPHQ7UXWNY732L56PJ4KOZT5XZGLZYFZPND2QYA, seg_pos={0 0}, upload=474110]: Done uploading piece. err: <nil> Dec 28 04:05:30.850526996 (540h36m30.721610031s) [noise=true , address=77.90.31.17:4779 , piece_id=GMVQKCFKGZU7AGQ7AT3ULHJCJM2B2VC2Z572ET3LZQJBD7X4M6NQ, seg_pos={0 0}, upload=474110]: Done uploading piece. err: <nil> Dec 28 04:05:30.896096206 (540h36m30.767179251s) [noise=true , address=77.90.31.13:4658 , piece_id=253DTNCSSNTDZR2Q56TT34OYAJPELSHIPO4PAOSNZ2KDQ3SUZXQA, seg_pos={0 0}, upload=474110]: Done uploading piece. err: <nil> Dec 28 04:05:31.410821180 (540h36m31.281904225s) [noise=true , address=145.79.227.11:4744 , piece_id=JTTI237QP4VHMZFDIX6MOUGYPFODL5GGUW3IDKZXQLVXCRHGN3YQ, seg_pos={0 0}, upload=474110]: Done uploading piece. err: <nil> Dec 28 04:05:31.494949853 (540h36m31.366032898s) [noise=true , address=145.79.227.40:4069 , piece_id=7YQ5FZTREX3LU3OYUHTXEMUISHIWPLXB74JUHF6O7CFCQBWN4QGQ, seg_pos={0 0}, upload=474110]: Done uploading piece. err: <nil> Dec 28 04:05:31.502794130 (540h36m31.373877175s) [noise=true , address=145.79.227.44:4221 , piece_id=JG45XU4WRCR6H343JZXFJ6JWNOJVNKBDREIHI6GXEENEAPI3LXUA, seg_pos={0 0}, upload=474110]: Done uploading piece. err: <nil> Dec 28 04:05:31.508696539 (540h36m31.379779574s) [noise=true , address=193.149.164.143:4695 , piece_id=HD5JUFDHKF5G4HA3CRIVTSAUH4MXW7HITIZLBD2J7FGGFMPQXOOQ, seg_pos={0 0}, upload=474110]: Done uploading piece. err: <nil> Dec 28 04:05:31.508716888 (540h36m31.379799923s) [seg_pos={0 0}, upload=474110]: Segment reached optimal threshold of 14 pieces. Dec 28 04:05:31.508997072 (540h36m31.380080117s) [noise=true , address=64.49.13.169:4709 , piece_id=A7RHDJ7GK5OZGUVBJW7X7FSYMVY6OIY3SHW66NCRU6SSISGCKYDA, seg_pos={0 0}, upload=474110]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:49772->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:49772->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49772->64.49.13.169:4709: use of closed network connection Dec 28 04:05:31.509004666 (540h36m31.380087701s) [noise=true , address=64.49.13.169:4709 , piece_id=A7RHDJ7GK5OZGUVBJW7X7FSYMVY6OIY3SHW66NCRU6SSISGCKYDA, seg_pos={0 0}, upload=474110]: Long tail cancellation due to optimal threshold Dec 28 04:05:31.509120086 (540h36m31.380203141s) [noise=true , address=45.91.116.12:4249 , piece_id=JM5XOZSORND2I3RGG2NVHN26CQDOK3VKNLJS6K4E2CKR62467RHQ, seg_pos={0 0}, upload=474110]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ghkFFJVqKzG9GbJjYVtcadUCr64bCY8dkEjbYGSShMG9e7WvA): protocol: read tcp 188.166.225.170:53796->45.91.116.12:4249: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 04:05:31.509136768 (540h36m31.380219823s) [noise=true , address=45.91.116.12:4249 , piece_id=JM5XOZSORND2I3RGG2NVHN26CQDOK3VKNLJS6K4E2CKR62467RHQ, seg_pos={0 0}, upload=474110]: Long tail cancellation due to optimal threshold Dec 28 04:05:31.509377987 (540h36m31.380461042s) [noise=true , address=145.79.227.34:4909 , piece_id=NDEZQCRKSSPYWQOQIDY7QMSKAUIL3O52MGDUQ3KVHTP35R75KWDQ, seg_pos={0 0}, upload=474110]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14Pqp5cbhmd44QnikAuD5TgsKFDR3gB2qtLuVSroaWVv4U2cDh): protocol: expected piece hash; context canceled; EOF Dec 28 04:05:31.509388978 (540h36m31.380472013s) [noise=true , address=145.79.227.34:4909 , piece_id=NDEZQCRKSSPYWQOQIDY7QMSKAUIL3O52MGDUQ3KVHTP35R75KWDQ, seg_pos={0 0}, upload=474110]: Long tail cancellation due to optimal threshold Dec 28 04:05:31.509451777 (540h36m31.380534832s) [noise=true , address=77.90.31.18:4819 , piece_id=NXICQKVF2XGMA7RTEL4DOWZ4Q5EKY2RIWL24LMJY47TS32L73F7Q, seg_pos={0 0}, upload=474110]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12nKVfrEuBKwyvtfYLJyfJ1Fo5gVVJU2Z6DNNvDEK8sKTmCW1Da): protocol: expected piece hash; context canceled; EOF Dec 28 04:05:31.509462578 (540h36m31.380545633s) [noise=true , address=77.90.31.18:4819 , piece_id=NXICQKVF2XGMA7RTEL4DOWZ4Q5EKY2RIWL24LMJY47TS32L73F7Q, seg_pos={0 0}, upload=474110]: Long tail cancellation due to optimal threshold Dec 28 04:05:31.509494629 (540h36m31.380577704s) [seg_pos={0 0}, upload=474110]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 04:05:31.509591002 (540h36m31.380674058s) [upload=474110]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:05:31.509597485 (540h36m31.38068053s) [upload=474110]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:05:31.807491599 (540h36m31.678574634s) [upload=474110]: Done uploading segments... Dec 28 04:05:31.807512519 (540h36m31.678595564s) [upload=474110]: Upload finished. err: <nil> Duration: 52.851368943s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 20 out of 60 is failed (threshold=6) [....X.XXXX....X.X.X...X..X.X.X..X...XX.....X......XX...X.X..] duration: 3.581440706s trace: https://jaeger.dev.storj.io/trace/693a9ad598fe2536 FAIL (2025-12-28 04:10:25 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.58152716s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 16.377254981s trace: https://jaeger.dev.storj.io/trace/3b8130f32e60318d FAIL (2025-12-28 04:09:54 UTC) orbiter warning: context deadline exceeded Duration: 16.377288283s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X...................................................X....] duration: 26.783768821s trace: https://jaeger.dev.storj.io/trace/591b5c08c3d8a813 FAIL (2025-12-28 04:09:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== A, seg_pos={0 0}, upload=474110]: Uploading piece... Dec 28 04:05:28.680332318 (540h36m28.551415353s) [noise=true , address=145.79.227.43:4182 , piece_id=VZEVVQ4T5SS2PLTVY4FQDGZVB2DRPG2VMDUCP5ECZPIQBKG2RCSA, seg_pos={0 0}, upload=474110]: Uploading piece... Dec 28 04:05:28.680692264 (540h36m28.551775309s) [noise=true , address=64.49.13.169:4709 , piece_id=A7RHDJ7GK5OZGUVBJW7X7FSYMVY6OIY3SHW66NCRU6SSISGCKYDA, seg_pos={0 0}, upload=474110]: Uploading piece... Dec 28 04:05:28.681012203 (540h36m28.552095238s) [noise=true , address=77.90.31.18:4819 , piece_id=NXICQKVF2XGMA7RTEL4DOWZ4Q5EKY2RIWL24LMJY47TS32L73F7Q, seg_pos={0 0}, upload=474110]: Uploading piece... Dec 28 04:05:28.681332643 (540h36m28.552415678s) [noise=true , address=64.49.13.140:4507 , piece_id=PT6VHNLKRVNYJKIEYXRSA57LIQJMA3TKG3FCKJI5DDSDOXZVZCLQ, seg_pos={0 0}, upload=474110]: Uploading piece... Dec 28 04:05:28.681662891 (540h36m28.552745936s) [noise=true , address=77.90.31.12:4597 , piece_id=V4ZPTTPBTO6OEZM6Z66P4WIQ3OEOL2EOV2C6TWDS6Q27MOC6M3CQ, seg_pos={0 0}, upload=474110]: Uploading piece... Dec 28 04:05:28.682067222 (540h36m28.553150257s) [noise=true , address=145.79.227.40:4069 , piece_id=7YQ5FZTREX3LU3OYUHTXEMUISHIWPLXB74JUHF6O7CFCQBWN4QGQ, seg_pos={0 0}, upload=474110]: Uploading piece... Dec 28 04:05:28.682393643 (540h36m28.553476678s) [noise=true , address=145.79.227.11:4744 , piece_id=JTTI237QP4VHMZFDIX6MOUGYPFODL5GGUW3IDKZXQLVXCRHGN3YQ, seg_pos={0 0}, upload=474110]: Uploading piece... Dec 28 04:05:28.682756655 (540h36m28.55383969s) [noise=true , address=77.90.31.17:4779 , piece_id=GMVQKCFKGZU7AGQ7AT3ULHJCJM2B2VC2Z572ET3LZQJBD7X4M6NQ, seg_pos={0 0}, upload=474110]: Uploading piece... Dec 28 04:05:28.683195209 (540h36m28.554278254s) [noise=true , address=145.79.227.34:4909 , piece_id=NDEZQCRKSSPYWQOQIDY7QMSKAUIL3O52MGDUQ3KVHTP35R75KWDQ, seg_pos={0 0}, upload=474110]: Uploading piece... Dec 28 04:05:28.683539605 (540h36m28.55462265s) [noise=true , address=77.90.31.13:4658 , piece_id=253DTNCSSNTDZR2Q56TT34OYAJPELSHIPO4PAOSNZ2KDQ3SUZXQA, seg_pos={0 0}, upload=474110]: Uploading piece... Dec 28 04:05:28.689846314 (540h36m28.560929359s) [noise=true , address=77.90.31.11:4577 , piece_id=NMWLIKEFIMQUHQVA4342PSF3RQ5WAYSNHJPZJ5VU5PEJIISQMEGA, seg_pos={0 0}, upload=474110]: Uploading piece... Dec 28 04:05:28.692671473 (540h36m28.563754528s) [noise=true , address=193.149.164.143:4695 , piece_id=HD5JUFDHKF5G4HA3CRIVTSAUH4MXW7HITIZLBD2J7FGGFMPQXOOQ, seg_pos={0 0}, upload=474110]: Uploading piece... Dec 28 04:05:28.695438943 (540h36m28.566521989s) [noise=true , address=45.91.116.11:4139 , piece_id=35R4AZATBN22P2KLD6VNWHJD5TAS4HA5GSV7OU5CCBD4YUYUWJTA, seg_pos={0 0}, upload=474110]: Uploading piece... Dec 28 04:05:28.698799452 (540h36m28.569882487s) [noise=true , address=45.91.116.12:4249 , piece_id=JM5XOZSORND2I3RGG2NVHN26CQDOK3VKNLJS6K4E2CKR62467RHQ, seg_pos={0 0}, upload=474110]: Uploading piece... Dec 28 04:05:28.706563095 (540h36m28.57764613s) [noise=true , address=145.79.227.44:4221 , piece_id=JG45XU4WRCR6H343JZXFJ6JWNOJVNKBDREIHI6GXEENEAPI3LXUA, seg_pos={0 0}, upload=474110]: Uploading piece... Dec 28 04:05:30.092893206 (540h36m29.963976251s) [upload=474110]: Next returned nil segment Dec 28 04:05:30.092906743 (540h36m29.963989778s) [upload=474110]: Waiting for error group managing segments... Dec 28 04:05:30.603591324 (540h36m30.474674369s) [noise=true , address=145.79.227.43:4182 , piece_id=VZEVVQ4T5SS2PLTVY4FQDGZVB2DRPG2VMDUCP5ECZPIQBKG2RCSA, seg_pos={0 0}, upload=474110]: Done uploading piece. err: <nil> Dec 28 04:05:30.605517772 (540h36m30.476600807s) [noise=true , address=145.79.227.51:4432 , piece_id=YN66BQJBHIRZW2CCPL55RLISG5LCM53MUUVNJNN3L5UYE2CHJPDQ, seg_pos={0 0}, upload=474110]: Done uploading piece. err: <nil> Dec 28 04:05:30.649134745 (540h36m30.52021778s) [noise=true , address=64.49.13.140:4507 , piece_id=PT6VHNLKRVNYJKIEYXRSA57LIQJMA3TKG3FCKJI5DDSDOXZVZCLQ, seg_pos={0 0}, upload=474110]: Done uploading piece. err: <nil> Dec 28 04:05:30.721256115 (540h36m30.59233915s) [noise=true , address=77.90.31.21:4891 , piece_id=VW4VGEYAY5JNK6F3P6G4B4JZ2AW4O5CWRYXFBAKJ2HXQJ7UW6IJA, seg_pos={0 0}, upload=474110]: Done uploading piece. err: <nil> Dec 28 04:05:30.751313910 (540h36m30.622396955s) [noise=true , address=77.90.31.11:4577 , piece_id=NMWLIKEFIMQUHQVA4342PSF3RQ5WAYSNHJPZJ5VU5PEJIISQMEGA, seg_pos={0 0}, upload=474110]: Done uploading piece. err: <nil> Dec 28 04:05:30.800719756 (540h36m30.671802831s) [noise=true , address=45.91.116.11:4139 , piece_id=35R4AZATBN22P2KLD6VNWHJD5TAS4HA5GSV7OU5CCBD4YUYUWJTA, seg_pos={0 0}, upload=474110]: Done uploading piece. err: <nil> Dec 28 04:05:30.823740571 (540h36m30.694823606s) [noise=true , address=77.90.31.12:4597 , piece_id=V4ZPTTPBTO6OEZM6Z66P4WIQ3OEOL2EOV2C6TWDS6Q27MOC6M3CQ, seg_pos={0 0}, upload=474110]: Done uploading piece. err: <nil> Dec 28 04:05:30.847552863 (540h36m30.718635898s) [noise=true , address=145.79.227.12:4795 , piece_id=AQNYUYTFXYD6BXPHQ7UXWNY732L56PJ4KOZT5XZGLZYFZPND2QYA, seg_pos={0 0}, upload=474110]: Done uploading piece. err: <nil> Dec 28 04:05:30.850526996 (540h36m30.721610031s) [noise=true , address=77.90.31.17:4779 , piece_id=GMVQKCFKGZU7AGQ7AT3ULHJCJM2B2VC2Z572ET3LZQJBD7X4M6NQ, seg_pos={0 0}, upload=474110]: Done uploading piece. err: <nil> Dec 28 04:05:30.896096206 (540h36m30.767179251s) [noise=true , address=77.90.31.13:4658 , piece_id=253DTNCSSNTDZR2Q56TT34OYAJPELSHIPO4PAOSNZ2KDQ3SUZXQA, seg_pos={0 0}, upload=474110]: Done uploading piece. err: <nil> Dec 28 04:05:31.410821180 (540h36m31.281904225s) [noise=true , address=145.79.227.11:4744 , piece_id=JTTI237QP4VHMZFDIX6MOUGYPFODL5GGUW3IDKZXQLVXCRHGN3YQ, seg_pos={0 0}, upload=474110]: Done uploading piece. err: <nil> Dec 28 04:05:31.494949853 (540h36m31.366032898s) [noise=true , address=145.79.227.40:4069 , piece_id=7YQ5FZTREX3LU3OYUHTXEMUISHIWPLXB74JUHF6O7CFCQBWN4QGQ, seg_pos={0 0}, upload=474110]: Done uploading piece. err: <nil> Dec 28 04:05:31.502794130 (540h36m31.373877175s) [noise=true , address=145.79.227.44:4221 , piece_id=JG45XU4WRCR6H343JZXFJ6JWNOJVNKBDREIHI6GXEENEAPI3LXUA, seg_pos={0 0}, upload=474110]: Done uploading piece. err: <nil> Dec 28 04:05:31.508696539 (540h36m31.379779574s) [noise=true , address=193.149.164.143:4695 , piece_id=HD5JUFDHKF5G4HA3CRIVTSAUH4MXW7HITIZLBD2J7FGGFMPQXOOQ, seg_pos={0 0}, upload=474110]: Done uploading piece. err: <nil> Dec 28 04:05:31.508716888 (540h36m31.379799923s) [seg_pos={0 0}, upload=474110]: Segment reached optimal threshold of 14 pieces. Dec 28 04:05:31.508997072 (540h36m31.380080117s) [noise=true , address=64.49.13.169:4709 , piece_id=A7RHDJ7GK5OZGUVBJW7X7FSYMVY6OIY3SHW66NCRU6SSISGCKYDA, seg_pos={0 0}, upload=474110]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:49772->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:49772->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49772->64.49.13.169:4709: use of closed network connection Dec 28 04:05:31.509004666 (540h36m31.380087701s) [noise=true , address=64.49.13.169:4709 , piece_id=A7RHDJ7GK5OZGUVBJW7X7FSYMVY6OIY3SHW66NCRU6SSISGCKYDA, seg_pos={0 0}, upload=474110]: Long tail cancellation due to optimal threshold Dec 28 04:05:31.509120086 (540h36m31.380203141s) [noise=true , address=45.91.116.12:4249 , piece_id=JM5XOZSORND2I3RGG2NVHN26CQDOK3VKNLJS6K4E2CKR62467RHQ, seg_pos={0 0}, upload=474110]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ghkFFJVqKzG9GbJjYVtcadUCr64bCY8dkEjbYGSShMG9e7WvA): protocol: read tcp 188.166.225.170:53796->45.91.116.12:4249: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 04:05:31.509136768 (540h36m31.380219823s) [noise=true , address=45.91.116.12:4249 , piece_id=JM5XOZSORND2I3RGG2NVHN26CQDOK3VKNLJS6K4E2CKR62467RHQ, seg_pos={0 0}, upload=474110]: Long tail cancellation due to optimal threshold Dec 28 04:05:31.509377987 (540h36m31.380461042s) [noise=true , address=145.79.227.34:4909 , piece_id=NDEZQCRKSSPYWQOQIDY7QMSKAUIL3O52MGDUQ3KVHTP35R75KWDQ, seg_pos={0 0}, upload=474110]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14Pqp5cbhmd44QnikAuD5TgsKFDR3gB2qtLuVSroaWVv4U2cDh): protocol: expected piece hash; context canceled; EOF Dec 28 04:05:31.509388978 (540h36m31.380472013s) [noise=true , address=145.79.227.34:4909 , piece_id=NDEZQCRKSSPYWQOQIDY7QMSKAUIL3O52MGDUQ3KVHTP35R75KWDQ, seg_pos={0 0}, upload=474110]: Long tail cancellation due to optimal threshold Dec 28 04:05:31.509451777 (540h36m31.380534832s) [noise=true , address=77.90.31.18:4819 , piece_id=NXICQKVF2XGMA7RTEL4DOWZ4Q5EKY2RIWL24LMJY47TS32L73F7Q, seg_pos={0 0}, upload=474110]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12nKVfrEuBKwyvtfYLJyfJ1Fo5gVVJU2Z6DNNvDEK8sKTmCW1Da): protocol: expected piece hash; context canceled; EOF Dec 28 04:05:31.509462578 (540h36m31.380545633s) [noise=true , address=77.90.31.18:4819 , piece_id=NXICQKVF2XGMA7RTEL4DOWZ4Q5EKY2RIWL24LMJY47TS32L73F7Q, seg_pos={0 0}, upload=474110]: Long tail cancellation due to optimal threshold Dec 28 04:05:31.509494629 (540h36m31.380577704s) [seg_pos={0 0}, upload=474110]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 04:05:31.509591002 (540h36m31.380674058s) [upload=474110]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:05:31.509597485 (540h36m31.38068053s) [upload=474110]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:05:31.807491599 (540h36m31.678574634s) [upload=474110]: Done uploading segments... Dec 28 04:05:31.807512519 (540h36m31.678595564s) [upload=474110]: Upload finished. err: <nil> Duration: 26.78380007s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 52 out of 60 is failed (threshold=3) [XXXXXXXX.XXXXX.XXXXXXXXXXXXXXXXX..XXXX.XXXX.XXXXX.XXXXXXX.XX] duration: 30.002738965s trace: https://jaeger.dev.storj.io/trace/2e4a44c3bcf2f9d2 FAIL (2025-12-28 04:09:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 04:09:01.746818283 (540h40m1.617901328s) [upload=474161]: Starting upload Dec 28 04:09:01.746921249 (540h40m1.618004294s) [upload=474161]: Done starting upload Dec 28 04:09:01.747045206 (540h40m1.618128251s) [upload=474161]: Uploading segments... Dec 28 04:09:01.747202946 (540h40m1.618285991s) [upload=474161]: Got next segment. Inline: false Dec 28 04:09:01.747265335 (540h40m1.61834838s) [upload=474161]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 04:09:01.747271196 (540h40m1.618354241s) [upload=474161]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 04:09:02.081447337 (540h40m1.952530402s) [seg_pos={0 0}, upload=474161]: Begin upload segment... Dec 28 04:09:02.081513463 (540h40m1.952596518s) [seg_pos={0 0}, upload=474161]: Done begin upload segment. Dec 28 04:09:02.081553238 (540h40m1.952636284s) [noise=true , address=77.90.31.15:4697 , piece_id=7OD5XY36BGG4PWLY73NCA6KECUTYAUQXMRB73FTYCIHZIBVJ3ZIQ, seg_pos={0 0}, upload=474161]: Uploading piece... Dec 28 04:09:02.081711009 (540h40m1.952794054s) [noise=true , address=145.79.227.14:4839 , piece_id=XYEF6BIGR4GZOXZG66FSXGXH2NH3DQOAKFI4QH4YV22S77HZLFCA, seg_pos={0 0}, upload=474161]: Uploading piece... Dec 28 04:09:02.083444591 (540h40m1.954527636s) [noise=true , address=145.79.227.11:4730 , piece_id=EQXWSOWYPT7F5L5WB5PQ6TWIRXQN5NLR6V3RR4IFXSU4EZ5ED7BQ, seg_pos={0 0}, upload=474161]: Uploading piece... Dec 28 04:09:02.083839703 (540h40m1.954922748s) [noise=true , address=145.79.227.38:4979 , piece_id=VE4NECQ376EWIKD7XFDNJSX7FX6LGFGH5B6VBRZBYPNTHVW7GYMA, seg_pos={0 0}, upload=474161]: Uploading piece... Dec 28 04:09:02.084712064 (540h40m1.955795119s) [noise=true , address=145.79.227.16:4910 , piece_id=Y4XKJG5ZITU3PI3V2KZV74MTVRWGNJKSFW3DI54H5ITINLSMYDXQ, seg_pos={0 0}, upload=474161]: Uploading piece... Dec 28 04:09:02.085441272 (540h40m1.956524317s) [noise=true , address=45.91.116.6:4976 , piece_id=X7F3HUALM2ON3PGESJWLFELYFR7LPQKRBS335N5QQ56IFVMQBQFQ, seg_pos={0 0}, upload=474161]: Uploading piece... Dec 28 04:09:02.086321268 (540h40m1.957404323s) [noise=true , address=145.79.227.17:4972 , piece_id=DMPB6O5SV6LBKNUW3DRXZVFRR27QECOJ6MNWMYER3YKD2MOD2GAA, seg_pos={0 0}, upload=474161]: Uploading piece... Dec 28 04:09:02.086464441 (540h40m1.957547486s) [noise=true , address=45.91.116.10:4084 , piece_id=MIHKDHFCCMRCEYNFHA37VVOMICEJLOAJFMMDN5LDVHROV3V7FO6Q, seg_pos={0 0}, upload=474161]: Uploading piece... Dec 28 04:09:02.087237743 (540h40m1.958320799s) [noise=true , address=145.79.227.15:4873 , piece_id=JW4ZABLIP6GGR7G2QGY4RWTD3XK7D5HY3PN2SCGSAVKTT7KDQIYQ, seg_pos={0 0}, upload=474161]: Uploading piece... Dec 28 04:09:02.087281807 (540h40m1.958364852s) [noise=true , address=145.79.227.29:4943 , piece_id=4ZO6COAKDVZ4OID3K4WPDYK7OTOYHTDVX3WCVZ4YETFXSI4WIRYA, seg_pos={0 0}, upload=474161]: Uploading piece... Dec 28 04:09:02.087949819 (540h40m1.959032874s) [noise=true , address=145.79.227.27:4691 , piece_id=KJA5AVWXM2CL4UWZEE3CMCJSNLK2TN2NWCBOCKKRBNBQ6X7BUWHA, seg_pos={0 0}, upload=474161]: Uploading piece... Dec 28 04:09:02.088005155 (540h40m1.9590882s) [noise=true , address=45.91.116.13:4154 , piece_id=IFK2U5AHWTWEO6LVM6OJDRUKOJNCDLPSV7UXE7R2WQIMW2BOTZNA, seg_pos={0 0}, upload=474161]: Uploading piece... Dec 28 04:09:02.088448559 (540h40m1.959531614s) [noise=true , address=77.90.31.18:4808 , piece_id=AMMUWO5R6OYMOAQLRBZKBPZ5PNFQMO3N6Y6VBZHMQIOG5TWR6E3Q, seg_pos={0 0}, upload=474161]: Uploading piece... Dec 28 04:09:02.088691532 (540h40m1.959774577s) [noise=true , address=77.90.31.10:4517 , piece_id=5RVHGUQOEGPFZ67CNISNMFXFVF5TB2ZODKOV4EKM6FL7JKINCNPQ, seg_pos={0 0}, upload=474161]: Uploading piece... Dec 28 04:09:02.088942570 (540h40m1.960025625s) [noise=true , address=77.90.31.14:4663 , piece_id=WWMEJZLUGLJB27Y7B5LVOZS2YEF2YCG373XRPUCH72RKHMIWMP7A, seg_pos={0 0}, upload=474161]: Uploading piece... Dec 28 04:09:02.089449795 (540h40m1.96053284s) [noise=true , address=145.79.227.46:4602 , piece_id=MBTC2M2PTMNTQZFCYBHYENSVXDEPZ6R6OYQ7LQSU3BTXIM6ZHBZA, seg_pos={0 0}, upload=474161]: Uploading piece... Dec 28 04:09:03.721180477 (540h40m3.592263522s) [upload=474161]: Next returned nil segment Dec 28 04:09:03.721209131 (540h40m3.592292207s) [upload=474161]: Waiting for error group managing segments... Dec 28 04:09:04.102038538 (540h40m3.973121583s) [noise=true , address=145.79.227.46:4602 , piece_id=MBTC2M2PTMNTQZFCYBHYENSVXDEPZ6R6OYQ7LQSU3BTXIM6ZHBZA, seg_pos={0 0}, upload=474161]: Done uploading piece. err: <nil> Dec 28 04:09:04.108234547 (540h40m3.979317592s) [noise=true , address=145.79.227.14:4839 , piece_id=XYEF6BIGR4GZOXZG66FSXGXH2NH3DQOAKFI4QH4YV22S77HZLFCA, seg_pos={0 0}, upload=474161]: Done uploading piece. err: <nil> Dec 28 04:09:04.123562433 (540h40m3.994645478s) [noise=true , address=145.79.227.16:4910 , piece_id=Y4XKJG5ZITU3PI3V2KZV74MTVRWGNJKSFW3DI54H5ITINLSMYDXQ, seg_pos={0 0}, upload=474161]: Done uploading piece. err: <nil> Dec 28 04:09:04.129360494 (540h40m4.00044368s) [noise=true , address=145.79.227.29:4943 , piece_id=4ZO6COAKDVZ4OID3K4WPDYK7OTOYHTDVX3WCVZ4YETFXSI4WIRYA, seg_pos={0 0}, upload=474161]: Done uploading piece. err: <nil> Dec 28 04:09:04.144127292 (540h40m4.015210357s) [noise=true , address=145.79.227.17:4972 , piece_id=DMPB6O5SV6LBKNUW3DRXZVFRR27QECOJ6MNWMYER3YKD2MOD2GAA, seg_pos={0 0}, upload=474161]: Done uploading piece. err: <nil> Dec 28 04:09:04.148193845 (540h40m4.01927689s) [noise=true , address=145.79.227.27:4691 , piece_id=KJA5AVWXM2CL4UWZEE3CMCJSNLK2TN2NWCBOCKKRBNBQ6X7BUWHA, seg_pos={0 0}, upload=474161]: Done uploading piece. err: <nil> Dec 28 04:09:04.154548656 (540h40m4.025631701s) [noise=true , address=145.79.227.11:4730 , piece_id=EQXWSOWYPT7F5L5WB5PQ6TWIRXQN5NLR6V3RR4IFXSU4EZ5ED7BQ, seg_pos={0 0}, upload=474161]: Done uploading piece. err: <nil> Dec 28 04:09:04.177601822 (540h40m4.048684867s) [noise=true , address=145.79.227.38:4979 , piece_id=VE4NECQ376EWIKD7XFDNJSX7FX6LGFGH5B6VBRZBYPNTHVW7GYMA, seg_pos={0 0}, upload=474161]: Done uploading piece. err: <nil> Dec 28 04:09:04.326213051 (540h40m4.197296096s) [noise=true , address=45.91.116.10:4084 , piece_id=MIHKDHFCCMRCEYNFHA37VVOMICEJLOAJFMMDN5LDVHROV3V7FO6Q, seg_pos={0 0}, upload=474161]: Done uploading piece. err: <nil> Dec 28 04:09:04.397256479 (540h40m4.268339524s) [noise=true , address=77.90.31.14:4663 , piece_id=WWMEJZLUGLJB27Y7B5LVOZS2YEF2YCG373XRPUCH72RKHMIWMP7A, seg_pos={0 0}, upload=474161]: Done uploading piece. err: <nil> Dec 28 04:09:04.398298974 (540h40m4.269382019s) [noise=true , address=77.90.31.18:4808 , piece_id=AMMUWO5R6OYMOAQLRBZKBPZ5PNFQMO3N6Y6VBZHMQIOG5TWR6E3Q, seg_pos={0 0}, upload=474161]: Done uploading piece. err: <nil> Dec 28 04:09:04.437306349 (540h40m4.308389394s) [noise=true , address=77.90.31.10:4517 , piece_id=5RVHGUQOEGPFZ67CNISNMFXFVF5TB2ZODKOV4EKM6FL7JKINCNPQ, seg_pos={0 0}, upload=474161]: Done uploading piece. err: <nil> Dec 28 04:09:04.689028086 (540h40m4.560111141s) [noise=true , address=45.91.116.13:4154 , piece_id=IFK2U5AHWTWEO6LVM6OJDRUKOJNCDLPSV7UXE7R2WQIMW2BOTZNA, seg_pos={0 0}, upload=474161]: Done uploading piece. err: <nil> Dec 28 04:09:04.689057021 (540h40m4.560140066s) [seg_pos={0 0}, upload=474161]: Segment reached optimal threshold of 13 pieces. Dec 28 04:09:04.689288442 (540h40m4.560371477s) [noise=true , address=45.91.116.6:4976 , piece_id=X7F3HUALM2ON3PGESJWLFELYFR7LPQKRBS335N5QQ56IFVMQBQFQ, seg_pos={0 0}, upload=474161]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: write tcp 188.166.225.170:40006->45.91.116.6:4976: use of closed network connection; write tcp 188.166.225.170:40006->45.91.116.6:4976: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40006->45.91.116.6:4976: use of closed network connection Dec 28 04:09:04.689298160 (540h40m4.560381205s) [noise=true , address=45.91.116.6:4976 , piece_id=X7F3HUALM2ON3PGESJWLFELYFR7LPQKRBS335N5QQ56IFVMQBQFQ, seg_pos={0 0}, upload=474161]: Long tail cancellation due to optimal threshold Dec 28 04:09:04.689346602 (540h40m4.560429647s) [noise=true , address=77.90.31.15:4697 , piece_id=7OD5XY36BGG4PWLY73NCA6KECUTYAUQXMRB73FTYCIHZIBVJ3ZIQ, seg_pos={0 0}, upload=474161]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Dec 28 04:09:04.689357173 (540h40m4.560440208s) [noise=true , address=77.90.31.15:4697 , piece_id=7OD5XY36BGG4PWLY73NCA6KECUTYAUQXMRB73FTYCIHZIBVJ3ZIQ, seg_pos={0 0}, upload=474161]: Long tail cancellation due to optimal threshold Dec 28 04:09:04.689539339 (540h40m4.560622384s) [noise=true , address=145.79.227.15:4873 , piece_id=JW4ZABLIP6GGR7G2QGY4RWTD3XK7D5HY3PN2SCGSAVKTT7KDQIYQ, seg_pos={0 0}, upload=474161]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3): protocol: expected piece hash; context canceled; EOF Dec 28 04:09:04.689546272 (540h40m4.560629317s) [noise=true , address=145.79.227.15:4873 , piece_id=JW4ZABLIP6GGR7G2QGY4RWTD3XK7D5HY3PN2SCGSAVKTT7KDQIYQ, seg_pos={0 0}, upload=474161]: Long tail cancellation due to optimal threshold Dec 28 04:09:04.689584325 (540h40m4.56066736s) [seg_pos={0 0}, upload=474161]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 04:09:04.689680648 (540h40m4.560763683s) [upload=474161]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:09:04.689686479 (540h40m4.560769524s) [upload=474161]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:09:05.039254891 (540h40m4.910337946s) [upload=474161]: Done uploading segments... Dec 28 04:09:05.039281702 (540h40m4.910364747s) [upload=474161]: Upload finished. err: <nil> Duration: 30.002760916s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 21 out of 60 is failed (threshold=6) [X....X.XXXX....X.X.X...X..X.X.X..X...XX.....X......XX...X.X.] duration: 4.7053846s trace: https://jaeger.dev.storj.io/trace/693a9ad598fe2536 FAIL (2025-12-28 04:09:30 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.705911233s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 11.939525148s trace: https://jaeger.dev.storj.io/trace/3b8130f32e60318d FAIL (2025-12-28 04:09:00 UTC) orbiter warning: context deadline exceeded Duration: 11.939555836s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [X...X...................................................X...] duration: 34.691282319s trace: https://jaeger.dev.storj.io/trace/591b5c08c3d8a813 FAIL (2025-12-28 04:08:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== A, seg_pos={0 0}, upload=474110]: Uploading piece... Dec 28 04:05:28.680332318 (540h36m28.551415353s) [noise=true , address=145.79.227.43:4182 , piece_id=VZEVVQ4T5SS2PLTVY4FQDGZVB2DRPG2VMDUCP5ECZPIQBKG2RCSA, seg_pos={0 0}, upload=474110]: Uploading piece... Dec 28 04:05:28.680692264 (540h36m28.551775309s) [noise=true , address=64.49.13.169:4709 , piece_id=A7RHDJ7GK5OZGUVBJW7X7FSYMVY6OIY3SHW66NCRU6SSISGCKYDA, seg_pos={0 0}, upload=474110]: Uploading piece... Dec 28 04:05:28.681012203 (540h36m28.552095238s) [noise=true , address=77.90.31.18:4819 , piece_id=NXICQKVF2XGMA7RTEL4DOWZ4Q5EKY2RIWL24LMJY47TS32L73F7Q, seg_pos={0 0}, upload=474110]: Uploading piece... Dec 28 04:05:28.681332643 (540h36m28.552415678s) [noise=true , address=64.49.13.140:4507 , piece_id=PT6VHNLKRVNYJKIEYXRSA57LIQJMA3TKG3FCKJI5DDSDOXZVZCLQ, seg_pos={0 0}, upload=474110]: Uploading piece... Dec 28 04:05:28.681662891 (540h36m28.552745936s) [noise=true , address=77.90.31.12:4597 , piece_id=V4ZPTTPBTO6OEZM6Z66P4WIQ3OEOL2EOV2C6TWDS6Q27MOC6M3CQ, seg_pos={0 0}, upload=474110]: Uploading piece... Dec 28 04:05:28.682067222 (540h36m28.553150257s) [noise=true , address=145.79.227.40:4069 , piece_id=7YQ5FZTREX3LU3OYUHTXEMUISHIWPLXB74JUHF6O7CFCQBWN4QGQ, seg_pos={0 0}, upload=474110]: Uploading piece... Dec 28 04:05:28.682393643 (540h36m28.553476678s) [noise=true , address=145.79.227.11:4744 , piece_id=JTTI237QP4VHMZFDIX6MOUGYPFODL5GGUW3IDKZXQLVXCRHGN3YQ, seg_pos={0 0}, upload=474110]: Uploading piece... Dec 28 04:05:28.682756655 (540h36m28.55383969s) [noise=true , address=77.90.31.17:4779 , piece_id=GMVQKCFKGZU7AGQ7AT3ULHJCJM2B2VC2Z572ET3LZQJBD7X4M6NQ, seg_pos={0 0}, upload=474110]: Uploading piece... Dec 28 04:05:28.683195209 (540h36m28.554278254s) [noise=true , address=145.79.227.34:4909 , piece_id=NDEZQCRKSSPYWQOQIDY7QMSKAUIL3O52MGDUQ3KVHTP35R75KWDQ, seg_pos={0 0}, upload=474110]: Uploading piece... Dec 28 04:05:28.683539605 (540h36m28.55462265s) [noise=true , address=77.90.31.13:4658 , piece_id=253DTNCSSNTDZR2Q56TT34OYAJPELSHIPO4PAOSNZ2KDQ3SUZXQA, seg_pos={0 0}, upload=474110]: Uploading piece... Dec 28 04:05:28.689846314 (540h36m28.560929359s) [noise=true , address=77.90.31.11:4577 , piece_id=NMWLIKEFIMQUHQVA4342PSF3RQ5WAYSNHJPZJ5VU5PEJIISQMEGA, seg_pos={0 0}, upload=474110]: Uploading piece... Dec 28 04:05:28.692671473 (540h36m28.563754528s) [noise=true , address=193.149.164.143:4695 , piece_id=HD5JUFDHKF5G4HA3CRIVTSAUH4MXW7HITIZLBD2J7FGGFMPQXOOQ, seg_pos={0 0}, upload=474110]: Uploading piece... Dec 28 04:05:28.695438943 (540h36m28.566521989s) [noise=true , address=45.91.116.11:4139 , piece_id=35R4AZATBN22P2KLD6VNWHJD5TAS4HA5GSV7OU5CCBD4YUYUWJTA, seg_pos={0 0}, upload=474110]: Uploading piece... Dec 28 04:05:28.698799452 (540h36m28.569882487s) [noise=true , address=45.91.116.12:4249 , piece_id=JM5XOZSORND2I3RGG2NVHN26CQDOK3VKNLJS6K4E2CKR62467RHQ, seg_pos={0 0}, upload=474110]: Uploading piece... Dec 28 04:05:28.706563095 (540h36m28.57764613s) [noise=true , address=145.79.227.44:4221 , piece_id=JG45XU4WRCR6H343JZXFJ6JWNOJVNKBDREIHI6GXEENEAPI3LXUA, seg_pos={0 0}, upload=474110]: Uploading piece... Dec 28 04:05:30.092893206 (540h36m29.963976251s) [upload=474110]: Next returned nil segment Dec 28 04:05:30.092906743 (540h36m29.963989778s) [upload=474110]: Waiting for error group managing segments... Dec 28 04:05:30.603591324 (540h36m30.474674369s) [noise=true , address=145.79.227.43:4182 , piece_id=VZEVVQ4T5SS2PLTVY4FQDGZVB2DRPG2VMDUCP5ECZPIQBKG2RCSA, seg_pos={0 0}, upload=474110]: Done uploading piece. err: <nil> Dec 28 04:05:30.605517772 (540h36m30.476600807s) [noise=true , address=145.79.227.51:4432 , piece_id=YN66BQJBHIRZW2CCPL55RLISG5LCM53MUUVNJNN3L5UYE2CHJPDQ, seg_pos={0 0}, upload=474110]: Done uploading piece. err: <nil> Dec 28 04:05:30.649134745 (540h36m30.52021778s) [noise=true , address=64.49.13.140:4507 , piece_id=PT6VHNLKRVNYJKIEYXRSA57LIQJMA3TKG3FCKJI5DDSDOXZVZCLQ, seg_pos={0 0}, upload=474110]: Done uploading piece. err: <nil> Dec 28 04:05:30.721256115 (540h36m30.59233915s) [noise=true , address=77.90.31.21:4891 , piece_id=VW4VGEYAY5JNK6F3P6G4B4JZ2AW4O5CWRYXFBAKJ2HXQJ7UW6IJA, seg_pos={0 0}, upload=474110]: Done uploading piece. err: <nil> Dec 28 04:05:30.751313910 (540h36m30.622396955s) [noise=true , address=77.90.31.11:4577 , piece_id=NMWLIKEFIMQUHQVA4342PSF3RQ5WAYSNHJPZJ5VU5PEJIISQMEGA, seg_pos={0 0}, upload=474110]: Done uploading piece. err: <nil> Dec 28 04:05:30.800719756 (540h36m30.671802831s) [noise=true , address=45.91.116.11:4139 , piece_id=35R4AZATBN22P2KLD6VNWHJD5TAS4HA5GSV7OU5CCBD4YUYUWJTA, seg_pos={0 0}, upload=474110]: Done uploading piece. err: <nil> Dec 28 04:05:30.823740571 (540h36m30.694823606s) [noise=true , address=77.90.31.12:4597 , piece_id=V4ZPTTPBTO6OEZM6Z66P4WIQ3OEOL2EOV2C6TWDS6Q27MOC6M3CQ, seg_pos={0 0}, upload=474110]: Done uploading piece. err: <nil> Dec 28 04:05:30.847552863 (540h36m30.718635898s) [noise=true , address=145.79.227.12:4795 , piece_id=AQNYUYTFXYD6BXPHQ7UXWNY732L56PJ4KOZT5XZGLZYFZPND2QYA, seg_pos={0 0}, upload=474110]: Done uploading piece. err: <nil> Dec 28 04:05:30.850526996 (540h36m30.721610031s) [noise=true , address=77.90.31.17:4779 , piece_id=GMVQKCFKGZU7AGQ7AT3ULHJCJM2B2VC2Z572ET3LZQJBD7X4M6NQ, seg_pos={0 0}, upload=474110]: Done uploading piece. err: <nil> Dec 28 04:05:30.896096206 (540h36m30.767179251s) [noise=true , address=77.90.31.13:4658 , piece_id=253DTNCSSNTDZR2Q56TT34OYAJPELSHIPO4PAOSNZ2KDQ3SUZXQA, seg_pos={0 0}, upload=474110]: Done uploading piece. err: <nil> Dec 28 04:05:31.410821180 (540h36m31.281904225s) [noise=true , address=145.79.227.11:4744 , piece_id=JTTI237QP4VHMZFDIX6MOUGYPFODL5GGUW3IDKZXQLVXCRHGN3YQ, seg_pos={0 0}, upload=474110]: Done uploading piece. err: <nil> Dec 28 04:05:31.494949853 (540h36m31.366032898s) [noise=true , address=145.79.227.40:4069 , piece_id=7YQ5FZTREX3LU3OYUHTXEMUISHIWPLXB74JUHF6O7CFCQBWN4QGQ, seg_pos={0 0}, upload=474110]: Done uploading piece. err: <nil> Dec 28 04:05:31.502794130 (540h36m31.373877175s) [noise=true , address=145.79.227.44:4221 , piece_id=JG45XU4WRCR6H343JZXFJ6JWNOJVNKBDREIHI6GXEENEAPI3LXUA, seg_pos={0 0}, upload=474110]: Done uploading piece. err: <nil> Dec 28 04:05:31.508696539 (540h36m31.379779574s) [noise=true , address=193.149.164.143:4695 , piece_id=HD5JUFDHKF5G4HA3CRIVTSAUH4MXW7HITIZLBD2J7FGGFMPQXOOQ, seg_pos={0 0}, upload=474110]: Done uploading piece. err: <nil> Dec 28 04:05:31.508716888 (540h36m31.379799923s) [seg_pos={0 0}, upload=474110]: Segment reached optimal threshold of 14 pieces. Dec 28 04:05:31.508997072 (540h36m31.380080117s) [noise=true , address=64.49.13.169:4709 , piece_id=A7RHDJ7GK5OZGUVBJW7X7FSYMVY6OIY3SHW66NCRU6SSISGCKYDA, seg_pos={0 0}, upload=474110]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:49772->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:49772->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49772->64.49.13.169:4709: use of closed network connection Dec 28 04:05:31.509004666 (540h36m31.380087701s) [noise=true , address=64.49.13.169:4709 , piece_id=A7RHDJ7GK5OZGUVBJW7X7FSYMVY6OIY3SHW66NCRU6SSISGCKYDA, seg_pos={0 0}, upload=474110]: Long tail cancellation due to optimal threshold Dec 28 04:05:31.509120086 (540h36m31.380203141s) [noise=true , address=45.91.116.12:4249 , piece_id=JM5XOZSORND2I3RGG2NVHN26CQDOK3VKNLJS6K4E2CKR62467RHQ, seg_pos={0 0}, upload=474110]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ghkFFJVqKzG9GbJjYVtcadUCr64bCY8dkEjbYGSShMG9e7WvA): protocol: read tcp 188.166.225.170:53796->45.91.116.12:4249: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 04:05:31.509136768 (540h36m31.380219823s) [noise=true , address=45.91.116.12:4249 , piece_id=JM5XOZSORND2I3RGG2NVHN26CQDOK3VKNLJS6K4E2CKR62467RHQ, seg_pos={0 0}, upload=474110]: Long tail cancellation due to optimal threshold Dec 28 04:05:31.509377987 (540h36m31.380461042s) [noise=true , address=145.79.227.34:4909 , piece_id=NDEZQCRKSSPYWQOQIDY7QMSKAUIL3O52MGDUQ3KVHTP35R75KWDQ, seg_pos={0 0}, upload=474110]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14Pqp5cbhmd44QnikAuD5TgsKFDR3gB2qtLuVSroaWVv4U2cDh): protocol: expected piece hash; context canceled; EOF Dec 28 04:05:31.509388978 (540h36m31.380472013s) [noise=true , address=145.79.227.34:4909 , piece_id=NDEZQCRKSSPYWQOQIDY7QMSKAUIL3O52MGDUQ3KVHTP35R75KWDQ, seg_pos={0 0}, upload=474110]: Long tail cancellation due to optimal threshold Dec 28 04:05:31.509451777 (540h36m31.380534832s) [noise=true , address=77.90.31.18:4819 , piece_id=NXICQKVF2XGMA7RTEL4DOWZ4Q5EKY2RIWL24LMJY47TS32L73F7Q, seg_pos={0 0}, upload=474110]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12nKVfrEuBKwyvtfYLJyfJ1Fo5gVVJU2Z6DNNvDEK8sKTmCW1Da): protocol: expected piece hash; context canceled; EOF Dec 28 04:05:31.509462578 (540h36m31.380545633s) [noise=true , address=77.90.31.18:4819 , piece_id=NXICQKVF2XGMA7RTEL4DOWZ4Q5EKY2RIWL24LMJY47TS32L73F7Q, seg_pos={0 0}, upload=474110]: Long tail cancellation due to optimal threshold Dec 28 04:05:31.509494629 (540h36m31.380577704s) [seg_pos={0 0}, upload=474110]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 04:05:31.509591002 (540h36m31.380674058s) [upload=474110]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:05:31.509597485 (540h36m31.38068053s) [upload=474110]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:05:31.807491599 (540h36m31.678574634s) [upload=474110]: Done uploading segments... Dec 28 04:05:31.807512519 (540h36m31.678595564s) [upload=474110]: Upload finished. err: <nil> Duration: 34.691318779s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 12.644016347s trace: https://jaeger.dev.storj.io/trace/3b8130f32e60318d FAIL (2025-12-28 04:08:36 UTC) orbiter warning: context deadline exceeded Duration: 12.644041305s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 52 out of 60 is failed (threshold=3) [XXXXXXXXX.XXXXX.XXXXXXXXXXXXXXXXX..XXXX.XXXX.XXXXX.XXXXXXX.X] duration: 30.007815479s trace: https://jaeger.dev.storj.io/trace/3a67b8708381e6f1 FAIL (2025-12-28 04:08:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: F46PT6K7APBMAX6TCWZUXKQPOUQEC3Q3J6X7OZCZCKPM6EVZJHNQ) context deadline exceeded; piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: MYFUEW5ITYHWW7I545K4FGAPUFS4FJZN5KU534DKMXS6TOXTGHPA) context deadline exceeded; piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: EXAWYBBDJFVTYG7C7E57JWE6OIVW62PCBIS7VPVOHHH7XPU33PAA) context deadline exceeded ======LOGS====== Dec 28 04:08:05.078159342 (540h39m4.949242387s) [upload=474146]: Starting upload Dec 28 04:08:05.078246888 (540h39m4.949329933s) [upload=474146]: Done starting upload Dec 28 04:08:05.078535277 (540h39m4.949618322s) [upload=474146]: Uploading segments... Dec 28 04:08:05.079065146 (540h39m4.950148202s) [upload=474146]: Got next segment. Inline: false Dec 28 04:08:05.079119980 (540h39m4.950203025s) [upload=474146]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 04:08:05.079127304 (540h39m4.950210349s) [upload=474146]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 04:08:05.330137065 (540h39m5.20122012s) [seg_pos={0 0}, upload=474146]: Begin upload segment... Dec 28 04:08:05.330194525 (540h39m5.20127756s) [seg_pos={0 0}, upload=474146]: Done begin upload segment. Dec 28 04:08:05.330226125 (540h39m5.20130917s) [noise=true , address=145.79.227.16:4911 , piece_id=66OMPRCW6X26BAFHBJ2MDYWP45TWOBHC3WYTYB7SN7LIK67HPV7A, seg_pos={0 0}, upload=474146]: Uploading piece... Dec 28 04:08:05.330285749 (540h39m5.201368784s) [noise=true , address=45.91.116.12:4226 , piece_id=F46PT6K7APBMAX6TCWZUXKQPOUQEC3Q3J6X7OZCZCKPM6EVZJHNQ, seg_pos={0 0}, upload=474146]: Uploading piece... Dec 28 04:08:05.330923092 (540h39m5.202006147s) [noise=true , address=145.79.227.37:4565 , piece_id=S7QJLIN53FJRIKA4O52D4AKMQUH6USA7JFEATI2PXDMSS7UI6O5A, seg_pos={0 0}, upload=474146]: Uploading piece... Dec 28 04:08:05.330936788 (540h39m5.202019833s) [noise=true , address=145.79.227.45:4231 , piece_id=TJTUAOHPPCTPPLHYLL5TM4YS3EIEXHI6DIWYRBNXKXOATSAZCDVA, seg_pos={0 0}, upload=474146]: Uploading piece... Dec 28 04:08:05.331251978 (540h39m5.202335023s) [noise=true , address=45.91.116.8:4047 , piece_id=BIGP7OU5BOKEWB26CJ5SUADM74FPSCPVTN25L42MN4HYPH7245TA, seg_pos={0 0}, upload=474146]: Uploading piece... Dec 28 04:08:05.331357038 (540h39m5.202440073s) [noise=true , address=45.91.116.9:4262 , piece_id=CHMXQ5TEOVEHUQJTAQZYYMQQ6RSC5HHOCABAPJE6DUIRMZN4W6OQ, seg_pos={0 0}, upload=474146]: Uploading piece... Dec 28 04:08:05.331535037 (540h39m5.202618072s) [noise=true , address=145.79.227.42:4122 , piece_id=4GLDIE5N4PWR7XKSIHDDX6V2KCUL5QUCDKQ2C26R62KJKENPGC5Q, seg_pos={0 0}, upload=474146]: Uploading piece... Dec 28 04:08:05.331754085 (540h39m5.20283713s) [noise=true , address=145.79.227.11:4729 , piece_id=JFAPZN5QX6LZEEOOVKAH4IZERSUBIYNE6GVE3XCAL62TLNZJRJJQ, seg_pos={0 0}, upload=474146]: Uploading piece... Dec 28 04:08:05.331930330 (540h39m5.203013365s) [noise=true , address=77.90.31.15:4697 , piece_id=GATOOK2B6KYAW6ZHIVH3OYNYPBNPTKTZ2QDHPINSA5O5TF7Q3D2Q, seg_pos={0 0}, upload=474146]: Uploading piece... Dec 28 04:08:05.332250600 (540h39m5.203333645s) [noise=true , address=145.79.227.17:4972 , piece_id=3ED5FPG4V4ERH744JF2J5RPBKZZUSRQAP53QZMJPEPLAZVJQ5LXQ, seg_pos={0 0}, upload=474146]: Uploading piece... Dec 28 04:08:05.332275528 (540h39m5.203358563s) [noise=true , address=145.79.227.36:4517 , piece_id=DN2VC2K6WS77TKJB2M4SUIZRYLAJ5AP6DCQH5WEOWPLTLZLR7V4Q, seg_pos={0 0}, upload=474146]: Uploading piece... Dec 28 04:08:05.332646884 (540h39m5.203729929s) [noise=true , address=77.90.31.13:4627 , piece_id=EXAWYBBDJFVTYG7C7E57JWE6OIVW62PCBIS7VPVOHHH7XPU33PAA, seg_pos={0 0}, upload=474146]: Uploading piece... Dec 28 04:08:05.332761734 (540h39m5.203844779s) [noise=true , address=145.79.227.47:4266 , piece_id=Q5Y6CY6T34XRR6XJB6D7RO3K4YX3UNCUQKEHD2LK2JJVBNACYHYA, seg_pos={0 0}, upload=474146]: Uploading piece... Dec 28 04:08:05.332980760 (540h39m5.204063795s) [noise=true , address=77.90.31.11:4556 , piece_id=3VTWZTBQIWJ42DIIRT3AU27YMUEUK7HKCXQUI6QK3R3OYM2JMFVA, seg_pos={0 0}, upload=474146]: Uploading piece... Dec 28 04:08:05.333259912 (540h39m5.204342967s) [noise=true , address=77.90.31.16:4736 , piece_id=YECD6XAVBMHRYRENRBZUYJ76LFMCSTGZ7EG5PJL4XGEHPCHMKOUQ, seg_pos={0 0}, upload=474146]: Uploading piece... Dec 28 04:08:05.333585932 (540h39m5.204668977s) [noise=true , address=77.90.31.14:4662 , piece_id=MYFUEW5ITYHWW7I545K4FGAPUFS4FJZN5KU534DKMXS6TOXTGHPA, seg_pos={0 0}, upload=474146]: Uploading piece... Dec 28 04:08:06.962753143 (540h39m6.833836178s) [upload=474146]: Next returned nil segment Dec 28 04:08:06.962770606 (540h39m6.833853651s) [upload=474146]: Waiting for error group managing segments... Dec 28 04:08:07.347995442 (540h39m7.219078488s) [noise=true , address=145.79.227.11:4729 , piece_id=JFAPZN5QX6LZEEOOVKAH4IZERSUBIYNE6GVE3XCAL62TLNZJRJJQ, seg_pos={0 0}, upload=474146]: Done uploading piece. err: <nil> Dec 28 04:08:07.370343427 (540h39m7.241426472s) [noise=true , address=145.79.227.16:4911 , piece_id=66OMPRCW6X26BAFHBJ2MDYWP45TWOBHC3WYTYB7SN7LIK67HPV7A, seg_pos={0 0}, upload=474146]: Done uploading piece. err: <nil> Dec 28 04:08:07.380670391 (540h39m7.251753426s) [noise=true , address=145.79.227.42:4122 , piece_id=4GLDIE5N4PWR7XKSIHDDX6V2KCUL5QUCDKQ2C26R62KJKENPGC5Q, seg_pos={0 0}, upload=474146]: Done uploading piece. err: <nil> Dec 28 04:08:07.380951957 (540h39m7.252034992s) [noise=true , address=145.79.227.45:4231 , piece_id=TJTUAOHPPCTPPLHYLL5TM4YS3EIEXHI6DIWYRBNXKXOATSAZCDVA, seg_pos={0 0}, upload=474146]: Done uploading piece. err: <nil> Dec 28 04:08:07.385138779 (540h39m7.256221824s) [noise=true , address=145.79.227.17:4972 , piece_id=3ED5FPG4V4ERH744JF2J5RPBKZZUSRQAP53QZMJPEPLAZVJQ5LXQ, seg_pos={0 0}, upload=474146]: Done uploading piece. err: <nil> Dec 28 04:08:07.413916786 (540h39m7.284999831s) [noise=true , address=145.79.227.47:4266 , piece_id=Q5Y6CY6T34XRR6XJB6D7RO3K4YX3UNCUQKEHD2LK2JJVBNACYHYA, seg_pos={0 0}, upload=474146]: Done uploading piece. err: <nil> Dec 28 04:08:07.498530735 (540h39m7.36961378s) [noise=true , address=77.90.31.14:4662 , piece_id=MYFUEW5ITYHWW7I545K4FGAPUFS4FJZN5KU534DKMXS6TOXTGHPA, seg_pos={0 0}, upload=474146]: Done uploading piece. err: <nil> Dec 28 04:08:07.638549713 (540h39m7.509632748s) [noise=true , address=77.90.31.13:4627 , piece_id=EXAWYBBDJFVTYG7C7E57JWE6OIVW62PCBIS7VPVOHHH7XPU33PAA, seg_pos={0 0}, upload=474146]: Done uploading piece. err: <nil> Dec 28 04:08:07.657383084 (540h39m7.528466129s) [noise=true , address=45.91.116.9:4262 , piece_id=CHMXQ5TEOVEHUQJTAQZYYMQQ6RSC5HHOCABAPJE6DUIRMZN4W6OQ, seg_pos={0 0}, upload=474146]: Done uploading piece. err: <nil> Dec 28 04:08:07.705357455 (540h39m7.5764405s) [noise=true , address=45.91.116.8:4047 , piece_id=BIGP7OU5BOKEWB26CJ5SUADM74FPSCPVTN25L42MN4HYPH7245TA, seg_pos={0 0}, upload=474146]: Done uploading piece. err: <nil> Dec 28 04:08:08.057181601 (540h39m7.928264646s) [noise=true , address=45.91.116.12:4226 , piece_id=F46PT6K7APBMAX6TCWZUXKQPOUQEC3Q3J6X7OZCZCKPM6EVZJHNQ, seg_pos={0 0}, upload=474146]: Done uploading piece. err: <nil> Dec 28 04:08:08.227383260 (540h39m8.098466335s) [noise=true , address=77.90.31.11:4556 , piece_id=3VTWZTBQIWJ42DIIRT3AU27YMUEUK7HKCXQUI6QK3R3OYM2JMFVA, seg_pos={0 0}, upload=474146]: Done uploading piece. err: <nil> Dec 28 04:08:08.270297585 (540h39m8.14138063s) [noise=true , address=145.79.227.36:4517 , piece_id=DN2VC2K6WS77TKJB2M4SUIZRYLAJ5AP6DCQH5WEOWPLTLZLR7V4Q, seg_pos={0 0}, upload=474146]: Done uploading piece. err: <nil> Dec 28 04:08:08.270324537 (540h39m8.141407572s) [seg_pos={0 0}, upload=474146]: Segment reached optimal threshold of 13 pieces. Dec 28 04:08:08.270534056 (540h39m8.141617111s) [noise=true , address=145.79.227.37:4565 , piece_id=S7QJLIN53FJRIKA4O52D4AKMQUH6USA7JFEATI2PXDMSS7UI6O5A, seg_pos={0 0}, upload=474146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nvvNczHtQrSEJuJbErnjiAWa4BRDzBrQmBNbB7RrrqN1JWNHT): protocol: expected piece hash; context canceled; EOF Dec 28 04:08:08.270543544 (540h39m8.141626579s) [noise=true , address=145.79.227.37:4565 , piece_id=S7QJLIN53FJRIKA4O52D4AKMQUH6USA7JFEATI2PXDMSS7UI6O5A, seg_pos={0 0}, upload=474146]: Long tail cancellation due to optimal threshold Dec 28 04:08:08.270949797 (540h39m8.142032832s) [noise=true , address=77.90.31.15:4697 , piece_id=GATOOK2B6KYAW6ZHIVH3OYNYPBNPTKTZ2QDHPINSA5O5TF7Q3D2Q, seg_pos={0 0}, upload=474146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Dec 28 04:08:08.270980074 (540h39m8.142063109s) [noise=true , address=77.90.31.15:4697 , piece_id=GATOOK2B6KYAW6ZHIVH3OYNYPBNPTKTZ2QDHPINSA5O5TF7Q3D2Q, seg_pos={0 0}, upload=474146]: Long tail cancellation due to optimal threshold Dec 28 04:08:08.271256240 (540h39m8.142339275s) [noise=true , address=77.90.31.16:4736 , piece_id=YECD6XAVBMHRYRENRBZUYJ76LFMCSTGZ7EG5PJL4XGEHPCHMKOUQ, seg_pos={0 0}, upload=474146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 28 04:08:08.271263645 (540h39m8.14234669s) [noise=true , address=77.90.31.16:4736 , piece_id=YECD6XAVBMHRYRENRBZUYJ76LFMCSTGZ7EG5PJL4XGEHPCHMKOUQ, seg_pos={0 0}, upload=474146]: Long tail cancellation due to optimal threshold Dec 28 04:08:08.271287750 (540h39m8.142370795s) [seg_pos={0 0}, upload=474146]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 04:08:08.271409182 (540h39m8.142492237s) [upload=474146]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:08:08.271415644 (540h39m8.142498689s) [upload=474146]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:08:08.557143523 (540h39m8.428226558s) [upload=474146]: Done uploading segments... Dec 28 04:08:08.557163291 (540h39m8.428246336s) [upload=474146]: Upload finished. err: <nil> Duration: 30.00784232s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 21 out of 60 is failed (threshold=6) [.X....X.XXXX....X.X.X...X..X.X.X..X...XX.....X......XX...X.X] duration: 5.001561766s trace: https://jaeger.dev.storj.io/trace/693a9ad598fe2536 FAIL (2025-12-28 04:08:19 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001593096s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 6.392349829s trace: https://jaeger.dev.storj.io/trace/3b8130f32e60318d FAIL (2025-12-28 04:08:02 UTC) orbiter warning: context deadline exceeded Duration: 6.392388242s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.X...X...................................................X..] duration: 31.143213224s trace: https://jaeger.dev.storj.io/trace/591b5c08c3d8a813 FAIL (2025-12-28 04:07:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== A, seg_pos={0 0}, upload=474110]: Uploading piece... Dec 28 04:05:28.680332318 (540h36m28.551415353s) [noise=true , address=145.79.227.43:4182 , piece_id=VZEVVQ4T5SS2PLTVY4FQDGZVB2DRPG2VMDUCP5ECZPIQBKG2RCSA, seg_pos={0 0}, upload=474110]: Uploading piece... Dec 28 04:05:28.680692264 (540h36m28.551775309s) [noise=true , address=64.49.13.169:4709 , piece_id=A7RHDJ7GK5OZGUVBJW7X7FSYMVY6OIY3SHW66NCRU6SSISGCKYDA, seg_pos={0 0}, upload=474110]: Uploading piece... Dec 28 04:05:28.681012203 (540h36m28.552095238s) [noise=true , address=77.90.31.18:4819 , piece_id=NXICQKVF2XGMA7RTEL4DOWZ4Q5EKY2RIWL24LMJY47TS32L73F7Q, seg_pos={0 0}, upload=474110]: Uploading piece... Dec 28 04:05:28.681332643 (540h36m28.552415678s) [noise=true , address=64.49.13.140:4507 , piece_id=PT6VHNLKRVNYJKIEYXRSA57LIQJMA3TKG3FCKJI5DDSDOXZVZCLQ, seg_pos={0 0}, upload=474110]: Uploading piece... Dec 28 04:05:28.681662891 (540h36m28.552745936s) [noise=true , address=77.90.31.12:4597 , piece_id=V4ZPTTPBTO6OEZM6Z66P4WIQ3OEOL2EOV2C6TWDS6Q27MOC6M3CQ, seg_pos={0 0}, upload=474110]: Uploading piece... Dec 28 04:05:28.682067222 (540h36m28.553150257s) [noise=true , address=145.79.227.40:4069 , piece_id=7YQ5FZTREX3LU3OYUHTXEMUISHIWPLXB74JUHF6O7CFCQBWN4QGQ, seg_pos={0 0}, upload=474110]: Uploading piece... Dec 28 04:05:28.682393643 (540h36m28.553476678s) [noise=true , address=145.79.227.11:4744 , piece_id=JTTI237QP4VHMZFDIX6MOUGYPFODL5GGUW3IDKZXQLVXCRHGN3YQ, seg_pos={0 0}, upload=474110]: Uploading piece... Dec 28 04:05:28.682756655 (540h36m28.55383969s) [noise=true , address=77.90.31.17:4779 , piece_id=GMVQKCFKGZU7AGQ7AT3ULHJCJM2B2VC2Z572ET3LZQJBD7X4M6NQ, seg_pos={0 0}, upload=474110]: Uploading piece... Dec 28 04:05:28.683195209 (540h36m28.554278254s) [noise=true , address=145.79.227.34:4909 , piece_id=NDEZQCRKSSPYWQOQIDY7QMSKAUIL3O52MGDUQ3KVHTP35R75KWDQ, seg_pos={0 0}, upload=474110]: Uploading piece... Dec 28 04:05:28.683539605 (540h36m28.55462265s) [noise=true , address=77.90.31.13:4658 , piece_id=253DTNCSSNTDZR2Q56TT34OYAJPELSHIPO4PAOSNZ2KDQ3SUZXQA, seg_pos={0 0}, upload=474110]: Uploading piece... Dec 28 04:05:28.689846314 (540h36m28.560929359s) [noise=true , address=77.90.31.11:4577 , piece_id=NMWLIKEFIMQUHQVA4342PSF3RQ5WAYSNHJPZJ5VU5PEJIISQMEGA, seg_pos={0 0}, upload=474110]: Uploading piece... Dec 28 04:05:28.692671473 (540h36m28.563754528s) [noise=true , address=193.149.164.143:4695 , piece_id=HD5JUFDHKF5G4HA3CRIVTSAUH4MXW7HITIZLBD2J7FGGFMPQXOOQ, seg_pos={0 0}, upload=474110]: Uploading piece... Dec 28 04:05:28.695438943 (540h36m28.566521989s) [noise=true , address=45.91.116.11:4139 , piece_id=35R4AZATBN22P2KLD6VNWHJD5TAS4HA5GSV7OU5CCBD4YUYUWJTA, seg_pos={0 0}, upload=474110]: Uploading piece... Dec 28 04:05:28.698799452 (540h36m28.569882487s) [noise=true , address=45.91.116.12:4249 , piece_id=JM5XOZSORND2I3RGG2NVHN26CQDOK3VKNLJS6K4E2CKR62467RHQ, seg_pos={0 0}, upload=474110]: Uploading piece... Dec 28 04:05:28.706563095 (540h36m28.57764613s) [noise=true , address=145.79.227.44:4221 , piece_id=JG45XU4WRCR6H343JZXFJ6JWNOJVNKBDREIHI6GXEENEAPI3LXUA, seg_pos={0 0}, upload=474110]: Uploading piece... Dec 28 04:05:30.092893206 (540h36m29.963976251s) [upload=474110]: Next returned nil segment Dec 28 04:05:30.092906743 (540h36m29.963989778s) [upload=474110]: Waiting for error group managing segments... Dec 28 04:05:30.603591324 (540h36m30.474674369s) [noise=true , address=145.79.227.43:4182 , piece_id=VZEVVQ4T5SS2PLTVY4FQDGZVB2DRPG2VMDUCP5ECZPIQBKG2RCSA, seg_pos={0 0}, upload=474110]: Done uploading piece. err: <nil> Dec 28 04:05:30.605517772 (540h36m30.476600807s) [noise=true , address=145.79.227.51:4432 , piece_id=YN66BQJBHIRZW2CCPL55RLISG5LCM53MUUVNJNN3L5UYE2CHJPDQ, seg_pos={0 0}, upload=474110]: Done uploading piece. err: <nil> Dec 28 04:05:30.649134745 (540h36m30.52021778s) [noise=true , address=64.49.13.140:4507 , piece_id=PT6VHNLKRVNYJKIEYXRSA57LIQJMA3TKG3FCKJI5DDSDOXZVZCLQ, seg_pos={0 0}, upload=474110]: Done uploading piece. err: <nil> Dec 28 04:05:30.721256115 (540h36m30.59233915s) [noise=true , address=77.90.31.21:4891 , piece_id=VW4VGEYAY5JNK6F3P6G4B4JZ2AW4O5CWRYXFBAKJ2HXQJ7UW6IJA, seg_pos={0 0}, upload=474110]: Done uploading piece. err: <nil> Dec 28 04:05:30.751313910 (540h36m30.622396955s) [noise=true , address=77.90.31.11:4577 , piece_id=NMWLIKEFIMQUHQVA4342PSF3RQ5WAYSNHJPZJ5VU5PEJIISQMEGA, seg_pos={0 0}, upload=474110]: Done uploading piece. err: <nil> Dec 28 04:05:30.800719756 (540h36m30.671802831s) [noise=true , address=45.91.116.11:4139 , piece_id=35R4AZATBN22P2KLD6VNWHJD5TAS4HA5GSV7OU5CCBD4YUYUWJTA, seg_pos={0 0}, upload=474110]: Done uploading piece. err: <nil> Dec 28 04:05:30.823740571 (540h36m30.694823606s) [noise=true , address=77.90.31.12:4597 , piece_id=V4ZPTTPBTO6OEZM6Z66P4WIQ3OEOL2EOV2C6TWDS6Q27MOC6M3CQ, seg_pos={0 0}, upload=474110]: Done uploading piece. err: <nil> Dec 28 04:05:30.847552863 (540h36m30.718635898s) [noise=true , address=145.79.227.12:4795 , piece_id=AQNYUYTFXYD6BXPHQ7UXWNY732L56PJ4KOZT5XZGLZYFZPND2QYA, seg_pos={0 0}, upload=474110]: Done uploading piece. err: <nil> Dec 28 04:05:30.850526996 (540h36m30.721610031s) [noise=true , address=77.90.31.17:4779 , piece_id=GMVQKCFKGZU7AGQ7AT3ULHJCJM2B2VC2Z572ET3LZQJBD7X4M6NQ, seg_pos={0 0}, upload=474110]: Done uploading piece. err: <nil> Dec 28 04:05:30.896096206 (540h36m30.767179251s) [noise=true , address=77.90.31.13:4658 , piece_id=253DTNCSSNTDZR2Q56TT34OYAJPELSHIPO4PAOSNZ2KDQ3SUZXQA, seg_pos={0 0}, upload=474110]: Done uploading piece. err: <nil> Dec 28 04:05:31.410821180 (540h36m31.281904225s) [noise=true , address=145.79.227.11:4744 , piece_id=JTTI237QP4VHMZFDIX6MOUGYPFODL5GGUW3IDKZXQLVXCRHGN3YQ, seg_pos={0 0}, upload=474110]: Done uploading piece. err: <nil> Dec 28 04:05:31.494949853 (540h36m31.366032898s) [noise=true , address=145.79.227.40:4069 , piece_id=7YQ5FZTREX3LU3OYUHTXEMUISHIWPLXB74JUHF6O7CFCQBWN4QGQ, seg_pos={0 0}, upload=474110]: Done uploading piece. err: <nil> Dec 28 04:05:31.502794130 (540h36m31.373877175s) [noise=true , address=145.79.227.44:4221 , piece_id=JG45XU4WRCR6H343JZXFJ6JWNOJVNKBDREIHI6GXEENEAPI3LXUA, seg_pos={0 0}, upload=474110]: Done uploading piece. err: <nil> Dec 28 04:05:31.508696539 (540h36m31.379779574s) [noise=true , address=193.149.164.143:4695 , piece_id=HD5JUFDHKF5G4HA3CRIVTSAUH4MXW7HITIZLBD2J7FGGFMPQXOOQ, seg_pos={0 0}, upload=474110]: Done uploading piece. err: <nil> Dec 28 04:05:31.508716888 (540h36m31.379799923s) [seg_pos={0 0}, upload=474110]: Segment reached optimal threshold of 14 pieces. Dec 28 04:05:31.508997072 (540h36m31.380080117s) [noise=true , address=64.49.13.169:4709 , piece_id=A7RHDJ7GK5OZGUVBJW7X7FSYMVY6OIY3SHW66NCRU6SSISGCKYDA, seg_pos={0 0}, upload=474110]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:49772->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:49772->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49772->64.49.13.169:4709: use of closed network connection Dec 28 04:05:31.509004666 (540h36m31.380087701s) [noise=true , address=64.49.13.169:4709 , piece_id=A7RHDJ7GK5OZGUVBJW7X7FSYMVY6OIY3SHW66NCRU6SSISGCKYDA, seg_pos={0 0}, upload=474110]: Long tail cancellation due to optimal threshold Dec 28 04:05:31.509120086 (540h36m31.380203141s) [noise=true , address=45.91.116.12:4249 , piece_id=JM5XOZSORND2I3RGG2NVHN26CQDOK3VKNLJS6K4E2CKR62467RHQ, seg_pos={0 0}, upload=474110]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ghkFFJVqKzG9GbJjYVtcadUCr64bCY8dkEjbYGSShMG9e7WvA): protocol: read tcp 188.166.225.170:53796->45.91.116.12:4249: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 04:05:31.509136768 (540h36m31.380219823s) [noise=true , address=45.91.116.12:4249 , piece_id=JM5XOZSORND2I3RGG2NVHN26CQDOK3VKNLJS6K4E2CKR62467RHQ, seg_pos={0 0}, upload=474110]: Long tail cancellation due to optimal threshold Dec 28 04:05:31.509377987 (540h36m31.380461042s) [noise=true , address=145.79.227.34:4909 , piece_id=NDEZQCRKSSPYWQOQIDY7QMSKAUIL3O52MGDUQ3KVHTP35R75KWDQ, seg_pos={0 0}, upload=474110]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14Pqp5cbhmd44QnikAuD5TgsKFDR3gB2qtLuVSroaWVv4U2cDh): protocol: expected piece hash; context canceled; EOF Dec 28 04:05:31.509388978 (540h36m31.380472013s) [noise=true , address=145.79.227.34:4909 , piece_id=NDEZQCRKSSPYWQOQIDY7QMSKAUIL3O52MGDUQ3KVHTP35R75KWDQ, seg_pos={0 0}, upload=474110]: Long tail cancellation due to optimal threshold Dec 28 04:05:31.509451777 (540h36m31.380534832s) [noise=true , address=77.90.31.18:4819 , piece_id=NXICQKVF2XGMA7RTEL4DOWZ4Q5EKY2RIWL24LMJY47TS32L73F7Q, seg_pos={0 0}, upload=474110]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12nKVfrEuBKwyvtfYLJyfJ1Fo5gVVJU2Z6DNNvDEK8sKTmCW1Da): protocol: expected piece hash; context canceled; EOF Dec 28 04:05:31.509462578 (540h36m31.380545633s) [noise=true , address=77.90.31.18:4819 , piece_id=NXICQKVF2XGMA7RTEL4DOWZ4Q5EKY2RIWL24LMJY47TS32L73F7Q, seg_pos={0 0}, upload=474110]: Long tail cancellation due to optimal threshold Dec 28 04:05:31.509494629 (540h36m31.380577704s) [seg_pos={0 0}, upload=474110]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 04:05:31.509591002 (540h36m31.380674058s) [upload=474110]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:05:31.509597485 (540h36m31.38068053s) [upload=474110]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:05:31.807491599 (540h36m31.678574634s) [upload=474110]: Done uploading segments... Dec 28 04:05:31.807512519 (540h36m31.678595564s) [upload=474110]: Upload finished. err: <nil> Duration: 31.143245757s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 52 out of 60 is failed (threshold=3) [XXXXXXXXXX.XXXXX.XXXXXXXXXXXXXXXXX..XXXX.XXXX.XXXXX.XXXXXXX.] duration: 7.953174023s trace: https://jaeger.dev.storj.io/trace/1fef9dc7e0dbbcc6 FAIL (2025-12-28 04:07:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: Z5CBXJXKPUMSVUMVQXV5EMBTH25NDD5M6LV7OBUJQVLIVBTV7Q7A) context deadline exceeded; piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: ODJ4HSCE6H3U3M6DTL6FEDYUZALGC4UAJUXZL6RPOY57S5L6BKMA) context deadline exceeded ======LOGS====== Dec 28 04:06:31.407007690 (540h37m31.278090735s) [upload=474126]: Starting upload Dec 28 04:06:31.407122488 (540h37m31.278205543s) [upload=474126]: Done starting upload Dec 28 04:06:31.407196689 (540h37m31.278279734s) [upload=474126]: Uploading segments... Dec 28 04:06:31.407428140 (540h37m31.278511185s) [upload=474126]: Got next segment. Inline: false Dec 28 04:06:31.407475069 (540h37m31.278558114s) [upload=474126]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 04:06:31.407480149 (540h37m31.278563194s) [upload=474126]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 04:06:31.652434850 (540h37m31.523517895s) [seg_pos={0 0}, upload=474126]: Begin upload segment... Dec 28 04:06:31.652483021 (540h37m31.523566066s) [seg_pos={0 0}, upload=474126]: Done begin upload segment. Dec 28 04:06:31.652520853 (540h37m31.523603928s) [noise=true , address=45.91.116.5:4299 , piece_id=KKFSQUEMBEEDHTEPNCPO7QAQSNORMGLP542XV3EIXNDZ7OVIECIA, seg_pos={0 0}, upload=474126]: Uploading piece... Dec 28 04:06:31.652544569 (540h37m31.523627604s) [noise=true , address=45.91.116.6:4976 , piece_id=PY3NMUSGLJTJEZ7NNXNBYCNL4FV4PSCX3GG7AG5UGO3CP3BEB7WQ, seg_pos={0 0}, upload=474126]: Uploading piece... Dec 28 04:06:31.653207681 (540h37m31.524290726s) [noise=true , address=45.91.116.7:4009 , piece_id=Z5CBXJXKPUMSVUMVQXV5EMBTH25NDD5M6LV7OBUJQVLIVBTV7Q7A, seg_pos={0 0}, upload=474126]: Uploading piece... Dec 28 04:06:31.653535565 (540h37m31.5246186s) [noise=true , address=145.79.227.48:4303 , piece_id=5QJ7U3ZY6QNK77WZZ6OUNS6YIVASOY6HHYDZQW4A5DVX7Z64OIUQ, seg_pos={0 0}, upload=474126]: Uploading piece... Dec 28 04:06:31.654000129 (540h37m31.525083165s) [noise=true , address=45.91.116.13:4153 , piece_id=74DC37C7RELWSGRMJ4K2CJVLHLVMHKFMTES6O5AJ7EISJIRAQEAQ, seg_pos={0 0}, upload=474126]: Uploading piece... Dec 28 04:06:31.654466929 (540h37m31.525549964s) [noise=true , address=145.79.227.52:4446 , piece_id=CEO4MQMTL47C5WO6UJOAWRCIV2BTAEEBJJVXGRZ6EJGYUWXWQNZQ, seg_pos={0 0}, upload=474126]: Uploading piece... Dec 28 04:06:31.655045771 (540h37m31.526128816s) [noise=true , address=145.79.227.16:4909 , piece_id=NHAYLGPBIHXSTDZDHV3FTLVNNHDHFG4AM5C6353YAKBTQ5Z5IOFQ, seg_pos={0 0}, upload=474126]: Uploading piece... Dec 28 04:06:31.655500477 (540h37m31.526583512s) [noise=true , address=145.79.227.15:4873 , piece_id=YCZIVC7NNWK6IEYXEQOTD7P5IGJ4KKC5OFY4C5G2E657BG5SKMEA, seg_pos={0 0}, upload=474126]: Uploading piece... Dec 28 04:06:31.655519403 (540h37m31.526602448s) [noise=true , address=145.79.227.42:4123 , piece_id=4XTLTS6K3CA2QTGNXHH4PNC7B5T7I5VMTXXNFBKYOOKAD5RAULHA, seg_pos={0 0}, upload=474126]: Uploading piece... Dec 28 04:06:31.655926679 (540h37m31.527009714s) [noise=true , address=77.90.31.11:4556 , piece_id=ODJ4HSCE6H3U3M6DTL6FEDYUZALGC4UAJUXZL6RPOY57S5L6BKMA, seg_pos={0 0}, upload=474126]: Uploading piece... Dec 28 04:06:31.655991312 (540h37m31.527074347s) [noise=true , address=145.79.227.29:4942 , piece_id=VSPVJJSIKGH5LRJWUZ2MIBKCNBDT6FJHVYEKUYGOAC52SRAAAYJQ, seg_pos={0 0}, upload=474126]: Uploading piece... Dec 28 04:06:31.656340857 (540h37m31.527423892s) [noise=true , address=145.79.227.5:4223 , piece_id=HQ5NRY5UMML4HJD66MLC52QFMLQU5DX65ORCUA2UYVEPZSQNML7Q, seg_pos={0 0}, upload=474126]: Uploading piece... Dec 28 04:06:31.656507895 (540h37m31.52759093s) [noise=true , address=145.79.227.12:4767 , piece_id=O2KD6NAI7VJUXELWEARALUCJDB3GUR5WLS6ZG33MSC2PKHYFDYCA, seg_pos={0 0}, upload=474126]: Uploading piece... Dec 28 04:06:31.656752901 (540h37m31.527835936s) [noise=true , address=145.79.227.7:4293 , piece_id=HO6BKNDFLXYLUAQXFM6HT2AYYXHX5R3ZNXG5PMK5IK2ZBJ2DVIHQ, seg_pos={0 0}, upload=474126]: Uploading piece... Dec 28 04:06:31.656875405 (540h37m31.52795845s) [noise=true , address=77.90.31.19:4841 , piece_id=GEWVRLICJH6TJZVOCTEHTNB7IOFLHV6OBSV6SELQ6NA6KYB47XMA, seg_pos={0 0}, upload=474126]: Uploading piece... Dec 28 04:06:31.657293661 (540h37m31.528376706s) [noise=true , address=77.90.31.21:4878 , piece_id=KYOBWODUBC65WHH2R36AENQ4ZHQJV7TPXAHI7DJVU44FEZPHKSPQ, seg_pos={0 0}, upload=474126]: Uploading piece... Dec 28 04:06:33.397997043 (540h37m33.269080088s) [upload=474126]: Next returned nil segment Dec 28 04:06:33.398014397 (540h37m33.269097432s) [upload=474126]: Waiting for error group managing segments... Dec 28 04:06:33.672298956 (540h37m33.543381991s) [noise=true , address=145.79.227.15:4873 , piece_id=YCZIVC7NNWK6IEYXEQOTD7P5IGJ4KKC5OFY4C5G2E657BG5SKMEA, seg_pos={0 0}, upload=474126]: Done uploading piece. err: <nil> Dec 28 04:06:33.690351560 (540h37m33.561434595s) [noise=true , address=145.79.227.16:4909 , piece_id=NHAYLGPBIHXSTDZDHV3FTLVNNHDHFG4AM5C6353YAKBTQ5Z5IOFQ, seg_pos={0 0}, upload=474126]: Done uploading piece. err: <nil> Dec 28 04:06:33.691297552 (540h37m33.562380597s) [noise=true , address=145.79.227.12:4767 , piece_id=O2KD6NAI7VJUXELWEARALUCJDB3GUR5WLS6ZG33MSC2PKHYFDYCA, seg_pos={0 0}, upload=474126]: Done uploading piece. err: <nil> Dec 28 04:06:33.692697688 (540h37m33.563780724s) [noise=true , address=145.79.227.52:4446 , piece_id=CEO4MQMTL47C5WO6UJOAWRCIV2BTAEEBJJVXGRZ6EJGYUWXWQNZQ, seg_pos={0 0}, upload=474126]: Done uploading piece. err: <nil> Dec 28 04:06:33.750739539 (540h37m33.621822584s) [noise=true , address=145.79.227.48:4303 , piece_id=5QJ7U3ZY6QNK77WZZ6OUNS6YIVASOY6HHYDZQW4A5DVX7Z64OIUQ, seg_pos={0 0}, upload=474126]: Done uploading piece. err: <nil> Dec 28 04:06:33.858827794 (540h37m33.72991084s) [noise=true , address=77.90.31.11:4556 , piece_id=ODJ4HSCE6H3U3M6DTL6FEDYUZALGC4UAJUXZL6RPOY57S5L6BKMA, seg_pos={0 0}, upload=474126]: Done uploading piece. err: <nil> Dec 28 04:06:33.904737233 (540h37m33.775820278s) [noise=true , address=45.91.116.7:4009 , piece_id=Z5CBXJXKPUMSVUMVQXV5EMBTH25NDD5M6LV7OBUJQVLIVBTV7Q7A, seg_pos={0 0}, upload=474126]: Done uploading piece. err: <nil> Dec 28 04:06:33.910266953 (540h37m33.781349998s) [noise=true , address=45.91.116.13:4153 , piece_id=74DC37C7RELWSGRMJ4K2CJVLHLVMHKFMTES6O5AJ7EISJIRAQEAQ, seg_pos={0 0}, upload=474126]: Done uploading piece. err: <nil> Dec 28 04:06:34.567956733 (540h37m34.439039779s) [noise=true , address=145.79.227.29:4942 , piece_id=VSPVJJSIKGH5LRJWUZ2MIBKCNBDT6FJHVYEKUYGOAC52SRAAAYJQ, seg_pos={0 0}, upload=474126]: Done uploading piece. err: <nil> Dec 28 04:06:34.574366128 (540h37m34.445449173s) [noise=true , address=145.79.227.5:4223 , piece_id=HQ5NRY5UMML4HJD66MLC52QFMLQU5DX65ORCUA2UYVEPZSQNML7Q, seg_pos={0 0}, upload=474126]: Done uploading piece. err: <nil> Dec 28 04:06:34.578206240 (540h37m34.449289285s) [noise=true , address=145.79.227.42:4123 , piece_id=4XTLTS6K3CA2QTGNXHH4PNC7B5T7I5VMTXXNFBKYOOKAD5RAULHA, seg_pos={0 0}, upload=474126]: Done uploading piece. err: <nil> Dec 28 04:06:34.604791702 (540h37m34.475874738s) [noise=true , address=145.79.227.7:4293 , piece_id=HO6BKNDFLXYLUAQXFM6HT2AYYXHX5R3ZNXG5PMK5IK2ZBJ2DVIHQ, seg_pos={0 0}, upload=474126]: Done uploading piece. err: <nil> Dec 28 04:06:34.698911531 (540h37m34.569994566s) [noise=true , address=45.91.116.5:4299 , piece_id=KKFSQUEMBEEDHTEPNCPO7QAQSNORMGLP542XV3EIXNDZ7OVIECIA, seg_pos={0 0}, upload=474126]: Done uploading piece. err: <nil> Dec 28 04:06:34.698930827 (540h37m34.570013862s) [seg_pos={0 0}, upload=474126]: Segment reached optimal threshold of 13 pieces. Dec 28 04:06:34.699107023 (540h37m34.570190058s) [noise=true , address=77.90.31.21:4878 , piece_id=KYOBWODUBC65WHH2R36AENQ4ZHQJV7TPXAHI7DJVU44FEZPHKSPQ, seg_pos={0 0}, upload=474126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 28 04:06:34.699113927 (540h37m34.570196962s) [noise=true , address=77.90.31.21:4878 , piece_id=KYOBWODUBC65WHH2R36AENQ4ZHQJV7TPXAHI7DJVU44FEZPHKSPQ, seg_pos={0 0}, upload=474126]: Long tail cancellation due to optimal threshold Dec 28 04:06:34.699166576 (540h37m34.570249611s) [noise=true , address=77.90.31.19:4841 , piece_id=GEWVRLICJH6TJZVOCTEHTNB7IOFLHV6OBSV6SELQ6NA6KYB47XMA, seg_pos={0 0}, upload=474126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: expected piece hash; context canceled; EOF Dec 28 04:06:34.699174942 (540h37m34.570257987s) [noise=true , address=77.90.31.19:4841 , piece_id=GEWVRLICJH6TJZVOCTEHTNB7IOFLHV6OBSV6SELQ6NA6KYB47XMA, seg_pos={0 0}, upload=474126]: Long tail cancellation due to optimal threshold Dec 28 04:06:34.699307374 (540h37m34.57039042s) [noise=true , address=45.91.116.6:4976 , piece_id=PY3NMUSGLJTJEZ7NNXNBYCNL4FV4PSCX3GG7AG5UGO3CP3BEB7WQ, seg_pos={0 0}, upload=474126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Dec 28 04:06:34.699313907 (540h37m34.570396942s) [noise=true , address=45.91.116.6:4976 , piece_id=PY3NMUSGLJTJEZ7NNXNBYCNL4FV4PSCX3GG7AG5UGO3CP3BEB7WQ, seg_pos={0 0}, upload=474126]: Long tail cancellation due to optimal threshold Dec 28 04:06:34.699332302 (540h37m34.570415337s) [seg_pos={0 0}, upload=474126]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 04:06:34.699400432 (540h37m34.570483467s) [upload=474126]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:06:34.699405291 (540h37m34.570488326s) [upload=474126]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:06:34.982927360 (540h37m34.854010406s) [upload=474126]: Done uploading segments... Dec 28 04:06:34.982946386 (540h37m34.854029421s) [upload=474126]: Upload finished. err: <nil> Duration: 7.953189954s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..X...X...................................................X.] duration: 45.004724927s trace: https://jaeger.dev.storj.io/trace/591b5c08c3d8a813 FAIL (2025-12-28 04:07:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== A, seg_pos={0 0}, upload=474110]: Uploading piece... Dec 28 04:05:28.680332318 (540h36m28.551415353s) [noise=true , address=145.79.227.43:4182 , piece_id=VZEVVQ4T5SS2PLTVY4FQDGZVB2DRPG2VMDUCP5ECZPIQBKG2RCSA, seg_pos={0 0}, upload=474110]: Uploading piece... Dec 28 04:05:28.680692264 (540h36m28.551775309s) [noise=true , address=64.49.13.169:4709 , piece_id=A7RHDJ7GK5OZGUVBJW7X7FSYMVY6OIY3SHW66NCRU6SSISGCKYDA, seg_pos={0 0}, upload=474110]: Uploading piece... Dec 28 04:05:28.681012203 (540h36m28.552095238s) [noise=true , address=77.90.31.18:4819 , piece_id=NXICQKVF2XGMA7RTEL4DOWZ4Q5EKY2RIWL24LMJY47TS32L73F7Q, seg_pos={0 0}, upload=474110]: Uploading piece... Dec 28 04:05:28.681332643 (540h36m28.552415678s) [noise=true , address=64.49.13.140:4507 , piece_id=PT6VHNLKRVNYJKIEYXRSA57LIQJMA3TKG3FCKJI5DDSDOXZVZCLQ, seg_pos={0 0}, upload=474110]: Uploading piece... Dec 28 04:05:28.681662891 (540h36m28.552745936s) [noise=true , address=77.90.31.12:4597 , piece_id=V4ZPTTPBTO6OEZM6Z66P4WIQ3OEOL2EOV2C6TWDS6Q27MOC6M3CQ, seg_pos={0 0}, upload=474110]: Uploading piece... Dec 28 04:05:28.682067222 (540h36m28.553150257s) [noise=true , address=145.79.227.40:4069 , piece_id=7YQ5FZTREX3LU3OYUHTXEMUISHIWPLXB74JUHF6O7CFCQBWN4QGQ, seg_pos={0 0}, upload=474110]: Uploading piece... Dec 28 04:05:28.682393643 (540h36m28.553476678s) [noise=true , address=145.79.227.11:4744 , piece_id=JTTI237QP4VHMZFDIX6MOUGYPFODL5GGUW3IDKZXQLVXCRHGN3YQ, seg_pos={0 0}, upload=474110]: Uploading piece... Dec 28 04:05:28.682756655 (540h36m28.55383969s) [noise=true , address=77.90.31.17:4779 , piece_id=GMVQKCFKGZU7AGQ7AT3ULHJCJM2B2VC2Z572ET3LZQJBD7X4M6NQ, seg_pos={0 0}, upload=474110]: Uploading piece... Dec 28 04:05:28.683195209 (540h36m28.554278254s) [noise=true , address=145.79.227.34:4909 , piece_id=NDEZQCRKSSPYWQOQIDY7QMSKAUIL3O52MGDUQ3KVHTP35R75KWDQ, seg_pos={0 0}, upload=474110]: Uploading piece... Dec 28 04:05:28.683539605 (540h36m28.55462265s) [noise=true , address=77.90.31.13:4658 , piece_id=253DTNCSSNTDZR2Q56TT34OYAJPELSHIPO4PAOSNZ2KDQ3SUZXQA, seg_pos={0 0}, upload=474110]: Uploading piece... Dec 28 04:05:28.689846314 (540h36m28.560929359s) [noise=true , address=77.90.31.11:4577 , piece_id=NMWLIKEFIMQUHQVA4342PSF3RQ5WAYSNHJPZJ5VU5PEJIISQMEGA, seg_pos={0 0}, upload=474110]: Uploading piece... Dec 28 04:05:28.692671473 (540h36m28.563754528s) [noise=true , address=193.149.164.143:4695 , piece_id=HD5JUFDHKF5G4HA3CRIVTSAUH4MXW7HITIZLBD2J7FGGFMPQXOOQ, seg_pos={0 0}, upload=474110]: Uploading piece... Dec 28 04:05:28.695438943 (540h36m28.566521989s) [noise=true , address=45.91.116.11:4139 , piece_id=35R4AZATBN22P2KLD6VNWHJD5TAS4HA5GSV7OU5CCBD4YUYUWJTA, seg_pos={0 0}, upload=474110]: Uploading piece... Dec 28 04:05:28.698799452 (540h36m28.569882487s) [noise=true , address=45.91.116.12:4249 , piece_id=JM5XOZSORND2I3RGG2NVHN26CQDOK3VKNLJS6K4E2CKR62467RHQ, seg_pos={0 0}, upload=474110]: Uploading piece... Dec 28 04:05:28.706563095 (540h36m28.57764613s) [noise=true , address=145.79.227.44:4221 , piece_id=JG45XU4WRCR6H343JZXFJ6JWNOJVNKBDREIHI6GXEENEAPI3LXUA, seg_pos={0 0}, upload=474110]: Uploading piece... Dec 28 04:05:30.092893206 (540h36m29.963976251s) [upload=474110]: Next returned nil segment Dec 28 04:05:30.092906743 (540h36m29.963989778s) [upload=474110]: Waiting for error group managing segments... Dec 28 04:05:30.603591324 (540h36m30.474674369s) [noise=true , address=145.79.227.43:4182 , piece_id=VZEVVQ4T5SS2PLTVY4FQDGZVB2DRPG2VMDUCP5ECZPIQBKG2RCSA, seg_pos={0 0}, upload=474110]: Done uploading piece. err: <nil> Dec 28 04:05:30.605517772 (540h36m30.476600807s) [noise=true , address=145.79.227.51:4432 , piece_id=YN66BQJBHIRZW2CCPL55RLISG5LCM53MUUVNJNN3L5UYE2CHJPDQ, seg_pos={0 0}, upload=474110]: Done uploading piece. err: <nil> Dec 28 04:05:30.649134745 (540h36m30.52021778s) [noise=true , address=64.49.13.140:4507 , piece_id=PT6VHNLKRVNYJKIEYXRSA57LIQJMA3TKG3FCKJI5DDSDOXZVZCLQ, seg_pos={0 0}, upload=474110]: Done uploading piece. err: <nil> Dec 28 04:05:30.721256115 (540h36m30.59233915s) [noise=true , address=77.90.31.21:4891 , piece_id=VW4VGEYAY5JNK6F3P6G4B4JZ2AW4O5CWRYXFBAKJ2HXQJ7UW6IJA, seg_pos={0 0}, upload=474110]: Done uploading piece. err: <nil> Dec 28 04:05:30.751313910 (540h36m30.622396955s) [noise=true , address=77.90.31.11:4577 , piece_id=NMWLIKEFIMQUHQVA4342PSF3RQ5WAYSNHJPZJ5VU5PEJIISQMEGA, seg_pos={0 0}, upload=474110]: Done uploading piece. err: <nil> Dec 28 04:05:30.800719756 (540h36m30.671802831s) [noise=true , address=45.91.116.11:4139 , piece_id=35R4AZATBN22P2KLD6VNWHJD5TAS4HA5GSV7OU5CCBD4YUYUWJTA, seg_pos={0 0}, upload=474110]: Done uploading piece. err: <nil> Dec 28 04:05:30.823740571 (540h36m30.694823606s) [noise=true , address=77.90.31.12:4597 , piece_id=V4ZPTTPBTO6OEZM6Z66P4WIQ3OEOL2EOV2C6TWDS6Q27MOC6M3CQ, seg_pos={0 0}, upload=474110]: Done uploading piece. err: <nil> Dec 28 04:05:30.847552863 (540h36m30.718635898s) [noise=true , address=145.79.227.12:4795 , piece_id=AQNYUYTFXYD6BXPHQ7UXWNY732L56PJ4KOZT5XZGLZYFZPND2QYA, seg_pos={0 0}, upload=474110]: Done uploading piece. err: <nil> Dec 28 04:05:30.850526996 (540h36m30.721610031s) [noise=true , address=77.90.31.17:4779 , piece_id=GMVQKCFKGZU7AGQ7AT3ULHJCJM2B2VC2Z572ET3LZQJBD7X4M6NQ, seg_pos={0 0}, upload=474110]: Done uploading piece. err: <nil> Dec 28 04:05:30.896096206 (540h36m30.767179251s) [noise=true , address=77.90.31.13:4658 , piece_id=253DTNCSSNTDZR2Q56TT34OYAJPELSHIPO4PAOSNZ2KDQ3SUZXQA, seg_pos={0 0}, upload=474110]: Done uploading piece. err: <nil> Dec 28 04:05:31.410821180 (540h36m31.281904225s) [noise=true , address=145.79.227.11:4744 , piece_id=JTTI237QP4VHMZFDIX6MOUGYPFODL5GGUW3IDKZXQLVXCRHGN3YQ, seg_pos={0 0}, upload=474110]: Done uploading piece. err: <nil> Dec 28 04:05:31.494949853 (540h36m31.366032898s) [noise=true , address=145.79.227.40:4069 , piece_id=7YQ5FZTREX3LU3OYUHTXEMUISHIWPLXB74JUHF6O7CFCQBWN4QGQ, seg_pos={0 0}, upload=474110]: Done uploading piece. err: <nil> Dec 28 04:05:31.502794130 (540h36m31.373877175s) [noise=true , address=145.79.227.44:4221 , piece_id=JG45XU4WRCR6H343JZXFJ6JWNOJVNKBDREIHI6GXEENEAPI3LXUA, seg_pos={0 0}, upload=474110]: Done uploading piece. err: <nil> Dec 28 04:05:31.508696539 (540h36m31.379779574s) [noise=true , address=193.149.164.143:4695 , piece_id=HD5JUFDHKF5G4HA3CRIVTSAUH4MXW7HITIZLBD2J7FGGFMPQXOOQ, seg_pos={0 0}, upload=474110]: Done uploading piece. err: <nil> Dec 28 04:05:31.508716888 (540h36m31.379799923s) [seg_pos={0 0}, upload=474110]: Segment reached optimal threshold of 14 pieces. Dec 28 04:05:31.508997072 (540h36m31.380080117s) [noise=true , address=64.49.13.169:4709 , piece_id=A7RHDJ7GK5OZGUVBJW7X7FSYMVY6OIY3SHW66NCRU6SSISGCKYDA, seg_pos={0 0}, upload=474110]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:49772->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:49772->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49772->64.49.13.169:4709: use of closed network connection Dec 28 04:05:31.509004666 (540h36m31.380087701s) [noise=true , address=64.49.13.169:4709 , piece_id=A7RHDJ7GK5OZGUVBJW7X7FSYMVY6OIY3SHW66NCRU6SSISGCKYDA, seg_pos={0 0}, upload=474110]: Long tail cancellation due to optimal threshold Dec 28 04:05:31.509120086 (540h36m31.380203141s) [noise=true , address=45.91.116.12:4249 , piece_id=JM5XOZSORND2I3RGG2NVHN26CQDOK3VKNLJS6K4E2CKR62467RHQ, seg_pos={0 0}, upload=474110]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ghkFFJVqKzG9GbJjYVtcadUCr64bCY8dkEjbYGSShMG9e7WvA): protocol: read tcp 188.166.225.170:53796->45.91.116.12:4249: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 04:05:31.509136768 (540h36m31.380219823s) [noise=true , address=45.91.116.12:4249 , piece_id=JM5XOZSORND2I3RGG2NVHN26CQDOK3VKNLJS6K4E2CKR62467RHQ, seg_pos={0 0}, upload=474110]: Long tail cancellation due to optimal threshold Dec 28 04:05:31.509377987 (540h36m31.380461042s) [noise=true , address=145.79.227.34:4909 , piece_id=NDEZQCRKSSPYWQOQIDY7QMSKAUIL3O52MGDUQ3KVHTP35R75KWDQ, seg_pos={0 0}, upload=474110]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14Pqp5cbhmd44QnikAuD5TgsKFDR3gB2qtLuVSroaWVv4U2cDh): protocol: expected piece hash; context canceled; EOF Dec 28 04:05:31.509388978 (540h36m31.380472013s) [noise=true , address=145.79.227.34:4909 , piece_id=NDEZQCRKSSPYWQOQIDY7QMSKAUIL3O52MGDUQ3KVHTP35R75KWDQ, seg_pos={0 0}, upload=474110]: Long tail cancellation due to optimal threshold Dec 28 04:05:31.509451777 (540h36m31.380534832s) [noise=true , address=77.90.31.18:4819 , piece_id=NXICQKVF2XGMA7RTEL4DOWZ4Q5EKY2RIWL24LMJY47TS32L73F7Q, seg_pos={0 0}, upload=474110]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12nKVfrEuBKwyvtfYLJyfJ1Fo5gVVJU2Z6DNNvDEK8sKTmCW1Da): protocol: expected piece hash; context canceled; EOF Dec 28 04:05:31.509462578 (540h36m31.380545633s) [noise=true , address=77.90.31.18:4819 , piece_id=NXICQKVF2XGMA7RTEL4DOWZ4Q5EKY2RIWL24LMJY47TS32L73F7Q, seg_pos={0 0}, upload=474110]: Long tail cancellation due to optimal threshold Dec 28 04:05:31.509494629 (540h36m31.380577704s) [seg_pos={0 0}, upload=474110]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 04:05:31.509591002 (540h36m31.380674058s) [upload=474110]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:05:31.509597485 (540h36m31.38068053s) [upload=474110]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:05:31.807491599 (540h36m31.678574634s) [upload=474110]: Done uploading segments... Dec 28 04:05:31.807512519 (540h36m31.678595564s) [upload=474110]: Upload finished. err: <nil> Duration: 45.004750738s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 21 out of 60 is failed (threshold=6) [X.X....X.XXXX....X.X.X...X..X.X.X..X...XX.....X......XX...X.] duration: 5.0005861s trace: https://jaeger.dev.storj.io/trace/67893059cc77910f FAIL (2025-12-28 04:07:07 UTC) orbiter warning: context deadline exceeded Duration: 5.000614053s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 53 out of 60 is failed (threshold=3) [XXXXXXXXXXX.XXXXX.XXXXXXXXXXXXXXXXX..XXXX.XXXX.XXXXX.XXXXXXX] duration: 30.00307817s trace: https://jaeger.dev.storj.io/trace/1fef9dc7e0dbbcc6 FAIL (2025-12-28 04:07:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: Z5CBXJXKPUMSVUMVQXV5EMBTH25NDD5M6LV7OBUJQVLIVBTV7Q7A) context deadline exceeded; piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: ODJ4HSCE6H3U3M6DTL6FEDYUZALGC4UAJUXZL6RPOY57S5L6BKMA) context deadline exceeded ======LOGS====== Dec 28 04:06:31.407007690 (540h37m31.278090735s) [upload=474126]: Starting upload Dec 28 04:06:31.407122488 (540h37m31.278205543s) [upload=474126]: Done starting upload Dec 28 04:06:31.407196689 (540h37m31.278279734s) [upload=474126]: Uploading segments... Dec 28 04:06:31.407428140 (540h37m31.278511185s) [upload=474126]: Got next segment. Inline: false Dec 28 04:06:31.407475069 (540h37m31.278558114s) [upload=474126]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 04:06:31.407480149 (540h37m31.278563194s) [upload=474126]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 04:06:31.652434850 (540h37m31.523517895s) [seg_pos={0 0}, upload=474126]: Begin upload segment... Dec 28 04:06:31.652483021 (540h37m31.523566066s) [seg_pos={0 0}, upload=474126]: Done begin upload segment. Dec 28 04:06:31.652520853 (540h37m31.523603928s) [noise=true , address=45.91.116.5:4299 , piece_id=KKFSQUEMBEEDHTEPNCPO7QAQSNORMGLP542XV3EIXNDZ7OVIECIA, seg_pos={0 0}, upload=474126]: Uploading piece... Dec 28 04:06:31.652544569 (540h37m31.523627604s) [noise=true , address=45.91.116.6:4976 , piece_id=PY3NMUSGLJTJEZ7NNXNBYCNL4FV4PSCX3GG7AG5UGO3CP3BEB7WQ, seg_pos={0 0}, upload=474126]: Uploading piece... Dec 28 04:06:31.653207681 (540h37m31.524290726s) [noise=true , address=45.91.116.7:4009 , piece_id=Z5CBXJXKPUMSVUMVQXV5EMBTH25NDD5M6LV7OBUJQVLIVBTV7Q7A, seg_pos={0 0}, upload=474126]: Uploading piece... Dec 28 04:06:31.653535565 (540h37m31.5246186s) [noise=true , address=145.79.227.48:4303 , piece_id=5QJ7U3ZY6QNK77WZZ6OUNS6YIVASOY6HHYDZQW4A5DVX7Z64OIUQ, seg_pos={0 0}, upload=474126]: Uploading piece... Dec 28 04:06:31.654000129 (540h37m31.525083165s) [noise=true , address=45.91.116.13:4153 , piece_id=74DC37C7RELWSGRMJ4K2CJVLHLVMHKFMTES6O5AJ7EISJIRAQEAQ, seg_pos={0 0}, upload=474126]: Uploading piece... Dec 28 04:06:31.654466929 (540h37m31.525549964s) [noise=true , address=145.79.227.52:4446 , piece_id=CEO4MQMTL47C5WO6UJOAWRCIV2BTAEEBJJVXGRZ6EJGYUWXWQNZQ, seg_pos={0 0}, upload=474126]: Uploading piece... Dec 28 04:06:31.655045771 (540h37m31.526128816s) [noise=true , address=145.79.227.16:4909 , piece_id=NHAYLGPBIHXSTDZDHV3FTLVNNHDHFG4AM5C6353YAKBTQ5Z5IOFQ, seg_pos={0 0}, upload=474126]: Uploading piece... Dec 28 04:06:31.655500477 (540h37m31.526583512s) [noise=true , address=145.79.227.15:4873 , piece_id=YCZIVC7NNWK6IEYXEQOTD7P5IGJ4KKC5OFY4C5G2E657BG5SKMEA, seg_pos={0 0}, upload=474126]: Uploading piece... Dec 28 04:06:31.655519403 (540h37m31.526602448s) [noise=true , address=145.79.227.42:4123 , piece_id=4XTLTS6K3CA2QTGNXHH4PNC7B5T7I5VMTXXNFBKYOOKAD5RAULHA, seg_pos={0 0}, upload=474126]: Uploading piece... Dec 28 04:06:31.655926679 (540h37m31.527009714s) [noise=true , address=77.90.31.11:4556 , piece_id=ODJ4HSCE6H3U3M6DTL6FEDYUZALGC4UAJUXZL6RPOY57S5L6BKMA, seg_pos={0 0}, upload=474126]: Uploading piece... Dec 28 04:06:31.655991312 (540h37m31.527074347s) [noise=true , address=145.79.227.29:4942 , piece_id=VSPVJJSIKGH5LRJWUZ2MIBKCNBDT6FJHVYEKUYGOAC52SRAAAYJQ, seg_pos={0 0}, upload=474126]: Uploading piece... Dec 28 04:06:31.656340857 (540h37m31.527423892s) [noise=true , address=145.79.227.5:4223 , piece_id=HQ5NRY5UMML4HJD66MLC52QFMLQU5DX65ORCUA2UYVEPZSQNML7Q, seg_pos={0 0}, upload=474126]: Uploading piece... Dec 28 04:06:31.656507895 (540h37m31.52759093s) [noise=true , address=145.79.227.12:4767 , piece_id=O2KD6NAI7VJUXELWEARALUCJDB3GUR5WLS6ZG33MSC2PKHYFDYCA, seg_pos={0 0}, upload=474126]: Uploading piece... Dec 28 04:06:31.656752901 (540h37m31.527835936s) [noise=true , address=145.79.227.7:4293 , piece_id=HO6BKNDFLXYLUAQXFM6HT2AYYXHX5R3ZNXG5PMK5IK2ZBJ2DVIHQ, seg_pos={0 0}, upload=474126]: Uploading piece... Dec 28 04:06:31.656875405 (540h37m31.52795845s) [noise=true , address=77.90.31.19:4841 , piece_id=GEWVRLICJH6TJZVOCTEHTNB7IOFLHV6OBSV6SELQ6NA6KYB47XMA, seg_pos={0 0}, upload=474126]: Uploading piece... Dec 28 04:06:31.657293661 (540h37m31.528376706s) [noise=true , address=77.90.31.21:4878 , piece_id=KYOBWODUBC65WHH2R36AENQ4ZHQJV7TPXAHI7DJVU44FEZPHKSPQ, seg_pos={0 0}, upload=474126]: Uploading piece... Dec 28 04:06:33.397997043 (540h37m33.269080088s) [upload=474126]: Next returned nil segment Dec 28 04:06:33.398014397 (540h37m33.269097432s) [upload=474126]: Waiting for error group managing segments... Dec 28 04:06:33.672298956 (540h37m33.543381991s) [noise=true , address=145.79.227.15:4873 , piece_id=YCZIVC7NNWK6IEYXEQOTD7P5IGJ4KKC5OFY4C5G2E657BG5SKMEA, seg_pos={0 0}, upload=474126]: Done uploading piece. err: <nil> Dec 28 04:06:33.690351560 (540h37m33.561434595s) [noise=true , address=145.79.227.16:4909 , piece_id=NHAYLGPBIHXSTDZDHV3FTLVNNHDHFG4AM5C6353YAKBTQ5Z5IOFQ, seg_pos={0 0}, upload=474126]: Done uploading piece. err: <nil> Dec 28 04:06:33.691297552 (540h37m33.562380597s) [noise=true , address=145.79.227.12:4767 , piece_id=O2KD6NAI7VJUXELWEARALUCJDB3GUR5WLS6ZG33MSC2PKHYFDYCA, seg_pos={0 0}, upload=474126]: Done uploading piece. err: <nil> Dec 28 04:06:33.692697688 (540h37m33.563780724s) [noise=true , address=145.79.227.52:4446 , piece_id=CEO4MQMTL47C5WO6UJOAWRCIV2BTAEEBJJVXGRZ6EJGYUWXWQNZQ, seg_pos={0 0}, upload=474126]: Done uploading piece. err: <nil> Dec 28 04:06:33.750739539 (540h37m33.621822584s) [noise=true , address=145.79.227.48:4303 , piece_id=5QJ7U3ZY6QNK77WZZ6OUNS6YIVASOY6HHYDZQW4A5DVX7Z64OIUQ, seg_pos={0 0}, upload=474126]: Done uploading piece. err: <nil> Dec 28 04:06:33.858827794 (540h37m33.72991084s) [noise=true , address=77.90.31.11:4556 , piece_id=ODJ4HSCE6H3U3M6DTL6FEDYUZALGC4UAJUXZL6RPOY57S5L6BKMA, seg_pos={0 0}, upload=474126]: Done uploading piece. err: <nil> Dec 28 04:06:33.904737233 (540h37m33.775820278s) [noise=true , address=45.91.116.7:4009 , piece_id=Z5CBXJXKPUMSVUMVQXV5EMBTH25NDD5M6LV7OBUJQVLIVBTV7Q7A, seg_pos={0 0}, upload=474126]: Done uploading piece. err: <nil> Dec 28 04:06:33.910266953 (540h37m33.781349998s) [noise=true , address=45.91.116.13:4153 , piece_id=74DC37C7RELWSGRMJ4K2CJVLHLVMHKFMTES6O5AJ7EISJIRAQEAQ, seg_pos={0 0}, upload=474126]: Done uploading piece. err: <nil> Dec 28 04:06:34.567956733 (540h37m34.439039779s) [noise=true , address=145.79.227.29:4942 , piece_id=VSPVJJSIKGH5LRJWUZ2MIBKCNBDT6FJHVYEKUYGOAC52SRAAAYJQ, seg_pos={0 0}, upload=474126]: Done uploading piece. err: <nil> Dec 28 04:06:34.574366128 (540h37m34.445449173s) [noise=true , address=145.79.227.5:4223 , piece_id=HQ5NRY5UMML4HJD66MLC52QFMLQU5DX65ORCUA2UYVEPZSQNML7Q, seg_pos={0 0}, upload=474126]: Done uploading piece. err: <nil> Dec 28 04:06:34.578206240 (540h37m34.449289285s) [noise=true , address=145.79.227.42:4123 , piece_id=4XTLTS6K3CA2QTGNXHH4PNC7B5T7I5VMTXXNFBKYOOKAD5RAULHA, seg_pos={0 0}, upload=474126]: Done uploading piece. err: <nil> Dec 28 04:06:34.604791702 (540h37m34.475874738s) [noise=true , address=145.79.227.7:4293 , piece_id=HO6BKNDFLXYLUAQXFM6HT2AYYXHX5R3ZNXG5PMK5IK2ZBJ2DVIHQ, seg_pos={0 0}, upload=474126]: Done uploading piece. err: <nil> Dec 28 04:06:34.698911531 (540h37m34.569994566s) [noise=true , address=45.91.116.5:4299 , piece_id=KKFSQUEMBEEDHTEPNCPO7QAQSNORMGLP542XV3EIXNDZ7OVIECIA, seg_pos={0 0}, upload=474126]: Done uploading piece. err: <nil> Dec 28 04:06:34.698930827 (540h37m34.570013862s) [seg_pos={0 0}, upload=474126]: Segment reached optimal threshold of 13 pieces. Dec 28 04:06:34.699107023 (540h37m34.570190058s) [noise=true , address=77.90.31.21:4878 , piece_id=KYOBWODUBC65WHH2R36AENQ4ZHQJV7TPXAHI7DJVU44FEZPHKSPQ, seg_pos={0 0}, upload=474126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 28 04:06:34.699113927 (540h37m34.570196962s) [noise=true , address=77.90.31.21:4878 , piece_id=KYOBWODUBC65WHH2R36AENQ4ZHQJV7TPXAHI7DJVU44FEZPHKSPQ, seg_pos={0 0}, upload=474126]: Long tail cancellation due to optimal threshold Dec 28 04:06:34.699166576 (540h37m34.570249611s) [noise=true , address=77.90.31.19:4841 , piece_id=GEWVRLICJH6TJZVOCTEHTNB7IOFLHV6OBSV6SELQ6NA6KYB47XMA, seg_pos={0 0}, upload=474126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: expected piece hash; context canceled; EOF Dec 28 04:06:34.699174942 (540h37m34.570257987s) [noise=true , address=77.90.31.19:4841 , piece_id=GEWVRLICJH6TJZVOCTEHTNB7IOFLHV6OBSV6SELQ6NA6KYB47XMA, seg_pos={0 0}, upload=474126]: Long tail cancellation due to optimal threshold Dec 28 04:06:34.699307374 (540h37m34.57039042s) [noise=true , address=45.91.116.6:4976 , piece_id=PY3NMUSGLJTJEZ7NNXNBYCNL4FV4PSCX3GG7AG5UGO3CP3BEB7WQ, seg_pos={0 0}, upload=474126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Dec 28 04:06:34.699313907 (540h37m34.570396942s) [noise=true , address=45.91.116.6:4976 , piece_id=PY3NMUSGLJTJEZ7NNXNBYCNL4FV4PSCX3GG7AG5UGO3CP3BEB7WQ, seg_pos={0 0}, upload=474126]: Long tail cancellation due to optimal threshold Dec 28 04:06:34.699332302 (540h37m34.570415337s) [seg_pos={0 0}, upload=474126]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 04:06:34.699400432 (540h37m34.570483467s) [upload=474126]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:06:34.699405291 (540h37m34.570488326s) [upload=474126]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:06:34.982927360 (540h37m34.854010406s) [upload=474126]: Done uploading segments... Dec 28 04:06:34.982946386 (540h37m34.854029421s) [upload=474126]: Upload finished. err: <nil> Duration: 30.003093349s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 7.972883914s trace: https://jaeger.dev.storj.io/trace/3b8130f32e60318d FAIL (2025-12-28 04:06:58 UTC) orbiter warning: context deadline exceeded Duration: 7.972911046s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 10.078604083s trace: https://jaeger.dev.storj.io/trace/3b8130f32e60318d FAIL (2025-12-28 04:06:39 UTC) orbiter warning: context deadline exceeded Duration: 10.078635693s edge gateway us-select-1

1
...
11347
11348
11349
11350
11351
...
17986