Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................................X..................] duration: 3.41151233s trace: https://jaeger.dev.storj.io/trace/1ac298e20b765731 FAIL (2025-12-30 02:09:01 UTC) orbiter warning: context deadline exceeded Duration: 3.411541044s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [X.....X.X......X.X..XX.X....X..XXX..XX.....X..X...XX....X..X] duration: 30.002744258s trace: https://jaeger.dev.storj.io/trace/479ce6f5b39959c5 FAIL (2025-12-30 02:08:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: N7JAMZR4MM2CLIE7TD6YNZ7LUSC3ZDK3PAJ7H7LTPO6CSQPAOREQ) context deadline exceeded; piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: 4MGEVSAFPE2LF53LVPUILGIWCM4262YPSHXIS5JR5NHF67EJE45A) context deadline exceeded ======LOGS====== Dec 30 02:08:07.606216755 (586h39m7.47729979s) [upload=514281]: Starting upload Dec 30 02:08:07.606292990 (586h39m7.477376045s) [upload=514281]: Done starting upload Dec 30 02:08:07.606639209 (586h39m7.477722244s) [upload=514281]: Uploading segments... Dec 30 02:08:07.606683122 (586h39m7.477766157s) [upload=514281]: Got next segment. Inline: false Dec 30 02:08:07.606709964 (586h39m7.477793009s) [upload=514281]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 02:08:07.606714913 (586h39m7.477797948s) [upload=514281]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 02:08:07.853588657 (586h39m7.724671702s) [seg_pos={0 0}, upload=514281]: Begin upload segment... Dec 30 02:08:07.853673920 (586h39m7.724756965s) [seg_pos={0 0}, upload=514281]: Done begin upload segment. Dec 30 02:08:07.853710370 (586h39m7.724793415s) [noise=true , address=77.90.31.14:4662 , piece_id=2VPRDYHOLAKBJOWCBAD54VY6KFXSOY4FTQJAUU35EK7H6YA4VEIQ, seg_pos={0 0}, upload=514281]: Uploading piece... Dec 30 02:08:07.858169751 (586h39m7.729252796s) [noise=true , address=45.91.116.12:4227 , piece_id=ZU7ECQPRZ432P4HQWE7GJT7AC5I3J5XSMYKFIIJJRVVP3UKOYCGQ, seg_pos={0 0}, upload=514281]: Uploading piece... Dec 30 02:08:07.859356701 (586h39m7.730439756s) [noise=true , address=145.79.227.51:4411 , piece_id=RM5TFCTODMNFJTMWUINYWOOTSZENJJK43CW3PIBGUEUSRE37HRGA, seg_pos={0 0}, upload=514281]: Uploading piece... Dec 30 02:08:07.859815905 (586h39m7.73089894s) [noise=true , address=45.91.116.8:4046 , piece_id=N7JAMZR4MM2CLIE7TD6YNZ7LUSC3ZDK3PAJ7H7LTPO6CSQPAOREQ, seg_pos={0 0}, upload=514281]: Uploading piece... Dec 30 02:08:07.859889195 (586h39m7.73097224s) [noise=true , address=45.91.116.9:4264 , piece_id=4MGEVSAFPE2LF53LVPUILGIWCM4262YPSHXIS5JR5NHF67EJE45A, seg_pos={0 0}, upload=514281]: Uploading piece... Dec 30 02:08:07.860999228 (586h39m7.732082263s) [noise=true , address=145.79.227.52:4445 , piece_id=JUDJXYZWZ6A56E3VEBUYQY7QQM5NJ3UI3KQ36OMZVKA5DQLFRF4Q, seg_pos={0 0}, upload=514281]: Uploading piece... Dec 30 02:08:07.861557912 (586h39m7.732640957s) [noise=true , address=145.79.227.23:4545 , piece_id=LEMUCRD2OP36PRCVR3M6JA5IDD4IE7EADQXAMEVOEDURAO4YQV2A, seg_pos={0 0}, upload=514281]: Uploading piece... Dec 30 02:08:07.862251252 (586h39m7.733334298s) [noise=true , address=145.79.227.17:4971 , piece_id=AGJQ7POJRWBIPVQM7OUBG76VG3RZOJFPG7XTIV3G4J5A7MTD53WQ, seg_pos={0 0}, upload=514281]: Uploading piece... Dec 30 02:08:07.863442120 (586h39m7.734525165s) [noise=true , address=145.79.227.45:4230 , piece_id=BFN4HACSYYTO7ONCYZJVYTMROVZLIJPF34SYDLKFMHM7BRB5VXFQ, seg_pos={0 0}, upload=514281]: Uploading piece... Dec 30 02:08:07.863482917 (586h39m7.734565952s) [noise=true , address=145.79.227.5:4221 , piece_id=5UZJMUH4OEO325NW5JMQFHB3ESZZ3GGB7JBKO3GGBY7NS6YCVEAQ, seg_pos={0 0}, upload=514281]: Uploading piece... Dec 30 02:08:07.864563896 (586h39m7.735646941s) [noise=true , address=145.79.227.31:4798 , piece_id=MROO5ZVQIK6NYZVVSRLYJAKABRLCFNGGB4VR4VL7MPUJ7EYNUAMQ, seg_pos={0 0}, upload=514281]: Uploading piece... Dec 30 02:08:07.865139372 (586h39m7.736222407s) [noise=true , address=77.90.31.10:4518 , piece_id=33X5ZAU6PVISIFLPDURENDMJ7FCRWTC7IVR2N757K6G6IOFIUWDQ, seg_pos={0 0}, upload=514281]: Uploading piece... Dec 30 02:08:07.865626168 (586h39m7.736709213s) [noise=true , address=145.79.227.42:4122 , piece_id=P3EGJBQSC5BJR7FNGLCDXRR5YM3ZOZB64IT3AVP2T5WWRXZK4QAQ, seg_pos={0 0}, upload=514281]: Uploading piece... Dec 30 02:08:07.866939940 (586h39m7.738022985s) [noise=true , address=145.79.227.16:4911 , piece_id=VZL7GPYLVNITXWGDYQAZXARLHKZKM7V2R7OWXGLIJKK4JLIXIJOA, seg_pos={0 0}, upload=514281]: Uploading piece... Dec 30 02:08:07.867045240 (586h39m7.738128275s) [noise=true , address=45.91.116.14:4189 , piece_id=UD6LU4AR53JTEHRE3KQ444MIQSJYMHBY3VJS7AR74FJFNAXMIC4Q, seg_pos={0 0}, upload=514281]: Uploading piece... Dec 30 02:08:07.869126614 (586h39m7.740209649s) [noise=true , address=77.90.31.15:4697 , piece_id=ZXVYS5HIDQZPO7ACAILFSW34OORV6TA5SMK26F4FAOGKL2IQDK2A, seg_pos={0 0}, upload=514281]: Uploading piece... Dec 30 02:08:09.565346160 (586h39m9.436429225s) [upload=514281]: Next returned nil segment Dec 30 02:08:09.565371488 (586h39m9.436454533s) [upload=514281]: Waiting for error group managing segments... Dec 30 02:08:09.846908156 (586h39m9.717991201s) [noise=true , address=145.79.227.51:4411 , piece_id=RM5TFCTODMNFJTMWUINYWOOTSZENJJK43CW3PIBGUEUSRE37HRGA, seg_pos={0 0}, upload=514281]: Done uploading piece. err: <nil> Dec 30 02:08:09.849824109 (586h39m9.720907144s) [noise=true , address=145.79.227.17:4971 , piece_id=AGJQ7POJRWBIPVQM7OUBG76VG3RZOJFPG7XTIV3G4J5A7MTD53WQ, seg_pos={0 0}, upload=514281]: Done uploading piece. err: <nil> Dec 30 02:08:09.856886456 (586h39m9.727969491s) [noise=true , address=145.79.227.16:4911 , piece_id=VZL7GPYLVNITXWGDYQAZXARLHKZKM7V2R7OWXGLIJKK4JLIXIJOA, seg_pos={0 0}, upload=514281]: Done uploading piece. err: <nil> Dec 30 02:08:09.862822580 (586h39m9.733905625s) [noise=true , address=145.79.227.31:4798 , piece_id=MROO5ZVQIK6NYZVVSRLYJAKABRLCFNGGB4VR4VL7MPUJ7EYNUAMQ, seg_pos={0 0}, upload=514281]: Done uploading piece. err: <nil> Dec 30 02:08:09.878895705 (586h39m9.74997874s) [noise=true , address=145.79.227.52:4445 , piece_id=JUDJXYZWZ6A56E3VEBUYQY7QQM5NJ3UI3KQ36OMZVKA5DQLFRF4Q, seg_pos={0 0}, upload=514281]: Done uploading piece. err: <nil> Dec 30 02:08:09.881665609 (586h39m9.752748644s) [noise=true , address=145.79.227.5:4221 , piece_id=5UZJMUH4OEO325NW5JMQFHB3ESZZ3GGB7JBKO3GGBY7NS6YCVEAQ, seg_pos={0 0}, upload=514281]: Done uploading piece. err: <nil> Dec 30 02:08:09.886000273 (586h39m9.757083318s) [noise=true , address=145.79.227.42:4122 , piece_id=P3EGJBQSC5BJR7FNGLCDXRR5YM3ZOZB64IT3AVP2T5WWRXZK4QAQ, seg_pos={0 0}, upload=514281]: Done uploading piece. err: <nil> Dec 30 02:08:09.891796790 (586h39m9.762879855s) [noise=true , address=145.79.227.23:4545 , piece_id=LEMUCRD2OP36PRCVR3M6JA5IDD4IE7EADQXAMEVOEDURAO4YQV2A, seg_pos={0 0}, upload=514281]: Done uploading piece. err: <nil> Dec 30 02:08:09.897477287 (586h39m9.768560322s) [noise=true , address=145.79.227.45:4230 , piece_id=BFN4HACSYYTO7ONCYZJVYTMROVZLIJPF34SYDLKFMHM7BRB5VXFQ, seg_pos={0 0}, upload=514281]: Done uploading piece. err: <nil> Dec 30 02:08:10.098844039 (586h39m9.969927074s) [noise=true , address=45.91.116.12:4227 , piece_id=ZU7ECQPRZ432P4HQWE7GJT7AC5I3J5XSMYKFIIJJRVVP3UKOYCGQ, seg_pos={0 0}, upload=514281]: Done uploading piece. err: <nil> Dec 30 02:08:10.120124961 (586h39m9.991207997s) [noise=true , address=77.90.31.14:4662 , piece_id=2VPRDYHOLAKBJOWCBAD54VY6KFXSOY4FTQJAUU35EK7H6YA4VEIQ, seg_pos={0 0}, upload=514281]: Done uploading piece. err: <nil> Dec 30 02:08:10.198041406 (586h39m10.069124441s) [noise=true , address=45.91.116.9:4264 , piece_id=4MGEVSAFPE2LF53LVPUILGIWCM4262YPSHXIS5JR5NHF67EJE45A, seg_pos={0 0}, upload=514281]: Done uploading piece. err: <nil> Dec 30 02:08:10.438560480 (586h39m10.309643515s) [noise=true , address=45.91.116.8:4046 , piece_id=N7JAMZR4MM2CLIE7TD6YNZ7LUSC3ZDK3PAJ7H7LTPO6CSQPAOREQ, seg_pos={0 0}, upload=514281]: Done uploading piece. err: <nil> Dec 30 02:08:10.438582171 (586h39m10.309665206s) [seg_pos={0 0}, upload=514281]: Segment reached optimal threshold of 13 pieces. Dec 30 02:08:10.438776682 (586h39m10.309859717s) [noise=true , address=77.90.31.15:4697 , piece_id=ZXVYS5HIDQZPO7ACAILFSW34OORV6TA5SMK26F4FAOGKL2IQDK2A, seg_pos={0 0}, upload=514281]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Dec 30 02:08:10.438783284 (586h39m10.309866319s) [noise=true , address=77.90.31.15:4697 , piece_id=ZXVYS5HIDQZPO7ACAILFSW34OORV6TA5SMK26F4FAOGKL2IQDK2A, seg_pos={0 0}, upload=514281]: Long tail cancellation due to optimal threshold Dec 30 02:08:10.438902010 (586h39m10.309985045s) [noise=true , address=77.90.31.10:4518 , piece_id=33X5ZAU6PVISIFLPDURENDMJ7FCRWTC7IVR2N757K6G6IOFIUWDQ, seg_pos={0 0}, upload=514281]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: write tcp 188.166.225.170:38642->77.90.31.10:4518: use of closed network connection; write tcp 188.166.225.170:38642->77.90.31.10:4518: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38642->77.90.31.10:4518: use of closed network connection Dec 30 02:08:10.438917169 (586h39m10.310000204s) [noise=true , address=77.90.31.10:4518 , piece_id=33X5ZAU6PVISIFLPDURENDMJ7FCRWTC7IVR2N757K6G6IOFIUWDQ, seg_pos={0 0}, upload=514281]: Long tail cancellation due to optimal threshold Dec 30 02:08:10.438993324 (586h39m10.310076359s) [noise=true , address=45.91.116.14:4189 , piece_id=UD6LU4AR53JTEHRE3KQ444MIQSJYMHBY3VJS7AR74FJFNAXMIC4Q, seg_pos={0 0}, upload=514281]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 30 02:08:10.439000217 (586h39m10.310083262s) [noise=true , address=45.91.116.14:4189 , piece_id=UD6LU4AR53JTEHRE3KQ444MIQSJYMHBY3VJS7AR74FJFNAXMIC4Q, seg_pos={0 0}, upload=514281]: Long tail cancellation due to optimal threshold Dec 30 02:08:10.439027739 (586h39m10.310110784s) [seg_pos={0 0}, upload=514281]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 02:08:10.439110547 (586h39m10.310193582s) [upload=514281]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 02:08:10.439115576 (586h39m10.310198611s) [upload=514281]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 02:08:10.764123169 (586h39m10.635206214s) [upload=514281]: Done uploading segments... Dec 30 02:08:10.764147987 (586h39m10.635231032s) [upload=514281]: Upload finished. err: <nil> Duration: 30.002768744s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................................X.................] duration: 3.453510869s trace: https://jaeger.dev.storj.io/trace/1ac298e20b765731 FAIL (2025-12-30 02:07:54 UTC) orbiter warning: context deadline exceeded Duration: 3.453534735s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [XX.....X.X......X.X..XX.X....X..XXX..XX.....X..X...XX....X..] duration: 5.917400165s trace: https://jaeger.dev.storj.io/trace/f0fe7c75b02f7a1 FAIL (2025-12-30 02:06:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: ROARONT7ZCGQ2PIKKUWT7QX4H2DVKKZMYQDOOELOBIOSGZQ4ERNQ) context deadline exceeded; piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: LHBKJDOLOFTCXNXHJYJNHQ7DXULKAZ4C2LZLPSFOXVUTX2BCLWGQ) context deadline exceeded ======LOGS====== m0.922847005s) [upload=514249]: Uploading segments... Dec 30 02:06:01.051859911 (586h37m0.922942957s) [upload=514249]: Got next segment. Inline: false Dec 30 02:06:01.051891702 (586h37m0.922974747s) [upload=514249]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 02:06:01.051898044 (586h37m0.922981089s) [upload=514249]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 02:06:01.302654583 (586h37m1.173737638s) [seg_pos={0 0}, upload=514249]: Begin upload segment... Dec 30 02:06:01.302725328 (586h37m1.173808373s) [seg_pos={0 0}, upload=514249]: Done begin upload segment. Dec 30 02:06:01.302783398 (586h37m1.173866433s) [noise=true , address=145.79.227.39:4015 , piece_id=QTCVXZ4G4H3G5F24P672PGHWEOTBTRI4BITTNJTDR2FA7UXBD24A, seg_pos={0 0}, upload=514249]: Uploading piece... Dec 30 02:06:01.302806723 (586h37m1.173889768s) [noise=true , address=45.91.116.12:4227 , piece_id=ROARONT7ZCGQ2PIKKUWT7QX4H2DVKKZMYQDOOELOBIOSGZQ4ERNQ, seg_pos={0 0}, upload=514249]: Uploading piece... Dec 30 02:06:01.303273912 (586h37m1.174356947s) [noise=true , address=145.79.227.5:4222 , piece_id=NVLSFK5BHTPAZ4NNAC77HI7F4FG6G76FH7YRRUWXEEEPC76LZITA, seg_pos={0 0}, upload=514249]: Uploading piece... Dec 30 02:06:01.303674144 (586h37m1.174757179s) [noise=true , address=77.90.31.14:4662 , piece_id=LHBKJDOLOFTCXNXHJYJNHQ7DXULKAZ4C2LZLPSFOXVUTX2BCLWGQ, seg_pos={0 0}, upload=514249]: Uploading piece... Dec 30 02:06:01.303966791 (586h37m1.175049836s) [noise=true , address=45.91.116.9:4261 , piece_id=UDKIRST2W6K66CFR2VYMEFFJTOAMXBUY33YNL46UKHBVBBXAVYXQ, seg_pos={0 0}, upload=514249]: Uploading piece... Dec 30 02:06:01.304810638 (586h37m1.175893683s) [noise=true , address=145.79.227.48:4303 , piece_id=GBTJXHMMLJNNZK5BVTP4YZTB4EMR63NYAQZX4PTVWCJMBORQ3QTA, seg_pos={0 0}, upload=514249]: Uploading piece... Dec 30 02:06:01.305153431 (586h37m1.176236476s) [noise=true , address=145.79.227.50:4374 , piece_id=B4N6P7VCN77WGCPRE6DZEXZWCRTB2MAR7NBD7MO6TZ22J5KYVTAA, seg_pos={0 0}, upload=514249]: Uploading piece... Dec 30 02:06:01.305476877 (586h37m1.176559922s) [noise=true , address=145.79.227.38:4977 , piece_id=FUCOSAYWSQWQQNP2MKTD52HPFKXG6MSHHQCPJ3M4QEI6GZ6R5KFA, seg_pos={0 0}, upload=514249]: Uploading piece... Dec 30 02:06:01.305932184 (586h37m1.177015229s) [noise=true , address=77.90.31.18:4808 , piece_id=LLBMPCLOMJIEHVGUPRHHFLZVRYOUW4YWXTPZT5I52SKOABLCLITQ, seg_pos={0 0}, upload=514249]: Uploading piece... Dec 30 02:06:01.306362273 (586h37m1.177445318s) [noise=true , address=45.91.116.6:4975 , piece_id=47YOLXUCYHRNDQQ5VOHJQV5ZLDARTZUZEKAA6WCLUI4LGGEQC6MQ, seg_pos={0 0}, upload=514249]: Uploading piece... Dec 30 02:06:01.306760431 (586h37m1.177843476s) [noise=true , address=145.79.227.17:4971 , piece_id=ABRF7TU6TL4DGUBSH263ETP6UPWMNDVQT7BZL4HJ5RVHKEKGXEGA, seg_pos={0 0}, upload=514249]: Uploading piece... Dec 30 02:06:01.307112712 (586h37m1.178195747s) [noise=true , address=145.79.227.7:4295 , piece_id=ITF76GE5NGAHZYARL4Z5YSQ2SDI2HNWMJ6F7VB6NCXDF4HVY2RSQ, seg_pos={0 0}, upload=514249]: Uploading piece... Dec 30 02:06:01.308313108 (586h37m1.179396143s) [noise=true , address=145.79.227.51:4410 , piece_id=GQQBCV4EB254OGAFFZZFAGUBMF6EIMVBRKA3MQOMV2DUOHWZ4UZQ, seg_pos={0 0}, upload=514249]: Uploading piece... Dec 30 02:06:01.309587164 (586h37m1.180670199s) [noise=true , address=145.79.227.27:4689 , piece_id=VMIEEWOEEPIMKHRTOSGCP4DYFGYQMRPXKIQDFE557OR3QSBSYY3Q, seg_pos={0 0}, upload=514249]: Uploading piece... Dec 30 02:06:01.312902697 (586h37m1.183985742s) [noise=true , address=145.79.227.36:4519 , piece_id=VSH2RCLD5WCE2MKKFKV2KTDACHNN5KQDHGEW5VS4IWDHFUPUY5YQ, seg_pos={0 0}, upload=514249]: Uploading piece... Dec 30 02:06:01.314093464 (586h37m1.185176499s) [noise=true , address=45.91.116.10:4081 , piece_id=FBQFKN3FG4YYVHRMEOEEQTZVX5NQ3M6SUM2L7BCABGLVXNGIWTFA, seg_pos={0 0}, upload=514249]: Uploading piece... Dec 30 02:06:02.984975804 (586h37m2.856058849s) [upload=514249]: Next returned nil segment Dec 30 02:06:02.985012594 (586h37m2.856095639s) [upload=514249]: Waiting for error group managing segments... Dec 30 02:06:03.293078868 (586h37m3.164161944s) [noise=true , address=145.79.227.5:4222 , piece_id=NVLSFK5BHTPAZ4NNAC77HI7F4FG6G76FH7YRRUWXEEEPC76LZITA, seg_pos={0 0}, upload=514249]: Done uploading piece. err: <nil> Dec 30 02:06:03.298386254 (586h37m3.169469299s) [noise=true , address=145.79.227.50:4374 , piece_id=B4N6P7VCN77WGCPRE6DZEXZWCRTB2MAR7NBD7MO6TZ22J5KYVTAA, seg_pos={0 0}, upload=514249]: Done uploading piece. err: <nil> Dec 30 02:06:03.298893039 (586h37m3.169976084s) [noise=true , address=145.79.227.17:4971 , piece_id=ABRF7TU6TL4DGUBSH263ETP6UPWMNDVQT7BZL4HJ5RVHKEKGXEGA, seg_pos={0 0}, upload=514249]: Done uploading piece. err: <nil> Dec 30 02:06:03.301241511 (586h37m3.172324556s) [noise=true , address=145.79.227.38:4977 , piece_id=FUCOSAYWSQWQQNP2MKTD52HPFKXG6MSHHQCPJ3M4QEI6GZ6R5KFA, seg_pos={0 0}, upload=514249]: Done uploading piece. err: <nil> Dec 30 02:06:03.304412709 (586h37m3.175495754s) [noise=true , address=145.79.227.7:4295 , piece_id=ITF76GE5NGAHZYARL4Z5YSQ2SDI2HNWMJ6F7VB6NCXDF4HVY2RSQ, seg_pos={0 0}, upload=514249]: Done uploading piece. err: <nil> Dec 30 02:06:03.305835589 (586h37m3.176918634s) [noise=true , address=145.79.227.48:4303 , piece_id=GBTJXHMMLJNNZK5BVTP4YZTB4EMR63NYAQZX4PTVWCJMBORQ3QTA, seg_pos={0 0}, upload=514249]: Done uploading piece. err: <nil> Dec 30 02:06:03.309669149 (586h37m3.180752194s) [noise=true , address=145.79.227.36:4519 , piece_id=VSH2RCLD5WCE2MKKFKV2KTDACHNN5KQDHGEW5VS4IWDHFUPUY5YQ, seg_pos={0 0}, upload=514249]: Done uploading piece. err: <nil> Dec 30 02:06:03.314772215 (586h37m3.18585526s) [noise=true , address=145.79.227.51:4410 , piece_id=GQQBCV4EB254OGAFFZZFAGUBMF6EIMVBRKA3MQOMV2DUOHWZ4UZQ, seg_pos={0 0}, upload=514249]: Done uploading piece. err: <nil> Dec 30 02:06:03.315495432 (586h37m3.186578478s) [noise=true , address=145.79.227.27:4689 , piece_id=VMIEEWOEEPIMKHRTOSGCP4DYFGYQMRPXKIQDFE557OR3QSBSYY3Q, seg_pos={0 0}, upload=514249]: Done uploading piece. err: <nil> Dec 30 02:06:03.335901369 (586h37m3.206984414s) [noise=true , address=145.79.227.39:4015 , piece_id=QTCVXZ4G4H3G5F24P672PGHWEOTBTRI4BITTNJTDR2FA7UXBD24A, seg_pos={0 0}, upload=514249]: Done uploading piece. err: <nil> Dec 30 02:06:03.543688236 (586h37m3.414771281s) [noise=true , address=77.90.31.14:4662 , piece_id=LHBKJDOLOFTCXNXHJYJNHQ7DXULKAZ4C2LZLPSFOXVUTX2BCLWGQ, seg_pos={0 0}, upload=514249]: Done uploading piece. err: <nil> Dec 30 02:06:03.553441247 (586h37m3.424524302s) [noise=true , address=45.91.116.9:4261 , piece_id=UDKIRST2W6K66CFR2VYMEFFJTOAMXBUY33YNL46UKHBVBBXAVYXQ, seg_pos={0 0}, upload=514249]: Done uploading piece. err: <nil> Dec 30 02:06:03.578893933 (586h37m3.449976978s) [noise=true , address=45.91.116.12:4227 , piece_id=ROARONT7ZCGQ2PIKKUWT7QX4H2DVKKZMYQDOOELOBIOSGZQ4ERNQ, seg_pos={0 0}, upload=514249]: Done uploading piece. err: <nil> Dec 30 02:06:03.578917427 (586h37m3.450000472s) [seg_pos={0 0}, upload=514249]: Segment reached optimal threshold of 13 pieces. Dec 30 02:06:03.579134831 (586h37m3.450217906s) [noise=true , address=77.90.31.18:4808 , piece_id=LLBMPCLOMJIEHVGUPRHHFLZVRYOUW4YWXTPZT5I52SKOABLCLITQ, seg_pos={0 0}, upload=514249]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: write tcp 188.166.225.170:32782->77.90.31.18:4808: use of closed network connection; write tcp 188.166.225.170:32782->77.90.31.18:4808: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32782->77.90.31.18:4808: use of closed network connection Dec 30 02:06:03.579148277 (586h37m3.450231322s) [noise=true , address=77.90.31.18:4808 , piece_id=LLBMPCLOMJIEHVGUPRHHFLZVRYOUW4YWXTPZT5I52SKOABLCLITQ, seg_pos={0 0}, upload=514249]: Long tail cancellation due to optimal threshold Dec 30 02:06:03.579358858 (586h37m3.450441893s) [noise=true , address=45.91.116.6:4975 , piece_id=47YOLXUCYHRNDQQ5VOHJQV5ZLDARTZUZEKAA6WCLUI4LGGEQC6MQ, seg_pos={0 0}, upload=514249]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 30 02:06:03.579376030 (586h37m3.450459075s) [noise=true , address=45.91.116.6:4975 , piece_id=47YOLXUCYHRNDQQ5VOHJQV5ZLDARTZUZEKAA6WCLUI4LGGEQC6MQ, seg_pos={0 0}, upload=514249]: Long tail cancellation due to optimal threshold Dec 30 02:06:03.579538440 (586h37m3.450621475s) [noise=true , address=45.91.116.10:4081 , piece_id=FBQFKN3FG4YYVHRMEOEEQTZVX5NQ3M6SUM2L7BCABGLVXNGIWTFA, seg_pos={0 0}, upload=514249]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: write tcp 188.166.225.170:45268->45.91.116.10:4081: use of closed network connection; write tcp 188.166.225.170:45268->45.91.116.10:4081: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45268->45.91.116.10:4081: use of closed network connection Dec 30 02:06:03.579545954 (586h37m3.450628989s) [noise=true , address=45.91.116.10:4081 , piece_id=FBQFKN3FG4YYVHRMEOEEQTZVX5NQ3M6SUM2L7BCABGLVXNGIWTFA, seg_pos={0 0}, upload=514249]: Long tail cancellation due to optimal threshold Dec 30 02:06:03.579563157 (586h37m3.450646192s) [seg_pos={0 0}, upload=514249]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 02:06:03.579647608 (586h37m3.450730653s) [upload=514249]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 02:06:03.579653098 (586h37m3.450736133s) [upload=514249]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 02:06:03.900027028 (586h37m3.771110073s) [upload=514249]: Done uploading segments... Dec 30 02:06:03.900047167 (586h37m3.771130202s) [upload=514249]: Upload finished. err: <nil> Duration: 5.91742862s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................................X................] duration: 3.535402835s trace: https://jaeger.dev.storj.io/trace/1ac298e20b765731 FAIL (2025-12-30 02:06:41 UTC) orbiter warning: context deadline exceeded Duration: 3.535429397s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [.XX.....X.X......X.X..XX.X....X..XXX..XX.....X..X...XX....X.] duration: 5.919917177s trace: https://jaeger.dev.storj.io/trace/f0fe7c75b02f7a1 FAIL (2025-12-30 02:06:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: ROARONT7ZCGQ2PIKKUWT7QX4H2DVKKZMYQDOOELOBIOSGZQ4ERNQ) context deadline exceeded; piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: LHBKJDOLOFTCXNXHJYJNHQ7DXULKAZ4C2LZLPSFOXVUTX2BCLWGQ) context deadline exceeded ======LOGS====== m0.922847005s) [upload=514249]: Uploading segments... Dec 30 02:06:01.051859911 (586h37m0.922942957s) [upload=514249]: Got next segment. Inline: false Dec 30 02:06:01.051891702 (586h37m0.922974747s) [upload=514249]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 02:06:01.051898044 (586h37m0.922981089s) [upload=514249]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 02:06:01.302654583 (586h37m1.173737638s) [seg_pos={0 0}, upload=514249]: Begin upload segment... Dec 30 02:06:01.302725328 (586h37m1.173808373s) [seg_pos={0 0}, upload=514249]: Done begin upload segment. Dec 30 02:06:01.302783398 (586h37m1.173866433s) [noise=true , address=145.79.227.39:4015 , piece_id=QTCVXZ4G4H3G5F24P672PGHWEOTBTRI4BITTNJTDR2FA7UXBD24A, seg_pos={0 0}, upload=514249]: Uploading piece... Dec 30 02:06:01.302806723 (586h37m1.173889768s) [noise=true , address=45.91.116.12:4227 , piece_id=ROARONT7ZCGQ2PIKKUWT7QX4H2DVKKZMYQDOOELOBIOSGZQ4ERNQ, seg_pos={0 0}, upload=514249]: Uploading piece... Dec 30 02:06:01.303273912 (586h37m1.174356947s) [noise=true , address=145.79.227.5:4222 , piece_id=NVLSFK5BHTPAZ4NNAC77HI7F4FG6G76FH7YRRUWXEEEPC76LZITA, seg_pos={0 0}, upload=514249]: Uploading piece... Dec 30 02:06:01.303674144 (586h37m1.174757179s) [noise=true , address=77.90.31.14:4662 , piece_id=LHBKJDOLOFTCXNXHJYJNHQ7DXULKAZ4C2LZLPSFOXVUTX2BCLWGQ, seg_pos={0 0}, upload=514249]: Uploading piece... Dec 30 02:06:01.303966791 (586h37m1.175049836s) [noise=true , address=45.91.116.9:4261 , piece_id=UDKIRST2W6K66CFR2VYMEFFJTOAMXBUY33YNL46UKHBVBBXAVYXQ, seg_pos={0 0}, upload=514249]: Uploading piece... Dec 30 02:06:01.304810638 (586h37m1.175893683s) [noise=true , address=145.79.227.48:4303 , piece_id=GBTJXHMMLJNNZK5BVTP4YZTB4EMR63NYAQZX4PTVWCJMBORQ3QTA, seg_pos={0 0}, upload=514249]: Uploading piece... Dec 30 02:06:01.305153431 (586h37m1.176236476s) [noise=true , address=145.79.227.50:4374 , piece_id=B4N6P7VCN77WGCPRE6DZEXZWCRTB2MAR7NBD7MO6TZ22J5KYVTAA, seg_pos={0 0}, upload=514249]: Uploading piece... Dec 30 02:06:01.305476877 (586h37m1.176559922s) [noise=true , address=145.79.227.38:4977 , piece_id=FUCOSAYWSQWQQNP2MKTD52HPFKXG6MSHHQCPJ3M4QEI6GZ6R5KFA, seg_pos={0 0}, upload=514249]: Uploading piece... Dec 30 02:06:01.305932184 (586h37m1.177015229s) [noise=true , address=77.90.31.18:4808 , piece_id=LLBMPCLOMJIEHVGUPRHHFLZVRYOUW4YWXTPZT5I52SKOABLCLITQ, seg_pos={0 0}, upload=514249]: Uploading piece... Dec 30 02:06:01.306362273 (586h37m1.177445318s) [noise=true , address=45.91.116.6:4975 , piece_id=47YOLXUCYHRNDQQ5VOHJQV5ZLDARTZUZEKAA6WCLUI4LGGEQC6MQ, seg_pos={0 0}, upload=514249]: Uploading piece... Dec 30 02:06:01.306760431 (586h37m1.177843476s) [noise=true , address=145.79.227.17:4971 , piece_id=ABRF7TU6TL4DGUBSH263ETP6UPWMNDVQT7BZL4HJ5RVHKEKGXEGA, seg_pos={0 0}, upload=514249]: Uploading piece... Dec 30 02:06:01.307112712 (586h37m1.178195747s) [noise=true , address=145.79.227.7:4295 , piece_id=ITF76GE5NGAHZYARL4Z5YSQ2SDI2HNWMJ6F7VB6NCXDF4HVY2RSQ, seg_pos={0 0}, upload=514249]: Uploading piece... Dec 30 02:06:01.308313108 (586h37m1.179396143s) [noise=true , address=145.79.227.51:4410 , piece_id=GQQBCV4EB254OGAFFZZFAGUBMF6EIMVBRKA3MQOMV2DUOHWZ4UZQ, seg_pos={0 0}, upload=514249]: Uploading piece... Dec 30 02:06:01.309587164 (586h37m1.180670199s) [noise=true , address=145.79.227.27:4689 , piece_id=VMIEEWOEEPIMKHRTOSGCP4DYFGYQMRPXKIQDFE557OR3QSBSYY3Q, seg_pos={0 0}, upload=514249]: Uploading piece... Dec 30 02:06:01.312902697 (586h37m1.183985742s) [noise=true , address=145.79.227.36:4519 , piece_id=VSH2RCLD5WCE2MKKFKV2KTDACHNN5KQDHGEW5VS4IWDHFUPUY5YQ, seg_pos={0 0}, upload=514249]: Uploading piece... Dec 30 02:06:01.314093464 (586h37m1.185176499s) [noise=true , address=45.91.116.10:4081 , piece_id=FBQFKN3FG4YYVHRMEOEEQTZVX5NQ3M6SUM2L7BCABGLVXNGIWTFA, seg_pos={0 0}, upload=514249]: Uploading piece... Dec 30 02:06:02.984975804 (586h37m2.856058849s) [upload=514249]: Next returned nil segment Dec 30 02:06:02.985012594 (586h37m2.856095639s) [upload=514249]: Waiting for error group managing segments... Dec 30 02:06:03.293078868 (586h37m3.164161944s) [noise=true , address=145.79.227.5:4222 , piece_id=NVLSFK5BHTPAZ4NNAC77HI7F4FG6G76FH7YRRUWXEEEPC76LZITA, seg_pos={0 0}, upload=514249]: Done uploading piece. err: <nil> Dec 30 02:06:03.298386254 (586h37m3.169469299s) [noise=true , address=145.79.227.50:4374 , piece_id=B4N6P7VCN77WGCPRE6DZEXZWCRTB2MAR7NBD7MO6TZ22J5KYVTAA, seg_pos={0 0}, upload=514249]: Done uploading piece. err: <nil> Dec 30 02:06:03.298893039 (586h37m3.169976084s) [noise=true , address=145.79.227.17:4971 , piece_id=ABRF7TU6TL4DGUBSH263ETP6UPWMNDVQT7BZL4HJ5RVHKEKGXEGA, seg_pos={0 0}, upload=514249]: Done uploading piece. err: <nil> Dec 30 02:06:03.301241511 (586h37m3.172324556s) [noise=true , address=145.79.227.38:4977 , piece_id=FUCOSAYWSQWQQNP2MKTD52HPFKXG6MSHHQCPJ3M4QEI6GZ6R5KFA, seg_pos={0 0}, upload=514249]: Done uploading piece. err: <nil> Dec 30 02:06:03.304412709 (586h37m3.175495754s) [noise=true , address=145.79.227.7:4295 , piece_id=ITF76GE5NGAHZYARL4Z5YSQ2SDI2HNWMJ6F7VB6NCXDF4HVY2RSQ, seg_pos={0 0}, upload=514249]: Done uploading piece. err: <nil> Dec 30 02:06:03.305835589 (586h37m3.176918634s) [noise=true , address=145.79.227.48:4303 , piece_id=GBTJXHMMLJNNZK5BVTP4YZTB4EMR63NYAQZX4PTVWCJMBORQ3QTA, seg_pos={0 0}, upload=514249]: Done uploading piece. err: <nil> Dec 30 02:06:03.309669149 (586h37m3.180752194s) [noise=true , address=145.79.227.36:4519 , piece_id=VSH2RCLD5WCE2MKKFKV2KTDACHNN5KQDHGEW5VS4IWDHFUPUY5YQ, seg_pos={0 0}, upload=514249]: Done uploading piece. err: <nil> Dec 30 02:06:03.314772215 (586h37m3.18585526s) [noise=true , address=145.79.227.51:4410 , piece_id=GQQBCV4EB254OGAFFZZFAGUBMF6EIMVBRKA3MQOMV2DUOHWZ4UZQ, seg_pos={0 0}, upload=514249]: Done uploading piece. err: <nil> Dec 30 02:06:03.315495432 (586h37m3.186578478s) [noise=true , address=145.79.227.27:4689 , piece_id=VMIEEWOEEPIMKHRTOSGCP4DYFGYQMRPXKIQDFE557OR3QSBSYY3Q, seg_pos={0 0}, upload=514249]: Done uploading piece. err: <nil> Dec 30 02:06:03.335901369 (586h37m3.206984414s) [noise=true , address=145.79.227.39:4015 , piece_id=QTCVXZ4G4H3G5F24P672PGHWEOTBTRI4BITTNJTDR2FA7UXBD24A, seg_pos={0 0}, upload=514249]: Done uploading piece. err: <nil> Dec 30 02:06:03.543688236 (586h37m3.414771281s) [noise=true , address=77.90.31.14:4662 , piece_id=LHBKJDOLOFTCXNXHJYJNHQ7DXULKAZ4C2LZLPSFOXVUTX2BCLWGQ, seg_pos={0 0}, upload=514249]: Done uploading piece. err: <nil> Dec 30 02:06:03.553441247 (586h37m3.424524302s) [noise=true , address=45.91.116.9:4261 , piece_id=UDKIRST2W6K66CFR2VYMEFFJTOAMXBUY33YNL46UKHBVBBXAVYXQ, seg_pos={0 0}, upload=514249]: Done uploading piece. err: <nil> Dec 30 02:06:03.578893933 (586h37m3.449976978s) [noise=true , address=45.91.116.12:4227 , piece_id=ROARONT7ZCGQ2PIKKUWT7QX4H2DVKKZMYQDOOELOBIOSGZQ4ERNQ, seg_pos={0 0}, upload=514249]: Done uploading piece. err: <nil> Dec 30 02:06:03.578917427 (586h37m3.450000472s) [seg_pos={0 0}, upload=514249]: Segment reached optimal threshold of 13 pieces. Dec 30 02:06:03.579134831 (586h37m3.450217906s) [noise=true , address=77.90.31.18:4808 , piece_id=LLBMPCLOMJIEHVGUPRHHFLZVRYOUW4YWXTPZT5I52SKOABLCLITQ, seg_pos={0 0}, upload=514249]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: write tcp 188.166.225.170:32782->77.90.31.18:4808: use of closed network connection; write tcp 188.166.225.170:32782->77.90.31.18:4808: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32782->77.90.31.18:4808: use of closed network connection Dec 30 02:06:03.579148277 (586h37m3.450231322s) [noise=true , address=77.90.31.18:4808 , piece_id=LLBMPCLOMJIEHVGUPRHHFLZVRYOUW4YWXTPZT5I52SKOABLCLITQ, seg_pos={0 0}, upload=514249]: Long tail cancellation due to optimal threshold Dec 30 02:06:03.579358858 (586h37m3.450441893s) [noise=true , address=45.91.116.6:4975 , piece_id=47YOLXUCYHRNDQQ5VOHJQV5ZLDARTZUZEKAA6WCLUI4LGGEQC6MQ, seg_pos={0 0}, upload=514249]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 30 02:06:03.579376030 (586h37m3.450459075s) [noise=true , address=45.91.116.6:4975 , piece_id=47YOLXUCYHRNDQQ5VOHJQV5ZLDARTZUZEKAA6WCLUI4LGGEQC6MQ, seg_pos={0 0}, upload=514249]: Long tail cancellation due to optimal threshold Dec 30 02:06:03.579538440 (586h37m3.450621475s) [noise=true , address=45.91.116.10:4081 , piece_id=FBQFKN3FG4YYVHRMEOEEQTZVX5NQ3M6SUM2L7BCABGLVXNGIWTFA, seg_pos={0 0}, upload=514249]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: write tcp 188.166.225.170:45268->45.91.116.10:4081: use of closed network connection; write tcp 188.166.225.170:45268->45.91.116.10:4081: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45268->45.91.116.10:4081: use of closed network connection Dec 30 02:06:03.579545954 (586h37m3.450628989s) [noise=true , address=45.91.116.10:4081 , piece_id=FBQFKN3FG4YYVHRMEOEEQTZVX5NQ3M6SUM2L7BCABGLVXNGIWTFA, seg_pos={0 0}, upload=514249]: Long tail cancellation due to optimal threshold Dec 30 02:06:03.579563157 (586h37m3.450646192s) [seg_pos={0 0}, upload=514249]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 02:06:03.579647608 (586h37m3.450730653s) [upload=514249]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 02:06:03.579653098 (586h37m3.450736133s) [upload=514249]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 02:06:03.900027028 (586h37m3.771110073s) [upload=514249]: Done uploading segments... Dec 30 02:06:03.900047167 (586h37m3.771130202s) [upload=514249]: Upload finished. err: <nil> Duration: 5.919947665s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [..XX.....X.X......X.X..XX.X....X..XXX..XX.....X..X...XX....X] duration: 30.004015087s trace: https://jaeger.dev.storj.io/trace/f0fe7c75b02f7a1 FAIL (2025-12-30 02:06:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: ROARONT7ZCGQ2PIKKUWT7QX4H2DVKKZMYQDOOELOBIOSGZQ4ERNQ) context deadline exceeded; piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: LHBKJDOLOFTCXNXHJYJNHQ7DXULKAZ4C2LZLPSFOXVUTX2BCLWGQ) context deadline exceeded ======LOGS====== m0.922847005s) [upload=514249]: Uploading segments... Dec 30 02:06:01.051859911 (586h37m0.922942957s) [upload=514249]: Got next segment. Inline: false Dec 30 02:06:01.051891702 (586h37m0.922974747s) [upload=514249]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 02:06:01.051898044 (586h37m0.922981089s) [upload=514249]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 02:06:01.302654583 (586h37m1.173737638s) [seg_pos={0 0}, upload=514249]: Begin upload segment... Dec 30 02:06:01.302725328 (586h37m1.173808373s) [seg_pos={0 0}, upload=514249]: Done begin upload segment. Dec 30 02:06:01.302783398 (586h37m1.173866433s) [noise=true , address=145.79.227.39:4015 , piece_id=QTCVXZ4G4H3G5F24P672PGHWEOTBTRI4BITTNJTDR2FA7UXBD24A, seg_pos={0 0}, upload=514249]: Uploading piece... Dec 30 02:06:01.302806723 (586h37m1.173889768s) [noise=true , address=45.91.116.12:4227 , piece_id=ROARONT7ZCGQ2PIKKUWT7QX4H2DVKKZMYQDOOELOBIOSGZQ4ERNQ, seg_pos={0 0}, upload=514249]: Uploading piece... Dec 30 02:06:01.303273912 (586h37m1.174356947s) [noise=true , address=145.79.227.5:4222 , piece_id=NVLSFK5BHTPAZ4NNAC77HI7F4FG6G76FH7YRRUWXEEEPC76LZITA, seg_pos={0 0}, upload=514249]: Uploading piece... Dec 30 02:06:01.303674144 (586h37m1.174757179s) [noise=true , address=77.90.31.14:4662 , piece_id=LHBKJDOLOFTCXNXHJYJNHQ7DXULKAZ4C2LZLPSFOXVUTX2BCLWGQ, seg_pos={0 0}, upload=514249]: Uploading piece... Dec 30 02:06:01.303966791 (586h37m1.175049836s) [noise=true , address=45.91.116.9:4261 , piece_id=UDKIRST2W6K66CFR2VYMEFFJTOAMXBUY33YNL46UKHBVBBXAVYXQ, seg_pos={0 0}, upload=514249]: Uploading piece... Dec 30 02:06:01.304810638 (586h37m1.175893683s) [noise=true , address=145.79.227.48:4303 , piece_id=GBTJXHMMLJNNZK5BVTP4YZTB4EMR63NYAQZX4PTVWCJMBORQ3QTA, seg_pos={0 0}, upload=514249]: Uploading piece... Dec 30 02:06:01.305153431 (586h37m1.176236476s) [noise=true , address=145.79.227.50:4374 , piece_id=B4N6P7VCN77WGCPRE6DZEXZWCRTB2MAR7NBD7MO6TZ22J5KYVTAA, seg_pos={0 0}, upload=514249]: Uploading piece... Dec 30 02:06:01.305476877 (586h37m1.176559922s) [noise=true , address=145.79.227.38:4977 , piece_id=FUCOSAYWSQWQQNP2MKTD52HPFKXG6MSHHQCPJ3M4QEI6GZ6R5KFA, seg_pos={0 0}, upload=514249]: Uploading piece... Dec 30 02:06:01.305932184 (586h37m1.177015229s) [noise=true , address=77.90.31.18:4808 , piece_id=LLBMPCLOMJIEHVGUPRHHFLZVRYOUW4YWXTPZT5I52SKOABLCLITQ, seg_pos={0 0}, upload=514249]: Uploading piece... Dec 30 02:06:01.306362273 (586h37m1.177445318s) [noise=true , address=45.91.116.6:4975 , piece_id=47YOLXUCYHRNDQQ5VOHJQV5ZLDARTZUZEKAA6WCLUI4LGGEQC6MQ, seg_pos={0 0}, upload=514249]: Uploading piece... Dec 30 02:06:01.306760431 (586h37m1.177843476s) [noise=true , address=145.79.227.17:4971 , piece_id=ABRF7TU6TL4DGUBSH263ETP6UPWMNDVQT7BZL4HJ5RVHKEKGXEGA, seg_pos={0 0}, upload=514249]: Uploading piece... Dec 30 02:06:01.307112712 (586h37m1.178195747s) [noise=true , address=145.79.227.7:4295 , piece_id=ITF76GE5NGAHZYARL4Z5YSQ2SDI2HNWMJ6F7VB6NCXDF4HVY2RSQ, seg_pos={0 0}, upload=514249]: Uploading piece... Dec 30 02:06:01.308313108 (586h37m1.179396143s) [noise=true , address=145.79.227.51:4410 , piece_id=GQQBCV4EB254OGAFFZZFAGUBMF6EIMVBRKA3MQOMV2DUOHWZ4UZQ, seg_pos={0 0}, upload=514249]: Uploading piece... Dec 30 02:06:01.309587164 (586h37m1.180670199s) [noise=true , address=145.79.227.27:4689 , piece_id=VMIEEWOEEPIMKHRTOSGCP4DYFGYQMRPXKIQDFE557OR3QSBSYY3Q, seg_pos={0 0}, upload=514249]: Uploading piece... Dec 30 02:06:01.312902697 (586h37m1.183985742s) [noise=true , address=145.79.227.36:4519 , piece_id=VSH2RCLD5WCE2MKKFKV2KTDACHNN5KQDHGEW5VS4IWDHFUPUY5YQ, seg_pos={0 0}, upload=514249]: Uploading piece... Dec 30 02:06:01.314093464 (586h37m1.185176499s) [noise=true , address=45.91.116.10:4081 , piece_id=FBQFKN3FG4YYVHRMEOEEQTZVX5NQ3M6SUM2L7BCABGLVXNGIWTFA, seg_pos={0 0}, upload=514249]: Uploading piece... Dec 30 02:06:02.984975804 (586h37m2.856058849s) [upload=514249]: Next returned nil segment Dec 30 02:06:02.985012594 (586h37m2.856095639s) [upload=514249]: Waiting for error group managing segments... Dec 30 02:06:03.293078868 (586h37m3.164161944s) [noise=true , address=145.79.227.5:4222 , piece_id=NVLSFK5BHTPAZ4NNAC77HI7F4FG6G76FH7YRRUWXEEEPC76LZITA, seg_pos={0 0}, upload=514249]: Done uploading piece. err: <nil> Dec 30 02:06:03.298386254 (586h37m3.169469299s) [noise=true , address=145.79.227.50:4374 , piece_id=B4N6P7VCN77WGCPRE6DZEXZWCRTB2MAR7NBD7MO6TZ22J5KYVTAA, seg_pos={0 0}, upload=514249]: Done uploading piece. err: <nil> Dec 30 02:06:03.298893039 (586h37m3.169976084s) [noise=true , address=145.79.227.17:4971 , piece_id=ABRF7TU6TL4DGUBSH263ETP6UPWMNDVQT7BZL4HJ5RVHKEKGXEGA, seg_pos={0 0}, upload=514249]: Done uploading piece. err: <nil> Dec 30 02:06:03.301241511 (586h37m3.172324556s) [noise=true , address=145.79.227.38:4977 , piece_id=FUCOSAYWSQWQQNP2MKTD52HPFKXG6MSHHQCPJ3M4QEI6GZ6R5KFA, seg_pos={0 0}, upload=514249]: Done uploading piece. err: <nil> Dec 30 02:06:03.304412709 (586h37m3.175495754s) [noise=true , address=145.79.227.7:4295 , piece_id=ITF76GE5NGAHZYARL4Z5YSQ2SDI2HNWMJ6F7VB6NCXDF4HVY2RSQ, seg_pos={0 0}, upload=514249]: Done uploading piece. err: <nil> Dec 30 02:06:03.305835589 (586h37m3.176918634s) [noise=true , address=145.79.227.48:4303 , piece_id=GBTJXHMMLJNNZK5BVTP4YZTB4EMR63NYAQZX4PTVWCJMBORQ3QTA, seg_pos={0 0}, upload=514249]: Done uploading piece. err: <nil> Dec 30 02:06:03.309669149 (586h37m3.180752194s) [noise=true , address=145.79.227.36:4519 , piece_id=VSH2RCLD5WCE2MKKFKV2KTDACHNN5KQDHGEW5VS4IWDHFUPUY5YQ, seg_pos={0 0}, upload=514249]: Done uploading piece. err: <nil> Dec 30 02:06:03.314772215 (586h37m3.18585526s) [noise=true , address=145.79.227.51:4410 , piece_id=GQQBCV4EB254OGAFFZZFAGUBMF6EIMVBRKA3MQOMV2DUOHWZ4UZQ, seg_pos={0 0}, upload=514249]: Done uploading piece. err: <nil> Dec 30 02:06:03.315495432 (586h37m3.186578478s) [noise=true , address=145.79.227.27:4689 , piece_id=VMIEEWOEEPIMKHRTOSGCP4DYFGYQMRPXKIQDFE557OR3QSBSYY3Q, seg_pos={0 0}, upload=514249]: Done uploading piece. err: <nil> Dec 30 02:06:03.335901369 (586h37m3.206984414s) [noise=true , address=145.79.227.39:4015 , piece_id=QTCVXZ4G4H3G5F24P672PGHWEOTBTRI4BITTNJTDR2FA7UXBD24A, seg_pos={0 0}, upload=514249]: Done uploading piece. err: <nil> Dec 30 02:06:03.543688236 (586h37m3.414771281s) [noise=true , address=77.90.31.14:4662 , piece_id=LHBKJDOLOFTCXNXHJYJNHQ7DXULKAZ4C2LZLPSFOXVUTX2BCLWGQ, seg_pos={0 0}, upload=514249]: Done uploading piece. err: <nil> Dec 30 02:06:03.553441247 (586h37m3.424524302s) [noise=true , address=45.91.116.9:4261 , piece_id=UDKIRST2W6K66CFR2VYMEFFJTOAMXBUY33YNL46UKHBVBBXAVYXQ, seg_pos={0 0}, upload=514249]: Done uploading piece. err: <nil> Dec 30 02:06:03.578893933 (586h37m3.449976978s) [noise=true , address=45.91.116.12:4227 , piece_id=ROARONT7ZCGQ2PIKKUWT7QX4H2DVKKZMYQDOOELOBIOSGZQ4ERNQ, seg_pos={0 0}, upload=514249]: Done uploading piece. err: <nil> Dec 30 02:06:03.578917427 (586h37m3.450000472s) [seg_pos={0 0}, upload=514249]: Segment reached optimal threshold of 13 pieces. Dec 30 02:06:03.579134831 (586h37m3.450217906s) [noise=true , address=77.90.31.18:4808 , piece_id=LLBMPCLOMJIEHVGUPRHHFLZVRYOUW4YWXTPZT5I52SKOABLCLITQ, seg_pos={0 0}, upload=514249]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: write tcp 188.166.225.170:32782->77.90.31.18:4808: use of closed network connection; write tcp 188.166.225.170:32782->77.90.31.18:4808: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32782->77.90.31.18:4808: use of closed network connection Dec 30 02:06:03.579148277 (586h37m3.450231322s) [noise=true , address=77.90.31.18:4808 , piece_id=LLBMPCLOMJIEHVGUPRHHFLZVRYOUW4YWXTPZT5I52SKOABLCLITQ, seg_pos={0 0}, upload=514249]: Long tail cancellation due to optimal threshold Dec 30 02:06:03.579358858 (586h37m3.450441893s) [noise=true , address=45.91.116.6:4975 , piece_id=47YOLXUCYHRNDQQ5VOHJQV5ZLDARTZUZEKAA6WCLUI4LGGEQC6MQ, seg_pos={0 0}, upload=514249]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 30 02:06:03.579376030 (586h37m3.450459075s) [noise=true , address=45.91.116.6:4975 , piece_id=47YOLXUCYHRNDQQ5VOHJQV5ZLDARTZUZEKAA6WCLUI4LGGEQC6MQ, seg_pos={0 0}, upload=514249]: Long tail cancellation due to optimal threshold Dec 30 02:06:03.579538440 (586h37m3.450621475s) [noise=true , address=45.91.116.10:4081 , piece_id=FBQFKN3FG4YYVHRMEOEEQTZVX5NQ3M6SUM2L7BCABGLVXNGIWTFA, seg_pos={0 0}, upload=514249]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: write tcp 188.166.225.170:45268->45.91.116.10:4081: use of closed network connection; write tcp 188.166.225.170:45268->45.91.116.10:4081: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45268->45.91.116.10:4081: use of closed network connection Dec 30 02:06:03.579545954 (586h37m3.450628989s) [noise=true , address=45.91.116.10:4081 , piece_id=FBQFKN3FG4YYVHRMEOEEQTZVX5NQ3M6SUM2L7BCABGLVXNGIWTFA, seg_pos={0 0}, upload=514249]: Long tail cancellation due to optimal threshold Dec 30 02:06:03.579563157 (586h37m3.450646192s) [seg_pos={0 0}, upload=514249]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 02:06:03.579647608 (586h37m3.450730653s) [upload=514249]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 02:06:03.579653098 (586h37m3.450736133s) [upload=514249]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 02:06:03.900027028 (586h37m3.771110073s) [upload=514249]: Done uploading segments... Dec 30 02:06:03.900047167 (586h37m3.771130202s) [upload=514249]: Upload finished. err: <nil> Duration: 30.004045774s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............................................X...............] duration: 3.477454553s trace: https://jaeger.dev.storj.io/trace/1ac298e20b765731 FAIL (2025-12-30 02:06:23 UTC) orbiter warning: context deadline exceeded Duration: 3.477482677s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [...XX.....X.X......X.X..XX.X....X..XXX..XX.....X..X...XX....] duration: 25.221619192s trace: https://jaeger.dev.storj.io/trace/532641d678d45a38 FAIL (2025-12-30 02:05:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 30 02:01:34.732123837 (586h32m34.603206882s) [upload=514181]: Starting upload Dec 30 02:01:34.732788423 (586h32m34.603871458s) [upload=514181]: Done starting upload Dec 30 02:01:34.733156514 (586h32m34.604239559s) [upload=514181]: Uploading segments... Dec 30 02:01:34.733346355 (586h32m34.6044294s) [upload=514181]: Got next segment. Inline: false Dec 30 02:01:34.733399156 (586h32m34.604482201s) [upload=514181]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 02:01:34.733405598 (586h32m34.604488633s) [upload=514181]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 02:01:34.981089223 (586h32m34.852172268s) [seg_pos={0 0}, upload=514181]: Begin upload segment... Dec 30 02:01:34.981141573 (586h32m34.852224619s) [seg_pos={0 0}, upload=514181]: Done begin upload segment. Dec 30 02:01:34.981176610 (586h32m34.852259645s) [noise=true , address=45.91.116.14:4192 , piece_id=OTPCOTLXEUGG6XGVYHZ2TI6RMPRPC5T3LMSBAZN37Q6MXTT22BCA, seg_pos={0 0}, upload=514181]: Uploading piece... Dec 30 02:01:34.981267323 (586h32m34.852350358s) [noise=true , address=77.90.31.13:4627 , piece_id=45IL2UTDXZKGFPNYOGH327WK2QFU7VNUYUFYACXDTEL22D2IWVSQ, seg_pos={0 0}, upload=514181]: Uploading piece... Dec 30 02:01:34.982134023 (586h32m34.853217068s) [noise=true , address=145.79.227.49:4339 , piece_id=GFPJ6JB23S5EBAGVCRQZKB7YAQVR3CTYIM6NN3NDCQJ4UJDWXDOA, seg_pos={0 0}, upload=514181]: Uploading piece... Dec 30 02:01:34.982569923 (586h32m34.853652968s) [noise=true , address=145.79.227.39:4014 , piece_id=ZTW6Y6X4RIAZKMZ3YGFJATRKDIGFAIABI6ESVEE3NBX4QTMJZLSQ, seg_pos={0 0}, upload=514181]: Uploading piece... Dec 30 02:01:34.982988961 (586h32m34.854071996s) [noise=true , address=145.79.227.7:4294 , piece_id=O77ASC66G5CHZE65SVWIDU644VEEUJVTQGVAZLHRP433SQKXWHRA, seg_pos={0 0}, upload=514181]: Uploading piece... Dec 30 02:01:34.983411956 (586h32m34.854495001s) [noise=true , address=77.90.31.10:4518 , piece_id=GOFOU2R5ZCM4QUZTBZ2XK2GEY3K6OCMHZVGNZ25TBTPBVB7LBCMA, seg_pos={0 0}, upload=514181]: Uploading piece... Dec 30 02:01:34.983848298 (586h32m34.854931333s) [noise=true , address=145.79.227.6:4259 , piece_id=S2O5VVGU57AEJJYGZG65HSZE45KPFMBJLY44OYSCYPQX4IL5XMWA, seg_pos={0 0}, upload=514181]: Uploading piece... Dec 30 02:01:34.984300238 (586h32m34.855383273s) [noise=true , address=45.91.116.6:4974 , piece_id=SRVYRQHGEJGRES4R5SSWLNZPRLJH7MMFN57INMGKNCRHATWEW75Q, seg_pos={0 0}, upload=514181]: Uploading piece... Dec 30 02:01:34.986365901 (586h32m34.857448936s) [noise=true , address=77.90.31.21:4880 , piece_id=GQ4P5ZEMX6IRPSXY5KGEYKLLX7KE73AL6HX2RWLMPZDZUIG4FH2Q, seg_pos={0 0}, upload=514181]: Uploading piece... Dec 30 02:01:34.986600188 (586h32m34.857683233s) [noise=true , address=145.79.227.16:4909 , piece_id=TRAUAVWRA2NHHCPPI6A3E4LSQ7P2U2CFGAC45XXUQZPEZUPNVH5Q, seg_pos={0 0}, upload=514181]: Uploading piece... Dec 30 02:01:34.988272051 (586h32m34.859355096s) [noise=true , address=145.79.227.29:4942 , piece_id=ZR72MSGGRQSWNKJKPRHYHPQ7OO2CTTSWRO7P6D34WHG4FGCGRLUQ, seg_pos={0 0}, upload=514181]: Uploading piece... Dec 30 02:01:34.989959885 (586h32m34.86104292s) [noise=true , address=77.90.31.16:4736 , piece_id=4RZTFN67VVUNF7ISNGAW6V4LKIG5HQX3VNHR6HL6XITQ4FM3U6OA, seg_pos={0 0}, upload=514181]: Uploading piece... Dec 30 02:01:34.992105891 (586h32m34.863188926s) [noise=true , address=45.91.116.10:4082 , piece_id=MPVJJ5AX75I4GHBWKG5D6TSWN5NI3C47YTISVSMS2HZSU6HFAKUA, seg_pos={0 0}, upload=514181]: Uploading piece... Dec 30 02:01:34.994081443 (586h32m34.865164489s) [noise=true , address=145.79.227.8:4331 , piece_id=PFO4K25EEQUE43YCOJXHBJLJ22ASFHYLBPLYZ4K37U7AMCTLIFNQ, seg_pos={0 0}, upload=514181]: Uploading piece... Dec 30 02:01:34.995799935 (586h32m34.86688298s) [noise=true , address=145.79.227.13:4803 , piece_id=DP6WODS2DTSCZRQDF73YEP3RAUE5F7JPBWETTTD5XDDEHDDA6FZA, seg_pos={0 0}, upload=514181]: Uploading piece... Dec 30 02:01:34.997462351 (586h32m34.868545386s) [noise=true , address=145.79.227.42:4121 , piece_id=TZSX3VWVFIPHKNDKVU44URH2FOSJ3YJTDKWH5QA2QZTJIF4DM7HQ, seg_pos={0 0}, upload=514181]: Uploading piece... Dec 30 02:01:36.687205603 (586h32m36.558288648s) [upload=514181]: Next returned nil segment Dec 30 02:01:36.687221724 (586h32m36.55830476s) [upload=514181]: Waiting for error group managing segments... Dec 30 02:01:36.966447461 (586h32m36.837530506s) [noise=true , address=145.79.227.39:4014 , piece_id=ZTW6Y6X4RIAZKMZ3YGFJATRKDIGFAIABI6ESVEE3NBX4QTMJZLSQ, seg_pos={0 0}, upload=514181]: Done uploading piece. err: <nil> Dec 30 02:01:36.969718400 (586h32m36.840801445s) [noise=true , address=145.79.227.7:4294 , piece_id=O77ASC66G5CHZE65SVWIDU644VEEUJVTQGVAZLHRP433SQKXWHRA, seg_pos={0 0}, upload=514181]: Done uploading piece. err: <nil> Dec 30 02:01:36.975611662 (586h32m36.846694707s) [noise=true , address=145.79.227.16:4909 , piece_id=TRAUAVWRA2NHHCPPI6A3E4LSQ7P2U2CFGAC45XXUQZPEZUPNVH5Q, seg_pos={0 0}, upload=514181]: Done uploading piece. err: <nil> Dec 30 02:01:36.980547260 (586h32m36.851630295s) [noise=true , address=145.79.227.6:4259 , piece_id=S2O5VVGU57AEJJYGZG65HSZE45KPFMBJLY44OYSCYPQX4IL5XMWA, seg_pos={0 0}, upload=514181]: Done uploading piece. err: <nil> Dec 30 02:01:36.985363039 (586h32m36.856446084s) [noise=true , address=145.79.227.13:4803 , piece_id=DP6WODS2DTSCZRQDF73YEP3RAUE5F7JPBWETTTD5XDDEHDDA6FZA, seg_pos={0 0}, upload=514181]: Done uploading piece. err: <nil> Dec 30 02:01:37.005192448 (586h32m36.876275483s) [noise=true , address=145.79.227.42:4121 , piece_id=TZSX3VWVFIPHKNDKVU44URH2FOSJ3YJTDKWH5QA2QZTJIF4DM7HQ, seg_pos={0 0}, upload=514181]: Done uploading piece. err: <nil> Dec 30 02:01:37.163718616 (586h32m37.034801661s) [noise=true , address=77.90.31.21:4880 , piece_id=GQ4P5ZEMX6IRPSXY5KGEYKLLX7KE73AL6HX2RWLMPZDZUIG4FH2Q, seg_pos={0 0}, upload=514181]: Done uploading piece. err: <nil> Dec 30 02:01:37.218883988 (586h32m37.089967023s) [noise=true , address=45.91.116.6:4974 , piece_id=SRVYRQHGEJGRES4R5SSWLNZPRLJH7MMFN57INMGKNCRHATWEW75Q, seg_pos={0 0}, upload=514181]: Done uploading piece. err: <nil> Dec 30 02:01:37.255428304 (586h32m37.126511339s) [noise=true , address=77.90.31.13:4627 , piece_id=45IL2UTDXZKGFPNYOGH327WK2QFU7VNUYUFYACXDTEL22D2IWVSQ, seg_pos={0 0}, upload=514181]: Done uploading piece. err: <nil> Dec 30 02:01:37.285705948 (586h32m37.156788983s) [noise=true , address=45.91.116.10:4082 , piece_id=MPVJJ5AX75I4GHBWKG5D6TSWN5NI3C47YTISVSMS2HZSU6HFAKUA, seg_pos={0 0}, upload=514181]: Done uploading piece. err: <nil> Dec 30 02:01:37.328629280 (586h32m37.199712335s) [noise=true , address=145.79.227.49:4339 , piece_id=GFPJ6JB23S5EBAGVCRQZKB7YAQVR3CTYIM6NN3NDCQJ4UJDWXDOA, seg_pos={0 0}, upload=514181]: Done uploading piece. err: <nil> Dec 30 02:01:37.385540347 (586h32m37.256623392s) [noise=true , address=45.91.116.14:4192 , piece_id=OTPCOTLXEUGG6XGVYHZ2TI6RMPRPC5T3LMSBAZN37Q6MXTT22BCA, seg_pos={0 0}, upload=514181]: Done uploading piece. err: <nil> Dec 30 02:01:37.852405909 (586h32m37.723488944s) [noise=true , address=77.90.31.10:4518 , piece_id=GOFOU2R5ZCM4QUZTBZ2XK2GEY3K6OCMHZVGNZ25TBTPBVB7LBCMA, seg_pos={0 0}, upload=514181]: Done uploading piece. err: <nil> Dec 30 02:01:37.852424484 (586h32m37.723507519s) [seg_pos={0 0}, upload=514181]: Segment reached optimal threshold of 13 pieces. Dec 30 02:01:37.852634875 (586h32m37.72371792s) [noise=true , address=145.79.227.29:4942 , piece_id=ZR72MSGGRQSWNKJKPRHYHPQ7OO2CTTSWRO7P6D34WHG4FGCGRLUQ, seg_pos={0 0}, upload=514181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7): protocol: expected piece hash; context canceled; EOF Dec 30 02:01:37.852641488 (586h32m37.723724523s) [noise=true , address=145.79.227.29:4942 , piece_id=ZR72MSGGRQSWNKJKPRHYHPQ7OO2CTTSWRO7P6D34WHG4FGCGRLUQ, seg_pos={0 0}, upload=514181]: Long tail cancellation due to optimal threshold Dec 30 02:01:37.852867298 (586h32m37.723950343s) [noise=true , address=145.79.227.8:4331 , piece_id=PFO4K25EEQUE43YCOJXHBJLJ22ASFHYLBPLYZ4K37U7AMCTLIFNQ, seg_pos={0 0}, upload=514181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: write tcp 188.166.225.170:59952->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:59952->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59952->145.79.227.8:4331: use of closed network connection Dec 30 02:01:37.852873239 (586h32m37.723956274s) [noise=true , address=145.79.227.8:4331 , piece_id=PFO4K25EEQUE43YCOJXHBJLJ22ASFHYLBPLYZ4K37U7AMCTLIFNQ, seg_pos={0 0}, upload=514181]: Long tail cancellation due to optimal threshold Dec 30 02:01:37.852925499 (586h32m37.724008534s) [noise=true , address=77.90.31.16:4736 , piece_id=4RZTFN67VVUNF7ISNGAW6V4LKIG5HQX3VNHR6HL6XITQ4FM3U6OA, seg_pos={0 0}, upload=514181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 30 02:01:37.852941509 (586h32m37.724024544s) [noise=true , address=77.90.31.16:4736 , piece_id=4RZTFN67VVUNF7ISNGAW6V4LKIG5HQX3VNHR6HL6XITQ4FM3U6OA, seg_pos={0 0}, upload=514181]: Long tail cancellation due to optimal threshold Dec 30 02:01:37.852975083 (586h32m37.724058118s) [seg_pos={0 0}, upload=514181]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 02:01:37.853062810 (586h32m37.724145845s) [upload=514181]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 02:01:37.853068361 (586h32m37.724151396s) [upload=514181]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 02:01:38.144036537 (586h32m38.015119592s) [upload=514181]: Done uploading segments... Dec 30 02:01:38.144066043 (586h32m38.015149098s) [upload=514181]: Upload finished. err: <nil> Duration: 25.221648256s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............................................X..............] duration: 3.353391198s trace: https://jaeger.dev.storj.io/trace/1ac298e20b765731 FAIL (2025-12-30 02:05:24 UTC) orbiter warning: context deadline exceeded Duration: 3.353416147s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............................................X.............] duration: 3.427780565s trace: https://jaeger.dev.storj.io/trace/1ac298e20b765731 FAIL (2025-12-30 02:04:53 UTC) orbiter warning: context deadline exceeded Duration: 3.427807206s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [....XX.....X.X......X.X..XX.X....X..XXX..XX.....X..X...XX...] duration: 5.93321674s trace: https://jaeger.dev.storj.io/trace/532641d678d45a38 FAIL (2025-12-30 02:04:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 30 02:01:34.732123837 (586h32m34.603206882s) [upload=514181]: Starting upload Dec 30 02:01:34.732788423 (586h32m34.603871458s) [upload=514181]: Done starting upload Dec 30 02:01:34.733156514 (586h32m34.604239559s) [upload=514181]: Uploading segments... Dec 30 02:01:34.733346355 (586h32m34.6044294s) [upload=514181]: Got next segment. Inline: false Dec 30 02:01:34.733399156 (586h32m34.604482201s) [upload=514181]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 02:01:34.733405598 (586h32m34.604488633s) [upload=514181]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 02:01:34.981089223 (586h32m34.852172268s) [seg_pos={0 0}, upload=514181]: Begin upload segment... Dec 30 02:01:34.981141573 (586h32m34.852224619s) [seg_pos={0 0}, upload=514181]: Done begin upload segment. Dec 30 02:01:34.981176610 (586h32m34.852259645s) [noise=true , address=45.91.116.14:4192 , piece_id=OTPCOTLXEUGG6XGVYHZ2TI6RMPRPC5T3LMSBAZN37Q6MXTT22BCA, seg_pos={0 0}, upload=514181]: Uploading piece... Dec 30 02:01:34.981267323 (586h32m34.852350358s) [noise=true , address=77.90.31.13:4627 , piece_id=45IL2UTDXZKGFPNYOGH327WK2QFU7VNUYUFYACXDTEL22D2IWVSQ, seg_pos={0 0}, upload=514181]: Uploading piece... Dec 30 02:01:34.982134023 (586h32m34.853217068s) [noise=true , address=145.79.227.49:4339 , piece_id=GFPJ6JB23S5EBAGVCRQZKB7YAQVR3CTYIM6NN3NDCQJ4UJDWXDOA, seg_pos={0 0}, upload=514181]: Uploading piece... Dec 30 02:01:34.982569923 (586h32m34.853652968s) [noise=true , address=145.79.227.39:4014 , piece_id=ZTW6Y6X4RIAZKMZ3YGFJATRKDIGFAIABI6ESVEE3NBX4QTMJZLSQ, seg_pos={0 0}, upload=514181]: Uploading piece... Dec 30 02:01:34.982988961 (586h32m34.854071996s) [noise=true , address=145.79.227.7:4294 , piece_id=O77ASC66G5CHZE65SVWIDU644VEEUJVTQGVAZLHRP433SQKXWHRA, seg_pos={0 0}, upload=514181]: Uploading piece... Dec 30 02:01:34.983411956 (586h32m34.854495001s) [noise=true , address=77.90.31.10:4518 , piece_id=GOFOU2R5ZCM4QUZTBZ2XK2GEY3K6OCMHZVGNZ25TBTPBVB7LBCMA, seg_pos={0 0}, upload=514181]: Uploading piece... Dec 30 02:01:34.983848298 (586h32m34.854931333s) [noise=true , address=145.79.227.6:4259 , piece_id=S2O5VVGU57AEJJYGZG65HSZE45KPFMBJLY44OYSCYPQX4IL5XMWA, seg_pos={0 0}, upload=514181]: Uploading piece... Dec 30 02:01:34.984300238 (586h32m34.855383273s) [noise=true , address=45.91.116.6:4974 , piece_id=SRVYRQHGEJGRES4R5SSWLNZPRLJH7MMFN57INMGKNCRHATWEW75Q, seg_pos={0 0}, upload=514181]: Uploading piece... Dec 30 02:01:34.986365901 (586h32m34.857448936s) [noise=true , address=77.90.31.21:4880 , piece_id=GQ4P5ZEMX6IRPSXY5KGEYKLLX7KE73AL6HX2RWLMPZDZUIG4FH2Q, seg_pos={0 0}, upload=514181]: Uploading piece... Dec 30 02:01:34.986600188 (586h32m34.857683233s) [noise=true , address=145.79.227.16:4909 , piece_id=TRAUAVWRA2NHHCPPI6A3E4LSQ7P2U2CFGAC45XXUQZPEZUPNVH5Q, seg_pos={0 0}, upload=514181]: Uploading piece... Dec 30 02:01:34.988272051 (586h32m34.859355096s) [noise=true , address=145.79.227.29:4942 , piece_id=ZR72MSGGRQSWNKJKPRHYHPQ7OO2CTTSWRO7P6D34WHG4FGCGRLUQ, seg_pos={0 0}, upload=514181]: Uploading piece... Dec 30 02:01:34.989959885 (586h32m34.86104292s) [noise=true , address=77.90.31.16:4736 , piece_id=4RZTFN67VVUNF7ISNGAW6V4LKIG5HQX3VNHR6HL6XITQ4FM3U6OA, seg_pos={0 0}, upload=514181]: Uploading piece... Dec 30 02:01:34.992105891 (586h32m34.863188926s) [noise=true , address=45.91.116.10:4082 , piece_id=MPVJJ5AX75I4GHBWKG5D6TSWN5NI3C47YTISVSMS2HZSU6HFAKUA, seg_pos={0 0}, upload=514181]: Uploading piece... Dec 30 02:01:34.994081443 (586h32m34.865164489s) [noise=true , address=145.79.227.8:4331 , piece_id=PFO4K25EEQUE43YCOJXHBJLJ22ASFHYLBPLYZ4K37U7AMCTLIFNQ, seg_pos={0 0}, upload=514181]: Uploading piece... Dec 30 02:01:34.995799935 (586h32m34.86688298s) [noise=true , address=145.79.227.13:4803 , piece_id=DP6WODS2DTSCZRQDF73YEP3RAUE5F7JPBWETTTD5XDDEHDDA6FZA, seg_pos={0 0}, upload=514181]: Uploading piece... Dec 30 02:01:34.997462351 (586h32m34.868545386s) [noise=true , address=145.79.227.42:4121 , piece_id=TZSX3VWVFIPHKNDKVU44URH2FOSJ3YJTDKWH5QA2QZTJIF4DM7HQ, seg_pos={0 0}, upload=514181]: Uploading piece... Dec 30 02:01:36.687205603 (586h32m36.558288648s) [upload=514181]: Next returned nil segment Dec 30 02:01:36.687221724 (586h32m36.55830476s) [upload=514181]: Waiting for error group managing segments... Dec 30 02:01:36.966447461 (586h32m36.837530506s) [noise=true , address=145.79.227.39:4014 , piece_id=ZTW6Y6X4RIAZKMZ3YGFJATRKDIGFAIABI6ESVEE3NBX4QTMJZLSQ, seg_pos={0 0}, upload=514181]: Done uploading piece. err: <nil> Dec 30 02:01:36.969718400 (586h32m36.840801445s) [noise=true , address=145.79.227.7:4294 , piece_id=O77ASC66G5CHZE65SVWIDU644VEEUJVTQGVAZLHRP433SQKXWHRA, seg_pos={0 0}, upload=514181]: Done uploading piece. err: <nil> Dec 30 02:01:36.975611662 (586h32m36.846694707s) [noise=true , address=145.79.227.16:4909 , piece_id=TRAUAVWRA2NHHCPPI6A3E4LSQ7P2U2CFGAC45XXUQZPEZUPNVH5Q, seg_pos={0 0}, upload=514181]: Done uploading piece. err: <nil> Dec 30 02:01:36.980547260 (586h32m36.851630295s) [noise=true , address=145.79.227.6:4259 , piece_id=S2O5VVGU57AEJJYGZG65HSZE45KPFMBJLY44OYSCYPQX4IL5XMWA, seg_pos={0 0}, upload=514181]: Done uploading piece. err: <nil> Dec 30 02:01:36.985363039 (586h32m36.856446084s) [noise=true , address=145.79.227.13:4803 , piece_id=DP6WODS2DTSCZRQDF73YEP3RAUE5F7JPBWETTTD5XDDEHDDA6FZA, seg_pos={0 0}, upload=514181]: Done uploading piece. err: <nil> Dec 30 02:01:37.005192448 (586h32m36.876275483s) [noise=true , address=145.79.227.42:4121 , piece_id=TZSX3VWVFIPHKNDKVU44URH2FOSJ3YJTDKWH5QA2QZTJIF4DM7HQ, seg_pos={0 0}, upload=514181]: Done uploading piece. err: <nil> Dec 30 02:01:37.163718616 (586h32m37.034801661s) [noise=true , address=77.90.31.21:4880 , piece_id=GQ4P5ZEMX6IRPSXY5KGEYKLLX7KE73AL6HX2RWLMPZDZUIG4FH2Q, seg_pos={0 0}, upload=514181]: Done uploading piece. err: <nil> Dec 30 02:01:37.218883988 (586h32m37.089967023s) [noise=true , address=45.91.116.6:4974 , piece_id=SRVYRQHGEJGRES4R5SSWLNZPRLJH7MMFN57INMGKNCRHATWEW75Q, seg_pos={0 0}, upload=514181]: Done uploading piece. err: <nil> Dec 30 02:01:37.255428304 (586h32m37.126511339s) [noise=true , address=77.90.31.13:4627 , piece_id=45IL2UTDXZKGFPNYOGH327WK2QFU7VNUYUFYACXDTEL22D2IWVSQ, seg_pos={0 0}, upload=514181]: Done uploading piece. err: <nil> Dec 30 02:01:37.285705948 (586h32m37.156788983s) [noise=true , address=45.91.116.10:4082 , piece_id=MPVJJ5AX75I4GHBWKG5D6TSWN5NI3C47YTISVSMS2HZSU6HFAKUA, seg_pos={0 0}, upload=514181]: Done uploading piece. err: <nil> Dec 30 02:01:37.328629280 (586h32m37.199712335s) [noise=true , address=145.79.227.49:4339 , piece_id=GFPJ6JB23S5EBAGVCRQZKB7YAQVR3CTYIM6NN3NDCQJ4UJDWXDOA, seg_pos={0 0}, upload=514181]: Done uploading piece. err: <nil> Dec 30 02:01:37.385540347 (586h32m37.256623392s) [noise=true , address=45.91.116.14:4192 , piece_id=OTPCOTLXEUGG6XGVYHZ2TI6RMPRPC5T3LMSBAZN37Q6MXTT22BCA, seg_pos={0 0}, upload=514181]: Done uploading piece. err: <nil> Dec 30 02:01:37.852405909 (586h32m37.723488944s) [noise=true , address=77.90.31.10:4518 , piece_id=GOFOU2R5ZCM4QUZTBZ2XK2GEY3K6OCMHZVGNZ25TBTPBVB7LBCMA, seg_pos={0 0}, upload=514181]: Done uploading piece. err: <nil> Dec 30 02:01:37.852424484 (586h32m37.723507519s) [seg_pos={0 0}, upload=514181]: Segment reached optimal threshold of 13 pieces. Dec 30 02:01:37.852634875 (586h32m37.72371792s) [noise=true , address=145.79.227.29:4942 , piece_id=ZR72MSGGRQSWNKJKPRHYHPQ7OO2CTTSWRO7P6D34WHG4FGCGRLUQ, seg_pos={0 0}, upload=514181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7): protocol: expected piece hash; context canceled; EOF Dec 30 02:01:37.852641488 (586h32m37.723724523s) [noise=true , address=145.79.227.29:4942 , piece_id=ZR72MSGGRQSWNKJKPRHYHPQ7OO2CTTSWRO7P6D34WHG4FGCGRLUQ, seg_pos={0 0}, upload=514181]: Long tail cancellation due to optimal threshold Dec 30 02:01:37.852867298 (586h32m37.723950343s) [noise=true , address=145.79.227.8:4331 , piece_id=PFO4K25EEQUE43YCOJXHBJLJ22ASFHYLBPLYZ4K37U7AMCTLIFNQ, seg_pos={0 0}, upload=514181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: write tcp 188.166.225.170:59952->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:59952->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59952->145.79.227.8:4331: use of closed network connection Dec 30 02:01:37.852873239 (586h32m37.723956274s) [noise=true , address=145.79.227.8:4331 , piece_id=PFO4K25EEQUE43YCOJXHBJLJ22ASFHYLBPLYZ4K37U7AMCTLIFNQ, seg_pos={0 0}, upload=514181]: Long tail cancellation due to optimal threshold Dec 30 02:01:37.852925499 (586h32m37.724008534s) [noise=true , address=77.90.31.16:4736 , piece_id=4RZTFN67VVUNF7ISNGAW6V4LKIG5HQX3VNHR6HL6XITQ4FM3U6OA, seg_pos={0 0}, upload=514181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 30 02:01:37.852941509 (586h32m37.724024544s) [noise=true , address=77.90.31.16:4736 , piece_id=4RZTFN67VVUNF7ISNGAW6V4LKIG5HQX3VNHR6HL6XITQ4FM3U6OA, seg_pos={0 0}, upload=514181]: Long tail cancellation due to optimal threshold Dec 30 02:01:37.852975083 (586h32m37.724058118s) [seg_pos={0 0}, upload=514181]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 02:01:37.853062810 (586h32m37.724145845s) [upload=514181]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 02:01:37.853068361 (586h32m37.724151396s) [upload=514181]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 02:01:38.144036537 (586h32m38.015119592s) [upload=514181]: Done uploading segments... Dec 30 02:01:38.144066043 (586h32m38.015149098s) [upload=514181]: Upload finished. err: <nil> Duration: 5.933245716s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............................................X............] duration: 3.494302254s trace: https://jaeger.dev.storj.io/trace/1ac298e20b765731 FAIL (2025-12-30 02:03:56 UTC) orbiter warning: context deadline exceeded Duration: 3.494342229s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.....XX.....X.X......X.X..XX.X....X..XXX..XX.....X..X...XX..] duration: 12.391134994s trace: https://jaeger.dev.storj.io/trace/532641d678d45a38 FAIL (2025-12-30 02:03:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 30 02:01:34.732123837 (586h32m34.603206882s) [upload=514181]: Starting upload Dec 30 02:01:34.732788423 (586h32m34.603871458s) [upload=514181]: Done starting upload Dec 30 02:01:34.733156514 (586h32m34.604239559s) [upload=514181]: Uploading segments... Dec 30 02:01:34.733346355 (586h32m34.6044294s) [upload=514181]: Got next segment. Inline: false Dec 30 02:01:34.733399156 (586h32m34.604482201s) [upload=514181]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 02:01:34.733405598 (586h32m34.604488633s) [upload=514181]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 02:01:34.981089223 (586h32m34.852172268s) [seg_pos={0 0}, upload=514181]: Begin upload segment... Dec 30 02:01:34.981141573 (586h32m34.852224619s) [seg_pos={0 0}, upload=514181]: Done begin upload segment. Dec 30 02:01:34.981176610 (586h32m34.852259645s) [noise=true , address=45.91.116.14:4192 , piece_id=OTPCOTLXEUGG6XGVYHZ2TI6RMPRPC5T3LMSBAZN37Q6MXTT22BCA, seg_pos={0 0}, upload=514181]: Uploading piece... Dec 30 02:01:34.981267323 (586h32m34.852350358s) [noise=true , address=77.90.31.13:4627 , piece_id=45IL2UTDXZKGFPNYOGH327WK2QFU7VNUYUFYACXDTEL22D2IWVSQ, seg_pos={0 0}, upload=514181]: Uploading piece... Dec 30 02:01:34.982134023 (586h32m34.853217068s) [noise=true , address=145.79.227.49:4339 , piece_id=GFPJ6JB23S5EBAGVCRQZKB7YAQVR3CTYIM6NN3NDCQJ4UJDWXDOA, seg_pos={0 0}, upload=514181]: Uploading piece... Dec 30 02:01:34.982569923 (586h32m34.853652968s) [noise=true , address=145.79.227.39:4014 , piece_id=ZTW6Y6X4RIAZKMZ3YGFJATRKDIGFAIABI6ESVEE3NBX4QTMJZLSQ, seg_pos={0 0}, upload=514181]: Uploading piece... Dec 30 02:01:34.982988961 (586h32m34.854071996s) [noise=true , address=145.79.227.7:4294 , piece_id=O77ASC66G5CHZE65SVWIDU644VEEUJVTQGVAZLHRP433SQKXWHRA, seg_pos={0 0}, upload=514181]: Uploading piece... Dec 30 02:01:34.983411956 (586h32m34.854495001s) [noise=true , address=77.90.31.10:4518 , piece_id=GOFOU2R5ZCM4QUZTBZ2XK2GEY3K6OCMHZVGNZ25TBTPBVB7LBCMA, seg_pos={0 0}, upload=514181]: Uploading piece... Dec 30 02:01:34.983848298 (586h32m34.854931333s) [noise=true , address=145.79.227.6:4259 , piece_id=S2O5VVGU57AEJJYGZG65HSZE45KPFMBJLY44OYSCYPQX4IL5XMWA, seg_pos={0 0}, upload=514181]: Uploading piece... Dec 30 02:01:34.984300238 (586h32m34.855383273s) [noise=true , address=45.91.116.6:4974 , piece_id=SRVYRQHGEJGRES4R5SSWLNZPRLJH7MMFN57INMGKNCRHATWEW75Q, seg_pos={0 0}, upload=514181]: Uploading piece... Dec 30 02:01:34.986365901 (586h32m34.857448936s) [noise=true , address=77.90.31.21:4880 , piece_id=GQ4P5ZEMX6IRPSXY5KGEYKLLX7KE73AL6HX2RWLMPZDZUIG4FH2Q, seg_pos={0 0}, upload=514181]: Uploading piece... Dec 30 02:01:34.986600188 (586h32m34.857683233s) [noise=true , address=145.79.227.16:4909 , piece_id=TRAUAVWRA2NHHCPPI6A3E4LSQ7P2U2CFGAC45XXUQZPEZUPNVH5Q, seg_pos={0 0}, upload=514181]: Uploading piece... Dec 30 02:01:34.988272051 (586h32m34.859355096s) [noise=true , address=145.79.227.29:4942 , piece_id=ZR72MSGGRQSWNKJKPRHYHPQ7OO2CTTSWRO7P6D34WHG4FGCGRLUQ, seg_pos={0 0}, upload=514181]: Uploading piece... Dec 30 02:01:34.989959885 (586h32m34.86104292s) [noise=true , address=77.90.31.16:4736 , piece_id=4RZTFN67VVUNF7ISNGAW6V4LKIG5HQX3VNHR6HL6XITQ4FM3U6OA, seg_pos={0 0}, upload=514181]: Uploading piece... Dec 30 02:01:34.992105891 (586h32m34.863188926s) [noise=true , address=45.91.116.10:4082 , piece_id=MPVJJ5AX75I4GHBWKG5D6TSWN5NI3C47YTISVSMS2HZSU6HFAKUA, seg_pos={0 0}, upload=514181]: Uploading piece... Dec 30 02:01:34.994081443 (586h32m34.865164489s) [noise=true , address=145.79.227.8:4331 , piece_id=PFO4K25EEQUE43YCOJXHBJLJ22ASFHYLBPLYZ4K37U7AMCTLIFNQ, seg_pos={0 0}, upload=514181]: Uploading piece... Dec 30 02:01:34.995799935 (586h32m34.86688298s) [noise=true , address=145.79.227.13:4803 , piece_id=DP6WODS2DTSCZRQDF73YEP3RAUE5F7JPBWETTTD5XDDEHDDA6FZA, seg_pos={0 0}, upload=514181]: Uploading piece... Dec 30 02:01:34.997462351 (586h32m34.868545386s) [noise=true , address=145.79.227.42:4121 , piece_id=TZSX3VWVFIPHKNDKVU44URH2FOSJ3YJTDKWH5QA2QZTJIF4DM7HQ, seg_pos={0 0}, upload=514181]: Uploading piece... Dec 30 02:01:36.687205603 (586h32m36.558288648s) [upload=514181]: Next returned nil segment Dec 30 02:01:36.687221724 (586h32m36.55830476s) [upload=514181]: Waiting for error group managing segments... Dec 30 02:01:36.966447461 (586h32m36.837530506s) [noise=true , address=145.79.227.39:4014 , piece_id=ZTW6Y6X4RIAZKMZ3YGFJATRKDIGFAIABI6ESVEE3NBX4QTMJZLSQ, seg_pos={0 0}, upload=514181]: Done uploading piece. err: <nil> Dec 30 02:01:36.969718400 (586h32m36.840801445s) [noise=true , address=145.79.227.7:4294 , piece_id=O77ASC66G5CHZE65SVWIDU644VEEUJVTQGVAZLHRP433SQKXWHRA, seg_pos={0 0}, upload=514181]: Done uploading piece. err: <nil> Dec 30 02:01:36.975611662 (586h32m36.846694707s) [noise=true , address=145.79.227.16:4909 , piece_id=TRAUAVWRA2NHHCPPI6A3E4LSQ7P2U2CFGAC45XXUQZPEZUPNVH5Q, seg_pos={0 0}, upload=514181]: Done uploading piece. err: <nil> Dec 30 02:01:36.980547260 (586h32m36.851630295s) [noise=true , address=145.79.227.6:4259 , piece_id=S2O5VVGU57AEJJYGZG65HSZE45KPFMBJLY44OYSCYPQX4IL5XMWA, seg_pos={0 0}, upload=514181]: Done uploading piece. err: <nil> Dec 30 02:01:36.985363039 (586h32m36.856446084s) [noise=true , address=145.79.227.13:4803 , piece_id=DP6WODS2DTSCZRQDF73YEP3RAUE5F7JPBWETTTD5XDDEHDDA6FZA, seg_pos={0 0}, upload=514181]: Done uploading piece. err: <nil> Dec 30 02:01:37.005192448 (586h32m36.876275483s) [noise=true , address=145.79.227.42:4121 , piece_id=TZSX3VWVFIPHKNDKVU44URH2FOSJ3YJTDKWH5QA2QZTJIF4DM7HQ, seg_pos={0 0}, upload=514181]: Done uploading piece. err: <nil> Dec 30 02:01:37.163718616 (586h32m37.034801661s) [noise=true , address=77.90.31.21:4880 , piece_id=GQ4P5ZEMX6IRPSXY5KGEYKLLX7KE73AL6HX2RWLMPZDZUIG4FH2Q, seg_pos={0 0}, upload=514181]: Done uploading piece. err: <nil> Dec 30 02:01:37.218883988 (586h32m37.089967023s) [noise=true , address=45.91.116.6:4974 , piece_id=SRVYRQHGEJGRES4R5SSWLNZPRLJH7MMFN57INMGKNCRHATWEW75Q, seg_pos={0 0}, upload=514181]: Done uploading piece. err: <nil> Dec 30 02:01:37.255428304 (586h32m37.126511339s) [noise=true , address=77.90.31.13:4627 , piece_id=45IL2UTDXZKGFPNYOGH327WK2QFU7VNUYUFYACXDTEL22D2IWVSQ, seg_pos={0 0}, upload=514181]: Done uploading piece. err: <nil> Dec 30 02:01:37.285705948 (586h32m37.156788983s) [noise=true , address=45.91.116.10:4082 , piece_id=MPVJJ5AX75I4GHBWKG5D6TSWN5NI3C47YTISVSMS2HZSU6HFAKUA, seg_pos={0 0}, upload=514181]: Done uploading piece. err: <nil> Dec 30 02:01:37.328629280 (586h32m37.199712335s) [noise=true , address=145.79.227.49:4339 , piece_id=GFPJ6JB23S5EBAGVCRQZKB7YAQVR3CTYIM6NN3NDCQJ4UJDWXDOA, seg_pos={0 0}, upload=514181]: Done uploading piece. err: <nil> Dec 30 02:01:37.385540347 (586h32m37.256623392s) [noise=true , address=45.91.116.14:4192 , piece_id=OTPCOTLXEUGG6XGVYHZ2TI6RMPRPC5T3LMSBAZN37Q6MXTT22BCA, seg_pos={0 0}, upload=514181]: Done uploading piece. err: <nil> Dec 30 02:01:37.852405909 (586h32m37.723488944s) [noise=true , address=77.90.31.10:4518 , piece_id=GOFOU2R5ZCM4QUZTBZ2XK2GEY3K6OCMHZVGNZ25TBTPBVB7LBCMA, seg_pos={0 0}, upload=514181]: Done uploading piece. err: <nil> Dec 30 02:01:37.852424484 (586h32m37.723507519s) [seg_pos={0 0}, upload=514181]: Segment reached optimal threshold of 13 pieces. Dec 30 02:01:37.852634875 (586h32m37.72371792s) [noise=true , address=145.79.227.29:4942 , piece_id=ZR72MSGGRQSWNKJKPRHYHPQ7OO2CTTSWRO7P6D34WHG4FGCGRLUQ, seg_pos={0 0}, upload=514181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7): protocol: expected piece hash; context canceled; EOF Dec 30 02:01:37.852641488 (586h32m37.723724523s) [noise=true , address=145.79.227.29:4942 , piece_id=ZR72MSGGRQSWNKJKPRHYHPQ7OO2CTTSWRO7P6D34WHG4FGCGRLUQ, seg_pos={0 0}, upload=514181]: Long tail cancellation due to optimal threshold Dec 30 02:01:37.852867298 (586h32m37.723950343s) [noise=true , address=145.79.227.8:4331 , piece_id=PFO4K25EEQUE43YCOJXHBJLJ22ASFHYLBPLYZ4K37U7AMCTLIFNQ, seg_pos={0 0}, upload=514181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: write tcp 188.166.225.170:59952->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:59952->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59952->145.79.227.8:4331: use of closed network connection Dec 30 02:01:37.852873239 (586h32m37.723956274s) [noise=true , address=145.79.227.8:4331 , piece_id=PFO4K25EEQUE43YCOJXHBJLJ22ASFHYLBPLYZ4K37U7AMCTLIFNQ, seg_pos={0 0}, upload=514181]: Long tail cancellation due to optimal threshold Dec 30 02:01:37.852925499 (586h32m37.724008534s) [noise=true , address=77.90.31.16:4736 , piece_id=4RZTFN67VVUNF7ISNGAW6V4LKIG5HQX3VNHR6HL6XITQ4FM3U6OA, seg_pos={0 0}, upload=514181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 30 02:01:37.852941509 (586h32m37.724024544s) [noise=true , address=77.90.31.16:4736 , piece_id=4RZTFN67VVUNF7ISNGAW6V4LKIG5HQX3VNHR6HL6XITQ4FM3U6OA, seg_pos={0 0}, upload=514181]: Long tail cancellation due to optimal threshold Dec 30 02:01:37.852975083 (586h32m37.724058118s) [seg_pos={0 0}, upload=514181]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 02:01:37.853062810 (586h32m37.724145845s) [upload=514181]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 02:01:37.853068361 (586h32m37.724151396s) [upload=514181]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 02:01:38.144036537 (586h32m38.015119592s) [upload=514181]: Done uploading segments... Dec 30 02:01:38.144066043 (586h32m38.015149098s) [upload=514181]: Upload finished. err: <nil> Duration: 12.391164571s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................................................X...........] duration: 3.521156757s trace: https://jaeger.dev.storj.io/trace/1ac298e20b765731 FAIL (2025-12-30 02:03:22 UTC) orbiter warning: context deadline exceeded Duration: 3.521182958s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [......XX.....X.X......X.X..XX.X....X..XXX..XX.....X..X...XX.] duration: 5.993634958s trace: https://jaeger.dev.storj.io/trace/532641d678d45a38 FAIL (2025-12-30 02:02:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 30 02:01:34.732123837 (586h32m34.603206882s) [upload=514181]: Starting upload Dec 30 02:01:34.732788423 (586h32m34.603871458s) [upload=514181]: Done starting upload Dec 30 02:01:34.733156514 (586h32m34.604239559s) [upload=514181]: Uploading segments... Dec 30 02:01:34.733346355 (586h32m34.6044294s) [upload=514181]: Got next segment. Inline: false Dec 30 02:01:34.733399156 (586h32m34.604482201s) [upload=514181]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 02:01:34.733405598 (586h32m34.604488633s) [upload=514181]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 02:01:34.981089223 (586h32m34.852172268s) [seg_pos={0 0}, upload=514181]: Begin upload segment... Dec 30 02:01:34.981141573 (586h32m34.852224619s) [seg_pos={0 0}, upload=514181]: Done begin upload segment. Dec 30 02:01:34.981176610 (586h32m34.852259645s) [noise=true , address=45.91.116.14:4192 , piece_id=OTPCOTLXEUGG6XGVYHZ2TI6RMPRPC5T3LMSBAZN37Q6MXTT22BCA, seg_pos={0 0}, upload=514181]: Uploading piece... Dec 30 02:01:34.981267323 (586h32m34.852350358s) [noise=true , address=77.90.31.13:4627 , piece_id=45IL2UTDXZKGFPNYOGH327WK2QFU7VNUYUFYACXDTEL22D2IWVSQ, seg_pos={0 0}, upload=514181]: Uploading piece... Dec 30 02:01:34.982134023 (586h32m34.853217068s) [noise=true , address=145.79.227.49:4339 , piece_id=GFPJ6JB23S5EBAGVCRQZKB7YAQVR3CTYIM6NN3NDCQJ4UJDWXDOA, seg_pos={0 0}, upload=514181]: Uploading piece... Dec 30 02:01:34.982569923 (586h32m34.853652968s) [noise=true , address=145.79.227.39:4014 , piece_id=ZTW6Y6X4RIAZKMZ3YGFJATRKDIGFAIABI6ESVEE3NBX4QTMJZLSQ, seg_pos={0 0}, upload=514181]: Uploading piece... Dec 30 02:01:34.982988961 (586h32m34.854071996s) [noise=true , address=145.79.227.7:4294 , piece_id=O77ASC66G5CHZE65SVWIDU644VEEUJVTQGVAZLHRP433SQKXWHRA, seg_pos={0 0}, upload=514181]: Uploading piece... Dec 30 02:01:34.983411956 (586h32m34.854495001s) [noise=true , address=77.90.31.10:4518 , piece_id=GOFOU2R5ZCM4QUZTBZ2XK2GEY3K6OCMHZVGNZ25TBTPBVB7LBCMA, seg_pos={0 0}, upload=514181]: Uploading piece... Dec 30 02:01:34.983848298 (586h32m34.854931333s) [noise=true , address=145.79.227.6:4259 , piece_id=S2O5VVGU57AEJJYGZG65HSZE45KPFMBJLY44OYSCYPQX4IL5XMWA, seg_pos={0 0}, upload=514181]: Uploading piece... Dec 30 02:01:34.984300238 (586h32m34.855383273s) [noise=true , address=45.91.116.6:4974 , piece_id=SRVYRQHGEJGRES4R5SSWLNZPRLJH7MMFN57INMGKNCRHATWEW75Q, seg_pos={0 0}, upload=514181]: Uploading piece... Dec 30 02:01:34.986365901 (586h32m34.857448936s) [noise=true , address=77.90.31.21:4880 , piece_id=GQ4P5ZEMX6IRPSXY5KGEYKLLX7KE73AL6HX2RWLMPZDZUIG4FH2Q, seg_pos={0 0}, upload=514181]: Uploading piece... Dec 30 02:01:34.986600188 (586h32m34.857683233s) [noise=true , address=145.79.227.16:4909 , piece_id=TRAUAVWRA2NHHCPPI6A3E4LSQ7P2U2CFGAC45XXUQZPEZUPNVH5Q, seg_pos={0 0}, upload=514181]: Uploading piece... Dec 30 02:01:34.988272051 (586h32m34.859355096s) [noise=true , address=145.79.227.29:4942 , piece_id=ZR72MSGGRQSWNKJKPRHYHPQ7OO2CTTSWRO7P6D34WHG4FGCGRLUQ, seg_pos={0 0}, upload=514181]: Uploading piece... Dec 30 02:01:34.989959885 (586h32m34.86104292s) [noise=true , address=77.90.31.16:4736 , piece_id=4RZTFN67VVUNF7ISNGAW6V4LKIG5HQX3VNHR6HL6XITQ4FM3U6OA, seg_pos={0 0}, upload=514181]: Uploading piece... Dec 30 02:01:34.992105891 (586h32m34.863188926s) [noise=true , address=45.91.116.10:4082 , piece_id=MPVJJ5AX75I4GHBWKG5D6TSWN5NI3C47YTISVSMS2HZSU6HFAKUA, seg_pos={0 0}, upload=514181]: Uploading piece... Dec 30 02:01:34.994081443 (586h32m34.865164489s) [noise=true , address=145.79.227.8:4331 , piece_id=PFO4K25EEQUE43YCOJXHBJLJ22ASFHYLBPLYZ4K37U7AMCTLIFNQ, seg_pos={0 0}, upload=514181]: Uploading piece... Dec 30 02:01:34.995799935 (586h32m34.86688298s) [noise=true , address=145.79.227.13:4803 , piece_id=DP6WODS2DTSCZRQDF73YEP3RAUE5F7JPBWETTTD5XDDEHDDA6FZA, seg_pos={0 0}, upload=514181]: Uploading piece... Dec 30 02:01:34.997462351 (586h32m34.868545386s) [noise=true , address=145.79.227.42:4121 , piece_id=TZSX3VWVFIPHKNDKVU44URH2FOSJ3YJTDKWH5QA2QZTJIF4DM7HQ, seg_pos={0 0}, upload=514181]: Uploading piece... Dec 30 02:01:36.687205603 (586h32m36.558288648s) [upload=514181]: Next returned nil segment Dec 30 02:01:36.687221724 (586h32m36.55830476s) [upload=514181]: Waiting for error group managing segments... Dec 30 02:01:36.966447461 (586h32m36.837530506s) [noise=true , address=145.79.227.39:4014 , piece_id=ZTW6Y6X4RIAZKMZ3YGFJATRKDIGFAIABI6ESVEE3NBX4QTMJZLSQ, seg_pos={0 0}, upload=514181]: Done uploading piece. err: <nil> Dec 30 02:01:36.969718400 (586h32m36.840801445s) [noise=true , address=145.79.227.7:4294 , piece_id=O77ASC66G5CHZE65SVWIDU644VEEUJVTQGVAZLHRP433SQKXWHRA, seg_pos={0 0}, upload=514181]: Done uploading piece. err: <nil> Dec 30 02:01:36.975611662 (586h32m36.846694707s) [noise=true , address=145.79.227.16:4909 , piece_id=TRAUAVWRA2NHHCPPI6A3E4LSQ7P2U2CFGAC45XXUQZPEZUPNVH5Q, seg_pos={0 0}, upload=514181]: Done uploading piece. err: <nil> Dec 30 02:01:36.980547260 (586h32m36.851630295s) [noise=true , address=145.79.227.6:4259 , piece_id=S2O5VVGU57AEJJYGZG65HSZE45KPFMBJLY44OYSCYPQX4IL5XMWA, seg_pos={0 0}, upload=514181]: Done uploading piece. err: <nil> Dec 30 02:01:36.985363039 (586h32m36.856446084s) [noise=true , address=145.79.227.13:4803 , piece_id=DP6WODS2DTSCZRQDF73YEP3RAUE5F7JPBWETTTD5XDDEHDDA6FZA, seg_pos={0 0}, upload=514181]: Done uploading piece. err: <nil> Dec 30 02:01:37.005192448 (586h32m36.876275483s) [noise=true , address=145.79.227.42:4121 , piece_id=TZSX3VWVFIPHKNDKVU44URH2FOSJ3YJTDKWH5QA2QZTJIF4DM7HQ, seg_pos={0 0}, upload=514181]: Done uploading piece. err: <nil> Dec 30 02:01:37.163718616 (586h32m37.034801661s) [noise=true , address=77.90.31.21:4880 , piece_id=GQ4P5ZEMX6IRPSXY5KGEYKLLX7KE73AL6HX2RWLMPZDZUIG4FH2Q, seg_pos={0 0}, upload=514181]: Done uploading piece. err: <nil> Dec 30 02:01:37.218883988 (586h32m37.089967023s) [noise=true , address=45.91.116.6:4974 , piece_id=SRVYRQHGEJGRES4R5SSWLNZPRLJH7MMFN57INMGKNCRHATWEW75Q, seg_pos={0 0}, upload=514181]: Done uploading piece. err: <nil> Dec 30 02:01:37.255428304 (586h32m37.126511339s) [noise=true , address=77.90.31.13:4627 , piece_id=45IL2UTDXZKGFPNYOGH327WK2QFU7VNUYUFYACXDTEL22D2IWVSQ, seg_pos={0 0}, upload=514181]: Done uploading piece. err: <nil> Dec 30 02:01:37.285705948 (586h32m37.156788983s) [noise=true , address=45.91.116.10:4082 , piece_id=MPVJJ5AX75I4GHBWKG5D6TSWN5NI3C47YTISVSMS2HZSU6HFAKUA, seg_pos={0 0}, upload=514181]: Done uploading piece. err: <nil> Dec 30 02:01:37.328629280 (586h32m37.199712335s) [noise=true , address=145.79.227.49:4339 , piece_id=GFPJ6JB23S5EBAGVCRQZKB7YAQVR3CTYIM6NN3NDCQJ4UJDWXDOA, seg_pos={0 0}, upload=514181]: Done uploading piece. err: <nil> Dec 30 02:01:37.385540347 (586h32m37.256623392s) [noise=true , address=45.91.116.14:4192 , piece_id=OTPCOTLXEUGG6XGVYHZ2TI6RMPRPC5T3LMSBAZN37Q6MXTT22BCA, seg_pos={0 0}, upload=514181]: Done uploading piece. err: <nil> Dec 30 02:01:37.852405909 (586h32m37.723488944s) [noise=true , address=77.90.31.10:4518 , piece_id=GOFOU2R5ZCM4QUZTBZ2XK2GEY3K6OCMHZVGNZ25TBTPBVB7LBCMA, seg_pos={0 0}, upload=514181]: Done uploading piece. err: <nil> Dec 30 02:01:37.852424484 (586h32m37.723507519s) [seg_pos={0 0}, upload=514181]: Segment reached optimal threshold of 13 pieces. Dec 30 02:01:37.852634875 (586h32m37.72371792s) [noise=true , address=145.79.227.29:4942 , piece_id=ZR72MSGGRQSWNKJKPRHYHPQ7OO2CTTSWRO7P6D34WHG4FGCGRLUQ, seg_pos={0 0}, upload=514181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7): protocol: expected piece hash; context canceled; EOF Dec 30 02:01:37.852641488 (586h32m37.723724523s) [noise=true , address=145.79.227.29:4942 , piece_id=ZR72MSGGRQSWNKJKPRHYHPQ7OO2CTTSWRO7P6D34WHG4FGCGRLUQ, seg_pos={0 0}, upload=514181]: Long tail cancellation due to optimal threshold Dec 30 02:01:37.852867298 (586h32m37.723950343s) [noise=true , address=145.79.227.8:4331 , piece_id=PFO4K25EEQUE43YCOJXHBJLJ22ASFHYLBPLYZ4K37U7AMCTLIFNQ, seg_pos={0 0}, upload=514181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: write tcp 188.166.225.170:59952->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:59952->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59952->145.79.227.8:4331: use of closed network connection Dec 30 02:01:37.852873239 (586h32m37.723956274s) [noise=true , address=145.79.227.8:4331 , piece_id=PFO4K25EEQUE43YCOJXHBJLJ22ASFHYLBPLYZ4K37U7AMCTLIFNQ, seg_pos={0 0}, upload=514181]: Long tail cancellation due to optimal threshold Dec 30 02:01:37.852925499 (586h32m37.724008534s) [noise=true , address=77.90.31.16:4736 , piece_id=4RZTFN67VVUNF7ISNGAW6V4LKIG5HQX3VNHR6HL6XITQ4FM3U6OA, seg_pos={0 0}, upload=514181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 30 02:01:37.852941509 (586h32m37.724024544s) [noise=true , address=77.90.31.16:4736 , piece_id=4RZTFN67VVUNF7ISNGAW6V4LKIG5HQX3VNHR6HL6XITQ4FM3U6OA, seg_pos={0 0}, upload=514181]: Long tail cancellation due to optimal threshold Dec 30 02:01:37.852975083 (586h32m37.724058118s) [seg_pos={0 0}, upload=514181]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 02:01:37.853062810 (586h32m37.724145845s) [upload=514181]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 02:01:37.853068361 (586h32m37.724151396s) [upload=514181]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 02:01:38.144036537 (586h32m38.015119592s) [upload=514181]: Done uploading segments... Dec 30 02:01:38.144066043 (586h32m38.015149098s) [upload=514181]: Upload finished. err: <nil> Duration: 5.99365181s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................................................X..........] duration: 5.001786189s trace: https://jaeger.dev.storj.io/trace/1ac298e20b765731 FAIL (2025-12-30 02:02:10 UTC) orbiter warning: context deadline exceeded Duration: 5.001842746s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.......XX.....X.X......X.X..XX.X....X..XXX..XX.....X..X...XX] duration: 30.003320001s trace: https://jaeger.dev.storj.io/trace/532641d678d45a38 FAIL (2025-12-30 02:02:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 30 02:01:34.732123837 (586h32m34.603206882s) [upload=514181]: Starting upload Dec 30 02:01:34.732788423 (586h32m34.603871458s) [upload=514181]: Done starting upload Dec 30 02:01:34.733156514 (586h32m34.604239559s) [upload=514181]: Uploading segments... Dec 30 02:01:34.733346355 (586h32m34.6044294s) [upload=514181]: Got next segment. Inline: false Dec 30 02:01:34.733399156 (586h32m34.604482201s) [upload=514181]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 02:01:34.733405598 (586h32m34.604488633s) [upload=514181]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 02:01:34.981089223 (586h32m34.852172268s) [seg_pos={0 0}, upload=514181]: Begin upload segment... Dec 30 02:01:34.981141573 (586h32m34.852224619s) [seg_pos={0 0}, upload=514181]: Done begin upload segment. Dec 30 02:01:34.981176610 (586h32m34.852259645s) [noise=true , address=45.91.116.14:4192 , piece_id=OTPCOTLXEUGG6XGVYHZ2TI6RMPRPC5T3LMSBAZN37Q6MXTT22BCA, seg_pos={0 0}, upload=514181]: Uploading piece... Dec 30 02:01:34.981267323 (586h32m34.852350358s) [noise=true , address=77.90.31.13:4627 , piece_id=45IL2UTDXZKGFPNYOGH327WK2QFU7VNUYUFYACXDTEL22D2IWVSQ, seg_pos={0 0}, upload=514181]: Uploading piece... Dec 30 02:01:34.982134023 (586h32m34.853217068s) [noise=true , address=145.79.227.49:4339 , piece_id=GFPJ6JB23S5EBAGVCRQZKB7YAQVR3CTYIM6NN3NDCQJ4UJDWXDOA, seg_pos={0 0}, upload=514181]: Uploading piece... Dec 30 02:01:34.982569923 (586h32m34.853652968s) [noise=true , address=145.79.227.39:4014 , piece_id=ZTW6Y6X4RIAZKMZ3YGFJATRKDIGFAIABI6ESVEE3NBX4QTMJZLSQ, seg_pos={0 0}, upload=514181]: Uploading piece... Dec 30 02:01:34.982988961 (586h32m34.854071996s) [noise=true , address=145.79.227.7:4294 , piece_id=O77ASC66G5CHZE65SVWIDU644VEEUJVTQGVAZLHRP433SQKXWHRA, seg_pos={0 0}, upload=514181]: Uploading piece... Dec 30 02:01:34.983411956 (586h32m34.854495001s) [noise=true , address=77.90.31.10:4518 , piece_id=GOFOU2R5ZCM4QUZTBZ2XK2GEY3K6OCMHZVGNZ25TBTPBVB7LBCMA, seg_pos={0 0}, upload=514181]: Uploading piece... Dec 30 02:01:34.983848298 (586h32m34.854931333s) [noise=true , address=145.79.227.6:4259 , piece_id=S2O5VVGU57AEJJYGZG65HSZE45KPFMBJLY44OYSCYPQX4IL5XMWA, seg_pos={0 0}, upload=514181]: Uploading piece... Dec 30 02:01:34.984300238 (586h32m34.855383273s) [noise=true , address=45.91.116.6:4974 , piece_id=SRVYRQHGEJGRES4R5SSWLNZPRLJH7MMFN57INMGKNCRHATWEW75Q, seg_pos={0 0}, upload=514181]: Uploading piece... Dec 30 02:01:34.986365901 (586h32m34.857448936s) [noise=true , address=77.90.31.21:4880 , piece_id=GQ4P5ZEMX6IRPSXY5KGEYKLLX7KE73AL6HX2RWLMPZDZUIG4FH2Q, seg_pos={0 0}, upload=514181]: Uploading piece... Dec 30 02:01:34.986600188 (586h32m34.857683233s) [noise=true , address=145.79.227.16:4909 , piece_id=TRAUAVWRA2NHHCPPI6A3E4LSQ7P2U2CFGAC45XXUQZPEZUPNVH5Q, seg_pos={0 0}, upload=514181]: Uploading piece... Dec 30 02:01:34.988272051 (586h32m34.859355096s) [noise=true , address=145.79.227.29:4942 , piece_id=ZR72MSGGRQSWNKJKPRHYHPQ7OO2CTTSWRO7P6D34WHG4FGCGRLUQ, seg_pos={0 0}, upload=514181]: Uploading piece... Dec 30 02:01:34.989959885 (586h32m34.86104292s) [noise=true , address=77.90.31.16:4736 , piece_id=4RZTFN67VVUNF7ISNGAW6V4LKIG5HQX3VNHR6HL6XITQ4FM3U6OA, seg_pos={0 0}, upload=514181]: Uploading piece... Dec 30 02:01:34.992105891 (586h32m34.863188926s) [noise=true , address=45.91.116.10:4082 , piece_id=MPVJJ5AX75I4GHBWKG5D6TSWN5NI3C47YTISVSMS2HZSU6HFAKUA, seg_pos={0 0}, upload=514181]: Uploading piece... Dec 30 02:01:34.994081443 (586h32m34.865164489s) [noise=true , address=145.79.227.8:4331 , piece_id=PFO4K25EEQUE43YCOJXHBJLJ22ASFHYLBPLYZ4K37U7AMCTLIFNQ, seg_pos={0 0}, upload=514181]: Uploading piece... Dec 30 02:01:34.995799935 (586h32m34.86688298s) [noise=true , address=145.79.227.13:4803 , piece_id=DP6WODS2DTSCZRQDF73YEP3RAUE5F7JPBWETTTD5XDDEHDDA6FZA, seg_pos={0 0}, upload=514181]: Uploading piece... Dec 30 02:01:34.997462351 (586h32m34.868545386s) [noise=true , address=145.79.227.42:4121 , piece_id=TZSX3VWVFIPHKNDKVU44URH2FOSJ3YJTDKWH5QA2QZTJIF4DM7HQ, seg_pos={0 0}, upload=514181]: Uploading piece... Dec 30 02:01:36.687205603 (586h32m36.558288648s) [upload=514181]: Next returned nil segment Dec 30 02:01:36.687221724 (586h32m36.55830476s) [upload=514181]: Waiting for error group managing segments... Dec 30 02:01:36.966447461 (586h32m36.837530506s) [noise=true , address=145.79.227.39:4014 , piece_id=ZTW6Y6X4RIAZKMZ3YGFJATRKDIGFAIABI6ESVEE3NBX4QTMJZLSQ, seg_pos={0 0}, upload=514181]: Done uploading piece. err: <nil> Dec 30 02:01:36.969718400 (586h32m36.840801445s) [noise=true , address=145.79.227.7:4294 , piece_id=O77ASC66G5CHZE65SVWIDU644VEEUJVTQGVAZLHRP433SQKXWHRA, seg_pos={0 0}, upload=514181]: Done uploading piece. err: <nil> Dec 30 02:01:36.975611662 (586h32m36.846694707s) [noise=true , address=145.79.227.16:4909 , piece_id=TRAUAVWRA2NHHCPPI6A3E4LSQ7P2U2CFGAC45XXUQZPEZUPNVH5Q, seg_pos={0 0}, upload=514181]: Done uploading piece. err: <nil> Dec 30 02:01:36.980547260 (586h32m36.851630295s) [noise=true , address=145.79.227.6:4259 , piece_id=S2O5VVGU57AEJJYGZG65HSZE45KPFMBJLY44OYSCYPQX4IL5XMWA, seg_pos={0 0}, upload=514181]: Done uploading piece. err: <nil> Dec 30 02:01:36.985363039 (586h32m36.856446084s) [noise=true , address=145.79.227.13:4803 , piece_id=DP6WODS2DTSCZRQDF73YEP3RAUE5F7JPBWETTTD5XDDEHDDA6FZA, seg_pos={0 0}, upload=514181]: Done uploading piece. err: <nil> Dec 30 02:01:37.005192448 (586h32m36.876275483s) [noise=true , address=145.79.227.42:4121 , piece_id=TZSX3VWVFIPHKNDKVU44URH2FOSJ3YJTDKWH5QA2QZTJIF4DM7HQ, seg_pos={0 0}, upload=514181]: Done uploading piece. err: <nil> Dec 30 02:01:37.163718616 (586h32m37.034801661s) [noise=true , address=77.90.31.21:4880 , piece_id=GQ4P5ZEMX6IRPSXY5KGEYKLLX7KE73AL6HX2RWLMPZDZUIG4FH2Q, seg_pos={0 0}, upload=514181]: Done uploading piece. err: <nil> Dec 30 02:01:37.218883988 (586h32m37.089967023s) [noise=true , address=45.91.116.6:4974 , piece_id=SRVYRQHGEJGRES4R5SSWLNZPRLJH7MMFN57INMGKNCRHATWEW75Q, seg_pos={0 0}, upload=514181]: Done uploading piece. err: <nil> Dec 30 02:01:37.255428304 (586h32m37.126511339s) [noise=true , address=77.90.31.13:4627 , piece_id=45IL2UTDXZKGFPNYOGH327WK2QFU7VNUYUFYACXDTEL22D2IWVSQ, seg_pos={0 0}, upload=514181]: Done uploading piece. err: <nil> Dec 30 02:01:37.285705948 (586h32m37.156788983s) [noise=true , address=45.91.116.10:4082 , piece_id=MPVJJ5AX75I4GHBWKG5D6TSWN5NI3C47YTISVSMS2HZSU6HFAKUA, seg_pos={0 0}, upload=514181]: Done uploading piece. err: <nil> Dec 30 02:01:37.328629280 (586h32m37.199712335s) [noise=true , address=145.79.227.49:4339 , piece_id=GFPJ6JB23S5EBAGVCRQZKB7YAQVR3CTYIM6NN3NDCQJ4UJDWXDOA, seg_pos={0 0}, upload=514181]: Done uploading piece. err: <nil> Dec 30 02:01:37.385540347 (586h32m37.256623392s) [noise=true , address=45.91.116.14:4192 , piece_id=OTPCOTLXEUGG6XGVYHZ2TI6RMPRPC5T3LMSBAZN37Q6MXTT22BCA, seg_pos={0 0}, upload=514181]: Done uploading piece. err: <nil> Dec 30 02:01:37.852405909 (586h32m37.723488944s) [noise=true , address=77.90.31.10:4518 , piece_id=GOFOU2R5ZCM4QUZTBZ2XK2GEY3K6OCMHZVGNZ25TBTPBVB7LBCMA, seg_pos={0 0}, upload=514181]: Done uploading piece. err: <nil> Dec 30 02:01:37.852424484 (586h32m37.723507519s) [seg_pos={0 0}, upload=514181]: Segment reached optimal threshold of 13 pieces. Dec 30 02:01:37.852634875 (586h32m37.72371792s) [noise=true , address=145.79.227.29:4942 , piece_id=ZR72MSGGRQSWNKJKPRHYHPQ7OO2CTTSWRO7P6D34WHG4FGCGRLUQ, seg_pos={0 0}, upload=514181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7): protocol: expected piece hash; context canceled; EOF Dec 30 02:01:37.852641488 (586h32m37.723724523s) [noise=true , address=145.79.227.29:4942 , piece_id=ZR72MSGGRQSWNKJKPRHYHPQ7OO2CTTSWRO7P6D34WHG4FGCGRLUQ, seg_pos={0 0}, upload=514181]: Long tail cancellation due to optimal threshold Dec 30 02:01:37.852867298 (586h32m37.723950343s) [noise=true , address=145.79.227.8:4331 , piece_id=PFO4K25EEQUE43YCOJXHBJLJ22ASFHYLBPLYZ4K37U7AMCTLIFNQ, seg_pos={0 0}, upload=514181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: write tcp 188.166.225.170:59952->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:59952->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59952->145.79.227.8:4331: use of closed network connection Dec 30 02:01:37.852873239 (586h32m37.723956274s) [noise=true , address=145.79.227.8:4331 , piece_id=PFO4K25EEQUE43YCOJXHBJLJ22ASFHYLBPLYZ4K37U7AMCTLIFNQ, seg_pos={0 0}, upload=514181]: Long tail cancellation due to optimal threshold Dec 30 02:01:37.852925499 (586h32m37.724008534s) [noise=true , address=77.90.31.16:4736 , piece_id=4RZTFN67VVUNF7ISNGAW6V4LKIG5HQX3VNHR6HL6XITQ4FM3U6OA, seg_pos={0 0}, upload=514181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 30 02:01:37.852941509 (586h32m37.724024544s) [noise=true , address=77.90.31.16:4736 , piece_id=4RZTFN67VVUNF7ISNGAW6V4LKIG5HQX3VNHR6HL6XITQ4FM3U6OA, seg_pos={0 0}, upload=514181]: Long tail cancellation due to optimal threshold Dec 30 02:01:37.852975083 (586h32m37.724058118s) [seg_pos={0 0}, upload=514181]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 02:01:37.853062810 (586h32m37.724145845s) [upload=514181]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 02:01:37.853068361 (586h32m37.724151396s) [upload=514181]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 02:01:38.144036537 (586h32m38.015119592s) [upload=514181]: Done uploading segments... Dec 30 02:01:38.144066043 (586h32m38.015149098s) [upload=514181]: Upload finished. err: <nil> Duration: 30.003339278s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 27.607056233s trace: https://jaeger.dev.storj.io/trace/58b94359c097ef0d FAIL (2025-12-30 02:01:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.221968961 (585h44m42.093052017s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222491837 (585h44m42.093574882s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222877261 (585h44m42.093960306s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223249098 (585h44m42.094332143s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223637930 (585h44m42.094720965s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224032200 (585h44m42.095115245s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224444876 (585h44m42.095527921s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224905332 (585h44m42.095988378s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225388433 (585h44m42.096471478s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225981221 (585h44m42.097064256s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227130389 (585h44m42.098213435s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227655670 (585h44m42.098738705s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228177764 (585h44m42.099260799s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228673828 (585h44m42.099756863s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:43.678892099 (585h44m43.549975134s) [upload=513482]: Next returned nil segment Dec 30 01:13:43.678910774 (585h44m43.549993819s) [upload=513482]: Waiting for error group managing segments... Dec 30 01:13:44.076462870 (585h44m43.947545905s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.083306310 (585h44m43.954389345s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.107556817 (585h44m43.978639852s) [noise=true , address=145.79.227.9:4391 , piece_id=4AVX4RQJ65H3VL7U3MW6MUYHGRK6YS7GOGZO4XVVG6YYTHFHAXDQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.147374754 (585h44m44.018457799s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.212624668 (585h44m44.083707703s) [noise=true , address=64.49.13.170:4721 , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.237525922 (585h44m44.108608957s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.242637575 (585h44m44.11372063s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.283510060 (585h44m44.154593095s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.340441716 (585h44m44.211524761s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.408662737 (585h44m44.279745772s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.962802591 (585h44m44.833885626s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.963549243 (585h44m44.834632288s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.017193407 (585h44m44.888276442s) [noise=true , address=145.79.227.44:4218 , piece_id=UBAPWL2SMR7FJ2AHQ5XHMX3H6NKDFZ23HEDFRMJLJFYZKGC4X4MA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093476639 (585h44m44.964559684s) [noise=true , address=145.79.227.49:4370 , piece_id=RZK3XQONM7S2XSOIO4EXITQ3XQGVYLIJNK66RI6LL5NMFG46NOPA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093501687 (585h44m44.964584722s) [seg_pos={0 0}, upload=513482]: Segment reached optimal threshold of 14 pieces. Dec 30 01:13:45.093670669 (585h44m44.964753704s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection Dec 30 01:13:45.093677973 (585h44m44.964761008s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093727678 (585h44m44.964810713s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093735993 (585h44m44.964819028s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093827067 (585h44m44.964910122s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UP8PxqeK7t24PXjopH1nziHYQFnsxhkamSHz8dmwcbuDxkAF6): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093833129 (585h44m44.964916174s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093916176 (585h44m44.964999211s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qZC1tmCvCadnm1fYZxR3kc4fonZUJPzvfTKkTFiDtccW5bo9T): protocol: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection Dec 30 01:13:45.093921096 (585h44m44.965004141s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093966583 (585h44m44.965049618s) [seg_pos={0 0}, upload=513482]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 01:13:45.094031926 (585h44m44.965114961s) [upload=513482]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:13:45.094036336 (585h44m44.965119371s) [upload=513482]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:13:45.389410141 (585h44m45.260493176s) [upload=513482]: Done uploading segments... Dec 30 01:13:45.389430891 (585h44m45.260513926s) [upload=513482]: Upload finished. err: <nil> Duration: 27.607084318s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................................................X.........] duration: 3.37621194s trace: https://jaeger.dev.storj.io/trace/1ac298e20b765731 FAIL (2025-12-30 02:01:36 UTC) orbiter warning: context deadline exceeded Duration: 3.376238199s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 15.382063196s trace: https://jaeger.dev.storj.io/trace/58b94359c097ef0d FAIL (2025-12-30 02:01:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.221968961 (585h44m42.093052017s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222491837 (585h44m42.093574882s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222877261 (585h44m42.093960306s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223249098 (585h44m42.094332143s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223637930 (585h44m42.094720965s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224032200 (585h44m42.095115245s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224444876 (585h44m42.095527921s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224905332 (585h44m42.095988378s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225388433 (585h44m42.096471478s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225981221 (585h44m42.097064256s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227130389 (585h44m42.098213435s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227655670 (585h44m42.098738705s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228177764 (585h44m42.099260799s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228673828 (585h44m42.099756863s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:43.678892099 (585h44m43.549975134s) [upload=513482]: Next returned nil segment Dec 30 01:13:43.678910774 (585h44m43.549993819s) [upload=513482]: Waiting for error group managing segments... Dec 30 01:13:44.076462870 (585h44m43.947545905s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.083306310 (585h44m43.954389345s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.107556817 (585h44m43.978639852s) [noise=true , address=145.79.227.9:4391 , piece_id=4AVX4RQJ65H3VL7U3MW6MUYHGRK6YS7GOGZO4XVVG6YYTHFHAXDQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.147374754 (585h44m44.018457799s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.212624668 (585h44m44.083707703s) [noise=true , address=64.49.13.170:4721 , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.237525922 (585h44m44.108608957s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.242637575 (585h44m44.11372063s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.283510060 (585h44m44.154593095s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.340441716 (585h44m44.211524761s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.408662737 (585h44m44.279745772s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.962802591 (585h44m44.833885626s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.963549243 (585h44m44.834632288s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.017193407 (585h44m44.888276442s) [noise=true , address=145.79.227.44:4218 , piece_id=UBAPWL2SMR7FJ2AHQ5XHMX3H6NKDFZ23HEDFRMJLJFYZKGC4X4MA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093476639 (585h44m44.964559684s) [noise=true , address=145.79.227.49:4370 , piece_id=RZK3XQONM7S2XSOIO4EXITQ3XQGVYLIJNK66RI6LL5NMFG46NOPA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093501687 (585h44m44.964584722s) [seg_pos={0 0}, upload=513482]: Segment reached optimal threshold of 14 pieces. Dec 30 01:13:45.093670669 (585h44m44.964753704s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection Dec 30 01:13:45.093677973 (585h44m44.964761008s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093727678 (585h44m44.964810713s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093735993 (585h44m44.964819028s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093827067 (585h44m44.964910122s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UP8PxqeK7t24PXjopH1nziHYQFnsxhkamSHz8dmwcbuDxkAF6): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093833129 (585h44m44.964916174s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093916176 (585h44m44.964999211s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qZC1tmCvCadnm1fYZxR3kc4fonZUJPzvfTKkTFiDtccW5bo9T): protocol: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection Dec 30 01:13:45.093921096 (585h44m44.965004141s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093966583 (585h44m44.965049618s) [seg_pos={0 0}, upload=513482]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 01:13:45.094031926 (585h44m44.965114961s) [upload=513482]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:13:45.094036336 (585h44m44.965119371s) [upload=513482]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:13:45.389410141 (585h44m45.260493176s) [upload=513482]: Done uploading segments... Dec 30 01:13:45.389430891 (585h44m45.260513926s) [upload=513482]: Upload finished. err: <nil> Duration: 15.382092121s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [........XX.....X.X......X.X..XX.X....X..XXX..XX.....X..X...X] duration: 30.002317771s trace: https://jaeger.dev.storj.io/trace/13a1f3cc2c38f2dd FAIL (2025-12-30 02:00:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: NZ5Z53XSM33QMRLM4RLLIWJIOGQG4GGSMFGBXCGO346GCZ23MNTQ) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: NCVLE6D2PUBBZ5GT6LF4YQSW2PTZXN762CMTQ3UBDNAX2UYZMSSQ) context deadline exceeded ======LOGS====== Dec 30 02:00:27.121144935 (586h31m26.99222797s) [upload=514163]: Starting upload Dec 30 02:00:27.121204078 (586h31m26.992287133s) [upload=514163]: Done starting upload Dec 30 02:00:27.121347591 (586h31m26.992430636s) [upload=514163]: Uploading segments... Dec 30 02:00:27.121507164 (586h31m26.99259021s) [upload=514163]: Got next segment. Inline: false Dec 30 02:00:27.121577348 (586h31m26.992660393s) [upload=514163]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 02:00:27.121584523 (586h31m26.992667558s) [upload=514163]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 02:00:27.369191142 (586h31m27.240274187s) [seg_pos={0 0}, upload=514163]: Begin upload segment... Dec 30 02:00:27.369249042 (586h31m27.240332077s) [seg_pos={0 0}, upload=514163]: Done begin upload segment. Dec 30 02:00:27.369267127 (586h31m27.240350172s) [noise=true , address=145.79.227.26:4653 , piece_id=ZBIQFEF3V7SJ6IFGRIPIHBVXAE6MYODI6D5CTYQGJA4SZJXGF5LA, seg_pos={0 0}, upload=514163]: Uploading piece... Dec 30 02:00:27.369282816 (586h31m27.240365861s) [noise=true , address=145.79.227.21:4475 , piece_id=7ANUD2JYGO3K4XKPRUDVMGNPAWFDSYQSJXGW4RUPJHF26BYVMCAQ, seg_pos={0 0}, upload=514163]: Uploading piece... Dec 30 02:00:27.370312227 (586h31m27.241395272s) [noise=true , address=145.79.227.8:4331 , piece_id=BPWR4GLNDIEAFBPLUSXOCLJRQ7RAWA74KQDKJ2M75FTGPGYJTI4Q, seg_pos={0 0}, upload=514163]: Uploading piece... Dec 30 02:00:27.370672242 (586h31m27.241755277s) [noise=true , address=145.79.227.15:4875 , piece_id=DXBZ6UYFOHLD7DWSF5F3N6TMUM6UA5CAY24WYEC5ISGVYUOB6BSA, seg_pos={0 0}, upload=514163]: Uploading piece... Dec 30 02:00:27.371010577 (586h31m27.242093612s) [noise=true , address=45.91.116.12:4225 , piece_id=NZ5Z53XSM33QMRLM4RLLIWJIOGQG4GGSMFGBXCGO346GCZ23MNTQ, seg_pos={0 0}, upload=514163]: Uploading piece... Dec 30 02:00:27.371277816 (586h31m27.242360861s) [noise=true , address=45.91.116.9:4264 , piece_id=S6OEHJVOJYQ3MSIOSJBCWBDXI2NY5B6YHPS5TSMNXEYBNK23K6VQ, seg_pos={0 0}, upload=514163]: Uploading piece... Dec 30 02:00:27.371682706 (586h31m27.242765741s) [noise=true , address=45.91.116.7:4009 , piece_id=NCVLE6D2PUBBZ5GT6LF4YQSW2PTZXN762CMTQ3UBDNAX2UYZMSSQ, seg_pos={0 0}, upload=514163]: Uploading piece... Dec 30 02:00:27.372603099 (586h31m27.243686144s) [noise=true , address=145.79.227.52:4445 , piece_id=ATD37F3AWTC6DT3GB46E3D7FVWKNYFF5LXEYSSHSTIRSPF3BFQCQ, seg_pos={0 0}, upload=514163]: Uploading piece... Dec 30 02:00:27.373143758 (586h31m27.244226803s) [noise=true , address=145.79.227.51:4410 , piece_id=MHBZSHX7XSQHPWFLXVSCAGMHJ3U7KHEF5UB5RPKOH762GF4RLYJQ, seg_pos={0 0}, upload=514163]: Uploading piece... Dec 30 02:00:27.373596320 (586h31m27.244679365s) [noise=true , address=145.79.227.41:4085 , piece_id=B72Y5POTAZPIFNPV67QAGEEUTZ4KKP36UDB4E6KOKA5POI735NXA, seg_pos={0 0}, upload=514163]: Uploading piece... Dec 30 02:00:27.374438183 (586h31m27.245521228s) [noise=true , address=77.90.31.12:4589 , piece_id=WIKACZKOLGI3RS5NDNS6OHWBE5U33WMAJOTNLEFMBEYFXJGPKWSA, seg_pos={0 0}, upload=514163]: Uploading piece... Dec 30 02:00:27.376333252 (586h31m27.247416287s) [noise=true , address=145.79.227.12:4765 , piece_id=O5NUKEJCUFX3S2Q5R4KNMO722SFLTAGTYBKS6CX5USHYPINQF6NA, seg_pos={0 0}, upload=514163]: Uploading piece... Dec 30 02:00:27.377923369 (586h31m27.249006404s) [noise=true , address=145.79.227.22:4509 , piece_id=SDZF6CQZQUF7EUU6MZFOKDKNOENHWSXDONS2GPONCUXL25FNK4UA, seg_pos={0 0}, upload=514163]: Uploading piece... Dec 30 02:00:27.379401134 (586h31m27.250484179s) [noise=true , address=77.90.31.11:4553 , piece_id=USWJSGWWTXB2K27ELZIWWC7BQ7AH2VLHTDMZVTQPTZXQQ75ZXL4A, seg_pos={0 0}, upload=514163]: Uploading piece... Dec 30 02:00:27.380755903 (586h31m27.251838969s) [noise=true , address=45.91.116.13:4153 , piece_id=FMGF4Y4KXG5JHWOTQLAS3KELNAGRSEPCSFOX47T3M7OBMG2D7IPA, seg_pos={0 0}, upload=514163]: Uploading piece... Dec 30 02:00:27.381960717 (586h31m27.253043762s) [noise=true , address=145.79.227.6:4258 , piece_id=5BAG4VACZHNC7AURO55EG4DHYFSZJHVXIT4ISOOVRK2YEX2FLNJQ, seg_pos={0 0}, upload=514163]: Uploading piece... Dec 30 02:00:29.070000637 (586h31m28.941083692s) [upload=514163]: Next returned nil segment Dec 30 02:00:29.070018772 (586h31m28.941101817s) [upload=514163]: Waiting for error group managing segments... Dec 30 02:00:29.353709200 (586h31m29.224792245s) [noise=true , address=145.79.227.26:4653 , piece_id=ZBIQFEF3V7SJ6IFGRIPIHBVXAE6MYODI6D5CTYQGJA4SZJXGF5LA, seg_pos={0 0}, upload=514163]: Done uploading piece. err: <nil> Dec 30 02:00:29.362761837 (586h31m29.233844882s) [noise=true , address=145.79.227.15:4875 , piece_id=DXBZ6UYFOHLD7DWSF5F3N6TMUM6UA5CAY24WYEC5ISGVYUOB6BSA, seg_pos={0 0}, upload=514163]: Done uploading piece. err: <nil> Dec 30 02:00:29.363543887 (586h31m29.234626932s) [noise=true , address=145.79.227.52:4445 , piece_id=ATD37F3AWTC6DT3GB46E3D7FVWKNYFF5LXEYSSHSTIRSPF3BFQCQ, seg_pos={0 0}, upload=514163]: Done uploading piece. err: <nil> Dec 30 02:00:29.365498359 (586h31m29.236581404s) [noise=true , address=145.79.227.41:4085 , piece_id=B72Y5POTAZPIFNPV67QAGEEUTZ4KKP36UDB4E6KOKA5POI735NXA, seg_pos={0 0}, upload=514163]: Done uploading piece. err: <nil> Dec 30 02:00:29.366946255 (586h31m29.23802931s) [noise=true , address=145.79.227.21:4475 , piece_id=7ANUD2JYGO3K4XKPRUDVMGNPAWFDSYQSJXGW4RUPJHF26BYVMCAQ, seg_pos={0 0}, upload=514163]: Done uploading piece. err: <nil> Dec 30 02:00:29.372921363 (586h31m29.244004398s) [noise=true , address=145.79.227.22:4509 , piece_id=SDZF6CQZQUF7EUU6MZFOKDKNOENHWSXDONS2GPONCUXL25FNK4UA, seg_pos={0 0}, upload=514163]: Done uploading piece. err: <nil> Dec 30 02:00:29.377071385 (586h31m29.24815443s) [noise=true , address=145.79.227.12:4765 , piece_id=O5NUKEJCUFX3S2Q5R4KNMO722SFLTAGTYBKS6CX5USHYPINQF6NA, seg_pos={0 0}, upload=514163]: Done uploading piece. err: <nil> Dec 30 02:00:29.382527014 (586h31m29.253610059s) [noise=true , address=145.79.227.8:4331 , piece_id=BPWR4GLNDIEAFBPLUSXOCLJRQ7RAWA74KQDKJ2M75FTGPGYJTI4Q, seg_pos={0 0}, upload=514163]: Done uploading piece. err: <nil> Dec 30 02:00:29.386289939 (586h31m29.257372984s) [noise=true , address=145.79.227.6:4258 , piece_id=5BAG4VACZHNC7AURO55EG4DHYFSZJHVXIT4ISOOVRK2YEX2FLNJQ, seg_pos={0 0}, upload=514163]: Done uploading piece. err: <nil> Dec 30 02:00:29.387884625 (586h31m29.25896767s) [noise=true , address=145.79.227.51:4410 , piece_id=MHBZSHX7XSQHPWFLXVSCAGMHJ3U7KHEF5UB5RPKOH762GF4RLYJQ, seg_pos={0 0}, upload=514163]: Done uploading piece. err: <nil> Dec 30 02:00:29.624601139 (586h31m29.495684184s) [noise=true , address=45.91.116.7:4009 , piece_id=NCVLE6D2PUBBZ5GT6LF4YQSW2PTZXN762CMTQ3UBDNAX2UYZMSSQ, seg_pos={0 0}, upload=514163]: Done uploading piece. err: <nil> Dec 30 02:00:29.726749294 (586h31m29.597832329s) [noise=true , address=45.91.116.9:4264 , piece_id=S6OEHJVOJYQ3MSIOSJBCWBDXI2NY5B6YHPS5TSMNXEYBNK23K6VQ, seg_pos={0 0}, upload=514163]: Done uploading piece. err: <nil> Dec 30 02:00:29.754987585 (586h31m29.62607063s) [noise=true , address=45.91.116.12:4225 , piece_id=NZ5Z53XSM33QMRLM4RLLIWJIOGQG4GGSMFGBXCGO346GCZ23MNTQ, seg_pos={0 0}, upload=514163]: Done uploading piece. err: <nil> Dec 30 02:00:29.755006981 (586h31m29.626090016s) [seg_pos={0 0}, upload=514163]: Segment reached optimal threshold of 13 pieces. Dec 30 02:00:29.755200680 (586h31m29.626283725s) [noise=true , address=45.91.116.13:4153 , piece_id=FMGF4Y4KXG5JHWOTQLAS3KELNAGRSEPCSFOX47T3M7OBMG2D7IPA, seg_pos={0 0}, upload=514163]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 30 02:00:29.755209837 (586h31m29.626292872s) [noise=true , address=45.91.116.13:4153 , piece_id=FMGF4Y4KXG5JHWOTQLAS3KELNAGRSEPCSFOX47T3M7OBMG2D7IPA, seg_pos={0 0}, upload=514163]: Long tail cancellation due to optimal threshold Dec 30 02:00:29.755319817 (586h31m29.626402852s) [noise=true , address=77.90.31.11:4553 , piece_id=USWJSGWWTXB2K27ELZIWWC7BQ7AH2VLHTDMZVTQPTZXQQ75ZXL4A, seg_pos={0 0}, upload=514163]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 30 02:00:29.755333874 (586h31m29.626416919s) [noise=true , address=77.90.31.11:4553 , piece_id=USWJSGWWTXB2K27ELZIWWC7BQ7AH2VLHTDMZVTQPTZXQQ75ZXL4A, seg_pos={0 0}, upload=514163]: Long tail cancellation due to optimal threshold Dec 30 02:00:29.755439886 (586h31m29.626522931s) [noise=true , address=77.90.31.12:4589 , piece_id=WIKACZKOLGI3RS5NDNS6OHWBE5U33WMAJOTNLEFMBEYFXJGPKWSA, seg_pos={0 0}, upload=514163]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: write tcp 188.166.225.170:59474->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:59474->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59474->77.90.31.12:4589: use of closed network connection Dec 30 02:00:29.755447119 (586h31m29.626530164s) [noise=true , address=77.90.31.12:4589 , piece_id=WIKACZKOLGI3RS5NDNS6OHWBE5U33WMAJOTNLEFMBEYFXJGPKWSA, seg_pos={0 0}, upload=514163]: Long tail cancellation due to optimal threshold Dec 30 02:00:29.755470745 (586h31m29.62655379s) [seg_pos={0 0}, upload=514163]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 02:00:29.755559303 (586h31m29.626642348s) [upload=514163]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 02:00:29.755564452 (586h31m29.626647498s) [upload=514163]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 02:00:30.037469812 (586h31m29.908552847s) [upload=514163]: Done uploading segments... Dec 30 02:00:30.037516040 (586h31m29.908599085s) [upload=514163]: Upload finished. err: <nil> Duration: 30.002341406s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................................................X........] duration: 3.452042633s trace: https://jaeger.dev.storj.io/trace/1ac298e20b765731 FAIL (2025-12-30 02:00:40 UTC) orbiter warning: context deadline exceeded Duration: 3.452065907s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 15.295865999s trace: https://jaeger.dev.storj.io/trace/58b94359c097ef0d FAIL (2025-12-30 02:00:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.221968961 (585h44m42.093052017s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222491837 (585h44m42.093574882s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222877261 (585h44m42.093960306s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223249098 (585h44m42.094332143s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223637930 (585h44m42.094720965s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224032200 (585h44m42.095115245s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224444876 (585h44m42.095527921s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224905332 (585h44m42.095988378s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225388433 (585h44m42.096471478s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225981221 (585h44m42.097064256s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227130389 (585h44m42.098213435s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227655670 (585h44m42.098738705s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228177764 (585h44m42.099260799s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228673828 (585h44m42.099756863s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:43.678892099 (585h44m43.549975134s) [upload=513482]: Next returned nil segment Dec 30 01:13:43.678910774 (585h44m43.549993819s) [upload=513482]: Waiting for error group managing segments... Dec 30 01:13:44.076462870 (585h44m43.947545905s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.083306310 (585h44m43.954389345s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.107556817 (585h44m43.978639852s) [noise=true , address=145.79.227.9:4391 , piece_id=4AVX4RQJ65H3VL7U3MW6MUYHGRK6YS7GOGZO4XVVG6YYTHFHAXDQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.147374754 (585h44m44.018457799s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.212624668 (585h44m44.083707703s) [noise=true , address=64.49.13.170:4721 , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.237525922 (585h44m44.108608957s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.242637575 (585h44m44.11372063s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.283510060 (585h44m44.154593095s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.340441716 (585h44m44.211524761s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.408662737 (585h44m44.279745772s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.962802591 (585h44m44.833885626s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.963549243 (585h44m44.834632288s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.017193407 (585h44m44.888276442s) [noise=true , address=145.79.227.44:4218 , piece_id=UBAPWL2SMR7FJ2AHQ5XHMX3H6NKDFZ23HEDFRMJLJFYZKGC4X4MA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093476639 (585h44m44.964559684s) [noise=true , address=145.79.227.49:4370 , piece_id=RZK3XQONM7S2XSOIO4EXITQ3XQGVYLIJNK66RI6LL5NMFG46NOPA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093501687 (585h44m44.964584722s) [seg_pos={0 0}, upload=513482]: Segment reached optimal threshold of 14 pieces. Dec 30 01:13:45.093670669 (585h44m44.964753704s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection Dec 30 01:13:45.093677973 (585h44m44.964761008s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093727678 (585h44m44.964810713s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093735993 (585h44m44.964819028s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093827067 (585h44m44.964910122s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UP8PxqeK7t24PXjopH1nziHYQFnsxhkamSHz8dmwcbuDxkAF6): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093833129 (585h44m44.964916174s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093916176 (585h44m44.964999211s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qZC1tmCvCadnm1fYZxR3kc4fonZUJPzvfTKkTFiDtccW5bo9T): protocol: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection Dec 30 01:13:45.093921096 (585h44m44.965004141s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093966583 (585h44m44.965049618s) [seg_pos={0 0}, upload=513482]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 01:13:45.094031926 (585h44m44.965114961s) [upload=513482]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:13:45.094036336 (585h44m44.965119371s) [upload=513482]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:13:45.389410141 (585h44m45.260493176s) [upload=513482]: Done uploading segments... Dec 30 01:13:45.389430891 (585h44m45.260513926s) [upload=513482]: Upload finished. err: <nil> Duration: 15.295911285s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 15.346397175s trace: https://jaeger.dev.storj.io/trace/58b94359c097ef0d FAIL (2025-12-30 01:59:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.221968961 (585h44m42.093052017s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222491837 (585h44m42.093574882s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222877261 (585h44m42.093960306s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223249098 (585h44m42.094332143s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223637930 (585h44m42.094720965s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224032200 (585h44m42.095115245s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224444876 (585h44m42.095527921s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224905332 (585h44m42.095988378s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225388433 (585h44m42.096471478s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225981221 (585h44m42.097064256s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227130389 (585h44m42.098213435s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227655670 (585h44m42.098738705s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228177764 (585h44m42.099260799s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228673828 (585h44m42.099756863s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:43.678892099 (585h44m43.549975134s) [upload=513482]: Next returned nil segment Dec 30 01:13:43.678910774 (585h44m43.549993819s) [upload=513482]: Waiting for error group managing segments... Dec 30 01:13:44.076462870 (585h44m43.947545905s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.083306310 (585h44m43.954389345s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.107556817 (585h44m43.978639852s) [noise=true , address=145.79.227.9:4391 , piece_id=4AVX4RQJ65H3VL7U3MW6MUYHGRK6YS7GOGZO4XVVG6YYTHFHAXDQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.147374754 (585h44m44.018457799s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.212624668 (585h44m44.083707703s) [noise=true , address=64.49.13.170:4721 , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.237525922 (585h44m44.108608957s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.242637575 (585h44m44.11372063s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.283510060 (585h44m44.154593095s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.340441716 (585h44m44.211524761s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.408662737 (585h44m44.279745772s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.962802591 (585h44m44.833885626s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.963549243 (585h44m44.834632288s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.017193407 (585h44m44.888276442s) [noise=true , address=145.79.227.44:4218 , piece_id=UBAPWL2SMR7FJ2AHQ5XHMX3H6NKDFZ23HEDFRMJLJFYZKGC4X4MA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093476639 (585h44m44.964559684s) [noise=true , address=145.79.227.49:4370 , piece_id=RZK3XQONM7S2XSOIO4EXITQ3XQGVYLIJNK66RI6LL5NMFG46NOPA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093501687 (585h44m44.964584722s) [seg_pos={0 0}, upload=513482]: Segment reached optimal threshold of 14 pieces. Dec 30 01:13:45.093670669 (585h44m44.964753704s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection Dec 30 01:13:45.093677973 (585h44m44.964761008s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093727678 (585h44m44.964810713s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093735993 (585h44m44.964819028s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093827067 (585h44m44.964910122s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UP8PxqeK7t24PXjopH1nziHYQFnsxhkamSHz8dmwcbuDxkAF6): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093833129 (585h44m44.964916174s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093916176 (585h44m44.964999211s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qZC1tmCvCadnm1fYZxR3kc4fonZUJPzvfTKkTFiDtccW5bo9T): protocol: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection Dec 30 01:13:45.093921096 (585h44m44.965004141s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093966583 (585h44m44.965049618s) [seg_pos={0 0}, upload=513482]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 01:13:45.094031926 (585h44m44.965114961s) [upload=513482]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:13:45.094036336 (585h44m44.965119371s) [upload=513482]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:13:45.389410141 (585h44m45.260493176s) [upload=513482]: Done uploading segments... Dec 30 01:13:45.389430891 (585h44m45.260513926s) [upload=513482]: Upload finished. err: <nil> Duration: 15.346412794s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................................................X.......] duration: 3.460006231s trace: https://jaeger.dev.storj.io/trace/1ac298e20b765731 FAIL (2025-12-30 01:59:56 UTC) orbiter warning: context deadline exceeded Duration: 3.460027271s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 41.773262481s trace: https://jaeger.dev.storj.io/trace/58b94359c097ef0d FAIL (2025-12-30 01:59:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.221968961 (585h44m42.093052017s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222491837 (585h44m42.093574882s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222877261 (585h44m42.093960306s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223249098 (585h44m42.094332143s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223637930 (585h44m42.094720965s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224032200 (585h44m42.095115245s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224444876 (585h44m42.095527921s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224905332 (585h44m42.095988378s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225388433 (585h44m42.096471478s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225981221 (585h44m42.097064256s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227130389 (585h44m42.098213435s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227655670 (585h44m42.098738705s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228177764 (585h44m42.099260799s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228673828 (585h44m42.099756863s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:43.678892099 (585h44m43.549975134s) [upload=513482]: Next returned nil segment Dec 30 01:13:43.678910774 (585h44m43.549993819s) [upload=513482]: Waiting for error group managing segments... Dec 30 01:13:44.076462870 (585h44m43.947545905s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.083306310 (585h44m43.954389345s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.107556817 (585h44m43.978639852s) [noise=true , address=145.79.227.9:4391 , piece_id=4AVX4RQJ65H3VL7U3MW6MUYHGRK6YS7GOGZO4XVVG6YYTHFHAXDQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.147374754 (585h44m44.018457799s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.212624668 (585h44m44.083707703s) [noise=true , address=64.49.13.170:4721 , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.237525922 (585h44m44.108608957s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.242637575 (585h44m44.11372063s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.283510060 (585h44m44.154593095s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.340441716 (585h44m44.211524761s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.408662737 (585h44m44.279745772s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.962802591 (585h44m44.833885626s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.963549243 (585h44m44.834632288s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.017193407 (585h44m44.888276442s) [noise=true , address=145.79.227.44:4218 , piece_id=UBAPWL2SMR7FJ2AHQ5XHMX3H6NKDFZ23HEDFRMJLJFYZKGC4X4MA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093476639 (585h44m44.964559684s) [noise=true , address=145.79.227.49:4370 , piece_id=RZK3XQONM7S2XSOIO4EXITQ3XQGVYLIJNK66RI6LL5NMFG46NOPA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093501687 (585h44m44.964584722s) [seg_pos={0 0}, upload=513482]: Segment reached optimal threshold of 14 pieces. Dec 30 01:13:45.093670669 (585h44m44.964753704s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection Dec 30 01:13:45.093677973 (585h44m44.964761008s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093727678 (585h44m44.964810713s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093735993 (585h44m44.964819028s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093827067 (585h44m44.964910122s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UP8PxqeK7t24PXjopH1nziHYQFnsxhkamSHz8dmwcbuDxkAF6): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093833129 (585h44m44.964916174s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093916176 (585h44m44.964999211s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qZC1tmCvCadnm1fYZxR3kc4fonZUJPzvfTKkTFiDtccW5bo9T): protocol: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection Dec 30 01:13:45.093921096 (585h44m44.965004141s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093966583 (585h44m44.965049618s) [seg_pos={0 0}, upload=513482]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 01:13:45.094031926 (585h44m44.965114961s) [upload=513482]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:13:45.094036336 (585h44m44.965119371s) [upload=513482]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:13:45.389410141 (585h44m45.260493176s) [upload=513482]: Done uploading segments... Dec 30 01:13:45.389430891 (585h44m45.260513926s) [upload=513482]: Upload finished. err: <nil> Duration: 41.773291025s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.........XX.....X.X......X.X..XX.X....X..XXX..XX.....X..X...] duration: 5.718591814s trace: https://jaeger.dev.storj.io/trace/54f1aa927f806c2c FAIL (2025-12-30 01:59:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: ICKQPUWVMJHIL6SX3ANBJQDVS3KBSTFK4A7SRHNUH2T5DYVNPVNA) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: BOQ5Y3TUGH32KHSCQV73J22E4QZJT7UJ5JSO42JFEAPXEJRLIMLQ) context deadline exceeded ======LOGS====== Dec 30 01:57:11.787337529 (586h28m11.658420574s) [upload=514117]: Starting upload Dec 30 01:57:11.787437048 (586h28m11.658520103s) [upload=514117]: Done starting upload Dec 30 01:57:11.787563730 (586h28m11.658646775s) [upload=514117]: Uploading segments... Dec 30 01:57:11.787681634 (586h28m11.658764669s) [upload=514117]: Got next segment. Inline: false Dec 30 01:57:11.787724315 (586h28m11.65880736s) [upload=514117]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 01:57:11.787730297 (586h28m11.658813332s) [upload=514117]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 01:57:12.031832714 (586h28m11.902915759s) [seg_pos={0 0}, upload=514117]: Begin upload segment... Dec 30 01:57:12.031909590 (586h28m11.902992635s) [seg_pos={0 0}, upload=514117]: Done begin upload segment. Dec 30 01:57:12.032054276 (586h28m11.903137321s) [noise=true , address=145.79.227.3:4149 , piece_id=AO37XAZ6DQLWGN65KZKLL4Z7XRKJSLIHIVMCYCTVBZTND254HBTQ, seg_pos={0 0}, upload=514117]: Uploading piece... Dec 30 01:57:12.032074805 (586h28m11.90315785s) [noise=true , address=145.79.227.6:4257 , piece_id=DH3S3IDPA7I66ZRKIFOXKLN2MDBMVMAWZTJG4PYKJZWK7NA7Z4VQ, seg_pos={0 0}, upload=514117]: Uploading piece... Dec 30 01:57:12.032718500 (586h28m11.903801535s) [noise=true , address=45.91.116.11:4119 , piece_id=ICKQPUWVMJHIL6SX3ANBJQDVS3KBSTFK4A7SRHNUH2T5DYVNPVNA, seg_pos={0 0}, upload=514117]: Uploading piece... Dec 30 01:57:12.033224033 (586h28m11.904307079s) [noise=true , address=145.79.227.7:4293 , piece_id=LSRJCQLXABLYH6NPKMPXAG6TDT6A3QKG2O3BXVTEVOOXTKXCEI5Q, seg_pos={0 0}, upload=514117]: Uploading piece... Dec 30 01:57:12.033756397 (586h28m11.904839442s) [noise=true , address=145.79.227.5:4221 , piece_id=KIKI266HETJCZQZ7QA67O5E3WVFQLTAAMGJJ74COADMX5HLZ5AYQ, seg_pos={0 0}, upload=514117]: Uploading piece... Dec 30 01:57:12.034508449 (586h28m11.905591494s) [noise=true , address=145.79.227.10:4401 , piece_id=LHZNK76RWLXKNORM57WAJIKGBLMWG76XJX47PUB2ANWWVEB2A25A, seg_pos={0 0}, upload=514117]: Uploading piece... Dec 30 01:57:12.038525228 (586h28m11.909608273s) [noise=true , address=77.90.31.14:4663 , piece_id=BOQ5Y3TUGH32KHSCQV73J22E4QZJT7UJ5JSO42JFEAPXEJRLIMLQ, seg_pos={0 0}, upload=514117]: Uploading piece... Dec 30 01:57:12.043483929 (586h28m11.914567014s) [noise=true , address=145.79.227.46:4602 , piece_id=AXSPIAOPJLYJHQPZ2NL5YS2XO4OJVYZ345S4HPX6WPCP46IAZ5QA, seg_pos={0 0}, upload=514117]: Uploading piece... Dec 30 01:57:12.050045313 (586h28m11.921128378s) [noise=true , address=145.79.227.32:4836 , piece_id=W5K62R7THVPLT2BBMLKQQPJTIBICAKJOHQCKZEKF4BKJXLMIAAOA, seg_pos={0 0}, upload=514117]: Uploading piece... Dec 30 01:57:12.051946353 (586h28m11.923029398s) [noise=true , address=145.79.227.19:4077 , piece_id=JWPVZW2PNM6I3HF5PWOWRNFC6C3O6FQTALILSON54QBQ3DPOWT7A, seg_pos={0 0}, upload=514117]: Uploading piece... Dec 30 01:57:12.054919544 (586h28m11.926002579s) [noise=true , address=77.90.31.11:4555 , piece_id=E4CXSIVAPSAOX2OTSLBCQRR3A7Z27I76QOIRPOPG6YJ2WITWHR5Q, seg_pos={0 0}, upload=514117]: Uploading piece... Dec 30 01:57:12.056504032 (586h28m11.927587067s) [noise=true , address=45.91.116.12:4228 , piece_id=NMK2B4GYZXVCBZZYZSQ4EV3H5MAH6CMU2WFGIEGJDNNN5OOSMRSA, seg_pos={0 0}, upload=514117]: Uploading piece... Dec 30 01:57:12.058074632 (586h28m11.929157677s) [noise=true , address=45.91.116.8:4048 , piece_id=UTE4QMRHNQQZE7B6M6TJ5WCRP3O3SD5DEVUCVHJJS5F7LPBR4ASQ, seg_pos={0 0}, upload=514117]: Uploading piece... Dec 30 01:57:12.059216897 (586h28m11.930299932s) [noise=true , address=145.79.227.8:4329 , piece_id=GIDL5TFVOT423QPESVGKLDZ5KGE46Z4YESWOLNRANRIO4BQW74PA, seg_pos={0 0}, upload=514117]: Uploading piece... Dec 30 01:57:12.059267724 (586h28m11.930350769s) [noise=true , address=145.79.227.48:4302 , piece_id=SEWV6427NUPSS46PYFHECUHF3TJMKGRUYOAUXYYIOLBUXFRECMPQ, seg_pos={0 0}, upload=514117]: Uploading piece... Dec 30 01:57:12.061185155 (586h28m11.93226819s) [noise=true , address=145.79.227.52:4447 , piece_id=DIKHFJKTE2JXNIIG55JGLP42AHZCX6XKI4JYBRI5BQHACHI3UX6Q, seg_pos={0 0}, upload=514117]: Uploading piece... Dec 30 01:57:13.688319383 (586h28m13.559402428s) [upload=514117]: Next returned nil segment Dec 30 01:57:13.688338599 (586h28m13.559421634s) [upload=514117]: Waiting for error group managing segments... Dec 30 01:57:14.019582918 (586h28m13.890665963s) [noise=true , address=145.79.227.10:4401 , piece_id=LHZNK76RWLXKNORM57WAJIKGBLMWG76XJX47PUB2ANWWVEB2A25A, seg_pos={0 0}, upload=514117]: Done uploading piece. err: <nil> Dec 30 01:57:14.019676136 (586h28m13.890759171s) [noise=true , address=145.79.227.3:4149 , piece_id=AO37XAZ6DQLWGN65KZKLL4Z7XRKJSLIHIVMCYCTVBZTND254HBTQ, seg_pos={0 0}, upload=514117]: Done uploading piece. err: <nil> Dec 30 01:57:14.020237825 (586h28m13.89132086s) [noise=true , address=145.79.227.5:4221 , piece_id=KIKI266HETJCZQZ7QA67O5E3WVFQLTAAMGJJ74COADMX5HLZ5AYQ, seg_pos={0 0}, upload=514117]: Done uploading piece. err: <nil> Dec 30 01:57:14.035649170 (586h28m13.906732216s) [noise=true , address=145.79.227.8:4329 , piece_id=GIDL5TFVOT423QPESVGKLDZ5KGE46Z4YESWOLNRANRIO4BQW74PA, seg_pos={0 0}, upload=514117]: Done uploading piece. err: <nil> Dec 30 01:57:14.041524819 (586h28m13.912607864s) [noise=true , address=145.79.227.6:4257 , piece_id=DH3S3IDPA7I66ZRKIFOXKLN2MDBMVMAWZTJG4PYKJZWK7NA7Z4VQ, seg_pos={0 0}, upload=514117]: Done uploading piece. err: <nil> Dec 30 01:57:14.046683191 (586h28m13.917766236s) [noise=true , address=145.79.227.7:4293 , piece_id=LSRJCQLXABLYH6NPKMPXAG6TDT6A3QKG2O3BXVTEVOOXTKXCEI5Q, seg_pos={0 0}, upload=514117]: Done uploading piece. err: <nil> Dec 30 01:57:14.048287225 (586h28m13.919370271s) [noise=true , address=145.79.227.46:4602 , piece_id=AXSPIAOPJLYJHQPZ2NL5YS2XO4OJVYZ345S4HPX6WPCP46IAZ5QA, seg_pos={0 0}, upload=514117]: Done uploading piece. err: <nil> Dec 30 01:57:14.058951151 (586h28m13.930034186s) [noise=true , address=145.79.227.48:4302 , piece_id=SEWV6427NUPSS46PYFHECUHF3TJMKGRUYOAUXYYIOLBUXFRECMPQ, seg_pos={0 0}, upload=514117]: Done uploading piece. err: <nil> Dec 30 01:57:14.059101707 (586h28m13.930184753s) [noise=true , address=145.79.227.52:4447 , piece_id=DIKHFJKTE2JXNIIG55JGLP42AHZCX6XKI4JYBRI5BQHACHI3UX6Q, seg_pos={0 0}, upload=514117]: Done uploading piece. err: <nil> Dec 30 01:57:14.060655236 (586h28m13.931738281s) [noise=true , address=145.79.227.19:4077 , piece_id=JWPVZW2PNM6I3HF5PWOWRNFC6C3O6FQTALILSON54QBQ3DPOWT7A, seg_pos={0 0}, upload=514117]: Done uploading piece. err: <nil> Dec 30 01:57:14.061044076 (586h28m13.932127111s) [noise=true , address=145.79.227.32:4836 , piece_id=W5K62R7THVPLT2BBMLKQQPJTIBICAKJOHQCKZEKF4BKJXLMIAAOA, seg_pos={0 0}, upload=514117]: Done uploading piece. err: <nil> Dec 30 01:57:14.210079021 (586h28m14.081162056s) [noise=true , address=77.90.31.14:4663 , piece_id=BOQ5Y3TUGH32KHSCQV73J22E4QZJT7UJ5JSO42JFEAPXEJRLIMLQ, seg_pos={0 0}, upload=514117]: Done uploading piece. err: <nil> Dec 30 01:57:14.318394851 (586h28m14.189477896s) [noise=true , address=45.91.116.11:4119 , piece_id=ICKQPUWVMJHIL6SX3ANBJQDVS3KBSTFK4A7SRHNUH2T5DYVNPVNA, seg_pos={0 0}, upload=514117]: Done uploading piece. err: <nil> Dec 30 01:57:14.318413846 (586h28m14.189496891s) [seg_pos={0 0}, upload=514117]: Segment reached optimal threshold of 13 pieces. Dec 30 01:57:14.318615931 (586h28m14.189698966s) [noise=true , address=45.91.116.12:4228 , piece_id=NMK2B4GYZXVCBZZYZSQ4EV3H5MAH6CMU2WFGIEGJDNNN5OOSMRSA, seg_pos={0 0}, upload=514117]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 30 01:57:14.318624157 (586h28m14.189707192s) [noise=true , address=45.91.116.12:4228 , piece_id=NMK2B4GYZXVCBZZYZSQ4EV3H5MAH6CMU2WFGIEGJDNNN5OOSMRSA, seg_pos={0 0}, upload=514117]: Long tail cancellation due to optimal threshold Dec 30 01:57:14.318738245 (586h28m14.18982128s) [noise=true , address=45.91.116.8:4048 , piece_id=UTE4QMRHNQQZE7B6M6TJ5WCRP3O3SD5DEVUCVHJJS5F7LPBR4ASQ, seg_pos={0 0}, upload=514117]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 30 01:57:14.318759765 (586h28m14.1898428s) [noise=true , address=45.91.116.8:4048 , piece_id=UTE4QMRHNQQZE7B6M6TJ5WCRP3O3SD5DEVUCVHJJS5F7LPBR4ASQ, seg_pos={0 0}, upload=514117]: Long tail cancellation due to optimal threshold Dec 30 01:57:14.318911545 (586h28m14.18999458s) [noise=true , address=77.90.31.11:4555 , piece_id=E4CXSIVAPSAOX2OTSLBCQRR3A7Z27I76QOIRPOPG6YJ2WITWHR5Q, seg_pos={0 0}, upload=514117]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Dec 30 01:57:14.318918949 (586h28m14.190001984s) [noise=true , address=77.90.31.11:4555 , piece_id=E4CXSIVAPSAOX2OTSLBCQRR3A7Z27I76QOIRPOPG6YJ2WITWHR5Q, seg_pos={0 0}, upload=514117]: Long tail cancellation due to optimal threshold Dec 30 01:57:14.318939818 (586h28m14.190022853s) [seg_pos={0 0}, upload=514117]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 01:57:14.319006966 (586h28m14.190090001s) [upload=514117]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:57:14.319011665 (586h28m14.1900947s) [upload=514117]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:57:14.602030556 (586h28m14.473113601s) [upload=514117]: Done uploading segments... Dec 30 01:57:14.602051265 (586h28m14.47313431s) [upload=514117]: Upload finished. err: <nil> Duration: 5.718616089s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................................................X......] duration: 3.364057911s trace: https://jaeger.dev.storj.io/trace/1ac298e20b765731 FAIL (2025-12-30 01:58:50 UTC) orbiter warning: context deadline exceeded Duration: 3.364082889s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 33.80237084s trace: https://jaeger.dev.storj.io/trace/58b94359c097ef0d FAIL (2025-12-30 01:58:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.221968961 (585h44m42.093052017s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222491837 (585h44m42.093574882s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222877261 (585h44m42.093960306s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223249098 (585h44m42.094332143s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223637930 (585h44m42.094720965s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224032200 (585h44m42.095115245s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224444876 (585h44m42.095527921s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224905332 (585h44m42.095988378s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225388433 (585h44m42.096471478s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225981221 (585h44m42.097064256s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227130389 (585h44m42.098213435s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227655670 (585h44m42.098738705s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228177764 (585h44m42.099260799s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228673828 (585h44m42.099756863s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:43.678892099 (585h44m43.549975134s) [upload=513482]: Next returned nil segment Dec 30 01:13:43.678910774 (585h44m43.549993819s) [upload=513482]: Waiting for error group managing segments... Dec 30 01:13:44.076462870 (585h44m43.947545905s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.083306310 (585h44m43.954389345s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.107556817 (585h44m43.978639852s) [noise=true , address=145.79.227.9:4391 , piece_id=4AVX4RQJ65H3VL7U3MW6MUYHGRK6YS7GOGZO4XVVG6YYTHFHAXDQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.147374754 (585h44m44.018457799s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.212624668 (585h44m44.083707703s) [noise=true , address=64.49.13.170:4721 , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.237525922 (585h44m44.108608957s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.242637575 (585h44m44.11372063s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.283510060 (585h44m44.154593095s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.340441716 (585h44m44.211524761s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.408662737 (585h44m44.279745772s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.962802591 (585h44m44.833885626s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.963549243 (585h44m44.834632288s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.017193407 (585h44m44.888276442s) [noise=true , address=145.79.227.44:4218 , piece_id=UBAPWL2SMR7FJ2AHQ5XHMX3H6NKDFZ23HEDFRMJLJFYZKGC4X4MA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093476639 (585h44m44.964559684s) [noise=true , address=145.79.227.49:4370 , piece_id=RZK3XQONM7S2XSOIO4EXITQ3XQGVYLIJNK66RI6LL5NMFG46NOPA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093501687 (585h44m44.964584722s) [seg_pos={0 0}, upload=513482]: Segment reached optimal threshold of 14 pieces. Dec 30 01:13:45.093670669 (585h44m44.964753704s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection Dec 30 01:13:45.093677973 (585h44m44.964761008s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093727678 (585h44m44.964810713s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093735993 (585h44m44.964819028s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093827067 (585h44m44.964910122s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UP8PxqeK7t24PXjopH1nziHYQFnsxhkamSHz8dmwcbuDxkAF6): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093833129 (585h44m44.964916174s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093916176 (585h44m44.964999211s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qZC1tmCvCadnm1fYZxR3kc4fonZUJPzvfTKkTFiDtccW5bo9T): protocol: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection Dec 30 01:13:45.093921096 (585h44m44.965004141s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093966583 (585h44m44.965049618s) [seg_pos={0 0}, upload=513482]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 01:13:45.094031926 (585h44m44.965114961s) [upload=513482]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:13:45.094036336 (585h44m44.965119371s) [upload=513482]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:13:45.389410141 (585h44m45.260493176s) [upload=513482]: Done uploading segments... Dec 30 01:13:45.389430891 (585h44m45.260513926s) [upload=513482]: Upload finished. err: <nil> Duration: 33.802404616s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [..........XX.....X.X......X.X..XX.X....X..XXX..XX.....X..X..] duration: 5.954989496s trace: https://jaeger.dev.storj.io/trace/54f1aa927f806c2c FAIL (2025-12-30 01:58:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: ICKQPUWVMJHIL6SX3ANBJQDVS3KBSTFK4A7SRHNUH2T5DYVNPVNA) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: BOQ5Y3TUGH32KHSCQV73J22E4QZJT7UJ5JSO42JFEAPXEJRLIMLQ) context deadline exceeded ======LOGS====== Dec 30 01:57:11.787337529 (586h28m11.658420574s) [upload=514117]: Starting upload Dec 30 01:57:11.787437048 (586h28m11.658520103s) [upload=514117]: Done starting upload Dec 30 01:57:11.787563730 (586h28m11.658646775s) [upload=514117]: Uploading segments... Dec 30 01:57:11.787681634 (586h28m11.658764669s) [upload=514117]: Got next segment. Inline: false Dec 30 01:57:11.787724315 (586h28m11.65880736s) [upload=514117]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 01:57:11.787730297 (586h28m11.658813332s) [upload=514117]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 01:57:12.031832714 (586h28m11.902915759s) [seg_pos={0 0}, upload=514117]: Begin upload segment... Dec 30 01:57:12.031909590 (586h28m11.902992635s) [seg_pos={0 0}, upload=514117]: Done begin upload segment. Dec 30 01:57:12.032054276 (586h28m11.903137321s) [noise=true , address=145.79.227.3:4149 , piece_id=AO37XAZ6DQLWGN65KZKLL4Z7XRKJSLIHIVMCYCTVBZTND254HBTQ, seg_pos={0 0}, upload=514117]: Uploading piece... Dec 30 01:57:12.032074805 (586h28m11.90315785s) [noise=true , address=145.79.227.6:4257 , piece_id=DH3S3IDPA7I66ZRKIFOXKLN2MDBMVMAWZTJG4PYKJZWK7NA7Z4VQ, seg_pos={0 0}, upload=514117]: Uploading piece... Dec 30 01:57:12.032718500 (586h28m11.903801535s) [noise=true , address=45.91.116.11:4119 , piece_id=ICKQPUWVMJHIL6SX3ANBJQDVS3KBSTFK4A7SRHNUH2T5DYVNPVNA, seg_pos={0 0}, upload=514117]: Uploading piece... Dec 30 01:57:12.033224033 (586h28m11.904307079s) [noise=true , address=145.79.227.7:4293 , piece_id=LSRJCQLXABLYH6NPKMPXAG6TDT6A3QKG2O3BXVTEVOOXTKXCEI5Q, seg_pos={0 0}, upload=514117]: Uploading piece... Dec 30 01:57:12.033756397 (586h28m11.904839442s) [noise=true , address=145.79.227.5:4221 , piece_id=KIKI266HETJCZQZ7QA67O5E3WVFQLTAAMGJJ74COADMX5HLZ5AYQ, seg_pos={0 0}, upload=514117]: Uploading piece... Dec 30 01:57:12.034508449 (586h28m11.905591494s) [noise=true , address=145.79.227.10:4401 , piece_id=LHZNK76RWLXKNORM57WAJIKGBLMWG76XJX47PUB2ANWWVEB2A25A, seg_pos={0 0}, upload=514117]: Uploading piece... Dec 30 01:57:12.038525228 (586h28m11.909608273s) [noise=true , address=77.90.31.14:4663 , piece_id=BOQ5Y3TUGH32KHSCQV73J22E4QZJT7UJ5JSO42JFEAPXEJRLIMLQ, seg_pos={0 0}, upload=514117]: Uploading piece... Dec 30 01:57:12.043483929 (586h28m11.914567014s) [noise=true , address=145.79.227.46:4602 , piece_id=AXSPIAOPJLYJHQPZ2NL5YS2XO4OJVYZ345S4HPX6WPCP46IAZ5QA, seg_pos={0 0}, upload=514117]: Uploading piece... Dec 30 01:57:12.050045313 (586h28m11.921128378s) [noise=true , address=145.79.227.32:4836 , piece_id=W5K62R7THVPLT2BBMLKQQPJTIBICAKJOHQCKZEKF4BKJXLMIAAOA, seg_pos={0 0}, upload=514117]: Uploading piece... Dec 30 01:57:12.051946353 (586h28m11.923029398s) [noise=true , address=145.79.227.19:4077 , piece_id=JWPVZW2PNM6I3HF5PWOWRNFC6C3O6FQTALILSON54QBQ3DPOWT7A, seg_pos={0 0}, upload=514117]: Uploading piece... Dec 30 01:57:12.054919544 (586h28m11.926002579s) [noise=true , address=77.90.31.11:4555 , piece_id=E4CXSIVAPSAOX2OTSLBCQRR3A7Z27I76QOIRPOPG6YJ2WITWHR5Q, seg_pos={0 0}, upload=514117]: Uploading piece... Dec 30 01:57:12.056504032 (586h28m11.927587067s) [noise=true , address=45.91.116.12:4228 , piece_id=NMK2B4GYZXVCBZZYZSQ4EV3H5MAH6CMU2WFGIEGJDNNN5OOSMRSA, seg_pos={0 0}, upload=514117]: Uploading piece... Dec 30 01:57:12.058074632 (586h28m11.929157677s) [noise=true , address=45.91.116.8:4048 , piece_id=UTE4QMRHNQQZE7B6M6TJ5WCRP3O3SD5DEVUCVHJJS5F7LPBR4ASQ, seg_pos={0 0}, upload=514117]: Uploading piece... Dec 30 01:57:12.059216897 (586h28m11.930299932s) [noise=true , address=145.79.227.8:4329 , piece_id=GIDL5TFVOT423QPESVGKLDZ5KGE46Z4YESWOLNRANRIO4BQW74PA, seg_pos={0 0}, upload=514117]: Uploading piece... Dec 30 01:57:12.059267724 (586h28m11.930350769s) [noise=true , address=145.79.227.48:4302 , piece_id=SEWV6427NUPSS46PYFHECUHF3TJMKGRUYOAUXYYIOLBUXFRECMPQ, seg_pos={0 0}, upload=514117]: Uploading piece... Dec 30 01:57:12.061185155 (586h28m11.93226819s) [noise=true , address=145.79.227.52:4447 , piece_id=DIKHFJKTE2JXNIIG55JGLP42AHZCX6XKI4JYBRI5BQHACHI3UX6Q, seg_pos={0 0}, upload=514117]: Uploading piece... Dec 30 01:57:13.688319383 (586h28m13.559402428s) [upload=514117]: Next returned nil segment Dec 30 01:57:13.688338599 (586h28m13.559421634s) [upload=514117]: Waiting for error group managing segments... Dec 30 01:57:14.019582918 (586h28m13.890665963s) [noise=true , address=145.79.227.10:4401 , piece_id=LHZNK76RWLXKNORM57WAJIKGBLMWG76XJX47PUB2ANWWVEB2A25A, seg_pos={0 0}, upload=514117]: Done uploading piece. err: <nil> Dec 30 01:57:14.019676136 (586h28m13.890759171s) [noise=true , address=145.79.227.3:4149 , piece_id=AO37XAZ6DQLWGN65KZKLL4Z7XRKJSLIHIVMCYCTVBZTND254HBTQ, seg_pos={0 0}, upload=514117]: Done uploading piece. err: <nil> Dec 30 01:57:14.020237825 (586h28m13.89132086s) [noise=true , address=145.79.227.5:4221 , piece_id=KIKI266HETJCZQZ7QA67O5E3WVFQLTAAMGJJ74COADMX5HLZ5AYQ, seg_pos={0 0}, upload=514117]: Done uploading piece. err: <nil> Dec 30 01:57:14.035649170 (586h28m13.906732216s) [noise=true , address=145.79.227.8:4329 , piece_id=GIDL5TFVOT423QPESVGKLDZ5KGE46Z4YESWOLNRANRIO4BQW74PA, seg_pos={0 0}, upload=514117]: Done uploading piece. err: <nil> Dec 30 01:57:14.041524819 (586h28m13.912607864s) [noise=true , address=145.79.227.6:4257 , piece_id=DH3S3IDPA7I66ZRKIFOXKLN2MDBMVMAWZTJG4PYKJZWK7NA7Z4VQ, seg_pos={0 0}, upload=514117]: Done uploading piece. err: <nil> Dec 30 01:57:14.046683191 (586h28m13.917766236s) [noise=true , address=145.79.227.7:4293 , piece_id=LSRJCQLXABLYH6NPKMPXAG6TDT6A3QKG2O3BXVTEVOOXTKXCEI5Q, seg_pos={0 0}, upload=514117]: Done uploading piece. err: <nil> Dec 30 01:57:14.048287225 (586h28m13.919370271s) [noise=true , address=145.79.227.46:4602 , piece_id=AXSPIAOPJLYJHQPZ2NL5YS2XO4OJVYZ345S4HPX6WPCP46IAZ5QA, seg_pos={0 0}, upload=514117]: Done uploading piece. err: <nil> Dec 30 01:57:14.058951151 (586h28m13.930034186s) [noise=true , address=145.79.227.48:4302 , piece_id=SEWV6427NUPSS46PYFHECUHF3TJMKGRUYOAUXYYIOLBUXFRECMPQ, seg_pos={0 0}, upload=514117]: Done uploading piece. err: <nil> Dec 30 01:57:14.059101707 (586h28m13.930184753s) [noise=true , address=145.79.227.52:4447 , piece_id=DIKHFJKTE2JXNIIG55JGLP42AHZCX6XKI4JYBRI5BQHACHI3UX6Q, seg_pos={0 0}, upload=514117]: Done uploading piece. err: <nil> Dec 30 01:57:14.060655236 (586h28m13.931738281s) [noise=true , address=145.79.227.19:4077 , piece_id=JWPVZW2PNM6I3HF5PWOWRNFC6C3O6FQTALILSON54QBQ3DPOWT7A, seg_pos={0 0}, upload=514117]: Done uploading piece. err: <nil> Dec 30 01:57:14.061044076 (586h28m13.932127111s) [noise=true , address=145.79.227.32:4836 , piece_id=W5K62R7THVPLT2BBMLKQQPJTIBICAKJOHQCKZEKF4BKJXLMIAAOA, seg_pos={0 0}, upload=514117]: Done uploading piece. err: <nil> Dec 30 01:57:14.210079021 (586h28m14.081162056s) [noise=true , address=77.90.31.14:4663 , piece_id=BOQ5Y3TUGH32KHSCQV73J22E4QZJT7UJ5JSO42JFEAPXEJRLIMLQ, seg_pos={0 0}, upload=514117]: Done uploading piece. err: <nil> Dec 30 01:57:14.318394851 (586h28m14.189477896s) [noise=true , address=45.91.116.11:4119 , piece_id=ICKQPUWVMJHIL6SX3ANBJQDVS3KBSTFK4A7SRHNUH2T5DYVNPVNA, seg_pos={0 0}, upload=514117]: Done uploading piece. err: <nil> Dec 30 01:57:14.318413846 (586h28m14.189496891s) [seg_pos={0 0}, upload=514117]: Segment reached optimal threshold of 13 pieces. Dec 30 01:57:14.318615931 (586h28m14.189698966s) [noise=true , address=45.91.116.12:4228 , piece_id=NMK2B4GYZXVCBZZYZSQ4EV3H5MAH6CMU2WFGIEGJDNNN5OOSMRSA, seg_pos={0 0}, upload=514117]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 30 01:57:14.318624157 (586h28m14.189707192s) [noise=true , address=45.91.116.12:4228 , piece_id=NMK2B4GYZXVCBZZYZSQ4EV3H5MAH6CMU2WFGIEGJDNNN5OOSMRSA, seg_pos={0 0}, upload=514117]: Long tail cancellation due to optimal threshold Dec 30 01:57:14.318738245 (586h28m14.18982128s) [noise=true , address=45.91.116.8:4048 , piece_id=UTE4QMRHNQQZE7B6M6TJ5WCRP3O3SD5DEVUCVHJJS5F7LPBR4ASQ, seg_pos={0 0}, upload=514117]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 30 01:57:14.318759765 (586h28m14.1898428s) [noise=true , address=45.91.116.8:4048 , piece_id=UTE4QMRHNQQZE7B6M6TJ5WCRP3O3SD5DEVUCVHJJS5F7LPBR4ASQ, seg_pos={0 0}, upload=514117]: Long tail cancellation due to optimal threshold Dec 30 01:57:14.318911545 (586h28m14.18999458s) [noise=true , address=77.90.31.11:4555 , piece_id=E4CXSIVAPSAOX2OTSLBCQRR3A7Z27I76QOIRPOPG6YJ2WITWHR5Q, seg_pos={0 0}, upload=514117]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Dec 30 01:57:14.318918949 (586h28m14.190001984s) [noise=true , address=77.90.31.11:4555 , piece_id=E4CXSIVAPSAOX2OTSLBCQRR3A7Z27I76QOIRPOPG6YJ2WITWHR5Q, seg_pos={0 0}, upload=514117]: Long tail cancellation due to optimal threshold Dec 30 01:57:14.318939818 (586h28m14.190022853s) [seg_pos={0 0}, upload=514117]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 01:57:14.319006966 (586h28m14.190090001s) [upload=514117]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:57:14.319011665 (586h28m14.1900947s) [upload=514117]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:57:14.602030556 (586h28m14.473113601s) [upload=514117]: Done uploading segments... Dec 30 01:57:14.602051265 (586h28m14.47313431s) [upload=514117]: Upload finished. err: <nil> Duration: 5.955012101s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................................................X.....] duration: 5.001009193s trace: https://jaeger.dev.storj.io/trace/1ac298e20b765731 FAIL (2025-12-30 01:58:15 UTC) orbiter warning: context deadline exceeded Duration: 5.001039752s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................................................X....] duration: 3.446927015s trace: https://jaeger.dev.storj.io/trace/1ac298e20b765731 FAIL (2025-12-30 01:57:48 UTC) orbiter warning: context deadline exceeded Duration: 3.446957452s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [...........XX.....X.X......X.X..XX.X....X..XXX..XX.....X..X.] duration: 5.877472866s trace: https://jaeger.dev.storj.io/trace/54f1aa927f806c2c FAIL (2025-12-30 01:57:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: ICKQPUWVMJHIL6SX3ANBJQDVS3KBSTFK4A7SRHNUH2T5DYVNPVNA) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: BOQ5Y3TUGH32KHSCQV73J22E4QZJT7UJ5JSO42JFEAPXEJRLIMLQ) context deadline exceeded ======LOGS====== Dec 30 01:57:11.787337529 (586h28m11.658420574s) [upload=514117]: Starting upload Dec 30 01:57:11.787437048 (586h28m11.658520103s) [upload=514117]: Done starting upload Dec 30 01:57:11.787563730 (586h28m11.658646775s) [upload=514117]: Uploading segments... Dec 30 01:57:11.787681634 (586h28m11.658764669s) [upload=514117]: Got next segment. Inline: false Dec 30 01:57:11.787724315 (586h28m11.65880736s) [upload=514117]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 01:57:11.787730297 (586h28m11.658813332s) [upload=514117]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 01:57:12.031832714 (586h28m11.902915759s) [seg_pos={0 0}, upload=514117]: Begin upload segment... Dec 30 01:57:12.031909590 (586h28m11.902992635s) [seg_pos={0 0}, upload=514117]: Done begin upload segment. Dec 30 01:57:12.032054276 (586h28m11.903137321s) [noise=true , address=145.79.227.3:4149 , piece_id=AO37XAZ6DQLWGN65KZKLL4Z7XRKJSLIHIVMCYCTVBZTND254HBTQ, seg_pos={0 0}, upload=514117]: Uploading piece... Dec 30 01:57:12.032074805 (586h28m11.90315785s) [noise=true , address=145.79.227.6:4257 , piece_id=DH3S3IDPA7I66ZRKIFOXKLN2MDBMVMAWZTJG4PYKJZWK7NA7Z4VQ, seg_pos={0 0}, upload=514117]: Uploading piece... Dec 30 01:57:12.032718500 (586h28m11.903801535s) [noise=true , address=45.91.116.11:4119 , piece_id=ICKQPUWVMJHIL6SX3ANBJQDVS3KBSTFK4A7SRHNUH2T5DYVNPVNA, seg_pos={0 0}, upload=514117]: Uploading piece... Dec 30 01:57:12.033224033 (586h28m11.904307079s) [noise=true , address=145.79.227.7:4293 , piece_id=LSRJCQLXABLYH6NPKMPXAG6TDT6A3QKG2O3BXVTEVOOXTKXCEI5Q, seg_pos={0 0}, upload=514117]: Uploading piece... Dec 30 01:57:12.033756397 (586h28m11.904839442s) [noise=true , address=145.79.227.5:4221 , piece_id=KIKI266HETJCZQZ7QA67O5E3WVFQLTAAMGJJ74COADMX5HLZ5AYQ, seg_pos={0 0}, upload=514117]: Uploading piece... Dec 30 01:57:12.034508449 (586h28m11.905591494s) [noise=true , address=145.79.227.10:4401 , piece_id=LHZNK76RWLXKNORM57WAJIKGBLMWG76XJX47PUB2ANWWVEB2A25A, seg_pos={0 0}, upload=514117]: Uploading piece... Dec 30 01:57:12.038525228 (586h28m11.909608273s) [noise=true , address=77.90.31.14:4663 , piece_id=BOQ5Y3TUGH32KHSCQV73J22E4QZJT7UJ5JSO42JFEAPXEJRLIMLQ, seg_pos={0 0}, upload=514117]: Uploading piece... Dec 30 01:57:12.043483929 (586h28m11.914567014s) [noise=true , address=145.79.227.46:4602 , piece_id=AXSPIAOPJLYJHQPZ2NL5YS2XO4OJVYZ345S4HPX6WPCP46IAZ5QA, seg_pos={0 0}, upload=514117]: Uploading piece... Dec 30 01:57:12.050045313 (586h28m11.921128378s) [noise=true , address=145.79.227.32:4836 , piece_id=W5K62R7THVPLT2BBMLKQQPJTIBICAKJOHQCKZEKF4BKJXLMIAAOA, seg_pos={0 0}, upload=514117]: Uploading piece... Dec 30 01:57:12.051946353 (586h28m11.923029398s) [noise=true , address=145.79.227.19:4077 , piece_id=JWPVZW2PNM6I3HF5PWOWRNFC6C3O6FQTALILSON54QBQ3DPOWT7A, seg_pos={0 0}, upload=514117]: Uploading piece... Dec 30 01:57:12.054919544 (586h28m11.926002579s) [noise=true , address=77.90.31.11:4555 , piece_id=E4CXSIVAPSAOX2OTSLBCQRR3A7Z27I76QOIRPOPG6YJ2WITWHR5Q, seg_pos={0 0}, upload=514117]: Uploading piece... Dec 30 01:57:12.056504032 (586h28m11.927587067s) [noise=true , address=45.91.116.12:4228 , piece_id=NMK2B4GYZXVCBZZYZSQ4EV3H5MAH6CMU2WFGIEGJDNNN5OOSMRSA, seg_pos={0 0}, upload=514117]: Uploading piece... Dec 30 01:57:12.058074632 (586h28m11.929157677s) [noise=true , address=45.91.116.8:4048 , piece_id=UTE4QMRHNQQZE7B6M6TJ5WCRP3O3SD5DEVUCVHJJS5F7LPBR4ASQ, seg_pos={0 0}, upload=514117]: Uploading piece... Dec 30 01:57:12.059216897 (586h28m11.930299932s) [noise=true , address=145.79.227.8:4329 , piece_id=GIDL5TFVOT423QPESVGKLDZ5KGE46Z4YESWOLNRANRIO4BQW74PA, seg_pos={0 0}, upload=514117]: Uploading piece... Dec 30 01:57:12.059267724 (586h28m11.930350769s) [noise=true , address=145.79.227.48:4302 , piece_id=SEWV6427NUPSS46PYFHECUHF3TJMKGRUYOAUXYYIOLBUXFRECMPQ, seg_pos={0 0}, upload=514117]: Uploading piece... Dec 30 01:57:12.061185155 (586h28m11.93226819s) [noise=true , address=145.79.227.52:4447 , piece_id=DIKHFJKTE2JXNIIG55JGLP42AHZCX6XKI4JYBRI5BQHACHI3UX6Q, seg_pos={0 0}, upload=514117]: Uploading piece... Dec 30 01:57:13.688319383 (586h28m13.559402428s) [upload=514117]: Next returned nil segment Dec 30 01:57:13.688338599 (586h28m13.559421634s) [upload=514117]: Waiting for error group managing segments... Dec 30 01:57:14.019582918 (586h28m13.890665963s) [noise=true , address=145.79.227.10:4401 , piece_id=LHZNK76RWLXKNORM57WAJIKGBLMWG76XJX47PUB2ANWWVEB2A25A, seg_pos={0 0}, upload=514117]: Done uploading piece. err: <nil> Dec 30 01:57:14.019676136 (586h28m13.890759171s) [noise=true , address=145.79.227.3:4149 , piece_id=AO37XAZ6DQLWGN65KZKLL4Z7XRKJSLIHIVMCYCTVBZTND254HBTQ, seg_pos={0 0}, upload=514117]: Done uploading piece. err: <nil> Dec 30 01:57:14.020237825 (586h28m13.89132086s) [noise=true , address=145.79.227.5:4221 , piece_id=KIKI266HETJCZQZ7QA67O5E3WVFQLTAAMGJJ74COADMX5HLZ5AYQ, seg_pos={0 0}, upload=514117]: Done uploading piece. err: <nil> Dec 30 01:57:14.035649170 (586h28m13.906732216s) [noise=true , address=145.79.227.8:4329 , piece_id=GIDL5TFVOT423QPESVGKLDZ5KGE46Z4YESWOLNRANRIO4BQW74PA, seg_pos={0 0}, upload=514117]: Done uploading piece. err: <nil> Dec 30 01:57:14.041524819 (586h28m13.912607864s) [noise=true , address=145.79.227.6:4257 , piece_id=DH3S3IDPA7I66ZRKIFOXKLN2MDBMVMAWZTJG4PYKJZWK7NA7Z4VQ, seg_pos={0 0}, upload=514117]: Done uploading piece. err: <nil> Dec 30 01:57:14.046683191 (586h28m13.917766236s) [noise=true , address=145.79.227.7:4293 , piece_id=LSRJCQLXABLYH6NPKMPXAG6TDT6A3QKG2O3BXVTEVOOXTKXCEI5Q, seg_pos={0 0}, upload=514117]: Done uploading piece. err: <nil> Dec 30 01:57:14.048287225 (586h28m13.919370271s) [noise=true , address=145.79.227.46:4602 , piece_id=AXSPIAOPJLYJHQPZ2NL5YS2XO4OJVYZ345S4HPX6WPCP46IAZ5QA, seg_pos={0 0}, upload=514117]: Done uploading piece. err: <nil> Dec 30 01:57:14.058951151 (586h28m13.930034186s) [noise=true , address=145.79.227.48:4302 , piece_id=SEWV6427NUPSS46PYFHECUHF3TJMKGRUYOAUXYYIOLBUXFRECMPQ, seg_pos={0 0}, upload=514117]: Done uploading piece. err: <nil> Dec 30 01:57:14.059101707 (586h28m13.930184753s) [noise=true , address=145.79.227.52:4447 , piece_id=DIKHFJKTE2JXNIIG55JGLP42AHZCX6XKI4JYBRI5BQHACHI3UX6Q, seg_pos={0 0}, upload=514117]: Done uploading piece. err: <nil> Dec 30 01:57:14.060655236 (586h28m13.931738281s) [noise=true , address=145.79.227.19:4077 , piece_id=JWPVZW2PNM6I3HF5PWOWRNFC6C3O6FQTALILSON54QBQ3DPOWT7A, seg_pos={0 0}, upload=514117]: Done uploading piece. err: <nil> Dec 30 01:57:14.061044076 (586h28m13.932127111s) [noise=true , address=145.79.227.32:4836 , piece_id=W5K62R7THVPLT2BBMLKQQPJTIBICAKJOHQCKZEKF4BKJXLMIAAOA, seg_pos={0 0}, upload=514117]: Done uploading piece. err: <nil> Dec 30 01:57:14.210079021 (586h28m14.081162056s) [noise=true , address=77.90.31.14:4663 , piece_id=BOQ5Y3TUGH32KHSCQV73J22E4QZJT7UJ5JSO42JFEAPXEJRLIMLQ, seg_pos={0 0}, upload=514117]: Done uploading piece. err: <nil> Dec 30 01:57:14.318394851 (586h28m14.189477896s) [noise=true , address=45.91.116.11:4119 , piece_id=ICKQPUWVMJHIL6SX3ANBJQDVS3KBSTFK4A7SRHNUH2T5DYVNPVNA, seg_pos={0 0}, upload=514117]: Done uploading piece. err: <nil> Dec 30 01:57:14.318413846 (586h28m14.189496891s) [seg_pos={0 0}, upload=514117]: Segment reached optimal threshold of 13 pieces. Dec 30 01:57:14.318615931 (586h28m14.189698966s) [noise=true , address=45.91.116.12:4228 , piece_id=NMK2B4GYZXVCBZZYZSQ4EV3H5MAH6CMU2WFGIEGJDNNN5OOSMRSA, seg_pos={0 0}, upload=514117]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 30 01:57:14.318624157 (586h28m14.189707192s) [noise=true , address=45.91.116.12:4228 , piece_id=NMK2B4GYZXVCBZZYZSQ4EV3H5MAH6CMU2WFGIEGJDNNN5OOSMRSA, seg_pos={0 0}, upload=514117]: Long tail cancellation due to optimal threshold Dec 30 01:57:14.318738245 (586h28m14.18982128s) [noise=true , address=45.91.116.8:4048 , piece_id=UTE4QMRHNQQZE7B6M6TJ5WCRP3O3SD5DEVUCVHJJS5F7LPBR4ASQ, seg_pos={0 0}, upload=514117]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 30 01:57:14.318759765 (586h28m14.1898428s) [noise=true , address=45.91.116.8:4048 , piece_id=UTE4QMRHNQQZE7B6M6TJ5WCRP3O3SD5DEVUCVHJJS5F7LPBR4ASQ, seg_pos={0 0}, upload=514117]: Long tail cancellation due to optimal threshold Dec 30 01:57:14.318911545 (586h28m14.18999458s) [noise=true , address=77.90.31.11:4555 , piece_id=E4CXSIVAPSAOX2OTSLBCQRR3A7Z27I76QOIRPOPG6YJ2WITWHR5Q, seg_pos={0 0}, upload=514117]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Dec 30 01:57:14.318918949 (586h28m14.190001984s) [noise=true , address=77.90.31.11:4555 , piece_id=E4CXSIVAPSAOX2OTSLBCQRR3A7Z27I76QOIRPOPG6YJ2WITWHR5Q, seg_pos={0 0}, upload=514117]: Long tail cancellation due to optimal threshold Dec 30 01:57:14.318939818 (586h28m14.190022853s) [seg_pos={0 0}, upload=514117]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 01:57:14.319006966 (586h28m14.190090001s) [upload=514117]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:57:14.319011665 (586h28m14.1900947s) [upload=514117]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:57:14.602030556 (586h28m14.473113601s) [upload=514117]: Done uploading segments... Dec 30 01:57:14.602051265 (586h28m14.47313431s) [upload=514117]: Upload finished. err: <nil> Duration: 5.877487985s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [............XX.....X.X......X.X..XX.X....X..XXX..XX.....X..X] duration: 30.002397389s trace: https://jaeger.dev.storj.io/trace/54f1aa927f806c2c FAIL (2025-12-30 01:57:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: ICKQPUWVMJHIL6SX3ANBJQDVS3KBSTFK4A7SRHNUH2T5DYVNPVNA) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: BOQ5Y3TUGH32KHSCQV73J22E4QZJT7UJ5JSO42JFEAPXEJRLIMLQ) context deadline exceeded ======LOGS====== Dec 30 01:57:11.787337529 (586h28m11.658420574s) [upload=514117]: Starting upload Dec 30 01:57:11.787437048 (586h28m11.658520103s) [upload=514117]: Done starting upload Dec 30 01:57:11.787563730 (586h28m11.658646775s) [upload=514117]: Uploading segments... Dec 30 01:57:11.787681634 (586h28m11.658764669s) [upload=514117]: Got next segment. Inline: false Dec 30 01:57:11.787724315 (586h28m11.65880736s) [upload=514117]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 01:57:11.787730297 (586h28m11.658813332s) [upload=514117]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 01:57:12.031832714 (586h28m11.902915759s) [seg_pos={0 0}, upload=514117]: Begin upload segment... Dec 30 01:57:12.031909590 (586h28m11.902992635s) [seg_pos={0 0}, upload=514117]: Done begin upload segment. Dec 30 01:57:12.032054276 (586h28m11.903137321s) [noise=true , address=145.79.227.3:4149 , piece_id=AO37XAZ6DQLWGN65KZKLL4Z7XRKJSLIHIVMCYCTVBZTND254HBTQ, seg_pos={0 0}, upload=514117]: Uploading piece... Dec 30 01:57:12.032074805 (586h28m11.90315785s) [noise=true , address=145.79.227.6:4257 , piece_id=DH3S3IDPA7I66ZRKIFOXKLN2MDBMVMAWZTJG4PYKJZWK7NA7Z4VQ, seg_pos={0 0}, upload=514117]: Uploading piece... Dec 30 01:57:12.032718500 (586h28m11.903801535s) [noise=true , address=45.91.116.11:4119 , piece_id=ICKQPUWVMJHIL6SX3ANBJQDVS3KBSTFK4A7SRHNUH2T5DYVNPVNA, seg_pos={0 0}, upload=514117]: Uploading piece... Dec 30 01:57:12.033224033 (586h28m11.904307079s) [noise=true , address=145.79.227.7:4293 , piece_id=LSRJCQLXABLYH6NPKMPXAG6TDT6A3QKG2O3BXVTEVOOXTKXCEI5Q, seg_pos={0 0}, upload=514117]: Uploading piece... Dec 30 01:57:12.033756397 (586h28m11.904839442s) [noise=true , address=145.79.227.5:4221 , piece_id=KIKI266HETJCZQZ7QA67O5E3WVFQLTAAMGJJ74COADMX5HLZ5AYQ, seg_pos={0 0}, upload=514117]: Uploading piece... Dec 30 01:57:12.034508449 (586h28m11.905591494s) [noise=true , address=145.79.227.10:4401 , piece_id=LHZNK76RWLXKNORM57WAJIKGBLMWG76XJX47PUB2ANWWVEB2A25A, seg_pos={0 0}, upload=514117]: Uploading piece... Dec 30 01:57:12.038525228 (586h28m11.909608273s) [noise=true , address=77.90.31.14:4663 , piece_id=BOQ5Y3TUGH32KHSCQV73J22E4QZJT7UJ5JSO42JFEAPXEJRLIMLQ, seg_pos={0 0}, upload=514117]: Uploading piece... Dec 30 01:57:12.043483929 (586h28m11.914567014s) [noise=true , address=145.79.227.46:4602 , piece_id=AXSPIAOPJLYJHQPZ2NL5YS2XO4OJVYZ345S4HPX6WPCP46IAZ5QA, seg_pos={0 0}, upload=514117]: Uploading piece... Dec 30 01:57:12.050045313 (586h28m11.921128378s) [noise=true , address=145.79.227.32:4836 , piece_id=W5K62R7THVPLT2BBMLKQQPJTIBICAKJOHQCKZEKF4BKJXLMIAAOA, seg_pos={0 0}, upload=514117]: Uploading piece... Dec 30 01:57:12.051946353 (586h28m11.923029398s) [noise=true , address=145.79.227.19:4077 , piece_id=JWPVZW2PNM6I3HF5PWOWRNFC6C3O6FQTALILSON54QBQ3DPOWT7A, seg_pos={0 0}, upload=514117]: Uploading piece... Dec 30 01:57:12.054919544 (586h28m11.926002579s) [noise=true , address=77.90.31.11:4555 , piece_id=E4CXSIVAPSAOX2OTSLBCQRR3A7Z27I76QOIRPOPG6YJ2WITWHR5Q, seg_pos={0 0}, upload=514117]: Uploading piece... Dec 30 01:57:12.056504032 (586h28m11.927587067s) [noise=true , address=45.91.116.12:4228 , piece_id=NMK2B4GYZXVCBZZYZSQ4EV3H5MAH6CMU2WFGIEGJDNNN5OOSMRSA, seg_pos={0 0}, upload=514117]: Uploading piece... Dec 30 01:57:12.058074632 (586h28m11.929157677s) [noise=true , address=45.91.116.8:4048 , piece_id=UTE4QMRHNQQZE7B6M6TJ5WCRP3O3SD5DEVUCVHJJS5F7LPBR4ASQ, seg_pos={0 0}, upload=514117]: Uploading piece... Dec 30 01:57:12.059216897 (586h28m11.930299932s) [noise=true , address=145.79.227.8:4329 , piece_id=GIDL5TFVOT423QPESVGKLDZ5KGE46Z4YESWOLNRANRIO4BQW74PA, seg_pos={0 0}, upload=514117]: Uploading piece... Dec 30 01:57:12.059267724 (586h28m11.930350769s) [noise=true , address=145.79.227.48:4302 , piece_id=SEWV6427NUPSS46PYFHECUHF3TJMKGRUYOAUXYYIOLBUXFRECMPQ, seg_pos={0 0}, upload=514117]: Uploading piece... Dec 30 01:57:12.061185155 (586h28m11.93226819s) [noise=true , address=145.79.227.52:4447 , piece_id=DIKHFJKTE2JXNIIG55JGLP42AHZCX6XKI4JYBRI5BQHACHI3UX6Q, seg_pos={0 0}, upload=514117]: Uploading piece... Dec 30 01:57:13.688319383 (586h28m13.559402428s) [upload=514117]: Next returned nil segment Dec 30 01:57:13.688338599 (586h28m13.559421634s) [upload=514117]: Waiting for error group managing segments... Dec 30 01:57:14.019582918 (586h28m13.890665963s) [noise=true , address=145.79.227.10:4401 , piece_id=LHZNK76RWLXKNORM57WAJIKGBLMWG76XJX47PUB2ANWWVEB2A25A, seg_pos={0 0}, upload=514117]: Done uploading piece. err: <nil> Dec 30 01:57:14.019676136 (586h28m13.890759171s) [noise=true , address=145.79.227.3:4149 , piece_id=AO37XAZ6DQLWGN65KZKLL4Z7XRKJSLIHIVMCYCTVBZTND254HBTQ, seg_pos={0 0}, upload=514117]: Done uploading piece. err: <nil> Dec 30 01:57:14.020237825 (586h28m13.89132086s) [noise=true , address=145.79.227.5:4221 , piece_id=KIKI266HETJCZQZ7QA67O5E3WVFQLTAAMGJJ74COADMX5HLZ5AYQ, seg_pos={0 0}, upload=514117]: Done uploading piece. err: <nil> Dec 30 01:57:14.035649170 (586h28m13.906732216s) [noise=true , address=145.79.227.8:4329 , piece_id=GIDL5TFVOT423QPESVGKLDZ5KGE46Z4YESWOLNRANRIO4BQW74PA, seg_pos={0 0}, upload=514117]: Done uploading piece. err: <nil> Dec 30 01:57:14.041524819 (586h28m13.912607864s) [noise=true , address=145.79.227.6:4257 , piece_id=DH3S3IDPA7I66ZRKIFOXKLN2MDBMVMAWZTJG4PYKJZWK7NA7Z4VQ, seg_pos={0 0}, upload=514117]: Done uploading piece. err: <nil> Dec 30 01:57:14.046683191 (586h28m13.917766236s) [noise=true , address=145.79.227.7:4293 , piece_id=LSRJCQLXABLYH6NPKMPXAG6TDT6A3QKG2O3BXVTEVOOXTKXCEI5Q, seg_pos={0 0}, upload=514117]: Done uploading piece. err: <nil> Dec 30 01:57:14.048287225 (586h28m13.919370271s) [noise=true , address=145.79.227.46:4602 , piece_id=AXSPIAOPJLYJHQPZ2NL5YS2XO4OJVYZ345S4HPX6WPCP46IAZ5QA, seg_pos={0 0}, upload=514117]: Done uploading piece. err: <nil> Dec 30 01:57:14.058951151 (586h28m13.930034186s) [noise=true , address=145.79.227.48:4302 , piece_id=SEWV6427NUPSS46PYFHECUHF3TJMKGRUYOAUXYYIOLBUXFRECMPQ, seg_pos={0 0}, upload=514117]: Done uploading piece. err: <nil> Dec 30 01:57:14.059101707 (586h28m13.930184753s) [noise=true , address=145.79.227.52:4447 , piece_id=DIKHFJKTE2JXNIIG55JGLP42AHZCX6XKI4JYBRI5BQHACHI3UX6Q, seg_pos={0 0}, upload=514117]: Done uploading piece. err: <nil> Dec 30 01:57:14.060655236 (586h28m13.931738281s) [noise=true , address=145.79.227.19:4077 , piece_id=JWPVZW2PNM6I3HF5PWOWRNFC6C3O6FQTALILSON54QBQ3DPOWT7A, seg_pos={0 0}, upload=514117]: Done uploading piece. err: <nil> Dec 30 01:57:14.061044076 (586h28m13.932127111s) [noise=true , address=145.79.227.32:4836 , piece_id=W5K62R7THVPLT2BBMLKQQPJTIBICAKJOHQCKZEKF4BKJXLMIAAOA, seg_pos={0 0}, upload=514117]: Done uploading piece. err: <nil> Dec 30 01:57:14.210079021 (586h28m14.081162056s) [noise=true , address=77.90.31.14:4663 , piece_id=BOQ5Y3TUGH32KHSCQV73J22E4QZJT7UJ5JSO42JFEAPXEJRLIMLQ, seg_pos={0 0}, upload=514117]: Done uploading piece. err: <nil> Dec 30 01:57:14.318394851 (586h28m14.189477896s) [noise=true , address=45.91.116.11:4119 , piece_id=ICKQPUWVMJHIL6SX3ANBJQDVS3KBSTFK4A7SRHNUH2T5DYVNPVNA, seg_pos={0 0}, upload=514117]: Done uploading piece. err: <nil> Dec 30 01:57:14.318413846 (586h28m14.189496891s) [seg_pos={0 0}, upload=514117]: Segment reached optimal threshold of 13 pieces. Dec 30 01:57:14.318615931 (586h28m14.189698966s) [noise=true , address=45.91.116.12:4228 , piece_id=NMK2B4GYZXVCBZZYZSQ4EV3H5MAH6CMU2WFGIEGJDNNN5OOSMRSA, seg_pos={0 0}, upload=514117]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 30 01:57:14.318624157 (586h28m14.189707192s) [noise=true , address=45.91.116.12:4228 , piece_id=NMK2B4GYZXVCBZZYZSQ4EV3H5MAH6CMU2WFGIEGJDNNN5OOSMRSA, seg_pos={0 0}, upload=514117]: Long tail cancellation due to optimal threshold Dec 30 01:57:14.318738245 (586h28m14.18982128s) [noise=true , address=45.91.116.8:4048 , piece_id=UTE4QMRHNQQZE7B6M6TJ5WCRP3O3SD5DEVUCVHJJS5F7LPBR4ASQ, seg_pos={0 0}, upload=514117]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 30 01:57:14.318759765 (586h28m14.1898428s) [noise=true , address=45.91.116.8:4048 , piece_id=UTE4QMRHNQQZE7B6M6TJ5WCRP3O3SD5DEVUCVHJJS5F7LPBR4ASQ, seg_pos={0 0}, upload=514117]: Long tail cancellation due to optimal threshold Dec 30 01:57:14.318911545 (586h28m14.18999458s) [noise=true , address=77.90.31.11:4555 , piece_id=E4CXSIVAPSAOX2OTSLBCQRR3A7Z27I76QOIRPOPG6YJ2WITWHR5Q, seg_pos={0 0}, upload=514117]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Dec 30 01:57:14.318918949 (586h28m14.190001984s) [noise=true , address=77.90.31.11:4555 , piece_id=E4CXSIVAPSAOX2OTSLBCQRR3A7Z27I76QOIRPOPG6YJ2WITWHR5Q, seg_pos={0 0}, upload=514117]: Long tail cancellation due to optimal threshold Dec 30 01:57:14.318939818 (586h28m14.190022853s) [seg_pos={0 0}, upload=514117]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 01:57:14.319006966 (586h28m14.190090001s) [upload=514117]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:57:14.319011665 (586h28m14.1900947s) [upload=514117]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:57:14.602030556 (586h28m14.473113601s) [upload=514117]: Done uploading segments... Dec 30 01:57:14.602051265 (586h28m14.47313431s) [upload=514117]: Upload finished. err: <nil> Duration: 30.002420332s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 30.379537556s trace: https://jaeger.dev.storj.io/trace/58b94359c097ef0d FAIL (2025-12-30 01:57:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.221968961 (585h44m42.093052017s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222491837 (585h44m42.093574882s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222877261 (585h44m42.093960306s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223249098 (585h44m42.094332143s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223637930 (585h44m42.094720965s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224032200 (585h44m42.095115245s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224444876 (585h44m42.095527921s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224905332 (585h44m42.095988378s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225388433 (585h44m42.096471478s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225981221 (585h44m42.097064256s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227130389 (585h44m42.098213435s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227655670 (585h44m42.098738705s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228177764 (585h44m42.099260799s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228673828 (585h44m42.099756863s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:43.678892099 (585h44m43.549975134s) [upload=513482]: Next returned nil segment Dec 30 01:13:43.678910774 (585h44m43.549993819s) [upload=513482]: Waiting for error group managing segments... Dec 30 01:13:44.076462870 (585h44m43.947545905s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.083306310 (585h44m43.954389345s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.107556817 (585h44m43.978639852s) [noise=true , address=145.79.227.9:4391 , piece_id=4AVX4RQJ65H3VL7U3MW6MUYHGRK6YS7GOGZO4XVVG6YYTHFHAXDQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.147374754 (585h44m44.018457799s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.212624668 (585h44m44.083707703s) [noise=true , address=64.49.13.170:4721 , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.237525922 (585h44m44.108608957s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.242637575 (585h44m44.11372063s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.283510060 (585h44m44.154593095s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.340441716 (585h44m44.211524761s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.408662737 (585h44m44.279745772s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.962802591 (585h44m44.833885626s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.963549243 (585h44m44.834632288s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.017193407 (585h44m44.888276442s) [noise=true , address=145.79.227.44:4218 , piece_id=UBAPWL2SMR7FJ2AHQ5XHMX3H6NKDFZ23HEDFRMJLJFYZKGC4X4MA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093476639 (585h44m44.964559684s) [noise=true , address=145.79.227.49:4370 , piece_id=RZK3XQONM7S2XSOIO4EXITQ3XQGVYLIJNK66RI6LL5NMFG46NOPA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093501687 (585h44m44.964584722s) [seg_pos={0 0}, upload=513482]: Segment reached optimal threshold of 14 pieces. Dec 30 01:13:45.093670669 (585h44m44.964753704s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection Dec 30 01:13:45.093677973 (585h44m44.964761008s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093727678 (585h44m44.964810713s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093735993 (585h44m44.964819028s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093827067 (585h44m44.964910122s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UP8PxqeK7t24PXjopH1nziHYQFnsxhkamSHz8dmwcbuDxkAF6): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093833129 (585h44m44.964916174s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093916176 (585h44m44.964999211s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qZC1tmCvCadnm1fYZxR3kc4fonZUJPzvfTKkTFiDtccW5bo9T): protocol: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection Dec 30 01:13:45.093921096 (585h44m44.965004141s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093966583 (585h44m44.965049618s) [seg_pos={0 0}, upload=513482]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 01:13:45.094031926 (585h44m44.965114961s) [upload=513482]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:13:45.094036336 (585h44m44.965119371s) [upload=513482]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:13:45.389410141 (585h44m45.260493176s) [upload=513482]: Done uploading segments... Dec 30 01:13:45.389430891 (585h44m45.260513926s) [upload=513482]: Upload finished. err: <nil> Duration: 30.379564398s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................................................X...] duration: 3.575976141s trace: https://jaeger.dev.storj.io/trace/1ac298e20b765731 FAIL (2025-12-30 01:57:28 UTC) orbiter warning: context deadline exceeded Duration: 3.576002571s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.............XX.....X.X......X.X..XX.X....X..XXX..XX.....X..] duration: 16.648459815s trace: https://jaeger.dev.storj.io/trace/1aa3b31e9fcd916c FAIL (2025-12-30 01:57:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: 2ENM5URTR3GCU4MACTJKNQZOBC7DH4EWXEJGAM5SHIBEFBRFDQ2Q) context deadline exceeded; piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: 6NLLVTJIMFPPUNHQVSGVYEKKMBIJTADJLRKK2FOYMAJHBTR7VFBQ) context deadline exceeded ======LOGS====== 01:55:26.264693797 (586h26m26.135776842s) [upload=514094]: Starting upload Dec 30 01:55:26.264768260 (586h26m26.135851295s) [upload=514094]: Done starting upload Dec 30 01:55:26.264868770 (586h26m26.135951826s) [upload=514094]: Uploading segments... Dec 30 01:55:26.265054515 (586h26m26.13613755s) [upload=514094]: Got next segment. Inline: false Dec 30 01:55:26.265090924 (586h26m26.136173959s) [upload=514094]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 01:55:26.265096034 (586h26m26.136179069s) [upload=514094]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 01:55:26.508102931 (586h26m26.379185966s) [seg_pos={0 0}, upload=514094]: Begin upload segment... Dec 30 01:55:26.508145091 (586h26m26.379228126s) [seg_pos={0 0}, upload=514094]: Done begin upload segment. Dec 30 01:55:26.508171071 (586h26m26.379254106s) [noise=true , address=77.90.31.14:4664 , piece_id=6NLLVTJIMFPPUNHQVSGVYEKKMBIJTADJLRKK2FOYMAJHBTR7VFBQ, seg_pos={0 0}, upload=514094]: Uploading piece... Dec 30 01:55:26.508250702 (586h26m26.379333767s) [noise=true , address=145.79.227.13:4804 , piece_id=6BEVDZ6A6BKLXHI2PLKZLRFCNASNVD7AIPM5AKPPKDFCWOVYLX7A, seg_pos={0 0}, upload=514094]: Uploading piece... Dec 30 01:55:26.508748820 (586h26m26.379831855s) [noise=true , address=145.79.227.17:4971 , piece_id=CEJPYJ54C4K7YS4BQ7L6BR4P2NCGEYVCE7UJFSF6IHXTHK2NO6IQ, seg_pos={0 0}, upload=514094]: Uploading piece... Dec 30 01:55:26.509099648 (586h26m26.380182683s) [noise=true , address=145.79.227.7:4293 , piece_id=KYDSY3QIIWMK3KWL7OTWVJPOQ6YJT5JFZALCCYDG3F7VKTHCESWA, seg_pos={0 0}, upload=514094]: Uploading piece... Dec 30 01:55:26.509393218 (586h26m26.380476253s) [noise=true , address=145.79.227.12:4767 , piece_id=H6IWHZQFIHZAK5QIHZMBJF37ZVHX4HHCMSJU35CD26EDWGXHXXIA, seg_pos={0 0}, upload=514094]: Uploading piece... Dec 30 01:55:26.509679443 (586h26m26.380762478s) [noise=true , address=145.79.227.42:4121 , piece_id=W637QJFKZ3I7RKI62AWX3RVHPU24LJGKH3Y75FQ2QKO7Z4F6MGGA, seg_pos={0 0}, upload=514094]: Uploading piece... Dec 30 01:55:26.510089403 (586h26m26.381172448s) [noise=true , address=145.79.227.3:4150 , piece_id=GU66EJDW2T6CUANS24JPNOBY5ZEH53EFIXN4S6ZIOKAJU6ZVN4MQ, seg_pos={0 0}, upload=514094]: Uploading piece... Dec 30 01:55:26.510402118 (586h26m26.381485163s) [noise=true , address=145.79.227.32:4834 , piece_id=EM44RVNXZ7MAW7RYLSRSEUT3HGBQDMPLBEPEOIQJZX4EAFDCIBDA, seg_pos={0 0}, upload=514094]: Uploading piece... Dec 30 01:55:26.510723771 (586h26m26.381806806s) [noise=true , address=145.79.227.46:4601 , piece_id=DBOEH5RTMFLF7GGS2YTGHATNSBT3ODYCR6B2WSZKHTHZJ67QEYZQ, seg_pos={0 0}, upload=514094]: Uploading piece... Dec 30 01:55:26.511615949 (586h26m26.382699005s) [noise=true , address=145.79.227.35:4481 , piece_id=XTHNACNP5IGF7EF7ERMZDT6A3FAXFWE6QXHBLZTMBCD6GSLZW7IQ, seg_pos={0 0}, upload=514094]: Uploading piece... Dec 30 01:55:26.513000306 (586h26m26.384083351s) [noise=true , address=145.79.227.45:4229 , piece_id=GJKDS3CZ5OOTIPMSP3LU3X6HIRRX2AR7TXOS44ZSSMHIPOW7WN6A, seg_pos={0 0}, upload=514094]: Uploading piece... Dec 30 01:55:26.514708609 (586h26m26.385791654s) [noise=true , address=145.79.227.39:4013 , piece_id=P7CTN2Q56TZAMQ7Z5MKP7NCWA73KXA6VVIYR54S4BNHMT2BFKO4Q, seg_pos={0 0}, upload=514094]: Uploading piece... Dec 30 01:55:26.516561266 (586h26m26.387644301s) [noise=true , address=45.91.116.5:4297 , piece_id=2ENM5URTR3GCU4MACTJKNQZOBC7DH4EWXEJGAM5SHIBEFBRFDQ2Q, seg_pos={0 0}, upload=514094]: Uploading piece... Dec 30 01:55:26.518322920 (586h26m26.389405955s) [noise=true , address=145.79.227.51:4410 , piece_id=N3JIXZE4TAAI2PBSHQ24NEEILLXM4YIFON5YVH7O5Z34ZLPNG5BQ, seg_pos={0 0}, upload=514094]: Uploading piece... Dec 30 01:55:26.519685224 (586h26m26.390768269s) [noise=true , address=45.91.116.6:4974 , piece_id=6ZEQRXS5TTQUXERD5FPHUBJ6YYDIVFKLBMQFSXD7F5DGN62UZZHA, seg_pos={0 0}, upload=514094]: Uploading piece... Dec 30 01:55:26.520165879 (586h26m26.391248914s) [noise=true , address=77.90.31.11:4553 , piece_id=4GCKYV7R7LGX2R53XBMVXFTG73XJU2EQ7XOUHYI4M3Q6PJT3TCZQ, seg_pos={0 0}, upload=514094]: Uploading piece... Dec 30 01:55:28.113351882 (586h26m27.984434917s) [upload=514094]: Next returned nil segment Dec 30 01:55:28.113366409 (586h26m27.984449444s) [upload=514094]: Waiting for error group managing segments... Dec 30 01:55:28.489714050 (586h26m28.360797085s) [noise=true , address=145.79.227.51:4410 , piece_id=N3JIXZE4TAAI2PBSHQ24NEEILLXM4YIFON5YVH7O5Z34ZLPNG5BQ, seg_pos={0 0}, upload=514094]: Done uploading piece. err: <nil> Dec 30 01:55:28.495852478 (586h26m28.366935513s) [noise=true , address=145.79.227.13:4804 , piece_id=6BEVDZ6A6BKLXHI2PLKZLRFCNASNVD7AIPM5AKPPKDFCWOVYLX7A, seg_pos={0 0}, upload=514094]: Done uploading piece. err: <nil> Dec 30 01:55:28.496659264 (586h26m28.367742299s) [noise=true , address=145.79.227.7:4293 , piece_id=KYDSY3QIIWMK3KWL7OTWVJPOQ6YJT5JFZALCCYDG3F7VKTHCESWA, seg_pos={0 0}, upload=514094]: Done uploading piece. err: <nil> Dec 30 01:55:28.499030169 (586h26m28.370113204s) [noise=true , address=145.79.227.17:4971 , piece_id=CEJPYJ54C4K7YS4BQ7L6BR4P2NCGEYVCE7UJFSF6IHXTHK2NO6IQ, seg_pos={0 0}, upload=514094]: Done uploading piece. err: <nil> Dec 30 01:55:28.500746907 (586h26m28.371829982s) [noise=true , address=145.79.227.32:4834 , piece_id=EM44RVNXZ7MAW7RYLSRSEUT3HGBQDMPLBEPEOIQJZX4EAFDCIBDA, seg_pos={0 0}, upload=514094]: Done uploading piece. err: <nil> Dec 30 01:55:28.504146861 (586h26m28.375229906s) [noise=true , address=145.79.227.39:4013 , piece_id=P7CTN2Q56TZAMQ7Z5MKP7NCWA73KXA6VVIYR54S4BNHMT2BFKO4Q, seg_pos={0 0}, upload=514094]: Done uploading piece. err: <nil> Dec 30 01:55:28.512015273 (586h26m28.383098309s) [noise=true , address=145.79.227.12:4767 , piece_id=H6IWHZQFIHZAK5QIHZMBJF37ZVHX4HHCMSJU35CD26EDWGXHXXIA, seg_pos={0 0}, upload=514094]: Done uploading piece. err: <nil> Dec 30 01:55:28.512474377 (586h26m28.383557413s) [noise=true , address=145.79.227.46:4601 , piece_id=DBOEH5RTMFLF7GGS2YTGHATNSBT3ODYCR6B2WSZKHTHZJ67QEYZQ, seg_pos={0 0}, upload=514094]: Done uploading piece. err: <nil> Dec 30 01:55:28.516709301 (586h26m28.387792346s) [noise=true , address=145.79.227.42:4121 , piece_id=W637QJFKZ3I7RKI62AWX3RVHPU24LJGKH3Y75FQ2QKO7Z4F6MGGA, seg_pos={0 0}, upload=514094]: Done uploading piece. err: <nil> Dec 30 01:55:28.519143556 (586h26m28.390226611s) [noise=true , address=145.79.227.3:4150 , piece_id=GU66EJDW2T6CUANS24JPNOBY5ZEH53EFIXN4S6ZIOKAJU6ZVN4MQ, seg_pos={0 0}, upload=514094]: Done uploading piece. err: <nil> Dec 30 01:55:28.522158937 (586h26m28.393241972s) [noise=true , address=145.79.227.45:4229 , piece_id=GJKDS3CZ5OOTIPMSP3LU3X6HIRRX2AR7TXOS44ZSSMHIPOW7WN6A, seg_pos={0 0}, upload=514094]: Done uploading piece. err: <nil> Dec 30 01:55:28.717343337 (586h26m28.588426402s) [noise=true , address=77.90.31.14:4664 , piece_id=6NLLVTJIMFPPUNHQVSGVYEKKMBIJTADJLRKK2FOYMAJHBTR7VFBQ, seg_pos={0 0}, upload=514094]: Done uploading piece. err: <nil> Dec 30 01:55:28.786975082 (586h26m28.658058117s) [noise=true , address=45.91.116.5:4297 , piece_id=2ENM5URTR3GCU4MACTJKNQZOBC7DH4EWXEJGAM5SHIBEFBRFDQ2Q, seg_pos={0 0}, upload=514094]: Done uploading piece. err: <nil> Dec 30 01:55:28.786995331 (586h26m28.658078376s) [seg_pos={0 0}, upload=514094]: Segment reached optimal threshold of 13 pieces. Dec 30 01:55:28.787182006 (586h26m28.658265051s) [noise=true , address=77.90.31.11:4553 , piece_id=4GCKYV7R7LGX2R53XBMVXFTG73XJU2EQ7XOUHYI4M3Q6PJT3TCZQ, seg_pos={0 0}, upload=514094]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 30 01:55:28.787188408 (586h26m28.658271453s) [noise=true , address=77.90.31.11:4553 , piece_id=4GCKYV7R7LGX2R53XBMVXFTG73XJU2EQ7XOUHYI4M3Q6PJT3TCZQ, seg_pos={0 0}, upload=514094]: Long tail cancellation due to optimal threshold Dec 30 01:55:28.787225378 (586h26m28.658308423s) [noise=true , address=45.91.116.6:4974 , piece_id=6ZEQRXS5TTQUXERD5FPHUBJ6YYDIVFKLBMQFSXD7F5DGN62UZZHA, seg_pos={0 0}, upload=514094]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:35214->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:35214->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35214->45.91.116.6:4974: use of closed network connection Dec 30 01:55:28.787234876 (586h26m28.658317921s) [noise=true , address=45.91.116.6:4974 , piece_id=6ZEQRXS5TTQUXERD5FPHUBJ6YYDIVFKLBMQFSXD7F5DGN62UZZHA, seg_pos={0 0}, upload=514094]: Long tail cancellation due to optimal threshold Dec 30 01:55:28.787382778 (586h26m28.658465823s) [noise=true , address=145.79.227.35:4481 , piece_id=XTHNACNP5IGF7EF7ERMZDT6A3FAXFWE6QXHBLZTMBCD6GSLZW7IQ, seg_pos={0 0}, upload=514094]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 30 01:55:28.787388960 (586h26m28.658471995s) [noise=true , address=145.79.227.35:4481 , piece_id=XTHNACNP5IGF7EF7ERMZDT6A3FAXFWE6QXHBLZTMBCD6GSLZW7IQ, seg_pos={0 0}, upload=514094]: Long tail cancellation due to optimal threshold Dec 30 01:55:28.787402736 (586h26m28.658485781s) [seg_pos={0 0}, upload=514094]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 01:55:28.787461819 (586h26m28.658544864s) [upload=514094]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:55:28.787467429 (586h26m28.658550464s) [upload=514094]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:55:29.079428950 (586h26m28.950511995s) [upload=514094]: Done uploading segments... Dec 30 01:55:29.079451724 (586h26m28.950534769s) [upload=514094]: Upload finished. err: <nil> Duration: 16.648484683s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................................................X..] duration: 3.454900988s trace: https://jaeger.dev.storj.io/trace/1ac298e20b765731 FAIL (2025-12-30 01:56:57 UTC) orbiter warning: context deadline exceeded Duration: 3.454934261s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [..............XX.....X.X......X.X..XX.X....X..XXX..XX.....X.] duration: 13.06857636s trace: https://jaeger.dev.storj.io/trace/1aa3b31e9fcd916c FAIL (2025-12-30 01:56:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: 2ENM5URTR3GCU4MACTJKNQZOBC7DH4EWXEJGAM5SHIBEFBRFDQ2Q) context deadline exceeded; piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: 6NLLVTJIMFPPUNHQVSGVYEKKMBIJTADJLRKK2FOYMAJHBTR7VFBQ) context deadline exceeded ======LOGS====== 01:55:26.264693797 (586h26m26.135776842s) [upload=514094]: Starting upload Dec 30 01:55:26.264768260 (586h26m26.135851295s) [upload=514094]: Done starting upload Dec 30 01:55:26.264868770 (586h26m26.135951826s) [upload=514094]: Uploading segments... Dec 30 01:55:26.265054515 (586h26m26.13613755s) [upload=514094]: Got next segment. Inline: false Dec 30 01:55:26.265090924 (586h26m26.136173959s) [upload=514094]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 01:55:26.265096034 (586h26m26.136179069s) [upload=514094]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 01:55:26.508102931 (586h26m26.379185966s) [seg_pos={0 0}, upload=514094]: Begin upload segment... Dec 30 01:55:26.508145091 (586h26m26.379228126s) [seg_pos={0 0}, upload=514094]: Done begin upload segment. Dec 30 01:55:26.508171071 (586h26m26.379254106s) [noise=true , address=77.90.31.14:4664 , piece_id=6NLLVTJIMFPPUNHQVSGVYEKKMBIJTADJLRKK2FOYMAJHBTR7VFBQ, seg_pos={0 0}, upload=514094]: Uploading piece... Dec 30 01:55:26.508250702 (586h26m26.379333767s) [noise=true , address=145.79.227.13:4804 , piece_id=6BEVDZ6A6BKLXHI2PLKZLRFCNASNVD7AIPM5AKPPKDFCWOVYLX7A, seg_pos={0 0}, upload=514094]: Uploading piece... Dec 30 01:55:26.508748820 (586h26m26.379831855s) [noise=true , address=145.79.227.17:4971 , piece_id=CEJPYJ54C4K7YS4BQ7L6BR4P2NCGEYVCE7UJFSF6IHXTHK2NO6IQ, seg_pos={0 0}, upload=514094]: Uploading piece... Dec 30 01:55:26.509099648 (586h26m26.380182683s) [noise=true , address=145.79.227.7:4293 , piece_id=KYDSY3QIIWMK3KWL7OTWVJPOQ6YJT5JFZALCCYDG3F7VKTHCESWA, seg_pos={0 0}, upload=514094]: Uploading piece... Dec 30 01:55:26.509393218 (586h26m26.380476253s) [noise=true , address=145.79.227.12:4767 , piece_id=H6IWHZQFIHZAK5QIHZMBJF37ZVHX4HHCMSJU35CD26EDWGXHXXIA, seg_pos={0 0}, upload=514094]: Uploading piece... Dec 30 01:55:26.509679443 (586h26m26.380762478s) [noise=true , address=145.79.227.42:4121 , piece_id=W637QJFKZ3I7RKI62AWX3RVHPU24LJGKH3Y75FQ2QKO7Z4F6MGGA, seg_pos={0 0}, upload=514094]: Uploading piece... Dec 30 01:55:26.510089403 (586h26m26.381172448s) [noise=true , address=145.79.227.3:4150 , piece_id=GU66EJDW2T6CUANS24JPNOBY5ZEH53EFIXN4S6ZIOKAJU6ZVN4MQ, seg_pos={0 0}, upload=514094]: Uploading piece... Dec 30 01:55:26.510402118 (586h26m26.381485163s) [noise=true , address=145.79.227.32:4834 , piece_id=EM44RVNXZ7MAW7RYLSRSEUT3HGBQDMPLBEPEOIQJZX4EAFDCIBDA, seg_pos={0 0}, upload=514094]: Uploading piece... Dec 30 01:55:26.510723771 (586h26m26.381806806s) [noise=true , address=145.79.227.46:4601 , piece_id=DBOEH5RTMFLF7GGS2YTGHATNSBT3ODYCR6B2WSZKHTHZJ67QEYZQ, seg_pos={0 0}, upload=514094]: Uploading piece... Dec 30 01:55:26.511615949 (586h26m26.382699005s) [noise=true , address=145.79.227.35:4481 , piece_id=XTHNACNP5IGF7EF7ERMZDT6A3FAXFWE6QXHBLZTMBCD6GSLZW7IQ, seg_pos={0 0}, upload=514094]: Uploading piece... Dec 30 01:55:26.513000306 (586h26m26.384083351s) [noise=true , address=145.79.227.45:4229 , piece_id=GJKDS3CZ5OOTIPMSP3LU3X6HIRRX2AR7TXOS44ZSSMHIPOW7WN6A, seg_pos={0 0}, upload=514094]: Uploading piece... Dec 30 01:55:26.514708609 (586h26m26.385791654s) [noise=true , address=145.79.227.39:4013 , piece_id=P7CTN2Q56TZAMQ7Z5MKP7NCWA73KXA6VVIYR54S4BNHMT2BFKO4Q, seg_pos={0 0}, upload=514094]: Uploading piece... Dec 30 01:55:26.516561266 (586h26m26.387644301s) [noise=true , address=45.91.116.5:4297 , piece_id=2ENM5URTR3GCU4MACTJKNQZOBC7DH4EWXEJGAM5SHIBEFBRFDQ2Q, seg_pos={0 0}, upload=514094]: Uploading piece... Dec 30 01:55:26.518322920 (586h26m26.389405955s) [noise=true , address=145.79.227.51:4410 , piece_id=N3JIXZE4TAAI2PBSHQ24NEEILLXM4YIFON5YVH7O5Z34ZLPNG5BQ, seg_pos={0 0}, upload=514094]: Uploading piece... Dec 30 01:55:26.519685224 (586h26m26.390768269s) [noise=true , address=45.91.116.6:4974 , piece_id=6ZEQRXS5TTQUXERD5FPHUBJ6YYDIVFKLBMQFSXD7F5DGN62UZZHA, seg_pos={0 0}, upload=514094]: Uploading piece... Dec 30 01:55:26.520165879 (586h26m26.391248914s) [noise=true , address=77.90.31.11:4553 , piece_id=4GCKYV7R7LGX2R53XBMVXFTG73XJU2EQ7XOUHYI4M3Q6PJT3TCZQ, seg_pos={0 0}, upload=514094]: Uploading piece... Dec 30 01:55:28.113351882 (586h26m27.984434917s) [upload=514094]: Next returned nil segment Dec 30 01:55:28.113366409 (586h26m27.984449444s) [upload=514094]: Waiting for error group managing segments... Dec 30 01:55:28.489714050 (586h26m28.360797085s) [noise=true , address=145.79.227.51:4410 , piece_id=N3JIXZE4TAAI2PBSHQ24NEEILLXM4YIFON5YVH7O5Z34ZLPNG5BQ, seg_pos={0 0}, upload=514094]: Done uploading piece. err: <nil> Dec 30 01:55:28.495852478 (586h26m28.366935513s) [noise=true , address=145.79.227.13:4804 , piece_id=6BEVDZ6A6BKLXHI2PLKZLRFCNASNVD7AIPM5AKPPKDFCWOVYLX7A, seg_pos={0 0}, upload=514094]: Done uploading piece. err: <nil> Dec 30 01:55:28.496659264 (586h26m28.367742299s) [noise=true , address=145.79.227.7:4293 , piece_id=KYDSY3QIIWMK3KWL7OTWVJPOQ6YJT5JFZALCCYDG3F7VKTHCESWA, seg_pos={0 0}, upload=514094]: Done uploading piece. err: <nil> Dec 30 01:55:28.499030169 (586h26m28.370113204s) [noise=true , address=145.79.227.17:4971 , piece_id=CEJPYJ54C4K7YS4BQ7L6BR4P2NCGEYVCE7UJFSF6IHXTHK2NO6IQ, seg_pos={0 0}, upload=514094]: Done uploading piece. err: <nil> Dec 30 01:55:28.500746907 (586h26m28.371829982s) [noise=true , address=145.79.227.32:4834 , piece_id=EM44RVNXZ7MAW7RYLSRSEUT3HGBQDMPLBEPEOIQJZX4EAFDCIBDA, seg_pos={0 0}, upload=514094]: Done uploading piece. err: <nil> Dec 30 01:55:28.504146861 (586h26m28.375229906s) [noise=true , address=145.79.227.39:4013 , piece_id=P7CTN2Q56TZAMQ7Z5MKP7NCWA73KXA6VVIYR54S4BNHMT2BFKO4Q, seg_pos={0 0}, upload=514094]: Done uploading piece. err: <nil> Dec 30 01:55:28.512015273 (586h26m28.383098309s) [noise=true , address=145.79.227.12:4767 , piece_id=H6IWHZQFIHZAK5QIHZMBJF37ZVHX4HHCMSJU35CD26EDWGXHXXIA, seg_pos={0 0}, upload=514094]: Done uploading piece. err: <nil> Dec 30 01:55:28.512474377 (586h26m28.383557413s) [noise=true , address=145.79.227.46:4601 , piece_id=DBOEH5RTMFLF7GGS2YTGHATNSBT3ODYCR6B2WSZKHTHZJ67QEYZQ, seg_pos={0 0}, upload=514094]: Done uploading piece. err: <nil> Dec 30 01:55:28.516709301 (586h26m28.387792346s) [noise=true , address=145.79.227.42:4121 , piece_id=W637QJFKZ3I7RKI62AWX3RVHPU24LJGKH3Y75FQ2QKO7Z4F6MGGA, seg_pos={0 0}, upload=514094]: Done uploading piece. err: <nil> Dec 30 01:55:28.519143556 (586h26m28.390226611s) [noise=true , address=145.79.227.3:4150 , piece_id=GU66EJDW2T6CUANS24JPNOBY5ZEH53EFIXN4S6ZIOKAJU6ZVN4MQ, seg_pos={0 0}, upload=514094]: Done uploading piece. err: <nil> Dec 30 01:55:28.522158937 (586h26m28.393241972s) [noise=true , address=145.79.227.45:4229 , piece_id=GJKDS3CZ5OOTIPMSP3LU3X6HIRRX2AR7TXOS44ZSSMHIPOW7WN6A, seg_pos={0 0}, upload=514094]: Done uploading piece. err: <nil> Dec 30 01:55:28.717343337 (586h26m28.588426402s) [noise=true , address=77.90.31.14:4664 , piece_id=6NLLVTJIMFPPUNHQVSGVYEKKMBIJTADJLRKK2FOYMAJHBTR7VFBQ, seg_pos={0 0}, upload=514094]: Done uploading piece. err: <nil> Dec 30 01:55:28.786975082 (586h26m28.658058117s) [noise=true , address=45.91.116.5:4297 , piece_id=2ENM5URTR3GCU4MACTJKNQZOBC7DH4EWXEJGAM5SHIBEFBRFDQ2Q, seg_pos={0 0}, upload=514094]: Done uploading piece. err: <nil> Dec 30 01:55:28.786995331 (586h26m28.658078376s) [seg_pos={0 0}, upload=514094]: Segment reached optimal threshold of 13 pieces. Dec 30 01:55:28.787182006 (586h26m28.658265051s) [noise=true , address=77.90.31.11:4553 , piece_id=4GCKYV7R7LGX2R53XBMVXFTG73XJU2EQ7XOUHYI4M3Q6PJT3TCZQ, seg_pos={0 0}, upload=514094]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 30 01:55:28.787188408 (586h26m28.658271453s) [noise=true , address=77.90.31.11:4553 , piece_id=4GCKYV7R7LGX2R53XBMVXFTG73XJU2EQ7XOUHYI4M3Q6PJT3TCZQ, seg_pos={0 0}, upload=514094]: Long tail cancellation due to optimal threshold Dec 30 01:55:28.787225378 (586h26m28.658308423s) [noise=true , address=45.91.116.6:4974 , piece_id=6ZEQRXS5TTQUXERD5FPHUBJ6YYDIVFKLBMQFSXD7F5DGN62UZZHA, seg_pos={0 0}, upload=514094]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:35214->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:35214->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35214->45.91.116.6:4974: use of closed network connection Dec 30 01:55:28.787234876 (586h26m28.658317921s) [noise=true , address=45.91.116.6:4974 , piece_id=6ZEQRXS5TTQUXERD5FPHUBJ6YYDIVFKLBMQFSXD7F5DGN62UZZHA, seg_pos={0 0}, upload=514094]: Long tail cancellation due to optimal threshold Dec 30 01:55:28.787382778 (586h26m28.658465823s) [noise=true , address=145.79.227.35:4481 , piece_id=XTHNACNP5IGF7EF7ERMZDT6A3FAXFWE6QXHBLZTMBCD6GSLZW7IQ, seg_pos={0 0}, upload=514094]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 30 01:55:28.787388960 (586h26m28.658471995s) [noise=true , address=145.79.227.35:4481 , piece_id=XTHNACNP5IGF7EF7ERMZDT6A3FAXFWE6QXHBLZTMBCD6GSLZW7IQ, seg_pos={0 0}, upload=514094]: Long tail cancellation due to optimal threshold Dec 30 01:55:28.787402736 (586h26m28.658485781s) [seg_pos={0 0}, upload=514094]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 01:55:28.787461819 (586h26m28.658544864s) [upload=514094]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:55:28.787467429 (586h26m28.658550464s) [upload=514094]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:55:29.079428950 (586h26m28.950511995s) [upload=514094]: Done uploading segments... Dec 30 01:55:29.079451724 (586h26m28.950534769s) [upload=514094]: Upload finished. err: <nil> Duration: 13.068600194s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 18.443838016s trace: https://jaeger.dev.storj.io/trace/58b94359c097ef0d FAIL (2025-12-30 01:56:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.221968961 (585h44m42.093052017s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222491837 (585h44m42.093574882s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222877261 (585h44m42.093960306s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223249098 (585h44m42.094332143s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223637930 (585h44m42.094720965s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224032200 (585h44m42.095115245s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224444876 (585h44m42.095527921s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224905332 (585h44m42.095988378s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225388433 (585h44m42.096471478s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225981221 (585h44m42.097064256s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227130389 (585h44m42.098213435s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227655670 (585h44m42.098738705s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228177764 (585h44m42.099260799s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228673828 (585h44m42.099756863s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:43.678892099 (585h44m43.549975134s) [upload=513482]: Next returned nil segment Dec 30 01:13:43.678910774 (585h44m43.549993819s) [upload=513482]: Waiting for error group managing segments... Dec 30 01:13:44.076462870 (585h44m43.947545905s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.083306310 (585h44m43.954389345s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.107556817 (585h44m43.978639852s) [noise=true , address=145.79.227.9:4391 , piece_id=4AVX4RQJ65H3VL7U3MW6MUYHGRK6YS7GOGZO4XVVG6YYTHFHAXDQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.147374754 (585h44m44.018457799s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.212624668 (585h44m44.083707703s) [noise=true , address=64.49.13.170:4721 , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.237525922 (585h44m44.108608957s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.242637575 (585h44m44.11372063s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.283510060 (585h44m44.154593095s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.340441716 (585h44m44.211524761s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.408662737 (585h44m44.279745772s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.962802591 (585h44m44.833885626s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.963549243 (585h44m44.834632288s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.017193407 (585h44m44.888276442s) [noise=true , address=145.79.227.44:4218 , piece_id=UBAPWL2SMR7FJ2AHQ5XHMX3H6NKDFZ23HEDFRMJLJFYZKGC4X4MA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093476639 (585h44m44.964559684s) [noise=true , address=145.79.227.49:4370 , piece_id=RZK3XQONM7S2XSOIO4EXITQ3XQGVYLIJNK66RI6LL5NMFG46NOPA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093501687 (585h44m44.964584722s) [seg_pos={0 0}, upload=513482]: Segment reached optimal threshold of 14 pieces. Dec 30 01:13:45.093670669 (585h44m44.964753704s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection Dec 30 01:13:45.093677973 (585h44m44.964761008s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093727678 (585h44m44.964810713s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093735993 (585h44m44.964819028s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093827067 (585h44m44.964910122s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UP8PxqeK7t24PXjopH1nziHYQFnsxhkamSHz8dmwcbuDxkAF6): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093833129 (585h44m44.964916174s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093916176 (585h44m44.964999211s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qZC1tmCvCadnm1fYZxR3kc4fonZUJPzvfTKkTFiDtccW5bo9T): protocol: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection Dec 30 01:13:45.093921096 (585h44m44.965004141s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093966583 (585h44m44.965049618s) [seg_pos={0 0}, upload=513482]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 01:13:45.094031926 (585h44m44.965114961s) [upload=513482]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:13:45.094036336 (585h44m44.965119371s) [upload=513482]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:13:45.389410141 (585h44m45.260493176s) [upload=513482]: Done uploading segments... Dec 30 01:13:45.389430891 (585h44m45.260513926s) [upload=513482]: Upload finished. err: <nil> Duration: 18.443858414s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................................................X.] duration: 3.514090991s trace: https://jaeger.dev.storj.io/trace/1ac298e20b765731 FAIL (2025-12-30 01:56:15 UTC) orbiter warning: context deadline exceeded Duration: 3.514115978s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [...............XX.....X.X......X.X..XX.X....X..XXX..XX.....X] duration: 30.002570112s trace: https://jaeger.dev.storj.io/trace/1aa3b31e9fcd916c FAIL (2025-12-30 01:55:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: 2ENM5URTR3GCU4MACTJKNQZOBC7DH4EWXEJGAM5SHIBEFBRFDQ2Q) context deadline exceeded; piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: 6NLLVTJIMFPPUNHQVSGVYEKKMBIJTADJLRKK2FOYMAJHBTR7VFBQ) context deadline exceeded ======LOGS====== 01:55:26.264693797 (586h26m26.135776842s) [upload=514094]: Starting upload Dec 30 01:55:26.264768260 (586h26m26.135851295s) [upload=514094]: Done starting upload Dec 30 01:55:26.264868770 (586h26m26.135951826s) [upload=514094]: Uploading segments... Dec 30 01:55:26.265054515 (586h26m26.13613755s) [upload=514094]: Got next segment. Inline: false Dec 30 01:55:26.265090924 (586h26m26.136173959s) [upload=514094]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 01:55:26.265096034 (586h26m26.136179069s) [upload=514094]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 01:55:26.508102931 (586h26m26.379185966s) [seg_pos={0 0}, upload=514094]: Begin upload segment... Dec 30 01:55:26.508145091 (586h26m26.379228126s) [seg_pos={0 0}, upload=514094]: Done begin upload segment. Dec 30 01:55:26.508171071 (586h26m26.379254106s) [noise=true , address=77.90.31.14:4664 , piece_id=6NLLVTJIMFPPUNHQVSGVYEKKMBIJTADJLRKK2FOYMAJHBTR7VFBQ, seg_pos={0 0}, upload=514094]: Uploading piece... Dec 30 01:55:26.508250702 (586h26m26.379333767s) [noise=true , address=145.79.227.13:4804 , piece_id=6BEVDZ6A6BKLXHI2PLKZLRFCNASNVD7AIPM5AKPPKDFCWOVYLX7A, seg_pos={0 0}, upload=514094]: Uploading piece... Dec 30 01:55:26.508748820 (586h26m26.379831855s) [noise=true , address=145.79.227.17:4971 , piece_id=CEJPYJ54C4K7YS4BQ7L6BR4P2NCGEYVCE7UJFSF6IHXTHK2NO6IQ, seg_pos={0 0}, upload=514094]: Uploading piece... Dec 30 01:55:26.509099648 (586h26m26.380182683s) [noise=true , address=145.79.227.7:4293 , piece_id=KYDSY3QIIWMK3KWL7OTWVJPOQ6YJT5JFZALCCYDG3F7VKTHCESWA, seg_pos={0 0}, upload=514094]: Uploading piece... Dec 30 01:55:26.509393218 (586h26m26.380476253s) [noise=true , address=145.79.227.12:4767 , piece_id=H6IWHZQFIHZAK5QIHZMBJF37ZVHX4HHCMSJU35CD26EDWGXHXXIA, seg_pos={0 0}, upload=514094]: Uploading piece... Dec 30 01:55:26.509679443 (586h26m26.380762478s) [noise=true , address=145.79.227.42:4121 , piece_id=W637QJFKZ3I7RKI62AWX3RVHPU24LJGKH3Y75FQ2QKO7Z4F6MGGA, seg_pos={0 0}, upload=514094]: Uploading piece... Dec 30 01:55:26.510089403 (586h26m26.381172448s) [noise=true , address=145.79.227.3:4150 , piece_id=GU66EJDW2T6CUANS24JPNOBY5ZEH53EFIXN4S6ZIOKAJU6ZVN4MQ, seg_pos={0 0}, upload=514094]: Uploading piece... Dec 30 01:55:26.510402118 (586h26m26.381485163s) [noise=true , address=145.79.227.32:4834 , piece_id=EM44RVNXZ7MAW7RYLSRSEUT3HGBQDMPLBEPEOIQJZX4EAFDCIBDA, seg_pos={0 0}, upload=514094]: Uploading piece... Dec 30 01:55:26.510723771 (586h26m26.381806806s) [noise=true , address=145.79.227.46:4601 , piece_id=DBOEH5RTMFLF7GGS2YTGHATNSBT3ODYCR6B2WSZKHTHZJ67QEYZQ, seg_pos={0 0}, upload=514094]: Uploading piece... Dec 30 01:55:26.511615949 (586h26m26.382699005s) [noise=true , address=145.79.227.35:4481 , piece_id=XTHNACNP5IGF7EF7ERMZDT6A3FAXFWE6QXHBLZTMBCD6GSLZW7IQ, seg_pos={0 0}, upload=514094]: Uploading piece... Dec 30 01:55:26.513000306 (586h26m26.384083351s) [noise=true , address=145.79.227.45:4229 , piece_id=GJKDS3CZ5OOTIPMSP3LU3X6HIRRX2AR7TXOS44ZSSMHIPOW7WN6A, seg_pos={0 0}, upload=514094]: Uploading piece... Dec 30 01:55:26.514708609 (586h26m26.385791654s) [noise=true , address=145.79.227.39:4013 , piece_id=P7CTN2Q56TZAMQ7Z5MKP7NCWA73KXA6VVIYR54S4BNHMT2BFKO4Q, seg_pos={0 0}, upload=514094]: Uploading piece... Dec 30 01:55:26.516561266 (586h26m26.387644301s) [noise=true , address=45.91.116.5:4297 , piece_id=2ENM5URTR3GCU4MACTJKNQZOBC7DH4EWXEJGAM5SHIBEFBRFDQ2Q, seg_pos={0 0}, upload=514094]: Uploading piece... Dec 30 01:55:26.518322920 (586h26m26.389405955s) [noise=true , address=145.79.227.51:4410 , piece_id=N3JIXZE4TAAI2PBSHQ24NEEILLXM4YIFON5YVH7O5Z34ZLPNG5BQ, seg_pos={0 0}, upload=514094]: Uploading piece... Dec 30 01:55:26.519685224 (586h26m26.390768269s) [noise=true , address=45.91.116.6:4974 , piece_id=6ZEQRXS5TTQUXERD5FPHUBJ6YYDIVFKLBMQFSXD7F5DGN62UZZHA, seg_pos={0 0}, upload=514094]: Uploading piece... Dec 30 01:55:26.520165879 (586h26m26.391248914s) [noise=true , address=77.90.31.11:4553 , piece_id=4GCKYV7R7LGX2R53XBMVXFTG73XJU2EQ7XOUHYI4M3Q6PJT3TCZQ, seg_pos={0 0}, upload=514094]: Uploading piece... Dec 30 01:55:28.113351882 (586h26m27.984434917s) [upload=514094]: Next returned nil segment Dec 30 01:55:28.113366409 (586h26m27.984449444s) [upload=514094]: Waiting for error group managing segments... Dec 30 01:55:28.489714050 (586h26m28.360797085s) [noise=true , address=145.79.227.51:4410 , piece_id=N3JIXZE4TAAI2PBSHQ24NEEILLXM4YIFON5YVH7O5Z34ZLPNG5BQ, seg_pos={0 0}, upload=514094]: Done uploading piece. err: <nil> Dec 30 01:55:28.495852478 (586h26m28.366935513s) [noise=true , address=145.79.227.13:4804 , piece_id=6BEVDZ6A6BKLXHI2PLKZLRFCNASNVD7AIPM5AKPPKDFCWOVYLX7A, seg_pos={0 0}, upload=514094]: Done uploading piece. err: <nil> Dec 30 01:55:28.496659264 (586h26m28.367742299s) [noise=true , address=145.79.227.7:4293 , piece_id=KYDSY3QIIWMK3KWL7OTWVJPOQ6YJT5JFZALCCYDG3F7VKTHCESWA, seg_pos={0 0}, upload=514094]: Done uploading piece. err: <nil> Dec 30 01:55:28.499030169 (586h26m28.370113204s) [noise=true , address=145.79.227.17:4971 , piece_id=CEJPYJ54C4K7YS4BQ7L6BR4P2NCGEYVCE7UJFSF6IHXTHK2NO6IQ, seg_pos={0 0}, upload=514094]: Done uploading piece. err: <nil> Dec 30 01:55:28.500746907 (586h26m28.371829982s) [noise=true , address=145.79.227.32:4834 , piece_id=EM44RVNXZ7MAW7RYLSRSEUT3HGBQDMPLBEPEOIQJZX4EAFDCIBDA, seg_pos={0 0}, upload=514094]: Done uploading piece. err: <nil> Dec 30 01:55:28.504146861 (586h26m28.375229906s) [noise=true , address=145.79.227.39:4013 , piece_id=P7CTN2Q56TZAMQ7Z5MKP7NCWA73KXA6VVIYR54S4BNHMT2BFKO4Q, seg_pos={0 0}, upload=514094]: Done uploading piece. err: <nil> Dec 30 01:55:28.512015273 (586h26m28.383098309s) [noise=true , address=145.79.227.12:4767 , piece_id=H6IWHZQFIHZAK5QIHZMBJF37ZVHX4HHCMSJU35CD26EDWGXHXXIA, seg_pos={0 0}, upload=514094]: Done uploading piece. err: <nil> Dec 30 01:55:28.512474377 (586h26m28.383557413s) [noise=true , address=145.79.227.46:4601 , piece_id=DBOEH5RTMFLF7GGS2YTGHATNSBT3ODYCR6B2WSZKHTHZJ67QEYZQ, seg_pos={0 0}, upload=514094]: Done uploading piece. err: <nil> Dec 30 01:55:28.516709301 (586h26m28.387792346s) [noise=true , address=145.79.227.42:4121 , piece_id=W637QJFKZ3I7RKI62AWX3RVHPU24LJGKH3Y75FQ2QKO7Z4F6MGGA, seg_pos={0 0}, upload=514094]: Done uploading piece. err: <nil> Dec 30 01:55:28.519143556 (586h26m28.390226611s) [noise=true , address=145.79.227.3:4150 , piece_id=GU66EJDW2T6CUANS24JPNOBY5ZEH53EFIXN4S6ZIOKAJU6ZVN4MQ, seg_pos={0 0}, upload=514094]: Done uploading piece. err: <nil> Dec 30 01:55:28.522158937 (586h26m28.393241972s) [noise=true , address=145.79.227.45:4229 , piece_id=GJKDS3CZ5OOTIPMSP3LU3X6HIRRX2AR7TXOS44ZSSMHIPOW7WN6A, seg_pos={0 0}, upload=514094]: Done uploading piece. err: <nil> Dec 30 01:55:28.717343337 (586h26m28.588426402s) [noise=true , address=77.90.31.14:4664 , piece_id=6NLLVTJIMFPPUNHQVSGVYEKKMBIJTADJLRKK2FOYMAJHBTR7VFBQ, seg_pos={0 0}, upload=514094]: Done uploading piece. err: <nil> Dec 30 01:55:28.786975082 (586h26m28.658058117s) [noise=true , address=45.91.116.5:4297 , piece_id=2ENM5URTR3GCU4MACTJKNQZOBC7DH4EWXEJGAM5SHIBEFBRFDQ2Q, seg_pos={0 0}, upload=514094]: Done uploading piece. err: <nil> Dec 30 01:55:28.786995331 (586h26m28.658078376s) [seg_pos={0 0}, upload=514094]: Segment reached optimal threshold of 13 pieces. Dec 30 01:55:28.787182006 (586h26m28.658265051s) [noise=true , address=77.90.31.11:4553 , piece_id=4GCKYV7R7LGX2R53XBMVXFTG73XJU2EQ7XOUHYI4M3Q6PJT3TCZQ, seg_pos={0 0}, upload=514094]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 30 01:55:28.787188408 (586h26m28.658271453s) [noise=true , address=77.90.31.11:4553 , piece_id=4GCKYV7R7LGX2R53XBMVXFTG73XJU2EQ7XOUHYI4M3Q6PJT3TCZQ, seg_pos={0 0}, upload=514094]: Long tail cancellation due to optimal threshold Dec 30 01:55:28.787225378 (586h26m28.658308423s) [noise=true , address=45.91.116.6:4974 , piece_id=6ZEQRXS5TTQUXERD5FPHUBJ6YYDIVFKLBMQFSXD7F5DGN62UZZHA, seg_pos={0 0}, upload=514094]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:35214->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:35214->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35214->45.91.116.6:4974: use of closed network connection Dec 30 01:55:28.787234876 (586h26m28.658317921s) [noise=true , address=45.91.116.6:4974 , piece_id=6ZEQRXS5TTQUXERD5FPHUBJ6YYDIVFKLBMQFSXD7F5DGN62UZZHA, seg_pos={0 0}, upload=514094]: Long tail cancellation due to optimal threshold Dec 30 01:55:28.787382778 (586h26m28.658465823s) [noise=true , address=145.79.227.35:4481 , piece_id=XTHNACNP5IGF7EF7ERMZDT6A3FAXFWE6QXHBLZTMBCD6GSLZW7IQ, seg_pos={0 0}, upload=514094]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 30 01:55:28.787388960 (586h26m28.658471995s) [noise=true , address=145.79.227.35:4481 , piece_id=XTHNACNP5IGF7EF7ERMZDT6A3FAXFWE6QXHBLZTMBCD6GSLZW7IQ, seg_pos={0 0}, upload=514094]: Long tail cancellation due to optimal threshold Dec 30 01:55:28.787402736 (586h26m28.658485781s) [seg_pos={0 0}, upload=514094]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 01:55:28.787461819 (586h26m28.658544864s) [upload=514094]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:55:28.787467429 (586h26m28.658550464s) [upload=514094]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:55:29.079428950 (586h26m28.950511995s) [upload=514094]: Done uploading segments... Dec 30 01:55:29.079451724 (586h26m28.950534769s) [upload=514094]: Upload finished. err: <nil> Duration: 30.002590641s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................................................X] duration: 5.000995467s trace: https://jaeger.dev.storj.io/trace/1ac298e20b765731 FAIL (2025-12-30 01:55:47 UTC) orbiter warning: context deadline exceeded Duration: 5.00101759s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 27.253976648s trace: https://jaeger.dev.storj.io/trace/58b94359c097ef0d FAIL (2025-12-30 01:55:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.221968961 (585h44m42.093052017s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222491837 (585h44m42.093574882s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222877261 (585h44m42.093960306s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223249098 (585h44m42.094332143s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223637930 (585h44m42.094720965s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224032200 (585h44m42.095115245s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224444876 (585h44m42.095527921s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224905332 (585h44m42.095988378s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225388433 (585h44m42.096471478s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225981221 (585h44m42.097064256s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227130389 (585h44m42.098213435s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227655670 (585h44m42.098738705s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228177764 (585h44m42.099260799s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228673828 (585h44m42.099756863s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:43.678892099 (585h44m43.549975134s) [upload=513482]: Next returned nil segment Dec 30 01:13:43.678910774 (585h44m43.549993819s) [upload=513482]: Waiting for error group managing segments... Dec 30 01:13:44.076462870 (585h44m43.947545905s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.083306310 (585h44m43.954389345s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.107556817 (585h44m43.978639852s) [noise=true , address=145.79.227.9:4391 , piece_id=4AVX4RQJ65H3VL7U3MW6MUYHGRK6YS7GOGZO4XVVG6YYTHFHAXDQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.147374754 (585h44m44.018457799s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.212624668 (585h44m44.083707703s) [noise=true , address=64.49.13.170:4721 , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.237525922 (585h44m44.108608957s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.242637575 (585h44m44.11372063s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.283510060 (585h44m44.154593095s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.340441716 (585h44m44.211524761s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.408662737 (585h44m44.279745772s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.962802591 (585h44m44.833885626s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.963549243 (585h44m44.834632288s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.017193407 (585h44m44.888276442s) [noise=true , address=145.79.227.44:4218 , piece_id=UBAPWL2SMR7FJ2AHQ5XHMX3H6NKDFZ23HEDFRMJLJFYZKGC4X4MA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093476639 (585h44m44.964559684s) [noise=true , address=145.79.227.49:4370 , piece_id=RZK3XQONM7S2XSOIO4EXITQ3XQGVYLIJNK66RI6LL5NMFG46NOPA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093501687 (585h44m44.964584722s) [seg_pos={0 0}, upload=513482]: Segment reached optimal threshold of 14 pieces. Dec 30 01:13:45.093670669 (585h44m44.964753704s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection Dec 30 01:13:45.093677973 (585h44m44.964761008s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093727678 (585h44m44.964810713s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093735993 (585h44m44.964819028s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093827067 (585h44m44.964910122s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UP8PxqeK7t24PXjopH1nziHYQFnsxhkamSHz8dmwcbuDxkAF6): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093833129 (585h44m44.964916174s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093916176 (585h44m44.964999211s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qZC1tmCvCadnm1fYZxR3kc4fonZUJPzvfTKkTFiDtccW5bo9T): protocol: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection Dec 30 01:13:45.093921096 (585h44m44.965004141s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093966583 (585h44m44.965049618s) [seg_pos={0 0}, upload=513482]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 01:13:45.094031926 (585h44m44.965114961s) [upload=513482]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:13:45.094036336 (585h44m44.965119371s) [upload=513482]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:13:45.389410141 (585h44m45.260493176s) [upload=513482]: Done uploading segments... Dec 30 01:13:45.389430891 (585h44m45.260513926s) [upload=513482]: Upload finished. err: <nil> Duration: 27.254007576s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [................XX.....X.X......X.X..XX.X....X..XXX..XX.....] duration: 5.999536087s trace: https://jaeger.dev.storj.io/trace/20f64a83037a2393 FAIL (2025-12-30 01:54:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: MPFSPCD6ASY2LCASLCT27Z657RVUMVJXENIMNHDRPZ5T7BBV37DA) context deadline exceeded; piecestore: (Node ID: 12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, Piece ID: HS3ABRPPVQFR57P7JITBF3246G3PRNFUH2ZMETP5CYA5NILCNO2A) context deadline exceeded; piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: JN32F2DIJK6I6VOMYTYV3T74BDBDLZDMWJYULBL7HUTGFKV65ZKQ) context deadline exceeded ======LOGS====== Dec 30 01:50:40.234985867 (586h21m40.106068912s) [upload=514021]: Starting upload Dec 30 01:50:40.235076680 (586h21m40.106159715s) [upload=514021]: Done starting upload Dec 30 01:50:40.235117789 (586h21m40.106200834s) [upload=514021]: Uploading segments... Dec 30 01:50:40.235287161 (586h21m40.106370196s) [upload=514021]: Got next segment. Inline: false Dec 30 01:50:40.235319232 (586h21m40.106402277s) [upload=514021]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 01:50:40.235324442 (586h21m40.106407487s) [upload=514021]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 01:50:40.483051707 (586h21m40.354134752s) [seg_pos={0 0}, upload=514021]: Begin upload segment... Dec 30 01:50:40.483093266 (586h21m40.354176301s) [seg_pos={0 0}, upload=514021]: Done begin upload segment. Dec 30 01:50:40.483115268 (586h21m40.354198313s) [noise=true , address=45.91.116.10:4082 , piece_id=WJT3ZSNQEMYXZN6QSXO2GQFAB4LFOFGB5AAESZWQOUJ73TBQDKBA, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.483227201 (586h21m40.354310246s) [noise=true , address=145.79.227.16:4911 , piece_id=JRL7QZGGYZAIFGZEBHEZAYQIJTFEXB6KKO7I6Y2DUS2SIKBCJAYA, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.483583540 (586h21m40.354666575s) [noise=true , address=45.91.116.9:4264 , piece_id=RXE5E3Z7AEDER6YEDU3H2W75EOGL2QY32EQDMLUI3DAP5VJVQUTA, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.483856730 (586h21m40.354939765s) [noise=true , address=145.79.227.51:4410 , piece_id=GEAU6ZOVWUMQ53FIM4YQ2BNVR7DUMAAR5QBORU5OWQBYLLDV5SVQ, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.484116876 (586h21m40.355199921s) [noise=true , address=145.79.227.36:4517 , piece_id=7PZIHPSP5YHU7XW7QHGBY5CD4WT4PC7F6TRUVYF335TTY2V64VEA, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.484379576 (586h21m40.355462611s) [noise=true , address=145.79.227.50:4373 , piece_id=L7NUZASKLC6WI4ATPDC3LEOQFKWFXBDM3XE6SKTOX3BNFMH5WLKQ, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.484684196 (586h21m40.355767231s) [noise=true , address=145.79.227.7:4293 , piece_id=RX7KO2NUK6YZ4KXVO65BWLDIHBXPIFFHDSQ456PJGZNYHM73BIQA, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.484997943 (586h21m40.356080978s) [noise=true , address=45.91.116.8:4047 , piece_id=MPFSPCD6ASY2LCASLCT27Z657RVUMVJXENIMNHDRPZ5T7BBV37DA, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.485305078 (586h21m40.356388113s) [noise=true , address=77.90.31.21:4877 , piece_id=HS3ABRPPVQFR57P7JITBF3246G3PRNFUH2ZMETP5CYA5NILCNO2A, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.485646458 (586h21m40.356729503s) [noise=true , address=145.79.227.39:4015 , piece_id=RMKHO2QMH2BB4XLLNPXCNKDW4RKI5ZQUCC6YA5PTC4NQC7HREOFA, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.486823519 (586h21m40.357906554s) [noise=true , address=145.79.227.29:4941 , piece_id=OL5DN2T5BKKCUDZ7GRKFYVO7MZYZGG6LX5G4XSUVWEOGPEWT4VGQ, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.487968209 (586h21m40.359051254s) [noise=true , address=77.90.31.19:4844 , piece_id=COAEIFAFOH5VP26MFZSXX5H2R2HESU6TCHGN2EKVBFJ3A52SIRVA, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.489030822 (586h21m40.360113857s) [noise=true , address=145.79.227.34:4906 , piece_id=T7LU4RO6KLZRF3YVRCLM6YRO45KWJMSYRRQC25BUM6LFI7O7SJJA, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.490339584 (586h21m40.361422619s) [noise=true , address=145.79.227.2:4113 , piece_id=DYZMSLAACLNG2TCDLEKFO7APCXBFVRHRY2FLQR5Q7ANSG3K3EJBA, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.491705184 (586h21m40.362788229s) [noise=true , address=45.91.116.13:4154 , piece_id=JN32F2DIJK6I6VOMYTYV3T74BDBDLZDMWJYULBL7HUTGFKV65ZKQ, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.498535069 (586h21m40.369618104s) [noise=true , address=45.91.116.7:4012 , piece_id=6V33KTBZE3IBFOBRYKHTNG7VNOSJCYV4FLCQ2T5PNEUZQF5KQEJQ, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:42.180702501 (586h21m42.051785546s) [upload=514021]: Next returned nil segment Dec 30 01:50:42.180721156 (586h21m42.051804191s) [upload=514021]: Waiting for error group managing segments... Dec 30 01:50:42.461140760 (586h21m42.332223795s) [noise=true , address=145.79.227.7:4293 , piece_id=RX7KO2NUK6YZ4KXVO65BWLDIHBXPIFFHDSQ456PJGZNYHM73BIQA, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:42.463300643 (586h21m42.334383688s) [noise=true , address=145.79.227.16:4911 , piece_id=JRL7QZGGYZAIFGZEBHEZAYQIJTFEXB6KKO7I6Y2DUS2SIKBCJAYA, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:42.467721662 (586h21m42.338804707s) [noise=true , address=145.79.227.50:4373 , piece_id=L7NUZASKLC6WI4ATPDC3LEOQFKWFXBDM3XE6SKTOX3BNFMH5WLKQ, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:42.477585643 (586h21m42.348668678s) [noise=true , address=145.79.227.51:4410 , piece_id=GEAU6ZOVWUMQ53FIM4YQ2BNVR7DUMAAR5QBORU5OWQBYLLDV5SVQ, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:42.490245268 (586h21m42.361328303s) [noise=true , address=145.79.227.29:4941 , piece_id=OL5DN2T5BKKCUDZ7GRKFYVO7MZYZGG6LX5G4XSUVWEOGPEWT4VGQ, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:42.494154722 (586h21m42.365237757s) [noise=true , address=145.79.227.39:4015 , piece_id=RMKHO2QMH2BB4XLLNPXCNKDW4RKI5ZQUCC6YA5PTC4NQC7HREOFA, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:42.501438722 (586h21m42.372521767s) [noise=true , address=145.79.227.36:4517 , piece_id=7PZIHPSP5YHU7XW7QHGBY5CD4WT4PC7F6TRUVYF335TTY2V64VEA, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:42.502707718 (586h21m42.373790753s) [noise=true , address=145.79.227.2:4113 , piece_id=DYZMSLAACLNG2TCDLEKFO7APCXBFVRHRY2FLQR5Q7ANSG3K3EJBA, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:42.905016712 (586h21m42.776099747s) [noise=true , address=45.91.116.8:4047 , piece_id=MPFSPCD6ASY2LCASLCT27Z657RVUMVJXENIMNHDRPZ5T7BBV37DA, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:43.050110884 (586h21m42.921193919s) [noise=true , address=45.91.116.9:4264 , piece_id=RXE5E3Z7AEDER6YEDU3H2W75EOGL2QY32EQDMLUI3DAP5VJVQUTA, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:43.440086393 (586h21m43.311169438s) [noise=true , address=45.91.116.13:4154 , piece_id=JN32F2DIJK6I6VOMYTYV3T74BDBDLZDMWJYULBL7HUTGFKV65ZKQ, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:43.442366294 (586h21m43.313449339s) [noise=true , address=77.90.31.21:4877 , piece_id=HS3ABRPPVQFR57P7JITBF3246G3PRNFUH2ZMETP5CYA5NILCNO2A, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:43.449133169 (586h21m43.320216224s) [noise=true , address=45.91.116.10:4082 , piece_id=WJT3ZSNQEMYXZN6QSXO2GQFAB4LFOFGB5AAESZWQOUJ73TBQDKBA, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:43.449154068 (586h21m43.320237113s) [seg_pos={0 0}, upload=514021]: Segment reached optimal threshold of 13 pieces. Dec 30 01:50:43.449402903 (586h21m43.320485958s) [noise=true , address=45.91.116.7:4012 , piece_id=6V33KTBZE3IBFOBRYKHTNG7VNOSJCYV4FLCQ2T5PNEUZQF5KQEJQ, seg_pos={0 0}, upload=514021]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: expected piece hash; context canceled; EOF Dec 30 01:50:43.449414875 (586h21m43.32049792s) [noise=true , address=45.91.116.7:4012 , piece_id=6V33KTBZE3IBFOBRYKHTNG7VNOSJCYV4FLCQ2T5PNEUZQF5KQEJQ, seg_pos={0 0}, upload=514021]: Long tail cancellation due to optimal threshold Dec 30 01:50:43.449570110 (586h21m43.320653155s) [noise=true , address=77.90.31.19:4844 , piece_id=COAEIFAFOH5VP26MFZSXX5H2R2HESU6TCHGN2EKVBFJ3A52SIRVA, seg_pos={0 0}, upload=514021]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF Dec 30 01:50:43.449579980 (586h21m43.320663015s) [noise=true , address=77.90.31.19:4844 , piece_id=COAEIFAFOH5VP26MFZSXX5H2R2HESU6TCHGN2EKVBFJ3A52SIRVA, seg_pos={0 0}, upload=514021]: Long tail cancellation due to optimal threshold Dec 30 01:50:43.449621318 (586h21m43.320704363s) [noise=true , address=145.79.227.34:4906 , piece_id=T7LU4RO6KLZRF3YVRCLM6YRO45KWJMSYRRQC25BUM6LFI7O7SJJA, seg_pos={0 0}, upload=514021]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 30 01:50:43.449635595 (586h21m43.32071863s) [noise=true , address=145.79.227.34:4906 , piece_id=T7LU4RO6KLZRF3YVRCLM6YRO45KWJMSYRRQC25BUM6LFI7O7SJJA, seg_pos={0 0}, upload=514021]: Long tail cancellation due to optimal threshold Dec 30 01:50:43.449668177 (586h21m43.320751212s) [seg_pos={0 0}, upload=514021]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 01:50:43.449783637 (586h21m43.320866682s) [upload=514021]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:50:43.449791002 (586h21m43.320874037s) [upload=514021]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:50:43.747593334 (586h21m43.618676369s) [upload=514021]: Done uploading segments... Dec 30 01:50:43.747607632 (586h21m43.618690667s) [upload=514021]: Upload finished. err: <nil> Duration: 5.99956347s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 34.578599096s trace: https://jaeger.dev.storj.io/trace/58b94359c097ef0d FAIL (2025-12-30 01:54:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.221968961 (585h44m42.093052017s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222491837 (585h44m42.093574882s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222877261 (585h44m42.093960306s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223249098 (585h44m42.094332143s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223637930 (585h44m42.094720965s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224032200 (585h44m42.095115245s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224444876 (585h44m42.095527921s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224905332 (585h44m42.095988378s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225388433 (585h44m42.096471478s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225981221 (585h44m42.097064256s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227130389 (585h44m42.098213435s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227655670 (585h44m42.098738705s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228177764 (585h44m42.099260799s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228673828 (585h44m42.099756863s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:43.678892099 (585h44m43.549975134s) [upload=513482]: Next returned nil segment Dec 30 01:13:43.678910774 (585h44m43.549993819s) [upload=513482]: Waiting for error group managing segments... Dec 30 01:13:44.076462870 (585h44m43.947545905s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.083306310 (585h44m43.954389345s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.107556817 (585h44m43.978639852s) [noise=true , address=145.79.227.9:4391 , piece_id=4AVX4RQJ65H3VL7U3MW6MUYHGRK6YS7GOGZO4XVVG6YYTHFHAXDQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.147374754 (585h44m44.018457799s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.212624668 (585h44m44.083707703s) [noise=true , address=64.49.13.170:4721 , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.237525922 (585h44m44.108608957s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.242637575 (585h44m44.11372063s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.283510060 (585h44m44.154593095s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.340441716 (585h44m44.211524761s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.408662737 (585h44m44.279745772s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.962802591 (585h44m44.833885626s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.963549243 (585h44m44.834632288s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.017193407 (585h44m44.888276442s) [noise=true , address=145.79.227.44:4218 , piece_id=UBAPWL2SMR7FJ2AHQ5XHMX3H6NKDFZ23HEDFRMJLJFYZKGC4X4MA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093476639 (585h44m44.964559684s) [noise=true , address=145.79.227.49:4370 , piece_id=RZK3XQONM7S2XSOIO4EXITQ3XQGVYLIJNK66RI6LL5NMFG46NOPA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093501687 (585h44m44.964584722s) [seg_pos={0 0}, upload=513482]: Segment reached optimal threshold of 14 pieces. Dec 30 01:13:45.093670669 (585h44m44.964753704s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection Dec 30 01:13:45.093677973 (585h44m44.964761008s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093727678 (585h44m44.964810713s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093735993 (585h44m44.964819028s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093827067 (585h44m44.964910122s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UP8PxqeK7t24PXjopH1nziHYQFnsxhkamSHz8dmwcbuDxkAF6): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093833129 (585h44m44.964916174s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093916176 (585h44m44.964999211s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qZC1tmCvCadnm1fYZxR3kc4fonZUJPzvfTKkTFiDtccW5bo9T): protocol: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection Dec 30 01:13:45.093921096 (585h44m44.965004141s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093966583 (585h44m44.965049618s) [seg_pos={0 0}, upload=513482]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 01:13:45.094031926 (585h44m44.965114961s) [upload=513482]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:13:45.094036336 (585h44m44.965119371s) [upload=513482]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:13:45.389410141 (585h44m45.260493176s) [upload=513482]: Done uploading segments... Dec 30 01:13:45.389430891 (585h44m45.260513926s) [upload=513482]: Upload finished. err: <nil> Duration: 34.578628822s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 15.116016245s trace: https://jaeger.dev.storj.io/trace/58b94359c097ef0d FAIL (2025-12-30 01:53:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.221968961 (585h44m42.093052017s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222491837 (585h44m42.093574882s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.222877261 (585h44m42.093960306s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223249098 (585h44m42.094332143s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.223637930 (585h44m42.094720965s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224032200 (585h44m42.095115245s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224444876 (585h44m42.095527921s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.224905332 (585h44m42.095988378s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225388433 (585h44m42.096471478s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.225981221 (585h44m42.097064256s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227130389 (585h44m42.098213435s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.227655670 (585h44m42.098738705s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228177764 (585h44m42.099260799s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:42.228673828 (585h44m42.099756863s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Uploading piece... Dec 30 01:13:43.678892099 (585h44m43.549975134s) [upload=513482]: Next returned nil segment Dec 30 01:13:43.678910774 (585h44m43.549993819s) [upload=513482]: Waiting for error group managing segments... Dec 30 01:13:44.076462870 (585h44m43.947545905s) [noise=true , address=145.79.227.8:4333 , piece_id=DUQPOFSQUA6VKFNEHAY3HJD5W64BVL6EJV3VJZYEINA66CE3Q2FA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.083306310 (585h44m43.954389345s) [noise=true , address=145.79.227.25:4649 , piece_id=DVPUGLEANG5SLO6VDLYKZ7VCYI2PQWE32VBCB4C3GI45HXVFQN7Q, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.107556817 (585h44m43.978639852s) [noise=true , address=145.79.227.9:4391 , piece_id=4AVX4RQJ65H3VL7U3MW6MUYHGRK6YS7GOGZO4XVVG6YYTHFHAXDQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.147374754 (585h44m44.018457799s) [noise=true , address=145.79.227.32:4847 , piece_id=I34JFA2NYNIV6MYJZSCOX76D5T4BZWQKXAVBTQWP4PUOQQ4QCXKA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.212624668 (585h44m44.083707703s) [noise=true , address=64.49.13.170:4721 , piece_id=KWC2VAYDNXTBDL7X4SO2FZGTL4AH5RTU3MMHVGACFV22OMGSQD5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.237525922 (585h44m44.108608957s) [noise=true , address=77.90.31.14:4690 , piece_id=PKLTDESNCHALXV6XICKY62K3X3RAO4HUIUPU36WMADHJ2FKX7U5A, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.242637575 (585h44m44.11372063s) [noise=true , address=77.90.31.13:4646 , piece_id=NAIM2W7345TXM7I4VWVKGX6VGNFECT4GWX3M3S5USTUWN754FXIQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.283510060 (585h44m44.154593095s) [noise=true , address=77.90.31.17:4794 , piece_id=CERYYPLWAEGBJ6YOVGPXSECAHZHICBJQ4BOXFELRLWOCMYZMRAGA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.340441716 (585h44m44.211524761s) [noise=true , address=45.91.116.8:4077 , piece_id=IH2TC5XXEBT2KPCS36PN33IAV5T5K67WWMBTT4YLP7DD7COFWSSA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.408662737 (585h44m44.279745772s) [noise=true , address=77.90.31.10:4536 , piece_id=TDBK5RZ4CBDI2XQB3WRKLIX3MX3XS6JDYKDQ6UP44UA625ZV55AQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.962802591 (585h44m44.833885626s) [noise=true , address=145.79.227.43:4180 , piece_id=WKLCSZUW72R2VMOJB2SF7TI3JZDE3PO7JWR7XF627HIYFRXVCWPQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:44.963549243 (585h44m44.834632288s) [noise=true , address=145.79.227.30:4771 , piece_id=UX73JCZBSQDRAJV6EFPGXCIOWDQIRJDMQ4PER4YK5S6EDLYTDLHQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.017193407 (585h44m44.888276442s) [noise=true , address=145.79.227.44:4218 , piece_id=UBAPWL2SMR7FJ2AHQ5XHMX3H6NKDFZ23HEDFRMJLJFYZKGC4X4MA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093476639 (585h44m44.964559684s) [noise=true , address=145.79.227.49:4370 , piece_id=RZK3XQONM7S2XSOIO4EXITQ3XQGVYLIJNK66RI6LL5NMFG46NOPA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: <nil> Dec 30 01:13:45.093501687 (585h44m44.964584722s) [seg_pos={0 0}, upload=513482]: Segment reached optimal threshold of 14 pieces. Dec 30 01:13:45.093670669 (585h44m44.964753704s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52690->64.49.13.169:4708: use of closed network connection Dec 30 01:13:45.093677973 (585h44m44.964761008s) [noise=true , address=64.49.13.169:4708 , piece_id=SEAMKPANK24TSFNLSS2C4VGUDY3OBFG6FNX5P2Z62UFI5PKCRIMQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093727678 (585h44m44.964810713s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093735993 (585h44m44.964819028s) [noise=true , address=145.79.227.40:4069 , piece_id=5Q732VY4S52YQSSQI62OZXC3FNLNDPDFVUDGE4FKQFI575BC7EVA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093827067 (585h44m44.964910122s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UP8PxqeK7t24PXjopH1nziHYQFnsxhkamSHz8dmwcbuDxkAF6): protocol: expected piece hash; context canceled; EOF Dec 30 01:13:45.093833129 (585h44m44.964916174s) [noise=true , address=145.79.227.28:4750 , piece_id=VKUC2JEZ7DAVTNWM45LUC3Z6JWNE4LAX7UBZNJ26RAW7QIRTR5HA, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093916176 (585h44m44.964999211s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qZC1tmCvCadnm1fYZxR3kc4fonZUJPzvfTKkTFiDtccW5bo9T): protocol: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39712->145.79.227.7:4327: use of closed network connection Dec 30 01:13:45.093921096 (585h44m44.965004141s) [noise=true , address=145.79.227.7:4327 , piece_id=HB4ALLZ7K3MPWV5F2ZSDZSQW6HPRLHSERS5LGCCVBADBZPYKS6MQ, seg_pos={0 0}, upload=513482]: Long tail cancellation due to optimal threshold Dec 30 01:13:45.093966583 (585h44m44.965049618s) [seg_pos={0 0}, upload=513482]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 01:13:45.094031926 (585h44m44.965114961s) [upload=513482]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:13:45.094036336 (585h44m44.965119371s) [upload=513482]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:13:45.389410141 (585h44m45.260493176s) [upload=513482]: Done uploading segments... Dec 30 01:13:45.389430891 (585h44m45.260513926s) [upload=513482]: Upload finished. err: <nil> Duration: 15.116047674s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.................XX.....X.X......X.X..XX.X....X..XXX..XX....] duration: 5.836291113s trace: https://jaeger.dev.storj.io/trace/20f64a83037a2393 FAIL (2025-12-30 01:53:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: MPFSPCD6ASY2LCASLCT27Z657RVUMVJXENIMNHDRPZ5T7BBV37DA) context deadline exceeded; piecestore: (Node ID: 12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, Piece ID: HS3ABRPPVQFR57P7JITBF3246G3PRNFUH2ZMETP5CYA5NILCNO2A) context deadline exceeded; piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: JN32F2DIJK6I6VOMYTYV3T74BDBDLZDMWJYULBL7HUTGFKV65ZKQ) context deadline exceeded ======LOGS====== Dec 30 01:50:40.234985867 (586h21m40.106068912s) [upload=514021]: Starting upload Dec 30 01:50:40.235076680 (586h21m40.106159715s) [upload=514021]: Done starting upload Dec 30 01:50:40.235117789 (586h21m40.106200834s) [upload=514021]: Uploading segments... Dec 30 01:50:40.235287161 (586h21m40.106370196s) [upload=514021]: Got next segment. Inline: false Dec 30 01:50:40.235319232 (586h21m40.106402277s) [upload=514021]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 01:50:40.235324442 (586h21m40.106407487s) [upload=514021]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 01:50:40.483051707 (586h21m40.354134752s) [seg_pos={0 0}, upload=514021]: Begin upload segment... Dec 30 01:50:40.483093266 (586h21m40.354176301s) [seg_pos={0 0}, upload=514021]: Done begin upload segment. Dec 30 01:50:40.483115268 (586h21m40.354198313s) [noise=true , address=45.91.116.10:4082 , piece_id=WJT3ZSNQEMYXZN6QSXO2GQFAB4LFOFGB5AAESZWQOUJ73TBQDKBA, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.483227201 (586h21m40.354310246s) [noise=true , address=145.79.227.16:4911 , piece_id=JRL7QZGGYZAIFGZEBHEZAYQIJTFEXB6KKO7I6Y2DUS2SIKBCJAYA, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.483583540 (586h21m40.354666575s) [noise=true , address=45.91.116.9:4264 , piece_id=RXE5E3Z7AEDER6YEDU3H2W75EOGL2QY32EQDMLUI3DAP5VJVQUTA, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.483856730 (586h21m40.354939765s) [noise=true , address=145.79.227.51:4410 , piece_id=GEAU6ZOVWUMQ53FIM4YQ2BNVR7DUMAAR5QBORU5OWQBYLLDV5SVQ, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.484116876 (586h21m40.355199921s) [noise=true , address=145.79.227.36:4517 , piece_id=7PZIHPSP5YHU7XW7QHGBY5CD4WT4PC7F6TRUVYF335TTY2V64VEA, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.484379576 (586h21m40.355462611s) [noise=true , address=145.79.227.50:4373 , piece_id=L7NUZASKLC6WI4ATPDC3LEOQFKWFXBDM3XE6SKTOX3BNFMH5WLKQ, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.484684196 (586h21m40.355767231s) [noise=true , address=145.79.227.7:4293 , piece_id=RX7KO2NUK6YZ4KXVO65BWLDIHBXPIFFHDSQ456PJGZNYHM73BIQA, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.484997943 (586h21m40.356080978s) [noise=true , address=45.91.116.8:4047 , piece_id=MPFSPCD6ASY2LCASLCT27Z657RVUMVJXENIMNHDRPZ5T7BBV37DA, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.485305078 (586h21m40.356388113s) [noise=true , address=77.90.31.21:4877 , piece_id=HS3ABRPPVQFR57P7JITBF3246G3PRNFUH2ZMETP5CYA5NILCNO2A, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.485646458 (586h21m40.356729503s) [noise=true , address=145.79.227.39:4015 , piece_id=RMKHO2QMH2BB4XLLNPXCNKDW4RKI5ZQUCC6YA5PTC4NQC7HREOFA, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.486823519 (586h21m40.357906554s) [noise=true , address=145.79.227.29:4941 , piece_id=OL5DN2T5BKKCUDZ7GRKFYVO7MZYZGG6LX5G4XSUVWEOGPEWT4VGQ, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.487968209 (586h21m40.359051254s) [noise=true , address=77.90.31.19:4844 , piece_id=COAEIFAFOH5VP26MFZSXX5H2R2HESU6TCHGN2EKVBFJ3A52SIRVA, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.489030822 (586h21m40.360113857s) [noise=true , address=145.79.227.34:4906 , piece_id=T7LU4RO6KLZRF3YVRCLM6YRO45KWJMSYRRQC25BUM6LFI7O7SJJA, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.490339584 (586h21m40.361422619s) [noise=true , address=145.79.227.2:4113 , piece_id=DYZMSLAACLNG2TCDLEKFO7APCXBFVRHRY2FLQR5Q7ANSG3K3EJBA, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.491705184 (586h21m40.362788229s) [noise=true , address=45.91.116.13:4154 , piece_id=JN32F2DIJK6I6VOMYTYV3T74BDBDLZDMWJYULBL7HUTGFKV65ZKQ, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.498535069 (586h21m40.369618104s) [noise=true , address=45.91.116.7:4012 , piece_id=6V33KTBZE3IBFOBRYKHTNG7VNOSJCYV4FLCQ2T5PNEUZQF5KQEJQ, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:42.180702501 (586h21m42.051785546s) [upload=514021]: Next returned nil segment Dec 30 01:50:42.180721156 (586h21m42.051804191s) [upload=514021]: Waiting for error group managing segments... Dec 30 01:50:42.461140760 (586h21m42.332223795s) [noise=true , address=145.79.227.7:4293 , piece_id=RX7KO2NUK6YZ4KXVO65BWLDIHBXPIFFHDSQ456PJGZNYHM73BIQA, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:42.463300643 (586h21m42.334383688s) [noise=true , address=145.79.227.16:4911 , piece_id=JRL7QZGGYZAIFGZEBHEZAYQIJTFEXB6KKO7I6Y2DUS2SIKBCJAYA, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:42.467721662 (586h21m42.338804707s) [noise=true , address=145.79.227.50:4373 , piece_id=L7NUZASKLC6WI4ATPDC3LEOQFKWFXBDM3XE6SKTOX3BNFMH5WLKQ, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:42.477585643 (586h21m42.348668678s) [noise=true , address=145.79.227.51:4410 , piece_id=GEAU6ZOVWUMQ53FIM4YQ2BNVR7DUMAAR5QBORU5OWQBYLLDV5SVQ, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:42.490245268 (586h21m42.361328303s) [noise=true , address=145.79.227.29:4941 , piece_id=OL5DN2T5BKKCUDZ7GRKFYVO7MZYZGG6LX5G4XSUVWEOGPEWT4VGQ, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:42.494154722 (586h21m42.365237757s) [noise=true , address=145.79.227.39:4015 , piece_id=RMKHO2QMH2BB4XLLNPXCNKDW4RKI5ZQUCC6YA5PTC4NQC7HREOFA, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:42.501438722 (586h21m42.372521767s) [noise=true , address=145.79.227.36:4517 , piece_id=7PZIHPSP5YHU7XW7QHGBY5CD4WT4PC7F6TRUVYF335TTY2V64VEA, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:42.502707718 (586h21m42.373790753s) [noise=true , address=145.79.227.2:4113 , piece_id=DYZMSLAACLNG2TCDLEKFO7APCXBFVRHRY2FLQR5Q7ANSG3K3EJBA, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:42.905016712 (586h21m42.776099747s) [noise=true , address=45.91.116.8:4047 , piece_id=MPFSPCD6ASY2LCASLCT27Z657RVUMVJXENIMNHDRPZ5T7BBV37DA, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:43.050110884 (586h21m42.921193919s) [noise=true , address=45.91.116.9:4264 , piece_id=RXE5E3Z7AEDER6YEDU3H2W75EOGL2QY32EQDMLUI3DAP5VJVQUTA, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:43.440086393 (586h21m43.311169438s) [noise=true , address=45.91.116.13:4154 , piece_id=JN32F2DIJK6I6VOMYTYV3T74BDBDLZDMWJYULBL7HUTGFKV65ZKQ, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:43.442366294 (586h21m43.313449339s) [noise=true , address=77.90.31.21:4877 , piece_id=HS3ABRPPVQFR57P7JITBF3246G3PRNFUH2ZMETP5CYA5NILCNO2A, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:43.449133169 (586h21m43.320216224s) [noise=true , address=45.91.116.10:4082 , piece_id=WJT3ZSNQEMYXZN6QSXO2GQFAB4LFOFGB5AAESZWQOUJ73TBQDKBA, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:43.449154068 (586h21m43.320237113s) [seg_pos={0 0}, upload=514021]: Segment reached optimal threshold of 13 pieces. Dec 30 01:50:43.449402903 (586h21m43.320485958s) [noise=true , address=45.91.116.7:4012 , piece_id=6V33KTBZE3IBFOBRYKHTNG7VNOSJCYV4FLCQ2T5PNEUZQF5KQEJQ, seg_pos={0 0}, upload=514021]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: expected piece hash; context canceled; EOF Dec 30 01:50:43.449414875 (586h21m43.32049792s) [noise=true , address=45.91.116.7:4012 , piece_id=6V33KTBZE3IBFOBRYKHTNG7VNOSJCYV4FLCQ2T5PNEUZQF5KQEJQ, seg_pos={0 0}, upload=514021]: Long tail cancellation due to optimal threshold Dec 30 01:50:43.449570110 (586h21m43.320653155s) [noise=true , address=77.90.31.19:4844 , piece_id=COAEIFAFOH5VP26MFZSXX5H2R2HESU6TCHGN2EKVBFJ3A52SIRVA, seg_pos={0 0}, upload=514021]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF Dec 30 01:50:43.449579980 (586h21m43.320663015s) [noise=true , address=77.90.31.19:4844 , piece_id=COAEIFAFOH5VP26MFZSXX5H2R2HESU6TCHGN2EKVBFJ3A52SIRVA, seg_pos={0 0}, upload=514021]: Long tail cancellation due to optimal threshold Dec 30 01:50:43.449621318 (586h21m43.320704363s) [noise=true , address=145.79.227.34:4906 , piece_id=T7LU4RO6KLZRF3YVRCLM6YRO45KWJMSYRRQC25BUM6LFI7O7SJJA, seg_pos={0 0}, upload=514021]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 30 01:50:43.449635595 (586h21m43.32071863s) [noise=true , address=145.79.227.34:4906 , piece_id=T7LU4RO6KLZRF3YVRCLM6YRO45KWJMSYRRQC25BUM6LFI7O7SJJA, seg_pos={0 0}, upload=514021]: Long tail cancellation due to optimal threshold Dec 30 01:50:43.449668177 (586h21m43.320751212s) [seg_pos={0 0}, upload=514021]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 01:50:43.449783637 (586h21m43.320866682s) [upload=514021]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:50:43.449791002 (586h21m43.320874037s) [upload=514021]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:50:43.747593334 (586h21m43.618676369s) [upload=514021]: Done uploading segments... Dec 30 01:50:43.747607632 (586h21m43.618690667s) [upload=514021]: Upload finished. err: <nil> Duration: 5.836316241s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [X.................XX.....X.X......X.X..XX.X....X..XXX..XX...] duration: 5.908668123s trace: https://jaeger.dev.storj.io/trace/20f64a83037a2393 FAIL (2025-12-30 01:53:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: MPFSPCD6ASY2LCASLCT27Z657RVUMVJXENIMNHDRPZ5T7BBV37DA) context deadline exceeded; piecestore: (Node ID: 12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, Piece ID: HS3ABRPPVQFR57P7JITBF3246G3PRNFUH2ZMETP5CYA5NILCNO2A) context deadline exceeded; piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: JN32F2DIJK6I6VOMYTYV3T74BDBDLZDMWJYULBL7HUTGFKV65ZKQ) context deadline exceeded ======LOGS====== Dec 30 01:50:40.234985867 (586h21m40.106068912s) [upload=514021]: Starting upload Dec 30 01:50:40.235076680 (586h21m40.106159715s) [upload=514021]: Done starting upload Dec 30 01:50:40.235117789 (586h21m40.106200834s) [upload=514021]: Uploading segments... Dec 30 01:50:40.235287161 (586h21m40.106370196s) [upload=514021]: Got next segment. Inline: false Dec 30 01:50:40.235319232 (586h21m40.106402277s) [upload=514021]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 01:50:40.235324442 (586h21m40.106407487s) [upload=514021]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 01:50:40.483051707 (586h21m40.354134752s) [seg_pos={0 0}, upload=514021]: Begin upload segment... Dec 30 01:50:40.483093266 (586h21m40.354176301s) [seg_pos={0 0}, upload=514021]: Done begin upload segment. Dec 30 01:50:40.483115268 (586h21m40.354198313s) [noise=true , address=45.91.116.10:4082 , piece_id=WJT3ZSNQEMYXZN6QSXO2GQFAB4LFOFGB5AAESZWQOUJ73TBQDKBA, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.483227201 (586h21m40.354310246s) [noise=true , address=145.79.227.16:4911 , piece_id=JRL7QZGGYZAIFGZEBHEZAYQIJTFEXB6KKO7I6Y2DUS2SIKBCJAYA, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.483583540 (586h21m40.354666575s) [noise=true , address=45.91.116.9:4264 , piece_id=RXE5E3Z7AEDER6YEDU3H2W75EOGL2QY32EQDMLUI3DAP5VJVQUTA, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.483856730 (586h21m40.354939765s) [noise=true , address=145.79.227.51:4410 , piece_id=GEAU6ZOVWUMQ53FIM4YQ2BNVR7DUMAAR5QBORU5OWQBYLLDV5SVQ, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.484116876 (586h21m40.355199921s) [noise=true , address=145.79.227.36:4517 , piece_id=7PZIHPSP5YHU7XW7QHGBY5CD4WT4PC7F6TRUVYF335TTY2V64VEA, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.484379576 (586h21m40.355462611s) [noise=true , address=145.79.227.50:4373 , piece_id=L7NUZASKLC6WI4ATPDC3LEOQFKWFXBDM3XE6SKTOX3BNFMH5WLKQ, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.484684196 (586h21m40.355767231s) [noise=true , address=145.79.227.7:4293 , piece_id=RX7KO2NUK6YZ4KXVO65BWLDIHBXPIFFHDSQ456PJGZNYHM73BIQA, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.484997943 (586h21m40.356080978s) [noise=true , address=45.91.116.8:4047 , piece_id=MPFSPCD6ASY2LCASLCT27Z657RVUMVJXENIMNHDRPZ5T7BBV37DA, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.485305078 (586h21m40.356388113s) [noise=true , address=77.90.31.21:4877 , piece_id=HS3ABRPPVQFR57P7JITBF3246G3PRNFUH2ZMETP5CYA5NILCNO2A, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.485646458 (586h21m40.356729503s) [noise=true , address=145.79.227.39:4015 , piece_id=RMKHO2QMH2BB4XLLNPXCNKDW4RKI5ZQUCC6YA5PTC4NQC7HREOFA, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.486823519 (586h21m40.357906554s) [noise=true , address=145.79.227.29:4941 , piece_id=OL5DN2T5BKKCUDZ7GRKFYVO7MZYZGG6LX5G4XSUVWEOGPEWT4VGQ, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.487968209 (586h21m40.359051254s) [noise=true , address=77.90.31.19:4844 , piece_id=COAEIFAFOH5VP26MFZSXX5H2R2HESU6TCHGN2EKVBFJ3A52SIRVA, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.489030822 (586h21m40.360113857s) [noise=true , address=145.79.227.34:4906 , piece_id=T7LU4RO6KLZRF3YVRCLM6YRO45KWJMSYRRQC25BUM6LFI7O7SJJA, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.490339584 (586h21m40.361422619s) [noise=true , address=145.79.227.2:4113 , piece_id=DYZMSLAACLNG2TCDLEKFO7APCXBFVRHRY2FLQR5Q7ANSG3K3EJBA, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.491705184 (586h21m40.362788229s) [noise=true , address=45.91.116.13:4154 , piece_id=JN32F2DIJK6I6VOMYTYV3T74BDBDLZDMWJYULBL7HUTGFKV65ZKQ, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:40.498535069 (586h21m40.369618104s) [noise=true , address=45.91.116.7:4012 , piece_id=6V33KTBZE3IBFOBRYKHTNG7VNOSJCYV4FLCQ2T5PNEUZQF5KQEJQ, seg_pos={0 0}, upload=514021]: Uploading piece... Dec 30 01:50:42.180702501 (586h21m42.051785546s) [upload=514021]: Next returned nil segment Dec 30 01:50:42.180721156 (586h21m42.051804191s) [upload=514021]: Waiting for error group managing segments... Dec 30 01:50:42.461140760 (586h21m42.332223795s) [noise=true , address=145.79.227.7:4293 , piece_id=RX7KO2NUK6YZ4KXVO65BWLDIHBXPIFFHDSQ456PJGZNYHM73BIQA, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:42.463300643 (586h21m42.334383688s) [noise=true , address=145.79.227.16:4911 , piece_id=JRL7QZGGYZAIFGZEBHEZAYQIJTFEXB6KKO7I6Y2DUS2SIKBCJAYA, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:42.467721662 (586h21m42.338804707s) [noise=true , address=145.79.227.50:4373 , piece_id=L7NUZASKLC6WI4ATPDC3LEOQFKWFXBDM3XE6SKTOX3BNFMH5WLKQ, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:42.477585643 (586h21m42.348668678s) [noise=true , address=145.79.227.51:4410 , piece_id=GEAU6ZOVWUMQ53FIM4YQ2BNVR7DUMAAR5QBORU5OWQBYLLDV5SVQ, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:42.490245268 (586h21m42.361328303s) [noise=true , address=145.79.227.29:4941 , piece_id=OL5DN2T5BKKCUDZ7GRKFYVO7MZYZGG6LX5G4XSUVWEOGPEWT4VGQ, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:42.494154722 (586h21m42.365237757s) [noise=true , address=145.79.227.39:4015 , piece_id=RMKHO2QMH2BB4XLLNPXCNKDW4RKI5ZQUCC6YA5PTC4NQC7HREOFA, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:42.501438722 (586h21m42.372521767s) [noise=true , address=145.79.227.36:4517 , piece_id=7PZIHPSP5YHU7XW7QHGBY5CD4WT4PC7F6TRUVYF335TTY2V64VEA, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:42.502707718 (586h21m42.373790753s) [noise=true , address=145.79.227.2:4113 , piece_id=DYZMSLAACLNG2TCDLEKFO7APCXBFVRHRY2FLQR5Q7ANSG3K3EJBA, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:42.905016712 (586h21m42.776099747s) [noise=true , address=45.91.116.8:4047 , piece_id=MPFSPCD6ASY2LCASLCT27Z657RVUMVJXENIMNHDRPZ5T7BBV37DA, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:43.050110884 (586h21m42.921193919s) [noise=true , address=45.91.116.9:4264 , piece_id=RXE5E3Z7AEDER6YEDU3H2W75EOGL2QY32EQDMLUI3DAP5VJVQUTA, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:43.440086393 (586h21m43.311169438s) [noise=true , address=45.91.116.13:4154 , piece_id=JN32F2DIJK6I6VOMYTYV3T74BDBDLZDMWJYULBL7HUTGFKV65ZKQ, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:43.442366294 (586h21m43.313449339s) [noise=true , address=77.90.31.21:4877 , piece_id=HS3ABRPPVQFR57P7JITBF3246G3PRNFUH2ZMETP5CYA5NILCNO2A, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:43.449133169 (586h21m43.320216224s) [noise=true , address=45.91.116.10:4082 , piece_id=WJT3ZSNQEMYXZN6QSXO2GQFAB4LFOFGB5AAESZWQOUJ73TBQDKBA, seg_pos={0 0}, upload=514021]: Done uploading piece. err: <nil> Dec 30 01:50:43.449154068 (586h21m43.320237113s) [seg_pos={0 0}, upload=514021]: Segment reached optimal threshold of 13 pieces. Dec 30 01:50:43.449402903 (586h21m43.320485958s) [noise=true , address=45.91.116.7:4012 , piece_id=6V33KTBZE3IBFOBRYKHTNG7VNOSJCYV4FLCQ2T5PNEUZQF5KQEJQ, seg_pos={0 0}, upload=514021]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: expected piece hash; context canceled; EOF Dec 30 01:50:43.449414875 (586h21m43.32049792s) [noise=true , address=45.91.116.7:4012 , piece_id=6V33KTBZE3IBFOBRYKHTNG7VNOSJCYV4FLCQ2T5PNEUZQF5KQEJQ, seg_pos={0 0}, upload=514021]: Long tail cancellation due to optimal threshold Dec 30 01:50:43.449570110 (586h21m43.320653155s) [noise=true , address=77.90.31.19:4844 , piece_id=COAEIFAFOH5VP26MFZSXX5H2R2HESU6TCHGN2EKVBFJ3A52SIRVA, seg_pos={0 0}, upload=514021]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF Dec 30 01:50:43.449579980 (586h21m43.320663015s) [noise=true , address=77.90.31.19:4844 , piece_id=COAEIFAFOH5VP26MFZSXX5H2R2HESU6TCHGN2EKVBFJ3A52SIRVA, seg_pos={0 0}, upload=514021]: Long tail cancellation due to optimal threshold Dec 30 01:50:43.449621318 (586h21m43.320704363s) [noise=true , address=145.79.227.34:4906 , piece_id=T7LU4RO6KLZRF3YVRCLM6YRO45KWJMSYRRQC25BUM6LFI7O7SJJA, seg_pos={0 0}, upload=514021]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 30 01:50:43.449635595 (586h21m43.32071863s) [noise=true , address=145.79.227.34:4906 , piece_id=T7LU4RO6KLZRF3YVRCLM6YRO45KWJMSYRRQC25BUM6LFI7O7SJJA, seg_pos={0 0}, upload=514021]: Long tail cancellation due to optimal threshold Dec 30 01:50:43.449668177 (586h21m43.320751212s) [seg_pos={0 0}, upload=514021]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 01:50:43.449783637 (586h21m43.320866682s) [upload=514021]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 01:50:43.449791002 (586h21m43.320874037s) [upload=514021]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 01:50:43.747593334 (586h21m43.618676369s) [upload=514021]: Done uploading segments... Dec 30 01:50:43.747607632 (586h21m43.618690667s) [upload=514021]: Upload finished. err: <nil> Duration: 5.908698811s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)