Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [.X.X..........XX...X.......................X..XXX...........] duration: 3.467016947s trace: https://jaeger.dev.storj.io/trace/532d591e051353a0 FAIL (2025-12-30 04:01:55 UTC) orbiter warning: context deadline exceeded Duration: 3.467048408s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [X.X...XXX.X.....XXXX........XX.XX...XXXX...XXXX...X.X...XX..] duration: 5.913009467s trace: https://jaeger.dev.storj.io/trace/15b4cf4615988abc FAIL (2025-12-30 04:01:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: R3PNCLCPRKOF5EJNUIVK4DDO7QKXFTQHOQTPIFOIFKUIPAKSFNIQ) context deadline exceeded; piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: L34HSQ2E6PJPQMBMI55PTN4QMKBMREI3NV3NLBUZREONPB6N77NA) context deadline exceeded ======LOGS====== Dec 30 04:00:13.165093016 (588h31m13.036176061s) [upload=515915]: Starting upload Dec 30 04:00:13.165195281 (588h31m13.036278326s) [upload=515915]: Done starting upload Dec 30 04:00:13.165260615 (588h31m13.03634366s) [upload=515915]: Uploading segments... Dec 30 04:00:13.165452350 (588h31m13.036535395s) [upload=515915]: Got next segment. Inline: false Dec 30 04:00:13.165524838 (588h31m13.036607893s) [upload=515915]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 04:00:13.165534637 (588h31m13.036617682s) [upload=515915]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 04:00:13.411787833 (588h31m13.282870888s) [seg_pos={0 0}, upload=515915]: Begin upload segment... Dec 30 04:00:13.411851905 (588h31m13.28293495s) [seg_pos={0 0}, upload=515915]: Done begin upload segment. Dec 30 04:00:13.411918562 (588h31m13.283001607s) [noise=true , address=45.91.116.14:4192 , piece_id=JRXJT2WPI3VPCS26OVNPNFJKDVTFF332F4BIJAHWQMTSZCBRPX4Q, seg_pos={0 0}, upload=515915]: Uploading piece... Dec 30 04:00:13.411947236 (588h31m13.283030281s) [noise=true , address=77.90.31.13:4626 , piece_id=L34HSQ2E6PJPQMBMI55PTN4QMKBMREI3NV3NLBUZREONPB6N77NA, seg_pos={0 0}, upload=515915]: Uploading piece... Dec 30 04:00:13.413006804 (588h31m13.284089849s) [noise=true , address=145.79.227.48:4303 , piece_id=GKDDXUWXTGIOAQMGKOFRQP2G5EQNCWX2DYOV7BRO3S5YTEGSSPPA, seg_pos={0 0}, upload=515915]: Uploading piece... Dec 30 04:00:13.413095222 (588h31m13.284178268s) [noise=true , address=145.79.227.46:4603 , piece_id=ESGLFXALOTXX5KKUAMXPUAI6JFEAIQDPLEFPAQHE44NVGN66YDHQ, seg_pos={0 0}, upload=515915]: Uploading piece... Dec 30 04:00:13.413605173 (588h31m13.284688218s) [noise=true , address=145.79.227.38:4979 , piece_id=NGOZUUNCELYZQGGEFXRSDC3TFBLVZLKLPQWWJXLLYVWXE6RORC7A, seg_pos={0 0}, upload=515915]: Uploading piece... Dec 30 04:00:13.416721317 (588h31m13.287804362s) [noise=true , address=77.90.31.12:4590 , piece_id=JUXYOK5VSNHHQGTDW3JD7YN75MSCLC3XDV4VTNEQMPJHELV2RROA, seg_pos={0 0}, upload=515915]: Uploading piece... Dec 30 04:00:13.417255474 (588h31m13.288338519s) [noise=true , address=145.79.227.36:4519 , piece_id=GDJ7VR575P2DGNOV7VAGMVCYIOMWLSH4XKBF75MIO5HMOEMVHG5A, seg_pos={0 0}, upload=515915]: Uploading piece... Dec 30 04:00:13.419062976 (588h31m13.290146021s) [noise=true , address=145.79.227.18:4006 , piece_id=RULFLOXGDF2FA7U75TIH6TB4R6JNFGLHJIJP4X3RGOU3BHEJAWBA, seg_pos={0 0}, upload=515915]: Uploading piece... Dec 30 04:00:13.419340053 (588h31m13.290423098s) [noise=true , address=45.91.116.6:4976 , piece_id=ZJINSWDTALZDL723UY6K7EV3LKIOFGBCXHNUQENXRODNDBKFMK3A, seg_pos={0 0}, upload=515915]: Uploading piece... Dec 30 04:00:13.420611755 (588h31m13.2916948s) [noise=true , address=45.91.116.12:4227 , piece_id=ZTTT5ONGFIKEEAFMGUKDCZP4YTC7STD7UDUE26UK6OT5TZM5TNFQ, seg_pos={0 0}, upload=515915]: Uploading piece... Dec 30 04:00:13.422017532 (588h31m13.293100577s) [noise=true , address=145.79.227.22:4509 , piece_id=6UV6HKTDAO4PB6OGYJX3KE3ECYH532ZRH3I6JFSWBJGBNW5K6BZA, seg_pos={0 0}, upload=515915]: Uploading piece... Dec 30 04:00:13.422373410 (588h31m13.293456455s) [noise=true , address=145.79.227.6:4258 , piece_id=KEDB55NEGVAIY5US2SS5PVTU23YTLXSDUXJCMMOIJKHEZPXPY64Q, seg_pos={0 0}, upload=515915]: Uploading piece... Dec 30 04:00:13.423506607 (588h31m13.294589653s) [noise=true , address=77.90.31.10:4519 , piece_id=2N2J2PXCARQI54VWOCYMXAMN6YXIRIWJYZP6OR4QDVYLXXF4DLXA, seg_pos={0 0}, upload=515915]: Uploading piece... Dec 30 04:00:13.424538543 (588h31m13.295621588s) [noise=true , address=45.91.116.5:4297 , piece_id=R3PNCLCPRKOF5EJNUIVK4DDO7QKXFTQHOQTPIFOIFKUIPAKSFNIQ, seg_pos={0 0}, upload=515915]: Uploading piece... Dec 30 04:00:13.427878192 (588h31m13.298961227s) [noise=true , address=145.79.227.9:4367 , piece_id=AZ2BN27PZYOF5GSFES2E6O2MOJ2OYBJRXSDGBSWKWSQXMUSSQ4KA, seg_pos={0 0}, upload=515915]: Uploading piece... Dec 30 04:00:13.429392917 (588h31m13.300475962s) [noise=true , address=145.79.227.42:4123 , piece_id=Q2EFDM6TAWZD5BESSWWLGA53KY4QL3HGY4D3AIVSSSJ3EMKLVR7Q, seg_pos={0 0}, upload=515915]: Uploading piece... Dec 30 04:00:15.022757128 (588h31m14.893840173s) [upload=515915]: Next returned nil segment Dec 30 04:00:15.022779361 (588h31m14.893862436s) [upload=515915]: Waiting for error group managing segments... Dec 30 04:00:15.412297827 (588h31m15.283380873s) [noise=true , address=145.79.227.46:4603 , piece_id=ESGLFXALOTXX5KKUAMXPUAI6JFEAIQDPLEFPAQHE44NVGN66YDHQ, seg_pos={0 0}, upload=515915]: Done uploading piece. err: <nil> Dec 30 04:00:15.415334210 (588h31m15.286417255s) [noise=true , address=145.79.227.36:4519 , piece_id=GDJ7VR575P2DGNOV7VAGMVCYIOMWLSH4XKBF75MIO5HMOEMVHG5A, seg_pos={0 0}, upload=515915]: Done uploading piece. err: <nil> Dec 30 04:00:15.416545888 (588h31m15.287628933s) [noise=true , address=145.79.227.48:4303 , piece_id=GKDDXUWXTGIOAQMGKOFRQP2G5EQNCWX2DYOV7BRO3S5YTEGSSPPA, seg_pos={0 0}, upload=515915]: Done uploading piece. err: <nil> Dec 30 04:00:15.428010749 (588h31m15.299093794s) [noise=true , address=145.79.227.9:4367 , piece_id=AZ2BN27PZYOF5GSFES2E6O2MOJ2OYBJRXSDGBSWKWSQXMUSSQ4KA, seg_pos={0 0}, upload=515915]: Done uploading piece. err: <nil> Dec 30 04:00:15.428579111 (588h31m15.299662156s) [noise=true , address=145.79.227.18:4006 , piece_id=RULFLOXGDF2FA7U75TIH6TB4R6JNFGLHJIJP4X3RGOU3BHEJAWBA, seg_pos={0 0}, upload=515915]: Done uploading piece. err: <nil> Dec 30 04:00:15.430778108 (588h31m15.301861163s) [noise=true , address=145.79.227.42:4123 , piece_id=Q2EFDM6TAWZD5BESSWWLGA53KY4QL3HGY4D3AIVSSSJ3EMKLVR7Q, seg_pos={0 0}, upload=515915]: Done uploading piece. err: <nil> Dec 30 04:00:15.433722836 (588h31m15.304805881s) [noise=true , address=145.79.227.22:4509 , piece_id=6UV6HKTDAO4PB6OGYJX3KE3ECYH532ZRH3I6JFSWBJGBNW5K6BZA, seg_pos={0 0}, upload=515915]: Done uploading piece. err: <nil> Dec 30 04:00:15.448367303 (588h31m15.319450348s) [noise=true , address=145.79.227.38:4979 , piece_id=NGOZUUNCELYZQGGEFXRSDC3TFBLVZLKLPQWWJXLLYVWXE6RORC7A, seg_pos={0 0}, upload=515915]: Done uploading piece. err: <nil> Dec 30 04:00:15.450738748 (588h31m15.321821783s) [noise=true , address=145.79.227.6:4258 , piece_id=KEDB55NEGVAIY5US2SS5PVTU23YTLXSDUXJCMMOIJKHEZPXPY64Q, seg_pos={0 0}, upload=515915]: Done uploading piece. err: <nil> Dec 30 04:00:15.658184446 (588h31m15.529267491s) [noise=true , address=45.91.116.5:4297 , piece_id=R3PNCLCPRKOF5EJNUIVK4DDO7QKXFTQHOQTPIFOIFKUIPAKSFNIQ, seg_pos={0 0}, upload=515915]: Done uploading piece. err: <nil> Dec 30 04:00:15.682045152 (588h31m15.553128197s) [noise=true , address=77.90.31.13:4626 , piece_id=L34HSQ2E6PJPQMBMI55PTN4QMKBMREI3NV3NLBUZREONPB6N77NA, seg_pos={0 0}, upload=515915]: Done uploading piece. err: <nil> Dec 30 04:00:15.747571129 (588h31m15.618654174s) [noise=true , address=77.90.31.10:4519 , piece_id=2N2J2PXCARQI54VWOCYMXAMN6YXIRIWJYZP6OR4QDVYLXXF4DLXA, seg_pos={0 0}, upload=515915]: Done uploading piece. err: <nil> Dec 30 04:00:16.112287537 (588h31m15.983370612s) [noise=true , address=45.91.116.6:4976 , piece_id=ZJINSWDTALZDL723UY6K7EV3LKIOFGBCXHNUQENXRODNDBKFMK3A, seg_pos={0 0}, upload=515915]: Done uploading piece. err: <nil> Dec 30 04:00:16.112315631 (588h31m15.983398666s) [seg_pos={0 0}, upload=515915]: Segment reached optimal threshold of 13 pieces. Dec 30 04:00:16.112590925 (588h31m15.98367398s) [noise=true , address=77.90.31.12:4590 , piece_id=JUXYOK5VSNHHQGTDW3JD7YN75MSCLC3XDV4VTNEQMPJHELV2RROA, seg_pos={0 0}, upload=515915]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 30 04:00:16.112602677 (588h31m15.983685722s) [noise=true , address=77.90.31.12:4590 , piece_id=JUXYOK5VSNHHQGTDW3JD7YN75MSCLC3XDV4VTNEQMPJHELV2RROA, seg_pos={0 0}, upload=515915]: Long tail cancellation due to optimal threshold Dec 30 04:00:16.112960609 (588h31m15.984043654s) [noise=true , address=45.91.116.12:4227 , piece_id=ZTTT5ONGFIKEEAFMGUKDCZP4YTC7STD7UDUE26UK6OT5TZM5TNFQ, seg_pos={0 0}, upload=515915]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Dec 30 04:00:16.112970908 (588h31m15.984053953s) [noise=true , address=45.91.116.12:4227 , piece_id=ZTTT5ONGFIKEEAFMGUKDCZP4YTC7STD7UDUE26UK6OT5TZM5TNFQ, seg_pos={0 0}, upload=515915]: Long tail cancellation due to optimal threshold Dec 30 04:00:16.113192330 (588h31m15.984275395s) [noise=true , address=45.91.116.14:4192 , piece_id=JRXJT2WPI3VPCS26OVNPNFJKDVTFF332F4BIJAHWQMTSZCBRPX4Q, seg_pos={0 0}, upload=515915]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF Dec 30 04:00:16.113203221 (588h31m15.984286266s) [noise=true , address=45.91.116.14:4192 , piece_id=JRXJT2WPI3VPCS26OVNPNFJKDVTFF332F4BIJAHWQMTSZCBRPX4Q, seg_pos={0 0}, upload=515915]: Long tail cancellation due to optimal threshold Dec 30 04:00:16.113227647 (588h31m15.984310692s) [seg_pos={0 0}, upload=515915]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 04:00:16.113335613 (588h31m15.984418688s) [upload=515915]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:00:16.113344680 (588h31m15.984427715s) [upload=515915]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:00:16.409991296 (588h31m16.281074332s) [upload=515915]: Done uploading segments... Dec 30 04:00:16.410012327 (588h31m16.281095362s) [upload=515915]: Upload finished. err: <nil> Duration: 5.913039363s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................................X............X] duration: 35.360667376s trace: https://jaeger.dev.storj.io/trace/637c536fc008fb78 FAIL (2025-12-30 04:01:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 2m8.645593264s) [noise=true , address=145.79.227.34:4933 , piece_id=FLSB6H6OYEN4TZQFPP6DJBVKR2D6EUPQRTNAN52E6H3OTCZWXBBA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.775911497 (588h32m8.646994542s) [noise=true , address=145.79.227.44:4209 , piece_id=CBBJPUQKEQ6XT2CLE24WY4XISA2KKOGBTDZXHRTUO4TVD2ALQKXA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.777318166 (588h32m8.648401212s) [noise=true , address=145.79.227.6:4285 , piece_id=P7OO3JQHCCYYN26PYMUGX2DROPI63B5BELMTI22GGARK2E7L4C5Q, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.778059658 (588h32m8.649142703s) [noise=true , address=145.79.227.40:4076 , piece_id=F63EK3TCIQFHVCCDV5DL5I2QTZJZYR7DRPOMFS6J3UEC5W6YX6BQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.778338178 (588h32m8.649421224s) [noise=true , address=77.90.31.13:4652 , piece_id=TNSSJ3SDTQUXNQ2DVRGRSVTEEYC3JWHVOS4JOTLJBKGM6522JSKQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.778723793 (588h32m8.649806838s) [noise=true , address=77.90.31.21:4908 , piece_id=WWHAPV3K3YCORFDS4GHD4U6C34PN25YUMDGMRIFBO4QVERGMH7GQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.779305430 (588h32m8.650388476s) [noise=true , address=145.79.227.30:4795 , piece_id=ECOXYC77MJSY3VWOOPNY3YSX7H723EW2CKQIIYIPQCKCSVD3FANQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.780090406 (588h32m8.651173441s) [noise=true , address=145.79.227.2:4125 , piece_id=BLLI3M637UHJRZP3YL52RE62RUEVGLHZXVAXPBZVJZMQPW5ZTIUA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.780617810 (588h32m8.651700855s) [noise=true , address=145.79.227.19:4083 , piece_id=MCQJI74FZW4KRT3ZLL2W226HF5LAZFUU5ING327XISXQYV3CQ5WQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.781367337 (588h32m8.652450373s) [noise=true , address=64.49.13.154:4926 , piece_id=T5REBW4C4OAX3NHQIQ4JNMAJHLHSL77P43OAGOAAVEQDELYSXOJA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.782194792 (588h32m8.653277837s) [noise=true , address=45.91.116.12:4253 , piece_id=5TOZUTOACBYBHCH62TBWINFAN6GF466FU2JYJQJMGIPM7Y5HML7Q, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.782953948 (588h32m8.654036993s) [noise=true , address=145.79.227.35:4501 , piece_id=CAPYASPEUC4KMTXKL7V5VWAYCRY5752BPH5RKWUZVVU3HCVIAVLQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.786503808 (588h32m8.657586853s) [noise=true , address=145.79.227.43:4160 , piece_id=4LCCWMEXVCUPMLLKG26S5EMIJP3MDEDNZNVLZRGJJOJJE7Q7AADA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.788837112 (588h32m8.659920157s) [noise=true , address=193.149.164.143:4701 , piece_id=3QVRO2CR2P3YINLVIEUJTC3OBOYLKHIDVVWX5DSSGIWVBMK6WAKQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.793855728 (588h32m8.664938773s) [noise=true , address=77.90.31.19:4856 , piece_id=YJLYJIBVIXIYKZ33TVLDL4MPCDIEN3OX7FETKLSGSZAD3EUY4BPQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.795240425 (588h32m8.66632346s) [noise=true , address=145.79.227.49:4358 , piece_id=TVR7OXHFNJN6J7J4HXPMHEC53NUX4NI74L6VGYITLHAQTSK54QIQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:10.180178281 (588h32m10.051261327s) [upload=515927]: Next returned nil segment Dec 30 04:01:10.180193331 (588h32m10.051276366s) [upload=515927]: Waiting for error group managing segments... Dec 30 04:01:10.630495634 (588h32m10.501578669s) [noise=true , address=145.79.227.2:4125 , piece_id=BLLI3M637UHJRZP3YL52RE62RUEVGLHZXVAXPBZVJZMQPW5ZTIUA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.659778345 (588h32m10.53086139s) [noise=true , address=145.79.227.43:4160 , piece_id=4LCCWMEXVCUPMLLKG26S5EMIJP3MDEDNZNVLZRGJJOJJE7Q7AADA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.671799526 (588h32m10.542882562s) [noise=true , address=145.79.227.19:4083 , piece_id=MCQJI74FZW4KRT3ZLL2W226HF5LAZFUU5ING327XISXQYV3CQ5WQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.676331005 (588h32m10.54741405s) [noise=true , address=145.79.227.40:4076 , piece_id=F63EK3TCIQFHVCCDV5DL5I2QTZJZYR7DRPOMFS6J3UEC5W6YX6BQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.820832219 (588h32m10.691915264s) [noise=true , address=77.90.31.15:4731 , piece_id=K3K6CN5V7SETRPMRCKHAVCNNK4DPSYJCHSAJ6HEFWVQS5TCWJZAQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.843204321 (588h32m10.714287356s) [noise=true , address=145.79.227.35:4501 , piece_id=CAPYASPEUC4KMTXKL7V5VWAYCRY5752BPH5RKWUZVVU3HCVIAVLQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.434118945 (588h32m11.30520199s) [noise=true , address=193.149.164.143:4701 , piece_id=3QVRO2CR2P3YINLVIEUJTC3OBOYLKHIDVVWX5DSSGIWVBMK6WAKQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.518819916 (588h32m11.389902961s) [noise=true , address=77.90.31.19:4856 , piece_id=YJLYJIBVIXIYKZ33TVLDL4MPCDIEN3OX7FETKLSGSZAD3EUY4BPQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.573041516 (588h32m11.444124551s) [noise=true , address=145.79.227.44:4209 , piece_id=CBBJPUQKEQ6XT2CLE24WY4XISA2KKOGBTDZXHRTUO4TVD2ALQKXA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.584552196 (588h32m11.455635241s) [noise=true , address=45.91.116.12:4253 , piece_id=5TOZUTOACBYBHCH62TBWINFAN6GF466FU2JYJQJMGIPM7Y5HML7Q, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.588999815 (588h32m11.46008285s) [noise=true , address=64.49.13.154:4926 , piece_id=T5REBW4C4OAX3NHQIQ4JNMAJHLHSL77P43OAGOAAVEQDELYSXOJA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.595442002 (588h32m11.466525037s) [noise=true , address=145.79.227.34:4933 , piece_id=FLSB6H6OYEN4TZQFPP6DJBVKR2D6EUPQRTNAN52E6H3OTCZWXBBA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.601502722 (588h32m11.472585767s) [noise=true , address=145.79.227.49:4358 , piece_id=TVR7OXHFNJN6J7J4HXPMHEC53NUX4NI74L6VGYITLHAQTSK54QIQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.603589166 (588h32m11.474672201s) [noise=true , address=145.79.227.30:4795 , piece_id=ECOXYC77MJSY3VWOOPNY3YSX7H723EW2CKQIIYIPQCKCSVD3FANQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.603605928 (588h32m11.474688963s) [seg_pos={0 0}, upload=515927]: Segment reached optimal threshold of 14 pieces. Dec 30 04:01:11.603761845 (588h32m11.47484489s) [noise=true , address=145.79.227.6:4285 , piece_id=P7OO3JQHCCYYN26PYMUGX2DROPI63B5BELMTI22GGARK2E7L4C5Q, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18UBLXtDVgwsDAPguXZ7eWhE9XKwfa8US7HXbWaXuhpDzCLa8Y): protocol: expected piece hash; context canceled; EOF Dec 30 04:01:11.603767786 (588h32m11.474850821s) [noise=true , address=145.79.227.6:4285 , piece_id=P7OO3JQHCCYYN26PYMUGX2DROPI63B5BELMTI22GGARK2E7L4C5Q, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604070503 (588h32m11.475153538s) [noise=true , address=77.90.31.13:4652 , piece_id=TNSSJ3SDTQUXNQ2DVRGRSVTEEYC3JWHVOS4JOTLJBKGM6522JSKQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KrMw5DoiMSiBHMZJ8Zf8ir7UnTzBwUt9XMhw3Ria6ASGkm3FA): protocol: expected piece hash; context canceled; EOF Dec 30 04:01:11.604079330 (588h32m11.475162375s) [noise=true , address=77.90.31.13:4652 , piece_id=TNSSJ3SDTQUXNQ2DVRGRSVTEEYC3JWHVOS4JOTLJBKGM6522JSKQ, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604224596 (588h32m11.475307631s) [noise=true , address=145.79.227.5:4253 , piece_id=TA4BHMWSSF55XGHUNRG6K2L5XWKITIZMUP62ZRZGWD46UWJP54IQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vJmwywMNTHh1ce5JGzbMiUBA3FijDecNEJ1TDU1Wyd4NWzA14): protocol: write tcp 188.166.225.170:43318->145.79.227.5:4253: use of closed network connection; write tcp 188.166.225.170:43318->145.79.227.5:4253: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43318->145.79.227.5:4253: use of closed network connection Dec 30 04:01:11.604233653 (588h32m11.475316698s) [noise=true , address=145.79.227.5:4253 , piece_id=TA4BHMWSSF55XGHUNRG6K2L5XWKITIZMUP62ZRZGWD46UWJP54IQ, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604355586 (588h32m11.475438621s) [noise=true , address=77.90.31.21:4908 , piece_id=WWHAPV3K3YCORFDS4GHD4U6C34PN25YUMDGMRIFBO4QVERGMH7GQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13hCBBV3cxyTWCAuG8QGGAoJRb6S8Ay23HpXYyBgRX4HZSSpga): protocol: expected piece hash; context canceled; EOF Dec 30 04:01:11.604361457 (588h32m11.475444502s) [noise=true , address=77.90.31.21:4908 , piece_id=WWHAPV3K3YCORFDS4GHD4U6C34PN25YUMDGMRIFBO4QVERGMH7GQ, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604390893 (588h32m11.475473928s) [seg_pos={0 0}, upload=515927]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 04:01:11.604455616 (588h32m11.475538661s) [upload=515927]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:01:11.604461698 (588h32m11.475544733s) [upload=515927]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:01:11.911203579 (588h32m11.782286634s) [upload=515927]: Done uploading segments... Dec 30 04:01:11.911228716 (588h32m11.782311761s) [upload=515927]: Upload finished. err: <nil> Duration: 35.360717551s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [..X.X..........XX...X.......................X..XXX..........] duration: 3.402435557s trace: https://jaeger.dev.storj.io/trace/532d591e051353a0 FAIL (2025-12-30 04:01:36 UTC) orbiter warning: context deadline exceeded Duration: 3.402461428s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [.X.X...XXX.X.....XXXX........XX.XX...XXXX...XXXX...X.X...XX.] duration: 6.105984724s trace: https://jaeger.dev.storj.io/trace/15b4cf4615988abc FAIL (2025-12-30 04:01:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: R3PNCLCPRKOF5EJNUIVK4DDO7QKXFTQHOQTPIFOIFKUIPAKSFNIQ) context deadline exceeded; piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: L34HSQ2E6PJPQMBMI55PTN4QMKBMREI3NV3NLBUZREONPB6N77NA) context deadline exceeded ======LOGS====== Dec 30 04:00:13.165093016 (588h31m13.036176061s) [upload=515915]: Starting upload Dec 30 04:00:13.165195281 (588h31m13.036278326s) [upload=515915]: Done starting upload Dec 30 04:00:13.165260615 (588h31m13.03634366s) [upload=515915]: Uploading segments... Dec 30 04:00:13.165452350 (588h31m13.036535395s) [upload=515915]: Got next segment. Inline: false Dec 30 04:00:13.165524838 (588h31m13.036607893s) [upload=515915]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 04:00:13.165534637 (588h31m13.036617682s) [upload=515915]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 04:00:13.411787833 (588h31m13.282870888s) [seg_pos={0 0}, upload=515915]: Begin upload segment... Dec 30 04:00:13.411851905 (588h31m13.28293495s) [seg_pos={0 0}, upload=515915]: Done begin upload segment. Dec 30 04:00:13.411918562 (588h31m13.283001607s) [noise=true , address=45.91.116.14:4192 , piece_id=JRXJT2WPI3VPCS26OVNPNFJKDVTFF332F4BIJAHWQMTSZCBRPX4Q, seg_pos={0 0}, upload=515915]: Uploading piece... Dec 30 04:00:13.411947236 (588h31m13.283030281s) [noise=true , address=77.90.31.13:4626 , piece_id=L34HSQ2E6PJPQMBMI55PTN4QMKBMREI3NV3NLBUZREONPB6N77NA, seg_pos={0 0}, upload=515915]: Uploading piece... Dec 30 04:00:13.413006804 (588h31m13.284089849s) [noise=true , address=145.79.227.48:4303 , piece_id=GKDDXUWXTGIOAQMGKOFRQP2G5EQNCWX2DYOV7BRO3S5YTEGSSPPA, seg_pos={0 0}, upload=515915]: Uploading piece... Dec 30 04:00:13.413095222 (588h31m13.284178268s) [noise=true , address=145.79.227.46:4603 , piece_id=ESGLFXALOTXX5KKUAMXPUAI6JFEAIQDPLEFPAQHE44NVGN66YDHQ, seg_pos={0 0}, upload=515915]: Uploading piece... Dec 30 04:00:13.413605173 (588h31m13.284688218s) [noise=true , address=145.79.227.38:4979 , piece_id=NGOZUUNCELYZQGGEFXRSDC3TFBLVZLKLPQWWJXLLYVWXE6RORC7A, seg_pos={0 0}, upload=515915]: Uploading piece... Dec 30 04:00:13.416721317 (588h31m13.287804362s) [noise=true , address=77.90.31.12:4590 , piece_id=JUXYOK5VSNHHQGTDW3JD7YN75MSCLC3XDV4VTNEQMPJHELV2RROA, seg_pos={0 0}, upload=515915]: Uploading piece... Dec 30 04:00:13.417255474 (588h31m13.288338519s) [noise=true , address=145.79.227.36:4519 , piece_id=GDJ7VR575P2DGNOV7VAGMVCYIOMWLSH4XKBF75MIO5HMOEMVHG5A, seg_pos={0 0}, upload=515915]: Uploading piece... Dec 30 04:00:13.419062976 (588h31m13.290146021s) [noise=true , address=145.79.227.18:4006 , piece_id=RULFLOXGDF2FA7U75TIH6TB4R6JNFGLHJIJP4X3RGOU3BHEJAWBA, seg_pos={0 0}, upload=515915]: Uploading piece... Dec 30 04:00:13.419340053 (588h31m13.290423098s) [noise=true , address=45.91.116.6:4976 , piece_id=ZJINSWDTALZDL723UY6K7EV3LKIOFGBCXHNUQENXRODNDBKFMK3A, seg_pos={0 0}, upload=515915]: Uploading piece... Dec 30 04:00:13.420611755 (588h31m13.2916948s) [noise=true , address=45.91.116.12:4227 , piece_id=ZTTT5ONGFIKEEAFMGUKDCZP4YTC7STD7UDUE26UK6OT5TZM5TNFQ, seg_pos={0 0}, upload=515915]: Uploading piece... Dec 30 04:00:13.422017532 (588h31m13.293100577s) [noise=true , address=145.79.227.22:4509 , piece_id=6UV6HKTDAO4PB6OGYJX3KE3ECYH532ZRH3I6JFSWBJGBNW5K6BZA, seg_pos={0 0}, upload=515915]: Uploading piece... Dec 30 04:00:13.422373410 (588h31m13.293456455s) [noise=true , address=145.79.227.6:4258 , piece_id=KEDB55NEGVAIY5US2SS5PVTU23YTLXSDUXJCMMOIJKHEZPXPY64Q, seg_pos={0 0}, upload=515915]: Uploading piece... Dec 30 04:00:13.423506607 (588h31m13.294589653s) [noise=true , address=77.90.31.10:4519 , piece_id=2N2J2PXCARQI54VWOCYMXAMN6YXIRIWJYZP6OR4QDVYLXXF4DLXA, seg_pos={0 0}, upload=515915]: Uploading piece... Dec 30 04:00:13.424538543 (588h31m13.295621588s) [noise=true , address=45.91.116.5:4297 , piece_id=R3PNCLCPRKOF5EJNUIVK4DDO7QKXFTQHOQTPIFOIFKUIPAKSFNIQ, seg_pos={0 0}, upload=515915]: Uploading piece... Dec 30 04:00:13.427878192 (588h31m13.298961227s) [noise=true , address=145.79.227.9:4367 , piece_id=AZ2BN27PZYOF5GSFES2E6O2MOJ2OYBJRXSDGBSWKWSQXMUSSQ4KA, seg_pos={0 0}, upload=515915]: Uploading piece... Dec 30 04:00:13.429392917 (588h31m13.300475962s) [noise=true , address=145.79.227.42:4123 , piece_id=Q2EFDM6TAWZD5BESSWWLGA53KY4QL3HGY4D3AIVSSSJ3EMKLVR7Q, seg_pos={0 0}, upload=515915]: Uploading piece... Dec 30 04:00:15.022757128 (588h31m14.893840173s) [upload=515915]: Next returned nil segment Dec 30 04:00:15.022779361 (588h31m14.893862436s) [upload=515915]: Waiting for error group managing segments... Dec 30 04:00:15.412297827 (588h31m15.283380873s) [noise=true , address=145.79.227.46:4603 , piece_id=ESGLFXALOTXX5KKUAMXPUAI6JFEAIQDPLEFPAQHE44NVGN66YDHQ, seg_pos={0 0}, upload=515915]: Done uploading piece. err: <nil> Dec 30 04:00:15.415334210 (588h31m15.286417255s) [noise=true , address=145.79.227.36:4519 , piece_id=GDJ7VR575P2DGNOV7VAGMVCYIOMWLSH4XKBF75MIO5HMOEMVHG5A, seg_pos={0 0}, upload=515915]: Done uploading piece. err: <nil> Dec 30 04:00:15.416545888 (588h31m15.287628933s) [noise=true , address=145.79.227.48:4303 , piece_id=GKDDXUWXTGIOAQMGKOFRQP2G5EQNCWX2DYOV7BRO3S5YTEGSSPPA, seg_pos={0 0}, upload=515915]: Done uploading piece. err: <nil> Dec 30 04:00:15.428010749 (588h31m15.299093794s) [noise=true , address=145.79.227.9:4367 , piece_id=AZ2BN27PZYOF5GSFES2E6O2MOJ2OYBJRXSDGBSWKWSQXMUSSQ4KA, seg_pos={0 0}, upload=515915]: Done uploading piece. err: <nil> Dec 30 04:00:15.428579111 (588h31m15.299662156s) [noise=true , address=145.79.227.18:4006 , piece_id=RULFLOXGDF2FA7U75TIH6TB4R6JNFGLHJIJP4X3RGOU3BHEJAWBA, seg_pos={0 0}, upload=515915]: Done uploading piece. err: <nil> Dec 30 04:00:15.430778108 (588h31m15.301861163s) [noise=true , address=145.79.227.42:4123 , piece_id=Q2EFDM6TAWZD5BESSWWLGA53KY4QL3HGY4D3AIVSSSJ3EMKLVR7Q, seg_pos={0 0}, upload=515915]: Done uploading piece. err: <nil> Dec 30 04:00:15.433722836 (588h31m15.304805881s) [noise=true , address=145.79.227.22:4509 , piece_id=6UV6HKTDAO4PB6OGYJX3KE3ECYH532ZRH3I6JFSWBJGBNW5K6BZA, seg_pos={0 0}, upload=515915]: Done uploading piece. err: <nil> Dec 30 04:00:15.448367303 (588h31m15.319450348s) [noise=true , address=145.79.227.38:4979 , piece_id=NGOZUUNCELYZQGGEFXRSDC3TFBLVZLKLPQWWJXLLYVWXE6RORC7A, seg_pos={0 0}, upload=515915]: Done uploading piece. err: <nil> Dec 30 04:00:15.450738748 (588h31m15.321821783s) [noise=true , address=145.79.227.6:4258 , piece_id=KEDB55NEGVAIY5US2SS5PVTU23YTLXSDUXJCMMOIJKHEZPXPY64Q, seg_pos={0 0}, upload=515915]: Done uploading piece. err: <nil> Dec 30 04:00:15.658184446 (588h31m15.529267491s) [noise=true , address=45.91.116.5:4297 , piece_id=R3PNCLCPRKOF5EJNUIVK4DDO7QKXFTQHOQTPIFOIFKUIPAKSFNIQ, seg_pos={0 0}, upload=515915]: Done uploading piece. err: <nil> Dec 30 04:00:15.682045152 (588h31m15.553128197s) [noise=true , address=77.90.31.13:4626 , piece_id=L34HSQ2E6PJPQMBMI55PTN4QMKBMREI3NV3NLBUZREONPB6N77NA, seg_pos={0 0}, upload=515915]: Done uploading piece. err: <nil> Dec 30 04:00:15.747571129 (588h31m15.618654174s) [noise=true , address=77.90.31.10:4519 , piece_id=2N2J2PXCARQI54VWOCYMXAMN6YXIRIWJYZP6OR4QDVYLXXF4DLXA, seg_pos={0 0}, upload=515915]: Done uploading piece. err: <nil> Dec 30 04:00:16.112287537 (588h31m15.983370612s) [noise=true , address=45.91.116.6:4976 , piece_id=ZJINSWDTALZDL723UY6K7EV3LKIOFGBCXHNUQENXRODNDBKFMK3A, seg_pos={0 0}, upload=515915]: Done uploading piece. err: <nil> Dec 30 04:00:16.112315631 (588h31m15.983398666s) [seg_pos={0 0}, upload=515915]: Segment reached optimal threshold of 13 pieces. Dec 30 04:00:16.112590925 (588h31m15.98367398s) [noise=true , address=77.90.31.12:4590 , piece_id=JUXYOK5VSNHHQGTDW3JD7YN75MSCLC3XDV4VTNEQMPJHELV2RROA, seg_pos={0 0}, upload=515915]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 30 04:00:16.112602677 (588h31m15.983685722s) [noise=true , address=77.90.31.12:4590 , piece_id=JUXYOK5VSNHHQGTDW3JD7YN75MSCLC3XDV4VTNEQMPJHELV2RROA, seg_pos={0 0}, upload=515915]: Long tail cancellation due to optimal threshold Dec 30 04:00:16.112960609 (588h31m15.984043654s) [noise=true , address=45.91.116.12:4227 , piece_id=ZTTT5ONGFIKEEAFMGUKDCZP4YTC7STD7UDUE26UK6OT5TZM5TNFQ, seg_pos={0 0}, upload=515915]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Dec 30 04:00:16.112970908 (588h31m15.984053953s) [noise=true , address=45.91.116.12:4227 , piece_id=ZTTT5ONGFIKEEAFMGUKDCZP4YTC7STD7UDUE26UK6OT5TZM5TNFQ, seg_pos={0 0}, upload=515915]: Long tail cancellation due to optimal threshold Dec 30 04:00:16.113192330 (588h31m15.984275395s) [noise=true , address=45.91.116.14:4192 , piece_id=JRXJT2WPI3VPCS26OVNPNFJKDVTFF332F4BIJAHWQMTSZCBRPX4Q, seg_pos={0 0}, upload=515915]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF Dec 30 04:00:16.113203221 (588h31m15.984286266s) [noise=true , address=45.91.116.14:4192 , piece_id=JRXJT2WPI3VPCS26OVNPNFJKDVTFF332F4BIJAHWQMTSZCBRPX4Q, seg_pos={0 0}, upload=515915]: Long tail cancellation due to optimal threshold Dec 30 04:00:16.113227647 (588h31m15.984310692s) [seg_pos={0 0}, upload=515915]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 04:00:16.113335613 (588h31m15.984418688s) [upload=515915]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:00:16.113344680 (588h31m15.984427715s) [upload=515915]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:00:16.409991296 (588h31m16.281074332s) [upload=515915]: Done uploading segments... Dec 30 04:00:16.410012327 (588h31m16.281095362s) [upload=515915]: Upload finished. err: <nil> Duration: 6.106013499s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 15.442749402s trace: https://jaeger.dev.storj.io/trace/1d81d68a7c17f75d FAIL (2025-12-30 04:01:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 15 , piece_id=6FWCMOTNEZLDIH2F5VRBC3H44WVYAKWTUIL364QUAIC7DGKURRSQ, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.829621721 (588h22m39.700704767s) [noise=true , address=193.149.164.143:4695 , piece_id=C3ZXZXKRGAI2FATBQ2PNRJFNGMV4XKBXU4MSKZVM2AEEH5GVBRBQ, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.829918227 (588h22m39.701001282s) [noise=true , address=145.79.227.17:4998 , piece_id=IMFG6HRRWRXUX3F76A5RTU54HXCFSTPTWH4YJQPHGIOMPALNS5AA, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.830292459 (588h22m39.701375504s) [noise=true , address=45.91.116.12:4249 , piece_id=YLNIBVQGVGZ5CYEZSGLORL3D2ULBBCXN23SSNXUU7LQ67E3E4G7A, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.830455549 (588h22m39.701538594s) [noise=true , address=145.79.227.43:4178 , piece_id=7DIUURHOKK53VG6RMELJUV3D3AWUM23SHXNNVDRFTIURMXETX6QQ, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.830740912 (588h22m39.701823948s) [noise=true , address=77.90.31.16:4759 , piece_id=W3RFMAKOXSYIDCA27H7CKEXSIKCMRUO72NITXH5WDM4FFK6XPDAQ, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.830892231 (588h22m39.701975266s) [noise=true , address=145.79.227.45:4250 , piece_id=MDFRH2T6ZDAOE25DJTGHA6YSXS6CRL5OBES5T2FDTGY6EOV7LINQ, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.831333561 (588h22m39.702416606s) [noise=true , address=145.79.227.40:4053 , piece_id=AKAAPJDTKY3BB2OB5EFW4IYMYPOAPL4VDS5VEHJF3U3TA7JHZ4FQ, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.831388847 (588h22m39.702471882s) [noise=true , address=145.79.227.34:4916 , piece_id=DMOYHOUOVSMTMPZBY7MV5LYQ5JQDV4MKRJHBH7FYV2TBL37AFYSA, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.831789770 (588h22m39.702872815s) [noise=true , address=64.49.13.26:4946 , piece_id=X7CGAXO4356EG4I52QB5LQEM33UD2FJFS56YFEH7FNHAG2EBEZFA, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.832180875 (588h22m39.70326391s) [noise=true , address=77.90.31.19:4871 , piece_id=HPPBTDGVLD62WLHSPO36VFYHP5YFORYCBT22PK2CEYRJJM4UQLYA, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.832576678 (588h22m39.703659723s) [noise=true , address=145.79.227.33:4901 , piece_id=ODRNWAOOWKB45HREGAAKMJYNIN2I4OPRLUN5BIVIW7HWFQRTR6NA, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.834204918 (588h22m39.705287963s) [noise=true , address=45.91.116.13:4164 , piece_id=QW7FBIO24JGOXJEGO7YHYFMMTKHAKF64UEGNDXX2KKQLSJ4MH3PA, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.834999141 (588h22m39.706082186s) [noise=true , address=77.90.31.11:4584 , piece_id=WKRB35UKP7L6Z6G5WSJKMFBOF3RPWJYVZJPFBP5G4UT4RQVQP6MA, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.835718220 (588h22m39.706801265s) [noise=true , address=145.79.227.49:4361 , piece_id=ARLKW5A6Y3DVQ25E2RDL5KRFCYBNVIW7WBDL7ZO2WOQDTF5RCRKQ, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:41.118730125 (588h22m40.98981316s) [upload=515790]: Next returned nil segment Dec 30 03:51:41.118750513 (588h22m40.989833558s) [upload=515790]: Waiting for error group managing segments... Dec 30 03:51:41.505594353 (588h22m41.376677388s) [noise=true , address=64.49.13.26:4946 , piece_id=X7CGAXO4356EG4I52QB5LQEM33UD2FJFS56YFEH7FNHAG2EBEZFA, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:41.711421644 (588h22m41.582504689s) [noise=true , address=145.79.227.17:4998 , piece_id=IMFG6HRRWRXUX3F76A5RTU54HXCFSTPTWH4YJQPHGIOMPALNS5AA, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:41.718129295 (588h22m41.58921233s) [noise=true , address=145.79.227.6:4285 , piece_id=WNJGANKULJAMPLIT3KTHRSSAYTUSIIUCW5ZVKAFNURMGNZ3N2P3A, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:41.729197852 (588h22m41.600280887s) [noise=true , address=145.79.227.14:4856 , piece_id=CBZ6JFWGDMBMR7XJBXIETWDBEEXV43HJWJFKH5FY5BF7ANACFDOQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:41.867535754 (588h22m41.738618799s) [noise=true , address=145.79.227.43:4178 , piece_id=7DIUURHOKK53VG6RMELJUV3D3AWUM23SHXNNVDRFTIURMXETX6QQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:41.894207901 (588h22m41.765290946s) [noise=true , address=77.90.31.12:4615 , piece_id=6FWCMOTNEZLDIH2F5VRBC3H44WVYAKWTUIL364QUAIC7DGKURRSQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:41.928246715 (588h22m41.79932975s) [noise=true , address=145.79.227.45:4250 , piece_id=MDFRH2T6ZDAOE25DJTGHA6YSXS6CRL5OBES5T2FDTGY6EOV7LINQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:41.940833472 (588h22m41.811916517s) [noise=true , address=45.91.116.13:4164 , piece_id=QW7FBIO24JGOXJEGO7YHYFMMTKHAKF64UEGNDXX2KKQLSJ4MH3PA, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:42.318460461 (588h22m42.189543506s) [noise=true , address=77.90.31.16:4759 , piece_id=W3RFMAKOXSYIDCA27H7CKEXSIKCMRUO72NITXH5WDM4FFK6XPDAQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:42.367315176 (588h22m42.238398221s) [noise=true , address=77.90.31.19:4871 , piece_id=HPPBTDGVLD62WLHSPO36VFYHP5YFORYCBT22PK2CEYRJJM4UQLYA, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:42.536538860 (588h22m42.407621905s) [noise=true , address=145.79.227.40:4053 , piece_id=AKAAPJDTKY3BB2OB5EFW4IYMYPOAPL4VDS5VEHJF3U3TA7JHZ4FQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:42.637143172 (588h22m42.508226217s) [noise=true , address=145.79.227.44:4200 , piece_id=2KRTBC4EQE7OCNJJ7TPM72KJFYBBXFHJUP2FRENF6KL3CQNAAUQQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:42.715818140 (588h22m42.586901185s) [noise=true , address=45.91.116.12:4249 , piece_id=YLNIBVQGVGZ5CYEZSGLORL3D2ULBBCXN23SSNXUU7LQ67E3E4G7A, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:43.127299661 (588h22m42.998382696s) [noise=true , address=145.79.227.34:4916 , piece_id=DMOYHOUOVSMTMPZBY7MV5LYQ5JQDV4MKRJHBH7FYV2TBL37AFYSA, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:43.127318687 (588h22m42.998401722s) [seg_pos={0 0}, upload=515790]: Segment reached optimal threshold of 14 pieces. Dec 30 03:51:43.127508449 (588h22m42.998591484s) [noise=true , address=145.79.227.49:4361 , piece_id=ARLKW5A6Y3DVQ25E2RDL5KRFCYBNVIW7WBDL7ZO2WOQDTF5RCRKQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k): protocol: expected piece hash; context canceled; EOF Dec 30 03:51:43.127514470 (588h22m42.998597505s) [noise=true , address=145.79.227.49:4361 , piece_id=ARLKW5A6Y3DVQ25E2RDL5KRFCYBNVIW7WBDL7ZO2WOQDTF5RCRKQ, seg_pos={0 0}, upload=515790]: Long tail cancellation due to optimal threshold Dec 30 03:51:43.127619871 (588h22m42.998702906s) [noise=true , address=77.90.31.11:4584 , piece_id=WKRB35UKP7L6Z6G5WSJKMFBOF3RPWJYVZJPFBP5G4UT4RQVQP6MA, seg_pos={0 0}, upload=515790]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iybcvQgVfqHcCSNFAEVpKZJyciSHbBfeAiuk3uNE2LGq4aLfJ): protocol: write tcp 188.166.225.170:43330->77.90.31.11:4584: use of closed network connection; write tcp 188.166.225.170:43330->77.90.31.11:4584: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43330->77.90.31.11:4584: use of closed network connection Dec 30 03:51:43.127624550 (588h22m42.998707585s) [noise=true , address=77.90.31.11:4584 , piece_id=WKRB35UKP7L6Z6G5WSJKMFBOF3RPWJYVZJPFBP5G4UT4RQVQP6MA, seg_pos={0 0}, upload=515790]: Long tail cancellation due to optimal threshold Dec 30 03:51:43.127738065 (588h22m42.998821111s) [noise=true , address=193.149.164.143:4695 , piece_id=C3ZXZXKRGAI2FATBQ2PNRJFNGMV4XKBXU4MSKZVM2AEEH5GVBRBQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh): protocol: expected piece hash; context canceled; EOF Dec 30 03:51:43.127743557 (588h22m42.998826592s) [noise=true , address=193.149.164.143:4695 , piece_id=C3ZXZXKRGAI2FATBQ2PNRJFNGMV4XKBXU4MSKZVM2AEEH5GVBRBQ, seg_pos={0 0}, upload=515790]: Long tail cancellation due to optimal threshold Dec 30 03:51:43.127899784 (588h22m42.998982829s) [noise=true , address=145.79.227.33:4901 , piece_id=ODRNWAOOWKB45HREGAAKMJYNIN2I4OPRLUN5BIVIW7HWFQRTR6NA, seg_pos={0 0}, upload=515790]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1): protocol: write tcp 188.166.225.170:48420->145.79.227.33:4901: use of closed network connection; write tcp 188.166.225.170:48420->145.79.227.33:4901: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48420->145.79.227.33:4901: use of closed network connection Dec 30 03:51:43.127920484 (588h22m42.999003539s) [noise=true , address=145.79.227.33:4901 , piece_id=ODRNWAOOWKB45HREGAAKMJYNIN2I4OPRLUN5BIVIW7HWFQRTR6NA, seg_pos={0 0}, upload=515790]: Long tail cancellation due to optimal threshold Dec 30 03:51:43.128047735 (588h22m42.99913077s) [seg_pos={0 0}, upload=515790]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 03:51:43.128161222 (588h22m42.999244257s) [upload=515790]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 03:51:43.128174597 (588h22m42.999257642s) [upload=515790]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 03:51:43.415948996 (588h22m43.287032052s) [upload=515790]: Done uploading segments... Dec 30 03:51:43.415976829 (588h22m43.287059864s) [upload=515790]: Upload finished. err: <nil> Duration: 15.442779631s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [X.X.X...XXX.X.....XXXX........XX.XX...XXXX...XXXX...X.X...XX] duration: 30.003290224s trace: https://jaeger.dev.storj.io/trace/15b4cf4615988abc FAIL (2025-12-30 04:00:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: R3PNCLCPRKOF5EJNUIVK4DDO7QKXFTQHOQTPIFOIFKUIPAKSFNIQ) context deadline exceeded; piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: L34HSQ2E6PJPQMBMI55PTN4QMKBMREI3NV3NLBUZREONPB6N77NA) context deadline exceeded ======LOGS====== Dec 30 04:00:13.165093016 (588h31m13.036176061s) [upload=515915]: Starting upload Dec 30 04:00:13.165195281 (588h31m13.036278326s) [upload=515915]: Done starting upload Dec 30 04:00:13.165260615 (588h31m13.03634366s) [upload=515915]: Uploading segments... Dec 30 04:00:13.165452350 (588h31m13.036535395s) [upload=515915]: Got next segment. Inline: false Dec 30 04:00:13.165524838 (588h31m13.036607893s) [upload=515915]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 04:00:13.165534637 (588h31m13.036617682s) [upload=515915]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 04:00:13.411787833 (588h31m13.282870888s) [seg_pos={0 0}, upload=515915]: Begin upload segment... Dec 30 04:00:13.411851905 (588h31m13.28293495s) [seg_pos={0 0}, upload=515915]: Done begin upload segment. Dec 30 04:00:13.411918562 (588h31m13.283001607s) [noise=true , address=45.91.116.14:4192 , piece_id=JRXJT2WPI3VPCS26OVNPNFJKDVTFF332F4BIJAHWQMTSZCBRPX4Q, seg_pos={0 0}, upload=515915]: Uploading piece... Dec 30 04:00:13.411947236 (588h31m13.283030281s) [noise=true , address=77.90.31.13:4626 , piece_id=L34HSQ2E6PJPQMBMI55PTN4QMKBMREI3NV3NLBUZREONPB6N77NA, seg_pos={0 0}, upload=515915]: Uploading piece... Dec 30 04:00:13.413006804 (588h31m13.284089849s) [noise=true , address=145.79.227.48:4303 , piece_id=GKDDXUWXTGIOAQMGKOFRQP2G5EQNCWX2DYOV7BRO3S5YTEGSSPPA, seg_pos={0 0}, upload=515915]: Uploading piece... Dec 30 04:00:13.413095222 (588h31m13.284178268s) [noise=true , address=145.79.227.46:4603 , piece_id=ESGLFXALOTXX5KKUAMXPUAI6JFEAIQDPLEFPAQHE44NVGN66YDHQ, seg_pos={0 0}, upload=515915]: Uploading piece... Dec 30 04:00:13.413605173 (588h31m13.284688218s) [noise=true , address=145.79.227.38:4979 , piece_id=NGOZUUNCELYZQGGEFXRSDC3TFBLVZLKLPQWWJXLLYVWXE6RORC7A, seg_pos={0 0}, upload=515915]: Uploading piece... Dec 30 04:00:13.416721317 (588h31m13.287804362s) [noise=true , address=77.90.31.12:4590 , piece_id=JUXYOK5VSNHHQGTDW3JD7YN75MSCLC3XDV4VTNEQMPJHELV2RROA, seg_pos={0 0}, upload=515915]: Uploading piece... Dec 30 04:00:13.417255474 (588h31m13.288338519s) [noise=true , address=145.79.227.36:4519 , piece_id=GDJ7VR575P2DGNOV7VAGMVCYIOMWLSH4XKBF75MIO5HMOEMVHG5A, seg_pos={0 0}, upload=515915]: Uploading piece... Dec 30 04:00:13.419062976 (588h31m13.290146021s) [noise=true , address=145.79.227.18:4006 , piece_id=RULFLOXGDF2FA7U75TIH6TB4R6JNFGLHJIJP4X3RGOU3BHEJAWBA, seg_pos={0 0}, upload=515915]: Uploading piece... Dec 30 04:00:13.419340053 (588h31m13.290423098s) [noise=true , address=45.91.116.6:4976 , piece_id=ZJINSWDTALZDL723UY6K7EV3LKIOFGBCXHNUQENXRODNDBKFMK3A, seg_pos={0 0}, upload=515915]: Uploading piece... Dec 30 04:00:13.420611755 (588h31m13.2916948s) [noise=true , address=45.91.116.12:4227 , piece_id=ZTTT5ONGFIKEEAFMGUKDCZP4YTC7STD7UDUE26UK6OT5TZM5TNFQ, seg_pos={0 0}, upload=515915]: Uploading piece... Dec 30 04:00:13.422017532 (588h31m13.293100577s) [noise=true , address=145.79.227.22:4509 , piece_id=6UV6HKTDAO4PB6OGYJX3KE3ECYH532ZRH3I6JFSWBJGBNW5K6BZA, seg_pos={0 0}, upload=515915]: Uploading piece... Dec 30 04:00:13.422373410 (588h31m13.293456455s) [noise=true , address=145.79.227.6:4258 , piece_id=KEDB55NEGVAIY5US2SS5PVTU23YTLXSDUXJCMMOIJKHEZPXPY64Q, seg_pos={0 0}, upload=515915]: Uploading piece... Dec 30 04:00:13.423506607 (588h31m13.294589653s) [noise=true , address=77.90.31.10:4519 , piece_id=2N2J2PXCARQI54VWOCYMXAMN6YXIRIWJYZP6OR4QDVYLXXF4DLXA, seg_pos={0 0}, upload=515915]: Uploading piece... Dec 30 04:00:13.424538543 (588h31m13.295621588s) [noise=true , address=45.91.116.5:4297 , piece_id=R3PNCLCPRKOF5EJNUIVK4DDO7QKXFTQHOQTPIFOIFKUIPAKSFNIQ, seg_pos={0 0}, upload=515915]: Uploading piece... Dec 30 04:00:13.427878192 (588h31m13.298961227s) [noise=true , address=145.79.227.9:4367 , piece_id=AZ2BN27PZYOF5GSFES2E6O2MOJ2OYBJRXSDGBSWKWSQXMUSSQ4KA, seg_pos={0 0}, upload=515915]: Uploading piece... Dec 30 04:00:13.429392917 (588h31m13.300475962s) [noise=true , address=145.79.227.42:4123 , piece_id=Q2EFDM6TAWZD5BESSWWLGA53KY4QL3HGY4D3AIVSSSJ3EMKLVR7Q, seg_pos={0 0}, upload=515915]: Uploading piece... Dec 30 04:00:15.022757128 (588h31m14.893840173s) [upload=515915]: Next returned nil segment Dec 30 04:00:15.022779361 (588h31m14.893862436s) [upload=515915]: Waiting for error group managing segments... Dec 30 04:00:15.412297827 (588h31m15.283380873s) [noise=true , address=145.79.227.46:4603 , piece_id=ESGLFXALOTXX5KKUAMXPUAI6JFEAIQDPLEFPAQHE44NVGN66YDHQ, seg_pos={0 0}, upload=515915]: Done uploading piece. err: <nil> Dec 30 04:00:15.415334210 (588h31m15.286417255s) [noise=true , address=145.79.227.36:4519 , piece_id=GDJ7VR575P2DGNOV7VAGMVCYIOMWLSH4XKBF75MIO5HMOEMVHG5A, seg_pos={0 0}, upload=515915]: Done uploading piece. err: <nil> Dec 30 04:00:15.416545888 (588h31m15.287628933s) [noise=true , address=145.79.227.48:4303 , piece_id=GKDDXUWXTGIOAQMGKOFRQP2G5EQNCWX2DYOV7BRO3S5YTEGSSPPA, seg_pos={0 0}, upload=515915]: Done uploading piece. err: <nil> Dec 30 04:00:15.428010749 (588h31m15.299093794s) [noise=true , address=145.79.227.9:4367 , piece_id=AZ2BN27PZYOF5GSFES2E6O2MOJ2OYBJRXSDGBSWKWSQXMUSSQ4KA, seg_pos={0 0}, upload=515915]: Done uploading piece. err: <nil> Dec 30 04:00:15.428579111 (588h31m15.299662156s) [noise=true , address=145.79.227.18:4006 , piece_id=RULFLOXGDF2FA7U75TIH6TB4R6JNFGLHJIJP4X3RGOU3BHEJAWBA, seg_pos={0 0}, upload=515915]: Done uploading piece. err: <nil> Dec 30 04:00:15.430778108 (588h31m15.301861163s) [noise=true , address=145.79.227.42:4123 , piece_id=Q2EFDM6TAWZD5BESSWWLGA53KY4QL3HGY4D3AIVSSSJ3EMKLVR7Q, seg_pos={0 0}, upload=515915]: Done uploading piece. err: <nil> Dec 30 04:00:15.433722836 (588h31m15.304805881s) [noise=true , address=145.79.227.22:4509 , piece_id=6UV6HKTDAO4PB6OGYJX3KE3ECYH532ZRH3I6JFSWBJGBNW5K6BZA, seg_pos={0 0}, upload=515915]: Done uploading piece. err: <nil> Dec 30 04:00:15.448367303 (588h31m15.319450348s) [noise=true , address=145.79.227.38:4979 , piece_id=NGOZUUNCELYZQGGEFXRSDC3TFBLVZLKLPQWWJXLLYVWXE6RORC7A, seg_pos={0 0}, upload=515915]: Done uploading piece. err: <nil> Dec 30 04:00:15.450738748 (588h31m15.321821783s) [noise=true , address=145.79.227.6:4258 , piece_id=KEDB55NEGVAIY5US2SS5PVTU23YTLXSDUXJCMMOIJKHEZPXPY64Q, seg_pos={0 0}, upload=515915]: Done uploading piece. err: <nil> Dec 30 04:00:15.658184446 (588h31m15.529267491s) [noise=true , address=45.91.116.5:4297 , piece_id=R3PNCLCPRKOF5EJNUIVK4DDO7QKXFTQHOQTPIFOIFKUIPAKSFNIQ, seg_pos={0 0}, upload=515915]: Done uploading piece. err: <nil> Dec 30 04:00:15.682045152 (588h31m15.553128197s) [noise=true , address=77.90.31.13:4626 , piece_id=L34HSQ2E6PJPQMBMI55PTN4QMKBMREI3NV3NLBUZREONPB6N77NA, seg_pos={0 0}, upload=515915]: Done uploading piece. err: <nil> Dec 30 04:00:15.747571129 (588h31m15.618654174s) [noise=true , address=77.90.31.10:4519 , piece_id=2N2J2PXCARQI54VWOCYMXAMN6YXIRIWJYZP6OR4QDVYLXXF4DLXA, seg_pos={0 0}, upload=515915]: Done uploading piece. err: <nil> Dec 30 04:00:16.112287537 (588h31m15.983370612s) [noise=true , address=45.91.116.6:4976 , piece_id=ZJINSWDTALZDL723UY6K7EV3LKIOFGBCXHNUQENXRODNDBKFMK3A, seg_pos={0 0}, upload=515915]: Done uploading piece. err: <nil> Dec 30 04:00:16.112315631 (588h31m15.983398666s) [seg_pos={0 0}, upload=515915]: Segment reached optimal threshold of 13 pieces. Dec 30 04:00:16.112590925 (588h31m15.98367398s) [noise=true , address=77.90.31.12:4590 , piece_id=JUXYOK5VSNHHQGTDW3JD7YN75MSCLC3XDV4VTNEQMPJHELV2RROA, seg_pos={0 0}, upload=515915]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 30 04:00:16.112602677 (588h31m15.983685722s) [noise=true , address=77.90.31.12:4590 , piece_id=JUXYOK5VSNHHQGTDW3JD7YN75MSCLC3XDV4VTNEQMPJHELV2RROA, seg_pos={0 0}, upload=515915]: Long tail cancellation due to optimal threshold Dec 30 04:00:16.112960609 (588h31m15.984043654s) [noise=true , address=45.91.116.12:4227 , piece_id=ZTTT5ONGFIKEEAFMGUKDCZP4YTC7STD7UDUE26UK6OT5TZM5TNFQ, seg_pos={0 0}, upload=515915]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Dec 30 04:00:16.112970908 (588h31m15.984053953s) [noise=true , address=45.91.116.12:4227 , piece_id=ZTTT5ONGFIKEEAFMGUKDCZP4YTC7STD7UDUE26UK6OT5TZM5TNFQ, seg_pos={0 0}, upload=515915]: Long tail cancellation due to optimal threshold Dec 30 04:00:16.113192330 (588h31m15.984275395s) [noise=true , address=45.91.116.14:4192 , piece_id=JRXJT2WPI3VPCS26OVNPNFJKDVTFF332F4BIJAHWQMTSZCBRPX4Q, seg_pos={0 0}, upload=515915]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF Dec 30 04:00:16.113203221 (588h31m15.984286266s) [noise=true , address=45.91.116.14:4192 , piece_id=JRXJT2WPI3VPCS26OVNPNFJKDVTFF332F4BIJAHWQMTSZCBRPX4Q, seg_pos={0 0}, upload=515915]: Long tail cancellation due to optimal threshold Dec 30 04:00:16.113227647 (588h31m15.984310692s) [seg_pos={0 0}, upload=515915]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 04:00:16.113335613 (588h31m15.984418688s) [upload=515915]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:00:16.113344680 (588h31m15.984427715s) [upload=515915]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:00:16.409991296 (588h31m16.281074332s) [upload=515915]: Done uploading segments... Dec 30 04:00:16.410012327 (588h31m16.281095362s) [upload=515915]: Upload finished. err: <nil> Duration: 30.003319971s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [...X.X..........XX...X.......................X..XXX.........] duration: 3.538848201s trace: https://jaeger.dev.storj.io/trace/532d591e051353a0 FAIL (2025-12-30 04:00:31 UTC) orbiter warning: context deadline exceeded Duration: 3.538877708s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 15.476905323s trace: https://jaeger.dev.storj.io/trace/1d81d68a7c17f75d FAIL (2025-12-30 04:00:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 15 , piece_id=6FWCMOTNEZLDIH2F5VRBC3H44WVYAKWTUIL364QUAIC7DGKURRSQ, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.829621721 (588h22m39.700704767s) [noise=true , address=193.149.164.143:4695 , piece_id=C3ZXZXKRGAI2FATBQ2PNRJFNGMV4XKBXU4MSKZVM2AEEH5GVBRBQ, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.829918227 (588h22m39.701001282s) [noise=true , address=145.79.227.17:4998 , piece_id=IMFG6HRRWRXUX3F76A5RTU54HXCFSTPTWH4YJQPHGIOMPALNS5AA, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.830292459 (588h22m39.701375504s) [noise=true , address=45.91.116.12:4249 , piece_id=YLNIBVQGVGZ5CYEZSGLORL3D2ULBBCXN23SSNXUU7LQ67E3E4G7A, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.830455549 (588h22m39.701538594s) [noise=true , address=145.79.227.43:4178 , piece_id=7DIUURHOKK53VG6RMELJUV3D3AWUM23SHXNNVDRFTIURMXETX6QQ, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.830740912 (588h22m39.701823948s) [noise=true , address=77.90.31.16:4759 , piece_id=W3RFMAKOXSYIDCA27H7CKEXSIKCMRUO72NITXH5WDM4FFK6XPDAQ, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.830892231 (588h22m39.701975266s) [noise=true , address=145.79.227.45:4250 , piece_id=MDFRH2T6ZDAOE25DJTGHA6YSXS6CRL5OBES5T2FDTGY6EOV7LINQ, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.831333561 (588h22m39.702416606s) [noise=true , address=145.79.227.40:4053 , piece_id=AKAAPJDTKY3BB2OB5EFW4IYMYPOAPL4VDS5VEHJF3U3TA7JHZ4FQ, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.831388847 (588h22m39.702471882s) [noise=true , address=145.79.227.34:4916 , piece_id=DMOYHOUOVSMTMPZBY7MV5LYQ5JQDV4MKRJHBH7FYV2TBL37AFYSA, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.831789770 (588h22m39.702872815s) [noise=true , address=64.49.13.26:4946 , piece_id=X7CGAXO4356EG4I52QB5LQEM33UD2FJFS56YFEH7FNHAG2EBEZFA, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.832180875 (588h22m39.70326391s) [noise=true , address=77.90.31.19:4871 , piece_id=HPPBTDGVLD62WLHSPO36VFYHP5YFORYCBT22PK2CEYRJJM4UQLYA, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.832576678 (588h22m39.703659723s) [noise=true , address=145.79.227.33:4901 , piece_id=ODRNWAOOWKB45HREGAAKMJYNIN2I4OPRLUN5BIVIW7HWFQRTR6NA, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.834204918 (588h22m39.705287963s) [noise=true , address=45.91.116.13:4164 , piece_id=QW7FBIO24JGOXJEGO7YHYFMMTKHAKF64UEGNDXX2KKQLSJ4MH3PA, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.834999141 (588h22m39.706082186s) [noise=true , address=77.90.31.11:4584 , piece_id=WKRB35UKP7L6Z6G5WSJKMFBOF3RPWJYVZJPFBP5G4UT4RQVQP6MA, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.835718220 (588h22m39.706801265s) [noise=true , address=145.79.227.49:4361 , piece_id=ARLKW5A6Y3DVQ25E2RDL5KRFCYBNVIW7WBDL7ZO2WOQDTF5RCRKQ, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:41.118730125 (588h22m40.98981316s) [upload=515790]: Next returned nil segment Dec 30 03:51:41.118750513 (588h22m40.989833558s) [upload=515790]: Waiting for error group managing segments... Dec 30 03:51:41.505594353 (588h22m41.376677388s) [noise=true , address=64.49.13.26:4946 , piece_id=X7CGAXO4356EG4I52QB5LQEM33UD2FJFS56YFEH7FNHAG2EBEZFA, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:41.711421644 (588h22m41.582504689s) [noise=true , address=145.79.227.17:4998 , piece_id=IMFG6HRRWRXUX3F76A5RTU54HXCFSTPTWH4YJQPHGIOMPALNS5AA, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:41.718129295 (588h22m41.58921233s) [noise=true , address=145.79.227.6:4285 , piece_id=WNJGANKULJAMPLIT3KTHRSSAYTUSIIUCW5ZVKAFNURMGNZ3N2P3A, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:41.729197852 (588h22m41.600280887s) [noise=true , address=145.79.227.14:4856 , piece_id=CBZ6JFWGDMBMR7XJBXIETWDBEEXV43HJWJFKH5FY5BF7ANACFDOQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:41.867535754 (588h22m41.738618799s) [noise=true , address=145.79.227.43:4178 , piece_id=7DIUURHOKK53VG6RMELJUV3D3AWUM23SHXNNVDRFTIURMXETX6QQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:41.894207901 (588h22m41.765290946s) [noise=true , address=77.90.31.12:4615 , piece_id=6FWCMOTNEZLDIH2F5VRBC3H44WVYAKWTUIL364QUAIC7DGKURRSQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:41.928246715 (588h22m41.79932975s) [noise=true , address=145.79.227.45:4250 , piece_id=MDFRH2T6ZDAOE25DJTGHA6YSXS6CRL5OBES5T2FDTGY6EOV7LINQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:41.940833472 (588h22m41.811916517s) [noise=true , address=45.91.116.13:4164 , piece_id=QW7FBIO24JGOXJEGO7YHYFMMTKHAKF64UEGNDXX2KKQLSJ4MH3PA, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:42.318460461 (588h22m42.189543506s) [noise=true , address=77.90.31.16:4759 , piece_id=W3RFMAKOXSYIDCA27H7CKEXSIKCMRUO72NITXH5WDM4FFK6XPDAQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:42.367315176 (588h22m42.238398221s) [noise=true , address=77.90.31.19:4871 , piece_id=HPPBTDGVLD62WLHSPO36VFYHP5YFORYCBT22PK2CEYRJJM4UQLYA, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:42.536538860 (588h22m42.407621905s) [noise=true , address=145.79.227.40:4053 , piece_id=AKAAPJDTKY3BB2OB5EFW4IYMYPOAPL4VDS5VEHJF3U3TA7JHZ4FQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:42.637143172 (588h22m42.508226217s) [noise=true , address=145.79.227.44:4200 , piece_id=2KRTBC4EQE7OCNJJ7TPM72KJFYBBXFHJUP2FRENF6KL3CQNAAUQQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:42.715818140 (588h22m42.586901185s) [noise=true , address=45.91.116.12:4249 , piece_id=YLNIBVQGVGZ5CYEZSGLORL3D2ULBBCXN23SSNXUU7LQ67E3E4G7A, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:43.127299661 (588h22m42.998382696s) [noise=true , address=145.79.227.34:4916 , piece_id=DMOYHOUOVSMTMPZBY7MV5LYQ5JQDV4MKRJHBH7FYV2TBL37AFYSA, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:43.127318687 (588h22m42.998401722s) [seg_pos={0 0}, upload=515790]: Segment reached optimal threshold of 14 pieces. Dec 30 03:51:43.127508449 (588h22m42.998591484s) [noise=true , address=145.79.227.49:4361 , piece_id=ARLKW5A6Y3DVQ25E2RDL5KRFCYBNVIW7WBDL7ZO2WOQDTF5RCRKQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k): protocol: expected piece hash; context canceled; EOF Dec 30 03:51:43.127514470 (588h22m42.998597505s) [noise=true , address=145.79.227.49:4361 , piece_id=ARLKW5A6Y3DVQ25E2RDL5KRFCYBNVIW7WBDL7ZO2WOQDTF5RCRKQ, seg_pos={0 0}, upload=515790]: Long tail cancellation due to optimal threshold Dec 30 03:51:43.127619871 (588h22m42.998702906s) [noise=true , address=77.90.31.11:4584 , piece_id=WKRB35UKP7L6Z6G5WSJKMFBOF3RPWJYVZJPFBP5G4UT4RQVQP6MA, seg_pos={0 0}, upload=515790]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iybcvQgVfqHcCSNFAEVpKZJyciSHbBfeAiuk3uNE2LGq4aLfJ): protocol: write tcp 188.166.225.170:43330->77.90.31.11:4584: use of closed network connection; write tcp 188.166.225.170:43330->77.90.31.11:4584: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43330->77.90.31.11:4584: use of closed network connection Dec 30 03:51:43.127624550 (588h22m42.998707585s) [noise=true , address=77.90.31.11:4584 , piece_id=WKRB35UKP7L6Z6G5WSJKMFBOF3RPWJYVZJPFBP5G4UT4RQVQP6MA, seg_pos={0 0}, upload=515790]: Long tail cancellation due to optimal threshold Dec 30 03:51:43.127738065 (588h22m42.998821111s) [noise=true , address=193.149.164.143:4695 , piece_id=C3ZXZXKRGAI2FATBQ2PNRJFNGMV4XKBXU4MSKZVM2AEEH5GVBRBQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh): protocol: expected piece hash; context canceled; EOF Dec 30 03:51:43.127743557 (588h22m42.998826592s) [noise=true , address=193.149.164.143:4695 , piece_id=C3ZXZXKRGAI2FATBQ2PNRJFNGMV4XKBXU4MSKZVM2AEEH5GVBRBQ, seg_pos={0 0}, upload=515790]: Long tail cancellation due to optimal threshold Dec 30 03:51:43.127899784 (588h22m42.998982829s) [noise=true , address=145.79.227.33:4901 , piece_id=ODRNWAOOWKB45HREGAAKMJYNIN2I4OPRLUN5BIVIW7HWFQRTR6NA, seg_pos={0 0}, upload=515790]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1): protocol: write tcp 188.166.225.170:48420->145.79.227.33:4901: use of closed network connection; write tcp 188.166.225.170:48420->145.79.227.33:4901: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48420->145.79.227.33:4901: use of closed network connection Dec 30 03:51:43.127920484 (588h22m42.999003539s) [noise=true , address=145.79.227.33:4901 , piece_id=ODRNWAOOWKB45HREGAAKMJYNIN2I4OPRLUN5BIVIW7HWFQRTR6NA, seg_pos={0 0}, upload=515790]: Long tail cancellation due to optimal threshold Dec 30 03:51:43.128047735 (588h22m42.99913077s) [seg_pos={0 0}, upload=515790]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 03:51:43.128161222 (588h22m42.999244257s) [upload=515790]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 03:51:43.128174597 (588h22m42.999257642s) [upload=515790]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 03:51:43.415948996 (588h22m43.287032052s) [upload=515790]: Done uploading segments... Dec 30 03:51:43.415976829 (588h22m43.287059864s) [upload=515790]: Upload finished. err: <nil> Duration: 15.477025121s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 31.007021511s trace: https://jaeger.dev.storj.io/trace/1d81d68a7c17f75d FAIL (2025-12-30 03:59:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 15 , piece_id=6FWCMOTNEZLDIH2F5VRBC3H44WVYAKWTUIL364QUAIC7DGKURRSQ, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.829621721 (588h22m39.700704767s) [noise=true , address=193.149.164.143:4695 , piece_id=C3ZXZXKRGAI2FATBQ2PNRJFNGMV4XKBXU4MSKZVM2AEEH5GVBRBQ, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.829918227 (588h22m39.701001282s) [noise=true , address=145.79.227.17:4998 , piece_id=IMFG6HRRWRXUX3F76A5RTU54HXCFSTPTWH4YJQPHGIOMPALNS5AA, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.830292459 (588h22m39.701375504s) [noise=true , address=45.91.116.12:4249 , piece_id=YLNIBVQGVGZ5CYEZSGLORL3D2ULBBCXN23SSNXUU7LQ67E3E4G7A, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.830455549 (588h22m39.701538594s) [noise=true , address=145.79.227.43:4178 , piece_id=7DIUURHOKK53VG6RMELJUV3D3AWUM23SHXNNVDRFTIURMXETX6QQ, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.830740912 (588h22m39.701823948s) [noise=true , address=77.90.31.16:4759 , piece_id=W3RFMAKOXSYIDCA27H7CKEXSIKCMRUO72NITXH5WDM4FFK6XPDAQ, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.830892231 (588h22m39.701975266s) [noise=true , address=145.79.227.45:4250 , piece_id=MDFRH2T6ZDAOE25DJTGHA6YSXS6CRL5OBES5T2FDTGY6EOV7LINQ, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.831333561 (588h22m39.702416606s) [noise=true , address=145.79.227.40:4053 , piece_id=AKAAPJDTKY3BB2OB5EFW4IYMYPOAPL4VDS5VEHJF3U3TA7JHZ4FQ, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.831388847 (588h22m39.702471882s) [noise=true , address=145.79.227.34:4916 , piece_id=DMOYHOUOVSMTMPZBY7MV5LYQ5JQDV4MKRJHBH7FYV2TBL37AFYSA, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.831789770 (588h22m39.702872815s) [noise=true , address=64.49.13.26:4946 , piece_id=X7CGAXO4356EG4I52QB5LQEM33UD2FJFS56YFEH7FNHAG2EBEZFA, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.832180875 (588h22m39.70326391s) [noise=true , address=77.90.31.19:4871 , piece_id=HPPBTDGVLD62WLHSPO36VFYHP5YFORYCBT22PK2CEYRJJM4UQLYA, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.832576678 (588h22m39.703659723s) [noise=true , address=145.79.227.33:4901 , piece_id=ODRNWAOOWKB45HREGAAKMJYNIN2I4OPRLUN5BIVIW7HWFQRTR6NA, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.834204918 (588h22m39.705287963s) [noise=true , address=45.91.116.13:4164 , piece_id=QW7FBIO24JGOXJEGO7YHYFMMTKHAKF64UEGNDXX2KKQLSJ4MH3PA, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.834999141 (588h22m39.706082186s) [noise=true , address=77.90.31.11:4584 , piece_id=WKRB35UKP7L6Z6G5WSJKMFBOF3RPWJYVZJPFBP5G4UT4RQVQP6MA, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.835718220 (588h22m39.706801265s) [noise=true , address=145.79.227.49:4361 , piece_id=ARLKW5A6Y3DVQ25E2RDL5KRFCYBNVIW7WBDL7ZO2WOQDTF5RCRKQ, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:41.118730125 (588h22m40.98981316s) [upload=515790]: Next returned nil segment Dec 30 03:51:41.118750513 (588h22m40.989833558s) [upload=515790]: Waiting for error group managing segments... Dec 30 03:51:41.505594353 (588h22m41.376677388s) [noise=true , address=64.49.13.26:4946 , piece_id=X7CGAXO4356EG4I52QB5LQEM33UD2FJFS56YFEH7FNHAG2EBEZFA, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:41.711421644 (588h22m41.582504689s) [noise=true , address=145.79.227.17:4998 , piece_id=IMFG6HRRWRXUX3F76A5RTU54HXCFSTPTWH4YJQPHGIOMPALNS5AA, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:41.718129295 (588h22m41.58921233s) [noise=true , address=145.79.227.6:4285 , piece_id=WNJGANKULJAMPLIT3KTHRSSAYTUSIIUCW5ZVKAFNURMGNZ3N2P3A, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:41.729197852 (588h22m41.600280887s) [noise=true , address=145.79.227.14:4856 , piece_id=CBZ6JFWGDMBMR7XJBXIETWDBEEXV43HJWJFKH5FY5BF7ANACFDOQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:41.867535754 (588h22m41.738618799s) [noise=true , address=145.79.227.43:4178 , piece_id=7DIUURHOKK53VG6RMELJUV3D3AWUM23SHXNNVDRFTIURMXETX6QQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:41.894207901 (588h22m41.765290946s) [noise=true , address=77.90.31.12:4615 , piece_id=6FWCMOTNEZLDIH2F5VRBC3H44WVYAKWTUIL364QUAIC7DGKURRSQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:41.928246715 (588h22m41.79932975s) [noise=true , address=145.79.227.45:4250 , piece_id=MDFRH2T6ZDAOE25DJTGHA6YSXS6CRL5OBES5T2FDTGY6EOV7LINQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:41.940833472 (588h22m41.811916517s) [noise=true , address=45.91.116.13:4164 , piece_id=QW7FBIO24JGOXJEGO7YHYFMMTKHAKF64UEGNDXX2KKQLSJ4MH3PA, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:42.318460461 (588h22m42.189543506s) [noise=true , address=77.90.31.16:4759 , piece_id=W3RFMAKOXSYIDCA27H7CKEXSIKCMRUO72NITXH5WDM4FFK6XPDAQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:42.367315176 (588h22m42.238398221s) [noise=true , address=77.90.31.19:4871 , piece_id=HPPBTDGVLD62WLHSPO36VFYHP5YFORYCBT22PK2CEYRJJM4UQLYA, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:42.536538860 (588h22m42.407621905s) [noise=true , address=145.79.227.40:4053 , piece_id=AKAAPJDTKY3BB2OB5EFW4IYMYPOAPL4VDS5VEHJF3U3TA7JHZ4FQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:42.637143172 (588h22m42.508226217s) [noise=true , address=145.79.227.44:4200 , piece_id=2KRTBC4EQE7OCNJJ7TPM72KJFYBBXFHJUP2FRENF6KL3CQNAAUQQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:42.715818140 (588h22m42.586901185s) [noise=true , address=45.91.116.12:4249 , piece_id=YLNIBVQGVGZ5CYEZSGLORL3D2ULBBCXN23SSNXUU7LQ67E3E4G7A, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:43.127299661 (588h22m42.998382696s) [noise=true , address=145.79.227.34:4916 , piece_id=DMOYHOUOVSMTMPZBY7MV5LYQ5JQDV4MKRJHBH7FYV2TBL37AFYSA, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:43.127318687 (588h22m42.998401722s) [seg_pos={0 0}, upload=515790]: Segment reached optimal threshold of 14 pieces. Dec 30 03:51:43.127508449 (588h22m42.998591484s) [noise=true , address=145.79.227.49:4361 , piece_id=ARLKW5A6Y3DVQ25E2RDL5KRFCYBNVIW7WBDL7ZO2WOQDTF5RCRKQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k): protocol: expected piece hash; context canceled; EOF Dec 30 03:51:43.127514470 (588h22m42.998597505s) [noise=true , address=145.79.227.49:4361 , piece_id=ARLKW5A6Y3DVQ25E2RDL5KRFCYBNVIW7WBDL7ZO2WOQDTF5RCRKQ, seg_pos={0 0}, upload=515790]: Long tail cancellation due to optimal threshold Dec 30 03:51:43.127619871 (588h22m42.998702906s) [noise=true , address=77.90.31.11:4584 , piece_id=WKRB35UKP7L6Z6G5WSJKMFBOF3RPWJYVZJPFBP5G4UT4RQVQP6MA, seg_pos={0 0}, upload=515790]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iybcvQgVfqHcCSNFAEVpKZJyciSHbBfeAiuk3uNE2LGq4aLfJ): protocol: write tcp 188.166.225.170:43330->77.90.31.11:4584: use of closed network connection; write tcp 188.166.225.170:43330->77.90.31.11:4584: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43330->77.90.31.11:4584: use of closed network connection Dec 30 03:51:43.127624550 (588h22m42.998707585s) [noise=true , address=77.90.31.11:4584 , piece_id=WKRB35UKP7L6Z6G5WSJKMFBOF3RPWJYVZJPFBP5G4UT4RQVQP6MA, seg_pos={0 0}, upload=515790]: Long tail cancellation due to optimal threshold Dec 30 03:51:43.127738065 (588h22m42.998821111s) [noise=true , address=193.149.164.143:4695 , piece_id=C3ZXZXKRGAI2FATBQ2PNRJFNGMV4XKBXU4MSKZVM2AEEH5GVBRBQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh): protocol: expected piece hash; context canceled; EOF Dec 30 03:51:43.127743557 (588h22m42.998826592s) [noise=true , address=193.149.164.143:4695 , piece_id=C3ZXZXKRGAI2FATBQ2PNRJFNGMV4XKBXU4MSKZVM2AEEH5GVBRBQ, seg_pos={0 0}, upload=515790]: Long tail cancellation due to optimal threshold Dec 30 03:51:43.127899784 (588h22m42.998982829s) [noise=true , address=145.79.227.33:4901 , piece_id=ODRNWAOOWKB45HREGAAKMJYNIN2I4OPRLUN5BIVIW7HWFQRTR6NA, seg_pos={0 0}, upload=515790]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1): protocol: write tcp 188.166.225.170:48420->145.79.227.33:4901: use of closed network connection; write tcp 188.166.225.170:48420->145.79.227.33:4901: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48420->145.79.227.33:4901: use of closed network connection Dec 30 03:51:43.127920484 (588h22m42.999003539s) [noise=true , address=145.79.227.33:4901 , piece_id=ODRNWAOOWKB45HREGAAKMJYNIN2I4OPRLUN5BIVIW7HWFQRTR6NA, seg_pos={0 0}, upload=515790]: Long tail cancellation due to optimal threshold Dec 30 03:51:43.128047735 (588h22m42.99913077s) [seg_pos={0 0}, upload=515790]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 03:51:43.128161222 (588h22m42.999244257s) [upload=515790]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 03:51:43.128174597 (588h22m42.999257642s) [upload=515790]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 03:51:43.415948996 (588h22m43.287032052s) [upload=515790]: Done uploading segments... Dec 30 03:51:43.415976829 (588h22m43.287059864s) [upload=515790]: Upload finished. err: <nil> Duration: 31.007042532s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [....X.X..........XX...X.......................X..XXX........] duration: 3.446733371s trace: https://jaeger.dev.storj.io/trace/532d591e051353a0 FAIL (2025-12-30 03:59:54 UTC) orbiter warning: context deadline exceeded Duration: 3.446757305s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [.X.X.X...XXX.X.....XXXX........XX.XX...XXXX...XXXX...X.X...X] duration: 30.003405918s trace: https://jaeger.dev.storj.io/trace/3392f045ecf7e796 FAIL (2025-12-30 03:59:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ts... Dec 30 03:59:13.232640292 (588h30m13.103723327s) [upload=515898]: Got next segment. Inline: false Dec 30 03:59:13.232687953 (588h30m13.103770988s) [upload=515898]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 03:59:13.232715506 (588h30m13.103798541s) [upload=515898]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 03:59:13.475499485 (588h30m13.34658253s) [seg_pos={0 0}, upload=515898]: Begin upload segment... Dec 30 03:59:13.475847237 (588h30m13.346930272s) [seg_pos={0 0}, upload=515898]: Done begin upload segment. Dec 30 03:59:13.476303346 (588h30m13.347386381s) [noise=true , address=45.91.116.11:4117 , piece_id=RURF3VH4IRO3SBFQIKNUWDY3E2SYOI7FOMOV6OYAAHY2QQETCFWQ, seg_pos={0 0}, upload=515898]: Uploading piece... Dec 30 03:59:13.477626375 (588h30m13.34870942s) [noise=true , address=145.79.227.45:4229 , piece_id=5TRCNG4YJAXMGSJ42NGO4YFGOUPWORUQ6S2ZFYCOB6UODKYHI3TQ, seg_pos={0 0}, upload=515898]: Uploading piece... Dec 30 03:59:13.478666586 (588h30m13.349749641s) [noise=true , address=45.91.116.9:4264 , piece_id=UTD373OHVAVEMMLG3RTMEVWQT5OHVZNT2JVVOIF74CLCBD2PGYTA, seg_pos={0 0}, upload=515898]: Uploading piece... Dec 30 03:59:13.479370868 (588h30m13.350453903s) [noise=true , address=145.79.227.22:4509 , piece_id=CG7JBXZ7BATMH4SZQE7UTHCPRI2JHGM72FBQSGZ6PFQSB5ZRBWRA, seg_pos={0 0}, upload=515898]: Uploading piece... Dec 30 03:59:13.480805970 (588h30m13.351889005s) [noise=true , address=145.79.227.15:4875 , piece_id=HJ5DCR76WNBVBNR7QHYXVRXIS5OF4ESXD7NCCJ2QLYKV42NJM3DA, seg_pos={0 0}, upload=515898]: Uploading piece... Dec 30 03:59:13.481892198 (588h30m13.352975233s) [noise=true , address=145.79.227.5:4221 , piece_id=JHIJXIYBYQTTLDXST56BLWAA4JFQRA34WPKU3MEGGKHHCN4JUH5A, seg_pos={0 0}, upload=515898]: Uploading piece... Dec 30 03:59:13.483526600 (588h30m13.354609645s) [noise=true , address=145.79.227.3:4149 , piece_id=3KHOKNMFR3FWCPVIWBIFHNZER5CTVZU5TN2NUAH7IQPZ2E5OG77Q, seg_pos={0 0}, upload=515898]: Uploading piece... Dec 30 03:59:13.485369309 (588h30m13.356452354s) [noise=true , address=45.91.116.12:4225 , piece_id=LSVV4UARPZW2M72GEQYRWPQ4ENG5665EPCQFV55XCLJPQUZCZT7Q, seg_pos={0 0}, upload=515898]: Uploading piece... Dec 30 03:59:13.486445118 (588h30m13.357528153s) [noise=true , address=145.79.227.46:4602 , piece_id=GHOP56I4QMAOAUHYE6VUNYS4I4WSVM37JGAU5X5UHJ6WVM4JEHPA, seg_pos={0 0}, upload=515898]: Uploading piece... Dec 30 03:59:13.487137677 (588h30m13.358220712s) [noise=true , address=145.79.227.16:4911 , piece_id=N77XYZJBOJLHOHQAYZ5XTSPGZZ2HU3SE6RECTL63MX4CS3GBLSDA, seg_pos={0 0}, upload=515898]: Uploading piece... Dec 30 03:59:13.489168253 (588h30m13.360251299s) [noise=true , address=145.79.227.17:4970 , piece_id=UQ2IBE7UFSBHK3QCMNHT2VA6MINHL3MQY3FNCXZH4M2RBJLVYP2A, seg_pos={0 0}, upload=515898]: Uploading piece... Dec 30 03:59:13.490029483 (588h30m13.361112528s) [noise=true , address=145.79.227.19:4077 , piece_id=36655QCGU4DJHY5FLXOSEHRBD7256EHW377X75KP77EHV3QTFOEA, seg_pos={0 0}, upload=515898]: Uploading piece... Dec 30 03:59:13.493769455 (588h30m13.3648525s) [noise=true , address=145.79.227.2:4114 , piece_id=MJAA3LN4YQKPHG4MZ6BCNYLC5ZEVD5I5CNRZIQNMERXSJBK3LZ5Q, seg_pos={0 0}, upload=515898]: Uploading piece... Dec 30 03:59:13.495504720 (588h30m13.366587805s) [noise=true , address=45.91.116.14:4190 , piece_id=67SHRAYWYVYAHGPVX2KFIYPTZMLRBEMWITR7RL3DZFDRGCADID3Q, seg_pos={0 0}, upload=515898]: Uploading piece... Dec 30 03:59:13.496471821 (588h30m13.367554857s) [noise=true , address=145.79.227.4:4186 , piece_id=GM3NSUGHUNHKJJZMXXJ5XKO322GMJ54MQID4MAM2MWHMPU7NHEMQ, seg_pos={0 0}, upload=515898]: Uploading piece... Dec 30 03:59:13.504576254 (588h30m13.375659289s) [noise=true , address=145.79.227.6:4259 , piece_id=HF4AORAHNYCL5GZH2OEJFJPRDZHTVSS2BNFYDCPXOGQYGWJRJ47Q, seg_pos={0 0}, upload=515898]: Uploading piece... Dec 30 03:59:15.087619070 (588h30m14.958702105s) [upload=515898]: Next returned nil segment Dec 30 03:59:15.087633438 (588h30m14.958716473s) [upload=515898]: Waiting for error group managing segments... Dec 30 03:59:15.470533952 (588h30m15.341616987s) [noise=true , address=145.79.227.15:4875 , piece_id=HJ5DCR76WNBVBNR7QHYXVRXIS5OF4ESXD7NCCJ2QLYKV42NJM3DA, seg_pos={0 0}, upload=515898]: Done uploading piece. err: <nil> Dec 30 03:59:15.470821160 (588h30m15.341904195s) [noise=true , address=145.79.227.22:4509 , piece_id=CG7JBXZ7BATMH4SZQE7UTHCPRI2JHGM72FBQSGZ6PFQSB5ZRBWRA, seg_pos={0 0}, upload=515898]: Done uploading piece. err: <nil> Dec 30 03:59:15.471722937 (588h30m15.342805972s) [noise=true , address=145.79.227.45:4229 , piece_id=5TRCNG4YJAXMGSJ42NGO4YFGOUPWORUQ6S2ZFYCOB6UODKYHI3TQ, seg_pos={0 0}, upload=515898]: Done uploading piece. err: <nil> Dec 30 03:59:15.478563894 (588h30m15.349646939s) [noise=true , address=145.79.227.17:4970 , piece_id=UQ2IBE7UFSBHK3QCMNHT2VA6MINHL3MQY3FNCXZH4M2RBJLVYP2A, seg_pos={0 0}, upload=515898]: Done uploading piece. err: <nil> Dec 30 03:59:15.481775569 (588h30m15.352858614s) [noise=true , address=145.79.227.5:4221 , piece_id=JHIJXIYBYQTTLDXST56BLWAA4JFQRA34WPKU3MEGGKHHCN4JUH5A, seg_pos={0 0}, upload=515898]: Done uploading piece. err: <nil> Dec 30 03:59:15.494500922 (588h30m15.365583967s) [noise=true , address=145.79.227.6:4259 , piece_id=HF4AORAHNYCL5GZH2OEJFJPRDZHTVSS2BNFYDCPXOGQYGWJRJ47Q, seg_pos={0 0}, upload=515898]: Done uploading piece. err: <nil> Dec 30 03:59:15.507229890 (588h30m15.378312935s) [noise=true , address=145.79.227.3:4149 , piece_id=3KHOKNMFR3FWCPVIWBIFHNZER5CTVZU5TN2NUAH7IQPZ2E5OG77Q, seg_pos={0 0}, upload=515898]: Done uploading piece. err: <nil> Dec 30 03:59:15.510810869 (588h30m15.381893904s) [noise=true , address=145.79.227.2:4114 , piece_id=MJAA3LN4YQKPHG4MZ6BCNYLC5ZEVD5I5CNRZIQNMERXSJBK3LZ5Q, seg_pos={0 0}, upload=515898]: Done uploading piece. err: <nil> Dec 30 03:59:15.512496920 (588h30m15.383579955s) [noise=true , address=145.79.227.46:4602 , piece_id=GHOP56I4QMAOAUHYE6VUNYS4I4WSVM37JGAU5X5UHJ6WVM4JEHPA, seg_pos={0 0}, upload=515898]: Done uploading piece. err: <nil> Dec 30 03:59:15.514505113 (588h30m15.385588148s) [noise=true , address=145.79.227.19:4077 , piece_id=36655QCGU4DJHY5FLXOSEHRBD7256EHW377X75KP77EHV3QTFOEA, seg_pos={0 0}, upload=515898]: Done uploading piece. err: <nil> Dec 30 03:59:15.515557268 (588h30m15.386640303s) [noise=true , address=145.79.227.16:4911 , piece_id=N77XYZJBOJLHOHQAYZ5XTSPGZZ2HU3SE6RECTL63MX4CS3GBLSDA, seg_pos={0 0}, upload=515898]: Done uploading piece. err: <nil> Dec 30 03:59:15.805602804 (588h30m15.676685839s) [noise=true , address=45.91.116.11:4117 , piece_id=RURF3VH4IRO3SBFQIKNUWDY3E2SYOI7FOMOV6OYAAHY2QQETCFWQ, seg_pos={0 0}, upload=515898]: Done uploading piece. err: <nil> Dec 30 03:59:15.828292911 (588h30m15.699375976s) [noise=true , address=45.91.116.14:4190 , piece_id=67SHRAYWYVYAHGPVX2KFIYPTZMLRBEMWITR7RL3DZFDRGCADID3Q, seg_pos={0 0}, upload=515898]: Done uploading piece. err: <nil> Dec 30 03:59:15.828319863 (588h30m15.699402908s) [seg_pos={0 0}, upload=515898]: Segment reached optimal threshold of 13 pieces. Dec 30 03:59:15.828566643 (588h30m15.699649698s) [noise=true , address=45.91.116.12:4225 , piece_id=LSVV4UARPZW2M72GEQYRWPQ4ENG5665EPCQFV55XCLJPQUZCZT7Q, seg_pos={0 0}, upload=515898]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:36298->45.91.116.12:4225: use of closed network connection; write tcp 188.166.225.170:36298->45.91.116.12:4225: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36298->45.91.116.12:4225: use of closed network connection Dec 30 03:59:15.828576803 (588h30m15.699659848s) [noise=true , address=45.91.116.12:4225 , piece_id=LSVV4UARPZW2M72GEQYRWPQ4ENG5665EPCQFV55XCLJPQUZCZT7Q, seg_pos={0 0}, upload=515898]: Long tail cancellation due to optimal threshold Dec 30 03:59:15.828603723 (588h30m15.699686758s) [noise=true , address=45.91.116.9:4264 , piece_id=UTD373OHVAVEMMLG3RTMEVWQT5OHVZNT2JVVOIF74CLCBD2PGYTA, seg_pos={0 0}, upload=515898]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 30 03:59:15.828613643 (588h30m15.699696678s) [noise=true , address=45.91.116.9:4264 , piece_id=UTD373OHVAVEMMLG3RTMEVWQT5OHVZNT2JVVOIF74CLCBD2PGYTA, seg_pos={0 0}, upload=515898]: Long tail cancellation due to optimal threshold Dec 30 03:59:15.828733791 (588h30m15.699816826s) [noise=true , address=145.79.227.4:4186 , piece_id=GM3NSUGHUNHKJJZMXXJ5XKO322GMJ54MQID4MAM2MWHMPU7NHEMQ, seg_pos={0 0}, upload=515898]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht): protocol: write tcp 188.166.225.170:55662->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:55662->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55662->145.79.227.4:4186: use of closed network connection Dec 30 03:59:15.828740153 (588h30m15.699823198s) [noise=true , address=145.79.227.4:4186 , piece_id=GM3NSUGHUNHKJJZMXXJ5XKO322GMJ54MQID4MAM2MWHMPU7NHEMQ, seg_pos={0 0}, upload=515898]: Long tail cancellation due to optimal threshold Dec 30 03:59:15.828754541 (588h30m15.699837586s) [seg_pos={0 0}, upload=515898]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 03:59:15.828822611 (588h30m15.699905646s) [upload=515898]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 03:59:15.828827750 (588h30m15.699910785s) [upload=515898]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 03:59:16.124072578 (588h30m15.995155613s) [upload=515898]: Done uploading segments... Dec 30 03:59:16.124092536 (588h30m15.995175571s) [upload=515898]: Upload finished. err: <nil> Duration: 30.00343289s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 44.92021756s trace: https://jaeger.dev.storj.io/trace/1d81d68a7c17f75d FAIL (2025-12-30 03:59:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 15 , piece_id=6FWCMOTNEZLDIH2F5VRBC3H44WVYAKWTUIL364QUAIC7DGKURRSQ, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.829621721 (588h22m39.700704767s) [noise=true , address=193.149.164.143:4695 , piece_id=C3ZXZXKRGAI2FATBQ2PNRJFNGMV4XKBXU4MSKZVM2AEEH5GVBRBQ, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.829918227 (588h22m39.701001282s) [noise=true , address=145.79.227.17:4998 , piece_id=IMFG6HRRWRXUX3F76A5RTU54HXCFSTPTWH4YJQPHGIOMPALNS5AA, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.830292459 (588h22m39.701375504s) [noise=true , address=45.91.116.12:4249 , piece_id=YLNIBVQGVGZ5CYEZSGLORL3D2ULBBCXN23SSNXUU7LQ67E3E4G7A, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.830455549 (588h22m39.701538594s) [noise=true , address=145.79.227.43:4178 , piece_id=7DIUURHOKK53VG6RMELJUV3D3AWUM23SHXNNVDRFTIURMXETX6QQ, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.830740912 (588h22m39.701823948s) [noise=true , address=77.90.31.16:4759 , piece_id=W3RFMAKOXSYIDCA27H7CKEXSIKCMRUO72NITXH5WDM4FFK6XPDAQ, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.830892231 (588h22m39.701975266s) [noise=true , address=145.79.227.45:4250 , piece_id=MDFRH2T6ZDAOE25DJTGHA6YSXS6CRL5OBES5T2FDTGY6EOV7LINQ, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.831333561 (588h22m39.702416606s) [noise=true , address=145.79.227.40:4053 , piece_id=AKAAPJDTKY3BB2OB5EFW4IYMYPOAPL4VDS5VEHJF3U3TA7JHZ4FQ, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.831388847 (588h22m39.702471882s) [noise=true , address=145.79.227.34:4916 , piece_id=DMOYHOUOVSMTMPZBY7MV5LYQ5JQDV4MKRJHBH7FYV2TBL37AFYSA, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.831789770 (588h22m39.702872815s) [noise=true , address=64.49.13.26:4946 , piece_id=X7CGAXO4356EG4I52QB5LQEM33UD2FJFS56YFEH7FNHAG2EBEZFA, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.832180875 (588h22m39.70326391s) [noise=true , address=77.90.31.19:4871 , piece_id=HPPBTDGVLD62WLHSPO36VFYHP5YFORYCBT22PK2CEYRJJM4UQLYA, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.832576678 (588h22m39.703659723s) [noise=true , address=145.79.227.33:4901 , piece_id=ODRNWAOOWKB45HREGAAKMJYNIN2I4OPRLUN5BIVIW7HWFQRTR6NA, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.834204918 (588h22m39.705287963s) [noise=true , address=45.91.116.13:4164 , piece_id=QW7FBIO24JGOXJEGO7YHYFMMTKHAKF64UEGNDXX2KKQLSJ4MH3PA, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.834999141 (588h22m39.706082186s) [noise=true , address=77.90.31.11:4584 , piece_id=WKRB35UKP7L6Z6G5WSJKMFBOF3RPWJYVZJPFBP5G4UT4RQVQP6MA, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.835718220 (588h22m39.706801265s) [noise=true , address=145.79.227.49:4361 , piece_id=ARLKW5A6Y3DVQ25E2RDL5KRFCYBNVIW7WBDL7ZO2WOQDTF5RCRKQ, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:41.118730125 (588h22m40.98981316s) [upload=515790]: Next returned nil segment Dec 30 03:51:41.118750513 (588h22m40.989833558s) [upload=515790]: Waiting for error group managing segments... Dec 30 03:51:41.505594353 (588h22m41.376677388s) [noise=true , address=64.49.13.26:4946 , piece_id=X7CGAXO4356EG4I52QB5LQEM33UD2FJFS56YFEH7FNHAG2EBEZFA, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:41.711421644 (588h22m41.582504689s) [noise=true , address=145.79.227.17:4998 , piece_id=IMFG6HRRWRXUX3F76A5RTU54HXCFSTPTWH4YJQPHGIOMPALNS5AA, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:41.718129295 (588h22m41.58921233s) [noise=true , address=145.79.227.6:4285 , piece_id=WNJGANKULJAMPLIT3KTHRSSAYTUSIIUCW5ZVKAFNURMGNZ3N2P3A, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:41.729197852 (588h22m41.600280887s) [noise=true , address=145.79.227.14:4856 , piece_id=CBZ6JFWGDMBMR7XJBXIETWDBEEXV43HJWJFKH5FY5BF7ANACFDOQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:41.867535754 (588h22m41.738618799s) [noise=true , address=145.79.227.43:4178 , piece_id=7DIUURHOKK53VG6RMELJUV3D3AWUM23SHXNNVDRFTIURMXETX6QQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:41.894207901 (588h22m41.765290946s) [noise=true , address=77.90.31.12:4615 , piece_id=6FWCMOTNEZLDIH2F5VRBC3H44WVYAKWTUIL364QUAIC7DGKURRSQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:41.928246715 (588h22m41.79932975s) [noise=true , address=145.79.227.45:4250 , piece_id=MDFRH2T6ZDAOE25DJTGHA6YSXS6CRL5OBES5T2FDTGY6EOV7LINQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:41.940833472 (588h22m41.811916517s) [noise=true , address=45.91.116.13:4164 , piece_id=QW7FBIO24JGOXJEGO7YHYFMMTKHAKF64UEGNDXX2KKQLSJ4MH3PA, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:42.318460461 (588h22m42.189543506s) [noise=true , address=77.90.31.16:4759 , piece_id=W3RFMAKOXSYIDCA27H7CKEXSIKCMRUO72NITXH5WDM4FFK6XPDAQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:42.367315176 (588h22m42.238398221s) [noise=true , address=77.90.31.19:4871 , piece_id=HPPBTDGVLD62WLHSPO36VFYHP5YFORYCBT22PK2CEYRJJM4UQLYA, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:42.536538860 (588h22m42.407621905s) [noise=true , address=145.79.227.40:4053 , piece_id=AKAAPJDTKY3BB2OB5EFW4IYMYPOAPL4VDS5VEHJF3U3TA7JHZ4FQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:42.637143172 (588h22m42.508226217s) [noise=true , address=145.79.227.44:4200 , piece_id=2KRTBC4EQE7OCNJJ7TPM72KJFYBBXFHJUP2FRENF6KL3CQNAAUQQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:42.715818140 (588h22m42.586901185s) [noise=true , address=45.91.116.12:4249 , piece_id=YLNIBVQGVGZ5CYEZSGLORL3D2ULBBCXN23SSNXUU7LQ67E3E4G7A, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:43.127299661 (588h22m42.998382696s) [noise=true , address=145.79.227.34:4916 , piece_id=DMOYHOUOVSMTMPZBY7MV5LYQ5JQDV4MKRJHBH7FYV2TBL37AFYSA, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:43.127318687 (588h22m42.998401722s) [seg_pos={0 0}, upload=515790]: Segment reached optimal threshold of 14 pieces. Dec 30 03:51:43.127508449 (588h22m42.998591484s) [noise=true , address=145.79.227.49:4361 , piece_id=ARLKW5A6Y3DVQ25E2RDL5KRFCYBNVIW7WBDL7ZO2WOQDTF5RCRKQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k): protocol: expected piece hash; context canceled; EOF Dec 30 03:51:43.127514470 (588h22m42.998597505s) [noise=true , address=145.79.227.49:4361 , piece_id=ARLKW5A6Y3DVQ25E2RDL5KRFCYBNVIW7WBDL7ZO2WOQDTF5RCRKQ, seg_pos={0 0}, upload=515790]: Long tail cancellation due to optimal threshold Dec 30 03:51:43.127619871 (588h22m42.998702906s) [noise=true , address=77.90.31.11:4584 , piece_id=WKRB35UKP7L6Z6G5WSJKMFBOF3RPWJYVZJPFBP5G4UT4RQVQP6MA, seg_pos={0 0}, upload=515790]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iybcvQgVfqHcCSNFAEVpKZJyciSHbBfeAiuk3uNE2LGq4aLfJ): protocol: write tcp 188.166.225.170:43330->77.90.31.11:4584: use of closed network connection; write tcp 188.166.225.170:43330->77.90.31.11:4584: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43330->77.90.31.11:4584: use of closed network connection Dec 30 03:51:43.127624550 (588h22m42.998707585s) [noise=true , address=77.90.31.11:4584 , piece_id=WKRB35UKP7L6Z6G5WSJKMFBOF3RPWJYVZJPFBP5G4UT4RQVQP6MA, seg_pos={0 0}, upload=515790]: Long tail cancellation due to optimal threshold Dec 30 03:51:43.127738065 (588h22m42.998821111s) [noise=true , address=193.149.164.143:4695 , piece_id=C3ZXZXKRGAI2FATBQ2PNRJFNGMV4XKBXU4MSKZVM2AEEH5GVBRBQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh): protocol: expected piece hash; context canceled; EOF Dec 30 03:51:43.127743557 (588h22m42.998826592s) [noise=true , address=193.149.164.143:4695 , piece_id=C3ZXZXKRGAI2FATBQ2PNRJFNGMV4XKBXU4MSKZVM2AEEH5GVBRBQ, seg_pos={0 0}, upload=515790]: Long tail cancellation due to optimal threshold Dec 30 03:51:43.127899784 (588h22m42.998982829s) [noise=true , address=145.79.227.33:4901 , piece_id=ODRNWAOOWKB45HREGAAKMJYNIN2I4OPRLUN5BIVIW7HWFQRTR6NA, seg_pos={0 0}, upload=515790]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1): protocol: write tcp 188.166.225.170:48420->145.79.227.33:4901: use of closed network connection; write tcp 188.166.225.170:48420->145.79.227.33:4901: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48420->145.79.227.33:4901: use of closed network connection Dec 30 03:51:43.127920484 (588h22m42.999003539s) [noise=true , address=145.79.227.33:4901 , piece_id=ODRNWAOOWKB45HREGAAKMJYNIN2I4OPRLUN5BIVIW7HWFQRTR6NA, seg_pos={0 0}, upload=515790]: Long tail cancellation due to optimal threshold Dec 30 03:51:43.128047735 (588h22m42.99913077s) [seg_pos={0 0}, upload=515790]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 03:51:43.128161222 (588h22m42.999244257s) [upload=515790]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 03:51:43.128174597 (588h22m42.999257642s) [upload=515790]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 03:51:43.415948996 (588h22m43.287032052s) [upload=515790]: Done uploading segments... Dec 30 03:51:43.415976829 (588h22m43.287059864s) [upload=515790]: Upload finished. err: <nil> Duration: 44.920242888s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [X....X.X..........XX...X.......................X..XXX.......] duration: 3.48773419s trace: https://jaeger.dev.storj.io/trace/532d591e051353a0 FAIL (2025-12-30 03:59:02 UTC) orbiter warning: context deadline exceeded Duration: 3.487757755s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 37.638030631s trace: https://jaeger.dev.storj.io/trace/1d81d68a7c17f75d FAIL (2025-12-30 03:58:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 15 , piece_id=6FWCMOTNEZLDIH2F5VRBC3H44WVYAKWTUIL364QUAIC7DGKURRSQ, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.829621721 (588h22m39.700704767s) [noise=true , address=193.149.164.143:4695 , piece_id=C3ZXZXKRGAI2FATBQ2PNRJFNGMV4XKBXU4MSKZVM2AEEH5GVBRBQ, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.829918227 (588h22m39.701001282s) [noise=true , address=145.79.227.17:4998 , piece_id=IMFG6HRRWRXUX3F76A5RTU54HXCFSTPTWH4YJQPHGIOMPALNS5AA, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.830292459 (588h22m39.701375504s) [noise=true , address=45.91.116.12:4249 , piece_id=YLNIBVQGVGZ5CYEZSGLORL3D2ULBBCXN23SSNXUU7LQ67E3E4G7A, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.830455549 (588h22m39.701538594s) [noise=true , address=145.79.227.43:4178 , piece_id=7DIUURHOKK53VG6RMELJUV3D3AWUM23SHXNNVDRFTIURMXETX6QQ, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.830740912 (588h22m39.701823948s) [noise=true , address=77.90.31.16:4759 , piece_id=W3RFMAKOXSYIDCA27H7CKEXSIKCMRUO72NITXH5WDM4FFK6XPDAQ, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.830892231 (588h22m39.701975266s) [noise=true , address=145.79.227.45:4250 , piece_id=MDFRH2T6ZDAOE25DJTGHA6YSXS6CRL5OBES5T2FDTGY6EOV7LINQ, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.831333561 (588h22m39.702416606s) [noise=true , address=145.79.227.40:4053 , piece_id=AKAAPJDTKY3BB2OB5EFW4IYMYPOAPL4VDS5VEHJF3U3TA7JHZ4FQ, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.831388847 (588h22m39.702471882s) [noise=true , address=145.79.227.34:4916 , piece_id=DMOYHOUOVSMTMPZBY7MV5LYQ5JQDV4MKRJHBH7FYV2TBL37AFYSA, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.831789770 (588h22m39.702872815s) [noise=true , address=64.49.13.26:4946 , piece_id=X7CGAXO4356EG4I52QB5LQEM33UD2FJFS56YFEH7FNHAG2EBEZFA, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.832180875 (588h22m39.70326391s) [noise=true , address=77.90.31.19:4871 , piece_id=HPPBTDGVLD62WLHSPO36VFYHP5YFORYCBT22PK2CEYRJJM4UQLYA, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.832576678 (588h22m39.703659723s) [noise=true , address=145.79.227.33:4901 , piece_id=ODRNWAOOWKB45HREGAAKMJYNIN2I4OPRLUN5BIVIW7HWFQRTR6NA, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.834204918 (588h22m39.705287963s) [noise=true , address=45.91.116.13:4164 , piece_id=QW7FBIO24JGOXJEGO7YHYFMMTKHAKF64UEGNDXX2KKQLSJ4MH3PA, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.834999141 (588h22m39.706082186s) [noise=true , address=77.90.31.11:4584 , piece_id=WKRB35UKP7L6Z6G5WSJKMFBOF3RPWJYVZJPFBP5G4UT4RQVQP6MA, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.835718220 (588h22m39.706801265s) [noise=true , address=145.79.227.49:4361 , piece_id=ARLKW5A6Y3DVQ25E2RDL5KRFCYBNVIW7WBDL7ZO2WOQDTF5RCRKQ, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:41.118730125 (588h22m40.98981316s) [upload=515790]: Next returned nil segment Dec 30 03:51:41.118750513 (588h22m40.989833558s) [upload=515790]: Waiting for error group managing segments... Dec 30 03:51:41.505594353 (588h22m41.376677388s) [noise=true , address=64.49.13.26:4946 , piece_id=X7CGAXO4356EG4I52QB5LQEM33UD2FJFS56YFEH7FNHAG2EBEZFA, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:41.711421644 (588h22m41.582504689s) [noise=true , address=145.79.227.17:4998 , piece_id=IMFG6HRRWRXUX3F76A5RTU54HXCFSTPTWH4YJQPHGIOMPALNS5AA, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:41.718129295 (588h22m41.58921233s) [noise=true , address=145.79.227.6:4285 , piece_id=WNJGANKULJAMPLIT3KTHRSSAYTUSIIUCW5ZVKAFNURMGNZ3N2P3A, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:41.729197852 (588h22m41.600280887s) [noise=true , address=145.79.227.14:4856 , piece_id=CBZ6JFWGDMBMR7XJBXIETWDBEEXV43HJWJFKH5FY5BF7ANACFDOQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:41.867535754 (588h22m41.738618799s) [noise=true , address=145.79.227.43:4178 , piece_id=7DIUURHOKK53VG6RMELJUV3D3AWUM23SHXNNVDRFTIURMXETX6QQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:41.894207901 (588h22m41.765290946s) [noise=true , address=77.90.31.12:4615 , piece_id=6FWCMOTNEZLDIH2F5VRBC3H44WVYAKWTUIL364QUAIC7DGKURRSQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:41.928246715 (588h22m41.79932975s) [noise=true , address=145.79.227.45:4250 , piece_id=MDFRH2T6ZDAOE25DJTGHA6YSXS6CRL5OBES5T2FDTGY6EOV7LINQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:41.940833472 (588h22m41.811916517s) [noise=true , address=45.91.116.13:4164 , piece_id=QW7FBIO24JGOXJEGO7YHYFMMTKHAKF64UEGNDXX2KKQLSJ4MH3PA, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:42.318460461 (588h22m42.189543506s) [noise=true , address=77.90.31.16:4759 , piece_id=W3RFMAKOXSYIDCA27H7CKEXSIKCMRUO72NITXH5WDM4FFK6XPDAQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:42.367315176 (588h22m42.238398221s) [noise=true , address=77.90.31.19:4871 , piece_id=HPPBTDGVLD62WLHSPO36VFYHP5YFORYCBT22PK2CEYRJJM4UQLYA, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:42.536538860 (588h22m42.407621905s) [noise=true , address=145.79.227.40:4053 , piece_id=AKAAPJDTKY3BB2OB5EFW4IYMYPOAPL4VDS5VEHJF3U3TA7JHZ4FQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:42.637143172 (588h22m42.508226217s) [noise=true , address=145.79.227.44:4200 , piece_id=2KRTBC4EQE7OCNJJ7TPM72KJFYBBXFHJUP2FRENF6KL3CQNAAUQQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:42.715818140 (588h22m42.586901185s) [noise=true , address=45.91.116.12:4249 , piece_id=YLNIBVQGVGZ5CYEZSGLORL3D2ULBBCXN23SSNXUU7LQ67E3E4G7A, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:43.127299661 (588h22m42.998382696s) [noise=true , address=145.79.227.34:4916 , piece_id=DMOYHOUOVSMTMPZBY7MV5LYQ5JQDV4MKRJHBH7FYV2TBL37AFYSA, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:43.127318687 (588h22m42.998401722s) [seg_pos={0 0}, upload=515790]: Segment reached optimal threshold of 14 pieces. Dec 30 03:51:43.127508449 (588h22m42.998591484s) [noise=true , address=145.79.227.49:4361 , piece_id=ARLKW5A6Y3DVQ25E2RDL5KRFCYBNVIW7WBDL7ZO2WOQDTF5RCRKQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k): protocol: expected piece hash; context canceled; EOF Dec 30 03:51:43.127514470 (588h22m42.998597505s) [noise=true , address=145.79.227.49:4361 , piece_id=ARLKW5A6Y3DVQ25E2RDL5KRFCYBNVIW7WBDL7ZO2WOQDTF5RCRKQ, seg_pos={0 0}, upload=515790]: Long tail cancellation due to optimal threshold Dec 30 03:51:43.127619871 (588h22m42.998702906s) [noise=true , address=77.90.31.11:4584 , piece_id=WKRB35UKP7L6Z6G5WSJKMFBOF3RPWJYVZJPFBP5G4UT4RQVQP6MA, seg_pos={0 0}, upload=515790]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iybcvQgVfqHcCSNFAEVpKZJyciSHbBfeAiuk3uNE2LGq4aLfJ): protocol: write tcp 188.166.225.170:43330->77.90.31.11:4584: use of closed network connection; write tcp 188.166.225.170:43330->77.90.31.11:4584: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43330->77.90.31.11:4584: use of closed network connection Dec 30 03:51:43.127624550 (588h22m42.998707585s) [noise=true , address=77.90.31.11:4584 , piece_id=WKRB35UKP7L6Z6G5WSJKMFBOF3RPWJYVZJPFBP5G4UT4RQVQP6MA, seg_pos={0 0}, upload=515790]: Long tail cancellation due to optimal threshold Dec 30 03:51:43.127738065 (588h22m42.998821111s) [noise=true , address=193.149.164.143:4695 , piece_id=C3ZXZXKRGAI2FATBQ2PNRJFNGMV4XKBXU4MSKZVM2AEEH5GVBRBQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh): protocol: expected piece hash; context canceled; EOF Dec 30 03:51:43.127743557 (588h22m42.998826592s) [noise=true , address=193.149.164.143:4695 , piece_id=C3ZXZXKRGAI2FATBQ2PNRJFNGMV4XKBXU4MSKZVM2AEEH5GVBRBQ, seg_pos={0 0}, upload=515790]: Long tail cancellation due to optimal threshold Dec 30 03:51:43.127899784 (588h22m42.998982829s) [noise=true , address=145.79.227.33:4901 , piece_id=ODRNWAOOWKB45HREGAAKMJYNIN2I4OPRLUN5BIVIW7HWFQRTR6NA, seg_pos={0 0}, upload=515790]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1): protocol: write tcp 188.166.225.170:48420->145.79.227.33:4901: use of closed network connection; write tcp 188.166.225.170:48420->145.79.227.33:4901: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48420->145.79.227.33:4901: use of closed network connection Dec 30 03:51:43.127920484 (588h22m42.999003539s) [noise=true , address=145.79.227.33:4901 , piece_id=ODRNWAOOWKB45HREGAAKMJYNIN2I4OPRLUN5BIVIW7HWFQRTR6NA, seg_pos={0 0}, upload=515790]: Long tail cancellation due to optimal threshold Dec 30 03:51:43.128047735 (588h22m42.99913077s) [seg_pos={0 0}, upload=515790]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 03:51:43.128161222 (588h22m42.999244257s) [upload=515790]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 03:51:43.128174597 (588h22m42.999257642s) [upload=515790]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 03:51:43.415948996 (588h22m43.287032052s) [upload=515790]: Done uploading segments... Dec 30 03:51:43.415976829 (588h22m43.287059864s) [upload=515790]: Upload finished. err: <nil> Duration: 37.639334694s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [.X....X.X..........XX...X.......................X..XXX......] duration: 4.270309901s trace: https://jaeger.dev.storj.io/trace/532d591e051353a0 FAIL (2025-12-30 03:58:28 UTC) orbiter warning: context deadline exceeded Duration: 4.270336823s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [..X.X.X...XXX.X.....XXXX........XX.XX...XXXX...XXXX...X.X...] duration: 6.034835654s trace: https://jaeger.dev.storj.io/trace/41ae83c3322c189c FAIL (2025-12-30 03:58:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: PQS2F3TYHD3H2HZFBQ46HAERCLKF2ZUUCGG2B6CKXH73BJIBOUSA) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: CASP7FZEK4B4E23NLRFWJDRM5V5JFXYVPHBSH6JUVCZVVNOZPK5A) context deadline exceeded; piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: CMRGKFBHMCTR2DQHQOM7GGDWHZRTUOJY42B6ZMJDDHSKPRUEXNRA) context deadline exceeded ======LOGS====== ng segments... Dec 30 03:55:25.346377833 (588h26m25.217460868s) [upload=515850]: Got next segment. Inline: false Dec 30 03:55:25.346450542 (588h26m25.217533597s) [upload=515850]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 03:55:25.346456974 (588h26m25.217540019s) [upload=515850]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 03:55:25.594479289 (588h26m25.465562324s) [seg_pos={0 0}, upload=515850]: Begin upload segment... Dec 30 03:55:25.594533653 (588h26m25.465616698s) [seg_pos={0 0}, upload=515850]: Done begin upload segment. Dec 30 03:55:25.594575863 (588h26m25.465658908s) [noise=true , address=77.90.31.16:4736 , piece_id=QVSY52ZKVXZMJA7E655NOHZ33E7D3WXFCG6Q3Z5QZLERKZ6WWQKA, seg_pos={0 0}, upload=515850]: Uploading piece... Dec 30 03:55:25.594599598 (588h26m25.465682633s) [noise=true , address=145.79.227.2:4114 , piece_id=DSA3A75Y7SSN42DSQ27VO7PWHNMSH6H7KE7JXJ6LGEIX56QW3G4Q, seg_pos={0 0}, upload=515850]: Uploading piece... Dec 30 03:55:25.595506324 (588h26m25.466589359s) [noise=true , address=145.79.227.27:4689 , piece_id=IXSWHEI4YPUAYCD7A2ZD7FFHG65ATOIFRIQEAPIXWPNC7E5ZXHOA, seg_pos={0 0}, upload=515850]: Uploading piece... Dec 30 03:55:25.596079456 (588h26m25.467162491s) [noise=true , address=45.91.116.7:4010 , piece_id=2EBKSY2QGFBDQK6FWH6C54WRAICUYK7BI5DTZ5ZMOONC7B47FOQQ, seg_pos={0 0}, upload=515850]: Uploading piece... Dec 30 03:55:25.596164848 (588h26m25.467247883s) [noise=true , address=145.79.227.3:4151 , piece_id=VMNYNCCO46MX7FUX6RHOKZWQDY3N4NMD2HF2N3HW5TVPSKFPD7UA, seg_pos={0 0}, upload=515850]: Uploading piece... Dec 30 03:55:25.596751285 (588h26m25.46783432s) [noise=true , address=45.91.116.4:4937 , piece_id=OZXMZWCQDFLBUPA36C2X4K27LIMHAFUMIA26ZIDNQ7WK5QMMMPGA, seg_pos={0 0}, upload=515850]: Uploading piece... Dec 30 03:55:25.597389460 (588h26m25.468472495s) [noise=true , address=77.90.31.21:4878 , piece_id=CMRGKFBHMCTR2DQHQOM7GGDWHZRTUOJY42B6ZMJDDHSKPRUEXNRA, seg_pos={0 0}, upload=515850]: Uploading piece... Dec 30 03:55:25.598414532 (588h26m25.469497577s) [noise=true , address=77.90.31.19:4841 , piece_id=BMJLFDSK7WX3HGQ7RLDTEDZGB4NW5MV47XS4FSAMAIF7NFU2QPNA, seg_pos={0 0}, upload=515850]: Uploading piece... Dec 30 03:55:25.599506091 (588h26m25.470589126s) [noise=true , address=145.79.227.39:4015 , piece_id=RAKLGLN4T2JJ3WERVBK7UE6DDM6OC653EEDFXIFJAFVGOD4N7FQQ, seg_pos={0 0}, upload=515850]: Uploading piece... Dec 30 03:55:25.599578068 (588h26m25.470661103s) [noise=true , address=77.90.31.10:4518 , piece_id=NPUTORIPATTOFGDBM7FEM5NBFAF4LRGTMNUX6ANV5MR3XIITFRDA, seg_pos={0 0}, upload=515850]: Uploading piece... Dec 30 03:55:25.600318988 (588h26m25.471402023s) [noise=true , address=45.91.116.14:4189 , piece_id=PQS2F3TYHD3H2HZFBQ46HAERCLKF2ZUUCGG2B6CKXH73BJIBOUSA, seg_pos={0 0}, upload=515850]: Uploading piece... Dec 30 03:55:25.601047295 (588h26m25.47213034s) [noise=true , address=145.79.227.42:4121 , piece_id=HDT35BDENUUQQ4LSLEPV2F4ERVQU4MVJU7VL52PJAGD45ABR5XXA, seg_pos={0 0}, upload=515850]: Uploading piece... Dec 30 03:55:25.601461624 (588h26m25.472544659s) [noise=true , address=45.91.116.8:4048 , piece_id=CASP7FZEK4B4E23NLRFWJDRM5V5JFXYVPHBSH6JUVCZVVNOZPK5A, seg_pos={0 0}, upload=515850]: Uploading piece... Dec 30 03:55:25.602910203 (588h26m25.473993238s) [noise=true , address=145.79.227.34:4907 , piece_id=GPMZYJ5Q2OD6KCE4T4WZ4EMYTMF52PG453RUYDGMI766QLQTIJWA, seg_pos={0 0}, upload=515850]: Uploading piece... Dec 30 03:55:25.603636665 (588h26m25.47471971s) [noise=true , address=45.91.116.6:4975 , piece_id=KKY6ANBWKXNQHBNJBM2NITWOYDSFBOMZNXXTFL7RUJILOTNT3B5A, seg_pos={0 0}, upload=515850]: Uploading piece... Dec 30 03:55:25.604181343 (588h26m25.475264388s) [noise=true , address=145.79.227.14:4839 , piece_id=7YUQXGKUXOWUBZJQNWYSZK2APWT65AS3ZRTSVBSYXT6B3423QBYQ, seg_pos={0 0}, upload=515850]: Uploading piece... Dec 30 03:55:27.273992732 (588h26m27.145075767s) [upload=515850]: Next returned nil segment Dec 30 03:55:27.274004635 (588h26m27.14508767s) [upload=515850]: Waiting for error group managing segments... Dec 30 03:55:27.586380428 (588h26m27.457463473s) [noise=true , address=145.79.227.27:4689 , piece_id=IXSWHEI4YPUAYCD7A2ZD7FFHG65ATOIFRIQEAPIXWPNC7E5ZXHOA, seg_pos={0 0}, upload=515850]: Done uploading piece. err: <nil> Dec 30 03:55:27.586454268 (588h26m27.457537314s) [noise=true , address=145.79.227.42:4121 , piece_id=HDT35BDENUUQQ4LSLEPV2F4ERVQU4MVJU7VL52PJAGD45ABR5XXA, seg_pos={0 0}, upload=515850]: Done uploading piece. err: <nil> Dec 30 03:55:27.587462528 (588h26m27.458545573s) [noise=true , address=145.79.227.14:4839 , piece_id=7YUQXGKUXOWUBZJQNWYSZK2APWT65AS3ZRTSVBSYXT6B3423QBYQ, seg_pos={0 0}, upload=515850]: Done uploading piece. err: <nil> Dec 30 03:55:27.589673147 (588h26m27.460756182s) [noise=true , address=145.79.227.39:4015 , piece_id=RAKLGLN4T2JJ3WERVBK7UE6DDM6OC653EEDFXIFJAFVGOD4N7FQQ, seg_pos={0 0}, upload=515850]: Done uploading piece. err: <nil> Dec 30 03:55:27.590415280 (588h26m27.461498315s) [noise=true , address=145.79.227.3:4151 , piece_id=VMNYNCCO46MX7FUX6RHOKZWQDY3N4NMD2HF2N3HW5TVPSKFPD7UA, seg_pos={0 0}, upload=515850]: Done uploading piece. err: <nil> Dec 30 03:55:27.874380536 (588h26m27.745463581s) [noise=true , address=45.91.116.8:4048 , piece_id=CASP7FZEK4B4E23NLRFWJDRM5V5JFXYVPHBSH6JUVCZVVNOZPK5A, seg_pos={0 0}, upload=515850]: Done uploading piece. err: <nil> Dec 30 03:55:27.889876592 (588h26m27.760959637s) [noise=true , address=77.90.31.21:4878 , piece_id=CMRGKFBHMCTR2DQHQOM7GGDWHZRTUOJY42B6ZMJDDHSKPRUEXNRA, seg_pos={0 0}, upload=515850]: Done uploading piece. err: <nil> Dec 30 03:55:27.911230463 (588h26m27.782313508s) [noise=true , address=145.79.227.34:4907 , piece_id=GPMZYJ5Q2OD6KCE4T4WZ4EMYTMF52PG453RUYDGMI766QLQTIJWA, seg_pos={0 0}, upload=515850]: Done uploading piece. err: <nil> Dec 30 03:55:27.925490335 (588h26m27.79657338s) [noise=true , address=45.91.116.7:4010 , piece_id=2EBKSY2QGFBDQK6FWH6C54WRAICUYK7BI5DTZ5ZMOONC7B47FOQQ, seg_pos={0 0}, upload=515850]: Done uploading piece. err: <nil> Dec 30 03:55:27.955525315 (588h26m27.82660835s) [noise=true , address=45.91.116.4:4937 , piece_id=OZXMZWCQDFLBUPA36C2X4K27LIMHAFUMIA26ZIDNQ7WK5QMMMPGA, seg_pos={0 0}, upload=515850]: Done uploading piece. err: <nil> Dec 30 03:55:27.967144099 (588h26m27.838227144s) [noise=true , address=45.91.116.14:4189 , piece_id=PQS2F3TYHD3H2HZFBQ46HAERCLKF2ZUUCGG2B6CKXH73BJIBOUSA, seg_pos={0 0}, upload=515850]: Done uploading piece. err: <nil> Dec 30 03:55:27.971694734 (588h26m27.842777779s) [noise=true , address=45.91.116.6:4975 , piece_id=KKY6ANBWKXNQHBNJBM2NITWOYDSFBOMZNXXTFL7RUJILOTNT3B5A, seg_pos={0 0}, upload=515850]: Done uploading piece. err: <nil> Dec 30 03:55:27.972455824 (588h26m27.843538859s) [noise=true , address=77.90.31.19:4841 , piece_id=BMJLFDSK7WX3HGQ7RLDTEDZGB4NW5MV47XS4FSAMAIF7NFU2QPNA, seg_pos={0 0}, upload=515850]: Done uploading piece. err: <nil> Dec 30 03:55:27.972468477 (588h26m27.843551522s) [seg_pos={0 0}, upload=515850]: Segment reached optimal threshold of 13 pieces. Dec 30 03:55:27.972639593 (588h26m27.843722628s) [noise=true , address=145.79.227.2:4114 , piece_id=DSA3A75Y7SSN42DSQ27VO7PWHNMSH6H7KE7JXJ6LGEIX56QW3G4Q, seg_pos={0 0}, upload=515850]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN): protocol: write tcp 188.166.225.170:44928->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:44928->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44928->145.79.227.2:4114: use of closed network connection Dec 30 03:55:27.972645264 (588h26m27.843728299s) [noise=true , address=145.79.227.2:4114 , piece_id=DSA3A75Y7SSN42DSQ27VO7PWHNMSH6H7KE7JXJ6LGEIX56QW3G4Q, seg_pos={0 0}, upload=515850]: Long tail cancellation due to optimal threshold Dec 30 03:55:27.972957088 (588h26m27.844040133s) [noise=true , address=77.90.31.16:4736 , piece_id=QVSY52ZKVXZMJA7E655NOHZ33E7D3WXFCG6Q3Z5QZLERKZ6WWQKA, seg_pos={0 0}, upload=515850]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: write tcp 188.166.225.170:53750->77.90.31.16:4736: use of closed network connection; write tcp 188.166.225.170:53750->77.90.31.16:4736: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53750->77.90.31.16:4736: use of closed network connection Dec 30 03:55:27.972971575 (588h26m27.84405461s) [noise=true , address=77.90.31.16:4736 , piece_id=QVSY52ZKVXZMJA7E655NOHZ33E7D3WXFCG6Q3Z5QZLERKZ6WWQKA, seg_pos={0 0}, upload=515850]: Long tail cancellation due to optimal threshold Dec 30 03:55:27.973168019 (588h26m27.844251054s) [noise=true , address=77.90.31.10:4518 , piece_id=NPUTORIPATTOFGDBM7FEM5NBFAF4LRGTMNUX6ANV5MR3XIITFRDA, seg_pos={0 0}, upload=515850]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 30 03:55:27.973173930 (588h26m27.844256965s) [noise=true , address=77.90.31.10:4518 , piece_id=NPUTORIPATTOFGDBM7FEM5NBFAF4LRGTMNUX6ANV5MR3XIITFRDA, seg_pos={0 0}, upload=515850]: Long tail cancellation due to optimal threshold Dec 30 03:55:27.973191504 (588h26m27.844274539s) [seg_pos={0 0}, upload=515850]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 03:55:27.973291655 (588h26m27.84437469s) [upload=515850]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 03:55:27.973301393 (588h26m27.844384428s) [upload=515850]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 03:55:28.268468558 (588h26m28.139551593s) [upload=515850]: Done uploading segments... Dec 30 03:55:28.268490931 (588h26m28.139573976s) [upload=515850]: Upload finished. err: <nil> Duration: 6.034855934s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 48.886278095s trace: https://jaeger.dev.storj.io/trace/1d81d68a7c17f75d FAIL (2025-12-30 03:57:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 15 , piece_id=6FWCMOTNEZLDIH2F5VRBC3H44WVYAKWTUIL364QUAIC7DGKURRSQ, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.829621721 (588h22m39.700704767s) [noise=true , address=193.149.164.143:4695 , piece_id=C3ZXZXKRGAI2FATBQ2PNRJFNGMV4XKBXU4MSKZVM2AEEH5GVBRBQ, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.829918227 (588h22m39.701001282s) [noise=true , address=145.79.227.17:4998 , piece_id=IMFG6HRRWRXUX3F76A5RTU54HXCFSTPTWH4YJQPHGIOMPALNS5AA, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.830292459 (588h22m39.701375504s) [noise=true , address=45.91.116.12:4249 , piece_id=YLNIBVQGVGZ5CYEZSGLORL3D2ULBBCXN23SSNXUU7LQ67E3E4G7A, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.830455549 (588h22m39.701538594s) [noise=true , address=145.79.227.43:4178 , piece_id=7DIUURHOKK53VG6RMELJUV3D3AWUM23SHXNNVDRFTIURMXETX6QQ, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.830740912 (588h22m39.701823948s) [noise=true , address=77.90.31.16:4759 , piece_id=W3RFMAKOXSYIDCA27H7CKEXSIKCMRUO72NITXH5WDM4FFK6XPDAQ, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.830892231 (588h22m39.701975266s) [noise=true , address=145.79.227.45:4250 , piece_id=MDFRH2T6ZDAOE25DJTGHA6YSXS6CRL5OBES5T2FDTGY6EOV7LINQ, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.831333561 (588h22m39.702416606s) [noise=true , address=145.79.227.40:4053 , piece_id=AKAAPJDTKY3BB2OB5EFW4IYMYPOAPL4VDS5VEHJF3U3TA7JHZ4FQ, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.831388847 (588h22m39.702471882s) [noise=true , address=145.79.227.34:4916 , piece_id=DMOYHOUOVSMTMPZBY7MV5LYQ5JQDV4MKRJHBH7FYV2TBL37AFYSA, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.831789770 (588h22m39.702872815s) [noise=true , address=64.49.13.26:4946 , piece_id=X7CGAXO4356EG4I52QB5LQEM33UD2FJFS56YFEH7FNHAG2EBEZFA, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.832180875 (588h22m39.70326391s) [noise=true , address=77.90.31.19:4871 , piece_id=HPPBTDGVLD62WLHSPO36VFYHP5YFORYCBT22PK2CEYRJJM4UQLYA, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.832576678 (588h22m39.703659723s) [noise=true , address=145.79.227.33:4901 , piece_id=ODRNWAOOWKB45HREGAAKMJYNIN2I4OPRLUN5BIVIW7HWFQRTR6NA, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.834204918 (588h22m39.705287963s) [noise=true , address=45.91.116.13:4164 , piece_id=QW7FBIO24JGOXJEGO7YHYFMMTKHAKF64UEGNDXX2KKQLSJ4MH3PA, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.834999141 (588h22m39.706082186s) [noise=true , address=77.90.31.11:4584 , piece_id=WKRB35UKP7L6Z6G5WSJKMFBOF3RPWJYVZJPFBP5G4UT4RQVQP6MA, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.835718220 (588h22m39.706801265s) [noise=true , address=145.79.227.49:4361 , piece_id=ARLKW5A6Y3DVQ25E2RDL5KRFCYBNVIW7WBDL7ZO2WOQDTF5RCRKQ, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:41.118730125 (588h22m40.98981316s) [upload=515790]: Next returned nil segment Dec 30 03:51:41.118750513 (588h22m40.989833558s) [upload=515790]: Waiting for error group managing segments... Dec 30 03:51:41.505594353 (588h22m41.376677388s) [noise=true , address=64.49.13.26:4946 , piece_id=X7CGAXO4356EG4I52QB5LQEM33UD2FJFS56YFEH7FNHAG2EBEZFA, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:41.711421644 (588h22m41.582504689s) [noise=true , address=145.79.227.17:4998 , piece_id=IMFG6HRRWRXUX3F76A5RTU54HXCFSTPTWH4YJQPHGIOMPALNS5AA, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:41.718129295 (588h22m41.58921233s) [noise=true , address=145.79.227.6:4285 , piece_id=WNJGANKULJAMPLIT3KTHRSSAYTUSIIUCW5ZVKAFNURMGNZ3N2P3A, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:41.729197852 (588h22m41.600280887s) [noise=true , address=145.79.227.14:4856 , piece_id=CBZ6JFWGDMBMR7XJBXIETWDBEEXV43HJWJFKH5FY5BF7ANACFDOQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:41.867535754 (588h22m41.738618799s) [noise=true , address=145.79.227.43:4178 , piece_id=7DIUURHOKK53VG6RMELJUV3D3AWUM23SHXNNVDRFTIURMXETX6QQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:41.894207901 (588h22m41.765290946s) [noise=true , address=77.90.31.12:4615 , piece_id=6FWCMOTNEZLDIH2F5VRBC3H44WVYAKWTUIL364QUAIC7DGKURRSQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:41.928246715 (588h22m41.79932975s) [noise=true , address=145.79.227.45:4250 , piece_id=MDFRH2T6ZDAOE25DJTGHA6YSXS6CRL5OBES5T2FDTGY6EOV7LINQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:41.940833472 (588h22m41.811916517s) [noise=true , address=45.91.116.13:4164 , piece_id=QW7FBIO24JGOXJEGO7YHYFMMTKHAKF64UEGNDXX2KKQLSJ4MH3PA, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:42.318460461 (588h22m42.189543506s) [noise=true , address=77.90.31.16:4759 , piece_id=W3RFMAKOXSYIDCA27H7CKEXSIKCMRUO72NITXH5WDM4FFK6XPDAQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:42.367315176 (588h22m42.238398221s) [noise=true , address=77.90.31.19:4871 , piece_id=HPPBTDGVLD62WLHSPO36VFYHP5YFORYCBT22PK2CEYRJJM4UQLYA, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:42.536538860 (588h22m42.407621905s) [noise=true , address=145.79.227.40:4053 , piece_id=AKAAPJDTKY3BB2OB5EFW4IYMYPOAPL4VDS5VEHJF3U3TA7JHZ4FQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:42.637143172 (588h22m42.508226217s) [noise=true , address=145.79.227.44:4200 , piece_id=2KRTBC4EQE7OCNJJ7TPM72KJFYBBXFHJUP2FRENF6KL3CQNAAUQQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:42.715818140 (588h22m42.586901185s) [noise=true , address=45.91.116.12:4249 , piece_id=YLNIBVQGVGZ5CYEZSGLORL3D2ULBBCXN23SSNXUU7LQ67E3E4G7A, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:43.127299661 (588h22m42.998382696s) [noise=true , address=145.79.227.34:4916 , piece_id=DMOYHOUOVSMTMPZBY7MV5LYQ5JQDV4MKRJHBH7FYV2TBL37AFYSA, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:43.127318687 (588h22m42.998401722s) [seg_pos={0 0}, upload=515790]: Segment reached optimal threshold of 14 pieces. Dec 30 03:51:43.127508449 (588h22m42.998591484s) [noise=true , address=145.79.227.49:4361 , piece_id=ARLKW5A6Y3DVQ25E2RDL5KRFCYBNVIW7WBDL7ZO2WOQDTF5RCRKQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k): protocol: expected piece hash; context canceled; EOF Dec 30 03:51:43.127514470 (588h22m42.998597505s) [noise=true , address=145.79.227.49:4361 , piece_id=ARLKW5A6Y3DVQ25E2RDL5KRFCYBNVIW7WBDL7ZO2WOQDTF5RCRKQ, seg_pos={0 0}, upload=515790]: Long tail cancellation due to optimal threshold Dec 30 03:51:43.127619871 (588h22m42.998702906s) [noise=true , address=77.90.31.11:4584 , piece_id=WKRB35UKP7L6Z6G5WSJKMFBOF3RPWJYVZJPFBP5G4UT4RQVQP6MA, seg_pos={0 0}, upload=515790]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iybcvQgVfqHcCSNFAEVpKZJyciSHbBfeAiuk3uNE2LGq4aLfJ): protocol: write tcp 188.166.225.170:43330->77.90.31.11:4584: use of closed network connection; write tcp 188.166.225.170:43330->77.90.31.11:4584: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43330->77.90.31.11:4584: use of closed network connection Dec 30 03:51:43.127624550 (588h22m42.998707585s) [noise=true , address=77.90.31.11:4584 , piece_id=WKRB35UKP7L6Z6G5WSJKMFBOF3RPWJYVZJPFBP5G4UT4RQVQP6MA, seg_pos={0 0}, upload=515790]: Long tail cancellation due to optimal threshold Dec 30 03:51:43.127738065 (588h22m42.998821111s) [noise=true , address=193.149.164.143:4695 , piece_id=C3ZXZXKRGAI2FATBQ2PNRJFNGMV4XKBXU4MSKZVM2AEEH5GVBRBQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh): protocol: expected piece hash; context canceled; EOF Dec 30 03:51:43.127743557 (588h22m42.998826592s) [noise=true , address=193.149.164.143:4695 , piece_id=C3ZXZXKRGAI2FATBQ2PNRJFNGMV4XKBXU4MSKZVM2AEEH5GVBRBQ, seg_pos={0 0}, upload=515790]: Long tail cancellation due to optimal threshold Dec 30 03:51:43.127899784 (588h22m42.998982829s) [noise=true , address=145.79.227.33:4901 , piece_id=ODRNWAOOWKB45HREGAAKMJYNIN2I4OPRLUN5BIVIW7HWFQRTR6NA, seg_pos={0 0}, upload=515790]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1): protocol: write tcp 188.166.225.170:48420->145.79.227.33:4901: use of closed network connection; write tcp 188.166.225.170:48420->145.79.227.33:4901: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48420->145.79.227.33:4901: use of closed network connection Dec 30 03:51:43.127920484 (588h22m42.999003539s) [noise=true , address=145.79.227.33:4901 , piece_id=ODRNWAOOWKB45HREGAAKMJYNIN2I4OPRLUN5BIVIW7HWFQRTR6NA, seg_pos={0 0}, upload=515790]: Long tail cancellation due to optimal threshold Dec 30 03:51:43.128047735 (588h22m42.99913077s) [seg_pos={0 0}, upload=515790]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 03:51:43.128161222 (588h22m42.999244257s) [upload=515790]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 03:51:43.128174597 (588h22m42.999257642s) [upload=515790]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 03:51:43.415948996 (588h22m43.287032052s) [upload=515790]: Done uploading segments... Dec 30 03:51:43.415976829 (588h22m43.287059864s) [upload=515790]: Upload finished. err: <nil> Duration: 48.8862967s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [..X....X.X..........XX...X.......................X..XXX.....] duration: 3.405216023s trace: https://jaeger.dev.storj.io/trace/532d591e051353a0 FAIL (2025-12-30 03:57:48 UTC) orbiter warning: context deadline exceeded Duration: 3.40523522s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [X..X.X.X...XXX.X.....XXXX........XX.XX...XXXX...XXXX...X.X..] duration: 6.048958837s trace: https://jaeger.dev.storj.io/trace/41ae83c3322c189c FAIL (2025-12-30 03:57:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: PQS2F3TYHD3H2HZFBQ46HAERCLKF2ZUUCGG2B6CKXH73BJIBOUSA) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: CASP7FZEK4B4E23NLRFWJDRM5V5JFXYVPHBSH6JUVCZVVNOZPK5A) context deadline exceeded; piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: CMRGKFBHMCTR2DQHQOM7GGDWHZRTUOJY42B6ZMJDDHSKPRUEXNRA) context deadline exceeded ======LOGS====== ng segments... Dec 30 03:55:25.346377833 (588h26m25.217460868s) [upload=515850]: Got next segment. Inline: false Dec 30 03:55:25.346450542 (588h26m25.217533597s) [upload=515850]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 03:55:25.346456974 (588h26m25.217540019s) [upload=515850]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 03:55:25.594479289 (588h26m25.465562324s) [seg_pos={0 0}, upload=515850]: Begin upload segment... Dec 30 03:55:25.594533653 (588h26m25.465616698s) [seg_pos={0 0}, upload=515850]: Done begin upload segment. Dec 30 03:55:25.594575863 (588h26m25.465658908s) [noise=true , address=77.90.31.16:4736 , piece_id=QVSY52ZKVXZMJA7E655NOHZ33E7D3WXFCG6Q3Z5QZLERKZ6WWQKA, seg_pos={0 0}, upload=515850]: Uploading piece... Dec 30 03:55:25.594599598 (588h26m25.465682633s) [noise=true , address=145.79.227.2:4114 , piece_id=DSA3A75Y7SSN42DSQ27VO7PWHNMSH6H7KE7JXJ6LGEIX56QW3G4Q, seg_pos={0 0}, upload=515850]: Uploading piece... Dec 30 03:55:25.595506324 (588h26m25.466589359s) [noise=true , address=145.79.227.27:4689 , piece_id=IXSWHEI4YPUAYCD7A2ZD7FFHG65ATOIFRIQEAPIXWPNC7E5ZXHOA, seg_pos={0 0}, upload=515850]: Uploading piece... Dec 30 03:55:25.596079456 (588h26m25.467162491s) [noise=true , address=45.91.116.7:4010 , piece_id=2EBKSY2QGFBDQK6FWH6C54WRAICUYK7BI5DTZ5ZMOONC7B47FOQQ, seg_pos={0 0}, upload=515850]: Uploading piece... Dec 30 03:55:25.596164848 (588h26m25.467247883s) [noise=true , address=145.79.227.3:4151 , piece_id=VMNYNCCO46MX7FUX6RHOKZWQDY3N4NMD2HF2N3HW5TVPSKFPD7UA, seg_pos={0 0}, upload=515850]: Uploading piece... Dec 30 03:55:25.596751285 (588h26m25.46783432s) [noise=true , address=45.91.116.4:4937 , piece_id=OZXMZWCQDFLBUPA36C2X4K27LIMHAFUMIA26ZIDNQ7WK5QMMMPGA, seg_pos={0 0}, upload=515850]: Uploading piece... Dec 30 03:55:25.597389460 (588h26m25.468472495s) [noise=true , address=77.90.31.21:4878 , piece_id=CMRGKFBHMCTR2DQHQOM7GGDWHZRTUOJY42B6ZMJDDHSKPRUEXNRA, seg_pos={0 0}, upload=515850]: Uploading piece... Dec 30 03:55:25.598414532 (588h26m25.469497577s) [noise=true , address=77.90.31.19:4841 , piece_id=BMJLFDSK7WX3HGQ7RLDTEDZGB4NW5MV47XS4FSAMAIF7NFU2QPNA, seg_pos={0 0}, upload=515850]: Uploading piece... Dec 30 03:55:25.599506091 (588h26m25.470589126s) [noise=true , address=145.79.227.39:4015 , piece_id=RAKLGLN4T2JJ3WERVBK7UE6DDM6OC653EEDFXIFJAFVGOD4N7FQQ, seg_pos={0 0}, upload=515850]: Uploading piece... Dec 30 03:55:25.599578068 (588h26m25.470661103s) [noise=true , address=77.90.31.10:4518 , piece_id=NPUTORIPATTOFGDBM7FEM5NBFAF4LRGTMNUX6ANV5MR3XIITFRDA, seg_pos={0 0}, upload=515850]: Uploading piece... Dec 30 03:55:25.600318988 (588h26m25.471402023s) [noise=true , address=45.91.116.14:4189 , piece_id=PQS2F3TYHD3H2HZFBQ46HAERCLKF2ZUUCGG2B6CKXH73BJIBOUSA, seg_pos={0 0}, upload=515850]: Uploading piece... Dec 30 03:55:25.601047295 (588h26m25.47213034s) [noise=true , address=145.79.227.42:4121 , piece_id=HDT35BDENUUQQ4LSLEPV2F4ERVQU4MVJU7VL52PJAGD45ABR5XXA, seg_pos={0 0}, upload=515850]: Uploading piece... Dec 30 03:55:25.601461624 (588h26m25.472544659s) [noise=true , address=45.91.116.8:4048 , piece_id=CASP7FZEK4B4E23NLRFWJDRM5V5JFXYVPHBSH6JUVCZVVNOZPK5A, seg_pos={0 0}, upload=515850]: Uploading piece... Dec 30 03:55:25.602910203 (588h26m25.473993238s) [noise=true , address=145.79.227.34:4907 , piece_id=GPMZYJ5Q2OD6KCE4T4WZ4EMYTMF52PG453RUYDGMI766QLQTIJWA, seg_pos={0 0}, upload=515850]: Uploading piece... Dec 30 03:55:25.603636665 (588h26m25.47471971s) [noise=true , address=45.91.116.6:4975 , piece_id=KKY6ANBWKXNQHBNJBM2NITWOYDSFBOMZNXXTFL7RUJILOTNT3B5A, seg_pos={0 0}, upload=515850]: Uploading piece... Dec 30 03:55:25.604181343 (588h26m25.475264388s) [noise=true , address=145.79.227.14:4839 , piece_id=7YUQXGKUXOWUBZJQNWYSZK2APWT65AS3ZRTSVBSYXT6B3423QBYQ, seg_pos={0 0}, upload=515850]: Uploading piece... Dec 30 03:55:27.273992732 (588h26m27.145075767s) [upload=515850]: Next returned nil segment Dec 30 03:55:27.274004635 (588h26m27.14508767s) [upload=515850]: Waiting for error group managing segments... Dec 30 03:55:27.586380428 (588h26m27.457463473s) [noise=true , address=145.79.227.27:4689 , piece_id=IXSWHEI4YPUAYCD7A2ZD7FFHG65ATOIFRIQEAPIXWPNC7E5ZXHOA, seg_pos={0 0}, upload=515850]: Done uploading piece. err: <nil> Dec 30 03:55:27.586454268 (588h26m27.457537314s) [noise=true , address=145.79.227.42:4121 , piece_id=HDT35BDENUUQQ4LSLEPV2F4ERVQU4MVJU7VL52PJAGD45ABR5XXA, seg_pos={0 0}, upload=515850]: Done uploading piece. err: <nil> Dec 30 03:55:27.587462528 (588h26m27.458545573s) [noise=true , address=145.79.227.14:4839 , piece_id=7YUQXGKUXOWUBZJQNWYSZK2APWT65AS3ZRTSVBSYXT6B3423QBYQ, seg_pos={0 0}, upload=515850]: Done uploading piece. err: <nil> Dec 30 03:55:27.589673147 (588h26m27.460756182s) [noise=true , address=145.79.227.39:4015 , piece_id=RAKLGLN4T2JJ3WERVBK7UE6DDM6OC653EEDFXIFJAFVGOD4N7FQQ, seg_pos={0 0}, upload=515850]: Done uploading piece. err: <nil> Dec 30 03:55:27.590415280 (588h26m27.461498315s) [noise=true , address=145.79.227.3:4151 , piece_id=VMNYNCCO46MX7FUX6RHOKZWQDY3N4NMD2HF2N3HW5TVPSKFPD7UA, seg_pos={0 0}, upload=515850]: Done uploading piece. err: <nil> Dec 30 03:55:27.874380536 (588h26m27.745463581s) [noise=true , address=45.91.116.8:4048 , piece_id=CASP7FZEK4B4E23NLRFWJDRM5V5JFXYVPHBSH6JUVCZVVNOZPK5A, seg_pos={0 0}, upload=515850]: Done uploading piece. err: <nil> Dec 30 03:55:27.889876592 (588h26m27.760959637s) [noise=true , address=77.90.31.21:4878 , piece_id=CMRGKFBHMCTR2DQHQOM7GGDWHZRTUOJY42B6ZMJDDHSKPRUEXNRA, seg_pos={0 0}, upload=515850]: Done uploading piece. err: <nil> Dec 30 03:55:27.911230463 (588h26m27.782313508s) [noise=true , address=145.79.227.34:4907 , piece_id=GPMZYJ5Q2OD6KCE4T4WZ4EMYTMF52PG453RUYDGMI766QLQTIJWA, seg_pos={0 0}, upload=515850]: Done uploading piece. err: <nil> Dec 30 03:55:27.925490335 (588h26m27.79657338s) [noise=true , address=45.91.116.7:4010 , piece_id=2EBKSY2QGFBDQK6FWH6C54WRAICUYK7BI5DTZ5ZMOONC7B47FOQQ, seg_pos={0 0}, upload=515850]: Done uploading piece. err: <nil> Dec 30 03:55:27.955525315 (588h26m27.82660835s) [noise=true , address=45.91.116.4:4937 , piece_id=OZXMZWCQDFLBUPA36C2X4K27LIMHAFUMIA26ZIDNQ7WK5QMMMPGA, seg_pos={0 0}, upload=515850]: Done uploading piece. err: <nil> Dec 30 03:55:27.967144099 (588h26m27.838227144s) [noise=true , address=45.91.116.14:4189 , piece_id=PQS2F3TYHD3H2HZFBQ46HAERCLKF2ZUUCGG2B6CKXH73BJIBOUSA, seg_pos={0 0}, upload=515850]: Done uploading piece. err: <nil> Dec 30 03:55:27.971694734 (588h26m27.842777779s) [noise=true , address=45.91.116.6:4975 , piece_id=KKY6ANBWKXNQHBNJBM2NITWOYDSFBOMZNXXTFL7RUJILOTNT3B5A, seg_pos={0 0}, upload=515850]: Done uploading piece. err: <nil> Dec 30 03:55:27.972455824 (588h26m27.843538859s) [noise=true , address=77.90.31.19:4841 , piece_id=BMJLFDSK7WX3HGQ7RLDTEDZGB4NW5MV47XS4FSAMAIF7NFU2QPNA, seg_pos={0 0}, upload=515850]: Done uploading piece. err: <nil> Dec 30 03:55:27.972468477 (588h26m27.843551522s) [seg_pos={0 0}, upload=515850]: Segment reached optimal threshold of 13 pieces. Dec 30 03:55:27.972639593 (588h26m27.843722628s) [noise=true , address=145.79.227.2:4114 , piece_id=DSA3A75Y7SSN42DSQ27VO7PWHNMSH6H7KE7JXJ6LGEIX56QW3G4Q, seg_pos={0 0}, upload=515850]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN): protocol: write tcp 188.166.225.170:44928->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:44928->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44928->145.79.227.2:4114: use of closed network connection Dec 30 03:55:27.972645264 (588h26m27.843728299s) [noise=true , address=145.79.227.2:4114 , piece_id=DSA3A75Y7SSN42DSQ27VO7PWHNMSH6H7KE7JXJ6LGEIX56QW3G4Q, seg_pos={0 0}, upload=515850]: Long tail cancellation due to optimal threshold Dec 30 03:55:27.972957088 (588h26m27.844040133s) [noise=true , address=77.90.31.16:4736 , piece_id=QVSY52ZKVXZMJA7E655NOHZ33E7D3WXFCG6Q3Z5QZLERKZ6WWQKA, seg_pos={0 0}, upload=515850]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: write tcp 188.166.225.170:53750->77.90.31.16:4736: use of closed network connection; write tcp 188.166.225.170:53750->77.90.31.16:4736: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53750->77.90.31.16:4736: use of closed network connection Dec 30 03:55:27.972971575 (588h26m27.84405461s) [noise=true , address=77.90.31.16:4736 , piece_id=QVSY52ZKVXZMJA7E655NOHZ33E7D3WXFCG6Q3Z5QZLERKZ6WWQKA, seg_pos={0 0}, upload=515850]: Long tail cancellation due to optimal threshold Dec 30 03:55:27.973168019 (588h26m27.844251054s) [noise=true , address=77.90.31.10:4518 , piece_id=NPUTORIPATTOFGDBM7FEM5NBFAF4LRGTMNUX6ANV5MR3XIITFRDA, seg_pos={0 0}, upload=515850]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 30 03:55:27.973173930 (588h26m27.844256965s) [noise=true , address=77.90.31.10:4518 , piece_id=NPUTORIPATTOFGDBM7FEM5NBFAF4LRGTMNUX6ANV5MR3XIITFRDA, seg_pos={0 0}, upload=515850]: Long tail cancellation due to optimal threshold Dec 30 03:55:27.973191504 (588h26m27.844274539s) [seg_pos={0 0}, upload=515850]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 03:55:27.973291655 (588h26m27.84437469s) [upload=515850]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 03:55:27.973301393 (588h26m27.844384428s) [upload=515850]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 03:55:28.268468558 (588h26m28.139551593s) [upload=515850]: Done uploading segments... Dec 30 03:55:28.268490931 (588h26m28.139573976s) [upload=515850]: Upload finished. err: <nil> Duration: 6.048985699s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [...X....X.X..........XX...X.......................X..XXX....] duration: 4.42465463s trace: https://jaeger.dev.storj.io/trace/532d591e051353a0 FAIL (2025-12-30 03:56:55 UTC) orbiter warning: context deadline exceeded Duration: 4.424682884s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [.X..X.X.X...XXX.X.....XXXX........XX.XX...XXXX...XXXX...X.X.] duration: 5.754599154s trace: https://jaeger.dev.storj.io/trace/41ae83c3322c189c FAIL (2025-12-30 03:56:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: PQS2F3TYHD3H2HZFBQ46HAERCLKF2ZUUCGG2B6CKXH73BJIBOUSA) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: CASP7FZEK4B4E23NLRFWJDRM5V5JFXYVPHBSH6JUVCZVVNOZPK5A) context deadline exceeded; piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: CMRGKFBHMCTR2DQHQOM7GGDWHZRTUOJY42B6ZMJDDHSKPRUEXNRA) context deadline exceeded ======LOGS====== ng segments... Dec 30 03:55:25.346377833 (588h26m25.217460868s) [upload=515850]: Got next segment. Inline: false Dec 30 03:55:25.346450542 (588h26m25.217533597s) [upload=515850]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 03:55:25.346456974 (588h26m25.217540019s) [upload=515850]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 03:55:25.594479289 (588h26m25.465562324s) [seg_pos={0 0}, upload=515850]: Begin upload segment... Dec 30 03:55:25.594533653 (588h26m25.465616698s) [seg_pos={0 0}, upload=515850]: Done begin upload segment. Dec 30 03:55:25.594575863 (588h26m25.465658908s) [noise=true , address=77.90.31.16:4736 , piece_id=QVSY52ZKVXZMJA7E655NOHZ33E7D3WXFCG6Q3Z5QZLERKZ6WWQKA, seg_pos={0 0}, upload=515850]: Uploading piece... Dec 30 03:55:25.594599598 (588h26m25.465682633s) [noise=true , address=145.79.227.2:4114 , piece_id=DSA3A75Y7SSN42DSQ27VO7PWHNMSH6H7KE7JXJ6LGEIX56QW3G4Q, seg_pos={0 0}, upload=515850]: Uploading piece... Dec 30 03:55:25.595506324 (588h26m25.466589359s) [noise=true , address=145.79.227.27:4689 , piece_id=IXSWHEI4YPUAYCD7A2ZD7FFHG65ATOIFRIQEAPIXWPNC7E5ZXHOA, seg_pos={0 0}, upload=515850]: Uploading piece... Dec 30 03:55:25.596079456 (588h26m25.467162491s) [noise=true , address=45.91.116.7:4010 , piece_id=2EBKSY2QGFBDQK6FWH6C54WRAICUYK7BI5DTZ5ZMOONC7B47FOQQ, seg_pos={0 0}, upload=515850]: Uploading piece... Dec 30 03:55:25.596164848 (588h26m25.467247883s) [noise=true , address=145.79.227.3:4151 , piece_id=VMNYNCCO46MX7FUX6RHOKZWQDY3N4NMD2HF2N3HW5TVPSKFPD7UA, seg_pos={0 0}, upload=515850]: Uploading piece... Dec 30 03:55:25.596751285 (588h26m25.46783432s) [noise=true , address=45.91.116.4:4937 , piece_id=OZXMZWCQDFLBUPA36C2X4K27LIMHAFUMIA26ZIDNQ7WK5QMMMPGA, seg_pos={0 0}, upload=515850]: Uploading piece... Dec 30 03:55:25.597389460 (588h26m25.468472495s) [noise=true , address=77.90.31.21:4878 , piece_id=CMRGKFBHMCTR2DQHQOM7GGDWHZRTUOJY42B6ZMJDDHSKPRUEXNRA, seg_pos={0 0}, upload=515850]: Uploading piece... Dec 30 03:55:25.598414532 (588h26m25.469497577s) [noise=true , address=77.90.31.19:4841 , piece_id=BMJLFDSK7WX3HGQ7RLDTEDZGB4NW5MV47XS4FSAMAIF7NFU2QPNA, seg_pos={0 0}, upload=515850]: Uploading piece... Dec 30 03:55:25.599506091 (588h26m25.470589126s) [noise=true , address=145.79.227.39:4015 , piece_id=RAKLGLN4T2JJ3WERVBK7UE6DDM6OC653EEDFXIFJAFVGOD4N7FQQ, seg_pos={0 0}, upload=515850]: Uploading piece... Dec 30 03:55:25.599578068 (588h26m25.470661103s) [noise=true , address=77.90.31.10:4518 , piece_id=NPUTORIPATTOFGDBM7FEM5NBFAF4LRGTMNUX6ANV5MR3XIITFRDA, seg_pos={0 0}, upload=515850]: Uploading piece... Dec 30 03:55:25.600318988 (588h26m25.471402023s) [noise=true , address=45.91.116.14:4189 , piece_id=PQS2F3TYHD3H2HZFBQ46HAERCLKF2ZUUCGG2B6CKXH73BJIBOUSA, seg_pos={0 0}, upload=515850]: Uploading piece... Dec 30 03:55:25.601047295 (588h26m25.47213034s) [noise=true , address=145.79.227.42:4121 , piece_id=HDT35BDENUUQQ4LSLEPV2F4ERVQU4MVJU7VL52PJAGD45ABR5XXA, seg_pos={0 0}, upload=515850]: Uploading piece... Dec 30 03:55:25.601461624 (588h26m25.472544659s) [noise=true , address=45.91.116.8:4048 , piece_id=CASP7FZEK4B4E23NLRFWJDRM5V5JFXYVPHBSH6JUVCZVVNOZPK5A, seg_pos={0 0}, upload=515850]: Uploading piece... Dec 30 03:55:25.602910203 (588h26m25.473993238s) [noise=true , address=145.79.227.34:4907 , piece_id=GPMZYJ5Q2OD6KCE4T4WZ4EMYTMF52PG453RUYDGMI766QLQTIJWA, seg_pos={0 0}, upload=515850]: Uploading piece... Dec 30 03:55:25.603636665 (588h26m25.47471971s) [noise=true , address=45.91.116.6:4975 , piece_id=KKY6ANBWKXNQHBNJBM2NITWOYDSFBOMZNXXTFL7RUJILOTNT3B5A, seg_pos={0 0}, upload=515850]: Uploading piece... Dec 30 03:55:25.604181343 (588h26m25.475264388s) [noise=true , address=145.79.227.14:4839 , piece_id=7YUQXGKUXOWUBZJQNWYSZK2APWT65AS3ZRTSVBSYXT6B3423QBYQ, seg_pos={0 0}, upload=515850]: Uploading piece... Dec 30 03:55:27.273992732 (588h26m27.145075767s) [upload=515850]: Next returned nil segment Dec 30 03:55:27.274004635 (588h26m27.14508767s) [upload=515850]: Waiting for error group managing segments... Dec 30 03:55:27.586380428 (588h26m27.457463473s) [noise=true , address=145.79.227.27:4689 , piece_id=IXSWHEI4YPUAYCD7A2ZD7FFHG65ATOIFRIQEAPIXWPNC7E5ZXHOA, seg_pos={0 0}, upload=515850]: Done uploading piece. err: <nil> Dec 30 03:55:27.586454268 (588h26m27.457537314s) [noise=true , address=145.79.227.42:4121 , piece_id=HDT35BDENUUQQ4LSLEPV2F4ERVQU4MVJU7VL52PJAGD45ABR5XXA, seg_pos={0 0}, upload=515850]: Done uploading piece. err: <nil> Dec 30 03:55:27.587462528 (588h26m27.458545573s) [noise=true , address=145.79.227.14:4839 , piece_id=7YUQXGKUXOWUBZJQNWYSZK2APWT65AS3ZRTSVBSYXT6B3423QBYQ, seg_pos={0 0}, upload=515850]: Done uploading piece. err: <nil> Dec 30 03:55:27.589673147 (588h26m27.460756182s) [noise=true , address=145.79.227.39:4015 , piece_id=RAKLGLN4T2JJ3WERVBK7UE6DDM6OC653EEDFXIFJAFVGOD4N7FQQ, seg_pos={0 0}, upload=515850]: Done uploading piece. err: <nil> Dec 30 03:55:27.590415280 (588h26m27.461498315s) [noise=true , address=145.79.227.3:4151 , piece_id=VMNYNCCO46MX7FUX6RHOKZWQDY3N4NMD2HF2N3HW5TVPSKFPD7UA, seg_pos={0 0}, upload=515850]: Done uploading piece. err: <nil> Dec 30 03:55:27.874380536 (588h26m27.745463581s) [noise=true , address=45.91.116.8:4048 , piece_id=CASP7FZEK4B4E23NLRFWJDRM5V5JFXYVPHBSH6JUVCZVVNOZPK5A, seg_pos={0 0}, upload=515850]: Done uploading piece. err: <nil> Dec 30 03:55:27.889876592 (588h26m27.760959637s) [noise=true , address=77.90.31.21:4878 , piece_id=CMRGKFBHMCTR2DQHQOM7GGDWHZRTUOJY42B6ZMJDDHSKPRUEXNRA, seg_pos={0 0}, upload=515850]: Done uploading piece. err: <nil> Dec 30 03:55:27.911230463 (588h26m27.782313508s) [noise=true , address=145.79.227.34:4907 , piece_id=GPMZYJ5Q2OD6KCE4T4WZ4EMYTMF52PG453RUYDGMI766QLQTIJWA, seg_pos={0 0}, upload=515850]: Done uploading piece. err: <nil> Dec 30 03:55:27.925490335 (588h26m27.79657338s) [noise=true , address=45.91.116.7:4010 , piece_id=2EBKSY2QGFBDQK6FWH6C54WRAICUYK7BI5DTZ5ZMOONC7B47FOQQ, seg_pos={0 0}, upload=515850]: Done uploading piece. err: <nil> Dec 30 03:55:27.955525315 (588h26m27.82660835s) [noise=true , address=45.91.116.4:4937 , piece_id=OZXMZWCQDFLBUPA36C2X4K27LIMHAFUMIA26ZIDNQ7WK5QMMMPGA, seg_pos={0 0}, upload=515850]: Done uploading piece. err: <nil> Dec 30 03:55:27.967144099 (588h26m27.838227144s) [noise=true , address=45.91.116.14:4189 , piece_id=PQS2F3TYHD3H2HZFBQ46HAERCLKF2ZUUCGG2B6CKXH73BJIBOUSA, seg_pos={0 0}, upload=515850]: Done uploading piece. err: <nil> Dec 30 03:55:27.971694734 (588h26m27.842777779s) [noise=true , address=45.91.116.6:4975 , piece_id=KKY6ANBWKXNQHBNJBM2NITWOYDSFBOMZNXXTFL7RUJILOTNT3B5A, seg_pos={0 0}, upload=515850]: Done uploading piece. err: <nil> Dec 30 03:55:27.972455824 (588h26m27.843538859s) [noise=true , address=77.90.31.19:4841 , piece_id=BMJLFDSK7WX3HGQ7RLDTEDZGB4NW5MV47XS4FSAMAIF7NFU2QPNA, seg_pos={0 0}, upload=515850]: Done uploading piece. err: <nil> Dec 30 03:55:27.972468477 (588h26m27.843551522s) [seg_pos={0 0}, upload=515850]: Segment reached optimal threshold of 13 pieces. Dec 30 03:55:27.972639593 (588h26m27.843722628s) [noise=true , address=145.79.227.2:4114 , piece_id=DSA3A75Y7SSN42DSQ27VO7PWHNMSH6H7KE7JXJ6LGEIX56QW3G4Q, seg_pos={0 0}, upload=515850]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN): protocol: write tcp 188.166.225.170:44928->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:44928->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44928->145.79.227.2:4114: use of closed network connection Dec 30 03:55:27.972645264 (588h26m27.843728299s) [noise=true , address=145.79.227.2:4114 , piece_id=DSA3A75Y7SSN42DSQ27VO7PWHNMSH6H7KE7JXJ6LGEIX56QW3G4Q, seg_pos={0 0}, upload=515850]: Long tail cancellation due to optimal threshold Dec 30 03:55:27.972957088 (588h26m27.844040133s) [noise=true , address=77.90.31.16:4736 , piece_id=QVSY52ZKVXZMJA7E655NOHZ33E7D3WXFCG6Q3Z5QZLERKZ6WWQKA, seg_pos={0 0}, upload=515850]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: write tcp 188.166.225.170:53750->77.90.31.16:4736: use of closed network connection; write tcp 188.166.225.170:53750->77.90.31.16:4736: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53750->77.90.31.16:4736: use of closed network connection Dec 30 03:55:27.972971575 (588h26m27.84405461s) [noise=true , address=77.90.31.16:4736 , piece_id=QVSY52ZKVXZMJA7E655NOHZ33E7D3WXFCG6Q3Z5QZLERKZ6WWQKA, seg_pos={0 0}, upload=515850]: Long tail cancellation due to optimal threshold Dec 30 03:55:27.973168019 (588h26m27.844251054s) [noise=true , address=77.90.31.10:4518 , piece_id=NPUTORIPATTOFGDBM7FEM5NBFAF4LRGTMNUX6ANV5MR3XIITFRDA, seg_pos={0 0}, upload=515850]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 30 03:55:27.973173930 (588h26m27.844256965s) [noise=true , address=77.90.31.10:4518 , piece_id=NPUTORIPATTOFGDBM7FEM5NBFAF4LRGTMNUX6ANV5MR3XIITFRDA, seg_pos={0 0}, upload=515850]: Long tail cancellation due to optimal threshold Dec 30 03:55:27.973191504 (588h26m27.844274539s) [seg_pos={0 0}, upload=515850]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 03:55:27.973291655 (588h26m27.84437469s) [upload=515850]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 03:55:27.973301393 (588h26m27.844384428s) [upload=515850]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 03:55:28.268468558 (588h26m28.139551593s) [upload=515850]: Done uploading segments... Dec 30 03:55:28.268490931 (588h26m28.139573976s) [upload=515850]: Upload finished. err: <nil> Duration: 5.754629942s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 37.533203301s trace: https://jaeger.dev.storj.io/trace/1d81d68a7c17f75d FAIL (2025-12-30 03:56:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 15 , piece_id=6FWCMOTNEZLDIH2F5VRBC3H44WVYAKWTUIL364QUAIC7DGKURRSQ, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.829621721 (588h22m39.700704767s) [noise=true , address=193.149.164.143:4695 , piece_id=C3ZXZXKRGAI2FATBQ2PNRJFNGMV4XKBXU4MSKZVM2AEEH5GVBRBQ, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.829918227 (588h22m39.701001282s) [noise=true , address=145.79.227.17:4998 , piece_id=IMFG6HRRWRXUX3F76A5RTU54HXCFSTPTWH4YJQPHGIOMPALNS5AA, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.830292459 (588h22m39.701375504s) [noise=true , address=45.91.116.12:4249 , piece_id=YLNIBVQGVGZ5CYEZSGLORL3D2ULBBCXN23SSNXUU7LQ67E3E4G7A, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.830455549 (588h22m39.701538594s) [noise=true , address=145.79.227.43:4178 , piece_id=7DIUURHOKK53VG6RMELJUV3D3AWUM23SHXNNVDRFTIURMXETX6QQ, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.830740912 (588h22m39.701823948s) [noise=true , address=77.90.31.16:4759 , piece_id=W3RFMAKOXSYIDCA27H7CKEXSIKCMRUO72NITXH5WDM4FFK6XPDAQ, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.830892231 (588h22m39.701975266s) [noise=true , address=145.79.227.45:4250 , piece_id=MDFRH2T6ZDAOE25DJTGHA6YSXS6CRL5OBES5T2FDTGY6EOV7LINQ, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.831333561 (588h22m39.702416606s) [noise=true , address=145.79.227.40:4053 , piece_id=AKAAPJDTKY3BB2OB5EFW4IYMYPOAPL4VDS5VEHJF3U3TA7JHZ4FQ, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.831388847 (588h22m39.702471882s) [noise=true , address=145.79.227.34:4916 , piece_id=DMOYHOUOVSMTMPZBY7MV5LYQ5JQDV4MKRJHBH7FYV2TBL37AFYSA, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.831789770 (588h22m39.702872815s) [noise=true , address=64.49.13.26:4946 , piece_id=X7CGAXO4356EG4I52QB5LQEM33UD2FJFS56YFEH7FNHAG2EBEZFA, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.832180875 (588h22m39.70326391s) [noise=true , address=77.90.31.19:4871 , piece_id=HPPBTDGVLD62WLHSPO36VFYHP5YFORYCBT22PK2CEYRJJM4UQLYA, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.832576678 (588h22m39.703659723s) [noise=true , address=145.79.227.33:4901 , piece_id=ODRNWAOOWKB45HREGAAKMJYNIN2I4OPRLUN5BIVIW7HWFQRTR6NA, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.834204918 (588h22m39.705287963s) [noise=true , address=45.91.116.13:4164 , piece_id=QW7FBIO24JGOXJEGO7YHYFMMTKHAKF64UEGNDXX2KKQLSJ4MH3PA, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.834999141 (588h22m39.706082186s) [noise=true , address=77.90.31.11:4584 , piece_id=WKRB35UKP7L6Z6G5WSJKMFBOF3RPWJYVZJPFBP5G4UT4RQVQP6MA, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.835718220 (588h22m39.706801265s) [noise=true , address=145.79.227.49:4361 , piece_id=ARLKW5A6Y3DVQ25E2RDL5KRFCYBNVIW7WBDL7ZO2WOQDTF5RCRKQ, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:41.118730125 (588h22m40.98981316s) [upload=515790]: Next returned nil segment Dec 30 03:51:41.118750513 (588h22m40.989833558s) [upload=515790]: Waiting for error group managing segments... Dec 30 03:51:41.505594353 (588h22m41.376677388s) [noise=true , address=64.49.13.26:4946 , piece_id=X7CGAXO4356EG4I52QB5LQEM33UD2FJFS56YFEH7FNHAG2EBEZFA, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:41.711421644 (588h22m41.582504689s) [noise=true , address=145.79.227.17:4998 , piece_id=IMFG6HRRWRXUX3F76A5RTU54HXCFSTPTWH4YJQPHGIOMPALNS5AA, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:41.718129295 (588h22m41.58921233s) [noise=true , address=145.79.227.6:4285 , piece_id=WNJGANKULJAMPLIT3KTHRSSAYTUSIIUCW5ZVKAFNURMGNZ3N2P3A, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:41.729197852 (588h22m41.600280887s) [noise=true , address=145.79.227.14:4856 , piece_id=CBZ6JFWGDMBMR7XJBXIETWDBEEXV43HJWJFKH5FY5BF7ANACFDOQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:41.867535754 (588h22m41.738618799s) [noise=true , address=145.79.227.43:4178 , piece_id=7DIUURHOKK53VG6RMELJUV3D3AWUM23SHXNNVDRFTIURMXETX6QQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:41.894207901 (588h22m41.765290946s) [noise=true , address=77.90.31.12:4615 , piece_id=6FWCMOTNEZLDIH2F5VRBC3H44WVYAKWTUIL364QUAIC7DGKURRSQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:41.928246715 (588h22m41.79932975s) [noise=true , address=145.79.227.45:4250 , piece_id=MDFRH2T6ZDAOE25DJTGHA6YSXS6CRL5OBES5T2FDTGY6EOV7LINQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:41.940833472 (588h22m41.811916517s) [noise=true , address=45.91.116.13:4164 , piece_id=QW7FBIO24JGOXJEGO7YHYFMMTKHAKF64UEGNDXX2KKQLSJ4MH3PA, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:42.318460461 (588h22m42.189543506s) [noise=true , address=77.90.31.16:4759 , piece_id=W3RFMAKOXSYIDCA27H7CKEXSIKCMRUO72NITXH5WDM4FFK6XPDAQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:42.367315176 (588h22m42.238398221s) [noise=true , address=77.90.31.19:4871 , piece_id=HPPBTDGVLD62WLHSPO36VFYHP5YFORYCBT22PK2CEYRJJM4UQLYA, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:42.536538860 (588h22m42.407621905s) [noise=true , address=145.79.227.40:4053 , piece_id=AKAAPJDTKY3BB2OB5EFW4IYMYPOAPL4VDS5VEHJF3U3TA7JHZ4FQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:42.637143172 (588h22m42.508226217s) [noise=true , address=145.79.227.44:4200 , piece_id=2KRTBC4EQE7OCNJJ7TPM72KJFYBBXFHJUP2FRENF6KL3CQNAAUQQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:42.715818140 (588h22m42.586901185s) [noise=true , address=45.91.116.12:4249 , piece_id=YLNIBVQGVGZ5CYEZSGLORL3D2ULBBCXN23SSNXUU7LQ67E3E4G7A, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:43.127299661 (588h22m42.998382696s) [noise=true , address=145.79.227.34:4916 , piece_id=DMOYHOUOVSMTMPZBY7MV5LYQ5JQDV4MKRJHBH7FYV2TBL37AFYSA, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:43.127318687 (588h22m42.998401722s) [seg_pos={0 0}, upload=515790]: Segment reached optimal threshold of 14 pieces. Dec 30 03:51:43.127508449 (588h22m42.998591484s) [noise=true , address=145.79.227.49:4361 , piece_id=ARLKW5A6Y3DVQ25E2RDL5KRFCYBNVIW7WBDL7ZO2WOQDTF5RCRKQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k): protocol: expected piece hash; context canceled; EOF Dec 30 03:51:43.127514470 (588h22m42.998597505s) [noise=true , address=145.79.227.49:4361 , piece_id=ARLKW5A6Y3DVQ25E2RDL5KRFCYBNVIW7WBDL7ZO2WOQDTF5RCRKQ, seg_pos={0 0}, upload=515790]: Long tail cancellation due to optimal threshold Dec 30 03:51:43.127619871 (588h22m42.998702906s) [noise=true , address=77.90.31.11:4584 , piece_id=WKRB35UKP7L6Z6G5WSJKMFBOF3RPWJYVZJPFBP5G4UT4RQVQP6MA, seg_pos={0 0}, upload=515790]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iybcvQgVfqHcCSNFAEVpKZJyciSHbBfeAiuk3uNE2LGq4aLfJ): protocol: write tcp 188.166.225.170:43330->77.90.31.11:4584: use of closed network connection; write tcp 188.166.225.170:43330->77.90.31.11:4584: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43330->77.90.31.11:4584: use of closed network connection Dec 30 03:51:43.127624550 (588h22m42.998707585s) [noise=true , address=77.90.31.11:4584 , piece_id=WKRB35UKP7L6Z6G5WSJKMFBOF3RPWJYVZJPFBP5G4UT4RQVQP6MA, seg_pos={0 0}, upload=515790]: Long tail cancellation due to optimal threshold Dec 30 03:51:43.127738065 (588h22m42.998821111s) [noise=true , address=193.149.164.143:4695 , piece_id=C3ZXZXKRGAI2FATBQ2PNRJFNGMV4XKBXU4MSKZVM2AEEH5GVBRBQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh): protocol: expected piece hash; context canceled; EOF Dec 30 03:51:43.127743557 (588h22m42.998826592s) [noise=true , address=193.149.164.143:4695 , piece_id=C3ZXZXKRGAI2FATBQ2PNRJFNGMV4XKBXU4MSKZVM2AEEH5GVBRBQ, seg_pos={0 0}, upload=515790]: Long tail cancellation due to optimal threshold Dec 30 03:51:43.127899784 (588h22m42.998982829s) [noise=true , address=145.79.227.33:4901 , piece_id=ODRNWAOOWKB45HREGAAKMJYNIN2I4OPRLUN5BIVIW7HWFQRTR6NA, seg_pos={0 0}, upload=515790]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1): protocol: write tcp 188.166.225.170:48420->145.79.227.33:4901: use of closed network connection; write tcp 188.166.225.170:48420->145.79.227.33:4901: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48420->145.79.227.33:4901: use of closed network connection Dec 30 03:51:43.127920484 (588h22m42.999003539s) [noise=true , address=145.79.227.33:4901 , piece_id=ODRNWAOOWKB45HREGAAKMJYNIN2I4OPRLUN5BIVIW7HWFQRTR6NA, seg_pos={0 0}, upload=515790]: Long tail cancellation due to optimal threshold Dec 30 03:51:43.128047735 (588h22m42.99913077s) [seg_pos={0 0}, upload=515790]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 03:51:43.128161222 (588h22m42.999244257s) [upload=515790]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 03:51:43.128174597 (588h22m42.999257642s) [upload=515790]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 03:51:43.415948996 (588h22m43.287032052s) [upload=515790]: Done uploading segments... Dec 30 03:51:43.415976829 (588h22m43.287059864s) [upload=515790]: Upload finished. err: <nil> Duration: 37.533217888s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [....X....X.X..........XX...X.......................X..XXX...] duration: 4.051460767s trace: https://jaeger.dev.storj.io/trace/532d591e051353a0 FAIL (2025-12-30 03:56:19 UTC) orbiter warning: context deadline exceeded Duration: 4.051491276s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [..X..X.X.X...XXX.X.....XXXX........XX.XX...XXXX...XXXX...X.X] duration: 30.00347536s trace: https://jaeger.dev.storj.io/trace/41ae83c3322c189c FAIL (2025-12-30 03:55:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: PQS2F3TYHD3H2HZFBQ46HAERCLKF2ZUUCGG2B6CKXH73BJIBOUSA) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: CASP7FZEK4B4E23NLRFWJDRM5V5JFXYVPHBSH6JUVCZVVNOZPK5A) context deadline exceeded; piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: CMRGKFBHMCTR2DQHQOM7GGDWHZRTUOJY42B6ZMJDDHSKPRUEXNRA) context deadline exceeded ======LOGS====== ng segments... Dec 30 03:55:25.346377833 (588h26m25.217460868s) [upload=515850]: Got next segment. Inline: false Dec 30 03:55:25.346450542 (588h26m25.217533597s) [upload=515850]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 03:55:25.346456974 (588h26m25.217540019s) [upload=515850]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 03:55:25.594479289 (588h26m25.465562324s) [seg_pos={0 0}, upload=515850]: Begin upload segment... Dec 30 03:55:25.594533653 (588h26m25.465616698s) [seg_pos={0 0}, upload=515850]: Done begin upload segment. Dec 30 03:55:25.594575863 (588h26m25.465658908s) [noise=true , address=77.90.31.16:4736 , piece_id=QVSY52ZKVXZMJA7E655NOHZ33E7D3WXFCG6Q3Z5QZLERKZ6WWQKA, seg_pos={0 0}, upload=515850]: Uploading piece... Dec 30 03:55:25.594599598 (588h26m25.465682633s) [noise=true , address=145.79.227.2:4114 , piece_id=DSA3A75Y7SSN42DSQ27VO7PWHNMSH6H7KE7JXJ6LGEIX56QW3G4Q, seg_pos={0 0}, upload=515850]: Uploading piece... Dec 30 03:55:25.595506324 (588h26m25.466589359s) [noise=true , address=145.79.227.27:4689 , piece_id=IXSWHEI4YPUAYCD7A2ZD7FFHG65ATOIFRIQEAPIXWPNC7E5ZXHOA, seg_pos={0 0}, upload=515850]: Uploading piece... Dec 30 03:55:25.596079456 (588h26m25.467162491s) [noise=true , address=45.91.116.7:4010 , piece_id=2EBKSY2QGFBDQK6FWH6C54WRAICUYK7BI5DTZ5ZMOONC7B47FOQQ, seg_pos={0 0}, upload=515850]: Uploading piece... Dec 30 03:55:25.596164848 (588h26m25.467247883s) [noise=true , address=145.79.227.3:4151 , piece_id=VMNYNCCO46MX7FUX6RHOKZWQDY3N4NMD2HF2N3HW5TVPSKFPD7UA, seg_pos={0 0}, upload=515850]: Uploading piece... Dec 30 03:55:25.596751285 (588h26m25.46783432s) [noise=true , address=45.91.116.4:4937 , piece_id=OZXMZWCQDFLBUPA36C2X4K27LIMHAFUMIA26ZIDNQ7WK5QMMMPGA, seg_pos={0 0}, upload=515850]: Uploading piece... Dec 30 03:55:25.597389460 (588h26m25.468472495s) [noise=true , address=77.90.31.21:4878 , piece_id=CMRGKFBHMCTR2DQHQOM7GGDWHZRTUOJY42B6ZMJDDHSKPRUEXNRA, seg_pos={0 0}, upload=515850]: Uploading piece... Dec 30 03:55:25.598414532 (588h26m25.469497577s) [noise=true , address=77.90.31.19:4841 , piece_id=BMJLFDSK7WX3HGQ7RLDTEDZGB4NW5MV47XS4FSAMAIF7NFU2QPNA, seg_pos={0 0}, upload=515850]: Uploading piece... Dec 30 03:55:25.599506091 (588h26m25.470589126s) [noise=true , address=145.79.227.39:4015 , piece_id=RAKLGLN4T2JJ3WERVBK7UE6DDM6OC653EEDFXIFJAFVGOD4N7FQQ, seg_pos={0 0}, upload=515850]: Uploading piece... Dec 30 03:55:25.599578068 (588h26m25.470661103s) [noise=true , address=77.90.31.10:4518 , piece_id=NPUTORIPATTOFGDBM7FEM5NBFAF4LRGTMNUX6ANV5MR3XIITFRDA, seg_pos={0 0}, upload=515850]: Uploading piece... Dec 30 03:55:25.600318988 (588h26m25.471402023s) [noise=true , address=45.91.116.14:4189 , piece_id=PQS2F3TYHD3H2HZFBQ46HAERCLKF2ZUUCGG2B6CKXH73BJIBOUSA, seg_pos={0 0}, upload=515850]: Uploading piece... Dec 30 03:55:25.601047295 (588h26m25.47213034s) [noise=true , address=145.79.227.42:4121 , piece_id=HDT35BDENUUQQ4LSLEPV2F4ERVQU4MVJU7VL52PJAGD45ABR5XXA, seg_pos={0 0}, upload=515850]: Uploading piece... Dec 30 03:55:25.601461624 (588h26m25.472544659s) [noise=true , address=45.91.116.8:4048 , piece_id=CASP7FZEK4B4E23NLRFWJDRM5V5JFXYVPHBSH6JUVCZVVNOZPK5A, seg_pos={0 0}, upload=515850]: Uploading piece... Dec 30 03:55:25.602910203 (588h26m25.473993238s) [noise=true , address=145.79.227.34:4907 , piece_id=GPMZYJ5Q2OD6KCE4T4WZ4EMYTMF52PG453RUYDGMI766QLQTIJWA, seg_pos={0 0}, upload=515850]: Uploading piece... Dec 30 03:55:25.603636665 (588h26m25.47471971s) [noise=true , address=45.91.116.6:4975 , piece_id=KKY6ANBWKXNQHBNJBM2NITWOYDSFBOMZNXXTFL7RUJILOTNT3B5A, seg_pos={0 0}, upload=515850]: Uploading piece... Dec 30 03:55:25.604181343 (588h26m25.475264388s) [noise=true , address=145.79.227.14:4839 , piece_id=7YUQXGKUXOWUBZJQNWYSZK2APWT65AS3ZRTSVBSYXT6B3423QBYQ, seg_pos={0 0}, upload=515850]: Uploading piece... Dec 30 03:55:27.273992732 (588h26m27.145075767s) [upload=515850]: Next returned nil segment Dec 30 03:55:27.274004635 (588h26m27.14508767s) [upload=515850]: Waiting for error group managing segments... Dec 30 03:55:27.586380428 (588h26m27.457463473s) [noise=true , address=145.79.227.27:4689 , piece_id=IXSWHEI4YPUAYCD7A2ZD7FFHG65ATOIFRIQEAPIXWPNC7E5ZXHOA, seg_pos={0 0}, upload=515850]: Done uploading piece. err: <nil> Dec 30 03:55:27.586454268 (588h26m27.457537314s) [noise=true , address=145.79.227.42:4121 , piece_id=HDT35BDENUUQQ4LSLEPV2F4ERVQU4MVJU7VL52PJAGD45ABR5XXA, seg_pos={0 0}, upload=515850]: Done uploading piece. err: <nil> Dec 30 03:55:27.587462528 (588h26m27.458545573s) [noise=true , address=145.79.227.14:4839 , piece_id=7YUQXGKUXOWUBZJQNWYSZK2APWT65AS3ZRTSVBSYXT6B3423QBYQ, seg_pos={0 0}, upload=515850]: Done uploading piece. err: <nil> Dec 30 03:55:27.589673147 (588h26m27.460756182s) [noise=true , address=145.79.227.39:4015 , piece_id=RAKLGLN4T2JJ3WERVBK7UE6DDM6OC653EEDFXIFJAFVGOD4N7FQQ, seg_pos={0 0}, upload=515850]: Done uploading piece. err: <nil> Dec 30 03:55:27.590415280 (588h26m27.461498315s) [noise=true , address=145.79.227.3:4151 , piece_id=VMNYNCCO46MX7FUX6RHOKZWQDY3N4NMD2HF2N3HW5TVPSKFPD7UA, seg_pos={0 0}, upload=515850]: Done uploading piece. err: <nil> Dec 30 03:55:27.874380536 (588h26m27.745463581s) [noise=true , address=45.91.116.8:4048 , piece_id=CASP7FZEK4B4E23NLRFWJDRM5V5JFXYVPHBSH6JUVCZVVNOZPK5A, seg_pos={0 0}, upload=515850]: Done uploading piece. err: <nil> Dec 30 03:55:27.889876592 (588h26m27.760959637s) [noise=true , address=77.90.31.21:4878 , piece_id=CMRGKFBHMCTR2DQHQOM7GGDWHZRTUOJY42B6ZMJDDHSKPRUEXNRA, seg_pos={0 0}, upload=515850]: Done uploading piece. err: <nil> Dec 30 03:55:27.911230463 (588h26m27.782313508s) [noise=true , address=145.79.227.34:4907 , piece_id=GPMZYJ5Q2OD6KCE4T4WZ4EMYTMF52PG453RUYDGMI766QLQTIJWA, seg_pos={0 0}, upload=515850]: Done uploading piece. err: <nil> Dec 30 03:55:27.925490335 (588h26m27.79657338s) [noise=true , address=45.91.116.7:4010 , piece_id=2EBKSY2QGFBDQK6FWH6C54WRAICUYK7BI5DTZ5ZMOONC7B47FOQQ, seg_pos={0 0}, upload=515850]: Done uploading piece. err: <nil> Dec 30 03:55:27.955525315 (588h26m27.82660835s) [noise=true , address=45.91.116.4:4937 , piece_id=OZXMZWCQDFLBUPA36C2X4K27LIMHAFUMIA26ZIDNQ7WK5QMMMPGA, seg_pos={0 0}, upload=515850]: Done uploading piece. err: <nil> Dec 30 03:55:27.967144099 (588h26m27.838227144s) [noise=true , address=45.91.116.14:4189 , piece_id=PQS2F3TYHD3H2HZFBQ46HAERCLKF2ZUUCGG2B6CKXH73BJIBOUSA, seg_pos={0 0}, upload=515850]: Done uploading piece. err: <nil> Dec 30 03:55:27.971694734 (588h26m27.842777779s) [noise=true , address=45.91.116.6:4975 , piece_id=KKY6ANBWKXNQHBNJBM2NITWOYDSFBOMZNXXTFL7RUJILOTNT3B5A, seg_pos={0 0}, upload=515850]: Done uploading piece. err: <nil> Dec 30 03:55:27.972455824 (588h26m27.843538859s) [noise=true , address=77.90.31.19:4841 , piece_id=BMJLFDSK7WX3HGQ7RLDTEDZGB4NW5MV47XS4FSAMAIF7NFU2QPNA, seg_pos={0 0}, upload=515850]: Done uploading piece. err: <nil> Dec 30 03:55:27.972468477 (588h26m27.843551522s) [seg_pos={0 0}, upload=515850]: Segment reached optimal threshold of 13 pieces. Dec 30 03:55:27.972639593 (588h26m27.843722628s) [noise=true , address=145.79.227.2:4114 , piece_id=DSA3A75Y7SSN42DSQ27VO7PWHNMSH6H7KE7JXJ6LGEIX56QW3G4Q, seg_pos={0 0}, upload=515850]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN): protocol: write tcp 188.166.225.170:44928->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:44928->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44928->145.79.227.2:4114: use of closed network connection Dec 30 03:55:27.972645264 (588h26m27.843728299s) [noise=true , address=145.79.227.2:4114 , piece_id=DSA3A75Y7SSN42DSQ27VO7PWHNMSH6H7KE7JXJ6LGEIX56QW3G4Q, seg_pos={0 0}, upload=515850]: Long tail cancellation due to optimal threshold Dec 30 03:55:27.972957088 (588h26m27.844040133s) [noise=true , address=77.90.31.16:4736 , piece_id=QVSY52ZKVXZMJA7E655NOHZ33E7D3WXFCG6Q3Z5QZLERKZ6WWQKA, seg_pos={0 0}, upload=515850]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: write tcp 188.166.225.170:53750->77.90.31.16:4736: use of closed network connection; write tcp 188.166.225.170:53750->77.90.31.16:4736: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53750->77.90.31.16:4736: use of closed network connection Dec 30 03:55:27.972971575 (588h26m27.84405461s) [noise=true , address=77.90.31.16:4736 , piece_id=QVSY52ZKVXZMJA7E655NOHZ33E7D3WXFCG6Q3Z5QZLERKZ6WWQKA, seg_pos={0 0}, upload=515850]: Long tail cancellation due to optimal threshold Dec 30 03:55:27.973168019 (588h26m27.844251054s) [noise=true , address=77.90.31.10:4518 , piece_id=NPUTORIPATTOFGDBM7FEM5NBFAF4LRGTMNUX6ANV5MR3XIITFRDA, seg_pos={0 0}, upload=515850]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 30 03:55:27.973173930 (588h26m27.844256965s) [noise=true , address=77.90.31.10:4518 , piece_id=NPUTORIPATTOFGDBM7FEM5NBFAF4LRGTMNUX6ANV5MR3XIITFRDA, seg_pos={0 0}, upload=515850]: Long tail cancellation due to optimal threshold Dec 30 03:55:27.973191504 (588h26m27.844274539s) [seg_pos={0 0}, upload=515850]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 03:55:27.973291655 (588h26m27.84437469s) [upload=515850]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 03:55:27.973301393 (588h26m27.844384428s) [upload=515850]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 03:55:28.268468558 (588h26m28.139551593s) [upload=515850]: Done uploading segments... Dec 30 03:55:28.268490931 (588h26m28.139573976s) [upload=515850]: Upload finished. err: <nil> Duration: 30.003493574s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 42.572964744s trace: https://jaeger.dev.storj.io/trace/1d81d68a7c17f75d FAIL (2025-12-30 03:55:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 15 , piece_id=6FWCMOTNEZLDIH2F5VRBC3H44WVYAKWTUIL364QUAIC7DGKURRSQ, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.829621721 (588h22m39.700704767s) [noise=true , address=193.149.164.143:4695 , piece_id=C3ZXZXKRGAI2FATBQ2PNRJFNGMV4XKBXU4MSKZVM2AEEH5GVBRBQ, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.829918227 (588h22m39.701001282s) [noise=true , address=145.79.227.17:4998 , piece_id=IMFG6HRRWRXUX3F76A5RTU54HXCFSTPTWH4YJQPHGIOMPALNS5AA, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.830292459 (588h22m39.701375504s) [noise=true , address=45.91.116.12:4249 , piece_id=YLNIBVQGVGZ5CYEZSGLORL3D2ULBBCXN23SSNXUU7LQ67E3E4G7A, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.830455549 (588h22m39.701538594s) [noise=true , address=145.79.227.43:4178 , piece_id=7DIUURHOKK53VG6RMELJUV3D3AWUM23SHXNNVDRFTIURMXETX6QQ, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.830740912 (588h22m39.701823948s) [noise=true , address=77.90.31.16:4759 , piece_id=W3RFMAKOXSYIDCA27H7CKEXSIKCMRUO72NITXH5WDM4FFK6XPDAQ, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.830892231 (588h22m39.701975266s) [noise=true , address=145.79.227.45:4250 , piece_id=MDFRH2T6ZDAOE25DJTGHA6YSXS6CRL5OBES5T2FDTGY6EOV7LINQ, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.831333561 (588h22m39.702416606s) [noise=true , address=145.79.227.40:4053 , piece_id=AKAAPJDTKY3BB2OB5EFW4IYMYPOAPL4VDS5VEHJF3U3TA7JHZ4FQ, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.831388847 (588h22m39.702471882s) [noise=true , address=145.79.227.34:4916 , piece_id=DMOYHOUOVSMTMPZBY7MV5LYQ5JQDV4MKRJHBH7FYV2TBL37AFYSA, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.831789770 (588h22m39.702872815s) [noise=true , address=64.49.13.26:4946 , piece_id=X7CGAXO4356EG4I52QB5LQEM33UD2FJFS56YFEH7FNHAG2EBEZFA, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.832180875 (588h22m39.70326391s) [noise=true , address=77.90.31.19:4871 , piece_id=HPPBTDGVLD62WLHSPO36VFYHP5YFORYCBT22PK2CEYRJJM4UQLYA, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.832576678 (588h22m39.703659723s) [noise=true , address=145.79.227.33:4901 , piece_id=ODRNWAOOWKB45HREGAAKMJYNIN2I4OPRLUN5BIVIW7HWFQRTR6NA, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.834204918 (588h22m39.705287963s) [noise=true , address=45.91.116.13:4164 , piece_id=QW7FBIO24JGOXJEGO7YHYFMMTKHAKF64UEGNDXX2KKQLSJ4MH3PA, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.834999141 (588h22m39.706082186s) [noise=true , address=77.90.31.11:4584 , piece_id=WKRB35UKP7L6Z6G5WSJKMFBOF3RPWJYVZJPFBP5G4UT4RQVQP6MA, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.835718220 (588h22m39.706801265s) [noise=true , address=145.79.227.49:4361 , piece_id=ARLKW5A6Y3DVQ25E2RDL5KRFCYBNVIW7WBDL7ZO2WOQDTF5RCRKQ, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:41.118730125 (588h22m40.98981316s) [upload=515790]: Next returned nil segment Dec 30 03:51:41.118750513 (588h22m40.989833558s) [upload=515790]: Waiting for error group managing segments... Dec 30 03:51:41.505594353 (588h22m41.376677388s) [noise=true , address=64.49.13.26:4946 , piece_id=X7CGAXO4356EG4I52QB5LQEM33UD2FJFS56YFEH7FNHAG2EBEZFA, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:41.711421644 (588h22m41.582504689s) [noise=true , address=145.79.227.17:4998 , piece_id=IMFG6HRRWRXUX3F76A5RTU54HXCFSTPTWH4YJQPHGIOMPALNS5AA, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:41.718129295 (588h22m41.58921233s) [noise=true , address=145.79.227.6:4285 , piece_id=WNJGANKULJAMPLIT3KTHRSSAYTUSIIUCW5ZVKAFNURMGNZ3N2P3A, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:41.729197852 (588h22m41.600280887s) [noise=true , address=145.79.227.14:4856 , piece_id=CBZ6JFWGDMBMR7XJBXIETWDBEEXV43HJWJFKH5FY5BF7ANACFDOQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:41.867535754 (588h22m41.738618799s) [noise=true , address=145.79.227.43:4178 , piece_id=7DIUURHOKK53VG6RMELJUV3D3AWUM23SHXNNVDRFTIURMXETX6QQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:41.894207901 (588h22m41.765290946s) [noise=true , address=77.90.31.12:4615 , piece_id=6FWCMOTNEZLDIH2F5VRBC3H44WVYAKWTUIL364QUAIC7DGKURRSQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:41.928246715 (588h22m41.79932975s) [noise=true , address=145.79.227.45:4250 , piece_id=MDFRH2T6ZDAOE25DJTGHA6YSXS6CRL5OBES5T2FDTGY6EOV7LINQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:41.940833472 (588h22m41.811916517s) [noise=true , address=45.91.116.13:4164 , piece_id=QW7FBIO24JGOXJEGO7YHYFMMTKHAKF64UEGNDXX2KKQLSJ4MH3PA, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:42.318460461 (588h22m42.189543506s) [noise=true , address=77.90.31.16:4759 , piece_id=W3RFMAKOXSYIDCA27H7CKEXSIKCMRUO72NITXH5WDM4FFK6XPDAQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:42.367315176 (588h22m42.238398221s) [noise=true , address=77.90.31.19:4871 , piece_id=HPPBTDGVLD62WLHSPO36VFYHP5YFORYCBT22PK2CEYRJJM4UQLYA, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:42.536538860 (588h22m42.407621905s) [noise=true , address=145.79.227.40:4053 , piece_id=AKAAPJDTKY3BB2OB5EFW4IYMYPOAPL4VDS5VEHJF3U3TA7JHZ4FQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:42.637143172 (588h22m42.508226217s) [noise=true , address=145.79.227.44:4200 , piece_id=2KRTBC4EQE7OCNJJ7TPM72KJFYBBXFHJUP2FRENF6KL3CQNAAUQQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:42.715818140 (588h22m42.586901185s) [noise=true , address=45.91.116.12:4249 , piece_id=YLNIBVQGVGZ5CYEZSGLORL3D2ULBBCXN23SSNXUU7LQ67E3E4G7A, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:43.127299661 (588h22m42.998382696s) [noise=true , address=145.79.227.34:4916 , piece_id=DMOYHOUOVSMTMPZBY7MV5LYQ5JQDV4MKRJHBH7FYV2TBL37AFYSA, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:43.127318687 (588h22m42.998401722s) [seg_pos={0 0}, upload=515790]: Segment reached optimal threshold of 14 pieces. Dec 30 03:51:43.127508449 (588h22m42.998591484s) [noise=true , address=145.79.227.49:4361 , piece_id=ARLKW5A6Y3DVQ25E2RDL5KRFCYBNVIW7WBDL7ZO2WOQDTF5RCRKQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k): protocol: expected piece hash; context canceled; EOF Dec 30 03:51:43.127514470 (588h22m42.998597505s) [noise=true , address=145.79.227.49:4361 , piece_id=ARLKW5A6Y3DVQ25E2RDL5KRFCYBNVIW7WBDL7ZO2WOQDTF5RCRKQ, seg_pos={0 0}, upload=515790]: Long tail cancellation due to optimal threshold Dec 30 03:51:43.127619871 (588h22m42.998702906s) [noise=true , address=77.90.31.11:4584 , piece_id=WKRB35UKP7L6Z6G5WSJKMFBOF3RPWJYVZJPFBP5G4UT4RQVQP6MA, seg_pos={0 0}, upload=515790]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iybcvQgVfqHcCSNFAEVpKZJyciSHbBfeAiuk3uNE2LGq4aLfJ): protocol: write tcp 188.166.225.170:43330->77.90.31.11:4584: use of closed network connection; write tcp 188.166.225.170:43330->77.90.31.11:4584: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43330->77.90.31.11:4584: use of closed network connection Dec 30 03:51:43.127624550 (588h22m42.998707585s) [noise=true , address=77.90.31.11:4584 , piece_id=WKRB35UKP7L6Z6G5WSJKMFBOF3RPWJYVZJPFBP5G4UT4RQVQP6MA, seg_pos={0 0}, upload=515790]: Long tail cancellation due to optimal threshold Dec 30 03:51:43.127738065 (588h22m42.998821111s) [noise=true , address=193.149.164.143:4695 , piece_id=C3ZXZXKRGAI2FATBQ2PNRJFNGMV4XKBXU4MSKZVM2AEEH5GVBRBQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh): protocol: expected piece hash; context canceled; EOF Dec 30 03:51:43.127743557 (588h22m42.998826592s) [noise=true , address=193.149.164.143:4695 , piece_id=C3ZXZXKRGAI2FATBQ2PNRJFNGMV4XKBXU4MSKZVM2AEEH5GVBRBQ, seg_pos={0 0}, upload=515790]: Long tail cancellation due to optimal threshold Dec 30 03:51:43.127899784 (588h22m42.998982829s) [noise=true , address=145.79.227.33:4901 , piece_id=ODRNWAOOWKB45HREGAAKMJYNIN2I4OPRLUN5BIVIW7HWFQRTR6NA, seg_pos={0 0}, upload=515790]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1): protocol: write tcp 188.166.225.170:48420->145.79.227.33:4901: use of closed network connection; write tcp 188.166.225.170:48420->145.79.227.33:4901: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48420->145.79.227.33:4901: use of closed network connection Dec 30 03:51:43.127920484 (588h22m42.999003539s) [noise=true , address=145.79.227.33:4901 , piece_id=ODRNWAOOWKB45HREGAAKMJYNIN2I4OPRLUN5BIVIW7HWFQRTR6NA, seg_pos={0 0}, upload=515790]: Long tail cancellation due to optimal threshold Dec 30 03:51:43.128047735 (588h22m42.99913077s) [seg_pos={0 0}, upload=515790]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 03:51:43.128161222 (588h22m42.999244257s) [upload=515790]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 03:51:43.128174597 (588h22m42.999257642s) [upload=515790]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 03:51:43.415948996 (588h22m43.287032052s) [upload=515790]: Done uploading segments... Dec 30 03:51:43.415976829 (588h22m43.287059864s) [upload=515790]: Upload finished. err: <nil> Duration: 42.572984802s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [.....X....X.X..........XX...X.......................X..XXX..] duration: 3.532118404s trace: https://jaeger.dev.storj.io/trace/532d591e051353a0 FAIL (2025-12-30 03:55:49 UTC) orbiter warning: context deadline exceeded Duration: 3.532147499s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [......X....X.X..........XX...X.......................X..XXX.] duration: 5.001681253s trace: https://jaeger.dev.storj.io/trace/532d591e051353a0 FAIL (2025-12-30 03:55:17 UTC) orbiter warning: context deadline exceeded Duration: 5.001715478s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [X..X..X.X.X...XXX.X.....XXXX........XX.XX...XXXX...XXXX...X.] duration: 5.964600172s trace: https://jaeger.dev.storj.io/trace/4d0089086b76f75 FAIL (2025-12-30 03:55:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: OYZIAG7S56VXD7G5QBBJ2XJIH5PCVCL5M6EMOYKLHUFJYNVAQ2WQ) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: P5GYWWVWHPSWB5UYKKAK3T444K2XLH5YDYJMJLGVNS6M3HN3WIWA) context deadline exceeded ======LOGS====== Dec 30 03:54:19.077600299 (588h25m18.948683344s) [upload=515832]: Starting upload Dec 30 03:54:19.077674681 (588h25m18.948757716s) [upload=515832]: Done starting upload Dec 30 03:54:19.077986164 (588h25m18.949069199s) [upload=515832]: Uploading segments... Dec 30 03:54:19.078073690 (588h25m18.949156755s) [upload=515832]: Got next segment. Inline: false Dec 30 03:54:19.078120881 (588h25m18.949203926s) [upload=515832]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 03:54:19.078126852 (588h25m18.949209897s) [upload=515832]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 03:54:19.322870864 (588h25m19.193953909s) [seg_pos={0 0}, upload=515832]: Begin upload segment... Dec 30 03:54:19.322960184 (588h25m19.194043219s) [seg_pos={0 0}, upload=515832]: Done begin upload segment. Dec 30 03:54:19.323009207 (588h25m19.194092242s) [noise=true , address=45.91.116.9:4262 , piece_id=Q5KSZD6HUY275FELKEJW3ZCC2HSYYXJFKOZ6SG4UABO6LTQOQGRA, seg_pos={0 0}, upload=515832]: Uploading piece... Dec 30 03:54:19.323015310 (588h25m19.194098345s) [noise=true , address=145.79.227.13:4804 , piece_id=3LDKTWW4ATIT5WEKICG4D4DNSZGVGWCJO3HJXRXQ333YEJNYYB7A, seg_pos={0 0}, upload=515832]: Uploading piece... Dec 30 03:54:19.323856081 (588h25m19.194939126s) [noise=true , address=45.91.116.5:4297 , piece_id=MGNZM5Q7LMHURCUZXYCOQFLNQZSZPEJYZYCDCOKLWCQO6QL2JLDA, seg_pos={0 0}, upload=515832]: Uploading piece... Dec 30 03:54:19.324216657 (588h25m19.195299692s) [noise=true , address=145.79.227.7:4293 , piece_id=R2WWRMVECI2SFCKCKYDFVXJUNFTQABE4LIYMEOIWOPWE2GRHN7XA, seg_pos={0 0}, upload=515832]: Uploading piece... Dec 30 03:54:19.325330868 (588h25m19.196413914s) [noise=true , address=145.79.227.21:4475 , piece_id=GJYHX3RK4HC7O3FNSRLFQPQFTU4WWQYIP5GEP255SEH63WZPAX2A, seg_pos={0 0}, upload=515832]: Uploading piece... Dec 30 03:54:19.327041065 (588h25m19.1981241s) [noise=true , address=145.79.227.3:4151 , piece_id=SPJCAMQAUVFARU4JC3J43ZXVL4DRWW2WUMA54WUQCVDWXQBXRTTQ, seg_pos={0 0}, upload=515832]: Uploading piece... Dec 30 03:54:19.327109285 (588h25m19.19819233s) [noise=true , address=77.90.31.11:4556 , piece_id=OYZIAG7S56VXD7G5QBBJ2XJIH5PCVCL5M6EMOYKLHUFJYNVAQ2WQ, seg_pos={0 0}, upload=515832]: Uploading piece... Dec 30 03:54:19.328485726 (588h25m19.199568761s) [noise=true , address=145.79.227.44:4194 , piece_id=SN2YIYGCO56QZ73KOSBLQO2KWNYG4KW6A2MY7YX3FS2RWSMQJFBA, seg_pos={0 0}, upload=515832]: Uploading piece... Dec 30 03:54:19.328507036 (588h25m19.199590071s) [noise=true , address=77.90.31.14:4663 , piece_id=P5GYWWVWHPSWB5UYKKAK3T444K2XLH5YDYJMJLGVNS6M3HN3WIWA, seg_pos={0 0}, upload=515832]: Uploading piece... Dec 30 03:54:19.330926764 (588h25m19.202009799s) [noise=true , address=145.79.227.19:4079 , piece_id=TTMEBLSZ44NAHHYLAG4CFFZWKM5CB4QBWBETBMZSG6VAZT6BTHBA, seg_pos={0 0}, upload=515832]: Uploading piece... Dec 30 03:54:19.332567148 (588h25m19.203650193s) [noise=true , address=145.79.227.14:4840 , piece_id=JII5UAY3GETLAETJU5JSYOYOF57A2LUNTQUKG3TSARD2J22IT6JQ, seg_pos={0 0}, upload=515832]: Uploading piece... Dec 30 03:54:19.333997421 (588h25m19.205080466s) [noise=true , address=145.79.227.46:4603 , piece_id=ECI4EOUWXDKHKZ6ZHIOVTZC5RCXS33RTM2LZSJWCM42UD2QRKR3A, seg_pos={0 0}, upload=515832]: Uploading piece... Dec 30 03:54:19.334649763 (588h25m19.205732808s) [noise=true , address=45.91.116.11:4119 , piece_id=HKZ3DW6LARRFHF4B6GFK3HGGCD2SVSEYILBIDP6FX5K3C4WJLILA, seg_pos={0 0}, upload=515832]: Uploading piece... Dec 30 03:54:19.336218079 (588h25m19.207301114s) [noise=true , address=145.79.227.17:4972 , piece_id=XBUYQ5AIXDRKXF6Q5MXSBPYZDT5SZVBV6TZDI545PI5DRAHN76UA, seg_pos={0 0}, upload=515832]: Uploading piece... Dec 30 03:54:19.337439705 (588h25m19.20852275s) [noise=true , address=45.91.116.14:4189 , piece_id=KSDPLD76WYKUAJHKTHUEMBDZINTRHJMTJSNEOLI2RZLM2GU376YQ, seg_pos={0 0}, upload=515832]: Uploading piece... Dec 30 03:54:19.338336172 (588h25m19.209419217s) [noise=true , address=77.90.31.21:4878 , piece_id=ZGHAIEN7HL2NKDTGL4HR4L2BYIQWFTH3OCZIL2FRY4UEAJM3TICQ, seg_pos={0 0}, upload=515832]: Uploading piece... Dec 30 03:54:21.023392130 (588h25m20.894475175s) [upload=515832]: Next returned nil segment Dec 30 03:54:21.023409061 (588h25m20.894492116s) [upload=515832]: Waiting for error group managing segments... Dec 30 03:54:21.305623253 (588h25m21.176706298s) [noise=true , address=145.79.227.46:4603 , piece_id=ECI4EOUWXDKHKZ6ZHIOVTZC5RCXS33RTM2LZSJWCM42UD2QRKR3A, seg_pos={0 0}, upload=515832]: Done uploading piece. err: <nil> Dec 30 03:54:21.305882195 (588h25m21.176965241s) [noise=true , address=145.79.227.13:4804 , piece_id=3LDKTWW4ATIT5WEKICG4D4DNSZGVGWCJO3HJXRXQ333YEJNYYB7A, seg_pos={0 0}, upload=515832]: Done uploading piece. err: <nil> Dec 30 03:54:21.315460915 (588h25m21.18654396s) [noise=true , address=145.79.227.14:4840 , piece_id=JII5UAY3GETLAETJU5JSYOYOF57A2LUNTQUKG3TSARD2J22IT6JQ, seg_pos={0 0}, upload=515832]: Done uploading piece. err: <nil> Dec 30 03:54:21.318602787 (588h25m21.189685822s) [noise=true , address=145.79.227.19:4079 , piece_id=TTMEBLSZ44NAHHYLAG4CFFZWKM5CB4QBWBETBMZSG6VAZT6BTHBA, seg_pos={0 0}, upload=515832]: Done uploading piece. err: <nil> Dec 30 03:54:21.319525885 (588h25m21.190608931s) [noise=true , address=145.79.227.3:4151 , piece_id=SPJCAMQAUVFARU4JC3J43ZXVL4DRWW2WUMA54WUQCVDWXQBXRTTQ, seg_pos={0 0}, upload=515832]: Done uploading piece. err: <nil> Dec 30 03:54:21.322358569 (588h25m21.193441614s) [noise=true , address=145.79.227.7:4293 , piece_id=R2WWRMVECI2SFCKCKYDFVXJUNFTQABE4LIYMEOIWOPWE2GRHN7XA, seg_pos={0 0}, upload=515832]: Done uploading piece. err: <nil> Dec 30 03:54:21.337090720 (588h25m21.208173755s) [noise=true , address=145.79.227.17:4972 , piece_id=XBUYQ5AIXDRKXF6Q5MXSBPYZDT5SZVBV6TZDI545PI5DRAHN76UA, seg_pos={0 0}, upload=515832]: Done uploading piece. err: <nil> Dec 30 03:54:21.346404566 (588h25m21.217487601s) [noise=true , address=145.79.227.21:4475 , piece_id=GJYHX3RK4HC7O3FNSRLFQPQFTU4WWQYIP5GEP255SEH63WZPAX2A, seg_pos={0 0}, upload=515832]: Done uploading piece. err: <nil> Dec 30 03:54:21.518161572 (588h25m21.389244617s) [noise=true , address=77.90.31.21:4878 , piece_id=ZGHAIEN7HL2NKDTGL4HR4L2BYIQWFTH3OCZIL2FRY4UEAJM3TICQ, seg_pos={0 0}, upload=515832]: Done uploading piece. err: <nil> Dec 30 03:54:21.601867768 (588h25m21.472950814s) [noise=true , address=77.90.31.14:4663 , piece_id=P5GYWWVWHPSWB5UYKKAK3T444K2XLH5YDYJMJLGVNS6M3HN3WIWA, seg_pos={0 0}, upload=515832]: Done uploading piece. err: <nil> Dec 30 03:54:21.914480202 (588h25m21.785563237s) [noise=true , address=45.91.116.9:4262 , piece_id=Q5KSZD6HUY275FELKEJW3ZCC2HSYYXJFKOZ6SG4UABO6LTQOQGRA, seg_pos={0 0}, upload=515832]: Done uploading piece. err: <nil> Dec 30 03:54:21.979145923 (588h25m21.850228968s) [noise=true , address=45.91.116.11:4119 , piece_id=HKZ3DW6LARRFHF4B6GFK3HGGCD2SVSEYILBIDP6FX5K3C4WJLILA, seg_pos={0 0}, upload=515832]: Done uploading piece. err: <nil> Dec 30 03:54:22.263571143 (588h25m22.134654178s) [noise=true , address=77.90.31.11:4556 , piece_id=OYZIAG7S56VXD7G5QBBJ2XJIH5PCVCL5M6EMOYKLHUFJYNVAQ2WQ, seg_pos={0 0}, upload=515832]: Done uploading piece. err: <nil> Dec 30 03:54:22.263591382 (588h25m22.134674427s) [seg_pos={0 0}, upload=515832]: Segment reached optimal threshold of 13 pieces. Dec 30 03:54:22.263819186 (588h25m22.134902221s) [noise=true , address=45.91.116.5:4297 , piece_id=MGNZM5Q7LMHURCUZXYCOQFLNQZSZPEJYZYCDCOKLWCQO6QL2JLDA, seg_pos={0 0}, upload=515832]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 30 03:54:22.263825848 (588h25m22.134908883s) [noise=true , address=45.91.116.5:4297 , piece_id=MGNZM5Q7LMHURCUZXYCOQFLNQZSZPEJYZYCDCOKLWCQO6QL2JLDA, seg_pos={0 0}, upload=515832]: Long tail cancellation due to optimal threshold Dec 30 03:54:22.263864472 (588h25m22.134947518s) [noise=true , address=45.91.116.14:4189 , piece_id=KSDPLD76WYKUAJHKTHUEMBDZINTRHJMTJSNEOLI2RZLM2GU376YQ, seg_pos={0 0}, upload=515832]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 30 03:54:22.263872668 (588h25m22.134955703s) [noise=true , address=45.91.116.14:4189 , piece_id=KSDPLD76WYKUAJHKTHUEMBDZINTRHJMTJSNEOLI2RZLM2GU376YQ, seg_pos={0 0}, upload=515832]: Long tail cancellation due to optimal threshold Dec 30 03:54:22.264049494 (588h25m22.13513253s) [noise=true , address=145.79.227.44:4194 , piece_id=SN2YIYGCO56QZ73KOSBLQO2KWNYG4KW6A2MY7YX3FS2RWSMQJFBA, seg_pos={0 0}, upload=515832]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 30 03:54:22.264056017 (588h25m22.135139052s) [noise=true , address=145.79.227.44:4194 , piece_id=SN2YIYGCO56QZ73KOSBLQO2KWNYG4KW6A2MY7YX3FS2RWSMQJFBA, seg_pos={0 0}, upload=515832]: Long tail cancellation due to optimal threshold Dec 30 03:54:22.264072639 (588h25m22.135155674s) [seg_pos={0 0}, upload=515832]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 03:54:22.264149345 (588h25m22.13523239s) [upload=515832]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 03:54:22.264154054 (588h25m22.135237089s) [upload=515832]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 03:54:22.557943865 (588h25m22.429026911s) [upload=515832]: Done uploading segments... Dec 30 03:54:22.557968182 (588h25m22.429051227s) [upload=515832]: Upload finished. err: <nil> Duration: 5.964711344s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 44.262911133s trace: https://jaeger.dev.storj.io/trace/1d81d68a7c17f75d FAIL (2025-12-30 03:55:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 15 , piece_id=6FWCMOTNEZLDIH2F5VRBC3H44WVYAKWTUIL364QUAIC7DGKURRSQ, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.829621721 (588h22m39.700704767s) [noise=true , address=193.149.164.143:4695 , piece_id=C3ZXZXKRGAI2FATBQ2PNRJFNGMV4XKBXU4MSKZVM2AEEH5GVBRBQ, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.829918227 (588h22m39.701001282s) [noise=true , address=145.79.227.17:4998 , piece_id=IMFG6HRRWRXUX3F76A5RTU54HXCFSTPTWH4YJQPHGIOMPALNS5AA, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.830292459 (588h22m39.701375504s) [noise=true , address=45.91.116.12:4249 , piece_id=YLNIBVQGVGZ5CYEZSGLORL3D2ULBBCXN23SSNXUU7LQ67E3E4G7A, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.830455549 (588h22m39.701538594s) [noise=true , address=145.79.227.43:4178 , piece_id=7DIUURHOKK53VG6RMELJUV3D3AWUM23SHXNNVDRFTIURMXETX6QQ, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.830740912 (588h22m39.701823948s) [noise=true , address=77.90.31.16:4759 , piece_id=W3RFMAKOXSYIDCA27H7CKEXSIKCMRUO72NITXH5WDM4FFK6XPDAQ, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.830892231 (588h22m39.701975266s) [noise=true , address=145.79.227.45:4250 , piece_id=MDFRH2T6ZDAOE25DJTGHA6YSXS6CRL5OBES5T2FDTGY6EOV7LINQ, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.831333561 (588h22m39.702416606s) [noise=true , address=145.79.227.40:4053 , piece_id=AKAAPJDTKY3BB2OB5EFW4IYMYPOAPL4VDS5VEHJF3U3TA7JHZ4FQ, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.831388847 (588h22m39.702471882s) [noise=true , address=145.79.227.34:4916 , piece_id=DMOYHOUOVSMTMPZBY7MV5LYQ5JQDV4MKRJHBH7FYV2TBL37AFYSA, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.831789770 (588h22m39.702872815s) [noise=true , address=64.49.13.26:4946 , piece_id=X7CGAXO4356EG4I52QB5LQEM33UD2FJFS56YFEH7FNHAG2EBEZFA, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.832180875 (588h22m39.70326391s) [noise=true , address=77.90.31.19:4871 , piece_id=HPPBTDGVLD62WLHSPO36VFYHP5YFORYCBT22PK2CEYRJJM4UQLYA, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.832576678 (588h22m39.703659723s) [noise=true , address=145.79.227.33:4901 , piece_id=ODRNWAOOWKB45HREGAAKMJYNIN2I4OPRLUN5BIVIW7HWFQRTR6NA, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.834204918 (588h22m39.705287963s) [noise=true , address=45.91.116.13:4164 , piece_id=QW7FBIO24JGOXJEGO7YHYFMMTKHAKF64UEGNDXX2KKQLSJ4MH3PA, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.834999141 (588h22m39.706082186s) [noise=true , address=77.90.31.11:4584 , piece_id=WKRB35UKP7L6Z6G5WSJKMFBOF3RPWJYVZJPFBP5G4UT4RQVQP6MA, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.835718220 (588h22m39.706801265s) [noise=true , address=145.79.227.49:4361 , piece_id=ARLKW5A6Y3DVQ25E2RDL5KRFCYBNVIW7WBDL7ZO2WOQDTF5RCRKQ, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:41.118730125 (588h22m40.98981316s) [upload=515790]: Next returned nil segment Dec 30 03:51:41.118750513 (588h22m40.989833558s) [upload=515790]: Waiting for error group managing segments... Dec 30 03:51:41.505594353 (588h22m41.376677388s) [noise=true , address=64.49.13.26:4946 , piece_id=X7CGAXO4356EG4I52QB5LQEM33UD2FJFS56YFEH7FNHAG2EBEZFA, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:41.711421644 (588h22m41.582504689s) [noise=true , address=145.79.227.17:4998 , piece_id=IMFG6HRRWRXUX3F76A5RTU54HXCFSTPTWH4YJQPHGIOMPALNS5AA, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:41.718129295 (588h22m41.58921233s) [noise=true , address=145.79.227.6:4285 , piece_id=WNJGANKULJAMPLIT3KTHRSSAYTUSIIUCW5ZVKAFNURMGNZ3N2P3A, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:41.729197852 (588h22m41.600280887s) [noise=true , address=145.79.227.14:4856 , piece_id=CBZ6JFWGDMBMR7XJBXIETWDBEEXV43HJWJFKH5FY5BF7ANACFDOQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:41.867535754 (588h22m41.738618799s) [noise=true , address=145.79.227.43:4178 , piece_id=7DIUURHOKK53VG6RMELJUV3D3AWUM23SHXNNVDRFTIURMXETX6QQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:41.894207901 (588h22m41.765290946s) [noise=true , address=77.90.31.12:4615 , piece_id=6FWCMOTNEZLDIH2F5VRBC3H44WVYAKWTUIL364QUAIC7DGKURRSQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:41.928246715 (588h22m41.79932975s) [noise=true , address=145.79.227.45:4250 , piece_id=MDFRH2T6ZDAOE25DJTGHA6YSXS6CRL5OBES5T2FDTGY6EOV7LINQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:41.940833472 (588h22m41.811916517s) [noise=true , address=45.91.116.13:4164 , piece_id=QW7FBIO24JGOXJEGO7YHYFMMTKHAKF64UEGNDXX2KKQLSJ4MH3PA, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:42.318460461 (588h22m42.189543506s) [noise=true , address=77.90.31.16:4759 , piece_id=W3RFMAKOXSYIDCA27H7CKEXSIKCMRUO72NITXH5WDM4FFK6XPDAQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:42.367315176 (588h22m42.238398221s) [noise=true , address=77.90.31.19:4871 , piece_id=HPPBTDGVLD62WLHSPO36VFYHP5YFORYCBT22PK2CEYRJJM4UQLYA, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:42.536538860 (588h22m42.407621905s) [noise=true , address=145.79.227.40:4053 , piece_id=AKAAPJDTKY3BB2OB5EFW4IYMYPOAPL4VDS5VEHJF3U3TA7JHZ4FQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:42.637143172 (588h22m42.508226217s) [noise=true , address=145.79.227.44:4200 , piece_id=2KRTBC4EQE7OCNJJ7TPM72KJFYBBXFHJUP2FRENF6KL3CQNAAUQQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:42.715818140 (588h22m42.586901185s) [noise=true , address=45.91.116.12:4249 , piece_id=YLNIBVQGVGZ5CYEZSGLORL3D2ULBBCXN23SSNXUU7LQ67E3E4G7A, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:43.127299661 (588h22m42.998382696s) [noise=true , address=145.79.227.34:4916 , piece_id=DMOYHOUOVSMTMPZBY7MV5LYQ5JQDV4MKRJHBH7FYV2TBL37AFYSA, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:43.127318687 (588h22m42.998401722s) [seg_pos={0 0}, upload=515790]: Segment reached optimal threshold of 14 pieces. Dec 30 03:51:43.127508449 (588h22m42.998591484s) [noise=true , address=145.79.227.49:4361 , piece_id=ARLKW5A6Y3DVQ25E2RDL5KRFCYBNVIW7WBDL7ZO2WOQDTF5RCRKQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k): protocol: expected piece hash; context canceled; EOF Dec 30 03:51:43.127514470 (588h22m42.998597505s) [noise=true , address=145.79.227.49:4361 , piece_id=ARLKW5A6Y3DVQ25E2RDL5KRFCYBNVIW7WBDL7ZO2WOQDTF5RCRKQ, seg_pos={0 0}, upload=515790]: Long tail cancellation due to optimal threshold Dec 30 03:51:43.127619871 (588h22m42.998702906s) [noise=true , address=77.90.31.11:4584 , piece_id=WKRB35UKP7L6Z6G5WSJKMFBOF3RPWJYVZJPFBP5G4UT4RQVQP6MA, seg_pos={0 0}, upload=515790]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iybcvQgVfqHcCSNFAEVpKZJyciSHbBfeAiuk3uNE2LGq4aLfJ): protocol: write tcp 188.166.225.170:43330->77.90.31.11:4584: use of closed network connection; write tcp 188.166.225.170:43330->77.90.31.11:4584: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43330->77.90.31.11:4584: use of closed network connection Dec 30 03:51:43.127624550 (588h22m42.998707585s) [noise=true , address=77.90.31.11:4584 , piece_id=WKRB35UKP7L6Z6G5WSJKMFBOF3RPWJYVZJPFBP5G4UT4RQVQP6MA, seg_pos={0 0}, upload=515790]: Long tail cancellation due to optimal threshold Dec 30 03:51:43.127738065 (588h22m42.998821111s) [noise=true , address=193.149.164.143:4695 , piece_id=C3ZXZXKRGAI2FATBQ2PNRJFNGMV4XKBXU4MSKZVM2AEEH5GVBRBQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh): protocol: expected piece hash; context canceled; EOF Dec 30 03:51:43.127743557 (588h22m42.998826592s) [noise=true , address=193.149.164.143:4695 , piece_id=C3ZXZXKRGAI2FATBQ2PNRJFNGMV4XKBXU4MSKZVM2AEEH5GVBRBQ, seg_pos={0 0}, upload=515790]: Long tail cancellation due to optimal threshold Dec 30 03:51:43.127899784 (588h22m42.998982829s) [noise=true , address=145.79.227.33:4901 , piece_id=ODRNWAOOWKB45HREGAAKMJYNIN2I4OPRLUN5BIVIW7HWFQRTR6NA, seg_pos={0 0}, upload=515790]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1): protocol: write tcp 188.166.225.170:48420->145.79.227.33:4901: use of closed network connection; write tcp 188.166.225.170:48420->145.79.227.33:4901: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48420->145.79.227.33:4901: use of closed network connection Dec 30 03:51:43.127920484 (588h22m42.999003539s) [noise=true , address=145.79.227.33:4901 , piece_id=ODRNWAOOWKB45HREGAAKMJYNIN2I4OPRLUN5BIVIW7HWFQRTR6NA, seg_pos={0 0}, upload=515790]: Long tail cancellation due to optimal threshold Dec 30 03:51:43.128047735 (588h22m42.99913077s) [seg_pos={0 0}, upload=515790]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 03:51:43.128161222 (588h22m42.999244257s) [upload=515790]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 03:51:43.128174597 (588h22m42.999257642s) [upload=515790]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 03:51:43.415948996 (588h22m43.287032052s) [upload=515790]: Done uploading segments... Dec 30 03:51:43.415976829 (588h22m43.287059864s) [upload=515790]: Upload finished. err: <nil> Duration: 44.262929167s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [XX..X..X.X.X...XXX.X.....XXXX........XX.XX...XXXX...XXXX...X] duration: 30.00338268s trace: https://jaeger.dev.storj.io/trace/4d0089086b76f75 FAIL (2025-12-30 03:54:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: OYZIAG7S56VXD7G5QBBJ2XJIH5PCVCL5M6EMOYKLHUFJYNVAQ2WQ) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: P5GYWWVWHPSWB5UYKKAK3T444K2XLH5YDYJMJLGVNS6M3HN3WIWA) context deadline exceeded ======LOGS====== Dec 30 03:54:19.077600299 (588h25m18.948683344s) [upload=515832]: Starting upload Dec 30 03:54:19.077674681 (588h25m18.948757716s) [upload=515832]: Done starting upload Dec 30 03:54:19.077986164 (588h25m18.949069199s) [upload=515832]: Uploading segments... Dec 30 03:54:19.078073690 (588h25m18.949156755s) [upload=515832]: Got next segment. Inline: false Dec 30 03:54:19.078120881 (588h25m18.949203926s) [upload=515832]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 03:54:19.078126852 (588h25m18.949209897s) [upload=515832]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 03:54:19.322870864 (588h25m19.193953909s) [seg_pos={0 0}, upload=515832]: Begin upload segment... Dec 30 03:54:19.322960184 (588h25m19.194043219s) [seg_pos={0 0}, upload=515832]: Done begin upload segment. Dec 30 03:54:19.323009207 (588h25m19.194092242s) [noise=true , address=45.91.116.9:4262 , piece_id=Q5KSZD6HUY275FELKEJW3ZCC2HSYYXJFKOZ6SG4UABO6LTQOQGRA, seg_pos={0 0}, upload=515832]: Uploading piece... Dec 30 03:54:19.323015310 (588h25m19.194098345s) [noise=true , address=145.79.227.13:4804 , piece_id=3LDKTWW4ATIT5WEKICG4D4DNSZGVGWCJO3HJXRXQ333YEJNYYB7A, seg_pos={0 0}, upload=515832]: Uploading piece... Dec 30 03:54:19.323856081 (588h25m19.194939126s) [noise=true , address=45.91.116.5:4297 , piece_id=MGNZM5Q7LMHURCUZXYCOQFLNQZSZPEJYZYCDCOKLWCQO6QL2JLDA, seg_pos={0 0}, upload=515832]: Uploading piece... Dec 30 03:54:19.324216657 (588h25m19.195299692s) [noise=true , address=145.79.227.7:4293 , piece_id=R2WWRMVECI2SFCKCKYDFVXJUNFTQABE4LIYMEOIWOPWE2GRHN7XA, seg_pos={0 0}, upload=515832]: Uploading piece... Dec 30 03:54:19.325330868 (588h25m19.196413914s) [noise=true , address=145.79.227.21:4475 , piece_id=GJYHX3RK4HC7O3FNSRLFQPQFTU4WWQYIP5GEP255SEH63WZPAX2A, seg_pos={0 0}, upload=515832]: Uploading piece... Dec 30 03:54:19.327041065 (588h25m19.1981241s) [noise=true , address=145.79.227.3:4151 , piece_id=SPJCAMQAUVFARU4JC3J43ZXVL4DRWW2WUMA54WUQCVDWXQBXRTTQ, seg_pos={0 0}, upload=515832]: Uploading piece... Dec 30 03:54:19.327109285 (588h25m19.19819233s) [noise=true , address=77.90.31.11:4556 , piece_id=OYZIAG7S56VXD7G5QBBJ2XJIH5PCVCL5M6EMOYKLHUFJYNVAQ2WQ, seg_pos={0 0}, upload=515832]: Uploading piece... Dec 30 03:54:19.328485726 (588h25m19.199568761s) [noise=true , address=145.79.227.44:4194 , piece_id=SN2YIYGCO56QZ73KOSBLQO2KWNYG4KW6A2MY7YX3FS2RWSMQJFBA, seg_pos={0 0}, upload=515832]: Uploading piece... Dec 30 03:54:19.328507036 (588h25m19.199590071s) [noise=true , address=77.90.31.14:4663 , piece_id=P5GYWWVWHPSWB5UYKKAK3T444K2XLH5YDYJMJLGVNS6M3HN3WIWA, seg_pos={0 0}, upload=515832]: Uploading piece... Dec 30 03:54:19.330926764 (588h25m19.202009799s) [noise=true , address=145.79.227.19:4079 , piece_id=TTMEBLSZ44NAHHYLAG4CFFZWKM5CB4QBWBETBMZSG6VAZT6BTHBA, seg_pos={0 0}, upload=515832]: Uploading piece... Dec 30 03:54:19.332567148 (588h25m19.203650193s) [noise=true , address=145.79.227.14:4840 , piece_id=JII5UAY3GETLAETJU5JSYOYOF57A2LUNTQUKG3TSARD2J22IT6JQ, seg_pos={0 0}, upload=515832]: Uploading piece... Dec 30 03:54:19.333997421 (588h25m19.205080466s) [noise=true , address=145.79.227.46:4603 , piece_id=ECI4EOUWXDKHKZ6ZHIOVTZC5RCXS33RTM2LZSJWCM42UD2QRKR3A, seg_pos={0 0}, upload=515832]: Uploading piece... Dec 30 03:54:19.334649763 (588h25m19.205732808s) [noise=true , address=45.91.116.11:4119 , piece_id=HKZ3DW6LARRFHF4B6GFK3HGGCD2SVSEYILBIDP6FX5K3C4WJLILA, seg_pos={0 0}, upload=515832]: Uploading piece... Dec 30 03:54:19.336218079 (588h25m19.207301114s) [noise=true , address=145.79.227.17:4972 , piece_id=XBUYQ5AIXDRKXF6Q5MXSBPYZDT5SZVBV6TZDI545PI5DRAHN76UA, seg_pos={0 0}, upload=515832]: Uploading piece... Dec 30 03:54:19.337439705 (588h25m19.20852275s) [noise=true , address=45.91.116.14:4189 , piece_id=KSDPLD76WYKUAJHKTHUEMBDZINTRHJMTJSNEOLI2RZLM2GU376YQ, seg_pos={0 0}, upload=515832]: Uploading piece... Dec 30 03:54:19.338336172 (588h25m19.209419217s) [noise=true , address=77.90.31.21:4878 , piece_id=ZGHAIEN7HL2NKDTGL4HR4L2BYIQWFTH3OCZIL2FRY4UEAJM3TICQ, seg_pos={0 0}, upload=515832]: Uploading piece... Dec 30 03:54:21.023392130 (588h25m20.894475175s) [upload=515832]: Next returned nil segment Dec 30 03:54:21.023409061 (588h25m20.894492116s) [upload=515832]: Waiting for error group managing segments... Dec 30 03:54:21.305623253 (588h25m21.176706298s) [noise=true , address=145.79.227.46:4603 , piece_id=ECI4EOUWXDKHKZ6ZHIOVTZC5RCXS33RTM2LZSJWCM42UD2QRKR3A, seg_pos={0 0}, upload=515832]: Done uploading piece. err: <nil> Dec 30 03:54:21.305882195 (588h25m21.176965241s) [noise=true , address=145.79.227.13:4804 , piece_id=3LDKTWW4ATIT5WEKICG4D4DNSZGVGWCJO3HJXRXQ333YEJNYYB7A, seg_pos={0 0}, upload=515832]: Done uploading piece. err: <nil> Dec 30 03:54:21.315460915 (588h25m21.18654396s) [noise=true , address=145.79.227.14:4840 , piece_id=JII5UAY3GETLAETJU5JSYOYOF57A2LUNTQUKG3TSARD2J22IT6JQ, seg_pos={0 0}, upload=515832]: Done uploading piece. err: <nil> Dec 30 03:54:21.318602787 (588h25m21.189685822s) [noise=true , address=145.79.227.19:4079 , piece_id=TTMEBLSZ44NAHHYLAG4CFFZWKM5CB4QBWBETBMZSG6VAZT6BTHBA, seg_pos={0 0}, upload=515832]: Done uploading piece. err: <nil> Dec 30 03:54:21.319525885 (588h25m21.190608931s) [noise=true , address=145.79.227.3:4151 , piece_id=SPJCAMQAUVFARU4JC3J43ZXVL4DRWW2WUMA54WUQCVDWXQBXRTTQ, seg_pos={0 0}, upload=515832]: Done uploading piece. err: <nil> Dec 30 03:54:21.322358569 (588h25m21.193441614s) [noise=true , address=145.79.227.7:4293 , piece_id=R2WWRMVECI2SFCKCKYDFVXJUNFTQABE4LIYMEOIWOPWE2GRHN7XA, seg_pos={0 0}, upload=515832]: Done uploading piece. err: <nil> Dec 30 03:54:21.337090720 (588h25m21.208173755s) [noise=true , address=145.79.227.17:4972 , piece_id=XBUYQ5AIXDRKXF6Q5MXSBPYZDT5SZVBV6TZDI545PI5DRAHN76UA, seg_pos={0 0}, upload=515832]: Done uploading piece. err: <nil> Dec 30 03:54:21.346404566 (588h25m21.217487601s) [noise=true , address=145.79.227.21:4475 , piece_id=GJYHX3RK4HC7O3FNSRLFQPQFTU4WWQYIP5GEP255SEH63WZPAX2A, seg_pos={0 0}, upload=515832]: Done uploading piece. err: <nil> Dec 30 03:54:21.518161572 (588h25m21.389244617s) [noise=true , address=77.90.31.21:4878 , piece_id=ZGHAIEN7HL2NKDTGL4HR4L2BYIQWFTH3OCZIL2FRY4UEAJM3TICQ, seg_pos={0 0}, upload=515832]: Done uploading piece. err: <nil> Dec 30 03:54:21.601867768 (588h25m21.472950814s) [noise=true , address=77.90.31.14:4663 , piece_id=P5GYWWVWHPSWB5UYKKAK3T444K2XLH5YDYJMJLGVNS6M3HN3WIWA, seg_pos={0 0}, upload=515832]: Done uploading piece. err: <nil> Dec 30 03:54:21.914480202 (588h25m21.785563237s) [noise=true , address=45.91.116.9:4262 , piece_id=Q5KSZD6HUY275FELKEJW3ZCC2HSYYXJFKOZ6SG4UABO6LTQOQGRA, seg_pos={0 0}, upload=515832]: Done uploading piece. err: <nil> Dec 30 03:54:21.979145923 (588h25m21.850228968s) [noise=true , address=45.91.116.11:4119 , piece_id=HKZ3DW6LARRFHF4B6GFK3HGGCD2SVSEYILBIDP6FX5K3C4WJLILA, seg_pos={0 0}, upload=515832]: Done uploading piece. err: <nil> Dec 30 03:54:22.263571143 (588h25m22.134654178s) [noise=true , address=77.90.31.11:4556 , piece_id=OYZIAG7S56VXD7G5QBBJ2XJIH5PCVCL5M6EMOYKLHUFJYNVAQ2WQ, seg_pos={0 0}, upload=515832]: Done uploading piece. err: <nil> Dec 30 03:54:22.263591382 (588h25m22.134674427s) [seg_pos={0 0}, upload=515832]: Segment reached optimal threshold of 13 pieces. Dec 30 03:54:22.263819186 (588h25m22.134902221s) [noise=true , address=45.91.116.5:4297 , piece_id=MGNZM5Q7LMHURCUZXYCOQFLNQZSZPEJYZYCDCOKLWCQO6QL2JLDA, seg_pos={0 0}, upload=515832]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 30 03:54:22.263825848 (588h25m22.134908883s) [noise=true , address=45.91.116.5:4297 , piece_id=MGNZM5Q7LMHURCUZXYCOQFLNQZSZPEJYZYCDCOKLWCQO6QL2JLDA, seg_pos={0 0}, upload=515832]: Long tail cancellation due to optimal threshold Dec 30 03:54:22.263864472 (588h25m22.134947518s) [noise=true , address=45.91.116.14:4189 , piece_id=KSDPLD76WYKUAJHKTHUEMBDZINTRHJMTJSNEOLI2RZLM2GU376YQ, seg_pos={0 0}, upload=515832]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 30 03:54:22.263872668 (588h25m22.134955703s) [noise=true , address=45.91.116.14:4189 , piece_id=KSDPLD76WYKUAJHKTHUEMBDZINTRHJMTJSNEOLI2RZLM2GU376YQ, seg_pos={0 0}, upload=515832]: Long tail cancellation due to optimal threshold Dec 30 03:54:22.264049494 (588h25m22.13513253s) [noise=true , address=145.79.227.44:4194 , piece_id=SN2YIYGCO56QZ73KOSBLQO2KWNYG4KW6A2MY7YX3FS2RWSMQJFBA, seg_pos={0 0}, upload=515832]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 30 03:54:22.264056017 (588h25m22.135139052s) [noise=true , address=145.79.227.44:4194 , piece_id=SN2YIYGCO56QZ73KOSBLQO2KWNYG4KW6A2MY7YX3FS2RWSMQJFBA, seg_pos={0 0}, upload=515832]: Long tail cancellation due to optimal threshold Dec 30 03:54:22.264072639 (588h25m22.135155674s) [seg_pos={0 0}, upload=515832]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 03:54:22.264149345 (588h25m22.13523239s) [upload=515832]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 03:54:22.264154054 (588h25m22.135237089s) [upload=515832]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 03:54:22.557943865 (588h25m22.429026911s) [upload=515832]: Done uploading segments... Dec 30 03:54:22.557968182 (588h25m22.429051227s) [upload=515832]: Upload finished. err: <nil> Duration: 30.003410623s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [.......X....X.X..........XX...X.......................X..XXX] duration: 5.000815157s trace: https://jaeger.dev.storj.io/trace/532d591e051353a0 FAIL (2025-12-30 03:54:37 UTC) orbiter warning: context deadline exceeded Duration: 5.000836628s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 41.501806422s trace: https://jaeger.dev.storj.io/trace/1d81d68a7c17f75d FAIL (2025-12-30 03:54:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 15 , piece_id=6FWCMOTNEZLDIH2F5VRBC3H44WVYAKWTUIL364QUAIC7DGKURRSQ, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.829621721 (588h22m39.700704767s) [noise=true , address=193.149.164.143:4695 , piece_id=C3ZXZXKRGAI2FATBQ2PNRJFNGMV4XKBXU4MSKZVM2AEEH5GVBRBQ, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.829918227 (588h22m39.701001282s) [noise=true , address=145.79.227.17:4998 , piece_id=IMFG6HRRWRXUX3F76A5RTU54HXCFSTPTWH4YJQPHGIOMPALNS5AA, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.830292459 (588h22m39.701375504s) [noise=true , address=45.91.116.12:4249 , piece_id=YLNIBVQGVGZ5CYEZSGLORL3D2ULBBCXN23SSNXUU7LQ67E3E4G7A, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.830455549 (588h22m39.701538594s) [noise=true , address=145.79.227.43:4178 , piece_id=7DIUURHOKK53VG6RMELJUV3D3AWUM23SHXNNVDRFTIURMXETX6QQ, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.830740912 (588h22m39.701823948s) [noise=true , address=77.90.31.16:4759 , piece_id=W3RFMAKOXSYIDCA27H7CKEXSIKCMRUO72NITXH5WDM4FFK6XPDAQ, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.830892231 (588h22m39.701975266s) [noise=true , address=145.79.227.45:4250 , piece_id=MDFRH2T6ZDAOE25DJTGHA6YSXS6CRL5OBES5T2FDTGY6EOV7LINQ, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.831333561 (588h22m39.702416606s) [noise=true , address=145.79.227.40:4053 , piece_id=AKAAPJDTKY3BB2OB5EFW4IYMYPOAPL4VDS5VEHJF3U3TA7JHZ4FQ, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.831388847 (588h22m39.702471882s) [noise=true , address=145.79.227.34:4916 , piece_id=DMOYHOUOVSMTMPZBY7MV5LYQ5JQDV4MKRJHBH7FYV2TBL37AFYSA, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.831789770 (588h22m39.702872815s) [noise=true , address=64.49.13.26:4946 , piece_id=X7CGAXO4356EG4I52QB5LQEM33UD2FJFS56YFEH7FNHAG2EBEZFA, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.832180875 (588h22m39.70326391s) [noise=true , address=77.90.31.19:4871 , piece_id=HPPBTDGVLD62WLHSPO36VFYHP5YFORYCBT22PK2CEYRJJM4UQLYA, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.832576678 (588h22m39.703659723s) [noise=true , address=145.79.227.33:4901 , piece_id=ODRNWAOOWKB45HREGAAKMJYNIN2I4OPRLUN5BIVIW7HWFQRTR6NA, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.834204918 (588h22m39.705287963s) [noise=true , address=45.91.116.13:4164 , piece_id=QW7FBIO24JGOXJEGO7YHYFMMTKHAKF64UEGNDXX2KKQLSJ4MH3PA, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.834999141 (588h22m39.706082186s) [noise=true , address=77.90.31.11:4584 , piece_id=WKRB35UKP7L6Z6G5WSJKMFBOF3RPWJYVZJPFBP5G4UT4RQVQP6MA, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.835718220 (588h22m39.706801265s) [noise=true , address=145.79.227.49:4361 , piece_id=ARLKW5A6Y3DVQ25E2RDL5KRFCYBNVIW7WBDL7ZO2WOQDTF5RCRKQ, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:41.118730125 (588h22m40.98981316s) [upload=515790]: Next returned nil segment Dec 30 03:51:41.118750513 (588h22m40.989833558s) [upload=515790]: Waiting for error group managing segments... Dec 30 03:51:41.505594353 (588h22m41.376677388s) [noise=true , address=64.49.13.26:4946 , piece_id=X7CGAXO4356EG4I52QB5LQEM33UD2FJFS56YFEH7FNHAG2EBEZFA, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:41.711421644 (588h22m41.582504689s) [noise=true , address=145.79.227.17:4998 , piece_id=IMFG6HRRWRXUX3F76A5RTU54HXCFSTPTWH4YJQPHGIOMPALNS5AA, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:41.718129295 (588h22m41.58921233s) [noise=true , address=145.79.227.6:4285 , piece_id=WNJGANKULJAMPLIT3KTHRSSAYTUSIIUCW5ZVKAFNURMGNZ3N2P3A, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:41.729197852 (588h22m41.600280887s) [noise=true , address=145.79.227.14:4856 , piece_id=CBZ6JFWGDMBMR7XJBXIETWDBEEXV43HJWJFKH5FY5BF7ANACFDOQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:41.867535754 (588h22m41.738618799s) [noise=true , address=145.79.227.43:4178 , piece_id=7DIUURHOKK53VG6RMELJUV3D3AWUM23SHXNNVDRFTIURMXETX6QQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:41.894207901 (588h22m41.765290946s) [noise=true , address=77.90.31.12:4615 , piece_id=6FWCMOTNEZLDIH2F5VRBC3H44WVYAKWTUIL364QUAIC7DGKURRSQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:41.928246715 (588h22m41.79932975s) [noise=true , address=145.79.227.45:4250 , piece_id=MDFRH2T6ZDAOE25DJTGHA6YSXS6CRL5OBES5T2FDTGY6EOV7LINQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:41.940833472 (588h22m41.811916517s) [noise=true , address=45.91.116.13:4164 , piece_id=QW7FBIO24JGOXJEGO7YHYFMMTKHAKF64UEGNDXX2KKQLSJ4MH3PA, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:42.318460461 (588h22m42.189543506s) [noise=true , address=77.90.31.16:4759 , piece_id=W3RFMAKOXSYIDCA27H7CKEXSIKCMRUO72NITXH5WDM4FFK6XPDAQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:42.367315176 (588h22m42.238398221s) [noise=true , address=77.90.31.19:4871 , piece_id=HPPBTDGVLD62WLHSPO36VFYHP5YFORYCBT22PK2CEYRJJM4UQLYA, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:42.536538860 (588h22m42.407621905s) [noise=true , address=145.79.227.40:4053 , piece_id=AKAAPJDTKY3BB2OB5EFW4IYMYPOAPL4VDS5VEHJF3U3TA7JHZ4FQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:42.637143172 (588h22m42.508226217s) [noise=true , address=145.79.227.44:4200 , piece_id=2KRTBC4EQE7OCNJJ7TPM72KJFYBBXFHJUP2FRENF6KL3CQNAAUQQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:42.715818140 (588h22m42.586901185s) [noise=true , address=45.91.116.12:4249 , piece_id=YLNIBVQGVGZ5CYEZSGLORL3D2ULBBCXN23SSNXUU7LQ67E3E4G7A, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:43.127299661 (588h22m42.998382696s) [noise=true , address=145.79.227.34:4916 , piece_id=DMOYHOUOVSMTMPZBY7MV5LYQ5JQDV4MKRJHBH7FYV2TBL37AFYSA, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:43.127318687 (588h22m42.998401722s) [seg_pos={0 0}, upload=515790]: Segment reached optimal threshold of 14 pieces. Dec 30 03:51:43.127508449 (588h22m42.998591484s) [noise=true , address=145.79.227.49:4361 , piece_id=ARLKW5A6Y3DVQ25E2RDL5KRFCYBNVIW7WBDL7ZO2WOQDTF5RCRKQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k): protocol: expected piece hash; context canceled; EOF Dec 30 03:51:43.127514470 (588h22m42.998597505s) [noise=true , address=145.79.227.49:4361 , piece_id=ARLKW5A6Y3DVQ25E2RDL5KRFCYBNVIW7WBDL7ZO2WOQDTF5RCRKQ, seg_pos={0 0}, upload=515790]: Long tail cancellation due to optimal threshold Dec 30 03:51:43.127619871 (588h22m42.998702906s) [noise=true , address=77.90.31.11:4584 , piece_id=WKRB35UKP7L6Z6G5WSJKMFBOF3RPWJYVZJPFBP5G4UT4RQVQP6MA, seg_pos={0 0}, upload=515790]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iybcvQgVfqHcCSNFAEVpKZJyciSHbBfeAiuk3uNE2LGq4aLfJ): protocol: write tcp 188.166.225.170:43330->77.90.31.11:4584: use of closed network connection; write tcp 188.166.225.170:43330->77.90.31.11:4584: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43330->77.90.31.11:4584: use of closed network connection Dec 30 03:51:43.127624550 (588h22m42.998707585s) [noise=true , address=77.90.31.11:4584 , piece_id=WKRB35UKP7L6Z6G5WSJKMFBOF3RPWJYVZJPFBP5G4UT4RQVQP6MA, seg_pos={0 0}, upload=515790]: Long tail cancellation due to optimal threshold Dec 30 03:51:43.127738065 (588h22m42.998821111s) [noise=true , address=193.149.164.143:4695 , piece_id=C3ZXZXKRGAI2FATBQ2PNRJFNGMV4XKBXU4MSKZVM2AEEH5GVBRBQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh): protocol: expected piece hash; context canceled; EOF Dec 30 03:51:43.127743557 (588h22m42.998826592s) [noise=true , address=193.149.164.143:4695 , piece_id=C3ZXZXKRGAI2FATBQ2PNRJFNGMV4XKBXU4MSKZVM2AEEH5GVBRBQ, seg_pos={0 0}, upload=515790]: Long tail cancellation due to optimal threshold Dec 30 03:51:43.127899784 (588h22m42.998982829s) [noise=true , address=145.79.227.33:4901 , piece_id=ODRNWAOOWKB45HREGAAKMJYNIN2I4OPRLUN5BIVIW7HWFQRTR6NA, seg_pos={0 0}, upload=515790]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1): protocol: write tcp 188.166.225.170:48420->145.79.227.33:4901: use of closed network connection; write tcp 188.166.225.170:48420->145.79.227.33:4901: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48420->145.79.227.33:4901: use of closed network connection Dec 30 03:51:43.127920484 (588h22m42.999003539s) [noise=true , address=145.79.227.33:4901 , piece_id=ODRNWAOOWKB45HREGAAKMJYNIN2I4OPRLUN5BIVIW7HWFQRTR6NA, seg_pos={0 0}, upload=515790]: Long tail cancellation due to optimal threshold Dec 30 03:51:43.128047735 (588h22m42.99913077s) [seg_pos={0 0}, upload=515790]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 03:51:43.128161222 (588h22m42.999244257s) [upload=515790]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 03:51:43.128174597 (588h22m42.999257642s) [upload=515790]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 03:51:43.415948996 (588h22m43.287032052s) [upload=515790]: Done uploading segments... Dec 30 03:51:43.415976829 (588h22m43.287059864s) [upload=515790]: Upload finished. err: <nil> Duration: 41.501826029s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [........X....X.X..........XX...X.......................X..XX] duration: 5.001719478s trace: https://jaeger.dev.storj.io/trace/774f27b465521e6b FAIL (2025-12-30 03:54:12 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001743534s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [.........X....X.X..........XX...X.......................X..X] duration: 5.001367128s trace: https://jaeger.dev.storj.io/trace/5b20bc52a641f38c FAIL (2025-12-30 03:53:53 UTC) orbiter warning: context deadline exceeded Duration: 5.001390863s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 25.441868573s trace: https://jaeger.dev.storj.io/trace/1d81d68a7c17f75d FAIL (2025-12-30 03:53:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 15 , piece_id=6FWCMOTNEZLDIH2F5VRBC3H44WVYAKWTUIL364QUAIC7DGKURRSQ, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.829621721 (588h22m39.700704767s) [noise=true , address=193.149.164.143:4695 , piece_id=C3ZXZXKRGAI2FATBQ2PNRJFNGMV4XKBXU4MSKZVM2AEEH5GVBRBQ, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.829918227 (588h22m39.701001282s) [noise=true , address=145.79.227.17:4998 , piece_id=IMFG6HRRWRXUX3F76A5RTU54HXCFSTPTWH4YJQPHGIOMPALNS5AA, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.830292459 (588h22m39.701375504s) [noise=true , address=45.91.116.12:4249 , piece_id=YLNIBVQGVGZ5CYEZSGLORL3D2ULBBCXN23SSNXUU7LQ67E3E4G7A, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.830455549 (588h22m39.701538594s) [noise=true , address=145.79.227.43:4178 , piece_id=7DIUURHOKK53VG6RMELJUV3D3AWUM23SHXNNVDRFTIURMXETX6QQ, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.830740912 (588h22m39.701823948s) [noise=true , address=77.90.31.16:4759 , piece_id=W3RFMAKOXSYIDCA27H7CKEXSIKCMRUO72NITXH5WDM4FFK6XPDAQ, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.830892231 (588h22m39.701975266s) [noise=true , address=145.79.227.45:4250 , piece_id=MDFRH2T6ZDAOE25DJTGHA6YSXS6CRL5OBES5T2FDTGY6EOV7LINQ, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.831333561 (588h22m39.702416606s) [noise=true , address=145.79.227.40:4053 , piece_id=AKAAPJDTKY3BB2OB5EFW4IYMYPOAPL4VDS5VEHJF3U3TA7JHZ4FQ, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.831388847 (588h22m39.702471882s) [noise=true , address=145.79.227.34:4916 , piece_id=DMOYHOUOVSMTMPZBY7MV5LYQ5JQDV4MKRJHBH7FYV2TBL37AFYSA, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.831789770 (588h22m39.702872815s) [noise=true , address=64.49.13.26:4946 , piece_id=X7CGAXO4356EG4I52QB5LQEM33UD2FJFS56YFEH7FNHAG2EBEZFA, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.832180875 (588h22m39.70326391s) [noise=true , address=77.90.31.19:4871 , piece_id=HPPBTDGVLD62WLHSPO36VFYHP5YFORYCBT22PK2CEYRJJM4UQLYA, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.832576678 (588h22m39.703659723s) [noise=true , address=145.79.227.33:4901 , piece_id=ODRNWAOOWKB45HREGAAKMJYNIN2I4OPRLUN5BIVIW7HWFQRTR6NA, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.834204918 (588h22m39.705287963s) [noise=true , address=45.91.116.13:4164 , piece_id=QW7FBIO24JGOXJEGO7YHYFMMTKHAKF64UEGNDXX2KKQLSJ4MH3PA, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.834999141 (588h22m39.706082186s) [noise=true , address=77.90.31.11:4584 , piece_id=WKRB35UKP7L6Z6G5WSJKMFBOF3RPWJYVZJPFBP5G4UT4RQVQP6MA, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.835718220 (588h22m39.706801265s) [noise=true , address=145.79.227.49:4361 , piece_id=ARLKW5A6Y3DVQ25E2RDL5KRFCYBNVIW7WBDL7ZO2WOQDTF5RCRKQ, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:41.118730125 (588h22m40.98981316s) [upload=515790]: Next returned nil segment Dec 30 03:51:41.118750513 (588h22m40.989833558s) [upload=515790]: Waiting for error group managing segments... Dec 30 03:51:41.505594353 (588h22m41.376677388s) [noise=true , address=64.49.13.26:4946 , piece_id=X7CGAXO4356EG4I52QB5LQEM33UD2FJFS56YFEH7FNHAG2EBEZFA, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:41.711421644 (588h22m41.582504689s) [noise=true , address=145.79.227.17:4998 , piece_id=IMFG6HRRWRXUX3F76A5RTU54HXCFSTPTWH4YJQPHGIOMPALNS5AA, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:41.718129295 (588h22m41.58921233s) [noise=true , address=145.79.227.6:4285 , piece_id=WNJGANKULJAMPLIT3KTHRSSAYTUSIIUCW5ZVKAFNURMGNZ3N2P3A, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:41.729197852 (588h22m41.600280887s) [noise=true , address=145.79.227.14:4856 , piece_id=CBZ6JFWGDMBMR7XJBXIETWDBEEXV43HJWJFKH5FY5BF7ANACFDOQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:41.867535754 (588h22m41.738618799s) [noise=true , address=145.79.227.43:4178 , piece_id=7DIUURHOKK53VG6RMELJUV3D3AWUM23SHXNNVDRFTIURMXETX6QQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:41.894207901 (588h22m41.765290946s) [noise=true , address=77.90.31.12:4615 , piece_id=6FWCMOTNEZLDIH2F5VRBC3H44WVYAKWTUIL364QUAIC7DGKURRSQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:41.928246715 (588h22m41.79932975s) [noise=true , address=145.79.227.45:4250 , piece_id=MDFRH2T6ZDAOE25DJTGHA6YSXS6CRL5OBES5T2FDTGY6EOV7LINQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:41.940833472 (588h22m41.811916517s) [noise=true , address=45.91.116.13:4164 , piece_id=QW7FBIO24JGOXJEGO7YHYFMMTKHAKF64UEGNDXX2KKQLSJ4MH3PA, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:42.318460461 (588h22m42.189543506s) [noise=true , address=77.90.31.16:4759 , piece_id=W3RFMAKOXSYIDCA27H7CKEXSIKCMRUO72NITXH5WDM4FFK6XPDAQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:42.367315176 (588h22m42.238398221s) [noise=true , address=77.90.31.19:4871 , piece_id=HPPBTDGVLD62WLHSPO36VFYHP5YFORYCBT22PK2CEYRJJM4UQLYA, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:42.536538860 (588h22m42.407621905s) [noise=true , address=145.79.227.40:4053 , piece_id=AKAAPJDTKY3BB2OB5EFW4IYMYPOAPL4VDS5VEHJF3U3TA7JHZ4FQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:42.637143172 (588h22m42.508226217s) [noise=true , address=145.79.227.44:4200 , piece_id=2KRTBC4EQE7OCNJJ7TPM72KJFYBBXFHJUP2FRENF6KL3CQNAAUQQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:42.715818140 (588h22m42.586901185s) [noise=true , address=45.91.116.12:4249 , piece_id=YLNIBVQGVGZ5CYEZSGLORL3D2ULBBCXN23SSNXUU7LQ67E3E4G7A, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:43.127299661 (588h22m42.998382696s) [noise=true , address=145.79.227.34:4916 , piece_id=DMOYHOUOVSMTMPZBY7MV5LYQ5JQDV4MKRJHBH7FYV2TBL37AFYSA, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:43.127318687 (588h22m42.998401722s) [seg_pos={0 0}, upload=515790]: Segment reached optimal threshold of 14 pieces. Dec 30 03:51:43.127508449 (588h22m42.998591484s) [noise=true , address=145.79.227.49:4361 , piece_id=ARLKW5A6Y3DVQ25E2RDL5KRFCYBNVIW7WBDL7ZO2WOQDTF5RCRKQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k): protocol: expected piece hash; context canceled; EOF Dec 30 03:51:43.127514470 (588h22m42.998597505s) [noise=true , address=145.79.227.49:4361 , piece_id=ARLKW5A6Y3DVQ25E2RDL5KRFCYBNVIW7WBDL7ZO2WOQDTF5RCRKQ, seg_pos={0 0}, upload=515790]: Long tail cancellation due to optimal threshold Dec 30 03:51:43.127619871 (588h22m42.998702906s) [noise=true , address=77.90.31.11:4584 , piece_id=WKRB35UKP7L6Z6G5WSJKMFBOF3RPWJYVZJPFBP5G4UT4RQVQP6MA, seg_pos={0 0}, upload=515790]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iybcvQgVfqHcCSNFAEVpKZJyciSHbBfeAiuk3uNE2LGq4aLfJ): protocol: write tcp 188.166.225.170:43330->77.90.31.11:4584: use of closed network connection; write tcp 188.166.225.170:43330->77.90.31.11:4584: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43330->77.90.31.11:4584: use of closed network connection Dec 30 03:51:43.127624550 (588h22m42.998707585s) [noise=true , address=77.90.31.11:4584 , piece_id=WKRB35UKP7L6Z6G5WSJKMFBOF3RPWJYVZJPFBP5G4UT4RQVQP6MA, seg_pos={0 0}, upload=515790]: Long tail cancellation due to optimal threshold Dec 30 03:51:43.127738065 (588h22m42.998821111s) [noise=true , address=193.149.164.143:4695 , piece_id=C3ZXZXKRGAI2FATBQ2PNRJFNGMV4XKBXU4MSKZVM2AEEH5GVBRBQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh): protocol: expected piece hash; context canceled; EOF Dec 30 03:51:43.127743557 (588h22m42.998826592s) [noise=true , address=193.149.164.143:4695 , piece_id=C3ZXZXKRGAI2FATBQ2PNRJFNGMV4XKBXU4MSKZVM2AEEH5GVBRBQ, seg_pos={0 0}, upload=515790]: Long tail cancellation due to optimal threshold Dec 30 03:51:43.127899784 (588h22m42.998982829s) [noise=true , address=145.79.227.33:4901 , piece_id=ODRNWAOOWKB45HREGAAKMJYNIN2I4OPRLUN5BIVIW7HWFQRTR6NA, seg_pos={0 0}, upload=515790]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1): protocol: write tcp 188.166.225.170:48420->145.79.227.33:4901: use of closed network connection; write tcp 188.166.225.170:48420->145.79.227.33:4901: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48420->145.79.227.33:4901: use of closed network connection Dec 30 03:51:43.127920484 (588h22m42.999003539s) [noise=true , address=145.79.227.33:4901 , piece_id=ODRNWAOOWKB45HREGAAKMJYNIN2I4OPRLUN5BIVIW7HWFQRTR6NA, seg_pos={0 0}, upload=515790]: Long tail cancellation due to optimal threshold Dec 30 03:51:43.128047735 (588h22m42.99913077s) [seg_pos={0 0}, upload=515790]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 03:51:43.128161222 (588h22m42.999244257s) [upload=515790]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 03:51:43.128174597 (588h22m42.999257642s) [upload=515790]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 03:51:43.415948996 (588h22m43.287032052s) [upload=515790]: Done uploading segments... Dec 30 03:51:43.415976829 (588h22m43.287059864s) [upload=515790]: Upload finished. err: <nil> Duration: 25.441883493s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [.XX..X..X.X.X...XXX.X.....XXXX........XX.XX...XXXX...XXXX...] duration: 5.690632644s trace: https://jaeger.dev.storj.io/trace/29cbd316ab2b2965 FAIL (2025-12-30 03:53:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: BGRIQ2C32XKNT7TYHITIPRIN7LDMA7ABWMPLAKXGERGCUAZ5XSBA) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: MULDAZEZVIZMW6UXEZNHM2ZF762GQWXYOJBHOEFJZEF2MEUSX2IA) context deadline exceeded ======LOGS====== 646402225 (588h21m44.51748527s) [upload=515778]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 03:50:44.894588582 (588h21m44.765671617s) [seg_pos={0 0}, upload=515778]: Begin upload segment... Dec 30 03:50:44.894641924 (588h21m44.765724959s) [seg_pos={0 0}, upload=515778]: Done begin upload segment. Dec 30 03:50:44.894677161 (588h21m44.765760206s) [noise=true , address=145.79.227.51:4411 , piece_id=ET37UNKINHDW4XSBHKPXWG2OFZQU6LW23UKCZCHHWQQJGWYCHOSA, seg_pos={0 0}, upload=515778]: Uploading piece... Dec 30 03:50:44.894724260 (588h21m44.765807295s) [noise=true , address=145.79.227.26:4655 , piece_id=F4PIAKSTVFOWKZ2DFSDZTSSVB7EHT4RAGO4I5GCNV6I5NI5OBEEA, seg_pos={0 0}, upload=515778]: Uploading piece... Dec 30 03:50:44.895676623 (588h21m44.766759668s) [noise=true , address=145.79.227.2:4115 , piece_id=H5FAMB5RLMZ3FO6ZMTV6GXCIML5LU5SLSDR2GNH5DCIR7IHAICXA, seg_pos={0 0}, upload=515778]: Uploading piece... Dec 30 03:50:44.896188809 (588h21m44.767271844s) [noise=true , address=77.90.31.19:4844 , piece_id=RDDQDSIBCJRUGXUNACXWJQPACPOMDAV34BAAAH7WTNXYMLRCI34A, seg_pos={0 0}, upload=515778]: Uploading piece... Dec 30 03:50:44.896308968 (588h21m44.767392003s) [noise=true , address=145.79.227.7:4294 , piece_id=7CBO7REVWCP4LUQZ57QV2OU7BGE4RHDYBOK5G6PI7BJXJAFFUZXA, seg_pos={0 0}, upload=515778]: Uploading piece... Dec 30 03:50:44.896584913 (588h21m44.767667958s) [noise=true , address=145.79.227.13:4804 , piece_id=LG4FAVRK2PWRX2TGHA7M3VWWGWPE5AHCI7CTZ4LLGHJ3QCMIICMA, seg_pos={0 0}, upload=515778]: Uploading piece... Dec 30 03:50:44.896774955 (588h21m44.76785799s) [noise=true , address=77.90.31.10:4519 , piece_id=TP2IFLWBFSJLWEXKJFIH7HDWXU3PXREOCVEPRALIZHPZLP52R5BQ, seg_pos={0 0}, upload=515778]: Uploading piece... Dec 30 03:50:44.896948726 (588h21m44.768031771s) [noise=true , address=45.91.116.9:4264 , piece_id=Y6W4IEFPQE2DZGUULIJGZ6RQU7LY5R6XNDQWWJTKBUYDCYQA4BVQ, seg_pos={0 0}, upload=515778]: Uploading piece... Dec 30 03:50:44.897151773 (588h21m44.768234808s) [noise=true , address=77.90.31.17:4770 , piece_id=BGRIQ2C32XKNT7TYHITIPRIN7LDMA7ABWMPLAKXGERGCUAZ5XSBA, seg_pos={0 0}, upload=515778]: Uploading piece... Dec 30 03:50:44.897351452 (588h21m44.768434497s) [noise=true , address=145.79.227.15:4874 , piece_id=CGX2M5WBQFAPMFUIG537L7572MTR5OU6CCF5TB64ANIAHJO3RGTQ, seg_pos={0 0}, upload=515778]: Uploading piece... Dec 30 03:50:44.897531506 (588h21m44.768614541s) [noise=true , address=145.79.227.11:4729 , piece_id=HQTRNF7SZ6TLONQC43SECGWWG6EH5UGSSZYI65IZJQLGJF7CTIWQ, seg_pos={0 0}, upload=515778]: Uploading piece... Dec 30 03:50:44.897752687 (588h21m44.768835722s) [noise=true , address=45.91.116.6:4973 , piece_id=MULDAZEZVIZMW6UXEZNHM2ZF762GQWXYOJBHOEFJZEF2MEUSX2IA, seg_pos={0 0}, upload=515778]: Uploading piece... Dec 30 03:50:44.898270172 (588h21m44.769353207s) [noise=true , address=77.90.31.14:4664 , piece_id=VEWPZRCZ3GHP4SVVH36V6ZLNZDRTCAQ6Y5ZMAW7PMLDM7KAOKDPQ, seg_pos={0 0}, upload=515778]: Uploading piece... Dec 30 03:50:44.898816181 (588h21m44.769899216s) [noise=true , address=145.79.227.46:4601 , piece_id=2NKPLJBTYPULRJHRZDUXZVNN6GUXN7UO6Y7G3DYQSFAWFUGICPMA, seg_pos={0 0}, upload=515778]: Uploading piece... Dec 30 03:50:44.899286587 (588h21m44.770369623s) [noise=true , address=45.91.116.7:4011 , piece_id=ADUSM7OBTNGRJOBAE2CTNM2FVDLTXHDV6ZANA3MQNTEOMJQPR7KA, seg_pos={0 0}, upload=515778]: Uploading piece... Dec 30 03:50:44.899559066 (588h21m44.770642102s) [noise=true , address=145.79.227.42:4121 , piece_id=NMQUIVFOXQXNO2HKCYMGV4KZ4UX6AP2RYY3U2JLXNCYKQWCUCP6A, seg_pos={0 0}, upload=515778]: Uploading piece... Dec 30 03:50:46.499996098 (588h21m46.371079133s) [upload=515778]: Next returned nil segment Dec 30 03:50:46.500006378 (588h21m46.371089413s) [upload=515778]: Waiting for error group managing segments... Dec 30 03:50:46.878577074 (588h21m46.749660109s) [noise=true , address=145.79.227.15:4874 , piece_id=CGX2M5WBQFAPMFUIG537L7572MTR5OU6CCF5TB64ANIAHJO3RGTQ, seg_pos={0 0}, upload=515778]: Done uploading piece. err: <nil> Dec 30 03:50:46.880225402 (588h21m46.751308447s) [noise=true , address=145.79.227.26:4655 , piece_id=F4PIAKSTVFOWKZ2DFSDZTSSVB7EHT4RAGO4I5GCNV6I5NI5OBEEA, seg_pos={0 0}, upload=515778]: Done uploading piece. err: <nil> Dec 30 03:50:46.880960001 (588h21m46.752043036s) [noise=true , address=145.79.227.7:4294 , piece_id=7CBO7REVWCP4LUQZ57QV2OU7BGE4RHDYBOK5G6PI7BJXJAFFUZXA, seg_pos={0 0}, upload=515778]: Done uploading piece. err: <nil> Dec 30 03:50:46.883262505 (588h21m46.75434554s) [noise=true , address=145.79.227.13:4804 , piece_id=LG4FAVRK2PWRX2TGHA7M3VWWGWPE5AHCI7CTZ4LLGHJ3QCMIICMA, seg_pos={0 0}, upload=515778]: Done uploading piece. err: <nil> Dec 30 03:50:46.884633585 (588h21m46.755716621s) [noise=true , address=145.79.227.11:4729 , piece_id=HQTRNF7SZ6TLONQC43SECGWWG6EH5UGSSZYI65IZJQLGJF7CTIWQ, seg_pos={0 0}, upload=515778]: Done uploading piece. err: <nil> Dec 30 03:50:46.890460090 (588h21m46.761543125s) [noise=true , address=145.79.227.46:4601 , piece_id=2NKPLJBTYPULRJHRZDUXZVNN6GUXN7UO6Y7G3DYQSFAWFUGICPMA, seg_pos={0 0}, upload=515778]: Done uploading piece. err: <nil> Dec 30 03:50:46.900643742 (588h21m46.771726787s) [noise=true , address=145.79.227.51:4411 , piece_id=ET37UNKINHDW4XSBHKPXWG2OFZQU6LW23UKCZCHHWQQJGWYCHOSA, seg_pos={0 0}, upload=515778]: Done uploading piece. err: <nil> Dec 30 03:50:46.911179513 (588h21m46.782262568s) [noise=true , address=145.79.227.42:4121 , piece_id=NMQUIVFOXQXNO2HKCYMGV4KZ4UX6AP2RYY3U2JLXNCYKQWCUCP6A, seg_pos={0 0}, upload=515778]: Done uploading piece. err: <nil> Dec 30 03:50:47.145138661 (588h21m47.016221696s) [noise=true , address=45.91.116.7:4011 , piece_id=ADUSM7OBTNGRJOBAE2CTNM2FVDLTXHDV6ZANA3MQNTEOMJQPR7KA, seg_pos={0 0}, upload=515778]: Done uploading piece. err: <nil> Dec 30 03:50:47.175069272 (588h21m47.046152307s) [noise=true , address=77.90.31.14:4664 , piece_id=VEWPZRCZ3GHP4SVVH36V6ZLNZDRTCAQ6Y5ZMAW7PMLDM7KAOKDPQ, seg_pos={0 0}, upload=515778]: Done uploading piece. err: <nil> Dec 30 03:50:47.178001555 (588h21m47.0490846s) [noise=true , address=77.90.31.10:4519 , piece_id=TP2IFLWBFSJLWEXKJFIH7HDWXU3PXREOCVEPRALIZHPZLP52R5BQ, seg_pos={0 0}, upload=515778]: Done uploading piece. err: <nil> Dec 30 03:50:47.281589540 (588h21m47.152672575s) [noise=true , address=77.90.31.17:4770 , piece_id=BGRIQ2C32XKNT7TYHITIPRIN7LDMA7ABWMPLAKXGERGCUAZ5XSBA, seg_pos={0 0}, upload=515778]: Done uploading piece. err: <nil> Dec 30 03:50:47.361085621 (588h21m47.232168656s) [noise=true , address=45.91.116.6:4973 , piece_id=MULDAZEZVIZMW6UXEZNHM2ZF762GQWXYOJBHOEFJZEF2MEUSX2IA, seg_pos={0 0}, upload=515778]: Done uploading piece. err: <nil> Dec 30 03:50:47.361104838 (588h21m47.232187883s) [seg_pos={0 0}, upload=515778]: Segment reached optimal threshold of 13 pieces. Dec 30 03:50:47.361259031 (588h21m47.232342066s) [noise=true , address=145.79.227.2:4115 , piece_id=H5FAMB5RLMZ3FO6ZMTV6GXCIML5LU5SLSDR2GNH5DCIR7IHAICXA, seg_pos={0 0}, upload=515778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:40066->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:40066->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40066->145.79.227.2:4115: use of closed network connection Dec 30 03:50:47.361264772 (588h21m47.232347807s) [noise=true , address=145.79.227.2:4115 , piece_id=H5FAMB5RLMZ3FO6ZMTV6GXCIML5LU5SLSDR2GNH5DCIR7IHAICXA, seg_pos={0 0}, upload=515778]: Long tail cancellation due to optimal threshold Dec 30 03:50:47.361281364 (588h21m47.232364399s) [noise=true , address=45.91.116.9:4264 , piece_id=Y6W4IEFPQE2DZGUULIJGZ6RQU7LY5R6XNDQWWJTKBUYDCYQA4BVQ, seg_pos={0 0}, upload=515778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: write tcp 188.166.225.170:43534->45.91.116.9:4264: use of closed network connection; write tcp 188.166.225.170:43534->45.91.116.9:4264: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43534->45.91.116.9:4264: use of closed network connection Dec 30 03:50:47.361290140 (588h21m47.232373175s) [noise=true , address=45.91.116.9:4264 , piece_id=Y6W4IEFPQE2DZGUULIJGZ6RQU7LY5R6XNDQWWJTKBUYDCYQA4BVQ, seg_pos={0 0}, upload=515778]: Long tail cancellation due to optimal threshold Dec 30 03:50:47.361351367 (588h21m47.232434412s) [noise=true , address=77.90.31.19:4844 , piece_id=RDDQDSIBCJRUGXUNACXWJQPACPOMDAV34BAAAH7WTNXYMLRCI34A, seg_pos={0 0}, upload=515778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: write tcp 188.166.225.170:34786->77.90.31.19:4844: use of closed network connection; write tcp 188.166.225.170:34786->77.90.31.19:4844: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34786->77.90.31.19:4844: use of closed network connection Dec 30 03:50:47.361356967 (588h21m47.232440002s) [noise=true , address=77.90.31.19:4844 , piece_id=RDDQDSIBCJRUGXUNACXWJQPACPOMDAV34BAAAH7WTNXYMLRCI34A, seg_pos={0 0}, upload=515778]: Long tail cancellation due to optimal threshold Dec 30 03:50:47.361482696 (588h21m47.232565731s) [seg_pos={0 0}, upload=515778]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 03:50:47.361547339 (588h21m47.232630375s) [upload=515778]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 03:50:47.361553822 (588h21m47.232636857s) [upload=515778]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 03:50:47.665512908 (588h21m47.536595944s) [upload=515778]: Done uploading segments... Dec 30 03:50:47.665535943 (588h21m47.536618978s) [upload=515778]: Upload finished. err: <nil> Duration:* 5.690660618s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 40.088194458s trace: https://jaeger.dev.storj.io/trace/1d81d68a7c17f75d FAIL (2025-12-30 03:53:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 15 , piece_id=6FWCMOTNEZLDIH2F5VRBC3H44WVYAKWTUIL364QUAIC7DGKURRSQ, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.829621721 (588h22m39.700704767s) [noise=true , address=193.149.164.143:4695 , piece_id=C3ZXZXKRGAI2FATBQ2PNRJFNGMV4XKBXU4MSKZVM2AEEH5GVBRBQ, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.829918227 (588h22m39.701001282s) [noise=true , address=145.79.227.17:4998 , piece_id=IMFG6HRRWRXUX3F76A5RTU54HXCFSTPTWH4YJQPHGIOMPALNS5AA, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.830292459 (588h22m39.701375504s) [noise=true , address=45.91.116.12:4249 , piece_id=YLNIBVQGVGZ5CYEZSGLORL3D2ULBBCXN23SSNXUU7LQ67E3E4G7A, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.830455549 (588h22m39.701538594s) [noise=true , address=145.79.227.43:4178 , piece_id=7DIUURHOKK53VG6RMELJUV3D3AWUM23SHXNNVDRFTIURMXETX6QQ, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.830740912 (588h22m39.701823948s) [noise=true , address=77.90.31.16:4759 , piece_id=W3RFMAKOXSYIDCA27H7CKEXSIKCMRUO72NITXH5WDM4FFK6XPDAQ, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.830892231 (588h22m39.701975266s) [noise=true , address=145.79.227.45:4250 , piece_id=MDFRH2T6ZDAOE25DJTGHA6YSXS6CRL5OBES5T2FDTGY6EOV7LINQ, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.831333561 (588h22m39.702416606s) [noise=true , address=145.79.227.40:4053 , piece_id=AKAAPJDTKY3BB2OB5EFW4IYMYPOAPL4VDS5VEHJF3U3TA7JHZ4FQ, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.831388847 (588h22m39.702471882s) [noise=true , address=145.79.227.34:4916 , piece_id=DMOYHOUOVSMTMPZBY7MV5LYQ5JQDV4MKRJHBH7FYV2TBL37AFYSA, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.831789770 (588h22m39.702872815s) [noise=true , address=64.49.13.26:4946 , piece_id=X7CGAXO4356EG4I52QB5LQEM33UD2FJFS56YFEH7FNHAG2EBEZFA, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.832180875 (588h22m39.70326391s) [noise=true , address=77.90.31.19:4871 , piece_id=HPPBTDGVLD62WLHSPO36VFYHP5YFORYCBT22PK2CEYRJJM4UQLYA, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.832576678 (588h22m39.703659723s) [noise=true , address=145.79.227.33:4901 , piece_id=ODRNWAOOWKB45HREGAAKMJYNIN2I4OPRLUN5BIVIW7HWFQRTR6NA, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.834204918 (588h22m39.705287963s) [noise=true , address=45.91.116.13:4164 , piece_id=QW7FBIO24JGOXJEGO7YHYFMMTKHAKF64UEGNDXX2KKQLSJ4MH3PA, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.834999141 (588h22m39.706082186s) [noise=true , address=77.90.31.11:4584 , piece_id=WKRB35UKP7L6Z6G5WSJKMFBOF3RPWJYVZJPFBP5G4UT4RQVQP6MA, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.835718220 (588h22m39.706801265s) [noise=true , address=145.79.227.49:4361 , piece_id=ARLKW5A6Y3DVQ25E2RDL5KRFCYBNVIW7WBDL7ZO2WOQDTF5RCRKQ, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:41.118730125 (588h22m40.98981316s) [upload=515790]: Next returned nil segment Dec 30 03:51:41.118750513 (588h22m40.989833558s) [upload=515790]: Waiting for error group managing segments... Dec 30 03:51:41.505594353 (588h22m41.376677388s) [noise=true , address=64.49.13.26:4946 , piece_id=X7CGAXO4356EG4I52QB5LQEM33UD2FJFS56YFEH7FNHAG2EBEZFA, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:41.711421644 (588h22m41.582504689s) [noise=true , address=145.79.227.17:4998 , piece_id=IMFG6HRRWRXUX3F76A5RTU54HXCFSTPTWH4YJQPHGIOMPALNS5AA, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:41.718129295 (588h22m41.58921233s) [noise=true , address=145.79.227.6:4285 , piece_id=WNJGANKULJAMPLIT3KTHRSSAYTUSIIUCW5ZVKAFNURMGNZ3N2P3A, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:41.729197852 (588h22m41.600280887s) [noise=true , address=145.79.227.14:4856 , piece_id=CBZ6JFWGDMBMR7XJBXIETWDBEEXV43HJWJFKH5FY5BF7ANACFDOQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:41.867535754 (588h22m41.738618799s) [noise=true , address=145.79.227.43:4178 , piece_id=7DIUURHOKK53VG6RMELJUV3D3AWUM23SHXNNVDRFTIURMXETX6QQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:41.894207901 (588h22m41.765290946s) [noise=true , address=77.90.31.12:4615 , piece_id=6FWCMOTNEZLDIH2F5VRBC3H44WVYAKWTUIL364QUAIC7DGKURRSQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:41.928246715 (588h22m41.79932975s) [noise=true , address=145.79.227.45:4250 , piece_id=MDFRH2T6ZDAOE25DJTGHA6YSXS6CRL5OBES5T2FDTGY6EOV7LINQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:41.940833472 (588h22m41.811916517s) [noise=true , address=45.91.116.13:4164 , piece_id=QW7FBIO24JGOXJEGO7YHYFMMTKHAKF64UEGNDXX2KKQLSJ4MH3PA, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:42.318460461 (588h22m42.189543506s) [noise=true , address=77.90.31.16:4759 , piece_id=W3RFMAKOXSYIDCA27H7CKEXSIKCMRUO72NITXH5WDM4FFK6XPDAQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:42.367315176 (588h22m42.238398221s) [noise=true , address=77.90.31.19:4871 , piece_id=HPPBTDGVLD62WLHSPO36VFYHP5YFORYCBT22PK2CEYRJJM4UQLYA, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:42.536538860 (588h22m42.407621905s) [noise=true , address=145.79.227.40:4053 , piece_id=AKAAPJDTKY3BB2OB5EFW4IYMYPOAPL4VDS5VEHJF3U3TA7JHZ4FQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:42.637143172 (588h22m42.508226217s) [noise=true , address=145.79.227.44:4200 , piece_id=2KRTBC4EQE7OCNJJ7TPM72KJFYBBXFHJUP2FRENF6KL3CQNAAUQQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:42.715818140 (588h22m42.586901185s) [noise=true , address=45.91.116.12:4249 , piece_id=YLNIBVQGVGZ5CYEZSGLORL3D2ULBBCXN23SSNXUU7LQ67E3E4G7A, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:43.127299661 (588h22m42.998382696s) [noise=true , address=145.79.227.34:4916 , piece_id=DMOYHOUOVSMTMPZBY7MV5LYQ5JQDV4MKRJHBH7FYV2TBL37AFYSA, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:43.127318687 (588h22m42.998401722s) [seg_pos={0 0}, upload=515790]: Segment reached optimal threshold of 14 pieces. Dec 30 03:51:43.127508449 (588h22m42.998591484s) [noise=true , address=145.79.227.49:4361 , piece_id=ARLKW5A6Y3DVQ25E2RDL5KRFCYBNVIW7WBDL7ZO2WOQDTF5RCRKQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k): protocol: expected piece hash; context canceled; EOF Dec 30 03:51:43.127514470 (588h22m42.998597505s) [noise=true , address=145.79.227.49:4361 , piece_id=ARLKW5A6Y3DVQ25E2RDL5KRFCYBNVIW7WBDL7ZO2WOQDTF5RCRKQ, seg_pos={0 0}, upload=515790]: Long tail cancellation due to optimal threshold Dec 30 03:51:43.127619871 (588h22m42.998702906s) [noise=true , address=77.90.31.11:4584 , piece_id=WKRB35UKP7L6Z6G5WSJKMFBOF3RPWJYVZJPFBP5G4UT4RQVQP6MA, seg_pos={0 0}, upload=515790]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iybcvQgVfqHcCSNFAEVpKZJyciSHbBfeAiuk3uNE2LGq4aLfJ): protocol: write tcp 188.166.225.170:43330->77.90.31.11:4584: use of closed network connection; write tcp 188.166.225.170:43330->77.90.31.11:4584: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43330->77.90.31.11:4584: use of closed network connection Dec 30 03:51:43.127624550 (588h22m42.998707585s) [noise=true , address=77.90.31.11:4584 , piece_id=WKRB35UKP7L6Z6G5WSJKMFBOF3RPWJYVZJPFBP5G4UT4RQVQP6MA, seg_pos={0 0}, upload=515790]: Long tail cancellation due to optimal threshold Dec 30 03:51:43.127738065 (588h22m42.998821111s) [noise=true , address=193.149.164.143:4695 , piece_id=C3ZXZXKRGAI2FATBQ2PNRJFNGMV4XKBXU4MSKZVM2AEEH5GVBRBQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh): protocol: expected piece hash; context canceled; EOF Dec 30 03:51:43.127743557 (588h22m42.998826592s) [noise=true , address=193.149.164.143:4695 , piece_id=C3ZXZXKRGAI2FATBQ2PNRJFNGMV4XKBXU4MSKZVM2AEEH5GVBRBQ, seg_pos={0 0}, upload=515790]: Long tail cancellation due to optimal threshold Dec 30 03:51:43.127899784 (588h22m42.998982829s) [noise=true , address=145.79.227.33:4901 , piece_id=ODRNWAOOWKB45HREGAAKMJYNIN2I4OPRLUN5BIVIW7HWFQRTR6NA, seg_pos={0 0}, upload=515790]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1): protocol: write tcp 188.166.225.170:48420->145.79.227.33:4901: use of closed network connection; write tcp 188.166.225.170:48420->145.79.227.33:4901: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48420->145.79.227.33:4901: use of closed network connection Dec 30 03:51:43.127920484 (588h22m42.999003539s) [noise=true , address=145.79.227.33:4901 , piece_id=ODRNWAOOWKB45HREGAAKMJYNIN2I4OPRLUN5BIVIW7HWFQRTR6NA, seg_pos={0 0}, upload=515790]: Long tail cancellation due to optimal threshold Dec 30 03:51:43.128047735 (588h22m42.99913077s) [seg_pos={0 0}, upload=515790]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 03:51:43.128161222 (588h22m42.999244257s) [upload=515790]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 03:51:43.128174597 (588h22m42.999257642s) [upload=515790]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 03:51:43.415948996 (588h22m43.287032052s) [upload=515790]: Done uploading segments... Dec 30 03:51:43.415976829 (588h22m43.287059864s) [upload=515790]: Upload finished. err: <nil> Duration: 40.088210047s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [X.XX..X..X.X.X...XXX.X.....XXXX........XX.XX...XXXX...XXXX..] duration: 5.980965401s trace: https://jaeger.dev.storj.io/trace/29cbd316ab2b2965 FAIL (2025-12-30 03:53:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: BGRIQ2C32XKNT7TYHITIPRIN7LDMA7ABWMPLAKXGERGCUAZ5XSBA) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: MULDAZEZVIZMW6UXEZNHM2ZF762GQWXYOJBHOEFJZEF2MEUSX2IA) context deadline exceeded ======LOGS====== 646402225 (588h21m44.51748527s) [upload=515778]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 03:50:44.894588582 (588h21m44.765671617s) [seg_pos={0 0}, upload=515778]: Begin upload segment... Dec 30 03:50:44.894641924 (588h21m44.765724959s) [seg_pos={0 0}, upload=515778]: Done begin upload segment. Dec 30 03:50:44.894677161 (588h21m44.765760206s) [noise=true , address=145.79.227.51:4411 , piece_id=ET37UNKINHDW4XSBHKPXWG2OFZQU6LW23UKCZCHHWQQJGWYCHOSA, seg_pos={0 0}, upload=515778]: Uploading piece... Dec 30 03:50:44.894724260 (588h21m44.765807295s) [noise=true , address=145.79.227.26:4655 , piece_id=F4PIAKSTVFOWKZ2DFSDZTSSVB7EHT4RAGO4I5GCNV6I5NI5OBEEA, seg_pos={0 0}, upload=515778]: Uploading piece... Dec 30 03:50:44.895676623 (588h21m44.766759668s) [noise=true , address=145.79.227.2:4115 , piece_id=H5FAMB5RLMZ3FO6ZMTV6GXCIML5LU5SLSDR2GNH5DCIR7IHAICXA, seg_pos={0 0}, upload=515778]: Uploading piece... Dec 30 03:50:44.896188809 (588h21m44.767271844s) [noise=true , address=77.90.31.19:4844 , piece_id=RDDQDSIBCJRUGXUNACXWJQPACPOMDAV34BAAAH7WTNXYMLRCI34A, seg_pos={0 0}, upload=515778]: Uploading piece... Dec 30 03:50:44.896308968 (588h21m44.767392003s) [noise=true , address=145.79.227.7:4294 , piece_id=7CBO7REVWCP4LUQZ57QV2OU7BGE4RHDYBOK5G6PI7BJXJAFFUZXA, seg_pos={0 0}, upload=515778]: Uploading piece... Dec 30 03:50:44.896584913 (588h21m44.767667958s) [noise=true , address=145.79.227.13:4804 , piece_id=LG4FAVRK2PWRX2TGHA7M3VWWGWPE5AHCI7CTZ4LLGHJ3QCMIICMA, seg_pos={0 0}, upload=515778]: Uploading piece... Dec 30 03:50:44.896774955 (588h21m44.76785799s) [noise=true , address=77.90.31.10:4519 , piece_id=TP2IFLWBFSJLWEXKJFIH7HDWXU3PXREOCVEPRALIZHPZLP52R5BQ, seg_pos={0 0}, upload=515778]: Uploading piece... Dec 30 03:50:44.896948726 (588h21m44.768031771s) [noise=true , address=45.91.116.9:4264 , piece_id=Y6W4IEFPQE2DZGUULIJGZ6RQU7LY5R6XNDQWWJTKBUYDCYQA4BVQ, seg_pos={0 0}, upload=515778]: Uploading piece... Dec 30 03:50:44.897151773 (588h21m44.768234808s) [noise=true , address=77.90.31.17:4770 , piece_id=BGRIQ2C32XKNT7TYHITIPRIN7LDMA7ABWMPLAKXGERGCUAZ5XSBA, seg_pos={0 0}, upload=515778]: Uploading piece... Dec 30 03:50:44.897351452 (588h21m44.768434497s) [noise=true , address=145.79.227.15:4874 , piece_id=CGX2M5WBQFAPMFUIG537L7572MTR5OU6CCF5TB64ANIAHJO3RGTQ, seg_pos={0 0}, upload=515778]: Uploading piece... Dec 30 03:50:44.897531506 (588h21m44.768614541s) [noise=true , address=145.79.227.11:4729 , piece_id=HQTRNF7SZ6TLONQC43SECGWWG6EH5UGSSZYI65IZJQLGJF7CTIWQ, seg_pos={0 0}, upload=515778]: Uploading piece... Dec 30 03:50:44.897752687 (588h21m44.768835722s) [noise=true , address=45.91.116.6:4973 , piece_id=MULDAZEZVIZMW6UXEZNHM2ZF762GQWXYOJBHOEFJZEF2MEUSX2IA, seg_pos={0 0}, upload=515778]: Uploading piece... Dec 30 03:50:44.898270172 (588h21m44.769353207s) [noise=true , address=77.90.31.14:4664 , piece_id=VEWPZRCZ3GHP4SVVH36V6ZLNZDRTCAQ6Y5ZMAW7PMLDM7KAOKDPQ, seg_pos={0 0}, upload=515778]: Uploading piece... Dec 30 03:50:44.898816181 (588h21m44.769899216s) [noise=true , address=145.79.227.46:4601 , piece_id=2NKPLJBTYPULRJHRZDUXZVNN6GUXN7UO6Y7G3DYQSFAWFUGICPMA, seg_pos={0 0}, upload=515778]: Uploading piece... Dec 30 03:50:44.899286587 (588h21m44.770369623s) [noise=true , address=45.91.116.7:4011 , piece_id=ADUSM7OBTNGRJOBAE2CTNM2FVDLTXHDV6ZANA3MQNTEOMJQPR7KA, seg_pos={0 0}, upload=515778]: Uploading piece... Dec 30 03:50:44.899559066 (588h21m44.770642102s) [noise=true , address=145.79.227.42:4121 , piece_id=NMQUIVFOXQXNO2HKCYMGV4KZ4UX6AP2RYY3U2JLXNCYKQWCUCP6A, seg_pos={0 0}, upload=515778]: Uploading piece... Dec 30 03:50:46.499996098 (588h21m46.371079133s) [upload=515778]: Next returned nil segment Dec 30 03:50:46.500006378 (588h21m46.371089413s) [upload=515778]: Waiting for error group managing segments... Dec 30 03:50:46.878577074 (588h21m46.749660109s) [noise=true , address=145.79.227.15:4874 , piece_id=CGX2M5WBQFAPMFUIG537L7572MTR5OU6CCF5TB64ANIAHJO3RGTQ, seg_pos={0 0}, upload=515778]: Done uploading piece. err: <nil> Dec 30 03:50:46.880225402 (588h21m46.751308447s) [noise=true , address=145.79.227.26:4655 , piece_id=F4PIAKSTVFOWKZ2DFSDZTSSVB7EHT4RAGO4I5GCNV6I5NI5OBEEA, seg_pos={0 0}, upload=515778]: Done uploading piece. err: <nil> Dec 30 03:50:46.880960001 (588h21m46.752043036s) [noise=true , address=145.79.227.7:4294 , piece_id=7CBO7REVWCP4LUQZ57QV2OU7BGE4RHDYBOK5G6PI7BJXJAFFUZXA, seg_pos={0 0}, upload=515778]: Done uploading piece. err: <nil> Dec 30 03:50:46.883262505 (588h21m46.75434554s) [noise=true , address=145.79.227.13:4804 , piece_id=LG4FAVRK2PWRX2TGHA7M3VWWGWPE5AHCI7CTZ4LLGHJ3QCMIICMA, seg_pos={0 0}, upload=515778]: Done uploading piece. err: <nil> Dec 30 03:50:46.884633585 (588h21m46.755716621s) [noise=true , address=145.79.227.11:4729 , piece_id=HQTRNF7SZ6TLONQC43SECGWWG6EH5UGSSZYI65IZJQLGJF7CTIWQ, seg_pos={0 0}, upload=515778]: Done uploading piece. err: <nil> Dec 30 03:50:46.890460090 (588h21m46.761543125s) [noise=true , address=145.79.227.46:4601 , piece_id=2NKPLJBTYPULRJHRZDUXZVNN6GUXN7UO6Y7G3DYQSFAWFUGICPMA, seg_pos={0 0}, upload=515778]: Done uploading piece. err: <nil> Dec 30 03:50:46.900643742 (588h21m46.771726787s) [noise=true , address=145.79.227.51:4411 , piece_id=ET37UNKINHDW4XSBHKPXWG2OFZQU6LW23UKCZCHHWQQJGWYCHOSA, seg_pos={0 0}, upload=515778]: Done uploading piece. err: <nil> Dec 30 03:50:46.911179513 (588h21m46.782262568s) [noise=true , address=145.79.227.42:4121 , piece_id=NMQUIVFOXQXNO2HKCYMGV4KZ4UX6AP2RYY3U2JLXNCYKQWCUCP6A, seg_pos={0 0}, upload=515778]: Done uploading piece. err: <nil> Dec 30 03:50:47.145138661 (588h21m47.016221696s) [noise=true , address=45.91.116.7:4011 , piece_id=ADUSM7OBTNGRJOBAE2CTNM2FVDLTXHDV6ZANA3MQNTEOMJQPR7KA, seg_pos={0 0}, upload=515778]: Done uploading piece. err: <nil> Dec 30 03:50:47.175069272 (588h21m47.046152307s) [noise=true , address=77.90.31.14:4664 , piece_id=VEWPZRCZ3GHP4SVVH36V6ZLNZDRTCAQ6Y5ZMAW7PMLDM7KAOKDPQ, seg_pos={0 0}, upload=515778]: Done uploading piece. err: <nil> Dec 30 03:50:47.178001555 (588h21m47.0490846s) [noise=true , address=77.90.31.10:4519 , piece_id=TP2IFLWBFSJLWEXKJFIH7HDWXU3PXREOCVEPRALIZHPZLP52R5BQ, seg_pos={0 0}, upload=515778]: Done uploading piece. err: <nil> Dec 30 03:50:47.281589540 (588h21m47.152672575s) [noise=true , address=77.90.31.17:4770 , piece_id=BGRIQ2C32XKNT7TYHITIPRIN7LDMA7ABWMPLAKXGERGCUAZ5XSBA, seg_pos={0 0}, upload=515778]: Done uploading piece. err: <nil> Dec 30 03:50:47.361085621 (588h21m47.232168656s) [noise=true , address=45.91.116.6:4973 , piece_id=MULDAZEZVIZMW6UXEZNHM2ZF762GQWXYOJBHOEFJZEF2MEUSX2IA, seg_pos={0 0}, upload=515778]: Done uploading piece. err: <nil> Dec 30 03:50:47.361104838 (588h21m47.232187883s) [seg_pos={0 0}, upload=515778]: Segment reached optimal threshold of 13 pieces. Dec 30 03:50:47.361259031 (588h21m47.232342066s) [noise=true , address=145.79.227.2:4115 , piece_id=H5FAMB5RLMZ3FO6ZMTV6GXCIML5LU5SLSDR2GNH5DCIR7IHAICXA, seg_pos={0 0}, upload=515778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:40066->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:40066->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40066->145.79.227.2:4115: use of closed network connection Dec 30 03:50:47.361264772 (588h21m47.232347807s) [noise=true , address=145.79.227.2:4115 , piece_id=H5FAMB5RLMZ3FO6ZMTV6GXCIML5LU5SLSDR2GNH5DCIR7IHAICXA, seg_pos={0 0}, upload=515778]: Long tail cancellation due to optimal threshold Dec 30 03:50:47.361281364 (588h21m47.232364399s) [noise=true , address=45.91.116.9:4264 , piece_id=Y6W4IEFPQE2DZGUULIJGZ6RQU7LY5R6XNDQWWJTKBUYDCYQA4BVQ, seg_pos={0 0}, upload=515778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: write tcp 188.166.225.170:43534->45.91.116.9:4264: use of closed network connection; write tcp 188.166.225.170:43534->45.91.116.9:4264: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43534->45.91.116.9:4264: use of closed network connection Dec 30 03:50:47.361290140 (588h21m47.232373175s) [noise=true , address=45.91.116.9:4264 , piece_id=Y6W4IEFPQE2DZGUULIJGZ6RQU7LY5R6XNDQWWJTKBUYDCYQA4BVQ, seg_pos={0 0}, upload=515778]: Long tail cancellation due to optimal threshold Dec 30 03:50:47.361351367 (588h21m47.232434412s) [noise=true , address=77.90.31.19:4844 , piece_id=RDDQDSIBCJRUGXUNACXWJQPACPOMDAV34BAAAH7WTNXYMLRCI34A, seg_pos={0 0}, upload=515778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: write tcp 188.166.225.170:34786->77.90.31.19:4844: use of closed network connection; write tcp 188.166.225.170:34786->77.90.31.19:4844: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34786->77.90.31.19:4844: use of closed network connection Dec 30 03:50:47.361356967 (588h21m47.232440002s) [noise=true , address=77.90.31.19:4844 , piece_id=RDDQDSIBCJRUGXUNACXWJQPACPOMDAV34BAAAH7WTNXYMLRCI34A, seg_pos={0 0}, upload=515778]: Long tail cancellation due to optimal threshold Dec 30 03:50:47.361482696 (588h21m47.232565731s) [seg_pos={0 0}, upload=515778]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 03:50:47.361547339 (588h21m47.232630375s) [upload=515778]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 03:50:47.361553822 (588h21m47.232636857s) [upload=515778]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 03:50:47.665512908 (588h21m47.536595944s) [upload=515778]: Done uploading segments... Dec 30 03:50:47.665535943 (588h21m47.536618978s) [upload=515778]: Upload finished. err: <nil> Duration:* 5.980997112s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [..........X....X.X..........XX...X.......................X..] duration: 3.517872577s trace: https://jaeger.dev.storj.io/trace/51699b18f550d6ea FAIL (2025-12-30 03:52:41 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.518307837s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.003347632s trace: https://jaeger.dev.storj.io/trace/1d81d68a7c17f75d FAIL (2025-12-30 03:52:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 15 , piece_id=6FWCMOTNEZLDIH2F5VRBC3H44WVYAKWTUIL364QUAIC7DGKURRSQ, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.829621721 (588h22m39.700704767s) [noise=true , address=193.149.164.143:4695 , piece_id=C3ZXZXKRGAI2FATBQ2PNRJFNGMV4XKBXU4MSKZVM2AEEH5GVBRBQ, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.829918227 (588h22m39.701001282s) [noise=true , address=145.79.227.17:4998 , piece_id=IMFG6HRRWRXUX3F76A5RTU54HXCFSTPTWH4YJQPHGIOMPALNS5AA, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.830292459 (588h22m39.701375504s) [noise=true , address=45.91.116.12:4249 , piece_id=YLNIBVQGVGZ5CYEZSGLORL3D2ULBBCXN23SSNXUU7LQ67E3E4G7A, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.830455549 (588h22m39.701538594s) [noise=true , address=145.79.227.43:4178 , piece_id=7DIUURHOKK53VG6RMELJUV3D3AWUM23SHXNNVDRFTIURMXETX6QQ, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.830740912 (588h22m39.701823948s) [noise=true , address=77.90.31.16:4759 , piece_id=W3RFMAKOXSYIDCA27H7CKEXSIKCMRUO72NITXH5WDM4FFK6XPDAQ, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.830892231 (588h22m39.701975266s) [noise=true , address=145.79.227.45:4250 , piece_id=MDFRH2T6ZDAOE25DJTGHA6YSXS6CRL5OBES5T2FDTGY6EOV7LINQ, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.831333561 (588h22m39.702416606s) [noise=true , address=145.79.227.40:4053 , piece_id=AKAAPJDTKY3BB2OB5EFW4IYMYPOAPL4VDS5VEHJF3U3TA7JHZ4FQ, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.831388847 (588h22m39.702471882s) [noise=true , address=145.79.227.34:4916 , piece_id=DMOYHOUOVSMTMPZBY7MV5LYQ5JQDV4MKRJHBH7FYV2TBL37AFYSA, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.831789770 (588h22m39.702872815s) [noise=true , address=64.49.13.26:4946 , piece_id=X7CGAXO4356EG4I52QB5LQEM33UD2FJFS56YFEH7FNHAG2EBEZFA, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.832180875 (588h22m39.70326391s) [noise=true , address=77.90.31.19:4871 , piece_id=HPPBTDGVLD62WLHSPO36VFYHP5YFORYCBT22PK2CEYRJJM4UQLYA, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.832576678 (588h22m39.703659723s) [noise=true , address=145.79.227.33:4901 , piece_id=ODRNWAOOWKB45HREGAAKMJYNIN2I4OPRLUN5BIVIW7HWFQRTR6NA, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.834204918 (588h22m39.705287963s) [noise=true , address=45.91.116.13:4164 , piece_id=QW7FBIO24JGOXJEGO7YHYFMMTKHAKF64UEGNDXX2KKQLSJ4MH3PA, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.834999141 (588h22m39.706082186s) [noise=true , address=77.90.31.11:4584 , piece_id=WKRB35UKP7L6Z6G5WSJKMFBOF3RPWJYVZJPFBP5G4UT4RQVQP6MA, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:39.835718220 (588h22m39.706801265s) [noise=true , address=145.79.227.49:4361 , piece_id=ARLKW5A6Y3DVQ25E2RDL5KRFCYBNVIW7WBDL7ZO2WOQDTF5RCRKQ, seg_pos={0 0}, upload=515790]: Uploading piece... Dec 30 03:51:41.118730125 (588h22m40.98981316s) [upload=515790]: Next returned nil segment Dec 30 03:51:41.118750513 (588h22m40.989833558s) [upload=515790]: Waiting for error group managing segments... Dec 30 03:51:41.505594353 (588h22m41.376677388s) [noise=true , address=64.49.13.26:4946 , piece_id=X7CGAXO4356EG4I52QB5LQEM33UD2FJFS56YFEH7FNHAG2EBEZFA, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:41.711421644 (588h22m41.582504689s) [noise=true , address=145.79.227.17:4998 , piece_id=IMFG6HRRWRXUX3F76A5RTU54HXCFSTPTWH4YJQPHGIOMPALNS5AA, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:41.718129295 (588h22m41.58921233s) [noise=true , address=145.79.227.6:4285 , piece_id=WNJGANKULJAMPLIT3KTHRSSAYTUSIIUCW5ZVKAFNURMGNZ3N2P3A, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:41.729197852 (588h22m41.600280887s) [noise=true , address=145.79.227.14:4856 , piece_id=CBZ6JFWGDMBMR7XJBXIETWDBEEXV43HJWJFKH5FY5BF7ANACFDOQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:41.867535754 (588h22m41.738618799s) [noise=true , address=145.79.227.43:4178 , piece_id=7DIUURHOKK53VG6RMELJUV3D3AWUM23SHXNNVDRFTIURMXETX6QQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:41.894207901 (588h22m41.765290946s) [noise=true , address=77.90.31.12:4615 , piece_id=6FWCMOTNEZLDIH2F5VRBC3H44WVYAKWTUIL364QUAIC7DGKURRSQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:41.928246715 (588h22m41.79932975s) [noise=true , address=145.79.227.45:4250 , piece_id=MDFRH2T6ZDAOE25DJTGHA6YSXS6CRL5OBES5T2FDTGY6EOV7LINQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:41.940833472 (588h22m41.811916517s) [noise=true , address=45.91.116.13:4164 , piece_id=QW7FBIO24JGOXJEGO7YHYFMMTKHAKF64UEGNDXX2KKQLSJ4MH3PA, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:42.318460461 (588h22m42.189543506s) [noise=true , address=77.90.31.16:4759 , piece_id=W3RFMAKOXSYIDCA27H7CKEXSIKCMRUO72NITXH5WDM4FFK6XPDAQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:42.367315176 (588h22m42.238398221s) [noise=true , address=77.90.31.19:4871 , piece_id=HPPBTDGVLD62WLHSPO36VFYHP5YFORYCBT22PK2CEYRJJM4UQLYA, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:42.536538860 (588h22m42.407621905s) [noise=true , address=145.79.227.40:4053 , piece_id=AKAAPJDTKY3BB2OB5EFW4IYMYPOAPL4VDS5VEHJF3U3TA7JHZ4FQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:42.637143172 (588h22m42.508226217s) [noise=true , address=145.79.227.44:4200 , piece_id=2KRTBC4EQE7OCNJJ7TPM72KJFYBBXFHJUP2FRENF6KL3CQNAAUQQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:42.715818140 (588h22m42.586901185s) [noise=true , address=45.91.116.12:4249 , piece_id=YLNIBVQGVGZ5CYEZSGLORL3D2ULBBCXN23SSNXUU7LQ67E3E4G7A, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:43.127299661 (588h22m42.998382696s) [noise=true , address=145.79.227.34:4916 , piece_id=DMOYHOUOVSMTMPZBY7MV5LYQ5JQDV4MKRJHBH7FYV2TBL37AFYSA, seg_pos={0 0}, upload=515790]: Done uploading piece. err: <nil> Dec 30 03:51:43.127318687 (588h22m42.998401722s) [seg_pos={0 0}, upload=515790]: Segment reached optimal threshold of 14 pieces. Dec 30 03:51:43.127508449 (588h22m42.998591484s) [noise=true , address=145.79.227.49:4361 , piece_id=ARLKW5A6Y3DVQ25E2RDL5KRFCYBNVIW7WBDL7ZO2WOQDTF5RCRKQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k): protocol: expected piece hash; context canceled; EOF Dec 30 03:51:43.127514470 (588h22m42.998597505s) [noise=true , address=145.79.227.49:4361 , piece_id=ARLKW5A6Y3DVQ25E2RDL5KRFCYBNVIW7WBDL7ZO2WOQDTF5RCRKQ, seg_pos={0 0}, upload=515790]: Long tail cancellation due to optimal threshold Dec 30 03:51:43.127619871 (588h22m42.998702906s) [noise=true , address=77.90.31.11:4584 , piece_id=WKRB35UKP7L6Z6G5WSJKMFBOF3RPWJYVZJPFBP5G4UT4RQVQP6MA, seg_pos={0 0}, upload=515790]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iybcvQgVfqHcCSNFAEVpKZJyciSHbBfeAiuk3uNE2LGq4aLfJ): protocol: write tcp 188.166.225.170:43330->77.90.31.11:4584: use of closed network connection; write tcp 188.166.225.170:43330->77.90.31.11:4584: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43330->77.90.31.11:4584: use of closed network connection Dec 30 03:51:43.127624550 (588h22m42.998707585s) [noise=true , address=77.90.31.11:4584 , piece_id=WKRB35UKP7L6Z6G5WSJKMFBOF3RPWJYVZJPFBP5G4UT4RQVQP6MA, seg_pos={0 0}, upload=515790]: Long tail cancellation due to optimal threshold Dec 30 03:51:43.127738065 (588h22m42.998821111s) [noise=true , address=193.149.164.143:4695 , piece_id=C3ZXZXKRGAI2FATBQ2PNRJFNGMV4XKBXU4MSKZVM2AEEH5GVBRBQ, seg_pos={0 0}, upload=515790]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh): protocol: expected piece hash; context canceled; EOF Dec 30 03:51:43.127743557 (588h22m42.998826592s) [noise=true , address=193.149.164.143:4695 , piece_id=C3ZXZXKRGAI2FATBQ2PNRJFNGMV4XKBXU4MSKZVM2AEEH5GVBRBQ, seg_pos={0 0}, upload=515790]: Long tail cancellation due to optimal threshold Dec 30 03:51:43.127899784 (588h22m42.998982829s) [noise=true , address=145.79.227.33:4901 , piece_id=ODRNWAOOWKB45HREGAAKMJYNIN2I4OPRLUN5BIVIW7HWFQRTR6NA, seg_pos={0 0}, upload=515790]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1): protocol: write tcp 188.166.225.170:48420->145.79.227.33:4901: use of closed network connection; write tcp 188.166.225.170:48420->145.79.227.33:4901: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48420->145.79.227.33:4901: use of closed network connection Dec 30 03:51:43.127920484 (588h22m42.999003539s) [noise=true , address=145.79.227.33:4901 , piece_id=ODRNWAOOWKB45HREGAAKMJYNIN2I4OPRLUN5BIVIW7HWFQRTR6NA, seg_pos={0 0}, upload=515790]: Long tail cancellation due to optimal threshold Dec 30 03:51:43.128047735 (588h22m42.99913077s) [seg_pos={0 0}, upload=515790]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 03:51:43.128161222 (588h22m42.999244257s) [upload=515790]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 03:51:43.128174597 (588h22m42.999257642s) [upload=515790]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 03:51:43.415948996 (588h22m43.287032052s) [upload=515790]: Done uploading segments... Dec 30 03:51:43.415976829 (588h22m43.287059864s) [upload=515790]: Upload finished. err: <nil> Duration: 1m0.00338266s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [.X.XX..X..X.X.X...XXX.X.....XXXX........XX.XX...XXXX...XXXX.] duration: 22.499755024s trace: https://jaeger.dev.storj.io/trace/29cbd316ab2b2965 FAIL (2025-12-30 03:52:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: BGRIQ2C32XKNT7TYHITIPRIN7LDMA7ABWMPLAKXGERGCUAZ5XSBA) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: MULDAZEZVIZMW6UXEZNHM2ZF762GQWXYOJBHOEFJZEF2MEUSX2IA) context deadline exceeded ======LOGS====== 646402225 (588h21m44.51748527s) [upload=515778]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 03:50:44.894588582 (588h21m44.765671617s) [seg_pos={0 0}, upload=515778]: Begin upload segment... Dec 30 03:50:44.894641924 (588h21m44.765724959s) [seg_pos={0 0}, upload=515778]: Done begin upload segment. Dec 30 03:50:44.894677161 (588h21m44.765760206s) [noise=true , address=145.79.227.51:4411 , piece_id=ET37UNKINHDW4XSBHKPXWG2OFZQU6LW23UKCZCHHWQQJGWYCHOSA, seg_pos={0 0}, upload=515778]: Uploading piece... Dec 30 03:50:44.894724260 (588h21m44.765807295s) [noise=true , address=145.79.227.26:4655 , piece_id=F4PIAKSTVFOWKZ2DFSDZTSSVB7EHT4RAGO4I5GCNV6I5NI5OBEEA, seg_pos={0 0}, upload=515778]: Uploading piece... Dec 30 03:50:44.895676623 (588h21m44.766759668s) [noise=true , address=145.79.227.2:4115 , piece_id=H5FAMB5RLMZ3FO6ZMTV6GXCIML5LU5SLSDR2GNH5DCIR7IHAICXA, seg_pos={0 0}, upload=515778]: Uploading piece... Dec 30 03:50:44.896188809 (588h21m44.767271844s) [noise=true , address=77.90.31.19:4844 , piece_id=RDDQDSIBCJRUGXUNACXWJQPACPOMDAV34BAAAH7WTNXYMLRCI34A, seg_pos={0 0}, upload=515778]: Uploading piece... Dec 30 03:50:44.896308968 (588h21m44.767392003s) [noise=true , address=145.79.227.7:4294 , piece_id=7CBO7REVWCP4LUQZ57QV2OU7BGE4RHDYBOK5G6PI7BJXJAFFUZXA, seg_pos={0 0}, upload=515778]: Uploading piece... Dec 30 03:50:44.896584913 (588h21m44.767667958s) [noise=true , address=145.79.227.13:4804 , piece_id=LG4FAVRK2PWRX2TGHA7M3VWWGWPE5AHCI7CTZ4LLGHJ3QCMIICMA, seg_pos={0 0}, upload=515778]: Uploading piece... Dec 30 03:50:44.896774955 (588h21m44.76785799s) [noise=true , address=77.90.31.10:4519 , piece_id=TP2IFLWBFSJLWEXKJFIH7HDWXU3PXREOCVEPRALIZHPZLP52R5BQ, seg_pos={0 0}, upload=515778]: Uploading piece... Dec 30 03:50:44.896948726 (588h21m44.768031771s) [noise=true , address=45.91.116.9:4264 , piece_id=Y6W4IEFPQE2DZGUULIJGZ6RQU7LY5R6XNDQWWJTKBUYDCYQA4BVQ, seg_pos={0 0}, upload=515778]: Uploading piece... Dec 30 03:50:44.897151773 (588h21m44.768234808s) [noise=true , address=77.90.31.17:4770 , piece_id=BGRIQ2C32XKNT7TYHITIPRIN7LDMA7ABWMPLAKXGERGCUAZ5XSBA, seg_pos={0 0}, upload=515778]: Uploading piece... Dec 30 03:50:44.897351452 (588h21m44.768434497s) [noise=true , address=145.79.227.15:4874 , piece_id=CGX2M5WBQFAPMFUIG537L7572MTR5OU6CCF5TB64ANIAHJO3RGTQ, seg_pos={0 0}, upload=515778]: Uploading piece... Dec 30 03:50:44.897531506 (588h21m44.768614541s) [noise=true , address=145.79.227.11:4729 , piece_id=HQTRNF7SZ6TLONQC43SECGWWG6EH5UGSSZYI65IZJQLGJF7CTIWQ, seg_pos={0 0}, upload=515778]: Uploading piece... Dec 30 03:50:44.897752687 (588h21m44.768835722s) [noise=true , address=45.91.116.6:4973 , piece_id=MULDAZEZVIZMW6UXEZNHM2ZF762GQWXYOJBHOEFJZEF2MEUSX2IA, seg_pos={0 0}, upload=515778]: Uploading piece... Dec 30 03:50:44.898270172 (588h21m44.769353207s) [noise=true , address=77.90.31.14:4664 , piece_id=VEWPZRCZ3GHP4SVVH36V6ZLNZDRTCAQ6Y5ZMAW7PMLDM7KAOKDPQ, seg_pos={0 0}, upload=515778]: Uploading piece... Dec 30 03:50:44.898816181 (588h21m44.769899216s) [noise=true , address=145.79.227.46:4601 , piece_id=2NKPLJBTYPULRJHRZDUXZVNN6GUXN7UO6Y7G3DYQSFAWFUGICPMA, seg_pos={0 0}, upload=515778]: Uploading piece... Dec 30 03:50:44.899286587 (588h21m44.770369623s) [noise=true , address=45.91.116.7:4011 , piece_id=ADUSM7OBTNGRJOBAE2CTNM2FVDLTXHDV6ZANA3MQNTEOMJQPR7KA, seg_pos={0 0}, upload=515778]: Uploading piece... Dec 30 03:50:44.899559066 (588h21m44.770642102s) [noise=true , address=145.79.227.42:4121 , piece_id=NMQUIVFOXQXNO2HKCYMGV4KZ4UX6AP2RYY3U2JLXNCYKQWCUCP6A, seg_pos={0 0}, upload=515778]: Uploading piece... Dec 30 03:50:46.499996098 (588h21m46.371079133s) [upload=515778]: Next returned nil segment Dec 30 03:50:46.500006378 (588h21m46.371089413s) [upload=515778]: Waiting for error group managing segments... Dec 30 03:50:46.878577074 (588h21m46.749660109s) [noise=true , address=145.79.227.15:4874 , piece_id=CGX2M5WBQFAPMFUIG537L7572MTR5OU6CCF5TB64ANIAHJO3RGTQ, seg_pos={0 0}, upload=515778]: Done uploading piece. err: <nil> Dec 30 03:50:46.880225402 (588h21m46.751308447s) [noise=true , address=145.79.227.26:4655 , piece_id=F4PIAKSTVFOWKZ2DFSDZTSSVB7EHT4RAGO4I5GCNV6I5NI5OBEEA, seg_pos={0 0}, upload=515778]: Done uploading piece. err: <nil> Dec 30 03:50:46.880960001 (588h21m46.752043036s) [noise=true , address=145.79.227.7:4294 , piece_id=7CBO7REVWCP4LUQZ57QV2OU7BGE4RHDYBOK5G6PI7BJXJAFFUZXA, seg_pos={0 0}, upload=515778]: Done uploading piece. err: <nil> Dec 30 03:50:46.883262505 (588h21m46.75434554s) [noise=true , address=145.79.227.13:4804 , piece_id=LG4FAVRK2PWRX2TGHA7M3VWWGWPE5AHCI7CTZ4LLGHJ3QCMIICMA, seg_pos={0 0}, upload=515778]: Done uploading piece. err: <nil> Dec 30 03:50:46.884633585 (588h21m46.755716621s) [noise=true , address=145.79.227.11:4729 , piece_id=HQTRNF7SZ6TLONQC43SECGWWG6EH5UGSSZYI65IZJQLGJF7CTIWQ, seg_pos={0 0}, upload=515778]: Done uploading piece. err: <nil> Dec 30 03:50:46.890460090 (588h21m46.761543125s) [noise=true , address=145.79.227.46:4601 , piece_id=2NKPLJBTYPULRJHRZDUXZVNN6GUXN7UO6Y7G3DYQSFAWFUGICPMA, seg_pos={0 0}, upload=515778]: Done uploading piece. err: <nil> Dec 30 03:50:46.900643742 (588h21m46.771726787s) [noise=true , address=145.79.227.51:4411 , piece_id=ET37UNKINHDW4XSBHKPXWG2OFZQU6LW23UKCZCHHWQQJGWYCHOSA, seg_pos={0 0}, upload=515778]: Done uploading piece. err: <nil> Dec 30 03:50:46.911179513 (588h21m46.782262568s) [noise=true , address=145.79.227.42:4121 , piece_id=NMQUIVFOXQXNO2HKCYMGV4KZ4UX6AP2RYY3U2JLXNCYKQWCUCP6A, seg_pos={0 0}, upload=515778]: Done uploading piece. err: <nil> Dec 30 03:50:47.145138661 (588h21m47.016221696s) [noise=true , address=45.91.116.7:4011 , piece_id=ADUSM7OBTNGRJOBAE2CTNM2FVDLTXHDV6ZANA3MQNTEOMJQPR7KA, seg_pos={0 0}, upload=515778]: Done uploading piece. err: <nil> Dec 30 03:50:47.175069272 (588h21m47.046152307s) [noise=true , address=77.90.31.14:4664 , piece_id=VEWPZRCZ3GHP4SVVH36V6ZLNZDRTCAQ6Y5ZMAW7PMLDM7KAOKDPQ, seg_pos={0 0}, upload=515778]: Done uploading piece. err: <nil> Dec 30 03:50:47.178001555 (588h21m47.0490846s) [noise=true , address=77.90.31.10:4519 , piece_id=TP2IFLWBFSJLWEXKJFIH7HDWXU3PXREOCVEPRALIZHPZLP52R5BQ, seg_pos={0 0}, upload=515778]: Done uploading piece. err: <nil> Dec 30 03:50:47.281589540 (588h21m47.152672575s) [noise=true , address=77.90.31.17:4770 , piece_id=BGRIQ2C32XKNT7TYHITIPRIN7LDMA7ABWMPLAKXGERGCUAZ5XSBA, seg_pos={0 0}, upload=515778]: Done uploading piece. err: <nil> Dec 30 03:50:47.361085621 (588h21m47.232168656s) [noise=true , address=45.91.116.6:4973 , piece_id=MULDAZEZVIZMW6UXEZNHM2ZF762GQWXYOJBHOEFJZEF2MEUSX2IA, seg_pos={0 0}, upload=515778]: Done uploading piece. err: <nil> Dec 30 03:50:47.361104838 (588h21m47.232187883s) [seg_pos={0 0}, upload=515778]: Segment reached optimal threshold of 13 pieces. Dec 30 03:50:47.361259031 (588h21m47.232342066s) [noise=true , address=145.79.227.2:4115 , piece_id=H5FAMB5RLMZ3FO6ZMTV6GXCIML5LU5SLSDR2GNH5DCIR7IHAICXA, seg_pos={0 0}, upload=515778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:40066->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:40066->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40066->145.79.227.2:4115: use of closed network connection Dec 30 03:50:47.361264772 (588h21m47.232347807s) [noise=true , address=145.79.227.2:4115 , piece_id=H5FAMB5RLMZ3FO6ZMTV6GXCIML5LU5SLSDR2GNH5DCIR7IHAICXA, seg_pos={0 0}, upload=515778]: Long tail cancellation due to optimal threshold Dec 30 03:50:47.361281364 (588h21m47.232364399s) [noise=true , address=45.91.116.9:4264 , piece_id=Y6W4IEFPQE2DZGUULIJGZ6RQU7LY5R6XNDQWWJTKBUYDCYQA4BVQ, seg_pos={0 0}, upload=515778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: write tcp 188.166.225.170:43534->45.91.116.9:4264: use of closed network connection; write tcp 188.166.225.170:43534->45.91.116.9:4264: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43534->45.91.116.9:4264: use of closed network connection Dec 30 03:50:47.361290140 (588h21m47.232373175s) [noise=true , address=45.91.116.9:4264 , piece_id=Y6W4IEFPQE2DZGUULIJGZ6RQU7LY5R6XNDQWWJTKBUYDCYQA4BVQ, seg_pos={0 0}, upload=515778]: Long tail cancellation due to optimal threshold Dec 30 03:50:47.361351367 (588h21m47.232434412s) [noise=true , address=77.90.31.19:4844 , piece_id=RDDQDSIBCJRUGXUNACXWJQPACPOMDAV34BAAAH7WTNXYMLRCI34A, seg_pos={0 0}, upload=515778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: write tcp 188.166.225.170:34786->77.90.31.19:4844: use of closed network connection; write tcp 188.166.225.170:34786->77.90.31.19:4844: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34786->77.90.31.19:4844: use of closed network connection Dec 30 03:50:47.361356967 (588h21m47.232440002s) [noise=true , address=77.90.31.19:4844 , piece_id=RDDQDSIBCJRUGXUNACXWJQPACPOMDAV34BAAAH7WTNXYMLRCI34A, seg_pos={0 0}, upload=515778]: Long tail cancellation due to optimal threshold Dec 30 03:50:47.361482696 (588h21m47.232565731s) [seg_pos={0 0}, upload=515778]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 03:50:47.361547339 (588h21m47.232630375s) [upload=515778]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 03:50:47.361553822 (588h21m47.232636857s) [upload=515778]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 03:50:47.665512908 (588h21m47.536595944s) [upload=515778]: Done uploading segments... Dec 30 03:50:47.665535943 (588h21m47.536618978s) [upload=515778]: Upload finished. err: <nil> Duration:* 22.499776325s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [...........X....X.X..........XX...X.......................X.] duration: 3.939214486s trace: https://jaeger.dev.storj.io/trace/51699b18f550d6ea FAIL (2025-12-30 03:51:55 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.939241667s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [X.X.XX..X..X.X.X...XXX.X.....XXXX........XX.XX...XXXX...XXXX] duration: 30.003376489s trace: https://jaeger.dev.storj.io/trace/29cbd316ab2b2965 FAIL (2025-12-30 03:51:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: BGRIQ2C32XKNT7TYHITIPRIN7LDMA7ABWMPLAKXGERGCUAZ5XSBA) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: MULDAZEZVIZMW6UXEZNHM2ZF762GQWXYOJBHOEFJZEF2MEUSX2IA) context deadline exceeded ======LOGS====== 646402225 (588h21m44.51748527s) [upload=515778]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 03:50:44.894588582 (588h21m44.765671617s) [seg_pos={0 0}, upload=515778]: Begin upload segment... Dec 30 03:50:44.894641924 (588h21m44.765724959s) [seg_pos={0 0}, upload=515778]: Done begin upload segment. Dec 30 03:50:44.894677161 (588h21m44.765760206s) [noise=true , address=145.79.227.51:4411 , piece_id=ET37UNKINHDW4XSBHKPXWG2OFZQU6LW23UKCZCHHWQQJGWYCHOSA, seg_pos={0 0}, upload=515778]: Uploading piece... Dec 30 03:50:44.894724260 (588h21m44.765807295s) [noise=true , address=145.79.227.26:4655 , piece_id=F4PIAKSTVFOWKZ2DFSDZTSSVB7EHT4RAGO4I5GCNV6I5NI5OBEEA, seg_pos={0 0}, upload=515778]: Uploading piece... Dec 30 03:50:44.895676623 (588h21m44.766759668s) [noise=true , address=145.79.227.2:4115 , piece_id=H5FAMB5RLMZ3FO6ZMTV6GXCIML5LU5SLSDR2GNH5DCIR7IHAICXA, seg_pos={0 0}, upload=515778]: Uploading piece... Dec 30 03:50:44.896188809 (588h21m44.767271844s) [noise=true , address=77.90.31.19:4844 , piece_id=RDDQDSIBCJRUGXUNACXWJQPACPOMDAV34BAAAH7WTNXYMLRCI34A, seg_pos={0 0}, upload=515778]: Uploading piece... Dec 30 03:50:44.896308968 (588h21m44.767392003s) [noise=true , address=145.79.227.7:4294 , piece_id=7CBO7REVWCP4LUQZ57QV2OU7BGE4RHDYBOK5G6PI7BJXJAFFUZXA, seg_pos={0 0}, upload=515778]: Uploading piece... Dec 30 03:50:44.896584913 (588h21m44.767667958s) [noise=true , address=145.79.227.13:4804 , piece_id=LG4FAVRK2PWRX2TGHA7M3VWWGWPE5AHCI7CTZ4LLGHJ3QCMIICMA, seg_pos={0 0}, upload=515778]: Uploading piece... Dec 30 03:50:44.896774955 (588h21m44.76785799s) [noise=true , address=77.90.31.10:4519 , piece_id=TP2IFLWBFSJLWEXKJFIH7HDWXU3PXREOCVEPRALIZHPZLP52R5BQ, seg_pos={0 0}, upload=515778]: Uploading piece... Dec 30 03:50:44.896948726 (588h21m44.768031771s) [noise=true , address=45.91.116.9:4264 , piece_id=Y6W4IEFPQE2DZGUULIJGZ6RQU7LY5R6XNDQWWJTKBUYDCYQA4BVQ, seg_pos={0 0}, upload=515778]: Uploading piece... Dec 30 03:50:44.897151773 (588h21m44.768234808s) [noise=true , address=77.90.31.17:4770 , piece_id=BGRIQ2C32XKNT7TYHITIPRIN7LDMA7ABWMPLAKXGERGCUAZ5XSBA, seg_pos={0 0}, upload=515778]: Uploading piece... Dec 30 03:50:44.897351452 (588h21m44.768434497s) [noise=true , address=145.79.227.15:4874 , piece_id=CGX2M5WBQFAPMFUIG537L7572MTR5OU6CCF5TB64ANIAHJO3RGTQ, seg_pos={0 0}, upload=515778]: Uploading piece... Dec 30 03:50:44.897531506 (588h21m44.768614541s) [noise=true , address=145.79.227.11:4729 , piece_id=HQTRNF7SZ6TLONQC43SECGWWG6EH5UGSSZYI65IZJQLGJF7CTIWQ, seg_pos={0 0}, upload=515778]: Uploading piece... Dec 30 03:50:44.897752687 (588h21m44.768835722s) [noise=true , address=45.91.116.6:4973 , piece_id=MULDAZEZVIZMW6UXEZNHM2ZF762GQWXYOJBHOEFJZEF2MEUSX2IA, seg_pos={0 0}, upload=515778]: Uploading piece... Dec 30 03:50:44.898270172 (588h21m44.769353207s) [noise=true , address=77.90.31.14:4664 , piece_id=VEWPZRCZ3GHP4SVVH36V6ZLNZDRTCAQ6Y5ZMAW7PMLDM7KAOKDPQ, seg_pos={0 0}, upload=515778]: Uploading piece... Dec 30 03:50:44.898816181 (588h21m44.769899216s) [noise=true , address=145.79.227.46:4601 , piece_id=2NKPLJBTYPULRJHRZDUXZVNN6GUXN7UO6Y7G3DYQSFAWFUGICPMA, seg_pos={0 0}, upload=515778]: Uploading piece... Dec 30 03:50:44.899286587 (588h21m44.770369623s) [noise=true , address=45.91.116.7:4011 , piece_id=ADUSM7OBTNGRJOBAE2CTNM2FVDLTXHDV6ZANA3MQNTEOMJQPR7KA, seg_pos={0 0}, upload=515778]: Uploading piece... Dec 30 03:50:44.899559066 (588h21m44.770642102s) [noise=true , address=145.79.227.42:4121 , piece_id=NMQUIVFOXQXNO2HKCYMGV4KZ4UX6AP2RYY3U2JLXNCYKQWCUCP6A, seg_pos={0 0}, upload=515778]: Uploading piece... Dec 30 03:50:46.499996098 (588h21m46.371079133s) [upload=515778]: Next returned nil segment Dec 30 03:50:46.500006378 (588h21m46.371089413s) [upload=515778]: Waiting for error group managing segments... Dec 30 03:50:46.878577074 (588h21m46.749660109s) [noise=true , address=145.79.227.15:4874 , piece_id=CGX2M5WBQFAPMFUIG537L7572MTR5OU6CCF5TB64ANIAHJO3RGTQ, seg_pos={0 0}, upload=515778]: Done uploading piece. err: <nil> Dec 30 03:50:46.880225402 (588h21m46.751308447s) [noise=true , address=145.79.227.26:4655 , piece_id=F4PIAKSTVFOWKZ2DFSDZTSSVB7EHT4RAGO4I5GCNV6I5NI5OBEEA, seg_pos={0 0}, upload=515778]: Done uploading piece. err: <nil> Dec 30 03:50:46.880960001 (588h21m46.752043036s) [noise=true , address=145.79.227.7:4294 , piece_id=7CBO7REVWCP4LUQZ57QV2OU7BGE4RHDYBOK5G6PI7BJXJAFFUZXA, seg_pos={0 0}, upload=515778]: Done uploading piece. err: <nil> Dec 30 03:50:46.883262505 (588h21m46.75434554s) [noise=true , address=145.79.227.13:4804 , piece_id=LG4FAVRK2PWRX2TGHA7M3VWWGWPE5AHCI7CTZ4LLGHJ3QCMIICMA, seg_pos={0 0}, upload=515778]: Done uploading piece. err: <nil> Dec 30 03:50:46.884633585 (588h21m46.755716621s) [noise=true , address=145.79.227.11:4729 , piece_id=HQTRNF7SZ6TLONQC43SECGWWG6EH5UGSSZYI65IZJQLGJF7CTIWQ, seg_pos={0 0}, upload=515778]: Done uploading piece. err: <nil> Dec 30 03:50:46.890460090 (588h21m46.761543125s) [noise=true , address=145.79.227.46:4601 , piece_id=2NKPLJBTYPULRJHRZDUXZVNN6GUXN7UO6Y7G3DYQSFAWFUGICPMA, seg_pos={0 0}, upload=515778]: Done uploading piece. err: <nil> Dec 30 03:50:46.900643742 (588h21m46.771726787s) [noise=true , address=145.79.227.51:4411 , piece_id=ET37UNKINHDW4XSBHKPXWG2OFZQU6LW23UKCZCHHWQQJGWYCHOSA, seg_pos={0 0}, upload=515778]: Done uploading piece. err: <nil> Dec 30 03:50:46.911179513 (588h21m46.782262568s) [noise=true , address=145.79.227.42:4121 , piece_id=NMQUIVFOXQXNO2HKCYMGV4KZ4UX6AP2RYY3U2JLXNCYKQWCUCP6A, seg_pos={0 0}, upload=515778]: Done uploading piece. err: <nil> Dec 30 03:50:47.145138661 (588h21m47.016221696s) [noise=true , address=45.91.116.7:4011 , piece_id=ADUSM7OBTNGRJOBAE2CTNM2FVDLTXHDV6ZANA3MQNTEOMJQPR7KA, seg_pos={0 0}, upload=515778]: Done uploading piece. err: <nil> Dec 30 03:50:47.175069272 (588h21m47.046152307s) [noise=true , address=77.90.31.14:4664 , piece_id=VEWPZRCZ3GHP4SVVH36V6ZLNZDRTCAQ6Y5ZMAW7PMLDM7KAOKDPQ, seg_pos={0 0}, upload=515778]: Done uploading piece. err: <nil> Dec 30 03:50:47.178001555 (588h21m47.0490846s) [noise=true , address=77.90.31.10:4519 , piece_id=TP2IFLWBFSJLWEXKJFIH7HDWXU3PXREOCVEPRALIZHPZLP52R5BQ, seg_pos={0 0}, upload=515778]: Done uploading piece. err: <nil> Dec 30 03:50:47.281589540 (588h21m47.152672575s) [noise=true , address=77.90.31.17:4770 , piece_id=BGRIQ2C32XKNT7TYHITIPRIN7LDMA7ABWMPLAKXGERGCUAZ5XSBA, seg_pos={0 0}, upload=515778]: Done uploading piece. err: <nil> Dec 30 03:50:47.361085621 (588h21m47.232168656s) [noise=true , address=45.91.116.6:4973 , piece_id=MULDAZEZVIZMW6UXEZNHM2ZF762GQWXYOJBHOEFJZEF2MEUSX2IA, seg_pos={0 0}, upload=515778]: Done uploading piece. err: <nil> Dec 30 03:50:47.361104838 (588h21m47.232187883s) [seg_pos={0 0}, upload=515778]: Segment reached optimal threshold of 13 pieces. Dec 30 03:50:47.361259031 (588h21m47.232342066s) [noise=true , address=145.79.227.2:4115 , piece_id=H5FAMB5RLMZ3FO6ZMTV6GXCIML5LU5SLSDR2GNH5DCIR7IHAICXA, seg_pos={0 0}, upload=515778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:40066->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:40066->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40066->145.79.227.2:4115: use of closed network connection Dec 30 03:50:47.361264772 (588h21m47.232347807s) [noise=true , address=145.79.227.2:4115 , piece_id=H5FAMB5RLMZ3FO6ZMTV6GXCIML5LU5SLSDR2GNH5DCIR7IHAICXA, seg_pos={0 0}, upload=515778]: Long tail cancellation due to optimal threshold Dec 30 03:50:47.361281364 (588h21m47.232364399s) [noise=true , address=45.91.116.9:4264 , piece_id=Y6W4IEFPQE2DZGUULIJGZ6RQU7LY5R6XNDQWWJTKBUYDCYQA4BVQ, seg_pos={0 0}, upload=515778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: write tcp 188.166.225.170:43534->45.91.116.9:4264: use of closed network connection; write tcp 188.166.225.170:43534->45.91.116.9:4264: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43534->45.91.116.9:4264: use of closed network connection Dec 30 03:50:47.361290140 (588h21m47.232373175s) [noise=true , address=45.91.116.9:4264 , piece_id=Y6W4IEFPQE2DZGUULIJGZ6RQU7LY5R6XNDQWWJTKBUYDCYQA4BVQ, seg_pos={0 0}, upload=515778]: Long tail cancellation due to optimal threshold Dec 30 03:50:47.361351367 (588h21m47.232434412s) [noise=true , address=77.90.31.19:4844 , piece_id=RDDQDSIBCJRUGXUNACXWJQPACPOMDAV34BAAAH7WTNXYMLRCI34A, seg_pos={0 0}, upload=515778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: write tcp 188.166.225.170:34786->77.90.31.19:4844: use of closed network connection; write tcp 188.166.225.170:34786->77.90.31.19:4844: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34786->77.90.31.19:4844: use of closed network connection Dec 30 03:50:47.361356967 (588h21m47.232440002s) [noise=true , address=77.90.31.19:4844 , piece_id=RDDQDSIBCJRUGXUNACXWJQPACPOMDAV34BAAAH7WTNXYMLRCI34A, seg_pos={0 0}, upload=515778]: Long tail cancellation due to optimal threshold Dec 30 03:50:47.361482696 (588h21m47.232565731s) [seg_pos={0 0}, upload=515778]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 03:50:47.361547339 (588h21m47.232630375s) [upload=515778]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 03:50:47.361553822 (588h21m47.232636857s) [upload=515778]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 03:50:47.665512908 (588h21m47.536595944s) [upload=515778]: Done uploading segments... Dec 30 03:50:47.665535943 (588h21m47.536618978s) [upload=515778]: Upload finished. err: <nil> Duration:* 30.003391737s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [............X....X.X..........XX...X.......................X] duration: 5.000807484s trace: https://jaeger.dev.storj.io/trace/51699b18f550d6ea FAIL (2025-12-30 03:50:57 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000829035s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [.X.X.XX..X..X.X.X...XXX.X.....XXXX........XX.XX...XXXX...XXX] duration: 30.002287635s trace: https://jaeger.dev.storj.io/trace/3f33f1d339f81ed5 FAIL (2025-12-30 03:50:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: 4GAYYR4VXCGHTDBV7K7OOPYJKFYS3CZXRECJINUS6TOWGLKO6KYQ) context deadline exceeded; piecestore: (Node ID: 12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, Piece ID: OX565K3EXYCIETTSJKAJKLAL75ETSTGVOJLHYJWDXSCIIJXEWROA) context deadline exceeded ======LOGS====== Dec 30 03:50:09.586899970 (588h21m9.457983015s) [upload=515768]: Starting upload Dec 30 03:50:09.586998978 (588h21m9.458082023s) [upload=515768]: Done starting upload Dec 30 03:50:09.587054204 (588h21m9.458137239s) [upload=515768]: Uploading segments... Dec 30 03:50:09.587274263 (588h21m9.458357308s) [upload=515768]: Got next segment. Inline: false Dec 30 03:50:09.587325220 (588h21m9.458408265s) [upload=515768]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 03:50:09.587331452 (588h21m9.458414497s) [upload=515768]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 03:50:09.835682252 (588h21m9.706765298s) [seg_pos={0 0}, upload=515768]: Begin upload segment... Dec 30 03:50:09.835743620 (588h21m9.706826655s) [seg_pos={0 0}, upload=515768]: Done begin upload segment. Dec 30 03:50:09.835760873 (588h21m9.706843918s) [noise=true , address=45.91.116.5:4297 , piece_id=TUUQZKEOQSXJ5EAMZL7LNXAKMDBS7OCYGKJ3B2KDLN3NX7MD5FXQ, seg_pos={0 0}, upload=515768]: Uploading piece... Dec 30 03:50:09.835774899 (588h21m9.706857944s) [noise=true , address=145.79.227.39:4015 , piece_id=NJ4AUJXVFRC5GA65SBNFZBGJAZRFTVBJ7MDFI4UAEYMFUDSRC63A, seg_pos={0 0}, upload=515768]: Uploading piece... Dec 30 03:50:09.835899838 (588h21m9.706982883s) [noise=true , address=145.79.227.42:4123 , piece_id=EO2S5U4FXO3T5XPZIJVNP2AWMNNSPUKZYCQCWABQXLOZXDJ3NSFQ, seg_pos={0 0}, upload=515768]: Uploading piece... Dec 30 03:50:09.836423885 (588h21m9.70750693s) [noise=true , address=77.90.31.11:4554 , piece_id=UO5BTPKXKMPB5MRM5I4RWMVDD27ZYTCNNYLOUBTE7V7VHVRC5L3A, seg_pos={0 0}, upload=515768]: Uploading piece... Dec 30 03:50:09.836611392 (588h21m9.707694437s) [noise=true , address=145.79.227.26:4655 , piece_id=BCZDKG6ZWUXYXZWF2COMN763KQETFHM26IXHK2RLVBN2JM6YPXHA, seg_pos={0 0}, upload=515768]: Uploading piece... Dec 30 03:50:09.836840749 (588h21m9.707923794s) [noise=true , address=145.79.227.32:4836 , piece_id=VXQMSP3X5FUF3M5QZZQN5ETGGN75WOE53EEFGMO7JX4IXJZNRN7A, seg_pos={0 0}, upload=515768]: Uploading piece... Dec 30 03:50:09.837227115 (588h21m9.70831016s) [noise=true , address=45.91.116.11:4119 , piece_id=4GAYYR4VXCGHTDBV7K7OOPYJKFYS3CZXRECJINUS6TOWGLKO6KYQ, seg_pos={0 0}, upload=515768]: Uploading piece... Dec 30 03:50:09.837596477 (588h21m9.708679523s) [noise=true , address=45.91.116.12:4225 , piece_id=ERGFF32ZHUBUJWMUQRDVAKD5HHPGO6FUBWXKLAMAG6GJHHAVP2ZA, seg_pos={0 0}, upload=515768]: Uploading piece... Dec 30 03:50:09.838026547 (588h21m9.709109592s) [noise=true , address=77.90.31.21:4877 , piece_id=OX565K3EXYCIETTSJKAJKLAL75ETSTGVOJLHYJWDXSCIIJXEWROA, seg_pos={0 0}, upload=515768]: Uploading piece... Dec 30 03:50:09.838463949 (588h21m9.709546984s) [noise=true , address=45.91.116.6:4974 , piece_id=RVHVJRZLZS5HNKMJRRGGEA4YFMX5DY2PYP5WMAE7ZNN6ZKJRASFQ, seg_pos={0 0}, upload=515768]: Uploading piece... Dec 30 03:50:09.838909217 (588h21m9.709992263s) [noise=true , address=45.91.116.9:4264 , piece_id=UZW6IAHSQN7FDQ6PQUSRXCQGDEJM7ZYXI4JLMQYOHCW67UHCYG4Q, seg_pos={0 0}, upload=515768]: Uploading piece... Dec 30 03:50:09.839458183 (588h21m9.710541218s) [noise=true , address=145.79.227.45:4231 , piece_id=DCFNGCPPBDAYA36A2LHYYM4U6LAINAVKXEHWY67PR6XUJ67A4TSQ, seg_pos={0 0}, upload=515768]: Uploading piece... Dec 30 03:50:09.839469715 (588h21m9.71055276s) [noise=true , address=145.79.227.7:4293 , piece_id=AV3J3OCZQGFWXS2OOTZONESESJ2ZBEFA7E2AGX5LRBWFCPOBTBYQ, seg_pos={0 0}, upload=515768]: Uploading piece... Dec 30 03:50:09.840039619 (588h21m9.711122665s) [noise=true , address=45.91.116.13:4153 , piece_id=PGRSNEB2XGPSMAHRZYZT3G2YHJZCDBW45P7ICEIEXZ4GN5GO4PCQ, seg_pos={0 0}, upload=515768]: Uploading piece... Dec 30 03:50:09.840456363 (588h21m9.711539408s) [noise=true , address=145.79.227.51:4410 , piece_id=25HMSKPYQB5SIYQBLZSNQBCFWZCY6WIBRYRDRQJRUNPBLAGWA6EQ, seg_pos={0 0}, upload=515768]: Uploading piece... Dec 30 03:50:09.840890811 (588h21m9.711973856s) [noise=true , address=145.79.227.46:4602 , piece_id=Q45PBB3LKULEDB6CWHSYRR4PZA7LLDY6JET6JTCVZSUWQDFICPQA, seg_pos={0 0}, upload=515768]: Uploading piece... Dec 30 03:50:11.534602077 (588h21m11.405685152s) [upload=515768]: Next returned nil segment Dec 30 03:50:11.534633477 (588h21m11.405716512s) [upload=515768]: Waiting for error group managing segments... Dec 30 03:50:11.809428611 (588h21m11.680511656s) [noise=true , address=145.79.227.32:4836 , piece_id=VXQMSP3X5FUF3M5QZZQN5ETGGN75WOE53EEFGMO7JX4IXJZNRN7A, seg_pos={0 0}, upload=515768]: Done uploading piece. err: <nil> Dec 30 03:50:11.815817666 (588h21m11.686900711s) [noise=true , address=145.79.227.45:4231 , piece_id=DCFNGCPPBDAYA36A2LHYYM4U6LAINAVKXEHWY67PR6XUJ67A4TSQ, seg_pos={0 0}, upload=515768]: Done uploading piece. err: <nil> Dec 30 03:50:11.823455600 (588h21m11.694538635s) [noise=true , address=145.79.227.26:4655 , piece_id=BCZDKG6ZWUXYXZWF2COMN763KQETFHM26IXHK2RLVBN2JM6YPXHA, seg_pos={0 0}, upload=515768]: Done uploading piece. err: <nil> Dec 30 03:50:11.825691557 (588h21m11.696774592s) [noise=true , address=145.79.227.7:4293 , piece_id=AV3J3OCZQGFWXS2OOTZONESESJ2ZBEFA7E2AGX5LRBWFCPOBTBYQ, seg_pos={0 0}, upload=515768]: Done uploading piece. err: <nil> Dec 30 03:50:11.825902078 (588h21m11.696985113s) [noise=true , address=145.79.227.51:4410 , piece_id=25HMSKPYQB5SIYQBLZSNQBCFWZCY6WIBRYRDRQJRUNPBLAGWA6EQ, seg_pos={0 0}, upload=515768]: Done uploading piece. err: <nil> Dec 30 03:50:11.836591212 (588h21m11.707674267s) [noise=true , address=145.79.227.39:4015 , piece_id=NJ4AUJXVFRC5GA65SBNFZBGJAZRFTVBJ7MDFI4UAEYMFUDSRC63A, seg_pos={0 0}, upload=515768]: Done uploading piece. err: <nil> Dec 30 03:50:11.842799033 (588h21m11.713882068s) [noise=true , address=145.79.227.42:4123 , piece_id=EO2S5U4FXO3T5XPZIJVNP2AWMNNSPUKZYCQCWABQXLOZXDJ3NSFQ, seg_pos={0 0}, upload=515768]: Done uploading piece. err: <nil> Dec 30 03:50:11.853746638 (588h21m11.724829673s) [noise=true , address=145.79.227.46:4602 , piece_id=Q45PBB3LKULEDB6CWHSYRR4PZA7LLDY6JET6JTCVZSUWQDFICPQA, seg_pos={0 0}, upload=515768]: Done uploading piece. err: <nil> Dec 30 03:50:11.995717228 (588h21m11.866800273s) [noise=true , address=77.90.31.21:4877 , piece_id=OX565K3EXYCIETTSJKAJKLAL75ETSTGVOJLHYJWDXSCIIJXEWROA, seg_pos={0 0}, upload=515768]: Done uploading piece. err: <nil> Dec 30 03:50:12.091466868 (588h21m11.962549913s) [noise=true , address=45.91.116.11:4119 , piece_id=4GAYYR4VXCGHTDBV7K7OOPYJKFYS3CZXRECJINUS6TOWGLKO6KYQ, seg_pos={0 0}, upload=515768]: Done uploading piece. err: <nil> Dec 30 03:50:12.117116136 (588h21m11.988199171s) [noise=true , address=45.91.116.13:4153 , piece_id=PGRSNEB2XGPSMAHRZYZT3G2YHJZCDBW45P7ICEIEXZ4GN5GO4PCQ, seg_pos={0 0}, upload=515768]: Done uploading piece. err: <nil> Dec 30 03:50:12.184847766 (588h21m12.055930811s) [noise=true , address=45.91.116.5:4297 , piece_id=TUUQZKEOQSXJ5EAMZL7LNXAKMDBS7OCYGKJ3B2KDLN3NX7MD5FXQ, seg_pos={0 0}, upload=515768]: Done uploading piece. err: <nil> Dec 30 03:50:12.185200057 (588h21m12.056283092s) [noise=true , address=45.91.116.9:4264 , piece_id=UZW6IAHSQN7FDQ6PQUSRXCQGDEJM7ZYXI4JLMQYOHCW67UHCYG4Q, seg_pos={0 0}, upload=515768]: Done uploading piece. err: <nil> Dec 30 03:50:12.185220967 (588h21m12.056304012s) [seg_pos={0 0}, upload=515768]: Segment reached optimal threshold of 13 pieces. Dec 30 03:50:12.185441747 (588h21m12.056524813s) [noise=true , address=45.91.116.12:4225 , piece_id=ERGFF32ZHUBUJWMUQRDVAKD5HHPGO6FUBWXKLAMAG6GJHHAVP2ZA, seg_pos={0 0}, upload=515768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:34452->45.91.116.12:4225: use of closed network connection; write tcp 188.166.225.170:34452->45.91.116.12:4225: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34452->45.91.116.12:4225: use of closed network connection Dec 30 03:50:12.185450704 (588h21m12.056533749s) [noise=true , address=45.91.116.12:4225 , piece_id=ERGFF32ZHUBUJWMUQRDVAKD5HHPGO6FUBWXKLAMAG6GJHHAVP2ZA, seg_pos={0 0}, upload=515768]: Long tail cancellation due to optimal threshold Dec 30 03:50:12.185621109 (588h21m12.056704154s) [noise=true , address=77.90.31.11:4554 , piece_id=UO5BTPKXKMPB5MRM5I4RWMVDD27ZYTCNNYLOUBTE7V7VHVRC5L3A, seg_pos={0 0}, upload=515768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Dec 30 03:50:12.185627330 (588h21m12.056710376s) [noise=true , address=77.90.31.11:4554 , piece_id=UO5BTPKXKMPB5MRM5I4RWMVDD27ZYTCNNYLOUBTE7V7VHVRC5L3A, seg_pos={0 0}, upload=515768]: Long tail cancellation due to optimal threshold Dec 30 03:50:12.185671414 (588h21m12.056754479s) [noise=true , address=45.91.116.6:4974 , piece_id=RVHVJRZLZS5HNKMJRRGGEA4YFMX5DY2PYP5WMAE7ZNN6ZKJRASFQ, seg_pos={0 0}, upload=515768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 30 03:50:12.185685792 (588h21m12.056768827s) [noise=true , address=45.91.116.6:4974 , piece_id=RVHVJRZLZS5HNKMJRRGGEA4YFMX5DY2PYP5WMAE7ZNN6ZKJRASFQ, seg_pos={0 0}, upload=515768]: Long tail cancellation due to optimal threshold Dec 30 03:50:12.185738662 (588h21m12.056821697s) [seg_pos={0 0}, upload=515768]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 03:50:12.185875102 (588h21m12.056958137s) [upload=515768]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 03:50:12.185883679 (588h21m12.056966714s) [upload=515768]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 03:50:12.478196118 (588h21m12.349279194s) [upload=515768]: Done uploading segments... Dec 30 03:50:12.478224673 (588h21m12.349307718s) [upload=515768]: Upload finished. err: <nil> Duration: 30.002331088s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [X............X....X.X..........XX...X.......................] duration: 4.485938164s trace: https://jaeger.dev.storj.io/trace/592ef51aa1a6f778 FAIL (2025-12-30 03:49:48 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.485962439s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [..X.X.XX..X..X.X.X...XXX.X.....XXXX........XX.XX...XXXX...XX] duration: 30.002839783s trace: https://jaeger.dev.storj.io/trace/a35ac80032a69aa FAIL (2025-12-30 03:49:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: G7EVOUXW5XW3EKVUY4IAM4P35HEG2SVMZ76XS37OVHBLFLBLJT6Q) context deadline exceeded; piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: DXABH7PCDPHQL3ESDDLZWEYTBTIFYKPAK6KFWORBYL7DD4GGUXUA) context deadline exceeded ======LOGS====== Dec 30 03:49:10.431662231 (588h20m10.302745276s) [upload=515755]: Starting upload Dec 30 03:49:10.431719901 (588h20m10.302802936s) [upload=515755]: Done starting upload Dec 30 03:49:10.431846041 (588h20m10.302929076s) [upload=515755]: Uploading segments... Dec 30 03:49:10.431950139 (588h20m10.303033174s) [upload=515755]: Got next segment. Inline: false Dec 30 03:49:10.431984755 (588h20m10.30306779s) [upload=515755]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 03:49:10.431990035 (588h20m10.30307308s) [upload=515755]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 03:49:10.673159593 (588h20m10.544242639s) [seg_pos={0 0}, upload=515755]: Begin upload segment... Dec 30 03:49:10.673244556 (588h20m10.544327591s) [seg_pos={0 0}, upload=515755]: Done begin upload segment. Dec 30 03:49:10.673309029 (588h20m10.544392074s) [noise=true , address=77.90.31.21:4877 , piece_id=TZMKY5GODDEE66LVZWBMFKZGHMPW6PNNF4SAWJGHNZRRP2OAO42Q, seg_pos={0 0}, upload=515755]: Uploading piece... Dec 30 03:49:10.673315250 (588h20m10.544398285s) [noise=true , address=45.91.116.8:4045 , piece_id=2HXK6H6Z3V2HZH7LZBWKSXX4DF5JNUY4YJGLGP7HYOYVINA3RB4Q, seg_pos={0 0}, upload=515755]: Uploading piece... Dec 30 03:49:10.673386636 (588h20m10.544469681s) [noise=true , address=45.91.116.9:4261 , piece_id=MY7WLBUK6HYOOEDVLBQTBMX6E7LWMYNF3VKT2SWPLXT2XECZKBCQ, seg_pos={0 0}, upload=515755]: Uploading piece... Dec 30 03:49:10.674005254 (588h20m10.54508832s) [noise=true , address=145.79.227.4:4186 , piece_id=KONRBTLBU6JVSJ2AM6XUJ445OOEOCFHZ2YINTESX2JKBCZP6GQYQ, seg_pos={0 0}, upload=515755]: Uploading piece... Dec 30 03:49:10.674540153 (588h20m10.545623198s) [noise=true , address=45.91.116.6:4976 , piece_id=F43SJQHSB6PZDVYDUWWRPQEZ3D7OH5GJOVZOJZ46IK24EU4UGGIQ, seg_pos={0 0}, upload=515755]: Uploading piece... Dec 30 03:49:10.674980652 (588h20m10.546063697s) [noise=true , address=45.91.116.12:4228 , piece_id=DXABH7PCDPHQL3ESDDLZWEYTBTIFYKPAK6KFWORBYL7DD4GGUXUA, seg_pos={0 0}, upload=515755]: Uploading piece... Dec 30 03:49:10.675362359 (588h20m10.546445404s) [noise=true , address=145.79.227.3:4149 , piece_id=7LFG5NILQ36LXF62MSGH4DEM5VX7UF5KI5L4XP4RUZID5QHDHKXQ, seg_pos={0 0}, upload=515755]: Uploading piece... Dec 30 03:49:10.675774162 (588h20m10.546857207s) [noise=true , address=145.79.227.16:4909 , piece_id=6MEAIIHTV5OB7SG3ZVC5RDTTIZG5FGXTLURM2OTYZ5MNU5HZIJCQ, seg_pos={0 0}, upload=515755]: Uploading piece... Dec 30 03:49:10.676228307 (588h20m10.547311352s) [noise=true , address=45.91.116.7:4012 , piece_id=CUIKINBW4Z3JSBN6E4XUJSBPN4LIWM6CDFENZCZOAFQXXLVVZWTA, seg_pos={0 0}, upload=515755]: Uploading piece... Dec 30 03:49:10.676718380 (588h20m10.547801425s) [noise=true , address=145.79.227.6:4258 , piece_id=QC3U4KIFTCOS6LKRWHRGTF2CDRGPBGIDUST3LNAMBSP3R2UOX73Q, seg_pos={0 0}, upload=515755]: Uploading piece... Dec 30 03:49:10.678488220 (588h20m10.549571255s) [noise=true , address=145.79.227.2:4113 , piece_id=U5DWCFTKHTB3W46HMD5OAXQHBMPFJEI3EBHNGMKSFWZYZCNPBKWA, seg_pos={0 0}, upload=515755]: Uploading piece... Dec 30 03:49:10.680099137 (588h20m10.551182182s) [noise=true , address=145.79.227.37:4565 , piece_id=4TK65GZVRJLJQEN3PTIM7JCBZT3OKBCNVIOXPF53OAZ2VZVUYBWA, seg_pos={0 0}, upload=515755]: Uploading piece... Dec 30 03:49:10.681939371 (588h20m10.553022417s) [noise=true , address=77.90.31.19:4842 , piece_id=G7EVOUXW5XW3EKVUY4IAM4P35HEG2SVMZ76XS37OVHBLFLBLJT6Q, seg_pos={0 0}, upload=515755]: Uploading piece... Dec 30 03:49:10.683646512 (588h20m10.554729547s) [noise=true , address=145.79.227.46:4603 , piece_id=KPZ4G77JGKSYUHB5E2IBKE3TIHJIYNBWRA7M33SKECWG76326HFQ, seg_pos={0 0}, upload=515755]: Uploading piece... Dec 30 03:49:10.685261136 (588h20m10.556344191s) [noise=true , address=145.79.227.43:4158 , piece_id=H2UHFCNYSFNY4B2HQSTXFP32VFMMLSGYN625H2OVHBRXCXYTYY5A, seg_pos={0 0}, upload=515755]: Uploading piece... Dec 30 03:49:10.686891591 (588h20m10.557974636s) [noise=true , address=145.79.227.29:4941 , piece_id=DE7QAS3ZSICQMBOPZENMEF5YJ7AMZSJCKN633VFWCZBGNTCOB4OA, seg_pos={0 0}, upload=515755]: Uploading piece... Dec 30 03:49:12.366852615 (588h20m12.23793566s) [upload=515755]: Next returned nil segment Dec 30 03:49:12.366867394 (588h20m12.237950439s) [upload=515755]: Waiting for error group managing segments... Dec 30 03:49:12.647378532 (588h20m12.518461577s) [noise=true , address=145.79.227.3:4149 , piece_id=7LFG5NILQ36LXF62MSGH4DEM5VX7UF5KI5L4XP4RUZID5QHDHKXQ, seg_pos={0 0}, upload=515755]: Done uploading piece. err: <nil> Dec 30 03:49:12.672330944 (588h20m12.543413979s) [noise=true , address=145.79.227.2:4113 , piece_id=U5DWCFTKHTB3W46HMD5OAXQHBMPFJEI3EBHNGMKSFWZYZCNPBKWA, seg_pos={0 0}, upload=515755]: Done uploading piece. err: <nil> Dec 30 03:49:12.673608407 (588h20m12.544691442s) [noise=true , address=145.79.227.4:4186 , piece_id=KONRBTLBU6JVSJ2AM6XUJ445OOEOCFHZ2YINTESX2JKBCZP6GQYQ, seg_pos={0 0}, upload=515755]: Done uploading piece. err: <nil> Dec 30 03:49:12.684428720 (588h20m12.555511755s) [noise=true , address=145.79.227.16:4909 , piece_id=6MEAIIHTV5OB7SG3ZVC5RDTTIZG5FGXTLURM2OTYZ5MNU5HZIJCQ, seg_pos={0 0}, upload=515755]: Done uploading piece. err: <nil> Dec 30 03:49:12.689931588 (588h20m12.561014633s) [noise=true , address=145.79.227.46:4603 , piece_id=KPZ4G77JGKSYUHB5E2IBKE3TIHJIYNBWRA7M33SKECWG76326HFQ, seg_pos={0 0}, upload=515755]: Done uploading piece. err: <nil> Dec 30 03:49:12.835491712 (588h20m12.706574748s) [noise=true , address=77.90.31.21:4877 , piece_id=TZMKY5GODDEE66LVZWBMFKZGHMPW6PNNF4SAWJGHNZRRP2OAO42Q, seg_pos={0 0}, upload=515755]: Done uploading piece. err: <nil> Dec 30 03:49:12.925002743 (588h20m12.796085788s) [noise=true , address=45.91.116.8:4045 , piece_id=2HXK6H6Z3V2HZH7LZBWKSXX4DF5JNUY4YJGLGP7HYOYVINA3RB4Q, seg_pos={0 0}, upload=515755]: Done uploading piece. err: <nil> Dec 30 03:49:12.945453594 (588h20m12.816536629s) [noise=true , address=45.91.116.9:4261 , piece_id=MY7WLBUK6HYOOEDVLBQTBMX6E7LWMYNF3VKT2SWPLXT2XECZKBCQ, seg_pos={0 0}, upload=515755]: Done uploading piece. err: <nil> Dec 30 03:49:12.963448008 (588h20m12.834531053s) [noise=true , address=145.79.227.29:4941 , piece_id=DE7QAS3ZSICQMBOPZENMEF5YJ7AMZSJCKN633VFWCZBGNTCOB4OA, seg_pos={0 0}, upload=515755]: Done uploading piece. err: <nil> Dec 30 03:49:13.030724351 (588h20m12.901807396s) [noise=true , address=45.91.116.7:4012 , piece_id=CUIKINBW4Z3JSBN6E4XUJSBPN4LIWM6CDFENZCZOAFQXXLVVZWTA, seg_pos={0 0}, upload=515755]: Done uploading piece. err: <nil> Dec 30 03:49:13.657401752 (588h20m13.528484797s) [noise=true , address=45.91.116.6:4976 , piece_id=F43SJQHSB6PZDVYDUWWRPQEZ3D7OH5GJOVZOJZ46IK24EU4UGGIQ, seg_pos={0 0}, upload=515755]: Done uploading piece. err: <nil> Dec 30 03:49:13.985304160 (588h20m13.856387205s) [noise=true , address=45.91.116.12:4228 , piece_id=DXABH7PCDPHQL3ESDDLZWEYTBTIFYKPAK6KFWORBYL7DD4GGUXUA, seg_pos={0 0}, upload=515755]: Done uploading piece. err: <nil> Dec 30 03:49:13.995037884 (588h20m13.866120929s) [noise=true , address=77.90.31.19:4842 , piece_id=G7EVOUXW5XW3EKVUY4IAM4P35HEG2SVMZ76XS37OVHBLFLBLJT6Q, seg_pos={0 0}, upload=515755]: Done uploading piece. err: <nil> Dec 30 03:49:13.995058283 (588h20m13.866141318s) [seg_pos={0 0}, upload=515755]: Segment reached optimal threshold of 13 pieces. Dec 30 03:49:13.995269605 (588h20m13.86635265s) [noise=true , address=145.79.227.43:4158 , piece_id=H2UHFCNYSFNY4B2HQSTXFP32VFMMLSGYN625H2OVHBRXCXYTYY5A, seg_pos={0 0}, upload=515755]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF Dec 30 03:49:13.995277300 (588h20m13.866360335s) [noise=true , address=145.79.227.43:4158 , piece_id=H2UHFCNYSFNY4B2HQSTXFP32VFMMLSGYN625H2OVHBRXCXYTYY5A, seg_pos={0 0}, upload=515755]: Long tail cancellation due to optimal threshold Dec 30 03:49:13.995405955 (588h20m13.86648899s) [noise=true , address=145.79.227.6:4258 , piece_id=QC3U4KIFTCOS6LKRWHRGTF2CDRGPBGIDUST3LNAMBSP3R2UOX73Q, seg_pos={0 0}, upload=515755]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF): protocol: expected piece hash; context canceled; EOF Dec 30 03:49:13.995411225 (588h20m13.86649426s) [noise=true , address=145.79.227.6:4258 , piece_id=QC3U4KIFTCOS6LKRWHRGTF2CDRGPBGIDUST3LNAMBSP3R2UOX73Q, seg_pos={0 0}, upload=515755]: Long tail cancellation due to optimal threshold Dec 30 03:49:13.995515293 (588h20m13.866598328s) [noise=true , address=145.79.227.37:4565 , piece_id=4TK65GZVRJLJQEN3PTIM7JCBZT3OKBCNVIOXPF53OAZ2VZVUYBWA, seg_pos={0 0}, upload=515755]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nvvNczHtQrSEJuJbErnjiAWa4BRDzBrQmBNbB7RrrqN1JWNHT): protocol: expected piece hash; context canceled; EOF Dec 30 03:49:13.995522757 (588h20m13.866605792s) [noise=true , address=145.79.227.37:4565 , piece_id=4TK65GZVRJLJQEN3PTIM7JCBZT3OKBCNVIOXPF53OAZ2VZVUYBWA, seg_pos={0 0}, upload=515755]: Long tail cancellation due to optimal threshold Dec 30 03:49:13.995541824 (588h20m13.866624859s) [seg_pos={0 0}, upload=515755]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 03:49:13.995825975 (588h20m13.86690901s) [upload=515755]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 03:49:13.995833539 (588h20m13.866916584s) [upload=515755]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 03:49:14.285678329 (588h20m14.156761364s) [upload=515755]: Done uploading segments... Dec 30 03:49:14.285703517 (588h20m14.156786562s) [upload=515755]: Upload finished. err: <nil> Duration: 30.002864309s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [XX............X....X.X..........XX...X......................] duration: 3.433464515s trace: https://jaeger.dev.storj.io/trace/592ef51aa1a6f778 FAIL (2025-12-30 03:49:16 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.433487177s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [X..X.X.XX..X..X.X.X...XXX.X.....XXXX........XX.XX...XXXX...X] duration: 30.002760824s trace: https://jaeger.dev.storj.io/trace/8cd25e2edd8c6b FAIL (2025-12-30 03:48:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== segments... Dec 30 03:48:20.043184475 (588h19m19.91426752s) [upload=515742]: Got next segment. Inline: false Dec 30 03:48:20.043224752 (588h19m19.914307787s) [upload=515742]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 03:48:20.043230593 (588h19m19.914313638s) [upload=515742]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 03:48:20.294579002 (588h19m20.165662047s) [seg_pos={0 0}, upload=515742]: Begin upload segment... Dec 30 03:48:20.294635250 (588h19m20.165718285s) [seg_pos={0 0}, upload=515742]: Done begin upload segment. Dec 30 03:48:20.294672090 (588h19m20.165755125s) [noise=true , address=145.79.227.4:4185 , piece_id=27HBD75WAL2VPJHVEKJA6LEX4PU7KHUOW6QGZX3KFRGRUCSSN53Q, seg_pos={0 0}, upload=515742]: Uploading piece... Dec 30 03:48:20.294705914 (588h19m20.165788979s) [noise=true , address=45.91.116.7:4012 , piece_id=NX5QOHN4KOKK7Y33ODJVHJGZGFUQ62MLM6PLMTQPWP7Q44P6ISQA, seg_pos={0 0}, upload=515742]: Uploading piece... Dec 30 03:48:20.295568958 (588h19m20.166652003s) [noise=true , address=45.91.116.9:4263 , piece_id=MC6M7AAL63LALSSA7U7XNWAXXWFLV3WDUWVFSOPT2PJVWIOV6PXQ, seg_pos={0 0}, upload=515742]: Uploading piece... Dec 30 03:48:20.295588745 (588h19m20.16667179s) [noise=true , address=45.91.116.11:4117 , piece_id=KPSPEIGTNPWESKHEVLS3DMFBO7X7T7FXGGQPWP7WHDD6M3VVUGHQ, seg_pos={0 0}, upload=515742]: Uploading piece... Dec 30 03:48:20.296017872 (588h19m20.167100917s) [noise=true , address=45.91.116.12:4227 , piece_id=RER6JFM344JUB373DGXY2GVW3EAAN4YKIOOD7CH4J5B4VKOSXVCQ, seg_pos={0 0}, upload=515742]: Uploading piece... Dec 30 03:48:20.296146307 (588h19m20.167229352s) [noise=true , address=45.91.116.13:4154 , piece_id=PDXTMJFOESXLO3E6ABBPZLA2365XAX35OUZPBP5KEEVFSN355BSQ, seg_pos={0 0}, upload=515742]: Uploading piece... Dec 30 03:48:20.296464001 (588h19m20.167547036s) [noise=true , address=77.90.31.18:4808 , piece_id=44PTMMU4Z7DIYM7A3UBZCFCBQQ5TD4G7UA5YK2GLLBGO6HJFF6XQ, seg_pos={0 0}, upload=515742]: Uploading piece... Dec 30 03:48:20.296638524 (588h19m20.167721559s) [noise=true , address=145.79.227.51:4411 , piece_id=IXVPOZNB6QAJGTKBU3K3G3QTRCEMABH4FK3Q46RNQE5L7N5XUDKQ, seg_pos={0 0}, upload=515742]: Uploading piece... Dec 30 03:48:20.296856138 (588h19m20.167939183s) [noise=true , address=145.79.227.46:4602 , piece_id=55XEMTSEL3YJNQCHKY52K4FNUVQHKVFOCEOXOCWGJR6SZV7KQDSA, seg_pos={0 0}, upload=515742]: Uploading piece... Dec 30 03:48:20.297172862 (588h19m20.168255907s) [noise=true , address=145.79.227.41:4087 , piece_id=5S5AIMGQJLX5CGV3KAKXLOZOKOCOAQLUGL2B3HLE3TJ54XF6565Q, seg_pos={0 0}, upload=515742]: Uploading piece... Dec 30 03:48:20.297245650 (588h19m20.168328695s) [noise=true , address=45.91.116.10:4084 , piece_id=VQNJMULEZQOKO2XQGJNTHVSQFHSHWGSBVLLIWR2NV2LCEACN4COQ, seg_pos={0 0}, upload=515742]: Uploading piece... Dec 30 03:48:20.297634951 (588h19m20.168717996s) [noise=true , address=145.79.227.22:4509 , piece_id=Y6FQJU5T44DJQJ4EYRC7CWLDBI4PCQNVLAUDTCDRWRQW3DV3EP4Q, seg_pos={0 0}, upload=515742]: Uploading piece... Dec 30 03:48:20.297720444 (588h19m20.168803479s) [noise=true , address=77.90.31.12:4589 , piece_id=HOQVZBAWHEONVEEPU3ED7BOOA6STCBSEBKGQ3QCPYI3N6CE2V2PQ, seg_pos={0 0}, upload=515742]: Uploading piece... Dec 30 03:48:20.298167936 (588h19m20.169250981s) [noise=true , address=45.91.116.5:4299 , piece_id=A2LXBXSWN5HMWEBQKMZ75QPZ5J4Q3ZIEPD6M5UZW5DF73FSVFCHA, seg_pos={0 0}, upload=515742]: Uploading piece... Dec 30 03:48:20.298352277 (588h19m20.169435312s) [noise=true , address=77.90.31.19:4842 , piece_id=NVXAMQAP2WIIEODH5QE3WQYKLJNDN22X3UOWEZDH6Y6NS5MHZHKA, seg_pos={0 0}, upload=515742]: Uploading piece... Dec 30 03:48:20.298724456 (588h19m20.169807501s) [noise=true , address=145.79.227.18:4006 , piece_id=EEB2I7X5MA55PDGKDXKWSZCZNQPZUHVOJASCNVA42PPPKZ2D4DNA, seg_pos={0 0}, upload=515742]: Uploading piece... Dec 30 03:48:22.001671940 (588h19m21.872754985s) [upload=515742]: Next returned nil segment Dec 30 03:48:22.001692779 (588h19m21.872775824s) [upload=515742]: Waiting for error group managing segments... Dec 30 03:48:22.280758205 (588h19m22.15184126s) [noise=true , address=145.79.227.4:4185 , piece_id=27HBD75WAL2VPJHVEKJA6LEX4PU7KHUOW6QGZX3KFRGRUCSSN53Q, seg_pos={0 0}, upload=515742]: Done uploading piece. err: <nil> Dec 30 03:48:22.282321240 (588h19m22.153404285s) [noise=true , address=145.79.227.51:4411 , piece_id=IXVPOZNB6QAJGTKBU3K3G3QTRCEMABH4FK3Q46RNQE5L7N5XUDKQ, seg_pos={0 0}, upload=515742]: Done uploading piece. err: <nil> Dec 30 03:48:22.283061420 (588h19m22.154144455s) [noise=true , address=145.79.227.46:4602 , piece_id=55XEMTSEL3YJNQCHKY52K4FNUVQHKVFOCEOXOCWGJR6SZV7KQDSA, seg_pos={0 0}, upload=515742]: Done uploading piece. err: <nil> Dec 30 03:48:22.288388152 (588h19m22.159471198s) [noise=true , address=145.79.227.41:4087 , piece_id=5S5AIMGQJLX5CGV3KAKXLOZOKOCOAQLUGL2B3HLE3TJ54XF6565Q, seg_pos={0 0}, upload=515742]: Done uploading piece. err: <nil> Dec 30 03:48:22.305147655 (588h19m22.17623069s) [noise=true , address=145.79.227.22:4509 , piece_id=Y6FQJU5T44DJQJ4EYRC7CWLDBI4PCQNVLAUDTCDRWRQW3DV3EP4Q, seg_pos={0 0}, upload=515742]: Done uploading piece. err: <nil> Dec 30 03:48:22.305259828 (588h19m22.176342863s) [noise=true , address=145.79.227.18:4006 , piece_id=EEB2I7X5MA55PDGKDXKWSZCZNQPZUHVOJASCNVA42PPPKZ2D4DNA, seg_pos={0 0}, upload=515742]: Done uploading piece. err: <nil> Dec 30 03:48:22.456941409 (588h19m22.328024454s) [noise=true , address=77.90.31.18:4808 , piece_id=44PTMMU4Z7DIYM7A3UBZCFCBQQ5TD4G7UA5YK2GLLBGO6HJFF6XQ, seg_pos={0 0}, upload=515742]: Done uploading piece. err: <nil> Dec 30 03:48:22.529759543 (588h19m22.400842589s) [noise=true , address=45.91.116.12:4227 , piece_id=RER6JFM344JUB373DGXY2GVW3EAAN4YKIOOD7CH4J5B4VKOSXVCQ, seg_pos={0 0}, upload=515742]: Done uploading piece. err: <nil> Dec 30 03:48:22.612892679 (588h19m22.483975724s) [noise=true , address=45.91.116.7:4012 , piece_id=NX5QOHN4KOKK7Y33ODJVHJGZGFUQ62MLM6PLMTQPWP7Q44P6ISQA, seg_pos={0 0}, upload=515742]: Done uploading piece. err: <nil> Dec 30 03:48:22.636534397 (588h19m22.507617432s) [noise=true , address=45.91.116.10:4084 , piece_id=VQNJMULEZQOKO2XQGJNTHVSQFHSHWGSBVLLIWR2NV2LCEACN4COQ, seg_pos={0 0}, upload=515742]: Done uploading piece. err: <nil> Dec 30 03:48:22.680323937 (588h19m22.551406982s) [noise=true , address=45.91.116.9:4263 , piece_id=MC6M7AAL63LALSSA7U7XNWAXXWFLV3WDUWVFSOPT2PJVWIOV6PXQ, seg_pos={0 0}, upload=515742]: Done uploading piece. err: <nil> Dec 30 03:48:22.681088603 (588h19m22.552171638s) [noise=true , address=45.91.116.5:4299 , piece_id=A2LXBXSWN5HMWEBQKMZ75QPZ5J4Q3ZIEPD6M5UZW5DF73FSVFCHA, seg_pos={0 0}, upload=515742]: Done uploading piece. err: <nil> Dec 30 03:48:22.714513458 (588h19m22.585596503s) [noise=true , address=45.91.116.13:4154 , piece_id=PDXTMJFOESXLO3E6ABBPZLA2365XAX35OUZPBP5KEEVFSN355BSQ, seg_pos={0 0}, upload=515742]: Done uploading piece. err: <nil> Dec 30 03:48:22.714549287 (588h19m22.585632332s) [seg_pos={0 0}, upload=515742]: Segment reached optimal threshold of 13 pieces. Dec 30 03:48:22.714697789 (588h19m22.585780824s) [noise=true , address=45.91.116.11:4117 , piece_id=KPSPEIGTNPWESKHEVLS3DMFBO7X7T7FXGGQPWP7WHDD6M3VVUGHQ, seg_pos={0 0}, upload=515742]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:53984->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:53984->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53984->45.91.116.11:4117: use of closed network connection Dec 30 03:48:22.714703791 (588h19m22.585786826s) [noise=true , address=45.91.116.11:4117 , piece_id=KPSPEIGTNPWESKHEVLS3DMFBO7X7T7FXGGQPWP7WHDD6M3VVUGHQ, seg_pos={0 0}, upload=515742]: Long tail cancellation due to optimal threshold Dec 30 03:48:22.714798691 (588h19m22.585881726s) [noise=true , address=77.90.31.19:4842 , piece_id=NVXAMQAP2WIIEODH5QE3WQYKLJNDN22X3UOWEZDH6Y6NS5MHZHKA, seg_pos={0 0}, upload=515742]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: write tcp 188.166.225.170:42888->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:42888->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42888->77.90.31.19:4842: use of closed network connection Dec 30 03:48:22.714803701 (588h19m22.585886736s) [noise=true , address=77.90.31.19:4842 , piece_id=NVXAMQAP2WIIEODH5QE3WQYKLJNDN22X3UOWEZDH6Y6NS5MHZHKA, seg_pos={0 0}, upload=515742]: Long tail cancellation due to optimal threshold Dec 30 03:48:22.714913711 (588h19m22.585996756s) [noise=true , address=77.90.31.12:4589 , piece_id=HOQVZBAWHEONVEEPU3ED7BOOA6STCBSEBKGQ3QCPYI3N6CE2V2PQ, seg_pos={0 0}, upload=515742]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 30 03:48:22.714948608 (588h19m22.586031653s) [noise=true , address=77.90.31.12:4589 , piece_id=HOQVZBAWHEONVEEPU3ED7BOOA6STCBSEBKGQ3QCPYI3N6CE2V2PQ, seg_pos={0 0}, upload=515742]: Long tail cancellation due to optimal threshold Dec 30 03:48:22.715142526 (588h19m22.586225571s) [seg_pos={0 0}, upload=515742]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 03:48:22.715269408 (588h19m22.586352453s) [upload=515742]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 03:48:22.715276712 (588h19m22.586359747s) [upload=515742]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 03:48:23.013554630 (588h19m22.884637665s) [upload=515742]: Done uploading segments... Dec 30 03:48:23.013574909 (588h19m22.884657954s) [upload=515742]: Upload finished. err: <nil> Duration: 30.002791452s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)