Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................................X..................] duration: 3.478337537s trace: https://jaeger.dev.storj.io/trace/7fc142ad3d711e34 FAIL (2025-12-28 02:20:06 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.47850255s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................X...................X] duration: 1m0.003367954s trace: https://jaeger.dev.storj.io/trace/1f68404b41391e4a FAIL (2025-12-28 02:20:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12CaBSQgwTru9NXaznMJ2XExKe3yhLoUCvDKFvqvLJGJev4mEPc, Piece ID: ULPCWUUHAL35F5WZYHDQLO7VBIGAUH2XHH4I2NYYHG7OAC5VJFOA) context deadline exceeded; piecestore: (Node ID: 1H1XywjaRt5ZRGqmZBgzfSowuCnPqFesL1Ch2jYYS1gV2oBgQX, Piece ID: EMRORAB74LXEB6222NUCRM5ZENJC2AZNK7Q2V2NSQMVD5TIRGVGQ) context deadline exceeded; piecestore: (Node ID: 129YE2Wd6LmDABJFaGVtYDwCHmStVDnPRCagD2cKPWA5PDTtZyh, Piece ID: UEKG3FOTTORJVQRPLAFCJEP2ZAVA5N7M2DFGH4BUTB2OGVC2KIZA) context deadline exceeded ======LOGS====== upload=472552]: Uploading piece... Dec 28 02:19:02.432783751 (538h50m2.303866796s) [noise=true , address=145.79.227.33:4896 , piece_id=D44WQB3G3NBNSB6ITIBHVV67V2EKPGXXONQU25IE6RVSGGTUJ44Q, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.432842112 (538h50m2.303925147s) [noise=true , address=77.90.31.21:4900 , piece_id=UEKG3FOTTORJVQRPLAFCJEP2ZAVA5N7M2DFGH4BUTB2OGVC2KIZA, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.433283032 (538h50m2.304366077s) [noise=true , address=77.90.31.15:4703 , piece_id=BERLD7G2DBBDBNE3MCS57VXAKXL3P4RQIV47ZOCU6VXMPJU4QITA, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.433734562 (538h50m2.304817617s) [noise=true , address=77.90.31.16:4741 , piece_id=CX66NBDSLZWQ34Z2UDYM6HM73TABJSVHSGBQRONAQ3S6ENVTY3KQ, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.434178757 (538h50m2.305261802s) [noise=true , address=77.90.31.13:4646 , piece_id=V5HTB4HVJVO7OG745JBRQSISFZU6UOORKLHEVDUDUR24EWRSSVOQ, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.434595160 (538h50m2.305678195s) [noise=true , address=145.79.227.37:4596 , piece_id=XBEKEUWGAAAWZGOV3PBH7NE2CX5VID75MFGWMPHTNR7MV66U3XNA, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.435217555 (538h50m2.3063006s) [noise=true , address=45.91.116.11:4136 , piece_id=JD6QHJCQ6D53BGKDPSPYBBHNCP7XDZCSV7H6ARD3JNBL75NULT6Q, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.435684624 (538h50m2.306767669s) [noise=true , address=145.79.227.32:4856 , piece_id=II5XB6TRHNQAAHBPP5PRMMA7BCBGR5QO4HUBL4YJ7ZLLMK4DJB5Q, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.436105054 (538h50m2.307188099s) [noise=true , address=145.79.227.14:4842 , piece_id=3F6EV4TB5TQNK5W5CJHRCCNORCEEAA6AXCOJQTQ6WJNRZG3YILOQ, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.436528130 (538h50m2.307611175s) [noise=true , address=77.90.31.12:4610 , piece_id=C7IRMHAHNXDZLKFBL7OVTNVI2KWMFREMFE5GR4USVMNT5QRB3HXQ, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.437497977 (538h50m2.308581022s) [noise=true , address=145.79.227.49:4340 , piece_id=5C75E4QGOX6G7QNOUUKX4GC7YL5MESWBOFSNCA6PO4PPSMQKIVFQ, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.439092573 (538h50m2.310175618s) [noise=true , address=45.91.116.11:4134 , piece_id=D4PXHXGBZGH3UU2CNKDRCWD44IJQR5BQFWHCNQRPZDNYVV755ICQ, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.440580235 (538h50m2.31166328s) [noise=true , address=45.91.116.5:4318 , piece_id=ULPCWUUHAL35F5WZYHDQLO7VBIGAUH2XHH4I2NYYHG7OAC5VJFOA, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.442091032 (538h50m2.313174077s) [noise=true , address=77.90.31.11:4558 , piece_id=TWOTPEDKIRIOR7NQNCEFM3Z5KGWVOFUYKIQYMEGHED5FKRDCB54Q, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.443305184 (538h50m2.314388229s) [noise=true , address=145.79.227.40:4078 , piece_id=3ICUQQR7NS6O2QYHBFCAQZS7A7UAWHTD3O2LOWLRZ3OCMU3WWBSA, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.447829959 (538h50m2.318913014s) [noise=true , address=145.79.227.14:4863 , piece_id=N3XS43U6I4WJYUCC5NVUBRLTTHVPYKDR3KRCNGJRYU3E26WDO3HA, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:02.451586160 (538h50m2.322669225s) [noise=true , address=45.91.116.9:4288 , piece_id=EMRORAB74LXEB6222NUCRM5ZENJC2AZNK7Q2V2NSQMVD5TIRGVGQ, seg_pos={0 0}, upload=472552]: Uploading piece... Dec 28 02:19:03.923272933 (538h50m3.794355978s) [upload=472552]: Next returned nil segment Dec 28 02:19:03.923289645 (538h50m3.79437269s) [upload=472552]: Waiting for error group managing segments... Dec 28 02:19:04.324829202 (538h50m4.195912237s) [noise=true , address=145.79.227.49:4340 , piece_id=5C75E4QGOX6G7QNOUUKX4GC7YL5MESWBOFSNCA6PO4PPSMQKIVFQ, seg_pos={0 0}, upload=472552]: Done uploading piece. err: <nil> Dec 28 02:19:04.335200317 (538h50m4.206283363s) [noise=true , address=145.79.227.14:4842 , piece_id=3F6EV4TB5TQNK5W5CJHRCCNORCEEAA6AXCOJQTQ6WJNRZG3YILOQ, seg_pos={0 0}, upload=472552]: Done uploading piece. err: <nil> Dec 28 02:19:04.343899841 (538h50m4.214982886s) [noise=true , address=145.79.227.14:4863 , piece_id=N3XS43U6I4WJYUCC5NVUBRLTTHVPYKDR3KRCNGJRYU3E26WDO3HA, seg_pos={0 0}, upload=472552]: Done uploading piece. err: <nil> Dec 28 02:19:04.378111328 (538h50m4.249194373s) [noise=true , address=145.79.227.37:4596 , piece_id=XBEKEUWGAAAWZGOV3PBH7NE2CX5VID75MFGWMPHTNR7MV66U3XNA, seg_pos={0 0}, upload=472552]: Done uploading piece. err: <nil> Dec 28 02:19:04.381135876 (538h50m4.252218921s) [noise=true , address=145.79.227.32:4856 , piece_id=II5XB6TRHNQAAHBPP5PRMMA7BCBGR5QO4HUBL4YJ7ZLLMK4DJB5Q, seg_pos={0 0}, upload=472552]: Done uploading piece. err: <nil> Dec 28 02:19:04.476390445 (538h50m4.34747348s) [noise=true , address=145.79.227.6:4290 , piece_id=K4OGXS2MXEFVQVW7HEAZJ6267HWETOUXWJAC77JNZADW76K2XMJA, seg_pos={0 0}, upload=472552]: Done uploading piece. err: <nil> Dec 28 02:19:04.533625848 (538h50m4.404708893s) [noise=true , address=145.79.227.33:4896 , piece_id=D44WQB3G3NBNSB6ITIBHVV67V2EKPGXXONQU25IE6RVSGGTUJ44Q, seg_pos={0 0}, upload=472552]: Done uploading piece. err: <nil> Dec 28 02:19:04.562532124 (538h50m4.433615159s) [noise=true , address=77.90.31.11:4558 , piece_id=TWOTPEDKIRIOR7NQNCEFM3Z5KGWVOFUYKIQYMEGHED5FKRDCB54Q, seg_pos={0 0}, upload=472552]: Done uploading piece. err: <nil> Dec 28 02:19:04.643414509 (538h50m4.514497544s) [noise=true , address=77.90.31.21:4900 , piece_id=UEKG3FOTTORJVQRPLAFCJEP2ZAVA5N7M2DFGH4BUTB2OGVC2KIZA, seg_pos={0 0}, upload=472552]: Done uploading piece. err: <nil> Dec 28 02:19:04.651013327 (538h50m4.522096372s) [noise=true , address=77.90.31.13:4646 , piece_id=V5HTB4HVJVO7OG745JBRQSISFZU6UOORKLHEVDUDUR24EWRSSVOQ, seg_pos={0 0}, upload=472552]: Done uploading piece. err: <nil> Dec 28 02:19:04.776667164 (538h50m4.647750209s) [noise=true , address=45.91.116.11:4136 , piece_id=JD6QHJCQ6D53BGKDPSPYBBHNCP7XDZCSV7H6ARD3JNBL75NULT6Q, seg_pos={0 0}, upload=472552]: Done uploading piece. err: <nil> Dec 28 02:19:04.919205529 (538h50m4.790288574s) [noise=true , address=45.91.116.11:4134 , piece_id=D4PXHXGBZGH3UU2CNKDRCWD44IJQR5BQFWHCNQRPZDNYVV755ICQ, seg_pos={0 0}, upload=472552]: Done uploading piece. err: <nil> Dec 28 02:19:04.927993490 (538h50m4.799076535s) [noise=true , address=45.91.116.9:4288 , piece_id=EMRORAB74LXEB6222NUCRM5ZENJC2AZNK7Q2V2NSQMVD5TIRGVGQ, seg_pos={0 0}, upload=472552]: Done uploading piece. err: <nil> Dec 28 02:19:05.041729347 (538h50m4.912812392s) [noise=true , address=45.91.116.5:4318 , piece_id=ULPCWUUHAL35F5WZYHDQLO7VBIGAUH2XHH4I2NYYHG7OAC5VJFOA, seg_pos={0 0}, upload=472552]: Done uploading piece. err: <nil> Dec 28 02:19:05.041751558 (538h50m4.912834593s) [seg_pos={0 0}, upload=472552]: Segment reached optimal threshold of 14 pieces. Dec 28 02:19:05.041986606 (538h50m4.913069651s) [noise=true , address=77.90.31.12:4610 , piece_id=C7IRMHAHNXDZLKFBL7OVTNVI2KWMFREMFE5GR4USVMNT5QRB3HXQ, seg_pos={0 0}, upload=472552]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi): protocol: expected piece hash; context canceled; EOF Dec 28 02:19:05.041995954 (538h50m4.913078999s) [noise=true , address=77.90.31.12:4610 , piece_id=C7IRMHAHNXDZLKFBL7OVTNVI2KWMFREMFE5GR4USVMNT5QRB3HXQ, seg_pos={0 0}, upload=472552]: Long tail cancellation due to optimal threshold Dec 28 02:19:05.042149837 (538h50m4.913232872s) [noise=true , address=77.90.31.15:4703 , piece_id=BERLD7G2DBBDBNE3MCS57VXAKXL3P4RQIV47ZOCU6VXMPJU4QITA, seg_pos={0 0}, upload=472552]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hMDD5K7WDjvN4ZgcpwpnPFb6X1ncCfyucfFrpTUshd8Ye1B6D): protocol: expected piece hash; context canceled; EOF Dec 28 02:19:05.042158714 (538h50m4.913241759s) [noise=true , address=77.90.31.15:4703 , piece_id=BERLD7G2DBBDBNE3MCS57VXAKXL3P4RQIV47ZOCU6VXMPJU4QITA, seg_pos={0 0}, upload=472552]: Long tail cancellation due to optimal threshold Dec 28 02:19:05.042299752 (538h50m4.913382788s) [noise=true , address=77.90.31.16:4741 , piece_id=CX66NBDSLZWQ34Z2UDYM6HM73TABJSVHSGBQRONAQ3S6ENVTY3KQ, seg_pos={0 0}, upload=472552]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR): protocol: expected piece hash; context canceled; EOF Dec 28 02:19:05.042304702 (538h50m4.913387737s) [noise=true , address=77.90.31.16:4741 , piece_id=CX66NBDSLZWQ34Z2UDYM6HM73TABJSVHSGBQRONAQ3S6ENVTY3KQ, seg_pos={0 0}, upload=472552]: Long tail cancellation due to optimal threshold Dec 28 02:19:05.042396276 (538h50m4.913479352s) [noise=true , address=145.79.227.40:4078 , piece_id=3ICUQQR7NS6O2QYHBFCAQZS7A7UAWHTD3O2LOWLRZ3OCMU3WWBSA, seg_pos={0 0}, upload=472552]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tzgtenV6us5ssD7xWGpZmVUoWEfbFhBNewbK9irHtZPkyDtRd): protocol: expected piece hash; context canceled; EOF Dec 28 02:19:05.042417406 (538h50m4.913500461s) [noise=true , address=145.79.227.40:4078 , piece_id=3ICUQQR7NS6O2QYHBFCAQZS7A7UAWHTD3O2LOWLRZ3OCMU3WWBSA, seg_pos={0 0}, upload=472552]: Long tail cancellation due to optimal threshold Dec 28 02:19:05.042612779 (538h50m4.913695814s) [seg_pos={0 0}, upload=472552]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 02:19:05.042821726 (538h50m4.913904771s) [upload=472552]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:19:05.042916818 (538h50m4.913999863s) [upload=472552]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:19:05.336925913 (538h50m5.208008958s) [upload=472552]: Done uploading segments... Dec 28 02:19:05.336953856 (538h50m5.208036901s) [upload=472552]: Upload finished. err: <nil> Duration: 1m0.003380327s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 31 out of 60 is failed (threshold=3) [.XXXXX..X.XXX.X.XX.XX.X..X.X.X.X.X..X...XX..X.XXX.XXX.X.....] duration: 6.014814775s trace: https://jaeger.dev.storj.io/trace/1c22d34f4ac03822 FAIL (2025-12-28 02:19:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: GU4X73Q6AFIBRIYAG4ASNF22MDMCKUOKIJCS3TTJT3FKDWLAMIKA) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: DQ4IHJQYEPBEW63HH3ELSFGDFZUHJ7HL6UHBLGQRHKOHTD7HP2SQ) context deadline exceeded ======LOGS====== Dec 28 02:15:53.759588852 (538h46m53.630671907s) [upload=472506]: Starting upload Dec 28 02:15:53.759803180 (538h46m53.630886235s) [upload=472506]: Done starting upload Dec 28 02:15:53.759860660 (538h46m53.630943705s) [upload=472506]: Uploading segments... Dec 28 02:15:53.760077182 (538h46m53.631160227s) [upload=472506]: Got next segment. Inline: false Dec 28 02:15:53.760126496 (538h46m53.631209541s) [upload=472506]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 02:15:53.760133219 (538h46m53.631216264s) [upload=472506]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 02:15:54.003115733 (538h46m53.874198788s) [seg_pos={0 0}, upload=472506]: Begin upload segment... Dec 28 02:15:54.003896579 (538h46m53.874979644s) [seg_pos={0 0}, upload=472506]: Done begin upload segment. Dec 28 02:15:54.003960000 (538h46m53.875043055s) [noise=true , address=145.79.227.16:4909 , piece_id=KSGRHS6G6JZ7IY23T277NMHN6HGG7DUODYBIXENFS2PSTYNUX37Q, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.003967354 (538h46m53.875050399s) [noise=true , address=145.79.227.52:4445 , piece_id=ZEY2QW4N6ZNGBILKIDQ7YBFSB7U2JVICZJV2W3GQU7D3QACB32FQ, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.004662528 (538h46m53.875745573s) [noise=true , address=145.79.227.31:4797 , piece_id=J5OPI7TBATORMGHB245L3YCWH2I56AVGHVAWJRLALMYDQIW7YAGQ, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.004851216 (538h46m53.875934261s) [noise=true , address=45.91.116.11:4118 , piece_id=LA7UXPHDDM25MJ4IKX4IGJXKZFA37TTXYBAGMA25C3Y57X6YFQ3Q, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.005240899 (538h46m53.876323944s) [noise=true , address=145.79.227.4:4185 , piece_id=SKBIGBFEIKWVJJFKWFM6TO5CYQ6RCM6RLV6AXCM4TKG7TS6TAVXQ, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.005594032 (538h46m53.876677067s) [noise=true , address=77.90.31.16:4735 , piece_id=VD6RDROPI6XQA2TYQUB4ENXFD2B6OCM2YH5AXP3OVGPQ5YBI37BA, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.005750930 (538h46m53.876833975s) [noise=true , address=145.79.227.18:4006 , piece_id=GYNQ6GGPHVJJYRV2RHRL63SFS4UPSONZY6PPPVC4ZTSOK5SVWVLQ, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.006059718 (538h46m53.877142793s) [noise=true , address=45.91.116.14:4190 , piece_id=ZDWBLJMWUZN4ZMK5X6FWN24VEWAFU2KODQJ7URBZBWJHIML4GKKA, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.006277232 (538h46m53.877360277s) [noise=true , address=145.79.227.17:4972 , piece_id=64UKV76NMUYNSLPZA3WLTQ6F7W2XP5Y564DFNQMLBFB3SG6S3IFA, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.006515977 (538h46m53.877599022s) [noise=true , address=45.91.116.6:4973 , piece_id=GU4X73Q6AFIBRIYAG4ASNF22MDMCKUOKIJCS3TTJT3FKDWLAMIKA, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.007002723 (538h46m53.878085789s) [noise=true , address=145.79.227.19:4077 , piece_id=4LCN23MUWAX2HK3J2OFAFLEGUQM4YYTJWZ7IPCN3D6QROORY5O6A, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.007095439 (538h46m53.878178484s) [noise=true , address=77.90.31.14:4661 , piece_id=U2S5KQX5UAKMMMIOZH6CP7SE7PR2DZIKHKX7OXBCJMSZKG4RUMMQ, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.007443692 (538h46m53.878526728s) [noise=true , address=45.91.116.7:4009 , piece_id=DQ4IHJQYEPBEW63HH3ELSFGDFZUHJ7HL6UHBLGQRHKOHTD7HP2SQ, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.007675033 (538h46m53.878758078s) [noise=true , address=145.79.227.32:4836 , piece_id=AZ5NCPW2VX2VSH7XGAUEWC2B25ZEQ2NVQVZZUL6CANSCBWIIEY3A, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.007858693 (538h46m53.878941738s) [noise=true , address=145.79.227.27:4691 , piece_id=MJMBV5LR4RA5ZP7Q7U34DXO2TWJ6LDKH4YMADM3K5CU2KRHMQWBA, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.008204601 (538h46m53.879287637s) [noise=true , address=145.79.227.12:4767 , piece_id=A3L6CAJZWVR4RYOZMKQ5C5UALP6FKW367UGSOCW3YEVLWMYJBBTQ, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:55.745297151 (538h46m55.616380186s) [upload=472506]: Next returned nil segment Dec 28 02:15:55.745351053 (538h46m55.616434108s) [upload=472506]: Waiting for error group managing segments... Dec 28 02:15:56.032845416 (538h46m55.903928451s) [noise=true , address=145.79.227.18:4006 , piece_id=GYNQ6GGPHVJJYRV2RHRL63SFS4UPSONZY6PPPVC4ZTSOK5SVWVLQ, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.202191721 (538h46m56.073274756s) [noise=true , address=145.79.227.19:4077 , piece_id=4LCN23MUWAX2HK3J2OFAFLEGUQM4YYTJWZ7IPCN3D6QROORY5O6A, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.205182144 (538h46m56.076265189s) [noise=true , address=145.79.227.31:4797 , piece_id=J5OPI7TBATORMGHB245L3YCWH2I56AVGHVAWJRLALMYDQIW7YAGQ, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.205433684 (538h46m56.076516719s) [noise=true , address=145.79.227.27:4691 , piece_id=MJMBV5LR4RA5ZP7Q7U34DXO2TWJ6LDKH4YMADM3K5CU2KRHMQWBA, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.214467002 (538h46m56.085550047s) [noise=true , address=145.79.227.16:4909 , piece_id=KSGRHS6G6JZ7IY23T277NMHN6HGG7DUODYBIXENFS2PSTYNUX37Q, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.214875340 (538h46m56.085958375s) [noise=true , address=145.79.227.4:4185 , piece_id=SKBIGBFEIKWVJJFKWFM6TO5CYQ6RCM6RLV6AXCM4TKG7TS6TAVXQ, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.217932621 (538h46m56.089015656s) [noise=true , address=145.79.227.12:4767 , piece_id=A3L6CAJZWVR4RYOZMKQ5C5UALP6FKW367UGSOCW3YEVLWMYJBBTQ, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.306475972 (538h46m56.177559007s) [noise=true , address=45.91.116.6:4973 , piece_id=GU4X73Q6AFIBRIYAG4ASNF22MDMCKUOKIJCS3TTJT3FKDWLAMIKA, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.385055362 (538h46m56.256138408s) [noise=true , address=45.91.116.7:4009 , piece_id=DQ4IHJQYEPBEW63HH3ELSFGDFZUHJ7HL6UHBLGQRHKOHTD7HP2SQ, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.395589739 (538h46m56.266672784s) [noise=true , address=45.91.116.14:4190 , piece_id=ZDWBLJMWUZN4ZMK5X6FWN24VEWAFU2KODQJ7URBZBWJHIML4GKKA, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.395681062 (538h46m56.266764107s) [noise=true , address=145.79.227.52:4445 , piece_id=ZEY2QW4N6ZNGBILKIDQ7YBFSB7U2JVICZJV2W3GQU7D3QACB32FQ, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.403044182 (538h46m56.274127217s) [noise=true , address=145.79.227.32:4836 , piece_id=AZ5NCPW2VX2VSH7XGAUEWC2B25ZEQ2NVQVZZUL6CANSCBWIIEY3A, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:57.110570880 (538h46m56.981653925s) [noise=true , address=45.91.116.11:4118 , piece_id=LA7UXPHDDM25MJ4IKX4IGJXKZFA37TTXYBAGMA25C3Y57X6YFQ3Q, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:57.110788534 (538h46m56.981871569s) [seg_pos={0 0}, upload=472506]: Segment reached optimal threshold of 13 pieces. Dec 28 02:15:57.111013823 (538h46m56.982096858s) [noise=true , address=77.90.31.14:4661 , piece_id=U2S5KQX5UAKMMMIOZH6CP7SE7PR2DZIKHKX7OXBCJMSZKG4RUMMQ, seg_pos={0 0}, upload=472506]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 28 02:15:57.111022289 (538h46m56.982105335s) [noise=true , address=77.90.31.14:4661 , piece_id=U2S5KQX5UAKMMMIOZH6CP7SE7PR2DZIKHKX7OXBCJMSZKG4RUMMQ, seg_pos={0 0}, upload=472506]: Long tail cancellation due to optimal threshold Dec 28 02:15:57.111365041 (538h46m56.982448076s) [noise=true , address=77.90.31.16:4735 , piece_id=VD6RDROPI6XQA2TYQUB4ENXFD2B6OCM2YH5AXP3OVGPQ5YBI37BA, seg_pos={0 0}, upload=472506]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: read tcp 188.166.225.170:47380->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 02:15:57.111372196 (538h46m56.982455241s) [noise=true , address=77.90.31.16:4735 , piece_id=VD6RDROPI6XQA2TYQUB4ENXFD2B6OCM2YH5AXP3OVGPQ5YBI37BA, seg_pos={0 0}, upload=472506]: Long tail cancellation due to optimal threshold Dec 28 02:15:57.111468879 (538h46m56.982551924s) [noise=true , address=145.79.227.17:4972 , piece_id=64UKV76NMUYNSLPZA3WLTQ6F7W2XP5Y564DFNQMLBFB3SG6S3IFA, seg_pos={0 0}, upload=472506]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: expected piece hash; context canceled; EOF Dec 28 02:15:57.111474270 (538h46m56.982557315s) [noise=true , address=145.79.227.17:4972 , piece_id=64UKV76NMUYNSLPZA3WLTQ6F7W2XP5Y564DFNQMLBFB3SG6S3IFA, seg_pos={0 0}, upload=472506]: Long tail cancellation due to optimal threshold Dec 28 02:15:57.111493035 (538h46m56.98257607s) [seg_pos={0 0}, upload=472506]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 02:15:57.111624576 (538h46m56.982707611s) [upload=472506]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:15:57.111630667 (538h46m56.982713713s) [upload=472506]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:15:57.409023835 (538h46m57.28010687s) [upload=472506]: Done uploading segments... Dec 28 02:15:57.409043784 (538h46m57.280126819s) [upload=472506]: Upload finished. err: <nil> Duration: 6.014947757s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 32 out of 60 is failed (threshold=3) [X.XXXXX..X.XXX.X.XX.XX.X..X.X.X.X.X..X...XX..X.XXX.XXX.X....] duration: 18.547683991s trace: https://jaeger.dev.storj.io/trace/1c22d34f4ac03822 FAIL (2025-12-28 02:19:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: GU4X73Q6AFIBRIYAG4ASNF22MDMCKUOKIJCS3TTJT3FKDWLAMIKA) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: DQ4IHJQYEPBEW63HH3ELSFGDFZUHJ7HL6UHBLGQRHKOHTD7HP2SQ) context deadline exceeded ======LOGS====== Dec 28 02:15:53.759588852 (538h46m53.630671907s) [upload=472506]: Starting upload Dec 28 02:15:53.759803180 (538h46m53.630886235s) [upload=472506]: Done starting upload Dec 28 02:15:53.759860660 (538h46m53.630943705s) [upload=472506]: Uploading segments... Dec 28 02:15:53.760077182 (538h46m53.631160227s) [upload=472506]: Got next segment. Inline: false Dec 28 02:15:53.760126496 (538h46m53.631209541s) [upload=472506]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 02:15:53.760133219 (538h46m53.631216264s) [upload=472506]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 02:15:54.003115733 (538h46m53.874198788s) [seg_pos={0 0}, upload=472506]: Begin upload segment... Dec 28 02:15:54.003896579 (538h46m53.874979644s) [seg_pos={0 0}, upload=472506]: Done begin upload segment. Dec 28 02:15:54.003960000 (538h46m53.875043055s) [noise=true , address=145.79.227.16:4909 , piece_id=KSGRHS6G6JZ7IY23T277NMHN6HGG7DUODYBIXENFS2PSTYNUX37Q, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.003967354 (538h46m53.875050399s) [noise=true , address=145.79.227.52:4445 , piece_id=ZEY2QW4N6ZNGBILKIDQ7YBFSB7U2JVICZJV2W3GQU7D3QACB32FQ, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.004662528 (538h46m53.875745573s) [noise=true , address=145.79.227.31:4797 , piece_id=J5OPI7TBATORMGHB245L3YCWH2I56AVGHVAWJRLALMYDQIW7YAGQ, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.004851216 (538h46m53.875934261s) [noise=true , address=45.91.116.11:4118 , piece_id=LA7UXPHDDM25MJ4IKX4IGJXKZFA37TTXYBAGMA25C3Y57X6YFQ3Q, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.005240899 (538h46m53.876323944s) [noise=true , address=145.79.227.4:4185 , piece_id=SKBIGBFEIKWVJJFKWFM6TO5CYQ6RCM6RLV6AXCM4TKG7TS6TAVXQ, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.005594032 (538h46m53.876677067s) [noise=true , address=77.90.31.16:4735 , piece_id=VD6RDROPI6XQA2TYQUB4ENXFD2B6OCM2YH5AXP3OVGPQ5YBI37BA, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.005750930 (538h46m53.876833975s) [noise=true , address=145.79.227.18:4006 , piece_id=GYNQ6GGPHVJJYRV2RHRL63SFS4UPSONZY6PPPVC4ZTSOK5SVWVLQ, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.006059718 (538h46m53.877142793s) [noise=true , address=45.91.116.14:4190 , piece_id=ZDWBLJMWUZN4ZMK5X6FWN24VEWAFU2KODQJ7URBZBWJHIML4GKKA, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.006277232 (538h46m53.877360277s) [noise=true , address=145.79.227.17:4972 , piece_id=64UKV76NMUYNSLPZA3WLTQ6F7W2XP5Y564DFNQMLBFB3SG6S3IFA, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.006515977 (538h46m53.877599022s) [noise=true , address=45.91.116.6:4973 , piece_id=GU4X73Q6AFIBRIYAG4ASNF22MDMCKUOKIJCS3TTJT3FKDWLAMIKA, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.007002723 (538h46m53.878085789s) [noise=true , address=145.79.227.19:4077 , piece_id=4LCN23MUWAX2HK3J2OFAFLEGUQM4YYTJWZ7IPCN3D6QROORY5O6A, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.007095439 (538h46m53.878178484s) [noise=true , address=77.90.31.14:4661 , piece_id=U2S5KQX5UAKMMMIOZH6CP7SE7PR2DZIKHKX7OXBCJMSZKG4RUMMQ, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.007443692 (538h46m53.878526728s) [noise=true , address=45.91.116.7:4009 , piece_id=DQ4IHJQYEPBEW63HH3ELSFGDFZUHJ7HL6UHBLGQRHKOHTD7HP2SQ, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.007675033 (538h46m53.878758078s) [noise=true , address=145.79.227.32:4836 , piece_id=AZ5NCPW2VX2VSH7XGAUEWC2B25ZEQ2NVQVZZUL6CANSCBWIIEY3A, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.007858693 (538h46m53.878941738s) [noise=true , address=145.79.227.27:4691 , piece_id=MJMBV5LR4RA5ZP7Q7U34DXO2TWJ6LDKH4YMADM3K5CU2KRHMQWBA, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.008204601 (538h46m53.879287637s) [noise=true , address=145.79.227.12:4767 , piece_id=A3L6CAJZWVR4RYOZMKQ5C5UALP6FKW367UGSOCW3YEVLWMYJBBTQ, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:55.745297151 (538h46m55.616380186s) [upload=472506]: Next returned nil segment Dec 28 02:15:55.745351053 (538h46m55.616434108s) [upload=472506]: Waiting for error group managing segments... Dec 28 02:15:56.032845416 (538h46m55.903928451s) [noise=true , address=145.79.227.18:4006 , piece_id=GYNQ6GGPHVJJYRV2RHRL63SFS4UPSONZY6PPPVC4ZTSOK5SVWVLQ, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.202191721 (538h46m56.073274756s) [noise=true , address=145.79.227.19:4077 , piece_id=4LCN23MUWAX2HK3J2OFAFLEGUQM4YYTJWZ7IPCN3D6QROORY5O6A, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.205182144 (538h46m56.076265189s) [noise=true , address=145.79.227.31:4797 , piece_id=J5OPI7TBATORMGHB245L3YCWH2I56AVGHVAWJRLALMYDQIW7YAGQ, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.205433684 (538h46m56.076516719s) [noise=true , address=145.79.227.27:4691 , piece_id=MJMBV5LR4RA5ZP7Q7U34DXO2TWJ6LDKH4YMADM3K5CU2KRHMQWBA, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.214467002 (538h46m56.085550047s) [noise=true , address=145.79.227.16:4909 , piece_id=KSGRHS6G6JZ7IY23T277NMHN6HGG7DUODYBIXENFS2PSTYNUX37Q, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.214875340 (538h46m56.085958375s) [noise=true , address=145.79.227.4:4185 , piece_id=SKBIGBFEIKWVJJFKWFM6TO5CYQ6RCM6RLV6AXCM4TKG7TS6TAVXQ, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.217932621 (538h46m56.089015656s) [noise=true , address=145.79.227.12:4767 , piece_id=A3L6CAJZWVR4RYOZMKQ5C5UALP6FKW367UGSOCW3YEVLWMYJBBTQ, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.306475972 (538h46m56.177559007s) [noise=true , address=45.91.116.6:4973 , piece_id=GU4X73Q6AFIBRIYAG4ASNF22MDMCKUOKIJCS3TTJT3FKDWLAMIKA, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.385055362 (538h46m56.256138408s) [noise=true , address=45.91.116.7:4009 , piece_id=DQ4IHJQYEPBEW63HH3ELSFGDFZUHJ7HL6UHBLGQRHKOHTD7HP2SQ, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.395589739 (538h46m56.266672784s) [noise=true , address=45.91.116.14:4190 , piece_id=ZDWBLJMWUZN4ZMK5X6FWN24VEWAFU2KODQJ7URBZBWJHIML4GKKA, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.395681062 (538h46m56.266764107s) [noise=true , address=145.79.227.52:4445 , piece_id=ZEY2QW4N6ZNGBILKIDQ7YBFSB7U2JVICZJV2W3GQU7D3QACB32FQ, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.403044182 (538h46m56.274127217s) [noise=true , address=145.79.227.32:4836 , piece_id=AZ5NCPW2VX2VSH7XGAUEWC2B25ZEQ2NVQVZZUL6CANSCBWIIEY3A, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:57.110570880 (538h46m56.981653925s) [noise=true , address=45.91.116.11:4118 , piece_id=LA7UXPHDDM25MJ4IKX4IGJXKZFA37TTXYBAGMA25C3Y57X6YFQ3Q, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:57.110788534 (538h46m56.981871569s) [seg_pos={0 0}, upload=472506]: Segment reached optimal threshold of 13 pieces. Dec 28 02:15:57.111013823 (538h46m56.982096858s) [noise=true , address=77.90.31.14:4661 , piece_id=U2S5KQX5UAKMMMIOZH6CP7SE7PR2DZIKHKX7OXBCJMSZKG4RUMMQ, seg_pos={0 0}, upload=472506]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 28 02:15:57.111022289 (538h46m56.982105335s) [noise=true , address=77.90.31.14:4661 , piece_id=U2S5KQX5UAKMMMIOZH6CP7SE7PR2DZIKHKX7OXBCJMSZKG4RUMMQ, seg_pos={0 0}, upload=472506]: Long tail cancellation due to optimal threshold Dec 28 02:15:57.111365041 (538h46m56.982448076s) [noise=true , address=77.90.31.16:4735 , piece_id=VD6RDROPI6XQA2TYQUB4ENXFD2B6OCM2YH5AXP3OVGPQ5YBI37BA, seg_pos={0 0}, upload=472506]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: read tcp 188.166.225.170:47380->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 02:15:57.111372196 (538h46m56.982455241s) [noise=true , address=77.90.31.16:4735 , piece_id=VD6RDROPI6XQA2TYQUB4ENXFD2B6OCM2YH5AXP3OVGPQ5YBI37BA, seg_pos={0 0}, upload=472506]: Long tail cancellation due to optimal threshold Dec 28 02:15:57.111468879 (538h46m56.982551924s) [noise=true , address=145.79.227.17:4972 , piece_id=64UKV76NMUYNSLPZA3WLTQ6F7W2XP5Y564DFNQMLBFB3SG6S3IFA, seg_pos={0 0}, upload=472506]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: expected piece hash; context canceled; EOF Dec 28 02:15:57.111474270 (538h46m56.982557315s) [noise=true , address=145.79.227.17:4972 , piece_id=64UKV76NMUYNSLPZA3WLTQ6F7W2XP5Y564DFNQMLBFB3SG6S3IFA, seg_pos={0 0}, upload=472506]: Long tail cancellation due to optimal threshold Dec 28 02:15:57.111493035 (538h46m56.98257607s) [seg_pos={0 0}, upload=472506]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 02:15:57.111624576 (538h46m56.982707611s) [upload=472506]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:15:57.111630667 (538h46m56.982713713s) [upload=472506]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:15:57.409023835 (538h46m57.28010687s) [upload=472506]: Done uploading segments... Dec 28 02:15:57.409043784 (538h46m57.280126819s) [upload=472506]: Upload finished. err: <nil> Duration: 18.547747333s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 43.400881464s trace: https://jaeger.dev.storj.io/trace/12a1d059b6f75af7 FAIL (2025-12-28 02:19:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1.145865281 (538h33m41.016948316s) [noise=true , address=77.90.31.10:4551 , piece_id=KBPO4MUDASHKMPRNJELQWLNMJNQBB3PZUOVNCEPP2CYVHU6X2PPA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.146408725 (538h33m41.01749177s) [noise=true , address=145.79.227.15:4908 , piece_id=I2FSDNVYBFZKRXZNUHH2APY7N2PI7DPAKFKGXRJW2UFU2PBKSSNQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.146778139 (538h33m41.017861184s) [noise=true , address=145.79.227.43:4161 , piece_id=ZN3T6JSPHJ57X4YYG5MHRJYYOFPH2T2CSGJ34MJQU55HDC7CMOXQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.147137874 (538h33m41.018220919s) [noise=true , address=77.90.31.21:4882 , piece_id=WQAGQTGLU5HRKEET36W34MQP4NPGSVCA4QZ6YEGVAJWBBPLZOUQQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.147426784 (538h33m41.018509819s) [noise=true , address=145.79.227.35:4503 , piece_id=MDYJ2TROT4FHLX4XHKHEQACC7ZXHZCQ5MUPJL5EDS3F2YMXSUGUA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.147466109 (538h33m41.018549154s) [noise=true , address=77.90.31.15:4730 , piece_id=EN7IGRJZ2WFN4HXJPMNLLXDUZPIN7KDUGKPPLJH7CSA4SDOUM2JA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.147817368 (538h33m41.018900413s) [noise=true , address=77.90.31.18:4833 , piece_id=3E57LWLI3WHROFGMXVZ7LHDHO5O7453LWQXA3OEKM52B2NIYM2JA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.147860690 (538h33m41.018943745s) [noise=true , address=145.79.227.44:4212 , piece_id=EFTTQM6323MUQ5WTGY6DFBTF3TGRM6XZWXIGT6DM4VS7GRGM4C2Q, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148163427 (538h33m41.019246472s) [noise=true , address=77.90.31.19:4862 , piece_id=PSBBBSGY7THVLMXJQCADIE7VYZ33OIIDVAEQWG3RWMNA6RSKELNQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148189276 (538h33m41.019272321s) [noise=true , address=77.90.31.10:4524 , piece_id=X2VCBS4PTPPWI6U64T4FVIULRLCDY3XUOJTXJNP34OHSO54ONIWA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148496531 (538h33m41.019579566s) [noise=true , address=145.79.227.23:4556 , piece_id=AKBXRSU5PM5QLHGHE3VXTJ3ZH23BECPGBLQSVZK6XVIYJGVDXT4Q, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148527380 (538h33m41.019610425s) [noise=true , address=145.79.227.33:4899 , piece_id=JFEXXK22EATV7UUH7LLXTFEPIUBW7EWSNYFXLE4AO5F6SLYFTGBA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148801512 (538h33m41.019884547s) [noise=true , address=77.90.31.14:4672 , piece_id=BZG2Y5X47J2WVFF5LBHEJZNUD2OQ3NOTVPTCAVT4TLFQB5FWESLA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148894920 (538h33m41.019977965s) [noise=true , address=5.231.73.5:4016 , piece_id=YCNKQEKJQSJHKKGCYWLOLEBKIVPJHZIJ6TYEG6DF4TRXMZ3ZTTIQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.149118806 (538h33m41.020201851s) [noise=true , address=145.79.227.47:4295 , piece_id=OM7B75MCPM7L3WP5MLQ7MAVBZ2NJUSISOWD56RSWI2AV2OWVDRTA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.149274993 (538h33m41.020358038s) [noise=true , address=145.79.227.50:4400 , piece_id=HZAT7YAVZYWK52FE4TIC6OA37U4VX5B35H55WMBO3MMWNIKTJLJQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.149625842 (538h33m41.020708877s) [noise=true , address=145.79.227.34:4915 , piece_id=RMXYZGGJA5EPJLQD6SQG677HFLOQWRY2HFTGEOICYAQG6QR557RQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:42.618098549 (538h33m42.489181584s) [upload=472313]: Next returned nil segment Dec 28 02:02:42.618120911 (538h33m42.489203946s) [upload=472313]: Waiting for error group managing segments... Dec 28 02:02:43.021704832 (538h33m42.892787877s) [noise=true , address=145.79.227.23:4556 , piece_id=AKBXRSU5PM5QLHGHE3VXTJ3ZH23BECPGBLQSVZK6XVIYJGVDXT4Q, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.027980492 (538h33m42.899063537s) [noise=true , address=145.79.227.50:4400 , piece_id=HZAT7YAVZYWK52FE4TIC6OA37U4VX5B35H55WMBO3MMWNIKTJLJQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.043329239 (538h33m42.914412284s) [noise=true , address=145.79.227.15:4908 , piece_id=I2FSDNVYBFZKRXZNUHH2APY7N2PI7DPAKFKGXRJW2UFU2PBKSSNQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.056308314 (538h33m42.927391349s) [noise=true , address=145.79.227.49:4347 , piece_id=WURIHGQASBLSVLZJVM5EJHWD7U4VFLLPGJIFYNOK6ALA5LYEQDSA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.074628196 (538h33m42.945711252s) [noise=true , address=145.79.227.47:4295 , piece_id=OM7B75MCPM7L3WP5MLQ7MAVBZ2NJUSISOWD56RSWI2AV2OWVDRTA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.080495740 (538h33m42.951578775s) [noise=true , address=5.231.73.5:4016 , piece_id=YCNKQEKJQSJHKKGCYWLOLEBKIVPJHZIJ6TYEG6DF4TRXMZ3ZTTIQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.193955176 (538h33m43.065038241s) [noise=true , address=77.90.31.10:4551 , piece_id=KBPO4MUDASHKMPRNJELQWLNMJNQBB3PZUOVNCEPP2CYVHU6X2PPA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.301513865 (538h33m43.1725969s) [noise=true , address=77.90.31.15:4730 , piece_id=EN7IGRJZ2WFN4HXJPMNLLXDUZPIN7KDUGKPPLJH7CSA4SDOUM2JA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.305020622 (538h33m43.176103657s) [noise=true , address=77.90.31.19:4862 , piece_id=PSBBBSGY7THVLMXJQCADIE7VYZ33OIIDVAEQWG3RWMNA6RSKELNQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.312187619 (538h33m43.183270654s) [noise=true , address=77.90.31.14:4672 , piece_id=BZG2Y5X47J2WVFF5LBHEJZNUD2OQ3NOTVPTCAVT4TLFQB5FWESLA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.319277390 (538h33m43.190360425s) [noise=true , address=77.90.31.21:4882 , piece_id=WQAGQTGLU5HRKEET36W34MQP4NPGSVCA4QZ6YEGVAJWBBPLZOUQQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.948916959 (538h33m43.820000004s) [noise=true , address=145.79.227.34:4915 , piece_id=RMXYZGGJA5EPJLQD6SQG677HFLOQWRY2HFTGEOICYAQG6QR557RQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.989823331 (538h33m43.860906366s) [noise=true , address=145.79.227.44:4212 , piece_id=EFTTQM6323MUQ5WTGY6DFBTF3TGRM6XZWXIGT6DM4VS7GRGM4C2Q, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:44.123218949 (538h33m43.994301984s) [noise=true , address=145.79.227.43:4161 , piece_id=ZN3T6JSPHJ57X4YYG5MHRJYYOFPH2T2CSGJ34MJQU55HDC7CMOXQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:44.123238386 (538h33m43.994321421s) [seg_pos={0 0}, upload=472313]: Segment reached optimal threshold of 14 pieces. Dec 28 02:02:44.123431754 (538h33m43.994514789s) [noise=true , address=145.79.227.33:4899 , piece_id=JFEXXK22EATV7UUH7LLXTFEPIUBW7EWSNYFXLE4AO5F6SLYFTGBA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b4pZVQRnjcQNn45VTgdsz5fpjYPMAzSXGR5KJ5BP9XHMkRnRw): protocol: expected piece hash; context canceled; EOF Dec 28 02:02:44.123437586 (538h33m43.994520621s) [noise=true , address=145.79.227.33:4899 , piece_id=JFEXXK22EATV7UUH7LLXTFEPIUBW7EWSNYFXLE4AO5F6SLYFTGBA, seg_pos={0 0}, upload=472313]: Long tail cancellation due to optimal threshold Dec 28 02:02:44.123631064 (538h33m43.994714109s) [noise=true , address=77.90.31.18:4833 , piece_id=3E57LWLI3WHROFGMXVZ7LHDHO5O7453LWQXA3OEKM52B2NIYM2JA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FbVbo8kGy1aaFyCk2noiXNUJHtosFJHFykBMRecGd5Kj5gjEp): protocol: expected piece hash; context canceled; EOF Dec 28 02:02:44.123675267 (538h33m43.994758313s) [noise=true , address=77.90.31.18:4833 , piece_id=3E57LWLI3WHROFGMXVZ7LHDHO5O7453LWQXA3OEKM52B2NIYM2JA, seg_pos={0 0}, upload=472313]: Long tail cancellation due to optimal threshold Dec 28 02:02:44.123774066 (538h33m43.994857101s) [noise=true , address=77.90.31.10:4524 , piece_id=X2VCBS4PTPPWI6U64T4FVIULRLCDY3XUOJTXJNP34OHSO54ONIWA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe): protocol: expected piece hash; context canceled; EOF Dec 28 02:02:44.123781761 (538h33m43.994864796s) [noise=true , address=77.90.31.10:4524 , piece_id=X2VCBS4PTPPWI6U64T4FVIULRLCDY3XUOJTXJNP34OHSO54ONIWA, seg_pos={0 0}, upload=472313]: Long tail cancellation due to optimal threshold Dec 28 02:02:44.123923341 (538h33m43.995006376s) [noise=true , address=145.79.227.35:4503 , piece_id=MDYJ2TROT4FHLX4XHKHEQACC7ZXHZCQ5MUPJL5EDS3F2YMXSUGUA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ctmSF1aMQasuXByFW8DppDantREmmWL2WUjmKGW5ZULVePyvm): protocol: expected piece hash; context canceled; EOF Dec 28 02:02:44.123928591 (538h33m43.995011636s) [noise=true , address=145.79.227.35:4503 , piece_id=MDYJ2TROT4FHLX4XHKHEQACC7ZXHZCQ5MUPJL5EDS3F2YMXSUGUA, seg_pos={0 0}, upload=472313]: Long tail cancellation due to optimal threshold Dec 28 02:02:44.123944731 (538h33m43.995027766s) [seg_pos={0 0}, upload=472313]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 02:02:44.124017179 (538h33m43.995100214s) [upload=472313]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:02:44.124022469 (538h33m43.995105504s) [upload=472313]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:02:44.409047274 (538h33m44.280130349s) [upload=472313]: Done uploading segments... Dec 28 02:02:44.409073354 (538h33m44.280156389s) [upload=472313]: Upload finished. err: <nil> Duration: 43.400901101s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................................X.................] duration: 3.629032461s trace: https://jaeger.dev.storj.io/trace/7fc142ad3d711e34 FAIL (2025-12-28 02:18:54 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.629060835s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 51.46678748s trace: https://jaeger.dev.storj.io/trace/12a1d059b6f75af7 FAIL (2025-12-28 02:18:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1.145865281 (538h33m41.016948316s) [noise=true , address=77.90.31.10:4551 , piece_id=KBPO4MUDASHKMPRNJELQWLNMJNQBB3PZUOVNCEPP2CYVHU6X2PPA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.146408725 (538h33m41.01749177s) [noise=true , address=145.79.227.15:4908 , piece_id=I2FSDNVYBFZKRXZNUHH2APY7N2PI7DPAKFKGXRJW2UFU2PBKSSNQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.146778139 (538h33m41.017861184s) [noise=true , address=145.79.227.43:4161 , piece_id=ZN3T6JSPHJ57X4YYG5MHRJYYOFPH2T2CSGJ34MJQU55HDC7CMOXQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.147137874 (538h33m41.018220919s) [noise=true , address=77.90.31.21:4882 , piece_id=WQAGQTGLU5HRKEET36W34MQP4NPGSVCA4QZ6YEGVAJWBBPLZOUQQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.147426784 (538h33m41.018509819s) [noise=true , address=145.79.227.35:4503 , piece_id=MDYJ2TROT4FHLX4XHKHEQACC7ZXHZCQ5MUPJL5EDS3F2YMXSUGUA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.147466109 (538h33m41.018549154s) [noise=true , address=77.90.31.15:4730 , piece_id=EN7IGRJZ2WFN4HXJPMNLLXDUZPIN7KDUGKPPLJH7CSA4SDOUM2JA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.147817368 (538h33m41.018900413s) [noise=true , address=77.90.31.18:4833 , piece_id=3E57LWLI3WHROFGMXVZ7LHDHO5O7453LWQXA3OEKM52B2NIYM2JA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.147860690 (538h33m41.018943745s) [noise=true , address=145.79.227.44:4212 , piece_id=EFTTQM6323MUQ5WTGY6DFBTF3TGRM6XZWXIGT6DM4VS7GRGM4C2Q, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148163427 (538h33m41.019246472s) [noise=true , address=77.90.31.19:4862 , piece_id=PSBBBSGY7THVLMXJQCADIE7VYZ33OIIDVAEQWG3RWMNA6RSKELNQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148189276 (538h33m41.019272321s) [noise=true , address=77.90.31.10:4524 , piece_id=X2VCBS4PTPPWI6U64T4FVIULRLCDY3XUOJTXJNP34OHSO54ONIWA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148496531 (538h33m41.019579566s) [noise=true , address=145.79.227.23:4556 , piece_id=AKBXRSU5PM5QLHGHE3VXTJ3ZH23BECPGBLQSVZK6XVIYJGVDXT4Q, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148527380 (538h33m41.019610425s) [noise=true , address=145.79.227.33:4899 , piece_id=JFEXXK22EATV7UUH7LLXTFEPIUBW7EWSNYFXLE4AO5F6SLYFTGBA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148801512 (538h33m41.019884547s) [noise=true , address=77.90.31.14:4672 , piece_id=BZG2Y5X47J2WVFF5LBHEJZNUD2OQ3NOTVPTCAVT4TLFQB5FWESLA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148894920 (538h33m41.019977965s) [noise=true , address=5.231.73.5:4016 , piece_id=YCNKQEKJQSJHKKGCYWLOLEBKIVPJHZIJ6TYEG6DF4TRXMZ3ZTTIQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.149118806 (538h33m41.020201851s) [noise=true , address=145.79.227.47:4295 , piece_id=OM7B75MCPM7L3WP5MLQ7MAVBZ2NJUSISOWD56RSWI2AV2OWVDRTA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.149274993 (538h33m41.020358038s) [noise=true , address=145.79.227.50:4400 , piece_id=HZAT7YAVZYWK52FE4TIC6OA37U4VX5B35H55WMBO3MMWNIKTJLJQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.149625842 (538h33m41.020708877s) [noise=true , address=145.79.227.34:4915 , piece_id=RMXYZGGJA5EPJLQD6SQG677HFLOQWRY2HFTGEOICYAQG6QR557RQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:42.618098549 (538h33m42.489181584s) [upload=472313]: Next returned nil segment Dec 28 02:02:42.618120911 (538h33m42.489203946s) [upload=472313]: Waiting for error group managing segments... Dec 28 02:02:43.021704832 (538h33m42.892787877s) [noise=true , address=145.79.227.23:4556 , piece_id=AKBXRSU5PM5QLHGHE3VXTJ3ZH23BECPGBLQSVZK6XVIYJGVDXT4Q, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.027980492 (538h33m42.899063537s) [noise=true , address=145.79.227.50:4400 , piece_id=HZAT7YAVZYWK52FE4TIC6OA37U4VX5B35H55WMBO3MMWNIKTJLJQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.043329239 (538h33m42.914412284s) [noise=true , address=145.79.227.15:4908 , piece_id=I2FSDNVYBFZKRXZNUHH2APY7N2PI7DPAKFKGXRJW2UFU2PBKSSNQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.056308314 (538h33m42.927391349s) [noise=true , address=145.79.227.49:4347 , piece_id=WURIHGQASBLSVLZJVM5EJHWD7U4VFLLPGJIFYNOK6ALA5LYEQDSA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.074628196 (538h33m42.945711252s) [noise=true , address=145.79.227.47:4295 , piece_id=OM7B75MCPM7L3WP5MLQ7MAVBZ2NJUSISOWD56RSWI2AV2OWVDRTA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.080495740 (538h33m42.951578775s) [noise=true , address=5.231.73.5:4016 , piece_id=YCNKQEKJQSJHKKGCYWLOLEBKIVPJHZIJ6TYEG6DF4TRXMZ3ZTTIQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.193955176 (538h33m43.065038241s) [noise=true , address=77.90.31.10:4551 , piece_id=KBPO4MUDASHKMPRNJELQWLNMJNQBB3PZUOVNCEPP2CYVHU6X2PPA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.301513865 (538h33m43.1725969s) [noise=true , address=77.90.31.15:4730 , piece_id=EN7IGRJZ2WFN4HXJPMNLLXDUZPIN7KDUGKPPLJH7CSA4SDOUM2JA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.305020622 (538h33m43.176103657s) [noise=true , address=77.90.31.19:4862 , piece_id=PSBBBSGY7THVLMXJQCADIE7VYZ33OIIDVAEQWG3RWMNA6RSKELNQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.312187619 (538h33m43.183270654s) [noise=true , address=77.90.31.14:4672 , piece_id=BZG2Y5X47J2WVFF5LBHEJZNUD2OQ3NOTVPTCAVT4TLFQB5FWESLA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.319277390 (538h33m43.190360425s) [noise=true , address=77.90.31.21:4882 , piece_id=WQAGQTGLU5HRKEET36W34MQP4NPGSVCA4QZ6YEGVAJWBBPLZOUQQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.948916959 (538h33m43.820000004s) [noise=true , address=145.79.227.34:4915 , piece_id=RMXYZGGJA5EPJLQD6SQG677HFLOQWRY2HFTGEOICYAQG6QR557RQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.989823331 (538h33m43.860906366s) [noise=true , address=145.79.227.44:4212 , piece_id=EFTTQM6323MUQ5WTGY6DFBTF3TGRM6XZWXIGT6DM4VS7GRGM4C2Q, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:44.123218949 (538h33m43.994301984s) [noise=true , address=145.79.227.43:4161 , piece_id=ZN3T6JSPHJ57X4YYG5MHRJYYOFPH2T2CSGJ34MJQU55HDC7CMOXQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:44.123238386 (538h33m43.994321421s) [seg_pos={0 0}, upload=472313]: Segment reached optimal threshold of 14 pieces. Dec 28 02:02:44.123431754 (538h33m43.994514789s) [noise=true , address=145.79.227.33:4899 , piece_id=JFEXXK22EATV7UUH7LLXTFEPIUBW7EWSNYFXLE4AO5F6SLYFTGBA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b4pZVQRnjcQNn45VTgdsz5fpjYPMAzSXGR5KJ5BP9XHMkRnRw): protocol: expected piece hash; context canceled; EOF Dec 28 02:02:44.123437586 (538h33m43.994520621s) [noise=true , address=145.79.227.33:4899 , piece_id=JFEXXK22EATV7UUH7LLXTFEPIUBW7EWSNYFXLE4AO5F6SLYFTGBA, seg_pos={0 0}, upload=472313]: Long tail cancellation due to optimal threshold Dec 28 02:02:44.123631064 (538h33m43.994714109s) [noise=true , address=77.90.31.18:4833 , piece_id=3E57LWLI3WHROFGMXVZ7LHDHO5O7453LWQXA3OEKM52B2NIYM2JA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FbVbo8kGy1aaFyCk2noiXNUJHtosFJHFykBMRecGd5Kj5gjEp): protocol: expected piece hash; context canceled; EOF Dec 28 02:02:44.123675267 (538h33m43.994758313s) [noise=true , address=77.90.31.18:4833 , piece_id=3E57LWLI3WHROFGMXVZ7LHDHO5O7453LWQXA3OEKM52B2NIYM2JA, seg_pos={0 0}, upload=472313]: Long tail cancellation due to optimal threshold Dec 28 02:02:44.123774066 (538h33m43.994857101s) [noise=true , address=77.90.31.10:4524 , piece_id=X2VCBS4PTPPWI6U64T4FVIULRLCDY3XUOJTXJNP34OHSO54ONIWA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe): protocol: expected piece hash; context canceled; EOF Dec 28 02:02:44.123781761 (538h33m43.994864796s) [noise=true , address=77.90.31.10:4524 , piece_id=X2VCBS4PTPPWI6U64T4FVIULRLCDY3XUOJTXJNP34OHSO54ONIWA, seg_pos={0 0}, upload=472313]: Long tail cancellation due to optimal threshold Dec 28 02:02:44.123923341 (538h33m43.995006376s) [noise=true , address=145.79.227.35:4503 , piece_id=MDYJ2TROT4FHLX4XHKHEQACC7ZXHZCQ5MUPJL5EDS3F2YMXSUGUA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ctmSF1aMQasuXByFW8DppDantREmmWL2WUjmKGW5ZULVePyvm): protocol: expected piece hash; context canceled; EOF Dec 28 02:02:44.123928591 (538h33m43.995011636s) [noise=true , address=145.79.227.35:4503 , piece_id=MDYJ2TROT4FHLX4XHKHEQACC7ZXHZCQ5MUPJL5EDS3F2YMXSUGUA, seg_pos={0 0}, upload=472313]: Long tail cancellation due to optimal threshold Dec 28 02:02:44.123944731 (538h33m43.995027766s) [seg_pos={0 0}, upload=472313]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 02:02:44.124017179 (538h33m43.995100214s) [upload=472313]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:02:44.124022469 (538h33m43.995105504s) [upload=472313]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:02:44.409047274 (538h33m44.280130349s) [upload=472313]: Done uploading segments... Dec 28 02:02:44.409073354 (538h33m44.280156389s) [upload=472313]: Upload finished. err: <nil> Duration: 51.466814892s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 33 out of 60 is failed (threshold=3) [XX.XXXXX..X.XXX.X.XX.XX.X..X.X.X.X.X..X...XX..X.XXX.XXX.X...] duration: 8.004482162s trace: https://jaeger.dev.storj.io/trace/1c22d34f4ac03822 FAIL (2025-12-28 02:18:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: GU4X73Q6AFIBRIYAG4ASNF22MDMCKUOKIJCS3TTJT3FKDWLAMIKA) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: DQ4IHJQYEPBEW63HH3ELSFGDFZUHJ7HL6UHBLGQRHKOHTD7HP2SQ) context deadline exceeded ======LOGS====== Dec 28 02:15:53.759588852 (538h46m53.630671907s) [upload=472506]: Starting upload Dec 28 02:15:53.759803180 (538h46m53.630886235s) [upload=472506]: Done starting upload Dec 28 02:15:53.759860660 (538h46m53.630943705s) [upload=472506]: Uploading segments... Dec 28 02:15:53.760077182 (538h46m53.631160227s) [upload=472506]: Got next segment. Inline: false Dec 28 02:15:53.760126496 (538h46m53.631209541s) [upload=472506]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 02:15:53.760133219 (538h46m53.631216264s) [upload=472506]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 02:15:54.003115733 (538h46m53.874198788s) [seg_pos={0 0}, upload=472506]: Begin upload segment... Dec 28 02:15:54.003896579 (538h46m53.874979644s) [seg_pos={0 0}, upload=472506]: Done begin upload segment. Dec 28 02:15:54.003960000 (538h46m53.875043055s) [noise=true , address=145.79.227.16:4909 , piece_id=KSGRHS6G6JZ7IY23T277NMHN6HGG7DUODYBIXENFS2PSTYNUX37Q, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.003967354 (538h46m53.875050399s) [noise=true , address=145.79.227.52:4445 , piece_id=ZEY2QW4N6ZNGBILKIDQ7YBFSB7U2JVICZJV2W3GQU7D3QACB32FQ, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.004662528 (538h46m53.875745573s) [noise=true , address=145.79.227.31:4797 , piece_id=J5OPI7TBATORMGHB245L3YCWH2I56AVGHVAWJRLALMYDQIW7YAGQ, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.004851216 (538h46m53.875934261s) [noise=true , address=45.91.116.11:4118 , piece_id=LA7UXPHDDM25MJ4IKX4IGJXKZFA37TTXYBAGMA25C3Y57X6YFQ3Q, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.005240899 (538h46m53.876323944s) [noise=true , address=145.79.227.4:4185 , piece_id=SKBIGBFEIKWVJJFKWFM6TO5CYQ6RCM6RLV6AXCM4TKG7TS6TAVXQ, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.005594032 (538h46m53.876677067s) [noise=true , address=77.90.31.16:4735 , piece_id=VD6RDROPI6XQA2TYQUB4ENXFD2B6OCM2YH5AXP3OVGPQ5YBI37BA, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.005750930 (538h46m53.876833975s) [noise=true , address=145.79.227.18:4006 , piece_id=GYNQ6GGPHVJJYRV2RHRL63SFS4UPSONZY6PPPVC4ZTSOK5SVWVLQ, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.006059718 (538h46m53.877142793s) [noise=true , address=45.91.116.14:4190 , piece_id=ZDWBLJMWUZN4ZMK5X6FWN24VEWAFU2KODQJ7URBZBWJHIML4GKKA, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.006277232 (538h46m53.877360277s) [noise=true , address=145.79.227.17:4972 , piece_id=64UKV76NMUYNSLPZA3WLTQ6F7W2XP5Y564DFNQMLBFB3SG6S3IFA, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.006515977 (538h46m53.877599022s) [noise=true , address=45.91.116.6:4973 , piece_id=GU4X73Q6AFIBRIYAG4ASNF22MDMCKUOKIJCS3TTJT3FKDWLAMIKA, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.007002723 (538h46m53.878085789s) [noise=true , address=145.79.227.19:4077 , piece_id=4LCN23MUWAX2HK3J2OFAFLEGUQM4YYTJWZ7IPCN3D6QROORY5O6A, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.007095439 (538h46m53.878178484s) [noise=true , address=77.90.31.14:4661 , piece_id=U2S5KQX5UAKMMMIOZH6CP7SE7PR2DZIKHKX7OXBCJMSZKG4RUMMQ, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.007443692 (538h46m53.878526728s) [noise=true , address=45.91.116.7:4009 , piece_id=DQ4IHJQYEPBEW63HH3ELSFGDFZUHJ7HL6UHBLGQRHKOHTD7HP2SQ, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.007675033 (538h46m53.878758078s) [noise=true , address=145.79.227.32:4836 , piece_id=AZ5NCPW2VX2VSH7XGAUEWC2B25ZEQ2NVQVZZUL6CANSCBWIIEY3A, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.007858693 (538h46m53.878941738s) [noise=true , address=145.79.227.27:4691 , piece_id=MJMBV5LR4RA5ZP7Q7U34DXO2TWJ6LDKH4YMADM3K5CU2KRHMQWBA, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.008204601 (538h46m53.879287637s) [noise=true , address=145.79.227.12:4767 , piece_id=A3L6CAJZWVR4RYOZMKQ5C5UALP6FKW367UGSOCW3YEVLWMYJBBTQ, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:55.745297151 (538h46m55.616380186s) [upload=472506]: Next returned nil segment Dec 28 02:15:55.745351053 (538h46m55.616434108s) [upload=472506]: Waiting for error group managing segments... Dec 28 02:15:56.032845416 (538h46m55.903928451s) [noise=true , address=145.79.227.18:4006 , piece_id=GYNQ6GGPHVJJYRV2RHRL63SFS4UPSONZY6PPPVC4ZTSOK5SVWVLQ, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.202191721 (538h46m56.073274756s) [noise=true , address=145.79.227.19:4077 , piece_id=4LCN23MUWAX2HK3J2OFAFLEGUQM4YYTJWZ7IPCN3D6QROORY5O6A, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.205182144 (538h46m56.076265189s) [noise=true , address=145.79.227.31:4797 , piece_id=J5OPI7TBATORMGHB245L3YCWH2I56AVGHVAWJRLALMYDQIW7YAGQ, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.205433684 (538h46m56.076516719s) [noise=true , address=145.79.227.27:4691 , piece_id=MJMBV5LR4RA5ZP7Q7U34DXO2TWJ6LDKH4YMADM3K5CU2KRHMQWBA, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.214467002 (538h46m56.085550047s) [noise=true , address=145.79.227.16:4909 , piece_id=KSGRHS6G6JZ7IY23T277NMHN6HGG7DUODYBIXENFS2PSTYNUX37Q, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.214875340 (538h46m56.085958375s) [noise=true , address=145.79.227.4:4185 , piece_id=SKBIGBFEIKWVJJFKWFM6TO5CYQ6RCM6RLV6AXCM4TKG7TS6TAVXQ, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.217932621 (538h46m56.089015656s) [noise=true , address=145.79.227.12:4767 , piece_id=A3L6CAJZWVR4RYOZMKQ5C5UALP6FKW367UGSOCW3YEVLWMYJBBTQ, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.306475972 (538h46m56.177559007s) [noise=true , address=45.91.116.6:4973 , piece_id=GU4X73Q6AFIBRIYAG4ASNF22MDMCKUOKIJCS3TTJT3FKDWLAMIKA, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.385055362 (538h46m56.256138408s) [noise=true , address=45.91.116.7:4009 , piece_id=DQ4IHJQYEPBEW63HH3ELSFGDFZUHJ7HL6UHBLGQRHKOHTD7HP2SQ, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.395589739 (538h46m56.266672784s) [noise=true , address=45.91.116.14:4190 , piece_id=ZDWBLJMWUZN4ZMK5X6FWN24VEWAFU2KODQJ7URBZBWJHIML4GKKA, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.395681062 (538h46m56.266764107s) [noise=true , address=145.79.227.52:4445 , piece_id=ZEY2QW4N6ZNGBILKIDQ7YBFSB7U2JVICZJV2W3GQU7D3QACB32FQ, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.403044182 (538h46m56.274127217s) [noise=true , address=145.79.227.32:4836 , piece_id=AZ5NCPW2VX2VSH7XGAUEWC2B25ZEQ2NVQVZZUL6CANSCBWIIEY3A, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:57.110570880 (538h46m56.981653925s) [noise=true , address=45.91.116.11:4118 , piece_id=LA7UXPHDDM25MJ4IKX4IGJXKZFA37TTXYBAGMA25C3Y57X6YFQ3Q, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:57.110788534 (538h46m56.981871569s) [seg_pos={0 0}, upload=472506]: Segment reached optimal threshold of 13 pieces. Dec 28 02:15:57.111013823 (538h46m56.982096858s) [noise=true , address=77.90.31.14:4661 , piece_id=U2S5KQX5UAKMMMIOZH6CP7SE7PR2DZIKHKX7OXBCJMSZKG4RUMMQ, seg_pos={0 0}, upload=472506]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 28 02:15:57.111022289 (538h46m56.982105335s) [noise=true , address=77.90.31.14:4661 , piece_id=U2S5KQX5UAKMMMIOZH6CP7SE7PR2DZIKHKX7OXBCJMSZKG4RUMMQ, seg_pos={0 0}, upload=472506]: Long tail cancellation due to optimal threshold Dec 28 02:15:57.111365041 (538h46m56.982448076s) [noise=true , address=77.90.31.16:4735 , piece_id=VD6RDROPI6XQA2TYQUB4ENXFD2B6OCM2YH5AXP3OVGPQ5YBI37BA, seg_pos={0 0}, upload=472506]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: read tcp 188.166.225.170:47380->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 02:15:57.111372196 (538h46m56.982455241s) [noise=true , address=77.90.31.16:4735 , piece_id=VD6RDROPI6XQA2TYQUB4ENXFD2B6OCM2YH5AXP3OVGPQ5YBI37BA, seg_pos={0 0}, upload=472506]: Long tail cancellation due to optimal threshold Dec 28 02:15:57.111468879 (538h46m56.982551924s) [noise=true , address=145.79.227.17:4972 , piece_id=64UKV76NMUYNSLPZA3WLTQ6F7W2XP5Y564DFNQMLBFB3SG6S3IFA, seg_pos={0 0}, upload=472506]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: expected piece hash; context canceled; EOF Dec 28 02:15:57.111474270 (538h46m56.982557315s) [noise=true , address=145.79.227.17:4972 , piece_id=64UKV76NMUYNSLPZA3WLTQ6F7W2XP5Y564DFNQMLBFB3SG6S3IFA, seg_pos={0 0}, upload=472506]: Long tail cancellation due to optimal threshold Dec 28 02:15:57.111493035 (538h46m56.98257607s) [seg_pos={0 0}, upload=472506]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 02:15:57.111624576 (538h46m56.982707611s) [upload=472506]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:15:57.111630667 (538h46m56.982713713s) [upload=472506]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:15:57.409023835 (538h46m57.28010687s) [upload=472506]: Done uploading segments... Dec 28 02:15:57.409043784 (538h46m57.280126819s) [upload=472506]: Upload finished. err: <nil> Duration: 8.004507841s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................................X................] duration: 3.773630435s trace: https://jaeger.dev.storj.io/trace/7fc142ad3d711e34 FAIL (2025-12-28 02:18:02 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.773667907s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 33 out of 60 is failed (threshold=3) [.XX.XXXXX..X.XXX.X.XX.XX.X..X.X.X.X.X..X...XX..X.XXX.XXX.X..] duration: 6.152997356s trace: https://jaeger.dev.storj.io/trace/1c22d34f4ac03822 FAIL (2025-12-28 02:17:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: GU4X73Q6AFIBRIYAG4ASNF22MDMCKUOKIJCS3TTJT3FKDWLAMIKA) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: DQ4IHJQYEPBEW63HH3ELSFGDFZUHJ7HL6UHBLGQRHKOHTD7HP2SQ) context deadline exceeded ======LOGS====== Dec 28 02:15:53.759588852 (538h46m53.630671907s) [upload=472506]: Starting upload Dec 28 02:15:53.759803180 (538h46m53.630886235s) [upload=472506]: Done starting upload Dec 28 02:15:53.759860660 (538h46m53.630943705s) [upload=472506]: Uploading segments... Dec 28 02:15:53.760077182 (538h46m53.631160227s) [upload=472506]: Got next segment. Inline: false Dec 28 02:15:53.760126496 (538h46m53.631209541s) [upload=472506]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 02:15:53.760133219 (538h46m53.631216264s) [upload=472506]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 02:15:54.003115733 (538h46m53.874198788s) [seg_pos={0 0}, upload=472506]: Begin upload segment... Dec 28 02:15:54.003896579 (538h46m53.874979644s) [seg_pos={0 0}, upload=472506]: Done begin upload segment. Dec 28 02:15:54.003960000 (538h46m53.875043055s) [noise=true , address=145.79.227.16:4909 , piece_id=KSGRHS6G6JZ7IY23T277NMHN6HGG7DUODYBIXENFS2PSTYNUX37Q, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.003967354 (538h46m53.875050399s) [noise=true , address=145.79.227.52:4445 , piece_id=ZEY2QW4N6ZNGBILKIDQ7YBFSB7U2JVICZJV2W3GQU7D3QACB32FQ, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.004662528 (538h46m53.875745573s) [noise=true , address=145.79.227.31:4797 , piece_id=J5OPI7TBATORMGHB245L3YCWH2I56AVGHVAWJRLALMYDQIW7YAGQ, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.004851216 (538h46m53.875934261s) [noise=true , address=45.91.116.11:4118 , piece_id=LA7UXPHDDM25MJ4IKX4IGJXKZFA37TTXYBAGMA25C3Y57X6YFQ3Q, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.005240899 (538h46m53.876323944s) [noise=true , address=145.79.227.4:4185 , piece_id=SKBIGBFEIKWVJJFKWFM6TO5CYQ6RCM6RLV6AXCM4TKG7TS6TAVXQ, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.005594032 (538h46m53.876677067s) [noise=true , address=77.90.31.16:4735 , piece_id=VD6RDROPI6XQA2TYQUB4ENXFD2B6OCM2YH5AXP3OVGPQ5YBI37BA, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.005750930 (538h46m53.876833975s) [noise=true , address=145.79.227.18:4006 , piece_id=GYNQ6GGPHVJJYRV2RHRL63SFS4UPSONZY6PPPVC4ZTSOK5SVWVLQ, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.006059718 (538h46m53.877142793s) [noise=true , address=45.91.116.14:4190 , piece_id=ZDWBLJMWUZN4ZMK5X6FWN24VEWAFU2KODQJ7URBZBWJHIML4GKKA, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.006277232 (538h46m53.877360277s) [noise=true , address=145.79.227.17:4972 , piece_id=64UKV76NMUYNSLPZA3WLTQ6F7W2XP5Y564DFNQMLBFB3SG6S3IFA, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.006515977 (538h46m53.877599022s) [noise=true , address=45.91.116.6:4973 , piece_id=GU4X73Q6AFIBRIYAG4ASNF22MDMCKUOKIJCS3TTJT3FKDWLAMIKA, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.007002723 (538h46m53.878085789s) [noise=true , address=145.79.227.19:4077 , piece_id=4LCN23MUWAX2HK3J2OFAFLEGUQM4YYTJWZ7IPCN3D6QROORY5O6A, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.007095439 (538h46m53.878178484s) [noise=true , address=77.90.31.14:4661 , piece_id=U2S5KQX5UAKMMMIOZH6CP7SE7PR2DZIKHKX7OXBCJMSZKG4RUMMQ, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.007443692 (538h46m53.878526728s) [noise=true , address=45.91.116.7:4009 , piece_id=DQ4IHJQYEPBEW63HH3ELSFGDFZUHJ7HL6UHBLGQRHKOHTD7HP2SQ, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.007675033 (538h46m53.878758078s) [noise=true , address=145.79.227.32:4836 , piece_id=AZ5NCPW2VX2VSH7XGAUEWC2B25ZEQ2NVQVZZUL6CANSCBWIIEY3A, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.007858693 (538h46m53.878941738s) [noise=true , address=145.79.227.27:4691 , piece_id=MJMBV5LR4RA5ZP7Q7U34DXO2TWJ6LDKH4YMADM3K5CU2KRHMQWBA, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.008204601 (538h46m53.879287637s) [noise=true , address=145.79.227.12:4767 , piece_id=A3L6CAJZWVR4RYOZMKQ5C5UALP6FKW367UGSOCW3YEVLWMYJBBTQ, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:55.745297151 (538h46m55.616380186s) [upload=472506]: Next returned nil segment Dec 28 02:15:55.745351053 (538h46m55.616434108s) [upload=472506]: Waiting for error group managing segments... Dec 28 02:15:56.032845416 (538h46m55.903928451s) [noise=true , address=145.79.227.18:4006 , piece_id=GYNQ6GGPHVJJYRV2RHRL63SFS4UPSONZY6PPPVC4ZTSOK5SVWVLQ, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.202191721 (538h46m56.073274756s) [noise=true , address=145.79.227.19:4077 , piece_id=4LCN23MUWAX2HK3J2OFAFLEGUQM4YYTJWZ7IPCN3D6QROORY5O6A, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.205182144 (538h46m56.076265189s) [noise=true , address=145.79.227.31:4797 , piece_id=J5OPI7TBATORMGHB245L3YCWH2I56AVGHVAWJRLALMYDQIW7YAGQ, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.205433684 (538h46m56.076516719s) [noise=true , address=145.79.227.27:4691 , piece_id=MJMBV5LR4RA5ZP7Q7U34DXO2TWJ6LDKH4YMADM3K5CU2KRHMQWBA, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.214467002 (538h46m56.085550047s) [noise=true , address=145.79.227.16:4909 , piece_id=KSGRHS6G6JZ7IY23T277NMHN6HGG7DUODYBIXENFS2PSTYNUX37Q, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.214875340 (538h46m56.085958375s) [noise=true , address=145.79.227.4:4185 , piece_id=SKBIGBFEIKWVJJFKWFM6TO5CYQ6RCM6RLV6AXCM4TKG7TS6TAVXQ, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.217932621 (538h46m56.089015656s) [noise=true , address=145.79.227.12:4767 , piece_id=A3L6CAJZWVR4RYOZMKQ5C5UALP6FKW367UGSOCW3YEVLWMYJBBTQ, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.306475972 (538h46m56.177559007s) [noise=true , address=45.91.116.6:4973 , piece_id=GU4X73Q6AFIBRIYAG4ASNF22MDMCKUOKIJCS3TTJT3FKDWLAMIKA, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.385055362 (538h46m56.256138408s) [noise=true , address=45.91.116.7:4009 , piece_id=DQ4IHJQYEPBEW63HH3ELSFGDFZUHJ7HL6UHBLGQRHKOHTD7HP2SQ, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.395589739 (538h46m56.266672784s) [noise=true , address=45.91.116.14:4190 , piece_id=ZDWBLJMWUZN4ZMK5X6FWN24VEWAFU2KODQJ7URBZBWJHIML4GKKA, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.395681062 (538h46m56.266764107s) [noise=true , address=145.79.227.52:4445 , piece_id=ZEY2QW4N6ZNGBILKIDQ7YBFSB7U2JVICZJV2W3GQU7D3QACB32FQ, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.403044182 (538h46m56.274127217s) [noise=true , address=145.79.227.32:4836 , piece_id=AZ5NCPW2VX2VSH7XGAUEWC2B25ZEQ2NVQVZZUL6CANSCBWIIEY3A, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:57.110570880 (538h46m56.981653925s) [noise=true , address=45.91.116.11:4118 , piece_id=LA7UXPHDDM25MJ4IKX4IGJXKZFA37TTXYBAGMA25C3Y57X6YFQ3Q, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:57.110788534 (538h46m56.981871569s) [seg_pos={0 0}, upload=472506]: Segment reached optimal threshold of 13 pieces. Dec 28 02:15:57.111013823 (538h46m56.982096858s) [noise=true , address=77.90.31.14:4661 , piece_id=U2S5KQX5UAKMMMIOZH6CP7SE7PR2DZIKHKX7OXBCJMSZKG4RUMMQ, seg_pos={0 0}, upload=472506]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 28 02:15:57.111022289 (538h46m56.982105335s) [noise=true , address=77.90.31.14:4661 , piece_id=U2S5KQX5UAKMMMIOZH6CP7SE7PR2DZIKHKX7OXBCJMSZKG4RUMMQ, seg_pos={0 0}, upload=472506]: Long tail cancellation due to optimal threshold Dec 28 02:15:57.111365041 (538h46m56.982448076s) [noise=true , address=77.90.31.16:4735 , piece_id=VD6RDROPI6XQA2TYQUB4ENXFD2B6OCM2YH5AXP3OVGPQ5YBI37BA, seg_pos={0 0}, upload=472506]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: read tcp 188.166.225.170:47380->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 02:15:57.111372196 (538h46m56.982455241s) [noise=true , address=77.90.31.16:4735 , piece_id=VD6RDROPI6XQA2TYQUB4ENXFD2B6OCM2YH5AXP3OVGPQ5YBI37BA, seg_pos={0 0}, upload=472506]: Long tail cancellation due to optimal threshold Dec 28 02:15:57.111468879 (538h46m56.982551924s) [noise=true , address=145.79.227.17:4972 , piece_id=64UKV76NMUYNSLPZA3WLTQ6F7W2XP5Y564DFNQMLBFB3SG6S3IFA, seg_pos={0 0}, upload=472506]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: expected piece hash; context canceled; EOF Dec 28 02:15:57.111474270 (538h46m56.982557315s) [noise=true , address=145.79.227.17:4972 , piece_id=64UKV76NMUYNSLPZA3WLTQ6F7W2XP5Y564DFNQMLBFB3SG6S3IFA, seg_pos={0 0}, upload=472506]: Long tail cancellation due to optimal threshold Dec 28 02:15:57.111493035 (538h46m56.98257607s) [seg_pos={0 0}, upload=472506]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 02:15:57.111624576 (538h46m56.982707611s) [upload=472506]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:15:57.111630667 (538h46m56.982713713s) [upload=472506]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:15:57.409023835 (538h46m57.28010687s) [upload=472506]: Done uploading segments... Dec 28 02:15:57.409043784 (538h46m57.280126819s) [upload=472506]: Upload finished. err: <nil> Duration: 6.153021833s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............................................X...............] duration: 4.299293537s trace: https://jaeger.dev.storj.io/trace/7fc142ad3d711e34 FAIL (2025-12-28 02:17:12 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.299320498s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 34 out of 60 is failed (threshold=3) [X.XX.XXXXX..X.XXX.X.XX.XX.X..X.X.X.X.X..X...XX..X.XXX.XXX.X.] duration: 9.662157017s trace: https://jaeger.dev.storj.io/trace/1c22d34f4ac03822 FAIL (2025-12-28 02:17:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: GU4X73Q6AFIBRIYAG4ASNF22MDMCKUOKIJCS3TTJT3FKDWLAMIKA) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: DQ4IHJQYEPBEW63HH3ELSFGDFZUHJ7HL6UHBLGQRHKOHTD7HP2SQ) context deadline exceeded ======LOGS====== Dec 28 02:15:53.759588852 (538h46m53.630671907s) [upload=472506]: Starting upload Dec 28 02:15:53.759803180 (538h46m53.630886235s) [upload=472506]: Done starting upload Dec 28 02:15:53.759860660 (538h46m53.630943705s) [upload=472506]: Uploading segments... Dec 28 02:15:53.760077182 (538h46m53.631160227s) [upload=472506]: Got next segment. Inline: false Dec 28 02:15:53.760126496 (538h46m53.631209541s) [upload=472506]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 02:15:53.760133219 (538h46m53.631216264s) [upload=472506]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 02:15:54.003115733 (538h46m53.874198788s) [seg_pos={0 0}, upload=472506]: Begin upload segment... Dec 28 02:15:54.003896579 (538h46m53.874979644s) [seg_pos={0 0}, upload=472506]: Done begin upload segment. Dec 28 02:15:54.003960000 (538h46m53.875043055s) [noise=true , address=145.79.227.16:4909 , piece_id=KSGRHS6G6JZ7IY23T277NMHN6HGG7DUODYBIXENFS2PSTYNUX37Q, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.003967354 (538h46m53.875050399s) [noise=true , address=145.79.227.52:4445 , piece_id=ZEY2QW4N6ZNGBILKIDQ7YBFSB7U2JVICZJV2W3GQU7D3QACB32FQ, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.004662528 (538h46m53.875745573s) [noise=true , address=145.79.227.31:4797 , piece_id=J5OPI7TBATORMGHB245L3YCWH2I56AVGHVAWJRLALMYDQIW7YAGQ, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.004851216 (538h46m53.875934261s) [noise=true , address=45.91.116.11:4118 , piece_id=LA7UXPHDDM25MJ4IKX4IGJXKZFA37TTXYBAGMA25C3Y57X6YFQ3Q, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.005240899 (538h46m53.876323944s) [noise=true , address=145.79.227.4:4185 , piece_id=SKBIGBFEIKWVJJFKWFM6TO5CYQ6RCM6RLV6AXCM4TKG7TS6TAVXQ, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.005594032 (538h46m53.876677067s) [noise=true , address=77.90.31.16:4735 , piece_id=VD6RDROPI6XQA2TYQUB4ENXFD2B6OCM2YH5AXP3OVGPQ5YBI37BA, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.005750930 (538h46m53.876833975s) [noise=true , address=145.79.227.18:4006 , piece_id=GYNQ6GGPHVJJYRV2RHRL63SFS4UPSONZY6PPPVC4ZTSOK5SVWVLQ, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.006059718 (538h46m53.877142793s) [noise=true , address=45.91.116.14:4190 , piece_id=ZDWBLJMWUZN4ZMK5X6FWN24VEWAFU2KODQJ7URBZBWJHIML4GKKA, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.006277232 (538h46m53.877360277s) [noise=true , address=145.79.227.17:4972 , piece_id=64UKV76NMUYNSLPZA3WLTQ6F7W2XP5Y564DFNQMLBFB3SG6S3IFA, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.006515977 (538h46m53.877599022s) [noise=true , address=45.91.116.6:4973 , piece_id=GU4X73Q6AFIBRIYAG4ASNF22MDMCKUOKIJCS3TTJT3FKDWLAMIKA, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.007002723 (538h46m53.878085789s) [noise=true , address=145.79.227.19:4077 , piece_id=4LCN23MUWAX2HK3J2OFAFLEGUQM4YYTJWZ7IPCN3D6QROORY5O6A, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.007095439 (538h46m53.878178484s) [noise=true , address=77.90.31.14:4661 , piece_id=U2S5KQX5UAKMMMIOZH6CP7SE7PR2DZIKHKX7OXBCJMSZKG4RUMMQ, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.007443692 (538h46m53.878526728s) [noise=true , address=45.91.116.7:4009 , piece_id=DQ4IHJQYEPBEW63HH3ELSFGDFZUHJ7HL6UHBLGQRHKOHTD7HP2SQ, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.007675033 (538h46m53.878758078s) [noise=true , address=145.79.227.32:4836 , piece_id=AZ5NCPW2VX2VSH7XGAUEWC2B25ZEQ2NVQVZZUL6CANSCBWIIEY3A, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.007858693 (538h46m53.878941738s) [noise=true , address=145.79.227.27:4691 , piece_id=MJMBV5LR4RA5ZP7Q7U34DXO2TWJ6LDKH4YMADM3K5CU2KRHMQWBA, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.008204601 (538h46m53.879287637s) [noise=true , address=145.79.227.12:4767 , piece_id=A3L6CAJZWVR4RYOZMKQ5C5UALP6FKW367UGSOCW3YEVLWMYJBBTQ, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:55.745297151 (538h46m55.616380186s) [upload=472506]: Next returned nil segment Dec 28 02:15:55.745351053 (538h46m55.616434108s) [upload=472506]: Waiting for error group managing segments... Dec 28 02:15:56.032845416 (538h46m55.903928451s) [noise=true , address=145.79.227.18:4006 , piece_id=GYNQ6GGPHVJJYRV2RHRL63SFS4UPSONZY6PPPVC4ZTSOK5SVWVLQ, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.202191721 (538h46m56.073274756s) [noise=true , address=145.79.227.19:4077 , piece_id=4LCN23MUWAX2HK3J2OFAFLEGUQM4YYTJWZ7IPCN3D6QROORY5O6A, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.205182144 (538h46m56.076265189s) [noise=true , address=145.79.227.31:4797 , piece_id=J5OPI7TBATORMGHB245L3YCWH2I56AVGHVAWJRLALMYDQIW7YAGQ, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.205433684 (538h46m56.076516719s) [noise=true , address=145.79.227.27:4691 , piece_id=MJMBV5LR4RA5ZP7Q7U34DXO2TWJ6LDKH4YMADM3K5CU2KRHMQWBA, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.214467002 (538h46m56.085550047s) [noise=true , address=145.79.227.16:4909 , piece_id=KSGRHS6G6JZ7IY23T277NMHN6HGG7DUODYBIXENFS2PSTYNUX37Q, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.214875340 (538h46m56.085958375s) [noise=true , address=145.79.227.4:4185 , piece_id=SKBIGBFEIKWVJJFKWFM6TO5CYQ6RCM6RLV6AXCM4TKG7TS6TAVXQ, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.217932621 (538h46m56.089015656s) [noise=true , address=145.79.227.12:4767 , piece_id=A3L6CAJZWVR4RYOZMKQ5C5UALP6FKW367UGSOCW3YEVLWMYJBBTQ, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.306475972 (538h46m56.177559007s) [noise=true , address=45.91.116.6:4973 , piece_id=GU4X73Q6AFIBRIYAG4ASNF22MDMCKUOKIJCS3TTJT3FKDWLAMIKA, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.385055362 (538h46m56.256138408s) [noise=true , address=45.91.116.7:4009 , piece_id=DQ4IHJQYEPBEW63HH3ELSFGDFZUHJ7HL6UHBLGQRHKOHTD7HP2SQ, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.395589739 (538h46m56.266672784s) [noise=true , address=45.91.116.14:4190 , piece_id=ZDWBLJMWUZN4ZMK5X6FWN24VEWAFU2KODQJ7URBZBWJHIML4GKKA, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.395681062 (538h46m56.266764107s) [noise=true , address=145.79.227.52:4445 , piece_id=ZEY2QW4N6ZNGBILKIDQ7YBFSB7U2JVICZJV2W3GQU7D3QACB32FQ, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.403044182 (538h46m56.274127217s) [noise=true , address=145.79.227.32:4836 , piece_id=AZ5NCPW2VX2VSH7XGAUEWC2B25ZEQ2NVQVZZUL6CANSCBWIIEY3A, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:57.110570880 (538h46m56.981653925s) [noise=true , address=45.91.116.11:4118 , piece_id=LA7UXPHDDM25MJ4IKX4IGJXKZFA37TTXYBAGMA25C3Y57X6YFQ3Q, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:57.110788534 (538h46m56.981871569s) [seg_pos={0 0}, upload=472506]: Segment reached optimal threshold of 13 pieces. Dec 28 02:15:57.111013823 (538h46m56.982096858s) [noise=true , address=77.90.31.14:4661 , piece_id=U2S5KQX5UAKMMMIOZH6CP7SE7PR2DZIKHKX7OXBCJMSZKG4RUMMQ, seg_pos={0 0}, upload=472506]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 28 02:15:57.111022289 (538h46m56.982105335s) [noise=true , address=77.90.31.14:4661 , piece_id=U2S5KQX5UAKMMMIOZH6CP7SE7PR2DZIKHKX7OXBCJMSZKG4RUMMQ, seg_pos={0 0}, upload=472506]: Long tail cancellation due to optimal threshold Dec 28 02:15:57.111365041 (538h46m56.982448076s) [noise=true , address=77.90.31.16:4735 , piece_id=VD6RDROPI6XQA2TYQUB4ENXFD2B6OCM2YH5AXP3OVGPQ5YBI37BA, seg_pos={0 0}, upload=472506]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: read tcp 188.166.225.170:47380->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 02:15:57.111372196 (538h46m56.982455241s) [noise=true , address=77.90.31.16:4735 , piece_id=VD6RDROPI6XQA2TYQUB4ENXFD2B6OCM2YH5AXP3OVGPQ5YBI37BA, seg_pos={0 0}, upload=472506]: Long tail cancellation due to optimal threshold Dec 28 02:15:57.111468879 (538h46m56.982551924s) [noise=true , address=145.79.227.17:4972 , piece_id=64UKV76NMUYNSLPZA3WLTQ6F7W2XP5Y564DFNQMLBFB3SG6S3IFA, seg_pos={0 0}, upload=472506]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: expected piece hash; context canceled; EOF Dec 28 02:15:57.111474270 (538h46m56.982557315s) [noise=true , address=145.79.227.17:4972 , piece_id=64UKV76NMUYNSLPZA3WLTQ6F7W2XP5Y564DFNQMLBFB3SG6S3IFA, seg_pos={0 0}, upload=472506]: Long tail cancellation due to optimal threshold Dec 28 02:15:57.111493035 (538h46m56.98257607s) [seg_pos={0 0}, upload=472506]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 02:15:57.111624576 (538h46m56.982707611s) [upload=472506]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:15:57.111630667 (538h46m56.982713713s) [upload=472506]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:15:57.409023835 (538h46m57.28010687s) [upload=472506]: Done uploading segments... Dec 28 02:15:57.409043784 (538h46m57.280126819s) [upload=472506]: Upload finished. err: <nil> Duration: 9.66218474s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............................................X..............] duration: 4.75798677s trace: https://jaeger.dev.storj.io/trace/7fc142ad3d711e34 FAIL (2025-12-28 02:16:55 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.758014704s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 32.554421083s trace: https://jaeger.dev.storj.io/trace/12a1d059b6f75af7 FAIL (2025-12-28 02:16:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1.145865281 (538h33m41.016948316s) [noise=true , address=77.90.31.10:4551 , piece_id=KBPO4MUDASHKMPRNJELQWLNMJNQBB3PZUOVNCEPP2CYVHU6X2PPA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.146408725 (538h33m41.01749177s) [noise=true , address=145.79.227.15:4908 , piece_id=I2FSDNVYBFZKRXZNUHH2APY7N2PI7DPAKFKGXRJW2UFU2PBKSSNQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.146778139 (538h33m41.017861184s) [noise=true , address=145.79.227.43:4161 , piece_id=ZN3T6JSPHJ57X4YYG5MHRJYYOFPH2T2CSGJ34MJQU55HDC7CMOXQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.147137874 (538h33m41.018220919s) [noise=true , address=77.90.31.21:4882 , piece_id=WQAGQTGLU5HRKEET36W34MQP4NPGSVCA4QZ6YEGVAJWBBPLZOUQQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.147426784 (538h33m41.018509819s) [noise=true , address=145.79.227.35:4503 , piece_id=MDYJ2TROT4FHLX4XHKHEQACC7ZXHZCQ5MUPJL5EDS3F2YMXSUGUA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.147466109 (538h33m41.018549154s) [noise=true , address=77.90.31.15:4730 , piece_id=EN7IGRJZ2WFN4HXJPMNLLXDUZPIN7KDUGKPPLJH7CSA4SDOUM2JA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.147817368 (538h33m41.018900413s) [noise=true , address=77.90.31.18:4833 , piece_id=3E57LWLI3WHROFGMXVZ7LHDHO5O7453LWQXA3OEKM52B2NIYM2JA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.147860690 (538h33m41.018943745s) [noise=true , address=145.79.227.44:4212 , piece_id=EFTTQM6323MUQ5WTGY6DFBTF3TGRM6XZWXIGT6DM4VS7GRGM4C2Q, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148163427 (538h33m41.019246472s) [noise=true , address=77.90.31.19:4862 , piece_id=PSBBBSGY7THVLMXJQCADIE7VYZ33OIIDVAEQWG3RWMNA6RSKELNQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148189276 (538h33m41.019272321s) [noise=true , address=77.90.31.10:4524 , piece_id=X2VCBS4PTPPWI6U64T4FVIULRLCDY3XUOJTXJNP34OHSO54ONIWA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148496531 (538h33m41.019579566s) [noise=true , address=145.79.227.23:4556 , piece_id=AKBXRSU5PM5QLHGHE3VXTJ3ZH23BECPGBLQSVZK6XVIYJGVDXT4Q, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148527380 (538h33m41.019610425s) [noise=true , address=145.79.227.33:4899 , piece_id=JFEXXK22EATV7UUH7LLXTFEPIUBW7EWSNYFXLE4AO5F6SLYFTGBA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148801512 (538h33m41.019884547s) [noise=true , address=77.90.31.14:4672 , piece_id=BZG2Y5X47J2WVFF5LBHEJZNUD2OQ3NOTVPTCAVT4TLFQB5FWESLA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148894920 (538h33m41.019977965s) [noise=true , address=5.231.73.5:4016 , piece_id=YCNKQEKJQSJHKKGCYWLOLEBKIVPJHZIJ6TYEG6DF4TRXMZ3ZTTIQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.149118806 (538h33m41.020201851s) [noise=true , address=145.79.227.47:4295 , piece_id=OM7B75MCPM7L3WP5MLQ7MAVBZ2NJUSISOWD56RSWI2AV2OWVDRTA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.149274993 (538h33m41.020358038s) [noise=true , address=145.79.227.50:4400 , piece_id=HZAT7YAVZYWK52FE4TIC6OA37U4VX5B35H55WMBO3MMWNIKTJLJQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.149625842 (538h33m41.020708877s) [noise=true , address=145.79.227.34:4915 , piece_id=RMXYZGGJA5EPJLQD6SQG677HFLOQWRY2HFTGEOICYAQG6QR557RQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:42.618098549 (538h33m42.489181584s) [upload=472313]: Next returned nil segment Dec 28 02:02:42.618120911 (538h33m42.489203946s) [upload=472313]: Waiting for error group managing segments... Dec 28 02:02:43.021704832 (538h33m42.892787877s) [noise=true , address=145.79.227.23:4556 , piece_id=AKBXRSU5PM5QLHGHE3VXTJ3ZH23BECPGBLQSVZK6XVIYJGVDXT4Q, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.027980492 (538h33m42.899063537s) [noise=true , address=145.79.227.50:4400 , piece_id=HZAT7YAVZYWK52FE4TIC6OA37U4VX5B35H55WMBO3MMWNIKTJLJQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.043329239 (538h33m42.914412284s) [noise=true , address=145.79.227.15:4908 , piece_id=I2FSDNVYBFZKRXZNUHH2APY7N2PI7DPAKFKGXRJW2UFU2PBKSSNQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.056308314 (538h33m42.927391349s) [noise=true , address=145.79.227.49:4347 , piece_id=WURIHGQASBLSVLZJVM5EJHWD7U4VFLLPGJIFYNOK6ALA5LYEQDSA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.074628196 (538h33m42.945711252s) [noise=true , address=145.79.227.47:4295 , piece_id=OM7B75MCPM7L3WP5MLQ7MAVBZ2NJUSISOWD56RSWI2AV2OWVDRTA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.080495740 (538h33m42.951578775s) [noise=true , address=5.231.73.5:4016 , piece_id=YCNKQEKJQSJHKKGCYWLOLEBKIVPJHZIJ6TYEG6DF4TRXMZ3ZTTIQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.193955176 (538h33m43.065038241s) [noise=true , address=77.90.31.10:4551 , piece_id=KBPO4MUDASHKMPRNJELQWLNMJNQBB3PZUOVNCEPP2CYVHU6X2PPA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.301513865 (538h33m43.1725969s) [noise=true , address=77.90.31.15:4730 , piece_id=EN7IGRJZ2WFN4HXJPMNLLXDUZPIN7KDUGKPPLJH7CSA4SDOUM2JA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.305020622 (538h33m43.176103657s) [noise=true , address=77.90.31.19:4862 , piece_id=PSBBBSGY7THVLMXJQCADIE7VYZ33OIIDVAEQWG3RWMNA6RSKELNQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.312187619 (538h33m43.183270654s) [noise=true , address=77.90.31.14:4672 , piece_id=BZG2Y5X47J2WVFF5LBHEJZNUD2OQ3NOTVPTCAVT4TLFQB5FWESLA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.319277390 (538h33m43.190360425s) [noise=true , address=77.90.31.21:4882 , piece_id=WQAGQTGLU5HRKEET36W34MQP4NPGSVCA4QZ6YEGVAJWBBPLZOUQQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.948916959 (538h33m43.820000004s) [noise=true , address=145.79.227.34:4915 , piece_id=RMXYZGGJA5EPJLQD6SQG677HFLOQWRY2HFTGEOICYAQG6QR557RQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.989823331 (538h33m43.860906366s) [noise=true , address=145.79.227.44:4212 , piece_id=EFTTQM6323MUQ5WTGY6DFBTF3TGRM6XZWXIGT6DM4VS7GRGM4C2Q, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:44.123218949 (538h33m43.994301984s) [noise=true , address=145.79.227.43:4161 , piece_id=ZN3T6JSPHJ57X4YYG5MHRJYYOFPH2T2CSGJ34MJQU55HDC7CMOXQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:44.123238386 (538h33m43.994321421s) [seg_pos={0 0}, upload=472313]: Segment reached optimal threshold of 14 pieces. Dec 28 02:02:44.123431754 (538h33m43.994514789s) [noise=true , address=145.79.227.33:4899 , piece_id=JFEXXK22EATV7UUH7LLXTFEPIUBW7EWSNYFXLE4AO5F6SLYFTGBA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b4pZVQRnjcQNn45VTgdsz5fpjYPMAzSXGR5KJ5BP9XHMkRnRw): protocol: expected piece hash; context canceled; EOF Dec 28 02:02:44.123437586 (538h33m43.994520621s) [noise=true , address=145.79.227.33:4899 , piece_id=JFEXXK22EATV7UUH7LLXTFEPIUBW7EWSNYFXLE4AO5F6SLYFTGBA, seg_pos={0 0}, upload=472313]: Long tail cancellation due to optimal threshold Dec 28 02:02:44.123631064 (538h33m43.994714109s) [noise=true , address=77.90.31.18:4833 , piece_id=3E57LWLI3WHROFGMXVZ7LHDHO5O7453LWQXA3OEKM52B2NIYM2JA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FbVbo8kGy1aaFyCk2noiXNUJHtosFJHFykBMRecGd5Kj5gjEp): protocol: expected piece hash; context canceled; EOF Dec 28 02:02:44.123675267 (538h33m43.994758313s) [noise=true , address=77.90.31.18:4833 , piece_id=3E57LWLI3WHROFGMXVZ7LHDHO5O7453LWQXA3OEKM52B2NIYM2JA, seg_pos={0 0}, upload=472313]: Long tail cancellation due to optimal threshold Dec 28 02:02:44.123774066 (538h33m43.994857101s) [noise=true , address=77.90.31.10:4524 , piece_id=X2VCBS4PTPPWI6U64T4FVIULRLCDY3XUOJTXJNP34OHSO54ONIWA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe): protocol: expected piece hash; context canceled; EOF Dec 28 02:02:44.123781761 (538h33m43.994864796s) [noise=true , address=77.90.31.10:4524 , piece_id=X2VCBS4PTPPWI6U64T4FVIULRLCDY3XUOJTXJNP34OHSO54ONIWA, seg_pos={0 0}, upload=472313]: Long tail cancellation due to optimal threshold Dec 28 02:02:44.123923341 (538h33m43.995006376s) [noise=true , address=145.79.227.35:4503 , piece_id=MDYJ2TROT4FHLX4XHKHEQACC7ZXHZCQ5MUPJL5EDS3F2YMXSUGUA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ctmSF1aMQasuXByFW8DppDantREmmWL2WUjmKGW5ZULVePyvm): protocol: expected piece hash; context canceled; EOF Dec 28 02:02:44.123928591 (538h33m43.995011636s) [noise=true , address=145.79.227.35:4503 , piece_id=MDYJ2TROT4FHLX4XHKHEQACC7ZXHZCQ5MUPJL5EDS3F2YMXSUGUA, seg_pos={0 0}, upload=472313]: Long tail cancellation due to optimal threshold Dec 28 02:02:44.123944731 (538h33m43.995027766s) [seg_pos={0 0}, upload=472313]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 02:02:44.124017179 (538h33m43.995100214s) [upload=472313]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:02:44.124022469 (538h33m43.995105504s) [upload=472313]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:02:44.409047274 (538h33m44.280130349s) [upload=472313]: Done uploading segments... Dec 28 02:02:44.409073354 (538h33m44.280156389s) [upload=472313]: Upload finished. err: <nil> Duration: 32.554438116s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 34 out of 60 is failed (threshold=3) [.X.XX.XXXXX..X.XXX.X.XX.XX.X..X.X.X.X.X..X...XX..X.XXX.XXX.X] duration: 30.003638409s trace: https://jaeger.dev.storj.io/trace/1c22d34f4ac03822 FAIL (2025-12-28 02:16:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: GU4X73Q6AFIBRIYAG4ASNF22MDMCKUOKIJCS3TTJT3FKDWLAMIKA) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: DQ4IHJQYEPBEW63HH3ELSFGDFZUHJ7HL6UHBLGQRHKOHTD7HP2SQ) context deadline exceeded ======LOGS====== Dec 28 02:15:53.759588852 (538h46m53.630671907s) [upload=472506]: Starting upload Dec 28 02:15:53.759803180 (538h46m53.630886235s) [upload=472506]: Done starting upload Dec 28 02:15:53.759860660 (538h46m53.630943705s) [upload=472506]: Uploading segments... Dec 28 02:15:53.760077182 (538h46m53.631160227s) [upload=472506]: Got next segment. Inline: false Dec 28 02:15:53.760126496 (538h46m53.631209541s) [upload=472506]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 02:15:53.760133219 (538h46m53.631216264s) [upload=472506]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 02:15:54.003115733 (538h46m53.874198788s) [seg_pos={0 0}, upload=472506]: Begin upload segment... Dec 28 02:15:54.003896579 (538h46m53.874979644s) [seg_pos={0 0}, upload=472506]: Done begin upload segment. Dec 28 02:15:54.003960000 (538h46m53.875043055s) [noise=true , address=145.79.227.16:4909 , piece_id=KSGRHS6G6JZ7IY23T277NMHN6HGG7DUODYBIXENFS2PSTYNUX37Q, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.003967354 (538h46m53.875050399s) [noise=true , address=145.79.227.52:4445 , piece_id=ZEY2QW4N6ZNGBILKIDQ7YBFSB7U2JVICZJV2W3GQU7D3QACB32FQ, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.004662528 (538h46m53.875745573s) [noise=true , address=145.79.227.31:4797 , piece_id=J5OPI7TBATORMGHB245L3YCWH2I56AVGHVAWJRLALMYDQIW7YAGQ, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.004851216 (538h46m53.875934261s) [noise=true , address=45.91.116.11:4118 , piece_id=LA7UXPHDDM25MJ4IKX4IGJXKZFA37TTXYBAGMA25C3Y57X6YFQ3Q, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.005240899 (538h46m53.876323944s) [noise=true , address=145.79.227.4:4185 , piece_id=SKBIGBFEIKWVJJFKWFM6TO5CYQ6RCM6RLV6AXCM4TKG7TS6TAVXQ, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.005594032 (538h46m53.876677067s) [noise=true , address=77.90.31.16:4735 , piece_id=VD6RDROPI6XQA2TYQUB4ENXFD2B6OCM2YH5AXP3OVGPQ5YBI37BA, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.005750930 (538h46m53.876833975s) [noise=true , address=145.79.227.18:4006 , piece_id=GYNQ6GGPHVJJYRV2RHRL63SFS4UPSONZY6PPPVC4ZTSOK5SVWVLQ, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.006059718 (538h46m53.877142793s) [noise=true , address=45.91.116.14:4190 , piece_id=ZDWBLJMWUZN4ZMK5X6FWN24VEWAFU2KODQJ7URBZBWJHIML4GKKA, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.006277232 (538h46m53.877360277s) [noise=true , address=145.79.227.17:4972 , piece_id=64UKV76NMUYNSLPZA3WLTQ6F7W2XP5Y564DFNQMLBFB3SG6S3IFA, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.006515977 (538h46m53.877599022s) [noise=true , address=45.91.116.6:4973 , piece_id=GU4X73Q6AFIBRIYAG4ASNF22MDMCKUOKIJCS3TTJT3FKDWLAMIKA, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.007002723 (538h46m53.878085789s) [noise=true , address=145.79.227.19:4077 , piece_id=4LCN23MUWAX2HK3J2OFAFLEGUQM4YYTJWZ7IPCN3D6QROORY5O6A, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.007095439 (538h46m53.878178484s) [noise=true , address=77.90.31.14:4661 , piece_id=U2S5KQX5UAKMMMIOZH6CP7SE7PR2DZIKHKX7OXBCJMSZKG4RUMMQ, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.007443692 (538h46m53.878526728s) [noise=true , address=45.91.116.7:4009 , piece_id=DQ4IHJQYEPBEW63HH3ELSFGDFZUHJ7HL6UHBLGQRHKOHTD7HP2SQ, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.007675033 (538h46m53.878758078s) [noise=true , address=145.79.227.32:4836 , piece_id=AZ5NCPW2VX2VSH7XGAUEWC2B25ZEQ2NVQVZZUL6CANSCBWIIEY3A, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.007858693 (538h46m53.878941738s) [noise=true , address=145.79.227.27:4691 , piece_id=MJMBV5LR4RA5ZP7Q7U34DXO2TWJ6LDKH4YMADM3K5CU2KRHMQWBA, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:54.008204601 (538h46m53.879287637s) [noise=true , address=145.79.227.12:4767 , piece_id=A3L6CAJZWVR4RYOZMKQ5C5UALP6FKW367UGSOCW3YEVLWMYJBBTQ, seg_pos={0 0}, upload=472506]: Uploading piece... Dec 28 02:15:55.745297151 (538h46m55.616380186s) [upload=472506]: Next returned nil segment Dec 28 02:15:55.745351053 (538h46m55.616434108s) [upload=472506]: Waiting for error group managing segments... Dec 28 02:15:56.032845416 (538h46m55.903928451s) [noise=true , address=145.79.227.18:4006 , piece_id=GYNQ6GGPHVJJYRV2RHRL63SFS4UPSONZY6PPPVC4ZTSOK5SVWVLQ, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.202191721 (538h46m56.073274756s) [noise=true , address=145.79.227.19:4077 , piece_id=4LCN23MUWAX2HK3J2OFAFLEGUQM4YYTJWZ7IPCN3D6QROORY5O6A, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.205182144 (538h46m56.076265189s) [noise=true , address=145.79.227.31:4797 , piece_id=J5OPI7TBATORMGHB245L3YCWH2I56AVGHVAWJRLALMYDQIW7YAGQ, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.205433684 (538h46m56.076516719s) [noise=true , address=145.79.227.27:4691 , piece_id=MJMBV5LR4RA5ZP7Q7U34DXO2TWJ6LDKH4YMADM3K5CU2KRHMQWBA, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.214467002 (538h46m56.085550047s) [noise=true , address=145.79.227.16:4909 , piece_id=KSGRHS6G6JZ7IY23T277NMHN6HGG7DUODYBIXENFS2PSTYNUX37Q, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.214875340 (538h46m56.085958375s) [noise=true , address=145.79.227.4:4185 , piece_id=SKBIGBFEIKWVJJFKWFM6TO5CYQ6RCM6RLV6AXCM4TKG7TS6TAVXQ, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.217932621 (538h46m56.089015656s) [noise=true , address=145.79.227.12:4767 , piece_id=A3L6CAJZWVR4RYOZMKQ5C5UALP6FKW367UGSOCW3YEVLWMYJBBTQ, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.306475972 (538h46m56.177559007s) [noise=true , address=45.91.116.6:4973 , piece_id=GU4X73Q6AFIBRIYAG4ASNF22MDMCKUOKIJCS3TTJT3FKDWLAMIKA, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.385055362 (538h46m56.256138408s) [noise=true , address=45.91.116.7:4009 , piece_id=DQ4IHJQYEPBEW63HH3ELSFGDFZUHJ7HL6UHBLGQRHKOHTD7HP2SQ, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.395589739 (538h46m56.266672784s) [noise=true , address=45.91.116.14:4190 , piece_id=ZDWBLJMWUZN4ZMK5X6FWN24VEWAFU2KODQJ7URBZBWJHIML4GKKA, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.395681062 (538h46m56.266764107s) [noise=true , address=145.79.227.52:4445 , piece_id=ZEY2QW4N6ZNGBILKIDQ7YBFSB7U2JVICZJV2W3GQU7D3QACB32FQ, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:56.403044182 (538h46m56.274127217s) [noise=true , address=145.79.227.32:4836 , piece_id=AZ5NCPW2VX2VSH7XGAUEWC2B25ZEQ2NVQVZZUL6CANSCBWIIEY3A, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:57.110570880 (538h46m56.981653925s) [noise=true , address=45.91.116.11:4118 , piece_id=LA7UXPHDDM25MJ4IKX4IGJXKZFA37TTXYBAGMA25C3Y57X6YFQ3Q, seg_pos={0 0}, upload=472506]: Done uploading piece. err: <nil> Dec 28 02:15:57.110788534 (538h46m56.981871569s) [seg_pos={0 0}, upload=472506]: Segment reached optimal threshold of 13 pieces. Dec 28 02:15:57.111013823 (538h46m56.982096858s) [noise=true , address=77.90.31.14:4661 , piece_id=U2S5KQX5UAKMMMIOZH6CP7SE7PR2DZIKHKX7OXBCJMSZKG4RUMMQ, seg_pos={0 0}, upload=472506]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 28 02:15:57.111022289 (538h46m56.982105335s) [noise=true , address=77.90.31.14:4661 , piece_id=U2S5KQX5UAKMMMIOZH6CP7SE7PR2DZIKHKX7OXBCJMSZKG4RUMMQ, seg_pos={0 0}, upload=472506]: Long tail cancellation due to optimal threshold Dec 28 02:15:57.111365041 (538h46m56.982448076s) [noise=true , address=77.90.31.16:4735 , piece_id=VD6RDROPI6XQA2TYQUB4ENXFD2B6OCM2YH5AXP3OVGPQ5YBI37BA, seg_pos={0 0}, upload=472506]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: read tcp 188.166.225.170:47380->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 02:15:57.111372196 (538h46m56.982455241s) [noise=true , address=77.90.31.16:4735 , piece_id=VD6RDROPI6XQA2TYQUB4ENXFD2B6OCM2YH5AXP3OVGPQ5YBI37BA, seg_pos={0 0}, upload=472506]: Long tail cancellation due to optimal threshold Dec 28 02:15:57.111468879 (538h46m56.982551924s) [noise=true , address=145.79.227.17:4972 , piece_id=64UKV76NMUYNSLPZA3WLTQ6F7W2XP5Y564DFNQMLBFB3SG6S3IFA, seg_pos={0 0}, upload=472506]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: expected piece hash; context canceled; EOF Dec 28 02:15:57.111474270 (538h46m56.982557315s) [noise=true , address=145.79.227.17:4972 , piece_id=64UKV76NMUYNSLPZA3WLTQ6F7W2XP5Y564DFNQMLBFB3SG6S3IFA, seg_pos={0 0}, upload=472506]: Long tail cancellation due to optimal threshold Dec 28 02:15:57.111493035 (538h46m56.98257607s) [seg_pos={0 0}, upload=472506]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 02:15:57.111624576 (538h46m56.982707611s) [upload=472506]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:15:57.111630667 (538h46m56.982713713s) [upload=472506]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:15:57.409023835 (538h46m57.28010687s) [upload=472506]: Done uploading segments... Dec 28 02:15:57.409043784 (538h46m57.280126819s) [upload=472506]: Upload finished. err: <nil> Duration: 30.003654049s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 44.601874219s trace: https://jaeger.dev.storj.io/trace/12a1d059b6f75af7 FAIL (2025-12-28 02:16:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1.145865281 (538h33m41.016948316s) [noise=true , address=77.90.31.10:4551 , piece_id=KBPO4MUDASHKMPRNJELQWLNMJNQBB3PZUOVNCEPP2CYVHU6X2PPA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.146408725 (538h33m41.01749177s) [noise=true , address=145.79.227.15:4908 , piece_id=I2FSDNVYBFZKRXZNUHH2APY7N2PI7DPAKFKGXRJW2UFU2PBKSSNQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.146778139 (538h33m41.017861184s) [noise=true , address=145.79.227.43:4161 , piece_id=ZN3T6JSPHJ57X4YYG5MHRJYYOFPH2T2CSGJ34MJQU55HDC7CMOXQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.147137874 (538h33m41.018220919s) [noise=true , address=77.90.31.21:4882 , piece_id=WQAGQTGLU5HRKEET36W34MQP4NPGSVCA4QZ6YEGVAJWBBPLZOUQQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.147426784 (538h33m41.018509819s) [noise=true , address=145.79.227.35:4503 , piece_id=MDYJ2TROT4FHLX4XHKHEQACC7ZXHZCQ5MUPJL5EDS3F2YMXSUGUA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.147466109 (538h33m41.018549154s) [noise=true , address=77.90.31.15:4730 , piece_id=EN7IGRJZ2WFN4HXJPMNLLXDUZPIN7KDUGKPPLJH7CSA4SDOUM2JA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.147817368 (538h33m41.018900413s) [noise=true , address=77.90.31.18:4833 , piece_id=3E57LWLI3WHROFGMXVZ7LHDHO5O7453LWQXA3OEKM52B2NIYM2JA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.147860690 (538h33m41.018943745s) [noise=true , address=145.79.227.44:4212 , piece_id=EFTTQM6323MUQ5WTGY6DFBTF3TGRM6XZWXIGT6DM4VS7GRGM4C2Q, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148163427 (538h33m41.019246472s) [noise=true , address=77.90.31.19:4862 , piece_id=PSBBBSGY7THVLMXJQCADIE7VYZ33OIIDVAEQWG3RWMNA6RSKELNQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148189276 (538h33m41.019272321s) [noise=true , address=77.90.31.10:4524 , piece_id=X2VCBS4PTPPWI6U64T4FVIULRLCDY3XUOJTXJNP34OHSO54ONIWA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148496531 (538h33m41.019579566s) [noise=true , address=145.79.227.23:4556 , piece_id=AKBXRSU5PM5QLHGHE3VXTJ3ZH23BECPGBLQSVZK6XVIYJGVDXT4Q, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148527380 (538h33m41.019610425s) [noise=true , address=145.79.227.33:4899 , piece_id=JFEXXK22EATV7UUH7LLXTFEPIUBW7EWSNYFXLE4AO5F6SLYFTGBA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148801512 (538h33m41.019884547s) [noise=true , address=77.90.31.14:4672 , piece_id=BZG2Y5X47J2WVFF5LBHEJZNUD2OQ3NOTVPTCAVT4TLFQB5FWESLA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148894920 (538h33m41.019977965s) [noise=true , address=5.231.73.5:4016 , piece_id=YCNKQEKJQSJHKKGCYWLOLEBKIVPJHZIJ6TYEG6DF4TRXMZ3ZTTIQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.149118806 (538h33m41.020201851s) [noise=true , address=145.79.227.47:4295 , piece_id=OM7B75MCPM7L3WP5MLQ7MAVBZ2NJUSISOWD56RSWI2AV2OWVDRTA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.149274993 (538h33m41.020358038s) [noise=true , address=145.79.227.50:4400 , piece_id=HZAT7YAVZYWK52FE4TIC6OA37U4VX5B35H55WMBO3MMWNIKTJLJQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.149625842 (538h33m41.020708877s) [noise=true , address=145.79.227.34:4915 , piece_id=RMXYZGGJA5EPJLQD6SQG677HFLOQWRY2HFTGEOICYAQG6QR557RQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:42.618098549 (538h33m42.489181584s) [upload=472313]: Next returned nil segment Dec 28 02:02:42.618120911 (538h33m42.489203946s) [upload=472313]: Waiting for error group managing segments... Dec 28 02:02:43.021704832 (538h33m42.892787877s) [noise=true , address=145.79.227.23:4556 , piece_id=AKBXRSU5PM5QLHGHE3VXTJ3ZH23BECPGBLQSVZK6XVIYJGVDXT4Q, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.027980492 (538h33m42.899063537s) [noise=true , address=145.79.227.50:4400 , piece_id=HZAT7YAVZYWK52FE4TIC6OA37U4VX5B35H55WMBO3MMWNIKTJLJQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.043329239 (538h33m42.914412284s) [noise=true , address=145.79.227.15:4908 , piece_id=I2FSDNVYBFZKRXZNUHH2APY7N2PI7DPAKFKGXRJW2UFU2PBKSSNQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.056308314 (538h33m42.927391349s) [noise=true , address=145.79.227.49:4347 , piece_id=WURIHGQASBLSVLZJVM5EJHWD7U4VFLLPGJIFYNOK6ALA5LYEQDSA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.074628196 (538h33m42.945711252s) [noise=true , address=145.79.227.47:4295 , piece_id=OM7B75MCPM7L3WP5MLQ7MAVBZ2NJUSISOWD56RSWI2AV2OWVDRTA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.080495740 (538h33m42.951578775s) [noise=true , address=5.231.73.5:4016 , piece_id=YCNKQEKJQSJHKKGCYWLOLEBKIVPJHZIJ6TYEG6DF4TRXMZ3ZTTIQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.193955176 (538h33m43.065038241s) [noise=true , address=77.90.31.10:4551 , piece_id=KBPO4MUDASHKMPRNJELQWLNMJNQBB3PZUOVNCEPP2CYVHU6X2PPA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.301513865 (538h33m43.1725969s) [noise=true , address=77.90.31.15:4730 , piece_id=EN7IGRJZ2WFN4HXJPMNLLXDUZPIN7KDUGKPPLJH7CSA4SDOUM2JA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.305020622 (538h33m43.176103657s) [noise=true , address=77.90.31.19:4862 , piece_id=PSBBBSGY7THVLMXJQCADIE7VYZ33OIIDVAEQWG3RWMNA6RSKELNQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.312187619 (538h33m43.183270654s) [noise=true , address=77.90.31.14:4672 , piece_id=BZG2Y5X47J2WVFF5LBHEJZNUD2OQ3NOTVPTCAVT4TLFQB5FWESLA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.319277390 (538h33m43.190360425s) [noise=true , address=77.90.31.21:4882 , piece_id=WQAGQTGLU5HRKEET36W34MQP4NPGSVCA4QZ6YEGVAJWBBPLZOUQQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.948916959 (538h33m43.820000004s) [noise=true , address=145.79.227.34:4915 , piece_id=RMXYZGGJA5EPJLQD6SQG677HFLOQWRY2HFTGEOICYAQG6QR557RQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.989823331 (538h33m43.860906366s) [noise=true , address=145.79.227.44:4212 , piece_id=EFTTQM6323MUQ5WTGY6DFBTF3TGRM6XZWXIGT6DM4VS7GRGM4C2Q, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:44.123218949 (538h33m43.994301984s) [noise=true , address=145.79.227.43:4161 , piece_id=ZN3T6JSPHJ57X4YYG5MHRJYYOFPH2T2CSGJ34MJQU55HDC7CMOXQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:44.123238386 (538h33m43.994321421s) [seg_pos={0 0}, upload=472313]: Segment reached optimal threshold of 14 pieces. Dec 28 02:02:44.123431754 (538h33m43.994514789s) [noise=true , address=145.79.227.33:4899 , piece_id=JFEXXK22EATV7UUH7LLXTFEPIUBW7EWSNYFXLE4AO5F6SLYFTGBA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b4pZVQRnjcQNn45VTgdsz5fpjYPMAzSXGR5KJ5BP9XHMkRnRw): protocol: expected piece hash; context canceled; EOF Dec 28 02:02:44.123437586 (538h33m43.994520621s) [noise=true , address=145.79.227.33:4899 , piece_id=JFEXXK22EATV7UUH7LLXTFEPIUBW7EWSNYFXLE4AO5F6SLYFTGBA, seg_pos={0 0}, upload=472313]: Long tail cancellation due to optimal threshold Dec 28 02:02:44.123631064 (538h33m43.994714109s) [noise=true , address=77.90.31.18:4833 , piece_id=3E57LWLI3WHROFGMXVZ7LHDHO5O7453LWQXA3OEKM52B2NIYM2JA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FbVbo8kGy1aaFyCk2noiXNUJHtosFJHFykBMRecGd5Kj5gjEp): protocol: expected piece hash; context canceled; EOF Dec 28 02:02:44.123675267 (538h33m43.994758313s) [noise=true , address=77.90.31.18:4833 , piece_id=3E57LWLI3WHROFGMXVZ7LHDHO5O7453LWQXA3OEKM52B2NIYM2JA, seg_pos={0 0}, upload=472313]: Long tail cancellation due to optimal threshold Dec 28 02:02:44.123774066 (538h33m43.994857101s) [noise=true , address=77.90.31.10:4524 , piece_id=X2VCBS4PTPPWI6U64T4FVIULRLCDY3XUOJTXJNP34OHSO54ONIWA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe): protocol: expected piece hash; context canceled; EOF Dec 28 02:02:44.123781761 (538h33m43.994864796s) [noise=true , address=77.90.31.10:4524 , piece_id=X2VCBS4PTPPWI6U64T4FVIULRLCDY3XUOJTXJNP34OHSO54ONIWA, seg_pos={0 0}, upload=472313]: Long tail cancellation due to optimal threshold Dec 28 02:02:44.123923341 (538h33m43.995006376s) [noise=true , address=145.79.227.35:4503 , piece_id=MDYJ2TROT4FHLX4XHKHEQACC7ZXHZCQ5MUPJL5EDS3F2YMXSUGUA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ctmSF1aMQasuXByFW8DppDantREmmWL2WUjmKGW5ZULVePyvm): protocol: expected piece hash; context canceled; EOF Dec 28 02:02:44.123928591 (538h33m43.995011636s) [noise=true , address=145.79.227.35:4503 , piece_id=MDYJ2TROT4FHLX4XHKHEQACC7ZXHZCQ5MUPJL5EDS3F2YMXSUGUA, seg_pos={0 0}, upload=472313]: Long tail cancellation due to optimal threshold Dec 28 02:02:44.123944731 (538h33m43.995027766s) [seg_pos={0 0}, upload=472313]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 02:02:44.124017179 (538h33m43.995100214s) [upload=472313]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:02:44.124022469 (538h33m43.995105504s) [upload=472313]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:02:44.409047274 (538h33m44.280130349s) [upload=472313]: Done uploading segments... Dec 28 02:02:44.409073354 (538h33m44.280156389s) [upload=472313]: Upload finished. err: <nil> Duration: 44.601900568s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............................................X.............] duration: 3.438670832s trace: https://jaeger.dev.storj.io/trace/7fc142ad3d711e34 FAIL (2025-12-28 02:16:01 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.438704587s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 48.533837308s trace: https://jaeger.dev.storj.io/trace/12a1d059b6f75af7 FAIL (2025-12-28 02:15:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1.145865281 (538h33m41.016948316s) [noise=true , address=77.90.31.10:4551 , piece_id=KBPO4MUDASHKMPRNJELQWLNMJNQBB3PZUOVNCEPP2CYVHU6X2PPA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.146408725 (538h33m41.01749177s) [noise=true , address=145.79.227.15:4908 , piece_id=I2FSDNVYBFZKRXZNUHH2APY7N2PI7DPAKFKGXRJW2UFU2PBKSSNQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.146778139 (538h33m41.017861184s) [noise=true , address=145.79.227.43:4161 , piece_id=ZN3T6JSPHJ57X4YYG5MHRJYYOFPH2T2CSGJ34MJQU55HDC7CMOXQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.147137874 (538h33m41.018220919s) [noise=true , address=77.90.31.21:4882 , piece_id=WQAGQTGLU5HRKEET36W34MQP4NPGSVCA4QZ6YEGVAJWBBPLZOUQQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.147426784 (538h33m41.018509819s) [noise=true , address=145.79.227.35:4503 , piece_id=MDYJ2TROT4FHLX4XHKHEQACC7ZXHZCQ5MUPJL5EDS3F2YMXSUGUA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.147466109 (538h33m41.018549154s) [noise=true , address=77.90.31.15:4730 , piece_id=EN7IGRJZ2WFN4HXJPMNLLXDUZPIN7KDUGKPPLJH7CSA4SDOUM2JA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.147817368 (538h33m41.018900413s) [noise=true , address=77.90.31.18:4833 , piece_id=3E57LWLI3WHROFGMXVZ7LHDHO5O7453LWQXA3OEKM52B2NIYM2JA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.147860690 (538h33m41.018943745s) [noise=true , address=145.79.227.44:4212 , piece_id=EFTTQM6323MUQ5WTGY6DFBTF3TGRM6XZWXIGT6DM4VS7GRGM4C2Q, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148163427 (538h33m41.019246472s) [noise=true , address=77.90.31.19:4862 , piece_id=PSBBBSGY7THVLMXJQCADIE7VYZ33OIIDVAEQWG3RWMNA6RSKELNQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148189276 (538h33m41.019272321s) [noise=true , address=77.90.31.10:4524 , piece_id=X2VCBS4PTPPWI6U64T4FVIULRLCDY3XUOJTXJNP34OHSO54ONIWA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148496531 (538h33m41.019579566s) [noise=true , address=145.79.227.23:4556 , piece_id=AKBXRSU5PM5QLHGHE3VXTJ3ZH23BECPGBLQSVZK6XVIYJGVDXT4Q, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148527380 (538h33m41.019610425s) [noise=true , address=145.79.227.33:4899 , piece_id=JFEXXK22EATV7UUH7LLXTFEPIUBW7EWSNYFXLE4AO5F6SLYFTGBA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148801512 (538h33m41.019884547s) [noise=true , address=77.90.31.14:4672 , piece_id=BZG2Y5X47J2WVFF5LBHEJZNUD2OQ3NOTVPTCAVT4TLFQB5FWESLA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148894920 (538h33m41.019977965s) [noise=true , address=5.231.73.5:4016 , piece_id=YCNKQEKJQSJHKKGCYWLOLEBKIVPJHZIJ6TYEG6DF4TRXMZ3ZTTIQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.149118806 (538h33m41.020201851s) [noise=true , address=145.79.227.47:4295 , piece_id=OM7B75MCPM7L3WP5MLQ7MAVBZ2NJUSISOWD56RSWI2AV2OWVDRTA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.149274993 (538h33m41.020358038s) [noise=true , address=145.79.227.50:4400 , piece_id=HZAT7YAVZYWK52FE4TIC6OA37U4VX5B35H55WMBO3MMWNIKTJLJQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.149625842 (538h33m41.020708877s) [noise=true , address=145.79.227.34:4915 , piece_id=RMXYZGGJA5EPJLQD6SQG677HFLOQWRY2HFTGEOICYAQG6QR557RQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:42.618098549 (538h33m42.489181584s) [upload=472313]: Next returned nil segment Dec 28 02:02:42.618120911 (538h33m42.489203946s) [upload=472313]: Waiting for error group managing segments... Dec 28 02:02:43.021704832 (538h33m42.892787877s) [noise=true , address=145.79.227.23:4556 , piece_id=AKBXRSU5PM5QLHGHE3VXTJ3ZH23BECPGBLQSVZK6XVIYJGVDXT4Q, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.027980492 (538h33m42.899063537s) [noise=true , address=145.79.227.50:4400 , piece_id=HZAT7YAVZYWK52FE4TIC6OA37U4VX5B35H55WMBO3MMWNIKTJLJQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.043329239 (538h33m42.914412284s) [noise=true , address=145.79.227.15:4908 , piece_id=I2FSDNVYBFZKRXZNUHH2APY7N2PI7DPAKFKGXRJW2UFU2PBKSSNQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.056308314 (538h33m42.927391349s) [noise=true , address=145.79.227.49:4347 , piece_id=WURIHGQASBLSVLZJVM5EJHWD7U4VFLLPGJIFYNOK6ALA5LYEQDSA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.074628196 (538h33m42.945711252s) [noise=true , address=145.79.227.47:4295 , piece_id=OM7B75MCPM7L3WP5MLQ7MAVBZ2NJUSISOWD56RSWI2AV2OWVDRTA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.080495740 (538h33m42.951578775s) [noise=true , address=5.231.73.5:4016 , piece_id=YCNKQEKJQSJHKKGCYWLOLEBKIVPJHZIJ6TYEG6DF4TRXMZ3ZTTIQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.193955176 (538h33m43.065038241s) [noise=true , address=77.90.31.10:4551 , piece_id=KBPO4MUDASHKMPRNJELQWLNMJNQBB3PZUOVNCEPP2CYVHU6X2PPA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.301513865 (538h33m43.1725969s) [noise=true , address=77.90.31.15:4730 , piece_id=EN7IGRJZ2WFN4HXJPMNLLXDUZPIN7KDUGKPPLJH7CSA4SDOUM2JA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.305020622 (538h33m43.176103657s) [noise=true , address=77.90.31.19:4862 , piece_id=PSBBBSGY7THVLMXJQCADIE7VYZ33OIIDVAEQWG3RWMNA6RSKELNQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.312187619 (538h33m43.183270654s) [noise=true , address=77.90.31.14:4672 , piece_id=BZG2Y5X47J2WVFF5LBHEJZNUD2OQ3NOTVPTCAVT4TLFQB5FWESLA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.319277390 (538h33m43.190360425s) [noise=true , address=77.90.31.21:4882 , piece_id=WQAGQTGLU5HRKEET36W34MQP4NPGSVCA4QZ6YEGVAJWBBPLZOUQQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.948916959 (538h33m43.820000004s) [noise=true , address=145.79.227.34:4915 , piece_id=RMXYZGGJA5EPJLQD6SQG677HFLOQWRY2HFTGEOICYAQG6QR557RQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.989823331 (538h33m43.860906366s) [noise=true , address=145.79.227.44:4212 , piece_id=EFTTQM6323MUQ5WTGY6DFBTF3TGRM6XZWXIGT6DM4VS7GRGM4C2Q, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:44.123218949 (538h33m43.994301984s) [noise=true , address=145.79.227.43:4161 , piece_id=ZN3T6JSPHJ57X4YYG5MHRJYYOFPH2T2CSGJ34MJQU55HDC7CMOXQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:44.123238386 (538h33m43.994321421s) [seg_pos={0 0}, upload=472313]: Segment reached optimal threshold of 14 pieces. Dec 28 02:02:44.123431754 (538h33m43.994514789s) [noise=true , address=145.79.227.33:4899 , piece_id=JFEXXK22EATV7UUH7LLXTFEPIUBW7EWSNYFXLE4AO5F6SLYFTGBA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b4pZVQRnjcQNn45VTgdsz5fpjYPMAzSXGR5KJ5BP9XHMkRnRw): protocol: expected piece hash; context canceled; EOF Dec 28 02:02:44.123437586 (538h33m43.994520621s) [noise=true , address=145.79.227.33:4899 , piece_id=JFEXXK22EATV7UUH7LLXTFEPIUBW7EWSNYFXLE4AO5F6SLYFTGBA, seg_pos={0 0}, upload=472313]: Long tail cancellation due to optimal threshold Dec 28 02:02:44.123631064 (538h33m43.994714109s) [noise=true , address=77.90.31.18:4833 , piece_id=3E57LWLI3WHROFGMXVZ7LHDHO5O7453LWQXA3OEKM52B2NIYM2JA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FbVbo8kGy1aaFyCk2noiXNUJHtosFJHFykBMRecGd5Kj5gjEp): protocol: expected piece hash; context canceled; EOF Dec 28 02:02:44.123675267 (538h33m43.994758313s) [noise=true , address=77.90.31.18:4833 , piece_id=3E57LWLI3WHROFGMXVZ7LHDHO5O7453LWQXA3OEKM52B2NIYM2JA, seg_pos={0 0}, upload=472313]: Long tail cancellation due to optimal threshold Dec 28 02:02:44.123774066 (538h33m43.994857101s) [noise=true , address=77.90.31.10:4524 , piece_id=X2VCBS4PTPPWI6U64T4FVIULRLCDY3XUOJTXJNP34OHSO54ONIWA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe): protocol: expected piece hash; context canceled; EOF Dec 28 02:02:44.123781761 (538h33m43.994864796s) [noise=true , address=77.90.31.10:4524 , piece_id=X2VCBS4PTPPWI6U64T4FVIULRLCDY3XUOJTXJNP34OHSO54ONIWA, seg_pos={0 0}, upload=472313]: Long tail cancellation due to optimal threshold Dec 28 02:02:44.123923341 (538h33m43.995006376s) [noise=true , address=145.79.227.35:4503 , piece_id=MDYJ2TROT4FHLX4XHKHEQACC7ZXHZCQ5MUPJL5EDS3F2YMXSUGUA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ctmSF1aMQasuXByFW8DppDantREmmWL2WUjmKGW5ZULVePyvm): protocol: expected piece hash; context canceled; EOF Dec 28 02:02:44.123928591 (538h33m43.995011636s) [noise=true , address=145.79.227.35:4503 , piece_id=MDYJ2TROT4FHLX4XHKHEQACC7ZXHZCQ5MUPJL5EDS3F2YMXSUGUA, seg_pos={0 0}, upload=472313]: Long tail cancellation due to optimal threshold Dec 28 02:02:44.123944731 (538h33m43.995027766s) [seg_pos={0 0}, upload=472313]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 02:02:44.124017179 (538h33m43.995100214s) [upload=472313]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:02:44.124022469 (538h33m43.995105504s) [upload=472313]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:02:44.409047274 (538h33m44.280130349s) [upload=472313]: Done uploading segments... Dec 28 02:02:44.409073354 (538h33m44.280156389s) [upload=472313]: Upload finished. err: <nil> Duration: 48.533879619s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............................................X............] duration: 4.066499265s trace: https://jaeger.dev.storj.io/trace/7fc142ad3d711e34 FAIL (2025-12-28 02:15:22 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.066535324s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 33 out of 60 is failed (threshold=3) [..X.XX.XXXXX..X.XXX.X.XX.XX.X..X.X.X.X.X..X...XX..X.XXX.XXX.] duration: 10.229819517s trace: https://jaeger.dev.storj.io/trace/441d78077f6ab18 FAIL (2025-12-28 02:15:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: V2S6ATZWTEIWJKI6UO6CSDA7IGBDYPMNIPP5ELFFVCEWRXEVJESA) context deadline exceeded; piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: KKGMUPP72RPAR37ENOVZYBNCFLA7HCKU4RPFVXRWJCWRS4A4OKVQ) context deadline exceeded ======LOGS====== 8660 (538h45m26.982611705s) [upload=472481]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 02:14:27.358929141 (538h45m27.230012177s) [seg_pos={0 0}, upload=472481]: Begin upload segment... Dec 28 02:14:27.358989497 (538h45m27.230072542s) [seg_pos={0 0}, upload=472481]: Done begin upload segment. Dec 28 02:14:27.359025255 (538h45m27.23010829s) [noise=true , address=145.79.227.5:4221 , piece_id=JLMITZ4MI3UXHLUQOPO7MIFQ3KSKFGO3S7A37OEBVAMMFQ6CY2WQ, seg_pos={0 0}, upload=472481]: Uploading piece... Dec 28 02:14:27.359068968 (538h45m27.230152003s) [noise=true , address=145.79.227.45:4231 , piece_id=HL5CQ2X6UVOHN4YEJGETJNZNMLYFDMKYNI3KMYS2D6L4ZE5QQDLA, seg_pos={0 0}, upload=472481]: Uploading piece... Dec 28 02:14:27.359899970 (538h45m27.230983015s) [noise=true , address=145.79.227.42:4121 , piece_id=O5HVHGMHOBCBWUT4F3XC5N27MG6S3X35TGRG4WFGFJ47RRKB7KQQ, seg_pos={0 0}, upload=472481]: Uploading piece... Dec 28 02:14:27.360433927 (538h45m27.231516972s) [noise=true , address=145.79.227.41:4086 , piece_id=VJY2OIJT5URPIC7NSVUUUFRBUUGA6NU2HAGEP4DW22Z2VBC67NVA, seg_pos={0 0}, upload=472481]: Uploading piece... Dec 28 02:14:27.360874366 (538h45m27.231957411s) [noise=true , address=145.79.227.12:4767 , piece_id=CBNTXRMGFN2HXAHPINMNEXTSDYPJZWV32A4ZF3C5BREC6UVKBDYA, seg_pos={0 0}, upload=472481]: Uploading piece... Dec 28 02:14:27.361315806 (538h45m27.232398851s) [noise=true , address=145.79.227.15:4874 , piece_id=CB4HEHSJK43BGCZLJ7QYJNEFUTM4JKRPSBF5SMTP2NZBXQKKZJGA, seg_pos={0 0}, upload=472481]: Uploading piece... Dec 28 02:14:27.361728001 (538h45m27.232811036s) [noise=true , address=145.79.227.48:4301 , piece_id=L4NDN2EGLXDIDNBXNX3EHD7ZQHC3FYVBZWNVUXJBYHT75H7XKGFQ, seg_pos={0 0}, upload=472481]: Uploading piece... Dec 28 02:14:27.362140446 (538h45m27.233223481s) [noise=true , address=45.91.116.7:4012 , piece_id=J3TKPTSNVJSF2DRAU7C2RZP67FGCANXQPH7AFA2R4ET3C32RYQSQ, seg_pos={0 0}, upload=472481]: Uploading piece... Dec 28 02:14:27.362216040 (538h45m27.233299085s) [noise=true , address=145.79.227.51:4410 , piece_id=PMDUD23X5YW6JAQP5WQTFQOZNIKO5AAGCLHBIZ7OOMQV3K67F2OA, seg_pos={0 0}, upload=472481]: Uploading piece... Dec 28 02:14:27.362651019 (538h45m27.233734054s) [noise=true , address=45.91.116.5:4298 , piece_id=V2S6ATZWTEIWJKI6UO6CSDA7IGBDYPMNIPP5ELFFVCEWRXEVJESA, seg_pos={0 0}, upload=472481]: Uploading piece... Dec 28 02:14:27.363158464 (538h45m27.234241509s) [noise=true , address=145.79.227.14:4840 , piece_id=CSUDVQWXZRUZZYN7F24M6NN64YZ6KRKI34LG2C7KRRXYR2UGNN7Q, seg_pos={0 0}, upload=472481]: Uploading piece... Dec 28 02:14:27.366054309 (538h45m27.237137344s) [noise=true , address=77.90.31.16:4735 , piece_id=KKGMUPP72RPAR37ENOVZYBNCFLA7HCKU4RPFVXRWJCWRS4A4OKVQ, seg_pos={0 0}, upload=472481]: Uploading piece... Dec 28 02:14:27.368328169 (538h45m27.239411214s) [noise=true , address=45.91.116.9:4264 , piece_id=ZEI7O4DIPJTWQLMDRJ3KDBA5FF7NDEM54JKUYA3Q56FZLUDTY5GQ, seg_pos={0 0}, upload=472481]: Uploading piece... Dec 28 02:14:27.370824743 (538h45m27.241907788s) [noise=true , address=77.90.31.21:4880 , piece_id=HQAEHPAKRIA3SDFAHSMIAQ7RQTRC7KBTSVDKIWAUEXS3FE6IZC2A, seg_pos={0 0}, upload=472481]: Uploading piece... Dec 28 02:14:27.372898772 (538h45m27.243981817s) [noise=true , address=145.79.227.31:4797 , piece_id=U5FGOR6NHM2NEKDLUJVLN6QLFNF23EYGBJ6OAYFUQFGWRQSTETIQ, seg_pos={0 0}, upload=472481]: Uploading piece... Dec 28 02:14:27.374981598 (538h45m27.246064643s) [noise=true , address=45.91.116.4:4939 , piece_id=O7TSZ4ITUR2EBWAVNHUUBRK3EOYM5LFIKK3WPRX32236MGT7634Q, seg_pos={0 0}, upload=472481]: Uploading piece... Dec 28 02:14:28.969586132 (538h45m28.840669167s) [upload=472481]: Next returned nil segment Dec 28 02:14:28.969600158 (538h45m28.840683193s) [upload=472481]: Waiting for error group managing segments... Dec 28 02:14:29.361916665 (538h45m29.23299971s) [noise=true , address=145.79.227.14:4840 , piece_id=CSUDVQWXZRUZZYN7F24M6NN64YZ6KRKI34LG2C7KRRXYR2UGNN7Q, seg_pos={0 0}, upload=472481]: Done uploading piece. err: <nil> Dec 28 02:14:29.365373437 (538h45m29.236456482s) [noise=true , address=145.79.227.5:4221 , piece_id=JLMITZ4MI3UXHLUQOPO7MIFQ3KSKFGO3S7A37OEBVAMMFQ6CY2WQ, seg_pos={0 0}, upload=472481]: Done uploading piece. err: <nil> Dec 28 02:14:29.375590472 (538h45m29.246673507s) [noise=true , address=145.79.227.51:4410 , piece_id=PMDUD23X5YW6JAQP5WQTFQOZNIKO5AAGCLHBIZ7OOMQV3K67F2OA, seg_pos={0 0}, upload=472481]: Done uploading piece. err: <nil> Dec 28 02:14:29.376279013 (538h45m29.247362048s) [noise=true , address=145.79.227.15:4874 , piece_id=CB4HEHSJK43BGCZLJ7QYJNEFUTM4JKRPSBF5SMTP2NZBXQKKZJGA, seg_pos={0 0}, upload=472481]: Done uploading piece. err: <nil> Dec 28 02:14:29.385230919 (538h45m29.256313954s) [noise=true , address=145.79.227.31:4797 , piece_id=U5FGOR6NHM2NEKDLUJVLN6QLFNF23EYGBJ6OAYFUQFGWRQSTETIQ, seg_pos={0 0}, upload=472481]: Done uploading piece. err: <nil> Dec 28 02:14:29.386211887 (538h45m29.257294922s) [noise=true , address=145.79.227.12:4767 , piece_id=CBNTXRMGFN2HXAHPINMNEXTSDYPJZWV32A4ZF3C5BREC6UVKBDYA, seg_pos={0 0}, upload=472481]: Done uploading piece. err: <nil> Dec 28 02:14:29.399698237 (538h45m29.270781273s) [noise=true , address=145.79.227.45:4231 , piece_id=HL5CQ2X6UVOHN4YEJGETJNZNMLYFDMKYNI3KMYS2D6L4ZE5QQDLA, seg_pos={0 0}, upload=472481]: Done uploading piece. err: <nil> Dec 28 02:14:29.629203533 (538h45m29.500286568s) [noise=true , address=45.91.116.5:4298 , piece_id=V2S6ATZWTEIWJKI6UO6CSDA7IGBDYPMNIPP5ELFFVCEWRXEVJESA, seg_pos={0 0}, upload=472481]: Done uploading piece. err: <nil> Dec 28 02:14:29.729121173 (538h45m29.600204208s) [noise=true , address=77.90.31.21:4880 , piece_id=HQAEHPAKRIA3SDFAHSMIAQ7RQTRC7KBTSVDKIWAUEXS3FE6IZC2A, seg_pos={0 0}, upload=472481]: Done uploading piece. err: <nil> Dec 28 02:14:30.386487558 (538h45m30.257570604s) [noise=true , address=77.90.31.16:4735 , piece_id=KKGMUPP72RPAR37ENOVZYBNCFLA7HCKU4RPFVXRWJCWRS4A4OKVQ, seg_pos={0 0}, upload=472481]: Done uploading piece. err: <nil> Dec 28 02:14:30.403094771 (538h45m30.274177806s) [noise=true , address=45.91.116.9:4264 , piece_id=ZEI7O4DIPJTWQLMDRJ3KDBA5FF7NDEM54JKUYA3Q56FZLUDTY5GQ, seg_pos={0 0}, upload=472481]: Done uploading piece. err: <nil> Dec 28 02:14:30.478364207 (538h45m30.349447242s) [noise=true , address=145.79.227.42:4121 , piece_id=O5HVHGMHOBCBWUT4F3XC5N27MG6S3X35TGRG4WFGFJ47RRKB7KQQ, seg_pos={0 0}, upload=472481]: Done uploading piece. err: <nil> Dec 28 02:14:30.482774816 (538h45m30.353857851s) [noise=true , address=145.79.227.48:4301 , piece_id=L4NDN2EGLXDIDNBXNX3EHD7ZQHC3FYVBZWNVUXJBYHT75H7XKGFQ, seg_pos={0 0}, upload=472481]: Done uploading piece. err: <nil> Dec 28 02:14:30.482787230 (538h45m30.353870265s) [seg_pos={0 0}, upload=472481]: Segment reached optimal threshold of 13 pieces. Dec 28 02:14:30.482954809 (538h45m30.354037854s) [noise=true , address=45.91.116.7:4012 , piece_id=J3TKPTSNVJSF2DRAU7C2RZP67FGCANXQPH7AFA2R4ET3C32RYQSQ, seg_pos={0 0}, upload=472481]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: write tcp 188.166.225.170:34034->45.91.116.7:4012: use of closed network connection; write tcp 188.166.225.170:34034->45.91.116.7:4012: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34034->45.91.116.7:4012: use of closed network connection Dec 28 02:14:30.482961151 (538h45m30.354044187s) [noise=true , address=45.91.116.7:4012 , piece_id=J3TKPTSNVJSF2DRAU7C2RZP67FGCANXQPH7AFA2R4ET3C32RYQSQ, seg_pos={0 0}, upload=472481]: Long tail cancellation due to optimal threshold Dec 28 02:14:30.483043949 (538h45m30.354126984s) [noise=true , address=45.91.116.4:4939 , piece_id=O7TSZ4ITUR2EBWAVNHUUBRK3EOYM5LFIKK3WPRX32236MGT7634Q, seg_pos={0 0}, upload=472481]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: write tcp 188.166.225.170:51982->45.91.116.4:4939: use of closed network connection; write tcp 188.166.225.170:51982->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51982->45.91.116.4:4939: use of closed network connection Dec 28 02:14:30.483051593 (538h45m30.354134638s) [noise=true , address=45.91.116.4:4939 , piece_id=O7TSZ4ITUR2EBWAVNHUUBRK3EOYM5LFIKK3WPRX32236MGT7634Q, seg_pos={0 0}, upload=472481]: Long tail cancellation due to optimal threshold Dec 28 02:14:30.483058576 (538h45m30.354141611s) [noise=true , address=145.79.227.41:4086 , piece_id=VJY2OIJT5URPIC7NSVUUUFRBUUGA6NU2HAGEP4DW22Z2VBC67NVA, seg_pos={0 0}, upload=472481]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: write tcp 188.166.225.170:60228->145.79.227.41:4086: use of closed network connection; write tcp 188.166.225.170:60228->145.79.227.41:4086: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60228->145.79.227.41:4086: use of closed network connection Dec 28 02:14:30.483064457 (538h45m30.354147492s) [noise=true , address=145.79.227.41:4086 , piece_id=VJY2OIJT5URPIC7NSVUUUFRBUUGA6NU2HAGEP4DW22Z2VBC67NVA, seg_pos={0 0}, upload=472481]: Long tail cancellation due to optimal threshold Dec 28 02:14:30.483079436 (538h45m30.354162481s) [seg_pos={0 0}, upload=472481]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 02:14:30.483138299 (538h45m30.354221334s) [upload=472481]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:14:30.483142446 (538h45m30.354225482s) [upload=472481]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:14:30.776656105 (538h45m30.647739151s) [upload=472481]: Done uploading segments... Dec 28 02:14:30.776680763 (538h45m30.647763798s) [upload=472481]: Upload finished. err: <nil> Duration:* 10.229838061s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................................................X...........] duration: 3.50324359s trace: https://jaeger.dev.storj.io/trace/7fc142ad3d711e34 FAIL (2025-12-28 02:15:00 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.503273198s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 33 out of 60 is failed (threshold=3) [...X.XX.XXXXX..X.XXX.X.XX.XX.X..X.X.X.X.X..X...XX..X.XXX.XXX] duration: 30.003985917s trace: https://jaeger.dev.storj.io/trace/441d78077f6ab18 FAIL (2025-12-28 02:14:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: V2S6ATZWTEIWJKI6UO6CSDA7IGBDYPMNIPP5ELFFVCEWRXEVJESA) context deadline exceeded; piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: KKGMUPP72RPAR37ENOVZYBNCFLA7HCKU4RPFVXRWJCWRS4A4OKVQ) context deadline exceeded ======LOGS====== 8660 (538h45m26.982611705s) [upload=472481]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 02:14:27.358929141 (538h45m27.230012177s) [seg_pos={0 0}, upload=472481]: Begin upload segment... Dec 28 02:14:27.358989497 (538h45m27.230072542s) [seg_pos={0 0}, upload=472481]: Done begin upload segment. Dec 28 02:14:27.359025255 (538h45m27.23010829s) [noise=true , address=145.79.227.5:4221 , piece_id=JLMITZ4MI3UXHLUQOPO7MIFQ3KSKFGO3S7A37OEBVAMMFQ6CY2WQ, seg_pos={0 0}, upload=472481]: Uploading piece... Dec 28 02:14:27.359068968 (538h45m27.230152003s) [noise=true , address=145.79.227.45:4231 , piece_id=HL5CQ2X6UVOHN4YEJGETJNZNMLYFDMKYNI3KMYS2D6L4ZE5QQDLA, seg_pos={0 0}, upload=472481]: Uploading piece... Dec 28 02:14:27.359899970 (538h45m27.230983015s) [noise=true , address=145.79.227.42:4121 , piece_id=O5HVHGMHOBCBWUT4F3XC5N27MG6S3X35TGRG4WFGFJ47RRKB7KQQ, seg_pos={0 0}, upload=472481]: Uploading piece... Dec 28 02:14:27.360433927 (538h45m27.231516972s) [noise=true , address=145.79.227.41:4086 , piece_id=VJY2OIJT5URPIC7NSVUUUFRBUUGA6NU2HAGEP4DW22Z2VBC67NVA, seg_pos={0 0}, upload=472481]: Uploading piece... Dec 28 02:14:27.360874366 (538h45m27.231957411s) [noise=true , address=145.79.227.12:4767 , piece_id=CBNTXRMGFN2HXAHPINMNEXTSDYPJZWV32A4ZF3C5BREC6UVKBDYA, seg_pos={0 0}, upload=472481]: Uploading piece... Dec 28 02:14:27.361315806 (538h45m27.232398851s) [noise=true , address=145.79.227.15:4874 , piece_id=CB4HEHSJK43BGCZLJ7QYJNEFUTM4JKRPSBF5SMTP2NZBXQKKZJGA, seg_pos={0 0}, upload=472481]: Uploading piece... Dec 28 02:14:27.361728001 (538h45m27.232811036s) [noise=true , address=145.79.227.48:4301 , piece_id=L4NDN2EGLXDIDNBXNX3EHD7ZQHC3FYVBZWNVUXJBYHT75H7XKGFQ, seg_pos={0 0}, upload=472481]: Uploading piece... Dec 28 02:14:27.362140446 (538h45m27.233223481s) [noise=true , address=45.91.116.7:4012 , piece_id=J3TKPTSNVJSF2DRAU7C2RZP67FGCANXQPH7AFA2R4ET3C32RYQSQ, seg_pos={0 0}, upload=472481]: Uploading piece... Dec 28 02:14:27.362216040 (538h45m27.233299085s) [noise=true , address=145.79.227.51:4410 , piece_id=PMDUD23X5YW6JAQP5WQTFQOZNIKO5AAGCLHBIZ7OOMQV3K67F2OA, seg_pos={0 0}, upload=472481]: Uploading piece... Dec 28 02:14:27.362651019 (538h45m27.233734054s) [noise=true , address=45.91.116.5:4298 , piece_id=V2S6ATZWTEIWJKI6UO6CSDA7IGBDYPMNIPP5ELFFVCEWRXEVJESA, seg_pos={0 0}, upload=472481]: Uploading piece... Dec 28 02:14:27.363158464 (538h45m27.234241509s) [noise=true , address=145.79.227.14:4840 , piece_id=CSUDVQWXZRUZZYN7F24M6NN64YZ6KRKI34LG2C7KRRXYR2UGNN7Q, seg_pos={0 0}, upload=472481]: Uploading piece... Dec 28 02:14:27.366054309 (538h45m27.237137344s) [noise=true , address=77.90.31.16:4735 , piece_id=KKGMUPP72RPAR37ENOVZYBNCFLA7HCKU4RPFVXRWJCWRS4A4OKVQ, seg_pos={0 0}, upload=472481]: Uploading piece... Dec 28 02:14:27.368328169 (538h45m27.239411214s) [noise=true , address=45.91.116.9:4264 , piece_id=ZEI7O4DIPJTWQLMDRJ3KDBA5FF7NDEM54JKUYA3Q56FZLUDTY5GQ, seg_pos={0 0}, upload=472481]: Uploading piece... Dec 28 02:14:27.370824743 (538h45m27.241907788s) [noise=true , address=77.90.31.21:4880 , piece_id=HQAEHPAKRIA3SDFAHSMIAQ7RQTRC7KBTSVDKIWAUEXS3FE6IZC2A, seg_pos={0 0}, upload=472481]: Uploading piece... Dec 28 02:14:27.372898772 (538h45m27.243981817s) [noise=true , address=145.79.227.31:4797 , piece_id=U5FGOR6NHM2NEKDLUJVLN6QLFNF23EYGBJ6OAYFUQFGWRQSTETIQ, seg_pos={0 0}, upload=472481]: Uploading piece... Dec 28 02:14:27.374981598 (538h45m27.246064643s) [noise=true , address=45.91.116.4:4939 , piece_id=O7TSZ4ITUR2EBWAVNHUUBRK3EOYM5LFIKK3WPRX32236MGT7634Q, seg_pos={0 0}, upload=472481]: Uploading piece... Dec 28 02:14:28.969586132 (538h45m28.840669167s) [upload=472481]: Next returned nil segment Dec 28 02:14:28.969600158 (538h45m28.840683193s) [upload=472481]: Waiting for error group managing segments... Dec 28 02:14:29.361916665 (538h45m29.23299971s) [noise=true , address=145.79.227.14:4840 , piece_id=CSUDVQWXZRUZZYN7F24M6NN64YZ6KRKI34LG2C7KRRXYR2UGNN7Q, seg_pos={0 0}, upload=472481]: Done uploading piece. err: <nil> Dec 28 02:14:29.365373437 (538h45m29.236456482s) [noise=true , address=145.79.227.5:4221 , piece_id=JLMITZ4MI3UXHLUQOPO7MIFQ3KSKFGO3S7A37OEBVAMMFQ6CY2WQ, seg_pos={0 0}, upload=472481]: Done uploading piece. err: <nil> Dec 28 02:14:29.375590472 (538h45m29.246673507s) [noise=true , address=145.79.227.51:4410 , piece_id=PMDUD23X5YW6JAQP5WQTFQOZNIKO5AAGCLHBIZ7OOMQV3K67F2OA, seg_pos={0 0}, upload=472481]: Done uploading piece. err: <nil> Dec 28 02:14:29.376279013 (538h45m29.247362048s) [noise=true , address=145.79.227.15:4874 , piece_id=CB4HEHSJK43BGCZLJ7QYJNEFUTM4JKRPSBF5SMTP2NZBXQKKZJGA, seg_pos={0 0}, upload=472481]: Done uploading piece. err: <nil> Dec 28 02:14:29.385230919 (538h45m29.256313954s) [noise=true , address=145.79.227.31:4797 , piece_id=U5FGOR6NHM2NEKDLUJVLN6QLFNF23EYGBJ6OAYFUQFGWRQSTETIQ, seg_pos={0 0}, upload=472481]: Done uploading piece. err: <nil> Dec 28 02:14:29.386211887 (538h45m29.257294922s) [noise=true , address=145.79.227.12:4767 , piece_id=CBNTXRMGFN2HXAHPINMNEXTSDYPJZWV32A4ZF3C5BREC6UVKBDYA, seg_pos={0 0}, upload=472481]: Done uploading piece. err: <nil> Dec 28 02:14:29.399698237 (538h45m29.270781273s) [noise=true , address=145.79.227.45:4231 , piece_id=HL5CQ2X6UVOHN4YEJGETJNZNMLYFDMKYNI3KMYS2D6L4ZE5QQDLA, seg_pos={0 0}, upload=472481]: Done uploading piece. err: <nil> Dec 28 02:14:29.629203533 (538h45m29.500286568s) [noise=true , address=45.91.116.5:4298 , piece_id=V2S6ATZWTEIWJKI6UO6CSDA7IGBDYPMNIPP5ELFFVCEWRXEVJESA, seg_pos={0 0}, upload=472481]: Done uploading piece. err: <nil> Dec 28 02:14:29.729121173 (538h45m29.600204208s) [noise=true , address=77.90.31.21:4880 , piece_id=HQAEHPAKRIA3SDFAHSMIAQ7RQTRC7KBTSVDKIWAUEXS3FE6IZC2A, seg_pos={0 0}, upload=472481]: Done uploading piece. err: <nil> Dec 28 02:14:30.386487558 (538h45m30.257570604s) [noise=true , address=77.90.31.16:4735 , piece_id=KKGMUPP72RPAR37ENOVZYBNCFLA7HCKU4RPFVXRWJCWRS4A4OKVQ, seg_pos={0 0}, upload=472481]: Done uploading piece. err: <nil> Dec 28 02:14:30.403094771 (538h45m30.274177806s) [noise=true , address=45.91.116.9:4264 , piece_id=ZEI7O4DIPJTWQLMDRJ3KDBA5FF7NDEM54JKUYA3Q56FZLUDTY5GQ, seg_pos={0 0}, upload=472481]: Done uploading piece. err: <nil> Dec 28 02:14:30.478364207 (538h45m30.349447242s) [noise=true , address=145.79.227.42:4121 , piece_id=O5HVHGMHOBCBWUT4F3XC5N27MG6S3X35TGRG4WFGFJ47RRKB7KQQ, seg_pos={0 0}, upload=472481]: Done uploading piece. err: <nil> Dec 28 02:14:30.482774816 (538h45m30.353857851s) [noise=true , address=145.79.227.48:4301 , piece_id=L4NDN2EGLXDIDNBXNX3EHD7ZQHC3FYVBZWNVUXJBYHT75H7XKGFQ, seg_pos={0 0}, upload=472481]: Done uploading piece. err: <nil> Dec 28 02:14:30.482787230 (538h45m30.353870265s) [seg_pos={0 0}, upload=472481]: Segment reached optimal threshold of 13 pieces. Dec 28 02:14:30.482954809 (538h45m30.354037854s) [noise=true , address=45.91.116.7:4012 , piece_id=J3TKPTSNVJSF2DRAU7C2RZP67FGCANXQPH7AFA2R4ET3C32RYQSQ, seg_pos={0 0}, upload=472481]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: write tcp 188.166.225.170:34034->45.91.116.7:4012: use of closed network connection; write tcp 188.166.225.170:34034->45.91.116.7:4012: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34034->45.91.116.7:4012: use of closed network connection Dec 28 02:14:30.482961151 (538h45m30.354044187s) [noise=true , address=45.91.116.7:4012 , piece_id=J3TKPTSNVJSF2DRAU7C2RZP67FGCANXQPH7AFA2R4ET3C32RYQSQ, seg_pos={0 0}, upload=472481]: Long tail cancellation due to optimal threshold Dec 28 02:14:30.483043949 (538h45m30.354126984s) [noise=true , address=45.91.116.4:4939 , piece_id=O7TSZ4ITUR2EBWAVNHUUBRK3EOYM5LFIKK3WPRX32236MGT7634Q, seg_pos={0 0}, upload=472481]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: write tcp 188.166.225.170:51982->45.91.116.4:4939: use of closed network connection; write tcp 188.166.225.170:51982->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51982->45.91.116.4:4939: use of closed network connection Dec 28 02:14:30.483051593 (538h45m30.354134638s) [noise=true , address=45.91.116.4:4939 , piece_id=O7TSZ4ITUR2EBWAVNHUUBRK3EOYM5LFIKK3WPRX32236MGT7634Q, seg_pos={0 0}, upload=472481]: Long tail cancellation due to optimal threshold Dec 28 02:14:30.483058576 (538h45m30.354141611s) [noise=true , address=145.79.227.41:4086 , piece_id=VJY2OIJT5URPIC7NSVUUUFRBUUGA6NU2HAGEP4DW22Z2VBC67NVA, seg_pos={0 0}, upload=472481]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: write tcp 188.166.225.170:60228->145.79.227.41:4086: use of closed network connection; write tcp 188.166.225.170:60228->145.79.227.41:4086: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60228->145.79.227.41:4086: use of closed network connection Dec 28 02:14:30.483064457 (538h45m30.354147492s) [noise=true , address=145.79.227.41:4086 , piece_id=VJY2OIJT5URPIC7NSVUUUFRBUUGA6NU2HAGEP4DW22Z2VBC67NVA, seg_pos={0 0}, upload=472481]: Long tail cancellation due to optimal threshold Dec 28 02:14:30.483079436 (538h45m30.354162481s) [seg_pos={0 0}, upload=472481]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 02:14:30.483138299 (538h45m30.354221334s) [upload=472481]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:14:30.483142446 (538h45m30.354225482s) [upload=472481]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:14:30.776656105 (538h45m30.647739151s) [upload=472481]: Done uploading segments... Dec 28 02:14:30.776680763 (538h45m30.647763798s) [upload=472481]: Upload finished. err: <nil> Duration:* 30.004021185s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 46.008343397s trace: https://jaeger.dev.storj.io/trace/12a1d059b6f75af7 FAIL (2025-12-28 02:14:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1.145865281 (538h33m41.016948316s) [noise=true , address=77.90.31.10:4551 , piece_id=KBPO4MUDASHKMPRNJELQWLNMJNQBB3PZUOVNCEPP2CYVHU6X2PPA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.146408725 (538h33m41.01749177s) [noise=true , address=145.79.227.15:4908 , piece_id=I2FSDNVYBFZKRXZNUHH2APY7N2PI7DPAKFKGXRJW2UFU2PBKSSNQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.146778139 (538h33m41.017861184s) [noise=true , address=145.79.227.43:4161 , piece_id=ZN3T6JSPHJ57X4YYG5MHRJYYOFPH2T2CSGJ34MJQU55HDC7CMOXQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.147137874 (538h33m41.018220919s) [noise=true , address=77.90.31.21:4882 , piece_id=WQAGQTGLU5HRKEET36W34MQP4NPGSVCA4QZ6YEGVAJWBBPLZOUQQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.147426784 (538h33m41.018509819s) [noise=true , address=145.79.227.35:4503 , piece_id=MDYJ2TROT4FHLX4XHKHEQACC7ZXHZCQ5MUPJL5EDS3F2YMXSUGUA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.147466109 (538h33m41.018549154s) [noise=true , address=77.90.31.15:4730 , piece_id=EN7IGRJZ2WFN4HXJPMNLLXDUZPIN7KDUGKPPLJH7CSA4SDOUM2JA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.147817368 (538h33m41.018900413s) [noise=true , address=77.90.31.18:4833 , piece_id=3E57LWLI3WHROFGMXVZ7LHDHO5O7453LWQXA3OEKM52B2NIYM2JA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.147860690 (538h33m41.018943745s) [noise=true , address=145.79.227.44:4212 , piece_id=EFTTQM6323MUQ5WTGY6DFBTF3TGRM6XZWXIGT6DM4VS7GRGM4C2Q, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148163427 (538h33m41.019246472s) [noise=true , address=77.90.31.19:4862 , piece_id=PSBBBSGY7THVLMXJQCADIE7VYZ33OIIDVAEQWG3RWMNA6RSKELNQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148189276 (538h33m41.019272321s) [noise=true , address=77.90.31.10:4524 , piece_id=X2VCBS4PTPPWI6U64T4FVIULRLCDY3XUOJTXJNP34OHSO54ONIWA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148496531 (538h33m41.019579566s) [noise=true , address=145.79.227.23:4556 , piece_id=AKBXRSU5PM5QLHGHE3VXTJ3ZH23BECPGBLQSVZK6XVIYJGVDXT4Q, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148527380 (538h33m41.019610425s) [noise=true , address=145.79.227.33:4899 , piece_id=JFEXXK22EATV7UUH7LLXTFEPIUBW7EWSNYFXLE4AO5F6SLYFTGBA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148801512 (538h33m41.019884547s) [noise=true , address=77.90.31.14:4672 , piece_id=BZG2Y5X47J2WVFF5LBHEJZNUD2OQ3NOTVPTCAVT4TLFQB5FWESLA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148894920 (538h33m41.019977965s) [noise=true , address=5.231.73.5:4016 , piece_id=YCNKQEKJQSJHKKGCYWLOLEBKIVPJHZIJ6TYEG6DF4TRXMZ3ZTTIQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.149118806 (538h33m41.020201851s) [noise=true , address=145.79.227.47:4295 , piece_id=OM7B75MCPM7L3WP5MLQ7MAVBZ2NJUSISOWD56RSWI2AV2OWVDRTA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.149274993 (538h33m41.020358038s) [noise=true , address=145.79.227.50:4400 , piece_id=HZAT7YAVZYWK52FE4TIC6OA37U4VX5B35H55WMBO3MMWNIKTJLJQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.149625842 (538h33m41.020708877s) [noise=true , address=145.79.227.34:4915 , piece_id=RMXYZGGJA5EPJLQD6SQG677HFLOQWRY2HFTGEOICYAQG6QR557RQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:42.618098549 (538h33m42.489181584s) [upload=472313]: Next returned nil segment Dec 28 02:02:42.618120911 (538h33m42.489203946s) [upload=472313]: Waiting for error group managing segments... Dec 28 02:02:43.021704832 (538h33m42.892787877s) [noise=true , address=145.79.227.23:4556 , piece_id=AKBXRSU5PM5QLHGHE3VXTJ3ZH23BECPGBLQSVZK6XVIYJGVDXT4Q, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.027980492 (538h33m42.899063537s) [noise=true , address=145.79.227.50:4400 , piece_id=HZAT7YAVZYWK52FE4TIC6OA37U4VX5B35H55WMBO3MMWNIKTJLJQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.043329239 (538h33m42.914412284s) [noise=true , address=145.79.227.15:4908 , piece_id=I2FSDNVYBFZKRXZNUHH2APY7N2PI7DPAKFKGXRJW2UFU2PBKSSNQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.056308314 (538h33m42.927391349s) [noise=true , address=145.79.227.49:4347 , piece_id=WURIHGQASBLSVLZJVM5EJHWD7U4VFLLPGJIFYNOK6ALA5LYEQDSA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.074628196 (538h33m42.945711252s) [noise=true , address=145.79.227.47:4295 , piece_id=OM7B75MCPM7L3WP5MLQ7MAVBZ2NJUSISOWD56RSWI2AV2OWVDRTA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.080495740 (538h33m42.951578775s) [noise=true , address=5.231.73.5:4016 , piece_id=YCNKQEKJQSJHKKGCYWLOLEBKIVPJHZIJ6TYEG6DF4TRXMZ3ZTTIQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.193955176 (538h33m43.065038241s) [noise=true , address=77.90.31.10:4551 , piece_id=KBPO4MUDASHKMPRNJELQWLNMJNQBB3PZUOVNCEPP2CYVHU6X2PPA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.301513865 (538h33m43.1725969s) [noise=true , address=77.90.31.15:4730 , piece_id=EN7IGRJZ2WFN4HXJPMNLLXDUZPIN7KDUGKPPLJH7CSA4SDOUM2JA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.305020622 (538h33m43.176103657s) [noise=true , address=77.90.31.19:4862 , piece_id=PSBBBSGY7THVLMXJQCADIE7VYZ33OIIDVAEQWG3RWMNA6RSKELNQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.312187619 (538h33m43.183270654s) [noise=true , address=77.90.31.14:4672 , piece_id=BZG2Y5X47J2WVFF5LBHEJZNUD2OQ3NOTVPTCAVT4TLFQB5FWESLA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.319277390 (538h33m43.190360425s) [noise=true , address=77.90.31.21:4882 , piece_id=WQAGQTGLU5HRKEET36W34MQP4NPGSVCA4QZ6YEGVAJWBBPLZOUQQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.948916959 (538h33m43.820000004s) [noise=true , address=145.79.227.34:4915 , piece_id=RMXYZGGJA5EPJLQD6SQG677HFLOQWRY2HFTGEOICYAQG6QR557RQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.989823331 (538h33m43.860906366s) [noise=true , address=145.79.227.44:4212 , piece_id=EFTTQM6323MUQ5WTGY6DFBTF3TGRM6XZWXIGT6DM4VS7GRGM4C2Q, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:44.123218949 (538h33m43.994301984s) [noise=true , address=145.79.227.43:4161 , piece_id=ZN3T6JSPHJ57X4YYG5MHRJYYOFPH2T2CSGJ34MJQU55HDC7CMOXQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:44.123238386 (538h33m43.994321421s) [seg_pos={0 0}, upload=472313]: Segment reached optimal threshold of 14 pieces. Dec 28 02:02:44.123431754 (538h33m43.994514789s) [noise=true , address=145.79.227.33:4899 , piece_id=JFEXXK22EATV7UUH7LLXTFEPIUBW7EWSNYFXLE4AO5F6SLYFTGBA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b4pZVQRnjcQNn45VTgdsz5fpjYPMAzSXGR5KJ5BP9XHMkRnRw): protocol: expected piece hash; context canceled; EOF Dec 28 02:02:44.123437586 (538h33m43.994520621s) [noise=true , address=145.79.227.33:4899 , piece_id=JFEXXK22EATV7UUH7LLXTFEPIUBW7EWSNYFXLE4AO5F6SLYFTGBA, seg_pos={0 0}, upload=472313]: Long tail cancellation due to optimal threshold Dec 28 02:02:44.123631064 (538h33m43.994714109s) [noise=true , address=77.90.31.18:4833 , piece_id=3E57LWLI3WHROFGMXVZ7LHDHO5O7453LWQXA3OEKM52B2NIYM2JA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FbVbo8kGy1aaFyCk2noiXNUJHtosFJHFykBMRecGd5Kj5gjEp): protocol: expected piece hash; context canceled; EOF Dec 28 02:02:44.123675267 (538h33m43.994758313s) [noise=true , address=77.90.31.18:4833 , piece_id=3E57LWLI3WHROFGMXVZ7LHDHO5O7453LWQXA3OEKM52B2NIYM2JA, seg_pos={0 0}, upload=472313]: Long tail cancellation due to optimal threshold Dec 28 02:02:44.123774066 (538h33m43.994857101s) [noise=true , address=77.90.31.10:4524 , piece_id=X2VCBS4PTPPWI6U64T4FVIULRLCDY3XUOJTXJNP34OHSO54ONIWA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe): protocol: expected piece hash; context canceled; EOF Dec 28 02:02:44.123781761 (538h33m43.994864796s) [noise=true , address=77.90.31.10:4524 , piece_id=X2VCBS4PTPPWI6U64T4FVIULRLCDY3XUOJTXJNP34OHSO54ONIWA, seg_pos={0 0}, upload=472313]: Long tail cancellation due to optimal threshold Dec 28 02:02:44.123923341 (538h33m43.995006376s) [noise=true , address=145.79.227.35:4503 , piece_id=MDYJ2TROT4FHLX4XHKHEQACC7ZXHZCQ5MUPJL5EDS3F2YMXSUGUA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ctmSF1aMQasuXByFW8DppDantREmmWL2WUjmKGW5ZULVePyvm): protocol: expected piece hash; context canceled; EOF Dec 28 02:02:44.123928591 (538h33m43.995011636s) [noise=true , address=145.79.227.35:4503 , piece_id=MDYJ2TROT4FHLX4XHKHEQACC7ZXHZCQ5MUPJL5EDS3F2YMXSUGUA, seg_pos={0 0}, upload=472313]: Long tail cancellation due to optimal threshold Dec 28 02:02:44.123944731 (538h33m43.995027766s) [seg_pos={0 0}, upload=472313]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 02:02:44.124017179 (538h33m43.995100214s) [upload=472313]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:02:44.124022469 (538h33m43.995105504s) [upload=472313]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:02:44.409047274 (538h33m44.280130349s) [upload=472313]: Done uploading segments... Dec 28 02:02:44.409073354 (538h33m44.280156389s) [upload=472313]: Upload finished. err: <nil> Duration: 46.008361011s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................................................X..........] duration: 3.426898239s trace: https://jaeger.dev.storj.io/trace/7fc142ad3d711e34 FAIL (2025-12-28 02:14:15 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.427200353s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 32 out of 60 is failed (threshold=3) [....X.XX.XXXXX..X.XXX.X.XX.XX.X..X.X.X.X.X..X...XX..X.XXX.XX] duration: 30.002745089s trace: https://jaeger.dev.storj.io/trace/739884be0e06d45b FAIL (2025-12-28 02:13:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ec 28 02:13:16.951992719 (538h44m16.823075764s) [upload=472467]: Starting upload Dec 28 02:13:16.952061289 (538h44m16.823144324s) [upload=472467]: Done starting upload Dec 28 02:13:16.952088682 (538h44m16.823171717s) [upload=472467]: Uploading segments... Dec 28 02:13:16.952242795 (538h44m16.82332584s) [upload=472467]: Got next segment. Inline: false Dec 28 02:13:16.952268534 (538h44m16.823351569s) [upload=472467]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 02:13:16.952272602 (538h44m16.823355657s) [upload=472467]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 02:13:17.199713541 (538h44m17.070796586s) [seg_pos={0 0}, upload=472467]: Begin upload segment... Dec 28 02:13:17.199780078 (538h44m17.070863113s) [seg_pos={0 0}, upload=472467]: Done begin upload segment. Dec 28 02:13:17.199834632 (538h44m17.070917667s) [noise=true , address=145.79.227.31:4798 , piece_id=HOYKSTDDWQUWIUIQMJHKCDL66RFIBQXPGURFLQRH3MNZVV6ITOSQ, seg_pos={0 0}, upload=472467]: Uploading piece... Dec 28 02:13:17.199893664 (538h44m17.070976699s) [noise=true , address=145.79.227.19:4079 , piece_id=65XSP3UPFJHNGPWM7QCOSXN5ZM3A474RARZMQ65TAIVIFVJJS64A, seg_pos={0 0}, upload=472467]: Uploading piece... Dec 28 02:13:17.200381122 (538h44m17.071464157s) [noise=true , address=145.79.227.29:4943 , piece_id=6M3NIGPQQPVJD4VVAYSYYU2BXP45R63W43I3E5K36VCS3QI7ECQQ, seg_pos={0 0}, upload=472467]: Uploading piece... Dec 28 02:13:17.200685602 (538h44m17.071768637s) [noise=true , address=145.79.227.5:4222 , piece_id=NLIOB2PCTLM5UNIABNNOO3FBHQKAFMS74CHWX3S525FRDWOFRYJQ, seg_pos={0 0}, upload=472467]: Uploading piece... Dec 28 02:13:17.200982538 (538h44m17.072065573s) [noise=true , address=145.79.227.45:4230 , piece_id=HT3OCVNTU4RQZQZNFIG5SG4BCS6U4CF4MFOVMSTII2UMDQY3ZUSA, seg_pos={0 0}, upload=472467]: Uploading piece... Dec 28 02:13:17.201312005 (538h44m17.07239505s) [noise=true , address=145.79.227.47:4267 , piece_id=HKOTMPNDD4CQOU2JWXNEXT4FZESGQ6VPRZQAV75XBHGSF3DXKJ7A, seg_pos={0 0}, upload=472467]: Uploading piece... Dec 28 02:13:17.201320741 (538h44m17.072403786s) [noise=true , address=145.79.227.51:4410 , piece_id=6SVJ44QTOJH2YKYYJKPYZURGJUSGWD2F2RGUIIEZQH2GH2YS2T2A, seg_pos={0 0}, upload=472467]: Uploading piece... Dec 28 02:13:17.201605314 (538h44m17.072688359s) [noise=true , address=145.79.227.36:4519 , piece_id=JUCQAE6U2VOYAXDVRON5T2PJKIXGVVXKKWLXVJLUAK2AOIECVM5Q, seg_pos={0 0}, upload=472467]: Uploading piece... Dec 28 02:13:17.201754247 (538h44m17.072837292s) [noise=true , address=45.91.116.9:4264 , piece_id=SK5DXIK3BHW7QYBK72QPTDBIIICKSV5NKKRFOGTMNQQ2RQ4YCH3Q, seg_pos={0 0}, upload=472467]: Uploading piece... Dec 28 02:13:17.201937416 (538h44m17.073020451s) [noise=true , address=45.91.116.8:4045 , piece_id=VDXKZ5WLZKKF2GFTMMQYOL7BD22XO357PF2UV2XAL27CKKJ5MNJQ, seg_pos={0 0}, upload=472467]: Uploading piece... Dec 28 02:13:17.202239060 (538h44m17.073322095s) [noise=true , address=145.79.227.48:4302 , piece_id=ELXMRPLA6SUP5YX3W6FY2P4HFBZAGPHMD7H64UU7PMBEZ3N5DUHQ, seg_pos={0 0}, upload=472467]: Uploading piece... Dec 28 02:13:17.202248768 (538h44m17.073331814s) [noise=true , address=45.91.116.10:4084 , piece_id=RRVJATQXQSZC66YGEPTV6ICXQ5C3PXPA4XH3IB5TIPRHUHNKLQYA, seg_pos={0 0}, upload=472467]: Uploading piece... Dec 28 02:13:17.202672575 (538h44m17.07375562s) [noise=true , address=77.90.31.10:4517 , piece_id=IGX47ZVM5MH4DPH5T5TQFHMFC5OKOTAF2TCIZCUF3S5AUZOBQFXQ, seg_pos={0 0}, upload=472467]: Uploading piece... Dec 28 02:13:17.203138442 (538h44m17.074221487s) [noise=true , address=45.91.116.13:4153 , piece_id=OIYGJXOYYHUVH5M6T57PVR6GUAYYAGCBIPIHZDB77KDIDQU6IA4A, seg_pos={0 0}, upload=472467]: Uploading piece... Dec 28 02:13:17.203567169 (538h44m17.074650214s) [noise=true , address=45.91.116.5:4299 , piece_id=RWDLMPVHCOUENEQHYQ3OIPJD7B4FOCLTKM77YQJRNZNRWMEFMDCA, seg_pos={0 0}, upload=472467]: Uploading piece... Dec 28 02:13:17.203905523 (538h44m17.074988558s) [noise=true , address=145.79.227.46:4603 , piece_id=YYPMVAQFAJX2UX2PVAOHCNLAM3XBRVLSX64O6TKFFMCFDKEJBMHA, seg_pos={0 0}, upload=472467]: Uploading piece... Dec 28 02:13:18.966204545 (538h44m18.83728759s) [upload=472467]: Next returned nil segment Dec 28 02:13:18.966222730 (538h44m18.837305765s) [upload=472467]: Waiting for error group managing segments... Dec 28 02:13:19.347488654 (538h44m19.218571699s) [noise=true , address=145.79.227.51:4410 , piece_id=6SVJ44QTOJH2YKYYJKPYZURGJUSGWD2F2RGUIIEZQH2GH2YS2T2A, seg_pos={0 0}, upload=472467]: Done uploading piece. err: <nil> Dec 28 02:13:19.355511651 (538h44m19.226594687s) [noise=true , address=145.79.227.46:4603 , piece_id=YYPMVAQFAJX2UX2PVAOHCNLAM3XBRVLSX64O6TKFFMCFDKEJBMHA, seg_pos={0 0}, upload=472467]: Done uploading piece. err: <nil> Dec 28 02:13:19.367254862 (538h44m19.238337907s) [noise=true , address=145.79.227.5:4222 , piece_id=NLIOB2PCTLM5UNIABNNOO3FBHQKAFMS74CHWX3S525FRDWOFRYJQ, seg_pos={0 0}, upload=472467]: Done uploading piece. err: <nil> Dec 28 02:13:19.382732494 (538h44m19.253815539s) [noise=true , address=145.79.227.29:4943 , piece_id=6M3NIGPQQPVJD4VVAYSYYU2BXP45R63W43I3E5K36VCS3QI7ECQQ, seg_pos={0 0}, upload=472467]: Done uploading piece. err: <nil> Dec 28 02:13:19.386533643 (538h44m19.257616688s) [noise=true , address=145.79.227.48:4302 , piece_id=ELXMRPLA6SUP5YX3W6FY2P4HFBZAGPHMD7H64UU7PMBEZ3N5DUHQ, seg_pos={0 0}, upload=472467]: Done uploading piece. err: <nil> Dec 28 02:13:19.386653150 (538h44m19.257736185s) [noise=true , address=145.79.227.19:4079 , piece_id=65XSP3UPFJHNGPWM7QCOSXN5ZM3A474RARZMQ65TAIVIFVJJS64A, seg_pos={0 0}, upload=472467]: Done uploading piece. err: <nil> Dec 28 02:13:19.387739318 (538h44m19.258822353s) [noise=true , address=145.79.227.31:4798 , piece_id=HOYKSTDDWQUWIUIQMJHKCDL66RFIBQXPGURFLQRH3MNZVV6ITOSQ, seg_pos={0 0}, upload=472467]: Done uploading piece. err: <nil> Dec 28 02:13:19.388367203 (538h44m19.259450248s) [noise=true , address=145.79.227.45:4230 , piece_id=HT3OCVNTU4RQZQZNFIG5SG4BCS6U4CF4MFOVMSTII2UMDQY3ZUSA, seg_pos={0 0}, upload=472467]: Done uploading piece. err: <nil> Dec 28 02:13:19.428250388 (538h44m19.299333423s) [noise=true , address=45.91.116.5:4299 , piece_id=RWDLMPVHCOUENEQHYQ3OIPJD7B4FOCLTKM77YQJRNZNRWMEFMDCA, seg_pos={0 0}, upload=472467]: Done uploading piece. err: <nil> Dec 28 02:13:19.448818894 (538h44m19.319901939s) [noise=true , address=45.91.116.8:4045 , piece_id=VDXKZ5WLZKKF2GFTMMQYOL7BD22XO357PF2UV2XAL27CKKJ5MNJQ, seg_pos={0 0}, upload=472467]: Done uploading piece. err: <nil> Dec 28 02:13:19.450210995 (538h44m19.32129403s) [noise=true , address=145.79.227.36:4519 , piece_id=JUCQAE6U2VOYAXDVRON5T2PJKIXGVVXKKWLXVJLUAK2AOIECVM5Q, seg_pos={0 0}, upload=472467]: Done uploading piece. err: <nil> Dec 28 02:13:19.460267654 (538h44m19.331350689s) [noise=true , address=77.90.31.10:4517 , piece_id=IGX47ZVM5MH4DPH5T5TQFHMFC5OKOTAF2TCIZCUF3S5AUZOBQFXQ, seg_pos={0 0}, upload=472467]: Done uploading piece. err: <nil> Dec 28 02:13:19.546104893 (538h44m19.417187938s) [noise=true , address=45.91.116.13:4153 , piece_id=OIYGJXOYYHUVH5M6T57PVR6GUAYYAGCBIPIHZDB77KDIDQU6IA4A, seg_pos={0 0}, upload=472467]: Done uploading piece. err: <nil> Dec 28 02:13:19.546129119 (538h44m19.417212154s) [seg_pos={0 0}, upload=472467]: Segment reached optimal threshold of 13 pieces. Dec 28 02:13:19.546395766 (538h44m19.417478831s) [noise=true , address=45.91.116.10:4084 , piece_id=RRVJATQXQSZC66YGEPTV6ICXQ5C3PXPA4XH3IB5TIPRHUHNKLQYA, seg_pos={0 0}, upload=472467]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 28 02:13:19.546407219 (538h44m19.417490264s) [noise=true , address=45.91.116.10:4084 , piece_id=RRVJATQXQSZC66YGEPTV6ICXQ5C3PXPA4XH3IB5TIPRHUHNKLQYA, seg_pos={0 0}, upload=472467]: Long tail cancellation due to optimal threshold Dec 28 02:13:19.546767324 (538h44m19.417850359s) [noise=true , address=145.79.227.47:4267 , piece_id=HKOTMPNDD4CQOU2JWXNEXT4FZESGQ6VPRZQAV75XBHGSF3DXKJ7A, seg_pos={0 0}, upload=472467]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8): protocol: expected piece hash; context canceled; EOF Dec 28 02:13:19.546783685 (538h44m19.41786672s) [noise=true , address=145.79.227.47:4267 , piece_id=HKOTMPNDD4CQOU2JWXNEXT4FZESGQ6VPRZQAV75XBHGSF3DXKJ7A, seg_pos={0 0}, upload=472467]: Long tail cancellation due to optimal threshold Dec 28 02:13:19.546833931 (538h44m19.417916966s) [noise=true , address=45.91.116.9:4264 , piece_id=SK5DXIK3BHW7QYBK72QPTDBIIICKSV5NKKRFOGTMNQQ2RQ4YCH3Q, seg_pos={0 0}, upload=472467]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: write tcp 188.166.225.170:46590->45.91.116.9:4264: use of closed network connection; write tcp 188.166.225.170:46590->45.91.116.9:4264: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46590->45.91.116.9:4264: use of closed network connection Dec 28 02:13:19.546874999 (538h44m19.417958054s) [noise=true , address=45.91.116.9:4264 , piece_id=SK5DXIK3BHW7QYBK72QPTDBIIICKSV5NKKRFOGTMNQQ2RQ4YCH3Q, seg_pos={0 0}, upload=472467]: Long tail cancellation due to optimal threshold Dec 28 02:13:19.546898514 (538h44m19.417981559s) [seg_pos={0 0}, upload=472467]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 02:13:19.546985159 (538h44m19.418068194s) [upload=472467]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:13:19.546991090 (538h44m19.418074125s) [upload=472467]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:13:19.844974991 (538h44m19.716058027s) [upload=472467]: Done uploading segments... Dec 28 02:13:19.844996412 (538h44m19.716079457s) [upload=472467]: Upload finished. err: <nil> Duration: 30.002759167s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 52.497042811s trace: https://jaeger.dev.storj.io/trace/12a1d059b6f75af7 FAIL (2025-12-28 02:13:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1.145865281 (538h33m41.016948316s) [noise=true , address=77.90.31.10:4551 , piece_id=KBPO4MUDASHKMPRNJELQWLNMJNQBB3PZUOVNCEPP2CYVHU6X2PPA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.146408725 (538h33m41.01749177s) [noise=true , address=145.79.227.15:4908 , piece_id=I2FSDNVYBFZKRXZNUHH2APY7N2PI7DPAKFKGXRJW2UFU2PBKSSNQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.146778139 (538h33m41.017861184s) [noise=true , address=145.79.227.43:4161 , piece_id=ZN3T6JSPHJ57X4YYG5MHRJYYOFPH2T2CSGJ34MJQU55HDC7CMOXQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.147137874 (538h33m41.018220919s) [noise=true , address=77.90.31.21:4882 , piece_id=WQAGQTGLU5HRKEET36W34MQP4NPGSVCA4QZ6YEGVAJWBBPLZOUQQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.147426784 (538h33m41.018509819s) [noise=true , address=145.79.227.35:4503 , piece_id=MDYJ2TROT4FHLX4XHKHEQACC7ZXHZCQ5MUPJL5EDS3F2YMXSUGUA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.147466109 (538h33m41.018549154s) [noise=true , address=77.90.31.15:4730 , piece_id=EN7IGRJZ2WFN4HXJPMNLLXDUZPIN7KDUGKPPLJH7CSA4SDOUM2JA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.147817368 (538h33m41.018900413s) [noise=true , address=77.90.31.18:4833 , piece_id=3E57LWLI3WHROFGMXVZ7LHDHO5O7453LWQXA3OEKM52B2NIYM2JA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.147860690 (538h33m41.018943745s) [noise=true , address=145.79.227.44:4212 , piece_id=EFTTQM6323MUQ5WTGY6DFBTF3TGRM6XZWXIGT6DM4VS7GRGM4C2Q, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148163427 (538h33m41.019246472s) [noise=true , address=77.90.31.19:4862 , piece_id=PSBBBSGY7THVLMXJQCADIE7VYZ33OIIDVAEQWG3RWMNA6RSKELNQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148189276 (538h33m41.019272321s) [noise=true , address=77.90.31.10:4524 , piece_id=X2VCBS4PTPPWI6U64T4FVIULRLCDY3XUOJTXJNP34OHSO54ONIWA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148496531 (538h33m41.019579566s) [noise=true , address=145.79.227.23:4556 , piece_id=AKBXRSU5PM5QLHGHE3VXTJ3ZH23BECPGBLQSVZK6XVIYJGVDXT4Q, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148527380 (538h33m41.019610425s) [noise=true , address=145.79.227.33:4899 , piece_id=JFEXXK22EATV7UUH7LLXTFEPIUBW7EWSNYFXLE4AO5F6SLYFTGBA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148801512 (538h33m41.019884547s) [noise=true , address=77.90.31.14:4672 , piece_id=BZG2Y5X47J2WVFF5LBHEJZNUD2OQ3NOTVPTCAVT4TLFQB5FWESLA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148894920 (538h33m41.019977965s) [noise=true , address=5.231.73.5:4016 , piece_id=YCNKQEKJQSJHKKGCYWLOLEBKIVPJHZIJ6TYEG6DF4TRXMZ3ZTTIQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.149118806 (538h33m41.020201851s) [noise=true , address=145.79.227.47:4295 , piece_id=OM7B75MCPM7L3WP5MLQ7MAVBZ2NJUSISOWD56RSWI2AV2OWVDRTA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.149274993 (538h33m41.020358038s) [noise=true , address=145.79.227.50:4400 , piece_id=HZAT7YAVZYWK52FE4TIC6OA37U4VX5B35H55WMBO3MMWNIKTJLJQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.149625842 (538h33m41.020708877s) [noise=true , address=145.79.227.34:4915 , piece_id=RMXYZGGJA5EPJLQD6SQG677HFLOQWRY2HFTGEOICYAQG6QR557RQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:42.618098549 (538h33m42.489181584s) [upload=472313]: Next returned nil segment Dec 28 02:02:42.618120911 (538h33m42.489203946s) [upload=472313]: Waiting for error group managing segments... Dec 28 02:02:43.021704832 (538h33m42.892787877s) [noise=true , address=145.79.227.23:4556 , piece_id=AKBXRSU5PM5QLHGHE3VXTJ3ZH23BECPGBLQSVZK6XVIYJGVDXT4Q, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.027980492 (538h33m42.899063537s) [noise=true , address=145.79.227.50:4400 , piece_id=HZAT7YAVZYWK52FE4TIC6OA37U4VX5B35H55WMBO3MMWNIKTJLJQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.043329239 (538h33m42.914412284s) [noise=true , address=145.79.227.15:4908 , piece_id=I2FSDNVYBFZKRXZNUHH2APY7N2PI7DPAKFKGXRJW2UFU2PBKSSNQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.056308314 (538h33m42.927391349s) [noise=true , address=145.79.227.49:4347 , piece_id=WURIHGQASBLSVLZJVM5EJHWD7U4VFLLPGJIFYNOK6ALA5LYEQDSA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.074628196 (538h33m42.945711252s) [noise=true , address=145.79.227.47:4295 , piece_id=OM7B75MCPM7L3WP5MLQ7MAVBZ2NJUSISOWD56RSWI2AV2OWVDRTA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.080495740 (538h33m42.951578775s) [noise=true , address=5.231.73.5:4016 , piece_id=YCNKQEKJQSJHKKGCYWLOLEBKIVPJHZIJ6TYEG6DF4TRXMZ3ZTTIQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.193955176 (538h33m43.065038241s) [noise=true , address=77.90.31.10:4551 , piece_id=KBPO4MUDASHKMPRNJELQWLNMJNQBB3PZUOVNCEPP2CYVHU6X2PPA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.301513865 (538h33m43.1725969s) [noise=true , address=77.90.31.15:4730 , piece_id=EN7IGRJZ2WFN4HXJPMNLLXDUZPIN7KDUGKPPLJH7CSA4SDOUM2JA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.305020622 (538h33m43.176103657s) [noise=true , address=77.90.31.19:4862 , piece_id=PSBBBSGY7THVLMXJQCADIE7VYZ33OIIDVAEQWG3RWMNA6RSKELNQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.312187619 (538h33m43.183270654s) [noise=true , address=77.90.31.14:4672 , piece_id=BZG2Y5X47J2WVFF5LBHEJZNUD2OQ3NOTVPTCAVT4TLFQB5FWESLA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.319277390 (538h33m43.190360425s) [noise=true , address=77.90.31.21:4882 , piece_id=WQAGQTGLU5HRKEET36W34MQP4NPGSVCA4QZ6YEGVAJWBBPLZOUQQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.948916959 (538h33m43.820000004s) [noise=true , address=145.79.227.34:4915 , piece_id=RMXYZGGJA5EPJLQD6SQG677HFLOQWRY2HFTGEOICYAQG6QR557RQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.989823331 (538h33m43.860906366s) [noise=true , address=145.79.227.44:4212 , piece_id=EFTTQM6323MUQ5WTGY6DFBTF3TGRM6XZWXIGT6DM4VS7GRGM4C2Q, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:44.123218949 (538h33m43.994301984s) [noise=true , address=145.79.227.43:4161 , piece_id=ZN3T6JSPHJ57X4YYG5MHRJYYOFPH2T2CSGJ34MJQU55HDC7CMOXQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:44.123238386 (538h33m43.994321421s) [seg_pos={0 0}, upload=472313]: Segment reached optimal threshold of 14 pieces. Dec 28 02:02:44.123431754 (538h33m43.994514789s) [noise=true , address=145.79.227.33:4899 , piece_id=JFEXXK22EATV7UUH7LLXTFEPIUBW7EWSNYFXLE4AO5F6SLYFTGBA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b4pZVQRnjcQNn45VTgdsz5fpjYPMAzSXGR5KJ5BP9XHMkRnRw): protocol: expected piece hash; context canceled; EOF Dec 28 02:02:44.123437586 (538h33m43.994520621s) [noise=true , address=145.79.227.33:4899 , piece_id=JFEXXK22EATV7UUH7LLXTFEPIUBW7EWSNYFXLE4AO5F6SLYFTGBA, seg_pos={0 0}, upload=472313]: Long tail cancellation due to optimal threshold Dec 28 02:02:44.123631064 (538h33m43.994714109s) [noise=true , address=77.90.31.18:4833 , piece_id=3E57LWLI3WHROFGMXVZ7LHDHO5O7453LWQXA3OEKM52B2NIYM2JA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FbVbo8kGy1aaFyCk2noiXNUJHtosFJHFykBMRecGd5Kj5gjEp): protocol: expected piece hash; context canceled; EOF Dec 28 02:02:44.123675267 (538h33m43.994758313s) [noise=true , address=77.90.31.18:4833 , piece_id=3E57LWLI3WHROFGMXVZ7LHDHO5O7453LWQXA3OEKM52B2NIYM2JA, seg_pos={0 0}, upload=472313]: Long tail cancellation due to optimal threshold Dec 28 02:02:44.123774066 (538h33m43.994857101s) [noise=true , address=77.90.31.10:4524 , piece_id=X2VCBS4PTPPWI6U64T4FVIULRLCDY3XUOJTXJNP34OHSO54ONIWA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe): protocol: expected piece hash; context canceled; EOF Dec 28 02:02:44.123781761 (538h33m43.994864796s) [noise=true , address=77.90.31.10:4524 , piece_id=X2VCBS4PTPPWI6U64T4FVIULRLCDY3XUOJTXJNP34OHSO54ONIWA, seg_pos={0 0}, upload=472313]: Long tail cancellation due to optimal threshold Dec 28 02:02:44.123923341 (538h33m43.995006376s) [noise=true , address=145.79.227.35:4503 , piece_id=MDYJ2TROT4FHLX4XHKHEQACC7ZXHZCQ5MUPJL5EDS3F2YMXSUGUA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ctmSF1aMQasuXByFW8DppDantREmmWL2WUjmKGW5ZULVePyvm): protocol: expected piece hash; context canceled; EOF Dec 28 02:02:44.123928591 (538h33m43.995011636s) [noise=true , address=145.79.227.35:4503 , piece_id=MDYJ2TROT4FHLX4XHKHEQACC7ZXHZCQ5MUPJL5EDS3F2YMXSUGUA, seg_pos={0 0}, upload=472313]: Long tail cancellation due to optimal threshold Dec 28 02:02:44.123944731 (538h33m43.995027766s) [seg_pos={0 0}, upload=472313]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 02:02:44.124017179 (538h33m43.995100214s) [upload=472313]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:02:44.124022469 (538h33m43.995105504s) [upload=472313]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:02:44.409047274 (538h33m44.280130349s) [upload=472313]: Done uploading segments... Dec 28 02:02:44.409073354 (538h33m44.280156389s) [upload=472313]: Upload finished. err: <nil> Duration: 52.497057309s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 32 out of 60 is failed (threshold=3) [X....X.XX.XXXXX..X.XXX.X.XX.XX.X..X.X.X.X.X..X...XX..X.XXX.X] duration: 30.002715755s trace: https://jaeger.dev.storj.io/trace/745356e45637dfc FAIL (2025-12-28 02:13:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e Dec 28 02:12:46.743244583 (538h43m46.614327628s) [upload=472460]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 02:12:46.743250824 (538h43m46.614333859s) [upload=472460]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 02:12:46.991616575 (538h43m46.86269963s) [seg_pos={0 0}, upload=472460]: Begin upload segment... Dec 28 02:12:46.991687510 (538h43m46.862770545s) [noise=true , address=145.79.227.38:4979 , piece_id=EMSQHZYQVH6SDH2TXTKGOYSFTKQTB3D62KETAJW5QVM6ZYVZ2F4Q, seg_pos={0 0}, upload=472460]: Uploading piece... Dec 28 02:12:46.991728718 (538h43m46.862811753s) [seg_pos={0 0}, upload=472460]: Done begin upload segment. Dec 28 02:12:46.991763995 (538h43m46.86284703s) [noise=true , address=45.91.116.6:4973 , piece_id=WYSCXDZMESSS5NPF7LS5XITYC6IO5WZ7TIHECPMDVQRNTM3RVUMQ, seg_pos={0 0}, upload=472460]: Uploading piece... Dec 28 02:12:46.992587313 (538h43m46.863670359s) [noise=true , address=145.79.227.42:4123 , piece_id=VNS4PO22KM2HRNB7BKZVU6XHF2ILB5MZTDSBHIIDYDVPKJYAG7NA, seg_pos={0 0}, upload=472460]: Uploading piece... Dec 28 02:12:46.992963911 (538h43m46.864046956s) [noise=true , address=77.90.31.18:4808 , piece_id=6QYAJZU4BLLMLBM25W5FAWIPWATX2ZGTJNQBUTQMCN24VDOK6HUA, seg_pos={0 0}, upload=472460]: Uploading piece... Dec 28 02:12:46.993301925 (538h43m46.86438496s) [noise=true , address=145.79.227.47:4265 , piece_id=3QNWEEKCV74HGGIDCVLSRR5GEVKVMJQT2WVH7UTQYZLR7BWUALWA, seg_pos={0 0}, upload=472460]: Uploading piece... Dec 28 02:12:46.993610812 (538h43m46.864693847s) [noise=true , address=145.79.227.48:4303 , piece_id=5KLEXFXFQA73KKZ47PDYXB5Y7E7C3FDPQC3PWXLPW233DDUG63YA, seg_pos={0 0}, upload=472460]: Uploading piece... Dec 28 02:12:46.994222247 (538h43m46.865305282s) [noise=true , address=145.79.227.11:4732 , piece_id=52DD7ARB6SIH3MJXVVF4FHUSERTFFWDYUGWARD25573IQCICX3PQ, seg_pos={0 0}, upload=472460]: Uploading piece... Dec 28 02:12:46.994873267 (538h43m46.865956312s) [noise=true , address=145.79.227.7:4295 , piece_id=Q4TBTQAETZPFSF5OHOKHD4X56GII55SH34MUIBKCPBDK4EG5V3SA, seg_pos={0 0}, upload=472460]: Uploading piece... Dec 28 02:12:46.995027029 (538h43m46.866110094s) [noise=true , address=45.91.116.13:4154 , piece_id=RPECLAKPSUSGKZY63HEVI7KS5BKZ55UMTIT2EC7WBJUIJXAJZ74Q, seg_pos={0 0}, upload=472460]: Uploading piece... Dec 28 02:12:46.995954636 (538h43m46.867037681s) [noise=true , address=77.90.31.13:4626 , piece_id=Q3Z5ZUYIXRU7JQXSEYPPWHKZYXE4CZJDR6HJKJWMLOQN7T2IMV3A, seg_pos={0 0}, upload=472460]: Uploading piece... Dec 28 02:12:46.995990794 (538h43m46.86707383s) [noise=true , address=77.90.31.10:4519 , piece_id=TEEOBI7TSQJGSJOVXL3JDO4CBWY7ZPBOUJBYS6BSA6TAHRI3YK5Q, seg_pos={0 0}, upload=472460]: Uploading piece... Dec 28 02:12:46.997054780 (538h43m46.868137825s) [noise=true , address=145.79.227.16:4911 , piece_id=7CSUT7DXLQYQG4NP7Y4ACLIMMB5ZLH6C5TBCFNYYHFX7AEFWHWRQ, seg_pos={0 0}, upload=472460]: Uploading piece... Dec 28 02:12:46.997105126 (538h43m46.868188181s) [noise=true , address=145.79.227.37:4566 , piece_id=232OR5DLVS6CAU22JM7B74GD4RGDQAQQYUY2HJ47VD3S2Q6B3PUQ, seg_pos={0 0}, upload=472460]: Uploading piece... Dec 28 02:12:46.998327945 (538h43m46.86941099s) [noise=true , address=145.79.227.51:4410 , piece_id=7PWCNRHITTIQ36L4LCTVA5QVQ2RN6LCOE3MPF3ZY5H3O7ZWTZN2A, seg_pos={0 0}, upload=472460]: Uploading piece... Dec 28 02:12:46.998782490 (538h43m46.869865525s) [noise=true , address=145.79.227.5:4222 , piece_id=BWSR6VMZNQWWO6ONNAN2X4UIIW5XJUQUZ75L564ZGPWP2GDT5WJQ, seg_pos={0 0}, upload=472460]: Uploading piece... Dec 28 02:12:46.999454730 (538h43m46.870537775s) [noise=true , address=145.79.227.14:4838 , piece_id=4BAZDPK3S3TNPN5QEBWLLBVSXLAFTBE3QDWOBD2MEGGEPXUOFCLA, seg_pos={0 0}, upload=472460]: Uploading piece... Dec 28 02:12:48.728153071 (538h43m48.599236106s) [upload=472460]: Next returned nil segment Dec 28 02:12:48.728167278 (538h43m48.599250313s) [upload=472460]: Waiting for error group managing segments... Dec 28 02:12:49.010120814 (538h43m48.881203869s) [noise=true , address=145.79.227.38:4979 , piece_id=EMSQHZYQVH6SDH2TXTKGOYSFTKQTB3D62KETAJW5QVM6ZYVZ2F4Q, seg_pos={0 0}, upload=472460]: Done uploading piece. err: <nil> Dec 28 02:12:49.027784971 (538h43m48.898868016s) [noise=true , address=145.79.227.5:4222 , piece_id=BWSR6VMZNQWWO6ONNAN2X4UIIW5XJUQUZ75L564ZGPWP2GDT5WJQ, seg_pos={0 0}, upload=472460]: Done uploading piece. err: <nil> Dec 28 02:12:49.046321747 (538h43m48.917404782s) [noise=true , address=145.79.227.51:4410 , piece_id=7PWCNRHITTIQ36L4LCTVA5QVQ2RN6LCOE3MPF3ZY5H3O7ZWTZN2A, seg_pos={0 0}, upload=472460]: Done uploading piece. err: <nil> Dec 28 02:12:49.054701885 (538h43m48.92578493s) [noise=true , address=145.79.227.11:4732 , piece_id=52DD7ARB6SIH3MJXVVF4FHUSERTFFWDYUGWARD25573IQCICX3PQ, seg_pos={0 0}, upload=472460]: Done uploading piece. err: <nil> Dec 28 02:12:49.223634030 (538h43m49.094717065s) [noise=true , address=145.79.227.14:4838 , piece_id=4BAZDPK3S3TNPN5QEBWLLBVSXLAFTBE3QDWOBD2MEGGEPXUOFCLA, seg_pos={0 0}, upload=472460]: Done uploading piece. err: <nil> Dec 28 02:12:49.288148817 (538h43m49.159231862s) [noise=true , address=77.90.31.18:4808 , piece_id=6QYAJZU4BLLMLBM25W5FAWIPWATX2ZGTJNQBUTQMCN24VDOK6HUA, seg_pos={0 0}, upload=472460]: Done uploading piece. err: <nil> Dec 28 02:12:49.381586647 (538h43m49.252669682s) [noise=true , address=45.91.116.6:4973 , piece_id=WYSCXDZMESSS5NPF7LS5XITYC6IO5WZ7TIHECPMDVQRNTM3RVUMQ, seg_pos={0 0}, upload=472460]: Done uploading piece. err: <nil> Dec 28 02:12:49.390530909 (538h43m49.261613944s) [noise=true , address=145.79.227.48:4303 , piece_id=5KLEXFXFQA73KKZ47PDYXB5Y7E7C3FDPQC3PWXLPW233DDUG63YA, seg_pos={0 0}, upload=472460]: Done uploading piece. err: <nil> Dec 28 02:12:49.401701600 (538h43m49.272784635s) [noise=true , address=145.79.227.7:4295 , piece_id=Q4TBTQAETZPFSF5OHOKHD4X56GII55SH34MUIBKCPBDK4EG5V3SA, seg_pos={0 0}, upload=472460]: Done uploading piece. err: <nil> Dec 28 02:12:49.588017867 (538h43m49.459100902s) [noise=true , address=145.79.227.47:4265 , piece_id=3QNWEEKCV74HGGIDCVLSRR5GEVKVMJQT2WVH7UTQYZLR7BWUALWA, seg_pos={0 0}, upload=472460]: Done uploading piece. err: <nil> Dec 28 02:12:49.607072311 (538h43m49.478155346s) [noise=true , address=145.79.227.42:4123 , piece_id=VNS4PO22KM2HRNB7BKZVU6XHF2ILB5MZTDSBHIIDYDVPKJYAG7NA, seg_pos={0 0}, upload=472460]: Done uploading piece. err: <nil> Dec 28 02:12:49.612037204 (538h43m49.483120239s) [noise=true , address=145.79.227.16:4911 , piece_id=7CSUT7DXLQYQG4NP7Y4ACLIMMB5ZLH6C5TBCFNYYHFX7AEFWHWRQ, seg_pos={0 0}, upload=472460]: Done uploading piece. err: <nil> Dec 28 02:12:50.007760578 (538h43m49.878843613s) [noise=true , address=77.90.31.13:4626 , piece_id=Q3Z5ZUYIXRU7JQXSEYPPWHKZYXE4CZJDR6HJKJWMLOQN7T2IMV3A, seg_pos={0 0}, upload=472460]: Done uploading piece. err: <nil> Dec 28 02:12:50.007779333 (538h43m49.878862378s) [seg_pos={0 0}, upload=472460]: Segment reached optimal threshold of 13 pieces. Dec 28 02:12:50.007949677 (538h43m49.879032722s) [noise=true , address=145.79.227.37:4566 , piece_id=232OR5DLVS6CAU22JM7B74GD4RGDQAQQYUY2HJ47VD3S2Q6B3PUQ, seg_pos={0 0}, upload=472460]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F): protocol: write tcp 188.166.225.170:59424->145.79.227.37:4566: use of closed network connection; write tcp 188.166.225.170:59424->145.79.227.37:4566: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59424->145.79.227.37:4566: use of closed network connection Dec 28 02:12:50.007957012 (538h43m49.879040057s) [noise=true , address=145.79.227.37:4566 , piece_id=232OR5DLVS6CAU22JM7B74GD4RGDQAQQYUY2HJ47VD3S2Q6B3PUQ, seg_pos={0 0}, upload=472460]: Long tail cancellation due to optimal threshold Dec 28 02:12:50.008044097 (538h43m49.879127143s) [noise=true , address=45.91.116.13:4154 , piece_id=RPECLAKPSUSGKZY63HEVI7KS5BKZ55UMTIT2EC7WBJUIJXAJZ74Q, seg_pos={0 0}, upload=472460]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: write tcp 188.166.225.170:45984->45.91.116.13:4154: use of closed network connection; write tcp 188.166.225.170:45984->45.91.116.13:4154: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45984->45.91.116.13:4154: use of closed network connection Dec 28 02:12:50.008048746 (538h43m49.879131781s) [noise=true , address=45.91.116.13:4154 , piece_id=RPECLAKPSUSGKZY63HEVI7KS5BKZ55UMTIT2EC7WBJUIJXAJZ74Q, seg_pos={0 0}, upload=472460]: Long tail cancellation due to optimal threshold Dec 28 02:12:50.008105625 (538h43m49.87918866s) [noise=true , address=77.90.31.10:4519 , piece_id=TEEOBI7TSQJGSJOVXL3JDO4CBWY7ZPBOUJBYS6BSA6TAHRI3YK5Q, seg_pos={0 0}, upload=472460]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: read tcp 188.166.225.170:56418->77.90.31.10:4519: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 02:12:50.008113690 (538h43m49.879196725s) [noise=true , address=77.90.31.10:4519 , piece_id=TEEOBI7TSQJGSJOVXL3JDO4CBWY7ZPBOUJBYS6BSA6TAHRI3YK5Q, seg_pos={0 0}, upload=472460]: Long tail cancellation due to optimal threshold Dec 28 02:12:50.008153055 (538h43m49.87923609s) [seg_pos={0 0}, upload=472460]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 02:12:50.008290176 (538h43m49.879373221s) [upload=472460]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:12:50.008296788 (538h43m49.879379833s) [upload=472460]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:12:50.313024370 (538h43m50.184107406s) [upload=472460]: Done uploading segments... Dec 28 02:12:50.313049729 (538h43m50.184132784s) [upload=472460]: Upload finished. err: <nil> Duration: 30.002727148s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................................................X.........] duration: 3.466236665s trace: https://jaeger.dev.storj.io/trace/7fc142ad3d711e34 FAIL (2025-12-28 02:13:05 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.466269638s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 41.796114591s trace: https://jaeger.dev.storj.io/trace/12a1d059b6f75af7 FAIL (2025-12-28 02:12:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1.145865281 (538h33m41.016948316s) [noise=true , address=77.90.31.10:4551 , piece_id=KBPO4MUDASHKMPRNJELQWLNMJNQBB3PZUOVNCEPP2CYVHU6X2PPA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.146408725 (538h33m41.01749177s) [noise=true , address=145.79.227.15:4908 , piece_id=I2FSDNVYBFZKRXZNUHH2APY7N2PI7DPAKFKGXRJW2UFU2PBKSSNQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.146778139 (538h33m41.017861184s) [noise=true , address=145.79.227.43:4161 , piece_id=ZN3T6JSPHJ57X4YYG5MHRJYYOFPH2T2CSGJ34MJQU55HDC7CMOXQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.147137874 (538h33m41.018220919s) [noise=true , address=77.90.31.21:4882 , piece_id=WQAGQTGLU5HRKEET36W34MQP4NPGSVCA4QZ6YEGVAJWBBPLZOUQQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.147426784 (538h33m41.018509819s) [noise=true , address=145.79.227.35:4503 , piece_id=MDYJ2TROT4FHLX4XHKHEQACC7ZXHZCQ5MUPJL5EDS3F2YMXSUGUA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.147466109 (538h33m41.018549154s) [noise=true , address=77.90.31.15:4730 , piece_id=EN7IGRJZ2WFN4HXJPMNLLXDUZPIN7KDUGKPPLJH7CSA4SDOUM2JA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.147817368 (538h33m41.018900413s) [noise=true , address=77.90.31.18:4833 , piece_id=3E57LWLI3WHROFGMXVZ7LHDHO5O7453LWQXA3OEKM52B2NIYM2JA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.147860690 (538h33m41.018943745s) [noise=true , address=145.79.227.44:4212 , piece_id=EFTTQM6323MUQ5WTGY6DFBTF3TGRM6XZWXIGT6DM4VS7GRGM4C2Q, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148163427 (538h33m41.019246472s) [noise=true , address=77.90.31.19:4862 , piece_id=PSBBBSGY7THVLMXJQCADIE7VYZ33OIIDVAEQWG3RWMNA6RSKELNQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148189276 (538h33m41.019272321s) [noise=true , address=77.90.31.10:4524 , piece_id=X2VCBS4PTPPWI6U64T4FVIULRLCDY3XUOJTXJNP34OHSO54ONIWA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148496531 (538h33m41.019579566s) [noise=true , address=145.79.227.23:4556 , piece_id=AKBXRSU5PM5QLHGHE3VXTJ3ZH23BECPGBLQSVZK6XVIYJGVDXT4Q, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148527380 (538h33m41.019610425s) [noise=true , address=145.79.227.33:4899 , piece_id=JFEXXK22EATV7UUH7LLXTFEPIUBW7EWSNYFXLE4AO5F6SLYFTGBA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148801512 (538h33m41.019884547s) [noise=true , address=77.90.31.14:4672 , piece_id=BZG2Y5X47J2WVFF5LBHEJZNUD2OQ3NOTVPTCAVT4TLFQB5FWESLA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148894920 (538h33m41.019977965s) [noise=true , address=5.231.73.5:4016 , piece_id=YCNKQEKJQSJHKKGCYWLOLEBKIVPJHZIJ6TYEG6DF4TRXMZ3ZTTIQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.149118806 (538h33m41.020201851s) [noise=true , address=145.79.227.47:4295 , piece_id=OM7B75MCPM7L3WP5MLQ7MAVBZ2NJUSISOWD56RSWI2AV2OWVDRTA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.149274993 (538h33m41.020358038s) [noise=true , address=145.79.227.50:4400 , piece_id=HZAT7YAVZYWK52FE4TIC6OA37U4VX5B35H55WMBO3MMWNIKTJLJQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.149625842 (538h33m41.020708877s) [noise=true , address=145.79.227.34:4915 , piece_id=RMXYZGGJA5EPJLQD6SQG677HFLOQWRY2HFTGEOICYAQG6QR557RQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:42.618098549 (538h33m42.489181584s) [upload=472313]: Next returned nil segment Dec 28 02:02:42.618120911 (538h33m42.489203946s) [upload=472313]: Waiting for error group managing segments... Dec 28 02:02:43.021704832 (538h33m42.892787877s) [noise=true , address=145.79.227.23:4556 , piece_id=AKBXRSU5PM5QLHGHE3VXTJ3ZH23BECPGBLQSVZK6XVIYJGVDXT4Q, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.027980492 (538h33m42.899063537s) [noise=true , address=145.79.227.50:4400 , piece_id=HZAT7YAVZYWK52FE4TIC6OA37U4VX5B35H55WMBO3MMWNIKTJLJQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.043329239 (538h33m42.914412284s) [noise=true , address=145.79.227.15:4908 , piece_id=I2FSDNVYBFZKRXZNUHH2APY7N2PI7DPAKFKGXRJW2UFU2PBKSSNQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.056308314 (538h33m42.927391349s) [noise=true , address=145.79.227.49:4347 , piece_id=WURIHGQASBLSVLZJVM5EJHWD7U4VFLLPGJIFYNOK6ALA5LYEQDSA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.074628196 (538h33m42.945711252s) [noise=true , address=145.79.227.47:4295 , piece_id=OM7B75MCPM7L3WP5MLQ7MAVBZ2NJUSISOWD56RSWI2AV2OWVDRTA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.080495740 (538h33m42.951578775s) [noise=true , address=5.231.73.5:4016 , piece_id=YCNKQEKJQSJHKKGCYWLOLEBKIVPJHZIJ6TYEG6DF4TRXMZ3ZTTIQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.193955176 (538h33m43.065038241s) [noise=true , address=77.90.31.10:4551 , piece_id=KBPO4MUDASHKMPRNJELQWLNMJNQBB3PZUOVNCEPP2CYVHU6X2PPA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.301513865 (538h33m43.1725969s) [noise=true , address=77.90.31.15:4730 , piece_id=EN7IGRJZ2WFN4HXJPMNLLXDUZPIN7KDUGKPPLJH7CSA4SDOUM2JA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.305020622 (538h33m43.176103657s) [noise=true , address=77.90.31.19:4862 , piece_id=PSBBBSGY7THVLMXJQCADIE7VYZ33OIIDVAEQWG3RWMNA6RSKELNQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.312187619 (538h33m43.183270654s) [noise=true , address=77.90.31.14:4672 , piece_id=BZG2Y5X47J2WVFF5LBHEJZNUD2OQ3NOTVPTCAVT4TLFQB5FWESLA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.319277390 (538h33m43.190360425s) [noise=true , address=77.90.31.21:4882 , piece_id=WQAGQTGLU5HRKEET36W34MQP4NPGSVCA4QZ6YEGVAJWBBPLZOUQQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.948916959 (538h33m43.820000004s) [noise=true , address=145.79.227.34:4915 , piece_id=RMXYZGGJA5EPJLQD6SQG677HFLOQWRY2HFTGEOICYAQG6QR557RQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.989823331 (538h33m43.860906366s) [noise=true , address=145.79.227.44:4212 , piece_id=EFTTQM6323MUQ5WTGY6DFBTF3TGRM6XZWXIGT6DM4VS7GRGM4C2Q, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:44.123218949 (538h33m43.994301984s) [noise=true , address=145.79.227.43:4161 , piece_id=ZN3T6JSPHJ57X4YYG5MHRJYYOFPH2T2CSGJ34MJQU55HDC7CMOXQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:44.123238386 (538h33m43.994321421s) [seg_pos={0 0}, upload=472313]: Segment reached optimal threshold of 14 pieces. Dec 28 02:02:44.123431754 (538h33m43.994514789s) [noise=true , address=145.79.227.33:4899 , piece_id=JFEXXK22EATV7UUH7LLXTFEPIUBW7EWSNYFXLE4AO5F6SLYFTGBA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b4pZVQRnjcQNn45VTgdsz5fpjYPMAzSXGR5KJ5BP9XHMkRnRw): protocol: expected piece hash; context canceled; EOF Dec 28 02:02:44.123437586 (538h33m43.994520621s) [noise=true , address=145.79.227.33:4899 , piece_id=JFEXXK22EATV7UUH7LLXTFEPIUBW7EWSNYFXLE4AO5F6SLYFTGBA, seg_pos={0 0}, upload=472313]: Long tail cancellation due to optimal threshold Dec 28 02:02:44.123631064 (538h33m43.994714109s) [noise=true , address=77.90.31.18:4833 , piece_id=3E57LWLI3WHROFGMXVZ7LHDHO5O7453LWQXA3OEKM52B2NIYM2JA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FbVbo8kGy1aaFyCk2noiXNUJHtosFJHFykBMRecGd5Kj5gjEp): protocol: expected piece hash; context canceled; EOF Dec 28 02:02:44.123675267 (538h33m43.994758313s) [noise=true , address=77.90.31.18:4833 , piece_id=3E57LWLI3WHROFGMXVZ7LHDHO5O7453LWQXA3OEKM52B2NIYM2JA, seg_pos={0 0}, upload=472313]: Long tail cancellation due to optimal threshold Dec 28 02:02:44.123774066 (538h33m43.994857101s) [noise=true , address=77.90.31.10:4524 , piece_id=X2VCBS4PTPPWI6U64T4FVIULRLCDY3XUOJTXJNP34OHSO54ONIWA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe): protocol: expected piece hash; context canceled; EOF Dec 28 02:02:44.123781761 (538h33m43.994864796s) [noise=true , address=77.90.31.10:4524 , piece_id=X2VCBS4PTPPWI6U64T4FVIULRLCDY3XUOJTXJNP34OHSO54ONIWA, seg_pos={0 0}, upload=472313]: Long tail cancellation due to optimal threshold Dec 28 02:02:44.123923341 (538h33m43.995006376s) [noise=true , address=145.79.227.35:4503 , piece_id=MDYJ2TROT4FHLX4XHKHEQACC7ZXHZCQ5MUPJL5EDS3F2YMXSUGUA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ctmSF1aMQasuXByFW8DppDantREmmWL2WUjmKGW5ZULVePyvm): protocol: expected piece hash; context canceled; EOF Dec 28 02:02:44.123928591 (538h33m43.995011636s) [noise=true , address=145.79.227.35:4503 , piece_id=MDYJ2TROT4FHLX4XHKHEQACC7ZXHZCQ5MUPJL5EDS3F2YMXSUGUA, seg_pos={0 0}, upload=472313]: Long tail cancellation due to optimal threshold Dec 28 02:02:44.123944731 (538h33m43.995027766s) [seg_pos={0 0}, upload=472313]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 02:02:44.124017179 (538h33m43.995100214s) [upload=472313]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:02:44.124022469 (538h33m43.995105504s) [upload=472313]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:02:44.409047274 (538h33m44.280130349s) [upload=472313]: Done uploading segments... Dec 28 02:02:44.409073354 (538h33m44.280156389s) [upload=472313]: Upload finished. err: <nil> Duration: 41.796132023s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 32 out of 60 is failed (threshold=3) [XX....X.XX.XXXXX..X.XXX.X.XX.XX.X..X.X.X.X.X..X...XX..X.XXX.] duration: 6.055917601s trace: https://jaeger.dev.storj.io/trace/19d45d58c66aa194 FAIL (2025-12-28 02:12:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: DK5FS77YXPC2YUKGLO2GHPCBRDAEIVDMK5ESJT4EDFWLLANNJVDA) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: 5XJ7LJPLRR7ALKWWHN3P2QE43QLD37I27GEL6ME4LU4GK462XZ3A) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: UIRV35YALI6NT3ERFGLIBIFOXRZD6LQ2MEOCK2DBZ7VLADELGYZA) context deadline exceeded ======LOGS====== Dec 28 02:11:35.885960014 (538h42m35.757043059s) [upload=472443]: Starting upload Dec 28 02:11:35.886044255 (538h42m35.75712731s) [upload=472443]: Done starting upload Dec 28 02:11:35.886131862 (538h42m35.757214907s) [upload=472443]: Uploading segments... Dec 28 02:11:35.886263483 (538h42m35.757346558s) [upload=472443]: Got next segment. Inline: false Dec 28 02:11:35.886307867 (538h42m35.757390902s) [upload=472443]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 02:11:35.886313287 (538h42m35.757396322s) [upload=472443]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 02:11:36.141046383 (538h42m36.012129418s) [seg_pos={0 0}, upload=472443]: Begin upload segment... Dec 28 02:11:36.141102439 (538h42m36.012185474s) [seg_pos={0 0}, upload=472443]: Done begin upload segment. Dec 28 02:11:36.141146322 (538h42m36.012229358s) [noise=true , address=145.79.227.13:4803 , piece_id=P6EACXBYOYLAFCKLIUQUUGXLONNC2KVQBRKN2EULFAX3RDVRYIDA, seg_pos={0 0}, upload=472443]: Uploading piece... Dec 28 02:11:36.141172863 (538h42m36.012255908s) [noise=true , address=45.91.116.14:4189 , piece_id=DBTL73IKPTJUDKGV3Q5AYZ5RAQXY35GYDM7KOAGNLZTLZS4K2XQQ, seg_pos={0 0}, upload=472443]: Uploading piece... Dec 28 02:11:36.141621868 (538h42m36.012704903s) [noise=true , address=145.79.227.42:4122 , piece_id=GBQ2XR2WUZNLKB6U7WWR33V22JXWSWJYCKJQ2UAQXKZ6JIPYZRIA, seg_pos={0 0}, upload=472443]: Uploading piece... Dec 28 02:11:36.142095601 (538h42m36.013178646s) [noise=true , address=145.79.227.33:4870 , piece_id=PTE3CQ4VJXJ6YHVOR72UACT22XQVCBEAI7VYFAKZQT3TLTUJ4OCQ, seg_pos={0 0}, upload=472443]: Uploading piece... Dec 28 02:11:36.142477468 (538h42m36.013560503s) [noise=true , address=145.79.227.52:4445 , piece_id=LSG3JJMDC5UX753JEDLQE3TZNBRS2JWAS3X3NQRODR6WS3YZS6ZQ, seg_pos={0 0}, upload=472443]: Uploading piece... Dec 28 02:11:36.142842683 (538h42m36.013925718s) [noise=true , address=145.79.227.49:4337 , piece_id=MRQYYVRT6EOINIS7L4MMB7HMUTK6U3WRR7HSFX25NSNRJ4KKRPOQ, seg_pos={0 0}, upload=472443]: Uploading piece... Dec 28 02:11:36.143139619 (538h42m36.014222654s) [noise=true , address=77.90.31.19:4844 , piece_id=WSJRGDLQFKKKR4BTIB5CKZFIL4SC33PEIKNLJ75LDZQK475OBPIA, seg_pos={0 0}, upload=472443]: Uploading piece... Dec 28 02:11:36.143477782 (538h42m36.014560827s) [noise=true , address=45.91.116.11:4120 , piece_id=MW2LUTCXDPAOHLKWFF4VLT4CNMHGZ4GLLC5JXQ52RIPX5Q23UUVA, seg_pos={0 0}, upload=472443]: Uploading piece... Dec 28 02:11:36.143821106 (538h42m36.014904141s) [noise=true , address=45.91.116.7:4009 , piece_id=5XJ7LJPLRR7ALKWWHN3P2QE43QLD37I27GEL6ME4LU4GK462XZ3A, seg_pos={0 0}, upload=472443]: Uploading piece... Dec 28 02:11:36.145093028 (538h42m36.016176063s) [noise=true , address=77.90.31.17:4773 , piece_id=4HOJERKNNECSJXZMTPWSX3YAIOA5E32XASJ3Q5VESE7XSOSA4C6Q, seg_pos={0 0}, upload=472443]: Uploading piece... Dec 28 02:11:36.146411950 (538h42m36.017494985s) [noise=true , address=45.91.116.4:4940 , piece_id=HSS7MBZFQNUAPGP6PMXNMP7KM7BNQ37GF6GSQUZGCDBUHMMJFIQA, seg_pos={0 0}, upload=472443]: Uploading piece... Dec 28 02:11:36.147915813 (538h42m36.018998848s) [noise=true , address=45.91.116.5:4300 , piece_id=UIRV35YALI6NT3ERFGLIBIFOXRZD6LQ2MEOCK2DBZ7VLADELGYZA, seg_pos={0 0}, upload=472443]: Uploading piece... Dec 28 02:11:36.149262688 (538h42m36.020345733s) [noise=true , address=145.79.227.51:4411 , piece_id=OOSIWGVDPTBB3LOWCM4W25T2KMBMQVLR3ROGGBGP7OQO4OEYGXOA, seg_pos={0 0}, upload=472443]: Uploading piece... Dec 28 02:11:36.150634360 (538h42m36.021717395s) [noise=true , address=77.90.31.13:4629 , piece_id=QLKSHQNI5N4ZZTF2NSUCDJA7GUWHYGFJBPQLAAAVGW6CMZGDN5EQ, seg_pos={0 0}, upload=472443]: Uploading piece... Dec 28 02:11:36.151801142 (538h42m36.022884177s) [noise=true , address=45.91.116.8:4045 , piece_id=DK5FS77YXPC2YUKGLO2GHPCBRDAEIVDMK5ESJT4EDFWLLANNJVDA, seg_pos={0 0}, upload=472443]: Uploading piece... Dec 28 02:11:36.152647123 (538h42m36.023730158s) [noise=true , address=145.79.227.9:4367 , piece_id=53WBZ2PZBQEZRJR5XKT47526FPRKKPDL434PPHTINPDJFTP5X7WQ, seg_pos={0 0}, upload=472443]: Uploading piece... Dec 28 02:11:37.862878909 (538h42m37.733961944s) [upload=472443]: Next returned nil segment Dec 28 02:11:37.862897123 (538h42m37.733980168s) [upload=472443]: Waiting for error group managing segments... Dec 28 02:11:38.154407318 (538h42m38.025490363s) [noise=true , address=145.79.227.9:4367 , piece_id=53WBZ2PZBQEZRJR5XKT47526FPRKKPDL434PPHTINPDJFTP5X7WQ, seg_pos={0 0}, upload=472443]: Done uploading piece. err: <nil> Dec 28 02:11:38.155631699 (538h42m38.026714734s) [noise=true , address=145.79.227.52:4445 , piece_id=LSG3JJMDC5UX753JEDLQE3TZNBRS2JWAS3X3NQRODR6WS3YZS6ZQ, seg_pos={0 0}, upload=472443]: Done uploading piece. err: <nil> Dec 28 02:11:38.160119925 (538h42m38.03120297s) [noise=true , address=145.79.227.13:4803 , piece_id=P6EACXBYOYLAFCKLIUQUUGXLONNC2KVQBRKN2EULFAX3RDVRYIDA, seg_pos={0 0}, upload=472443]: Done uploading piece. err: <nil> Dec 28 02:11:38.171207337 (538h42m38.042290372s) [noise=true , address=145.79.227.42:4122 , piece_id=GBQ2XR2WUZNLKB6U7WWR33V22JXWSWJYCKJQ2UAQXKZ6JIPYZRIA, seg_pos={0 0}, upload=472443]: Done uploading piece. err: <nil> Dec 28 02:11:38.175984393 (538h42m38.047067478s) [noise=true , address=145.79.227.51:4411 , piece_id=OOSIWGVDPTBB3LOWCM4W25T2KMBMQVLR3ROGGBGP7OQO4OEYGXOA, seg_pos={0 0}, upload=472443]: Done uploading piece. err: <nil> Dec 28 02:11:38.338320298 (538h42m38.209403333s) [noise=true , address=77.90.31.13:4629 , piece_id=QLKSHQNI5N4ZZTF2NSUCDJA7GUWHYGFJBPQLAAAVGW6CMZGDN5EQ, seg_pos={0 0}, upload=472443]: Done uploading piece. err: <nil> Dec 28 02:11:38.375693494 (538h42m38.246776539s) [noise=true , address=45.91.116.7:4009 , piece_id=5XJ7LJPLRR7ALKWWHN3P2QE43QLD37I27GEL6ME4LU4GK462XZ3A, seg_pos={0 0}, upload=472443]: Done uploading piece. err: <nil> Dec 28 02:11:38.501546308 (538h42m38.372629353s) [noise=true , address=77.90.31.17:4773 , piece_id=4HOJERKNNECSJXZMTPWSX3YAIOA5E32XASJ3Q5VESE7XSOSA4C6Q, seg_pos={0 0}, upload=472443]: Done uploading piece. err: <nil> Dec 28 02:11:38.502263985 (538h42m38.37334702s) [noise=true , address=45.91.116.14:4189 , piece_id=DBTL73IKPTJUDKGV3Q5AYZ5RAQXY35GYDM7KOAGNLZTLZS4K2XQQ, seg_pos={0 0}, upload=472443]: Done uploading piece. err: <nil> Dec 28 02:11:38.539662568 (538h42m38.410745613s) [noise=true , address=45.91.116.5:4300 , piece_id=UIRV35YALI6NT3ERFGLIBIFOXRZD6LQ2MEOCK2DBZ7VLADELGYZA, seg_pos={0 0}, upload=472443]: Done uploading piece. err: <nil> Dec 28 02:11:38.551702023 (538h42m38.422785068s) [noise=true , address=45.91.116.11:4120 , piece_id=MW2LUTCXDPAOHLKWFF4VLT4CNMHGZ4GLLC5JXQ52RIPX5Q23UUVA, seg_pos={0 0}, upload=472443]: Done uploading piece. err: <nil> Dec 28 02:11:39.141406751 (538h42m39.012489796s) [noise=true , address=45.91.116.8:4045 , piece_id=DK5FS77YXPC2YUKGLO2GHPCBRDAEIVDMK5ESJT4EDFWLLANNJVDA, seg_pos={0 0}, upload=472443]: Done uploading piece. err: <nil> Dec 28 02:11:39.190288981 (538h42m39.061372016s) [noise=true , address=45.91.116.4:4940 , piece_id=HSS7MBZFQNUAPGP6PMXNMP7KM7BNQ37GF6GSQUZGCDBUHMMJFIQA, seg_pos={0 0}, upload=472443]: Done uploading piece. err: <nil> Dec 28 02:11:39.190308687 (538h42m39.061391723s) [seg_pos={0 0}, upload=472443]: Segment reached optimal threshold of 13 pieces. Dec 28 02:11:39.190511303 (538h42m39.061594349s) [noise=true , address=77.90.31.19:4844 , piece_id=WSJRGDLQFKKKR4BTIB5CKZFIL4SC33PEIKNLJ75LDZQK475OBPIA, seg_pos={0 0}, upload=472443]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF Dec 28 02:11:39.190521493 (538h42m39.061604538s) [noise=true , address=77.90.31.19:4844 , piece_id=WSJRGDLQFKKKR4BTIB5CKZFIL4SC33PEIKNLJ75LDZQK475OBPIA, seg_pos={0 0}, upload=472443]: Long tail cancellation due to optimal threshold Dec 28 02:11:39.190660537 (538h42m39.061743623s) [noise=true , address=145.79.227.49:4337 , piece_id=MRQYYVRT6EOINIS7L4MMB7HMUTK6U3WRR7HSFX25NSNRJ4KKRPOQ, seg_pos={0 0}, upload=472443]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE): protocol: expected piece hash; context canceled; EOF Dec 28 02:11:39.190771799 (538h42m39.061854864s) [noise=true , address=145.79.227.49:4337 , piece_id=MRQYYVRT6EOINIS7L4MMB7HMUTK6U3WRR7HSFX25NSNRJ4KKRPOQ, seg_pos={0 0}, upload=472443]: Long tail cancellation due to optimal threshold Dec 28 02:11:39.190834849 (538h42m39.061917894s) [noise=true , address=145.79.227.33:4870 , piece_id=PTE3CQ4VJXJ6YHVOR72UACT22XQVCBEAI7VYFAKZQT3TLTUJ4OCQ, seg_pos={0 0}, upload=472443]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 28 02:11:39.190849658 (538h42m39.061932693s) [noise=true , address=145.79.227.33:4870 , piece_id=PTE3CQ4VJXJ6YHVOR72UACT22XQVCBEAI7VYFAKZQT3TLTUJ4OCQ, seg_pos={0 0}, upload=472443]: Long tail cancellation due to optimal threshold Dec 28 02:11:39.190878502 (538h42m39.061961537s) [seg_pos={0 0}, upload=472443]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 02:11:39.190990967 (538h42m39.062074012s) [upload=472443]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:11:39.190998141 (538h42m39.062081176s) [upload=472443]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:11:39.489285610 (538h42m39.360368645s) [upload=472443]: Done uploading segments... Dec 28 02:11:39.489303233 (538h42m39.360386268s) [upload=472443]: Upload finished. err: <nil> Duration: 6.055945764s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................................................X........] duration: 3.496184468s trace: https://jaeger.dev.storj.io/trace/7fc142ad3d711e34 FAIL (2025-12-28 02:12:15 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.496210147s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 32 out of 60 is failed (threshold=3) [.XX....X.XX.XXXXX..X.XXX.X.XX.XX.X..X.X.X.X.X..X...XX..X.XXX] duration: 30.004398502s trace: https://jaeger.dev.storj.io/trace/19d45d58c66aa194 FAIL (2025-12-28 02:12:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: DK5FS77YXPC2YUKGLO2GHPCBRDAEIVDMK5ESJT4EDFWLLANNJVDA) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: 5XJ7LJPLRR7ALKWWHN3P2QE43QLD37I27GEL6ME4LU4GK462XZ3A) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: UIRV35YALI6NT3ERFGLIBIFOXRZD6LQ2MEOCK2DBZ7VLADELGYZA) context deadline exceeded ======LOGS====== Dec 28 02:11:35.885960014 (538h42m35.757043059s) [upload=472443]: Starting upload Dec 28 02:11:35.886044255 (538h42m35.75712731s) [upload=472443]: Done starting upload Dec 28 02:11:35.886131862 (538h42m35.757214907s) [upload=472443]: Uploading segments... Dec 28 02:11:35.886263483 (538h42m35.757346558s) [upload=472443]: Got next segment. Inline: false Dec 28 02:11:35.886307867 (538h42m35.757390902s) [upload=472443]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 02:11:35.886313287 (538h42m35.757396322s) [upload=472443]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 02:11:36.141046383 (538h42m36.012129418s) [seg_pos={0 0}, upload=472443]: Begin upload segment... Dec 28 02:11:36.141102439 (538h42m36.012185474s) [seg_pos={0 0}, upload=472443]: Done begin upload segment. Dec 28 02:11:36.141146322 (538h42m36.012229358s) [noise=true , address=145.79.227.13:4803 , piece_id=P6EACXBYOYLAFCKLIUQUUGXLONNC2KVQBRKN2EULFAX3RDVRYIDA, seg_pos={0 0}, upload=472443]: Uploading piece... Dec 28 02:11:36.141172863 (538h42m36.012255908s) [noise=true , address=45.91.116.14:4189 , piece_id=DBTL73IKPTJUDKGV3Q5AYZ5RAQXY35GYDM7KOAGNLZTLZS4K2XQQ, seg_pos={0 0}, upload=472443]: Uploading piece... Dec 28 02:11:36.141621868 (538h42m36.012704903s) [noise=true , address=145.79.227.42:4122 , piece_id=GBQ2XR2WUZNLKB6U7WWR33V22JXWSWJYCKJQ2UAQXKZ6JIPYZRIA, seg_pos={0 0}, upload=472443]: Uploading piece... Dec 28 02:11:36.142095601 (538h42m36.013178646s) [noise=true , address=145.79.227.33:4870 , piece_id=PTE3CQ4VJXJ6YHVOR72UACT22XQVCBEAI7VYFAKZQT3TLTUJ4OCQ, seg_pos={0 0}, upload=472443]: Uploading piece... Dec 28 02:11:36.142477468 (538h42m36.013560503s) [noise=true , address=145.79.227.52:4445 , piece_id=LSG3JJMDC5UX753JEDLQE3TZNBRS2JWAS3X3NQRODR6WS3YZS6ZQ, seg_pos={0 0}, upload=472443]: Uploading piece... Dec 28 02:11:36.142842683 (538h42m36.013925718s) [noise=true , address=145.79.227.49:4337 , piece_id=MRQYYVRT6EOINIS7L4MMB7HMUTK6U3WRR7HSFX25NSNRJ4KKRPOQ, seg_pos={0 0}, upload=472443]: Uploading piece... Dec 28 02:11:36.143139619 (538h42m36.014222654s) [noise=true , address=77.90.31.19:4844 , piece_id=WSJRGDLQFKKKR4BTIB5CKZFIL4SC33PEIKNLJ75LDZQK475OBPIA, seg_pos={0 0}, upload=472443]: Uploading piece... Dec 28 02:11:36.143477782 (538h42m36.014560827s) [noise=true , address=45.91.116.11:4120 , piece_id=MW2LUTCXDPAOHLKWFF4VLT4CNMHGZ4GLLC5JXQ52RIPX5Q23UUVA, seg_pos={0 0}, upload=472443]: Uploading piece... Dec 28 02:11:36.143821106 (538h42m36.014904141s) [noise=true , address=45.91.116.7:4009 , piece_id=5XJ7LJPLRR7ALKWWHN3P2QE43QLD37I27GEL6ME4LU4GK462XZ3A, seg_pos={0 0}, upload=472443]: Uploading piece... Dec 28 02:11:36.145093028 (538h42m36.016176063s) [noise=true , address=77.90.31.17:4773 , piece_id=4HOJERKNNECSJXZMTPWSX3YAIOA5E32XASJ3Q5VESE7XSOSA4C6Q, seg_pos={0 0}, upload=472443]: Uploading piece... Dec 28 02:11:36.146411950 (538h42m36.017494985s) [noise=true , address=45.91.116.4:4940 , piece_id=HSS7MBZFQNUAPGP6PMXNMP7KM7BNQ37GF6GSQUZGCDBUHMMJFIQA, seg_pos={0 0}, upload=472443]: Uploading piece... Dec 28 02:11:36.147915813 (538h42m36.018998848s) [noise=true , address=45.91.116.5:4300 , piece_id=UIRV35YALI6NT3ERFGLIBIFOXRZD6LQ2MEOCK2DBZ7VLADELGYZA, seg_pos={0 0}, upload=472443]: Uploading piece... Dec 28 02:11:36.149262688 (538h42m36.020345733s) [noise=true , address=145.79.227.51:4411 , piece_id=OOSIWGVDPTBB3LOWCM4W25T2KMBMQVLR3ROGGBGP7OQO4OEYGXOA, seg_pos={0 0}, upload=472443]: Uploading piece... Dec 28 02:11:36.150634360 (538h42m36.021717395s) [noise=true , address=77.90.31.13:4629 , piece_id=QLKSHQNI5N4ZZTF2NSUCDJA7GUWHYGFJBPQLAAAVGW6CMZGDN5EQ, seg_pos={0 0}, upload=472443]: Uploading piece... Dec 28 02:11:36.151801142 (538h42m36.022884177s) [noise=true , address=45.91.116.8:4045 , piece_id=DK5FS77YXPC2YUKGLO2GHPCBRDAEIVDMK5ESJT4EDFWLLANNJVDA, seg_pos={0 0}, upload=472443]: Uploading piece... Dec 28 02:11:36.152647123 (538h42m36.023730158s) [noise=true , address=145.79.227.9:4367 , piece_id=53WBZ2PZBQEZRJR5XKT47526FPRKKPDL434PPHTINPDJFTP5X7WQ, seg_pos={0 0}, upload=472443]: Uploading piece... Dec 28 02:11:37.862878909 (538h42m37.733961944s) [upload=472443]: Next returned nil segment Dec 28 02:11:37.862897123 (538h42m37.733980168s) [upload=472443]: Waiting for error group managing segments... Dec 28 02:11:38.154407318 (538h42m38.025490363s) [noise=true , address=145.79.227.9:4367 , piece_id=53WBZ2PZBQEZRJR5XKT47526FPRKKPDL434PPHTINPDJFTP5X7WQ, seg_pos={0 0}, upload=472443]: Done uploading piece. err: <nil> Dec 28 02:11:38.155631699 (538h42m38.026714734s) [noise=true , address=145.79.227.52:4445 , piece_id=LSG3JJMDC5UX753JEDLQE3TZNBRS2JWAS3X3NQRODR6WS3YZS6ZQ, seg_pos={0 0}, upload=472443]: Done uploading piece. err: <nil> Dec 28 02:11:38.160119925 (538h42m38.03120297s) [noise=true , address=145.79.227.13:4803 , piece_id=P6EACXBYOYLAFCKLIUQUUGXLONNC2KVQBRKN2EULFAX3RDVRYIDA, seg_pos={0 0}, upload=472443]: Done uploading piece. err: <nil> Dec 28 02:11:38.171207337 (538h42m38.042290372s) [noise=true , address=145.79.227.42:4122 , piece_id=GBQ2XR2WUZNLKB6U7WWR33V22JXWSWJYCKJQ2UAQXKZ6JIPYZRIA, seg_pos={0 0}, upload=472443]: Done uploading piece. err: <nil> Dec 28 02:11:38.175984393 (538h42m38.047067478s) [noise=true , address=145.79.227.51:4411 , piece_id=OOSIWGVDPTBB3LOWCM4W25T2KMBMQVLR3ROGGBGP7OQO4OEYGXOA, seg_pos={0 0}, upload=472443]: Done uploading piece. err: <nil> Dec 28 02:11:38.338320298 (538h42m38.209403333s) [noise=true , address=77.90.31.13:4629 , piece_id=QLKSHQNI5N4ZZTF2NSUCDJA7GUWHYGFJBPQLAAAVGW6CMZGDN5EQ, seg_pos={0 0}, upload=472443]: Done uploading piece. err: <nil> Dec 28 02:11:38.375693494 (538h42m38.246776539s) [noise=true , address=45.91.116.7:4009 , piece_id=5XJ7LJPLRR7ALKWWHN3P2QE43QLD37I27GEL6ME4LU4GK462XZ3A, seg_pos={0 0}, upload=472443]: Done uploading piece. err: <nil> Dec 28 02:11:38.501546308 (538h42m38.372629353s) [noise=true , address=77.90.31.17:4773 , piece_id=4HOJERKNNECSJXZMTPWSX3YAIOA5E32XASJ3Q5VESE7XSOSA4C6Q, seg_pos={0 0}, upload=472443]: Done uploading piece. err: <nil> Dec 28 02:11:38.502263985 (538h42m38.37334702s) [noise=true , address=45.91.116.14:4189 , piece_id=DBTL73IKPTJUDKGV3Q5AYZ5RAQXY35GYDM7KOAGNLZTLZS4K2XQQ, seg_pos={0 0}, upload=472443]: Done uploading piece. err: <nil> Dec 28 02:11:38.539662568 (538h42m38.410745613s) [noise=true , address=45.91.116.5:4300 , piece_id=UIRV35YALI6NT3ERFGLIBIFOXRZD6LQ2MEOCK2DBZ7VLADELGYZA, seg_pos={0 0}, upload=472443]: Done uploading piece. err: <nil> Dec 28 02:11:38.551702023 (538h42m38.422785068s) [noise=true , address=45.91.116.11:4120 , piece_id=MW2LUTCXDPAOHLKWFF4VLT4CNMHGZ4GLLC5JXQ52RIPX5Q23UUVA, seg_pos={0 0}, upload=472443]: Done uploading piece. err: <nil> Dec 28 02:11:39.141406751 (538h42m39.012489796s) [noise=true , address=45.91.116.8:4045 , piece_id=DK5FS77YXPC2YUKGLO2GHPCBRDAEIVDMK5ESJT4EDFWLLANNJVDA, seg_pos={0 0}, upload=472443]: Done uploading piece. err: <nil> Dec 28 02:11:39.190288981 (538h42m39.061372016s) [noise=true , address=45.91.116.4:4940 , piece_id=HSS7MBZFQNUAPGP6PMXNMP7KM7BNQ37GF6GSQUZGCDBUHMMJFIQA, seg_pos={0 0}, upload=472443]: Done uploading piece. err: <nil> Dec 28 02:11:39.190308687 (538h42m39.061391723s) [seg_pos={0 0}, upload=472443]: Segment reached optimal threshold of 13 pieces. Dec 28 02:11:39.190511303 (538h42m39.061594349s) [noise=true , address=77.90.31.19:4844 , piece_id=WSJRGDLQFKKKR4BTIB5CKZFIL4SC33PEIKNLJ75LDZQK475OBPIA, seg_pos={0 0}, upload=472443]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF Dec 28 02:11:39.190521493 (538h42m39.061604538s) [noise=true , address=77.90.31.19:4844 , piece_id=WSJRGDLQFKKKR4BTIB5CKZFIL4SC33PEIKNLJ75LDZQK475OBPIA, seg_pos={0 0}, upload=472443]: Long tail cancellation due to optimal threshold Dec 28 02:11:39.190660537 (538h42m39.061743623s) [noise=true , address=145.79.227.49:4337 , piece_id=MRQYYVRT6EOINIS7L4MMB7HMUTK6U3WRR7HSFX25NSNRJ4KKRPOQ, seg_pos={0 0}, upload=472443]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE): protocol: expected piece hash; context canceled; EOF Dec 28 02:11:39.190771799 (538h42m39.061854864s) [noise=true , address=145.79.227.49:4337 , piece_id=MRQYYVRT6EOINIS7L4MMB7HMUTK6U3WRR7HSFX25NSNRJ4KKRPOQ, seg_pos={0 0}, upload=472443]: Long tail cancellation due to optimal threshold Dec 28 02:11:39.190834849 (538h42m39.061917894s) [noise=true , address=145.79.227.33:4870 , piece_id=PTE3CQ4VJXJ6YHVOR72UACT22XQVCBEAI7VYFAKZQT3TLTUJ4OCQ, seg_pos={0 0}, upload=472443]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 28 02:11:39.190849658 (538h42m39.061932693s) [noise=true , address=145.79.227.33:4870 , piece_id=PTE3CQ4VJXJ6YHVOR72UACT22XQVCBEAI7VYFAKZQT3TLTUJ4OCQ, seg_pos={0 0}, upload=472443]: Long tail cancellation due to optimal threshold Dec 28 02:11:39.190878502 (538h42m39.061961537s) [seg_pos={0 0}, upload=472443]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 02:11:39.190990967 (538h42m39.062074012s) [upload=472443]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:11:39.190998141 (538h42m39.062081176s) [upload=472443]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:11:39.489285610 (538h42m39.360368645s) [upload=472443]: Done uploading segments... Dec 28 02:11:39.489303233 (538h42m39.360386268s) [upload=472443]: Upload finished. err: <nil> Duration: 30.004533018s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 51.829365642s trace: https://jaeger.dev.storj.io/trace/12a1d059b6f75af7 FAIL (2025-12-28 02:12:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1.145865281 (538h33m41.016948316s) [noise=true , address=77.90.31.10:4551 , piece_id=KBPO4MUDASHKMPRNJELQWLNMJNQBB3PZUOVNCEPP2CYVHU6X2PPA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.146408725 (538h33m41.01749177s) [noise=true , address=145.79.227.15:4908 , piece_id=I2FSDNVYBFZKRXZNUHH2APY7N2PI7DPAKFKGXRJW2UFU2PBKSSNQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.146778139 (538h33m41.017861184s) [noise=true , address=145.79.227.43:4161 , piece_id=ZN3T6JSPHJ57X4YYG5MHRJYYOFPH2T2CSGJ34MJQU55HDC7CMOXQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.147137874 (538h33m41.018220919s) [noise=true , address=77.90.31.21:4882 , piece_id=WQAGQTGLU5HRKEET36W34MQP4NPGSVCA4QZ6YEGVAJWBBPLZOUQQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.147426784 (538h33m41.018509819s) [noise=true , address=145.79.227.35:4503 , piece_id=MDYJ2TROT4FHLX4XHKHEQACC7ZXHZCQ5MUPJL5EDS3F2YMXSUGUA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.147466109 (538h33m41.018549154s) [noise=true , address=77.90.31.15:4730 , piece_id=EN7IGRJZ2WFN4HXJPMNLLXDUZPIN7KDUGKPPLJH7CSA4SDOUM2JA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.147817368 (538h33m41.018900413s) [noise=true , address=77.90.31.18:4833 , piece_id=3E57LWLI3WHROFGMXVZ7LHDHO5O7453LWQXA3OEKM52B2NIYM2JA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.147860690 (538h33m41.018943745s) [noise=true , address=145.79.227.44:4212 , piece_id=EFTTQM6323MUQ5WTGY6DFBTF3TGRM6XZWXIGT6DM4VS7GRGM4C2Q, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148163427 (538h33m41.019246472s) [noise=true , address=77.90.31.19:4862 , piece_id=PSBBBSGY7THVLMXJQCADIE7VYZ33OIIDVAEQWG3RWMNA6RSKELNQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148189276 (538h33m41.019272321s) [noise=true , address=77.90.31.10:4524 , piece_id=X2VCBS4PTPPWI6U64T4FVIULRLCDY3XUOJTXJNP34OHSO54ONIWA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148496531 (538h33m41.019579566s) [noise=true , address=145.79.227.23:4556 , piece_id=AKBXRSU5PM5QLHGHE3VXTJ3ZH23BECPGBLQSVZK6XVIYJGVDXT4Q, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148527380 (538h33m41.019610425s) [noise=true , address=145.79.227.33:4899 , piece_id=JFEXXK22EATV7UUH7LLXTFEPIUBW7EWSNYFXLE4AO5F6SLYFTGBA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148801512 (538h33m41.019884547s) [noise=true , address=77.90.31.14:4672 , piece_id=BZG2Y5X47J2WVFF5LBHEJZNUD2OQ3NOTVPTCAVT4TLFQB5FWESLA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148894920 (538h33m41.019977965s) [noise=true , address=5.231.73.5:4016 , piece_id=YCNKQEKJQSJHKKGCYWLOLEBKIVPJHZIJ6TYEG6DF4TRXMZ3ZTTIQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.149118806 (538h33m41.020201851s) [noise=true , address=145.79.227.47:4295 , piece_id=OM7B75MCPM7L3WP5MLQ7MAVBZ2NJUSISOWD56RSWI2AV2OWVDRTA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.149274993 (538h33m41.020358038s) [noise=true , address=145.79.227.50:4400 , piece_id=HZAT7YAVZYWK52FE4TIC6OA37U4VX5B35H55WMBO3MMWNIKTJLJQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.149625842 (538h33m41.020708877s) [noise=true , address=145.79.227.34:4915 , piece_id=RMXYZGGJA5EPJLQD6SQG677HFLOQWRY2HFTGEOICYAQG6QR557RQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:42.618098549 (538h33m42.489181584s) [upload=472313]: Next returned nil segment Dec 28 02:02:42.618120911 (538h33m42.489203946s) [upload=472313]: Waiting for error group managing segments... Dec 28 02:02:43.021704832 (538h33m42.892787877s) [noise=true , address=145.79.227.23:4556 , piece_id=AKBXRSU5PM5QLHGHE3VXTJ3ZH23BECPGBLQSVZK6XVIYJGVDXT4Q, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.027980492 (538h33m42.899063537s) [noise=true , address=145.79.227.50:4400 , piece_id=HZAT7YAVZYWK52FE4TIC6OA37U4VX5B35H55WMBO3MMWNIKTJLJQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.043329239 (538h33m42.914412284s) [noise=true , address=145.79.227.15:4908 , piece_id=I2FSDNVYBFZKRXZNUHH2APY7N2PI7DPAKFKGXRJW2UFU2PBKSSNQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.056308314 (538h33m42.927391349s) [noise=true , address=145.79.227.49:4347 , piece_id=WURIHGQASBLSVLZJVM5EJHWD7U4VFLLPGJIFYNOK6ALA5LYEQDSA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.074628196 (538h33m42.945711252s) [noise=true , address=145.79.227.47:4295 , piece_id=OM7B75MCPM7L3WP5MLQ7MAVBZ2NJUSISOWD56RSWI2AV2OWVDRTA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.080495740 (538h33m42.951578775s) [noise=true , address=5.231.73.5:4016 , piece_id=YCNKQEKJQSJHKKGCYWLOLEBKIVPJHZIJ6TYEG6DF4TRXMZ3ZTTIQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.193955176 (538h33m43.065038241s) [noise=true , address=77.90.31.10:4551 , piece_id=KBPO4MUDASHKMPRNJELQWLNMJNQBB3PZUOVNCEPP2CYVHU6X2PPA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.301513865 (538h33m43.1725969s) [noise=true , address=77.90.31.15:4730 , piece_id=EN7IGRJZ2WFN4HXJPMNLLXDUZPIN7KDUGKPPLJH7CSA4SDOUM2JA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.305020622 (538h33m43.176103657s) [noise=true , address=77.90.31.19:4862 , piece_id=PSBBBSGY7THVLMXJQCADIE7VYZ33OIIDVAEQWG3RWMNA6RSKELNQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.312187619 (538h33m43.183270654s) [noise=true , address=77.90.31.14:4672 , piece_id=BZG2Y5X47J2WVFF5LBHEJZNUD2OQ3NOTVPTCAVT4TLFQB5FWESLA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.319277390 (538h33m43.190360425s) [noise=true , address=77.90.31.21:4882 , piece_id=WQAGQTGLU5HRKEET36W34MQP4NPGSVCA4QZ6YEGVAJWBBPLZOUQQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.948916959 (538h33m43.820000004s) [noise=true , address=145.79.227.34:4915 , piece_id=RMXYZGGJA5EPJLQD6SQG677HFLOQWRY2HFTGEOICYAQG6QR557RQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.989823331 (538h33m43.860906366s) [noise=true , address=145.79.227.44:4212 , piece_id=EFTTQM6323MUQ5WTGY6DFBTF3TGRM6XZWXIGT6DM4VS7GRGM4C2Q, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:44.123218949 (538h33m43.994301984s) [noise=true , address=145.79.227.43:4161 , piece_id=ZN3T6JSPHJ57X4YYG5MHRJYYOFPH2T2CSGJ34MJQU55HDC7CMOXQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:44.123238386 (538h33m43.994321421s) [seg_pos={0 0}, upload=472313]: Segment reached optimal threshold of 14 pieces. Dec 28 02:02:44.123431754 (538h33m43.994514789s) [noise=true , address=145.79.227.33:4899 , piece_id=JFEXXK22EATV7UUH7LLXTFEPIUBW7EWSNYFXLE4AO5F6SLYFTGBA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b4pZVQRnjcQNn45VTgdsz5fpjYPMAzSXGR5KJ5BP9XHMkRnRw): protocol: expected piece hash; context canceled; EOF Dec 28 02:02:44.123437586 (538h33m43.994520621s) [noise=true , address=145.79.227.33:4899 , piece_id=JFEXXK22EATV7UUH7LLXTFEPIUBW7EWSNYFXLE4AO5F6SLYFTGBA, seg_pos={0 0}, upload=472313]: Long tail cancellation due to optimal threshold Dec 28 02:02:44.123631064 (538h33m43.994714109s) [noise=true , address=77.90.31.18:4833 , piece_id=3E57LWLI3WHROFGMXVZ7LHDHO5O7453LWQXA3OEKM52B2NIYM2JA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FbVbo8kGy1aaFyCk2noiXNUJHtosFJHFykBMRecGd5Kj5gjEp): protocol: expected piece hash; context canceled; EOF Dec 28 02:02:44.123675267 (538h33m43.994758313s) [noise=true , address=77.90.31.18:4833 , piece_id=3E57LWLI3WHROFGMXVZ7LHDHO5O7453LWQXA3OEKM52B2NIYM2JA, seg_pos={0 0}, upload=472313]: Long tail cancellation due to optimal threshold Dec 28 02:02:44.123774066 (538h33m43.994857101s) [noise=true , address=77.90.31.10:4524 , piece_id=X2VCBS4PTPPWI6U64T4FVIULRLCDY3XUOJTXJNP34OHSO54ONIWA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe): protocol: expected piece hash; context canceled; EOF Dec 28 02:02:44.123781761 (538h33m43.994864796s) [noise=true , address=77.90.31.10:4524 , piece_id=X2VCBS4PTPPWI6U64T4FVIULRLCDY3XUOJTXJNP34OHSO54ONIWA, seg_pos={0 0}, upload=472313]: Long tail cancellation due to optimal threshold Dec 28 02:02:44.123923341 (538h33m43.995006376s) [noise=true , address=145.79.227.35:4503 , piece_id=MDYJ2TROT4FHLX4XHKHEQACC7ZXHZCQ5MUPJL5EDS3F2YMXSUGUA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ctmSF1aMQasuXByFW8DppDantREmmWL2WUjmKGW5ZULVePyvm): protocol: expected piece hash; context canceled; EOF Dec 28 02:02:44.123928591 (538h33m43.995011636s) [noise=true , address=145.79.227.35:4503 , piece_id=MDYJ2TROT4FHLX4XHKHEQACC7ZXHZCQ5MUPJL5EDS3F2YMXSUGUA, seg_pos={0 0}, upload=472313]: Long tail cancellation due to optimal threshold Dec 28 02:02:44.123944731 (538h33m43.995027766s) [seg_pos={0 0}, upload=472313]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 02:02:44.124017179 (538h33m43.995100214s) [upload=472313]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:02:44.124022469 (538h33m43.995105504s) [upload=472313]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:02:44.409047274 (538h33m44.280130349s) [upload=472313]: Done uploading segments... Dec 28 02:02:44.409073354 (538h33m44.280156389s) [upload=472313]: Upload finished. err: <nil> Duration: 51.829380682s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................................................X.......] duration: 3.751586749s trace: https://jaeger.dev.storj.io/trace/7fc142ad3d711e34 FAIL (2025-12-28 02:11:48 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.751613689s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 32 out of 60 is failed (threshold=3) [X.XX....X.XX.XXXXX..X.XXX.X.XX.XX.X..X.X.X.X.X..X...XX..X.XX] duration: 30.003542748s trace: https://jaeger.dev.storj.io/trace/515b7a6df070b47e FAIL (2025-12-28 02:11:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: GE5L5MUMZ4LB2UDWB6MGNLMHPV2ORLSVPU732K2437QLMRY5ZMJA) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: SBEMMCVA4BHCSSRRAKNNUNVME2BMSW2Q7XYBXK6ZEISRWNCZRJUA) context deadline exceeded ======LOGS====== ments... Dec 28 02:10:44.501865108 (538h41m44.372948143s) [upload=472427]: Got next segment. Inline: false Dec 28 02:10:44.501887852 (538h41m44.372970887s) [upload=472427]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 02:10:44.501891729 (538h41m44.372974764s) [upload=472427]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 02:10:44.744986593 (538h41m44.616069638s) [seg_pos={0 0}, upload=472427]: Begin upload segment... Dec 28 02:10:44.745048410 (538h41m44.616131455s) [seg_pos={0 0}, upload=472427]: Done begin upload segment. Dec 28 02:10:44.745076623 (538h41m44.616159658s) [noise=true , address=77.90.31.14:4663 , piece_id=GE5L5MUMZ4LB2UDWB6MGNLMHPV2ORLSVPU732K2437QLMRY5ZMJA, seg_pos={0 0}, upload=472427]: Uploading piece... Dec 28 02:10:44.745098435 (538h41m44.61618147s) [noise=true , address=45.91.116.8:4046 , piece_id=YZI5LW6JT6NT4IIURSBU22SWYIS5NJBCUIAX7QPJHYWNAA3AB3MQ, seg_pos={0 0}, upload=472427]: Uploading piece... Dec 28 02:10:44.745146486 (538h41m44.616229532s) [noise=true , address=45.91.116.5:4300 , piece_id=SBEMMCVA4BHCSSRRAKNNUNVME2BMSW2Q7XYBXK6ZEISRWNCZRJUA, seg_pos={0 0}, upload=472427]: Uploading piece... Dec 28 02:10:44.745581966 (538h41m44.616665001s) [noise=true , address=145.79.227.31:4797 , piece_id=LA3OSEVIEXQ7ZV7EIKVDW356RYNR7ULFLU2WYW7TGT3JKV4QGS5Q, seg_pos={0 0}, upload=472427]: Uploading piece... Dec 28 02:10:44.746041541 (538h41m44.617124586s) [noise=true , address=45.91.116.6:4973 , piece_id=IFST4HCE4V67HOC575L2UZWB4KAYXIXLK2ZT4DXHQNYWGHLB5HWQ, seg_pos={0 0}, upload=472427]: Uploading piece... Dec 28 02:10:44.746447353 (538h41m44.617530388s) [noise=true , address=145.79.227.16:4911 , piece_id=I5XY6ZP4YMBGKJYN4JA34T7RW22VBBNHMF4EDJSDGOQBDHVPUPQQ, seg_pos={0 0}, upload=472427]: Uploading piece... Dec 28 02:10:44.746916347 (538h41m44.617999392s) [noise=true , address=145.79.227.18:4006 , piece_id=DTHV66CWJFUVRREVTC6N2CLQTP42NXD5PO37YRHXBNBTGNCMMACA, seg_pos={0 0}, upload=472427]: Uploading piece... Dec 28 02:10:44.747415166 (538h41m44.618498211s) [noise=true , address=145.79.227.11:4731 , piece_id=S33VOZ63OQFBEQL5MTAQ3VPA4OAX7JI5OG53CSJKNVUT6OAH5NJA, seg_pos={0 0}, upload=472427]: Uploading piece... Dec 28 02:10:44.748581478 (538h41m44.619664523s) [noise=true , address=45.91.116.11:4117 , piece_id=TTFSK3QCSUR4UKSO7W2OTMSULDQPOIJERQ7BYF2GPG4KPAVUVR5Q, seg_pos={0 0}, upload=472427]: Uploading piece... Dec 28 02:10:44.749690700 (538h41m44.620773745s) [noise=true , address=45.91.116.12:4227 , piece_id=EV2EGV5MGAU6O3J53IVDXKSWY5XV65B5GIRXPGABPNVBQRQHX2OQ, seg_pos={0 0}, upload=472427]: Uploading piece... Dec 28 02:10:44.750397335 (538h41m44.62148038s) [noise=true , address=145.79.227.47:4265 , piece_id=7ZGKZPUUREAJ6TRVDCUDGBG6V5RYM6LYY3GH5GGDWGJ4VKW3HTNA, seg_pos={0 0}, upload=472427]: Uploading piece... Dec 28 02:10:44.752078306 (538h41m44.623161351s) [noise=true , address=145.79.227.2:4114 , piece_id=VC3EJZE4ITBMYZHQJLYHSV3QLSP4MCOJ2PYTFPDGB23CBAVHOFJA, seg_pos={0 0}, upload=472427]: Uploading piece... Dec 28 02:10:44.753684023 (538h41m44.624767068s) [noise=true , address=145.79.227.49:4339 , piece_id=5SSR642TBCB7UAUAI6OF4E6HX4E5N4PNSKY67H3EYCDHQSLZJNWQ, seg_pos={0 0}, upload=472427]: Uploading piece... Dec 28 02:10:44.755286545 (538h41m44.62636959s) [noise=true , address=145.79.227.3:4151 , piece_id=FXR2ERVSBEOHQZCFFCNLYTMOKTGHC76MIDLVYQHZY7TZ65QUIZAQ, seg_pos={0 0}, upload=472427]: Uploading piece... Dec 28 02:10:44.756157152 (538h41m44.627240197s) [noise=true , address=145.79.227.38:4979 , piece_id=TKCXF43KOOAQ37S5R5OLOTOBIUY4AYSQANU6DOU42DMYSDTQOBQA, seg_pos={0 0}, upload=472427]: Uploading piece... Dec 28 02:10:44.757275984 (538h41m44.628359029s) [noise=true , address=45.91.116.13:4153 , piece_id=5EDDY2YTV7NFA2J7PVAM4F7VOWQXXTYTWYCWF3MRXFAA5TPWIEZA, seg_pos={0 0}, upload=472427]: Uploading piece... Dec 28 02:10:46.466344125 (538h41m46.337427161s) [upload=472427]: Next returned nil segment Dec 28 02:10:46.466357972 (538h41m46.337441008s) [upload=472427]: Waiting for error group managing segments... Dec 28 02:10:46.752799435 (538h41m46.62388247s) [noise=true , address=145.79.227.18:4006 , piece_id=DTHV66CWJFUVRREVTC6N2CLQTP42NXD5PO37YRHXBNBTGNCMMACA, seg_pos={0 0}, upload=472427]: Done uploading piece. err: <nil> Dec 28 02:10:46.767670482 (538h41m46.638753517s) [noise=true , address=145.79.227.38:4979 , piece_id=TKCXF43KOOAQ37S5R5OLOTOBIUY4AYSQANU6DOU42DMYSDTQOBQA, seg_pos={0 0}, upload=472427]: Done uploading piece. err: <nil> Dec 28 02:10:46.775507234 (538h41m46.646590269s) [noise=true , address=145.79.227.47:4265 , piece_id=7ZGKZPUUREAJ6TRVDCUDGBG6V5RYM6LYY3GH5GGDWGJ4VKW3HTNA, seg_pos={0 0}, upload=472427]: Done uploading piece. err: <nil> Dec 28 02:10:46.776818421 (538h41m46.647901456s) [noise=true , address=145.79.227.31:4797 , piece_id=LA3OSEVIEXQ7ZV7EIKVDW356RYNR7ULFLU2WYW7TGT3JKV4QGS5Q, seg_pos={0 0}, upload=472427]: Done uploading piece. err: <nil> Dec 28 02:10:46.840399741 (538h41m46.711482776s) [noise=true , address=145.79.227.16:4911 , piece_id=I5XY6ZP4YMBGKJYN4JA34T7RW22VBBNHMF4EDJSDGOQBDHVPUPQQ, seg_pos={0 0}, upload=472427]: Done uploading piece. err: <nil> Dec 28 02:10:46.965087427 (538h41m46.836170462s) [noise=true , address=45.91.116.5:4300 , piece_id=SBEMMCVA4BHCSSRRAKNNUNVME2BMSW2Q7XYBXK6ZEISRWNCZRJUA, seg_pos={0 0}, upload=472427]: Done uploading piece. err: <nil> Dec 28 02:10:47.066070695 (538h41m46.93715373s) [noise=true , address=145.79.227.11:4731 , piece_id=S33VOZ63OQFBEQL5MTAQ3VPA4OAX7JI5OG53CSJKNVUT6OAH5NJA, seg_pos={0 0}, upload=472427]: Done uploading piece. err: <nil> Dec 28 02:10:47.077764843 (538h41m46.948847888s) [noise=true , address=45.91.116.6:4973 , piece_id=IFST4HCE4V67HOC575L2UZWB4KAYXIXLK2ZT4DXHQNYWGHLB5HWQ, seg_pos={0 0}, upload=472427]: Done uploading piece. err: <nil> Dec 28 02:10:47.081956706 (538h41m46.953039741s) [noise=true , address=145.79.227.49:4339 , piece_id=5SSR642TBCB7UAUAI6OF4E6HX4E5N4PNSKY67H3EYCDHQSLZJNWQ, seg_pos={0 0}, upload=472427]: Done uploading piece. err: <nil> Dec 28 02:10:47.084947920 (538h41m46.956030955s) [noise=true , address=45.91.116.11:4117 , piece_id=TTFSK3QCSUR4UKSO7W2OTMSULDQPOIJERQ7BYF2GPG4KPAVUVR5Q, seg_pos={0 0}, upload=472427]: Done uploading piece. err: <nil> Dec 28 02:10:47.091889259 (538h41m46.962972294s) [noise=true , address=77.90.31.14:4663 , piece_id=GE5L5MUMZ4LB2UDWB6MGNLMHPV2ORLSVPU732K2437QLMRY5ZMJA, seg_pos={0 0}, upload=472427]: Done uploading piece. err: <nil> Dec 28 02:10:47.112906359 (538h41m46.983989404s) [noise=true , address=45.91.116.8:4046 , piece_id=YZI5LW6JT6NT4IIURSBU22SWYIS5NJBCUIAX7QPJHYWNAA3AB3MQ, seg_pos={0 0}, upload=472427]: Done uploading piece. err: <nil> Dec 28 02:10:47.638605117 (538h41m47.509688162s) [noise=true , address=45.91.116.12:4227 , piece_id=EV2EGV5MGAU6O3J53IVDXKSWY5XV65B5GIRXPGABPNVBQRQHX2OQ, seg_pos={0 0}, upload=472427]: Done uploading piece. err: <nil> Dec 28 02:10:47.638627640 (538h41m47.509710675s) [seg_pos={0 0}, upload=472427]: Segment reached optimal threshold of 13 pieces. Dec 28 02:10:47.638918022 (538h41m47.510001057s) [noise=true , address=145.79.227.2:4114 , piece_id=VC3EJZE4ITBMYZHQJLYHSV3QLSP4MCOJ2PYTFPDGB23CBAVHOFJA, seg_pos={0 0}, upload=472427]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN): protocol: write tcp 188.166.225.170:54116->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:54116->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54116->145.79.227.2:4114: use of closed network connection Dec 28 02:10:47.638924686 (538h41m47.510007721s) [noise=true , address=145.79.227.2:4114 , piece_id=VC3EJZE4ITBMYZHQJLYHSV3QLSP4MCOJ2PYTFPDGB23CBAVHOFJA, seg_pos={0 0}, upload=472427]: Long tail cancellation due to optimal threshold Dec 28 02:10:47.639236570 (538h41m47.510319625s) [noise=true , address=145.79.227.3:4151 , piece_id=FXR2ERVSBEOHQZCFFCNLYTMOKTGHC76MIDLVYQHZY7TZ65QUIZAQ, seg_pos={0 0}, upload=472427]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: write tcp 188.166.225.170:51772->145.79.227.3:4151: use of closed network connection; write tcp 188.166.225.170:51772->145.79.227.3:4151: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51772->145.79.227.3:4151: use of closed network connection Dec 28 02:10:47.639259212 (538h41m47.510342257s) [noise=true , address=145.79.227.3:4151 , piece_id=FXR2ERVSBEOHQZCFFCNLYTMOKTGHC76MIDLVYQHZY7TZ65QUIZAQ, seg_pos={0 0}, upload=472427]: Long tail cancellation due to optimal threshold Dec 28 02:10:47.639329696 (538h41m47.510412731s) [noise=true , address=45.91.116.13:4153 , piece_id=5EDDY2YTV7NFA2J7PVAM4F7VOWQXXTYTWYCWF3MRXFAA5TPWIEZA, seg_pos={0 0}, upload=472427]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 28 02:10:47.639342251 (538h41m47.510425286s) [noise=true , address=45.91.116.13:4153 , piece_id=5EDDY2YTV7NFA2J7PVAM4F7VOWQXXTYTWYCWF3MRXFAA5TPWIEZA, seg_pos={0 0}, upload=472427]: Long tail cancellation due to optimal threshold Dec 28 02:10:47.639363431 (538h41m47.510446466s) [seg_pos={0 0}, upload=472427]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 02:10:47.639449074 (538h41m47.510532119s) [upload=472427]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:10:47.639454004 (538h41m47.510537039s) [upload=472427]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:10:47.931228641 (538h41m47.802311676s) [upload=472427]: Done uploading segments... Dec 28 02:10:47.931246816 (538h41m47.802329851s) [upload=472427]: Upload finished. err: <nil> Duration: 30.003561182s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 47.865543366s trace: https://jaeger.dev.storj.io/trace/12a1d059b6f75af7 FAIL (2025-12-28 02:11:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1.145865281 (538h33m41.016948316s) [noise=true , address=77.90.31.10:4551 , piece_id=KBPO4MUDASHKMPRNJELQWLNMJNQBB3PZUOVNCEPP2CYVHU6X2PPA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.146408725 (538h33m41.01749177s) [noise=true , address=145.79.227.15:4908 , piece_id=I2FSDNVYBFZKRXZNUHH2APY7N2PI7DPAKFKGXRJW2UFU2PBKSSNQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.146778139 (538h33m41.017861184s) [noise=true , address=145.79.227.43:4161 , piece_id=ZN3T6JSPHJ57X4YYG5MHRJYYOFPH2T2CSGJ34MJQU55HDC7CMOXQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.147137874 (538h33m41.018220919s) [noise=true , address=77.90.31.21:4882 , piece_id=WQAGQTGLU5HRKEET36W34MQP4NPGSVCA4QZ6YEGVAJWBBPLZOUQQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.147426784 (538h33m41.018509819s) [noise=true , address=145.79.227.35:4503 , piece_id=MDYJ2TROT4FHLX4XHKHEQACC7ZXHZCQ5MUPJL5EDS3F2YMXSUGUA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.147466109 (538h33m41.018549154s) [noise=true , address=77.90.31.15:4730 , piece_id=EN7IGRJZ2WFN4HXJPMNLLXDUZPIN7KDUGKPPLJH7CSA4SDOUM2JA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.147817368 (538h33m41.018900413s) [noise=true , address=77.90.31.18:4833 , piece_id=3E57LWLI3WHROFGMXVZ7LHDHO5O7453LWQXA3OEKM52B2NIYM2JA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.147860690 (538h33m41.018943745s) [noise=true , address=145.79.227.44:4212 , piece_id=EFTTQM6323MUQ5WTGY6DFBTF3TGRM6XZWXIGT6DM4VS7GRGM4C2Q, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148163427 (538h33m41.019246472s) [noise=true , address=77.90.31.19:4862 , piece_id=PSBBBSGY7THVLMXJQCADIE7VYZ33OIIDVAEQWG3RWMNA6RSKELNQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148189276 (538h33m41.019272321s) [noise=true , address=77.90.31.10:4524 , piece_id=X2VCBS4PTPPWI6U64T4FVIULRLCDY3XUOJTXJNP34OHSO54ONIWA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148496531 (538h33m41.019579566s) [noise=true , address=145.79.227.23:4556 , piece_id=AKBXRSU5PM5QLHGHE3VXTJ3ZH23BECPGBLQSVZK6XVIYJGVDXT4Q, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148527380 (538h33m41.019610425s) [noise=true , address=145.79.227.33:4899 , piece_id=JFEXXK22EATV7UUH7LLXTFEPIUBW7EWSNYFXLE4AO5F6SLYFTGBA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148801512 (538h33m41.019884547s) [noise=true , address=77.90.31.14:4672 , piece_id=BZG2Y5X47J2WVFF5LBHEJZNUD2OQ3NOTVPTCAVT4TLFQB5FWESLA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148894920 (538h33m41.019977965s) [noise=true , address=5.231.73.5:4016 , piece_id=YCNKQEKJQSJHKKGCYWLOLEBKIVPJHZIJ6TYEG6DF4TRXMZ3ZTTIQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.149118806 (538h33m41.020201851s) [noise=true , address=145.79.227.47:4295 , piece_id=OM7B75MCPM7L3WP5MLQ7MAVBZ2NJUSISOWD56RSWI2AV2OWVDRTA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.149274993 (538h33m41.020358038s) [noise=true , address=145.79.227.50:4400 , piece_id=HZAT7YAVZYWK52FE4TIC6OA37U4VX5B35H55WMBO3MMWNIKTJLJQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.149625842 (538h33m41.020708877s) [noise=true , address=145.79.227.34:4915 , piece_id=RMXYZGGJA5EPJLQD6SQG677HFLOQWRY2HFTGEOICYAQG6QR557RQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:42.618098549 (538h33m42.489181584s) [upload=472313]: Next returned nil segment Dec 28 02:02:42.618120911 (538h33m42.489203946s) [upload=472313]: Waiting for error group managing segments... Dec 28 02:02:43.021704832 (538h33m42.892787877s) [noise=true , address=145.79.227.23:4556 , piece_id=AKBXRSU5PM5QLHGHE3VXTJ3ZH23BECPGBLQSVZK6XVIYJGVDXT4Q, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.027980492 (538h33m42.899063537s) [noise=true , address=145.79.227.50:4400 , piece_id=HZAT7YAVZYWK52FE4TIC6OA37U4VX5B35H55WMBO3MMWNIKTJLJQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.043329239 (538h33m42.914412284s) [noise=true , address=145.79.227.15:4908 , piece_id=I2FSDNVYBFZKRXZNUHH2APY7N2PI7DPAKFKGXRJW2UFU2PBKSSNQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.056308314 (538h33m42.927391349s) [noise=true , address=145.79.227.49:4347 , piece_id=WURIHGQASBLSVLZJVM5EJHWD7U4VFLLPGJIFYNOK6ALA5LYEQDSA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.074628196 (538h33m42.945711252s) [noise=true , address=145.79.227.47:4295 , piece_id=OM7B75MCPM7L3WP5MLQ7MAVBZ2NJUSISOWD56RSWI2AV2OWVDRTA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.080495740 (538h33m42.951578775s) [noise=true , address=5.231.73.5:4016 , piece_id=YCNKQEKJQSJHKKGCYWLOLEBKIVPJHZIJ6TYEG6DF4TRXMZ3ZTTIQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.193955176 (538h33m43.065038241s) [noise=true , address=77.90.31.10:4551 , piece_id=KBPO4MUDASHKMPRNJELQWLNMJNQBB3PZUOVNCEPP2CYVHU6X2PPA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.301513865 (538h33m43.1725969s) [noise=true , address=77.90.31.15:4730 , piece_id=EN7IGRJZ2WFN4HXJPMNLLXDUZPIN7KDUGKPPLJH7CSA4SDOUM2JA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.305020622 (538h33m43.176103657s) [noise=true , address=77.90.31.19:4862 , piece_id=PSBBBSGY7THVLMXJQCADIE7VYZ33OIIDVAEQWG3RWMNA6RSKELNQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.312187619 (538h33m43.183270654s) [noise=true , address=77.90.31.14:4672 , piece_id=BZG2Y5X47J2WVFF5LBHEJZNUD2OQ3NOTVPTCAVT4TLFQB5FWESLA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.319277390 (538h33m43.190360425s) [noise=true , address=77.90.31.21:4882 , piece_id=WQAGQTGLU5HRKEET36W34MQP4NPGSVCA4QZ6YEGVAJWBBPLZOUQQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.948916959 (538h33m43.820000004s) [noise=true , address=145.79.227.34:4915 , piece_id=RMXYZGGJA5EPJLQD6SQG677HFLOQWRY2HFTGEOICYAQG6QR557RQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.989823331 (538h33m43.860906366s) [noise=true , address=145.79.227.44:4212 , piece_id=EFTTQM6323MUQ5WTGY6DFBTF3TGRM6XZWXIGT6DM4VS7GRGM4C2Q, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:44.123218949 (538h33m43.994301984s) [noise=true , address=145.79.227.43:4161 , piece_id=ZN3T6JSPHJ57X4YYG5MHRJYYOFPH2T2CSGJ34MJQU55HDC7CMOXQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:44.123238386 (538h33m43.994321421s) [seg_pos={0 0}, upload=472313]: Segment reached optimal threshold of 14 pieces. Dec 28 02:02:44.123431754 (538h33m43.994514789s) [noise=true , address=145.79.227.33:4899 , piece_id=JFEXXK22EATV7UUH7LLXTFEPIUBW7EWSNYFXLE4AO5F6SLYFTGBA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b4pZVQRnjcQNn45VTgdsz5fpjYPMAzSXGR5KJ5BP9XHMkRnRw): protocol: expected piece hash; context canceled; EOF Dec 28 02:02:44.123437586 (538h33m43.994520621s) [noise=true , address=145.79.227.33:4899 , piece_id=JFEXXK22EATV7UUH7LLXTFEPIUBW7EWSNYFXLE4AO5F6SLYFTGBA, seg_pos={0 0}, upload=472313]: Long tail cancellation due to optimal threshold Dec 28 02:02:44.123631064 (538h33m43.994714109s) [noise=true , address=77.90.31.18:4833 , piece_id=3E57LWLI3WHROFGMXVZ7LHDHO5O7453LWQXA3OEKM52B2NIYM2JA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FbVbo8kGy1aaFyCk2noiXNUJHtosFJHFykBMRecGd5Kj5gjEp): protocol: expected piece hash; context canceled; EOF Dec 28 02:02:44.123675267 (538h33m43.994758313s) [noise=true , address=77.90.31.18:4833 , piece_id=3E57LWLI3WHROFGMXVZ7LHDHO5O7453LWQXA3OEKM52B2NIYM2JA, seg_pos={0 0}, upload=472313]: Long tail cancellation due to optimal threshold Dec 28 02:02:44.123774066 (538h33m43.994857101s) [noise=true , address=77.90.31.10:4524 , piece_id=X2VCBS4PTPPWI6U64T4FVIULRLCDY3XUOJTXJNP34OHSO54ONIWA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe): protocol: expected piece hash; context canceled; EOF Dec 28 02:02:44.123781761 (538h33m43.994864796s) [noise=true , address=77.90.31.10:4524 , piece_id=X2VCBS4PTPPWI6U64T4FVIULRLCDY3XUOJTXJNP34OHSO54ONIWA, seg_pos={0 0}, upload=472313]: Long tail cancellation due to optimal threshold Dec 28 02:02:44.123923341 (538h33m43.995006376s) [noise=true , address=145.79.227.35:4503 , piece_id=MDYJ2TROT4FHLX4XHKHEQACC7ZXHZCQ5MUPJL5EDS3F2YMXSUGUA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ctmSF1aMQasuXByFW8DppDantREmmWL2WUjmKGW5ZULVePyvm): protocol: expected piece hash; context canceled; EOF Dec 28 02:02:44.123928591 (538h33m43.995011636s) [noise=true , address=145.79.227.35:4503 , piece_id=MDYJ2TROT4FHLX4XHKHEQACC7ZXHZCQ5MUPJL5EDS3F2YMXSUGUA, seg_pos={0 0}, upload=472313]: Long tail cancellation due to optimal threshold Dec 28 02:02:44.123944731 (538h33m43.995027766s) [seg_pos={0 0}, upload=472313]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 02:02:44.124017179 (538h33m43.995100214s) [upload=472313]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:02:44.124022469 (538h33m43.995105504s) [upload=472313]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:02:44.409047274 (538h33m44.280130349s) [upload=472313]: Done uploading segments... Dec 28 02:02:44.409073354 (538h33m44.280156389s) [upload=472313]: Upload finished. err: <nil> Duration: 47.865604093s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................................................X......] duration: 3.977264387s trace: https://jaeger.dev.storj.io/trace/7fc142ad3d711e34 FAIL (2025-12-28 02:10:41 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.977456984s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 15.441818152s trace: https://jaeger.dev.storj.io/trace/12a1d059b6f75af7 FAIL (2025-12-28 02:10:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1.145865281 (538h33m41.016948316s) [noise=true , address=77.90.31.10:4551 , piece_id=KBPO4MUDASHKMPRNJELQWLNMJNQBB3PZUOVNCEPP2CYVHU6X2PPA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.146408725 (538h33m41.01749177s) [noise=true , address=145.79.227.15:4908 , piece_id=I2FSDNVYBFZKRXZNUHH2APY7N2PI7DPAKFKGXRJW2UFU2PBKSSNQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.146778139 (538h33m41.017861184s) [noise=true , address=145.79.227.43:4161 , piece_id=ZN3T6JSPHJ57X4YYG5MHRJYYOFPH2T2CSGJ34MJQU55HDC7CMOXQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.147137874 (538h33m41.018220919s) [noise=true , address=77.90.31.21:4882 , piece_id=WQAGQTGLU5HRKEET36W34MQP4NPGSVCA4QZ6YEGVAJWBBPLZOUQQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.147426784 (538h33m41.018509819s) [noise=true , address=145.79.227.35:4503 , piece_id=MDYJ2TROT4FHLX4XHKHEQACC7ZXHZCQ5MUPJL5EDS3F2YMXSUGUA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.147466109 (538h33m41.018549154s) [noise=true , address=77.90.31.15:4730 , piece_id=EN7IGRJZ2WFN4HXJPMNLLXDUZPIN7KDUGKPPLJH7CSA4SDOUM2JA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.147817368 (538h33m41.018900413s) [noise=true , address=77.90.31.18:4833 , piece_id=3E57LWLI3WHROFGMXVZ7LHDHO5O7453LWQXA3OEKM52B2NIYM2JA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.147860690 (538h33m41.018943745s) [noise=true , address=145.79.227.44:4212 , piece_id=EFTTQM6323MUQ5WTGY6DFBTF3TGRM6XZWXIGT6DM4VS7GRGM4C2Q, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148163427 (538h33m41.019246472s) [noise=true , address=77.90.31.19:4862 , piece_id=PSBBBSGY7THVLMXJQCADIE7VYZ33OIIDVAEQWG3RWMNA6RSKELNQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148189276 (538h33m41.019272321s) [noise=true , address=77.90.31.10:4524 , piece_id=X2VCBS4PTPPWI6U64T4FVIULRLCDY3XUOJTXJNP34OHSO54ONIWA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148496531 (538h33m41.019579566s) [noise=true , address=145.79.227.23:4556 , piece_id=AKBXRSU5PM5QLHGHE3VXTJ3ZH23BECPGBLQSVZK6XVIYJGVDXT4Q, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148527380 (538h33m41.019610425s) [noise=true , address=145.79.227.33:4899 , piece_id=JFEXXK22EATV7UUH7LLXTFEPIUBW7EWSNYFXLE4AO5F6SLYFTGBA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148801512 (538h33m41.019884547s) [noise=true , address=77.90.31.14:4672 , piece_id=BZG2Y5X47J2WVFF5LBHEJZNUD2OQ3NOTVPTCAVT4TLFQB5FWESLA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148894920 (538h33m41.019977965s) [noise=true , address=5.231.73.5:4016 , piece_id=YCNKQEKJQSJHKKGCYWLOLEBKIVPJHZIJ6TYEG6DF4TRXMZ3ZTTIQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.149118806 (538h33m41.020201851s) [noise=true , address=145.79.227.47:4295 , piece_id=OM7B75MCPM7L3WP5MLQ7MAVBZ2NJUSISOWD56RSWI2AV2OWVDRTA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.149274993 (538h33m41.020358038s) [noise=true , address=145.79.227.50:4400 , piece_id=HZAT7YAVZYWK52FE4TIC6OA37U4VX5B35H55WMBO3MMWNIKTJLJQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.149625842 (538h33m41.020708877s) [noise=true , address=145.79.227.34:4915 , piece_id=RMXYZGGJA5EPJLQD6SQG677HFLOQWRY2HFTGEOICYAQG6QR557RQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:42.618098549 (538h33m42.489181584s) [upload=472313]: Next returned nil segment Dec 28 02:02:42.618120911 (538h33m42.489203946s) [upload=472313]: Waiting for error group managing segments... Dec 28 02:02:43.021704832 (538h33m42.892787877s) [noise=true , address=145.79.227.23:4556 , piece_id=AKBXRSU5PM5QLHGHE3VXTJ3ZH23BECPGBLQSVZK6XVIYJGVDXT4Q, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.027980492 (538h33m42.899063537s) [noise=true , address=145.79.227.50:4400 , piece_id=HZAT7YAVZYWK52FE4TIC6OA37U4VX5B35H55WMBO3MMWNIKTJLJQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.043329239 (538h33m42.914412284s) [noise=true , address=145.79.227.15:4908 , piece_id=I2FSDNVYBFZKRXZNUHH2APY7N2PI7DPAKFKGXRJW2UFU2PBKSSNQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.056308314 (538h33m42.927391349s) [noise=true , address=145.79.227.49:4347 , piece_id=WURIHGQASBLSVLZJVM5EJHWD7U4VFLLPGJIFYNOK6ALA5LYEQDSA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.074628196 (538h33m42.945711252s) [noise=true , address=145.79.227.47:4295 , piece_id=OM7B75MCPM7L3WP5MLQ7MAVBZ2NJUSISOWD56RSWI2AV2OWVDRTA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.080495740 (538h33m42.951578775s) [noise=true , address=5.231.73.5:4016 , piece_id=YCNKQEKJQSJHKKGCYWLOLEBKIVPJHZIJ6TYEG6DF4TRXMZ3ZTTIQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.193955176 (538h33m43.065038241s) [noise=true , address=77.90.31.10:4551 , piece_id=KBPO4MUDASHKMPRNJELQWLNMJNQBB3PZUOVNCEPP2CYVHU6X2PPA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.301513865 (538h33m43.1725969s) [noise=true , address=77.90.31.15:4730 , piece_id=EN7IGRJZ2WFN4HXJPMNLLXDUZPIN7KDUGKPPLJH7CSA4SDOUM2JA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.305020622 (538h33m43.176103657s) [noise=true , address=77.90.31.19:4862 , piece_id=PSBBBSGY7THVLMXJQCADIE7VYZ33OIIDVAEQWG3RWMNA6RSKELNQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.312187619 (538h33m43.183270654s) [noise=true , address=77.90.31.14:4672 , piece_id=BZG2Y5X47J2WVFF5LBHEJZNUD2OQ3NOTVPTCAVT4TLFQB5FWESLA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.319277390 (538h33m43.190360425s) [noise=true , address=77.90.31.21:4882 , piece_id=WQAGQTGLU5HRKEET36W34MQP4NPGSVCA4QZ6YEGVAJWBBPLZOUQQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.948916959 (538h33m43.820000004s) [noise=true , address=145.79.227.34:4915 , piece_id=RMXYZGGJA5EPJLQD6SQG677HFLOQWRY2HFTGEOICYAQG6QR557RQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.989823331 (538h33m43.860906366s) [noise=true , address=145.79.227.44:4212 , piece_id=EFTTQM6323MUQ5WTGY6DFBTF3TGRM6XZWXIGT6DM4VS7GRGM4C2Q, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:44.123218949 (538h33m43.994301984s) [noise=true , address=145.79.227.43:4161 , piece_id=ZN3T6JSPHJ57X4YYG5MHRJYYOFPH2T2CSGJ34MJQU55HDC7CMOXQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:44.123238386 (538h33m43.994321421s) [seg_pos={0 0}, upload=472313]: Segment reached optimal threshold of 14 pieces. Dec 28 02:02:44.123431754 (538h33m43.994514789s) [noise=true , address=145.79.227.33:4899 , piece_id=JFEXXK22EATV7UUH7LLXTFEPIUBW7EWSNYFXLE4AO5F6SLYFTGBA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b4pZVQRnjcQNn45VTgdsz5fpjYPMAzSXGR5KJ5BP9XHMkRnRw): protocol: expected piece hash; context canceled; EOF Dec 28 02:02:44.123437586 (538h33m43.994520621s) [noise=true , address=145.79.227.33:4899 , piece_id=JFEXXK22EATV7UUH7LLXTFEPIUBW7EWSNYFXLE4AO5F6SLYFTGBA, seg_pos={0 0}, upload=472313]: Long tail cancellation due to optimal threshold Dec 28 02:02:44.123631064 (538h33m43.994714109s) [noise=true , address=77.90.31.18:4833 , piece_id=3E57LWLI3WHROFGMXVZ7LHDHO5O7453LWQXA3OEKM52B2NIYM2JA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FbVbo8kGy1aaFyCk2noiXNUJHtosFJHFykBMRecGd5Kj5gjEp): protocol: expected piece hash; context canceled; EOF Dec 28 02:02:44.123675267 (538h33m43.994758313s) [noise=true , address=77.90.31.18:4833 , piece_id=3E57LWLI3WHROFGMXVZ7LHDHO5O7453LWQXA3OEKM52B2NIYM2JA, seg_pos={0 0}, upload=472313]: Long tail cancellation due to optimal threshold Dec 28 02:02:44.123774066 (538h33m43.994857101s) [noise=true , address=77.90.31.10:4524 , piece_id=X2VCBS4PTPPWI6U64T4FVIULRLCDY3XUOJTXJNP34OHSO54ONIWA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe): protocol: expected piece hash; context canceled; EOF Dec 28 02:02:44.123781761 (538h33m43.994864796s) [noise=true , address=77.90.31.10:4524 , piece_id=X2VCBS4PTPPWI6U64T4FVIULRLCDY3XUOJTXJNP34OHSO54ONIWA, seg_pos={0 0}, upload=472313]: Long tail cancellation due to optimal threshold Dec 28 02:02:44.123923341 (538h33m43.995006376s) [noise=true , address=145.79.227.35:4503 , piece_id=MDYJ2TROT4FHLX4XHKHEQACC7ZXHZCQ5MUPJL5EDS3F2YMXSUGUA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ctmSF1aMQasuXByFW8DppDantREmmWL2WUjmKGW5ZULVePyvm): protocol: expected piece hash; context canceled; EOF Dec 28 02:02:44.123928591 (538h33m43.995011636s) [noise=true , address=145.79.227.35:4503 , piece_id=MDYJ2TROT4FHLX4XHKHEQACC7ZXHZCQ5MUPJL5EDS3F2YMXSUGUA, seg_pos={0 0}, upload=472313]: Long tail cancellation due to optimal threshold Dec 28 02:02:44.123944731 (538h33m43.995027766s) [seg_pos={0 0}, upload=472313]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 02:02:44.124017179 (538h33m43.995100214s) [upload=472313]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:02:44.124022469 (538h33m43.995105504s) [upload=472313]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:02:44.409047274 (538h33m44.280130349s) [upload=472313]: Done uploading segments... Dec 28 02:02:44.409073354 (538h33m44.280156389s) [upload=472313]: Upload finished. err: <nil> Duration: 15.441834814s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 31 out of 60 is failed (threshold=3) [.X.XX....X.XX.XXXXX..X.XXX.X.XX.XX.X..X.X.X.X.X..X...XX..X.X] duration: 30.006378153s trace: https://jaeger.dev.storj.io/trace/6f59fef8fc8d5abc FAIL (2025-12-28 02:10:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: 2EYJL73PC4NDTZ4SY7L3MHUROESODWK2RFVDOZWGWK24XIKH7EDQ) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: ZEX4CDARSQXCTOYS3SABYXOWBCDELCBQUGBPVOMZFCMXCZLPNOFA) context deadline exceeded ======LOGS====== 28 02:09:35.379340087 (538h40m35.250423122s) [upload=472409]: Starting upload Dec 28 02:09:35.379442121 (538h40m35.250525186s) [upload=472409]: Done starting upload Dec 28 02:09:35.379521131 (538h40m35.250604186s) [upload=472409]: Uploading segments... Dec 28 02:09:35.379807246 (538h40m35.250890291s) [upload=472409]: Got next segment. Inline: false Dec 28 02:09:35.379847293 (538h40m35.250930348s) [upload=472409]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 02:09:35.379853114 (538h40m35.250936159s) [upload=472409]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 02:09:35.633489390 (538h40m35.504572435s) [seg_pos={0 0}, upload=472409]: Begin upload segment... Dec 28 02:09:35.633560455 (538h40m35.5046435s) [seg_pos={0 0}, upload=472409]: Done begin upload segment. Dec 28 02:09:35.633596935 (538h40m35.50467999s) [noise=true , address=45.91.116.10:4082 , piece_id=EMOMICWKDK4TJKWBBYVJJMIXTIYWSFLOOCWJ5Q5BBEAZEXK7RBEA, seg_pos={0 0}, upload=472409]: Uploading piece... Dec 28 02:09:35.633622634 (538h40m35.504705669s) [noise=true , address=145.79.227.13:4804 , piece_id=EJQDDQEJDEI6UUQLPDGFKFELNZYSGQ4YZ5WOJOYGHH3Q6N4CG7LQ, seg_pos={0 0}, upload=472409]: Uploading piece... Dec 28 02:09:35.634523369 (538h40m35.505606404s) [noise=true , address=77.90.31.18:4808 , piece_id=2EYJL73PC4NDTZ4SY7L3MHUROESODWK2RFVDOZWGWK24XIKH7EDQ, seg_pos={0 0}, upload=472409]: Uploading piece... Dec 28 02:09:35.634537796 (538h40m35.505620831s) [noise=true , address=145.79.227.15:4876 , piece_id=PND7V446FV2ZUE746PUE3WAHRFXNI5SUI6RBVR22ZON6U4YRN4PA, seg_pos={0 0}, upload=472409]: Uploading piece... Dec 28 02:09:35.635109865 (538h40m35.50619292s) [noise=true , address=145.79.227.9:4367 , piece_id=DEAQKI4BMYGTUR2QB7325K44ZKX34WBV5HS3BPROWMTM43JPS7VA, seg_pos={0 0}, upload=472409]: Uploading piece... Dec 28 02:09:35.635306049 (538h40m35.506389094s) [noise=true , address=145.79.227.18:4006 , piece_id=RNAZFI6IYPHMYZVRMEP2VZD3PH7FHDH7BWKGNBUU4FWSJIOF5HQQ, seg_pos={0 0}, upload=472409]: Uploading piece... Dec 28 02:09:35.635549583 (538h40m35.506632628s) [noise=true , address=145.79.227.33:4870 , piece_id=JR2IIOEXQVBBIH34A54BMT23V6TETENNYBRWH3G3GVCON3DMHAJQ, seg_pos={0 0}, upload=472409]: Uploading piece... Dec 28 02:09:35.635859343 (538h40m35.506942378s) [noise=true , address=145.79.227.2:4114 , piece_id=EX2VXCNLXE7QT36KTN6GQRL2QLLRWBSZ4CDTB36SYLPYL25L7Z4A, seg_pos={0 0}, upload=472409]: Uploading piece... Dec 28 02:09:35.635957830 (538h40m35.507040866s) [noise=true , address=45.91.116.6:4976 , piece_id=EAF63WQMMQI6A354IP5IOMGJ22J3ABH4CW4ZROJWENTFV3C65K3A, seg_pos={0 0}, upload=472409]: Uploading piece... Dec 28 02:09:35.636425150 (538h40m35.507508195s) [noise=true , address=77.90.31.21:4877 , piece_id=T7IJ6HCTPO6FK5NQZDUTZZORHKPUV2LTERFLXSKIOOKR6HUISIEQ, seg_pos={0 0}, upload=472409]: Uploading piece... Dec 28 02:09:35.636545109 (538h40m35.507628144s) [noise=true , address=45.91.116.14:4192 , piece_id=ZEX4CDARSQXCTOYS3SABYXOWBCDELCBQUGBPVOMZFCMXCZLPNOFA, seg_pos={0 0}, upload=472409]: Uploading piece... Dec 28 02:09:35.636902980 (538h40m35.507986025s) [noise=true , address=145.79.227.43:4158 , piece_id=ISG6WTUIMT6K4UVWGMSTLLKHFDFOOXECJ25FQ3OV43MPGQNKTOHQ, seg_pos={0 0}, upload=472409]: Uploading piece... Dec 28 02:09:35.637122408 (538h40m35.508205463s) [noise=true , address=145.79.227.45:4231 , piece_id=ER7XWQTXM2YE5JOQINUQ2ESAAHAJOEVJ2C6L4IRY7MQUS7XSNLMA, seg_pos={0 0}, upload=472409]: Uploading piece... Dec 28 02:09:35.637351414 (538h40m35.508434449s) [noise=true , address=145.79.227.51:4410 , piece_id=R22RLSDDVBLW2ZAXKHM2W4VP2VGSRIUTOOTH3IY5DVY2243RXFXA, seg_pos={0 0}, upload=472409]: Uploading piece... Dec 28 02:09:35.638972110 (538h40m35.510055155s) [noise=true , address=77.90.31.11:4556 , piece_id=EIBTVEAFADM46FLUZEIKN4T5DCX3ZAK4Y7DLTZ6SZZOMNA52XXRA, seg_pos={0 0}, upload=472409]: Uploading piece... Dec 28 02:09:35.639430113 (538h40m35.510513158s) [noise=true , address=45.91.116.5:4298 , piece_id=DJOEYKEU6QMPI4X6SWKAMUOGVOQLUV3FTM7SRUAWVDS5INLBY6KQ, seg_pos={0 0}, upload=472409]: Uploading piece... Dec 28 02:09:37.245920889 (538h40m37.117003924s) [upload=472409]: Next returned nil segment Dec 28 02:09:37.245937972 (538h40m37.117021007s) [upload=472409]: Waiting for error group managing segments... Dec 28 02:09:37.642567892 (538h40m37.513650927s) [noise=true , address=145.79.227.18:4006 , piece_id=RNAZFI6IYPHMYZVRMEP2VZD3PH7FHDH7BWKGNBUU4FWSJIOF5HQQ, seg_pos={0 0}, upload=472409]: Done uploading piece. err: <nil> Dec 28 02:09:37.646027429 (538h40m37.517110464s) [noise=true , address=145.79.227.15:4876 , piece_id=PND7V446FV2ZUE746PUE3WAHRFXNI5SUI6RBVR22ZON6U4YRN4PA, seg_pos={0 0}, upload=472409]: Done uploading piece. err: <nil> Dec 28 02:09:37.667646025 (538h40m37.53872907s) [noise=true , address=145.79.227.13:4804 , piece_id=EJQDDQEJDEI6UUQLPDGFKFELNZYSGQ4YZ5WOJOYGHH3Q6N4CG7LQ, seg_pos={0 0}, upload=472409]: Done uploading piece. err: <nil> Dec 28 02:09:37.825865892 (538h40m37.696948927s) [noise=true , address=77.90.31.21:4877 , piece_id=T7IJ6HCTPO6FK5NQZDUTZZORHKPUV2LTERFLXSKIOOKR6HUISIEQ, seg_pos={0 0}, upload=472409]: Done uploading piece. err: <nil> Dec 28 02:09:37.906786838 (538h40m37.777869873s) [noise=true , address=145.79.227.45:4231 , piece_id=ER7XWQTXM2YE5JOQINUQ2ESAAHAJOEVJ2C6L4IRY7MQUS7XSNLMA, seg_pos={0 0}, upload=472409]: Done uploading piece. err: <nil> Dec 28 02:09:37.919862988 (538h40m37.790946023s) [noise=true , address=145.79.227.51:4410 , piece_id=R22RLSDDVBLW2ZAXKHM2W4VP2VGSRIUTOOTH3IY5DVY2243RXFXA, seg_pos={0 0}, upload=472409]: Done uploading piece. err: <nil> Dec 28 02:09:37.941457547 (538h40m37.812540582s) [noise=true , address=77.90.31.18:4808 , piece_id=2EYJL73PC4NDTZ4SY7L3MHUROESODWK2RFVDOZWGWK24XIKH7EDQ, seg_pos={0 0}, upload=472409]: Done uploading piece. err: <nil> Dec 28 02:09:37.951886126 (538h40m37.822969171s) [noise=true , address=45.91.116.6:4976 , piece_id=EAF63WQMMQI6A354IP5IOMGJ22J3ABH4CW4ZROJWENTFV3C65K3A, seg_pos={0 0}, upload=472409]: Done uploading piece. err: <nil> Dec 28 02:09:38.033688681 (538h40m37.904771716s) [noise=true , address=145.79.227.9:4367 , piece_id=DEAQKI4BMYGTUR2QB7325K44ZKX34WBV5HS3BPROWMTM43JPS7VA, seg_pos={0 0}, upload=472409]: Done uploading piece. err: <nil> Dec 28 02:09:38.333521734 (538h40m38.204604769s) [noise=true , address=145.79.227.33:4870 , piece_id=JR2IIOEXQVBBIH34A54BMT23V6TETENNYBRWH3G3GVCON3DMHAJQ, seg_pos={0 0}, upload=472409]: Done uploading piece. err: <nil> Dec 28 02:09:38.575157388 (538h40m38.446240443s) [noise=true , address=77.90.31.11:4556 , piece_id=EIBTVEAFADM46FLUZEIKN4T5DCX3ZAK4Y7DLTZ6SZZOMNA52XXRA, seg_pos={0 0}, upload=472409]: Done uploading piece. err: <nil> Dec 28 02:09:38.589027198 (538h40m38.460110233s) [noise=true , address=45.91.116.10:4082 , piece_id=EMOMICWKDK4TJKWBBYVJJMIXTIYWSFLOOCWJ5Q5BBEAZEXK7RBEA, seg_pos={0 0}, upload=472409]: Done uploading piece. err: <nil> Dec 28 02:09:38.702920891 (538h40m38.574003936s) [noise=true , address=45.91.116.14:4192 , piece_id=ZEX4CDARSQXCTOYS3SABYXOWBCDELCBQUGBPVOMZFCMXCZLPNOFA, seg_pos={0 0}, upload=472409]: Done uploading piece. err: <nil> Dec 28 02:09:38.702941851 (538h40m38.574024896s) [seg_pos={0 0}, upload=472409]: Segment reached optimal threshold of 13 pieces. Dec 28 02:09:38.703215582 (538h40m38.574298617s) [noise=true , address=45.91.116.5:4298 , piece_id=DJOEYKEU6QMPI4X6SWKAMUOGVOQLUV3FTM7SRUAWVDS5INLBY6KQ, seg_pos={0 0}, upload=472409]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 28 02:09:38.703231622 (538h40m38.574314668s) [noise=true , address=45.91.116.5:4298 , piece_id=DJOEYKEU6QMPI4X6SWKAMUOGVOQLUV3FTM7SRUAWVDS5INLBY6KQ, seg_pos={0 0}, upload=472409]: Long tail cancellation due to optimal threshold Dec 28 02:09:38.703237534 (538h40m38.574320569s) [noise=true , address=145.79.227.43:4158 , piece_id=ISG6WTUIMT6K4UVWGMSTLLKHFDFOOXECJ25FQ3OV43MPGQNKTOHQ, seg_pos={0 0}, upload=472409]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF Dec 28 02:09:38.703249978 (538h40m38.574333023s) [noise=true , address=145.79.227.43:4158 , piece_id=ISG6WTUIMT6K4UVWGMSTLLKHFDFOOXECJ25FQ3OV43MPGQNKTOHQ, seg_pos={0 0}, upload=472409]: Long tail cancellation due to optimal threshold Dec 28 02:09:38.703389353 (538h40m38.574472388s) [noise=true , address=145.79.227.2:4114 , piece_id=EX2VXCNLXE7QT36KTN6GQRL2QLLRWBSZ4CDTB36SYLPYL25L7Z4A, seg_pos={0 0}, upload=472409]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN): protocol: write tcp 188.166.225.170:52408->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:52408->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52408->145.79.227.2:4114: use of closed network connection Dec 28 02:09:38.703395435 (538h40m38.57447847s) [noise=true , address=145.79.227.2:4114 , piece_id=EX2VXCNLXE7QT36KTN6GQRL2QLLRWBSZ4CDTB36SYLPYL25L7Z4A, seg_pos={0 0}, upload=472409]: Long tail cancellation due to optimal threshold Dec 28 02:09:38.703411916 (538h40m38.574494951s) [seg_pos={0 0}, upload=472409]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 02:09:38.703543376 (538h40m38.574626411s) [upload=472409]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:09:38.703549738 (538h40m38.574632773s) [upload=472409]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:09:39.090614906 (538h40m38.961697941s) [upload=472409]: Done uploading segments... Dec 28 02:09:39.090633411 (538h40m38.961716446s) [upload=472409]: Upload finished. err: <nil> Duration: 30.006404804s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 37.371412301s trace: https://jaeger.dev.storj.io/trace/12a1d059b6f75af7 FAIL (2025-12-28 02:09:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1.145865281 (538h33m41.016948316s) [noise=true , address=77.90.31.10:4551 , piece_id=KBPO4MUDASHKMPRNJELQWLNMJNQBB3PZUOVNCEPP2CYVHU6X2PPA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.146408725 (538h33m41.01749177s) [noise=true , address=145.79.227.15:4908 , piece_id=I2FSDNVYBFZKRXZNUHH2APY7N2PI7DPAKFKGXRJW2UFU2PBKSSNQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.146778139 (538h33m41.017861184s) [noise=true , address=145.79.227.43:4161 , piece_id=ZN3T6JSPHJ57X4YYG5MHRJYYOFPH2T2CSGJ34MJQU55HDC7CMOXQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.147137874 (538h33m41.018220919s) [noise=true , address=77.90.31.21:4882 , piece_id=WQAGQTGLU5HRKEET36W34MQP4NPGSVCA4QZ6YEGVAJWBBPLZOUQQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.147426784 (538h33m41.018509819s) [noise=true , address=145.79.227.35:4503 , piece_id=MDYJ2TROT4FHLX4XHKHEQACC7ZXHZCQ5MUPJL5EDS3F2YMXSUGUA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.147466109 (538h33m41.018549154s) [noise=true , address=77.90.31.15:4730 , piece_id=EN7IGRJZ2WFN4HXJPMNLLXDUZPIN7KDUGKPPLJH7CSA4SDOUM2JA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.147817368 (538h33m41.018900413s) [noise=true , address=77.90.31.18:4833 , piece_id=3E57LWLI3WHROFGMXVZ7LHDHO5O7453LWQXA3OEKM52B2NIYM2JA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.147860690 (538h33m41.018943745s) [noise=true , address=145.79.227.44:4212 , piece_id=EFTTQM6323MUQ5WTGY6DFBTF3TGRM6XZWXIGT6DM4VS7GRGM4C2Q, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148163427 (538h33m41.019246472s) [noise=true , address=77.90.31.19:4862 , piece_id=PSBBBSGY7THVLMXJQCADIE7VYZ33OIIDVAEQWG3RWMNA6RSKELNQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148189276 (538h33m41.019272321s) [noise=true , address=77.90.31.10:4524 , piece_id=X2VCBS4PTPPWI6U64T4FVIULRLCDY3XUOJTXJNP34OHSO54ONIWA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148496531 (538h33m41.019579566s) [noise=true , address=145.79.227.23:4556 , piece_id=AKBXRSU5PM5QLHGHE3VXTJ3ZH23BECPGBLQSVZK6XVIYJGVDXT4Q, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148527380 (538h33m41.019610425s) [noise=true , address=145.79.227.33:4899 , piece_id=JFEXXK22EATV7UUH7LLXTFEPIUBW7EWSNYFXLE4AO5F6SLYFTGBA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148801512 (538h33m41.019884547s) [noise=true , address=77.90.31.14:4672 , piece_id=BZG2Y5X47J2WVFF5LBHEJZNUD2OQ3NOTVPTCAVT4TLFQB5FWESLA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148894920 (538h33m41.019977965s) [noise=true , address=5.231.73.5:4016 , piece_id=YCNKQEKJQSJHKKGCYWLOLEBKIVPJHZIJ6TYEG6DF4TRXMZ3ZTTIQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.149118806 (538h33m41.020201851s) [noise=true , address=145.79.227.47:4295 , piece_id=OM7B75MCPM7L3WP5MLQ7MAVBZ2NJUSISOWD56RSWI2AV2OWVDRTA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.149274993 (538h33m41.020358038s) [noise=true , address=145.79.227.50:4400 , piece_id=HZAT7YAVZYWK52FE4TIC6OA37U4VX5B35H55WMBO3MMWNIKTJLJQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.149625842 (538h33m41.020708877s) [noise=true , address=145.79.227.34:4915 , piece_id=RMXYZGGJA5EPJLQD6SQG677HFLOQWRY2HFTGEOICYAQG6QR557RQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:42.618098549 (538h33m42.489181584s) [upload=472313]: Next returned nil segment Dec 28 02:02:42.618120911 (538h33m42.489203946s) [upload=472313]: Waiting for error group managing segments... Dec 28 02:02:43.021704832 (538h33m42.892787877s) [noise=true , address=145.79.227.23:4556 , piece_id=AKBXRSU5PM5QLHGHE3VXTJ3ZH23BECPGBLQSVZK6XVIYJGVDXT4Q, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.027980492 (538h33m42.899063537s) [noise=true , address=145.79.227.50:4400 , piece_id=HZAT7YAVZYWK52FE4TIC6OA37U4VX5B35H55WMBO3MMWNIKTJLJQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.043329239 (538h33m42.914412284s) [noise=true , address=145.79.227.15:4908 , piece_id=I2FSDNVYBFZKRXZNUHH2APY7N2PI7DPAKFKGXRJW2UFU2PBKSSNQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.056308314 (538h33m42.927391349s) [noise=true , address=145.79.227.49:4347 , piece_id=WURIHGQASBLSVLZJVM5EJHWD7U4VFLLPGJIFYNOK6ALA5LYEQDSA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.074628196 (538h33m42.945711252s) [noise=true , address=145.79.227.47:4295 , piece_id=OM7B75MCPM7L3WP5MLQ7MAVBZ2NJUSISOWD56RSWI2AV2OWVDRTA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.080495740 (538h33m42.951578775s) [noise=true , address=5.231.73.5:4016 , piece_id=YCNKQEKJQSJHKKGCYWLOLEBKIVPJHZIJ6TYEG6DF4TRXMZ3ZTTIQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.193955176 (538h33m43.065038241s) [noise=true , address=77.90.31.10:4551 , piece_id=KBPO4MUDASHKMPRNJELQWLNMJNQBB3PZUOVNCEPP2CYVHU6X2PPA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.301513865 (538h33m43.1725969s) [noise=true , address=77.90.31.15:4730 , piece_id=EN7IGRJZ2WFN4HXJPMNLLXDUZPIN7KDUGKPPLJH7CSA4SDOUM2JA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.305020622 (538h33m43.176103657s) [noise=true , address=77.90.31.19:4862 , piece_id=PSBBBSGY7THVLMXJQCADIE7VYZ33OIIDVAEQWG3RWMNA6RSKELNQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.312187619 (538h33m43.183270654s) [noise=true , address=77.90.31.14:4672 , piece_id=BZG2Y5X47J2WVFF5LBHEJZNUD2OQ3NOTVPTCAVT4TLFQB5FWESLA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.319277390 (538h33m43.190360425s) [noise=true , address=77.90.31.21:4882 , piece_id=WQAGQTGLU5HRKEET36W34MQP4NPGSVCA4QZ6YEGVAJWBBPLZOUQQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.948916959 (538h33m43.820000004s) [noise=true , address=145.79.227.34:4915 , piece_id=RMXYZGGJA5EPJLQD6SQG677HFLOQWRY2HFTGEOICYAQG6QR557RQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.989823331 (538h33m43.860906366s) [noise=true , address=145.79.227.44:4212 , piece_id=EFTTQM6323MUQ5WTGY6DFBTF3TGRM6XZWXIGT6DM4VS7GRGM4C2Q, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:44.123218949 (538h33m43.994301984s) [noise=true , address=145.79.227.43:4161 , piece_id=ZN3T6JSPHJ57X4YYG5MHRJYYOFPH2T2CSGJ34MJQU55HDC7CMOXQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:44.123238386 (538h33m43.994321421s) [seg_pos={0 0}, upload=472313]: Segment reached optimal threshold of 14 pieces. Dec 28 02:02:44.123431754 (538h33m43.994514789s) [noise=true , address=145.79.227.33:4899 , piece_id=JFEXXK22EATV7UUH7LLXTFEPIUBW7EWSNYFXLE4AO5F6SLYFTGBA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b4pZVQRnjcQNn45VTgdsz5fpjYPMAzSXGR5KJ5BP9XHMkRnRw): protocol: expected piece hash; context canceled; EOF Dec 28 02:02:44.123437586 (538h33m43.994520621s) [noise=true , address=145.79.227.33:4899 , piece_id=JFEXXK22EATV7UUH7LLXTFEPIUBW7EWSNYFXLE4AO5F6SLYFTGBA, seg_pos={0 0}, upload=472313]: Long tail cancellation due to optimal threshold Dec 28 02:02:44.123631064 (538h33m43.994714109s) [noise=true , address=77.90.31.18:4833 , piece_id=3E57LWLI3WHROFGMXVZ7LHDHO5O7453LWQXA3OEKM52B2NIYM2JA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FbVbo8kGy1aaFyCk2noiXNUJHtosFJHFykBMRecGd5Kj5gjEp): protocol: expected piece hash; context canceled; EOF Dec 28 02:02:44.123675267 (538h33m43.994758313s) [noise=true , address=77.90.31.18:4833 , piece_id=3E57LWLI3WHROFGMXVZ7LHDHO5O7453LWQXA3OEKM52B2NIYM2JA, seg_pos={0 0}, upload=472313]: Long tail cancellation due to optimal threshold Dec 28 02:02:44.123774066 (538h33m43.994857101s) [noise=true , address=77.90.31.10:4524 , piece_id=X2VCBS4PTPPWI6U64T4FVIULRLCDY3XUOJTXJNP34OHSO54ONIWA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe): protocol: expected piece hash; context canceled; EOF Dec 28 02:02:44.123781761 (538h33m43.994864796s) [noise=true , address=77.90.31.10:4524 , piece_id=X2VCBS4PTPPWI6U64T4FVIULRLCDY3XUOJTXJNP34OHSO54ONIWA, seg_pos={0 0}, upload=472313]: Long tail cancellation due to optimal threshold Dec 28 02:02:44.123923341 (538h33m43.995006376s) [noise=true , address=145.79.227.35:4503 , piece_id=MDYJ2TROT4FHLX4XHKHEQACC7ZXHZCQ5MUPJL5EDS3F2YMXSUGUA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ctmSF1aMQasuXByFW8DppDantREmmWL2WUjmKGW5ZULVePyvm): protocol: expected piece hash; context canceled; EOF Dec 28 02:02:44.123928591 (538h33m43.995011636s) [noise=true , address=145.79.227.35:4503 , piece_id=MDYJ2TROT4FHLX4XHKHEQACC7ZXHZCQ5MUPJL5EDS3F2YMXSUGUA, seg_pos={0 0}, upload=472313]: Long tail cancellation due to optimal threshold Dec 28 02:02:44.123944731 (538h33m43.995027766s) [seg_pos={0 0}, upload=472313]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 02:02:44.124017179 (538h33m43.995100214s) [upload=472313]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:02:44.124022469 (538h33m43.995105504s) [upload=472313]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:02:44.409047274 (538h33m44.280130349s) [upload=472313]: Done uploading segments... Dec 28 02:02:44.409073354 (538h33m44.280156389s) [upload=472313]: Upload finished. err: <nil> Duration: 37.371428912s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................................................X.....] duration: 3.500046395s trace: https://jaeger.dev.storj.io/trace/7fc142ad3d711e34 FAIL (2025-12-28 02:09:39 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.500062155s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 31 out of 60 is failed (threshold=3) [X.X.XX....X.XX.XXXXX..X.XXX.X.XX.XX.X..X.X.X.X.X..X...XX..X.] duration: 18.149823379s trace: https://jaeger.dev.storj.io/trace/2140711179d51097 FAIL (2025-12-28 02:09:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: H4ERAYCDVULAC733FW64VTEB6UY6J4WV4R4NEAOTGBMNAWUVIIDA) context deadline exceeded; piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: ZPNKHLD4BEQV6MNLI6LTRZE5SUYLUJ747AQTTILBKKCBOV2CZURA) context deadline exceeded; piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: WMJH4AOJKOKVZVAGPOC4RJG6WYWKNMQV65NLIRTH2WKAWN327NZQ) context deadline exceeded ======LOGS====== Dec 28 02:08:08.096883081 (538h39m7.967966126s) [upload=472390]: Starting upload Dec 28 02:08:08.096959186 (538h39m7.968042221s) [upload=472390]: Done starting upload Dec 28 02:08:08.097115213 (538h39m7.968198248s) [upload=472390]: Uploading segments... Dec 28 02:08:08.097228669 (538h39m7.968311714s) [upload=472390]: Got next segment. Inline: false Dec 28 02:08:08.097274376 (538h39m7.968357422s) [upload=472390]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 02:08:08.097279767 (538h39m7.968362802s) [upload=472390]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 02:08:08.343782458 (538h39m8.214865503s) [seg_pos={0 0}, upload=472390]: Begin upload segment... Dec 28 02:08:08.343845468 (538h39m8.214928513s) [seg_pos={0 0}, upload=472390]: Done begin upload segment. Dec 28 02:08:08.343887899 (538h39m8.214970934s) [noise=true , address=145.79.227.15:4874 , piece_id=KCKZAV3ENC6UJ44UYKMNLAVMHZ73FDVZBLAUKYN4QNIIAIT4SWNQ, seg_pos={0 0}, upload=472390]: Uploading piece... Dec 28 02:08:08.343910532 (538h39m8.214993567s) [noise=true , address=45.91.116.13:4153 , piece_id=TAVZ43N4CXCNFRCLII4BO2U3FX74QXO7EE7CTTT2JXUZQGQVCJMA, seg_pos={0 0}, upload=472390]: Uploading piece... Dec 28 02:08:08.344482381 (538h39m8.215565426s) [noise=true , address=45.91.116.4:4937 , piece_id=YQFM4J43LA32NYG7JKRB3RS4ZWAQV4W4C5A2N75BJLGMDJGIJZ4A, seg_pos={0 0}, upload=472390]: Uploading piece... Dec 28 02:08:08.344492741 (538h39m8.215575786s) [noise=true , address=145.79.227.52:4446 , piece_id=6S5F5BMRM3CSI2S5Z6WWR7RXBDF6PWPSEPKGICKVRQUD7BMAAEBQ, seg_pos={0 0}, upload=472390]: Uploading piece... Dec 28 02:08:08.344847446 (538h39m8.215930481s) [noise=true , address=145.79.227.7:4293 , piece_id=QPRU6FUMCQF2V2MZU6CCGYCJGBL6IJAW2SGM4BT3ZYWKIRKMBI4A, seg_pos={0 0}, upload=472390]: Uploading piece... Dec 28 02:08:08.344877273 (538h39m8.215960308s) [noise=true , address=77.90.31.18:4808 , piece_id=WMJH4AOJKOKVZVAGPOC4RJG6WYWKNMQV65NLIRTH2WKAWN327NZQ, seg_pos={0 0}, upload=472390]: Uploading piece... Dec 28 02:08:08.345186011 (538h39m8.216269046s) [noise=true , address=145.79.227.27:4691 , piece_id=CYAKJNKCSXHQMJB7SEAFJOO73PQOFYVSS4ZSZVD46LIQB2BGGSUA, seg_pos={0 0}, upload=472390]: Uploading piece... Dec 28 02:08:08.345294017 (538h39m8.216377052s) [noise=true , address=145.79.227.51:4411 , piece_id=YGH4ONL6VBFVOKRT3TGHUPCKR5ULDLLN5U4CWMG6VCY3Z7WBSIAA, seg_pos={0 0}, upload=472390]: Uploading piece... Dec 28 02:08:08.345707364 (538h39m8.216790409s) [noise=true , address=77.90.31.19:4844 , piece_id=6YP7YMB5XVLEJY73QFDPL2QG73GX2YFMLDTKBNKH66WWBDEA6Q6A, seg_pos={0 0}, upload=472390]: Uploading piece... Dec 28 02:08:08.345871506 (538h39m8.216954551s) [noise=true , address=77.90.31.10:4519 , piece_id=ZPNKHLD4BEQV6MNLI6LTRZE5SUYLUJ747AQTTILBKKCBOV2CZURA, seg_pos={0 0}, upload=472390]: Uploading piece... Dec 28 02:08:08.346155416 (538h39m8.217238461s) [noise=true , address=145.79.227.45:4229 , piece_id=SBXMFCXNFXKIFN6HFI7IDAHZNM6BWZHIFQAFJTMC22HCWPVXRRXA, seg_pos={0 0}, upload=472390]: Uploading piece... Dec 28 02:08:08.346543847 (538h39m8.217626892s) [noise=true , address=77.90.31.21:4877 , piece_id=2MHERE2XKTVEAX6LRKYYF4FV7JOSKAF3FJEAV5ZWC4ODXLQP3KBQ, seg_pos={0 0}, upload=472390]: Uploading piece... Dec 28 02:08:08.347125494 (538h39m8.218208529s) [noise=true , address=145.79.227.32:4836 , piece_id=DRKL7KL5VZXHUKE2QGM6YNIBA3SGOKMASLSTAYQWGVO27KJQ7JKQ, seg_pos={0 0}, upload=472390]: Uploading piece... Dec 28 02:08:08.347281912 (538h39m8.218364947s) [noise=true , address=145.79.227.47:4266 , piece_id=E3ZBG7GUMC4JIR7CZXAC56TMDSTOIPZSKOPRBTSADY3X2ANLXTZA, seg_pos={0 0}, upload=472390]: Uploading piece... Dec 28 02:08:08.347741938 (538h39m8.218824983s) [noise=true , address=45.91.116.6:4973 , piece_id=H4ERAYCDVULAC733FW64VTEB6UY6J4WV4R4NEAOTGBMNAWUVIIDA, seg_pos={0 0}, upload=472390]: Uploading piece... Dec 28 02:08:08.348318466 (538h39m8.219401501s) [noise=true , address=77.90.31.15:4700 , piece_id=ELIKNM64ZGW35I6N7ZOFHRCMOK43CL4W2BWUGCN6TEXYI23V3KPA, seg_pos={0 0}, upload=472390]: Uploading piece... Dec 28 02:08:10.056936841 (538h39m9.928019876s) [upload=472390]: Next returned nil segment Dec 28 02:08:10.056951479 (538h39m9.928034524s) [upload=472390]: Waiting for error group managing segments... Dec 28 02:08:10.333760791 (538h39m10.204843836s) [noise=true , address=145.79.227.15:4874 , piece_id=KCKZAV3ENC6UJ44UYKMNLAVMHZ73FDVZBLAUKYN4QNIIAIT4SWNQ, seg_pos={0 0}, upload=472390]: Done uploading piece. err: <nil> Dec 28 02:08:10.338359948 (538h39m10.209442983s) [noise=true , address=145.79.227.7:4293 , piece_id=QPRU6FUMCQF2V2MZU6CCGYCJGBL6IJAW2SGM4BT3ZYWKIRKMBI4A, seg_pos={0 0}, upload=472390]: Done uploading piece. err: <nil> Dec 28 02:08:10.349637483 (538h39m10.220720518s) [noise=true , address=145.79.227.27:4691 , piece_id=CYAKJNKCSXHQMJB7SEAFJOO73PQOFYVSS4ZSZVD46LIQB2BGGSUA, seg_pos={0 0}, upload=472390]: Done uploading piece. err: <nil> Dec 28 02:08:10.365965794 (538h39m10.237048839s) [noise=true , address=145.79.227.47:4266 , piece_id=E3ZBG7GUMC4JIR7CZXAC56TMDSTOIPZSKOPRBTSADY3X2ANLXTZA, seg_pos={0 0}, upload=472390]: Done uploading piece. err: <nil> Dec 28 02:08:10.374047483 (538h39m10.245130518s) [noise=true , address=145.79.227.51:4411 , piece_id=YGH4ONL6VBFVOKRT3TGHUPCKR5ULDLLN5U4CWMG6VCY3Z7WBSIAA, seg_pos={0 0}, upload=472390]: Done uploading piece. err: <nil> Dec 28 02:08:10.378783000 (538h39m10.249866035s) [noise=true , address=145.79.227.52:4446 , piece_id=6S5F5BMRM3CSI2S5Z6WWR7RXBDF6PWPSEPKGICKVRQUD7BMAAEBQ, seg_pos={0 0}, upload=472390]: Done uploading piece. err: <nil> Dec 28 02:08:10.382402763 (538h39m10.253485798s) [noise=true , address=145.79.227.45:4229 , piece_id=SBXMFCXNFXKIFN6HFI7IDAHZNM6BWZHIFQAFJTMC22HCWPVXRRXA, seg_pos={0 0}, upload=472390]: Done uploading piece. err: <nil> Dec 28 02:08:10.384796972 (538h39m10.255880007s) [noise=true , address=145.79.227.32:4836 , piece_id=DRKL7KL5VZXHUKE2QGM6YNIBA3SGOKMASLSTAYQWGVO27KJQ7JKQ, seg_pos={0 0}, upload=472390]: Done uploading piece. err: <nil> Dec 28 02:08:10.583803704 (538h39m10.454886749s) [noise=true , address=45.91.116.6:4973 , piece_id=H4ERAYCDVULAC733FW64VTEB6UY6J4WV4R4NEAOTGBMNAWUVIIDA, seg_pos={0 0}, upload=472390]: Done uploading piece. err: <nil> Dec 28 02:08:10.644860707 (538h39m10.515943752s) [noise=true , address=77.90.31.15:4700 , piece_id=ELIKNM64ZGW35I6N7ZOFHRCMOK43CL4W2BWUGCN6TEXYI23V3KPA, seg_pos={0 0}, upload=472390]: Done uploading piece. err: <nil> Dec 28 02:08:10.712079693 (538h39m10.583162738s) [noise=true , address=77.90.31.19:4844 , piece_id=6YP7YMB5XVLEJY73QFDPL2QG73GX2YFMLDTKBNKH66WWBDEA6Q6A, seg_pos={0 0}, upload=472390]: Done uploading piece. err: <nil> Dec 28 02:08:11.233977033 (538h39m11.105060078s) [noise=true , address=77.90.31.18:4808 , piece_id=WMJH4AOJKOKVZVAGPOC4RJG6WYWKNMQV65NLIRTH2WKAWN327NZQ, seg_pos={0 0}, upload=472390]: Done uploading piece. err: <nil> Dec 28 02:08:11.236266232 (538h39m11.107349287s) [noise=true , address=77.90.31.10:4519 , piece_id=ZPNKHLD4BEQV6MNLI6LTRZE5SUYLUJ747AQTTILBKKCBOV2CZURA, seg_pos={0 0}, upload=472390]: Done uploading piece. err: <nil> Dec 28 02:08:11.236286351 (538h39m11.107369386s) [seg_pos={0 0}, upload=472390]: Segment reached optimal threshold of 13 pieces. Dec 28 02:08:11.236504065 (538h39m11.10758711s) [noise=true , address=77.90.31.21:4877 , piece_id=2MHERE2XKTVEAX6LRKYYF4FV7JOSKAF3FJEAV5ZWC4ODXLQP3KBQ, seg_pos={0 0}, upload=472390]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 28 02:08:11.236511469 (538h39m11.107594514s) [noise=true , address=77.90.31.21:4877 , piece_id=2MHERE2XKTVEAX6LRKYYF4FV7JOSKAF3FJEAV5ZWC4ODXLQP3KBQ, seg_pos={0 0}, upload=472390]: Long tail cancellation due to optimal threshold Dec 28 02:08:11.236563629 (538h39m11.107646664s) [noise=true , address=45.91.116.4:4937 , piece_id=YQFM4J43LA32NYG7JKRB3RS4ZWAQV4W4C5A2N75BJLGMDJGIJZ4A, seg_pos={0 0}, upload=472390]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 28 02:08:11.236587634 (538h39m11.107670669s) [noise=true , address=45.91.116.4:4937 , piece_id=YQFM4J43LA32NYG7JKRB3RS4ZWAQV4W4C5A2N75BJLGMDJGIJZ4A, seg_pos={0 0}, upload=472390]: Long tail cancellation due to optimal threshold Dec 28 02:08:11.236771925 (538h39m11.10785498s) [noise=true , address=45.91.116.13:4153 , piece_id=TAVZ43N4CXCNFRCLII4BO2U3FX74QXO7EE7CTTT2JXUZQGQVCJMA, seg_pos={0 0}, upload=472390]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 28 02:08:11.236780601 (538h39m11.107863647s) [noise=true , address=45.91.116.13:4153 , piece_id=TAVZ43N4CXCNFRCLII4BO2U3FX74QXO7EE7CTTT2JXUZQGQVCJMA, seg_pos={0 0}, upload=472390]: Long tail cancellation due to optimal threshold Dec 28 02:08:11.236800189 (538h39m11.107883234s) [seg_pos={0 0}, upload=472390]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 02:08:11.236879280 (538h39m11.107962325s) [upload=472390]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:08:11.236884599 (538h39m11.107967645s) [upload=472390]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:08:11.536994098 (538h39m11.408077143s) [upload=472390]: Done uploading segments... Dec 28 02:08:11.537017663 (538h39m11.408100708s) [upload=472390]: Upload finished. err: <nil> Duration: 18.149852813s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 44.168118633s trace: https://jaeger.dev.storj.io/trace/12a1d059b6f75af7 FAIL (2025-12-28 02:09:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1.145865281 (538h33m41.016948316s) [noise=true , address=77.90.31.10:4551 , piece_id=KBPO4MUDASHKMPRNJELQWLNMJNQBB3PZUOVNCEPP2CYVHU6X2PPA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.146408725 (538h33m41.01749177s) [noise=true , address=145.79.227.15:4908 , piece_id=I2FSDNVYBFZKRXZNUHH2APY7N2PI7DPAKFKGXRJW2UFU2PBKSSNQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.146778139 (538h33m41.017861184s) [noise=true , address=145.79.227.43:4161 , piece_id=ZN3T6JSPHJ57X4YYG5MHRJYYOFPH2T2CSGJ34MJQU55HDC7CMOXQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.147137874 (538h33m41.018220919s) [noise=true , address=77.90.31.21:4882 , piece_id=WQAGQTGLU5HRKEET36W34MQP4NPGSVCA4QZ6YEGVAJWBBPLZOUQQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.147426784 (538h33m41.018509819s) [noise=true , address=145.79.227.35:4503 , piece_id=MDYJ2TROT4FHLX4XHKHEQACC7ZXHZCQ5MUPJL5EDS3F2YMXSUGUA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.147466109 (538h33m41.018549154s) [noise=true , address=77.90.31.15:4730 , piece_id=EN7IGRJZ2WFN4HXJPMNLLXDUZPIN7KDUGKPPLJH7CSA4SDOUM2JA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.147817368 (538h33m41.018900413s) [noise=true , address=77.90.31.18:4833 , piece_id=3E57LWLI3WHROFGMXVZ7LHDHO5O7453LWQXA3OEKM52B2NIYM2JA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.147860690 (538h33m41.018943745s) [noise=true , address=145.79.227.44:4212 , piece_id=EFTTQM6323MUQ5WTGY6DFBTF3TGRM6XZWXIGT6DM4VS7GRGM4C2Q, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148163427 (538h33m41.019246472s) [noise=true , address=77.90.31.19:4862 , piece_id=PSBBBSGY7THVLMXJQCADIE7VYZ33OIIDVAEQWG3RWMNA6RSKELNQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148189276 (538h33m41.019272321s) [noise=true , address=77.90.31.10:4524 , piece_id=X2VCBS4PTPPWI6U64T4FVIULRLCDY3XUOJTXJNP34OHSO54ONIWA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148496531 (538h33m41.019579566s) [noise=true , address=145.79.227.23:4556 , piece_id=AKBXRSU5PM5QLHGHE3VXTJ3ZH23BECPGBLQSVZK6XVIYJGVDXT4Q, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148527380 (538h33m41.019610425s) [noise=true , address=145.79.227.33:4899 , piece_id=JFEXXK22EATV7UUH7LLXTFEPIUBW7EWSNYFXLE4AO5F6SLYFTGBA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148801512 (538h33m41.019884547s) [noise=true , address=77.90.31.14:4672 , piece_id=BZG2Y5X47J2WVFF5LBHEJZNUD2OQ3NOTVPTCAVT4TLFQB5FWESLA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148894920 (538h33m41.019977965s) [noise=true , address=5.231.73.5:4016 , piece_id=YCNKQEKJQSJHKKGCYWLOLEBKIVPJHZIJ6TYEG6DF4TRXMZ3ZTTIQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.149118806 (538h33m41.020201851s) [noise=true , address=145.79.227.47:4295 , piece_id=OM7B75MCPM7L3WP5MLQ7MAVBZ2NJUSISOWD56RSWI2AV2OWVDRTA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.149274993 (538h33m41.020358038s) [noise=true , address=145.79.227.50:4400 , piece_id=HZAT7YAVZYWK52FE4TIC6OA37U4VX5B35H55WMBO3MMWNIKTJLJQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.149625842 (538h33m41.020708877s) [noise=true , address=145.79.227.34:4915 , piece_id=RMXYZGGJA5EPJLQD6SQG677HFLOQWRY2HFTGEOICYAQG6QR557RQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:42.618098549 (538h33m42.489181584s) [upload=472313]: Next returned nil segment Dec 28 02:02:42.618120911 (538h33m42.489203946s) [upload=472313]: Waiting for error group managing segments... Dec 28 02:02:43.021704832 (538h33m42.892787877s) [noise=true , address=145.79.227.23:4556 , piece_id=AKBXRSU5PM5QLHGHE3VXTJ3ZH23BECPGBLQSVZK6XVIYJGVDXT4Q, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.027980492 (538h33m42.899063537s) [noise=true , address=145.79.227.50:4400 , piece_id=HZAT7YAVZYWK52FE4TIC6OA37U4VX5B35H55WMBO3MMWNIKTJLJQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.043329239 (538h33m42.914412284s) [noise=true , address=145.79.227.15:4908 , piece_id=I2FSDNVYBFZKRXZNUHH2APY7N2PI7DPAKFKGXRJW2UFU2PBKSSNQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.056308314 (538h33m42.927391349s) [noise=true , address=145.79.227.49:4347 , piece_id=WURIHGQASBLSVLZJVM5EJHWD7U4VFLLPGJIFYNOK6ALA5LYEQDSA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.074628196 (538h33m42.945711252s) [noise=true , address=145.79.227.47:4295 , piece_id=OM7B75MCPM7L3WP5MLQ7MAVBZ2NJUSISOWD56RSWI2AV2OWVDRTA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.080495740 (538h33m42.951578775s) [noise=true , address=5.231.73.5:4016 , piece_id=YCNKQEKJQSJHKKGCYWLOLEBKIVPJHZIJ6TYEG6DF4TRXMZ3ZTTIQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.193955176 (538h33m43.065038241s) [noise=true , address=77.90.31.10:4551 , piece_id=KBPO4MUDASHKMPRNJELQWLNMJNQBB3PZUOVNCEPP2CYVHU6X2PPA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.301513865 (538h33m43.1725969s) [noise=true , address=77.90.31.15:4730 , piece_id=EN7IGRJZ2WFN4HXJPMNLLXDUZPIN7KDUGKPPLJH7CSA4SDOUM2JA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.305020622 (538h33m43.176103657s) [noise=true , address=77.90.31.19:4862 , piece_id=PSBBBSGY7THVLMXJQCADIE7VYZ33OIIDVAEQWG3RWMNA6RSKELNQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.312187619 (538h33m43.183270654s) [noise=true , address=77.90.31.14:4672 , piece_id=BZG2Y5X47J2WVFF5LBHEJZNUD2OQ3NOTVPTCAVT4TLFQB5FWESLA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.319277390 (538h33m43.190360425s) [noise=true , address=77.90.31.21:4882 , piece_id=WQAGQTGLU5HRKEET36W34MQP4NPGSVCA4QZ6YEGVAJWBBPLZOUQQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.948916959 (538h33m43.820000004s) [noise=true , address=145.79.227.34:4915 , piece_id=RMXYZGGJA5EPJLQD6SQG677HFLOQWRY2HFTGEOICYAQG6QR557RQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.989823331 (538h33m43.860906366s) [noise=true , address=145.79.227.44:4212 , piece_id=EFTTQM6323MUQ5WTGY6DFBTF3TGRM6XZWXIGT6DM4VS7GRGM4C2Q, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:44.123218949 (538h33m43.994301984s) [noise=true , address=145.79.227.43:4161 , piece_id=ZN3T6JSPHJ57X4YYG5MHRJYYOFPH2T2CSGJ34MJQU55HDC7CMOXQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:44.123238386 (538h33m43.994321421s) [seg_pos={0 0}, upload=472313]: Segment reached optimal threshold of 14 pieces. Dec 28 02:02:44.123431754 (538h33m43.994514789s) [noise=true , address=145.79.227.33:4899 , piece_id=JFEXXK22EATV7UUH7LLXTFEPIUBW7EWSNYFXLE4AO5F6SLYFTGBA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b4pZVQRnjcQNn45VTgdsz5fpjYPMAzSXGR5KJ5BP9XHMkRnRw): protocol: expected piece hash; context canceled; EOF Dec 28 02:02:44.123437586 (538h33m43.994520621s) [noise=true , address=145.79.227.33:4899 , piece_id=JFEXXK22EATV7UUH7LLXTFEPIUBW7EWSNYFXLE4AO5F6SLYFTGBA, seg_pos={0 0}, upload=472313]: Long tail cancellation due to optimal threshold Dec 28 02:02:44.123631064 (538h33m43.994714109s) [noise=true , address=77.90.31.18:4833 , piece_id=3E57LWLI3WHROFGMXVZ7LHDHO5O7453LWQXA3OEKM52B2NIYM2JA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FbVbo8kGy1aaFyCk2noiXNUJHtosFJHFykBMRecGd5Kj5gjEp): protocol: expected piece hash; context canceled; EOF Dec 28 02:02:44.123675267 (538h33m43.994758313s) [noise=true , address=77.90.31.18:4833 , piece_id=3E57LWLI3WHROFGMXVZ7LHDHO5O7453LWQXA3OEKM52B2NIYM2JA, seg_pos={0 0}, upload=472313]: Long tail cancellation due to optimal threshold Dec 28 02:02:44.123774066 (538h33m43.994857101s) [noise=true , address=77.90.31.10:4524 , piece_id=X2VCBS4PTPPWI6U64T4FVIULRLCDY3XUOJTXJNP34OHSO54ONIWA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe): protocol: expected piece hash; context canceled; EOF Dec 28 02:02:44.123781761 (538h33m43.994864796s) [noise=true , address=77.90.31.10:4524 , piece_id=X2VCBS4PTPPWI6U64T4FVIULRLCDY3XUOJTXJNP34OHSO54ONIWA, seg_pos={0 0}, upload=472313]: Long tail cancellation due to optimal threshold Dec 28 02:02:44.123923341 (538h33m43.995006376s) [noise=true , address=145.79.227.35:4503 , piece_id=MDYJ2TROT4FHLX4XHKHEQACC7ZXHZCQ5MUPJL5EDS3F2YMXSUGUA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ctmSF1aMQasuXByFW8DppDantREmmWL2WUjmKGW5ZULVePyvm): protocol: expected piece hash; context canceled; EOF Dec 28 02:02:44.123928591 (538h33m43.995011636s) [noise=true , address=145.79.227.35:4503 , piece_id=MDYJ2TROT4FHLX4XHKHEQACC7ZXHZCQ5MUPJL5EDS3F2YMXSUGUA, seg_pos={0 0}, upload=472313]: Long tail cancellation due to optimal threshold Dec 28 02:02:44.123944731 (538h33m43.995027766s) [seg_pos={0 0}, upload=472313]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 02:02:44.124017179 (538h33m43.995100214s) [upload=472313]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:02:44.124022469 (538h33m43.995105504s) [upload=472313]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:02:44.409047274 (538h33m44.280130349s) [upload=472313]: Done uploading segments... Dec 28 02:02:44.409073354 (538h33m44.280156389s) [upload=472313]: Upload finished. err: <nil> Duration: 44.168141949s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................................................X....] duration: 3.430657897s trace: https://jaeger.dev.storj.io/trace/7fc142ad3d711e34 FAIL (2025-12-28 02:08:51 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.430680621s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 32 out of 60 is failed (threshold=3) [XX.X.XX....X.XX.XXXXX..X.XXX.X.XX.XX.X..X.X.X.X.X..X...XX..X] duration: 30.005107137s trace: https://jaeger.dev.storj.io/trace/2140711179d51097 FAIL (2025-12-28 02:08:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: H4ERAYCDVULAC733FW64VTEB6UY6J4WV4R4NEAOTGBMNAWUVIIDA) context deadline exceeded; piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: ZPNKHLD4BEQV6MNLI6LTRZE5SUYLUJ747AQTTILBKKCBOV2CZURA) context deadline exceeded; piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: WMJH4AOJKOKVZVAGPOC4RJG6WYWKNMQV65NLIRTH2WKAWN327NZQ) context deadline exceeded ======LOGS====== Dec 28 02:08:08.096883081 (538h39m7.967966126s) [upload=472390]: Starting upload Dec 28 02:08:08.096959186 (538h39m7.968042221s) [upload=472390]: Done starting upload Dec 28 02:08:08.097115213 (538h39m7.968198248s) [upload=472390]: Uploading segments... Dec 28 02:08:08.097228669 (538h39m7.968311714s) [upload=472390]: Got next segment. Inline: false Dec 28 02:08:08.097274376 (538h39m7.968357422s) [upload=472390]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 02:08:08.097279767 (538h39m7.968362802s) [upload=472390]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 02:08:08.343782458 (538h39m8.214865503s) [seg_pos={0 0}, upload=472390]: Begin upload segment... Dec 28 02:08:08.343845468 (538h39m8.214928513s) [seg_pos={0 0}, upload=472390]: Done begin upload segment. Dec 28 02:08:08.343887899 (538h39m8.214970934s) [noise=true , address=145.79.227.15:4874 , piece_id=KCKZAV3ENC6UJ44UYKMNLAVMHZ73FDVZBLAUKYN4QNIIAIT4SWNQ, seg_pos={0 0}, upload=472390]: Uploading piece... Dec 28 02:08:08.343910532 (538h39m8.214993567s) [noise=true , address=45.91.116.13:4153 , piece_id=TAVZ43N4CXCNFRCLII4BO2U3FX74QXO7EE7CTTT2JXUZQGQVCJMA, seg_pos={0 0}, upload=472390]: Uploading piece... Dec 28 02:08:08.344482381 (538h39m8.215565426s) [noise=true , address=45.91.116.4:4937 , piece_id=YQFM4J43LA32NYG7JKRB3RS4ZWAQV4W4C5A2N75BJLGMDJGIJZ4A, seg_pos={0 0}, upload=472390]: Uploading piece... Dec 28 02:08:08.344492741 (538h39m8.215575786s) [noise=true , address=145.79.227.52:4446 , piece_id=6S5F5BMRM3CSI2S5Z6WWR7RXBDF6PWPSEPKGICKVRQUD7BMAAEBQ, seg_pos={0 0}, upload=472390]: Uploading piece... Dec 28 02:08:08.344847446 (538h39m8.215930481s) [noise=true , address=145.79.227.7:4293 , piece_id=QPRU6FUMCQF2V2MZU6CCGYCJGBL6IJAW2SGM4BT3ZYWKIRKMBI4A, seg_pos={0 0}, upload=472390]: Uploading piece... Dec 28 02:08:08.344877273 (538h39m8.215960308s) [noise=true , address=77.90.31.18:4808 , piece_id=WMJH4AOJKOKVZVAGPOC4RJG6WYWKNMQV65NLIRTH2WKAWN327NZQ, seg_pos={0 0}, upload=472390]: Uploading piece... Dec 28 02:08:08.345186011 (538h39m8.216269046s) [noise=true , address=145.79.227.27:4691 , piece_id=CYAKJNKCSXHQMJB7SEAFJOO73PQOFYVSS4ZSZVD46LIQB2BGGSUA, seg_pos={0 0}, upload=472390]: Uploading piece... Dec 28 02:08:08.345294017 (538h39m8.216377052s) [noise=true , address=145.79.227.51:4411 , piece_id=YGH4ONL6VBFVOKRT3TGHUPCKR5ULDLLN5U4CWMG6VCY3Z7WBSIAA, seg_pos={0 0}, upload=472390]: Uploading piece... Dec 28 02:08:08.345707364 (538h39m8.216790409s) [noise=true , address=77.90.31.19:4844 , piece_id=6YP7YMB5XVLEJY73QFDPL2QG73GX2YFMLDTKBNKH66WWBDEA6Q6A, seg_pos={0 0}, upload=472390]: Uploading piece... Dec 28 02:08:08.345871506 (538h39m8.216954551s) [noise=true , address=77.90.31.10:4519 , piece_id=ZPNKHLD4BEQV6MNLI6LTRZE5SUYLUJ747AQTTILBKKCBOV2CZURA, seg_pos={0 0}, upload=472390]: Uploading piece... Dec 28 02:08:08.346155416 (538h39m8.217238461s) [noise=true , address=145.79.227.45:4229 , piece_id=SBXMFCXNFXKIFN6HFI7IDAHZNM6BWZHIFQAFJTMC22HCWPVXRRXA, seg_pos={0 0}, upload=472390]: Uploading piece... Dec 28 02:08:08.346543847 (538h39m8.217626892s) [noise=true , address=77.90.31.21:4877 , piece_id=2MHERE2XKTVEAX6LRKYYF4FV7JOSKAF3FJEAV5ZWC4ODXLQP3KBQ, seg_pos={0 0}, upload=472390]: Uploading piece... Dec 28 02:08:08.347125494 (538h39m8.218208529s) [noise=true , address=145.79.227.32:4836 , piece_id=DRKL7KL5VZXHUKE2QGM6YNIBA3SGOKMASLSTAYQWGVO27KJQ7JKQ, seg_pos={0 0}, upload=472390]: Uploading piece... Dec 28 02:08:08.347281912 (538h39m8.218364947s) [noise=true , address=145.79.227.47:4266 , piece_id=E3ZBG7GUMC4JIR7CZXAC56TMDSTOIPZSKOPRBTSADY3X2ANLXTZA, seg_pos={0 0}, upload=472390]: Uploading piece... Dec 28 02:08:08.347741938 (538h39m8.218824983s) [noise=true , address=45.91.116.6:4973 , piece_id=H4ERAYCDVULAC733FW64VTEB6UY6J4WV4R4NEAOTGBMNAWUVIIDA, seg_pos={0 0}, upload=472390]: Uploading piece... Dec 28 02:08:08.348318466 (538h39m8.219401501s) [noise=true , address=77.90.31.15:4700 , piece_id=ELIKNM64ZGW35I6N7ZOFHRCMOK43CL4W2BWUGCN6TEXYI23V3KPA, seg_pos={0 0}, upload=472390]: Uploading piece... Dec 28 02:08:10.056936841 (538h39m9.928019876s) [upload=472390]: Next returned nil segment Dec 28 02:08:10.056951479 (538h39m9.928034524s) [upload=472390]: Waiting for error group managing segments... Dec 28 02:08:10.333760791 (538h39m10.204843836s) [noise=true , address=145.79.227.15:4874 , piece_id=KCKZAV3ENC6UJ44UYKMNLAVMHZ73FDVZBLAUKYN4QNIIAIT4SWNQ, seg_pos={0 0}, upload=472390]: Done uploading piece. err: <nil> Dec 28 02:08:10.338359948 (538h39m10.209442983s) [noise=true , address=145.79.227.7:4293 , piece_id=QPRU6FUMCQF2V2MZU6CCGYCJGBL6IJAW2SGM4BT3ZYWKIRKMBI4A, seg_pos={0 0}, upload=472390]: Done uploading piece. err: <nil> Dec 28 02:08:10.349637483 (538h39m10.220720518s) [noise=true , address=145.79.227.27:4691 , piece_id=CYAKJNKCSXHQMJB7SEAFJOO73PQOFYVSS4ZSZVD46LIQB2BGGSUA, seg_pos={0 0}, upload=472390]: Done uploading piece. err: <nil> Dec 28 02:08:10.365965794 (538h39m10.237048839s) [noise=true , address=145.79.227.47:4266 , piece_id=E3ZBG7GUMC4JIR7CZXAC56TMDSTOIPZSKOPRBTSADY3X2ANLXTZA, seg_pos={0 0}, upload=472390]: Done uploading piece. err: <nil> Dec 28 02:08:10.374047483 (538h39m10.245130518s) [noise=true , address=145.79.227.51:4411 , piece_id=YGH4ONL6VBFVOKRT3TGHUPCKR5ULDLLN5U4CWMG6VCY3Z7WBSIAA, seg_pos={0 0}, upload=472390]: Done uploading piece. err: <nil> Dec 28 02:08:10.378783000 (538h39m10.249866035s) [noise=true , address=145.79.227.52:4446 , piece_id=6S5F5BMRM3CSI2S5Z6WWR7RXBDF6PWPSEPKGICKVRQUD7BMAAEBQ, seg_pos={0 0}, upload=472390]: Done uploading piece. err: <nil> Dec 28 02:08:10.382402763 (538h39m10.253485798s) [noise=true , address=145.79.227.45:4229 , piece_id=SBXMFCXNFXKIFN6HFI7IDAHZNM6BWZHIFQAFJTMC22HCWPVXRRXA, seg_pos={0 0}, upload=472390]: Done uploading piece. err: <nil> Dec 28 02:08:10.384796972 (538h39m10.255880007s) [noise=true , address=145.79.227.32:4836 , piece_id=DRKL7KL5VZXHUKE2QGM6YNIBA3SGOKMASLSTAYQWGVO27KJQ7JKQ, seg_pos={0 0}, upload=472390]: Done uploading piece. err: <nil> Dec 28 02:08:10.583803704 (538h39m10.454886749s) [noise=true , address=45.91.116.6:4973 , piece_id=H4ERAYCDVULAC733FW64VTEB6UY6J4WV4R4NEAOTGBMNAWUVIIDA, seg_pos={0 0}, upload=472390]: Done uploading piece. err: <nil> Dec 28 02:08:10.644860707 (538h39m10.515943752s) [noise=true , address=77.90.31.15:4700 , piece_id=ELIKNM64ZGW35I6N7ZOFHRCMOK43CL4W2BWUGCN6TEXYI23V3KPA, seg_pos={0 0}, upload=472390]: Done uploading piece. err: <nil> Dec 28 02:08:10.712079693 (538h39m10.583162738s) [noise=true , address=77.90.31.19:4844 , piece_id=6YP7YMB5XVLEJY73QFDPL2QG73GX2YFMLDTKBNKH66WWBDEA6Q6A, seg_pos={0 0}, upload=472390]: Done uploading piece. err: <nil> Dec 28 02:08:11.233977033 (538h39m11.105060078s) [noise=true , address=77.90.31.18:4808 , piece_id=WMJH4AOJKOKVZVAGPOC4RJG6WYWKNMQV65NLIRTH2WKAWN327NZQ, seg_pos={0 0}, upload=472390]: Done uploading piece. err: <nil> Dec 28 02:08:11.236266232 (538h39m11.107349287s) [noise=true , address=77.90.31.10:4519 , piece_id=ZPNKHLD4BEQV6MNLI6LTRZE5SUYLUJ747AQTTILBKKCBOV2CZURA, seg_pos={0 0}, upload=472390]: Done uploading piece. err: <nil> Dec 28 02:08:11.236286351 (538h39m11.107369386s) [seg_pos={0 0}, upload=472390]: Segment reached optimal threshold of 13 pieces. Dec 28 02:08:11.236504065 (538h39m11.10758711s) [noise=true , address=77.90.31.21:4877 , piece_id=2MHERE2XKTVEAX6LRKYYF4FV7JOSKAF3FJEAV5ZWC4ODXLQP3KBQ, seg_pos={0 0}, upload=472390]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 28 02:08:11.236511469 (538h39m11.107594514s) [noise=true , address=77.90.31.21:4877 , piece_id=2MHERE2XKTVEAX6LRKYYF4FV7JOSKAF3FJEAV5ZWC4ODXLQP3KBQ, seg_pos={0 0}, upload=472390]: Long tail cancellation due to optimal threshold Dec 28 02:08:11.236563629 (538h39m11.107646664s) [noise=true , address=45.91.116.4:4937 , piece_id=YQFM4J43LA32NYG7JKRB3RS4ZWAQV4W4C5A2N75BJLGMDJGIJZ4A, seg_pos={0 0}, upload=472390]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 28 02:08:11.236587634 (538h39m11.107670669s) [noise=true , address=45.91.116.4:4937 , piece_id=YQFM4J43LA32NYG7JKRB3RS4ZWAQV4W4C5A2N75BJLGMDJGIJZ4A, seg_pos={0 0}, upload=472390]: Long tail cancellation due to optimal threshold Dec 28 02:08:11.236771925 (538h39m11.10785498s) [noise=true , address=45.91.116.13:4153 , piece_id=TAVZ43N4CXCNFRCLII4BO2U3FX74QXO7EE7CTTT2JXUZQGQVCJMA, seg_pos={0 0}, upload=472390]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 28 02:08:11.236780601 (538h39m11.107863647s) [noise=true , address=45.91.116.13:4153 , piece_id=TAVZ43N4CXCNFRCLII4BO2U3FX74QXO7EE7CTTT2JXUZQGQVCJMA, seg_pos={0 0}, upload=472390]: Long tail cancellation due to optimal threshold Dec 28 02:08:11.236800189 (538h39m11.107883234s) [seg_pos={0 0}, upload=472390]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 02:08:11.236879280 (538h39m11.107962325s) [upload=472390]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:08:11.236884599 (538h39m11.107967645s) [upload=472390]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:08:11.536994098 (538h39m11.408077143s) [upload=472390]: Done uploading segments... Dec 28 02:08:11.537017663 (538h39m11.408100708s) [upload=472390]: Upload finished. err: <nil> Duration: 30.005132283s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 40.13771839s trace: https://jaeger.dev.storj.io/trace/12a1d059b6f75af7 FAIL (2025-12-28 02:08:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1.145865281 (538h33m41.016948316s) [noise=true , address=77.90.31.10:4551 , piece_id=KBPO4MUDASHKMPRNJELQWLNMJNQBB3PZUOVNCEPP2CYVHU6X2PPA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.146408725 (538h33m41.01749177s) [noise=true , address=145.79.227.15:4908 , piece_id=I2FSDNVYBFZKRXZNUHH2APY7N2PI7DPAKFKGXRJW2UFU2PBKSSNQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.146778139 (538h33m41.017861184s) [noise=true , address=145.79.227.43:4161 , piece_id=ZN3T6JSPHJ57X4YYG5MHRJYYOFPH2T2CSGJ34MJQU55HDC7CMOXQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.147137874 (538h33m41.018220919s) [noise=true , address=77.90.31.21:4882 , piece_id=WQAGQTGLU5HRKEET36W34MQP4NPGSVCA4QZ6YEGVAJWBBPLZOUQQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.147426784 (538h33m41.018509819s) [noise=true , address=145.79.227.35:4503 , piece_id=MDYJ2TROT4FHLX4XHKHEQACC7ZXHZCQ5MUPJL5EDS3F2YMXSUGUA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.147466109 (538h33m41.018549154s) [noise=true , address=77.90.31.15:4730 , piece_id=EN7IGRJZ2WFN4HXJPMNLLXDUZPIN7KDUGKPPLJH7CSA4SDOUM2JA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.147817368 (538h33m41.018900413s) [noise=true , address=77.90.31.18:4833 , piece_id=3E57LWLI3WHROFGMXVZ7LHDHO5O7453LWQXA3OEKM52B2NIYM2JA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.147860690 (538h33m41.018943745s) [noise=true , address=145.79.227.44:4212 , piece_id=EFTTQM6323MUQ5WTGY6DFBTF3TGRM6XZWXIGT6DM4VS7GRGM4C2Q, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148163427 (538h33m41.019246472s) [noise=true , address=77.90.31.19:4862 , piece_id=PSBBBSGY7THVLMXJQCADIE7VYZ33OIIDVAEQWG3RWMNA6RSKELNQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148189276 (538h33m41.019272321s) [noise=true , address=77.90.31.10:4524 , piece_id=X2VCBS4PTPPWI6U64T4FVIULRLCDY3XUOJTXJNP34OHSO54ONIWA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148496531 (538h33m41.019579566s) [noise=true , address=145.79.227.23:4556 , piece_id=AKBXRSU5PM5QLHGHE3VXTJ3ZH23BECPGBLQSVZK6XVIYJGVDXT4Q, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148527380 (538h33m41.019610425s) [noise=true , address=145.79.227.33:4899 , piece_id=JFEXXK22EATV7UUH7LLXTFEPIUBW7EWSNYFXLE4AO5F6SLYFTGBA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148801512 (538h33m41.019884547s) [noise=true , address=77.90.31.14:4672 , piece_id=BZG2Y5X47J2WVFF5LBHEJZNUD2OQ3NOTVPTCAVT4TLFQB5FWESLA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.148894920 (538h33m41.019977965s) [noise=true , address=5.231.73.5:4016 , piece_id=YCNKQEKJQSJHKKGCYWLOLEBKIVPJHZIJ6TYEG6DF4TRXMZ3ZTTIQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.149118806 (538h33m41.020201851s) [noise=true , address=145.79.227.47:4295 , piece_id=OM7B75MCPM7L3WP5MLQ7MAVBZ2NJUSISOWD56RSWI2AV2OWVDRTA, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.149274993 (538h33m41.020358038s) [noise=true , address=145.79.227.50:4400 , piece_id=HZAT7YAVZYWK52FE4TIC6OA37U4VX5B35H55WMBO3MMWNIKTJLJQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:41.149625842 (538h33m41.020708877s) [noise=true , address=145.79.227.34:4915 , piece_id=RMXYZGGJA5EPJLQD6SQG677HFLOQWRY2HFTGEOICYAQG6QR557RQ, seg_pos={0 0}, upload=472313]: Uploading piece... Dec 28 02:02:42.618098549 (538h33m42.489181584s) [upload=472313]: Next returned nil segment Dec 28 02:02:42.618120911 (538h33m42.489203946s) [upload=472313]: Waiting for error group managing segments... Dec 28 02:02:43.021704832 (538h33m42.892787877s) [noise=true , address=145.79.227.23:4556 , piece_id=AKBXRSU5PM5QLHGHE3VXTJ3ZH23BECPGBLQSVZK6XVIYJGVDXT4Q, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.027980492 (538h33m42.899063537s) [noise=true , address=145.79.227.50:4400 , piece_id=HZAT7YAVZYWK52FE4TIC6OA37U4VX5B35H55WMBO3MMWNIKTJLJQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.043329239 (538h33m42.914412284s) [noise=true , address=145.79.227.15:4908 , piece_id=I2FSDNVYBFZKRXZNUHH2APY7N2PI7DPAKFKGXRJW2UFU2PBKSSNQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.056308314 (538h33m42.927391349s) [noise=true , address=145.79.227.49:4347 , piece_id=WURIHGQASBLSVLZJVM5EJHWD7U4VFLLPGJIFYNOK6ALA5LYEQDSA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.074628196 (538h33m42.945711252s) [noise=true , address=145.79.227.47:4295 , piece_id=OM7B75MCPM7L3WP5MLQ7MAVBZ2NJUSISOWD56RSWI2AV2OWVDRTA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.080495740 (538h33m42.951578775s) [noise=true , address=5.231.73.5:4016 , piece_id=YCNKQEKJQSJHKKGCYWLOLEBKIVPJHZIJ6TYEG6DF4TRXMZ3ZTTIQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.193955176 (538h33m43.065038241s) [noise=true , address=77.90.31.10:4551 , piece_id=KBPO4MUDASHKMPRNJELQWLNMJNQBB3PZUOVNCEPP2CYVHU6X2PPA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.301513865 (538h33m43.1725969s) [noise=true , address=77.90.31.15:4730 , piece_id=EN7IGRJZ2WFN4HXJPMNLLXDUZPIN7KDUGKPPLJH7CSA4SDOUM2JA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.305020622 (538h33m43.176103657s) [noise=true , address=77.90.31.19:4862 , piece_id=PSBBBSGY7THVLMXJQCADIE7VYZ33OIIDVAEQWG3RWMNA6RSKELNQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.312187619 (538h33m43.183270654s) [noise=true , address=77.90.31.14:4672 , piece_id=BZG2Y5X47J2WVFF5LBHEJZNUD2OQ3NOTVPTCAVT4TLFQB5FWESLA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.319277390 (538h33m43.190360425s) [noise=true , address=77.90.31.21:4882 , piece_id=WQAGQTGLU5HRKEET36W34MQP4NPGSVCA4QZ6YEGVAJWBBPLZOUQQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.948916959 (538h33m43.820000004s) [noise=true , address=145.79.227.34:4915 , piece_id=RMXYZGGJA5EPJLQD6SQG677HFLOQWRY2HFTGEOICYAQG6QR557RQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:43.989823331 (538h33m43.860906366s) [noise=true , address=145.79.227.44:4212 , piece_id=EFTTQM6323MUQ5WTGY6DFBTF3TGRM6XZWXIGT6DM4VS7GRGM4C2Q, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:44.123218949 (538h33m43.994301984s) [noise=true , address=145.79.227.43:4161 , piece_id=ZN3T6JSPHJ57X4YYG5MHRJYYOFPH2T2CSGJ34MJQU55HDC7CMOXQ, seg_pos={0 0}, upload=472313]: Done uploading piece. err: <nil> Dec 28 02:02:44.123238386 (538h33m43.994321421s) [seg_pos={0 0}, upload=472313]: Segment reached optimal threshold of 14 pieces. Dec 28 02:02:44.123431754 (538h33m43.994514789s) [noise=true , address=145.79.227.33:4899 , piece_id=JFEXXK22EATV7UUH7LLXTFEPIUBW7EWSNYFXLE4AO5F6SLYFTGBA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b4pZVQRnjcQNn45VTgdsz5fpjYPMAzSXGR5KJ5BP9XHMkRnRw): protocol: expected piece hash; context canceled; EOF Dec 28 02:02:44.123437586 (538h33m43.994520621s) [noise=true , address=145.79.227.33:4899 , piece_id=JFEXXK22EATV7UUH7LLXTFEPIUBW7EWSNYFXLE4AO5F6SLYFTGBA, seg_pos={0 0}, upload=472313]: Long tail cancellation due to optimal threshold Dec 28 02:02:44.123631064 (538h33m43.994714109s) [noise=true , address=77.90.31.18:4833 , piece_id=3E57LWLI3WHROFGMXVZ7LHDHO5O7453LWQXA3OEKM52B2NIYM2JA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FbVbo8kGy1aaFyCk2noiXNUJHtosFJHFykBMRecGd5Kj5gjEp): protocol: expected piece hash; context canceled; EOF Dec 28 02:02:44.123675267 (538h33m43.994758313s) [noise=true , address=77.90.31.18:4833 , piece_id=3E57LWLI3WHROFGMXVZ7LHDHO5O7453LWQXA3OEKM52B2NIYM2JA, seg_pos={0 0}, upload=472313]: Long tail cancellation due to optimal threshold Dec 28 02:02:44.123774066 (538h33m43.994857101s) [noise=true , address=77.90.31.10:4524 , piece_id=X2VCBS4PTPPWI6U64T4FVIULRLCDY3XUOJTXJNP34OHSO54ONIWA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe): protocol: expected piece hash; context canceled; EOF Dec 28 02:02:44.123781761 (538h33m43.994864796s) [noise=true , address=77.90.31.10:4524 , piece_id=X2VCBS4PTPPWI6U64T4FVIULRLCDY3XUOJTXJNP34OHSO54ONIWA, seg_pos={0 0}, upload=472313]: Long tail cancellation due to optimal threshold Dec 28 02:02:44.123923341 (538h33m43.995006376s) [noise=true , address=145.79.227.35:4503 , piece_id=MDYJ2TROT4FHLX4XHKHEQACC7ZXHZCQ5MUPJL5EDS3F2YMXSUGUA, seg_pos={0 0}, upload=472313]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ctmSF1aMQasuXByFW8DppDantREmmWL2WUjmKGW5ZULVePyvm): protocol: expected piece hash; context canceled; EOF Dec 28 02:02:44.123928591 (538h33m43.995011636s) [noise=true , address=145.79.227.35:4503 , piece_id=MDYJ2TROT4FHLX4XHKHEQACC7ZXHZCQ5MUPJL5EDS3F2YMXSUGUA, seg_pos={0 0}, upload=472313]: Long tail cancellation due to optimal threshold Dec 28 02:02:44.123944731 (538h33m43.995027766s) [seg_pos={0 0}, upload=472313]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 02:02:44.124017179 (538h33m43.995100214s) [upload=472313]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:02:44.124022469 (538h33m43.995105504s) [upload=472313]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:02:44.409047274 (538h33m44.280130349s) [upload=472313]: Done uploading segments... Dec 28 02:02:44.409073354 (538h33m44.280156389s) [upload=472313]: Upload finished. err: <nil> Duration: 40.137756513s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 32 out of 60 is failed (threshold=3) [XXX.X.XX....X.XX.XXXXX..X.XXX.X.XX.XX.X..X.X.X.X.X..X...XX..] duration: 18.434621714s trace: https://jaeger.dev.storj.io/trace/76e37aad528c0be4 FAIL (2025-12-28 02:08:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: QUEWIBEIARLTC2I4X7DYROTND45JEFKT46XHXUMZNLPTUPPRASKQ) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: SVFUZUJX2AFGM63NAPZELLFWKZXKLWHKZO3NCHBUOBBGDJMYAV4Q) context deadline exceeded; piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: YU44UT26XUN7BHIDE24PLQZ7DIP4P4CWIXT3PVC4DUYVBIYXMM4A) context deadline exceeded ======LOGS====== 26 (538h37m25.081596381s) [upload=472366]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 02:06:25.457825870 (538h37m25.328908915s) [seg_pos={0 0}, upload=472366]: Begin upload segment... Dec 28 02:06:25.457886896 (538h37m25.328969941s) [seg_pos={0 0}, upload=472366]: Done begin upload segment. Dec 28 02:06:25.457921783 (538h37m25.329004828s) [noise=true , address=45.91.116.10:4082 , piece_id=EBRIPLQD3K3OKFGJY4HP6WR63BTPV442H2FXLOHEBKXTZXUI3AHA, seg_pos={0 0}, upload=472366]: Uploading piece... Dec 28 02:06:25.458423057 (538h37m25.329506102s) [noise=true , address=145.79.227.36:4519 , piece_id=P6K24YRNO4AS2RZ5EOW2AR62VXCYG3NXHG7DKFV4ZETNTVSLQ7RQ, seg_pos={0 0}, upload=472366]: Uploading piece... Dec 28 02:06:25.458717086 (538h37m25.329800121s) [noise=true , address=145.79.227.38:4977 , piece_id=PB6TUSJR5RRLE3ZYYHCIOSED7AUYBEBNGWWBEELNSEQIPBEO5LHA, seg_pos={0 0}, upload=472366]: Uploading piece... Dec 28 02:06:25.459198994 (538h37m25.330282039s) [noise=true , address=145.79.227.42:4121 , piece_id=5HWESY6JNEEED2JTXPKAL2DAQXVQSFFBGGU7OBM3RHFY5BZ6RLKQ, seg_pos={0 0}, upload=472366]: Uploading piece... Dec 28 02:06:25.459289517 (538h37m25.330372562s) [noise=true , address=145.79.227.5:4223 , piece_id=3TWREZCDN652ZJCISV2WLTALSDAHUV2CZ2YPSANTK7CIR4HXQYSA, seg_pos={0 0}, upload=472366]: Uploading piece... Dec 28 02:06:25.459637540 (538h37m25.330720575s) [noise=true , address=145.79.227.11:4732 , piece_id=3KJFXY2YOFIQEPYKXNV2BWNKO5IFHAXOQVBYXANNWFLTSCSVOZTA, seg_pos={0 0}, upload=472366]: Uploading piece... Dec 28 02:06:25.459770723 (538h37m25.330853768s) [noise=true , address=45.91.116.5:4298 , piece_id=QUEWIBEIARLTC2I4X7DYROTND45JEFKT46XHXUMZNLPTUPPRASKQ, seg_pos={0 0}, upload=472366]: Uploading piece... Dec 28 02:06:25.460029616 (538h37m25.331112651s) [noise=true , address=145.79.227.52:4447 , piece_id=QGMJTUWZOYTR6BVRIGURBE6L7YLILV3CZ6G26SZ6WPDBD3UZW4GQ, seg_pos={0 0}, upload=472366]: Uploading piece... Dec 28 02:06:25.460164132 (538h37m25.331247167s) [noise=true , address=145.79.227.3:4151 , piece_id=4DZU4WML2HZPB32WPQFXCSE526XIZD47PWPAOYPQEABHMTPX2TSQ, seg_pos={0 0}, upload=472366]: Uploading piece... Dec 28 02:06:25.460400853 (538h37m25.331483898s) [noise=true , address=45.91.116.11:4117 , piece_id=SVFUZUJX2AFGM63NAPZELLFWKZXKLWHKZO3NCHBUOBBGDJMYAV4Q, seg_pos={0 0}, upload=472366]: Uploading piece... Dec 28 02:06:25.460677590 (538h37m25.331760625s) [noise=true , address=45.91.116.13:4156 , piece_id=HS5XHKMOB5CKFJ7QB5PUSXK273WR77AQZI35AGXORYRCMNXGLXOQ, seg_pos={0 0}, upload=472366]: Uploading piece... Dec 28 02:06:25.460774515 (538h37m25.33185755s) [noise=true , address=77.90.31.21:4878 , piece_id=YU44UT26XUN7BHIDE24PLQZ7DIP4P4CWIXT3PVC4DUYVBIYXMM4A, seg_pos={0 0}, upload=472366]: Uploading piece... Dec 28 02:06:25.461130663 (538h37m25.332213708s) [noise=true , address=45.91.116.6:4974 , piece_id=ZI2WE7IT42PSDZ2O27YOGGTESZHI3RQZ7IIMC4GD3FGJWZK2RVKQ, seg_pos={0 0}, upload=472366]: Uploading piece... Dec 28 02:06:25.461155529 (538h37m25.332238575s) [noise=true , address=45.91.116.4:4939 , piece_id=BFMRZAEILMEWURGHKTEXWPINPXBI7NF6WRRBNTVXVPQ5TOVFZBFQ, seg_pos={0 0}, upload=472366]: Uploading piece... Dec 28 02:06:25.461579478 (538h37m25.332662523s) [noise=true , address=145.79.227.46:4602 , piece_id=VWZDLIRLZLBTFA6SGUAAFSYDQID66H3SIUMP4Z53JPUS7Y6DAPIA, seg_pos={0 0}, upload=472366]: Uploading piece... Dec 28 02:06:25.461672424 (538h37m25.332755459s) [noise=true , address=145.79.227.37:4565 , piece_id=DK542XMJJALTFK2DLLYALZLFJDGZFRYVHNFNBRZGPHZ4QKGCX4KQ, seg_pos={0 0}, upload=472366]: Uploading piece... Dec 28 02:06:27.169998904 (538h37m27.041081939s) [upload=472366]: Next returned nil segment Dec 28 02:06:27.170020946 (538h37m27.041103991s) [upload=472366]: Waiting for error group managing segments... Dec 28 02:06:27.463469060 (538h37m27.334552105s) [noise=true , address=145.79.227.36:4519 , piece_id=P6K24YRNO4AS2RZ5EOW2AR62VXCYG3NXHG7DKFV4ZETNTVSLQ7RQ, seg_pos={0 0}, upload=472366]: Done uploading piece. err: <nil> Dec 28 02:06:27.465105817 (538h37m27.336188852s) [noise=true , address=145.79.227.11:4732 , piece_id=3KJFXY2YOFIQEPYKXNV2BWNKO5IFHAXOQVBYXANNWFLTSCSVOZTA, seg_pos={0 0}, upload=472366]: Done uploading piece. err: <nil> Dec 28 02:06:27.472501379 (538h37m27.343584414s) [noise=true , address=145.79.227.37:4565 , piece_id=DK542XMJJALTFK2DLLYALZLFJDGZFRYVHNFNBRZGPHZ4QKGCX4KQ, seg_pos={0 0}, upload=472366]: Done uploading piece. err: <nil> Dec 28 02:06:27.486021213 (538h37m27.357104289s) [noise=true , address=145.79.227.52:4447 , piece_id=QGMJTUWZOYTR6BVRIGURBE6L7YLILV3CZ6G26SZ6WPDBD3UZW4GQ, seg_pos={0 0}, upload=472366]: Done uploading piece. err: <nil> Dec 28 02:06:27.486303821 (538h37m27.357386857s) [noise=true , address=145.79.227.46:4602 , piece_id=VWZDLIRLZLBTFA6SGUAAFSYDQID66H3SIUMP4Z53JPUS7Y6DAPIA, seg_pos={0 0}, upload=472366]: Done uploading piece. err: <nil> Dec 28 02:06:27.493106595 (538h37m27.36418964s) [noise=true , address=145.79.227.42:4121 , piece_id=5HWESY6JNEEED2JTXPKAL2DAQXVQSFFBGGU7OBM3RHFY5BZ6RLKQ, seg_pos={0 0}, upload=472366]: Done uploading piece. err: <nil> Dec 28 02:06:27.505383713 (538h37m27.376466758s) [noise=true , address=145.79.227.38:4977 , piece_id=PB6TUSJR5RRLE3ZYYHCIOSED7AUYBEBNGWWBEELNSEQIPBEO5LHA, seg_pos={0 0}, upload=472366]: Done uploading piece. err: <nil> Dec 28 02:06:27.692603320 (538h37m27.563686365s) [noise=true , address=45.91.116.6:4974 , piece_id=ZI2WE7IT42PSDZ2O27YOGGTESZHI3RQZ7IIMC4GD3FGJWZK2RVKQ, seg_pos={0 0}, upload=472366]: Done uploading piece. err: <nil> Dec 28 02:06:27.724516999 (538h37m27.595600034s) [noise=true , address=45.91.116.4:4939 , piece_id=BFMRZAEILMEWURGHKTEXWPINPXBI7NF6WRRBNTVXVPQ5TOVFZBFQ, seg_pos={0 0}, upload=472366]: Done uploading piece. err: <nil> Dec 28 02:06:27.753017799 (538h37m27.624100834s) [noise=true , address=45.91.116.5:4298 , piece_id=QUEWIBEIARLTC2I4X7DYROTND45JEFKT46XHXUMZNLPTUPPRASKQ, seg_pos={0 0}, upload=472366]: Done uploading piece. err: <nil> Dec 28 02:06:27.764587140 (538h37m27.635670175s) [noise=true , address=45.91.116.10:4082 , piece_id=EBRIPLQD3K3OKFGJY4HP6WR63BTPV442H2FXLOHEBKXTZXUI3AHA, seg_pos={0 0}, upload=472366]: Done uploading piece. err: <nil> Dec 28 02:06:27.800607448 (538h37m27.671690483s) [noise=true , address=45.91.116.11:4117 , piece_id=SVFUZUJX2AFGM63NAPZELLFWKZXKLWHKZO3NCHBUOBBGDJMYAV4Q, seg_pos={0 0}, upload=472366]: Done uploading piece. err: <nil> Dec 28 02:06:28.039356947 (538h37m27.910439992s) [noise=true , address=77.90.31.21:4878 , piece_id=YU44UT26XUN7BHIDE24PLQZ7DIP4P4CWIXT3PVC4DUYVBIYXMM4A, seg_pos={0 0}, upload=472366]: Done uploading piece. err: <nil> Dec 28 02:06:28.039378568 (538h37m27.910461613s) [seg_pos={0 0}, upload=472366]: Segment reached optimal threshold of 13 pieces. Dec 28 02:06:28.039572838 (538h37m27.910655883s) [noise=true , address=145.79.227.5:4223 , piece_id=3TWREZCDN652ZJCISV2WLTALSDAHUV2CZ2YPSANTK7CIR4HXQYSA, seg_pos={0 0}, upload=472366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE): protocol: write tcp 188.166.225.170:50600->145.79.227.5:4223: use of closed network connection; write tcp 188.166.225.170:50600->145.79.227.5:4223: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50600->145.79.227.5:4223: use of closed network connection Dec 28 02:06:28.039580082 (538h37m27.910663127s) [noise=true , address=145.79.227.5:4223 , piece_id=3TWREZCDN652ZJCISV2WLTALSDAHUV2CZ2YPSANTK7CIR4HXQYSA, seg_pos={0 0}, upload=472366]: Long tail cancellation due to optimal threshold Dec 28 02:06:28.039698728 (538h37m27.910781763s) [noise=true , address=145.79.227.3:4151 , piece_id=4DZU4WML2HZPB32WPQFXCSE526XIZD47PWPAOYPQEABHMTPX2TSQ, seg_pos={0 0}, upload=472366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: write tcp 188.166.225.170:58896->145.79.227.3:4151: use of closed network connection; write tcp 188.166.225.170:58896->145.79.227.3:4151: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58896->145.79.227.3:4151: use of closed network connection Dec 28 02:06:28.039704408 (538h37m27.910787453s) [noise=true , address=145.79.227.3:4151 , piece_id=4DZU4WML2HZPB32WPQFXCSE526XIZD47PWPAOYPQEABHMTPX2TSQ, seg_pos={0 0}, upload=472366]: Long tail cancellation due to optimal threshold Dec 28 02:06:28.039764934 (538h37m27.910847999s) [noise=true , address=45.91.116.13:4156 , piece_id=HS5XHKMOB5CKFJ7QB5PUSXK273WR77AQZI35AGXORYRCMNXGLXOQ, seg_pos={0 0}, upload=472366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: write tcp 188.166.225.170:48116->45.91.116.13:4156: use of closed network connection; write tcp 188.166.225.170:48116->45.91.116.13:4156: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48116->45.91.116.13:4156: use of closed network connection Dec 28 02:06:28.039776776 (538h37m27.910859821s) [noise=true , address=45.91.116.13:4156 , piece_id=HS5XHKMOB5CKFJ7QB5PUSXK273WR77AQZI35AGXORYRCMNXGLXOQ, seg_pos={0 0}, upload=472366]: Long tail cancellation due to optimal threshold Dec 28 02:06:28.039802115 (538h37m27.91088516s) [seg_pos={0 0}, upload=472366]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 02:06:28.039984792 (538h37m27.911067837s) [upload=472366]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:06:28.039992768 (538h37m27.911075803s) [upload=472366]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:06:28.336343158 (538h37m28.207426203s) [upload=472366]: Done uploading segments... Dec 28 02:06:28.336412921 (538h37m28.207495966s) [upload=472366]: Upload finished. err: <nil> Duration:* 18.434648014s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................................................X...] duration: 3.483476983s trace: https://jaeger.dev.storj.io/trace/7fc142ad3d711e34 FAIL (2025-12-28 02:07:59 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.483505647s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................................................X..] duration: 3.532210058s trace: https://jaeger.dev.storj.io/trace/7fc142ad3d711e34 FAIL (2025-12-28 02:07:35 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.532235617s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 32 out of 60 is failed (threshold=3) [.XXX.X.XX....X.XX.XXXXX..X.XXX.X.XX.XX.X..X.X.X.X.X..X...XX.] duration: 8.511448623s trace: https://jaeger.dev.storj.io/trace/76e37aad528c0be4 FAIL (2025-12-28 02:07:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: QUEWIBEIARLTC2I4X7DYROTND45JEFKT46XHXUMZNLPTUPPRASKQ) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: SVFUZUJX2AFGM63NAPZELLFWKZXKLWHKZO3NCHBUOBBGDJMYAV4Q) context deadline exceeded; piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: YU44UT26XUN7BHIDE24PLQZ7DIP4P4CWIXT3PVC4DUYVBIYXMM4A) context deadline exceeded ======LOGS====== 26 (538h37m25.081596381s) [upload=472366]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 02:06:25.457825870 (538h37m25.328908915s) [seg_pos={0 0}, upload=472366]: Begin upload segment... Dec 28 02:06:25.457886896 (538h37m25.328969941s) [seg_pos={0 0}, upload=472366]: Done begin upload segment. Dec 28 02:06:25.457921783 (538h37m25.329004828s) [noise=true , address=45.91.116.10:4082 , piece_id=EBRIPLQD3K3OKFGJY4HP6WR63BTPV442H2FXLOHEBKXTZXUI3AHA, seg_pos={0 0}, upload=472366]: Uploading piece... Dec 28 02:06:25.458423057 (538h37m25.329506102s) [noise=true , address=145.79.227.36:4519 , piece_id=P6K24YRNO4AS2RZ5EOW2AR62VXCYG3NXHG7DKFV4ZETNTVSLQ7RQ, seg_pos={0 0}, upload=472366]: Uploading piece... Dec 28 02:06:25.458717086 (538h37m25.329800121s) [noise=true , address=145.79.227.38:4977 , piece_id=PB6TUSJR5RRLE3ZYYHCIOSED7AUYBEBNGWWBEELNSEQIPBEO5LHA, seg_pos={0 0}, upload=472366]: Uploading piece... Dec 28 02:06:25.459198994 (538h37m25.330282039s) [noise=true , address=145.79.227.42:4121 , piece_id=5HWESY6JNEEED2JTXPKAL2DAQXVQSFFBGGU7OBM3RHFY5BZ6RLKQ, seg_pos={0 0}, upload=472366]: Uploading piece... Dec 28 02:06:25.459289517 (538h37m25.330372562s) [noise=true , address=145.79.227.5:4223 , piece_id=3TWREZCDN652ZJCISV2WLTALSDAHUV2CZ2YPSANTK7CIR4HXQYSA, seg_pos={0 0}, upload=472366]: Uploading piece... Dec 28 02:06:25.459637540 (538h37m25.330720575s) [noise=true , address=145.79.227.11:4732 , piece_id=3KJFXY2YOFIQEPYKXNV2BWNKO5IFHAXOQVBYXANNWFLTSCSVOZTA, seg_pos={0 0}, upload=472366]: Uploading piece... Dec 28 02:06:25.459770723 (538h37m25.330853768s) [noise=true , address=45.91.116.5:4298 , piece_id=QUEWIBEIARLTC2I4X7DYROTND45JEFKT46XHXUMZNLPTUPPRASKQ, seg_pos={0 0}, upload=472366]: Uploading piece... Dec 28 02:06:25.460029616 (538h37m25.331112651s) [noise=true , address=145.79.227.52:4447 , piece_id=QGMJTUWZOYTR6BVRIGURBE6L7YLILV3CZ6G26SZ6WPDBD3UZW4GQ, seg_pos={0 0}, upload=472366]: Uploading piece... Dec 28 02:06:25.460164132 (538h37m25.331247167s) [noise=true , address=145.79.227.3:4151 , piece_id=4DZU4WML2HZPB32WPQFXCSE526XIZD47PWPAOYPQEABHMTPX2TSQ, seg_pos={0 0}, upload=472366]: Uploading piece... Dec 28 02:06:25.460400853 (538h37m25.331483898s) [noise=true , address=45.91.116.11:4117 , piece_id=SVFUZUJX2AFGM63NAPZELLFWKZXKLWHKZO3NCHBUOBBGDJMYAV4Q, seg_pos={0 0}, upload=472366]: Uploading piece... Dec 28 02:06:25.460677590 (538h37m25.331760625s) [noise=true , address=45.91.116.13:4156 , piece_id=HS5XHKMOB5CKFJ7QB5PUSXK273WR77AQZI35AGXORYRCMNXGLXOQ, seg_pos={0 0}, upload=472366]: Uploading piece... Dec 28 02:06:25.460774515 (538h37m25.33185755s) [noise=true , address=77.90.31.21:4878 , piece_id=YU44UT26XUN7BHIDE24PLQZ7DIP4P4CWIXT3PVC4DUYVBIYXMM4A, seg_pos={0 0}, upload=472366]: Uploading piece... Dec 28 02:06:25.461130663 (538h37m25.332213708s) [noise=true , address=45.91.116.6:4974 , piece_id=ZI2WE7IT42PSDZ2O27YOGGTESZHI3RQZ7IIMC4GD3FGJWZK2RVKQ, seg_pos={0 0}, upload=472366]: Uploading piece... Dec 28 02:06:25.461155529 (538h37m25.332238575s) [noise=true , address=45.91.116.4:4939 , piece_id=BFMRZAEILMEWURGHKTEXWPINPXBI7NF6WRRBNTVXVPQ5TOVFZBFQ, seg_pos={0 0}, upload=472366]: Uploading piece... Dec 28 02:06:25.461579478 (538h37m25.332662523s) [noise=true , address=145.79.227.46:4602 , piece_id=VWZDLIRLZLBTFA6SGUAAFSYDQID66H3SIUMP4Z53JPUS7Y6DAPIA, seg_pos={0 0}, upload=472366]: Uploading piece... Dec 28 02:06:25.461672424 (538h37m25.332755459s) [noise=true , address=145.79.227.37:4565 , piece_id=DK542XMJJALTFK2DLLYALZLFJDGZFRYVHNFNBRZGPHZ4QKGCX4KQ, seg_pos={0 0}, upload=472366]: Uploading piece... Dec 28 02:06:27.169998904 (538h37m27.041081939s) [upload=472366]: Next returned nil segment Dec 28 02:06:27.170020946 (538h37m27.041103991s) [upload=472366]: Waiting for error group managing segments... Dec 28 02:06:27.463469060 (538h37m27.334552105s) [noise=true , address=145.79.227.36:4519 , piece_id=P6K24YRNO4AS2RZ5EOW2AR62VXCYG3NXHG7DKFV4ZETNTVSLQ7RQ, seg_pos={0 0}, upload=472366]: Done uploading piece. err: <nil> Dec 28 02:06:27.465105817 (538h37m27.336188852s) [noise=true , address=145.79.227.11:4732 , piece_id=3KJFXY2YOFIQEPYKXNV2BWNKO5IFHAXOQVBYXANNWFLTSCSVOZTA, seg_pos={0 0}, upload=472366]: Done uploading piece. err: <nil> Dec 28 02:06:27.472501379 (538h37m27.343584414s) [noise=true , address=145.79.227.37:4565 , piece_id=DK542XMJJALTFK2DLLYALZLFJDGZFRYVHNFNBRZGPHZ4QKGCX4KQ, seg_pos={0 0}, upload=472366]: Done uploading piece. err: <nil> Dec 28 02:06:27.486021213 (538h37m27.357104289s) [noise=true , address=145.79.227.52:4447 , piece_id=QGMJTUWZOYTR6BVRIGURBE6L7YLILV3CZ6G26SZ6WPDBD3UZW4GQ, seg_pos={0 0}, upload=472366]: Done uploading piece. err: <nil> Dec 28 02:06:27.486303821 (538h37m27.357386857s) [noise=true , address=145.79.227.46:4602 , piece_id=VWZDLIRLZLBTFA6SGUAAFSYDQID66H3SIUMP4Z53JPUS7Y6DAPIA, seg_pos={0 0}, upload=472366]: Done uploading piece. err: <nil> Dec 28 02:06:27.493106595 (538h37m27.36418964s) [noise=true , address=145.79.227.42:4121 , piece_id=5HWESY6JNEEED2JTXPKAL2DAQXVQSFFBGGU7OBM3RHFY5BZ6RLKQ, seg_pos={0 0}, upload=472366]: Done uploading piece. err: <nil> Dec 28 02:06:27.505383713 (538h37m27.376466758s) [noise=true , address=145.79.227.38:4977 , piece_id=PB6TUSJR5RRLE3ZYYHCIOSED7AUYBEBNGWWBEELNSEQIPBEO5LHA, seg_pos={0 0}, upload=472366]: Done uploading piece. err: <nil> Dec 28 02:06:27.692603320 (538h37m27.563686365s) [noise=true , address=45.91.116.6:4974 , piece_id=ZI2WE7IT42PSDZ2O27YOGGTESZHI3RQZ7IIMC4GD3FGJWZK2RVKQ, seg_pos={0 0}, upload=472366]: Done uploading piece. err: <nil> Dec 28 02:06:27.724516999 (538h37m27.595600034s) [noise=true , address=45.91.116.4:4939 , piece_id=BFMRZAEILMEWURGHKTEXWPINPXBI7NF6WRRBNTVXVPQ5TOVFZBFQ, seg_pos={0 0}, upload=472366]: Done uploading piece. err: <nil> Dec 28 02:06:27.753017799 (538h37m27.624100834s) [noise=true , address=45.91.116.5:4298 , piece_id=QUEWIBEIARLTC2I4X7DYROTND45JEFKT46XHXUMZNLPTUPPRASKQ, seg_pos={0 0}, upload=472366]: Done uploading piece. err: <nil> Dec 28 02:06:27.764587140 (538h37m27.635670175s) [noise=true , address=45.91.116.10:4082 , piece_id=EBRIPLQD3K3OKFGJY4HP6WR63BTPV442H2FXLOHEBKXTZXUI3AHA, seg_pos={0 0}, upload=472366]: Done uploading piece. err: <nil> Dec 28 02:06:27.800607448 (538h37m27.671690483s) [noise=true , address=45.91.116.11:4117 , piece_id=SVFUZUJX2AFGM63NAPZELLFWKZXKLWHKZO3NCHBUOBBGDJMYAV4Q, seg_pos={0 0}, upload=472366]: Done uploading piece. err: <nil> Dec 28 02:06:28.039356947 (538h37m27.910439992s) [noise=true , address=77.90.31.21:4878 , piece_id=YU44UT26XUN7BHIDE24PLQZ7DIP4P4CWIXT3PVC4DUYVBIYXMM4A, seg_pos={0 0}, upload=472366]: Done uploading piece. err: <nil> Dec 28 02:06:28.039378568 (538h37m27.910461613s) [seg_pos={0 0}, upload=472366]: Segment reached optimal threshold of 13 pieces. Dec 28 02:06:28.039572838 (538h37m27.910655883s) [noise=true , address=145.79.227.5:4223 , piece_id=3TWREZCDN652ZJCISV2WLTALSDAHUV2CZ2YPSANTK7CIR4HXQYSA, seg_pos={0 0}, upload=472366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE): protocol: write tcp 188.166.225.170:50600->145.79.227.5:4223: use of closed network connection; write tcp 188.166.225.170:50600->145.79.227.5:4223: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50600->145.79.227.5:4223: use of closed network connection Dec 28 02:06:28.039580082 (538h37m27.910663127s) [noise=true , address=145.79.227.5:4223 , piece_id=3TWREZCDN652ZJCISV2WLTALSDAHUV2CZ2YPSANTK7CIR4HXQYSA, seg_pos={0 0}, upload=472366]: Long tail cancellation due to optimal threshold Dec 28 02:06:28.039698728 (538h37m27.910781763s) [noise=true , address=145.79.227.3:4151 , piece_id=4DZU4WML2HZPB32WPQFXCSE526XIZD47PWPAOYPQEABHMTPX2TSQ, seg_pos={0 0}, upload=472366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: write tcp 188.166.225.170:58896->145.79.227.3:4151: use of closed network connection; write tcp 188.166.225.170:58896->145.79.227.3:4151: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58896->145.79.227.3:4151: use of closed network connection Dec 28 02:06:28.039704408 (538h37m27.910787453s) [noise=true , address=145.79.227.3:4151 , piece_id=4DZU4WML2HZPB32WPQFXCSE526XIZD47PWPAOYPQEABHMTPX2TSQ, seg_pos={0 0}, upload=472366]: Long tail cancellation due to optimal threshold Dec 28 02:06:28.039764934 (538h37m27.910847999s) [noise=true , address=45.91.116.13:4156 , piece_id=HS5XHKMOB5CKFJ7QB5PUSXK273WR77AQZI35AGXORYRCMNXGLXOQ, seg_pos={0 0}, upload=472366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: write tcp 188.166.225.170:48116->45.91.116.13:4156: use of closed network connection; write tcp 188.166.225.170:48116->45.91.116.13:4156: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48116->45.91.116.13:4156: use of closed network connection Dec 28 02:06:28.039776776 (538h37m27.910859821s) [noise=true , address=45.91.116.13:4156 , piece_id=HS5XHKMOB5CKFJ7QB5PUSXK273WR77AQZI35AGXORYRCMNXGLXOQ, seg_pos={0 0}, upload=472366]: Long tail cancellation due to optimal threshold Dec 28 02:06:28.039802115 (538h37m27.91088516s) [seg_pos={0 0}, upload=472366]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 02:06:28.039984792 (538h37m27.911067837s) [upload=472366]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 02:06:28.039992768 (538h37m27.911075803s) [upload=472366]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 02:06:28.336343158 (538h37m28.207426203s) [upload=472366]: Done uploading segments... Dec 28 02:06:28.336412921 (538h37m28.207495966s) [upload=472366]: Upload finished. err: <nil> Duration:* 8.51146736s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)