Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........X................X......X........................] duration: 43.723074994s trace: https://jaeger.dev.storj.io/trace/6c3caa8adcfacbb9 FAIL (2025-12-30 03:13:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== true , address=145.79.227.35:4496 , piece_id=5X54ZGU4OIVI3DT622QE7IL66SVFLZYUICPODGMDKN7U4D4OJNSA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.040303259 (587h24m33.911386294s) [noise=true , address=77.90.31.12:4601 , piece_id=QQFK4PCL27YYF3UATGLV5KDENRNX6NHE7RXAZ2K66CC3WLHEBMWQ, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.040670117 (587h24m33.911753152s) [noise=true , address=77.90.31.21:4881 , piece_id=EMV4LHFK2VRHSW6PD3BBYGOXDAAYJKX6GEOXGGLOT3IXWNDQMU4A, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041006247 (587h24m33.912089292s) [noise=true , address=145.79.227.52:4472 , piece_id=DT2DAIC4ZI3WL7OGY55ZTTXJEI4IDRXPSJIQA3LTRC7K2F6EVPCA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041183314 (587h24m33.912266349s) [noise=true , address=145.79.227.26:4674 , piece_id=3BAN2ZXOFTSGZITYYHMZUS4AD35X6ZBOYA2HKOJCJI7VBWPJ3LSA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041313653 (587h24m33.912396688s) [noise=true , address=145.79.227.51:4429 , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041535194 (587h24m33.912618229s) [noise=true , address=145.79.227.33:4885 , piece_id=LA755L4RA4VJBHNCQBXOCFROZA3W6W6PUPKGJ5J772WAN44YMY2A, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041639693 (587h24m33.912722728s) [noise=true , address=145.79.227.43:4187 , piece_id=CAAFD3XIWWHZN7L7SYDMMSC5G5RKRLFYYHQZEEN4GAC2D6Y2EARA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041877216 (587h24m33.912960261s) [noise=true , address=145.79.227.46:4623 , piece_id=V4TEM7RRTTGBW5ISZOEAOPFIOZZ5EABO7BPWNRH5U5UPM2X7554Q, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041986634 (587h24m33.913069669s) [noise=true , address=45.91.116.13:4176 , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.042345657 (587h24m33.913428692s) [noise=true , address=45.91.116.10:4104 , piece_id=MRIWH6WGACE4UDNX55MIY6ZDWBAHS6E5KQS4QCOQN2BU7JI7ODCA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.042362019 (587h24m33.913445054s) [noise=true , address=77.90.31.11:4573 , piece_id=TORCC4W7DS5TJ7H7NWONEPNJOFULSZ6F5OHOP4BEVMGEH2MJA4DQ, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.042684192 (587h24m33.913767237s) [noise=true , address=145.79.227.49:4340 , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.042707327 (587h24m33.913790362s) [noise=true , address=145.79.227.32:4837 , piece_id=SQTHTF5KWZOKFYX6CGL6KCDRDAY7OJY6NA5OM3N4YCICDPM6DKXA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.043091528 (587h24m33.914174563s) [noise=true , address=45.91.116.5:4309 , piece_id=Y5HCMU6Z2FPBV3BNTUOC4F6JJWKNBKPOXWFVAFQTXUKQNHTLA4TA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.043544771 (587h24m33.914627816s) [noise=true , address=145.79.227.40:4053 , piece_id=3IZP7JBGSWBRBQHMJWXIDD73B5R3COWU5H3IYFGVBCTTLW7OUJ5A, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:35.347637489 (587h24m35.218720524s) [upload=514937]: Next returned nil segment Dec 30 02:53:35.347733591 (587h24m35.218816626s) [upload=514937]: Waiting for error group managing segments... Dec 30 02:53:35.674169656 (587h24m35.545252721s) [noise=true , address=64.49.13.27:4959 , piece_id=B4UZ67DIWNRHV77VTQONS6ZBUI2BPDDBM3ZS6FVCTABQZ6OHRJJA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:35.891295347 (587h24m35.762378382s) [noise=true , address=145.79.227.32:4837 , piece_id=SQTHTF5KWZOKFYX6CGL6KCDRDAY7OJY6NA5OM3N4YCICDPM6DKXA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:35.912619632 (587h24m35.783702677s) [noise=true , address=145.79.227.26:4674 , piece_id=3BAN2ZXOFTSGZITYYHMZUS4AD35X6ZBOYA2HKOJCJI7VBWPJ3LSA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:35.913054741 (587h24m35.784137786s) [noise=true , address=145.79.227.46:4623 , piece_id=V4TEM7RRTTGBW5ISZOEAOPFIOZZ5EABO7BPWNRH5U5UPM2X7554Q, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:35.913485270 (587h24m35.784568305s) [noise=true , address=145.79.227.35:4496 , piece_id=5X54ZGU4OIVI3DT622QE7IL66SVFLZYUICPODGMDKN7U4D4OJNSA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:35.923790864 (587h24m35.794873909s) [noise=true , address=145.79.227.52:4472 , piece_id=DT2DAIC4ZI3WL7OGY55ZTTXJEI4IDRXPSJIQA3LTRC7K2F6EVPCA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.109605534 (587h24m35.980688569s) [noise=true , address=77.90.31.21:4881 , piece_id=EMV4LHFK2VRHSW6PD3BBYGOXDAAYJKX6GEOXGGLOT3IXWNDQMU4A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.122502652 (587h24m35.993585697s) [noise=true , address=77.90.31.11:4573 , piece_id=TORCC4W7DS5TJ7H7NWONEPNJOFULSZ6F5OHOP4BEVMGEH2MJA4DQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.151131466 (587h24m36.022214511s) [noise=true , address=45.91.116.10:4104 , piece_id=MRIWH6WGACE4UDNX55MIY6ZDWBAHS6E5KQS4QCOQN2BU7JI7ODCA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.159562149 (587h24m36.030645184s) [noise=true , address=77.90.31.12:4601 , piece_id=QQFK4PCL27YYF3UATGLV5KDENRNX6NHE7RXAZ2K66CC3WLHEBMWQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.238339784 (587h24m36.10942282s) [noise=true , address=45.91.116.5:4309 , piece_id=Y5HCMU6Z2FPBV3BNTUOC4F6JJWKNBKPOXWFVAFQTXUKQNHTLA4TA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.813503456 (587h24m36.684586501s) [noise=true , address=145.79.227.33:4885 , piece_id=LA755L4RA4VJBHNCQBXOCFROZA3W6W6PUPKGJ5J772WAN44YMY2A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.826964278 (587h24m36.698047313s) [noise=true , address=145.79.227.43:4187 , piece_id=CAAFD3XIWWHZN7L7SYDMMSC5G5RKRLFYYHQZEEN4GAC2D6Y2EARA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.828562510 (587h24m36.699645545s) [noise=true , address=145.79.227.40:4053 , piece_id=3IZP7JBGSWBRBQHMJWXIDD73B5R3COWU5H3IYFGVBCTTLW7OUJ5A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.828575566 (587h24m36.699658601s) [seg_pos={0 0}, upload=514937]: Segment reached optimal threshold of 14 pieces. Dec 30 02:53:36.828752202 (587h24m36.699835237s) [noise=true , address=145.79.227.30:4780 , piece_id=DZQ5LIBMWXPJ4XXWORRDLQIAFXJDNDQEEBRMSBS44UX3D2HPTSWQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125WCbZ6a2NaewiLAHhHuPo91sQEfC9CZzEBV6rwF1JY7z13AEY): protocol: expected piece hash; context canceled; EOF Dec 30 02:53:36.828758714 (587h24m36.699841749s) [noise=true , address=145.79.227.30:4780 , piece_id=DZQ5LIBMWXPJ4XXWORRDLQIAFXJDNDQEEBRMSBS44UX3D2HPTSWQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold Dec 30 02:53:36.828762672 (587h24m36.699845707s) [noise=true , address=145.79.227.51:4429 , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qytmzwy7sTmWXskEbLWsWMniPz55zgho2akE7oh6ahAiT5SXf): protocol: write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection; write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection Dec 30 02:53:36.828774034 (587h24m36.699857069s) [noise=true , address=145.79.227.51:4429 , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold Dec 30 02:53:36.828889934 (587h24m36.699972969s) [noise=true , address=45.91.116.13:4176 , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ejsmBd4tAVwTHjamPYbryQb5x2vDnk9pBuX3CBF9WYFhTrFVJ): protocol: expected piece hash; context canceled; EOF Dec 30 02:53:36.828895144 (587h24m36.699978179s) [noise=true , address=45.91.116.13:4176 , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold Dec 30 02:53:36.828994173 (587h24m36.700077208s) [noise=true , address=145.79.227.49:4340 , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF Dec 30 02:53:36.828999793 (587h24m36.700082828s) [noise=true , address=145.79.227.49:4340 , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold Dec 30 02:53:36.829030332 (587h24m36.700113367s) [seg_pos={0 0}, upload=514937]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 02:53:36.829095486 (587h24m36.700178621s) [upload=514937]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 02:53:36.829100105 (587h24m36.70018315s) [upload=514937]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 02:53:37.128695270 (587h24m36.999778315s) [upload=514937]: Done uploading segments... Dec 30 02:53:37.128717683 (587h24m36.999800728s) [upload=514937]: Upload finished. err: <nil> Duration: 43.723101745s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [.....X...X..XX..X.X......XXX.X.X..X........X.X.XX..X.X.XX..X] duration: 30.00593005s trace: https://jaeger.dev.storj.io/trace/12186bf18ed51589 FAIL (2025-12-30 03:13:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 03:12:40.164940898 (587h43m40.036023933s) [upload=515213]: Starting upload Dec 30 03:12:40.165050406 (587h43m40.036133461s) [upload=515213]: Done starting upload Dec 30 03:12:40.165124849 (587h43m40.036207884s) [upload=515213]: Uploading segments... Dec 30 03:12:40.165440490 (587h43m40.036523535s) [upload=515213]: Got next segment. Inline: false Dec 30 03:12:40.165485375 (587h43m40.03656841s) [upload=515213]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 03:12:40.165491366 (587h43m40.036574401s) [upload=515213]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 03:12:40.418472013 (587h43m40.289555048s) [seg_pos={0 0}, upload=515213]: Begin upload segment... Dec 30 03:12:40.418535334 (587h43m40.289618379s) [seg_pos={0 0}, upload=515213]: Done begin upload segment. Dec 30 03:12:40.418582583 (587h43m40.289665618s) [noise=true , address=45.91.116.12:4228 , piece_id=IY5RKRFJJBPA2MX7C5ENRS47NWLYWKAFPKBLGK2ZFAC5QBVTNB5A, seg_pos={0 0}, upload=515213]: Uploading piece... Dec 30 03:12:40.418650362 (587h43m40.289733407s) [noise=true , address=145.79.227.15:4876 , piece_id=TMFKJEE3B5I7KBW4UGUGL35QH7WJXY6PWCP4IAOY63DX4ZX6JXYQ, seg_pos={0 0}, upload=515213]: Uploading piece... Dec 30 03:12:40.419272888 (587h43m40.290355923s) [noise=true , address=145.79.227.42:4123 , piece_id=WKWF4LQIQAIPMBTLPGSZKPN5DQTAWETH2JAUFF6XKKBP6BN3FX5Q, seg_pos={0 0}, upload=515213]: Uploading piece... Dec 30 03:12:40.419752922 (587h43m40.290835967s) [noise=true , address=45.91.116.8:4048 , piece_id=EMX2FFPR63V6TX2EFINO7CRMBLMEB26OTP2JYMKJJXKEZ2MSDOFA, seg_pos={0 0}, upload=515213]: Uploading piece... Dec 30 03:12:40.420289003 (587h43m40.291372048s) [noise=true , address=145.79.227.32:4836 , piece_id=6JBYTCQRY26XBMMJKJ4PNAWHSKZPPGWAQDOIDE7PNWQOZYBATJNQ, seg_pos={0 0}, upload=515213]: Uploading piece... Dec 30 03:12:40.421578167 (587h43m40.292661212s) [noise=true , address=145.79.227.13:4803 , piece_id=BWOZMKEJOMP4FTJYEZAJLUTLM5W7BEQSB3NRMPVUGWILSJTTCRGA, seg_pos={0 0}, upload=515213]: Uploading piece... Dec 30 03:12:40.423655233 (587h43m40.294738278s) [noise=true , address=145.79.227.35:4482 , piece_id=PU3M3NBOAFFELJO3YQXOPY7QHNY2YVXQGI5SMBUCDBPPRIATSNSA, seg_pos={0 0}, upload=515213]: Uploading piece... Dec 30 03:12:40.426910461 (587h43m40.297993506s) [noise=true , address=77.90.31.13:4626 , piece_id=YC7FSZCZKTUZAE3UCUAKPRYMYMLDNX4V64EU3F67GHLXC72ANZRQ, seg_pos={0 0}, upload=515213]: Uploading piece... Dec 30 03:12:40.430745895 (587h43m40.30182896s) [noise=true , address=45.91.116.7:4011 , piece_id=ETL5GTRG2XJBIPJ4ACE2MC7SLBH3AA6BCUZMAI5ZHEWTV3WOQO7Q, seg_pos={0 0}, upload=515213]: Uploading piece... Dec 30 03:12:40.430982735 (587h43m40.30206577s) [noise=true , address=145.79.227.48:4302 , piece_id=QGU5PJINPKFUOFORMBZFUYYKBWFZGAJT2LEMTWFLJVOWA34XGVJA, seg_pos={0 0}, upload=515213]: Uploading piece... Dec 30 03:12:40.432874588 (587h43m40.303957633s) [noise=true , address=145.79.227.14:4839 , piece_id=6LUFU5DX4V6P72BK2KFATJ42PJ75ZFQF6UOJYKYIHYDDILMX4AXQ, seg_pos={0 0}, upload=515213]: Uploading piece... Dec 30 03:12:40.435392342 (587h43m40.306475377s) [noise=true , address=145.79.227.9:4367 , piece_id=BXRV2BDPTDB3GP2FGZNQKSCJ2VB25FSUOHQZXVC5LTY6YRJH2HBA, seg_pos={0 0}, upload=515213]: Uploading piece... Dec 30 03:12:40.436761078 (587h43m40.307844123s) [noise=true , address=45.91.116.10:4084 , piece_id=S7ELCREVROMSIU7SLSYEPFRE4POXIFP3JAH3G6LCKQ2Y5DTNCPFA, seg_pos={0 0}, upload=515213]: Uploading piece... Dec 30 03:12:40.442470289 (587h43m40.313553324s) [noise=true , address=145.79.227.2:4114 , piece_id=5CD5IHM2HKG4QS3ZD5YRS5EQTHA6KDRCXC2QM7J66LB4NXIPJQ7A, seg_pos={0 0}, upload=515213]: Uploading piece... Dec 30 03:12:40.444326955 (587h43m40.31541s) [noise=true , address=145.79.227.19:4077 , piece_id=CGKRXP37UECASDCVQXZ4JKV3WXYVTZEBJHHAIO2DEZAKSWOQH22A, seg_pos={0 0}, upload=515213]: Uploading piece... Dec 30 03:12:40.445001990 (587h43m40.316085025s) [noise=true , address=45.91.116.9:4263 , piece_id=LFRXNHFEP3QFDNOVXGSH6FOZC7S3DIMSG7ORTNBKHWX3S77ZSSMA, seg_pos={0 0}, upload=515213]: Uploading piece... Dec 30 03:12:42.118851413 (587h43m41.989934448s) [upload=515213]: Next returned nil segment Dec 30 03:12:42.118863427 (587h43m41.989946472s) [upload=515213]: Waiting for error group managing segments... Dec 30 03:12:42.405720249 (587h43m42.276803284s) [noise=true , address=145.79.227.32:4836 , piece_id=6JBYTCQRY26XBMMJKJ4PNAWHSKZPPGWAQDOIDE7PNWQOZYBATJNQ, seg_pos={0 0}, upload=515213]: Done uploading piece. err: <nil> Dec 30 03:12:42.411195465 (587h43m42.2822785s) [noise=true , address=145.79.227.42:4123 , piece_id=WKWF4LQIQAIPMBTLPGSZKPN5DQTAWETH2JAUFF6XKKBP6BN3FX5Q, seg_pos={0 0}, upload=515213]: Done uploading piece. err: <nil> Dec 30 03:12:42.415202575 (587h43m42.28628561s) [noise=true , address=145.79.227.13:4803 , piece_id=BWOZMKEJOMP4FTJYEZAJLUTLM5W7BEQSB3NRMPVUGWILSJTTCRGA, seg_pos={0 0}, upload=515213]: Done uploading piece. err: <nil> Dec 30 03:12:42.420472730 (587h43m42.291555795s) [noise=true , address=145.79.227.9:4367 , piece_id=BXRV2BDPTDB3GP2FGZNQKSCJ2VB25FSUOHQZXVC5LTY6YRJH2HBA, seg_pos={0 0}, upload=515213]: Done uploading piece. err: <nil> Dec 30 03:12:42.446510851 (587h43m42.317593896s) [noise=true , address=145.79.227.15:4876 , piece_id=TMFKJEE3B5I7KBW4UGUGL35QH7WJXY6PWCP4IAOY63DX4ZX6JXYQ, seg_pos={0 0}, upload=515213]: Done uploading piece. err: <nil> Dec 30 03:12:42.447650281 (587h43m42.318733326s) [noise=true , address=145.79.227.19:4077 , piece_id=CGKRXP37UECASDCVQXZ4JKV3WXYVTZEBJHHAIO2DEZAKSWOQH22A, seg_pos={0 0}, upload=515213]: Done uploading piece. err: <nil> Dec 30 03:12:42.448275281 (587h43m42.319358326s) [noise=true , address=145.79.227.48:4302 , piece_id=QGU5PJINPKFUOFORMBZFUYYKBWFZGAJT2LEMTWFLJVOWA34XGVJA, seg_pos={0 0}, upload=515213]: Done uploading piece. err: <nil> Dec 30 03:12:42.453726491 (587h43m42.324809526s) [noise=true , address=145.79.227.14:4839 , piece_id=6LUFU5DX4V6P72BK2KFATJ42PJ75ZFQF6UOJYKYIHYDDILMX4AXQ, seg_pos={0 0}, upload=515213]: Done uploading piece. err: <nil> Dec 30 03:12:42.456498640 (587h43m42.327581675s) [noise=true , address=145.79.227.2:4114 , piece_id=5CD5IHM2HKG4QS3ZD5YRS5EQTHA6KDRCXC2QM7J66LB4NXIPJQ7A, seg_pos={0 0}, upload=515213]: Done uploading piece. err: <nil> Dec 30 03:12:42.633626620 (587h43m42.504709666s) [noise=true , address=45.91.116.10:4084 , piece_id=S7ELCREVROMSIU7SLSYEPFRE4POXIFP3JAH3G6LCKQ2Y5DTNCPFA, seg_pos={0 0}, upload=515213]: Done uploading piece. err: <nil> Dec 30 03:12:42.689063291 (587h43m42.560146326s) [noise=true , address=45.91.116.12:4228 , piece_id=IY5RKRFJJBPA2MX7C5ENRS47NWLYWKAFPKBLGK2ZFAC5QBVTNB5A, seg_pos={0 0}, upload=515213]: Done uploading piece. err: <nil> Dec 30 03:12:42.838513469 (587h43m42.709596504s) [noise=true , address=45.91.116.7:4011 , piece_id=ETL5GTRG2XJBIPJ4ACE2MC7SLBH3AA6BCUZMAI5ZHEWTV3WOQO7Q, seg_pos={0 0}, upload=515213]: Done uploading piece. err: <nil> Dec 30 03:12:42.996672750 (587h43m42.867755785s) [noise=true , address=45.91.116.9:4263 , piece_id=LFRXNHFEP3QFDNOVXGSH6FOZC7S3DIMSG7ORTNBKHWX3S77ZSSMA, seg_pos={0 0}, upload=515213]: Done uploading piece. err: <nil> Dec 30 03:12:42.996690483 (587h43m42.867773518s) [seg_pos={0 0}, upload=515213]: Segment reached optimal threshold of 13 pieces. Dec 30 03:12:42.996877650 (587h43m42.867960685s) [noise=true , address=145.79.227.35:4482 , piece_id=PU3M3NBOAFFELJO3YQXOPY7QHNY2YVXQGI5SMBUCDBPPRIATSNSA, seg_pos={0 0}, upload=515213]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: expected piece hash; context canceled; EOF Dec 30 03:12:42.996885345 (587h43m42.86796838s) [noise=true , address=145.79.227.35:4482 , piece_id=PU3M3NBOAFFELJO3YQXOPY7QHNY2YVXQGI5SMBUCDBPPRIATSNSA, seg_pos={0 0}, upload=515213]: Long tail cancellation due to optimal threshold Dec 30 03:12:42.996999732 (587h43m42.868082777s) [noise=true , address=77.90.31.13:4626 , piece_id=YC7FSZCZKTUZAE3UCUAKPRYMYMLDNX4V64EU3F67GHLXC72ANZRQ, seg_pos={0 0}, upload=515213]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 188.166.225.170:44614->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:44614->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44614->77.90.31.13:4626: use of closed network connection Dec 30 03:12:42.997010552 (587h43m42.868093597s) [noise=true , address=77.90.31.13:4626 , piece_id=YC7FSZCZKTUZAE3UCUAKPRYMYMLDNX4V64EU3F67GHLXC72ANZRQ, seg_pos={0 0}, upload=515213]: Long tail cancellation due to optimal threshold Dec 30 03:12:42.997111405 (587h43m42.86819444s) [noise=true , address=45.91.116.8:4048 , piece_id=EMX2FFPR63V6TX2EFINO7CRMBLMEB26OTP2JYMKJJXKEZ2MSDOFA, seg_pos={0 0}, upload=515213]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 30 03:12:42.997117786 (587h43m42.868200822s) [noise=true , address=45.91.116.8:4048 , piece_id=EMX2FFPR63V6TX2EFINO7CRMBLMEB26OTP2JYMKJJXKEZ2MSDOFA, seg_pos={0 0}, upload=515213]: Long tail cancellation due to optimal threshold Dec 30 03:12:42.997133918 (587h43m42.868216963s) [seg_pos={0 0}, upload=515213]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 03:12:42.997261691 (587h43m42.868344726s) [upload=515213]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 03:12:42.997268534 (587h43m42.868351579s) [upload=515213]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 03:12:43.280439328 (587h43m43.151522373s) [upload=515213]: Done uploading segments... Dec 30 03:12:43.280462953 (587h43m43.151545998s) [upload=515213]: Upload finished. err: <nil> Duration: 30.005946041s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [.........................X...X.X.XX................XX.......] duration: 5.000742623s trace: https://jaeger.dev.storj.io/trace/55b84c463909ac27 FAIL (2025-12-30 03:12:54 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000766839s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [............X................X......X.......................] duration: 32.687059968s trace: https://jaeger.dev.storj.io/trace/6c3caa8adcfacbb9 FAIL (2025-12-30 03:12:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== true , address=145.79.227.35:4496 , piece_id=5X54ZGU4OIVI3DT622QE7IL66SVFLZYUICPODGMDKN7U4D4OJNSA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.040303259 (587h24m33.911386294s) [noise=true , address=77.90.31.12:4601 , piece_id=QQFK4PCL27YYF3UATGLV5KDENRNX6NHE7RXAZ2K66CC3WLHEBMWQ, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.040670117 (587h24m33.911753152s) [noise=true , address=77.90.31.21:4881 , piece_id=EMV4LHFK2VRHSW6PD3BBYGOXDAAYJKX6GEOXGGLOT3IXWNDQMU4A, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041006247 (587h24m33.912089292s) [noise=true , address=145.79.227.52:4472 , piece_id=DT2DAIC4ZI3WL7OGY55ZTTXJEI4IDRXPSJIQA3LTRC7K2F6EVPCA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041183314 (587h24m33.912266349s) [noise=true , address=145.79.227.26:4674 , piece_id=3BAN2ZXOFTSGZITYYHMZUS4AD35X6ZBOYA2HKOJCJI7VBWPJ3LSA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041313653 (587h24m33.912396688s) [noise=true , address=145.79.227.51:4429 , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041535194 (587h24m33.912618229s) [noise=true , address=145.79.227.33:4885 , piece_id=LA755L4RA4VJBHNCQBXOCFROZA3W6W6PUPKGJ5J772WAN44YMY2A, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041639693 (587h24m33.912722728s) [noise=true , address=145.79.227.43:4187 , piece_id=CAAFD3XIWWHZN7L7SYDMMSC5G5RKRLFYYHQZEEN4GAC2D6Y2EARA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041877216 (587h24m33.912960261s) [noise=true , address=145.79.227.46:4623 , piece_id=V4TEM7RRTTGBW5ISZOEAOPFIOZZ5EABO7BPWNRH5U5UPM2X7554Q, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041986634 (587h24m33.913069669s) [noise=true , address=45.91.116.13:4176 , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.042345657 (587h24m33.913428692s) [noise=true , address=45.91.116.10:4104 , piece_id=MRIWH6WGACE4UDNX55MIY6ZDWBAHS6E5KQS4QCOQN2BU7JI7ODCA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.042362019 (587h24m33.913445054s) [noise=true , address=77.90.31.11:4573 , piece_id=TORCC4W7DS5TJ7H7NWONEPNJOFULSZ6F5OHOP4BEVMGEH2MJA4DQ, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.042684192 (587h24m33.913767237s) [noise=true , address=145.79.227.49:4340 , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.042707327 (587h24m33.913790362s) [noise=true , address=145.79.227.32:4837 , piece_id=SQTHTF5KWZOKFYX6CGL6KCDRDAY7OJY6NA5OM3N4YCICDPM6DKXA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.043091528 (587h24m33.914174563s) [noise=true , address=45.91.116.5:4309 , piece_id=Y5HCMU6Z2FPBV3BNTUOC4F6JJWKNBKPOXWFVAFQTXUKQNHTLA4TA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.043544771 (587h24m33.914627816s) [noise=true , address=145.79.227.40:4053 , piece_id=3IZP7JBGSWBRBQHMJWXIDD73B5R3COWU5H3IYFGVBCTTLW7OUJ5A, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:35.347637489 (587h24m35.218720524s) [upload=514937]: Next returned nil segment Dec 30 02:53:35.347733591 (587h24m35.218816626s) [upload=514937]: Waiting for error group managing segments... Dec 30 02:53:35.674169656 (587h24m35.545252721s) [noise=true , address=64.49.13.27:4959 , piece_id=B4UZ67DIWNRHV77VTQONS6ZBUI2BPDDBM3ZS6FVCTABQZ6OHRJJA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:35.891295347 (587h24m35.762378382s) [noise=true , address=145.79.227.32:4837 , piece_id=SQTHTF5KWZOKFYX6CGL6KCDRDAY7OJY6NA5OM3N4YCICDPM6DKXA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:35.912619632 (587h24m35.783702677s) [noise=true , address=145.79.227.26:4674 , piece_id=3BAN2ZXOFTSGZITYYHMZUS4AD35X6ZBOYA2HKOJCJI7VBWPJ3LSA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:35.913054741 (587h24m35.784137786s) [noise=true , address=145.79.227.46:4623 , piece_id=V4TEM7RRTTGBW5ISZOEAOPFIOZZ5EABO7BPWNRH5U5UPM2X7554Q, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:35.913485270 (587h24m35.784568305s) [noise=true , address=145.79.227.35:4496 , piece_id=5X54ZGU4OIVI3DT622QE7IL66SVFLZYUICPODGMDKN7U4D4OJNSA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:35.923790864 (587h24m35.794873909s) [noise=true , address=145.79.227.52:4472 , piece_id=DT2DAIC4ZI3WL7OGY55ZTTXJEI4IDRXPSJIQA3LTRC7K2F6EVPCA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.109605534 (587h24m35.980688569s) [noise=true , address=77.90.31.21:4881 , piece_id=EMV4LHFK2VRHSW6PD3BBYGOXDAAYJKX6GEOXGGLOT3IXWNDQMU4A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.122502652 (587h24m35.993585697s) [noise=true , address=77.90.31.11:4573 , piece_id=TORCC4W7DS5TJ7H7NWONEPNJOFULSZ6F5OHOP4BEVMGEH2MJA4DQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.151131466 (587h24m36.022214511s) [noise=true , address=45.91.116.10:4104 , piece_id=MRIWH6WGACE4UDNX55MIY6ZDWBAHS6E5KQS4QCOQN2BU7JI7ODCA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.159562149 (587h24m36.030645184s) [noise=true , address=77.90.31.12:4601 , piece_id=QQFK4PCL27YYF3UATGLV5KDENRNX6NHE7RXAZ2K66CC3WLHEBMWQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.238339784 (587h24m36.10942282s) [noise=true , address=45.91.116.5:4309 , piece_id=Y5HCMU6Z2FPBV3BNTUOC4F6JJWKNBKPOXWFVAFQTXUKQNHTLA4TA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.813503456 (587h24m36.684586501s) [noise=true , address=145.79.227.33:4885 , piece_id=LA755L4RA4VJBHNCQBXOCFROZA3W6W6PUPKGJ5J772WAN44YMY2A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.826964278 (587h24m36.698047313s) [noise=true , address=145.79.227.43:4187 , piece_id=CAAFD3XIWWHZN7L7SYDMMSC5G5RKRLFYYHQZEEN4GAC2D6Y2EARA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.828562510 (587h24m36.699645545s) [noise=true , address=145.79.227.40:4053 , piece_id=3IZP7JBGSWBRBQHMJWXIDD73B5R3COWU5H3IYFGVBCTTLW7OUJ5A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.828575566 (587h24m36.699658601s) [seg_pos={0 0}, upload=514937]: Segment reached optimal threshold of 14 pieces. Dec 30 02:53:36.828752202 (587h24m36.699835237s) [noise=true , address=145.79.227.30:4780 , piece_id=DZQ5LIBMWXPJ4XXWORRDLQIAFXJDNDQEEBRMSBS44UX3D2HPTSWQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125WCbZ6a2NaewiLAHhHuPo91sQEfC9CZzEBV6rwF1JY7z13AEY): protocol: expected piece hash; context canceled; EOF Dec 30 02:53:36.828758714 (587h24m36.699841749s) [noise=true , address=145.79.227.30:4780 , piece_id=DZQ5LIBMWXPJ4XXWORRDLQIAFXJDNDQEEBRMSBS44UX3D2HPTSWQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold Dec 30 02:53:36.828762672 (587h24m36.699845707s) [noise=true , address=145.79.227.51:4429 , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qytmzwy7sTmWXskEbLWsWMniPz55zgho2akE7oh6ahAiT5SXf): protocol: write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection; write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection Dec 30 02:53:36.828774034 (587h24m36.699857069s) [noise=true , address=145.79.227.51:4429 , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold Dec 30 02:53:36.828889934 (587h24m36.699972969s) [noise=true , address=45.91.116.13:4176 , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ejsmBd4tAVwTHjamPYbryQb5x2vDnk9pBuX3CBF9WYFhTrFVJ): protocol: expected piece hash; context canceled; EOF Dec 30 02:53:36.828895144 (587h24m36.699978179s) [noise=true , address=45.91.116.13:4176 , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold Dec 30 02:53:36.828994173 (587h24m36.700077208s) [noise=true , address=145.79.227.49:4340 , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF Dec 30 02:53:36.828999793 (587h24m36.700082828s) [noise=true , address=145.79.227.49:4340 , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold Dec 30 02:53:36.829030332 (587h24m36.700113367s) [seg_pos={0 0}, upload=514937]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 02:53:36.829095486 (587h24m36.700178621s) [upload=514937]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 02:53:36.829100105 (587h24m36.70018315s) [upload=514937]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 02:53:37.128695270 (587h24m36.999778315s) [upload=514937]: Done uploading segments... Dec 30 02:53:37.128717683 (587h24m36.999800728s) [upload=514937]: Upload finished. err: <nil> Duration: 32.687089434s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [..........................X...X.X.XX................XX......] duration: 3.661545485s trace: https://jaeger.dev.storj.io/trace/55b84c463909ac27 FAIL (2025-12-30 03:12:11 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.661566053s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [......X...X..XX..X.X......XXX.X.X..X........X.X.XX..X.X.XX..] duration: 5.924276173s trace: https://jaeger.dev.storj.io/trace/cd54c0b0e5ccd2b FAIL (2025-12-30 03:11:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 30 03:10:24.791854941 (587h41m24.662937976s) [upload=515181]: Starting upload Dec 30 03:10:24.791916989 (587h41m24.663000035s) [upload=515181]: Done starting upload Dec 30 03:10:24.792134443 (587h41m24.663217478s) [upload=515181]: Uploading segments... Dec 30 03:10:24.792188065 (587h41m24.6632711s) [upload=515181]: Got next segment. Inline: false Dec 30 03:10:24.792213754 (587h41m24.663296789s) [upload=515181]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 03:10:24.792218343 (587h41m24.663301388s) [upload=515181]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 03:10:25.040864557 (587h41m24.911947612s) [seg_pos={0 0}, upload=515181]: Begin upload segment... Dec 30 03:10:25.040917447 (587h41m24.912000492s) [seg_pos={0 0}, upload=515181]: Done begin upload segment. Dec 30 03:10:25.040948176 (587h41m24.912031221s) [noise=true , address=145.79.227.39:4014 , piece_id=D7LVR5LWLRGJNC76BATWBO3GFFL56Y64MOCXNHFGGKL2UUJ7PVGA, seg_pos={0 0}, upload=515181]: Uploading piece... Dec 30 03:10:25.041722661 (587h41m24.912805706s) [noise=true , address=45.91.116.14:4189 , piece_id=JNIUV7WE2YWU6UXGOWWUH7PIVZONMPUT2WK7R3MDUIMKWPYWRBWQ, seg_pos={0 0}, upload=515181]: Uploading piece... Dec 30 03:10:25.041881263 (587h41m24.912964308s) [noise=true , address=77.90.31.16:4735 , piece_id=YLJKK3EZF4A3DQGBQZFK7GIYOBUVDNFZWUMAVHKGHPFUZZBGN4RA, seg_pos={0 0}, upload=515181]: Uploading piece... Dec 30 03:10:25.042287336 (587h41m24.913370381s) [noise=true , address=145.79.227.14:4839 , piece_id=UXNJ5SUIF5RUK4XOO5UHCUTRD42IFFDQBQMWSNNSPQJIDWOJRWRQ, seg_pos={0 0}, upload=515181]: Uploading piece... Dec 30 03:10:25.042378269 (587h41m24.913461304s) [noise=true , address=145.79.227.36:4519 , piece_id=HAVZ6MCZHYFEUVHECN3BBCW3RQADF3D64VZH2AAZT7V2KTBKDIEQ, seg_pos={0 0}, upload=515181]: Uploading piece... Dec 30 03:10:25.042656148 (587h41m24.913739193s) [noise=true , address=145.79.227.46:4603 , piece_id=EQBACEDADMEZTPWD3UFXZ7HSNQ3X647O2D6KDQJULCFLKWMO7BTA, seg_pos={0 0}, upload=515181]: Uploading piece... Dec 30 03:10:25.042684613 (587h41m24.913767648s) [noise=true , address=145.79.227.17:4972 , piece_id=LWABVX33XMVXJQKQAIE6TFR2PHMHZGRJGF3NVLDWNZNBSEBJ4FLQ, seg_pos={0 0}, upload=515181]: Uploading piece... Dec 30 03:10:25.042917566 (587h41m24.914000602s) [noise=true , address=45.91.116.7:4009 , piece_id=SP7M5LMNHGEBZJ5IMXA6IDCQYUC5GUGCVFZDTMAZCAPU5FJDHNXA, seg_pos={0 0}, upload=515181]: Uploading piece... Dec 30 03:10:25.043061230 (587h41m24.914144265s) [noise=true , address=145.79.227.25:4619 , piece_id=YVOH6TCNPNPO2AXWP3NN6KN7WZBIBP23JNK6TOG3VUQEA72YZUGQ, seg_pos={0 0}, upload=515181]: Uploading piece... Dec 30 03:10:25.043182250 (587h41m24.914265285s) [noise=true , address=45.91.116.5:4298 , piece_id=W34NKHDFMGIHOVEUGOHNV3SPTFIINIESSKEHB74JSMST2CASOBXA, seg_pos={0 0}, upload=515181]: Uploading piece... Dec 30 03:10:25.043445151 (587h41m24.914528196s) [noise=true , address=145.79.227.42:4121 , piece_id=YZRW5773RMYVB2BUGV4HAE35JRCRTAPGNRRHW4M4MMKTJVYEABTA, seg_pos={0 0}, upload=515181]: Uploading piece... Dec 30 03:10:25.043453988 (587h41m24.914537033s) [noise=true , address=145.79.227.6:4259 , piece_id=7CD7TBZGXR6HBZ4TFYJFDP3IA5IZXKU7GUDDSF46MJWMUKXCGDKQ, seg_pos={0 0}, upload=515181]: Uploading piece... Dec 30 03:10:25.043851905 (587h41m24.91493495s) [noise=true , address=45.91.116.10:4084 , piece_id=UDSGCWMHWKPXMUFDQHZQOLYP3LPXZYZYWRE6GHOC6AGCQPD7D7JA, seg_pos={0 0}, upload=515181]: Uploading piece... Dec 30 03:10:25.044684911 (587h41m24.915767946s) [noise=true , address=45.91.116.6:4976 , piece_id=CLGQSMNBXE63EIPI3OS27QXCFQIC6COUP6Q4NIQUFTLNZIVBIYAA, seg_pos={0 0}, upload=515181]: Uploading piece... Dec 30 03:10:25.045245007 (587h41m24.916328053s) [noise=true , address=145.79.227.2:4113 , piece_id=P4VHP3ZVC2FABHWSCIO6522JFXSOE7EZMZWZILAHAWSRBG6UJCIQ, seg_pos={0 0}, upload=515181]: Uploading piece... Dec 30 03:10:25.045257662 (587h41m24.916340707s) [noise=true , address=145.79.227.50:4374 , piece_id=3SM3QTVE3YUHJUFBOK6ZBCMUUSX4PRDTQNTSOJHIRPWTZOALUVNA, seg_pos={0 0}, upload=515181]: Uploading piece... Dec 30 03:10:26.662003563 (587h41m26.533086598s) [upload=515181]: Next returned nil segment Dec 30 03:10:26.662018661 (587h41m26.533101706s) [upload=515181]: Waiting for error group managing segments... Dec 30 03:10:27.023175193 (587h41m26.894258238s) [noise=true , address=145.79.227.6:4259 , piece_id=7CD7TBZGXR6HBZ4TFYJFDP3IA5IZXKU7GUDDSF46MJWMUKXCGDKQ, seg_pos={0 0}, upload=515181]: Done uploading piece. err: <nil> Dec 30 03:10:27.023924228 (587h41m26.895007263s) [noise=true , address=145.79.227.17:4972 , piece_id=LWABVX33XMVXJQKQAIE6TFR2PHMHZGRJGF3NVLDWNZNBSEBJ4FLQ, seg_pos={0 0}, upload=515181]: Done uploading piece. err: <nil> Dec 30 03:10:27.030513275 (587h41m26.90159632s) [noise=true , address=145.79.227.14:4839 , piece_id=UXNJ5SUIF5RUK4XOO5UHCUTRD42IFFDQBQMWSNNSPQJIDWOJRWRQ, seg_pos={0 0}, upload=515181]: Done uploading piece. err: <nil> Dec 30 03:10:27.031052863 (587h41m26.902135898s) [noise=true , address=145.79.227.25:4619 , piece_id=YVOH6TCNPNPO2AXWP3NN6KN7WZBIBP23JNK6TOG3VUQEA72YZUGQ, seg_pos={0 0}, upload=515181]: Done uploading piece. err: <nil> Dec 30 03:10:27.038813891 (587h41m26.909896926s) [noise=true , address=145.79.227.36:4519 , piece_id=HAVZ6MCZHYFEUVHECN3BBCW3RQADF3D64VZH2AAZT7V2KTBKDIEQ, seg_pos={0 0}, upload=515181]: Done uploading piece. err: <nil> Dec 30 03:10:27.047944667 (587h41m26.919027712s) [noise=true , address=145.79.227.2:4113 , piece_id=P4VHP3ZVC2FABHWSCIO6522JFXSOE7EZMZWZILAHAWSRBG6UJCIQ, seg_pos={0 0}, upload=515181]: Done uploading piece. err: <nil> Dec 30 03:10:27.049040363 (587h41m26.920123398s) [noise=true , address=145.79.227.50:4374 , piece_id=3SM3QTVE3YUHJUFBOK6ZBCMUUSX4PRDTQNTSOJHIRPWTZOALUVNA, seg_pos={0 0}, upload=515181]: Done uploading piece. err: <nil> Dec 30 03:10:27.052510261 (587h41m26.923593306s) [noise=true , address=145.79.227.42:4121 , piece_id=YZRW5773RMYVB2BUGV4HAE35JRCRTAPGNRRHW4M4MMKTJVYEABTA, seg_pos={0 0}, upload=515181]: Done uploading piece. err: <nil> Dec 30 03:10:27.059180272 (587h41m26.930263317s) [noise=true , address=145.79.227.46:4603 , piece_id=EQBACEDADMEZTPWD3UFXZ7HSNQ3X647O2D6KDQJULCFLKWMO7BTA, seg_pos={0 0}, upload=515181]: Done uploading piece. err: <nil> Dec 30 03:10:27.068649223 (587h41m26.939732278s) [noise=true , address=145.79.227.39:4014 , piece_id=D7LVR5LWLRGJNC76BATWBO3GFFL56Y64MOCXNHFGGKL2UUJ7PVGA, seg_pos={0 0}, upload=515181]: Done uploading piece. err: <nil> Dec 30 03:10:27.219987115 (587h41m27.09107018s) [noise=true , address=77.90.31.16:4735 , piece_id=YLJKK3EZF4A3DQGBQZFK7GIYOBUVDNFZWUMAVHKGHPFUZZBGN4RA, seg_pos={0 0}, upload=515181]: Done uploading piece. err: <nil> Dec 30 03:10:27.335729688 (587h41m27.206812733s) [noise=true , address=45.91.116.7:4009 , piece_id=SP7M5LMNHGEBZJ5IMXA6IDCQYUC5GUGCVFZDTMAZCAPU5FJDHNXA, seg_pos={0 0}, upload=515181]: Done uploading piece. err: <nil> Dec 30 03:10:27.425829402 (587h41m27.296912437s) [noise=true , address=45.91.116.6:4976 , piece_id=CLGQSMNBXE63EIPI3OS27QXCFQIC6COUP6Q4NIQUFTLNZIVBIYAA, seg_pos={0 0}, upload=515181]: Done uploading piece. err: <nil> Dec 30 03:10:27.425849661 (587h41m27.296932696s) [seg_pos={0 0}, upload=515181]: Segment reached optimal threshold of 13 pieces. Dec 30 03:10:27.426016377 (587h41m27.297099412s) [noise=true , address=45.91.116.14:4189 , piece_id=JNIUV7WE2YWU6UXGOWWUH7PIVZONMPUT2WK7R3MDUIMKWPYWRBWQ, seg_pos={0 0}, upload=515181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 30 03:10:27.426022680 (587h41m27.297105715s) [noise=true , address=45.91.116.14:4189 , piece_id=JNIUV7WE2YWU6UXGOWWUH7PIVZONMPUT2WK7R3MDUIMKWPYWRBWQ, seg_pos={0 0}, upload=515181]: Long tail cancellation due to optimal threshold Dec 30 03:10:27.426288365 (587h41m27.297371411s) [noise=true , address=45.91.116.10:4084 , piece_id=UDSGCWMHWKPXMUFDQHZQOLYP3LPXZYZYWRE6GHOC6AGCQPD7D7JA, seg_pos={0 0}, upload=515181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 30 03:10:27.426308154 (587h41m27.297391199s) [noise=true , address=45.91.116.10:4084 , piece_id=UDSGCWMHWKPXMUFDQHZQOLYP3LPXZYZYWRE6GHOC6AGCQPD7D7JA, seg_pos={0 0}, upload=515181]: Long tail cancellation due to optimal threshold Dec 30 03:10:27.426326067 (587h41m27.297409112s) [noise=true , address=45.91.116.5:4298 , piece_id=W34NKHDFMGIHOVEUGOHNV3SPTFIINIESSKEHB74JSMST2CASOBXA, seg_pos={0 0}, upload=515181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:52292->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:52292->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52292->45.91.116.5:4298: use of closed network connection Dec 30 03:10:27.426334734 (587h41m27.297417779s) [noise=true , address=45.91.116.5:4298 , piece_id=W34NKHDFMGIHOVEUGOHNV3SPTFIINIESSKEHB74JSMST2CASOBXA, seg_pos={0 0}, upload=515181]: Long tail cancellation due to optimal threshold Dec 30 03:10:27.426353079 (587h41m27.297436114s) [seg_pos={0 0}, upload=515181]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 03:10:27.426438783 (587h41m27.297521828s) [upload=515181]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 03:10:27.426443992 (587h41m27.297527027s) [upload=515181]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 03:10:27.723842097 (587h41m27.594925142s) [upload=515181]: Done uploading segments... Dec 30 03:10:27.723866343 (587h41m27.594949388s) [upload=515181]: Upload finished. err: <nil> Duration: 5.924305249s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [.......X...X..XX..X.X......XXX.X.X..X........X.X.XX..X.X.XX.] duration: 5.731272533s trace: https://jaeger.dev.storj.io/trace/cd54c0b0e5ccd2b FAIL (2025-12-30 03:11:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 30 03:10:24.791854941 (587h41m24.662937976s) [upload=515181]: Starting upload Dec 30 03:10:24.791916989 (587h41m24.663000035s) [upload=515181]: Done starting upload Dec 30 03:10:24.792134443 (587h41m24.663217478s) [upload=515181]: Uploading segments... Dec 30 03:10:24.792188065 (587h41m24.6632711s) [upload=515181]: Got next segment. Inline: false Dec 30 03:10:24.792213754 (587h41m24.663296789s) [upload=515181]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 03:10:24.792218343 (587h41m24.663301388s) [upload=515181]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 03:10:25.040864557 (587h41m24.911947612s) [seg_pos={0 0}, upload=515181]: Begin upload segment... Dec 30 03:10:25.040917447 (587h41m24.912000492s) [seg_pos={0 0}, upload=515181]: Done begin upload segment. Dec 30 03:10:25.040948176 (587h41m24.912031221s) [noise=true , address=145.79.227.39:4014 , piece_id=D7LVR5LWLRGJNC76BATWBO3GFFL56Y64MOCXNHFGGKL2UUJ7PVGA, seg_pos={0 0}, upload=515181]: Uploading piece... Dec 30 03:10:25.041722661 (587h41m24.912805706s) [noise=true , address=45.91.116.14:4189 , piece_id=JNIUV7WE2YWU6UXGOWWUH7PIVZONMPUT2WK7R3MDUIMKWPYWRBWQ, seg_pos={0 0}, upload=515181]: Uploading piece... Dec 30 03:10:25.041881263 (587h41m24.912964308s) [noise=true , address=77.90.31.16:4735 , piece_id=YLJKK3EZF4A3DQGBQZFK7GIYOBUVDNFZWUMAVHKGHPFUZZBGN4RA, seg_pos={0 0}, upload=515181]: Uploading piece... Dec 30 03:10:25.042287336 (587h41m24.913370381s) [noise=true , address=145.79.227.14:4839 , piece_id=UXNJ5SUIF5RUK4XOO5UHCUTRD42IFFDQBQMWSNNSPQJIDWOJRWRQ, seg_pos={0 0}, upload=515181]: Uploading piece... Dec 30 03:10:25.042378269 (587h41m24.913461304s) [noise=true , address=145.79.227.36:4519 , piece_id=HAVZ6MCZHYFEUVHECN3BBCW3RQADF3D64VZH2AAZT7V2KTBKDIEQ, seg_pos={0 0}, upload=515181]: Uploading piece... Dec 30 03:10:25.042656148 (587h41m24.913739193s) [noise=true , address=145.79.227.46:4603 , piece_id=EQBACEDADMEZTPWD3UFXZ7HSNQ3X647O2D6KDQJULCFLKWMO7BTA, seg_pos={0 0}, upload=515181]: Uploading piece... Dec 30 03:10:25.042684613 (587h41m24.913767648s) [noise=true , address=145.79.227.17:4972 , piece_id=LWABVX33XMVXJQKQAIE6TFR2PHMHZGRJGF3NVLDWNZNBSEBJ4FLQ, seg_pos={0 0}, upload=515181]: Uploading piece... Dec 30 03:10:25.042917566 (587h41m24.914000602s) [noise=true , address=45.91.116.7:4009 , piece_id=SP7M5LMNHGEBZJ5IMXA6IDCQYUC5GUGCVFZDTMAZCAPU5FJDHNXA, seg_pos={0 0}, upload=515181]: Uploading piece... Dec 30 03:10:25.043061230 (587h41m24.914144265s) [noise=true , address=145.79.227.25:4619 , piece_id=YVOH6TCNPNPO2AXWP3NN6KN7WZBIBP23JNK6TOG3VUQEA72YZUGQ, seg_pos={0 0}, upload=515181]: Uploading piece... Dec 30 03:10:25.043182250 (587h41m24.914265285s) [noise=true , address=45.91.116.5:4298 , piece_id=W34NKHDFMGIHOVEUGOHNV3SPTFIINIESSKEHB74JSMST2CASOBXA, seg_pos={0 0}, upload=515181]: Uploading piece... Dec 30 03:10:25.043445151 (587h41m24.914528196s) [noise=true , address=145.79.227.42:4121 , piece_id=YZRW5773RMYVB2BUGV4HAE35JRCRTAPGNRRHW4M4MMKTJVYEABTA, seg_pos={0 0}, upload=515181]: Uploading piece... Dec 30 03:10:25.043453988 (587h41m24.914537033s) [noise=true , address=145.79.227.6:4259 , piece_id=7CD7TBZGXR6HBZ4TFYJFDP3IA5IZXKU7GUDDSF46MJWMUKXCGDKQ, seg_pos={0 0}, upload=515181]: Uploading piece... Dec 30 03:10:25.043851905 (587h41m24.91493495s) [noise=true , address=45.91.116.10:4084 , piece_id=UDSGCWMHWKPXMUFDQHZQOLYP3LPXZYZYWRE6GHOC6AGCQPD7D7JA, seg_pos={0 0}, upload=515181]: Uploading piece... Dec 30 03:10:25.044684911 (587h41m24.915767946s) [noise=true , address=45.91.116.6:4976 , piece_id=CLGQSMNBXE63EIPI3OS27QXCFQIC6COUP6Q4NIQUFTLNZIVBIYAA, seg_pos={0 0}, upload=515181]: Uploading piece... Dec 30 03:10:25.045245007 (587h41m24.916328053s) [noise=true , address=145.79.227.2:4113 , piece_id=P4VHP3ZVC2FABHWSCIO6522JFXSOE7EZMZWZILAHAWSRBG6UJCIQ, seg_pos={0 0}, upload=515181]: Uploading piece... Dec 30 03:10:25.045257662 (587h41m24.916340707s) [noise=true , address=145.79.227.50:4374 , piece_id=3SM3QTVE3YUHJUFBOK6ZBCMUUSX4PRDTQNTSOJHIRPWTZOALUVNA, seg_pos={0 0}, upload=515181]: Uploading piece... Dec 30 03:10:26.662003563 (587h41m26.533086598s) [upload=515181]: Next returned nil segment Dec 30 03:10:26.662018661 (587h41m26.533101706s) [upload=515181]: Waiting for error group managing segments... Dec 30 03:10:27.023175193 (587h41m26.894258238s) [noise=true , address=145.79.227.6:4259 , piece_id=7CD7TBZGXR6HBZ4TFYJFDP3IA5IZXKU7GUDDSF46MJWMUKXCGDKQ, seg_pos={0 0}, upload=515181]: Done uploading piece. err: <nil> Dec 30 03:10:27.023924228 (587h41m26.895007263s) [noise=true , address=145.79.227.17:4972 , piece_id=LWABVX33XMVXJQKQAIE6TFR2PHMHZGRJGF3NVLDWNZNBSEBJ4FLQ, seg_pos={0 0}, upload=515181]: Done uploading piece. err: <nil> Dec 30 03:10:27.030513275 (587h41m26.90159632s) [noise=true , address=145.79.227.14:4839 , piece_id=UXNJ5SUIF5RUK4XOO5UHCUTRD42IFFDQBQMWSNNSPQJIDWOJRWRQ, seg_pos={0 0}, upload=515181]: Done uploading piece. err: <nil> Dec 30 03:10:27.031052863 (587h41m26.902135898s) [noise=true , address=145.79.227.25:4619 , piece_id=YVOH6TCNPNPO2AXWP3NN6KN7WZBIBP23JNK6TOG3VUQEA72YZUGQ, seg_pos={0 0}, upload=515181]: Done uploading piece. err: <nil> Dec 30 03:10:27.038813891 (587h41m26.909896926s) [noise=true , address=145.79.227.36:4519 , piece_id=HAVZ6MCZHYFEUVHECN3BBCW3RQADF3D64VZH2AAZT7V2KTBKDIEQ, seg_pos={0 0}, upload=515181]: Done uploading piece. err: <nil> Dec 30 03:10:27.047944667 (587h41m26.919027712s) [noise=true , address=145.79.227.2:4113 , piece_id=P4VHP3ZVC2FABHWSCIO6522JFXSOE7EZMZWZILAHAWSRBG6UJCIQ, seg_pos={0 0}, upload=515181]: Done uploading piece. err: <nil> Dec 30 03:10:27.049040363 (587h41m26.920123398s) [noise=true , address=145.79.227.50:4374 , piece_id=3SM3QTVE3YUHJUFBOK6ZBCMUUSX4PRDTQNTSOJHIRPWTZOALUVNA, seg_pos={0 0}, upload=515181]: Done uploading piece. err: <nil> Dec 30 03:10:27.052510261 (587h41m26.923593306s) [noise=true , address=145.79.227.42:4121 , piece_id=YZRW5773RMYVB2BUGV4HAE35JRCRTAPGNRRHW4M4MMKTJVYEABTA, seg_pos={0 0}, upload=515181]: Done uploading piece. err: <nil> Dec 30 03:10:27.059180272 (587h41m26.930263317s) [noise=true , address=145.79.227.46:4603 , piece_id=EQBACEDADMEZTPWD3UFXZ7HSNQ3X647O2D6KDQJULCFLKWMO7BTA, seg_pos={0 0}, upload=515181]: Done uploading piece. err: <nil> Dec 30 03:10:27.068649223 (587h41m26.939732278s) [noise=true , address=145.79.227.39:4014 , piece_id=D7LVR5LWLRGJNC76BATWBO3GFFL56Y64MOCXNHFGGKL2UUJ7PVGA, seg_pos={0 0}, upload=515181]: Done uploading piece. err: <nil> Dec 30 03:10:27.219987115 (587h41m27.09107018s) [noise=true , address=77.90.31.16:4735 , piece_id=YLJKK3EZF4A3DQGBQZFK7GIYOBUVDNFZWUMAVHKGHPFUZZBGN4RA, seg_pos={0 0}, upload=515181]: Done uploading piece. err: <nil> Dec 30 03:10:27.335729688 (587h41m27.206812733s) [noise=true , address=45.91.116.7:4009 , piece_id=SP7M5LMNHGEBZJ5IMXA6IDCQYUC5GUGCVFZDTMAZCAPU5FJDHNXA, seg_pos={0 0}, upload=515181]: Done uploading piece. err: <nil> Dec 30 03:10:27.425829402 (587h41m27.296912437s) [noise=true , address=45.91.116.6:4976 , piece_id=CLGQSMNBXE63EIPI3OS27QXCFQIC6COUP6Q4NIQUFTLNZIVBIYAA, seg_pos={0 0}, upload=515181]: Done uploading piece. err: <nil> Dec 30 03:10:27.425849661 (587h41m27.296932696s) [seg_pos={0 0}, upload=515181]: Segment reached optimal threshold of 13 pieces. Dec 30 03:10:27.426016377 (587h41m27.297099412s) [noise=true , address=45.91.116.14:4189 , piece_id=JNIUV7WE2YWU6UXGOWWUH7PIVZONMPUT2WK7R3MDUIMKWPYWRBWQ, seg_pos={0 0}, upload=515181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 30 03:10:27.426022680 (587h41m27.297105715s) [noise=true , address=45.91.116.14:4189 , piece_id=JNIUV7WE2YWU6UXGOWWUH7PIVZONMPUT2WK7R3MDUIMKWPYWRBWQ, seg_pos={0 0}, upload=515181]: Long tail cancellation due to optimal threshold Dec 30 03:10:27.426288365 (587h41m27.297371411s) [noise=true , address=45.91.116.10:4084 , piece_id=UDSGCWMHWKPXMUFDQHZQOLYP3LPXZYZYWRE6GHOC6AGCQPD7D7JA, seg_pos={0 0}, upload=515181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 30 03:10:27.426308154 (587h41m27.297391199s) [noise=true , address=45.91.116.10:4084 , piece_id=UDSGCWMHWKPXMUFDQHZQOLYP3LPXZYZYWRE6GHOC6AGCQPD7D7JA, seg_pos={0 0}, upload=515181]: Long tail cancellation due to optimal threshold Dec 30 03:10:27.426326067 (587h41m27.297409112s) [noise=true , address=45.91.116.5:4298 , piece_id=W34NKHDFMGIHOVEUGOHNV3SPTFIINIESSKEHB74JSMST2CASOBXA, seg_pos={0 0}, upload=515181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:52292->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:52292->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52292->45.91.116.5:4298: use of closed network connection Dec 30 03:10:27.426334734 (587h41m27.297417779s) [noise=true , address=45.91.116.5:4298 , piece_id=W34NKHDFMGIHOVEUGOHNV3SPTFIINIESSKEHB74JSMST2CASOBXA, seg_pos={0 0}, upload=515181]: Long tail cancellation due to optimal threshold Dec 30 03:10:27.426353079 (587h41m27.297436114s) [seg_pos={0 0}, upload=515181]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 03:10:27.426438783 (587h41m27.297521828s) [upload=515181]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 03:10:27.426443992 (587h41m27.297527027s) [upload=515181]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 03:10:27.723842097 (587h41m27.594925142s) [upload=515181]: Done uploading segments... Dec 30 03:10:27.723866343 (587h41m27.594949388s) [upload=515181]: Upload finished. err: <nil> Duration: 5.731300365s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [...........................X...X.X.XX................XX.....] duration: 5.001591572s trace: https://jaeger.dev.storj.io/trace/55b84c463909ac27 FAIL (2025-12-30 03:11:21 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00161683s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............X................X......X......................] duration: 25.557695253s trace: https://jaeger.dev.storj.io/trace/6c3caa8adcfacbb9 FAIL (2025-12-30 03:11:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== true , address=145.79.227.35:4496 , piece_id=5X54ZGU4OIVI3DT622QE7IL66SVFLZYUICPODGMDKN7U4D4OJNSA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.040303259 (587h24m33.911386294s) [noise=true , address=77.90.31.12:4601 , piece_id=QQFK4PCL27YYF3UATGLV5KDENRNX6NHE7RXAZ2K66CC3WLHEBMWQ, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.040670117 (587h24m33.911753152s) [noise=true , address=77.90.31.21:4881 , piece_id=EMV4LHFK2VRHSW6PD3BBYGOXDAAYJKX6GEOXGGLOT3IXWNDQMU4A, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041006247 (587h24m33.912089292s) [noise=true , address=145.79.227.52:4472 , piece_id=DT2DAIC4ZI3WL7OGY55ZTTXJEI4IDRXPSJIQA3LTRC7K2F6EVPCA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041183314 (587h24m33.912266349s) [noise=true , address=145.79.227.26:4674 , piece_id=3BAN2ZXOFTSGZITYYHMZUS4AD35X6ZBOYA2HKOJCJI7VBWPJ3LSA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041313653 (587h24m33.912396688s) [noise=true , address=145.79.227.51:4429 , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041535194 (587h24m33.912618229s) [noise=true , address=145.79.227.33:4885 , piece_id=LA755L4RA4VJBHNCQBXOCFROZA3W6W6PUPKGJ5J772WAN44YMY2A, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041639693 (587h24m33.912722728s) [noise=true , address=145.79.227.43:4187 , piece_id=CAAFD3XIWWHZN7L7SYDMMSC5G5RKRLFYYHQZEEN4GAC2D6Y2EARA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041877216 (587h24m33.912960261s) [noise=true , address=145.79.227.46:4623 , piece_id=V4TEM7RRTTGBW5ISZOEAOPFIOZZ5EABO7BPWNRH5U5UPM2X7554Q, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041986634 (587h24m33.913069669s) [noise=true , address=45.91.116.13:4176 , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.042345657 (587h24m33.913428692s) [noise=true , address=45.91.116.10:4104 , piece_id=MRIWH6WGACE4UDNX55MIY6ZDWBAHS6E5KQS4QCOQN2BU7JI7ODCA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.042362019 (587h24m33.913445054s) [noise=true , address=77.90.31.11:4573 , piece_id=TORCC4W7DS5TJ7H7NWONEPNJOFULSZ6F5OHOP4BEVMGEH2MJA4DQ, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.042684192 (587h24m33.913767237s) [noise=true , address=145.79.227.49:4340 , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.042707327 (587h24m33.913790362s) [noise=true , address=145.79.227.32:4837 , piece_id=SQTHTF5KWZOKFYX6CGL6KCDRDAY7OJY6NA5OM3N4YCICDPM6DKXA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.043091528 (587h24m33.914174563s) [noise=true , address=45.91.116.5:4309 , piece_id=Y5HCMU6Z2FPBV3BNTUOC4F6JJWKNBKPOXWFVAFQTXUKQNHTLA4TA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.043544771 (587h24m33.914627816s) [noise=true , address=145.79.227.40:4053 , piece_id=3IZP7JBGSWBRBQHMJWXIDD73B5R3COWU5H3IYFGVBCTTLW7OUJ5A, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:35.347637489 (587h24m35.218720524s) [upload=514937]: Next returned nil segment Dec 30 02:53:35.347733591 (587h24m35.218816626s) [upload=514937]: Waiting for error group managing segments... Dec 30 02:53:35.674169656 (587h24m35.545252721s) [noise=true , address=64.49.13.27:4959 , piece_id=B4UZ67DIWNRHV77VTQONS6ZBUI2BPDDBM3ZS6FVCTABQZ6OHRJJA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:35.891295347 (587h24m35.762378382s) [noise=true , address=145.79.227.32:4837 , piece_id=SQTHTF5KWZOKFYX6CGL6KCDRDAY7OJY6NA5OM3N4YCICDPM6DKXA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:35.912619632 (587h24m35.783702677s) [noise=true , address=145.79.227.26:4674 , piece_id=3BAN2ZXOFTSGZITYYHMZUS4AD35X6ZBOYA2HKOJCJI7VBWPJ3LSA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:35.913054741 (587h24m35.784137786s) [noise=true , address=145.79.227.46:4623 , piece_id=V4TEM7RRTTGBW5ISZOEAOPFIOZZ5EABO7BPWNRH5U5UPM2X7554Q, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:35.913485270 (587h24m35.784568305s) [noise=true , address=145.79.227.35:4496 , piece_id=5X54ZGU4OIVI3DT622QE7IL66SVFLZYUICPODGMDKN7U4D4OJNSA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:35.923790864 (587h24m35.794873909s) [noise=true , address=145.79.227.52:4472 , piece_id=DT2DAIC4ZI3WL7OGY55ZTTXJEI4IDRXPSJIQA3LTRC7K2F6EVPCA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.109605534 (587h24m35.980688569s) [noise=true , address=77.90.31.21:4881 , piece_id=EMV4LHFK2VRHSW6PD3BBYGOXDAAYJKX6GEOXGGLOT3IXWNDQMU4A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.122502652 (587h24m35.993585697s) [noise=true , address=77.90.31.11:4573 , piece_id=TORCC4W7DS5TJ7H7NWONEPNJOFULSZ6F5OHOP4BEVMGEH2MJA4DQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.151131466 (587h24m36.022214511s) [noise=true , address=45.91.116.10:4104 , piece_id=MRIWH6WGACE4UDNX55MIY6ZDWBAHS6E5KQS4QCOQN2BU7JI7ODCA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.159562149 (587h24m36.030645184s) [noise=true , address=77.90.31.12:4601 , piece_id=QQFK4PCL27YYF3UATGLV5KDENRNX6NHE7RXAZ2K66CC3WLHEBMWQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.238339784 (587h24m36.10942282s) [noise=true , address=45.91.116.5:4309 , piece_id=Y5HCMU6Z2FPBV3BNTUOC4F6JJWKNBKPOXWFVAFQTXUKQNHTLA4TA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.813503456 (587h24m36.684586501s) [noise=true , address=145.79.227.33:4885 , piece_id=LA755L4RA4VJBHNCQBXOCFROZA3W6W6PUPKGJ5J772WAN44YMY2A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.826964278 (587h24m36.698047313s) [noise=true , address=145.79.227.43:4187 , piece_id=CAAFD3XIWWHZN7L7SYDMMSC5G5RKRLFYYHQZEEN4GAC2D6Y2EARA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.828562510 (587h24m36.699645545s) [noise=true , address=145.79.227.40:4053 , piece_id=3IZP7JBGSWBRBQHMJWXIDD73B5R3COWU5H3IYFGVBCTTLW7OUJ5A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.828575566 (587h24m36.699658601s) [seg_pos={0 0}, upload=514937]: Segment reached optimal threshold of 14 pieces. Dec 30 02:53:36.828752202 (587h24m36.699835237s) [noise=true , address=145.79.227.30:4780 , piece_id=DZQ5LIBMWXPJ4XXWORRDLQIAFXJDNDQEEBRMSBS44UX3D2HPTSWQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125WCbZ6a2NaewiLAHhHuPo91sQEfC9CZzEBV6rwF1JY7z13AEY): protocol: expected piece hash; context canceled; EOF Dec 30 02:53:36.828758714 (587h24m36.699841749s) [noise=true , address=145.79.227.30:4780 , piece_id=DZQ5LIBMWXPJ4XXWORRDLQIAFXJDNDQEEBRMSBS44UX3D2HPTSWQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold Dec 30 02:53:36.828762672 (587h24m36.699845707s) [noise=true , address=145.79.227.51:4429 , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qytmzwy7sTmWXskEbLWsWMniPz55zgho2akE7oh6ahAiT5SXf): protocol: write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection; write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection Dec 30 02:53:36.828774034 (587h24m36.699857069s) [noise=true , address=145.79.227.51:4429 , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold Dec 30 02:53:36.828889934 (587h24m36.699972969s) [noise=true , address=45.91.116.13:4176 , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ejsmBd4tAVwTHjamPYbryQb5x2vDnk9pBuX3CBF9WYFhTrFVJ): protocol: expected piece hash; context canceled; EOF Dec 30 02:53:36.828895144 (587h24m36.699978179s) [noise=true , address=45.91.116.13:4176 , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold Dec 30 02:53:36.828994173 (587h24m36.700077208s) [noise=true , address=145.79.227.49:4340 , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF Dec 30 02:53:36.828999793 (587h24m36.700082828s) [noise=true , address=145.79.227.49:4340 , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold Dec 30 02:53:36.829030332 (587h24m36.700113367s) [seg_pos={0 0}, upload=514937]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 02:53:36.829095486 (587h24m36.700178621s) [upload=514937]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 02:53:36.829100105 (587h24m36.70018315s) [upload=514937]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 02:53:37.128695270 (587h24m36.999778315s) [upload=514937]: Done uploading segments... Dec 30 02:53:37.128717683 (587h24m36.999800728s) [upload=514937]: Upload finished. err: <nil> Duration: 25.557721063s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [........X...X..XX..X.X......XXX.X.X..X........X.X.XX..X.X.XX] duration: 30.002559537s trace: https://jaeger.dev.storj.io/trace/cd54c0b0e5ccd2b FAIL (2025-12-30 03:10:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 30 03:10:24.791854941 (587h41m24.662937976s) [upload=515181]: Starting upload Dec 30 03:10:24.791916989 (587h41m24.663000035s) [upload=515181]: Done starting upload Dec 30 03:10:24.792134443 (587h41m24.663217478s) [upload=515181]: Uploading segments... Dec 30 03:10:24.792188065 (587h41m24.6632711s) [upload=515181]: Got next segment. Inline: false Dec 30 03:10:24.792213754 (587h41m24.663296789s) [upload=515181]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 03:10:24.792218343 (587h41m24.663301388s) [upload=515181]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 03:10:25.040864557 (587h41m24.911947612s) [seg_pos={0 0}, upload=515181]: Begin upload segment... Dec 30 03:10:25.040917447 (587h41m24.912000492s) [seg_pos={0 0}, upload=515181]: Done begin upload segment. Dec 30 03:10:25.040948176 (587h41m24.912031221s) [noise=true , address=145.79.227.39:4014 , piece_id=D7LVR5LWLRGJNC76BATWBO3GFFL56Y64MOCXNHFGGKL2UUJ7PVGA, seg_pos={0 0}, upload=515181]: Uploading piece... Dec 30 03:10:25.041722661 (587h41m24.912805706s) [noise=true , address=45.91.116.14:4189 , piece_id=JNIUV7WE2YWU6UXGOWWUH7PIVZONMPUT2WK7R3MDUIMKWPYWRBWQ, seg_pos={0 0}, upload=515181]: Uploading piece... Dec 30 03:10:25.041881263 (587h41m24.912964308s) [noise=true , address=77.90.31.16:4735 , piece_id=YLJKK3EZF4A3DQGBQZFK7GIYOBUVDNFZWUMAVHKGHPFUZZBGN4RA, seg_pos={0 0}, upload=515181]: Uploading piece... Dec 30 03:10:25.042287336 (587h41m24.913370381s) [noise=true , address=145.79.227.14:4839 , piece_id=UXNJ5SUIF5RUK4XOO5UHCUTRD42IFFDQBQMWSNNSPQJIDWOJRWRQ, seg_pos={0 0}, upload=515181]: Uploading piece... Dec 30 03:10:25.042378269 (587h41m24.913461304s) [noise=true , address=145.79.227.36:4519 , piece_id=HAVZ6MCZHYFEUVHECN3BBCW3RQADF3D64VZH2AAZT7V2KTBKDIEQ, seg_pos={0 0}, upload=515181]: Uploading piece... Dec 30 03:10:25.042656148 (587h41m24.913739193s) [noise=true , address=145.79.227.46:4603 , piece_id=EQBACEDADMEZTPWD3UFXZ7HSNQ3X647O2D6KDQJULCFLKWMO7BTA, seg_pos={0 0}, upload=515181]: Uploading piece... Dec 30 03:10:25.042684613 (587h41m24.913767648s) [noise=true , address=145.79.227.17:4972 , piece_id=LWABVX33XMVXJQKQAIE6TFR2PHMHZGRJGF3NVLDWNZNBSEBJ4FLQ, seg_pos={0 0}, upload=515181]: Uploading piece... Dec 30 03:10:25.042917566 (587h41m24.914000602s) [noise=true , address=45.91.116.7:4009 , piece_id=SP7M5LMNHGEBZJ5IMXA6IDCQYUC5GUGCVFZDTMAZCAPU5FJDHNXA, seg_pos={0 0}, upload=515181]: Uploading piece... Dec 30 03:10:25.043061230 (587h41m24.914144265s) [noise=true , address=145.79.227.25:4619 , piece_id=YVOH6TCNPNPO2AXWP3NN6KN7WZBIBP23JNK6TOG3VUQEA72YZUGQ, seg_pos={0 0}, upload=515181]: Uploading piece... Dec 30 03:10:25.043182250 (587h41m24.914265285s) [noise=true , address=45.91.116.5:4298 , piece_id=W34NKHDFMGIHOVEUGOHNV3SPTFIINIESSKEHB74JSMST2CASOBXA, seg_pos={0 0}, upload=515181]: Uploading piece... Dec 30 03:10:25.043445151 (587h41m24.914528196s) [noise=true , address=145.79.227.42:4121 , piece_id=YZRW5773RMYVB2BUGV4HAE35JRCRTAPGNRRHW4M4MMKTJVYEABTA, seg_pos={0 0}, upload=515181]: Uploading piece... Dec 30 03:10:25.043453988 (587h41m24.914537033s) [noise=true , address=145.79.227.6:4259 , piece_id=7CD7TBZGXR6HBZ4TFYJFDP3IA5IZXKU7GUDDSF46MJWMUKXCGDKQ, seg_pos={0 0}, upload=515181]: Uploading piece... Dec 30 03:10:25.043851905 (587h41m24.91493495s) [noise=true , address=45.91.116.10:4084 , piece_id=UDSGCWMHWKPXMUFDQHZQOLYP3LPXZYZYWRE6GHOC6AGCQPD7D7JA, seg_pos={0 0}, upload=515181]: Uploading piece... Dec 30 03:10:25.044684911 (587h41m24.915767946s) [noise=true , address=45.91.116.6:4976 , piece_id=CLGQSMNBXE63EIPI3OS27QXCFQIC6COUP6Q4NIQUFTLNZIVBIYAA, seg_pos={0 0}, upload=515181]: Uploading piece... Dec 30 03:10:25.045245007 (587h41m24.916328053s) [noise=true , address=145.79.227.2:4113 , piece_id=P4VHP3ZVC2FABHWSCIO6522JFXSOE7EZMZWZILAHAWSRBG6UJCIQ, seg_pos={0 0}, upload=515181]: Uploading piece... Dec 30 03:10:25.045257662 (587h41m24.916340707s) [noise=true , address=145.79.227.50:4374 , piece_id=3SM3QTVE3YUHJUFBOK6ZBCMUUSX4PRDTQNTSOJHIRPWTZOALUVNA, seg_pos={0 0}, upload=515181]: Uploading piece... Dec 30 03:10:26.662003563 (587h41m26.533086598s) [upload=515181]: Next returned nil segment Dec 30 03:10:26.662018661 (587h41m26.533101706s) [upload=515181]: Waiting for error group managing segments... Dec 30 03:10:27.023175193 (587h41m26.894258238s) [noise=true , address=145.79.227.6:4259 , piece_id=7CD7TBZGXR6HBZ4TFYJFDP3IA5IZXKU7GUDDSF46MJWMUKXCGDKQ, seg_pos={0 0}, upload=515181]: Done uploading piece. err: <nil> Dec 30 03:10:27.023924228 (587h41m26.895007263s) [noise=true , address=145.79.227.17:4972 , piece_id=LWABVX33XMVXJQKQAIE6TFR2PHMHZGRJGF3NVLDWNZNBSEBJ4FLQ, seg_pos={0 0}, upload=515181]: Done uploading piece. err: <nil> Dec 30 03:10:27.030513275 (587h41m26.90159632s) [noise=true , address=145.79.227.14:4839 , piece_id=UXNJ5SUIF5RUK4XOO5UHCUTRD42IFFDQBQMWSNNSPQJIDWOJRWRQ, seg_pos={0 0}, upload=515181]: Done uploading piece. err: <nil> Dec 30 03:10:27.031052863 (587h41m26.902135898s) [noise=true , address=145.79.227.25:4619 , piece_id=YVOH6TCNPNPO2AXWP3NN6KN7WZBIBP23JNK6TOG3VUQEA72YZUGQ, seg_pos={0 0}, upload=515181]: Done uploading piece. err: <nil> Dec 30 03:10:27.038813891 (587h41m26.909896926s) [noise=true , address=145.79.227.36:4519 , piece_id=HAVZ6MCZHYFEUVHECN3BBCW3RQADF3D64VZH2AAZT7V2KTBKDIEQ, seg_pos={0 0}, upload=515181]: Done uploading piece. err: <nil> Dec 30 03:10:27.047944667 (587h41m26.919027712s) [noise=true , address=145.79.227.2:4113 , piece_id=P4VHP3ZVC2FABHWSCIO6522JFXSOE7EZMZWZILAHAWSRBG6UJCIQ, seg_pos={0 0}, upload=515181]: Done uploading piece. err: <nil> Dec 30 03:10:27.049040363 (587h41m26.920123398s) [noise=true , address=145.79.227.50:4374 , piece_id=3SM3QTVE3YUHJUFBOK6ZBCMUUSX4PRDTQNTSOJHIRPWTZOALUVNA, seg_pos={0 0}, upload=515181]: Done uploading piece. err: <nil> Dec 30 03:10:27.052510261 (587h41m26.923593306s) [noise=true , address=145.79.227.42:4121 , piece_id=YZRW5773RMYVB2BUGV4HAE35JRCRTAPGNRRHW4M4MMKTJVYEABTA, seg_pos={0 0}, upload=515181]: Done uploading piece. err: <nil> Dec 30 03:10:27.059180272 (587h41m26.930263317s) [noise=true , address=145.79.227.46:4603 , piece_id=EQBACEDADMEZTPWD3UFXZ7HSNQ3X647O2D6KDQJULCFLKWMO7BTA, seg_pos={0 0}, upload=515181]: Done uploading piece. err: <nil> Dec 30 03:10:27.068649223 (587h41m26.939732278s) [noise=true , address=145.79.227.39:4014 , piece_id=D7LVR5LWLRGJNC76BATWBO3GFFL56Y64MOCXNHFGGKL2UUJ7PVGA, seg_pos={0 0}, upload=515181]: Done uploading piece. err: <nil> Dec 30 03:10:27.219987115 (587h41m27.09107018s) [noise=true , address=77.90.31.16:4735 , piece_id=YLJKK3EZF4A3DQGBQZFK7GIYOBUVDNFZWUMAVHKGHPFUZZBGN4RA, seg_pos={0 0}, upload=515181]: Done uploading piece. err: <nil> Dec 30 03:10:27.335729688 (587h41m27.206812733s) [noise=true , address=45.91.116.7:4009 , piece_id=SP7M5LMNHGEBZJ5IMXA6IDCQYUC5GUGCVFZDTMAZCAPU5FJDHNXA, seg_pos={0 0}, upload=515181]: Done uploading piece. err: <nil> Dec 30 03:10:27.425829402 (587h41m27.296912437s) [noise=true , address=45.91.116.6:4976 , piece_id=CLGQSMNBXE63EIPI3OS27QXCFQIC6COUP6Q4NIQUFTLNZIVBIYAA, seg_pos={0 0}, upload=515181]: Done uploading piece. err: <nil> Dec 30 03:10:27.425849661 (587h41m27.296932696s) [seg_pos={0 0}, upload=515181]: Segment reached optimal threshold of 13 pieces. Dec 30 03:10:27.426016377 (587h41m27.297099412s) [noise=true , address=45.91.116.14:4189 , piece_id=JNIUV7WE2YWU6UXGOWWUH7PIVZONMPUT2WK7R3MDUIMKWPYWRBWQ, seg_pos={0 0}, upload=515181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 30 03:10:27.426022680 (587h41m27.297105715s) [noise=true , address=45.91.116.14:4189 , piece_id=JNIUV7WE2YWU6UXGOWWUH7PIVZONMPUT2WK7R3MDUIMKWPYWRBWQ, seg_pos={0 0}, upload=515181]: Long tail cancellation due to optimal threshold Dec 30 03:10:27.426288365 (587h41m27.297371411s) [noise=true , address=45.91.116.10:4084 , piece_id=UDSGCWMHWKPXMUFDQHZQOLYP3LPXZYZYWRE6GHOC6AGCQPD7D7JA, seg_pos={0 0}, upload=515181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 30 03:10:27.426308154 (587h41m27.297391199s) [noise=true , address=45.91.116.10:4084 , piece_id=UDSGCWMHWKPXMUFDQHZQOLYP3LPXZYZYWRE6GHOC6AGCQPD7D7JA, seg_pos={0 0}, upload=515181]: Long tail cancellation due to optimal threshold Dec 30 03:10:27.426326067 (587h41m27.297409112s) [noise=true , address=45.91.116.5:4298 , piece_id=W34NKHDFMGIHOVEUGOHNV3SPTFIINIESSKEHB74JSMST2CASOBXA, seg_pos={0 0}, upload=515181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:52292->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:52292->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52292->45.91.116.5:4298: use of closed network connection Dec 30 03:10:27.426334734 (587h41m27.297417779s) [noise=true , address=45.91.116.5:4298 , piece_id=W34NKHDFMGIHOVEUGOHNV3SPTFIINIESSKEHB74JSMST2CASOBXA, seg_pos={0 0}, upload=515181]: Long tail cancellation due to optimal threshold Dec 30 03:10:27.426353079 (587h41m27.297436114s) [seg_pos={0 0}, upload=515181]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 03:10:27.426438783 (587h41m27.297521828s) [upload=515181]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 03:10:27.426443992 (587h41m27.297527027s) [upload=515181]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 03:10:27.723842097 (587h41m27.594925142s) [upload=515181]: Done uploading segments... Dec 30 03:10:27.723866343 (587h41m27.594949388s) [upload=515181]: Upload finished. err: <nil> Duration: 30.002585125s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [............................X...X.X.XX................XX....] duration: 3.48852937s trace: https://jaeger.dev.storj.io/trace/55b84c463909ac27 FAIL (2025-12-30 03:10:50 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.488556002s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [X.............X................X......X.....................] duration: 36.005021595s trace: https://jaeger.dev.storj.io/trace/6c3caa8adcfacbb9 FAIL (2025-12-30 03:10:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== true , address=145.79.227.35:4496 , piece_id=5X54ZGU4OIVI3DT622QE7IL66SVFLZYUICPODGMDKN7U4D4OJNSA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.040303259 (587h24m33.911386294s) [noise=true , address=77.90.31.12:4601 , piece_id=QQFK4PCL27YYF3UATGLV5KDENRNX6NHE7RXAZ2K66CC3WLHEBMWQ, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.040670117 (587h24m33.911753152s) [noise=true , address=77.90.31.21:4881 , piece_id=EMV4LHFK2VRHSW6PD3BBYGOXDAAYJKX6GEOXGGLOT3IXWNDQMU4A, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041006247 (587h24m33.912089292s) [noise=true , address=145.79.227.52:4472 , piece_id=DT2DAIC4ZI3WL7OGY55ZTTXJEI4IDRXPSJIQA3LTRC7K2F6EVPCA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041183314 (587h24m33.912266349s) [noise=true , address=145.79.227.26:4674 , piece_id=3BAN2ZXOFTSGZITYYHMZUS4AD35X6ZBOYA2HKOJCJI7VBWPJ3LSA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041313653 (587h24m33.912396688s) [noise=true , address=145.79.227.51:4429 , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041535194 (587h24m33.912618229s) [noise=true , address=145.79.227.33:4885 , piece_id=LA755L4RA4VJBHNCQBXOCFROZA3W6W6PUPKGJ5J772WAN44YMY2A, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041639693 (587h24m33.912722728s) [noise=true , address=145.79.227.43:4187 , piece_id=CAAFD3XIWWHZN7L7SYDMMSC5G5RKRLFYYHQZEEN4GAC2D6Y2EARA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041877216 (587h24m33.912960261s) [noise=true , address=145.79.227.46:4623 , piece_id=V4TEM7RRTTGBW5ISZOEAOPFIOZZ5EABO7BPWNRH5U5UPM2X7554Q, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041986634 (587h24m33.913069669s) [noise=true , address=45.91.116.13:4176 , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.042345657 (587h24m33.913428692s) [noise=true , address=45.91.116.10:4104 , piece_id=MRIWH6WGACE4UDNX55MIY6ZDWBAHS6E5KQS4QCOQN2BU7JI7ODCA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.042362019 (587h24m33.913445054s) [noise=true , address=77.90.31.11:4573 , piece_id=TORCC4W7DS5TJ7H7NWONEPNJOFULSZ6F5OHOP4BEVMGEH2MJA4DQ, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.042684192 (587h24m33.913767237s) [noise=true , address=145.79.227.49:4340 , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.042707327 (587h24m33.913790362s) [noise=true , address=145.79.227.32:4837 , piece_id=SQTHTF5KWZOKFYX6CGL6KCDRDAY7OJY6NA5OM3N4YCICDPM6DKXA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.043091528 (587h24m33.914174563s) [noise=true , address=45.91.116.5:4309 , piece_id=Y5HCMU6Z2FPBV3BNTUOC4F6JJWKNBKPOXWFVAFQTXUKQNHTLA4TA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.043544771 (587h24m33.914627816s) [noise=true , address=145.79.227.40:4053 , piece_id=3IZP7JBGSWBRBQHMJWXIDD73B5R3COWU5H3IYFGVBCTTLW7OUJ5A, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:35.347637489 (587h24m35.218720524s) [upload=514937]: Next returned nil segment Dec 30 02:53:35.347733591 (587h24m35.218816626s) [upload=514937]: Waiting for error group managing segments... Dec 30 02:53:35.674169656 (587h24m35.545252721s) [noise=true , address=64.49.13.27:4959 , piece_id=B4UZ67DIWNRHV77VTQONS6ZBUI2BPDDBM3ZS6FVCTABQZ6OHRJJA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:35.891295347 (587h24m35.762378382s) [noise=true , address=145.79.227.32:4837 , piece_id=SQTHTF5KWZOKFYX6CGL6KCDRDAY7OJY6NA5OM3N4YCICDPM6DKXA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:35.912619632 (587h24m35.783702677s) [noise=true , address=145.79.227.26:4674 , piece_id=3BAN2ZXOFTSGZITYYHMZUS4AD35X6ZBOYA2HKOJCJI7VBWPJ3LSA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:35.913054741 (587h24m35.784137786s) [noise=true , address=145.79.227.46:4623 , piece_id=V4TEM7RRTTGBW5ISZOEAOPFIOZZ5EABO7BPWNRH5U5UPM2X7554Q, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:35.913485270 (587h24m35.784568305s) [noise=true , address=145.79.227.35:4496 , piece_id=5X54ZGU4OIVI3DT622QE7IL66SVFLZYUICPODGMDKN7U4D4OJNSA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:35.923790864 (587h24m35.794873909s) [noise=true , address=145.79.227.52:4472 , piece_id=DT2DAIC4ZI3WL7OGY55ZTTXJEI4IDRXPSJIQA3LTRC7K2F6EVPCA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.109605534 (587h24m35.980688569s) [noise=true , address=77.90.31.21:4881 , piece_id=EMV4LHFK2VRHSW6PD3BBYGOXDAAYJKX6GEOXGGLOT3IXWNDQMU4A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.122502652 (587h24m35.993585697s) [noise=true , address=77.90.31.11:4573 , piece_id=TORCC4W7DS5TJ7H7NWONEPNJOFULSZ6F5OHOP4BEVMGEH2MJA4DQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.151131466 (587h24m36.022214511s) [noise=true , address=45.91.116.10:4104 , piece_id=MRIWH6WGACE4UDNX55MIY6ZDWBAHS6E5KQS4QCOQN2BU7JI7ODCA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.159562149 (587h24m36.030645184s) [noise=true , address=77.90.31.12:4601 , piece_id=QQFK4PCL27YYF3UATGLV5KDENRNX6NHE7RXAZ2K66CC3WLHEBMWQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.238339784 (587h24m36.10942282s) [noise=true , address=45.91.116.5:4309 , piece_id=Y5HCMU6Z2FPBV3BNTUOC4F6JJWKNBKPOXWFVAFQTXUKQNHTLA4TA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.813503456 (587h24m36.684586501s) [noise=true , address=145.79.227.33:4885 , piece_id=LA755L4RA4VJBHNCQBXOCFROZA3W6W6PUPKGJ5J772WAN44YMY2A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.826964278 (587h24m36.698047313s) [noise=true , address=145.79.227.43:4187 , piece_id=CAAFD3XIWWHZN7L7SYDMMSC5G5RKRLFYYHQZEEN4GAC2D6Y2EARA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.828562510 (587h24m36.699645545s) [noise=true , address=145.79.227.40:4053 , piece_id=3IZP7JBGSWBRBQHMJWXIDD73B5R3COWU5H3IYFGVBCTTLW7OUJ5A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.828575566 (587h24m36.699658601s) [seg_pos={0 0}, upload=514937]: Segment reached optimal threshold of 14 pieces. Dec 30 02:53:36.828752202 (587h24m36.699835237s) [noise=true , address=145.79.227.30:4780 , piece_id=DZQ5LIBMWXPJ4XXWORRDLQIAFXJDNDQEEBRMSBS44UX3D2HPTSWQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125WCbZ6a2NaewiLAHhHuPo91sQEfC9CZzEBV6rwF1JY7z13AEY): protocol: expected piece hash; context canceled; EOF Dec 30 02:53:36.828758714 (587h24m36.699841749s) [noise=true , address=145.79.227.30:4780 , piece_id=DZQ5LIBMWXPJ4XXWORRDLQIAFXJDNDQEEBRMSBS44UX3D2HPTSWQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold Dec 30 02:53:36.828762672 (587h24m36.699845707s) [noise=true , address=145.79.227.51:4429 , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qytmzwy7sTmWXskEbLWsWMniPz55zgho2akE7oh6ahAiT5SXf): protocol: write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection; write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection Dec 30 02:53:36.828774034 (587h24m36.699857069s) [noise=true , address=145.79.227.51:4429 , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold Dec 30 02:53:36.828889934 (587h24m36.699972969s) [noise=true , address=45.91.116.13:4176 , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ejsmBd4tAVwTHjamPYbryQb5x2vDnk9pBuX3CBF9WYFhTrFVJ): protocol: expected piece hash; context canceled; EOF Dec 30 02:53:36.828895144 (587h24m36.699978179s) [noise=true , address=45.91.116.13:4176 , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold Dec 30 02:53:36.828994173 (587h24m36.700077208s) [noise=true , address=145.79.227.49:4340 , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF Dec 30 02:53:36.828999793 (587h24m36.700082828s) [noise=true , address=145.79.227.49:4340 , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold Dec 30 02:53:36.829030332 (587h24m36.700113367s) [seg_pos={0 0}, upload=514937]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 02:53:36.829095486 (587h24m36.700178621s) [upload=514937]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 02:53:36.829100105 (587h24m36.70018315s) [upload=514937]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 02:53:37.128695270 (587h24m36.999778315s) [upload=514937]: Done uploading segments... Dec 30 02:53:37.128717683 (587h24m36.999800728s) [upload=514937]: Upload finished. err: <nil> Duration: 36.005049147s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [.............................X...X.X.XX................XX...] duration: 3.492592179s trace: https://jaeger.dev.storj.io/trace/55b84c463909ac27 FAIL (2025-12-30 03:10:34 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.492612688s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [X........X...X..XX..X.X......XXX.X.X..X........X.X.XX..X.X.X] duration: 30.002544145s trace: https://jaeger.dev.storj.io/trace/d4ec1551422a062 FAIL (2025-12-30 03:10:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: 6HEFPQ22RA5A7BAWS45WNNLPAAXGH7QDTFD7L3RJMJF6MTJ3LGWQ) context deadline exceeded; piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: 3V6CWBPXKHQGD22OCSGFEDURTG5YWAWEV24SAFDEZM2JP4XIYOVA) context deadline exceeded ======LOGS====== Dec 30 03:09:53.704645759 (587h40m53.575728804s) [upload=515173]: Starting upload Dec 30 03:09:53.704703469 (587h40m53.575786504s) [upload=515173]: Done starting upload Dec 30 03:09:53.704754906 (587h40m53.575837951s) [upload=515173]: Uploading segments... Dec 30 03:09:53.705080497 (587h40m53.576163552s) [upload=515173]: Got next segment. Inline: false Dec 30 03:09:53.705130412 (587h40m53.576213447s) [upload=515173]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 03:09:53.705136002 (587h40m53.576219047s) [upload=515173]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 03:09:53.951308966 (587h40m53.822392001s) [seg_pos={0 0}, upload=515173]: Begin upload segment... Dec 30 03:09:53.951361907 (587h40m53.822444942s) [seg_pos={0 0}, upload=515173]: Done begin upload segment. Dec 30 03:09:53.951403937 (587h40m53.822486972s) [noise=true , address=145.79.227.36:4518 , piece_id=46FDH7OL5MZCHG5O55XDQCGDA3FZRN7X6QLMBXNBGPHEEGPNS5CA, seg_pos={0 0}, upload=515173]: Uploading piece... Dec 30 03:09:53.951447961 (587h40m53.822531006s) [noise=true , address=145.79.227.11:4732 , piece_id=CDTCTH76ANVFIGWZKCSXMBBQVJ7WAIICRS4TL6OH3Q2FXB6NRCZQ, seg_pos={0 0}, upload=515173]: Uploading piece... Dec 30 03:09:53.952165337 (587h40m53.823248372s) [noise=true , address=45.91.116.5:4298 , piece_id=XFKEMGXHMKS75BYMRZBYBKZSHK2UE3DEDM7G7ZYGA7NTOGC6UGTQ, seg_pos={0 0}, upload=515173]: Uploading piece... Dec 30 03:09:53.952604944 (587h40m53.823687979s) [noise=true , address=145.79.227.13:4802 , piece_id=4NI54ZPNYTE7J7WW3K2TPXY5GC3C3VYP4OPSBXCEFBZNO2RRA6OA, seg_pos={0 0}, upload=515173]: Uploading piece... Dec 30 03:09:53.952787170 (587h40m53.823870205s) [noise=true , address=77.90.31.13:4626 , piece_id=6HEFPQ22RA5A7BAWS45WNNLPAAXGH7QDTFD7L3RJMJF6MTJ3LGWQ, seg_pos={0 0}, upload=515173]: Uploading piece... Dec 30 03:09:53.953097702 (587h40m53.824180737s) [noise=true , address=45.91.116.10:4081 , piece_id=3V6CWBPXKHQGD22OCSGFEDURTG5YWAWEV24SAFDEZM2JP4XIYOVA, seg_pos={0 0}, upload=515173]: Uploading piece... Dec 30 03:09:53.953243620 (587h40m53.824326656s) [noise=true , address=145.79.227.17:4970 , piece_id=BSARQ2UV2XNP4OFC6TYBGPDINUJQ54BWBEYCW73GCPVCDXNB53HA, seg_pos={0 0}, upload=515173]: Uploading piece... Dec 30 03:09:53.953424194 (587h40m53.824507239s) [noise=true , address=145.79.227.52:4446 , piece_id=ZAJ3O5EUCPWNHMOW4GS6OGHW26XA3JYXTLCJZMOBAF7NUMDO7H3A, seg_pos={0 0}, upload=515173]: Uploading piece... Dec 30 03:09:53.953837822 (587h40m53.824920867s) [noise=true , address=145.79.227.41:4086 , piece_id=CC33BXJAS6LQKX6CKNT4BT4ZV32LXFBR4SMLN6TX3IPTKTBS3MPA, seg_pos={0 0}, upload=515173]: Uploading piece... Dec 30 03:09:53.954314920 (587h40m53.825397955s) [noise=true , address=145.79.227.42:4121 , piece_id=C3OHJWG2HE4BMDO7PJRZZLTNPAIRGEM5L3U45YXOMSAN7SO2PVIA, seg_pos={0 0}, upload=515173]: Uploading piece... Dec 30 03:09:53.954675437 (587h40m53.825758472s) [noise=true , address=45.91.116.9:4261 , piece_id=PAIN2E2RURT43WUABHXEDRZBY2LBOIZSRX7EF66JDBKFBNDR6L4Q, seg_pos={0 0}, upload=515173]: Uploading piece... Dec 30 03:09:53.955007468 (587h40m53.826090504s) [noise=true , address=145.79.227.7:4295 , piece_id=MID5QCQAQDTW22LW3SD4OL4EPPKYGDDHDDKRVF7RNRVNCOKRH6FQ, seg_pos={0 0}, upload=515173]: Uploading piece... Dec 30 03:09:53.955196749 (587h40m53.826279795s) [noise=true , address=145.79.227.25:4619 , piece_id=KVOYOUZQM245OBWNPBZXXJW4H2RT4PV2C3RQN3HSKHQ5HVMZZBEQ, seg_pos={0 0}, upload=515173]: Uploading piece... Dec 30 03:09:53.955852819 (587h40m53.826935864s) [noise=true , address=77.90.31.10:4517 , piece_id=J5LFGKULKNCN6ZCCKJGDB3OUONWK747VNSYRMIL7H6DZ24C4B3BA, seg_pos={0 0}, upload=515173]: Uploading piece... Dec 30 03:09:53.956044854 (587h40m53.827127899s) [noise=true , address=145.79.227.43:4158 , piece_id=OOTGKRCGYJB3XBOPS62JPYH6YMGVE7NHPDDNTYZMJNTGYEK44KLQ, seg_pos={0 0}, upload=515173]: Uploading piece... Dec 30 03:09:53.956430598 (587h40m53.827513644s) [noise=true , address=45.91.116.13:4155 , piece_id=D3A2LHAR6RFQRSB6WWY3NO7QKBUGT6YXODMACCORINTDEOVTYETQ, seg_pos={0 0}, upload=515173]: Uploading piece... Dec 30 03:09:55.657369114 (587h40m55.528452149s) [upload=515173]: Next returned nil segment Dec 30 03:09:55.657381828 (587h40m55.528464874s) [upload=515173]: Waiting for error group managing segments... Dec 30 03:09:55.941079956 (587h40m55.812163001s) [noise=true , address=145.79.227.17:4970 , piece_id=BSARQ2UV2XNP4OFC6TYBGPDINUJQ54BWBEYCW73GCPVCDXNB53HA, seg_pos={0 0}, upload=515173]: Done uploading piece. err: <nil> Dec 30 03:09:55.941960814 (587h40m55.813043849s) [noise=true , address=145.79.227.52:4446 , piece_id=ZAJ3O5EUCPWNHMOW4GS6OGHW26XA3JYXTLCJZMOBAF7NUMDO7H3A, seg_pos={0 0}, upload=515173]: Done uploading piece. err: <nil> Dec 30 03:09:55.942446940 (587h40m55.813529985s) [noise=true , address=145.79.227.13:4802 , piece_id=4NI54ZPNYTE7J7WW3K2TPXY5GC3C3VYP4OPSBXCEFBZNO2RRA6OA, seg_pos={0 0}, upload=515173]: Done uploading piece. err: <nil> Dec 30 03:09:55.944440515 (587h40m55.815523551s) [noise=true , address=145.79.227.11:4732 , piece_id=CDTCTH76ANVFIGWZKCSXMBBQVJ7WAIICRS4TL6OH3Q2FXB6NRCZQ, seg_pos={0 0}, upload=515173]: Done uploading piece. err: <nil> Dec 30 03:09:55.945949398 (587h40m55.817032443s) [noise=true , address=145.79.227.7:4295 , piece_id=MID5QCQAQDTW22LW3SD4OL4EPPKYGDDHDDKRVF7RNRVNCOKRH6FQ, seg_pos={0 0}, upload=515173]: Done uploading piece. err: <nil> Dec 30 03:09:55.949651298 (587h40m55.820734333s) [noise=true , address=145.79.227.36:4518 , piece_id=46FDH7OL5MZCHG5O55XDQCGDA3FZRN7X6QLMBXNBGPHEEGPNS5CA, seg_pos={0 0}, upload=515173]: Done uploading piece. err: <nil> Dec 30 03:09:55.952044795 (587h40m55.82312784s) [noise=true , address=145.79.227.41:4086 , piece_id=CC33BXJAS6LQKX6CKNT4BT4ZV32LXFBR4SMLN6TX3IPTKTBS3MPA, seg_pos={0 0}, upload=515173]: Done uploading piece. err: <nil> Dec 30 03:09:55.957722687 (587h40m55.828805723s) [noise=true , address=145.79.227.42:4121 , piece_id=C3OHJWG2HE4BMDO7PJRZZLTNPAIRGEM5L3U45YXOMSAN7SO2PVIA, seg_pos={0 0}, upload=515173]: Done uploading piece. err: <nil> Dec 30 03:09:55.962722908 (587h40m55.833805943s) [noise=true , address=145.79.227.25:4619 , piece_id=KVOYOUZQM245OBWNPBZXXJW4H2RT4PV2C3RQN3HSKHQ5HVMZZBEQ, seg_pos={0 0}, upload=515173]: Done uploading piece. err: <nil> Dec 30 03:09:56.164947107 (587h40m56.036030152s) [noise=true , address=45.91.116.10:4081 , piece_id=3V6CWBPXKHQGD22OCSGFEDURTG5YWAWEV24SAFDEZM2JP4XIYOVA, seg_pos={0 0}, upload=515173]: Done uploading piece. err: <nil> Dec 30 03:09:56.175771448 (587h40m56.046854493s) [noise=true , address=45.91.116.13:4155 , piece_id=D3A2LHAR6RFQRSB6WWY3NO7QKBUGT6YXODMACCORINTDEOVTYETQ, seg_pos={0 0}, upload=515173]: Done uploading piece. err: <nil> Dec 30 03:09:56.311928244 (587h40m56.183011279s) [noise=true , address=45.91.116.5:4298 , piece_id=XFKEMGXHMKS75BYMRZBYBKZSHK2UE3DEDM7G7ZYGA7NTOGC6UGTQ, seg_pos={0 0}, upload=515173]: Done uploading piece. err: <nil> Dec 30 03:09:56.602901164 (587h40m56.473984199s) [noise=true , address=77.90.31.13:4626 , piece_id=6HEFPQ22RA5A7BAWS45WNNLPAAXGH7QDTFD7L3RJMJF6MTJ3LGWQ, seg_pos={0 0}, upload=515173]: Done uploading piece. err: <nil> Dec 30 03:09:56.602920310 (587h40m56.474003355s) [seg_pos={0 0}, upload=515173]: Segment reached optimal threshold of 13 pieces. Dec 30 03:09:56.603087989 (587h40m56.474171024s) [noise=true , address=77.90.31.10:4517 , piece_id=J5LFGKULKNCN6ZCCKJGDB3OUONWK747VNSYRMIL7H6DZ24C4B3BA, seg_pos={0 0}, upload=515173]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 30 03:09:56.603095483 (587h40m56.474178518s) [noise=true , address=77.90.31.10:4517 , piece_id=J5LFGKULKNCN6ZCCKJGDB3OUONWK747VNSYRMIL7H6DZ24C4B3BA, seg_pos={0 0}, upload=515173]: Long tail cancellation due to optimal threshold Dec 30 03:09:56.603168744 (587h40m56.474251789s) [noise=true , address=145.79.227.43:4158 , piece_id=OOTGKRCGYJB3XBOPS62JPYH6YMGVE7NHPDDNTYZMJNTGYEK44KLQ, seg_pos={0 0}, upload=515173]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF Dec 30 03:09:56.603176408 (587h40m56.474259443s) [noise=true , address=145.79.227.43:4158 , piece_id=OOTGKRCGYJB3XBOPS62JPYH6YMGVE7NHPDDNTYZMJNTGYEK44KLQ, seg_pos={0 0}, upload=515173]: Long tail cancellation due to optimal threshold Dec 30 03:09:56.603266279 (587h40m56.474349324s) [noise=true , address=45.91.116.9:4261 , piece_id=PAIN2E2RURT43WUABHXEDRZBY2LBOIZSRX7EF66JDBKFBNDR6L4Q, seg_pos={0 0}, upload=515173]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF Dec 30 03:09:56.603271840 (587h40m56.474354885s) [noise=true , address=45.91.116.9:4261 , piece_id=PAIN2E2RURT43WUABHXEDRZBY2LBOIZSRX7EF66JDBKFBNDR6L4Q, seg_pos={0 0}, upload=515173]: Long tail cancellation due to optimal threshold Dec 30 03:09:56.603287630 (587h40m56.474370665s) [seg_pos={0 0}, upload=515173]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 03:09:56.603355689 (587h40m56.474438724s) [upload=515173]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 03:09:56.603360578 (587h40m56.474443613s) [upload=515173]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 03:09:56.889705656 (587h40m56.760788701s) [upload=515173]: Done uploading segments... Dec 30 03:09:56.889729612 (587h40m56.760812657s) [upload=515173]: Upload finished. err: <nil> Duration: 30.002566176s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [..............................X...X.X.XX................XX..] duration: 3.762439041s trace: https://jaeger.dev.storj.io/trace/55b84c463909ac27 FAIL (2025-12-30 03:10:00 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.762466684s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.X.............X................X......X....................] duration: 35.920877033s trace: https://jaeger.dev.storj.io/trace/6c3caa8adcfacbb9 FAIL (2025-12-30 03:09:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== true , address=145.79.227.35:4496 , piece_id=5X54ZGU4OIVI3DT622QE7IL66SVFLZYUICPODGMDKN7U4D4OJNSA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.040303259 (587h24m33.911386294s) [noise=true , address=77.90.31.12:4601 , piece_id=QQFK4PCL27YYF3UATGLV5KDENRNX6NHE7RXAZ2K66CC3WLHEBMWQ, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.040670117 (587h24m33.911753152s) [noise=true , address=77.90.31.21:4881 , piece_id=EMV4LHFK2VRHSW6PD3BBYGOXDAAYJKX6GEOXGGLOT3IXWNDQMU4A, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041006247 (587h24m33.912089292s) [noise=true , address=145.79.227.52:4472 , piece_id=DT2DAIC4ZI3WL7OGY55ZTTXJEI4IDRXPSJIQA3LTRC7K2F6EVPCA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041183314 (587h24m33.912266349s) [noise=true , address=145.79.227.26:4674 , piece_id=3BAN2ZXOFTSGZITYYHMZUS4AD35X6ZBOYA2HKOJCJI7VBWPJ3LSA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041313653 (587h24m33.912396688s) [noise=true , address=145.79.227.51:4429 , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041535194 (587h24m33.912618229s) [noise=true , address=145.79.227.33:4885 , piece_id=LA755L4RA4VJBHNCQBXOCFROZA3W6W6PUPKGJ5J772WAN44YMY2A, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041639693 (587h24m33.912722728s) [noise=true , address=145.79.227.43:4187 , piece_id=CAAFD3XIWWHZN7L7SYDMMSC5G5RKRLFYYHQZEEN4GAC2D6Y2EARA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041877216 (587h24m33.912960261s) [noise=true , address=145.79.227.46:4623 , piece_id=V4TEM7RRTTGBW5ISZOEAOPFIOZZ5EABO7BPWNRH5U5UPM2X7554Q, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041986634 (587h24m33.913069669s) [noise=true , address=45.91.116.13:4176 , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.042345657 (587h24m33.913428692s) [noise=true , address=45.91.116.10:4104 , piece_id=MRIWH6WGACE4UDNX55MIY6ZDWBAHS6E5KQS4QCOQN2BU7JI7ODCA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.042362019 (587h24m33.913445054s) [noise=true , address=77.90.31.11:4573 , piece_id=TORCC4W7DS5TJ7H7NWONEPNJOFULSZ6F5OHOP4BEVMGEH2MJA4DQ, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.042684192 (587h24m33.913767237s) [noise=true , address=145.79.227.49:4340 , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.042707327 (587h24m33.913790362s) [noise=true , address=145.79.227.32:4837 , piece_id=SQTHTF5KWZOKFYX6CGL6KCDRDAY7OJY6NA5OM3N4YCICDPM6DKXA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.043091528 (587h24m33.914174563s) [noise=true , address=45.91.116.5:4309 , piece_id=Y5HCMU6Z2FPBV3BNTUOC4F6JJWKNBKPOXWFVAFQTXUKQNHTLA4TA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.043544771 (587h24m33.914627816s) [noise=true , address=145.79.227.40:4053 , piece_id=3IZP7JBGSWBRBQHMJWXIDD73B5R3COWU5H3IYFGVBCTTLW7OUJ5A, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:35.347637489 (587h24m35.218720524s) [upload=514937]: Next returned nil segment Dec 30 02:53:35.347733591 (587h24m35.218816626s) [upload=514937]: Waiting for error group managing segments... Dec 30 02:53:35.674169656 (587h24m35.545252721s) [noise=true , address=64.49.13.27:4959 , piece_id=B4UZ67DIWNRHV77VTQONS6ZBUI2BPDDBM3ZS6FVCTABQZ6OHRJJA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:35.891295347 (587h24m35.762378382s) [noise=true , address=145.79.227.32:4837 , piece_id=SQTHTF5KWZOKFYX6CGL6KCDRDAY7OJY6NA5OM3N4YCICDPM6DKXA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:35.912619632 (587h24m35.783702677s) [noise=true , address=145.79.227.26:4674 , piece_id=3BAN2ZXOFTSGZITYYHMZUS4AD35X6ZBOYA2HKOJCJI7VBWPJ3LSA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:35.913054741 (587h24m35.784137786s) [noise=true , address=145.79.227.46:4623 , piece_id=V4TEM7RRTTGBW5ISZOEAOPFIOZZ5EABO7BPWNRH5U5UPM2X7554Q, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:35.913485270 (587h24m35.784568305s) [noise=true , address=145.79.227.35:4496 , piece_id=5X54ZGU4OIVI3DT622QE7IL66SVFLZYUICPODGMDKN7U4D4OJNSA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:35.923790864 (587h24m35.794873909s) [noise=true , address=145.79.227.52:4472 , piece_id=DT2DAIC4ZI3WL7OGY55ZTTXJEI4IDRXPSJIQA3LTRC7K2F6EVPCA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.109605534 (587h24m35.980688569s) [noise=true , address=77.90.31.21:4881 , piece_id=EMV4LHFK2VRHSW6PD3BBYGOXDAAYJKX6GEOXGGLOT3IXWNDQMU4A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.122502652 (587h24m35.993585697s) [noise=true , address=77.90.31.11:4573 , piece_id=TORCC4W7DS5TJ7H7NWONEPNJOFULSZ6F5OHOP4BEVMGEH2MJA4DQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.151131466 (587h24m36.022214511s) [noise=true , address=45.91.116.10:4104 , piece_id=MRIWH6WGACE4UDNX55MIY6ZDWBAHS6E5KQS4QCOQN2BU7JI7ODCA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.159562149 (587h24m36.030645184s) [noise=true , address=77.90.31.12:4601 , piece_id=QQFK4PCL27YYF3UATGLV5KDENRNX6NHE7RXAZ2K66CC3WLHEBMWQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.238339784 (587h24m36.10942282s) [noise=true , address=45.91.116.5:4309 , piece_id=Y5HCMU6Z2FPBV3BNTUOC4F6JJWKNBKPOXWFVAFQTXUKQNHTLA4TA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.813503456 (587h24m36.684586501s) [noise=true , address=145.79.227.33:4885 , piece_id=LA755L4RA4VJBHNCQBXOCFROZA3W6W6PUPKGJ5J772WAN44YMY2A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.826964278 (587h24m36.698047313s) [noise=true , address=145.79.227.43:4187 , piece_id=CAAFD3XIWWHZN7L7SYDMMSC5G5RKRLFYYHQZEEN4GAC2D6Y2EARA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.828562510 (587h24m36.699645545s) [noise=true , address=145.79.227.40:4053 , piece_id=3IZP7JBGSWBRBQHMJWXIDD73B5R3COWU5H3IYFGVBCTTLW7OUJ5A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.828575566 (587h24m36.699658601s) [seg_pos={0 0}, upload=514937]: Segment reached optimal threshold of 14 pieces. Dec 30 02:53:36.828752202 (587h24m36.699835237s) [noise=true , address=145.79.227.30:4780 , piece_id=DZQ5LIBMWXPJ4XXWORRDLQIAFXJDNDQEEBRMSBS44UX3D2HPTSWQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125WCbZ6a2NaewiLAHhHuPo91sQEfC9CZzEBV6rwF1JY7z13AEY): protocol: expected piece hash; context canceled; EOF Dec 30 02:53:36.828758714 (587h24m36.699841749s) [noise=true , address=145.79.227.30:4780 , piece_id=DZQ5LIBMWXPJ4XXWORRDLQIAFXJDNDQEEBRMSBS44UX3D2HPTSWQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold Dec 30 02:53:36.828762672 (587h24m36.699845707s) [noise=true , address=145.79.227.51:4429 , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qytmzwy7sTmWXskEbLWsWMniPz55zgho2akE7oh6ahAiT5SXf): protocol: write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection; write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection Dec 30 02:53:36.828774034 (587h24m36.699857069s) [noise=true , address=145.79.227.51:4429 , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold Dec 30 02:53:36.828889934 (587h24m36.699972969s) [noise=true , address=45.91.116.13:4176 , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ejsmBd4tAVwTHjamPYbryQb5x2vDnk9pBuX3CBF9WYFhTrFVJ): protocol: expected piece hash; context canceled; EOF Dec 30 02:53:36.828895144 (587h24m36.699978179s) [noise=true , address=45.91.116.13:4176 , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold Dec 30 02:53:36.828994173 (587h24m36.700077208s) [noise=true , address=145.79.227.49:4340 , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF Dec 30 02:53:36.828999793 (587h24m36.700082828s) [noise=true , address=145.79.227.49:4340 , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold Dec 30 02:53:36.829030332 (587h24m36.700113367s) [seg_pos={0 0}, upload=514937]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 02:53:36.829095486 (587h24m36.700178621s) [upload=514937]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 02:53:36.829100105 (587h24m36.70018315s) [upload=514937]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 02:53:37.128695270 (587h24m36.999778315s) [upload=514937]: Done uploading segments... Dec 30 02:53:37.128717683 (587h24m36.999800728s) [upload=514937]: Upload finished. err: <nil> Duration: 35.920895488s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [..X.............X................X......X...................] duration: 20.900817755s trace: https://jaeger.dev.storj.io/trace/6c3caa8adcfacbb9 FAIL (2025-12-30 03:09:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== true , address=145.79.227.35:4496 , piece_id=5X54ZGU4OIVI3DT622QE7IL66SVFLZYUICPODGMDKN7U4D4OJNSA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.040303259 (587h24m33.911386294s) [noise=true , address=77.90.31.12:4601 , piece_id=QQFK4PCL27YYF3UATGLV5KDENRNX6NHE7RXAZ2K66CC3WLHEBMWQ, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.040670117 (587h24m33.911753152s) [noise=true , address=77.90.31.21:4881 , piece_id=EMV4LHFK2VRHSW6PD3BBYGOXDAAYJKX6GEOXGGLOT3IXWNDQMU4A, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041006247 (587h24m33.912089292s) [noise=true , address=145.79.227.52:4472 , piece_id=DT2DAIC4ZI3WL7OGY55ZTTXJEI4IDRXPSJIQA3LTRC7K2F6EVPCA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041183314 (587h24m33.912266349s) [noise=true , address=145.79.227.26:4674 , piece_id=3BAN2ZXOFTSGZITYYHMZUS4AD35X6ZBOYA2HKOJCJI7VBWPJ3LSA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041313653 (587h24m33.912396688s) [noise=true , address=145.79.227.51:4429 , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041535194 (587h24m33.912618229s) [noise=true , address=145.79.227.33:4885 , piece_id=LA755L4RA4VJBHNCQBXOCFROZA3W6W6PUPKGJ5J772WAN44YMY2A, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041639693 (587h24m33.912722728s) [noise=true , address=145.79.227.43:4187 , piece_id=CAAFD3XIWWHZN7L7SYDMMSC5G5RKRLFYYHQZEEN4GAC2D6Y2EARA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041877216 (587h24m33.912960261s) [noise=true , address=145.79.227.46:4623 , piece_id=V4TEM7RRTTGBW5ISZOEAOPFIOZZ5EABO7BPWNRH5U5UPM2X7554Q, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041986634 (587h24m33.913069669s) [noise=true , address=45.91.116.13:4176 , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.042345657 (587h24m33.913428692s) [noise=true , address=45.91.116.10:4104 , piece_id=MRIWH6WGACE4UDNX55MIY6ZDWBAHS6E5KQS4QCOQN2BU7JI7ODCA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.042362019 (587h24m33.913445054s) [noise=true , address=77.90.31.11:4573 , piece_id=TORCC4W7DS5TJ7H7NWONEPNJOFULSZ6F5OHOP4BEVMGEH2MJA4DQ, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.042684192 (587h24m33.913767237s) [noise=true , address=145.79.227.49:4340 , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.042707327 (587h24m33.913790362s) [noise=true , address=145.79.227.32:4837 , piece_id=SQTHTF5KWZOKFYX6CGL6KCDRDAY7OJY6NA5OM3N4YCICDPM6DKXA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.043091528 (587h24m33.914174563s) [noise=true , address=45.91.116.5:4309 , piece_id=Y5HCMU6Z2FPBV3BNTUOC4F6JJWKNBKPOXWFVAFQTXUKQNHTLA4TA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.043544771 (587h24m33.914627816s) [noise=true , address=145.79.227.40:4053 , piece_id=3IZP7JBGSWBRBQHMJWXIDD73B5R3COWU5H3IYFGVBCTTLW7OUJ5A, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:35.347637489 (587h24m35.218720524s) [upload=514937]: Next returned nil segment Dec 30 02:53:35.347733591 (587h24m35.218816626s) [upload=514937]: Waiting for error group managing segments... Dec 30 02:53:35.674169656 (587h24m35.545252721s) [noise=true , address=64.49.13.27:4959 , piece_id=B4UZ67DIWNRHV77VTQONS6ZBUI2BPDDBM3ZS6FVCTABQZ6OHRJJA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:35.891295347 (587h24m35.762378382s) [noise=true , address=145.79.227.32:4837 , piece_id=SQTHTF5KWZOKFYX6CGL6KCDRDAY7OJY6NA5OM3N4YCICDPM6DKXA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:35.912619632 (587h24m35.783702677s) [noise=true , address=145.79.227.26:4674 , piece_id=3BAN2ZXOFTSGZITYYHMZUS4AD35X6ZBOYA2HKOJCJI7VBWPJ3LSA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:35.913054741 (587h24m35.784137786s) [noise=true , address=145.79.227.46:4623 , piece_id=V4TEM7RRTTGBW5ISZOEAOPFIOZZ5EABO7BPWNRH5U5UPM2X7554Q, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:35.913485270 (587h24m35.784568305s) [noise=true , address=145.79.227.35:4496 , piece_id=5X54ZGU4OIVI3DT622QE7IL66SVFLZYUICPODGMDKN7U4D4OJNSA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:35.923790864 (587h24m35.794873909s) [noise=true , address=145.79.227.52:4472 , piece_id=DT2DAIC4ZI3WL7OGY55ZTTXJEI4IDRXPSJIQA3LTRC7K2F6EVPCA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.109605534 (587h24m35.980688569s) [noise=true , address=77.90.31.21:4881 , piece_id=EMV4LHFK2VRHSW6PD3BBYGOXDAAYJKX6GEOXGGLOT3IXWNDQMU4A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.122502652 (587h24m35.993585697s) [noise=true , address=77.90.31.11:4573 , piece_id=TORCC4W7DS5TJ7H7NWONEPNJOFULSZ6F5OHOP4BEVMGEH2MJA4DQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.151131466 (587h24m36.022214511s) [noise=true , address=45.91.116.10:4104 , piece_id=MRIWH6WGACE4UDNX55MIY6ZDWBAHS6E5KQS4QCOQN2BU7JI7ODCA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.159562149 (587h24m36.030645184s) [noise=true , address=77.90.31.12:4601 , piece_id=QQFK4PCL27YYF3UATGLV5KDENRNX6NHE7RXAZ2K66CC3WLHEBMWQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.238339784 (587h24m36.10942282s) [noise=true , address=45.91.116.5:4309 , piece_id=Y5HCMU6Z2FPBV3BNTUOC4F6JJWKNBKPOXWFVAFQTXUKQNHTLA4TA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.813503456 (587h24m36.684586501s) [noise=true , address=145.79.227.33:4885 , piece_id=LA755L4RA4VJBHNCQBXOCFROZA3W6W6PUPKGJ5J772WAN44YMY2A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.826964278 (587h24m36.698047313s) [noise=true , address=145.79.227.43:4187 , piece_id=CAAFD3XIWWHZN7L7SYDMMSC5G5RKRLFYYHQZEEN4GAC2D6Y2EARA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.828562510 (587h24m36.699645545s) [noise=true , address=145.79.227.40:4053 , piece_id=3IZP7JBGSWBRBQHMJWXIDD73B5R3COWU5H3IYFGVBCTTLW7OUJ5A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.828575566 (587h24m36.699658601s) [seg_pos={0 0}, upload=514937]: Segment reached optimal threshold of 14 pieces. Dec 30 02:53:36.828752202 (587h24m36.699835237s) [noise=true , address=145.79.227.30:4780 , piece_id=DZQ5LIBMWXPJ4XXWORRDLQIAFXJDNDQEEBRMSBS44UX3D2HPTSWQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125WCbZ6a2NaewiLAHhHuPo91sQEfC9CZzEBV6rwF1JY7z13AEY): protocol: expected piece hash; context canceled; EOF Dec 30 02:53:36.828758714 (587h24m36.699841749s) [noise=true , address=145.79.227.30:4780 , piece_id=DZQ5LIBMWXPJ4XXWORRDLQIAFXJDNDQEEBRMSBS44UX3D2HPTSWQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold Dec 30 02:53:36.828762672 (587h24m36.699845707s) [noise=true , address=145.79.227.51:4429 , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qytmzwy7sTmWXskEbLWsWMniPz55zgho2akE7oh6ahAiT5SXf): protocol: write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection; write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection Dec 30 02:53:36.828774034 (587h24m36.699857069s) [noise=true , address=145.79.227.51:4429 , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold Dec 30 02:53:36.828889934 (587h24m36.699972969s) [noise=true , address=45.91.116.13:4176 , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ejsmBd4tAVwTHjamPYbryQb5x2vDnk9pBuX3CBF9WYFhTrFVJ): protocol: expected piece hash; context canceled; EOF Dec 30 02:53:36.828895144 (587h24m36.699978179s) [noise=true , address=45.91.116.13:4176 , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold Dec 30 02:53:36.828994173 (587h24m36.700077208s) [noise=true , address=145.79.227.49:4340 , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF Dec 30 02:53:36.828999793 (587h24m36.700082828s) [noise=true , address=145.79.227.49:4340 , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold Dec 30 02:53:36.829030332 (587h24m36.700113367s) [seg_pos={0 0}, upload=514937]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 02:53:36.829095486 (587h24m36.700178621s) [upload=514937]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 02:53:36.829100105 (587h24m36.70018315s) [upload=514937]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 02:53:37.128695270 (587h24m36.999778315s) [upload=514937]: Done uploading segments... Dec 30 02:53:37.128717683 (587h24m36.999800728s) [upload=514937]: Upload finished. err: <nil> Duration: 20.900836471s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [...............................X...X.X.XX................XX.] duration: 4.884454743s trace: https://jaeger.dev.storj.io/trace/55b84c463909ac27 FAIL (2025-12-30 03:08:56 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.884473831s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.X........X...X..XX..X.X......XXX.X.X..X........X.X.XX..X.X.] duration: 6.875696063s trace: https://jaeger.dev.storj.io/trace/71acf892875ff4a9 FAIL (2025-12-30 03:08:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: 72P4Z6YNWVPU2YCCQNZKLMSJIVPPZYOOD6PMBRPZGKMNQSLXVUHQ) context deadline exceeded; piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: IS3RKIM6GAKIO47S7PKILBTCKWVUV2O6CEYFLUMDA7DXQ7HRADMA) context deadline exceeded; piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: SBE2PATH3SBS6TEWYLR6V7KYTP3OC5PZMF5BFMLXV7GW2VWGPXKQ) context deadline exceeded ======LOGS====== . Dec 30 03:07:45.235256899 (587h38m45.106339954s) [upload=515144]: Got next segment. Inline: false Dec 30 03:07:45.235306404 (587h38m45.106389459s) [upload=515144]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 03:07:45.235313888 (587h38m45.106396943s) [upload=515144]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 03:07:45.480535530 (587h38m45.351618565s) [seg_pos={0 0}, upload=515144]: Begin upload segment... Dec 30 03:07:45.480591326 (587h38m45.351674372s) [seg_pos={0 0}, upload=515144]: Done begin upload segment. Dec 30 03:07:45.480627806 (587h38m45.351710851s) [noise=true , address=145.79.227.49:4338 , piece_id=4564TSOKRG7LFRKOPLSM42PSWNOYXHNQ6RVPHFFTXQB322I3XU5Q, seg_pos={0 0}, upload=515144]: Uploading piece... Dec 30 03:07:45.480657172 (587h38m45.351740207s) [noise=true , address=45.91.116.13:4156 , piece_id=SBE2PATH3SBS6TEWYLR6V7KYTP3OC5PZMF5BFMLXV7GW2VWGPXKQ, seg_pos={0 0}, upload=515144]: Uploading piece... Dec 30 03:07:45.481129341 (587h38m45.352212376s) [noise=true , address=145.79.227.39:4015 , piece_id=ECWADQ244MB4JQRKF66MVUOOES7KOHDPKFLWU73TZYIZMOMH3CTQ, seg_pos={0 0}, upload=515144]: Uploading piece... Dec 30 03:07:45.481295116 (587h38m45.352378151s) [noise=true , address=45.91.116.10:4081 , piece_id=UUBTUUMX7FNNCZLNAB3YIE66QTMZRNP5UDH2ARCGNRIZ5SERRSGQ, seg_pos={0 0}, upload=515144]: Uploading piece... Dec 30 03:07:45.481624623 (587h38m45.352707658s) [noise=true , address=45.91.116.6:4974 , piece_id=72P4Z6YNWVPU2YCCQNZKLMSJIVPPZYOOD6PMBRPZGKMNQSLXVUHQ, seg_pos={0 0}, upload=515144]: Uploading piece... Dec 30 03:07:45.482002944 (587h38m45.353085979s) [noise=true , address=77.90.31.11:4553 , piece_id=Y36XLKTM7RHY7F5K2OABEPFLAZPSN5DVEA5BL727WTIHXJPHYO3Q, seg_pos={0 0}, upload=515144]: Uploading piece... Dec 30 03:07:45.482239715 (587h38m45.35332277s) [noise=true , address=145.79.227.27:4689 , piece_id=KYSAEHNB5FAW5RSES2MN64OEAWWBYW7FZ32AH632XIVYUA22DXJQ, seg_pos={0 0}, upload=515144]: Uploading piece... Dec 30 03:07:45.482457029 (587h38m45.353540064s) [noise=true , address=145.79.227.33:4869 , piece_id=AJIEURE66DN7W477R3RO5JTAXHKTCZM5PMRHLGKUYB6GCPKGKAAA, seg_pos={0 0}, upload=515144]: Uploading piece... Dec 30 03:07:45.482638824 (587h38m45.35372187s) [noise=true , address=145.79.227.47:4267 , piece_id=Q2KKULZTUHB6IFSUTWYXFI7BLHX5ATX6IUPAX5QQ3AQENJS7ZRIQ, seg_pos={0 0}, upload=515144]: Uploading piece... Dec 30 03:07:45.482814349 (587h38m45.353897384s) [noise=true , address=145.79.227.25:4619 , piece_id=MUYFZ6EKVWQXEAYRFZZXW6DGT553UP2DHKIXQ4XB5NF4OOUFHELA, seg_pos={0 0}, upload=515144]: Uploading piece... Dec 30 03:07:45.483050268 (587h38m45.354133303s) [noise=true , address=145.79.227.50:4374 , piece_id=JT472PTMK3SMTXO3Z7FMPT52OEMONNKIL2R4AAD4RHRMBL54VBVQ, seg_pos={0 0}, upload=515144]: Uploading piece... Dec 30 03:07:45.483158143 (587h38m45.354241188s) [noise=true , address=145.79.227.37:4567 , piece_id=4TXCSROSF6OUJX6GRGWGZEG7LWXUZRWRKFF7TPNELSVKXTF2UYIQ, seg_pos={0 0}, upload=515144]: Uploading piece... Dec 30 03:07:45.483476299 (587h38m45.354559334s) [noise=true , address=77.90.31.21:4878 , piece_id=IS3RKIM6GAKIO47S7PKILBTCKWVUV2O6CEYFLUMDA7DXQ7HRADMA, seg_pos={0 0}, upload=515144]: Uploading piece... Dec 30 03:07:45.483491779 (587h38m45.354574814s) [noise=true , address=145.79.227.15:4873 , piece_id=J3HPX6M7265YDTXWDK5LHOAL7YC3AIFT7EZ7DGOZS5FKCE4EYOQQ, seg_pos={0 0}, upload=515144]: Uploading piece... Dec 30 03:07:45.483885088 (587h38m45.354968123s) [noise=true , address=145.79.227.35:4483 , piece_id=N6PJVBURZPPAEC4RKMEAF7RHJH63PBLXZF5DHJH576I2A3SHBDOQ, seg_pos={0 0}, upload=515144]: Uploading piece... Dec 30 03:07:45.484316158 (587h38m45.355399193s) [noise=true , address=145.79.227.52:4446 , piece_id=OR55ADZXLCP6T2X4ZFN3T3ENF2X3AFR5ULIE2UM7MFHXZZ3HISOA, seg_pos={0 0}, upload=515144]: Uploading piece... Dec 30 03:07:47.136488717 (587h38m47.007571763s) [upload=515144]: Next returned nil segment Dec 30 03:07:47.136502834 (587h38m47.007585869s) [upload=515144]: Waiting for error group managing segments... Dec 30 03:07:47.467278385 (587h38m47.33836143s) [noise=true , address=145.79.227.15:4873 , piece_id=J3HPX6M7265YDTXWDK5LHOAL7YC3AIFT7EZ7DGOZS5FKCE4EYOQQ, seg_pos={0 0}, upload=515144]: Done uploading piece. err: <nil> Dec 30 03:07:47.467592613 (587h38m47.338675648s) [noise=true , address=145.79.227.52:4446 , piece_id=OR55ADZXLCP6T2X4ZFN3T3ENF2X3AFR5ULIE2UM7MFHXZZ3HISOA, seg_pos={0 0}, upload=515144]: Done uploading piece. err: <nil> Dec 30 03:07:47.471011272 (587h38m47.342094307s) [noise=true , address=145.79.227.25:4619 , piece_id=MUYFZ6EKVWQXEAYRFZZXW6DGT553UP2DHKIXQ4XB5NF4OOUFHELA, seg_pos={0 0}, upload=515144]: Done uploading piece. err: <nil> Dec 30 03:07:47.474913623 (587h38m47.345996658s) [noise=true , address=145.79.227.47:4267 , piece_id=Q2KKULZTUHB6IFSUTWYXFI7BLHX5ATX6IUPAX5QQ3AQENJS7ZRIQ, seg_pos={0 0}, upload=515144]: Done uploading piece. err: <nil> Dec 30 03:07:47.486578916 (587h38m47.357661961s) [noise=true , address=145.79.227.37:4567 , piece_id=4TXCSROSF6OUJX6GRGWGZEG7LWXUZRWRKFF7TPNELSVKXTF2UYIQ, seg_pos={0 0}, upload=515144]: Done uploading piece. err: <nil> Dec 30 03:07:47.492201893 (587h38m47.363284928s) [noise=true , address=145.79.227.27:4689 , piece_id=KYSAEHNB5FAW5RSES2MN64OEAWWBYW7FZ32AH632XIVYUA22DXJQ, seg_pos={0 0}, upload=515144]: Done uploading piece. err: <nil> Dec 30 03:07:47.767877234 (587h38m47.638960269s) [noise=true , address=45.91.116.13:4156 , piece_id=SBE2PATH3SBS6TEWYLR6V7KYTP3OC5PZMF5BFMLXV7GW2VWGPXKQ, seg_pos={0 0}, upload=515144]: Done uploading piece. err: <nil> Dec 30 03:07:47.769785027 (587h38m47.640868062s) [noise=true , address=45.91.116.6:4974 , piece_id=72P4Z6YNWVPU2YCCQNZKLMSJIVPPZYOOD6PMBRPZGKMNQSLXVUHQ, seg_pos={0 0}, upload=515144]: Done uploading piece. err: <nil> Dec 30 03:07:47.785540959 (587h38m47.656623994s) [noise=true , address=145.79.227.49:4338 , piece_id=4564TSOKRG7LFRKOPLSM42PSWNOYXHNQ6RVPHFFTXQB322I3XU5Q, seg_pos={0 0}, upload=515144]: Done uploading piece. err: <nil> Dec 30 03:07:47.787560675 (587h38m47.65864371s) [noise=true , address=77.90.31.21:4878 , piece_id=IS3RKIM6GAKIO47S7PKILBTCKWVUV2O6CEYFLUMDA7DXQ7HRADMA, seg_pos={0 0}, upload=515144]: Done uploading piece. err: <nil> Dec 30 03:07:48.418936950 (587h38m48.290019995s) [noise=true , address=45.91.116.10:4081 , piece_id=UUBTUUMX7FNNCZLNAB3YIE66QTMZRNP5UDH2ARCGNRIZ5SERRSGQ, seg_pos={0 0}, upload=515144]: Done uploading piece. err: <nil> Dec 30 03:07:48.696576180 (587h38m48.567659235s) [noise=true , address=145.79.227.33:4869 , piece_id=AJIEURE66DN7W477R3RO5JTAXHKTCZM5PMRHLGKUYB6GCPKGKAAA, seg_pos={0 0}, upload=515144]: Done uploading piece. err: <nil> Dec 30 03:07:48.857650632 (587h38m48.728733668s) [noise=true , address=145.79.227.35:4483 , piece_id=N6PJVBURZPPAEC4RKMEAF7RHJH63PBLXZF5DHJH576I2A3SHBDOQ, seg_pos={0 0}, upload=515144]: Done uploading piece. err: <nil> Dec 30 03:07:48.857676502 (587h38m48.728759537s) [seg_pos={0 0}, upload=515144]: Segment reached optimal threshold of 13 pieces. Dec 30 03:07:48.857879940 (587h38m48.728962975s) [noise=true , address=145.79.227.50:4374 , piece_id=JT472PTMK3SMTXO3Z7FMPT52OEMONNKIL2R4AAD4RHRMBL54VBVQ, seg_pos={0 0}, upload=515144]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K): protocol: write tcp 188.166.225.170:44734->145.79.227.50:4374: use of closed network connection; write tcp 188.166.225.170:44734->145.79.227.50:4374: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44734->145.79.227.50:4374: use of closed network connection Dec 30 03:07:48.857886913 (587h38m48.728969948s) [noise=true , address=145.79.227.50:4374 , piece_id=JT472PTMK3SMTXO3Z7FMPT52OEMONNKIL2R4AAD4RHRMBL54VBVQ, seg_pos={0 0}, upload=515144]: Long tail cancellation due to optimal threshold Dec 30 03:07:48.858018012 (587h38m48.729101048s) [noise=true , address=77.90.31.11:4553 , piece_id=Y36XLKTM7RHY7F5K2OABEPFLAZPSN5DVEA5BL727WTIHXJPHYO3Q, seg_pos={0 0}, upload=515144]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 30 03:07:48.858057467 (587h38m48.729140503s) [noise=true , address=77.90.31.11:4553 , piece_id=Y36XLKTM7RHY7F5K2OABEPFLAZPSN5DVEA5BL727WTIHXJPHYO3Q, seg_pos={0 0}, upload=515144]: Long tail cancellation due to optimal threshold Dec 30 03:07:48.858190551 (587h38m48.729273586s) [noise=true , address=145.79.227.39:4015 , piece_id=ECWADQ244MB4JQRKF66MVUOOES7KOHDPKFLWU73TZYIZMOMH3CTQ, seg_pos={0 0}, upload=515144]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE): protocol: write tcp 188.166.225.170:37124->145.79.227.39:4015: use of closed network connection; write tcp 188.166.225.170:37124->145.79.227.39:4015: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37124->145.79.227.39:4015: use of closed network connection Dec 30 03:07:48.858198125 (587h38m48.72928116s) [noise=true , address=145.79.227.39:4015 , piece_id=ECWADQ244MB4JQRKF66MVUOOES7KOHDPKFLWU73TZYIZMOMH3CTQ, seg_pos={0 0}, upload=515144]: Long tail cancellation due to optimal threshold Dec 30 03:07:48.858266636 (587h38m48.729349671s) [seg_pos={0 0}, upload=515144]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 03:07:48.858400080 (587h38m48.729483125s) [upload=515144]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 03:07:48.858405590 (587h38m48.729488635s) [upload=515144]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 03:07:49.141334163 (587h38m49.012417198s) [upload=515144]: Done uploading segments... Dec 30 03:07:49.141353911 (587h38m49.012436956s) [upload=515144]: Upload finished. err: <nil> Duration: 6.875730649s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [...X.............X................X......X..................] duration: 40.293383087s trace: https://jaeger.dev.storj.io/trace/6c3caa8adcfacbb9 FAIL (2025-12-30 03:08:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== true , address=145.79.227.35:4496 , piece_id=5X54ZGU4OIVI3DT622QE7IL66SVFLZYUICPODGMDKN7U4D4OJNSA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.040303259 (587h24m33.911386294s) [noise=true , address=77.90.31.12:4601 , piece_id=QQFK4PCL27YYF3UATGLV5KDENRNX6NHE7RXAZ2K66CC3WLHEBMWQ, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.040670117 (587h24m33.911753152s) [noise=true , address=77.90.31.21:4881 , piece_id=EMV4LHFK2VRHSW6PD3BBYGOXDAAYJKX6GEOXGGLOT3IXWNDQMU4A, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041006247 (587h24m33.912089292s) [noise=true , address=145.79.227.52:4472 , piece_id=DT2DAIC4ZI3WL7OGY55ZTTXJEI4IDRXPSJIQA3LTRC7K2F6EVPCA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041183314 (587h24m33.912266349s) [noise=true , address=145.79.227.26:4674 , piece_id=3BAN2ZXOFTSGZITYYHMZUS4AD35X6ZBOYA2HKOJCJI7VBWPJ3LSA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041313653 (587h24m33.912396688s) [noise=true , address=145.79.227.51:4429 , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041535194 (587h24m33.912618229s) [noise=true , address=145.79.227.33:4885 , piece_id=LA755L4RA4VJBHNCQBXOCFROZA3W6W6PUPKGJ5J772WAN44YMY2A, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041639693 (587h24m33.912722728s) [noise=true , address=145.79.227.43:4187 , piece_id=CAAFD3XIWWHZN7L7SYDMMSC5G5RKRLFYYHQZEEN4GAC2D6Y2EARA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041877216 (587h24m33.912960261s) [noise=true , address=145.79.227.46:4623 , piece_id=V4TEM7RRTTGBW5ISZOEAOPFIOZZ5EABO7BPWNRH5U5UPM2X7554Q, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041986634 (587h24m33.913069669s) [noise=true , address=45.91.116.13:4176 , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.042345657 (587h24m33.913428692s) [noise=true , address=45.91.116.10:4104 , piece_id=MRIWH6WGACE4UDNX55MIY6ZDWBAHS6E5KQS4QCOQN2BU7JI7ODCA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.042362019 (587h24m33.913445054s) [noise=true , address=77.90.31.11:4573 , piece_id=TORCC4W7DS5TJ7H7NWONEPNJOFULSZ6F5OHOP4BEVMGEH2MJA4DQ, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.042684192 (587h24m33.913767237s) [noise=true , address=145.79.227.49:4340 , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.042707327 (587h24m33.913790362s) [noise=true , address=145.79.227.32:4837 , piece_id=SQTHTF5KWZOKFYX6CGL6KCDRDAY7OJY6NA5OM3N4YCICDPM6DKXA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.043091528 (587h24m33.914174563s) [noise=true , address=45.91.116.5:4309 , piece_id=Y5HCMU6Z2FPBV3BNTUOC4F6JJWKNBKPOXWFVAFQTXUKQNHTLA4TA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.043544771 (587h24m33.914627816s) [noise=true , address=145.79.227.40:4053 , piece_id=3IZP7JBGSWBRBQHMJWXIDD73B5R3COWU5H3IYFGVBCTTLW7OUJ5A, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:35.347637489 (587h24m35.218720524s) [upload=514937]: Next returned nil segment Dec 30 02:53:35.347733591 (587h24m35.218816626s) [upload=514937]: Waiting for error group managing segments... Dec 30 02:53:35.674169656 (587h24m35.545252721s) [noise=true , address=64.49.13.27:4959 , piece_id=B4UZ67DIWNRHV77VTQONS6ZBUI2BPDDBM3ZS6FVCTABQZ6OHRJJA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:35.891295347 (587h24m35.762378382s) [noise=true , address=145.79.227.32:4837 , piece_id=SQTHTF5KWZOKFYX6CGL6KCDRDAY7OJY6NA5OM3N4YCICDPM6DKXA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:35.912619632 (587h24m35.783702677s) [noise=true , address=145.79.227.26:4674 , piece_id=3BAN2ZXOFTSGZITYYHMZUS4AD35X6ZBOYA2HKOJCJI7VBWPJ3LSA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:35.913054741 (587h24m35.784137786s) [noise=true , address=145.79.227.46:4623 , piece_id=V4TEM7RRTTGBW5ISZOEAOPFIOZZ5EABO7BPWNRH5U5UPM2X7554Q, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:35.913485270 (587h24m35.784568305s) [noise=true , address=145.79.227.35:4496 , piece_id=5X54ZGU4OIVI3DT622QE7IL66SVFLZYUICPODGMDKN7U4D4OJNSA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:35.923790864 (587h24m35.794873909s) [noise=true , address=145.79.227.52:4472 , piece_id=DT2DAIC4ZI3WL7OGY55ZTTXJEI4IDRXPSJIQA3LTRC7K2F6EVPCA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.109605534 (587h24m35.980688569s) [noise=true , address=77.90.31.21:4881 , piece_id=EMV4LHFK2VRHSW6PD3BBYGOXDAAYJKX6GEOXGGLOT3IXWNDQMU4A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.122502652 (587h24m35.993585697s) [noise=true , address=77.90.31.11:4573 , piece_id=TORCC4W7DS5TJ7H7NWONEPNJOFULSZ6F5OHOP4BEVMGEH2MJA4DQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.151131466 (587h24m36.022214511s) [noise=true , address=45.91.116.10:4104 , piece_id=MRIWH6WGACE4UDNX55MIY6ZDWBAHS6E5KQS4QCOQN2BU7JI7ODCA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.159562149 (587h24m36.030645184s) [noise=true , address=77.90.31.12:4601 , piece_id=QQFK4PCL27YYF3UATGLV5KDENRNX6NHE7RXAZ2K66CC3WLHEBMWQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.238339784 (587h24m36.10942282s) [noise=true , address=45.91.116.5:4309 , piece_id=Y5HCMU6Z2FPBV3BNTUOC4F6JJWKNBKPOXWFVAFQTXUKQNHTLA4TA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.813503456 (587h24m36.684586501s) [noise=true , address=145.79.227.33:4885 , piece_id=LA755L4RA4VJBHNCQBXOCFROZA3W6W6PUPKGJ5J772WAN44YMY2A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.826964278 (587h24m36.698047313s) [noise=true , address=145.79.227.43:4187 , piece_id=CAAFD3XIWWHZN7L7SYDMMSC5G5RKRLFYYHQZEEN4GAC2D6Y2EARA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.828562510 (587h24m36.699645545s) [noise=true , address=145.79.227.40:4053 , piece_id=3IZP7JBGSWBRBQHMJWXIDD73B5R3COWU5H3IYFGVBCTTLW7OUJ5A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.828575566 (587h24m36.699658601s) [seg_pos={0 0}, upload=514937]: Segment reached optimal threshold of 14 pieces. Dec 30 02:53:36.828752202 (587h24m36.699835237s) [noise=true , address=145.79.227.30:4780 , piece_id=DZQ5LIBMWXPJ4XXWORRDLQIAFXJDNDQEEBRMSBS44UX3D2HPTSWQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125WCbZ6a2NaewiLAHhHuPo91sQEfC9CZzEBV6rwF1JY7z13AEY): protocol: expected piece hash; context canceled; EOF Dec 30 02:53:36.828758714 (587h24m36.699841749s) [noise=true , address=145.79.227.30:4780 , piece_id=DZQ5LIBMWXPJ4XXWORRDLQIAFXJDNDQEEBRMSBS44UX3D2HPTSWQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold Dec 30 02:53:36.828762672 (587h24m36.699845707s) [noise=true , address=145.79.227.51:4429 , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qytmzwy7sTmWXskEbLWsWMniPz55zgho2akE7oh6ahAiT5SXf): protocol: write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection; write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection Dec 30 02:53:36.828774034 (587h24m36.699857069s) [noise=true , address=145.79.227.51:4429 , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold Dec 30 02:53:36.828889934 (587h24m36.699972969s) [noise=true , address=45.91.116.13:4176 , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ejsmBd4tAVwTHjamPYbryQb5x2vDnk9pBuX3CBF9WYFhTrFVJ): protocol: expected piece hash; context canceled; EOF Dec 30 02:53:36.828895144 (587h24m36.699978179s) [noise=true , address=45.91.116.13:4176 , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold Dec 30 02:53:36.828994173 (587h24m36.700077208s) [noise=true , address=145.79.227.49:4340 , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF Dec 30 02:53:36.828999793 (587h24m36.700082828s) [noise=true , address=145.79.227.49:4340 , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold Dec 30 02:53:36.829030332 (587h24m36.700113367s) [seg_pos={0 0}, upload=514937]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 02:53:36.829095486 (587h24m36.700178621s) [upload=514937]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 02:53:36.829100105 (587h24m36.70018315s) [upload=514937]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 02:53:37.128695270 (587h24m36.999778315s) [upload=514937]: Done uploading segments... Dec 30 02:53:37.128717683 (587h24m36.999800728s) [upload=514937]: Upload finished. err: <nil> Duration: 40.293411633s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [..X........X...X..XX..X.X......XXX.X.X..X........X.X.XX..X.X] duration: 30.003766461s trace: https://jaeger.dev.storj.io/trace/71acf892875ff4a9 FAIL (2025-12-30 03:08:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: 72P4Z6YNWVPU2YCCQNZKLMSJIVPPZYOOD6PMBRPZGKMNQSLXVUHQ) context deadline exceeded; piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: IS3RKIM6GAKIO47S7PKILBTCKWVUV2O6CEYFLUMDA7DXQ7HRADMA) context deadline exceeded; piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: SBE2PATH3SBS6TEWYLR6V7KYTP3OC5PZMF5BFMLXV7GW2VWGPXKQ) context deadline exceeded ======LOGS====== . Dec 30 03:07:45.235256899 (587h38m45.106339954s) [upload=515144]: Got next segment. Inline: false Dec 30 03:07:45.235306404 (587h38m45.106389459s) [upload=515144]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 03:07:45.235313888 (587h38m45.106396943s) [upload=515144]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 03:07:45.480535530 (587h38m45.351618565s) [seg_pos={0 0}, upload=515144]: Begin upload segment... Dec 30 03:07:45.480591326 (587h38m45.351674372s) [seg_pos={0 0}, upload=515144]: Done begin upload segment. Dec 30 03:07:45.480627806 (587h38m45.351710851s) [noise=true , address=145.79.227.49:4338 , piece_id=4564TSOKRG7LFRKOPLSM42PSWNOYXHNQ6RVPHFFTXQB322I3XU5Q, seg_pos={0 0}, upload=515144]: Uploading piece... Dec 30 03:07:45.480657172 (587h38m45.351740207s) [noise=true , address=45.91.116.13:4156 , piece_id=SBE2PATH3SBS6TEWYLR6V7KYTP3OC5PZMF5BFMLXV7GW2VWGPXKQ, seg_pos={0 0}, upload=515144]: Uploading piece... Dec 30 03:07:45.481129341 (587h38m45.352212376s) [noise=true , address=145.79.227.39:4015 , piece_id=ECWADQ244MB4JQRKF66MVUOOES7KOHDPKFLWU73TZYIZMOMH3CTQ, seg_pos={0 0}, upload=515144]: Uploading piece... Dec 30 03:07:45.481295116 (587h38m45.352378151s) [noise=true , address=45.91.116.10:4081 , piece_id=UUBTUUMX7FNNCZLNAB3YIE66QTMZRNP5UDH2ARCGNRIZ5SERRSGQ, seg_pos={0 0}, upload=515144]: Uploading piece... Dec 30 03:07:45.481624623 (587h38m45.352707658s) [noise=true , address=45.91.116.6:4974 , piece_id=72P4Z6YNWVPU2YCCQNZKLMSJIVPPZYOOD6PMBRPZGKMNQSLXVUHQ, seg_pos={0 0}, upload=515144]: Uploading piece... Dec 30 03:07:45.482002944 (587h38m45.353085979s) [noise=true , address=77.90.31.11:4553 , piece_id=Y36XLKTM7RHY7F5K2OABEPFLAZPSN5DVEA5BL727WTIHXJPHYO3Q, seg_pos={0 0}, upload=515144]: Uploading piece... Dec 30 03:07:45.482239715 (587h38m45.35332277s) [noise=true , address=145.79.227.27:4689 , piece_id=KYSAEHNB5FAW5RSES2MN64OEAWWBYW7FZ32AH632XIVYUA22DXJQ, seg_pos={0 0}, upload=515144]: Uploading piece... Dec 30 03:07:45.482457029 (587h38m45.353540064s) [noise=true , address=145.79.227.33:4869 , piece_id=AJIEURE66DN7W477R3RO5JTAXHKTCZM5PMRHLGKUYB6GCPKGKAAA, seg_pos={0 0}, upload=515144]: Uploading piece... Dec 30 03:07:45.482638824 (587h38m45.35372187s) [noise=true , address=145.79.227.47:4267 , piece_id=Q2KKULZTUHB6IFSUTWYXFI7BLHX5ATX6IUPAX5QQ3AQENJS7ZRIQ, seg_pos={0 0}, upload=515144]: Uploading piece... Dec 30 03:07:45.482814349 (587h38m45.353897384s) [noise=true , address=145.79.227.25:4619 , piece_id=MUYFZ6EKVWQXEAYRFZZXW6DGT553UP2DHKIXQ4XB5NF4OOUFHELA, seg_pos={0 0}, upload=515144]: Uploading piece... Dec 30 03:07:45.483050268 (587h38m45.354133303s) [noise=true , address=145.79.227.50:4374 , piece_id=JT472PTMK3SMTXO3Z7FMPT52OEMONNKIL2R4AAD4RHRMBL54VBVQ, seg_pos={0 0}, upload=515144]: Uploading piece... Dec 30 03:07:45.483158143 (587h38m45.354241188s) [noise=true , address=145.79.227.37:4567 , piece_id=4TXCSROSF6OUJX6GRGWGZEG7LWXUZRWRKFF7TPNELSVKXTF2UYIQ, seg_pos={0 0}, upload=515144]: Uploading piece... Dec 30 03:07:45.483476299 (587h38m45.354559334s) [noise=true , address=77.90.31.21:4878 , piece_id=IS3RKIM6GAKIO47S7PKILBTCKWVUV2O6CEYFLUMDA7DXQ7HRADMA, seg_pos={0 0}, upload=515144]: Uploading piece... Dec 30 03:07:45.483491779 (587h38m45.354574814s) [noise=true , address=145.79.227.15:4873 , piece_id=J3HPX6M7265YDTXWDK5LHOAL7YC3AIFT7EZ7DGOZS5FKCE4EYOQQ, seg_pos={0 0}, upload=515144]: Uploading piece... Dec 30 03:07:45.483885088 (587h38m45.354968123s) [noise=true , address=145.79.227.35:4483 , piece_id=N6PJVBURZPPAEC4RKMEAF7RHJH63PBLXZF5DHJH576I2A3SHBDOQ, seg_pos={0 0}, upload=515144]: Uploading piece... Dec 30 03:07:45.484316158 (587h38m45.355399193s) [noise=true , address=145.79.227.52:4446 , piece_id=OR55ADZXLCP6T2X4ZFN3T3ENF2X3AFR5ULIE2UM7MFHXZZ3HISOA, seg_pos={0 0}, upload=515144]: Uploading piece... Dec 30 03:07:47.136488717 (587h38m47.007571763s) [upload=515144]: Next returned nil segment Dec 30 03:07:47.136502834 (587h38m47.007585869s) [upload=515144]: Waiting for error group managing segments... Dec 30 03:07:47.467278385 (587h38m47.33836143s) [noise=true , address=145.79.227.15:4873 , piece_id=J3HPX6M7265YDTXWDK5LHOAL7YC3AIFT7EZ7DGOZS5FKCE4EYOQQ, seg_pos={0 0}, upload=515144]: Done uploading piece. err: <nil> Dec 30 03:07:47.467592613 (587h38m47.338675648s) [noise=true , address=145.79.227.52:4446 , piece_id=OR55ADZXLCP6T2X4ZFN3T3ENF2X3AFR5ULIE2UM7MFHXZZ3HISOA, seg_pos={0 0}, upload=515144]: Done uploading piece. err: <nil> Dec 30 03:07:47.471011272 (587h38m47.342094307s) [noise=true , address=145.79.227.25:4619 , piece_id=MUYFZ6EKVWQXEAYRFZZXW6DGT553UP2DHKIXQ4XB5NF4OOUFHELA, seg_pos={0 0}, upload=515144]: Done uploading piece. err: <nil> Dec 30 03:07:47.474913623 (587h38m47.345996658s) [noise=true , address=145.79.227.47:4267 , piece_id=Q2KKULZTUHB6IFSUTWYXFI7BLHX5ATX6IUPAX5QQ3AQENJS7ZRIQ, seg_pos={0 0}, upload=515144]: Done uploading piece. err: <nil> Dec 30 03:07:47.486578916 (587h38m47.357661961s) [noise=true , address=145.79.227.37:4567 , piece_id=4TXCSROSF6OUJX6GRGWGZEG7LWXUZRWRKFF7TPNELSVKXTF2UYIQ, seg_pos={0 0}, upload=515144]: Done uploading piece. err: <nil> Dec 30 03:07:47.492201893 (587h38m47.363284928s) [noise=true , address=145.79.227.27:4689 , piece_id=KYSAEHNB5FAW5RSES2MN64OEAWWBYW7FZ32AH632XIVYUA22DXJQ, seg_pos={0 0}, upload=515144]: Done uploading piece. err: <nil> Dec 30 03:07:47.767877234 (587h38m47.638960269s) [noise=true , address=45.91.116.13:4156 , piece_id=SBE2PATH3SBS6TEWYLR6V7KYTP3OC5PZMF5BFMLXV7GW2VWGPXKQ, seg_pos={0 0}, upload=515144]: Done uploading piece. err: <nil> Dec 30 03:07:47.769785027 (587h38m47.640868062s) [noise=true , address=45.91.116.6:4974 , piece_id=72P4Z6YNWVPU2YCCQNZKLMSJIVPPZYOOD6PMBRPZGKMNQSLXVUHQ, seg_pos={0 0}, upload=515144]: Done uploading piece. err: <nil> Dec 30 03:07:47.785540959 (587h38m47.656623994s) [noise=true , address=145.79.227.49:4338 , piece_id=4564TSOKRG7LFRKOPLSM42PSWNOYXHNQ6RVPHFFTXQB322I3XU5Q, seg_pos={0 0}, upload=515144]: Done uploading piece. err: <nil> Dec 30 03:07:47.787560675 (587h38m47.65864371s) [noise=true , address=77.90.31.21:4878 , piece_id=IS3RKIM6GAKIO47S7PKILBTCKWVUV2O6CEYFLUMDA7DXQ7HRADMA, seg_pos={0 0}, upload=515144]: Done uploading piece. err: <nil> Dec 30 03:07:48.418936950 (587h38m48.290019995s) [noise=true , address=45.91.116.10:4081 , piece_id=UUBTUUMX7FNNCZLNAB3YIE66QTMZRNP5UDH2ARCGNRIZ5SERRSGQ, seg_pos={0 0}, upload=515144]: Done uploading piece. err: <nil> Dec 30 03:07:48.696576180 (587h38m48.567659235s) [noise=true , address=145.79.227.33:4869 , piece_id=AJIEURE66DN7W477R3RO5JTAXHKTCZM5PMRHLGKUYB6GCPKGKAAA, seg_pos={0 0}, upload=515144]: Done uploading piece. err: <nil> Dec 30 03:07:48.857650632 (587h38m48.728733668s) [noise=true , address=145.79.227.35:4483 , piece_id=N6PJVBURZPPAEC4RKMEAF7RHJH63PBLXZF5DHJH576I2A3SHBDOQ, seg_pos={0 0}, upload=515144]: Done uploading piece. err: <nil> Dec 30 03:07:48.857676502 (587h38m48.728759537s) [seg_pos={0 0}, upload=515144]: Segment reached optimal threshold of 13 pieces. Dec 30 03:07:48.857879940 (587h38m48.728962975s) [noise=true , address=145.79.227.50:4374 , piece_id=JT472PTMK3SMTXO3Z7FMPT52OEMONNKIL2R4AAD4RHRMBL54VBVQ, seg_pos={0 0}, upload=515144]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K): protocol: write tcp 188.166.225.170:44734->145.79.227.50:4374: use of closed network connection; write tcp 188.166.225.170:44734->145.79.227.50:4374: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44734->145.79.227.50:4374: use of closed network connection Dec 30 03:07:48.857886913 (587h38m48.728969948s) [noise=true , address=145.79.227.50:4374 , piece_id=JT472PTMK3SMTXO3Z7FMPT52OEMONNKIL2R4AAD4RHRMBL54VBVQ, seg_pos={0 0}, upload=515144]: Long tail cancellation due to optimal threshold Dec 30 03:07:48.858018012 (587h38m48.729101048s) [noise=true , address=77.90.31.11:4553 , piece_id=Y36XLKTM7RHY7F5K2OABEPFLAZPSN5DVEA5BL727WTIHXJPHYO3Q, seg_pos={0 0}, upload=515144]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 30 03:07:48.858057467 (587h38m48.729140503s) [noise=true , address=77.90.31.11:4553 , piece_id=Y36XLKTM7RHY7F5K2OABEPFLAZPSN5DVEA5BL727WTIHXJPHYO3Q, seg_pos={0 0}, upload=515144]: Long tail cancellation due to optimal threshold Dec 30 03:07:48.858190551 (587h38m48.729273586s) [noise=true , address=145.79.227.39:4015 , piece_id=ECWADQ244MB4JQRKF66MVUOOES7KOHDPKFLWU73TZYIZMOMH3CTQ, seg_pos={0 0}, upload=515144]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE): protocol: write tcp 188.166.225.170:37124->145.79.227.39:4015: use of closed network connection; write tcp 188.166.225.170:37124->145.79.227.39:4015: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37124->145.79.227.39:4015: use of closed network connection Dec 30 03:07:48.858198125 (587h38m48.72928116s) [noise=true , address=145.79.227.39:4015 , piece_id=ECWADQ244MB4JQRKF66MVUOOES7KOHDPKFLWU73TZYIZMOMH3CTQ, seg_pos={0 0}, upload=515144]: Long tail cancellation due to optimal threshold Dec 30 03:07:48.858266636 (587h38m48.729349671s) [seg_pos={0 0}, upload=515144]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 03:07:48.858400080 (587h38m48.729483125s) [upload=515144]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 03:07:48.858405590 (587h38m48.729488635s) [upload=515144]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 03:07:49.141334163 (587h38m49.012417198s) [upload=515144]: Done uploading segments... Dec 30 03:07:49.141353911 (587h38m49.012436956s) [upload=515144]: Upload finished. err: <nil> Duration: 30.003788694s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [................................X...X.X.XX................XX] duration: 5.001613825s trace: https://jaeger.dev.storj.io/trace/55b84c463909ac27 FAIL (2025-12-30 03:08:07 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00163781s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [.................................X...X.X.XX................X] duration: 5.000906417s trace: https://jaeger.dev.storj.io/trace/4da263c2ba407efb FAIL (2025-12-30 03:07:14 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000937556s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [....X.............X................X......X.................] duration: 15.344826048s trace: https://jaeger.dev.storj.io/trace/6c3caa8adcfacbb9 FAIL (2025-12-30 03:07:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== true , address=145.79.227.35:4496 , piece_id=5X54ZGU4OIVI3DT622QE7IL66SVFLZYUICPODGMDKN7U4D4OJNSA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.040303259 (587h24m33.911386294s) [noise=true , address=77.90.31.12:4601 , piece_id=QQFK4PCL27YYF3UATGLV5KDENRNX6NHE7RXAZ2K66CC3WLHEBMWQ, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.040670117 (587h24m33.911753152s) [noise=true , address=77.90.31.21:4881 , piece_id=EMV4LHFK2VRHSW6PD3BBYGOXDAAYJKX6GEOXGGLOT3IXWNDQMU4A, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041006247 (587h24m33.912089292s) [noise=true , address=145.79.227.52:4472 , piece_id=DT2DAIC4ZI3WL7OGY55ZTTXJEI4IDRXPSJIQA3LTRC7K2F6EVPCA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041183314 (587h24m33.912266349s) [noise=true , address=145.79.227.26:4674 , piece_id=3BAN2ZXOFTSGZITYYHMZUS4AD35X6ZBOYA2HKOJCJI7VBWPJ3LSA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041313653 (587h24m33.912396688s) [noise=true , address=145.79.227.51:4429 , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041535194 (587h24m33.912618229s) [noise=true , address=145.79.227.33:4885 , piece_id=LA755L4RA4VJBHNCQBXOCFROZA3W6W6PUPKGJ5J772WAN44YMY2A, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041639693 (587h24m33.912722728s) [noise=true , address=145.79.227.43:4187 , piece_id=CAAFD3XIWWHZN7L7SYDMMSC5G5RKRLFYYHQZEEN4GAC2D6Y2EARA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041877216 (587h24m33.912960261s) [noise=true , address=145.79.227.46:4623 , piece_id=V4TEM7RRTTGBW5ISZOEAOPFIOZZ5EABO7BPWNRH5U5UPM2X7554Q, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041986634 (587h24m33.913069669s) [noise=true , address=45.91.116.13:4176 , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.042345657 (587h24m33.913428692s) [noise=true , address=45.91.116.10:4104 , piece_id=MRIWH6WGACE4UDNX55MIY6ZDWBAHS6E5KQS4QCOQN2BU7JI7ODCA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.042362019 (587h24m33.913445054s) [noise=true , address=77.90.31.11:4573 , piece_id=TORCC4W7DS5TJ7H7NWONEPNJOFULSZ6F5OHOP4BEVMGEH2MJA4DQ, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.042684192 (587h24m33.913767237s) [noise=true , address=145.79.227.49:4340 , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.042707327 (587h24m33.913790362s) [noise=true , address=145.79.227.32:4837 , piece_id=SQTHTF5KWZOKFYX6CGL6KCDRDAY7OJY6NA5OM3N4YCICDPM6DKXA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.043091528 (587h24m33.914174563s) [noise=true , address=45.91.116.5:4309 , piece_id=Y5HCMU6Z2FPBV3BNTUOC4F6JJWKNBKPOXWFVAFQTXUKQNHTLA4TA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.043544771 (587h24m33.914627816s) [noise=true , address=145.79.227.40:4053 , piece_id=3IZP7JBGSWBRBQHMJWXIDD73B5R3COWU5H3IYFGVBCTTLW7OUJ5A, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:35.347637489 (587h24m35.218720524s) [upload=514937]: Next returned nil segment Dec 30 02:53:35.347733591 (587h24m35.218816626s) [upload=514937]: Waiting for error group managing segments... Dec 30 02:53:35.674169656 (587h24m35.545252721s) [noise=true , address=64.49.13.27:4959 , piece_id=B4UZ67DIWNRHV77VTQONS6ZBUI2BPDDBM3ZS6FVCTABQZ6OHRJJA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:35.891295347 (587h24m35.762378382s) [noise=true , address=145.79.227.32:4837 , piece_id=SQTHTF5KWZOKFYX6CGL6KCDRDAY7OJY6NA5OM3N4YCICDPM6DKXA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:35.912619632 (587h24m35.783702677s) [noise=true , address=145.79.227.26:4674 , piece_id=3BAN2ZXOFTSGZITYYHMZUS4AD35X6ZBOYA2HKOJCJI7VBWPJ3LSA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:35.913054741 (587h24m35.784137786s) [noise=true , address=145.79.227.46:4623 , piece_id=V4TEM7RRTTGBW5ISZOEAOPFIOZZ5EABO7BPWNRH5U5UPM2X7554Q, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:35.913485270 (587h24m35.784568305s) [noise=true , address=145.79.227.35:4496 , piece_id=5X54ZGU4OIVI3DT622QE7IL66SVFLZYUICPODGMDKN7U4D4OJNSA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:35.923790864 (587h24m35.794873909s) [noise=true , address=145.79.227.52:4472 , piece_id=DT2DAIC4ZI3WL7OGY55ZTTXJEI4IDRXPSJIQA3LTRC7K2F6EVPCA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.109605534 (587h24m35.980688569s) [noise=true , address=77.90.31.21:4881 , piece_id=EMV4LHFK2VRHSW6PD3BBYGOXDAAYJKX6GEOXGGLOT3IXWNDQMU4A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.122502652 (587h24m35.993585697s) [noise=true , address=77.90.31.11:4573 , piece_id=TORCC4W7DS5TJ7H7NWONEPNJOFULSZ6F5OHOP4BEVMGEH2MJA4DQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.151131466 (587h24m36.022214511s) [noise=true , address=45.91.116.10:4104 , piece_id=MRIWH6WGACE4UDNX55MIY6ZDWBAHS6E5KQS4QCOQN2BU7JI7ODCA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.159562149 (587h24m36.030645184s) [noise=true , address=77.90.31.12:4601 , piece_id=QQFK4PCL27YYF3UATGLV5KDENRNX6NHE7RXAZ2K66CC3WLHEBMWQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.238339784 (587h24m36.10942282s) [noise=true , address=45.91.116.5:4309 , piece_id=Y5HCMU6Z2FPBV3BNTUOC4F6JJWKNBKPOXWFVAFQTXUKQNHTLA4TA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.813503456 (587h24m36.684586501s) [noise=true , address=145.79.227.33:4885 , piece_id=LA755L4RA4VJBHNCQBXOCFROZA3W6W6PUPKGJ5J772WAN44YMY2A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.826964278 (587h24m36.698047313s) [noise=true , address=145.79.227.43:4187 , piece_id=CAAFD3XIWWHZN7L7SYDMMSC5G5RKRLFYYHQZEEN4GAC2D6Y2EARA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.828562510 (587h24m36.699645545s) [noise=true , address=145.79.227.40:4053 , piece_id=3IZP7JBGSWBRBQHMJWXIDD73B5R3COWU5H3IYFGVBCTTLW7OUJ5A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.828575566 (587h24m36.699658601s) [seg_pos={0 0}, upload=514937]: Segment reached optimal threshold of 14 pieces. Dec 30 02:53:36.828752202 (587h24m36.699835237s) [noise=true , address=145.79.227.30:4780 , piece_id=DZQ5LIBMWXPJ4XXWORRDLQIAFXJDNDQEEBRMSBS44UX3D2HPTSWQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125WCbZ6a2NaewiLAHhHuPo91sQEfC9CZzEBV6rwF1JY7z13AEY): protocol: expected piece hash; context canceled; EOF Dec 30 02:53:36.828758714 (587h24m36.699841749s) [noise=true , address=145.79.227.30:4780 , piece_id=DZQ5LIBMWXPJ4XXWORRDLQIAFXJDNDQEEBRMSBS44UX3D2HPTSWQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold Dec 30 02:53:36.828762672 (587h24m36.699845707s) [noise=true , address=145.79.227.51:4429 , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qytmzwy7sTmWXskEbLWsWMniPz55zgho2akE7oh6ahAiT5SXf): protocol: write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection; write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection Dec 30 02:53:36.828774034 (587h24m36.699857069s) [noise=true , address=145.79.227.51:4429 , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold Dec 30 02:53:36.828889934 (587h24m36.699972969s) [noise=true , address=45.91.116.13:4176 , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ejsmBd4tAVwTHjamPYbryQb5x2vDnk9pBuX3CBF9WYFhTrFVJ): protocol: expected piece hash; context canceled; EOF Dec 30 02:53:36.828895144 (587h24m36.699978179s) [noise=true , address=45.91.116.13:4176 , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold Dec 30 02:53:36.828994173 (587h24m36.700077208s) [noise=true , address=145.79.227.49:4340 , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF Dec 30 02:53:36.828999793 (587h24m36.700082828s) [noise=true , address=145.79.227.49:4340 , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold Dec 30 02:53:36.829030332 (587h24m36.700113367s) [seg_pos={0 0}, upload=514937]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 02:53:36.829095486 (587h24m36.700178621s) [upload=514937]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 02:53:36.829100105 (587h24m36.70018315s) [upload=514937]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 02:53:37.128695270 (587h24m36.999778315s) [upload=514937]: Done uploading segments... Dec 30 02:53:37.128717683 (587h24m36.999800728s) [upload=514937]: Upload finished. err: <nil> Duration: 15.344842398s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [...X........X...X..XX..X.X......XXX.X.X..X........X.X.XX..X.] duration: 5.966164759s trace: https://jaeger.dev.storj.io/trace/1037b2d6d124385e FAIL (2025-12-30 03:07:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: OL4HTVKLCIES44PFNINL73SANMMZXOPDITY5MHTTAKRAU6GILO2A) context deadline exceeded; piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: Q5CBEFIJGIMQHG7YSAGYTGLVNS6LCM7A2SAWFUA6BOEYY7LYZGLA) context deadline exceeded ======LOGS====== 03:06:23.270054960 (587h37m23.141137996s) [upload=515126]: Starting upload Dec 30 03:06:23.270236226 (587h37m23.141319261s) [upload=515126]: Done starting upload Dec 30 03:06:23.270335655 (587h37m23.14141869s) [upload=515126]: Uploading segments... Dec 30 03:06:23.270431488 (587h37m23.141514533s) [upload=515126]: Got next segment. Inline: false Dec 30 03:06:23.270469229 (587h37m23.141552264s) [upload=515126]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 03:06:23.270474038 (587h37m23.141557073s) [upload=515126]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 03:06:23.520231147 (587h37m23.391314192s) [seg_pos={0 0}, upload=515126]: Begin upload segment... Dec 30 03:06:23.520288526 (587h37m23.391371571s) [seg_pos={0 0}, upload=515126]: Done begin upload segment. Dec 30 03:06:23.520323923 (587h37m23.391406968s) [noise=true , address=145.79.227.15:4875 , piece_id=WH3BING6GOTFGPGSDWRNAMOO2IWXEU2JNEU27TF2D5H4ONMZJSEA, seg_pos={0 0}, upload=515126]: Uploading piece... Dec 30 03:06:23.520446587 (587h37m23.391529632s) [noise=true , address=145.79.227.36:4519 , piece_id=5LIYTVSF24PAA755GKZ54R43PNH6UROZ3FHBJI4O73EFPYGY45SQ, seg_pos={0 0}, upload=515126]: Uploading piece... Dec 30 03:06:23.521174153 (587h37m23.392257188s) [noise=true , address=145.79.227.6:4259 , piece_id=W3OS72MMCHKZ325XTIVS5JRO7BMWJ6VF6CAOVHIVAUBIQLUTF3BQ, seg_pos={0 0}, upload=515126]: Uploading piece... Dec 30 03:06:23.521600925 (587h37m23.39268397s) [noise=true , address=145.79.227.45:4230 , piece_id=GL5I3XGMM3N2CDU7YCRDWUZ4GV7SR6ACO3QIU46X4S4FER33DMZA, seg_pos={0 0}, upload=515126]: Uploading piece... Dec 30 03:06:23.522296991 (587h37m23.393380066s) [noise=true , address=145.79.227.9:4367 , piece_id=IAK7XLIYRSIJBKKIOCBO3JIGP46M5GKYSIBXIU5XAU54M27HEEWA, seg_pos={0 0}, upload=515126]: Uploading piece... Dec 30 03:06:23.522482954 (587h37m23.39356599s) [noise=true , address=45.91.116.8:4047 , piece_id=QWT5FOPZB2IJAV6I33EHWG22BYNU6HUWKWTCUWLKJR7HLECSBACQ, seg_pos={0 0}, upload=515126]: Uploading piece... Dec 30 03:06:23.522806581 (587h37m23.393889616s) [noise=true , address=145.79.227.37:4566 , piece_id=VTRRLYPUK23LPFYI773LOFSOFAD2RM2SPCEKL3XYGJS2NZOMNBCQ, seg_pos={0 0}, upload=515126]: Uploading piece... Dec 30 03:06:23.523276786 (587h37m23.394359821s) [noise=true , address=145.79.227.40:4051 , piece_id=3WVBWZ6AQ6SKWZRDYWYXA4MJ4XQ475UGSUY4UDZ3BBYLW6CBAKDQ, seg_pos={0 0}, upload=515126]: Uploading piece... Dec 30 03:06:23.523388138 (587h37m23.394471183s) [noise=true , address=77.90.31.10:4518 , piece_id=OL4HTVKLCIES44PFNINL73SANMMZXOPDITY5MHTTAKRAU6GILO2A, seg_pos={0 0}, upload=515126]: Uploading piece... Dec 30 03:06:23.523797608 (587h37m23.394880653s) [noise=true , address=77.90.31.13:4629 , piece_id=JDTETOQG46SAHSNLW6DV7QUK6HBN4ZUOANSSNTZW2JKM2JTISSOQ, seg_pos={0 0}, upload=515126]: Uploading piece... Dec 30 03:06:23.524197220 (587h37m23.395280265s) [noise=true , address=145.79.227.8:4331 , piece_id=7J3NHSJ6YZD6M5LWSMN3KGRNVCSOK2PESNIFN7U3XJY2R24V62UQ, seg_pos={0 0}, upload=515126]: Uploading piece... Dec 30 03:06:23.524282201 (587h37m23.395365246s) [noise=true , address=145.79.227.50:4375 , piece_id=Z35SPEM5DY5MQDFY5IGZWHNOJIM7HEQFPBG52BHTOYGD3ILIXNRA, seg_pos={0 0}, upload=515126]: Uploading piece... Dec 30 03:06:23.524350661 (587h37m23.395433706s) [noise=true , address=45.91.116.13:4154 , piece_id=QRX4CH2DIMWNG7OUDL5IWEG6NYBA22XNS2EOUBMB4PBAMFVCUNLQ, seg_pos={0 0}, upload=515126]: Uploading piece... Dec 30 03:06:23.524867195 (587h37m23.39595023s) [noise=true , address=77.90.31.12:4589 , piece_id=Q5CBEFIJGIMQHG7YSAGYTGLVNS6LCM7A2SAWFUA6BOEYY7LYZGLA, seg_pos={0 0}, upload=515126]: Uploading piece... Dec 30 03:06:23.525877238 (587h37m23.396960273s) [noise=true , address=145.79.227.48:4301 , piece_id=YY36IULPEBE2VDFGLTYA47ABQD4FGTMA557I4JUODYKXK6HZ75CQ, seg_pos={0 0}, upload=515126]: Uploading piece... Dec 30 03:06:23.526766030 (587h37m23.397849075s) [noise=true , address=145.79.227.11:4732 , piece_id=7KJWUG256CVRH4JWV36WA7JYB2V7XSUJXDBWYHFIMAAZUTDIWKQQ, seg_pos={0 0}, upload=515126]: Uploading piece... Dec 30 03:06:25.213449403 (587h37m25.084532448s) [upload=515126]: Next returned nil segment Dec 30 03:06:25.213473068 (587h37m25.084556113s) [upload=515126]: Waiting for error group managing segments... Dec 30 03:06:25.497405021 (587h37m25.368488056s) [noise=true , address=145.79.227.45:4230 , piece_id=GL5I3XGMM3N2CDU7YCRDWUZ4GV7SR6ACO3QIU46X4S4FER33DMZA, seg_pos={0 0}, upload=515126]: Done uploading piece. err: <nil> Dec 30 03:06:25.505392360 (587h37m25.376475405s) [noise=true , address=145.79.227.6:4259 , piece_id=W3OS72MMCHKZ325XTIVS5JRO7BMWJ6VF6CAOVHIVAUBIQLUTF3BQ, seg_pos={0 0}, upload=515126]: Done uploading piece. err: <nil> Dec 30 03:06:25.512193912 (587h37m25.383276947s) [noise=true , address=145.79.227.37:4566 , piece_id=VTRRLYPUK23LPFYI773LOFSOFAD2RM2SPCEKL3XYGJS2NZOMNBCQ, seg_pos={0 0}, upload=515126]: Done uploading piece. err: <nil> Dec 30 03:06:25.513404586 (587h37m25.384487631s) [noise=true , address=145.79.227.15:4875 , piece_id=WH3BING6GOTFGPGSDWRNAMOO2IWXEU2JNEU27TF2D5H4ONMZJSEA, seg_pos={0 0}, upload=515126]: Done uploading piece. err: <nil> Dec 30 03:06:25.522354448 (587h37m25.393437483s) [noise=true , address=145.79.227.36:4519 , piece_id=5LIYTVSF24PAA755GKZ54R43PNH6UROZ3FHBJI4O73EFPYGY45SQ, seg_pos={0 0}, upload=515126]: Done uploading piece. err: <nil> Dec 30 03:06:25.528152790 (587h37m25.399235825s) [noise=true , address=145.79.227.9:4367 , piece_id=IAK7XLIYRSIJBKKIOCBO3JIGP46M5GKYSIBXIU5XAU54M27HEEWA, seg_pos={0 0}, upload=515126]: Done uploading piece. err: <nil> Dec 30 03:06:25.531653766 (587h37m25.402736801s) [noise=true , address=145.79.227.11:4732 , piece_id=7KJWUG256CVRH4JWV36WA7JYB2V7XSUJXDBWYHFIMAAZUTDIWKQQ, seg_pos={0 0}, upload=515126]: Done uploading piece. err: <nil> Dec 30 03:06:25.541399072 (587h37m25.412482117s) [noise=true , address=145.79.227.50:4375 , piece_id=Z35SPEM5DY5MQDFY5IGZWHNOJIM7HEQFPBG52BHTOYGD3ILIXNRA, seg_pos={0 0}, upload=515126]: Done uploading piece. err: <nil> Dec 30 03:06:25.547662299 (587h37m25.418745334s) [noise=true , address=145.79.227.8:4331 , piece_id=7J3NHSJ6YZD6M5LWSMN3KGRNVCSOK2PESNIFN7U3XJY2R24V62UQ, seg_pos={0 0}, upload=515126]: Done uploading piece. err: <nil> Dec 30 03:06:25.690728931 (587h37m25.561811977s) [noise=true , address=77.90.31.13:4629 , piece_id=JDTETOQG46SAHSNLW6DV7QUK6HBN4ZUOANSSNTZW2JKM2JTISSOQ, seg_pos={0 0}, upload=515126]: Done uploading piece. err: <nil> Dec 30 03:06:25.874204940 (587h37m25.745287975s) [noise=true , address=77.90.31.12:4589 , piece_id=Q5CBEFIJGIMQHG7YSAGYTGLVNS6LCM7A2SAWFUA6BOEYY7LYZGLA, seg_pos={0 0}, upload=515126]: Done uploading piece. err: <nil> Dec 30 03:06:25.884770398 (587h37m25.755853433s) [noise=true , address=45.91.116.8:4047 , piece_id=QWT5FOPZB2IJAV6I33EHWG22BYNU6HUWKWTCUWLKJR7HLECSBACQ, seg_pos={0 0}, upload=515126]: Done uploading piece. err: <nil> Dec 30 03:06:26.103519923 (587h37m25.974602968s) [noise=true , address=77.90.31.10:4518 , piece_id=OL4HTVKLCIES44PFNINL73SANMMZXOPDITY5MHTTAKRAU6GILO2A, seg_pos={0 0}, upload=515126]: Done uploading piece. err: <nil> Dec 30 03:06:26.103544951 (587h37m25.974627986s) [seg_pos={0 0}, upload=515126]: Segment reached optimal threshold of 13 pieces. Dec 30 03:06:26.103768025 (587h37m25.97485109s) [noise=true , address=45.91.116.13:4154 , piece_id=QRX4CH2DIMWNG7OUDL5IWEG6NYBA22XNS2EOUBMB4PBAMFVCUNLQ, seg_pos={0 0}, upload=515126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 30 03:06:26.103776802 (587h37m25.974859837s) [noise=true , address=45.91.116.13:4154 , piece_id=QRX4CH2DIMWNG7OUDL5IWEG6NYBA22XNS2EOUBMB4PBAMFVCUNLQ, seg_pos={0 0}, upload=515126]: Long tail cancellation due to optimal threshold Dec 30 03:06:26.103803623 (587h37m25.974886658s) [noise=true , address=145.79.227.40:4051 , piece_id=3WVBWZ6AQ6SKWZRDYWYXA4MJ4XQ475UGSUY4UDZ3BBYLW6CBAKDQ, seg_pos={0 0}, upload=515126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 30 03:06:26.103811769 (587h37m25.974894814s) [noise=true , address=145.79.227.40:4051 , piece_id=3WVBWZ6AQ6SKWZRDYWYXA4MJ4XQ475UGSUY4UDZ3BBYLW6CBAKDQ, seg_pos={0 0}, upload=515126]: Long tail cancellation due to optimal threshold Dec 30 03:06:26.104027329 (587h37m25.975110364s) [noise=true , address=145.79.227.48:4301 , piece_id=YY36IULPEBE2VDFGLTYA47ABQD4FGTMA557I4JUODYKXK6HZ75CQ, seg_pos={0 0}, upload=515126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18HkaSrdHHnvmeNz1ormfcSBZH8bMUgkX7X3JmGuqkmtBoak29): protocol: write tcp 188.166.225.170:45176->145.79.227.48:4301: use of closed network connection; write tcp 188.166.225.170:45176->145.79.227.48:4301: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45176->145.79.227.48:4301: use of closed network connection Dec 30 03:06:26.104035194 (587h37m25.975118239s) [noise=true , address=145.79.227.48:4301 , piece_id=YY36IULPEBE2VDFGLTYA47ABQD4FGTMA557I4JUODYKXK6HZ75CQ, seg_pos={0 0}, upload=515126]: Long tail cancellation due to optimal threshold Dec 30 03:06:26.104069509 (587h37m25.975152544s) [seg_pos={0 0}, upload=515126]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 03:06:26.104161124 (587h37m25.975244159s) [upload=515126]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 03:06:26.104167045 (587h37m25.97525008s) [upload=515126]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 03:06:26.406388955 (587h37m26.27747199s) [upload=515126]: Done uploading segments... Dec 30 03:06:26.406412971 (587h37m26.277496006s) [upload=515126]: Upload finished. err: <nil> Duration: 5.966210667s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....X.............X................X......X................] duration: 36.478610054s trace: https://jaeger.dev.storj.io/trace/6c3caa8adcfacbb9 FAIL (2025-12-30 03:06:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== true , address=145.79.227.35:4496 , piece_id=5X54ZGU4OIVI3DT622QE7IL66SVFLZYUICPODGMDKN7U4D4OJNSA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.040303259 (587h24m33.911386294s) [noise=true , address=77.90.31.12:4601 , piece_id=QQFK4PCL27YYF3UATGLV5KDENRNX6NHE7RXAZ2K66CC3WLHEBMWQ, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.040670117 (587h24m33.911753152s) [noise=true , address=77.90.31.21:4881 , piece_id=EMV4LHFK2VRHSW6PD3BBYGOXDAAYJKX6GEOXGGLOT3IXWNDQMU4A, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041006247 (587h24m33.912089292s) [noise=true , address=145.79.227.52:4472 , piece_id=DT2DAIC4ZI3WL7OGY55ZTTXJEI4IDRXPSJIQA3LTRC7K2F6EVPCA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041183314 (587h24m33.912266349s) [noise=true , address=145.79.227.26:4674 , piece_id=3BAN2ZXOFTSGZITYYHMZUS4AD35X6ZBOYA2HKOJCJI7VBWPJ3LSA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041313653 (587h24m33.912396688s) [noise=true , address=145.79.227.51:4429 , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041535194 (587h24m33.912618229s) [noise=true , address=145.79.227.33:4885 , piece_id=LA755L4RA4VJBHNCQBXOCFROZA3W6W6PUPKGJ5J772WAN44YMY2A, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041639693 (587h24m33.912722728s) [noise=true , address=145.79.227.43:4187 , piece_id=CAAFD3XIWWHZN7L7SYDMMSC5G5RKRLFYYHQZEEN4GAC2D6Y2EARA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041877216 (587h24m33.912960261s) [noise=true , address=145.79.227.46:4623 , piece_id=V4TEM7RRTTGBW5ISZOEAOPFIOZZ5EABO7BPWNRH5U5UPM2X7554Q, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041986634 (587h24m33.913069669s) [noise=true , address=45.91.116.13:4176 , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.042345657 (587h24m33.913428692s) [noise=true , address=45.91.116.10:4104 , piece_id=MRIWH6WGACE4UDNX55MIY6ZDWBAHS6E5KQS4QCOQN2BU7JI7ODCA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.042362019 (587h24m33.913445054s) [noise=true , address=77.90.31.11:4573 , piece_id=TORCC4W7DS5TJ7H7NWONEPNJOFULSZ6F5OHOP4BEVMGEH2MJA4DQ, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.042684192 (587h24m33.913767237s) [noise=true , address=145.79.227.49:4340 , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.042707327 (587h24m33.913790362s) [noise=true , address=145.79.227.32:4837 , piece_id=SQTHTF5KWZOKFYX6CGL6KCDRDAY7OJY6NA5OM3N4YCICDPM6DKXA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.043091528 (587h24m33.914174563s) [noise=true , address=45.91.116.5:4309 , piece_id=Y5HCMU6Z2FPBV3BNTUOC4F6JJWKNBKPOXWFVAFQTXUKQNHTLA4TA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.043544771 (587h24m33.914627816s) [noise=true , address=145.79.227.40:4053 , piece_id=3IZP7JBGSWBRBQHMJWXIDD73B5R3COWU5H3IYFGVBCTTLW7OUJ5A, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:35.347637489 (587h24m35.218720524s) [upload=514937]: Next returned nil segment Dec 30 02:53:35.347733591 (587h24m35.218816626s) [upload=514937]: Waiting for error group managing segments... Dec 30 02:53:35.674169656 (587h24m35.545252721s) [noise=true , address=64.49.13.27:4959 , piece_id=B4UZ67DIWNRHV77VTQONS6ZBUI2BPDDBM3ZS6FVCTABQZ6OHRJJA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:35.891295347 (587h24m35.762378382s) [noise=true , address=145.79.227.32:4837 , piece_id=SQTHTF5KWZOKFYX6CGL6KCDRDAY7OJY6NA5OM3N4YCICDPM6DKXA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:35.912619632 (587h24m35.783702677s) [noise=true , address=145.79.227.26:4674 , piece_id=3BAN2ZXOFTSGZITYYHMZUS4AD35X6ZBOYA2HKOJCJI7VBWPJ3LSA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:35.913054741 (587h24m35.784137786s) [noise=true , address=145.79.227.46:4623 , piece_id=V4TEM7RRTTGBW5ISZOEAOPFIOZZ5EABO7BPWNRH5U5UPM2X7554Q, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:35.913485270 (587h24m35.784568305s) [noise=true , address=145.79.227.35:4496 , piece_id=5X54ZGU4OIVI3DT622QE7IL66SVFLZYUICPODGMDKN7U4D4OJNSA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:35.923790864 (587h24m35.794873909s) [noise=true , address=145.79.227.52:4472 , piece_id=DT2DAIC4ZI3WL7OGY55ZTTXJEI4IDRXPSJIQA3LTRC7K2F6EVPCA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.109605534 (587h24m35.980688569s) [noise=true , address=77.90.31.21:4881 , piece_id=EMV4LHFK2VRHSW6PD3BBYGOXDAAYJKX6GEOXGGLOT3IXWNDQMU4A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.122502652 (587h24m35.993585697s) [noise=true , address=77.90.31.11:4573 , piece_id=TORCC4W7DS5TJ7H7NWONEPNJOFULSZ6F5OHOP4BEVMGEH2MJA4DQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.151131466 (587h24m36.022214511s) [noise=true , address=45.91.116.10:4104 , piece_id=MRIWH6WGACE4UDNX55MIY6ZDWBAHS6E5KQS4QCOQN2BU7JI7ODCA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.159562149 (587h24m36.030645184s) [noise=true , address=77.90.31.12:4601 , piece_id=QQFK4PCL27YYF3UATGLV5KDENRNX6NHE7RXAZ2K66CC3WLHEBMWQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.238339784 (587h24m36.10942282s) [noise=true , address=45.91.116.5:4309 , piece_id=Y5HCMU6Z2FPBV3BNTUOC4F6JJWKNBKPOXWFVAFQTXUKQNHTLA4TA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.813503456 (587h24m36.684586501s) [noise=true , address=145.79.227.33:4885 , piece_id=LA755L4RA4VJBHNCQBXOCFROZA3W6W6PUPKGJ5J772WAN44YMY2A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.826964278 (587h24m36.698047313s) [noise=true , address=145.79.227.43:4187 , piece_id=CAAFD3XIWWHZN7L7SYDMMSC5G5RKRLFYYHQZEEN4GAC2D6Y2EARA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.828562510 (587h24m36.699645545s) [noise=true , address=145.79.227.40:4053 , piece_id=3IZP7JBGSWBRBQHMJWXIDD73B5R3COWU5H3IYFGVBCTTLW7OUJ5A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.828575566 (587h24m36.699658601s) [seg_pos={0 0}, upload=514937]: Segment reached optimal threshold of 14 pieces. Dec 30 02:53:36.828752202 (587h24m36.699835237s) [noise=true , address=145.79.227.30:4780 , piece_id=DZQ5LIBMWXPJ4XXWORRDLQIAFXJDNDQEEBRMSBS44UX3D2HPTSWQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125WCbZ6a2NaewiLAHhHuPo91sQEfC9CZzEBV6rwF1JY7z13AEY): protocol: expected piece hash; context canceled; EOF Dec 30 02:53:36.828758714 (587h24m36.699841749s) [noise=true , address=145.79.227.30:4780 , piece_id=DZQ5LIBMWXPJ4XXWORRDLQIAFXJDNDQEEBRMSBS44UX3D2HPTSWQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold Dec 30 02:53:36.828762672 (587h24m36.699845707s) [noise=true , address=145.79.227.51:4429 , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qytmzwy7sTmWXskEbLWsWMniPz55zgho2akE7oh6ahAiT5SXf): protocol: write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection; write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection Dec 30 02:53:36.828774034 (587h24m36.699857069s) [noise=true , address=145.79.227.51:4429 , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold Dec 30 02:53:36.828889934 (587h24m36.699972969s) [noise=true , address=45.91.116.13:4176 , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ejsmBd4tAVwTHjamPYbryQb5x2vDnk9pBuX3CBF9WYFhTrFVJ): protocol: expected piece hash; context canceled; EOF Dec 30 02:53:36.828895144 (587h24m36.699978179s) [noise=true , address=45.91.116.13:4176 , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold Dec 30 02:53:36.828994173 (587h24m36.700077208s) [noise=true , address=145.79.227.49:4340 , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF Dec 30 02:53:36.828999793 (587h24m36.700082828s) [noise=true , address=145.79.227.49:4340 , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold Dec 30 02:53:36.829030332 (587h24m36.700113367s) [seg_pos={0 0}, upload=514937]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 02:53:36.829095486 (587h24m36.700178621s) [upload=514937]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 02:53:36.829100105 (587h24m36.70018315s) [upload=514937]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 02:53:37.128695270 (587h24m36.999778315s) [upload=514937]: Done uploading segments... Dec 30 02:53:37.128717683 (587h24m36.999800728s) [upload=514937]: Upload finished. err: <nil> Duration: 36.478636555s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [....X........X...X..XX..X.X......XXX.X.X..X........X.X.XX..X] duration: 30.002991743s trace: https://jaeger.dev.storj.io/trace/1037b2d6d124385e FAIL (2025-12-30 03:06:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: OL4HTVKLCIES44PFNINL73SANMMZXOPDITY5MHTTAKRAU6GILO2A) context deadline exceeded; piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: Q5CBEFIJGIMQHG7YSAGYTGLVNS6LCM7A2SAWFUA6BOEYY7LYZGLA) context deadline exceeded ======LOGS====== 03:06:23.270054960 (587h37m23.141137996s) [upload=515126]: Starting upload Dec 30 03:06:23.270236226 (587h37m23.141319261s) [upload=515126]: Done starting upload Dec 30 03:06:23.270335655 (587h37m23.14141869s) [upload=515126]: Uploading segments... Dec 30 03:06:23.270431488 (587h37m23.141514533s) [upload=515126]: Got next segment. Inline: false Dec 30 03:06:23.270469229 (587h37m23.141552264s) [upload=515126]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 03:06:23.270474038 (587h37m23.141557073s) [upload=515126]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 03:06:23.520231147 (587h37m23.391314192s) [seg_pos={0 0}, upload=515126]: Begin upload segment... Dec 30 03:06:23.520288526 (587h37m23.391371571s) [seg_pos={0 0}, upload=515126]: Done begin upload segment. Dec 30 03:06:23.520323923 (587h37m23.391406968s) [noise=true , address=145.79.227.15:4875 , piece_id=WH3BING6GOTFGPGSDWRNAMOO2IWXEU2JNEU27TF2D5H4ONMZJSEA, seg_pos={0 0}, upload=515126]: Uploading piece... Dec 30 03:06:23.520446587 (587h37m23.391529632s) [noise=true , address=145.79.227.36:4519 , piece_id=5LIYTVSF24PAA755GKZ54R43PNH6UROZ3FHBJI4O73EFPYGY45SQ, seg_pos={0 0}, upload=515126]: Uploading piece... Dec 30 03:06:23.521174153 (587h37m23.392257188s) [noise=true , address=145.79.227.6:4259 , piece_id=W3OS72MMCHKZ325XTIVS5JRO7BMWJ6VF6CAOVHIVAUBIQLUTF3BQ, seg_pos={0 0}, upload=515126]: Uploading piece... Dec 30 03:06:23.521600925 (587h37m23.39268397s) [noise=true , address=145.79.227.45:4230 , piece_id=GL5I3XGMM3N2CDU7YCRDWUZ4GV7SR6ACO3QIU46X4S4FER33DMZA, seg_pos={0 0}, upload=515126]: Uploading piece... Dec 30 03:06:23.522296991 (587h37m23.393380066s) [noise=true , address=145.79.227.9:4367 , piece_id=IAK7XLIYRSIJBKKIOCBO3JIGP46M5GKYSIBXIU5XAU54M27HEEWA, seg_pos={0 0}, upload=515126]: Uploading piece... Dec 30 03:06:23.522482954 (587h37m23.39356599s) [noise=true , address=45.91.116.8:4047 , piece_id=QWT5FOPZB2IJAV6I33EHWG22BYNU6HUWKWTCUWLKJR7HLECSBACQ, seg_pos={0 0}, upload=515126]: Uploading piece... Dec 30 03:06:23.522806581 (587h37m23.393889616s) [noise=true , address=145.79.227.37:4566 , piece_id=VTRRLYPUK23LPFYI773LOFSOFAD2RM2SPCEKL3XYGJS2NZOMNBCQ, seg_pos={0 0}, upload=515126]: Uploading piece... Dec 30 03:06:23.523276786 (587h37m23.394359821s) [noise=true , address=145.79.227.40:4051 , piece_id=3WVBWZ6AQ6SKWZRDYWYXA4MJ4XQ475UGSUY4UDZ3BBYLW6CBAKDQ, seg_pos={0 0}, upload=515126]: Uploading piece... Dec 30 03:06:23.523388138 (587h37m23.394471183s) [noise=true , address=77.90.31.10:4518 , piece_id=OL4HTVKLCIES44PFNINL73SANMMZXOPDITY5MHTTAKRAU6GILO2A, seg_pos={0 0}, upload=515126]: Uploading piece... Dec 30 03:06:23.523797608 (587h37m23.394880653s) [noise=true , address=77.90.31.13:4629 , piece_id=JDTETOQG46SAHSNLW6DV7QUK6HBN4ZUOANSSNTZW2JKM2JTISSOQ, seg_pos={0 0}, upload=515126]: Uploading piece... Dec 30 03:06:23.524197220 (587h37m23.395280265s) [noise=true , address=145.79.227.8:4331 , piece_id=7J3NHSJ6YZD6M5LWSMN3KGRNVCSOK2PESNIFN7U3XJY2R24V62UQ, seg_pos={0 0}, upload=515126]: Uploading piece... Dec 30 03:06:23.524282201 (587h37m23.395365246s) [noise=true , address=145.79.227.50:4375 , piece_id=Z35SPEM5DY5MQDFY5IGZWHNOJIM7HEQFPBG52BHTOYGD3ILIXNRA, seg_pos={0 0}, upload=515126]: Uploading piece... Dec 30 03:06:23.524350661 (587h37m23.395433706s) [noise=true , address=45.91.116.13:4154 , piece_id=QRX4CH2DIMWNG7OUDL5IWEG6NYBA22XNS2EOUBMB4PBAMFVCUNLQ, seg_pos={0 0}, upload=515126]: Uploading piece... Dec 30 03:06:23.524867195 (587h37m23.39595023s) [noise=true , address=77.90.31.12:4589 , piece_id=Q5CBEFIJGIMQHG7YSAGYTGLVNS6LCM7A2SAWFUA6BOEYY7LYZGLA, seg_pos={0 0}, upload=515126]: Uploading piece... Dec 30 03:06:23.525877238 (587h37m23.396960273s) [noise=true , address=145.79.227.48:4301 , piece_id=YY36IULPEBE2VDFGLTYA47ABQD4FGTMA557I4JUODYKXK6HZ75CQ, seg_pos={0 0}, upload=515126]: Uploading piece... Dec 30 03:06:23.526766030 (587h37m23.397849075s) [noise=true , address=145.79.227.11:4732 , piece_id=7KJWUG256CVRH4JWV36WA7JYB2V7XSUJXDBWYHFIMAAZUTDIWKQQ, seg_pos={0 0}, upload=515126]: Uploading piece... Dec 30 03:06:25.213449403 (587h37m25.084532448s) [upload=515126]: Next returned nil segment Dec 30 03:06:25.213473068 (587h37m25.084556113s) [upload=515126]: Waiting for error group managing segments... Dec 30 03:06:25.497405021 (587h37m25.368488056s) [noise=true , address=145.79.227.45:4230 , piece_id=GL5I3XGMM3N2CDU7YCRDWUZ4GV7SR6ACO3QIU46X4S4FER33DMZA, seg_pos={0 0}, upload=515126]: Done uploading piece. err: <nil> Dec 30 03:06:25.505392360 (587h37m25.376475405s) [noise=true , address=145.79.227.6:4259 , piece_id=W3OS72MMCHKZ325XTIVS5JRO7BMWJ6VF6CAOVHIVAUBIQLUTF3BQ, seg_pos={0 0}, upload=515126]: Done uploading piece. err: <nil> Dec 30 03:06:25.512193912 (587h37m25.383276947s) [noise=true , address=145.79.227.37:4566 , piece_id=VTRRLYPUK23LPFYI773LOFSOFAD2RM2SPCEKL3XYGJS2NZOMNBCQ, seg_pos={0 0}, upload=515126]: Done uploading piece. err: <nil> Dec 30 03:06:25.513404586 (587h37m25.384487631s) [noise=true , address=145.79.227.15:4875 , piece_id=WH3BING6GOTFGPGSDWRNAMOO2IWXEU2JNEU27TF2D5H4ONMZJSEA, seg_pos={0 0}, upload=515126]: Done uploading piece. err: <nil> Dec 30 03:06:25.522354448 (587h37m25.393437483s) [noise=true , address=145.79.227.36:4519 , piece_id=5LIYTVSF24PAA755GKZ54R43PNH6UROZ3FHBJI4O73EFPYGY45SQ, seg_pos={0 0}, upload=515126]: Done uploading piece. err: <nil> Dec 30 03:06:25.528152790 (587h37m25.399235825s) [noise=true , address=145.79.227.9:4367 , piece_id=IAK7XLIYRSIJBKKIOCBO3JIGP46M5GKYSIBXIU5XAU54M27HEEWA, seg_pos={0 0}, upload=515126]: Done uploading piece. err: <nil> Dec 30 03:06:25.531653766 (587h37m25.402736801s) [noise=true , address=145.79.227.11:4732 , piece_id=7KJWUG256CVRH4JWV36WA7JYB2V7XSUJXDBWYHFIMAAZUTDIWKQQ, seg_pos={0 0}, upload=515126]: Done uploading piece. err: <nil> Dec 30 03:06:25.541399072 (587h37m25.412482117s) [noise=true , address=145.79.227.50:4375 , piece_id=Z35SPEM5DY5MQDFY5IGZWHNOJIM7HEQFPBG52BHTOYGD3ILIXNRA, seg_pos={0 0}, upload=515126]: Done uploading piece. err: <nil> Dec 30 03:06:25.547662299 (587h37m25.418745334s) [noise=true , address=145.79.227.8:4331 , piece_id=7J3NHSJ6YZD6M5LWSMN3KGRNVCSOK2PESNIFN7U3XJY2R24V62UQ, seg_pos={0 0}, upload=515126]: Done uploading piece. err: <nil> Dec 30 03:06:25.690728931 (587h37m25.561811977s) [noise=true , address=77.90.31.13:4629 , piece_id=JDTETOQG46SAHSNLW6DV7QUK6HBN4ZUOANSSNTZW2JKM2JTISSOQ, seg_pos={0 0}, upload=515126]: Done uploading piece. err: <nil> Dec 30 03:06:25.874204940 (587h37m25.745287975s) [noise=true , address=77.90.31.12:4589 , piece_id=Q5CBEFIJGIMQHG7YSAGYTGLVNS6LCM7A2SAWFUA6BOEYY7LYZGLA, seg_pos={0 0}, upload=515126]: Done uploading piece. err: <nil> Dec 30 03:06:25.884770398 (587h37m25.755853433s) [noise=true , address=45.91.116.8:4047 , piece_id=QWT5FOPZB2IJAV6I33EHWG22BYNU6HUWKWTCUWLKJR7HLECSBACQ, seg_pos={0 0}, upload=515126]: Done uploading piece. err: <nil> Dec 30 03:06:26.103519923 (587h37m25.974602968s) [noise=true , address=77.90.31.10:4518 , piece_id=OL4HTVKLCIES44PFNINL73SANMMZXOPDITY5MHTTAKRAU6GILO2A, seg_pos={0 0}, upload=515126]: Done uploading piece. err: <nil> Dec 30 03:06:26.103544951 (587h37m25.974627986s) [seg_pos={0 0}, upload=515126]: Segment reached optimal threshold of 13 pieces. Dec 30 03:06:26.103768025 (587h37m25.97485109s) [noise=true , address=45.91.116.13:4154 , piece_id=QRX4CH2DIMWNG7OUDL5IWEG6NYBA22XNS2EOUBMB4PBAMFVCUNLQ, seg_pos={0 0}, upload=515126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 30 03:06:26.103776802 (587h37m25.974859837s) [noise=true , address=45.91.116.13:4154 , piece_id=QRX4CH2DIMWNG7OUDL5IWEG6NYBA22XNS2EOUBMB4PBAMFVCUNLQ, seg_pos={0 0}, upload=515126]: Long tail cancellation due to optimal threshold Dec 30 03:06:26.103803623 (587h37m25.974886658s) [noise=true , address=145.79.227.40:4051 , piece_id=3WVBWZ6AQ6SKWZRDYWYXA4MJ4XQ475UGSUY4UDZ3BBYLW6CBAKDQ, seg_pos={0 0}, upload=515126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 30 03:06:26.103811769 (587h37m25.974894814s) [noise=true , address=145.79.227.40:4051 , piece_id=3WVBWZ6AQ6SKWZRDYWYXA4MJ4XQ475UGSUY4UDZ3BBYLW6CBAKDQ, seg_pos={0 0}, upload=515126]: Long tail cancellation due to optimal threshold Dec 30 03:06:26.104027329 (587h37m25.975110364s) [noise=true , address=145.79.227.48:4301 , piece_id=YY36IULPEBE2VDFGLTYA47ABQD4FGTMA557I4JUODYKXK6HZ75CQ, seg_pos={0 0}, upload=515126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18HkaSrdHHnvmeNz1ormfcSBZH8bMUgkX7X3JmGuqkmtBoak29): protocol: write tcp 188.166.225.170:45176->145.79.227.48:4301: use of closed network connection; write tcp 188.166.225.170:45176->145.79.227.48:4301: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45176->145.79.227.48:4301: use of closed network connection Dec 30 03:06:26.104035194 (587h37m25.975118239s) [noise=true , address=145.79.227.48:4301 , piece_id=YY36IULPEBE2VDFGLTYA47ABQD4FGTMA557I4JUODYKXK6HZ75CQ, seg_pos={0 0}, upload=515126]: Long tail cancellation due to optimal threshold Dec 30 03:06:26.104069509 (587h37m25.975152544s) [seg_pos={0 0}, upload=515126]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 03:06:26.104161124 (587h37m25.975244159s) [upload=515126]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 03:06:26.104167045 (587h37m25.97525008s) [upload=515126]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 03:06:26.406388955 (587h37m26.27747199s) [upload=515126]: Done uploading segments... Dec 30 03:06:26.406412971 (587h37m26.277496006s) [upload=515126]: Upload finished. err: <nil> Duration: 30.00300609s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [......X.............X................X......X...............] duration: 58.654443261s trace: https://jaeger.dev.storj.io/trace/6c3caa8adcfacbb9 FAIL (2025-12-30 03:06:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== true , address=145.79.227.35:4496 , piece_id=5X54ZGU4OIVI3DT622QE7IL66SVFLZYUICPODGMDKN7U4D4OJNSA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.040303259 (587h24m33.911386294s) [noise=true , address=77.90.31.12:4601 , piece_id=QQFK4PCL27YYF3UATGLV5KDENRNX6NHE7RXAZ2K66CC3WLHEBMWQ, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.040670117 (587h24m33.911753152s) [noise=true , address=77.90.31.21:4881 , piece_id=EMV4LHFK2VRHSW6PD3BBYGOXDAAYJKX6GEOXGGLOT3IXWNDQMU4A, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041006247 (587h24m33.912089292s) [noise=true , address=145.79.227.52:4472 , piece_id=DT2DAIC4ZI3WL7OGY55ZTTXJEI4IDRXPSJIQA3LTRC7K2F6EVPCA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041183314 (587h24m33.912266349s) [noise=true , address=145.79.227.26:4674 , piece_id=3BAN2ZXOFTSGZITYYHMZUS4AD35X6ZBOYA2HKOJCJI7VBWPJ3LSA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041313653 (587h24m33.912396688s) [noise=true , address=145.79.227.51:4429 , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041535194 (587h24m33.912618229s) [noise=true , address=145.79.227.33:4885 , piece_id=LA755L4RA4VJBHNCQBXOCFROZA3W6W6PUPKGJ5J772WAN44YMY2A, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041639693 (587h24m33.912722728s) [noise=true , address=145.79.227.43:4187 , piece_id=CAAFD3XIWWHZN7L7SYDMMSC5G5RKRLFYYHQZEEN4GAC2D6Y2EARA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041877216 (587h24m33.912960261s) [noise=true , address=145.79.227.46:4623 , piece_id=V4TEM7RRTTGBW5ISZOEAOPFIOZZ5EABO7BPWNRH5U5UPM2X7554Q, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041986634 (587h24m33.913069669s) [noise=true , address=45.91.116.13:4176 , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.042345657 (587h24m33.913428692s) [noise=true , address=45.91.116.10:4104 , piece_id=MRIWH6WGACE4UDNX55MIY6ZDWBAHS6E5KQS4QCOQN2BU7JI7ODCA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.042362019 (587h24m33.913445054s) [noise=true , address=77.90.31.11:4573 , piece_id=TORCC4W7DS5TJ7H7NWONEPNJOFULSZ6F5OHOP4BEVMGEH2MJA4DQ, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.042684192 (587h24m33.913767237s) [noise=true , address=145.79.227.49:4340 , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.042707327 (587h24m33.913790362s) [noise=true , address=145.79.227.32:4837 , piece_id=SQTHTF5KWZOKFYX6CGL6KCDRDAY7OJY6NA5OM3N4YCICDPM6DKXA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.043091528 (587h24m33.914174563s) [noise=true , address=45.91.116.5:4309 , piece_id=Y5HCMU6Z2FPBV3BNTUOC4F6JJWKNBKPOXWFVAFQTXUKQNHTLA4TA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.043544771 (587h24m33.914627816s) [noise=true , address=145.79.227.40:4053 , piece_id=3IZP7JBGSWBRBQHMJWXIDD73B5R3COWU5H3IYFGVBCTTLW7OUJ5A, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:35.347637489 (587h24m35.218720524s) [upload=514937]: Next returned nil segment Dec 30 02:53:35.347733591 (587h24m35.218816626s) [upload=514937]: Waiting for error group managing segments... Dec 30 02:53:35.674169656 (587h24m35.545252721s) [noise=true , address=64.49.13.27:4959 , piece_id=B4UZ67DIWNRHV77VTQONS6ZBUI2BPDDBM3ZS6FVCTABQZ6OHRJJA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:35.891295347 (587h24m35.762378382s) [noise=true , address=145.79.227.32:4837 , piece_id=SQTHTF5KWZOKFYX6CGL6KCDRDAY7OJY6NA5OM3N4YCICDPM6DKXA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:35.912619632 (587h24m35.783702677s) [noise=true , address=145.79.227.26:4674 , piece_id=3BAN2ZXOFTSGZITYYHMZUS4AD35X6ZBOYA2HKOJCJI7VBWPJ3LSA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:35.913054741 (587h24m35.784137786s) [noise=true , address=145.79.227.46:4623 , piece_id=V4TEM7RRTTGBW5ISZOEAOPFIOZZ5EABO7BPWNRH5U5UPM2X7554Q, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:35.913485270 (587h24m35.784568305s) [noise=true , address=145.79.227.35:4496 , piece_id=5X54ZGU4OIVI3DT622QE7IL66SVFLZYUICPODGMDKN7U4D4OJNSA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:35.923790864 (587h24m35.794873909s) [noise=true , address=145.79.227.52:4472 , piece_id=DT2DAIC4ZI3WL7OGY55ZTTXJEI4IDRXPSJIQA3LTRC7K2F6EVPCA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.109605534 (587h24m35.980688569s) [noise=true , address=77.90.31.21:4881 , piece_id=EMV4LHFK2VRHSW6PD3BBYGOXDAAYJKX6GEOXGGLOT3IXWNDQMU4A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.122502652 (587h24m35.993585697s) [noise=true , address=77.90.31.11:4573 , piece_id=TORCC4W7DS5TJ7H7NWONEPNJOFULSZ6F5OHOP4BEVMGEH2MJA4DQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.151131466 (587h24m36.022214511s) [noise=true , address=45.91.116.10:4104 , piece_id=MRIWH6WGACE4UDNX55MIY6ZDWBAHS6E5KQS4QCOQN2BU7JI7ODCA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.159562149 (587h24m36.030645184s) [noise=true , address=77.90.31.12:4601 , piece_id=QQFK4PCL27YYF3UATGLV5KDENRNX6NHE7RXAZ2K66CC3WLHEBMWQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.238339784 (587h24m36.10942282s) [noise=true , address=45.91.116.5:4309 , piece_id=Y5HCMU6Z2FPBV3BNTUOC4F6JJWKNBKPOXWFVAFQTXUKQNHTLA4TA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.813503456 (587h24m36.684586501s) [noise=true , address=145.79.227.33:4885 , piece_id=LA755L4RA4VJBHNCQBXOCFROZA3W6W6PUPKGJ5J772WAN44YMY2A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.826964278 (587h24m36.698047313s) [noise=true , address=145.79.227.43:4187 , piece_id=CAAFD3XIWWHZN7L7SYDMMSC5G5RKRLFYYHQZEEN4GAC2D6Y2EARA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.828562510 (587h24m36.699645545s) [noise=true , address=145.79.227.40:4053 , piece_id=3IZP7JBGSWBRBQHMJWXIDD73B5R3COWU5H3IYFGVBCTTLW7OUJ5A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.828575566 (587h24m36.699658601s) [seg_pos={0 0}, upload=514937]: Segment reached optimal threshold of 14 pieces. Dec 30 02:53:36.828752202 (587h24m36.699835237s) [noise=true , address=145.79.227.30:4780 , piece_id=DZQ5LIBMWXPJ4XXWORRDLQIAFXJDNDQEEBRMSBS44UX3D2HPTSWQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125WCbZ6a2NaewiLAHhHuPo91sQEfC9CZzEBV6rwF1JY7z13AEY): protocol: expected piece hash; context canceled; EOF Dec 30 02:53:36.828758714 (587h24m36.699841749s) [noise=true , address=145.79.227.30:4780 , piece_id=DZQ5LIBMWXPJ4XXWORRDLQIAFXJDNDQEEBRMSBS44UX3D2HPTSWQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold Dec 30 02:53:36.828762672 (587h24m36.699845707s) [noise=true , address=145.79.227.51:4429 , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qytmzwy7sTmWXskEbLWsWMniPz55zgho2akE7oh6ahAiT5SXf): protocol: write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection; write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection Dec 30 02:53:36.828774034 (587h24m36.699857069s) [noise=true , address=145.79.227.51:4429 , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold Dec 30 02:53:36.828889934 (587h24m36.699972969s) [noise=true , address=45.91.116.13:4176 , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ejsmBd4tAVwTHjamPYbryQb5x2vDnk9pBuX3CBF9WYFhTrFVJ): protocol: expected piece hash; context canceled; EOF Dec 30 02:53:36.828895144 (587h24m36.699978179s) [noise=true , address=45.91.116.13:4176 , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold Dec 30 02:53:36.828994173 (587h24m36.700077208s) [noise=true , address=145.79.227.49:4340 , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF Dec 30 02:53:36.828999793 (587h24m36.700082828s) [noise=true , address=145.79.227.49:4340 , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold Dec 30 02:53:36.829030332 (587h24m36.700113367s) [seg_pos={0 0}, upload=514937]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 02:53:36.829095486 (587h24m36.700178621s) [upload=514937]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 02:53:36.829100105 (587h24m36.70018315s) [upload=514937]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 02:53:37.128695270 (587h24m36.999778315s) [upload=514937]: Done uploading segments... Dec 30 02:53:37.128717683 (587h24m36.999800728s) [upload=514937]: Upload finished. err: <nil> Duration: 58.654458691s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [..................................X...X.X.XX................] duration: 3.604506441s trace: https://jaeger.dev.storj.io/trace/14b283cf2d9b0313 FAIL (2025-12-30 03:06:06 UTC) orbiter warning: context deadline exceeded Duration: 3.604534333s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.....X........X...X..XX..X.X......XXX.X.X..X........X.X.XX..] duration: 6.03429642s trace: https://jaeger.dev.storj.io/trace/1f609d54324f4aba FAIL (2025-12-30 03:05:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 03:03:42.664733818 (587h34m42.535816853s) [upload=515086]: Starting upload Dec 30 03:03:42.664803922 (587h34m42.535886957s) [upload=515086]: Done starting upload Dec 30 03:03:42.664926486 (587h34m42.536009531s) [upload=515086]: Uploading segments... Dec 30 03:03:42.665104645 (587h34m42.53618769s) [upload=515086]: Got next segment. Inline: false Dec 30 03:03:42.665157715 (587h34m42.53624077s) [upload=515086]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 03:03:42.665164197 (587h34m42.536247242s) [upload=515086]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 03:03:42.916348402 (587h34m42.787431447s) [seg_pos={0 0}, upload=515086]: Begin upload segment... Dec 30 03:03:42.916399158 (587h34m42.787482203s) [seg_pos={0 0}, upload=515086]: Done begin upload segment. Dec 30 03:03:42.916461577 (587h34m42.787544612s) [noise=true , address=145.79.227.5:4222 , piece_id=TLYHW5ZI4A7BNPGD46HWFTNEWGWEO4DWI455GNSCBXJUIQY4FA2Q, seg_pos={0 0}, upload=515086]: Uploading piece... Dec 30 03:03:42.916489440 (587h34m42.787572475s) [noise=true , address=45.91.116.11:4119 , piece_id=KEZ2CIX3PCLO4MPD367AEIY5B4G7KGGDBVJKATLDPMTRT5RDBSPQ, seg_pos={0 0}, upload=515086]: Uploading piece... Dec 30 03:03:42.917030490 (587h34m42.788113525s) [noise=true , address=145.79.227.34:4906 , piece_id=GXI6LC45KZ5UA4KPWMXZJ66MA7RUPM4YDX6E2LSNQ7DABUCFFVLA, seg_pos={0 0}, upload=515086]: Uploading piece... Dec 30 03:03:42.917755180 (587h34m42.788838215s) [noise=true , address=145.79.227.13:4803 , piece_id=2NFC5GJ7FPRYQ6QVGE7CMIXNHEB4VDB2TAWWSQA5LM5RZUDMLKBA, seg_pos={0 0}, upload=515086]: Uploading piece... Dec 30 03:03:42.918253569 (587h34m42.789336604s) [noise=true , address=145.79.227.15:4873 , piece_id=DOPL3LC6OXAYLCTIZWBUBY2EOY2EYEV46NG4AHHS776I653WX25A, seg_pos={0 0}, upload=515086]: Uploading piece... Dec 30 03:03:42.919431933 (587h34m42.790514978s) [noise=true , address=77.90.31.13:4627 , piece_id=6QHJQ3SE47DC5UYSAZBPDDW7FYMD6V3HIN5MZ6MOHD4GBYJ6PU6Q, seg_pos={0 0}, upload=515086]: Uploading piece... Dec 30 03:03:42.920977887 (587h34m42.792060922s) [noise=true , address=145.79.227.3:4151 , piece_id=GCMHNSC6JQLJAB3L5PVZFYD4UKMPXZKEWJEOVL6VAFRNELPH7XUA, seg_pos={0 0}, upload=515086]: Uploading piece... Dec 30 03:03:42.922352504 (587h34m42.793435539s) [noise=true , address=145.79.227.14:4837 , piece_id=GJKO67C2D34PEWE7N6E4UDXGK6X2SKZ5CELLKZTV7XMG6TFFQRYQ, seg_pos={0 0}, upload=515086]: Uploading piece... Dec 30 03:03:42.923837903 (587h34m42.794920948s) [noise=true , address=145.79.227.44:4193 , piece_id=XWVAX5Y53A55ZQWOCOL6E5AZGJPBDVXMKTZCPT46J7KT6DN7ATTQ, seg_pos={0 0}, upload=515086]: Uploading piece... Dec 30 03:03:42.925195508 (587h34m42.796278553s) [noise=true , address=145.79.227.21:4475 , piece_id=W2KUQ5YXLWPXDCKDKLEYSU57Z37NT3K3V6FOLW2543S66RQRC2MA, seg_pos={0 0}, upload=515086]: Uploading piece... Dec 30 03:03:42.925309184 (587h34m42.796392229s) [noise=true , address=145.79.227.52:4445 , piece_id=SCEZ4SNBWWKSJYXDACVUEUW3VGGXDXDAO3PD2K2ILSKW6FNBSM2A, seg_pos={0 0}, upload=515086]: Uploading piece... Dec 30 03:03:42.927045460 (587h34m42.798128506s) [noise=true , address=45.91.116.5:4300 , piece_id=ODDRH7AM5TANWKSYGLESZU5LUACCPVU7TI2PV7PPOZUHGBGNT35A, seg_pos={0 0}, upload=515086]: Uploading piece... Dec 30 03:03:42.927814815 (587h34m42.79889786s) [noise=true , address=45.91.116.10:4083 , piece_id=L37FM6KLC2IOPOB6QLI4WYNXNWK2DIZPZXYGDPRT4P6ZBO4RLO4Q, seg_pos={0 0}, upload=515086]: Uploading piece... Dec 30 03:03:42.928481585 (587h34m42.79956462s) [noise=true , address=45.91.116.8:4048 , piece_id=FAFGXWER5ICDY4BA7LTTFT34BXPY2MYWYOIYRLRHE3LPSKBQMK7Q, seg_pos={0 0}, upload=515086]: Uploading piece... Dec 30 03:03:42.929498541 (587h34m42.800581576s) [noise=true , address=145.79.227.10:4401 , piece_id=IF3T5AIHAP5SLDU4BL6A7KYM32MHVFDNTPU7ADI5UOUYL72FPPBQ, seg_pos={0 0}, upload=515086]: Uploading piece... Dec 30 03:03:42.929784767 (587h34m42.800867802s) [noise=true , address=77.90.31.12:4590 , piece_id=AIURUSSEUNDHTXREDBQZBRMBVRZBERPRNERXEOIPYYJZQZ7XOD7A, seg_pos={0 0}, upload=515086]: Uploading piece... Dec 30 03:03:44.609888847 (587h34m44.480971882s) [upload=515086]: Next returned nil segment Dec 30 03:03:44.609903665 (587h34m44.4809867s) [upload=515086]: Waiting for error group managing segments... Dec 30 03:03:44.889262778 (587h34m44.760345813s) [noise=true , address=145.79.227.3:4151 , piece_id=GCMHNSC6JQLJAB3L5PVZFYD4UKMPXZKEWJEOVL6VAFRNELPH7XUA, seg_pos={0 0}, upload=515086]: Done uploading piece. err: <nil> Dec 30 03:03:44.905294826 (587h34m44.776377861s) [noise=true , address=145.79.227.13:4803 , piece_id=2NFC5GJ7FPRYQ6QVGE7CMIXNHEB4VDB2TAWWSQA5LM5RZUDMLKBA, seg_pos={0 0}, upload=515086]: Done uploading piece. err: <nil> Dec 30 03:03:44.911669865 (587h34m44.7827529s) [noise=true , address=145.79.227.21:4475 , piece_id=W2KUQ5YXLWPXDCKDKLEYSU57Z37NT3K3V6FOLW2543S66RQRC2MA, seg_pos={0 0}, upload=515086]: Done uploading piece. err: <nil> Dec 30 03:03:44.913028953 (587h34m44.784111988s) [noise=true , address=145.79.227.14:4837 , piece_id=GJKO67C2D34PEWE7N6E4UDXGK6X2SKZ5CELLKZTV7XMG6TFFQRYQ, seg_pos={0 0}, upload=515086]: Done uploading piece. err: <nil> Dec 30 03:03:44.923221862 (587h34m44.794304897s) [noise=true , address=145.79.227.52:4445 , piece_id=SCEZ4SNBWWKSJYXDACVUEUW3VGGXDXDAO3PD2K2ILSKW6FNBSM2A, seg_pos={0 0}, upload=515086]: Done uploading piece. err: <nil> Dec 30 03:03:44.929698846 (587h34m44.800781891s) [noise=true , address=145.79.227.15:4873 , piece_id=DOPL3LC6OXAYLCTIZWBUBY2EOY2EYEV46NG4AHHS776I653WX25A, seg_pos={0 0}, upload=515086]: Done uploading piece. err: <nil> Dec 30 03:03:44.933930954 (587h34m44.805013999s) [noise=true , address=145.79.227.10:4401 , piece_id=IF3T5AIHAP5SLDU4BL6A7KYM32MHVFDNTPU7ADI5UOUYL72FPPBQ, seg_pos={0 0}, upload=515086]: Done uploading piece. err: <nil> Dec 30 03:03:45.105306018 (587h34m44.976389053s) [noise=true , address=77.90.31.13:4627 , piece_id=6QHJQ3SE47DC5UYSAZBPDDW7FYMD6V3HIN5MZ6MOHD4GBYJ6PU6Q, seg_pos={0 0}, upload=515086]: Done uploading piece. err: <nil> Dec 30 03:03:45.110644804 (587h34m44.981727849s) [noise=true , address=77.90.31.12:4590 , piece_id=AIURUSSEUNDHTXREDBQZBRMBVRZBERPRNERXEOIPYYJZQZ7XOD7A, seg_pos={0 0}, upload=515086]: Done uploading piece. err: <nil> Dec 30 03:03:45.180587856 (587h34m45.051670901s) [noise=true , address=45.91.116.5:4300 , piece_id=ODDRH7AM5TANWKSYGLESZU5LUACCPVU7TI2PV7PPOZUHGBGNT35A, seg_pos={0 0}, upload=515086]: Done uploading piece. err: <nil> Dec 30 03:03:45.199847840 (587h34m45.070930875s) [noise=true , address=45.91.116.10:4083 , piece_id=L37FM6KLC2IOPOB6QLI4WYNXNWK2DIZPZXYGDPRT4P6ZBO4RLO4Q, seg_pos={0 0}, upload=515086]: Done uploading piece. err: <nil> Dec 30 03:03:45.903412778 (587h34m45.774495823s) [noise=true , address=45.91.116.8:4048 , piece_id=FAFGXWER5ICDY4BA7LTTFT34BXPY2MYWYOIYRLRHE3LPSKBQMK7Q, seg_pos={0 0}, upload=515086]: Done uploading piece. err: <nil> Dec 30 03:03:46.235528288 (587h34m46.106611323s) [noise=true , address=145.79.227.34:4906 , piece_id=GXI6LC45KZ5UA4KPWMXZJ66MA7RUPM4YDX6E2LSNQ7DABUCFFVLA, seg_pos={0 0}, upload=515086]: Done uploading piece. err: <nil> Dec 30 03:03:46.235553807 (587h34m46.106636862s) [seg_pos={0 0}, upload=515086]: Segment reached optimal threshold of 13 pieces. Dec 30 03:03:46.235780979 (587h34m46.106864024s) [noise=true , address=45.91.116.11:4119 , piece_id=KEZ2CIX3PCLO4MPD367AEIY5B4G7KGGDBVJKATLDPMTRT5RDBSPQ, seg_pos={0 0}, upload=515086]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 30 03:03:46.235790848 (587h34m46.106873883s) [noise=true , address=45.91.116.11:4119 , piece_id=KEZ2CIX3PCLO4MPD367AEIY5B4G7KGGDBVJKATLDPMTRT5RDBSPQ, seg_pos={0 0}, upload=515086]: Long tail cancellation due to optimal threshold Dec 30 03:03:46.236012901 (587h34m46.107095966s) [noise=true , address=145.79.227.44:4193 , piece_id=XWVAX5Y53A55ZQWOCOL6E5AZGJPBDVXMKTZCPT46J7KT6DN7ATTQ, seg_pos={0 0}, upload=515086]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 30 03:03:46.236024332 (587h34m46.107107377s) [noise=true , address=145.79.227.44:4193 , piece_id=XWVAX5Y53A55ZQWOCOL6E5AZGJPBDVXMKTZCPT46J7KT6DN7ATTQ, seg_pos={0 0}, upload=515086]: Long tail cancellation due to optimal threshold Dec 30 03:03:46.236088344 (587h34m46.107171379s) [noise=true , address=145.79.227.5:4222 , piece_id=TLYHW5ZI4A7BNPGD46HWFTNEWGWEO4DWI455GNSCBXJUIQY4FA2Q, seg_pos={0 0}, upload=515086]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: expected piece hash; context canceled; EOF Dec 30 03:03:46.236104355 (587h34m46.1071874s) [noise=true , address=145.79.227.5:4222 , piece_id=TLYHW5ZI4A7BNPGD46HWFTNEWGWEO4DWI455GNSCBXJUIQY4FA2Q, seg_pos={0 0}, upload=515086]: Long tail cancellation due to optimal threshold Dec 30 03:03:46.236165651 (587h34m46.107248686s) [seg_pos={0 0}, upload=515086]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 03:03:46.236274089 (587h34m46.107357134s) [upload=515086]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 03:03:46.236280160 (587h34m46.107363195s) [upload=515086]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 03:03:46.536344670 (587h34m46.407427715s) [upload=515086]: Done uploading segments... Dec 30 03:03:46.536367172 (587h34m46.407450217s) [upload=515086]: Upload finished. err: <nil> Duration: 6.034387713s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......X.............X................X......X..............] duration: 28.888749107s trace: https://jaeger.dev.storj.io/trace/6c3caa8adcfacbb9 FAIL (2025-12-30 03:05:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== true , address=145.79.227.35:4496 , piece_id=5X54ZGU4OIVI3DT622QE7IL66SVFLZYUICPODGMDKN7U4D4OJNSA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.040303259 (587h24m33.911386294s) [noise=true , address=77.90.31.12:4601 , piece_id=QQFK4PCL27YYF3UATGLV5KDENRNX6NHE7RXAZ2K66CC3WLHEBMWQ, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.040670117 (587h24m33.911753152s) [noise=true , address=77.90.31.21:4881 , piece_id=EMV4LHFK2VRHSW6PD3BBYGOXDAAYJKX6GEOXGGLOT3IXWNDQMU4A, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041006247 (587h24m33.912089292s) [noise=true , address=145.79.227.52:4472 , piece_id=DT2DAIC4ZI3WL7OGY55ZTTXJEI4IDRXPSJIQA3LTRC7K2F6EVPCA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041183314 (587h24m33.912266349s) [noise=true , address=145.79.227.26:4674 , piece_id=3BAN2ZXOFTSGZITYYHMZUS4AD35X6ZBOYA2HKOJCJI7VBWPJ3LSA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041313653 (587h24m33.912396688s) [noise=true , address=145.79.227.51:4429 , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041535194 (587h24m33.912618229s) [noise=true , address=145.79.227.33:4885 , piece_id=LA755L4RA4VJBHNCQBXOCFROZA3W6W6PUPKGJ5J772WAN44YMY2A, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041639693 (587h24m33.912722728s) [noise=true , address=145.79.227.43:4187 , piece_id=CAAFD3XIWWHZN7L7SYDMMSC5G5RKRLFYYHQZEEN4GAC2D6Y2EARA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041877216 (587h24m33.912960261s) [noise=true , address=145.79.227.46:4623 , piece_id=V4TEM7RRTTGBW5ISZOEAOPFIOZZ5EABO7BPWNRH5U5UPM2X7554Q, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041986634 (587h24m33.913069669s) [noise=true , address=45.91.116.13:4176 , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.042345657 (587h24m33.913428692s) [noise=true , address=45.91.116.10:4104 , piece_id=MRIWH6WGACE4UDNX55MIY6ZDWBAHS6E5KQS4QCOQN2BU7JI7ODCA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.042362019 (587h24m33.913445054s) [noise=true , address=77.90.31.11:4573 , piece_id=TORCC4W7DS5TJ7H7NWONEPNJOFULSZ6F5OHOP4BEVMGEH2MJA4DQ, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.042684192 (587h24m33.913767237s) [noise=true , address=145.79.227.49:4340 , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.042707327 (587h24m33.913790362s) [noise=true , address=145.79.227.32:4837 , piece_id=SQTHTF5KWZOKFYX6CGL6KCDRDAY7OJY6NA5OM3N4YCICDPM6DKXA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.043091528 (587h24m33.914174563s) [noise=true , address=45.91.116.5:4309 , piece_id=Y5HCMU6Z2FPBV3BNTUOC4F6JJWKNBKPOXWFVAFQTXUKQNHTLA4TA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.043544771 (587h24m33.914627816s) [noise=true , address=145.79.227.40:4053 , piece_id=3IZP7JBGSWBRBQHMJWXIDD73B5R3COWU5H3IYFGVBCTTLW7OUJ5A, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:35.347637489 (587h24m35.218720524s) [upload=514937]: Next returned nil segment Dec 30 02:53:35.347733591 (587h24m35.218816626s) [upload=514937]: Waiting for error group managing segments... Dec 30 02:53:35.674169656 (587h24m35.545252721s) [noise=true , address=64.49.13.27:4959 , piece_id=B4UZ67DIWNRHV77VTQONS6ZBUI2BPDDBM3ZS6FVCTABQZ6OHRJJA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:35.891295347 (587h24m35.762378382s) [noise=true , address=145.79.227.32:4837 , piece_id=SQTHTF5KWZOKFYX6CGL6KCDRDAY7OJY6NA5OM3N4YCICDPM6DKXA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:35.912619632 (587h24m35.783702677s) [noise=true , address=145.79.227.26:4674 , piece_id=3BAN2ZXOFTSGZITYYHMZUS4AD35X6ZBOYA2HKOJCJI7VBWPJ3LSA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:35.913054741 (587h24m35.784137786s) [noise=true , address=145.79.227.46:4623 , piece_id=V4TEM7RRTTGBW5ISZOEAOPFIOZZ5EABO7BPWNRH5U5UPM2X7554Q, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:35.913485270 (587h24m35.784568305s) [noise=true , address=145.79.227.35:4496 , piece_id=5X54ZGU4OIVI3DT622QE7IL66SVFLZYUICPODGMDKN7U4D4OJNSA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:35.923790864 (587h24m35.794873909s) [noise=true , address=145.79.227.52:4472 , piece_id=DT2DAIC4ZI3WL7OGY55ZTTXJEI4IDRXPSJIQA3LTRC7K2F6EVPCA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.109605534 (587h24m35.980688569s) [noise=true , address=77.90.31.21:4881 , piece_id=EMV4LHFK2VRHSW6PD3BBYGOXDAAYJKX6GEOXGGLOT3IXWNDQMU4A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.122502652 (587h24m35.993585697s) [noise=true , address=77.90.31.11:4573 , piece_id=TORCC4W7DS5TJ7H7NWONEPNJOFULSZ6F5OHOP4BEVMGEH2MJA4DQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.151131466 (587h24m36.022214511s) [noise=true , address=45.91.116.10:4104 , piece_id=MRIWH6WGACE4UDNX55MIY6ZDWBAHS6E5KQS4QCOQN2BU7JI7ODCA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.159562149 (587h24m36.030645184s) [noise=true , address=77.90.31.12:4601 , piece_id=QQFK4PCL27YYF3UATGLV5KDENRNX6NHE7RXAZ2K66CC3WLHEBMWQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.238339784 (587h24m36.10942282s) [noise=true , address=45.91.116.5:4309 , piece_id=Y5HCMU6Z2FPBV3BNTUOC4F6JJWKNBKPOXWFVAFQTXUKQNHTLA4TA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.813503456 (587h24m36.684586501s) [noise=true , address=145.79.227.33:4885 , piece_id=LA755L4RA4VJBHNCQBXOCFROZA3W6W6PUPKGJ5J772WAN44YMY2A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.826964278 (587h24m36.698047313s) [noise=true , address=145.79.227.43:4187 , piece_id=CAAFD3XIWWHZN7L7SYDMMSC5G5RKRLFYYHQZEEN4GAC2D6Y2EARA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.828562510 (587h24m36.699645545s) [noise=true , address=145.79.227.40:4053 , piece_id=3IZP7JBGSWBRBQHMJWXIDD73B5R3COWU5H3IYFGVBCTTLW7OUJ5A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.828575566 (587h24m36.699658601s) [seg_pos={0 0}, upload=514937]: Segment reached optimal threshold of 14 pieces. Dec 30 02:53:36.828752202 (587h24m36.699835237s) [noise=true , address=145.79.227.30:4780 , piece_id=DZQ5LIBMWXPJ4XXWORRDLQIAFXJDNDQEEBRMSBS44UX3D2HPTSWQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125WCbZ6a2NaewiLAHhHuPo91sQEfC9CZzEBV6rwF1JY7z13AEY): protocol: expected piece hash; context canceled; EOF Dec 30 02:53:36.828758714 (587h24m36.699841749s) [noise=true , address=145.79.227.30:4780 , piece_id=DZQ5LIBMWXPJ4XXWORRDLQIAFXJDNDQEEBRMSBS44UX3D2HPTSWQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold Dec 30 02:53:36.828762672 (587h24m36.699845707s) [noise=true , address=145.79.227.51:4429 , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qytmzwy7sTmWXskEbLWsWMniPz55zgho2akE7oh6ahAiT5SXf): protocol: write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection; write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection Dec 30 02:53:36.828774034 (587h24m36.699857069s) [noise=true , address=145.79.227.51:4429 , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold Dec 30 02:53:36.828889934 (587h24m36.699972969s) [noise=true , address=45.91.116.13:4176 , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ejsmBd4tAVwTHjamPYbryQb5x2vDnk9pBuX3CBF9WYFhTrFVJ): protocol: expected piece hash; context canceled; EOF Dec 30 02:53:36.828895144 (587h24m36.699978179s) [noise=true , address=45.91.116.13:4176 , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold Dec 30 02:53:36.828994173 (587h24m36.700077208s) [noise=true , address=145.79.227.49:4340 , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF Dec 30 02:53:36.828999793 (587h24m36.700082828s) [noise=true , address=145.79.227.49:4340 , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold Dec 30 02:53:36.829030332 (587h24m36.700113367s) [seg_pos={0 0}, upload=514937]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 02:53:36.829095486 (587h24m36.700178621s) [upload=514937]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 02:53:36.829100105 (587h24m36.70018315s) [upload=514937]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 02:53:37.128695270 (587h24m36.999778315s) [upload=514937]: Done uploading segments... Dec 30 02:53:37.128717683 (587h24m36.999800728s) [upload=514937]: Upload finished. err: <nil> Duration: 28.888762602s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [...................................X...X.X.XX...............] duration: 4.432723185s trace: https://jaeger.dev.storj.io/trace/14b283cf2d9b0313 FAIL (2025-12-30 03:05:12 UTC) orbiter warning: context deadline exceeded Duration: 4.432785062s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [........X.............X................X......X.............] duration: 44.11939617s trace: https://jaeger.dev.storj.io/trace/6c3caa8adcfacbb9 FAIL (2025-12-30 03:04:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== true , address=145.79.227.35:4496 , piece_id=5X54ZGU4OIVI3DT622QE7IL66SVFLZYUICPODGMDKN7U4D4OJNSA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.040303259 (587h24m33.911386294s) [noise=true , address=77.90.31.12:4601 , piece_id=QQFK4PCL27YYF3UATGLV5KDENRNX6NHE7RXAZ2K66CC3WLHEBMWQ, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.040670117 (587h24m33.911753152s) [noise=true , address=77.90.31.21:4881 , piece_id=EMV4LHFK2VRHSW6PD3BBYGOXDAAYJKX6GEOXGGLOT3IXWNDQMU4A, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041006247 (587h24m33.912089292s) [noise=true , address=145.79.227.52:4472 , piece_id=DT2DAIC4ZI3WL7OGY55ZTTXJEI4IDRXPSJIQA3LTRC7K2F6EVPCA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041183314 (587h24m33.912266349s) [noise=true , address=145.79.227.26:4674 , piece_id=3BAN2ZXOFTSGZITYYHMZUS4AD35X6ZBOYA2HKOJCJI7VBWPJ3LSA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041313653 (587h24m33.912396688s) [noise=true , address=145.79.227.51:4429 , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041535194 (587h24m33.912618229s) [noise=true , address=145.79.227.33:4885 , piece_id=LA755L4RA4VJBHNCQBXOCFROZA3W6W6PUPKGJ5J772WAN44YMY2A, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041639693 (587h24m33.912722728s) [noise=true , address=145.79.227.43:4187 , piece_id=CAAFD3XIWWHZN7L7SYDMMSC5G5RKRLFYYHQZEEN4GAC2D6Y2EARA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041877216 (587h24m33.912960261s) [noise=true , address=145.79.227.46:4623 , piece_id=V4TEM7RRTTGBW5ISZOEAOPFIOZZ5EABO7BPWNRH5U5UPM2X7554Q, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041986634 (587h24m33.913069669s) [noise=true , address=45.91.116.13:4176 , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.042345657 (587h24m33.913428692s) [noise=true , address=45.91.116.10:4104 , piece_id=MRIWH6WGACE4UDNX55MIY6ZDWBAHS6E5KQS4QCOQN2BU7JI7ODCA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.042362019 (587h24m33.913445054s) [noise=true , address=77.90.31.11:4573 , piece_id=TORCC4W7DS5TJ7H7NWONEPNJOFULSZ6F5OHOP4BEVMGEH2MJA4DQ, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.042684192 (587h24m33.913767237s) [noise=true , address=145.79.227.49:4340 , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.042707327 (587h24m33.913790362s) [noise=true , address=145.79.227.32:4837 , piece_id=SQTHTF5KWZOKFYX6CGL6KCDRDAY7OJY6NA5OM3N4YCICDPM6DKXA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.043091528 (587h24m33.914174563s) [noise=true , address=45.91.116.5:4309 , piece_id=Y5HCMU6Z2FPBV3BNTUOC4F6JJWKNBKPOXWFVAFQTXUKQNHTLA4TA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.043544771 (587h24m33.914627816s) [noise=true , address=145.79.227.40:4053 , piece_id=3IZP7JBGSWBRBQHMJWXIDD73B5R3COWU5H3IYFGVBCTTLW7OUJ5A, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:35.347637489 (587h24m35.218720524s) [upload=514937]: Next returned nil segment Dec 30 02:53:35.347733591 (587h24m35.218816626s) [upload=514937]: Waiting for error group managing segments... Dec 30 02:53:35.674169656 (587h24m35.545252721s) [noise=true , address=64.49.13.27:4959 , piece_id=B4UZ67DIWNRHV77VTQONS6ZBUI2BPDDBM3ZS6FVCTABQZ6OHRJJA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:35.891295347 (587h24m35.762378382s) [noise=true , address=145.79.227.32:4837 , piece_id=SQTHTF5KWZOKFYX6CGL6KCDRDAY7OJY6NA5OM3N4YCICDPM6DKXA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:35.912619632 (587h24m35.783702677s) [noise=true , address=145.79.227.26:4674 , piece_id=3BAN2ZXOFTSGZITYYHMZUS4AD35X6ZBOYA2HKOJCJI7VBWPJ3LSA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:35.913054741 (587h24m35.784137786s) [noise=true , address=145.79.227.46:4623 , piece_id=V4TEM7RRTTGBW5ISZOEAOPFIOZZ5EABO7BPWNRH5U5UPM2X7554Q, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:35.913485270 (587h24m35.784568305s) [noise=true , address=145.79.227.35:4496 , piece_id=5X54ZGU4OIVI3DT622QE7IL66SVFLZYUICPODGMDKN7U4D4OJNSA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:35.923790864 (587h24m35.794873909s) [noise=true , address=145.79.227.52:4472 , piece_id=DT2DAIC4ZI3WL7OGY55ZTTXJEI4IDRXPSJIQA3LTRC7K2F6EVPCA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.109605534 (587h24m35.980688569s) [noise=true , address=77.90.31.21:4881 , piece_id=EMV4LHFK2VRHSW6PD3BBYGOXDAAYJKX6GEOXGGLOT3IXWNDQMU4A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.122502652 (587h24m35.993585697s) [noise=true , address=77.90.31.11:4573 , piece_id=TORCC4W7DS5TJ7H7NWONEPNJOFULSZ6F5OHOP4BEVMGEH2MJA4DQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.151131466 (587h24m36.022214511s) [noise=true , address=45.91.116.10:4104 , piece_id=MRIWH6WGACE4UDNX55MIY6ZDWBAHS6E5KQS4QCOQN2BU7JI7ODCA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.159562149 (587h24m36.030645184s) [noise=true , address=77.90.31.12:4601 , piece_id=QQFK4PCL27YYF3UATGLV5KDENRNX6NHE7RXAZ2K66CC3WLHEBMWQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.238339784 (587h24m36.10942282s) [noise=true , address=45.91.116.5:4309 , piece_id=Y5HCMU6Z2FPBV3BNTUOC4F6JJWKNBKPOXWFVAFQTXUKQNHTLA4TA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.813503456 (587h24m36.684586501s) [noise=true , address=145.79.227.33:4885 , piece_id=LA755L4RA4VJBHNCQBXOCFROZA3W6W6PUPKGJ5J772WAN44YMY2A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.826964278 (587h24m36.698047313s) [noise=true , address=145.79.227.43:4187 , piece_id=CAAFD3XIWWHZN7L7SYDMMSC5G5RKRLFYYHQZEEN4GAC2D6Y2EARA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.828562510 (587h24m36.699645545s) [noise=true , address=145.79.227.40:4053 , piece_id=3IZP7JBGSWBRBQHMJWXIDD73B5R3COWU5H3IYFGVBCTTLW7OUJ5A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.828575566 (587h24m36.699658601s) [seg_pos={0 0}, upload=514937]: Segment reached optimal threshold of 14 pieces. Dec 30 02:53:36.828752202 (587h24m36.699835237s) [noise=true , address=145.79.227.30:4780 , piece_id=DZQ5LIBMWXPJ4XXWORRDLQIAFXJDNDQEEBRMSBS44UX3D2HPTSWQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125WCbZ6a2NaewiLAHhHuPo91sQEfC9CZzEBV6rwF1JY7z13AEY): protocol: expected piece hash; context canceled; EOF Dec 30 02:53:36.828758714 (587h24m36.699841749s) [noise=true , address=145.79.227.30:4780 , piece_id=DZQ5LIBMWXPJ4XXWORRDLQIAFXJDNDQEEBRMSBS44UX3D2HPTSWQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold Dec 30 02:53:36.828762672 (587h24m36.699845707s) [noise=true , address=145.79.227.51:4429 , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qytmzwy7sTmWXskEbLWsWMniPz55zgho2akE7oh6ahAiT5SXf): protocol: write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection; write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection Dec 30 02:53:36.828774034 (587h24m36.699857069s) [noise=true , address=145.79.227.51:4429 , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold Dec 30 02:53:36.828889934 (587h24m36.699972969s) [noise=true , address=45.91.116.13:4176 , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ejsmBd4tAVwTHjamPYbryQb5x2vDnk9pBuX3CBF9WYFhTrFVJ): protocol: expected piece hash; context canceled; EOF Dec 30 02:53:36.828895144 (587h24m36.699978179s) [noise=true , address=45.91.116.13:4176 , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold Dec 30 02:53:36.828994173 (587h24m36.700077208s) [noise=true , address=145.79.227.49:4340 , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF Dec 30 02:53:36.828999793 (587h24m36.700082828s) [noise=true , address=145.79.227.49:4340 , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold Dec 30 02:53:36.829030332 (587h24m36.700113367s) [seg_pos={0 0}, upload=514937]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 02:53:36.829095486 (587h24m36.700178621s) [upload=514937]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 02:53:36.829100105 (587h24m36.70018315s) [upload=514937]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 02:53:37.128695270 (587h24m36.999778315s) [upload=514937]: Done uploading segments... Dec 30 02:53:37.128717683 (587h24m36.999800728s) [upload=514937]: Upload finished. err: <nil> Duration: 44.119425896s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [X...................................X...X.X.XX..............] duration: 4.186850598s trace: https://jaeger.dev.storj.io/trace/14b283cf2d9b0313 FAIL (2025-12-30 03:04:31 UTC) orbiter warning: context deadline exceeded Duration: 4.18688294s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [......X........X...X..XX..X.X......XXX.X.X..X........X.X.XX.] duration: 6.200029154s trace: https://jaeger.dev.storj.io/trace/1f609d54324f4aba FAIL (2025-12-30 03:04:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 03:03:42.664733818 (587h34m42.535816853s) [upload=515086]: Starting upload Dec 30 03:03:42.664803922 (587h34m42.535886957s) [upload=515086]: Done starting upload Dec 30 03:03:42.664926486 (587h34m42.536009531s) [upload=515086]: Uploading segments... Dec 30 03:03:42.665104645 (587h34m42.53618769s) [upload=515086]: Got next segment. Inline: false Dec 30 03:03:42.665157715 (587h34m42.53624077s) [upload=515086]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 03:03:42.665164197 (587h34m42.536247242s) [upload=515086]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 03:03:42.916348402 (587h34m42.787431447s) [seg_pos={0 0}, upload=515086]: Begin upload segment... Dec 30 03:03:42.916399158 (587h34m42.787482203s) [seg_pos={0 0}, upload=515086]: Done begin upload segment. Dec 30 03:03:42.916461577 (587h34m42.787544612s) [noise=true , address=145.79.227.5:4222 , piece_id=TLYHW5ZI4A7BNPGD46HWFTNEWGWEO4DWI455GNSCBXJUIQY4FA2Q, seg_pos={0 0}, upload=515086]: Uploading piece... Dec 30 03:03:42.916489440 (587h34m42.787572475s) [noise=true , address=45.91.116.11:4119 , piece_id=KEZ2CIX3PCLO4MPD367AEIY5B4G7KGGDBVJKATLDPMTRT5RDBSPQ, seg_pos={0 0}, upload=515086]: Uploading piece... Dec 30 03:03:42.917030490 (587h34m42.788113525s) [noise=true , address=145.79.227.34:4906 , piece_id=GXI6LC45KZ5UA4KPWMXZJ66MA7RUPM4YDX6E2LSNQ7DABUCFFVLA, seg_pos={0 0}, upload=515086]: Uploading piece... Dec 30 03:03:42.917755180 (587h34m42.788838215s) [noise=true , address=145.79.227.13:4803 , piece_id=2NFC5GJ7FPRYQ6QVGE7CMIXNHEB4VDB2TAWWSQA5LM5RZUDMLKBA, seg_pos={0 0}, upload=515086]: Uploading piece... Dec 30 03:03:42.918253569 (587h34m42.789336604s) [noise=true , address=145.79.227.15:4873 , piece_id=DOPL3LC6OXAYLCTIZWBUBY2EOY2EYEV46NG4AHHS776I653WX25A, seg_pos={0 0}, upload=515086]: Uploading piece... Dec 30 03:03:42.919431933 (587h34m42.790514978s) [noise=true , address=77.90.31.13:4627 , piece_id=6QHJQ3SE47DC5UYSAZBPDDW7FYMD6V3HIN5MZ6MOHD4GBYJ6PU6Q, seg_pos={0 0}, upload=515086]: Uploading piece... Dec 30 03:03:42.920977887 (587h34m42.792060922s) [noise=true , address=145.79.227.3:4151 , piece_id=GCMHNSC6JQLJAB3L5PVZFYD4UKMPXZKEWJEOVL6VAFRNELPH7XUA, seg_pos={0 0}, upload=515086]: Uploading piece... Dec 30 03:03:42.922352504 (587h34m42.793435539s) [noise=true , address=145.79.227.14:4837 , piece_id=GJKO67C2D34PEWE7N6E4UDXGK6X2SKZ5CELLKZTV7XMG6TFFQRYQ, seg_pos={0 0}, upload=515086]: Uploading piece... Dec 30 03:03:42.923837903 (587h34m42.794920948s) [noise=true , address=145.79.227.44:4193 , piece_id=XWVAX5Y53A55ZQWOCOL6E5AZGJPBDVXMKTZCPT46J7KT6DN7ATTQ, seg_pos={0 0}, upload=515086]: Uploading piece... Dec 30 03:03:42.925195508 (587h34m42.796278553s) [noise=true , address=145.79.227.21:4475 , piece_id=W2KUQ5YXLWPXDCKDKLEYSU57Z37NT3K3V6FOLW2543S66RQRC2MA, seg_pos={0 0}, upload=515086]: Uploading piece... Dec 30 03:03:42.925309184 (587h34m42.796392229s) [noise=true , address=145.79.227.52:4445 , piece_id=SCEZ4SNBWWKSJYXDACVUEUW3VGGXDXDAO3PD2K2ILSKW6FNBSM2A, seg_pos={0 0}, upload=515086]: Uploading piece... Dec 30 03:03:42.927045460 (587h34m42.798128506s) [noise=true , address=45.91.116.5:4300 , piece_id=ODDRH7AM5TANWKSYGLESZU5LUACCPVU7TI2PV7PPOZUHGBGNT35A, seg_pos={0 0}, upload=515086]: Uploading piece... Dec 30 03:03:42.927814815 (587h34m42.79889786s) [noise=true , address=45.91.116.10:4083 , piece_id=L37FM6KLC2IOPOB6QLI4WYNXNWK2DIZPZXYGDPRT4P6ZBO4RLO4Q, seg_pos={0 0}, upload=515086]: Uploading piece... Dec 30 03:03:42.928481585 (587h34m42.79956462s) [noise=true , address=45.91.116.8:4048 , piece_id=FAFGXWER5ICDY4BA7LTTFT34BXPY2MYWYOIYRLRHE3LPSKBQMK7Q, seg_pos={0 0}, upload=515086]: Uploading piece... Dec 30 03:03:42.929498541 (587h34m42.800581576s) [noise=true , address=145.79.227.10:4401 , piece_id=IF3T5AIHAP5SLDU4BL6A7KYM32MHVFDNTPU7ADI5UOUYL72FPPBQ, seg_pos={0 0}, upload=515086]: Uploading piece... Dec 30 03:03:42.929784767 (587h34m42.800867802s) [noise=true , address=77.90.31.12:4590 , piece_id=AIURUSSEUNDHTXREDBQZBRMBVRZBERPRNERXEOIPYYJZQZ7XOD7A, seg_pos={0 0}, upload=515086]: Uploading piece... Dec 30 03:03:44.609888847 (587h34m44.480971882s) [upload=515086]: Next returned nil segment Dec 30 03:03:44.609903665 (587h34m44.4809867s) [upload=515086]: Waiting for error group managing segments... Dec 30 03:03:44.889262778 (587h34m44.760345813s) [noise=true , address=145.79.227.3:4151 , piece_id=GCMHNSC6JQLJAB3L5PVZFYD4UKMPXZKEWJEOVL6VAFRNELPH7XUA, seg_pos={0 0}, upload=515086]: Done uploading piece. err: <nil> Dec 30 03:03:44.905294826 (587h34m44.776377861s) [noise=true , address=145.79.227.13:4803 , piece_id=2NFC5GJ7FPRYQ6QVGE7CMIXNHEB4VDB2TAWWSQA5LM5RZUDMLKBA, seg_pos={0 0}, upload=515086]: Done uploading piece. err: <nil> Dec 30 03:03:44.911669865 (587h34m44.7827529s) [noise=true , address=145.79.227.21:4475 , piece_id=W2KUQ5YXLWPXDCKDKLEYSU57Z37NT3K3V6FOLW2543S66RQRC2MA, seg_pos={0 0}, upload=515086]: Done uploading piece. err: <nil> Dec 30 03:03:44.913028953 (587h34m44.784111988s) [noise=true , address=145.79.227.14:4837 , piece_id=GJKO67C2D34PEWE7N6E4UDXGK6X2SKZ5CELLKZTV7XMG6TFFQRYQ, seg_pos={0 0}, upload=515086]: Done uploading piece. err: <nil> Dec 30 03:03:44.923221862 (587h34m44.794304897s) [noise=true , address=145.79.227.52:4445 , piece_id=SCEZ4SNBWWKSJYXDACVUEUW3VGGXDXDAO3PD2K2ILSKW6FNBSM2A, seg_pos={0 0}, upload=515086]: Done uploading piece. err: <nil> Dec 30 03:03:44.929698846 (587h34m44.800781891s) [noise=true , address=145.79.227.15:4873 , piece_id=DOPL3LC6OXAYLCTIZWBUBY2EOY2EYEV46NG4AHHS776I653WX25A, seg_pos={0 0}, upload=515086]: Done uploading piece. err: <nil> Dec 30 03:03:44.933930954 (587h34m44.805013999s) [noise=true , address=145.79.227.10:4401 , piece_id=IF3T5AIHAP5SLDU4BL6A7KYM32MHVFDNTPU7ADI5UOUYL72FPPBQ, seg_pos={0 0}, upload=515086]: Done uploading piece. err: <nil> Dec 30 03:03:45.105306018 (587h34m44.976389053s) [noise=true , address=77.90.31.13:4627 , piece_id=6QHJQ3SE47DC5UYSAZBPDDW7FYMD6V3HIN5MZ6MOHD4GBYJ6PU6Q, seg_pos={0 0}, upload=515086]: Done uploading piece. err: <nil> Dec 30 03:03:45.110644804 (587h34m44.981727849s) [noise=true , address=77.90.31.12:4590 , piece_id=AIURUSSEUNDHTXREDBQZBRMBVRZBERPRNERXEOIPYYJZQZ7XOD7A, seg_pos={0 0}, upload=515086]: Done uploading piece. err: <nil> Dec 30 03:03:45.180587856 (587h34m45.051670901s) [noise=true , address=45.91.116.5:4300 , piece_id=ODDRH7AM5TANWKSYGLESZU5LUACCPVU7TI2PV7PPOZUHGBGNT35A, seg_pos={0 0}, upload=515086]: Done uploading piece. err: <nil> Dec 30 03:03:45.199847840 (587h34m45.070930875s) [noise=true , address=45.91.116.10:4083 , piece_id=L37FM6KLC2IOPOB6QLI4WYNXNWK2DIZPZXYGDPRT4P6ZBO4RLO4Q, seg_pos={0 0}, upload=515086]: Done uploading piece. err: <nil> Dec 30 03:03:45.903412778 (587h34m45.774495823s) [noise=true , address=45.91.116.8:4048 , piece_id=FAFGXWER5ICDY4BA7LTTFT34BXPY2MYWYOIYRLRHE3LPSKBQMK7Q, seg_pos={0 0}, upload=515086]: Done uploading piece. err: <nil> Dec 30 03:03:46.235528288 (587h34m46.106611323s) [noise=true , address=145.79.227.34:4906 , piece_id=GXI6LC45KZ5UA4KPWMXZJ66MA7RUPM4YDX6E2LSNQ7DABUCFFVLA, seg_pos={0 0}, upload=515086]: Done uploading piece. err: <nil> Dec 30 03:03:46.235553807 (587h34m46.106636862s) [seg_pos={0 0}, upload=515086]: Segment reached optimal threshold of 13 pieces. Dec 30 03:03:46.235780979 (587h34m46.106864024s) [noise=true , address=45.91.116.11:4119 , piece_id=KEZ2CIX3PCLO4MPD367AEIY5B4G7KGGDBVJKATLDPMTRT5RDBSPQ, seg_pos={0 0}, upload=515086]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 30 03:03:46.235790848 (587h34m46.106873883s) [noise=true , address=45.91.116.11:4119 , piece_id=KEZ2CIX3PCLO4MPD367AEIY5B4G7KGGDBVJKATLDPMTRT5RDBSPQ, seg_pos={0 0}, upload=515086]: Long tail cancellation due to optimal threshold Dec 30 03:03:46.236012901 (587h34m46.107095966s) [noise=true , address=145.79.227.44:4193 , piece_id=XWVAX5Y53A55ZQWOCOL6E5AZGJPBDVXMKTZCPT46J7KT6DN7ATTQ, seg_pos={0 0}, upload=515086]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 30 03:03:46.236024332 (587h34m46.107107377s) [noise=true , address=145.79.227.44:4193 , piece_id=XWVAX5Y53A55ZQWOCOL6E5AZGJPBDVXMKTZCPT46J7KT6DN7ATTQ, seg_pos={0 0}, upload=515086]: Long tail cancellation due to optimal threshold Dec 30 03:03:46.236088344 (587h34m46.107171379s) [noise=true , address=145.79.227.5:4222 , piece_id=TLYHW5ZI4A7BNPGD46HWFTNEWGWEO4DWI455GNSCBXJUIQY4FA2Q, seg_pos={0 0}, upload=515086]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: expected piece hash; context canceled; EOF Dec 30 03:03:46.236104355 (587h34m46.1071874s) [noise=true , address=145.79.227.5:4222 , piece_id=TLYHW5ZI4A7BNPGD46HWFTNEWGWEO4DWI455GNSCBXJUIQY4FA2Q, seg_pos={0 0}, upload=515086]: Long tail cancellation due to optimal threshold Dec 30 03:03:46.236165651 (587h34m46.107248686s) [seg_pos={0 0}, upload=515086]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 03:03:46.236274089 (587h34m46.107357134s) [upload=515086]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 03:03:46.236280160 (587h34m46.107363195s) [upload=515086]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 03:03:46.536344670 (587h34m46.407427715s) [upload=515086]: Done uploading segments... Dec 30 03:03:46.536367172 (587h34m46.407450217s) [upload=515086]: Upload finished. err: <nil> Duration: 6.200047428s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [X......X........X...X..XX..X.X......XXX.X.X..X........X.X.XX] duration: 30.004736284s trace: https://jaeger.dev.storj.io/trace/1f609d54324f4aba FAIL (2025-12-30 03:04:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 03:03:42.664733818 (587h34m42.535816853s) [upload=515086]: Starting upload Dec 30 03:03:42.664803922 (587h34m42.535886957s) [upload=515086]: Done starting upload Dec 30 03:03:42.664926486 (587h34m42.536009531s) [upload=515086]: Uploading segments... Dec 30 03:03:42.665104645 (587h34m42.53618769s) [upload=515086]: Got next segment. Inline: false Dec 30 03:03:42.665157715 (587h34m42.53624077s) [upload=515086]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 03:03:42.665164197 (587h34m42.536247242s) [upload=515086]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 03:03:42.916348402 (587h34m42.787431447s) [seg_pos={0 0}, upload=515086]: Begin upload segment... Dec 30 03:03:42.916399158 (587h34m42.787482203s) [seg_pos={0 0}, upload=515086]: Done begin upload segment. Dec 30 03:03:42.916461577 (587h34m42.787544612s) [noise=true , address=145.79.227.5:4222 , piece_id=TLYHW5ZI4A7BNPGD46HWFTNEWGWEO4DWI455GNSCBXJUIQY4FA2Q, seg_pos={0 0}, upload=515086]: Uploading piece... Dec 30 03:03:42.916489440 (587h34m42.787572475s) [noise=true , address=45.91.116.11:4119 , piece_id=KEZ2CIX3PCLO4MPD367AEIY5B4G7KGGDBVJKATLDPMTRT5RDBSPQ, seg_pos={0 0}, upload=515086]: Uploading piece... Dec 30 03:03:42.917030490 (587h34m42.788113525s) [noise=true , address=145.79.227.34:4906 , piece_id=GXI6LC45KZ5UA4KPWMXZJ66MA7RUPM4YDX6E2LSNQ7DABUCFFVLA, seg_pos={0 0}, upload=515086]: Uploading piece... Dec 30 03:03:42.917755180 (587h34m42.788838215s) [noise=true , address=145.79.227.13:4803 , piece_id=2NFC5GJ7FPRYQ6QVGE7CMIXNHEB4VDB2TAWWSQA5LM5RZUDMLKBA, seg_pos={0 0}, upload=515086]: Uploading piece... Dec 30 03:03:42.918253569 (587h34m42.789336604s) [noise=true , address=145.79.227.15:4873 , piece_id=DOPL3LC6OXAYLCTIZWBUBY2EOY2EYEV46NG4AHHS776I653WX25A, seg_pos={0 0}, upload=515086]: Uploading piece... Dec 30 03:03:42.919431933 (587h34m42.790514978s) [noise=true , address=77.90.31.13:4627 , piece_id=6QHJQ3SE47DC5UYSAZBPDDW7FYMD6V3HIN5MZ6MOHD4GBYJ6PU6Q, seg_pos={0 0}, upload=515086]: Uploading piece... Dec 30 03:03:42.920977887 (587h34m42.792060922s) [noise=true , address=145.79.227.3:4151 , piece_id=GCMHNSC6JQLJAB3L5PVZFYD4UKMPXZKEWJEOVL6VAFRNELPH7XUA, seg_pos={0 0}, upload=515086]: Uploading piece... Dec 30 03:03:42.922352504 (587h34m42.793435539s) [noise=true , address=145.79.227.14:4837 , piece_id=GJKO67C2D34PEWE7N6E4UDXGK6X2SKZ5CELLKZTV7XMG6TFFQRYQ, seg_pos={0 0}, upload=515086]: Uploading piece... Dec 30 03:03:42.923837903 (587h34m42.794920948s) [noise=true , address=145.79.227.44:4193 , piece_id=XWVAX5Y53A55ZQWOCOL6E5AZGJPBDVXMKTZCPT46J7KT6DN7ATTQ, seg_pos={0 0}, upload=515086]: Uploading piece... Dec 30 03:03:42.925195508 (587h34m42.796278553s) [noise=true , address=145.79.227.21:4475 , piece_id=W2KUQ5YXLWPXDCKDKLEYSU57Z37NT3K3V6FOLW2543S66RQRC2MA, seg_pos={0 0}, upload=515086]: Uploading piece... Dec 30 03:03:42.925309184 (587h34m42.796392229s) [noise=true , address=145.79.227.52:4445 , piece_id=SCEZ4SNBWWKSJYXDACVUEUW3VGGXDXDAO3PD2K2ILSKW6FNBSM2A, seg_pos={0 0}, upload=515086]: Uploading piece... Dec 30 03:03:42.927045460 (587h34m42.798128506s) [noise=true , address=45.91.116.5:4300 , piece_id=ODDRH7AM5TANWKSYGLESZU5LUACCPVU7TI2PV7PPOZUHGBGNT35A, seg_pos={0 0}, upload=515086]: Uploading piece... Dec 30 03:03:42.927814815 (587h34m42.79889786s) [noise=true , address=45.91.116.10:4083 , piece_id=L37FM6KLC2IOPOB6QLI4WYNXNWK2DIZPZXYGDPRT4P6ZBO4RLO4Q, seg_pos={0 0}, upload=515086]: Uploading piece... Dec 30 03:03:42.928481585 (587h34m42.79956462s) [noise=true , address=45.91.116.8:4048 , piece_id=FAFGXWER5ICDY4BA7LTTFT34BXPY2MYWYOIYRLRHE3LPSKBQMK7Q, seg_pos={0 0}, upload=515086]: Uploading piece... Dec 30 03:03:42.929498541 (587h34m42.800581576s) [noise=true , address=145.79.227.10:4401 , piece_id=IF3T5AIHAP5SLDU4BL6A7KYM32MHVFDNTPU7ADI5UOUYL72FPPBQ, seg_pos={0 0}, upload=515086]: Uploading piece... Dec 30 03:03:42.929784767 (587h34m42.800867802s) [noise=true , address=77.90.31.12:4590 , piece_id=AIURUSSEUNDHTXREDBQZBRMBVRZBERPRNERXEOIPYYJZQZ7XOD7A, seg_pos={0 0}, upload=515086]: Uploading piece... Dec 30 03:03:44.609888847 (587h34m44.480971882s) [upload=515086]: Next returned nil segment Dec 30 03:03:44.609903665 (587h34m44.4809867s) [upload=515086]: Waiting for error group managing segments... Dec 30 03:03:44.889262778 (587h34m44.760345813s) [noise=true , address=145.79.227.3:4151 , piece_id=GCMHNSC6JQLJAB3L5PVZFYD4UKMPXZKEWJEOVL6VAFRNELPH7XUA, seg_pos={0 0}, upload=515086]: Done uploading piece. err: <nil> Dec 30 03:03:44.905294826 (587h34m44.776377861s) [noise=true , address=145.79.227.13:4803 , piece_id=2NFC5GJ7FPRYQ6QVGE7CMIXNHEB4VDB2TAWWSQA5LM5RZUDMLKBA, seg_pos={0 0}, upload=515086]: Done uploading piece. err: <nil> Dec 30 03:03:44.911669865 (587h34m44.7827529s) [noise=true , address=145.79.227.21:4475 , piece_id=W2KUQ5YXLWPXDCKDKLEYSU57Z37NT3K3V6FOLW2543S66RQRC2MA, seg_pos={0 0}, upload=515086]: Done uploading piece. err: <nil> Dec 30 03:03:44.913028953 (587h34m44.784111988s) [noise=true , address=145.79.227.14:4837 , piece_id=GJKO67C2D34PEWE7N6E4UDXGK6X2SKZ5CELLKZTV7XMG6TFFQRYQ, seg_pos={0 0}, upload=515086]: Done uploading piece. err: <nil> Dec 30 03:03:44.923221862 (587h34m44.794304897s) [noise=true , address=145.79.227.52:4445 , piece_id=SCEZ4SNBWWKSJYXDACVUEUW3VGGXDXDAO3PD2K2ILSKW6FNBSM2A, seg_pos={0 0}, upload=515086]: Done uploading piece. err: <nil> Dec 30 03:03:44.929698846 (587h34m44.800781891s) [noise=true , address=145.79.227.15:4873 , piece_id=DOPL3LC6OXAYLCTIZWBUBY2EOY2EYEV46NG4AHHS776I653WX25A, seg_pos={0 0}, upload=515086]: Done uploading piece. err: <nil> Dec 30 03:03:44.933930954 (587h34m44.805013999s) [noise=true , address=145.79.227.10:4401 , piece_id=IF3T5AIHAP5SLDU4BL6A7KYM32MHVFDNTPU7ADI5UOUYL72FPPBQ, seg_pos={0 0}, upload=515086]: Done uploading piece. err: <nil> Dec 30 03:03:45.105306018 (587h34m44.976389053s) [noise=true , address=77.90.31.13:4627 , piece_id=6QHJQ3SE47DC5UYSAZBPDDW7FYMD6V3HIN5MZ6MOHD4GBYJ6PU6Q, seg_pos={0 0}, upload=515086]: Done uploading piece. err: <nil> Dec 30 03:03:45.110644804 (587h34m44.981727849s) [noise=true , address=77.90.31.12:4590 , piece_id=AIURUSSEUNDHTXREDBQZBRMBVRZBERPRNERXEOIPYYJZQZ7XOD7A, seg_pos={0 0}, upload=515086]: Done uploading piece. err: <nil> Dec 30 03:03:45.180587856 (587h34m45.051670901s) [noise=true , address=45.91.116.5:4300 , piece_id=ODDRH7AM5TANWKSYGLESZU5LUACCPVU7TI2PV7PPOZUHGBGNT35A, seg_pos={0 0}, upload=515086]: Done uploading piece. err: <nil> Dec 30 03:03:45.199847840 (587h34m45.070930875s) [noise=true , address=45.91.116.10:4083 , piece_id=L37FM6KLC2IOPOB6QLI4WYNXNWK2DIZPZXYGDPRT4P6ZBO4RLO4Q, seg_pos={0 0}, upload=515086]: Done uploading piece. err: <nil> Dec 30 03:03:45.903412778 (587h34m45.774495823s) [noise=true , address=45.91.116.8:4048 , piece_id=FAFGXWER5ICDY4BA7LTTFT34BXPY2MYWYOIYRLRHE3LPSKBQMK7Q, seg_pos={0 0}, upload=515086]: Done uploading piece. err: <nil> Dec 30 03:03:46.235528288 (587h34m46.106611323s) [noise=true , address=145.79.227.34:4906 , piece_id=GXI6LC45KZ5UA4KPWMXZJ66MA7RUPM4YDX6E2LSNQ7DABUCFFVLA, seg_pos={0 0}, upload=515086]: Done uploading piece. err: <nil> Dec 30 03:03:46.235553807 (587h34m46.106636862s) [seg_pos={0 0}, upload=515086]: Segment reached optimal threshold of 13 pieces. Dec 30 03:03:46.235780979 (587h34m46.106864024s) [noise=true , address=45.91.116.11:4119 , piece_id=KEZ2CIX3PCLO4MPD367AEIY5B4G7KGGDBVJKATLDPMTRT5RDBSPQ, seg_pos={0 0}, upload=515086]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 30 03:03:46.235790848 (587h34m46.106873883s) [noise=true , address=45.91.116.11:4119 , piece_id=KEZ2CIX3PCLO4MPD367AEIY5B4G7KGGDBVJKATLDPMTRT5RDBSPQ, seg_pos={0 0}, upload=515086]: Long tail cancellation due to optimal threshold Dec 30 03:03:46.236012901 (587h34m46.107095966s) [noise=true , address=145.79.227.44:4193 , piece_id=XWVAX5Y53A55ZQWOCOL6E5AZGJPBDVXMKTZCPT46J7KT6DN7ATTQ, seg_pos={0 0}, upload=515086]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 30 03:03:46.236024332 (587h34m46.107107377s) [noise=true , address=145.79.227.44:4193 , piece_id=XWVAX5Y53A55ZQWOCOL6E5AZGJPBDVXMKTZCPT46J7KT6DN7ATTQ, seg_pos={0 0}, upload=515086]: Long tail cancellation due to optimal threshold Dec 30 03:03:46.236088344 (587h34m46.107171379s) [noise=true , address=145.79.227.5:4222 , piece_id=TLYHW5ZI4A7BNPGD46HWFTNEWGWEO4DWI455GNSCBXJUIQY4FA2Q, seg_pos={0 0}, upload=515086]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: expected piece hash; context canceled; EOF Dec 30 03:03:46.236104355 (587h34m46.1071874s) [noise=true , address=145.79.227.5:4222 , piece_id=TLYHW5ZI4A7BNPGD46HWFTNEWGWEO4DWI455GNSCBXJUIQY4FA2Q, seg_pos={0 0}, upload=515086]: Long tail cancellation due to optimal threshold Dec 30 03:03:46.236165651 (587h34m46.107248686s) [seg_pos={0 0}, upload=515086]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 03:03:46.236274089 (587h34m46.107357134s) [upload=515086]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 03:03:46.236280160 (587h34m46.107363195s) [upload=515086]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 03:03:46.536344670 (587h34m46.407427715s) [upload=515086]: Done uploading segments... Dec 30 03:03:46.536367172 (587h34m46.407450217s) [upload=515086]: Upload finished. err: <nil> Duration: 30.004749318s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [.X...................................X...X.X.XX.............] duration: 4.518464312s trace: https://jaeger.dev.storj.io/trace/14b283cf2d9b0313 FAIL (2025-12-30 03:03:45 UTC) orbiter warning: context deadline exceeded Duration: 4.518492695s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.X......X........X...X..XX..X.X......XXX.X.X..X........X.X.X] duration: 30.004021042s trace: https://jaeger.dev.storj.io/trace/17c53afc8dba92b3 FAIL (2025-12-30 03:03:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: 36MKVBR677D2F6EYUMZWZOLHZLQQRVDF7L2YOYFVNTDFDKTLJ5RA) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: WDR735FUQA4N7Y67V2ORMHBXYRM2YNVNN4QURCOZB35E5ZK6U37A) context deadline exceeded ======LOGS====== 3:12.366015084 (587h34m12.237098129s) [upload=515081]: Starting upload Dec 30 03:03:12.366186521 (587h34m12.237269576s) [upload=515081]: Done starting upload Dec 30 03:03:12.366277554 (587h34m12.237360599s) [upload=515081]: Uploading segments... Dec 30 03:03:12.366433040 (587h34m12.237516085s) [upload=515081]: Got next segment. Inline: false Dec 30 03:03:12.366478076 (587h34m12.237561121s) [upload=515081]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 03:03:12.366505849 (587h34m12.237588894s) [upload=515081]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 03:03:12.609985672 (587h34m12.481068747s) [seg_pos={0 0}, upload=515081]: Begin upload segment... Dec 30 03:03:12.610041639 (587h34m12.481124674s) [seg_pos={0 0}, upload=515081]: Done begin upload segment. Dec 30 03:03:12.610074962 (587h34m12.481157998s) [noise=true , address=145.79.227.41:4085 , piece_id=GBHIEYNGRBRGIHZCLQIEHDVE6VPBU4HRSAW5GLEA7Q2KUZBSX5ZA, seg_pos={0 0}, upload=515081]: Uploading piece... Dec 30 03:03:12.610189771 (587h34m12.481272816s) [noise=true , address=45.91.116.13:4156 , piece_id=36MKVBR677D2F6EYUMZWZOLHZLQQRVDF7L2YOYFVNTDFDKTLJ5RA, seg_pos={0 0}, upload=515081]: Uploading piece... Dec 30 03:03:12.611020222 (587h34m12.482103257s) [noise=true , address=77.90.31.14:4664 , piece_id=SQKCNWEBOACQIX2OS3FS5EVZ6RIDWZP7YWN666VRLVSEFXH2PNZQ, seg_pos={0 0}, upload=515081]: Uploading piece... Dec 30 03:03:12.611224091 (587h34m12.482307136s) [noise=true , address=145.79.227.4:4187 , piece_id=SMX2YAVKM75J3NWO46F4UI77A46H4G6SZZC2YYJFTRA2TCY2FFTA, seg_pos={0 0}, upload=515081]: Uploading piece... Dec 30 03:03:12.611419141 (587h34m12.482502176s) [noise=true , address=77.90.31.13:4626 , piece_id=UXGS6BXDLIODGIQSMSIN53XNUH52C4WO342XJ4MK7Y6QZF3UPMXQ, seg_pos={0 0}, upload=515081]: Uploading piece... Dec 30 03:03:12.611588254 (587h34m12.482671299s) [noise=true , address=77.90.31.11:4553 , piece_id=TWJPYF326CKYYJMBMVHHTKV3EOOGZMNYXUNDXQ5CWLB7SG6MS4WA, seg_pos={0 0}, upload=515081]: Uploading piece... Dec 30 03:03:12.611848509 (587h34m12.482931544s) [noise=true , address=45.91.116.10:4082 , piece_id=WDR735FUQA4N7Y67V2ORMHBXYRM2YNVNN4QURCOZB35E5ZK6U37A, seg_pos={0 0}, upload=515081]: Uploading piece... Dec 30 03:03:12.611960423 (587h34m12.483043468s) [noise=true , address=145.79.227.48:4301 , piece_id=PDIPXHWCIAND3QMP472M6XS3OYNT6ZIX2PNXCJPY6MT6H57B3HEA, seg_pos={0 0}, upload=515081]: Uploading piece... Dec 30 03:03:12.612273529 (587h34m12.483356574s) [noise=true , address=145.79.227.42:4122 , piece_id=WDG4KHPHFOLAGKJDQEPNUCK56NYZZ6UVFWDK5NZD6VWXPUDTSWCQ, seg_pos={0 0}, upload=515081]: Uploading piece... Dec 30 03:03:12.612318474 (587h34m12.483401519s) [noise=true , address=145.79.227.52:4445 , piece_id=4KGBRJBRI7VSPTDTZNPNCM6AT5BSLXUZP2TRUWLEUO66UJFSV5HA, seg_pos={0 0}, upload=515081]: Uploading piece... Dec 30 03:03:12.612645066 (587h34m12.483728101s) [noise=true , address=145.79.227.2:4114 , piece_id=KZEREEUHYR4YUPQSNTFXZRNIL6BL4OJ2DHMFGZCYMITJNZBQB6QA, seg_pos={0 0}, upload=515081]: Uploading piece... Dec 30 03:03:12.612727663 (587h34m12.483810709s) [noise=true , address=145.79.227.25:4619 , piece_id=D6DU46545QPCCUY362TXFPOOJO56HR25J6LBN7KFPVLGMIU6KGFA, seg_pos={0 0}, upload=515081]: Uploading piece... Dec 30 03:03:12.613013708 (587h34m12.484096743s) [noise=true , address=45.91.116.11:4117 , piece_id=HCRGJHEUBRZVS6I5BKTHNTD6SZXOKKE7ZCRSF2K32FF4NRNAZ5SQ, seg_pos={0 0}, upload=515081]: Uploading piece... Dec 30 03:03:12.613120281 (587h34m12.484203316s) [noise=true , address=45.91.116.9:4262 , piece_id=XISR4HZHXCGVYDNXBZHTBIEPD2H7PVSE2HZK4THBUWMZ2PDSHJMQ, seg_pos={0 0}, upload=515081]: Uploading piece... Dec 30 03:03:12.613382300 (587h34m12.484465345s) [noise=true , address=45.91.116.7:4009 , piece_id=GX4YFPHLGKDCL34TGOAVRG3O2R5M3TD4ZO54Y6TWSGBMKAOHBFNQ, seg_pos={0 0}, upload=515081]: Uploading piece... Dec 30 03:03:12.613732677 (587h34m12.484815712s) [noise=true , address=145.79.227.32:4836 , piece_id=NYWNLLA6LPVEHQGVVF42PJE3SHBGWNS6A67SP4LLKNQ6DRXK7EMA, seg_pos={0 0}, upload=515081]: Uploading piece... Dec 30 03:03:14.280149183 (587h34m14.151232218s) [upload=515081]: Next returned nil segment Dec 30 03:03:14.280162048 (587h34m14.151245094s) [upload=515081]: Waiting for error group managing segments... Dec 30 03:03:14.585500630 (587h34m14.456583676s) [noise=true , address=145.79.227.48:4301 , piece_id=PDIPXHWCIAND3QMP472M6XS3OYNT6ZIX2PNXCJPY6MT6H57B3HEA, seg_pos={0 0}, upload=515081]: Done uploading piece. err: <nil> Dec 30 03:03:14.593883533 (587h34m14.464966578s) [noise=true , address=145.79.227.2:4114 , piece_id=KZEREEUHYR4YUPQSNTFXZRNIL6BL4OJ2DHMFGZCYMITJNZBQB6QA, seg_pos={0 0}, upload=515081]: Done uploading piece. err: <nil> Dec 30 03:03:14.598714372 (587h34m14.469797417s) [noise=true , address=145.79.227.52:4445 , piece_id=4KGBRJBRI7VSPTDTZNPNCM6AT5BSLXUZP2TRUWLEUO66UJFSV5HA, seg_pos={0 0}, upload=515081]: Done uploading piece. err: <nil> Dec 30 03:03:14.612148953 (587h34m14.483231998s) [noise=true , address=145.79.227.42:4122 , piece_id=WDG4KHPHFOLAGKJDQEPNUCK56NYZZ6UVFWDK5NZD6VWXPUDTSWCQ, seg_pos={0 0}, upload=515081]: Done uploading piece. err: <nil> Dec 30 03:03:14.630554010 (587h34m14.501637045s) [noise=true , address=145.79.227.32:4836 , piece_id=NYWNLLA6LPVEHQGVVF42PJE3SHBGWNS6A67SP4LLKNQ6DRXK7EMA, seg_pos={0 0}, upload=515081]: Done uploading piece. err: <nil> Dec 30 03:03:14.640542308 (587h34m14.511625353s) [noise=true , address=145.79.227.25:4619 , piece_id=D6DU46545QPCCUY362TXFPOOJO56HR25J6LBN7KFPVLGMIU6KGFA, seg_pos={0 0}, upload=515081]: Done uploading piece. err: <nil> Dec 30 03:03:14.782481093 (587h34m14.653564128s) [noise=true , address=77.90.31.13:4626 , piece_id=UXGS6BXDLIODGIQSMSIN53XNUH52C4WO342XJ4MK7Y6QZF3UPMXQ, seg_pos={0 0}, upload=515081]: Done uploading piece. err: <nil> Dec 30 03:03:14.836510593 (587h34m14.707593668s) [noise=true , address=45.91.116.13:4156 , piece_id=36MKVBR677D2F6EYUMZWZOLHZLQQRVDF7L2YOYFVNTDFDKTLJ5RA, seg_pos={0 0}, upload=515081]: Done uploading piece. err: <nil> Dec 30 03:03:14.847124864 (587h34m14.718207909s) [noise=true , address=45.91.116.9:4262 , piece_id=XISR4HZHXCGVYDNXBZHTBIEPD2H7PVSE2HZK4THBUWMZ2PDSHJMQ, seg_pos={0 0}, upload=515081]: Done uploading piece. err: <nil> Dec 30 03:03:14.866067835 (587h34m14.73715088s) [noise=true , address=45.91.116.10:4082 , piece_id=WDR735FUQA4N7Y67V2ORMHBXYRM2YNVNN4QURCOZB35E5ZK6U37A, seg_pos={0 0}, upload=515081]: Done uploading piece. err: <nil> Dec 30 03:03:14.872608639 (587h34m14.743691684s) [noise=true , address=45.91.116.11:4117 , piece_id=HCRGJHEUBRZVS6I5BKTHNTD6SZXOKKE7ZCRSF2K32FF4NRNAZ5SQ, seg_pos={0 0}, upload=515081]: Done uploading piece. err: <nil> Dec 30 03:03:14.881990894 (587h34m14.753073929s) [noise=true , address=77.90.31.11:4553 , piece_id=TWJPYF326CKYYJMBMVHHTKV3EOOGZMNYXUNDXQ5CWLB7SG6MS4WA, seg_pos={0 0}, upload=515081]: Done uploading piece. err: <nil> Dec 30 03:03:15.035544725 (587h34m14.90662777s) [noise=true , address=45.91.116.7:4009 , piece_id=GX4YFPHLGKDCL34TGOAVRG3O2R5M3TD4ZO54Y6TWSGBMKAOHBFNQ, seg_pos={0 0}, upload=515081]: Done uploading piece. err: <nil> Dec 30 03:03:15.035564041 (587h34m14.906647086s) [seg_pos={0 0}, upload=515081]: Segment reached optimal threshold of 13 pieces. Dec 30 03:03:15.035839315 (587h34m14.90692235s) [noise=true , address=145.79.227.4:4187 , piece_id=SMX2YAVKM75J3NWO46F4UI77A46H4G6SZZC2YYJFTRA2TCY2FFTA, seg_pos={0 0}, upload=515081]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: expected piece hash; context canceled; EOF Dec 30 03:03:15.035846659 (587h34m14.906929694s) [noise=true , address=145.79.227.4:4187 , piece_id=SMX2YAVKM75J3NWO46F4UI77A46H4G6SZZC2YYJFTRA2TCY2FFTA, seg_pos={0 0}, upload=515081]: Long tail cancellation due to optimal threshold Dec 30 03:03:15.035984031 (587h34m14.907067076s) [noise=true , address=145.79.227.41:4085 , piece_id=GBHIEYNGRBRGIHZCLQIEHDVE6VPBU4HRSAW5GLEA7Q2KUZBSX5ZA, seg_pos={0 0}, upload=515081]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C): protocol: write tcp 188.166.225.170:56696->145.79.227.41:4085: use of closed network connection; write tcp 188.166.225.170:56696->145.79.227.41:4085: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56696->145.79.227.41:4085: use of closed network connection Dec 30 03:03:15.035998389 (587h34m14.907081424s) [noise=true , address=145.79.227.41:4085 , piece_id=GBHIEYNGRBRGIHZCLQIEHDVE6VPBU4HRSAW5GLEA7Q2KUZBSX5ZA, seg_pos={0 0}, upload=515081]: Long tail cancellation due to optimal threshold Dec 30 03:03:15.036072790 (587h34m14.907155835s) [noise=true , address=77.90.31.14:4664 , piece_id=SQKCNWEBOACQIX2OS3FS5EVZ6RIDWZP7YWN666VRLVSEFXH2PNZQ, seg_pos={0 0}, upload=515081]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: expected piece hash; context canceled; EOF Dec 30 03:03:15.036079463 (587h34m14.907162498s) [noise=true , address=77.90.31.14:4664 , piece_id=SQKCNWEBOACQIX2OS3FS5EVZ6RIDWZP7YWN666VRLVSEFXH2PNZQ, seg_pos={0 0}, upload=515081]: Long tail cancellation due to optimal threshold Dec 30 03:03:15.036096506 (587h34m14.907179541s) [seg_pos={0 0}, upload=515081]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 03:03:15.036166499 (587h34m14.907249534s) [upload=515081]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 03:03:15.036171077 (587h34m14.907254113s) [upload=515081]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 03:03:15.323352395 (587h34m15.194435431s) [upload=515081]: Done uploading segments... Dec 30 03:03:15.323371962 (587h34m15.194454997s) [upload=515081]: Upload finished. err: <nil> Duration: 30.004045229s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.........X.............X................X......X............] duration: 32.252810313s trace: https://jaeger.dev.storj.io/trace/6c3caa8adcfacbb9 FAIL (2025-12-30 03:03:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== true , address=145.79.227.35:4496 , piece_id=5X54ZGU4OIVI3DT622QE7IL66SVFLZYUICPODGMDKN7U4D4OJNSA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.040303259 (587h24m33.911386294s) [noise=true , address=77.90.31.12:4601 , piece_id=QQFK4PCL27YYF3UATGLV5KDENRNX6NHE7RXAZ2K66CC3WLHEBMWQ, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.040670117 (587h24m33.911753152s) [noise=true , address=77.90.31.21:4881 , piece_id=EMV4LHFK2VRHSW6PD3BBYGOXDAAYJKX6GEOXGGLOT3IXWNDQMU4A, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041006247 (587h24m33.912089292s) [noise=true , address=145.79.227.52:4472 , piece_id=DT2DAIC4ZI3WL7OGY55ZTTXJEI4IDRXPSJIQA3LTRC7K2F6EVPCA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041183314 (587h24m33.912266349s) [noise=true , address=145.79.227.26:4674 , piece_id=3BAN2ZXOFTSGZITYYHMZUS4AD35X6ZBOYA2HKOJCJI7VBWPJ3LSA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041313653 (587h24m33.912396688s) [noise=true , address=145.79.227.51:4429 , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041535194 (587h24m33.912618229s) [noise=true , address=145.79.227.33:4885 , piece_id=LA755L4RA4VJBHNCQBXOCFROZA3W6W6PUPKGJ5J772WAN44YMY2A, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041639693 (587h24m33.912722728s) [noise=true , address=145.79.227.43:4187 , piece_id=CAAFD3XIWWHZN7L7SYDMMSC5G5RKRLFYYHQZEEN4GAC2D6Y2EARA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041877216 (587h24m33.912960261s) [noise=true , address=145.79.227.46:4623 , piece_id=V4TEM7RRTTGBW5ISZOEAOPFIOZZ5EABO7BPWNRH5U5UPM2X7554Q, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041986634 (587h24m33.913069669s) [noise=true , address=45.91.116.13:4176 , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.042345657 (587h24m33.913428692s) [noise=true , address=45.91.116.10:4104 , piece_id=MRIWH6WGACE4UDNX55MIY6ZDWBAHS6E5KQS4QCOQN2BU7JI7ODCA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.042362019 (587h24m33.913445054s) [noise=true , address=77.90.31.11:4573 , piece_id=TORCC4W7DS5TJ7H7NWONEPNJOFULSZ6F5OHOP4BEVMGEH2MJA4DQ, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.042684192 (587h24m33.913767237s) [noise=true , address=145.79.227.49:4340 , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.042707327 (587h24m33.913790362s) [noise=true , address=145.79.227.32:4837 , piece_id=SQTHTF5KWZOKFYX6CGL6KCDRDAY7OJY6NA5OM3N4YCICDPM6DKXA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.043091528 (587h24m33.914174563s) [noise=true , address=45.91.116.5:4309 , piece_id=Y5HCMU6Z2FPBV3BNTUOC4F6JJWKNBKPOXWFVAFQTXUKQNHTLA4TA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.043544771 (587h24m33.914627816s) [noise=true , address=145.79.227.40:4053 , piece_id=3IZP7JBGSWBRBQHMJWXIDD73B5R3COWU5H3IYFGVBCTTLW7OUJ5A, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:35.347637489 (587h24m35.218720524s) [upload=514937]: Next returned nil segment Dec 30 02:53:35.347733591 (587h24m35.218816626s) [upload=514937]: Waiting for error group managing segments... Dec 30 02:53:35.674169656 (587h24m35.545252721s) [noise=true , address=64.49.13.27:4959 , piece_id=B4UZ67DIWNRHV77VTQONS6ZBUI2BPDDBM3ZS6FVCTABQZ6OHRJJA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:35.891295347 (587h24m35.762378382s) [noise=true , address=145.79.227.32:4837 , piece_id=SQTHTF5KWZOKFYX6CGL6KCDRDAY7OJY6NA5OM3N4YCICDPM6DKXA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:35.912619632 (587h24m35.783702677s) [noise=true , address=145.79.227.26:4674 , piece_id=3BAN2ZXOFTSGZITYYHMZUS4AD35X6ZBOYA2HKOJCJI7VBWPJ3LSA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:35.913054741 (587h24m35.784137786s) [noise=true , address=145.79.227.46:4623 , piece_id=V4TEM7RRTTGBW5ISZOEAOPFIOZZ5EABO7BPWNRH5U5UPM2X7554Q, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:35.913485270 (587h24m35.784568305s) [noise=true , address=145.79.227.35:4496 , piece_id=5X54ZGU4OIVI3DT622QE7IL66SVFLZYUICPODGMDKN7U4D4OJNSA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:35.923790864 (587h24m35.794873909s) [noise=true , address=145.79.227.52:4472 , piece_id=DT2DAIC4ZI3WL7OGY55ZTTXJEI4IDRXPSJIQA3LTRC7K2F6EVPCA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.109605534 (587h24m35.980688569s) [noise=true , address=77.90.31.21:4881 , piece_id=EMV4LHFK2VRHSW6PD3BBYGOXDAAYJKX6GEOXGGLOT3IXWNDQMU4A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.122502652 (587h24m35.993585697s) [noise=true , address=77.90.31.11:4573 , piece_id=TORCC4W7DS5TJ7H7NWONEPNJOFULSZ6F5OHOP4BEVMGEH2MJA4DQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.151131466 (587h24m36.022214511s) [noise=true , address=45.91.116.10:4104 , piece_id=MRIWH6WGACE4UDNX55MIY6ZDWBAHS6E5KQS4QCOQN2BU7JI7ODCA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.159562149 (587h24m36.030645184s) [noise=true , address=77.90.31.12:4601 , piece_id=QQFK4PCL27YYF3UATGLV5KDENRNX6NHE7RXAZ2K66CC3WLHEBMWQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.238339784 (587h24m36.10942282s) [noise=true , address=45.91.116.5:4309 , piece_id=Y5HCMU6Z2FPBV3BNTUOC4F6JJWKNBKPOXWFVAFQTXUKQNHTLA4TA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.813503456 (587h24m36.684586501s) [noise=true , address=145.79.227.33:4885 , piece_id=LA755L4RA4VJBHNCQBXOCFROZA3W6W6PUPKGJ5J772WAN44YMY2A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.826964278 (587h24m36.698047313s) [noise=true , address=145.79.227.43:4187 , piece_id=CAAFD3XIWWHZN7L7SYDMMSC5G5RKRLFYYHQZEEN4GAC2D6Y2EARA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.828562510 (587h24m36.699645545s) [noise=true , address=145.79.227.40:4053 , piece_id=3IZP7JBGSWBRBQHMJWXIDD73B5R3COWU5H3IYFGVBCTTLW7OUJ5A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.828575566 (587h24m36.699658601s) [seg_pos={0 0}, upload=514937]: Segment reached optimal threshold of 14 pieces. Dec 30 02:53:36.828752202 (587h24m36.699835237s) [noise=true , address=145.79.227.30:4780 , piece_id=DZQ5LIBMWXPJ4XXWORRDLQIAFXJDNDQEEBRMSBS44UX3D2HPTSWQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125WCbZ6a2NaewiLAHhHuPo91sQEfC9CZzEBV6rwF1JY7z13AEY): protocol: expected piece hash; context canceled; EOF Dec 30 02:53:36.828758714 (587h24m36.699841749s) [noise=true , address=145.79.227.30:4780 , piece_id=DZQ5LIBMWXPJ4XXWORRDLQIAFXJDNDQEEBRMSBS44UX3D2HPTSWQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold Dec 30 02:53:36.828762672 (587h24m36.699845707s) [noise=true , address=145.79.227.51:4429 , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qytmzwy7sTmWXskEbLWsWMniPz55zgho2akE7oh6ahAiT5SXf): protocol: write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection; write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection Dec 30 02:53:36.828774034 (587h24m36.699857069s) [noise=true , address=145.79.227.51:4429 , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold Dec 30 02:53:36.828889934 (587h24m36.699972969s) [noise=true , address=45.91.116.13:4176 , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ejsmBd4tAVwTHjamPYbryQb5x2vDnk9pBuX3CBF9WYFhTrFVJ): protocol: expected piece hash; context canceled; EOF Dec 30 02:53:36.828895144 (587h24m36.699978179s) [noise=true , address=45.91.116.13:4176 , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold Dec 30 02:53:36.828994173 (587h24m36.700077208s) [noise=true , address=145.79.227.49:4340 , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF Dec 30 02:53:36.828999793 (587h24m36.700082828s) [noise=true , address=145.79.227.49:4340 , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold Dec 30 02:53:36.829030332 (587h24m36.700113367s) [seg_pos={0 0}, upload=514937]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 02:53:36.829095486 (587h24m36.700178621s) [upload=514937]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 02:53:36.829100105 (587h24m36.70018315s) [upload=514937]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 02:53:37.128695270 (587h24m36.999778315s) [upload=514937]: Done uploading segments... Dec 30 02:53:37.128717683 (587h24m36.999800728s) [upload=514937]: Upload finished. err: <nil> Duration: 32.252836885s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [..X...................................X...X.X.XX............] duration: 5.000501097s trace: https://jaeger.dev.storj.io/trace/14b283cf2d9b0313 FAIL (2025-12-30 03:02:47 UTC) orbiter warning: context deadline exceeded Duration: 5.000524482s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [..........X.............X................X......X...........] duration: 28.013354678s trace: https://jaeger.dev.storj.io/trace/6c3caa8adcfacbb9 FAIL (2025-12-30 03:02:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== true , address=145.79.227.35:4496 , piece_id=5X54ZGU4OIVI3DT622QE7IL66SVFLZYUICPODGMDKN7U4D4OJNSA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.040303259 (587h24m33.911386294s) [noise=true , address=77.90.31.12:4601 , piece_id=QQFK4PCL27YYF3UATGLV5KDENRNX6NHE7RXAZ2K66CC3WLHEBMWQ, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.040670117 (587h24m33.911753152s) [noise=true , address=77.90.31.21:4881 , piece_id=EMV4LHFK2VRHSW6PD3BBYGOXDAAYJKX6GEOXGGLOT3IXWNDQMU4A, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041006247 (587h24m33.912089292s) [noise=true , address=145.79.227.52:4472 , piece_id=DT2DAIC4ZI3WL7OGY55ZTTXJEI4IDRXPSJIQA3LTRC7K2F6EVPCA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041183314 (587h24m33.912266349s) [noise=true , address=145.79.227.26:4674 , piece_id=3BAN2ZXOFTSGZITYYHMZUS4AD35X6ZBOYA2HKOJCJI7VBWPJ3LSA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041313653 (587h24m33.912396688s) [noise=true , address=145.79.227.51:4429 , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041535194 (587h24m33.912618229s) [noise=true , address=145.79.227.33:4885 , piece_id=LA755L4RA4VJBHNCQBXOCFROZA3W6W6PUPKGJ5J772WAN44YMY2A, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041639693 (587h24m33.912722728s) [noise=true , address=145.79.227.43:4187 , piece_id=CAAFD3XIWWHZN7L7SYDMMSC5G5RKRLFYYHQZEEN4GAC2D6Y2EARA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041877216 (587h24m33.912960261s) [noise=true , address=145.79.227.46:4623 , piece_id=V4TEM7RRTTGBW5ISZOEAOPFIOZZ5EABO7BPWNRH5U5UPM2X7554Q, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041986634 (587h24m33.913069669s) [noise=true , address=45.91.116.13:4176 , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.042345657 (587h24m33.913428692s) [noise=true , address=45.91.116.10:4104 , piece_id=MRIWH6WGACE4UDNX55MIY6ZDWBAHS6E5KQS4QCOQN2BU7JI7ODCA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.042362019 (587h24m33.913445054s) [noise=true , address=77.90.31.11:4573 , piece_id=TORCC4W7DS5TJ7H7NWONEPNJOFULSZ6F5OHOP4BEVMGEH2MJA4DQ, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.042684192 (587h24m33.913767237s) [noise=true , address=145.79.227.49:4340 , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.042707327 (587h24m33.913790362s) [noise=true , address=145.79.227.32:4837 , piece_id=SQTHTF5KWZOKFYX6CGL6KCDRDAY7OJY6NA5OM3N4YCICDPM6DKXA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.043091528 (587h24m33.914174563s) [noise=true , address=45.91.116.5:4309 , piece_id=Y5HCMU6Z2FPBV3BNTUOC4F6JJWKNBKPOXWFVAFQTXUKQNHTLA4TA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.043544771 (587h24m33.914627816s) [noise=true , address=145.79.227.40:4053 , piece_id=3IZP7JBGSWBRBQHMJWXIDD73B5R3COWU5H3IYFGVBCTTLW7OUJ5A, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:35.347637489 (587h24m35.218720524s) [upload=514937]: Next returned nil segment Dec 30 02:53:35.347733591 (587h24m35.218816626s) [upload=514937]: Waiting for error group managing segments... Dec 30 02:53:35.674169656 (587h24m35.545252721s) [noise=true , address=64.49.13.27:4959 , piece_id=B4UZ67DIWNRHV77VTQONS6ZBUI2BPDDBM3ZS6FVCTABQZ6OHRJJA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:35.891295347 (587h24m35.762378382s) [noise=true , address=145.79.227.32:4837 , piece_id=SQTHTF5KWZOKFYX6CGL6KCDRDAY7OJY6NA5OM3N4YCICDPM6DKXA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:35.912619632 (587h24m35.783702677s) [noise=true , address=145.79.227.26:4674 , piece_id=3BAN2ZXOFTSGZITYYHMZUS4AD35X6ZBOYA2HKOJCJI7VBWPJ3LSA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:35.913054741 (587h24m35.784137786s) [noise=true , address=145.79.227.46:4623 , piece_id=V4TEM7RRTTGBW5ISZOEAOPFIOZZ5EABO7BPWNRH5U5UPM2X7554Q, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:35.913485270 (587h24m35.784568305s) [noise=true , address=145.79.227.35:4496 , piece_id=5X54ZGU4OIVI3DT622QE7IL66SVFLZYUICPODGMDKN7U4D4OJNSA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:35.923790864 (587h24m35.794873909s) [noise=true , address=145.79.227.52:4472 , piece_id=DT2DAIC4ZI3WL7OGY55ZTTXJEI4IDRXPSJIQA3LTRC7K2F6EVPCA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.109605534 (587h24m35.980688569s) [noise=true , address=77.90.31.21:4881 , piece_id=EMV4LHFK2VRHSW6PD3BBYGOXDAAYJKX6GEOXGGLOT3IXWNDQMU4A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.122502652 (587h24m35.993585697s) [noise=true , address=77.90.31.11:4573 , piece_id=TORCC4W7DS5TJ7H7NWONEPNJOFULSZ6F5OHOP4BEVMGEH2MJA4DQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.151131466 (587h24m36.022214511s) [noise=true , address=45.91.116.10:4104 , piece_id=MRIWH6WGACE4UDNX55MIY6ZDWBAHS6E5KQS4QCOQN2BU7JI7ODCA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.159562149 (587h24m36.030645184s) [noise=true , address=77.90.31.12:4601 , piece_id=QQFK4PCL27YYF3UATGLV5KDENRNX6NHE7RXAZ2K66CC3WLHEBMWQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.238339784 (587h24m36.10942282s) [noise=true , address=45.91.116.5:4309 , piece_id=Y5HCMU6Z2FPBV3BNTUOC4F6JJWKNBKPOXWFVAFQTXUKQNHTLA4TA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.813503456 (587h24m36.684586501s) [noise=true , address=145.79.227.33:4885 , piece_id=LA755L4RA4VJBHNCQBXOCFROZA3W6W6PUPKGJ5J772WAN44YMY2A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.826964278 (587h24m36.698047313s) [noise=true , address=145.79.227.43:4187 , piece_id=CAAFD3XIWWHZN7L7SYDMMSC5G5RKRLFYYHQZEEN4GAC2D6Y2EARA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.828562510 (587h24m36.699645545s) [noise=true , address=145.79.227.40:4053 , piece_id=3IZP7JBGSWBRBQHMJWXIDD73B5R3COWU5H3IYFGVBCTTLW7OUJ5A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.828575566 (587h24m36.699658601s) [seg_pos={0 0}, upload=514937]: Segment reached optimal threshold of 14 pieces. Dec 30 02:53:36.828752202 (587h24m36.699835237s) [noise=true , address=145.79.227.30:4780 , piece_id=DZQ5LIBMWXPJ4XXWORRDLQIAFXJDNDQEEBRMSBS44UX3D2HPTSWQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125WCbZ6a2NaewiLAHhHuPo91sQEfC9CZzEBV6rwF1JY7z13AEY): protocol: expected piece hash; context canceled; EOF Dec 30 02:53:36.828758714 (587h24m36.699841749s) [noise=true , address=145.79.227.30:4780 , piece_id=DZQ5LIBMWXPJ4XXWORRDLQIAFXJDNDQEEBRMSBS44UX3D2HPTSWQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold Dec 30 02:53:36.828762672 (587h24m36.699845707s) [noise=true , address=145.79.227.51:4429 , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qytmzwy7sTmWXskEbLWsWMniPz55zgho2akE7oh6ahAiT5SXf): protocol: write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection; write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection Dec 30 02:53:36.828774034 (587h24m36.699857069s) [noise=true , address=145.79.227.51:4429 , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold Dec 30 02:53:36.828889934 (587h24m36.699972969s) [noise=true , address=45.91.116.13:4176 , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ejsmBd4tAVwTHjamPYbryQb5x2vDnk9pBuX3CBF9WYFhTrFVJ): protocol: expected piece hash; context canceled; EOF Dec 30 02:53:36.828895144 (587h24m36.699978179s) [noise=true , address=45.91.116.13:4176 , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold Dec 30 02:53:36.828994173 (587h24m36.700077208s) [noise=true , address=145.79.227.49:4340 , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF Dec 30 02:53:36.828999793 (587h24m36.700082828s) [noise=true , address=145.79.227.49:4340 , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold Dec 30 02:53:36.829030332 (587h24m36.700113367s) [seg_pos={0 0}, upload=514937]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 02:53:36.829095486 (587h24m36.700178621s) [upload=514937]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 02:53:36.829100105 (587h24m36.70018315s) [upload=514937]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 02:53:37.128695270 (587h24m36.999778315s) [upload=514937]: Done uploading segments... Dec 30 02:53:37.128717683 (587h24m36.999800728s) [upload=514937]: Upload finished. err: <nil> Duration: 28.013386327s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [..X......X........X...X..XX..X.X......XXX.X.X..X........X.X.] duration: 6.008078009s trace: https://jaeger.dev.storj.io/trace/6f3bfd4d256fd0db FAIL (2025-12-30 03:02:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :01:30.790193548 (587h32m30.661276584s) [upload=515056]: Starting upload Dec 30 03:01:30.790289301 (587h32m30.661372336s) [upload=515056]: Done starting upload Dec 30 03:01:30.790376558 (587h32m30.661459593s) [upload=515056]: Uploading segments... Dec 30 03:01:30.790597177 (587h32m30.661680212s) [upload=515056]: Got next segment. Inline: false Dec 30 03:01:30.790647995 (587h32m30.66173103s) [upload=515056]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 03:01:30.790654317 (587h32m30.661737362s) [upload=515056]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 03:01:31.037317862 (587h32m30.908400907s) [seg_pos={0 0}, upload=515056]: Begin upload segment... Dec 30 03:01:31.037386412 (587h32m30.908469447s) [seg_pos={0 0}, upload=515056]: Done begin upload segment. Dec 30 03:01:31.037443722 (587h32m30.908526767s) [noise=true , address=145.79.227.19:4079 , piece_id=ALA7ILFGDKEOLX7K5CQN474NF5HBGELEGJGZBEEL3ZKOBC3YIDKQ, seg_pos={0 0}, upload=515056]: Uploading piece... Dec 30 03:01:31.037477586 (587h32m30.908560621s) [noise=true , address=45.91.116.6:4976 , piece_id=XZOTFFMCQOILBARF64YIYBL26YNVUCHDHK6YPKNCR777FZV5VH5A, seg_pos={0 0}, upload=515056]: Uploading piece... Dec 30 03:01:31.037943072 (587h32m30.909026107s) [noise=true , address=145.79.227.37:4567 , piece_id=2ROPNJTG63T25PTLYLUTYC2JCJJJUXRHX266UFVGY223ZBGIPRLQ, seg_pos={0 0}, upload=515056]: Uploading piece... Dec 30 03:01:31.038401074 (587h32m30.909484119s) [noise=true , address=145.79.227.16:4912 , piece_id=YFXSOO7U6DYHCD4ION66NQ323ESJUIFBKSHDTU7EAZZP7NFM4DVA, seg_pos={0 0}, upload=515056]: Uploading piece... Dec 30 03:01:31.038807739 (587h32m30.909890774s) [noise=true , address=145.79.227.28:4727 , piece_id=EEXKOAM5LZ2ZURSJ7BYZMXWKDHXIAIPJEM6636ZT3MSK7WFS42EA, seg_pos={0 0}, upload=515056]: Uploading piece... Dec 30 03:01:31.039149600 (587h32m30.910232635s) [noise=true , address=145.79.227.34:4905 , piece_id=NSSGPJ3KGZXKHUAMLCXA7PXEOZWH6EFBOYC3RVJEKHLNEU6R25PA, seg_pos={0 0}, upload=515056]: Uploading piece... Dec 30 03:01:31.039459550 (587h32m30.910542585s) [noise=true , address=145.79.227.10:4402 , piece_id=Q7FGLABYRIWMY4CCB2KJA2BA4U6V3HSOXGNB6QDY3R44L7IBOV6Q, seg_pos={0 0}, upload=515056]: Uploading piece... Dec 30 03:01:31.039797724 (587h32m30.910880769s) [noise=true , address=145.79.227.5:4222 , piece_id=Q5PSW7OHW4SIUKAICJMANFFBTIC6LXXMCFDL6MVBVVJ3OKDBIGDQ, seg_pos={0 0}, upload=515056]: Uploading piece... Dec 30 03:01:31.040165144 (587h32m30.911248179s) [noise=true , address=145.79.227.46:4603 , piece_id=GHRYHC6HQW6UQTDUVBEEN6FUYOG7H6ZHNK7DGZCAGAPNO7WP5AKQ, seg_pos={0 0}, upload=515056]: Uploading piece... Dec 30 03:01:31.041435292 (587h32m30.912518327s) [noise=true , address=145.79.227.14:4839 , piece_id=ZLN5B2OB6UEURYWXWCHGU6Q7PYMR33AI6HI4CJ34ZP6METMXL2PQ, seg_pos={0 0}, upload=515056]: Uploading piece... Dec 30 03:01:31.042805612 (587h32m30.913888647s) [noise=true , address=145.79.227.12:4767 , piece_id=OOQKMKFBMPEG3DRLLLLKYMPDPF4UWOW2Y2YDC2SBKEOMOUU3WUJA, seg_pos={0 0}, upload=515056]: Uploading piece... Dec 30 03:01:31.043957525 (587h32m30.91504057s) [noise=true , address=45.91.116.5:4299 , piece_id=4ETGQI2DFT2EF473DQBLYFYZFVVX3GX5PTMX4F7ZCGVJ66PHXQAA, seg_pos={0 0}, upload=515056]: Uploading piece... Dec 30 03:01:31.045114969 (587h32m30.916198004s) [noise=true , address=145.79.227.45:4230 , piece_id=C33SO4Q2BDHQ5OL7W6IRNTVE7ZEVRVZG2MXVYWHCO2UUB2NNPBBA, seg_pos={0 0}, upload=515056]: Uploading piece... Dec 30 03:01:31.046472343 (587h32m30.917555389s) [noise=true , address=45.91.116.7:4011 , piece_id=XTDQS3FWMPLARRIRFLALSOIDAMK6VOZ4RSRWZJMF5ESIEKA7K3SQ, seg_pos={0 0}, upload=515056]: Uploading piece... Dec 30 03:01:31.046549650 (587h32m30.917632686s) [noise=true , address=145.79.227.26:4655 , piece_id=SF326TKBTCXPBD5KCYJ2DOIJ2MUGSZPZSQQPUWAH7XOQHFILT6GA, seg_pos={0 0}, upload=515056]: Uploading piece... Dec 30 03:01:31.058387201 (587h32m30.929470236s) [noise=true , address=45.91.116.9:4261 , piece_id=WWJ3S3FCCQ6KTYHW4IRAVHT56VXZSVJGX43BPXBFVDB5THRJJDCA, seg_pos={0 0}, upload=515056]: Uploading piece... Dec 30 03:01:32.645269112 (587h32m32.516352147s) [upload=515056]: Next returned nil segment Dec 30 03:01:32.645279483 (587h32m32.516362528s) [upload=515056]: Waiting for error group managing segments... Dec 30 03:01:33.025385394 (587h32m32.896468439s) [noise=true , address=145.79.227.46:4603 , piece_id=GHRYHC6HQW6UQTDUVBEEN6FUYOG7H6ZHNK7DGZCAGAPNO7WP5AKQ, seg_pos={0 0}, upload=515056]: Done uploading piece. err: <nil> Dec 30 03:01:33.027232090 (587h32m32.898315125s) [noise=true , address=145.79.227.14:4839 , piece_id=ZLN5B2OB6UEURYWXWCHGU6Q7PYMR33AI6HI4CJ34ZP6METMXL2PQ, seg_pos={0 0}, upload=515056]: Done uploading piece. err: <nil> Dec 30 03:01:33.027742612 (587h32m32.898825657s) [noise=true , address=145.79.227.16:4912 , piece_id=YFXSOO7U6DYHCD4ION66NQ323ESJUIFBKSHDTU7EAZZP7NFM4DVA, seg_pos={0 0}, upload=515056]: Done uploading piece. err: <nil> Dec 30 03:01:33.031832290 (587h32m32.902915335s) [noise=true , address=145.79.227.5:4222 , piece_id=Q5PSW7OHW4SIUKAICJMANFFBTIC6LXXMCFDL6MVBVVJ3OKDBIGDQ, seg_pos={0 0}, upload=515056]: Done uploading piece. err: <nil> Dec 30 03:01:33.035842285 (587h32m32.90692533s) [noise=true , address=145.79.227.19:4079 , piece_id=ALA7ILFGDKEOLX7K5CQN474NF5HBGELEGJGZBEEL3ZKOBC3YIDKQ, seg_pos={0 0}, upload=515056]: Done uploading piece. err: <nil> Dec 30 03:01:33.038642507 (587h32m32.909725542s) [noise=true , address=145.79.227.45:4230 , piece_id=C33SO4Q2BDHQ5OL7W6IRNTVE7ZEVRVZG2MXVYWHCO2UUB2NNPBBA, seg_pos={0 0}, upload=515056]: Done uploading piece. err: <nil> Dec 30 03:01:33.047324690 (587h32m32.918407735s) [noise=true , address=145.79.227.26:4655 , piece_id=SF326TKBTCXPBD5KCYJ2DOIJ2MUGSZPZSQQPUWAH7XOQHFILT6GA, seg_pos={0 0}, upload=515056]: Done uploading piece. err: <nil> Dec 30 03:01:33.050767214 (587h32m32.921850249s) [noise=true , address=145.79.227.28:4727 , piece_id=EEXKOAM5LZ2ZURSJ7BYZMXWKDHXIAIPJEM6636ZT3MSK7WFS42EA, seg_pos={0 0}, upload=515056]: Done uploading piece. err: <nil> Dec 30 03:01:33.071618338 (587h32m32.942701373s) [noise=true , address=145.79.227.10:4402 , piece_id=Q7FGLABYRIWMY4CCB2KJA2BA4U6V3HSOXGNB6QDY3R44L7IBOV6Q, seg_pos={0 0}, upload=515056]: Done uploading piece. err: <nil> Dec 30 03:01:33.402498867 (587h32m33.273581912s) [noise=true , address=45.91.116.5:4299 , piece_id=4ETGQI2DFT2EF473DQBLYFYZFVVX3GX5PTMX4F7ZCGVJ66PHXQAA, seg_pos={0 0}, upload=515056]: Done uploading piece. err: <nil> Dec 30 03:01:34.053963436 (587h32m33.925046471s) [noise=true , address=45.91.116.6:4976 , piece_id=XZOTFFMCQOILBARF64YIYBL26YNVUCHDHK6YPKNCR777FZV5VH5A, seg_pos={0 0}, upload=515056]: Done uploading piece. err: <nil> Dec 30 03:01:34.154566471 (587h32m34.025649516s) [noise=true , address=145.79.227.12:4767 , piece_id=OOQKMKFBMPEG3DRLLLLKYMPDPF4UWOW2Y2YDC2SBKEOMOUU3WUJA, seg_pos={0 0}, upload=515056]: Done uploading piece. err: <nil> Dec 30 03:01:34.166289793 (587h32m34.037372828s) [noise=true , address=45.91.116.9:4261 , piece_id=WWJ3S3FCCQ6KTYHW4IRAVHT56VXZSVJGX43BPXBFVDB5THRJJDCA, seg_pos={0 0}, upload=515056]: Done uploading piece. err: <nil> Dec 30 03:01:34.166316595 (587h32m34.03739963s) [seg_pos={0 0}, upload=515056]: Segment reached optimal threshold of 13 pieces. Dec 30 03:01:34.166630082 (587h32m34.037713147s) [noise=true , address=45.91.116.7:4011 , piece_id=XTDQS3FWMPLARRIRFLALSOIDAMK6VOZ4RSRWZJMF5ESIEKA7K3SQ, seg_pos={0 0}, upload=515056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 30 03:01:34.166641544 (587h32m34.037724589s) [noise=true , address=45.91.116.7:4011 , piece_id=XTDQS3FWMPLARRIRFLALSOIDAMK6VOZ4RSRWZJMF5ESIEKA7K3SQ, seg_pos={0 0}, upload=515056]: Long tail cancellation due to optimal threshold Dec 30 03:01:34.166722768 (587h32m34.037805803s) [noise=true , address=145.79.227.37:4567 , piece_id=2ROPNJTG63T25PTLYLUTYC2JCJJJUXRHX266UFVGY223ZBGIPRLQ, seg_pos={0 0}, upload=515056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: write tcp 188.166.225.170:43386->145.79.227.37:4567: use of closed network connection; write tcp 188.166.225.170:43386->145.79.227.37:4567: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43386->145.79.227.37:4567: use of closed network connection Dec 30 03:01:34.166736494 (587h32m34.037819529s) [noise=true , address=145.79.227.37:4567 , piece_id=2ROPNJTG63T25PTLYLUTYC2JCJJJUXRHX266UFVGY223ZBGIPRLQ, seg_pos={0 0}, upload=515056]: Long tail cancellation due to optimal threshold Dec 30 03:01:34.166794224 (587h32m34.037877279s) [noise=true , address=145.79.227.34:4905 , piece_id=NSSGPJ3KGZXKHUAMLCXA7PXEOZWH6EFBOYC3RVJEKHLNEU6R25PA, seg_pos={0 0}, upload=515056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 30 03:01:34.166803652 (587h32m34.037886687s) [noise=true , address=145.79.227.34:4905 , piece_id=NSSGPJ3KGZXKHUAMLCXA7PXEOZWH6EFBOYC3RVJEKHLNEU6R25PA, seg_pos={0 0}, upload=515056]: Long tail cancellation due to optimal threshold Dec 30 03:01:34.166832527 (587h32m34.037915572s) [seg_pos={0 0}, upload=515056]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 03:01:34.166959700 (587h32m34.038042745s) [upload=515056]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 03:01:34.166968556 (587h32m34.038051591s) [upload=515056]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 03:01:34.454039895 (587h32m34.32512294s) [upload=515056]: Done uploading segments... Dec 30 03:01:34.454209839 (587h32m34.325292894s) [upload=515056]: Upload finished. err: <nil> Duration: 6.008094671s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [X..X......X........X...X..XX..X.X......XXX.X.X..X........X.X] duration: 30.00356161s trace: https://jaeger.dev.storj.io/trace/6f3bfd4d256fd0db FAIL (2025-12-30 03:02:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :01:30.790193548 (587h32m30.661276584s) [upload=515056]: Starting upload Dec 30 03:01:30.790289301 (587h32m30.661372336s) [upload=515056]: Done starting upload Dec 30 03:01:30.790376558 (587h32m30.661459593s) [upload=515056]: Uploading segments... Dec 30 03:01:30.790597177 (587h32m30.661680212s) [upload=515056]: Got next segment. Inline: false Dec 30 03:01:30.790647995 (587h32m30.66173103s) [upload=515056]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 03:01:30.790654317 (587h32m30.661737362s) [upload=515056]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 03:01:31.037317862 (587h32m30.908400907s) [seg_pos={0 0}, upload=515056]: Begin upload segment... Dec 30 03:01:31.037386412 (587h32m30.908469447s) [seg_pos={0 0}, upload=515056]: Done begin upload segment. Dec 30 03:01:31.037443722 (587h32m30.908526767s) [noise=true , address=145.79.227.19:4079 , piece_id=ALA7ILFGDKEOLX7K5CQN474NF5HBGELEGJGZBEEL3ZKOBC3YIDKQ, seg_pos={0 0}, upload=515056]: Uploading piece... Dec 30 03:01:31.037477586 (587h32m30.908560621s) [noise=true , address=45.91.116.6:4976 , piece_id=XZOTFFMCQOILBARF64YIYBL26YNVUCHDHK6YPKNCR777FZV5VH5A, seg_pos={0 0}, upload=515056]: Uploading piece... Dec 30 03:01:31.037943072 (587h32m30.909026107s) [noise=true , address=145.79.227.37:4567 , piece_id=2ROPNJTG63T25PTLYLUTYC2JCJJJUXRHX266UFVGY223ZBGIPRLQ, seg_pos={0 0}, upload=515056]: Uploading piece... Dec 30 03:01:31.038401074 (587h32m30.909484119s) [noise=true , address=145.79.227.16:4912 , piece_id=YFXSOO7U6DYHCD4ION66NQ323ESJUIFBKSHDTU7EAZZP7NFM4DVA, seg_pos={0 0}, upload=515056]: Uploading piece... Dec 30 03:01:31.038807739 (587h32m30.909890774s) [noise=true , address=145.79.227.28:4727 , piece_id=EEXKOAM5LZ2ZURSJ7BYZMXWKDHXIAIPJEM6636ZT3MSK7WFS42EA, seg_pos={0 0}, upload=515056]: Uploading piece... Dec 30 03:01:31.039149600 (587h32m30.910232635s) [noise=true , address=145.79.227.34:4905 , piece_id=NSSGPJ3KGZXKHUAMLCXA7PXEOZWH6EFBOYC3RVJEKHLNEU6R25PA, seg_pos={0 0}, upload=515056]: Uploading piece... Dec 30 03:01:31.039459550 (587h32m30.910542585s) [noise=true , address=145.79.227.10:4402 , piece_id=Q7FGLABYRIWMY4CCB2KJA2BA4U6V3HSOXGNB6QDY3R44L7IBOV6Q, seg_pos={0 0}, upload=515056]: Uploading piece... Dec 30 03:01:31.039797724 (587h32m30.910880769s) [noise=true , address=145.79.227.5:4222 , piece_id=Q5PSW7OHW4SIUKAICJMANFFBTIC6LXXMCFDL6MVBVVJ3OKDBIGDQ, seg_pos={0 0}, upload=515056]: Uploading piece... Dec 30 03:01:31.040165144 (587h32m30.911248179s) [noise=true , address=145.79.227.46:4603 , piece_id=GHRYHC6HQW6UQTDUVBEEN6FUYOG7H6ZHNK7DGZCAGAPNO7WP5AKQ, seg_pos={0 0}, upload=515056]: Uploading piece... Dec 30 03:01:31.041435292 (587h32m30.912518327s) [noise=true , address=145.79.227.14:4839 , piece_id=ZLN5B2OB6UEURYWXWCHGU6Q7PYMR33AI6HI4CJ34ZP6METMXL2PQ, seg_pos={0 0}, upload=515056]: Uploading piece... Dec 30 03:01:31.042805612 (587h32m30.913888647s) [noise=true , address=145.79.227.12:4767 , piece_id=OOQKMKFBMPEG3DRLLLLKYMPDPF4UWOW2Y2YDC2SBKEOMOUU3WUJA, seg_pos={0 0}, upload=515056]: Uploading piece... Dec 30 03:01:31.043957525 (587h32m30.91504057s) [noise=true , address=45.91.116.5:4299 , piece_id=4ETGQI2DFT2EF473DQBLYFYZFVVX3GX5PTMX4F7ZCGVJ66PHXQAA, seg_pos={0 0}, upload=515056]: Uploading piece... Dec 30 03:01:31.045114969 (587h32m30.916198004s) [noise=true , address=145.79.227.45:4230 , piece_id=C33SO4Q2BDHQ5OL7W6IRNTVE7ZEVRVZG2MXVYWHCO2UUB2NNPBBA, seg_pos={0 0}, upload=515056]: Uploading piece... Dec 30 03:01:31.046472343 (587h32m30.917555389s) [noise=true , address=45.91.116.7:4011 , piece_id=XTDQS3FWMPLARRIRFLALSOIDAMK6VOZ4RSRWZJMF5ESIEKA7K3SQ, seg_pos={0 0}, upload=515056]: Uploading piece... Dec 30 03:01:31.046549650 (587h32m30.917632686s) [noise=true , address=145.79.227.26:4655 , piece_id=SF326TKBTCXPBD5KCYJ2DOIJ2MUGSZPZSQQPUWAH7XOQHFILT6GA, seg_pos={0 0}, upload=515056]: Uploading piece... Dec 30 03:01:31.058387201 (587h32m30.929470236s) [noise=true , address=45.91.116.9:4261 , piece_id=WWJ3S3FCCQ6KTYHW4IRAVHT56VXZSVJGX43BPXBFVDB5THRJJDCA, seg_pos={0 0}, upload=515056]: Uploading piece... Dec 30 03:01:32.645269112 (587h32m32.516352147s) [upload=515056]: Next returned nil segment Dec 30 03:01:32.645279483 (587h32m32.516362528s) [upload=515056]: Waiting for error group managing segments... Dec 30 03:01:33.025385394 (587h32m32.896468439s) [noise=true , address=145.79.227.46:4603 , piece_id=GHRYHC6HQW6UQTDUVBEEN6FUYOG7H6ZHNK7DGZCAGAPNO7WP5AKQ, seg_pos={0 0}, upload=515056]: Done uploading piece. err: <nil> Dec 30 03:01:33.027232090 (587h32m32.898315125s) [noise=true , address=145.79.227.14:4839 , piece_id=ZLN5B2OB6UEURYWXWCHGU6Q7PYMR33AI6HI4CJ34ZP6METMXL2PQ, seg_pos={0 0}, upload=515056]: Done uploading piece. err: <nil> Dec 30 03:01:33.027742612 (587h32m32.898825657s) [noise=true , address=145.79.227.16:4912 , piece_id=YFXSOO7U6DYHCD4ION66NQ323ESJUIFBKSHDTU7EAZZP7NFM4DVA, seg_pos={0 0}, upload=515056]: Done uploading piece. err: <nil> Dec 30 03:01:33.031832290 (587h32m32.902915335s) [noise=true , address=145.79.227.5:4222 , piece_id=Q5PSW7OHW4SIUKAICJMANFFBTIC6LXXMCFDL6MVBVVJ3OKDBIGDQ, seg_pos={0 0}, upload=515056]: Done uploading piece. err: <nil> Dec 30 03:01:33.035842285 (587h32m32.90692533s) [noise=true , address=145.79.227.19:4079 , piece_id=ALA7ILFGDKEOLX7K5CQN474NF5HBGELEGJGZBEEL3ZKOBC3YIDKQ, seg_pos={0 0}, upload=515056]: Done uploading piece. err: <nil> Dec 30 03:01:33.038642507 (587h32m32.909725542s) [noise=true , address=145.79.227.45:4230 , piece_id=C33SO4Q2BDHQ5OL7W6IRNTVE7ZEVRVZG2MXVYWHCO2UUB2NNPBBA, seg_pos={0 0}, upload=515056]: Done uploading piece. err: <nil> Dec 30 03:01:33.047324690 (587h32m32.918407735s) [noise=true , address=145.79.227.26:4655 , piece_id=SF326TKBTCXPBD5KCYJ2DOIJ2MUGSZPZSQQPUWAH7XOQHFILT6GA, seg_pos={0 0}, upload=515056]: Done uploading piece. err: <nil> Dec 30 03:01:33.050767214 (587h32m32.921850249s) [noise=true , address=145.79.227.28:4727 , piece_id=EEXKOAM5LZ2ZURSJ7BYZMXWKDHXIAIPJEM6636ZT3MSK7WFS42EA, seg_pos={0 0}, upload=515056]: Done uploading piece. err: <nil> Dec 30 03:01:33.071618338 (587h32m32.942701373s) [noise=true , address=145.79.227.10:4402 , piece_id=Q7FGLABYRIWMY4CCB2KJA2BA4U6V3HSOXGNB6QDY3R44L7IBOV6Q, seg_pos={0 0}, upload=515056]: Done uploading piece. err: <nil> Dec 30 03:01:33.402498867 (587h32m33.273581912s) [noise=true , address=45.91.116.5:4299 , piece_id=4ETGQI2DFT2EF473DQBLYFYZFVVX3GX5PTMX4F7ZCGVJ66PHXQAA, seg_pos={0 0}, upload=515056]: Done uploading piece. err: <nil> Dec 30 03:01:34.053963436 (587h32m33.925046471s) [noise=true , address=45.91.116.6:4976 , piece_id=XZOTFFMCQOILBARF64YIYBL26YNVUCHDHK6YPKNCR777FZV5VH5A, seg_pos={0 0}, upload=515056]: Done uploading piece. err: <nil> Dec 30 03:01:34.154566471 (587h32m34.025649516s) [noise=true , address=145.79.227.12:4767 , piece_id=OOQKMKFBMPEG3DRLLLLKYMPDPF4UWOW2Y2YDC2SBKEOMOUU3WUJA, seg_pos={0 0}, upload=515056]: Done uploading piece. err: <nil> Dec 30 03:01:34.166289793 (587h32m34.037372828s) [noise=true , address=45.91.116.9:4261 , piece_id=WWJ3S3FCCQ6KTYHW4IRAVHT56VXZSVJGX43BPXBFVDB5THRJJDCA, seg_pos={0 0}, upload=515056]: Done uploading piece. err: <nil> Dec 30 03:01:34.166316595 (587h32m34.03739963s) [seg_pos={0 0}, upload=515056]: Segment reached optimal threshold of 13 pieces. Dec 30 03:01:34.166630082 (587h32m34.037713147s) [noise=true , address=45.91.116.7:4011 , piece_id=XTDQS3FWMPLARRIRFLALSOIDAMK6VOZ4RSRWZJMF5ESIEKA7K3SQ, seg_pos={0 0}, upload=515056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 30 03:01:34.166641544 (587h32m34.037724589s) [noise=true , address=45.91.116.7:4011 , piece_id=XTDQS3FWMPLARRIRFLALSOIDAMK6VOZ4RSRWZJMF5ESIEKA7K3SQ, seg_pos={0 0}, upload=515056]: Long tail cancellation due to optimal threshold Dec 30 03:01:34.166722768 (587h32m34.037805803s) [noise=true , address=145.79.227.37:4567 , piece_id=2ROPNJTG63T25PTLYLUTYC2JCJJJUXRHX266UFVGY223ZBGIPRLQ, seg_pos={0 0}, upload=515056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: write tcp 188.166.225.170:43386->145.79.227.37:4567: use of closed network connection; write tcp 188.166.225.170:43386->145.79.227.37:4567: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43386->145.79.227.37:4567: use of closed network connection Dec 30 03:01:34.166736494 (587h32m34.037819529s) [noise=true , address=145.79.227.37:4567 , piece_id=2ROPNJTG63T25PTLYLUTYC2JCJJJUXRHX266UFVGY223ZBGIPRLQ, seg_pos={0 0}, upload=515056]: Long tail cancellation due to optimal threshold Dec 30 03:01:34.166794224 (587h32m34.037877279s) [noise=true , address=145.79.227.34:4905 , piece_id=NSSGPJ3KGZXKHUAMLCXA7PXEOZWH6EFBOYC3RVJEKHLNEU6R25PA, seg_pos={0 0}, upload=515056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 30 03:01:34.166803652 (587h32m34.037886687s) [noise=true , address=145.79.227.34:4905 , piece_id=NSSGPJ3KGZXKHUAMLCXA7PXEOZWH6EFBOYC3RVJEKHLNEU6R25PA, seg_pos={0 0}, upload=515056]: Long tail cancellation due to optimal threshold Dec 30 03:01:34.166832527 (587h32m34.037915572s) [seg_pos={0 0}, upload=515056]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 03:01:34.166959700 (587h32m34.038042745s) [upload=515056]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 03:01:34.166968556 (587h32m34.038051591s) [upload=515056]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 03:01:34.454039895 (587h32m34.32512294s) [upload=515056]: Done uploading segments... Dec 30 03:01:34.454209839 (587h32m34.325292894s) [upload=515056]: Upload finished. err: <nil> Duration: 30.003587549s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [...........X.............X................X......X..........] duration: 27.828879844s trace: https://jaeger.dev.storj.io/trace/6c3caa8adcfacbb9 FAIL (2025-12-30 03:01:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== true , address=145.79.227.35:4496 , piece_id=5X54ZGU4OIVI3DT622QE7IL66SVFLZYUICPODGMDKN7U4D4OJNSA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.040303259 (587h24m33.911386294s) [noise=true , address=77.90.31.12:4601 , piece_id=QQFK4PCL27YYF3UATGLV5KDENRNX6NHE7RXAZ2K66CC3WLHEBMWQ, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.040670117 (587h24m33.911753152s) [noise=true , address=77.90.31.21:4881 , piece_id=EMV4LHFK2VRHSW6PD3BBYGOXDAAYJKX6GEOXGGLOT3IXWNDQMU4A, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041006247 (587h24m33.912089292s) [noise=true , address=145.79.227.52:4472 , piece_id=DT2DAIC4ZI3WL7OGY55ZTTXJEI4IDRXPSJIQA3LTRC7K2F6EVPCA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041183314 (587h24m33.912266349s) [noise=true , address=145.79.227.26:4674 , piece_id=3BAN2ZXOFTSGZITYYHMZUS4AD35X6ZBOYA2HKOJCJI7VBWPJ3LSA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041313653 (587h24m33.912396688s) [noise=true , address=145.79.227.51:4429 , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041535194 (587h24m33.912618229s) [noise=true , address=145.79.227.33:4885 , piece_id=LA755L4RA4VJBHNCQBXOCFROZA3W6W6PUPKGJ5J772WAN44YMY2A, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041639693 (587h24m33.912722728s) [noise=true , address=145.79.227.43:4187 , piece_id=CAAFD3XIWWHZN7L7SYDMMSC5G5RKRLFYYHQZEEN4GAC2D6Y2EARA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041877216 (587h24m33.912960261s) [noise=true , address=145.79.227.46:4623 , piece_id=V4TEM7RRTTGBW5ISZOEAOPFIOZZ5EABO7BPWNRH5U5UPM2X7554Q, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041986634 (587h24m33.913069669s) [noise=true , address=45.91.116.13:4176 , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.042345657 (587h24m33.913428692s) [noise=true , address=45.91.116.10:4104 , piece_id=MRIWH6WGACE4UDNX55MIY6ZDWBAHS6E5KQS4QCOQN2BU7JI7ODCA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.042362019 (587h24m33.913445054s) [noise=true , address=77.90.31.11:4573 , piece_id=TORCC4W7DS5TJ7H7NWONEPNJOFULSZ6F5OHOP4BEVMGEH2MJA4DQ, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.042684192 (587h24m33.913767237s) [noise=true , address=145.79.227.49:4340 , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.042707327 (587h24m33.913790362s) [noise=true , address=145.79.227.32:4837 , piece_id=SQTHTF5KWZOKFYX6CGL6KCDRDAY7OJY6NA5OM3N4YCICDPM6DKXA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.043091528 (587h24m33.914174563s) [noise=true , address=45.91.116.5:4309 , piece_id=Y5HCMU6Z2FPBV3BNTUOC4F6JJWKNBKPOXWFVAFQTXUKQNHTLA4TA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.043544771 (587h24m33.914627816s) [noise=true , address=145.79.227.40:4053 , piece_id=3IZP7JBGSWBRBQHMJWXIDD73B5R3COWU5H3IYFGVBCTTLW7OUJ5A, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:35.347637489 (587h24m35.218720524s) [upload=514937]: Next returned nil segment Dec 30 02:53:35.347733591 (587h24m35.218816626s) [upload=514937]: Waiting for error group managing segments... Dec 30 02:53:35.674169656 (587h24m35.545252721s) [noise=true , address=64.49.13.27:4959 , piece_id=B4UZ67DIWNRHV77VTQONS6ZBUI2BPDDBM3ZS6FVCTABQZ6OHRJJA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:35.891295347 (587h24m35.762378382s) [noise=true , address=145.79.227.32:4837 , piece_id=SQTHTF5KWZOKFYX6CGL6KCDRDAY7OJY6NA5OM3N4YCICDPM6DKXA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:35.912619632 (587h24m35.783702677s) [noise=true , address=145.79.227.26:4674 , piece_id=3BAN2ZXOFTSGZITYYHMZUS4AD35X6ZBOYA2HKOJCJI7VBWPJ3LSA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:35.913054741 (587h24m35.784137786s) [noise=true , address=145.79.227.46:4623 , piece_id=V4TEM7RRTTGBW5ISZOEAOPFIOZZ5EABO7BPWNRH5U5UPM2X7554Q, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:35.913485270 (587h24m35.784568305s) [noise=true , address=145.79.227.35:4496 , piece_id=5X54ZGU4OIVI3DT622QE7IL66SVFLZYUICPODGMDKN7U4D4OJNSA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:35.923790864 (587h24m35.794873909s) [noise=true , address=145.79.227.52:4472 , piece_id=DT2DAIC4ZI3WL7OGY55ZTTXJEI4IDRXPSJIQA3LTRC7K2F6EVPCA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.109605534 (587h24m35.980688569s) [noise=true , address=77.90.31.21:4881 , piece_id=EMV4LHFK2VRHSW6PD3BBYGOXDAAYJKX6GEOXGGLOT3IXWNDQMU4A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.122502652 (587h24m35.993585697s) [noise=true , address=77.90.31.11:4573 , piece_id=TORCC4W7DS5TJ7H7NWONEPNJOFULSZ6F5OHOP4BEVMGEH2MJA4DQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.151131466 (587h24m36.022214511s) [noise=true , address=45.91.116.10:4104 , piece_id=MRIWH6WGACE4UDNX55MIY6ZDWBAHS6E5KQS4QCOQN2BU7JI7ODCA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.159562149 (587h24m36.030645184s) [noise=true , address=77.90.31.12:4601 , piece_id=QQFK4PCL27YYF3UATGLV5KDENRNX6NHE7RXAZ2K66CC3WLHEBMWQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.238339784 (587h24m36.10942282s) [noise=true , address=45.91.116.5:4309 , piece_id=Y5HCMU6Z2FPBV3BNTUOC4F6JJWKNBKPOXWFVAFQTXUKQNHTLA4TA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.813503456 (587h24m36.684586501s) [noise=true , address=145.79.227.33:4885 , piece_id=LA755L4RA4VJBHNCQBXOCFROZA3W6W6PUPKGJ5J772WAN44YMY2A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.826964278 (587h24m36.698047313s) [noise=true , address=145.79.227.43:4187 , piece_id=CAAFD3XIWWHZN7L7SYDMMSC5G5RKRLFYYHQZEEN4GAC2D6Y2EARA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.828562510 (587h24m36.699645545s) [noise=true , address=145.79.227.40:4053 , piece_id=3IZP7JBGSWBRBQHMJWXIDD73B5R3COWU5H3IYFGVBCTTLW7OUJ5A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.828575566 (587h24m36.699658601s) [seg_pos={0 0}, upload=514937]: Segment reached optimal threshold of 14 pieces. Dec 30 02:53:36.828752202 (587h24m36.699835237s) [noise=true , address=145.79.227.30:4780 , piece_id=DZQ5LIBMWXPJ4XXWORRDLQIAFXJDNDQEEBRMSBS44UX3D2HPTSWQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125WCbZ6a2NaewiLAHhHuPo91sQEfC9CZzEBV6rwF1JY7z13AEY): protocol: expected piece hash; context canceled; EOF Dec 30 02:53:36.828758714 (587h24m36.699841749s) [noise=true , address=145.79.227.30:4780 , piece_id=DZQ5LIBMWXPJ4XXWORRDLQIAFXJDNDQEEBRMSBS44UX3D2HPTSWQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold Dec 30 02:53:36.828762672 (587h24m36.699845707s) [noise=true , address=145.79.227.51:4429 , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qytmzwy7sTmWXskEbLWsWMniPz55zgho2akE7oh6ahAiT5SXf): protocol: write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection; write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection Dec 30 02:53:36.828774034 (587h24m36.699857069s) [noise=true , address=145.79.227.51:4429 , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold Dec 30 02:53:36.828889934 (587h24m36.699972969s) [noise=true , address=45.91.116.13:4176 , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ejsmBd4tAVwTHjamPYbryQb5x2vDnk9pBuX3CBF9WYFhTrFVJ): protocol: expected piece hash; context canceled; EOF Dec 30 02:53:36.828895144 (587h24m36.699978179s) [noise=true , address=45.91.116.13:4176 , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold Dec 30 02:53:36.828994173 (587h24m36.700077208s) [noise=true , address=145.79.227.49:4340 , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF Dec 30 02:53:36.828999793 (587h24m36.700082828s) [noise=true , address=145.79.227.49:4340 , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold Dec 30 02:53:36.829030332 (587h24m36.700113367s) [seg_pos={0 0}, upload=514937]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 02:53:36.829095486 (587h24m36.700178621s) [upload=514937]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 02:53:36.829100105 (587h24m36.70018315s) [upload=514937]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 02:53:37.128695270 (587h24m36.999778315s) [upload=514937]: Done uploading segments... Dec 30 02:53:37.128717683 (587h24m36.999800728s) [upload=514937]: Upload finished. err: <nil> Duration: 27.828907117s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [...X...................................X...X.X.XX...........] duration: 3.338734061s trace: https://jaeger.dev.storj.io/trace/14b283cf2d9b0313 FAIL (2025-12-30 03:01:34 UTC) orbiter warning: context deadline exceeded Duration: 3.338761564s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [............X.............X................X......X.........] duration: 45.986273857s trace: https://jaeger.dev.storj.io/trace/6c3caa8adcfacbb9 FAIL (2025-12-30 03:01:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== true , address=145.79.227.35:4496 , piece_id=5X54ZGU4OIVI3DT622QE7IL66SVFLZYUICPODGMDKN7U4D4OJNSA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.040303259 (587h24m33.911386294s) [noise=true , address=77.90.31.12:4601 , piece_id=QQFK4PCL27YYF3UATGLV5KDENRNX6NHE7RXAZ2K66CC3WLHEBMWQ, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.040670117 (587h24m33.911753152s) [noise=true , address=77.90.31.21:4881 , piece_id=EMV4LHFK2VRHSW6PD3BBYGOXDAAYJKX6GEOXGGLOT3IXWNDQMU4A, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041006247 (587h24m33.912089292s) [noise=true , address=145.79.227.52:4472 , piece_id=DT2DAIC4ZI3WL7OGY55ZTTXJEI4IDRXPSJIQA3LTRC7K2F6EVPCA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041183314 (587h24m33.912266349s) [noise=true , address=145.79.227.26:4674 , piece_id=3BAN2ZXOFTSGZITYYHMZUS4AD35X6ZBOYA2HKOJCJI7VBWPJ3LSA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041313653 (587h24m33.912396688s) [noise=true , address=145.79.227.51:4429 , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041535194 (587h24m33.912618229s) [noise=true , address=145.79.227.33:4885 , piece_id=LA755L4RA4VJBHNCQBXOCFROZA3W6W6PUPKGJ5J772WAN44YMY2A, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041639693 (587h24m33.912722728s) [noise=true , address=145.79.227.43:4187 , piece_id=CAAFD3XIWWHZN7L7SYDMMSC5G5RKRLFYYHQZEEN4GAC2D6Y2EARA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041877216 (587h24m33.912960261s) [noise=true , address=145.79.227.46:4623 , piece_id=V4TEM7RRTTGBW5ISZOEAOPFIOZZ5EABO7BPWNRH5U5UPM2X7554Q, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041986634 (587h24m33.913069669s) [noise=true , address=45.91.116.13:4176 , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.042345657 (587h24m33.913428692s) [noise=true , address=45.91.116.10:4104 , piece_id=MRIWH6WGACE4UDNX55MIY6ZDWBAHS6E5KQS4QCOQN2BU7JI7ODCA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.042362019 (587h24m33.913445054s) [noise=true , address=77.90.31.11:4573 , piece_id=TORCC4W7DS5TJ7H7NWONEPNJOFULSZ6F5OHOP4BEVMGEH2MJA4DQ, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.042684192 (587h24m33.913767237s) [noise=true , address=145.79.227.49:4340 , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.042707327 (587h24m33.913790362s) [noise=true , address=145.79.227.32:4837 , piece_id=SQTHTF5KWZOKFYX6CGL6KCDRDAY7OJY6NA5OM3N4YCICDPM6DKXA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.043091528 (587h24m33.914174563s) [noise=true , address=45.91.116.5:4309 , piece_id=Y5HCMU6Z2FPBV3BNTUOC4F6JJWKNBKPOXWFVAFQTXUKQNHTLA4TA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.043544771 (587h24m33.914627816s) [noise=true , address=145.79.227.40:4053 , piece_id=3IZP7JBGSWBRBQHMJWXIDD73B5R3COWU5H3IYFGVBCTTLW7OUJ5A, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:35.347637489 (587h24m35.218720524s) [upload=514937]: Next returned nil segment Dec 30 02:53:35.347733591 (587h24m35.218816626s) [upload=514937]: Waiting for error group managing segments... Dec 30 02:53:35.674169656 (587h24m35.545252721s) [noise=true , address=64.49.13.27:4959 , piece_id=B4UZ67DIWNRHV77VTQONS6ZBUI2BPDDBM3ZS6FVCTABQZ6OHRJJA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:35.891295347 (587h24m35.762378382s) [noise=true , address=145.79.227.32:4837 , piece_id=SQTHTF5KWZOKFYX6CGL6KCDRDAY7OJY6NA5OM3N4YCICDPM6DKXA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:35.912619632 (587h24m35.783702677s) [noise=true , address=145.79.227.26:4674 , piece_id=3BAN2ZXOFTSGZITYYHMZUS4AD35X6ZBOYA2HKOJCJI7VBWPJ3LSA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:35.913054741 (587h24m35.784137786s) [noise=true , address=145.79.227.46:4623 , piece_id=V4TEM7RRTTGBW5ISZOEAOPFIOZZ5EABO7BPWNRH5U5UPM2X7554Q, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:35.913485270 (587h24m35.784568305s) [noise=true , address=145.79.227.35:4496 , piece_id=5X54ZGU4OIVI3DT622QE7IL66SVFLZYUICPODGMDKN7U4D4OJNSA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:35.923790864 (587h24m35.794873909s) [noise=true , address=145.79.227.52:4472 , piece_id=DT2DAIC4ZI3WL7OGY55ZTTXJEI4IDRXPSJIQA3LTRC7K2F6EVPCA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.109605534 (587h24m35.980688569s) [noise=true , address=77.90.31.21:4881 , piece_id=EMV4LHFK2VRHSW6PD3BBYGOXDAAYJKX6GEOXGGLOT3IXWNDQMU4A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.122502652 (587h24m35.993585697s) [noise=true , address=77.90.31.11:4573 , piece_id=TORCC4W7DS5TJ7H7NWONEPNJOFULSZ6F5OHOP4BEVMGEH2MJA4DQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.151131466 (587h24m36.022214511s) [noise=true , address=45.91.116.10:4104 , piece_id=MRIWH6WGACE4UDNX55MIY6ZDWBAHS6E5KQS4QCOQN2BU7JI7ODCA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.159562149 (587h24m36.030645184s) [noise=true , address=77.90.31.12:4601 , piece_id=QQFK4PCL27YYF3UATGLV5KDENRNX6NHE7RXAZ2K66CC3WLHEBMWQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.238339784 (587h24m36.10942282s) [noise=true , address=45.91.116.5:4309 , piece_id=Y5HCMU6Z2FPBV3BNTUOC4F6JJWKNBKPOXWFVAFQTXUKQNHTLA4TA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.813503456 (587h24m36.684586501s) [noise=true , address=145.79.227.33:4885 , piece_id=LA755L4RA4VJBHNCQBXOCFROZA3W6W6PUPKGJ5J772WAN44YMY2A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.826964278 (587h24m36.698047313s) [noise=true , address=145.79.227.43:4187 , piece_id=CAAFD3XIWWHZN7L7SYDMMSC5G5RKRLFYYHQZEEN4GAC2D6Y2EARA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.828562510 (587h24m36.699645545s) [noise=true , address=145.79.227.40:4053 , piece_id=3IZP7JBGSWBRBQHMJWXIDD73B5R3COWU5H3IYFGVBCTTLW7OUJ5A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.828575566 (587h24m36.699658601s) [seg_pos={0 0}, upload=514937]: Segment reached optimal threshold of 14 pieces. Dec 30 02:53:36.828752202 (587h24m36.699835237s) [noise=true , address=145.79.227.30:4780 , piece_id=DZQ5LIBMWXPJ4XXWORRDLQIAFXJDNDQEEBRMSBS44UX3D2HPTSWQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125WCbZ6a2NaewiLAHhHuPo91sQEfC9CZzEBV6rwF1JY7z13AEY): protocol: expected piece hash; context canceled; EOF Dec 30 02:53:36.828758714 (587h24m36.699841749s) [noise=true , address=145.79.227.30:4780 , piece_id=DZQ5LIBMWXPJ4XXWORRDLQIAFXJDNDQEEBRMSBS44UX3D2HPTSWQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold Dec 30 02:53:36.828762672 (587h24m36.699845707s) [noise=true , address=145.79.227.51:4429 , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qytmzwy7sTmWXskEbLWsWMniPz55zgho2akE7oh6ahAiT5SXf): protocol: write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection; write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection Dec 30 02:53:36.828774034 (587h24m36.699857069s) [noise=true , address=145.79.227.51:4429 , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold Dec 30 02:53:36.828889934 (587h24m36.699972969s) [noise=true , address=45.91.116.13:4176 , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ejsmBd4tAVwTHjamPYbryQb5x2vDnk9pBuX3CBF9WYFhTrFVJ): protocol: expected piece hash; context canceled; EOF Dec 30 02:53:36.828895144 (587h24m36.699978179s) [noise=true , address=45.91.116.13:4176 , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold Dec 30 02:53:36.828994173 (587h24m36.700077208s) [noise=true , address=145.79.227.49:4340 , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF Dec 30 02:53:36.828999793 (587h24m36.700082828s) [noise=true , address=145.79.227.49:4340 , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold Dec 30 02:53:36.829030332 (587h24m36.700113367s) [seg_pos={0 0}, upload=514937]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 02:53:36.829095486 (587h24m36.700178621s) [upload=514937]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 02:53:36.829100105 (587h24m36.70018315s) [upload=514937]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 02:53:37.128695270 (587h24m36.999778315s) [upload=514937]: Done uploading segments... Dec 30 02:53:37.128717683 (587h24m36.999800728s) [upload=514937]: Upload finished. err: <nil> Duration: 45.986310819s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [XX..X......X........X...X..XX..X.X......XXX.X.X..X........X.] duration: 5.968855112s trace: https://jaeger.dev.storj.io/trace/624cdc63715e4a13 FAIL (2025-12-30 03:01:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: TKAPVEKQV7VN3RJLMOFVPIXFLIQYW7IVO7R57UBDZIPMSZ4WXAZA) context deadline exceeded; piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: MLNF25NMPYX6SLQE6LYZEUO7LWEZUN5CXG2M2BJPGQR34BIQKYHQ) context deadline exceeded; piecestore: (Node ID: 12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, Piece ID: CZYOWDPIA76LDZZNLMZ6QGCWG4WKO5ZZKUHPLNRDZNPJTHFHVMDQ) context deadline exceeded ======LOGS====== Dec 30 03:00:32.524759643 (587h31m32.395842678s) [upload=515040]: Starting upload Dec 30 03:00:32.524842481 (587h31m32.395925526s) [upload=515040]: Done starting upload Dec 30 03:00:32.524949444 (587h31m32.396032489s) [upload=515040]: Uploading segments... Dec 30 03:00:32.525074613 (587h31m32.396157658s) [upload=515040]: Got next segment. Inline: false Dec 30 03:00:32.525108918 (587h31m32.396191963s) [upload=515040]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 03:00:32.525112936 (587h31m32.396195981s) [upload=515040]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 03:00:32.779208656 (587h31m32.650291691s) [seg_pos={0 0}, upload=515040]: Begin upload segment... Dec 30 03:00:32.779264722 (587h31m32.650347767s) [seg_pos={0 0}, upload=515040]: Done begin upload segment. Dec 30 03:00:32.779324766 (587h31m32.650407802s) [noise=true , address=45.91.116.12:4228 , piece_id=PDFE253J6E7ENLOCSLTICZUZWU6HSJ3MXMG7SDPKEQID4NOV3GDQ, seg_pos={0 0}, upload=515040]: Uploading piece... Dec 30 03:00:32.779336699 (587h31m32.650419744s) [noise=true , address=145.79.227.9:4367 , piece_id=VGW4A2GFHWLB7AKCALQ7TUSGUKNRSIX2AFVFF4GVYBBT7X6LUBVQ, seg_pos={0 0}, upload=515040]: Uploading piece... Dec 30 03:00:32.779403737 (587h31m32.650486772s) [noise=true , address=45.91.116.10:4081 , piece_id=MLNF25NMPYX6SLQE6LYZEUO7LWEZUN5CXG2M2BJPGQR34BIQKYHQ, seg_pos={0 0}, upload=515040]: Uploading piece... Dec 30 03:00:32.779599981 (587h31m32.650683026s) [noise=true , address=45.91.116.5:4299 , piece_id=JGBFYTCNZWADXUYRDKMNHEIS4YEKOPMVNJDMEMZS4WFZQUQ2B2MA, seg_pos={0 0}, upload=515040]: Uploading piece... Dec 30 03:00:32.780044998 (587h31m32.651128043s) [noise=true , address=45.91.116.14:4192 , piece_id=KUHWBGJMOQUNBQLKBFPXZYOER47AAXH7WZZ23DRNYHJI4A7QQ3SQ, seg_pos={0 0}, upload=515040]: Uploading piece... Dec 30 03:00:32.780471721 (587h31m32.651554766s) [noise=true , address=145.79.227.36:4519 , piece_id=3JTCWUQCUFAO2WPUYXEWCJ7PWFORKE64EREJDAOV7LIJZ2HHJFIQ, seg_pos={0 0}, upload=515040]: Uploading piece... Dec 30 03:00:32.780817649 (587h31m32.651900684s) [noise=true , address=145.79.227.51:4409 , piece_id=LL7TCMAGQLEF3SKXSP4HB6MN42NUPWBXWVCPFXUJQKYWZARU27GQ, seg_pos={0 0}, upload=515040]: Uploading piece... Dec 30 03:00:32.781240204 (587h31m32.652323249s) [noise=true , address=145.79.227.26:4653 , piece_id=5ZUGLOU7Z6IO5M6RJE7HNVHSMU2VS3OTURKRO6SC2EDNGSTS7AIQ, seg_pos={0 0}, upload=515040]: Uploading piece... Dec 30 03:00:32.781601191 (587h31m32.652684236s) [noise=true , address=45.91.116.6:4974 , piece_id=TZPCW6JXEHP24O5HOGZT52DOYJLHYZIIFBVY7GQBY3XTUPY7RZEA, seg_pos={0 0}, upload=515040]: Uploading piece... Dec 30 03:00:32.782205652 (587h31m32.653288708s) [noise=true , address=77.90.31.17:4770 , piece_id=LTCWAFC5AUI4YTBYER2CXT33CS4NKTF7T3ZNGEO7JSWH3STM5ECA, seg_pos={0 0}, upload=515040]: Uploading piece... Dec 30 03:00:32.783536396 (587h31m32.654619441s) [noise=true , address=145.79.227.15:4875 , piece_id=AFQFUDKDEHNVETSPYSCQYKXWJ6ZBCP4LSXEJXGMBIFSUKAVY2UGQ, seg_pos={0 0}, upload=515040]: Uploading piece... Dec 30 03:00:32.783714786 (587h31m32.654797831s) [noise=true , address=77.90.31.14:4662 , piece_id=TKAPVEKQV7VN3RJLMOFVPIXFLIQYW7IVO7R57UBDZIPMSZ4WXAZA, seg_pos={0 0}, upload=515040]: Uploading piece... Dec 30 03:00:32.784939718 (587h31m32.656022763s) [noise=true , address=77.90.31.21:4877 , piece_id=CZYOWDPIA76LDZZNLMZ6QGCWG4WKO5ZZKUHPLNRDZNPJTHFHVMDQ, seg_pos={0 0}, upload=515040]: Uploading piece... Dec 30 03:00:32.786513915 (587h31m32.65759696s) [noise=true , address=145.79.227.46:4601 , piece_id=K6ZGWNDMF455AYO4CVUNFAH4RXIDFM46X2F6CGAIA62ULAWK4LAQ, seg_pos={0 0}, upload=515040]: Uploading piece... Dec 30 03:00:32.788250482 (587h31m32.659333517s) [noise=true , address=77.90.31.19:4841 , piece_id=C7YYO3GEXHLI4VSOKEMBNY5LOJKXNZ2NH2ZITL4LRDFTHRXSA3NA, seg_pos={0 0}, upload=515040]: Uploading piece... Dec 30 03:00:32.789584874 (587h31m32.660667909s) [noise=true , address=145.79.227.2:4114 , piece_id=BY7KSMJITGJ676OMTN7AEI2FULTBNPYGEVPPQ57R7O3YS6D46ZDA, seg_pos={0 0}, upload=515040]: Uploading piece... Dec 30 03:00:34.483331001 (587h31m34.354414036s) [upload=515040]: Next returned nil segment Dec 30 03:00:34.483345248 (587h31m34.354428293s) [upload=515040]: Waiting for error group managing segments... Dec 30 03:00:34.772196065 (587h31m34.64327911s) [noise=true , address=145.79.227.51:4409 , piece_id=LL7TCMAGQLEF3SKXSP4HB6MN42NUPWBXWVCPFXUJQKYWZARU27GQ, seg_pos={0 0}, upload=515040]: Done uploading piece. err: <nil> Dec 30 03:00:34.775524764 (587h31m34.646607809s) [noise=true , address=145.79.227.26:4653 , piece_id=5ZUGLOU7Z6IO5M6RJE7HNVHSMU2VS3OTURKRO6SC2EDNGSTS7AIQ, seg_pos={0 0}, upload=515040]: Done uploading piece. err: <nil> Dec 30 03:00:34.779275755 (587h31m34.65035879s) [noise=true , address=145.79.227.2:4114 , piece_id=BY7KSMJITGJ676OMTN7AEI2FULTBNPYGEVPPQ57R7O3YS6D46ZDA, seg_pos={0 0}, upload=515040]: Done uploading piece. err: <nil> Dec 30 03:00:34.785444191 (587h31m34.656527226s) [noise=true , address=145.79.227.36:4519 , piece_id=3JTCWUQCUFAO2WPUYXEWCJ7PWFORKE64EREJDAOV7LIJZ2HHJFIQ, seg_pos={0 0}, upload=515040]: Done uploading piece. err: <nil> Dec 30 03:00:34.785957889 (587h31m34.657040934s) [noise=true , address=145.79.227.9:4367 , piece_id=VGW4A2GFHWLB7AKCALQ7TUSGUKNRSIX2AFVFF4GVYBBT7X6LUBVQ, seg_pos={0 0}, upload=515040]: Done uploading piece. err: <nil> Dec 30 03:00:34.803653535 (587h31m34.67473657s) [noise=true , address=145.79.227.15:4875 , piece_id=AFQFUDKDEHNVETSPYSCQYKXWJ6ZBCP4LSXEJXGMBIFSUKAVY2UGQ, seg_pos={0 0}, upload=515040]: Done uploading piece. err: <nil> Dec 30 03:00:34.826913716 (587h31m34.697996761s) [noise=true , address=145.79.227.46:4601 , piece_id=K6ZGWNDMF455AYO4CVUNFAH4RXIDFM46X2F6CGAIA62ULAWK4LAQ, seg_pos={0 0}, upload=515040]: Done uploading piece. err: <nil> Dec 30 03:00:34.956582195 (587h31m34.82766523s) [noise=true , address=77.90.31.21:4877 , piece_id=CZYOWDPIA76LDZZNLMZ6QGCWG4WKO5ZZKUHPLNRDZNPJTHFHVMDQ, seg_pos={0 0}, upload=515040]: Done uploading piece. err: <nil> Dec 30 03:00:35.009621380 (587h31m34.880704425s) [noise=true , address=45.91.116.6:4974 , piece_id=TZPCW6JXEHP24O5HOGZT52DOYJLHYZIIFBVY7GQBY3XTUPY7RZEA, seg_pos={0 0}, upload=515040]: Done uploading piece. err: <nil> Dec 30 03:00:35.018580239 (587h31m34.889663274s) [noise=true , address=45.91.116.10:4081 , piece_id=MLNF25NMPYX6SLQE6LYZEUO7LWEZUN5CXG2M2BJPGQR34BIQKYHQ, seg_pos={0 0}, upload=515040]: Done uploading piece. err: <nil> Dec 30 03:00:35.080684325 (587h31m34.95176737s) [noise=true , address=77.90.31.14:4662 , piece_id=TKAPVEKQV7VN3RJLMOFVPIXFLIQYW7IVO7R57UBDZIPMSZ4WXAZA, seg_pos={0 0}, upload=515040]: Done uploading piece. err: <nil> Dec 30 03:00:35.152848707 (587h31m35.023931742s) [noise=true , address=45.91.116.5:4299 , piece_id=JGBFYTCNZWADXUYRDKMNHEIS4YEKOPMVNJDMEMZS4WFZQUQ2B2MA, seg_pos={0 0}, upload=515040]: Done uploading piece. err: <nil> Dec 30 03:00:35.198388662 (587h31m35.069471697s) [noise=true , address=45.91.116.12:4228 , piece_id=PDFE253J6E7ENLOCSLTICZUZWU6HSJ3MXMG7SDPKEQID4NOV3GDQ, seg_pos={0 0}, upload=515040]: Done uploading piece. err: <nil> Dec 30 03:00:35.198475989 (587h31m35.069559034s) [seg_pos={0 0}, upload=515040]: Segment reached optimal threshold of 13 pieces. Dec 30 03:00:35.198711848 (587h31m35.069794903s) [noise=true , address=45.91.116.14:4192 , piece_id=KUHWBGJMOQUNBQLKBFPXZYOER47AAXH7WZZ23DRNYHJI4A7QQ3SQ, seg_pos={0 0}, upload=515040]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF Dec 30 03:00:35.198722588 (587h31m35.069805623s) [noise=true , address=45.91.116.14:4192 , piece_id=KUHWBGJMOQUNBQLKBFPXZYOER47AAXH7WZZ23DRNYHJI4A7QQ3SQ, seg_pos={0 0}, upload=515040]: Long tail cancellation due to optimal threshold Dec 30 03:00:35.199050963 (587h31m35.070133998s) [noise=true , address=77.90.31.17:4770 , piece_id=LTCWAFC5AUI4YTBYER2CXT33CS4NKTF7T3ZNGEO7JSWH3STM5ECA, seg_pos={0 0}, upload=515040]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Dec 30 03:00:35.199070932 (587h31m35.070153967s) [noise=true , address=77.90.31.17:4770 , piece_id=LTCWAFC5AUI4YTBYER2CXT33CS4NKTF7T3ZNGEO7JSWH3STM5ECA, seg_pos={0 0}, upload=515040]: Long tail cancellation due to optimal threshold Dec 30 03:00:35.199113041 (587h31m35.070196157s) [noise=true , address=77.90.31.19:4841 , piece_id=C7YYO3GEXHLI4VSOKEMBNY5LOJKXNZ2NH2ZITL4LRDFTHRXSA3NA, seg_pos={0 0}, upload=515040]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: read tcp 188.166.225.170:34682->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 30 03:00:35.199122389 (587h31m35.070205434s) [noise=true , address=77.90.31.19:4841 , piece_id=C7YYO3GEXHLI4VSOKEMBNY5LOJKXNZ2NH2ZITL4LRDFTHRXSA3NA, seg_pos={0 0}, upload=515040]: Long tail cancellation due to optimal threshold Dec 30 03:00:35.199145143 (587h31m35.070228178s) [seg_pos={0 0}, upload=515040]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 03:00:35.199232910 (587h31m35.070315945s) [upload=515040]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 03:00:35.199239082 (587h31m35.070322127s) [upload=515040]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 03:00:35.494663716 (587h31m35.365746751s) [upload=515040]: Done uploading segments... Dec 30 03:00:35.494678374 (587h31m35.365761419s) [upload=515040]: Upload finished. err: <nil> Duration: 5.968873587s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [XXX..X......X........X...X..XX..X.X......XXX.X.X..X........X] duration: 30.003476177s trace: https://jaeger.dev.storj.io/trace/624cdc63715e4a13 FAIL (2025-12-30 03:01:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: TKAPVEKQV7VN3RJLMOFVPIXFLIQYW7IVO7R57UBDZIPMSZ4WXAZA) context deadline exceeded; piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: MLNF25NMPYX6SLQE6LYZEUO7LWEZUN5CXG2M2BJPGQR34BIQKYHQ) context deadline exceeded; piecestore: (Node ID: 12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, Piece ID: CZYOWDPIA76LDZZNLMZ6QGCWG4WKO5ZZKUHPLNRDZNPJTHFHVMDQ) context deadline exceeded ======LOGS====== Dec 30 03:00:32.524759643 (587h31m32.395842678s) [upload=515040]: Starting upload Dec 30 03:00:32.524842481 (587h31m32.395925526s) [upload=515040]: Done starting upload Dec 30 03:00:32.524949444 (587h31m32.396032489s) [upload=515040]: Uploading segments... Dec 30 03:00:32.525074613 (587h31m32.396157658s) [upload=515040]: Got next segment. Inline: false Dec 30 03:00:32.525108918 (587h31m32.396191963s) [upload=515040]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 03:00:32.525112936 (587h31m32.396195981s) [upload=515040]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 03:00:32.779208656 (587h31m32.650291691s) [seg_pos={0 0}, upload=515040]: Begin upload segment... Dec 30 03:00:32.779264722 (587h31m32.650347767s) [seg_pos={0 0}, upload=515040]: Done begin upload segment. Dec 30 03:00:32.779324766 (587h31m32.650407802s) [noise=true , address=45.91.116.12:4228 , piece_id=PDFE253J6E7ENLOCSLTICZUZWU6HSJ3MXMG7SDPKEQID4NOV3GDQ, seg_pos={0 0}, upload=515040]: Uploading piece... Dec 30 03:00:32.779336699 (587h31m32.650419744s) [noise=true , address=145.79.227.9:4367 , piece_id=VGW4A2GFHWLB7AKCALQ7TUSGUKNRSIX2AFVFF4GVYBBT7X6LUBVQ, seg_pos={0 0}, upload=515040]: Uploading piece... Dec 30 03:00:32.779403737 (587h31m32.650486772s) [noise=true , address=45.91.116.10:4081 , piece_id=MLNF25NMPYX6SLQE6LYZEUO7LWEZUN5CXG2M2BJPGQR34BIQKYHQ, seg_pos={0 0}, upload=515040]: Uploading piece... Dec 30 03:00:32.779599981 (587h31m32.650683026s) [noise=true , address=45.91.116.5:4299 , piece_id=JGBFYTCNZWADXUYRDKMNHEIS4YEKOPMVNJDMEMZS4WFZQUQ2B2MA, seg_pos={0 0}, upload=515040]: Uploading piece... Dec 30 03:00:32.780044998 (587h31m32.651128043s) [noise=true , address=45.91.116.14:4192 , piece_id=KUHWBGJMOQUNBQLKBFPXZYOER47AAXH7WZZ23DRNYHJI4A7QQ3SQ, seg_pos={0 0}, upload=515040]: Uploading piece... Dec 30 03:00:32.780471721 (587h31m32.651554766s) [noise=true , address=145.79.227.36:4519 , piece_id=3JTCWUQCUFAO2WPUYXEWCJ7PWFORKE64EREJDAOV7LIJZ2HHJFIQ, seg_pos={0 0}, upload=515040]: Uploading piece... Dec 30 03:00:32.780817649 (587h31m32.651900684s) [noise=true , address=145.79.227.51:4409 , piece_id=LL7TCMAGQLEF3SKXSP4HB6MN42NUPWBXWVCPFXUJQKYWZARU27GQ, seg_pos={0 0}, upload=515040]: Uploading piece... Dec 30 03:00:32.781240204 (587h31m32.652323249s) [noise=true , address=145.79.227.26:4653 , piece_id=5ZUGLOU7Z6IO5M6RJE7HNVHSMU2VS3OTURKRO6SC2EDNGSTS7AIQ, seg_pos={0 0}, upload=515040]: Uploading piece... Dec 30 03:00:32.781601191 (587h31m32.652684236s) [noise=true , address=45.91.116.6:4974 , piece_id=TZPCW6JXEHP24O5HOGZT52DOYJLHYZIIFBVY7GQBY3XTUPY7RZEA, seg_pos={0 0}, upload=515040]: Uploading piece... Dec 30 03:00:32.782205652 (587h31m32.653288708s) [noise=true , address=77.90.31.17:4770 , piece_id=LTCWAFC5AUI4YTBYER2CXT33CS4NKTF7T3ZNGEO7JSWH3STM5ECA, seg_pos={0 0}, upload=515040]: Uploading piece... Dec 30 03:00:32.783536396 (587h31m32.654619441s) [noise=true , address=145.79.227.15:4875 , piece_id=AFQFUDKDEHNVETSPYSCQYKXWJ6ZBCP4LSXEJXGMBIFSUKAVY2UGQ, seg_pos={0 0}, upload=515040]: Uploading piece... Dec 30 03:00:32.783714786 (587h31m32.654797831s) [noise=true , address=77.90.31.14:4662 , piece_id=TKAPVEKQV7VN3RJLMOFVPIXFLIQYW7IVO7R57UBDZIPMSZ4WXAZA, seg_pos={0 0}, upload=515040]: Uploading piece... Dec 30 03:00:32.784939718 (587h31m32.656022763s) [noise=true , address=77.90.31.21:4877 , piece_id=CZYOWDPIA76LDZZNLMZ6QGCWG4WKO5ZZKUHPLNRDZNPJTHFHVMDQ, seg_pos={0 0}, upload=515040]: Uploading piece... Dec 30 03:00:32.786513915 (587h31m32.65759696s) [noise=true , address=145.79.227.46:4601 , piece_id=K6ZGWNDMF455AYO4CVUNFAH4RXIDFM46X2F6CGAIA62ULAWK4LAQ, seg_pos={0 0}, upload=515040]: Uploading piece... Dec 30 03:00:32.788250482 (587h31m32.659333517s) [noise=true , address=77.90.31.19:4841 , piece_id=C7YYO3GEXHLI4VSOKEMBNY5LOJKXNZ2NH2ZITL4LRDFTHRXSA3NA, seg_pos={0 0}, upload=515040]: Uploading piece... Dec 30 03:00:32.789584874 (587h31m32.660667909s) [noise=true , address=145.79.227.2:4114 , piece_id=BY7KSMJITGJ676OMTN7AEI2FULTBNPYGEVPPQ57R7O3YS6D46ZDA, seg_pos={0 0}, upload=515040]: Uploading piece... Dec 30 03:00:34.483331001 (587h31m34.354414036s) [upload=515040]: Next returned nil segment Dec 30 03:00:34.483345248 (587h31m34.354428293s) [upload=515040]: Waiting for error group managing segments... Dec 30 03:00:34.772196065 (587h31m34.64327911s) [noise=true , address=145.79.227.51:4409 , piece_id=LL7TCMAGQLEF3SKXSP4HB6MN42NUPWBXWVCPFXUJQKYWZARU27GQ, seg_pos={0 0}, upload=515040]: Done uploading piece. err: <nil> Dec 30 03:00:34.775524764 (587h31m34.646607809s) [noise=true , address=145.79.227.26:4653 , piece_id=5ZUGLOU7Z6IO5M6RJE7HNVHSMU2VS3OTURKRO6SC2EDNGSTS7AIQ, seg_pos={0 0}, upload=515040]: Done uploading piece. err: <nil> Dec 30 03:00:34.779275755 (587h31m34.65035879s) [noise=true , address=145.79.227.2:4114 , piece_id=BY7KSMJITGJ676OMTN7AEI2FULTBNPYGEVPPQ57R7O3YS6D46ZDA, seg_pos={0 0}, upload=515040]: Done uploading piece. err: <nil> Dec 30 03:00:34.785444191 (587h31m34.656527226s) [noise=true , address=145.79.227.36:4519 , piece_id=3JTCWUQCUFAO2WPUYXEWCJ7PWFORKE64EREJDAOV7LIJZ2HHJFIQ, seg_pos={0 0}, upload=515040]: Done uploading piece. err: <nil> Dec 30 03:00:34.785957889 (587h31m34.657040934s) [noise=true , address=145.79.227.9:4367 , piece_id=VGW4A2GFHWLB7AKCALQ7TUSGUKNRSIX2AFVFF4GVYBBT7X6LUBVQ, seg_pos={0 0}, upload=515040]: Done uploading piece. err: <nil> Dec 30 03:00:34.803653535 (587h31m34.67473657s) [noise=true , address=145.79.227.15:4875 , piece_id=AFQFUDKDEHNVETSPYSCQYKXWJ6ZBCP4LSXEJXGMBIFSUKAVY2UGQ, seg_pos={0 0}, upload=515040]: Done uploading piece. err: <nil> Dec 30 03:00:34.826913716 (587h31m34.697996761s) [noise=true , address=145.79.227.46:4601 , piece_id=K6ZGWNDMF455AYO4CVUNFAH4RXIDFM46X2F6CGAIA62ULAWK4LAQ, seg_pos={0 0}, upload=515040]: Done uploading piece. err: <nil> Dec 30 03:00:34.956582195 (587h31m34.82766523s) [noise=true , address=77.90.31.21:4877 , piece_id=CZYOWDPIA76LDZZNLMZ6QGCWG4WKO5ZZKUHPLNRDZNPJTHFHVMDQ, seg_pos={0 0}, upload=515040]: Done uploading piece. err: <nil> Dec 30 03:00:35.009621380 (587h31m34.880704425s) [noise=true , address=45.91.116.6:4974 , piece_id=TZPCW6JXEHP24O5HOGZT52DOYJLHYZIIFBVY7GQBY3XTUPY7RZEA, seg_pos={0 0}, upload=515040]: Done uploading piece. err: <nil> Dec 30 03:00:35.018580239 (587h31m34.889663274s) [noise=true , address=45.91.116.10:4081 , piece_id=MLNF25NMPYX6SLQE6LYZEUO7LWEZUN5CXG2M2BJPGQR34BIQKYHQ, seg_pos={0 0}, upload=515040]: Done uploading piece. err: <nil> Dec 30 03:00:35.080684325 (587h31m34.95176737s) [noise=true , address=77.90.31.14:4662 , piece_id=TKAPVEKQV7VN3RJLMOFVPIXFLIQYW7IVO7R57UBDZIPMSZ4WXAZA, seg_pos={0 0}, upload=515040]: Done uploading piece. err: <nil> Dec 30 03:00:35.152848707 (587h31m35.023931742s) [noise=true , address=45.91.116.5:4299 , piece_id=JGBFYTCNZWADXUYRDKMNHEIS4YEKOPMVNJDMEMZS4WFZQUQ2B2MA, seg_pos={0 0}, upload=515040]: Done uploading piece. err: <nil> Dec 30 03:00:35.198388662 (587h31m35.069471697s) [noise=true , address=45.91.116.12:4228 , piece_id=PDFE253J6E7ENLOCSLTICZUZWU6HSJ3MXMG7SDPKEQID4NOV3GDQ, seg_pos={0 0}, upload=515040]: Done uploading piece. err: <nil> Dec 30 03:00:35.198475989 (587h31m35.069559034s) [seg_pos={0 0}, upload=515040]: Segment reached optimal threshold of 13 pieces. Dec 30 03:00:35.198711848 (587h31m35.069794903s) [noise=true , address=45.91.116.14:4192 , piece_id=KUHWBGJMOQUNBQLKBFPXZYOER47AAXH7WZZ23DRNYHJI4A7QQ3SQ, seg_pos={0 0}, upload=515040]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF Dec 30 03:00:35.198722588 (587h31m35.069805623s) [noise=true , address=45.91.116.14:4192 , piece_id=KUHWBGJMOQUNBQLKBFPXZYOER47AAXH7WZZ23DRNYHJI4A7QQ3SQ, seg_pos={0 0}, upload=515040]: Long tail cancellation due to optimal threshold Dec 30 03:00:35.199050963 (587h31m35.070133998s) [noise=true , address=77.90.31.17:4770 , piece_id=LTCWAFC5AUI4YTBYER2CXT33CS4NKTF7T3ZNGEO7JSWH3STM5ECA, seg_pos={0 0}, upload=515040]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Dec 30 03:00:35.199070932 (587h31m35.070153967s) [noise=true , address=77.90.31.17:4770 , piece_id=LTCWAFC5AUI4YTBYER2CXT33CS4NKTF7T3ZNGEO7JSWH3STM5ECA, seg_pos={0 0}, upload=515040]: Long tail cancellation due to optimal threshold Dec 30 03:00:35.199113041 (587h31m35.070196157s) [noise=true , address=77.90.31.19:4841 , piece_id=C7YYO3GEXHLI4VSOKEMBNY5LOJKXNZ2NH2ZITL4LRDFTHRXSA3NA, seg_pos={0 0}, upload=515040]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: read tcp 188.166.225.170:34682->77.90.31.19:4841: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 30 03:00:35.199122389 (587h31m35.070205434s) [noise=true , address=77.90.31.19:4841 , piece_id=C7YYO3GEXHLI4VSOKEMBNY5LOJKXNZ2NH2ZITL4LRDFTHRXSA3NA, seg_pos={0 0}, upload=515040]: Long tail cancellation due to optimal threshold Dec 30 03:00:35.199145143 (587h31m35.070228178s) [seg_pos={0 0}, upload=515040]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 03:00:35.199232910 (587h31m35.070315945s) [upload=515040]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 03:00:35.199239082 (587h31m35.070322127s) [upload=515040]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 03:00:35.494663716 (587h31m35.365746751s) [upload=515040]: Done uploading segments... Dec 30 03:00:35.494678374 (587h31m35.365761419s) [upload=515040]: Upload finished. err: <nil> Duration: 30.003516235s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.............X.............X................X......X........] duration: 31.973816942s trace: https://jaeger.dev.storj.io/trace/6c3caa8adcfacbb9 FAIL (2025-12-30 03:00:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== true , address=145.79.227.35:4496 , piece_id=5X54ZGU4OIVI3DT622QE7IL66SVFLZYUICPODGMDKN7U4D4OJNSA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.040303259 (587h24m33.911386294s) [noise=true , address=77.90.31.12:4601 , piece_id=QQFK4PCL27YYF3UATGLV5KDENRNX6NHE7RXAZ2K66CC3WLHEBMWQ, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.040670117 (587h24m33.911753152s) [noise=true , address=77.90.31.21:4881 , piece_id=EMV4LHFK2VRHSW6PD3BBYGOXDAAYJKX6GEOXGGLOT3IXWNDQMU4A, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041006247 (587h24m33.912089292s) [noise=true , address=145.79.227.52:4472 , piece_id=DT2DAIC4ZI3WL7OGY55ZTTXJEI4IDRXPSJIQA3LTRC7K2F6EVPCA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041183314 (587h24m33.912266349s) [noise=true , address=145.79.227.26:4674 , piece_id=3BAN2ZXOFTSGZITYYHMZUS4AD35X6ZBOYA2HKOJCJI7VBWPJ3LSA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041313653 (587h24m33.912396688s) [noise=true , address=145.79.227.51:4429 , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041535194 (587h24m33.912618229s) [noise=true , address=145.79.227.33:4885 , piece_id=LA755L4RA4VJBHNCQBXOCFROZA3W6W6PUPKGJ5J772WAN44YMY2A, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041639693 (587h24m33.912722728s) [noise=true , address=145.79.227.43:4187 , piece_id=CAAFD3XIWWHZN7L7SYDMMSC5G5RKRLFYYHQZEEN4GAC2D6Y2EARA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041877216 (587h24m33.912960261s) [noise=true , address=145.79.227.46:4623 , piece_id=V4TEM7RRTTGBW5ISZOEAOPFIOZZ5EABO7BPWNRH5U5UPM2X7554Q, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.041986634 (587h24m33.913069669s) [noise=true , address=45.91.116.13:4176 , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.042345657 (587h24m33.913428692s) [noise=true , address=45.91.116.10:4104 , piece_id=MRIWH6WGACE4UDNX55MIY6ZDWBAHS6E5KQS4QCOQN2BU7JI7ODCA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.042362019 (587h24m33.913445054s) [noise=true , address=77.90.31.11:4573 , piece_id=TORCC4W7DS5TJ7H7NWONEPNJOFULSZ6F5OHOP4BEVMGEH2MJA4DQ, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.042684192 (587h24m33.913767237s) [noise=true , address=145.79.227.49:4340 , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.042707327 (587h24m33.913790362s) [noise=true , address=145.79.227.32:4837 , piece_id=SQTHTF5KWZOKFYX6CGL6KCDRDAY7OJY6NA5OM3N4YCICDPM6DKXA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.043091528 (587h24m33.914174563s) [noise=true , address=45.91.116.5:4309 , piece_id=Y5HCMU6Z2FPBV3BNTUOC4F6JJWKNBKPOXWFVAFQTXUKQNHTLA4TA, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:34.043544771 (587h24m33.914627816s) [noise=true , address=145.79.227.40:4053 , piece_id=3IZP7JBGSWBRBQHMJWXIDD73B5R3COWU5H3IYFGVBCTTLW7OUJ5A, seg_pos={0 0}, upload=514937]: Uploading piece... Dec 30 02:53:35.347637489 (587h24m35.218720524s) [upload=514937]: Next returned nil segment Dec 30 02:53:35.347733591 (587h24m35.218816626s) [upload=514937]: Waiting for error group managing segments... Dec 30 02:53:35.674169656 (587h24m35.545252721s) [noise=true , address=64.49.13.27:4959 , piece_id=B4UZ67DIWNRHV77VTQONS6ZBUI2BPDDBM3ZS6FVCTABQZ6OHRJJA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:35.891295347 (587h24m35.762378382s) [noise=true , address=145.79.227.32:4837 , piece_id=SQTHTF5KWZOKFYX6CGL6KCDRDAY7OJY6NA5OM3N4YCICDPM6DKXA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:35.912619632 (587h24m35.783702677s) [noise=true , address=145.79.227.26:4674 , piece_id=3BAN2ZXOFTSGZITYYHMZUS4AD35X6ZBOYA2HKOJCJI7VBWPJ3LSA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:35.913054741 (587h24m35.784137786s) [noise=true , address=145.79.227.46:4623 , piece_id=V4TEM7RRTTGBW5ISZOEAOPFIOZZ5EABO7BPWNRH5U5UPM2X7554Q, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:35.913485270 (587h24m35.784568305s) [noise=true , address=145.79.227.35:4496 , piece_id=5X54ZGU4OIVI3DT622QE7IL66SVFLZYUICPODGMDKN7U4D4OJNSA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:35.923790864 (587h24m35.794873909s) [noise=true , address=145.79.227.52:4472 , piece_id=DT2DAIC4ZI3WL7OGY55ZTTXJEI4IDRXPSJIQA3LTRC7K2F6EVPCA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.109605534 (587h24m35.980688569s) [noise=true , address=77.90.31.21:4881 , piece_id=EMV4LHFK2VRHSW6PD3BBYGOXDAAYJKX6GEOXGGLOT3IXWNDQMU4A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.122502652 (587h24m35.993585697s) [noise=true , address=77.90.31.11:4573 , piece_id=TORCC4W7DS5TJ7H7NWONEPNJOFULSZ6F5OHOP4BEVMGEH2MJA4DQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.151131466 (587h24m36.022214511s) [noise=true , address=45.91.116.10:4104 , piece_id=MRIWH6WGACE4UDNX55MIY6ZDWBAHS6E5KQS4QCOQN2BU7JI7ODCA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.159562149 (587h24m36.030645184s) [noise=true , address=77.90.31.12:4601 , piece_id=QQFK4PCL27YYF3UATGLV5KDENRNX6NHE7RXAZ2K66CC3WLHEBMWQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.238339784 (587h24m36.10942282s) [noise=true , address=45.91.116.5:4309 , piece_id=Y5HCMU6Z2FPBV3BNTUOC4F6JJWKNBKPOXWFVAFQTXUKQNHTLA4TA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.813503456 (587h24m36.684586501s) [noise=true , address=145.79.227.33:4885 , piece_id=LA755L4RA4VJBHNCQBXOCFROZA3W6W6PUPKGJ5J772WAN44YMY2A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.826964278 (587h24m36.698047313s) [noise=true , address=145.79.227.43:4187 , piece_id=CAAFD3XIWWHZN7L7SYDMMSC5G5RKRLFYYHQZEEN4GAC2D6Y2EARA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.828562510 (587h24m36.699645545s) [noise=true , address=145.79.227.40:4053 , piece_id=3IZP7JBGSWBRBQHMJWXIDD73B5R3COWU5H3IYFGVBCTTLW7OUJ5A, seg_pos={0 0}, upload=514937]: Done uploading piece. err: <nil> Dec 30 02:53:36.828575566 (587h24m36.699658601s) [seg_pos={0 0}, upload=514937]: Segment reached optimal threshold of 14 pieces. Dec 30 02:53:36.828752202 (587h24m36.699835237s) [noise=true , address=145.79.227.30:4780 , piece_id=DZQ5LIBMWXPJ4XXWORRDLQIAFXJDNDQEEBRMSBS44UX3D2HPTSWQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125WCbZ6a2NaewiLAHhHuPo91sQEfC9CZzEBV6rwF1JY7z13AEY): protocol: expected piece hash; context canceled; EOF Dec 30 02:53:36.828758714 (587h24m36.699841749s) [noise=true , address=145.79.227.30:4780 , piece_id=DZQ5LIBMWXPJ4XXWORRDLQIAFXJDNDQEEBRMSBS44UX3D2HPTSWQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold Dec 30 02:53:36.828762672 (587h24m36.699845707s) [noise=true , address=145.79.227.51:4429 , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qytmzwy7sTmWXskEbLWsWMniPz55zgho2akE7oh6ahAiT5SXf): protocol: write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection; write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56932->145.79.227.51:4429: use of closed network connection Dec 30 02:53:36.828774034 (587h24m36.699857069s) [noise=true , address=145.79.227.51:4429 , piece_id=G3DHZWANX5KTQRUV6KT5VDGMRGYT67UMN6ZLKHGVSJ6VTSLIJVOQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold Dec 30 02:53:36.828889934 (587h24m36.699972969s) [noise=true , address=45.91.116.13:4176 , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ejsmBd4tAVwTHjamPYbryQb5x2vDnk9pBuX3CBF9WYFhTrFVJ): protocol: expected piece hash; context canceled; EOF Dec 30 02:53:36.828895144 (587h24m36.699978179s) [noise=true , address=45.91.116.13:4176 , piece_id=PKOAFJRRTLH2PLTIJ4ZOY73CLLWSVKUCKWOT62SUGQDUMLXWRNUQ, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold Dec 30 02:53:36.828994173 (587h24m36.700077208s) [noise=true , address=145.79.227.49:4340 , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF Dec 30 02:53:36.828999793 (587h24m36.700082828s) [noise=true , address=145.79.227.49:4340 , piece_id=GN66Y532O6KFQWJ2CPTXAA4PQ6LKCAX7FVB7XMAPQ6SGQS6D7VAA, seg_pos={0 0}, upload=514937]: Long tail cancellation due to optimal threshold Dec 30 02:53:36.829030332 (587h24m36.700113367s) [seg_pos={0 0}, upload=514937]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 02:53:36.829095486 (587h24m36.700178621s) [upload=514937]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 02:53:36.829100105 (587h24m36.70018315s) [upload=514937]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 02:53:37.128695270 (587h24m36.999778315s) [upload=514937]: Done uploading segments... Dec 30 02:53:37.128717683 (587h24m36.999800728s) [upload=514937]: Upload finished. err: <nil> Duration: 31.973839926s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [....X...................................X...X.X.XX..........] duration: 3.464904342s trace: https://jaeger.dev.storj.io/trace/14b283cf2d9b0313 FAIL (2025-12-30 03:00:28 UTC) orbiter warning: context deadline exceeded Duration: 3.464937436s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)