Orbiter History

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 4.932252653s trace: https://jaeger.dev.storj.io/trace/49fc2034659f6881 FAIL (2025-12-08 20:56:45 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.932280555s core uplink eu1
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 466.369196ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/51c3a25722f21601 FAIL (2025-12-08 20:56:43 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 466.396187ms core satellite eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 6.258540295s trace: https://jaeger.dev.storj.io/trace/3996915dd8f30735 FAIL (2025-12-08 20:56:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== PGM26LWTYTWGZGQDPBL7UDX263DFBXY7WZRXGMYEFPWXQQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.569231809 (76h54m34.440315054s) [noise=true , address=89.149.201.147:4022 , piece_id=2VR3E4SXAAOR5RAVHFVUQQ7MBZQEEQYFZ2ZW5Z7YSCVOETZNFQWQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.569369882 (76h54m34.440452917s) [noise=true , address=185.167.204.206:4037 , piece_id=XYY5NAATQJPHOK6JAMHXR3JH3QTLQF7OTA32BMST5HYQDYJ76UMA, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.569672879 (76h54m34.440755924s) [noise=true , address=185.167.204.206:4056 , piece_id=IPDW6RCHJFLMB5LOZM3FG645CCW6TT7KJW4PBQXBUSAIE2ZZM74A, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.569779642 (76h54m34.440862677s) [noise=true , address=185.167.204.205:4024 , piece_id=2UHV7KDAKJJ4DDUU5MPBNLRP24OK5RQU6GGAVYUV5MEFLDDPGUZQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.570073130 (76h54m34.441156176s) [noise=true , address=81.171.14.70:4035 , piece_id=ZKLJ5VUPALKZBSPFCJQBZZUTMJ3PX2H3XMUZ5JVSW5BJBACRUYHA, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.570428618 (76h54m34.441511653s) [noise=true , address=81.171.14.70:4043 , piece_id=FVUGWZXJLQAOLUSTRKJAULSHDWKX5GGWA2BB52YP7WQEKU4NKS5Q, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.570785237 (76h54m34.441868282s) [noise=true , address=81.171.14.70:4044 , piece_id=QPYQGR62JDWWV3GM3CVDB6BLNSGNUTQUEHHQF74TYXFN25PMNPVQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.571235034 (76h54m34.442318069s) [noise=true , address=185.167.204.205:4035 , piece_id=LXUP2T5SN5SSV7IWZLGTSLCRWLFRZBODXX7W4FFVHZFO74I2CQHQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.571655133 (76h54m34.442738169s) [noise=true , address=81.171.14.70:4031 , piece_id=53QOUYYU4KS53TELJUM3FPEKOCB464OR5GKYGYB54RVRPD52RFWA, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.572102666 (76h54m34.443185701s) [noise=true , address=185.167.204.205:4034 , piece_id=4VH6HEDD7CTPMEOHYGKFXSYXHKOLPYB2N6BCCJT2DXVAMADI3KJQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:35.395419290 (76h54m35.266502335s) [upload=67229]: Next returned nil segment Dec 8 20:23:35.395432614 (76h54m35.26651565s) [upload=67229]: Waiting for error group managing segments... Dec 8 20:23:35.780649683 (76h54m35.651732718s) [noise=true , address=185.167.204.206:4037 , piece_id=XYY5NAATQJPHOK6JAMHXR3JH3QTLQF7OTA32BMST5HYQDYJ76UMA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.781236982 (76h54m35.652320027s) [noise=true , address=185.167.204.206:4056 , piece_id=IPDW6RCHJFLMB5LOZM3FG645CCW6TT7KJW4PBQXBUSAIE2ZZM74A, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.784267011 (76h54m35.655350046s) [noise=true , address=185.167.204.205:4030 , piece_id=O3QKGK2G5I5YYPRP5K57RTG4H35H2UDFROFU2JMSHKTFNZJDVUIA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.785164540 (76h54m35.656247585s) [noise=true , address=185.167.204.205:4012 , piece_id=QCJD2IHH4DEKTFN5ZPMZXPECBQ2IX2QPSFUZCSFR5HKCDOVYGGQA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.785585301 (76h54m35.656668336s) [noise=true , address=185.167.204.205:4002 , piece_id=PDSJ5KDCFQBURILRTWU2WSEG3UH7GX4GK57W4J75QADBPKQXRSYA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.787925598 (76h54m35.659008643s) [noise=true , address=185.167.204.206:4047 , piece_id=THE6CUP46D2OSXS34UR3TTBUP6FIVMGDGT7DVJDBB57BHMWBCZ7A, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.787994289 (76h54m35.659077324s) [noise=true , address=185.167.204.205:4034 , piece_id=4VH6HEDD7CTPMEOHYGKFXSYXHKOLPYB2N6BCCJT2DXVAMADI3KJQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.788899402 (76h54m35.659982437s) [noise=true , address=185.167.204.206:4050 , piece_id=7B4GLDQZUPQPLGPZPPN5CO3VOX3LLBMLIYA6IMALYITCHE4CZBOA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.791033055 (76h54m35.66211609s) [noise=true , address=185.167.204.205:4031 , piece_id=5XPLPLO4SPMX7ZBYHLUZTH7MBGCRESEEMP2OEI7LTTT5BN67XHMQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.792163116 (76h54m35.663246161s) [noise=true , address=185.167.204.206:4055 , piece_id=JESF36BPW3YEPXVI2GP4OALECNLYSASJHQRZVLNXKV5BQRDGOWWQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.792693356 (76h54m35.663776392s) [noise=true , address=185.167.204.206:4063 , piece_id=BRUAUJTOCPWNUOB6D5G6AJCTMJBRZRJXXXBZBIUUZZXXQB5OEX2Q, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.793171606 (76h54m35.664254662s) [noise=true , address=185.167.204.206:4045 , piece_id=IXV6HTIOVQR4HCGJIWTBQFLO733BMMNNMZ24Y34H6V6KYUQYPQRA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.798151289 (76h54m35.669234344s) [noise=true , address=185.167.204.206:4053 , piece_id=VGK5AKCI23P5ZDS427NS6PNNRIJ3QUXLRZ4ECLYN4UDSL3KAQAKA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.798189562 (76h54m35.669272597s) [noise=true , address=185.167.204.206:4040 , piece_id=4KFICIELLLPCLWQ7RIB7UUSWUK7SFX2PJK7JWCRU5S4HQNHBEWRQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.798633607 (76h54m35.669716642s) [noise=true , address=185.167.204.205:4035 , piece_id=LXUP2T5SN5SSV7IWZLGTSLCRWLFRZBODXX7W4FFVHZFO74I2CQHQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.803059726 (76h54m35.674142761s) [noise=true , address=185.167.204.205:4024 , piece_id=2UHV7KDAKJJ4DDUU5MPBNLRP24OK5RQU6GGAVYUV5MEFLDDPGUZQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.344154907 (76h54m36.215237952s) [noise=true , address=81.171.14.70:4036 , piece_id=FB3SB3XQDNT2HSK5FDGELGYWODFLU655QW4D23GEXIWQEZYFNFRA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.359261713 (76h54m36.230344758s) [noise=true , address=81.171.14.70:4031 , piece_id=53QOUYYU4KS53TELJUM3FPEKOCB464OR5GKYGYB54RVRPD52RFWA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.393865115 (76h54m36.26494815s) [noise=true , address=81.171.14.70:4035 , piece_id=ZKLJ5VUPALKZBSPFCJQBZZUTMJ3PX2H3XMUZ5JVSW5BJBACRUYHA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.394302157 (76h54m36.265385202s) [noise=true , address=81.171.14.70:4044 , piece_id=QPYQGR62JDWWV3GM3CVDB6BLNSGNUTQUEHHQF74TYXFN25PMNPVQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.430200674 (76h54m36.301283709s) [noise=true , address=81.171.14.70:4046 , piece_id=ZHZUULNXPFDSTOXXH6T7XIEPN23OOWCNAAUWPCZYNNDTLPRWXVKQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.800799297 (76h54m36.671882332s) [noise=true , address=81.171.14.70:4040 , piece_id=W2SOTP5SPLQ6J3PM57NZOLRVZNMTNSGPWDZKZC2KV2PBPH4TUBLA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.862864259 (76h54m36.733947295s) [noise=true , address=81.171.14.70:4038 , piece_id=CKN2RF2YSTEOK6HUXGVBVECYAP2CLGUF6KYNR5JTHRUZPTGVI73Q, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.953769972 (76h54m36.824853007s) [noise=true , address=81.171.14.70:4043 , piece_id=FVUGWZXJLQAOLUSTRKJAULSHDWKX5GGWA2BB52YP7WQEKU4NKS5Q, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.996603376 (76h54m36.867686421s) [noise=true , address=89.149.201.147:4006 , piece_id=UGREZBZRVKXMGBOLQEV2NC4COTXOTTRUEAGSPNWRWP22MCPHQP7Q, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:37.037518916 (76h54m36.908601951s) [noise=true , address=89.149.201.147:4014 , piece_id=AEG6KGPGM26LWTYTWGZGQDPBL7UDX263DFBXY7WZRXGMYEFPWXQQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:37.037537541 (76h54m36.908620586s) [seg_pos={0 0}, upload=67229]: Segment reached optimal threshold of 26 pieces. Dec 8 20:23:37.037730880 (76h54m36.908813915s) [noise=true , address=89.149.201.147:4023 , piece_id=5EPGXWBK6GNTKLQLWEIWYBC2OJHFR3M7F4X3X4WODJWNXID3PCQQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BaeSxjXneBKoGKVrRM686sq386Bhn1YzzBFG9we8fHbcc9tbv): protocol: expected piece hash; context canceled; EOF Dec 8 20:23:37.037737452 (76h54m36.908820497s) [noise=true , address=89.149.201.147:4023 , piece_id=5EPGXWBK6GNTKLQLWEIWYBC2OJHFR3M7F4X3X4WODJWNXID3PCQQ, seg_pos={0 0}, upload=67229]: Long tail cancellation due to optimal threshold Dec 8 20:23:37.037893039 (76h54m36.908976074s) [noise=true , address=89.149.201.147:4022 , piece_id=2VR3E4SXAAOR5RAVHFVUQQ7MBZQEEQYFZ2ZW5Z7YSCVOETZNFQWQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122Vj8XxpS6RnhVweQV3cZWFhj8nrzy3hZ9P7JsuTzEaZcD6iFE): protocol: expected piece hash; context canceled; EOF Dec 8 20:23:37.037900964 (76h54m36.908984019s) [noise=true , address=89.149.201.147:4022 , piece_id=2VR3E4SXAAOR5RAVHFVUQQ7MBZQEEQYFZ2ZW5Z7YSCVOETZNFQWQ, seg_pos={0 0}, upload=67229]: Long tail cancellation due to optimal threshold Dec 8 20:23:37.037926843 (76h54m36.909009889s) [seg_pos={0 0}, upload=67229]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 8 20:23:37.038007697 (76h54m36.909090732s) [upload=67229]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 20:23:37.038013398 (76h54m36.909096433s) [upload=67229]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 20:26:49.742746371 (76h57m49.613829416s) [upload=67229]: Done uploading segments... Dec 8 20:26:49.742860908 (76h57m49.613943943s) [upload=67229]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.258573979s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........X................................X..................] duration: 5.64778958s trace: https://jaeger.dev.storj.io/trace/2eed98fdb4249ce9 FAIL (2025-12-08 20:56:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 20:43:00.763765824 (77h14m0.634848859s) [upload=67501]: Starting upload Dec 8 20:43:00.763879119 (77h14m0.634962164s) [upload=67501]: Done starting upload Dec 8 20:43:00.763936168 (77h14m0.635019203s) [upload=67501]: Uploading segments... Dec 8 20:43:00.764117523 (77h14m0.635200568s) [upload=67501]: Got next segment. Inline: false Dec 8 20:43:00.764157569 (77h14m0.635240604s) [upload=67501]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 20:43:00.764162089 (77h14m0.635245134s) [upload=67501]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 20:43:01.015312549 (77h14m0.886395594s) [seg_pos={0 0}, upload=67501]: Begin upload segment... Dec 8 20:43:01.015378243 (77h14m0.886461288s) [seg_pos={0 0}, upload=67501]: Done begin upload segment. Dec 8 20:43:01.015411717 (77h14m0.886494762s) [noise=true , address=145.79.227.40:4050 , piece_id=K2RW6IQERLFKEUD2ZTAZEX767U77PQWEXMXEH2SYDBQ7BT26WAHQ, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.018044751 (77h14m0.889127796s) [noise=true , address=145.79.227.33:4870 , piece_id=KY2FLO3KDFPWRXPUUFINFLIRLZDMHPUCSPKRCN7RYPVAXXUYMXWQ, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.019100842 (77h14m0.890183887s) [noise=true , address=145.79.227.31:4797 , piece_id=ECY3CM3GYNMM5RIEDHYI5NVWDCKEXRRKH4NCK3FV5ILW7W54LIIA, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.020021405 (77h14m0.89110444s) [noise=true , address=145.79.227.41:4087 , piece_id=QYRY35JJKPQEK4BAVFZ6DEBKJN7SBUHG66S44ZA7XRNOVKAVQ33Q, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.020812742 (77h14m0.891895787s) [noise=true , address=145.79.227.20:4437 , piece_id=CCMQ7GX5NBRAUKHEHERGP5SHETIGELARKU5AE5ETKZWQRJP7OOAQ, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.021487076 (77h14m0.892570111s) [noise=true , address=77.90.31.21:4879 , piece_id=WF6YF2HQIXM26JI5TQ3YW7OXHNYZH3E2VTHAQVWIUHZ3ME3CCG6Q, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.022677542 (77h14m0.893760588s) [noise=true , address=145.79.227.42:4123 , piece_id=NBMNEIGLWZHFGHY674WDGEDH4CGDCKYERXLB7UKXXI4MTZJ2HVEQ, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.022691720 (77h14m0.893774755s) [noise=true , address=77.90.31.18:4807 , piece_id=KDIWMU2XEJ7NTRCSG5J37ZUWIFURSJVK2BSDM7VJFKOB3ROANRGA, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.023821471 (77h14m0.894904506s) [noise=true , address=145.79.227.41:4085 , piece_id=LS54ZFJC7DEGOKQRUSZOY3I665N43TNXICDJTXQ3ZJ62OHPGETSQ, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.024740981 (77h14m0.895824026s) [noise=true , address=77.90.31.12:4589 , piece_id=WUVA4G7EEN4J4HF47KQ7FZKFINAN2CCYDXMRNHDZVBF6EAPVFS4Q, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.025197331 (77h14m0.896280366s) [noise=true , address=145.79.227.34:4906 , piece_id=VPDAFAMOLFSYUGFZIHRTADXWLYAE7IXILSPLT6QL2IPCOVUDPA3A, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.026609700 (77h14m0.897692745s) [noise=true , address=145.79.227.25:4618 , piece_id=AE7LMOAKNNIKVS4STMQJKZER7YUSFSUTSE7P5KXYTG4AQLFNQHLQ, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.027084214 (77h14m0.898167259s) [noise=true , address=145.79.227.44:4195 , piece_id=EYY7B3AXUMA63GQLGKQWPWGDZUBORTXR6QFYSEOB26ATZK2GJC6Q, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.028812575 (77h14m0.89989562s) [noise=true , address=145.79.227.38:4977 , piece_id=JW72CEO3WDGEB6BSSAR3M4Z5ANDBH63BLJZRSYAJDK7F7EXCEOZQ, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.030193064 (77h14m0.90127611s) [noise=true , address=145.79.227.24:4583 , piece_id=DRLVD3TA2Z6TJQ4WKA67KDTXK7Q32YJH2ES5QEQG7Z37DYBJBB6Q, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.036513970 (77h14m0.907597045s) [noise=true , address=77.90.31.13:4627 , piece_id=XMI4HVR2MQQ5KHN4GQOUCKFYJYJ2YZEGQGTAAXYKGS3OI6XPJLVA, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:02.093045104 (77h14m1.964128159s) [noise=true , address=77.90.31.21:4879 , piece_id=WF6YF2HQIXM26JI5TQ3YW7OXHNYZH3E2VTHAQVWIUHZ3ME3CCG6Q, seg_pos={0 0}, upload=67501]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: storage node overloaded, request limit: 2000 Dec 8 20:43:02.127847935 (77h14m1.99893098s) [noise=true , address=77.90.31.13:4627 , piece_id=XMI4HVR2MQQ5KHN4GQOUCKFYJYJ2YZEGQGTAAXYKGS3OI6XPJLVA, seg_pos={0 0}, upload=67501]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 8 20:43:02.156386568 (77h14m2.027469613s) [noise=true , address=77.90.31.12:4589 , piece_id=WUVA4G7EEN4J4HF47KQ7FZKFINAN2CCYDXMRNHDZVBF6EAPVFS4Q, seg_pos={0 0}, upload=67501]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 20:43:02.719927441 (77h14m2.591010486s) [upload=67501]: Next returned nil segment Dec 8 20:43:02.719942970 (77h14m2.591026005s) [upload=67501]: Waiting for error group managing segments... Dec 8 20:43:03.005012008 (77h14m2.876095063s) [noise=true , address=145.79.227.42:4123 , piece_id=NBMNEIGLWZHFGHY674WDGEDH4CGDCKYERXLB7UKXXI4MTZJ2HVEQ, seg_pos={0 0}, upload=67501]: Done uploading piece. err: <nil> Dec 8 20:43:03.012525866 (77h14m2.883608911s) [noise=true , address=145.79.227.41:4085 , piece_id=LS54ZFJC7DEGOKQRUSZOY3I665N43TNXICDJTXQ3ZJ62OHPGETSQ, seg_pos={0 0}, upload=67501]: Done uploading piece. err: <nil> Dec 8 20:43:03.015459803 (77h14m2.886542848s) [noise=true , address=145.79.227.41:4087 , piece_id=QYRY35JJKPQEK4BAVFZ6DEBKJN7SBUHG66S44ZA7XRNOVKAVQ33Q, seg_pos={0 0}, upload=67501]: Done uploading piece. err: <nil> Dec 8 20:43:03.020621582 (77h14m2.891704617s) [noise=true , address=145.79.227.20:4437 , piece_id=CCMQ7GX5NBRAUKHEHERGP5SHETIGELARKU5AE5ETKZWQRJP7OOAQ, seg_pos={0 0}, upload=67501]: Done uploading piece. err: <nil> Dec 8 20:43:03.020672098 (77h14m2.891755153s) [noise=true , address=145.79.227.31:4797 , piece_id=ECY3CM3GYNMM5RIEDHYI5NVWDCKEXRRKH4NCK3FV5ILW7W54LIIA, seg_pos={0 0}, upload=67501]: Done uploading piece. err: <nil> Dec 8 20:43:03.028139237 (77h14m2.899222282s) [noise=true , address=145.79.227.24:4583 , piece_id=DRLVD3TA2Z6TJQ4WKA67KDTXK7Q32YJH2ES5QEQG7Z37DYBJBB6Q, seg_pos={0 0}, upload=67501]: Done uploading piece. err: <nil> Dec 8 20:43:03.031290167 (77h14m2.902373202s) [noise=true , address=145.79.227.33:4870 , piece_id=KY2FLO3KDFPWRXPUUFINFLIRLZDMHPUCSPKRCN7RYPVAXXUYMXWQ, seg_pos={0 0}, upload=67501]: Done uploading piece. err: <nil> Dec 8 20:43:03.033756633 (77h14m2.904839668s) [noise=true , address=145.79.227.25:4618 , piece_id=AE7LMOAKNNIKVS4STMQJKZER7YUSFSUTSE7P5KXYTG4AQLFNQHLQ, seg_pos={0 0}, upload=67501]: Done uploading piece. err: <nil> Dec 8 20:43:03.039153600 (77h14m2.910236655s) [noise=true , address=145.79.227.40:4050 , piece_id=K2RW6IQERLFKEUD2ZTAZEX767U77PQWEXMXEH2SYDBQ7BT26WAHQ, seg_pos={0 0}, upload=67501]: Done uploading piece. err: <nil> Dec 8 20:43:03.040008869 (77h14m2.911091914s) [noise=true , address=145.79.227.38:4977 , piece_id=JW72CEO3WDGEB6BSSAR3M4Z5ANDBH63BLJZRSYAJDK7F7EXCEOZQ, seg_pos={0 0}, upload=67501]: Done uploading piece. err: <nil> Dec 8 20:43:03.178911548 (77h14m3.049994593s) [noise=true , address=77.90.31.18:4807 , piece_id=KDIWMU2XEJ7NTRCSG5J37ZUWIFURSJVK2BSDM7VJFKOB3ROANRGA, seg_pos={0 0}, upload=67501]: Done uploading piece. err: <nil> Dec 8 20:43:04.409564163 (77h14m4.280647208s) [noise=true , address=145.79.227.44:4195 , piece_id=EYY7B3AXUMA63GQLGKQWPWGDZUBORTXR6QFYSEOB26ATZK2GJC6Q, seg_pos={0 0}, upload=67501]: Done uploading piece. err: <nil> Dec 8 20:43:30.067753910 (77h14m29.938836955s) [noise=true , address=145.79.227.34:4906 , piece_id=VPDAFAMOLFSYUGFZIHRTADXWLYAE7IXILSPLT6QL2IPCOVUDPA3A, seg_pos={0 0}, upload=67501]: Done uploading piece. err: ecclient: upload failed (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw, address:145.79.227.34:4906): protocol: read tcp 188.166.225.170:36576->145.79.227.34:4906: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 20:43:30.067797914 (77h14m29.938880959s) [seg_pos={0 0}, upload=67501]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 20:43:30.323605733 (77h14m30.194688778s) [upload=67501]: Done uploading segments... Dec 8 20:43:30.323629097 (77h14m30.194712142s) [upload=67501]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.647815921s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [...X....X....X..............................................] duration: 5.580105788s trace: https://jaeger.dev.storj.io/trace/4600f1753cc382a2 FAIL (2025-12-08 20:56:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.580133229s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................................................X.....] duration: 3.394906305s trace: https://jaeger.dev.storj.io/trace/65b6db6cde985dc2 FAIL (2025-12-08 20:56:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.394934829s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 5.032151291s trace: https://jaeger.dev.storj.io/trace/49fc2034659f6881 FAIL (2025-12-08 20:56:20 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.032181397s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [X...............X...........................................] duration: 5.223741073s trace: https://jaeger.dev.storj.io/trace/3996915dd8f30735 FAIL (2025-12-08 20:56:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== PGM26LWTYTWGZGQDPBL7UDX263DFBXY7WZRXGMYEFPWXQQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.569231809 (76h54m34.440315054s) [noise=true , address=89.149.201.147:4022 , piece_id=2VR3E4SXAAOR5RAVHFVUQQ7MBZQEEQYFZ2ZW5Z7YSCVOETZNFQWQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.569369882 (76h54m34.440452917s) [noise=true , address=185.167.204.206:4037 , piece_id=XYY5NAATQJPHOK6JAMHXR3JH3QTLQF7OTA32BMST5HYQDYJ76UMA, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.569672879 (76h54m34.440755924s) [noise=true , address=185.167.204.206:4056 , piece_id=IPDW6RCHJFLMB5LOZM3FG645CCW6TT7KJW4PBQXBUSAIE2ZZM74A, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.569779642 (76h54m34.440862677s) [noise=true , address=185.167.204.205:4024 , piece_id=2UHV7KDAKJJ4DDUU5MPBNLRP24OK5RQU6GGAVYUV5MEFLDDPGUZQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.570073130 (76h54m34.441156176s) [noise=true , address=81.171.14.70:4035 , piece_id=ZKLJ5VUPALKZBSPFCJQBZZUTMJ3PX2H3XMUZ5JVSW5BJBACRUYHA, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.570428618 (76h54m34.441511653s) [noise=true , address=81.171.14.70:4043 , piece_id=FVUGWZXJLQAOLUSTRKJAULSHDWKX5GGWA2BB52YP7WQEKU4NKS5Q, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.570785237 (76h54m34.441868282s) [noise=true , address=81.171.14.70:4044 , piece_id=QPYQGR62JDWWV3GM3CVDB6BLNSGNUTQUEHHQF74TYXFN25PMNPVQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.571235034 (76h54m34.442318069s) [noise=true , address=185.167.204.205:4035 , piece_id=LXUP2T5SN5SSV7IWZLGTSLCRWLFRZBODXX7W4FFVHZFO74I2CQHQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.571655133 (76h54m34.442738169s) [noise=true , address=81.171.14.70:4031 , piece_id=53QOUYYU4KS53TELJUM3FPEKOCB464OR5GKYGYB54RVRPD52RFWA, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.572102666 (76h54m34.443185701s) [noise=true , address=185.167.204.205:4034 , piece_id=4VH6HEDD7CTPMEOHYGKFXSYXHKOLPYB2N6BCCJT2DXVAMADI3KJQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:35.395419290 (76h54m35.266502335s) [upload=67229]: Next returned nil segment Dec 8 20:23:35.395432614 (76h54m35.26651565s) [upload=67229]: Waiting for error group managing segments... Dec 8 20:23:35.780649683 (76h54m35.651732718s) [noise=true , address=185.167.204.206:4037 , piece_id=XYY5NAATQJPHOK6JAMHXR3JH3QTLQF7OTA32BMST5HYQDYJ76UMA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.781236982 (76h54m35.652320027s) [noise=true , address=185.167.204.206:4056 , piece_id=IPDW6RCHJFLMB5LOZM3FG645CCW6TT7KJW4PBQXBUSAIE2ZZM74A, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.784267011 (76h54m35.655350046s) [noise=true , address=185.167.204.205:4030 , piece_id=O3QKGK2G5I5YYPRP5K57RTG4H35H2UDFROFU2JMSHKTFNZJDVUIA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.785164540 (76h54m35.656247585s) [noise=true , address=185.167.204.205:4012 , piece_id=QCJD2IHH4DEKTFN5ZPMZXPECBQ2IX2QPSFUZCSFR5HKCDOVYGGQA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.785585301 (76h54m35.656668336s) [noise=true , address=185.167.204.205:4002 , piece_id=PDSJ5KDCFQBURILRTWU2WSEG3UH7GX4GK57W4J75QADBPKQXRSYA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.787925598 (76h54m35.659008643s) [noise=true , address=185.167.204.206:4047 , piece_id=THE6CUP46D2OSXS34UR3TTBUP6FIVMGDGT7DVJDBB57BHMWBCZ7A, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.787994289 (76h54m35.659077324s) [noise=true , address=185.167.204.205:4034 , piece_id=4VH6HEDD7CTPMEOHYGKFXSYXHKOLPYB2N6BCCJT2DXVAMADI3KJQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.788899402 (76h54m35.659982437s) [noise=true , address=185.167.204.206:4050 , piece_id=7B4GLDQZUPQPLGPZPPN5CO3VOX3LLBMLIYA6IMALYITCHE4CZBOA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.791033055 (76h54m35.66211609s) [noise=true , address=185.167.204.205:4031 , piece_id=5XPLPLO4SPMX7ZBYHLUZTH7MBGCRESEEMP2OEI7LTTT5BN67XHMQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.792163116 (76h54m35.663246161s) [noise=true , address=185.167.204.206:4055 , piece_id=JESF36BPW3YEPXVI2GP4OALECNLYSASJHQRZVLNXKV5BQRDGOWWQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.792693356 (76h54m35.663776392s) [noise=true , address=185.167.204.206:4063 , piece_id=BRUAUJTOCPWNUOB6D5G6AJCTMJBRZRJXXXBZBIUUZZXXQB5OEX2Q, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.793171606 (76h54m35.664254662s) [noise=true , address=185.167.204.206:4045 , piece_id=IXV6HTIOVQR4HCGJIWTBQFLO733BMMNNMZ24Y34H6V6KYUQYPQRA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.798151289 (76h54m35.669234344s) [noise=true , address=185.167.204.206:4053 , piece_id=VGK5AKCI23P5ZDS427NS6PNNRIJ3QUXLRZ4ECLYN4UDSL3KAQAKA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.798189562 (76h54m35.669272597s) [noise=true , address=185.167.204.206:4040 , piece_id=4KFICIELLLPCLWQ7RIB7UUSWUK7SFX2PJK7JWCRU5S4HQNHBEWRQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.798633607 (76h54m35.669716642s) [noise=true , address=185.167.204.205:4035 , piece_id=LXUP2T5SN5SSV7IWZLGTSLCRWLFRZBODXX7W4FFVHZFO74I2CQHQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.803059726 (76h54m35.674142761s) [noise=true , address=185.167.204.205:4024 , piece_id=2UHV7KDAKJJ4DDUU5MPBNLRP24OK5RQU6GGAVYUV5MEFLDDPGUZQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.344154907 (76h54m36.215237952s) [noise=true , address=81.171.14.70:4036 , piece_id=FB3SB3XQDNT2HSK5FDGELGYWODFLU655QW4D23GEXIWQEZYFNFRA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.359261713 (76h54m36.230344758s) [noise=true , address=81.171.14.70:4031 , piece_id=53QOUYYU4KS53TELJUM3FPEKOCB464OR5GKYGYB54RVRPD52RFWA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.393865115 (76h54m36.26494815s) [noise=true , address=81.171.14.70:4035 , piece_id=ZKLJ5VUPALKZBSPFCJQBZZUTMJ3PX2H3XMUZ5JVSW5BJBACRUYHA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.394302157 (76h54m36.265385202s) [noise=true , address=81.171.14.70:4044 , piece_id=QPYQGR62JDWWV3GM3CVDB6BLNSGNUTQUEHHQF74TYXFN25PMNPVQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.430200674 (76h54m36.301283709s) [noise=true , address=81.171.14.70:4046 , piece_id=ZHZUULNXPFDSTOXXH6T7XIEPN23OOWCNAAUWPCZYNNDTLPRWXVKQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.800799297 (76h54m36.671882332s) [noise=true , address=81.171.14.70:4040 , piece_id=W2SOTP5SPLQ6J3PM57NZOLRVZNMTNSGPWDZKZC2KV2PBPH4TUBLA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.862864259 (76h54m36.733947295s) [noise=true , address=81.171.14.70:4038 , piece_id=CKN2RF2YSTEOK6HUXGVBVECYAP2CLGUF6KYNR5JTHRUZPTGVI73Q, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.953769972 (76h54m36.824853007s) [noise=true , address=81.171.14.70:4043 , piece_id=FVUGWZXJLQAOLUSTRKJAULSHDWKX5GGWA2BB52YP7WQEKU4NKS5Q, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.996603376 (76h54m36.867686421s) [noise=true , address=89.149.201.147:4006 , piece_id=UGREZBZRVKXMGBOLQEV2NC4COTXOTTRUEAGSPNWRWP22MCPHQP7Q, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:37.037518916 (76h54m36.908601951s) [noise=true , address=89.149.201.147:4014 , piece_id=AEG6KGPGM26LWTYTWGZGQDPBL7UDX263DFBXY7WZRXGMYEFPWXQQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:37.037537541 (76h54m36.908620586s) [seg_pos={0 0}, upload=67229]: Segment reached optimal threshold of 26 pieces. Dec 8 20:23:37.037730880 (76h54m36.908813915s) [noise=true , address=89.149.201.147:4023 , piece_id=5EPGXWBK6GNTKLQLWEIWYBC2OJHFR3M7F4X3X4WODJWNXID3PCQQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BaeSxjXneBKoGKVrRM686sq386Bhn1YzzBFG9we8fHbcc9tbv): protocol: expected piece hash; context canceled; EOF Dec 8 20:23:37.037737452 (76h54m36.908820497s) [noise=true , address=89.149.201.147:4023 , piece_id=5EPGXWBK6GNTKLQLWEIWYBC2OJHFR3M7F4X3X4WODJWNXID3PCQQ, seg_pos={0 0}, upload=67229]: Long tail cancellation due to optimal threshold Dec 8 20:23:37.037893039 (76h54m36.908976074s) [noise=true , address=89.149.201.147:4022 , piece_id=2VR3E4SXAAOR5RAVHFVUQQ7MBZQEEQYFZ2ZW5Z7YSCVOETZNFQWQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122Vj8XxpS6RnhVweQV3cZWFhj8nrzy3hZ9P7JsuTzEaZcD6iFE): protocol: expected piece hash; context canceled; EOF Dec 8 20:23:37.037900964 (76h54m36.908984019s) [noise=true , address=89.149.201.147:4022 , piece_id=2VR3E4SXAAOR5RAVHFVUQQ7MBZQEEQYFZ2ZW5Z7YSCVOETZNFQWQ, seg_pos={0 0}, upload=67229]: Long tail cancellation due to optimal threshold Dec 8 20:23:37.037926843 (76h54m36.909009889s) [seg_pos={0 0}, upload=67229]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 8 20:23:37.038007697 (76h54m36.909090732s) [upload=67229]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 20:23:37.038013398 (76h54m36.909096433s) [upload=67229]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 20:26:49.742746371 (76h57m49.613829416s) [upload=67229]: Done uploading segments... Dec 8 20:26:49.742860908 (76h57m49.613943943s) [upload=67229]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.223765149s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 5.063366637s trace: https://jaeger.dev.storj.io/trace/49fc2034659f6881 FAIL (2025-12-08 20:56:05 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.063397746s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [.X...............X..........................................] duration: 5.234215539s trace: https://jaeger.dev.storj.io/trace/3996915dd8f30735 FAIL (2025-12-08 20:55:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== PGM26LWTYTWGZGQDPBL7UDX263DFBXY7WZRXGMYEFPWXQQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.569231809 (76h54m34.440315054s) [noise=true , address=89.149.201.147:4022 , piece_id=2VR3E4SXAAOR5RAVHFVUQQ7MBZQEEQYFZ2ZW5Z7YSCVOETZNFQWQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.569369882 (76h54m34.440452917s) [noise=true , address=185.167.204.206:4037 , piece_id=XYY5NAATQJPHOK6JAMHXR3JH3QTLQF7OTA32BMST5HYQDYJ76UMA, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.569672879 (76h54m34.440755924s) [noise=true , address=185.167.204.206:4056 , piece_id=IPDW6RCHJFLMB5LOZM3FG645CCW6TT7KJW4PBQXBUSAIE2ZZM74A, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.569779642 (76h54m34.440862677s) [noise=true , address=185.167.204.205:4024 , piece_id=2UHV7KDAKJJ4DDUU5MPBNLRP24OK5RQU6GGAVYUV5MEFLDDPGUZQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.570073130 (76h54m34.441156176s) [noise=true , address=81.171.14.70:4035 , piece_id=ZKLJ5VUPALKZBSPFCJQBZZUTMJ3PX2H3XMUZ5JVSW5BJBACRUYHA, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.570428618 (76h54m34.441511653s) [noise=true , address=81.171.14.70:4043 , piece_id=FVUGWZXJLQAOLUSTRKJAULSHDWKX5GGWA2BB52YP7WQEKU4NKS5Q, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.570785237 (76h54m34.441868282s) [noise=true , address=81.171.14.70:4044 , piece_id=QPYQGR62JDWWV3GM3CVDB6BLNSGNUTQUEHHQF74TYXFN25PMNPVQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.571235034 (76h54m34.442318069s) [noise=true , address=185.167.204.205:4035 , piece_id=LXUP2T5SN5SSV7IWZLGTSLCRWLFRZBODXX7W4FFVHZFO74I2CQHQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.571655133 (76h54m34.442738169s) [noise=true , address=81.171.14.70:4031 , piece_id=53QOUYYU4KS53TELJUM3FPEKOCB464OR5GKYGYB54RVRPD52RFWA, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.572102666 (76h54m34.443185701s) [noise=true , address=185.167.204.205:4034 , piece_id=4VH6HEDD7CTPMEOHYGKFXSYXHKOLPYB2N6BCCJT2DXVAMADI3KJQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:35.395419290 (76h54m35.266502335s) [upload=67229]: Next returned nil segment Dec 8 20:23:35.395432614 (76h54m35.26651565s) [upload=67229]: Waiting for error group managing segments... Dec 8 20:23:35.780649683 (76h54m35.651732718s) [noise=true , address=185.167.204.206:4037 , piece_id=XYY5NAATQJPHOK6JAMHXR3JH3QTLQF7OTA32BMST5HYQDYJ76UMA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.781236982 (76h54m35.652320027s) [noise=true , address=185.167.204.206:4056 , piece_id=IPDW6RCHJFLMB5LOZM3FG645CCW6TT7KJW4PBQXBUSAIE2ZZM74A, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.784267011 (76h54m35.655350046s) [noise=true , address=185.167.204.205:4030 , piece_id=O3QKGK2G5I5YYPRP5K57RTG4H35H2UDFROFU2JMSHKTFNZJDVUIA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.785164540 (76h54m35.656247585s) [noise=true , address=185.167.204.205:4012 , piece_id=QCJD2IHH4DEKTFN5ZPMZXPECBQ2IX2QPSFUZCSFR5HKCDOVYGGQA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.785585301 (76h54m35.656668336s) [noise=true , address=185.167.204.205:4002 , piece_id=PDSJ5KDCFQBURILRTWU2WSEG3UH7GX4GK57W4J75QADBPKQXRSYA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.787925598 (76h54m35.659008643s) [noise=true , address=185.167.204.206:4047 , piece_id=THE6CUP46D2OSXS34UR3TTBUP6FIVMGDGT7DVJDBB57BHMWBCZ7A, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.787994289 (76h54m35.659077324s) [noise=true , address=185.167.204.205:4034 , piece_id=4VH6HEDD7CTPMEOHYGKFXSYXHKOLPYB2N6BCCJT2DXVAMADI3KJQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.788899402 (76h54m35.659982437s) [noise=true , address=185.167.204.206:4050 , piece_id=7B4GLDQZUPQPLGPZPPN5CO3VOX3LLBMLIYA6IMALYITCHE4CZBOA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.791033055 (76h54m35.66211609s) [noise=true , address=185.167.204.205:4031 , piece_id=5XPLPLO4SPMX7ZBYHLUZTH7MBGCRESEEMP2OEI7LTTT5BN67XHMQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.792163116 (76h54m35.663246161s) [noise=true , address=185.167.204.206:4055 , piece_id=JESF36BPW3YEPXVI2GP4OALECNLYSASJHQRZVLNXKV5BQRDGOWWQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.792693356 (76h54m35.663776392s) [noise=true , address=185.167.204.206:4063 , piece_id=BRUAUJTOCPWNUOB6D5G6AJCTMJBRZRJXXXBZBIUUZZXXQB5OEX2Q, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.793171606 (76h54m35.664254662s) [noise=true , address=185.167.204.206:4045 , piece_id=IXV6HTIOVQR4HCGJIWTBQFLO733BMMNNMZ24Y34H6V6KYUQYPQRA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.798151289 (76h54m35.669234344s) [noise=true , address=185.167.204.206:4053 , piece_id=VGK5AKCI23P5ZDS427NS6PNNRIJ3QUXLRZ4ECLYN4UDSL3KAQAKA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.798189562 (76h54m35.669272597s) [noise=true , address=185.167.204.206:4040 , piece_id=4KFICIELLLPCLWQ7RIB7UUSWUK7SFX2PJK7JWCRU5S4HQNHBEWRQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.798633607 (76h54m35.669716642s) [noise=true , address=185.167.204.205:4035 , piece_id=LXUP2T5SN5SSV7IWZLGTSLCRWLFRZBODXX7W4FFVHZFO74I2CQHQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.803059726 (76h54m35.674142761s) [noise=true , address=185.167.204.205:4024 , piece_id=2UHV7KDAKJJ4DDUU5MPBNLRP24OK5RQU6GGAVYUV5MEFLDDPGUZQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.344154907 (76h54m36.215237952s) [noise=true , address=81.171.14.70:4036 , piece_id=FB3SB3XQDNT2HSK5FDGELGYWODFLU655QW4D23GEXIWQEZYFNFRA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.359261713 (76h54m36.230344758s) [noise=true , address=81.171.14.70:4031 , piece_id=53QOUYYU4KS53TELJUM3FPEKOCB464OR5GKYGYB54RVRPD52RFWA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.393865115 (76h54m36.26494815s) [noise=true , address=81.171.14.70:4035 , piece_id=ZKLJ5VUPALKZBSPFCJQBZZUTMJ3PX2H3XMUZ5JVSW5BJBACRUYHA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.394302157 (76h54m36.265385202s) [noise=true , address=81.171.14.70:4044 , piece_id=QPYQGR62JDWWV3GM3CVDB6BLNSGNUTQUEHHQF74TYXFN25PMNPVQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.430200674 (76h54m36.301283709s) [noise=true , address=81.171.14.70:4046 , piece_id=ZHZUULNXPFDSTOXXH6T7XIEPN23OOWCNAAUWPCZYNNDTLPRWXVKQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.800799297 (76h54m36.671882332s) [noise=true , address=81.171.14.70:4040 , piece_id=W2SOTP5SPLQ6J3PM57NZOLRVZNMTNSGPWDZKZC2KV2PBPH4TUBLA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.862864259 (76h54m36.733947295s) [noise=true , address=81.171.14.70:4038 , piece_id=CKN2RF2YSTEOK6HUXGVBVECYAP2CLGUF6KYNR5JTHRUZPTGVI73Q, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.953769972 (76h54m36.824853007s) [noise=true , address=81.171.14.70:4043 , piece_id=FVUGWZXJLQAOLUSTRKJAULSHDWKX5GGWA2BB52YP7WQEKU4NKS5Q, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.996603376 (76h54m36.867686421s) [noise=true , address=89.149.201.147:4006 , piece_id=UGREZBZRVKXMGBOLQEV2NC4COTXOTTRUEAGSPNWRWP22MCPHQP7Q, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:37.037518916 (76h54m36.908601951s) [noise=true , address=89.149.201.147:4014 , piece_id=AEG6KGPGM26LWTYTWGZGQDPBL7UDX263DFBXY7WZRXGMYEFPWXQQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:37.037537541 (76h54m36.908620586s) [seg_pos={0 0}, upload=67229]: Segment reached optimal threshold of 26 pieces. Dec 8 20:23:37.037730880 (76h54m36.908813915s) [noise=true , address=89.149.201.147:4023 , piece_id=5EPGXWBK6GNTKLQLWEIWYBC2OJHFR3M7F4X3X4WODJWNXID3PCQQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BaeSxjXneBKoGKVrRM686sq386Bhn1YzzBFG9we8fHbcc9tbv): protocol: expected piece hash; context canceled; EOF Dec 8 20:23:37.037737452 (76h54m36.908820497s) [noise=true , address=89.149.201.147:4023 , piece_id=5EPGXWBK6GNTKLQLWEIWYBC2OJHFR3M7F4X3X4WODJWNXID3PCQQ, seg_pos={0 0}, upload=67229]: Long tail cancellation due to optimal threshold Dec 8 20:23:37.037893039 (76h54m36.908976074s) [noise=true , address=89.149.201.147:4022 , piece_id=2VR3E4SXAAOR5RAVHFVUQQ7MBZQEEQYFZ2ZW5Z7YSCVOETZNFQWQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122Vj8XxpS6RnhVweQV3cZWFhj8nrzy3hZ9P7JsuTzEaZcD6iFE): protocol: expected piece hash; context canceled; EOF Dec 8 20:23:37.037900964 (76h54m36.908984019s) [noise=true , address=89.149.201.147:4022 , piece_id=2VR3E4SXAAOR5RAVHFVUQQ7MBZQEEQYFZ2ZW5Z7YSCVOETZNFQWQ, seg_pos={0 0}, upload=67229]: Long tail cancellation due to optimal threshold Dec 8 20:23:37.037926843 (76h54m36.909009889s) [seg_pos={0 0}, upload=67229]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 8 20:23:37.038007697 (76h54m36.909090732s) [upload=67229]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 20:23:37.038013398 (76h54m36.909096433s) [upload=67229]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 20:26:49.742746371 (76h57m49.613829416s) [upload=67229]: Done uploading segments... Dec 8 20:26:49.742860908 (76h57m49.613943943s) [upload=67229]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.234243373s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [....X....X....X.............................................] duration: 5.936973911s trace: https://jaeger.dev.storj.io/trace/4600f1753cc382a2 FAIL (2025-12-08 20:55:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.937002586s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 475.770948ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/51c3a25722f21601 FAIL (2025-12-08 20:55:43 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 475.79832ms core satellite eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........X................................X.................] duration: 5.715439377s trace: https://jaeger.dev.storj.io/trace/2eed98fdb4249ce9 FAIL (2025-12-08 20:55:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 20:43:00.763765824 (77h14m0.634848859s) [upload=67501]: Starting upload Dec 8 20:43:00.763879119 (77h14m0.634962164s) [upload=67501]: Done starting upload Dec 8 20:43:00.763936168 (77h14m0.635019203s) [upload=67501]: Uploading segments... Dec 8 20:43:00.764117523 (77h14m0.635200568s) [upload=67501]: Got next segment. Inline: false Dec 8 20:43:00.764157569 (77h14m0.635240604s) [upload=67501]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 20:43:00.764162089 (77h14m0.635245134s) [upload=67501]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 20:43:01.015312549 (77h14m0.886395594s) [seg_pos={0 0}, upload=67501]: Begin upload segment... Dec 8 20:43:01.015378243 (77h14m0.886461288s) [seg_pos={0 0}, upload=67501]: Done begin upload segment. Dec 8 20:43:01.015411717 (77h14m0.886494762s) [noise=true , address=145.79.227.40:4050 , piece_id=K2RW6IQERLFKEUD2ZTAZEX767U77PQWEXMXEH2SYDBQ7BT26WAHQ, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.018044751 (77h14m0.889127796s) [noise=true , address=145.79.227.33:4870 , piece_id=KY2FLO3KDFPWRXPUUFINFLIRLZDMHPUCSPKRCN7RYPVAXXUYMXWQ, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.019100842 (77h14m0.890183887s) [noise=true , address=145.79.227.31:4797 , piece_id=ECY3CM3GYNMM5RIEDHYI5NVWDCKEXRRKH4NCK3FV5ILW7W54LIIA, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.020021405 (77h14m0.89110444s) [noise=true , address=145.79.227.41:4087 , piece_id=QYRY35JJKPQEK4BAVFZ6DEBKJN7SBUHG66S44ZA7XRNOVKAVQ33Q, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.020812742 (77h14m0.891895787s) [noise=true , address=145.79.227.20:4437 , piece_id=CCMQ7GX5NBRAUKHEHERGP5SHETIGELARKU5AE5ETKZWQRJP7OOAQ, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.021487076 (77h14m0.892570111s) [noise=true , address=77.90.31.21:4879 , piece_id=WF6YF2HQIXM26JI5TQ3YW7OXHNYZH3E2VTHAQVWIUHZ3ME3CCG6Q, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.022677542 (77h14m0.893760588s) [noise=true , address=145.79.227.42:4123 , piece_id=NBMNEIGLWZHFGHY674WDGEDH4CGDCKYERXLB7UKXXI4MTZJ2HVEQ, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.022691720 (77h14m0.893774755s) [noise=true , address=77.90.31.18:4807 , piece_id=KDIWMU2XEJ7NTRCSG5J37ZUWIFURSJVK2BSDM7VJFKOB3ROANRGA, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.023821471 (77h14m0.894904506s) [noise=true , address=145.79.227.41:4085 , piece_id=LS54ZFJC7DEGOKQRUSZOY3I665N43TNXICDJTXQ3ZJ62OHPGETSQ, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.024740981 (77h14m0.895824026s) [noise=true , address=77.90.31.12:4589 , piece_id=WUVA4G7EEN4J4HF47KQ7FZKFINAN2CCYDXMRNHDZVBF6EAPVFS4Q, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.025197331 (77h14m0.896280366s) [noise=true , address=145.79.227.34:4906 , piece_id=VPDAFAMOLFSYUGFZIHRTADXWLYAE7IXILSPLT6QL2IPCOVUDPA3A, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.026609700 (77h14m0.897692745s) [noise=true , address=145.79.227.25:4618 , piece_id=AE7LMOAKNNIKVS4STMQJKZER7YUSFSUTSE7P5KXYTG4AQLFNQHLQ, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.027084214 (77h14m0.898167259s) [noise=true , address=145.79.227.44:4195 , piece_id=EYY7B3AXUMA63GQLGKQWPWGDZUBORTXR6QFYSEOB26ATZK2GJC6Q, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.028812575 (77h14m0.89989562s) [noise=true , address=145.79.227.38:4977 , piece_id=JW72CEO3WDGEB6BSSAR3M4Z5ANDBH63BLJZRSYAJDK7F7EXCEOZQ, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.030193064 (77h14m0.90127611s) [noise=true , address=145.79.227.24:4583 , piece_id=DRLVD3TA2Z6TJQ4WKA67KDTXK7Q32YJH2ES5QEQG7Z37DYBJBB6Q, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.036513970 (77h14m0.907597045s) [noise=true , address=77.90.31.13:4627 , piece_id=XMI4HVR2MQQ5KHN4GQOUCKFYJYJ2YZEGQGTAAXYKGS3OI6XPJLVA, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:02.093045104 (77h14m1.964128159s) [noise=true , address=77.90.31.21:4879 , piece_id=WF6YF2HQIXM26JI5TQ3YW7OXHNYZH3E2VTHAQVWIUHZ3ME3CCG6Q, seg_pos={0 0}, upload=67501]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: storage node overloaded, request limit: 2000 Dec 8 20:43:02.127847935 (77h14m1.99893098s) [noise=true , address=77.90.31.13:4627 , piece_id=XMI4HVR2MQQ5KHN4GQOUCKFYJYJ2YZEGQGTAAXYKGS3OI6XPJLVA, seg_pos={0 0}, upload=67501]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 8 20:43:02.156386568 (77h14m2.027469613s) [noise=true , address=77.90.31.12:4589 , piece_id=WUVA4G7EEN4J4HF47KQ7FZKFINAN2CCYDXMRNHDZVBF6EAPVFS4Q, seg_pos={0 0}, upload=67501]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 20:43:02.719927441 (77h14m2.591010486s) [upload=67501]: Next returned nil segment Dec 8 20:43:02.719942970 (77h14m2.591026005s) [upload=67501]: Waiting for error group managing segments... Dec 8 20:43:03.005012008 (77h14m2.876095063s) [noise=true , address=145.79.227.42:4123 , piece_id=NBMNEIGLWZHFGHY674WDGEDH4CGDCKYERXLB7UKXXI4MTZJ2HVEQ, seg_pos={0 0}, upload=67501]: Done uploading piece. err: <nil> Dec 8 20:43:03.012525866 (77h14m2.883608911s) [noise=true , address=145.79.227.41:4085 , piece_id=LS54ZFJC7DEGOKQRUSZOY3I665N43TNXICDJTXQ3ZJ62OHPGETSQ, seg_pos={0 0}, upload=67501]: Done uploading piece. err: <nil> Dec 8 20:43:03.015459803 (77h14m2.886542848s) [noise=true , address=145.79.227.41:4087 , piece_id=QYRY35JJKPQEK4BAVFZ6DEBKJN7SBUHG66S44ZA7XRNOVKAVQ33Q, seg_pos={0 0}, upload=67501]: Done uploading piece. err: <nil> Dec 8 20:43:03.020621582 (77h14m2.891704617s) [noise=true , address=145.79.227.20:4437 , piece_id=CCMQ7GX5NBRAUKHEHERGP5SHETIGELARKU5AE5ETKZWQRJP7OOAQ, seg_pos={0 0}, upload=67501]: Done uploading piece. err: <nil> Dec 8 20:43:03.020672098 (77h14m2.891755153s) [noise=true , address=145.79.227.31:4797 , piece_id=ECY3CM3GYNMM5RIEDHYI5NVWDCKEXRRKH4NCK3FV5ILW7W54LIIA, seg_pos={0 0}, upload=67501]: Done uploading piece. err: <nil> Dec 8 20:43:03.028139237 (77h14m2.899222282s) [noise=true , address=145.79.227.24:4583 , piece_id=DRLVD3TA2Z6TJQ4WKA67KDTXK7Q32YJH2ES5QEQG7Z37DYBJBB6Q, seg_pos={0 0}, upload=67501]: Done uploading piece. err: <nil> Dec 8 20:43:03.031290167 (77h14m2.902373202s) [noise=true , address=145.79.227.33:4870 , piece_id=KY2FLO3KDFPWRXPUUFINFLIRLZDMHPUCSPKRCN7RYPVAXXUYMXWQ, seg_pos={0 0}, upload=67501]: Done uploading piece. err: <nil> Dec 8 20:43:03.033756633 (77h14m2.904839668s) [noise=true , address=145.79.227.25:4618 , piece_id=AE7LMOAKNNIKVS4STMQJKZER7YUSFSUTSE7P5KXYTG4AQLFNQHLQ, seg_pos={0 0}, upload=67501]: Done uploading piece. err: <nil> Dec 8 20:43:03.039153600 (77h14m2.910236655s) [noise=true , address=145.79.227.40:4050 , piece_id=K2RW6IQERLFKEUD2ZTAZEX767U77PQWEXMXEH2SYDBQ7BT26WAHQ, seg_pos={0 0}, upload=67501]: Done uploading piece. err: <nil> Dec 8 20:43:03.040008869 (77h14m2.911091914s) [noise=true , address=145.79.227.38:4977 , piece_id=JW72CEO3WDGEB6BSSAR3M4Z5ANDBH63BLJZRSYAJDK7F7EXCEOZQ, seg_pos={0 0}, upload=67501]: Done uploading piece. err: <nil> Dec 8 20:43:03.178911548 (77h14m3.049994593s) [noise=true , address=77.90.31.18:4807 , piece_id=KDIWMU2XEJ7NTRCSG5J37ZUWIFURSJVK2BSDM7VJFKOB3ROANRGA, seg_pos={0 0}, upload=67501]: Done uploading piece. err: <nil> Dec 8 20:43:04.409564163 (77h14m4.280647208s) [noise=true , address=145.79.227.44:4195 , piece_id=EYY7B3AXUMA63GQLGKQWPWGDZUBORTXR6QFYSEOB26ATZK2GJC6Q, seg_pos={0 0}, upload=67501]: Done uploading piece. err: <nil> Dec 8 20:43:30.067753910 (77h14m29.938836955s) [noise=true , address=145.79.227.34:4906 , piece_id=VPDAFAMOLFSYUGFZIHRTADXWLYAE7IXILSPLT6QL2IPCOVUDPA3A, seg_pos={0 0}, upload=67501]: Done uploading piece. err: ecclient: upload failed (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw, address:145.79.227.34:4906): protocol: read tcp 188.166.225.170:36576->145.79.227.34:4906: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 20:43:30.067797914 (77h14m29.938880959s) [seg_pos={0 0}, upload=67501]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 20:43:30.323605733 (77h14m30.194688778s) [upload=67501]: Done uploading segments... Dec 8 20:43:30.323629097 (77h14m30.194712142s) [upload=67501]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.715457763s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................................................X....] duration: 3.46829443s trace: https://jaeger.dev.storj.io/trace/65b6db6cde985dc2 FAIL (2025-12-08 20:55:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.468322044s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 4.956164043s trace: https://jaeger.dev.storj.io/trace/49fc2034659f6881 FAIL (2025-12-08 20:54:59 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.956196275s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [..X...............X.........................................] duration: 5.942091166s trace: https://jaeger.dev.storj.io/trace/3996915dd8f30735 FAIL (2025-12-08 20:54:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== PGM26LWTYTWGZGQDPBL7UDX263DFBXY7WZRXGMYEFPWXQQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.569231809 (76h54m34.440315054s) [noise=true , address=89.149.201.147:4022 , piece_id=2VR3E4SXAAOR5RAVHFVUQQ7MBZQEEQYFZ2ZW5Z7YSCVOETZNFQWQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.569369882 (76h54m34.440452917s) [noise=true , address=185.167.204.206:4037 , piece_id=XYY5NAATQJPHOK6JAMHXR3JH3QTLQF7OTA32BMST5HYQDYJ76UMA, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.569672879 (76h54m34.440755924s) [noise=true , address=185.167.204.206:4056 , piece_id=IPDW6RCHJFLMB5LOZM3FG645CCW6TT7KJW4PBQXBUSAIE2ZZM74A, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.569779642 (76h54m34.440862677s) [noise=true , address=185.167.204.205:4024 , piece_id=2UHV7KDAKJJ4DDUU5MPBNLRP24OK5RQU6GGAVYUV5MEFLDDPGUZQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.570073130 (76h54m34.441156176s) [noise=true , address=81.171.14.70:4035 , piece_id=ZKLJ5VUPALKZBSPFCJQBZZUTMJ3PX2H3XMUZ5JVSW5BJBACRUYHA, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.570428618 (76h54m34.441511653s) [noise=true , address=81.171.14.70:4043 , piece_id=FVUGWZXJLQAOLUSTRKJAULSHDWKX5GGWA2BB52YP7WQEKU4NKS5Q, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.570785237 (76h54m34.441868282s) [noise=true , address=81.171.14.70:4044 , piece_id=QPYQGR62JDWWV3GM3CVDB6BLNSGNUTQUEHHQF74TYXFN25PMNPVQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.571235034 (76h54m34.442318069s) [noise=true , address=185.167.204.205:4035 , piece_id=LXUP2T5SN5SSV7IWZLGTSLCRWLFRZBODXX7W4FFVHZFO74I2CQHQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.571655133 (76h54m34.442738169s) [noise=true , address=81.171.14.70:4031 , piece_id=53QOUYYU4KS53TELJUM3FPEKOCB464OR5GKYGYB54RVRPD52RFWA, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.572102666 (76h54m34.443185701s) [noise=true , address=185.167.204.205:4034 , piece_id=4VH6HEDD7CTPMEOHYGKFXSYXHKOLPYB2N6BCCJT2DXVAMADI3KJQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:35.395419290 (76h54m35.266502335s) [upload=67229]: Next returned nil segment Dec 8 20:23:35.395432614 (76h54m35.26651565s) [upload=67229]: Waiting for error group managing segments... Dec 8 20:23:35.780649683 (76h54m35.651732718s) [noise=true , address=185.167.204.206:4037 , piece_id=XYY5NAATQJPHOK6JAMHXR3JH3QTLQF7OTA32BMST5HYQDYJ76UMA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.781236982 (76h54m35.652320027s) [noise=true , address=185.167.204.206:4056 , piece_id=IPDW6RCHJFLMB5LOZM3FG645CCW6TT7KJW4PBQXBUSAIE2ZZM74A, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.784267011 (76h54m35.655350046s) [noise=true , address=185.167.204.205:4030 , piece_id=O3QKGK2G5I5YYPRP5K57RTG4H35H2UDFROFU2JMSHKTFNZJDVUIA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.785164540 (76h54m35.656247585s) [noise=true , address=185.167.204.205:4012 , piece_id=QCJD2IHH4DEKTFN5ZPMZXPECBQ2IX2QPSFUZCSFR5HKCDOVYGGQA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.785585301 (76h54m35.656668336s) [noise=true , address=185.167.204.205:4002 , piece_id=PDSJ5KDCFQBURILRTWU2WSEG3UH7GX4GK57W4J75QADBPKQXRSYA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.787925598 (76h54m35.659008643s) [noise=true , address=185.167.204.206:4047 , piece_id=THE6CUP46D2OSXS34UR3TTBUP6FIVMGDGT7DVJDBB57BHMWBCZ7A, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.787994289 (76h54m35.659077324s) [noise=true , address=185.167.204.205:4034 , piece_id=4VH6HEDD7CTPMEOHYGKFXSYXHKOLPYB2N6BCCJT2DXVAMADI3KJQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.788899402 (76h54m35.659982437s) [noise=true , address=185.167.204.206:4050 , piece_id=7B4GLDQZUPQPLGPZPPN5CO3VOX3LLBMLIYA6IMALYITCHE4CZBOA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.791033055 (76h54m35.66211609s) [noise=true , address=185.167.204.205:4031 , piece_id=5XPLPLO4SPMX7ZBYHLUZTH7MBGCRESEEMP2OEI7LTTT5BN67XHMQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.792163116 (76h54m35.663246161s) [noise=true , address=185.167.204.206:4055 , piece_id=JESF36BPW3YEPXVI2GP4OALECNLYSASJHQRZVLNXKV5BQRDGOWWQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.792693356 (76h54m35.663776392s) [noise=true , address=185.167.204.206:4063 , piece_id=BRUAUJTOCPWNUOB6D5G6AJCTMJBRZRJXXXBZBIUUZZXXQB5OEX2Q, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.793171606 (76h54m35.664254662s) [noise=true , address=185.167.204.206:4045 , piece_id=IXV6HTIOVQR4HCGJIWTBQFLO733BMMNNMZ24Y34H6V6KYUQYPQRA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.798151289 (76h54m35.669234344s) [noise=true , address=185.167.204.206:4053 , piece_id=VGK5AKCI23P5ZDS427NS6PNNRIJ3QUXLRZ4ECLYN4UDSL3KAQAKA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.798189562 (76h54m35.669272597s) [noise=true , address=185.167.204.206:4040 , piece_id=4KFICIELLLPCLWQ7RIB7UUSWUK7SFX2PJK7JWCRU5S4HQNHBEWRQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.798633607 (76h54m35.669716642s) [noise=true , address=185.167.204.205:4035 , piece_id=LXUP2T5SN5SSV7IWZLGTSLCRWLFRZBODXX7W4FFVHZFO74I2CQHQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.803059726 (76h54m35.674142761s) [noise=true , address=185.167.204.205:4024 , piece_id=2UHV7KDAKJJ4DDUU5MPBNLRP24OK5RQU6GGAVYUV5MEFLDDPGUZQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.344154907 (76h54m36.215237952s) [noise=true , address=81.171.14.70:4036 , piece_id=FB3SB3XQDNT2HSK5FDGELGYWODFLU655QW4D23GEXIWQEZYFNFRA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.359261713 (76h54m36.230344758s) [noise=true , address=81.171.14.70:4031 , piece_id=53QOUYYU4KS53TELJUM3FPEKOCB464OR5GKYGYB54RVRPD52RFWA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.393865115 (76h54m36.26494815s) [noise=true , address=81.171.14.70:4035 , piece_id=ZKLJ5VUPALKZBSPFCJQBZZUTMJ3PX2H3XMUZ5JVSW5BJBACRUYHA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.394302157 (76h54m36.265385202s) [noise=true , address=81.171.14.70:4044 , piece_id=QPYQGR62JDWWV3GM3CVDB6BLNSGNUTQUEHHQF74TYXFN25PMNPVQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.430200674 (76h54m36.301283709s) [noise=true , address=81.171.14.70:4046 , piece_id=ZHZUULNXPFDSTOXXH6T7XIEPN23OOWCNAAUWPCZYNNDTLPRWXVKQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.800799297 (76h54m36.671882332s) [noise=true , address=81.171.14.70:4040 , piece_id=W2SOTP5SPLQ6J3PM57NZOLRVZNMTNSGPWDZKZC2KV2PBPH4TUBLA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.862864259 (76h54m36.733947295s) [noise=true , address=81.171.14.70:4038 , piece_id=CKN2RF2YSTEOK6HUXGVBVECYAP2CLGUF6KYNR5JTHRUZPTGVI73Q, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.953769972 (76h54m36.824853007s) [noise=true , address=81.171.14.70:4043 , piece_id=FVUGWZXJLQAOLUSTRKJAULSHDWKX5GGWA2BB52YP7WQEKU4NKS5Q, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.996603376 (76h54m36.867686421s) [noise=true , address=89.149.201.147:4006 , piece_id=UGREZBZRVKXMGBOLQEV2NC4COTXOTTRUEAGSPNWRWP22MCPHQP7Q, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:37.037518916 (76h54m36.908601951s) [noise=true , address=89.149.201.147:4014 , piece_id=AEG6KGPGM26LWTYTWGZGQDPBL7UDX263DFBXY7WZRXGMYEFPWXQQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:37.037537541 (76h54m36.908620586s) [seg_pos={0 0}, upload=67229]: Segment reached optimal threshold of 26 pieces. Dec 8 20:23:37.037730880 (76h54m36.908813915s) [noise=true , address=89.149.201.147:4023 , piece_id=5EPGXWBK6GNTKLQLWEIWYBC2OJHFR3M7F4X3X4WODJWNXID3PCQQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BaeSxjXneBKoGKVrRM686sq386Bhn1YzzBFG9we8fHbcc9tbv): protocol: expected piece hash; context canceled; EOF Dec 8 20:23:37.037737452 (76h54m36.908820497s) [noise=true , address=89.149.201.147:4023 , piece_id=5EPGXWBK6GNTKLQLWEIWYBC2OJHFR3M7F4X3X4WODJWNXID3PCQQ, seg_pos={0 0}, upload=67229]: Long tail cancellation due to optimal threshold Dec 8 20:23:37.037893039 (76h54m36.908976074s) [noise=true , address=89.149.201.147:4022 , piece_id=2VR3E4SXAAOR5RAVHFVUQQ7MBZQEEQYFZ2ZW5Z7YSCVOETZNFQWQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122Vj8XxpS6RnhVweQV3cZWFhj8nrzy3hZ9P7JsuTzEaZcD6iFE): protocol: expected piece hash; context canceled; EOF Dec 8 20:23:37.037900964 (76h54m36.908984019s) [noise=true , address=89.149.201.147:4022 , piece_id=2VR3E4SXAAOR5RAVHFVUQQ7MBZQEEQYFZ2ZW5Z7YSCVOETZNFQWQ, seg_pos={0 0}, upload=67229]: Long tail cancellation due to optimal threshold Dec 8 20:23:37.037926843 (76h54m36.909009889s) [seg_pos={0 0}, upload=67229]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 8 20:23:37.038007697 (76h54m36.909090732s) [upload=67229]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 20:23:37.038013398 (76h54m36.909096433s) [upload=67229]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 20:26:49.742746371 (76h57m49.613829416s) [upload=67229]: Done uploading segments... Dec 8 20:26:49.742860908 (76h57m49.613943943s) [upload=67229]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.94215041s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................................................X...] duration: 3.54295632s trace: https://jaeger.dev.storj.io/trace/65b6db6cde985dc2 FAIL (2025-12-08 20:54:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.543004091s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.....X....X....X............................................] duration: 5.769036283s trace: https://jaeger.dev.storj.io/trace/4600f1753cc382a2 FAIL (2025-12-08 20:54:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.769056621s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 2 out of 60 is failed (threshold=3) [X..................................................X........] duration: 466.099192ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/51c3a25722f21601 FAIL (2025-12-08 20:54:44 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 466.144678ms core satellite eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........X................................X................] duration: 5.688752525s trace: https://jaeger.dev.storj.io/trace/2eed98fdb4249ce9 FAIL (2025-12-08 20:54:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 20:43:00.763765824 (77h14m0.634848859s) [upload=67501]: Starting upload Dec 8 20:43:00.763879119 (77h14m0.634962164s) [upload=67501]: Done starting upload Dec 8 20:43:00.763936168 (77h14m0.635019203s) [upload=67501]: Uploading segments... Dec 8 20:43:00.764117523 (77h14m0.635200568s) [upload=67501]: Got next segment. Inline: false Dec 8 20:43:00.764157569 (77h14m0.635240604s) [upload=67501]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 20:43:00.764162089 (77h14m0.635245134s) [upload=67501]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 20:43:01.015312549 (77h14m0.886395594s) [seg_pos={0 0}, upload=67501]: Begin upload segment... Dec 8 20:43:01.015378243 (77h14m0.886461288s) [seg_pos={0 0}, upload=67501]: Done begin upload segment. Dec 8 20:43:01.015411717 (77h14m0.886494762s) [noise=true , address=145.79.227.40:4050 , piece_id=K2RW6IQERLFKEUD2ZTAZEX767U77PQWEXMXEH2SYDBQ7BT26WAHQ, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.018044751 (77h14m0.889127796s) [noise=true , address=145.79.227.33:4870 , piece_id=KY2FLO3KDFPWRXPUUFINFLIRLZDMHPUCSPKRCN7RYPVAXXUYMXWQ, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.019100842 (77h14m0.890183887s) [noise=true , address=145.79.227.31:4797 , piece_id=ECY3CM3GYNMM5RIEDHYI5NVWDCKEXRRKH4NCK3FV5ILW7W54LIIA, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.020021405 (77h14m0.89110444s) [noise=true , address=145.79.227.41:4087 , piece_id=QYRY35JJKPQEK4BAVFZ6DEBKJN7SBUHG66S44ZA7XRNOVKAVQ33Q, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.020812742 (77h14m0.891895787s) [noise=true , address=145.79.227.20:4437 , piece_id=CCMQ7GX5NBRAUKHEHERGP5SHETIGELARKU5AE5ETKZWQRJP7OOAQ, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.021487076 (77h14m0.892570111s) [noise=true , address=77.90.31.21:4879 , piece_id=WF6YF2HQIXM26JI5TQ3YW7OXHNYZH3E2VTHAQVWIUHZ3ME3CCG6Q, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.022677542 (77h14m0.893760588s) [noise=true , address=145.79.227.42:4123 , piece_id=NBMNEIGLWZHFGHY674WDGEDH4CGDCKYERXLB7UKXXI4MTZJ2HVEQ, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.022691720 (77h14m0.893774755s) [noise=true , address=77.90.31.18:4807 , piece_id=KDIWMU2XEJ7NTRCSG5J37ZUWIFURSJVK2BSDM7VJFKOB3ROANRGA, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.023821471 (77h14m0.894904506s) [noise=true , address=145.79.227.41:4085 , piece_id=LS54ZFJC7DEGOKQRUSZOY3I665N43TNXICDJTXQ3ZJ62OHPGETSQ, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.024740981 (77h14m0.895824026s) [noise=true , address=77.90.31.12:4589 , piece_id=WUVA4G7EEN4J4HF47KQ7FZKFINAN2CCYDXMRNHDZVBF6EAPVFS4Q, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.025197331 (77h14m0.896280366s) [noise=true , address=145.79.227.34:4906 , piece_id=VPDAFAMOLFSYUGFZIHRTADXWLYAE7IXILSPLT6QL2IPCOVUDPA3A, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.026609700 (77h14m0.897692745s) [noise=true , address=145.79.227.25:4618 , piece_id=AE7LMOAKNNIKVS4STMQJKZER7YUSFSUTSE7P5KXYTG4AQLFNQHLQ, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.027084214 (77h14m0.898167259s) [noise=true , address=145.79.227.44:4195 , piece_id=EYY7B3AXUMA63GQLGKQWPWGDZUBORTXR6QFYSEOB26ATZK2GJC6Q, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.028812575 (77h14m0.89989562s) [noise=true , address=145.79.227.38:4977 , piece_id=JW72CEO3WDGEB6BSSAR3M4Z5ANDBH63BLJZRSYAJDK7F7EXCEOZQ, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.030193064 (77h14m0.90127611s) [noise=true , address=145.79.227.24:4583 , piece_id=DRLVD3TA2Z6TJQ4WKA67KDTXK7Q32YJH2ES5QEQG7Z37DYBJBB6Q, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.036513970 (77h14m0.907597045s) [noise=true , address=77.90.31.13:4627 , piece_id=XMI4HVR2MQQ5KHN4GQOUCKFYJYJ2YZEGQGTAAXYKGS3OI6XPJLVA, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:02.093045104 (77h14m1.964128159s) [noise=true , address=77.90.31.21:4879 , piece_id=WF6YF2HQIXM26JI5TQ3YW7OXHNYZH3E2VTHAQVWIUHZ3ME3CCG6Q, seg_pos={0 0}, upload=67501]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: storage node overloaded, request limit: 2000 Dec 8 20:43:02.127847935 (77h14m1.99893098s) [noise=true , address=77.90.31.13:4627 , piece_id=XMI4HVR2MQQ5KHN4GQOUCKFYJYJ2YZEGQGTAAXYKGS3OI6XPJLVA, seg_pos={0 0}, upload=67501]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 8 20:43:02.156386568 (77h14m2.027469613s) [noise=true , address=77.90.31.12:4589 , piece_id=WUVA4G7EEN4J4HF47KQ7FZKFINAN2CCYDXMRNHDZVBF6EAPVFS4Q, seg_pos={0 0}, upload=67501]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 20:43:02.719927441 (77h14m2.591010486s) [upload=67501]: Next returned nil segment Dec 8 20:43:02.719942970 (77h14m2.591026005s) [upload=67501]: Waiting for error group managing segments... Dec 8 20:43:03.005012008 (77h14m2.876095063s) [noise=true , address=145.79.227.42:4123 , piece_id=NBMNEIGLWZHFGHY674WDGEDH4CGDCKYERXLB7UKXXI4MTZJ2HVEQ, seg_pos={0 0}, upload=67501]: Done uploading piece. err: <nil> Dec 8 20:43:03.012525866 (77h14m2.883608911s) [noise=true , address=145.79.227.41:4085 , piece_id=LS54ZFJC7DEGOKQRUSZOY3I665N43TNXICDJTXQ3ZJ62OHPGETSQ, seg_pos={0 0}, upload=67501]: Done uploading piece. err: <nil> Dec 8 20:43:03.015459803 (77h14m2.886542848s) [noise=true , address=145.79.227.41:4087 , piece_id=QYRY35JJKPQEK4BAVFZ6DEBKJN7SBUHG66S44ZA7XRNOVKAVQ33Q, seg_pos={0 0}, upload=67501]: Done uploading piece. err: <nil> Dec 8 20:43:03.020621582 (77h14m2.891704617s) [noise=true , address=145.79.227.20:4437 , piece_id=CCMQ7GX5NBRAUKHEHERGP5SHETIGELARKU5AE5ETKZWQRJP7OOAQ, seg_pos={0 0}, upload=67501]: Done uploading piece. err: <nil> Dec 8 20:43:03.020672098 (77h14m2.891755153s) [noise=true , address=145.79.227.31:4797 , piece_id=ECY3CM3GYNMM5RIEDHYI5NVWDCKEXRRKH4NCK3FV5ILW7W54LIIA, seg_pos={0 0}, upload=67501]: Done uploading piece. err: <nil> Dec 8 20:43:03.028139237 (77h14m2.899222282s) [noise=true , address=145.79.227.24:4583 , piece_id=DRLVD3TA2Z6TJQ4WKA67KDTXK7Q32YJH2ES5QEQG7Z37DYBJBB6Q, seg_pos={0 0}, upload=67501]: Done uploading piece. err: <nil> Dec 8 20:43:03.031290167 (77h14m2.902373202s) [noise=true , address=145.79.227.33:4870 , piece_id=KY2FLO3KDFPWRXPUUFINFLIRLZDMHPUCSPKRCN7RYPVAXXUYMXWQ, seg_pos={0 0}, upload=67501]: Done uploading piece. err: <nil> Dec 8 20:43:03.033756633 (77h14m2.904839668s) [noise=true , address=145.79.227.25:4618 , piece_id=AE7LMOAKNNIKVS4STMQJKZER7YUSFSUTSE7P5KXYTG4AQLFNQHLQ, seg_pos={0 0}, upload=67501]: Done uploading piece. err: <nil> Dec 8 20:43:03.039153600 (77h14m2.910236655s) [noise=true , address=145.79.227.40:4050 , piece_id=K2RW6IQERLFKEUD2ZTAZEX767U77PQWEXMXEH2SYDBQ7BT26WAHQ, seg_pos={0 0}, upload=67501]: Done uploading piece. err: <nil> Dec 8 20:43:03.040008869 (77h14m2.911091914s) [noise=true , address=145.79.227.38:4977 , piece_id=JW72CEO3WDGEB6BSSAR3M4Z5ANDBH63BLJZRSYAJDK7F7EXCEOZQ, seg_pos={0 0}, upload=67501]: Done uploading piece. err: <nil> Dec 8 20:43:03.178911548 (77h14m3.049994593s) [noise=true , address=77.90.31.18:4807 , piece_id=KDIWMU2XEJ7NTRCSG5J37ZUWIFURSJVK2BSDM7VJFKOB3ROANRGA, seg_pos={0 0}, upload=67501]: Done uploading piece. err: <nil> Dec 8 20:43:04.409564163 (77h14m4.280647208s) [noise=true , address=145.79.227.44:4195 , piece_id=EYY7B3AXUMA63GQLGKQWPWGDZUBORTXR6QFYSEOB26ATZK2GJC6Q, seg_pos={0 0}, upload=67501]: Done uploading piece. err: <nil> Dec 8 20:43:30.067753910 (77h14m29.938836955s) [noise=true , address=145.79.227.34:4906 , piece_id=VPDAFAMOLFSYUGFZIHRTADXWLYAE7IXILSPLT6QL2IPCOVUDPA3A, seg_pos={0 0}, upload=67501]: Done uploading piece. err: ecclient: upload failed (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw, address:145.79.227.34:4906): protocol: read tcp 188.166.225.170:36576->145.79.227.34:4906: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 20:43:30.067797914 (77h14m29.938880959s) [seg_pos={0 0}, upload=67501]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 20:43:30.323605733 (77h14m30.194688778s) [upload=67501]: Done uploading segments... Dec 8 20:43:30.323629097 (77h14m30.194712142s) [upload=67501]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.688780247s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [...X...............X........................................] duration: 5.347611733s trace: https://jaeger.dev.storj.io/trace/3996915dd8f30735 FAIL (2025-12-08 20:54:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== PGM26LWTYTWGZGQDPBL7UDX263DFBXY7WZRXGMYEFPWXQQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.569231809 (76h54m34.440315054s) [noise=true , address=89.149.201.147:4022 , piece_id=2VR3E4SXAAOR5RAVHFVUQQ7MBZQEEQYFZ2ZW5Z7YSCVOETZNFQWQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.569369882 (76h54m34.440452917s) [noise=true , address=185.167.204.206:4037 , piece_id=XYY5NAATQJPHOK6JAMHXR3JH3QTLQF7OTA32BMST5HYQDYJ76UMA, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.569672879 (76h54m34.440755924s) [noise=true , address=185.167.204.206:4056 , piece_id=IPDW6RCHJFLMB5LOZM3FG645CCW6TT7KJW4PBQXBUSAIE2ZZM74A, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.569779642 (76h54m34.440862677s) [noise=true , address=185.167.204.205:4024 , piece_id=2UHV7KDAKJJ4DDUU5MPBNLRP24OK5RQU6GGAVYUV5MEFLDDPGUZQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.570073130 (76h54m34.441156176s) [noise=true , address=81.171.14.70:4035 , piece_id=ZKLJ5VUPALKZBSPFCJQBZZUTMJ3PX2H3XMUZ5JVSW5BJBACRUYHA, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.570428618 (76h54m34.441511653s) [noise=true , address=81.171.14.70:4043 , piece_id=FVUGWZXJLQAOLUSTRKJAULSHDWKX5GGWA2BB52YP7WQEKU4NKS5Q, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.570785237 (76h54m34.441868282s) [noise=true , address=81.171.14.70:4044 , piece_id=QPYQGR62JDWWV3GM3CVDB6BLNSGNUTQUEHHQF74TYXFN25PMNPVQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.571235034 (76h54m34.442318069s) [noise=true , address=185.167.204.205:4035 , piece_id=LXUP2T5SN5SSV7IWZLGTSLCRWLFRZBODXX7W4FFVHZFO74I2CQHQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.571655133 (76h54m34.442738169s) [noise=true , address=81.171.14.70:4031 , piece_id=53QOUYYU4KS53TELJUM3FPEKOCB464OR5GKYGYB54RVRPD52RFWA, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.572102666 (76h54m34.443185701s) [noise=true , address=185.167.204.205:4034 , piece_id=4VH6HEDD7CTPMEOHYGKFXSYXHKOLPYB2N6BCCJT2DXVAMADI3KJQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:35.395419290 (76h54m35.266502335s) [upload=67229]: Next returned nil segment Dec 8 20:23:35.395432614 (76h54m35.26651565s) [upload=67229]: Waiting for error group managing segments... Dec 8 20:23:35.780649683 (76h54m35.651732718s) [noise=true , address=185.167.204.206:4037 , piece_id=XYY5NAATQJPHOK6JAMHXR3JH3QTLQF7OTA32BMST5HYQDYJ76UMA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.781236982 (76h54m35.652320027s) [noise=true , address=185.167.204.206:4056 , piece_id=IPDW6RCHJFLMB5LOZM3FG645CCW6TT7KJW4PBQXBUSAIE2ZZM74A, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.784267011 (76h54m35.655350046s) [noise=true , address=185.167.204.205:4030 , piece_id=O3QKGK2G5I5YYPRP5K57RTG4H35H2UDFROFU2JMSHKTFNZJDVUIA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.785164540 (76h54m35.656247585s) [noise=true , address=185.167.204.205:4012 , piece_id=QCJD2IHH4DEKTFN5ZPMZXPECBQ2IX2QPSFUZCSFR5HKCDOVYGGQA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.785585301 (76h54m35.656668336s) [noise=true , address=185.167.204.205:4002 , piece_id=PDSJ5KDCFQBURILRTWU2WSEG3UH7GX4GK57W4J75QADBPKQXRSYA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.787925598 (76h54m35.659008643s) [noise=true , address=185.167.204.206:4047 , piece_id=THE6CUP46D2OSXS34UR3TTBUP6FIVMGDGT7DVJDBB57BHMWBCZ7A, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.787994289 (76h54m35.659077324s) [noise=true , address=185.167.204.205:4034 , piece_id=4VH6HEDD7CTPMEOHYGKFXSYXHKOLPYB2N6BCCJT2DXVAMADI3KJQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.788899402 (76h54m35.659982437s) [noise=true , address=185.167.204.206:4050 , piece_id=7B4GLDQZUPQPLGPZPPN5CO3VOX3LLBMLIYA6IMALYITCHE4CZBOA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.791033055 (76h54m35.66211609s) [noise=true , address=185.167.204.205:4031 , piece_id=5XPLPLO4SPMX7ZBYHLUZTH7MBGCRESEEMP2OEI7LTTT5BN67XHMQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.792163116 (76h54m35.663246161s) [noise=true , address=185.167.204.206:4055 , piece_id=JESF36BPW3YEPXVI2GP4OALECNLYSASJHQRZVLNXKV5BQRDGOWWQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.792693356 (76h54m35.663776392s) [noise=true , address=185.167.204.206:4063 , piece_id=BRUAUJTOCPWNUOB6D5G6AJCTMJBRZRJXXXBZBIUUZZXXQB5OEX2Q, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.793171606 (76h54m35.664254662s) [noise=true , address=185.167.204.206:4045 , piece_id=IXV6HTIOVQR4HCGJIWTBQFLO733BMMNNMZ24Y34H6V6KYUQYPQRA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.798151289 (76h54m35.669234344s) [noise=true , address=185.167.204.206:4053 , piece_id=VGK5AKCI23P5ZDS427NS6PNNRIJ3QUXLRZ4ECLYN4UDSL3KAQAKA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.798189562 (76h54m35.669272597s) [noise=true , address=185.167.204.206:4040 , piece_id=4KFICIELLLPCLWQ7RIB7UUSWUK7SFX2PJK7JWCRU5S4HQNHBEWRQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.798633607 (76h54m35.669716642s) [noise=true , address=185.167.204.205:4035 , piece_id=LXUP2T5SN5SSV7IWZLGTSLCRWLFRZBODXX7W4FFVHZFO74I2CQHQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.803059726 (76h54m35.674142761s) [noise=true , address=185.167.204.205:4024 , piece_id=2UHV7KDAKJJ4DDUU5MPBNLRP24OK5RQU6GGAVYUV5MEFLDDPGUZQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.344154907 (76h54m36.215237952s) [noise=true , address=81.171.14.70:4036 , piece_id=FB3SB3XQDNT2HSK5FDGELGYWODFLU655QW4D23GEXIWQEZYFNFRA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.359261713 (76h54m36.230344758s) [noise=true , address=81.171.14.70:4031 , piece_id=53QOUYYU4KS53TELJUM3FPEKOCB464OR5GKYGYB54RVRPD52RFWA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.393865115 (76h54m36.26494815s) [noise=true , address=81.171.14.70:4035 , piece_id=ZKLJ5VUPALKZBSPFCJQBZZUTMJ3PX2H3XMUZ5JVSW5BJBACRUYHA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.394302157 (76h54m36.265385202s) [noise=true , address=81.171.14.70:4044 , piece_id=QPYQGR62JDWWV3GM3CVDB6BLNSGNUTQUEHHQF74TYXFN25PMNPVQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.430200674 (76h54m36.301283709s) [noise=true , address=81.171.14.70:4046 , piece_id=ZHZUULNXPFDSTOXXH6T7XIEPN23OOWCNAAUWPCZYNNDTLPRWXVKQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.800799297 (76h54m36.671882332s) [noise=true , address=81.171.14.70:4040 , piece_id=W2SOTP5SPLQ6J3PM57NZOLRVZNMTNSGPWDZKZC2KV2PBPH4TUBLA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.862864259 (76h54m36.733947295s) [noise=true , address=81.171.14.70:4038 , piece_id=CKN2RF2YSTEOK6HUXGVBVECYAP2CLGUF6KYNR5JTHRUZPTGVI73Q, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.953769972 (76h54m36.824853007s) [noise=true , address=81.171.14.70:4043 , piece_id=FVUGWZXJLQAOLUSTRKJAULSHDWKX5GGWA2BB52YP7WQEKU4NKS5Q, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.996603376 (76h54m36.867686421s) [noise=true , address=89.149.201.147:4006 , piece_id=UGREZBZRVKXMGBOLQEV2NC4COTXOTTRUEAGSPNWRWP22MCPHQP7Q, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:37.037518916 (76h54m36.908601951s) [noise=true , address=89.149.201.147:4014 , piece_id=AEG6KGPGM26LWTYTWGZGQDPBL7UDX263DFBXY7WZRXGMYEFPWXQQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:37.037537541 (76h54m36.908620586s) [seg_pos={0 0}, upload=67229]: Segment reached optimal threshold of 26 pieces. Dec 8 20:23:37.037730880 (76h54m36.908813915s) [noise=true , address=89.149.201.147:4023 , piece_id=5EPGXWBK6GNTKLQLWEIWYBC2OJHFR3M7F4X3X4WODJWNXID3PCQQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BaeSxjXneBKoGKVrRM686sq386Bhn1YzzBFG9we8fHbcc9tbv): protocol: expected piece hash; context canceled; EOF Dec 8 20:23:37.037737452 (76h54m36.908820497s) [noise=true , address=89.149.201.147:4023 , piece_id=5EPGXWBK6GNTKLQLWEIWYBC2OJHFR3M7F4X3X4WODJWNXID3PCQQ, seg_pos={0 0}, upload=67229]: Long tail cancellation due to optimal threshold Dec 8 20:23:37.037893039 (76h54m36.908976074s) [noise=true , address=89.149.201.147:4022 , piece_id=2VR3E4SXAAOR5RAVHFVUQQ7MBZQEEQYFZ2ZW5Z7YSCVOETZNFQWQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122Vj8XxpS6RnhVweQV3cZWFhj8nrzy3hZ9P7JsuTzEaZcD6iFE): protocol: expected piece hash; context canceled; EOF Dec 8 20:23:37.037900964 (76h54m36.908984019s) [noise=true , address=89.149.201.147:4022 , piece_id=2VR3E4SXAAOR5RAVHFVUQQ7MBZQEEQYFZ2ZW5Z7YSCVOETZNFQWQ, seg_pos={0 0}, upload=67229]: Long tail cancellation due to optimal threshold Dec 8 20:23:37.037926843 (76h54m36.909009889s) [seg_pos={0 0}, upload=67229]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 8 20:23:37.038007697 (76h54m36.909090732s) [upload=67229]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 20:23:37.038013398 (76h54m36.909096433s) [upload=67229]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 20:26:49.742746371 (76h57m49.613829416s) [upload=67229]: Done uploading segments... Dec 8 20:26:49.742860908 (76h57m49.613943943s) [upload=67229]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.347646669s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........X................................X...............] duration: 5.758886352s trace: https://jaeger.dev.storj.io/trace/2eed98fdb4249ce9 FAIL (2025-12-08 20:54:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 20:43:00.763765824 (77h14m0.634848859s) [upload=67501]: Starting upload Dec 8 20:43:00.763879119 (77h14m0.634962164s) [upload=67501]: Done starting upload Dec 8 20:43:00.763936168 (77h14m0.635019203s) [upload=67501]: Uploading segments... Dec 8 20:43:00.764117523 (77h14m0.635200568s) [upload=67501]: Got next segment. Inline: false Dec 8 20:43:00.764157569 (77h14m0.635240604s) [upload=67501]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 20:43:00.764162089 (77h14m0.635245134s) [upload=67501]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 20:43:01.015312549 (77h14m0.886395594s) [seg_pos={0 0}, upload=67501]: Begin upload segment... Dec 8 20:43:01.015378243 (77h14m0.886461288s) [seg_pos={0 0}, upload=67501]: Done begin upload segment. Dec 8 20:43:01.015411717 (77h14m0.886494762s) [noise=true , address=145.79.227.40:4050 , piece_id=K2RW6IQERLFKEUD2ZTAZEX767U77PQWEXMXEH2SYDBQ7BT26WAHQ, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.018044751 (77h14m0.889127796s) [noise=true , address=145.79.227.33:4870 , piece_id=KY2FLO3KDFPWRXPUUFINFLIRLZDMHPUCSPKRCN7RYPVAXXUYMXWQ, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.019100842 (77h14m0.890183887s) [noise=true , address=145.79.227.31:4797 , piece_id=ECY3CM3GYNMM5RIEDHYI5NVWDCKEXRRKH4NCK3FV5ILW7W54LIIA, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.020021405 (77h14m0.89110444s) [noise=true , address=145.79.227.41:4087 , piece_id=QYRY35JJKPQEK4BAVFZ6DEBKJN7SBUHG66S44ZA7XRNOVKAVQ33Q, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.020812742 (77h14m0.891895787s) [noise=true , address=145.79.227.20:4437 , piece_id=CCMQ7GX5NBRAUKHEHERGP5SHETIGELARKU5AE5ETKZWQRJP7OOAQ, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.021487076 (77h14m0.892570111s) [noise=true , address=77.90.31.21:4879 , piece_id=WF6YF2HQIXM26JI5TQ3YW7OXHNYZH3E2VTHAQVWIUHZ3ME3CCG6Q, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.022677542 (77h14m0.893760588s) [noise=true , address=145.79.227.42:4123 , piece_id=NBMNEIGLWZHFGHY674WDGEDH4CGDCKYERXLB7UKXXI4MTZJ2HVEQ, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.022691720 (77h14m0.893774755s) [noise=true , address=77.90.31.18:4807 , piece_id=KDIWMU2XEJ7NTRCSG5J37ZUWIFURSJVK2BSDM7VJFKOB3ROANRGA, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.023821471 (77h14m0.894904506s) [noise=true , address=145.79.227.41:4085 , piece_id=LS54ZFJC7DEGOKQRUSZOY3I665N43TNXICDJTXQ3ZJ62OHPGETSQ, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.024740981 (77h14m0.895824026s) [noise=true , address=77.90.31.12:4589 , piece_id=WUVA4G7EEN4J4HF47KQ7FZKFINAN2CCYDXMRNHDZVBF6EAPVFS4Q, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.025197331 (77h14m0.896280366s) [noise=true , address=145.79.227.34:4906 , piece_id=VPDAFAMOLFSYUGFZIHRTADXWLYAE7IXILSPLT6QL2IPCOVUDPA3A, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.026609700 (77h14m0.897692745s) [noise=true , address=145.79.227.25:4618 , piece_id=AE7LMOAKNNIKVS4STMQJKZER7YUSFSUTSE7P5KXYTG4AQLFNQHLQ, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.027084214 (77h14m0.898167259s) [noise=true , address=145.79.227.44:4195 , piece_id=EYY7B3AXUMA63GQLGKQWPWGDZUBORTXR6QFYSEOB26ATZK2GJC6Q, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.028812575 (77h14m0.89989562s) [noise=true , address=145.79.227.38:4977 , piece_id=JW72CEO3WDGEB6BSSAR3M4Z5ANDBH63BLJZRSYAJDK7F7EXCEOZQ, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.030193064 (77h14m0.90127611s) [noise=true , address=145.79.227.24:4583 , piece_id=DRLVD3TA2Z6TJQ4WKA67KDTXK7Q32YJH2ES5QEQG7Z37DYBJBB6Q, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.036513970 (77h14m0.907597045s) [noise=true , address=77.90.31.13:4627 , piece_id=XMI4HVR2MQQ5KHN4GQOUCKFYJYJ2YZEGQGTAAXYKGS3OI6XPJLVA, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:02.093045104 (77h14m1.964128159s) [noise=true , address=77.90.31.21:4879 , piece_id=WF6YF2HQIXM26JI5TQ3YW7OXHNYZH3E2VTHAQVWIUHZ3ME3CCG6Q, seg_pos={0 0}, upload=67501]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: storage node overloaded, request limit: 2000 Dec 8 20:43:02.127847935 (77h14m1.99893098s) [noise=true , address=77.90.31.13:4627 , piece_id=XMI4HVR2MQQ5KHN4GQOUCKFYJYJ2YZEGQGTAAXYKGS3OI6XPJLVA, seg_pos={0 0}, upload=67501]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 8 20:43:02.156386568 (77h14m2.027469613s) [noise=true , address=77.90.31.12:4589 , piece_id=WUVA4G7EEN4J4HF47KQ7FZKFINAN2CCYDXMRNHDZVBF6EAPVFS4Q, seg_pos={0 0}, upload=67501]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 20:43:02.719927441 (77h14m2.591010486s) [upload=67501]: Next returned nil segment Dec 8 20:43:02.719942970 (77h14m2.591026005s) [upload=67501]: Waiting for error group managing segments... Dec 8 20:43:03.005012008 (77h14m2.876095063s) [noise=true , address=145.79.227.42:4123 , piece_id=NBMNEIGLWZHFGHY674WDGEDH4CGDCKYERXLB7UKXXI4MTZJ2HVEQ, seg_pos={0 0}, upload=67501]: Done uploading piece. err: <nil> Dec 8 20:43:03.012525866 (77h14m2.883608911s) [noise=true , address=145.79.227.41:4085 , piece_id=LS54ZFJC7DEGOKQRUSZOY3I665N43TNXICDJTXQ3ZJ62OHPGETSQ, seg_pos={0 0}, upload=67501]: Done uploading piece. err: <nil> Dec 8 20:43:03.015459803 (77h14m2.886542848s) [noise=true , address=145.79.227.41:4087 , piece_id=QYRY35JJKPQEK4BAVFZ6DEBKJN7SBUHG66S44ZA7XRNOVKAVQ33Q, seg_pos={0 0}, upload=67501]: Done uploading piece. err: <nil> Dec 8 20:43:03.020621582 (77h14m2.891704617s) [noise=true , address=145.79.227.20:4437 , piece_id=CCMQ7GX5NBRAUKHEHERGP5SHETIGELARKU5AE5ETKZWQRJP7OOAQ, seg_pos={0 0}, upload=67501]: Done uploading piece. err: <nil> Dec 8 20:43:03.020672098 (77h14m2.891755153s) [noise=true , address=145.79.227.31:4797 , piece_id=ECY3CM3GYNMM5RIEDHYI5NVWDCKEXRRKH4NCK3FV5ILW7W54LIIA, seg_pos={0 0}, upload=67501]: Done uploading piece. err: <nil> Dec 8 20:43:03.028139237 (77h14m2.899222282s) [noise=true , address=145.79.227.24:4583 , piece_id=DRLVD3TA2Z6TJQ4WKA67KDTXK7Q32YJH2ES5QEQG7Z37DYBJBB6Q, seg_pos={0 0}, upload=67501]: Done uploading piece. err: <nil> Dec 8 20:43:03.031290167 (77h14m2.902373202s) [noise=true , address=145.79.227.33:4870 , piece_id=KY2FLO3KDFPWRXPUUFINFLIRLZDMHPUCSPKRCN7RYPVAXXUYMXWQ, seg_pos={0 0}, upload=67501]: Done uploading piece. err: <nil> Dec 8 20:43:03.033756633 (77h14m2.904839668s) [noise=true , address=145.79.227.25:4618 , piece_id=AE7LMOAKNNIKVS4STMQJKZER7YUSFSUTSE7P5KXYTG4AQLFNQHLQ, seg_pos={0 0}, upload=67501]: Done uploading piece. err: <nil> Dec 8 20:43:03.039153600 (77h14m2.910236655s) [noise=true , address=145.79.227.40:4050 , piece_id=K2RW6IQERLFKEUD2ZTAZEX767U77PQWEXMXEH2SYDBQ7BT26WAHQ, seg_pos={0 0}, upload=67501]: Done uploading piece. err: <nil> Dec 8 20:43:03.040008869 (77h14m2.911091914s) [noise=true , address=145.79.227.38:4977 , piece_id=JW72CEO3WDGEB6BSSAR3M4Z5ANDBH63BLJZRSYAJDK7F7EXCEOZQ, seg_pos={0 0}, upload=67501]: Done uploading piece. err: <nil> Dec 8 20:43:03.178911548 (77h14m3.049994593s) [noise=true , address=77.90.31.18:4807 , piece_id=KDIWMU2XEJ7NTRCSG5J37ZUWIFURSJVK2BSDM7VJFKOB3ROANRGA, seg_pos={0 0}, upload=67501]: Done uploading piece. err: <nil> Dec 8 20:43:04.409564163 (77h14m4.280647208s) [noise=true , address=145.79.227.44:4195 , piece_id=EYY7B3AXUMA63GQLGKQWPWGDZUBORTXR6QFYSEOB26ATZK2GJC6Q, seg_pos={0 0}, upload=67501]: Done uploading piece. err: <nil> Dec 8 20:43:30.067753910 (77h14m29.938836955s) [noise=true , address=145.79.227.34:4906 , piece_id=VPDAFAMOLFSYUGFZIHRTADXWLYAE7IXILSPLT6QL2IPCOVUDPA3A, seg_pos={0 0}, upload=67501]: Done uploading piece. err: ecclient: upload failed (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw, address:145.79.227.34:4906): protocol: read tcp 188.166.225.170:36576->145.79.227.34:4906: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 20:43:30.067797914 (77h14m29.938880959s) [seg_pos={0 0}, upload=67501]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 20:43:30.323605733 (77h14m30.194688778s) [upload=67501]: Done uploading segments... Dec 8 20:43:30.323629097 (77h14m30.194712142s) [upload=67501]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.758923091s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................................................X..] duration: 3.434198146s trace: https://jaeger.dev.storj.io/trace/65b6db6cde985dc2 FAIL (2025-12-08 20:54:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.434215699s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 2 out of 60 is failed (threshold=3) [.X..................................................X.......] duration: 470.055926ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/51c3a25722f21601 FAIL (2025-12-08 20:53:57 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 470.091994ms core satellite eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 5.266995238s trace: https://jaeger.dev.storj.io/trace/49fc2034659f6881 FAIL (2025-12-08 20:53:49 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.267020266s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [....X...............X.......................................] duration: 5.1257503s trace: https://jaeger.dev.storj.io/trace/3996915dd8f30735 FAIL (2025-12-08 20:53:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== PGM26LWTYTWGZGQDPBL7UDX263DFBXY7WZRXGMYEFPWXQQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.569231809 (76h54m34.440315054s) [noise=true , address=89.149.201.147:4022 , piece_id=2VR3E4SXAAOR5RAVHFVUQQ7MBZQEEQYFZ2ZW5Z7YSCVOETZNFQWQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.569369882 (76h54m34.440452917s) [noise=true , address=185.167.204.206:4037 , piece_id=XYY5NAATQJPHOK6JAMHXR3JH3QTLQF7OTA32BMST5HYQDYJ76UMA, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.569672879 (76h54m34.440755924s) [noise=true , address=185.167.204.206:4056 , piece_id=IPDW6RCHJFLMB5LOZM3FG645CCW6TT7KJW4PBQXBUSAIE2ZZM74A, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.569779642 (76h54m34.440862677s) [noise=true , address=185.167.204.205:4024 , piece_id=2UHV7KDAKJJ4DDUU5MPBNLRP24OK5RQU6GGAVYUV5MEFLDDPGUZQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.570073130 (76h54m34.441156176s) [noise=true , address=81.171.14.70:4035 , piece_id=ZKLJ5VUPALKZBSPFCJQBZZUTMJ3PX2H3XMUZ5JVSW5BJBACRUYHA, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.570428618 (76h54m34.441511653s) [noise=true , address=81.171.14.70:4043 , piece_id=FVUGWZXJLQAOLUSTRKJAULSHDWKX5GGWA2BB52YP7WQEKU4NKS5Q, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.570785237 (76h54m34.441868282s) [noise=true , address=81.171.14.70:4044 , piece_id=QPYQGR62JDWWV3GM3CVDB6BLNSGNUTQUEHHQF74TYXFN25PMNPVQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.571235034 (76h54m34.442318069s) [noise=true , address=185.167.204.205:4035 , piece_id=LXUP2T5SN5SSV7IWZLGTSLCRWLFRZBODXX7W4FFVHZFO74I2CQHQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.571655133 (76h54m34.442738169s) [noise=true , address=81.171.14.70:4031 , piece_id=53QOUYYU4KS53TELJUM3FPEKOCB464OR5GKYGYB54RVRPD52RFWA, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.572102666 (76h54m34.443185701s) [noise=true , address=185.167.204.205:4034 , piece_id=4VH6HEDD7CTPMEOHYGKFXSYXHKOLPYB2N6BCCJT2DXVAMADI3KJQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:35.395419290 (76h54m35.266502335s) [upload=67229]: Next returned nil segment Dec 8 20:23:35.395432614 (76h54m35.26651565s) [upload=67229]: Waiting for error group managing segments... Dec 8 20:23:35.780649683 (76h54m35.651732718s) [noise=true , address=185.167.204.206:4037 , piece_id=XYY5NAATQJPHOK6JAMHXR3JH3QTLQF7OTA32BMST5HYQDYJ76UMA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.781236982 (76h54m35.652320027s) [noise=true , address=185.167.204.206:4056 , piece_id=IPDW6RCHJFLMB5LOZM3FG645CCW6TT7KJW4PBQXBUSAIE2ZZM74A, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.784267011 (76h54m35.655350046s) [noise=true , address=185.167.204.205:4030 , piece_id=O3QKGK2G5I5YYPRP5K57RTG4H35H2UDFROFU2JMSHKTFNZJDVUIA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.785164540 (76h54m35.656247585s) [noise=true , address=185.167.204.205:4012 , piece_id=QCJD2IHH4DEKTFN5ZPMZXPECBQ2IX2QPSFUZCSFR5HKCDOVYGGQA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.785585301 (76h54m35.656668336s) [noise=true , address=185.167.204.205:4002 , piece_id=PDSJ5KDCFQBURILRTWU2WSEG3UH7GX4GK57W4J75QADBPKQXRSYA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.787925598 (76h54m35.659008643s) [noise=true , address=185.167.204.206:4047 , piece_id=THE6CUP46D2OSXS34UR3TTBUP6FIVMGDGT7DVJDBB57BHMWBCZ7A, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.787994289 (76h54m35.659077324s) [noise=true , address=185.167.204.205:4034 , piece_id=4VH6HEDD7CTPMEOHYGKFXSYXHKOLPYB2N6BCCJT2DXVAMADI3KJQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.788899402 (76h54m35.659982437s) [noise=true , address=185.167.204.206:4050 , piece_id=7B4GLDQZUPQPLGPZPPN5CO3VOX3LLBMLIYA6IMALYITCHE4CZBOA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.791033055 (76h54m35.66211609s) [noise=true , address=185.167.204.205:4031 , piece_id=5XPLPLO4SPMX7ZBYHLUZTH7MBGCRESEEMP2OEI7LTTT5BN67XHMQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.792163116 (76h54m35.663246161s) [noise=true , address=185.167.204.206:4055 , piece_id=JESF36BPW3YEPXVI2GP4OALECNLYSASJHQRZVLNXKV5BQRDGOWWQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.792693356 (76h54m35.663776392s) [noise=true , address=185.167.204.206:4063 , piece_id=BRUAUJTOCPWNUOB6D5G6AJCTMJBRZRJXXXBZBIUUZZXXQB5OEX2Q, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.793171606 (76h54m35.664254662s) [noise=true , address=185.167.204.206:4045 , piece_id=IXV6HTIOVQR4HCGJIWTBQFLO733BMMNNMZ24Y34H6V6KYUQYPQRA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.798151289 (76h54m35.669234344s) [noise=true , address=185.167.204.206:4053 , piece_id=VGK5AKCI23P5ZDS427NS6PNNRIJ3QUXLRZ4ECLYN4UDSL3KAQAKA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.798189562 (76h54m35.669272597s) [noise=true , address=185.167.204.206:4040 , piece_id=4KFICIELLLPCLWQ7RIB7UUSWUK7SFX2PJK7JWCRU5S4HQNHBEWRQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.798633607 (76h54m35.669716642s) [noise=true , address=185.167.204.205:4035 , piece_id=LXUP2T5SN5SSV7IWZLGTSLCRWLFRZBODXX7W4FFVHZFO74I2CQHQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.803059726 (76h54m35.674142761s) [noise=true , address=185.167.204.205:4024 , piece_id=2UHV7KDAKJJ4DDUU5MPBNLRP24OK5RQU6GGAVYUV5MEFLDDPGUZQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.344154907 (76h54m36.215237952s) [noise=true , address=81.171.14.70:4036 , piece_id=FB3SB3XQDNT2HSK5FDGELGYWODFLU655QW4D23GEXIWQEZYFNFRA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.359261713 (76h54m36.230344758s) [noise=true , address=81.171.14.70:4031 , piece_id=53QOUYYU4KS53TELJUM3FPEKOCB464OR5GKYGYB54RVRPD52RFWA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.393865115 (76h54m36.26494815s) [noise=true , address=81.171.14.70:4035 , piece_id=ZKLJ5VUPALKZBSPFCJQBZZUTMJ3PX2H3XMUZ5JVSW5BJBACRUYHA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.394302157 (76h54m36.265385202s) [noise=true , address=81.171.14.70:4044 , piece_id=QPYQGR62JDWWV3GM3CVDB6BLNSGNUTQUEHHQF74TYXFN25PMNPVQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.430200674 (76h54m36.301283709s) [noise=true , address=81.171.14.70:4046 , piece_id=ZHZUULNXPFDSTOXXH6T7XIEPN23OOWCNAAUWPCZYNNDTLPRWXVKQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.800799297 (76h54m36.671882332s) [noise=true , address=81.171.14.70:4040 , piece_id=W2SOTP5SPLQ6J3PM57NZOLRVZNMTNSGPWDZKZC2KV2PBPH4TUBLA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.862864259 (76h54m36.733947295s) [noise=true , address=81.171.14.70:4038 , piece_id=CKN2RF2YSTEOK6HUXGVBVECYAP2CLGUF6KYNR5JTHRUZPTGVI73Q, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.953769972 (76h54m36.824853007s) [noise=true , address=81.171.14.70:4043 , piece_id=FVUGWZXJLQAOLUSTRKJAULSHDWKX5GGWA2BB52YP7WQEKU4NKS5Q, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.996603376 (76h54m36.867686421s) [noise=true , address=89.149.201.147:4006 , piece_id=UGREZBZRVKXMGBOLQEV2NC4COTXOTTRUEAGSPNWRWP22MCPHQP7Q, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:37.037518916 (76h54m36.908601951s) [noise=true , address=89.149.201.147:4014 , piece_id=AEG6KGPGM26LWTYTWGZGQDPBL7UDX263DFBXY7WZRXGMYEFPWXQQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:37.037537541 (76h54m36.908620586s) [seg_pos={0 0}, upload=67229]: Segment reached optimal threshold of 26 pieces. Dec 8 20:23:37.037730880 (76h54m36.908813915s) [noise=true , address=89.149.201.147:4023 , piece_id=5EPGXWBK6GNTKLQLWEIWYBC2OJHFR3M7F4X3X4WODJWNXID3PCQQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BaeSxjXneBKoGKVrRM686sq386Bhn1YzzBFG9we8fHbcc9tbv): protocol: expected piece hash; context canceled; EOF Dec 8 20:23:37.037737452 (76h54m36.908820497s) [noise=true , address=89.149.201.147:4023 , piece_id=5EPGXWBK6GNTKLQLWEIWYBC2OJHFR3M7F4X3X4WODJWNXID3PCQQ, seg_pos={0 0}, upload=67229]: Long tail cancellation due to optimal threshold Dec 8 20:23:37.037893039 (76h54m36.908976074s) [noise=true , address=89.149.201.147:4022 , piece_id=2VR3E4SXAAOR5RAVHFVUQQ7MBZQEEQYFZ2ZW5Z7YSCVOETZNFQWQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122Vj8XxpS6RnhVweQV3cZWFhj8nrzy3hZ9P7JsuTzEaZcD6iFE): protocol: expected piece hash; context canceled; EOF Dec 8 20:23:37.037900964 (76h54m36.908984019s) [noise=true , address=89.149.201.147:4022 , piece_id=2VR3E4SXAAOR5RAVHFVUQQ7MBZQEEQYFZ2ZW5Z7YSCVOETZNFQWQ, seg_pos={0 0}, upload=67229]: Long tail cancellation due to optimal threshold Dec 8 20:23:37.037926843 (76h54m36.909009889s) [seg_pos={0 0}, upload=67229]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 8 20:23:37.038007697 (76h54m36.909090732s) [upload=67229]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 20:23:37.038013398 (76h54m36.909096433s) [upload=67229]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 20:26:49.742746371 (76h57m49.613829416s) [upload=67229]: Done uploading segments... Dec 8 20:26:49.742860908 (76h57m49.613943943s) [upload=67229]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.125779526s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [......X....X....X...........................................] duration: 5.684116583s trace: https://jaeger.dev.storj.io/trace/4600f1753cc382a2 FAIL (2025-12-08 20:53:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.684142472s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 4.871568329s trace: https://jaeger.dev.storj.io/trace/49fc2034659f6881 FAIL (2025-12-08 20:53:23 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.871613825s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [.....X...............X......................................] duration: 5.260544925s trace: https://jaeger.dev.storj.io/trace/3996915dd8f30735 FAIL (2025-12-08 20:53:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== PGM26LWTYTWGZGQDPBL7UDX263DFBXY7WZRXGMYEFPWXQQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.569231809 (76h54m34.440315054s) [noise=true , address=89.149.201.147:4022 , piece_id=2VR3E4SXAAOR5RAVHFVUQQ7MBZQEEQYFZ2ZW5Z7YSCVOETZNFQWQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.569369882 (76h54m34.440452917s) [noise=true , address=185.167.204.206:4037 , piece_id=XYY5NAATQJPHOK6JAMHXR3JH3QTLQF7OTA32BMST5HYQDYJ76UMA, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.569672879 (76h54m34.440755924s) [noise=true , address=185.167.204.206:4056 , piece_id=IPDW6RCHJFLMB5LOZM3FG645CCW6TT7KJW4PBQXBUSAIE2ZZM74A, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.569779642 (76h54m34.440862677s) [noise=true , address=185.167.204.205:4024 , piece_id=2UHV7KDAKJJ4DDUU5MPBNLRP24OK5RQU6GGAVYUV5MEFLDDPGUZQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.570073130 (76h54m34.441156176s) [noise=true , address=81.171.14.70:4035 , piece_id=ZKLJ5VUPALKZBSPFCJQBZZUTMJ3PX2H3XMUZ5JVSW5BJBACRUYHA, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.570428618 (76h54m34.441511653s) [noise=true , address=81.171.14.70:4043 , piece_id=FVUGWZXJLQAOLUSTRKJAULSHDWKX5GGWA2BB52YP7WQEKU4NKS5Q, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.570785237 (76h54m34.441868282s) [noise=true , address=81.171.14.70:4044 , piece_id=QPYQGR62JDWWV3GM3CVDB6BLNSGNUTQUEHHQF74TYXFN25PMNPVQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.571235034 (76h54m34.442318069s) [noise=true , address=185.167.204.205:4035 , piece_id=LXUP2T5SN5SSV7IWZLGTSLCRWLFRZBODXX7W4FFVHZFO74I2CQHQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.571655133 (76h54m34.442738169s) [noise=true , address=81.171.14.70:4031 , piece_id=53QOUYYU4KS53TELJUM3FPEKOCB464OR5GKYGYB54RVRPD52RFWA, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.572102666 (76h54m34.443185701s) [noise=true , address=185.167.204.205:4034 , piece_id=4VH6HEDD7CTPMEOHYGKFXSYXHKOLPYB2N6BCCJT2DXVAMADI3KJQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:35.395419290 (76h54m35.266502335s) [upload=67229]: Next returned nil segment Dec 8 20:23:35.395432614 (76h54m35.26651565s) [upload=67229]: Waiting for error group managing segments... Dec 8 20:23:35.780649683 (76h54m35.651732718s) [noise=true , address=185.167.204.206:4037 , piece_id=XYY5NAATQJPHOK6JAMHXR3JH3QTLQF7OTA32BMST5HYQDYJ76UMA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.781236982 (76h54m35.652320027s) [noise=true , address=185.167.204.206:4056 , piece_id=IPDW6RCHJFLMB5LOZM3FG645CCW6TT7KJW4PBQXBUSAIE2ZZM74A, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.784267011 (76h54m35.655350046s) [noise=true , address=185.167.204.205:4030 , piece_id=O3QKGK2G5I5YYPRP5K57RTG4H35H2UDFROFU2JMSHKTFNZJDVUIA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.785164540 (76h54m35.656247585s) [noise=true , address=185.167.204.205:4012 , piece_id=QCJD2IHH4DEKTFN5ZPMZXPECBQ2IX2QPSFUZCSFR5HKCDOVYGGQA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.785585301 (76h54m35.656668336s) [noise=true , address=185.167.204.205:4002 , piece_id=PDSJ5KDCFQBURILRTWU2WSEG3UH7GX4GK57W4J75QADBPKQXRSYA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.787925598 (76h54m35.659008643s) [noise=true , address=185.167.204.206:4047 , piece_id=THE6CUP46D2OSXS34UR3TTBUP6FIVMGDGT7DVJDBB57BHMWBCZ7A, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.787994289 (76h54m35.659077324s) [noise=true , address=185.167.204.205:4034 , piece_id=4VH6HEDD7CTPMEOHYGKFXSYXHKOLPYB2N6BCCJT2DXVAMADI3KJQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.788899402 (76h54m35.659982437s) [noise=true , address=185.167.204.206:4050 , piece_id=7B4GLDQZUPQPLGPZPPN5CO3VOX3LLBMLIYA6IMALYITCHE4CZBOA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.791033055 (76h54m35.66211609s) [noise=true , address=185.167.204.205:4031 , piece_id=5XPLPLO4SPMX7ZBYHLUZTH7MBGCRESEEMP2OEI7LTTT5BN67XHMQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.792163116 (76h54m35.663246161s) [noise=true , address=185.167.204.206:4055 , piece_id=JESF36BPW3YEPXVI2GP4OALECNLYSASJHQRZVLNXKV5BQRDGOWWQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.792693356 (76h54m35.663776392s) [noise=true , address=185.167.204.206:4063 , piece_id=BRUAUJTOCPWNUOB6D5G6AJCTMJBRZRJXXXBZBIUUZZXXQB5OEX2Q, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.793171606 (76h54m35.664254662s) [noise=true , address=185.167.204.206:4045 , piece_id=IXV6HTIOVQR4HCGJIWTBQFLO733BMMNNMZ24Y34H6V6KYUQYPQRA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.798151289 (76h54m35.669234344s) [noise=true , address=185.167.204.206:4053 , piece_id=VGK5AKCI23P5ZDS427NS6PNNRIJ3QUXLRZ4ECLYN4UDSL3KAQAKA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.798189562 (76h54m35.669272597s) [noise=true , address=185.167.204.206:4040 , piece_id=4KFICIELLLPCLWQ7RIB7UUSWUK7SFX2PJK7JWCRU5S4HQNHBEWRQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.798633607 (76h54m35.669716642s) [noise=true , address=185.167.204.205:4035 , piece_id=LXUP2T5SN5SSV7IWZLGTSLCRWLFRZBODXX7W4FFVHZFO74I2CQHQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.803059726 (76h54m35.674142761s) [noise=true , address=185.167.204.205:4024 , piece_id=2UHV7KDAKJJ4DDUU5MPBNLRP24OK5RQU6GGAVYUV5MEFLDDPGUZQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.344154907 (76h54m36.215237952s) [noise=true , address=81.171.14.70:4036 , piece_id=FB3SB3XQDNT2HSK5FDGELGYWODFLU655QW4D23GEXIWQEZYFNFRA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.359261713 (76h54m36.230344758s) [noise=true , address=81.171.14.70:4031 , piece_id=53QOUYYU4KS53TELJUM3FPEKOCB464OR5GKYGYB54RVRPD52RFWA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.393865115 (76h54m36.26494815s) [noise=true , address=81.171.14.70:4035 , piece_id=ZKLJ5VUPALKZBSPFCJQBZZUTMJ3PX2H3XMUZ5JVSW5BJBACRUYHA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.394302157 (76h54m36.265385202s) [noise=true , address=81.171.14.70:4044 , piece_id=QPYQGR62JDWWV3GM3CVDB6BLNSGNUTQUEHHQF74TYXFN25PMNPVQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.430200674 (76h54m36.301283709s) [noise=true , address=81.171.14.70:4046 , piece_id=ZHZUULNXPFDSTOXXH6T7XIEPN23OOWCNAAUWPCZYNNDTLPRWXVKQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.800799297 (76h54m36.671882332s) [noise=true , address=81.171.14.70:4040 , piece_id=W2SOTP5SPLQ6J3PM57NZOLRVZNMTNSGPWDZKZC2KV2PBPH4TUBLA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.862864259 (76h54m36.733947295s) [noise=true , address=81.171.14.70:4038 , piece_id=CKN2RF2YSTEOK6HUXGVBVECYAP2CLGUF6KYNR5JTHRUZPTGVI73Q, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.953769972 (76h54m36.824853007s) [noise=true , address=81.171.14.70:4043 , piece_id=FVUGWZXJLQAOLUSTRKJAULSHDWKX5GGWA2BB52YP7WQEKU4NKS5Q, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.996603376 (76h54m36.867686421s) [noise=true , address=89.149.201.147:4006 , piece_id=UGREZBZRVKXMGBOLQEV2NC4COTXOTTRUEAGSPNWRWP22MCPHQP7Q, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:37.037518916 (76h54m36.908601951s) [noise=true , address=89.149.201.147:4014 , piece_id=AEG6KGPGM26LWTYTWGZGQDPBL7UDX263DFBXY7WZRXGMYEFPWXQQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:37.037537541 (76h54m36.908620586s) [seg_pos={0 0}, upload=67229]: Segment reached optimal threshold of 26 pieces. Dec 8 20:23:37.037730880 (76h54m36.908813915s) [noise=true , address=89.149.201.147:4023 , piece_id=5EPGXWBK6GNTKLQLWEIWYBC2OJHFR3M7F4X3X4WODJWNXID3PCQQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BaeSxjXneBKoGKVrRM686sq386Bhn1YzzBFG9we8fHbcc9tbv): protocol: expected piece hash; context canceled; EOF Dec 8 20:23:37.037737452 (76h54m36.908820497s) [noise=true , address=89.149.201.147:4023 , piece_id=5EPGXWBK6GNTKLQLWEIWYBC2OJHFR3M7F4X3X4WODJWNXID3PCQQ, seg_pos={0 0}, upload=67229]: Long tail cancellation due to optimal threshold Dec 8 20:23:37.037893039 (76h54m36.908976074s) [noise=true , address=89.149.201.147:4022 , piece_id=2VR3E4SXAAOR5RAVHFVUQQ7MBZQEEQYFZ2ZW5Z7YSCVOETZNFQWQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122Vj8XxpS6RnhVweQV3cZWFhj8nrzy3hZ9P7JsuTzEaZcD6iFE): protocol: expected piece hash; context canceled; EOF Dec 8 20:23:37.037900964 (76h54m36.908984019s) [noise=true , address=89.149.201.147:4022 , piece_id=2VR3E4SXAAOR5RAVHFVUQQ7MBZQEEQYFZ2ZW5Z7YSCVOETZNFQWQ, seg_pos={0 0}, upload=67229]: Long tail cancellation due to optimal threshold Dec 8 20:23:37.037926843 (76h54m36.909009889s) [seg_pos={0 0}, upload=67229]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 8 20:23:37.038007697 (76h54m36.909090732s) [upload=67229]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 20:23:37.038013398 (76h54m36.909096433s) [upload=67229]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 20:26:49.742746371 (76h57m49.613829416s) [upload=67229]: Done uploading segments... Dec 8 20:26:49.742860908 (76h57m49.613943943s) [upload=67229]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.26057389s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.......X....X....X..........................................] duration: 5.774622305s trace: https://jaeger.dev.storj.io/trace/4600f1753cc382a2 FAIL (2025-12-08 20:53:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.774709591s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................................................X.] duration: 3.51882408s trace: https://jaeger.dev.storj.io/trace/65b6db6cde985dc2 FAIL (2025-12-08 20:53:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.518854859s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............X................................X..............] duration: 5.852526459s trace: https://jaeger.dev.storj.io/trace/2eed98fdb4249ce9 FAIL (2025-12-08 20:53:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 20:43:00.763765824 (77h14m0.634848859s) [upload=67501]: Starting upload Dec 8 20:43:00.763879119 (77h14m0.634962164s) [upload=67501]: Done starting upload Dec 8 20:43:00.763936168 (77h14m0.635019203s) [upload=67501]: Uploading segments... Dec 8 20:43:00.764117523 (77h14m0.635200568s) [upload=67501]: Got next segment. Inline: false Dec 8 20:43:00.764157569 (77h14m0.635240604s) [upload=67501]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 20:43:00.764162089 (77h14m0.635245134s) [upload=67501]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 20:43:01.015312549 (77h14m0.886395594s) [seg_pos={0 0}, upload=67501]: Begin upload segment... Dec 8 20:43:01.015378243 (77h14m0.886461288s) [seg_pos={0 0}, upload=67501]: Done begin upload segment. Dec 8 20:43:01.015411717 (77h14m0.886494762s) [noise=true , address=145.79.227.40:4050 , piece_id=K2RW6IQERLFKEUD2ZTAZEX767U77PQWEXMXEH2SYDBQ7BT26WAHQ, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.018044751 (77h14m0.889127796s) [noise=true , address=145.79.227.33:4870 , piece_id=KY2FLO3KDFPWRXPUUFINFLIRLZDMHPUCSPKRCN7RYPVAXXUYMXWQ, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.019100842 (77h14m0.890183887s) [noise=true , address=145.79.227.31:4797 , piece_id=ECY3CM3GYNMM5RIEDHYI5NVWDCKEXRRKH4NCK3FV5ILW7W54LIIA, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.020021405 (77h14m0.89110444s) [noise=true , address=145.79.227.41:4087 , piece_id=QYRY35JJKPQEK4BAVFZ6DEBKJN7SBUHG66S44ZA7XRNOVKAVQ33Q, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.020812742 (77h14m0.891895787s) [noise=true , address=145.79.227.20:4437 , piece_id=CCMQ7GX5NBRAUKHEHERGP5SHETIGELARKU5AE5ETKZWQRJP7OOAQ, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.021487076 (77h14m0.892570111s) [noise=true , address=77.90.31.21:4879 , piece_id=WF6YF2HQIXM26JI5TQ3YW7OXHNYZH3E2VTHAQVWIUHZ3ME3CCG6Q, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.022677542 (77h14m0.893760588s) [noise=true , address=145.79.227.42:4123 , piece_id=NBMNEIGLWZHFGHY674WDGEDH4CGDCKYERXLB7UKXXI4MTZJ2HVEQ, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.022691720 (77h14m0.893774755s) [noise=true , address=77.90.31.18:4807 , piece_id=KDIWMU2XEJ7NTRCSG5J37ZUWIFURSJVK2BSDM7VJFKOB3ROANRGA, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.023821471 (77h14m0.894904506s) [noise=true , address=145.79.227.41:4085 , piece_id=LS54ZFJC7DEGOKQRUSZOY3I665N43TNXICDJTXQ3ZJ62OHPGETSQ, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.024740981 (77h14m0.895824026s) [noise=true , address=77.90.31.12:4589 , piece_id=WUVA4G7EEN4J4HF47KQ7FZKFINAN2CCYDXMRNHDZVBF6EAPVFS4Q, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.025197331 (77h14m0.896280366s) [noise=true , address=145.79.227.34:4906 , piece_id=VPDAFAMOLFSYUGFZIHRTADXWLYAE7IXILSPLT6QL2IPCOVUDPA3A, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.026609700 (77h14m0.897692745s) [noise=true , address=145.79.227.25:4618 , piece_id=AE7LMOAKNNIKVS4STMQJKZER7YUSFSUTSE7P5KXYTG4AQLFNQHLQ, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.027084214 (77h14m0.898167259s) [noise=true , address=145.79.227.44:4195 , piece_id=EYY7B3AXUMA63GQLGKQWPWGDZUBORTXR6QFYSEOB26ATZK2GJC6Q, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.028812575 (77h14m0.89989562s) [noise=true , address=145.79.227.38:4977 , piece_id=JW72CEO3WDGEB6BSSAR3M4Z5ANDBH63BLJZRSYAJDK7F7EXCEOZQ, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.030193064 (77h14m0.90127611s) [noise=true , address=145.79.227.24:4583 , piece_id=DRLVD3TA2Z6TJQ4WKA67KDTXK7Q32YJH2ES5QEQG7Z37DYBJBB6Q, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.036513970 (77h14m0.907597045s) [noise=true , address=77.90.31.13:4627 , piece_id=XMI4HVR2MQQ5KHN4GQOUCKFYJYJ2YZEGQGTAAXYKGS3OI6XPJLVA, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:02.093045104 (77h14m1.964128159s) [noise=true , address=77.90.31.21:4879 , piece_id=WF6YF2HQIXM26JI5TQ3YW7OXHNYZH3E2VTHAQVWIUHZ3ME3CCG6Q, seg_pos={0 0}, upload=67501]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: storage node overloaded, request limit: 2000 Dec 8 20:43:02.127847935 (77h14m1.99893098s) [noise=true , address=77.90.31.13:4627 , piece_id=XMI4HVR2MQQ5KHN4GQOUCKFYJYJ2YZEGQGTAAXYKGS3OI6XPJLVA, seg_pos={0 0}, upload=67501]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 8 20:43:02.156386568 (77h14m2.027469613s) [noise=true , address=77.90.31.12:4589 , piece_id=WUVA4G7EEN4J4HF47KQ7FZKFINAN2CCYDXMRNHDZVBF6EAPVFS4Q, seg_pos={0 0}, upload=67501]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 20:43:02.719927441 (77h14m2.591010486s) [upload=67501]: Next returned nil segment Dec 8 20:43:02.719942970 (77h14m2.591026005s) [upload=67501]: Waiting for error group managing segments... Dec 8 20:43:03.005012008 (77h14m2.876095063s) [noise=true , address=145.79.227.42:4123 , piece_id=NBMNEIGLWZHFGHY674WDGEDH4CGDCKYERXLB7UKXXI4MTZJ2HVEQ, seg_pos={0 0}, upload=67501]: Done uploading piece. err: <nil> Dec 8 20:43:03.012525866 (77h14m2.883608911s) [noise=true , address=145.79.227.41:4085 , piece_id=LS54ZFJC7DEGOKQRUSZOY3I665N43TNXICDJTXQ3ZJ62OHPGETSQ, seg_pos={0 0}, upload=67501]: Done uploading piece. err: <nil> Dec 8 20:43:03.015459803 (77h14m2.886542848s) [noise=true , address=145.79.227.41:4087 , piece_id=QYRY35JJKPQEK4BAVFZ6DEBKJN7SBUHG66S44ZA7XRNOVKAVQ33Q, seg_pos={0 0}, upload=67501]: Done uploading piece. err: <nil> Dec 8 20:43:03.020621582 (77h14m2.891704617s) [noise=true , address=145.79.227.20:4437 , piece_id=CCMQ7GX5NBRAUKHEHERGP5SHETIGELARKU5AE5ETKZWQRJP7OOAQ, seg_pos={0 0}, upload=67501]: Done uploading piece. err: <nil> Dec 8 20:43:03.020672098 (77h14m2.891755153s) [noise=true , address=145.79.227.31:4797 , piece_id=ECY3CM3GYNMM5RIEDHYI5NVWDCKEXRRKH4NCK3FV5ILW7W54LIIA, seg_pos={0 0}, upload=67501]: Done uploading piece. err: <nil> Dec 8 20:43:03.028139237 (77h14m2.899222282s) [noise=true , address=145.79.227.24:4583 , piece_id=DRLVD3TA2Z6TJQ4WKA67KDTXK7Q32YJH2ES5QEQG7Z37DYBJBB6Q, seg_pos={0 0}, upload=67501]: Done uploading piece. err: <nil> Dec 8 20:43:03.031290167 (77h14m2.902373202s) [noise=true , address=145.79.227.33:4870 , piece_id=KY2FLO3KDFPWRXPUUFINFLIRLZDMHPUCSPKRCN7RYPVAXXUYMXWQ, seg_pos={0 0}, upload=67501]: Done uploading piece. err: <nil> Dec 8 20:43:03.033756633 (77h14m2.904839668s) [noise=true , address=145.79.227.25:4618 , piece_id=AE7LMOAKNNIKVS4STMQJKZER7YUSFSUTSE7P5KXYTG4AQLFNQHLQ, seg_pos={0 0}, upload=67501]: Done uploading piece. err: <nil> Dec 8 20:43:03.039153600 (77h14m2.910236655s) [noise=true , address=145.79.227.40:4050 , piece_id=K2RW6IQERLFKEUD2ZTAZEX767U77PQWEXMXEH2SYDBQ7BT26WAHQ, seg_pos={0 0}, upload=67501]: Done uploading piece. err: <nil> Dec 8 20:43:03.040008869 (77h14m2.911091914s) [noise=true , address=145.79.227.38:4977 , piece_id=JW72CEO3WDGEB6BSSAR3M4Z5ANDBH63BLJZRSYAJDK7F7EXCEOZQ, seg_pos={0 0}, upload=67501]: Done uploading piece. err: <nil> Dec 8 20:43:03.178911548 (77h14m3.049994593s) [noise=true , address=77.90.31.18:4807 , piece_id=KDIWMU2XEJ7NTRCSG5J37ZUWIFURSJVK2BSDM7VJFKOB3ROANRGA, seg_pos={0 0}, upload=67501]: Done uploading piece. err: <nil> Dec 8 20:43:04.409564163 (77h14m4.280647208s) [noise=true , address=145.79.227.44:4195 , piece_id=EYY7B3AXUMA63GQLGKQWPWGDZUBORTXR6QFYSEOB26ATZK2GJC6Q, seg_pos={0 0}, upload=67501]: Done uploading piece. err: <nil> Dec 8 20:43:30.067753910 (77h14m29.938836955s) [noise=true , address=145.79.227.34:4906 , piece_id=VPDAFAMOLFSYUGFZIHRTADXWLYAE7IXILSPLT6QL2IPCOVUDPA3A, seg_pos={0 0}, upload=67501]: Done uploading piece. err: ecclient: upload failed (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw, address:145.79.227.34:4906): protocol: read tcp 188.166.225.170:36576->145.79.227.34:4906: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 20:43:30.067797914 (77h14m29.938880959s) [seg_pos={0 0}, upload=67501]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 20:43:30.323605733 (77h14m30.194688778s) [upload=67501]: Done uploading segments... Dec 8 20:43:30.323629097 (77h14m30.194712142s) [upload=67501]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.852551535s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 4.942721388s trace: https://jaeger.dev.storj.io/trace/49fc2034659f6881 FAIL (2025-12-08 20:52:55 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.942757706s core uplink eu1
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 2 out of 60 is failed (threshold=3) [..X..................................................X......] duration: 475.180583ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/51c3a25722f21601 FAIL (2025-12-08 20:52:46 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 475.204068ms core satellite eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [......X...............X.....................................] duration: 4.670770551s trace: https://jaeger.dev.storj.io/trace/3996915dd8f30735 FAIL (2025-12-08 20:52:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== PGM26LWTYTWGZGQDPBL7UDX263DFBXY7WZRXGMYEFPWXQQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.569231809 (76h54m34.440315054s) [noise=true , address=89.149.201.147:4022 , piece_id=2VR3E4SXAAOR5RAVHFVUQQ7MBZQEEQYFZ2ZW5Z7YSCVOETZNFQWQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.569369882 (76h54m34.440452917s) [noise=true , address=185.167.204.206:4037 , piece_id=XYY5NAATQJPHOK6JAMHXR3JH3QTLQF7OTA32BMST5HYQDYJ76UMA, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.569672879 (76h54m34.440755924s) [noise=true , address=185.167.204.206:4056 , piece_id=IPDW6RCHJFLMB5LOZM3FG645CCW6TT7KJW4PBQXBUSAIE2ZZM74A, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.569779642 (76h54m34.440862677s) [noise=true , address=185.167.204.205:4024 , piece_id=2UHV7KDAKJJ4DDUU5MPBNLRP24OK5RQU6GGAVYUV5MEFLDDPGUZQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.570073130 (76h54m34.441156176s) [noise=true , address=81.171.14.70:4035 , piece_id=ZKLJ5VUPALKZBSPFCJQBZZUTMJ3PX2H3XMUZ5JVSW5BJBACRUYHA, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.570428618 (76h54m34.441511653s) [noise=true , address=81.171.14.70:4043 , piece_id=FVUGWZXJLQAOLUSTRKJAULSHDWKX5GGWA2BB52YP7WQEKU4NKS5Q, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.570785237 (76h54m34.441868282s) [noise=true , address=81.171.14.70:4044 , piece_id=QPYQGR62JDWWV3GM3CVDB6BLNSGNUTQUEHHQF74TYXFN25PMNPVQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.571235034 (76h54m34.442318069s) [noise=true , address=185.167.204.205:4035 , piece_id=LXUP2T5SN5SSV7IWZLGTSLCRWLFRZBODXX7W4FFVHZFO74I2CQHQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.571655133 (76h54m34.442738169s) [noise=true , address=81.171.14.70:4031 , piece_id=53QOUYYU4KS53TELJUM3FPEKOCB464OR5GKYGYB54RVRPD52RFWA, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.572102666 (76h54m34.443185701s) [noise=true , address=185.167.204.205:4034 , piece_id=4VH6HEDD7CTPMEOHYGKFXSYXHKOLPYB2N6BCCJT2DXVAMADI3KJQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:35.395419290 (76h54m35.266502335s) [upload=67229]: Next returned nil segment Dec 8 20:23:35.395432614 (76h54m35.26651565s) [upload=67229]: Waiting for error group managing segments... Dec 8 20:23:35.780649683 (76h54m35.651732718s) [noise=true , address=185.167.204.206:4037 , piece_id=XYY5NAATQJPHOK6JAMHXR3JH3QTLQF7OTA32BMST5HYQDYJ76UMA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.781236982 (76h54m35.652320027s) [noise=true , address=185.167.204.206:4056 , piece_id=IPDW6RCHJFLMB5LOZM3FG645CCW6TT7KJW4PBQXBUSAIE2ZZM74A, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.784267011 (76h54m35.655350046s) [noise=true , address=185.167.204.205:4030 , piece_id=O3QKGK2G5I5YYPRP5K57RTG4H35H2UDFROFU2JMSHKTFNZJDVUIA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.785164540 (76h54m35.656247585s) [noise=true , address=185.167.204.205:4012 , piece_id=QCJD2IHH4DEKTFN5ZPMZXPECBQ2IX2QPSFUZCSFR5HKCDOVYGGQA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.785585301 (76h54m35.656668336s) [noise=true , address=185.167.204.205:4002 , piece_id=PDSJ5KDCFQBURILRTWU2WSEG3UH7GX4GK57W4J75QADBPKQXRSYA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.787925598 (76h54m35.659008643s) [noise=true , address=185.167.204.206:4047 , piece_id=THE6CUP46D2OSXS34UR3TTBUP6FIVMGDGT7DVJDBB57BHMWBCZ7A, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.787994289 (76h54m35.659077324s) [noise=true , address=185.167.204.205:4034 , piece_id=4VH6HEDD7CTPMEOHYGKFXSYXHKOLPYB2N6BCCJT2DXVAMADI3KJQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.788899402 (76h54m35.659982437s) [noise=true , address=185.167.204.206:4050 , piece_id=7B4GLDQZUPQPLGPZPPN5CO3VOX3LLBMLIYA6IMALYITCHE4CZBOA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.791033055 (76h54m35.66211609s) [noise=true , address=185.167.204.205:4031 , piece_id=5XPLPLO4SPMX7ZBYHLUZTH7MBGCRESEEMP2OEI7LTTT5BN67XHMQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.792163116 (76h54m35.663246161s) [noise=true , address=185.167.204.206:4055 , piece_id=JESF36BPW3YEPXVI2GP4OALECNLYSASJHQRZVLNXKV5BQRDGOWWQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.792693356 (76h54m35.663776392s) [noise=true , address=185.167.204.206:4063 , piece_id=BRUAUJTOCPWNUOB6D5G6AJCTMJBRZRJXXXBZBIUUZZXXQB5OEX2Q, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.793171606 (76h54m35.664254662s) [noise=true , address=185.167.204.206:4045 , piece_id=IXV6HTIOVQR4HCGJIWTBQFLO733BMMNNMZ24Y34H6V6KYUQYPQRA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.798151289 (76h54m35.669234344s) [noise=true , address=185.167.204.206:4053 , piece_id=VGK5AKCI23P5ZDS427NS6PNNRIJ3QUXLRZ4ECLYN4UDSL3KAQAKA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.798189562 (76h54m35.669272597s) [noise=true , address=185.167.204.206:4040 , piece_id=4KFICIELLLPCLWQ7RIB7UUSWUK7SFX2PJK7JWCRU5S4HQNHBEWRQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.798633607 (76h54m35.669716642s) [noise=true , address=185.167.204.205:4035 , piece_id=LXUP2T5SN5SSV7IWZLGTSLCRWLFRZBODXX7W4FFVHZFO74I2CQHQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.803059726 (76h54m35.674142761s) [noise=true , address=185.167.204.205:4024 , piece_id=2UHV7KDAKJJ4DDUU5MPBNLRP24OK5RQU6GGAVYUV5MEFLDDPGUZQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.344154907 (76h54m36.215237952s) [noise=true , address=81.171.14.70:4036 , piece_id=FB3SB3XQDNT2HSK5FDGELGYWODFLU655QW4D23GEXIWQEZYFNFRA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.359261713 (76h54m36.230344758s) [noise=true , address=81.171.14.70:4031 , piece_id=53QOUYYU4KS53TELJUM3FPEKOCB464OR5GKYGYB54RVRPD52RFWA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.393865115 (76h54m36.26494815s) [noise=true , address=81.171.14.70:4035 , piece_id=ZKLJ5VUPALKZBSPFCJQBZZUTMJ3PX2H3XMUZ5JVSW5BJBACRUYHA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.394302157 (76h54m36.265385202s) [noise=true , address=81.171.14.70:4044 , piece_id=QPYQGR62JDWWV3GM3CVDB6BLNSGNUTQUEHHQF74TYXFN25PMNPVQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.430200674 (76h54m36.301283709s) [noise=true , address=81.171.14.70:4046 , piece_id=ZHZUULNXPFDSTOXXH6T7XIEPN23OOWCNAAUWPCZYNNDTLPRWXVKQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.800799297 (76h54m36.671882332s) [noise=true , address=81.171.14.70:4040 , piece_id=W2SOTP5SPLQ6J3PM57NZOLRVZNMTNSGPWDZKZC2KV2PBPH4TUBLA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.862864259 (76h54m36.733947295s) [noise=true , address=81.171.14.70:4038 , piece_id=CKN2RF2YSTEOK6HUXGVBVECYAP2CLGUF6KYNR5JTHRUZPTGVI73Q, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.953769972 (76h54m36.824853007s) [noise=true , address=81.171.14.70:4043 , piece_id=FVUGWZXJLQAOLUSTRKJAULSHDWKX5GGWA2BB52YP7WQEKU4NKS5Q, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.996603376 (76h54m36.867686421s) [noise=true , address=89.149.201.147:4006 , piece_id=UGREZBZRVKXMGBOLQEV2NC4COTXOTTRUEAGSPNWRWP22MCPHQP7Q, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:37.037518916 (76h54m36.908601951s) [noise=true , address=89.149.201.147:4014 , piece_id=AEG6KGPGM26LWTYTWGZGQDPBL7UDX263DFBXY7WZRXGMYEFPWXQQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:37.037537541 (76h54m36.908620586s) [seg_pos={0 0}, upload=67229]: Segment reached optimal threshold of 26 pieces. Dec 8 20:23:37.037730880 (76h54m36.908813915s) [noise=true , address=89.149.201.147:4023 , piece_id=5EPGXWBK6GNTKLQLWEIWYBC2OJHFR3M7F4X3X4WODJWNXID3PCQQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BaeSxjXneBKoGKVrRM686sq386Bhn1YzzBFG9we8fHbcc9tbv): protocol: expected piece hash; context canceled; EOF Dec 8 20:23:37.037737452 (76h54m36.908820497s) [noise=true , address=89.149.201.147:4023 , piece_id=5EPGXWBK6GNTKLQLWEIWYBC2OJHFR3M7F4X3X4WODJWNXID3PCQQ, seg_pos={0 0}, upload=67229]: Long tail cancellation due to optimal threshold Dec 8 20:23:37.037893039 (76h54m36.908976074s) [noise=true , address=89.149.201.147:4022 , piece_id=2VR3E4SXAAOR5RAVHFVUQQ7MBZQEEQYFZ2ZW5Z7YSCVOETZNFQWQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122Vj8XxpS6RnhVweQV3cZWFhj8nrzy3hZ9P7JsuTzEaZcD6iFE): protocol: expected piece hash; context canceled; EOF Dec 8 20:23:37.037900964 (76h54m36.908984019s) [noise=true , address=89.149.201.147:4022 , piece_id=2VR3E4SXAAOR5RAVHFVUQQ7MBZQEEQYFZ2ZW5Z7YSCVOETZNFQWQ, seg_pos={0 0}, upload=67229]: Long tail cancellation due to optimal threshold Dec 8 20:23:37.037926843 (76h54m36.909009889s) [seg_pos={0 0}, upload=67229]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 8 20:23:37.038007697 (76h54m36.909090732s) [upload=67229]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 20:23:37.038013398 (76h54m36.909096433s) [upload=67229]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 20:26:49.742746371 (76h57m49.613829416s) [upload=67229]: Done uploading segments... Dec 8 20:26:49.742860908 (76h57m49.613943943s) [upload=67229]: Upload finished. err: metaclient: context deadline exceeded Duration: 4.670796631s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 4.957169082s trace: https://jaeger.dev.storj.io/trace/49fc2034659f6881 FAIL (2025-12-08 20:52:24 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.957197955s core uplink eu1
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 2 out of 60 is failed (threshold=3) [...X..................................................X.....] duration: 467.497204ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/51c3a25722f21601 FAIL (2025-12-08 20:52:21 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 467.521088ms core satellite eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [........X....X....X.........................................] duration: 5.705035679s trace: https://jaeger.dev.storj.io/trace/4600f1753cc382a2 FAIL (2025-12-08 20:52:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.705063232s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............X................................X.............] duration: 5.742426556s trace: https://jaeger.dev.storj.io/trace/2eed98fdb4249ce9 FAIL (2025-12-08 20:52:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 20:43:00.763765824 (77h14m0.634848859s) [upload=67501]: Starting upload Dec 8 20:43:00.763879119 (77h14m0.634962164s) [upload=67501]: Done starting upload Dec 8 20:43:00.763936168 (77h14m0.635019203s) [upload=67501]: Uploading segments... Dec 8 20:43:00.764117523 (77h14m0.635200568s) [upload=67501]: Got next segment. Inline: false Dec 8 20:43:00.764157569 (77h14m0.635240604s) [upload=67501]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 20:43:00.764162089 (77h14m0.635245134s) [upload=67501]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 20:43:01.015312549 (77h14m0.886395594s) [seg_pos={0 0}, upload=67501]: Begin upload segment... Dec 8 20:43:01.015378243 (77h14m0.886461288s) [seg_pos={0 0}, upload=67501]: Done begin upload segment. Dec 8 20:43:01.015411717 (77h14m0.886494762s) [noise=true , address=145.79.227.40:4050 , piece_id=K2RW6IQERLFKEUD2ZTAZEX767U77PQWEXMXEH2SYDBQ7BT26WAHQ, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.018044751 (77h14m0.889127796s) [noise=true , address=145.79.227.33:4870 , piece_id=KY2FLO3KDFPWRXPUUFINFLIRLZDMHPUCSPKRCN7RYPVAXXUYMXWQ, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.019100842 (77h14m0.890183887s) [noise=true , address=145.79.227.31:4797 , piece_id=ECY3CM3GYNMM5RIEDHYI5NVWDCKEXRRKH4NCK3FV5ILW7W54LIIA, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.020021405 (77h14m0.89110444s) [noise=true , address=145.79.227.41:4087 , piece_id=QYRY35JJKPQEK4BAVFZ6DEBKJN7SBUHG66S44ZA7XRNOVKAVQ33Q, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.020812742 (77h14m0.891895787s) [noise=true , address=145.79.227.20:4437 , piece_id=CCMQ7GX5NBRAUKHEHERGP5SHETIGELARKU5AE5ETKZWQRJP7OOAQ, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.021487076 (77h14m0.892570111s) [noise=true , address=77.90.31.21:4879 , piece_id=WF6YF2HQIXM26JI5TQ3YW7OXHNYZH3E2VTHAQVWIUHZ3ME3CCG6Q, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.022677542 (77h14m0.893760588s) [noise=true , address=145.79.227.42:4123 , piece_id=NBMNEIGLWZHFGHY674WDGEDH4CGDCKYERXLB7UKXXI4MTZJ2HVEQ, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.022691720 (77h14m0.893774755s) [noise=true , address=77.90.31.18:4807 , piece_id=KDIWMU2XEJ7NTRCSG5J37ZUWIFURSJVK2BSDM7VJFKOB3ROANRGA, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.023821471 (77h14m0.894904506s) [noise=true , address=145.79.227.41:4085 , piece_id=LS54ZFJC7DEGOKQRUSZOY3I665N43TNXICDJTXQ3ZJ62OHPGETSQ, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.024740981 (77h14m0.895824026s) [noise=true , address=77.90.31.12:4589 , piece_id=WUVA4G7EEN4J4HF47KQ7FZKFINAN2CCYDXMRNHDZVBF6EAPVFS4Q, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.025197331 (77h14m0.896280366s) [noise=true , address=145.79.227.34:4906 , piece_id=VPDAFAMOLFSYUGFZIHRTADXWLYAE7IXILSPLT6QL2IPCOVUDPA3A, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.026609700 (77h14m0.897692745s) [noise=true , address=145.79.227.25:4618 , piece_id=AE7LMOAKNNIKVS4STMQJKZER7YUSFSUTSE7P5KXYTG4AQLFNQHLQ, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.027084214 (77h14m0.898167259s) [noise=true , address=145.79.227.44:4195 , piece_id=EYY7B3AXUMA63GQLGKQWPWGDZUBORTXR6QFYSEOB26ATZK2GJC6Q, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.028812575 (77h14m0.89989562s) [noise=true , address=145.79.227.38:4977 , piece_id=JW72CEO3WDGEB6BSSAR3M4Z5ANDBH63BLJZRSYAJDK7F7EXCEOZQ, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.030193064 (77h14m0.90127611s) [noise=true , address=145.79.227.24:4583 , piece_id=DRLVD3TA2Z6TJQ4WKA67KDTXK7Q32YJH2ES5QEQG7Z37DYBJBB6Q, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.036513970 (77h14m0.907597045s) [noise=true , address=77.90.31.13:4627 , piece_id=XMI4HVR2MQQ5KHN4GQOUCKFYJYJ2YZEGQGTAAXYKGS3OI6XPJLVA, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:02.093045104 (77h14m1.964128159s) [noise=true , address=77.90.31.21:4879 , piece_id=WF6YF2HQIXM26JI5TQ3YW7OXHNYZH3E2VTHAQVWIUHZ3ME3CCG6Q, seg_pos={0 0}, upload=67501]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: storage node overloaded, request limit: 2000 Dec 8 20:43:02.127847935 (77h14m1.99893098s) [noise=true , address=77.90.31.13:4627 , piece_id=XMI4HVR2MQQ5KHN4GQOUCKFYJYJ2YZEGQGTAAXYKGS3OI6XPJLVA, seg_pos={0 0}, upload=67501]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 8 20:43:02.156386568 (77h14m2.027469613s) [noise=true , address=77.90.31.12:4589 , piece_id=WUVA4G7EEN4J4HF47KQ7FZKFINAN2CCYDXMRNHDZVBF6EAPVFS4Q, seg_pos={0 0}, upload=67501]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 20:43:02.719927441 (77h14m2.591010486s) [upload=67501]: Next returned nil segment Dec 8 20:43:02.719942970 (77h14m2.591026005s) [upload=67501]: Waiting for error group managing segments... Dec 8 20:43:03.005012008 (77h14m2.876095063s) [noise=true , address=145.79.227.42:4123 , piece_id=NBMNEIGLWZHFGHY674WDGEDH4CGDCKYERXLB7UKXXI4MTZJ2HVEQ, seg_pos={0 0}, upload=67501]: Done uploading piece. err: <nil> Dec 8 20:43:03.012525866 (77h14m2.883608911s) [noise=true , address=145.79.227.41:4085 , piece_id=LS54ZFJC7DEGOKQRUSZOY3I665N43TNXICDJTXQ3ZJ62OHPGETSQ, seg_pos={0 0}, upload=67501]: Done uploading piece. err: <nil> Dec 8 20:43:03.015459803 (77h14m2.886542848s) [noise=true , address=145.79.227.41:4087 , piece_id=QYRY35JJKPQEK4BAVFZ6DEBKJN7SBUHG66S44ZA7XRNOVKAVQ33Q, seg_pos={0 0}, upload=67501]: Done uploading piece. err: <nil> Dec 8 20:43:03.020621582 (77h14m2.891704617s) [noise=true , address=145.79.227.20:4437 , piece_id=CCMQ7GX5NBRAUKHEHERGP5SHETIGELARKU5AE5ETKZWQRJP7OOAQ, seg_pos={0 0}, upload=67501]: Done uploading piece. err: <nil> Dec 8 20:43:03.020672098 (77h14m2.891755153s) [noise=true , address=145.79.227.31:4797 , piece_id=ECY3CM3GYNMM5RIEDHYI5NVWDCKEXRRKH4NCK3FV5ILW7W54LIIA, seg_pos={0 0}, upload=67501]: Done uploading piece. err: <nil> Dec 8 20:43:03.028139237 (77h14m2.899222282s) [noise=true , address=145.79.227.24:4583 , piece_id=DRLVD3TA2Z6TJQ4WKA67KDTXK7Q32YJH2ES5QEQG7Z37DYBJBB6Q, seg_pos={0 0}, upload=67501]: Done uploading piece. err: <nil> Dec 8 20:43:03.031290167 (77h14m2.902373202s) [noise=true , address=145.79.227.33:4870 , piece_id=KY2FLO3KDFPWRXPUUFINFLIRLZDMHPUCSPKRCN7RYPVAXXUYMXWQ, seg_pos={0 0}, upload=67501]: Done uploading piece. err: <nil> Dec 8 20:43:03.033756633 (77h14m2.904839668s) [noise=true , address=145.79.227.25:4618 , piece_id=AE7LMOAKNNIKVS4STMQJKZER7YUSFSUTSE7P5KXYTG4AQLFNQHLQ, seg_pos={0 0}, upload=67501]: Done uploading piece. err: <nil> Dec 8 20:43:03.039153600 (77h14m2.910236655s) [noise=true , address=145.79.227.40:4050 , piece_id=K2RW6IQERLFKEUD2ZTAZEX767U77PQWEXMXEH2SYDBQ7BT26WAHQ, seg_pos={0 0}, upload=67501]: Done uploading piece. err: <nil> Dec 8 20:43:03.040008869 (77h14m2.911091914s) [noise=true , address=145.79.227.38:4977 , piece_id=JW72CEO3WDGEB6BSSAR3M4Z5ANDBH63BLJZRSYAJDK7F7EXCEOZQ, seg_pos={0 0}, upload=67501]: Done uploading piece. err: <nil> Dec 8 20:43:03.178911548 (77h14m3.049994593s) [noise=true , address=77.90.31.18:4807 , piece_id=KDIWMU2XEJ7NTRCSG5J37ZUWIFURSJVK2BSDM7VJFKOB3ROANRGA, seg_pos={0 0}, upload=67501]: Done uploading piece. err: <nil> Dec 8 20:43:04.409564163 (77h14m4.280647208s) [noise=true , address=145.79.227.44:4195 , piece_id=EYY7B3AXUMA63GQLGKQWPWGDZUBORTXR6QFYSEOB26ATZK2GJC6Q, seg_pos={0 0}, upload=67501]: Done uploading piece. err: <nil> Dec 8 20:43:30.067753910 (77h14m29.938836955s) [noise=true , address=145.79.227.34:4906 , piece_id=VPDAFAMOLFSYUGFZIHRTADXWLYAE7IXILSPLT6QL2IPCOVUDPA3A, seg_pos={0 0}, upload=67501]: Done uploading piece. err: ecclient: upload failed (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw, address:145.79.227.34:4906): protocol: read tcp 188.166.225.170:36576->145.79.227.34:4906: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 20:43:30.067797914 (77h14m29.938880959s) [seg_pos={0 0}, upload=67501]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 20:43:30.323605733 (77h14m30.194688778s) [upload=67501]: Done uploading segments... Dec 8 20:43:30.323629097 (77h14m30.194712142s) [upload=67501]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.742452927s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.........X....X....X........................................] duration: 5.920113986s trace: https://jaeger.dev.storj.io/trace/4600f1753cc382a2 FAIL (2025-12-08 20:52:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.920207453s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 2 out of 60 is failed (threshold=3) [....X..................................................X....] duration: 469.578686ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/51c3a25722f21601 FAIL (2025-12-08 20:51:59 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 469.605116ms core satellite eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................................................X] duration: 5.001360085s trace: https://jaeger.dev.storj.io/trace/65b6db6cde985dc2 FAIL (2025-12-08 20:51:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001383049s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..........X....X....X.......................................] duration: 5.824525488s trace: https://jaeger.dev.storj.io/trace/4600f1753cc382a2 FAIL (2025-12-08 20:51:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.824549714s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [.......X...............X....................................] duration: 5.759444885s trace: https://jaeger.dev.storj.io/trace/3996915dd8f30735 FAIL (2025-12-08 20:51:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== PGM26LWTYTWGZGQDPBL7UDX263DFBXY7WZRXGMYEFPWXQQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.569231809 (76h54m34.440315054s) [noise=true , address=89.149.201.147:4022 , piece_id=2VR3E4SXAAOR5RAVHFVUQQ7MBZQEEQYFZ2ZW5Z7YSCVOETZNFQWQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.569369882 (76h54m34.440452917s) [noise=true , address=185.167.204.206:4037 , piece_id=XYY5NAATQJPHOK6JAMHXR3JH3QTLQF7OTA32BMST5HYQDYJ76UMA, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.569672879 (76h54m34.440755924s) [noise=true , address=185.167.204.206:4056 , piece_id=IPDW6RCHJFLMB5LOZM3FG645CCW6TT7KJW4PBQXBUSAIE2ZZM74A, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.569779642 (76h54m34.440862677s) [noise=true , address=185.167.204.205:4024 , piece_id=2UHV7KDAKJJ4DDUU5MPBNLRP24OK5RQU6GGAVYUV5MEFLDDPGUZQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.570073130 (76h54m34.441156176s) [noise=true , address=81.171.14.70:4035 , piece_id=ZKLJ5VUPALKZBSPFCJQBZZUTMJ3PX2H3XMUZ5JVSW5BJBACRUYHA, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.570428618 (76h54m34.441511653s) [noise=true , address=81.171.14.70:4043 , piece_id=FVUGWZXJLQAOLUSTRKJAULSHDWKX5GGWA2BB52YP7WQEKU4NKS5Q, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.570785237 (76h54m34.441868282s) [noise=true , address=81.171.14.70:4044 , piece_id=QPYQGR62JDWWV3GM3CVDB6BLNSGNUTQUEHHQF74TYXFN25PMNPVQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.571235034 (76h54m34.442318069s) [noise=true , address=185.167.204.205:4035 , piece_id=LXUP2T5SN5SSV7IWZLGTSLCRWLFRZBODXX7W4FFVHZFO74I2CQHQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.571655133 (76h54m34.442738169s) [noise=true , address=81.171.14.70:4031 , piece_id=53QOUYYU4KS53TELJUM3FPEKOCB464OR5GKYGYB54RVRPD52RFWA, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.572102666 (76h54m34.443185701s) [noise=true , address=185.167.204.205:4034 , piece_id=4VH6HEDD7CTPMEOHYGKFXSYXHKOLPYB2N6BCCJT2DXVAMADI3KJQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:35.395419290 (76h54m35.266502335s) [upload=67229]: Next returned nil segment Dec 8 20:23:35.395432614 (76h54m35.26651565s) [upload=67229]: Waiting for error group managing segments... Dec 8 20:23:35.780649683 (76h54m35.651732718s) [noise=true , address=185.167.204.206:4037 , piece_id=XYY5NAATQJPHOK6JAMHXR3JH3QTLQF7OTA32BMST5HYQDYJ76UMA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.781236982 (76h54m35.652320027s) [noise=true , address=185.167.204.206:4056 , piece_id=IPDW6RCHJFLMB5LOZM3FG645CCW6TT7KJW4PBQXBUSAIE2ZZM74A, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.784267011 (76h54m35.655350046s) [noise=true , address=185.167.204.205:4030 , piece_id=O3QKGK2G5I5YYPRP5K57RTG4H35H2UDFROFU2JMSHKTFNZJDVUIA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.785164540 (76h54m35.656247585s) [noise=true , address=185.167.204.205:4012 , piece_id=QCJD2IHH4DEKTFN5ZPMZXPECBQ2IX2QPSFUZCSFR5HKCDOVYGGQA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.785585301 (76h54m35.656668336s) [noise=true , address=185.167.204.205:4002 , piece_id=PDSJ5KDCFQBURILRTWU2WSEG3UH7GX4GK57W4J75QADBPKQXRSYA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.787925598 (76h54m35.659008643s) [noise=true , address=185.167.204.206:4047 , piece_id=THE6CUP46D2OSXS34UR3TTBUP6FIVMGDGT7DVJDBB57BHMWBCZ7A, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.787994289 (76h54m35.659077324s) [noise=true , address=185.167.204.205:4034 , piece_id=4VH6HEDD7CTPMEOHYGKFXSYXHKOLPYB2N6BCCJT2DXVAMADI3KJQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.788899402 (76h54m35.659982437s) [noise=true , address=185.167.204.206:4050 , piece_id=7B4GLDQZUPQPLGPZPPN5CO3VOX3LLBMLIYA6IMALYITCHE4CZBOA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.791033055 (76h54m35.66211609s) [noise=true , address=185.167.204.205:4031 , piece_id=5XPLPLO4SPMX7ZBYHLUZTH7MBGCRESEEMP2OEI7LTTT5BN67XHMQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.792163116 (76h54m35.663246161s) [noise=true , address=185.167.204.206:4055 , piece_id=JESF36BPW3YEPXVI2GP4OALECNLYSASJHQRZVLNXKV5BQRDGOWWQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.792693356 (76h54m35.663776392s) [noise=true , address=185.167.204.206:4063 , piece_id=BRUAUJTOCPWNUOB6D5G6AJCTMJBRZRJXXXBZBIUUZZXXQB5OEX2Q, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.793171606 (76h54m35.664254662s) [noise=true , address=185.167.204.206:4045 , piece_id=IXV6HTIOVQR4HCGJIWTBQFLO733BMMNNMZ24Y34H6V6KYUQYPQRA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.798151289 (76h54m35.669234344s) [noise=true , address=185.167.204.206:4053 , piece_id=VGK5AKCI23P5ZDS427NS6PNNRIJ3QUXLRZ4ECLYN4UDSL3KAQAKA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.798189562 (76h54m35.669272597s) [noise=true , address=185.167.204.206:4040 , piece_id=4KFICIELLLPCLWQ7RIB7UUSWUK7SFX2PJK7JWCRU5S4HQNHBEWRQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.798633607 (76h54m35.669716642s) [noise=true , address=185.167.204.205:4035 , piece_id=LXUP2T5SN5SSV7IWZLGTSLCRWLFRZBODXX7W4FFVHZFO74I2CQHQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.803059726 (76h54m35.674142761s) [noise=true , address=185.167.204.205:4024 , piece_id=2UHV7KDAKJJ4DDUU5MPBNLRP24OK5RQU6GGAVYUV5MEFLDDPGUZQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.344154907 (76h54m36.215237952s) [noise=true , address=81.171.14.70:4036 , piece_id=FB3SB3XQDNT2HSK5FDGELGYWODFLU655QW4D23GEXIWQEZYFNFRA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.359261713 (76h54m36.230344758s) [noise=true , address=81.171.14.70:4031 , piece_id=53QOUYYU4KS53TELJUM3FPEKOCB464OR5GKYGYB54RVRPD52RFWA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.393865115 (76h54m36.26494815s) [noise=true , address=81.171.14.70:4035 , piece_id=ZKLJ5VUPALKZBSPFCJQBZZUTMJ3PX2H3XMUZ5JVSW5BJBACRUYHA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.394302157 (76h54m36.265385202s) [noise=true , address=81.171.14.70:4044 , piece_id=QPYQGR62JDWWV3GM3CVDB6BLNSGNUTQUEHHQF74TYXFN25PMNPVQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.430200674 (76h54m36.301283709s) [noise=true , address=81.171.14.70:4046 , piece_id=ZHZUULNXPFDSTOXXH6T7XIEPN23OOWCNAAUWPCZYNNDTLPRWXVKQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.800799297 (76h54m36.671882332s) [noise=true , address=81.171.14.70:4040 , piece_id=W2SOTP5SPLQ6J3PM57NZOLRVZNMTNSGPWDZKZC2KV2PBPH4TUBLA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.862864259 (76h54m36.733947295s) [noise=true , address=81.171.14.70:4038 , piece_id=CKN2RF2YSTEOK6HUXGVBVECYAP2CLGUF6KYNR5JTHRUZPTGVI73Q, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.953769972 (76h54m36.824853007s) [noise=true , address=81.171.14.70:4043 , piece_id=FVUGWZXJLQAOLUSTRKJAULSHDWKX5GGWA2BB52YP7WQEKU4NKS5Q, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.996603376 (76h54m36.867686421s) [noise=true , address=89.149.201.147:4006 , piece_id=UGREZBZRVKXMGBOLQEV2NC4COTXOTTRUEAGSPNWRWP22MCPHQP7Q, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:37.037518916 (76h54m36.908601951s) [noise=true , address=89.149.201.147:4014 , piece_id=AEG6KGPGM26LWTYTWGZGQDPBL7UDX263DFBXY7WZRXGMYEFPWXQQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:37.037537541 (76h54m36.908620586s) [seg_pos={0 0}, upload=67229]: Segment reached optimal threshold of 26 pieces. Dec 8 20:23:37.037730880 (76h54m36.908813915s) [noise=true , address=89.149.201.147:4023 , piece_id=5EPGXWBK6GNTKLQLWEIWYBC2OJHFR3M7F4X3X4WODJWNXID3PCQQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BaeSxjXneBKoGKVrRM686sq386Bhn1YzzBFG9we8fHbcc9tbv): protocol: expected piece hash; context canceled; EOF Dec 8 20:23:37.037737452 (76h54m36.908820497s) [noise=true , address=89.149.201.147:4023 , piece_id=5EPGXWBK6GNTKLQLWEIWYBC2OJHFR3M7F4X3X4WODJWNXID3PCQQ, seg_pos={0 0}, upload=67229]: Long tail cancellation due to optimal threshold Dec 8 20:23:37.037893039 (76h54m36.908976074s) [noise=true , address=89.149.201.147:4022 , piece_id=2VR3E4SXAAOR5RAVHFVUQQ7MBZQEEQYFZ2ZW5Z7YSCVOETZNFQWQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122Vj8XxpS6RnhVweQV3cZWFhj8nrzy3hZ9P7JsuTzEaZcD6iFE): protocol: expected piece hash; context canceled; EOF Dec 8 20:23:37.037900964 (76h54m36.908984019s) [noise=true , address=89.149.201.147:4022 , piece_id=2VR3E4SXAAOR5RAVHFVUQQ7MBZQEEQYFZ2ZW5Z7YSCVOETZNFQWQ, seg_pos={0 0}, upload=67229]: Long tail cancellation due to optimal threshold Dec 8 20:23:37.037926843 (76h54m36.909009889s) [seg_pos={0 0}, upload=67229]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 8 20:23:37.038007697 (76h54m36.909090732s) [upload=67229]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 20:23:37.038013398 (76h54m36.909096433s) [upload=67229]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 20:26:49.742746371 (76h57m49.613829416s) [upload=67229]: Done uploading segments... Dec 8 20:26:49.742860908 (76h57m49.613943943s) [upload=67229]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.759471687s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............X................................X............] duration: 5.713250612s trace: https://jaeger.dev.storj.io/trace/2eed98fdb4249ce9 FAIL (2025-12-08 20:51:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 20:43:00.763765824 (77h14m0.634848859s) [upload=67501]: Starting upload Dec 8 20:43:00.763879119 (77h14m0.634962164s) [upload=67501]: Done starting upload Dec 8 20:43:00.763936168 (77h14m0.635019203s) [upload=67501]: Uploading segments... Dec 8 20:43:00.764117523 (77h14m0.635200568s) [upload=67501]: Got next segment. Inline: false Dec 8 20:43:00.764157569 (77h14m0.635240604s) [upload=67501]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 20:43:00.764162089 (77h14m0.635245134s) [upload=67501]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 20:43:01.015312549 (77h14m0.886395594s) [seg_pos={0 0}, upload=67501]: Begin upload segment... Dec 8 20:43:01.015378243 (77h14m0.886461288s) [seg_pos={0 0}, upload=67501]: Done begin upload segment. Dec 8 20:43:01.015411717 (77h14m0.886494762s) [noise=true , address=145.79.227.40:4050 , piece_id=K2RW6IQERLFKEUD2ZTAZEX767U77PQWEXMXEH2SYDBQ7BT26WAHQ, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.018044751 (77h14m0.889127796s) [noise=true , address=145.79.227.33:4870 , piece_id=KY2FLO3KDFPWRXPUUFINFLIRLZDMHPUCSPKRCN7RYPVAXXUYMXWQ, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.019100842 (77h14m0.890183887s) [noise=true , address=145.79.227.31:4797 , piece_id=ECY3CM3GYNMM5RIEDHYI5NVWDCKEXRRKH4NCK3FV5ILW7W54LIIA, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.020021405 (77h14m0.89110444s) [noise=true , address=145.79.227.41:4087 , piece_id=QYRY35JJKPQEK4BAVFZ6DEBKJN7SBUHG66S44ZA7XRNOVKAVQ33Q, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.020812742 (77h14m0.891895787s) [noise=true , address=145.79.227.20:4437 , piece_id=CCMQ7GX5NBRAUKHEHERGP5SHETIGELARKU5AE5ETKZWQRJP7OOAQ, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.021487076 (77h14m0.892570111s) [noise=true , address=77.90.31.21:4879 , piece_id=WF6YF2HQIXM26JI5TQ3YW7OXHNYZH3E2VTHAQVWIUHZ3ME3CCG6Q, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.022677542 (77h14m0.893760588s) [noise=true , address=145.79.227.42:4123 , piece_id=NBMNEIGLWZHFGHY674WDGEDH4CGDCKYERXLB7UKXXI4MTZJ2HVEQ, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.022691720 (77h14m0.893774755s) [noise=true , address=77.90.31.18:4807 , piece_id=KDIWMU2XEJ7NTRCSG5J37ZUWIFURSJVK2BSDM7VJFKOB3ROANRGA, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.023821471 (77h14m0.894904506s) [noise=true , address=145.79.227.41:4085 , piece_id=LS54ZFJC7DEGOKQRUSZOY3I665N43TNXICDJTXQ3ZJ62OHPGETSQ, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.024740981 (77h14m0.895824026s) [noise=true , address=77.90.31.12:4589 , piece_id=WUVA4G7EEN4J4HF47KQ7FZKFINAN2CCYDXMRNHDZVBF6EAPVFS4Q, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.025197331 (77h14m0.896280366s) [noise=true , address=145.79.227.34:4906 , piece_id=VPDAFAMOLFSYUGFZIHRTADXWLYAE7IXILSPLT6QL2IPCOVUDPA3A, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.026609700 (77h14m0.897692745s) [noise=true , address=145.79.227.25:4618 , piece_id=AE7LMOAKNNIKVS4STMQJKZER7YUSFSUTSE7P5KXYTG4AQLFNQHLQ, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.027084214 (77h14m0.898167259s) [noise=true , address=145.79.227.44:4195 , piece_id=EYY7B3AXUMA63GQLGKQWPWGDZUBORTXR6QFYSEOB26ATZK2GJC6Q, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.028812575 (77h14m0.89989562s) [noise=true , address=145.79.227.38:4977 , piece_id=JW72CEO3WDGEB6BSSAR3M4Z5ANDBH63BLJZRSYAJDK7F7EXCEOZQ, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.030193064 (77h14m0.90127611s) [noise=true , address=145.79.227.24:4583 , piece_id=DRLVD3TA2Z6TJQ4WKA67KDTXK7Q32YJH2ES5QEQG7Z37DYBJBB6Q, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.036513970 (77h14m0.907597045s) [noise=true , address=77.90.31.13:4627 , piece_id=XMI4HVR2MQQ5KHN4GQOUCKFYJYJ2YZEGQGTAAXYKGS3OI6XPJLVA, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:02.093045104 (77h14m1.964128159s) [noise=true , address=77.90.31.21:4879 , piece_id=WF6YF2HQIXM26JI5TQ3YW7OXHNYZH3E2VTHAQVWIUHZ3ME3CCG6Q, seg_pos={0 0}, upload=67501]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: storage node overloaded, request limit: 2000 Dec 8 20:43:02.127847935 (77h14m1.99893098s) [noise=true , address=77.90.31.13:4627 , piece_id=XMI4HVR2MQQ5KHN4GQOUCKFYJYJ2YZEGQGTAAXYKGS3OI6XPJLVA, seg_pos={0 0}, upload=67501]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 8 20:43:02.156386568 (77h14m2.027469613s) [noise=true , address=77.90.31.12:4589 , piece_id=WUVA4G7EEN4J4HF47KQ7FZKFINAN2CCYDXMRNHDZVBF6EAPVFS4Q, seg_pos={0 0}, upload=67501]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 20:43:02.719927441 (77h14m2.591010486s) [upload=67501]: Next returned nil segment Dec 8 20:43:02.719942970 (77h14m2.591026005s) [upload=67501]: Waiting for error group managing segments... Dec 8 20:43:03.005012008 (77h14m2.876095063s) [noise=true , address=145.79.227.42:4123 , piece_id=NBMNEIGLWZHFGHY674WDGEDH4CGDCKYERXLB7UKXXI4MTZJ2HVEQ, seg_pos={0 0}, upload=67501]: Done uploading piece. err: <nil> Dec 8 20:43:03.012525866 (77h14m2.883608911s) [noise=true , address=145.79.227.41:4085 , piece_id=LS54ZFJC7DEGOKQRUSZOY3I665N43TNXICDJTXQ3ZJ62OHPGETSQ, seg_pos={0 0}, upload=67501]: Done uploading piece. err: <nil> Dec 8 20:43:03.015459803 (77h14m2.886542848s) [noise=true , address=145.79.227.41:4087 , piece_id=QYRY35JJKPQEK4BAVFZ6DEBKJN7SBUHG66S44ZA7XRNOVKAVQ33Q, seg_pos={0 0}, upload=67501]: Done uploading piece. err: <nil> Dec 8 20:43:03.020621582 (77h14m2.891704617s) [noise=true , address=145.79.227.20:4437 , piece_id=CCMQ7GX5NBRAUKHEHERGP5SHETIGELARKU5AE5ETKZWQRJP7OOAQ, seg_pos={0 0}, upload=67501]: Done uploading piece. err: <nil> Dec 8 20:43:03.020672098 (77h14m2.891755153s) [noise=true , address=145.79.227.31:4797 , piece_id=ECY3CM3GYNMM5RIEDHYI5NVWDCKEXRRKH4NCK3FV5ILW7W54LIIA, seg_pos={0 0}, upload=67501]: Done uploading piece. err: <nil> Dec 8 20:43:03.028139237 (77h14m2.899222282s) [noise=true , address=145.79.227.24:4583 , piece_id=DRLVD3TA2Z6TJQ4WKA67KDTXK7Q32YJH2ES5QEQG7Z37DYBJBB6Q, seg_pos={0 0}, upload=67501]: Done uploading piece. err: <nil> Dec 8 20:43:03.031290167 (77h14m2.902373202s) [noise=true , address=145.79.227.33:4870 , piece_id=KY2FLO3KDFPWRXPUUFINFLIRLZDMHPUCSPKRCN7RYPVAXXUYMXWQ, seg_pos={0 0}, upload=67501]: Done uploading piece. err: <nil> Dec 8 20:43:03.033756633 (77h14m2.904839668s) [noise=true , address=145.79.227.25:4618 , piece_id=AE7LMOAKNNIKVS4STMQJKZER7YUSFSUTSE7P5KXYTG4AQLFNQHLQ, seg_pos={0 0}, upload=67501]: Done uploading piece. err: <nil> Dec 8 20:43:03.039153600 (77h14m2.910236655s) [noise=true , address=145.79.227.40:4050 , piece_id=K2RW6IQERLFKEUD2ZTAZEX767U77PQWEXMXEH2SYDBQ7BT26WAHQ, seg_pos={0 0}, upload=67501]: Done uploading piece. err: <nil> Dec 8 20:43:03.040008869 (77h14m2.911091914s) [noise=true , address=145.79.227.38:4977 , piece_id=JW72CEO3WDGEB6BSSAR3M4Z5ANDBH63BLJZRSYAJDK7F7EXCEOZQ, seg_pos={0 0}, upload=67501]: Done uploading piece. err: <nil> Dec 8 20:43:03.178911548 (77h14m3.049994593s) [noise=true , address=77.90.31.18:4807 , piece_id=KDIWMU2XEJ7NTRCSG5J37ZUWIFURSJVK2BSDM7VJFKOB3ROANRGA, seg_pos={0 0}, upload=67501]: Done uploading piece. err: <nil> Dec 8 20:43:04.409564163 (77h14m4.280647208s) [noise=true , address=145.79.227.44:4195 , piece_id=EYY7B3AXUMA63GQLGKQWPWGDZUBORTXR6QFYSEOB26ATZK2GJC6Q, seg_pos={0 0}, upload=67501]: Done uploading piece. err: <nil> Dec 8 20:43:30.067753910 (77h14m29.938836955s) [noise=true , address=145.79.227.34:4906 , piece_id=VPDAFAMOLFSYUGFZIHRTADXWLYAE7IXILSPLT6QL2IPCOVUDPA3A, seg_pos={0 0}, upload=67501]: Done uploading piece. err: ecclient: upload failed (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw, address:145.79.227.34:4906): protocol: read tcp 188.166.225.170:36576->145.79.227.34:4906: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 20:43:30.067797914 (77h14m29.938880959s) [seg_pos={0 0}, upload=67501]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 20:43:30.323605733 (77h14m30.194688778s) [upload=67501]: Done uploading segments... Dec 8 20:43:30.323629097 (77h14m30.194712142s) [upload=67501]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.713278706s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 5.118426253s trace: https://jaeger.dev.storj.io/trace/49fc2034659f6881 FAIL (2025-12-08 20:51:18 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.118452464s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [........X...............X...................................] duration: 5.787546434s trace: https://jaeger.dev.storj.io/trace/3996915dd8f30735 FAIL (2025-12-08 20:51:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== PGM26LWTYTWGZGQDPBL7UDX263DFBXY7WZRXGMYEFPWXQQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.569231809 (76h54m34.440315054s) [noise=true , address=89.149.201.147:4022 , piece_id=2VR3E4SXAAOR5RAVHFVUQQ7MBZQEEQYFZ2ZW5Z7YSCVOETZNFQWQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.569369882 (76h54m34.440452917s) [noise=true , address=185.167.204.206:4037 , piece_id=XYY5NAATQJPHOK6JAMHXR3JH3QTLQF7OTA32BMST5HYQDYJ76UMA, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.569672879 (76h54m34.440755924s) [noise=true , address=185.167.204.206:4056 , piece_id=IPDW6RCHJFLMB5LOZM3FG645CCW6TT7KJW4PBQXBUSAIE2ZZM74A, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.569779642 (76h54m34.440862677s) [noise=true , address=185.167.204.205:4024 , piece_id=2UHV7KDAKJJ4DDUU5MPBNLRP24OK5RQU6GGAVYUV5MEFLDDPGUZQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.570073130 (76h54m34.441156176s) [noise=true , address=81.171.14.70:4035 , piece_id=ZKLJ5VUPALKZBSPFCJQBZZUTMJ3PX2H3XMUZ5JVSW5BJBACRUYHA, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.570428618 (76h54m34.441511653s) [noise=true , address=81.171.14.70:4043 , piece_id=FVUGWZXJLQAOLUSTRKJAULSHDWKX5GGWA2BB52YP7WQEKU4NKS5Q, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.570785237 (76h54m34.441868282s) [noise=true , address=81.171.14.70:4044 , piece_id=QPYQGR62JDWWV3GM3CVDB6BLNSGNUTQUEHHQF74TYXFN25PMNPVQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.571235034 (76h54m34.442318069s) [noise=true , address=185.167.204.205:4035 , piece_id=LXUP2T5SN5SSV7IWZLGTSLCRWLFRZBODXX7W4FFVHZFO74I2CQHQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.571655133 (76h54m34.442738169s) [noise=true , address=81.171.14.70:4031 , piece_id=53QOUYYU4KS53TELJUM3FPEKOCB464OR5GKYGYB54RVRPD52RFWA, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.572102666 (76h54m34.443185701s) [noise=true , address=185.167.204.205:4034 , piece_id=4VH6HEDD7CTPMEOHYGKFXSYXHKOLPYB2N6BCCJT2DXVAMADI3KJQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:35.395419290 (76h54m35.266502335s) [upload=67229]: Next returned nil segment Dec 8 20:23:35.395432614 (76h54m35.26651565s) [upload=67229]: Waiting for error group managing segments... Dec 8 20:23:35.780649683 (76h54m35.651732718s) [noise=true , address=185.167.204.206:4037 , piece_id=XYY5NAATQJPHOK6JAMHXR3JH3QTLQF7OTA32BMST5HYQDYJ76UMA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.781236982 (76h54m35.652320027s) [noise=true , address=185.167.204.206:4056 , piece_id=IPDW6RCHJFLMB5LOZM3FG645CCW6TT7KJW4PBQXBUSAIE2ZZM74A, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.784267011 (76h54m35.655350046s) [noise=true , address=185.167.204.205:4030 , piece_id=O3QKGK2G5I5YYPRP5K57RTG4H35H2UDFROFU2JMSHKTFNZJDVUIA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.785164540 (76h54m35.656247585s) [noise=true , address=185.167.204.205:4012 , piece_id=QCJD2IHH4DEKTFN5ZPMZXPECBQ2IX2QPSFUZCSFR5HKCDOVYGGQA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.785585301 (76h54m35.656668336s) [noise=true , address=185.167.204.205:4002 , piece_id=PDSJ5KDCFQBURILRTWU2WSEG3UH7GX4GK57W4J75QADBPKQXRSYA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.787925598 (76h54m35.659008643s) [noise=true , address=185.167.204.206:4047 , piece_id=THE6CUP46D2OSXS34UR3TTBUP6FIVMGDGT7DVJDBB57BHMWBCZ7A, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.787994289 (76h54m35.659077324s) [noise=true , address=185.167.204.205:4034 , piece_id=4VH6HEDD7CTPMEOHYGKFXSYXHKOLPYB2N6BCCJT2DXVAMADI3KJQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.788899402 (76h54m35.659982437s) [noise=true , address=185.167.204.206:4050 , piece_id=7B4GLDQZUPQPLGPZPPN5CO3VOX3LLBMLIYA6IMALYITCHE4CZBOA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.791033055 (76h54m35.66211609s) [noise=true , address=185.167.204.205:4031 , piece_id=5XPLPLO4SPMX7ZBYHLUZTH7MBGCRESEEMP2OEI7LTTT5BN67XHMQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.792163116 (76h54m35.663246161s) [noise=true , address=185.167.204.206:4055 , piece_id=JESF36BPW3YEPXVI2GP4OALECNLYSASJHQRZVLNXKV5BQRDGOWWQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.792693356 (76h54m35.663776392s) [noise=true , address=185.167.204.206:4063 , piece_id=BRUAUJTOCPWNUOB6D5G6AJCTMJBRZRJXXXBZBIUUZZXXQB5OEX2Q, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.793171606 (76h54m35.664254662s) [noise=true , address=185.167.204.206:4045 , piece_id=IXV6HTIOVQR4HCGJIWTBQFLO733BMMNNMZ24Y34H6V6KYUQYPQRA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.798151289 (76h54m35.669234344s) [noise=true , address=185.167.204.206:4053 , piece_id=VGK5AKCI23P5ZDS427NS6PNNRIJ3QUXLRZ4ECLYN4UDSL3KAQAKA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.798189562 (76h54m35.669272597s) [noise=true , address=185.167.204.206:4040 , piece_id=4KFICIELLLPCLWQ7RIB7UUSWUK7SFX2PJK7JWCRU5S4HQNHBEWRQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.798633607 (76h54m35.669716642s) [noise=true , address=185.167.204.205:4035 , piece_id=LXUP2T5SN5SSV7IWZLGTSLCRWLFRZBODXX7W4FFVHZFO74I2CQHQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.803059726 (76h54m35.674142761s) [noise=true , address=185.167.204.205:4024 , piece_id=2UHV7KDAKJJ4DDUU5MPBNLRP24OK5RQU6GGAVYUV5MEFLDDPGUZQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.344154907 (76h54m36.215237952s) [noise=true , address=81.171.14.70:4036 , piece_id=FB3SB3XQDNT2HSK5FDGELGYWODFLU655QW4D23GEXIWQEZYFNFRA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.359261713 (76h54m36.230344758s) [noise=true , address=81.171.14.70:4031 , piece_id=53QOUYYU4KS53TELJUM3FPEKOCB464OR5GKYGYB54RVRPD52RFWA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.393865115 (76h54m36.26494815s) [noise=true , address=81.171.14.70:4035 , piece_id=ZKLJ5VUPALKZBSPFCJQBZZUTMJ3PX2H3XMUZ5JVSW5BJBACRUYHA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.394302157 (76h54m36.265385202s) [noise=true , address=81.171.14.70:4044 , piece_id=QPYQGR62JDWWV3GM3CVDB6BLNSGNUTQUEHHQF74TYXFN25PMNPVQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.430200674 (76h54m36.301283709s) [noise=true , address=81.171.14.70:4046 , piece_id=ZHZUULNXPFDSTOXXH6T7XIEPN23OOWCNAAUWPCZYNNDTLPRWXVKQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.800799297 (76h54m36.671882332s) [noise=true , address=81.171.14.70:4040 , piece_id=W2SOTP5SPLQ6J3PM57NZOLRVZNMTNSGPWDZKZC2KV2PBPH4TUBLA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.862864259 (76h54m36.733947295s) [noise=true , address=81.171.14.70:4038 , piece_id=CKN2RF2YSTEOK6HUXGVBVECYAP2CLGUF6KYNR5JTHRUZPTGVI73Q, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.953769972 (76h54m36.824853007s) [noise=true , address=81.171.14.70:4043 , piece_id=FVUGWZXJLQAOLUSTRKJAULSHDWKX5GGWA2BB52YP7WQEKU4NKS5Q, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.996603376 (76h54m36.867686421s) [noise=true , address=89.149.201.147:4006 , piece_id=UGREZBZRVKXMGBOLQEV2NC4COTXOTTRUEAGSPNWRWP22MCPHQP7Q, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:37.037518916 (76h54m36.908601951s) [noise=true , address=89.149.201.147:4014 , piece_id=AEG6KGPGM26LWTYTWGZGQDPBL7UDX263DFBXY7WZRXGMYEFPWXQQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:37.037537541 (76h54m36.908620586s) [seg_pos={0 0}, upload=67229]: Segment reached optimal threshold of 26 pieces. Dec 8 20:23:37.037730880 (76h54m36.908813915s) [noise=true , address=89.149.201.147:4023 , piece_id=5EPGXWBK6GNTKLQLWEIWYBC2OJHFR3M7F4X3X4WODJWNXID3PCQQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BaeSxjXneBKoGKVrRM686sq386Bhn1YzzBFG9we8fHbcc9tbv): protocol: expected piece hash; context canceled; EOF Dec 8 20:23:37.037737452 (76h54m36.908820497s) [noise=true , address=89.149.201.147:4023 , piece_id=5EPGXWBK6GNTKLQLWEIWYBC2OJHFR3M7F4X3X4WODJWNXID3PCQQ, seg_pos={0 0}, upload=67229]: Long tail cancellation due to optimal threshold Dec 8 20:23:37.037893039 (76h54m36.908976074s) [noise=true , address=89.149.201.147:4022 , piece_id=2VR3E4SXAAOR5RAVHFVUQQ7MBZQEEQYFZ2ZW5Z7YSCVOETZNFQWQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122Vj8XxpS6RnhVweQV3cZWFhj8nrzy3hZ9P7JsuTzEaZcD6iFE): protocol: expected piece hash; context canceled; EOF Dec 8 20:23:37.037900964 (76h54m36.908984019s) [noise=true , address=89.149.201.147:4022 , piece_id=2VR3E4SXAAOR5RAVHFVUQQ7MBZQEEQYFZ2ZW5Z7YSCVOETZNFQWQ, seg_pos={0 0}, upload=67229]: Long tail cancellation due to optimal threshold Dec 8 20:23:37.037926843 (76h54m36.909009889s) [seg_pos={0 0}, upload=67229]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 8 20:23:37.038007697 (76h54m36.909090732s) [upload=67229]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 20:23:37.038013398 (76h54m36.909096433s) [upload=67229]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 20:26:49.742746371 (76h57m49.613829416s) [upload=67229]: Done uploading segments... Dec 8 20:26:49.742860908 (76h57m49.613943943s) [upload=67229]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.787581672s core uplink eu-select-1 eu1-placement-15
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 2 out of 60 is failed (threshold=3) [.....X..................................................X...] duration: 476.654119ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/51c3a25722f21601 FAIL (2025-12-08 20:50:59 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 476.684588ms core satellite eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [.........X...............X..................................] duration: 4.581784495s trace: https://jaeger.dev.storj.io/trace/3996915dd8f30735 FAIL (2025-12-08 20:50:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== PGM26LWTYTWGZGQDPBL7UDX263DFBXY7WZRXGMYEFPWXQQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.569231809 (76h54m34.440315054s) [noise=true , address=89.149.201.147:4022 , piece_id=2VR3E4SXAAOR5RAVHFVUQQ7MBZQEEQYFZ2ZW5Z7YSCVOETZNFQWQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.569369882 (76h54m34.440452917s) [noise=true , address=185.167.204.206:4037 , piece_id=XYY5NAATQJPHOK6JAMHXR3JH3QTLQF7OTA32BMST5HYQDYJ76UMA, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.569672879 (76h54m34.440755924s) [noise=true , address=185.167.204.206:4056 , piece_id=IPDW6RCHJFLMB5LOZM3FG645CCW6TT7KJW4PBQXBUSAIE2ZZM74A, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.569779642 (76h54m34.440862677s) [noise=true , address=185.167.204.205:4024 , piece_id=2UHV7KDAKJJ4DDUU5MPBNLRP24OK5RQU6GGAVYUV5MEFLDDPGUZQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.570073130 (76h54m34.441156176s) [noise=true , address=81.171.14.70:4035 , piece_id=ZKLJ5VUPALKZBSPFCJQBZZUTMJ3PX2H3XMUZ5JVSW5BJBACRUYHA, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.570428618 (76h54m34.441511653s) [noise=true , address=81.171.14.70:4043 , piece_id=FVUGWZXJLQAOLUSTRKJAULSHDWKX5GGWA2BB52YP7WQEKU4NKS5Q, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.570785237 (76h54m34.441868282s) [noise=true , address=81.171.14.70:4044 , piece_id=QPYQGR62JDWWV3GM3CVDB6BLNSGNUTQUEHHQF74TYXFN25PMNPVQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.571235034 (76h54m34.442318069s) [noise=true , address=185.167.204.205:4035 , piece_id=LXUP2T5SN5SSV7IWZLGTSLCRWLFRZBODXX7W4FFVHZFO74I2CQHQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.571655133 (76h54m34.442738169s) [noise=true , address=81.171.14.70:4031 , piece_id=53QOUYYU4KS53TELJUM3FPEKOCB464OR5GKYGYB54RVRPD52RFWA, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.572102666 (76h54m34.443185701s) [noise=true , address=185.167.204.205:4034 , piece_id=4VH6HEDD7CTPMEOHYGKFXSYXHKOLPYB2N6BCCJT2DXVAMADI3KJQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:35.395419290 (76h54m35.266502335s) [upload=67229]: Next returned nil segment Dec 8 20:23:35.395432614 (76h54m35.26651565s) [upload=67229]: Waiting for error group managing segments... Dec 8 20:23:35.780649683 (76h54m35.651732718s) [noise=true , address=185.167.204.206:4037 , piece_id=XYY5NAATQJPHOK6JAMHXR3JH3QTLQF7OTA32BMST5HYQDYJ76UMA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.781236982 (76h54m35.652320027s) [noise=true , address=185.167.204.206:4056 , piece_id=IPDW6RCHJFLMB5LOZM3FG645CCW6TT7KJW4PBQXBUSAIE2ZZM74A, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.784267011 (76h54m35.655350046s) [noise=true , address=185.167.204.205:4030 , piece_id=O3QKGK2G5I5YYPRP5K57RTG4H35H2UDFROFU2JMSHKTFNZJDVUIA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.785164540 (76h54m35.656247585s) [noise=true , address=185.167.204.205:4012 , piece_id=QCJD2IHH4DEKTFN5ZPMZXPECBQ2IX2QPSFUZCSFR5HKCDOVYGGQA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.785585301 (76h54m35.656668336s) [noise=true , address=185.167.204.205:4002 , piece_id=PDSJ5KDCFQBURILRTWU2WSEG3UH7GX4GK57W4J75QADBPKQXRSYA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.787925598 (76h54m35.659008643s) [noise=true , address=185.167.204.206:4047 , piece_id=THE6CUP46D2OSXS34UR3TTBUP6FIVMGDGT7DVJDBB57BHMWBCZ7A, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.787994289 (76h54m35.659077324s) [noise=true , address=185.167.204.205:4034 , piece_id=4VH6HEDD7CTPMEOHYGKFXSYXHKOLPYB2N6BCCJT2DXVAMADI3KJQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.788899402 (76h54m35.659982437s) [noise=true , address=185.167.204.206:4050 , piece_id=7B4GLDQZUPQPLGPZPPN5CO3VOX3LLBMLIYA6IMALYITCHE4CZBOA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.791033055 (76h54m35.66211609s) [noise=true , address=185.167.204.205:4031 , piece_id=5XPLPLO4SPMX7ZBYHLUZTH7MBGCRESEEMP2OEI7LTTT5BN67XHMQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.792163116 (76h54m35.663246161s) [noise=true , address=185.167.204.206:4055 , piece_id=JESF36BPW3YEPXVI2GP4OALECNLYSASJHQRZVLNXKV5BQRDGOWWQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.792693356 (76h54m35.663776392s) [noise=true , address=185.167.204.206:4063 , piece_id=BRUAUJTOCPWNUOB6D5G6AJCTMJBRZRJXXXBZBIUUZZXXQB5OEX2Q, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.793171606 (76h54m35.664254662s) [noise=true , address=185.167.204.206:4045 , piece_id=IXV6HTIOVQR4HCGJIWTBQFLO733BMMNNMZ24Y34H6V6KYUQYPQRA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.798151289 (76h54m35.669234344s) [noise=true , address=185.167.204.206:4053 , piece_id=VGK5AKCI23P5ZDS427NS6PNNRIJ3QUXLRZ4ECLYN4UDSL3KAQAKA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.798189562 (76h54m35.669272597s) [noise=true , address=185.167.204.206:4040 , piece_id=4KFICIELLLPCLWQ7RIB7UUSWUK7SFX2PJK7JWCRU5S4HQNHBEWRQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.798633607 (76h54m35.669716642s) [noise=true , address=185.167.204.205:4035 , piece_id=LXUP2T5SN5SSV7IWZLGTSLCRWLFRZBODXX7W4FFVHZFO74I2CQHQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.803059726 (76h54m35.674142761s) [noise=true , address=185.167.204.205:4024 , piece_id=2UHV7KDAKJJ4DDUU5MPBNLRP24OK5RQU6GGAVYUV5MEFLDDPGUZQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.344154907 (76h54m36.215237952s) [noise=true , address=81.171.14.70:4036 , piece_id=FB3SB3XQDNT2HSK5FDGELGYWODFLU655QW4D23GEXIWQEZYFNFRA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.359261713 (76h54m36.230344758s) [noise=true , address=81.171.14.70:4031 , piece_id=53QOUYYU4KS53TELJUM3FPEKOCB464OR5GKYGYB54RVRPD52RFWA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.393865115 (76h54m36.26494815s) [noise=true , address=81.171.14.70:4035 , piece_id=ZKLJ5VUPALKZBSPFCJQBZZUTMJ3PX2H3XMUZ5JVSW5BJBACRUYHA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.394302157 (76h54m36.265385202s) [noise=true , address=81.171.14.70:4044 , piece_id=QPYQGR62JDWWV3GM3CVDB6BLNSGNUTQUEHHQF74TYXFN25PMNPVQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.430200674 (76h54m36.301283709s) [noise=true , address=81.171.14.70:4046 , piece_id=ZHZUULNXPFDSTOXXH6T7XIEPN23OOWCNAAUWPCZYNNDTLPRWXVKQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.800799297 (76h54m36.671882332s) [noise=true , address=81.171.14.70:4040 , piece_id=W2SOTP5SPLQ6J3PM57NZOLRVZNMTNSGPWDZKZC2KV2PBPH4TUBLA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.862864259 (76h54m36.733947295s) [noise=true , address=81.171.14.70:4038 , piece_id=CKN2RF2YSTEOK6HUXGVBVECYAP2CLGUF6KYNR5JTHRUZPTGVI73Q, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.953769972 (76h54m36.824853007s) [noise=true , address=81.171.14.70:4043 , piece_id=FVUGWZXJLQAOLUSTRKJAULSHDWKX5GGWA2BB52YP7WQEKU4NKS5Q, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.996603376 (76h54m36.867686421s) [noise=true , address=89.149.201.147:4006 , piece_id=UGREZBZRVKXMGBOLQEV2NC4COTXOTTRUEAGSPNWRWP22MCPHQP7Q, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:37.037518916 (76h54m36.908601951s) [noise=true , address=89.149.201.147:4014 , piece_id=AEG6KGPGM26LWTYTWGZGQDPBL7UDX263DFBXY7WZRXGMYEFPWXQQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:37.037537541 (76h54m36.908620586s) [seg_pos={0 0}, upload=67229]: Segment reached optimal threshold of 26 pieces. Dec 8 20:23:37.037730880 (76h54m36.908813915s) [noise=true , address=89.149.201.147:4023 , piece_id=5EPGXWBK6GNTKLQLWEIWYBC2OJHFR3M7F4X3X4WODJWNXID3PCQQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BaeSxjXneBKoGKVrRM686sq386Bhn1YzzBFG9we8fHbcc9tbv): protocol: expected piece hash; context canceled; EOF Dec 8 20:23:37.037737452 (76h54m36.908820497s) [noise=true , address=89.149.201.147:4023 , piece_id=5EPGXWBK6GNTKLQLWEIWYBC2OJHFR3M7F4X3X4WODJWNXID3PCQQ, seg_pos={0 0}, upload=67229]: Long tail cancellation due to optimal threshold Dec 8 20:23:37.037893039 (76h54m36.908976074s) [noise=true , address=89.149.201.147:4022 , piece_id=2VR3E4SXAAOR5RAVHFVUQQ7MBZQEEQYFZ2ZW5Z7YSCVOETZNFQWQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122Vj8XxpS6RnhVweQV3cZWFhj8nrzy3hZ9P7JsuTzEaZcD6iFE): protocol: expected piece hash; context canceled; EOF Dec 8 20:23:37.037900964 (76h54m36.908984019s) [noise=true , address=89.149.201.147:4022 , piece_id=2VR3E4SXAAOR5RAVHFVUQQ7MBZQEEQYFZ2ZW5Z7YSCVOETZNFQWQ, seg_pos={0 0}, upload=67229]: Long tail cancellation due to optimal threshold Dec 8 20:23:37.037926843 (76h54m36.909009889s) [seg_pos={0 0}, upload=67229]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 8 20:23:37.038007697 (76h54m36.909090732s) [upload=67229]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 20:23:37.038013398 (76h54m36.909096433s) [upload=67229]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 20:26:49.742746371 (76h57m49.613829416s) [upload=67229]: Done uploading segments... Dec 8 20:26:49.742860908 (76h57m49.613943943s) [upload=67229]: Upload finished. err: metaclient: context deadline exceeded Duration: 4.581816977s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............X................................X...........] duration: 5.858235271s trace: https://jaeger.dev.storj.io/trace/2eed98fdb4249ce9 FAIL (2025-12-08 20:50:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 8 20:43:00.763765824 (77h14m0.634848859s) [upload=67501]: Starting upload Dec 8 20:43:00.763879119 (77h14m0.634962164s) [upload=67501]: Done starting upload Dec 8 20:43:00.763936168 (77h14m0.635019203s) [upload=67501]: Uploading segments... Dec 8 20:43:00.764117523 (77h14m0.635200568s) [upload=67501]: Got next segment. Inline: false Dec 8 20:43:00.764157569 (77h14m0.635240604s) [upload=67501]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 20:43:00.764162089 (77h14m0.635245134s) [upload=67501]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 20:43:01.015312549 (77h14m0.886395594s) [seg_pos={0 0}, upload=67501]: Begin upload segment... Dec 8 20:43:01.015378243 (77h14m0.886461288s) [seg_pos={0 0}, upload=67501]: Done begin upload segment. Dec 8 20:43:01.015411717 (77h14m0.886494762s) [noise=true , address=145.79.227.40:4050 , piece_id=K2RW6IQERLFKEUD2ZTAZEX767U77PQWEXMXEH2SYDBQ7BT26WAHQ, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.018044751 (77h14m0.889127796s) [noise=true , address=145.79.227.33:4870 , piece_id=KY2FLO3KDFPWRXPUUFINFLIRLZDMHPUCSPKRCN7RYPVAXXUYMXWQ, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.019100842 (77h14m0.890183887s) [noise=true , address=145.79.227.31:4797 , piece_id=ECY3CM3GYNMM5RIEDHYI5NVWDCKEXRRKH4NCK3FV5ILW7W54LIIA, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.020021405 (77h14m0.89110444s) [noise=true , address=145.79.227.41:4087 , piece_id=QYRY35JJKPQEK4BAVFZ6DEBKJN7SBUHG66S44ZA7XRNOVKAVQ33Q, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.020812742 (77h14m0.891895787s) [noise=true , address=145.79.227.20:4437 , piece_id=CCMQ7GX5NBRAUKHEHERGP5SHETIGELARKU5AE5ETKZWQRJP7OOAQ, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.021487076 (77h14m0.892570111s) [noise=true , address=77.90.31.21:4879 , piece_id=WF6YF2HQIXM26JI5TQ3YW7OXHNYZH3E2VTHAQVWIUHZ3ME3CCG6Q, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.022677542 (77h14m0.893760588s) [noise=true , address=145.79.227.42:4123 , piece_id=NBMNEIGLWZHFGHY674WDGEDH4CGDCKYERXLB7UKXXI4MTZJ2HVEQ, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.022691720 (77h14m0.893774755s) [noise=true , address=77.90.31.18:4807 , piece_id=KDIWMU2XEJ7NTRCSG5J37ZUWIFURSJVK2BSDM7VJFKOB3ROANRGA, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.023821471 (77h14m0.894904506s) [noise=true , address=145.79.227.41:4085 , piece_id=LS54ZFJC7DEGOKQRUSZOY3I665N43TNXICDJTXQ3ZJ62OHPGETSQ, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.024740981 (77h14m0.895824026s) [noise=true , address=77.90.31.12:4589 , piece_id=WUVA4G7EEN4J4HF47KQ7FZKFINAN2CCYDXMRNHDZVBF6EAPVFS4Q, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.025197331 (77h14m0.896280366s) [noise=true , address=145.79.227.34:4906 , piece_id=VPDAFAMOLFSYUGFZIHRTADXWLYAE7IXILSPLT6QL2IPCOVUDPA3A, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.026609700 (77h14m0.897692745s) [noise=true , address=145.79.227.25:4618 , piece_id=AE7LMOAKNNIKVS4STMQJKZER7YUSFSUTSE7P5KXYTG4AQLFNQHLQ, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.027084214 (77h14m0.898167259s) [noise=true , address=145.79.227.44:4195 , piece_id=EYY7B3AXUMA63GQLGKQWPWGDZUBORTXR6QFYSEOB26ATZK2GJC6Q, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.028812575 (77h14m0.89989562s) [noise=true , address=145.79.227.38:4977 , piece_id=JW72CEO3WDGEB6BSSAR3M4Z5ANDBH63BLJZRSYAJDK7F7EXCEOZQ, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.030193064 (77h14m0.90127611s) [noise=true , address=145.79.227.24:4583 , piece_id=DRLVD3TA2Z6TJQ4WKA67KDTXK7Q32YJH2ES5QEQG7Z37DYBJBB6Q, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:01.036513970 (77h14m0.907597045s) [noise=true , address=77.90.31.13:4627 , piece_id=XMI4HVR2MQQ5KHN4GQOUCKFYJYJ2YZEGQGTAAXYKGS3OI6XPJLVA, seg_pos={0 0}, upload=67501]: Uploading piece... Dec 8 20:43:02.093045104 (77h14m1.964128159s) [noise=true , address=77.90.31.21:4879 , piece_id=WF6YF2HQIXM26JI5TQ3YW7OXHNYZH3E2VTHAQVWIUHZ3ME3CCG6Q, seg_pos={0 0}, upload=67501]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: storage node overloaded, request limit: 2000 Dec 8 20:43:02.127847935 (77h14m1.99893098s) [noise=true , address=77.90.31.13:4627 , piece_id=XMI4HVR2MQQ5KHN4GQOUCKFYJYJ2YZEGQGTAAXYKGS3OI6XPJLVA, seg_pos={0 0}, upload=67501]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 8 20:43:02.156386568 (77h14m2.027469613s) [noise=true , address=77.90.31.12:4589 , piece_id=WUVA4G7EEN4J4HF47KQ7FZKFINAN2CCYDXMRNHDZVBF6EAPVFS4Q, seg_pos={0 0}, upload=67501]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 8 20:43:02.719927441 (77h14m2.591010486s) [upload=67501]: Next returned nil segment Dec 8 20:43:02.719942970 (77h14m2.591026005s) [upload=67501]: Waiting for error group managing segments... Dec 8 20:43:03.005012008 (77h14m2.876095063s) [noise=true , address=145.79.227.42:4123 , piece_id=NBMNEIGLWZHFGHY674WDGEDH4CGDCKYERXLB7UKXXI4MTZJ2HVEQ, seg_pos={0 0}, upload=67501]: Done uploading piece. err: <nil> Dec 8 20:43:03.012525866 (77h14m2.883608911s) [noise=true , address=145.79.227.41:4085 , piece_id=LS54ZFJC7DEGOKQRUSZOY3I665N43TNXICDJTXQ3ZJ62OHPGETSQ, seg_pos={0 0}, upload=67501]: Done uploading piece. err: <nil> Dec 8 20:43:03.015459803 (77h14m2.886542848s) [noise=true , address=145.79.227.41:4087 , piece_id=QYRY35JJKPQEK4BAVFZ6DEBKJN7SBUHG66S44ZA7XRNOVKAVQ33Q, seg_pos={0 0}, upload=67501]: Done uploading piece. err: <nil> Dec 8 20:43:03.020621582 (77h14m2.891704617s) [noise=true , address=145.79.227.20:4437 , piece_id=CCMQ7GX5NBRAUKHEHERGP5SHETIGELARKU5AE5ETKZWQRJP7OOAQ, seg_pos={0 0}, upload=67501]: Done uploading piece. err: <nil> Dec 8 20:43:03.020672098 (77h14m2.891755153s) [noise=true , address=145.79.227.31:4797 , piece_id=ECY3CM3GYNMM5RIEDHYI5NVWDCKEXRRKH4NCK3FV5ILW7W54LIIA, seg_pos={0 0}, upload=67501]: Done uploading piece. err: <nil> Dec 8 20:43:03.028139237 (77h14m2.899222282s) [noise=true , address=145.79.227.24:4583 , piece_id=DRLVD3TA2Z6TJQ4WKA67KDTXK7Q32YJH2ES5QEQG7Z37DYBJBB6Q, seg_pos={0 0}, upload=67501]: Done uploading piece. err: <nil> Dec 8 20:43:03.031290167 (77h14m2.902373202s) [noise=true , address=145.79.227.33:4870 , piece_id=KY2FLO3KDFPWRXPUUFINFLIRLZDMHPUCSPKRCN7RYPVAXXUYMXWQ, seg_pos={0 0}, upload=67501]: Done uploading piece. err: <nil> Dec 8 20:43:03.033756633 (77h14m2.904839668s) [noise=true , address=145.79.227.25:4618 , piece_id=AE7LMOAKNNIKVS4STMQJKZER7YUSFSUTSE7P5KXYTG4AQLFNQHLQ, seg_pos={0 0}, upload=67501]: Done uploading piece. err: <nil> Dec 8 20:43:03.039153600 (77h14m2.910236655s) [noise=true , address=145.79.227.40:4050 , piece_id=K2RW6IQERLFKEUD2ZTAZEX767U77PQWEXMXEH2SYDBQ7BT26WAHQ, seg_pos={0 0}, upload=67501]: Done uploading piece. err: <nil> Dec 8 20:43:03.040008869 (77h14m2.911091914s) [noise=true , address=145.79.227.38:4977 , piece_id=JW72CEO3WDGEB6BSSAR3M4Z5ANDBH63BLJZRSYAJDK7F7EXCEOZQ, seg_pos={0 0}, upload=67501]: Done uploading piece. err: <nil> Dec 8 20:43:03.178911548 (77h14m3.049994593s) [noise=true , address=77.90.31.18:4807 , piece_id=KDIWMU2XEJ7NTRCSG5J37ZUWIFURSJVK2BSDM7VJFKOB3ROANRGA, seg_pos={0 0}, upload=67501]: Done uploading piece. err: <nil> Dec 8 20:43:04.409564163 (77h14m4.280647208s) [noise=true , address=145.79.227.44:4195 , piece_id=EYY7B3AXUMA63GQLGKQWPWGDZUBORTXR6QFYSEOB26ATZK2GJC6Q, seg_pos={0 0}, upload=67501]: Done uploading piece. err: <nil> Dec 8 20:43:30.067753910 (77h14m29.938836955s) [noise=true , address=145.79.227.34:4906 , piece_id=VPDAFAMOLFSYUGFZIHRTADXWLYAE7IXILSPLT6QL2IPCOVUDPA3A, seg_pos={0 0}, upload=67501]: Done uploading piece. err: ecclient: upload failed (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw, address:145.79.227.34:4906): protocol: read tcp 188.166.225.170:36576->145.79.227.34:4906: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 20:43:30.067797914 (77h14m29.938880959s) [seg_pos={0 0}, upload=67501]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 8 20:43:30.323605733 (77h14m30.194688778s) [upload=67501]: Done uploading segments... Dec 8 20:43:30.323629097 (77h14m30.194712142s) [upload=67501]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.85829811s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)