Orbiter History

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 473.25385ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/4b87d2bbd73f899c FAIL (2025-12-08 20:40:57 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 473.296771ms core satellite eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............X.........X....................................] duration: 5.924479208s trace: https://jaeger.dev.storj.io/trace/6fb9d317d1dcf919 FAIL (2025-12-08 20:40:54 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.924517231s core uplink us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 5.01173397s trace: https://jaeger.dev.storj.io/trace/49fc2034659f6881 FAIL (2025-12-08 20:40:45 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.011766411s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.X.......................X....X....X........................] duration: 5.719697467s trace: https://jaeger.dev.storj.io/trace/4600f1753cc382a2 FAIL (2025-12-08 20:40:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.719711884s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [.......................X...............X....................] duration: 5.814169959s trace: https://jaeger.dev.storj.io/trace/3996915dd8f30735 FAIL (2025-12-08 20:40:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== PGM26LWTYTWGZGQDPBL7UDX263DFBXY7WZRXGMYEFPWXQQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.569231809 (76h54m34.440315054s) [noise=true , address=89.149.201.147:4022 , piece_id=2VR3E4SXAAOR5RAVHFVUQQ7MBZQEEQYFZ2ZW5Z7YSCVOETZNFQWQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.569369882 (76h54m34.440452917s) [noise=true , address=185.167.204.206:4037 , piece_id=XYY5NAATQJPHOK6JAMHXR3JH3QTLQF7OTA32BMST5HYQDYJ76UMA, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.569672879 (76h54m34.440755924s) [noise=true , address=185.167.204.206:4056 , piece_id=IPDW6RCHJFLMB5LOZM3FG645CCW6TT7KJW4PBQXBUSAIE2ZZM74A, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.569779642 (76h54m34.440862677s) [noise=true , address=185.167.204.205:4024 , piece_id=2UHV7KDAKJJ4DDUU5MPBNLRP24OK5RQU6GGAVYUV5MEFLDDPGUZQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.570073130 (76h54m34.441156176s) [noise=true , address=81.171.14.70:4035 , piece_id=ZKLJ5VUPALKZBSPFCJQBZZUTMJ3PX2H3XMUZ5JVSW5BJBACRUYHA, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.570428618 (76h54m34.441511653s) [noise=true , address=81.171.14.70:4043 , piece_id=FVUGWZXJLQAOLUSTRKJAULSHDWKX5GGWA2BB52YP7WQEKU4NKS5Q, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.570785237 (76h54m34.441868282s) [noise=true , address=81.171.14.70:4044 , piece_id=QPYQGR62JDWWV3GM3CVDB6BLNSGNUTQUEHHQF74TYXFN25PMNPVQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.571235034 (76h54m34.442318069s) [noise=true , address=185.167.204.205:4035 , piece_id=LXUP2T5SN5SSV7IWZLGTSLCRWLFRZBODXX7W4FFVHZFO74I2CQHQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.571655133 (76h54m34.442738169s) [noise=true , address=81.171.14.70:4031 , piece_id=53QOUYYU4KS53TELJUM3FPEKOCB464OR5GKYGYB54RVRPD52RFWA, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.572102666 (76h54m34.443185701s) [noise=true , address=185.167.204.205:4034 , piece_id=4VH6HEDD7CTPMEOHYGKFXSYXHKOLPYB2N6BCCJT2DXVAMADI3KJQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:35.395419290 (76h54m35.266502335s) [upload=67229]: Next returned nil segment Dec 8 20:23:35.395432614 (76h54m35.26651565s) [upload=67229]: Waiting for error group managing segments... Dec 8 20:23:35.780649683 (76h54m35.651732718s) [noise=true , address=185.167.204.206:4037 , piece_id=XYY5NAATQJPHOK6JAMHXR3JH3QTLQF7OTA32BMST5HYQDYJ76UMA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.781236982 (76h54m35.652320027s) [noise=true , address=185.167.204.206:4056 , piece_id=IPDW6RCHJFLMB5LOZM3FG645CCW6TT7KJW4PBQXBUSAIE2ZZM74A, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.784267011 (76h54m35.655350046s) [noise=true , address=185.167.204.205:4030 , piece_id=O3QKGK2G5I5YYPRP5K57RTG4H35H2UDFROFU2JMSHKTFNZJDVUIA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.785164540 (76h54m35.656247585s) [noise=true , address=185.167.204.205:4012 , piece_id=QCJD2IHH4DEKTFN5ZPMZXPECBQ2IX2QPSFUZCSFR5HKCDOVYGGQA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.785585301 (76h54m35.656668336s) [noise=true , address=185.167.204.205:4002 , piece_id=PDSJ5KDCFQBURILRTWU2WSEG3UH7GX4GK57W4J75QADBPKQXRSYA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.787925598 (76h54m35.659008643s) [noise=true , address=185.167.204.206:4047 , piece_id=THE6CUP46D2OSXS34UR3TTBUP6FIVMGDGT7DVJDBB57BHMWBCZ7A, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.787994289 (76h54m35.659077324s) [noise=true , address=185.167.204.205:4034 , piece_id=4VH6HEDD7CTPMEOHYGKFXSYXHKOLPYB2N6BCCJT2DXVAMADI3KJQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.788899402 (76h54m35.659982437s) [noise=true , address=185.167.204.206:4050 , piece_id=7B4GLDQZUPQPLGPZPPN5CO3VOX3LLBMLIYA6IMALYITCHE4CZBOA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.791033055 (76h54m35.66211609s) [noise=true , address=185.167.204.205:4031 , piece_id=5XPLPLO4SPMX7ZBYHLUZTH7MBGCRESEEMP2OEI7LTTT5BN67XHMQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.792163116 (76h54m35.663246161s) [noise=true , address=185.167.204.206:4055 , piece_id=JESF36BPW3YEPXVI2GP4OALECNLYSASJHQRZVLNXKV5BQRDGOWWQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.792693356 (76h54m35.663776392s) [noise=true , address=185.167.204.206:4063 , piece_id=BRUAUJTOCPWNUOB6D5G6AJCTMJBRZRJXXXBZBIUUZZXXQB5OEX2Q, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.793171606 (76h54m35.664254662s) [noise=true , address=185.167.204.206:4045 , piece_id=IXV6HTIOVQR4HCGJIWTBQFLO733BMMNNMZ24Y34H6V6KYUQYPQRA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.798151289 (76h54m35.669234344s) [noise=true , address=185.167.204.206:4053 , piece_id=VGK5AKCI23P5ZDS427NS6PNNRIJ3QUXLRZ4ECLYN4UDSL3KAQAKA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.798189562 (76h54m35.669272597s) [noise=true , address=185.167.204.206:4040 , piece_id=4KFICIELLLPCLWQ7RIB7UUSWUK7SFX2PJK7JWCRU5S4HQNHBEWRQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.798633607 (76h54m35.669716642s) [noise=true , address=185.167.204.205:4035 , piece_id=LXUP2T5SN5SSV7IWZLGTSLCRWLFRZBODXX7W4FFVHZFO74I2CQHQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.803059726 (76h54m35.674142761s) [noise=true , address=185.167.204.205:4024 , piece_id=2UHV7KDAKJJ4DDUU5MPBNLRP24OK5RQU6GGAVYUV5MEFLDDPGUZQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.344154907 (76h54m36.215237952s) [noise=true , address=81.171.14.70:4036 , piece_id=FB3SB3XQDNT2HSK5FDGELGYWODFLU655QW4D23GEXIWQEZYFNFRA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.359261713 (76h54m36.230344758s) [noise=true , address=81.171.14.70:4031 , piece_id=53QOUYYU4KS53TELJUM3FPEKOCB464OR5GKYGYB54RVRPD52RFWA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.393865115 (76h54m36.26494815s) [noise=true , address=81.171.14.70:4035 , piece_id=ZKLJ5VUPALKZBSPFCJQBZZUTMJ3PX2H3XMUZ5JVSW5BJBACRUYHA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.394302157 (76h54m36.265385202s) [noise=true , address=81.171.14.70:4044 , piece_id=QPYQGR62JDWWV3GM3CVDB6BLNSGNUTQUEHHQF74TYXFN25PMNPVQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.430200674 (76h54m36.301283709s) [noise=true , address=81.171.14.70:4046 , piece_id=ZHZUULNXPFDSTOXXH6T7XIEPN23OOWCNAAUWPCZYNNDTLPRWXVKQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.800799297 (76h54m36.671882332s) [noise=true , address=81.171.14.70:4040 , piece_id=W2SOTP5SPLQ6J3PM57NZOLRVZNMTNSGPWDZKZC2KV2PBPH4TUBLA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.862864259 (76h54m36.733947295s) [noise=true , address=81.171.14.70:4038 , piece_id=CKN2RF2YSTEOK6HUXGVBVECYAP2CLGUF6KYNR5JTHRUZPTGVI73Q, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.953769972 (76h54m36.824853007s) [noise=true , address=81.171.14.70:4043 , piece_id=FVUGWZXJLQAOLUSTRKJAULSHDWKX5GGWA2BB52YP7WQEKU4NKS5Q, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.996603376 (76h54m36.867686421s) [noise=true , address=89.149.201.147:4006 , piece_id=UGREZBZRVKXMGBOLQEV2NC4COTXOTTRUEAGSPNWRWP22MCPHQP7Q, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:37.037518916 (76h54m36.908601951s) [noise=true , address=89.149.201.147:4014 , piece_id=AEG6KGPGM26LWTYTWGZGQDPBL7UDX263DFBXY7WZRXGMYEFPWXQQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:37.037537541 (76h54m36.908620586s) [seg_pos={0 0}, upload=67229]: Segment reached optimal threshold of 26 pieces. Dec 8 20:23:37.037730880 (76h54m36.908813915s) [noise=true , address=89.149.201.147:4023 , piece_id=5EPGXWBK6GNTKLQLWEIWYBC2OJHFR3M7F4X3X4WODJWNXID3PCQQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BaeSxjXneBKoGKVrRM686sq386Bhn1YzzBFG9we8fHbcc9tbv): protocol: expected piece hash; context canceled; EOF Dec 8 20:23:37.037737452 (76h54m36.908820497s) [noise=true , address=89.149.201.147:4023 , piece_id=5EPGXWBK6GNTKLQLWEIWYBC2OJHFR3M7F4X3X4WODJWNXID3PCQQ, seg_pos={0 0}, upload=67229]: Long tail cancellation due to optimal threshold Dec 8 20:23:37.037893039 (76h54m36.908976074s) [noise=true , address=89.149.201.147:4022 , piece_id=2VR3E4SXAAOR5RAVHFVUQQ7MBZQEEQYFZ2ZW5Z7YSCVOETZNFQWQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122Vj8XxpS6RnhVweQV3cZWFhj8nrzy3hZ9P7JsuTzEaZcD6iFE): protocol: expected piece hash; context canceled; EOF Dec 8 20:23:37.037900964 (76h54m36.908984019s) [noise=true , address=89.149.201.147:4022 , piece_id=2VR3E4SXAAOR5RAVHFVUQQ7MBZQEEQYFZ2ZW5Z7YSCVOETZNFQWQ, seg_pos={0 0}, upload=67229]: Long tail cancellation due to optimal threshold Dec 8 20:23:37.037926843 (76h54m36.909009889s) [seg_pos={0 0}, upload=67229]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 8 20:23:37.038007697 (76h54m36.909090732s) [upload=67229]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 20:23:37.038013398 (76h54m36.909096433s) [upload=67229]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 20:26:49.742746371 (76h57m49.613829416s) [upload=67229]: Done uploading segments... Dec 8 20:26:49.742860908 (76h57m49.613943943s) [upload=67229]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.81419644s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 5.907627557s trace: https://jaeger.dev.storj.io/trace/690c0830f44a9942 FAIL (2025-12-08 20:40:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 20:13:16.312406346 (76h44m16.183489391s) [upload=67101]: Starting upload Dec 8 20:13:16.312474916 (76h44m16.183557952s) [upload=67101]: Done starting upload Dec 8 20:13:16.312661411 (76h44m16.183744446s) [upload=67101]: Uploading segments... Dec 8 20:13:16.313579429 (76h44m16.184662474s) [upload=67101]: Got next segment. Inline: false Dec 8 20:13:16.313623854 (76h44m16.184706889s) [upload=67101]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 20:13:16.313629435 (76h44m16.18471247s) [upload=67101]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 20:13:18.037990476 (76h44m17.909073521s) [seg_pos={0 0}, upload=67101]: Begin upload segment... Dec 8 20:13:18.038044659 (76h44m17.909127694s) [seg_pos={0 0}, upload=67101]: Done begin upload segment. Dec 8 20:13:18.038075859 (76h44m17.909158914s) [noise=true , address=145.79.227.31:4797 , piece_id=IHEVQ2JUPASTR3CU7QMNAQVSYQ65LEV5SR2KO4GCEOKAKQR5Y5NA, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.038191428 (76h44m17.909274463s) [noise=true , address=145.79.227.41:4085 , piece_id=FFFFOS7PH3SN3S4NWJJRYP772D725TSGCBKXIZ7FTEDBPO3TNR3Q, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.038908023 (76h44m17.909991068s) [noise=true , address=145.79.227.24:4583 , piece_id=MTNWT2AOTPYXLNVPSOKK5ZOAY7U3UH376V232TZKE56UM2CMDVAQ, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.039373890 (76h44m17.910456925s) [noise=true , address=145.79.227.29:4942 , piece_id=ZYT75AZ7X56NBH3OVLWL27J5FRRN4WKV5VEIKEXN32E6P5OQNOZQ, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.039816273 (76h44m17.910899318s) [noise=true , address=145.79.227.42:4121 , piece_id=ICWYGS3AGWC47JS4PUC4GHT2YOV5UTD5RYL73JFTWZRNEMAGMRZA, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.040292149 (76h44m17.911375194s) [noise=true , address=145.79.227.32:4834 , piece_id=WDZ3N3A3CNPAECT7HW2VZ2K7JEJVOZJCZR3BKHRBVR3XHGPGTKPQ, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.040745432 (76h44m17.911828477s) [noise=true , address=145.79.227.31:4799 , piece_id=RNPR7HC6YANTTQW7STMNTHYANE7ZD5XC2N5DMLQQ5YBULNUSFK7A, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.041266104 (76h44m17.912349139s) [noise=true , address=145.79.227.20:4437 , piece_id=G76GCXUI5YH3CMQV47V6FSFDDSOBIIFMS24UJVTNXRR2MM4UZ5ZQ, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.041787887 (76h44m17.912870952s) [noise=true , address=145.79.227.25:4619 , piece_id=5M5PTNON4YZ7INE34X7I6BIJVQF543VQWALEZBCZEEEQ26UMGYQA, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.042946814 (76h44m17.914029849s) [noise=true , address=145.79.227.16:4912 , piece_id=J7BVXTHOSEPXKDDWXPV7JIUWW5NNWLETI7S4IOVPDG6ZUVPATOXQ, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.044580184 (76h44m17.915663219s) [noise=true , address=145.79.227.27:4691 , piece_id=R5VVIJNPYZIDQW5WSJMEYS4D55AHXXMACQY5GM5IOJXVKNOGBEAQ, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.047199231 (76h44m17.918282266s) [noise=true , address=145.79.227.16:4910 , piece_id=YIHYKALCHWYGMAUKS3RPUQ2WOAQKCRQV6GZPFM27K37CKTTJSDPA, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.049620632 (76h44m17.920703677s) [noise=true , address=145.79.227.20:4438 , piece_id=QZQULVSGCAOTQEE5MBUONJYAJIO3GPUNKB4CEVMK3NA2J3BYZ6GA, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.051625078 (76h44m17.922708143s) [noise=true , address=145.79.227.22:4511 , piece_id=CFUDAP3RXMYJCS5GQJVYCKXTN6RLXG7NSQH6EHTZPSYCXPSPBEFQ, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.053186752 (76h44m17.924269787s) [noise=true , address=145.79.227.39:4015 , piece_id=UX4HGNQIHC3FCNGUGIAZYPNYYTLNXMYIND7H7PZ6DCD5EXS67Y7Q, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.054291115 (76h44m17.92537415s) [noise=true , address=77.90.31.10:4517 , piece_id=BMXTXAXNYZ5U5T54U57PMGP4KZO332K2VEP4KO5NEYC7TZ2S53SA, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:19.111879942 (76h44m18.982962987s) [noise=true , address=77.90.31.10:4517 , piece_id=BMXTXAXNYZ5U5T54U57PMGP4KZO332K2VEP4KO5NEYC7TZ2S53SA, seg_pos={0 0}, upload=67101]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: storage node overloaded, request limit: 2000 Dec 8 20:13:19.643615670 (76h44m19.514698706s) [upload=67101]: Next returned nil segment Dec 8 20:13:19.643629617 (76h44m19.514712652s) [upload=67101]: Waiting for error group managing segments... Dec 8 20:13:20.026246645 (76h44m19.89732969s) [noise=true , address=145.79.227.32:4834 , piece_id=WDZ3N3A3CNPAECT7HW2VZ2K7JEJVOZJCZR3BKHRBVR3XHGPGTKPQ, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.027742514 (76h44m19.898825559s) [noise=true , address=145.79.227.25:4619 , piece_id=5M5PTNON4YZ7INE34X7I6BIJVQF543VQWALEZBCZEEEQ26UMGYQA, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.028492581 (76h44m19.899575616s) [noise=true , address=145.79.227.41:4085 , piece_id=FFFFOS7PH3SN3S4NWJJRYP772D725TSGCBKXIZ7FTEDBPO3TNR3Q, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.029004767 (76h44m19.900087802s) [noise=true , address=145.79.227.16:4912 , piece_id=J7BVXTHOSEPXKDDWXPV7JIUWW5NNWLETI7S4IOVPDG6ZUVPATOXQ, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.029438392 (76h44m19.900521427s) [noise=true , address=145.79.227.24:4583 , piece_id=MTNWT2AOTPYXLNVPSOKK5ZOAY7U3UH376V232TZKE56UM2CMDVAQ, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.031410738 (76h44m19.902493783s) [noise=true , address=145.79.227.42:4121 , piece_id=ICWYGS3AGWC47JS4PUC4GHT2YOV5UTD5RYL73JFTWZRNEMAGMRZA, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.034141027 (76h44m19.905224072s) [noise=true , address=145.79.227.31:4799 , piece_id=RNPR7HC6YANTTQW7STMNTHYANE7ZD5XC2N5DMLQQ5YBULNUSFK7A, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.037360508 (76h44m19.908443543s) [noise=true , address=145.79.227.22:4511 , piece_id=CFUDAP3RXMYJCS5GQJVYCKXTN6RLXG7NSQH6EHTZPSYCXPSPBEFQ, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.038059108 (76h44m19.909142153s) [noise=true , address=145.79.227.39:4015 , piece_id=UX4HGNQIHC3FCNGUGIAZYPNYYTLNXMYIND7H7PZ6DCD5EXS67Y7Q, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.053205520 (76h44m19.924288565s) [noise=true , address=145.79.227.16:4910 , piece_id=YIHYKALCHWYGMAUKS3RPUQ2WOAQKCRQV6GZPFM27K37CKTTJSDPA, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.053898059 (76h44m19.924981114s) [noise=true , address=145.79.227.20:4437 , piece_id=G76GCXUI5YH3CMQV47V6FSFDDSOBIIFMS24UJVTNXRR2MM4UZ5ZQ, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.054182971 (76h44m19.925266006s) [noise=true , address=145.79.227.27:4691 , piece_id=R5VVIJNPYZIDQW5WSJMEYS4D55AHXXMACQY5GM5IOJXVKNOGBEAQ, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.058658112 (76h44m19.929741157s) [noise=true , address=145.79.227.29:4942 , piece_id=ZYT75AZ7X56NBH3OVLWL27J5FRRN4WKV5VEIKEXN32E6P5OQNOZQ, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.058679072 (76h44m19.929762107s) [seg_pos={0 0}, upload=67101]: Segment reached optimal threshold of 13 pieces. Dec 8 20:13:20.058873853 (76h44m19.929956888s) [noise=true , address=145.79.227.31:4797 , piece_id=IHEVQ2JUPASTR3CU7QMNAQVSYQ65LEV5SR2KO4GCEOKAKQR5Y5NA, seg_pos={0 0}, upload=67101]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9): protocol: expected piece hash; context canceled; EOF Dec 8 20:13:20.058880526 (76h44m19.929963561s) [noise=true , address=145.79.227.31:4797 , piece_id=IHEVQ2JUPASTR3CU7QMNAQVSYQ65LEV5SR2KO4GCEOKAKQR5Y5NA, seg_pos={0 0}, upload=67101]: Long tail cancellation due to optimal threshold Dec 8 20:13:20.059077551 (76h44m19.930160586s) [noise=true , address=145.79.227.20:4438 , piece_id=QZQULVSGCAOTQEE5MBUONJYAJIO3GPUNKB4CEVMK3NA2J3BYZ6GA, seg_pos={0 0}, upload=67101]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT): protocol: expected piece hash; context canceled; EOF Dec 8 20:13:20.059093532 (76h44m19.930176567s) [noise=true , address=145.79.227.20:4438 , piece_id=QZQULVSGCAOTQEE5MBUONJYAJIO3GPUNKB4CEVMK3NA2J3BYZ6GA, seg_pos={0 0}, upload=67101]: Long tail cancellation due to optimal threshold Dec 8 20:13:20.059116485 (76h44m19.93019953s) [seg_pos={0 0}, upload=67101]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 20:13:20.059191879 (76h44m19.930274924s) [upload=67101]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 20:13:20.059196798 (76h44m19.930279833s) [upload=67101]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 20:19:41.003042505 (76h50m40.87412554s) [upload=67101]: Done uploading segments... Dec 8 20:19:41.003155129 (76h50m40.874238175s) [upload=67101]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.90767646s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............X.........X...................................] duration: 5.800180948s trace: https://jaeger.dev.storj.io/trace/6fb9d317d1dcf919 FAIL (2025-12-08 20:40:18 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.800216345s core uplink us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [........................X...............X...................] duration: 5.226157693s trace: https://jaeger.dev.storj.io/trace/3996915dd8f30735 FAIL (2025-12-08 20:39:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== PGM26LWTYTWGZGQDPBL7UDX263DFBXY7WZRXGMYEFPWXQQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.569231809 (76h54m34.440315054s) [noise=true , address=89.149.201.147:4022 , piece_id=2VR3E4SXAAOR5RAVHFVUQQ7MBZQEEQYFZ2ZW5Z7YSCVOETZNFQWQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.569369882 (76h54m34.440452917s) [noise=true , address=185.167.204.206:4037 , piece_id=XYY5NAATQJPHOK6JAMHXR3JH3QTLQF7OTA32BMST5HYQDYJ76UMA, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.569672879 (76h54m34.440755924s) [noise=true , address=185.167.204.206:4056 , piece_id=IPDW6RCHJFLMB5LOZM3FG645CCW6TT7KJW4PBQXBUSAIE2ZZM74A, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.569779642 (76h54m34.440862677s) [noise=true , address=185.167.204.205:4024 , piece_id=2UHV7KDAKJJ4DDUU5MPBNLRP24OK5RQU6GGAVYUV5MEFLDDPGUZQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.570073130 (76h54m34.441156176s) [noise=true , address=81.171.14.70:4035 , piece_id=ZKLJ5VUPALKZBSPFCJQBZZUTMJ3PX2H3XMUZ5JVSW5BJBACRUYHA, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.570428618 (76h54m34.441511653s) [noise=true , address=81.171.14.70:4043 , piece_id=FVUGWZXJLQAOLUSTRKJAULSHDWKX5GGWA2BB52YP7WQEKU4NKS5Q, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.570785237 (76h54m34.441868282s) [noise=true , address=81.171.14.70:4044 , piece_id=QPYQGR62JDWWV3GM3CVDB6BLNSGNUTQUEHHQF74TYXFN25PMNPVQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.571235034 (76h54m34.442318069s) [noise=true , address=185.167.204.205:4035 , piece_id=LXUP2T5SN5SSV7IWZLGTSLCRWLFRZBODXX7W4FFVHZFO74I2CQHQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.571655133 (76h54m34.442738169s) [noise=true , address=81.171.14.70:4031 , piece_id=53QOUYYU4KS53TELJUM3FPEKOCB464OR5GKYGYB54RVRPD52RFWA, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.572102666 (76h54m34.443185701s) [noise=true , address=185.167.204.205:4034 , piece_id=4VH6HEDD7CTPMEOHYGKFXSYXHKOLPYB2N6BCCJT2DXVAMADI3KJQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:35.395419290 (76h54m35.266502335s) [upload=67229]: Next returned nil segment Dec 8 20:23:35.395432614 (76h54m35.26651565s) [upload=67229]: Waiting for error group managing segments... Dec 8 20:23:35.780649683 (76h54m35.651732718s) [noise=true , address=185.167.204.206:4037 , piece_id=XYY5NAATQJPHOK6JAMHXR3JH3QTLQF7OTA32BMST5HYQDYJ76UMA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.781236982 (76h54m35.652320027s) [noise=true , address=185.167.204.206:4056 , piece_id=IPDW6RCHJFLMB5LOZM3FG645CCW6TT7KJW4PBQXBUSAIE2ZZM74A, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.784267011 (76h54m35.655350046s) [noise=true , address=185.167.204.205:4030 , piece_id=O3QKGK2G5I5YYPRP5K57RTG4H35H2UDFROFU2JMSHKTFNZJDVUIA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.785164540 (76h54m35.656247585s) [noise=true , address=185.167.204.205:4012 , piece_id=QCJD2IHH4DEKTFN5ZPMZXPECBQ2IX2QPSFUZCSFR5HKCDOVYGGQA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.785585301 (76h54m35.656668336s) [noise=true , address=185.167.204.205:4002 , piece_id=PDSJ5KDCFQBURILRTWU2WSEG3UH7GX4GK57W4J75QADBPKQXRSYA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.787925598 (76h54m35.659008643s) [noise=true , address=185.167.204.206:4047 , piece_id=THE6CUP46D2OSXS34UR3TTBUP6FIVMGDGT7DVJDBB57BHMWBCZ7A, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.787994289 (76h54m35.659077324s) [noise=true , address=185.167.204.205:4034 , piece_id=4VH6HEDD7CTPMEOHYGKFXSYXHKOLPYB2N6BCCJT2DXVAMADI3KJQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.788899402 (76h54m35.659982437s) [noise=true , address=185.167.204.206:4050 , piece_id=7B4GLDQZUPQPLGPZPPN5CO3VOX3LLBMLIYA6IMALYITCHE4CZBOA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.791033055 (76h54m35.66211609s) [noise=true , address=185.167.204.205:4031 , piece_id=5XPLPLO4SPMX7ZBYHLUZTH7MBGCRESEEMP2OEI7LTTT5BN67XHMQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.792163116 (76h54m35.663246161s) [noise=true , address=185.167.204.206:4055 , piece_id=JESF36BPW3YEPXVI2GP4OALECNLYSASJHQRZVLNXKV5BQRDGOWWQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.792693356 (76h54m35.663776392s) [noise=true , address=185.167.204.206:4063 , piece_id=BRUAUJTOCPWNUOB6D5G6AJCTMJBRZRJXXXBZBIUUZZXXQB5OEX2Q, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.793171606 (76h54m35.664254662s) [noise=true , address=185.167.204.206:4045 , piece_id=IXV6HTIOVQR4HCGJIWTBQFLO733BMMNNMZ24Y34H6V6KYUQYPQRA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.798151289 (76h54m35.669234344s) [noise=true , address=185.167.204.206:4053 , piece_id=VGK5AKCI23P5ZDS427NS6PNNRIJ3QUXLRZ4ECLYN4UDSL3KAQAKA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.798189562 (76h54m35.669272597s) [noise=true , address=185.167.204.206:4040 , piece_id=4KFICIELLLPCLWQ7RIB7UUSWUK7SFX2PJK7JWCRU5S4HQNHBEWRQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.798633607 (76h54m35.669716642s) [noise=true , address=185.167.204.205:4035 , piece_id=LXUP2T5SN5SSV7IWZLGTSLCRWLFRZBODXX7W4FFVHZFO74I2CQHQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.803059726 (76h54m35.674142761s) [noise=true , address=185.167.204.205:4024 , piece_id=2UHV7KDAKJJ4DDUU5MPBNLRP24OK5RQU6GGAVYUV5MEFLDDPGUZQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.344154907 (76h54m36.215237952s) [noise=true , address=81.171.14.70:4036 , piece_id=FB3SB3XQDNT2HSK5FDGELGYWODFLU655QW4D23GEXIWQEZYFNFRA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.359261713 (76h54m36.230344758s) [noise=true , address=81.171.14.70:4031 , piece_id=53QOUYYU4KS53TELJUM3FPEKOCB464OR5GKYGYB54RVRPD52RFWA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.393865115 (76h54m36.26494815s) [noise=true , address=81.171.14.70:4035 , piece_id=ZKLJ5VUPALKZBSPFCJQBZZUTMJ3PX2H3XMUZ5JVSW5BJBACRUYHA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.394302157 (76h54m36.265385202s) [noise=true , address=81.171.14.70:4044 , piece_id=QPYQGR62JDWWV3GM3CVDB6BLNSGNUTQUEHHQF74TYXFN25PMNPVQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.430200674 (76h54m36.301283709s) [noise=true , address=81.171.14.70:4046 , piece_id=ZHZUULNXPFDSTOXXH6T7XIEPN23OOWCNAAUWPCZYNNDTLPRWXVKQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.800799297 (76h54m36.671882332s) [noise=true , address=81.171.14.70:4040 , piece_id=W2SOTP5SPLQ6J3PM57NZOLRVZNMTNSGPWDZKZC2KV2PBPH4TUBLA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.862864259 (76h54m36.733947295s) [noise=true , address=81.171.14.70:4038 , piece_id=CKN2RF2YSTEOK6HUXGVBVECYAP2CLGUF6KYNR5JTHRUZPTGVI73Q, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.953769972 (76h54m36.824853007s) [noise=true , address=81.171.14.70:4043 , piece_id=FVUGWZXJLQAOLUSTRKJAULSHDWKX5GGWA2BB52YP7WQEKU4NKS5Q, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.996603376 (76h54m36.867686421s) [noise=true , address=89.149.201.147:4006 , piece_id=UGREZBZRVKXMGBOLQEV2NC4COTXOTTRUEAGSPNWRWP22MCPHQP7Q, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:37.037518916 (76h54m36.908601951s) [noise=true , address=89.149.201.147:4014 , piece_id=AEG6KGPGM26LWTYTWGZGQDPBL7UDX263DFBXY7WZRXGMYEFPWXQQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:37.037537541 (76h54m36.908620586s) [seg_pos={0 0}, upload=67229]: Segment reached optimal threshold of 26 pieces. Dec 8 20:23:37.037730880 (76h54m36.908813915s) [noise=true , address=89.149.201.147:4023 , piece_id=5EPGXWBK6GNTKLQLWEIWYBC2OJHFR3M7F4X3X4WODJWNXID3PCQQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BaeSxjXneBKoGKVrRM686sq386Bhn1YzzBFG9we8fHbcc9tbv): protocol: expected piece hash; context canceled; EOF Dec 8 20:23:37.037737452 (76h54m36.908820497s) [noise=true , address=89.149.201.147:4023 , piece_id=5EPGXWBK6GNTKLQLWEIWYBC2OJHFR3M7F4X3X4WODJWNXID3PCQQ, seg_pos={0 0}, upload=67229]: Long tail cancellation due to optimal threshold Dec 8 20:23:37.037893039 (76h54m36.908976074s) [noise=true , address=89.149.201.147:4022 , piece_id=2VR3E4SXAAOR5RAVHFVUQQ7MBZQEEQYFZ2ZW5Z7YSCVOETZNFQWQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122Vj8XxpS6RnhVweQV3cZWFhj8nrzy3hZ9P7JsuTzEaZcD6iFE): protocol: expected piece hash; context canceled; EOF Dec 8 20:23:37.037900964 (76h54m36.908984019s) [noise=true , address=89.149.201.147:4022 , piece_id=2VR3E4SXAAOR5RAVHFVUQQ7MBZQEEQYFZ2ZW5Z7YSCVOETZNFQWQ, seg_pos={0 0}, upload=67229]: Long tail cancellation due to optimal threshold Dec 8 20:23:37.037926843 (76h54m36.909009889s) [seg_pos={0 0}, upload=67229]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 8 20:23:37.038007697 (76h54m36.909090732s) [upload=67229]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 20:23:37.038013398 (76h54m36.909096433s) [upload=67229]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 20:26:49.742746371 (76h57m49.613829416s) [upload=67229]: Done uploading segments... Dec 8 20:26:49.742860908 (76h57m49.613943943s) [upload=67229]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.226185647s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [..X.......................X....X....X.......................] duration: 5.939548139s trace: https://jaeger.dev.storj.io/trace/4600f1753cc382a2 FAIL (2025-12-08 20:39:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.939584238s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 473.481131ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/4b87d2bbd73f899c FAIL (2025-12-08 20:39:48 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 473.51194ms core satellite eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 5.781497141s trace: https://jaeger.dev.storj.io/trace/690c0830f44a9942 FAIL (2025-12-08 20:39:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 20:13:16.312406346 (76h44m16.183489391s) [upload=67101]: Starting upload Dec 8 20:13:16.312474916 (76h44m16.183557952s) [upload=67101]: Done starting upload Dec 8 20:13:16.312661411 (76h44m16.183744446s) [upload=67101]: Uploading segments... Dec 8 20:13:16.313579429 (76h44m16.184662474s) [upload=67101]: Got next segment. Inline: false Dec 8 20:13:16.313623854 (76h44m16.184706889s) [upload=67101]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 20:13:16.313629435 (76h44m16.18471247s) [upload=67101]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 20:13:18.037990476 (76h44m17.909073521s) [seg_pos={0 0}, upload=67101]: Begin upload segment... Dec 8 20:13:18.038044659 (76h44m17.909127694s) [seg_pos={0 0}, upload=67101]: Done begin upload segment. Dec 8 20:13:18.038075859 (76h44m17.909158914s) [noise=true , address=145.79.227.31:4797 , piece_id=IHEVQ2JUPASTR3CU7QMNAQVSYQ65LEV5SR2KO4GCEOKAKQR5Y5NA, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.038191428 (76h44m17.909274463s) [noise=true , address=145.79.227.41:4085 , piece_id=FFFFOS7PH3SN3S4NWJJRYP772D725TSGCBKXIZ7FTEDBPO3TNR3Q, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.038908023 (76h44m17.909991068s) [noise=true , address=145.79.227.24:4583 , piece_id=MTNWT2AOTPYXLNVPSOKK5ZOAY7U3UH376V232TZKE56UM2CMDVAQ, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.039373890 (76h44m17.910456925s) [noise=true , address=145.79.227.29:4942 , piece_id=ZYT75AZ7X56NBH3OVLWL27J5FRRN4WKV5VEIKEXN32E6P5OQNOZQ, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.039816273 (76h44m17.910899318s) [noise=true , address=145.79.227.42:4121 , piece_id=ICWYGS3AGWC47JS4PUC4GHT2YOV5UTD5RYL73JFTWZRNEMAGMRZA, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.040292149 (76h44m17.911375194s) [noise=true , address=145.79.227.32:4834 , piece_id=WDZ3N3A3CNPAECT7HW2VZ2K7JEJVOZJCZR3BKHRBVR3XHGPGTKPQ, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.040745432 (76h44m17.911828477s) [noise=true , address=145.79.227.31:4799 , piece_id=RNPR7HC6YANTTQW7STMNTHYANE7ZD5XC2N5DMLQQ5YBULNUSFK7A, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.041266104 (76h44m17.912349139s) [noise=true , address=145.79.227.20:4437 , piece_id=G76GCXUI5YH3CMQV47V6FSFDDSOBIIFMS24UJVTNXRR2MM4UZ5ZQ, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.041787887 (76h44m17.912870952s) [noise=true , address=145.79.227.25:4619 , piece_id=5M5PTNON4YZ7INE34X7I6BIJVQF543VQWALEZBCZEEEQ26UMGYQA, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.042946814 (76h44m17.914029849s) [noise=true , address=145.79.227.16:4912 , piece_id=J7BVXTHOSEPXKDDWXPV7JIUWW5NNWLETI7S4IOVPDG6ZUVPATOXQ, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.044580184 (76h44m17.915663219s) [noise=true , address=145.79.227.27:4691 , piece_id=R5VVIJNPYZIDQW5WSJMEYS4D55AHXXMACQY5GM5IOJXVKNOGBEAQ, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.047199231 (76h44m17.918282266s) [noise=true , address=145.79.227.16:4910 , piece_id=YIHYKALCHWYGMAUKS3RPUQ2WOAQKCRQV6GZPFM27K37CKTTJSDPA, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.049620632 (76h44m17.920703677s) [noise=true , address=145.79.227.20:4438 , piece_id=QZQULVSGCAOTQEE5MBUONJYAJIO3GPUNKB4CEVMK3NA2J3BYZ6GA, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.051625078 (76h44m17.922708143s) [noise=true , address=145.79.227.22:4511 , piece_id=CFUDAP3RXMYJCS5GQJVYCKXTN6RLXG7NSQH6EHTZPSYCXPSPBEFQ, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.053186752 (76h44m17.924269787s) [noise=true , address=145.79.227.39:4015 , piece_id=UX4HGNQIHC3FCNGUGIAZYPNYYTLNXMYIND7H7PZ6DCD5EXS67Y7Q, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.054291115 (76h44m17.92537415s) [noise=true , address=77.90.31.10:4517 , piece_id=BMXTXAXNYZ5U5T54U57PMGP4KZO332K2VEP4KO5NEYC7TZ2S53SA, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:19.111879942 (76h44m18.982962987s) [noise=true , address=77.90.31.10:4517 , piece_id=BMXTXAXNYZ5U5T54U57PMGP4KZO332K2VEP4KO5NEYC7TZ2S53SA, seg_pos={0 0}, upload=67101]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: storage node overloaded, request limit: 2000 Dec 8 20:13:19.643615670 (76h44m19.514698706s) [upload=67101]: Next returned nil segment Dec 8 20:13:19.643629617 (76h44m19.514712652s) [upload=67101]: Waiting for error group managing segments... Dec 8 20:13:20.026246645 (76h44m19.89732969s) [noise=true , address=145.79.227.32:4834 , piece_id=WDZ3N3A3CNPAECT7HW2VZ2K7JEJVOZJCZR3BKHRBVR3XHGPGTKPQ, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.027742514 (76h44m19.898825559s) [noise=true , address=145.79.227.25:4619 , piece_id=5M5PTNON4YZ7INE34X7I6BIJVQF543VQWALEZBCZEEEQ26UMGYQA, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.028492581 (76h44m19.899575616s) [noise=true , address=145.79.227.41:4085 , piece_id=FFFFOS7PH3SN3S4NWJJRYP772D725TSGCBKXIZ7FTEDBPO3TNR3Q, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.029004767 (76h44m19.900087802s) [noise=true , address=145.79.227.16:4912 , piece_id=J7BVXTHOSEPXKDDWXPV7JIUWW5NNWLETI7S4IOVPDG6ZUVPATOXQ, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.029438392 (76h44m19.900521427s) [noise=true , address=145.79.227.24:4583 , piece_id=MTNWT2AOTPYXLNVPSOKK5ZOAY7U3UH376V232TZKE56UM2CMDVAQ, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.031410738 (76h44m19.902493783s) [noise=true , address=145.79.227.42:4121 , piece_id=ICWYGS3AGWC47JS4PUC4GHT2YOV5UTD5RYL73JFTWZRNEMAGMRZA, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.034141027 (76h44m19.905224072s) [noise=true , address=145.79.227.31:4799 , piece_id=RNPR7HC6YANTTQW7STMNTHYANE7ZD5XC2N5DMLQQ5YBULNUSFK7A, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.037360508 (76h44m19.908443543s) [noise=true , address=145.79.227.22:4511 , piece_id=CFUDAP3RXMYJCS5GQJVYCKXTN6RLXG7NSQH6EHTZPSYCXPSPBEFQ, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.038059108 (76h44m19.909142153s) [noise=true , address=145.79.227.39:4015 , piece_id=UX4HGNQIHC3FCNGUGIAZYPNYYTLNXMYIND7H7PZ6DCD5EXS67Y7Q, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.053205520 (76h44m19.924288565s) [noise=true , address=145.79.227.16:4910 , piece_id=YIHYKALCHWYGMAUKS3RPUQ2WOAQKCRQV6GZPFM27K37CKTTJSDPA, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.053898059 (76h44m19.924981114s) [noise=true , address=145.79.227.20:4437 , piece_id=G76GCXUI5YH3CMQV47V6FSFDDSOBIIFMS24UJVTNXRR2MM4UZ5ZQ, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.054182971 (76h44m19.925266006s) [noise=true , address=145.79.227.27:4691 , piece_id=R5VVIJNPYZIDQW5WSJMEYS4D55AHXXMACQY5GM5IOJXVKNOGBEAQ, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.058658112 (76h44m19.929741157s) [noise=true , address=145.79.227.29:4942 , piece_id=ZYT75AZ7X56NBH3OVLWL27J5FRRN4WKV5VEIKEXN32E6P5OQNOZQ, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.058679072 (76h44m19.929762107s) [seg_pos={0 0}, upload=67101]: Segment reached optimal threshold of 13 pieces. Dec 8 20:13:20.058873853 (76h44m19.929956888s) [noise=true , address=145.79.227.31:4797 , piece_id=IHEVQ2JUPASTR3CU7QMNAQVSYQ65LEV5SR2KO4GCEOKAKQR5Y5NA, seg_pos={0 0}, upload=67101]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9): protocol: expected piece hash; context canceled; EOF Dec 8 20:13:20.058880526 (76h44m19.929963561s) [noise=true , address=145.79.227.31:4797 , piece_id=IHEVQ2JUPASTR3CU7QMNAQVSYQ65LEV5SR2KO4GCEOKAKQR5Y5NA, seg_pos={0 0}, upload=67101]: Long tail cancellation due to optimal threshold Dec 8 20:13:20.059077551 (76h44m19.930160586s) [noise=true , address=145.79.227.20:4438 , piece_id=QZQULVSGCAOTQEE5MBUONJYAJIO3GPUNKB4CEVMK3NA2J3BYZ6GA, seg_pos={0 0}, upload=67101]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT): protocol: expected piece hash; context canceled; EOF Dec 8 20:13:20.059093532 (76h44m19.930176567s) [noise=true , address=145.79.227.20:4438 , piece_id=QZQULVSGCAOTQEE5MBUONJYAJIO3GPUNKB4CEVMK3NA2J3BYZ6GA, seg_pos={0 0}, upload=67101]: Long tail cancellation due to optimal threshold Dec 8 20:13:20.059116485 (76h44m19.93019953s) [seg_pos={0 0}, upload=67101]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 20:13:20.059191879 (76h44m19.930274924s) [upload=67101]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 20:13:20.059196798 (76h44m19.930279833s) [upload=67101]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 20:19:41.003042505 (76h50m40.87412554s) [upload=67101]: Done uploading segments... Dec 8 20:19:41.003155129 (76h50m40.874238175s) [upload=67101]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.781524052s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 5.205457036s trace: https://jaeger.dev.storj.io/trace/49fc2034659f6881 FAIL (2025-12-08 20:39:40 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.205480701s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [...X.......................X....X....X......................] duration: 5.624003652s trace: https://jaeger.dev.storj.io/trace/4600f1753cc382a2 FAIL (2025-12-08 20:39:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.624019672s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 5.010761628s trace: https://jaeger.dev.storj.io/trace/49fc2034659f6881 FAIL (2025-12-08 20:39:24 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.010841551s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............X.........X..................................] duration: 6.09722987s trace: https://jaeger.dev.storj.io/trace/6fb9d317d1dcf919 FAIL (2025-12-08 20:39:04 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.097258053s core uplink us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [.........................X...............X..................] duration: 6.240387248s trace: https://jaeger.dev.storj.io/trace/3996915dd8f30735 FAIL (2025-12-08 20:38:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== PGM26LWTYTWGZGQDPBL7UDX263DFBXY7WZRXGMYEFPWXQQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.569231809 (76h54m34.440315054s) [noise=true , address=89.149.201.147:4022 , piece_id=2VR3E4SXAAOR5RAVHFVUQQ7MBZQEEQYFZ2ZW5Z7YSCVOETZNFQWQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.569369882 (76h54m34.440452917s) [noise=true , address=185.167.204.206:4037 , piece_id=XYY5NAATQJPHOK6JAMHXR3JH3QTLQF7OTA32BMST5HYQDYJ76UMA, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.569672879 (76h54m34.440755924s) [noise=true , address=185.167.204.206:4056 , piece_id=IPDW6RCHJFLMB5LOZM3FG645CCW6TT7KJW4PBQXBUSAIE2ZZM74A, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.569779642 (76h54m34.440862677s) [noise=true , address=185.167.204.205:4024 , piece_id=2UHV7KDAKJJ4DDUU5MPBNLRP24OK5RQU6GGAVYUV5MEFLDDPGUZQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.570073130 (76h54m34.441156176s) [noise=true , address=81.171.14.70:4035 , piece_id=ZKLJ5VUPALKZBSPFCJQBZZUTMJ3PX2H3XMUZ5JVSW5BJBACRUYHA, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.570428618 (76h54m34.441511653s) [noise=true , address=81.171.14.70:4043 , piece_id=FVUGWZXJLQAOLUSTRKJAULSHDWKX5GGWA2BB52YP7WQEKU4NKS5Q, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.570785237 (76h54m34.441868282s) [noise=true , address=81.171.14.70:4044 , piece_id=QPYQGR62JDWWV3GM3CVDB6BLNSGNUTQUEHHQF74TYXFN25PMNPVQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.571235034 (76h54m34.442318069s) [noise=true , address=185.167.204.205:4035 , piece_id=LXUP2T5SN5SSV7IWZLGTSLCRWLFRZBODXX7W4FFVHZFO74I2CQHQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.571655133 (76h54m34.442738169s) [noise=true , address=81.171.14.70:4031 , piece_id=53QOUYYU4KS53TELJUM3FPEKOCB464OR5GKYGYB54RVRPD52RFWA, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.572102666 (76h54m34.443185701s) [noise=true , address=185.167.204.205:4034 , piece_id=4VH6HEDD7CTPMEOHYGKFXSYXHKOLPYB2N6BCCJT2DXVAMADI3KJQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:35.395419290 (76h54m35.266502335s) [upload=67229]: Next returned nil segment Dec 8 20:23:35.395432614 (76h54m35.26651565s) [upload=67229]: Waiting for error group managing segments... Dec 8 20:23:35.780649683 (76h54m35.651732718s) [noise=true , address=185.167.204.206:4037 , piece_id=XYY5NAATQJPHOK6JAMHXR3JH3QTLQF7OTA32BMST5HYQDYJ76UMA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.781236982 (76h54m35.652320027s) [noise=true , address=185.167.204.206:4056 , piece_id=IPDW6RCHJFLMB5LOZM3FG645CCW6TT7KJW4PBQXBUSAIE2ZZM74A, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.784267011 (76h54m35.655350046s) [noise=true , address=185.167.204.205:4030 , piece_id=O3QKGK2G5I5YYPRP5K57RTG4H35H2UDFROFU2JMSHKTFNZJDVUIA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.785164540 (76h54m35.656247585s) [noise=true , address=185.167.204.205:4012 , piece_id=QCJD2IHH4DEKTFN5ZPMZXPECBQ2IX2QPSFUZCSFR5HKCDOVYGGQA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.785585301 (76h54m35.656668336s) [noise=true , address=185.167.204.205:4002 , piece_id=PDSJ5KDCFQBURILRTWU2WSEG3UH7GX4GK57W4J75QADBPKQXRSYA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.787925598 (76h54m35.659008643s) [noise=true , address=185.167.204.206:4047 , piece_id=THE6CUP46D2OSXS34UR3TTBUP6FIVMGDGT7DVJDBB57BHMWBCZ7A, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.787994289 (76h54m35.659077324s) [noise=true , address=185.167.204.205:4034 , piece_id=4VH6HEDD7CTPMEOHYGKFXSYXHKOLPYB2N6BCCJT2DXVAMADI3KJQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.788899402 (76h54m35.659982437s) [noise=true , address=185.167.204.206:4050 , piece_id=7B4GLDQZUPQPLGPZPPN5CO3VOX3LLBMLIYA6IMALYITCHE4CZBOA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.791033055 (76h54m35.66211609s) [noise=true , address=185.167.204.205:4031 , piece_id=5XPLPLO4SPMX7ZBYHLUZTH7MBGCRESEEMP2OEI7LTTT5BN67XHMQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.792163116 (76h54m35.663246161s) [noise=true , address=185.167.204.206:4055 , piece_id=JESF36BPW3YEPXVI2GP4OALECNLYSASJHQRZVLNXKV5BQRDGOWWQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.792693356 (76h54m35.663776392s) [noise=true , address=185.167.204.206:4063 , piece_id=BRUAUJTOCPWNUOB6D5G6AJCTMJBRZRJXXXBZBIUUZZXXQB5OEX2Q, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.793171606 (76h54m35.664254662s) [noise=true , address=185.167.204.206:4045 , piece_id=IXV6HTIOVQR4HCGJIWTBQFLO733BMMNNMZ24Y34H6V6KYUQYPQRA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.798151289 (76h54m35.669234344s) [noise=true , address=185.167.204.206:4053 , piece_id=VGK5AKCI23P5ZDS427NS6PNNRIJ3QUXLRZ4ECLYN4UDSL3KAQAKA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.798189562 (76h54m35.669272597s) [noise=true , address=185.167.204.206:4040 , piece_id=4KFICIELLLPCLWQ7RIB7UUSWUK7SFX2PJK7JWCRU5S4HQNHBEWRQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.798633607 (76h54m35.669716642s) [noise=true , address=185.167.204.205:4035 , piece_id=LXUP2T5SN5SSV7IWZLGTSLCRWLFRZBODXX7W4FFVHZFO74I2CQHQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.803059726 (76h54m35.674142761s) [noise=true , address=185.167.204.205:4024 , piece_id=2UHV7KDAKJJ4DDUU5MPBNLRP24OK5RQU6GGAVYUV5MEFLDDPGUZQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.344154907 (76h54m36.215237952s) [noise=true , address=81.171.14.70:4036 , piece_id=FB3SB3XQDNT2HSK5FDGELGYWODFLU655QW4D23GEXIWQEZYFNFRA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.359261713 (76h54m36.230344758s) [noise=true , address=81.171.14.70:4031 , piece_id=53QOUYYU4KS53TELJUM3FPEKOCB464OR5GKYGYB54RVRPD52RFWA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.393865115 (76h54m36.26494815s) [noise=true , address=81.171.14.70:4035 , piece_id=ZKLJ5VUPALKZBSPFCJQBZZUTMJ3PX2H3XMUZ5JVSW5BJBACRUYHA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.394302157 (76h54m36.265385202s) [noise=true , address=81.171.14.70:4044 , piece_id=QPYQGR62JDWWV3GM3CVDB6BLNSGNUTQUEHHQF74TYXFN25PMNPVQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.430200674 (76h54m36.301283709s) [noise=true , address=81.171.14.70:4046 , piece_id=ZHZUULNXPFDSTOXXH6T7XIEPN23OOWCNAAUWPCZYNNDTLPRWXVKQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.800799297 (76h54m36.671882332s) [noise=true , address=81.171.14.70:4040 , piece_id=W2SOTP5SPLQ6J3PM57NZOLRVZNMTNSGPWDZKZC2KV2PBPH4TUBLA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.862864259 (76h54m36.733947295s) [noise=true , address=81.171.14.70:4038 , piece_id=CKN2RF2YSTEOK6HUXGVBVECYAP2CLGUF6KYNR5JTHRUZPTGVI73Q, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.953769972 (76h54m36.824853007s) [noise=true , address=81.171.14.70:4043 , piece_id=FVUGWZXJLQAOLUSTRKJAULSHDWKX5GGWA2BB52YP7WQEKU4NKS5Q, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.996603376 (76h54m36.867686421s) [noise=true , address=89.149.201.147:4006 , piece_id=UGREZBZRVKXMGBOLQEV2NC4COTXOTTRUEAGSPNWRWP22MCPHQP7Q, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:37.037518916 (76h54m36.908601951s) [noise=true , address=89.149.201.147:4014 , piece_id=AEG6KGPGM26LWTYTWGZGQDPBL7UDX263DFBXY7WZRXGMYEFPWXQQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:37.037537541 (76h54m36.908620586s) [seg_pos={0 0}, upload=67229]: Segment reached optimal threshold of 26 pieces. Dec 8 20:23:37.037730880 (76h54m36.908813915s) [noise=true , address=89.149.201.147:4023 , piece_id=5EPGXWBK6GNTKLQLWEIWYBC2OJHFR3M7F4X3X4WODJWNXID3PCQQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BaeSxjXneBKoGKVrRM686sq386Bhn1YzzBFG9we8fHbcc9tbv): protocol: expected piece hash; context canceled; EOF Dec 8 20:23:37.037737452 (76h54m36.908820497s) [noise=true , address=89.149.201.147:4023 , piece_id=5EPGXWBK6GNTKLQLWEIWYBC2OJHFR3M7F4X3X4WODJWNXID3PCQQ, seg_pos={0 0}, upload=67229]: Long tail cancellation due to optimal threshold Dec 8 20:23:37.037893039 (76h54m36.908976074s) [noise=true , address=89.149.201.147:4022 , piece_id=2VR3E4SXAAOR5RAVHFVUQQ7MBZQEEQYFZ2ZW5Z7YSCVOETZNFQWQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122Vj8XxpS6RnhVweQV3cZWFhj8nrzy3hZ9P7JsuTzEaZcD6iFE): protocol: expected piece hash; context canceled; EOF Dec 8 20:23:37.037900964 (76h54m36.908984019s) [noise=true , address=89.149.201.147:4022 , piece_id=2VR3E4SXAAOR5RAVHFVUQQ7MBZQEEQYFZ2ZW5Z7YSCVOETZNFQWQ, seg_pos={0 0}, upload=67229]: Long tail cancellation due to optimal threshold Dec 8 20:23:37.037926843 (76h54m36.909009889s) [seg_pos={0 0}, upload=67229]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 8 20:23:37.038007697 (76h54m36.909090732s) [upload=67229]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 20:23:37.038013398 (76h54m36.909096433s) [upload=67229]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 20:26:49.742746371 (76h57m49.613829416s) [upload=67229]: Done uploading segments... Dec 8 20:26:49.742860908 (76h57m49.613943943s) [upload=67229]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.240416093s core uplink eu-select-1 eu1-placement-15
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 468.500247ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/4b87d2bbd73f899c FAIL (2025-12-08 20:38:53 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 468.54441ms core satellite eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 5.652660123s trace: https://jaeger.dev.storj.io/trace/690c0830f44a9942 FAIL (2025-12-08 20:38:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 20:13:16.312406346 (76h44m16.183489391s) [upload=67101]: Starting upload Dec 8 20:13:16.312474916 (76h44m16.183557952s) [upload=67101]: Done starting upload Dec 8 20:13:16.312661411 (76h44m16.183744446s) [upload=67101]: Uploading segments... Dec 8 20:13:16.313579429 (76h44m16.184662474s) [upload=67101]: Got next segment. Inline: false Dec 8 20:13:16.313623854 (76h44m16.184706889s) [upload=67101]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 20:13:16.313629435 (76h44m16.18471247s) [upload=67101]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 20:13:18.037990476 (76h44m17.909073521s) [seg_pos={0 0}, upload=67101]: Begin upload segment... Dec 8 20:13:18.038044659 (76h44m17.909127694s) [seg_pos={0 0}, upload=67101]: Done begin upload segment. Dec 8 20:13:18.038075859 (76h44m17.909158914s) [noise=true , address=145.79.227.31:4797 , piece_id=IHEVQ2JUPASTR3CU7QMNAQVSYQ65LEV5SR2KO4GCEOKAKQR5Y5NA, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.038191428 (76h44m17.909274463s) [noise=true , address=145.79.227.41:4085 , piece_id=FFFFOS7PH3SN3S4NWJJRYP772D725TSGCBKXIZ7FTEDBPO3TNR3Q, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.038908023 (76h44m17.909991068s) [noise=true , address=145.79.227.24:4583 , piece_id=MTNWT2AOTPYXLNVPSOKK5ZOAY7U3UH376V232TZKE56UM2CMDVAQ, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.039373890 (76h44m17.910456925s) [noise=true , address=145.79.227.29:4942 , piece_id=ZYT75AZ7X56NBH3OVLWL27J5FRRN4WKV5VEIKEXN32E6P5OQNOZQ, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.039816273 (76h44m17.910899318s) [noise=true , address=145.79.227.42:4121 , piece_id=ICWYGS3AGWC47JS4PUC4GHT2YOV5UTD5RYL73JFTWZRNEMAGMRZA, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.040292149 (76h44m17.911375194s) [noise=true , address=145.79.227.32:4834 , piece_id=WDZ3N3A3CNPAECT7HW2VZ2K7JEJVOZJCZR3BKHRBVR3XHGPGTKPQ, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.040745432 (76h44m17.911828477s) [noise=true , address=145.79.227.31:4799 , piece_id=RNPR7HC6YANTTQW7STMNTHYANE7ZD5XC2N5DMLQQ5YBULNUSFK7A, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.041266104 (76h44m17.912349139s) [noise=true , address=145.79.227.20:4437 , piece_id=G76GCXUI5YH3CMQV47V6FSFDDSOBIIFMS24UJVTNXRR2MM4UZ5ZQ, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.041787887 (76h44m17.912870952s) [noise=true , address=145.79.227.25:4619 , piece_id=5M5PTNON4YZ7INE34X7I6BIJVQF543VQWALEZBCZEEEQ26UMGYQA, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.042946814 (76h44m17.914029849s) [noise=true , address=145.79.227.16:4912 , piece_id=J7BVXTHOSEPXKDDWXPV7JIUWW5NNWLETI7S4IOVPDG6ZUVPATOXQ, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.044580184 (76h44m17.915663219s) [noise=true , address=145.79.227.27:4691 , piece_id=R5VVIJNPYZIDQW5WSJMEYS4D55AHXXMACQY5GM5IOJXVKNOGBEAQ, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.047199231 (76h44m17.918282266s) [noise=true , address=145.79.227.16:4910 , piece_id=YIHYKALCHWYGMAUKS3RPUQ2WOAQKCRQV6GZPFM27K37CKTTJSDPA, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.049620632 (76h44m17.920703677s) [noise=true , address=145.79.227.20:4438 , piece_id=QZQULVSGCAOTQEE5MBUONJYAJIO3GPUNKB4CEVMK3NA2J3BYZ6GA, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.051625078 (76h44m17.922708143s) [noise=true , address=145.79.227.22:4511 , piece_id=CFUDAP3RXMYJCS5GQJVYCKXTN6RLXG7NSQH6EHTZPSYCXPSPBEFQ, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.053186752 (76h44m17.924269787s) [noise=true , address=145.79.227.39:4015 , piece_id=UX4HGNQIHC3FCNGUGIAZYPNYYTLNXMYIND7H7PZ6DCD5EXS67Y7Q, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.054291115 (76h44m17.92537415s) [noise=true , address=77.90.31.10:4517 , piece_id=BMXTXAXNYZ5U5T54U57PMGP4KZO332K2VEP4KO5NEYC7TZ2S53SA, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:19.111879942 (76h44m18.982962987s) [noise=true , address=77.90.31.10:4517 , piece_id=BMXTXAXNYZ5U5T54U57PMGP4KZO332K2VEP4KO5NEYC7TZ2S53SA, seg_pos={0 0}, upload=67101]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: storage node overloaded, request limit: 2000 Dec 8 20:13:19.643615670 (76h44m19.514698706s) [upload=67101]: Next returned nil segment Dec 8 20:13:19.643629617 (76h44m19.514712652s) [upload=67101]: Waiting for error group managing segments... Dec 8 20:13:20.026246645 (76h44m19.89732969s) [noise=true , address=145.79.227.32:4834 , piece_id=WDZ3N3A3CNPAECT7HW2VZ2K7JEJVOZJCZR3BKHRBVR3XHGPGTKPQ, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.027742514 (76h44m19.898825559s) [noise=true , address=145.79.227.25:4619 , piece_id=5M5PTNON4YZ7INE34X7I6BIJVQF543VQWALEZBCZEEEQ26UMGYQA, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.028492581 (76h44m19.899575616s) [noise=true , address=145.79.227.41:4085 , piece_id=FFFFOS7PH3SN3S4NWJJRYP772D725TSGCBKXIZ7FTEDBPO3TNR3Q, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.029004767 (76h44m19.900087802s) [noise=true , address=145.79.227.16:4912 , piece_id=J7BVXTHOSEPXKDDWXPV7JIUWW5NNWLETI7S4IOVPDG6ZUVPATOXQ, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.029438392 (76h44m19.900521427s) [noise=true , address=145.79.227.24:4583 , piece_id=MTNWT2AOTPYXLNVPSOKK5ZOAY7U3UH376V232TZKE56UM2CMDVAQ, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.031410738 (76h44m19.902493783s) [noise=true , address=145.79.227.42:4121 , piece_id=ICWYGS3AGWC47JS4PUC4GHT2YOV5UTD5RYL73JFTWZRNEMAGMRZA, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.034141027 (76h44m19.905224072s) [noise=true , address=145.79.227.31:4799 , piece_id=RNPR7HC6YANTTQW7STMNTHYANE7ZD5XC2N5DMLQQ5YBULNUSFK7A, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.037360508 (76h44m19.908443543s) [noise=true , address=145.79.227.22:4511 , piece_id=CFUDAP3RXMYJCS5GQJVYCKXTN6RLXG7NSQH6EHTZPSYCXPSPBEFQ, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.038059108 (76h44m19.909142153s) [noise=true , address=145.79.227.39:4015 , piece_id=UX4HGNQIHC3FCNGUGIAZYPNYYTLNXMYIND7H7PZ6DCD5EXS67Y7Q, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.053205520 (76h44m19.924288565s) [noise=true , address=145.79.227.16:4910 , piece_id=YIHYKALCHWYGMAUKS3RPUQ2WOAQKCRQV6GZPFM27K37CKTTJSDPA, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.053898059 (76h44m19.924981114s) [noise=true , address=145.79.227.20:4437 , piece_id=G76GCXUI5YH3CMQV47V6FSFDDSOBIIFMS24UJVTNXRR2MM4UZ5ZQ, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.054182971 (76h44m19.925266006s) [noise=true , address=145.79.227.27:4691 , piece_id=R5VVIJNPYZIDQW5WSJMEYS4D55AHXXMACQY5GM5IOJXVKNOGBEAQ, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.058658112 (76h44m19.929741157s) [noise=true , address=145.79.227.29:4942 , piece_id=ZYT75AZ7X56NBH3OVLWL27J5FRRN4WKV5VEIKEXN32E6P5OQNOZQ, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.058679072 (76h44m19.929762107s) [seg_pos={0 0}, upload=67101]: Segment reached optimal threshold of 13 pieces. Dec 8 20:13:20.058873853 (76h44m19.929956888s) [noise=true , address=145.79.227.31:4797 , piece_id=IHEVQ2JUPASTR3CU7QMNAQVSYQ65LEV5SR2KO4GCEOKAKQR5Y5NA, seg_pos={0 0}, upload=67101]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9): protocol: expected piece hash; context canceled; EOF Dec 8 20:13:20.058880526 (76h44m19.929963561s) [noise=true , address=145.79.227.31:4797 , piece_id=IHEVQ2JUPASTR3CU7QMNAQVSYQ65LEV5SR2KO4GCEOKAKQR5Y5NA, seg_pos={0 0}, upload=67101]: Long tail cancellation due to optimal threshold Dec 8 20:13:20.059077551 (76h44m19.930160586s) [noise=true , address=145.79.227.20:4438 , piece_id=QZQULVSGCAOTQEE5MBUONJYAJIO3GPUNKB4CEVMK3NA2J3BYZ6GA, seg_pos={0 0}, upload=67101]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT): protocol: expected piece hash; context canceled; EOF Dec 8 20:13:20.059093532 (76h44m19.930176567s) [noise=true , address=145.79.227.20:4438 , piece_id=QZQULVSGCAOTQEE5MBUONJYAJIO3GPUNKB4CEVMK3NA2J3BYZ6GA, seg_pos={0 0}, upload=67101]: Long tail cancellation due to optimal threshold Dec 8 20:13:20.059116485 (76h44m19.93019953s) [seg_pos={0 0}, upload=67101]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 20:13:20.059191879 (76h44m19.930274924s) [upload=67101]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 20:13:20.059196798 (76h44m19.930279833s) [upload=67101]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 20:19:41.003042505 (76h50m40.87412554s) [upload=67101]: Done uploading segments... Dec 8 20:19:41.003155129 (76h50m40.874238175s) [upload=67101]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.652692504s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................X.........X.................................] duration: 5.655982811s trace: https://jaeger.dev.storj.io/trace/6fb9d317d1dcf919 FAIL (2025-12-08 20:38:41 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.656014652s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [....X.......................X....X....X.....................] duration: 5.675440793s trace: https://jaeger.dev.storj.io/trace/4600f1753cc382a2 FAIL (2025-12-08 20:38:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.675466493s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 4.909559554s trace: https://jaeger.dev.storj.io/trace/49fc2034659f6881 FAIL (2025-12-08 20:38:34 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.909585473s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [..........................X...............X.................] duration: 6.20701769s trace: https://jaeger.dev.storj.io/trace/3996915dd8f30735 FAIL (2025-12-08 20:38:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== PGM26LWTYTWGZGQDPBL7UDX263DFBXY7WZRXGMYEFPWXQQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.569231809 (76h54m34.440315054s) [noise=true , address=89.149.201.147:4022 , piece_id=2VR3E4SXAAOR5RAVHFVUQQ7MBZQEEQYFZ2ZW5Z7YSCVOETZNFQWQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.569369882 (76h54m34.440452917s) [noise=true , address=185.167.204.206:4037 , piece_id=XYY5NAATQJPHOK6JAMHXR3JH3QTLQF7OTA32BMST5HYQDYJ76UMA, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.569672879 (76h54m34.440755924s) [noise=true , address=185.167.204.206:4056 , piece_id=IPDW6RCHJFLMB5LOZM3FG645CCW6TT7KJW4PBQXBUSAIE2ZZM74A, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.569779642 (76h54m34.440862677s) [noise=true , address=185.167.204.205:4024 , piece_id=2UHV7KDAKJJ4DDUU5MPBNLRP24OK5RQU6GGAVYUV5MEFLDDPGUZQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.570073130 (76h54m34.441156176s) [noise=true , address=81.171.14.70:4035 , piece_id=ZKLJ5VUPALKZBSPFCJQBZZUTMJ3PX2H3XMUZ5JVSW5BJBACRUYHA, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.570428618 (76h54m34.441511653s) [noise=true , address=81.171.14.70:4043 , piece_id=FVUGWZXJLQAOLUSTRKJAULSHDWKX5GGWA2BB52YP7WQEKU4NKS5Q, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.570785237 (76h54m34.441868282s) [noise=true , address=81.171.14.70:4044 , piece_id=QPYQGR62JDWWV3GM3CVDB6BLNSGNUTQUEHHQF74TYXFN25PMNPVQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.571235034 (76h54m34.442318069s) [noise=true , address=185.167.204.205:4035 , piece_id=LXUP2T5SN5SSV7IWZLGTSLCRWLFRZBODXX7W4FFVHZFO74I2CQHQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.571655133 (76h54m34.442738169s) [noise=true , address=81.171.14.70:4031 , piece_id=53QOUYYU4KS53TELJUM3FPEKOCB464OR5GKYGYB54RVRPD52RFWA, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.572102666 (76h54m34.443185701s) [noise=true , address=185.167.204.205:4034 , piece_id=4VH6HEDD7CTPMEOHYGKFXSYXHKOLPYB2N6BCCJT2DXVAMADI3KJQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:35.395419290 (76h54m35.266502335s) [upload=67229]: Next returned nil segment Dec 8 20:23:35.395432614 (76h54m35.26651565s) [upload=67229]: Waiting for error group managing segments... Dec 8 20:23:35.780649683 (76h54m35.651732718s) [noise=true , address=185.167.204.206:4037 , piece_id=XYY5NAATQJPHOK6JAMHXR3JH3QTLQF7OTA32BMST5HYQDYJ76UMA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.781236982 (76h54m35.652320027s) [noise=true , address=185.167.204.206:4056 , piece_id=IPDW6RCHJFLMB5LOZM3FG645CCW6TT7KJW4PBQXBUSAIE2ZZM74A, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.784267011 (76h54m35.655350046s) [noise=true , address=185.167.204.205:4030 , piece_id=O3QKGK2G5I5YYPRP5K57RTG4H35H2UDFROFU2JMSHKTFNZJDVUIA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.785164540 (76h54m35.656247585s) [noise=true , address=185.167.204.205:4012 , piece_id=QCJD2IHH4DEKTFN5ZPMZXPECBQ2IX2QPSFUZCSFR5HKCDOVYGGQA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.785585301 (76h54m35.656668336s) [noise=true , address=185.167.204.205:4002 , piece_id=PDSJ5KDCFQBURILRTWU2WSEG3UH7GX4GK57W4J75QADBPKQXRSYA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.787925598 (76h54m35.659008643s) [noise=true , address=185.167.204.206:4047 , piece_id=THE6CUP46D2OSXS34UR3TTBUP6FIVMGDGT7DVJDBB57BHMWBCZ7A, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.787994289 (76h54m35.659077324s) [noise=true , address=185.167.204.205:4034 , piece_id=4VH6HEDD7CTPMEOHYGKFXSYXHKOLPYB2N6BCCJT2DXVAMADI3KJQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.788899402 (76h54m35.659982437s) [noise=true , address=185.167.204.206:4050 , piece_id=7B4GLDQZUPQPLGPZPPN5CO3VOX3LLBMLIYA6IMALYITCHE4CZBOA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.791033055 (76h54m35.66211609s) [noise=true , address=185.167.204.205:4031 , piece_id=5XPLPLO4SPMX7ZBYHLUZTH7MBGCRESEEMP2OEI7LTTT5BN67XHMQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.792163116 (76h54m35.663246161s) [noise=true , address=185.167.204.206:4055 , piece_id=JESF36BPW3YEPXVI2GP4OALECNLYSASJHQRZVLNXKV5BQRDGOWWQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.792693356 (76h54m35.663776392s) [noise=true , address=185.167.204.206:4063 , piece_id=BRUAUJTOCPWNUOB6D5G6AJCTMJBRZRJXXXBZBIUUZZXXQB5OEX2Q, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.793171606 (76h54m35.664254662s) [noise=true , address=185.167.204.206:4045 , piece_id=IXV6HTIOVQR4HCGJIWTBQFLO733BMMNNMZ24Y34H6V6KYUQYPQRA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.798151289 (76h54m35.669234344s) [noise=true , address=185.167.204.206:4053 , piece_id=VGK5AKCI23P5ZDS427NS6PNNRIJ3QUXLRZ4ECLYN4UDSL3KAQAKA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.798189562 (76h54m35.669272597s) [noise=true , address=185.167.204.206:4040 , piece_id=4KFICIELLLPCLWQ7RIB7UUSWUK7SFX2PJK7JWCRU5S4HQNHBEWRQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.798633607 (76h54m35.669716642s) [noise=true , address=185.167.204.205:4035 , piece_id=LXUP2T5SN5SSV7IWZLGTSLCRWLFRZBODXX7W4FFVHZFO74I2CQHQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.803059726 (76h54m35.674142761s) [noise=true , address=185.167.204.205:4024 , piece_id=2UHV7KDAKJJ4DDUU5MPBNLRP24OK5RQU6GGAVYUV5MEFLDDPGUZQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.344154907 (76h54m36.215237952s) [noise=true , address=81.171.14.70:4036 , piece_id=FB3SB3XQDNT2HSK5FDGELGYWODFLU655QW4D23GEXIWQEZYFNFRA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.359261713 (76h54m36.230344758s) [noise=true , address=81.171.14.70:4031 , piece_id=53QOUYYU4KS53TELJUM3FPEKOCB464OR5GKYGYB54RVRPD52RFWA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.393865115 (76h54m36.26494815s) [noise=true , address=81.171.14.70:4035 , piece_id=ZKLJ5VUPALKZBSPFCJQBZZUTMJ3PX2H3XMUZ5JVSW5BJBACRUYHA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.394302157 (76h54m36.265385202s) [noise=true , address=81.171.14.70:4044 , piece_id=QPYQGR62JDWWV3GM3CVDB6BLNSGNUTQUEHHQF74TYXFN25PMNPVQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.430200674 (76h54m36.301283709s) [noise=true , address=81.171.14.70:4046 , piece_id=ZHZUULNXPFDSTOXXH6T7XIEPN23OOWCNAAUWPCZYNNDTLPRWXVKQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.800799297 (76h54m36.671882332s) [noise=true , address=81.171.14.70:4040 , piece_id=W2SOTP5SPLQ6J3PM57NZOLRVZNMTNSGPWDZKZC2KV2PBPH4TUBLA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.862864259 (76h54m36.733947295s) [noise=true , address=81.171.14.70:4038 , piece_id=CKN2RF2YSTEOK6HUXGVBVECYAP2CLGUF6KYNR5JTHRUZPTGVI73Q, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.953769972 (76h54m36.824853007s) [noise=true , address=81.171.14.70:4043 , piece_id=FVUGWZXJLQAOLUSTRKJAULSHDWKX5GGWA2BB52YP7WQEKU4NKS5Q, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.996603376 (76h54m36.867686421s) [noise=true , address=89.149.201.147:4006 , piece_id=UGREZBZRVKXMGBOLQEV2NC4COTXOTTRUEAGSPNWRWP22MCPHQP7Q, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:37.037518916 (76h54m36.908601951s) [noise=true , address=89.149.201.147:4014 , piece_id=AEG6KGPGM26LWTYTWGZGQDPBL7UDX263DFBXY7WZRXGMYEFPWXQQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:37.037537541 (76h54m36.908620586s) [seg_pos={0 0}, upload=67229]: Segment reached optimal threshold of 26 pieces. Dec 8 20:23:37.037730880 (76h54m36.908813915s) [noise=true , address=89.149.201.147:4023 , piece_id=5EPGXWBK6GNTKLQLWEIWYBC2OJHFR3M7F4X3X4WODJWNXID3PCQQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BaeSxjXneBKoGKVrRM686sq386Bhn1YzzBFG9we8fHbcc9tbv): protocol: expected piece hash; context canceled; EOF Dec 8 20:23:37.037737452 (76h54m36.908820497s) [noise=true , address=89.149.201.147:4023 , piece_id=5EPGXWBK6GNTKLQLWEIWYBC2OJHFR3M7F4X3X4WODJWNXID3PCQQ, seg_pos={0 0}, upload=67229]: Long tail cancellation due to optimal threshold Dec 8 20:23:37.037893039 (76h54m36.908976074s) [noise=true , address=89.149.201.147:4022 , piece_id=2VR3E4SXAAOR5RAVHFVUQQ7MBZQEEQYFZ2ZW5Z7YSCVOETZNFQWQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122Vj8XxpS6RnhVweQV3cZWFhj8nrzy3hZ9P7JsuTzEaZcD6iFE): protocol: expected piece hash; context canceled; EOF Dec 8 20:23:37.037900964 (76h54m36.908984019s) [noise=true , address=89.149.201.147:4022 , piece_id=2VR3E4SXAAOR5RAVHFVUQQ7MBZQEEQYFZ2ZW5Z7YSCVOETZNFQWQ, seg_pos={0 0}, upload=67229]: Long tail cancellation due to optimal threshold Dec 8 20:23:37.037926843 (76h54m36.909009889s) [seg_pos={0 0}, upload=67229]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 8 20:23:37.038007697 (76h54m36.909090732s) [upload=67229]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 20:23:37.038013398 (76h54m36.909096433s) [upload=67229]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 20:26:49.742746371 (76h57m49.613829416s) [upload=67229]: Done uploading segments... Dec 8 20:26:49.742860908 (76h57m49.613943943s) [upload=67229]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.207052358s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 5.674931809s trace: https://jaeger.dev.storj.io/trace/690c0830f44a9942 FAIL (2025-12-08 20:38:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 20:13:16.312406346 (76h44m16.183489391s) [upload=67101]: Starting upload Dec 8 20:13:16.312474916 (76h44m16.183557952s) [upload=67101]: Done starting upload Dec 8 20:13:16.312661411 (76h44m16.183744446s) [upload=67101]: Uploading segments... Dec 8 20:13:16.313579429 (76h44m16.184662474s) [upload=67101]: Got next segment. Inline: false Dec 8 20:13:16.313623854 (76h44m16.184706889s) [upload=67101]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 20:13:16.313629435 (76h44m16.18471247s) [upload=67101]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 20:13:18.037990476 (76h44m17.909073521s) [seg_pos={0 0}, upload=67101]: Begin upload segment... Dec 8 20:13:18.038044659 (76h44m17.909127694s) [seg_pos={0 0}, upload=67101]: Done begin upload segment. Dec 8 20:13:18.038075859 (76h44m17.909158914s) [noise=true , address=145.79.227.31:4797 , piece_id=IHEVQ2JUPASTR3CU7QMNAQVSYQ65LEV5SR2KO4GCEOKAKQR5Y5NA, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.038191428 (76h44m17.909274463s) [noise=true , address=145.79.227.41:4085 , piece_id=FFFFOS7PH3SN3S4NWJJRYP772D725TSGCBKXIZ7FTEDBPO3TNR3Q, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.038908023 (76h44m17.909991068s) [noise=true , address=145.79.227.24:4583 , piece_id=MTNWT2AOTPYXLNVPSOKK5ZOAY7U3UH376V232TZKE56UM2CMDVAQ, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.039373890 (76h44m17.910456925s) [noise=true , address=145.79.227.29:4942 , piece_id=ZYT75AZ7X56NBH3OVLWL27J5FRRN4WKV5VEIKEXN32E6P5OQNOZQ, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.039816273 (76h44m17.910899318s) [noise=true , address=145.79.227.42:4121 , piece_id=ICWYGS3AGWC47JS4PUC4GHT2YOV5UTD5RYL73JFTWZRNEMAGMRZA, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.040292149 (76h44m17.911375194s) [noise=true , address=145.79.227.32:4834 , piece_id=WDZ3N3A3CNPAECT7HW2VZ2K7JEJVOZJCZR3BKHRBVR3XHGPGTKPQ, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.040745432 (76h44m17.911828477s) [noise=true , address=145.79.227.31:4799 , piece_id=RNPR7HC6YANTTQW7STMNTHYANE7ZD5XC2N5DMLQQ5YBULNUSFK7A, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.041266104 (76h44m17.912349139s) [noise=true , address=145.79.227.20:4437 , piece_id=G76GCXUI5YH3CMQV47V6FSFDDSOBIIFMS24UJVTNXRR2MM4UZ5ZQ, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.041787887 (76h44m17.912870952s) [noise=true , address=145.79.227.25:4619 , piece_id=5M5PTNON4YZ7INE34X7I6BIJVQF543VQWALEZBCZEEEQ26UMGYQA, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.042946814 (76h44m17.914029849s) [noise=true , address=145.79.227.16:4912 , piece_id=J7BVXTHOSEPXKDDWXPV7JIUWW5NNWLETI7S4IOVPDG6ZUVPATOXQ, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.044580184 (76h44m17.915663219s) [noise=true , address=145.79.227.27:4691 , piece_id=R5VVIJNPYZIDQW5WSJMEYS4D55AHXXMACQY5GM5IOJXVKNOGBEAQ, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.047199231 (76h44m17.918282266s) [noise=true , address=145.79.227.16:4910 , piece_id=YIHYKALCHWYGMAUKS3RPUQ2WOAQKCRQV6GZPFM27K37CKTTJSDPA, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.049620632 (76h44m17.920703677s) [noise=true , address=145.79.227.20:4438 , piece_id=QZQULVSGCAOTQEE5MBUONJYAJIO3GPUNKB4CEVMK3NA2J3BYZ6GA, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.051625078 (76h44m17.922708143s) [noise=true , address=145.79.227.22:4511 , piece_id=CFUDAP3RXMYJCS5GQJVYCKXTN6RLXG7NSQH6EHTZPSYCXPSPBEFQ, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.053186752 (76h44m17.924269787s) [noise=true , address=145.79.227.39:4015 , piece_id=UX4HGNQIHC3FCNGUGIAZYPNYYTLNXMYIND7H7PZ6DCD5EXS67Y7Q, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.054291115 (76h44m17.92537415s) [noise=true , address=77.90.31.10:4517 , piece_id=BMXTXAXNYZ5U5T54U57PMGP4KZO332K2VEP4KO5NEYC7TZ2S53SA, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:19.111879942 (76h44m18.982962987s) [noise=true , address=77.90.31.10:4517 , piece_id=BMXTXAXNYZ5U5T54U57PMGP4KZO332K2VEP4KO5NEYC7TZ2S53SA, seg_pos={0 0}, upload=67101]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: storage node overloaded, request limit: 2000 Dec 8 20:13:19.643615670 (76h44m19.514698706s) [upload=67101]: Next returned nil segment Dec 8 20:13:19.643629617 (76h44m19.514712652s) [upload=67101]: Waiting for error group managing segments... Dec 8 20:13:20.026246645 (76h44m19.89732969s) [noise=true , address=145.79.227.32:4834 , piece_id=WDZ3N3A3CNPAECT7HW2VZ2K7JEJVOZJCZR3BKHRBVR3XHGPGTKPQ, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.027742514 (76h44m19.898825559s) [noise=true , address=145.79.227.25:4619 , piece_id=5M5PTNON4YZ7INE34X7I6BIJVQF543VQWALEZBCZEEEQ26UMGYQA, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.028492581 (76h44m19.899575616s) [noise=true , address=145.79.227.41:4085 , piece_id=FFFFOS7PH3SN3S4NWJJRYP772D725TSGCBKXIZ7FTEDBPO3TNR3Q, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.029004767 (76h44m19.900087802s) [noise=true , address=145.79.227.16:4912 , piece_id=J7BVXTHOSEPXKDDWXPV7JIUWW5NNWLETI7S4IOVPDG6ZUVPATOXQ, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.029438392 (76h44m19.900521427s) [noise=true , address=145.79.227.24:4583 , piece_id=MTNWT2AOTPYXLNVPSOKK5ZOAY7U3UH376V232TZKE56UM2CMDVAQ, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.031410738 (76h44m19.902493783s) [noise=true , address=145.79.227.42:4121 , piece_id=ICWYGS3AGWC47JS4PUC4GHT2YOV5UTD5RYL73JFTWZRNEMAGMRZA, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.034141027 (76h44m19.905224072s) [noise=true , address=145.79.227.31:4799 , piece_id=RNPR7HC6YANTTQW7STMNTHYANE7ZD5XC2N5DMLQQ5YBULNUSFK7A, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.037360508 (76h44m19.908443543s) [noise=true , address=145.79.227.22:4511 , piece_id=CFUDAP3RXMYJCS5GQJVYCKXTN6RLXG7NSQH6EHTZPSYCXPSPBEFQ, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.038059108 (76h44m19.909142153s) [noise=true , address=145.79.227.39:4015 , piece_id=UX4HGNQIHC3FCNGUGIAZYPNYYTLNXMYIND7H7PZ6DCD5EXS67Y7Q, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.053205520 (76h44m19.924288565s) [noise=true , address=145.79.227.16:4910 , piece_id=YIHYKALCHWYGMAUKS3RPUQ2WOAQKCRQV6GZPFM27K37CKTTJSDPA, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.053898059 (76h44m19.924981114s) [noise=true , address=145.79.227.20:4437 , piece_id=G76GCXUI5YH3CMQV47V6FSFDDSOBIIFMS24UJVTNXRR2MM4UZ5ZQ, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.054182971 (76h44m19.925266006s) [noise=true , address=145.79.227.27:4691 , piece_id=R5VVIJNPYZIDQW5WSJMEYS4D55AHXXMACQY5GM5IOJXVKNOGBEAQ, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.058658112 (76h44m19.929741157s) [noise=true , address=145.79.227.29:4942 , piece_id=ZYT75AZ7X56NBH3OVLWL27J5FRRN4WKV5VEIKEXN32E6P5OQNOZQ, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.058679072 (76h44m19.929762107s) [seg_pos={0 0}, upload=67101]: Segment reached optimal threshold of 13 pieces. Dec 8 20:13:20.058873853 (76h44m19.929956888s) [noise=true , address=145.79.227.31:4797 , piece_id=IHEVQ2JUPASTR3CU7QMNAQVSYQ65LEV5SR2KO4GCEOKAKQR5Y5NA, seg_pos={0 0}, upload=67101]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9): protocol: expected piece hash; context canceled; EOF Dec 8 20:13:20.058880526 (76h44m19.929963561s) [noise=true , address=145.79.227.31:4797 , piece_id=IHEVQ2JUPASTR3CU7QMNAQVSYQ65LEV5SR2KO4GCEOKAKQR5Y5NA, seg_pos={0 0}, upload=67101]: Long tail cancellation due to optimal threshold Dec 8 20:13:20.059077551 (76h44m19.930160586s) [noise=true , address=145.79.227.20:4438 , piece_id=QZQULVSGCAOTQEE5MBUONJYAJIO3GPUNKB4CEVMK3NA2J3BYZ6GA, seg_pos={0 0}, upload=67101]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT): protocol: expected piece hash; context canceled; EOF Dec 8 20:13:20.059093532 (76h44m19.930176567s) [noise=true , address=145.79.227.20:4438 , piece_id=QZQULVSGCAOTQEE5MBUONJYAJIO3GPUNKB4CEVMK3NA2J3BYZ6GA, seg_pos={0 0}, upload=67101]: Long tail cancellation due to optimal threshold Dec 8 20:13:20.059116485 (76h44m19.93019953s) [seg_pos={0 0}, upload=67101]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 20:13:20.059191879 (76h44m19.930274924s) [upload=67101]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 20:13:20.059196798 (76h44m19.930279833s) [upload=67101]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 20:19:41.003042505 (76h50m40.87412554s) [upload=67101]: Done uploading segments... Dec 8 20:19:41.003155129 (76h50m40.874238175s) [upload=67101]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.674959612s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................X.........X................................] duration: 5.820136977s trace: https://jaeger.dev.storj.io/trace/6fb9d317d1dcf919 FAIL (2025-12-08 20:38:01 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.820167233s core uplink us1
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 474.639678ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/4b87d2bbd73f899c FAIL (2025-12-08 20:37:54 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 474.695564ms core satellite eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 5.102807751s trace: https://jaeger.dev.storj.io/trace/49fc2034659f6881 FAIL (2025-12-08 20:37:53 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.10284369s core uplink eu1
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 468.697855ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/4b87d2bbd73f899c FAIL (2025-12-08 20:37:35 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 468.721709ms core satellite eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 5.778456212s trace: https://jaeger.dev.storj.io/trace/690c0830f44a9942 FAIL (2025-12-08 20:37:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 20:13:16.312406346 (76h44m16.183489391s) [upload=67101]: Starting upload Dec 8 20:13:16.312474916 (76h44m16.183557952s) [upload=67101]: Done starting upload Dec 8 20:13:16.312661411 (76h44m16.183744446s) [upload=67101]: Uploading segments... Dec 8 20:13:16.313579429 (76h44m16.184662474s) [upload=67101]: Got next segment. Inline: false Dec 8 20:13:16.313623854 (76h44m16.184706889s) [upload=67101]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 20:13:16.313629435 (76h44m16.18471247s) [upload=67101]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 20:13:18.037990476 (76h44m17.909073521s) [seg_pos={0 0}, upload=67101]: Begin upload segment... Dec 8 20:13:18.038044659 (76h44m17.909127694s) [seg_pos={0 0}, upload=67101]: Done begin upload segment. Dec 8 20:13:18.038075859 (76h44m17.909158914s) [noise=true , address=145.79.227.31:4797 , piece_id=IHEVQ2JUPASTR3CU7QMNAQVSYQ65LEV5SR2KO4GCEOKAKQR5Y5NA, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.038191428 (76h44m17.909274463s) [noise=true , address=145.79.227.41:4085 , piece_id=FFFFOS7PH3SN3S4NWJJRYP772D725TSGCBKXIZ7FTEDBPO3TNR3Q, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.038908023 (76h44m17.909991068s) [noise=true , address=145.79.227.24:4583 , piece_id=MTNWT2AOTPYXLNVPSOKK5ZOAY7U3UH376V232TZKE56UM2CMDVAQ, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.039373890 (76h44m17.910456925s) [noise=true , address=145.79.227.29:4942 , piece_id=ZYT75AZ7X56NBH3OVLWL27J5FRRN4WKV5VEIKEXN32E6P5OQNOZQ, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.039816273 (76h44m17.910899318s) [noise=true , address=145.79.227.42:4121 , piece_id=ICWYGS3AGWC47JS4PUC4GHT2YOV5UTD5RYL73JFTWZRNEMAGMRZA, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.040292149 (76h44m17.911375194s) [noise=true , address=145.79.227.32:4834 , piece_id=WDZ3N3A3CNPAECT7HW2VZ2K7JEJVOZJCZR3BKHRBVR3XHGPGTKPQ, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.040745432 (76h44m17.911828477s) [noise=true , address=145.79.227.31:4799 , piece_id=RNPR7HC6YANTTQW7STMNTHYANE7ZD5XC2N5DMLQQ5YBULNUSFK7A, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.041266104 (76h44m17.912349139s) [noise=true , address=145.79.227.20:4437 , piece_id=G76GCXUI5YH3CMQV47V6FSFDDSOBIIFMS24UJVTNXRR2MM4UZ5ZQ, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.041787887 (76h44m17.912870952s) [noise=true , address=145.79.227.25:4619 , piece_id=5M5PTNON4YZ7INE34X7I6BIJVQF543VQWALEZBCZEEEQ26UMGYQA, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.042946814 (76h44m17.914029849s) [noise=true , address=145.79.227.16:4912 , piece_id=J7BVXTHOSEPXKDDWXPV7JIUWW5NNWLETI7S4IOVPDG6ZUVPATOXQ, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.044580184 (76h44m17.915663219s) [noise=true , address=145.79.227.27:4691 , piece_id=R5VVIJNPYZIDQW5WSJMEYS4D55AHXXMACQY5GM5IOJXVKNOGBEAQ, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.047199231 (76h44m17.918282266s) [noise=true , address=145.79.227.16:4910 , piece_id=YIHYKALCHWYGMAUKS3RPUQ2WOAQKCRQV6GZPFM27K37CKTTJSDPA, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.049620632 (76h44m17.920703677s) [noise=true , address=145.79.227.20:4438 , piece_id=QZQULVSGCAOTQEE5MBUONJYAJIO3GPUNKB4CEVMK3NA2J3BYZ6GA, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.051625078 (76h44m17.922708143s) [noise=true , address=145.79.227.22:4511 , piece_id=CFUDAP3RXMYJCS5GQJVYCKXTN6RLXG7NSQH6EHTZPSYCXPSPBEFQ, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.053186752 (76h44m17.924269787s) [noise=true , address=145.79.227.39:4015 , piece_id=UX4HGNQIHC3FCNGUGIAZYPNYYTLNXMYIND7H7PZ6DCD5EXS67Y7Q, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.054291115 (76h44m17.92537415s) [noise=true , address=77.90.31.10:4517 , piece_id=BMXTXAXNYZ5U5T54U57PMGP4KZO332K2VEP4KO5NEYC7TZ2S53SA, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:19.111879942 (76h44m18.982962987s) [noise=true , address=77.90.31.10:4517 , piece_id=BMXTXAXNYZ5U5T54U57PMGP4KZO332K2VEP4KO5NEYC7TZ2S53SA, seg_pos={0 0}, upload=67101]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: storage node overloaded, request limit: 2000 Dec 8 20:13:19.643615670 (76h44m19.514698706s) [upload=67101]: Next returned nil segment Dec 8 20:13:19.643629617 (76h44m19.514712652s) [upload=67101]: Waiting for error group managing segments... Dec 8 20:13:20.026246645 (76h44m19.89732969s) [noise=true , address=145.79.227.32:4834 , piece_id=WDZ3N3A3CNPAECT7HW2VZ2K7JEJVOZJCZR3BKHRBVR3XHGPGTKPQ, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.027742514 (76h44m19.898825559s) [noise=true , address=145.79.227.25:4619 , piece_id=5M5PTNON4YZ7INE34X7I6BIJVQF543VQWALEZBCZEEEQ26UMGYQA, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.028492581 (76h44m19.899575616s) [noise=true , address=145.79.227.41:4085 , piece_id=FFFFOS7PH3SN3S4NWJJRYP772D725TSGCBKXIZ7FTEDBPO3TNR3Q, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.029004767 (76h44m19.900087802s) [noise=true , address=145.79.227.16:4912 , piece_id=J7BVXTHOSEPXKDDWXPV7JIUWW5NNWLETI7S4IOVPDG6ZUVPATOXQ, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.029438392 (76h44m19.900521427s) [noise=true , address=145.79.227.24:4583 , piece_id=MTNWT2AOTPYXLNVPSOKK5ZOAY7U3UH376V232TZKE56UM2CMDVAQ, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.031410738 (76h44m19.902493783s) [noise=true , address=145.79.227.42:4121 , piece_id=ICWYGS3AGWC47JS4PUC4GHT2YOV5UTD5RYL73JFTWZRNEMAGMRZA, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.034141027 (76h44m19.905224072s) [noise=true , address=145.79.227.31:4799 , piece_id=RNPR7HC6YANTTQW7STMNTHYANE7ZD5XC2N5DMLQQ5YBULNUSFK7A, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.037360508 (76h44m19.908443543s) [noise=true , address=145.79.227.22:4511 , piece_id=CFUDAP3RXMYJCS5GQJVYCKXTN6RLXG7NSQH6EHTZPSYCXPSPBEFQ, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.038059108 (76h44m19.909142153s) [noise=true , address=145.79.227.39:4015 , piece_id=UX4HGNQIHC3FCNGUGIAZYPNYYTLNXMYIND7H7PZ6DCD5EXS67Y7Q, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.053205520 (76h44m19.924288565s) [noise=true , address=145.79.227.16:4910 , piece_id=YIHYKALCHWYGMAUKS3RPUQ2WOAQKCRQV6GZPFM27K37CKTTJSDPA, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.053898059 (76h44m19.924981114s) [noise=true , address=145.79.227.20:4437 , piece_id=G76GCXUI5YH3CMQV47V6FSFDDSOBIIFMS24UJVTNXRR2MM4UZ5ZQ, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.054182971 (76h44m19.925266006s) [noise=true , address=145.79.227.27:4691 , piece_id=R5VVIJNPYZIDQW5WSJMEYS4D55AHXXMACQY5GM5IOJXVKNOGBEAQ, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.058658112 (76h44m19.929741157s) [noise=true , address=145.79.227.29:4942 , piece_id=ZYT75AZ7X56NBH3OVLWL27J5FRRN4WKV5VEIKEXN32E6P5OQNOZQ, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.058679072 (76h44m19.929762107s) [seg_pos={0 0}, upload=67101]: Segment reached optimal threshold of 13 pieces. Dec 8 20:13:20.058873853 (76h44m19.929956888s) [noise=true , address=145.79.227.31:4797 , piece_id=IHEVQ2JUPASTR3CU7QMNAQVSYQ65LEV5SR2KO4GCEOKAKQR5Y5NA, seg_pos={0 0}, upload=67101]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9): protocol: expected piece hash; context canceled; EOF Dec 8 20:13:20.058880526 (76h44m19.929963561s) [noise=true , address=145.79.227.31:4797 , piece_id=IHEVQ2JUPASTR3CU7QMNAQVSYQ65LEV5SR2KO4GCEOKAKQR5Y5NA, seg_pos={0 0}, upload=67101]: Long tail cancellation due to optimal threshold Dec 8 20:13:20.059077551 (76h44m19.930160586s) [noise=true , address=145.79.227.20:4438 , piece_id=QZQULVSGCAOTQEE5MBUONJYAJIO3GPUNKB4CEVMK3NA2J3BYZ6GA, seg_pos={0 0}, upload=67101]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT): protocol: expected piece hash; context canceled; EOF Dec 8 20:13:20.059093532 (76h44m19.930176567s) [noise=true , address=145.79.227.20:4438 , piece_id=QZQULVSGCAOTQEE5MBUONJYAJIO3GPUNKB4CEVMK3NA2J3BYZ6GA, seg_pos={0 0}, upload=67101]: Long tail cancellation due to optimal threshold Dec 8 20:13:20.059116485 (76h44m19.93019953s) [seg_pos={0 0}, upload=67101]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 20:13:20.059191879 (76h44m19.930274924s) [upload=67101]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 20:13:20.059196798 (76h44m19.930279833s) [upload=67101]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 20:19:41.003042505 (76h50m40.87412554s) [upload=67101]: Done uploading segments... Dec 8 20:19:41.003155129 (76h50m40.874238175s) [upload=67101]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.778488614s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 4.979065843s trace: https://jaeger.dev.storj.io/trace/49fc2034659f6881 FAIL (2025-12-08 20:37:30 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.979094608s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [...........................X...............X................] duration: 5.461121182s trace: https://jaeger.dev.storj.io/trace/3996915dd8f30735 FAIL (2025-12-08 20:37:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== PGM26LWTYTWGZGQDPBL7UDX263DFBXY7WZRXGMYEFPWXQQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.569231809 (76h54m34.440315054s) [noise=true , address=89.149.201.147:4022 , piece_id=2VR3E4SXAAOR5RAVHFVUQQ7MBZQEEQYFZ2ZW5Z7YSCVOETZNFQWQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.569369882 (76h54m34.440452917s) [noise=true , address=185.167.204.206:4037 , piece_id=XYY5NAATQJPHOK6JAMHXR3JH3QTLQF7OTA32BMST5HYQDYJ76UMA, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.569672879 (76h54m34.440755924s) [noise=true , address=185.167.204.206:4056 , piece_id=IPDW6RCHJFLMB5LOZM3FG645CCW6TT7KJW4PBQXBUSAIE2ZZM74A, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.569779642 (76h54m34.440862677s) [noise=true , address=185.167.204.205:4024 , piece_id=2UHV7KDAKJJ4DDUU5MPBNLRP24OK5RQU6GGAVYUV5MEFLDDPGUZQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.570073130 (76h54m34.441156176s) [noise=true , address=81.171.14.70:4035 , piece_id=ZKLJ5VUPALKZBSPFCJQBZZUTMJ3PX2H3XMUZ5JVSW5BJBACRUYHA, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.570428618 (76h54m34.441511653s) [noise=true , address=81.171.14.70:4043 , piece_id=FVUGWZXJLQAOLUSTRKJAULSHDWKX5GGWA2BB52YP7WQEKU4NKS5Q, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.570785237 (76h54m34.441868282s) [noise=true , address=81.171.14.70:4044 , piece_id=QPYQGR62JDWWV3GM3CVDB6BLNSGNUTQUEHHQF74TYXFN25PMNPVQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.571235034 (76h54m34.442318069s) [noise=true , address=185.167.204.205:4035 , piece_id=LXUP2T5SN5SSV7IWZLGTSLCRWLFRZBODXX7W4FFVHZFO74I2CQHQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.571655133 (76h54m34.442738169s) [noise=true , address=81.171.14.70:4031 , piece_id=53QOUYYU4KS53TELJUM3FPEKOCB464OR5GKYGYB54RVRPD52RFWA, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.572102666 (76h54m34.443185701s) [noise=true , address=185.167.204.205:4034 , piece_id=4VH6HEDD7CTPMEOHYGKFXSYXHKOLPYB2N6BCCJT2DXVAMADI3KJQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:35.395419290 (76h54m35.266502335s) [upload=67229]: Next returned nil segment Dec 8 20:23:35.395432614 (76h54m35.26651565s) [upload=67229]: Waiting for error group managing segments... Dec 8 20:23:35.780649683 (76h54m35.651732718s) [noise=true , address=185.167.204.206:4037 , piece_id=XYY5NAATQJPHOK6JAMHXR3JH3QTLQF7OTA32BMST5HYQDYJ76UMA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.781236982 (76h54m35.652320027s) [noise=true , address=185.167.204.206:4056 , piece_id=IPDW6RCHJFLMB5LOZM3FG645CCW6TT7KJW4PBQXBUSAIE2ZZM74A, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.784267011 (76h54m35.655350046s) [noise=true , address=185.167.204.205:4030 , piece_id=O3QKGK2G5I5YYPRP5K57RTG4H35H2UDFROFU2JMSHKTFNZJDVUIA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.785164540 (76h54m35.656247585s) [noise=true , address=185.167.204.205:4012 , piece_id=QCJD2IHH4DEKTFN5ZPMZXPECBQ2IX2QPSFUZCSFR5HKCDOVYGGQA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.785585301 (76h54m35.656668336s) [noise=true , address=185.167.204.205:4002 , piece_id=PDSJ5KDCFQBURILRTWU2WSEG3UH7GX4GK57W4J75QADBPKQXRSYA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.787925598 (76h54m35.659008643s) [noise=true , address=185.167.204.206:4047 , piece_id=THE6CUP46D2OSXS34UR3TTBUP6FIVMGDGT7DVJDBB57BHMWBCZ7A, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.787994289 (76h54m35.659077324s) [noise=true , address=185.167.204.205:4034 , piece_id=4VH6HEDD7CTPMEOHYGKFXSYXHKOLPYB2N6BCCJT2DXVAMADI3KJQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.788899402 (76h54m35.659982437s) [noise=true , address=185.167.204.206:4050 , piece_id=7B4GLDQZUPQPLGPZPPN5CO3VOX3LLBMLIYA6IMALYITCHE4CZBOA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.791033055 (76h54m35.66211609s) [noise=true , address=185.167.204.205:4031 , piece_id=5XPLPLO4SPMX7ZBYHLUZTH7MBGCRESEEMP2OEI7LTTT5BN67XHMQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.792163116 (76h54m35.663246161s) [noise=true , address=185.167.204.206:4055 , piece_id=JESF36BPW3YEPXVI2GP4OALECNLYSASJHQRZVLNXKV5BQRDGOWWQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.792693356 (76h54m35.663776392s) [noise=true , address=185.167.204.206:4063 , piece_id=BRUAUJTOCPWNUOB6D5G6AJCTMJBRZRJXXXBZBIUUZZXXQB5OEX2Q, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.793171606 (76h54m35.664254662s) [noise=true , address=185.167.204.206:4045 , piece_id=IXV6HTIOVQR4HCGJIWTBQFLO733BMMNNMZ24Y34H6V6KYUQYPQRA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.798151289 (76h54m35.669234344s) [noise=true , address=185.167.204.206:4053 , piece_id=VGK5AKCI23P5ZDS427NS6PNNRIJ3QUXLRZ4ECLYN4UDSL3KAQAKA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.798189562 (76h54m35.669272597s) [noise=true , address=185.167.204.206:4040 , piece_id=4KFICIELLLPCLWQ7RIB7UUSWUK7SFX2PJK7JWCRU5S4HQNHBEWRQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.798633607 (76h54m35.669716642s) [noise=true , address=185.167.204.205:4035 , piece_id=LXUP2T5SN5SSV7IWZLGTSLCRWLFRZBODXX7W4FFVHZFO74I2CQHQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.803059726 (76h54m35.674142761s) [noise=true , address=185.167.204.205:4024 , piece_id=2UHV7KDAKJJ4DDUU5MPBNLRP24OK5RQU6GGAVYUV5MEFLDDPGUZQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.344154907 (76h54m36.215237952s) [noise=true , address=81.171.14.70:4036 , piece_id=FB3SB3XQDNT2HSK5FDGELGYWODFLU655QW4D23GEXIWQEZYFNFRA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.359261713 (76h54m36.230344758s) [noise=true , address=81.171.14.70:4031 , piece_id=53QOUYYU4KS53TELJUM3FPEKOCB464OR5GKYGYB54RVRPD52RFWA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.393865115 (76h54m36.26494815s) [noise=true , address=81.171.14.70:4035 , piece_id=ZKLJ5VUPALKZBSPFCJQBZZUTMJ3PX2H3XMUZ5JVSW5BJBACRUYHA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.394302157 (76h54m36.265385202s) [noise=true , address=81.171.14.70:4044 , piece_id=QPYQGR62JDWWV3GM3CVDB6BLNSGNUTQUEHHQF74TYXFN25PMNPVQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.430200674 (76h54m36.301283709s) [noise=true , address=81.171.14.70:4046 , piece_id=ZHZUULNXPFDSTOXXH6T7XIEPN23OOWCNAAUWPCZYNNDTLPRWXVKQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.800799297 (76h54m36.671882332s) [noise=true , address=81.171.14.70:4040 , piece_id=W2SOTP5SPLQ6J3PM57NZOLRVZNMTNSGPWDZKZC2KV2PBPH4TUBLA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.862864259 (76h54m36.733947295s) [noise=true , address=81.171.14.70:4038 , piece_id=CKN2RF2YSTEOK6HUXGVBVECYAP2CLGUF6KYNR5JTHRUZPTGVI73Q, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.953769972 (76h54m36.824853007s) [noise=true , address=81.171.14.70:4043 , piece_id=FVUGWZXJLQAOLUSTRKJAULSHDWKX5GGWA2BB52YP7WQEKU4NKS5Q, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.996603376 (76h54m36.867686421s) [noise=true , address=89.149.201.147:4006 , piece_id=UGREZBZRVKXMGBOLQEV2NC4COTXOTTRUEAGSPNWRWP22MCPHQP7Q, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:37.037518916 (76h54m36.908601951s) [noise=true , address=89.149.201.147:4014 , piece_id=AEG6KGPGM26LWTYTWGZGQDPBL7UDX263DFBXY7WZRXGMYEFPWXQQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:37.037537541 (76h54m36.908620586s) [seg_pos={0 0}, upload=67229]: Segment reached optimal threshold of 26 pieces. Dec 8 20:23:37.037730880 (76h54m36.908813915s) [noise=true , address=89.149.201.147:4023 , piece_id=5EPGXWBK6GNTKLQLWEIWYBC2OJHFR3M7F4X3X4WODJWNXID3PCQQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BaeSxjXneBKoGKVrRM686sq386Bhn1YzzBFG9we8fHbcc9tbv): protocol: expected piece hash; context canceled; EOF Dec 8 20:23:37.037737452 (76h54m36.908820497s) [noise=true , address=89.149.201.147:4023 , piece_id=5EPGXWBK6GNTKLQLWEIWYBC2OJHFR3M7F4X3X4WODJWNXID3PCQQ, seg_pos={0 0}, upload=67229]: Long tail cancellation due to optimal threshold Dec 8 20:23:37.037893039 (76h54m36.908976074s) [noise=true , address=89.149.201.147:4022 , piece_id=2VR3E4SXAAOR5RAVHFVUQQ7MBZQEEQYFZ2ZW5Z7YSCVOETZNFQWQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122Vj8XxpS6RnhVweQV3cZWFhj8nrzy3hZ9P7JsuTzEaZcD6iFE): protocol: expected piece hash; context canceled; EOF Dec 8 20:23:37.037900964 (76h54m36.908984019s) [noise=true , address=89.149.201.147:4022 , piece_id=2VR3E4SXAAOR5RAVHFVUQQ7MBZQEEQYFZ2ZW5Z7YSCVOETZNFQWQ, seg_pos={0 0}, upload=67229]: Long tail cancellation due to optimal threshold Dec 8 20:23:37.037926843 (76h54m36.909009889s) [seg_pos={0 0}, upload=67229]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 8 20:23:37.038007697 (76h54m36.909090732s) [upload=67229]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 20:23:37.038013398 (76h54m36.909096433s) [upload=67229]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 20:26:49.742746371 (76h57m49.613829416s) [upload=67229]: Done uploading segments... Dec 8 20:26:49.742860908 (76h57m49.613943943s) [upload=67229]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.461148535s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.....X.......................X....X....X....................] duration: 6.00108984s trace: https://jaeger.dev.storj.io/trace/4600f1753cc382a2 FAIL (2025-12-08 20:37:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.001112983s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................X.........X...............................] duration: 6.126963166s trace: https://jaeger.dev.storj.io/trace/6fb9d317d1dcf919 FAIL (2025-12-08 20:37:14 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.126993686s core uplink us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 4.768267021s trace: https://jaeger.dev.storj.io/trace/49fc2034659f6881 FAIL (2025-12-08 20:36:58 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.768285927s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [............................X...............X...............] duration: 5.281209244s trace: https://jaeger.dev.storj.io/trace/3996915dd8f30735 FAIL (2025-12-08 20:36:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== PGM26LWTYTWGZGQDPBL7UDX263DFBXY7WZRXGMYEFPWXQQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.569231809 (76h54m34.440315054s) [noise=true , address=89.149.201.147:4022 , piece_id=2VR3E4SXAAOR5RAVHFVUQQ7MBZQEEQYFZ2ZW5Z7YSCVOETZNFQWQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.569369882 (76h54m34.440452917s) [noise=true , address=185.167.204.206:4037 , piece_id=XYY5NAATQJPHOK6JAMHXR3JH3QTLQF7OTA32BMST5HYQDYJ76UMA, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.569672879 (76h54m34.440755924s) [noise=true , address=185.167.204.206:4056 , piece_id=IPDW6RCHJFLMB5LOZM3FG645CCW6TT7KJW4PBQXBUSAIE2ZZM74A, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.569779642 (76h54m34.440862677s) [noise=true , address=185.167.204.205:4024 , piece_id=2UHV7KDAKJJ4DDUU5MPBNLRP24OK5RQU6GGAVYUV5MEFLDDPGUZQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.570073130 (76h54m34.441156176s) [noise=true , address=81.171.14.70:4035 , piece_id=ZKLJ5VUPALKZBSPFCJQBZZUTMJ3PX2H3XMUZ5JVSW5BJBACRUYHA, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.570428618 (76h54m34.441511653s) [noise=true , address=81.171.14.70:4043 , piece_id=FVUGWZXJLQAOLUSTRKJAULSHDWKX5GGWA2BB52YP7WQEKU4NKS5Q, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.570785237 (76h54m34.441868282s) [noise=true , address=81.171.14.70:4044 , piece_id=QPYQGR62JDWWV3GM3CVDB6BLNSGNUTQUEHHQF74TYXFN25PMNPVQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.571235034 (76h54m34.442318069s) [noise=true , address=185.167.204.205:4035 , piece_id=LXUP2T5SN5SSV7IWZLGTSLCRWLFRZBODXX7W4FFVHZFO74I2CQHQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.571655133 (76h54m34.442738169s) [noise=true , address=81.171.14.70:4031 , piece_id=53QOUYYU4KS53TELJUM3FPEKOCB464OR5GKYGYB54RVRPD52RFWA, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.572102666 (76h54m34.443185701s) [noise=true , address=185.167.204.205:4034 , piece_id=4VH6HEDD7CTPMEOHYGKFXSYXHKOLPYB2N6BCCJT2DXVAMADI3KJQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:35.395419290 (76h54m35.266502335s) [upload=67229]: Next returned nil segment Dec 8 20:23:35.395432614 (76h54m35.26651565s) [upload=67229]: Waiting for error group managing segments... Dec 8 20:23:35.780649683 (76h54m35.651732718s) [noise=true , address=185.167.204.206:4037 , piece_id=XYY5NAATQJPHOK6JAMHXR3JH3QTLQF7OTA32BMST5HYQDYJ76UMA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.781236982 (76h54m35.652320027s) [noise=true , address=185.167.204.206:4056 , piece_id=IPDW6RCHJFLMB5LOZM3FG645CCW6TT7KJW4PBQXBUSAIE2ZZM74A, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.784267011 (76h54m35.655350046s) [noise=true , address=185.167.204.205:4030 , piece_id=O3QKGK2G5I5YYPRP5K57RTG4H35H2UDFROFU2JMSHKTFNZJDVUIA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.785164540 (76h54m35.656247585s) [noise=true , address=185.167.204.205:4012 , piece_id=QCJD2IHH4DEKTFN5ZPMZXPECBQ2IX2QPSFUZCSFR5HKCDOVYGGQA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.785585301 (76h54m35.656668336s) [noise=true , address=185.167.204.205:4002 , piece_id=PDSJ5KDCFQBURILRTWU2WSEG3UH7GX4GK57W4J75QADBPKQXRSYA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.787925598 (76h54m35.659008643s) [noise=true , address=185.167.204.206:4047 , piece_id=THE6CUP46D2OSXS34UR3TTBUP6FIVMGDGT7DVJDBB57BHMWBCZ7A, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.787994289 (76h54m35.659077324s) [noise=true , address=185.167.204.205:4034 , piece_id=4VH6HEDD7CTPMEOHYGKFXSYXHKOLPYB2N6BCCJT2DXVAMADI3KJQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.788899402 (76h54m35.659982437s) [noise=true , address=185.167.204.206:4050 , piece_id=7B4GLDQZUPQPLGPZPPN5CO3VOX3LLBMLIYA6IMALYITCHE4CZBOA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.791033055 (76h54m35.66211609s) [noise=true , address=185.167.204.205:4031 , piece_id=5XPLPLO4SPMX7ZBYHLUZTH7MBGCRESEEMP2OEI7LTTT5BN67XHMQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.792163116 (76h54m35.663246161s) [noise=true , address=185.167.204.206:4055 , piece_id=JESF36BPW3YEPXVI2GP4OALECNLYSASJHQRZVLNXKV5BQRDGOWWQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.792693356 (76h54m35.663776392s) [noise=true , address=185.167.204.206:4063 , piece_id=BRUAUJTOCPWNUOB6D5G6AJCTMJBRZRJXXXBZBIUUZZXXQB5OEX2Q, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.793171606 (76h54m35.664254662s) [noise=true , address=185.167.204.206:4045 , piece_id=IXV6HTIOVQR4HCGJIWTBQFLO733BMMNNMZ24Y34H6V6KYUQYPQRA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.798151289 (76h54m35.669234344s) [noise=true , address=185.167.204.206:4053 , piece_id=VGK5AKCI23P5ZDS427NS6PNNRIJ3QUXLRZ4ECLYN4UDSL3KAQAKA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.798189562 (76h54m35.669272597s) [noise=true , address=185.167.204.206:4040 , piece_id=4KFICIELLLPCLWQ7RIB7UUSWUK7SFX2PJK7JWCRU5S4HQNHBEWRQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.798633607 (76h54m35.669716642s) [noise=true , address=185.167.204.205:4035 , piece_id=LXUP2T5SN5SSV7IWZLGTSLCRWLFRZBODXX7W4FFVHZFO74I2CQHQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.803059726 (76h54m35.674142761s) [noise=true , address=185.167.204.205:4024 , piece_id=2UHV7KDAKJJ4DDUU5MPBNLRP24OK5RQU6GGAVYUV5MEFLDDPGUZQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.344154907 (76h54m36.215237952s) [noise=true , address=81.171.14.70:4036 , piece_id=FB3SB3XQDNT2HSK5FDGELGYWODFLU655QW4D23GEXIWQEZYFNFRA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.359261713 (76h54m36.230344758s) [noise=true , address=81.171.14.70:4031 , piece_id=53QOUYYU4KS53TELJUM3FPEKOCB464OR5GKYGYB54RVRPD52RFWA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.393865115 (76h54m36.26494815s) [noise=true , address=81.171.14.70:4035 , piece_id=ZKLJ5VUPALKZBSPFCJQBZZUTMJ3PX2H3XMUZ5JVSW5BJBACRUYHA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.394302157 (76h54m36.265385202s) [noise=true , address=81.171.14.70:4044 , piece_id=QPYQGR62JDWWV3GM3CVDB6BLNSGNUTQUEHHQF74TYXFN25PMNPVQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.430200674 (76h54m36.301283709s) [noise=true , address=81.171.14.70:4046 , piece_id=ZHZUULNXPFDSTOXXH6T7XIEPN23OOWCNAAUWPCZYNNDTLPRWXVKQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.800799297 (76h54m36.671882332s) [noise=true , address=81.171.14.70:4040 , piece_id=W2SOTP5SPLQ6J3PM57NZOLRVZNMTNSGPWDZKZC2KV2PBPH4TUBLA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.862864259 (76h54m36.733947295s) [noise=true , address=81.171.14.70:4038 , piece_id=CKN2RF2YSTEOK6HUXGVBVECYAP2CLGUF6KYNR5JTHRUZPTGVI73Q, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.953769972 (76h54m36.824853007s) [noise=true , address=81.171.14.70:4043 , piece_id=FVUGWZXJLQAOLUSTRKJAULSHDWKX5GGWA2BB52YP7WQEKU4NKS5Q, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.996603376 (76h54m36.867686421s) [noise=true , address=89.149.201.147:4006 , piece_id=UGREZBZRVKXMGBOLQEV2NC4COTXOTTRUEAGSPNWRWP22MCPHQP7Q, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:37.037518916 (76h54m36.908601951s) [noise=true , address=89.149.201.147:4014 , piece_id=AEG6KGPGM26LWTYTWGZGQDPBL7UDX263DFBXY7WZRXGMYEFPWXQQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:37.037537541 (76h54m36.908620586s) [seg_pos={0 0}, upload=67229]: Segment reached optimal threshold of 26 pieces. Dec 8 20:23:37.037730880 (76h54m36.908813915s) [noise=true , address=89.149.201.147:4023 , piece_id=5EPGXWBK6GNTKLQLWEIWYBC2OJHFR3M7F4X3X4WODJWNXID3PCQQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BaeSxjXneBKoGKVrRM686sq386Bhn1YzzBFG9we8fHbcc9tbv): protocol: expected piece hash; context canceled; EOF Dec 8 20:23:37.037737452 (76h54m36.908820497s) [noise=true , address=89.149.201.147:4023 , piece_id=5EPGXWBK6GNTKLQLWEIWYBC2OJHFR3M7F4X3X4WODJWNXID3PCQQ, seg_pos={0 0}, upload=67229]: Long tail cancellation due to optimal threshold Dec 8 20:23:37.037893039 (76h54m36.908976074s) [noise=true , address=89.149.201.147:4022 , piece_id=2VR3E4SXAAOR5RAVHFVUQQ7MBZQEEQYFZ2ZW5Z7YSCVOETZNFQWQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122Vj8XxpS6RnhVweQV3cZWFhj8nrzy3hZ9P7JsuTzEaZcD6iFE): protocol: expected piece hash; context canceled; EOF Dec 8 20:23:37.037900964 (76h54m36.908984019s) [noise=true , address=89.149.201.147:4022 , piece_id=2VR3E4SXAAOR5RAVHFVUQQ7MBZQEEQYFZ2ZW5Z7YSCVOETZNFQWQ, seg_pos={0 0}, upload=67229]: Long tail cancellation due to optimal threshold Dec 8 20:23:37.037926843 (76h54m36.909009889s) [seg_pos={0 0}, upload=67229]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 8 20:23:37.038007697 (76h54m36.909090732s) [upload=67229]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 20:23:37.038013398 (76h54m36.909096433s) [upload=67229]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 20:26:49.742746371 (76h57m49.613829416s) [upload=67229]: Done uploading segments... Dec 8 20:26:49.742860908 (76h57m49.613943943s) [upload=67229]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.281228441s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [.............................X...............X..............] duration: 28.904174504s trace: https://jaeger.dev.storj.io/trace/3996915dd8f30735 FAIL (2025-12-08 20:36:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== PGM26LWTYTWGZGQDPBL7UDX263DFBXY7WZRXGMYEFPWXQQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.569231809 (76h54m34.440315054s) [noise=true , address=89.149.201.147:4022 , piece_id=2VR3E4SXAAOR5RAVHFVUQQ7MBZQEEQYFZ2ZW5Z7YSCVOETZNFQWQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.569369882 (76h54m34.440452917s) [noise=true , address=185.167.204.206:4037 , piece_id=XYY5NAATQJPHOK6JAMHXR3JH3QTLQF7OTA32BMST5HYQDYJ76UMA, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.569672879 (76h54m34.440755924s) [noise=true , address=185.167.204.206:4056 , piece_id=IPDW6RCHJFLMB5LOZM3FG645CCW6TT7KJW4PBQXBUSAIE2ZZM74A, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.569779642 (76h54m34.440862677s) [noise=true , address=185.167.204.205:4024 , piece_id=2UHV7KDAKJJ4DDUU5MPBNLRP24OK5RQU6GGAVYUV5MEFLDDPGUZQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.570073130 (76h54m34.441156176s) [noise=true , address=81.171.14.70:4035 , piece_id=ZKLJ5VUPALKZBSPFCJQBZZUTMJ3PX2H3XMUZ5JVSW5BJBACRUYHA, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.570428618 (76h54m34.441511653s) [noise=true , address=81.171.14.70:4043 , piece_id=FVUGWZXJLQAOLUSTRKJAULSHDWKX5GGWA2BB52YP7WQEKU4NKS5Q, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.570785237 (76h54m34.441868282s) [noise=true , address=81.171.14.70:4044 , piece_id=QPYQGR62JDWWV3GM3CVDB6BLNSGNUTQUEHHQF74TYXFN25PMNPVQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.571235034 (76h54m34.442318069s) [noise=true , address=185.167.204.205:4035 , piece_id=LXUP2T5SN5SSV7IWZLGTSLCRWLFRZBODXX7W4FFVHZFO74I2CQHQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.571655133 (76h54m34.442738169s) [noise=true , address=81.171.14.70:4031 , piece_id=53QOUYYU4KS53TELJUM3FPEKOCB464OR5GKYGYB54RVRPD52RFWA, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.572102666 (76h54m34.443185701s) [noise=true , address=185.167.204.205:4034 , piece_id=4VH6HEDD7CTPMEOHYGKFXSYXHKOLPYB2N6BCCJT2DXVAMADI3KJQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:35.395419290 (76h54m35.266502335s) [upload=67229]: Next returned nil segment Dec 8 20:23:35.395432614 (76h54m35.26651565s) [upload=67229]: Waiting for error group managing segments... Dec 8 20:23:35.780649683 (76h54m35.651732718s) [noise=true , address=185.167.204.206:4037 , piece_id=XYY5NAATQJPHOK6JAMHXR3JH3QTLQF7OTA32BMST5HYQDYJ76UMA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.781236982 (76h54m35.652320027s) [noise=true , address=185.167.204.206:4056 , piece_id=IPDW6RCHJFLMB5LOZM3FG645CCW6TT7KJW4PBQXBUSAIE2ZZM74A, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.784267011 (76h54m35.655350046s) [noise=true , address=185.167.204.205:4030 , piece_id=O3QKGK2G5I5YYPRP5K57RTG4H35H2UDFROFU2JMSHKTFNZJDVUIA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.785164540 (76h54m35.656247585s) [noise=true , address=185.167.204.205:4012 , piece_id=QCJD2IHH4DEKTFN5ZPMZXPECBQ2IX2QPSFUZCSFR5HKCDOVYGGQA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.785585301 (76h54m35.656668336s) [noise=true , address=185.167.204.205:4002 , piece_id=PDSJ5KDCFQBURILRTWU2WSEG3UH7GX4GK57W4J75QADBPKQXRSYA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.787925598 (76h54m35.659008643s) [noise=true , address=185.167.204.206:4047 , piece_id=THE6CUP46D2OSXS34UR3TTBUP6FIVMGDGT7DVJDBB57BHMWBCZ7A, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.787994289 (76h54m35.659077324s) [noise=true , address=185.167.204.205:4034 , piece_id=4VH6HEDD7CTPMEOHYGKFXSYXHKOLPYB2N6BCCJT2DXVAMADI3KJQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.788899402 (76h54m35.659982437s) [noise=true , address=185.167.204.206:4050 , piece_id=7B4GLDQZUPQPLGPZPPN5CO3VOX3LLBMLIYA6IMALYITCHE4CZBOA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.791033055 (76h54m35.66211609s) [noise=true , address=185.167.204.205:4031 , piece_id=5XPLPLO4SPMX7ZBYHLUZTH7MBGCRESEEMP2OEI7LTTT5BN67XHMQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.792163116 (76h54m35.663246161s) [noise=true , address=185.167.204.206:4055 , piece_id=JESF36BPW3YEPXVI2GP4OALECNLYSASJHQRZVLNXKV5BQRDGOWWQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.792693356 (76h54m35.663776392s) [noise=true , address=185.167.204.206:4063 , piece_id=BRUAUJTOCPWNUOB6D5G6AJCTMJBRZRJXXXBZBIUUZZXXQB5OEX2Q, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.793171606 (76h54m35.664254662s) [noise=true , address=185.167.204.206:4045 , piece_id=IXV6HTIOVQR4HCGJIWTBQFLO733BMMNNMZ24Y34H6V6KYUQYPQRA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.798151289 (76h54m35.669234344s) [noise=true , address=185.167.204.206:4053 , piece_id=VGK5AKCI23P5ZDS427NS6PNNRIJ3QUXLRZ4ECLYN4UDSL3KAQAKA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.798189562 (76h54m35.669272597s) [noise=true , address=185.167.204.206:4040 , piece_id=4KFICIELLLPCLWQ7RIB7UUSWUK7SFX2PJK7JWCRU5S4HQNHBEWRQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.798633607 (76h54m35.669716642s) [noise=true , address=185.167.204.205:4035 , piece_id=LXUP2T5SN5SSV7IWZLGTSLCRWLFRZBODXX7W4FFVHZFO74I2CQHQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.803059726 (76h54m35.674142761s) [noise=true , address=185.167.204.205:4024 , piece_id=2UHV7KDAKJJ4DDUU5MPBNLRP24OK5RQU6GGAVYUV5MEFLDDPGUZQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.344154907 (76h54m36.215237952s) [noise=true , address=81.171.14.70:4036 , piece_id=FB3SB3XQDNT2HSK5FDGELGYWODFLU655QW4D23GEXIWQEZYFNFRA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.359261713 (76h54m36.230344758s) [noise=true , address=81.171.14.70:4031 , piece_id=53QOUYYU4KS53TELJUM3FPEKOCB464OR5GKYGYB54RVRPD52RFWA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.393865115 (76h54m36.26494815s) [noise=true , address=81.171.14.70:4035 , piece_id=ZKLJ5VUPALKZBSPFCJQBZZUTMJ3PX2H3XMUZ5JVSW5BJBACRUYHA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.394302157 (76h54m36.265385202s) [noise=true , address=81.171.14.70:4044 , piece_id=QPYQGR62JDWWV3GM3CVDB6BLNSGNUTQUEHHQF74TYXFN25PMNPVQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.430200674 (76h54m36.301283709s) [noise=true , address=81.171.14.70:4046 , piece_id=ZHZUULNXPFDSTOXXH6T7XIEPN23OOWCNAAUWPCZYNNDTLPRWXVKQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.800799297 (76h54m36.671882332s) [noise=true , address=81.171.14.70:4040 , piece_id=W2SOTP5SPLQ6J3PM57NZOLRVZNMTNSGPWDZKZC2KV2PBPH4TUBLA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.862864259 (76h54m36.733947295s) [noise=true , address=81.171.14.70:4038 , piece_id=CKN2RF2YSTEOK6HUXGVBVECYAP2CLGUF6KYNR5JTHRUZPTGVI73Q, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.953769972 (76h54m36.824853007s) [noise=true , address=81.171.14.70:4043 , piece_id=FVUGWZXJLQAOLUSTRKJAULSHDWKX5GGWA2BB52YP7WQEKU4NKS5Q, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.996603376 (76h54m36.867686421s) [noise=true , address=89.149.201.147:4006 , piece_id=UGREZBZRVKXMGBOLQEV2NC4COTXOTTRUEAGSPNWRWP22MCPHQP7Q, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:37.037518916 (76h54m36.908601951s) [noise=true , address=89.149.201.147:4014 , piece_id=AEG6KGPGM26LWTYTWGZGQDPBL7UDX263DFBXY7WZRXGMYEFPWXQQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:37.037537541 (76h54m36.908620586s) [seg_pos={0 0}, upload=67229]: Segment reached optimal threshold of 26 pieces. Dec 8 20:23:37.037730880 (76h54m36.908813915s) [noise=true , address=89.149.201.147:4023 , piece_id=5EPGXWBK6GNTKLQLWEIWYBC2OJHFR3M7F4X3X4WODJWNXID3PCQQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BaeSxjXneBKoGKVrRM686sq386Bhn1YzzBFG9we8fHbcc9tbv): protocol: expected piece hash; context canceled; EOF Dec 8 20:23:37.037737452 (76h54m36.908820497s) [noise=true , address=89.149.201.147:4023 , piece_id=5EPGXWBK6GNTKLQLWEIWYBC2OJHFR3M7F4X3X4WODJWNXID3PCQQ, seg_pos={0 0}, upload=67229]: Long tail cancellation due to optimal threshold Dec 8 20:23:37.037893039 (76h54m36.908976074s) [noise=true , address=89.149.201.147:4022 , piece_id=2VR3E4SXAAOR5RAVHFVUQQ7MBZQEEQYFZ2ZW5Z7YSCVOETZNFQWQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122Vj8XxpS6RnhVweQV3cZWFhj8nrzy3hZ9P7JsuTzEaZcD6iFE): protocol: expected piece hash; context canceled; EOF Dec 8 20:23:37.037900964 (76h54m36.908984019s) [noise=true , address=89.149.201.147:4022 , piece_id=2VR3E4SXAAOR5RAVHFVUQQ7MBZQEEQYFZ2ZW5Z7YSCVOETZNFQWQ, seg_pos={0 0}, upload=67229]: Long tail cancellation due to optimal threshold Dec 8 20:23:37.037926843 (76h54m36.909009889s) [seg_pos={0 0}, upload=67229]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 8 20:23:37.038007697 (76h54m36.909090732s) [upload=67229]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 20:23:37.038013398 (76h54m36.909096433s) [upload=67229]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 20:26:49.742746371 (76h57m49.613829416s) [upload=67229]: Done uploading segments... Dec 8 20:26:49.742860908 (76h57m49.613943943s) [upload=67229]: Upload finished. err: metaclient: context deadline exceeded Duration: 28.904200135s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 5.77704788s trace: https://jaeger.dev.storj.io/trace/690c0830f44a9942 FAIL (2025-12-08 20:36:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 20:13:16.312406346 (76h44m16.183489391s) [upload=67101]: Starting upload Dec 8 20:13:16.312474916 (76h44m16.183557952s) [upload=67101]: Done starting upload Dec 8 20:13:16.312661411 (76h44m16.183744446s) [upload=67101]: Uploading segments... Dec 8 20:13:16.313579429 (76h44m16.184662474s) [upload=67101]: Got next segment. Inline: false Dec 8 20:13:16.313623854 (76h44m16.184706889s) [upload=67101]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 20:13:16.313629435 (76h44m16.18471247s) [upload=67101]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 20:13:18.037990476 (76h44m17.909073521s) [seg_pos={0 0}, upload=67101]: Begin upload segment... Dec 8 20:13:18.038044659 (76h44m17.909127694s) [seg_pos={0 0}, upload=67101]: Done begin upload segment. Dec 8 20:13:18.038075859 (76h44m17.909158914s) [noise=true , address=145.79.227.31:4797 , piece_id=IHEVQ2JUPASTR3CU7QMNAQVSYQ65LEV5SR2KO4GCEOKAKQR5Y5NA, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.038191428 (76h44m17.909274463s) [noise=true , address=145.79.227.41:4085 , piece_id=FFFFOS7PH3SN3S4NWJJRYP772D725TSGCBKXIZ7FTEDBPO3TNR3Q, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.038908023 (76h44m17.909991068s) [noise=true , address=145.79.227.24:4583 , piece_id=MTNWT2AOTPYXLNVPSOKK5ZOAY7U3UH376V232TZKE56UM2CMDVAQ, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.039373890 (76h44m17.910456925s) [noise=true , address=145.79.227.29:4942 , piece_id=ZYT75AZ7X56NBH3OVLWL27J5FRRN4WKV5VEIKEXN32E6P5OQNOZQ, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.039816273 (76h44m17.910899318s) [noise=true , address=145.79.227.42:4121 , piece_id=ICWYGS3AGWC47JS4PUC4GHT2YOV5UTD5RYL73JFTWZRNEMAGMRZA, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.040292149 (76h44m17.911375194s) [noise=true , address=145.79.227.32:4834 , piece_id=WDZ3N3A3CNPAECT7HW2VZ2K7JEJVOZJCZR3BKHRBVR3XHGPGTKPQ, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.040745432 (76h44m17.911828477s) [noise=true , address=145.79.227.31:4799 , piece_id=RNPR7HC6YANTTQW7STMNTHYANE7ZD5XC2N5DMLQQ5YBULNUSFK7A, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.041266104 (76h44m17.912349139s) [noise=true , address=145.79.227.20:4437 , piece_id=G76GCXUI5YH3CMQV47V6FSFDDSOBIIFMS24UJVTNXRR2MM4UZ5ZQ, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.041787887 (76h44m17.912870952s) [noise=true , address=145.79.227.25:4619 , piece_id=5M5PTNON4YZ7INE34X7I6BIJVQF543VQWALEZBCZEEEQ26UMGYQA, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.042946814 (76h44m17.914029849s) [noise=true , address=145.79.227.16:4912 , piece_id=J7BVXTHOSEPXKDDWXPV7JIUWW5NNWLETI7S4IOVPDG6ZUVPATOXQ, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.044580184 (76h44m17.915663219s) [noise=true , address=145.79.227.27:4691 , piece_id=R5VVIJNPYZIDQW5WSJMEYS4D55AHXXMACQY5GM5IOJXVKNOGBEAQ, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.047199231 (76h44m17.918282266s) [noise=true , address=145.79.227.16:4910 , piece_id=YIHYKALCHWYGMAUKS3RPUQ2WOAQKCRQV6GZPFM27K37CKTTJSDPA, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.049620632 (76h44m17.920703677s) [noise=true , address=145.79.227.20:4438 , piece_id=QZQULVSGCAOTQEE5MBUONJYAJIO3GPUNKB4CEVMK3NA2J3BYZ6GA, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.051625078 (76h44m17.922708143s) [noise=true , address=145.79.227.22:4511 , piece_id=CFUDAP3RXMYJCS5GQJVYCKXTN6RLXG7NSQH6EHTZPSYCXPSPBEFQ, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.053186752 (76h44m17.924269787s) [noise=true , address=145.79.227.39:4015 , piece_id=UX4HGNQIHC3FCNGUGIAZYPNYYTLNXMYIND7H7PZ6DCD5EXS67Y7Q, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.054291115 (76h44m17.92537415s) [noise=true , address=77.90.31.10:4517 , piece_id=BMXTXAXNYZ5U5T54U57PMGP4KZO332K2VEP4KO5NEYC7TZ2S53SA, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:19.111879942 (76h44m18.982962987s) [noise=true , address=77.90.31.10:4517 , piece_id=BMXTXAXNYZ5U5T54U57PMGP4KZO332K2VEP4KO5NEYC7TZ2S53SA, seg_pos={0 0}, upload=67101]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: storage node overloaded, request limit: 2000 Dec 8 20:13:19.643615670 (76h44m19.514698706s) [upload=67101]: Next returned nil segment Dec 8 20:13:19.643629617 (76h44m19.514712652s) [upload=67101]: Waiting for error group managing segments... Dec 8 20:13:20.026246645 (76h44m19.89732969s) [noise=true , address=145.79.227.32:4834 , piece_id=WDZ3N3A3CNPAECT7HW2VZ2K7JEJVOZJCZR3BKHRBVR3XHGPGTKPQ, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.027742514 (76h44m19.898825559s) [noise=true , address=145.79.227.25:4619 , piece_id=5M5PTNON4YZ7INE34X7I6BIJVQF543VQWALEZBCZEEEQ26UMGYQA, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.028492581 (76h44m19.899575616s) [noise=true , address=145.79.227.41:4085 , piece_id=FFFFOS7PH3SN3S4NWJJRYP772D725TSGCBKXIZ7FTEDBPO3TNR3Q, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.029004767 (76h44m19.900087802s) [noise=true , address=145.79.227.16:4912 , piece_id=J7BVXTHOSEPXKDDWXPV7JIUWW5NNWLETI7S4IOVPDG6ZUVPATOXQ, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.029438392 (76h44m19.900521427s) [noise=true , address=145.79.227.24:4583 , piece_id=MTNWT2AOTPYXLNVPSOKK5ZOAY7U3UH376V232TZKE56UM2CMDVAQ, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.031410738 (76h44m19.902493783s) [noise=true , address=145.79.227.42:4121 , piece_id=ICWYGS3AGWC47JS4PUC4GHT2YOV5UTD5RYL73JFTWZRNEMAGMRZA, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.034141027 (76h44m19.905224072s) [noise=true , address=145.79.227.31:4799 , piece_id=RNPR7HC6YANTTQW7STMNTHYANE7ZD5XC2N5DMLQQ5YBULNUSFK7A, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.037360508 (76h44m19.908443543s) [noise=true , address=145.79.227.22:4511 , piece_id=CFUDAP3RXMYJCS5GQJVYCKXTN6RLXG7NSQH6EHTZPSYCXPSPBEFQ, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.038059108 (76h44m19.909142153s) [noise=true , address=145.79.227.39:4015 , piece_id=UX4HGNQIHC3FCNGUGIAZYPNYYTLNXMYIND7H7PZ6DCD5EXS67Y7Q, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.053205520 (76h44m19.924288565s) [noise=true , address=145.79.227.16:4910 , piece_id=YIHYKALCHWYGMAUKS3RPUQ2WOAQKCRQV6GZPFM27K37CKTTJSDPA, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.053898059 (76h44m19.924981114s) [noise=true , address=145.79.227.20:4437 , piece_id=G76GCXUI5YH3CMQV47V6FSFDDSOBIIFMS24UJVTNXRR2MM4UZ5ZQ, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.054182971 (76h44m19.925266006s) [noise=true , address=145.79.227.27:4691 , piece_id=R5VVIJNPYZIDQW5WSJMEYS4D55AHXXMACQY5GM5IOJXVKNOGBEAQ, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.058658112 (76h44m19.929741157s) [noise=true , address=145.79.227.29:4942 , piece_id=ZYT75AZ7X56NBH3OVLWL27J5FRRN4WKV5VEIKEXN32E6P5OQNOZQ, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.058679072 (76h44m19.929762107s) [seg_pos={0 0}, upload=67101]: Segment reached optimal threshold of 13 pieces. Dec 8 20:13:20.058873853 (76h44m19.929956888s) [noise=true , address=145.79.227.31:4797 , piece_id=IHEVQ2JUPASTR3CU7QMNAQVSYQ65LEV5SR2KO4GCEOKAKQR5Y5NA, seg_pos={0 0}, upload=67101]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9): protocol: expected piece hash; context canceled; EOF Dec 8 20:13:20.058880526 (76h44m19.929963561s) [noise=true , address=145.79.227.31:4797 , piece_id=IHEVQ2JUPASTR3CU7QMNAQVSYQ65LEV5SR2KO4GCEOKAKQR5Y5NA, seg_pos={0 0}, upload=67101]: Long tail cancellation due to optimal threshold Dec 8 20:13:20.059077551 (76h44m19.930160586s) [noise=true , address=145.79.227.20:4438 , piece_id=QZQULVSGCAOTQEE5MBUONJYAJIO3GPUNKB4CEVMK3NA2J3BYZ6GA, seg_pos={0 0}, upload=67101]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT): protocol: expected piece hash; context canceled; EOF Dec 8 20:13:20.059093532 (76h44m19.930176567s) [noise=true , address=145.79.227.20:4438 , piece_id=QZQULVSGCAOTQEE5MBUONJYAJIO3GPUNKB4CEVMK3NA2J3BYZ6GA, seg_pos={0 0}, upload=67101]: Long tail cancellation due to optimal threshold Dec 8 20:13:20.059116485 (76h44m19.93019953s) [seg_pos={0 0}, upload=67101]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 20:13:20.059191879 (76h44m19.930274924s) [upload=67101]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 20:13:20.059196798 (76h44m19.930279833s) [upload=67101]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 20:19:41.003042505 (76h50m40.87412554s) [upload=67101]: Done uploading segments... Dec 8 20:19:41.003155129 (76h50m40.874238175s) [upload=67101]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.777080151s core uplink us-select-1
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 466.468789ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/4b87d2bbd73f899c FAIL (2025-12-08 20:36:21 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 466.492253ms core satellite eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [......X.......................X....X....X...................] duration: 5.576569718s trace: https://jaeger.dev.storj.io/trace/4600f1753cc382a2 FAIL (2025-12-08 20:36:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.57659681s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 5.355203278s trace: https://jaeger.dev.storj.io/trace/49fc2034659f6881 FAIL (2025-12-08 20:36:05 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.355232183s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................X.........X..............................] duration: 5.887549113s trace: https://jaeger.dev.storj.io/trace/6fb9d317d1dcf919 FAIL (2025-12-08 20:36:00 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.887572058s core uplink us1
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 470.666783ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/4b87d2bbd73f899c FAIL (2025-12-08 20:35:49 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 470.715044ms core satellite eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 5.829246074s trace: https://jaeger.dev.storj.io/trace/690c0830f44a9942 FAIL (2025-12-08 20:35:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 20:13:16.312406346 (76h44m16.183489391s) [upload=67101]: Starting upload Dec 8 20:13:16.312474916 (76h44m16.183557952s) [upload=67101]: Done starting upload Dec 8 20:13:16.312661411 (76h44m16.183744446s) [upload=67101]: Uploading segments... Dec 8 20:13:16.313579429 (76h44m16.184662474s) [upload=67101]: Got next segment. Inline: false Dec 8 20:13:16.313623854 (76h44m16.184706889s) [upload=67101]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 20:13:16.313629435 (76h44m16.18471247s) [upload=67101]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 20:13:18.037990476 (76h44m17.909073521s) [seg_pos={0 0}, upload=67101]: Begin upload segment... Dec 8 20:13:18.038044659 (76h44m17.909127694s) [seg_pos={0 0}, upload=67101]: Done begin upload segment. Dec 8 20:13:18.038075859 (76h44m17.909158914s) [noise=true , address=145.79.227.31:4797 , piece_id=IHEVQ2JUPASTR3CU7QMNAQVSYQ65LEV5SR2KO4GCEOKAKQR5Y5NA, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.038191428 (76h44m17.909274463s) [noise=true , address=145.79.227.41:4085 , piece_id=FFFFOS7PH3SN3S4NWJJRYP772D725TSGCBKXIZ7FTEDBPO3TNR3Q, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.038908023 (76h44m17.909991068s) [noise=true , address=145.79.227.24:4583 , piece_id=MTNWT2AOTPYXLNVPSOKK5ZOAY7U3UH376V232TZKE56UM2CMDVAQ, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.039373890 (76h44m17.910456925s) [noise=true , address=145.79.227.29:4942 , piece_id=ZYT75AZ7X56NBH3OVLWL27J5FRRN4WKV5VEIKEXN32E6P5OQNOZQ, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.039816273 (76h44m17.910899318s) [noise=true , address=145.79.227.42:4121 , piece_id=ICWYGS3AGWC47JS4PUC4GHT2YOV5UTD5RYL73JFTWZRNEMAGMRZA, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.040292149 (76h44m17.911375194s) [noise=true , address=145.79.227.32:4834 , piece_id=WDZ3N3A3CNPAECT7HW2VZ2K7JEJVOZJCZR3BKHRBVR3XHGPGTKPQ, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.040745432 (76h44m17.911828477s) [noise=true , address=145.79.227.31:4799 , piece_id=RNPR7HC6YANTTQW7STMNTHYANE7ZD5XC2N5DMLQQ5YBULNUSFK7A, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.041266104 (76h44m17.912349139s) [noise=true , address=145.79.227.20:4437 , piece_id=G76GCXUI5YH3CMQV47V6FSFDDSOBIIFMS24UJVTNXRR2MM4UZ5ZQ, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.041787887 (76h44m17.912870952s) [noise=true , address=145.79.227.25:4619 , piece_id=5M5PTNON4YZ7INE34X7I6BIJVQF543VQWALEZBCZEEEQ26UMGYQA, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.042946814 (76h44m17.914029849s) [noise=true , address=145.79.227.16:4912 , piece_id=J7BVXTHOSEPXKDDWXPV7JIUWW5NNWLETI7S4IOVPDG6ZUVPATOXQ, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.044580184 (76h44m17.915663219s) [noise=true , address=145.79.227.27:4691 , piece_id=R5VVIJNPYZIDQW5WSJMEYS4D55AHXXMACQY5GM5IOJXVKNOGBEAQ, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.047199231 (76h44m17.918282266s) [noise=true , address=145.79.227.16:4910 , piece_id=YIHYKALCHWYGMAUKS3RPUQ2WOAQKCRQV6GZPFM27K37CKTTJSDPA, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.049620632 (76h44m17.920703677s) [noise=true , address=145.79.227.20:4438 , piece_id=QZQULVSGCAOTQEE5MBUONJYAJIO3GPUNKB4CEVMK3NA2J3BYZ6GA, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.051625078 (76h44m17.922708143s) [noise=true , address=145.79.227.22:4511 , piece_id=CFUDAP3RXMYJCS5GQJVYCKXTN6RLXG7NSQH6EHTZPSYCXPSPBEFQ, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.053186752 (76h44m17.924269787s) [noise=true , address=145.79.227.39:4015 , piece_id=UX4HGNQIHC3FCNGUGIAZYPNYYTLNXMYIND7H7PZ6DCD5EXS67Y7Q, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.054291115 (76h44m17.92537415s) [noise=true , address=77.90.31.10:4517 , piece_id=BMXTXAXNYZ5U5T54U57PMGP4KZO332K2VEP4KO5NEYC7TZ2S53SA, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:19.111879942 (76h44m18.982962987s) [noise=true , address=77.90.31.10:4517 , piece_id=BMXTXAXNYZ5U5T54U57PMGP4KZO332K2VEP4KO5NEYC7TZ2S53SA, seg_pos={0 0}, upload=67101]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: storage node overloaded, request limit: 2000 Dec 8 20:13:19.643615670 (76h44m19.514698706s) [upload=67101]: Next returned nil segment Dec 8 20:13:19.643629617 (76h44m19.514712652s) [upload=67101]: Waiting for error group managing segments... Dec 8 20:13:20.026246645 (76h44m19.89732969s) [noise=true , address=145.79.227.32:4834 , piece_id=WDZ3N3A3CNPAECT7HW2VZ2K7JEJVOZJCZR3BKHRBVR3XHGPGTKPQ, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.027742514 (76h44m19.898825559s) [noise=true , address=145.79.227.25:4619 , piece_id=5M5PTNON4YZ7INE34X7I6BIJVQF543VQWALEZBCZEEEQ26UMGYQA, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.028492581 (76h44m19.899575616s) [noise=true , address=145.79.227.41:4085 , piece_id=FFFFOS7PH3SN3S4NWJJRYP772D725TSGCBKXIZ7FTEDBPO3TNR3Q, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.029004767 (76h44m19.900087802s) [noise=true , address=145.79.227.16:4912 , piece_id=J7BVXTHOSEPXKDDWXPV7JIUWW5NNWLETI7S4IOVPDG6ZUVPATOXQ, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.029438392 (76h44m19.900521427s) [noise=true , address=145.79.227.24:4583 , piece_id=MTNWT2AOTPYXLNVPSOKK5ZOAY7U3UH376V232TZKE56UM2CMDVAQ, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.031410738 (76h44m19.902493783s) [noise=true , address=145.79.227.42:4121 , piece_id=ICWYGS3AGWC47JS4PUC4GHT2YOV5UTD5RYL73JFTWZRNEMAGMRZA, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.034141027 (76h44m19.905224072s) [noise=true , address=145.79.227.31:4799 , piece_id=RNPR7HC6YANTTQW7STMNTHYANE7ZD5XC2N5DMLQQ5YBULNUSFK7A, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.037360508 (76h44m19.908443543s) [noise=true , address=145.79.227.22:4511 , piece_id=CFUDAP3RXMYJCS5GQJVYCKXTN6RLXG7NSQH6EHTZPSYCXPSPBEFQ, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.038059108 (76h44m19.909142153s) [noise=true , address=145.79.227.39:4015 , piece_id=UX4HGNQIHC3FCNGUGIAZYPNYYTLNXMYIND7H7PZ6DCD5EXS67Y7Q, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.053205520 (76h44m19.924288565s) [noise=true , address=145.79.227.16:4910 , piece_id=YIHYKALCHWYGMAUKS3RPUQ2WOAQKCRQV6GZPFM27K37CKTTJSDPA, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.053898059 (76h44m19.924981114s) [noise=true , address=145.79.227.20:4437 , piece_id=G76GCXUI5YH3CMQV47V6FSFDDSOBIIFMS24UJVTNXRR2MM4UZ5ZQ, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.054182971 (76h44m19.925266006s) [noise=true , address=145.79.227.27:4691 , piece_id=R5VVIJNPYZIDQW5WSJMEYS4D55AHXXMACQY5GM5IOJXVKNOGBEAQ, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.058658112 (76h44m19.929741157s) [noise=true , address=145.79.227.29:4942 , piece_id=ZYT75AZ7X56NBH3OVLWL27J5FRRN4WKV5VEIKEXN32E6P5OQNOZQ, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.058679072 (76h44m19.929762107s) [seg_pos={0 0}, upload=67101]: Segment reached optimal threshold of 13 pieces. Dec 8 20:13:20.058873853 (76h44m19.929956888s) [noise=true , address=145.79.227.31:4797 , piece_id=IHEVQ2JUPASTR3CU7QMNAQVSYQ65LEV5SR2KO4GCEOKAKQR5Y5NA, seg_pos={0 0}, upload=67101]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9): protocol: expected piece hash; context canceled; EOF Dec 8 20:13:20.058880526 (76h44m19.929963561s) [noise=true , address=145.79.227.31:4797 , piece_id=IHEVQ2JUPASTR3CU7QMNAQVSYQ65LEV5SR2KO4GCEOKAKQR5Y5NA, seg_pos={0 0}, upload=67101]: Long tail cancellation due to optimal threshold Dec 8 20:13:20.059077551 (76h44m19.930160586s) [noise=true , address=145.79.227.20:4438 , piece_id=QZQULVSGCAOTQEE5MBUONJYAJIO3GPUNKB4CEVMK3NA2J3BYZ6GA, seg_pos={0 0}, upload=67101]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT): protocol: expected piece hash; context canceled; EOF Dec 8 20:13:20.059093532 (76h44m19.930176567s) [noise=true , address=145.79.227.20:4438 , piece_id=QZQULVSGCAOTQEE5MBUONJYAJIO3GPUNKB4CEVMK3NA2J3BYZ6GA, seg_pos={0 0}, upload=67101]: Long tail cancellation due to optimal threshold Dec 8 20:13:20.059116485 (76h44m19.93019953s) [seg_pos={0 0}, upload=67101]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 20:13:20.059191879 (76h44m19.930274924s) [upload=67101]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 20:13:20.059196798 (76h44m19.930279833s) [upload=67101]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 20:19:41.003042505 (76h50m40.87412554s) [upload=67101]: Done uploading segments... Dec 8 20:19:41.003155129 (76h50m40.874238175s) [upload=67101]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.829282183s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [..............................X...............X.............] duration: 5.103669743s trace: https://jaeger.dev.storj.io/trace/3996915dd8f30735 FAIL (2025-12-08 20:35:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== PGM26LWTYTWGZGQDPBL7UDX263DFBXY7WZRXGMYEFPWXQQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.569231809 (76h54m34.440315054s) [noise=true , address=89.149.201.147:4022 , piece_id=2VR3E4SXAAOR5RAVHFVUQQ7MBZQEEQYFZ2ZW5Z7YSCVOETZNFQWQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.569369882 (76h54m34.440452917s) [noise=true , address=185.167.204.206:4037 , piece_id=XYY5NAATQJPHOK6JAMHXR3JH3QTLQF7OTA32BMST5HYQDYJ76UMA, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.569672879 (76h54m34.440755924s) [noise=true , address=185.167.204.206:4056 , piece_id=IPDW6RCHJFLMB5LOZM3FG645CCW6TT7KJW4PBQXBUSAIE2ZZM74A, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.569779642 (76h54m34.440862677s) [noise=true , address=185.167.204.205:4024 , piece_id=2UHV7KDAKJJ4DDUU5MPBNLRP24OK5RQU6GGAVYUV5MEFLDDPGUZQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.570073130 (76h54m34.441156176s) [noise=true , address=81.171.14.70:4035 , piece_id=ZKLJ5VUPALKZBSPFCJQBZZUTMJ3PX2H3XMUZ5JVSW5BJBACRUYHA, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.570428618 (76h54m34.441511653s) [noise=true , address=81.171.14.70:4043 , piece_id=FVUGWZXJLQAOLUSTRKJAULSHDWKX5GGWA2BB52YP7WQEKU4NKS5Q, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.570785237 (76h54m34.441868282s) [noise=true , address=81.171.14.70:4044 , piece_id=QPYQGR62JDWWV3GM3CVDB6BLNSGNUTQUEHHQF74TYXFN25PMNPVQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.571235034 (76h54m34.442318069s) [noise=true , address=185.167.204.205:4035 , piece_id=LXUP2T5SN5SSV7IWZLGTSLCRWLFRZBODXX7W4FFVHZFO74I2CQHQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.571655133 (76h54m34.442738169s) [noise=true , address=81.171.14.70:4031 , piece_id=53QOUYYU4KS53TELJUM3FPEKOCB464OR5GKYGYB54RVRPD52RFWA, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.572102666 (76h54m34.443185701s) [noise=true , address=185.167.204.205:4034 , piece_id=4VH6HEDD7CTPMEOHYGKFXSYXHKOLPYB2N6BCCJT2DXVAMADI3KJQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:35.395419290 (76h54m35.266502335s) [upload=67229]: Next returned nil segment Dec 8 20:23:35.395432614 (76h54m35.26651565s) [upload=67229]: Waiting for error group managing segments... Dec 8 20:23:35.780649683 (76h54m35.651732718s) [noise=true , address=185.167.204.206:4037 , piece_id=XYY5NAATQJPHOK6JAMHXR3JH3QTLQF7OTA32BMST5HYQDYJ76UMA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.781236982 (76h54m35.652320027s) [noise=true , address=185.167.204.206:4056 , piece_id=IPDW6RCHJFLMB5LOZM3FG645CCW6TT7KJW4PBQXBUSAIE2ZZM74A, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.784267011 (76h54m35.655350046s) [noise=true , address=185.167.204.205:4030 , piece_id=O3QKGK2G5I5YYPRP5K57RTG4H35H2UDFROFU2JMSHKTFNZJDVUIA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.785164540 (76h54m35.656247585s) [noise=true , address=185.167.204.205:4012 , piece_id=QCJD2IHH4DEKTFN5ZPMZXPECBQ2IX2QPSFUZCSFR5HKCDOVYGGQA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.785585301 (76h54m35.656668336s) [noise=true , address=185.167.204.205:4002 , piece_id=PDSJ5KDCFQBURILRTWU2WSEG3UH7GX4GK57W4J75QADBPKQXRSYA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.787925598 (76h54m35.659008643s) [noise=true , address=185.167.204.206:4047 , piece_id=THE6CUP46D2OSXS34UR3TTBUP6FIVMGDGT7DVJDBB57BHMWBCZ7A, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.787994289 (76h54m35.659077324s) [noise=true , address=185.167.204.205:4034 , piece_id=4VH6HEDD7CTPMEOHYGKFXSYXHKOLPYB2N6BCCJT2DXVAMADI3KJQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.788899402 (76h54m35.659982437s) [noise=true , address=185.167.204.206:4050 , piece_id=7B4GLDQZUPQPLGPZPPN5CO3VOX3LLBMLIYA6IMALYITCHE4CZBOA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.791033055 (76h54m35.66211609s) [noise=true , address=185.167.204.205:4031 , piece_id=5XPLPLO4SPMX7ZBYHLUZTH7MBGCRESEEMP2OEI7LTTT5BN67XHMQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.792163116 (76h54m35.663246161s) [noise=true , address=185.167.204.206:4055 , piece_id=JESF36BPW3YEPXVI2GP4OALECNLYSASJHQRZVLNXKV5BQRDGOWWQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.792693356 (76h54m35.663776392s) [noise=true , address=185.167.204.206:4063 , piece_id=BRUAUJTOCPWNUOB6D5G6AJCTMJBRZRJXXXBZBIUUZZXXQB5OEX2Q, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.793171606 (76h54m35.664254662s) [noise=true , address=185.167.204.206:4045 , piece_id=IXV6HTIOVQR4HCGJIWTBQFLO733BMMNNMZ24Y34H6V6KYUQYPQRA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.798151289 (76h54m35.669234344s) [noise=true , address=185.167.204.206:4053 , piece_id=VGK5AKCI23P5ZDS427NS6PNNRIJ3QUXLRZ4ECLYN4UDSL3KAQAKA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.798189562 (76h54m35.669272597s) [noise=true , address=185.167.204.206:4040 , piece_id=4KFICIELLLPCLWQ7RIB7UUSWUK7SFX2PJK7JWCRU5S4HQNHBEWRQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.798633607 (76h54m35.669716642s) [noise=true , address=185.167.204.205:4035 , piece_id=LXUP2T5SN5SSV7IWZLGTSLCRWLFRZBODXX7W4FFVHZFO74I2CQHQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.803059726 (76h54m35.674142761s) [noise=true , address=185.167.204.205:4024 , piece_id=2UHV7KDAKJJ4DDUU5MPBNLRP24OK5RQU6GGAVYUV5MEFLDDPGUZQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.344154907 (76h54m36.215237952s) [noise=true , address=81.171.14.70:4036 , piece_id=FB3SB3XQDNT2HSK5FDGELGYWODFLU655QW4D23GEXIWQEZYFNFRA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.359261713 (76h54m36.230344758s) [noise=true , address=81.171.14.70:4031 , piece_id=53QOUYYU4KS53TELJUM3FPEKOCB464OR5GKYGYB54RVRPD52RFWA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.393865115 (76h54m36.26494815s) [noise=true , address=81.171.14.70:4035 , piece_id=ZKLJ5VUPALKZBSPFCJQBZZUTMJ3PX2H3XMUZ5JVSW5BJBACRUYHA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.394302157 (76h54m36.265385202s) [noise=true , address=81.171.14.70:4044 , piece_id=QPYQGR62JDWWV3GM3CVDB6BLNSGNUTQUEHHQF74TYXFN25PMNPVQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.430200674 (76h54m36.301283709s) [noise=true , address=81.171.14.70:4046 , piece_id=ZHZUULNXPFDSTOXXH6T7XIEPN23OOWCNAAUWPCZYNNDTLPRWXVKQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.800799297 (76h54m36.671882332s) [noise=true , address=81.171.14.70:4040 , piece_id=W2SOTP5SPLQ6J3PM57NZOLRVZNMTNSGPWDZKZC2KV2PBPH4TUBLA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.862864259 (76h54m36.733947295s) [noise=true , address=81.171.14.70:4038 , piece_id=CKN2RF2YSTEOK6HUXGVBVECYAP2CLGUF6KYNR5JTHRUZPTGVI73Q, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.953769972 (76h54m36.824853007s) [noise=true , address=81.171.14.70:4043 , piece_id=FVUGWZXJLQAOLUSTRKJAULSHDWKX5GGWA2BB52YP7WQEKU4NKS5Q, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.996603376 (76h54m36.867686421s) [noise=true , address=89.149.201.147:4006 , piece_id=UGREZBZRVKXMGBOLQEV2NC4COTXOTTRUEAGSPNWRWP22MCPHQP7Q, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:37.037518916 (76h54m36.908601951s) [noise=true , address=89.149.201.147:4014 , piece_id=AEG6KGPGM26LWTYTWGZGQDPBL7UDX263DFBXY7WZRXGMYEFPWXQQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:37.037537541 (76h54m36.908620586s) [seg_pos={0 0}, upload=67229]: Segment reached optimal threshold of 26 pieces. Dec 8 20:23:37.037730880 (76h54m36.908813915s) [noise=true , address=89.149.201.147:4023 , piece_id=5EPGXWBK6GNTKLQLWEIWYBC2OJHFR3M7F4X3X4WODJWNXID3PCQQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BaeSxjXneBKoGKVrRM686sq386Bhn1YzzBFG9we8fHbcc9tbv): protocol: expected piece hash; context canceled; EOF Dec 8 20:23:37.037737452 (76h54m36.908820497s) [noise=true , address=89.149.201.147:4023 , piece_id=5EPGXWBK6GNTKLQLWEIWYBC2OJHFR3M7F4X3X4WODJWNXID3PCQQ, seg_pos={0 0}, upload=67229]: Long tail cancellation due to optimal threshold Dec 8 20:23:37.037893039 (76h54m36.908976074s) [noise=true , address=89.149.201.147:4022 , piece_id=2VR3E4SXAAOR5RAVHFVUQQ7MBZQEEQYFZ2ZW5Z7YSCVOETZNFQWQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122Vj8XxpS6RnhVweQV3cZWFhj8nrzy3hZ9P7JsuTzEaZcD6iFE): protocol: expected piece hash; context canceled; EOF Dec 8 20:23:37.037900964 (76h54m36.908984019s) [noise=true , address=89.149.201.147:4022 , piece_id=2VR3E4SXAAOR5RAVHFVUQQ7MBZQEEQYFZ2ZW5Z7YSCVOETZNFQWQ, seg_pos={0 0}, upload=67229]: Long tail cancellation due to optimal threshold Dec 8 20:23:37.037926843 (76h54m36.909009889s) [seg_pos={0 0}, upload=67229]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 8 20:23:37.038007697 (76h54m36.909090732s) [upload=67229]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 20:23:37.038013398 (76h54m36.909096433s) [upload=67229]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 20:26:49.742746371 (76h57m49.613829416s) [upload=67229]: Done uploading segments... Dec 8 20:26:49.742860908 (76h57m49.613943943s) [upload=67229]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.103698748s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.......X.......................X....X....X..................] duration: 5.639885565s trace: https://jaeger.dev.storj.io/trace/4600f1753cc382a2 FAIL (2025-12-08 20:35:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.639912916s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 4.939459182s trace: https://jaeger.dev.storj.io/trace/49fc2034659f6881 FAIL (2025-12-08 20:35:28 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.939488119s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 5.859999484s trace: https://jaeger.dev.storj.io/trace/690c0830f44a9942 FAIL (2025-12-08 20:35:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 20:13:16.312406346 (76h44m16.183489391s) [upload=67101]: Starting upload Dec 8 20:13:16.312474916 (76h44m16.183557952s) [upload=67101]: Done starting upload Dec 8 20:13:16.312661411 (76h44m16.183744446s) [upload=67101]: Uploading segments... Dec 8 20:13:16.313579429 (76h44m16.184662474s) [upload=67101]: Got next segment. Inline: false Dec 8 20:13:16.313623854 (76h44m16.184706889s) [upload=67101]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 20:13:16.313629435 (76h44m16.18471247s) [upload=67101]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 20:13:18.037990476 (76h44m17.909073521s) [seg_pos={0 0}, upload=67101]: Begin upload segment... Dec 8 20:13:18.038044659 (76h44m17.909127694s) [seg_pos={0 0}, upload=67101]: Done begin upload segment. Dec 8 20:13:18.038075859 (76h44m17.909158914s) [noise=true , address=145.79.227.31:4797 , piece_id=IHEVQ2JUPASTR3CU7QMNAQVSYQ65LEV5SR2KO4GCEOKAKQR5Y5NA, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.038191428 (76h44m17.909274463s) [noise=true , address=145.79.227.41:4085 , piece_id=FFFFOS7PH3SN3S4NWJJRYP772D725TSGCBKXIZ7FTEDBPO3TNR3Q, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.038908023 (76h44m17.909991068s) [noise=true , address=145.79.227.24:4583 , piece_id=MTNWT2AOTPYXLNVPSOKK5ZOAY7U3UH376V232TZKE56UM2CMDVAQ, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.039373890 (76h44m17.910456925s) [noise=true , address=145.79.227.29:4942 , piece_id=ZYT75AZ7X56NBH3OVLWL27J5FRRN4WKV5VEIKEXN32E6P5OQNOZQ, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.039816273 (76h44m17.910899318s) [noise=true , address=145.79.227.42:4121 , piece_id=ICWYGS3AGWC47JS4PUC4GHT2YOV5UTD5RYL73JFTWZRNEMAGMRZA, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.040292149 (76h44m17.911375194s) [noise=true , address=145.79.227.32:4834 , piece_id=WDZ3N3A3CNPAECT7HW2VZ2K7JEJVOZJCZR3BKHRBVR3XHGPGTKPQ, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.040745432 (76h44m17.911828477s) [noise=true , address=145.79.227.31:4799 , piece_id=RNPR7HC6YANTTQW7STMNTHYANE7ZD5XC2N5DMLQQ5YBULNUSFK7A, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.041266104 (76h44m17.912349139s) [noise=true , address=145.79.227.20:4437 , piece_id=G76GCXUI5YH3CMQV47V6FSFDDSOBIIFMS24UJVTNXRR2MM4UZ5ZQ, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.041787887 (76h44m17.912870952s) [noise=true , address=145.79.227.25:4619 , piece_id=5M5PTNON4YZ7INE34X7I6BIJVQF543VQWALEZBCZEEEQ26UMGYQA, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.042946814 (76h44m17.914029849s) [noise=true , address=145.79.227.16:4912 , piece_id=J7BVXTHOSEPXKDDWXPV7JIUWW5NNWLETI7S4IOVPDG6ZUVPATOXQ, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.044580184 (76h44m17.915663219s) [noise=true , address=145.79.227.27:4691 , piece_id=R5VVIJNPYZIDQW5WSJMEYS4D55AHXXMACQY5GM5IOJXVKNOGBEAQ, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.047199231 (76h44m17.918282266s) [noise=true , address=145.79.227.16:4910 , piece_id=YIHYKALCHWYGMAUKS3RPUQ2WOAQKCRQV6GZPFM27K37CKTTJSDPA, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.049620632 (76h44m17.920703677s) [noise=true , address=145.79.227.20:4438 , piece_id=QZQULVSGCAOTQEE5MBUONJYAJIO3GPUNKB4CEVMK3NA2J3BYZ6GA, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.051625078 (76h44m17.922708143s) [noise=true , address=145.79.227.22:4511 , piece_id=CFUDAP3RXMYJCS5GQJVYCKXTN6RLXG7NSQH6EHTZPSYCXPSPBEFQ, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.053186752 (76h44m17.924269787s) [noise=true , address=145.79.227.39:4015 , piece_id=UX4HGNQIHC3FCNGUGIAZYPNYYTLNXMYIND7H7PZ6DCD5EXS67Y7Q, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.054291115 (76h44m17.92537415s) [noise=true , address=77.90.31.10:4517 , piece_id=BMXTXAXNYZ5U5T54U57PMGP4KZO332K2VEP4KO5NEYC7TZ2S53SA, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:19.111879942 (76h44m18.982962987s) [noise=true , address=77.90.31.10:4517 , piece_id=BMXTXAXNYZ5U5T54U57PMGP4KZO332K2VEP4KO5NEYC7TZ2S53SA, seg_pos={0 0}, upload=67101]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: storage node overloaded, request limit: 2000 Dec 8 20:13:19.643615670 (76h44m19.514698706s) [upload=67101]: Next returned nil segment Dec 8 20:13:19.643629617 (76h44m19.514712652s) [upload=67101]: Waiting for error group managing segments... Dec 8 20:13:20.026246645 (76h44m19.89732969s) [noise=true , address=145.79.227.32:4834 , piece_id=WDZ3N3A3CNPAECT7HW2VZ2K7JEJVOZJCZR3BKHRBVR3XHGPGTKPQ, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.027742514 (76h44m19.898825559s) [noise=true , address=145.79.227.25:4619 , piece_id=5M5PTNON4YZ7INE34X7I6BIJVQF543VQWALEZBCZEEEQ26UMGYQA, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.028492581 (76h44m19.899575616s) [noise=true , address=145.79.227.41:4085 , piece_id=FFFFOS7PH3SN3S4NWJJRYP772D725TSGCBKXIZ7FTEDBPO3TNR3Q, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.029004767 (76h44m19.900087802s) [noise=true , address=145.79.227.16:4912 , piece_id=J7BVXTHOSEPXKDDWXPV7JIUWW5NNWLETI7S4IOVPDG6ZUVPATOXQ, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.029438392 (76h44m19.900521427s) [noise=true , address=145.79.227.24:4583 , piece_id=MTNWT2AOTPYXLNVPSOKK5ZOAY7U3UH376V232TZKE56UM2CMDVAQ, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.031410738 (76h44m19.902493783s) [noise=true , address=145.79.227.42:4121 , piece_id=ICWYGS3AGWC47JS4PUC4GHT2YOV5UTD5RYL73JFTWZRNEMAGMRZA, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.034141027 (76h44m19.905224072s) [noise=true , address=145.79.227.31:4799 , piece_id=RNPR7HC6YANTTQW7STMNTHYANE7ZD5XC2N5DMLQQ5YBULNUSFK7A, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.037360508 (76h44m19.908443543s) [noise=true , address=145.79.227.22:4511 , piece_id=CFUDAP3RXMYJCS5GQJVYCKXTN6RLXG7NSQH6EHTZPSYCXPSPBEFQ, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.038059108 (76h44m19.909142153s) [noise=true , address=145.79.227.39:4015 , piece_id=UX4HGNQIHC3FCNGUGIAZYPNYYTLNXMYIND7H7PZ6DCD5EXS67Y7Q, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.053205520 (76h44m19.924288565s) [noise=true , address=145.79.227.16:4910 , piece_id=YIHYKALCHWYGMAUKS3RPUQ2WOAQKCRQV6GZPFM27K37CKTTJSDPA, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.053898059 (76h44m19.924981114s) [noise=true , address=145.79.227.20:4437 , piece_id=G76GCXUI5YH3CMQV47V6FSFDDSOBIIFMS24UJVTNXRR2MM4UZ5ZQ, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.054182971 (76h44m19.925266006s) [noise=true , address=145.79.227.27:4691 , piece_id=R5VVIJNPYZIDQW5WSJMEYS4D55AHXXMACQY5GM5IOJXVKNOGBEAQ, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.058658112 (76h44m19.929741157s) [noise=true , address=145.79.227.29:4942 , piece_id=ZYT75AZ7X56NBH3OVLWL27J5FRRN4WKV5VEIKEXN32E6P5OQNOZQ, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.058679072 (76h44m19.929762107s) [seg_pos={0 0}, upload=67101]: Segment reached optimal threshold of 13 pieces. Dec 8 20:13:20.058873853 (76h44m19.929956888s) [noise=true , address=145.79.227.31:4797 , piece_id=IHEVQ2JUPASTR3CU7QMNAQVSYQ65LEV5SR2KO4GCEOKAKQR5Y5NA, seg_pos={0 0}, upload=67101]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9): protocol: expected piece hash; context canceled; EOF Dec 8 20:13:20.058880526 (76h44m19.929963561s) [noise=true , address=145.79.227.31:4797 , piece_id=IHEVQ2JUPASTR3CU7QMNAQVSYQ65LEV5SR2KO4GCEOKAKQR5Y5NA, seg_pos={0 0}, upload=67101]: Long tail cancellation due to optimal threshold Dec 8 20:13:20.059077551 (76h44m19.930160586s) [noise=true , address=145.79.227.20:4438 , piece_id=QZQULVSGCAOTQEE5MBUONJYAJIO3GPUNKB4CEVMK3NA2J3BYZ6GA, seg_pos={0 0}, upload=67101]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT): protocol: expected piece hash; context canceled; EOF Dec 8 20:13:20.059093532 (76h44m19.930176567s) [noise=true , address=145.79.227.20:4438 , piece_id=QZQULVSGCAOTQEE5MBUONJYAJIO3GPUNKB4CEVMK3NA2J3BYZ6GA, seg_pos={0 0}, upload=67101]: Long tail cancellation due to optimal threshold Dec 8 20:13:20.059116485 (76h44m19.93019953s) [seg_pos={0 0}, upload=67101]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 20:13:20.059191879 (76h44m19.930274924s) [upload=67101]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 20:13:20.059196798 (76h44m19.930279833s) [upload=67101]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 20:19:41.003042505 (76h50m40.87412554s) [upload=67101]: Done uploading segments... Dec 8 20:19:41.003155129 (76h50m40.874238175s) [upload=67101]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.860026414s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................X.........X.............................] duration: 5.978999653s trace: https://jaeger.dev.storj.io/trace/6fb9d317d1dcf919 FAIL (2025-12-08 20:35:10 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.979168184s core uplink us1
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 481.44681ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/4b87d2bbd73f899c FAIL (2025-12-08 20:34:53 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 481.472188ms core satellite eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 5.732804751s trace: https://jaeger.dev.storj.io/trace/690c0830f44a9942 FAIL (2025-12-08 20:34:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 20:13:16.312406346 (76h44m16.183489391s) [upload=67101]: Starting upload Dec 8 20:13:16.312474916 (76h44m16.183557952s) [upload=67101]: Done starting upload Dec 8 20:13:16.312661411 (76h44m16.183744446s) [upload=67101]: Uploading segments... Dec 8 20:13:16.313579429 (76h44m16.184662474s) [upload=67101]: Got next segment. Inline: false Dec 8 20:13:16.313623854 (76h44m16.184706889s) [upload=67101]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 20:13:16.313629435 (76h44m16.18471247s) [upload=67101]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 20:13:18.037990476 (76h44m17.909073521s) [seg_pos={0 0}, upload=67101]: Begin upload segment... Dec 8 20:13:18.038044659 (76h44m17.909127694s) [seg_pos={0 0}, upload=67101]: Done begin upload segment. Dec 8 20:13:18.038075859 (76h44m17.909158914s) [noise=true , address=145.79.227.31:4797 , piece_id=IHEVQ2JUPASTR3CU7QMNAQVSYQ65LEV5SR2KO4GCEOKAKQR5Y5NA, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.038191428 (76h44m17.909274463s) [noise=true , address=145.79.227.41:4085 , piece_id=FFFFOS7PH3SN3S4NWJJRYP772D725TSGCBKXIZ7FTEDBPO3TNR3Q, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.038908023 (76h44m17.909991068s) [noise=true , address=145.79.227.24:4583 , piece_id=MTNWT2AOTPYXLNVPSOKK5ZOAY7U3UH376V232TZKE56UM2CMDVAQ, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.039373890 (76h44m17.910456925s) [noise=true , address=145.79.227.29:4942 , piece_id=ZYT75AZ7X56NBH3OVLWL27J5FRRN4WKV5VEIKEXN32E6P5OQNOZQ, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.039816273 (76h44m17.910899318s) [noise=true , address=145.79.227.42:4121 , piece_id=ICWYGS3AGWC47JS4PUC4GHT2YOV5UTD5RYL73JFTWZRNEMAGMRZA, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.040292149 (76h44m17.911375194s) [noise=true , address=145.79.227.32:4834 , piece_id=WDZ3N3A3CNPAECT7HW2VZ2K7JEJVOZJCZR3BKHRBVR3XHGPGTKPQ, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.040745432 (76h44m17.911828477s) [noise=true , address=145.79.227.31:4799 , piece_id=RNPR7HC6YANTTQW7STMNTHYANE7ZD5XC2N5DMLQQ5YBULNUSFK7A, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.041266104 (76h44m17.912349139s) [noise=true , address=145.79.227.20:4437 , piece_id=G76GCXUI5YH3CMQV47V6FSFDDSOBIIFMS24UJVTNXRR2MM4UZ5ZQ, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.041787887 (76h44m17.912870952s) [noise=true , address=145.79.227.25:4619 , piece_id=5M5PTNON4YZ7INE34X7I6BIJVQF543VQWALEZBCZEEEQ26UMGYQA, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.042946814 (76h44m17.914029849s) [noise=true , address=145.79.227.16:4912 , piece_id=J7BVXTHOSEPXKDDWXPV7JIUWW5NNWLETI7S4IOVPDG6ZUVPATOXQ, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.044580184 (76h44m17.915663219s) [noise=true , address=145.79.227.27:4691 , piece_id=R5VVIJNPYZIDQW5WSJMEYS4D55AHXXMACQY5GM5IOJXVKNOGBEAQ, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.047199231 (76h44m17.918282266s) [noise=true , address=145.79.227.16:4910 , piece_id=YIHYKALCHWYGMAUKS3RPUQ2WOAQKCRQV6GZPFM27K37CKTTJSDPA, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.049620632 (76h44m17.920703677s) [noise=true , address=145.79.227.20:4438 , piece_id=QZQULVSGCAOTQEE5MBUONJYAJIO3GPUNKB4CEVMK3NA2J3BYZ6GA, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.051625078 (76h44m17.922708143s) [noise=true , address=145.79.227.22:4511 , piece_id=CFUDAP3RXMYJCS5GQJVYCKXTN6RLXG7NSQH6EHTZPSYCXPSPBEFQ, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.053186752 (76h44m17.924269787s) [noise=true , address=145.79.227.39:4015 , piece_id=UX4HGNQIHC3FCNGUGIAZYPNYYTLNXMYIND7H7PZ6DCD5EXS67Y7Q, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:18.054291115 (76h44m17.92537415s) [noise=true , address=77.90.31.10:4517 , piece_id=BMXTXAXNYZ5U5T54U57PMGP4KZO332K2VEP4KO5NEYC7TZ2S53SA, seg_pos={0 0}, upload=67101]: Uploading piece... Dec 8 20:13:19.111879942 (76h44m18.982962987s) [noise=true , address=77.90.31.10:4517 , piece_id=BMXTXAXNYZ5U5T54U57PMGP4KZO332K2VEP4KO5NEYC7TZ2S53SA, seg_pos={0 0}, upload=67101]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: storage node overloaded, request limit: 2000 Dec 8 20:13:19.643615670 (76h44m19.514698706s) [upload=67101]: Next returned nil segment Dec 8 20:13:19.643629617 (76h44m19.514712652s) [upload=67101]: Waiting for error group managing segments... Dec 8 20:13:20.026246645 (76h44m19.89732969s) [noise=true , address=145.79.227.32:4834 , piece_id=WDZ3N3A3CNPAECT7HW2VZ2K7JEJVOZJCZR3BKHRBVR3XHGPGTKPQ, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.027742514 (76h44m19.898825559s) [noise=true , address=145.79.227.25:4619 , piece_id=5M5PTNON4YZ7INE34X7I6BIJVQF543VQWALEZBCZEEEQ26UMGYQA, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.028492581 (76h44m19.899575616s) [noise=true , address=145.79.227.41:4085 , piece_id=FFFFOS7PH3SN3S4NWJJRYP772D725TSGCBKXIZ7FTEDBPO3TNR3Q, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.029004767 (76h44m19.900087802s) [noise=true , address=145.79.227.16:4912 , piece_id=J7BVXTHOSEPXKDDWXPV7JIUWW5NNWLETI7S4IOVPDG6ZUVPATOXQ, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.029438392 (76h44m19.900521427s) [noise=true , address=145.79.227.24:4583 , piece_id=MTNWT2AOTPYXLNVPSOKK5ZOAY7U3UH376V232TZKE56UM2CMDVAQ, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.031410738 (76h44m19.902493783s) [noise=true , address=145.79.227.42:4121 , piece_id=ICWYGS3AGWC47JS4PUC4GHT2YOV5UTD5RYL73JFTWZRNEMAGMRZA, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.034141027 (76h44m19.905224072s) [noise=true , address=145.79.227.31:4799 , piece_id=RNPR7HC6YANTTQW7STMNTHYANE7ZD5XC2N5DMLQQ5YBULNUSFK7A, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.037360508 (76h44m19.908443543s) [noise=true , address=145.79.227.22:4511 , piece_id=CFUDAP3RXMYJCS5GQJVYCKXTN6RLXG7NSQH6EHTZPSYCXPSPBEFQ, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.038059108 (76h44m19.909142153s) [noise=true , address=145.79.227.39:4015 , piece_id=UX4HGNQIHC3FCNGUGIAZYPNYYTLNXMYIND7H7PZ6DCD5EXS67Y7Q, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.053205520 (76h44m19.924288565s) [noise=true , address=145.79.227.16:4910 , piece_id=YIHYKALCHWYGMAUKS3RPUQ2WOAQKCRQV6GZPFM27K37CKTTJSDPA, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.053898059 (76h44m19.924981114s) [noise=true , address=145.79.227.20:4437 , piece_id=G76GCXUI5YH3CMQV47V6FSFDDSOBIIFMS24UJVTNXRR2MM4UZ5ZQ, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.054182971 (76h44m19.925266006s) [noise=true , address=145.79.227.27:4691 , piece_id=R5VVIJNPYZIDQW5WSJMEYS4D55AHXXMACQY5GM5IOJXVKNOGBEAQ, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.058658112 (76h44m19.929741157s) [noise=true , address=145.79.227.29:4942 , piece_id=ZYT75AZ7X56NBH3OVLWL27J5FRRN4WKV5VEIKEXN32E6P5OQNOZQ, seg_pos={0 0}, upload=67101]: Done uploading piece. err: <nil> Dec 8 20:13:20.058679072 (76h44m19.929762107s) [seg_pos={0 0}, upload=67101]: Segment reached optimal threshold of 13 pieces. Dec 8 20:13:20.058873853 (76h44m19.929956888s) [noise=true , address=145.79.227.31:4797 , piece_id=IHEVQ2JUPASTR3CU7QMNAQVSYQ65LEV5SR2KO4GCEOKAKQR5Y5NA, seg_pos={0 0}, upload=67101]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9): protocol: expected piece hash; context canceled; EOF Dec 8 20:13:20.058880526 (76h44m19.929963561s) [noise=true , address=145.79.227.31:4797 , piece_id=IHEVQ2JUPASTR3CU7QMNAQVSYQ65LEV5SR2KO4GCEOKAKQR5Y5NA, seg_pos={0 0}, upload=67101]: Long tail cancellation due to optimal threshold Dec 8 20:13:20.059077551 (76h44m19.930160586s) [noise=true , address=145.79.227.20:4438 , piece_id=QZQULVSGCAOTQEE5MBUONJYAJIO3GPUNKB4CEVMK3NA2J3BYZ6GA, seg_pos={0 0}, upload=67101]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT): protocol: expected piece hash; context canceled; EOF Dec 8 20:13:20.059093532 (76h44m19.930176567s) [noise=true , address=145.79.227.20:4438 , piece_id=QZQULVSGCAOTQEE5MBUONJYAJIO3GPUNKB4CEVMK3NA2J3BYZ6GA, seg_pos={0 0}, upload=67101]: Long tail cancellation due to optimal threshold Dec 8 20:13:20.059116485 (76h44m19.93019953s) [seg_pos={0 0}, upload=67101]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 20:13:20.059191879 (76h44m19.930274924s) [upload=67101]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 20:13:20.059196798 (76h44m19.930279833s) [upload=67101]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 20:19:41.003042505 (76h50m40.87412554s) [upload=67101]: Done uploading segments... Dec 8 20:19:41.003155129 (76h50m40.874238175s) [upload=67101]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.732835037s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 2 out of 60 is failed (threshold=3) [...............................X...............X............] duration: 5.781233442s trace: https://jaeger.dev.storj.io/trace/3996915dd8f30735 FAIL (2025-12-08 20:34:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== PGM26LWTYTWGZGQDPBL7UDX263DFBXY7WZRXGMYEFPWXQQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.569231809 (76h54m34.440315054s) [noise=true , address=89.149.201.147:4022 , piece_id=2VR3E4SXAAOR5RAVHFVUQQ7MBZQEEQYFZ2ZW5Z7YSCVOETZNFQWQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.569369882 (76h54m34.440452917s) [noise=true , address=185.167.204.206:4037 , piece_id=XYY5NAATQJPHOK6JAMHXR3JH3QTLQF7OTA32BMST5HYQDYJ76UMA, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.569672879 (76h54m34.440755924s) [noise=true , address=185.167.204.206:4056 , piece_id=IPDW6RCHJFLMB5LOZM3FG645CCW6TT7KJW4PBQXBUSAIE2ZZM74A, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.569779642 (76h54m34.440862677s) [noise=true , address=185.167.204.205:4024 , piece_id=2UHV7KDAKJJ4DDUU5MPBNLRP24OK5RQU6GGAVYUV5MEFLDDPGUZQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.570073130 (76h54m34.441156176s) [noise=true , address=81.171.14.70:4035 , piece_id=ZKLJ5VUPALKZBSPFCJQBZZUTMJ3PX2H3XMUZ5JVSW5BJBACRUYHA, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.570428618 (76h54m34.441511653s) [noise=true , address=81.171.14.70:4043 , piece_id=FVUGWZXJLQAOLUSTRKJAULSHDWKX5GGWA2BB52YP7WQEKU4NKS5Q, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.570785237 (76h54m34.441868282s) [noise=true , address=81.171.14.70:4044 , piece_id=QPYQGR62JDWWV3GM3CVDB6BLNSGNUTQUEHHQF74TYXFN25PMNPVQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.571235034 (76h54m34.442318069s) [noise=true , address=185.167.204.205:4035 , piece_id=LXUP2T5SN5SSV7IWZLGTSLCRWLFRZBODXX7W4FFVHZFO74I2CQHQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.571655133 (76h54m34.442738169s) [noise=true , address=81.171.14.70:4031 , piece_id=53QOUYYU4KS53TELJUM3FPEKOCB464OR5GKYGYB54RVRPD52RFWA, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:34.572102666 (76h54m34.443185701s) [noise=true , address=185.167.204.205:4034 , piece_id=4VH6HEDD7CTPMEOHYGKFXSYXHKOLPYB2N6BCCJT2DXVAMADI3KJQ, seg_pos={0 0}, upload=67229]: Uploading piece... Dec 8 20:23:35.395419290 (76h54m35.266502335s) [upload=67229]: Next returned nil segment Dec 8 20:23:35.395432614 (76h54m35.26651565s) [upload=67229]: Waiting for error group managing segments... Dec 8 20:23:35.780649683 (76h54m35.651732718s) [noise=true , address=185.167.204.206:4037 , piece_id=XYY5NAATQJPHOK6JAMHXR3JH3QTLQF7OTA32BMST5HYQDYJ76UMA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.781236982 (76h54m35.652320027s) [noise=true , address=185.167.204.206:4056 , piece_id=IPDW6RCHJFLMB5LOZM3FG645CCW6TT7KJW4PBQXBUSAIE2ZZM74A, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.784267011 (76h54m35.655350046s) [noise=true , address=185.167.204.205:4030 , piece_id=O3QKGK2G5I5YYPRP5K57RTG4H35H2UDFROFU2JMSHKTFNZJDVUIA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.785164540 (76h54m35.656247585s) [noise=true , address=185.167.204.205:4012 , piece_id=QCJD2IHH4DEKTFN5ZPMZXPECBQ2IX2QPSFUZCSFR5HKCDOVYGGQA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.785585301 (76h54m35.656668336s) [noise=true , address=185.167.204.205:4002 , piece_id=PDSJ5KDCFQBURILRTWU2WSEG3UH7GX4GK57W4J75QADBPKQXRSYA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.787925598 (76h54m35.659008643s) [noise=true , address=185.167.204.206:4047 , piece_id=THE6CUP46D2OSXS34UR3TTBUP6FIVMGDGT7DVJDBB57BHMWBCZ7A, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.787994289 (76h54m35.659077324s) [noise=true , address=185.167.204.205:4034 , piece_id=4VH6HEDD7CTPMEOHYGKFXSYXHKOLPYB2N6BCCJT2DXVAMADI3KJQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.788899402 (76h54m35.659982437s) [noise=true , address=185.167.204.206:4050 , piece_id=7B4GLDQZUPQPLGPZPPN5CO3VOX3LLBMLIYA6IMALYITCHE4CZBOA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.791033055 (76h54m35.66211609s) [noise=true , address=185.167.204.205:4031 , piece_id=5XPLPLO4SPMX7ZBYHLUZTH7MBGCRESEEMP2OEI7LTTT5BN67XHMQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.792163116 (76h54m35.663246161s) [noise=true , address=185.167.204.206:4055 , piece_id=JESF36BPW3YEPXVI2GP4OALECNLYSASJHQRZVLNXKV5BQRDGOWWQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.792693356 (76h54m35.663776392s) [noise=true , address=185.167.204.206:4063 , piece_id=BRUAUJTOCPWNUOB6D5G6AJCTMJBRZRJXXXBZBIUUZZXXQB5OEX2Q, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.793171606 (76h54m35.664254662s) [noise=true , address=185.167.204.206:4045 , piece_id=IXV6HTIOVQR4HCGJIWTBQFLO733BMMNNMZ24Y34H6V6KYUQYPQRA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.798151289 (76h54m35.669234344s) [noise=true , address=185.167.204.206:4053 , piece_id=VGK5AKCI23P5ZDS427NS6PNNRIJ3QUXLRZ4ECLYN4UDSL3KAQAKA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.798189562 (76h54m35.669272597s) [noise=true , address=185.167.204.206:4040 , piece_id=4KFICIELLLPCLWQ7RIB7UUSWUK7SFX2PJK7JWCRU5S4HQNHBEWRQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.798633607 (76h54m35.669716642s) [noise=true , address=185.167.204.205:4035 , piece_id=LXUP2T5SN5SSV7IWZLGTSLCRWLFRZBODXX7W4FFVHZFO74I2CQHQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:35.803059726 (76h54m35.674142761s) [noise=true , address=185.167.204.205:4024 , piece_id=2UHV7KDAKJJ4DDUU5MPBNLRP24OK5RQU6GGAVYUV5MEFLDDPGUZQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.344154907 (76h54m36.215237952s) [noise=true , address=81.171.14.70:4036 , piece_id=FB3SB3XQDNT2HSK5FDGELGYWODFLU655QW4D23GEXIWQEZYFNFRA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.359261713 (76h54m36.230344758s) [noise=true , address=81.171.14.70:4031 , piece_id=53QOUYYU4KS53TELJUM3FPEKOCB464OR5GKYGYB54RVRPD52RFWA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.393865115 (76h54m36.26494815s) [noise=true , address=81.171.14.70:4035 , piece_id=ZKLJ5VUPALKZBSPFCJQBZZUTMJ3PX2H3XMUZ5JVSW5BJBACRUYHA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.394302157 (76h54m36.265385202s) [noise=true , address=81.171.14.70:4044 , piece_id=QPYQGR62JDWWV3GM3CVDB6BLNSGNUTQUEHHQF74TYXFN25PMNPVQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.430200674 (76h54m36.301283709s) [noise=true , address=81.171.14.70:4046 , piece_id=ZHZUULNXPFDSTOXXH6T7XIEPN23OOWCNAAUWPCZYNNDTLPRWXVKQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.800799297 (76h54m36.671882332s) [noise=true , address=81.171.14.70:4040 , piece_id=W2SOTP5SPLQ6J3PM57NZOLRVZNMTNSGPWDZKZC2KV2PBPH4TUBLA, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.862864259 (76h54m36.733947295s) [noise=true , address=81.171.14.70:4038 , piece_id=CKN2RF2YSTEOK6HUXGVBVECYAP2CLGUF6KYNR5JTHRUZPTGVI73Q, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.953769972 (76h54m36.824853007s) [noise=true , address=81.171.14.70:4043 , piece_id=FVUGWZXJLQAOLUSTRKJAULSHDWKX5GGWA2BB52YP7WQEKU4NKS5Q, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:36.996603376 (76h54m36.867686421s) [noise=true , address=89.149.201.147:4006 , piece_id=UGREZBZRVKXMGBOLQEV2NC4COTXOTTRUEAGSPNWRWP22MCPHQP7Q, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:37.037518916 (76h54m36.908601951s) [noise=true , address=89.149.201.147:4014 , piece_id=AEG6KGPGM26LWTYTWGZGQDPBL7UDX263DFBXY7WZRXGMYEFPWXQQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: <nil> Dec 8 20:23:37.037537541 (76h54m36.908620586s) [seg_pos={0 0}, upload=67229]: Segment reached optimal threshold of 26 pieces. Dec 8 20:23:37.037730880 (76h54m36.908813915s) [noise=true , address=89.149.201.147:4023 , piece_id=5EPGXWBK6GNTKLQLWEIWYBC2OJHFR3M7F4X3X4WODJWNXID3PCQQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BaeSxjXneBKoGKVrRM686sq386Bhn1YzzBFG9we8fHbcc9tbv): protocol: expected piece hash; context canceled; EOF Dec 8 20:23:37.037737452 (76h54m36.908820497s) [noise=true , address=89.149.201.147:4023 , piece_id=5EPGXWBK6GNTKLQLWEIWYBC2OJHFR3M7F4X3X4WODJWNXID3PCQQ, seg_pos={0 0}, upload=67229]: Long tail cancellation due to optimal threshold Dec 8 20:23:37.037893039 (76h54m36.908976074s) [noise=true , address=89.149.201.147:4022 , piece_id=2VR3E4SXAAOR5RAVHFVUQQ7MBZQEEQYFZ2ZW5Z7YSCVOETZNFQWQ, seg_pos={0 0}, upload=67229]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122Vj8XxpS6RnhVweQV3cZWFhj8nrzy3hZ9P7JsuTzEaZcD6iFE): protocol: expected piece hash; context canceled; EOF Dec 8 20:23:37.037900964 (76h54m36.908984019s) [noise=true , address=89.149.201.147:4022 , piece_id=2VR3E4SXAAOR5RAVHFVUQQ7MBZQEEQYFZ2ZW5Z7YSCVOETZNFQWQ, seg_pos={0 0}, upload=67229]: Long tail cancellation due to optimal threshold Dec 8 20:23:37.037926843 (76h54m36.909009889s) [seg_pos={0 0}, upload=67229]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 8 20:23:37.038007697 (76h54m36.909090732s) [upload=67229]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 20:23:37.038013398 (76h54m36.909096433s) [upload=67229]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 20:26:49.742746371 (76h57m49.613829416s) [upload=67229]: Done uploading segments... Dec 8 20:26:49.742860908 (76h57m49.613943943s) [upload=67229]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.781262355s core uplink eu-select-1 eu1-placement-15

Orbiter Check History

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)