Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [...............X.X.....X....................................] duration: 5.862414483s trace: https://jaeger.dev.storj.io/trace/5280cd334ed3ed6b FAIL (2025-12-09 02:32:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.86243434s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 5.642597002s trace: https://jaeger.dev.storj.io/trace/1ddfa89c12e62214 FAIL (2025-12-09 02:31:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: manager closed: closed: read tcp 188.166.225.170:38662->35.212.10.183:7777: read: connection reset by peer ======LOGS====== Dec 9 02:24:03.838353395 (82h55m3.70943644s) [upload=72429]: Starting upload Dec 9 02:24:03.838420683 (82h55m3.709503729s) [upload=72429]: Done starting upload Dec 9 02:24:03.838549719 (82h55m3.709632755s) [upload=72429]: Uploading segments... Dec 9 02:24:03.838688734 (82h55m3.709771779s) [upload=72429]: Got next segment. Inline: false Dec 9 02:24:03.838734311 (82h55m3.709817356s) [upload=72429]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 02:24:03.838741164 (82h55m3.709824209s) [upload=72429]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 02:24:04.086906602 (82h55m3.957989647s) [seg_pos={0 0}, upload=72429]: Begin upload segment... Dec 9 02:24:04.086974701 (82h55m3.958057746s) [seg_pos={0 0}, upload=72429]: Done begin upload segment. Dec 9 02:24:04.087029476 (82h55m3.958112511s) [noise=true , address=145.79.227.24:4581 , piece_id=ETKQGTKTMEKLK42KNN72MBQF62TFTOQO357XKF4WS6NTVBSOANDA, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.087057810 (82h55m3.958140855s) [noise=true , address=145.79.227.16:4911 , piece_id=Q62EUAKMIIB6QZ3KTNQH3PJAZIX7YNRYXJPTATNCZ4W6RWELTLXQ, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.087925792 (82h55m3.959008837s) [noise=true , address=145.79.227.42:4122 , piece_id=ZWY35JIEVFUL3LADBSS4EKYMWMFBTHJWMD6YQ7M4OGKXYAU4LNWQ, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.088596299 (82h55m3.959679344s) [noise=true , address=145.79.227.23:4545 , piece_id=NPORRF6TACVJIVBM3IABBQF7NBYSDRGJTPGDREFMOWNNT3B325UQ, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.089090329 (82h55m3.960173374s) [noise=true , address=145.79.227.43:4158 , piece_id=DS3UAJZAYVR4GXPHKPMJ2RO7765FUPASW7T7GQH6ATVCM4FAPYLQ, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.089582366 (82h55m3.960665411s) [noise=true , address=145.79.227.11:4732 , piece_id=CA5XB6ODOYLW3RQU5F7BMUDDSAGEJGIKOP7IZBHPNUTUXYDJ4JWQ, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.090135129 (82h55m3.961218174s) [noise=true , address=145.79.227.34:4906 , piece_id=GTGD6FTBFJ6VMETYMRJPTO7P77B4TBQT4VSRAEMUCJ6TTW4VBHDA, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.090616965 (82h55m3.96170001s) [noise=true , address=145.79.227.24:4583 , piece_id=MFGUNU4E6DBCQ5TUNPSWBXRSEQ7QGTNWTWCUTZK5NAVLMIBB2KKQ, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.091103331 (82h55m3.962186376s) [noise=true , address=145.79.227.25:4619 , piece_id=RALQO5F6I5M4Q56PLG47D57UH4WHRQWB47N45NASEPILQGC26VQQ, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.091901141 (82h55m3.962984186s) [noise=true , address=145.79.227.38:4978 , piece_id=XBHD6MMZECN7T7QTC3HVN7OS5XY7CJJJEFERUVK2ZIXTVWSVNLXA, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.094168697 (82h55m3.965251742s) [noise=true , address=77.90.31.16:4734 , piece_id=WSIBQS2XW3PENG55ORDRTUYXPZ6HCSTAX7DP5JFDVVCLCX6JGLTA, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.094774330 (82h55m3.965857375s) [noise=true , address=145.79.227.40:4049 , piece_id=VH3PRUHK4OR4MCGVX3VHMXKWTVE3KQLKH5RJHI4MLJMITRHILR2A, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.095474052 (82h55m3.966557097s) [noise=true , address=145.79.227.31:4797 , piece_id=XDQGJD7JELZ3D73GMYRVHKWEFIZGYF6IWDBZ54NIJTGA3G7WWMFQ, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.096054146 (82h55m3.967137191s) [noise=true , address=145.79.227.21:4474 , piece_id=TTDRH27TBF34WUNSWZL3XL2RR6SJGEHP7J4YCHAZB4SH3ZWH4PVA, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.096661643 (82h55m3.967744688s) [noise=true , address=77.90.31.10:4520 , piece_id=GW2ECPZB4XCEXAWWCBVUM2VEUXXMSBPJUZSKTULZYCHK4DZL63XA, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.097203715 (82h55m3.96828676s) [noise=true , address=145.79.227.22:4511 , piece_id=QVG66WWAN3TCESLCDW2ZG7SQADAJPUIILVYGFGTCKRLGVXJWSQ7Q, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:05.228786599 (82h55m5.099869634s) [noise=true , address=77.90.31.16:4734 , piece_id=WSIBQS2XW3PENG55ORDRTUYXPZ6HCSTAX7DP5JFDVVCLCX6JGLTA, seg_pos={0 0}, upload=72429]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 9 02:24:05.771060198 (82h55m5.642143243s) [upload=72429]: Next returned nil segment Dec 9 02:24:05.771079454 (82h55m5.642162529s) [upload=72429]: Waiting for error group managing segments... Dec 9 02:24:06.084934630 (82h55m5.956017675s) [noise=true , address=145.79.227.24:4583 , piece_id=MFGUNU4E6DBCQ5TUNPSWBXRSEQ7QGTNWTWCUTZK5NAVLMIBB2KKQ, seg_pos={0 0}, upload=72429]: Done uploading piece. err: <nil> Dec 9 02:24:06.086488888 (82h55m5.957571934s) [noise=true , address=145.79.227.42:4122 , piece_id=ZWY35JIEVFUL3LADBSS4EKYMWMFBTHJWMD6YQ7M4OGKXYAU4LNWQ, seg_pos={0 0}, upload=72429]: Done uploading piece. err: <nil> Dec 9 02:24:06.087432816 (82h55m5.958515861s) [noise=true , address=145.79.227.25:4619 , piece_id=RALQO5F6I5M4Q56PLG47D57UH4WHRQWB47N45NASEPILQGC26VQQ, seg_pos={0 0}, upload=72429]: Done uploading piece. err: <nil> Dec 9 02:24:06.088224853 (82h55m5.959307898s) [noise=true , address=145.79.227.24:4581 , piece_id=ETKQGTKTMEKLK42KNN72MBQF62TFTOQO357XKF4WS6NTVBSOANDA, seg_pos={0 0}, upload=72429]: Done uploading piece. err: <nil> Dec 9 02:24:06.090286018 (82h55m5.961369053s) [noise=true , address=145.79.227.31:4797 , piece_id=XDQGJD7JELZ3D73GMYRVHKWEFIZGYF6IWDBZ54NIJTGA3G7WWMFQ, seg_pos={0 0}, upload=72429]: Done uploading piece. err: <nil> Dec 9 02:24:06.094149082 (82h55m5.965232127s) [noise=true , address=145.79.227.34:4906 , piece_id=GTGD6FTBFJ6VMETYMRJPTO7P77B4TBQT4VSRAEMUCJ6TTW4VBHDA, seg_pos={0 0}, upload=72429]: Done uploading piece. err: <nil> Dec 9 02:24:06.096430375 (82h55m5.96751342s) [noise=true , address=145.79.227.21:4474 , piece_id=TTDRH27TBF34WUNSWZL3XL2RR6SJGEHP7J4YCHAZB4SH3ZWH4PVA, seg_pos={0 0}, upload=72429]: Done uploading piece. err: <nil> Dec 9 02:24:06.096707993 (82h55m5.967791028s) [noise=true , address=145.79.227.11:4732 , piece_id=CA5XB6ODOYLW3RQU5F7BMUDDSAGEJGIKOP7IZBHPNUTUXYDJ4JWQ, seg_pos={0 0}, upload=72429]: Done uploading piece. err: <nil> Dec 9 02:24:06.103696047 (82h55m5.974779092s) [noise=true , address=145.79.227.16:4911 , piece_id=Q62EUAKMIIB6QZ3KTNQH3PJAZIX7YNRYXJPTATNCZ4W6RWELTLXQ, seg_pos={0 0}, upload=72429]: Done uploading piece. err: <nil> Dec 9 02:24:06.105006020 (82h55m5.976089065s) [noise=true , address=145.79.227.23:4545 , piece_id=NPORRF6TACVJIVBM3IABBQF7NBYSDRGJTPGDREFMOWNNT3B325UQ, seg_pos={0 0}, upload=72429]: Done uploading piece. err: <nil> Dec 9 02:24:06.105250326 (82h55m5.976333381s) [noise=true , address=145.79.227.38:4978 , piece_id=XBHD6MMZECN7T7QTC3HVN7OS5XY7CJJJEFERUVK2ZIXTVWSVNLXA, seg_pos={0 0}, upload=72429]: Done uploading piece. err: <nil> Dec 9 02:24:06.105501193 (82h55m5.976584238s) [noise=true , address=145.79.227.22:4511 , piece_id=QVG66WWAN3TCESLCDW2ZG7SQADAJPUIILVYGFGTCKRLGVXJWSQ7Q, seg_pos={0 0}, upload=72429]: Done uploading piece. err: <nil> Dec 9 02:24:06.108138354 (82h55m5.979221399s) [noise=true , address=145.79.227.40:4049 , piece_id=VH3PRUHK4OR4MCGVX3VHMXKWTVE3KQLKH5RJHI4MLJMITRHILR2A, seg_pos={0 0}, upload=72429]: Done uploading piece. err: <nil> Dec 9 02:24:06.108158543 (82h55m5.979241588s) [seg_pos={0 0}, upload=72429]: Segment reached optimal threshold of 13 pieces. Dec 9 02:24:06.108688862 (82h55m5.979771907s) [noise=true , address=77.90.31.10:4520 , piece_id=GW2ECPZB4XCEXAWWCBVUM2VEUXXMSBPJUZSKTULZYCHK4DZL63XA, seg_pos={0 0}, upload=72429]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Dec 9 02:24:06.108700394 (82h55m5.979783449s) [noise=true , address=77.90.31.10:4520 , piece_id=GW2ECPZB4XCEXAWWCBVUM2VEUXXMSBPJUZSKTULZYCHK4DZL63XA, seg_pos={0 0}, upload=72429]: Long tail cancellation due to optimal threshold Dec 9 02:24:06.108930402 (82h55m5.980013447s) [noise=true , address=145.79.227.43:4158 , piece_id=DS3UAJZAYVR4GXPHKPMJ2RO7765FUPASW7T7GQH6ATVCM4FAPYLQ, seg_pos={0 0}, upload=72429]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF Dec 9 02:24:06.108941323 (82h55m5.980024368s) [noise=true , address=145.79.227.43:4158 , piece_id=DS3UAJZAYVR4GXPHKPMJ2RO7765FUPASW7T7GQH6ATVCM4FAPYLQ, seg_pos={0 0}, upload=72429]: Long tail cancellation due to optimal threshold Dec 9 02:24:06.108964487 (82h55m5.980047532s) [seg_pos={0 0}, upload=72429]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 02:24:06.109072202 (82h55m5.980155257s) [upload=72429]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 02:24:06.109080047 (82h55m5.980163092s) [upload=72429]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 02:24:24.569429416 (82h55m24.440512451s) [upload=72429]: Done uploading segments... Dec 9 02:24:24.569477928 (82h55m24.440560963s) [upload=72429]: Upload finished. err: metaclient: manager closed: closed: read tcp 188.166.225.170:38662->35.212.10.183:7777: read: connection reset by peer Duration: 5.642627129s core uplink us-select-1
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 2 out of 60 is failed (threshold=3) [....................X......X................................] duration: 476.154043ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/4e3d2b60e3f4041b FAIL (2025-12-09 02:31:32 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 476.183128ms core satellite eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 6.179822363s trace: https://jaeger.dev.storj.io/trace/6f1ff10066b1d8ea FAIL (2025-12-09 02:31:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== rue , address=185.167.204.206:4052 , piece_id=WY7PGT7EC5IUBTSHFEAMHXONCIKPFPRF3Y4YSL2IJNY2MOO6U7HA, seg_pos={0 0}, upload=72112]: Uploading piece... Dec 9 02:01:05.223188835 (82h32m5.09427188s) [noise=true , address=89.149.201.147:4011 , piece_id=I77RBTAA7KPTBE27RPPPCNYU65WBBUORVCNYAJOV5OHOBGUFEJKA, seg_pos={0 0}, upload=72112]: Uploading piece... Dec 9 02:01:05.223808365 (82h32m5.09489141s) [noise=true , address=185.167.204.206:4050 , piece_id=YKLEM6U44VR3GM45KKKJF7JYOFA4YQE6P5UAZZEMXIA2LBZW5UKA, seg_pos={0 0}, upload=72112]: Uploading piece... Dec 9 02:01:05.224324407 (82h32m5.095407452s) [noise=true , address=185.167.204.205:4011 , piece_id=6JR2JQYKTRLCSBJDUWWSJ6CRYGWKWN3AW3KJCPXMYLYPUN6W2DRQ, seg_pos={0 0}, upload=72112]: Uploading piece... Dec 9 02:01:05.224877841 (82h32m5.095960886s) [noise=true , address=81.171.14.70:4032 , piece_id=APFBMO7KRDDWH6WT3H4ZLCXBAAMZTZQYQBN3XX6JZDGOBAMSGVCA, seg_pos={0 0}, upload=72112]: Uploading piece... Dec 9 02:01:05.225476030 (82h32m5.096559075s) [noise=true , address=89.149.201.147:4014 , piece_id=PFXVAHNZ2Q6YC2MGNSUZMJIL5Q2O2HPDSONU3FD3E472H2G7AG5A, seg_pos={0 0}, upload=72112]: Uploading piece... Dec 9 02:01:05.226119545 (82h32m5.09720258s) [noise=true , address=89.149.201.147:4015 , piece_id=XBCH6ONGHITOZ2Z5GDPPKDV45TILCDFTOZ44OKMSZANCGVBGQ7ZQ, seg_pos={0 0}, upload=72112]: Uploading piece... Dec 9 02:01:05.226816121 (82h32m5.097899166s) [noise=true , address=89.149.201.147:4012 , piece_id=7UFT32WUAW2JUETM2Y4SFGMSPTNLVVH5FKPQJSK6S3DR6E64ITKQ, seg_pos={0 0}, upload=72112]: Uploading piece... Dec 9 02:01:05.227549758 (82h32m5.098632813s) [noise=true , address=185.167.204.206:4067 , piece_id=PK6GGQNT4DQ2ZMF2IEEO5X32IMA26ULKDN7HAAG4CWOC2UAUKMMA, seg_pos={0 0}, upload=72112]: Uploading piece... Dec 9 02:01:05.228376403 (82h32m5.099459448s) [noise=true , address=81.171.14.70:4046 , piece_id=RR4RRWZ6YO3EK62O7TE2QUC3ZJ762VGUTVXVVHPGMQRPWEFTB7VA, seg_pos={0 0}, upload=72112]: Uploading piece... Dec 9 02:01:05.228990242 (82h32m5.100073297s) [noise=true , address=185.167.204.205:4013 , piece_id=7TBB5J7PYW23YFBOESEZ5Q6YOD3FKALVJMUQRMC5JSQ5JYKGDQSA, seg_pos={0 0}, upload=72112]: Uploading piece... Dec 9 02:01:06.054905806 (82h32m5.925988851s) [upload=72112]: Next returned nil segment Dec 9 02:01:06.054930403 (82h32m5.926013448s) [upload=72112]: Waiting for error group managing segments... Dec 9 02:01:06.441184862 (82h32m6.312267897s) [noise=true , address=185.167.204.205:4004 , piece_id=6CC2A4V4TJPRJBN4IMJ7VGDVFYTJIVY6J3ZIPZWYL5TIBLGT744A, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:06.444455871 (82h32m6.315538916s) [noise=true , address=185.167.204.205:4005 , piece_id=QUO2YNK2MJJ63CKWADK5K64GYKHDOSBPFHKYKQSKH2TJQUZTNEVA, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:06.447822552 (82h32m6.318905597s) [noise=true , address=185.167.204.206:4066 , piece_id=SSGUSCBUP2YNFN7OMUZQIXJJMSHV6GDHQYMN475LJFQQ7SV2FV2A, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:06.451954059 (82h32m6.323037104s) [noise=true , address=185.167.204.205:4008 , piece_id=54IT2P63UNQKCHG4HB7CX7UWJFWDLJXXPOGPNAEN36ZAC4UQ6VQQ, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:06.454212810 (82h32m6.325295856s) [noise=true , address=185.167.204.205:4011 , piece_id=6JR2JQYKTRLCSBJDUWWSJ6CRYGWKWN3AW3KJCPXMYLYPUN6W2DRQ, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:06.454331416 (82h32m6.325414461s) [noise=true , address=185.167.204.206:4052 , piece_id=WY7PGT7EC5IUBTSHFEAMHXONCIKPFPRF3Y4YSL2IJNY2MOO6U7HA, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:06.455138002 (82h32m6.326221047s) [noise=true , address=185.167.204.206:4050 , piece_id=YKLEM6U44VR3GM45KKKJF7JYOFA4YQE6P5UAZZEMXIA2LBZW5UKA, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:06.459504045 (82h32m6.33058709s) [noise=true , address=185.167.204.205:4013 , piece_id=7TBB5J7PYW23YFBOESEZ5Q6YOD3FKALVJMUQRMC5JSQ5JYKGDQSA, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:06.465474574 (82h32m6.336557629s) [noise=true , address=185.167.204.206:4067 , piece_id=PK6GGQNT4DQ2ZMF2IEEO5X32IMA26ULKDN7HAAG4CWOC2UAUKMMA, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.060075359 (82h32m6.931158404s) [noise=true , address=81.171.14.70:4034 , piece_id=DNHPDP5YCQLS5AN6OHWRWVOGOIGE53YB7CJVUHNPJK35AEM455GQ, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.111399558 (82h32m6.982482603s) [noise=true , address=81.171.14.70:4026 , piece_id=CRRJW4OCPQUSWT6BERGQJPYYQ4DHSZIBMIBCD22QWCGGHUISCFXQ, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.133611052 (82h32m7.004694097s) [noise=true , address=81.171.14.70:4032 , piece_id=APFBMO7KRDDWH6WT3H4ZLCXBAAMZTZQYQBN3XX6JZDGOBAMSGVCA, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.149778297 (82h32m7.020861342s) [noise=true , address=81.171.14.70:4030 , piece_id=VOYOWF6Y5DZOJ6DLW6QFYUI6DF5YA7FI4VI57LHMNCZ2PZLRY4BQ, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.243693687 (82h32m7.114776732s) [noise=true , address=81.171.14.70:4040 , piece_id=RYDZ6DCJ3Z5YKJPIFMLR7EHPH55S7HABKAJCKGRGHF2TCRKEYHXQ, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.624315751 (82h32m7.495398796s) [noise=true , address=81.171.14.70:4046 , piece_id=RR4RRWZ6YO3EK62O7TE2QUC3ZJ762VGUTVXVVHPGMQRPWEFTB7VA, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.682160867 (82h32m7.553243912s) [noise=true , address=89.149.201.147:4011 , piece_id=I77RBTAA7KPTBE27RPPPCNYU65WBBUORVCNYAJOV5OHOBGUFEJKA, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.704962114 (82h32m7.576045159s) [noise=true , address=89.149.201.147:4003 , piece_id=KQDYYB3S7PMHBAZTSQDATYYBAUYHMHEQDI6PW4GM4HKXBK4XHIVQ, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.723657624 (82h32m7.594740669s) [noise=true , address=89.149.201.147:4008 , piece_id=JCJNV47ZC7LJPXSERGUXP2QRLHVPOAQNOPI25RQZTA52IYKRZK4Q, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.729409366 (82h32m7.600492411s) [noise=true , address=89.149.201.147:4002 , piece_id=YFUKN6FRXESUYMS6ZV53MCQTZV5UXYHFF72WS2XJWGF3K3I72GTQ, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.761878943 (82h32m7.632961988s) [noise=true , address=89.149.201.147:4004 , piece_id=NGEHOFVBOCBBCYIVGSYLMKTFQWTDJ2EOGKYEW4X2OIWYH5TVGX2Q, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.825863449 (82h32m7.696946494s) [noise=true , address=89.149.201.147:4005 , piece_id=K6WGQSQZ32ZSAD6KN5IWB66MJ3YP76IKULPEHTFSODYFBSESSU4A, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.866343990 (82h32m7.737427035s) [noise=true , address=89.149.201.147:4015 , piece_id=XBCH6ONGHITOZ2Z5GDPPKDV45TILCDFTOZ44OKMSZANCGVBGQ7ZQ, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:08.228780506 (82h32m8.099863541s) [noise=true , address=89.149.201.147:4017 , piece_id=5W66LIGVHUXIA4AVNKUNCOAJAP5TCYAKVFKCL6NDF2OVXNI62UMQ, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:08.230729618 (82h32m8.101812653s) [noise=true , address=89.149.201.147:4019 , piece_id=JTQ44O6RFCMIPK7LAEWXO2NTXAZAJK7XR3XAJGVQXLOLM7SFSBCA, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:08.257428165 (82h32m8.12851121s) [noise=true , address=89.149.201.147:4020 , piece_id=T5RN3NMY35NXWM55YN3F7TWPPP7KXEER5R63W4KZV6Y2UNORJX4A, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:08.267970901 (82h32m8.139053946s) [noise=true , address=89.149.201.147:4006 , piece_id=M3VDXP334RID2FJK7ZYNUHJBRMS57EXIMVMLER25MVJWM4EHJ2WA, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:08.267990939 (82h32m8.139073974s) [seg_pos={0 0}, upload=72112]: Segment reached optimal threshold of 26 pieces. Dec 9 02:01:08.268172084 (82h32m8.139255129s) [noise=true , address=89.149.201.147:4012 , piece_id=7UFT32WUAW2JUETM2Y4SFGMSPTNLVVH5FKPQJSK6S3DR6E64ITKQ, seg_pos={0 0}, upload=72112]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hJ5jT6S2d3DJe9DoVDjHMkpwSTpeyTMyNLaK8Pk8SWcUVRKnv): protocol: expected piece hash; context canceled; EOF Dec 9 02:01:08.268182163 (82h32m8.139265198s) [noise=true , address=89.149.201.147:4012 , piece_id=7UFT32WUAW2JUETM2Y4SFGMSPTNLVVH5FKPQJSK6S3DR6E64ITKQ, seg_pos={0 0}, upload=72112]: Long tail cancellation due to optimal threshold Dec 9 02:01:08.268290639 (82h32m8.139373684s) [noise=true , address=89.149.201.147:4014 , piece_id=PFXVAHNZ2Q6YC2MGNSUZMJIL5Q2O2HPDSONU3FD3E472H2G7AG5A, seg_pos={0 0}, upload=72112]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wgtphHKaTejC1qrJ98sDysBwtj2Zeqw2c5UFk1HMqfY8Buvr): protocol: expected piece hash; context canceled; EOF Dec 9 02:01:08.268299597 (82h32m8.139382642s) [noise=true , address=89.149.201.147:4014 , piece_id=PFXVAHNZ2Q6YC2MGNSUZMJIL5Q2O2HPDSONU3FD3E472H2G7AG5A, seg_pos={0 0}, upload=72112]: Long tail cancellation due to optimal threshold Dec 9 02:01:08.268340294 (82h32m8.139423329s) [seg_pos={0 0}, upload=72112]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 9 02:01:08.268435756 (82h32m8.139518802s) [upload=72112]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 02:01:08.268441557 (82h32m8.139524592s) [upload=72112]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 02:12:41.424380970 (82h43m41.295464025s) [upload=72112]: Done uploading segments... Dec 9 02:12:41.424504095 (82h43m41.29558713s) [upload=72112]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.17985212s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.......X.......XX........X..................................] duration: 3.487082396s trace: https://jaeger.dev.storj.io/trace/dbaa73a4151be2e FAIL (2025-12-09 02:31:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.487109928s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [................X.X.....X...................................] duration: 5.880231462s trace: https://jaeger.dev.storj.io/trace/5280cd334ed3ed6b FAIL (2025-12-09 02:31:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.880259706s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 5.985769316s trace: https://jaeger.dev.storj.io/trace/40019ff87d088ff7 FAIL (2025-12-09 02:31:18 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.985791096s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 5.735402831s trace: https://jaeger.dev.storj.io/trace/1ddfa89c12e62214 FAIL (2025-12-09 02:31:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: manager closed: closed: read tcp 188.166.225.170:38662->35.212.10.183:7777: read: connection reset by peer ======LOGS====== Dec 9 02:24:03.838353395 (82h55m3.70943644s) [upload=72429]: Starting upload Dec 9 02:24:03.838420683 (82h55m3.709503729s) [upload=72429]: Done starting upload Dec 9 02:24:03.838549719 (82h55m3.709632755s) [upload=72429]: Uploading segments... Dec 9 02:24:03.838688734 (82h55m3.709771779s) [upload=72429]: Got next segment. Inline: false Dec 9 02:24:03.838734311 (82h55m3.709817356s) [upload=72429]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 02:24:03.838741164 (82h55m3.709824209s) [upload=72429]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 02:24:04.086906602 (82h55m3.957989647s) [seg_pos={0 0}, upload=72429]: Begin upload segment... Dec 9 02:24:04.086974701 (82h55m3.958057746s) [seg_pos={0 0}, upload=72429]: Done begin upload segment. Dec 9 02:24:04.087029476 (82h55m3.958112511s) [noise=true , address=145.79.227.24:4581 , piece_id=ETKQGTKTMEKLK42KNN72MBQF62TFTOQO357XKF4WS6NTVBSOANDA, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.087057810 (82h55m3.958140855s) [noise=true , address=145.79.227.16:4911 , piece_id=Q62EUAKMIIB6QZ3KTNQH3PJAZIX7YNRYXJPTATNCZ4W6RWELTLXQ, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.087925792 (82h55m3.959008837s) [noise=true , address=145.79.227.42:4122 , piece_id=ZWY35JIEVFUL3LADBSS4EKYMWMFBTHJWMD6YQ7M4OGKXYAU4LNWQ, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.088596299 (82h55m3.959679344s) [noise=true , address=145.79.227.23:4545 , piece_id=NPORRF6TACVJIVBM3IABBQF7NBYSDRGJTPGDREFMOWNNT3B325UQ, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.089090329 (82h55m3.960173374s) [noise=true , address=145.79.227.43:4158 , piece_id=DS3UAJZAYVR4GXPHKPMJ2RO7765FUPASW7T7GQH6ATVCM4FAPYLQ, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.089582366 (82h55m3.960665411s) [noise=true , address=145.79.227.11:4732 , piece_id=CA5XB6ODOYLW3RQU5F7BMUDDSAGEJGIKOP7IZBHPNUTUXYDJ4JWQ, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.090135129 (82h55m3.961218174s) [noise=true , address=145.79.227.34:4906 , piece_id=GTGD6FTBFJ6VMETYMRJPTO7P77B4TBQT4VSRAEMUCJ6TTW4VBHDA, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.090616965 (82h55m3.96170001s) [noise=true , address=145.79.227.24:4583 , piece_id=MFGUNU4E6DBCQ5TUNPSWBXRSEQ7QGTNWTWCUTZK5NAVLMIBB2KKQ, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.091103331 (82h55m3.962186376s) [noise=true , address=145.79.227.25:4619 , piece_id=RALQO5F6I5M4Q56PLG47D57UH4WHRQWB47N45NASEPILQGC26VQQ, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.091901141 (82h55m3.962984186s) [noise=true , address=145.79.227.38:4978 , piece_id=XBHD6MMZECN7T7QTC3HVN7OS5XY7CJJJEFERUVK2ZIXTVWSVNLXA, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.094168697 (82h55m3.965251742s) [noise=true , address=77.90.31.16:4734 , piece_id=WSIBQS2XW3PENG55ORDRTUYXPZ6HCSTAX7DP5JFDVVCLCX6JGLTA, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.094774330 (82h55m3.965857375s) [noise=true , address=145.79.227.40:4049 , piece_id=VH3PRUHK4OR4MCGVX3VHMXKWTVE3KQLKH5RJHI4MLJMITRHILR2A, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.095474052 (82h55m3.966557097s) [noise=true , address=145.79.227.31:4797 , piece_id=XDQGJD7JELZ3D73GMYRVHKWEFIZGYF6IWDBZ54NIJTGA3G7WWMFQ, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.096054146 (82h55m3.967137191s) [noise=true , address=145.79.227.21:4474 , piece_id=TTDRH27TBF34WUNSWZL3XL2RR6SJGEHP7J4YCHAZB4SH3ZWH4PVA, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.096661643 (82h55m3.967744688s) [noise=true , address=77.90.31.10:4520 , piece_id=GW2ECPZB4XCEXAWWCBVUM2VEUXXMSBPJUZSKTULZYCHK4DZL63XA, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.097203715 (82h55m3.96828676s) [noise=true , address=145.79.227.22:4511 , piece_id=QVG66WWAN3TCESLCDW2ZG7SQADAJPUIILVYGFGTCKRLGVXJWSQ7Q, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:05.228786599 (82h55m5.099869634s) [noise=true , address=77.90.31.16:4734 , piece_id=WSIBQS2XW3PENG55ORDRTUYXPZ6HCSTAX7DP5JFDVVCLCX6JGLTA, seg_pos={0 0}, upload=72429]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 9 02:24:05.771060198 (82h55m5.642143243s) [upload=72429]: Next returned nil segment Dec 9 02:24:05.771079454 (82h55m5.642162529s) [upload=72429]: Waiting for error group managing segments... Dec 9 02:24:06.084934630 (82h55m5.956017675s) [noise=true , address=145.79.227.24:4583 , piece_id=MFGUNU4E6DBCQ5TUNPSWBXRSEQ7QGTNWTWCUTZK5NAVLMIBB2KKQ, seg_pos={0 0}, upload=72429]: Done uploading piece. err: <nil> Dec 9 02:24:06.086488888 (82h55m5.957571934s) [noise=true , address=145.79.227.42:4122 , piece_id=ZWY35JIEVFUL3LADBSS4EKYMWMFBTHJWMD6YQ7M4OGKXYAU4LNWQ, seg_pos={0 0}, upload=72429]: Done uploading piece. err: <nil> Dec 9 02:24:06.087432816 (82h55m5.958515861s) [noise=true , address=145.79.227.25:4619 , piece_id=RALQO5F6I5M4Q56PLG47D57UH4WHRQWB47N45NASEPILQGC26VQQ, seg_pos={0 0}, upload=72429]: Done uploading piece. err: <nil> Dec 9 02:24:06.088224853 (82h55m5.959307898s) [noise=true , address=145.79.227.24:4581 , piece_id=ETKQGTKTMEKLK42KNN72MBQF62TFTOQO357XKF4WS6NTVBSOANDA, seg_pos={0 0}, upload=72429]: Done uploading piece. err: <nil> Dec 9 02:24:06.090286018 (82h55m5.961369053s) [noise=true , address=145.79.227.31:4797 , piece_id=XDQGJD7JELZ3D73GMYRVHKWEFIZGYF6IWDBZ54NIJTGA3G7WWMFQ, seg_pos={0 0}, upload=72429]: Done uploading piece. err: <nil> Dec 9 02:24:06.094149082 (82h55m5.965232127s) [noise=true , address=145.79.227.34:4906 , piece_id=GTGD6FTBFJ6VMETYMRJPTO7P77B4TBQT4VSRAEMUCJ6TTW4VBHDA, seg_pos={0 0}, upload=72429]: Done uploading piece. err: <nil> Dec 9 02:24:06.096430375 (82h55m5.96751342s) [noise=true , address=145.79.227.21:4474 , piece_id=TTDRH27TBF34WUNSWZL3XL2RR6SJGEHP7J4YCHAZB4SH3ZWH4PVA, seg_pos={0 0}, upload=72429]: Done uploading piece. err: <nil> Dec 9 02:24:06.096707993 (82h55m5.967791028s) [noise=true , address=145.79.227.11:4732 , piece_id=CA5XB6ODOYLW3RQU5F7BMUDDSAGEJGIKOP7IZBHPNUTUXYDJ4JWQ, seg_pos={0 0}, upload=72429]: Done uploading piece. err: <nil> Dec 9 02:24:06.103696047 (82h55m5.974779092s) [noise=true , address=145.79.227.16:4911 , piece_id=Q62EUAKMIIB6QZ3KTNQH3PJAZIX7YNRYXJPTATNCZ4W6RWELTLXQ, seg_pos={0 0}, upload=72429]: Done uploading piece. err: <nil> Dec 9 02:24:06.105006020 (82h55m5.976089065s) [noise=true , address=145.79.227.23:4545 , piece_id=NPORRF6TACVJIVBM3IABBQF7NBYSDRGJTPGDREFMOWNNT3B325UQ, seg_pos={0 0}, upload=72429]: Done uploading piece. err: <nil> Dec 9 02:24:06.105250326 (82h55m5.976333381s) [noise=true , address=145.79.227.38:4978 , piece_id=XBHD6MMZECN7T7QTC3HVN7OS5XY7CJJJEFERUVK2ZIXTVWSVNLXA, seg_pos={0 0}, upload=72429]: Done uploading piece. err: <nil> Dec 9 02:24:06.105501193 (82h55m5.976584238s) [noise=true , address=145.79.227.22:4511 , piece_id=QVG66WWAN3TCESLCDW2ZG7SQADAJPUIILVYGFGTCKRLGVXJWSQ7Q, seg_pos={0 0}, upload=72429]: Done uploading piece. err: <nil> Dec 9 02:24:06.108138354 (82h55m5.979221399s) [noise=true , address=145.79.227.40:4049 , piece_id=VH3PRUHK4OR4MCGVX3VHMXKWTVE3KQLKH5RJHI4MLJMITRHILR2A, seg_pos={0 0}, upload=72429]: Done uploading piece. err: <nil> Dec 9 02:24:06.108158543 (82h55m5.979241588s) [seg_pos={0 0}, upload=72429]: Segment reached optimal threshold of 13 pieces. Dec 9 02:24:06.108688862 (82h55m5.979771907s) [noise=true , address=77.90.31.10:4520 , piece_id=GW2ECPZB4XCEXAWWCBVUM2VEUXXMSBPJUZSKTULZYCHK4DZL63XA, seg_pos={0 0}, upload=72429]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Dec 9 02:24:06.108700394 (82h55m5.979783449s) [noise=true , address=77.90.31.10:4520 , piece_id=GW2ECPZB4XCEXAWWCBVUM2VEUXXMSBPJUZSKTULZYCHK4DZL63XA, seg_pos={0 0}, upload=72429]: Long tail cancellation due to optimal threshold Dec 9 02:24:06.108930402 (82h55m5.980013447s) [noise=true , address=145.79.227.43:4158 , piece_id=DS3UAJZAYVR4GXPHKPMJ2RO7765FUPASW7T7GQH6ATVCM4FAPYLQ, seg_pos={0 0}, upload=72429]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF Dec 9 02:24:06.108941323 (82h55m5.980024368s) [noise=true , address=145.79.227.43:4158 , piece_id=DS3UAJZAYVR4GXPHKPMJ2RO7765FUPASW7T7GQH6ATVCM4FAPYLQ, seg_pos={0 0}, upload=72429]: Long tail cancellation due to optimal threshold Dec 9 02:24:06.108964487 (82h55m5.980047532s) [seg_pos={0 0}, upload=72429]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 02:24:06.109072202 (82h55m5.980155257s) [upload=72429]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 02:24:06.109080047 (82h55m5.980163092s) [upload=72429]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 02:24:24.569429416 (82h55m24.440512451s) [upload=72429]: Done uploading segments... Dec 9 02:24:24.569477928 (82h55m24.440560963s) [upload=72429]: Upload finished. err: metaclient: manager closed: closed: read tcp 188.166.225.170:38662->35.212.10.183:7777: read: connection reset by peer Duration: 5.735433107s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 5.870234137s trace: https://jaeger.dev.storj.io/trace/6f1ff10066b1d8ea FAIL (2025-12-09 02:31:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== rue , address=185.167.204.206:4052 , piece_id=WY7PGT7EC5IUBTSHFEAMHXONCIKPFPRF3Y4YSL2IJNY2MOO6U7HA, seg_pos={0 0}, upload=72112]: Uploading piece... Dec 9 02:01:05.223188835 (82h32m5.09427188s) [noise=true , address=89.149.201.147:4011 , piece_id=I77RBTAA7KPTBE27RPPPCNYU65WBBUORVCNYAJOV5OHOBGUFEJKA, seg_pos={0 0}, upload=72112]: Uploading piece... Dec 9 02:01:05.223808365 (82h32m5.09489141s) [noise=true , address=185.167.204.206:4050 , piece_id=YKLEM6U44VR3GM45KKKJF7JYOFA4YQE6P5UAZZEMXIA2LBZW5UKA, seg_pos={0 0}, upload=72112]: Uploading piece... Dec 9 02:01:05.224324407 (82h32m5.095407452s) [noise=true , address=185.167.204.205:4011 , piece_id=6JR2JQYKTRLCSBJDUWWSJ6CRYGWKWN3AW3KJCPXMYLYPUN6W2DRQ, seg_pos={0 0}, upload=72112]: Uploading piece... Dec 9 02:01:05.224877841 (82h32m5.095960886s) [noise=true , address=81.171.14.70:4032 , piece_id=APFBMO7KRDDWH6WT3H4ZLCXBAAMZTZQYQBN3XX6JZDGOBAMSGVCA, seg_pos={0 0}, upload=72112]: Uploading piece... Dec 9 02:01:05.225476030 (82h32m5.096559075s) [noise=true , address=89.149.201.147:4014 , piece_id=PFXVAHNZ2Q6YC2MGNSUZMJIL5Q2O2HPDSONU3FD3E472H2G7AG5A, seg_pos={0 0}, upload=72112]: Uploading piece... Dec 9 02:01:05.226119545 (82h32m5.09720258s) [noise=true , address=89.149.201.147:4015 , piece_id=XBCH6ONGHITOZ2Z5GDPPKDV45TILCDFTOZ44OKMSZANCGVBGQ7ZQ, seg_pos={0 0}, upload=72112]: Uploading piece... Dec 9 02:01:05.226816121 (82h32m5.097899166s) [noise=true , address=89.149.201.147:4012 , piece_id=7UFT32WUAW2JUETM2Y4SFGMSPTNLVVH5FKPQJSK6S3DR6E64ITKQ, seg_pos={0 0}, upload=72112]: Uploading piece... Dec 9 02:01:05.227549758 (82h32m5.098632813s) [noise=true , address=185.167.204.206:4067 , piece_id=PK6GGQNT4DQ2ZMF2IEEO5X32IMA26ULKDN7HAAG4CWOC2UAUKMMA, seg_pos={0 0}, upload=72112]: Uploading piece... Dec 9 02:01:05.228376403 (82h32m5.099459448s) [noise=true , address=81.171.14.70:4046 , piece_id=RR4RRWZ6YO3EK62O7TE2QUC3ZJ762VGUTVXVVHPGMQRPWEFTB7VA, seg_pos={0 0}, upload=72112]: Uploading piece... Dec 9 02:01:05.228990242 (82h32m5.100073297s) [noise=true , address=185.167.204.205:4013 , piece_id=7TBB5J7PYW23YFBOESEZ5Q6YOD3FKALVJMUQRMC5JSQ5JYKGDQSA, seg_pos={0 0}, upload=72112]: Uploading piece... Dec 9 02:01:06.054905806 (82h32m5.925988851s) [upload=72112]: Next returned nil segment Dec 9 02:01:06.054930403 (82h32m5.926013448s) [upload=72112]: Waiting for error group managing segments... Dec 9 02:01:06.441184862 (82h32m6.312267897s) [noise=true , address=185.167.204.205:4004 , piece_id=6CC2A4V4TJPRJBN4IMJ7VGDVFYTJIVY6J3ZIPZWYL5TIBLGT744A, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:06.444455871 (82h32m6.315538916s) [noise=true , address=185.167.204.205:4005 , piece_id=QUO2YNK2MJJ63CKWADK5K64GYKHDOSBPFHKYKQSKH2TJQUZTNEVA, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:06.447822552 (82h32m6.318905597s) [noise=true , address=185.167.204.206:4066 , piece_id=SSGUSCBUP2YNFN7OMUZQIXJJMSHV6GDHQYMN475LJFQQ7SV2FV2A, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:06.451954059 (82h32m6.323037104s) [noise=true , address=185.167.204.205:4008 , piece_id=54IT2P63UNQKCHG4HB7CX7UWJFWDLJXXPOGPNAEN36ZAC4UQ6VQQ, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:06.454212810 (82h32m6.325295856s) [noise=true , address=185.167.204.205:4011 , piece_id=6JR2JQYKTRLCSBJDUWWSJ6CRYGWKWN3AW3KJCPXMYLYPUN6W2DRQ, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:06.454331416 (82h32m6.325414461s) [noise=true , address=185.167.204.206:4052 , piece_id=WY7PGT7EC5IUBTSHFEAMHXONCIKPFPRF3Y4YSL2IJNY2MOO6U7HA, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:06.455138002 (82h32m6.326221047s) [noise=true , address=185.167.204.206:4050 , piece_id=YKLEM6U44VR3GM45KKKJF7JYOFA4YQE6P5UAZZEMXIA2LBZW5UKA, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:06.459504045 (82h32m6.33058709s) [noise=true , address=185.167.204.205:4013 , piece_id=7TBB5J7PYW23YFBOESEZ5Q6YOD3FKALVJMUQRMC5JSQ5JYKGDQSA, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:06.465474574 (82h32m6.336557629s) [noise=true , address=185.167.204.206:4067 , piece_id=PK6GGQNT4DQ2ZMF2IEEO5X32IMA26ULKDN7HAAG4CWOC2UAUKMMA, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.060075359 (82h32m6.931158404s) [noise=true , address=81.171.14.70:4034 , piece_id=DNHPDP5YCQLS5AN6OHWRWVOGOIGE53YB7CJVUHNPJK35AEM455GQ, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.111399558 (82h32m6.982482603s) [noise=true , address=81.171.14.70:4026 , piece_id=CRRJW4OCPQUSWT6BERGQJPYYQ4DHSZIBMIBCD22QWCGGHUISCFXQ, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.133611052 (82h32m7.004694097s) [noise=true , address=81.171.14.70:4032 , piece_id=APFBMO7KRDDWH6WT3H4ZLCXBAAMZTZQYQBN3XX6JZDGOBAMSGVCA, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.149778297 (82h32m7.020861342s) [noise=true , address=81.171.14.70:4030 , piece_id=VOYOWF6Y5DZOJ6DLW6QFYUI6DF5YA7FI4VI57LHMNCZ2PZLRY4BQ, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.243693687 (82h32m7.114776732s) [noise=true , address=81.171.14.70:4040 , piece_id=RYDZ6DCJ3Z5YKJPIFMLR7EHPH55S7HABKAJCKGRGHF2TCRKEYHXQ, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.624315751 (82h32m7.495398796s) [noise=true , address=81.171.14.70:4046 , piece_id=RR4RRWZ6YO3EK62O7TE2QUC3ZJ762VGUTVXVVHPGMQRPWEFTB7VA, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.682160867 (82h32m7.553243912s) [noise=true , address=89.149.201.147:4011 , piece_id=I77RBTAA7KPTBE27RPPPCNYU65WBBUORVCNYAJOV5OHOBGUFEJKA, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.704962114 (82h32m7.576045159s) [noise=true , address=89.149.201.147:4003 , piece_id=KQDYYB3S7PMHBAZTSQDATYYBAUYHMHEQDI6PW4GM4HKXBK4XHIVQ, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.723657624 (82h32m7.594740669s) [noise=true , address=89.149.201.147:4008 , piece_id=JCJNV47ZC7LJPXSERGUXP2QRLHVPOAQNOPI25RQZTA52IYKRZK4Q, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.729409366 (82h32m7.600492411s) [noise=true , address=89.149.201.147:4002 , piece_id=YFUKN6FRXESUYMS6ZV53MCQTZV5UXYHFF72WS2XJWGF3K3I72GTQ, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.761878943 (82h32m7.632961988s) [noise=true , address=89.149.201.147:4004 , piece_id=NGEHOFVBOCBBCYIVGSYLMKTFQWTDJ2EOGKYEW4X2OIWYH5TVGX2Q, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.825863449 (82h32m7.696946494s) [noise=true , address=89.149.201.147:4005 , piece_id=K6WGQSQZ32ZSAD6KN5IWB66MJ3YP76IKULPEHTFSODYFBSESSU4A, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.866343990 (82h32m7.737427035s) [noise=true , address=89.149.201.147:4015 , piece_id=XBCH6ONGHITOZ2Z5GDPPKDV45TILCDFTOZ44OKMSZANCGVBGQ7ZQ, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:08.228780506 (82h32m8.099863541s) [noise=true , address=89.149.201.147:4017 , piece_id=5W66LIGVHUXIA4AVNKUNCOAJAP5TCYAKVFKCL6NDF2OVXNI62UMQ, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:08.230729618 (82h32m8.101812653s) [noise=true , address=89.149.201.147:4019 , piece_id=JTQ44O6RFCMIPK7LAEWXO2NTXAZAJK7XR3XAJGVQXLOLM7SFSBCA, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:08.257428165 (82h32m8.12851121s) [noise=true , address=89.149.201.147:4020 , piece_id=T5RN3NMY35NXWM55YN3F7TWPPP7KXEER5R63W4KZV6Y2UNORJX4A, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:08.267970901 (82h32m8.139053946s) [noise=true , address=89.149.201.147:4006 , piece_id=M3VDXP334RID2FJK7ZYNUHJBRMS57EXIMVMLER25MVJWM4EHJ2WA, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:08.267990939 (82h32m8.139073974s) [seg_pos={0 0}, upload=72112]: Segment reached optimal threshold of 26 pieces. Dec 9 02:01:08.268172084 (82h32m8.139255129s) [noise=true , address=89.149.201.147:4012 , piece_id=7UFT32WUAW2JUETM2Y4SFGMSPTNLVVH5FKPQJSK6S3DR6E64ITKQ, seg_pos={0 0}, upload=72112]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hJ5jT6S2d3DJe9DoVDjHMkpwSTpeyTMyNLaK8Pk8SWcUVRKnv): protocol: expected piece hash; context canceled; EOF Dec 9 02:01:08.268182163 (82h32m8.139265198s) [noise=true , address=89.149.201.147:4012 , piece_id=7UFT32WUAW2JUETM2Y4SFGMSPTNLVVH5FKPQJSK6S3DR6E64ITKQ, seg_pos={0 0}, upload=72112]: Long tail cancellation due to optimal threshold Dec 9 02:01:08.268290639 (82h32m8.139373684s) [noise=true , address=89.149.201.147:4014 , piece_id=PFXVAHNZ2Q6YC2MGNSUZMJIL5Q2O2HPDSONU3FD3E472H2G7AG5A, seg_pos={0 0}, upload=72112]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wgtphHKaTejC1qrJ98sDysBwtj2Zeqw2c5UFk1HMqfY8Buvr): protocol: expected piece hash; context canceled; EOF Dec 9 02:01:08.268299597 (82h32m8.139382642s) [noise=true , address=89.149.201.147:4014 , piece_id=PFXVAHNZ2Q6YC2MGNSUZMJIL5Q2O2HPDSONU3FD3E472H2G7AG5A, seg_pos={0 0}, upload=72112]: Long tail cancellation due to optimal threshold Dec 9 02:01:08.268340294 (82h32m8.139423329s) [seg_pos={0 0}, upload=72112]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 9 02:01:08.268435756 (82h32m8.139518802s) [upload=72112]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 02:01:08.268441557 (82h32m8.139524592s) [upload=72112]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 02:12:41.424380970 (82h43m41.295464025s) [upload=72112]: Done uploading segments... Dec 9 02:12:41.424504095 (82h43m41.29558713s) [upload=72112]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.870262983s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 6.385919294s trace: https://jaeger.dev.storj.io/trace/40019ff87d088ff7 FAIL (2025-12-09 02:30:45 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.386046826s core uplink us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 6.651091509s trace: https://jaeger.dev.storj.io/trace/6f1ff10066b1d8ea FAIL (2025-12-09 02:30:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== rue , address=185.167.204.206:4052 , piece_id=WY7PGT7EC5IUBTSHFEAMHXONCIKPFPRF3Y4YSL2IJNY2MOO6U7HA, seg_pos={0 0}, upload=72112]: Uploading piece... Dec 9 02:01:05.223188835 (82h32m5.09427188s) [noise=true , address=89.149.201.147:4011 , piece_id=I77RBTAA7KPTBE27RPPPCNYU65WBBUORVCNYAJOV5OHOBGUFEJKA, seg_pos={0 0}, upload=72112]: Uploading piece... Dec 9 02:01:05.223808365 (82h32m5.09489141s) [noise=true , address=185.167.204.206:4050 , piece_id=YKLEM6U44VR3GM45KKKJF7JYOFA4YQE6P5UAZZEMXIA2LBZW5UKA, seg_pos={0 0}, upload=72112]: Uploading piece... Dec 9 02:01:05.224324407 (82h32m5.095407452s) [noise=true , address=185.167.204.205:4011 , piece_id=6JR2JQYKTRLCSBJDUWWSJ6CRYGWKWN3AW3KJCPXMYLYPUN6W2DRQ, seg_pos={0 0}, upload=72112]: Uploading piece... Dec 9 02:01:05.224877841 (82h32m5.095960886s) [noise=true , address=81.171.14.70:4032 , piece_id=APFBMO7KRDDWH6WT3H4ZLCXBAAMZTZQYQBN3XX6JZDGOBAMSGVCA, seg_pos={0 0}, upload=72112]: Uploading piece... Dec 9 02:01:05.225476030 (82h32m5.096559075s) [noise=true , address=89.149.201.147:4014 , piece_id=PFXVAHNZ2Q6YC2MGNSUZMJIL5Q2O2HPDSONU3FD3E472H2G7AG5A, seg_pos={0 0}, upload=72112]: Uploading piece... Dec 9 02:01:05.226119545 (82h32m5.09720258s) [noise=true , address=89.149.201.147:4015 , piece_id=XBCH6ONGHITOZ2Z5GDPPKDV45TILCDFTOZ44OKMSZANCGVBGQ7ZQ, seg_pos={0 0}, upload=72112]: Uploading piece... Dec 9 02:01:05.226816121 (82h32m5.097899166s) [noise=true , address=89.149.201.147:4012 , piece_id=7UFT32WUAW2JUETM2Y4SFGMSPTNLVVH5FKPQJSK6S3DR6E64ITKQ, seg_pos={0 0}, upload=72112]: Uploading piece... Dec 9 02:01:05.227549758 (82h32m5.098632813s) [noise=true , address=185.167.204.206:4067 , piece_id=PK6GGQNT4DQ2ZMF2IEEO5X32IMA26ULKDN7HAAG4CWOC2UAUKMMA, seg_pos={0 0}, upload=72112]: Uploading piece... Dec 9 02:01:05.228376403 (82h32m5.099459448s) [noise=true , address=81.171.14.70:4046 , piece_id=RR4RRWZ6YO3EK62O7TE2QUC3ZJ762VGUTVXVVHPGMQRPWEFTB7VA, seg_pos={0 0}, upload=72112]: Uploading piece... Dec 9 02:01:05.228990242 (82h32m5.100073297s) [noise=true , address=185.167.204.205:4013 , piece_id=7TBB5J7PYW23YFBOESEZ5Q6YOD3FKALVJMUQRMC5JSQ5JYKGDQSA, seg_pos={0 0}, upload=72112]: Uploading piece... Dec 9 02:01:06.054905806 (82h32m5.925988851s) [upload=72112]: Next returned nil segment Dec 9 02:01:06.054930403 (82h32m5.926013448s) [upload=72112]: Waiting for error group managing segments... Dec 9 02:01:06.441184862 (82h32m6.312267897s) [noise=true , address=185.167.204.205:4004 , piece_id=6CC2A4V4TJPRJBN4IMJ7VGDVFYTJIVY6J3ZIPZWYL5TIBLGT744A, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:06.444455871 (82h32m6.315538916s) [noise=true , address=185.167.204.205:4005 , piece_id=QUO2YNK2MJJ63CKWADK5K64GYKHDOSBPFHKYKQSKH2TJQUZTNEVA, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:06.447822552 (82h32m6.318905597s) [noise=true , address=185.167.204.206:4066 , piece_id=SSGUSCBUP2YNFN7OMUZQIXJJMSHV6GDHQYMN475LJFQQ7SV2FV2A, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:06.451954059 (82h32m6.323037104s) [noise=true , address=185.167.204.205:4008 , piece_id=54IT2P63UNQKCHG4HB7CX7UWJFWDLJXXPOGPNAEN36ZAC4UQ6VQQ, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:06.454212810 (82h32m6.325295856s) [noise=true , address=185.167.204.205:4011 , piece_id=6JR2JQYKTRLCSBJDUWWSJ6CRYGWKWN3AW3KJCPXMYLYPUN6W2DRQ, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:06.454331416 (82h32m6.325414461s) [noise=true , address=185.167.204.206:4052 , piece_id=WY7PGT7EC5IUBTSHFEAMHXONCIKPFPRF3Y4YSL2IJNY2MOO6U7HA, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:06.455138002 (82h32m6.326221047s) [noise=true , address=185.167.204.206:4050 , piece_id=YKLEM6U44VR3GM45KKKJF7JYOFA4YQE6P5UAZZEMXIA2LBZW5UKA, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:06.459504045 (82h32m6.33058709s) [noise=true , address=185.167.204.205:4013 , piece_id=7TBB5J7PYW23YFBOESEZ5Q6YOD3FKALVJMUQRMC5JSQ5JYKGDQSA, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:06.465474574 (82h32m6.336557629s) [noise=true , address=185.167.204.206:4067 , piece_id=PK6GGQNT4DQ2ZMF2IEEO5X32IMA26ULKDN7HAAG4CWOC2UAUKMMA, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.060075359 (82h32m6.931158404s) [noise=true , address=81.171.14.70:4034 , piece_id=DNHPDP5YCQLS5AN6OHWRWVOGOIGE53YB7CJVUHNPJK35AEM455GQ, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.111399558 (82h32m6.982482603s) [noise=true , address=81.171.14.70:4026 , piece_id=CRRJW4OCPQUSWT6BERGQJPYYQ4DHSZIBMIBCD22QWCGGHUISCFXQ, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.133611052 (82h32m7.004694097s) [noise=true , address=81.171.14.70:4032 , piece_id=APFBMO7KRDDWH6WT3H4ZLCXBAAMZTZQYQBN3XX6JZDGOBAMSGVCA, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.149778297 (82h32m7.020861342s) [noise=true , address=81.171.14.70:4030 , piece_id=VOYOWF6Y5DZOJ6DLW6QFYUI6DF5YA7FI4VI57LHMNCZ2PZLRY4BQ, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.243693687 (82h32m7.114776732s) [noise=true , address=81.171.14.70:4040 , piece_id=RYDZ6DCJ3Z5YKJPIFMLR7EHPH55S7HABKAJCKGRGHF2TCRKEYHXQ, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.624315751 (82h32m7.495398796s) [noise=true , address=81.171.14.70:4046 , piece_id=RR4RRWZ6YO3EK62O7TE2QUC3ZJ762VGUTVXVVHPGMQRPWEFTB7VA, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.682160867 (82h32m7.553243912s) [noise=true , address=89.149.201.147:4011 , piece_id=I77RBTAA7KPTBE27RPPPCNYU65WBBUORVCNYAJOV5OHOBGUFEJKA, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.704962114 (82h32m7.576045159s) [noise=true , address=89.149.201.147:4003 , piece_id=KQDYYB3S7PMHBAZTSQDATYYBAUYHMHEQDI6PW4GM4HKXBK4XHIVQ, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.723657624 (82h32m7.594740669s) [noise=true , address=89.149.201.147:4008 , piece_id=JCJNV47ZC7LJPXSERGUXP2QRLHVPOAQNOPI25RQZTA52IYKRZK4Q, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.729409366 (82h32m7.600492411s) [noise=true , address=89.149.201.147:4002 , piece_id=YFUKN6FRXESUYMS6ZV53MCQTZV5UXYHFF72WS2XJWGF3K3I72GTQ, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.761878943 (82h32m7.632961988s) [noise=true , address=89.149.201.147:4004 , piece_id=NGEHOFVBOCBBCYIVGSYLMKTFQWTDJ2EOGKYEW4X2OIWYH5TVGX2Q, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.825863449 (82h32m7.696946494s) [noise=true , address=89.149.201.147:4005 , piece_id=K6WGQSQZ32ZSAD6KN5IWB66MJ3YP76IKULPEHTFSODYFBSESSU4A, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.866343990 (82h32m7.737427035s) [noise=true , address=89.149.201.147:4015 , piece_id=XBCH6ONGHITOZ2Z5GDPPKDV45TILCDFTOZ44OKMSZANCGVBGQ7ZQ, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:08.228780506 (82h32m8.099863541s) [noise=true , address=89.149.201.147:4017 , piece_id=5W66LIGVHUXIA4AVNKUNCOAJAP5TCYAKVFKCL6NDF2OVXNI62UMQ, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:08.230729618 (82h32m8.101812653s) [noise=true , address=89.149.201.147:4019 , piece_id=JTQ44O6RFCMIPK7LAEWXO2NTXAZAJK7XR3XAJGVQXLOLM7SFSBCA, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:08.257428165 (82h32m8.12851121s) [noise=true , address=89.149.201.147:4020 , piece_id=T5RN3NMY35NXWM55YN3F7TWPPP7KXEER5R63W4KZV6Y2UNORJX4A, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:08.267970901 (82h32m8.139053946s) [noise=true , address=89.149.201.147:4006 , piece_id=M3VDXP334RID2FJK7ZYNUHJBRMS57EXIMVMLER25MVJWM4EHJ2WA, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:08.267990939 (82h32m8.139073974s) [seg_pos={0 0}, upload=72112]: Segment reached optimal threshold of 26 pieces. Dec 9 02:01:08.268172084 (82h32m8.139255129s) [noise=true , address=89.149.201.147:4012 , piece_id=7UFT32WUAW2JUETM2Y4SFGMSPTNLVVH5FKPQJSK6S3DR6E64ITKQ, seg_pos={0 0}, upload=72112]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hJ5jT6S2d3DJe9DoVDjHMkpwSTpeyTMyNLaK8Pk8SWcUVRKnv): protocol: expected piece hash; context canceled; EOF Dec 9 02:01:08.268182163 (82h32m8.139265198s) [noise=true , address=89.149.201.147:4012 , piece_id=7UFT32WUAW2JUETM2Y4SFGMSPTNLVVH5FKPQJSK6S3DR6E64ITKQ, seg_pos={0 0}, upload=72112]: Long tail cancellation due to optimal threshold Dec 9 02:01:08.268290639 (82h32m8.139373684s) [noise=true , address=89.149.201.147:4014 , piece_id=PFXVAHNZ2Q6YC2MGNSUZMJIL5Q2O2HPDSONU3FD3E472H2G7AG5A, seg_pos={0 0}, upload=72112]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wgtphHKaTejC1qrJ98sDysBwtj2Zeqw2c5UFk1HMqfY8Buvr): protocol: expected piece hash; context canceled; EOF Dec 9 02:01:08.268299597 (82h32m8.139382642s) [noise=true , address=89.149.201.147:4014 , piece_id=PFXVAHNZ2Q6YC2MGNSUZMJIL5Q2O2HPDSONU3FD3E472H2G7AG5A, seg_pos={0 0}, upload=72112]: Long tail cancellation due to optimal threshold Dec 9 02:01:08.268340294 (82h32m8.139423329s) [seg_pos={0 0}, upload=72112]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 9 02:01:08.268435756 (82h32m8.139518802s) [upload=72112]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 02:01:08.268441557 (82h32m8.139524592s) [upload=72112]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 02:12:41.424380970 (82h43m41.295464025s) [upload=72112]: Done uploading segments... Dec 9 02:12:41.424504095 (82h43m41.29558713s) [upload=72112]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.651122549s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.................X.X.....X..................................] duration: 6.894357201s trace: https://jaeger.dev.storj.io/trace/5280cd334ed3ed6b FAIL (2025-12-09 02:30:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.894385644s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 5.82413269s trace: https://jaeger.dev.storj.io/trace/1ddfa89c12e62214 FAIL (2025-12-09 02:30:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: manager closed: closed: read tcp 188.166.225.170:38662->35.212.10.183:7777: read: connection reset by peer ======LOGS====== Dec 9 02:24:03.838353395 (82h55m3.70943644s) [upload=72429]: Starting upload Dec 9 02:24:03.838420683 (82h55m3.709503729s) [upload=72429]: Done starting upload Dec 9 02:24:03.838549719 (82h55m3.709632755s) [upload=72429]: Uploading segments... Dec 9 02:24:03.838688734 (82h55m3.709771779s) [upload=72429]: Got next segment. Inline: false Dec 9 02:24:03.838734311 (82h55m3.709817356s) [upload=72429]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 02:24:03.838741164 (82h55m3.709824209s) [upload=72429]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 02:24:04.086906602 (82h55m3.957989647s) [seg_pos={0 0}, upload=72429]: Begin upload segment... Dec 9 02:24:04.086974701 (82h55m3.958057746s) [seg_pos={0 0}, upload=72429]: Done begin upload segment. Dec 9 02:24:04.087029476 (82h55m3.958112511s) [noise=true , address=145.79.227.24:4581 , piece_id=ETKQGTKTMEKLK42KNN72MBQF62TFTOQO357XKF4WS6NTVBSOANDA, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.087057810 (82h55m3.958140855s) [noise=true , address=145.79.227.16:4911 , piece_id=Q62EUAKMIIB6QZ3KTNQH3PJAZIX7YNRYXJPTATNCZ4W6RWELTLXQ, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.087925792 (82h55m3.959008837s) [noise=true , address=145.79.227.42:4122 , piece_id=ZWY35JIEVFUL3LADBSS4EKYMWMFBTHJWMD6YQ7M4OGKXYAU4LNWQ, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.088596299 (82h55m3.959679344s) [noise=true , address=145.79.227.23:4545 , piece_id=NPORRF6TACVJIVBM3IABBQF7NBYSDRGJTPGDREFMOWNNT3B325UQ, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.089090329 (82h55m3.960173374s) [noise=true , address=145.79.227.43:4158 , piece_id=DS3UAJZAYVR4GXPHKPMJ2RO7765FUPASW7T7GQH6ATVCM4FAPYLQ, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.089582366 (82h55m3.960665411s) [noise=true , address=145.79.227.11:4732 , piece_id=CA5XB6ODOYLW3RQU5F7BMUDDSAGEJGIKOP7IZBHPNUTUXYDJ4JWQ, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.090135129 (82h55m3.961218174s) [noise=true , address=145.79.227.34:4906 , piece_id=GTGD6FTBFJ6VMETYMRJPTO7P77B4TBQT4VSRAEMUCJ6TTW4VBHDA, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.090616965 (82h55m3.96170001s) [noise=true , address=145.79.227.24:4583 , piece_id=MFGUNU4E6DBCQ5TUNPSWBXRSEQ7QGTNWTWCUTZK5NAVLMIBB2KKQ, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.091103331 (82h55m3.962186376s) [noise=true , address=145.79.227.25:4619 , piece_id=RALQO5F6I5M4Q56PLG47D57UH4WHRQWB47N45NASEPILQGC26VQQ, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.091901141 (82h55m3.962984186s) [noise=true , address=145.79.227.38:4978 , piece_id=XBHD6MMZECN7T7QTC3HVN7OS5XY7CJJJEFERUVK2ZIXTVWSVNLXA, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.094168697 (82h55m3.965251742s) [noise=true , address=77.90.31.16:4734 , piece_id=WSIBQS2XW3PENG55ORDRTUYXPZ6HCSTAX7DP5JFDVVCLCX6JGLTA, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.094774330 (82h55m3.965857375s) [noise=true , address=145.79.227.40:4049 , piece_id=VH3PRUHK4OR4MCGVX3VHMXKWTVE3KQLKH5RJHI4MLJMITRHILR2A, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.095474052 (82h55m3.966557097s) [noise=true , address=145.79.227.31:4797 , piece_id=XDQGJD7JELZ3D73GMYRVHKWEFIZGYF6IWDBZ54NIJTGA3G7WWMFQ, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.096054146 (82h55m3.967137191s) [noise=true , address=145.79.227.21:4474 , piece_id=TTDRH27TBF34WUNSWZL3XL2RR6SJGEHP7J4YCHAZB4SH3ZWH4PVA, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.096661643 (82h55m3.967744688s) [noise=true , address=77.90.31.10:4520 , piece_id=GW2ECPZB4XCEXAWWCBVUM2VEUXXMSBPJUZSKTULZYCHK4DZL63XA, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.097203715 (82h55m3.96828676s) [noise=true , address=145.79.227.22:4511 , piece_id=QVG66WWAN3TCESLCDW2ZG7SQADAJPUIILVYGFGTCKRLGVXJWSQ7Q, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:05.228786599 (82h55m5.099869634s) [noise=true , address=77.90.31.16:4734 , piece_id=WSIBQS2XW3PENG55ORDRTUYXPZ6HCSTAX7DP5JFDVVCLCX6JGLTA, seg_pos={0 0}, upload=72429]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 9 02:24:05.771060198 (82h55m5.642143243s) [upload=72429]: Next returned nil segment Dec 9 02:24:05.771079454 (82h55m5.642162529s) [upload=72429]: Waiting for error group managing segments... Dec 9 02:24:06.084934630 (82h55m5.956017675s) [noise=true , address=145.79.227.24:4583 , piece_id=MFGUNU4E6DBCQ5TUNPSWBXRSEQ7QGTNWTWCUTZK5NAVLMIBB2KKQ, seg_pos={0 0}, upload=72429]: Done uploading piece. err: <nil> Dec 9 02:24:06.086488888 (82h55m5.957571934s) [noise=true , address=145.79.227.42:4122 , piece_id=ZWY35JIEVFUL3LADBSS4EKYMWMFBTHJWMD6YQ7M4OGKXYAU4LNWQ, seg_pos={0 0}, upload=72429]: Done uploading piece. err: <nil> Dec 9 02:24:06.087432816 (82h55m5.958515861s) [noise=true , address=145.79.227.25:4619 , piece_id=RALQO5F6I5M4Q56PLG47D57UH4WHRQWB47N45NASEPILQGC26VQQ, seg_pos={0 0}, upload=72429]: Done uploading piece. err: <nil> Dec 9 02:24:06.088224853 (82h55m5.959307898s) [noise=true , address=145.79.227.24:4581 , piece_id=ETKQGTKTMEKLK42KNN72MBQF62TFTOQO357XKF4WS6NTVBSOANDA, seg_pos={0 0}, upload=72429]: Done uploading piece. err: <nil> Dec 9 02:24:06.090286018 (82h55m5.961369053s) [noise=true , address=145.79.227.31:4797 , piece_id=XDQGJD7JELZ3D73GMYRVHKWEFIZGYF6IWDBZ54NIJTGA3G7WWMFQ, seg_pos={0 0}, upload=72429]: Done uploading piece. err: <nil> Dec 9 02:24:06.094149082 (82h55m5.965232127s) [noise=true , address=145.79.227.34:4906 , piece_id=GTGD6FTBFJ6VMETYMRJPTO7P77B4TBQT4VSRAEMUCJ6TTW4VBHDA, seg_pos={0 0}, upload=72429]: Done uploading piece. err: <nil> Dec 9 02:24:06.096430375 (82h55m5.96751342s) [noise=true , address=145.79.227.21:4474 , piece_id=TTDRH27TBF34WUNSWZL3XL2RR6SJGEHP7J4YCHAZB4SH3ZWH4PVA, seg_pos={0 0}, upload=72429]: Done uploading piece. err: <nil> Dec 9 02:24:06.096707993 (82h55m5.967791028s) [noise=true , address=145.79.227.11:4732 , piece_id=CA5XB6ODOYLW3RQU5F7BMUDDSAGEJGIKOP7IZBHPNUTUXYDJ4JWQ, seg_pos={0 0}, upload=72429]: Done uploading piece. err: <nil> Dec 9 02:24:06.103696047 (82h55m5.974779092s) [noise=true , address=145.79.227.16:4911 , piece_id=Q62EUAKMIIB6QZ3KTNQH3PJAZIX7YNRYXJPTATNCZ4W6RWELTLXQ, seg_pos={0 0}, upload=72429]: Done uploading piece. err: <nil> Dec 9 02:24:06.105006020 (82h55m5.976089065s) [noise=true , address=145.79.227.23:4545 , piece_id=NPORRF6TACVJIVBM3IABBQF7NBYSDRGJTPGDREFMOWNNT3B325UQ, seg_pos={0 0}, upload=72429]: Done uploading piece. err: <nil> Dec 9 02:24:06.105250326 (82h55m5.976333381s) [noise=true , address=145.79.227.38:4978 , piece_id=XBHD6MMZECN7T7QTC3HVN7OS5XY7CJJJEFERUVK2ZIXTVWSVNLXA, seg_pos={0 0}, upload=72429]: Done uploading piece. err: <nil> Dec 9 02:24:06.105501193 (82h55m5.976584238s) [noise=true , address=145.79.227.22:4511 , piece_id=QVG66WWAN3TCESLCDW2ZG7SQADAJPUIILVYGFGTCKRLGVXJWSQ7Q, seg_pos={0 0}, upload=72429]: Done uploading piece. err: <nil> Dec 9 02:24:06.108138354 (82h55m5.979221399s) [noise=true , address=145.79.227.40:4049 , piece_id=VH3PRUHK4OR4MCGVX3VHMXKWTVE3KQLKH5RJHI4MLJMITRHILR2A, seg_pos={0 0}, upload=72429]: Done uploading piece. err: <nil> Dec 9 02:24:06.108158543 (82h55m5.979241588s) [seg_pos={0 0}, upload=72429]: Segment reached optimal threshold of 13 pieces. Dec 9 02:24:06.108688862 (82h55m5.979771907s) [noise=true , address=77.90.31.10:4520 , piece_id=GW2ECPZB4XCEXAWWCBVUM2VEUXXMSBPJUZSKTULZYCHK4DZL63XA, seg_pos={0 0}, upload=72429]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Dec 9 02:24:06.108700394 (82h55m5.979783449s) [noise=true , address=77.90.31.10:4520 , piece_id=GW2ECPZB4XCEXAWWCBVUM2VEUXXMSBPJUZSKTULZYCHK4DZL63XA, seg_pos={0 0}, upload=72429]: Long tail cancellation due to optimal threshold Dec 9 02:24:06.108930402 (82h55m5.980013447s) [noise=true , address=145.79.227.43:4158 , piece_id=DS3UAJZAYVR4GXPHKPMJ2RO7765FUPASW7T7GQH6ATVCM4FAPYLQ, seg_pos={0 0}, upload=72429]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF Dec 9 02:24:06.108941323 (82h55m5.980024368s) [noise=true , address=145.79.227.43:4158 , piece_id=DS3UAJZAYVR4GXPHKPMJ2RO7765FUPASW7T7GQH6ATVCM4FAPYLQ, seg_pos={0 0}, upload=72429]: Long tail cancellation due to optimal threshold Dec 9 02:24:06.108964487 (82h55m5.980047532s) [seg_pos={0 0}, upload=72429]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 02:24:06.109072202 (82h55m5.980155257s) [upload=72429]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 02:24:06.109080047 (82h55m5.980163092s) [upload=72429]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 02:24:24.569429416 (82h55m24.440512451s) [upload=72429]: Done uploading segments... Dec 9 02:24:24.569477928 (82h55m24.440560963s) [upload=72429]: Upload finished. err: metaclient: manager closed: closed: read tcp 188.166.225.170:38662->35.212.10.183:7777: read: connection reset by peer Duration: 5.824165333s core uplink us-select-1
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 2 out of 60 is failed (threshold=3) [.....................X......X...............................] duration: 469.627855ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/4e3d2b60e3f4041b FAIL (2025-12-09 02:30:23 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 469.658365ms core satellite eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [........X.......XX........X.................................] duration: 3.619863012s trace: https://jaeger.dev.storj.io/trace/dbaa73a4151be2e FAIL (2025-12-09 02:30:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.619895375s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.........X.......XX........X................................] duration: 3.498982003s trace: https://jaeger.dev.storj.io/trace/dbaa73a4151be2e FAIL (2025-12-09 02:29:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.499013855s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 6.648587543s trace: https://jaeger.dev.storj.io/trace/40019ff87d088ff7 FAIL (2025-12-09 02:29:54 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.648617739s core uplink us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 5.881969149s trace: https://jaeger.dev.storj.io/trace/6f1ff10066b1d8ea FAIL (2025-12-09 02:29:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== rue , address=185.167.204.206:4052 , piece_id=WY7PGT7EC5IUBTSHFEAMHXONCIKPFPRF3Y4YSL2IJNY2MOO6U7HA, seg_pos={0 0}, upload=72112]: Uploading piece... Dec 9 02:01:05.223188835 (82h32m5.09427188s) [noise=true , address=89.149.201.147:4011 , piece_id=I77RBTAA7KPTBE27RPPPCNYU65WBBUORVCNYAJOV5OHOBGUFEJKA, seg_pos={0 0}, upload=72112]: Uploading piece... Dec 9 02:01:05.223808365 (82h32m5.09489141s) [noise=true , address=185.167.204.206:4050 , piece_id=YKLEM6U44VR3GM45KKKJF7JYOFA4YQE6P5UAZZEMXIA2LBZW5UKA, seg_pos={0 0}, upload=72112]: Uploading piece... Dec 9 02:01:05.224324407 (82h32m5.095407452s) [noise=true , address=185.167.204.205:4011 , piece_id=6JR2JQYKTRLCSBJDUWWSJ6CRYGWKWN3AW3KJCPXMYLYPUN6W2DRQ, seg_pos={0 0}, upload=72112]: Uploading piece... Dec 9 02:01:05.224877841 (82h32m5.095960886s) [noise=true , address=81.171.14.70:4032 , piece_id=APFBMO7KRDDWH6WT3H4ZLCXBAAMZTZQYQBN3XX6JZDGOBAMSGVCA, seg_pos={0 0}, upload=72112]: Uploading piece... Dec 9 02:01:05.225476030 (82h32m5.096559075s) [noise=true , address=89.149.201.147:4014 , piece_id=PFXVAHNZ2Q6YC2MGNSUZMJIL5Q2O2HPDSONU3FD3E472H2G7AG5A, seg_pos={0 0}, upload=72112]: Uploading piece... Dec 9 02:01:05.226119545 (82h32m5.09720258s) [noise=true , address=89.149.201.147:4015 , piece_id=XBCH6ONGHITOZ2Z5GDPPKDV45TILCDFTOZ44OKMSZANCGVBGQ7ZQ, seg_pos={0 0}, upload=72112]: Uploading piece... Dec 9 02:01:05.226816121 (82h32m5.097899166s) [noise=true , address=89.149.201.147:4012 , piece_id=7UFT32WUAW2JUETM2Y4SFGMSPTNLVVH5FKPQJSK6S3DR6E64ITKQ, seg_pos={0 0}, upload=72112]: Uploading piece... Dec 9 02:01:05.227549758 (82h32m5.098632813s) [noise=true , address=185.167.204.206:4067 , piece_id=PK6GGQNT4DQ2ZMF2IEEO5X32IMA26ULKDN7HAAG4CWOC2UAUKMMA, seg_pos={0 0}, upload=72112]: Uploading piece... Dec 9 02:01:05.228376403 (82h32m5.099459448s) [noise=true , address=81.171.14.70:4046 , piece_id=RR4RRWZ6YO3EK62O7TE2QUC3ZJ762VGUTVXVVHPGMQRPWEFTB7VA, seg_pos={0 0}, upload=72112]: Uploading piece... Dec 9 02:01:05.228990242 (82h32m5.100073297s) [noise=true , address=185.167.204.205:4013 , piece_id=7TBB5J7PYW23YFBOESEZ5Q6YOD3FKALVJMUQRMC5JSQ5JYKGDQSA, seg_pos={0 0}, upload=72112]: Uploading piece... Dec 9 02:01:06.054905806 (82h32m5.925988851s) [upload=72112]: Next returned nil segment Dec 9 02:01:06.054930403 (82h32m5.926013448s) [upload=72112]: Waiting for error group managing segments... Dec 9 02:01:06.441184862 (82h32m6.312267897s) [noise=true , address=185.167.204.205:4004 , piece_id=6CC2A4V4TJPRJBN4IMJ7VGDVFYTJIVY6J3ZIPZWYL5TIBLGT744A, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:06.444455871 (82h32m6.315538916s) [noise=true , address=185.167.204.205:4005 , piece_id=QUO2YNK2MJJ63CKWADK5K64GYKHDOSBPFHKYKQSKH2TJQUZTNEVA, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:06.447822552 (82h32m6.318905597s) [noise=true , address=185.167.204.206:4066 , piece_id=SSGUSCBUP2YNFN7OMUZQIXJJMSHV6GDHQYMN475LJFQQ7SV2FV2A, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:06.451954059 (82h32m6.323037104s) [noise=true , address=185.167.204.205:4008 , piece_id=54IT2P63UNQKCHG4HB7CX7UWJFWDLJXXPOGPNAEN36ZAC4UQ6VQQ, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:06.454212810 (82h32m6.325295856s) [noise=true , address=185.167.204.205:4011 , piece_id=6JR2JQYKTRLCSBJDUWWSJ6CRYGWKWN3AW3KJCPXMYLYPUN6W2DRQ, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:06.454331416 (82h32m6.325414461s) [noise=true , address=185.167.204.206:4052 , piece_id=WY7PGT7EC5IUBTSHFEAMHXONCIKPFPRF3Y4YSL2IJNY2MOO6U7HA, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:06.455138002 (82h32m6.326221047s) [noise=true , address=185.167.204.206:4050 , piece_id=YKLEM6U44VR3GM45KKKJF7JYOFA4YQE6P5UAZZEMXIA2LBZW5UKA, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:06.459504045 (82h32m6.33058709s) [noise=true , address=185.167.204.205:4013 , piece_id=7TBB5J7PYW23YFBOESEZ5Q6YOD3FKALVJMUQRMC5JSQ5JYKGDQSA, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:06.465474574 (82h32m6.336557629s) [noise=true , address=185.167.204.206:4067 , piece_id=PK6GGQNT4DQ2ZMF2IEEO5X32IMA26ULKDN7HAAG4CWOC2UAUKMMA, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.060075359 (82h32m6.931158404s) [noise=true , address=81.171.14.70:4034 , piece_id=DNHPDP5YCQLS5AN6OHWRWVOGOIGE53YB7CJVUHNPJK35AEM455GQ, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.111399558 (82h32m6.982482603s) [noise=true , address=81.171.14.70:4026 , piece_id=CRRJW4OCPQUSWT6BERGQJPYYQ4DHSZIBMIBCD22QWCGGHUISCFXQ, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.133611052 (82h32m7.004694097s) [noise=true , address=81.171.14.70:4032 , piece_id=APFBMO7KRDDWH6WT3H4ZLCXBAAMZTZQYQBN3XX6JZDGOBAMSGVCA, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.149778297 (82h32m7.020861342s) [noise=true , address=81.171.14.70:4030 , piece_id=VOYOWF6Y5DZOJ6DLW6QFYUI6DF5YA7FI4VI57LHMNCZ2PZLRY4BQ, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.243693687 (82h32m7.114776732s) [noise=true , address=81.171.14.70:4040 , piece_id=RYDZ6DCJ3Z5YKJPIFMLR7EHPH55S7HABKAJCKGRGHF2TCRKEYHXQ, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.624315751 (82h32m7.495398796s) [noise=true , address=81.171.14.70:4046 , piece_id=RR4RRWZ6YO3EK62O7TE2QUC3ZJ762VGUTVXVVHPGMQRPWEFTB7VA, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.682160867 (82h32m7.553243912s) [noise=true , address=89.149.201.147:4011 , piece_id=I77RBTAA7KPTBE27RPPPCNYU65WBBUORVCNYAJOV5OHOBGUFEJKA, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.704962114 (82h32m7.576045159s) [noise=true , address=89.149.201.147:4003 , piece_id=KQDYYB3S7PMHBAZTSQDATYYBAUYHMHEQDI6PW4GM4HKXBK4XHIVQ, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.723657624 (82h32m7.594740669s) [noise=true , address=89.149.201.147:4008 , piece_id=JCJNV47ZC7LJPXSERGUXP2QRLHVPOAQNOPI25RQZTA52IYKRZK4Q, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.729409366 (82h32m7.600492411s) [noise=true , address=89.149.201.147:4002 , piece_id=YFUKN6FRXESUYMS6ZV53MCQTZV5UXYHFF72WS2XJWGF3K3I72GTQ, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.761878943 (82h32m7.632961988s) [noise=true , address=89.149.201.147:4004 , piece_id=NGEHOFVBOCBBCYIVGSYLMKTFQWTDJ2EOGKYEW4X2OIWYH5TVGX2Q, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.825863449 (82h32m7.696946494s) [noise=true , address=89.149.201.147:4005 , piece_id=K6WGQSQZ32ZSAD6KN5IWB66MJ3YP76IKULPEHTFSODYFBSESSU4A, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.866343990 (82h32m7.737427035s) [noise=true , address=89.149.201.147:4015 , piece_id=XBCH6ONGHITOZ2Z5GDPPKDV45TILCDFTOZ44OKMSZANCGVBGQ7ZQ, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:08.228780506 (82h32m8.099863541s) [noise=true , address=89.149.201.147:4017 , piece_id=5W66LIGVHUXIA4AVNKUNCOAJAP5TCYAKVFKCL6NDF2OVXNI62UMQ, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:08.230729618 (82h32m8.101812653s) [noise=true , address=89.149.201.147:4019 , piece_id=JTQ44O6RFCMIPK7LAEWXO2NTXAZAJK7XR3XAJGVQXLOLM7SFSBCA, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:08.257428165 (82h32m8.12851121s) [noise=true , address=89.149.201.147:4020 , piece_id=T5RN3NMY35NXWM55YN3F7TWPPP7KXEER5R63W4KZV6Y2UNORJX4A, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:08.267970901 (82h32m8.139053946s) [noise=true , address=89.149.201.147:4006 , piece_id=M3VDXP334RID2FJK7ZYNUHJBRMS57EXIMVMLER25MVJWM4EHJ2WA, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:08.267990939 (82h32m8.139073974s) [seg_pos={0 0}, upload=72112]: Segment reached optimal threshold of 26 pieces. Dec 9 02:01:08.268172084 (82h32m8.139255129s) [noise=true , address=89.149.201.147:4012 , piece_id=7UFT32WUAW2JUETM2Y4SFGMSPTNLVVH5FKPQJSK6S3DR6E64ITKQ, seg_pos={0 0}, upload=72112]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hJ5jT6S2d3DJe9DoVDjHMkpwSTpeyTMyNLaK8Pk8SWcUVRKnv): protocol: expected piece hash; context canceled; EOF Dec 9 02:01:08.268182163 (82h32m8.139265198s) [noise=true , address=89.149.201.147:4012 , piece_id=7UFT32WUAW2JUETM2Y4SFGMSPTNLVVH5FKPQJSK6S3DR6E64ITKQ, seg_pos={0 0}, upload=72112]: Long tail cancellation due to optimal threshold Dec 9 02:01:08.268290639 (82h32m8.139373684s) [noise=true , address=89.149.201.147:4014 , piece_id=PFXVAHNZ2Q6YC2MGNSUZMJIL5Q2O2HPDSONU3FD3E472H2G7AG5A, seg_pos={0 0}, upload=72112]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wgtphHKaTejC1qrJ98sDysBwtj2Zeqw2c5UFk1HMqfY8Buvr): protocol: expected piece hash; context canceled; EOF Dec 9 02:01:08.268299597 (82h32m8.139382642s) [noise=true , address=89.149.201.147:4014 , piece_id=PFXVAHNZ2Q6YC2MGNSUZMJIL5Q2O2HPDSONU3FD3E472H2G7AG5A, seg_pos={0 0}, upload=72112]: Long tail cancellation due to optimal threshold Dec 9 02:01:08.268340294 (82h32m8.139423329s) [seg_pos={0 0}, upload=72112]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 9 02:01:08.268435756 (82h32m8.139518802s) [upload=72112]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 02:01:08.268441557 (82h32m8.139524592s) [upload=72112]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 02:12:41.424380970 (82h43m41.295464025s) [upload=72112]: Done uploading segments... Dec 9 02:12:41.424504095 (82h43m41.29558713s) [upload=72112]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.882000008s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 5.982052317s trace: https://jaeger.dev.storj.io/trace/1ddfa89c12e62214 FAIL (2025-12-09 02:29:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: manager closed: closed: read tcp 188.166.225.170:38662->35.212.10.183:7777: read: connection reset by peer ======LOGS====== Dec 9 02:24:03.838353395 (82h55m3.70943644s) [upload=72429]: Starting upload Dec 9 02:24:03.838420683 (82h55m3.709503729s) [upload=72429]: Done starting upload Dec 9 02:24:03.838549719 (82h55m3.709632755s) [upload=72429]: Uploading segments... Dec 9 02:24:03.838688734 (82h55m3.709771779s) [upload=72429]: Got next segment. Inline: false Dec 9 02:24:03.838734311 (82h55m3.709817356s) [upload=72429]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 02:24:03.838741164 (82h55m3.709824209s) [upload=72429]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 02:24:04.086906602 (82h55m3.957989647s) [seg_pos={0 0}, upload=72429]: Begin upload segment... Dec 9 02:24:04.086974701 (82h55m3.958057746s) [seg_pos={0 0}, upload=72429]: Done begin upload segment. Dec 9 02:24:04.087029476 (82h55m3.958112511s) [noise=true , address=145.79.227.24:4581 , piece_id=ETKQGTKTMEKLK42KNN72MBQF62TFTOQO357XKF4WS6NTVBSOANDA, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.087057810 (82h55m3.958140855s) [noise=true , address=145.79.227.16:4911 , piece_id=Q62EUAKMIIB6QZ3KTNQH3PJAZIX7YNRYXJPTATNCZ4W6RWELTLXQ, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.087925792 (82h55m3.959008837s) [noise=true , address=145.79.227.42:4122 , piece_id=ZWY35JIEVFUL3LADBSS4EKYMWMFBTHJWMD6YQ7M4OGKXYAU4LNWQ, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.088596299 (82h55m3.959679344s) [noise=true , address=145.79.227.23:4545 , piece_id=NPORRF6TACVJIVBM3IABBQF7NBYSDRGJTPGDREFMOWNNT3B325UQ, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.089090329 (82h55m3.960173374s) [noise=true , address=145.79.227.43:4158 , piece_id=DS3UAJZAYVR4GXPHKPMJ2RO7765FUPASW7T7GQH6ATVCM4FAPYLQ, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.089582366 (82h55m3.960665411s) [noise=true , address=145.79.227.11:4732 , piece_id=CA5XB6ODOYLW3RQU5F7BMUDDSAGEJGIKOP7IZBHPNUTUXYDJ4JWQ, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.090135129 (82h55m3.961218174s) [noise=true , address=145.79.227.34:4906 , piece_id=GTGD6FTBFJ6VMETYMRJPTO7P77B4TBQT4VSRAEMUCJ6TTW4VBHDA, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.090616965 (82h55m3.96170001s) [noise=true , address=145.79.227.24:4583 , piece_id=MFGUNU4E6DBCQ5TUNPSWBXRSEQ7QGTNWTWCUTZK5NAVLMIBB2KKQ, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.091103331 (82h55m3.962186376s) [noise=true , address=145.79.227.25:4619 , piece_id=RALQO5F6I5M4Q56PLG47D57UH4WHRQWB47N45NASEPILQGC26VQQ, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.091901141 (82h55m3.962984186s) [noise=true , address=145.79.227.38:4978 , piece_id=XBHD6MMZECN7T7QTC3HVN7OS5XY7CJJJEFERUVK2ZIXTVWSVNLXA, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.094168697 (82h55m3.965251742s) [noise=true , address=77.90.31.16:4734 , piece_id=WSIBQS2XW3PENG55ORDRTUYXPZ6HCSTAX7DP5JFDVVCLCX6JGLTA, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.094774330 (82h55m3.965857375s) [noise=true , address=145.79.227.40:4049 , piece_id=VH3PRUHK4OR4MCGVX3VHMXKWTVE3KQLKH5RJHI4MLJMITRHILR2A, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.095474052 (82h55m3.966557097s) [noise=true , address=145.79.227.31:4797 , piece_id=XDQGJD7JELZ3D73GMYRVHKWEFIZGYF6IWDBZ54NIJTGA3G7WWMFQ, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.096054146 (82h55m3.967137191s) [noise=true , address=145.79.227.21:4474 , piece_id=TTDRH27TBF34WUNSWZL3XL2RR6SJGEHP7J4YCHAZB4SH3ZWH4PVA, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.096661643 (82h55m3.967744688s) [noise=true , address=77.90.31.10:4520 , piece_id=GW2ECPZB4XCEXAWWCBVUM2VEUXXMSBPJUZSKTULZYCHK4DZL63XA, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.097203715 (82h55m3.96828676s) [noise=true , address=145.79.227.22:4511 , piece_id=QVG66WWAN3TCESLCDW2ZG7SQADAJPUIILVYGFGTCKRLGVXJWSQ7Q, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:05.228786599 (82h55m5.099869634s) [noise=true , address=77.90.31.16:4734 , piece_id=WSIBQS2XW3PENG55ORDRTUYXPZ6HCSTAX7DP5JFDVVCLCX6JGLTA, seg_pos={0 0}, upload=72429]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 9 02:24:05.771060198 (82h55m5.642143243s) [upload=72429]: Next returned nil segment Dec 9 02:24:05.771079454 (82h55m5.642162529s) [upload=72429]: Waiting for error group managing segments... Dec 9 02:24:06.084934630 (82h55m5.956017675s) [noise=true , address=145.79.227.24:4583 , piece_id=MFGUNU4E6DBCQ5TUNPSWBXRSEQ7QGTNWTWCUTZK5NAVLMIBB2KKQ, seg_pos={0 0}, upload=72429]: Done uploading piece. err: <nil> Dec 9 02:24:06.086488888 (82h55m5.957571934s) [noise=true , address=145.79.227.42:4122 , piece_id=ZWY35JIEVFUL3LADBSS4EKYMWMFBTHJWMD6YQ7M4OGKXYAU4LNWQ, seg_pos={0 0}, upload=72429]: Done uploading piece. err: <nil> Dec 9 02:24:06.087432816 (82h55m5.958515861s) [noise=true , address=145.79.227.25:4619 , piece_id=RALQO5F6I5M4Q56PLG47D57UH4WHRQWB47N45NASEPILQGC26VQQ, seg_pos={0 0}, upload=72429]: Done uploading piece. err: <nil> Dec 9 02:24:06.088224853 (82h55m5.959307898s) [noise=true , address=145.79.227.24:4581 , piece_id=ETKQGTKTMEKLK42KNN72MBQF62TFTOQO357XKF4WS6NTVBSOANDA, seg_pos={0 0}, upload=72429]: Done uploading piece. err: <nil> Dec 9 02:24:06.090286018 (82h55m5.961369053s) [noise=true , address=145.79.227.31:4797 , piece_id=XDQGJD7JELZ3D73GMYRVHKWEFIZGYF6IWDBZ54NIJTGA3G7WWMFQ, seg_pos={0 0}, upload=72429]: Done uploading piece. err: <nil> Dec 9 02:24:06.094149082 (82h55m5.965232127s) [noise=true , address=145.79.227.34:4906 , piece_id=GTGD6FTBFJ6VMETYMRJPTO7P77B4TBQT4VSRAEMUCJ6TTW4VBHDA, seg_pos={0 0}, upload=72429]: Done uploading piece. err: <nil> Dec 9 02:24:06.096430375 (82h55m5.96751342s) [noise=true , address=145.79.227.21:4474 , piece_id=TTDRH27TBF34WUNSWZL3XL2RR6SJGEHP7J4YCHAZB4SH3ZWH4PVA, seg_pos={0 0}, upload=72429]: Done uploading piece. err: <nil> Dec 9 02:24:06.096707993 (82h55m5.967791028s) [noise=true , address=145.79.227.11:4732 , piece_id=CA5XB6ODOYLW3RQU5F7BMUDDSAGEJGIKOP7IZBHPNUTUXYDJ4JWQ, seg_pos={0 0}, upload=72429]: Done uploading piece. err: <nil> Dec 9 02:24:06.103696047 (82h55m5.974779092s) [noise=true , address=145.79.227.16:4911 , piece_id=Q62EUAKMIIB6QZ3KTNQH3PJAZIX7YNRYXJPTATNCZ4W6RWELTLXQ, seg_pos={0 0}, upload=72429]: Done uploading piece. err: <nil> Dec 9 02:24:06.105006020 (82h55m5.976089065s) [noise=true , address=145.79.227.23:4545 , piece_id=NPORRF6TACVJIVBM3IABBQF7NBYSDRGJTPGDREFMOWNNT3B325UQ, seg_pos={0 0}, upload=72429]: Done uploading piece. err: <nil> Dec 9 02:24:06.105250326 (82h55m5.976333381s) [noise=true , address=145.79.227.38:4978 , piece_id=XBHD6MMZECN7T7QTC3HVN7OS5XY7CJJJEFERUVK2ZIXTVWSVNLXA, seg_pos={0 0}, upload=72429]: Done uploading piece. err: <nil> Dec 9 02:24:06.105501193 (82h55m5.976584238s) [noise=true , address=145.79.227.22:4511 , piece_id=QVG66WWAN3TCESLCDW2ZG7SQADAJPUIILVYGFGTCKRLGVXJWSQ7Q, seg_pos={0 0}, upload=72429]: Done uploading piece. err: <nil> Dec 9 02:24:06.108138354 (82h55m5.979221399s) [noise=true , address=145.79.227.40:4049 , piece_id=VH3PRUHK4OR4MCGVX3VHMXKWTVE3KQLKH5RJHI4MLJMITRHILR2A, seg_pos={0 0}, upload=72429]: Done uploading piece. err: <nil> Dec 9 02:24:06.108158543 (82h55m5.979241588s) [seg_pos={0 0}, upload=72429]: Segment reached optimal threshold of 13 pieces. Dec 9 02:24:06.108688862 (82h55m5.979771907s) [noise=true , address=77.90.31.10:4520 , piece_id=GW2ECPZB4XCEXAWWCBVUM2VEUXXMSBPJUZSKTULZYCHK4DZL63XA, seg_pos={0 0}, upload=72429]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Dec 9 02:24:06.108700394 (82h55m5.979783449s) [noise=true , address=77.90.31.10:4520 , piece_id=GW2ECPZB4XCEXAWWCBVUM2VEUXXMSBPJUZSKTULZYCHK4DZL63XA, seg_pos={0 0}, upload=72429]: Long tail cancellation due to optimal threshold Dec 9 02:24:06.108930402 (82h55m5.980013447s) [noise=true , address=145.79.227.43:4158 , piece_id=DS3UAJZAYVR4GXPHKPMJ2RO7765FUPASW7T7GQH6ATVCM4FAPYLQ, seg_pos={0 0}, upload=72429]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF Dec 9 02:24:06.108941323 (82h55m5.980024368s) [noise=true , address=145.79.227.43:4158 , piece_id=DS3UAJZAYVR4GXPHKPMJ2RO7765FUPASW7T7GQH6ATVCM4FAPYLQ, seg_pos={0 0}, upload=72429]: Long tail cancellation due to optimal threshold Dec 9 02:24:06.108964487 (82h55m5.980047532s) [seg_pos={0 0}, upload=72429]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 02:24:06.109072202 (82h55m5.980155257s) [upload=72429]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 02:24:06.109080047 (82h55m5.980163092s) [upload=72429]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 02:24:24.569429416 (82h55m24.440512451s) [upload=72429]: Done uploading segments... Dec 9 02:24:24.569477928 (82h55m24.440560963s) [upload=72429]: Upload finished. err: metaclient: manager closed: closed: read tcp 188.166.225.170:38662->35.212.10.183:7777: read: connection reset by peer Duration: 5.982078688s core uplink us-select-1
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 2 out of 60 is failed (threshold=3) [......................X......X..............................] duration: 469.717998ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/4e3d2b60e3f4041b FAIL (2025-12-09 02:29:30 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 469.746862ms core satellite eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [..........X.......XX........X...............................] duration: 3.591995539s trace: https://jaeger.dev.storj.io/trace/dbaa73a4151be2e FAIL (2025-12-09 02:29:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.592063188s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..................X.X.....X.................................] duration: 5.835008912s trace: https://jaeger.dev.storj.io/trace/5280cd334ed3ed6b FAIL (2025-12-09 02:29:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.83502881s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 6.229829071s trace: https://jaeger.dev.storj.io/trace/6f1ff10066b1d8ea FAIL (2025-12-09 02:29:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== rue , address=185.167.204.206:4052 , piece_id=WY7PGT7EC5IUBTSHFEAMHXONCIKPFPRF3Y4YSL2IJNY2MOO6U7HA, seg_pos={0 0}, upload=72112]: Uploading piece... Dec 9 02:01:05.223188835 (82h32m5.09427188s) [noise=true , address=89.149.201.147:4011 , piece_id=I77RBTAA7KPTBE27RPPPCNYU65WBBUORVCNYAJOV5OHOBGUFEJKA, seg_pos={0 0}, upload=72112]: Uploading piece... Dec 9 02:01:05.223808365 (82h32m5.09489141s) [noise=true , address=185.167.204.206:4050 , piece_id=YKLEM6U44VR3GM45KKKJF7JYOFA4YQE6P5UAZZEMXIA2LBZW5UKA, seg_pos={0 0}, upload=72112]: Uploading piece... Dec 9 02:01:05.224324407 (82h32m5.095407452s) [noise=true , address=185.167.204.205:4011 , piece_id=6JR2JQYKTRLCSBJDUWWSJ6CRYGWKWN3AW3KJCPXMYLYPUN6W2DRQ, seg_pos={0 0}, upload=72112]: Uploading piece... Dec 9 02:01:05.224877841 (82h32m5.095960886s) [noise=true , address=81.171.14.70:4032 , piece_id=APFBMO7KRDDWH6WT3H4ZLCXBAAMZTZQYQBN3XX6JZDGOBAMSGVCA, seg_pos={0 0}, upload=72112]: Uploading piece... Dec 9 02:01:05.225476030 (82h32m5.096559075s) [noise=true , address=89.149.201.147:4014 , piece_id=PFXVAHNZ2Q6YC2MGNSUZMJIL5Q2O2HPDSONU3FD3E472H2G7AG5A, seg_pos={0 0}, upload=72112]: Uploading piece... Dec 9 02:01:05.226119545 (82h32m5.09720258s) [noise=true , address=89.149.201.147:4015 , piece_id=XBCH6ONGHITOZ2Z5GDPPKDV45TILCDFTOZ44OKMSZANCGVBGQ7ZQ, seg_pos={0 0}, upload=72112]: Uploading piece... Dec 9 02:01:05.226816121 (82h32m5.097899166s) [noise=true , address=89.149.201.147:4012 , piece_id=7UFT32WUAW2JUETM2Y4SFGMSPTNLVVH5FKPQJSK6S3DR6E64ITKQ, seg_pos={0 0}, upload=72112]: Uploading piece... Dec 9 02:01:05.227549758 (82h32m5.098632813s) [noise=true , address=185.167.204.206:4067 , piece_id=PK6GGQNT4DQ2ZMF2IEEO5X32IMA26ULKDN7HAAG4CWOC2UAUKMMA, seg_pos={0 0}, upload=72112]: Uploading piece... Dec 9 02:01:05.228376403 (82h32m5.099459448s) [noise=true , address=81.171.14.70:4046 , piece_id=RR4RRWZ6YO3EK62O7TE2QUC3ZJ762VGUTVXVVHPGMQRPWEFTB7VA, seg_pos={0 0}, upload=72112]: Uploading piece... Dec 9 02:01:05.228990242 (82h32m5.100073297s) [noise=true , address=185.167.204.205:4013 , piece_id=7TBB5J7PYW23YFBOESEZ5Q6YOD3FKALVJMUQRMC5JSQ5JYKGDQSA, seg_pos={0 0}, upload=72112]: Uploading piece... Dec 9 02:01:06.054905806 (82h32m5.925988851s) [upload=72112]: Next returned nil segment Dec 9 02:01:06.054930403 (82h32m5.926013448s) [upload=72112]: Waiting for error group managing segments... Dec 9 02:01:06.441184862 (82h32m6.312267897s) [noise=true , address=185.167.204.205:4004 , piece_id=6CC2A4V4TJPRJBN4IMJ7VGDVFYTJIVY6J3ZIPZWYL5TIBLGT744A, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:06.444455871 (82h32m6.315538916s) [noise=true , address=185.167.204.205:4005 , piece_id=QUO2YNK2MJJ63CKWADK5K64GYKHDOSBPFHKYKQSKH2TJQUZTNEVA, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:06.447822552 (82h32m6.318905597s) [noise=true , address=185.167.204.206:4066 , piece_id=SSGUSCBUP2YNFN7OMUZQIXJJMSHV6GDHQYMN475LJFQQ7SV2FV2A, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:06.451954059 (82h32m6.323037104s) [noise=true , address=185.167.204.205:4008 , piece_id=54IT2P63UNQKCHG4HB7CX7UWJFWDLJXXPOGPNAEN36ZAC4UQ6VQQ, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:06.454212810 (82h32m6.325295856s) [noise=true , address=185.167.204.205:4011 , piece_id=6JR2JQYKTRLCSBJDUWWSJ6CRYGWKWN3AW3KJCPXMYLYPUN6W2DRQ, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:06.454331416 (82h32m6.325414461s) [noise=true , address=185.167.204.206:4052 , piece_id=WY7PGT7EC5IUBTSHFEAMHXONCIKPFPRF3Y4YSL2IJNY2MOO6U7HA, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:06.455138002 (82h32m6.326221047s) [noise=true , address=185.167.204.206:4050 , piece_id=YKLEM6U44VR3GM45KKKJF7JYOFA4YQE6P5UAZZEMXIA2LBZW5UKA, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:06.459504045 (82h32m6.33058709s) [noise=true , address=185.167.204.205:4013 , piece_id=7TBB5J7PYW23YFBOESEZ5Q6YOD3FKALVJMUQRMC5JSQ5JYKGDQSA, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:06.465474574 (82h32m6.336557629s) [noise=true , address=185.167.204.206:4067 , piece_id=PK6GGQNT4DQ2ZMF2IEEO5X32IMA26ULKDN7HAAG4CWOC2UAUKMMA, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.060075359 (82h32m6.931158404s) [noise=true , address=81.171.14.70:4034 , piece_id=DNHPDP5YCQLS5AN6OHWRWVOGOIGE53YB7CJVUHNPJK35AEM455GQ, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.111399558 (82h32m6.982482603s) [noise=true , address=81.171.14.70:4026 , piece_id=CRRJW4OCPQUSWT6BERGQJPYYQ4DHSZIBMIBCD22QWCGGHUISCFXQ, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.133611052 (82h32m7.004694097s) [noise=true , address=81.171.14.70:4032 , piece_id=APFBMO7KRDDWH6WT3H4ZLCXBAAMZTZQYQBN3XX6JZDGOBAMSGVCA, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.149778297 (82h32m7.020861342s) [noise=true , address=81.171.14.70:4030 , piece_id=VOYOWF6Y5DZOJ6DLW6QFYUI6DF5YA7FI4VI57LHMNCZ2PZLRY4BQ, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.243693687 (82h32m7.114776732s) [noise=true , address=81.171.14.70:4040 , piece_id=RYDZ6DCJ3Z5YKJPIFMLR7EHPH55S7HABKAJCKGRGHF2TCRKEYHXQ, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.624315751 (82h32m7.495398796s) [noise=true , address=81.171.14.70:4046 , piece_id=RR4RRWZ6YO3EK62O7TE2QUC3ZJ762VGUTVXVVHPGMQRPWEFTB7VA, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.682160867 (82h32m7.553243912s) [noise=true , address=89.149.201.147:4011 , piece_id=I77RBTAA7KPTBE27RPPPCNYU65WBBUORVCNYAJOV5OHOBGUFEJKA, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.704962114 (82h32m7.576045159s) [noise=true , address=89.149.201.147:4003 , piece_id=KQDYYB3S7PMHBAZTSQDATYYBAUYHMHEQDI6PW4GM4HKXBK4XHIVQ, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.723657624 (82h32m7.594740669s) [noise=true , address=89.149.201.147:4008 , piece_id=JCJNV47ZC7LJPXSERGUXP2QRLHVPOAQNOPI25RQZTA52IYKRZK4Q, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.729409366 (82h32m7.600492411s) [noise=true , address=89.149.201.147:4002 , piece_id=YFUKN6FRXESUYMS6ZV53MCQTZV5UXYHFF72WS2XJWGF3K3I72GTQ, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.761878943 (82h32m7.632961988s) [noise=true , address=89.149.201.147:4004 , piece_id=NGEHOFVBOCBBCYIVGSYLMKTFQWTDJ2EOGKYEW4X2OIWYH5TVGX2Q, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.825863449 (82h32m7.696946494s) [noise=true , address=89.149.201.147:4005 , piece_id=K6WGQSQZ32ZSAD6KN5IWB66MJ3YP76IKULPEHTFSODYFBSESSU4A, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.866343990 (82h32m7.737427035s) [noise=true , address=89.149.201.147:4015 , piece_id=XBCH6ONGHITOZ2Z5GDPPKDV45TILCDFTOZ44OKMSZANCGVBGQ7ZQ, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:08.228780506 (82h32m8.099863541s) [noise=true , address=89.149.201.147:4017 , piece_id=5W66LIGVHUXIA4AVNKUNCOAJAP5TCYAKVFKCL6NDF2OVXNI62UMQ, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:08.230729618 (82h32m8.101812653s) [noise=true , address=89.149.201.147:4019 , piece_id=JTQ44O6RFCMIPK7LAEWXO2NTXAZAJK7XR3XAJGVQXLOLM7SFSBCA, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:08.257428165 (82h32m8.12851121s) [noise=true , address=89.149.201.147:4020 , piece_id=T5RN3NMY35NXWM55YN3F7TWPPP7KXEER5R63W4KZV6Y2UNORJX4A, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:08.267970901 (82h32m8.139053946s) [noise=true , address=89.149.201.147:4006 , piece_id=M3VDXP334RID2FJK7ZYNUHJBRMS57EXIMVMLER25MVJWM4EHJ2WA, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:08.267990939 (82h32m8.139073974s) [seg_pos={0 0}, upload=72112]: Segment reached optimal threshold of 26 pieces. Dec 9 02:01:08.268172084 (82h32m8.139255129s) [noise=true , address=89.149.201.147:4012 , piece_id=7UFT32WUAW2JUETM2Y4SFGMSPTNLVVH5FKPQJSK6S3DR6E64ITKQ, seg_pos={0 0}, upload=72112]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hJ5jT6S2d3DJe9DoVDjHMkpwSTpeyTMyNLaK8Pk8SWcUVRKnv): protocol: expected piece hash; context canceled; EOF Dec 9 02:01:08.268182163 (82h32m8.139265198s) [noise=true , address=89.149.201.147:4012 , piece_id=7UFT32WUAW2JUETM2Y4SFGMSPTNLVVH5FKPQJSK6S3DR6E64ITKQ, seg_pos={0 0}, upload=72112]: Long tail cancellation due to optimal threshold Dec 9 02:01:08.268290639 (82h32m8.139373684s) [noise=true , address=89.149.201.147:4014 , piece_id=PFXVAHNZ2Q6YC2MGNSUZMJIL5Q2O2HPDSONU3FD3E472H2G7AG5A, seg_pos={0 0}, upload=72112]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wgtphHKaTejC1qrJ98sDysBwtj2Zeqw2c5UFk1HMqfY8Buvr): protocol: expected piece hash; context canceled; EOF Dec 9 02:01:08.268299597 (82h32m8.139382642s) [noise=true , address=89.149.201.147:4014 , piece_id=PFXVAHNZ2Q6YC2MGNSUZMJIL5Q2O2HPDSONU3FD3E472H2G7AG5A, seg_pos={0 0}, upload=72112]: Long tail cancellation due to optimal threshold Dec 9 02:01:08.268340294 (82h32m8.139423329s) [seg_pos={0 0}, upload=72112]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 9 02:01:08.268435756 (82h32m8.139518802s) [upload=72112]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 02:01:08.268441557 (82h32m8.139524592s) [upload=72112]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 02:12:41.424380970 (82h43m41.295464025s) [upload=72112]: Done uploading segments... Dec 9 02:12:41.424504095 (82h43m41.29558713s) [upload=72112]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.229859941s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 6.130491973s trace: https://jaeger.dev.storj.io/trace/40019ff87d088ff7 FAIL (2025-12-09 02:29:06 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.13053206s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 5.795674523s trace: https://jaeger.dev.storj.io/trace/1ddfa89c12e62214 FAIL (2025-12-09 02:28:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: manager closed: closed: read tcp 188.166.225.170:38662->35.212.10.183:7777: read: connection reset by peer ======LOGS====== Dec 9 02:24:03.838353395 (82h55m3.70943644s) [upload=72429]: Starting upload Dec 9 02:24:03.838420683 (82h55m3.709503729s) [upload=72429]: Done starting upload Dec 9 02:24:03.838549719 (82h55m3.709632755s) [upload=72429]: Uploading segments... Dec 9 02:24:03.838688734 (82h55m3.709771779s) [upload=72429]: Got next segment. Inline: false Dec 9 02:24:03.838734311 (82h55m3.709817356s) [upload=72429]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 02:24:03.838741164 (82h55m3.709824209s) [upload=72429]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 02:24:04.086906602 (82h55m3.957989647s) [seg_pos={0 0}, upload=72429]: Begin upload segment... Dec 9 02:24:04.086974701 (82h55m3.958057746s) [seg_pos={0 0}, upload=72429]: Done begin upload segment. Dec 9 02:24:04.087029476 (82h55m3.958112511s) [noise=true , address=145.79.227.24:4581 , piece_id=ETKQGTKTMEKLK42KNN72MBQF62TFTOQO357XKF4WS6NTVBSOANDA, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.087057810 (82h55m3.958140855s) [noise=true , address=145.79.227.16:4911 , piece_id=Q62EUAKMIIB6QZ3KTNQH3PJAZIX7YNRYXJPTATNCZ4W6RWELTLXQ, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.087925792 (82h55m3.959008837s) [noise=true , address=145.79.227.42:4122 , piece_id=ZWY35JIEVFUL3LADBSS4EKYMWMFBTHJWMD6YQ7M4OGKXYAU4LNWQ, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.088596299 (82h55m3.959679344s) [noise=true , address=145.79.227.23:4545 , piece_id=NPORRF6TACVJIVBM3IABBQF7NBYSDRGJTPGDREFMOWNNT3B325UQ, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.089090329 (82h55m3.960173374s) [noise=true , address=145.79.227.43:4158 , piece_id=DS3UAJZAYVR4GXPHKPMJ2RO7765FUPASW7T7GQH6ATVCM4FAPYLQ, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.089582366 (82h55m3.960665411s) [noise=true , address=145.79.227.11:4732 , piece_id=CA5XB6ODOYLW3RQU5F7BMUDDSAGEJGIKOP7IZBHPNUTUXYDJ4JWQ, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.090135129 (82h55m3.961218174s) [noise=true , address=145.79.227.34:4906 , piece_id=GTGD6FTBFJ6VMETYMRJPTO7P77B4TBQT4VSRAEMUCJ6TTW4VBHDA, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.090616965 (82h55m3.96170001s) [noise=true , address=145.79.227.24:4583 , piece_id=MFGUNU4E6DBCQ5TUNPSWBXRSEQ7QGTNWTWCUTZK5NAVLMIBB2KKQ, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.091103331 (82h55m3.962186376s) [noise=true , address=145.79.227.25:4619 , piece_id=RALQO5F6I5M4Q56PLG47D57UH4WHRQWB47N45NASEPILQGC26VQQ, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.091901141 (82h55m3.962984186s) [noise=true , address=145.79.227.38:4978 , piece_id=XBHD6MMZECN7T7QTC3HVN7OS5XY7CJJJEFERUVK2ZIXTVWSVNLXA, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.094168697 (82h55m3.965251742s) [noise=true , address=77.90.31.16:4734 , piece_id=WSIBQS2XW3PENG55ORDRTUYXPZ6HCSTAX7DP5JFDVVCLCX6JGLTA, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.094774330 (82h55m3.965857375s) [noise=true , address=145.79.227.40:4049 , piece_id=VH3PRUHK4OR4MCGVX3VHMXKWTVE3KQLKH5RJHI4MLJMITRHILR2A, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.095474052 (82h55m3.966557097s) [noise=true , address=145.79.227.31:4797 , piece_id=XDQGJD7JELZ3D73GMYRVHKWEFIZGYF6IWDBZ54NIJTGA3G7WWMFQ, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.096054146 (82h55m3.967137191s) [noise=true , address=145.79.227.21:4474 , piece_id=TTDRH27TBF34WUNSWZL3XL2RR6SJGEHP7J4YCHAZB4SH3ZWH4PVA, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.096661643 (82h55m3.967744688s) [noise=true , address=77.90.31.10:4520 , piece_id=GW2ECPZB4XCEXAWWCBVUM2VEUXXMSBPJUZSKTULZYCHK4DZL63XA, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.097203715 (82h55m3.96828676s) [noise=true , address=145.79.227.22:4511 , piece_id=QVG66WWAN3TCESLCDW2ZG7SQADAJPUIILVYGFGTCKRLGVXJWSQ7Q, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:05.228786599 (82h55m5.099869634s) [noise=true , address=77.90.31.16:4734 , piece_id=WSIBQS2XW3PENG55ORDRTUYXPZ6HCSTAX7DP5JFDVVCLCX6JGLTA, seg_pos={0 0}, upload=72429]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 9 02:24:05.771060198 (82h55m5.642143243s) [upload=72429]: Next returned nil segment Dec 9 02:24:05.771079454 (82h55m5.642162529s) [upload=72429]: Waiting for error group managing segments... Dec 9 02:24:06.084934630 (82h55m5.956017675s) [noise=true , address=145.79.227.24:4583 , piece_id=MFGUNU4E6DBCQ5TUNPSWBXRSEQ7QGTNWTWCUTZK5NAVLMIBB2KKQ, seg_pos={0 0}, upload=72429]: Done uploading piece. err: <nil> Dec 9 02:24:06.086488888 (82h55m5.957571934s) [noise=true , address=145.79.227.42:4122 , piece_id=ZWY35JIEVFUL3LADBSS4EKYMWMFBTHJWMD6YQ7M4OGKXYAU4LNWQ, seg_pos={0 0}, upload=72429]: Done uploading piece. err: <nil> Dec 9 02:24:06.087432816 (82h55m5.958515861s) [noise=true , address=145.79.227.25:4619 , piece_id=RALQO5F6I5M4Q56PLG47D57UH4WHRQWB47N45NASEPILQGC26VQQ, seg_pos={0 0}, upload=72429]: Done uploading piece. err: <nil> Dec 9 02:24:06.088224853 (82h55m5.959307898s) [noise=true , address=145.79.227.24:4581 , piece_id=ETKQGTKTMEKLK42KNN72MBQF62TFTOQO357XKF4WS6NTVBSOANDA, seg_pos={0 0}, upload=72429]: Done uploading piece. err: <nil> Dec 9 02:24:06.090286018 (82h55m5.961369053s) [noise=true , address=145.79.227.31:4797 , piece_id=XDQGJD7JELZ3D73GMYRVHKWEFIZGYF6IWDBZ54NIJTGA3G7WWMFQ, seg_pos={0 0}, upload=72429]: Done uploading piece. err: <nil> Dec 9 02:24:06.094149082 (82h55m5.965232127s) [noise=true , address=145.79.227.34:4906 , piece_id=GTGD6FTBFJ6VMETYMRJPTO7P77B4TBQT4VSRAEMUCJ6TTW4VBHDA, seg_pos={0 0}, upload=72429]: Done uploading piece. err: <nil> Dec 9 02:24:06.096430375 (82h55m5.96751342s) [noise=true , address=145.79.227.21:4474 , piece_id=TTDRH27TBF34WUNSWZL3XL2RR6SJGEHP7J4YCHAZB4SH3ZWH4PVA, seg_pos={0 0}, upload=72429]: Done uploading piece. err: <nil> Dec 9 02:24:06.096707993 (82h55m5.967791028s) [noise=true , address=145.79.227.11:4732 , piece_id=CA5XB6ODOYLW3RQU5F7BMUDDSAGEJGIKOP7IZBHPNUTUXYDJ4JWQ, seg_pos={0 0}, upload=72429]: Done uploading piece. err: <nil> Dec 9 02:24:06.103696047 (82h55m5.974779092s) [noise=true , address=145.79.227.16:4911 , piece_id=Q62EUAKMIIB6QZ3KTNQH3PJAZIX7YNRYXJPTATNCZ4W6RWELTLXQ, seg_pos={0 0}, upload=72429]: Done uploading piece. err: <nil> Dec 9 02:24:06.105006020 (82h55m5.976089065s) [noise=true , address=145.79.227.23:4545 , piece_id=NPORRF6TACVJIVBM3IABBQF7NBYSDRGJTPGDREFMOWNNT3B325UQ, seg_pos={0 0}, upload=72429]: Done uploading piece. err: <nil> Dec 9 02:24:06.105250326 (82h55m5.976333381s) [noise=true , address=145.79.227.38:4978 , piece_id=XBHD6MMZECN7T7QTC3HVN7OS5XY7CJJJEFERUVK2ZIXTVWSVNLXA, seg_pos={0 0}, upload=72429]: Done uploading piece. err: <nil> Dec 9 02:24:06.105501193 (82h55m5.976584238s) [noise=true , address=145.79.227.22:4511 , piece_id=QVG66WWAN3TCESLCDW2ZG7SQADAJPUIILVYGFGTCKRLGVXJWSQ7Q, seg_pos={0 0}, upload=72429]: Done uploading piece. err: <nil> Dec 9 02:24:06.108138354 (82h55m5.979221399s) [noise=true , address=145.79.227.40:4049 , piece_id=VH3PRUHK4OR4MCGVX3VHMXKWTVE3KQLKH5RJHI4MLJMITRHILR2A, seg_pos={0 0}, upload=72429]: Done uploading piece. err: <nil> Dec 9 02:24:06.108158543 (82h55m5.979241588s) [seg_pos={0 0}, upload=72429]: Segment reached optimal threshold of 13 pieces. Dec 9 02:24:06.108688862 (82h55m5.979771907s) [noise=true , address=77.90.31.10:4520 , piece_id=GW2ECPZB4XCEXAWWCBVUM2VEUXXMSBPJUZSKTULZYCHK4DZL63XA, seg_pos={0 0}, upload=72429]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Dec 9 02:24:06.108700394 (82h55m5.979783449s) [noise=true , address=77.90.31.10:4520 , piece_id=GW2ECPZB4XCEXAWWCBVUM2VEUXXMSBPJUZSKTULZYCHK4DZL63XA, seg_pos={0 0}, upload=72429]: Long tail cancellation due to optimal threshold Dec 9 02:24:06.108930402 (82h55m5.980013447s) [noise=true , address=145.79.227.43:4158 , piece_id=DS3UAJZAYVR4GXPHKPMJ2RO7765FUPASW7T7GQH6ATVCM4FAPYLQ, seg_pos={0 0}, upload=72429]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF Dec 9 02:24:06.108941323 (82h55m5.980024368s) [noise=true , address=145.79.227.43:4158 , piece_id=DS3UAJZAYVR4GXPHKPMJ2RO7765FUPASW7T7GQH6ATVCM4FAPYLQ, seg_pos={0 0}, upload=72429]: Long tail cancellation due to optimal threshold Dec 9 02:24:06.108964487 (82h55m5.980047532s) [seg_pos={0 0}, upload=72429]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 02:24:06.109072202 (82h55m5.980155257s) [upload=72429]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 02:24:06.109080047 (82h55m5.980163092s) [upload=72429]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 02:24:24.569429416 (82h55m24.440512451s) [upload=72429]: Done uploading segments... Dec 9 02:24:24.569477928 (82h55m24.440560963s) [upload=72429]: Upload finished. err: metaclient: manager closed: closed: read tcp 188.166.225.170:38662->35.212.10.183:7777: read: connection reset by peer Duration: 5.795705752s core uplink us-select-1
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 2 out of 60 is failed (threshold=3) [.......................X......X.............................] duration: 470.597972ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/4e3d2b60e3f4041b FAIL (2025-12-09 02:28:52 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 470.627028ms core satellite eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [...........X.......XX........X..............................] duration: 3.515082352s trace: https://jaeger.dev.storj.io/trace/dbaa73a4151be2e FAIL (2025-12-09 02:28:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.515178004s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 5.88079603s trace: https://jaeger.dev.storj.io/trace/40019ff87d088ff7 FAIL (2025-12-09 02:28:24 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.880829634s core uplink us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 6.258128274s trace: https://jaeger.dev.storj.io/trace/6f1ff10066b1d8ea FAIL (2025-12-09 02:28:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== rue , address=185.167.204.206:4052 , piece_id=WY7PGT7EC5IUBTSHFEAMHXONCIKPFPRF3Y4YSL2IJNY2MOO6U7HA, seg_pos={0 0}, upload=72112]: Uploading piece... Dec 9 02:01:05.223188835 (82h32m5.09427188s) [noise=true , address=89.149.201.147:4011 , piece_id=I77RBTAA7KPTBE27RPPPCNYU65WBBUORVCNYAJOV5OHOBGUFEJKA, seg_pos={0 0}, upload=72112]: Uploading piece... Dec 9 02:01:05.223808365 (82h32m5.09489141s) [noise=true , address=185.167.204.206:4050 , piece_id=YKLEM6U44VR3GM45KKKJF7JYOFA4YQE6P5UAZZEMXIA2LBZW5UKA, seg_pos={0 0}, upload=72112]: Uploading piece... Dec 9 02:01:05.224324407 (82h32m5.095407452s) [noise=true , address=185.167.204.205:4011 , piece_id=6JR2JQYKTRLCSBJDUWWSJ6CRYGWKWN3AW3KJCPXMYLYPUN6W2DRQ, seg_pos={0 0}, upload=72112]: Uploading piece... Dec 9 02:01:05.224877841 (82h32m5.095960886s) [noise=true , address=81.171.14.70:4032 , piece_id=APFBMO7KRDDWH6WT3H4ZLCXBAAMZTZQYQBN3XX6JZDGOBAMSGVCA, seg_pos={0 0}, upload=72112]: Uploading piece... Dec 9 02:01:05.225476030 (82h32m5.096559075s) [noise=true , address=89.149.201.147:4014 , piece_id=PFXVAHNZ2Q6YC2MGNSUZMJIL5Q2O2HPDSONU3FD3E472H2G7AG5A, seg_pos={0 0}, upload=72112]: Uploading piece... Dec 9 02:01:05.226119545 (82h32m5.09720258s) [noise=true , address=89.149.201.147:4015 , piece_id=XBCH6ONGHITOZ2Z5GDPPKDV45TILCDFTOZ44OKMSZANCGVBGQ7ZQ, seg_pos={0 0}, upload=72112]: Uploading piece... Dec 9 02:01:05.226816121 (82h32m5.097899166s) [noise=true , address=89.149.201.147:4012 , piece_id=7UFT32WUAW2JUETM2Y4SFGMSPTNLVVH5FKPQJSK6S3DR6E64ITKQ, seg_pos={0 0}, upload=72112]: Uploading piece... Dec 9 02:01:05.227549758 (82h32m5.098632813s) [noise=true , address=185.167.204.206:4067 , piece_id=PK6GGQNT4DQ2ZMF2IEEO5X32IMA26ULKDN7HAAG4CWOC2UAUKMMA, seg_pos={0 0}, upload=72112]: Uploading piece... Dec 9 02:01:05.228376403 (82h32m5.099459448s) [noise=true , address=81.171.14.70:4046 , piece_id=RR4RRWZ6YO3EK62O7TE2QUC3ZJ762VGUTVXVVHPGMQRPWEFTB7VA, seg_pos={0 0}, upload=72112]: Uploading piece... Dec 9 02:01:05.228990242 (82h32m5.100073297s) [noise=true , address=185.167.204.205:4013 , piece_id=7TBB5J7PYW23YFBOESEZ5Q6YOD3FKALVJMUQRMC5JSQ5JYKGDQSA, seg_pos={0 0}, upload=72112]: Uploading piece... Dec 9 02:01:06.054905806 (82h32m5.925988851s) [upload=72112]: Next returned nil segment Dec 9 02:01:06.054930403 (82h32m5.926013448s) [upload=72112]: Waiting for error group managing segments... Dec 9 02:01:06.441184862 (82h32m6.312267897s) [noise=true , address=185.167.204.205:4004 , piece_id=6CC2A4V4TJPRJBN4IMJ7VGDVFYTJIVY6J3ZIPZWYL5TIBLGT744A, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:06.444455871 (82h32m6.315538916s) [noise=true , address=185.167.204.205:4005 , piece_id=QUO2YNK2MJJ63CKWADK5K64GYKHDOSBPFHKYKQSKH2TJQUZTNEVA, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:06.447822552 (82h32m6.318905597s) [noise=true , address=185.167.204.206:4066 , piece_id=SSGUSCBUP2YNFN7OMUZQIXJJMSHV6GDHQYMN475LJFQQ7SV2FV2A, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:06.451954059 (82h32m6.323037104s) [noise=true , address=185.167.204.205:4008 , piece_id=54IT2P63UNQKCHG4HB7CX7UWJFWDLJXXPOGPNAEN36ZAC4UQ6VQQ, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:06.454212810 (82h32m6.325295856s) [noise=true , address=185.167.204.205:4011 , piece_id=6JR2JQYKTRLCSBJDUWWSJ6CRYGWKWN3AW3KJCPXMYLYPUN6W2DRQ, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:06.454331416 (82h32m6.325414461s) [noise=true , address=185.167.204.206:4052 , piece_id=WY7PGT7EC5IUBTSHFEAMHXONCIKPFPRF3Y4YSL2IJNY2MOO6U7HA, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:06.455138002 (82h32m6.326221047s) [noise=true , address=185.167.204.206:4050 , piece_id=YKLEM6U44VR3GM45KKKJF7JYOFA4YQE6P5UAZZEMXIA2LBZW5UKA, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:06.459504045 (82h32m6.33058709s) [noise=true , address=185.167.204.205:4013 , piece_id=7TBB5J7PYW23YFBOESEZ5Q6YOD3FKALVJMUQRMC5JSQ5JYKGDQSA, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:06.465474574 (82h32m6.336557629s) [noise=true , address=185.167.204.206:4067 , piece_id=PK6GGQNT4DQ2ZMF2IEEO5X32IMA26ULKDN7HAAG4CWOC2UAUKMMA, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.060075359 (82h32m6.931158404s) [noise=true , address=81.171.14.70:4034 , piece_id=DNHPDP5YCQLS5AN6OHWRWVOGOIGE53YB7CJVUHNPJK35AEM455GQ, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.111399558 (82h32m6.982482603s) [noise=true , address=81.171.14.70:4026 , piece_id=CRRJW4OCPQUSWT6BERGQJPYYQ4DHSZIBMIBCD22QWCGGHUISCFXQ, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.133611052 (82h32m7.004694097s) [noise=true , address=81.171.14.70:4032 , piece_id=APFBMO7KRDDWH6WT3H4ZLCXBAAMZTZQYQBN3XX6JZDGOBAMSGVCA, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.149778297 (82h32m7.020861342s) [noise=true , address=81.171.14.70:4030 , piece_id=VOYOWF6Y5DZOJ6DLW6QFYUI6DF5YA7FI4VI57LHMNCZ2PZLRY4BQ, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.243693687 (82h32m7.114776732s) [noise=true , address=81.171.14.70:4040 , piece_id=RYDZ6DCJ3Z5YKJPIFMLR7EHPH55S7HABKAJCKGRGHF2TCRKEYHXQ, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.624315751 (82h32m7.495398796s) [noise=true , address=81.171.14.70:4046 , piece_id=RR4RRWZ6YO3EK62O7TE2QUC3ZJ762VGUTVXVVHPGMQRPWEFTB7VA, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.682160867 (82h32m7.553243912s) [noise=true , address=89.149.201.147:4011 , piece_id=I77RBTAA7KPTBE27RPPPCNYU65WBBUORVCNYAJOV5OHOBGUFEJKA, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.704962114 (82h32m7.576045159s) [noise=true , address=89.149.201.147:4003 , piece_id=KQDYYB3S7PMHBAZTSQDATYYBAUYHMHEQDI6PW4GM4HKXBK4XHIVQ, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.723657624 (82h32m7.594740669s) [noise=true , address=89.149.201.147:4008 , piece_id=JCJNV47ZC7LJPXSERGUXP2QRLHVPOAQNOPI25RQZTA52IYKRZK4Q, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.729409366 (82h32m7.600492411s) [noise=true , address=89.149.201.147:4002 , piece_id=YFUKN6FRXESUYMS6ZV53MCQTZV5UXYHFF72WS2XJWGF3K3I72GTQ, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.761878943 (82h32m7.632961988s) [noise=true , address=89.149.201.147:4004 , piece_id=NGEHOFVBOCBBCYIVGSYLMKTFQWTDJ2EOGKYEW4X2OIWYH5TVGX2Q, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.825863449 (82h32m7.696946494s) [noise=true , address=89.149.201.147:4005 , piece_id=K6WGQSQZ32ZSAD6KN5IWB66MJ3YP76IKULPEHTFSODYFBSESSU4A, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.866343990 (82h32m7.737427035s) [noise=true , address=89.149.201.147:4015 , piece_id=XBCH6ONGHITOZ2Z5GDPPKDV45TILCDFTOZ44OKMSZANCGVBGQ7ZQ, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:08.228780506 (82h32m8.099863541s) [noise=true , address=89.149.201.147:4017 , piece_id=5W66LIGVHUXIA4AVNKUNCOAJAP5TCYAKVFKCL6NDF2OVXNI62UMQ, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:08.230729618 (82h32m8.101812653s) [noise=true , address=89.149.201.147:4019 , piece_id=JTQ44O6RFCMIPK7LAEWXO2NTXAZAJK7XR3XAJGVQXLOLM7SFSBCA, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:08.257428165 (82h32m8.12851121s) [noise=true , address=89.149.201.147:4020 , piece_id=T5RN3NMY35NXWM55YN3F7TWPPP7KXEER5R63W4KZV6Y2UNORJX4A, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:08.267970901 (82h32m8.139053946s) [noise=true , address=89.149.201.147:4006 , piece_id=M3VDXP334RID2FJK7ZYNUHJBRMS57EXIMVMLER25MVJWM4EHJ2WA, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:08.267990939 (82h32m8.139073974s) [seg_pos={0 0}, upload=72112]: Segment reached optimal threshold of 26 pieces. Dec 9 02:01:08.268172084 (82h32m8.139255129s) [noise=true , address=89.149.201.147:4012 , piece_id=7UFT32WUAW2JUETM2Y4SFGMSPTNLVVH5FKPQJSK6S3DR6E64ITKQ, seg_pos={0 0}, upload=72112]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hJ5jT6S2d3DJe9DoVDjHMkpwSTpeyTMyNLaK8Pk8SWcUVRKnv): protocol: expected piece hash; context canceled; EOF Dec 9 02:01:08.268182163 (82h32m8.139265198s) [noise=true , address=89.149.201.147:4012 , piece_id=7UFT32WUAW2JUETM2Y4SFGMSPTNLVVH5FKPQJSK6S3DR6E64ITKQ, seg_pos={0 0}, upload=72112]: Long tail cancellation due to optimal threshold Dec 9 02:01:08.268290639 (82h32m8.139373684s) [noise=true , address=89.149.201.147:4014 , piece_id=PFXVAHNZ2Q6YC2MGNSUZMJIL5Q2O2HPDSONU3FD3E472H2G7AG5A, seg_pos={0 0}, upload=72112]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wgtphHKaTejC1qrJ98sDysBwtj2Zeqw2c5UFk1HMqfY8Buvr): protocol: expected piece hash; context canceled; EOF Dec 9 02:01:08.268299597 (82h32m8.139382642s) [noise=true , address=89.149.201.147:4014 , piece_id=PFXVAHNZ2Q6YC2MGNSUZMJIL5Q2O2HPDSONU3FD3E472H2G7AG5A, seg_pos={0 0}, upload=72112]: Long tail cancellation due to optimal threshold Dec 9 02:01:08.268340294 (82h32m8.139423329s) [seg_pos={0 0}, upload=72112]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 9 02:01:08.268435756 (82h32m8.139518802s) [upload=72112]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 02:01:08.268441557 (82h32m8.139524592s) [upload=72112]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 02:12:41.424380970 (82h43m41.295464025s) [upload=72112]: Done uploading segments... Dec 9 02:12:41.424504095 (82h43m41.29558713s) [upload=72112]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.258167649s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [............X.......XX........X.............................] duration: 3.568977121s trace: https://jaeger.dev.storj.io/trace/dbaa73a4151be2e FAIL (2025-12-09 02:28:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.569004712s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 2 out of 60 is failed (threshold=3) [........................X......X............................] duration: 469.491973ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/4e3d2b60e3f4041b FAIL (2025-12-09 02:28:04 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 469.527729ms core satellite eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [...................X.X.....X................................] duration: 5.859566736s trace: https://jaeger.dev.storj.io/trace/5280cd334ed3ed6b FAIL (2025-12-09 02:28:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.859592145s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 5.815106513s trace: https://jaeger.dev.storj.io/trace/1ddfa89c12e62214 FAIL (2025-12-09 02:27:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: manager closed: closed: read tcp 188.166.225.170:38662->35.212.10.183:7777: read: connection reset by peer ======LOGS====== Dec 9 02:24:03.838353395 (82h55m3.70943644s) [upload=72429]: Starting upload Dec 9 02:24:03.838420683 (82h55m3.709503729s) [upload=72429]: Done starting upload Dec 9 02:24:03.838549719 (82h55m3.709632755s) [upload=72429]: Uploading segments... Dec 9 02:24:03.838688734 (82h55m3.709771779s) [upload=72429]: Got next segment. Inline: false Dec 9 02:24:03.838734311 (82h55m3.709817356s) [upload=72429]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 02:24:03.838741164 (82h55m3.709824209s) [upload=72429]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 02:24:04.086906602 (82h55m3.957989647s) [seg_pos={0 0}, upload=72429]: Begin upload segment... Dec 9 02:24:04.086974701 (82h55m3.958057746s) [seg_pos={0 0}, upload=72429]: Done begin upload segment. Dec 9 02:24:04.087029476 (82h55m3.958112511s) [noise=true , address=145.79.227.24:4581 , piece_id=ETKQGTKTMEKLK42KNN72MBQF62TFTOQO357XKF4WS6NTVBSOANDA, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.087057810 (82h55m3.958140855s) [noise=true , address=145.79.227.16:4911 , piece_id=Q62EUAKMIIB6QZ3KTNQH3PJAZIX7YNRYXJPTATNCZ4W6RWELTLXQ, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.087925792 (82h55m3.959008837s) [noise=true , address=145.79.227.42:4122 , piece_id=ZWY35JIEVFUL3LADBSS4EKYMWMFBTHJWMD6YQ7M4OGKXYAU4LNWQ, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.088596299 (82h55m3.959679344s) [noise=true , address=145.79.227.23:4545 , piece_id=NPORRF6TACVJIVBM3IABBQF7NBYSDRGJTPGDREFMOWNNT3B325UQ, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.089090329 (82h55m3.960173374s) [noise=true , address=145.79.227.43:4158 , piece_id=DS3UAJZAYVR4GXPHKPMJ2RO7765FUPASW7T7GQH6ATVCM4FAPYLQ, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.089582366 (82h55m3.960665411s) [noise=true , address=145.79.227.11:4732 , piece_id=CA5XB6ODOYLW3RQU5F7BMUDDSAGEJGIKOP7IZBHPNUTUXYDJ4JWQ, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.090135129 (82h55m3.961218174s) [noise=true , address=145.79.227.34:4906 , piece_id=GTGD6FTBFJ6VMETYMRJPTO7P77B4TBQT4VSRAEMUCJ6TTW4VBHDA, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.090616965 (82h55m3.96170001s) [noise=true , address=145.79.227.24:4583 , piece_id=MFGUNU4E6DBCQ5TUNPSWBXRSEQ7QGTNWTWCUTZK5NAVLMIBB2KKQ, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.091103331 (82h55m3.962186376s) [noise=true , address=145.79.227.25:4619 , piece_id=RALQO5F6I5M4Q56PLG47D57UH4WHRQWB47N45NASEPILQGC26VQQ, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.091901141 (82h55m3.962984186s) [noise=true , address=145.79.227.38:4978 , piece_id=XBHD6MMZECN7T7QTC3HVN7OS5XY7CJJJEFERUVK2ZIXTVWSVNLXA, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.094168697 (82h55m3.965251742s) [noise=true , address=77.90.31.16:4734 , piece_id=WSIBQS2XW3PENG55ORDRTUYXPZ6HCSTAX7DP5JFDVVCLCX6JGLTA, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.094774330 (82h55m3.965857375s) [noise=true , address=145.79.227.40:4049 , piece_id=VH3PRUHK4OR4MCGVX3VHMXKWTVE3KQLKH5RJHI4MLJMITRHILR2A, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.095474052 (82h55m3.966557097s) [noise=true , address=145.79.227.31:4797 , piece_id=XDQGJD7JELZ3D73GMYRVHKWEFIZGYF6IWDBZ54NIJTGA3G7WWMFQ, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.096054146 (82h55m3.967137191s) [noise=true , address=145.79.227.21:4474 , piece_id=TTDRH27TBF34WUNSWZL3XL2RR6SJGEHP7J4YCHAZB4SH3ZWH4PVA, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.096661643 (82h55m3.967744688s) [noise=true , address=77.90.31.10:4520 , piece_id=GW2ECPZB4XCEXAWWCBVUM2VEUXXMSBPJUZSKTULZYCHK4DZL63XA, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.097203715 (82h55m3.96828676s) [noise=true , address=145.79.227.22:4511 , piece_id=QVG66WWAN3TCESLCDW2ZG7SQADAJPUIILVYGFGTCKRLGVXJWSQ7Q, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:05.228786599 (82h55m5.099869634s) [noise=true , address=77.90.31.16:4734 , piece_id=WSIBQS2XW3PENG55ORDRTUYXPZ6HCSTAX7DP5JFDVVCLCX6JGLTA, seg_pos={0 0}, upload=72429]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 9 02:24:05.771060198 (82h55m5.642143243s) [upload=72429]: Next returned nil segment Dec 9 02:24:05.771079454 (82h55m5.642162529s) [upload=72429]: Waiting for error group managing segments... Dec 9 02:24:06.084934630 (82h55m5.956017675s) [noise=true , address=145.79.227.24:4583 , piece_id=MFGUNU4E6DBCQ5TUNPSWBXRSEQ7QGTNWTWCUTZK5NAVLMIBB2KKQ, seg_pos={0 0}, upload=72429]: Done uploading piece. err: <nil> Dec 9 02:24:06.086488888 (82h55m5.957571934s) [noise=true , address=145.79.227.42:4122 , piece_id=ZWY35JIEVFUL3LADBSS4EKYMWMFBTHJWMD6YQ7M4OGKXYAU4LNWQ, seg_pos={0 0}, upload=72429]: Done uploading piece. err: <nil> Dec 9 02:24:06.087432816 (82h55m5.958515861s) [noise=true , address=145.79.227.25:4619 , piece_id=RALQO5F6I5M4Q56PLG47D57UH4WHRQWB47N45NASEPILQGC26VQQ, seg_pos={0 0}, upload=72429]: Done uploading piece. err: <nil> Dec 9 02:24:06.088224853 (82h55m5.959307898s) [noise=true , address=145.79.227.24:4581 , piece_id=ETKQGTKTMEKLK42KNN72MBQF62TFTOQO357XKF4WS6NTVBSOANDA, seg_pos={0 0}, upload=72429]: Done uploading piece. err: <nil> Dec 9 02:24:06.090286018 (82h55m5.961369053s) [noise=true , address=145.79.227.31:4797 , piece_id=XDQGJD7JELZ3D73GMYRVHKWEFIZGYF6IWDBZ54NIJTGA3G7WWMFQ, seg_pos={0 0}, upload=72429]: Done uploading piece. err: <nil> Dec 9 02:24:06.094149082 (82h55m5.965232127s) [noise=true , address=145.79.227.34:4906 , piece_id=GTGD6FTBFJ6VMETYMRJPTO7P77B4TBQT4VSRAEMUCJ6TTW4VBHDA, seg_pos={0 0}, upload=72429]: Done uploading piece. err: <nil> Dec 9 02:24:06.096430375 (82h55m5.96751342s) [noise=true , address=145.79.227.21:4474 , piece_id=TTDRH27TBF34WUNSWZL3XL2RR6SJGEHP7J4YCHAZB4SH3ZWH4PVA, seg_pos={0 0}, upload=72429]: Done uploading piece. err: <nil> Dec 9 02:24:06.096707993 (82h55m5.967791028s) [noise=true , address=145.79.227.11:4732 , piece_id=CA5XB6ODOYLW3RQU5F7BMUDDSAGEJGIKOP7IZBHPNUTUXYDJ4JWQ, seg_pos={0 0}, upload=72429]: Done uploading piece. err: <nil> Dec 9 02:24:06.103696047 (82h55m5.974779092s) [noise=true , address=145.79.227.16:4911 , piece_id=Q62EUAKMIIB6QZ3KTNQH3PJAZIX7YNRYXJPTATNCZ4W6RWELTLXQ, seg_pos={0 0}, upload=72429]: Done uploading piece. err: <nil> Dec 9 02:24:06.105006020 (82h55m5.976089065s) [noise=true , address=145.79.227.23:4545 , piece_id=NPORRF6TACVJIVBM3IABBQF7NBYSDRGJTPGDREFMOWNNT3B325UQ, seg_pos={0 0}, upload=72429]: Done uploading piece. err: <nil> Dec 9 02:24:06.105250326 (82h55m5.976333381s) [noise=true , address=145.79.227.38:4978 , piece_id=XBHD6MMZECN7T7QTC3HVN7OS5XY7CJJJEFERUVK2ZIXTVWSVNLXA, seg_pos={0 0}, upload=72429]: Done uploading piece. err: <nil> Dec 9 02:24:06.105501193 (82h55m5.976584238s) [noise=true , address=145.79.227.22:4511 , piece_id=QVG66WWAN3TCESLCDW2ZG7SQADAJPUIILVYGFGTCKRLGVXJWSQ7Q, seg_pos={0 0}, upload=72429]: Done uploading piece. err: <nil> Dec 9 02:24:06.108138354 (82h55m5.979221399s) [noise=true , address=145.79.227.40:4049 , piece_id=VH3PRUHK4OR4MCGVX3VHMXKWTVE3KQLKH5RJHI4MLJMITRHILR2A, seg_pos={0 0}, upload=72429]: Done uploading piece. err: <nil> Dec 9 02:24:06.108158543 (82h55m5.979241588s) [seg_pos={0 0}, upload=72429]: Segment reached optimal threshold of 13 pieces. Dec 9 02:24:06.108688862 (82h55m5.979771907s) [noise=true , address=77.90.31.10:4520 , piece_id=GW2ECPZB4XCEXAWWCBVUM2VEUXXMSBPJUZSKTULZYCHK4DZL63XA, seg_pos={0 0}, upload=72429]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Dec 9 02:24:06.108700394 (82h55m5.979783449s) [noise=true , address=77.90.31.10:4520 , piece_id=GW2ECPZB4XCEXAWWCBVUM2VEUXXMSBPJUZSKTULZYCHK4DZL63XA, seg_pos={0 0}, upload=72429]: Long tail cancellation due to optimal threshold Dec 9 02:24:06.108930402 (82h55m5.980013447s) [noise=true , address=145.79.227.43:4158 , piece_id=DS3UAJZAYVR4GXPHKPMJ2RO7765FUPASW7T7GQH6ATVCM4FAPYLQ, seg_pos={0 0}, upload=72429]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF Dec 9 02:24:06.108941323 (82h55m5.980024368s) [noise=true , address=145.79.227.43:4158 , piece_id=DS3UAJZAYVR4GXPHKPMJ2RO7765FUPASW7T7GQH6ATVCM4FAPYLQ, seg_pos={0 0}, upload=72429]: Long tail cancellation due to optimal threshold Dec 9 02:24:06.108964487 (82h55m5.980047532s) [seg_pos={0 0}, upload=72429]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 02:24:06.109072202 (82h55m5.980155257s) [upload=72429]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 02:24:06.109080047 (82h55m5.980163092s) [upload=72429]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 02:24:24.569429416 (82h55m24.440512451s) [upload=72429]: Done uploading segments... Dec 9 02:24:24.569477928 (82h55m24.440560963s) [upload=72429]: Upload finished. err: metaclient: manager closed: closed: read tcp 188.166.225.170:38662->35.212.10.183:7777: read: connection reset by peer Duration: 5.815134678s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 5.77595914s trace: https://jaeger.dev.storj.io/trace/40019ff87d088ff7 FAIL (2025-12-09 02:27:29 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.775993026s core uplink us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 5.778186951s trace: https://jaeger.dev.storj.io/trace/6f1ff10066b1d8ea FAIL (2025-12-09 02:27:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== rue , address=185.167.204.206:4052 , piece_id=WY7PGT7EC5IUBTSHFEAMHXONCIKPFPRF3Y4YSL2IJNY2MOO6U7HA, seg_pos={0 0}, upload=72112]: Uploading piece... Dec 9 02:01:05.223188835 (82h32m5.09427188s) [noise=true , address=89.149.201.147:4011 , piece_id=I77RBTAA7KPTBE27RPPPCNYU65WBBUORVCNYAJOV5OHOBGUFEJKA, seg_pos={0 0}, upload=72112]: Uploading piece... Dec 9 02:01:05.223808365 (82h32m5.09489141s) [noise=true , address=185.167.204.206:4050 , piece_id=YKLEM6U44VR3GM45KKKJF7JYOFA4YQE6P5UAZZEMXIA2LBZW5UKA, seg_pos={0 0}, upload=72112]: Uploading piece... Dec 9 02:01:05.224324407 (82h32m5.095407452s) [noise=true , address=185.167.204.205:4011 , piece_id=6JR2JQYKTRLCSBJDUWWSJ6CRYGWKWN3AW3KJCPXMYLYPUN6W2DRQ, seg_pos={0 0}, upload=72112]: Uploading piece... Dec 9 02:01:05.224877841 (82h32m5.095960886s) [noise=true , address=81.171.14.70:4032 , piece_id=APFBMO7KRDDWH6WT3H4ZLCXBAAMZTZQYQBN3XX6JZDGOBAMSGVCA, seg_pos={0 0}, upload=72112]: Uploading piece... Dec 9 02:01:05.225476030 (82h32m5.096559075s) [noise=true , address=89.149.201.147:4014 , piece_id=PFXVAHNZ2Q6YC2MGNSUZMJIL5Q2O2HPDSONU3FD3E472H2G7AG5A, seg_pos={0 0}, upload=72112]: Uploading piece... Dec 9 02:01:05.226119545 (82h32m5.09720258s) [noise=true , address=89.149.201.147:4015 , piece_id=XBCH6ONGHITOZ2Z5GDPPKDV45TILCDFTOZ44OKMSZANCGVBGQ7ZQ, seg_pos={0 0}, upload=72112]: Uploading piece... Dec 9 02:01:05.226816121 (82h32m5.097899166s) [noise=true , address=89.149.201.147:4012 , piece_id=7UFT32WUAW2JUETM2Y4SFGMSPTNLVVH5FKPQJSK6S3DR6E64ITKQ, seg_pos={0 0}, upload=72112]: Uploading piece... Dec 9 02:01:05.227549758 (82h32m5.098632813s) [noise=true , address=185.167.204.206:4067 , piece_id=PK6GGQNT4DQ2ZMF2IEEO5X32IMA26ULKDN7HAAG4CWOC2UAUKMMA, seg_pos={0 0}, upload=72112]: Uploading piece... Dec 9 02:01:05.228376403 (82h32m5.099459448s) [noise=true , address=81.171.14.70:4046 , piece_id=RR4RRWZ6YO3EK62O7TE2QUC3ZJ762VGUTVXVVHPGMQRPWEFTB7VA, seg_pos={0 0}, upload=72112]: Uploading piece... Dec 9 02:01:05.228990242 (82h32m5.100073297s) [noise=true , address=185.167.204.205:4013 , piece_id=7TBB5J7PYW23YFBOESEZ5Q6YOD3FKALVJMUQRMC5JSQ5JYKGDQSA, seg_pos={0 0}, upload=72112]: Uploading piece... Dec 9 02:01:06.054905806 (82h32m5.925988851s) [upload=72112]: Next returned nil segment Dec 9 02:01:06.054930403 (82h32m5.926013448s) [upload=72112]: Waiting for error group managing segments... Dec 9 02:01:06.441184862 (82h32m6.312267897s) [noise=true , address=185.167.204.205:4004 , piece_id=6CC2A4V4TJPRJBN4IMJ7VGDVFYTJIVY6J3ZIPZWYL5TIBLGT744A, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:06.444455871 (82h32m6.315538916s) [noise=true , address=185.167.204.205:4005 , piece_id=QUO2YNK2MJJ63CKWADK5K64GYKHDOSBPFHKYKQSKH2TJQUZTNEVA, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:06.447822552 (82h32m6.318905597s) [noise=true , address=185.167.204.206:4066 , piece_id=SSGUSCBUP2YNFN7OMUZQIXJJMSHV6GDHQYMN475LJFQQ7SV2FV2A, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:06.451954059 (82h32m6.323037104s) [noise=true , address=185.167.204.205:4008 , piece_id=54IT2P63UNQKCHG4HB7CX7UWJFWDLJXXPOGPNAEN36ZAC4UQ6VQQ, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:06.454212810 (82h32m6.325295856s) [noise=true , address=185.167.204.205:4011 , piece_id=6JR2JQYKTRLCSBJDUWWSJ6CRYGWKWN3AW3KJCPXMYLYPUN6W2DRQ, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:06.454331416 (82h32m6.325414461s) [noise=true , address=185.167.204.206:4052 , piece_id=WY7PGT7EC5IUBTSHFEAMHXONCIKPFPRF3Y4YSL2IJNY2MOO6U7HA, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:06.455138002 (82h32m6.326221047s) [noise=true , address=185.167.204.206:4050 , piece_id=YKLEM6U44VR3GM45KKKJF7JYOFA4YQE6P5UAZZEMXIA2LBZW5UKA, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:06.459504045 (82h32m6.33058709s) [noise=true , address=185.167.204.205:4013 , piece_id=7TBB5J7PYW23YFBOESEZ5Q6YOD3FKALVJMUQRMC5JSQ5JYKGDQSA, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:06.465474574 (82h32m6.336557629s) [noise=true , address=185.167.204.206:4067 , piece_id=PK6GGQNT4DQ2ZMF2IEEO5X32IMA26ULKDN7HAAG4CWOC2UAUKMMA, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.060075359 (82h32m6.931158404s) [noise=true , address=81.171.14.70:4034 , piece_id=DNHPDP5YCQLS5AN6OHWRWVOGOIGE53YB7CJVUHNPJK35AEM455GQ, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.111399558 (82h32m6.982482603s) [noise=true , address=81.171.14.70:4026 , piece_id=CRRJW4OCPQUSWT6BERGQJPYYQ4DHSZIBMIBCD22QWCGGHUISCFXQ, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.133611052 (82h32m7.004694097s) [noise=true , address=81.171.14.70:4032 , piece_id=APFBMO7KRDDWH6WT3H4ZLCXBAAMZTZQYQBN3XX6JZDGOBAMSGVCA, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.149778297 (82h32m7.020861342s) [noise=true , address=81.171.14.70:4030 , piece_id=VOYOWF6Y5DZOJ6DLW6QFYUI6DF5YA7FI4VI57LHMNCZ2PZLRY4BQ, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.243693687 (82h32m7.114776732s) [noise=true , address=81.171.14.70:4040 , piece_id=RYDZ6DCJ3Z5YKJPIFMLR7EHPH55S7HABKAJCKGRGHF2TCRKEYHXQ, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.624315751 (82h32m7.495398796s) [noise=true , address=81.171.14.70:4046 , piece_id=RR4RRWZ6YO3EK62O7TE2QUC3ZJ762VGUTVXVVHPGMQRPWEFTB7VA, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.682160867 (82h32m7.553243912s) [noise=true , address=89.149.201.147:4011 , piece_id=I77RBTAA7KPTBE27RPPPCNYU65WBBUORVCNYAJOV5OHOBGUFEJKA, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.704962114 (82h32m7.576045159s) [noise=true , address=89.149.201.147:4003 , piece_id=KQDYYB3S7PMHBAZTSQDATYYBAUYHMHEQDI6PW4GM4HKXBK4XHIVQ, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.723657624 (82h32m7.594740669s) [noise=true , address=89.149.201.147:4008 , piece_id=JCJNV47ZC7LJPXSERGUXP2QRLHVPOAQNOPI25RQZTA52IYKRZK4Q, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.729409366 (82h32m7.600492411s) [noise=true , address=89.149.201.147:4002 , piece_id=YFUKN6FRXESUYMS6ZV53MCQTZV5UXYHFF72WS2XJWGF3K3I72GTQ, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.761878943 (82h32m7.632961988s) [noise=true , address=89.149.201.147:4004 , piece_id=NGEHOFVBOCBBCYIVGSYLMKTFQWTDJ2EOGKYEW4X2OIWYH5TVGX2Q, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.825863449 (82h32m7.696946494s) [noise=true , address=89.149.201.147:4005 , piece_id=K6WGQSQZ32ZSAD6KN5IWB66MJ3YP76IKULPEHTFSODYFBSESSU4A, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.866343990 (82h32m7.737427035s) [noise=true , address=89.149.201.147:4015 , piece_id=XBCH6ONGHITOZ2Z5GDPPKDV45TILCDFTOZ44OKMSZANCGVBGQ7ZQ, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:08.228780506 (82h32m8.099863541s) [noise=true , address=89.149.201.147:4017 , piece_id=5W66LIGVHUXIA4AVNKUNCOAJAP5TCYAKVFKCL6NDF2OVXNI62UMQ, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:08.230729618 (82h32m8.101812653s) [noise=true , address=89.149.201.147:4019 , piece_id=JTQ44O6RFCMIPK7LAEWXO2NTXAZAJK7XR3XAJGVQXLOLM7SFSBCA, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:08.257428165 (82h32m8.12851121s) [noise=true , address=89.149.201.147:4020 , piece_id=T5RN3NMY35NXWM55YN3F7TWPPP7KXEER5R63W4KZV6Y2UNORJX4A, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:08.267970901 (82h32m8.139053946s) [noise=true , address=89.149.201.147:4006 , piece_id=M3VDXP334RID2FJK7ZYNUHJBRMS57EXIMVMLER25MVJWM4EHJ2WA, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:08.267990939 (82h32m8.139073974s) [seg_pos={0 0}, upload=72112]: Segment reached optimal threshold of 26 pieces. Dec 9 02:01:08.268172084 (82h32m8.139255129s) [noise=true , address=89.149.201.147:4012 , piece_id=7UFT32WUAW2JUETM2Y4SFGMSPTNLVVH5FKPQJSK6S3DR6E64ITKQ, seg_pos={0 0}, upload=72112]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hJ5jT6S2d3DJe9DoVDjHMkpwSTpeyTMyNLaK8Pk8SWcUVRKnv): protocol: expected piece hash; context canceled; EOF Dec 9 02:01:08.268182163 (82h32m8.139265198s) [noise=true , address=89.149.201.147:4012 , piece_id=7UFT32WUAW2JUETM2Y4SFGMSPTNLVVH5FKPQJSK6S3DR6E64ITKQ, seg_pos={0 0}, upload=72112]: Long tail cancellation due to optimal threshold Dec 9 02:01:08.268290639 (82h32m8.139373684s) [noise=true , address=89.149.201.147:4014 , piece_id=PFXVAHNZ2Q6YC2MGNSUZMJIL5Q2O2HPDSONU3FD3E472H2G7AG5A, seg_pos={0 0}, upload=72112]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wgtphHKaTejC1qrJ98sDysBwtj2Zeqw2c5UFk1HMqfY8Buvr): protocol: expected piece hash; context canceled; EOF Dec 9 02:01:08.268299597 (82h32m8.139382642s) [noise=true , address=89.149.201.147:4014 , piece_id=PFXVAHNZ2Q6YC2MGNSUZMJIL5Q2O2HPDSONU3FD3E472H2G7AG5A, seg_pos={0 0}, upload=72112]: Long tail cancellation due to optimal threshold Dec 9 02:01:08.268340294 (82h32m8.139423329s) [seg_pos={0 0}, upload=72112]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 9 02:01:08.268435756 (82h32m8.139518802s) [upload=72112]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 02:01:08.268441557 (82h32m8.139524592s) [upload=72112]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 02:12:41.424380970 (82h43m41.295464025s) [upload=72112]: Done uploading segments... Dec 9 02:12:41.424504095 (82h43m41.29558713s) [upload=72112]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.778216697s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.............X.......XX........X............................] duration: 3.490515905s trace: https://jaeger.dev.storj.io/trace/dbaa73a4151be2e FAIL (2025-12-09 02:27:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.490546936s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [....................X.X.....X...............................] duration: 6.572280793s trace: https://jaeger.dev.storj.io/trace/5280cd334ed3ed6b FAIL (2025-12-09 02:27:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.572305131s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 2 out of 60 is failed (threshold=3) [.........................X......X...........................] duration: 487.859207ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/4e3d2b60e3f4041b FAIL (2025-12-09 02:26:56 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 487.890838ms core satellite eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 5.765063025s trace: https://jaeger.dev.storj.io/trace/1ddfa89c12e62214 FAIL (2025-12-09 02:26:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: manager closed: closed: read tcp 188.166.225.170:38662->35.212.10.183:7777: read: connection reset by peer ======LOGS====== Dec 9 02:24:03.838353395 (82h55m3.70943644s) [upload=72429]: Starting upload Dec 9 02:24:03.838420683 (82h55m3.709503729s) [upload=72429]: Done starting upload Dec 9 02:24:03.838549719 (82h55m3.709632755s) [upload=72429]: Uploading segments... Dec 9 02:24:03.838688734 (82h55m3.709771779s) [upload=72429]: Got next segment. Inline: false Dec 9 02:24:03.838734311 (82h55m3.709817356s) [upload=72429]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 02:24:03.838741164 (82h55m3.709824209s) [upload=72429]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 02:24:04.086906602 (82h55m3.957989647s) [seg_pos={0 0}, upload=72429]: Begin upload segment... Dec 9 02:24:04.086974701 (82h55m3.958057746s) [seg_pos={0 0}, upload=72429]: Done begin upload segment. Dec 9 02:24:04.087029476 (82h55m3.958112511s) [noise=true , address=145.79.227.24:4581 , piece_id=ETKQGTKTMEKLK42KNN72MBQF62TFTOQO357XKF4WS6NTVBSOANDA, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.087057810 (82h55m3.958140855s) [noise=true , address=145.79.227.16:4911 , piece_id=Q62EUAKMIIB6QZ3KTNQH3PJAZIX7YNRYXJPTATNCZ4W6RWELTLXQ, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.087925792 (82h55m3.959008837s) [noise=true , address=145.79.227.42:4122 , piece_id=ZWY35JIEVFUL3LADBSS4EKYMWMFBTHJWMD6YQ7M4OGKXYAU4LNWQ, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.088596299 (82h55m3.959679344s) [noise=true , address=145.79.227.23:4545 , piece_id=NPORRF6TACVJIVBM3IABBQF7NBYSDRGJTPGDREFMOWNNT3B325UQ, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.089090329 (82h55m3.960173374s) [noise=true , address=145.79.227.43:4158 , piece_id=DS3UAJZAYVR4GXPHKPMJ2RO7765FUPASW7T7GQH6ATVCM4FAPYLQ, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.089582366 (82h55m3.960665411s) [noise=true , address=145.79.227.11:4732 , piece_id=CA5XB6ODOYLW3RQU5F7BMUDDSAGEJGIKOP7IZBHPNUTUXYDJ4JWQ, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.090135129 (82h55m3.961218174s) [noise=true , address=145.79.227.34:4906 , piece_id=GTGD6FTBFJ6VMETYMRJPTO7P77B4TBQT4VSRAEMUCJ6TTW4VBHDA, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.090616965 (82h55m3.96170001s) [noise=true , address=145.79.227.24:4583 , piece_id=MFGUNU4E6DBCQ5TUNPSWBXRSEQ7QGTNWTWCUTZK5NAVLMIBB2KKQ, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.091103331 (82h55m3.962186376s) [noise=true , address=145.79.227.25:4619 , piece_id=RALQO5F6I5M4Q56PLG47D57UH4WHRQWB47N45NASEPILQGC26VQQ, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.091901141 (82h55m3.962984186s) [noise=true , address=145.79.227.38:4978 , piece_id=XBHD6MMZECN7T7QTC3HVN7OS5XY7CJJJEFERUVK2ZIXTVWSVNLXA, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.094168697 (82h55m3.965251742s) [noise=true , address=77.90.31.16:4734 , piece_id=WSIBQS2XW3PENG55ORDRTUYXPZ6HCSTAX7DP5JFDVVCLCX6JGLTA, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.094774330 (82h55m3.965857375s) [noise=true , address=145.79.227.40:4049 , piece_id=VH3PRUHK4OR4MCGVX3VHMXKWTVE3KQLKH5RJHI4MLJMITRHILR2A, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.095474052 (82h55m3.966557097s) [noise=true , address=145.79.227.31:4797 , piece_id=XDQGJD7JELZ3D73GMYRVHKWEFIZGYF6IWDBZ54NIJTGA3G7WWMFQ, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.096054146 (82h55m3.967137191s) [noise=true , address=145.79.227.21:4474 , piece_id=TTDRH27TBF34WUNSWZL3XL2RR6SJGEHP7J4YCHAZB4SH3ZWH4PVA, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.096661643 (82h55m3.967744688s) [noise=true , address=77.90.31.10:4520 , piece_id=GW2ECPZB4XCEXAWWCBVUM2VEUXXMSBPJUZSKTULZYCHK4DZL63XA, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.097203715 (82h55m3.96828676s) [noise=true , address=145.79.227.22:4511 , piece_id=QVG66WWAN3TCESLCDW2ZG7SQADAJPUIILVYGFGTCKRLGVXJWSQ7Q, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:05.228786599 (82h55m5.099869634s) [noise=true , address=77.90.31.16:4734 , piece_id=WSIBQS2XW3PENG55ORDRTUYXPZ6HCSTAX7DP5JFDVVCLCX6JGLTA, seg_pos={0 0}, upload=72429]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 9 02:24:05.771060198 (82h55m5.642143243s) [upload=72429]: Next returned nil segment Dec 9 02:24:05.771079454 (82h55m5.642162529s) [upload=72429]: Waiting for error group managing segments... Dec 9 02:24:06.084934630 (82h55m5.956017675s) [noise=true , address=145.79.227.24:4583 , piece_id=MFGUNU4E6DBCQ5TUNPSWBXRSEQ7QGTNWTWCUTZK5NAVLMIBB2KKQ, seg_pos={0 0}, upload=72429]: Done uploading piece. err: <nil> Dec 9 02:24:06.086488888 (82h55m5.957571934s) [noise=true , address=145.79.227.42:4122 , piece_id=ZWY35JIEVFUL3LADBSS4EKYMWMFBTHJWMD6YQ7M4OGKXYAU4LNWQ, seg_pos={0 0}, upload=72429]: Done uploading piece. err: <nil> Dec 9 02:24:06.087432816 (82h55m5.958515861s) [noise=true , address=145.79.227.25:4619 , piece_id=RALQO5F6I5M4Q56PLG47D57UH4WHRQWB47N45NASEPILQGC26VQQ, seg_pos={0 0}, upload=72429]: Done uploading piece. err: <nil> Dec 9 02:24:06.088224853 (82h55m5.959307898s) [noise=true , address=145.79.227.24:4581 , piece_id=ETKQGTKTMEKLK42KNN72MBQF62TFTOQO357XKF4WS6NTVBSOANDA, seg_pos={0 0}, upload=72429]: Done uploading piece. err: <nil> Dec 9 02:24:06.090286018 (82h55m5.961369053s) [noise=true , address=145.79.227.31:4797 , piece_id=XDQGJD7JELZ3D73GMYRVHKWEFIZGYF6IWDBZ54NIJTGA3G7WWMFQ, seg_pos={0 0}, upload=72429]: Done uploading piece. err: <nil> Dec 9 02:24:06.094149082 (82h55m5.965232127s) [noise=true , address=145.79.227.34:4906 , piece_id=GTGD6FTBFJ6VMETYMRJPTO7P77B4TBQT4VSRAEMUCJ6TTW4VBHDA, seg_pos={0 0}, upload=72429]: Done uploading piece. err: <nil> Dec 9 02:24:06.096430375 (82h55m5.96751342s) [noise=true , address=145.79.227.21:4474 , piece_id=TTDRH27TBF34WUNSWZL3XL2RR6SJGEHP7J4YCHAZB4SH3ZWH4PVA, seg_pos={0 0}, upload=72429]: Done uploading piece. err: <nil> Dec 9 02:24:06.096707993 (82h55m5.967791028s) [noise=true , address=145.79.227.11:4732 , piece_id=CA5XB6ODOYLW3RQU5F7BMUDDSAGEJGIKOP7IZBHPNUTUXYDJ4JWQ, seg_pos={0 0}, upload=72429]: Done uploading piece. err: <nil> Dec 9 02:24:06.103696047 (82h55m5.974779092s) [noise=true , address=145.79.227.16:4911 , piece_id=Q62EUAKMIIB6QZ3KTNQH3PJAZIX7YNRYXJPTATNCZ4W6RWELTLXQ, seg_pos={0 0}, upload=72429]: Done uploading piece. err: <nil> Dec 9 02:24:06.105006020 (82h55m5.976089065s) [noise=true , address=145.79.227.23:4545 , piece_id=NPORRF6TACVJIVBM3IABBQF7NBYSDRGJTPGDREFMOWNNT3B325UQ, seg_pos={0 0}, upload=72429]: Done uploading piece. err: <nil> Dec 9 02:24:06.105250326 (82h55m5.976333381s) [noise=true , address=145.79.227.38:4978 , piece_id=XBHD6MMZECN7T7QTC3HVN7OS5XY7CJJJEFERUVK2ZIXTVWSVNLXA, seg_pos={0 0}, upload=72429]: Done uploading piece. err: <nil> Dec 9 02:24:06.105501193 (82h55m5.976584238s) [noise=true , address=145.79.227.22:4511 , piece_id=QVG66WWAN3TCESLCDW2ZG7SQADAJPUIILVYGFGTCKRLGVXJWSQ7Q, seg_pos={0 0}, upload=72429]: Done uploading piece. err: <nil> Dec 9 02:24:06.108138354 (82h55m5.979221399s) [noise=true , address=145.79.227.40:4049 , piece_id=VH3PRUHK4OR4MCGVX3VHMXKWTVE3KQLKH5RJHI4MLJMITRHILR2A, seg_pos={0 0}, upload=72429]: Done uploading piece. err: <nil> Dec 9 02:24:06.108158543 (82h55m5.979241588s) [seg_pos={0 0}, upload=72429]: Segment reached optimal threshold of 13 pieces. Dec 9 02:24:06.108688862 (82h55m5.979771907s) [noise=true , address=77.90.31.10:4520 , piece_id=GW2ECPZB4XCEXAWWCBVUM2VEUXXMSBPJUZSKTULZYCHK4DZL63XA, seg_pos={0 0}, upload=72429]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Dec 9 02:24:06.108700394 (82h55m5.979783449s) [noise=true , address=77.90.31.10:4520 , piece_id=GW2ECPZB4XCEXAWWCBVUM2VEUXXMSBPJUZSKTULZYCHK4DZL63XA, seg_pos={0 0}, upload=72429]: Long tail cancellation due to optimal threshold Dec 9 02:24:06.108930402 (82h55m5.980013447s) [noise=true , address=145.79.227.43:4158 , piece_id=DS3UAJZAYVR4GXPHKPMJ2RO7765FUPASW7T7GQH6ATVCM4FAPYLQ, seg_pos={0 0}, upload=72429]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF Dec 9 02:24:06.108941323 (82h55m5.980024368s) [noise=true , address=145.79.227.43:4158 , piece_id=DS3UAJZAYVR4GXPHKPMJ2RO7765FUPASW7T7GQH6ATVCM4FAPYLQ, seg_pos={0 0}, upload=72429]: Long tail cancellation due to optimal threshold Dec 9 02:24:06.108964487 (82h55m5.980047532s) [seg_pos={0 0}, upload=72429]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 02:24:06.109072202 (82h55m5.980155257s) [upload=72429]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 02:24:06.109080047 (82h55m5.980163092s) [upload=72429]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 02:24:24.569429416 (82h55m24.440512451s) [upload=72429]: Done uploading segments... Dec 9 02:24:24.569477928 (82h55m24.440560963s) [upload=72429]: Upload finished. err: metaclient: manager closed: closed: read tcp 188.166.225.170:38662->35.212.10.183:7777: read: connection reset by peer Duration: 5.765088955s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 6.093594084s trace: https://jaeger.dev.storj.io/trace/40019ff87d088ff7 FAIL (2025-12-09 02:26:40 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.09361872s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [X.............X.......XX........X...........................] duration: 3.475198809s trace: https://jaeger.dev.storj.io/trace/dbaa73a4151be2e FAIL (2025-12-09 02:26:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.47521496s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 2 out of 60 is failed (threshold=3) [..........................X......X..........................] duration: 484.179602ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/4e3d2b60e3f4041b FAIL (2025-12-09 02:26:29 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 484.223046ms core satellite eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 5.819848801s trace: https://jaeger.dev.storj.io/trace/6f1ff10066b1d8ea FAIL (2025-12-09 02:26:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== rue , address=185.167.204.206:4052 , piece_id=WY7PGT7EC5IUBTSHFEAMHXONCIKPFPRF3Y4YSL2IJNY2MOO6U7HA, seg_pos={0 0}, upload=72112]: Uploading piece... Dec 9 02:01:05.223188835 (82h32m5.09427188s) [noise=true , address=89.149.201.147:4011 , piece_id=I77RBTAA7KPTBE27RPPPCNYU65WBBUORVCNYAJOV5OHOBGUFEJKA, seg_pos={0 0}, upload=72112]: Uploading piece... Dec 9 02:01:05.223808365 (82h32m5.09489141s) [noise=true , address=185.167.204.206:4050 , piece_id=YKLEM6U44VR3GM45KKKJF7JYOFA4YQE6P5UAZZEMXIA2LBZW5UKA, seg_pos={0 0}, upload=72112]: Uploading piece... Dec 9 02:01:05.224324407 (82h32m5.095407452s) [noise=true , address=185.167.204.205:4011 , piece_id=6JR2JQYKTRLCSBJDUWWSJ6CRYGWKWN3AW3KJCPXMYLYPUN6W2DRQ, seg_pos={0 0}, upload=72112]: Uploading piece... Dec 9 02:01:05.224877841 (82h32m5.095960886s) [noise=true , address=81.171.14.70:4032 , piece_id=APFBMO7KRDDWH6WT3H4ZLCXBAAMZTZQYQBN3XX6JZDGOBAMSGVCA, seg_pos={0 0}, upload=72112]: Uploading piece... Dec 9 02:01:05.225476030 (82h32m5.096559075s) [noise=true , address=89.149.201.147:4014 , piece_id=PFXVAHNZ2Q6YC2MGNSUZMJIL5Q2O2HPDSONU3FD3E472H2G7AG5A, seg_pos={0 0}, upload=72112]: Uploading piece... Dec 9 02:01:05.226119545 (82h32m5.09720258s) [noise=true , address=89.149.201.147:4015 , piece_id=XBCH6ONGHITOZ2Z5GDPPKDV45TILCDFTOZ44OKMSZANCGVBGQ7ZQ, seg_pos={0 0}, upload=72112]: Uploading piece... Dec 9 02:01:05.226816121 (82h32m5.097899166s) [noise=true , address=89.149.201.147:4012 , piece_id=7UFT32WUAW2JUETM2Y4SFGMSPTNLVVH5FKPQJSK6S3DR6E64ITKQ, seg_pos={0 0}, upload=72112]: Uploading piece... Dec 9 02:01:05.227549758 (82h32m5.098632813s) [noise=true , address=185.167.204.206:4067 , piece_id=PK6GGQNT4DQ2ZMF2IEEO5X32IMA26ULKDN7HAAG4CWOC2UAUKMMA, seg_pos={0 0}, upload=72112]: Uploading piece... Dec 9 02:01:05.228376403 (82h32m5.099459448s) [noise=true , address=81.171.14.70:4046 , piece_id=RR4RRWZ6YO3EK62O7TE2QUC3ZJ762VGUTVXVVHPGMQRPWEFTB7VA, seg_pos={0 0}, upload=72112]: Uploading piece... Dec 9 02:01:05.228990242 (82h32m5.100073297s) [noise=true , address=185.167.204.205:4013 , piece_id=7TBB5J7PYW23YFBOESEZ5Q6YOD3FKALVJMUQRMC5JSQ5JYKGDQSA, seg_pos={0 0}, upload=72112]: Uploading piece... Dec 9 02:01:06.054905806 (82h32m5.925988851s) [upload=72112]: Next returned nil segment Dec 9 02:01:06.054930403 (82h32m5.926013448s) [upload=72112]: Waiting for error group managing segments... Dec 9 02:01:06.441184862 (82h32m6.312267897s) [noise=true , address=185.167.204.205:4004 , piece_id=6CC2A4V4TJPRJBN4IMJ7VGDVFYTJIVY6J3ZIPZWYL5TIBLGT744A, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:06.444455871 (82h32m6.315538916s) [noise=true , address=185.167.204.205:4005 , piece_id=QUO2YNK2MJJ63CKWADK5K64GYKHDOSBPFHKYKQSKH2TJQUZTNEVA, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:06.447822552 (82h32m6.318905597s) [noise=true , address=185.167.204.206:4066 , piece_id=SSGUSCBUP2YNFN7OMUZQIXJJMSHV6GDHQYMN475LJFQQ7SV2FV2A, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:06.451954059 (82h32m6.323037104s) [noise=true , address=185.167.204.205:4008 , piece_id=54IT2P63UNQKCHG4HB7CX7UWJFWDLJXXPOGPNAEN36ZAC4UQ6VQQ, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:06.454212810 (82h32m6.325295856s) [noise=true , address=185.167.204.205:4011 , piece_id=6JR2JQYKTRLCSBJDUWWSJ6CRYGWKWN3AW3KJCPXMYLYPUN6W2DRQ, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:06.454331416 (82h32m6.325414461s) [noise=true , address=185.167.204.206:4052 , piece_id=WY7PGT7EC5IUBTSHFEAMHXONCIKPFPRF3Y4YSL2IJNY2MOO6U7HA, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:06.455138002 (82h32m6.326221047s) [noise=true , address=185.167.204.206:4050 , piece_id=YKLEM6U44VR3GM45KKKJF7JYOFA4YQE6P5UAZZEMXIA2LBZW5UKA, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:06.459504045 (82h32m6.33058709s) [noise=true , address=185.167.204.205:4013 , piece_id=7TBB5J7PYW23YFBOESEZ5Q6YOD3FKALVJMUQRMC5JSQ5JYKGDQSA, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:06.465474574 (82h32m6.336557629s) [noise=true , address=185.167.204.206:4067 , piece_id=PK6GGQNT4DQ2ZMF2IEEO5X32IMA26ULKDN7HAAG4CWOC2UAUKMMA, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.060075359 (82h32m6.931158404s) [noise=true , address=81.171.14.70:4034 , piece_id=DNHPDP5YCQLS5AN6OHWRWVOGOIGE53YB7CJVUHNPJK35AEM455GQ, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.111399558 (82h32m6.982482603s) [noise=true , address=81.171.14.70:4026 , piece_id=CRRJW4OCPQUSWT6BERGQJPYYQ4DHSZIBMIBCD22QWCGGHUISCFXQ, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.133611052 (82h32m7.004694097s) [noise=true , address=81.171.14.70:4032 , piece_id=APFBMO7KRDDWH6WT3H4ZLCXBAAMZTZQYQBN3XX6JZDGOBAMSGVCA, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.149778297 (82h32m7.020861342s) [noise=true , address=81.171.14.70:4030 , piece_id=VOYOWF6Y5DZOJ6DLW6QFYUI6DF5YA7FI4VI57LHMNCZ2PZLRY4BQ, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.243693687 (82h32m7.114776732s) [noise=true , address=81.171.14.70:4040 , piece_id=RYDZ6DCJ3Z5YKJPIFMLR7EHPH55S7HABKAJCKGRGHF2TCRKEYHXQ, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.624315751 (82h32m7.495398796s) [noise=true , address=81.171.14.70:4046 , piece_id=RR4RRWZ6YO3EK62O7TE2QUC3ZJ762VGUTVXVVHPGMQRPWEFTB7VA, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.682160867 (82h32m7.553243912s) [noise=true , address=89.149.201.147:4011 , piece_id=I77RBTAA7KPTBE27RPPPCNYU65WBBUORVCNYAJOV5OHOBGUFEJKA, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.704962114 (82h32m7.576045159s) [noise=true , address=89.149.201.147:4003 , piece_id=KQDYYB3S7PMHBAZTSQDATYYBAUYHMHEQDI6PW4GM4HKXBK4XHIVQ, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.723657624 (82h32m7.594740669s) [noise=true , address=89.149.201.147:4008 , piece_id=JCJNV47ZC7LJPXSERGUXP2QRLHVPOAQNOPI25RQZTA52IYKRZK4Q, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.729409366 (82h32m7.600492411s) [noise=true , address=89.149.201.147:4002 , piece_id=YFUKN6FRXESUYMS6ZV53MCQTZV5UXYHFF72WS2XJWGF3K3I72GTQ, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.761878943 (82h32m7.632961988s) [noise=true , address=89.149.201.147:4004 , piece_id=NGEHOFVBOCBBCYIVGSYLMKTFQWTDJ2EOGKYEW4X2OIWYH5TVGX2Q, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.825863449 (82h32m7.696946494s) [noise=true , address=89.149.201.147:4005 , piece_id=K6WGQSQZ32ZSAD6KN5IWB66MJ3YP76IKULPEHTFSODYFBSESSU4A, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.866343990 (82h32m7.737427035s) [noise=true , address=89.149.201.147:4015 , piece_id=XBCH6ONGHITOZ2Z5GDPPKDV45TILCDFTOZ44OKMSZANCGVBGQ7ZQ, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:08.228780506 (82h32m8.099863541s) [noise=true , address=89.149.201.147:4017 , piece_id=5W66LIGVHUXIA4AVNKUNCOAJAP5TCYAKVFKCL6NDF2OVXNI62UMQ, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:08.230729618 (82h32m8.101812653s) [noise=true , address=89.149.201.147:4019 , piece_id=JTQ44O6RFCMIPK7LAEWXO2NTXAZAJK7XR3XAJGVQXLOLM7SFSBCA, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:08.257428165 (82h32m8.12851121s) [noise=true , address=89.149.201.147:4020 , piece_id=T5RN3NMY35NXWM55YN3F7TWPPP7KXEER5R63W4KZV6Y2UNORJX4A, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:08.267970901 (82h32m8.139053946s) [noise=true , address=89.149.201.147:4006 , piece_id=M3VDXP334RID2FJK7ZYNUHJBRMS57EXIMVMLER25MVJWM4EHJ2WA, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:08.267990939 (82h32m8.139073974s) [seg_pos={0 0}, upload=72112]: Segment reached optimal threshold of 26 pieces. Dec 9 02:01:08.268172084 (82h32m8.139255129s) [noise=true , address=89.149.201.147:4012 , piece_id=7UFT32WUAW2JUETM2Y4SFGMSPTNLVVH5FKPQJSK6S3DR6E64ITKQ, seg_pos={0 0}, upload=72112]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hJ5jT6S2d3DJe9DoVDjHMkpwSTpeyTMyNLaK8Pk8SWcUVRKnv): protocol: expected piece hash; context canceled; EOF Dec 9 02:01:08.268182163 (82h32m8.139265198s) [noise=true , address=89.149.201.147:4012 , piece_id=7UFT32WUAW2JUETM2Y4SFGMSPTNLVVH5FKPQJSK6S3DR6E64ITKQ, seg_pos={0 0}, upload=72112]: Long tail cancellation due to optimal threshold Dec 9 02:01:08.268290639 (82h32m8.139373684s) [noise=true , address=89.149.201.147:4014 , piece_id=PFXVAHNZ2Q6YC2MGNSUZMJIL5Q2O2HPDSONU3FD3E472H2G7AG5A, seg_pos={0 0}, upload=72112]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wgtphHKaTejC1qrJ98sDysBwtj2Zeqw2c5UFk1HMqfY8Buvr): protocol: expected piece hash; context canceled; EOF Dec 9 02:01:08.268299597 (82h32m8.139382642s) [noise=true , address=89.149.201.147:4014 , piece_id=PFXVAHNZ2Q6YC2MGNSUZMJIL5Q2O2HPDSONU3FD3E472H2G7AG5A, seg_pos={0 0}, upload=72112]: Long tail cancellation due to optimal threshold Dec 9 02:01:08.268340294 (82h32m8.139423329s) [seg_pos={0 0}, upload=72112]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 9 02:01:08.268435756 (82h32m8.139518802s) [upload=72112]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 02:01:08.268441557 (82h32m8.139524592s) [upload=72112]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 02:12:41.424380970 (82h43m41.295464025s) [upload=72112]: Done uploading segments... Dec 9 02:12:41.424504095 (82h43m41.29558713s) [upload=72112]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.819877144s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 23.92955456s trace: https://jaeger.dev.storj.io/trace/40019ff87d088ff7 FAIL (2025-12-09 02:26:15 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 23.929613782s core uplink us1
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 2 out of 60 is failed (threshold=3) [...........................X......X.........................] duration: 471.201406ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/4e3d2b60e3f4041b FAIL (2025-12-09 02:26:05 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 471.329961ms core satellite eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 5.949927763s trace: https://jaeger.dev.storj.io/trace/1ddfa89c12e62214 FAIL (2025-12-09 02:25:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: manager closed: closed: read tcp 188.166.225.170:38662->35.212.10.183:7777: read: connection reset by peer ======LOGS====== Dec 9 02:24:03.838353395 (82h55m3.70943644s) [upload=72429]: Starting upload Dec 9 02:24:03.838420683 (82h55m3.709503729s) [upload=72429]: Done starting upload Dec 9 02:24:03.838549719 (82h55m3.709632755s) [upload=72429]: Uploading segments... Dec 9 02:24:03.838688734 (82h55m3.709771779s) [upload=72429]: Got next segment. Inline: false Dec 9 02:24:03.838734311 (82h55m3.709817356s) [upload=72429]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 02:24:03.838741164 (82h55m3.709824209s) [upload=72429]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 02:24:04.086906602 (82h55m3.957989647s) [seg_pos={0 0}, upload=72429]: Begin upload segment... Dec 9 02:24:04.086974701 (82h55m3.958057746s) [seg_pos={0 0}, upload=72429]: Done begin upload segment. Dec 9 02:24:04.087029476 (82h55m3.958112511s) [noise=true , address=145.79.227.24:4581 , piece_id=ETKQGTKTMEKLK42KNN72MBQF62TFTOQO357XKF4WS6NTVBSOANDA, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.087057810 (82h55m3.958140855s) [noise=true , address=145.79.227.16:4911 , piece_id=Q62EUAKMIIB6QZ3KTNQH3PJAZIX7YNRYXJPTATNCZ4W6RWELTLXQ, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.087925792 (82h55m3.959008837s) [noise=true , address=145.79.227.42:4122 , piece_id=ZWY35JIEVFUL3LADBSS4EKYMWMFBTHJWMD6YQ7M4OGKXYAU4LNWQ, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.088596299 (82h55m3.959679344s) [noise=true , address=145.79.227.23:4545 , piece_id=NPORRF6TACVJIVBM3IABBQF7NBYSDRGJTPGDREFMOWNNT3B325UQ, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.089090329 (82h55m3.960173374s) [noise=true , address=145.79.227.43:4158 , piece_id=DS3UAJZAYVR4GXPHKPMJ2RO7765FUPASW7T7GQH6ATVCM4FAPYLQ, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.089582366 (82h55m3.960665411s) [noise=true , address=145.79.227.11:4732 , piece_id=CA5XB6ODOYLW3RQU5F7BMUDDSAGEJGIKOP7IZBHPNUTUXYDJ4JWQ, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.090135129 (82h55m3.961218174s) [noise=true , address=145.79.227.34:4906 , piece_id=GTGD6FTBFJ6VMETYMRJPTO7P77B4TBQT4VSRAEMUCJ6TTW4VBHDA, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.090616965 (82h55m3.96170001s) [noise=true , address=145.79.227.24:4583 , piece_id=MFGUNU4E6DBCQ5TUNPSWBXRSEQ7QGTNWTWCUTZK5NAVLMIBB2KKQ, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.091103331 (82h55m3.962186376s) [noise=true , address=145.79.227.25:4619 , piece_id=RALQO5F6I5M4Q56PLG47D57UH4WHRQWB47N45NASEPILQGC26VQQ, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.091901141 (82h55m3.962984186s) [noise=true , address=145.79.227.38:4978 , piece_id=XBHD6MMZECN7T7QTC3HVN7OS5XY7CJJJEFERUVK2ZIXTVWSVNLXA, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.094168697 (82h55m3.965251742s) [noise=true , address=77.90.31.16:4734 , piece_id=WSIBQS2XW3PENG55ORDRTUYXPZ6HCSTAX7DP5JFDVVCLCX6JGLTA, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.094774330 (82h55m3.965857375s) [noise=true , address=145.79.227.40:4049 , piece_id=VH3PRUHK4OR4MCGVX3VHMXKWTVE3KQLKH5RJHI4MLJMITRHILR2A, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.095474052 (82h55m3.966557097s) [noise=true , address=145.79.227.31:4797 , piece_id=XDQGJD7JELZ3D73GMYRVHKWEFIZGYF6IWDBZ54NIJTGA3G7WWMFQ, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.096054146 (82h55m3.967137191s) [noise=true , address=145.79.227.21:4474 , piece_id=TTDRH27TBF34WUNSWZL3XL2RR6SJGEHP7J4YCHAZB4SH3ZWH4PVA, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.096661643 (82h55m3.967744688s) [noise=true , address=77.90.31.10:4520 , piece_id=GW2ECPZB4XCEXAWWCBVUM2VEUXXMSBPJUZSKTULZYCHK4DZL63XA, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:04.097203715 (82h55m3.96828676s) [noise=true , address=145.79.227.22:4511 , piece_id=QVG66WWAN3TCESLCDW2ZG7SQADAJPUIILVYGFGTCKRLGVXJWSQ7Q, seg_pos={0 0}, upload=72429]: Uploading piece... Dec 9 02:24:05.228786599 (82h55m5.099869634s) [noise=true , address=77.90.31.16:4734 , piece_id=WSIBQS2XW3PENG55ORDRTUYXPZ6HCSTAX7DP5JFDVVCLCX6JGLTA, seg_pos={0 0}, upload=72429]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: storage node overloaded, request limit: 2000 Dec 9 02:24:05.771060198 (82h55m5.642143243s) [upload=72429]: Next returned nil segment Dec 9 02:24:05.771079454 (82h55m5.642162529s) [upload=72429]: Waiting for error group managing segments... Dec 9 02:24:06.084934630 (82h55m5.956017675s) [noise=true , address=145.79.227.24:4583 , piece_id=MFGUNU4E6DBCQ5TUNPSWBXRSEQ7QGTNWTWCUTZK5NAVLMIBB2KKQ, seg_pos={0 0}, upload=72429]: Done uploading piece. err: <nil> Dec 9 02:24:06.086488888 (82h55m5.957571934s) [noise=true , address=145.79.227.42:4122 , piece_id=ZWY35JIEVFUL3LADBSS4EKYMWMFBTHJWMD6YQ7M4OGKXYAU4LNWQ, seg_pos={0 0}, upload=72429]: Done uploading piece. err: <nil> Dec 9 02:24:06.087432816 (82h55m5.958515861s) [noise=true , address=145.79.227.25:4619 , piece_id=RALQO5F6I5M4Q56PLG47D57UH4WHRQWB47N45NASEPILQGC26VQQ, seg_pos={0 0}, upload=72429]: Done uploading piece. err: <nil> Dec 9 02:24:06.088224853 (82h55m5.959307898s) [noise=true , address=145.79.227.24:4581 , piece_id=ETKQGTKTMEKLK42KNN72MBQF62TFTOQO357XKF4WS6NTVBSOANDA, seg_pos={0 0}, upload=72429]: Done uploading piece. err: <nil> Dec 9 02:24:06.090286018 (82h55m5.961369053s) [noise=true , address=145.79.227.31:4797 , piece_id=XDQGJD7JELZ3D73GMYRVHKWEFIZGYF6IWDBZ54NIJTGA3G7WWMFQ, seg_pos={0 0}, upload=72429]: Done uploading piece. err: <nil> Dec 9 02:24:06.094149082 (82h55m5.965232127s) [noise=true , address=145.79.227.34:4906 , piece_id=GTGD6FTBFJ6VMETYMRJPTO7P77B4TBQT4VSRAEMUCJ6TTW4VBHDA, seg_pos={0 0}, upload=72429]: Done uploading piece. err: <nil> Dec 9 02:24:06.096430375 (82h55m5.96751342s) [noise=true , address=145.79.227.21:4474 , piece_id=TTDRH27TBF34WUNSWZL3XL2RR6SJGEHP7J4YCHAZB4SH3ZWH4PVA, seg_pos={0 0}, upload=72429]: Done uploading piece. err: <nil> Dec 9 02:24:06.096707993 (82h55m5.967791028s) [noise=true , address=145.79.227.11:4732 , piece_id=CA5XB6ODOYLW3RQU5F7BMUDDSAGEJGIKOP7IZBHPNUTUXYDJ4JWQ, seg_pos={0 0}, upload=72429]: Done uploading piece. err: <nil> Dec 9 02:24:06.103696047 (82h55m5.974779092s) [noise=true , address=145.79.227.16:4911 , piece_id=Q62EUAKMIIB6QZ3KTNQH3PJAZIX7YNRYXJPTATNCZ4W6RWELTLXQ, seg_pos={0 0}, upload=72429]: Done uploading piece. err: <nil> Dec 9 02:24:06.105006020 (82h55m5.976089065s) [noise=true , address=145.79.227.23:4545 , piece_id=NPORRF6TACVJIVBM3IABBQF7NBYSDRGJTPGDREFMOWNNT3B325UQ, seg_pos={0 0}, upload=72429]: Done uploading piece. err: <nil> Dec 9 02:24:06.105250326 (82h55m5.976333381s) [noise=true , address=145.79.227.38:4978 , piece_id=XBHD6MMZECN7T7QTC3HVN7OS5XY7CJJJEFERUVK2ZIXTVWSVNLXA, seg_pos={0 0}, upload=72429]: Done uploading piece. err: <nil> Dec 9 02:24:06.105501193 (82h55m5.976584238s) [noise=true , address=145.79.227.22:4511 , piece_id=QVG66WWAN3TCESLCDW2ZG7SQADAJPUIILVYGFGTCKRLGVXJWSQ7Q, seg_pos={0 0}, upload=72429]: Done uploading piece. err: <nil> Dec 9 02:24:06.108138354 (82h55m5.979221399s) [noise=true , address=145.79.227.40:4049 , piece_id=VH3PRUHK4OR4MCGVX3VHMXKWTVE3KQLKH5RJHI4MLJMITRHILR2A, seg_pos={0 0}, upload=72429]: Done uploading piece. err: <nil> Dec 9 02:24:06.108158543 (82h55m5.979241588s) [seg_pos={0 0}, upload=72429]: Segment reached optimal threshold of 13 pieces. Dec 9 02:24:06.108688862 (82h55m5.979771907s) [noise=true , address=77.90.31.10:4520 , piece_id=GW2ECPZB4XCEXAWWCBVUM2VEUXXMSBPJUZSKTULZYCHK4DZL63XA, seg_pos={0 0}, upload=72429]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Dec 9 02:24:06.108700394 (82h55m5.979783449s) [noise=true , address=77.90.31.10:4520 , piece_id=GW2ECPZB4XCEXAWWCBVUM2VEUXXMSBPJUZSKTULZYCHK4DZL63XA, seg_pos={0 0}, upload=72429]: Long tail cancellation due to optimal threshold Dec 9 02:24:06.108930402 (82h55m5.980013447s) [noise=true , address=145.79.227.43:4158 , piece_id=DS3UAJZAYVR4GXPHKPMJ2RO7765FUPASW7T7GQH6ATVCM4FAPYLQ, seg_pos={0 0}, upload=72429]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF Dec 9 02:24:06.108941323 (82h55m5.980024368s) [noise=true , address=145.79.227.43:4158 , piece_id=DS3UAJZAYVR4GXPHKPMJ2RO7765FUPASW7T7GQH6ATVCM4FAPYLQ, seg_pos={0 0}, upload=72429]: Long tail cancellation due to optimal threshold Dec 9 02:24:06.108964487 (82h55m5.980047532s) [seg_pos={0 0}, upload=72429]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 02:24:06.109072202 (82h55m5.980155257s) [upload=72429]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 02:24:06.109080047 (82h55m5.980163092s) [upload=72429]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 02:24:24.569429416 (82h55m24.440512451s) [upload=72429]: Done uploading segments... Dec 9 02:24:24.569477928 (82h55m24.440560963s) [upload=72429]: Upload finished. err: metaclient: manager closed: closed: read tcp 188.166.225.170:38662->35.212.10.183:7777: read: connection reset by peer Duration: 5.949956048s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.....................X.X.....X..............................] duration: 5.842190086s trace: https://jaeger.dev.storj.io/trace/5280cd334ed3ed6b FAIL (2025-12-09 02:25:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.842221335s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 2 out of 60 is failed (threshold=3) [............................X......X........................] duration: 468.182417ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/4e3d2b60e3f4041b FAIL (2025-12-09 02:25:40 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 468.206002ms core satellite eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 6.398339162s trace: https://jaeger.dev.storj.io/trace/6f1ff10066b1d8ea FAIL (2025-12-09 02:25:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== rue , address=185.167.204.206:4052 , piece_id=WY7PGT7EC5IUBTSHFEAMHXONCIKPFPRF3Y4YSL2IJNY2MOO6U7HA, seg_pos={0 0}, upload=72112]: Uploading piece... Dec 9 02:01:05.223188835 (82h32m5.09427188s) [noise=true , address=89.149.201.147:4011 , piece_id=I77RBTAA7KPTBE27RPPPCNYU65WBBUORVCNYAJOV5OHOBGUFEJKA, seg_pos={0 0}, upload=72112]: Uploading piece... Dec 9 02:01:05.223808365 (82h32m5.09489141s) [noise=true , address=185.167.204.206:4050 , piece_id=YKLEM6U44VR3GM45KKKJF7JYOFA4YQE6P5UAZZEMXIA2LBZW5UKA, seg_pos={0 0}, upload=72112]: Uploading piece... Dec 9 02:01:05.224324407 (82h32m5.095407452s) [noise=true , address=185.167.204.205:4011 , piece_id=6JR2JQYKTRLCSBJDUWWSJ6CRYGWKWN3AW3KJCPXMYLYPUN6W2DRQ, seg_pos={0 0}, upload=72112]: Uploading piece... Dec 9 02:01:05.224877841 (82h32m5.095960886s) [noise=true , address=81.171.14.70:4032 , piece_id=APFBMO7KRDDWH6WT3H4ZLCXBAAMZTZQYQBN3XX6JZDGOBAMSGVCA, seg_pos={0 0}, upload=72112]: Uploading piece... Dec 9 02:01:05.225476030 (82h32m5.096559075s) [noise=true , address=89.149.201.147:4014 , piece_id=PFXVAHNZ2Q6YC2MGNSUZMJIL5Q2O2HPDSONU3FD3E472H2G7AG5A, seg_pos={0 0}, upload=72112]: Uploading piece... Dec 9 02:01:05.226119545 (82h32m5.09720258s) [noise=true , address=89.149.201.147:4015 , piece_id=XBCH6ONGHITOZ2Z5GDPPKDV45TILCDFTOZ44OKMSZANCGVBGQ7ZQ, seg_pos={0 0}, upload=72112]: Uploading piece... Dec 9 02:01:05.226816121 (82h32m5.097899166s) [noise=true , address=89.149.201.147:4012 , piece_id=7UFT32WUAW2JUETM2Y4SFGMSPTNLVVH5FKPQJSK6S3DR6E64ITKQ, seg_pos={0 0}, upload=72112]: Uploading piece... Dec 9 02:01:05.227549758 (82h32m5.098632813s) [noise=true , address=185.167.204.206:4067 , piece_id=PK6GGQNT4DQ2ZMF2IEEO5X32IMA26ULKDN7HAAG4CWOC2UAUKMMA, seg_pos={0 0}, upload=72112]: Uploading piece... Dec 9 02:01:05.228376403 (82h32m5.099459448s) [noise=true , address=81.171.14.70:4046 , piece_id=RR4RRWZ6YO3EK62O7TE2QUC3ZJ762VGUTVXVVHPGMQRPWEFTB7VA, seg_pos={0 0}, upload=72112]: Uploading piece... Dec 9 02:01:05.228990242 (82h32m5.100073297s) [noise=true , address=185.167.204.205:4013 , piece_id=7TBB5J7PYW23YFBOESEZ5Q6YOD3FKALVJMUQRMC5JSQ5JYKGDQSA, seg_pos={0 0}, upload=72112]: Uploading piece... Dec 9 02:01:06.054905806 (82h32m5.925988851s) [upload=72112]: Next returned nil segment Dec 9 02:01:06.054930403 (82h32m5.926013448s) [upload=72112]: Waiting for error group managing segments... Dec 9 02:01:06.441184862 (82h32m6.312267897s) [noise=true , address=185.167.204.205:4004 , piece_id=6CC2A4V4TJPRJBN4IMJ7VGDVFYTJIVY6J3ZIPZWYL5TIBLGT744A, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:06.444455871 (82h32m6.315538916s) [noise=true , address=185.167.204.205:4005 , piece_id=QUO2YNK2MJJ63CKWADK5K64GYKHDOSBPFHKYKQSKH2TJQUZTNEVA, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:06.447822552 (82h32m6.318905597s) [noise=true , address=185.167.204.206:4066 , piece_id=SSGUSCBUP2YNFN7OMUZQIXJJMSHV6GDHQYMN475LJFQQ7SV2FV2A, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:06.451954059 (82h32m6.323037104s) [noise=true , address=185.167.204.205:4008 , piece_id=54IT2P63UNQKCHG4HB7CX7UWJFWDLJXXPOGPNAEN36ZAC4UQ6VQQ, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:06.454212810 (82h32m6.325295856s) [noise=true , address=185.167.204.205:4011 , piece_id=6JR2JQYKTRLCSBJDUWWSJ6CRYGWKWN3AW3KJCPXMYLYPUN6W2DRQ, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:06.454331416 (82h32m6.325414461s) [noise=true , address=185.167.204.206:4052 , piece_id=WY7PGT7EC5IUBTSHFEAMHXONCIKPFPRF3Y4YSL2IJNY2MOO6U7HA, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:06.455138002 (82h32m6.326221047s) [noise=true , address=185.167.204.206:4050 , piece_id=YKLEM6U44VR3GM45KKKJF7JYOFA4YQE6P5UAZZEMXIA2LBZW5UKA, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:06.459504045 (82h32m6.33058709s) [noise=true , address=185.167.204.205:4013 , piece_id=7TBB5J7PYW23YFBOESEZ5Q6YOD3FKALVJMUQRMC5JSQ5JYKGDQSA, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:06.465474574 (82h32m6.336557629s) [noise=true , address=185.167.204.206:4067 , piece_id=PK6GGQNT4DQ2ZMF2IEEO5X32IMA26ULKDN7HAAG4CWOC2UAUKMMA, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.060075359 (82h32m6.931158404s) [noise=true , address=81.171.14.70:4034 , piece_id=DNHPDP5YCQLS5AN6OHWRWVOGOIGE53YB7CJVUHNPJK35AEM455GQ, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.111399558 (82h32m6.982482603s) [noise=true , address=81.171.14.70:4026 , piece_id=CRRJW4OCPQUSWT6BERGQJPYYQ4DHSZIBMIBCD22QWCGGHUISCFXQ, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.133611052 (82h32m7.004694097s) [noise=true , address=81.171.14.70:4032 , piece_id=APFBMO7KRDDWH6WT3H4ZLCXBAAMZTZQYQBN3XX6JZDGOBAMSGVCA, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.149778297 (82h32m7.020861342s) [noise=true , address=81.171.14.70:4030 , piece_id=VOYOWF6Y5DZOJ6DLW6QFYUI6DF5YA7FI4VI57LHMNCZ2PZLRY4BQ, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.243693687 (82h32m7.114776732s) [noise=true , address=81.171.14.70:4040 , piece_id=RYDZ6DCJ3Z5YKJPIFMLR7EHPH55S7HABKAJCKGRGHF2TCRKEYHXQ, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.624315751 (82h32m7.495398796s) [noise=true , address=81.171.14.70:4046 , piece_id=RR4RRWZ6YO3EK62O7TE2QUC3ZJ762VGUTVXVVHPGMQRPWEFTB7VA, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.682160867 (82h32m7.553243912s) [noise=true , address=89.149.201.147:4011 , piece_id=I77RBTAA7KPTBE27RPPPCNYU65WBBUORVCNYAJOV5OHOBGUFEJKA, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.704962114 (82h32m7.576045159s) [noise=true , address=89.149.201.147:4003 , piece_id=KQDYYB3S7PMHBAZTSQDATYYBAUYHMHEQDI6PW4GM4HKXBK4XHIVQ, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.723657624 (82h32m7.594740669s) [noise=true , address=89.149.201.147:4008 , piece_id=JCJNV47ZC7LJPXSERGUXP2QRLHVPOAQNOPI25RQZTA52IYKRZK4Q, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.729409366 (82h32m7.600492411s) [noise=true , address=89.149.201.147:4002 , piece_id=YFUKN6FRXESUYMS6ZV53MCQTZV5UXYHFF72WS2XJWGF3K3I72GTQ, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.761878943 (82h32m7.632961988s) [noise=true , address=89.149.201.147:4004 , piece_id=NGEHOFVBOCBBCYIVGSYLMKTFQWTDJ2EOGKYEW4X2OIWYH5TVGX2Q, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.825863449 (82h32m7.696946494s) [noise=true , address=89.149.201.147:4005 , piece_id=K6WGQSQZ32ZSAD6KN5IWB66MJ3YP76IKULPEHTFSODYFBSESSU4A, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:07.866343990 (82h32m7.737427035s) [noise=true , address=89.149.201.147:4015 , piece_id=XBCH6ONGHITOZ2Z5GDPPKDV45TILCDFTOZ44OKMSZANCGVBGQ7ZQ, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:08.228780506 (82h32m8.099863541s) [noise=true , address=89.149.201.147:4017 , piece_id=5W66LIGVHUXIA4AVNKUNCOAJAP5TCYAKVFKCL6NDF2OVXNI62UMQ, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:08.230729618 (82h32m8.101812653s) [noise=true , address=89.149.201.147:4019 , piece_id=JTQ44O6RFCMIPK7LAEWXO2NTXAZAJK7XR3XAJGVQXLOLM7SFSBCA, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:08.257428165 (82h32m8.12851121s) [noise=true , address=89.149.201.147:4020 , piece_id=T5RN3NMY35NXWM55YN3F7TWPPP7KXEER5R63W4KZV6Y2UNORJX4A, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:08.267970901 (82h32m8.139053946s) [noise=true , address=89.149.201.147:4006 , piece_id=M3VDXP334RID2FJK7ZYNUHJBRMS57EXIMVMLER25MVJWM4EHJ2WA, seg_pos={0 0}, upload=72112]: Done uploading piece. err: <nil> Dec 9 02:01:08.267990939 (82h32m8.139073974s) [seg_pos={0 0}, upload=72112]: Segment reached optimal threshold of 26 pieces. Dec 9 02:01:08.268172084 (82h32m8.139255129s) [noise=true , address=89.149.201.147:4012 , piece_id=7UFT32WUAW2JUETM2Y4SFGMSPTNLVVH5FKPQJSK6S3DR6E64ITKQ, seg_pos={0 0}, upload=72112]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hJ5jT6S2d3DJe9DoVDjHMkpwSTpeyTMyNLaK8Pk8SWcUVRKnv): protocol: expected piece hash; context canceled; EOF Dec 9 02:01:08.268182163 (82h32m8.139265198s) [noise=true , address=89.149.201.147:4012 , piece_id=7UFT32WUAW2JUETM2Y4SFGMSPTNLVVH5FKPQJSK6S3DR6E64ITKQ, seg_pos={0 0}, upload=72112]: Long tail cancellation due to optimal threshold Dec 9 02:01:08.268290639 (82h32m8.139373684s) [noise=true , address=89.149.201.147:4014 , piece_id=PFXVAHNZ2Q6YC2MGNSUZMJIL5Q2O2HPDSONU3FD3E472H2G7AG5A, seg_pos={0 0}, upload=72112]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wgtphHKaTejC1qrJ98sDysBwtj2Zeqw2c5UFk1HMqfY8Buvr): protocol: expected piece hash; context canceled; EOF Dec 9 02:01:08.268299597 (82h32m8.139382642s) [noise=true , address=89.149.201.147:4014 , piece_id=PFXVAHNZ2Q6YC2MGNSUZMJIL5Q2O2HPDSONU3FD3E472H2G7AG5A, seg_pos={0 0}, upload=72112]: Long tail cancellation due to optimal threshold Dec 9 02:01:08.268340294 (82h32m8.139423329s) [seg_pos={0 0}, upload=72112]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 9 02:01:08.268435756 (82h32m8.139518802s) [upload=72112]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 02:01:08.268441557 (82h32m8.139524592s) [upload=72112]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 02:12:41.424380970 (82h43m41.295464025s) [upload=72112]: Done uploading segments... Dec 9 02:12:41.424504095 (82h43m41.29558713s) [upload=72112]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.398368678s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [.X.............X.......XX........X..........................] duration: 3.453577525s trace: https://jaeger.dev.storj.io/trace/dbaa73a4151be2e FAIL (2025-12-09 02:25:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.45366926s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)