Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [...............X.......................................XX...] duration: 7.30536186s trace: https://jaeger.dev.storj.io/trace/70c2b0731de5fb9c FAIL (2026-01-16 17:05:44 UTC) orbiter warning: context deadline exceeded Duration: 7.305381566s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................................................XXX....] duration: 8.883795715s trace: https://jaeger.dev.storj.io/trace/521596a1b50467d9 FAIL (2026-01-16 17:05:38 UTC) orbiter warning: context deadline exceeded Duration: 8.883811976s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 53 out of 60 is failed (threshold=6) [XX.XXXX.XXXXXXXXXXXXXXXXXXXX.XXXXXX.X..XXXXXXX.XXXXXXXXXXXXX] duration: 5.000629238s trace: https://jaeger.dev.storj.io/trace/76af2ef8cdd41ec0 FAIL (2026-01-16 17:05:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000689412s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......................................................XXX...] duration: 14.417710876s trace: https://jaeger.dev.storj.io/trace/521596a1b50467d9 FAIL (2026-01-16 17:05:08 UTC) orbiter warning: context deadline exceeded Duration: 14.417737285s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [................X.......................................XX..] duration: 7.749578808s trace: https://jaeger.dev.storj.io/trace/70c2b0731de5fb9c FAIL (2026-01-16 17:05:05 UTC) orbiter warning: context deadline exceeded Duration: 7.749604809s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 53 out of 60 is failed (threshold=6) [XXX.XXXX.XXXXXXXXXXXXXXXXXXXX.XXXXXX.X..XXXXXXX.XXXXXXXXXXXX] duration: 5.000848505s trace: https://jaeger.dev.storj.io/trace/4fc93e0998efb113 FAIL (2026-01-16 17:05:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000883931s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................................XX.............] duration: 5.696014575s trace: https://jaeger.dev.storj.io/trace/ff1eaafc0d95989 FAIL (2026-01-16 17:05:01 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 16 16:56:06.586316499 (25h0m3.480025108s) [upload=21929]: Starting upload Jan 16 16:56:06.586371244 (25h0m3.480079853s) [upload=21929]: Done starting upload Jan 16 16:56:06.586529134 (25h0m3.480237763s) [upload=21929]: Uploading segments... Jan 16 16:56:06.586687636 (25h0m3.480396275s) [upload=21929]: Got next segment. Inline: false Jan 16 16:56:06.586750094 (25h0m3.480458723s) [upload=21929]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 16 16:56:06.586757138 (25h0m3.480465767s) [upload=21929]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 16 16:56:08.120364931 (25h0m5.01407354s) [seg_pos={0 0}, upload=21929]: Begin upload segment... Jan 16 16:56:08.120413474 (25h0m5.014122083s) [seg_pos={0 0}, upload=21929]: Done begin upload segment. Jan 16 16:56:08.120440615 (25h0m5.014149224s) [noise=true , address=145.79.227.2:4115 , piece_id=YMME5WUV7XGQZOE2I5H7UWQ7TSTP2LGV7D4AZO4T62XOWBE4ZEKA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.120799800 (25h0m5.014508419s) [noise=true , address=145.79.227.32:4835 , piece_id=6PP7J34OCTKYEDEFQJWDUHO5H4Z6VG7LNG3SL7WLITSKMX43HAYQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.121695454 (25h0m5.015404084s) [noise=true , address=45.91.116.4:4937 , piece_id=MJHRB2I4KWRJJ3MHGW657D53JFDARTHOBRUEHQZEUA4DRFVK7ZUA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.122317739 (25h0m5.016026349s) [noise=true , address=145.79.227.30:4762 , piece_id=YLBYKHJNJ2J3YYVYHRWEZTFLXZVL5WCKCBWXM526XF4KKCYA4RAQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.123161636 (25h0m5.016870245s) [noise=true , address=45.91.116.14:4190 , piece_id=ZBC7C5E5V7B4VAGI2JHD7T4E6BGVV74YNNAROZYBITS2N23BJDWA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.123395732 (25h0m5.017104351s) [noise=true , address=145.79.227.28:4727 , piece_id=U5CDDUXQZ2BB7SPFAIWQXRGALJQTOWFNWGJLU3CGXKBUHYIQBIGQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.123848494 (25h0m5.017557113s) [noise=true , address=77.90.31.19:4842 , piece_id=3VJ3J4JQPSJEVFSNSYDVHRFZMMCBMNZRBYEGJOOU4Q4HFEDDOKHQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.123868092 (25h0m5.017576731s) [noise=true , address=145.79.227.15:4873 , piece_id=EVJGQ5JNJI37QIPQDJPFOPMT3JAHPBZXQG7DBRUZXJMKXAIADWSA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.124249808 (25h0m5.017958417s) [noise=true , address=145.79.227.38:4977 , piece_id=JXV6MGOVSAA2URSIF2KFFQJJEGWQWFVT65PWVDW5MEH5ITSDYFAA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.124389284 (25h0m5.018097893s) [noise=true , address=145.79.227.49:4339 , piece_id=NWDULUWHMV4PIMWMNO5XYXSAGVLNPLSIPYSDA32OZB2MBGBYPEUA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.124619222 (25h0m5.018327831s) [noise=true , address=145.79.227.33:4870 , piece_id=7I3KU7EKC2MHBJ6TOGO5ZBMJSPYU4E26D5373M2R53IZAZJLWVOA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.124999566 (25h0m5.018708175s) [noise=true , address=145.79.227.3:4151 , piece_id=QVFK6RJITYJIW6ZJRUYFU3P2NEMMVANA36O6WXPS7Z7FNFSYQWUA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.125202182 (25h0m5.018910801s) [noise=true , address=77.90.31.13:4627 , piece_id=TJFFMEVYN7M52FH5UP3WNAGC7HWF5GSEUPQJZ5KJWTJZWKZNAWXQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.125576805 (25h0m5.019285414s) [noise=true , address=45.91.116.5:4300 , piece_id=M2F4W7IGAEKEXJXZRENON55A2E7DNSCZTPKWYJFY3NKNTBIWG2YA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.125952470 (25h0m5.019661079s) [noise=true , address=145.79.227.10:4401 , piece_id=C2X52SPGSNM2CB6TFGBCT6HO3MXSEMX3CCZQASDNFGWIEW2EGIGQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.126492900 (25h0m5.020201509s) [noise=true , address=145.79.227.34:4906 , piece_id=G2GHF4S5NVJP5R43EN6NDEJ4FCPCLSL5K3TO4UADJ6ZANME54KOQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:10.730578935 (25h0m7.624287544s) [upload=21929]: Next returned nil segment Jan 16 16:56:10.730594134 (25h0m7.624302753s) [upload=21929]: Waiting for error group managing segments... Jan 16 16:56:11.329683279 (25h0m8.223391888s) [noise=true , address=145.79.227.15:4873 , piece_id=EVJGQ5JNJI37QIPQDJPFOPMT3JAHPBZXQG7DBRUZXJMKXAIADWSA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.338893335 (25h0m8.232601944s) [noise=true , address=45.91.116.14:4190 , piece_id=ZBC7C5E5V7B4VAGI2JHD7T4E6BGVV74YNNAROZYBITS2N23BJDWA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.400223651 (25h0m8.29393226s) [noise=true , address=145.79.227.33:4870 , piece_id=7I3KU7EKC2MHBJ6TOGO5ZBMJSPYU4E26D5373M2R53IZAZJLWVOA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.400340974 (25h0m8.294049583s) [noise=true , address=145.79.227.2:4115 , piece_id=YMME5WUV7XGQZOE2I5H7UWQ7TSTP2LGV7D4AZO4T62XOWBE4ZEKA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.445450294 (25h0m8.339158904s) [noise=true , address=145.79.227.30:4762 , piece_id=YLBYKHJNJ2J3YYVYHRWEZTFLXZVL5WCKCBWXM526XF4KKCYA4RAQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.964879212 (25h0m8.858587821s) [noise=true , address=145.79.227.3:4151 , piece_id=QVFK6RJITYJIW6ZJRUYFU3P2NEMMVANA36O6WXPS7Z7FNFSYQWUA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.027795357 (25h0m8.921503966s) [noise=true , address=145.79.227.49:4339 , piece_id=NWDULUWHMV4PIMWMNO5XYXSAGVLNPLSIPYSDA32OZB2MBGBYPEUA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.267543450 (25h0m9.161252059s) [noise=true , address=145.79.227.28:4727 , piece_id=U5CDDUXQZ2BB7SPFAIWQXRGALJQTOWFNWGJLU3CGXKBUHYIQBIGQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.337643945 (25h0m9.231352554s) [noise=true , address=145.79.227.10:4401 , piece_id=C2X52SPGSNM2CB6TFGBCT6HO3MXSEMX3CCZQASDNFGWIEW2EGIGQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.380001967 (25h0m9.273710576s) [noise=true , address=145.79.227.32:4835 , piece_id=6PP7J34OCTKYEDEFQJWDUHO5H4Z6VG7LNG3SL7WLITSKMX43HAYQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.850164228 (25h0m9.743872837s) [noise=true , address=77.90.31.13:4627 , piece_id=TJFFMEVYN7M52FH5UP3WNAGC7HWF5GSEUPQJZ5KJWTJZWKZNAWXQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:13.183873887 (25h0m10.077582496s) [noise=true , address=77.90.31.19:4842 , piece_id=3VJ3J4JQPSJEVFSNSYDVHRFZMMCBMNZRBYEGJOOU4Q4HFEDDOKHQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:13.361354387 (25h0m10.255063026s) [noise=true , address=45.91.116.4:4937 , piece_id=MJHRB2I4KWRJJ3MHGW657D53JFDARTHOBRUEHQZEUA4DRFVK7ZUA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:13.361383082 (25h0m10.255091691s) [seg_pos={0 0}, upload=21929]: Satisfied cohort requirements after 13 pieces Jan 16 16:56:13.361615564 (25h0m10.255324173s) [noise=true , address=145.79.227.34:4906 , piece_id=G2GHF4S5NVJP5R43EN6NDEJ4FCPCLSL5K3TO4UADJ6ZANME54KOQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Jan 16 16:56:13.361624061 (25h0m10.25533267s) [noise=true , address=145.79.227.34:4906 , piece_id=G2GHF4S5NVJP5R43EN6NDEJ4FCPCLSL5K3TO4UADJ6ZANME54KOQ, seg_pos={0 0}, upload=21929]: Long tail cancellation due to optimal threshold Jan 16 16:56:13.361904004 (25h0m10.255612623s) [noise=true , address=145.79.227.38:4977 , piece_id=JXV6MGOVSAA2URSIF2KFFQJJEGWQWFVT65PWVDW5MEH5ITSDYFAA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8): protocol: read tcp 188.166.225.170:42244->145.79.227.38:4977: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 16 16:56:13.361925344 (25h0m10.255633963s) [noise=true , address=145.79.227.38:4977 , piece_id=JXV6MGOVSAA2URSIF2KFFQJJEGWQWFVT65PWVDW5MEH5ITSDYFAA, seg_pos={0 0}, upload=21929]: Long tail cancellation due to optimal threshold Jan 16 16:56:13.361971022 (25h0m10.255679631s) [noise=true , address=45.91.116.5:4300 , piece_id=M2F4W7IGAEKEXJXZRENON55A2E7DNSCZTPKWYJFY3NKNTBIWG2YA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Jan 16 16:56:13.361978977 (25h0m10.255687596s) [noise=true , address=45.91.116.5:4300 , piece_id=M2F4W7IGAEKEXJXZRENON55A2E7DNSCZTPKWYJFY3NKNTBIWG2YA, seg_pos={0 0}, upload=21929]: Long tail cancellation due to optimal threshold Jan 16 16:56:13.362041817 (25h0m10.255750426s) [seg_pos={0 0}, upload=21929]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 16 16:56:13.362197272 (25h0m10.255905881s) [upload=21929]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 16 16:56:13.362205868 (25h0m10.255914477s) [upload=21929]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 16 16:56:13.965177888 (25h0m10.858886497s) [upload=21929]: Done uploading segments... Jan 16 16:56:13.965204158 (25h0m10.858912767s) [upload=21929]: Upload finished. err: <nil> Duration: 5.696042097s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.................X.......................................XX.] duration: 12.591798011s trace: https://jaeger.dev.storj.io/trace/70c2b0731de5fb9c FAIL (2026-01-16 17:04:35 UTC) orbiter warning: context deadline exceeded Duration: 12.591819543s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......................................................XXX..] duration: 26.718691096s trace: https://jaeger.dev.storj.io/trace/521596a1b50467d9 FAIL (2026-01-16 17:04:21 UTC) orbiter warning: context deadline exceeded Duration: 26.718707509s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 53 out of 60 is failed (threshold=6) [XXXX.XXXX.XXXXXXXXXXXXXXXXXXXX.XXXXXX.X..XXXXXXX.XXXXXXXXXXX] duration: 5.001214152s trace: https://jaeger.dev.storj.io/trace/5fa5ec7ddcc17260 FAIL (2026-01-16 17:04:21 UTC) orbiter warning: context deadline exceeded Duration: 5.001236506s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..................X.......................................XX] duration: 1m0.001402443s trace: https://jaeger.dev.storj.io/trace/70c2b0731de5fb9c FAIL (2026-01-16 17:04:20 UTC) orbiter warning: context deadline exceeded Duration: 1m0.001412291s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 53 out of 60 is failed (threshold=6) [XXXXX.XXXX.XXXXXXXXXXXXXXXXXXXX.XXXXXX.X..XXXXXXX.XXXXXXXXXX] duration: 5.001349742s trace: https://jaeger.dev.storj.io/trace/38ebd0dcc4c61521 FAIL (2026-01-16 17:04:01 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001377935s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........................................................XXX.] duration: 21.357625397s trace: https://jaeger.dev.storj.io/trace/521596a1b50467d9 FAIL (2026-01-16 17:03:52 UTC) orbiter warning: context deadline exceeded Duration: 21.357654502s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................................XX............] duration: 5.726903931s trace: https://jaeger.dev.storj.io/trace/ff1eaafc0d95989 FAIL (2026-01-16 17:03:48 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 16 16:56:06.586316499 (25h0m3.480025108s) [upload=21929]: Starting upload Jan 16 16:56:06.586371244 (25h0m3.480079853s) [upload=21929]: Done starting upload Jan 16 16:56:06.586529134 (25h0m3.480237763s) [upload=21929]: Uploading segments... Jan 16 16:56:06.586687636 (25h0m3.480396275s) [upload=21929]: Got next segment. Inline: false Jan 16 16:56:06.586750094 (25h0m3.480458723s) [upload=21929]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 16 16:56:06.586757138 (25h0m3.480465767s) [upload=21929]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 16 16:56:08.120364931 (25h0m5.01407354s) [seg_pos={0 0}, upload=21929]: Begin upload segment... Jan 16 16:56:08.120413474 (25h0m5.014122083s) [seg_pos={0 0}, upload=21929]: Done begin upload segment. Jan 16 16:56:08.120440615 (25h0m5.014149224s) [noise=true , address=145.79.227.2:4115 , piece_id=YMME5WUV7XGQZOE2I5H7UWQ7TSTP2LGV7D4AZO4T62XOWBE4ZEKA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.120799800 (25h0m5.014508419s) [noise=true , address=145.79.227.32:4835 , piece_id=6PP7J34OCTKYEDEFQJWDUHO5H4Z6VG7LNG3SL7WLITSKMX43HAYQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.121695454 (25h0m5.015404084s) [noise=true , address=45.91.116.4:4937 , piece_id=MJHRB2I4KWRJJ3MHGW657D53JFDARTHOBRUEHQZEUA4DRFVK7ZUA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.122317739 (25h0m5.016026349s) [noise=true , address=145.79.227.30:4762 , piece_id=YLBYKHJNJ2J3YYVYHRWEZTFLXZVL5WCKCBWXM526XF4KKCYA4RAQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.123161636 (25h0m5.016870245s) [noise=true , address=45.91.116.14:4190 , piece_id=ZBC7C5E5V7B4VAGI2JHD7T4E6BGVV74YNNAROZYBITS2N23BJDWA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.123395732 (25h0m5.017104351s) [noise=true , address=145.79.227.28:4727 , piece_id=U5CDDUXQZ2BB7SPFAIWQXRGALJQTOWFNWGJLU3CGXKBUHYIQBIGQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.123848494 (25h0m5.017557113s) [noise=true , address=77.90.31.19:4842 , piece_id=3VJ3J4JQPSJEVFSNSYDVHRFZMMCBMNZRBYEGJOOU4Q4HFEDDOKHQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.123868092 (25h0m5.017576731s) [noise=true , address=145.79.227.15:4873 , piece_id=EVJGQ5JNJI37QIPQDJPFOPMT3JAHPBZXQG7DBRUZXJMKXAIADWSA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.124249808 (25h0m5.017958417s) [noise=true , address=145.79.227.38:4977 , piece_id=JXV6MGOVSAA2URSIF2KFFQJJEGWQWFVT65PWVDW5MEH5ITSDYFAA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.124389284 (25h0m5.018097893s) [noise=true , address=145.79.227.49:4339 , piece_id=NWDULUWHMV4PIMWMNO5XYXSAGVLNPLSIPYSDA32OZB2MBGBYPEUA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.124619222 (25h0m5.018327831s) [noise=true , address=145.79.227.33:4870 , piece_id=7I3KU7EKC2MHBJ6TOGO5ZBMJSPYU4E26D5373M2R53IZAZJLWVOA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.124999566 (25h0m5.018708175s) [noise=true , address=145.79.227.3:4151 , piece_id=QVFK6RJITYJIW6ZJRUYFU3P2NEMMVANA36O6WXPS7Z7FNFSYQWUA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.125202182 (25h0m5.018910801s) [noise=true , address=77.90.31.13:4627 , piece_id=TJFFMEVYN7M52FH5UP3WNAGC7HWF5GSEUPQJZ5KJWTJZWKZNAWXQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.125576805 (25h0m5.019285414s) [noise=true , address=45.91.116.5:4300 , piece_id=M2F4W7IGAEKEXJXZRENON55A2E7DNSCZTPKWYJFY3NKNTBIWG2YA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.125952470 (25h0m5.019661079s) [noise=true , address=145.79.227.10:4401 , piece_id=C2X52SPGSNM2CB6TFGBCT6HO3MXSEMX3CCZQASDNFGWIEW2EGIGQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.126492900 (25h0m5.020201509s) [noise=true , address=145.79.227.34:4906 , piece_id=G2GHF4S5NVJP5R43EN6NDEJ4FCPCLSL5K3TO4UADJ6ZANME54KOQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:10.730578935 (25h0m7.624287544s) [upload=21929]: Next returned nil segment Jan 16 16:56:10.730594134 (25h0m7.624302753s) [upload=21929]: Waiting for error group managing segments... Jan 16 16:56:11.329683279 (25h0m8.223391888s) [noise=true , address=145.79.227.15:4873 , piece_id=EVJGQ5JNJI37QIPQDJPFOPMT3JAHPBZXQG7DBRUZXJMKXAIADWSA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.338893335 (25h0m8.232601944s) [noise=true , address=45.91.116.14:4190 , piece_id=ZBC7C5E5V7B4VAGI2JHD7T4E6BGVV74YNNAROZYBITS2N23BJDWA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.400223651 (25h0m8.29393226s) [noise=true , address=145.79.227.33:4870 , piece_id=7I3KU7EKC2MHBJ6TOGO5ZBMJSPYU4E26D5373M2R53IZAZJLWVOA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.400340974 (25h0m8.294049583s) [noise=true , address=145.79.227.2:4115 , piece_id=YMME5WUV7XGQZOE2I5H7UWQ7TSTP2LGV7D4AZO4T62XOWBE4ZEKA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.445450294 (25h0m8.339158904s) [noise=true , address=145.79.227.30:4762 , piece_id=YLBYKHJNJ2J3YYVYHRWEZTFLXZVL5WCKCBWXM526XF4KKCYA4RAQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.964879212 (25h0m8.858587821s) [noise=true , address=145.79.227.3:4151 , piece_id=QVFK6RJITYJIW6ZJRUYFU3P2NEMMVANA36O6WXPS7Z7FNFSYQWUA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.027795357 (25h0m8.921503966s) [noise=true , address=145.79.227.49:4339 , piece_id=NWDULUWHMV4PIMWMNO5XYXSAGVLNPLSIPYSDA32OZB2MBGBYPEUA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.267543450 (25h0m9.161252059s) [noise=true , address=145.79.227.28:4727 , piece_id=U5CDDUXQZ2BB7SPFAIWQXRGALJQTOWFNWGJLU3CGXKBUHYIQBIGQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.337643945 (25h0m9.231352554s) [noise=true , address=145.79.227.10:4401 , piece_id=C2X52SPGSNM2CB6TFGBCT6HO3MXSEMX3CCZQASDNFGWIEW2EGIGQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.380001967 (25h0m9.273710576s) [noise=true , address=145.79.227.32:4835 , piece_id=6PP7J34OCTKYEDEFQJWDUHO5H4Z6VG7LNG3SL7WLITSKMX43HAYQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.850164228 (25h0m9.743872837s) [noise=true , address=77.90.31.13:4627 , piece_id=TJFFMEVYN7M52FH5UP3WNAGC7HWF5GSEUPQJZ5KJWTJZWKZNAWXQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:13.183873887 (25h0m10.077582496s) [noise=true , address=77.90.31.19:4842 , piece_id=3VJ3J4JQPSJEVFSNSYDVHRFZMMCBMNZRBYEGJOOU4Q4HFEDDOKHQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:13.361354387 (25h0m10.255063026s) [noise=true , address=45.91.116.4:4937 , piece_id=MJHRB2I4KWRJJ3MHGW657D53JFDARTHOBRUEHQZEUA4DRFVK7ZUA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:13.361383082 (25h0m10.255091691s) [seg_pos={0 0}, upload=21929]: Satisfied cohort requirements after 13 pieces Jan 16 16:56:13.361615564 (25h0m10.255324173s) [noise=true , address=145.79.227.34:4906 , piece_id=G2GHF4S5NVJP5R43EN6NDEJ4FCPCLSL5K3TO4UADJ6ZANME54KOQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Jan 16 16:56:13.361624061 (25h0m10.25533267s) [noise=true , address=145.79.227.34:4906 , piece_id=G2GHF4S5NVJP5R43EN6NDEJ4FCPCLSL5K3TO4UADJ6ZANME54KOQ, seg_pos={0 0}, upload=21929]: Long tail cancellation due to optimal threshold Jan 16 16:56:13.361904004 (25h0m10.255612623s) [noise=true , address=145.79.227.38:4977 , piece_id=JXV6MGOVSAA2URSIF2KFFQJJEGWQWFVT65PWVDW5MEH5ITSDYFAA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8): protocol: read tcp 188.166.225.170:42244->145.79.227.38:4977: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 16 16:56:13.361925344 (25h0m10.255633963s) [noise=true , address=145.79.227.38:4977 , piece_id=JXV6MGOVSAA2URSIF2KFFQJJEGWQWFVT65PWVDW5MEH5ITSDYFAA, seg_pos={0 0}, upload=21929]: Long tail cancellation due to optimal threshold Jan 16 16:56:13.361971022 (25h0m10.255679631s) [noise=true , address=45.91.116.5:4300 , piece_id=M2F4W7IGAEKEXJXZRENON55A2E7DNSCZTPKWYJFY3NKNTBIWG2YA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Jan 16 16:56:13.361978977 (25h0m10.255687596s) [noise=true , address=45.91.116.5:4300 , piece_id=M2F4W7IGAEKEXJXZRENON55A2E7DNSCZTPKWYJFY3NKNTBIWG2YA, seg_pos={0 0}, upload=21929]: Long tail cancellation due to optimal threshold Jan 16 16:56:13.362041817 (25h0m10.255750426s) [seg_pos={0 0}, upload=21929]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 16 16:56:13.362197272 (25h0m10.255905881s) [upload=21929]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 16 16:56:13.362205868 (25h0m10.255914477s) [upload=21929]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 16 16:56:13.965177888 (25h0m10.858886497s) [upload=21929]: Done uploading segments... Jan 16 16:56:13.965204158 (25h0m10.858912767s) [upload=21929]: Upload finished. err: <nil> Duration: 5.726931403s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 53 out of 60 is failed (threshold=6) [XXXXXX.XXXX.XXXXXXXXXXXXXXXXXXXX.XXXXXX.X..XXXXXXX.XXXXXXXXX] duration: 5.001104174s trace: https://jaeger.dev.storj.io/trace/6b5904aaaad99f20 FAIL (2026-01-16 17:03:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001126867s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................................XX...........] duration: 5.821057085s trace: https://jaeger.dev.storj.io/trace/ff1eaafc0d95989 FAIL (2026-01-16 17:03:29 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 16 16:56:06.586316499 (25h0m3.480025108s) [upload=21929]: Starting upload Jan 16 16:56:06.586371244 (25h0m3.480079853s) [upload=21929]: Done starting upload Jan 16 16:56:06.586529134 (25h0m3.480237763s) [upload=21929]: Uploading segments... Jan 16 16:56:06.586687636 (25h0m3.480396275s) [upload=21929]: Got next segment. Inline: false Jan 16 16:56:06.586750094 (25h0m3.480458723s) [upload=21929]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 16 16:56:06.586757138 (25h0m3.480465767s) [upload=21929]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 16 16:56:08.120364931 (25h0m5.01407354s) [seg_pos={0 0}, upload=21929]: Begin upload segment... Jan 16 16:56:08.120413474 (25h0m5.014122083s) [seg_pos={0 0}, upload=21929]: Done begin upload segment. Jan 16 16:56:08.120440615 (25h0m5.014149224s) [noise=true , address=145.79.227.2:4115 , piece_id=YMME5WUV7XGQZOE2I5H7UWQ7TSTP2LGV7D4AZO4T62XOWBE4ZEKA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.120799800 (25h0m5.014508419s) [noise=true , address=145.79.227.32:4835 , piece_id=6PP7J34OCTKYEDEFQJWDUHO5H4Z6VG7LNG3SL7WLITSKMX43HAYQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.121695454 (25h0m5.015404084s) [noise=true , address=45.91.116.4:4937 , piece_id=MJHRB2I4KWRJJ3MHGW657D53JFDARTHOBRUEHQZEUA4DRFVK7ZUA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.122317739 (25h0m5.016026349s) [noise=true , address=145.79.227.30:4762 , piece_id=YLBYKHJNJ2J3YYVYHRWEZTFLXZVL5WCKCBWXM526XF4KKCYA4RAQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.123161636 (25h0m5.016870245s) [noise=true , address=45.91.116.14:4190 , piece_id=ZBC7C5E5V7B4VAGI2JHD7T4E6BGVV74YNNAROZYBITS2N23BJDWA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.123395732 (25h0m5.017104351s) [noise=true , address=145.79.227.28:4727 , piece_id=U5CDDUXQZ2BB7SPFAIWQXRGALJQTOWFNWGJLU3CGXKBUHYIQBIGQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.123848494 (25h0m5.017557113s) [noise=true , address=77.90.31.19:4842 , piece_id=3VJ3J4JQPSJEVFSNSYDVHRFZMMCBMNZRBYEGJOOU4Q4HFEDDOKHQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.123868092 (25h0m5.017576731s) [noise=true , address=145.79.227.15:4873 , piece_id=EVJGQ5JNJI37QIPQDJPFOPMT3JAHPBZXQG7DBRUZXJMKXAIADWSA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.124249808 (25h0m5.017958417s) [noise=true , address=145.79.227.38:4977 , piece_id=JXV6MGOVSAA2URSIF2KFFQJJEGWQWFVT65PWVDW5MEH5ITSDYFAA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.124389284 (25h0m5.018097893s) [noise=true , address=145.79.227.49:4339 , piece_id=NWDULUWHMV4PIMWMNO5XYXSAGVLNPLSIPYSDA32OZB2MBGBYPEUA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.124619222 (25h0m5.018327831s) [noise=true , address=145.79.227.33:4870 , piece_id=7I3KU7EKC2MHBJ6TOGO5ZBMJSPYU4E26D5373M2R53IZAZJLWVOA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.124999566 (25h0m5.018708175s) [noise=true , address=145.79.227.3:4151 , piece_id=QVFK6RJITYJIW6ZJRUYFU3P2NEMMVANA36O6WXPS7Z7FNFSYQWUA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.125202182 (25h0m5.018910801s) [noise=true , address=77.90.31.13:4627 , piece_id=TJFFMEVYN7M52FH5UP3WNAGC7HWF5GSEUPQJZ5KJWTJZWKZNAWXQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.125576805 (25h0m5.019285414s) [noise=true , address=45.91.116.5:4300 , piece_id=M2F4W7IGAEKEXJXZRENON55A2E7DNSCZTPKWYJFY3NKNTBIWG2YA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.125952470 (25h0m5.019661079s) [noise=true , address=145.79.227.10:4401 , piece_id=C2X52SPGSNM2CB6TFGBCT6HO3MXSEMX3CCZQASDNFGWIEW2EGIGQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.126492900 (25h0m5.020201509s) [noise=true , address=145.79.227.34:4906 , piece_id=G2GHF4S5NVJP5R43EN6NDEJ4FCPCLSL5K3TO4UADJ6ZANME54KOQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:10.730578935 (25h0m7.624287544s) [upload=21929]: Next returned nil segment Jan 16 16:56:10.730594134 (25h0m7.624302753s) [upload=21929]: Waiting for error group managing segments... Jan 16 16:56:11.329683279 (25h0m8.223391888s) [noise=true , address=145.79.227.15:4873 , piece_id=EVJGQ5JNJI37QIPQDJPFOPMT3JAHPBZXQG7DBRUZXJMKXAIADWSA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.338893335 (25h0m8.232601944s) [noise=true , address=45.91.116.14:4190 , piece_id=ZBC7C5E5V7B4VAGI2JHD7T4E6BGVV74YNNAROZYBITS2N23BJDWA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.400223651 (25h0m8.29393226s) [noise=true , address=145.79.227.33:4870 , piece_id=7I3KU7EKC2MHBJ6TOGO5ZBMJSPYU4E26D5373M2R53IZAZJLWVOA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.400340974 (25h0m8.294049583s) [noise=true , address=145.79.227.2:4115 , piece_id=YMME5WUV7XGQZOE2I5H7UWQ7TSTP2LGV7D4AZO4T62XOWBE4ZEKA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.445450294 (25h0m8.339158904s) [noise=true , address=145.79.227.30:4762 , piece_id=YLBYKHJNJ2J3YYVYHRWEZTFLXZVL5WCKCBWXM526XF4KKCYA4RAQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.964879212 (25h0m8.858587821s) [noise=true , address=145.79.227.3:4151 , piece_id=QVFK6RJITYJIW6ZJRUYFU3P2NEMMVANA36O6WXPS7Z7FNFSYQWUA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.027795357 (25h0m8.921503966s) [noise=true , address=145.79.227.49:4339 , piece_id=NWDULUWHMV4PIMWMNO5XYXSAGVLNPLSIPYSDA32OZB2MBGBYPEUA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.267543450 (25h0m9.161252059s) [noise=true , address=145.79.227.28:4727 , piece_id=U5CDDUXQZ2BB7SPFAIWQXRGALJQTOWFNWGJLU3CGXKBUHYIQBIGQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.337643945 (25h0m9.231352554s) [noise=true , address=145.79.227.10:4401 , piece_id=C2X52SPGSNM2CB6TFGBCT6HO3MXSEMX3CCZQASDNFGWIEW2EGIGQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.380001967 (25h0m9.273710576s) [noise=true , address=145.79.227.32:4835 , piece_id=6PP7J34OCTKYEDEFQJWDUHO5H4Z6VG7LNG3SL7WLITSKMX43HAYQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.850164228 (25h0m9.743872837s) [noise=true , address=77.90.31.13:4627 , piece_id=TJFFMEVYN7M52FH5UP3WNAGC7HWF5GSEUPQJZ5KJWTJZWKZNAWXQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:13.183873887 (25h0m10.077582496s) [noise=true , address=77.90.31.19:4842 , piece_id=3VJ3J4JQPSJEVFSNSYDVHRFZMMCBMNZRBYEGJOOU4Q4HFEDDOKHQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:13.361354387 (25h0m10.255063026s) [noise=true , address=45.91.116.4:4937 , piece_id=MJHRB2I4KWRJJ3MHGW657D53JFDARTHOBRUEHQZEUA4DRFVK7ZUA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:13.361383082 (25h0m10.255091691s) [seg_pos={0 0}, upload=21929]: Satisfied cohort requirements after 13 pieces Jan 16 16:56:13.361615564 (25h0m10.255324173s) [noise=true , address=145.79.227.34:4906 , piece_id=G2GHF4S5NVJP5R43EN6NDEJ4FCPCLSL5K3TO4UADJ6ZANME54KOQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Jan 16 16:56:13.361624061 (25h0m10.25533267s) [noise=true , address=145.79.227.34:4906 , piece_id=G2GHF4S5NVJP5R43EN6NDEJ4FCPCLSL5K3TO4UADJ6ZANME54KOQ, seg_pos={0 0}, upload=21929]: Long tail cancellation due to optimal threshold Jan 16 16:56:13.361904004 (25h0m10.255612623s) [noise=true , address=145.79.227.38:4977 , piece_id=JXV6MGOVSAA2URSIF2KFFQJJEGWQWFVT65PWVDW5MEH5ITSDYFAA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8): protocol: read tcp 188.166.225.170:42244->145.79.227.38:4977: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 16 16:56:13.361925344 (25h0m10.255633963s) [noise=true , address=145.79.227.38:4977 , piece_id=JXV6MGOVSAA2URSIF2KFFQJJEGWQWFVT65PWVDW5MEH5ITSDYFAA, seg_pos={0 0}, upload=21929]: Long tail cancellation due to optimal threshold Jan 16 16:56:13.361971022 (25h0m10.255679631s) [noise=true , address=45.91.116.5:4300 , piece_id=M2F4W7IGAEKEXJXZRENON55A2E7DNSCZTPKWYJFY3NKNTBIWG2YA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Jan 16 16:56:13.361978977 (25h0m10.255687596s) [noise=true , address=45.91.116.5:4300 , piece_id=M2F4W7IGAEKEXJXZRENON55A2E7DNSCZTPKWYJFY3NKNTBIWG2YA, seg_pos={0 0}, upload=21929]: Long tail cancellation due to optimal threshold Jan 16 16:56:13.362041817 (25h0m10.255750426s) [seg_pos={0 0}, upload=21929]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 16 16:56:13.362197272 (25h0m10.255905881s) [upload=21929]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 16 16:56:13.362205868 (25h0m10.255914477s) [upload=21929]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 16 16:56:13.965177888 (25h0m10.858886497s) [upload=21929]: Done uploading segments... Jan 16 16:56:13.965204158 (25h0m10.858912767s) [upload=21929]: Upload finished. err: <nil> Duration: 5.821084577s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........................................................XXX] duration: 30.001528613s trace: https://jaeger.dev.storj.io/trace/521596a1b50467d9 FAIL (2026-01-16 17:03:22 UTC) orbiter warning: context deadline exceeded Duration: 30.001549842s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...................X.......................................X] duration: 1m0.001578175s trace: https://jaeger.dev.storj.io/trace/614fd9a52b38870a FAIL (2026-01-16 17:03:20 UTC) orbiter warning: context deadline exceeded Duration: 1m0.001599474s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................................XX..........] duration: 5.794441146s trace: https://jaeger.dev.storj.io/trace/ff1eaafc0d95989 FAIL (2026-01-16 17:02:50 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 16 16:56:06.586316499 (25h0m3.480025108s) [upload=21929]: Starting upload Jan 16 16:56:06.586371244 (25h0m3.480079853s) [upload=21929]: Done starting upload Jan 16 16:56:06.586529134 (25h0m3.480237763s) [upload=21929]: Uploading segments... Jan 16 16:56:06.586687636 (25h0m3.480396275s) [upload=21929]: Got next segment. Inline: false Jan 16 16:56:06.586750094 (25h0m3.480458723s) [upload=21929]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 16 16:56:06.586757138 (25h0m3.480465767s) [upload=21929]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 16 16:56:08.120364931 (25h0m5.01407354s) [seg_pos={0 0}, upload=21929]: Begin upload segment... Jan 16 16:56:08.120413474 (25h0m5.014122083s) [seg_pos={0 0}, upload=21929]: Done begin upload segment. Jan 16 16:56:08.120440615 (25h0m5.014149224s) [noise=true , address=145.79.227.2:4115 , piece_id=YMME5WUV7XGQZOE2I5H7UWQ7TSTP2LGV7D4AZO4T62XOWBE4ZEKA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.120799800 (25h0m5.014508419s) [noise=true , address=145.79.227.32:4835 , piece_id=6PP7J34OCTKYEDEFQJWDUHO5H4Z6VG7LNG3SL7WLITSKMX43HAYQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.121695454 (25h0m5.015404084s) [noise=true , address=45.91.116.4:4937 , piece_id=MJHRB2I4KWRJJ3MHGW657D53JFDARTHOBRUEHQZEUA4DRFVK7ZUA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.122317739 (25h0m5.016026349s) [noise=true , address=145.79.227.30:4762 , piece_id=YLBYKHJNJ2J3YYVYHRWEZTFLXZVL5WCKCBWXM526XF4KKCYA4RAQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.123161636 (25h0m5.016870245s) [noise=true , address=45.91.116.14:4190 , piece_id=ZBC7C5E5V7B4VAGI2JHD7T4E6BGVV74YNNAROZYBITS2N23BJDWA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.123395732 (25h0m5.017104351s) [noise=true , address=145.79.227.28:4727 , piece_id=U5CDDUXQZ2BB7SPFAIWQXRGALJQTOWFNWGJLU3CGXKBUHYIQBIGQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.123848494 (25h0m5.017557113s) [noise=true , address=77.90.31.19:4842 , piece_id=3VJ3J4JQPSJEVFSNSYDVHRFZMMCBMNZRBYEGJOOU4Q4HFEDDOKHQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.123868092 (25h0m5.017576731s) [noise=true , address=145.79.227.15:4873 , piece_id=EVJGQ5JNJI37QIPQDJPFOPMT3JAHPBZXQG7DBRUZXJMKXAIADWSA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.124249808 (25h0m5.017958417s) [noise=true , address=145.79.227.38:4977 , piece_id=JXV6MGOVSAA2URSIF2KFFQJJEGWQWFVT65PWVDW5MEH5ITSDYFAA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.124389284 (25h0m5.018097893s) [noise=true , address=145.79.227.49:4339 , piece_id=NWDULUWHMV4PIMWMNO5XYXSAGVLNPLSIPYSDA32OZB2MBGBYPEUA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.124619222 (25h0m5.018327831s) [noise=true , address=145.79.227.33:4870 , piece_id=7I3KU7EKC2MHBJ6TOGO5ZBMJSPYU4E26D5373M2R53IZAZJLWVOA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.124999566 (25h0m5.018708175s) [noise=true , address=145.79.227.3:4151 , piece_id=QVFK6RJITYJIW6ZJRUYFU3P2NEMMVANA36O6WXPS7Z7FNFSYQWUA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.125202182 (25h0m5.018910801s) [noise=true , address=77.90.31.13:4627 , piece_id=TJFFMEVYN7M52FH5UP3WNAGC7HWF5GSEUPQJZ5KJWTJZWKZNAWXQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.125576805 (25h0m5.019285414s) [noise=true , address=45.91.116.5:4300 , piece_id=M2F4W7IGAEKEXJXZRENON55A2E7DNSCZTPKWYJFY3NKNTBIWG2YA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.125952470 (25h0m5.019661079s) [noise=true , address=145.79.227.10:4401 , piece_id=C2X52SPGSNM2CB6TFGBCT6HO3MXSEMX3CCZQASDNFGWIEW2EGIGQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.126492900 (25h0m5.020201509s) [noise=true , address=145.79.227.34:4906 , piece_id=G2GHF4S5NVJP5R43EN6NDEJ4FCPCLSL5K3TO4UADJ6ZANME54KOQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:10.730578935 (25h0m7.624287544s) [upload=21929]: Next returned nil segment Jan 16 16:56:10.730594134 (25h0m7.624302753s) [upload=21929]: Waiting for error group managing segments... Jan 16 16:56:11.329683279 (25h0m8.223391888s) [noise=true , address=145.79.227.15:4873 , piece_id=EVJGQ5JNJI37QIPQDJPFOPMT3JAHPBZXQG7DBRUZXJMKXAIADWSA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.338893335 (25h0m8.232601944s) [noise=true , address=45.91.116.14:4190 , piece_id=ZBC7C5E5V7B4VAGI2JHD7T4E6BGVV74YNNAROZYBITS2N23BJDWA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.400223651 (25h0m8.29393226s) [noise=true , address=145.79.227.33:4870 , piece_id=7I3KU7EKC2MHBJ6TOGO5ZBMJSPYU4E26D5373M2R53IZAZJLWVOA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.400340974 (25h0m8.294049583s) [noise=true , address=145.79.227.2:4115 , piece_id=YMME5WUV7XGQZOE2I5H7UWQ7TSTP2LGV7D4AZO4T62XOWBE4ZEKA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.445450294 (25h0m8.339158904s) [noise=true , address=145.79.227.30:4762 , piece_id=YLBYKHJNJ2J3YYVYHRWEZTFLXZVL5WCKCBWXM526XF4KKCYA4RAQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.964879212 (25h0m8.858587821s) [noise=true , address=145.79.227.3:4151 , piece_id=QVFK6RJITYJIW6ZJRUYFU3P2NEMMVANA36O6WXPS7Z7FNFSYQWUA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.027795357 (25h0m8.921503966s) [noise=true , address=145.79.227.49:4339 , piece_id=NWDULUWHMV4PIMWMNO5XYXSAGVLNPLSIPYSDA32OZB2MBGBYPEUA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.267543450 (25h0m9.161252059s) [noise=true , address=145.79.227.28:4727 , piece_id=U5CDDUXQZ2BB7SPFAIWQXRGALJQTOWFNWGJLU3CGXKBUHYIQBIGQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.337643945 (25h0m9.231352554s) [noise=true , address=145.79.227.10:4401 , piece_id=C2X52SPGSNM2CB6TFGBCT6HO3MXSEMX3CCZQASDNFGWIEW2EGIGQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.380001967 (25h0m9.273710576s) [noise=true , address=145.79.227.32:4835 , piece_id=6PP7J34OCTKYEDEFQJWDUHO5H4Z6VG7LNG3SL7WLITSKMX43HAYQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.850164228 (25h0m9.743872837s) [noise=true , address=77.90.31.13:4627 , piece_id=TJFFMEVYN7M52FH5UP3WNAGC7HWF5GSEUPQJZ5KJWTJZWKZNAWXQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:13.183873887 (25h0m10.077582496s) [noise=true , address=77.90.31.19:4842 , piece_id=3VJ3J4JQPSJEVFSNSYDVHRFZMMCBMNZRBYEGJOOU4Q4HFEDDOKHQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:13.361354387 (25h0m10.255063026s) [noise=true , address=45.91.116.4:4937 , piece_id=MJHRB2I4KWRJJ3MHGW657D53JFDARTHOBRUEHQZEUA4DRFVK7ZUA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:13.361383082 (25h0m10.255091691s) [seg_pos={0 0}, upload=21929]: Satisfied cohort requirements after 13 pieces Jan 16 16:56:13.361615564 (25h0m10.255324173s) [noise=true , address=145.79.227.34:4906 , piece_id=G2GHF4S5NVJP5R43EN6NDEJ4FCPCLSL5K3TO4UADJ6ZANME54KOQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Jan 16 16:56:13.361624061 (25h0m10.25533267s) [noise=true , address=145.79.227.34:4906 , piece_id=G2GHF4S5NVJP5R43EN6NDEJ4FCPCLSL5K3TO4UADJ6ZANME54KOQ, seg_pos={0 0}, upload=21929]: Long tail cancellation due to optimal threshold Jan 16 16:56:13.361904004 (25h0m10.255612623s) [noise=true , address=145.79.227.38:4977 , piece_id=JXV6MGOVSAA2URSIF2KFFQJJEGWQWFVT65PWVDW5MEH5ITSDYFAA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8): protocol: read tcp 188.166.225.170:42244->145.79.227.38:4977: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 16 16:56:13.361925344 (25h0m10.255633963s) [noise=true , address=145.79.227.38:4977 , piece_id=JXV6MGOVSAA2URSIF2KFFQJJEGWQWFVT65PWVDW5MEH5ITSDYFAA, seg_pos={0 0}, upload=21929]: Long tail cancellation due to optimal threshold Jan 16 16:56:13.361971022 (25h0m10.255679631s) [noise=true , address=45.91.116.5:4300 , piece_id=M2F4W7IGAEKEXJXZRENON55A2E7DNSCZTPKWYJFY3NKNTBIWG2YA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Jan 16 16:56:13.361978977 (25h0m10.255687596s) [noise=true , address=45.91.116.5:4300 , piece_id=M2F4W7IGAEKEXJXZRENON55A2E7DNSCZTPKWYJFY3NKNTBIWG2YA, seg_pos={0 0}, upload=21929]: Long tail cancellation due to optimal threshold Jan 16 16:56:13.362041817 (25h0m10.255750426s) [seg_pos={0 0}, upload=21929]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 16 16:56:13.362197272 (25h0m10.255905881s) [upload=21929]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 16 16:56:13.362205868 (25h0m10.255914477s) [upload=21929]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 16 16:56:13.965177888 (25h0m10.858886497s) [upload=21929]: Done uploading segments... Jan 16 16:56:13.965204158 (25h0m10.858912767s) [upload=21929]: Upload finished. err: <nil> Duration: 5.794468538s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 53 out of 60 is failed (threshold=6) [XXXXXXX.XXXX.XXXXXXXXXXXXXXXXXXXX.XXXXXX.X..XXXXXXX.XXXXXXXX] duration: 5.000923792s trace: https://jaeger.dev.storj.io/trace/4e741659ed782638 FAIL (2026-01-16 17:02:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000947849s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 53 out of 60 is failed (threshold=6) [XXXXXXXX.XXXX.XXXXXXXXXXXXXXXXXXXX.XXXXXX.X..XXXXXXX.XXXXXXX] duration: 5.001099041s trace: https://jaeger.dev.storj.io/trace/358827a1eafeb524 FAIL (2026-01-16 17:02:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001153625s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................................................XX] duration: 30.00128189s trace: https://jaeger.dev.storj.io/trace/666f1b2f97661797 FAIL (2026-01-16 17:02:29 UTC) orbiter warning: context deadline exceeded Duration: 30.001303791s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................................XX.........] duration: 5.7803418s trace: https://jaeger.dev.storj.io/trace/ff1eaafc0d95989 FAIL (2026-01-16 17:01:58 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 16 16:56:06.586316499 (25h0m3.480025108s) [upload=21929]: Starting upload Jan 16 16:56:06.586371244 (25h0m3.480079853s) [upload=21929]: Done starting upload Jan 16 16:56:06.586529134 (25h0m3.480237763s) [upload=21929]: Uploading segments... Jan 16 16:56:06.586687636 (25h0m3.480396275s) [upload=21929]: Got next segment. Inline: false Jan 16 16:56:06.586750094 (25h0m3.480458723s) [upload=21929]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 16 16:56:06.586757138 (25h0m3.480465767s) [upload=21929]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 16 16:56:08.120364931 (25h0m5.01407354s) [seg_pos={0 0}, upload=21929]: Begin upload segment... Jan 16 16:56:08.120413474 (25h0m5.014122083s) [seg_pos={0 0}, upload=21929]: Done begin upload segment. Jan 16 16:56:08.120440615 (25h0m5.014149224s) [noise=true , address=145.79.227.2:4115 , piece_id=YMME5WUV7XGQZOE2I5H7UWQ7TSTP2LGV7D4AZO4T62XOWBE4ZEKA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.120799800 (25h0m5.014508419s) [noise=true , address=145.79.227.32:4835 , piece_id=6PP7J34OCTKYEDEFQJWDUHO5H4Z6VG7LNG3SL7WLITSKMX43HAYQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.121695454 (25h0m5.015404084s) [noise=true , address=45.91.116.4:4937 , piece_id=MJHRB2I4KWRJJ3MHGW657D53JFDARTHOBRUEHQZEUA4DRFVK7ZUA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.122317739 (25h0m5.016026349s) [noise=true , address=145.79.227.30:4762 , piece_id=YLBYKHJNJ2J3YYVYHRWEZTFLXZVL5WCKCBWXM526XF4KKCYA4RAQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.123161636 (25h0m5.016870245s) [noise=true , address=45.91.116.14:4190 , piece_id=ZBC7C5E5V7B4VAGI2JHD7T4E6BGVV74YNNAROZYBITS2N23BJDWA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.123395732 (25h0m5.017104351s) [noise=true , address=145.79.227.28:4727 , piece_id=U5CDDUXQZ2BB7SPFAIWQXRGALJQTOWFNWGJLU3CGXKBUHYIQBIGQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.123848494 (25h0m5.017557113s) [noise=true , address=77.90.31.19:4842 , piece_id=3VJ3J4JQPSJEVFSNSYDVHRFZMMCBMNZRBYEGJOOU4Q4HFEDDOKHQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.123868092 (25h0m5.017576731s) [noise=true , address=145.79.227.15:4873 , piece_id=EVJGQ5JNJI37QIPQDJPFOPMT3JAHPBZXQG7DBRUZXJMKXAIADWSA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.124249808 (25h0m5.017958417s) [noise=true , address=145.79.227.38:4977 , piece_id=JXV6MGOVSAA2URSIF2KFFQJJEGWQWFVT65PWVDW5MEH5ITSDYFAA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.124389284 (25h0m5.018097893s) [noise=true , address=145.79.227.49:4339 , piece_id=NWDULUWHMV4PIMWMNO5XYXSAGVLNPLSIPYSDA32OZB2MBGBYPEUA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.124619222 (25h0m5.018327831s) [noise=true , address=145.79.227.33:4870 , piece_id=7I3KU7EKC2MHBJ6TOGO5ZBMJSPYU4E26D5373M2R53IZAZJLWVOA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.124999566 (25h0m5.018708175s) [noise=true , address=145.79.227.3:4151 , piece_id=QVFK6RJITYJIW6ZJRUYFU3P2NEMMVANA36O6WXPS7Z7FNFSYQWUA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.125202182 (25h0m5.018910801s) [noise=true , address=77.90.31.13:4627 , piece_id=TJFFMEVYN7M52FH5UP3WNAGC7HWF5GSEUPQJZ5KJWTJZWKZNAWXQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.125576805 (25h0m5.019285414s) [noise=true , address=45.91.116.5:4300 , piece_id=M2F4W7IGAEKEXJXZRENON55A2E7DNSCZTPKWYJFY3NKNTBIWG2YA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.125952470 (25h0m5.019661079s) [noise=true , address=145.79.227.10:4401 , piece_id=C2X52SPGSNM2CB6TFGBCT6HO3MXSEMX3CCZQASDNFGWIEW2EGIGQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.126492900 (25h0m5.020201509s) [noise=true , address=145.79.227.34:4906 , piece_id=G2GHF4S5NVJP5R43EN6NDEJ4FCPCLSL5K3TO4UADJ6ZANME54KOQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:10.730578935 (25h0m7.624287544s) [upload=21929]: Next returned nil segment Jan 16 16:56:10.730594134 (25h0m7.624302753s) [upload=21929]: Waiting for error group managing segments... Jan 16 16:56:11.329683279 (25h0m8.223391888s) [noise=true , address=145.79.227.15:4873 , piece_id=EVJGQ5JNJI37QIPQDJPFOPMT3JAHPBZXQG7DBRUZXJMKXAIADWSA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.338893335 (25h0m8.232601944s) [noise=true , address=45.91.116.14:4190 , piece_id=ZBC7C5E5V7B4VAGI2JHD7T4E6BGVV74YNNAROZYBITS2N23BJDWA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.400223651 (25h0m8.29393226s) [noise=true , address=145.79.227.33:4870 , piece_id=7I3KU7EKC2MHBJ6TOGO5ZBMJSPYU4E26D5373M2R53IZAZJLWVOA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.400340974 (25h0m8.294049583s) [noise=true , address=145.79.227.2:4115 , piece_id=YMME5WUV7XGQZOE2I5H7UWQ7TSTP2LGV7D4AZO4T62XOWBE4ZEKA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.445450294 (25h0m8.339158904s) [noise=true , address=145.79.227.30:4762 , piece_id=YLBYKHJNJ2J3YYVYHRWEZTFLXZVL5WCKCBWXM526XF4KKCYA4RAQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.964879212 (25h0m8.858587821s) [noise=true , address=145.79.227.3:4151 , piece_id=QVFK6RJITYJIW6ZJRUYFU3P2NEMMVANA36O6WXPS7Z7FNFSYQWUA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.027795357 (25h0m8.921503966s) [noise=true , address=145.79.227.49:4339 , piece_id=NWDULUWHMV4PIMWMNO5XYXSAGVLNPLSIPYSDA32OZB2MBGBYPEUA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.267543450 (25h0m9.161252059s) [noise=true , address=145.79.227.28:4727 , piece_id=U5CDDUXQZ2BB7SPFAIWQXRGALJQTOWFNWGJLU3CGXKBUHYIQBIGQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.337643945 (25h0m9.231352554s) [noise=true , address=145.79.227.10:4401 , piece_id=C2X52SPGSNM2CB6TFGBCT6HO3MXSEMX3CCZQASDNFGWIEW2EGIGQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.380001967 (25h0m9.273710576s) [noise=true , address=145.79.227.32:4835 , piece_id=6PP7J34OCTKYEDEFQJWDUHO5H4Z6VG7LNG3SL7WLITSKMX43HAYQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.850164228 (25h0m9.743872837s) [noise=true , address=77.90.31.13:4627 , piece_id=TJFFMEVYN7M52FH5UP3WNAGC7HWF5GSEUPQJZ5KJWTJZWKZNAWXQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:13.183873887 (25h0m10.077582496s) [noise=true , address=77.90.31.19:4842 , piece_id=3VJ3J4JQPSJEVFSNSYDVHRFZMMCBMNZRBYEGJOOU4Q4HFEDDOKHQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:13.361354387 (25h0m10.255063026s) [noise=true , address=45.91.116.4:4937 , piece_id=MJHRB2I4KWRJJ3MHGW657D53JFDARTHOBRUEHQZEUA4DRFVK7ZUA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:13.361383082 (25h0m10.255091691s) [seg_pos={0 0}, upload=21929]: Satisfied cohort requirements after 13 pieces Jan 16 16:56:13.361615564 (25h0m10.255324173s) [noise=true , address=145.79.227.34:4906 , piece_id=G2GHF4S5NVJP5R43EN6NDEJ4FCPCLSL5K3TO4UADJ6ZANME54KOQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Jan 16 16:56:13.361624061 (25h0m10.25533267s) [noise=true , address=145.79.227.34:4906 , piece_id=G2GHF4S5NVJP5R43EN6NDEJ4FCPCLSL5K3TO4UADJ6ZANME54KOQ, seg_pos={0 0}, upload=21929]: Long tail cancellation due to optimal threshold Jan 16 16:56:13.361904004 (25h0m10.255612623s) [noise=true , address=145.79.227.38:4977 , piece_id=JXV6MGOVSAA2URSIF2KFFQJJEGWQWFVT65PWVDW5MEH5ITSDYFAA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8): protocol: read tcp 188.166.225.170:42244->145.79.227.38:4977: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 16 16:56:13.361925344 (25h0m10.255633963s) [noise=true , address=145.79.227.38:4977 , piece_id=JXV6MGOVSAA2URSIF2KFFQJJEGWQWFVT65PWVDW5MEH5ITSDYFAA, seg_pos={0 0}, upload=21929]: Long tail cancellation due to optimal threshold Jan 16 16:56:13.361971022 (25h0m10.255679631s) [noise=true , address=45.91.116.5:4300 , piece_id=M2F4W7IGAEKEXJXZRENON55A2E7DNSCZTPKWYJFY3NKNTBIWG2YA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Jan 16 16:56:13.361978977 (25h0m10.255687596s) [noise=true , address=45.91.116.5:4300 , piece_id=M2F4W7IGAEKEXJXZRENON55A2E7DNSCZTPKWYJFY3NKNTBIWG2YA, seg_pos={0 0}, upload=21929]: Long tail cancellation due to optimal threshold Jan 16 16:56:13.362041817 (25h0m10.255750426s) [seg_pos={0 0}, upload=21929]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 16 16:56:13.362197272 (25h0m10.255905881s) [upload=21929]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 16 16:56:13.362205868 (25h0m10.255914477s) [upload=21929]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 16 16:56:13.965177888 (25h0m10.858886497s) [upload=21929]: Done uploading segments... Jan 16 16:56:13.965204158 (25h0m10.858912767s) [upload=21929]: Upload finished. err: <nil> Duration: 5.7803677s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 52 out of 60 is failed (threshold=6) [.XXXXXXXX.XXXX.XXXXXXXXXXXXXXXXXXXX.XXXXXX.X..XXXXXXX.XXXXXX] duration: 5.000929301s trace: https://jaeger.dev.storj.io/trace/69e086aa7783dc79 FAIL (2026-01-16 17:01:52 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000948427s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.000822575s trace: https://jaeger.dev.storj.io/trace/5fce5ac58f74dbc8 FAIL (2026-01-16 17:01:38 UTC) orbiter warning: context deadline exceeded Duration: 30.000849437s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 24.145596352s trace: https://jaeger.dev.storj.io/trace/7c21759b617df8f1 FAIL (2026-01-16 17:01:37 UTC) orbiter warning: context deadline exceeded Duration: 24.145624504s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 52 out of 60 is failed (threshold=6) [X.XXXXXXXX.XXXX.XXXXXXXXXXXXXXXXXXXX.XXXXXX.X..XXXXXXX.XXXXX] duration: 5.001247074s trace: https://jaeger.dev.storj.io/trace/61ae1235f298ff4 FAIL (2026-01-16 17:01:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001268526s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................................XX........] duration: 5.784821749s trace: https://jaeger.dev.storj.io/trace/ff1eaafc0d95989 FAIL (2026-01-16 17:00:49 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 16 16:56:06.586316499 (25h0m3.480025108s) [upload=21929]: Starting upload Jan 16 16:56:06.586371244 (25h0m3.480079853s) [upload=21929]: Done starting upload Jan 16 16:56:06.586529134 (25h0m3.480237763s) [upload=21929]: Uploading segments... Jan 16 16:56:06.586687636 (25h0m3.480396275s) [upload=21929]: Got next segment. Inline: false Jan 16 16:56:06.586750094 (25h0m3.480458723s) [upload=21929]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 16 16:56:06.586757138 (25h0m3.480465767s) [upload=21929]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 16 16:56:08.120364931 (25h0m5.01407354s) [seg_pos={0 0}, upload=21929]: Begin upload segment... Jan 16 16:56:08.120413474 (25h0m5.014122083s) [seg_pos={0 0}, upload=21929]: Done begin upload segment. Jan 16 16:56:08.120440615 (25h0m5.014149224s) [noise=true , address=145.79.227.2:4115 , piece_id=YMME5WUV7XGQZOE2I5H7UWQ7TSTP2LGV7D4AZO4T62XOWBE4ZEKA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.120799800 (25h0m5.014508419s) [noise=true , address=145.79.227.32:4835 , piece_id=6PP7J34OCTKYEDEFQJWDUHO5H4Z6VG7LNG3SL7WLITSKMX43HAYQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.121695454 (25h0m5.015404084s) [noise=true , address=45.91.116.4:4937 , piece_id=MJHRB2I4KWRJJ3MHGW657D53JFDARTHOBRUEHQZEUA4DRFVK7ZUA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.122317739 (25h0m5.016026349s) [noise=true , address=145.79.227.30:4762 , piece_id=YLBYKHJNJ2J3YYVYHRWEZTFLXZVL5WCKCBWXM526XF4KKCYA4RAQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.123161636 (25h0m5.016870245s) [noise=true , address=45.91.116.14:4190 , piece_id=ZBC7C5E5V7B4VAGI2JHD7T4E6BGVV74YNNAROZYBITS2N23BJDWA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.123395732 (25h0m5.017104351s) [noise=true , address=145.79.227.28:4727 , piece_id=U5CDDUXQZ2BB7SPFAIWQXRGALJQTOWFNWGJLU3CGXKBUHYIQBIGQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.123848494 (25h0m5.017557113s) [noise=true , address=77.90.31.19:4842 , piece_id=3VJ3J4JQPSJEVFSNSYDVHRFZMMCBMNZRBYEGJOOU4Q4HFEDDOKHQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.123868092 (25h0m5.017576731s) [noise=true , address=145.79.227.15:4873 , piece_id=EVJGQ5JNJI37QIPQDJPFOPMT3JAHPBZXQG7DBRUZXJMKXAIADWSA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.124249808 (25h0m5.017958417s) [noise=true , address=145.79.227.38:4977 , piece_id=JXV6MGOVSAA2URSIF2KFFQJJEGWQWFVT65PWVDW5MEH5ITSDYFAA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.124389284 (25h0m5.018097893s) [noise=true , address=145.79.227.49:4339 , piece_id=NWDULUWHMV4PIMWMNO5XYXSAGVLNPLSIPYSDA32OZB2MBGBYPEUA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.124619222 (25h0m5.018327831s) [noise=true , address=145.79.227.33:4870 , piece_id=7I3KU7EKC2MHBJ6TOGO5ZBMJSPYU4E26D5373M2R53IZAZJLWVOA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.124999566 (25h0m5.018708175s) [noise=true , address=145.79.227.3:4151 , piece_id=QVFK6RJITYJIW6ZJRUYFU3P2NEMMVANA36O6WXPS7Z7FNFSYQWUA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.125202182 (25h0m5.018910801s) [noise=true , address=77.90.31.13:4627 , piece_id=TJFFMEVYN7M52FH5UP3WNAGC7HWF5GSEUPQJZ5KJWTJZWKZNAWXQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.125576805 (25h0m5.019285414s) [noise=true , address=45.91.116.5:4300 , piece_id=M2F4W7IGAEKEXJXZRENON55A2E7DNSCZTPKWYJFY3NKNTBIWG2YA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.125952470 (25h0m5.019661079s) [noise=true , address=145.79.227.10:4401 , piece_id=C2X52SPGSNM2CB6TFGBCT6HO3MXSEMX3CCZQASDNFGWIEW2EGIGQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.126492900 (25h0m5.020201509s) [noise=true , address=145.79.227.34:4906 , piece_id=G2GHF4S5NVJP5R43EN6NDEJ4FCPCLSL5K3TO4UADJ6ZANME54KOQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:10.730578935 (25h0m7.624287544s) [upload=21929]: Next returned nil segment Jan 16 16:56:10.730594134 (25h0m7.624302753s) [upload=21929]: Waiting for error group managing segments... Jan 16 16:56:11.329683279 (25h0m8.223391888s) [noise=true , address=145.79.227.15:4873 , piece_id=EVJGQ5JNJI37QIPQDJPFOPMT3JAHPBZXQG7DBRUZXJMKXAIADWSA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.338893335 (25h0m8.232601944s) [noise=true , address=45.91.116.14:4190 , piece_id=ZBC7C5E5V7B4VAGI2JHD7T4E6BGVV74YNNAROZYBITS2N23BJDWA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.400223651 (25h0m8.29393226s) [noise=true , address=145.79.227.33:4870 , piece_id=7I3KU7EKC2MHBJ6TOGO5ZBMJSPYU4E26D5373M2R53IZAZJLWVOA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.400340974 (25h0m8.294049583s) [noise=true , address=145.79.227.2:4115 , piece_id=YMME5WUV7XGQZOE2I5H7UWQ7TSTP2LGV7D4AZO4T62XOWBE4ZEKA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.445450294 (25h0m8.339158904s) [noise=true , address=145.79.227.30:4762 , piece_id=YLBYKHJNJ2J3YYVYHRWEZTFLXZVL5WCKCBWXM526XF4KKCYA4RAQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.964879212 (25h0m8.858587821s) [noise=true , address=145.79.227.3:4151 , piece_id=QVFK6RJITYJIW6ZJRUYFU3P2NEMMVANA36O6WXPS7Z7FNFSYQWUA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.027795357 (25h0m8.921503966s) [noise=true , address=145.79.227.49:4339 , piece_id=NWDULUWHMV4PIMWMNO5XYXSAGVLNPLSIPYSDA32OZB2MBGBYPEUA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.267543450 (25h0m9.161252059s) [noise=true , address=145.79.227.28:4727 , piece_id=U5CDDUXQZ2BB7SPFAIWQXRGALJQTOWFNWGJLU3CGXKBUHYIQBIGQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.337643945 (25h0m9.231352554s) [noise=true , address=145.79.227.10:4401 , piece_id=C2X52SPGSNM2CB6TFGBCT6HO3MXSEMX3CCZQASDNFGWIEW2EGIGQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.380001967 (25h0m9.273710576s) [noise=true , address=145.79.227.32:4835 , piece_id=6PP7J34OCTKYEDEFQJWDUHO5H4Z6VG7LNG3SL7WLITSKMX43HAYQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.850164228 (25h0m9.743872837s) [noise=true , address=77.90.31.13:4627 , piece_id=TJFFMEVYN7M52FH5UP3WNAGC7HWF5GSEUPQJZ5KJWTJZWKZNAWXQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:13.183873887 (25h0m10.077582496s) [noise=true , address=77.90.31.19:4842 , piece_id=3VJ3J4JQPSJEVFSNSYDVHRFZMMCBMNZRBYEGJOOU4Q4HFEDDOKHQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:13.361354387 (25h0m10.255063026s) [noise=true , address=45.91.116.4:4937 , piece_id=MJHRB2I4KWRJJ3MHGW657D53JFDARTHOBRUEHQZEUA4DRFVK7ZUA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:13.361383082 (25h0m10.255091691s) [seg_pos={0 0}, upload=21929]: Satisfied cohort requirements after 13 pieces Jan 16 16:56:13.361615564 (25h0m10.255324173s) [noise=true , address=145.79.227.34:4906 , piece_id=G2GHF4S5NVJP5R43EN6NDEJ4FCPCLSL5K3TO4UADJ6ZANME54KOQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Jan 16 16:56:13.361624061 (25h0m10.25533267s) [noise=true , address=145.79.227.34:4906 , piece_id=G2GHF4S5NVJP5R43EN6NDEJ4FCPCLSL5K3TO4UADJ6ZANME54KOQ, seg_pos={0 0}, upload=21929]: Long tail cancellation due to optimal threshold Jan 16 16:56:13.361904004 (25h0m10.255612623s) [noise=true , address=145.79.227.38:4977 , piece_id=JXV6MGOVSAA2URSIF2KFFQJJEGWQWFVT65PWVDW5MEH5ITSDYFAA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8): protocol: read tcp 188.166.225.170:42244->145.79.227.38:4977: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 16 16:56:13.361925344 (25h0m10.255633963s) [noise=true , address=145.79.227.38:4977 , piece_id=JXV6MGOVSAA2URSIF2KFFQJJEGWQWFVT65PWVDW5MEH5ITSDYFAA, seg_pos={0 0}, upload=21929]: Long tail cancellation due to optimal threshold Jan 16 16:56:13.361971022 (25h0m10.255679631s) [noise=true , address=45.91.116.5:4300 , piece_id=M2F4W7IGAEKEXJXZRENON55A2E7DNSCZTPKWYJFY3NKNTBIWG2YA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Jan 16 16:56:13.361978977 (25h0m10.255687596s) [noise=true , address=45.91.116.5:4300 , piece_id=M2F4W7IGAEKEXJXZRENON55A2E7DNSCZTPKWYJFY3NKNTBIWG2YA, seg_pos={0 0}, upload=21929]: Long tail cancellation due to optimal threshold Jan 16 16:56:13.362041817 (25h0m10.255750426s) [seg_pos={0 0}, upload=21929]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 16 16:56:13.362197272 (25h0m10.255905881s) [upload=21929]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 16 16:56:13.362205868 (25h0m10.255914477s) [upload=21929]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 16 16:56:13.965177888 (25h0m10.858886497s) [upload=21929]: Done uploading segments... Jan 16 16:56:13.965204158 (25h0m10.858912767s) [upload=21929]: Upload finished. err: <nil> Duration: 5.784852597s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 52 out of 60 is failed (threshold=6) [XX.XXXXXXXX.XXXX.XXXXXXXXXXXXXXXXXXXX.XXXXXX.X..XXXXXXX.XXXX] duration: 5.001737429s trace: https://jaeger.dev.storj.io/trace/69e9b2d526681fbd FAIL (2026-01-16 17:00:42 UTC) orbiter warning: context deadline exceeded Duration: 5.001761885s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 29.217275069s trace: https://jaeger.dev.storj.io/trace/10b8cc1148adf877 FAIL (2026-01-16 17:00:36 UTC) orbiter warning: context deadline exceeded Duration: 29.217299777s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [X....................X......................................] duration: 16.765543939s trace: https://jaeger.dev.storj.io/trace/7c21759b617df8f1 FAIL (2026-01-16 17:00:30 UTC) orbiter warning: context deadline exceeded Duration: 16.76557614s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 51 out of 60 is failed (threshold=6) [.XX.XXXXXXXX.XXXX.XXXXXXXXXXXXXXXXXXXX.XXXXXX.X..XXXXXXX.XXX] duration: 5.002745607s trace: https://jaeger.dev.storj.io/trace/6666f2c88f47f446 FAIL (2026-01-16 17:00:21 UTC) orbiter warning: context deadline exceeded Duration: 5.002768831s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................................XX.......] duration: 6.104620001s trace: https://jaeger.dev.storj.io/trace/ff1eaafc0d95989 FAIL (2026-01-16 17:00:04 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 16 16:56:06.586316499 (25h0m3.480025108s) [upload=21929]: Starting upload Jan 16 16:56:06.586371244 (25h0m3.480079853s) [upload=21929]: Done starting upload Jan 16 16:56:06.586529134 (25h0m3.480237763s) [upload=21929]: Uploading segments... Jan 16 16:56:06.586687636 (25h0m3.480396275s) [upload=21929]: Got next segment. Inline: false Jan 16 16:56:06.586750094 (25h0m3.480458723s) [upload=21929]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 16 16:56:06.586757138 (25h0m3.480465767s) [upload=21929]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 16 16:56:08.120364931 (25h0m5.01407354s) [seg_pos={0 0}, upload=21929]: Begin upload segment... Jan 16 16:56:08.120413474 (25h0m5.014122083s) [seg_pos={0 0}, upload=21929]: Done begin upload segment. Jan 16 16:56:08.120440615 (25h0m5.014149224s) [noise=true , address=145.79.227.2:4115 , piece_id=YMME5WUV7XGQZOE2I5H7UWQ7TSTP2LGV7D4AZO4T62XOWBE4ZEKA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.120799800 (25h0m5.014508419s) [noise=true , address=145.79.227.32:4835 , piece_id=6PP7J34OCTKYEDEFQJWDUHO5H4Z6VG7LNG3SL7WLITSKMX43HAYQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.121695454 (25h0m5.015404084s) [noise=true , address=45.91.116.4:4937 , piece_id=MJHRB2I4KWRJJ3MHGW657D53JFDARTHOBRUEHQZEUA4DRFVK7ZUA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.122317739 (25h0m5.016026349s) [noise=true , address=145.79.227.30:4762 , piece_id=YLBYKHJNJ2J3YYVYHRWEZTFLXZVL5WCKCBWXM526XF4KKCYA4RAQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.123161636 (25h0m5.016870245s) [noise=true , address=45.91.116.14:4190 , piece_id=ZBC7C5E5V7B4VAGI2JHD7T4E6BGVV74YNNAROZYBITS2N23BJDWA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.123395732 (25h0m5.017104351s) [noise=true , address=145.79.227.28:4727 , piece_id=U5CDDUXQZ2BB7SPFAIWQXRGALJQTOWFNWGJLU3CGXKBUHYIQBIGQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.123848494 (25h0m5.017557113s) [noise=true , address=77.90.31.19:4842 , piece_id=3VJ3J4JQPSJEVFSNSYDVHRFZMMCBMNZRBYEGJOOU4Q4HFEDDOKHQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.123868092 (25h0m5.017576731s) [noise=true , address=145.79.227.15:4873 , piece_id=EVJGQ5JNJI37QIPQDJPFOPMT3JAHPBZXQG7DBRUZXJMKXAIADWSA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.124249808 (25h0m5.017958417s) [noise=true , address=145.79.227.38:4977 , piece_id=JXV6MGOVSAA2URSIF2KFFQJJEGWQWFVT65PWVDW5MEH5ITSDYFAA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.124389284 (25h0m5.018097893s) [noise=true , address=145.79.227.49:4339 , piece_id=NWDULUWHMV4PIMWMNO5XYXSAGVLNPLSIPYSDA32OZB2MBGBYPEUA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.124619222 (25h0m5.018327831s) [noise=true , address=145.79.227.33:4870 , piece_id=7I3KU7EKC2MHBJ6TOGO5ZBMJSPYU4E26D5373M2R53IZAZJLWVOA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.124999566 (25h0m5.018708175s) [noise=true , address=145.79.227.3:4151 , piece_id=QVFK6RJITYJIW6ZJRUYFU3P2NEMMVANA36O6WXPS7Z7FNFSYQWUA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.125202182 (25h0m5.018910801s) [noise=true , address=77.90.31.13:4627 , piece_id=TJFFMEVYN7M52FH5UP3WNAGC7HWF5GSEUPQJZ5KJWTJZWKZNAWXQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.125576805 (25h0m5.019285414s) [noise=true , address=45.91.116.5:4300 , piece_id=M2F4W7IGAEKEXJXZRENON55A2E7DNSCZTPKWYJFY3NKNTBIWG2YA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.125952470 (25h0m5.019661079s) [noise=true , address=145.79.227.10:4401 , piece_id=C2X52SPGSNM2CB6TFGBCT6HO3MXSEMX3CCZQASDNFGWIEW2EGIGQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.126492900 (25h0m5.020201509s) [noise=true , address=145.79.227.34:4906 , piece_id=G2GHF4S5NVJP5R43EN6NDEJ4FCPCLSL5K3TO4UADJ6ZANME54KOQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:10.730578935 (25h0m7.624287544s) [upload=21929]: Next returned nil segment Jan 16 16:56:10.730594134 (25h0m7.624302753s) [upload=21929]: Waiting for error group managing segments... Jan 16 16:56:11.329683279 (25h0m8.223391888s) [noise=true , address=145.79.227.15:4873 , piece_id=EVJGQ5JNJI37QIPQDJPFOPMT3JAHPBZXQG7DBRUZXJMKXAIADWSA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.338893335 (25h0m8.232601944s) [noise=true , address=45.91.116.14:4190 , piece_id=ZBC7C5E5V7B4VAGI2JHD7T4E6BGVV74YNNAROZYBITS2N23BJDWA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.400223651 (25h0m8.29393226s) [noise=true , address=145.79.227.33:4870 , piece_id=7I3KU7EKC2MHBJ6TOGO5ZBMJSPYU4E26D5373M2R53IZAZJLWVOA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.400340974 (25h0m8.294049583s) [noise=true , address=145.79.227.2:4115 , piece_id=YMME5WUV7XGQZOE2I5H7UWQ7TSTP2LGV7D4AZO4T62XOWBE4ZEKA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.445450294 (25h0m8.339158904s) [noise=true , address=145.79.227.30:4762 , piece_id=YLBYKHJNJ2J3YYVYHRWEZTFLXZVL5WCKCBWXM526XF4KKCYA4RAQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.964879212 (25h0m8.858587821s) [noise=true , address=145.79.227.3:4151 , piece_id=QVFK6RJITYJIW6ZJRUYFU3P2NEMMVANA36O6WXPS7Z7FNFSYQWUA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.027795357 (25h0m8.921503966s) [noise=true , address=145.79.227.49:4339 , piece_id=NWDULUWHMV4PIMWMNO5XYXSAGVLNPLSIPYSDA32OZB2MBGBYPEUA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.267543450 (25h0m9.161252059s) [noise=true , address=145.79.227.28:4727 , piece_id=U5CDDUXQZ2BB7SPFAIWQXRGALJQTOWFNWGJLU3CGXKBUHYIQBIGQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.337643945 (25h0m9.231352554s) [noise=true , address=145.79.227.10:4401 , piece_id=C2X52SPGSNM2CB6TFGBCT6HO3MXSEMX3CCZQASDNFGWIEW2EGIGQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.380001967 (25h0m9.273710576s) [noise=true , address=145.79.227.32:4835 , piece_id=6PP7J34OCTKYEDEFQJWDUHO5H4Z6VG7LNG3SL7WLITSKMX43HAYQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.850164228 (25h0m9.743872837s) [noise=true , address=77.90.31.13:4627 , piece_id=TJFFMEVYN7M52FH5UP3WNAGC7HWF5GSEUPQJZ5KJWTJZWKZNAWXQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:13.183873887 (25h0m10.077582496s) [noise=true , address=77.90.31.19:4842 , piece_id=3VJ3J4JQPSJEVFSNSYDVHRFZMMCBMNZRBYEGJOOU4Q4HFEDDOKHQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:13.361354387 (25h0m10.255063026s) [noise=true , address=45.91.116.4:4937 , piece_id=MJHRB2I4KWRJJ3MHGW657D53JFDARTHOBRUEHQZEUA4DRFVK7ZUA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:13.361383082 (25h0m10.255091691s) [seg_pos={0 0}, upload=21929]: Satisfied cohort requirements after 13 pieces Jan 16 16:56:13.361615564 (25h0m10.255324173s) [noise=true , address=145.79.227.34:4906 , piece_id=G2GHF4S5NVJP5R43EN6NDEJ4FCPCLSL5K3TO4UADJ6ZANME54KOQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Jan 16 16:56:13.361624061 (25h0m10.25533267s) [noise=true , address=145.79.227.34:4906 , piece_id=G2GHF4S5NVJP5R43EN6NDEJ4FCPCLSL5K3TO4UADJ6ZANME54KOQ, seg_pos={0 0}, upload=21929]: Long tail cancellation due to optimal threshold Jan 16 16:56:13.361904004 (25h0m10.255612623s) [noise=true , address=145.79.227.38:4977 , piece_id=JXV6MGOVSAA2URSIF2KFFQJJEGWQWFVT65PWVDW5MEH5ITSDYFAA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8): protocol: read tcp 188.166.225.170:42244->145.79.227.38:4977: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 16 16:56:13.361925344 (25h0m10.255633963s) [noise=true , address=145.79.227.38:4977 , piece_id=JXV6MGOVSAA2URSIF2KFFQJJEGWQWFVT65PWVDW5MEH5ITSDYFAA, seg_pos={0 0}, upload=21929]: Long tail cancellation due to optimal threshold Jan 16 16:56:13.361971022 (25h0m10.255679631s) [noise=true , address=45.91.116.5:4300 , piece_id=M2F4W7IGAEKEXJXZRENON55A2E7DNSCZTPKWYJFY3NKNTBIWG2YA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Jan 16 16:56:13.361978977 (25h0m10.255687596s) [noise=true , address=45.91.116.5:4300 , piece_id=M2F4W7IGAEKEXJXZRENON55A2E7DNSCZTPKWYJFY3NKNTBIWG2YA, seg_pos={0 0}, upload=21929]: Long tail cancellation due to optimal threshold Jan 16 16:56:13.362041817 (25h0m10.255750426s) [seg_pos={0 0}, upload=21929]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 16 16:56:13.362197272 (25h0m10.255905881s) [upload=21929]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 16 16:56:13.362205868 (25h0m10.255914477s) [upload=21929]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 16 16:56:13.965177888 (25h0m10.858886497s) [upload=21929]: Done uploading segments... Jan 16 16:56:13.965204158 (25h0m10.858912767s) [upload=21929]: Upload finished. err: <nil> Duration: 6.104654967s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................................XX......] duration: 5.830400123s trace: https://jaeger.dev.storj.io/trace/ff1eaafc0d95989 FAIL (2026-01-16 16:59:48 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 16 16:56:06.586316499 (25h0m3.480025108s) [upload=21929]: Starting upload Jan 16 16:56:06.586371244 (25h0m3.480079853s) [upload=21929]: Done starting upload Jan 16 16:56:06.586529134 (25h0m3.480237763s) [upload=21929]: Uploading segments... Jan 16 16:56:06.586687636 (25h0m3.480396275s) [upload=21929]: Got next segment. Inline: false Jan 16 16:56:06.586750094 (25h0m3.480458723s) [upload=21929]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 16 16:56:06.586757138 (25h0m3.480465767s) [upload=21929]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 16 16:56:08.120364931 (25h0m5.01407354s) [seg_pos={0 0}, upload=21929]: Begin upload segment... Jan 16 16:56:08.120413474 (25h0m5.014122083s) [seg_pos={0 0}, upload=21929]: Done begin upload segment. Jan 16 16:56:08.120440615 (25h0m5.014149224s) [noise=true , address=145.79.227.2:4115 , piece_id=YMME5WUV7XGQZOE2I5H7UWQ7TSTP2LGV7D4AZO4T62XOWBE4ZEKA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.120799800 (25h0m5.014508419s) [noise=true , address=145.79.227.32:4835 , piece_id=6PP7J34OCTKYEDEFQJWDUHO5H4Z6VG7LNG3SL7WLITSKMX43HAYQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.121695454 (25h0m5.015404084s) [noise=true , address=45.91.116.4:4937 , piece_id=MJHRB2I4KWRJJ3MHGW657D53JFDARTHOBRUEHQZEUA4DRFVK7ZUA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.122317739 (25h0m5.016026349s) [noise=true , address=145.79.227.30:4762 , piece_id=YLBYKHJNJ2J3YYVYHRWEZTFLXZVL5WCKCBWXM526XF4KKCYA4RAQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.123161636 (25h0m5.016870245s) [noise=true , address=45.91.116.14:4190 , piece_id=ZBC7C5E5V7B4VAGI2JHD7T4E6BGVV74YNNAROZYBITS2N23BJDWA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.123395732 (25h0m5.017104351s) [noise=true , address=145.79.227.28:4727 , piece_id=U5CDDUXQZ2BB7SPFAIWQXRGALJQTOWFNWGJLU3CGXKBUHYIQBIGQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.123848494 (25h0m5.017557113s) [noise=true , address=77.90.31.19:4842 , piece_id=3VJ3J4JQPSJEVFSNSYDVHRFZMMCBMNZRBYEGJOOU4Q4HFEDDOKHQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.123868092 (25h0m5.017576731s) [noise=true , address=145.79.227.15:4873 , piece_id=EVJGQ5JNJI37QIPQDJPFOPMT3JAHPBZXQG7DBRUZXJMKXAIADWSA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.124249808 (25h0m5.017958417s) [noise=true , address=145.79.227.38:4977 , piece_id=JXV6MGOVSAA2URSIF2KFFQJJEGWQWFVT65PWVDW5MEH5ITSDYFAA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.124389284 (25h0m5.018097893s) [noise=true , address=145.79.227.49:4339 , piece_id=NWDULUWHMV4PIMWMNO5XYXSAGVLNPLSIPYSDA32OZB2MBGBYPEUA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.124619222 (25h0m5.018327831s) [noise=true , address=145.79.227.33:4870 , piece_id=7I3KU7EKC2MHBJ6TOGO5ZBMJSPYU4E26D5373M2R53IZAZJLWVOA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.124999566 (25h0m5.018708175s) [noise=true , address=145.79.227.3:4151 , piece_id=QVFK6RJITYJIW6ZJRUYFU3P2NEMMVANA36O6WXPS7Z7FNFSYQWUA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.125202182 (25h0m5.018910801s) [noise=true , address=77.90.31.13:4627 , piece_id=TJFFMEVYN7M52FH5UP3WNAGC7HWF5GSEUPQJZ5KJWTJZWKZNAWXQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.125576805 (25h0m5.019285414s) [noise=true , address=45.91.116.5:4300 , piece_id=M2F4W7IGAEKEXJXZRENON55A2E7DNSCZTPKWYJFY3NKNTBIWG2YA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.125952470 (25h0m5.019661079s) [noise=true , address=145.79.227.10:4401 , piece_id=C2X52SPGSNM2CB6TFGBCT6HO3MXSEMX3CCZQASDNFGWIEW2EGIGQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.126492900 (25h0m5.020201509s) [noise=true , address=145.79.227.34:4906 , piece_id=G2GHF4S5NVJP5R43EN6NDEJ4FCPCLSL5K3TO4UADJ6ZANME54KOQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:10.730578935 (25h0m7.624287544s) [upload=21929]: Next returned nil segment Jan 16 16:56:10.730594134 (25h0m7.624302753s) [upload=21929]: Waiting for error group managing segments... Jan 16 16:56:11.329683279 (25h0m8.223391888s) [noise=true , address=145.79.227.15:4873 , piece_id=EVJGQ5JNJI37QIPQDJPFOPMT3JAHPBZXQG7DBRUZXJMKXAIADWSA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.338893335 (25h0m8.232601944s) [noise=true , address=45.91.116.14:4190 , piece_id=ZBC7C5E5V7B4VAGI2JHD7T4E6BGVV74YNNAROZYBITS2N23BJDWA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.400223651 (25h0m8.29393226s) [noise=true , address=145.79.227.33:4870 , piece_id=7I3KU7EKC2MHBJ6TOGO5ZBMJSPYU4E26D5373M2R53IZAZJLWVOA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.400340974 (25h0m8.294049583s) [noise=true , address=145.79.227.2:4115 , piece_id=YMME5WUV7XGQZOE2I5H7UWQ7TSTP2LGV7D4AZO4T62XOWBE4ZEKA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.445450294 (25h0m8.339158904s) [noise=true , address=145.79.227.30:4762 , piece_id=YLBYKHJNJ2J3YYVYHRWEZTFLXZVL5WCKCBWXM526XF4KKCYA4RAQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.964879212 (25h0m8.858587821s) [noise=true , address=145.79.227.3:4151 , piece_id=QVFK6RJITYJIW6ZJRUYFU3P2NEMMVANA36O6WXPS7Z7FNFSYQWUA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.027795357 (25h0m8.921503966s) [noise=true , address=145.79.227.49:4339 , piece_id=NWDULUWHMV4PIMWMNO5XYXSAGVLNPLSIPYSDA32OZB2MBGBYPEUA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.267543450 (25h0m9.161252059s) [noise=true , address=145.79.227.28:4727 , piece_id=U5CDDUXQZ2BB7SPFAIWQXRGALJQTOWFNWGJLU3CGXKBUHYIQBIGQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.337643945 (25h0m9.231352554s) [noise=true , address=145.79.227.10:4401 , piece_id=C2X52SPGSNM2CB6TFGBCT6HO3MXSEMX3CCZQASDNFGWIEW2EGIGQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.380001967 (25h0m9.273710576s) [noise=true , address=145.79.227.32:4835 , piece_id=6PP7J34OCTKYEDEFQJWDUHO5H4Z6VG7LNG3SL7WLITSKMX43HAYQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.850164228 (25h0m9.743872837s) [noise=true , address=77.90.31.13:4627 , piece_id=TJFFMEVYN7M52FH5UP3WNAGC7HWF5GSEUPQJZ5KJWTJZWKZNAWXQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:13.183873887 (25h0m10.077582496s) [noise=true , address=77.90.31.19:4842 , piece_id=3VJ3J4JQPSJEVFSNSYDVHRFZMMCBMNZRBYEGJOOU4Q4HFEDDOKHQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:13.361354387 (25h0m10.255063026s) [noise=true , address=45.91.116.4:4937 , piece_id=MJHRB2I4KWRJJ3MHGW657D53JFDARTHOBRUEHQZEUA4DRFVK7ZUA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:13.361383082 (25h0m10.255091691s) [seg_pos={0 0}, upload=21929]: Satisfied cohort requirements after 13 pieces Jan 16 16:56:13.361615564 (25h0m10.255324173s) [noise=true , address=145.79.227.34:4906 , piece_id=G2GHF4S5NVJP5R43EN6NDEJ4FCPCLSL5K3TO4UADJ6ZANME54KOQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Jan 16 16:56:13.361624061 (25h0m10.25533267s) [noise=true , address=145.79.227.34:4906 , piece_id=G2GHF4S5NVJP5R43EN6NDEJ4FCPCLSL5K3TO4UADJ6ZANME54KOQ, seg_pos={0 0}, upload=21929]: Long tail cancellation due to optimal threshold Jan 16 16:56:13.361904004 (25h0m10.255612623s) [noise=true , address=145.79.227.38:4977 , piece_id=JXV6MGOVSAA2URSIF2KFFQJJEGWQWFVT65PWVDW5MEH5ITSDYFAA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8): protocol: read tcp 188.166.225.170:42244->145.79.227.38:4977: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 16 16:56:13.361925344 (25h0m10.255633963s) [noise=true , address=145.79.227.38:4977 , piece_id=JXV6MGOVSAA2URSIF2KFFQJJEGWQWFVT65PWVDW5MEH5ITSDYFAA, seg_pos={0 0}, upload=21929]: Long tail cancellation due to optimal threshold Jan 16 16:56:13.361971022 (25h0m10.255679631s) [noise=true , address=45.91.116.5:4300 , piece_id=M2F4W7IGAEKEXJXZRENON55A2E7DNSCZTPKWYJFY3NKNTBIWG2YA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Jan 16 16:56:13.361978977 (25h0m10.255687596s) [noise=true , address=45.91.116.5:4300 , piece_id=M2F4W7IGAEKEXJXZRENON55A2E7DNSCZTPKWYJFY3NKNTBIWG2YA, seg_pos={0 0}, upload=21929]: Long tail cancellation due to optimal threshold Jan 16 16:56:13.362041817 (25h0m10.255750426s) [seg_pos={0 0}, upload=21929]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 16 16:56:13.362197272 (25h0m10.255905881s) [upload=21929]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 16 16:56:13.362205868 (25h0m10.255914477s) [upload=21929]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 16 16:56:13.965177888 (25h0m10.858886497s) [upload=21929]: Done uploading segments... Jan 16 16:56:13.965204158 (25h0m10.858912767s) [upload=21929]: Upload finished. err: <nil> Duration: 5.830444656s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 12.391337164s trace: https://jaeger.dev.storj.io/trace/10b8cc1148adf877 FAIL (2026-01-16 16:59:35 UTC) orbiter warning: context deadline exceeded Duration: 12.39136167s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 51 out of 60 is failed (threshold=6) [X.XX.XXXXXXXX.XXXX.XXXXXXXXXXXXXXXXXXXX.XXXXXX.X..XXXXXXX.XX] duration: 5.000936875s trace: https://jaeger.dev.storj.io/trace/4a7344f0b75eb063 FAIL (2026-01-16 16:59:30 UTC) orbiter warning: context deadline exceeded Duration: 5.001083886s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.X....................X.....................................] duration: 20.325097726s trace: https://jaeger.dev.storj.io/trace/7c21759b617df8f1 FAIL (2026-01-16 16:59:21 UTC) orbiter warning: context deadline exceeded Duration: 20.325111722s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................................XX.....] duration: 5.758214258s trace: https://jaeger.dev.storj.io/trace/ff1eaafc0d95989 FAIL (2026-01-16 16:59:11 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 16 16:56:06.586316499 (25h0m3.480025108s) [upload=21929]: Starting upload Jan 16 16:56:06.586371244 (25h0m3.480079853s) [upload=21929]: Done starting upload Jan 16 16:56:06.586529134 (25h0m3.480237763s) [upload=21929]: Uploading segments... Jan 16 16:56:06.586687636 (25h0m3.480396275s) [upload=21929]: Got next segment. Inline: false Jan 16 16:56:06.586750094 (25h0m3.480458723s) [upload=21929]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 16 16:56:06.586757138 (25h0m3.480465767s) [upload=21929]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 16 16:56:08.120364931 (25h0m5.01407354s) [seg_pos={0 0}, upload=21929]: Begin upload segment... Jan 16 16:56:08.120413474 (25h0m5.014122083s) [seg_pos={0 0}, upload=21929]: Done begin upload segment. Jan 16 16:56:08.120440615 (25h0m5.014149224s) [noise=true , address=145.79.227.2:4115 , piece_id=YMME5WUV7XGQZOE2I5H7UWQ7TSTP2LGV7D4AZO4T62XOWBE4ZEKA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.120799800 (25h0m5.014508419s) [noise=true , address=145.79.227.32:4835 , piece_id=6PP7J34OCTKYEDEFQJWDUHO5H4Z6VG7LNG3SL7WLITSKMX43HAYQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.121695454 (25h0m5.015404084s) [noise=true , address=45.91.116.4:4937 , piece_id=MJHRB2I4KWRJJ3MHGW657D53JFDARTHOBRUEHQZEUA4DRFVK7ZUA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.122317739 (25h0m5.016026349s) [noise=true , address=145.79.227.30:4762 , piece_id=YLBYKHJNJ2J3YYVYHRWEZTFLXZVL5WCKCBWXM526XF4KKCYA4RAQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.123161636 (25h0m5.016870245s) [noise=true , address=45.91.116.14:4190 , piece_id=ZBC7C5E5V7B4VAGI2JHD7T4E6BGVV74YNNAROZYBITS2N23BJDWA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.123395732 (25h0m5.017104351s) [noise=true , address=145.79.227.28:4727 , piece_id=U5CDDUXQZ2BB7SPFAIWQXRGALJQTOWFNWGJLU3CGXKBUHYIQBIGQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.123848494 (25h0m5.017557113s) [noise=true , address=77.90.31.19:4842 , piece_id=3VJ3J4JQPSJEVFSNSYDVHRFZMMCBMNZRBYEGJOOU4Q4HFEDDOKHQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.123868092 (25h0m5.017576731s) [noise=true , address=145.79.227.15:4873 , piece_id=EVJGQ5JNJI37QIPQDJPFOPMT3JAHPBZXQG7DBRUZXJMKXAIADWSA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.124249808 (25h0m5.017958417s) [noise=true , address=145.79.227.38:4977 , piece_id=JXV6MGOVSAA2URSIF2KFFQJJEGWQWFVT65PWVDW5MEH5ITSDYFAA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.124389284 (25h0m5.018097893s) [noise=true , address=145.79.227.49:4339 , piece_id=NWDULUWHMV4PIMWMNO5XYXSAGVLNPLSIPYSDA32OZB2MBGBYPEUA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.124619222 (25h0m5.018327831s) [noise=true , address=145.79.227.33:4870 , piece_id=7I3KU7EKC2MHBJ6TOGO5ZBMJSPYU4E26D5373M2R53IZAZJLWVOA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.124999566 (25h0m5.018708175s) [noise=true , address=145.79.227.3:4151 , piece_id=QVFK6RJITYJIW6ZJRUYFU3P2NEMMVANA36O6WXPS7Z7FNFSYQWUA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.125202182 (25h0m5.018910801s) [noise=true , address=77.90.31.13:4627 , piece_id=TJFFMEVYN7M52FH5UP3WNAGC7HWF5GSEUPQJZ5KJWTJZWKZNAWXQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.125576805 (25h0m5.019285414s) [noise=true , address=45.91.116.5:4300 , piece_id=M2F4W7IGAEKEXJXZRENON55A2E7DNSCZTPKWYJFY3NKNTBIWG2YA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.125952470 (25h0m5.019661079s) [noise=true , address=145.79.227.10:4401 , piece_id=C2X52SPGSNM2CB6TFGBCT6HO3MXSEMX3CCZQASDNFGWIEW2EGIGQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.126492900 (25h0m5.020201509s) [noise=true , address=145.79.227.34:4906 , piece_id=G2GHF4S5NVJP5R43EN6NDEJ4FCPCLSL5K3TO4UADJ6ZANME54KOQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:10.730578935 (25h0m7.624287544s) [upload=21929]: Next returned nil segment Jan 16 16:56:10.730594134 (25h0m7.624302753s) [upload=21929]: Waiting for error group managing segments... Jan 16 16:56:11.329683279 (25h0m8.223391888s) [noise=true , address=145.79.227.15:4873 , piece_id=EVJGQ5JNJI37QIPQDJPFOPMT3JAHPBZXQG7DBRUZXJMKXAIADWSA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.338893335 (25h0m8.232601944s) [noise=true , address=45.91.116.14:4190 , piece_id=ZBC7C5E5V7B4VAGI2JHD7T4E6BGVV74YNNAROZYBITS2N23BJDWA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.400223651 (25h0m8.29393226s) [noise=true , address=145.79.227.33:4870 , piece_id=7I3KU7EKC2MHBJ6TOGO5ZBMJSPYU4E26D5373M2R53IZAZJLWVOA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.400340974 (25h0m8.294049583s) [noise=true , address=145.79.227.2:4115 , piece_id=YMME5WUV7XGQZOE2I5H7UWQ7TSTP2LGV7D4AZO4T62XOWBE4ZEKA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.445450294 (25h0m8.339158904s) [noise=true , address=145.79.227.30:4762 , piece_id=YLBYKHJNJ2J3YYVYHRWEZTFLXZVL5WCKCBWXM526XF4KKCYA4RAQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.964879212 (25h0m8.858587821s) [noise=true , address=145.79.227.3:4151 , piece_id=QVFK6RJITYJIW6ZJRUYFU3P2NEMMVANA36O6WXPS7Z7FNFSYQWUA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.027795357 (25h0m8.921503966s) [noise=true , address=145.79.227.49:4339 , piece_id=NWDULUWHMV4PIMWMNO5XYXSAGVLNPLSIPYSDA32OZB2MBGBYPEUA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.267543450 (25h0m9.161252059s) [noise=true , address=145.79.227.28:4727 , piece_id=U5CDDUXQZ2BB7SPFAIWQXRGALJQTOWFNWGJLU3CGXKBUHYIQBIGQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.337643945 (25h0m9.231352554s) [noise=true , address=145.79.227.10:4401 , piece_id=C2X52SPGSNM2CB6TFGBCT6HO3MXSEMX3CCZQASDNFGWIEW2EGIGQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.380001967 (25h0m9.273710576s) [noise=true , address=145.79.227.32:4835 , piece_id=6PP7J34OCTKYEDEFQJWDUHO5H4Z6VG7LNG3SL7WLITSKMX43HAYQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.850164228 (25h0m9.743872837s) [noise=true , address=77.90.31.13:4627 , piece_id=TJFFMEVYN7M52FH5UP3WNAGC7HWF5GSEUPQJZ5KJWTJZWKZNAWXQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:13.183873887 (25h0m10.077582496s) [noise=true , address=77.90.31.19:4842 , piece_id=3VJ3J4JQPSJEVFSNSYDVHRFZMMCBMNZRBYEGJOOU4Q4HFEDDOKHQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:13.361354387 (25h0m10.255063026s) [noise=true , address=45.91.116.4:4937 , piece_id=MJHRB2I4KWRJJ3MHGW657D53JFDARTHOBRUEHQZEUA4DRFVK7ZUA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:13.361383082 (25h0m10.255091691s) [seg_pos={0 0}, upload=21929]: Satisfied cohort requirements after 13 pieces Jan 16 16:56:13.361615564 (25h0m10.255324173s) [noise=true , address=145.79.227.34:4906 , piece_id=G2GHF4S5NVJP5R43EN6NDEJ4FCPCLSL5K3TO4UADJ6ZANME54KOQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Jan 16 16:56:13.361624061 (25h0m10.25533267s) [noise=true , address=145.79.227.34:4906 , piece_id=G2GHF4S5NVJP5R43EN6NDEJ4FCPCLSL5K3TO4UADJ6ZANME54KOQ, seg_pos={0 0}, upload=21929]: Long tail cancellation due to optimal threshold Jan 16 16:56:13.361904004 (25h0m10.255612623s) [noise=true , address=145.79.227.38:4977 , piece_id=JXV6MGOVSAA2URSIF2KFFQJJEGWQWFVT65PWVDW5MEH5ITSDYFAA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8): protocol: read tcp 188.166.225.170:42244->145.79.227.38:4977: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 16 16:56:13.361925344 (25h0m10.255633963s) [noise=true , address=145.79.227.38:4977 , piece_id=JXV6MGOVSAA2URSIF2KFFQJJEGWQWFVT65PWVDW5MEH5ITSDYFAA, seg_pos={0 0}, upload=21929]: Long tail cancellation due to optimal threshold Jan 16 16:56:13.361971022 (25h0m10.255679631s) [noise=true , address=45.91.116.5:4300 , piece_id=M2F4W7IGAEKEXJXZRENON55A2E7DNSCZTPKWYJFY3NKNTBIWG2YA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Jan 16 16:56:13.361978977 (25h0m10.255687596s) [noise=true , address=45.91.116.5:4300 , piece_id=M2F4W7IGAEKEXJXZRENON55A2E7DNSCZTPKWYJFY3NKNTBIWG2YA, seg_pos={0 0}, upload=21929]: Long tail cancellation due to optimal threshold Jan 16 16:56:13.362041817 (25h0m10.255750426s) [seg_pos={0 0}, upload=21929]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 16 16:56:13.362197272 (25h0m10.255905881s) [upload=21929]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 16 16:56:13.362205868 (25h0m10.255914477s) [upload=21929]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 16 16:56:13.965177888 (25h0m10.858886497s) [upload=21929]: Done uploading segments... Jan 16 16:56:13.965204158 (25h0m10.858912767s) [upload=21929]: Upload finished. err: <nil> Duration: 5.75824095s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 51 out of 60 is failed (threshold=6) [XX.XX.XXXXXXXX.XXXX.XXXXXXXXXXXXXXXXXXXX.XXXXXX.X..XXXXXXX.X] duration: 5.000580166s trace: https://jaeger.dev.storj.io/trace/33352c4840f50af9 FAIL (2026-01-16 16:59:11 UTC) orbiter warning: context deadline exceeded Duration: 5.00060275s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 19.455326186s trace: https://jaeger.dev.storj.io/trace/10b8cc1148adf877 FAIL (2026-01-16 16:59:03 UTC) orbiter warning: context deadline exceeded Duration: 19.455458037s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................................XX....] duration: 5.72786421s trace: https://jaeger.dev.storj.io/trace/ff1eaafc0d95989 FAIL (2026-01-16 16:58:47 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 16 16:56:06.586316499 (25h0m3.480025108s) [upload=21929]: Starting upload Jan 16 16:56:06.586371244 (25h0m3.480079853s) [upload=21929]: Done starting upload Jan 16 16:56:06.586529134 (25h0m3.480237763s) [upload=21929]: Uploading segments... Jan 16 16:56:06.586687636 (25h0m3.480396275s) [upload=21929]: Got next segment. Inline: false Jan 16 16:56:06.586750094 (25h0m3.480458723s) [upload=21929]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 16 16:56:06.586757138 (25h0m3.480465767s) [upload=21929]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 16 16:56:08.120364931 (25h0m5.01407354s) [seg_pos={0 0}, upload=21929]: Begin upload segment... Jan 16 16:56:08.120413474 (25h0m5.014122083s) [seg_pos={0 0}, upload=21929]: Done begin upload segment. Jan 16 16:56:08.120440615 (25h0m5.014149224s) [noise=true , address=145.79.227.2:4115 , piece_id=YMME5WUV7XGQZOE2I5H7UWQ7TSTP2LGV7D4AZO4T62XOWBE4ZEKA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.120799800 (25h0m5.014508419s) [noise=true , address=145.79.227.32:4835 , piece_id=6PP7J34OCTKYEDEFQJWDUHO5H4Z6VG7LNG3SL7WLITSKMX43HAYQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.121695454 (25h0m5.015404084s) [noise=true , address=45.91.116.4:4937 , piece_id=MJHRB2I4KWRJJ3MHGW657D53JFDARTHOBRUEHQZEUA4DRFVK7ZUA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.122317739 (25h0m5.016026349s) [noise=true , address=145.79.227.30:4762 , piece_id=YLBYKHJNJ2J3YYVYHRWEZTFLXZVL5WCKCBWXM526XF4KKCYA4RAQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.123161636 (25h0m5.016870245s) [noise=true , address=45.91.116.14:4190 , piece_id=ZBC7C5E5V7B4VAGI2JHD7T4E6BGVV74YNNAROZYBITS2N23BJDWA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.123395732 (25h0m5.017104351s) [noise=true , address=145.79.227.28:4727 , piece_id=U5CDDUXQZ2BB7SPFAIWQXRGALJQTOWFNWGJLU3CGXKBUHYIQBIGQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.123848494 (25h0m5.017557113s) [noise=true , address=77.90.31.19:4842 , piece_id=3VJ3J4JQPSJEVFSNSYDVHRFZMMCBMNZRBYEGJOOU4Q4HFEDDOKHQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.123868092 (25h0m5.017576731s) [noise=true , address=145.79.227.15:4873 , piece_id=EVJGQ5JNJI37QIPQDJPFOPMT3JAHPBZXQG7DBRUZXJMKXAIADWSA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.124249808 (25h0m5.017958417s) [noise=true , address=145.79.227.38:4977 , piece_id=JXV6MGOVSAA2URSIF2KFFQJJEGWQWFVT65PWVDW5MEH5ITSDYFAA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.124389284 (25h0m5.018097893s) [noise=true , address=145.79.227.49:4339 , piece_id=NWDULUWHMV4PIMWMNO5XYXSAGVLNPLSIPYSDA32OZB2MBGBYPEUA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.124619222 (25h0m5.018327831s) [noise=true , address=145.79.227.33:4870 , piece_id=7I3KU7EKC2MHBJ6TOGO5ZBMJSPYU4E26D5373M2R53IZAZJLWVOA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.124999566 (25h0m5.018708175s) [noise=true , address=145.79.227.3:4151 , piece_id=QVFK6RJITYJIW6ZJRUYFU3P2NEMMVANA36O6WXPS7Z7FNFSYQWUA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.125202182 (25h0m5.018910801s) [noise=true , address=77.90.31.13:4627 , piece_id=TJFFMEVYN7M52FH5UP3WNAGC7HWF5GSEUPQJZ5KJWTJZWKZNAWXQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.125576805 (25h0m5.019285414s) [noise=true , address=45.91.116.5:4300 , piece_id=M2F4W7IGAEKEXJXZRENON55A2E7DNSCZTPKWYJFY3NKNTBIWG2YA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.125952470 (25h0m5.019661079s) [noise=true , address=145.79.227.10:4401 , piece_id=C2X52SPGSNM2CB6TFGBCT6HO3MXSEMX3CCZQASDNFGWIEW2EGIGQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.126492900 (25h0m5.020201509s) [noise=true , address=145.79.227.34:4906 , piece_id=G2GHF4S5NVJP5R43EN6NDEJ4FCPCLSL5K3TO4UADJ6ZANME54KOQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:10.730578935 (25h0m7.624287544s) [upload=21929]: Next returned nil segment Jan 16 16:56:10.730594134 (25h0m7.624302753s) [upload=21929]: Waiting for error group managing segments... Jan 16 16:56:11.329683279 (25h0m8.223391888s) [noise=true , address=145.79.227.15:4873 , piece_id=EVJGQ5JNJI37QIPQDJPFOPMT3JAHPBZXQG7DBRUZXJMKXAIADWSA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.338893335 (25h0m8.232601944s) [noise=true , address=45.91.116.14:4190 , piece_id=ZBC7C5E5V7B4VAGI2JHD7T4E6BGVV74YNNAROZYBITS2N23BJDWA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.400223651 (25h0m8.29393226s) [noise=true , address=145.79.227.33:4870 , piece_id=7I3KU7EKC2MHBJ6TOGO5ZBMJSPYU4E26D5373M2R53IZAZJLWVOA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.400340974 (25h0m8.294049583s) [noise=true , address=145.79.227.2:4115 , piece_id=YMME5WUV7XGQZOE2I5H7UWQ7TSTP2LGV7D4AZO4T62XOWBE4ZEKA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.445450294 (25h0m8.339158904s) [noise=true , address=145.79.227.30:4762 , piece_id=YLBYKHJNJ2J3YYVYHRWEZTFLXZVL5WCKCBWXM526XF4KKCYA4RAQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.964879212 (25h0m8.858587821s) [noise=true , address=145.79.227.3:4151 , piece_id=QVFK6RJITYJIW6ZJRUYFU3P2NEMMVANA36O6WXPS7Z7FNFSYQWUA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.027795357 (25h0m8.921503966s) [noise=true , address=145.79.227.49:4339 , piece_id=NWDULUWHMV4PIMWMNO5XYXSAGVLNPLSIPYSDA32OZB2MBGBYPEUA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.267543450 (25h0m9.161252059s) [noise=true , address=145.79.227.28:4727 , piece_id=U5CDDUXQZ2BB7SPFAIWQXRGALJQTOWFNWGJLU3CGXKBUHYIQBIGQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.337643945 (25h0m9.231352554s) [noise=true , address=145.79.227.10:4401 , piece_id=C2X52SPGSNM2CB6TFGBCT6HO3MXSEMX3CCZQASDNFGWIEW2EGIGQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.380001967 (25h0m9.273710576s) [noise=true , address=145.79.227.32:4835 , piece_id=6PP7J34OCTKYEDEFQJWDUHO5H4Z6VG7LNG3SL7WLITSKMX43HAYQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.850164228 (25h0m9.743872837s) [noise=true , address=77.90.31.13:4627 , piece_id=TJFFMEVYN7M52FH5UP3WNAGC7HWF5GSEUPQJZ5KJWTJZWKZNAWXQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:13.183873887 (25h0m10.077582496s) [noise=true , address=77.90.31.19:4842 , piece_id=3VJ3J4JQPSJEVFSNSYDVHRFZMMCBMNZRBYEGJOOU4Q4HFEDDOKHQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:13.361354387 (25h0m10.255063026s) [noise=true , address=45.91.116.4:4937 , piece_id=MJHRB2I4KWRJJ3MHGW657D53JFDARTHOBRUEHQZEUA4DRFVK7ZUA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:13.361383082 (25h0m10.255091691s) [seg_pos={0 0}, upload=21929]: Satisfied cohort requirements after 13 pieces Jan 16 16:56:13.361615564 (25h0m10.255324173s) [noise=true , address=145.79.227.34:4906 , piece_id=G2GHF4S5NVJP5R43EN6NDEJ4FCPCLSL5K3TO4UADJ6ZANME54KOQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Jan 16 16:56:13.361624061 (25h0m10.25533267s) [noise=true , address=145.79.227.34:4906 , piece_id=G2GHF4S5NVJP5R43EN6NDEJ4FCPCLSL5K3TO4UADJ6ZANME54KOQ, seg_pos={0 0}, upload=21929]: Long tail cancellation due to optimal threshold Jan 16 16:56:13.361904004 (25h0m10.255612623s) [noise=true , address=145.79.227.38:4977 , piece_id=JXV6MGOVSAA2URSIF2KFFQJJEGWQWFVT65PWVDW5MEH5ITSDYFAA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8): protocol: read tcp 188.166.225.170:42244->145.79.227.38:4977: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 16 16:56:13.361925344 (25h0m10.255633963s) [noise=true , address=145.79.227.38:4977 , piece_id=JXV6MGOVSAA2URSIF2KFFQJJEGWQWFVT65PWVDW5MEH5ITSDYFAA, seg_pos={0 0}, upload=21929]: Long tail cancellation due to optimal threshold Jan 16 16:56:13.361971022 (25h0m10.255679631s) [noise=true , address=45.91.116.5:4300 , piece_id=M2F4W7IGAEKEXJXZRENON55A2E7DNSCZTPKWYJFY3NKNTBIWG2YA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Jan 16 16:56:13.361978977 (25h0m10.255687596s) [noise=true , address=45.91.116.5:4300 , piece_id=M2F4W7IGAEKEXJXZRENON55A2E7DNSCZTPKWYJFY3NKNTBIWG2YA, seg_pos={0 0}, upload=21929]: Long tail cancellation due to optimal threshold Jan 16 16:56:13.362041817 (25h0m10.255750426s) [seg_pos={0 0}, upload=21929]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 16 16:56:13.362197272 (25h0m10.255905881s) [upload=21929]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 16 16:56:13.362205868 (25h0m10.255914477s) [upload=21929]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 16 16:56:13.965177888 (25h0m10.858886497s) [upload=21929]: Done uploading segments... Jan 16 16:56:13.965204158 (25h0m10.858912767s) [upload=21929]: Upload finished. err: <nil> Duration: 5.727897383s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 51 out of 60 is failed (threshold=6) [XXX.XX.XXXXXXXX.XXXX.XXXXXXXXXXXXXXXXXXXX.XXXXXX.X..XXXXXXX.] duration: 4.356384647s trace: https://jaeger.dev.storj.io/trace/15adfdd1fc6c4a9d FAIL (2026-01-16 16:58:31 UTC) orbiter warning: context deadline exceeded Duration: 4.356506639s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..X....................X....................................] duration: 13.58156378s trace: https://jaeger.dev.storj.io/trace/7c21759b617df8f1 FAIL (2026-01-16 16:58:26 UTC) orbiter warning: context deadline exceeded Duration: 13.581588167s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 52 out of 60 is failed (threshold=6) [XXXX.XX.XXXXXXXX.XXXX.XXXXXXXXXXXXXXXXXXXX.XXXXXX.X..XXXXXXX] duration: 5.001546687s trace: https://jaeger.dev.storj.io/trace/15adfdd1fc6c4a9d FAIL (2026-01-16 16:58:14 UTC) orbiter warning: context deadline exceeded Duration: 5.001567758s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...X....................X...................................] duration: 16.029650533s trace: https://jaeger.dev.storj.io/trace/7c21759b617df8f1 FAIL (2026-01-16 16:58:12 UTC) orbiter warning: context deadline exceeded Duration: 16.029676052s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 16.452565832s trace: https://jaeger.dev.storj.io/trace/10b8cc1148adf877 FAIL (2026-01-16 16:58:02 UTC) orbiter warning: context deadline exceeded Duration: 16.452592825s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 51 out of 60 is failed (threshold=6) [.XXXX.XX.XXXXXXXX.XXXX.XXXXXXXXXXXXXXXXXXXX.XXXXXX.X..XXXXXX] duration: 5.000999411s trace: https://jaeger.dev.storj.io/trace/c1ae70d5a1cb43c FAIL (2026-01-16 16:57:54 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001014659s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................................XX...] duration: 5.803574945s trace: https://jaeger.dev.storj.io/trace/ff1eaafc0d95989 FAIL (2026-01-16 16:57:47 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 16 16:56:06.586316499 (25h0m3.480025108s) [upload=21929]: Starting upload Jan 16 16:56:06.586371244 (25h0m3.480079853s) [upload=21929]: Done starting upload Jan 16 16:56:06.586529134 (25h0m3.480237763s) [upload=21929]: Uploading segments... Jan 16 16:56:06.586687636 (25h0m3.480396275s) [upload=21929]: Got next segment. Inline: false Jan 16 16:56:06.586750094 (25h0m3.480458723s) [upload=21929]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 16 16:56:06.586757138 (25h0m3.480465767s) [upload=21929]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 16 16:56:08.120364931 (25h0m5.01407354s) [seg_pos={0 0}, upload=21929]: Begin upload segment... Jan 16 16:56:08.120413474 (25h0m5.014122083s) [seg_pos={0 0}, upload=21929]: Done begin upload segment. Jan 16 16:56:08.120440615 (25h0m5.014149224s) [noise=true , address=145.79.227.2:4115 , piece_id=YMME5WUV7XGQZOE2I5H7UWQ7TSTP2LGV7D4AZO4T62XOWBE4ZEKA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.120799800 (25h0m5.014508419s) [noise=true , address=145.79.227.32:4835 , piece_id=6PP7J34OCTKYEDEFQJWDUHO5H4Z6VG7LNG3SL7WLITSKMX43HAYQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.121695454 (25h0m5.015404084s) [noise=true , address=45.91.116.4:4937 , piece_id=MJHRB2I4KWRJJ3MHGW657D53JFDARTHOBRUEHQZEUA4DRFVK7ZUA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.122317739 (25h0m5.016026349s) [noise=true , address=145.79.227.30:4762 , piece_id=YLBYKHJNJ2J3YYVYHRWEZTFLXZVL5WCKCBWXM526XF4KKCYA4RAQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.123161636 (25h0m5.016870245s) [noise=true , address=45.91.116.14:4190 , piece_id=ZBC7C5E5V7B4VAGI2JHD7T4E6BGVV74YNNAROZYBITS2N23BJDWA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.123395732 (25h0m5.017104351s) [noise=true , address=145.79.227.28:4727 , piece_id=U5CDDUXQZ2BB7SPFAIWQXRGALJQTOWFNWGJLU3CGXKBUHYIQBIGQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.123848494 (25h0m5.017557113s) [noise=true , address=77.90.31.19:4842 , piece_id=3VJ3J4JQPSJEVFSNSYDVHRFZMMCBMNZRBYEGJOOU4Q4HFEDDOKHQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.123868092 (25h0m5.017576731s) [noise=true , address=145.79.227.15:4873 , piece_id=EVJGQ5JNJI37QIPQDJPFOPMT3JAHPBZXQG7DBRUZXJMKXAIADWSA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.124249808 (25h0m5.017958417s) [noise=true , address=145.79.227.38:4977 , piece_id=JXV6MGOVSAA2URSIF2KFFQJJEGWQWFVT65PWVDW5MEH5ITSDYFAA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.124389284 (25h0m5.018097893s) [noise=true , address=145.79.227.49:4339 , piece_id=NWDULUWHMV4PIMWMNO5XYXSAGVLNPLSIPYSDA32OZB2MBGBYPEUA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.124619222 (25h0m5.018327831s) [noise=true , address=145.79.227.33:4870 , piece_id=7I3KU7EKC2MHBJ6TOGO5ZBMJSPYU4E26D5373M2R53IZAZJLWVOA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.124999566 (25h0m5.018708175s) [noise=true , address=145.79.227.3:4151 , piece_id=QVFK6RJITYJIW6ZJRUYFU3P2NEMMVANA36O6WXPS7Z7FNFSYQWUA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.125202182 (25h0m5.018910801s) [noise=true , address=77.90.31.13:4627 , piece_id=TJFFMEVYN7M52FH5UP3WNAGC7HWF5GSEUPQJZ5KJWTJZWKZNAWXQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.125576805 (25h0m5.019285414s) [noise=true , address=45.91.116.5:4300 , piece_id=M2F4W7IGAEKEXJXZRENON55A2E7DNSCZTPKWYJFY3NKNTBIWG2YA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.125952470 (25h0m5.019661079s) [noise=true , address=145.79.227.10:4401 , piece_id=C2X52SPGSNM2CB6TFGBCT6HO3MXSEMX3CCZQASDNFGWIEW2EGIGQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.126492900 (25h0m5.020201509s) [noise=true , address=145.79.227.34:4906 , piece_id=G2GHF4S5NVJP5R43EN6NDEJ4FCPCLSL5K3TO4UADJ6ZANME54KOQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:10.730578935 (25h0m7.624287544s) [upload=21929]: Next returned nil segment Jan 16 16:56:10.730594134 (25h0m7.624302753s) [upload=21929]: Waiting for error group managing segments... Jan 16 16:56:11.329683279 (25h0m8.223391888s) [noise=true , address=145.79.227.15:4873 , piece_id=EVJGQ5JNJI37QIPQDJPFOPMT3JAHPBZXQG7DBRUZXJMKXAIADWSA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.338893335 (25h0m8.232601944s) [noise=true , address=45.91.116.14:4190 , piece_id=ZBC7C5E5V7B4VAGI2JHD7T4E6BGVV74YNNAROZYBITS2N23BJDWA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.400223651 (25h0m8.29393226s) [noise=true , address=145.79.227.33:4870 , piece_id=7I3KU7EKC2MHBJ6TOGO5ZBMJSPYU4E26D5373M2R53IZAZJLWVOA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.400340974 (25h0m8.294049583s) [noise=true , address=145.79.227.2:4115 , piece_id=YMME5WUV7XGQZOE2I5H7UWQ7TSTP2LGV7D4AZO4T62XOWBE4ZEKA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.445450294 (25h0m8.339158904s) [noise=true , address=145.79.227.30:4762 , piece_id=YLBYKHJNJ2J3YYVYHRWEZTFLXZVL5WCKCBWXM526XF4KKCYA4RAQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.964879212 (25h0m8.858587821s) [noise=true , address=145.79.227.3:4151 , piece_id=QVFK6RJITYJIW6ZJRUYFU3P2NEMMVANA36O6WXPS7Z7FNFSYQWUA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.027795357 (25h0m8.921503966s) [noise=true , address=145.79.227.49:4339 , piece_id=NWDULUWHMV4PIMWMNO5XYXSAGVLNPLSIPYSDA32OZB2MBGBYPEUA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.267543450 (25h0m9.161252059s) [noise=true , address=145.79.227.28:4727 , piece_id=U5CDDUXQZ2BB7SPFAIWQXRGALJQTOWFNWGJLU3CGXKBUHYIQBIGQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.337643945 (25h0m9.231352554s) [noise=true , address=145.79.227.10:4401 , piece_id=C2X52SPGSNM2CB6TFGBCT6HO3MXSEMX3CCZQASDNFGWIEW2EGIGQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.380001967 (25h0m9.273710576s) [noise=true , address=145.79.227.32:4835 , piece_id=6PP7J34OCTKYEDEFQJWDUHO5H4Z6VG7LNG3SL7WLITSKMX43HAYQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.850164228 (25h0m9.743872837s) [noise=true , address=77.90.31.13:4627 , piece_id=TJFFMEVYN7M52FH5UP3WNAGC7HWF5GSEUPQJZ5KJWTJZWKZNAWXQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:13.183873887 (25h0m10.077582496s) [noise=true , address=77.90.31.19:4842 , piece_id=3VJ3J4JQPSJEVFSNSYDVHRFZMMCBMNZRBYEGJOOU4Q4HFEDDOKHQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:13.361354387 (25h0m10.255063026s) [noise=true , address=45.91.116.4:4937 , piece_id=MJHRB2I4KWRJJ3MHGW657D53JFDARTHOBRUEHQZEUA4DRFVK7ZUA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:13.361383082 (25h0m10.255091691s) [seg_pos={0 0}, upload=21929]: Satisfied cohort requirements after 13 pieces Jan 16 16:56:13.361615564 (25h0m10.255324173s) [noise=true , address=145.79.227.34:4906 , piece_id=G2GHF4S5NVJP5R43EN6NDEJ4FCPCLSL5K3TO4UADJ6ZANME54KOQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Jan 16 16:56:13.361624061 (25h0m10.25533267s) [noise=true , address=145.79.227.34:4906 , piece_id=G2GHF4S5NVJP5R43EN6NDEJ4FCPCLSL5K3TO4UADJ6ZANME54KOQ, seg_pos={0 0}, upload=21929]: Long tail cancellation due to optimal threshold Jan 16 16:56:13.361904004 (25h0m10.255612623s) [noise=true , address=145.79.227.38:4977 , piece_id=JXV6MGOVSAA2URSIF2KFFQJJEGWQWFVT65PWVDW5MEH5ITSDYFAA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8): protocol: read tcp 188.166.225.170:42244->145.79.227.38:4977: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 16 16:56:13.361925344 (25h0m10.255633963s) [noise=true , address=145.79.227.38:4977 , piece_id=JXV6MGOVSAA2URSIF2KFFQJJEGWQWFVT65PWVDW5MEH5ITSDYFAA, seg_pos={0 0}, upload=21929]: Long tail cancellation due to optimal threshold Jan 16 16:56:13.361971022 (25h0m10.255679631s) [noise=true , address=45.91.116.5:4300 , piece_id=M2F4W7IGAEKEXJXZRENON55A2E7DNSCZTPKWYJFY3NKNTBIWG2YA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Jan 16 16:56:13.361978977 (25h0m10.255687596s) [noise=true , address=45.91.116.5:4300 , piece_id=M2F4W7IGAEKEXJXZRENON55A2E7DNSCZTPKWYJFY3NKNTBIWG2YA, seg_pos={0 0}, upload=21929]: Long tail cancellation due to optimal threshold Jan 16 16:56:13.362041817 (25h0m10.255750426s) [seg_pos={0 0}, upload=21929]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 16 16:56:13.362197272 (25h0m10.255905881s) [upload=21929]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 16 16:56:13.362205868 (25h0m10.255914477s) [upload=21929]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 16 16:56:13.965177888 (25h0m10.858886497s) [upload=21929]: Done uploading segments... Jan 16 16:56:13.965204158 (25h0m10.858912767s) [upload=21929]: Upload finished. err: <nil> Duration: 5.803601807s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [....X....................X..................................] duration: 23.149511747s trace: https://jaeger.dev.storj.io/trace/7c21759b617df8f1 FAIL (2026-01-16 16:57:45 UTC) orbiter warning: context deadline exceeded Duration: 23.149538728s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................................................XX..] duration: 5.724469207s trace: https://jaeger.dev.storj.io/trace/ff1eaafc0d95989 FAIL (2026-01-16 16:57:09 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 16 16:56:06.586316499 (25h0m3.480025108s) [upload=21929]: Starting upload Jan 16 16:56:06.586371244 (25h0m3.480079853s) [upload=21929]: Done starting upload Jan 16 16:56:06.586529134 (25h0m3.480237763s) [upload=21929]: Uploading segments... Jan 16 16:56:06.586687636 (25h0m3.480396275s) [upload=21929]: Got next segment. Inline: false Jan 16 16:56:06.586750094 (25h0m3.480458723s) [upload=21929]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 16 16:56:06.586757138 (25h0m3.480465767s) [upload=21929]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 16 16:56:08.120364931 (25h0m5.01407354s) [seg_pos={0 0}, upload=21929]: Begin upload segment... Jan 16 16:56:08.120413474 (25h0m5.014122083s) [seg_pos={0 0}, upload=21929]: Done begin upload segment. Jan 16 16:56:08.120440615 (25h0m5.014149224s) [noise=true , address=145.79.227.2:4115 , piece_id=YMME5WUV7XGQZOE2I5H7UWQ7TSTP2LGV7D4AZO4T62XOWBE4ZEKA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.120799800 (25h0m5.014508419s) [noise=true , address=145.79.227.32:4835 , piece_id=6PP7J34OCTKYEDEFQJWDUHO5H4Z6VG7LNG3SL7WLITSKMX43HAYQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.121695454 (25h0m5.015404084s) [noise=true , address=45.91.116.4:4937 , piece_id=MJHRB2I4KWRJJ3MHGW657D53JFDARTHOBRUEHQZEUA4DRFVK7ZUA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.122317739 (25h0m5.016026349s) [noise=true , address=145.79.227.30:4762 , piece_id=YLBYKHJNJ2J3YYVYHRWEZTFLXZVL5WCKCBWXM526XF4KKCYA4RAQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.123161636 (25h0m5.016870245s) [noise=true , address=45.91.116.14:4190 , piece_id=ZBC7C5E5V7B4VAGI2JHD7T4E6BGVV74YNNAROZYBITS2N23BJDWA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.123395732 (25h0m5.017104351s) [noise=true , address=145.79.227.28:4727 , piece_id=U5CDDUXQZ2BB7SPFAIWQXRGALJQTOWFNWGJLU3CGXKBUHYIQBIGQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.123848494 (25h0m5.017557113s) [noise=true , address=77.90.31.19:4842 , piece_id=3VJ3J4JQPSJEVFSNSYDVHRFZMMCBMNZRBYEGJOOU4Q4HFEDDOKHQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.123868092 (25h0m5.017576731s) [noise=true , address=145.79.227.15:4873 , piece_id=EVJGQ5JNJI37QIPQDJPFOPMT3JAHPBZXQG7DBRUZXJMKXAIADWSA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.124249808 (25h0m5.017958417s) [noise=true , address=145.79.227.38:4977 , piece_id=JXV6MGOVSAA2URSIF2KFFQJJEGWQWFVT65PWVDW5MEH5ITSDYFAA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.124389284 (25h0m5.018097893s) [noise=true , address=145.79.227.49:4339 , piece_id=NWDULUWHMV4PIMWMNO5XYXSAGVLNPLSIPYSDA32OZB2MBGBYPEUA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.124619222 (25h0m5.018327831s) [noise=true , address=145.79.227.33:4870 , piece_id=7I3KU7EKC2MHBJ6TOGO5ZBMJSPYU4E26D5373M2R53IZAZJLWVOA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.124999566 (25h0m5.018708175s) [noise=true , address=145.79.227.3:4151 , piece_id=QVFK6RJITYJIW6ZJRUYFU3P2NEMMVANA36O6WXPS7Z7FNFSYQWUA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.125202182 (25h0m5.018910801s) [noise=true , address=77.90.31.13:4627 , piece_id=TJFFMEVYN7M52FH5UP3WNAGC7HWF5GSEUPQJZ5KJWTJZWKZNAWXQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.125576805 (25h0m5.019285414s) [noise=true , address=45.91.116.5:4300 , piece_id=M2F4W7IGAEKEXJXZRENON55A2E7DNSCZTPKWYJFY3NKNTBIWG2YA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.125952470 (25h0m5.019661079s) [noise=true , address=145.79.227.10:4401 , piece_id=C2X52SPGSNM2CB6TFGBCT6HO3MXSEMX3CCZQASDNFGWIEW2EGIGQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.126492900 (25h0m5.020201509s) [noise=true , address=145.79.227.34:4906 , piece_id=G2GHF4S5NVJP5R43EN6NDEJ4FCPCLSL5K3TO4UADJ6ZANME54KOQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:10.730578935 (25h0m7.624287544s) [upload=21929]: Next returned nil segment Jan 16 16:56:10.730594134 (25h0m7.624302753s) [upload=21929]: Waiting for error group managing segments... Jan 16 16:56:11.329683279 (25h0m8.223391888s) [noise=true , address=145.79.227.15:4873 , piece_id=EVJGQ5JNJI37QIPQDJPFOPMT3JAHPBZXQG7DBRUZXJMKXAIADWSA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.338893335 (25h0m8.232601944s) [noise=true , address=45.91.116.14:4190 , piece_id=ZBC7C5E5V7B4VAGI2JHD7T4E6BGVV74YNNAROZYBITS2N23BJDWA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.400223651 (25h0m8.29393226s) [noise=true , address=145.79.227.33:4870 , piece_id=7I3KU7EKC2MHBJ6TOGO5ZBMJSPYU4E26D5373M2R53IZAZJLWVOA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.400340974 (25h0m8.294049583s) [noise=true , address=145.79.227.2:4115 , piece_id=YMME5WUV7XGQZOE2I5H7UWQ7TSTP2LGV7D4AZO4T62XOWBE4ZEKA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.445450294 (25h0m8.339158904s) [noise=true , address=145.79.227.30:4762 , piece_id=YLBYKHJNJ2J3YYVYHRWEZTFLXZVL5WCKCBWXM526XF4KKCYA4RAQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.964879212 (25h0m8.858587821s) [noise=true , address=145.79.227.3:4151 , piece_id=QVFK6RJITYJIW6ZJRUYFU3P2NEMMVANA36O6WXPS7Z7FNFSYQWUA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.027795357 (25h0m8.921503966s) [noise=true , address=145.79.227.49:4339 , piece_id=NWDULUWHMV4PIMWMNO5XYXSAGVLNPLSIPYSDA32OZB2MBGBYPEUA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.267543450 (25h0m9.161252059s) [noise=true , address=145.79.227.28:4727 , piece_id=U5CDDUXQZ2BB7SPFAIWQXRGALJQTOWFNWGJLU3CGXKBUHYIQBIGQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.337643945 (25h0m9.231352554s) [noise=true , address=145.79.227.10:4401 , piece_id=C2X52SPGSNM2CB6TFGBCT6HO3MXSEMX3CCZQASDNFGWIEW2EGIGQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.380001967 (25h0m9.273710576s) [noise=true , address=145.79.227.32:4835 , piece_id=6PP7J34OCTKYEDEFQJWDUHO5H4Z6VG7LNG3SL7WLITSKMX43HAYQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.850164228 (25h0m9.743872837s) [noise=true , address=77.90.31.13:4627 , piece_id=TJFFMEVYN7M52FH5UP3WNAGC7HWF5GSEUPQJZ5KJWTJZWKZNAWXQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:13.183873887 (25h0m10.077582496s) [noise=true , address=77.90.31.19:4842 , piece_id=3VJ3J4JQPSJEVFSNSYDVHRFZMMCBMNZRBYEGJOOU4Q4HFEDDOKHQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:13.361354387 (25h0m10.255063026s) [noise=true , address=45.91.116.4:4937 , piece_id=MJHRB2I4KWRJJ3MHGW657D53JFDARTHOBRUEHQZEUA4DRFVK7ZUA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:13.361383082 (25h0m10.255091691s) [seg_pos={0 0}, upload=21929]: Satisfied cohort requirements after 13 pieces Jan 16 16:56:13.361615564 (25h0m10.255324173s) [noise=true , address=145.79.227.34:4906 , piece_id=G2GHF4S5NVJP5R43EN6NDEJ4FCPCLSL5K3TO4UADJ6ZANME54KOQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Jan 16 16:56:13.361624061 (25h0m10.25533267s) [noise=true , address=145.79.227.34:4906 , piece_id=G2GHF4S5NVJP5R43EN6NDEJ4FCPCLSL5K3TO4UADJ6ZANME54KOQ, seg_pos={0 0}, upload=21929]: Long tail cancellation due to optimal threshold Jan 16 16:56:13.361904004 (25h0m10.255612623s) [noise=true , address=145.79.227.38:4977 , piece_id=JXV6MGOVSAA2URSIF2KFFQJJEGWQWFVT65PWVDW5MEH5ITSDYFAA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8): protocol: read tcp 188.166.225.170:42244->145.79.227.38:4977: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 16 16:56:13.361925344 (25h0m10.255633963s) [noise=true , address=145.79.227.38:4977 , piece_id=JXV6MGOVSAA2URSIF2KFFQJJEGWQWFVT65PWVDW5MEH5ITSDYFAA, seg_pos={0 0}, upload=21929]: Long tail cancellation due to optimal threshold Jan 16 16:56:13.361971022 (25h0m10.255679631s) [noise=true , address=45.91.116.5:4300 , piece_id=M2F4W7IGAEKEXJXZRENON55A2E7DNSCZTPKWYJFY3NKNTBIWG2YA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Jan 16 16:56:13.361978977 (25h0m10.255687596s) [noise=true , address=45.91.116.5:4300 , piece_id=M2F4W7IGAEKEXJXZRENON55A2E7DNSCZTPKWYJFY3NKNTBIWG2YA, seg_pos={0 0}, upload=21929]: Long tail cancellation due to optimal threshold Jan 16 16:56:13.362041817 (25h0m10.255750426s) [seg_pos={0 0}, upload=21929]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 16 16:56:13.362197272 (25h0m10.255905881s) [upload=21929]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 16 16:56:13.362205868 (25h0m10.255914477s) [upload=21929]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 16 16:56:13.965177888 (25h0m10.858886497s) [upload=21929]: Done uploading segments... Jan 16 16:56:13.965204158 (25h0m10.858912767s) [upload=21929]: Upload finished. err: <nil> Duration: 5.724501469s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)