Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.......X.......................................XX...........] duration: 26.534582826s trace: https://jaeger.dev.storj.io/trace/70c2b0731de5fb9c FAIL (2026-01-16 17:12:01 UTC) orbiter warning: context deadline exceeded Duration: 26.53460583s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 53 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXX.XXXXXX.X..XXXXXXX.XXXXXXXXXXXXXX.XXXXXXXX.] duration: 3.808937409s trace: https://jaeger.dev.storj.io/trace/6d90503901c024d7 FAIL (2026-01-16 17:11:39 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.808962296s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [........X.......................................XX..........] duration: 17.450756905s trace: https://jaeger.dev.storj.io/trace/70c2b0731de5fb9c FAIL (2026-01-16 17:11:28 UTC) orbiter warning: context deadline exceeded Duration: 17.450777404s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............................................XXX...........] duration: 13.533681053s trace: https://jaeger.dev.storj.io/trace/521596a1b50467d9 FAIL (2026-01-16 17:11:22 UTC) orbiter warning: context deadline exceeded Duration: 13.533704528s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................XX......................] duration: 5.790675817s trace: https://jaeger.dev.storj.io/trace/ff1eaafc0d95989 FAIL (2026-01-16 17:11:17 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 16 16:56:06.586316499 (25h0m3.480025108s) [upload=21929]: Starting upload Jan 16 16:56:06.586371244 (25h0m3.480079853s) [upload=21929]: Done starting upload Jan 16 16:56:06.586529134 (25h0m3.480237763s) [upload=21929]: Uploading segments... Jan 16 16:56:06.586687636 (25h0m3.480396275s) [upload=21929]: Got next segment. Inline: false Jan 16 16:56:06.586750094 (25h0m3.480458723s) [upload=21929]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 16 16:56:06.586757138 (25h0m3.480465767s) [upload=21929]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 16 16:56:08.120364931 (25h0m5.01407354s) [seg_pos={0 0}, upload=21929]: Begin upload segment... Jan 16 16:56:08.120413474 (25h0m5.014122083s) [seg_pos={0 0}, upload=21929]: Done begin upload segment. Jan 16 16:56:08.120440615 (25h0m5.014149224s) [noise=true , address=145.79.227.2:4115 , piece_id=YMME5WUV7XGQZOE2I5H7UWQ7TSTP2LGV7D4AZO4T62XOWBE4ZEKA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.120799800 (25h0m5.014508419s) [noise=true , address=145.79.227.32:4835 , piece_id=6PP7J34OCTKYEDEFQJWDUHO5H4Z6VG7LNG3SL7WLITSKMX43HAYQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.121695454 (25h0m5.015404084s) [noise=true , address=45.91.116.4:4937 , piece_id=MJHRB2I4KWRJJ3MHGW657D53JFDARTHOBRUEHQZEUA4DRFVK7ZUA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.122317739 (25h0m5.016026349s) [noise=true , address=145.79.227.30:4762 , piece_id=YLBYKHJNJ2J3YYVYHRWEZTFLXZVL5WCKCBWXM526XF4KKCYA4RAQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.123161636 (25h0m5.016870245s) [noise=true , address=45.91.116.14:4190 , piece_id=ZBC7C5E5V7B4VAGI2JHD7T4E6BGVV74YNNAROZYBITS2N23BJDWA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.123395732 (25h0m5.017104351s) [noise=true , address=145.79.227.28:4727 , piece_id=U5CDDUXQZ2BB7SPFAIWQXRGALJQTOWFNWGJLU3CGXKBUHYIQBIGQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.123848494 (25h0m5.017557113s) [noise=true , address=77.90.31.19:4842 , piece_id=3VJ3J4JQPSJEVFSNSYDVHRFZMMCBMNZRBYEGJOOU4Q4HFEDDOKHQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.123868092 (25h0m5.017576731s) [noise=true , address=145.79.227.15:4873 , piece_id=EVJGQ5JNJI37QIPQDJPFOPMT3JAHPBZXQG7DBRUZXJMKXAIADWSA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.124249808 (25h0m5.017958417s) [noise=true , address=145.79.227.38:4977 , piece_id=JXV6MGOVSAA2URSIF2KFFQJJEGWQWFVT65PWVDW5MEH5ITSDYFAA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.124389284 (25h0m5.018097893s) [noise=true , address=145.79.227.49:4339 , piece_id=NWDULUWHMV4PIMWMNO5XYXSAGVLNPLSIPYSDA32OZB2MBGBYPEUA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.124619222 (25h0m5.018327831s) [noise=true , address=145.79.227.33:4870 , piece_id=7I3KU7EKC2MHBJ6TOGO5ZBMJSPYU4E26D5373M2R53IZAZJLWVOA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.124999566 (25h0m5.018708175s) [noise=true , address=145.79.227.3:4151 , piece_id=QVFK6RJITYJIW6ZJRUYFU3P2NEMMVANA36O6WXPS7Z7FNFSYQWUA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.125202182 (25h0m5.018910801s) [noise=true , address=77.90.31.13:4627 , piece_id=TJFFMEVYN7M52FH5UP3WNAGC7HWF5GSEUPQJZ5KJWTJZWKZNAWXQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.125576805 (25h0m5.019285414s) [noise=true , address=45.91.116.5:4300 , piece_id=M2F4W7IGAEKEXJXZRENON55A2E7DNSCZTPKWYJFY3NKNTBIWG2YA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.125952470 (25h0m5.019661079s) [noise=true , address=145.79.227.10:4401 , piece_id=C2X52SPGSNM2CB6TFGBCT6HO3MXSEMX3CCZQASDNFGWIEW2EGIGQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.126492900 (25h0m5.020201509s) [noise=true , address=145.79.227.34:4906 , piece_id=G2GHF4S5NVJP5R43EN6NDEJ4FCPCLSL5K3TO4UADJ6ZANME54KOQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:10.730578935 (25h0m7.624287544s) [upload=21929]: Next returned nil segment Jan 16 16:56:10.730594134 (25h0m7.624302753s) [upload=21929]: Waiting for error group managing segments... Jan 16 16:56:11.329683279 (25h0m8.223391888s) [noise=true , address=145.79.227.15:4873 , piece_id=EVJGQ5JNJI37QIPQDJPFOPMT3JAHPBZXQG7DBRUZXJMKXAIADWSA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.338893335 (25h0m8.232601944s) [noise=true , address=45.91.116.14:4190 , piece_id=ZBC7C5E5V7B4VAGI2JHD7T4E6BGVV74YNNAROZYBITS2N23BJDWA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.400223651 (25h0m8.29393226s) [noise=true , address=145.79.227.33:4870 , piece_id=7I3KU7EKC2MHBJ6TOGO5ZBMJSPYU4E26D5373M2R53IZAZJLWVOA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.400340974 (25h0m8.294049583s) [noise=true , address=145.79.227.2:4115 , piece_id=YMME5WUV7XGQZOE2I5H7UWQ7TSTP2LGV7D4AZO4T62XOWBE4ZEKA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.445450294 (25h0m8.339158904s) [noise=true , address=145.79.227.30:4762 , piece_id=YLBYKHJNJ2J3YYVYHRWEZTFLXZVL5WCKCBWXM526XF4KKCYA4RAQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.964879212 (25h0m8.858587821s) [noise=true , address=145.79.227.3:4151 , piece_id=QVFK6RJITYJIW6ZJRUYFU3P2NEMMVANA36O6WXPS7Z7FNFSYQWUA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.027795357 (25h0m8.921503966s) [noise=true , address=145.79.227.49:4339 , piece_id=NWDULUWHMV4PIMWMNO5XYXSAGVLNPLSIPYSDA32OZB2MBGBYPEUA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.267543450 (25h0m9.161252059s) [noise=true , address=145.79.227.28:4727 , piece_id=U5CDDUXQZ2BB7SPFAIWQXRGALJQTOWFNWGJLU3CGXKBUHYIQBIGQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.337643945 (25h0m9.231352554s) [noise=true , address=145.79.227.10:4401 , piece_id=C2X52SPGSNM2CB6TFGBCT6HO3MXSEMX3CCZQASDNFGWIEW2EGIGQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.380001967 (25h0m9.273710576s) [noise=true , address=145.79.227.32:4835 , piece_id=6PP7J34OCTKYEDEFQJWDUHO5H4Z6VG7LNG3SL7WLITSKMX43HAYQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.850164228 (25h0m9.743872837s) [noise=true , address=77.90.31.13:4627 , piece_id=TJFFMEVYN7M52FH5UP3WNAGC7HWF5GSEUPQJZ5KJWTJZWKZNAWXQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:13.183873887 (25h0m10.077582496s) [noise=true , address=77.90.31.19:4842 , piece_id=3VJ3J4JQPSJEVFSNSYDVHRFZMMCBMNZRBYEGJOOU4Q4HFEDDOKHQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:13.361354387 (25h0m10.255063026s) [noise=true , address=45.91.116.4:4937 , piece_id=MJHRB2I4KWRJJ3MHGW657D53JFDARTHOBRUEHQZEUA4DRFVK7ZUA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:13.361383082 (25h0m10.255091691s) [seg_pos={0 0}, upload=21929]: Satisfied cohort requirements after 13 pieces Jan 16 16:56:13.361615564 (25h0m10.255324173s) [noise=true , address=145.79.227.34:4906 , piece_id=G2GHF4S5NVJP5R43EN6NDEJ4FCPCLSL5K3TO4UADJ6ZANME54KOQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Jan 16 16:56:13.361624061 (25h0m10.25533267s) [noise=true , address=145.79.227.34:4906 , piece_id=G2GHF4S5NVJP5R43EN6NDEJ4FCPCLSL5K3TO4UADJ6ZANME54KOQ, seg_pos={0 0}, upload=21929]: Long tail cancellation due to optimal threshold Jan 16 16:56:13.361904004 (25h0m10.255612623s) [noise=true , address=145.79.227.38:4977 , piece_id=JXV6MGOVSAA2URSIF2KFFQJJEGWQWFVT65PWVDW5MEH5ITSDYFAA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8): protocol: read tcp 188.166.225.170:42244->145.79.227.38:4977: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 16 16:56:13.361925344 (25h0m10.255633963s) [noise=true , address=145.79.227.38:4977 , piece_id=JXV6MGOVSAA2URSIF2KFFQJJEGWQWFVT65PWVDW5MEH5ITSDYFAA, seg_pos={0 0}, upload=21929]: Long tail cancellation due to optimal threshold Jan 16 16:56:13.361971022 (25h0m10.255679631s) [noise=true , address=45.91.116.5:4300 , piece_id=M2F4W7IGAEKEXJXZRENON55A2E7DNSCZTPKWYJFY3NKNTBIWG2YA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Jan 16 16:56:13.361978977 (25h0m10.255687596s) [noise=true , address=45.91.116.5:4300 , piece_id=M2F4W7IGAEKEXJXZRENON55A2E7DNSCZTPKWYJFY3NKNTBIWG2YA, seg_pos={0 0}, upload=21929]: Long tail cancellation due to optimal threshold Jan 16 16:56:13.362041817 (25h0m10.255750426s) [seg_pos={0 0}, upload=21929]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 16 16:56:13.362197272 (25h0m10.255905881s) [upload=21929]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 16 16:56:13.362205868 (25h0m10.255914477s) [upload=21929]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 16 16:56:13.965177888 (25h0m10.858886497s) [upload=21929]: Done uploading segments... Jan 16 16:56:13.965204158 (25h0m10.858912767s) [upload=21929]: Upload finished. err: <nil> Duration: 5.790696094s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.........X.......................................XX.........] duration: 34.905507104s trace: https://jaeger.dev.storj.io/trace/70c2b0731de5fb9c FAIL (2026-01-16 17:11:10 UTC) orbiter warning: context deadline exceeded Duration: 34.905551679s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 54 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXX.XXXXXX.X..XXXXXXX.XXXXXXXXXXXXXX.XXXXXXXX] duration: 5.001647687s trace: https://jaeger.dev.storj.io/trace/6d90503901c024d7 FAIL (2026-01-16 17:10:55 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001669188s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................XX.....................] duration: 5.775639616s trace: https://jaeger.dev.storj.io/trace/ff1eaafc0d95989 FAIL (2026-01-16 17:10:20 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 16 16:56:06.586316499 (25h0m3.480025108s) [upload=21929]: Starting upload Jan 16 16:56:06.586371244 (25h0m3.480079853s) [upload=21929]: Done starting upload Jan 16 16:56:06.586529134 (25h0m3.480237763s) [upload=21929]: Uploading segments... Jan 16 16:56:06.586687636 (25h0m3.480396275s) [upload=21929]: Got next segment. Inline: false Jan 16 16:56:06.586750094 (25h0m3.480458723s) [upload=21929]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 16 16:56:06.586757138 (25h0m3.480465767s) [upload=21929]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 16 16:56:08.120364931 (25h0m5.01407354s) [seg_pos={0 0}, upload=21929]: Begin upload segment... Jan 16 16:56:08.120413474 (25h0m5.014122083s) [seg_pos={0 0}, upload=21929]: Done begin upload segment. Jan 16 16:56:08.120440615 (25h0m5.014149224s) [noise=true , address=145.79.227.2:4115 , piece_id=YMME5WUV7XGQZOE2I5H7UWQ7TSTP2LGV7D4AZO4T62XOWBE4ZEKA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.120799800 (25h0m5.014508419s) [noise=true , address=145.79.227.32:4835 , piece_id=6PP7J34OCTKYEDEFQJWDUHO5H4Z6VG7LNG3SL7WLITSKMX43HAYQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.121695454 (25h0m5.015404084s) [noise=true , address=45.91.116.4:4937 , piece_id=MJHRB2I4KWRJJ3MHGW657D53JFDARTHOBRUEHQZEUA4DRFVK7ZUA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.122317739 (25h0m5.016026349s) [noise=true , address=145.79.227.30:4762 , piece_id=YLBYKHJNJ2J3YYVYHRWEZTFLXZVL5WCKCBWXM526XF4KKCYA4RAQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.123161636 (25h0m5.016870245s) [noise=true , address=45.91.116.14:4190 , piece_id=ZBC7C5E5V7B4VAGI2JHD7T4E6BGVV74YNNAROZYBITS2N23BJDWA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.123395732 (25h0m5.017104351s) [noise=true , address=145.79.227.28:4727 , piece_id=U5CDDUXQZ2BB7SPFAIWQXRGALJQTOWFNWGJLU3CGXKBUHYIQBIGQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.123848494 (25h0m5.017557113s) [noise=true , address=77.90.31.19:4842 , piece_id=3VJ3J4JQPSJEVFSNSYDVHRFZMMCBMNZRBYEGJOOU4Q4HFEDDOKHQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.123868092 (25h0m5.017576731s) [noise=true , address=145.79.227.15:4873 , piece_id=EVJGQ5JNJI37QIPQDJPFOPMT3JAHPBZXQG7DBRUZXJMKXAIADWSA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.124249808 (25h0m5.017958417s) [noise=true , address=145.79.227.38:4977 , piece_id=JXV6MGOVSAA2URSIF2KFFQJJEGWQWFVT65PWVDW5MEH5ITSDYFAA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.124389284 (25h0m5.018097893s) [noise=true , address=145.79.227.49:4339 , piece_id=NWDULUWHMV4PIMWMNO5XYXSAGVLNPLSIPYSDA32OZB2MBGBYPEUA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.124619222 (25h0m5.018327831s) [noise=true , address=145.79.227.33:4870 , piece_id=7I3KU7EKC2MHBJ6TOGO5ZBMJSPYU4E26D5373M2R53IZAZJLWVOA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.124999566 (25h0m5.018708175s) [noise=true , address=145.79.227.3:4151 , piece_id=QVFK6RJITYJIW6ZJRUYFU3P2NEMMVANA36O6WXPS7Z7FNFSYQWUA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.125202182 (25h0m5.018910801s) [noise=true , address=77.90.31.13:4627 , piece_id=TJFFMEVYN7M52FH5UP3WNAGC7HWF5GSEUPQJZ5KJWTJZWKZNAWXQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.125576805 (25h0m5.019285414s) [noise=true , address=45.91.116.5:4300 , piece_id=M2F4W7IGAEKEXJXZRENON55A2E7DNSCZTPKWYJFY3NKNTBIWG2YA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.125952470 (25h0m5.019661079s) [noise=true , address=145.79.227.10:4401 , piece_id=C2X52SPGSNM2CB6TFGBCT6HO3MXSEMX3CCZQASDNFGWIEW2EGIGQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.126492900 (25h0m5.020201509s) [noise=true , address=145.79.227.34:4906 , piece_id=G2GHF4S5NVJP5R43EN6NDEJ4FCPCLSL5K3TO4UADJ6ZANME54KOQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:10.730578935 (25h0m7.624287544s) [upload=21929]: Next returned nil segment Jan 16 16:56:10.730594134 (25h0m7.624302753s) [upload=21929]: Waiting for error group managing segments... Jan 16 16:56:11.329683279 (25h0m8.223391888s) [noise=true , address=145.79.227.15:4873 , piece_id=EVJGQ5JNJI37QIPQDJPFOPMT3JAHPBZXQG7DBRUZXJMKXAIADWSA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.338893335 (25h0m8.232601944s) [noise=true , address=45.91.116.14:4190 , piece_id=ZBC7C5E5V7B4VAGI2JHD7T4E6BGVV74YNNAROZYBITS2N23BJDWA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.400223651 (25h0m8.29393226s) [noise=true , address=145.79.227.33:4870 , piece_id=7I3KU7EKC2MHBJ6TOGO5ZBMJSPYU4E26D5373M2R53IZAZJLWVOA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.400340974 (25h0m8.294049583s) [noise=true , address=145.79.227.2:4115 , piece_id=YMME5WUV7XGQZOE2I5H7UWQ7TSTP2LGV7D4AZO4T62XOWBE4ZEKA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.445450294 (25h0m8.339158904s) [noise=true , address=145.79.227.30:4762 , piece_id=YLBYKHJNJ2J3YYVYHRWEZTFLXZVL5WCKCBWXM526XF4KKCYA4RAQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.964879212 (25h0m8.858587821s) [noise=true , address=145.79.227.3:4151 , piece_id=QVFK6RJITYJIW6ZJRUYFU3P2NEMMVANA36O6WXPS7Z7FNFSYQWUA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.027795357 (25h0m8.921503966s) [noise=true , address=145.79.227.49:4339 , piece_id=NWDULUWHMV4PIMWMNO5XYXSAGVLNPLSIPYSDA32OZB2MBGBYPEUA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.267543450 (25h0m9.161252059s) [noise=true , address=145.79.227.28:4727 , piece_id=U5CDDUXQZ2BB7SPFAIWQXRGALJQTOWFNWGJLU3CGXKBUHYIQBIGQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.337643945 (25h0m9.231352554s) [noise=true , address=145.79.227.10:4401 , piece_id=C2X52SPGSNM2CB6TFGBCT6HO3MXSEMX3CCZQASDNFGWIEW2EGIGQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.380001967 (25h0m9.273710576s) [noise=true , address=145.79.227.32:4835 , piece_id=6PP7J34OCTKYEDEFQJWDUHO5H4Z6VG7LNG3SL7WLITSKMX43HAYQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.850164228 (25h0m9.743872837s) [noise=true , address=77.90.31.13:4627 , piece_id=TJFFMEVYN7M52FH5UP3WNAGC7HWF5GSEUPQJZ5KJWTJZWKZNAWXQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:13.183873887 (25h0m10.077582496s) [noise=true , address=77.90.31.19:4842 , piece_id=3VJ3J4JQPSJEVFSNSYDVHRFZMMCBMNZRBYEGJOOU4Q4HFEDDOKHQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:13.361354387 (25h0m10.255063026s) [noise=true , address=45.91.116.4:4937 , piece_id=MJHRB2I4KWRJJ3MHGW657D53JFDARTHOBRUEHQZEUA4DRFVK7ZUA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:13.361383082 (25h0m10.255091691s) [seg_pos={0 0}, upload=21929]: Satisfied cohort requirements after 13 pieces Jan 16 16:56:13.361615564 (25h0m10.255324173s) [noise=true , address=145.79.227.34:4906 , piece_id=G2GHF4S5NVJP5R43EN6NDEJ4FCPCLSL5K3TO4UADJ6ZANME54KOQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Jan 16 16:56:13.361624061 (25h0m10.25533267s) [noise=true , address=145.79.227.34:4906 , piece_id=G2GHF4S5NVJP5R43EN6NDEJ4FCPCLSL5K3TO4UADJ6ZANME54KOQ, seg_pos={0 0}, upload=21929]: Long tail cancellation due to optimal threshold Jan 16 16:56:13.361904004 (25h0m10.255612623s) [noise=true , address=145.79.227.38:4977 , piece_id=JXV6MGOVSAA2URSIF2KFFQJJEGWQWFVT65PWVDW5MEH5ITSDYFAA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8): protocol: read tcp 188.166.225.170:42244->145.79.227.38:4977: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 16 16:56:13.361925344 (25h0m10.255633963s) [noise=true , address=145.79.227.38:4977 , piece_id=JXV6MGOVSAA2URSIF2KFFQJJEGWQWFVT65PWVDW5MEH5ITSDYFAA, seg_pos={0 0}, upload=21929]: Long tail cancellation due to optimal threshold Jan 16 16:56:13.361971022 (25h0m10.255679631s) [noise=true , address=45.91.116.5:4300 , piece_id=M2F4W7IGAEKEXJXZRENON55A2E7DNSCZTPKWYJFY3NKNTBIWG2YA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Jan 16 16:56:13.361978977 (25h0m10.255687596s) [noise=true , address=45.91.116.5:4300 , piece_id=M2F4W7IGAEKEXJXZRENON55A2E7DNSCZTPKWYJFY3NKNTBIWG2YA, seg_pos={0 0}, upload=21929]: Long tail cancellation due to optimal threshold Jan 16 16:56:13.362041817 (25h0m10.255750426s) [seg_pos={0 0}, upload=21929]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 16 16:56:13.362197272 (25h0m10.255905881s) [upload=21929]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 16 16:56:13.362205868 (25h0m10.255914477s) [upload=21929]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 16 16:56:13.965177888 (25h0m10.858886497s) [upload=21929]: Done uploading segments... Jan 16 16:56:13.965204158 (25h0m10.858912767s) [upload=21929]: Upload finished. err: <nil> Duration: 5.775666046s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............................................XXX..........] duration: 21.617636159s trace: https://jaeger.dev.storj.io/trace/521596a1b50467d9 FAIL (2026-01-16 17:10:17 UTC) orbiter warning: context deadline exceeded Duration: 21.617676786s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..........X.......................................XX........] duration: 16.479891305s trace: https://jaeger.dev.storj.io/trace/70c2b0731de5fb9c FAIL (2026-01-16 17:10:11 UTC) orbiter warning: context deadline exceeded Duration: 16.479916063s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 54 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXX.XXXXXX.X..XXXXXXX.XXXXXXXXXXXXXX.XXXXXXX] duration: 5.001841606s trace: https://jaeger.dev.storj.io/trace/36fe72cf8a51dd3d FAIL (2026-01-16 17:10:07 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001868416s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 54 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXX.XXXXXX.X..XXXXXXX.XXXXXXXXXXXXXX.XXXXXX] duration: 5.00104479s trace: https://jaeger.dev.storj.io/trace/3fd51f48cc8037a2 FAIL (2026-01-16 17:09:50 UTC) orbiter warning: context deadline exceeded Duration: 5.001068484s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [................................................XXX.........] duration: 23.439126386s trace: https://jaeger.dev.storj.io/trace/521596a1b50467d9 FAIL (2026-01-16 17:09:33 UTC) orbiter warning: context deadline exceeded Duration: 23.439150754s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 53 out of 60 is failed (threshold=6) [.XXXXXXXXXXXXXXXXXXXX.XXXXXX.X..XXXXXXX.XXXXXXXXXXXXXX.XXXXX] duration: 5.001233781s trace: https://jaeger.dev.storj.io/trace/306f5d8dc64fbd54 FAIL (2026-01-16 17:09:26 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001257566s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................XX....................] duration: 5.764979159s trace: https://jaeger.dev.storj.io/trace/ff1eaafc0d95989 FAIL (2026-01-16 17:09:14 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 16 16:56:06.586316499 (25h0m3.480025108s) [upload=21929]: Starting upload Jan 16 16:56:06.586371244 (25h0m3.480079853s) [upload=21929]: Done starting upload Jan 16 16:56:06.586529134 (25h0m3.480237763s) [upload=21929]: Uploading segments... Jan 16 16:56:06.586687636 (25h0m3.480396275s) [upload=21929]: Got next segment. Inline: false Jan 16 16:56:06.586750094 (25h0m3.480458723s) [upload=21929]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 16 16:56:06.586757138 (25h0m3.480465767s) [upload=21929]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 16 16:56:08.120364931 (25h0m5.01407354s) [seg_pos={0 0}, upload=21929]: Begin upload segment... Jan 16 16:56:08.120413474 (25h0m5.014122083s) [seg_pos={0 0}, upload=21929]: Done begin upload segment. Jan 16 16:56:08.120440615 (25h0m5.014149224s) [noise=true , address=145.79.227.2:4115 , piece_id=YMME5WUV7XGQZOE2I5H7UWQ7TSTP2LGV7D4AZO4T62XOWBE4ZEKA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.120799800 (25h0m5.014508419s) [noise=true , address=145.79.227.32:4835 , piece_id=6PP7J34OCTKYEDEFQJWDUHO5H4Z6VG7LNG3SL7WLITSKMX43HAYQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.121695454 (25h0m5.015404084s) [noise=true , address=45.91.116.4:4937 , piece_id=MJHRB2I4KWRJJ3MHGW657D53JFDARTHOBRUEHQZEUA4DRFVK7ZUA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.122317739 (25h0m5.016026349s) [noise=true , address=145.79.227.30:4762 , piece_id=YLBYKHJNJ2J3YYVYHRWEZTFLXZVL5WCKCBWXM526XF4KKCYA4RAQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.123161636 (25h0m5.016870245s) [noise=true , address=45.91.116.14:4190 , piece_id=ZBC7C5E5V7B4VAGI2JHD7T4E6BGVV74YNNAROZYBITS2N23BJDWA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.123395732 (25h0m5.017104351s) [noise=true , address=145.79.227.28:4727 , piece_id=U5CDDUXQZ2BB7SPFAIWQXRGALJQTOWFNWGJLU3CGXKBUHYIQBIGQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.123848494 (25h0m5.017557113s) [noise=true , address=77.90.31.19:4842 , piece_id=3VJ3J4JQPSJEVFSNSYDVHRFZMMCBMNZRBYEGJOOU4Q4HFEDDOKHQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.123868092 (25h0m5.017576731s) [noise=true , address=145.79.227.15:4873 , piece_id=EVJGQ5JNJI37QIPQDJPFOPMT3JAHPBZXQG7DBRUZXJMKXAIADWSA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.124249808 (25h0m5.017958417s) [noise=true , address=145.79.227.38:4977 , piece_id=JXV6MGOVSAA2URSIF2KFFQJJEGWQWFVT65PWVDW5MEH5ITSDYFAA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.124389284 (25h0m5.018097893s) [noise=true , address=145.79.227.49:4339 , piece_id=NWDULUWHMV4PIMWMNO5XYXSAGVLNPLSIPYSDA32OZB2MBGBYPEUA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.124619222 (25h0m5.018327831s) [noise=true , address=145.79.227.33:4870 , piece_id=7I3KU7EKC2MHBJ6TOGO5ZBMJSPYU4E26D5373M2R53IZAZJLWVOA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.124999566 (25h0m5.018708175s) [noise=true , address=145.79.227.3:4151 , piece_id=QVFK6RJITYJIW6ZJRUYFU3P2NEMMVANA36O6WXPS7Z7FNFSYQWUA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.125202182 (25h0m5.018910801s) [noise=true , address=77.90.31.13:4627 , piece_id=TJFFMEVYN7M52FH5UP3WNAGC7HWF5GSEUPQJZ5KJWTJZWKZNAWXQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.125576805 (25h0m5.019285414s) [noise=true , address=45.91.116.5:4300 , piece_id=M2F4W7IGAEKEXJXZRENON55A2E7DNSCZTPKWYJFY3NKNTBIWG2YA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.125952470 (25h0m5.019661079s) [noise=true , address=145.79.227.10:4401 , piece_id=C2X52SPGSNM2CB6TFGBCT6HO3MXSEMX3CCZQASDNFGWIEW2EGIGQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.126492900 (25h0m5.020201509s) [noise=true , address=145.79.227.34:4906 , piece_id=G2GHF4S5NVJP5R43EN6NDEJ4FCPCLSL5K3TO4UADJ6ZANME54KOQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:10.730578935 (25h0m7.624287544s) [upload=21929]: Next returned nil segment Jan 16 16:56:10.730594134 (25h0m7.624302753s) [upload=21929]: Waiting for error group managing segments... Jan 16 16:56:11.329683279 (25h0m8.223391888s) [noise=true , address=145.79.227.15:4873 , piece_id=EVJGQ5JNJI37QIPQDJPFOPMT3JAHPBZXQG7DBRUZXJMKXAIADWSA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.338893335 (25h0m8.232601944s) [noise=true , address=45.91.116.14:4190 , piece_id=ZBC7C5E5V7B4VAGI2JHD7T4E6BGVV74YNNAROZYBITS2N23BJDWA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.400223651 (25h0m8.29393226s) [noise=true , address=145.79.227.33:4870 , piece_id=7I3KU7EKC2MHBJ6TOGO5ZBMJSPYU4E26D5373M2R53IZAZJLWVOA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.400340974 (25h0m8.294049583s) [noise=true , address=145.79.227.2:4115 , piece_id=YMME5WUV7XGQZOE2I5H7UWQ7TSTP2LGV7D4AZO4T62XOWBE4ZEKA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.445450294 (25h0m8.339158904s) [noise=true , address=145.79.227.30:4762 , piece_id=YLBYKHJNJ2J3YYVYHRWEZTFLXZVL5WCKCBWXM526XF4KKCYA4RAQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.964879212 (25h0m8.858587821s) [noise=true , address=145.79.227.3:4151 , piece_id=QVFK6RJITYJIW6ZJRUYFU3P2NEMMVANA36O6WXPS7Z7FNFSYQWUA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.027795357 (25h0m8.921503966s) [noise=true , address=145.79.227.49:4339 , piece_id=NWDULUWHMV4PIMWMNO5XYXSAGVLNPLSIPYSDA32OZB2MBGBYPEUA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.267543450 (25h0m9.161252059s) [noise=true , address=145.79.227.28:4727 , piece_id=U5CDDUXQZ2BB7SPFAIWQXRGALJQTOWFNWGJLU3CGXKBUHYIQBIGQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.337643945 (25h0m9.231352554s) [noise=true , address=145.79.227.10:4401 , piece_id=C2X52SPGSNM2CB6TFGBCT6HO3MXSEMX3CCZQASDNFGWIEW2EGIGQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.380001967 (25h0m9.273710576s) [noise=true , address=145.79.227.32:4835 , piece_id=6PP7J34OCTKYEDEFQJWDUHO5H4Z6VG7LNG3SL7WLITSKMX43HAYQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.850164228 (25h0m9.743872837s) [noise=true , address=77.90.31.13:4627 , piece_id=TJFFMEVYN7M52FH5UP3WNAGC7HWF5GSEUPQJZ5KJWTJZWKZNAWXQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:13.183873887 (25h0m10.077582496s) [noise=true , address=77.90.31.19:4842 , piece_id=3VJ3J4JQPSJEVFSNSYDVHRFZMMCBMNZRBYEGJOOU4Q4HFEDDOKHQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:13.361354387 (25h0m10.255063026s) [noise=true , address=45.91.116.4:4937 , piece_id=MJHRB2I4KWRJJ3MHGW657D53JFDARTHOBRUEHQZEUA4DRFVK7ZUA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:13.361383082 (25h0m10.255091691s) [seg_pos={0 0}, upload=21929]: Satisfied cohort requirements after 13 pieces Jan 16 16:56:13.361615564 (25h0m10.255324173s) [noise=true , address=145.79.227.34:4906 , piece_id=G2GHF4S5NVJP5R43EN6NDEJ4FCPCLSL5K3TO4UADJ6ZANME54KOQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Jan 16 16:56:13.361624061 (25h0m10.25533267s) [noise=true , address=145.79.227.34:4906 , piece_id=G2GHF4S5NVJP5R43EN6NDEJ4FCPCLSL5K3TO4UADJ6ZANME54KOQ, seg_pos={0 0}, upload=21929]: Long tail cancellation due to optimal threshold Jan 16 16:56:13.361904004 (25h0m10.255612623s) [noise=true , address=145.79.227.38:4977 , piece_id=JXV6MGOVSAA2URSIF2KFFQJJEGWQWFVT65PWVDW5MEH5ITSDYFAA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8): protocol: read tcp 188.166.225.170:42244->145.79.227.38:4977: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 16 16:56:13.361925344 (25h0m10.255633963s) [noise=true , address=145.79.227.38:4977 , piece_id=JXV6MGOVSAA2URSIF2KFFQJJEGWQWFVT65PWVDW5MEH5ITSDYFAA, seg_pos={0 0}, upload=21929]: Long tail cancellation due to optimal threshold Jan 16 16:56:13.361971022 (25h0m10.255679631s) [noise=true , address=45.91.116.5:4300 , piece_id=M2F4W7IGAEKEXJXZRENON55A2E7DNSCZTPKWYJFY3NKNTBIWG2YA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Jan 16 16:56:13.361978977 (25h0m10.255687596s) [noise=true , address=45.91.116.5:4300 , piece_id=M2F4W7IGAEKEXJXZRENON55A2E7DNSCZTPKWYJFY3NKNTBIWG2YA, seg_pos={0 0}, upload=21929]: Long tail cancellation due to optimal threshold Jan 16 16:56:13.362041817 (25h0m10.255750426s) [seg_pos={0 0}, upload=21929]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 16 16:56:13.362197272 (25h0m10.255905881s) [upload=21929]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 16 16:56:13.362205868 (25h0m10.255914477s) [upload=21929]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 16 16:56:13.965177888 (25h0m10.858886497s) [upload=21929]: Done uploading segments... Jan 16 16:56:13.965204158 (25h0m10.858912767s) [upload=21929]: Upload finished. err: <nil> Duration: 5.765006441s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [...........X.......................................XX.......] duration: 7.126071745s trace: https://jaeger.dev.storj.io/trace/70c2b0731de5fb9c FAIL (2026-01-16 17:09:08 UTC) orbiter warning: context deadline exceeded Duration: 7.126101581s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................XX...................] duration: 5.767496622s trace: https://jaeger.dev.storj.io/trace/ff1eaafc0d95989 FAIL (2026-01-16 17:08:52 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 16 16:56:06.586316499 (25h0m3.480025108s) [upload=21929]: Starting upload Jan 16 16:56:06.586371244 (25h0m3.480079853s) [upload=21929]: Done starting upload Jan 16 16:56:06.586529134 (25h0m3.480237763s) [upload=21929]: Uploading segments... Jan 16 16:56:06.586687636 (25h0m3.480396275s) [upload=21929]: Got next segment. Inline: false Jan 16 16:56:06.586750094 (25h0m3.480458723s) [upload=21929]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 16 16:56:06.586757138 (25h0m3.480465767s) [upload=21929]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 16 16:56:08.120364931 (25h0m5.01407354s) [seg_pos={0 0}, upload=21929]: Begin upload segment... Jan 16 16:56:08.120413474 (25h0m5.014122083s) [seg_pos={0 0}, upload=21929]: Done begin upload segment. Jan 16 16:56:08.120440615 (25h0m5.014149224s) [noise=true , address=145.79.227.2:4115 , piece_id=YMME5WUV7XGQZOE2I5H7UWQ7TSTP2LGV7D4AZO4T62XOWBE4ZEKA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.120799800 (25h0m5.014508419s) [noise=true , address=145.79.227.32:4835 , piece_id=6PP7J34OCTKYEDEFQJWDUHO5H4Z6VG7LNG3SL7WLITSKMX43HAYQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.121695454 (25h0m5.015404084s) [noise=true , address=45.91.116.4:4937 , piece_id=MJHRB2I4KWRJJ3MHGW657D53JFDARTHOBRUEHQZEUA4DRFVK7ZUA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.122317739 (25h0m5.016026349s) [noise=true , address=145.79.227.30:4762 , piece_id=YLBYKHJNJ2J3YYVYHRWEZTFLXZVL5WCKCBWXM526XF4KKCYA4RAQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.123161636 (25h0m5.016870245s) [noise=true , address=45.91.116.14:4190 , piece_id=ZBC7C5E5V7B4VAGI2JHD7T4E6BGVV74YNNAROZYBITS2N23BJDWA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.123395732 (25h0m5.017104351s) [noise=true , address=145.79.227.28:4727 , piece_id=U5CDDUXQZ2BB7SPFAIWQXRGALJQTOWFNWGJLU3CGXKBUHYIQBIGQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.123848494 (25h0m5.017557113s) [noise=true , address=77.90.31.19:4842 , piece_id=3VJ3J4JQPSJEVFSNSYDVHRFZMMCBMNZRBYEGJOOU4Q4HFEDDOKHQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.123868092 (25h0m5.017576731s) [noise=true , address=145.79.227.15:4873 , piece_id=EVJGQ5JNJI37QIPQDJPFOPMT3JAHPBZXQG7DBRUZXJMKXAIADWSA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.124249808 (25h0m5.017958417s) [noise=true , address=145.79.227.38:4977 , piece_id=JXV6MGOVSAA2URSIF2KFFQJJEGWQWFVT65PWVDW5MEH5ITSDYFAA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.124389284 (25h0m5.018097893s) [noise=true , address=145.79.227.49:4339 , piece_id=NWDULUWHMV4PIMWMNO5XYXSAGVLNPLSIPYSDA32OZB2MBGBYPEUA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.124619222 (25h0m5.018327831s) [noise=true , address=145.79.227.33:4870 , piece_id=7I3KU7EKC2MHBJ6TOGO5ZBMJSPYU4E26D5373M2R53IZAZJLWVOA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.124999566 (25h0m5.018708175s) [noise=true , address=145.79.227.3:4151 , piece_id=QVFK6RJITYJIW6ZJRUYFU3P2NEMMVANA36O6WXPS7Z7FNFSYQWUA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.125202182 (25h0m5.018910801s) [noise=true , address=77.90.31.13:4627 , piece_id=TJFFMEVYN7M52FH5UP3WNAGC7HWF5GSEUPQJZ5KJWTJZWKZNAWXQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.125576805 (25h0m5.019285414s) [noise=true , address=45.91.116.5:4300 , piece_id=M2F4W7IGAEKEXJXZRENON55A2E7DNSCZTPKWYJFY3NKNTBIWG2YA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.125952470 (25h0m5.019661079s) [noise=true , address=145.79.227.10:4401 , piece_id=C2X52SPGSNM2CB6TFGBCT6HO3MXSEMX3CCZQASDNFGWIEW2EGIGQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.126492900 (25h0m5.020201509s) [noise=true , address=145.79.227.34:4906 , piece_id=G2GHF4S5NVJP5R43EN6NDEJ4FCPCLSL5K3TO4UADJ6ZANME54KOQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:10.730578935 (25h0m7.624287544s) [upload=21929]: Next returned nil segment Jan 16 16:56:10.730594134 (25h0m7.624302753s) [upload=21929]: Waiting for error group managing segments... Jan 16 16:56:11.329683279 (25h0m8.223391888s) [noise=true , address=145.79.227.15:4873 , piece_id=EVJGQ5JNJI37QIPQDJPFOPMT3JAHPBZXQG7DBRUZXJMKXAIADWSA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.338893335 (25h0m8.232601944s) [noise=true , address=45.91.116.14:4190 , piece_id=ZBC7C5E5V7B4VAGI2JHD7T4E6BGVV74YNNAROZYBITS2N23BJDWA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.400223651 (25h0m8.29393226s) [noise=true , address=145.79.227.33:4870 , piece_id=7I3KU7EKC2MHBJ6TOGO5ZBMJSPYU4E26D5373M2R53IZAZJLWVOA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.400340974 (25h0m8.294049583s) [noise=true , address=145.79.227.2:4115 , piece_id=YMME5WUV7XGQZOE2I5H7UWQ7TSTP2LGV7D4AZO4T62XOWBE4ZEKA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.445450294 (25h0m8.339158904s) [noise=true , address=145.79.227.30:4762 , piece_id=YLBYKHJNJ2J3YYVYHRWEZTFLXZVL5WCKCBWXM526XF4KKCYA4RAQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.964879212 (25h0m8.858587821s) [noise=true , address=145.79.227.3:4151 , piece_id=QVFK6RJITYJIW6ZJRUYFU3P2NEMMVANA36O6WXPS7Z7FNFSYQWUA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.027795357 (25h0m8.921503966s) [noise=true , address=145.79.227.49:4339 , piece_id=NWDULUWHMV4PIMWMNO5XYXSAGVLNPLSIPYSDA32OZB2MBGBYPEUA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.267543450 (25h0m9.161252059s) [noise=true , address=145.79.227.28:4727 , piece_id=U5CDDUXQZ2BB7SPFAIWQXRGALJQTOWFNWGJLU3CGXKBUHYIQBIGQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.337643945 (25h0m9.231352554s) [noise=true , address=145.79.227.10:4401 , piece_id=C2X52SPGSNM2CB6TFGBCT6HO3MXSEMX3CCZQASDNFGWIEW2EGIGQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.380001967 (25h0m9.273710576s) [noise=true , address=145.79.227.32:4835 , piece_id=6PP7J34OCTKYEDEFQJWDUHO5H4Z6VG7LNG3SL7WLITSKMX43HAYQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.850164228 (25h0m9.743872837s) [noise=true , address=77.90.31.13:4627 , piece_id=TJFFMEVYN7M52FH5UP3WNAGC7HWF5GSEUPQJZ5KJWTJZWKZNAWXQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:13.183873887 (25h0m10.077582496s) [noise=true , address=77.90.31.19:4842 , piece_id=3VJ3J4JQPSJEVFSNSYDVHRFZMMCBMNZRBYEGJOOU4Q4HFEDDOKHQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:13.361354387 (25h0m10.255063026s) [noise=true , address=45.91.116.4:4937 , piece_id=MJHRB2I4KWRJJ3MHGW657D53JFDARTHOBRUEHQZEUA4DRFVK7ZUA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:13.361383082 (25h0m10.255091691s) [seg_pos={0 0}, upload=21929]: Satisfied cohort requirements after 13 pieces Jan 16 16:56:13.361615564 (25h0m10.255324173s) [noise=true , address=145.79.227.34:4906 , piece_id=G2GHF4S5NVJP5R43EN6NDEJ4FCPCLSL5K3TO4UADJ6ZANME54KOQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Jan 16 16:56:13.361624061 (25h0m10.25533267s) [noise=true , address=145.79.227.34:4906 , piece_id=G2GHF4S5NVJP5R43EN6NDEJ4FCPCLSL5K3TO4UADJ6ZANME54KOQ, seg_pos={0 0}, upload=21929]: Long tail cancellation due to optimal threshold Jan 16 16:56:13.361904004 (25h0m10.255612623s) [noise=true , address=145.79.227.38:4977 , piece_id=JXV6MGOVSAA2URSIF2KFFQJJEGWQWFVT65PWVDW5MEH5ITSDYFAA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8): protocol: read tcp 188.166.225.170:42244->145.79.227.38:4977: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 16 16:56:13.361925344 (25h0m10.255633963s) [noise=true , address=145.79.227.38:4977 , piece_id=JXV6MGOVSAA2URSIF2KFFQJJEGWQWFVT65PWVDW5MEH5ITSDYFAA, seg_pos={0 0}, upload=21929]: Long tail cancellation due to optimal threshold Jan 16 16:56:13.361971022 (25h0m10.255679631s) [noise=true , address=45.91.116.5:4300 , piece_id=M2F4W7IGAEKEXJXZRENON55A2E7DNSCZTPKWYJFY3NKNTBIWG2YA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Jan 16 16:56:13.361978977 (25h0m10.255687596s) [noise=true , address=45.91.116.5:4300 , piece_id=M2F4W7IGAEKEXJXZRENON55A2E7DNSCZTPKWYJFY3NKNTBIWG2YA, seg_pos={0 0}, upload=21929]: Long tail cancellation due to optimal threshold Jan 16 16:56:13.362041817 (25h0m10.255750426s) [seg_pos={0 0}, upload=21929]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 16 16:56:13.362197272 (25h0m10.255905881s) [upload=21929]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 16 16:56:13.362205868 (25h0m10.255914477s) [upload=21929]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 16 16:56:13.965177888 (25h0m10.858886497s) [upload=21929]: Done uploading segments... Jan 16 16:56:13.965204158 (25h0m10.858912767s) [upload=21929]: Upload finished. err: <nil> Duration: 5.76753295s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 53 out of 60 is failed (threshold=6) [X.XXXXXXXXXXXXXXXXXXXX.XXXXXX.X..XXXXXXX.XXXXXXXXXXXXXX.XXXX] duration: 5.000683533s trace: https://jaeger.dev.storj.io/trace/79556a24a0b422ce FAIL (2026-01-16 17:08:51 UTC) orbiter warning: context deadline exceeded Duration: 5.000702819s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 53 out of 60 is failed (threshold=6) [XX.XXXXXXXXXXXXXXXXXXXX.XXXXXX.X..XXXXXXX.XXXXXXXXXXXXXX.XXX] duration: 5.001219022s trace: https://jaeger.dev.storj.io/trace/70043ea070387fba FAIL (2026-01-16 17:08:32 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001241996s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................................XX..................] duration: 5.728190931s trace: https://jaeger.dev.storj.io/trace/ff1eaafc0d95989 FAIL (2026-01-16 17:08:28 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 16 16:56:06.586316499 (25h0m3.480025108s) [upload=21929]: Starting upload Jan 16 16:56:06.586371244 (25h0m3.480079853s) [upload=21929]: Done starting upload Jan 16 16:56:06.586529134 (25h0m3.480237763s) [upload=21929]: Uploading segments... Jan 16 16:56:06.586687636 (25h0m3.480396275s) [upload=21929]: Got next segment. Inline: false Jan 16 16:56:06.586750094 (25h0m3.480458723s) [upload=21929]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 16 16:56:06.586757138 (25h0m3.480465767s) [upload=21929]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 16 16:56:08.120364931 (25h0m5.01407354s) [seg_pos={0 0}, upload=21929]: Begin upload segment... Jan 16 16:56:08.120413474 (25h0m5.014122083s) [seg_pos={0 0}, upload=21929]: Done begin upload segment. Jan 16 16:56:08.120440615 (25h0m5.014149224s) [noise=true , address=145.79.227.2:4115 , piece_id=YMME5WUV7XGQZOE2I5H7UWQ7TSTP2LGV7D4AZO4T62XOWBE4ZEKA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.120799800 (25h0m5.014508419s) [noise=true , address=145.79.227.32:4835 , piece_id=6PP7J34OCTKYEDEFQJWDUHO5H4Z6VG7LNG3SL7WLITSKMX43HAYQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.121695454 (25h0m5.015404084s) [noise=true , address=45.91.116.4:4937 , piece_id=MJHRB2I4KWRJJ3MHGW657D53JFDARTHOBRUEHQZEUA4DRFVK7ZUA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.122317739 (25h0m5.016026349s) [noise=true , address=145.79.227.30:4762 , piece_id=YLBYKHJNJ2J3YYVYHRWEZTFLXZVL5WCKCBWXM526XF4KKCYA4RAQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.123161636 (25h0m5.016870245s) [noise=true , address=45.91.116.14:4190 , piece_id=ZBC7C5E5V7B4VAGI2JHD7T4E6BGVV74YNNAROZYBITS2N23BJDWA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.123395732 (25h0m5.017104351s) [noise=true , address=145.79.227.28:4727 , piece_id=U5CDDUXQZ2BB7SPFAIWQXRGALJQTOWFNWGJLU3CGXKBUHYIQBIGQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.123848494 (25h0m5.017557113s) [noise=true , address=77.90.31.19:4842 , piece_id=3VJ3J4JQPSJEVFSNSYDVHRFZMMCBMNZRBYEGJOOU4Q4HFEDDOKHQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.123868092 (25h0m5.017576731s) [noise=true , address=145.79.227.15:4873 , piece_id=EVJGQ5JNJI37QIPQDJPFOPMT3JAHPBZXQG7DBRUZXJMKXAIADWSA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.124249808 (25h0m5.017958417s) [noise=true , address=145.79.227.38:4977 , piece_id=JXV6MGOVSAA2URSIF2KFFQJJEGWQWFVT65PWVDW5MEH5ITSDYFAA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.124389284 (25h0m5.018097893s) [noise=true , address=145.79.227.49:4339 , piece_id=NWDULUWHMV4PIMWMNO5XYXSAGVLNPLSIPYSDA32OZB2MBGBYPEUA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.124619222 (25h0m5.018327831s) [noise=true , address=145.79.227.33:4870 , piece_id=7I3KU7EKC2MHBJ6TOGO5ZBMJSPYU4E26D5373M2R53IZAZJLWVOA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.124999566 (25h0m5.018708175s) [noise=true , address=145.79.227.3:4151 , piece_id=QVFK6RJITYJIW6ZJRUYFU3P2NEMMVANA36O6WXPS7Z7FNFSYQWUA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.125202182 (25h0m5.018910801s) [noise=true , address=77.90.31.13:4627 , piece_id=TJFFMEVYN7M52FH5UP3WNAGC7HWF5GSEUPQJZ5KJWTJZWKZNAWXQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.125576805 (25h0m5.019285414s) [noise=true , address=45.91.116.5:4300 , piece_id=M2F4W7IGAEKEXJXZRENON55A2E7DNSCZTPKWYJFY3NKNTBIWG2YA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.125952470 (25h0m5.019661079s) [noise=true , address=145.79.227.10:4401 , piece_id=C2X52SPGSNM2CB6TFGBCT6HO3MXSEMX3CCZQASDNFGWIEW2EGIGQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.126492900 (25h0m5.020201509s) [noise=true , address=145.79.227.34:4906 , piece_id=G2GHF4S5NVJP5R43EN6NDEJ4FCPCLSL5K3TO4UADJ6ZANME54KOQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:10.730578935 (25h0m7.624287544s) [upload=21929]: Next returned nil segment Jan 16 16:56:10.730594134 (25h0m7.624302753s) [upload=21929]: Waiting for error group managing segments... Jan 16 16:56:11.329683279 (25h0m8.223391888s) [noise=true , address=145.79.227.15:4873 , piece_id=EVJGQ5JNJI37QIPQDJPFOPMT3JAHPBZXQG7DBRUZXJMKXAIADWSA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.338893335 (25h0m8.232601944s) [noise=true , address=45.91.116.14:4190 , piece_id=ZBC7C5E5V7B4VAGI2JHD7T4E6BGVV74YNNAROZYBITS2N23BJDWA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.400223651 (25h0m8.29393226s) [noise=true , address=145.79.227.33:4870 , piece_id=7I3KU7EKC2MHBJ6TOGO5ZBMJSPYU4E26D5373M2R53IZAZJLWVOA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.400340974 (25h0m8.294049583s) [noise=true , address=145.79.227.2:4115 , piece_id=YMME5WUV7XGQZOE2I5H7UWQ7TSTP2LGV7D4AZO4T62XOWBE4ZEKA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.445450294 (25h0m8.339158904s) [noise=true , address=145.79.227.30:4762 , piece_id=YLBYKHJNJ2J3YYVYHRWEZTFLXZVL5WCKCBWXM526XF4KKCYA4RAQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.964879212 (25h0m8.858587821s) [noise=true , address=145.79.227.3:4151 , piece_id=QVFK6RJITYJIW6ZJRUYFU3P2NEMMVANA36O6WXPS7Z7FNFSYQWUA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.027795357 (25h0m8.921503966s) [noise=true , address=145.79.227.49:4339 , piece_id=NWDULUWHMV4PIMWMNO5XYXSAGVLNPLSIPYSDA32OZB2MBGBYPEUA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.267543450 (25h0m9.161252059s) [noise=true , address=145.79.227.28:4727 , piece_id=U5CDDUXQZ2BB7SPFAIWQXRGALJQTOWFNWGJLU3CGXKBUHYIQBIGQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.337643945 (25h0m9.231352554s) [noise=true , address=145.79.227.10:4401 , piece_id=C2X52SPGSNM2CB6TFGBCT6HO3MXSEMX3CCZQASDNFGWIEW2EGIGQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.380001967 (25h0m9.273710576s) [noise=true , address=145.79.227.32:4835 , piece_id=6PP7J34OCTKYEDEFQJWDUHO5H4Z6VG7LNG3SL7WLITSKMX43HAYQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.850164228 (25h0m9.743872837s) [noise=true , address=77.90.31.13:4627 , piece_id=TJFFMEVYN7M52FH5UP3WNAGC7HWF5GSEUPQJZ5KJWTJZWKZNAWXQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:13.183873887 (25h0m10.077582496s) [noise=true , address=77.90.31.19:4842 , piece_id=3VJ3J4JQPSJEVFSNSYDVHRFZMMCBMNZRBYEGJOOU4Q4HFEDDOKHQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:13.361354387 (25h0m10.255063026s) [noise=true , address=45.91.116.4:4937 , piece_id=MJHRB2I4KWRJJ3MHGW657D53JFDARTHOBRUEHQZEUA4DRFVK7ZUA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:13.361383082 (25h0m10.255091691s) [seg_pos={0 0}, upload=21929]: Satisfied cohort requirements after 13 pieces Jan 16 16:56:13.361615564 (25h0m10.255324173s) [noise=true , address=145.79.227.34:4906 , piece_id=G2GHF4S5NVJP5R43EN6NDEJ4FCPCLSL5K3TO4UADJ6ZANME54KOQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Jan 16 16:56:13.361624061 (25h0m10.25533267s) [noise=true , address=145.79.227.34:4906 , piece_id=G2GHF4S5NVJP5R43EN6NDEJ4FCPCLSL5K3TO4UADJ6ZANME54KOQ, seg_pos={0 0}, upload=21929]: Long tail cancellation due to optimal threshold Jan 16 16:56:13.361904004 (25h0m10.255612623s) [noise=true , address=145.79.227.38:4977 , piece_id=JXV6MGOVSAA2URSIF2KFFQJJEGWQWFVT65PWVDW5MEH5ITSDYFAA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8): protocol: read tcp 188.166.225.170:42244->145.79.227.38:4977: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 16 16:56:13.361925344 (25h0m10.255633963s) [noise=true , address=145.79.227.38:4977 , piece_id=JXV6MGOVSAA2URSIF2KFFQJJEGWQWFVT65PWVDW5MEH5ITSDYFAA, seg_pos={0 0}, upload=21929]: Long tail cancellation due to optimal threshold Jan 16 16:56:13.361971022 (25h0m10.255679631s) [noise=true , address=45.91.116.5:4300 , piece_id=M2F4W7IGAEKEXJXZRENON55A2E7DNSCZTPKWYJFY3NKNTBIWG2YA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Jan 16 16:56:13.361978977 (25h0m10.255687596s) [noise=true , address=45.91.116.5:4300 , piece_id=M2F4W7IGAEKEXJXZRENON55A2E7DNSCZTPKWYJFY3NKNTBIWG2YA, seg_pos={0 0}, upload=21929]: Long tail cancellation due to optimal threshold Jan 16 16:56:13.362041817 (25h0m10.255750426s) [seg_pos={0 0}, upload=21929]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 16 16:56:13.362197272 (25h0m10.255905881s) [upload=21929]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 16 16:56:13.362205868 (25h0m10.255914477s) [upload=21929]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 16 16:56:13.965177888 (25h0m10.858886497s) [upload=21929]: Done uploading segments... Jan 16 16:56:13.965204158 (25h0m10.858912767s) [upload=21929]: Upload finished. err: <nil> Duration: 5.728220838s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [............X.......................................XX......] duration: 22.432176255s trace: https://jaeger.dev.storj.io/trace/70c2b0731de5fb9c FAIL (2026-01-16 17:08:21 UTC) orbiter warning: context deadline exceeded Duration: 22.432202186s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................................................XXX........] duration: 17.852066467s trace: https://jaeger.dev.storj.io/trace/521596a1b50467d9 FAIL (2026-01-16 17:08:13 UTC) orbiter warning: context deadline exceeded Duration: 17.85209443s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................................XX.................] duration: 5.719014209s trace: https://jaeger.dev.storj.io/trace/ff1eaafc0d95989 FAIL (2026-01-16 17:08:06 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 16 16:56:06.586316499 (25h0m3.480025108s) [upload=21929]: Starting upload Jan 16 16:56:06.586371244 (25h0m3.480079853s) [upload=21929]: Done starting upload Jan 16 16:56:06.586529134 (25h0m3.480237763s) [upload=21929]: Uploading segments... Jan 16 16:56:06.586687636 (25h0m3.480396275s) [upload=21929]: Got next segment. Inline: false Jan 16 16:56:06.586750094 (25h0m3.480458723s) [upload=21929]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 16 16:56:06.586757138 (25h0m3.480465767s) [upload=21929]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 16 16:56:08.120364931 (25h0m5.01407354s) [seg_pos={0 0}, upload=21929]: Begin upload segment... Jan 16 16:56:08.120413474 (25h0m5.014122083s) [seg_pos={0 0}, upload=21929]: Done begin upload segment. Jan 16 16:56:08.120440615 (25h0m5.014149224s) [noise=true , address=145.79.227.2:4115 , piece_id=YMME5WUV7XGQZOE2I5H7UWQ7TSTP2LGV7D4AZO4T62XOWBE4ZEKA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.120799800 (25h0m5.014508419s) [noise=true , address=145.79.227.32:4835 , piece_id=6PP7J34OCTKYEDEFQJWDUHO5H4Z6VG7LNG3SL7WLITSKMX43HAYQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.121695454 (25h0m5.015404084s) [noise=true , address=45.91.116.4:4937 , piece_id=MJHRB2I4KWRJJ3MHGW657D53JFDARTHOBRUEHQZEUA4DRFVK7ZUA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.122317739 (25h0m5.016026349s) [noise=true , address=145.79.227.30:4762 , piece_id=YLBYKHJNJ2J3YYVYHRWEZTFLXZVL5WCKCBWXM526XF4KKCYA4RAQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.123161636 (25h0m5.016870245s) [noise=true , address=45.91.116.14:4190 , piece_id=ZBC7C5E5V7B4VAGI2JHD7T4E6BGVV74YNNAROZYBITS2N23BJDWA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.123395732 (25h0m5.017104351s) [noise=true , address=145.79.227.28:4727 , piece_id=U5CDDUXQZ2BB7SPFAIWQXRGALJQTOWFNWGJLU3CGXKBUHYIQBIGQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.123848494 (25h0m5.017557113s) [noise=true , address=77.90.31.19:4842 , piece_id=3VJ3J4JQPSJEVFSNSYDVHRFZMMCBMNZRBYEGJOOU4Q4HFEDDOKHQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.123868092 (25h0m5.017576731s) [noise=true , address=145.79.227.15:4873 , piece_id=EVJGQ5JNJI37QIPQDJPFOPMT3JAHPBZXQG7DBRUZXJMKXAIADWSA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.124249808 (25h0m5.017958417s) [noise=true , address=145.79.227.38:4977 , piece_id=JXV6MGOVSAA2URSIF2KFFQJJEGWQWFVT65PWVDW5MEH5ITSDYFAA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.124389284 (25h0m5.018097893s) [noise=true , address=145.79.227.49:4339 , piece_id=NWDULUWHMV4PIMWMNO5XYXSAGVLNPLSIPYSDA32OZB2MBGBYPEUA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.124619222 (25h0m5.018327831s) [noise=true , address=145.79.227.33:4870 , piece_id=7I3KU7EKC2MHBJ6TOGO5ZBMJSPYU4E26D5373M2R53IZAZJLWVOA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.124999566 (25h0m5.018708175s) [noise=true , address=145.79.227.3:4151 , piece_id=QVFK6RJITYJIW6ZJRUYFU3P2NEMMVANA36O6WXPS7Z7FNFSYQWUA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.125202182 (25h0m5.018910801s) [noise=true , address=77.90.31.13:4627 , piece_id=TJFFMEVYN7M52FH5UP3WNAGC7HWF5GSEUPQJZ5KJWTJZWKZNAWXQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.125576805 (25h0m5.019285414s) [noise=true , address=45.91.116.5:4300 , piece_id=M2F4W7IGAEKEXJXZRENON55A2E7DNSCZTPKWYJFY3NKNTBIWG2YA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.125952470 (25h0m5.019661079s) [noise=true , address=145.79.227.10:4401 , piece_id=C2X52SPGSNM2CB6TFGBCT6HO3MXSEMX3CCZQASDNFGWIEW2EGIGQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.126492900 (25h0m5.020201509s) [noise=true , address=145.79.227.34:4906 , piece_id=G2GHF4S5NVJP5R43EN6NDEJ4FCPCLSL5K3TO4UADJ6ZANME54KOQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:10.730578935 (25h0m7.624287544s) [upload=21929]: Next returned nil segment Jan 16 16:56:10.730594134 (25h0m7.624302753s) [upload=21929]: Waiting for error group managing segments... Jan 16 16:56:11.329683279 (25h0m8.223391888s) [noise=true , address=145.79.227.15:4873 , piece_id=EVJGQ5JNJI37QIPQDJPFOPMT3JAHPBZXQG7DBRUZXJMKXAIADWSA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.338893335 (25h0m8.232601944s) [noise=true , address=45.91.116.14:4190 , piece_id=ZBC7C5E5V7B4VAGI2JHD7T4E6BGVV74YNNAROZYBITS2N23BJDWA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.400223651 (25h0m8.29393226s) [noise=true , address=145.79.227.33:4870 , piece_id=7I3KU7EKC2MHBJ6TOGO5ZBMJSPYU4E26D5373M2R53IZAZJLWVOA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.400340974 (25h0m8.294049583s) [noise=true , address=145.79.227.2:4115 , piece_id=YMME5WUV7XGQZOE2I5H7UWQ7TSTP2LGV7D4AZO4T62XOWBE4ZEKA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.445450294 (25h0m8.339158904s) [noise=true , address=145.79.227.30:4762 , piece_id=YLBYKHJNJ2J3YYVYHRWEZTFLXZVL5WCKCBWXM526XF4KKCYA4RAQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.964879212 (25h0m8.858587821s) [noise=true , address=145.79.227.3:4151 , piece_id=QVFK6RJITYJIW6ZJRUYFU3P2NEMMVANA36O6WXPS7Z7FNFSYQWUA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.027795357 (25h0m8.921503966s) [noise=true , address=145.79.227.49:4339 , piece_id=NWDULUWHMV4PIMWMNO5XYXSAGVLNPLSIPYSDA32OZB2MBGBYPEUA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.267543450 (25h0m9.161252059s) [noise=true , address=145.79.227.28:4727 , piece_id=U5CDDUXQZ2BB7SPFAIWQXRGALJQTOWFNWGJLU3CGXKBUHYIQBIGQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.337643945 (25h0m9.231352554s) [noise=true , address=145.79.227.10:4401 , piece_id=C2X52SPGSNM2CB6TFGBCT6HO3MXSEMX3CCZQASDNFGWIEW2EGIGQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.380001967 (25h0m9.273710576s) [noise=true , address=145.79.227.32:4835 , piece_id=6PP7J34OCTKYEDEFQJWDUHO5H4Z6VG7LNG3SL7WLITSKMX43HAYQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.850164228 (25h0m9.743872837s) [noise=true , address=77.90.31.13:4627 , piece_id=TJFFMEVYN7M52FH5UP3WNAGC7HWF5GSEUPQJZ5KJWTJZWKZNAWXQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:13.183873887 (25h0m10.077582496s) [noise=true , address=77.90.31.19:4842 , piece_id=3VJ3J4JQPSJEVFSNSYDVHRFZMMCBMNZRBYEGJOOU4Q4HFEDDOKHQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:13.361354387 (25h0m10.255063026s) [noise=true , address=45.91.116.4:4937 , piece_id=MJHRB2I4KWRJJ3MHGW657D53JFDARTHOBRUEHQZEUA4DRFVK7ZUA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:13.361383082 (25h0m10.255091691s) [seg_pos={0 0}, upload=21929]: Satisfied cohort requirements after 13 pieces Jan 16 16:56:13.361615564 (25h0m10.255324173s) [noise=true , address=145.79.227.34:4906 , piece_id=G2GHF4S5NVJP5R43EN6NDEJ4FCPCLSL5K3TO4UADJ6ZANME54KOQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Jan 16 16:56:13.361624061 (25h0m10.25533267s) [noise=true , address=145.79.227.34:4906 , piece_id=G2GHF4S5NVJP5R43EN6NDEJ4FCPCLSL5K3TO4UADJ6ZANME54KOQ, seg_pos={0 0}, upload=21929]: Long tail cancellation due to optimal threshold Jan 16 16:56:13.361904004 (25h0m10.255612623s) [noise=true , address=145.79.227.38:4977 , piece_id=JXV6MGOVSAA2URSIF2KFFQJJEGWQWFVT65PWVDW5MEH5ITSDYFAA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8): protocol: read tcp 188.166.225.170:42244->145.79.227.38:4977: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 16 16:56:13.361925344 (25h0m10.255633963s) [noise=true , address=145.79.227.38:4977 , piece_id=JXV6MGOVSAA2URSIF2KFFQJJEGWQWFVT65PWVDW5MEH5ITSDYFAA, seg_pos={0 0}, upload=21929]: Long tail cancellation due to optimal threshold Jan 16 16:56:13.361971022 (25h0m10.255679631s) [noise=true , address=45.91.116.5:4300 , piece_id=M2F4W7IGAEKEXJXZRENON55A2E7DNSCZTPKWYJFY3NKNTBIWG2YA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Jan 16 16:56:13.361978977 (25h0m10.255687596s) [noise=true , address=45.91.116.5:4300 , piece_id=M2F4W7IGAEKEXJXZRENON55A2E7DNSCZTPKWYJFY3NKNTBIWG2YA, seg_pos={0 0}, upload=21929]: Long tail cancellation due to optimal threshold Jan 16 16:56:13.362041817 (25h0m10.255750426s) [seg_pos={0 0}, upload=21929]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 16 16:56:13.362197272 (25h0m10.255905881s) [upload=21929]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 16 16:56:13.362205868 (25h0m10.255914477s) [upload=21929]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 16 16:56:13.965177888 (25h0m10.858886497s) [upload=21929]: Done uploading segments... Jan 16 16:56:13.965204158 (25h0m10.858912767s) [upload=21929]: Upload finished. err: <nil> Duration: 5.719037983s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 53 out of 60 is failed (threshold=6) [XXX.XXXXXXXXXXXXXXXXXXXX.XXXXXX.X..XXXXXXX.XXXXXXXXXXXXXX.XX] duration: 5.000974547s trace: https://jaeger.dev.storj.io/trace/3c040c15f0e8893b FAIL (2026-01-16 17:08:00 UTC) orbiter warning: context deadline exceeded Duration: 5.000993473s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................................................XXX.......] duration: 29.645250174s trace: https://jaeger.dev.storj.io/trace/521596a1b50467d9 FAIL (2026-01-16 17:07:49 UTC) orbiter warning: context deadline exceeded Duration: 29.645278899s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................................XX................] duration: 5.821408122s trace: https://jaeger.dev.storj.io/trace/ff1eaafc0d95989 FAIL (2026-01-16 17:07:45 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 16 16:56:06.586316499 (25h0m3.480025108s) [upload=21929]: Starting upload Jan 16 16:56:06.586371244 (25h0m3.480079853s) [upload=21929]: Done starting upload Jan 16 16:56:06.586529134 (25h0m3.480237763s) [upload=21929]: Uploading segments... Jan 16 16:56:06.586687636 (25h0m3.480396275s) [upload=21929]: Got next segment. Inline: false Jan 16 16:56:06.586750094 (25h0m3.480458723s) [upload=21929]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 16 16:56:06.586757138 (25h0m3.480465767s) [upload=21929]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 16 16:56:08.120364931 (25h0m5.01407354s) [seg_pos={0 0}, upload=21929]: Begin upload segment... Jan 16 16:56:08.120413474 (25h0m5.014122083s) [seg_pos={0 0}, upload=21929]: Done begin upload segment. Jan 16 16:56:08.120440615 (25h0m5.014149224s) [noise=true , address=145.79.227.2:4115 , piece_id=YMME5WUV7XGQZOE2I5H7UWQ7TSTP2LGV7D4AZO4T62XOWBE4ZEKA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.120799800 (25h0m5.014508419s) [noise=true , address=145.79.227.32:4835 , piece_id=6PP7J34OCTKYEDEFQJWDUHO5H4Z6VG7LNG3SL7WLITSKMX43HAYQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.121695454 (25h0m5.015404084s) [noise=true , address=45.91.116.4:4937 , piece_id=MJHRB2I4KWRJJ3MHGW657D53JFDARTHOBRUEHQZEUA4DRFVK7ZUA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.122317739 (25h0m5.016026349s) [noise=true , address=145.79.227.30:4762 , piece_id=YLBYKHJNJ2J3YYVYHRWEZTFLXZVL5WCKCBWXM526XF4KKCYA4RAQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.123161636 (25h0m5.016870245s) [noise=true , address=45.91.116.14:4190 , piece_id=ZBC7C5E5V7B4VAGI2JHD7T4E6BGVV74YNNAROZYBITS2N23BJDWA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.123395732 (25h0m5.017104351s) [noise=true , address=145.79.227.28:4727 , piece_id=U5CDDUXQZ2BB7SPFAIWQXRGALJQTOWFNWGJLU3CGXKBUHYIQBIGQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.123848494 (25h0m5.017557113s) [noise=true , address=77.90.31.19:4842 , piece_id=3VJ3J4JQPSJEVFSNSYDVHRFZMMCBMNZRBYEGJOOU4Q4HFEDDOKHQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.123868092 (25h0m5.017576731s) [noise=true , address=145.79.227.15:4873 , piece_id=EVJGQ5JNJI37QIPQDJPFOPMT3JAHPBZXQG7DBRUZXJMKXAIADWSA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.124249808 (25h0m5.017958417s) [noise=true , address=145.79.227.38:4977 , piece_id=JXV6MGOVSAA2URSIF2KFFQJJEGWQWFVT65PWVDW5MEH5ITSDYFAA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.124389284 (25h0m5.018097893s) [noise=true , address=145.79.227.49:4339 , piece_id=NWDULUWHMV4PIMWMNO5XYXSAGVLNPLSIPYSDA32OZB2MBGBYPEUA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.124619222 (25h0m5.018327831s) [noise=true , address=145.79.227.33:4870 , piece_id=7I3KU7EKC2MHBJ6TOGO5ZBMJSPYU4E26D5373M2R53IZAZJLWVOA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.124999566 (25h0m5.018708175s) [noise=true , address=145.79.227.3:4151 , piece_id=QVFK6RJITYJIW6ZJRUYFU3P2NEMMVANA36O6WXPS7Z7FNFSYQWUA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.125202182 (25h0m5.018910801s) [noise=true , address=77.90.31.13:4627 , piece_id=TJFFMEVYN7M52FH5UP3WNAGC7HWF5GSEUPQJZ5KJWTJZWKZNAWXQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.125576805 (25h0m5.019285414s) [noise=true , address=45.91.116.5:4300 , piece_id=M2F4W7IGAEKEXJXZRENON55A2E7DNSCZTPKWYJFY3NKNTBIWG2YA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.125952470 (25h0m5.019661079s) [noise=true , address=145.79.227.10:4401 , piece_id=C2X52SPGSNM2CB6TFGBCT6HO3MXSEMX3CCZQASDNFGWIEW2EGIGQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.126492900 (25h0m5.020201509s) [noise=true , address=145.79.227.34:4906 , piece_id=G2GHF4S5NVJP5R43EN6NDEJ4FCPCLSL5K3TO4UADJ6ZANME54KOQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:10.730578935 (25h0m7.624287544s) [upload=21929]: Next returned nil segment Jan 16 16:56:10.730594134 (25h0m7.624302753s) [upload=21929]: Waiting for error group managing segments... Jan 16 16:56:11.329683279 (25h0m8.223391888s) [noise=true , address=145.79.227.15:4873 , piece_id=EVJGQ5JNJI37QIPQDJPFOPMT3JAHPBZXQG7DBRUZXJMKXAIADWSA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.338893335 (25h0m8.232601944s) [noise=true , address=45.91.116.14:4190 , piece_id=ZBC7C5E5V7B4VAGI2JHD7T4E6BGVV74YNNAROZYBITS2N23BJDWA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.400223651 (25h0m8.29393226s) [noise=true , address=145.79.227.33:4870 , piece_id=7I3KU7EKC2MHBJ6TOGO5ZBMJSPYU4E26D5373M2R53IZAZJLWVOA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.400340974 (25h0m8.294049583s) [noise=true , address=145.79.227.2:4115 , piece_id=YMME5WUV7XGQZOE2I5H7UWQ7TSTP2LGV7D4AZO4T62XOWBE4ZEKA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.445450294 (25h0m8.339158904s) [noise=true , address=145.79.227.30:4762 , piece_id=YLBYKHJNJ2J3YYVYHRWEZTFLXZVL5WCKCBWXM526XF4KKCYA4RAQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.964879212 (25h0m8.858587821s) [noise=true , address=145.79.227.3:4151 , piece_id=QVFK6RJITYJIW6ZJRUYFU3P2NEMMVANA36O6WXPS7Z7FNFSYQWUA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.027795357 (25h0m8.921503966s) [noise=true , address=145.79.227.49:4339 , piece_id=NWDULUWHMV4PIMWMNO5XYXSAGVLNPLSIPYSDA32OZB2MBGBYPEUA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.267543450 (25h0m9.161252059s) [noise=true , address=145.79.227.28:4727 , piece_id=U5CDDUXQZ2BB7SPFAIWQXRGALJQTOWFNWGJLU3CGXKBUHYIQBIGQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.337643945 (25h0m9.231352554s) [noise=true , address=145.79.227.10:4401 , piece_id=C2X52SPGSNM2CB6TFGBCT6HO3MXSEMX3CCZQASDNFGWIEW2EGIGQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.380001967 (25h0m9.273710576s) [noise=true , address=145.79.227.32:4835 , piece_id=6PP7J34OCTKYEDEFQJWDUHO5H4Z6VG7LNG3SL7WLITSKMX43HAYQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.850164228 (25h0m9.743872837s) [noise=true , address=77.90.31.13:4627 , piece_id=TJFFMEVYN7M52FH5UP3WNAGC7HWF5GSEUPQJZ5KJWTJZWKZNAWXQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:13.183873887 (25h0m10.077582496s) [noise=true , address=77.90.31.19:4842 , piece_id=3VJ3J4JQPSJEVFSNSYDVHRFZMMCBMNZRBYEGJOOU4Q4HFEDDOKHQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:13.361354387 (25h0m10.255063026s) [noise=true , address=45.91.116.4:4937 , piece_id=MJHRB2I4KWRJJ3MHGW657D53JFDARTHOBRUEHQZEUA4DRFVK7ZUA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:13.361383082 (25h0m10.255091691s) [seg_pos={0 0}, upload=21929]: Satisfied cohort requirements after 13 pieces Jan 16 16:56:13.361615564 (25h0m10.255324173s) [noise=true , address=145.79.227.34:4906 , piece_id=G2GHF4S5NVJP5R43EN6NDEJ4FCPCLSL5K3TO4UADJ6ZANME54KOQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Jan 16 16:56:13.361624061 (25h0m10.25533267s) [noise=true , address=145.79.227.34:4906 , piece_id=G2GHF4S5NVJP5R43EN6NDEJ4FCPCLSL5K3TO4UADJ6ZANME54KOQ, seg_pos={0 0}, upload=21929]: Long tail cancellation due to optimal threshold Jan 16 16:56:13.361904004 (25h0m10.255612623s) [noise=true , address=145.79.227.38:4977 , piece_id=JXV6MGOVSAA2URSIF2KFFQJJEGWQWFVT65PWVDW5MEH5ITSDYFAA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8): protocol: read tcp 188.166.225.170:42244->145.79.227.38:4977: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 16 16:56:13.361925344 (25h0m10.255633963s) [noise=true , address=145.79.227.38:4977 , piece_id=JXV6MGOVSAA2URSIF2KFFQJJEGWQWFVT65PWVDW5MEH5ITSDYFAA, seg_pos={0 0}, upload=21929]: Long tail cancellation due to optimal threshold Jan 16 16:56:13.361971022 (25h0m10.255679631s) [noise=true , address=45.91.116.5:4300 , piece_id=M2F4W7IGAEKEXJXZRENON55A2E7DNSCZTPKWYJFY3NKNTBIWG2YA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Jan 16 16:56:13.361978977 (25h0m10.255687596s) [noise=true , address=45.91.116.5:4300 , piece_id=M2F4W7IGAEKEXJXZRENON55A2E7DNSCZTPKWYJFY3NKNTBIWG2YA, seg_pos={0 0}, upload=21929]: Long tail cancellation due to optimal threshold Jan 16 16:56:13.362041817 (25h0m10.255750426s) [seg_pos={0 0}, upload=21929]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 16 16:56:13.362197272 (25h0m10.255905881s) [upload=21929]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 16 16:56:13.362205868 (25h0m10.255914477s) [upload=21929]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 16 16:56:13.965177888 (25h0m10.858886497s) [upload=21929]: Done uploading segments... Jan 16 16:56:13.965204158 (25h0m10.858912767s) [upload=21929]: Upload finished. err: <nil> Duration: 5.821434171s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.............X.......................................XX.....] duration: 13.113538058s trace: https://jaeger.dev.storj.io/trace/70c2b0731de5fb9c FAIL (2026-01-16 17:07:45 UTC) orbiter warning: context deadline exceeded Duration: 13.113695408s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 53 out of 60 is failed (threshold=6) [XXXX.XXXXXXXXXXXXXXXXXXXX.XXXXXX.X..XXXXXXX.XXXXXXXXXXXXXX.X] duration: 5.001352185s trace: https://jaeger.dev.storj.io/trace/6f482aeb9d03f90f FAIL (2026-01-16 17:07:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001380379s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................................................XXX......] duration: 18.824869622s trace: https://jaeger.dev.storj.io/trace/521596a1b50467d9 FAIL (2026-01-16 17:07:14 UTC) orbiter warning: context deadline exceeded Duration: 18.824906503s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................................XX...............] duration: 5.818815535s trace: https://jaeger.dev.storj.io/trace/ff1eaafc0d95989 FAIL (2026-01-16 17:06:52 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 16 16:56:06.586316499 (25h0m3.480025108s) [upload=21929]: Starting upload Jan 16 16:56:06.586371244 (25h0m3.480079853s) [upload=21929]: Done starting upload Jan 16 16:56:06.586529134 (25h0m3.480237763s) [upload=21929]: Uploading segments... Jan 16 16:56:06.586687636 (25h0m3.480396275s) [upload=21929]: Got next segment. Inline: false Jan 16 16:56:06.586750094 (25h0m3.480458723s) [upload=21929]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 16 16:56:06.586757138 (25h0m3.480465767s) [upload=21929]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 16 16:56:08.120364931 (25h0m5.01407354s) [seg_pos={0 0}, upload=21929]: Begin upload segment... Jan 16 16:56:08.120413474 (25h0m5.014122083s) [seg_pos={0 0}, upload=21929]: Done begin upload segment. Jan 16 16:56:08.120440615 (25h0m5.014149224s) [noise=true , address=145.79.227.2:4115 , piece_id=YMME5WUV7XGQZOE2I5H7UWQ7TSTP2LGV7D4AZO4T62XOWBE4ZEKA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.120799800 (25h0m5.014508419s) [noise=true , address=145.79.227.32:4835 , piece_id=6PP7J34OCTKYEDEFQJWDUHO5H4Z6VG7LNG3SL7WLITSKMX43HAYQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.121695454 (25h0m5.015404084s) [noise=true , address=45.91.116.4:4937 , piece_id=MJHRB2I4KWRJJ3MHGW657D53JFDARTHOBRUEHQZEUA4DRFVK7ZUA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.122317739 (25h0m5.016026349s) [noise=true , address=145.79.227.30:4762 , piece_id=YLBYKHJNJ2J3YYVYHRWEZTFLXZVL5WCKCBWXM526XF4KKCYA4RAQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.123161636 (25h0m5.016870245s) [noise=true , address=45.91.116.14:4190 , piece_id=ZBC7C5E5V7B4VAGI2JHD7T4E6BGVV74YNNAROZYBITS2N23BJDWA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.123395732 (25h0m5.017104351s) [noise=true , address=145.79.227.28:4727 , piece_id=U5CDDUXQZ2BB7SPFAIWQXRGALJQTOWFNWGJLU3CGXKBUHYIQBIGQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.123848494 (25h0m5.017557113s) [noise=true , address=77.90.31.19:4842 , piece_id=3VJ3J4JQPSJEVFSNSYDVHRFZMMCBMNZRBYEGJOOU4Q4HFEDDOKHQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.123868092 (25h0m5.017576731s) [noise=true , address=145.79.227.15:4873 , piece_id=EVJGQ5JNJI37QIPQDJPFOPMT3JAHPBZXQG7DBRUZXJMKXAIADWSA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.124249808 (25h0m5.017958417s) [noise=true , address=145.79.227.38:4977 , piece_id=JXV6MGOVSAA2URSIF2KFFQJJEGWQWFVT65PWVDW5MEH5ITSDYFAA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.124389284 (25h0m5.018097893s) [noise=true , address=145.79.227.49:4339 , piece_id=NWDULUWHMV4PIMWMNO5XYXSAGVLNPLSIPYSDA32OZB2MBGBYPEUA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.124619222 (25h0m5.018327831s) [noise=true , address=145.79.227.33:4870 , piece_id=7I3KU7EKC2MHBJ6TOGO5ZBMJSPYU4E26D5373M2R53IZAZJLWVOA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.124999566 (25h0m5.018708175s) [noise=true , address=145.79.227.3:4151 , piece_id=QVFK6RJITYJIW6ZJRUYFU3P2NEMMVANA36O6WXPS7Z7FNFSYQWUA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.125202182 (25h0m5.018910801s) [noise=true , address=77.90.31.13:4627 , piece_id=TJFFMEVYN7M52FH5UP3WNAGC7HWF5GSEUPQJZ5KJWTJZWKZNAWXQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.125576805 (25h0m5.019285414s) [noise=true , address=45.91.116.5:4300 , piece_id=M2F4W7IGAEKEXJXZRENON55A2E7DNSCZTPKWYJFY3NKNTBIWG2YA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.125952470 (25h0m5.019661079s) [noise=true , address=145.79.227.10:4401 , piece_id=C2X52SPGSNM2CB6TFGBCT6HO3MXSEMX3CCZQASDNFGWIEW2EGIGQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.126492900 (25h0m5.020201509s) [noise=true , address=145.79.227.34:4906 , piece_id=G2GHF4S5NVJP5R43EN6NDEJ4FCPCLSL5K3TO4UADJ6ZANME54KOQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:10.730578935 (25h0m7.624287544s) [upload=21929]: Next returned nil segment Jan 16 16:56:10.730594134 (25h0m7.624302753s) [upload=21929]: Waiting for error group managing segments... Jan 16 16:56:11.329683279 (25h0m8.223391888s) [noise=true , address=145.79.227.15:4873 , piece_id=EVJGQ5JNJI37QIPQDJPFOPMT3JAHPBZXQG7DBRUZXJMKXAIADWSA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.338893335 (25h0m8.232601944s) [noise=true , address=45.91.116.14:4190 , piece_id=ZBC7C5E5V7B4VAGI2JHD7T4E6BGVV74YNNAROZYBITS2N23BJDWA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.400223651 (25h0m8.29393226s) [noise=true , address=145.79.227.33:4870 , piece_id=7I3KU7EKC2MHBJ6TOGO5ZBMJSPYU4E26D5373M2R53IZAZJLWVOA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.400340974 (25h0m8.294049583s) [noise=true , address=145.79.227.2:4115 , piece_id=YMME5WUV7XGQZOE2I5H7UWQ7TSTP2LGV7D4AZO4T62XOWBE4ZEKA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.445450294 (25h0m8.339158904s) [noise=true , address=145.79.227.30:4762 , piece_id=YLBYKHJNJ2J3YYVYHRWEZTFLXZVL5WCKCBWXM526XF4KKCYA4RAQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.964879212 (25h0m8.858587821s) [noise=true , address=145.79.227.3:4151 , piece_id=QVFK6RJITYJIW6ZJRUYFU3P2NEMMVANA36O6WXPS7Z7FNFSYQWUA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.027795357 (25h0m8.921503966s) [noise=true , address=145.79.227.49:4339 , piece_id=NWDULUWHMV4PIMWMNO5XYXSAGVLNPLSIPYSDA32OZB2MBGBYPEUA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.267543450 (25h0m9.161252059s) [noise=true , address=145.79.227.28:4727 , piece_id=U5CDDUXQZ2BB7SPFAIWQXRGALJQTOWFNWGJLU3CGXKBUHYIQBIGQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.337643945 (25h0m9.231352554s) [noise=true , address=145.79.227.10:4401 , piece_id=C2X52SPGSNM2CB6TFGBCT6HO3MXSEMX3CCZQASDNFGWIEW2EGIGQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.380001967 (25h0m9.273710576s) [noise=true , address=145.79.227.32:4835 , piece_id=6PP7J34OCTKYEDEFQJWDUHO5H4Z6VG7LNG3SL7WLITSKMX43HAYQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.850164228 (25h0m9.743872837s) [noise=true , address=77.90.31.13:4627 , piece_id=TJFFMEVYN7M52FH5UP3WNAGC7HWF5GSEUPQJZ5KJWTJZWKZNAWXQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:13.183873887 (25h0m10.077582496s) [noise=true , address=77.90.31.19:4842 , piece_id=3VJ3J4JQPSJEVFSNSYDVHRFZMMCBMNZRBYEGJOOU4Q4HFEDDOKHQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:13.361354387 (25h0m10.255063026s) [noise=true , address=45.91.116.4:4937 , piece_id=MJHRB2I4KWRJJ3MHGW657D53JFDARTHOBRUEHQZEUA4DRFVK7ZUA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:13.361383082 (25h0m10.255091691s) [seg_pos={0 0}, upload=21929]: Satisfied cohort requirements after 13 pieces Jan 16 16:56:13.361615564 (25h0m10.255324173s) [noise=true , address=145.79.227.34:4906 , piece_id=G2GHF4S5NVJP5R43EN6NDEJ4FCPCLSL5K3TO4UADJ6ZANME54KOQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Jan 16 16:56:13.361624061 (25h0m10.25533267s) [noise=true , address=145.79.227.34:4906 , piece_id=G2GHF4S5NVJP5R43EN6NDEJ4FCPCLSL5K3TO4UADJ6ZANME54KOQ, seg_pos={0 0}, upload=21929]: Long tail cancellation due to optimal threshold Jan 16 16:56:13.361904004 (25h0m10.255612623s) [noise=true , address=145.79.227.38:4977 , piece_id=JXV6MGOVSAA2URSIF2KFFQJJEGWQWFVT65PWVDW5MEH5ITSDYFAA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8): protocol: read tcp 188.166.225.170:42244->145.79.227.38:4977: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 16 16:56:13.361925344 (25h0m10.255633963s) [noise=true , address=145.79.227.38:4977 , piece_id=JXV6MGOVSAA2URSIF2KFFQJJEGWQWFVT65PWVDW5MEH5ITSDYFAA, seg_pos={0 0}, upload=21929]: Long tail cancellation due to optimal threshold Jan 16 16:56:13.361971022 (25h0m10.255679631s) [noise=true , address=45.91.116.5:4300 , piece_id=M2F4W7IGAEKEXJXZRENON55A2E7DNSCZTPKWYJFY3NKNTBIWG2YA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Jan 16 16:56:13.361978977 (25h0m10.255687596s) [noise=true , address=45.91.116.5:4300 , piece_id=M2F4W7IGAEKEXJXZRENON55A2E7DNSCZTPKWYJFY3NKNTBIWG2YA, seg_pos={0 0}, upload=21929]: Long tail cancellation due to optimal threshold Jan 16 16:56:13.362041817 (25h0m10.255750426s) [seg_pos={0 0}, upload=21929]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 16 16:56:13.362197272 (25h0m10.255905881s) [upload=21929]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 16 16:56:13.362205868 (25h0m10.255914477s) [upload=21929]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 16 16:56:13.965177888 (25h0m10.858886497s) [upload=21929]: Done uploading segments... Jan 16 16:56:13.965204158 (25h0m10.858912767s) [upload=21929]: Upload finished. err: <nil> Duration: 5.81884456s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..............X.......................................XX....] duration: 12.040409649s trace: https://jaeger.dev.storj.io/trace/70c2b0731de5fb9c FAIL (2026-01-16 17:06:43 UTC) orbiter warning: context deadline exceeded Duration: 12.04043641s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................................................XXX.....] duration: 19.427628589s trace: https://jaeger.dev.storj.io/trace/521596a1b50467d9 FAIL (2026-01-16 17:06:42 UTC) orbiter warning: context deadline exceeded Duration: 19.427652015s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 52 out of 60 is failed (threshold=6) [.XXXX.XXXXXXXXXXXXXXXXXXXX.XXXXXX.X..XXXXXXX.XXXXXXXXXXXXXX.] duration: 4.399735167s trace: https://jaeger.dev.storj.io/trace/31caf78ed04f4f0e FAIL (2026-01-16 17:06:17 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.399764013s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................................XX..............] duration: 5.775514s trace: https://jaeger.dev.storj.io/trace/ff1eaafc0d95989 FAIL (2026-01-16 17:06:16 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 16 16:56:06.586316499 (25h0m3.480025108s) [upload=21929]: Starting upload Jan 16 16:56:06.586371244 (25h0m3.480079853s) [upload=21929]: Done starting upload Jan 16 16:56:06.586529134 (25h0m3.480237763s) [upload=21929]: Uploading segments... Jan 16 16:56:06.586687636 (25h0m3.480396275s) [upload=21929]: Got next segment. Inline: false Jan 16 16:56:06.586750094 (25h0m3.480458723s) [upload=21929]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 16 16:56:06.586757138 (25h0m3.480465767s) [upload=21929]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 16 16:56:08.120364931 (25h0m5.01407354s) [seg_pos={0 0}, upload=21929]: Begin upload segment... Jan 16 16:56:08.120413474 (25h0m5.014122083s) [seg_pos={0 0}, upload=21929]: Done begin upload segment. Jan 16 16:56:08.120440615 (25h0m5.014149224s) [noise=true , address=145.79.227.2:4115 , piece_id=YMME5WUV7XGQZOE2I5H7UWQ7TSTP2LGV7D4AZO4T62XOWBE4ZEKA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.120799800 (25h0m5.014508419s) [noise=true , address=145.79.227.32:4835 , piece_id=6PP7J34OCTKYEDEFQJWDUHO5H4Z6VG7LNG3SL7WLITSKMX43HAYQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.121695454 (25h0m5.015404084s) [noise=true , address=45.91.116.4:4937 , piece_id=MJHRB2I4KWRJJ3MHGW657D53JFDARTHOBRUEHQZEUA4DRFVK7ZUA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.122317739 (25h0m5.016026349s) [noise=true , address=145.79.227.30:4762 , piece_id=YLBYKHJNJ2J3YYVYHRWEZTFLXZVL5WCKCBWXM526XF4KKCYA4RAQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.123161636 (25h0m5.016870245s) [noise=true , address=45.91.116.14:4190 , piece_id=ZBC7C5E5V7B4VAGI2JHD7T4E6BGVV74YNNAROZYBITS2N23BJDWA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.123395732 (25h0m5.017104351s) [noise=true , address=145.79.227.28:4727 , piece_id=U5CDDUXQZ2BB7SPFAIWQXRGALJQTOWFNWGJLU3CGXKBUHYIQBIGQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.123848494 (25h0m5.017557113s) [noise=true , address=77.90.31.19:4842 , piece_id=3VJ3J4JQPSJEVFSNSYDVHRFZMMCBMNZRBYEGJOOU4Q4HFEDDOKHQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.123868092 (25h0m5.017576731s) [noise=true , address=145.79.227.15:4873 , piece_id=EVJGQ5JNJI37QIPQDJPFOPMT3JAHPBZXQG7DBRUZXJMKXAIADWSA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.124249808 (25h0m5.017958417s) [noise=true , address=145.79.227.38:4977 , piece_id=JXV6MGOVSAA2URSIF2KFFQJJEGWQWFVT65PWVDW5MEH5ITSDYFAA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.124389284 (25h0m5.018097893s) [noise=true , address=145.79.227.49:4339 , piece_id=NWDULUWHMV4PIMWMNO5XYXSAGVLNPLSIPYSDA32OZB2MBGBYPEUA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.124619222 (25h0m5.018327831s) [noise=true , address=145.79.227.33:4870 , piece_id=7I3KU7EKC2MHBJ6TOGO5ZBMJSPYU4E26D5373M2R53IZAZJLWVOA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.124999566 (25h0m5.018708175s) [noise=true , address=145.79.227.3:4151 , piece_id=QVFK6RJITYJIW6ZJRUYFU3P2NEMMVANA36O6WXPS7Z7FNFSYQWUA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.125202182 (25h0m5.018910801s) [noise=true , address=77.90.31.13:4627 , piece_id=TJFFMEVYN7M52FH5UP3WNAGC7HWF5GSEUPQJZ5KJWTJZWKZNAWXQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.125576805 (25h0m5.019285414s) [noise=true , address=45.91.116.5:4300 , piece_id=M2F4W7IGAEKEXJXZRENON55A2E7DNSCZTPKWYJFY3NKNTBIWG2YA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.125952470 (25h0m5.019661079s) [noise=true , address=145.79.227.10:4401 , piece_id=C2X52SPGSNM2CB6TFGBCT6HO3MXSEMX3CCZQASDNFGWIEW2EGIGQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.126492900 (25h0m5.020201509s) [noise=true , address=145.79.227.34:4906 , piece_id=G2GHF4S5NVJP5R43EN6NDEJ4FCPCLSL5K3TO4UADJ6ZANME54KOQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:10.730578935 (25h0m7.624287544s) [upload=21929]: Next returned nil segment Jan 16 16:56:10.730594134 (25h0m7.624302753s) [upload=21929]: Waiting for error group managing segments... Jan 16 16:56:11.329683279 (25h0m8.223391888s) [noise=true , address=145.79.227.15:4873 , piece_id=EVJGQ5JNJI37QIPQDJPFOPMT3JAHPBZXQG7DBRUZXJMKXAIADWSA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.338893335 (25h0m8.232601944s) [noise=true , address=45.91.116.14:4190 , piece_id=ZBC7C5E5V7B4VAGI2JHD7T4E6BGVV74YNNAROZYBITS2N23BJDWA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.400223651 (25h0m8.29393226s) [noise=true , address=145.79.227.33:4870 , piece_id=7I3KU7EKC2MHBJ6TOGO5ZBMJSPYU4E26D5373M2R53IZAZJLWVOA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.400340974 (25h0m8.294049583s) [noise=true , address=145.79.227.2:4115 , piece_id=YMME5WUV7XGQZOE2I5H7UWQ7TSTP2LGV7D4AZO4T62XOWBE4ZEKA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.445450294 (25h0m8.339158904s) [noise=true , address=145.79.227.30:4762 , piece_id=YLBYKHJNJ2J3YYVYHRWEZTFLXZVL5WCKCBWXM526XF4KKCYA4RAQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.964879212 (25h0m8.858587821s) [noise=true , address=145.79.227.3:4151 , piece_id=QVFK6RJITYJIW6ZJRUYFU3P2NEMMVANA36O6WXPS7Z7FNFSYQWUA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.027795357 (25h0m8.921503966s) [noise=true , address=145.79.227.49:4339 , piece_id=NWDULUWHMV4PIMWMNO5XYXSAGVLNPLSIPYSDA32OZB2MBGBYPEUA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.267543450 (25h0m9.161252059s) [noise=true , address=145.79.227.28:4727 , piece_id=U5CDDUXQZ2BB7SPFAIWQXRGALJQTOWFNWGJLU3CGXKBUHYIQBIGQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.337643945 (25h0m9.231352554s) [noise=true , address=145.79.227.10:4401 , piece_id=C2X52SPGSNM2CB6TFGBCT6HO3MXSEMX3CCZQASDNFGWIEW2EGIGQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.380001967 (25h0m9.273710576s) [noise=true , address=145.79.227.32:4835 , piece_id=6PP7J34OCTKYEDEFQJWDUHO5H4Z6VG7LNG3SL7WLITSKMX43HAYQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.850164228 (25h0m9.743872837s) [noise=true , address=77.90.31.13:4627 , piece_id=TJFFMEVYN7M52FH5UP3WNAGC7HWF5GSEUPQJZ5KJWTJZWKZNAWXQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:13.183873887 (25h0m10.077582496s) [noise=true , address=77.90.31.19:4842 , piece_id=3VJ3J4JQPSJEVFSNSYDVHRFZMMCBMNZRBYEGJOOU4Q4HFEDDOKHQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:13.361354387 (25h0m10.255063026s) [noise=true , address=45.91.116.4:4937 , piece_id=MJHRB2I4KWRJJ3MHGW657D53JFDARTHOBRUEHQZEUA4DRFVK7ZUA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:13.361383082 (25h0m10.255091691s) [seg_pos={0 0}, upload=21929]: Satisfied cohort requirements after 13 pieces Jan 16 16:56:13.361615564 (25h0m10.255324173s) [noise=true , address=145.79.227.34:4906 , piece_id=G2GHF4S5NVJP5R43EN6NDEJ4FCPCLSL5K3TO4UADJ6ZANME54KOQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Jan 16 16:56:13.361624061 (25h0m10.25533267s) [noise=true , address=145.79.227.34:4906 , piece_id=G2GHF4S5NVJP5R43EN6NDEJ4FCPCLSL5K3TO4UADJ6ZANME54KOQ, seg_pos={0 0}, upload=21929]: Long tail cancellation due to optimal threshold Jan 16 16:56:13.361904004 (25h0m10.255612623s) [noise=true , address=145.79.227.38:4977 , piece_id=JXV6MGOVSAA2URSIF2KFFQJJEGWQWFVT65PWVDW5MEH5ITSDYFAA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8): protocol: read tcp 188.166.225.170:42244->145.79.227.38:4977: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 16 16:56:13.361925344 (25h0m10.255633963s) [noise=true , address=145.79.227.38:4977 , piece_id=JXV6MGOVSAA2URSIF2KFFQJJEGWQWFVT65PWVDW5MEH5ITSDYFAA, seg_pos={0 0}, upload=21929]: Long tail cancellation due to optimal threshold Jan 16 16:56:13.361971022 (25h0m10.255679631s) [noise=true , address=45.91.116.5:4300 , piece_id=M2F4W7IGAEKEXJXZRENON55A2E7DNSCZTPKWYJFY3NKNTBIWG2YA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Jan 16 16:56:13.361978977 (25h0m10.255687596s) [noise=true , address=45.91.116.5:4300 , piece_id=M2F4W7IGAEKEXJXZRENON55A2E7DNSCZTPKWYJFY3NKNTBIWG2YA, seg_pos={0 0}, upload=21929]: Long tail cancellation due to optimal threshold Jan 16 16:56:13.362041817 (25h0m10.255750426s) [seg_pos={0 0}, upload=21929]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 16 16:56:13.362197272 (25h0m10.255905881s) [upload=21929]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 16 16:56:13.362205868 (25h0m10.255914477s) [upload=21929]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 16 16:56:13.965177888 (25h0m10.858886497s) [upload=21929]: Done uploading segments... Jan 16 16:56:13.965204158 (25h0m10.858912767s) [upload=21929]: Upload finished. err: <nil> Duration: 5.775547834s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 53 out of 60 is failed (threshold=6) [X.XXXX.XXXXXXXXXXXXXXXXXXXX.XXXXXX.X..XXXXXXX.XXXXXXXXXXXXXX] duration: 5.000729658s trace: https://jaeger.dev.storj.io/trace/31caf78ed04f4f0e FAIL (2026-01-16 17:06:01 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000753835s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [...............X.......................................XX...] duration: 7.30536186s trace: https://jaeger.dev.storj.io/trace/70c2b0731de5fb9c FAIL (2026-01-16 17:05:44 UTC) orbiter warning: context deadline exceeded Duration: 7.305381566s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................................................XXX....] duration: 8.883795715s trace: https://jaeger.dev.storj.io/trace/521596a1b50467d9 FAIL (2026-01-16 17:05:38 UTC) orbiter warning: context deadline exceeded Duration: 8.883811976s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 53 out of 60 is failed (threshold=6) [XX.XXXX.XXXXXXXXXXXXXXXXXXXX.XXXXXX.X..XXXXXXX.XXXXXXXXXXXXX] duration: 5.000629238s trace: https://jaeger.dev.storj.io/trace/76af2ef8cdd41ec0 FAIL (2026-01-16 17:05:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000689412s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......................................................XXX...] duration: 14.417710876s trace: https://jaeger.dev.storj.io/trace/521596a1b50467d9 FAIL (2026-01-16 17:05:08 UTC) orbiter warning: context deadline exceeded Duration: 14.417737285s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [................X.......................................XX..] duration: 7.749578808s trace: https://jaeger.dev.storj.io/trace/70c2b0731de5fb9c FAIL (2026-01-16 17:05:05 UTC) orbiter warning: context deadline exceeded Duration: 7.749604809s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 53 out of 60 is failed (threshold=6) [XXX.XXXX.XXXXXXXXXXXXXXXXXXXX.XXXXXX.X..XXXXXXX.XXXXXXXXXXXX] duration: 5.000848505s trace: https://jaeger.dev.storj.io/trace/4fc93e0998efb113 FAIL (2026-01-16 17:05:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000883931s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................................XX.............] duration: 5.696014575s trace: https://jaeger.dev.storj.io/trace/ff1eaafc0d95989 FAIL (2026-01-16 17:05:01 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 16 16:56:06.586316499 (25h0m3.480025108s) [upload=21929]: Starting upload Jan 16 16:56:06.586371244 (25h0m3.480079853s) [upload=21929]: Done starting upload Jan 16 16:56:06.586529134 (25h0m3.480237763s) [upload=21929]: Uploading segments... Jan 16 16:56:06.586687636 (25h0m3.480396275s) [upload=21929]: Got next segment. Inline: false Jan 16 16:56:06.586750094 (25h0m3.480458723s) [upload=21929]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 16 16:56:06.586757138 (25h0m3.480465767s) [upload=21929]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 16 16:56:08.120364931 (25h0m5.01407354s) [seg_pos={0 0}, upload=21929]: Begin upload segment... Jan 16 16:56:08.120413474 (25h0m5.014122083s) [seg_pos={0 0}, upload=21929]: Done begin upload segment. Jan 16 16:56:08.120440615 (25h0m5.014149224s) [noise=true , address=145.79.227.2:4115 , piece_id=YMME5WUV7XGQZOE2I5H7UWQ7TSTP2LGV7D4AZO4T62XOWBE4ZEKA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.120799800 (25h0m5.014508419s) [noise=true , address=145.79.227.32:4835 , piece_id=6PP7J34OCTKYEDEFQJWDUHO5H4Z6VG7LNG3SL7WLITSKMX43HAYQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.121695454 (25h0m5.015404084s) [noise=true , address=45.91.116.4:4937 , piece_id=MJHRB2I4KWRJJ3MHGW657D53JFDARTHOBRUEHQZEUA4DRFVK7ZUA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.122317739 (25h0m5.016026349s) [noise=true , address=145.79.227.30:4762 , piece_id=YLBYKHJNJ2J3YYVYHRWEZTFLXZVL5WCKCBWXM526XF4KKCYA4RAQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.123161636 (25h0m5.016870245s) [noise=true , address=45.91.116.14:4190 , piece_id=ZBC7C5E5V7B4VAGI2JHD7T4E6BGVV74YNNAROZYBITS2N23BJDWA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.123395732 (25h0m5.017104351s) [noise=true , address=145.79.227.28:4727 , piece_id=U5CDDUXQZ2BB7SPFAIWQXRGALJQTOWFNWGJLU3CGXKBUHYIQBIGQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.123848494 (25h0m5.017557113s) [noise=true , address=77.90.31.19:4842 , piece_id=3VJ3J4JQPSJEVFSNSYDVHRFZMMCBMNZRBYEGJOOU4Q4HFEDDOKHQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.123868092 (25h0m5.017576731s) [noise=true , address=145.79.227.15:4873 , piece_id=EVJGQ5JNJI37QIPQDJPFOPMT3JAHPBZXQG7DBRUZXJMKXAIADWSA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.124249808 (25h0m5.017958417s) [noise=true , address=145.79.227.38:4977 , piece_id=JXV6MGOVSAA2URSIF2KFFQJJEGWQWFVT65PWVDW5MEH5ITSDYFAA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.124389284 (25h0m5.018097893s) [noise=true , address=145.79.227.49:4339 , piece_id=NWDULUWHMV4PIMWMNO5XYXSAGVLNPLSIPYSDA32OZB2MBGBYPEUA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.124619222 (25h0m5.018327831s) [noise=true , address=145.79.227.33:4870 , piece_id=7I3KU7EKC2MHBJ6TOGO5ZBMJSPYU4E26D5373M2R53IZAZJLWVOA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.124999566 (25h0m5.018708175s) [noise=true , address=145.79.227.3:4151 , piece_id=QVFK6RJITYJIW6ZJRUYFU3P2NEMMVANA36O6WXPS7Z7FNFSYQWUA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.125202182 (25h0m5.018910801s) [noise=true , address=77.90.31.13:4627 , piece_id=TJFFMEVYN7M52FH5UP3WNAGC7HWF5GSEUPQJZ5KJWTJZWKZNAWXQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.125576805 (25h0m5.019285414s) [noise=true , address=45.91.116.5:4300 , piece_id=M2F4W7IGAEKEXJXZRENON55A2E7DNSCZTPKWYJFY3NKNTBIWG2YA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.125952470 (25h0m5.019661079s) [noise=true , address=145.79.227.10:4401 , piece_id=C2X52SPGSNM2CB6TFGBCT6HO3MXSEMX3CCZQASDNFGWIEW2EGIGQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.126492900 (25h0m5.020201509s) [noise=true , address=145.79.227.34:4906 , piece_id=G2GHF4S5NVJP5R43EN6NDEJ4FCPCLSL5K3TO4UADJ6ZANME54KOQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:10.730578935 (25h0m7.624287544s) [upload=21929]: Next returned nil segment Jan 16 16:56:10.730594134 (25h0m7.624302753s) [upload=21929]: Waiting for error group managing segments... Jan 16 16:56:11.329683279 (25h0m8.223391888s) [noise=true , address=145.79.227.15:4873 , piece_id=EVJGQ5JNJI37QIPQDJPFOPMT3JAHPBZXQG7DBRUZXJMKXAIADWSA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.338893335 (25h0m8.232601944s) [noise=true , address=45.91.116.14:4190 , piece_id=ZBC7C5E5V7B4VAGI2JHD7T4E6BGVV74YNNAROZYBITS2N23BJDWA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.400223651 (25h0m8.29393226s) [noise=true , address=145.79.227.33:4870 , piece_id=7I3KU7EKC2MHBJ6TOGO5ZBMJSPYU4E26D5373M2R53IZAZJLWVOA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.400340974 (25h0m8.294049583s) [noise=true , address=145.79.227.2:4115 , piece_id=YMME5WUV7XGQZOE2I5H7UWQ7TSTP2LGV7D4AZO4T62XOWBE4ZEKA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.445450294 (25h0m8.339158904s) [noise=true , address=145.79.227.30:4762 , piece_id=YLBYKHJNJ2J3YYVYHRWEZTFLXZVL5WCKCBWXM526XF4KKCYA4RAQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.964879212 (25h0m8.858587821s) [noise=true , address=145.79.227.3:4151 , piece_id=QVFK6RJITYJIW6ZJRUYFU3P2NEMMVANA36O6WXPS7Z7FNFSYQWUA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.027795357 (25h0m8.921503966s) [noise=true , address=145.79.227.49:4339 , piece_id=NWDULUWHMV4PIMWMNO5XYXSAGVLNPLSIPYSDA32OZB2MBGBYPEUA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.267543450 (25h0m9.161252059s) [noise=true , address=145.79.227.28:4727 , piece_id=U5CDDUXQZ2BB7SPFAIWQXRGALJQTOWFNWGJLU3CGXKBUHYIQBIGQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.337643945 (25h0m9.231352554s) [noise=true , address=145.79.227.10:4401 , piece_id=C2X52SPGSNM2CB6TFGBCT6HO3MXSEMX3CCZQASDNFGWIEW2EGIGQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.380001967 (25h0m9.273710576s) [noise=true , address=145.79.227.32:4835 , piece_id=6PP7J34OCTKYEDEFQJWDUHO5H4Z6VG7LNG3SL7WLITSKMX43HAYQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.850164228 (25h0m9.743872837s) [noise=true , address=77.90.31.13:4627 , piece_id=TJFFMEVYN7M52FH5UP3WNAGC7HWF5GSEUPQJZ5KJWTJZWKZNAWXQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:13.183873887 (25h0m10.077582496s) [noise=true , address=77.90.31.19:4842 , piece_id=3VJ3J4JQPSJEVFSNSYDVHRFZMMCBMNZRBYEGJOOU4Q4HFEDDOKHQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:13.361354387 (25h0m10.255063026s) [noise=true , address=45.91.116.4:4937 , piece_id=MJHRB2I4KWRJJ3MHGW657D53JFDARTHOBRUEHQZEUA4DRFVK7ZUA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:13.361383082 (25h0m10.255091691s) [seg_pos={0 0}, upload=21929]: Satisfied cohort requirements after 13 pieces Jan 16 16:56:13.361615564 (25h0m10.255324173s) [noise=true , address=145.79.227.34:4906 , piece_id=G2GHF4S5NVJP5R43EN6NDEJ4FCPCLSL5K3TO4UADJ6ZANME54KOQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Jan 16 16:56:13.361624061 (25h0m10.25533267s) [noise=true , address=145.79.227.34:4906 , piece_id=G2GHF4S5NVJP5R43EN6NDEJ4FCPCLSL5K3TO4UADJ6ZANME54KOQ, seg_pos={0 0}, upload=21929]: Long tail cancellation due to optimal threshold Jan 16 16:56:13.361904004 (25h0m10.255612623s) [noise=true , address=145.79.227.38:4977 , piece_id=JXV6MGOVSAA2URSIF2KFFQJJEGWQWFVT65PWVDW5MEH5ITSDYFAA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8): protocol: read tcp 188.166.225.170:42244->145.79.227.38:4977: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 16 16:56:13.361925344 (25h0m10.255633963s) [noise=true , address=145.79.227.38:4977 , piece_id=JXV6MGOVSAA2URSIF2KFFQJJEGWQWFVT65PWVDW5MEH5ITSDYFAA, seg_pos={0 0}, upload=21929]: Long tail cancellation due to optimal threshold Jan 16 16:56:13.361971022 (25h0m10.255679631s) [noise=true , address=45.91.116.5:4300 , piece_id=M2F4W7IGAEKEXJXZRENON55A2E7DNSCZTPKWYJFY3NKNTBIWG2YA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Jan 16 16:56:13.361978977 (25h0m10.255687596s) [noise=true , address=45.91.116.5:4300 , piece_id=M2F4W7IGAEKEXJXZRENON55A2E7DNSCZTPKWYJFY3NKNTBIWG2YA, seg_pos={0 0}, upload=21929]: Long tail cancellation due to optimal threshold Jan 16 16:56:13.362041817 (25h0m10.255750426s) [seg_pos={0 0}, upload=21929]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 16 16:56:13.362197272 (25h0m10.255905881s) [upload=21929]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 16 16:56:13.362205868 (25h0m10.255914477s) [upload=21929]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 16 16:56:13.965177888 (25h0m10.858886497s) [upload=21929]: Done uploading segments... Jan 16 16:56:13.965204158 (25h0m10.858912767s) [upload=21929]: Upload finished. err: <nil> Duration: 5.696042097s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.................X.......................................XX.] duration: 12.591798011s trace: https://jaeger.dev.storj.io/trace/70c2b0731de5fb9c FAIL (2026-01-16 17:04:35 UTC) orbiter warning: context deadline exceeded Duration: 12.591819543s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......................................................XXX..] duration: 26.718691096s trace: https://jaeger.dev.storj.io/trace/521596a1b50467d9 FAIL (2026-01-16 17:04:21 UTC) orbiter warning: context deadline exceeded Duration: 26.718707509s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 53 out of 60 is failed (threshold=6) [XXXX.XXXX.XXXXXXXXXXXXXXXXXXXX.XXXXXX.X..XXXXXXX.XXXXXXXXXXX] duration: 5.001214152s trace: https://jaeger.dev.storj.io/trace/5fa5ec7ddcc17260 FAIL (2026-01-16 17:04:21 UTC) orbiter warning: context deadline exceeded Duration: 5.001236506s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..................X.......................................XX] duration: 1m0.001402443s trace: https://jaeger.dev.storj.io/trace/70c2b0731de5fb9c FAIL (2026-01-16 17:04:20 UTC) orbiter warning: context deadline exceeded Duration: 1m0.001412291s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 53 out of 60 is failed (threshold=6) [XXXXX.XXXX.XXXXXXXXXXXXXXXXXXXX.XXXXXX.X..XXXXXXX.XXXXXXXXXX] duration: 5.001349742s trace: https://jaeger.dev.storj.io/trace/38ebd0dcc4c61521 FAIL (2026-01-16 17:04:01 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001377935s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........................................................XXX.] duration: 21.357625397s trace: https://jaeger.dev.storj.io/trace/521596a1b50467d9 FAIL (2026-01-16 17:03:52 UTC) orbiter warning: context deadline exceeded Duration: 21.357654502s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................................XX............] duration: 5.726903931s trace: https://jaeger.dev.storj.io/trace/ff1eaafc0d95989 FAIL (2026-01-16 17:03:48 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 16 16:56:06.586316499 (25h0m3.480025108s) [upload=21929]: Starting upload Jan 16 16:56:06.586371244 (25h0m3.480079853s) [upload=21929]: Done starting upload Jan 16 16:56:06.586529134 (25h0m3.480237763s) [upload=21929]: Uploading segments... Jan 16 16:56:06.586687636 (25h0m3.480396275s) [upload=21929]: Got next segment. Inline: false Jan 16 16:56:06.586750094 (25h0m3.480458723s) [upload=21929]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 16 16:56:06.586757138 (25h0m3.480465767s) [upload=21929]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 16 16:56:08.120364931 (25h0m5.01407354s) [seg_pos={0 0}, upload=21929]: Begin upload segment... Jan 16 16:56:08.120413474 (25h0m5.014122083s) [seg_pos={0 0}, upload=21929]: Done begin upload segment. Jan 16 16:56:08.120440615 (25h0m5.014149224s) [noise=true , address=145.79.227.2:4115 , piece_id=YMME5WUV7XGQZOE2I5H7UWQ7TSTP2LGV7D4AZO4T62XOWBE4ZEKA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.120799800 (25h0m5.014508419s) [noise=true , address=145.79.227.32:4835 , piece_id=6PP7J34OCTKYEDEFQJWDUHO5H4Z6VG7LNG3SL7WLITSKMX43HAYQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.121695454 (25h0m5.015404084s) [noise=true , address=45.91.116.4:4937 , piece_id=MJHRB2I4KWRJJ3MHGW657D53JFDARTHOBRUEHQZEUA4DRFVK7ZUA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.122317739 (25h0m5.016026349s) [noise=true , address=145.79.227.30:4762 , piece_id=YLBYKHJNJ2J3YYVYHRWEZTFLXZVL5WCKCBWXM526XF4KKCYA4RAQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.123161636 (25h0m5.016870245s) [noise=true , address=45.91.116.14:4190 , piece_id=ZBC7C5E5V7B4VAGI2JHD7T4E6BGVV74YNNAROZYBITS2N23BJDWA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.123395732 (25h0m5.017104351s) [noise=true , address=145.79.227.28:4727 , piece_id=U5CDDUXQZ2BB7SPFAIWQXRGALJQTOWFNWGJLU3CGXKBUHYIQBIGQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.123848494 (25h0m5.017557113s) [noise=true , address=77.90.31.19:4842 , piece_id=3VJ3J4JQPSJEVFSNSYDVHRFZMMCBMNZRBYEGJOOU4Q4HFEDDOKHQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.123868092 (25h0m5.017576731s) [noise=true , address=145.79.227.15:4873 , piece_id=EVJGQ5JNJI37QIPQDJPFOPMT3JAHPBZXQG7DBRUZXJMKXAIADWSA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.124249808 (25h0m5.017958417s) [noise=true , address=145.79.227.38:4977 , piece_id=JXV6MGOVSAA2URSIF2KFFQJJEGWQWFVT65PWVDW5MEH5ITSDYFAA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.124389284 (25h0m5.018097893s) [noise=true , address=145.79.227.49:4339 , piece_id=NWDULUWHMV4PIMWMNO5XYXSAGVLNPLSIPYSDA32OZB2MBGBYPEUA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.124619222 (25h0m5.018327831s) [noise=true , address=145.79.227.33:4870 , piece_id=7I3KU7EKC2MHBJ6TOGO5ZBMJSPYU4E26D5373M2R53IZAZJLWVOA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.124999566 (25h0m5.018708175s) [noise=true , address=145.79.227.3:4151 , piece_id=QVFK6RJITYJIW6ZJRUYFU3P2NEMMVANA36O6WXPS7Z7FNFSYQWUA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.125202182 (25h0m5.018910801s) [noise=true , address=77.90.31.13:4627 , piece_id=TJFFMEVYN7M52FH5UP3WNAGC7HWF5GSEUPQJZ5KJWTJZWKZNAWXQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.125576805 (25h0m5.019285414s) [noise=true , address=45.91.116.5:4300 , piece_id=M2F4W7IGAEKEXJXZRENON55A2E7DNSCZTPKWYJFY3NKNTBIWG2YA, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.125952470 (25h0m5.019661079s) [noise=true , address=145.79.227.10:4401 , piece_id=C2X52SPGSNM2CB6TFGBCT6HO3MXSEMX3CCZQASDNFGWIEW2EGIGQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:08.126492900 (25h0m5.020201509s) [noise=true , address=145.79.227.34:4906 , piece_id=G2GHF4S5NVJP5R43EN6NDEJ4FCPCLSL5K3TO4UADJ6ZANME54KOQ, seg_pos={0 0}, upload=21929]: Uploading piece... Jan 16 16:56:10.730578935 (25h0m7.624287544s) [upload=21929]: Next returned nil segment Jan 16 16:56:10.730594134 (25h0m7.624302753s) [upload=21929]: Waiting for error group managing segments... Jan 16 16:56:11.329683279 (25h0m8.223391888s) [noise=true , address=145.79.227.15:4873 , piece_id=EVJGQ5JNJI37QIPQDJPFOPMT3JAHPBZXQG7DBRUZXJMKXAIADWSA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.338893335 (25h0m8.232601944s) [noise=true , address=45.91.116.14:4190 , piece_id=ZBC7C5E5V7B4VAGI2JHD7T4E6BGVV74YNNAROZYBITS2N23BJDWA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.400223651 (25h0m8.29393226s) [noise=true , address=145.79.227.33:4870 , piece_id=7I3KU7EKC2MHBJ6TOGO5ZBMJSPYU4E26D5373M2R53IZAZJLWVOA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.400340974 (25h0m8.294049583s) [noise=true , address=145.79.227.2:4115 , piece_id=YMME5WUV7XGQZOE2I5H7UWQ7TSTP2LGV7D4AZO4T62XOWBE4ZEKA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.445450294 (25h0m8.339158904s) [noise=true , address=145.79.227.30:4762 , piece_id=YLBYKHJNJ2J3YYVYHRWEZTFLXZVL5WCKCBWXM526XF4KKCYA4RAQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:11.964879212 (25h0m8.858587821s) [noise=true , address=145.79.227.3:4151 , piece_id=QVFK6RJITYJIW6ZJRUYFU3P2NEMMVANA36O6WXPS7Z7FNFSYQWUA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.027795357 (25h0m8.921503966s) [noise=true , address=145.79.227.49:4339 , piece_id=NWDULUWHMV4PIMWMNO5XYXSAGVLNPLSIPYSDA32OZB2MBGBYPEUA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.267543450 (25h0m9.161252059s) [noise=true , address=145.79.227.28:4727 , piece_id=U5CDDUXQZ2BB7SPFAIWQXRGALJQTOWFNWGJLU3CGXKBUHYIQBIGQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.337643945 (25h0m9.231352554s) [noise=true , address=145.79.227.10:4401 , piece_id=C2X52SPGSNM2CB6TFGBCT6HO3MXSEMX3CCZQASDNFGWIEW2EGIGQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.380001967 (25h0m9.273710576s) [noise=true , address=145.79.227.32:4835 , piece_id=6PP7J34OCTKYEDEFQJWDUHO5H4Z6VG7LNG3SL7WLITSKMX43HAYQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:12.850164228 (25h0m9.743872837s) [noise=true , address=77.90.31.13:4627 , piece_id=TJFFMEVYN7M52FH5UP3WNAGC7HWF5GSEUPQJZ5KJWTJZWKZNAWXQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:13.183873887 (25h0m10.077582496s) [noise=true , address=77.90.31.19:4842 , piece_id=3VJ3J4JQPSJEVFSNSYDVHRFZMMCBMNZRBYEGJOOU4Q4HFEDDOKHQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:13.361354387 (25h0m10.255063026s) [noise=true , address=45.91.116.4:4937 , piece_id=MJHRB2I4KWRJJ3MHGW657D53JFDARTHOBRUEHQZEUA4DRFVK7ZUA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: <nil> Jan 16 16:56:13.361383082 (25h0m10.255091691s) [seg_pos={0 0}, upload=21929]: Satisfied cohort requirements after 13 pieces Jan 16 16:56:13.361615564 (25h0m10.255324173s) [noise=true , address=145.79.227.34:4906 , piece_id=G2GHF4S5NVJP5R43EN6NDEJ4FCPCLSL5K3TO4UADJ6ZANME54KOQ, seg_pos={0 0}, upload=21929]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Jan 16 16:56:13.361624061 (25h0m10.25533267s) [noise=true , address=145.79.227.34:4906 , piece_id=G2GHF4S5NVJP5R43EN6NDEJ4FCPCLSL5K3TO4UADJ6ZANME54KOQ, seg_pos={0 0}, upload=21929]: Long tail cancellation due to optimal threshold Jan 16 16:56:13.361904004 (25h0m10.255612623s) [noise=true , address=145.79.227.38:4977 , piece_id=JXV6MGOVSAA2URSIF2KFFQJJEGWQWFVT65PWVDW5MEH5ITSDYFAA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8): protocol: read tcp 188.166.225.170:42244->145.79.227.38:4977: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 16 16:56:13.361925344 (25h0m10.255633963s) [noise=true , address=145.79.227.38:4977 , piece_id=JXV6MGOVSAA2URSIF2KFFQJJEGWQWFVT65PWVDW5MEH5ITSDYFAA, seg_pos={0 0}, upload=21929]: Long tail cancellation due to optimal threshold Jan 16 16:56:13.361971022 (25h0m10.255679631s) [noise=true , address=45.91.116.5:4300 , piece_id=M2F4W7IGAEKEXJXZRENON55A2E7DNSCZTPKWYJFY3NKNTBIWG2YA, seg_pos={0 0}, upload=21929]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Jan 16 16:56:13.361978977 (25h0m10.255687596s) [noise=true , address=45.91.116.5:4300 , piece_id=M2F4W7IGAEKEXJXZRENON55A2E7DNSCZTPKWYJFY3NKNTBIWG2YA, seg_pos={0 0}, upload=21929]: Long tail cancellation due to optimal threshold Jan 16 16:56:13.362041817 (25h0m10.255750426s) [seg_pos={0 0}, upload=21929]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 16 16:56:13.362197272 (25h0m10.255905881s) [upload=21929]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 16 16:56:13.362205868 (25h0m10.255914477s) [upload=21929]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 16 16:56:13.965177888 (25h0m10.858886497s) [upload=21929]: Done uploading segments... Jan 16 16:56:13.965204158 (25h0m10.858912767s) [upload=21929]: Upload finished. err: <nil> Duration: 5.726931403s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 53 out of 60 is failed (threshold=6) [XXXXXX.XXXX.XXXXXXXXXXXXXXXXXXXX.XXXXXX.X..XXXXXXX.XXXXXXXXX] duration: 5.001104174s trace: https://jaeger.dev.storj.io/trace/6b5904aaaad99f20 FAIL (2026-01-16 17:03:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001126867s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)