Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 20 out of 60 is failed (threshold=6) [.X..XX..X......X.X.XX...X.X.X.X.....X....XXXX......XX...X...] duration: 3.640531258s trace: https://jaeger.dev.storj.io/trace/450d7df9d8d51f5c FAIL (2025-12-08 16:47:04 UTC) orbiter warning: context deadline exceeded Duration: 3.640558802s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...X.X......................................................] duration: 5.787283403s trace: https://jaeger.dev.storj.io/trace/690f79ef204fee2 FAIL (2025-12-08 16:46:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.787300255s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............XX...........................................] duration: 5.960753152s trace: https://jaeger.dev.storj.io/trace/13568c7eb15b2369 FAIL (2025-12-08 16:46:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, Piece ID: U3FCNUY4ASY3ZGX7ZY26GE2NA52GJNKJMW7XKZKALT2MWBCZMHFA) context deadline exceeded; piecestore: (Node ID: 1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, Piece ID: QN7JC2XQK6FKMIYD2WXQ7GMDILEWDUO2HXZLZERVJ24WWIMHFOVA) context deadline exceeded; piecestore: (Node ID: 12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, Piece ID: KRCKO7NXT2XOYK4QHVJK5UIU246TVB45EE7ELVUT365XTJ7RHKJQ) context deadline exceeded; piecestore: (Node ID: 122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, Piece ID: U7SKMZGV33HMSQLQIFIBZ6WNYC7JJ2CENSK5LCW3KVMP53T7467Q) context deadline exceeded; piecestore: (Node ID: 1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, Piece ID: FGYLRBVDMMLYWTTKLC3B2P2MHTECHEBH2DJOOKAHBR6TBAMKFBXA) context deadline exceeded ======LOGS====== Dec 8 16:15:19.690262294 (72h46m19.561345339s) [upload=63694]: Starting upload Dec 8 16:15:19.690357154 (72h46m19.56144022s) [upload=63694]: Done starting upload Dec 8 16:15:19.690425354 (72h46m19.561508409s) [upload=63694]: Uploading segments... Dec 8 16:15:19.690652296 (72h46m19.561735341s) [upload=63694]: Got next segment. Inline: false Dec 8 16:15:19.690693895 (72h46m19.56177694s) [upload=63694]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 16:15:19.690700889 (72h46m19.561783934s) [upload=63694]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 16:15:20.501121517 (72h46m20.372204572s) [seg_pos={0 0}, upload=63694]: Begin upload segment... Dec 8 16:15:20.501188935 (72h46m20.372271981s) [seg_pos={0 0}, upload=63694]: Done begin upload segment. Dec 8 16:15:20.501225655 (72h46m20.3723087s) [noise=true , address=145.79.227.16:4912 , piece_id=NIW3LT24E4LPJ5QYJVQOAR43S23JLDDOXZ6NVOURZNDTCCTLLPJA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.501311268 (72h46m20.372394313s) [noise=true , address=145.79.227.34:4906 , piece_id=AIZQB5ZLXPCLPT5Q7NLNSI3FPBPAMBVUIXKZCTA3ML7L3XP2UY6A, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.501987676 (72h46m20.373070722s) [noise=true , address=145.79.227.17:4971 , piece_id=KRCKO7NXT2XOYK4QHVJK5UIU246TVB45EE7ELVUT365XTJ7RHKJQ, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.502073379 (72h46m20.373156424s) [noise=true , address=145.79.227.38:4979 , piece_id=HCELMHS3GNLCKHFCQU3V45M4OETQZPL5MQJBUOUF5VPA44HXZW2Q, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.502559184 (72h46m20.373642229s) [noise=true , address=145.79.227.33:4870 , piece_id=IK7NDJD5GNFMWB26JYOMCSNIAT5TTTQOUXB2VFYSNRGIQ2MVMXAQ, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.502903350 (72h46m20.373986385s) [noise=true , address=145.79.227.22:4511 , piece_id=RT3ULMIM24LMC4UASFJYVAOHIKJBBGLFYJP3LBJCUL6I532MTSGA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503199825 (72h46m20.37428287s) [noise=true , address=145.79.227.29:4941 , piece_id=GJQXPA7WEZUTVJNELRBJVTU4LIGXYD7XLEPR5SWF4NBWWMXBW2SA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503205956 (72h46m20.374289001s) [noise=true , address=77.90.31.11:4555 , piece_id=LZ4YCICB4QQVLQAUZD6ZFABXLYSRKC4OFZESDVUGIOFJMZBRMMSA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503484487 (72h46m20.374567532s) [noise=true , address=145.79.227.39:4014 , piece_id=U7SKMZGV33HMSQLQIFIBZ6WNYC7JJ2CENSK5LCW3KVMP53T7467Q, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503757176 (72h46m20.374840221s) [noise=true , address=145.79.227.24:4582 , piece_id=FGYLRBVDMMLYWTTKLC3B2P2MHTECHEBH2DJOOKAHBR6TBAMKFBXA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503773727 (72h46m20.374856772s) [noise=true , address=145.79.227.40:4049 , piece_id=VX3PNKMHR75TNJLKRBLYS3KT42ZTPT4X6I5QS2AKNMGDQDVE4WDA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.504114126 (72h46m20.375197161s) [noise=true , address=77.90.31.17:4771 , piece_id=R6PLE6VLZNL23TJC6V6MRBLJP6PKB2DDGQMPPGLMM23WN6J7SH5Q, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.504441298 (72h46m20.375524343s) [noise=true , address=77.90.31.18:4807 , piece_id=7FXKUHBJT7AHDC4PHCZ73Q62GDSOUFRFIVAD337ORGE7YUTAULVA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.505240540 (72h46m20.376323585s) [noise=true , address=77.90.31.14:4662 , piece_id=BO6DQNLFNMG6HAPKOAG2DHWDD3YMX46IAYGDVQG7THC5EVZORJRA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.506022409 (72h46m20.377105454s) [noise=true , address=145.79.227.20:4438 , piece_id=U3FCNUY4ASY3ZGX7ZY26GE2NA52GJNKJMW7XKZKALT2MWBCZMHFA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.506709457 (72h46m20.377792502s) [noise=true , address=145.79.227.43:4158 , piece_id=QN7JC2XQK6FKMIYD2WXQ7GMDILEWDUO2HXZLZERVJ24WWIMHFOVA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:22.339915069 (72h46m22.210998114s) [upload=63694]: Next returned nil segment Dec 8 16:15:22.339932321 (72h46m22.211015366s) [upload=63694]: Waiting for error group managing segments... Dec 8 16:15:22.655133540 (72h46m22.526216575s) [noise=true , address=145.79.227.43:4158 , piece_id=QN7JC2XQK6FKMIYD2WXQ7GMDILEWDUO2HXZLZERVJ24WWIMHFOVA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.660462928 (72h46m22.531545983s) [noise=true , address=145.79.227.17:4971 , piece_id=KRCKO7NXT2XOYK4QHVJK5UIU246TVB45EE7ELVUT365XTJ7RHKJQ, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.683969820 (72h46m22.555052855s) [noise=true , address=77.90.31.11:4555 , piece_id=LZ4YCICB4QQVLQAUZD6ZFABXLYSRKC4OFZESDVUGIOFJMZBRMMSA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.685188641 (72h46m22.556271686s) [noise=true , address=145.79.227.22:4511 , piece_id=RT3ULMIM24LMC4UASFJYVAOHIKJBBGLFYJP3LBJCUL6I532MTSGA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.686713104 (72h46m22.557796149s) [noise=true , address=145.79.227.16:4912 , piece_id=NIW3LT24E4LPJ5QYJVQOAR43S23JLDDOXZ6NVOURZNDTCCTLLPJA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.698627942 (72h46m22.569710987s) [noise=true , address=145.79.227.39:4014 , piece_id=U7SKMZGV33HMSQLQIFIBZ6WNYC7JJ2CENSK5LCW3KVMP53T7467Q, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.713359143 (72h46m22.584442188s) [noise=true , address=145.79.227.40:4049 , piece_id=VX3PNKMHR75TNJLKRBLYS3KT42ZTPT4X6I5QS2AKNMGDQDVE4WDA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.731975212 (72h46m22.603058257s) [noise=true , address=77.90.31.17:4771 , piece_id=R6PLE6VLZNL23TJC6V6MRBLJP6PKB2DDGQMPPGLMM23WN6J7SH5Q, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.741125236 (72h46m22.612208281s) [noise=true , address=145.79.227.20:4438 , piece_id=U3FCNUY4ASY3ZGX7ZY26GE2NA52GJNKJMW7XKZKALT2MWBCZMHFA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.773771311 (72h46m22.644854346s) [noise=true , address=77.90.31.18:4807 , piece_id=7FXKUHBJT7AHDC4PHCZ73Q62GDSOUFRFIVAD337ORGE7YUTAULVA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.906364795 (72h46m22.777447841s) [noise=true , address=77.90.31.14:4662 , piece_id=BO6DQNLFNMG6HAPKOAG2DHWDD3YMX46IAYGDVQG7THC5EVZORJRA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:23.367795705 (72h46m23.23887874s) [noise=true , address=145.79.227.24:4582 , piece_id=FGYLRBVDMMLYWTTKLC3B2P2MHTECHEBH2DJOOKAHBR6TBAMKFBXA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:24.582400944 (72h46m24.453483979s) [noise=true , address=145.79.227.33:4870 , piece_id=IK7NDJD5GNFMWB26JYOMCSNIAT5TTTQOUXB2VFYSNRGIQ2MVMXAQ, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:24.582421023 (72h46m24.453504068s) [seg_pos={0 0}, upload=63694]: Segment reached optimal threshold of 13 pieces. Dec 8 16:15:24.582652363 (72h46m24.453735408s) [noise=true , address=145.79.227.38:4979 , piece_id=HCELMHS3GNLCKHFCQU3V45M4OETQZPL5MQJBUOUF5VPA44HXZW2Q, seg_pos={0 0}, upload=63694]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma): protocol: expected piece hash; context canceled; EOF Dec 8 16:15:24.582659086 (72h46m24.453742121s) [noise=true , address=145.79.227.38:4979 , piece_id=HCELMHS3GNLCKHFCQU3V45M4OETQZPL5MQJBUOUF5VPA44HXZW2Q, seg_pos={0 0}, upload=63694]: Long tail cancellation due to optimal threshold Dec 8 16:15:24.582740842 (72h46m24.453823877s) [noise=true , address=145.79.227.34:4906 , piece_id=AIZQB5ZLXPCLPT5Q7NLNSI3FPBPAMBVUIXKZCTA3ML7L3XP2UY6A, seg_pos={0 0}, upload=63694]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 8 16:15:24.582753055 (72h46m24.45383609s) [noise=true , address=145.79.227.34:4906 , piece_id=AIZQB5ZLXPCLPT5Q7NLNSI3FPBPAMBVUIXKZCTA3ML7L3XP2UY6A, seg_pos={0 0}, upload=63694]: Long tail cancellation due to optimal threshold Dec 8 16:15:24.583044900 (72h46m24.454127935s) [noise=true , address=145.79.227.29:4941 , piece_id=GJQXPA7WEZUTVJNELRBJVTU4LIGXYD7XLEPR5SWF4NBWWMXBW2SA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: context canceled; context canceled Dec 8 16:15:24.583052926 (72h46m24.454135961s) [noise=true , address=145.79.227.29:4941 , piece_id=GJQXPA7WEZUTVJNELRBJVTU4LIGXYD7XLEPR5SWF4NBWWMXBW2SA, seg_pos={0 0}, upload=63694]: Long tail cancellation due to optimal threshold Dec 8 16:15:24.583071441 (72h46m24.454154486s) [seg_pos={0 0}, upload=63694]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 16:15:24.583162014 (72h46m24.454245059s) [upload=63694]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 16:15:24.583171562 (72h46m24.454254627s) [upload=63694]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 16:15:25.042230695 (72h46m24.91331374s) [upload=63694]: Done uploading segments... Dec 8 16:15:25.042258869 (72h46m24.913341914s) [upload=63694]: Upload finished. err: <nil> Duration: 5.960779562s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 5.008936368s trace: https://jaeger.dev.storj.io/trace/5d817b557064ae10 FAIL (2025-12-08 16:46:40 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.00896304s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 20 out of 60 is failed (threshold=6) [..X..XX..X......X.X.XX...X.X.X.X.....X....XXXX......XX...X..] duration: 3.421163309s trace: https://jaeger.dev.storj.io/trace/450d7df9d8d51f5c FAIL (2025-12-08 16:46:40 UTC) orbiter warning: context deadline exceeded Duration: 3.421191181s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 2.117023823s trace: https://jaeger.dev.storj.io/trace/4dad09bb52e33232 FAIL (2025-12-08 16:46:38 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 2.117060642s core uplink us1
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 468.119544ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/6c02152d3764cef5 FAIL (2025-12-08 16:46:38 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 468.1487ms core satellite eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 2.010791185s trace: https://jaeger.dev.storj.io/trace/4dad09bb52e33232 FAIL (2025-12-08 16:46:23 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 2.01082065s core uplink us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 5.194933994s trace: https://jaeger.dev.storj.io/trace/5d817b557064ae10 FAIL (2025-12-08 16:46:14 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.194972869s core uplink eu1
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 468.153539ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/6c02152d3764cef5 FAIL (2025-12-08 16:46:09 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 468.194989ms core satellite eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 5.765747251s trace: https://jaeger.dev.storj.io/trace/1e9e467d23cfcdb3 FAIL (2025-12-08 16:46:06 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.765780024s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 2.010261056s trace: https://jaeger.dev.storj.io/trace/4dad09bb52e33232 FAIL (2025-12-08 16:46:01 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 2.010295621s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................X...X.X...................................] duration: 6.75135076s trace: https://jaeger.dev.storj.io/trace/20c661c94f160c60 FAIL (2025-12-08 16:45:54 UTC) orbiter warning: context deadline exceeded Duration: 6.751384614s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [....X.X.....................................................] duration: 9.228732875s trace: https://jaeger.dev.storj.io/trace/690f79ef204fee2 FAIL (2025-12-08 16:45:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 9.228791357s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 20 out of 60 is failed (threshold=6) [...X..XX..X......X.X.XX...X.X.X.X.....X....XXXX......XX...X.] duration: 3.546430761s trace: https://jaeger.dev.storj.io/trace/450d7df9d8d51f5c FAIL (2025-12-08 16:45:45 UTC) orbiter warning: context deadline exceeded Duration: 3.546454946s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................XX..........................................] duration: 5.771303561s trace: https://jaeger.dev.storj.io/trace/13568c7eb15b2369 FAIL (2025-12-08 16:45:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, Piece ID: U3FCNUY4ASY3ZGX7ZY26GE2NA52GJNKJMW7XKZKALT2MWBCZMHFA) context deadline exceeded; piecestore: (Node ID: 1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, Piece ID: QN7JC2XQK6FKMIYD2WXQ7GMDILEWDUO2HXZLZERVJ24WWIMHFOVA) context deadline exceeded; piecestore: (Node ID: 12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, Piece ID: KRCKO7NXT2XOYK4QHVJK5UIU246TVB45EE7ELVUT365XTJ7RHKJQ) context deadline exceeded; piecestore: (Node ID: 122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, Piece ID: U7SKMZGV33HMSQLQIFIBZ6WNYC7JJ2CENSK5LCW3KVMP53T7467Q) context deadline exceeded; piecestore: (Node ID: 1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, Piece ID: FGYLRBVDMMLYWTTKLC3B2P2MHTECHEBH2DJOOKAHBR6TBAMKFBXA) context deadline exceeded ======LOGS====== Dec 8 16:15:19.690262294 (72h46m19.561345339s) [upload=63694]: Starting upload Dec 8 16:15:19.690357154 (72h46m19.56144022s) [upload=63694]: Done starting upload Dec 8 16:15:19.690425354 (72h46m19.561508409s) [upload=63694]: Uploading segments... Dec 8 16:15:19.690652296 (72h46m19.561735341s) [upload=63694]: Got next segment. Inline: false Dec 8 16:15:19.690693895 (72h46m19.56177694s) [upload=63694]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 16:15:19.690700889 (72h46m19.561783934s) [upload=63694]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 16:15:20.501121517 (72h46m20.372204572s) [seg_pos={0 0}, upload=63694]: Begin upload segment... Dec 8 16:15:20.501188935 (72h46m20.372271981s) [seg_pos={0 0}, upload=63694]: Done begin upload segment. Dec 8 16:15:20.501225655 (72h46m20.3723087s) [noise=true , address=145.79.227.16:4912 , piece_id=NIW3LT24E4LPJ5QYJVQOAR43S23JLDDOXZ6NVOURZNDTCCTLLPJA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.501311268 (72h46m20.372394313s) [noise=true , address=145.79.227.34:4906 , piece_id=AIZQB5ZLXPCLPT5Q7NLNSI3FPBPAMBVUIXKZCTA3ML7L3XP2UY6A, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.501987676 (72h46m20.373070722s) [noise=true , address=145.79.227.17:4971 , piece_id=KRCKO7NXT2XOYK4QHVJK5UIU246TVB45EE7ELVUT365XTJ7RHKJQ, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.502073379 (72h46m20.373156424s) [noise=true , address=145.79.227.38:4979 , piece_id=HCELMHS3GNLCKHFCQU3V45M4OETQZPL5MQJBUOUF5VPA44HXZW2Q, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.502559184 (72h46m20.373642229s) [noise=true , address=145.79.227.33:4870 , piece_id=IK7NDJD5GNFMWB26JYOMCSNIAT5TTTQOUXB2VFYSNRGIQ2MVMXAQ, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.502903350 (72h46m20.373986385s) [noise=true , address=145.79.227.22:4511 , piece_id=RT3ULMIM24LMC4UASFJYVAOHIKJBBGLFYJP3LBJCUL6I532MTSGA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503199825 (72h46m20.37428287s) [noise=true , address=145.79.227.29:4941 , piece_id=GJQXPA7WEZUTVJNELRBJVTU4LIGXYD7XLEPR5SWF4NBWWMXBW2SA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503205956 (72h46m20.374289001s) [noise=true , address=77.90.31.11:4555 , piece_id=LZ4YCICB4QQVLQAUZD6ZFABXLYSRKC4OFZESDVUGIOFJMZBRMMSA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503484487 (72h46m20.374567532s) [noise=true , address=145.79.227.39:4014 , piece_id=U7SKMZGV33HMSQLQIFIBZ6WNYC7JJ2CENSK5LCW3KVMP53T7467Q, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503757176 (72h46m20.374840221s) [noise=true , address=145.79.227.24:4582 , piece_id=FGYLRBVDMMLYWTTKLC3B2P2MHTECHEBH2DJOOKAHBR6TBAMKFBXA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503773727 (72h46m20.374856772s) [noise=true , address=145.79.227.40:4049 , piece_id=VX3PNKMHR75TNJLKRBLYS3KT42ZTPT4X6I5QS2AKNMGDQDVE4WDA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.504114126 (72h46m20.375197161s) [noise=true , address=77.90.31.17:4771 , piece_id=R6PLE6VLZNL23TJC6V6MRBLJP6PKB2DDGQMPPGLMM23WN6J7SH5Q, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.504441298 (72h46m20.375524343s) [noise=true , address=77.90.31.18:4807 , piece_id=7FXKUHBJT7AHDC4PHCZ73Q62GDSOUFRFIVAD337ORGE7YUTAULVA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.505240540 (72h46m20.376323585s) [noise=true , address=77.90.31.14:4662 , piece_id=BO6DQNLFNMG6HAPKOAG2DHWDD3YMX46IAYGDVQG7THC5EVZORJRA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.506022409 (72h46m20.377105454s) [noise=true , address=145.79.227.20:4438 , piece_id=U3FCNUY4ASY3ZGX7ZY26GE2NA52GJNKJMW7XKZKALT2MWBCZMHFA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.506709457 (72h46m20.377792502s) [noise=true , address=145.79.227.43:4158 , piece_id=QN7JC2XQK6FKMIYD2WXQ7GMDILEWDUO2HXZLZERVJ24WWIMHFOVA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:22.339915069 (72h46m22.210998114s) [upload=63694]: Next returned nil segment Dec 8 16:15:22.339932321 (72h46m22.211015366s) [upload=63694]: Waiting for error group managing segments... Dec 8 16:15:22.655133540 (72h46m22.526216575s) [noise=true , address=145.79.227.43:4158 , piece_id=QN7JC2XQK6FKMIYD2WXQ7GMDILEWDUO2HXZLZERVJ24WWIMHFOVA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.660462928 (72h46m22.531545983s) [noise=true , address=145.79.227.17:4971 , piece_id=KRCKO7NXT2XOYK4QHVJK5UIU246TVB45EE7ELVUT365XTJ7RHKJQ, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.683969820 (72h46m22.555052855s) [noise=true , address=77.90.31.11:4555 , piece_id=LZ4YCICB4QQVLQAUZD6ZFABXLYSRKC4OFZESDVUGIOFJMZBRMMSA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.685188641 (72h46m22.556271686s) [noise=true , address=145.79.227.22:4511 , piece_id=RT3ULMIM24LMC4UASFJYVAOHIKJBBGLFYJP3LBJCUL6I532MTSGA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.686713104 (72h46m22.557796149s) [noise=true , address=145.79.227.16:4912 , piece_id=NIW3LT24E4LPJ5QYJVQOAR43S23JLDDOXZ6NVOURZNDTCCTLLPJA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.698627942 (72h46m22.569710987s) [noise=true , address=145.79.227.39:4014 , piece_id=U7SKMZGV33HMSQLQIFIBZ6WNYC7JJ2CENSK5LCW3KVMP53T7467Q, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.713359143 (72h46m22.584442188s) [noise=true , address=145.79.227.40:4049 , piece_id=VX3PNKMHR75TNJLKRBLYS3KT42ZTPT4X6I5QS2AKNMGDQDVE4WDA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.731975212 (72h46m22.603058257s) [noise=true , address=77.90.31.17:4771 , piece_id=R6PLE6VLZNL23TJC6V6MRBLJP6PKB2DDGQMPPGLMM23WN6J7SH5Q, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.741125236 (72h46m22.612208281s) [noise=true , address=145.79.227.20:4438 , piece_id=U3FCNUY4ASY3ZGX7ZY26GE2NA52GJNKJMW7XKZKALT2MWBCZMHFA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.773771311 (72h46m22.644854346s) [noise=true , address=77.90.31.18:4807 , piece_id=7FXKUHBJT7AHDC4PHCZ73Q62GDSOUFRFIVAD337ORGE7YUTAULVA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.906364795 (72h46m22.777447841s) [noise=true , address=77.90.31.14:4662 , piece_id=BO6DQNLFNMG6HAPKOAG2DHWDD3YMX46IAYGDVQG7THC5EVZORJRA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:23.367795705 (72h46m23.23887874s) [noise=true , address=145.79.227.24:4582 , piece_id=FGYLRBVDMMLYWTTKLC3B2P2MHTECHEBH2DJOOKAHBR6TBAMKFBXA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:24.582400944 (72h46m24.453483979s) [noise=true , address=145.79.227.33:4870 , piece_id=IK7NDJD5GNFMWB26JYOMCSNIAT5TTTQOUXB2VFYSNRGIQ2MVMXAQ, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:24.582421023 (72h46m24.453504068s) [seg_pos={0 0}, upload=63694]: Segment reached optimal threshold of 13 pieces. Dec 8 16:15:24.582652363 (72h46m24.453735408s) [noise=true , address=145.79.227.38:4979 , piece_id=HCELMHS3GNLCKHFCQU3V45M4OETQZPL5MQJBUOUF5VPA44HXZW2Q, seg_pos={0 0}, upload=63694]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma): protocol: expected piece hash; context canceled; EOF Dec 8 16:15:24.582659086 (72h46m24.453742121s) [noise=true , address=145.79.227.38:4979 , piece_id=HCELMHS3GNLCKHFCQU3V45M4OETQZPL5MQJBUOUF5VPA44HXZW2Q, seg_pos={0 0}, upload=63694]: Long tail cancellation due to optimal threshold Dec 8 16:15:24.582740842 (72h46m24.453823877s) [noise=true , address=145.79.227.34:4906 , piece_id=AIZQB5ZLXPCLPT5Q7NLNSI3FPBPAMBVUIXKZCTA3ML7L3XP2UY6A, seg_pos={0 0}, upload=63694]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 8 16:15:24.582753055 (72h46m24.45383609s) [noise=true , address=145.79.227.34:4906 , piece_id=AIZQB5ZLXPCLPT5Q7NLNSI3FPBPAMBVUIXKZCTA3ML7L3XP2UY6A, seg_pos={0 0}, upload=63694]: Long tail cancellation due to optimal threshold Dec 8 16:15:24.583044900 (72h46m24.454127935s) [noise=true , address=145.79.227.29:4941 , piece_id=GJQXPA7WEZUTVJNELRBJVTU4LIGXYD7XLEPR5SWF4NBWWMXBW2SA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: context canceled; context canceled Dec 8 16:15:24.583052926 (72h46m24.454135961s) [noise=true , address=145.79.227.29:4941 , piece_id=GJQXPA7WEZUTVJNELRBJVTU4LIGXYD7XLEPR5SWF4NBWWMXBW2SA, seg_pos={0 0}, upload=63694]: Long tail cancellation due to optimal threshold Dec 8 16:15:24.583071441 (72h46m24.454154486s) [seg_pos={0 0}, upload=63694]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 16:15:24.583162014 (72h46m24.454245059s) [upload=63694]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 16:15:24.583171562 (72h46m24.454254627s) [upload=63694]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 16:15:25.042230695 (72h46m24.91331374s) [upload=63694]: Done uploading segments... Dec 8 16:15:25.042258869 (72h46m24.913341914s) [upload=63694]: Upload finished. err: <nil> Duration: 5.771331053s core uplink us-select-1
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 477.634062ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/6c02152d3764cef5 FAIL (2025-12-08 16:45:41 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 477.661844ms core satellite eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.....X.X....................................................] duration: 5.807417467s trace: https://jaeger.dev.storj.io/trace/690f79ef204fee2 FAIL (2025-12-08 16:45:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.807446443s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 24.873914128s trace: https://jaeger.dev.storj.io/trace/5d817b557064ae10 FAIL (2025-12-08 16:45:26 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 24.873944575s core uplink eu1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................X...X.X..................................] duration: 7.049452523s trace: https://jaeger.dev.storj.io/trace/20c661c94f160c60 FAIL (2025-12-08 16:45:23 UTC) orbiter warning: context deadline exceeded Duration: 7.049474485s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................XX.........................................] duration: 5.931374356s trace: https://jaeger.dev.storj.io/trace/13568c7eb15b2369 FAIL (2025-12-08 16:45:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, Piece ID: U3FCNUY4ASY3ZGX7ZY26GE2NA52GJNKJMW7XKZKALT2MWBCZMHFA) context deadline exceeded; piecestore: (Node ID: 1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, Piece ID: QN7JC2XQK6FKMIYD2WXQ7GMDILEWDUO2HXZLZERVJ24WWIMHFOVA) context deadline exceeded; piecestore: (Node ID: 12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, Piece ID: KRCKO7NXT2XOYK4QHVJK5UIU246TVB45EE7ELVUT365XTJ7RHKJQ) context deadline exceeded; piecestore: (Node ID: 122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, Piece ID: U7SKMZGV33HMSQLQIFIBZ6WNYC7JJ2CENSK5LCW3KVMP53T7467Q) context deadline exceeded; piecestore: (Node ID: 1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, Piece ID: FGYLRBVDMMLYWTTKLC3B2P2MHTECHEBH2DJOOKAHBR6TBAMKFBXA) context deadline exceeded ======LOGS====== Dec 8 16:15:19.690262294 (72h46m19.561345339s) [upload=63694]: Starting upload Dec 8 16:15:19.690357154 (72h46m19.56144022s) [upload=63694]: Done starting upload Dec 8 16:15:19.690425354 (72h46m19.561508409s) [upload=63694]: Uploading segments... Dec 8 16:15:19.690652296 (72h46m19.561735341s) [upload=63694]: Got next segment. Inline: false Dec 8 16:15:19.690693895 (72h46m19.56177694s) [upload=63694]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 16:15:19.690700889 (72h46m19.561783934s) [upload=63694]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 16:15:20.501121517 (72h46m20.372204572s) [seg_pos={0 0}, upload=63694]: Begin upload segment... Dec 8 16:15:20.501188935 (72h46m20.372271981s) [seg_pos={0 0}, upload=63694]: Done begin upload segment. Dec 8 16:15:20.501225655 (72h46m20.3723087s) [noise=true , address=145.79.227.16:4912 , piece_id=NIW3LT24E4LPJ5QYJVQOAR43S23JLDDOXZ6NVOURZNDTCCTLLPJA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.501311268 (72h46m20.372394313s) [noise=true , address=145.79.227.34:4906 , piece_id=AIZQB5ZLXPCLPT5Q7NLNSI3FPBPAMBVUIXKZCTA3ML7L3XP2UY6A, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.501987676 (72h46m20.373070722s) [noise=true , address=145.79.227.17:4971 , piece_id=KRCKO7NXT2XOYK4QHVJK5UIU246TVB45EE7ELVUT365XTJ7RHKJQ, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.502073379 (72h46m20.373156424s) [noise=true , address=145.79.227.38:4979 , piece_id=HCELMHS3GNLCKHFCQU3V45M4OETQZPL5MQJBUOUF5VPA44HXZW2Q, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.502559184 (72h46m20.373642229s) [noise=true , address=145.79.227.33:4870 , piece_id=IK7NDJD5GNFMWB26JYOMCSNIAT5TTTQOUXB2VFYSNRGIQ2MVMXAQ, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.502903350 (72h46m20.373986385s) [noise=true , address=145.79.227.22:4511 , piece_id=RT3ULMIM24LMC4UASFJYVAOHIKJBBGLFYJP3LBJCUL6I532MTSGA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503199825 (72h46m20.37428287s) [noise=true , address=145.79.227.29:4941 , piece_id=GJQXPA7WEZUTVJNELRBJVTU4LIGXYD7XLEPR5SWF4NBWWMXBW2SA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503205956 (72h46m20.374289001s) [noise=true , address=77.90.31.11:4555 , piece_id=LZ4YCICB4QQVLQAUZD6ZFABXLYSRKC4OFZESDVUGIOFJMZBRMMSA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503484487 (72h46m20.374567532s) [noise=true , address=145.79.227.39:4014 , piece_id=U7SKMZGV33HMSQLQIFIBZ6WNYC7JJ2CENSK5LCW3KVMP53T7467Q, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503757176 (72h46m20.374840221s) [noise=true , address=145.79.227.24:4582 , piece_id=FGYLRBVDMMLYWTTKLC3B2P2MHTECHEBH2DJOOKAHBR6TBAMKFBXA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503773727 (72h46m20.374856772s) [noise=true , address=145.79.227.40:4049 , piece_id=VX3PNKMHR75TNJLKRBLYS3KT42ZTPT4X6I5QS2AKNMGDQDVE4WDA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.504114126 (72h46m20.375197161s) [noise=true , address=77.90.31.17:4771 , piece_id=R6PLE6VLZNL23TJC6V6MRBLJP6PKB2DDGQMPPGLMM23WN6J7SH5Q, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.504441298 (72h46m20.375524343s) [noise=true , address=77.90.31.18:4807 , piece_id=7FXKUHBJT7AHDC4PHCZ73Q62GDSOUFRFIVAD337ORGE7YUTAULVA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.505240540 (72h46m20.376323585s) [noise=true , address=77.90.31.14:4662 , piece_id=BO6DQNLFNMG6HAPKOAG2DHWDD3YMX46IAYGDVQG7THC5EVZORJRA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.506022409 (72h46m20.377105454s) [noise=true , address=145.79.227.20:4438 , piece_id=U3FCNUY4ASY3ZGX7ZY26GE2NA52GJNKJMW7XKZKALT2MWBCZMHFA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.506709457 (72h46m20.377792502s) [noise=true , address=145.79.227.43:4158 , piece_id=QN7JC2XQK6FKMIYD2WXQ7GMDILEWDUO2HXZLZERVJ24WWIMHFOVA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:22.339915069 (72h46m22.210998114s) [upload=63694]: Next returned nil segment Dec 8 16:15:22.339932321 (72h46m22.211015366s) [upload=63694]: Waiting for error group managing segments... Dec 8 16:15:22.655133540 (72h46m22.526216575s) [noise=true , address=145.79.227.43:4158 , piece_id=QN7JC2XQK6FKMIYD2WXQ7GMDILEWDUO2HXZLZERVJ24WWIMHFOVA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.660462928 (72h46m22.531545983s) [noise=true , address=145.79.227.17:4971 , piece_id=KRCKO7NXT2XOYK4QHVJK5UIU246TVB45EE7ELVUT365XTJ7RHKJQ, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.683969820 (72h46m22.555052855s) [noise=true , address=77.90.31.11:4555 , piece_id=LZ4YCICB4QQVLQAUZD6ZFABXLYSRKC4OFZESDVUGIOFJMZBRMMSA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.685188641 (72h46m22.556271686s) [noise=true , address=145.79.227.22:4511 , piece_id=RT3ULMIM24LMC4UASFJYVAOHIKJBBGLFYJP3LBJCUL6I532MTSGA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.686713104 (72h46m22.557796149s) [noise=true , address=145.79.227.16:4912 , piece_id=NIW3LT24E4LPJ5QYJVQOAR43S23JLDDOXZ6NVOURZNDTCCTLLPJA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.698627942 (72h46m22.569710987s) [noise=true , address=145.79.227.39:4014 , piece_id=U7SKMZGV33HMSQLQIFIBZ6WNYC7JJ2CENSK5LCW3KVMP53T7467Q, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.713359143 (72h46m22.584442188s) [noise=true , address=145.79.227.40:4049 , piece_id=VX3PNKMHR75TNJLKRBLYS3KT42ZTPT4X6I5QS2AKNMGDQDVE4WDA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.731975212 (72h46m22.603058257s) [noise=true , address=77.90.31.17:4771 , piece_id=R6PLE6VLZNL23TJC6V6MRBLJP6PKB2DDGQMPPGLMM23WN6J7SH5Q, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.741125236 (72h46m22.612208281s) [noise=true , address=145.79.227.20:4438 , piece_id=U3FCNUY4ASY3ZGX7ZY26GE2NA52GJNKJMW7XKZKALT2MWBCZMHFA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.773771311 (72h46m22.644854346s) [noise=true , address=77.90.31.18:4807 , piece_id=7FXKUHBJT7AHDC4PHCZ73Q62GDSOUFRFIVAD337ORGE7YUTAULVA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.906364795 (72h46m22.777447841s) [noise=true , address=77.90.31.14:4662 , piece_id=BO6DQNLFNMG6HAPKOAG2DHWDD3YMX46IAYGDVQG7THC5EVZORJRA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:23.367795705 (72h46m23.23887874s) [noise=true , address=145.79.227.24:4582 , piece_id=FGYLRBVDMMLYWTTKLC3B2P2MHTECHEBH2DJOOKAHBR6TBAMKFBXA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:24.582400944 (72h46m24.453483979s) [noise=true , address=145.79.227.33:4870 , piece_id=IK7NDJD5GNFMWB26JYOMCSNIAT5TTTQOUXB2VFYSNRGIQ2MVMXAQ, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:24.582421023 (72h46m24.453504068s) [seg_pos={0 0}, upload=63694]: Segment reached optimal threshold of 13 pieces. Dec 8 16:15:24.582652363 (72h46m24.453735408s) [noise=true , address=145.79.227.38:4979 , piece_id=HCELMHS3GNLCKHFCQU3V45M4OETQZPL5MQJBUOUF5VPA44HXZW2Q, seg_pos={0 0}, upload=63694]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma): protocol: expected piece hash; context canceled; EOF Dec 8 16:15:24.582659086 (72h46m24.453742121s) [noise=true , address=145.79.227.38:4979 , piece_id=HCELMHS3GNLCKHFCQU3V45M4OETQZPL5MQJBUOUF5VPA44HXZW2Q, seg_pos={0 0}, upload=63694]: Long tail cancellation due to optimal threshold Dec 8 16:15:24.582740842 (72h46m24.453823877s) [noise=true , address=145.79.227.34:4906 , piece_id=AIZQB5ZLXPCLPT5Q7NLNSI3FPBPAMBVUIXKZCTA3ML7L3XP2UY6A, seg_pos={0 0}, upload=63694]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 8 16:15:24.582753055 (72h46m24.45383609s) [noise=true , address=145.79.227.34:4906 , piece_id=AIZQB5ZLXPCLPT5Q7NLNSI3FPBPAMBVUIXKZCTA3ML7L3XP2UY6A, seg_pos={0 0}, upload=63694]: Long tail cancellation due to optimal threshold Dec 8 16:15:24.583044900 (72h46m24.454127935s) [noise=true , address=145.79.227.29:4941 , piece_id=GJQXPA7WEZUTVJNELRBJVTU4LIGXYD7XLEPR5SWF4NBWWMXBW2SA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: context canceled; context canceled Dec 8 16:15:24.583052926 (72h46m24.454135961s) [noise=true , address=145.79.227.29:4941 , piece_id=GJQXPA7WEZUTVJNELRBJVTU4LIGXYD7XLEPR5SWF4NBWWMXBW2SA, seg_pos={0 0}, upload=63694]: Long tail cancellation due to optimal threshold Dec 8 16:15:24.583071441 (72h46m24.454154486s) [seg_pos={0 0}, upload=63694]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 16:15:24.583162014 (72h46m24.454245059s) [upload=63694]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 16:15:24.583171562 (72h46m24.454254627s) [upload=63694]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 16:15:25.042230695 (72h46m24.91331374s) [upload=63694]: Done uploading segments... Dec 8 16:15:25.042258869 (72h46m24.913341914s) [upload=63694]: Upload finished. err: <nil> Duration: 5.931403382s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 21 out of 60 is failed (threshold=6) [X...X..XX..X......X.X.XX...X.X.X.X.....X....XXXX......XX...X] duration: 5.001302273s trace: https://jaeger.dev.storj.io/trace/450d7df9d8d51f5c FAIL (2025-12-08 16:45:20 UTC) orbiter warning: context deadline exceeded Duration: 5.001330075s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................XX........................................] duration: 5.943450505s trace: https://jaeger.dev.storj.io/trace/13568c7eb15b2369 FAIL (2025-12-08 16:45:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, Piece ID: U3FCNUY4ASY3ZGX7ZY26GE2NA52GJNKJMW7XKZKALT2MWBCZMHFA) context deadline exceeded; piecestore: (Node ID: 1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, Piece ID: QN7JC2XQK6FKMIYD2WXQ7GMDILEWDUO2HXZLZERVJ24WWIMHFOVA) context deadline exceeded; piecestore: (Node ID: 12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, Piece ID: KRCKO7NXT2XOYK4QHVJK5UIU246TVB45EE7ELVUT365XTJ7RHKJQ) context deadline exceeded; piecestore: (Node ID: 122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, Piece ID: U7SKMZGV33HMSQLQIFIBZ6WNYC7JJ2CENSK5LCW3KVMP53T7467Q) context deadline exceeded; piecestore: (Node ID: 1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, Piece ID: FGYLRBVDMMLYWTTKLC3B2P2MHTECHEBH2DJOOKAHBR6TBAMKFBXA) context deadline exceeded ======LOGS====== Dec 8 16:15:19.690262294 (72h46m19.561345339s) [upload=63694]: Starting upload Dec 8 16:15:19.690357154 (72h46m19.56144022s) [upload=63694]: Done starting upload Dec 8 16:15:19.690425354 (72h46m19.561508409s) [upload=63694]: Uploading segments... Dec 8 16:15:19.690652296 (72h46m19.561735341s) [upload=63694]: Got next segment. Inline: false Dec 8 16:15:19.690693895 (72h46m19.56177694s) [upload=63694]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 16:15:19.690700889 (72h46m19.561783934s) [upload=63694]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 16:15:20.501121517 (72h46m20.372204572s) [seg_pos={0 0}, upload=63694]: Begin upload segment... Dec 8 16:15:20.501188935 (72h46m20.372271981s) [seg_pos={0 0}, upload=63694]: Done begin upload segment. Dec 8 16:15:20.501225655 (72h46m20.3723087s) [noise=true , address=145.79.227.16:4912 , piece_id=NIW3LT24E4LPJ5QYJVQOAR43S23JLDDOXZ6NVOURZNDTCCTLLPJA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.501311268 (72h46m20.372394313s) [noise=true , address=145.79.227.34:4906 , piece_id=AIZQB5ZLXPCLPT5Q7NLNSI3FPBPAMBVUIXKZCTA3ML7L3XP2UY6A, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.501987676 (72h46m20.373070722s) [noise=true , address=145.79.227.17:4971 , piece_id=KRCKO7NXT2XOYK4QHVJK5UIU246TVB45EE7ELVUT365XTJ7RHKJQ, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.502073379 (72h46m20.373156424s) [noise=true , address=145.79.227.38:4979 , piece_id=HCELMHS3GNLCKHFCQU3V45M4OETQZPL5MQJBUOUF5VPA44HXZW2Q, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.502559184 (72h46m20.373642229s) [noise=true , address=145.79.227.33:4870 , piece_id=IK7NDJD5GNFMWB26JYOMCSNIAT5TTTQOUXB2VFYSNRGIQ2MVMXAQ, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.502903350 (72h46m20.373986385s) [noise=true , address=145.79.227.22:4511 , piece_id=RT3ULMIM24LMC4UASFJYVAOHIKJBBGLFYJP3LBJCUL6I532MTSGA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503199825 (72h46m20.37428287s) [noise=true , address=145.79.227.29:4941 , piece_id=GJQXPA7WEZUTVJNELRBJVTU4LIGXYD7XLEPR5SWF4NBWWMXBW2SA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503205956 (72h46m20.374289001s) [noise=true , address=77.90.31.11:4555 , piece_id=LZ4YCICB4QQVLQAUZD6ZFABXLYSRKC4OFZESDVUGIOFJMZBRMMSA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503484487 (72h46m20.374567532s) [noise=true , address=145.79.227.39:4014 , piece_id=U7SKMZGV33HMSQLQIFIBZ6WNYC7JJ2CENSK5LCW3KVMP53T7467Q, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503757176 (72h46m20.374840221s) [noise=true , address=145.79.227.24:4582 , piece_id=FGYLRBVDMMLYWTTKLC3B2P2MHTECHEBH2DJOOKAHBR6TBAMKFBXA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503773727 (72h46m20.374856772s) [noise=true , address=145.79.227.40:4049 , piece_id=VX3PNKMHR75TNJLKRBLYS3KT42ZTPT4X6I5QS2AKNMGDQDVE4WDA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.504114126 (72h46m20.375197161s) [noise=true , address=77.90.31.17:4771 , piece_id=R6PLE6VLZNL23TJC6V6MRBLJP6PKB2DDGQMPPGLMM23WN6J7SH5Q, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.504441298 (72h46m20.375524343s) [noise=true , address=77.90.31.18:4807 , piece_id=7FXKUHBJT7AHDC4PHCZ73Q62GDSOUFRFIVAD337ORGE7YUTAULVA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.505240540 (72h46m20.376323585s) [noise=true , address=77.90.31.14:4662 , piece_id=BO6DQNLFNMG6HAPKOAG2DHWDD3YMX46IAYGDVQG7THC5EVZORJRA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.506022409 (72h46m20.377105454s) [noise=true , address=145.79.227.20:4438 , piece_id=U3FCNUY4ASY3ZGX7ZY26GE2NA52GJNKJMW7XKZKALT2MWBCZMHFA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.506709457 (72h46m20.377792502s) [noise=true , address=145.79.227.43:4158 , piece_id=QN7JC2XQK6FKMIYD2WXQ7GMDILEWDUO2HXZLZERVJ24WWIMHFOVA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:22.339915069 (72h46m22.210998114s) [upload=63694]: Next returned nil segment Dec 8 16:15:22.339932321 (72h46m22.211015366s) [upload=63694]: Waiting for error group managing segments... Dec 8 16:15:22.655133540 (72h46m22.526216575s) [noise=true , address=145.79.227.43:4158 , piece_id=QN7JC2XQK6FKMIYD2WXQ7GMDILEWDUO2HXZLZERVJ24WWIMHFOVA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.660462928 (72h46m22.531545983s) [noise=true , address=145.79.227.17:4971 , piece_id=KRCKO7NXT2XOYK4QHVJK5UIU246TVB45EE7ELVUT365XTJ7RHKJQ, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.683969820 (72h46m22.555052855s) [noise=true , address=77.90.31.11:4555 , piece_id=LZ4YCICB4QQVLQAUZD6ZFABXLYSRKC4OFZESDVUGIOFJMZBRMMSA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.685188641 (72h46m22.556271686s) [noise=true , address=145.79.227.22:4511 , piece_id=RT3ULMIM24LMC4UASFJYVAOHIKJBBGLFYJP3LBJCUL6I532MTSGA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.686713104 (72h46m22.557796149s) [noise=true , address=145.79.227.16:4912 , piece_id=NIW3LT24E4LPJ5QYJVQOAR43S23JLDDOXZ6NVOURZNDTCCTLLPJA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.698627942 (72h46m22.569710987s) [noise=true , address=145.79.227.39:4014 , piece_id=U7SKMZGV33HMSQLQIFIBZ6WNYC7JJ2CENSK5LCW3KVMP53T7467Q, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.713359143 (72h46m22.584442188s) [noise=true , address=145.79.227.40:4049 , piece_id=VX3PNKMHR75TNJLKRBLYS3KT42ZTPT4X6I5QS2AKNMGDQDVE4WDA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.731975212 (72h46m22.603058257s) [noise=true , address=77.90.31.17:4771 , piece_id=R6PLE6VLZNL23TJC6V6MRBLJP6PKB2DDGQMPPGLMM23WN6J7SH5Q, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.741125236 (72h46m22.612208281s) [noise=true , address=145.79.227.20:4438 , piece_id=U3FCNUY4ASY3ZGX7ZY26GE2NA52GJNKJMW7XKZKALT2MWBCZMHFA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.773771311 (72h46m22.644854346s) [noise=true , address=77.90.31.18:4807 , piece_id=7FXKUHBJT7AHDC4PHCZ73Q62GDSOUFRFIVAD337ORGE7YUTAULVA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.906364795 (72h46m22.777447841s) [noise=true , address=77.90.31.14:4662 , piece_id=BO6DQNLFNMG6HAPKOAG2DHWDD3YMX46IAYGDVQG7THC5EVZORJRA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:23.367795705 (72h46m23.23887874s) [noise=true , address=145.79.227.24:4582 , piece_id=FGYLRBVDMMLYWTTKLC3B2P2MHTECHEBH2DJOOKAHBR6TBAMKFBXA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:24.582400944 (72h46m24.453483979s) [noise=true , address=145.79.227.33:4870 , piece_id=IK7NDJD5GNFMWB26JYOMCSNIAT5TTTQOUXB2VFYSNRGIQ2MVMXAQ, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:24.582421023 (72h46m24.453504068s) [seg_pos={0 0}, upload=63694]: Segment reached optimal threshold of 13 pieces. Dec 8 16:15:24.582652363 (72h46m24.453735408s) [noise=true , address=145.79.227.38:4979 , piece_id=HCELMHS3GNLCKHFCQU3V45M4OETQZPL5MQJBUOUF5VPA44HXZW2Q, seg_pos={0 0}, upload=63694]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma): protocol: expected piece hash; context canceled; EOF Dec 8 16:15:24.582659086 (72h46m24.453742121s) [noise=true , address=145.79.227.38:4979 , piece_id=HCELMHS3GNLCKHFCQU3V45M4OETQZPL5MQJBUOUF5VPA44HXZW2Q, seg_pos={0 0}, upload=63694]: Long tail cancellation due to optimal threshold Dec 8 16:15:24.582740842 (72h46m24.453823877s) [noise=true , address=145.79.227.34:4906 , piece_id=AIZQB5ZLXPCLPT5Q7NLNSI3FPBPAMBVUIXKZCTA3ML7L3XP2UY6A, seg_pos={0 0}, upload=63694]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 8 16:15:24.582753055 (72h46m24.45383609s) [noise=true , address=145.79.227.34:4906 , piece_id=AIZQB5ZLXPCLPT5Q7NLNSI3FPBPAMBVUIXKZCTA3ML7L3XP2UY6A, seg_pos={0 0}, upload=63694]: Long tail cancellation due to optimal threshold Dec 8 16:15:24.583044900 (72h46m24.454127935s) [noise=true , address=145.79.227.29:4941 , piece_id=GJQXPA7WEZUTVJNELRBJVTU4LIGXYD7XLEPR5SWF4NBWWMXBW2SA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: context canceled; context canceled Dec 8 16:15:24.583052926 (72h46m24.454135961s) [noise=true , address=145.79.227.29:4941 , piece_id=GJQXPA7WEZUTVJNELRBJVTU4LIGXYD7XLEPR5SWF4NBWWMXBW2SA, seg_pos={0 0}, upload=63694]: Long tail cancellation due to optimal threshold Dec 8 16:15:24.583071441 (72h46m24.454154486s) [seg_pos={0 0}, upload=63694]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 16:15:24.583162014 (72h46m24.454245059s) [upload=63694]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 16:15:24.583171562 (72h46m24.454254627s) [upload=63694]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 16:15:25.042230695 (72h46m24.91331374s) [upload=63694]: Done uploading segments... Dec 8 16:15:25.042258869 (72h46m24.913341914s) [upload=63694]: Upload finished. err: <nil> Duration: 5.943484769s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 6.639364548s trace: https://jaeger.dev.storj.io/trace/1e9e467d23cfcdb3 FAIL (2025-12-08 16:44:52 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.63939712s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 2.032879264s trace: https://jaeger.dev.storj.io/trace/4dad09bb52e33232 FAIL (2025-12-08 16:44:49 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 2.032911956s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................XX.......................................] duration: 6.171461662s trace: https://jaeger.dev.storj.io/trace/13568c7eb15b2369 FAIL (2025-12-08 16:44:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, Piece ID: U3FCNUY4ASY3ZGX7ZY26GE2NA52GJNKJMW7XKZKALT2MWBCZMHFA) context deadline exceeded; piecestore: (Node ID: 1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, Piece ID: QN7JC2XQK6FKMIYD2WXQ7GMDILEWDUO2HXZLZERVJ24WWIMHFOVA) context deadline exceeded; piecestore: (Node ID: 12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, Piece ID: KRCKO7NXT2XOYK4QHVJK5UIU246TVB45EE7ELVUT365XTJ7RHKJQ) context deadline exceeded; piecestore: (Node ID: 122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, Piece ID: U7SKMZGV33HMSQLQIFIBZ6WNYC7JJ2CENSK5LCW3KVMP53T7467Q) context deadline exceeded; piecestore: (Node ID: 1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, Piece ID: FGYLRBVDMMLYWTTKLC3B2P2MHTECHEBH2DJOOKAHBR6TBAMKFBXA) context deadline exceeded ======LOGS====== Dec 8 16:15:19.690262294 (72h46m19.561345339s) [upload=63694]: Starting upload Dec 8 16:15:19.690357154 (72h46m19.56144022s) [upload=63694]: Done starting upload Dec 8 16:15:19.690425354 (72h46m19.561508409s) [upload=63694]: Uploading segments... Dec 8 16:15:19.690652296 (72h46m19.561735341s) [upload=63694]: Got next segment. Inline: false Dec 8 16:15:19.690693895 (72h46m19.56177694s) [upload=63694]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 16:15:19.690700889 (72h46m19.561783934s) [upload=63694]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 16:15:20.501121517 (72h46m20.372204572s) [seg_pos={0 0}, upload=63694]: Begin upload segment... Dec 8 16:15:20.501188935 (72h46m20.372271981s) [seg_pos={0 0}, upload=63694]: Done begin upload segment. Dec 8 16:15:20.501225655 (72h46m20.3723087s) [noise=true , address=145.79.227.16:4912 , piece_id=NIW3LT24E4LPJ5QYJVQOAR43S23JLDDOXZ6NVOURZNDTCCTLLPJA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.501311268 (72h46m20.372394313s) [noise=true , address=145.79.227.34:4906 , piece_id=AIZQB5ZLXPCLPT5Q7NLNSI3FPBPAMBVUIXKZCTA3ML7L3XP2UY6A, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.501987676 (72h46m20.373070722s) [noise=true , address=145.79.227.17:4971 , piece_id=KRCKO7NXT2XOYK4QHVJK5UIU246TVB45EE7ELVUT365XTJ7RHKJQ, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.502073379 (72h46m20.373156424s) [noise=true , address=145.79.227.38:4979 , piece_id=HCELMHS3GNLCKHFCQU3V45M4OETQZPL5MQJBUOUF5VPA44HXZW2Q, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.502559184 (72h46m20.373642229s) [noise=true , address=145.79.227.33:4870 , piece_id=IK7NDJD5GNFMWB26JYOMCSNIAT5TTTQOUXB2VFYSNRGIQ2MVMXAQ, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.502903350 (72h46m20.373986385s) [noise=true , address=145.79.227.22:4511 , piece_id=RT3ULMIM24LMC4UASFJYVAOHIKJBBGLFYJP3LBJCUL6I532MTSGA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503199825 (72h46m20.37428287s) [noise=true , address=145.79.227.29:4941 , piece_id=GJQXPA7WEZUTVJNELRBJVTU4LIGXYD7XLEPR5SWF4NBWWMXBW2SA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503205956 (72h46m20.374289001s) [noise=true , address=77.90.31.11:4555 , piece_id=LZ4YCICB4QQVLQAUZD6ZFABXLYSRKC4OFZESDVUGIOFJMZBRMMSA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503484487 (72h46m20.374567532s) [noise=true , address=145.79.227.39:4014 , piece_id=U7SKMZGV33HMSQLQIFIBZ6WNYC7JJ2CENSK5LCW3KVMP53T7467Q, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503757176 (72h46m20.374840221s) [noise=true , address=145.79.227.24:4582 , piece_id=FGYLRBVDMMLYWTTKLC3B2P2MHTECHEBH2DJOOKAHBR6TBAMKFBXA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503773727 (72h46m20.374856772s) [noise=true , address=145.79.227.40:4049 , piece_id=VX3PNKMHR75TNJLKRBLYS3KT42ZTPT4X6I5QS2AKNMGDQDVE4WDA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.504114126 (72h46m20.375197161s) [noise=true , address=77.90.31.17:4771 , piece_id=R6PLE6VLZNL23TJC6V6MRBLJP6PKB2DDGQMPPGLMM23WN6J7SH5Q, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.504441298 (72h46m20.375524343s) [noise=true , address=77.90.31.18:4807 , piece_id=7FXKUHBJT7AHDC4PHCZ73Q62GDSOUFRFIVAD337ORGE7YUTAULVA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.505240540 (72h46m20.376323585s) [noise=true , address=77.90.31.14:4662 , piece_id=BO6DQNLFNMG6HAPKOAG2DHWDD3YMX46IAYGDVQG7THC5EVZORJRA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.506022409 (72h46m20.377105454s) [noise=true , address=145.79.227.20:4438 , piece_id=U3FCNUY4ASY3ZGX7ZY26GE2NA52GJNKJMW7XKZKALT2MWBCZMHFA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.506709457 (72h46m20.377792502s) [noise=true , address=145.79.227.43:4158 , piece_id=QN7JC2XQK6FKMIYD2WXQ7GMDILEWDUO2HXZLZERVJ24WWIMHFOVA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:22.339915069 (72h46m22.210998114s) [upload=63694]: Next returned nil segment Dec 8 16:15:22.339932321 (72h46m22.211015366s) [upload=63694]: Waiting for error group managing segments... Dec 8 16:15:22.655133540 (72h46m22.526216575s) [noise=true , address=145.79.227.43:4158 , piece_id=QN7JC2XQK6FKMIYD2WXQ7GMDILEWDUO2HXZLZERVJ24WWIMHFOVA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.660462928 (72h46m22.531545983s) [noise=true , address=145.79.227.17:4971 , piece_id=KRCKO7NXT2XOYK4QHVJK5UIU246TVB45EE7ELVUT365XTJ7RHKJQ, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.683969820 (72h46m22.555052855s) [noise=true , address=77.90.31.11:4555 , piece_id=LZ4YCICB4QQVLQAUZD6ZFABXLYSRKC4OFZESDVUGIOFJMZBRMMSA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.685188641 (72h46m22.556271686s) [noise=true , address=145.79.227.22:4511 , piece_id=RT3ULMIM24LMC4UASFJYVAOHIKJBBGLFYJP3LBJCUL6I532MTSGA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.686713104 (72h46m22.557796149s) [noise=true , address=145.79.227.16:4912 , piece_id=NIW3LT24E4LPJ5QYJVQOAR43S23JLDDOXZ6NVOURZNDTCCTLLPJA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.698627942 (72h46m22.569710987s) [noise=true , address=145.79.227.39:4014 , piece_id=U7SKMZGV33HMSQLQIFIBZ6WNYC7JJ2CENSK5LCW3KVMP53T7467Q, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.713359143 (72h46m22.584442188s) [noise=true , address=145.79.227.40:4049 , piece_id=VX3PNKMHR75TNJLKRBLYS3KT42ZTPT4X6I5QS2AKNMGDQDVE4WDA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.731975212 (72h46m22.603058257s) [noise=true , address=77.90.31.17:4771 , piece_id=R6PLE6VLZNL23TJC6V6MRBLJP6PKB2DDGQMPPGLMM23WN6J7SH5Q, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.741125236 (72h46m22.612208281s) [noise=true , address=145.79.227.20:4438 , piece_id=U3FCNUY4ASY3ZGX7ZY26GE2NA52GJNKJMW7XKZKALT2MWBCZMHFA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.773771311 (72h46m22.644854346s) [noise=true , address=77.90.31.18:4807 , piece_id=7FXKUHBJT7AHDC4PHCZ73Q62GDSOUFRFIVAD337ORGE7YUTAULVA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.906364795 (72h46m22.777447841s) [noise=true , address=77.90.31.14:4662 , piece_id=BO6DQNLFNMG6HAPKOAG2DHWDD3YMX46IAYGDVQG7THC5EVZORJRA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:23.367795705 (72h46m23.23887874s) [noise=true , address=145.79.227.24:4582 , piece_id=FGYLRBVDMMLYWTTKLC3B2P2MHTECHEBH2DJOOKAHBR6TBAMKFBXA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:24.582400944 (72h46m24.453483979s) [noise=true , address=145.79.227.33:4870 , piece_id=IK7NDJD5GNFMWB26JYOMCSNIAT5TTTQOUXB2VFYSNRGIQ2MVMXAQ, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:24.582421023 (72h46m24.453504068s) [seg_pos={0 0}, upload=63694]: Segment reached optimal threshold of 13 pieces. Dec 8 16:15:24.582652363 (72h46m24.453735408s) [noise=true , address=145.79.227.38:4979 , piece_id=HCELMHS3GNLCKHFCQU3V45M4OETQZPL5MQJBUOUF5VPA44HXZW2Q, seg_pos={0 0}, upload=63694]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma): protocol: expected piece hash; context canceled; EOF Dec 8 16:15:24.582659086 (72h46m24.453742121s) [noise=true , address=145.79.227.38:4979 , piece_id=HCELMHS3GNLCKHFCQU3V45M4OETQZPL5MQJBUOUF5VPA44HXZW2Q, seg_pos={0 0}, upload=63694]: Long tail cancellation due to optimal threshold Dec 8 16:15:24.582740842 (72h46m24.453823877s) [noise=true , address=145.79.227.34:4906 , piece_id=AIZQB5ZLXPCLPT5Q7NLNSI3FPBPAMBVUIXKZCTA3ML7L3XP2UY6A, seg_pos={0 0}, upload=63694]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 8 16:15:24.582753055 (72h46m24.45383609s) [noise=true , address=145.79.227.34:4906 , piece_id=AIZQB5ZLXPCLPT5Q7NLNSI3FPBPAMBVUIXKZCTA3ML7L3XP2UY6A, seg_pos={0 0}, upload=63694]: Long tail cancellation due to optimal threshold Dec 8 16:15:24.583044900 (72h46m24.454127935s) [noise=true , address=145.79.227.29:4941 , piece_id=GJQXPA7WEZUTVJNELRBJVTU4LIGXYD7XLEPR5SWF4NBWWMXBW2SA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: context canceled; context canceled Dec 8 16:15:24.583052926 (72h46m24.454135961s) [noise=true , address=145.79.227.29:4941 , piece_id=GJQXPA7WEZUTVJNELRBJVTU4LIGXYD7XLEPR5SWF4NBWWMXBW2SA, seg_pos={0 0}, upload=63694]: Long tail cancellation due to optimal threshold Dec 8 16:15:24.583071441 (72h46m24.454154486s) [seg_pos={0 0}, upload=63694]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 16:15:24.583162014 (72h46m24.454245059s) [upload=63694]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 16:15:24.583171562 (72h46m24.454254627s) [upload=63694]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 16:15:25.042230695 (72h46m24.91331374s) [upload=63694]: Done uploading segments... Dec 8 16:15:25.042258869 (72h46m24.913341914s) [upload=63694]: Upload finished. err: <nil> Duration: 6.171489876s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................X...X.X.................................] duration: 6.452501483s trace: https://jaeger.dev.storj.io/trace/20c661c94f160c60 FAIL (2025-12-08 16:44:42 UTC) orbiter warning: context deadline exceeded Duration: 6.452534065s edge gateway us-select-1
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 470.602465ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/6c02152d3764cef5 FAIL (2025-12-08 16:44:39 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 470.618255ms core satellite eu1
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.316154036s remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/6c02152d3764cef5 FAIL (2025-12-08 16:44:38 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 1m0.316181629s core satellite eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 5.342495322s trace: https://jaeger.dev.storj.io/trace/5d817b557064ae10 FAIL (2025-12-08 16:44:34 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.342526162s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 20 out of 60 is failed (threshold=6) [.X...X..XX..X......X.X.XX...X.X.X.X.....X....XXXX......XX...] duration: 3.501446823s trace: https://jaeger.dev.storj.io/trace/200a3d46e55af7e7 FAIL (2025-12-08 16:44:26 UTC) orbiter warning: context deadline exceeded Duration: 3.501465217s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [......X.X...................................................] duration: 5.700896696s trace: https://jaeger.dev.storj.io/trace/690f79ef204fee2 FAIL (2025-12-08 16:44:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.700920813s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 6.137864433s trace: https://jaeger.dev.storj.io/trace/1e9e467d23cfcdb3 FAIL (2025-12-08 16:44:14 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.137896745s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................XX......................................] duration: 5.94222618s trace: https://jaeger.dev.storj.io/trace/13568c7eb15b2369 FAIL (2025-12-08 16:44:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, Piece ID: U3FCNUY4ASY3ZGX7ZY26GE2NA52GJNKJMW7XKZKALT2MWBCZMHFA) context deadline exceeded; piecestore: (Node ID: 1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, Piece ID: QN7JC2XQK6FKMIYD2WXQ7GMDILEWDUO2HXZLZERVJ24WWIMHFOVA) context deadline exceeded; piecestore: (Node ID: 12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, Piece ID: KRCKO7NXT2XOYK4QHVJK5UIU246TVB45EE7ELVUT365XTJ7RHKJQ) context deadline exceeded; piecestore: (Node ID: 122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, Piece ID: U7SKMZGV33HMSQLQIFIBZ6WNYC7JJ2CENSK5LCW3KVMP53T7467Q) context deadline exceeded; piecestore: (Node ID: 1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, Piece ID: FGYLRBVDMMLYWTTKLC3B2P2MHTECHEBH2DJOOKAHBR6TBAMKFBXA) context deadline exceeded ======LOGS====== Dec 8 16:15:19.690262294 (72h46m19.561345339s) [upload=63694]: Starting upload Dec 8 16:15:19.690357154 (72h46m19.56144022s) [upload=63694]: Done starting upload Dec 8 16:15:19.690425354 (72h46m19.561508409s) [upload=63694]: Uploading segments... Dec 8 16:15:19.690652296 (72h46m19.561735341s) [upload=63694]: Got next segment. Inline: false Dec 8 16:15:19.690693895 (72h46m19.56177694s) [upload=63694]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 16:15:19.690700889 (72h46m19.561783934s) [upload=63694]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 16:15:20.501121517 (72h46m20.372204572s) [seg_pos={0 0}, upload=63694]: Begin upload segment... Dec 8 16:15:20.501188935 (72h46m20.372271981s) [seg_pos={0 0}, upload=63694]: Done begin upload segment. Dec 8 16:15:20.501225655 (72h46m20.3723087s) [noise=true , address=145.79.227.16:4912 , piece_id=NIW3LT24E4LPJ5QYJVQOAR43S23JLDDOXZ6NVOURZNDTCCTLLPJA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.501311268 (72h46m20.372394313s) [noise=true , address=145.79.227.34:4906 , piece_id=AIZQB5ZLXPCLPT5Q7NLNSI3FPBPAMBVUIXKZCTA3ML7L3XP2UY6A, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.501987676 (72h46m20.373070722s) [noise=true , address=145.79.227.17:4971 , piece_id=KRCKO7NXT2XOYK4QHVJK5UIU246TVB45EE7ELVUT365XTJ7RHKJQ, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.502073379 (72h46m20.373156424s) [noise=true , address=145.79.227.38:4979 , piece_id=HCELMHS3GNLCKHFCQU3V45M4OETQZPL5MQJBUOUF5VPA44HXZW2Q, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.502559184 (72h46m20.373642229s) [noise=true , address=145.79.227.33:4870 , piece_id=IK7NDJD5GNFMWB26JYOMCSNIAT5TTTQOUXB2VFYSNRGIQ2MVMXAQ, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.502903350 (72h46m20.373986385s) [noise=true , address=145.79.227.22:4511 , piece_id=RT3ULMIM24LMC4UASFJYVAOHIKJBBGLFYJP3LBJCUL6I532MTSGA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503199825 (72h46m20.37428287s) [noise=true , address=145.79.227.29:4941 , piece_id=GJQXPA7WEZUTVJNELRBJVTU4LIGXYD7XLEPR5SWF4NBWWMXBW2SA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503205956 (72h46m20.374289001s) [noise=true , address=77.90.31.11:4555 , piece_id=LZ4YCICB4QQVLQAUZD6ZFABXLYSRKC4OFZESDVUGIOFJMZBRMMSA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503484487 (72h46m20.374567532s) [noise=true , address=145.79.227.39:4014 , piece_id=U7SKMZGV33HMSQLQIFIBZ6WNYC7JJ2CENSK5LCW3KVMP53T7467Q, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503757176 (72h46m20.374840221s) [noise=true , address=145.79.227.24:4582 , piece_id=FGYLRBVDMMLYWTTKLC3B2P2MHTECHEBH2DJOOKAHBR6TBAMKFBXA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503773727 (72h46m20.374856772s) [noise=true , address=145.79.227.40:4049 , piece_id=VX3PNKMHR75TNJLKRBLYS3KT42ZTPT4X6I5QS2AKNMGDQDVE4WDA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.504114126 (72h46m20.375197161s) [noise=true , address=77.90.31.17:4771 , piece_id=R6PLE6VLZNL23TJC6V6MRBLJP6PKB2DDGQMPPGLMM23WN6J7SH5Q, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.504441298 (72h46m20.375524343s) [noise=true , address=77.90.31.18:4807 , piece_id=7FXKUHBJT7AHDC4PHCZ73Q62GDSOUFRFIVAD337ORGE7YUTAULVA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.505240540 (72h46m20.376323585s) [noise=true , address=77.90.31.14:4662 , piece_id=BO6DQNLFNMG6HAPKOAG2DHWDD3YMX46IAYGDVQG7THC5EVZORJRA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.506022409 (72h46m20.377105454s) [noise=true , address=145.79.227.20:4438 , piece_id=U3FCNUY4ASY3ZGX7ZY26GE2NA52GJNKJMW7XKZKALT2MWBCZMHFA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.506709457 (72h46m20.377792502s) [noise=true , address=145.79.227.43:4158 , piece_id=QN7JC2XQK6FKMIYD2WXQ7GMDILEWDUO2HXZLZERVJ24WWIMHFOVA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:22.339915069 (72h46m22.210998114s) [upload=63694]: Next returned nil segment Dec 8 16:15:22.339932321 (72h46m22.211015366s) [upload=63694]: Waiting for error group managing segments... Dec 8 16:15:22.655133540 (72h46m22.526216575s) [noise=true , address=145.79.227.43:4158 , piece_id=QN7JC2XQK6FKMIYD2WXQ7GMDILEWDUO2HXZLZERVJ24WWIMHFOVA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.660462928 (72h46m22.531545983s) [noise=true , address=145.79.227.17:4971 , piece_id=KRCKO7NXT2XOYK4QHVJK5UIU246TVB45EE7ELVUT365XTJ7RHKJQ, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.683969820 (72h46m22.555052855s) [noise=true , address=77.90.31.11:4555 , piece_id=LZ4YCICB4QQVLQAUZD6ZFABXLYSRKC4OFZESDVUGIOFJMZBRMMSA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.685188641 (72h46m22.556271686s) [noise=true , address=145.79.227.22:4511 , piece_id=RT3ULMIM24LMC4UASFJYVAOHIKJBBGLFYJP3LBJCUL6I532MTSGA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.686713104 (72h46m22.557796149s) [noise=true , address=145.79.227.16:4912 , piece_id=NIW3LT24E4LPJ5QYJVQOAR43S23JLDDOXZ6NVOURZNDTCCTLLPJA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.698627942 (72h46m22.569710987s) [noise=true , address=145.79.227.39:4014 , piece_id=U7SKMZGV33HMSQLQIFIBZ6WNYC7JJ2CENSK5LCW3KVMP53T7467Q, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.713359143 (72h46m22.584442188s) [noise=true , address=145.79.227.40:4049 , piece_id=VX3PNKMHR75TNJLKRBLYS3KT42ZTPT4X6I5QS2AKNMGDQDVE4WDA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.731975212 (72h46m22.603058257s) [noise=true , address=77.90.31.17:4771 , piece_id=R6PLE6VLZNL23TJC6V6MRBLJP6PKB2DDGQMPPGLMM23WN6J7SH5Q, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.741125236 (72h46m22.612208281s) [noise=true , address=145.79.227.20:4438 , piece_id=U3FCNUY4ASY3ZGX7ZY26GE2NA52GJNKJMW7XKZKALT2MWBCZMHFA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.773771311 (72h46m22.644854346s) [noise=true , address=77.90.31.18:4807 , piece_id=7FXKUHBJT7AHDC4PHCZ73Q62GDSOUFRFIVAD337ORGE7YUTAULVA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.906364795 (72h46m22.777447841s) [noise=true , address=77.90.31.14:4662 , piece_id=BO6DQNLFNMG6HAPKOAG2DHWDD3YMX46IAYGDVQG7THC5EVZORJRA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:23.367795705 (72h46m23.23887874s) [noise=true , address=145.79.227.24:4582 , piece_id=FGYLRBVDMMLYWTTKLC3B2P2MHTECHEBH2DJOOKAHBR6TBAMKFBXA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:24.582400944 (72h46m24.453483979s) [noise=true , address=145.79.227.33:4870 , piece_id=IK7NDJD5GNFMWB26JYOMCSNIAT5TTTQOUXB2VFYSNRGIQ2MVMXAQ, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:24.582421023 (72h46m24.453504068s) [seg_pos={0 0}, upload=63694]: Segment reached optimal threshold of 13 pieces. Dec 8 16:15:24.582652363 (72h46m24.453735408s) [noise=true , address=145.79.227.38:4979 , piece_id=HCELMHS3GNLCKHFCQU3V45M4OETQZPL5MQJBUOUF5VPA44HXZW2Q, seg_pos={0 0}, upload=63694]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma): protocol: expected piece hash; context canceled; EOF Dec 8 16:15:24.582659086 (72h46m24.453742121s) [noise=true , address=145.79.227.38:4979 , piece_id=HCELMHS3GNLCKHFCQU3V45M4OETQZPL5MQJBUOUF5VPA44HXZW2Q, seg_pos={0 0}, upload=63694]: Long tail cancellation due to optimal threshold Dec 8 16:15:24.582740842 (72h46m24.453823877s) [noise=true , address=145.79.227.34:4906 , piece_id=AIZQB5ZLXPCLPT5Q7NLNSI3FPBPAMBVUIXKZCTA3ML7L3XP2UY6A, seg_pos={0 0}, upload=63694]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 8 16:15:24.582753055 (72h46m24.45383609s) [noise=true , address=145.79.227.34:4906 , piece_id=AIZQB5ZLXPCLPT5Q7NLNSI3FPBPAMBVUIXKZCTA3ML7L3XP2UY6A, seg_pos={0 0}, upload=63694]: Long tail cancellation due to optimal threshold Dec 8 16:15:24.583044900 (72h46m24.454127935s) [noise=true , address=145.79.227.29:4941 , piece_id=GJQXPA7WEZUTVJNELRBJVTU4LIGXYD7XLEPR5SWF4NBWWMXBW2SA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: context canceled; context canceled Dec 8 16:15:24.583052926 (72h46m24.454135961s) [noise=true , address=145.79.227.29:4941 , piece_id=GJQXPA7WEZUTVJNELRBJVTU4LIGXYD7XLEPR5SWF4NBWWMXBW2SA, seg_pos={0 0}, upload=63694]: Long tail cancellation due to optimal threshold Dec 8 16:15:24.583071441 (72h46m24.454154486s) [seg_pos={0 0}, upload=63694]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 16:15:24.583162014 (72h46m24.454245059s) [upload=63694]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 16:15:24.583171562 (72h46m24.454254627s) [upload=63694]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 16:15:25.042230695 (72h46m24.91331374s) [upload=63694]: Done uploading segments... Dec 8 16:15:25.042258869 (72h46m24.913341914s) [upload=63694]: Upload finished. err: <nil> Duration: 5.942252731s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 20 out of 60 is failed (threshold=6) [..X...X..XX..X......X.X.XX...X.X.X.X.....X....XXXX......XX..] duration: 3.449201498s trace: https://jaeger.dev.storj.io/trace/200a3d46e55af7e7 FAIL (2025-12-08 16:43:53 UTC) orbiter warning: context deadline exceeded Duration: 3.44922937s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 5.010118805s trace: https://jaeger.dev.storj.io/trace/5d817b557064ae10 FAIL (2025-12-08 16:43:50 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.010148272s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 1.472442575s trace: https://jaeger.dev.storj.io/trace/3bdb8560e64b821d FAIL (2025-12-08 16:43:41 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 1.472468975s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 1.979113079s trace: https://jaeger.dev.storj.io/trace/4dad09bb52e33232 FAIL (2025-12-08 16:43:40 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.97916609s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................X...X.X................................] duration: 6.68121807s trace: https://jaeger.dev.storj.io/trace/20c661c94f160c60 FAIL (2025-12-08 16:43:38 UTC) orbiter warning: context deadline exceeded Duration: 6.681244349s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................XX.....................................] duration: 5.87912047s trace: https://jaeger.dev.storj.io/trace/13568c7eb15b2369 FAIL (2025-12-08 16:43:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, Piece ID: U3FCNUY4ASY3ZGX7ZY26GE2NA52GJNKJMW7XKZKALT2MWBCZMHFA) context deadline exceeded; piecestore: (Node ID: 1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, Piece ID: QN7JC2XQK6FKMIYD2WXQ7GMDILEWDUO2HXZLZERVJ24WWIMHFOVA) context deadline exceeded; piecestore: (Node ID: 12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, Piece ID: KRCKO7NXT2XOYK4QHVJK5UIU246TVB45EE7ELVUT365XTJ7RHKJQ) context deadline exceeded; piecestore: (Node ID: 122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, Piece ID: U7SKMZGV33HMSQLQIFIBZ6WNYC7JJ2CENSK5LCW3KVMP53T7467Q) context deadline exceeded; piecestore: (Node ID: 1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, Piece ID: FGYLRBVDMMLYWTTKLC3B2P2MHTECHEBH2DJOOKAHBR6TBAMKFBXA) context deadline exceeded ======LOGS====== Dec 8 16:15:19.690262294 (72h46m19.561345339s) [upload=63694]: Starting upload Dec 8 16:15:19.690357154 (72h46m19.56144022s) [upload=63694]: Done starting upload Dec 8 16:15:19.690425354 (72h46m19.561508409s) [upload=63694]: Uploading segments... Dec 8 16:15:19.690652296 (72h46m19.561735341s) [upload=63694]: Got next segment. Inline: false Dec 8 16:15:19.690693895 (72h46m19.56177694s) [upload=63694]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 16:15:19.690700889 (72h46m19.561783934s) [upload=63694]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 16:15:20.501121517 (72h46m20.372204572s) [seg_pos={0 0}, upload=63694]: Begin upload segment... Dec 8 16:15:20.501188935 (72h46m20.372271981s) [seg_pos={0 0}, upload=63694]: Done begin upload segment. Dec 8 16:15:20.501225655 (72h46m20.3723087s) [noise=true , address=145.79.227.16:4912 , piece_id=NIW3LT24E4LPJ5QYJVQOAR43S23JLDDOXZ6NVOURZNDTCCTLLPJA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.501311268 (72h46m20.372394313s) [noise=true , address=145.79.227.34:4906 , piece_id=AIZQB5ZLXPCLPT5Q7NLNSI3FPBPAMBVUIXKZCTA3ML7L3XP2UY6A, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.501987676 (72h46m20.373070722s) [noise=true , address=145.79.227.17:4971 , piece_id=KRCKO7NXT2XOYK4QHVJK5UIU246TVB45EE7ELVUT365XTJ7RHKJQ, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.502073379 (72h46m20.373156424s) [noise=true , address=145.79.227.38:4979 , piece_id=HCELMHS3GNLCKHFCQU3V45M4OETQZPL5MQJBUOUF5VPA44HXZW2Q, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.502559184 (72h46m20.373642229s) [noise=true , address=145.79.227.33:4870 , piece_id=IK7NDJD5GNFMWB26JYOMCSNIAT5TTTQOUXB2VFYSNRGIQ2MVMXAQ, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.502903350 (72h46m20.373986385s) [noise=true , address=145.79.227.22:4511 , piece_id=RT3ULMIM24LMC4UASFJYVAOHIKJBBGLFYJP3LBJCUL6I532MTSGA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503199825 (72h46m20.37428287s) [noise=true , address=145.79.227.29:4941 , piece_id=GJQXPA7WEZUTVJNELRBJVTU4LIGXYD7XLEPR5SWF4NBWWMXBW2SA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503205956 (72h46m20.374289001s) [noise=true , address=77.90.31.11:4555 , piece_id=LZ4YCICB4QQVLQAUZD6ZFABXLYSRKC4OFZESDVUGIOFJMZBRMMSA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503484487 (72h46m20.374567532s) [noise=true , address=145.79.227.39:4014 , piece_id=U7SKMZGV33HMSQLQIFIBZ6WNYC7JJ2CENSK5LCW3KVMP53T7467Q, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503757176 (72h46m20.374840221s) [noise=true , address=145.79.227.24:4582 , piece_id=FGYLRBVDMMLYWTTKLC3B2P2MHTECHEBH2DJOOKAHBR6TBAMKFBXA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503773727 (72h46m20.374856772s) [noise=true , address=145.79.227.40:4049 , piece_id=VX3PNKMHR75TNJLKRBLYS3KT42ZTPT4X6I5QS2AKNMGDQDVE4WDA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.504114126 (72h46m20.375197161s) [noise=true , address=77.90.31.17:4771 , piece_id=R6PLE6VLZNL23TJC6V6MRBLJP6PKB2DDGQMPPGLMM23WN6J7SH5Q, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.504441298 (72h46m20.375524343s) [noise=true , address=77.90.31.18:4807 , piece_id=7FXKUHBJT7AHDC4PHCZ73Q62GDSOUFRFIVAD337ORGE7YUTAULVA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.505240540 (72h46m20.376323585s) [noise=true , address=77.90.31.14:4662 , piece_id=BO6DQNLFNMG6HAPKOAG2DHWDD3YMX46IAYGDVQG7THC5EVZORJRA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.506022409 (72h46m20.377105454s) [noise=true , address=145.79.227.20:4438 , piece_id=U3FCNUY4ASY3ZGX7ZY26GE2NA52GJNKJMW7XKZKALT2MWBCZMHFA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.506709457 (72h46m20.377792502s) [noise=true , address=145.79.227.43:4158 , piece_id=QN7JC2XQK6FKMIYD2WXQ7GMDILEWDUO2HXZLZERVJ24WWIMHFOVA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:22.339915069 (72h46m22.210998114s) [upload=63694]: Next returned nil segment Dec 8 16:15:22.339932321 (72h46m22.211015366s) [upload=63694]: Waiting for error group managing segments... Dec 8 16:15:22.655133540 (72h46m22.526216575s) [noise=true , address=145.79.227.43:4158 , piece_id=QN7JC2XQK6FKMIYD2WXQ7GMDILEWDUO2HXZLZERVJ24WWIMHFOVA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.660462928 (72h46m22.531545983s) [noise=true , address=145.79.227.17:4971 , piece_id=KRCKO7NXT2XOYK4QHVJK5UIU246TVB45EE7ELVUT365XTJ7RHKJQ, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.683969820 (72h46m22.555052855s) [noise=true , address=77.90.31.11:4555 , piece_id=LZ4YCICB4QQVLQAUZD6ZFABXLYSRKC4OFZESDVUGIOFJMZBRMMSA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.685188641 (72h46m22.556271686s) [noise=true , address=145.79.227.22:4511 , piece_id=RT3ULMIM24LMC4UASFJYVAOHIKJBBGLFYJP3LBJCUL6I532MTSGA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.686713104 (72h46m22.557796149s) [noise=true , address=145.79.227.16:4912 , piece_id=NIW3LT24E4LPJ5QYJVQOAR43S23JLDDOXZ6NVOURZNDTCCTLLPJA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.698627942 (72h46m22.569710987s) [noise=true , address=145.79.227.39:4014 , piece_id=U7SKMZGV33HMSQLQIFIBZ6WNYC7JJ2CENSK5LCW3KVMP53T7467Q, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.713359143 (72h46m22.584442188s) [noise=true , address=145.79.227.40:4049 , piece_id=VX3PNKMHR75TNJLKRBLYS3KT42ZTPT4X6I5QS2AKNMGDQDVE4WDA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.731975212 (72h46m22.603058257s) [noise=true , address=77.90.31.17:4771 , piece_id=R6PLE6VLZNL23TJC6V6MRBLJP6PKB2DDGQMPPGLMM23WN6J7SH5Q, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.741125236 (72h46m22.612208281s) [noise=true , address=145.79.227.20:4438 , piece_id=U3FCNUY4ASY3ZGX7ZY26GE2NA52GJNKJMW7XKZKALT2MWBCZMHFA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.773771311 (72h46m22.644854346s) [noise=true , address=77.90.31.18:4807 , piece_id=7FXKUHBJT7AHDC4PHCZ73Q62GDSOUFRFIVAD337ORGE7YUTAULVA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.906364795 (72h46m22.777447841s) [noise=true , address=77.90.31.14:4662 , piece_id=BO6DQNLFNMG6HAPKOAG2DHWDD3YMX46IAYGDVQG7THC5EVZORJRA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:23.367795705 (72h46m23.23887874s) [noise=true , address=145.79.227.24:4582 , piece_id=FGYLRBVDMMLYWTTKLC3B2P2MHTECHEBH2DJOOKAHBR6TBAMKFBXA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:24.582400944 (72h46m24.453483979s) [noise=true , address=145.79.227.33:4870 , piece_id=IK7NDJD5GNFMWB26JYOMCSNIAT5TTTQOUXB2VFYSNRGIQ2MVMXAQ, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:24.582421023 (72h46m24.453504068s) [seg_pos={0 0}, upload=63694]: Segment reached optimal threshold of 13 pieces. Dec 8 16:15:24.582652363 (72h46m24.453735408s) [noise=true , address=145.79.227.38:4979 , piece_id=HCELMHS3GNLCKHFCQU3V45M4OETQZPL5MQJBUOUF5VPA44HXZW2Q, seg_pos={0 0}, upload=63694]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma): protocol: expected piece hash; context canceled; EOF Dec 8 16:15:24.582659086 (72h46m24.453742121s) [noise=true , address=145.79.227.38:4979 , piece_id=HCELMHS3GNLCKHFCQU3V45M4OETQZPL5MQJBUOUF5VPA44HXZW2Q, seg_pos={0 0}, upload=63694]: Long tail cancellation due to optimal threshold Dec 8 16:15:24.582740842 (72h46m24.453823877s) [noise=true , address=145.79.227.34:4906 , piece_id=AIZQB5ZLXPCLPT5Q7NLNSI3FPBPAMBVUIXKZCTA3ML7L3XP2UY6A, seg_pos={0 0}, upload=63694]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 8 16:15:24.582753055 (72h46m24.45383609s) [noise=true , address=145.79.227.34:4906 , piece_id=AIZQB5ZLXPCLPT5Q7NLNSI3FPBPAMBVUIXKZCTA3ML7L3XP2UY6A, seg_pos={0 0}, upload=63694]: Long tail cancellation due to optimal threshold Dec 8 16:15:24.583044900 (72h46m24.454127935s) [noise=true , address=145.79.227.29:4941 , piece_id=GJQXPA7WEZUTVJNELRBJVTU4LIGXYD7XLEPR5SWF4NBWWMXBW2SA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: context canceled; context canceled Dec 8 16:15:24.583052926 (72h46m24.454135961s) [noise=true , address=145.79.227.29:4941 , piece_id=GJQXPA7WEZUTVJNELRBJVTU4LIGXYD7XLEPR5SWF4NBWWMXBW2SA, seg_pos={0 0}, upload=63694]: Long tail cancellation due to optimal threshold Dec 8 16:15:24.583071441 (72h46m24.454154486s) [seg_pos={0 0}, upload=63694]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 16:15:24.583162014 (72h46m24.454245059s) [upload=63694]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 16:15:24.583171562 (72h46m24.454254627s) [upload=63694]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 16:15:25.042230695 (72h46m24.91331374s) [upload=63694]: Done uploading segments... Dec 8 16:15:25.042258869 (72h46m24.913341914s) [upload=63694]: Upload finished. err: <nil> Duration: 5.879144717s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 5.866834062s trace: https://jaeger.dev.storj.io/trace/1e9e467d23cfcdb3 FAIL (2025-12-08 16:43:32 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.866866484s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.......X.X..................................................] duration: 12.410020812s trace: https://jaeger.dev.storj.io/trace/690f79ef204fee2 FAIL (2025-12-08 16:43:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 12.410049666s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 4.913652498s trace: https://jaeger.dev.storj.io/trace/5d817b557064ae10 FAIL (2025-12-08 16:43:16 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.913681493s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 20 out of 60 is failed (threshold=6) [...X...X..XX..X......X.X.XX...X.X.X.X.....X....XXXX......XX.] duration: 3.380993875s trace: https://jaeger.dev.storj.io/trace/200a3d46e55af7e7 FAIL (2025-12-08 16:43:14 UTC) orbiter warning: context deadline exceeded Duration: 3.381025105s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......................X...X.X...............................] duration: 14.400075225s trace: https://jaeger.dev.storj.io/trace/20c661c94f160c60 FAIL (2025-12-08 16:43:07 UTC) orbiter warning: context deadline exceeded Duration: 14.40010358s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 20 out of 60 is failed (threshold=6) [....X...X..XX..X......X.X.XX...X.X.X.X.....X....XXXX......XX] duration: 5.001387662s trace: https://jaeger.dev.storj.io/trace/200a3d46e55af7e7 FAIL (2025-12-08 16:42:46 UTC) orbiter warning: context deadline exceeded Duration: 5.001409293s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................XX....................................] duration: 5.788369096s trace: https://jaeger.dev.storj.io/trace/13568c7eb15b2369 FAIL (2025-12-08 16:42:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, Piece ID: U3FCNUY4ASY3ZGX7ZY26GE2NA52GJNKJMW7XKZKALT2MWBCZMHFA) context deadline exceeded; piecestore: (Node ID: 1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, Piece ID: QN7JC2XQK6FKMIYD2WXQ7GMDILEWDUO2HXZLZERVJ24WWIMHFOVA) context deadline exceeded; piecestore: (Node ID: 12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, Piece ID: KRCKO7NXT2XOYK4QHVJK5UIU246TVB45EE7ELVUT365XTJ7RHKJQ) context deadline exceeded; piecestore: (Node ID: 122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, Piece ID: U7SKMZGV33HMSQLQIFIBZ6WNYC7JJ2CENSK5LCW3KVMP53T7467Q) context deadline exceeded; piecestore: (Node ID: 1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, Piece ID: FGYLRBVDMMLYWTTKLC3B2P2MHTECHEBH2DJOOKAHBR6TBAMKFBXA) context deadline exceeded ======LOGS====== Dec 8 16:15:19.690262294 (72h46m19.561345339s) [upload=63694]: Starting upload Dec 8 16:15:19.690357154 (72h46m19.56144022s) [upload=63694]: Done starting upload Dec 8 16:15:19.690425354 (72h46m19.561508409s) [upload=63694]: Uploading segments... Dec 8 16:15:19.690652296 (72h46m19.561735341s) [upload=63694]: Got next segment. Inline: false Dec 8 16:15:19.690693895 (72h46m19.56177694s) [upload=63694]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 16:15:19.690700889 (72h46m19.561783934s) [upload=63694]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 16:15:20.501121517 (72h46m20.372204572s) [seg_pos={0 0}, upload=63694]: Begin upload segment... Dec 8 16:15:20.501188935 (72h46m20.372271981s) [seg_pos={0 0}, upload=63694]: Done begin upload segment. Dec 8 16:15:20.501225655 (72h46m20.3723087s) [noise=true , address=145.79.227.16:4912 , piece_id=NIW3LT24E4LPJ5QYJVQOAR43S23JLDDOXZ6NVOURZNDTCCTLLPJA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.501311268 (72h46m20.372394313s) [noise=true , address=145.79.227.34:4906 , piece_id=AIZQB5ZLXPCLPT5Q7NLNSI3FPBPAMBVUIXKZCTA3ML7L3XP2UY6A, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.501987676 (72h46m20.373070722s) [noise=true , address=145.79.227.17:4971 , piece_id=KRCKO7NXT2XOYK4QHVJK5UIU246TVB45EE7ELVUT365XTJ7RHKJQ, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.502073379 (72h46m20.373156424s) [noise=true , address=145.79.227.38:4979 , piece_id=HCELMHS3GNLCKHFCQU3V45M4OETQZPL5MQJBUOUF5VPA44HXZW2Q, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.502559184 (72h46m20.373642229s) [noise=true , address=145.79.227.33:4870 , piece_id=IK7NDJD5GNFMWB26JYOMCSNIAT5TTTQOUXB2VFYSNRGIQ2MVMXAQ, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.502903350 (72h46m20.373986385s) [noise=true , address=145.79.227.22:4511 , piece_id=RT3ULMIM24LMC4UASFJYVAOHIKJBBGLFYJP3LBJCUL6I532MTSGA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503199825 (72h46m20.37428287s) [noise=true , address=145.79.227.29:4941 , piece_id=GJQXPA7WEZUTVJNELRBJVTU4LIGXYD7XLEPR5SWF4NBWWMXBW2SA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503205956 (72h46m20.374289001s) [noise=true , address=77.90.31.11:4555 , piece_id=LZ4YCICB4QQVLQAUZD6ZFABXLYSRKC4OFZESDVUGIOFJMZBRMMSA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503484487 (72h46m20.374567532s) [noise=true , address=145.79.227.39:4014 , piece_id=U7SKMZGV33HMSQLQIFIBZ6WNYC7JJ2CENSK5LCW3KVMP53T7467Q, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503757176 (72h46m20.374840221s) [noise=true , address=145.79.227.24:4582 , piece_id=FGYLRBVDMMLYWTTKLC3B2P2MHTECHEBH2DJOOKAHBR6TBAMKFBXA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503773727 (72h46m20.374856772s) [noise=true , address=145.79.227.40:4049 , piece_id=VX3PNKMHR75TNJLKRBLYS3KT42ZTPT4X6I5QS2AKNMGDQDVE4WDA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.504114126 (72h46m20.375197161s) [noise=true , address=77.90.31.17:4771 , piece_id=R6PLE6VLZNL23TJC6V6MRBLJP6PKB2DDGQMPPGLMM23WN6J7SH5Q, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.504441298 (72h46m20.375524343s) [noise=true , address=77.90.31.18:4807 , piece_id=7FXKUHBJT7AHDC4PHCZ73Q62GDSOUFRFIVAD337ORGE7YUTAULVA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.505240540 (72h46m20.376323585s) [noise=true , address=77.90.31.14:4662 , piece_id=BO6DQNLFNMG6HAPKOAG2DHWDD3YMX46IAYGDVQG7THC5EVZORJRA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.506022409 (72h46m20.377105454s) [noise=true , address=145.79.227.20:4438 , piece_id=U3FCNUY4ASY3ZGX7ZY26GE2NA52GJNKJMW7XKZKALT2MWBCZMHFA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.506709457 (72h46m20.377792502s) [noise=true , address=145.79.227.43:4158 , piece_id=QN7JC2XQK6FKMIYD2WXQ7GMDILEWDUO2HXZLZERVJ24WWIMHFOVA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:22.339915069 (72h46m22.210998114s) [upload=63694]: Next returned nil segment Dec 8 16:15:22.339932321 (72h46m22.211015366s) [upload=63694]: Waiting for error group managing segments... Dec 8 16:15:22.655133540 (72h46m22.526216575s) [noise=true , address=145.79.227.43:4158 , piece_id=QN7JC2XQK6FKMIYD2WXQ7GMDILEWDUO2HXZLZERVJ24WWIMHFOVA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.660462928 (72h46m22.531545983s) [noise=true , address=145.79.227.17:4971 , piece_id=KRCKO7NXT2XOYK4QHVJK5UIU246TVB45EE7ELVUT365XTJ7RHKJQ, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.683969820 (72h46m22.555052855s) [noise=true , address=77.90.31.11:4555 , piece_id=LZ4YCICB4QQVLQAUZD6ZFABXLYSRKC4OFZESDVUGIOFJMZBRMMSA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.685188641 (72h46m22.556271686s) [noise=true , address=145.79.227.22:4511 , piece_id=RT3ULMIM24LMC4UASFJYVAOHIKJBBGLFYJP3LBJCUL6I532MTSGA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.686713104 (72h46m22.557796149s) [noise=true , address=145.79.227.16:4912 , piece_id=NIW3LT24E4LPJ5QYJVQOAR43S23JLDDOXZ6NVOURZNDTCCTLLPJA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.698627942 (72h46m22.569710987s) [noise=true , address=145.79.227.39:4014 , piece_id=U7SKMZGV33HMSQLQIFIBZ6WNYC7JJ2CENSK5LCW3KVMP53T7467Q, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.713359143 (72h46m22.584442188s) [noise=true , address=145.79.227.40:4049 , piece_id=VX3PNKMHR75TNJLKRBLYS3KT42ZTPT4X6I5QS2AKNMGDQDVE4WDA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.731975212 (72h46m22.603058257s) [noise=true , address=77.90.31.17:4771 , piece_id=R6PLE6VLZNL23TJC6V6MRBLJP6PKB2DDGQMPPGLMM23WN6J7SH5Q, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.741125236 (72h46m22.612208281s) [noise=true , address=145.79.227.20:4438 , piece_id=U3FCNUY4ASY3ZGX7ZY26GE2NA52GJNKJMW7XKZKALT2MWBCZMHFA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.773771311 (72h46m22.644854346s) [noise=true , address=77.90.31.18:4807 , piece_id=7FXKUHBJT7AHDC4PHCZ73Q62GDSOUFRFIVAD337ORGE7YUTAULVA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.906364795 (72h46m22.777447841s) [noise=true , address=77.90.31.14:4662 , piece_id=BO6DQNLFNMG6HAPKOAG2DHWDD3YMX46IAYGDVQG7THC5EVZORJRA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:23.367795705 (72h46m23.23887874s) [noise=true , address=145.79.227.24:4582 , piece_id=FGYLRBVDMMLYWTTKLC3B2P2MHTECHEBH2DJOOKAHBR6TBAMKFBXA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:24.582400944 (72h46m24.453483979s) [noise=true , address=145.79.227.33:4870 , piece_id=IK7NDJD5GNFMWB26JYOMCSNIAT5TTTQOUXB2VFYSNRGIQ2MVMXAQ, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:24.582421023 (72h46m24.453504068s) [seg_pos={0 0}, upload=63694]: Segment reached optimal threshold of 13 pieces. Dec 8 16:15:24.582652363 (72h46m24.453735408s) [noise=true , address=145.79.227.38:4979 , piece_id=HCELMHS3GNLCKHFCQU3V45M4OETQZPL5MQJBUOUF5VPA44HXZW2Q, seg_pos={0 0}, upload=63694]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma): protocol: expected piece hash; context canceled; EOF Dec 8 16:15:24.582659086 (72h46m24.453742121s) [noise=true , address=145.79.227.38:4979 , piece_id=HCELMHS3GNLCKHFCQU3V45M4OETQZPL5MQJBUOUF5VPA44HXZW2Q, seg_pos={0 0}, upload=63694]: Long tail cancellation due to optimal threshold Dec 8 16:15:24.582740842 (72h46m24.453823877s) [noise=true , address=145.79.227.34:4906 , piece_id=AIZQB5ZLXPCLPT5Q7NLNSI3FPBPAMBVUIXKZCTA3ML7L3XP2UY6A, seg_pos={0 0}, upload=63694]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 8 16:15:24.582753055 (72h46m24.45383609s) [noise=true , address=145.79.227.34:4906 , piece_id=AIZQB5ZLXPCLPT5Q7NLNSI3FPBPAMBVUIXKZCTA3ML7L3XP2UY6A, seg_pos={0 0}, upload=63694]: Long tail cancellation due to optimal threshold Dec 8 16:15:24.583044900 (72h46m24.454127935s) [noise=true , address=145.79.227.29:4941 , piece_id=GJQXPA7WEZUTVJNELRBJVTU4LIGXYD7XLEPR5SWF4NBWWMXBW2SA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: context canceled; context canceled Dec 8 16:15:24.583052926 (72h46m24.454135961s) [noise=true , address=145.79.227.29:4941 , piece_id=GJQXPA7WEZUTVJNELRBJVTU4LIGXYD7XLEPR5SWF4NBWWMXBW2SA, seg_pos={0 0}, upload=63694]: Long tail cancellation due to optimal threshold Dec 8 16:15:24.583071441 (72h46m24.454154486s) [seg_pos={0 0}, upload=63694]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 16:15:24.583162014 (72h46m24.454245059s) [upload=63694]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 16:15:24.583171562 (72h46m24.454254627s) [upload=63694]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 16:15:25.042230695 (72h46m24.91331374s) [upload=63694]: Done uploading segments... Dec 8 16:15:25.042258869 (72h46m24.913341914s) [upload=63694]: Upload finished. err: <nil> Duration: 5.788394525s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 1.454682778s trace: https://jaeger.dev.storj.io/trace/3bdb8560e64b821d FAIL (2025-12-08 16:42:38 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 1.454718507s core uplink eu1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......................X...X.X..............................] duration: 16.7887427s trace: https://jaeger.dev.storj.io/trace/20c661c94f160c60 FAIL (2025-12-08 16:42:35 UTC) orbiter warning: context deadline exceeded Duration: 16.788804377s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 2.018094179s trace: https://jaeger.dev.storj.io/trace/4dad09bb52e33232 FAIL (2025-12-08 16:42:30 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 2.01812627s core uplink us1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)