Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [............X.X.............................................] duration: 13.266031843s trace: https://jaeger.dev.storj.io/trace/690f79ef204fee2 FAIL (2025-12-08 16:38:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 13.266065267s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................XX..............................] duration: 5.99796889s trace: https://jaeger.dev.storj.io/trace/13568c7eb15b2369 FAIL (2025-12-08 16:38:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, Piece ID: U3FCNUY4ASY3ZGX7ZY26GE2NA52GJNKJMW7XKZKALT2MWBCZMHFA) context deadline exceeded; piecestore: (Node ID: 1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, Piece ID: QN7JC2XQK6FKMIYD2WXQ7GMDILEWDUO2HXZLZERVJ24WWIMHFOVA) context deadline exceeded; piecestore: (Node ID: 12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, Piece ID: KRCKO7NXT2XOYK4QHVJK5UIU246TVB45EE7ELVUT365XTJ7RHKJQ) context deadline exceeded; piecestore: (Node ID: 122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, Piece ID: U7SKMZGV33HMSQLQIFIBZ6WNYC7JJ2CENSK5LCW3KVMP53T7467Q) context deadline exceeded; piecestore: (Node ID: 1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, Piece ID: FGYLRBVDMMLYWTTKLC3B2P2MHTECHEBH2DJOOKAHBR6TBAMKFBXA) context deadline exceeded ======LOGS====== Dec 8 16:15:19.690262294 (72h46m19.561345339s) [upload=63694]: Starting upload Dec 8 16:15:19.690357154 (72h46m19.56144022s) [upload=63694]: Done starting upload Dec 8 16:15:19.690425354 (72h46m19.561508409s) [upload=63694]: Uploading segments... Dec 8 16:15:19.690652296 (72h46m19.561735341s) [upload=63694]: Got next segment. Inline: false Dec 8 16:15:19.690693895 (72h46m19.56177694s) [upload=63694]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 16:15:19.690700889 (72h46m19.561783934s) [upload=63694]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 16:15:20.501121517 (72h46m20.372204572s) [seg_pos={0 0}, upload=63694]: Begin upload segment... Dec 8 16:15:20.501188935 (72h46m20.372271981s) [seg_pos={0 0}, upload=63694]: Done begin upload segment. Dec 8 16:15:20.501225655 (72h46m20.3723087s) [noise=true , address=145.79.227.16:4912 , piece_id=NIW3LT24E4LPJ5QYJVQOAR43S23JLDDOXZ6NVOURZNDTCCTLLPJA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.501311268 (72h46m20.372394313s) [noise=true , address=145.79.227.34:4906 , piece_id=AIZQB5ZLXPCLPT5Q7NLNSI3FPBPAMBVUIXKZCTA3ML7L3XP2UY6A, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.501987676 (72h46m20.373070722s) [noise=true , address=145.79.227.17:4971 , piece_id=KRCKO7NXT2XOYK4QHVJK5UIU246TVB45EE7ELVUT365XTJ7RHKJQ, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.502073379 (72h46m20.373156424s) [noise=true , address=145.79.227.38:4979 , piece_id=HCELMHS3GNLCKHFCQU3V45M4OETQZPL5MQJBUOUF5VPA44HXZW2Q, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.502559184 (72h46m20.373642229s) [noise=true , address=145.79.227.33:4870 , piece_id=IK7NDJD5GNFMWB26JYOMCSNIAT5TTTQOUXB2VFYSNRGIQ2MVMXAQ, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.502903350 (72h46m20.373986385s) [noise=true , address=145.79.227.22:4511 , piece_id=RT3ULMIM24LMC4UASFJYVAOHIKJBBGLFYJP3LBJCUL6I532MTSGA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503199825 (72h46m20.37428287s) [noise=true , address=145.79.227.29:4941 , piece_id=GJQXPA7WEZUTVJNELRBJVTU4LIGXYD7XLEPR5SWF4NBWWMXBW2SA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503205956 (72h46m20.374289001s) [noise=true , address=77.90.31.11:4555 , piece_id=LZ4YCICB4QQVLQAUZD6ZFABXLYSRKC4OFZESDVUGIOFJMZBRMMSA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503484487 (72h46m20.374567532s) [noise=true , address=145.79.227.39:4014 , piece_id=U7SKMZGV33HMSQLQIFIBZ6WNYC7JJ2CENSK5LCW3KVMP53T7467Q, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503757176 (72h46m20.374840221s) [noise=true , address=145.79.227.24:4582 , piece_id=FGYLRBVDMMLYWTTKLC3B2P2MHTECHEBH2DJOOKAHBR6TBAMKFBXA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503773727 (72h46m20.374856772s) [noise=true , address=145.79.227.40:4049 , piece_id=VX3PNKMHR75TNJLKRBLYS3KT42ZTPT4X6I5QS2AKNMGDQDVE4WDA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.504114126 (72h46m20.375197161s) [noise=true , address=77.90.31.17:4771 , piece_id=R6PLE6VLZNL23TJC6V6MRBLJP6PKB2DDGQMPPGLMM23WN6J7SH5Q, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.504441298 (72h46m20.375524343s) [noise=true , address=77.90.31.18:4807 , piece_id=7FXKUHBJT7AHDC4PHCZ73Q62GDSOUFRFIVAD337ORGE7YUTAULVA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.505240540 (72h46m20.376323585s) [noise=true , address=77.90.31.14:4662 , piece_id=BO6DQNLFNMG6HAPKOAG2DHWDD3YMX46IAYGDVQG7THC5EVZORJRA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.506022409 (72h46m20.377105454s) [noise=true , address=145.79.227.20:4438 , piece_id=U3FCNUY4ASY3ZGX7ZY26GE2NA52GJNKJMW7XKZKALT2MWBCZMHFA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.506709457 (72h46m20.377792502s) [noise=true , address=145.79.227.43:4158 , piece_id=QN7JC2XQK6FKMIYD2WXQ7GMDILEWDUO2HXZLZERVJ24WWIMHFOVA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:22.339915069 (72h46m22.210998114s) [upload=63694]: Next returned nil segment Dec 8 16:15:22.339932321 (72h46m22.211015366s) [upload=63694]: Waiting for error group managing segments... Dec 8 16:15:22.655133540 (72h46m22.526216575s) [noise=true , address=145.79.227.43:4158 , piece_id=QN7JC2XQK6FKMIYD2WXQ7GMDILEWDUO2HXZLZERVJ24WWIMHFOVA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.660462928 (72h46m22.531545983s) [noise=true , address=145.79.227.17:4971 , piece_id=KRCKO7NXT2XOYK4QHVJK5UIU246TVB45EE7ELVUT365XTJ7RHKJQ, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.683969820 (72h46m22.555052855s) [noise=true , address=77.90.31.11:4555 , piece_id=LZ4YCICB4QQVLQAUZD6ZFABXLYSRKC4OFZESDVUGIOFJMZBRMMSA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.685188641 (72h46m22.556271686s) [noise=true , address=145.79.227.22:4511 , piece_id=RT3ULMIM24LMC4UASFJYVAOHIKJBBGLFYJP3LBJCUL6I532MTSGA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.686713104 (72h46m22.557796149s) [noise=true , address=145.79.227.16:4912 , piece_id=NIW3LT24E4LPJ5QYJVQOAR43S23JLDDOXZ6NVOURZNDTCCTLLPJA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.698627942 (72h46m22.569710987s) [noise=true , address=145.79.227.39:4014 , piece_id=U7SKMZGV33HMSQLQIFIBZ6WNYC7JJ2CENSK5LCW3KVMP53T7467Q, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.713359143 (72h46m22.584442188s) [noise=true , address=145.79.227.40:4049 , piece_id=VX3PNKMHR75TNJLKRBLYS3KT42ZTPT4X6I5QS2AKNMGDQDVE4WDA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.731975212 (72h46m22.603058257s) [noise=true , address=77.90.31.17:4771 , piece_id=R6PLE6VLZNL23TJC6V6MRBLJP6PKB2DDGQMPPGLMM23WN6J7SH5Q, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.741125236 (72h46m22.612208281s) [noise=true , address=145.79.227.20:4438 , piece_id=U3FCNUY4ASY3ZGX7ZY26GE2NA52GJNKJMW7XKZKALT2MWBCZMHFA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.773771311 (72h46m22.644854346s) [noise=true , address=77.90.31.18:4807 , piece_id=7FXKUHBJT7AHDC4PHCZ73Q62GDSOUFRFIVAD337ORGE7YUTAULVA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.906364795 (72h46m22.777447841s) [noise=true , address=77.90.31.14:4662 , piece_id=BO6DQNLFNMG6HAPKOAG2DHWDD3YMX46IAYGDVQG7THC5EVZORJRA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:23.367795705 (72h46m23.23887874s) [noise=true , address=145.79.227.24:4582 , piece_id=FGYLRBVDMMLYWTTKLC3B2P2MHTECHEBH2DJOOKAHBR6TBAMKFBXA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:24.582400944 (72h46m24.453483979s) [noise=true , address=145.79.227.33:4870 , piece_id=IK7NDJD5GNFMWB26JYOMCSNIAT5TTTQOUXB2VFYSNRGIQ2MVMXAQ, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:24.582421023 (72h46m24.453504068s) [seg_pos={0 0}, upload=63694]: Segment reached optimal threshold of 13 pieces. Dec 8 16:15:24.582652363 (72h46m24.453735408s) [noise=true , address=145.79.227.38:4979 , piece_id=HCELMHS3GNLCKHFCQU3V45M4OETQZPL5MQJBUOUF5VPA44HXZW2Q, seg_pos={0 0}, upload=63694]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma): protocol: expected piece hash; context canceled; EOF Dec 8 16:15:24.582659086 (72h46m24.453742121s) [noise=true , address=145.79.227.38:4979 , piece_id=HCELMHS3GNLCKHFCQU3V45M4OETQZPL5MQJBUOUF5VPA44HXZW2Q, seg_pos={0 0}, upload=63694]: Long tail cancellation due to optimal threshold Dec 8 16:15:24.582740842 (72h46m24.453823877s) [noise=true , address=145.79.227.34:4906 , piece_id=AIZQB5ZLXPCLPT5Q7NLNSI3FPBPAMBVUIXKZCTA3ML7L3XP2UY6A, seg_pos={0 0}, upload=63694]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 8 16:15:24.582753055 (72h46m24.45383609s) [noise=true , address=145.79.227.34:4906 , piece_id=AIZQB5ZLXPCLPT5Q7NLNSI3FPBPAMBVUIXKZCTA3ML7L3XP2UY6A, seg_pos={0 0}, upload=63694]: Long tail cancellation due to optimal threshold Dec 8 16:15:24.583044900 (72h46m24.454127935s) [noise=true , address=145.79.227.29:4941 , piece_id=GJQXPA7WEZUTVJNELRBJVTU4LIGXYD7XLEPR5SWF4NBWWMXBW2SA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: context canceled; context canceled Dec 8 16:15:24.583052926 (72h46m24.454135961s) [noise=true , address=145.79.227.29:4941 , piece_id=GJQXPA7WEZUTVJNELRBJVTU4LIGXYD7XLEPR5SWF4NBWWMXBW2SA, seg_pos={0 0}, upload=63694]: Long tail cancellation due to optimal threshold Dec 8 16:15:24.583071441 (72h46m24.454154486s) [seg_pos={0 0}, upload=63694]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 16:15:24.583162014 (72h46m24.454245059s) [upload=63694]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 16:15:24.583171562 (72h46m24.454254627s) [upload=63694]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 16:15:25.042230695 (72h46m24.91331374s) [upload=63694]: Done uploading segments... Dec 8 16:15:25.042258869 (72h46m24.913341914s) [upload=63694]: Upload finished. err: <nil> Duration: 5.99799026s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 5.625618506s trace: https://jaeger.dev.storj.io/trace/1e9e467d23cfcdb3 FAIL (2025-12-08 16:38:50 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.625638575s core uplink us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 5.683955561s trace: https://jaeger.dev.storj.io/trace/5d817b557064ae10 FAIL (2025-12-08 16:38:45 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.683983395s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 1.524116233s trace: https://jaeger.dev.storj.io/trace/3bdb8560e64b821d FAIL (2025-12-08 16:38:31 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 1.52414111s core uplink eu1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..X..........................X...X.X........................] duration: 13.262558961s trace: https://jaeger.dev.storj.io/trace/20c661c94f160c60 FAIL (2025-12-08 16:38:24 UTC) orbiter warning: context deadline exceeded Duration: 13.262575593s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 19 out of 60 is failed (threshold=6) [....X.....X...X..XX..X......X.X.XX...X.X.X.X.....X....XXXX..] duration: 3.437646122s trace: https://jaeger.dev.storj.io/trace/69fa5d6c40dbd9ac FAIL (2025-12-08 16:38:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.437714392s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 5.9964248s trace: https://jaeger.dev.storj.io/trace/1e9e467d23cfcdb3 FAIL (2025-12-08 16:38:15 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.996562382s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 2.013879125s trace: https://jaeger.dev.storj.io/trace/4dad09bb52e33232 FAIL (2025-12-08 16:38:14 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 2.013908249s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [X............................XX.............................] duration: 5.777048722s trace: https://jaeger.dev.storj.io/trace/13568c7eb15b2369 FAIL (2025-12-08 16:38:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, Piece ID: U3FCNUY4ASY3ZGX7ZY26GE2NA52GJNKJMW7XKZKALT2MWBCZMHFA) context deadline exceeded; piecestore: (Node ID: 1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, Piece ID: QN7JC2XQK6FKMIYD2WXQ7GMDILEWDUO2HXZLZERVJ24WWIMHFOVA) context deadline exceeded; piecestore: (Node ID: 12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, Piece ID: KRCKO7NXT2XOYK4QHVJK5UIU246TVB45EE7ELVUT365XTJ7RHKJQ) context deadline exceeded; piecestore: (Node ID: 122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, Piece ID: U7SKMZGV33HMSQLQIFIBZ6WNYC7JJ2CENSK5LCW3KVMP53T7467Q) context deadline exceeded; piecestore: (Node ID: 1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, Piece ID: FGYLRBVDMMLYWTTKLC3B2P2MHTECHEBH2DJOOKAHBR6TBAMKFBXA) context deadline exceeded ======LOGS====== Dec 8 16:15:19.690262294 (72h46m19.561345339s) [upload=63694]: Starting upload Dec 8 16:15:19.690357154 (72h46m19.56144022s) [upload=63694]: Done starting upload Dec 8 16:15:19.690425354 (72h46m19.561508409s) [upload=63694]: Uploading segments... Dec 8 16:15:19.690652296 (72h46m19.561735341s) [upload=63694]: Got next segment. Inline: false Dec 8 16:15:19.690693895 (72h46m19.56177694s) [upload=63694]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 16:15:19.690700889 (72h46m19.561783934s) [upload=63694]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 16:15:20.501121517 (72h46m20.372204572s) [seg_pos={0 0}, upload=63694]: Begin upload segment... Dec 8 16:15:20.501188935 (72h46m20.372271981s) [seg_pos={0 0}, upload=63694]: Done begin upload segment. Dec 8 16:15:20.501225655 (72h46m20.3723087s) [noise=true , address=145.79.227.16:4912 , piece_id=NIW3LT24E4LPJ5QYJVQOAR43S23JLDDOXZ6NVOURZNDTCCTLLPJA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.501311268 (72h46m20.372394313s) [noise=true , address=145.79.227.34:4906 , piece_id=AIZQB5ZLXPCLPT5Q7NLNSI3FPBPAMBVUIXKZCTA3ML7L3XP2UY6A, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.501987676 (72h46m20.373070722s) [noise=true , address=145.79.227.17:4971 , piece_id=KRCKO7NXT2XOYK4QHVJK5UIU246TVB45EE7ELVUT365XTJ7RHKJQ, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.502073379 (72h46m20.373156424s) [noise=true , address=145.79.227.38:4979 , piece_id=HCELMHS3GNLCKHFCQU3V45M4OETQZPL5MQJBUOUF5VPA44HXZW2Q, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.502559184 (72h46m20.373642229s) [noise=true , address=145.79.227.33:4870 , piece_id=IK7NDJD5GNFMWB26JYOMCSNIAT5TTTQOUXB2VFYSNRGIQ2MVMXAQ, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.502903350 (72h46m20.373986385s) [noise=true , address=145.79.227.22:4511 , piece_id=RT3ULMIM24LMC4UASFJYVAOHIKJBBGLFYJP3LBJCUL6I532MTSGA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503199825 (72h46m20.37428287s) [noise=true , address=145.79.227.29:4941 , piece_id=GJQXPA7WEZUTVJNELRBJVTU4LIGXYD7XLEPR5SWF4NBWWMXBW2SA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503205956 (72h46m20.374289001s) [noise=true , address=77.90.31.11:4555 , piece_id=LZ4YCICB4QQVLQAUZD6ZFABXLYSRKC4OFZESDVUGIOFJMZBRMMSA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503484487 (72h46m20.374567532s) [noise=true , address=145.79.227.39:4014 , piece_id=U7SKMZGV33HMSQLQIFIBZ6WNYC7JJ2CENSK5LCW3KVMP53T7467Q, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503757176 (72h46m20.374840221s) [noise=true , address=145.79.227.24:4582 , piece_id=FGYLRBVDMMLYWTTKLC3B2P2MHTECHEBH2DJOOKAHBR6TBAMKFBXA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503773727 (72h46m20.374856772s) [noise=true , address=145.79.227.40:4049 , piece_id=VX3PNKMHR75TNJLKRBLYS3KT42ZTPT4X6I5QS2AKNMGDQDVE4WDA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.504114126 (72h46m20.375197161s) [noise=true , address=77.90.31.17:4771 , piece_id=R6PLE6VLZNL23TJC6V6MRBLJP6PKB2DDGQMPPGLMM23WN6J7SH5Q, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.504441298 (72h46m20.375524343s) [noise=true , address=77.90.31.18:4807 , piece_id=7FXKUHBJT7AHDC4PHCZ73Q62GDSOUFRFIVAD337ORGE7YUTAULVA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.505240540 (72h46m20.376323585s) [noise=true , address=77.90.31.14:4662 , piece_id=BO6DQNLFNMG6HAPKOAG2DHWDD3YMX46IAYGDVQG7THC5EVZORJRA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.506022409 (72h46m20.377105454s) [noise=true , address=145.79.227.20:4438 , piece_id=U3FCNUY4ASY3ZGX7ZY26GE2NA52GJNKJMW7XKZKALT2MWBCZMHFA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.506709457 (72h46m20.377792502s) [noise=true , address=145.79.227.43:4158 , piece_id=QN7JC2XQK6FKMIYD2WXQ7GMDILEWDUO2HXZLZERVJ24WWIMHFOVA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:22.339915069 (72h46m22.210998114s) [upload=63694]: Next returned nil segment Dec 8 16:15:22.339932321 (72h46m22.211015366s) [upload=63694]: Waiting for error group managing segments... Dec 8 16:15:22.655133540 (72h46m22.526216575s) [noise=true , address=145.79.227.43:4158 , piece_id=QN7JC2XQK6FKMIYD2WXQ7GMDILEWDUO2HXZLZERVJ24WWIMHFOVA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.660462928 (72h46m22.531545983s) [noise=true , address=145.79.227.17:4971 , piece_id=KRCKO7NXT2XOYK4QHVJK5UIU246TVB45EE7ELVUT365XTJ7RHKJQ, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.683969820 (72h46m22.555052855s) [noise=true , address=77.90.31.11:4555 , piece_id=LZ4YCICB4QQVLQAUZD6ZFABXLYSRKC4OFZESDVUGIOFJMZBRMMSA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.685188641 (72h46m22.556271686s) [noise=true , address=145.79.227.22:4511 , piece_id=RT3ULMIM24LMC4UASFJYVAOHIKJBBGLFYJP3LBJCUL6I532MTSGA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.686713104 (72h46m22.557796149s) [noise=true , address=145.79.227.16:4912 , piece_id=NIW3LT24E4LPJ5QYJVQOAR43S23JLDDOXZ6NVOURZNDTCCTLLPJA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.698627942 (72h46m22.569710987s) [noise=true , address=145.79.227.39:4014 , piece_id=U7SKMZGV33HMSQLQIFIBZ6WNYC7JJ2CENSK5LCW3KVMP53T7467Q, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.713359143 (72h46m22.584442188s) [noise=true , address=145.79.227.40:4049 , piece_id=VX3PNKMHR75TNJLKRBLYS3KT42ZTPT4X6I5QS2AKNMGDQDVE4WDA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.731975212 (72h46m22.603058257s) [noise=true , address=77.90.31.17:4771 , piece_id=R6PLE6VLZNL23TJC6V6MRBLJP6PKB2DDGQMPPGLMM23WN6J7SH5Q, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.741125236 (72h46m22.612208281s) [noise=true , address=145.79.227.20:4438 , piece_id=U3FCNUY4ASY3ZGX7ZY26GE2NA52GJNKJMW7XKZKALT2MWBCZMHFA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.773771311 (72h46m22.644854346s) [noise=true , address=77.90.31.18:4807 , piece_id=7FXKUHBJT7AHDC4PHCZ73Q62GDSOUFRFIVAD337ORGE7YUTAULVA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.906364795 (72h46m22.777447841s) [noise=true , address=77.90.31.14:4662 , piece_id=BO6DQNLFNMG6HAPKOAG2DHWDD3YMX46IAYGDVQG7THC5EVZORJRA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:23.367795705 (72h46m23.23887874s) [noise=true , address=145.79.227.24:4582 , piece_id=FGYLRBVDMMLYWTTKLC3B2P2MHTECHEBH2DJOOKAHBR6TBAMKFBXA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:24.582400944 (72h46m24.453483979s) [noise=true , address=145.79.227.33:4870 , piece_id=IK7NDJD5GNFMWB26JYOMCSNIAT5TTTQOUXB2VFYSNRGIQ2MVMXAQ, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:24.582421023 (72h46m24.453504068s) [seg_pos={0 0}, upload=63694]: Segment reached optimal threshold of 13 pieces. Dec 8 16:15:24.582652363 (72h46m24.453735408s) [noise=true , address=145.79.227.38:4979 , piece_id=HCELMHS3GNLCKHFCQU3V45M4OETQZPL5MQJBUOUF5VPA44HXZW2Q, seg_pos={0 0}, upload=63694]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma): protocol: expected piece hash; context canceled; EOF Dec 8 16:15:24.582659086 (72h46m24.453742121s) [noise=true , address=145.79.227.38:4979 , piece_id=HCELMHS3GNLCKHFCQU3V45M4OETQZPL5MQJBUOUF5VPA44HXZW2Q, seg_pos={0 0}, upload=63694]: Long tail cancellation due to optimal threshold Dec 8 16:15:24.582740842 (72h46m24.453823877s) [noise=true , address=145.79.227.34:4906 , piece_id=AIZQB5ZLXPCLPT5Q7NLNSI3FPBPAMBVUIXKZCTA3ML7L3XP2UY6A, seg_pos={0 0}, upload=63694]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 8 16:15:24.582753055 (72h46m24.45383609s) [noise=true , address=145.79.227.34:4906 , piece_id=AIZQB5ZLXPCLPT5Q7NLNSI3FPBPAMBVUIXKZCTA3ML7L3XP2UY6A, seg_pos={0 0}, upload=63694]: Long tail cancellation due to optimal threshold Dec 8 16:15:24.583044900 (72h46m24.454127935s) [noise=true , address=145.79.227.29:4941 , piece_id=GJQXPA7WEZUTVJNELRBJVTU4LIGXYD7XLEPR5SWF4NBWWMXBW2SA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: context canceled; context canceled Dec 8 16:15:24.583052926 (72h46m24.454135961s) [noise=true , address=145.79.227.29:4941 , piece_id=GJQXPA7WEZUTVJNELRBJVTU4LIGXYD7XLEPR5SWF4NBWWMXBW2SA, seg_pos={0 0}, upload=63694]: Long tail cancellation due to optimal threshold Dec 8 16:15:24.583071441 (72h46m24.454154486s) [seg_pos={0 0}, upload=63694]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 16:15:24.583162014 (72h46m24.454245059s) [upload=63694]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 16:15:24.583171562 (72h46m24.454254627s) [upload=63694]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 16:15:25.042230695 (72h46m24.91331374s) [upload=63694]: Done uploading segments... Dec 8 16:15:25.042258869 (72h46m24.913341914s) [upload=63694]: Upload finished. err: <nil> Duration: 5.77707962s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.............X.X............................................] duration: 10.936199337s trace: https://jaeger.dev.storj.io/trace/690f79ef204fee2 FAIL (2025-12-08 16:37:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 10.93622739s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 19 out of 60 is failed (threshold=6) [.....X.....X...X..XX..X......X.X.XX...X.X.X.X.....X....XXXX.] duration: 3.520914503s trace: https://jaeger.dev.storj.io/trace/69fa5d6c40dbd9ac FAIL (2025-12-08 16:37:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.52096023s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 1.463010096s trace: https://jaeger.dev.storj.io/trace/3bdb8560e64b821d FAIL (2025-12-08 16:37:48 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 1.463045314s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 2.033966933s trace: https://jaeger.dev.storj.io/trace/4dad09bb52e33232 FAIL (2025-12-08 16:37:43 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 2.033993994s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...X..........................X...X.X.......................] duration: 6.703459629s trace: https://jaeger.dev.storj.io/trace/20c661c94f160c60 FAIL (2025-12-08 16:37:39 UTC) orbiter warning: context deadline exceeded Duration: 6.70348137s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.X............................XX............................] duration: 5.721690983s trace: https://jaeger.dev.storj.io/trace/13568c7eb15b2369 FAIL (2025-12-08 16:37:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, Piece ID: U3FCNUY4ASY3ZGX7ZY26GE2NA52GJNKJMW7XKZKALT2MWBCZMHFA) context deadline exceeded; piecestore: (Node ID: 1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, Piece ID: QN7JC2XQK6FKMIYD2WXQ7GMDILEWDUO2HXZLZERVJ24WWIMHFOVA) context deadline exceeded; piecestore: (Node ID: 12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, Piece ID: KRCKO7NXT2XOYK4QHVJK5UIU246TVB45EE7ELVUT365XTJ7RHKJQ) context deadline exceeded; piecestore: (Node ID: 122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, Piece ID: U7SKMZGV33HMSQLQIFIBZ6WNYC7JJ2CENSK5LCW3KVMP53T7467Q) context deadline exceeded; piecestore: (Node ID: 1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, Piece ID: FGYLRBVDMMLYWTTKLC3B2P2MHTECHEBH2DJOOKAHBR6TBAMKFBXA) context deadline exceeded ======LOGS====== Dec 8 16:15:19.690262294 (72h46m19.561345339s) [upload=63694]: Starting upload Dec 8 16:15:19.690357154 (72h46m19.56144022s) [upload=63694]: Done starting upload Dec 8 16:15:19.690425354 (72h46m19.561508409s) [upload=63694]: Uploading segments... Dec 8 16:15:19.690652296 (72h46m19.561735341s) [upload=63694]: Got next segment. Inline: false Dec 8 16:15:19.690693895 (72h46m19.56177694s) [upload=63694]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 16:15:19.690700889 (72h46m19.561783934s) [upload=63694]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 16:15:20.501121517 (72h46m20.372204572s) [seg_pos={0 0}, upload=63694]: Begin upload segment... Dec 8 16:15:20.501188935 (72h46m20.372271981s) [seg_pos={0 0}, upload=63694]: Done begin upload segment. Dec 8 16:15:20.501225655 (72h46m20.3723087s) [noise=true , address=145.79.227.16:4912 , piece_id=NIW3LT24E4LPJ5QYJVQOAR43S23JLDDOXZ6NVOURZNDTCCTLLPJA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.501311268 (72h46m20.372394313s) [noise=true , address=145.79.227.34:4906 , piece_id=AIZQB5ZLXPCLPT5Q7NLNSI3FPBPAMBVUIXKZCTA3ML7L3XP2UY6A, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.501987676 (72h46m20.373070722s) [noise=true , address=145.79.227.17:4971 , piece_id=KRCKO7NXT2XOYK4QHVJK5UIU246TVB45EE7ELVUT365XTJ7RHKJQ, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.502073379 (72h46m20.373156424s) [noise=true , address=145.79.227.38:4979 , piece_id=HCELMHS3GNLCKHFCQU3V45M4OETQZPL5MQJBUOUF5VPA44HXZW2Q, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.502559184 (72h46m20.373642229s) [noise=true , address=145.79.227.33:4870 , piece_id=IK7NDJD5GNFMWB26JYOMCSNIAT5TTTQOUXB2VFYSNRGIQ2MVMXAQ, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.502903350 (72h46m20.373986385s) [noise=true , address=145.79.227.22:4511 , piece_id=RT3ULMIM24LMC4UASFJYVAOHIKJBBGLFYJP3LBJCUL6I532MTSGA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503199825 (72h46m20.37428287s) [noise=true , address=145.79.227.29:4941 , piece_id=GJQXPA7WEZUTVJNELRBJVTU4LIGXYD7XLEPR5SWF4NBWWMXBW2SA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503205956 (72h46m20.374289001s) [noise=true , address=77.90.31.11:4555 , piece_id=LZ4YCICB4QQVLQAUZD6ZFABXLYSRKC4OFZESDVUGIOFJMZBRMMSA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503484487 (72h46m20.374567532s) [noise=true , address=145.79.227.39:4014 , piece_id=U7SKMZGV33HMSQLQIFIBZ6WNYC7JJ2CENSK5LCW3KVMP53T7467Q, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503757176 (72h46m20.374840221s) [noise=true , address=145.79.227.24:4582 , piece_id=FGYLRBVDMMLYWTTKLC3B2P2MHTECHEBH2DJOOKAHBR6TBAMKFBXA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503773727 (72h46m20.374856772s) [noise=true , address=145.79.227.40:4049 , piece_id=VX3PNKMHR75TNJLKRBLYS3KT42ZTPT4X6I5QS2AKNMGDQDVE4WDA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.504114126 (72h46m20.375197161s) [noise=true , address=77.90.31.17:4771 , piece_id=R6PLE6VLZNL23TJC6V6MRBLJP6PKB2DDGQMPPGLMM23WN6J7SH5Q, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.504441298 (72h46m20.375524343s) [noise=true , address=77.90.31.18:4807 , piece_id=7FXKUHBJT7AHDC4PHCZ73Q62GDSOUFRFIVAD337ORGE7YUTAULVA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.505240540 (72h46m20.376323585s) [noise=true , address=77.90.31.14:4662 , piece_id=BO6DQNLFNMG6HAPKOAG2DHWDD3YMX46IAYGDVQG7THC5EVZORJRA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.506022409 (72h46m20.377105454s) [noise=true , address=145.79.227.20:4438 , piece_id=U3FCNUY4ASY3ZGX7ZY26GE2NA52GJNKJMW7XKZKALT2MWBCZMHFA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.506709457 (72h46m20.377792502s) [noise=true , address=145.79.227.43:4158 , piece_id=QN7JC2XQK6FKMIYD2WXQ7GMDILEWDUO2HXZLZERVJ24WWIMHFOVA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:22.339915069 (72h46m22.210998114s) [upload=63694]: Next returned nil segment Dec 8 16:15:22.339932321 (72h46m22.211015366s) [upload=63694]: Waiting for error group managing segments... Dec 8 16:15:22.655133540 (72h46m22.526216575s) [noise=true , address=145.79.227.43:4158 , piece_id=QN7JC2XQK6FKMIYD2WXQ7GMDILEWDUO2HXZLZERVJ24WWIMHFOVA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.660462928 (72h46m22.531545983s) [noise=true , address=145.79.227.17:4971 , piece_id=KRCKO7NXT2XOYK4QHVJK5UIU246TVB45EE7ELVUT365XTJ7RHKJQ, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.683969820 (72h46m22.555052855s) [noise=true , address=77.90.31.11:4555 , piece_id=LZ4YCICB4QQVLQAUZD6ZFABXLYSRKC4OFZESDVUGIOFJMZBRMMSA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.685188641 (72h46m22.556271686s) [noise=true , address=145.79.227.22:4511 , piece_id=RT3ULMIM24LMC4UASFJYVAOHIKJBBGLFYJP3LBJCUL6I532MTSGA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.686713104 (72h46m22.557796149s) [noise=true , address=145.79.227.16:4912 , piece_id=NIW3LT24E4LPJ5QYJVQOAR43S23JLDDOXZ6NVOURZNDTCCTLLPJA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.698627942 (72h46m22.569710987s) [noise=true , address=145.79.227.39:4014 , piece_id=U7SKMZGV33HMSQLQIFIBZ6WNYC7JJ2CENSK5LCW3KVMP53T7467Q, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.713359143 (72h46m22.584442188s) [noise=true , address=145.79.227.40:4049 , piece_id=VX3PNKMHR75TNJLKRBLYS3KT42ZTPT4X6I5QS2AKNMGDQDVE4WDA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.731975212 (72h46m22.603058257s) [noise=true , address=77.90.31.17:4771 , piece_id=R6PLE6VLZNL23TJC6V6MRBLJP6PKB2DDGQMPPGLMM23WN6J7SH5Q, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.741125236 (72h46m22.612208281s) [noise=true , address=145.79.227.20:4438 , piece_id=U3FCNUY4ASY3ZGX7ZY26GE2NA52GJNKJMW7XKZKALT2MWBCZMHFA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.773771311 (72h46m22.644854346s) [noise=true , address=77.90.31.18:4807 , piece_id=7FXKUHBJT7AHDC4PHCZ73Q62GDSOUFRFIVAD337ORGE7YUTAULVA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.906364795 (72h46m22.777447841s) [noise=true , address=77.90.31.14:4662 , piece_id=BO6DQNLFNMG6HAPKOAG2DHWDD3YMX46IAYGDVQG7THC5EVZORJRA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:23.367795705 (72h46m23.23887874s) [noise=true , address=145.79.227.24:4582 , piece_id=FGYLRBVDMMLYWTTKLC3B2P2MHTECHEBH2DJOOKAHBR6TBAMKFBXA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:24.582400944 (72h46m24.453483979s) [noise=true , address=145.79.227.33:4870 , piece_id=IK7NDJD5GNFMWB26JYOMCSNIAT5TTTQOUXB2VFYSNRGIQ2MVMXAQ, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:24.582421023 (72h46m24.453504068s) [seg_pos={0 0}, upload=63694]: Segment reached optimal threshold of 13 pieces. Dec 8 16:15:24.582652363 (72h46m24.453735408s) [noise=true , address=145.79.227.38:4979 , piece_id=HCELMHS3GNLCKHFCQU3V45M4OETQZPL5MQJBUOUF5VPA44HXZW2Q, seg_pos={0 0}, upload=63694]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma): protocol: expected piece hash; context canceled; EOF Dec 8 16:15:24.582659086 (72h46m24.453742121s) [noise=true , address=145.79.227.38:4979 , piece_id=HCELMHS3GNLCKHFCQU3V45M4OETQZPL5MQJBUOUF5VPA44HXZW2Q, seg_pos={0 0}, upload=63694]: Long tail cancellation due to optimal threshold Dec 8 16:15:24.582740842 (72h46m24.453823877s) [noise=true , address=145.79.227.34:4906 , piece_id=AIZQB5ZLXPCLPT5Q7NLNSI3FPBPAMBVUIXKZCTA3ML7L3XP2UY6A, seg_pos={0 0}, upload=63694]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 8 16:15:24.582753055 (72h46m24.45383609s) [noise=true , address=145.79.227.34:4906 , piece_id=AIZQB5ZLXPCLPT5Q7NLNSI3FPBPAMBVUIXKZCTA3ML7L3XP2UY6A, seg_pos={0 0}, upload=63694]: Long tail cancellation due to optimal threshold Dec 8 16:15:24.583044900 (72h46m24.454127935s) [noise=true , address=145.79.227.29:4941 , piece_id=GJQXPA7WEZUTVJNELRBJVTU4LIGXYD7XLEPR5SWF4NBWWMXBW2SA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: context canceled; context canceled Dec 8 16:15:24.583052926 (72h46m24.454135961s) [noise=true , address=145.79.227.29:4941 , piece_id=GJQXPA7WEZUTVJNELRBJVTU4LIGXYD7XLEPR5SWF4NBWWMXBW2SA, seg_pos={0 0}, upload=63694]: Long tail cancellation due to optimal threshold Dec 8 16:15:24.583071441 (72h46m24.454154486s) [seg_pos={0 0}, upload=63694]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 16:15:24.583162014 (72h46m24.454245059s) [upload=63694]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 16:15:24.583171562 (72h46m24.454254627s) [upload=63694]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 16:15:25.042230695 (72h46m24.91331374s) [upload=63694]: Done uploading segments... Dec 8 16:15:25.042258869 (72h46m24.913341914s) [upload=63694]: Upload finished. err: <nil> Duration: 5.721714897s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 4.890773029s trace: https://jaeger.dev.storj.io/trace/5d817b557064ae10 FAIL (2025-12-08 16:37:32 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.890801332s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 5.878342269s trace: https://jaeger.dev.storj.io/trace/1e9e467d23cfcdb3 FAIL (2025-12-08 16:37:21 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.878371514s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 20 out of 60 is failed (threshold=6) [X.....X.....X...X..XX..X......X.X.XX...X.X.X.X.....X....XXXX] duration: 4.998955253s trace: https://jaeger.dev.storj.io/trace/69fa5d6c40dbd9ac FAIL (2025-12-08 16:37:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001305308s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 2.029978171s trace: https://jaeger.dev.storj.io/trace/4dad09bb52e33232 FAIL (2025-12-08 16:37:04 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 2.030008008s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [....X..........................X...X.X......................] duration: 8.362069155s trace: https://jaeger.dev.storj.io/trace/20c661c94f160c60 FAIL (2025-12-08 16:36:59 UTC) orbiter warning: context deadline exceeded Duration: 8.362100344s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..............X.X...........................................] duration: 12.765085698s trace: https://jaeger.dev.storj.io/trace/690f79ef204fee2 FAIL (2025-12-08 16:36:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 12.765114783s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..X............................XX...........................] duration: 5.940978639s trace: https://jaeger.dev.storj.io/trace/13568c7eb15b2369 FAIL (2025-12-08 16:36:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, Piece ID: U3FCNUY4ASY3ZGX7ZY26GE2NA52GJNKJMW7XKZKALT2MWBCZMHFA) context deadline exceeded; piecestore: (Node ID: 1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, Piece ID: QN7JC2XQK6FKMIYD2WXQ7GMDILEWDUO2HXZLZERVJ24WWIMHFOVA) context deadline exceeded; piecestore: (Node ID: 12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, Piece ID: KRCKO7NXT2XOYK4QHVJK5UIU246TVB45EE7ELVUT365XTJ7RHKJQ) context deadline exceeded; piecestore: (Node ID: 122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, Piece ID: U7SKMZGV33HMSQLQIFIBZ6WNYC7JJ2CENSK5LCW3KVMP53T7467Q) context deadline exceeded; piecestore: (Node ID: 1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, Piece ID: FGYLRBVDMMLYWTTKLC3B2P2MHTECHEBH2DJOOKAHBR6TBAMKFBXA) context deadline exceeded ======LOGS====== Dec 8 16:15:19.690262294 (72h46m19.561345339s) [upload=63694]: Starting upload Dec 8 16:15:19.690357154 (72h46m19.56144022s) [upload=63694]: Done starting upload Dec 8 16:15:19.690425354 (72h46m19.561508409s) [upload=63694]: Uploading segments... Dec 8 16:15:19.690652296 (72h46m19.561735341s) [upload=63694]: Got next segment. Inline: false Dec 8 16:15:19.690693895 (72h46m19.56177694s) [upload=63694]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 16:15:19.690700889 (72h46m19.561783934s) [upload=63694]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 16:15:20.501121517 (72h46m20.372204572s) [seg_pos={0 0}, upload=63694]: Begin upload segment... Dec 8 16:15:20.501188935 (72h46m20.372271981s) [seg_pos={0 0}, upload=63694]: Done begin upload segment. Dec 8 16:15:20.501225655 (72h46m20.3723087s) [noise=true , address=145.79.227.16:4912 , piece_id=NIW3LT24E4LPJ5QYJVQOAR43S23JLDDOXZ6NVOURZNDTCCTLLPJA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.501311268 (72h46m20.372394313s) [noise=true , address=145.79.227.34:4906 , piece_id=AIZQB5ZLXPCLPT5Q7NLNSI3FPBPAMBVUIXKZCTA3ML7L3XP2UY6A, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.501987676 (72h46m20.373070722s) [noise=true , address=145.79.227.17:4971 , piece_id=KRCKO7NXT2XOYK4QHVJK5UIU246TVB45EE7ELVUT365XTJ7RHKJQ, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.502073379 (72h46m20.373156424s) [noise=true , address=145.79.227.38:4979 , piece_id=HCELMHS3GNLCKHFCQU3V45M4OETQZPL5MQJBUOUF5VPA44HXZW2Q, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.502559184 (72h46m20.373642229s) [noise=true , address=145.79.227.33:4870 , piece_id=IK7NDJD5GNFMWB26JYOMCSNIAT5TTTQOUXB2VFYSNRGIQ2MVMXAQ, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.502903350 (72h46m20.373986385s) [noise=true , address=145.79.227.22:4511 , piece_id=RT3ULMIM24LMC4UASFJYVAOHIKJBBGLFYJP3LBJCUL6I532MTSGA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503199825 (72h46m20.37428287s) [noise=true , address=145.79.227.29:4941 , piece_id=GJQXPA7WEZUTVJNELRBJVTU4LIGXYD7XLEPR5SWF4NBWWMXBW2SA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503205956 (72h46m20.374289001s) [noise=true , address=77.90.31.11:4555 , piece_id=LZ4YCICB4QQVLQAUZD6ZFABXLYSRKC4OFZESDVUGIOFJMZBRMMSA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503484487 (72h46m20.374567532s) [noise=true , address=145.79.227.39:4014 , piece_id=U7SKMZGV33HMSQLQIFIBZ6WNYC7JJ2CENSK5LCW3KVMP53T7467Q, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503757176 (72h46m20.374840221s) [noise=true , address=145.79.227.24:4582 , piece_id=FGYLRBVDMMLYWTTKLC3B2P2MHTECHEBH2DJOOKAHBR6TBAMKFBXA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503773727 (72h46m20.374856772s) [noise=true , address=145.79.227.40:4049 , piece_id=VX3PNKMHR75TNJLKRBLYS3KT42ZTPT4X6I5QS2AKNMGDQDVE4WDA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.504114126 (72h46m20.375197161s) [noise=true , address=77.90.31.17:4771 , piece_id=R6PLE6VLZNL23TJC6V6MRBLJP6PKB2DDGQMPPGLMM23WN6J7SH5Q, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.504441298 (72h46m20.375524343s) [noise=true , address=77.90.31.18:4807 , piece_id=7FXKUHBJT7AHDC4PHCZ73Q62GDSOUFRFIVAD337ORGE7YUTAULVA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.505240540 (72h46m20.376323585s) [noise=true , address=77.90.31.14:4662 , piece_id=BO6DQNLFNMG6HAPKOAG2DHWDD3YMX46IAYGDVQG7THC5EVZORJRA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.506022409 (72h46m20.377105454s) [noise=true , address=145.79.227.20:4438 , piece_id=U3FCNUY4ASY3ZGX7ZY26GE2NA52GJNKJMW7XKZKALT2MWBCZMHFA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.506709457 (72h46m20.377792502s) [noise=true , address=145.79.227.43:4158 , piece_id=QN7JC2XQK6FKMIYD2WXQ7GMDILEWDUO2HXZLZERVJ24WWIMHFOVA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:22.339915069 (72h46m22.210998114s) [upload=63694]: Next returned nil segment Dec 8 16:15:22.339932321 (72h46m22.211015366s) [upload=63694]: Waiting for error group managing segments... Dec 8 16:15:22.655133540 (72h46m22.526216575s) [noise=true , address=145.79.227.43:4158 , piece_id=QN7JC2XQK6FKMIYD2WXQ7GMDILEWDUO2HXZLZERVJ24WWIMHFOVA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.660462928 (72h46m22.531545983s) [noise=true , address=145.79.227.17:4971 , piece_id=KRCKO7NXT2XOYK4QHVJK5UIU246TVB45EE7ELVUT365XTJ7RHKJQ, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.683969820 (72h46m22.555052855s) [noise=true , address=77.90.31.11:4555 , piece_id=LZ4YCICB4QQVLQAUZD6ZFABXLYSRKC4OFZESDVUGIOFJMZBRMMSA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.685188641 (72h46m22.556271686s) [noise=true , address=145.79.227.22:4511 , piece_id=RT3ULMIM24LMC4UASFJYVAOHIKJBBGLFYJP3LBJCUL6I532MTSGA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.686713104 (72h46m22.557796149s) [noise=true , address=145.79.227.16:4912 , piece_id=NIW3LT24E4LPJ5QYJVQOAR43S23JLDDOXZ6NVOURZNDTCCTLLPJA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.698627942 (72h46m22.569710987s) [noise=true , address=145.79.227.39:4014 , piece_id=U7SKMZGV33HMSQLQIFIBZ6WNYC7JJ2CENSK5LCW3KVMP53T7467Q, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.713359143 (72h46m22.584442188s) [noise=true , address=145.79.227.40:4049 , piece_id=VX3PNKMHR75TNJLKRBLYS3KT42ZTPT4X6I5QS2AKNMGDQDVE4WDA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.731975212 (72h46m22.603058257s) [noise=true , address=77.90.31.17:4771 , piece_id=R6PLE6VLZNL23TJC6V6MRBLJP6PKB2DDGQMPPGLMM23WN6J7SH5Q, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.741125236 (72h46m22.612208281s) [noise=true , address=145.79.227.20:4438 , piece_id=U3FCNUY4ASY3ZGX7ZY26GE2NA52GJNKJMW7XKZKALT2MWBCZMHFA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.773771311 (72h46m22.644854346s) [noise=true , address=77.90.31.18:4807 , piece_id=7FXKUHBJT7AHDC4PHCZ73Q62GDSOUFRFIVAD337ORGE7YUTAULVA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.906364795 (72h46m22.777447841s) [noise=true , address=77.90.31.14:4662 , piece_id=BO6DQNLFNMG6HAPKOAG2DHWDD3YMX46IAYGDVQG7THC5EVZORJRA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:23.367795705 (72h46m23.23887874s) [noise=true , address=145.79.227.24:4582 , piece_id=FGYLRBVDMMLYWTTKLC3B2P2MHTECHEBH2DJOOKAHBR6TBAMKFBXA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:24.582400944 (72h46m24.453483979s) [noise=true , address=145.79.227.33:4870 , piece_id=IK7NDJD5GNFMWB26JYOMCSNIAT5TTTQOUXB2VFYSNRGIQ2MVMXAQ, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:24.582421023 (72h46m24.453504068s) [seg_pos={0 0}, upload=63694]: Segment reached optimal threshold of 13 pieces. Dec 8 16:15:24.582652363 (72h46m24.453735408s) [noise=true , address=145.79.227.38:4979 , piece_id=HCELMHS3GNLCKHFCQU3V45M4OETQZPL5MQJBUOUF5VPA44HXZW2Q, seg_pos={0 0}, upload=63694]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma): protocol: expected piece hash; context canceled; EOF Dec 8 16:15:24.582659086 (72h46m24.453742121s) [noise=true , address=145.79.227.38:4979 , piece_id=HCELMHS3GNLCKHFCQU3V45M4OETQZPL5MQJBUOUF5VPA44HXZW2Q, seg_pos={0 0}, upload=63694]: Long tail cancellation due to optimal threshold Dec 8 16:15:24.582740842 (72h46m24.453823877s) [noise=true , address=145.79.227.34:4906 , piece_id=AIZQB5ZLXPCLPT5Q7NLNSI3FPBPAMBVUIXKZCTA3ML7L3XP2UY6A, seg_pos={0 0}, upload=63694]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 8 16:15:24.582753055 (72h46m24.45383609s) [noise=true , address=145.79.227.34:4906 , piece_id=AIZQB5ZLXPCLPT5Q7NLNSI3FPBPAMBVUIXKZCTA3ML7L3XP2UY6A, seg_pos={0 0}, upload=63694]: Long tail cancellation due to optimal threshold Dec 8 16:15:24.583044900 (72h46m24.454127935s) [noise=true , address=145.79.227.29:4941 , piece_id=GJQXPA7WEZUTVJNELRBJVTU4LIGXYD7XLEPR5SWF4NBWWMXBW2SA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: context canceled; context canceled Dec 8 16:15:24.583052926 (72h46m24.454135961s) [noise=true , address=145.79.227.29:4941 , piece_id=GJQXPA7WEZUTVJNELRBJVTU4LIGXYD7XLEPR5SWF4NBWWMXBW2SA, seg_pos={0 0}, upload=63694]: Long tail cancellation due to optimal threshold Dec 8 16:15:24.583071441 (72h46m24.454154486s) [seg_pos={0 0}, upload=63694]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 16:15:24.583162014 (72h46m24.454245059s) [upload=63694]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 16:15:24.583171562 (72h46m24.454254627s) [upload=63694]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 16:15:25.042230695 (72h46m24.91331374s) [upload=63694]: Done uploading segments... Dec 8 16:15:25.042258869 (72h46m24.913341914s) [upload=63694]: Upload finished. err: <nil> Duration: 5.941010299s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 1.4058553s trace: https://jaeger.dev.storj.io/trace/3bdb8560e64b821d FAIL (2025-12-08 16:36:47 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 1.405885568s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 20 out of 60 is failed (threshold=6) [XX.....X.....X...X..XX..X......X.X.XX...X.X.X.X.....X....XXX] duration: 5.0017018s trace: https://jaeger.dev.storj.io/trace/37d50eeb17d3f658 FAIL (2025-12-08 16:36:41 UTC) orbiter warning: context deadline exceeded Duration: 5.001726758s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 2.082764577s trace: https://jaeger.dev.storj.io/trace/4dad09bb52e33232 FAIL (2025-12-08 16:36:35 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 2.082798382s core uplink us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 5.016269258s trace: https://jaeger.dev.storj.io/trace/5d817b557064ae10 FAIL (2025-12-08 16:36:35 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.016300838s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 1.578247029s trace: https://jaeger.dev.storj.io/trace/3bdb8560e64b821d FAIL (2025-12-08 16:36:20 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 1.578276424s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...............X.X..........................................] duration: 15.110221853s trace: https://jaeger.dev.storj.io/trace/690f79ef204fee2 FAIL (2025-12-08 16:36:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 15.110386616s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...X............................XX..........................] duration: 5.799851794s trace: https://jaeger.dev.storj.io/trace/13568c7eb15b2369 FAIL (2025-12-08 16:36:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, Piece ID: U3FCNUY4ASY3ZGX7ZY26GE2NA52GJNKJMW7XKZKALT2MWBCZMHFA) context deadline exceeded; piecestore: (Node ID: 1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, Piece ID: QN7JC2XQK6FKMIYD2WXQ7GMDILEWDUO2HXZLZERVJ24WWIMHFOVA) context deadline exceeded; piecestore: (Node ID: 12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, Piece ID: KRCKO7NXT2XOYK4QHVJK5UIU246TVB45EE7ELVUT365XTJ7RHKJQ) context deadline exceeded; piecestore: (Node ID: 122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, Piece ID: U7SKMZGV33HMSQLQIFIBZ6WNYC7JJ2CENSK5LCW3KVMP53T7467Q) context deadline exceeded; piecestore: (Node ID: 1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, Piece ID: FGYLRBVDMMLYWTTKLC3B2P2MHTECHEBH2DJOOKAHBR6TBAMKFBXA) context deadline exceeded ======LOGS====== Dec 8 16:15:19.690262294 (72h46m19.561345339s) [upload=63694]: Starting upload Dec 8 16:15:19.690357154 (72h46m19.56144022s) [upload=63694]: Done starting upload Dec 8 16:15:19.690425354 (72h46m19.561508409s) [upload=63694]: Uploading segments... Dec 8 16:15:19.690652296 (72h46m19.561735341s) [upload=63694]: Got next segment. Inline: false Dec 8 16:15:19.690693895 (72h46m19.56177694s) [upload=63694]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 16:15:19.690700889 (72h46m19.561783934s) [upload=63694]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 16:15:20.501121517 (72h46m20.372204572s) [seg_pos={0 0}, upload=63694]: Begin upload segment... Dec 8 16:15:20.501188935 (72h46m20.372271981s) [seg_pos={0 0}, upload=63694]: Done begin upload segment. Dec 8 16:15:20.501225655 (72h46m20.3723087s) [noise=true , address=145.79.227.16:4912 , piece_id=NIW3LT24E4LPJ5QYJVQOAR43S23JLDDOXZ6NVOURZNDTCCTLLPJA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.501311268 (72h46m20.372394313s) [noise=true , address=145.79.227.34:4906 , piece_id=AIZQB5ZLXPCLPT5Q7NLNSI3FPBPAMBVUIXKZCTA3ML7L3XP2UY6A, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.501987676 (72h46m20.373070722s) [noise=true , address=145.79.227.17:4971 , piece_id=KRCKO7NXT2XOYK4QHVJK5UIU246TVB45EE7ELVUT365XTJ7RHKJQ, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.502073379 (72h46m20.373156424s) [noise=true , address=145.79.227.38:4979 , piece_id=HCELMHS3GNLCKHFCQU3V45M4OETQZPL5MQJBUOUF5VPA44HXZW2Q, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.502559184 (72h46m20.373642229s) [noise=true , address=145.79.227.33:4870 , piece_id=IK7NDJD5GNFMWB26JYOMCSNIAT5TTTQOUXB2VFYSNRGIQ2MVMXAQ, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.502903350 (72h46m20.373986385s) [noise=true , address=145.79.227.22:4511 , piece_id=RT3ULMIM24LMC4UASFJYVAOHIKJBBGLFYJP3LBJCUL6I532MTSGA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503199825 (72h46m20.37428287s) [noise=true , address=145.79.227.29:4941 , piece_id=GJQXPA7WEZUTVJNELRBJVTU4LIGXYD7XLEPR5SWF4NBWWMXBW2SA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503205956 (72h46m20.374289001s) [noise=true , address=77.90.31.11:4555 , piece_id=LZ4YCICB4QQVLQAUZD6ZFABXLYSRKC4OFZESDVUGIOFJMZBRMMSA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503484487 (72h46m20.374567532s) [noise=true , address=145.79.227.39:4014 , piece_id=U7SKMZGV33HMSQLQIFIBZ6WNYC7JJ2CENSK5LCW3KVMP53T7467Q, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503757176 (72h46m20.374840221s) [noise=true , address=145.79.227.24:4582 , piece_id=FGYLRBVDMMLYWTTKLC3B2P2MHTECHEBH2DJOOKAHBR6TBAMKFBXA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503773727 (72h46m20.374856772s) [noise=true , address=145.79.227.40:4049 , piece_id=VX3PNKMHR75TNJLKRBLYS3KT42ZTPT4X6I5QS2AKNMGDQDVE4WDA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.504114126 (72h46m20.375197161s) [noise=true , address=77.90.31.17:4771 , piece_id=R6PLE6VLZNL23TJC6V6MRBLJP6PKB2DDGQMPPGLMM23WN6J7SH5Q, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.504441298 (72h46m20.375524343s) [noise=true , address=77.90.31.18:4807 , piece_id=7FXKUHBJT7AHDC4PHCZ73Q62GDSOUFRFIVAD337ORGE7YUTAULVA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.505240540 (72h46m20.376323585s) [noise=true , address=77.90.31.14:4662 , piece_id=BO6DQNLFNMG6HAPKOAG2DHWDD3YMX46IAYGDVQG7THC5EVZORJRA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.506022409 (72h46m20.377105454s) [noise=true , address=145.79.227.20:4438 , piece_id=U3FCNUY4ASY3ZGX7ZY26GE2NA52GJNKJMW7XKZKALT2MWBCZMHFA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.506709457 (72h46m20.377792502s) [noise=true , address=145.79.227.43:4158 , piece_id=QN7JC2XQK6FKMIYD2WXQ7GMDILEWDUO2HXZLZERVJ24WWIMHFOVA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:22.339915069 (72h46m22.210998114s) [upload=63694]: Next returned nil segment Dec 8 16:15:22.339932321 (72h46m22.211015366s) [upload=63694]: Waiting for error group managing segments... Dec 8 16:15:22.655133540 (72h46m22.526216575s) [noise=true , address=145.79.227.43:4158 , piece_id=QN7JC2XQK6FKMIYD2WXQ7GMDILEWDUO2HXZLZERVJ24WWIMHFOVA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.660462928 (72h46m22.531545983s) [noise=true , address=145.79.227.17:4971 , piece_id=KRCKO7NXT2XOYK4QHVJK5UIU246TVB45EE7ELVUT365XTJ7RHKJQ, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.683969820 (72h46m22.555052855s) [noise=true , address=77.90.31.11:4555 , piece_id=LZ4YCICB4QQVLQAUZD6ZFABXLYSRKC4OFZESDVUGIOFJMZBRMMSA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.685188641 (72h46m22.556271686s) [noise=true , address=145.79.227.22:4511 , piece_id=RT3ULMIM24LMC4UASFJYVAOHIKJBBGLFYJP3LBJCUL6I532MTSGA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.686713104 (72h46m22.557796149s) [noise=true , address=145.79.227.16:4912 , piece_id=NIW3LT24E4LPJ5QYJVQOAR43S23JLDDOXZ6NVOURZNDTCCTLLPJA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.698627942 (72h46m22.569710987s) [noise=true , address=145.79.227.39:4014 , piece_id=U7SKMZGV33HMSQLQIFIBZ6WNYC7JJ2CENSK5LCW3KVMP53T7467Q, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.713359143 (72h46m22.584442188s) [noise=true , address=145.79.227.40:4049 , piece_id=VX3PNKMHR75TNJLKRBLYS3KT42ZTPT4X6I5QS2AKNMGDQDVE4WDA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.731975212 (72h46m22.603058257s) [noise=true , address=77.90.31.17:4771 , piece_id=R6PLE6VLZNL23TJC6V6MRBLJP6PKB2DDGQMPPGLMM23WN6J7SH5Q, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.741125236 (72h46m22.612208281s) [noise=true , address=145.79.227.20:4438 , piece_id=U3FCNUY4ASY3ZGX7ZY26GE2NA52GJNKJMW7XKZKALT2MWBCZMHFA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.773771311 (72h46m22.644854346s) [noise=true , address=77.90.31.18:4807 , piece_id=7FXKUHBJT7AHDC4PHCZ73Q62GDSOUFRFIVAD337ORGE7YUTAULVA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.906364795 (72h46m22.777447841s) [noise=true , address=77.90.31.14:4662 , piece_id=BO6DQNLFNMG6HAPKOAG2DHWDD3YMX46IAYGDVQG7THC5EVZORJRA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:23.367795705 (72h46m23.23887874s) [noise=true , address=145.79.227.24:4582 , piece_id=FGYLRBVDMMLYWTTKLC3B2P2MHTECHEBH2DJOOKAHBR6TBAMKFBXA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:24.582400944 (72h46m24.453483979s) [noise=true , address=145.79.227.33:4870 , piece_id=IK7NDJD5GNFMWB26JYOMCSNIAT5TTTQOUXB2VFYSNRGIQ2MVMXAQ, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:24.582421023 (72h46m24.453504068s) [seg_pos={0 0}, upload=63694]: Segment reached optimal threshold of 13 pieces. Dec 8 16:15:24.582652363 (72h46m24.453735408s) [noise=true , address=145.79.227.38:4979 , piece_id=HCELMHS3GNLCKHFCQU3V45M4OETQZPL5MQJBUOUF5VPA44HXZW2Q, seg_pos={0 0}, upload=63694]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma): protocol: expected piece hash; context canceled; EOF Dec 8 16:15:24.582659086 (72h46m24.453742121s) [noise=true , address=145.79.227.38:4979 , piece_id=HCELMHS3GNLCKHFCQU3V45M4OETQZPL5MQJBUOUF5VPA44HXZW2Q, seg_pos={0 0}, upload=63694]: Long tail cancellation due to optimal threshold Dec 8 16:15:24.582740842 (72h46m24.453823877s) [noise=true , address=145.79.227.34:4906 , piece_id=AIZQB5ZLXPCLPT5Q7NLNSI3FPBPAMBVUIXKZCTA3ML7L3XP2UY6A, seg_pos={0 0}, upload=63694]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 8 16:15:24.582753055 (72h46m24.45383609s) [noise=true , address=145.79.227.34:4906 , piece_id=AIZQB5ZLXPCLPT5Q7NLNSI3FPBPAMBVUIXKZCTA3ML7L3XP2UY6A, seg_pos={0 0}, upload=63694]: Long tail cancellation due to optimal threshold Dec 8 16:15:24.583044900 (72h46m24.454127935s) [noise=true , address=145.79.227.29:4941 , piece_id=GJQXPA7WEZUTVJNELRBJVTU4LIGXYD7XLEPR5SWF4NBWWMXBW2SA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: context canceled; context canceled Dec 8 16:15:24.583052926 (72h46m24.454135961s) [noise=true , address=145.79.227.29:4941 , piece_id=GJQXPA7WEZUTVJNELRBJVTU4LIGXYD7XLEPR5SWF4NBWWMXBW2SA, seg_pos={0 0}, upload=63694]: Long tail cancellation due to optimal threshold Dec 8 16:15:24.583071441 (72h46m24.454154486s) [seg_pos={0 0}, upload=63694]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 16:15:24.583162014 (72h46m24.454245059s) [upload=63694]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 16:15:24.583171562 (72h46m24.454254627s) [upload=63694]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 16:15:25.042230695 (72h46m24.91331374s) [upload=63694]: Done uploading segments... Dec 8 16:15:25.042258869 (72h46m24.913341914s) [upload=63694]: Upload finished. err: <nil> Duration: 5.799879307s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 6.046933358s trace: https://jaeger.dev.storj.io/trace/1e9e467d23cfcdb3 FAIL (2025-12-08 16:36:10 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.046977141s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [X....X..........................X...X.X.....................] duration: 5.951893712s trace: https://jaeger.dev.storj.io/trace/20c661c94f160c60 FAIL (2025-12-08 16:36:08 UTC) orbiter warning: context deadline exceeded Duration: 5.951920203s edge gateway us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 4.881358009s trace: https://jaeger.dev.storj.io/trace/5d817b557064ae10 FAIL (2025-12-08 16:36:07 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.881380582s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 2.135768926s trace: https://jaeger.dev.storj.io/trace/4dad09bb52e33232 FAIL (2025-12-08 16:35:52 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 2.135797881s core uplink us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 1.600473433s trace: https://jaeger.dev.storj.io/trace/3bdb8560e64b821d FAIL (2025-12-08 16:35:51 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 1.600502218s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 19 out of 60 is failed (threshold=6) [.XX.....X.....X...X..XX..X......X.X.XX...X.X.X.X.....X....XX] duration: 5.001668099s trace: https://jaeger.dev.storj.io/trace/177a046795e058b8 FAIL (2025-12-08 16:35:31 UTC) orbiter warning: context deadline exceeded Duration: 5.001691524s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.X....X..........................X...X.X....................] duration: 16.562999014s trace: https://jaeger.dev.storj.io/trace/20c661c94f160c60 FAIL (2025-12-08 16:35:19 UTC) orbiter warning: context deadline exceeded Duration: 16.563029813s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 2.018140292s trace: https://jaeger.dev.storj.io/trace/4dad09bb52e33232 FAIL (2025-12-08 16:35:14 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 2.018169316s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....X............................XX.........................] duration: 5.819294116s trace: https://jaeger.dev.storj.io/trace/13568c7eb15b2369 FAIL (2025-12-08 16:35:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, Piece ID: U3FCNUY4ASY3ZGX7ZY26GE2NA52GJNKJMW7XKZKALT2MWBCZMHFA) context deadline exceeded; piecestore: (Node ID: 1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, Piece ID: QN7JC2XQK6FKMIYD2WXQ7GMDILEWDUO2HXZLZERVJ24WWIMHFOVA) context deadline exceeded; piecestore: (Node ID: 12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, Piece ID: KRCKO7NXT2XOYK4QHVJK5UIU246TVB45EE7ELVUT365XTJ7RHKJQ) context deadline exceeded; piecestore: (Node ID: 122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, Piece ID: U7SKMZGV33HMSQLQIFIBZ6WNYC7JJ2CENSK5LCW3KVMP53T7467Q) context deadline exceeded; piecestore: (Node ID: 1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, Piece ID: FGYLRBVDMMLYWTTKLC3B2P2MHTECHEBH2DJOOKAHBR6TBAMKFBXA) context deadline exceeded ======LOGS====== Dec 8 16:15:19.690262294 (72h46m19.561345339s) [upload=63694]: Starting upload Dec 8 16:15:19.690357154 (72h46m19.56144022s) [upload=63694]: Done starting upload Dec 8 16:15:19.690425354 (72h46m19.561508409s) [upload=63694]: Uploading segments... Dec 8 16:15:19.690652296 (72h46m19.561735341s) [upload=63694]: Got next segment. Inline: false Dec 8 16:15:19.690693895 (72h46m19.56177694s) [upload=63694]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 16:15:19.690700889 (72h46m19.561783934s) [upload=63694]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 16:15:20.501121517 (72h46m20.372204572s) [seg_pos={0 0}, upload=63694]: Begin upload segment... Dec 8 16:15:20.501188935 (72h46m20.372271981s) [seg_pos={0 0}, upload=63694]: Done begin upload segment. Dec 8 16:15:20.501225655 (72h46m20.3723087s) [noise=true , address=145.79.227.16:4912 , piece_id=NIW3LT24E4LPJ5QYJVQOAR43S23JLDDOXZ6NVOURZNDTCCTLLPJA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.501311268 (72h46m20.372394313s) [noise=true , address=145.79.227.34:4906 , piece_id=AIZQB5ZLXPCLPT5Q7NLNSI3FPBPAMBVUIXKZCTA3ML7L3XP2UY6A, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.501987676 (72h46m20.373070722s) [noise=true , address=145.79.227.17:4971 , piece_id=KRCKO7NXT2XOYK4QHVJK5UIU246TVB45EE7ELVUT365XTJ7RHKJQ, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.502073379 (72h46m20.373156424s) [noise=true , address=145.79.227.38:4979 , piece_id=HCELMHS3GNLCKHFCQU3V45M4OETQZPL5MQJBUOUF5VPA44HXZW2Q, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.502559184 (72h46m20.373642229s) [noise=true , address=145.79.227.33:4870 , piece_id=IK7NDJD5GNFMWB26JYOMCSNIAT5TTTQOUXB2VFYSNRGIQ2MVMXAQ, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.502903350 (72h46m20.373986385s) [noise=true , address=145.79.227.22:4511 , piece_id=RT3ULMIM24LMC4UASFJYVAOHIKJBBGLFYJP3LBJCUL6I532MTSGA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503199825 (72h46m20.37428287s) [noise=true , address=145.79.227.29:4941 , piece_id=GJQXPA7WEZUTVJNELRBJVTU4LIGXYD7XLEPR5SWF4NBWWMXBW2SA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503205956 (72h46m20.374289001s) [noise=true , address=77.90.31.11:4555 , piece_id=LZ4YCICB4QQVLQAUZD6ZFABXLYSRKC4OFZESDVUGIOFJMZBRMMSA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503484487 (72h46m20.374567532s) [noise=true , address=145.79.227.39:4014 , piece_id=U7SKMZGV33HMSQLQIFIBZ6WNYC7JJ2CENSK5LCW3KVMP53T7467Q, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503757176 (72h46m20.374840221s) [noise=true , address=145.79.227.24:4582 , piece_id=FGYLRBVDMMLYWTTKLC3B2P2MHTECHEBH2DJOOKAHBR6TBAMKFBXA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503773727 (72h46m20.374856772s) [noise=true , address=145.79.227.40:4049 , piece_id=VX3PNKMHR75TNJLKRBLYS3KT42ZTPT4X6I5QS2AKNMGDQDVE4WDA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.504114126 (72h46m20.375197161s) [noise=true , address=77.90.31.17:4771 , piece_id=R6PLE6VLZNL23TJC6V6MRBLJP6PKB2DDGQMPPGLMM23WN6J7SH5Q, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.504441298 (72h46m20.375524343s) [noise=true , address=77.90.31.18:4807 , piece_id=7FXKUHBJT7AHDC4PHCZ73Q62GDSOUFRFIVAD337ORGE7YUTAULVA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.505240540 (72h46m20.376323585s) [noise=true , address=77.90.31.14:4662 , piece_id=BO6DQNLFNMG6HAPKOAG2DHWDD3YMX46IAYGDVQG7THC5EVZORJRA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.506022409 (72h46m20.377105454s) [noise=true , address=145.79.227.20:4438 , piece_id=U3FCNUY4ASY3ZGX7ZY26GE2NA52GJNKJMW7XKZKALT2MWBCZMHFA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.506709457 (72h46m20.377792502s) [noise=true , address=145.79.227.43:4158 , piece_id=QN7JC2XQK6FKMIYD2WXQ7GMDILEWDUO2HXZLZERVJ24WWIMHFOVA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:22.339915069 (72h46m22.210998114s) [upload=63694]: Next returned nil segment Dec 8 16:15:22.339932321 (72h46m22.211015366s) [upload=63694]: Waiting for error group managing segments... Dec 8 16:15:22.655133540 (72h46m22.526216575s) [noise=true , address=145.79.227.43:4158 , piece_id=QN7JC2XQK6FKMIYD2WXQ7GMDILEWDUO2HXZLZERVJ24WWIMHFOVA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.660462928 (72h46m22.531545983s) [noise=true , address=145.79.227.17:4971 , piece_id=KRCKO7NXT2XOYK4QHVJK5UIU246TVB45EE7ELVUT365XTJ7RHKJQ, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.683969820 (72h46m22.555052855s) [noise=true , address=77.90.31.11:4555 , piece_id=LZ4YCICB4QQVLQAUZD6ZFABXLYSRKC4OFZESDVUGIOFJMZBRMMSA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.685188641 (72h46m22.556271686s) [noise=true , address=145.79.227.22:4511 , piece_id=RT3ULMIM24LMC4UASFJYVAOHIKJBBGLFYJP3LBJCUL6I532MTSGA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.686713104 (72h46m22.557796149s) [noise=true , address=145.79.227.16:4912 , piece_id=NIW3LT24E4LPJ5QYJVQOAR43S23JLDDOXZ6NVOURZNDTCCTLLPJA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.698627942 (72h46m22.569710987s) [noise=true , address=145.79.227.39:4014 , piece_id=U7SKMZGV33HMSQLQIFIBZ6WNYC7JJ2CENSK5LCW3KVMP53T7467Q, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.713359143 (72h46m22.584442188s) [noise=true , address=145.79.227.40:4049 , piece_id=VX3PNKMHR75TNJLKRBLYS3KT42ZTPT4X6I5QS2AKNMGDQDVE4WDA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.731975212 (72h46m22.603058257s) [noise=true , address=77.90.31.17:4771 , piece_id=R6PLE6VLZNL23TJC6V6MRBLJP6PKB2DDGQMPPGLMM23WN6J7SH5Q, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.741125236 (72h46m22.612208281s) [noise=true , address=145.79.227.20:4438 , piece_id=U3FCNUY4ASY3ZGX7ZY26GE2NA52GJNKJMW7XKZKALT2MWBCZMHFA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.773771311 (72h46m22.644854346s) [noise=true , address=77.90.31.18:4807 , piece_id=7FXKUHBJT7AHDC4PHCZ73Q62GDSOUFRFIVAD337ORGE7YUTAULVA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.906364795 (72h46m22.777447841s) [noise=true , address=77.90.31.14:4662 , piece_id=BO6DQNLFNMG6HAPKOAG2DHWDD3YMX46IAYGDVQG7THC5EVZORJRA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:23.367795705 (72h46m23.23887874s) [noise=true , address=145.79.227.24:4582 , piece_id=FGYLRBVDMMLYWTTKLC3B2P2MHTECHEBH2DJOOKAHBR6TBAMKFBXA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:24.582400944 (72h46m24.453483979s) [noise=true , address=145.79.227.33:4870 , piece_id=IK7NDJD5GNFMWB26JYOMCSNIAT5TTTQOUXB2VFYSNRGIQ2MVMXAQ, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:24.582421023 (72h46m24.453504068s) [seg_pos={0 0}, upload=63694]: Segment reached optimal threshold of 13 pieces. Dec 8 16:15:24.582652363 (72h46m24.453735408s) [noise=true , address=145.79.227.38:4979 , piece_id=HCELMHS3GNLCKHFCQU3V45M4OETQZPL5MQJBUOUF5VPA44HXZW2Q, seg_pos={0 0}, upload=63694]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma): protocol: expected piece hash; context canceled; EOF Dec 8 16:15:24.582659086 (72h46m24.453742121s) [noise=true , address=145.79.227.38:4979 , piece_id=HCELMHS3GNLCKHFCQU3V45M4OETQZPL5MQJBUOUF5VPA44HXZW2Q, seg_pos={0 0}, upload=63694]: Long tail cancellation due to optimal threshold Dec 8 16:15:24.582740842 (72h46m24.453823877s) [noise=true , address=145.79.227.34:4906 , piece_id=AIZQB5ZLXPCLPT5Q7NLNSI3FPBPAMBVUIXKZCTA3ML7L3XP2UY6A, seg_pos={0 0}, upload=63694]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 8 16:15:24.582753055 (72h46m24.45383609s) [noise=true , address=145.79.227.34:4906 , piece_id=AIZQB5ZLXPCLPT5Q7NLNSI3FPBPAMBVUIXKZCTA3ML7L3XP2UY6A, seg_pos={0 0}, upload=63694]: Long tail cancellation due to optimal threshold Dec 8 16:15:24.583044900 (72h46m24.454127935s) [noise=true , address=145.79.227.29:4941 , piece_id=GJQXPA7WEZUTVJNELRBJVTU4LIGXYD7XLEPR5SWF4NBWWMXBW2SA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: context canceled; context canceled Dec 8 16:15:24.583052926 (72h46m24.454135961s) [noise=true , address=145.79.227.29:4941 , piece_id=GJQXPA7WEZUTVJNELRBJVTU4LIGXYD7XLEPR5SWF4NBWWMXBW2SA, seg_pos={0 0}, upload=63694]: Long tail cancellation due to optimal threshold Dec 8 16:15:24.583071441 (72h46m24.454154486s) [seg_pos={0 0}, upload=63694]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 16:15:24.583162014 (72h46m24.454245059s) [upload=63694]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 16:15:24.583171562 (72h46m24.454254627s) [upload=63694]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 16:15:25.042230695 (72h46m24.91331374s) [upload=63694]: Done uploading segments... Dec 8 16:15:25.042258869 (72h46m24.913341914s) [upload=63694]: Upload finished. err: <nil> Duration: 5.819323822s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 5.216556316s trace: https://jaeger.dev.storj.io/trace/5d817b557064ae10 FAIL (2025-12-08 16:35:05 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.216587856s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [................X.X.........................................] duration: 11.965589449s trace: https://jaeger.dev.storj.io/trace/690f79ef204fee2 FAIL (2025-12-08 16:35:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 11.965624046s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 5.940381543s trace: https://jaeger.dev.storj.io/trace/1e9e467d23cfcdb3 FAIL (2025-12-08 16:34:56 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.94041125s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 2.016201174s trace: https://jaeger.dev.storj.io/trace/4dad09bb52e33232 FAIL (2025-12-08 16:34:45 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 2.026714812s core uplink us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 1.403390687s trace: https://jaeger.dev.storj.io/trace/3bdb8560e64b821d FAIL (2025-12-08 16:34:41 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 1.403416366s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....X............................XX........................] duration: 5.7963183s trace: https://jaeger.dev.storj.io/trace/13568c7eb15b2369 FAIL (2025-12-08 16:34:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, Piece ID: U3FCNUY4ASY3ZGX7ZY26GE2NA52GJNKJMW7XKZKALT2MWBCZMHFA) context deadline exceeded; piecestore: (Node ID: 1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, Piece ID: QN7JC2XQK6FKMIYD2WXQ7GMDILEWDUO2HXZLZERVJ24WWIMHFOVA) context deadline exceeded; piecestore: (Node ID: 12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, Piece ID: KRCKO7NXT2XOYK4QHVJK5UIU246TVB45EE7ELVUT365XTJ7RHKJQ) context deadline exceeded; piecestore: (Node ID: 122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, Piece ID: U7SKMZGV33HMSQLQIFIBZ6WNYC7JJ2CENSK5LCW3KVMP53T7467Q) context deadline exceeded; piecestore: (Node ID: 1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, Piece ID: FGYLRBVDMMLYWTTKLC3B2P2MHTECHEBH2DJOOKAHBR6TBAMKFBXA) context deadline exceeded ======LOGS====== Dec 8 16:15:19.690262294 (72h46m19.561345339s) [upload=63694]: Starting upload Dec 8 16:15:19.690357154 (72h46m19.56144022s) [upload=63694]: Done starting upload Dec 8 16:15:19.690425354 (72h46m19.561508409s) [upload=63694]: Uploading segments... Dec 8 16:15:19.690652296 (72h46m19.561735341s) [upload=63694]: Got next segment. Inline: false Dec 8 16:15:19.690693895 (72h46m19.56177694s) [upload=63694]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 16:15:19.690700889 (72h46m19.561783934s) [upload=63694]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 16:15:20.501121517 (72h46m20.372204572s) [seg_pos={0 0}, upload=63694]: Begin upload segment... Dec 8 16:15:20.501188935 (72h46m20.372271981s) [seg_pos={0 0}, upload=63694]: Done begin upload segment. Dec 8 16:15:20.501225655 (72h46m20.3723087s) [noise=true , address=145.79.227.16:4912 , piece_id=NIW3LT24E4LPJ5QYJVQOAR43S23JLDDOXZ6NVOURZNDTCCTLLPJA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.501311268 (72h46m20.372394313s) [noise=true , address=145.79.227.34:4906 , piece_id=AIZQB5ZLXPCLPT5Q7NLNSI3FPBPAMBVUIXKZCTA3ML7L3XP2UY6A, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.501987676 (72h46m20.373070722s) [noise=true , address=145.79.227.17:4971 , piece_id=KRCKO7NXT2XOYK4QHVJK5UIU246TVB45EE7ELVUT365XTJ7RHKJQ, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.502073379 (72h46m20.373156424s) [noise=true , address=145.79.227.38:4979 , piece_id=HCELMHS3GNLCKHFCQU3V45M4OETQZPL5MQJBUOUF5VPA44HXZW2Q, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.502559184 (72h46m20.373642229s) [noise=true , address=145.79.227.33:4870 , piece_id=IK7NDJD5GNFMWB26JYOMCSNIAT5TTTQOUXB2VFYSNRGIQ2MVMXAQ, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.502903350 (72h46m20.373986385s) [noise=true , address=145.79.227.22:4511 , piece_id=RT3ULMIM24LMC4UASFJYVAOHIKJBBGLFYJP3LBJCUL6I532MTSGA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503199825 (72h46m20.37428287s) [noise=true , address=145.79.227.29:4941 , piece_id=GJQXPA7WEZUTVJNELRBJVTU4LIGXYD7XLEPR5SWF4NBWWMXBW2SA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503205956 (72h46m20.374289001s) [noise=true , address=77.90.31.11:4555 , piece_id=LZ4YCICB4QQVLQAUZD6ZFABXLYSRKC4OFZESDVUGIOFJMZBRMMSA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503484487 (72h46m20.374567532s) [noise=true , address=145.79.227.39:4014 , piece_id=U7SKMZGV33HMSQLQIFIBZ6WNYC7JJ2CENSK5LCW3KVMP53T7467Q, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503757176 (72h46m20.374840221s) [noise=true , address=145.79.227.24:4582 , piece_id=FGYLRBVDMMLYWTTKLC3B2P2MHTECHEBH2DJOOKAHBR6TBAMKFBXA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503773727 (72h46m20.374856772s) [noise=true , address=145.79.227.40:4049 , piece_id=VX3PNKMHR75TNJLKRBLYS3KT42ZTPT4X6I5QS2AKNMGDQDVE4WDA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.504114126 (72h46m20.375197161s) [noise=true , address=77.90.31.17:4771 , piece_id=R6PLE6VLZNL23TJC6V6MRBLJP6PKB2DDGQMPPGLMM23WN6J7SH5Q, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.504441298 (72h46m20.375524343s) [noise=true , address=77.90.31.18:4807 , piece_id=7FXKUHBJT7AHDC4PHCZ73Q62GDSOUFRFIVAD337ORGE7YUTAULVA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.505240540 (72h46m20.376323585s) [noise=true , address=77.90.31.14:4662 , piece_id=BO6DQNLFNMG6HAPKOAG2DHWDD3YMX46IAYGDVQG7THC5EVZORJRA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.506022409 (72h46m20.377105454s) [noise=true , address=145.79.227.20:4438 , piece_id=U3FCNUY4ASY3ZGX7ZY26GE2NA52GJNKJMW7XKZKALT2MWBCZMHFA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.506709457 (72h46m20.377792502s) [noise=true , address=145.79.227.43:4158 , piece_id=QN7JC2XQK6FKMIYD2WXQ7GMDILEWDUO2HXZLZERVJ24WWIMHFOVA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:22.339915069 (72h46m22.210998114s) [upload=63694]: Next returned nil segment Dec 8 16:15:22.339932321 (72h46m22.211015366s) [upload=63694]: Waiting for error group managing segments... Dec 8 16:15:22.655133540 (72h46m22.526216575s) [noise=true , address=145.79.227.43:4158 , piece_id=QN7JC2XQK6FKMIYD2WXQ7GMDILEWDUO2HXZLZERVJ24WWIMHFOVA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.660462928 (72h46m22.531545983s) [noise=true , address=145.79.227.17:4971 , piece_id=KRCKO7NXT2XOYK4QHVJK5UIU246TVB45EE7ELVUT365XTJ7RHKJQ, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.683969820 (72h46m22.555052855s) [noise=true , address=77.90.31.11:4555 , piece_id=LZ4YCICB4QQVLQAUZD6ZFABXLYSRKC4OFZESDVUGIOFJMZBRMMSA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.685188641 (72h46m22.556271686s) [noise=true , address=145.79.227.22:4511 , piece_id=RT3ULMIM24LMC4UASFJYVAOHIKJBBGLFYJP3LBJCUL6I532MTSGA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.686713104 (72h46m22.557796149s) [noise=true , address=145.79.227.16:4912 , piece_id=NIW3LT24E4LPJ5QYJVQOAR43S23JLDDOXZ6NVOURZNDTCCTLLPJA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.698627942 (72h46m22.569710987s) [noise=true , address=145.79.227.39:4014 , piece_id=U7SKMZGV33HMSQLQIFIBZ6WNYC7JJ2CENSK5LCW3KVMP53T7467Q, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.713359143 (72h46m22.584442188s) [noise=true , address=145.79.227.40:4049 , piece_id=VX3PNKMHR75TNJLKRBLYS3KT42ZTPT4X6I5QS2AKNMGDQDVE4WDA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.731975212 (72h46m22.603058257s) [noise=true , address=77.90.31.17:4771 , piece_id=R6PLE6VLZNL23TJC6V6MRBLJP6PKB2DDGQMPPGLMM23WN6J7SH5Q, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.741125236 (72h46m22.612208281s) [noise=true , address=145.79.227.20:4438 , piece_id=U3FCNUY4ASY3ZGX7ZY26GE2NA52GJNKJMW7XKZKALT2MWBCZMHFA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.773771311 (72h46m22.644854346s) [noise=true , address=77.90.31.18:4807 , piece_id=7FXKUHBJT7AHDC4PHCZ73Q62GDSOUFRFIVAD337ORGE7YUTAULVA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.906364795 (72h46m22.777447841s) [noise=true , address=77.90.31.14:4662 , piece_id=BO6DQNLFNMG6HAPKOAG2DHWDD3YMX46IAYGDVQG7THC5EVZORJRA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:23.367795705 (72h46m23.23887874s) [noise=true , address=145.79.227.24:4582 , piece_id=FGYLRBVDMMLYWTTKLC3B2P2MHTECHEBH2DJOOKAHBR6TBAMKFBXA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:24.582400944 (72h46m24.453483979s) [noise=true , address=145.79.227.33:4870 , piece_id=IK7NDJD5GNFMWB26JYOMCSNIAT5TTTQOUXB2VFYSNRGIQ2MVMXAQ, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:24.582421023 (72h46m24.453504068s) [seg_pos={0 0}, upload=63694]: Segment reached optimal threshold of 13 pieces. Dec 8 16:15:24.582652363 (72h46m24.453735408s) [noise=true , address=145.79.227.38:4979 , piece_id=HCELMHS3GNLCKHFCQU3V45M4OETQZPL5MQJBUOUF5VPA44HXZW2Q, seg_pos={0 0}, upload=63694]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma): protocol: expected piece hash; context canceled; EOF Dec 8 16:15:24.582659086 (72h46m24.453742121s) [noise=true , address=145.79.227.38:4979 , piece_id=HCELMHS3GNLCKHFCQU3V45M4OETQZPL5MQJBUOUF5VPA44HXZW2Q, seg_pos={0 0}, upload=63694]: Long tail cancellation due to optimal threshold Dec 8 16:15:24.582740842 (72h46m24.453823877s) [noise=true , address=145.79.227.34:4906 , piece_id=AIZQB5ZLXPCLPT5Q7NLNSI3FPBPAMBVUIXKZCTA3ML7L3XP2UY6A, seg_pos={0 0}, upload=63694]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 8 16:15:24.582753055 (72h46m24.45383609s) [noise=true , address=145.79.227.34:4906 , piece_id=AIZQB5ZLXPCLPT5Q7NLNSI3FPBPAMBVUIXKZCTA3ML7L3XP2UY6A, seg_pos={0 0}, upload=63694]: Long tail cancellation due to optimal threshold Dec 8 16:15:24.583044900 (72h46m24.454127935s) [noise=true , address=145.79.227.29:4941 , piece_id=GJQXPA7WEZUTVJNELRBJVTU4LIGXYD7XLEPR5SWF4NBWWMXBW2SA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: context canceled; context canceled Dec 8 16:15:24.583052926 (72h46m24.454135961s) [noise=true , address=145.79.227.29:4941 , piece_id=GJQXPA7WEZUTVJNELRBJVTU4LIGXYD7XLEPR5SWF4NBWWMXBW2SA, seg_pos={0 0}, upload=63694]: Long tail cancellation due to optimal threshold Dec 8 16:15:24.583071441 (72h46m24.454154486s) [seg_pos={0 0}, upload=63694]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 16:15:24.583162014 (72h46m24.454245059s) [upload=63694]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 16:15:24.583171562 (72h46m24.454254627s) [upload=63694]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 16:15:25.042230695 (72h46m24.91331374s) [upload=63694]: Done uploading segments... Dec 8 16:15:25.042258869 (72h46m24.913341914s) [upload=63694]: Upload finished. err: <nil> Duration: 5.796345391s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 18 out of 60 is failed (threshold=6) [..XX.....X.....X...X..XX..X......X.X.XX...X.X.X.X.....X....X] duration: 5.001251301s trace: https://jaeger.dev.storj.io/trace/1768c1c027131217 FAIL (2025-12-08 16:34:34 UTC) orbiter warning: context deadline exceeded Duration: 5.001274594s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 5.561923679s trace: https://jaeger.dev.storj.io/trace/5d817b557064ae10 FAIL (2025-12-08 16:34:28 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.561953647s core uplink eu1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..X....X..........................X...X.X...................] duration: 6.082311835s trace: https://jaeger.dev.storj.io/trace/20c661c94f160c60 FAIL (2025-12-08 16:34:23 UTC) orbiter warning: context deadline exceeded Duration: 6.082351089s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 18 out of 60 is failed (threshold=6) [X..XX.....X.....X...X..XX..X......X.X.XX...X.X.X.X.....X....] duration: 3.671205978s trace: https://jaeger.dev.storj.io/trace/73b9295ce8b68b8b FAIL (2025-12-08 16:34:17 UTC) orbiter warning: context deadline exceeded Duration: 3.671234572s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......X............................XX.......................] duration: 5.788928139s trace: https://jaeger.dev.storj.io/trace/13568c7eb15b2369 FAIL (2025-12-08 16:34:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, Piece ID: U3FCNUY4ASY3ZGX7ZY26GE2NA52GJNKJMW7XKZKALT2MWBCZMHFA) context deadline exceeded; piecestore: (Node ID: 1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, Piece ID: QN7JC2XQK6FKMIYD2WXQ7GMDILEWDUO2HXZLZERVJ24WWIMHFOVA) context deadline exceeded; piecestore: (Node ID: 12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, Piece ID: KRCKO7NXT2XOYK4QHVJK5UIU246TVB45EE7ELVUT365XTJ7RHKJQ) context deadline exceeded; piecestore: (Node ID: 122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, Piece ID: U7SKMZGV33HMSQLQIFIBZ6WNYC7JJ2CENSK5LCW3KVMP53T7467Q) context deadline exceeded; piecestore: (Node ID: 1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, Piece ID: FGYLRBVDMMLYWTTKLC3B2P2MHTECHEBH2DJOOKAHBR6TBAMKFBXA) context deadline exceeded ======LOGS====== Dec 8 16:15:19.690262294 (72h46m19.561345339s) [upload=63694]: Starting upload Dec 8 16:15:19.690357154 (72h46m19.56144022s) [upload=63694]: Done starting upload Dec 8 16:15:19.690425354 (72h46m19.561508409s) [upload=63694]: Uploading segments... Dec 8 16:15:19.690652296 (72h46m19.561735341s) [upload=63694]: Got next segment. Inline: false Dec 8 16:15:19.690693895 (72h46m19.56177694s) [upload=63694]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 16:15:19.690700889 (72h46m19.561783934s) [upload=63694]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 16:15:20.501121517 (72h46m20.372204572s) [seg_pos={0 0}, upload=63694]: Begin upload segment... Dec 8 16:15:20.501188935 (72h46m20.372271981s) [seg_pos={0 0}, upload=63694]: Done begin upload segment. Dec 8 16:15:20.501225655 (72h46m20.3723087s) [noise=true , address=145.79.227.16:4912 , piece_id=NIW3LT24E4LPJ5QYJVQOAR43S23JLDDOXZ6NVOURZNDTCCTLLPJA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.501311268 (72h46m20.372394313s) [noise=true , address=145.79.227.34:4906 , piece_id=AIZQB5ZLXPCLPT5Q7NLNSI3FPBPAMBVUIXKZCTA3ML7L3XP2UY6A, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.501987676 (72h46m20.373070722s) [noise=true , address=145.79.227.17:4971 , piece_id=KRCKO7NXT2XOYK4QHVJK5UIU246TVB45EE7ELVUT365XTJ7RHKJQ, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.502073379 (72h46m20.373156424s) [noise=true , address=145.79.227.38:4979 , piece_id=HCELMHS3GNLCKHFCQU3V45M4OETQZPL5MQJBUOUF5VPA44HXZW2Q, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.502559184 (72h46m20.373642229s) [noise=true , address=145.79.227.33:4870 , piece_id=IK7NDJD5GNFMWB26JYOMCSNIAT5TTTQOUXB2VFYSNRGIQ2MVMXAQ, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.502903350 (72h46m20.373986385s) [noise=true , address=145.79.227.22:4511 , piece_id=RT3ULMIM24LMC4UASFJYVAOHIKJBBGLFYJP3LBJCUL6I532MTSGA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503199825 (72h46m20.37428287s) [noise=true , address=145.79.227.29:4941 , piece_id=GJQXPA7WEZUTVJNELRBJVTU4LIGXYD7XLEPR5SWF4NBWWMXBW2SA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503205956 (72h46m20.374289001s) [noise=true , address=77.90.31.11:4555 , piece_id=LZ4YCICB4QQVLQAUZD6ZFABXLYSRKC4OFZESDVUGIOFJMZBRMMSA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503484487 (72h46m20.374567532s) [noise=true , address=145.79.227.39:4014 , piece_id=U7SKMZGV33HMSQLQIFIBZ6WNYC7JJ2CENSK5LCW3KVMP53T7467Q, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503757176 (72h46m20.374840221s) [noise=true , address=145.79.227.24:4582 , piece_id=FGYLRBVDMMLYWTTKLC3B2P2MHTECHEBH2DJOOKAHBR6TBAMKFBXA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503773727 (72h46m20.374856772s) [noise=true , address=145.79.227.40:4049 , piece_id=VX3PNKMHR75TNJLKRBLYS3KT42ZTPT4X6I5QS2AKNMGDQDVE4WDA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.504114126 (72h46m20.375197161s) [noise=true , address=77.90.31.17:4771 , piece_id=R6PLE6VLZNL23TJC6V6MRBLJP6PKB2DDGQMPPGLMM23WN6J7SH5Q, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.504441298 (72h46m20.375524343s) [noise=true , address=77.90.31.18:4807 , piece_id=7FXKUHBJT7AHDC4PHCZ73Q62GDSOUFRFIVAD337ORGE7YUTAULVA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.505240540 (72h46m20.376323585s) [noise=true , address=77.90.31.14:4662 , piece_id=BO6DQNLFNMG6HAPKOAG2DHWDD3YMX46IAYGDVQG7THC5EVZORJRA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.506022409 (72h46m20.377105454s) [noise=true , address=145.79.227.20:4438 , piece_id=U3FCNUY4ASY3ZGX7ZY26GE2NA52GJNKJMW7XKZKALT2MWBCZMHFA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.506709457 (72h46m20.377792502s) [noise=true , address=145.79.227.43:4158 , piece_id=QN7JC2XQK6FKMIYD2WXQ7GMDILEWDUO2HXZLZERVJ24WWIMHFOVA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:22.339915069 (72h46m22.210998114s) [upload=63694]: Next returned nil segment Dec 8 16:15:22.339932321 (72h46m22.211015366s) [upload=63694]: Waiting for error group managing segments... Dec 8 16:15:22.655133540 (72h46m22.526216575s) [noise=true , address=145.79.227.43:4158 , piece_id=QN7JC2XQK6FKMIYD2WXQ7GMDILEWDUO2HXZLZERVJ24WWIMHFOVA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.660462928 (72h46m22.531545983s) [noise=true , address=145.79.227.17:4971 , piece_id=KRCKO7NXT2XOYK4QHVJK5UIU246TVB45EE7ELVUT365XTJ7RHKJQ, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.683969820 (72h46m22.555052855s) [noise=true , address=77.90.31.11:4555 , piece_id=LZ4YCICB4QQVLQAUZD6ZFABXLYSRKC4OFZESDVUGIOFJMZBRMMSA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.685188641 (72h46m22.556271686s) [noise=true , address=145.79.227.22:4511 , piece_id=RT3ULMIM24LMC4UASFJYVAOHIKJBBGLFYJP3LBJCUL6I532MTSGA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.686713104 (72h46m22.557796149s) [noise=true , address=145.79.227.16:4912 , piece_id=NIW3LT24E4LPJ5QYJVQOAR43S23JLDDOXZ6NVOURZNDTCCTLLPJA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.698627942 (72h46m22.569710987s) [noise=true , address=145.79.227.39:4014 , piece_id=U7SKMZGV33HMSQLQIFIBZ6WNYC7JJ2CENSK5LCW3KVMP53T7467Q, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.713359143 (72h46m22.584442188s) [noise=true , address=145.79.227.40:4049 , piece_id=VX3PNKMHR75TNJLKRBLYS3KT42ZTPT4X6I5QS2AKNMGDQDVE4WDA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.731975212 (72h46m22.603058257s) [noise=true , address=77.90.31.17:4771 , piece_id=R6PLE6VLZNL23TJC6V6MRBLJP6PKB2DDGQMPPGLMM23WN6J7SH5Q, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.741125236 (72h46m22.612208281s) [noise=true , address=145.79.227.20:4438 , piece_id=U3FCNUY4ASY3ZGX7ZY26GE2NA52GJNKJMW7XKZKALT2MWBCZMHFA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.773771311 (72h46m22.644854346s) [noise=true , address=77.90.31.18:4807 , piece_id=7FXKUHBJT7AHDC4PHCZ73Q62GDSOUFRFIVAD337ORGE7YUTAULVA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.906364795 (72h46m22.777447841s) [noise=true , address=77.90.31.14:4662 , piece_id=BO6DQNLFNMG6HAPKOAG2DHWDD3YMX46IAYGDVQG7THC5EVZORJRA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:23.367795705 (72h46m23.23887874s) [noise=true , address=145.79.227.24:4582 , piece_id=FGYLRBVDMMLYWTTKLC3B2P2MHTECHEBH2DJOOKAHBR6TBAMKFBXA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:24.582400944 (72h46m24.453483979s) [noise=true , address=145.79.227.33:4870 , piece_id=IK7NDJD5GNFMWB26JYOMCSNIAT5TTTQOUXB2VFYSNRGIQ2MVMXAQ, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:24.582421023 (72h46m24.453504068s) [seg_pos={0 0}, upload=63694]: Segment reached optimal threshold of 13 pieces. Dec 8 16:15:24.582652363 (72h46m24.453735408s) [noise=true , address=145.79.227.38:4979 , piece_id=HCELMHS3GNLCKHFCQU3V45M4OETQZPL5MQJBUOUF5VPA44HXZW2Q, seg_pos={0 0}, upload=63694]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma): protocol: expected piece hash; context canceled; EOF Dec 8 16:15:24.582659086 (72h46m24.453742121s) [noise=true , address=145.79.227.38:4979 , piece_id=HCELMHS3GNLCKHFCQU3V45M4OETQZPL5MQJBUOUF5VPA44HXZW2Q, seg_pos={0 0}, upload=63694]: Long tail cancellation due to optimal threshold Dec 8 16:15:24.582740842 (72h46m24.453823877s) [noise=true , address=145.79.227.34:4906 , piece_id=AIZQB5ZLXPCLPT5Q7NLNSI3FPBPAMBVUIXKZCTA3ML7L3XP2UY6A, seg_pos={0 0}, upload=63694]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 8 16:15:24.582753055 (72h46m24.45383609s) [noise=true , address=145.79.227.34:4906 , piece_id=AIZQB5ZLXPCLPT5Q7NLNSI3FPBPAMBVUIXKZCTA3ML7L3XP2UY6A, seg_pos={0 0}, upload=63694]: Long tail cancellation due to optimal threshold Dec 8 16:15:24.583044900 (72h46m24.454127935s) [noise=true , address=145.79.227.29:4941 , piece_id=GJQXPA7WEZUTVJNELRBJVTU4LIGXYD7XLEPR5SWF4NBWWMXBW2SA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: context canceled; context canceled Dec 8 16:15:24.583052926 (72h46m24.454135961s) [noise=true , address=145.79.227.29:4941 , piece_id=GJQXPA7WEZUTVJNELRBJVTU4LIGXYD7XLEPR5SWF4NBWWMXBW2SA, seg_pos={0 0}, upload=63694]: Long tail cancellation due to optimal threshold Dec 8 16:15:24.583071441 (72h46m24.454154486s) [seg_pos={0 0}, upload=63694]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 16:15:24.583162014 (72h46m24.454245059s) [upload=63694]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 16:15:24.583171562 (72h46m24.454254627s) [upload=63694]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 16:15:25.042230695 (72h46m24.91331374s) [upload=63694]: Done uploading segments... Dec 8 16:15:25.042258869 (72h46m24.913341914s) [upload=63694]: Upload finished. err: <nil> Duration: 5.788959299s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)