Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [...X..X...X.................X.X.............................] duration: 5.853419408s trace: https://jaeger.dev.storj.io/trace/690f79ef204fee2 FAIL (2025-12-08 16:24:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.853459324s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 21 out of 60 is failed (threshold=6) [...XX.X..XX.XX.X..XX.....X.....X...X..XX..X......X.X.XX...X.] duration: 3.446930138s trace: https://jaeger.dev.storj.io/trace/19f01a746f9710cc FAIL (2025-12-08 16:24:25 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.447000021s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 5.104386336s trace: https://jaeger.dev.storj.io/trace/5d817b557064ae10 FAIL (2025-12-08 16:24:21 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.104413077s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 1.999335367s trace: https://jaeger.dev.storj.io/trace/4dad09bb52e33232 FAIL (2025-12-08 16:24:17 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.999439124s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [X.X....XX......X....X..........................X...X.X......] duration: 6.09562557s trace: https://jaeger.dev.storj.io/trace/20c661c94f160c60 FAIL (2025-12-08 16:24:16 UTC) orbiter warning: context deadline exceeded Duration: 6.095655697s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [......X.............X............................XX.........] duration: 5.884349809s trace: https://jaeger.dev.storj.io/trace/13568c7eb15b2369 FAIL (2025-12-08 16:24:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, Piece ID: U3FCNUY4ASY3ZGX7ZY26GE2NA52GJNKJMW7XKZKALT2MWBCZMHFA) context deadline exceeded; piecestore: (Node ID: 1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, Piece ID: QN7JC2XQK6FKMIYD2WXQ7GMDILEWDUO2HXZLZERVJ24WWIMHFOVA) context deadline exceeded; piecestore: (Node ID: 12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, Piece ID: KRCKO7NXT2XOYK4QHVJK5UIU246TVB45EE7ELVUT365XTJ7RHKJQ) context deadline exceeded; piecestore: (Node ID: 122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, Piece ID: U7SKMZGV33HMSQLQIFIBZ6WNYC7JJ2CENSK5LCW3KVMP53T7467Q) context deadline exceeded; piecestore: (Node ID: 1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, Piece ID: FGYLRBVDMMLYWTTKLC3B2P2MHTECHEBH2DJOOKAHBR6TBAMKFBXA) context deadline exceeded ======LOGS====== Dec 8 16:15:19.690262294 (72h46m19.561345339s) [upload=63694]: Starting upload Dec 8 16:15:19.690357154 (72h46m19.56144022s) [upload=63694]: Done starting upload Dec 8 16:15:19.690425354 (72h46m19.561508409s) [upload=63694]: Uploading segments... Dec 8 16:15:19.690652296 (72h46m19.561735341s) [upload=63694]: Got next segment. Inline: false Dec 8 16:15:19.690693895 (72h46m19.56177694s) [upload=63694]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 16:15:19.690700889 (72h46m19.561783934s) [upload=63694]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 16:15:20.501121517 (72h46m20.372204572s) [seg_pos={0 0}, upload=63694]: Begin upload segment... Dec 8 16:15:20.501188935 (72h46m20.372271981s) [seg_pos={0 0}, upload=63694]: Done begin upload segment. Dec 8 16:15:20.501225655 (72h46m20.3723087s) [noise=true , address=145.79.227.16:4912 , piece_id=NIW3LT24E4LPJ5QYJVQOAR43S23JLDDOXZ6NVOURZNDTCCTLLPJA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.501311268 (72h46m20.372394313s) [noise=true , address=145.79.227.34:4906 , piece_id=AIZQB5ZLXPCLPT5Q7NLNSI3FPBPAMBVUIXKZCTA3ML7L3XP2UY6A, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.501987676 (72h46m20.373070722s) [noise=true , address=145.79.227.17:4971 , piece_id=KRCKO7NXT2XOYK4QHVJK5UIU246TVB45EE7ELVUT365XTJ7RHKJQ, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.502073379 (72h46m20.373156424s) [noise=true , address=145.79.227.38:4979 , piece_id=HCELMHS3GNLCKHFCQU3V45M4OETQZPL5MQJBUOUF5VPA44HXZW2Q, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.502559184 (72h46m20.373642229s) [noise=true , address=145.79.227.33:4870 , piece_id=IK7NDJD5GNFMWB26JYOMCSNIAT5TTTQOUXB2VFYSNRGIQ2MVMXAQ, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.502903350 (72h46m20.373986385s) [noise=true , address=145.79.227.22:4511 , piece_id=RT3ULMIM24LMC4UASFJYVAOHIKJBBGLFYJP3LBJCUL6I532MTSGA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503199825 (72h46m20.37428287s) [noise=true , address=145.79.227.29:4941 , piece_id=GJQXPA7WEZUTVJNELRBJVTU4LIGXYD7XLEPR5SWF4NBWWMXBW2SA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503205956 (72h46m20.374289001s) [noise=true , address=77.90.31.11:4555 , piece_id=LZ4YCICB4QQVLQAUZD6ZFABXLYSRKC4OFZESDVUGIOFJMZBRMMSA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503484487 (72h46m20.374567532s) [noise=true , address=145.79.227.39:4014 , piece_id=U7SKMZGV33HMSQLQIFIBZ6WNYC7JJ2CENSK5LCW3KVMP53T7467Q, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503757176 (72h46m20.374840221s) [noise=true , address=145.79.227.24:4582 , piece_id=FGYLRBVDMMLYWTTKLC3B2P2MHTECHEBH2DJOOKAHBR6TBAMKFBXA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503773727 (72h46m20.374856772s) [noise=true , address=145.79.227.40:4049 , piece_id=VX3PNKMHR75TNJLKRBLYS3KT42ZTPT4X6I5QS2AKNMGDQDVE4WDA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.504114126 (72h46m20.375197161s) [noise=true , address=77.90.31.17:4771 , piece_id=R6PLE6VLZNL23TJC6V6MRBLJP6PKB2DDGQMPPGLMM23WN6J7SH5Q, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.504441298 (72h46m20.375524343s) [noise=true , address=77.90.31.18:4807 , piece_id=7FXKUHBJT7AHDC4PHCZ73Q62GDSOUFRFIVAD337ORGE7YUTAULVA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.505240540 (72h46m20.376323585s) [noise=true , address=77.90.31.14:4662 , piece_id=BO6DQNLFNMG6HAPKOAG2DHWDD3YMX46IAYGDVQG7THC5EVZORJRA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.506022409 (72h46m20.377105454s) [noise=true , address=145.79.227.20:4438 , piece_id=U3FCNUY4ASY3ZGX7ZY26GE2NA52GJNKJMW7XKZKALT2MWBCZMHFA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.506709457 (72h46m20.377792502s) [noise=true , address=145.79.227.43:4158 , piece_id=QN7JC2XQK6FKMIYD2WXQ7GMDILEWDUO2HXZLZERVJ24WWIMHFOVA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:22.339915069 (72h46m22.210998114s) [upload=63694]: Next returned nil segment Dec 8 16:15:22.339932321 (72h46m22.211015366s) [upload=63694]: Waiting for error group managing segments... Dec 8 16:15:22.655133540 (72h46m22.526216575s) [noise=true , address=145.79.227.43:4158 , piece_id=QN7JC2XQK6FKMIYD2WXQ7GMDILEWDUO2HXZLZERVJ24WWIMHFOVA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.660462928 (72h46m22.531545983s) [noise=true , address=145.79.227.17:4971 , piece_id=KRCKO7NXT2XOYK4QHVJK5UIU246TVB45EE7ELVUT365XTJ7RHKJQ, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.683969820 (72h46m22.555052855s) [noise=true , address=77.90.31.11:4555 , piece_id=LZ4YCICB4QQVLQAUZD6ZFABXLYSRKC4OFZESDVUGIOFJMZBRMMSA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.685188641 (72h46m22.556271686s) [noise=true , address=145.79.227.22:4511 , piece_id=RT3ULMIM24LMC4UASFJYVAOHIKJBBGLFYJP3LBJCUL6I532MTSGA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.686713104 (72h46m22.557796149s) [noise=true , address=145.79.227.16:4912 , piece_id=NIW3LT24E4LPJ5QYJVQOAR43S23JLDDOXZ6NVOURZNDTCCTLLPJA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.698627942 (72h46m22.569710987s) [noise=true , address=145.79.227.39:4014 , piece_id=U7SKMZGV33HMSQLQIFIBZ6WNYC7JJ2CENSK5LCW3KVMP53T7467Q, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.713359143 (72h46m22.584442188s) [noise=true , address=145.79.227.40:4049 , piece_id=VX3PNKMHR75TNJLKRBLYS3KT42ZTPT4X6I5QS2AKNMGDQDVE4WDA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.731975212 (72h46m22.603058257s) [noise=true , address=77.90.31.17:4771 , piece_id=R6PLE6VLZNL23TJC6V6MRBLJP6PKB2DDGQMPPGLMM23WN6J7SH5Q, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.741125236 (72h46m22.612208281s) [noise=true , address=145.79.227.20:4438 , piece_id=U3FCNUY4ASY3ZGX7ZY26GE2NA52GJNKJMW7XKZKALT2MWBCZMHFA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.773771311 (72h46m22.644854346s) [noise=true , address=77.90.31.18:4807 , piece_id=7FXKUHBJT7AHDC4PHCZ73Q62GDSOUFRFIVAD337ORGE7YUTAULVA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.906364795 (72h46m22.777447841s) [noise=true , address=77.90.31.14:4662 , piece_id=BO6DQNLFNMG6HAPKOAG2DHWDD3YMX46IAYGDVQG7THC5EVZORJRA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:23.367795705 (72h46m23.23887874s) [noise=true , address=145.79.227.24:4582 , piece_id=FGYLRBVDMMLYWTTKLC3B2P2MHTECHEBH2DJOOKAHBR6TBAMKFBXA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:24.582400944 (72h46m24.453483979s) [noise=true , address=145.79.227.33:4870 , piece_id=IK7NDJD5GNFMWB26JYOMCSNIAT5TTTQOUXB2VFYSNRGIQ2MVMXAQ, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:24.582421023 (72h46m24.453504068s) [seg_pos={0 0}, upload=63694]: Segment reached optimal threshold of 13 pieces. Dec 8 16:15:24.582652363 (72h46m24.453735408s) [noise=true , address=145.79.227.38:4979 , piece_id=HCELMHS3GNLCKHFCQU3V45M4OETQZPL5MQJBUOUF5VPA44HXZW2Q, seg_pos={0 0}, upload=63694]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma): protocol: expected piece hash; context canceled; EOF Dec 8 16:15:24.582659086 (72h46m24.453742121s) [noise=true , address=145.79.227.38:4979 , piece_id=HCELMHS3GNLCKHFCQU3V45M4OETQZPL5MQJBUOUF5VPA44HXZW2Q, seg_pos={0 0}, upload=63694]: Long tail cancellation due to optimal threshold Dec 8 16:15:24.582740842 (72h46m24.453823877s) [noise=true , address=145.79.227.34:4906 , piece_id=AIZQB5ZLXPCLPT5Q7NLNSI3FPBPAMBVUIXKZCTA3ML7L3XP2UY6A, seg_pos={0 0}, upload=63694]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 8 16:15:24.582753055 (72h46m24.45383609s) [noise=true , address=145.79.227.34:4906 , piece_id=AIZQB5ZLXPCLPT5Q7NLNSI3FPBPAMBVUIXKZCTA3ML7L3XP2UY6A, seg_pos={0 0}, upload=63694]: Long tail cancellation due to optimal threshold Dec 8 16:15:24.583044900 (72h46m24.454127935s) [noise=true , address=145.79.227.29:4941 , piece_id=GJQXPA7WEZUTVJNELRBJVTU4LIGXYD7XLEPR5SWF4NBWWMXBW2SA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: context canceled; context canceled Dec 8 16:15:24.583052926 (72h46m24.454135961s) [noise=true , address=145.79.227.29:4941 , piece_id=GJQXPA7WEZUTVJNELRBJVTU4LIGXYD7XLEPR5SWF4NBWWMXBW2SA, seg_pos={0 0}, upload=63694]: Long tail cancellation due to optimal threshold Dec 8 16:15:24.583071441 (72h46m24.454154486s) [seg_pos={0 0}, upload=63694]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 16:15:24.583162014 (72h46m24.454245059s) [upload=63694]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 16:15:24.583171562 (72h46m24.454254627s) [upload=63694]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 16:15:25.042230695 (72h46m24.91331374s) [upload=63694]: Done uploading segments... Dec 8 16:15:25.042258869 (72h46m24.913341914s) [upload=63694]: Upload finished. err: <nil> Duration: 5.884376118s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 6.634024361s trace: https://jaeger.dev.storj.io/trace/4195524620a3e2c2 FAIL (2025-12-08 16:24:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.771167108 (72h9m11.642250153s) [noise=true , address=81.171.14.70:4032 , piece_id=XAMDRY73RIJV4LC2W2FRMH6L4WL3VAFUYKB3QD4TXOQWWZUYGPOQ, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.771462930 (72h9m11.642545976s) [noise=true , address=81.171.14.70:4035 , piece_id=VYAAT75X4R4J22MLY7RK4VHOEFMYZTLWUWTWVAMOOIIDSEXTQV7Q, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.771589993 (72h9m11.642673038s) [noise=true , address=185.167.204.205:4036 , piece_id=X2TWRAEKE6FEH3ITYY74A76ZZQ2KCB63ETZNW5TAHQ3MLEM34PZQ, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.772028248 (72h9m11.643111293s) [noise=true , address=185.167.204.206:4067 , piece_id=3YOG2SUYBHBE6MMIPMQWMBDZTLL7HPSBPEKFHMGOICBTAQU5HA4A, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.772041683 (72h9m11.643124728s) [noise=true , address=185.167.204.206:4069 , piece_id=GH2ZEUIEIUZ6OLM6XH5ZJVXKAMHUKQ6L3GZ7NJLUZFY623YH5GDQ, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.772603803 (72h9m11.643686848s) [noise=true , address=89.149.201.147:4022 , piece_id=CWWNRXZNWQOUTZEA4WPHGA6BYEPO35D2YDU5VSTHFZ7GRC67Y7VQ, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.773136077 (72h9m11.644219122s) [noise=true , address=185.167.204.206:4051 , piece_id=5JMUWS6LLPRENPBAZ74MDELLIZMWTMI3GOZFUWLCOCUNLDTPETWQ, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.773659244 (72h9m11.644742289s) [noise=true , address=185.167.204.206:4062 , piece_id=KYFUGH5EAVFGEJ2J7VDRDWBDP2R2MX4JSCIU7RP3VJOK4LCM6D2Q, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.774276288 (72h9m11.645359333s) [noise=true , address=81.171.14.70:4026 , piece_id=4BLGOA4GSQVOKZ6UBUIXANALBDHDHDXPLDA3TDQ44ACTIC2IHSEA, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.774811648 (72h9m11.645894693s) [noise=true , address=185.167.204.205:4008 , piece_id=JWZAQAYIUNBFRDVQHADARS5NTEWET7UI2GR73ZMIACA5VFUS3D6Q, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:12.600277390 (72h9m12.471360435s) [upload=63171]: Next returned nil segment Dec 8 15:38:12.600300113 (72h9m12.471383158s) [upload=63171]: Waiting for error group managing segments... Dec 8 15:38:12.984985581 (72h9m12.856068647s) [noise=true , address=185.167.204.205:4028 , piece_id=DCGYFXSKGK6GFLLFU7UQONAX643EA6CDTYWEZ42YQPLEERINL3KQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.985372348 (72h9m12.856455383s) [noise=true , address=185.167.204.205:4008 , piece_id=JWZAQAYIUNBFRDVQHADARS5NTEWET7UI2GR73ZMIACA5VFUS3D6Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.993832749 (72h9m12.864915784s) [noise=true , address=185.167.204.205:4015 , piece_id=GK2HV5AGF2Q6RYHDZBZQ3SY537WQXB7E7UF4SWF5X3LFM4PMCZPQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.994210087 (72h9m12.865293122s) [noise=true , address=185.167.204.206:4053 , piece_id=DGSLZPKB474OADK43GBZO2FGAR2F62PBL364EWVNPCZE3PWZGQPA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.994561226 (72h9m12.865644271s) [noise=true , address=185.167.204.205:4036 , piece_id=X2TWRAEKE6FEH3ITYY74A76ZZQ2KCB63ETZNW5TAHQ3MLEM34PZQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.994944356 (72h9m12.866027401s) [noise=true , address=185.167.204.205:4023 , piece_id=5RNGVQ3LUTPKX4RAVK7Y4WRQC2Y6GT5QWV2G3MTY2LW2MU6IMKLQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.995089352 (72h9m12.866172397s) [noise=true , address=185.167.204.205:4007 , piece_id=WEZIGATHDFL7G3S2VG7BEFGGT7JJIXBDWHGIPHBTPSKDJQUTROIQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.995157291 (72h9m12.866240336s) [noise=true , address=185.167.204.205:4033 , piece_id=4V3DXW42ZCPJFF3PPJEBMBYFW354WEZYLVFUSPGEZ4E4HZ7MFI4Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.000801098 (72h9m12.871884163s) [noise=true , address=185.167.204.206:4062 , piece_id=KYFUGH5EAVFGEJ2J7VDRDWBDP2R2MX4JSCIU7RP3VJOK4LCM6D2Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.000862164 (72h9m12.871945209s) [noise=true , address=185.167.204.206:4051 , piece_id=5JMUWS6LLPRENPBAZ74MDELLIZMWTMI3GOZFUWLCOCUNLDTPETWQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.002117705 (72h9m12.87320074s) [noise=true , address=185.167.204.206:4064 , piece_id=LUITKU5LPQ67MFA4VWE6CDV46QOU7RXOFCGKKAYPCLFGWFSLYLTA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.003015134 (72h9m12.874098179s) [noise=true , address=185.167.204.205:4034 , piece_id=OJPEC5SBHQDKJW3LDBPHMEOPDG6NPNNMO3W54C7XIXTNUYBTMUOA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.003432589 (72h9m12.874515624s) [noise=true , address=185.167.204.206:4037 , piece_id=OPF6NTGFZ7FSN7LIEKHOGIJRYUWX27RTI3KKGCKPCOQLK4HC62LQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.004078648 (72h9m12.875161694s) [noise=true , address=185.167.204.206:4069 , piece_id=GH2ZEUIEIUZ6OLM6XH5ZJVXKAMHUKQ6L3GZ7NJLUZFY623YH5GDQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.009788571 (72h9m12.880871616s) [noise=true , address=185.167.204.206:4067 , piece_id=3YOG2SUYBHBE6MMIPMQWMBDZTLL7HPSBPEKFHMGOICBTAQU5HA4A, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.012083561 (72h9m12.883166596s) [noise=true , address=185.167.204.205:4030 , piece_id=EFAUOQCDA25VE55ZHJOSVXTOQD2VTQCW55MEH5DBEAY7N364GMXA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.584995241 (72h9m13.456078286s) [noise=true , address=81.171.14.70:4046 , piece_id=77RPCXTIQW7MO6UKAVJUD4Y7XR674IDZE2TL2SB77LJUHRE3ACDA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.651263008 (72h9m13.522346063s) [noise=true , address=81.171.14.70:4026 , piece_id=4BLGOA4GSQVOKZ6UBUIXANALBDHDHDXPLDA3TDQ44ACTIC2IHSEA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.696266204 (72h9m13.567349249s) [noise=true , address=81.171.14.70:4035 , piece_id=VYAAT75X4R4J22MLY7RK4VHOEFMYZTLWUWTWVAMOOIIDSEXTQV7Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.123999844 (72h9m13.995082889s) [noise=true , address=81.171.14.70:4025 , piece_id=XML4X66PHAT33VTUTEV22HSTIHMG4WFZU6YSDPCZZZ4SEBL3NWUA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.147808461 (72h9m14.018891506s) [noise=true , address=81.171.14.70:4032 , piece_id=XAMDRY73RIJV4LC2W2FRMH6L4WL3VAFUYKB3QD4TXOQWWZUYGPOQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.255119813 (72h9m14.126202858s) [noise=true , address=89.149.201.147:4005 , piece_id=MBR3CYKQ3YSPVXIVSL6ABGZNG7FB5HGP45OA3YE2IRKPJMOSM3KQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.271452473 (72h9m14.142535518s) [noise=true , address=89.149.201.147:4013 , piece_id=KIPMWZVYZADDS2LAOY5ASMIWP32HIRTNGYQ65SNLYR2EQDQCN3HQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.775652542 (72h9m14.646735587s) [noise=true , address=89.149.201.147:4016 , piece_id=5N5HKBPMDQUEQIZZVFFRQGE4LQAGSJ4H6INPDXHYQMDK5YTHUVKQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.785904314 (72h9m14.656987359s) [noise=true , address=89.149.201.147:4017 , piece_id=MBD47F6TJHZ76PAGU7ZIFRQ4M5HUDT5AQULHE6RPQPBUBAVKSTKQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.827349233 (72h9m14.698432278s) [noise=true , address=89.149.201.147:4022 , piece_id=CWWNRXZNWQOUTZEA4WPHGA6BYEPO35D2YDU5VSTHFZ7GRC67Y7VQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.827365925 (72h9m14.69844897s) [seg_pos={0 0}, upload=63171]: Segment reached optimal threshold of 26 pieces. Dec 8 15:38:14.827539125 (72h9m14.69862217s) [noise=true , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hJ5jT6S2d3DJe9DoVDjHMkpwSTpeyTMyNLaK8Pk8SWcUVRKnv): protocol: expected piece hash; context canceled; EOF Dec 8 15:38:14.827545567 (72h9m14.698628612s) [noise=true , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Long tail cancellation due to optimal threshold Dec 8 15:38:14.827903338 (72h9m14.698986373s) [noise=true , address=89.149.201.147:4018 , piece_id=BFJH5ELBTZD767FJNFT2ZBUS3GI62IZIRSKYARLVBCLWZONSURMQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua): protocol: expected piece hash; context canceled; EOF Dec 8 15:38:14.827910673 (72h9m14.698993708s) [noise=true , address=89.149.201.147:4018 , piece_id=BFJH5ELBTZD767FJNFT2ZBUS3GI62IZIRSKYARLVBCLWZONSURMQ, seg_pos={0 0}, upload=63171]: Long tail cancellation due to optimal threshold Dec 8 15:38:14.827928065 (72h9m14.69901111s) [seg_pos={0 0}, upload=63171]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 8 15:38:14.828037083 (72h9m14.699120128s) [upload=63171]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 15:38:14.828053224 (72h9m14.699136269s) [upload=63171]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 15:39:18.272006268 (72h10m18.143089313s) [upload=63171]: Done uploading segments... Dec 8 15:39:18.272087593 (72h10m18.143170628s) [upload=63171]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.634052525s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [....X..X...X.................X.X............................] duration: 6.228569714s trace: https://jaeger.dev.storj.io/trace/690f79ef204fee2 FAIL (2025-12-08 16:23:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.228596245s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 2.452153046s trace: https://jaeger.dev.storj.io/trace/4dad09bb52e33232 FAIL (2025-12-08 16:23:32 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 2.452184456s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......X.............X............................XX........] duration: 6.567746255s trace: https://jaeger.dev.storj.io/trace/13568c7eb15b2369 FAIL (2025-12-08 16:23:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, Piece ID: U3FCNUY4ASY3ZGX7ZY26GE2NA52GJNKJMW7XKZKALT2MWBCZMHFA) context deadline exceeded; piecestore: (Node ID: 1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, Piece ID: QN7JC2XQK6FKMIYD2WXQ7GMDILEWDUO2HXZLZERVJ24WWIMHFOVA) context deadline exceeded; piecestore: (Node ID: 12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, Piece ID: KRCKO7NXT2XOYK4QHVJK5UIU246TVB45EE7ELVUT365XTJ7RHKJQ) context deadline exceeded; piecestore: (Node ID: 122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, Piece ID: U7SKMZGV33HMSQLQIFIBZ6WNYC7JJ2CENSK5LCW3KVMP53T7467Q) context deadline exceeded; piecestore: (Node ID: 1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, Piece ID: FGYLRBVDMMLYWTTKLC3B2P2MHTECHEBH2DJOOKAHBR6TBAMKFBXA) context deadline exceeded ======LOGS====== Dec 8 16:15:19.690262294 (72h46m19.561345339s) [upload=63694]: Starting upload Dec 8 16:15:19.690357154 (72h46m19.56144022s) [upload=63694]: Done starting upload Dec 8 16:15:19.690425354 (72h46m19.561508409s) [upload=63694]: Uploading segments... Dec 8 16:15:19.690652296 (72h46m19.561735341s) [upload=63694]: Got next segment. Inline: false Dec 8 16:15:19.690693895 (72h46m19.56177694s) [upload=63694]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 16:15:19.690700889 (72h46m19.561783934s) [upload=63694]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 16:15:20.501121517 (72h46m20.372204572s) [seg_pos={0 0}, upload=63694]: Begin upload segment... Dec 8 16:15:20.501188935 (72h46m20.372271981s) [seg_pos={0 0}, upload=63694]: Done begin upload segment. Dec 8 16:15:20.501225655 (72h46m20.3723087s) [noise=true , address=145.79.227.16:4912 , piece_id=NIW3LT24E4LPJ5QYJVQOAR43S23JLDDOXZ6NVOURZNDTCCTLLPJA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.501311268 (72h46m20.372394313s) [noise=true , address=145.79.227.34:4906 , piece_id=AIZQB5ZLXPCLPT5Q7NLNSI3FPBPAMBVUIXKZCTA3ML7L3XP2UY6A, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.501987676 (72h46m20.373070722s) [noise=true , address=145.79.227.17:4971 , piece_id=KRCKO7NXT2XOYK4QHVJK5UIU246TVB45EE7ELVUT365XTJ7RHKJQ, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.502073379 (72h46m20.373156424s) [noise=true , address=145.79.227.38:4979 , piece_id=HCELMHS3GNLCKHFCQU3V45M4OETQZPL5MQJBUOUF5VPA44HXZW2Q, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.502559184 (72h46m20.373642229s) [noise=true , address=145.79.227.33:4870 , piece_id=IK7NDJD5GNFMWB26JYOMCSNIAT5TTTQOUXB2VFYSNRGIQ2MVMXAQ, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.502903350 (72h46m20.373986385s) [noise=true , address=145.79.227.22:4511 , piece_id=RT3ULMIM24LMC4UASFJYVAOHIKJBBGLFYJP3LBJCUL6I532MTSGA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503199825 (72h46m20.37428287s) [noise=true , address=145.79.227.29:4941 , piece_id=GJQXPA7WEZUTVJNELRBJVTU4LIGXYD7XLEPR5SWF4NBWWMXBW2SA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503205956 (72h46m20.374289001s) [noise=true , address=77.90.31.11:4555 , piece_id=LZ4YCICB4QQVLQAUZD6ZFABXLYSRKC4OFZESDVUGIOFJMZBRMMSA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503484487 (72h46m20.374567532s) [noise=true , address=145.79.227.39:4014 , piece_id=U7SKMZGV33HMSQLQIFIBZ6WNYC7JJ2CENSK5LCW3KVMP53T7467Q, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503757176 (72h46m20.374840221s) [noise=true , address=145.79.227.24:4582 , piece_id=FGYLRBVDMMLYWTTKLC3B2P2MHTECHEBH2DJOOKAHBR6TBAMKFBXA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503773727 (72h46m20.374856772s) [noise=true , address=145.79.227.40:4049 , piece_id=VX3PNKMHR75TNJLKRBLYS3KT42ZTPT4X6I5QS2AKNMGDQDVE4WDA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.504114126 (72h46m20.375197161s) [noise=true , address=77.90.31.17:4771 , piece_id=R6PLE6VLZNL23TJC6V6MRBLJP6PKB2DDGQMPPGLMM23WN6J7SH5Q, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.504441298 (72h46m20.375524343s) [noise=true , address=77.90.31.18:4807 , piece_id=7FXKUHBJT7AHDC4PHCZ73Q62GDSOUFRFIVAD337ORGE7YUTAULVA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.505240540 (72h46m20.376323585s) [noise=true , address=77.90.31.14:4662 , piece_id=BO6DQNLFNMG6HAPKOAG2DHWDD3YMX46IAYGDVQG7THC5EVZORJRA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.506022409 (72h46m20.377105454s) [noise=true , address=145.79.227.20:4438 , piece_id=U3FCNUY4ASY3ZGX7ZY26GE2NA52GJNKJMW7XKZKALT2MWBCZMHFA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.506709457 (72h46m20.377792502s) [noise=true , address=145.79.227.43:4158 , piece_id=QN7JC2XQK6FKMIYD2WXQ7GMDILEWDUO2HXZLZERVJ24WWIMHFOVA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:22.339915069 (72h46m22.210998114s) [upload=63694]: Next returned nil segment Dec 8 16:15:22.339932321 (72h46m22.211015366s) [upload=63694]: Waiting for error group managing segments... Dec 8 16:15:22.655133540 (72h46m22.526216575s) [noise=true , address=145.79.227.43:4158 , piece_id=QN7JC2XQK6FKMIYD2WXQ7GMDILEWDUO2HXZLZERVJ24WWIMHFOVA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.660462928 (72h46m22.531545983s) [noise=true , address=145.79.227.17:4971 , piece_id=KRCKO7NXT2XOYK4QHVJK5UIU246TVB45EE7ELVUT365XTJ7RHKJQ, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.683969820 (72h46m22.555052855s) [noise=true , address=77.90.31.11:4555 , piece_id=LZ4YCICB4QQVLQAUZD6ZFABXLYSRKC4OFZESDVUGIOFJMZBRMMSA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.685188641 (72h46m22.556271686s) [noise=true , address=145.79.227.22:4511 , piece_id=RT3ULMIM24LMC4UASFJYVAOHIKJBBGLFYJP3LBJCUL6I532MTSGA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.686713104 (72h46m22.557796149s) [noise=true , address=145.79.227.16:4912 , piece_id=NIW3LT24E4LPJ5QYJVQOAR43S23JLDDOXZ6NVOURZNDTCCTLLPJA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.698627942 (72h46m22.569710987s) [noise=true , address=145.79.227.39:4014 , piece_id=U7SKMZGV33HMSQLQIFIBZ6WNYC7JJ2CENSK5LCW3KVMP53T7467Q, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.713359143 (72h46m22.584442188s) [noise=true , address=145.79.227.40:4049 , piece_id=VX3PNKMHR75TNJLKRBLYS3KT42ZTPT4X6I5QS2AKNMGDQDVE4WDA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.731975212 (72h46m22.603058257s) [noise=true , address=77.90.31.17:4771 , piece_id=R6PLE6VLZNL23TJC6V6MRBLJP6PKB2DDGQMPPGLMM23WN6J7SH5Q, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.741125236 (72h46m22.612208281s) [noise=true , address=145.79.227.20:4438 , piece_id=U3FCNUY4ASY3ZGX7ZY26GE2NA52GJNKJMW7XKZKALT2MWBCZMHFA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.773771311 (72h46m22.644854346s) [noise=true , address=77.90.31.18:4807 , piece_id=7FXKUHBJT7AHDC4PHCZ73Q62GDSOUFRFIVAD337ORGE7YUTAULVA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.906364795 (72h46m22.777447841s) [noise=true , address=77.90.31.14:4662 , piece_id=BO6DQNLFNMG6HAPKOAG2DHWDD3YMX46IAYGDVQG7THC5EVZORJRA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:23.367795705 (72h46m23.23887874s) [noise=true , address=145.79.227.24:4582 , piece_id=FGYLRBVDMMLYWTTKLC3B2P2MHTECHEBH2DJOOKAHBR6TBAMKFBXA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:24.582400944 (72h46m24.453483979s) [noise=true , address=145.79.227.33:4870 , piece_id=IK7NDJD5GNFMWB26JYOMCSNIAT5TTTQOUXB2VFYSNRGIQ2MVMXAQ, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:24.582421023 (72h46m24.453504068s) [seg_pos={0 0}, upload=63694]: Segment reached optimal threshold of 13 pieces. Dec 8 16:15:24.582652363 (72h46m24.453735408s) [noise=true , address=145.79.227.38:4979 , piece_id=HCELMHS3GNLCKHFCQU3V45M4OETQZPL5MQJBUOUF5VPA44HXZW2Q, seg_pos={0 0}, upload=63694]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma): protocol: expected piece hash; context canceled; EOF Dec 8 16:15:24.582659086 (72h46m24.453742121s) [noise=true , address=145.79.227.38:4979 , piece_id=HCELMHS3GNLCKHFCQU3V45M4OETQZPL5MQJBUOUF5VPA44HXZW2Q, seg_pos={0 0}, upload=63694]: Long tail cancellation due to optimal threshold Dec 8 16:15:24.582740842 (72h46m24.453823877s) [noise=true , address=145.79.227.34:4906 , piece_id=AIZQB5ZLXPCLPT5Q7NLNSI3FPBPAMBVUIXKZCTA3ML7L3XP2UY6A, seg_pos={0 0}, upload=63694]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 8 16:15:24.582753055 (72h46m24.45383609s) [noise=true , address=145.79.227.34:4906 , piece_id=AIZQB5ZLXPCLPT5Q7NLNSI3FPBPAMBVUIXKZCTA3ML7L3XP2UY6A, seg_pos={0 0}, upload=63694]: Long tail cancellation due to optimal threshold Dec 8 16:15:24.583044900 (72h46m24.454127935s) [noise=true , address=145.79.227.29:4941 , piece_id=GJQXPA7WEZUTVJNELRBJVTU4LIGXYD7XLEPR5SWF4NBWWMXBW2SA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: context canceled; context canceled Dec 8 16:15:24.583052926 (72h46m24.454135961s) [noise=true , address=145.79.227.29:4941 , piece_id=GJQXPA7WEZUTVJNELRBJVTU4LIGXYD7XLEPR5SWF4NBWWMXBW2SA, seg_pos={0 0}, upload=63694]: Long tail cancellation due to optimal threshold Dec 8 16:15:24.583071441 (72h46m24.454154486s) [seg_pos={0 0}, upload=63694]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 16:15:24.583162014 (72h46m24.454245059s) [upload=63694]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 16:15:24.583171562 (72h46m24.454254627s) [upload=63694]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 16:15:25.042230695 (72h46m24.91331374s) [upload=63694]: Done uploading segments... Dec 8 16:15:25.042258869 (72h46m24.913341914s) [upload=63694]: Upload finished. err: <nil> Duration: 6.567774718s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 5.679406139s trace: https://jaeger.dev.storj.io/trace/4195524620a3e2c2 FAIL (2025-12-08 16:23:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.771167108 (72h9m11.642250153s) [noise=true , address=81.171.14.70:4032 , piece_id=XAMDRY73RIJV4LC2W2FRMH6L4WL3VAFUYKB3QD4TXOQWWZUYGPOQ, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.771462930 (72h9m11.642545976s) [noise=true , address=81.171.14.70:4035 , piece_id=VYAAT75X4R4J22MLY7RK4VHOEFMYZTLWUWTWVAMOOIIDSEXTQV7Q, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.771589993 (72h9m11.642673038s) [noise=true , address=185.167.204.205:4036 , piece_id=X2TWRAEKE6FEH3ITYY74A76ZZQ2KCB63ETZNW5TAHQ3MLEM34PZQ, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.772028248 (72h9m11.643111293s) [noise=true , address=185.167.204.206:4067 , piece_id=3YOG2SUYBHBE6MMIPMQWMBDZTLL7HPSBPEKFHMGOICBTAQU5HA4A, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.772041683 (72h9m11.643124728s) [noise=true , address=185.167.204.206:4069 , piece_id=GH2ZEUIEIUZ6OLM6XH5ZJVXKAMHUKQ6L3GZ7NJLUZFY623YH5GDQ, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.772603803 (72h9m11.643686848s) [noise=true , address=89.149.201.147:4022 , piece_id=CWWNRXZNWQOUTZEA4WPHGA6BYEPO35D2YDU5VSTHFZ7GRC67Y7VQ, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.773136077 (72h9m11.644219122s) [noise=true , address=185.167.204.206:4051 , piece_id=5JMUWS6LLPRENPBAZ74MDELLIZMWTMI3GOZFUWLCOCUNLDTPETWQ, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.773659244 (72h9m11.644742289s) [noise=true , address=185.167.204.206:4062 , piece_id=KYFUGH5EAVFGEJ2J7VDRDWBDP2R2MX4JSCIU7RP3VJOK4LCM6D2Q, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.774276288 (72h9m11.645359333s) [noise=true , address=81.171.14.70:4026 , piece_id=4BLGOA4GSQVOKZ6UBUIXANALBDHDHDXPLDA3TDQ44ACTIC2IHSEA, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.774811648 (72h9m11.645894693s) [noise=true , address=185.167.204.205:4008 , piece_id=JWZAQAYIUNBFRDVQHADARS5NTEWET7UI2GR73ZMIACA5VFUS3D6Q, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:12.600277390 (72h9m12.471360435s) [upload=63171]: Next returned nil segment Dec 8 15:38:12.600300113 (72h9m12.471383158s) [upload=63171]: Waiting for error group managing segments... Dec 8 15:38:12.984985581 (72h9m12.856068647s) [noise=true , address=185.167.204.205:4028 , piece_id=DCGYFXSKGK6GFLLFU7UQONAX643EA6CDTYWEZ42YQPLEERINL3KQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.985372348 (72h9m12.856455383s) [noise=true , address=185.167.204.205:4008 , piece_id=JWZAQAYIUNBFRDVQHADARS5NTEWET7UI2GR73ZMIACA5VFUS3D6Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.993832749 (72h9m12.864915784s) [noise=true , address=185.167.204.205:4015 , piece_id=GK2HV5AGF2Q6RYHDZBZQ3SY537WQXB7E7UF4SWF5X3LFM4PMCZPQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.994210087 (72h9m12.865293122s) [noise=true , address=185.167.204.206:4053 , piece_id=DGSLZPKB474OADK43GBZO2FGAR2F62PBL364EWVNPCZE3PWZGQPA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.994561226 (72h9m12.865644271s) [noise=true , address=185.167.204.205:4036 , piece_id=X2TWRAEKE6FEH3ITYY74A76ZZQ2KCB63ETZNW5TAHQ3MLEM34PZQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.994944356 (72h9m12.866027401s) [noise=true , address=185.167.204.205:4023 , piece_id=5RNGVQ3LUTPKX4RAVK7Y4WRQC2Y6GT5QWV2G3MTY2LW2MU6IMKLQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.995089352 (72h9m12.866172397s) [noise=true , address=185.167.204.205:4007 , piece_id=WEZIGATHDFL7G3S2VG7BEFGGT7JJIXBDWHGIPHBTPSKDJQUTROIQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.995157291 (72h9m12.866240336s) [noise=true , address=185.167.204.205:4033 , piece_id=4V3DXW42ZCPJFF3PPJEBMBYFW354WEZYLVFUSPGEZ4E4HZ7MFI4Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.000801098 (72h9m12.871884163s) [noise=true , address=185.167.204.206:4062 , piece_id=KYFUGH5EAVFGEJ2J7VDRDWBDP2R2MX4JSCIU7RP3VJOK4LCM6D2Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.000862164 (72h9m12.871945209s) [noise=true , address=185.167.204.206:4051 , piece_id=5JMUWS6LLPRENPBAZ74MDELLIZMWTMI3GOZFUWLCOCUNLDTPETWQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.002117705 (72h9m12.87320074s) [noise=true , address=185.167.204.206:4064 , piece_id=LUITKU5LPQ67MFA4VWE6CDV46QOU7RXOFCGKKAYPCLFGWFSLYLTA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.003015134 (72h9m12.874098179s) [noise=true , address=185.167.204.205:4034 , piece_id=OJPEC5SBHQDKJW3LDBPHMEOPDG6NPNNMO3W54C7XIXTNUYBTMUOA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.003432589 (72h9m12.874515624s) [noise=true , address=185.167.204.206:4037 , piece_id=OPF6NTGFZ7FSN7LIEKHOGIJRYUWX27RTI3KKGCKPCOQLK4HC62LQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.004078648 (72h9m12.875161694s) [noise=true , address=185.167.204.206:4069 , piece_id=GH2ZEUIEIUZ6OLM6XH5ZJVXKAMHUKQ6L3GZ7NJLUZFY623YH5GDQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.009788571 (72h9m12.880871616s) [noise=true , address=185.167.204.206:4067 , piece_id=3YOG2SUYBHBE6MMIPMQWMBDZTLL7HPSBPEKFHMGOICBTAQU5HA4A, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.012083561 (72h9m12.883166596s) [noise=true , address=185.167.204.205:4030 , piece_id=EFAUOQCDA25VE55ZHJOSVXTOQD2VTQCW55MEH5DBEAY7N364GMXA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.584995241 (72h9m13.456078286s) [noise=true , address=81.171.14.70:4046 , piece_id=77RPCXTIQW7MO6UKAVJUD4Y7XR674IDZE2TL2SB77LJUHRE3ACDA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.651263008 (72h9m13.522346063s) [noise=true , address=81.171.14.70:4026 , piece_id=4BLGOA4GSQVOKZ6UBUIXANALBDHDHDXPLDA3TDQ44ACTIC2IHSEA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.696266204 (72h9m13.567349249s) [noise=true , address=81.171.14.70:4035 , piece_id=VYAAT75X4R4J22MLY7RK4VHOEFMYZTLWUWTWVAMOOIIDSEXTQV7Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.123999844 (72h9m13.995082889s) [noise=true , address=81.171.14.70:4025 , piece_id=XML4X66PHAT33VTUTEV22HSTIHMG4WFZU6YSDPCZZZ4SEBL3NWUA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.147808461 (72h9m14.018891506s) [noise=true , address=81.171.14.70:4032 , piece_id=XAMDRY73RIJV4LC2W2FRMH6L4WL3VAFUYKB3QD4TXOQWWZUYGPOQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.255119813 (72h9m14.126202858s) [noise=true , address=89.149.201.147:4005 , piece_id=MBR3CYKQ3YSPVXIVSL6ABGZNG7FB5HGP45OA3YE2IRKPJMOSM3KQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.271452473 (72h9m14.142535518s) [noise=true , address=89.149.201.147:4013 , piece_id=KIPMWZVYZADDS2LAOY5ASMIWP32HIRTNGYQ65SNLYR2EQDQCN3HQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.775652542 (72h9m14.646735587s) [noise=true , address=89.149.201.147:4016 , piece_id=5N5HKBPMDQUEQIZZVFFRQGE4LQAGSJ4H6INPDXHYQMDK5YTHUVKQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.785904314 (72h9m14.656987359s) [noise=true , address=89.149.201.147:4017 , piece_id=MBD47F6TJHZ76PAGU7ZIFRQ4M5HUDT5AQULHE6RPQPBUBAVKSTKQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.827349233 (72h9m14.698432278s) [noise=true , address=89.149.201.147:4022 , piece_id=CWWNRXZNWQOUTZEA4WPHGA6BYEPO35D2YDU5VSTHFZ7GRC67Y7VQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.827365925 (72h9m14.69844897s) [seg_pos={0 0}, upload=63171]: Segment reached optimal threshold of 26 pieces. Dec 8 15:38:14.827539125 (72h9m14.69862217s) [noise=true , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hJ5jT6S2d3DJe9DoVDjHMkpwSTpeyTMyNLaK8Pk8SWcUVRKnv): protocol: expected piece hash; context canceled; EOF Dec 8 15:38:14.827545567 (72h9m14.698628612s) [noise=true , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Long tail cancellation due to optimal threshold Dec 8 15:38:14.827903338 (72h9m14.698986373s) [noise=true , address=89.149.201.147:4018 , piece_id=BFJH5ELBTZD767FJNFT2ZBUS3GI62IZIRSKYARLVBCLWZONSURMQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua): protocol: expected piece hash; context canceled; EOF Dec 8 15:38:14.827910673 (72h9m14.698993708s) [noise=true , address=89.149.201.147:4018 , piece_id=BFJH5ELBTZD767FJNFT2ZBUS3GI62IZIRSKYARLVBCLWZONSURMQ, seg_pos={0 0}, upload=63171]: Long tail cancellation due to optimal threshold Dec 8 15:38:14.827928065 (72h9m14.69901111s) [seg_pos={0 0}, upload=63171]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 8 15:38:14.828037083 (72h9m14.699120128s) [upload=63171]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 15:38:14.828053224 (72h9m14.699136269s) [upload=63171]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 15:39:18.272006268 (72h10m18.143089313s) [upload=63171]: Done uploading segments... Dec 8 15:39:18.272087593 (72h10m18.143170628s) [upload=63171]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.679446646s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 1.521720431s trace: https://jaeger.dev.storj.io/trace/3bdb8560e64b821d FAIL (2025-12-08 16:23:24 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 1.52175158s core uplink eu1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.X.X....XX......X....X..........................X...X.X.....] duration: 5.874521751s trace: https://jaeger.dev.storj.io/trace/20c661c94f160c60 FAIL (2025-12-08 16:23:22 UTC) orbiter warning: context deadline exceeded Duration: 5.874550585s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 22 out of 60 is failed (threshold=6) [X...XX.X..XX.XX.X..XX.....X.....X...X..XX..X......X.X.XX...X] duration: 5.002179866s trace: https://jaeger.dev.storj.io/trace/19f01a746f9710cc FAIL (2025-12-08 16:23:14 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002196236s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 5.100296176s trace: https://jaeger.dev.storj.io/trace/5d817b557064ae10 FAIL (2025-12-08 16:23:09 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.100328818s core uplink eu1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [X.X.X....XX......X....X..........................X...X.X....] duration: 20.880495703s trace: https://jaeger.dev.storj.io/trace/20c661c94f160c60 FAIL (2025-12-08 16:22:47 UTC) orbiter warning: context deadline exceeded Duration: 20.880521462s edge gateway us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 5.280565167s trace: https://jaeger.dev.storj.io/trace/5d817b557064ae10 FAIL (2025-12-08 16:22:41 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.2805986s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 5.943225747s trace: https://jaeger.dev.storj.io/trace/4195524620a3e2c2 FAIL (2025-12-08 16:22:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.771167108 (72h9m11.642250153s) [noise=true , address=81.171.14.70:4032 , piece_id=XAMDRY73RIJV4LC2W2FRMH6L4WL3VAFUYKB3QD4TXOQWWZUYGPOQ, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.771462930 (72h9m11.642545976s) [noise=true , address=81.171.14.70:4035 , piece_id=VYAAT75X4R4J22MLY7RK4VHOEFMYZTLWUWTWVAMOOIIDSEXTQV7Q, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.771589993 (72h9m11.642673038s) [noise=true , address=185.167.204.205:4036 , piece_id=X2TWRAEKE6FEH3ITYY74A76ZZQ2KCB63ETZNW5TAHQ3MLEM34PZQ, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.772028248 (72h9m11.643111293s) [noise=true , address=185.167.204.206:4067 , piece_id=3YOG2SUYBHBE6MMIPMQWMBDZTLL7HPSBPEKFHMGOICBTAQU5HA4A, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.772041683 (72h9m11.643124728s) [noise=true , address=185.167.204.206:4069 , piece_id=GH2ZEUIEIUZ6OLM6XH5ZJVXKAMHUKQ6L3GZ7NJLUZFY623YH5GDQ, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.772603803 (72h9m11.643686848s) [noise=true , address=89.149.201.147:4022 , piece_id=CWWNRXZNWQOUTZEA4WPHGA6BYEPO35D2YDU5VSTHFZ7GRC67Y7VQ, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.773136077 (72h9m11.644219122s) [noise=true , address=185.167.204.206:4051 , piece_id=5JMUWS6LLPRENPBAZ74MDELLIZMWTMI3GOZFUWLCOCUNLDTPETWQ, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.773659244 (72h9m11.644742289s) [noise=true , address=185.167.204.206:4062 , piece_id=KYFUGH5EAVFGEJ2J7VDRDWBDP2R2MX4JSCIU7RP3VJOK4LCM6D2Q, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.774276288 (72h9m11.645359333s) [noise=true , address=81.171.14.70:4026 , piece_id=4BLGOA4GSQVOKZ6UBUIXANALBDHDHDXPLDA3TDQ44ACTIC2IHSEA, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.774811648 (72h9m11.645894693s) [noise=true , address=185.167.204.205:4008 , piece_id=JWZAQAYIUNBFRDVQHADARS5NTEWET7UI2GR73ZMIACA5VFUS3D6Q, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:12.600277390 (72h9m12.471360435s) [upload=63171]: Next returned nil segment Dec 8 15:38:12.600300113 (72h9m12.471383158s) [upload=63171]: Waiting for error group managing segments... Dec 8 15:38:12.984985581 (72h9m12.856068647s) [noise=true , address=185.167.204.205:4028 , piece_id=DCGYFXSKGK6GFLLFU7UQONAX643EA6CDTYWEZ42YQPLEERINL3KQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.985372348 (72h9m12.856455383s) [noise=true , address=185.167.204.205:4008 , piece_id=JWZAQAYIUNBFRDVQHADARS5NTEWET7UI2GR73ZMIACA5VFUS3D6Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.993832749 (72h9m12.864915784s) [noise=true , address=185.167.204.205:4015 , piece_id=GK2HV5AGF2Q6RYHDZBZQ3SY537WQXB7E7UF4SWF5X3LFM4PMCZPQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.994210087 (72h9m12.865293122s) [noise=true , address=185.167.204.206:4053 , piece_id=DGSLZPKB474OADK43GBZO2FGAR2F62PBL364EWVNPCZE3PWZGQPA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.994561226 (72h9m12.865644271s) [noise=true , address=185.167.204.205:4036 , piece_id=X2TWRAEKE6FEH3ITYY74A76ZZQ2KCB63ETZNW5TAHQ3MLEM34PZQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.994944356 (72h9m12.866027401s) [noise=true , address=185.167.204.205:4023 , piece_id=5RNGVQ3LUTPKX4RAVK7Y4WRQC2Y6GT5QWV2G3MTY2LW2MU6IMKLQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.995089352 (72h9m12.866172397s) [noise=true , address=185.167.204.205:4007 , piece_id=WEZIGATHDFL7G3S2VG7BEFGGT7JJIXBDWHGIPHBTPSKDJQUTROIQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.995157291 (72h9m12.866240336s) [noise=true , address=185.167.204.205:4033 , piece_id=4V3DXW42ZCPJFF3PPJEBMBYFW354WEZYLVFUSPGEZ4E4HZ7MFI4Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.000801098 (72h9m12.871884163s) [noise=true , address=185.167.204.206:4062 , piece_id=KYFUGH5EAVFGEJ2J7VDRDWBDP2R2MX4JSCIU7RP3VJOK4LCM6D2Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.000862164 (72h9m12.871945209s) [noise=true , address=185.167.204.206:4051 , piece_id=5JMUWS6LLPRENPBAZ74MDELLIZMWTMI3GOZFUWLCOCUNLDTPETWQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.002117705 (72h9m12.87320074s) [noise=true , address=185.167.204.206:4064 , piece_id=LUITKU5LPQ67MFA4VWE6CDV46QOU7RXOFCGKKAYPCLFGWFSLYLTA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.003015134 (72h9m12.874098179s) [noise=true , address=185.167.204.205:4034 , piece_id=OJPEC5SBHQDKJW3LDBPHMEOPDG6NPNNMO3W54C7XIXTNUYBTMUOA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.003432589 (72h9m12.874515624s) [noise=true , address=185.167.204.206:4037 , piece_id=OPF6NTGFZ7FSN7LIEKHOGIJRYUWX27RTI3KKGCKPCOQLK4HC62LQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.004078648 (72h9m12.875161694s) [noise=true , address=185.167.204.206:4069 , piece_id=GH2ZEUIEIUZ6OLM6XH5ZJVXKAMHUKQ6L3GZ7NJLUZFY623YH5GDQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.009788571 (72h9m12.880871616s) [noise=true , address=185.167.204.206:4067 , piece_id=3YOG2SUYBHBE6MMIPMQWMBDZTLL7HPSBPEKFHMGOICBTAQU5HA4A, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.012083561 (72h9m12.883166596s) [noise=true , address=185.167.204.205:4030 , piece_id=EFAUOQCDA25VE55ZHJOSVXTOQD2VTQCW55MEH5DBEAY7N364GMXA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.584995241 (72h9m13.456078286s) [noise=true , address=81.171.14.70:4046 , piece_id=77RPCXTIQW7MO6UKAVJUD4Y7XR674IDZE2TL2SB77LJUHRE3ACDA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.651263008 (72h9m13.522346063s) [noise=true , address=81.171.14.70:4026 , piece_id=4BLGOA4GSQVOKZ6UBUIXANALBDHDHDXPLDA3TDQ44ACTIC2IHSEA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.696266204 (72h9m13.567349249s) [noise=true , address=81.171.14.70:4035 , piece_id=VYAAT75X4R4J22MLY7RK4VHOEFMYZTLWUWTWVAMOOIIDSEXTQV7Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.123999844 (72h9m13.995082889s) [noise=true , address=81.171.14.70:4025 , piece_id=XML4X66PHAT33VTUTEV22HSTIHMG4WFZU6YSDPCZZZ4SEBL3NWUA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.147808461 (72h9m14.018891506s) [noise=true , address=81.171.14.70:4032 , piece_id=XAMDRY73RIJV4LC2W2FRMH6L4WL3VAFUYKB3QD4TXOQWWZUYGPOQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.255119813 (72h9m14.126202858s) [noise=true , address=89.149.201.147:4005 , piece_id=MBR3CYKQ3YSPVXIVSL6ABGZNG7FB5HGP45OA3YE2IRKPJMOSM3KQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.271452473 (72h9m14.142535518s) [noise=true , address=89.149.201.147:4013 , piece_id=KIPMWZVYZADDS2LAOY5ASMIWP32HIRTNGYQ65SNLYR2EQDQCN3HQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.775652542 (72h9m14.646735587s) [noise=true , address=89.149.201.147:4016 , piece_id=5N5HKBPMDQUEQIZZVFFRQGE4LQAGSJ4H6INPDXHYQMDK5YTHUVKQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.785904314 (72h9m14.656987359s) [noise=true , address=89.149.201.147:4017 , piece_id=MBD47F6TJHZ76PAGU7ZIFRQ4M5HUDT5AQULHE6RPQPBUBAVKSTKQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.827349233 (72h9m14.698432278s) [noise=true , address=89.149.201.147:4022 , piece_id=CWWNRXZNWQOUTZEA4WPHGA6BYEPO35D2YDU5VSTHFZ7GRC67Y7VQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.827365925 (72h9m14.69844897s) [seg_pos={0 0}, upload=63171]: Segment reached optimal threshold of 26 pieces. Dec 8 15:38:14.827539125 (72h9m14.69862217s) [noise=true , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hJ5jT6S2d3DJe9DoVDjHMkpwSTpeyTMyNLaK8Pk8SWcUVRKnv): protocol: expected piece hash; context canceled; EOF Dec 8 15:38:14.827545567 (72h9m14.698628612s) [noise=true , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Long tail cancellation due to optimal threshold Dec 8 15:38:14.827903338 (72h9m14.698986373s) [noise=true , address=89.149.201.147:4018 , piece_id=BFJH5ELBTZD767FJNFT2ZBUS3GI62IZIRSKYARLVBCLWZONSURMQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua): protocol: expected piece hash; context canceled; EOF Dec 8 15:38:14.827910673 (72h9m14.698993708s) [noise=true , address=89.149.201.147:4018 , piece_id=BFJH5ELBTZD767FJNFT2ZBUS3GI62IZIRSKYARLVBCLWZONSURMQ, seg_pos={0 0}, upload=63171]: Long tail cancellation due to optimal threshold Dec 8 15:38:14.827928065 (72h9m14.69901111s) [seg_pos={0 0}, upload=63171]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 8 15:38:14.828037083 (72h9m14.699120128s) [upload=63171]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 15:38:14.828053224 (72h9m14.699136269s) [upload=63171]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 15:39:18.272006268 (72h10m18.143089313s) [upload=63171]: Done uploading segments... Dec 8 15:39:18.272087593 (72h10m18.143170628s) [upload=63171]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.943327943s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 1.447192413s trace: https://jaeger.dev.storj.io/trace/3bdb8560e64b821d FAIL (2025-12-08 16:22:36 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 1.447223112s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [........X.............X............................XX.......] duration: 7.247753166s trace: https://jaeger.dev.storj.io/trace/13568c7eb15b2369 FAIL (2025-12-08 16:22:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, Piece ID: U3FCNUY4ASY3ZGX7ZY26GE2NA52GJNKJMW7XKZKALT2MWBCZMHFA) context deadline exceeded; piecestore: (Node ID: 1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, Piece ID: QN7JC2XQK6FKMIYD2WXQ7GMDILEWDUO2HXZLZERVJ24WWIMHFOVA) context deadline exceeded; piecestore: (Node ID: 12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, Piece ID: KRCKO7NXT2XOYK4QHVJK5UIU246TVB45EE7ELVUT365XTJ7RHKJQ) context deadline exceeded; piecestore: (Node ID: 122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, Piece ID: U7SKMZGV33HMSQLQIFIBZ6WNYC7JJ2CENSK5LCW3KVMP53T7467Q) context deadline exceeded; piecestore: (Node ID: 1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, Piece ID: FGYLRBVDMMLYWTTKLC3B2P2MHTECHEBH2DJOOKAHBR6TBAMKFBXA) context deadline exceeded ======LOGS====== Dec 8 16:15:19.690262294 (72h46m19.561345339s) [upload=63694]: Starting upload Dec 8 16:15:19.690357154 (72h46m19.56144022s) [upload=63694]: Done starting upload Dec 8 16:15:19.690425354 (72h46m19.561508409s) [upload=63694]: Uploading segments... Dec 8 16:15:19.690652296 (72h46m19.561735341s) [upload=63694]: Got next segment. Inline: false Dec 8 16:15:19.690693895 (72h46m19.56177694s) [upload=63694]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 16:15:19.690700889 (72h46m19.561783934s) [upload=63694]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 16:15:20.501121517 (72h46m20.372204572s) [seg_pos={0 0}, upload=63694]: Begin upload segment... Dec 8 16:15:20.501188935 (72h46m20.372271981s) [seg_pos={0 0}, upload=63694]: Done begin upload segment. Dec 8 16:15:20.501225655 (72h46m20.3723087s) [noise=true , address=145.79.227.16:4912 , piece_id=NIW3LT24E4LPJ5QYJVQOAR43S23JLDDOXZ6NVOURZNDTCCTLLPJA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.501311268 (72h46m20.372394313s) [noise=true , address=145.79.227.34:4906 , piece_id=AIZQB5ZLXPCLPT5Q7NLNSI3FPBPAMBVUIXKZCTA3ML7L3XP2UY6A, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.501987676 (72h46m20.373070722s) [noise=true , address=145.79.227.17:4971 , piece_id=KRCKO7NXT2XOYK4QHVJK5UIU246TVB45EE7ELVUT365XTJ7RHKJQ, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.502073379 (72h46m20.373156424s) [noise=true , address=145.79.227.38:4979 , piece_id=HCELMHS3GNLCKHFCQU3V45M4OETQZPL5MQJBUOUF5VPA44HXZW2Q, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.502559184 (72h46m20.373642229s) [noise=true , address=145.79.227.33:4870 , piece_id=IK7NDJD5GNFMWB26JYOMCSNIAT5TTTQOUXB2VFYSNRGIQ2MVMXAQ, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.502903350 (72h46m20.373986385s) [noise=true , address=145.79.227.22:4511 , piece_id=RT3ULMIM24LMC4UASFJYVAOHIKJBBGLFYJP3LBJCUL6I532MTSGA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503199825 (72h46m20.37428287s) [noise=true , address=145.79.227.29:4941 , piece_id=GJQXPA7WEZUTVJNELRBJVTU4LIGXYD7XLEPR5SWF4NBWWMXBW2SA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503205956 (72h46m20.374289001s) [noise=true , address=77.90.31.11:4555 , piece_id=LZ4YCICB4QQVLQAUZD6ZFABXLYSRKC4OFZESDVUGIOFJMZBRMMSA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503484487 (72h46m20.374567532s) [noise=true , address=145.79.227.39:4014 , piece_id=U7SKMZGV33HMSQLQIFIBZ6WNYC7JJ2CENSK5LCW3KVMP53T7467Q, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503757176 (72h46m20.374840221s) [noise=true , address=145.79.227.24:4582 , piece_id=FGYLRBVDMMLYWTTKLC3B2P2MHTECHEBH2DJOOKAHBR6TBAMKFBXA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503773727 (72h46m20.374856772s) [noise=true , address=145.79.227.40:4049 , piece_id=VX3PNKMHR75TNJLKRBLYS3KT42ZTPT4X6I5QS2AKNMGDQDVE4WDA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.504114126 (72h46m20.375197161s) [noise=true , address=77.90.31.17:4771 , piece_id=R6PLE6VLZNL23TJC6V6MRBLJP6PKB2DDGQMPPGLMM23WN6J7SH5Q, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.504441298 (72h46m20.375524343s) [noise=true , address=77.90.31.18:4807 , piece_id=7FXKUHBJT7AHDC4PHCZ73Q62GDSOUFRFIVAD337ORGE7YUTAULVA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.505240540 (72h46m20.376323585s) [noise=true , address=77.90.31.14:4662 , piece_id=BO6DQNLFNMG6HAPKOAG2DHWDD3YMX46IAYGDVQG7THC5EVZORJRA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.506022409 (72h46m20.377105454s) [noise=true , address=145.79.227.20:4438 , piece_id=U3FCNUY4ASY3ZGX7ZY26GE2NA52GJNKJMW7XKZKALT2MWBCZMHFA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.506709457 (72h46m20.377792502s) [noise=true , address=145.79.227.43:4158 , piece_id=QN7JC2XQK6FKMIYD2WXQ7GMDILEWDUO2HXZLZERVJ24WWIMHFOVA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:22.339915069 (72h46m22.210998114s) [upload=63694]: Next returned nil segment Dec 8 16:15:22.339932321 (72h46m22.211015366s) [upload=63694]: Waiting for error group managing segments... Dec 8 16:15:22.655133540 (72h46m22.526216575s) [noise=true , address=145.79.227.43:4158 , piece_id=QN7JC2XQK6FKMIYD2WXQ7GMDILEWDUO2HXZLZERVJ24WWIMHFOVA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.660462928 (72h46m22.531545983s) [noise=true , address=145.79.227.17:4971 , piece_id=KRCKO7NXT2XOYK4QHVJK5UIU246TVB45EE7ELVUT365XTJ7RHKJQ, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.683969820 (72h46m22.555052855s) [noise=true , address=77.90.31.11:4555 , piece_id=LZ4YCICB4QQVLQAUZD6ZFABXLYSRKC4OFZESDVUGIOFJMZBRMMSA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.685188641 (72h46m22.556271686s) [noise=true , address=145.79.227.22:4511 , piece_id=RT3ULMIM24LMC4UASFJYVAOHIKJBBGLFYJP3LBJCUL6I532MTSGA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.686713104 (72h46m22.557796149s) [noise=true , address=145.79.227.16:4912 , piece_id=NIW3LT24E4LPJ5QYJVQOAR43S23JLDDOXZ6NVOURZNDTCCTLLPJA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.698627942 (72h46m22.569710987s) [noise=true , address=145.79.227.39:4014 , piece_id=U7SKMZGV33HMSQLQIFIBZ6WNYC7JJ2CENSK5LCW3KVMP53T7467Q, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.713359143 (72h46m22.584442188s) [noise=true , address=145.79.227.40:4049 , piece_id=VX3PNKMHR75TNJLKRBLYS3KT42ZTPT4X6I5QS2AKNMGDQDVE4WDA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.731975212 (72h46m22.603058257s) [noise=true , address=77.90.31.17:4771 , piece_id=R6PLE6VLZNL23TJC6V6MRBLJP6PKB2DDGQMPPGLMM23WN6J7SH5Q, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.741125236 (72h46m22.612208281s) [noise=true , address=145.79.227.20:4438 , piece_id=U3FCNUY4ASY3ZGX7ZY26GE2NA52GJNKJMW7XKZKALT2MWBCZMHFA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.773771311 (72h46m22.644854346s) [noise=true , address=77.90.31.18:4807 , piece_id=7FXKUHBJT7AHDC4PHCZ73Q62GDSOUFRFIVAD337ORGE7YUTAULVA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.906364795 (72h46m22.777447841s) [noise=true , address=77.90.31.14:4662 , piece_id=BO6DQNLFNMG6HAPKOAG2DHWDD3YMX46IAYGDVQG7THC5EVZORJRA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:23.367795705 (72h46m23.23887874s) [noise=true , address=145.79.227.24:4582 , piece_id=FGYLRBVDMMLYWTTKLC3B2P2MHTECHEBH2DJOOKAHBR6TBAMKFBXA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:24.582400944 (72h46m24.453483979s) [noise=true , address=145.79.227.33:4870 , piece_id=IK7NDJD5GNFMWB26JYOMCSNIAT5TTTQOUXB2VFYSNRGIQ2MVMXAQ, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:24.582421023 (72h46m24.453504068s) [seg_pos={0 0}, upload=63694]: Segment reached optimal threshold of 13 pieces. Dec 8 16:15:24.582652363 (72h46m24.453735408s) [noise=true , address=145.79.227.38:4979 , piece_id=HCELMHS3GNLCKHFCQU3V45M4OETQZPL5MQJBUOUF5VPA44HXZW2Q, seg_pos={0 0}, upload=63694]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma): protocol: expected piece hash; context canceled; EOF Dec 8 16:15:24.582659086 (72h46m24.453742121s) [noise=true , address=145.79.227.38:4979 , piece_id=HCELMHS3GNLCKHFCQU3V45M4OETQZPL5MQJBUOUF5VPA44HXZW2Q, seg_pos={0 0}, upload=63694]: Long tail cancellation due to optimal threshold Dec 8 16:15:24.582740842 (72h46m24.453823877s) [noise=true , address=145.79.227.34:4906 , piece_id=AIZQB5ZLXPCLPT5Q7NLNSI3FPBPAMBVUIXKZCTA3ML7L3XP2UY6A, seg_pos={0 0}, upload=63694]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 8 16:15:24.582753055 (72h46m24.45383609s) [noise=true , address=145.79.227.34:4906 , piece_id=AIZQB5ZLXPCLPT5Q7NLNSI3FPBPAMBVUIXKZCTA3ML7L3XP2UY6A, seg_pos={0 0}, upload=63694]: Long tail cancellation due to optimal threshold Dec 8 16:15:24.583044900 (72h46m24.454127935s) [noise=true , address=145.79.227.29:4941 , piece_id=GJQXPA7WEZUTVJNELRBJVTU4LIGXYD7XLEPR5SWF4NBWWMXBW2SA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: context canceled; context canceled Dec 8 16:15:24.583052926 (72h46m24.454135961s) [noise=true , address=145.79.227.29:4941 , piece_id=GJQXPA7WEZUTVJNELRBJVTU4LIGXYD7XLEPR5SWF4NBWWMXBW2SA, seg_pos={0 0}, upload=63694]: Long tail cancellation due to optimal threshold Dec 8 16:15:24.583071441 (72h46m24.454154486s) [seg_pos={0 0}, upload=63694]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 16:15:24.583162014 (72h46m24.454245059s) [upload=63694]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 16:15:24.583171562 (72h46m24.454254627s) [upload=63694]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 16:15:25.042230695 (72h46m24.91331374s) [upload=63694]: Done uploading segments... Dec 8 16:15:25.042258869 (72h46m24.913341914s) [upload=63694]: Upload finished. err: <nil> Duration: 7.247859267s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 21 out of 60 is failed (threshold=6) [.X...XX.X..XX.XX.X..XX.....X.....X...X..XX..X......X.X.XX...] duration: 3.456298837s trace: https://jaeger.dev.storj.io/trace/53840b57e98c88b0 FAIL (2025-12-08 16:22:31 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.456325818s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 2.026664948s trace: https://jaeger.dev.storj.io/trace/4dad09bb52e33232 FAIL (2025-12-08 16:22:25 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 2.026696157s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.....X..X...X.................X.X...........................] duration: 5.773046622s trace: https://jaeger.dev.storj.io/trace/690f79ef204fee2 FAIL (2025-12-08 16:22:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.773075537s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 4.872474218s trace: https://jaeger.dev.storj.io/trace/5d817b557064ae10 FAIL (2025-12-08 16:22:20 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.872499695s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 1.517070805s trace: https://jaeger.dev.storj.io/trace/3bdb8560e64b821d FAIL (2025-12-08 16:22:18 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 1.517143044s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 5.547014388s trace: https://jaeger.dev.storj.io/trace/5d817b557064ae10 FAIL (2025-12-08 16:22:01 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.547044315s core uplink eu1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.X.X.X....XX......X....X..........................X...X.X...] duration: 5.957074559s trace: https://jaeger.dev.storj.io/trace/20c661c94f160c60 FAIL (2025-12-08 16:21:56 UTC) orbiter warning: context deadline exceeded Duration: 5.957095588s edge gateway us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 5.976969202s trace: https://jaeger.dev.storj.io/trace/4195524620a3e2c2 FAIL (2025-12-08 16:21:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.771167108 (72h9m11.642250153s) [noise=true , address=81.171.14.70:4032 , piece_id=XAMDRY73RIJV4LC2W2FRMH6L4WL3VAFUYKB3QD4TXOQWWZUYGPOQ, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.771462930 (72h9m11.642545976s) [noise=true , address=81.171.14.70:4035 , piece_id=VYAAT75X4R4J22MLY7RK4VHOEFMYZTLWUWTWVAMOOIIDSEXTQV7Q, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.771589993 (72h9m11.642673038s) [noise=true , address=185.167.204.205:4036 , piece_id=X2TWRAEKE6FEH3ITYY74A76ZZQ2KCB63ETZNW5TAHQ3MLEM34PZQ, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.772028248 (72h9m11.643111293s) [noise=true , address=185.167.204.206:4067 , piece_id=3YOG2SUYBHBE6MMIPMQWMBDZTLL7HPSBPEKFHMGOICBTAQU5HA4A, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.772041683 (72h9m11.643124728s) [noise=true , address=185.167.204.206:4069 , piece_id=GH2ZEUIEIUZ6OLM6XH5ZJVXKAMHUKQ6L3GZ7NJLUZFY623YH5GDQ, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.772603803 (72h9m11.643686848s) [noise=true , address=89.149.201.147:4022 , piece_id=CWWNRXZNWQOUTZEA4WPHGA6BYEPO35D2YDU5VSTHFZ7GRC67Y7VQ, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.773136077 (72h9m11.644219122s) [noise=true , address=185.167.204.206:4051 , piece_id=5JMUWS6LLPRENPBAZ74MDELLIZMWTMI3GOZFUWLCOCUNLDTPETWQ, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.773659244 (72h9m11.644742289s) [noise=true , address=185.167.204.206:4062 , piece_id=KYFUGH5EAVFGEJ2J7VDRDWBDP2R2MX4JSCIU7RP3VJOK4LCM6D2Q, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.774276288 (72h9m11.645359333s) [noise=true , address=81.171.14.70:4026 , piece_id=4BLGOA4GSQVOKZ6UBUIXANALBDHDHDXPLDA3TDQ44ACTIC2IHSEA, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.774811648 (72h9m11.645894693s) [noise=true , address=185.167.204.205:4008 , piece_id=JWZAQAYIUNBFRDVQHADARS5NTEWET7UI2GR73ZMIACA5VFUS3D6Q, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:12.600277390 (72h9m12.471360435s) [upload=63171]: Next returned nil segment Dec 8 15:38:12.600300113 (72h9m12.471383158s) [upload=63171]: Waiting for error group managing segments... Dec 8 15:38:12.984985581 (72h9m12.856068647s) [noise=true , address=185.167.204.205:4028 , piece_id=DCGYFXSKGK6GFLLFU7UQONAX643EA6CDTYWEZ42YQPLEERINL3KQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.985372348 (72h9m12.856455383s) [noise=true , address=185.167.204.205:4008 , piece_id=JWZAQAYIUNBFRDVQHADARS5NTEWET7UI2GR73ZMIACA5VFUS3D6Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.993832749 (72h9m12.864915784s) [noise=true , address=185.167.204.205:4015 , piece_id=GK2HV5AGF2Q6RYHDZBZQ3SY537WQXB7E7UF4SWF5X3LFM4PMCZPQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.994210087 (72h9m12.865293122s) [noise=true , address=185.167.204.206:4053 , piece_id=DGSLZPKB474OADK43GBZO2FGAR2F62PBL364EWVNPCZE3PWZGQPA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.994561226 (72h9m12.865644271s) [noise=true , address=185.167.204.205:4036 , piece_id=X2TWRAEKE6FEH3ITYY74A76ZZQ2KCB63ETZNW5TAHQ3MLEM34PZQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.994944356 (72h9m12.866027401s) [noise=true , address=185.167.204.205:4023 , piece_id=5RNGVQ3LUTPKX4RAVK7Y4WRQC2Y6GT5QWV2G3MTY2LW2MU6IMKLQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.995089352 (72h9m12.866172397s) [noise=true , address=185.167.204.205:4007 , piece_id=WEZIGATHDFL7G3S2VG7BEFGGT7JJIXBDWHGIPHBTPSKDJQUTROIQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.995157291 (72h9m12.866240336s) [noise=true , address=185.167.204.205:4033 , piece_id=4V3DXW42ZCPJFF3PPJEBMBYFW354WEZYLVFUSPGEZ4E4HZ7MFI4Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.000801098 (72h9m12.871884163s) [noise=true , address=185.167.204.206:4062 , piece_id=KYFUGH5EAVFGEJ2J7VDRDWBDP2R2MX4JSCIU7RP3VJOK4LCM6D2Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.000862164 (72h9m12.871945209s) [noise=true , address=185.167.204.206:4051 , piece_id=5JMUWS6LLPRENPBAZ74MDELLIZMWTMI3GOZFUWLCOCUNLDTPETWQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.002117705 (72h9m12.87320074s) [noise=true , address=185.167.204.206:4064 , piece_id=LUITKU5LPQ67MFA4VWE6CDV46QOU7RXOFCGKKAYPCLFGWFSLYLTA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.003015134 (72h9m12.874098179s) [noise=true , address=185.167.204.205:4034 , piece_id=OJPEC5SBHQDKJW3LDBPHMEOPDG6NPNNMO3W54C7XIXTNUYBTMUOA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.003432589 (72h9m12.874515624s) [noise=true , address=185.167.204.206:4037 , piece_id=OPF6NTGFZ7FSN7LIEKHOGIJRYUWX27RTI3KKGCKPCOQLK4HC62LQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.004078648 (72h9m12.875161694s) [noise=true , address=185.167.204.206:4069 , piece_id=GH2ZEUIEIUZ6OLM6XH5ZJVXKAMHUKQ6L3GZ7NJLUZFY623YH5GDQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.009788571 (72h9m12.880871616s) [noise=true , address=185.167.204.206:4067 , piece_id=3YOG2SUYBHBE6MMIPMQWMBDZTLL7HPSBPEKFHMGOICBTAQU5HA4A, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.012083561 (72h9m12.883166596s) [noise=true , address=185.167.204.205:4030 , piece_id=EFAUOQCDA25VE55ZHJOSVXTOQD2VTQCW55MEH5DBEAY7N364GMXA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.584995241 (72h9m13.456078286s) [noise=true , address=81.171.14.70:4046 , piece_id=77RPCXTIQW7MO6UKAVJUD4Y7XR674IDZE2TL2SB77LJUHRE3ACDA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.651263008 (72h9m13.522346063s) [noise=true , address=81.171.14.70:4026 , piece_id=4BLGOA4GSQVOKZ6UBUIXANALBDHDHDXPLDA3TDQ44ACTIC2IHSEA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.696266204 (72h9m13.567349249s) [noise=true , address=81.171.14.70:4035 , piece_id=VYAAT75X4R4J22MLY7RK4VHOEFMYZTLWUWTWVAMOOIIDSEXTQV7Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.123999844 (72h9m13.995082889s) [noise=true , address=81.171.14.70:4025 , piece_id=XML4X66PHAT33VTUTEV22HSTIHMG4WFZU6YSDPCZZZ4SEBL3NWUA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.147808461 (72h9m14.018891506s) [noise=true , address=81.171.14.70:4032 , piece_id=XAMDRY73RIJV4LC2W2FRMH6L4WL3VAFUYKB3QD4TXOQWWZUYGPOQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.255119813 (72h9m14.126202858s) [noise=true , address=89.149.201.147:4005 , piece_id=MBR3CYKQ3YSPVXIVSL6ABGZNG7FB5HGP45OA3YE2IRKPJMOSM3KQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.271452473 (72h9m14.142535518s) [noise=true , address=89.149.201.147:4013 , piece_id=KIPMWZVYZADDS2LAOY5ASMIWP32HIRTNGYQ65SNLYR2EQDQCN3HQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.775652542 (72h9m14.646735587s) [noise=true , address=89.149.201.147:4016 , piece_id=5N5HKBPMDQUEQIZZVFFRQGE4LQAGSJ4H6INPDXHYQMDK5YTHUVKQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.785904314 (72h9m14.656987359s) [noise=true , address=89.149.201.147:4017 , piece_id=MBD47F6TJHZ76PAGU7ZIFRQ4M5HUDT5AQULHE6RPQPBUBAVKSTKQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.827349233 (72h9m14.698432278s) [noise=true , address=89.149.201.147:4022 , piece_id=CWWNRXZNWQOUTZEA4WPHGA6BYEPO35D2YDU5VSTHFZ7GRC67Y7VQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.827365925 (72h9m14.69844897s) [seg_pos={0 0}, upload=63171]: Segment reached optimal threshold of 26 pieces. Dec 8 15:38:14.827539125 (72h9m14.69862217s) [noise=true , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hJ5jT6S2d3DJe9DoVDjHMkpwSTpeyTMyNLaK8Pk8SWcUVRKnv): protocol: expected piece hash; context canceled; EOF Dec 8 15:38:14.827545567 (72h9m14.698628612s) [noise=true , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Long tail cancellation due to optimal threshold Dec 8 15:38:14.827903338 (72h9m14.698986373s) [noise=true , address=89.149.201.147:4018 , piece_id=BFJH5ELBTZD767FJNFT2ZBUS3GI62IZIRSKYARLVBCLWZONSURMQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua): protocol: expected piece hash; context canceled; EOF Dec 8 15:38:14.827910673 (72h9m14.698993708s) [noise=true , address=89.149.201.147:4018 , piece_id=BFJH5ELBTZD767FJNFT2ZBUS3GI62IZIRSKYARLVBCLWZONSURMQ, seg_pos={0 0}, upload=63171]: Long tail cancellation due to optimal threshold Dec 8 15:38:14.827928065 (72h9m14.69901111s) [seg_pos={0 0}, upload=63171]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 8 15:38:14.828037083 (72h9m14.699120128s) [upload=63171]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 15:38:14.828053224 (72h9m14.699136269s) [upload=63171]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 15:39:18.272006268 (72h10m18.143089313s) [upload=63171]: Done uploading segments... Dec 8 15:39:18.272087593 (72h10m18.143170628s) [upload=63171]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.977001023s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.........X.............X............................XX......] duration: 6.060088961s trace: https://jaeger.dev.storj.io/trace/13568c7eb15b2369 FAIL (2025-12-08 16:21:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, Piece ID: U3FCNUY4ASY3ZGX7ZY26GE2NA52GJNKJMW7XKZKALT2MWBCZMHFA) context deadline exceeded; piecestore: (Node ID: 1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, Piece ID: QN7JC2XQK6FKMIYD2WXQ7GMDILEWDUO2HXZLZERVJ24WWIMHFOVA) context deadline exceeded; piecestore: (Node ID: 12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, Piece ID: KRCKO7NXT2XOYK4QHVJK5UIU246TVB45EE7ELVUT365XTJ7RHKJQ) context deadline exceeded; piecestore: (Node ID: 122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, Piece ID: U7SKMZGV33HMSQLQIFIBZ6WNYC7JJ2CENSK5LCW3KVMP53T7467Q) context deadline exceeded; piecestore: (Node ID: 1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, Piece ID: FGYLRBVDMMLYWTTKLC3B2P2MHTECHEBH2DJOOKAHBR6TBAMKFBXA) context deadline exceeded ======LOGS====== Dec 8 16:15:19.690262294 (72h46m19.561345339s) [upload=63694]: Starting upload Dec 8 16:15:19.690357154 (72h46m19.56144022s) [upload=63694]: Done starting upload Dec 8 16:15:19.690425354 (72h46m19.561508409s) [upload=63694]: Uploading segments... Dec 8 16:15:19.690652296 (72h46m19.561735341s) [upload=63694]: Got next segment. Inline: false Dec 8 16:15:19.690693895 (72h46m19.56177694s) [upload=63694]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 16:15:19.690700889 (72h46m19.561783934s) [upload=63694]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 16:15:20.501121517 (72h46m20.372204572s) [seg_pos={0 0}, upload=63694]: Begin upload segment... Dec 8 16:15:20.501188935 (72h46m20.372271981s) [seg_pos={0 0}, upload=63694]: Done begin upload segment. Dec 8 16:15:20.501225655 (72h46m20.3723087s) [noise=true , address=145.79.227.16:4912 , piece_id=NIW3LT24E4LPJ5QYJVQOAR43S23JLDDOXZ6NVOURZNDTCCTLLPJA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.501311268 (72h46m20.372394313s) [noise=true , address=145.79.227.34:4906 , piece_id=AIZQB5ZLXPCLPT5Q7NLNSI3FPBPAMBVUIXKZCTA3ML7L3XP2UY6A, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.501987676 (72h46m20.373070722s) [noise=true , address=145.79.227.17:4971 , piece_id=KRCKO7NXT2XOYK4QHVJK5UIU246TVB45EE7ELVUT365XTJ7RHKJQ, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.502073379 (72h46m20.373156424s) [noise=true , address=145.79.227.38:4979 , piece_id=HCELMHS3GNLCKHFCQU3V45M4OETQZPL5MQJBUOUF5VPA44HXZW2Q, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.502559184 (72h46m20.373642229s) [noise=true , address=145.79.227.33:4870 , piece_id=IK7NDJD5GNFMWB26JYOMCSNIAT5TTTQOUXB2VFYSNRGIQ2MVMXAQ, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.502903350 (72h46m20.373986385s) [noise=true , address=145.79.227.22:4511 , piece_id=RT3ULMIM24LMC4UASFJYVAOHIKJBBGLFYJP3LBJCUL6I532MTSGA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503199825 (72h46m20.37428287s) [noise=true , address=145.79.227.29:4941 , piece_id=GJQXPA7WEZUTVJNELRBJVTU4LIGXYD7XLEPR5SWF4NBWWMXBW2SA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503205956 (72h46m20.374289001s) [noise=true , address=77.90.31.11:4555 , piece_id=LZ4YCICB4QQVLQAUZD6ZFABXLYSRKC4OFZESDVUGIOFJMZBRMMSA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503484487 (72h46m20.374567532s) [noise=true , address=145.79.227.39:4014 , piece_id=U7SKMZGV33HMSQLQIFIBZ6WNYC7JJ2CENSK5LCW3KVMP53T7467Q, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503757176 (72h46m20.374840221s) [noise=true , address=145.79.227.24:4582 , piece_id=FGYLRBVDMMLYWTTKLC3B2P2MHTECHEBH2DJOOKAHBR6TBAMKFBXA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503773727 (72h46m20.374856772s) [noise=true , address=145.79.227.40:4049 , piece_id=VX3PNKMHR75TNJLKRBLYS3KT42ZTPT4X6I5QS2AKNMGDQDVE4WDA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.504114126 (72h46m20.375197161s) [noise=true , address=77.90.31.17:4771 , piece_id=R6PLE6VLZNL23TJC6V6MRBLJP6PKB2DDGQMPPGLMM23WN6J7SH5Q, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.504441298 (72h46m20.375524343s) [noise=true , address=77.90.31.18:4807 , piece_id=7FXKUHBJT7AHDC4PHCZ73Q62GDSOUFRFIVAD337ORGE7YUTAULVA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.505240540 (72h46m20.376323585s) [noise=true , address=77.90.31.14:4662 , piece_id=BO6DQNLFNMG6HAPKOAG2DHWDD3YMX46IAYGDVQG7THC5EVZORJRA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.506022409 (72h46m20.377105454s) [noise=true , address=145.79.227.20:4438 , piece_id=U3FCNUY4ASY3ZGX7ZY26GE2NA52GJNKJMW7XKZKALT2MWBCZMHFA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.506709457 (72h46m20.377792502s) [noise=true , address=145.79.227.43:4158 , piece_id=QN7JC2XQK6FKMIYD2WXQ7GMDILEWDUO2HXZLZERVJ24WWIMHFOVA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:22.339915069 (72h46m22.210998114s) [upload=63694]: Next returned nil segment Dec 8 16:15:22.339932321 (72h46m22.211015366s) [upload=63694]: Waiting for error group managing segments... Dec 8 16:15:22.655133540 (72h46m22.526216575s) [noise=true , address=145.79.227.43:4158 , piece_id=QN7JC2XQK6FKMIYD2WXQ7GMDILEWDUO2HXZLZERVJ24WWIMHFOVA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.660462928 (72h46m22.531545983s) [noise=true , address=145.79.227.17:4971 , piece_id=KRCKO7NXT2XOYK4QHVJK5UIU246TVB45EE7ELVUT365XTJ7RHKJQ, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.683969820 (72h46m22.555052855s) [noise=true , address=77.90.31.11:4555 , piece_id=LZ4YCICB4QQVLQAUZD6ZFABXLYSRKC4OFZESDVUGIOFJMZBRMMSA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.685188641 (72h46m22.556271686s) [noise=true , address=145.79.227.22:4511 , piece_id=RT3ULMIM24LMC4UASFJYVAOHIKJBBGLFYJP3LBJCUL6I532MTSGA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.686713104 (72h46m22.557796149s) [noise=true , address=145.79.227.16:4912 , piece_id=NIW3LT24E4LPJ5QYJVQOAR43S23JLDDOXZ6NVOURZNDTCCTLLPJA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.698627942 (72h46m22.569710987s) [noise=true , address=145.79.227.39:4014 , piece_id=U7SKMZGV33HMSQLQIFIBZ6WNYC7JJ2CENSK5LCW3KVMP53T7467Q, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.713359143 (72h46m22.584442188s) [noise=true , address=145.79.227.40:4049 , piece_id=VX3PNKMHR75TNJLKRBLYS3KT42ZTPT4X6I5QS2AKNMGDQDVE4WDA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.731975212 (72h46m22.603058257s) [noise=true , address=77.90.31.17:4771 , piece_id=R6PLE6VLZNL23TJC6V6MRBLJP6PKB2DDGQMPPGLMM23WN6J7SH5Q, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.741125236 (72h46m22.612208281s) [noise=true , address=145.79.227.20:4438 , piece_id=U3FCNUY4ASY3ZGX7ZY26GE2NA52GJNKJMW7XKZKALT2MWBCZMHFA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.773771311 (72h46m22.644854346s) [noise=true , address=77.90.31.18:4807 , piece_id=7FXKUHBJT7AHDC4PHCZ73Q62GDSOUFRFIVAD337ORGE7YUTAULVA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.906364795 (72h46m22.777447841s) [noise=true , address=77.90.31.14:4662 , piece_id=BO6DQNLFNMG6HAPKOAG2DHWDD3YMX46IAYGDVQG7THC5EVZORJRA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:23.367795705 (72h46m23.23887874s) [noise=true , address=145.79.227.24:4582 , piece_id=FGYLRBVDMMLYWTTKLC3B2P2MHTECHEBH2DJOOKAHBR6TBAMKFBXA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:24.582400944 (72h46m24.453483979s) [noise=true , address=145.79.227.33:4870 , piece_id=IK7NDJD5GNFMWB26JYOMCSNIAT5TTTQOUXB2VFYSNRGIQ2MVMXAQ, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:24.582421023 (72h46m24.453504068s) [seg_pos={0 0}, upload=63694]: Segment reached optimal threshold of 13 pieces. Dec 8 16:15:24.582652363 (72h46m24.453735408s) [noise=true , address=145.79.227.38:4979 , piece_id=HCELMHS3GNLCKHFCQU3V45M4OETQZPL5MQJBUOUF5VPA44HXZW2Q, seg_pos={0 0}, upload=63694]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma): protocol: expected piece hash; context canceled; EOF Dec 8 16:15:24.582659086 (72h46m24.453742121s) [noise=true , address=145.79.227.38:4979 , piece_id=HCELMHS3GNLCKHFCQU3V45M4OETQZPL5MQJBUOUF5VPA44HXZW2Q, seg_pos={0 0}, upload=63694]: Long tail cancellation due to optimal threshold Dec 8 16:15:24.582740842 (72h46m24.453823877s) [noise=true , address=145.79.227.34:4906 , piece_id=AIZQB5ZLXPCLPT5Q7NLNSI3FPBPAMBVUIXKZCTA3ML7L3XP2UY6A, seg_pos={0 0}, upload=63694]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 8 16:15:24.582753055 (72h46m24.45383609s) [noise=true , address=145.79.227.34:4906 , piece_id=AIZQB5ZLXPCLPT5Q7NLNSI3FPBPAMBVUIXKZCTA3ML7L3XP2UY6A, seg_pos={0 0}, upload=63694]: Long tail cancellation due to optimal threshold Dec 8 16:15:24.583044900 (72h46m24.454127935s) [noise=true , address=145.79.227.29:4941 , piece_id=GJQXPA7WEZUTVJNELRBJVTU4LIGXYD7XLEPR5SWF4NBWWMXBW2SA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: context canceled; context canceled Dec 8 16:15:24.583052926 (72h46m24.454135961s) [noise=true , address=145.79.227.29:4941 , piece_id=GJQXPA7WEZUTVJNELRBJVTU4LIGXYD7XLEPR5SWF4NBWWMXBW2SA, seg_pos={0 0}, upload=63694]: Long tail cancellation due to optimal threshold Dec 8 16:15:24.583071441 (72h46m24.454154486s) [seg_pos={0 0}, upload=63694]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 16:15:24.583162014 (72h46m24.454245059s) [upload=63694]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 16:15:24.583171562 (72h46m24.454254627s) [upload=63694]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 16:15:25.042230695 (72h46m24.91331374s) [upload=63694]: Done uploading segments... Dec 8 16:15:25.042258869 (72h46m24.913341914s) [upload=63694]: Upload finished. err: <nil> Duration: 6.060120592s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 5.416028936s trace: https://jaeger.dev.storj.io/trace/4195524620a3e2c2 FAIL (2025-12-08 16:21:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.771167108 (72h9m11.642250153s) [noise=true , address=81.171.14.70:4032 , piece_id=XAMDRY73RIJV4LC2W2FRMH6L4WL3VAFUYKB3QD4TXOQWWZUYGPOQ, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.771462930 (72h9m11.642545976s) [noise=true , address=81.171.14.70:4035 , piece_id=VYAAT75X4R4J22MLY7RK4VHOEFMYZTLWUWTWVAMOOIIDSEXTQV7Q, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.771589993 (72h9m11.642673038s) [noise=true , address=185.167.204.205:4036 , piece_id=X2TWRAEKE6FEH3ITYY74A76ZZQ2KCB63ETZNW5TAHQ3MLEM34PZQ, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.772028248 (72h9m11.643111293s) [noise=true , address=185.167.204.206:4067 , piece_id=3YOG2SUYBHBE6MMIPMQWMBDZTLL7HPSBPEKFHMGOICBTAQU5HA4A, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.772041683 (72h9m11.643124728s) [noise=true , address=185.167.204.206:4069 , piece_id=GH2ZEUIEIUZ6OLM6XH5ZJVXKAMHUKQ6L3GZ7NJLUZFY623YH5GDQ, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.772603803 (72h9m11.643686848s) [noise=true , address=89.149.201.147:4022 , piece_id=CWWNRXZNWQOUTZEA4WPHGA6BYEPO35D2YDU5VSTHFZ7GRC67Y7VQ, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.773136077 (72h9m11.644219122s) [noise=true , address=185.167.204.206:4051 , piece_id=5JMUWS6LLPRENPBAZ74MDELLIZMWTMI3GOZFUWLCOCUNLDTPETWQ, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.773659244 (72h9m11.644742289s) [noise=true , address=185.167.204.206:4062 , piece_id=KYFUGH5EAVFGEJ2J7VDRDWBDP2R2MX4JSCIU7RP3VJOK4LCM6D2Q, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.774276288 (72h9m11.645359333s) [noise=true , address=81.171.14.70:4026 , piece_id=4BLGOA4GSQVOKZ6UBUIXANALBDHDHDXPLDA3TDQ44ACTIC2IHSEA, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.774811648 (72h9m11.645894693s) [noise=true , address=185.167.204.205:4008 , piece_id=JWZAQAYIUNBFRDVQHADARS5NTEWET7UI2GR73ZMIACA5VFUS3D6Q, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:12.600277390 (72h9m12.471360435s) [upload=63171]: Next returned nil segment Dec 8 15:38:12.600300113 (72h9m12.471383158s) [upload=63171]: Waiting for error group managing segments... Dec 8 15:38:12.984985581 (72h9m12.856068647s) [noise=true , address=185.167.204.205:4028 , piece_id=DCGYFXSKGK6GFLLFU7UQONAX643EA6CDTYWEZ42YQPLEERINL3KQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.985372348 (72h9m12.856455383s) [noise=true , address=185.167.204.205:4008 , piece_id=JWZAQAYIUNBFRDVQHADARS5NTEWET7UI2GR73ZMIACA5VFUS3D6Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.993832749 (72h9m12.864915784s) [noise=true , address=185.167.204.205:4015 , piece_id=GK2HV5AGF2Q6RYHDZBZQ3SY537WQXB7E7UF4SWF5X3LFM4PMCZPQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.994210087 (72h9m12.865293122s) [noise=true , address=185.167.204.206:4053 , piece_id=DGSLZPKB474OADK43GBZO2FGAR2F62PBL364EWVNPCZE3PWZGQPA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.994561226 (72h9m12.865644271s) [noise=true , address=185.167.204.205:4036 , piece_id=X2TWRAEKE6FEH3ITYY74A76ZZQ2KCB63ETZNW5TAHQ3MLEM34PZQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.994944356 (72h9m12.866027401s) [noise=true , address=185.167.204.205:4023 , piece_id=5RNGVQ3LUTPKX4RAVK7Y4WRQC2Y6GT5QWV2G3MTY2LW2MU6IMKLQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.995089352 (72h9m12.866172397s) [noise=true , address=185.167.204.205:4007 , piece_id=WEZIGATHDFL7G3S2VG7BEFGGT7JJIXBDWHGIPHBTPSKDJQUTROIQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.995157291 (72h9m12.866240336s) [noise=true , address=185.167.204.205:4033 , piece_id=4V3DXW42ZCPJFF3PPJEBMBYFW354WEZYLVFUSPGEZ4E4HZ7MFI4Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.000801098 (72h9m12.871884163s) [noise=true , address=185.167.204.206:4062 , piece_id=KYFUGH5EAVFGEJ2J7VDRDWBDP2R2MX4JSCIU7RP3VJOK4LCM6D2Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.000862164 (72h9m12.871945209s) [noise=true , address=185.167.204.206:4051 , piece_id=5JMUWS6LLPRENPBAZ74MDELLIZMWTMI3GOZFUWLCOCUNLDTPETWQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.002117705 (72h9m12.87320074s) [noise=true , address=185.167.204.206:4064 , piece_id=LUITKU5LPQ67MFA4VWE6CDV46QOU7RXOFCGKKAYPCLFGWFSLYLTA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.003015134 (72h9m12.874098179s) [noise=true , address=185.167.204.205:4034 , piece_id=OJPEC5SBHQDKJW3LDBPHMEOPDG6NPNNMO3W54C7XIXTNUYBTMUOA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.003432589 (72h9m12.874515624s) [noise=true , address=185.167.204.206:4037 , piece_id=OPF6NTGFZ7FSN7LIEKHOGIJRYUWX27RTI3KKGCKPCOQLK4HC62LQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.004078648 (72h9m12.875161694s) [noise=true , address=185.167.204.206:4069 , piece_id=GH2ZEUIEIUZ6OLM6XH5ZJVXKAMHUKQ6L3GZ7NJLUZFY623YH5GDQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.009788571 (72h9m12.880871616s) [noise=true , address=185.167.204.206:4067 , piece_id=3YOG2SUYBHBE6MMIPMQWMBDZTLL7HPSBPEKFHMGOICBTAQU5HA4A, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.012083561 (72h9m12.883166596s) [noise=true , address=185.167.204.205:4030 , piece_id=EFAUOQCDA25VE55ZHJOSVXTOQD2VTQCW55MEH5DBEAY7N364GMXA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.584995241 (72h9m13.456078286s) [noise=true , address=81.171.14.70:4046 , piece_id=77RPCXTIQW7MO6UKAVJUD4Y7XR674IDZE2TL2SB77LJUHRE3ACDA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.651263008 (72h9m13.522346063s) [noise=true , address=81.171.14.70:4026 , piece_id=4BLGOA4GSQVOKZ6UBUIXANALBDHDHDXPLDA3TDQ44ACTIC2IHSEA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.696266204 (72h9m13.567349249s) [noise=true , address=81.171.14.70:4035 , piece_id=VYAAT75X4R4J22MLY7RK4VHOEFMYZTLWUWTWVAMOOIIDSEXTQV7Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.123999844 (72h9m13.995082889s) [noise=true , address=81.171.14.70:4025 , piece_id=XML4X66PHAT33VTUTEV22HSTIHMG4WFZU6YSDPCZZZ4SEBL3NWUA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.147808461 (72h9m14.018891506s) [noise=true , address=81.171.14.70:4032 , piece_id=XAMDRY73RIJV4LC2W2FRMH6L4WL3VAFUYKB3QD4TXOQWWZUYGPOQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.255119813 (72h9m14.126202858s) [noise=true , address=89.149.201.147:4005 , piece_id=MBR3CYKQ3YSPVXIVSL6ABGZNG7FB5HGP45OA3YE2IRKPJMOSM3KQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.271452473 (72h9m14.142535518s) [noise=true , address=89.149.201.147:4013 , piece_id=KIPMWZVYZADDS2LAOY5ASMIWP32HIRTNGYQ65SNLYR2EQDQCN3HQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.775652542 (72h9m14.646735587s) [noise=true , address=89.149.201.147:4016 , piece_id=5N5HKBPMDQUEQIZZVFFRQGE4LQAGSJ4H6INPDXHYQMDK5YTHUVKQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.785904314 (72h9m14.656987359s) [noise=true , address=89.149.201.147:4017 , piece_id=MBD47F6TJHZ76PAGU7ZIFRQ4M5HUDT5AQULHE6RPQPBUBAVKSTKQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.827349233 (72h9m14.698432278s) [noise=true , address=89.149.201.147:4022 , piece_id=CWWNRXZNWQOUTZEA4WPHGA6BYEPO35D2YDU5VSTHFZ7GRC67Y7VQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.827365925 (72h9m14.69844897s) [seg_pos={0 0}, upload=63171]: Segment reached optimal threshold of 26 pieces. Dec 8 15:38:14.827539125 (72h9m14.69862217s) [noise=true , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hJ5jT6S2d3DJe9DoVDjHMkpwSTpeyTMyNLaK8Pk8SWcUVRKnv): protocol: expected piece hash; context canceled; EOF Dec 8 15:38:14.827545567 (72h9m14.698628612s) [noise=true , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Long tail cancellation due to optimal threshold Dec 8 15:38:14.827903338 (72h9m14.698986373s) [noise=true , address=89.149.201.147:4018 , piece_id=BFJH5ELBTZD767FJNFT2ZBUS3GI62IZIRSKYARLVBCLWZONSURMQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua): protocol: expected piece hash; context canceled; EOF Dec 8 15:38:14.827910673 (72h9m14.698993708s) [noise=true , address=89.149.201.147:4018 , piece_id=BFJH5ELBTZD767FJNFT2ZBUS3GI62IZIRSKYARLVBCLWZONSURMQ, seg_pos={0 0}, upload=63171]: Long tail cancellation due to optimal threshold Dec 8 15:38:14.827928065 (72h9m14.69901111s) [seg_pos={0 0}, upload=63171]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 8 15:38:14.828037083 (72h9m14.699120128s) [upload=63171]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 15:38:14.828053224 (72h9m14.699136269s) [upload=63171]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 15:39:18.272006268 (72h10m18.143089313s) [upload=63171]: Done uploading segments... Dec 8 15:39:18.272087593 (72h10m18.143170628s) [upload=63171]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.416073241s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [......X..X...X.................X.X..........................] duration: 5.622224378s trace: https://jaeger.dev.storj.io/trace/690f79ef204fee2 FAIL (2025-12-08 16:21:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.62225167s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 22 out of 60 is failed (threshold=6) [X.X...XX.X..XX.XX.X..XX.....X.....X...X..XX..X......X.X.XX..] duration: 3.549082987s trace: https://jaeger.dev.storj.io/trace/53840b57e98c88b0 FAIL (2025-12-08 16:21:19 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.549111432s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 1.492885744s trace: https://jaeger.dev.storj.io/trace/3bdb8560e64b821d FAIL (2025-12-08 16:21:18 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 1.49291538s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 2.03126632s trace: https://jaeger.dev.storj.io/trace/4dad09bb52e33232 FAIL (2025-12-08 16:21:14 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 2.031365961s core uplink us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 5.048199394s trace: https://jaeger.dev.storj.io/trace/5d817b557064ae10 FAIL (2025-12-08 16:21:14 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.048231295s core uplink eu1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [X.X.X.X....XX......X....X..........................X...X.X..] duration: 23.431094172s trace: https://jaeger.dev.storj.io/trace/20c661c94f160c60 FAIL (2025-12-08 16:21:14 UTC) orbiter warning: context deadline exceeded Duration: 23.431347516s edge gateway us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 6.747749595s trace: https://jaeger.dev.storj.io/trace/4195524620a3e2c2 FAIL (2025-12-08 16:21:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.771167108 (72h9m11.642250153s) [noise=true , address=81.171.14.70:4032 , piece_id=XAMDRY73RIJV4LC2W2FRMH6L4WL3VAFUYKB3QD4TXOQWWZUYGPOQ, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.771462930 (72h9m11.642545976s) [noise=true , address=81.171.14.70:4035 , piece_id=VYAAT75X4R4J22MLY7RK4VHOEFMYZTLWUWTWVAMOOIIDSEXTQV7Q, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.771589993 (72h9m11.642673038s) [noise=true , address=185.167.204.205:4036 , piece_id=X2TWRAEKE6FEH3ITYY74A76ZZQ2KCB63ETZNW5TAHQ3MLEM34PZQ, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.772028248 (72h9m11.643111293s) [noise=true , address=185.167.204.206:4067 , piece_id=3YOG2SUYBHBE6MMIPMQWMBDZTLL7HPSBPEKFHMGOICBTAQU5HA4A, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.772041683 (72h9m11.643124728s) [noise=true , address=185.167.204.206:4069 , piece_id=GH2ZEUIEIUZ6OLM6XH5ZJVXKAMHUKQ6L3GZ7NJLUZFY623YH5GDQ, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.772603803 (72h9m11.643686848s) [noise=true , address=89.149.201.147:4022 , piece_id=CWWNRXZNWQOUTZEA4WPHGA6BYEPO35D2YDU5VSTHFZ7GRC67Y7VQ, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.773136077 (72h9m11.644219122s) [noise=true , address=185.167.204.206:4051 , piece_id=5JMUWS6LLPRENPBAZ74MDELLIZMWTMI3GOZFUWLCOCUNLDTPETWQ, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.773659244 (72h9m11.644742289s) [noise=true , address=185.167.204.206:4062 , piece_id=KYFUGH5EAVFGEJ2J7VDRDWBDP2R2MX4JSCIU7RP3VJOK4LCM6D2Q, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.774276288 (72h9m11.645359333s) [noise=true , address=81.171.14.70:4026 , piece_id=4BLGOA4GSQVOKZ6UBUIXANALBDHDHDXPLDA3TDQ44ACTIC2IHSEA, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.774811648 (72h9m11.645894693s) [noise=true , address=185.167.204.205:4008 , piece_id=JWZAQAYIUNBFRDVQHADARS5NTEWET7UI2GR73ZMIACA5VFUS3D6Q, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:12.600277390 (72h9m12.471360435s) [upload=63171]: Next returned nil segment Dec 8 15:38:12.600300113 (72h9m12.471383158s) [upload=63171]: Waiting for error group managing segments... Dec 8 15:38:12.984985581 (72h9m12.856068647s) [noise=true , address=185.167.204.205:4028 , piece_id=DCGYFXSKGK6GFLLFU7UQONAX643EA6CDTYWEZ42YQPLEERINL3KQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.985372348 (72h9m12.856455383s) [noise=true , address=185.167.204.205:4008 , piece_id=JWZAQAYIUNBFRDVQHADARS5NTEWET7UI2GR73ZMIACA5VFUS3D6Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.993832749 (72h9m12.864915784s) [noise=true , address=185.167.204.205:4015 , piece_id=GK2HV5AGF2Q6RYHDZBZQ3SY537WQXB7E7UF4SWF5X3LFM4PMCZPQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.994210087 (72h9m12.865293122s) [noise=true , address=185.167.204.206:4053 , piece_id=DGSLZPKB474OADK43GBZO2FGAR2F62PBL364EWVNPCZE3PWZGQPA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.994561226 (72h9m12.865644271s) [noise=true , address=185.167.204.205:4036 , piece_id=X2TWRAEKE6FEH3ITYY74A76ZZQ2KCB63ETZNW5TAHQ3MLEM34PZQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.994944356 (72h9m12.866027401s) [noise=true , address=185.167.204.205:4023 , piece_id=5RNGVQ3LUTPKX4RAVK7Y4WRQC2Y6GT5QWV2G3MTY2LW2MU6IMKLQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.995089352 (72h9m12.866172397s) [noise=true , address=185.167.204.205:4007 , piece_id=WEZIGATHDFL7G3S2VG7BEFGGT7JJIXBDWHGIPHBTPSKDJQUTROIQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.995157291 (72h9m12.866240336s) [noise=true , address=185.167.204.205:4033 , piece_id=4V3DXW42ZCPJFF3PPJEBMBYFW354WEZYLVFUSPGEZ4E4HZ7MFI4Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.000801098 (72h9m12.871884163s) [noise=true , address=185.167.204.206:4062 , piece_id=KYFUGH5EAVFGEJ2J7VDRDWBDP2R2MX4JSCIU7RP3VJOK4LCM6D2Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.000862164 (72h9m12.871945209s) [noise=true , address=185.167.204.206:4051 , piece_id=5JMUWS6LLPRENPBAZ74MDELLIZMWTMI3GOZFUWLCOCUNLDTPETWQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.002117705 (72h9m12.87320074s) [noise=true , address=185.167.204.206:4064 , piece_id=LUITKU5LPQ67MFA4VWE6CDV46QOU7RXOFCGKKAYPCLFGWFSLYLTA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.003015134 (72h9m12.874098179s) [noise=true , address=185.167.204.205:4034 , piece_id=OJPEC5SBHQDKJW3LDBPHMEOPDG6NPNNMO3W54C7XIXTNUYBTMUOA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.003432589 (72h9m12.874515624s) [noise=true , address=185.167.204.206:4037 , piece_id=OPF6NTGFZ7FSN7LIEKHOGIJRYUWX27RTI3KKGCKPCOQLK4HC62LQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.004078648 (72h9m12.875161694s) [noise=true , address=185.167.204.206:4069 , piece_id=GH2ZEUIEIUZ6OLM6XH5ZJVXKAMHUKQ6L3GZ7NJLUZFY623YH5GDQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.009788571 (72h9m12.880871616s) [noise=true , address=185.167.204.206:4067 , piece_id=3YOG2SUYBHBE6MMIPMQWMBDZTLL7HPSBPEKFHMGOICBTAQU5HA4A, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.012083561 (72h9m12.883166596s) [noise=true , address=185.167.204.205:4030 , piece_id=EFAUOQCDA25VE55ZHJOSVXTOQD2VTQCW55MEH5DBEAY7N364GMXA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.584995241 (72h9m13.456078286s) [noise=true , address=81.171.14.70:4046 , piece_id=77RPCXTIQW7MO6UKAVJUD4Y7XR674IDZE2TL2SB77LJUHRE3ACDA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.651263008 (72h9m13.522346063s) [noise=true , address=81.171.14.70:4026 , piece_id=4BLGOA4GSQVOKZ6UBUIXANALBDHDHDXPLDA3TDQ44ACTIC2IHSEA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.696266204 (72h9m13.567349249s) [noise=true , address=81.171.14.70:4035 , piece_id=VYAAT75X4R4J22MLY7RK4VHOEFMYZTLWUWTWVAMOOIIDSEXTQV7Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.123999844 (72h9m13.995082889s) [noise=true , address=81.171.14.70:4025 , piece_id=XML4X66PHAT33VTUTEV22HSTIHMG4WFZU6YSDPCZZZ4SEBL3NWUA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.147808461 (72h9m14.018891506s) [noise=true , address=81.171.14.70:4032 , piece_id=XAMDRY73RIJV4LC2W2FRMH6L4WL3VAFUYKB3QD4TXOQWWZUYGPOQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.255119813 (72h9m14.126202858s) [noise=true , address=89.149.201.147:4005 , piece_id=MBR3CYKQ3YSPVXIVSL6ABGZNG7FB5HGP45OA3YE2IRKPJMOSM3KQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.271452473 (72h9m14.142535518s) [noise=true , address=89.149.201.147:4013 , piece_id=KIPMWZVYZADDS2LAOY5ASMIWP32HIRTNGYQ65SNLYR2EQDQCN3HQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.775652542 (72h9m14.646735587s) [noise=true , address=89.149.201.147:4016 , piece_id=5N5HKBPMDQUEQIZZVFFRQGE4LQAGSJ4H6INPDXHYQMDK5YTHUVKQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.785904314 (72h9m14.656987359s) [noise=true , address=89.149.201.147:4017 , piece_id=MBD47F6TJHZ76PAGU7ZIFRQ4M5HUDT5AQULHE6RPQPBUBAVKSTKQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.827349233 (72h9m14.698432278s) [noise=true , address=89.149.201.147:4022 , piece_id=CWWNRXZNWQOUTZEA4WPHGA6BYEPO35D2YDU5VSTHFZ7GRC67Y7VQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.827365925 (72h9m14.69844897s) [seg_pos={0 0}, upload=63171]: Segment reached optimal threshold of 26 pieces. Dec 8 15:38:14.827539125 (72h9m14.69862217s) [noise=true , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hJ5jT6S2d3DJe9DoVDjHMkpwSTpeyTMyNLaK8Pk8SWcUVRKnv): protocol: expected piece hash; context canceled; EOF Dec 8 15:38:14.827545567 (72h9m14.698628612s) [noise=true , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Long tail cancellation due to optimal threshold Dec 8 15:38:14.827903338 (72h9m14.698986373s) [noise=true , address=89.149.201.147:4018 , piece_id=BFJH5ELBTZD767FJNFT2ZBUS3GI62IZIRSKYARLVBCLWZONSURMQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua): protocol: expected piece hash; context canceled; EOF Dec 8 15:38:14.827910673 (72h9m14.698993708s) [noise=true , address=89.149.201.147:4018 , piece_id=BFJH5ELBTZD767FJNFT2ZBUS3GI62IZIRSKYARLVBCLWZONSURMQ, seg_pos={0 0}, upload=63171]: Long tail cancellation due to optimal threshold Dec 8 15:38:14.827928065 (72h9m14.69901111s) [seg_pos={0 0}, upload=63171]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 8 15:38:14.828037083 (72h9m14.699120128s) [upload=63171]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 15:38:14.828053224 (72h9m14.699136269s) [upload=63171]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 15:39:18.272006268 (72h10m18.143089313s) [upload=63171]: Done uploading segments... Dec 8 15:39:18.272087593 (72h10m18.143170628s) [upload=63171]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.747791835s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.......X..X...X.................X.X.........................] duration: 18.30984421s trace: https://jaeger.dev.storj.io/trace/690f79ef204fee2 FAIL (2025-12-08 16:20:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 18.309865661s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 5.122107329s trace: https://jaeger.dev.storj.io/trace/5d817b557064ae10 FAIL (2025-12-08 16:20:43 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.122141824s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 1.434703761s trace: https://jaeger.dev.storj.io/trace/3bdb8560e64b821d FAIL (2025-12-08 16:20:43 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 1.434728298s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..........X.............X............................XX.....] duration: 6.779210263s trace: https://jaeger.dev.storj.io/trace/13568c7eb15b2369 FAIL (2025-12-08 16:20:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, Piece ID: U3FCNUY4ASY3ZGX7ZY26GE2NA52GJNKJMW7XKZKALT2MWBCZMHFA) context deadline exceeded; piecestore: (Node ID: 1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, Piece ID: QN7JC2XQK6FKMIYD2WXQ7GMDILEWDUO2HXZLZERVJ24WWIMHFOVA) context deadline exceeded; piecestore: (Node ID: 12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, Piece ID: KRCKO7NXT2XOYK4QHVJK5UIU246TVB45EE7ELVUT365XTJ7RHKJQ) context deadline exceeded; piecestore: (Node ID: 122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, Piece ID: U7SKMZGV33HMSQLQIFIBZ6WNYC7JJ2CENSK5LCW3KVMP53T7467Q) context deadline exceeded; piecestore: (Node ID: 1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, Piece ID: FGYLRBVDMMLYWTTKLC3B2P2MHTECHEBH2DJOOKAHBR6TBAMKFBXA) context deadline exceeded ======LOGS====== Dec 8 16:15:19.690262294 (72h46m19.561345339s) [upload=63694]: Starting upload Dec 8 16:15:19.690357154 (72h46m19.56144022s) [upload=63694]: Done starting upload Dec 8 16:15:19.690425354 (72h46m19.561508409s) [upload=63694]: Uploading segments... Dec 8 16:15:19.690652296 (72h46m19.561735341s) [upload=63694]: Got next segment. Inline: false Dec 8 16:15:19.690693895 (72h46m19.56177694s) [upload=63694]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 16:15:19.690700889 (72h46m19.561783934s) [upload=63694]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 16:15:20.501121517 (72h46m20.372204572s) [seg_pos={0 0}, upload=63694]: Begin upload segment... Dec 8 16:15:20.501188935 (72h46m20.372271981s) [seg_pos={0 0}, upload=63694]: Done begin upload segment. Dec 8 16:15:20.501225655 (72h46m20.3723087s) [noise=true , address=145.79.227.16:4912 , piece_id=NIW3LT24E4LPJ5QYJVQOAR43S23JLDDOXZ6NVOURZNDTCCTLLPJA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.501311268 (72h46m20.372394313s) [noise=true , address=145.79.227.34:4906 , piece_id=AIZQB5ZLXPCLPT5Q7NLNSI3FPBPAMBVUIXKZCTA3ML7L3XP2UY6A, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.501987676 (72h46m20.373070722s) [noise=true , address=145.79.227.17:4971 , piece_id=KRCKO7NXT2XOYK4QHVJK5UIU246TVB45EE7ELVUT365XTJ7RHKJQ, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.502073379 (72h46m20.373156424s) [noise=true , address=145.79.227.38:4979 , piece_id=HCELMHS3GNLCKHFCQU3V45M4OETQZPL5MQJBUOUF5VPA44HXZW2Q, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.502559184 (72h46m20.373642229s) [noise=true , address=145.79.227.33:4870 , piece_id=IK7NDJD5GNFMWB26JYOMCSNIAT5TTTQOUXB2VFYSNRGIQ2MVMXAQ, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.502903350 (72h46m20.373986385s) [noise=true , address=145.79.227.22:4511 , piece_id=RT3ULMIM24LMC4UASFJYVAOHIKJBBGLFYJP3LBJCUL6I532MTSGA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503199825 (72h46m20.37428287s) [noise=true , address=145.79.227.29:4941 , piece_id=GJQXPA7WEZUTVJNELRBJVTU4LIGXYD7XLEPR5SWF4NBWWMXBW2SA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503205956 (72h46m20.374289001s) [noise=true , address=77.90.31.11:4555 , piece_id=LZ4YCICB4QQVLQAUZD6ZFABXLYSRKC4OFZESDVUGIOFJMZBRMMSA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503484487 (72h46m20.374567532s) [noise=true , address=145.79.227.39:4014 , piece_id=U7SKMZGV33HMSQLQIFIBZ6WNYC7JJ2CENSK5LCW3KVMP53T7467Q, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503757176 (72h46m20.374840221s) [noise=true , address=145.79.227.24:4582 , piece_id=FGYLRBVDMMLYWTTKLC3B2P2MHTECHEBH2DJOOKAHBR6TBAMKFBXA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503773727 (72h46m20.374856772s) [noise=true , address=145.79.227.40:4049 , piece_id=VX3PNKMHR75TNJLKRBLYS3KT42ZTPT4X6I5QS2AKNMGDQDVE4WDA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.504114126 (72h46m20.375197161s) [noise=true , address=77.90.31.17:4771 , piece_id=R6PLE6VLZNL23TJC6V6MRBLJP6PKB2DDGQMPPGLMM23WN6J7SH5Q, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.504441298 (72h46m20.375524343s) [noise=true , address=77.90.31.18:4807 , piece_id=7FXKUHBJT7AHDC4PHCZ73Q62GDSOUFRFIVAD337ORGE7YUTAULVA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.505240540 (72h46m20.376323585s) [noise=true , address=77.90.31.14:4662 , piece_id=BO6DQNLFNMG6HAPKOAG2DHWDD3YMX46IAYGDVQG7THC5EVZORJRA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.506022409 (72h46m20.377105454s) [noise=true , address=145.79.227.20:4438 , piece_id=U3FCNUY4ASY3ZGX7ZY26GE2NA52GJNKJMW7XKZKALT2MWBCZMHFA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.506709457 (72h46m20.377792502s) [noise=true , address=145.79.227.43:4158 , piece_id=QN7JC2XQK6FKMIYD2WXQ7GMDILEWDUO2HXZLZERVJ24WWIMHFOVA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:22.339915069 (72h46m22.210998114s) [upload=63694]: Next returned nil segment Dec 8 16:15:22.339932321 (72h46m22.211015366s) [upload=63694]: Waiting for error group managing segments... Dec 8 16:15:22.655133540 (72h46m22.526216575s) [noise=true , address=145.79.227.43:4158 , piece_id=QN7JC2XQK6FKMIYD2WXQ7GMDILEWDUO2HXZLZERVJ24WWIMHFOVA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.660462928 (72h46m22.531545983s) [noise=true , address=145.79.227.17:4971 , piece_id=KRCKO7NXT2XOYK4QHVJK5UIU246TVB45EE7ELVUT365XTJ7RHKJQ, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.683969820 (72h46m22.555052855s) [noise=true , address=77.90.31.11:4555 , piece_id=LZ4YCICB4QQVLQAUZD6ZFABXLYSRKC4OFZESDVUGIOFJMZBRMMSA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.685188641 (72h46m22.556271686s) [noise=true , address=145.79.227.22:4511 , piece_id=RT3ULMIM24LMC4UASFJYVAOHIKJBBGLFYJP3LBJCUL6I532MTSGA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.686713104 (72h46m22.557796149s) [noise=true , address=145.79.227.16:4912 , piece_id=NIW3LT24E4LPJ5QYJVQOAR43S23JLDDOXZ6NVOURZNDTCCTLLPJA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.698627942 (72h46m22.569710987s) [noise=true , address=145.79.227.39:4014 , piece_id=U7SKMZGV33HMSQLQIFIBZ6WNYC7JJ2CENSK5LCW3KVMP53T7467Q, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.713359143 (72h46m22.584442188s) [noise=true , address=145.79.227.40:4049 , piece_id=VX3PNKMHR75TNJLKRBLYS3KT42ZTPT4X6I5QS2AKNMGDQDVE4WDA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.731975212 (72h46m22.603058257s) [noise=true , address=77.90.31.17:4771 , piece_id=R6PLE6VLZNL23TJC6V6MRBLJP6PKB2DDGQMPPGLMM23WN6J7SH5Q, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.741125236 (72h46m22.612208281s) [noise=true , address=145.79.227.20:4438 , piece_id=U3FCNUY4ASY3ZGX7ZY26GE2NA52GJNKJMW7XKZKALT2MWBCZMHFA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.773771311 (72h46m22.644854346s) [noise=true , address=77.90.31.18:4807 , piece_id=7FXKUHBJT7AHDC4PHCZ73Q62GDSOUFRFIVAD337ORGE7YUTAULVA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.906364795 (72h46m22.777447841s) [noise=true , address=77.90.31.14:4662 , piece_id=BO6DQNLFNMG6HAPKOAG2DHWDD3YMX46IAYGDVQG7THC5EVZORJRA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:23.367795705 (72h46m23.23887874s) [noise=true , address=145.79.227.24:4582 , piece_id=FGYLRBVDMMLYWTTKLC3B2P2MHTECHEBH2DJOOKAHBR6TBAMKFBXA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:24.582400944 (72h46m24.453483979s) [noise=true , address=145.79.227.33:4870 , piece_id=IK7NDJD5GNFMWB26JYOMCSNIAT5TTTQOUXB2VFYSNRGIQ2MVMXAQ, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:24.582421023 (72h46m24.453504068s) [seg_pos={0 0}, upload=63694]: Segment reached optimal threshold of 13 pieces. Dec 8 16:15:24.582652363 (72h46m24.453735408s) [noise=true , address=145.79.227.38:4979 , piece_id=HCELMHS3GNLCKHFCQU3V45M4OETQZPL5MQJBUOUF5VPA44HXZW2Q, seg_pos={0 0}, upload=63694]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma): protocol: expected piece hash; context canceled; EOF Dec 8 16:15:24.582659086 (72h46m24.453742121s) [noise=true , address=145.79.227.38:4979 , piece_id=HCELMHS3GNLCKHFCQU3V45M4OETQZPL5MQJBUOUF5VPA44HXZW2Q, seg_pos={0 0}, upload=63694]: Long tail cancellation due to optimal threshold Dec 8 16:15:24.582740842 (72h46m24.453823877s) [noise=true , address=145.79.227.34:4906 , piece_id=AIZQB5ZLXPCLPT5Q7NLNSI3FPBPAMBVUIXKZCTA3ML7L3XP2UY6A, seg_pos={0 0}, upload=63694]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 8 16:15:24.582753055 (72h46m24.45383609s) [noise=true , address=145.79.227.34:4906 , piece_id=AIZQB5ZLXPCLPT5Q7NLNSI3FPBPAMBVUIXKZCTA3ML7L3XP2UY6A, seg_pos={0 0}, upload=63694]: Long tail cancellation due to optimal threshold Dec 8 16:15:24.583044900 (72h46m24.454127935s) [noise=true , address=145.79.227.29:4941 , piece_id=GJQXPA7WEZUTVJNELRBJVTU4LIGXYD7XLEPR5SWF4NBWWMXBW2SA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: context canceled; context canceled Dec 8 16:15:24.583052926 (72h46m24.454135961s) [noise=true , address=145.79.227.29:4941 , piece_id=GJQXPA7WEZUTVJNELRBJVTU4LIGXYD7XLEPR5SWF4NBWWMXBW2SA, seg_pos={0 0}, upload=63694]: Long tail cancellation due to optimal threshold Dec 8 16:15:24.583071441 (72h46m24.454154486s) [seg_pos={0 0}, upload=63694]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 16:15:24.583162014 (72h46m24.454245059s) [upload=63694]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 16:15:24.583171562 (72h46m24.454254627s) [upload=63694]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 16:15:25.042230695 (72h46m24.91331374s) [upload=63694]: Done uploading segments... Dec 8 16:15:25.042258869 (72h46m24.913341914s) [upload=63694]: Upload finished. err: <nil> Duration: 6.779236973s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 23 out of 60 is failed (threshold=6) [XX.X...XX.X..XX.XX.X..XX.....X.....X...X..XX..X......X.X.XX.] duration: 3.44960606s trace: https://jaeger.dev.storj.io/trace/53840b57e98c88b0 FAIL (2025-12-08 16:20:35 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.449630848s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.X.X.X.X....XX......X....X..........................X...X.X.] duration: 7.238035621s trace: https://jaeger.dev.storj.io/trace/20c661c94f160c60 FAIL (2025-12-08 16:20:32 UTC) orbiter warning: context deadline exceeded Duration: 7.238063485s edge gateway us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 1.533170091s trace: https://jaeger.dev.storj.io/trace/3bdb8560e64b821d FAIL (2025-12-08 16:20:26 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 1.533199056s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 2.081012389s trace: https://jaeger.dev.storj.io/trace/4dad09bb52e33232 FAIL (2025-12-08 16:20:16 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 2.081053597s core uplink us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 7.070120196s trace: https://jaeger.dev.storj.io/trace/4195524620a3e2c2 FAIL (2025-12-08 16:20:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.771167108 (72h9m11.642250153s) [noise=true , address=81.171.14.70:4032 , piece_id=XAMDRY73RIJV4LC2W2FRMH6L4WL3VAFUYKB3QD4TXOQWWZUYGPOQ, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.771462930 (72h9m11.642545976s) [noise=true , address=81.171.14.70:4035 , piece_id=VYAAT75X4R4J22MLY7RK4VHOEFMYZTLWUWTWVAMOOIIDSEXTQV7Q, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.771589993 (72h9m11.642673038s) [noise=true , address=185.167.204.205:4036 , piece_id=X2TWRAEKE6FEH3ITYY74A76ZZQ2KCB63ETZNW5TAHQ3MLEM34PZQ, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.772028248 (72h9m11.643111293s) [noise=true , address=185.167.204.206:4067 , piece_id=3YOG2SUYBHBE6MMIPMQWMBDZTLL7HPSBPEKFHMGOICBTAQU5HA4A, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.772041683 (72h9m11.643124728s) [noise=true , address=185.167.204.206:4069 , piece_id=GH2ZEUIEIUZ6OLM6XH5ZJVXKAMHUKQ6L3GZ7NJLUZFY623YH5GDQ, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.772603803 (72h9m11.643686848s) [noise=true , address=89.149.201.147:4022 , piece_id=CWWNRXZNWQOUTZEA4WPHGA6BYEPO35D2YDU5VSTHFZ7GRC67Y7VQ, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.773136077 (72h9m11.644219122s) [noise=true , address=185.167.204.206:4051 , piece_id=5JMUWS6LLPRENPBAZ74MDELLIZMWTMI3GOZFUWLCOCUNLDTPETWQ, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.773659244 (72h9m11.644742289s) [noise=true , address=185.167.204.206:4062 , piece_id=KYFUGH5EAVFGEJ2J7VDRDWBDP2R2MX4JSCIU7RP3VJOK4LCM6D2Q, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.774276288 (72h9m11.645359333s) [noise=true , address=81.171.14.70:4026 , piece_id=4BLGOA4GSQVOKZ6UBUIXANALBDHDHDXPLDA3TDQ44ACTIC2IHSEA, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.774811648 (72h9m11.645894693s) [noise=true , address=185.167.204.205:4008 , piece_id=JWZAQAYIUNBFRDVQHADARS5NTEWET7UI2GR73ZMIACA5VFUS3D6Q, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:12.600277390 (72h9m12.471360435s) [upload=63171]: Next returned nil segment Dec 8 15:38:12.600300113 (72h9m12.471383158s) [upload=63171]: Waiting for error group managing segments... Dec 8 15:38:12.984985581 (72h9m12.856068647s) [noise=true , address=185.167.204.205:4028 , piece_id=DCGYFXSKGK6GFLLFU7UQONAX643EA6CDTYWEZ42YQPLEERINL3KQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.985372348 (72h9m12.856455383s) [noise=true , address=185.167.204.205:4008 , piece_id=JWZAQAYIUNBFRDVQHADARS5NTEWET7UI2GR73ZMIACA5VFUS3D6Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.993832749 (72h9m12.864915784s) [noise=true , address=185.167.204.205:4015 , piece_id=GK2HV5AGF2Q6RYHDZBZQ3SY537WQXB7E7UF4SWF5X3LFM4PMCZPQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.994210087 (72h9m12.865293122s) [noise=true , address=185.167.204.206:4053 , piece_id=DGSLZPKB474OADK43GBZO2FGAR2F62PBL364EWVNPCZE3PWZGQPA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.994561226 (72h9m12.865644271s) [noise=true , address=185.167.204.205:4036 , piece_id=X2TWRAEKE6FEH3ITYY74A76ZZQ2KCB63ETZNW5TAHQ3MLEM34PZQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.994944356 (72h9m12.866027401s) [noise=true , address=185.167.204.205:4023 , piece_id=5RNGVQ3LUTPKX4RAVK7Y4WRQC2Y6GT5QWV2G3MTY2LW2MU6IMKLQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.995089352 (72h9m12.866172397s) [noise=true , address=185.167.204.205:4007 , piece_id=WEZIGATHDFL7G3S2VG7BEFGGT7JJIXBDWHGIPHBTPSKDJQUTROIQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.995157291 (72h9m12.866240336s) [noise=true , address=185.167.204.205:4033 , piece_id=4V3DXW42ZCPJFF3PPJEBMBYFW354WEZYLVFUSPGEZ4E4HZ7MFI4Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.000801098 (72h9m12.871884163s) [noise=true , address=185.167.204.206:4062 , piece_id=KYFUGH5EAVFGEJ2J7VDRDWBDP2R2MX4JSCIU7RP3VJOK4LCM6D2Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.000862164 (72h9m12.871945209s) [noise=true , address=185.167.204.206:4051 , piece_id=5JMUWS6LLPRENPBAZ74MDELLIZMWTMI3GOZFUWLCOCUNLDTPETWQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.002117705 (72h9m12.87320074s) [noise=true , address=185.167.204.206:4064 , piece_id=LUITKU5LPQ67MFA4VWE6CDV46QOU7RXOFCGKKAYPCLFGWFSLYLTA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.003015134 (72h9m12.874098179s) [noise=true , address=185.167.204.205:4034 , piece_id=OJPEC5SBHQDKJW3LDBPHMEOPDG6NPNNMO3W54C7XIXTNUYBTMUOA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.003432589 (72h9m12.874515624s) [noise=true , address=185.167.204.206:4037 , piece_id=OPF6NTGFZ7FSN7LIEKHOGIJRYUWX27RTI3KKGCKPCOQLK4HC62LQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.004078648 (72h9m12.875161694s) [noise=true , address=185.167.204.206:4069 , piece_id=GH2ZEUIEIUZ6OLM6XH5ZJVXKAMHUKQ6L3GZ7NJLUZFY623YH5GDQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.009788571 (72h9m12.880871616s) [noise=true , address=185.167.204.206:4067 , piece_id=3YOG2SUYBHBE6MMIPMQWMBDZTLL7HPSBPEKFHMGOICBTAQU5HA4A, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.012083561 (72h9m12.883166596s) [noise=true , address=185.167.204.205:4030 , piece_id=EFAUOQCDA25VE55ZHJOSVXTOQD2VTQCW55MEH5DBEAY7N364GMXA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.584995241 (72h9m13.456078286s) [noise=true , address=81.171.14.70:4046 , piece_id=77RPCXTIQW7MO6UKAVJUD4Y7XR674IDZE2TL2SB77LJUHRE3ACDA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.651263008 (72h9m13.522346063s) [noise=true , address=81.171.14.70:4026 , piece_id=4BLGOA4GSQVOKZ6UBUIXANALBDHDHDXPLDA3TDQ44ACTIC2IHSEA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.696266204 (72h9m13.567349249s) [noise=true , address=81.171.14.70:4035 , piece_id=VYAAT75X4R4J22MLY7RK4VHOEFMYZTLWUWTWVAMOOIIDSEXTQV7Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.123999844 (72h9m13.995082889s) [noise=true , address=81.171.14.70:4025 , piece_id=XML4X66PHAT33VTUTEV22HSTIHMG4WFZU6YSDPCZZZ4SEBL3NWUA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.147808461 (72h9m14.018891506s) [noise=true , address=81.171.14.70:4032 , piece_id=XAMDRY73RIJV4LC2W2FRMH6L4WL3VAFUYKB3QD4TXOQWWZUYGPOQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.255119813 (72h9m14.126202858s) [noise=true , address=89.149.201.147:4005 , piece_id=MBR3CYKQ3YSPVXIVSL6ABGZNG7FB5HGP45OA3YE2IRKPJMOSM3KQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.271452473 (72h9m14.142535518s) [noise=true , address=89.149.201.147:4013 , piece_id=KIPMWZVYZADDS2LAOY5ASMIWP32HIRTNGYQ65SNLYR2EQDQCN3HQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.775652542 (72h9m14.646735587s) [noise=true , address=89.149.201.147:4016 , piece_id=5N5HKBPMDQUEQIZZVFFRQGE4LQAGSJ4H6INPDXHYQMDK5YTHUVKQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.785904314 (72h9m14.656987359s) [noise=true , address=89.149.201.147:4017 , piece_id=MBD47F6TJHZ76PAGU7ZIFRQ4M5HUDT5AQULHE6RPQPBUBAVKSTKQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.827349233 (72h9m14.698432278s) [noise=true , address=89.149.201.147:4022 , piece_id=CWWNRXZNWQOUTZEA4WPHGA6BYEPO35D2YDU5VSTHFZ7GRC67Y7VQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.827365925 (72h9m14.69844897s) [seg_pos={0 0}, upload=63171]: Segment reached optimal threshold of 26 pieces. Dec 8 15:38:14.827539125 (72h9m14.69862217s) [noise=true , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hJ5jT6S2d3DJe9DoVDjHMkpwSTpeyTMyNLaK8Pk8SWcUVRKnv): protocol: expected piece hash; context canceled; EOF Dec 8 15:38:14.827545567 (72h9m14.698628612s) [noise=true , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Long tail cancellation due to optimal threshold Dec 8 15:38:14.827903338 (72h9m14.698986373s) [noise=true , address=89.149.201.147:4018 , piece_id=BFJH5ELBTZD767FJNFT2ZBUS3GI62IZIRSKYARLVBCLWZONSURMQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua): protocol: expected piece hash; context canceled; EOF Dec 8 15:38:14.827910673 (72h9m14.698993708s) [noise=true , address=89.149.201.147:4018 , piece_id=BFJH5ELBTZD767FJNFT2ZBUS3GI62IZIRSKYARLVBCLWZONSURMQ, seg_pos={0 0}, upload=63171]: Long tail cancellation due to optimal threshold Dec 8 15:38:14.827928065 (72h9m14.69901111s) [seg_pos={0 0}, upload=63171]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 8 15:38:14.828037083 (72h9m14.699120128s) [upload=63171]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 15:38:14.828053224 (72h9m14.699136269s) [upload=63171]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 15:39:18.272006268 (72h10m18.143089313s) [upload=63171]: Done uploading segments... Dec 8 15:39:18.272087593 (72h10m18.143170628s) [upload=63171]: Upload finished. err: metaclient: context deadline exceeded Duration: 7.070145845s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 5.332618809s trace: https://jaeger.dev.storj.io/trace/5d817b557064ae10 FAIL (2025-12-08 16:19:50 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.332650569s core uplink eu1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [..X.X.X.X....XX......X....X..........................X...X.X] duration: 30.000875505s trace: https://jaeger.dev.storj.io/trace/20c661c94f160c60 FAIL (2025-12-08 16:19:47 UTC) orbiter warning: context deadline exceeded Duration: 30.000896195s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 1.98450795s trace: https://jaeger.dev.storj.io/trace/4dad09bb52e33232 FAIL (2025-12-08 16:19:42 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.984534802s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [........X..X...X.................X.X........................] duration: 5.932335142s trace: https://jaeger.dev.storj.io/trace/690f79ef204fee2 FAIL (2025-12-08 16:19:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.932355961s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)