Orbiter History

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 4.957673203s trace: https://jaeger.dev.storj.io/trace/5d817b557064ae10 FAIL (2025-12-08 16:17:55 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.957702999s core uplink eu1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [X...X.X.X.X....XX......X....X..........................X...X] duration: 30.001204661s trace: https://jaeger.dev.storj.io/trace/45d30210edb12b40 FAIL (2025-12-08 16:17:51 UTC) orbiter warning: context deadline exceeded Duration: 30.001226223s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 23 out of 60 is failed (threshold=6) [.XXXX.X...XX.X..XX.XX.X..XX.....X.....X...X..XX..X......X.X.] duration: 3.452710475s trace: https://jaeger.dev.storj.io/trace/470fdd80045558be FAIL (2025-12-08 16:17:47 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.452748509s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [..........X..X...X.................X.X......................] duration: 5.810623824s trace: https://jaeger.dev.storj.io/trace/690f79ef204fee2 FAIL (2025-12-08 16:17:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.810639143s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 1.447325918s trace: https://jaeger.dev.storj.io/trace/3bdb8560e64b821d FAIL (2025-12-08 16:17:41 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 1.447353901s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 5.716753697s trace: https://jaeger.dev.storj.io/trace/5d817b557064ae10 FAIL (2025-12-08 16:17:36 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.716782292s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 2.031942339s trace: https://jaeger.dev.storj.io/trace/4dad09bb52e33232 FAIL (2025-12-08 16:17:24 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 2.031972725s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.............X.............X............................XX..] duration: 6.4545682s trace: https://jaeger.dev.storj.io/trace/13568c7eb15b2369 FAIL (2025-12-08 16:17:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, Piece ID: U3FCNUY4ASY3ZGX7ZY26GE2NA52GJNKJMW7XKZKALT2MWBCZMHFA) context deadline exceeded; piecestore: (Node ID: 1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, Piece ID: QN7JC2XQK6FKMIYD2WXQ7GMDILEWDUO2HXZLZERVJ24WWIMHFOVA) context deadline exceeded; piecestore: (Node ID: 12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, Piece ID: KRCKO7NXT2XOYK4QHVJK5UIU246TVB45EE7ELVUT365XTJ7RHKJQ) context deadline exceeded; piecestore: (Node ID: 122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, Piece ID: U7SKMZGV33HMSQLQIFIBZ6WNYC7JJ2CENSK5LCW3KVMP53T7467Q) context deadline exceeded; piecestore: (Node ID: 1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, Piece ID: FGYLRBVDMMLYWTTKLC3B2P2MHTECHEBH2DJOOKAHBR6TBAMKFBXA) context deadline exceeded ======LOGS====== Dec 8 16:15:19.690262294 (72h46m19.561345339s) [upload=63694]: Starting upload Dec 8 16:15:19.690357154 (72h46m19.56144022s) [upload=63694]: Done starting upload Dec 8 16:15:19.690425354 (72h46m19.561508409s) [upload=63694]: Uploading segments... Dec 8 16:15:19.690652296 (72h46m19.561735341s) [upload=63694]: Got next segment. Inline: false Dec 8 16:15:19.690693895 (72h46m19.56177694s) [upload=63694]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 16:15:19.690700889 (72h46m19.561783934s) [upload=63694]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 16:15:20.501121517 (72h46m20.372204572s) [seg_pos={0 0}, upload=63694]: Begin upload segment... Dec 8 16:15:20.501188935 (72h46m20.372271981s) [seg_pos={0 0}, upload=63694]: Done begin upload segment. Dec 8 16:15:20.501225655 (72h46m20.3723087s) [noise=true , address=145.79.227.16:4912 , piece_id=NIW3LT24E4LPJ5QYJVQOAR43S23JLDDOXZ6NVOURZNDTCCTLLPJA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.501311268 (72h46m20.372394313s) [noise=true , address=145.79.227.34:4906 , piece_id=AIZQB5ZLXPCLPT5Q7NLNSI3FPBPAMBVUIXKZCTA3ML7L3XP2UY6A, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.501987676 (72h46m20.373070722s) [noise=true , address=145.79.227.17:4971 , piece_id=KRCKO7NXT2XOYK4QHVJK5UIU246TVB45EE7ELVUT365XTJ7RHKJQ, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.502073379 (72h46m20.373156424s) [noise=true , address=145.79.227.38:4979 , piece_id=HCELMHS3GNLCKHFCQU3V45M4OETQZPL5MQJBUOUF5VPA44HXZW2Q, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.502559184 (72h46m20.373642229s) [noise=true , address=145.79.227.33:4870 , piece_id=IK7NDJD5GNFMWB26JYOMCSNIAT5TTTQOUXB2VFYSNRGIQ2MVMXAQ, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.502903350 (72h46m20.373986385s) [noise=true , address=145.79.227.22:4511 , piece_id=RT3ULMIM24LMC4UASFJYVAOHIKJBBGLFYJP3LBJCUL6I532MTSGA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503199825 (72h46m20.37428287s) [noise=true , address=145.79.227.29:4941 , piece_id=GJQXPA7WEZUTVJNELRBJVTU4LIGXYD7XLEPR5SWF4NBWWMXBW2SA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503205956 (72h46m20.374289001s) [noise=true , address=77.90.31.11:4555 , piece_id=LZ4YCICB4QQVLQAUZD6ZFABXLYSRKC4OFZESDVUGIOFJMZBRMMSA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503484487 (72h46m20.374567532s) [noise=true , address=145.79.227.39:4014 , piece_id=U7SKMZGV33HMSQLQIFIBZ6WNYC7JJ2CENSK5LCW3KVMP53T7467Q, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503757176 (72h46m20.374840221s) [noise=true , address=145.79.227.24:4582 , piece_id=FGYLRBVDMMLYWTTKLC3B2P2MHTECHEBH2DJOOKAHBR6TBAMKFBXA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503773727 (72h46m20.374856772s) [noise=true , address=145.79.227.40:4049 , piece_id=VX3PNKMHR75TNJLKRBLYS3KT42ZTPT4X6I5QS2AKNMGDQDVE4WDA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.504114126 (72h46m20.375197161s) [noise=true , address=77.90.31.17:4771 , piece_id=R6PLE6VLZNL23TJC6V6MRBLJP6PKB2DDGQMPPGLMM23WN6J7SH5Q, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.504441298 (72h46m20.375524343s) [noise=true , address=77.90.31.18:4807 , piece_id=7FXKUHBJT7AHDC4PHCZ73Q62GDSOUFRFIVAD337ORGE7YUTAULVA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.505240540 (72h46m20.376323585s) [noise=true , address=77.90.31.14:4662 , piece_id=BO6DQNLFNMG6HAPKOAG2DHWDD3YMX46IAYGDVQG7THC5EVZORJRA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.506022409 (72h46m20.377105454s) [noise=true , address=145.79.227.20:4438 , piece_id=U3FCNUY4ASY3ZGX7ZY26GE2NA52GJNKJMW7XKZKALT2MWBCZMHFA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.506709457 (72h46m20.377792502s) [noise=true , address=145.79.227.43:4158 , piece_id=QN7JC2XQK6FKMIYD2WXQ7GMDILEWDUO2HXZLZERVJ24WWIMHFOVA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:22.339915069 (72h46m22.210998114s) [upload=63694]: Next returned nil segment Dec 8 16:15:22.339932321 (72h46m22.211015366s) [upload=63694]: Waiting for error group managing segments... Dec 8 16:15:22.655133540 (72h46m22.526216575s) [noise=true , address=145.79.227.43:4158 , piece_id=QN7JC2XQK6FKMIYD2WXQ7GMDILEWDUO2HXZLZERVJ24WWIMHFOVA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.660462928 (72h46m22.531545983s) [noise=true , address=145.79.227.17:4971 , piece_id=KRCKO7NXT2XOYK4QHVJK5UIU246TVB45EE7ELVUT365XTJ7RHKJQ, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.683969820 (72h46m22.555052855s) [noise=true , address=77.90.31.11:4555 , piece_id=LZ4YCICB4QQVLQAUZD6ZFABXLYSRKC4OFZESDVUGIOFJMZBRMMSA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.685188641 (72h46m22.556271686s) [noise=true , address=145.79.227.22:4511 , piece_id=RT3ULMIM24LMC4UASFJYVAOHIKJBBGLFYJP3LBJCUL6I532MTSGA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.686713104 (72h46m22.557796149s) [noise=true , address=145.79.227.16:4912 , piece_id=NIW3LT24E4LPJ5QYJVQOAR43S23JLDDOXZ6NVOURZNDTCCTLLPJA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.698627942 (72h46m22.569710987s) [noise=true , address=145.79.227.39:4014 , piece_id=U7SKMZGV33HMSQLQIFIBZ6WNYC7JJ2CENSK5LCW3KVMP53T7467Q, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.713359143 (72h46m22.584442188s) [noise=true , address=145.79.227.40:4049 , piece_id=VX3PNKMHR75TNJLKRBLYS3KT42ZTPT4X6I5QS2AKNMGDQDVE4WDA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.731975212 (72h46m22.603058257s) [noise=true , address=77.90.31.17:4771 , piece_id=R6PLE6VLZNL23TJC6V6MRBLJP6PKB2DDGQMPPGLMM23WN6J7SH5Q, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.741125236 (72h46m22.612208281s) [noise=true , address=145.79.227.20:4438 , piece_id=U3FCNUY4ASY3ZGX7ZY26GE2NA52GJNKJMW7XKZKALT2MWBCZMHFA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.773771311 (72h46m22.644854346s) [noise=true , address=77.90.31.18:4807 , piece_id=7FXKUHBJT7AHDC4PHCZ73Q62GDSOUFRFIVAD337ORGE7YUTAULVA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.906364795 (72h46m22.777447841s) [noise=true , address=77.90.31.14:4662 , piece_id=BO6DQNLFNMG6HAPKOAG2DHWDD3YMX46IAYGDVQG7THC5EVZORJRA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:23.367795705 (72h46m23.23887874s) [noise=true , address=145.79.227.24:4582 , piece_id=FGYLRBVDMMLYWTTKLC3B2P2MHTECHEBH2DJOOKAHBR6TBAMKFBXA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:24.582400944 (72h46m24.453483979s) [noise=true , address=145.79.227.33:4870 , piece_id=IK7NDJD5GNFMWB26JYOMCSNIAT5TTTQOUXB2VFYSNRGIQ2MVMXAQ, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:24.582421023 (72h46m24.453504068s) [seg_pos={0 0}, upload=63694]: Segment reached optimal threshold of 13 pieces. Dec 8 16:15:24.582652363 (72h46m24.453735408s) [noise=true , address=145.79.227.38:4979 , piece_id=HCELMHS3GNLCKHFCQU3V45M4OETQZPL5MQJBUOUF5VPA44HXZW2Q, seg_pos={0 0}, upload=63694]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma): protocol: expected piece hash; context canceled; EOF Dec 8 16:15:24.582659086 (72h46m24.453742121s) [noise=true , address=145.79.227.38:4979 , piece_id=HCELMHS3GNLCKHFCQU3V45M4OETQZPL5MQJBUOUF5VPA44HXZW2Q, seg_pos={0 0}, upload=63694]: Long tail cancellation due to optimal threshold Dec 8 16:15:24.582740842 (72h46m24.453823877s) [noise=true , address=145.79.227.34:4906 , piece_id=AIZQB5ZLXPCLPT5Q7NLNSI3FPBPAMBVUIXKZCTA3ML7L3XP2UY6A, seg_pos={0 0}, upload=63694]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 8 16:15:24.582753055 (72h46m24.45383609s) [noise=true , address=145.79.227.34:4906 , piece_id=AIZQB5ZLXPCLPT5Q7NLNSI3FPBPAMBVUIXKZCTA3ML7L3XP2UY6A, seg_pos={0 0}, upload=63694]: Long tail cancellation due to optimal threshold Dec 8 16:15:24.583044900 (72h46m24.454127935s) [noise=true , address=145.79.227.29:4941 , piece_id=GJQXPA7WEZUTVJNELRBJVTU4LIGXYD7XLEPR5SWF4NBWWMXBW2SA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: context canceled; context canceled Dec 8 16:15:24.583052926 (72h46m24.454135961s) [noise=true , address=145.79.227.29:4941 , piece_id=GJQXPA7WEZUTVJNELRBJVTU4LIGXYD7XLEPR5SWF4NBWWMXBW2SA, seg_pos={0 0}, upload=63694]: Long tail cancellation due to optimal threshold Dec 8 16:15:24.583071441 (72h46m24.454154486s) [seg_pos={0 0}, upload=63694]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 16:15:24.583162014 (72h46m24.454245059s) [upload=63694]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 16:15:24.583171562 (72h46m24.454254627s) [upload=63694]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 16:15:25.042230695 (72h46m24.91331374s) [upload=63694]: Done uploading segments... Dec 8 16:15:25.042258869 (72h46m24.913341914s) [upload=63694]: Upload finished. err: <nil> Duration: 6.454591273s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 6.324757935s trace: https://jaeger.dev.storj.io/trace/4195524620a3e2c2 FAIL (2025-12-08 16:17:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.771167108 (72h9m11.642250153s) [noise=true , address=81.171.14.70:4032 , piece_id=XAMDRY73RIJV4LC2W2FRMH6L4WL3VAFUYKB3QD4TXOQWWZUYGPOQ, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.771462930 (72h9m11.642545976s) [noise=true , address=81.171.14.70:4035 , piece_id=VYAAT75X4R4J22MLY7RK4VHOEFMYZTLWUWTWVAMOOIIDSEXTQV7Q, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.771589993 (72h9m11.642673038s) [noise=true , address=185.167.204.205:4036 , piece_id=X2TWRAEKE6FEH3ITYY74A76ZZQ2KCB63ETZNW5TAHQ3MLEM34PZQ, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.772028248 (72h9m11.643111293s) [noise=true , address=185.167.204.206:4067 , piece_id=3YOG2SUYBHBE6MMIPMQWMBDZTLL7HPSBPEKFHMGOICBTAQU5HA4A, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.772041683 (72h9m11.643124728s) [noise=true , address=185.167.204.206:4069 , piece_id=GH2ZEUIEIUZ6OLM6XH5ZJVXKAMHUKQ6L3GZ7NJLUZFY623YH5GDQ, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.772603803 (72h9m11.643686848s) [noise=true , address=89.149.201.147:4022 , piece_id=CWWNRXZNWQOUTZEA4WPHGA6BYEPO35D2YDU5VSTHFZ7GRC67Y7VQ, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.773136077 (72h9m11.644219122s) [noise=true , address=185.167.204.206:4051 , piece_id=5JMUWS6LLPRENPBAZ74MDELLIZMWTMI3GOZFUWLCOCUNLDTPETWQ, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.773659244 (72h9m11.644742289s) [noise=true , address=185.167.204.206:4062 , piece_id=KYFUGH5EAVFGEJ2J7VDRDWBDP2R2MX4JSCIU7RP3VJOK4LCM6D2Q, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.774276288 (72h9m11.645359333s) [noise=true , address=81.171.14.70:4026 , piece_id=4BLGOA4GSQVOKZ6UBUIXANALBDHDHDXPLDA3TDQ44ACTIC2IHSEA, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.774811648 (72h9m11.645894693s) [noise=true , address=185.167.204.205:4008 , piece_id=JWZAQAYIUNBFRDVQHADARS5NTEWET7UI2GR73ZMIACA5VFUS3D6Q, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:12.600277390 (72h9m12.471360435s) [upload=63171]: Next returned nil segment Dec 8 15:38:12.600300113 (72h9m12.471383158s) [upload=63171]: Waiting for error group managing segments... Dec 8 15:38:12.984985581 (72h9m12.856068647s) [noise=true , address=185.167.204.205:4028 , piece_id=DCGYFXSKGK6GFLLFU7UQONAX643EA6CDTYWEZ42YQPLEERINL3KQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.985372348 (72h9m12.856455383s) [noise=true , address=185.167.204.205:4008 , piece_id=JWZAQAYIUNBFRDVQHADARS5NTEWET7UI2GR73ZMIACA5VFUS3D6Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.993832749 (72h9m12.864915784s) [noise=true , address=185.167.204.205:4015 , piece_id=GK2HV5AGF2Q6RYHDZBZQ3SY537WQXB7E7UF4SWF5X3LFM4PMCZPQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.994210087 (72h9m12.865293122s) [noise=true , address=185.167.204.206:4053 , piece_id=DGSLZPKB474OADK43GBZO2FGAR2F62PBL364EWVNPCZE3PWZGQPA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.994561226 (72h9m12.865644271s) [noise=true , address=185.167.204.205:4036 , piece_id=X2TWRAEKE6FEH3ITYY74A76ZZQ2KCB63ETZNW5TAHQ3MLEM34PZQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.994944356 (72h9m12.866027401s) [noise=true , address=185.167.204.205:4023 , piece_id=5RNGVQ3LUTPKX4RAVK7Y4WRQC2Y6GT5QWV2G3MTY2LW2MU6IMKLQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.995089352 (72h9m12.866172397s) [noise=true , address=185.167.204.205:4007 , piece_id=WEZIGATHDFL7G3S2VG7BEFGGT7JJIXBDWHGIPHBTPSKDJQUTROIQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.995157291 (72h9m12.866240336s) [noise=true , address=185.167.204.205:4033 , piece_id=4V3DXW42ZCPJFF3PPJEBMBYFW354WEZYLVFUSPGEZ4E4HZ7MFI4Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.000801098 (72h9m12.871884163s) [noise=true , address=185.167.204.206:4062 , piece_id=KYFUGH5EAVFGEJ2J7VDRDWBDP2R2MX4JSCIU7RP3VJOK4LCM6D2Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.000862164 (72h9m12.871945209s) [noise=true , address=185.167.204.206:4051 , piece_id=5JMUWS6LLPRENPBAZ74MDELLIZMWTMI3GOZFUWLCOCUNLDTPETWQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.002117705 (72h9m12.87320074s) [noise=true , address=185.167.204.206:4064 , piece_id=LUITKU5LPQ67MFA4VWE6CDV46QOU7RXOFCGKKAYPCLFGWFSLYLTA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.003015134 (72h9m12.874098179s) [noise=true , address=185.167.204.205:4034 , piece_id=OJPEC5SBHQDKJW3LDBPHMEOPDG6NPNNMO3W54C7XIXTNUYBTMUOA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.003432589 (72h9m12.874515624s) [noise=true , address=185.167.204.206:4037 , piece_id=OPF6NTGFZ7FSN7LIEKHOGIJRYUWX27RTI3KKGCKPCOQLK4HC62LQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.004078648 (72h9m12.875161694s) [noise=true , address=185.167.204.206:4069 , piece_id=GH2ZEUIEIUZ6OLM6XH5ZJVXKAMHUKQ6L3GZ7NJLUZFY623YH5GDQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.009788571 (72h9m12.880871616s) [noise=true , address=185.167.204.206:4067 , piece_id=3YOG2SUYBHBE6MMIPMQWMBDZTLL7HPSBPEKFHMGOICBTAQU5HA4A, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.012083561 (72h9m12.883166596s) [noise=true , address=185.167.204.205:4030 , piece_id=EFAUOQCDA25VE55ZHJOSVXTOQD2VTQCW55MEH5DBEAY7N364GMXA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.584995241 (72h9m13.456078286s) [noise=true , address=81.171.14.70:4046 , piece_id=77RPCXTIQW7MO6UKAVJUD4Y7XR674IDZE2TL2SB77LJUHRE3ACDA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.651263008 (72h9m13.522346063s) [noise=true , address=81.171.14.70:4026 , piece_id=4BLGOA4GSQVOKZ6UBUIXANALBDHDHDXPLDA3TDQ44ACTIC2IHSEA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.696266204 (72h9m13.567349249s) [noise=true , address=81.171.14.70:4035 , piece_id=VYAAT75X4R4J22MLY7RK4VHOEFMYZTLWUWTWVAMOOIIDSEXTQV7Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.123999844 (72h9m13.995082889s) [noise=true , address=81.171.14.70:4025 , piece_id=XML4X66PHAT33VTUTEV22HSTIHMG4WFZU6YSDPCZZZ4SEBL3NWUA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.147808461 (72h9m14.018891506s) [noise=true , address=81.171.14.70:4032 , piece_id=XAMDRY73RIJV4LC2W2FRMH6L4WL3VAFUYKB3QD4TXOQWWZUYGPOQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.255119813 (72h9m14.126202858s) [noise=true , address=89.149.201.147:4005 , piece_id=MBR3CYKQ3YSPVXIVSL6ABGZNG7FB5HGP45OA3YE2IRKPJMOSM3KQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.271452473 (72h9m14.142535518s) [noise=true , address=89.149.201.147:4013 , piece_id=KIPMWZVYZADDS2LAOY5ASMIWP32HIRTNGYQ65SNLYR2EQDQCN3HQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.775652542 (72h9m14.646735587s) [noise=true , address=89.149.201.147:4016 , piece_id=5N5HKBPMDQUEQIZZVFFRQGE4LQAGSJ4H6INPDXHYQMDK5YTHUVKQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.785904314 (72h9m14.656987359s) [noise=true , address=89.149.201.147:4017 , piece_id=MBD47F6TJHZ76PAGU7ZIFRQ4M5HUDT5AQULHE6RPQPBUBAVKSTKQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.827349233 (72h9m14.698432278s) [noise=true , address=89.149.201.147:4022 , piece_id=CWWNRXZNWQOUTZEA4WPHGA6BYEPO35D2YDU5VSTHFZ7GRC67Y7VQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.827365925 (72h9m14.69844897s) [seg_pos={0 0}, upload=63171]: Segment reached optimal threshold of 26 pieces. Dec 8 15:38:14.827539125 (72h9m14.69862217s) [noise=true , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hJ5jT6S2d3DJe9DoVDjHMkpwSTpeyTMyNLaK8Pk8SWcUVRKnv): protocol: expected piece hash; context canceled; EOF Dec 8 15:38:14.827545567 (72h9m14.698628612s) [noise=true , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Long tail cancellation due to optimal threshold Dec 8 15:38:14.827903338 (72h9m14.698986373s) [noise=true , address=89.149.201.147:4018 , piece_id=BFJH5ELBTZD767FJNFT2ZBUS3GI62IZIRSKYARLVBCLWZONSURMQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua): protocol: expected piece hash; context canceled; EOF Dec 8 15:38:14.827910673 (72h9m14.698993708s) [noise=true , address=89.149.201.147:4018 , piece_id=BFJH5ELBTZD767FJNFT2ZBUS3GI62IZIRSKYARLVBCLWZONSURMQ, seg_pos={0 0}, upload=63171]: Long tail cancellation due to optimal threshold Dec 8 15:38:14.827928065 (72h9m14.69901111s) [seg_pos={0 0}, upload=63171]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 8 15:38:14.828037083 (72h9m14.699120128s) [upload=63171]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 15:38:14.828053224 (72h9m14.699136269s) [upload=63171]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 15:39:18.272006268 (72h10m18.143089313s) [upload=63171]: Done uploading segments... Dec 8 15:39:18.272087593 (72h10m18.143170628s) [upload=63171]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.32478636s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [...........X..X...X.................X.X.....................] duration: 5.956073084s trace: https://jaeger.dev.storj.io/trace/690f79ef204fee2 FAIL (2025-12-08 16:17:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.956133199s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 4.818639324s trace: https://jaeger.dev.storj.io/trace/5d817b557064ae10 FAIL (2025-12-08 16:17:06 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.818666365s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 24 out of 60 is failed (threshold=6) [X.XXXX.X...XX.X..XX.XX.X..XX.....X.....X...X..XX..X......X.X] duration: 5.002013267s trace: https://jaeger.dev.storj.io/trace/470fdd80045558be FAIL (2025-12-08 16:17:04 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002036321s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 2.037351958s trace: https://jaeger.dev.storj.io/trace/4dad09bb52e33232 FAIL (2025-12-08 16:17:03 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 2.037371937s core uplink us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 6.471769445s trace: https://jaeger.dev.storj.io/trace/4195524620a3e2c2 FAIL (2025-12-08 16:16:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.771167108 (72h9m11.642250153s) [noise=true , address=81.171.14.70:4032 , piece_id=XAMDRY73RIJV4LC2W2FRMH6L4WL3VAFUYKB3QD4TXOQWWZUYGPOQ, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.771462930 (72h9m11.642545976s) [noise=true , address=81.171.14.70:4035 , piece_id=VYAAT75X4R4J22MLY7RK4VHOEFMYZTLWUWTWVAMOOIIDSEXTQV7Q, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.771589993 (72h9m11.642673038s) [noise=true , address=185.167.204.205:4036 , piece_id=X2TWRAEKE6FEH3ITYY74A76ZZQ2KCB63ETZNW5TAHQ3MLEM34PZQ, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.772028248 (72h9m11.643111293s) [noise=true , address=185.167.204.206:4067 , piece_id=3YOG2SUYBHBE6MMIPMQWMBDZTLL7HPSBPEKFHMGOICBTAQU5HA4A, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.772041683 (72h9m11.643124728s) [noise=true , address=185.167.204.206:4069 , piece_id=GH2ZEUIEIUZ6OLM6XH5ZJVXKAMHUKQ6L3GZ7NJLUZFY623YH5GDQ, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.772603803 (72h9m11.643686848s) [noise=true , address=89.149.201.147:4022 , piece_id=CWWNRXZNWQOUTZEA4WPHGA6BYEPO35D2YDU5VSTHFZ7GRC67Y7VQ, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.773136077 (72h9m11.644219122s) [noise=true , address=185.167.204.206:4051 , piece_id=5JMUWS6LLPRENPBAZ74MDELLIZMWTMI3GOZFUWLCOCUNLDTPETWQ, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.773659244 (72h9m11.644742289s) [noise=true , address=185.167.204.206:4062 , piece_id=KYFUGH5EAVFGEJ2J7VDRDWBDP2R2MX4JSCIU7RP3VJOK4LCM6D2Q, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.774276288 (72h9m11.645359333s) [noise=true , address=81.171.14.70:4026 , piece_id=4BLGOA4GSQVOKZ6UBUIXANALBDHDHDXPLDA3TDQ44ACTIC2IHSEA, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.774811648 (72h9m11.645894693s) [noise=true , address=185.167.204.205:4008 , piece_id=JWZAQAYIUNBFRDVQHADARS5NTEWET7UI2GR73ZMIACA5VFUS3D6Q, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:12.600277390 (72h9m12.471360435s) [upload=63171]: Next returned nil segment Dec 8 15:38:12.600300113 (72h9m12.471383158s) [upload=63171]: Waiting for error group managing segments... Dec 8 15:38:12.984985581 (72h9m12.856068647s) [noise=true , address=185.167.204.205:4028 , piece_id=DCGYFXSKGK6GFLLFU7UQONAX643EA6CDTYWEZ42YQPLEERINL3KQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.985372348 (72h9m12.856455383s) [noise=true , address=185.167.204.205:4008 , piece_id=JWZAQAYIUNBFRDVQHADARS5NTEWET7UI2GR73ZMIACA5VFUS3D6Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.993832749 (72h9m12.864915784s) [noise=true , address=185.167.204.205:4015 , piece_id=GK2HV5AGF2Q6RYHDZBZQ3SY537WQXB7E7UF4SWF5X3LFM4PMCZPQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.994210087 (72h9m12.865293122s) [noise=true , address=185.167.204.206:4053 , piece_id=DGSLZPKB474OADK43GBZO2FGAR2F62PBL364EWVNPCZE3PWZGQPA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.994561226 (72h9m12.865644271s) [noise=true , address=185.167.204.205:4036 , piece_id=X2TWRAEKE6FEH3ITYY74A76ZZQ2KCB63ETZNW5TAHQ3MLEM34PZQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.994944356 (72h9m12.866027401s) [noise=true , address=185.167.204.205:4023 , piece_id=5RNGVQ3LUTPKX4RAVK7Y4WRQC2Y6GT5QWV2G3MTY2LW2MU6IMKLQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.995089352 (72h9m12.866172397s) [noise=true , address=185.167.204.205:4007 , piece_id=WEZIGATHDFL7G3S2VG7BEFGGT7JJIXBDWHGIPHBTPSKDJQUTROIQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.995157291 (72h9m12.866240336s) [noise=true , address=185.167.204.205:4033 , piece_id=4V3DXW42ZCPJFF3PPJEBMBYFW354WEZYLVFUSPGEZ4E4HZ7MFI4Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.000801098 (72h9m12.871884163s) [noise=true , address=185.167.204.206:4062 , piece_id=KYFUGH5EAVFGEJ2J7VDRDWBDP2R2MX4JSCIU7RP3VJOK4LCM6D2Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.000862164 (72h9m12.871945209s) [noise=true , address=185.167.204.206:4051 , piece_id=5JMUWS6LLPRENPBAZ74MDELLIZMWTMI3GOZFUWLCOCUNLDTPETWQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.002117705 (72h9m12.87320074s) [noise=true , address=185.167.204.206:4064 , piece_id=LUITKU5LPQ67MFA4VWE6CDV46QOU7RXOFCGKKAYPCLFGWFSLYLTA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.003015134 (72h9m12.874098179s) [noise=true , address=185.167.204.205:4034 , piece_id=OJPEC5SBHQDKJW3LDBPHMEOPDG6NPNNMO3W54C7XIXTNUYBTMUOA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.003432589 (72h9m12.874515624s) [noise=true , address=185.167.204.206:4037 , piece_id=OPF6NTGFZ7FSN7LIEKHOGIJRYUWX27RTI3KKGCKPCOQLK4HC62LQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.004078648 (72h9m12.875161694s) [noise=true , address=185.167.204.206:4069 , piece_id=GH2ZEUIEIUZ6OLM6XH5ZJVXKAMHUKQ6L3GZ7NJLUZFY623YH5GDQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.009788571 (72h9m12.880871616s) [noise=true , address=185.167.204.206:4067 , piece_id=3YOG2SUYBHBE6MMIPMQWMBDZTLL7HPSBPEKFHMGOICBTAQU5HA4A, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.012083561 (72h9m12.883166596s) [noise=true , address=185.167.204.205:4030 , piece_id=EFAUOQCDA25VE55ZHJOSVXTOQD2VTQCW55MEH5DBEAY7N364GMXA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.584995241 (72h9m13.456078286s) [noise=true , address=81.171.14.70:4046 , piece_id=77RPCXTIQW7MO6UKAVJUD4Y7XR674IDZE2TL2SB77LJUHRE3ACDA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.651263008 (72h9m13.522346063s) [noise=true , address=81.171.14.70:4026 , piece_id=4BLGOA4GSQVOKZ6UBUIXANALBDHDHDXPLDA3TDQ44ACTIC2IHSEA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.696266204 (72h9m13.567349249s) [noise=true , address=81.171.14.70:4035 , piece_id=VYAAT75X4R4J22MLY7RK4VHOEFMYZTLWUWTWVAMOOIIDSEXTQV7Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.123999844 (72h9m13.995082889s) [noise=true , address=81.171.14.70:4025 , piece_id=XML4X66PHAT33VTUTEV22HSTIHMG4WFZU6YSDPCZZZ4SEBL3NWUA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.147808461 (72h9m14.018891506s) [noise=true , address=81.171.14.70:4032 , piece_id=XAMDRY73RIJV4LC2W2FRMH6L4WL3VAFUYKB3QD4TXOQWWZUYGPOQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.255119813 (72h9m14.126202858s) [noise=true , address=89.149.201.147:4005 , piece_id=MBR3CYKQ3YSPVXIVSL6ABGZNG7FB5HGP45OA3YE2IRKPJMOSM3KQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.271452473 (72h9m14.142535518s) [noise=true , address=89.149.201.147:4013 , piece_id=KIPMWZVYZADDS2LAOY5ASMIWP32HIRTNGYQ65SNLYR2EQDQCN3HQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.775652542 (72h9m14.646735587s) [noise=true , address=89.149.201.147:4016 , piece_id=5N5HKBPMDQUEQIZZVFFRQGE4LQAGSJ4H6INPDXHYQMDK5YTHUVKQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.785904314 (72h9m14.656987359s) [noise=true , address=89.149.201.147:4017 , piece_id=MBD47F6TJHZ76PAGU7ZIFRQ4M5HUDT5AQULHE6RPQPBUBAVKSTKQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.827349233 (72h9m14.698432278s) [noise=true , address=89.149.201.147:4022 , piece_id=CWWNRXZNWQOUTZEA4WPHGA6BYEPO35D2YDU5VSTHFZ7GRC67Y7VQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.827365925 (72h9m14.69844897s) [seg_pos={0 0}, upload=63171]: Segment reached optimal threshold of 26 pieces. Dec 8 15:38:14.827539125 (72h9m14.69862217s) [noise=true , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hJ5jT6S2d3DJe9DoVDjHMkpwSTpeyTMyNLaK8Pk8SWcUVRKnv): protocol: expected piece hash; context canceled; EOF Dec 8 15:38:14.827545567 (72h9m14.698628612s) [noise=true , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Long tail cancellation due to optimal threshold Dec 8 15:38:14.827903338 (72h9m14.698986373s) [noise=true , address=89.149.201.147:4018 , piece_id=BFJH5ELBTZD767FJNFT2ZBUS3GI62IZIRSKYARLVBCLWZONSURMQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua): protocol: expected piece hash; context canceled; EOF Dec 8 15:38:14.827910673 (72h9m14.698993708s) [noise=true , address=89.149.201.147:4018 , piece_id=BFJH5ELBTZD767FJNFT2ZBUS3GI62IZIRSKYARLVBCLWZONSURMQ, seg_pos={0 0}, upload=63171]: Long tail cancellation due to optimal threshold Dec 8 15:38:14.827928065 (72h9m14.69901111s) [seg_pos={0 0}, upload=63171]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 8 15:38:14.828037083 (72h9m14.699120128s) [upload=63171]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 15:38:14.828053224 (72h9m14.699136269s) [upload=63171]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 15:39:18.272006268 (72h10m18.143089313s) [upload=63171]: Done uploading segments... Dec 8 15:39:18.272087593 (72h10m18.143170628s) [upload=63171]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.471798059s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 1.506646387s trace: https://jaeger.dev.storj.io/trace/3bdb8560e64b821d FAIL (2025-12-08 16:16:51 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 1.506688806s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 5.095578822s trace: https://jaeger.dev.storj.io/trace/5d817b557064ae10 FAIL (2025-12-08 16:16:44 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.095602879s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 1.990719551s trace: https://jaeger.dev.storj.io/trace/4dad09bb52e33232 FAIL (2025-12-08 16:16:36 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.990747153s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [XX...X.X.X.X....XX......X....X..........................X...] duration: 17.486646476s trace: https://jaeger.dev.storj.io/trace/12eb27369d16b107 FAIL (2025-12-08 16:16:34 UTC) orbiter warning: context deadline exceeded Duration: 17.486673717s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 23 out of 60 is failed (threshold=6) [.X.XXXX.X...XX.X..XX.XX.X..XX.....X.....X...X..XX..X......X.] duration: 3.593281536s trace: https://jaeger.dev.storj.io/trace/29a7923ec0ca079f FAIL (2025-12-08 16:16:24 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.593322834s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 5.821031392s trace: https://jaeger.dev.storj.io/trace/4195524620a3e2c2 FAIL (2025-12-08 16:16:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.771167108 (72h9m11.642250153s) [noise=true , address=81.171.14.70:4032 , piece_id=XAMDRY73RIJV4LC2W2FRMH6L4WL3VAFUYKB3QD4TXOQWWZUYGPOQ, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.771462930 (72h9m11.642545976s) [noise=true , address=81.171.14.70:4035 , piece_id=VYAAT75X4R4J22MLY7RK4VHOEFMYZTLWUWTWVAMOOIIDSEXTQV7Q, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.771589993 (72h9m11.642673038s) [noise=true , address=185.167.204.205:4036 , piece_id=X2TWRAEKE6FEH3ITYY74A76ZZQ2KCB63ETZNW5TAHQ3MLEM34PZQ, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.772028248 (72h9m11.643111293s) [noise=true , address=185.167.204.206:4067 , piece_id=3YOG2SUYBHBE6MMIPMQWMBDZTLL7HPSBPEKFHMGOICBTAQU5HA4A, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.772041683 (72h9m11.643124728s) [noise=true , address=185.167.204.206:4069 , piece_id=GH2ZEUIEIUZ6OLM6XH5ZJVXKAMHUKQ6L3GZ7NJLUZFY623YH5GDQ, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.772603803 (72h9m11.643686848s) [noise=true , address=89.149.201.147:4022 , piece_id=CWWNRXZNWQOUTZEA4WPHGA6BYEPO35D2YDU5VSTHFZ7GRC67Y7VQ, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.773136077 (72h9m11.644219122s) [noise=true , address=185.167.204.206:4051 , piece_id=5JMUWS6LLPRENPBAZ74MDELLIZMWTMI3GOZFUWLCOCUNLDTPETWQ, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.773659244 (72h9m11.644742289s) [noise=true , address=185.167.204.206:4062 , piece_id=KYFUGH5EAVFGEJ2J7VDRDWBDP2R2MX4JSCIU7RP3VJOK4LCM6D2Q, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.774276288 (72h9m11.645359333s) [noise=true , address=81.171.14.70:4026 , piece_id=4BLGOA4GSQVOKZ6UBUIXANALBDHDHDXPLDA3TDQ44ACTIC2IHSEA, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.774811648 (72h9m11.645894693s) [noise=true , address=185.167.204.205:4008 , piece_id=JWZAQAYIUNBFRDVQHADARS5NTEWET7UI2GR73ZMIACA5VFUS3D6Q, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:12.600277390 (72h9m12.471360435s) [upload=63171]: Next returned nil segment Dec 8 15:38:12.600300113 (72h9m12.471383158s) [upload=63171]: Waiting for error group managing segments... Dec 8 15:38:12.984985581 (72h9m12.856068647s) [noise=true , address=185.167.204.205:4028 , piece_id=DCGYFXSKGK6GFLLFU7UQONAX643EA6CDTYWEZ42YQPLEERINL3KQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.985372348 (72h9m12.856455383s) [noise=true , address=185.167.204.205:4008 , piece_id=JWZAQAYIUNBFRDVQHADARS5NTEWET7UI2GR73ZMIACA5VFUS3D6Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.993832749 (72h9m12.864915784s) [noise=true , address=185.167.204.205:4015 , piece_id=GK2HV5AGF2Q6RYHDZBZQ3SY537WQXB7E7UF4SWF5X3LFM4PMCZPQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.994210087 (72h9m12.865293122s) [noise=true , address=185.167.204.206:4053 , piece_id=DGSLZPKB474OADK43GBZO2FGAR2F62PBL364EWVNPCZE3PWZGQPA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.994561226 (72h9m12.865644271s) [noise=true , address=185.167.204.205:4036 , piece_id=X2TWRAEKE6FEH3ITYY74A76ZZQ2KCB63ETZNW5TAHQ3MLEM34PZQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.994944356 (72h9m12.866027401s) [noise=true , address=185.167.204.205:4023 , piece_id=5RNGVQ3LUTPKX4RAVK7Y4WRQC2Y6GT5QWV2G3MTY2LW2MU6IMKLQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.995089352 (72h9m12.866172397s) [noise=true , address=185.167.204.205:4007 , piece_id=WEZIGATHDFL7G3S2VG7BEFGGT7JJIXBDWHGIPHBTPSKDJQUTROIQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.995157291 (72h9m12.866240336s) [noise=true , address=185.167.204.205:4033 , piece_id=4V3DXW42ZCPJFF3PPJEBMBYFW354WEZYLVFUSPGEZ4E4HZ7MFI4Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.000801098 (72h9m12.871884163s) [noise=true , address=185.167.204.206:4062 , piece_id=KYFUGH5EAVFGEJ2J7VDRDWBDP2R2MX4JSCIU7RP3VJOK4LCM6D2Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.000862164 (72h9m12.871945209s) [noise=true , address=185.167.204.206:4051 , piece_id=5JMUWS6LLPRENPBAZ74MDELLIZMWTMI3GOZFUWLCOCUNLDTPETWQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.002117705 (72h9m12.87320074s) [noise=true , address=185.167.204.206:4064 , piece_id=LUITKU5LPQ67MFA4VWE6CDV46QOU7RXOFCGKKAYPCLFGWFSLYLTA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.003015134 (72h9m12.874098179s) [noise=true , address=185.167.204.205:4034 , piece_id=OJPEC5SBHQDKJW3LDBPHMEOPDG6NPNNMO3W54C7XIXTNUYBTMUOA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.003432589 (72h9m12.874515624s) [noise=true , address=185.167.204.206:4037 , piece_id=OPF6NTGFZ7FSN7LIEKHOGIJRYUWX27RTI3KKGCKPCOQLK4HC62LQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.004078648 (72h9m12.875161694s) [noise=true , address=185.167.204.206:4069 , piece_id=GH2ZEUIEIUZ6OLM6XH5ZJVXKAMHUKQ6L3GZ7NJLUZFY623YH5GDQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.009788571 (72h9m12.880871616s) [noise=true , address=185.167.204.206:4067 , piece_id=3YOG2SUYBHBE6MMIPMQWMBDZTLL7HPSBPEKFHMGOICBTAQU5HA4A, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.012083561 (72h9m12.883166596s) [noise=true , address=185.167.204.205:4030 , piece_id=EFAUOQCDA25VE55ZHJOSVXTOQD2VTQCW55MEH5DBEAY7N364GMXA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.584995241 (72h9m13.456078286s) [noise=true , address=81.171.14.70:4046 , piece_id=77RPCXTIQW7MO6UKAVJUD4Y7XR674IDZE2TL2SB77LJUHRE3ACDA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.651263008 (72h9m13.522346063s) [noise=true , address=81.171.14.70:4026 , piece_id=4BLGOA4GSQVOKZ6UBUIXANALBDHDHDXPLDA3TDQ44ACTIC2IHSEA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.696266204 (72h9m13.567349249s) [noise=true , address=81.171.14.70:4035 , piece_id=VYAAT75X4R4J22MLY7RK4VHOEFMYZTLWUWTWVAMOOIIDSEXTQV7Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.123999844 (72h9m13.995082889s) [noise=true , address=81.171.14.70:4025 , piece_id=XML4X66PHAT33VTUTEV22HSTIHMG4WFZU6YSDPCZZZ4SEBL3NWUA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.147808461 (72h9m14.018891506s) [noise=true , address=81.171.14.70:4032 , piece_id=XAMDRY73RIJV4LC2W2FRMH6L4WL3VAFUYKB3QD4TXOQWWZUYGPOQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.255119813 (72h9m14.126202858s) [noise=true , address=89.149.201.147:4005 , piece_id=MBR3CYKQ3YSPVXIVSL6ABGZNG7FB5HGP45OA3YE2IRKPJMOSM3KQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.271452473 (72h9m14.142535518s) [noise=true , address=89.149.201.147:4013 , piece_id=KIPMWZVYZADDS2LAOY5ASMIWP32HIRTNGYQ65SNLYR2EQDQCN3HQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.775652542 (72h9m14.646735587s) [noise=true , address=89.149.201.147:4016 , piece_id=5N5HKBPMDQUEQIZZVFFRQGE4LQAGSJ4H6INPDXHYQMDK5YTHUVKQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.785904314 (72h9m14.656987359s) [noise=true , address=89.149.201.147:4017 , piece_id=MBD47F6TJHZ76PAGU7ZIFRQ4M5HUDT5AQULHE6RPQPBUBAVKSTKQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.827349233 (72h9m14.698432278s) [noise=true , address=89.149.201.147:4022 , piece_id=CWWNRXZNWQOUTZEA4WPHGA6BYEPO35D2YDU5VSTHFZ7GRC67Y7VQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.827365925 (72h9m14.69844897s) [seg_pos={0 0}, upload=63171]: Segment reached optimal threshold of 26 pieces. Dec 8 15:38:14.827539125 (72h9m14.69862217s) [noise=true , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hJ5jT6S2d3DJe9DoVDjHMkpwSTpeyTMyNLaK8Pk8SWcUVRKnv): protocol: expected piece hash; context canceled; EOF Dec 8 15:38:14.827545567 (72h9m14.698628612s) [noise=true , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Long tail cancellation due to optimal threshold Dec 8 15:38:14.827903338 (72h9m14.698986373s) [noise=true , address=89.149.201.147:4018 , piece_id=BFJH5ELBTZD767FJNFT2ZBUS3GI62IZIRSKYARLVBCLWZONSURMQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua): protocol: expected piece hash; context canceled; EOF Dec 8 15:38:14.827910673 (72h9m14.698993708s) [noise=true , address=89.149.201.147:4018 , piece_id=BFJH5ELBTZD767FJNFT2ZBUS3GI62IZIRSKYARLVBCLWZONSURMQ, seg_pos={0 0}, upload=63171]: Long tail cancellation due to optimal threshold Dec 8 15:38:14.827928065 (72h9m14.69901111s) [seg_pos={0 0}, upload=63171]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 8 15:38:14.828037083 (72h9m14.699120128s) [upload=63171]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 15:38:14.828053224 (72h9m14.699136269s) [upload=63171]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 15:39:18.272006268 (72h10m18.143089313s) [upload=63171]: Done uploading segments... Dec 8 15:39:18.272087593 (72h10m18.143170628s) [upload=63171]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.821062692s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [............X..X...X.................X.X....................] duration: 5.676541282s trace: https://jaeger.dev.storj.io/trace/690f79ef204fee2 FAIL (2025-12-08 16:16:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.676571311s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..............X.............X............................XX.] duration: 7.604961972s trace: https://jaeger.dev.storj.io/trace/13568c7eb15b2369 FAIL (2025-12-08 16:16:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, Piece ID: U3FCNUY4ASY3ZGX7ZY26GE2NA52GJNKJMW7XKZKALT2MWBCZMHFA) context deadline exceeded; piecestore: (Node ID: 1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, Piece ID: QN7JC2XQK6FKMIYD2WXQ7GMDILEWDUO2HXZLZERVJ24WWIMHFOVA) context deadline exceeded; piecestore: (Node ID: 12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, Piece ID: KRCKO7NXT2XOYK4QHVJK5UIU246TVB45EE7ELVUT365XTJ7RHKJQ) context deadline exceeded; piecestore: (Node ID: 122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, Piece ID: U7SKMZGV33HMSQLQIFIBZ6WNYC7JJ2CENSK5LCW3KVMP53T7467Q) context deadline exceeded; piecestore: (Node ID: 1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, Piece ID: FGYLRBVDMMLYWTTKLC3B2P2MHTECHEBH2DJOOKAHBR6TBAMKFBXA) context deadline exceeded ======LOGS====== Dec 8 16:15:19.690262294 (72h46m19.561345339s) [upload=63694]: Starting upload Dec 8 16:15:19.690357154 (72h46m19.56144022s) [upload=63694]: Done starting upload Dec 8 16:15:19.690425354 (72h46m19.561508409s) [upload=63694]: Uploading segments... Dec 8 16:15:19.690652296 (72h46m19.561735341s) [upload=63694]: Got next segment. Inline: false Dec 8 16:15:19.690693895 (72h46m19.56177694s) [upload=63694]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 16:15:19.690700889 (72h46m19.561783934s) [upload=63694]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 16:15:20.501121517 (72h46m20.372204572s) [seg_pos={0 0}, upload=63694]: Begin upload segment... Dec 8 16:15:20.501188935 (72h46m20.372271981s) [seg_pos={0 0}, upload=63694]: Done begin upload segment. Dec 8 16:15:20.501225655 (72h46m20.3723087s) [noise=true , address=145.79.227.16:4912 , piece_id=NIW3LT24E4LPJ5QYJVQOAR43S23JLDDOXZ6NVOURZNDTCCTLLPJA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.501311268 (72h46m20.372394313s) [noise=true , address=145.79.227.34:4906 , piece_id=AIZQB5ZLXPCLPT5Q7NLNSI3FPBPAMBVUIXKZCTA3ML7L3XP2UY6A, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.501987676 (72h46m20.373070722s) [noise=true , address=145.79.227.17:4971 , piece_id=KRCKO7NXT2XOYK4QHVJK5UIU246TVB45EE7ELVUT365XTJ7RHKJQ, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.502073379 (72h46m20.373156424s) [noise=true , address=145.79.227.38:4979 , piece_id=HCELMHS3GNLCKHFCQU3V45M4OETQZPL5MQJBUOUF5VPA44HXZW2Q, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.502559184 (72h46m20.373642229s) [noise=true , address=145.79.227.33:4870 , piece_id=IK7NDJD5GNFMWB26JYOMCSNIAT5TTTQOUXB2VFYSNRGIQ2MVMXAQ, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.502903350 (72h46m20.373986385s) [noise=true , address=145.79.227.22:4511 , piece_id=RT3ULMIM24LMC4UASFJYVAOHIKJBBGLFYJP3LBJCUL6I532MTSGA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503199825 (72h46m20.37428287s) [noise=true , address=145.79.227.29:4941 , piece_id=GJQXPA7WEZUTVJNELRBJVTU4LIGXYD7XLEPR5SWF4NBWWMXBW2SA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503205956 (72h46m20.374289001s) [noise=true , address=77.90.31.11:4555 , piece_id=LZ4YCICB4QQVLQAUZD6ZFABXLYSRKC4OFZESDVUGIOFJMZBRMMSA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503484487 (72h46m20.374567532s) [noise=true , address=145.79.227.39:4014 , piece_id=U7SKMZGV33HMSQLQIFIBZ6WNYC7JJ2CENSK5LCW3KVMP53T7467Q, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503757176 (72h46m20.374840221s) [noise=true , address=145.79.227.24:4582 , piece_id=FGYLRBVDMMLYWTTKLC3B2P2MHTECHEBH2DJOOKAHBR6TBAMKFBXA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503773727 (72h46m20.374856772s) [noise=true , address=145.79.227.40:4049 , piece_id=VX3PNKMHR75TNJLKRBLYS3KT42ZTPT4X6I5QS2AKNMGDQDVE4WDA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.504114126 (72h46m20.375197161s) [noise=true , address=77.90.31.17:4771 , piece_id=R6PLE6VLZNL23TJC6V6MRBLJP6PKB2DDGQMPPGLMM23WN6J7SH5Q, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.504441298 (72h46m20.375524343s) [noise=true , address=77.90.31.18:4807 , piece_id=7FXKUHBJT7AHDC4PHCZ73Q62GDSOUFRFIVAD337ORGE7YUTAULVA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.505240540 (72h46m20.376323585s) [noise=true , address=77.90.31.14:4662 , piece_id=BO6DQNLFNMG6HAPKOAG2DHWDD3YMX46IAYGDVQG7THC5EVZORJRA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.506022409 (72h46m20.377105454s) [noise=true , address=145.79.227.20:4438 , piece_id=U3FCNUY4ASY3ZGX7ZY26GE2NA52GJNKJMW7XKZKALT2MWBCZMHFA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.506709457 (72h46m20.377792502s) [noise=true , address=145.79.227.43:4158 , piece_id=QN7JC2XQK6FKMIYD2WXQ7GMDILEWDUO2HXZLZERVJ24WWIMHFOVA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:22.339915069 (72h46m22.210998114s) [upload=63694]: Next returned nil segment Dec 8 16:15:22.339932321 (72h46m22.211015366s) [upload=63694]: Waiting for error group managing segments... Dec 8 16:15:22.655133540 (72h46m22.526216575s) [noise=true , address=145.79.227.43:4158 , piece_id=QN7JC2XQK6FKMIYD2WXQ7GMDILEWDUO2HXZLZERVJ24WWIMHFOVA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.660462928 (72h46m22.531545983s) [noise=true , address=145.79.227.17:4971 , piece_id=KRCKO7NXT2XOYK4QHVJK5UIU246TVB45EE7ELVUT365XTJ7RHKJQ, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.683969820 (72h46m22.555052855s) [noise=true , address=77.90.31.11:4555 , piece_id=LZ4YCICB4QQVLQAUZD6ZFABXLYSRKC4OFZESDVUGIOFJMZBRMMSA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.685188641 (72h46m22.556271686s) [noise=true , address=145.79.227.22:4511 , piece_id=RT3ULMIM24LMC4UASFJYVAOHIKJBBGLFYJP3LBJCUL6I532MTSGA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.686713104 (72h46m22.557796149s) [noise=true , address=145.79.227.16:4912 , piece_id=NIW3LT24E4LPJ5QYJVQOAR43S23JLDDOXZ6NVOURZNDTCCTLLPJA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.698627942 (72h46m22.569710987s) [noise=true , address=145.79.227.39:4014 , piece_id=U7SKMZGV33HMSQLQIFIBZ6WNYC7JJ2CENSK5LCW3KVMP53T7467Q, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.713359143 (72h46m22.584442188s) [noise=true , address=145.79.227.40:4049 , piece_id=VX3PNKMHR75TNJLKRBLYS3KT42ZTPT4X6I5QS2AKNMGDQDVE4WDA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.731975212 (72h46m22.603058257s) [noise=true , address=77.90.31.17:4771 , piece_id=R6PLE6VLZNL23TJC6V6MRBLJP6PKB2DDGQMPPGLMM23WN6J7SH5Q, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.741125236 (72h46m22.612208281s) [noise=true , address=145.79.227.20:4438 , piece_id=U3FCNUY4ASY3ZGX7ZY26GE2NA52GJNKJMW7XKZKALT2MWBCZMHFA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.773771311 (72h46m22.644854346s) [noise=true , address=77.90.31.18:4807 , piece_id=7FXKUHBJT7AHDC4PHCZ73Q62GDSOUFRFIVAD337ORGE7YUTAULVA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.906364795 (72h46m22.777447841s) [noise=true , address=77.90.31.14:4662 , piece_id=BO6DQNLFNMG6HAPKOAG2DHWDD3YMX46IAYGDVQG7THC5EVZORJRA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:23.367795705 (72h46m23.23887874s) [noise=true , address=145.79.227.24:4582 , piece_id=FGYLRBVDMMLYWTTKLC3B2P2MHTECHEBH2DJOOKAHBR6TBAMKFBXA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:24.582400944 (72h46m24.453483979s) [noise=true , address=145.79.227.33:4870 , piece_id=IK7NDJD5GNFMWB26JYOMCSNIAT5TTTQOUXB2VFYSNRGIQ2MVMXAQ, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:24.582421023 (72h46m24.453504068s) [seg_pos={0 0}, upload=63694]: Segment reached optimal threshold of 13 pieces. Dec 8 16:15:24.582652363 (72h46m24.453735408s) [noise=true , address=145.79.227.38:4979 , piece_id=HCELMHS3GNLCKHFCQU3V45M4OETQZPL5MQJBUOUF5VPA44HXZW2Q, seg_pos={0 0}, upload=63694]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma): protocol: expected piece hash; context canceled; EOF Dec 8 16:15:24.582659086 (72h46m24.453742121s) [noise=true , address=145.79.227.38:4979 , piece_id=HCELMHS3GNLCKHFCQU3V45M4OETQZPL5MQJBUOUF5VPA44HXZW2Q, seg_pos={0 0}, upload=63694]: Long tail cancellation due to optimal threshold Dec 8 16:15:24.582740842 (72h46m24.453823877s) [noise=true , address=145.79.227.34:4906 , piece_id=AIZQB5ZLXPCLPT5Q7NLNSI3FPBPAMBVUIXKZCTA3ML7L3XP2UY6A, seg_pos={0 0}, upload=63694]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 8 16:15:24.582753055 (72h46m24.45383609s) [noise=true , address=145.79.227.34:4906 , piece_id=AIZQB5ZLXPCLPT5Q7NLNSI3FPBPAMBVUIXKZCTA3ML7L3XP2UY6A, seg_pos={0 0}, upload=63694]: Long tail cancellation due to optimal threshold Dec 8 16:15:24.583044900 (72h46m24.454127935s) [noise=true , address=145.79.227.29:4941 , piece_id=GJQXPA7WEZUTVJNELRBJVTU4LIGXYD7XLEPR5SWF4NBWWMXBW2SA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: context canceled; context canceled Dec 8 16:15:24.583052926 (72h46m24.454135961s) [noise=true , address=145.79.227.29:4941 , piece_id=GJQXPA7WEZUTVJNELRBJVTU4LIGXYD7XLEPR5SWF4NBWWMXBW2SA, seg_pos={0 0}, upload=63694]: Long tail cancellation due to optimal threshold Dec 8 16:15:24.583071441 (72h46m24.454154486s) [seg_pos={0 0}, upload=63694]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 16:15:24.583162014 (72h46m24.454245059s) [upload=63694]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 16:15:24.583171562 (72h46m24.454254627s) [upload=63694]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 16:15:25.042230695 (72h46m24.91331374s) [upload=63694]: Done uploading segments... Dec 8 16:15:25.042258869 (72h46m24.913341914s) [upload=63694]: Upload finished. err: <nil> Duration: 7.605016517s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 1.442802393s trace: https://jaeger.dev.storj.io/trace/3bdb8560e64b821d FAIL (2025-12-08 16:16:03 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 1.442872496s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.............X..X...X.................X.X...................] duration: 17.437583836s trace: https://jaeger.dev.storj.io/trace/690f79ef204fee2 FAIL (2025-12-08 16:15:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 17.437616879s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 24 out of 60 is failed (threshold=6) [X.X.XXXX.X...XX.X..XX.XX.X..XX.....X.....X...X..XX..X......X] duration: 5.002088376s trace: https://jaeger.dev.storj.io/trace/29a7923ec0ca079f FAIL (2025-12-08 16:15:54 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002112261s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 4.668967318s trace: https://jaeger.dev.storj.io/trace/4dad09bb52e33232 FAIL (2025-12-08 16:15:44 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 4.669001793s core uplink us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 5.614535674s trace: https://jaeger.dev.storj.io/trace/5d817b557064ae10 FAIL (2025-12-08 16:15:44 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.614560933s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [X..............X.............X............................XX] duration: 30.003092026s trace: https://jaeger.dev.storj.io/trace/13568c7eb15b2369 FAIL (2025-12-08 16:15:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, Piece ID: U3FCNUY4ASY3ZGX7ZY26GE2NA52GJNKJMW7XKZKALT2MWBCZMHFA) context deadline exceeded; piecestore: (Node ID: 1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, Piece ID: QN7JC2XQK6FKMIYD2WXQ7GMDILEWDUO2HXZLZERVJ24WWIMHFOVA) context deadline exceeded; piecestore: (Node ID: 12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, Piece ID: KRCKO7NXT2XOYK4QHVJK5UIU246TVB45EE7ELVUT365XTJ7RHKJQ) context deadline exceeded; piecestore: (Node ID: 122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, Piece ID: U7SKMZGV33HMSQLQIFIBZ6WNYC7JJ2CENSK5LCW3KVMP53T7467Q) context deadline exceeded; piecestore: (Node ID: 1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, Piece ID: FGYLRBVDMMLYWTTKLC3B2P2MHTECHEBH2DJOOKAHBR6TBAMKFBXA) context deadline exceeded ======LOGS====== Dec 8 16:15:19.690262294 (72h46m19.561345339s) [upload=63694]: Starting upload Dec 8 16:15:19.690357154 (72h46m19.56144022s) [upload=63694]: Done starting upload Dec 8 16:15:19.690425354 (72h46m19.561508409s) [upload=63694]: Uploading segments... Dec 8 16:15:19.690652296 (72h46m19.561735341s) [upload=63694]: Got next segment. Inline: false Dec 8 16:15:19.690693895 (72h46m19.56177694s) [upload=63694]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 16:15:19.690700889 (72h46m19.561783934s) [upload=63694]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 16:15:20.501121517 (72h46m20.372204572s) [seg_pos={0 0}, upload=63694]: Begin upload segment... Dec 8 16:15:20.501188935 (72h46m20.372271981s) [seg_pos={0 0}, upload=63694]: Done begin upload segment. Dec 8 16:15:20.501225655 (72h46m20.3723087s) [noise=true , address=145.79.227.16:4912 , piece_id=NIW3LT24E4LPJ5QYJVQOAR43S23JLDDOXZ6NVOURZNDTCCTLLPJA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.501311268 (72h46m20.372394313s) [noise=true , address=145.79.227.34:4906 , piece_id=AIZQB5ZLXPCLPT5Q7NLNSI3FPBPAMBVUIXKZCTA3ML7L3XP2UY6A, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.501987676 (72h46m20.373070722s) [noise=true , address=145.79.227.17:4971 , piece_id=KRCKO7NXT2XOYK4QHVJK5UIU246TVB45EE7ELVUT365XTJ7RHKJQ, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.502073379 (72h46m20.373156424s) [noise=true , address=145.79.227.38:4979 , piece_id=HCELMHS3GNLCKHFCQU3V45M4OETQZPL5MQJBUOUF5VPA44HXZW2Q, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.502559184 (72h46m20.373642229s) [noise=true , address=145.79.227.33:4870 , piece_id=IK7NDJD5GNFMWB26JYOMCSNIAT5TTTQOUXB2VFYSNRGIQ2MVMXAQ, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.502903350 (72h46m20.373986385s) [noise=true , address=145.79.227.22:4511 , piece_id=RT3ULMIM24LMC4UASFJYVAOHIKJBBGLFYJP3LBJCUL6I532MTSGA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503199825 (72h46m20.37428287s) [noise=true , address=145.79.227.29:4941 , piece_id=GJQXPA7WEZUTVJNELRBJVTU4LIGXYD7XLEPR5SWF4NBWWMXBW2SA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503205956 (72h46m20.374289001s) [noise=true , address=77.90.31.11:4555 , piece_id=LZ4YCICB4QQVLQAUZD6ZFABXLYSRKC4OFZESDVUGIOFJMZBRMMSA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503484487 (72h46m20.374567532s) [noise=true , address=145.79.227.39:4014 , piece_id=U7SKMZGV33HMSQLQIFIBZ6WNYC7JJ2CENSK5LCW3KVMP53T7467Q, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503757176 (72h46m20.374840221s) [noise=true , address=145.79.227.24:4582 , piece_id=FGYLRBVDMMLYWTTKLC3B2P2MHTECHEBH2DJOOKAHBR6TBAMKFBXA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.503773727 (72h46m20.374856772s) [noise=true , address=145.79.227.40:4049 , piece_id=VX3PNKMHR75TNJLKRBLYS3KT42ZTPT4X6I5QS2AKNMGDQDVE4WDA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.504114126 (72h46m20.375197161s) [noise=true , address=77.90.31.17:4771 , piece_id=R6PLE6VLZNL23TJC6V6MRBLJP6PKB2DDGQMPPGLMM23WN6J7SH5Q, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.504441298 (72h46m20.375524343s) [noise=true , address=77.90.31.18:4807 , piece_id=7FXKUHBJT7AHDC4PHCZ73Q62GDSOUFRFIVAD337ORGE7YUTAULVA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.505240540 (72h46m20.376323585s) [noise=true , address=77.90.31.14:4662 , piece_id=BO6DQNLFNMG6HAPKOAG2DHWDD3YMX46IAYGDVQG7THC5EVZORJRA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.506022409 (72h46m20.377105454s) [noise=true , address=145.79.227.20:4438 , piece_id=U3FCNUY4ASY3ZGX7ZY26GE2NA52GJNKJMW7XKZKALT2MWBCZMHFA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:20.506709457 (72h46m20.377792502s) [noise=true , address=145.79.227.43:4158 , piece_id=QN7JC2XQK6FKMIYD2WXQ7GMDILEWDUO2HXZLZERVJ24WWIMHFOVA, seg_pos={0 0}, upload=63694]: Uploading piece... Dec 8 16:15:22.339915069 (72h46m22.210998114s) [upload=63694]: Next returned nil segment Dec 8 16:15:22.339932321 (72h46m22.211015366s) [upload=63694]: Waiting for error group managing segments... Dec 8 16:15:22.655133540 (72h46m22.526216575s) [noise=true , address=145.79.227.43:4158 , piece_id=QN7JC2XQK6FKMIYD2WXQ7GMDILEWDUO2HXZLZERVJ24WWIMHFOVA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.660462928 (72h46m22.531545983s) [noise=true , address=145.79.227.17:4971 , piece_id=KRCKO7NXT2XOYK4QHVJK5UIU246TVB45EE7ELVUT365XTJ7RHKJQ, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.683969820 (72h46m22.555052855s) [noise=true , address=77.90.31.11:4555 , piece_id=LZ4YCICB4QQVLQAUZD6ZFABXLYSRKC4OFZESDVUGIOFJMZBRMMSA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.685188641 (72h46m22.556271686s) [noise=true , address=145.79.227.22:4511 , piece_id=RT3ULMIM24LMC4UASFJYVAOHIKJBBGLFYJP3LBJCUL6I532MTSGA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.686713104 (72h46m22.557796149s) [noise=true , address=145.79.227.16:4912 , piece_id=NIW3LT24E4LPJ5QYJVQOAR43S23JLDDOXZ6NVOURZNDTCCTLLPJA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.698627942 (72h46m22.569710987s) [noise=true , address=145.79.227.39:4014 , piece_id=U7SKMZGV33HMSQLQIFIBZ6WNYC7JJ2CENSK5LCW3KVMP53T7467Q, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.713359143 (72h46m22.584442188s) [noise=true , address=145.79.227.40:4049 , piece_id=VX3PNKMHR75TNJLKRBLYS3KT42ZTPT4X6I5QS2AKNMGDQDVE4WDA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.731975212 (72h46m22.603058257s) [noise=true , address=77.90.31.17:4771 , piece_id=R6PLE6VLZNL23TJC6V6MRBLJP6PKB2DDGQMPPGLMM23WN6J7SH5Q, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.741125236 (72h46m22.612208281s) [noise=true , address=145.79.227.20:4438 , piece_id=U3FCNUY4ASY3ZGX7ZY26GE2NA52GJNKJMW7XKZKALT2MWBCZMHFA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.773771311 (72h46m22.644854346s) [noise=true , address=77.90.31.18:4807 , piece_id=7FXKUHBJT7AHDC4PHCZ73Q62GDSOUFRFIVAD337ORGE7YUTAULVA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:22.906364795 (72h46m22.777447841s) [noise=true , address=77.90.31.14:4662 , piece_id=BO6DQNLFNMG6HAPKOAG2DHWDD3YMX46IAYGDVQG7THC5EVZORJRA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:23.367795705 (72h46m23.23887874s) [noise=true , address=145.79.227.24:4582 , piece_id=FGYLRBVDMMLYWTTKLC3B2P2MHTECHEBH2DJOOKAHBR6TBAMKFBXA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:24.582400944 (72h46m24.453483979s) [noise=true , address=145.79.227.33:4870 , piece_id=IK7NDJD5GNFMWB26JYOMCSNIAT5TTTQOUXB2VFYSNRGIQ2MVMXAQ, seg_pos={0 0}, upload=63694]: Done uploading piece. err: <nil> Dec 8 16:15:24.582421023 (72h46m24.453504068s) [seg_pos={0 0}, upload=63694]: Segment reached optimal threshold of 13 pieces. Dec 8 16:15:24.582652363 (72h46m24.453735408s) [noise=true , address=145.79.227.38:4979 , piece_id=HCELMHS3GNLCKHFCQU3V45M4OETQZPL5MQJBUOUF5VPA44HXZW2Q, seg_pos={0 0}, upload=63694]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma): protocol: expected piece hash; context canceled; EOF Dec 8 16:15:24.582659086 (72h46m24.453742121s) [noise=true , address=145.79.227.38:4979 , piece_id=HCELMHS3GNLCKHFCQU3V45M4OETQZPL5MQJBUOUF5VPA44HXZW2Q, seg_pos={0 0}, upload=63694]: Long tail cancellation due to optimal threshold Dec 8 16:15:24.582740842 (72h46m24.453823877s) [noise=true , address=145.79.227.34:4906 , piece_id=AIZQB5ZLXPCLPT5Q7NLNSI3FPBPAMBVUIXKZCTA3ML7L3XP2UY6A, seg_pos={0 0}, upload=63694]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 8 16:15:24.582753055 (72h46m24.45383609s) [noise=true , address=145.79.227.34:4906 , piece_id=AIZQB5ZLXPCLPT5Q7NLNSI3FPBPAMBVUIXKZCTA3ML7L3XP2UY6A, seg_pos={0 0}, upload=63694]: Long tail cancellation due to optimal threshold Dec 8 16:15:24.583044900 (72h46m24.454127935s) [noise=true , address=145.79.227.29:4941 , piece_id=GJQXPA7WEZUTVJNELRBJVTU4LIGXYD7XLEPR5SWF4NBWWMXBW2SA, seg_pos={0 0}, upload=63694]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: context canceled; context canceled Dec 8 16:15:24.583052926 (72h46m24.454135961s) [noise=true , address=145.79.227.29:4941 , piece_id=GJQXPA7WEZUTVJNELRBJVTU4LIGXYD7XLEPR5SWF4NBWWMXBW2SA, seg_pos={0 0}, upload=63694]: Long tail cancellation due to optimal threshold Dec 8 16:15:24.583071441 (72h46m24.454154486s) [seg_pos={0 0}, upload=63694]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 16:15:24.583162014 (72h46m24.454245059s) [upload=63694]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 16:15:24.583171562 (72h46m24.454254627s) [upload=63694]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 16:15:25.042230695 (72h46m24.91331374s) [upload=63694]: Done uploading segments... Dec 8 16:15:25.042258869 (72h46m24.913341914s) [upload=63694]: Upload finished. err: <nil> Duration: 30.003108878s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.001535505s trace: https://jaeger.dev.storj.io/trace/4dad09bb52e33232 FAIL (2025-12-08 16:15:39 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 30.001560552s core uplink us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 6.534470922s trace: https://jaeger.dev.storj.io/trace/4195524620a3e2c2 FAIL (2025-12-08 16:15:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.771167108 (72h9m11.642250153s) [noise=true , address=81.171.14.70:4032 , piece_id=XAMDRY73RIJV4LC2W2FRMH6L4WL3VAFUYKB3QD4TXOQWWZUYGPOQ, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.771462930 (72h9m11.642545976s) [noise=true , address=81.171.14.70:4035 , piece_id=VYAAT75X4R4J22MLY7RK4VHOEFMYZTLWUWTWVAMOOIIDSEXTQV7Q, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.771589993 (72h9m11.642673038s) [noise=true , address=185.167.204.205:4036 , piece_id=X2TWRAEKE6FEH3ITYY74A76ZZQ2KCB63ETZNW5TAHQ3MLEM34PZQ, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.772028248 (72h9m11.643111293s) [noise=true , address=185.167.204.206:4067 , piece_id=3YOG2SUYBHBE6MMIPMQWMBDZTLL7HPSBPEKFHMGOICBTAQU5HA4A, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.772041683 (72h9m11.643124728s) [noise=true , address=185.167.204.206:4069 , piece_id=GH2ZEUIEIUZ6OLM6XH5ZJVXKAMHUKQ6L3GZ7NJLUZFY623YH5GDQ, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.772603803 (72h9m11.643686848s) [noise=true , address=89.149.201.147:4022 , piece_id=CWWNRXZNWQOUTZEA4WPHGA6BYEPO35D2YDU5VSTHFZ7GRC67Y7VQ, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.773136077 (72h9m11.644219122s) [noise=true , address=185.167.204.206:4051 , piece_id=5JMUWS6LLPRENPBAZ74MDELLIZMWTMI3GOZFUWLCOCUNLDTPETWQ, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.773659244 (72h9m11.644742289s) [noise=true , address=185.167.204.206:4062 , piece_id=KYFUGH5EAVFGEJ2J7VDRDWBDP2R2MX4JSCIU7RP3VJOK4LCM6D2Q, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.774276288 (72h9m11.645359333s) [noise=true , address=81.171.14.70:4026 , piece_id=4BLGOA4GSQVOKZ6UBUIXANALBDHDHDXPLDA3TDQ44ACTIC2IHSEA, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.774811648 (72h9m11.645894693s) [noise=true , address=185.167.204.205:4008 , piece_id=JWZAQAYIUNBFRDVQHADARS5NTEWET7UI2GR73ZMIACA5VFUS3D6Q, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:12.600277390 (72h9m12.471360435s) [upload=63171]: Next returned nil segment Dec 8 15:38:12.600300113 (72h9m12.471383158s) [upload=63171]: Waiting for error group managing segments... Dec 8 15:38:12.984985581 (72h9m12.856068647s) [noise=true , address=185.167.204.205:4028 , piece_id=DCGYFXSKGK6GFLLFU7UQONAX643EA6CDTYWEZ42YQPLEERINL3KQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.985372348 (72h9m12.856455383s) [noise=true , address=185.167.204.205:4008 , piece_id=JWZAQAYIUNBFRDVQHADARS5NTEWET7UI2GR73ZMIACA5VFUS3D6Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.993832749 (72h9m12.864915784s) [noise=true , address=185.167.204.205:4015 , piece_id=GK2HV5AGF2Q6RYHDZBZQ3SY537WQXB7E7UF4SWF5X3LFM4PMCZPQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.994210087 (72h9m12.865293122s) [noise=true , address=185.167.204.206:4053 , piece_id=DGSLZPKB474OADK43GBZO2FGAR2F62PBL364EWVNPCZE3PWZGQPA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.994561226 (72h9m12.865644271s) [noise=true , address=185.167.204.205:4036 , piece_id=X2TWRAEKE6FEH3ITYY74A76ZZQ2KCB63ETZNW5TAHQ3MLEM34PZQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.994944356 (72h9m12.866027401s) [noise=true , address=185.167.204.205:4023 , piece_id=5RNGVQ3LUTPKX4RAVK7Y4WRQC2Y6GT5QWV2G3MTY2LW2MU6IMKLQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.995089352 (72h9m12.866172397s) [noise=true , address=185.167.204.205:4007 , piece_id=WEZIGATHDFL7G3S2VG7BEFGGT7JJIXBDWHGIPHBTPSKDJQUTROIQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.995157291 (72h9m12.866240336s) [noise=true , address=185.167.204.205:4033 , piece_id=4V3DXW42ZCPJFF3PPJEBMBYFW354WEZYLVFUSPGEZ4E4HZ7MFI4Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.000801098 (72h9m12.871884163s) [noise=true , address=185.167.204.206:4062 , piece_id=KYFUGH5EAVFGEJ2J7VDRDWBDP2R2MX4JSCIU7RP3VJOK4LCM6D2Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.000862164 (72h9m12.871945209s) [noise=true , address=185.167.204.206:4051 , piece_id=5JMUWS6LLPRENPBAZ74MDELLIZMWTMI3GOZFUWLCOCUNLDTPETWQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.002117705 (72h9m12.87320074s) [noise=true , address=185.167.204.206:4064 , piece_id=LUITKU5LPQ67MFA4VWE6CDV46QOU7RXOFCGKKAYPCLFGWFSLYLTA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.003015134 (72h9m12.874098179s) [noise=true , address=185.167.204.205:4034 , piece_id=OJPEC5SBHQDKJW3LDBPHMEOPDG6NPNNMO3W54C7XIXTNUYBTMUOA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.003432589 (72h9m12.874515624s) [noise=true , address=185.167.204.206:4037 , piece_id=OPF6NTGFZ7FSN7LIEKHOGIJRYUWX27RTI3KKGCKPCOQLK4HC62LQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.004078648 (72h9m12.875161694s) [noise=true , address=185.167.204.206:4069 , piece_id=GH2ZEUIEIUZ6OLM6XH5ZJVXKAMHUKQ6L3GZ7NJLUZFY623YH5GDQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.009788571 (72h9m12.880871616s) [noise=true , address=185.167.204.206:4067 , piece_id=3YOG2SUYBHBE6MMIPMQWMBDZTLL7HPSBPEKFHMGOICBTAQU5HA4A, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.012083561 (72h9m12.883166596s) [noise=true , address=185.167.204.205:4030 , piece_id=EFAUOQCDA25VE55ZHJOSVXTOQD2VTQCW55MEH5DBEAY7N364GMXA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.584995241 (72h9m13.456078286s) [noise=true , address=81.171.14.70:4046 , piece_id=77RPCXTIQW7MO6UKAVJUD4Y7XR674IDZE2TL2SB77LJUHRE3ACDA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.651263008 (72h9m13.522346063s) [noise=true , address=81.171.14.70:4026 , piece_id=4BLGOA4GSQVOKZ6UBUIXANALBDHDHDXPLDA3TDQ44ACTIC2IHSEA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.696266204 (72h9m13.567349249s) [noise=true , address=81.171.14.70:4035 , piece_id=VYAAT75X4R4J22MLY7RK4VHOEFMYZTLWUWTWVAMOOIIDSEXTQV7Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.123999844 (72h9m13.995082889s) [noise=true , address=81.171.14.70:4025 , piece_id=XML4X66PHAT33VTUTEV22HSTIHMG4WFZU6YSDPCZZZ4SEBL3NWUA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.147808461 (72h9m14.018891506s) [noise=true , address=81.171.14.70:4032 , piece_id=XAMDRY73RIJV4LC2W2FRMH6L4WL3VAFUYKB3QD4TXOQWWZUYGPOQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.255119813 (72h9m14.126202858s) [noise=true , address=89.149.201.147:4005 , piece_id=MBR3CYKQ3YSPVXIVSL6ABGZNG7FB5HGP45OA3YE2IRKPJMOSM3KQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.271452473 (72h9m14.142535518s) [noise=true , address=89.149.201.147:4013 , piece_id=KIPMWZVYZADDS2LAOY5ASMIWP32HIRTNGYQ65SNLYR2EQDQCN3HQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.775652542 (72h9m14.646735587s) [noise=true , address=89.149.201.147:4016 , piece_id=5N5HKBPMDQUEQIZZVFFRQGE4LQAGSJ4H6INPDXHYQMDK5YTHUVKQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.785904314 (72h9m14.656987359s) [noise=true , address=89.149.201.147:4017 , piece_id=MBD47F6TJHZ76PAGU7ZIFRQ4M5HUDT5AQULHE6RPQPBUBAVKSTKQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.827349233 (72h9m14.698432278s) [noise=true , address=89.149.201.147:4022 , piece_id=CWWNRXZNWQOUTZEA4WPHGA6BYEPO35D2YDU5VSTHFZ7GRC67Y7VQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.827365925 (72h9m14.69844897s) [seg_pos={0 0}, upload=63171]: Segment reached optimal threshold of 26 pieces. Dec 8 15:38:14.827539125 (72h9m14.69862217s) [noise=true , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hJ5jT6S2d3DJe9DoVDjHMkpwSTpeyTMyNLaK8Pk8SWcUVRKnv): protocol: expected piece hash; context canceled; EOF Dec 8 15:38:14.827545567 (72h9m14.698628612s) [noise=true , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Long tail cancellation due to optimal threshold Dec 8 15:38:14.827903338 (72h9m14.698986373s) [noise=true , address=89.149.201.147:4018 , piece_id=BFJH5ELBTZD767FJNFT2ZBUS3GI62IZIRSKYARLVBCLWZONSURMQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua): protocol: expected piece hash; context canceled; EOF Dec 8 15:38:14.827910673 (72h9m14.698993708s) [noise=true , address=89.149.201.147:4018 , piece_id=BFJH5ELBTZD767FJNFT2ZBUS3GI62IZIRSKYARLVBCLWZONSURMQ, seg_pos={0 0}, upload=63171]: Long tail cancellation due to optimal threshold Dec 8 15:38:14.827928065 (72h9m14.69901111s) [seg_pos={0 0}, upload=63171]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 8 15:38:14.828037083 (72h9m14.699120128s) [upload=63171]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 15:38:14.828053224 (72h9m14.699136269s) [upload=63171]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 15:39:18.272006268 (72h10m18.143089313s) [upload=63171]: Done uploading segments... Dec 8 15:39:18.272087593 (72h10m18.143170628s) [upload=63171]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.534498353s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.XX...X.X.X.X....XX......X....X..........................X..] duration: 17.135026657s trace: https://jaeger.dev.storj.io/trace/12eb27369d16b107 FAIL (2025-12-08 16:15:38 UTC) orbiter warning: context deadline exceeded Duration: 17.135057987s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 23 out of 60 is failed (threshold=6) [.X.X.XXXX.X...XX.X..XX.XX.X..XX.....X.....X...X..XX..X......] duration: 3.54443333s trace: https://jaeger.dev.storj.io/trace/44072b2910bf5c31 FAIL (2025-12-08 16:15:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.544458459s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [..............X..X...X.................X.X..................] duration: 5.835058534s trace: https://jaeger.dev.storj.io/trace/690f79ef204fee2 FAIL (2025-12-08 16:15:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.835139588s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 1.42429876s trace: https://jaeger.dev.storj.io/trace/3bdb8560e64b821d FAIL (2025-12-08 16:15:18 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 1.424333955s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.X..............X.............X............................X] duration: 30.002797718s trace: https://jaeger.dev.storj.io/trace/5b80613b41363e55 FAIL (2025-12-08 16:15:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, Piece ID: HEYMJJAQOUAAD34SM2BUU26RZEPJUIGUUSQHKQTX5ZZFH2WIVWGQ) context deadline exceeded; piecestore: (Node ID: 1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, Piece ID: A4OYPDKRD33AUNMH5PHLYFRQUWMPZSY5KVBKAQLW5TDIWVYUHVVA) context deadline exceeded ======LOGS====== Dec 8 16:14:42.881174216 (72h45m42.752257261s) [upload=63686]: Starting upload Dec 8 16:14:42.881235553 (72h45m42.752318588s) [upload=63686]: Done starting upload Dec 8 16:14:42.881366752 (72h45m42.752449797s) [upload=63686]: Uploading segments... Dec 8 16:14:42.881468365 (72h45m42.75255141s) [upload=63686]: Got next segment. Inline: false Dec 8 16:14:42.881512620 (72h45m42.752595665s) [upload=63686]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 16:14:42.881540132 (72h45m42.752623177s) [upload=63686]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 16:14:43.131206078 (72h45m43.002289123s) [seg_pos={0 0}, upload=63686]: Begin upload segment... Dec 8 16:14:43.131263728 (72h45m43.002346773s) [seg_pos={0 0}, upload=63686]: Done begin upload segment. Dec 8 16:14:43.131309685 (72h45m43.00239272s) [noise=true , address=77.90.31.19:4842 , piece_id=RIFEEW4GSL2EHVQMIJAJJHOPVJRJX4LYE2IPXXOLDKKQJW4T5RJA, seg_pos={0 0}, upload=63686]: Uploading piece... Dec 8 16:14:43.131322028 (72h45m43.002405073s) [noise=true , address=77.90.31.12:4590 , piece_id=NO27GUAAM6PD4XHCM7ZDHRE7FDZZT4SKOFN7OWW3ZPWRCFMMVMKA, seg_pos={0 0}, upload=63686]: Uploading piece... Dec 8 16:14:43.131962879 (72h45m43.003045924s) [noise=true , address=145.79.227.41:4086 , piece_id=MVUTPTLCMQPXVEJ37XTSXXTIIXMZTOGTH6XXUTYGZSPN4V2UYKSQ, seg_pos={0 0}, upload=63686]: Uploading piece... Dec 8 16:14:43.132367249 (72h45m43.003450294s) [noise=true , address=145.79.227.39:4013 , piece_id=PZOOHYLWUF5ODWA6EVCUFQFOGNQQTDPTGPLBLJSBERW37KH7NSOQ, seg_pos={0 0}, upload=63686]: Uploading piece... Dec 8 16:14:43.132781167 (72h45m43.003864212s) [noise=true , address=145.79.227.31:4797 , piece_id=6MWZ57CJBOTQE3JSSQ55VO6YDZVUF6MZWHT574KX4IDI2MX7INKA, seg_pos={0 0}, upload=63686]: Uploading piece... Dec 8 16:14:43.133917581 (72h45m43.005000616s) [noise=true , address=145.79.227.38:4977 , piece_id=GRXNN2XHDT2GSCCGA63EOOUYHQNE35BSD5AOZ3G3JFW5IT7SZ2QQ, seg_pos={0 0}, upload=63686]: Uploading piece... Dec 8 16:14:43.134482166 (72h45m43.005565211s) [noise=true , address=145.79.227.29:4942 , piece_id=45RBCVENKUBAOM4UDNCEOFJOLUB7WXPDDPLBJNF7K4PFOC67PJWQ, seg_pos={0 0}, upload=63686]: Uploading piece... Dec 8 16:14:43.134940328 (72h45m43.006023363s) [noise=true , address=145.79.227.9:4366 , piece_id=A4OYPDKRD33AUNMH5PHLYFRQUWMPZSY5KVBKAQLW5TDIWVYUHVVA, seg_pos={0 0}, upload=63686]: Uploading piece... Dec 8 16:14:43.135873976 (72h45m43.006957021s) [noise=true , address=145.79.227.33:4870 , piece_id=XLUQ75TQX4DNGW3MN6NG52HXC7IT6VA3OEAF2MBRA2RTNGGA34WQ, seg_pos={0 0}, upload=63686]: Uploading piece... Dec 8 16:14:43.136599037 (72h45m43.007682072s) [noise=true , address=77.90.31.14:4661 , piece_id=CSE5RXIKSTYE7XZ6L7A3M3XDIYWRKUVOZBYD3TC2KAPL7GWJIXPQ, seg_pos={0 0}, upload=63686]: Uploading piece... Dec 8 16:14:43.137929009 (72h45m43.009012054s) [noise=true , address=145.79.227.33:4869 , piece_id=B3S73OMH7QNVWVU6KB6ZA2QA2LZZOXXK27COG6ZWVQNC2OKKEFSQ, seg_pos={0 0}, upload=63686]: Uploading piece... Dec 8 16:14:43.139585915 (72h45m43.01066896s) [noise=true , address=145.79.227.26:4653 , piece_id=WI3XJAP5FO3N7XSG5MRLDRV47ENKUZGN2A3YJ4ZTBDKFRZZ4DRBA, seg_pos={0 0}, upload=63686]: Uploading piece... Dec 8 16:14:43.141140445 (72h45m43.01222349s) [noise=true , address=145.79.227.25:4618 , piece_id=HEYMJJAQOUAAD34SM2BUU26RZEPJUIGUUSQHKQTX5ZZFH2WIVWGQ, seg_pos={0 0}, upload=63686]: Uploading piece... Dec 8 16:14:43.142700315 (72h45m43.01378336s) [noise=true , address=77.90.31.18:4807 , piece_id=UWTOXKYTRO6UJDV4DWCHFAB2KFPW32ADVMZGBLRALCJP554DYAHA, seg_pos={0 0}, upload=63686]: Uploading piece... Dec 8 16:14:43.144120569 (72h45m43.015203614s) [noise=true , address=145.79.227.21:4474 , piece_id=DVGNRAIJRGZXR62YOKJ44NX3XQ2VF5HYG6PCMZY3DDUD7QGMOSNA, seg_pos={0 0}, upload=63686]: Uploading piece... Dec 8 16:14:43.145665472 (72h45m43.016748507s) [noise=true , address=145.79.227.43:4157 , piece_id=BZTOVFD3SBYJIMRU7LG4HPWMF2OVYY6CMR5DHVBPWGAXQZEZDEWQ, seg_pos={0 0}, upload=63686]: Uploading piece... Dec 8 16:14:44.936607565 (72h45m44.80769062s) [upload=63686]: Next returned nil segment Dec 8 16:14:44.936642581 (72h45m44.807725626s) [upload=63686]: Waiting for error group managing segments... Dec 8 16:14:45.238683580 (72h45m45.109766615s) [noise=true , address=145.79.227.21:4474 , piece_id=DVGNRAIJRGZXR62YOKJ44NX3XQ2VF5HYG6PCMZY3DDUD7QGMOSNA, seg_pos={0 0}, upload=63686]: Done uploading piece. err: <nil> Dec 8 16:14:45.249393923 (72h45m45.120476958s) [noise=true , address=145.79.227.39:4013 , piece_id=PZOOHYLWUF5ODWA6EVCUFQFOGNQQTDPTGPLBLJSBERW37KH7NSOQ, seg_pos={0 0}, upload=63686]: Done uploading piece. err: <nil> Dec 8 16:14:45.251493813 (72h45m45.122576848s) [noise=true , address=145.79.227.29:4942 , piece_id=45RBCVENKUBAOM4UDNCEOFJOLUB7WXPDDPLBJNF7K4PFOC67PJWQ, seg_pos={0 0}, upload=63686]: Done uploading piece. err: <nil> Dec 8 16:14:45.255551379 (72h45m45.126634414s) [noise=true , address=145.79.227.9:4366 , piece_id=A4OYPDKRD33AUNMH5PHLYFRQUWMPZSY5KVBKAQLW5TDIWVYUHVVA, seg_pos={0 0}, upload=63686]: Done uploading piece. err: <nil> Dec 8 16:14:45.302943874 (72h45m45.17402691s) [noise=true , address=145.79.227.33:4870 , piece_id=XLUQ75TQX4DNGW3MN6NG52HXC7IT6VA3OEAF2MBRA2RTNGGA34WQ, seg_pos={0 0}, upload=63686]: Done uploading piece. err: <nil> Dec 8 16:14:45.333453150 (72h45m45.204536185s) [noise=true , address=77.90.31.19:4842 , piece_id=RIFEEW4GSL2EHVQMIJAJJHOPVJRJX4LYE2IPXXOLDKKQJW4T5RJA, seg_pos={0 0}, upload=63686]: Done uploading piece. err: <nil> Dec 8 16:14:45.345207072 (72h45m45.216290107s) [noise=true , address=145.79.227.41:4086 , piece_id=MVUTPTLCMQPXVEJ37XTSXXTIIXMZTOGTH6XXUTYGZSPN4V2UYKSQ, seg_pos={0 0}, upload=63686]: Done uploading piece. err: <nil> Dec 8 16:14:45.365471299 (72h45m45.236554344s) [noise=true , address=77.90.31.14:4661 , piece_id=CSE5RXIKSTYE7XZ6L7A3M3XDIYWRKUVOZBYD3TC2KAPL7GWJIXPQ, seg_pos={0 0}, upload=63686]: Done uploading piece. err: <nil> Dec 8 16:14:45.381668081 (72h45m45.252751127s) [noise=true , address=77.90.31.18:4807 , piece_id=UWTOXKYTRO6UJDV4DWCHFAB2KFPW32ADVMZGBLRALCJP554DYAHA, seg_pos={0 0}, upload=63686]: Done uploading piece. err: <nil> Dec 8 16:14:45.426700432 (72h45m45.297783477s) [noise=true , address=77.90.31.12:4590 , piece_id=NO27GUAAM6PD4XHCM7ZDHRE7FDZZT4SKOFN7OWW3ZPWRCFMMVMKA, seg_pos={0 0}, upload=63686]: Done uploading piece. err: <nil> Dec 8 16:14:46.314856303 (72h45m46.185939348s) [noise=true , address=145.79.227.25:4618 , piece_id=HEYMJJAQOUAAD34SM2BUU26RZEPJUIGUUSQHKQTX5ZZFH2WIVWGQ, seg_pos={0 0}, upload=63686]: Done uploading piece. err: <nil> Dec 8 16:14:46.487603220 (72h45m46.358686255s) [noise=true , address=145.79.227.38:4977 , piece_id=GRXNN2XHDT2GSCCGA63EOOUYHQNE35BSD5AOZ3G3JFW5IT7SZ2QQ, seg_pos={0 0}, upload=63686]: Done uploading piece. err: <nil> Dec 8 16:14:46.901578664 (72h45m46.772661699s) [noise=true , address=145.79.227.31:4797 , piece_id=6MWZ57CJBOTQE3JSSQ55VO6YDZVUF6MZWHT574KX4IDI2MX7INKA, seg_pos={0 0}, upload=63686]: Done uploading piece. err: <nil> Dec 8 16:14:46.901600666 (72h45m46.772683701s) [seg_pos={0 0}, upload=63686]: Segment reached optimal threshold of 13 pieces. Dec 8 16:14:46.901794064 (72h45m46.772877099s) [noise=true , address=145.79.227.33:4869 , piece_id=B3S73OMH7QNVWVU6KB6ZA2QA2LZZOXXK27COG6ZWVQNC2OKKEFSQ, seg_pos={0 0}, upload=63686]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Dec 8 16:14:46.901836805 (72h45m46.772919841s) [noise=true , address=145.79.227.33:4869 , piece_id=B3S73OMH7QNVWVU6KB6ZA2QA2LZZOXXK27COG6ZWVQNC2OKKEFSQ, seg_pos={0 0}, upload=63686]: Long tail cancellation due to optimal threshold Dec 8 16:14:46.902083986 (72h45m46.773167021s) [noise=true , address=145.79.227.26:4653 , piece_id=WI3XJAP5FO3N7XSG5MRLDRV47ENKUZGN2A3YJ4ZTBDKFRZZ4DRBA, seg_pos={0 0}, upload=63686]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ): protocol: expected piece hash; context canceled; EOF Dec 8 16:14:46.902092343 (72h45m46.773175388s) [noise=true , address=145.79.227.26:4653 , piece_id=WI3XJAP5FO3N7XSG5MRLDRV47ENKUZGN2A3YJ4ZTBDKFRZZ4DRBA, seg_pos={0 0}, upload=63686]: Long tail cancellation due to optimal threshold Dec 8 16:14:46.902148018 (72h45m46.773231053s) [noise=true , address=145.79.227.43:4157 , piece_id=BZTOVFD3SBYJIMRU7LG4HPWMF2OVYY6CMR5DHVBPWGAXQZEZDEWQ, seg_pos={0 0}, upload=63686]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz): protocol: read tcp 188.166.225.170:37658->145.79.227.43:4157: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 16:14:46.902155703 (72h45m46.773238738s) [noise=true , address=145.79.227.43:4157 , piece_id=BZTOVFD3SBYJIMRU7LG4HPWMF2OVYY6CMR5DHVBPWGAXQZEZDEWQ, seg_pos={0 0}, upload=63686]: Long tail cancellation due to optimal threshold Dec 8 16:14:46.902182885 (72h45m46.77326592s) [seg_pos={0 0}, upload=63686]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 16:14:46.902249732 (72h45m46.773332767s) [upload=63686]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 16:14:46.902255763 (72h45m46.773338798s) [upload=63686]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 16:14:47.217193791 (72h45m47.088276847s) [upload=63686]: Done uploading segments... Dec 8 16:14:47.217215984 (72h45m47.088299019s) [upload=63686]: Upload finished. err: <nil> Duration: 30.002823907s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [...............X..X...X.................X.X.................] duration: 22.966008225s trace: https://jaeger.dev.storj.io/trace/690f79ef204fee2 FAIL (2025-12-08 16:15:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 22.966022442s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 24 out of 60 is failed (threshold=6) [X.X.X.XXXX.X...XX.X..XX.XX.X..XX.....X.....X...X..XX..X.....] duration: 3.450503199s trace: https://jaeger.dev.storj.io/trace/44072b2910bf5c31 FAIL (2025-12-08 16:14:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.450553064s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 4.99694755s trace: https://jaeger.dev.storj.io/trace/5d817b557064ae10 FAIL (2025-12-08 16:14:54 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.996975284s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 1.418512651s trace: https://jaeger.dev.storj.io/trace/3bdb8560e64b821d FAIL (2025-12-08 16:14:50 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 1.418540764s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 6.04960291s trace: https://jaeger.dev.storj.io/trace/4195524620a3e2c2 FAIL (2025-12-08 16:14:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.771167108 (72h9m11.642250153s) [noise=true , address=81.171.14.70:4032 , piece_id=XAMDRY73RIJV4LC2W2FRMH6L4WL3VAFUYKB3QD4TXOQWWZUYGPOQ, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.771462930 (72h9m11.642545976s) [noise=true , address=81.171.14.70:4035 , piece_id=VYAAT75X4R4J22MLY7RK4VHOEFMYZTLWUWTWVAMOOIIDSEXTQV7Q, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.771589993 (72h9m11.642673038s) [noise=true , address=185.167.204.205:4036 , piece_id=X2TWRAEKE6FEH3ITYY74A76ZZQ2KCB63ETZNW5TAHQ3MLEM34PZQ, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.772028248 (72h9m11.643111293s) [noise=true , address=185.167.204.206:4067 , piece_id=3YOG2SUYBHBE6MMIPMQWMBDZTLL7HPSBPEKFHMGOICBTAQU5HA4A, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.772041683 (72h9m11.643124728s) [noise=true , address=185.167.204.206:4069 , piece_id=GH2ZEUIEIUZ6OLM6XH5ZJVXKAMHUKQ6L3GZ7NJLUZFY623YH5GDQ, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.772603803 (72h9m11.643686848s) [noise=true , address=89.149.201.147:4022 , piece_id=CWWNRXZNWQOUTZEA4WPHGA6BYEPO35D2YDU5VSTHFZ7GRC67Y7VQ, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.773136077 (72h9m11.644219122s) [noise=true , address=185.167.204.206:4051 , piece_id=5JMUWS6LLPRENPBAZ74MDELLIZMWTMI3GOZFUWLCOCUNLDTPETWQ, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.773659244 (72h9m11.644742289s) [noise=true , address=185.167.204.206:4062 , piece_id=KYFUGH5EAVFGEJ2J7VDRDWBDP2R2MX4JSCIU7RP3VJOK4LCM6D2Q, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.774276288 (72h9m11.645359333s) [noise=true , address=81.171.14.70:4026 , piece_id=4BLGOA4GSQVOKZ6UBUIXANALBDHDHDXPLDA3TDQ44ACTIC2IHSEA, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.774811648 (72h9m11.645894693s) [noise=true , address=185.167.204.205:4008 , piece_id=JWZAQAYIUNBFRDVQHADARS5NTEWET7UI2GR73ZMIACA5VFUS3D6Q, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:12.600277390 (72h9m12.471360435s) [upload=63171]: Next returned nil segment Dec 8 15:38:12.600300113 (72h9m12.471383158s) [upload=63171]: Waiting for error group managing segments... Dec 8 15:38:12.984985581 (72h9m12.856068647s) [noise=true , address=185.167.204.205:4028 , piece_id=DCGYFXSKGK6GFLLFU7UQONAX643EA6CDTYWEZ42YQPLEERINL3KQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.985372348 (72h9m12.856455383s) [noise=true , address=185.167.204.205:4008 , piece_id=JWZAQAYIUNBFRDVQHADARS5NTEWET7UI2GR73ZMIACA5VFUS3D6Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.993832749 (72h9m12.864915784s) [noise=true , address=185.167.204.205:4015 , piece_id=GK2HV5AGF2Q6RYHDZBZQ3SY537WQXB7E7UF4SWF5X3LFM4PMCZPQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.994210087 (72h9m12.865293122s) [noise=true , address=185.167.204.206:4053 , piece_id=DGSLZPKB474OADK43GBZO2FGAR2F62PBL364EWVNPCZE3PWZGQPA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.994561226 (72h9m12.865644271s) [noise=true , address=185.167.204.205:4036 , piece_id=X2TWRAEKE6FEH3ITYY74A76ZZQ2KCB63ETZNW5TAHQ3MLEM34PZQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.994944356 (72h9m12.866027401s) [noise=true , address=185.167.204.205:4023 , piece_id=5RNGVQ3LUTPKX4RAVK7Y4WRQC2Y6GT5QWV2G3MTY2LW2MU6IMKLQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.995089352 (72h9m12.866172397s) [noise=true , address=185.167.204.205:4007 , piece_id=WEZIGATHDFL7G3S2VG7BEFGGT7JJIXBDWHGIPHBTPSKDJQUTROIQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.995157291 (72h9m12.866240336s) [noise=true , address=185.167.204.205:4033 , piece_id=4V3DXW42ZCPJFF3PPJEBMBYFW354WEZYLVFUSPGEZ4E4HZ7MFI4Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.000801098 (72h9m12.871884163s) [noise=true , address=185.167.204.206:4062 , piece_id=KYFUGH5EAVFGEJ2J7VDRDWBDP2R2MX4JSCIU7RP3VJOK4LCM6D2Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.000862164 (72h9m12.871945209s) [noise=true , address=185.167.204.206:4051 , piece_id=5JMUWS6LLPRENPBAZ74MDELLIZMWTMI3GOZFUWLCOCUNLDTPETWQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.002117705 (72h9m12.87320074s) [noise=true , address=185.167.204.206:4064 , piece_id=LUITKU5LPQ67MFA4VWE6CDV46QOU7RXOFCGKKAYPCLFGWFSLYLTA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.003015134 (72h9m12.874098179s) [noise=true , address=185.167.204.205:4034 , piece_id=OJPEC5SBHQDKJW3LDBPHMEOPDG6NPNNMO3W54C7XIXTNUYBTMUOA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.003432589 (72h9m12.874515624s) [noise=true , address=185.167.204.206:4037 , piece_id=OPF6NTGFZ7FSN7LIEKHOGIJRYUWX27RTI3KKGCKPCOQLK4HC62LQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.004078648 (72h9m12.875161694s) [noise=true , address=185.167.204.206:4069 , piece_id=GH2ZEUIEIUZ6OLM6XH5ZJVXKAMHUKQ6L3GZ7NJLUZFY623YH5GDQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.009788571 (72h9m12.880871616s) [noise=true , address=185.167.204.206:4067 , piece_id=3YOG2SUYBHBE6MMIPMQWMBDZTLL7HPSBPEKFHMGOICBTAQU5HA4A, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.012083561 (72h9m12.883166596s) [noise=true , address=185.167.204.205:4030 , piece_id=EFAUOQCDA25VE55ZHJOSVXTOQD2VTQCW55MEH5DBEAY7N364GMXA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.584995241 (72h9m13.456078286s) [noise=true , address=81.171.14.70:4046 , piece_id=77RPCXTIQW7MO6UKAVJUD4Y7XR674IDZE2TL2SB77LJUHRE3ACDA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.651263008 (72h9m13.522346063s) [noise=true , address=81.171.14.70:4026 , piece_id=4BLGOA4GSQVOKZ6UBUIXANALBDHDHDXPLDA3TDQ44ACTIC2IHSEA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.696266204 (72h9m13.567349249s) [noise=true , address=81.171.14.70:4035 , piece_id=VYAAT75X4R4J22MLY7RK4VHOEFMYZTLWUWTWVAMOOIIDSEXTQV7Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.123999844 (72h9m13.995082889s) [noise=true , address=81.171.14.70:4025 , piece_id=XML4X66PHAT33VTUTEV22HSTIHMG4WFZU6YSDPCZZZ4SEBL3NWUA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.147808461 (72h9m14.018891506s) [noise=true , address=81.171.14.70:4032 , piece_id=XAMDRY73RIJV4LC2W2FRMH6L4WL3VAFUYKB3QD4TXOQWWZUYGPOQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.255119813 (72h9m14.126202858s) [noise=true , address=89.149.201.147:4005 , piece_id=MBR3CYKQ3YSPVXIVSL6ABGZNG7FB5HGP45OA3YE2IRKPJMOSM3KQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.271452473 (72h9m14.142535518s) [noise=true , address=89.149.201.147:4013 , piece_id=KIPMWZVYZADDS2LAOY5ASMIWP32HIRTNGYQ65SNLYR2EQDQCN3HQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.775652542 (72h9m14.646735587s) [noise=true , address=89.149.201.147:4016 , piece_id=5N5HKBPMDQUEQIZZVFFRQGE4LQAGSJ4H6INPDXHYQMDK5YTHUVKQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.785904314 (72h9m14.656987359s) [noise=true , address=89.149.201.147:4017 , piece_id=MBD47F6TJHZ76PAGU7ZIFRQ4M5HUDT5AQULHE6RPQPBUBAVKSTKQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.827349233 (72h9m14.698432278s) [noise=true , address=89.149.201.147:4022 , piece_id=CWWNRXZNWQOUTZEA4WPHGA6BYEPO35D2YDU5VSTHFZ7GRC67Y7VQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.827365925 (72h9m14.69844897s) [seg_pos={0 0}, upload=63171]: Segment reached optimal threshold of 26 pieces. Dec 8 15:38:14.827539125 (72h9m14.69862217s) [noise=true , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hJ5jT6S2d3DJe9DoVDjHMkpwSTpeyTMyNLaK8Pk8SWcUVRKnv): protocol: expected piece hash; context canceled; EOF Dec 8 15:38:14.827545567 (72h9m14.698628612s) [noise=true , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Long tail cancellation due to optimal threshold Dec 8 15:38:14.827903338 (72h9m14.698986373s) [noise=true , address=89.149.201.147:4018 , piece_id=BFJH5ELBTZD767FJNFT2ZBUS3GI62IZIRSKYARLVBCLWZONSURMQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua): protocol: expected piece hash; context canceled; EOF Dec 8 15:38:14.827910673 (72h9m14.698993708s) [noise=true , address=89.149.201.147:4018 , piece_id=BFJH5ELBTZD767FJNFT2ZBUS3GI62IZIRSKYARLVBCLWZONSURMQ, seg_pos={0 0}, upload=63171]: Long tail cancellation due to optimal threshold Dec 8 15:38:14.827928065 (72h9m14.69901111s) [seg_pos={0 0}, upload=63171]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 8 15:38:14.828037083 (72h9m14.699120128s) [upload=63171]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 15:38:14.828053224 (72h9m14.699136269s) [upload=63171]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 15:39:18.272006268 (72h10m18.143089313s) [upload=63171]: Done uploading segments... Dec 8 15:39:18.272087593 (72h10m18.143170628s) [upload=63171]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.049630974s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [XX.X.X.XXXX.X...XX.X..XX.XX.X..XX.....X.....X...X..XX..X....] duration: 3.4816633s trace: https://jaeger.dev.storj.io/trace/44072b2910bf5c31 FAIL (2025-12-08 16:14:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.481731169s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [..XX...X.X.X.X....XX......X....X..........................X.] duration: 6.333391778s trace: https://jaeger.dev.storj.io/trace/12eb27369d16b107 FAIL (2025-12-08 16:14:13 UTC) orbiter warning: context deadline exceeded Duration: 6.333420221s edge gateway us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 5.065227804s trace: https://jaeger.dev.storj.io/trace/5d817b557064ae10 FAIL (2025-12-08 16:14:06 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.065247161s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 1.448605227s trace: https://jaeger.dev.storj.io/trace/3bdb8560e64b821d FAIL (2025-12-08 16:14:02 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 1.448633892s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 15m5.542350681s trace: https://jaeger.dev.storj.io/trace/5d817b557064ae10 FAIL (2025-12-08 16:14:01 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 15m5.542365169s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..X..............X.............X............................] duration: 5.836223232s trace: https://jaeger.dev.storj.io/trace/4213b059398cd95c FAIL (2025-12-08 16:13:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, Piece ID: PRU4WN2RSHGSMLJQUJFHG7J5OGN5EPJ6H5K2PWGFSDXGLRAPRZ7A) context deadline exceeded; piecestore: (Node ID: 1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, Piece ID: ZL3BLXO3N44PN32GABZY7LCUJCI2OTOALU3UNVA2PR7VLNBRKLPQ) context deadline exceeded; piecestore: (Node ID: 1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV, Piece ID: H3A7JCGL7NYGYKEIMKBSSVX7RPONMWFIKYWJRQ6RI7BS324BFYRA) context deadline exceeded ======LOGS====== Dec 8 15:53:18.157362917 (72h24m18.028445962s) [upload=63391]: Starting upload Dec 8 15:53:18.157453920 (72h24m18.028536985s) [upload=63391]: Done starting upload Dec 8 15:53:18.157573969 (72h24m18.028657014s) [upload=63391]: Uploading segments... Dec 8 15:53:18.157710549 (72h24m18.028793584s) [upload=63391]: Got next segment. Inline: false Dec 8 15:53:18.157780753 (72h24m18.028863788s) [upload=63391]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 15:53:18.157787826 (72h24m18.028870871s) [upload=63391]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 15:53:18.404301207 (72h24m18.275384262s) [seg_pos={0 0}, upload=63391]: Begin upload segment... Dec 8 15:53:18.404366131 (72h24m18.275449176s) [seg_pos={0 0}, upload=63391]: Done begin upload segment. Dec 8 15:53:18.404406237 (72h24m18.275489282s) [noise=true , address=145.79.227.32:4835 , piece_id=AVX3WXR3GKZGMSZTK6KFWCAZTEXSODMLATQVH5O3LDQI5UK7R5JQ, seg_pos={0 0}, upload=63391]: Uploading piece... Dec 8 15:53:18.404413581 (72h24m18.275496616s) [noise=true , address=77.90.31.10:4517 , piece_id=F6XFHVULC43K2X35O6TCPEHDKLHHB2O7G6IYQNLXC7LITHVC32PA, seg_pos={0 0}, upload=63391]: Uploading piece... Dec 8 15:53:18.404802713 (72h24m18.275885758s) [noise=true , address=145.79.227.42:4121 , piece_id=4ASZMA5IVHSELKSS2M6S3H5NLFCG7FI63PDP7DP6YZAVO4VMBQMQ, seg_pos={0 0}, upload=63391]: Uploading piece... Dec 8 15:53:18.405306271 (72h24m18.276389316s) [noise=true , address=145.79.227.26:4654 , piece_id=HMSWSSE2KTZ7YD3PT7KGBM4EK26R34RTOIZMJFAWXLK7IWUNALJQ, seg_pos={0 0}, upload=63391]: Uploading piece... Dec 8 15:53:18.405765786 (72h24m18.276848821s) [noise=true , address=145.79.227.21:4473 , piece_id=X5NWPG5J5VWO2H6JO2WQG474GIIEKMBATL4ZCHBJVXBIJ6EC6UJA, seg_pos={0 0}, upload=63391]: Uploading piece... Dec 8 15:53:18.406268273 (72h24m18.277351319s) [noise=true , address=145.79.227.20:4437 , piece_id=H3A7JCGL7NYGYKEIMKBSSVX7RPONMWFIKYWJRQ6RI7BS324BFYRA, seg_pos={0 0}, upload=63391]: Uploading piece... Dec 8 15:53:18.406748387 (72h24m18.277831432s) [noise=true , address=77.90.31.11:4555 , piece_id=YCU5C7D5SRVMV2ZRNR63NIJ344PSTN4L2D4UQOGPXWWBPYQIUZRQ, seg_pos={0 0}, upload=63391]: Uploading piece... Dec 8 15:53:18.407223522 (72h24m18.278306567s) [noise=true , address=145.79.227.39:4015 , piece_id=PRU4WN2RSHGSMLJQUJFHG7J5OGN5EPJ6H5K2PWGFSDXGLRAPRZ7A, seg_pos={0 0}, upload=63391]: Uploading piece... Dec 8 15:53:18.407763771 (72h24m18.278846816s) [noise=true , address=145.79.227.25:4618 , piece_id=7SURNL5URXOPNLN6LMAWSD5AJUJQMM6JLOWMUMG43ONXB7B4XHZQ, seg_pos={0 0}, upload=63391]: Uploading piece... Dec 8 15:53:18.409872616 (72h24m18.280955671s) [noise=true , address=77.90.31.18:4806 , piece_id=H27DLSIFHPGEEPD772CXBTPGHSKFFJKYC5EZBSFHITASPSYJ2GZA, seg_pos={0 0}, upload=63391]: Uploading piece... Dec 8 15:53:18.410599110 (72h24m18.281682145s) [noise=true , address=145.79.227.31:4797 , piece_id=FFONIBKZ4433V2Q3LBVCAP6PEDDMV4MHLKOYEHDTQ6UIBAGHZUMQ, seg_pos={0 0}, upload=63391]: Uploading piece... Dec 8 15:53:18.412764604 (72h24m18.283847659s) [noise=true , address=145.79.227.21:4474 , piece_id=ZL3BLXO3N44PN32GABZY7LCUJCI2OTOALU3UNVA2PR7VLNBRKLPQ, seg_pos={0 0}, upload=63391]: Uploading piece... Dec 8 15:53:18.415050376 (72h24m18.286133421s) [noise=true , address=145.79.227.43:4159 , piece_id=BIXIBLUU2IBMUFNHMM4X4WMUD4FUICY6ENXGUBDXYYZW5CNSW4XA, seg_pos={0 0}, upload=63391]: Uploading piece... Dec 8 15:53:18.418752035 (72h24m18.28983509s) [noise=true , address=145.79.227.44:4195 , piece_id=HLHR4HM53WUWQB3HIWMFIHNXW4YUKMEVZE4DPK3HUK6ESQUCMMVA, seg_pos={0 0}, upload=63391]: Uploading piece... Dec 8 15:53:18.422974876 (72h24m18.294057941s) [noise=true , address=77.90.31.10:4518 , piece_id=6EFXAYNL6RY2RHFPH6DOKVZUA3GDRIPVHZZBLYL36XWRHIVPHX4Q, seg_pos={0 0}, upload=63391]: Uploading piece... Dec 8 15:53:18.423256632 (72h24m18.294339677s) [noise=true , address=145.79.227.22:4510 , piece_id=WQL7Q5FJFKYASAFPSDM23G75LP3YMZV7D6J4IV3PMVHUGEFNE3WQ, seg_pos={0 0}, upload=63391]: Uploading piece... Dec 8 15:53:20.254544139 (72h24m20.125627174s) [upload=63391]: Next returned nil segment Dec 8 15:53:20.254561903 (72h24m20.125644958s) [upload=63391]: Waiting for error group managing segments... Dec 8 15:53:20.589422337 (72h24m20.460505372s) [noise=true , address=77.90.31.18:4806 , piece_id=H27DLSIFHPGEEPD772CXBTPGHSKFFJKYC5EZBSFHITASPSYJ2GZA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil> Dec 8 15:53:20.591013766 (72h24m20.462096821s) [noise=true , address=145.79.227.21:4473 , piece_id=X5NWPG5J5VWO2H6JO2WQG474GIIEKMBATL4ZCHBJVXBIJ6EC6UJA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil> Dec 8 15:53:20.605852342 (72h24m20.476935387s) [noise=true , address=145.79.227.39:4015 , piece_id=PRU4WN2RSHGSMLJQUJFHG7J5OGN5EPJ6H5K2PWGFSDXGLRAPRZ7A, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil> Dec 8 15:53:20.606166761 (72h24m20.477249806s) [noise=true , address=77.90.31.11:4555 , piece_id=YCU5C7D5SRVMV2ZRNR63NIJ344PSTN4L2D4UQOGPXWWBPYQIUZRQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil> Dec 8 15:53:20.609717352 (72h24m20.480800407s) [noise=true , address=145.79.227.21:4474 , piece_id=ZL3BLXO3N44PN32GABZY7LCUJCI2OTOALU3UNVA2PR7VLNBRKLPQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil> Dec 8 15:53:20.633303055 (72h24m20.50438611s) [noise=true , address=77.90.31.10:4517 , piece_id=F6XFHVULC43K2X35O6TCPEHDKLHHB2O7G6IYQNLXC7LITHVC32PA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil> Dec 8 15:53:20.719688660 (72h24m20.590771695s) [noise=true , address=77.90.31.10:4518 , piece_id=6EFXAYNL6RY2RHFPH6DOKVZUA3GDRIPVHZZBLYL36XWRHIVPHX4Q, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil> Dec 8 15:53:20.750210560 (72h24m20.621293605s) [noise=true , address=145.79.227.43:4159 , piece_id=BIXIBLUU2IBMUFNHMM4X4WMUD4FUICY6ENXGUBDXYYZW5CNSW4XA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil> Dec 8 15:53:21.271868414 (72h24m21.142951459s) [noise=true , address=145.79.227.26:4654 , piece_id=HMSWSSE2KTZ7YD3PT7KGBM4EK26R34RTOIZMJFAWXLK7IWUNALJQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil> Dec 8 15:53:21.640978165 (72h24m21.5120612s) [noise=true , address=145.79.227.31:4797 , piece_id=FFONIBKZ4433V2Q3LBVCAP6PEDDMV4MHLKOYEHDTQ6UIBAGHZUMQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil> Dec 8 15:53:21.644561458 (72h24m21.515644493s) [noise=true , address=145.79.227.22:4510 , piece_id=WQL7Q5FJFKYASAFPSDM23G75LP3YMZV7D6J4IV3PMVHUGEFNE3WQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil> Dec 8 15:53:22.261710996 (72h24m22.132794031s) [noise=true , address=145.79.227.42:4121 , piece_id=4ASZMA5IVHSELKSS2M6S3H5NLFCG7FI63PDP7DP6YZAVO4VMBQMQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil> Dec 8 15:53:22.325854689 (72h24m22.196937734s) [noise=true , address=145.79.227.20:4437 , piece_id=H3A7JCGL7NYGYKEIMKBSSVX7RPONMWFIKYWJRQ6RI7BS324BFYRA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: <nil> Dec 8 15:53:22.325877934 (72h24m22.196960989s) [seg_pos={0 0}, upload=63391]: Segment reached optimal threshold of 13 pieces. Dec 8 15:53:22.326097201 (72h24m22.197180236s) [noise=true , address=145.79.227.32:4835 , piece_id=AVX3WXR3GKZGMSZTK6KFWCAZTEXSODMLATQVH5O3LDQI5UK7R5JQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y): protocol: expected piece hash; context canceled; EOF Dec 8 15:53:22.326105578 (72h24m22.197188623s) [noise=true , address=145.79.227.32:4835 , piece_id=AVX3WXR3GKZGMSZTK6KFWCAZTEXSODMLATQVH5O3LDQI5UK7R5JQ, seg_pos={0 0}, upload=63391]: Long tail cancellation due to optimal threshold Dec 8 15:53:22.326541157 (72h24m22.197624192s) [noise=true , address=145.79.227.25:4618 , piece_id=7SURNL5URXOPNLN6LMAWSD5AJUJQMM6JLOWMUMG43ONXB7B4XHZQ, seg_pos={0 0}, upload=63391]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: read tcp 188.166.225.170:57610->145.79.227.25:4618: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 8 15:53:22.326550484 (72h24m22.197633529s) [noise=true , address=145.79.227.25:4618 , piece_id=7SURNL5URXOPNLN6LMAWSD5AJUJQMM6JLOWMUMG43ONXB7B4XHZQ, seg_pos={0 0}, upload=63391]: Long tail cancellation due to optimal threshold Dec 8 15:53:22.326708185 (72h24m22.19779122s) [noise=true , address=145.79.227.44:4195 , piece_id=HLHR4HM53WUWQB3HIWMFIHNXW4YUKMEVZE4DPK3HUK6ESQUCMMVA, seg_pos={0 0}, upload=63391]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 8 15:53:22.326716230 (72h24m22.197799275s) [noise=true , address=145.79.227.44:4195 , piece_id=HLHR4HM53WUWQB3HIWMFIHNXW4YUKMEVZE4DPK3HUK6ESQUCMMVA, seg_pos={0 0}, upload=63391]: Long tail cancellation due to optimal threshold Dec 8 15:53:22.326738102 (72h24m22.197821137s) [seg_pos={0 0}, upload=63391]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 15:53:22.326820439 (72h24m22.197903484s) [upload=63391]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 15:53:22.326826189 (72h24m22.197909244s) [upload=63391]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 15:53:22.616008101 (72h24m22.487091156s) [upload=63391]: Done uploading segments... Dec 8 15:53:22.616034280 (72h24m22.487117326s) [upload=63391]: Upload finished. err: <nil> Duration: 5.836250354s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [...XX...X.X.X.X....XX......X....X..........................X] duration: 30.001587191s trace: https://jaeger.dev.storj.io/trace/12eb27369d16b107 FAIL (2025-12-08 16:13:46 UTC) orbiter warning: context deadline exceeded Duration: 30.001608782s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [................X..X...X.................X.X................] duration: 7.197066756s trace: https://jaeger.dev.storj.io/trace/690f79ef204fee2 FAIL (2025-12-08 16:13:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.197097817s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [.XX.X.X.XXXX.X...XX.X..XX.XX.X..XX.....X.....X...X..XX..X...] duration: 3.490012785s trace: https://jaeger.dev.storj.io/trace/44072b2910bf5c31 FAIL (2025-12-08 16:13:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.490049355s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 4.324219554s trace: https://jaeger.dev.storj.io/trace/4195524620a3e2c2 FAIL (2025-12-08 16:13:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.771167108 (72h9m11.642250153s) [noise=true , address=81.171.14.70:4032 , piece_id=XAMDRY73RIJV4LC2W2FRMH6L4WL3VAFUYKB3QD4TXOQWWZUYGPOQ, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.771462930 (72h9m11.642545976s) [noise=true , address=81.171.14.70:4035 , piece_id=VYAAT75X4R4J22MLY7RK4VHOEFMYZTLWUWTWVAMOOIIDSEXTQV7Q, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.771589993 (72h9m11.642673038s) [noise=true , address=185.167.204.205:4036 , piece_id=X2TWRAEKE6FEH3ITYY74A76ZZQ2KCB63ETZNW5TAHQ3MLEM34PZQ, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.772028248 (72h9m11.643111293s) [noise=true , address=185.167.204.206:4067 , piece_id=3YOG2SUYBHBE6MMIPMQWMBDZTLL7HPSBPEKFHMGOICBTAQU5HA4A, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.772041683 (72h9m11.643124728s) [noise=true , address=185.167.204.206:4069 , piece_id=GH2ZEUIEIUZ6OLM6XH5ZJVXKAMHUKQ6L3GZ7NJLUZFY623YH5GDQ, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.772603803 (72h9m11.643686848s) [noise=true , address=89.149.201.147:4022 , piece_id=CWWNRXZNWQOUTZEA4WPHGA6BYEPO35D2YDU5VSTHFZ7GRC67Y7VQ, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.773136077 (72h9m11.644219122s) [noise=true , address=185.167.204.206:4051 , piece_id=5JMUWS6LLPRENPBAZ74MDELLIZMWTMI3GOZFUWLCOCUNLDTPETWQ, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.773659244 (72h9m11.644742289s) [noise=true , address=185.167.204.206:4062 , piece_id=KYFUGH5EAVFGEJ2J7VDRDWBDP2R2MX4JSCIU7RP3VJOK4LCM6D2Q, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.774276288 (72h9m11.645359333s) [noise=true , address=81.171.14.70:4026 , piece_id=4BLGOA4GSQVOKZ6UBUIXANALBDHDHDXPLDA3TDQ44ACTIC2IHSEA, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:11.774811648 (72h9m11.645894693s) [noise=true , address=185.167.204.205:4008 , piece_id=JWZAQAYIUNBFRDVQHADARS5NTEWET7UI2GR73ZMIACA5VFUS3D6Q, seg_pos={0 0}, upload=63171]: Uploading piece... Dec 8 15:38:12.600277390 (72h9m12.471360435s) [upload=63171]: Next returned nil segment Dec 8 15:38:12.600300113 (72h9m12.471383158s) [upload=63171]: Waiting for error group managing segments... Dec 8 15:38:12.984985581 (72h9m12.856068647s) [noise=true , address=185.167.204.205:4028 , piece_id=DCGYFXSKGK6GFLLFU7UQONAX643EA6CDTYWEZ42YQPLEERINL3KQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.985372348 (72h9m12.856455383s) [noise=true , address=185.167.204.205:4008 , piece_id=JWZAQAYIUNBFRDVQHADARS5NTEWET7UI2GR73ZMIACA5VFUS3D6Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.993832749 (72h9m12.864915784s) [noise=true , address=185.167.204.205:4015 , piece_id=GK2HV5AGF2Q6RYHDZBZQ3SY537WQXB7E7UF4SWF5X3LFM4PMCZPQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.994210087 (72h9m12.865293122s) [noise=true , address=185.167.204.206:4053 , piece_id=DGSLZPKB474OADK43GBZO2FGAR2F62PBL364EWVNPCZE3PWZGQPA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.994561226 (72h9m12.865644271s) [noise=true , address=185.167.204.205:4036 , piece_id=X2TWRAEKE6FEH3ITYY74A76ZZQ2KCB63ETZNW5TAHQ3MLEM34PZQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.994944356 (72h9m12.866027401s) [noise=true , address=185.167.204.205:4023 , piece_id=5RNGVQ3LUTPKX4RAVK7Y4WRQC2Y6GT5QWV2G3MTY2LW2MU6IMKLQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.995089352 (72h9m12.866172397s) [noise=true , address=185.167.204.205:4007 , piece_id=WEZIGATHDFL7G3S2VG7BEFGGT7JJIXBDWHGIPHBTPSKDJQUTROIQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:12.995157291 (72h9m12.866240336s) [noise=true , address=185.167.204.205:4033 , piece_id=4V3DXW42ZCPJFF3PPJEBMBYFW354WEZYLVFUSPGEZ4E4HZ7MFI4Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.000801098 (72h9m12.871884163s) [noise=true , address=185.167.204.206:4062 , piece_id=KYFUGH5EAVFGEJ2J7VDRDWBDP2R2MX4JSCIU7RP3VJOK4LCM6D2Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.000862164 (72h9m12.871945209s) [noise=true , address=185.167.204.206:4051 , piece_id=5JMUWS6LLPRENPBAZ74MDELLIZMWTMI3GOZFUWLCOCUNLDTPETWQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.002117705 (72h9m12.87320074s) [noise=true , address=185.167.204.206:4064 , piece_id=LUITKU5LPQ67MFA4VWE6CDV46QOU7RXOFCGKKAYPCLFGWFSLYLTA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.003015134 (72h9m12.874098179s) [noise=true , address=185.167.204.205:4034 , piece_id=OJPEC5SBHQDKJW3LDBPHMEOPDG6NPNNMO3W54C7XIXTNUYBTMUOA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.003432589 (72h9m12.874515624s) [noise=true , address=185.167.204.206:4037 , piece_id=OPF6NTGFZ7FSN7LIEKHOGIJRYUWX27RTI3KKGCKPCOQLK4HC62LQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.004078648 (72h9m12.875161694s) [noise=true , address=185.167.204.206:4069 , piece_id=GH2ZEUIEIUZ6OLM6XH5ZJVXKAMHUKQ6L3GZ7NJLUZFY623YH5GDQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.009788571 (72h9m12.880871616s) [noise=true , address=185.167.204.206:4067 , piece_id=3YOG2SUYBHBE6MMIPMQWMBDZTLL7HPSBPEKFHMGOICBTAQU5HA4A, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.012083561 (72h9m12.883166596s) [noise=true , address=185.167.204.205:4030 , piece_id=EFAUOQCDA25VE55ZHJOSVXTOQD2VTQCW55MEH5DBEAY7N364GMXA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.584995241 (72h9m13.456078286s) [noise=true , address=81.171.14.70:4046 , piece_id=77RPCXTIQW7MO6UKAVJUD4Y7XR674IDZE2TL2SB77LJUHRE3ACDA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.651263008 (72h9m13.522346063s) [noise=true , address=81.171.14.70:4026 , piece_id=4BLGOA4GSQVOKZ6UBUIXANALBDHDHDXPLDA3TDQ44ACTIC2IHSEA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:13.696266204 (72h9m13.567349249s) [noise=true , address=81.171.14.70:4035 , piece_id=VYAAT75X4R4J22MLY7RK4VHOEFMYZTLWUWTWVAMOOIIDSEXTQV7Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.123999844 (72h9m13.995082889s) [noise=true , address=81.171.14.70:4025 , piece_id=XML4X66PHAT33VTUTEV22HSTIHMG4WFZU6YSDPCZZZ4SEBL3NWUA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.147808461 (72h9m14.018891506s) [noise=true , address=81.171.14.70:4032 , piece_id=XAMDRY73RIJV4LC2W2FRMH6L4WL3VAFUYKB3QD4TXOQWWZUYGPOQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.255119813 (72h9m14.126202858s) [noise=true , address=89.149.201.147:4005 , piece_id=MBR3CYKQ3YSPVXIVSL6ABGZNG7FB5HGP45OA3YE2IRKPJMOSM3KQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.271452473 (72h9m14.142535518s) [noise=true , address=89.149.201.147:4013 , piece_id=KIPMWZVYZADDS2LAOY5ASMIWP32HIRTNGYQ65SNLYR2EQDQCN3HQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.775652542 (72h9m14.646735587s) [noise=true , address=89.149.201.147:4016 , piece_id=5N5HKBPMDQUEQIZZVFFRQGE4LQAGSJ4H6INPDXHYQMDK5YTHUVKQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.785904314 (72h9m14.656987359s) [noise=true , address=89.149.201.147:4017 , piece_id=MBD47F6TJHZ76PAGU7ZIFRQ4M5HUDT5AQULHE6RPQPBUBAVKSTKQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.827349233 (72h9m14.698432278s) [noise=true , address=89.149.201.147:4022 , piece_id=CWWNRXZNWQOUTZEA4WPHGA6BYEPO35D2YDU5VSTHFZ7GRC67Y7VQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil> Dec 8 15:38:14.827365925 (72h9m14.69844897s) [seg_pos={0 0}, upload=63171]: Segment reached optimal threshold of 26 pieces. Dec 8 15:38:14.827539125 (72h9m14.69862217s) [noise=true , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hJ5jT6S2d3DJe9DoVDjHMkpwSTpeyTMyNLaK8Pk8SWcUVRKnv): protocol: expected piece hash; context canceled; EOF Dec 8 15:38:14.827545567 (72h9m14.698628612s) [noise=true , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Long tail cancellation due to optimal threshold Dec 8 15:38:14.827903338 (72h9m14.698986373s) [noise=true , address=89.149.201.147:4018 , piece_id=BFJH5ELBTZD767FJNFT2ZBUS3GI62IZIRSKYARLVBCLWZONSURMQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua): protocol: expected piece hash; context canceled; EOF Dec 8 15:38:14.827910673 (72h9m14.698993708s) [noise=true , address=89.149.201.147:4018 , piece_id=BFJH5ELBTZD767FJNFT2ZBUS3GI62IZIRSKYARLVBCLWZONSURMQ, seg_pos={0 0}, upload=63171]: Long tail cancellation due to optimal threshold Dec 8 15:38:14.827928065 (72h9m14.69901111s) [seg_pos={0 0}, upload=63171]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 8 15:38:14.828037083 (72h9m14.699120128s) [upload=63171]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 15:38:14.828053224 (72h9m14.699136269s) [upload=63171]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 15:39:18.272006268 (72h10m18.143089313s) [upload=63171]: Done uploading segments... Dec 8 15:39:18.272087593 (72h10m18.143170628s) [upload=63171]: Upload finished. err: metaclient: context deadline exceeded Duration: 4.324251315s core uplink eu-select-1 eu1-placement-15

Orbiter Check History

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)