Orbiter History

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................X.........X.............] duration: 5.452971799s trace: https://jaeger.dev.storj.io/trace/7e72d81acd8b3600 FAIL (2025-12-22 14:29:18 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.452997647s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [X....X.X....XX..X..............X.....X.X.......X............] duration: 3.905006021s trace: https://jaeger.dev.storj.io/trace/1c84c110f15266bb FAIL (2025-12-22 14:29:14 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.905058219s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................X...X..................X..................] duration: 24.082539192s trace: https://jaeger.dev.storj.io/trace/30697bf8e531908d FAIL (2025-12-22 14:28:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 22 14:16:07.103235978 (406h47m6.974319013s) [upload=357142]: Starting upload Dec 22 14:16:07.103295332 (406h47m6.974378377s) [upload=357142]: Done starting upload Dec 22 14:16:07.103399340 (406h47m6.974482375s) [upload=357142]: Uploading segments... Dec 22 14:16:07.103487287 (406h47m6.974570332s) [upload=357142]: Got next segment. Inline: false Dec 22 14:16:07.103515991 (406h47m6.974599026s) [upload=357142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 14:16:07.103520289 (406h47m6.974603325s) [upload=357142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 14:16:07.415289927 (406h47m7.286372962s) [seg_pos={0 0}, upload=357142]: Begin upload segment... Dec 22 14:16:07.415336014 (406h47m7.286419049s) [seg_pos={0 0}, upload=357142]: Done begin upload segment. Dec 22 14:16:07.415363607 (406h47m7.286446652s) [noise=true , address=145.79.227.17:4972 , piece_id=XOKLGQXCXZCOWYK5H4TPHRDHLLJXTII36XFZBX4XWYKHFE24WVOQ, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.415526457 (406h47m7.286609492s) [noise=true , address=145.79.227.18:4007 , piece_id=UMT4AQFVRTREIJDRAPCZTJY3URZYOHLX37QQTG5L53JFDW63ULQQ, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.416519017 (406h47m7.287602062s) [noise=true , address=145.79.227.48:4302 , piece_id=WYPWNL6UUKRVETGRAKJJMS5US3B4LQBJ6TRDYBUTRLXKGPFPPEEA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.417017777 (406h47m7.288100822s) [noise=true , address=145.79.227.50:4374 , piece_id=2XXNMOGMBDEQLIBGHJ6E25INEXYZR7JWYKA76WB5CKLK42IU5L4Q, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.417027014 (406h47m7.288110059s) [noise=true , address=145.79.227.43:4157 , piece_id=PLKGYWWJYEHSEMRCRFVQUTXTGYN6LYHFF4HPL2TOW6F7Q4K6QL7Q, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.417966864 (406h47m7.289049909s) [noise=true , address=45.91.116.8:4048 , piece_id=HQWNONHI5SJYLPULF2H75PVYLOPOPJTEC5ZPOZUT2HHLA6IC5ALA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.418744234 (406h47m7.289827269s) [noise=true , address=145.79.227.49:4339 , piece_id=ANLDLFM6BWPIJJIKHCY7CMWU2KIYTTNWA4ETR3DGNZG6GO7XM5QQ, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.419502227 (406h47m7.290585263s) [noise=true , address=77.90.31.10:4519 , piece_id=FMJHUEO445SYIHAV47SFERXWEFDSCCN36WPT4RJTXABWLJ6IO2VQ, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.419641743 (406h47m7.290724788s) [noise=true , address=145.79.227.10:4401 , piece_id=HVP7A7MFY7IM2S4P5QFS3T2BNYINUQRSF5TJ3OZLWH5REX2QCNJA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.420879160 (406h47m7.291962205s) [noise=true , address=145.79.227.29:4942 , piece_id=OMRNVS4BDDHYEFK5BEKTMKHTJUMJ4MY55FXPWOHZ3NVGHNST7KZA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.420914838 (406h47m7.291997883s) [noise=true , address=145.79.227.41:4085 , piece_id=OGRSMMM4JHNTZZZURJHFOBUAOPGDC6T2PCHDEX7YL7SADHG6AN6A, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.421340087 (406h47m7.292423132s) [noise=true , address=145.79.227.36:4517 , piece_id=DKVCILBC2JWXLG6G23ZMKEWQNDXYR4UNVD2XESJ57JIKIS5AXACA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.421402066 (406h47m7.292485101s) [noise=true , address=145.79.227.46:4603 , piece_id=5KQGOFJZAEFI5OGUVUIK2HEBRMMUVGXEOD47OQIALAIMVCPEMLFA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.422719935 (406h47m7.29380297s) [noise=true , address=145.79.227.45:4231 , piece_id=I7F2TC2A6E35C3LPHT4D62ZOAZRSMF6XZZWVKJEI6SUHWTRQ6DFA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.423220067 (406h47m7.294303112s) [noise=true , address=145.79.227.37:4566 , piece_id=4H2NM22RU4Y7H334JVNOTYO2FOUUXTKOCVYEHNJJ4HRVXVVXWOVA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.423702486 (406h47m7.294785531s) [noise=true , address=77.90.31.14:4663 , piece_id=X2NSIUKNVP2Y2MABYR3UPGZJSPF3CPS3EU6JYVNHX74RZTSGEQMA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:09.117304385 (406h47m8.988387421s) [upload=357142]: Next returned nil segment Dec 22 14:16:09.117321889 (406h47m8.988404924s) [upload=357142]: Waiting for error group managing segments... Dec 22 14:16:09.403723791 (406h47m9.274806826s) [noise=true , address=145.79.227.43:4157 , piece_id=PLKGYWWJYEHSEMRCRFVQUTXTGYN6LYHFF4HPL2TOW6F7Q4K6QL7Q, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:09.416402182 (406h47m9.287485247s) [noise=true , address=145.79.227.50:4374 , piece_id=2XXNMOGMBDEQLIBGHJ6E25INEXYZR7JWYKA76WB5CKLK42IU5L4Q, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:09.488678941 (406h47m9.359761976s) [noise=true , address=145.79.227.36:4517 , piece_id=DKVCILBC2JWXLG6G23ZMKEWQNDXYR4UNVD2XESJ57JIKIS5AXACA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:09.591529928 (406h47m9.462612963s) [noise=true , address=77.90.31.14:4663 , piece_id=X2NSIUKNVP2Y2MABYR3UPGZJSPF3CPS3EU6JYVNHX74RZTSGEQMA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:10.003825194 (406h47m9.874908229s) [noise=true , address=77.90.31.10:4519 , piece_id=FMJHUEO445SYIHAV47SFERXWEFDSCCN36WPT4RJTXABWLJ6IO2VQ, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.151289256 (406h47m11.022372301s) [noise=true , address=145.79.227.10:4401 , piece_id=HVP7A7MFY7IM2S4P5QFS3T2BNYINUQRSF5TJ3OZLWH5REX2QCNJA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.190801796 (406h47m11.061884841s) [noise=true , address=145.79.227.48:4302 , piece_id=WYPWNL6UUKRVETGRAKJJMS5US3B4LQBJ6TRDYBUTRLXKGPFPPEEA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.216210448 (406h47m11.087293483s) [noise=true , address=145.79.227.49:4339 , piece_id=ANLDLFM6BWPIJJIKHCY7CMWU2KIYTTNWA4ETR3DGNZG6GO7XM5QQ, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.447886943 (406h47m11.318969978s) [noise=true , address=145.79.227.17:4972 , piece_id=XOKLGQXCXZCOWYK5H4TPHRDHLLJXTII36XFZBX4XWYKHFE24WVOQ, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.513411914 (406h47m11.384494949s) [noise=true , address=145.79.227.41:4085 , piece_id=OGRSMMM4JHNTZZZURJHFOBUAOPGDC6T2PCHDEX7YL7SADHG6AN6A, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.768195260 (406h47m11.639278305s) [noise=true , address=145.79.227.29:4942 , piece_id=OMRNVS4BDDHYEFK5BEKTMKHTJUMJ4MY55FXPWOHZ3NVGHNST7KZA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.791646034 (406h47m11.66272908s) [noise=true , address=145.79.227.45:4231 , piece_id=I7F2TC2A6E35C3LPHT4D62ZOAZRSMF6XZZWVKJEI6SUHWTRQ6DFA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.807399862 (406h47m11.678482907s) [noise=true , address=145.79.227.46:4603 , piece_id=5KQGOFJZAEFI5OGUVUIK2HEBRMMUVGXEOD47OQIALAIMVCPEMLFA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.807424800 (406h47m11.678507845s) [seg_pos={0 0}, upload=357142]: Segment reached optimal threshold of 13 pieces. Dec 22 14:16:11.807688312 (406h47m11.678771357s) [noise=true , address=145.79.227.37:4566 , piece_id=4H2NM22RU4Y7H334JVNOTYO2FOUUXTKOCVYEHNJJ4HRVXVVXWOVA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F): protocol: expected piece hash; context canceled; EOF Dec 22 14:16:11.807698351 (406h47m11.678781396s) [noise=true , address=145.79.227.37:4566 , piece_id=4H2NM22RU4Y7H334JVNOTYO2FOUUXTKOCVYEHNJJ4HRVXVVXWOVA, seg_pos={0 0}, upload=357142]: Long tail cancellation due to optimal threshold Dec 22 14:16:11.807723459 (406h47m11.678806514s) [noise=true , address=145.79.227.18:4007 , piece_id=UMT4AQFVRTREIJDRAPCZTJY3URZYOHLX37QQTG5L53JFDW63ULQQ, seg_pos={0 0}, upload=357142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp): protocol: expected piece hash; context canceled; EOF Dec 22 14:16:11.807736343 (406h47m11.678819398s) [noise=true , address=145.79.227.18:4007 , piece_id=UMT4AQFVRTREIJDRAPCZTJY3URZYOHLX37QQTG5L53JFDW63ULQQ, seg_pos={0 0}, upload=357142]: Long tail cancellation due to optimal threshold Dec 22 14:16:11.807862974 (406h47m11.678946019s) [noise=true , address=45.91.116.8:4048 , piece_id=HQWNONHI5SJYLPULF2H75PVYLOPOPJTEC5ZPOZUT2HHLA6IC5ALA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 22 14:16:11.807871991 (406h47m11.678955026s) [noise=true , address=45.91.116.8:4048 , piece_id=HQWNONHI5SJYLPULF2H75PVYLOPOPJTEC5ZPOZUT2HHLA6IC5ALA, seg_pos={0 0}, upload=357142]: Long tail cancellation due to optimal threshold Dec 22 14:16:11.807896659 (406h47m11.678979704s) [seg_pos={0 0}, upload=357142]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 14:16:11.808114984 (406h47m11.679198019s) [upload=357142]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 14:16:11.808121817 (406h47m11.679204852s) [upload=357142]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 14:16:12.460702542 (406h47m12.331785577s) [upload=357142]: Done uploading segments... Dec 22 14:16:12.460721377 (406h47m12.331804412s) [upload=357142]: Upload finished. err: <nil> Duration: 24.082569179s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [.X....X.X....XX..X..............X.....X.X.......X...........] duration: 3.549265524s trace: https://jaeger.dev.storj.io/trace/1c84c110f15266bb FAIL (2025-12-22 14:28:32 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.549289931s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 1.510704776s trace: https://jaeger.dev.storj.io/trace/1e7f77260788e649 FAIL (2025-12-22 14:28:24 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.510737067s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................X.........X............] duration: 5.57630596s trace: https://jaeger.dev.storj.io/trace/7e72d81acd8b3600 FAIL (2025-12-22 14:28:21 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.576334494s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [X..................X...X..................X.................] duration: 5.888409041s trace: https://jaeger.dev.storj.io/trace/30697bf8e531908d FAIL (2025-12-22 14:27:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 22 14:16:07.103235978 (406h47m6.974319013s) [upload=357142]: Starting upload Dec 22 14:16:07.103295332 (406h47m6.974378377s) [upload=357142]: Done starting upload Dec 22 14:16:07.103399340 (406h47m6.974482375s) [upload=357142]: Uploading segments... Dec 22 14:16:07.103487287 (406h47m6.974570332s) [upload=357142]: Got next segment. Inline: false Dec 22 14:16:07.103515991 (406h47m6.974599026s) [upload=357142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 14:16:07.103520289 (406h47m6.974603325s) [upload=357142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 14:16:07.415289927 (406h47m7.286372962s) [seg_pos={0 0}, upload=357142]: Begin upload segment... Dec 22 14:16:07.415336014 (406h47m7.286419049s) [seg_pos={0 0}, upload=357142]: Done begin upload segment. Dec 22 14:16:07.415363607 (406h47m7.286446652s) [noise=true , address=145.79.227.17:4972 , piece_id=XOKLGQXCXZCOWYK5H4TPHRDHLLJXTII36XFZBX4XWYKHFE24WVOQ, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.415526457 (406h47m7.286609492s) [noise=true , address=145.79.227.18:4007 , piece_id=UMT4AQFVRTREIJDRAPCZTJY3URZYOHLX37QQTG5L53JFDW63ULQQ, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.416519017 (406h47m7.287602062s) [noise=true , address=145.79.227.48:4302 , piece_id=WYPWNL6UUKRVETGRAKJJMS5US3B4LQBJ6TRDYBUTRLXKGPFPPEEA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.417017777 (406h47m7.288100822s) [noise=true , address=145.79.227.50:4374 , piece_id=2XXNMOGMBDEQLIBGHJ6E25INEXYZR7JWYKA76WB5CKLK42IU5L4Q, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.417027014 (406h47m7.288110059s) [noise=true , address=145.79.227.43:4157 , piece_id=PLKGYWWJYEHSEMRCRFVQUTXTGYN6LYHFF4HPL2TOW6F7Q4K6QL7Q, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.417966864 (406h47m7.289049909s) [noise=true , address=45.91.116.8:4048 , piece_id=HQWNONHI5SJYLPULF2H75PVYLOPOPJTEC5ZPOZUT2HHLA6IC5ALA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.418744234 (406h47m7.289827269s) [noise=true , address=145.79.227.49:4339 , piece_id=ANLDLFM6BWPIJJIKHCY7CMWU2KIYTTNWA4ETR3DGNZG6GO7XM5QQ, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.419502227 (406h47m7.290585263s) [noise=true , address=77.90.31.10:4519 , piece_id=FMJHUEO445SYIHAV47SFERXWEFDSCCN36WPT4RJTXABWLJ6IO2VQ, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.419641743 (406h47m7.290724788s) [noise=true , address=145.79.227.10:4401 , piece_id=HVP7A7MFY7IM2S4P5QFS3T2BNYINUQRSF5TJ3OZLWH5REX2QCNJA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.420879160 (406h47m7.291962205s) [noise=true , address=145.79.227.29:4942 , piece_id=OMRNVS4BDDHYEFK5BEKTMKHTJUMJ4MY55FXPWOHZ3NVGHNST7KZA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.420914838 (406h47m7.291997883s) [noise=true , address=145.79.227.41:4085 , piece_id=OGRSMMM4JHNTZZZURJHFOBUAOPGDC6T2PCHDEX7YL7SADHG6AN6A, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.421340087 (406h47m7.292423132s) [noise=true , address=145.79.227.36:4517 , piece_id=DKVCILBC2JWXLG6G23ZMKEWQNDXYR4UNVD2XESJ57JIKIS5AXACA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.421402066 (406h47m7.292485101s) [noise=true , address=145.79.227.46:4603 , piece_id=5KQGOFJZAEFI5OGUVUIK2HEBRMMUVGXEOD47OQIALAIMVCPEMLFA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.422719935 (406h47m7.29380297s) [noise=true , address=145.79.227.45:4231 , piece_id=I7F2TC2A6E35C3LPHT4D62ZOAZRSMF6XZZWVKJEI6SUHWTRQ6DFA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.423220067 (406h47m7.294303112s) [noise=true , address=145.79.227.37:4566 , piece_id=4H2NM22RU4Y7H334JVNOTYO2FOUUXTKOCVYEHNJJ4HRVXVVXWOVA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.423702486 (406h47m7.294785531s) [noise=true , address=77.90.31.14:4663 , piece_id=X2NSIUKNVP2Y2MABYR3UPGZJSPF3CPS3EU6JYVNHX74RZTSGEQMA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:09.117304385 (406h47m8.988387421s) [upload=357142]: Next returned nil segment Dec 22 14:16:09.117321889 (406h47m8.988404924s) [upload=357142]: Waiting for error group managing segments... Dec 22 14:16:09.403723791 (406h47m9.274806826s) [noise=true , address=145.79.227.43:4157 , piece_id=PLKGYWWJYEHSEMRCRFVQUTXTGYN6LYHFF4HPL2TOW6F7Q4K6QL7Q, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:09.416402182 (406h47m9.287485247s) [noise=true , address=145.79.227.50:4374 , piece_id=2XXNMOGMBDEQLIBGHJ6E25INEXYZR7JWYKA76WB5CKLK42IU5L4Q, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:09.488678941 (406h47m9.359761976s) [noise=true , address=145.79.227.36:4517 , piece_id=DKVCILBC2JWXLG6G23ZMKEWQNDXYR4UNVD2XESJ57JIKIS5AXACA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:09.591529928 (406h47m9.462612963s) [noise=true , address=77.90.31.14:4663 , piece_id=X2NSIUKNVP2Y2MABYR3UPGZJSPF3CPS3EU6JYVNHX74RZTSGEQMA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:10.003825194 (406h47m9.874908229s) [noise=true , address=77.90.31.10:4519 , piece_id=FMJHUEO445SYIHAV47SFERXWEFDSCCN36WPT4RJTXABWLJ6IO2VQ, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.151289256 (406h47m11.022372301s) [noise=true , address=145.79.227.10:4401 , piece_id=HVP7A7MFY7IM2S4P5QFS3T2BNYINUQRSF5TJ3OZLWH5REX2QCNJA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.190801796 (406h47m11.061884841s) [noise=true , address=145.79.227.48:4302 , piece_id=WYPWNL6UUKRVETGRAKJJMS5US3B4LQBJ6TRDYBUTRLXKGPFPPEEA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.216210448 (406h47m11.087293483s) [noise=true , address=145.79.227.49:4339 , piece_id=ANLDLFM6BWPIJJIKHCY7CMWU2KIYTTNWA4ETR3DGNZG6GO7XM5QQ, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.447886943 (406h47m11.318969978s) [noise=true , address=145.79.227.17:4972 , piece_id=XOKLGQXCXZCOWYK5H4TPHRDHLLJXTII36XFZBX4XWYKHFE24WVOQ, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.513411914 (406h47m11.384494949s) [noise=true , address=145.79.227.41:4085 , piece_id=OGRSMMM4JHNTZZZURJHFOBUAOPGDC6T2PCHDEX7YL7SADHG6AN6A, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.768195260 (406h47m11.639278305s) [noise=true , address=145.79.227.29:4942 , piece_id=OMRNVS4BDDHYEFK5BEKTMKHTJUMJ4MY55FXPWOHZ3NVGHNST7KZA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.791646034 (406h47m11.66272908s) [noise=true , address=145.79.227.45:4231 , piece_id=I7F2TC2A6E35C3LPHT4D62ZOAZRSMF6XZZWVKJEI6SUHWTRQ6DFA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.807399862 (406h47m11.678482907s) [noise=true , address=145.79.227.46:4603 , piece_id=5KQGOFJZAEFI5OGUVUIK2HEBRMMUVGXEOD47OQIALAIMVCPEMLFA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.807424800 (406h47m11.678507845s) [seg_pos={0 0}, upload=357142]: Segment reached optimal threshold of 13 pieces. Dec 22 14:16:11.807688312 (406h47m11.678771357s) [noise=true , address=145.79.227.37:4566 , piece_id=4H2NM22RU4Y7H334JVNOTYO2FOUUXTKOCVYEHNJJ4HRVXVVXWOVA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F): protocol: expected piece hash; context canceled; EOF Dec 22 14:16:11.807698351 (406h47m11.678781396s) [noise=true , address=145.79.227.37:4566 , piece_id=4H2NM22RU4Y7H334JVNOTYO2FOUUXTKOCVYEHNJJ4HRVXVVXWOVA, seg_pos={0 0}, upload=357142]: Long tail cancellation due to optimal threshold Dec 22 14:16:11.807723459 (406h47m11.678806514s) [noise=true , address=145.79.227.18:4007 , piece_id=UMT4AQFVRTREIJDRAPCZTJY3URZYOHLX37QQTG5L53JFDW63ULQQ, seg_pos={0 0}, upload=357142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp): protocol: expected piece hash; context canceled; EOF Dec 22 14:16:11.807736343 (406h47m11.678819398s) [noise=true , address=145.79.227.18:4007 , piece_id=UMT4AQFVRTREIJDRAPCZTJY3URZYOHLX37QQTG5L53JFDW63ULQQ, seg_pos={0 0}, upload=357142]: Long tail cancellation due to optimal threshold Dec 22 14:16:11.807862974 (406h47m11.678946019s) [noise=true , address=45.91.116.8:4048 , piece_id=HQWNONHI5SJYLPULF2H75PVYLOPOPJTEC5ZPOZUT2HHLA6IC5ALA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 22 14:16:11.807871991 (406h47m11.678955026s) [noise=true , address=45.91.116.8:4048 , piece_id=HQWNONHI5SJYLPULF2H75PVYLOPOPJTEC5ZPOZUT2HHLA6IC5ALA, seg_pos={0 0}, upload=357142]: Long tail cancellation due to optimal threshold Dec 22 14:16:11.807896659 (406h47m11.678979704s) [seg_pos={0 0}, upload=357142]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 14:16:11.808114984 (406h47m11.679198019s) [upload=357142]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 14:16:11.808121817 (406h47m11.679204852s) [upload=357142]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 14:16:12.460702542 (406h47m12.331785577s) [upload=357142]: Done uploading segments... Dec 22 14:16:12.460721377 (406h47m12.331804412s) [upload=357142]: Upload finished. err: <nil> Duration: 5.888436894s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [..X....X.X....XX..X..............X.....X.X.......X..........] duration: 3.775640208s trace: https://jaeger.dev.storj.io/trace/1c84c110f15266bb FAIL (2025-12-22 14:27:24 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.775665896s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 1.485855279s trace: https://jaeger.dev.storj.io/trace/1e7f77260788e649 FAIL (2025-12-22 14:27:18 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.48588113s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................X.........X...........] duration: 5.223500176s trace: https://jaeger.dev.storj.io/trace/7e72d81acd8b3600 FAIL (2025-12-22 14:27:13 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.223529702s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.X..................X...X..................X................] duration: 5.868375965s trace: https://jaeger.dev.storj.io/trace/30697bf8e531908d FAIL (2025-12-22 14:26:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 22 14:16:07.103235978 (406h47m6.974319013s) [upload=357142]: Starting upload Dec 22 14:16:07.103295332 (406h47m6.974378377s) [upload=357142]: Done starting upload Dec 22 14:16:07.103399340 (406h47m6.974482375s) [upload=357142]: Uploading segments... Dec 22 14:16:07.103487287 (406h47m6.974570332s) [upload=357142]: Got next segment. Inline: false Dec 22 14:16:07.103515991 (406h47m6.974599026s) [upload=357142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 14:16:07.103520289 (406h47m6.974603325s) [upload=357142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 14:16:07.415289927 (406h47m7.286372962s) [seg_pos={0 0}, upload=357142]: Begin upload segment... Dec 22 14:16:07.415336014 (406h47m7.286419049s) [seg_pos={0 0}, upload=357142]: Done begin upload segment. Dec 22 14:16:07.415363607 (406h47m7.286446652s) [noise=true , address=145.79.227.17:4972 , piece_id=XOKLGQXCXZCOWYK5H4TPHRDHLLJXTII36XFZBX4XWYKHFE24WVOQ, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.415526457 (406h47m7.286609492s) [noise=true , address=145.79.227.18:4007 , piece_id=UMT4AQFVRTREIJDRAPCZTJY3URZYOHLX37QQTG5L53JFDW63ULQQ, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.416519017 (406h47m7.287602062s) [noise=true , address=145.79.227.48:4302 , piece_id=WYPWNL6UUKRVETGRAKJJMS5US3B4LQBJ6TRDYBUTRLXKGPFPPEEA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.417017777 (406h47m7.288100822s) [noise=true , address=145.79.227.50:4374 , piece_id=2XXNMOGMBDEQLIBGHJ6E25INEXYZR7JWYKA76WB5CKLK42IU5L4Q, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.417027014 (406h47m7.288110059s) [noise=true , address=145.79.227.43:4157 , piece_id=PLKGYWWJYEHSEMRCRFVQUTXTGYN6LYHFF4HPL2TOW6F7Q4K6QL7Q, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.417966864 (406h47m7.289049909s) [noise=true , address=45.91.116.8:4048 , piece_id=HQWNONHI5SJYLPULF2H75PVYLOPOPJTEC5ZPOZUT2HHLA6IC5ALA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.418744234 (406h47m7.289827269s) [noise=true , address=145.79.227.49:4339 , piece_id=ANLDLFM6BWPIJJIKHCY7CMWU2KIYTTNWA4ETR3DGNZG6GO7XM5QQ, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.419502227 (406h47m7.290585263s) [noise=true , address=77.90.31.10:4519 , piece_id=FMJHUEO445SYIHAV47SFERXWEFDSCCN36WPT4RJTXABWLJ6IO2VQ, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.419641743 (406h47m7.290724788s) [noise=true , address=145.79.227.10:4401 , piece_id=HVP7A7MFY7IM2S4P5QFS3T2BNYINUQRSF5TJ3OZLWH5REX2QCNJA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.420879160 (406h47m7.291962205s) [noise=true , address=145.79.227.29:4942 , piece_id=OMRNVS4BDDHYEFK5BEKTMKHTJUMJ4MY55FXPWOHZ3NVGHNST7KZA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.420914838 (406h47m7.291997883s) [noise=true , address=145.79.227.41:4085 , piece_id=OGRSMMM4JHNTZZZURJHFOBUAOPGDC6T2PCHDEX7YL7SADHG6AN6A, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.421340087 (406h47m7.292423132s) [noise=true , address=145.79.227.36:4517 , piece_id=DKVCILBC2JWXLG6G23ZMKEWQNDXYR4UNVD2XESJ57JIKIS5AXACA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.421402066 (406h47m7.292485101s) [noise=true , address=145.79.227.46:4603 , piece_id=5KQGOFJZAEFI5OGUVUIK2HEBRMMUVGXEOD47OQIALAIMVCPEMLFA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.422719935 (406h47m7.29380297s) [noise=true , address=145.79.227.45:4231 , piece_id=I7F2TC2A6E35C3LPHT4D62ZOAZRSMF6XZZWVKJEI6SUHWTRQ6DFA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.423220067 (406h47m7.294303112s) [noise=true , address=145.79.227.37:4566 , piece_id=4H2NM22RU4Y7H334JVNOTYO2FOUUXTKOCVYEHNJJ4HRVXVVXWOVA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.423702486 (406h47m7.294785531s) [noise=true , address=77.90.31.14:4663 , piece_id=X2NSIUKNVP2Y2MABYR3UPGZJSPF3CPS3EU6JYVNHX74RZTSGEQMA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:09.117304385 (406h47m8.988387421s) [upload=357142]: Next returned nil segment Dec 22 14:16:09.117321889 (406h47m8.988404924s) [upload=357142]: Waiting for error group managing segments... Dec 22 14:16:09.403723791 (406h47m9.274806826s) [noise=true , address=145.79.227.43:4157 , piece_id=PLKGYWWJYEHSEMRCRFVQUTXTGYN6LYHFF4HPL2TOW6F7Q4K6QL7Q, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:09.416402182 (406h47m9.287485247s) [noise=true , address=145.79.227.50:4374 , piece_id=2XXNMOGMBDEQLIBGHJ6E25INEXYZR7JWYKA76WB5CKLK42IU5L4Q, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:09.488678941 (406h47m9.359761976s) [noise=true , address=145.79.227.36:4517 , piece_id=DKVCILBC2JWXLG6G23ZMKEWQNDXYR4UNVD2XESJ57JIKIS5AXACA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:09.591529928 (406h47m9.462612963s) [noise=true , address=77.90.31.14:4663 , piece_id=X2NSIUKNVP2Y2MABYR3UPGZJSPF3CPS3EU6JYVNHX74RZTSGEQMA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:10.003825194 (406h47m9.874908229s) [noise=true , address=77.90.31.10:4519 , piece_id=FMJHUEO445SYIHAV47SFERXWEFDSCCN36WPT4RJTXABWLJ6IO2VQ, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.151289256 (406h47m11.022372301s) [noise=true , address=145.79.227.10:4401 , piece_id=HVP7A7MFY7IM2S4P5QFS3T2BNYINUQRSF5TJ3OZLWH5REX2QCNJA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.190801796 (406h47m11.061884841s) [noise=true , address=145.79.227.48:4302 , piece_id=WYPWNL6UUKRVETGRAKJJMS5US3B4LQBJ6TRDYBUTRLXKGPFPPEEA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.216210448 (406h47m11.087293483s) [noise=true , address=145.79.227.49:4339 , piece_id=ANLDLFM6BWPIJJIKHCY7CMWU2KIYTTNWA4ETR3DGNZG6GO7XM5QQ, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.447886943 (406h47m11.318969978s) [noise=true , address=145.79.227.17:4972 , piece_id=XOKLGQXCXZCOWYK5H4TPHRDHLLJXTII36XFZBX4XWYKHFE24WVOQ, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.513411914 (406h47m11.384494949s) [noise=true , address=145.79.227.41:4085 , piece_id=OGRSMMM4JHNTZZZURJHFOBUAOPGDC6T2PCHDEX7YL7SADHG6AN6A, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.768195260 (406h47m11.639278305s) [noise=true , address=145.79.227.29:4942 , piece_id=OMRNVS4BDDHYEFK5BEKTMKHTJUMJ4MY55FXPWOHZ3NVGHNST7KZA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.791646034 (406h47m11.66272908s) [noise=true , address=145.79.227.45:4231 , piece_id=I7F2TC2A6E35C3LPHT4D62ZOAZRSMF6XZZWVKJEI6SUHWTRQ6DFA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.807399862 (406h47m11.678482907s) [noise=true , address=145.79.227.46:4603 , piece_id=5KQGOFJZAEFI5OGUVUIK2HEBRMMUVGXEOD47OQIALAIMVCPEMLFA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.807424800 (406h47m11.678507845s) [seg_pos={0 0}, upload=357142]: Segment reached optimal threshold of 13 pieces. Dec 22 14:16:11.807688312 (406h47m11.678771357s) [noise=true , address=145.79.227.37:4566 , piece_id=4H2NM22RU4Y7H334JVNOTYO2FOUUXTKOCVYEHNJJ4HRVXVVXWOVA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F): protocol: expected piece hash; context canceled; EOF Dec 22 14:16:11.807698351 (406h47m11.678781396s) [noise=true , address=145.79.227.37:4566 , piece_id=4H2NM22RU4Y7H334JVNOTYO2FOUUXTKOCVYEHNJJ4HRVXVVXWOVA, seg_pos={0 0}, upload=357142]: Long tail cancellation due to optimal threshold Dec 22 14:16:11.807723459 (406h47m11.678806514s) [noise=true , address=145.79.227.18:4007 , piece_id=UMT4AQFVRTREIJDRAPCZTJY3URZYOHLX37QQTG5L53JFDW63ULQQ, seg_pos={0 0}, upload=357142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp): protocol: expected piece hash; context canceled; EOF Dec 22 14:16:11.807736343 (406h47m11.678819398s) [noise=true , address=145.79.227.18:4007 , piece_id=UMT4AQFVRTREIJDRAPCZTJY3URZYOHLX37QQTG5L53JFDW63ULQQ, seg_pos={0 0}, upload=357142]: Long tail cancellation due to optimal threshold Dec 22 14:16:11.807862974 (406h47m11.678946019s) [noise=true , address=45.91.116.8:4048 , piece_id=HQWNONHI5SJYLPULF2H75PVYLOPOPJTEC5ZPOZUT2HHLA6IC5ALA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 22 14:16:11.807871991 (406h47m11.678955026s) [noise=true , address=45.91.116.8:4048 , piece_id=HQWNONHI5SJYLPULF2H75PVYLOPOPJTEC5ZPOZUT2HHLA6IC5ALA, seg_pos={0 0}, upload=357142]: Long tail cancellation due to optimal threshold Dec 22 14:16:11.807896659 (406h47m11.678979704s) [seg_pos={0 0}, upload=357142]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 14:16:11.808114984 (406h47m11.679198019s) [upload=357142]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 14:16:11.808121817 (406h47m11.679204852s) [upload=357142]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 14:16:12.460702542 (406h47m12.331785577s) [upload=357142]: Done uploading segments... Dec 22 14:16:12.460721377 (406h47m12.331804412s) [upload=357142]: Upload finished. err: <nil> Duration: 5.868408026s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 1.367382395s trace: https://jaeger.dev.storj.io/trace/1e7f77260788e649 FAIL (2025-12-22 14:26:41 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.367404857s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................X.........X..........] duration: 5.54995373s trace: https://jaeger.dev.storj.io/trace/7e72d81acd8b3600 FAIL (2025-12-22 14:26:39 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.549979731s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [...X....X.X....XX..X..............X.....X.X.......X.........] duration: 3.62594556s trace: https://jaeger.dev.storj.io/trace/1c84c110f15266bb FAIL (2025-12-22 14:26:36 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.625969766s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................................X.........X.........] duration: 5.378285002s trace: https://jaeger.dev.storj.io/trace/7e72d81acd8b3600 FAIL (2025-12-22 14:26:19 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.378310921s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 1.453140072s trace: https://jaeger.dev.storj.io/trace/1e7f77260788e649 FAIL (2025-12-22 14:26:13 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.453161524s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..X..................X...X..................X...............] duration: 8.784373704s trace: https://jaeger.dev.storj.io/trace/30697bf8e531908d FAIL (2025-12-22 14:26:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 22 14:16:07.103235978 (406h47m6.974319013s) [upload=357142]: Starting upload Dec 22 14:16:07.103295332 (406h47m6.974378377s) [upload=357142]: Done starting upload Dec 22 14:16:07.103399340 (406h47m6.974482375s) [upload=357142]: Uploading segments... Dec 22 14:16:07.103487287 (406h47m6.974570332s) [upload=357142]: Got next segment. Inline: false Dec 22 14:16:07.103515991 (406h47m6.974599026s) [upload=357142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 14:16:07.103520289 (406h47m6.974603325s) [upload=357142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 14:16:07.415289927 (406h47m7.286372962s) [seg_pos={0 0}, upload=357142]: Begin upload segment... Dec 22 14:16:07.415336014 (406h47m7.286419049s) [seg_pos={0 0}, upload=357142]: Done begin upload segment. Dec 22 14:16:07.415363607 (406h47m7.286446652s) [noise=true , address=145.79.227.17:4972 , piece_id=XOKLGQXCXZCOWYK5H4TPHRDHLLJXTII36XFZBX4XWYKHFE24WVOQ, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.415526457 (406h47m7.286609492s) [noise=true , address=145.79.227.18:4007 , piece_id=UMT4AQFVRTREIJDRAPCZTJY3URZYOHLX37QQTG5L53JFDW63ULQQ, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.416519017 (406h47m7.287602062s) [noise=true , address=145.79.227.48:4302 , piece_id=WYPWNL6UUKRVETGRAKJJMS5US3B4LQBJ6TRDYBUTRLXKGPFPPEEA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.417017777 (406h47m7.288100822s) [noise=true , address=145.79.227.50:4374 , piece_id=2XXNMOGMBDEQLIBGHJ6E25INEXYZR7JWYKA76WB5CKLK42IU5L4Q, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.417027014 (406h47m7.288110059s) [noise=true , address=145.79.227.43:4157 , piece_id=PLKGYWWJYEHSEMRCRFVQUTXTGYN6LYHFF4HPL2TOW6F7Q4K6QL7Q, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.417966864 (406h47m7.289049909s) [noise=true , address=45.91.116.8:4048 , piece_id=HQWNONHI5SJYLPULF2H75PVYLOPOPJTEC5ZPOZUT2HHLA6IC5ALA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.418744234 (406h47m7.289827269s) [noise=true , address=145.79.227.49:4339 , piece_id=ANLDLFM6BWPIJJIKHCY7CMWU2KIYTTNWA4ETR3DGNZG6GO7XM5QQ, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.419502227 (406h47m7.290585263s) [noise=true , address=77.90.31.10:4519 , piece_id=FMJHUEO445SYIHAV47SFERXWEFDSCCN36WPT4RJTXABWLJ6IO2VQ, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.419641743 (406h47m7.290724788s) [noise=true , address=145.79.227.10:4401 , piece_id=HVP7A7MFY7IM2S4P5QFS3T2BNYINUQRSF5TJ3OZLWH5REX2QCNJA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.420879160 (406h47m7.291962205s) [noise=true , address=145.79.227.29:4942 , piece_id=OMRNVS4BDDHYEFK5BEKTMKHTJUMJ4MY55FXPWOHZ3NVGHNST7KZA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.420914838 (406h47m7.291997883s) [noise=true , address=145.79.227.41:4085 , piece_id=OGRSMMM4JHNTZZZURJHFOBUAOPGDC6T2PCHDEX7YL7SADHG6AN6A, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.421340087 (406h47m7.292423132s) [noise=true , address=145.79.227.36:4517 , piece_id=DKVCILBC2JWXLG6G23ZMKEWQNDXYR4UNVD2XESJ57JIKIS5AXACA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.421402066 (406h47m7.292485101s) [noise=true , address=145.79.227.46:4603 , piece_id=5KQGOFJZAEFI5OGUVUIK2HEBRMMUVGXEOD47OQIALAIMVCPEMLFA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.422719935 (406h47m7.29380297s) [noise=true , address=145.79.227.45:4231 , piece_id=I7F2TC2A6E35C3LPHT4D62ZOAZRSMF6XZZWVKJEI6SUHWTRQ6DFA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.423220067 (406h47m7.294303112s) [noise=true , address=145.79.227.37:4566 , piece_id=4H2NM22RU4Y7H334JVNOTYO2FOUUXTKOCVYEHNJJ4HRVXVVXWOVA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.423702486 (406h47m7.294785531s) [noise=true , address=77.90.31.14:4663 , piece_id=X2NSIUKNVP2Y2MABYR3UPGZJSPF3CPS3EU6JYVNHX74RZTSGEQMA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:09.117304385 (406h47m8.988387421s) [upload=357142]: Next returned nil segment Dec 22 14:16:09.117321889 (406h47m8.988404924s) [upload=357142]: Waiting for error group managing segments... Dec 22 14:16:09.403723791 (406h47m9.274806826s) [noise=true , address=145.79.227.43:4157 , piece_id=PLKGYWWJYEHSEMRCRFVQUTXTGYN6LYHFF4HPL2TOW6F7Q4K6QL7Q, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:09.416402182 (406h47m9.287485247s) [noise=true , address=145.79.227.50:4374 , piece_id=2XXNMOGMBDEQLIBGHJ6E25INEXYZR7JWYKA76WB5CKLK42IU5L4Q, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:09.488678941 (406h47m9.359761976s) [noise=true , address=145.79.227.36:4517 , piece_id=DKVCILBC2JWXLG6G23ZMKEWQNDXYR4UNVD2XESJ57JIKIS5AXACA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:09.591529928 (406h47m9.462612963s) [noise=true , address=77.90.31.14:4663 , piece_id=X2NSIUKNVP2Y2MABYR3UPGZJSPF3CPS3EU6JYVNHX74RZTSGEQMA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:10.003825194 (406h47m9.874908229s) [noise=true , address=77.90.31.10:4519 , piece_id=FMJHUEO445SYIHAV47SFERXWEFDSCCN36WPT4RJTXABWLJ6IO2VQ, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.151289256 (406h47m11.022372301s) [noise=true , address=145.79.227.10:4401 , piece_id=HVP7A7MFY7IM2S4P5QFS3T2BNYINUQRSF5TJ3OZLWH5REX2QCNJA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.190801796 (406h47m11.061884841s) [noise=true , address=145.79.227.48:4302 , piece_id=WYPWNL6UUKRVETGRAKJJMS5US3B4LQBJ6TRDYBUTRLXKGPFPPEEA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.216210448 (406h47m11.087293483s) [noise=true , address=145.79.227.49:4339 , piece_id=ANLDLFM6BWPIJJIKHCY7CMWU2KIYTTNWA4ETR3DGNZG6GO7XM5QQ, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.447886943 (406h47m11.318969978s) [noise=true , address=145.79.227.17:4972 , piece_id=XOKLGQXCXZCOWYK5H4TPHRDHLLJXTII36XFZBX4XWYKHFE24WVOQ, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.513411914 (406h47m11.384494949s) [noise=true , address=145.79.227.41:4085 , piece_id=OGRSMMM4JHNTZZZURJHFOBUAOPGDC6T2PCHDEX7YL7SADHG6AN6A, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.768195260 (406h47m11.639278305s) [noise=true , address=145.79.227.29:4942 , piece_id=OMRNVS4BDDHYEFK5BEKTMKHTJUMJ4MY55FXPWOHZ3NVGHNST7KZA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.791646034 (406h47m11.66272908s) [noise=true , address=145.79.227.45:4231 , piece_id=I7F2TC2A6E35C3LPHT4D62ZOAZRSMF6XZZWVKJEI6SUHWTRQ6DFA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.807399862 (406h47m11.678482907s) [noise=true , address=145.79.227.46:4603 , piece_id=5KQGOFJZAEFI5OGUVUIK2HEBRMMUVGXEOD47OQIALAIMVCPEMLFA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.807424800 (406h47m11.678507845s) [seg_pos={0 0}, upload=357142]: Segment reached optimal threshold of 13 pieces. Dec 22 14:16:11.807688312 (406h47m11.678771357s) [noise=true , address=145.79.227.37:4566 , piece_id=4H2NM22RU4Y7H334JVNOTYO2FOUUXTKOCVYEHNJJ4HRVXVVXWOVA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F): protocol: expected piece hash; context canceled; EOF Dec 22 14:16:11.807698351 (406h47m11.678781396s) [noise=true , address=145.79.227.37:4566 , piece_id=4H2NM22RU4Y7H334JVNOTYO2FOUUXTKOCVYEHNJJ4HRVXVVXWOVA, seg_pos={0 0}, upload=357142]: Long tail cancellation due to optimal threshold Dec 22 14:16:11.807723459 (406h47m11.678806514s) [noise=true , address=145.79.227.18:4007 , piece_id=UMT4AQFVRTREIJDRAPCZTJY3URZYOHLX37QQTG5L53JFDW63ULQQ, seg_pos={0 0}, upload=357142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp): protocol: expected piece hash; context canceled; EOF Dec 22 14:16:11.807736343 (406h47m11.678819398s) [noise=true , address=145.79.227.18:4007 , piece_id=UMT4AQFVRTREIJDRAPCZTJY3URZYOHLX37QQTG5L53JFDW63ULQQ, seg_pos={0 0}, upload=357142]: Long tail cancellation due to optimal threshold Dec 22 14:16:11.807862974 (406h47m11.678946019s) [noise=true , address=45.91.116.8:4048 , piece_id=HQWNONHI5SJYLPULF2H75PVYLOPOPJTEC5ZPOZUT2HHLA6IC5ALA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 22 14:16:11.807871991 (406h47m11.678955026s) [noise=true , address=45.91.116.8:4048 , piece_id=HQWNONHI5SJYLPULF2H75PVYLOPOPJTEC5ZPOZUT2HHLA6IC5ALA, seg_pos={0 0}, upload=357142]: Long tail cancellation due to optimal threshold Dec 22 14:16:11.807896659 (406h47m11.678979704s) [seg_pos={0 0}, upload=357142]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 14:16:11.808114984 (406h47m11.679198019s) [upload=357142]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 14:16:11.808121817 (406h47m11.679204852s) [upload=357142]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 14:16:12.460702542 (406h47m12.331785577s) [upload=357142]: Done uploading segments... Dec 22 14:16:12.460721377 (406h47m12.331804412s) [upload=357142]: Upload finished. err: <nil> Duration: 8.784408491s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................................X.........X........] duration: 5.406427459s trace: https://jaeger.dev.storj.io/trace/7e72d81acd8b3600 FAIL (2025-12-22 14:25:40 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.406459851s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [....X....X.X....XX..X..............X.....X.X.......X........] duration: 3.423426976s trace: https://jaeger.dev.storj.io/trace/1c84c110f15266bb FAIL (2025-12-22 14:25:25 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.423456131s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 1.468812014s trace: https://jaeger.dev.storj.io/trace/1e7f77260788e649 FAIL (2025-12-22 14:25:03 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.46884633s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...X..................X...X..................X..............] duration: 5.809984336s trace: https://jaeger.dev.storj.io/trace/30697bf8e531908d FAIL (2025-12-22 14:24:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 22 14:16:07.103235978 (406h47m6.974319013s) [upload=357142]: Starting upload Dec 22 14:16:07.103295332 (406h47m6.974378377s) [upload=357142]: Done starting upload Dec 22 14:16:07.103399340 (406h47m6.974482375s) [upload=357142]: Uploading segments... Dec 22 14:16:07.103487287 (406h47m6.974570332s) [upload=357142]: Got next segment. Inline: false Dec 22 14:16:07.103515991 (406h47m6.974599026s) [upload=357142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 14:16:07.103520289 (406h47m6.974603325s) [upload=357142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 14:16:07.415289927 (406h47m7.286372962s) [seg_pos={0 0}, upload=357142]: Begin upload segment... Dec 22 14:16:07.415336014 (406h47m7.286419049s) [seg_pos={0 0}, upload=357142]: Done begin upload segment. Dec 22 14:16:07.415363607 (406h47m7.286446652s) [noise=true , address=145.79.227.17:4972 , piece_id=XOKLGQXCXZCOWYK5H4TPHRDHLLJXTII36XFZBX4XWYKHFE24WVOQ, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.415526457 (406h47m7.286609492s) [noise=true , address=145.79.227.18:4007 , piece_id=UMT4AQFVRTREIJDRAPCZTJY3URZYOHLX37QQTG5L53JFDW63ULQQ, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.416519017 (406h47m7.287602062s) [noise=true , address=145.79.227.48:4302 , piece_id=WYPWNL6UUKRVETGRAKJJMS5US3B4LQBJ6TRDYBUTRLXKGPFPPEEA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.417017777 (406h47m7.288100822s) [noise=true , address=145.79.227.50:4374 , piece_id=2XXNMOGMBDEQLIBGHJ6E25INEXYZR7JWYKA76WB5CKLK42IU5L4Q, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.417027014 (406h47m7.288110059s) [noise=true , address=145.79.227.43:4157 , piece_id=PLKGYWWJYEHSEMRCRFVQUTXTGYN6LYHFF4HPL2TOW6F7Q4K6QL7Q, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.417966864 (406h47m7.289049909s) [noise=true , address=45.91.116.8:4048 , piece_id=HQWNONHI5SJYLPULF2H75PVYLOPOPJTEC5ZPOZUT2HHLA6IC5ALA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.418744234 (406h47m7.289827269s) [noise=true , address=145.79.227.49:4339 , piece_id=ANLDLFM6BWPIJJIKHCY7CMWU2KIYTTNWA4ETR3DGNZG6GO7XM5QQ, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.419502227 (406h47m7.290585263s) [noise=true , address=77.90.31.10:4519 , piece_id=FMJHUEO445SYIHAV47SFERXWEFDSCCN36WPT4RJTXABWLJ6IO2VQ, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.419641743 (406h47m7.290724788s) [noise=true , address=145.79.227.10:4401 , piece_id=HVP7A7MFY7IM2S4P5QFS3T2BNYINUQRSF5TJ3OZLWH5REX2QCNJA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.420879160 (406h47m7.291962205s) [noise=true , address=145.79.227.29:4942 , piece_id=OMRNVS4BDDHYEFK5BEKTMKHTJUMJ4MY55FXPWOHZ3NVGHNST7KZA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.420914838 (406h47m7.291997883s) [noise=true , address=145.79.227.41:4085 , piece_id=OGRSMMM4JHNTZZZURJHFOBUAOPGDC6T2PCHDEX7YL7SADHG6AN6A, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.421340087 (406h47m7.292423132s) [noise=true , address=145.79.227.36:4517 , piece_id=DKVCILBC2JWXLG6G23ZMKEWQNDXYR4UNVD2XESJ57JIKIS5AXACA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.421402066 (406h47m7.292485101s) [noise=true , address=145.79.227.46:4603 , piece_id=5KQGOFJZAEFI5OGUVUIK2HEBRMMUVGXEOD47OQIALAIMVCPEMLFA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.422719935 (406h47m7.29380297s) [noise=true , address=145.79.227.45:4231 , piece_id=I7F2TC2A6E35C3LPHT4D62ZOAZRSMF6XZZWVKJEI6SUHWTRQ6DFA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.423220067 (406h47m7.294303112s) [noise=true , address=145.79.227.37:4566 , piece_id=4H2NM22RU4Y7H334JVNOTYO2FOUUXTKOCVYEHNJJ4HRVXVVXWOVA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.423702486 (406h47m7.294785531s) [noise=true , address=77.90.31.14:4663 , piece_id=X2NSIUKNVP2Y2MABYR3UPGZJSPF3CPS3EU6JYVNHX74RZTSGEQMA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:09.117304385 (406h47m8.988387421s) [upload=357142]: Next returned nil segment Dec 22 14:16:09.117321889 (406h47m8.988404924s) [upload=357142]: Waiting for error group managing segments... Dec 22 14:16:09.403723791 (406h47m9.274806826s) [noise=true , address=145.79.227.43:4157 , piece_id=PLKGYWWJYEHSEMRCRFVQUTXTGYN6LYHFF4HPL2TOW6F7Q4K6QL7Q, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:09.416402182 (406h47m9.287485247s) [noise=true , address=145.79.227.50:4374 , piece_id=2XXNMOGMBDEQLIBGHJ6E25INEXYZR7JWYKA76WB5CKLK42IU5L4Q, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:09.488678941 (406h47m9.359761976s) [noise=true , address=145.79.227.36:4517 , piece_id=DKVCILBC2JWXLG6G23ZMKEWQNDXYR4UNVD2XESJ57JIKIS5AXACA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:09.591529928 (406h47m9.462612963s) [noise=true , address=77.90.31.14:4663 , piece_id=X2NSIUKNVP2Y2MABYR3UPGZJSPF3CPS3EU6JYVNHX74RZTSGEQMA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:10.003825194 (406h47m9.874908229s) [noise=true , address=77.90.31.10:4519 , piece_id=FMJHUEO445SYIHAV47SFERXWEFDSCCN36WPT4RJTXABWLJ6IO2VQ, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.151289256 (406h47m11.022372301s) [noise=true , address=145.79.227.10:4401 , piece_id=HVP7A7MFY7IM2S4P5QFS3T2BNYINUQRSF5TJ3OZLWH5REX2QCNJA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.190801796 (406h47m11.061884841s) [noise=true , address=145.79.227.48:4302 , piece_id=WYPWNL6UUKRVETGRAKJJMS5US3B4LQBJ6TRDYBUTRLXKGPFPPEEA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.216210448 (406h47m11.087293483s) [noise=true , address=145.79.227.49:4339 , piece_id=ANLDLFM6BWPIJJIKHCY7CMWU2KIYTTNWA4ETR3DGNZG6GO7XM5QQ, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.447886943 (406h47m11.318969978s) [noise=true , address=145.79.227.17:4972 , piece_id=XOKLGQXCXZCOWYK5H4TPHRDHLLJXTII36XFZBX4XWYKHFE24WVOQ, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.513411914 (406h47m11.384494949s) [noise=true , address=145.79.227.41:4085 , piece_id=OGRSMMM4JHNTZZZURJHFOBUAOPGDC6T2PCHDEX7YL7SADHG6AN6A, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.768195260 (406h47m11.639278305s) [noise=true , address=145.79.227.29:4942 , piece_id=OMRNVS4BDDHYEFK5BEKTMKHTJUMJ4MY55FXPWOHZ3NVGHNST7KZA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.791646034 (406h47m11.66272908s) [noise=true , address=145.79.227.45:4231 , piece_id=I7F2TC2A6E35C3LPHT4D62ZOAZRSMF6XZZWVKJEI6SUHWTRQ6DFA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.807399862 (406h47m11.678482907s) [noise=true , address=145.79.227.46:4603 , piece_id=5KQGOFJZAEFI5OGUVUIK2HEBRMMUVGXEOD47OQIALAIMVCPEMLFA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.807424800 (406h47m11.678507845s) [seg_pos={0 0}, upload=357142]: Segment reached optimal threshold of 13 pieces. Dec 22 14:16:11.807688312 (406h47m11.678771357s) [noise=true , address=145.79.227.37:4566 , piece_id=4H2NM22RU4Y7H334JVNOTYO2FOUUXTKOCVYEHNJJ4HRVXVVXWOVA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F): protocol: expected piece hash; context canceled; EOF Dec 22 14:16:11.807698351 (406h47m11.678781396s) [noise=true , address=145.79.227.37:4566 , piece_id=4H2NM22RU4Y7H334JVNOTYO2FOUUXTKOCVYEHNJJ4HRVXVVXWOVA, seg_pos={0 0}, upload=357142]: Long tail cancellation due to optimal threshold Dec 22 14:16:11.807723459 (406h47m11.678806514s) [noise=true , address=145.79.227.18:4007 , piece_id=UMT4AQFVRTREIJDRAPCZTJY3URZYOHLX37QQTG5L53JFDW63ULQQ, seg_pos={0 0}, upload=357142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp): protocol: expected piece hash; context canceled; EOF Dec 22 14:16:11.807736343 (406h47m11.678819398s) [noise=true , address=145.79.227.18:4007 , piece_id=UMT4AQFVRTREIJDRAPCZTJY3URZYOHLX37QQTG5L53JFDW63ULQQ, seg_pos={0 0}, upload=357142]: Long tail cancellation due to optimal threshold Dec 22 14:16:11.807862974 (406h47m11.678946019s) [noise=true , address=45.91.116.8:4048 , piece_id=HQWNONHI5SJYLPULF2H75PVYLOPOPJTEC5ZPOZUT2HHLA6IC5ALA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 22 14:16:11.807871991 (406h47m11.678955026s) [noise=true , address=45.91.116.8:4048 , piece_id=HQWNONHI5SJYLPULF2H75PVYLOPOPJTEC5ZPOZUT2HHLA6IC5ALA, seg_pos={0 0}, upload=357142]: Long tail cancellation due to optimal threshold Dec 22 14:16:11.807896659 (406h47m11.678979704s) [seg_pos={0 0}, upload=357142]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 14:16:11.808114984 (406h47m11.679198019s) [upload=357142]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 14:16:11.808121817 (406h47m11.679204852s) [upload=357142]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 14:16:12.460702542 (406h47m12.331785577s) [upload=357142]: Done uploading segments... Dec 22 14:16:12.460721377 (406h47m12.331804412s) [upload=357142]: Upload finished. err: <nil> Duration: 5.810009835s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [.....X....X.X....XX..X..............X.....X.X.......X.......] duration: 4.698087975s trace: https://jaeger.dev.storj.io/trace/1c84c110f15266bb FAIL (2025-12-22 14:24:42 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.698114756s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................................X.........X.......] duration: 5.350891691s trace: https://jaeger.dev.storj.io/trace/7e72d81acd8b3600 FAIL (2025-12-22 14:24:27 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.350914894s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [......X....X.X....XX..X..............X.....X.X.......X......] duration: 3.761474812s trace: https://jaeger.dev.storj.io/trace/1c84c110f15266bb FAIL (2025-12-22 14:24:25 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.761563632s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................................X.........X......] duration: 5.187461512s trace: https://jaeger.dev.storj.io/trace/7e72d81acd8b3600 FAIL (2025-12-22 14:24:10 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.187500137s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 1.402169155s trace: https://jaeger.dev.storj.io/trace/1e7f77260788e649 FAIL (2025-12-22 14:24:10 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.402194352s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [....X..................X...X..................X.............] duration: 6.800062526s trace: https://jaeger.dev.storj.io/trace/30697bf8e531908d FAIL (2025-12-22 14:24:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 22 14:16:07.103235978 (406h47m6.974319013s) [upload=357142]: Starting upload Dec 22 14:16:07.103295332 (406h47m6.974378377s) [upload=357142]: Done starting upload Dec 22 14:16:07.103399340 (406h47m6.974482375s) [upload=357142]: Uploading segments... Dec 22 14:16:07.103487287 (406h47m6.974570332s) [upload=357142]: Got next segment. Inline: false Dec 22 14:16:07.103515991 (406h47m6.974599026s) [upload=357142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 14:16:07.103520289 (406h47m6.974603325s) [upload=357142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 14:16:07.415289927 (406h47m7.286372962s) [seg_pos={0 0}, upload=357142]: Begin upload segment... Dec 22 14:16:07.415336014 (406h47m7.286419049s) [seg_pos={0 0}, upload=357142]: Done begin upload segment. Dec 22 14:16:07.415363607 (406h47m7.286446652s) [noise=true , address=145.79.227.17:4972 , piece_id=XOKLGQXCXZCOWYK5H4TPHRDHLLJXTII36XFZBX4XWYKHFE24WVOQ, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.415526457 (406h47m7.286609492s) [noise=true , address=145.79.227.18:4007 , piece_id=UMT4AQFVRTREIJDRAPCZTJY3URZYOHLX37QQTG5L53JFDW63ULQQ, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.416519017 (406h47m7.287602062s) [noise=true , address=145.79.227.48:4302 , piece_id=WYPWNL6UUKRVETGRAKJJMS5US3B4LQBJ6TRDYBUTRLXKGPFPPEEA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.417017777 (406h47m7.288100822s) [noise=true , address=145.79.227.50:4374 , piece_id=2XXNMOGMBDEQLIBGHJ6E25INEXYZR7JWYKA76WB5CKLK42IU5L4Q, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.417027014 (406h47m7.288110059s) [noise=true , address=145.79.227.43:4157 , piece_id=PLKGYWWJYEHSEMRCRFVQUTXTGYN6LYHFF4HPL2TOW6F7Q4K6QL7Q, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.417966864 (406h47m7.289049909s) [noise=true , address=45.91.116.8:4048 , piece_id=HQWNONHI5SJYLPULF2H75PVYLOPOPJTEC5ZPOZUT2HHLA6IC5ALA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.418744234 (406h47m7.289827269s) [noise=true , address=145.79.227.49:4339 , piece_id=ANLDLFM6BWPIJJIKHCY7CMWU2KIYTTNWA4ETR3DGNZG6GO7XM5QQ, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.419502227 (406h47m7.290585263s) [noise=true , address=77.90.31.10:4519 , piece_id=FMJHUEO445SYIHAV47SFERXWEFDSCCN36WPT4RJTXABWLJ6IO2VQ, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.419641743 (406h47m7.290724788s) [noise=true , address=145.79.227.10:4401 , piece_id=HVP7A7MFY7IM2S4P5QFS3T2BNYINUQRSF5TJ3OZLWH5REX2QCNJA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.420879160 (406h47m7.291962205s) [noise=true , address=145.79.227.29:4942 , piece_id=OMRNVS4BDDHYEFK5BEKTMKHTJUMJ4MY55FXPWOHZ3NVGHNST7KZA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.420914838 (406h47m7.291997883s) [noise=true , address=145.79.227.41:4085 , piece_id=OGRSMMM4JHNTZZZURJHFOBUAOPGDC6T2PCHDEX7YL7SADHG6AN6A, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.421340087 (406h47m7.292423132s) [noise=true , address=145.79.227.36:4517 , piece_id=DKVCILBC2JWXLG6G23ZMKEWQNDXYR4UNVD2XESJ57JIKIS5AXACA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.421402066 (406h47m7.292485101s) [noise=true , address=145.79.227.46:4603 , piece_id=5KQGOFJZAEFI5OGUVUIK2HEBRMMUVGXEOD47OQIALAIMVCPEMLFA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.422719935 (406h47m7.29380297s) [noise=true , address=145.79.227.45:4231 , piece_id=I7F2TC2A6E35C3LPHT4D62ZOAZRSMF6XZZWVKJEI6SUHWTRQ6DFA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.423220067 (406h47m7.294303112s) [noise=true , address=145.79.227.37:4566 , piece_id=4H2NM22RU4Y7H334JVNOTYO2FOUUXTKOCVYEHNJJ4HRVXVVXWOVA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.423702486 (406h47m7.294785531s) [noise=true , address=77.90.31.14:4663 , piece_id=X2NSIUKNVP2Y2MABYR3UPGZJSPF3CPS3EU6JYVNHX74RZTSGEQMA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:09.117304385 (406h47m8.988387421s) [upload=357142]: Next returned nil segment Dec 22 14:16:09.117321889 (406h47m8.988404924s) [upload=357142]: Waiting for error group managing segments... Dec 22 14:16:09.403723791 (406h47m9.274806826s) [noise=true , address=145.79.227.43:4157 , piece_id=PLKGYWWJYEHSEMRCRFVQUTXTGYN6LYHFF4HPL2TOW6F7Q4K6QL7Q, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:09.416402182 (406h47m9.287485247s) [noise=true , address=145.79.227.50:4374 , piece_id=2XXNMOGMBDEQLIBGHJ6E25INEXYZR7JWYKA76WB5CKLK42IU5L4Q, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:09.488678941 (406h47m9.359761976s) [noise=true , address=145.79.227.36:4517 , piece_id=DKVCILBC2JWXLG6G23ZMKEWQNDXYR4UNVD2XESJ57JIKIS5AXACA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:09.591529928 (406h47m9.462612963s) [noise=true , address=77.90.31.14:4663 , piece_id=X2NSIUKNVP2Y2MABYR3UPGZJSPF3CPS3EU6JYVNHX74RZTSGEQMA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:10.003825194 (406h47m9.874908229s) [noise=true , address=77.90.31.10:4519 , piece_id=FMJHUEO445SYIHAV47SFERXWEFDSCCN36WPT4RJTXABWLJ6IO2VQ, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.151289256 (406h47m11.022372301s) [noise=true , address=145.79.227.10:4401 , piece_id=HVP7A7MFY7IM2S4P5QFS3T2BNYINUQRSF5TJ3OZLWH5REX2QCNJA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.190801796 (406h47m11.061884841s) [noise=true , address=145.79.227.48:4302 , piece_id=WYPWNL6UUKRVETGRAKJJMS5US3B4LQBJ6TRDYBUTRLXKGPFPPEEA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.216210448 (406h47m11.087293483s) [noise=true , address=145.79.227.49:4339 , piece_id=ANLDLFM6BWPIJJIKHCY7CMWU2KIYTTNWA4ETR3DGNZG6GO7XM5QQ, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.447886943 (406h47m11.318969978s) [noise=true , address=145.79.227.17:4972 , piece_id=XOKLGQXCXZCOWYK5H4TPHRDHLLJXTII36XFZBX4XWYKHFE24WVOQ, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.513411914 (406h47m11.384494949s) [noise=true , address=145.79.227.41:4085 , piece_id=OGRSMMM4JHNTZZZURJHFOBUAOPGDC6T2PCHDEX7YL7SADHG6AN6A, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.768195260 (406h47m11.639278305s) [noise=true , address=145.79.227.29:4942 , piece_id=OMRNVS4BDDHYEFK5BEKTMKHTJUMJ4MY55FXPWOHZ3NVGHNST7KZA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.791646034 (406h47m11.66272908s) [noise=true , address=145.79.227.45:4231 , piece_id=I7F2TC2A6E35C3LPHT4D62ZOAZRSMF6XZZWVKJEI6SUHWTRQ6DFA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.807399862 (406h47m11.678482907s) [noise=true , address=145.79.227.46:4603 , piece_id=5KQGOFJZAEFI5OGUVUIK2HEBRMMUVGXEOD47OQIALAIMVCPEMLFA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.807424800 (406h47m11.678507845s) [seg_pos={0 0}, upload=357142]: Segment reached optimal threshold of 13 pieces. Dec 22 14:16:11.807688312 (406h47m11.678771357s) [noise=true , address=145.79.227.37:4566 , piece_id=4H2NM22RU4Y7H334JVNOTYO2FOUUXTKOCVYEHNJJ4HRVXVVXWOVA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F): protocol: expected piece hash; context canceled; EOF Dec 22 14:16:11.807698351 (406h47m11.678781396s) [noise=true , address=145.79.227.37:4566 , piece_id=4H2NM22RU4Y7H334JVNOTYO2FOUUXTKOCVYEHNJJ4HRVXVVXWOVA, seg_pos={0 0}, upload=357142]: Long tail cancellation due to optimal threshold Dec 22 14:16:11.807723459 (406h47m11.678806514s) [noise=true , address=145.79.227.18:4007 , piece_id=UMT4AQFVRTREIJDRAPCZTJY3URZYOHLX37QQTG5L53JFDW63ULQQ, seg_pos={0 0}, upload=357142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp): protocol: expected piece hash; context canceled; EOF Dec 22 14:16:11.807736343 (406h47m11.678819398s) [noise=true , address=145.79.227.18:4007 , piece_id=UMT4AQFVRTREIJDRAPCZTJY3URZYOHLX37QQTG5L53JFDW63ULQQ, seg_pos={0 0}, upload=357142]: Long tail cancellation due to optimal threshold Dec 22 14:16:11.807862974 (406h47m11.678946019s) [noise=true , address=45.91.116.8:4048 , piece_id=HQWNONHI5SJYLPULF2H75PVYLOPOPJTEC5ZPOZUT2HHLA6IC5ALA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 22 14:16:11.807871991 (406h47m11.678955026s) [noise=true , address=45.91.116.8:4048 , piece_id=HQWNONHI5SJYLPULF2H75PVYLOPOPJTEC5ZPOZUT2HHLA6IC5ALA, seg_pos={0 0}, upload=357142]: Long tail cancellation due to optimal threshold Dec 22 14:16:11.807896659 (406h47m11.678979704s) [seg_pos={0 0}, upload=357142]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 14:16:11.808114984 (406h47m11.679198019s) [upload=357142]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 14:16:11.808121817 (406h47m11.679204852s) [upload=357142]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 14:16:12.460702542 (406h47m12.331785577s) [upload=357142]: Done uploading segments... Dec 22 14:16:12.460721377 (406h47m12.331804412s) [upload=357142]: Upload finished. err: <nil> Duration: 6.800092994s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [.......X....X.X....XX..X..............X.....X.X.......X.....] duration: 3.721440307s trace: https://jaeger.dev.storj.io/trace/1c84c110f15266bb FAIL (2025-12-22 14:23:35 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.72146862s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....X..................X...X..................X............] duration: 5.86868438s trace: https://jaeger.dev.storj.io/trace/30697bf8e531908d FAIL (2025-12-22 14:23:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 22 14:16:07.103235978 (406h47m6.974319013s) [upload=357142]: Starting upload Dec 22 14:16:07.103295332 (406h47m6.974378377s) [upload=357142]: Done starting upload Dec 22 14:16:07.103399340 (406h47m6.974482375s) [upload=357142]: Uploading segments... Dec 22 14:16:07.103487287 (406h47m6.974570332s) [upload=357142]: Got next segment. Inline: false Dec 22 14:16:07.103515991 (406h47m6.974599026s) [upload=357142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 14:16:07.103520289 (406h47m6.974603325s) [upload=357142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 14:16:07.415289927 (406h47m7.286372962s) [seg_pos={0 0}, upload=357142]: Begin upload segment... Dec 22 14:16:07.415336014 (406h47m7.286419049s) [seg_pos={0 0}, upload=357142]: Done begin upload segment. Dec 22 14:16:07.415363607 (406h47m7.286446652s) [noise=true , address=145.79.227.17:4972 , piece_id=XOKLGQXCXZCOWYK5H4TPHRDHLLJXTII36XFZBX4XWYKHFE24WVOQ, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.415526457 (406h47m7.286609492s) [noise=true , address=145.79.227.18:4007 , piece_id=UMT4AQFVRTREIJDRAPCZTJY3URZYOHLX37QQTG5L53JFDW63ULQQ, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.416519017 (406h47m7.287602062s) [noise=true , address=145.79.227.48:4302 , piece_id=WYPWNL6UUKRVETGRAKJJMS5US3B4LQBJ6TRDYBUTRLXKGPFPPEEA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.417017777 (406h47m7.288100822s) [noise=true , address=145.79.227.50:4374 , piece_id=2XXNMOGMBDEQLIBGHJ6E25INEXYZR7JWYKA76WB5CKLK42IU5L4Q, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.417027014 (406h47m7.288110059s) [noise=true , address=145.79.227.43:4157 , piece_id=PLKGYWWJYEHSEMRCRFVQUTXTGYN6LYHFF4HPL2TOW6F7Q4K6QL7Q, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.417966864 (406h47m7.289049909s) [noise=true , address=45.91.116.8:4048 , piece_id=HQWNONHI5SJYLPULF2H75PVYLOPOPJTEC5ZPOZUT2HHLA6IC5ALA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.418744234 (406h47m7.289827269s) [noise=true , address=145.79.227.49:4339 , piece_id=ANLDLFM6BWPIJJIKHCY7CMWU2KIYTTNWA4ETR3DGNZG6GO7XM5QQ, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.419502227 (406h47m7.290585263s) [noise=true , address=77.90.31.10:4519 , piece_id=FMJHUEO445SYIHAV47SFERXWEFDSCCN36WPT4RJTXABWLJ6IO2VQ, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.419641743 (406h47m7.290724788s) [noise=true , address=145.79.227.10:4401 , piece_id=HVP7A7MFY7IM2S4P5QFS3T2BNYINUQRSF5TJ3OZLWH5REX2QCNJA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.420879160 (406h47m7.291962205s) [noise=true , address=145.79.227.29:4942 , piece_id=OMRNVS4BDDHYEFK5BEKTMKHTJUMJ4MY55FXPWOHZ3NVGHNST7KZA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.420914838 (406h47m7.291997883s) [noise=true , address=145.79.227.41:4085 , piece_id=OGRSMMM4JHNTZZZURJHFOBUAOPGDC6T2PCHDEX7YL7SADHG6AN6A, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.421340087 (406h47m7.292423132s) [noise=true , address=145.79.227.36:4517 , piece_id=DKVCILBC2JWXLG6G23ZMKEWQNDXYR4UNVD2XESJ57JIKIS5AXACA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.421402066 (406h47m7.292485101s) [noise=true , address=145.79.227.46:4603 , piece_id=5KQGOFJZAEFI5OGUVUIK2HEBRMMUVGXEOD47OQIALAIMVCPEMLFA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.422719935 (406h47m7.29380297s) [noise=true , address=145.79.227.45:4231 , piece_id=I7F2TC2A6E35C3LPHT4D62ZOAZRSMF6XZZWVKJEI6SUHWTRQ6DFA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.423220067 (406h47m7.294303112s) [noise=true , address=145.79.227.37:4566 , piece_id=4H2NM22RU4Y7H334JVNOTYO2FOUUXTKOCVYEHNJJ4HRVXVVXWOVA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.423702486 (406h47m7.294785531s) [noise=true , address=77.90.31.14:4663 , piece_id=X2NSIUKNVP2Y2MABYR3UPGZJSPF3CPS3EU6JYVNHX74RZTSGEQMA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:09.117304385 (406h47m8.988387421s) [upload=357142]: Next returned nil segment Dec 22 14:16:09.117321889 (406h47m8.988404924s) [upload=357142]: Waiting for error group managing segments... Dec 22 14:16:09.403723791 (406h47m9.274806826s) [noise=true , address=145.79.227.43:4157 , piece_id=PLKGYWWJYEHSEMRCRFVQUTXTGYN6LYHFF4HPL2TOW6F7Q4K6QL7Q, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:09.416402182 (406h47m9.287485247s) [noise=true , address=145.79.227.50:4374 , piece_id=2XXNMOGMBDEQLIBGHJ6E25INEXYZR7JWYKA76WB5CKLK42IU5L4Q, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:09.488678941 (406h47m9.359761976s) [noise=true , address=145.79.227.36:4517 , piece_id=DKVCILBC2JWXLG6G23ZMKEWQNDXYR4UNVD2XESJ57JIKIS5AXACA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:09.591529928 (406h47m9.462612963s) [noise=true , address=77.90.31.14:4663 , piece_id=X2NSIUKNVP2Y2MABYR3UPGZJSPF3CPS3EU6JYVNHX74RZTSGEQMA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:10.003825194 (406h47m9.874908229s) [noise=true , address=77.90.31.10:4519 , piece_id=FMJHUEO445SYIHAV47SFERXWEFDSCCN36WPT4RJTXABWLJ6IO2VQ, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.151289256 (406h47m11.022372301s) [noise=true , address=145.79.227.10:4401 , piece_id=HVP7A7MFY7IM2S4P5QFS3T2BNYINUQRSF5TJ3OZLWH5REX2QCNJA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.190801796 (406h47m11.061884841s) [noise=true , address=145.79.227.48:4302 , piece_id=WYPWNL6UUKRVETGRAKJJMS5US3B4LQBJ6TRDYBUTRLXKGPFPPEEA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.216210448 (406h47m11.087293483s) [noise=true , address=145.79.227.49:4339 , piece_id=ANLDLFM6BWPIJJIKHCY7CMWU2KIYTTNWA4ETR3DGNZG6GO7XM5QQ, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.447886943 (406h47m11.318969978s) [noise=true , address=145.79.227.17:4972 , piece_id=XOKLGQXCXZCOWYK5H4TPHRDHLLJXTII36XFZBX4XWYKHFE24WVOQ, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.513411914 (406h47m11.384494949s) [noise=true , address=145.79.227.41:4085 , piece_id=OGRSMMM4JHNTZZZURJHFOBUAOPGDC6T2PCHDEX7YL7SADHG6AN6A, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.768195260 (406h47m11.639278305s) [noise=true , address=145.79.227.29:4942 , piece_id=OMRNVS4BDDHYEFK5BEKTMKHTJUMJ4MY55FXPWOHZ3NVGHNST7KZA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.791646034 (406h47m11.66272908s) [noise=true , address=145.79.227.45:4231 , piece_id=I7F2TC2A6E35C3LPHT4D62ZOAZRSMF6XZZWVKJEI6SUHWTRQ6DFA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.807399862 (406h47m11.678482907s) [noise=true , address=145.79.227.46:4603 , piece_id=5KQGOFJZAEFI5OGUVUIK2HEBRMMUVGXEOD47OQIALAIMVCPEMLFA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.807424800 (406h47m11.678507845s) [seg_pos={0 0}, upload=357142]: Segment reached optimal threshold of 13 pieces. Dec 22 14:16:11.807688312 (406h47m11.678771357s) [noise=true , address=145.79.227.37:4566 , piece_id=4H2NM22RU4Y7H334JVNOTYO2FOUUXTKOCVYEHNJJ4HRVXVVXWOVA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F): protocol: expected piece hash; context canceled; EOF Dec 22 14:16:11.807698351 (406h47m11.678781396s) [noise=true , address=145.79.227.37:4566 , piece_id=4H2NM22RU4Y7H334JVNOTYO2FOUUXTKOCVYEHNJJ4HRVXVVXWOVA, seg_pos={0 0}, upload=357142]: Long tail cancellation due to optimal threshold Dec 22 14:16:11.807723459 (406h47m11.678806514s) [noise=true , address=145.79.227.18:4007 , piece_id=UMT4AQFVRTREIJDRAPCZTJY3URZYOHLX37QQTG5L53JFDW63ULQQ, seg_pos={0 0}, upload=357142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp): protocol: expected piece hash; context canceled; EOF Dec 22 14:16:11.807736343 (406h47m11.678819398s) [noise=true , address=145.79.227.18:4007 , piece_id=UMT4AQFVRTREIJDRAPCZTJY3URZYOHLX37QQTG5L53JFDW63ULQQ, seg_pos={0 0}, upload=357142]: Long tail cancellation due to optimal threshold Dec 22 14:16:11.807862974 (406h47m11.678946019s) [noise=true , address=45.91.116.8:4048 , piece_id=HQWNONHI5SJYLPULF2H75PVYLOPOPJTEC5ZPOZUT2HHLA6IC5ALA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 22 14:16:11.807871991 (406h47m11.678955026s) [noise=true , address=45.91.116.8:4048 , piece_id=HQWNONHI5SJYLPULF2H75PVYLOPOPJTEC5ZPOZUT2HHLA6IC5ALA, seg_pos={0 0}, upload=357142]: Long tail cancellation due to optimal threshold Dec 22 14:16:11.807896659 (406h47m11.678979704s) [seg_pos={0 0}, upload=357142]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 14:16:11.808114984 (406h47m11.679198019s) [upload=357142]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 14:16:11.808121817 (406h47m11.679204852s) [upload=357142]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 14:16:12.460702542 (406h47m12.331785577s) [upload=357142]: Done uploading segments... Dec 22 14:16:12.460721377 (406h47m12.331804412s) [upload=357142]: Upload finished. err: <nil> Duration: 5.868709458s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [........X....X.X....XX..X..............X.....X.X.......X....] duration: 4.683111143s trace: https://jaeger.dev.storj.io/trace/1c84c110f15266bb FAIL (2025-12-22 14:23:04 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.683144818s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 1.478323426s trace: https://jaeger.dev.storj.io/trace/1e7f77260788e649 FAIL (2025-12-22 14:23:01 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.478352942s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................................X.........X.....] duration: 5.205688596s trace: https://jaeger.dev.storj.io/trace/7e72d81acd8b3600 FAIL (2025-12-22 14:22:56 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.205719865s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [......X..................X...X..................X...........] duration: 5.709599936s trace: https://jaeger.dev.storj.io/trace/30697bf8e531908d FAIL (2025-12-22 14:22:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 22 14:16:07.103235978 (406h47m6.974319013s) [upload=357142]: Starting upload Dec 22 14:16:07.103295332 (406h47m6.974378377s) [upload=357142]: Done starting upload Dec 22 14:16:07.103399340 (406h47m6.974482375s) [upload=357142]: Uploading segments... Dec 22 14:16:07.103487287 (406h47m6.974570332s) [upload=357142]: Got next segment. Inline: false Dec 22 14:16:07.103515991 (406h47m6.974599026s) [upload=357142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 14:16:07.103520289 (406h47m6.974603325s) [upload=357142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 14:16:07.415289927 (406h47m7.286372962s) [seg_pos={0 0}, upload=357142]: Begin upload segment... Dec 22 14:16:07.415336014 (406h47m7.286419049s) [seg_pos={0 0}, upload=357142]: Done begin upload segment. Dec 22 14:16:07.415363607 (406h47m7.286446652s) [noise=true , address=145.79.227.17:4972 , piece_id=XOKLGQXCXZCOWYK5H4TPHRDHLLJXTII36XFZBX4XWYKHFE24WVOQ, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.415526457 (406h47m7.286609492s) [noise=true , address=145.79.227.18:4007 , piece_id=UMT4AQFVRTREIJDRAPCZTJY3URZYOHLX37QQTG5L53JFDW63ULQQ, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.416519017 (406h47m7.287602062s) [noise=true , address=145.79.227.48:4302 , piece_id=WYPWNL6UUKRVETGRAKJJMS5US3B4LQBJ6TRDYBUTRLXKGPFPPEEA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.417017777 (406h47m7.288100822s) [noise=true , address=145.79.227.50:4374 , piece_id=2XXNMOGMBDEQLIBGHJ6E25INEXYZR7JWYKA76WB5CKLK42IU5L4Q, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.417027014 (406h47m7.288110059s) [noise=true , address=145.79.227.43:4157 , piece_id=PLKGYWWJYEHSEMRCRFVQUTXTGYN6LYHFF4HPL2TOW6F7Q4K6QL7Q, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.417966864 (406h47m7.289049909s) [noise=true , address=45.91.116.8:4048 , piece_id=HQWNONHI5SJYLPULF2H75PVYLOPOPJTEC5ZPOZUT2HHLA6IC5ALA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.418744234 (406h47m7.289827269s) [noise=true , address=145.79.227.49:4339 , piece_id=ANLDLFM6BWPIJJIKHCY7CMWU2KIYTTNWA4ETR3DGNZG6GO7XM5QQ, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.419502227 (406h47m7.290585263s) [noise=true , address=77.90.31.10:4519 , piece_id=FMJHUEO445SYIHAV47SFERXWEFDSCCN36WPT4RJTXABWLJ6IO2VQ, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.419641743 (406h47m7.290724788s) [noise=true , address=145.79.227.10:4401 , piece_id=HVP7A7MFY7IM2S4P5QFS3T2BNYINUQRSF5TJ3OZLWH5REX2QCNJA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.420879160 (406h47m7.291962205s) [noise=true , address=145.79.227.29:4942 , piece_id=OMRNVS4BDDHYEFK5BEKTMKHTJUMJ4MY55FXPWOHZ3NVGHNST7KZA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.420914838 (406h47m7.291997883s) [noise=true , address=145.79.227.41:4085 , piece_id=OGRSMMM4JHNTZZZURJHFOBUAOPGDC6T2PCHDEX7YL7SADHG6AN6A, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.421340087 (406h47m7.292423132s) [noise=true , address=145.79.227.36:4517 , piece_id=DKVCILBC2JWXLG6G23ZMKEWQNDXYR4UNVD2XESJ57JIKIS5AXACA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.421402066 (406h47m7.292485101s) [noise=true , address=145.79.227.46:4603 , piece_id=5KQGOFJZAEFI5OGUVUIK2HEBRMMUVGXEOD47OQIALAIMVCPEMLFA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.422719935 (406h47m7.29380297s) [noise=true , address=145.79.227.45:4231 , piece_id=I7F2TC2A6E35C3LPHT4D62ZOAZRSMF6XZZWVKJEI6SUHWTRQ6DFA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.423220067 (406h47m7.294303112s) [noise=true , address=145.79.227.37:4566 , piece_id=4H2NM22RU4Y7H334JVNOTYO2FOUUXTKOCVYEHNJJ4HRVXVVXWOVA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.423702486 (406h47m7.294785531s) [noise=true , address=77.90.31.14:4663 , piece_id=X2NSIUKNVP2Y2MABYR3UPGZJSPF3CPS3EU6JYVNHX74RZTSGEQMA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:09.117304385 (406h47m8.988387421s) [upload=357142]: Next returned nil segment Dec 22 14:16:09.117321889 (406h47m8.988404924s) [upload=357142]: Waiting for error group managing segments... Dec 22 14:16:09.403723791 (406h47m9.274806826s) [noise=true , address=145.79.227.43:4157 , piece_id=PLKGYWWJYEHSEMRCRFVQUTXTGYN6LYHFF4HPL2TOW6F7Q4K6QL7Q, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:09.416402182 (406h47m9.287485247s) [noise=true , address=145.79.227.50:4374 , piece_id=2XXNMOGMBDEQLIBGHJ6E25INEXYZR7JWYKA76WB5CKLK42IU5L4Q, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:09.488678941 (406h47m9.359761976s) [noise=true , address=145.79.227.36:4517 , piece_id=DKVCILBC2JWXLG6G23ZMKEWQNDXYR4UNVD2XESJ57JIKIS5AXACA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:09.591529928 (406h47m9.462612963s) [noise=true , address=77.90.31.14:4663 , piece_id=X2NSIUKNVP2Y2MABYR3UPGZJSPF3CPS3EU6JYVNHX74RZTSGEQMA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:10.003825194 (406h47m9.874908229s) [noise=true , address=77.90.31.10:4519 , piece_id=FMJHUEO445SYIHAV47SFERXWEFDSCCN36WPT4RJTXABWLJ6IO2VQ, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.151289256 (406h47m11.022372301s) [noise=true , address=145.79.227.10:4401 , piece_id=HVP7A7MFY7IM2S4P5QFS3T2BNYINUQRSF5TJ3OZLWH5REX2QCNJA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.190801796 (406h47m11.061884841s) [noise=true , address=145.79.227.48:4302 , piece_id=WYPWNL6UUKRVETGRAKJJMS5US3B4LQBJ6TRDYBUTRLXKGPFPPEEA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.216210448 (406h47m11.087293483s) [noise=true , address=145.79.227.49:4339 , piece_id=ANLDLFM6BWPIJJIKHCY7CMWU2KIYTTNWA4ETR3DGNZG6GO7XM5QQ, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.447886943 (406h47m11.318969978s) [noise=true , address=145.79.227.17:4972 , piece_id=XOKLGQXCXZCOWYK5H4TPHRDHLLJXTII36XFZBX4XWYKHFE24WVOQ, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.513411914 (406h47m11.384494949s) [noise=true , address=145.79.227.41:4085 , piece_id=OGRSMMM4JHNTZZZURJHFOBUAOPGDC6T2PCHDEX7YL7SADHG6AN6A, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.768195260 (406h47m11.639278305s) [noise=true , address=145.79.227.29:4942 , piece_id=OMRNVS4BDDHYEFK5BEKTMKHTJUMJ4MY55FXPWOHZ3NVGHNST7KZA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.791646034 (406h47m11.66272908s) [noise=true , address=145.79.227.45:4231 , piece_id=I7F2TC2A6E35C3LPHT4D62ZOAZRSMF6XZZWVKJEI6SUHWTRQ6DFA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.807399862 (406h47m11.678482907s) [noise=true , address=145.79.227.46:4603 , piece_id=5KQGOFJZAEFI5OGUVUIK2HEBRMMUVGXEOD47OQIALAIMVCPEMLFA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.807424800 (406h47m11.678507845s) [seg_pos={0 0}, upload=357142]: Segment reached optimal threshold of 13 pieces. Dec 22 14:16:11.807688312 (406h47m11.678771357s) [noise=true , address=145.79.227.37:4566 , piece_id=4H2NM22RU4Y7H334JVNOTYO2FOUUXTKOCVYEHNJJ4HRVXVVXWOVA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F): protocol: expected piece hash; context canceled; EOF Dec 22 14:16:11.807698351 (406h47m11.678781396s) [noise=true , address=145.79.227.37:4566 , piece_id=4H2NM22RU4Y7H334JVNOTYO2FOUUXTKOCVYEHNJJ4HRVXVVXWOVA, seg_pos={0 0}, upload=357142]: Long tail cancellation due to optimal threshold Dec 22 14:16:11.807723459 (406h47m11.678806514s) [noise=true , address=145.79.227.18:4007 , piece_id=UMT4AQFVRTREIJDRAPCZTJY3URZYOHLX37QQTG5L53JFDW63ULQQ, seg_pos={0 0}, upload=357142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp): protocol: expected piece hash; context canceled; EOF Dec 22 14:16:11.807736343 (406h47m11.678819398s) [noise=true , address=145.79.227.18:4007 , piece_id=UMT4AQFVRTREIJDRAPCZTJY3URZYOHLX37QQTG5L53JFDW63ULQQ, seg_pos={0 0}, upload=357142]: Long tail cancellation due to optimal threshold Dec 22 14:16:11.807862974 (406h47m11.678946019s) [noise=true , address=45.91.116.8:4048 , piece_id=HQWNONHI5SJYLPULF2H75PVYLOPOPJTEC5ZPOZUT2HHLA6IC5ALA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 22 14:16:11.807871991 (406h47m11.678955026s) [noise=true , address=45.91.116.8:4048 , piece_id=HQWNONHI5SJYLPULF2H75PVYLOPOPJTEC5ZPOZUT2HHLA6IC5ALA, seg_pos={0 0}, upload=357142]: Long tail cancellation due to optimal threshold Dec 22 14:16:11.807896659 (406h47m11.678979704s) [seg_pos={0 0}, upload=357142]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 14:16:11.808114984 (406h47m11.679198019s) [upload=357142]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 14:16:11.808121817 (406h47m11.679204852s) [upload=357142]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 14:16:12.460702542 (406h47m12.331785577s) [upload=357142]: Done uploading segments... Dec 22 14:16:12.460721377 (406h47m12.331804412s) [upload=357142]: Upload finished. err: <nil> Duration: 5.709631916s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [.........X....X.X....XX..X..............X.....X.X.......X...] duration: 3.422244562s trace: https://jaeger.dev.storj.io/trace/1c84c110f15266bb FAIL (2025-12-22 14:22:31 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.422288735s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 1.433040614s trace: https://jaeger.dev.storj.io/trace/1e7f77260788e649 FAIL (2025-12-22 14:22:18 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.433069388s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [..........X....X.X....XX..X..............X.....X.X.......X..] duration: 3.556025058s trace: https://jaeger.dev.storj.io/trace/1c84c110f15266bb FAIL (2025-12-22 14:22:16 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.556057259s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......X..................X...X..................X..........] duration: 6.730955109s trace: https://jaeger.dev.storj.io/trace/30697bf8e531908d FAIL (2025-12-22 14:22:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 22 14:16:07.103235978 (406h47m6.974319013s) [upload=357142]: Starting upload Dec 22 14:16:07.103295332 (406h47m6.974378377s) [upload=357142]: Done starting upload Dec 22 14:16:07.103399340 (406h47m6.974482375s) [upload=357142]: Uploading segments... Dec 22 14:16:07.103487287 (406h47m6.974570332s) [upload=357142]: Got next segment. Inline: false Dec 22 14:16:07.103515991 (406h47m6.974599026s) [upload=357142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 14:16:07.103520289 (406h47m6.974603325s) [upload=357142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 14:16:07.415289927 (406h47m7.286372962s) [seg_pos={0 0}, upload=357142]: Begin upload segment... Dec 22 14:16:07.415336014 (406h47m7.286419049s) [seg_pos={0 0}, upload=357142]: Done begin upload segment. Dec 22 14:16:07.415363607 (406h47m7.286446652s) [noise=true , address=145.79.227.17:4972 , piece_id=XOKLGQXCXZCOWYK5H4TPHRDHLLJXTII36XFZBX4XWYKHFE24WVOQ, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.415526457 (406h47m7.286609492s) [noise=true , address=145.79.227.18:4007 , piece_id=UMT4AQFVRTREIJDRAPCZTJY3URZYOHLX37QQTG5L53JFDW63ULQQ, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.416519017 (406h47m7.287602062s) [noise=true , address=145.79.227.48:4302 , piece_id=WYPWNL6UUKRVETGRAKJJMS5US3B4LQBJ6TRDYBUTRLXKGPFPPEEA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.417017777 (406h47m7.288100822s) [noise=true , address=145.79.227.50:4374 , piece_id=2XXNMOGMBDEQLIBGHJ6E25INEXYZR7JWYKA76WB5CKLK42IU5L4Q, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.417027014 (406h47m7.288110059s) [noise=true , address=145.79.227.43:4157 , piece_id=PLKGYWWJYEHSEMRCRFVQUTXTGYN6LYHFF4HPL2TOW6F7Q4K6QL7Q, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.417966864 (406h47m7.289049909s) [noise=true , address=45.91.116.8:4048 , piece_id=HQWNONHI5SJYLPULF2H75PVYLOPOPJTEC5ZPOZUT2HHLA6IC5ALA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.418744234 (406h47m7.289827269s) [noise=true , address=145.79.227.49:4339 , piece_id=ANLDLFM6BWPIJJIKHCY7CMWU2KIYTTNWA4ETR3DGNZG6GO7XM5QQ, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.419502227 (406h47m7.290585263s) [noise=true , address=77.90.31.10:4519 , piece_id=FMJHUEO445SYIHAV47SFERXWEFDSCCN36WPT4RJTXABWLJ6IO2VQ, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.419641743 (406h47m7.290724788s) [noise=true , address=145.79.227.10:4401 , piece_id=HVP7A7MFY7IM2S4P5QFS3T2BNYINUQRSF5TJ3OZLWH5REX2QCNJA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.420879160 (406h47m7.291962205s) [noise=true , address=145.79.227.29:4942 , piece_id=OMRNVS4BDDHYEFK5BEKTMKHTJUMJ4MY55FXPWOHZ3NVGHNST7KZA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.420914838 (406h47m7.291997883s) [noise=true , address=145.79.227.41:4085 , piece_id=OGRSMMM4JHNTZZZURJHFOBUAOPGDC6T2PCHDEX7YL7SADHG6AN6A, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.421340087 (406h47m7.292423132s) [noise=true , address=145.79.227.36:4517 , piece_id=DKVCILBC2JWXLG6G23ZMKEWQNDXYR4UNVD2XESJ57JIKIS5AXACA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.421402066 (406h47m7.292485101s) [noise=true , address=145.79.227.46:4603 , piece_id=5KQGOFJZAEFI5OGUVUIK2HEBRMMUVGXEOD47OQIALAIMVCPEMLFA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.422719935 (406h47m7.29380297s) [noise=true , address=145.79.227.45:4231 , piece_id=I7F2TC2A6E35C3LPHT4D62ZOAZRSMF6XZZWVKJEI6SUHWTRQ6DFA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.423220067 (406h47m7.294303112s) [noise=true , address=145.79.227.37:4566 , piece_id=4H2NM22RU4Y7H334JVNOTYO2FOUUXTKOCVYEHNJJ4HRVXVVXWOVA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.423702486 (406h47m7.294785531s) [noise=true , address=77.90.31.14:4663 , piece_id=X2NSIUKNVP2Y2MABYR3UPGZJSPF3CPS3EU6JYVNHX74RZTSGEQMA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:09.117304385 (406h47m8.988387421s) [upload=357142]: Next returned nil segment Dec 22 14:16:09.117321889 (406h47m8.988404924s) [upload=357142]: Waiting for error group managing segments... Dec 22 14:16:09.403723791 (406h47m9.274806826s) [noise=true , address=145.79.227.43:4157 , piece_id=PLKGYWWJYEHSEMRCRFVQUTXTGYN6LYHFF4HPL2TOW6F7Q4K6QL7Q, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:09.416402182 (406h47m9.287485247s) [noise=true , address=145.79.227.50:4374 , piece_id=2XXNMOGMBDEQLIBGHJ6E25INEXYZR7JWYKA76WB5CKLK42IU5L4Q, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:09.488678941 (406h47m9.359761976s) [noise=true , address=145.79.227.36:4517 , piece_id=DKVCILBC2JWXLG6G23ZMKEWQNDXYR4UNVD2XESJ57JIKIS5AXACA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:09.591529928 (406h47m9.462612963s) [noise=true , address=77.90.31.14:4663 , piece_id=X2NSIUKNVP2Y2MABYR3UPGZJSPF3CPS3EU6JYVNHX74RZTSGEQMA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:10.003825194 (406h47m9.874908229s) [noise=true , address=77.90.31.10:4519 , piece_id=FMJHUEO445SYIHAV47SFERXWEFDSCCN36WPT4RJTXABWLJ6IO2VQ, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.151289256 (406h47m11.022372301s) [noise=true , address=145.79.227.10:4401 , piece_id=HVP7A7MFY7IM2S4P5QFS3T2BNYINUQRSF5TJ3OZLWH5REX2QCNJA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.190801796 (406h47m11.061884841s) [noise=true , address=145.79.227.48:4302 , piece_id=WYPWNL6UUKRVETGRAKJJMS5US3B4LQBJ6TRDYBUTRLXKGPFPPEEA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.216210448 (406h47m11.087293483s) [noise=true , address=145.79.227.49:4339 , piece_id=ANLDLFM6BWPIJJIKHCY7CMWU2KIYTTNWA4ETR3DGNZG6GO7XM5QQ, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.447886943 (406h47m11.318969978s) [noise=true , address=145.79.227.17:4972 , piece_id=XOKLGQXCXZCOWYK5H4TPHRDHLLJXTII36XFZBX4XWYKHFE24WVOQ, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.513411914 (406h47m11.384494949s) [noise=true , address=145.79.227.41:4085 , piece_id=OGRSMMM4JHNTZZZURJHFOBUAOPGDC6T2PCHDEX7YL7SADHG6AN6A, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.768195260 (406h47m11.639278305s) [noise=true , address=145.79.227.29:4942 , piece_id=OMRNVS4BDDHYEFK5BEKTMKHTJUMJ4MY55FXPWOHZ3NVGHNST7KZA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.791646034 (406h47m11.66272908s) [noise=true , address=145.79.227.45:4231 , piece_id=I7F2TC2A6E35C3LPHT4D62ZOAZRSMF6XZZWVKJEI6SUHWTRQ6DFA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.807399862 (406h47m11.678482907s) [noise=true , address=145.79.227.46:4603 , piece_id=5KQGOFJZAEFI5OGUVUIK2HEBRMMUVGXEOD47OQIALAIMVCPEMLFA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.807424800 (406h47m11.678507845s) [seg_pos={0 0}, upload=357142]: Segment reached optimal threshold of 13 pieces. Dec 22 14:16:11.807688312 (406h47m11.678771357s) [noise=true , address=145.79.227.37:4566 , piece_id=4H2NM22RU4Y7H334JVNOTYO2FOUUXTKOCVYEHNJJ4HRVXVVXWOVA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F): protocol: expected piece hash; context canceled; EOF Dec 22 14:16:11.807698351 (406h47m11.678781396s) [noise=true , address=145.79.227.37:4566 , piece_id=4H2NM22RU4Y7H334JVNOTYO2FOUUXTKOCVYEHNJJ4HRVXVVXWOVA, seg_pos={0 0}, upload=357142]: Long tail cancellation due to optimal threshold Dec 22 14:16:11.807723459 (406h47m11.678806514s) [noise=true , address=145.79.227.18:4007 , piece_id=UMT4AQFVRTREIJDRAPCZTJY3URZYOHLX37QQTG5L53JFDW63ULQQ, seg_pos={0 0}, upload=357142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp): protocol: expected piece hash; context canceled; EOF Dec 22 14:16:11.807736343 (406h47m11.678819398s) [noise=true , address=145.79.227.18:4007 , piece_id=UMT4AQFVRTREIJDRAPCZTJY3URZYOHLX37QQTG5L53JFDW63ULQQ, seg_pos={0 0}, upload=357142]: Long tail cancellation due to optimal threshold Dec 22 14:16:11.807862974 (406h47m11.678946019s) [noise=true , address=45.91.116.8:4048 , piece_id=HQWNONHI5SJYLPULF2H75PVYLOPOPJTEC5ZPOZUT2HHLA6IC5ALA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 22 14:16:11.807871991 (406h47m11.678955026s) [noise=true , address=45.91.116.8:4048 , piece_id=HQWNONHI5SJYLPULF2H75PVYLOPOPJTEC5ZPOZUT2HHLA6IC5ALA, seg_pos={0 0}, upload=357142]: Long tail cancellation due to optimal threshold Dec 22 14:16:11.807896659 (406h47m11.678979704s) [seg_pos={0 0}, upload=357142]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 14:16:11.808114984 (406h47m11.679198019s) [upload=357142]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 14:16:11.808121817 (406h47m11.679204852s) [upload=357142]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 14:16:12.460702542 (406h47m12.331785577s) [upload=357142]: Done uploading segments... Dec 22 14:16:12.460721377 (406h47m12.331804412s) [upload=357142]: Upload finished. err: <nil> Duration: 6.730986009s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................................X.........X....] duration: 5.132929153s trace: https://jaeger.dev.storj.io/trace/7e72d81acd8b3600 FAIL (2025-12-22 14:21:58 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.132967355s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [........X..................X...X..................X.........] duration: 22.898721182s trace: https://jaeger.dev.storj.io/trace/30697bf8e531908d FAIL (2025-12-22 14:21:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 22 14:16:07.103235978 (406h47m6.974319013s) [upload=357142]: Starting upload Dec 22 14:16:07.103295332 (406h47m6.974378377s) [upload=357142]: Done starting upload Dec 22 14:16:07.103399340 (406h47m6.974482375s) [upload=357142]: Uploading segments... Dec 22 14:16:07.103487287 (406h47m6.974570332s) [upload=357142]: Got next segment. Inline: false Dec 22 14:16:07.103515991 (406h47m6.974599026s) [upload=357142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 14:16:07.103520289 (406h47m6.974603325s) [upload=357142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 14:16:07.415289927 (406h47m7.286372962s) [seg_pos={0 0}, upload=357142]: Begin upload segment... Dec 22 14:16:07.415336014 (406h47m7.286419049s) [seg_pos={0 0}, upload=357142]: Done begin upload segment. Dec 22 14:16:07.415363607 (406h47m7.286446652s) [noise=true , address=145.79.227.17:4972 , piece_id=XOKLGQXCXZCOWYK5H4TPHRDHLLJXTII36XFZBX4XWYKHFE24WVOQ, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.415526457 (406h47m7.286609492s) [noise=true , address=145.79.227.18:4007 , piece_id=UMT4AQFVRTREIJDRAPCZTJY3URZYOHLX37QQTG5L53JFDW63ULQQ, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.416519017 (406h47m7.287602062s) [noise=true , address=145.79.227.48:4302 , piece_id=WYPWNL6UUKRVETGRAKJJMS5US3B4LQBJ6TRDYBUTRLXKGPFPPEEA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.417017777 (406h47m7.288100822s) [noise=true , address=145.79.227.50:4374 , piece_id=2XXNMOGMBDEQLIBGHJ6E25INEXYZR7JWYKA76WB5CKLK42IU5L4Q, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.417027014 (406h47m7.288110059s) [noise=true , address=145.79.227.43:4157 , piece_id=PLKGYWWJYEHSEMRCRFVQUTXTGYN6LYHFF4HPL2TOW6F7Q4K6QL7Q, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.417966864 (406h47m7.289049909s) [noise=true , address=45.91.116.8:4048 , piece_id=HQWNONHI5SJYLPULF2H75PVYLOPOPJTEC5ZPOZUT2HHLA6IC5ALA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.418744234 (406h47m7.289827269s) [noise=true , address=145.79.227.49:4339 , piece_id=ANLDLFM6BWPIJJIKHCY7CMWU2KIYTTNWA4ETR3DGNZG6GO7XM5QQ, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.419502227 (406h47m7.290585263s) [noise=true , address=77.90.31.10:4519 , piece_id=FMJHUEO445SYIHAV47SFERXWEFDSCCN36WPT4RJTXABWLJ6IO2VQ, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.419641743 (406h47m7.290724788s) [noise=true , address=145.79.227.10:4401 , piece_id=HVP7A7MFY7IM2S4P5QFS3T2BNYINUQRSF5TJ3OZLWH5REX2QCNJA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.420879160 (406h47m7.291962205s) [noise=true , address=145.79.227.29:4942 , piece_id=OMRNVS4BDDHYEFK5BEKTMKHTJUMJ4MY55FXPWOHZ3NVGHNST7KZA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.420914838 (406h47m7.291997883s) [noise=true , address=145.79.227.41:4085 , piece_id=OGRSMMM4JHNTZZZURJHFOBUAOPGDC6T2PCHDEX7YL7SADHG6AN6A, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.421340087 (406h47m7.292423132s) [noise=true , address=145.79.227.36:4517 , piece_id=DKVCILBC2JWXLG6G23ZMKEWQNDXYR4UNVD2XESJ57JIKIS5AXACA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.421402066 (406h47m7.292485101s) [noise=true , address=145.79.227.46:4603 , piece_id=5KQGOFJZAEFI5OGUVUIK2HEBRMMUVGXEOD47OQIALAIMVCPEMLFA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.422719935 (406h47m7.29380297s) [noise=true , address=145.79.227.45:4231 , piece_id=I7F2TC2A6E35C3LPHT4D62ZOAZRSMF6XZZWVKJEI6SUHWTRQ6DFA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.423220067 (406h47m7.294303112s) [noise=true , address=145.79.227.37:4566 , piece_id=4H2NM22RU4Y7H334JVNOTYO2FOUUXTKOCVYEHNJJ4HRVXVVXWOVA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.423702486 (406h47m7.294785531s) [noise=true , address=77.90.31.14:4663 , piece_id=X2NSIUKNVP2Y2MABYR3UPGZJSPF3CPS3EU6JYVNHX74RZTSGEQMA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:09.117304385 (406h47m8.988387421s) [upload=357142]: Next returned nil segment Dec 22 14:16:09.117321889 (406h47m8.988404924s) [upload=357142]: Waiting for error group managing segments... Dec 22 14:16:09.403723791 (406h47m9.274806826s) [noise=true , address=145.79.227.43:4157 , piece_id=PLKGYWWJYEHSEMRCRFVQUTXTGYN6LYHFF4HPL2TOW6F7Q4K6QL7Q, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:09.416402182 (406h47m9.287485247s) [noise=true , address=145.79.227.50:4374 , piece_id=2XXNMOGMBDEQLIBGHJ6E25INEXYZR7JWYKA76WB5CKLK42IU5L4Q, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:09.488678941 (406h47m9.359761976s) [noise=true , address=145.79.227.36:4517 , piece_id=DKVCILBC2JWXLG6G23ZMKEWQNDXYR4UNVD2XESJ57JIKIS5AXACA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:09.591529928 (406h47m9.462612963s) [noise=true , address=77.90.31.14:4663 , piece_id=X2NSIUKNVP2Y2MABYR3UPGZJSPF3CPS3EU6JYVNHX74RZTSGEQMA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:10.003825194 (406h47m9.874908229s) [noise=true , address=77.90.31.10:4519 , piece_id=FMJHUEO445SYIHAV47SFERXWEFDSCCN36WPT4RJTXABWLJ6IO2VQ, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.151289256 (406h47m11.022372301s) [noise=true , address=145.79.227.10:4401 , piece_id=HVP7A7MFY7IM2S4P5QFS3T2BNYINUQRSF5TJ3OZLWH5REX2QCNJA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.190801796 (406h47m11.061884841s) [noise=true , address=145.79.227.48:4302 , piece_id=WYPWNL6UUKRVETGRAKJJMS5US3B4LQBJ6TRDYBUTRLXKGPFPPEEA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.216210448 (406h47m11.087293483s) [noise=true , address=145.79.227.49:4339 , piece_id=ANLDLFM6BWPIJJIKHCY7CMWU2KIYTTNWA4ETR3DGNZG6GO7XM5QQ, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.447886943 (406h47m11.318969978s) [noise=true , address=145.79.227.17:4972 , piece_id=XOKLGQXCXZCOWYK5H4TPHRDHLLJXTII36XFZBX4XWYKHFE24WVOQ, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.513411914 (406h47m11.384494949s) [noise=true , address=145.79.227.41:4085 , piece_id=OGRSMMM4JHNTZZZURJHFOBUAOPGDC6T2PCHDEX7YL7SADHG6AN6A, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.768195260 (406h47m11.639278305s) [noise=true , address=145.79.227.29:4942 , piece_id=OMRNVS4BDDHYEFK5BEKTMKHTJUMJ4MY55FXPWOHZ3NVGHNST7KZA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.791646034 (406h47m11.66272908s) [noise=true , address=145.79.227.45:4231 , piece_id=I7F2TC2A6E35C3LPHT4D62ZOAZRSMF6XZZWVKJEI6SUHWTRQ6DFA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.807399862 (406h47m11.678482907s) [noise=true , address=145.79.227.46:4603 , piece_id=5KQGOFJZAEFI5OGUVUIK2HEBRMMUVGXEOD47OQIALAIMVCPEMLFA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.807424800 (406h47m11.678507845s) [seg_pos={0 0}, upload=357142]: Segment reached optimal threshold of 13 pieces. Dec 22 14:16:11.807688312 (406h47m11.678771357s) [noise=true , address=145.79.227.37:4566 , piece_id=4H2NM22RU4Y7H334JVNOTYO2FOUUXTKOCVYEHNJJ4HRVXVVXWOVA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F): protocol: expected piece hash; context canceled; EOF Dec 22 14:16:11.807698351 (406h47m11.678781396s) [noise=true , address=145.79.227.37:4566 , piece_id=4H2NM22RU4Y7H334JVNOTYO2FOUUXTKOCVYEHNJJ4HRVXVVXWOVA, seg_pos={0 0}, upload=357142]: Long tail cancellation due to optimal threshold Dec 22 14:16:11.807723459 (406h47m11.678806514s) [noise=true , address=145.79.227.18:4007 , piece_id=UMT4AQFVRTREIJDRAPCZTJY3URZYOHLX37QQTG5L53JFDW63ULQQ, seg_pos={0 0}, upload=357142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp): protocol: expected piece hash; context canceled; EOF Dec 22 14:16:11.807736343 (406h47m11.678819398s) [noise=true , address=145.79.227.18:4007 , piece_id=UMT4AQFVRTREIJDRAPCZTJY3URZYOHLX37QQTG5L53JFDW63ULQQ, seg_pos={0 0}, upload=357142]: Long tail cancellation due to optimal threshold Dec 22 14:16:11.807862974 (406h47m11.678946019s) [noise=true , address=45.91.116.8:4048 , piece_id=HQWNONHI5SJYLPULF2H75PVYLOPOPJTEC5ZPOZUT2HHLA6IC5ALA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 22 14:16:11.807871991 (406h47m11.678955026s) [noise=true , address=45.91.116.8:4048 , piece_id=HQWNONHI5SJYLPULF2H75PVYLOPOPJTEC5ZPOZUT2HHLA6IC5ALA, seg_pos={0 0}, upload=357142]: Long tail cancellation due to optimal threshold Dec 22 14:16:11.807896659 (406h47m11.678979704s) [seg_pos={0 0}, upload=357142]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 14:16:11.808114984 (406h47m11.679198019s) [upload=357142]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 14:16:11.808121817 (406h47m11.679204852s) [upload=357142]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 14:16:12.460702542 (406h47m12.331785577s) [upload=357142]: Done uploading segments... Dec 22 14:16:12.460721377 (406h47m12.331804412s) [upload=357142]: Upload finished. err: <nil> Duration: 22.8988146s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [...........X....X.X....XX..X..............X.....X.X.......X.] duration: 4.470898652s trace: https://jaeger.dev.storj.io/trace/1c84c110f15266bb FAIL (2025-12-22 14:21:21 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.47092368s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 1.450379601s trace: https://jaeger.dev.storj.io/trace/1e7f77260788e649 FAIL (2025-12-22 14:21:15 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.450433472s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................................X.........X...] duration: 6.102315942s trace: https://jaeger.dev.storj.io/trace/7e72d81acd8b3600 FAIL (2025-12-22 14:20:55 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6.102348945s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 1.507873629s trace: https://jaeger.dev.storj.io/trace/1e7f77260788e649 FAIL (2025-12-22 14:20:54 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.507912694s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.........X..................X...X..................X........] duration: 5.812099194s trace: https://jaeger.dev.storj.io/trace/30697bf8e531908d FAIL (2025-12-22 14:20:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 22 14:16:07.103235978 (406h47m6.974319013s) [upload=357142]: Starting upload Dec 22 14:16:07.103295332 (406h47m6.974378377s) [upload=357142]: Done starting upload Dec 22 14:16:07.103399340 (406h47m6.974482375s) [upload=357142]: Uploading segments... Dec 22 14:16:07.103487287 (406h47m6.974570332s) [upload=357142]: Got next segment. Inline: false Dec 22 14:16:07.103515991 (406h47m6.974599026s) [upload=357142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 14:16:07.103520289 (406h47m6.974603325s) [upload=357142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 14:16:07.415289927 (406h47m7.286372962s) [seg_pos={0 0}, upload=357142]: Begin upload segment... Dec 22 14:16:07.415336014 (406h47m7.286419049s) [seg_pos={0 0}, upload=357142]: Done begin upload segment. Dec 22 14:16:07.415363607 (406h47m7.286446652s) [noise=true , address=145.79.227.17:4972 , piece_id=XOKLGQXCXZCOWYK5H4TPHRDHLLJXTII36XFZBX4XWYKHFE24WVOQ, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.415526457 (406h47m7.286609492s) [noise=true , address=145.79.227.18:4007 , piece_id=UMT4AQFVRTREIJDRAPCZTJY3URZYOHLX37QQTG5L53JFDW63ULQQ, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.416519017 (406h47m7.287602062s) [noise=true , address=145.79.227.48:4302 , piece_id=WYPWNL6UUKRVETGRAKJJMS5US3B4LQBJ6TRDYBUTRLXKGPFPPEEA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.417017777 (406h47m7.288100822s) [noise=true , address=145.79.227.50:4374 , piece_id=2XXNMOGMBDEQLIBGHJ6E25INEXYZR7JWYKA76WB5CKLK42IU5L4Q, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.417027014 (406h47m7.288110059s) [noise=true , address=145.79.227.43:4157 , piece_id=PLKGYWWJYEHSEMRCRFVQUTXTGYN6LYHFF4HPL2TOW6F7Q4K6QL7Q, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.417966864 (406h47m7.289049909s) [noise=true , address=45.91.116.8:4048 , piece_id=HQWNONHI5SJYLPULF2H75PVYLOPOPJTEC5ZPOZUT2HHLA6IC5ALA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.418744234 (406h47m7.289827269s) [noise=true , address=145.79.227.49:4339 , piece_id=ANLDLFM6BWPIJJIKHCY7CMWU2KIYTTNWA4ETR3DGNZG6GO7XM5QQ, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.419502227 (406h47m7.290585263s) [noise=true , address=77.90.31.10:4519 , piece_id=FMJHUEO445SYIHAV47SFERXWEFDSCCN36WPT4RJTXABWLJ6IO2VQ, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.419641743 (406h47m7.290724788s) [noise=true , address=145.79.227.10:4401 , piece_id=HVP7A7MFY7IM2S4P5QFS3T2BNYINUQRSF5TJ3OZLWH5REX2QCNJA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.420879160 (406h47m7.291962205s) [noise=true , address=145.79.227.29:4942 , piece_id=OMRNVS4BDDHYEFK5BEKTMKHTJUMJ4MY55FXPWOHZ3NVGHNST7KZA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.420914838 (406h47m7.291997883s) [noise=true , address=145.79.227.41:4085 , piece_id=OGRSMMM4JHNTZZZURJHFOBUAOPGDC6T2PCHDEX7YL7SADHG6AN6A, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.421340087 (406h47m7.292423132s) [noise=true , address=145.79.227.36:4517 , piece_id=DKVCILBC2JWXLG6G23ZMKEWQNDXYR4UNVD2XESJ57JIKIS5AXACA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.421402066 (406h47m7.292485101s) [noise=true , address=145.79.227.46:4603 , piece_id=5KQGOFJZAEFI5OGUVUIK2HEBRMMUVGXEOD47OQIALAIMVCPEMLFA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.422719935 (406h47m7.29380297s) [noise=true , address=145.79.227.45:4231 , piece_id=I7F2TC2A6E35C3LPHT4D62ZOAZRSMF6XZZWVKJEI6SUHWTRQ6DFA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.423220067 (406h47m7.294303112s) [noise=true , address=145.79.227.37:4566 , piece_id=4H2NM22RU4Y7H334JVNOTYO2FOUUXTKOCVYEHNJJ4HRVXVVXWOVA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.423702486 (406h47m7.294785531s) [noise=true , address=77.90.31.14:4663 , piece_id=X2NSIUKNVP2Y2MABYR3UPGZJSPF3CPS3EU6JYVNHX74RZTSGEQMA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:09.117304385 (406h47m8.988387421s) [upload=357142]: Next returned nil segment Dec 22 14:16:09.117321889 (406h47m8.988404924s) [upload=357142]: Waiting for error group managing segments... Dec 22 14:16:09.403723791 (406h47m9.274806826s) [noise=true , address=145.79.227.43:4157 , piece_id=PLKGYWWJYEHSEMRCRFVQUTXTGYN6LYHFF4HPL2TOW6F7Q4K6QL7Q, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:09.416402182 (406h47m9.287485247s) [noise=true , address=145.79.227.50:4374 , piece_id=2XXNMOGMBDEQLIBGHJ6E25INEXYZR7JWYKA76WB5CKLK42IU5L4Q, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:09.488678941 (406h47m9.359761976s) [noise=true , address=145.79.227.36:4517 , piece_id=DKVCILBC2JWXLG6G23ZMKEWQNDXYR4UNVD2XESJ57JIKIS5AXACA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:09.591529928 (406h47m9.462612963s) [noise=true , address=77.90.31.14:4663 , piece_id=X2NSIUKNVP2Y2MABYR3UPGZJSPF3CPS3EU6JYVNHX74RZTSGEQMA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:10.003825194 (406h47m9.874908229s) [noise=true , address=77.90.31.10:4519 , piece_id=FMJHUEO445SYIHAV47SFERXWEFDSCCN36WPT4RJTXABWLJ6IO2VQ, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.151289256 (406h47m11.022372301s) [noise=true , address=145.79.227.10:4401 , piece_id=HVP7A7MFY7IM2S4P5QFS3T2BNYINUQRSF5TJ3OZLWH5REX2QCNJA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.190801796 (406h47m11.061884841s) [noise=true , address=145.79.227.48:4302 , piece_id=WYPWNL6UUKRVETGRAKJJMS5US3B4LQBJ6TRDYBUTRLXKGPFPPEEA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.216210448 (406h47m11.087293483s) [noise=true , address=145.79.227.49:4339 , piece_id=ANLDLFM6BWPIJJIKHCY7CMWU2KIYTTNWA4ETR3DGNZG6GO7XM5QQ, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.447886943 (406h47m11.318969978s) [noise=true , address=145.79.227.17:4972 , piece_id=XOKLGQXCXZCOWYK5H4TPHRDHLLJXTII36XFZBX4XWYKHFE24WVOQ, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.513411914 (406h47m11.384494949s) [noise=true , address=145.79.227.41:4085 , piece_id=OGRSMMM4JHNTZZZURJHFOBUAOPGDC6T2PCHDEX7YL7SADHG6AN6A, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.768195260 (406h47m11.639278305s) [noise=true , address=145.79.227.29:4942 , piece_id=OMRNVS4BDDHYEFK5BEKTMKHTJUMJ4MY55FXPWOHZ3NVGHNST7KZA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.791646034 (406h47m11.66272908s) [noise=true , address=145.79.227.45:4231 , piece_id=I7F2TC2A6E35C3LPHT4D62ZOAZRSMF6XZZWVKJEI6SUHWTRQ6DFA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.807399862 (406h47m11.678482907s) [noise=true , address=145.79.227.46:4603 , piece_id=5KQGOFJZAEFI5OGUVUIK2HEBRMMUVGXEOD47OQIALAIMVCPEMLFA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.807424800 (406h47m11.678507845s) [seg_pos={0 0}, upload=357142]: Segment reached optimal threshold of 13 pieces. Dec 22 14:16:11.807688312 (406h47m11.678771357s) [noise=true , address=145.79.227.37:4566 , piece_id=4H2NM22RU4Y7H334JVNOTYO2FOUUXTKOCVYEHNJJ4HRVXVVXWOVA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F): protocol: expected piece hash; context canceled; EOF Dec 22 14:16:11.807698351 (406h47m11.678781396s) [noise=true , address=145.79.227.37:4566 , piece_id=4H2NM22RU4Y7H334JVNOTYO2FOUUXTKOCVYEHNJJ4HRVXVVXWOVA, seg_pos={0 0}, upload=357142]: Long tail cancellation due to optimal threshold Dec 22 14:16:11.807723459 (406h47m11.678806514s) [noise=true , address=145.79.227.18:4007 , piece_id=UMT4AQFVRTREIJDRAPCZTJY3URZYOHLX37QQTG5L53JFDW63ULQQ, seg_pos={0 0}, upload=357142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp): protocol: expected piece hash; context canceled; EOF Dec 22 14:16:11.807736343 (406h47m11.678819398s) [noise=true , address=145.79.227.18:4007 , piece_id=UMT4AQFVRTREIJDRAPCZTJY3URZYOHLX37QQTG5L53JFDW63ULQQ, seg_pos={0 0}, upload=357142]: Long tail cancellation due to optimal threshold Dec 22 14:16:11.807862974 (406h47m11.678946019s) [noise=true , address=45.91.116.8:4048 , piece_id=HQWNONHI5SJYLPULF2H75PVYLOPOPJTEC5ZPOZUT2HHLA6IC5ALA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 22 14:16:11.807871991 (406h47m11.678955026s) [noise=true , address=45.91.116.8:4048 , piece_id=HQWNONHI5SJYLPULF2H75PVYLOPOPJTEC5ZPOZUT2HHLA6IC5ALA, seg_pos={0 0}, upload=357142]: Long tail cancellation due to optimal threshold Dec 22 14:16:11.807896659 (406h47m11.678979704s) [seg_pos={0 0}, upload=357142]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 14:16:11.808114984 (406h47m11.679198019s) [upload=357142]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 14:16:11.808121817 (406h47m11.679204852s) [upload=357142]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 14:16:12.460702542 (406h47m12.331785577s) [upload=357142]: Done uploading segments... Dec 22 14:16:12.460721377 (406h47m12.331804412s) [upload=357142]: Upload finished. err: <nil> Duration: 5.812127249s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................................X.........X..] duration: 5.400263876s trace: https://jaeger.dev.storj.io/trace/7e72d81acd8b3600 FAIL (2025-12-22 14:20:10 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.400292961s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [............X....X.X....XX..X..............X.....X.X.......X] duration: 5.000765617s trace: https://jaeger.dev.storj.io/trace/1c84c110f15266bb FAIL (2025-12-22 14:20:10 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000788752s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 1.416288103s trace: https://jaeger.dev.storj.io/trace/1e7f77260788e649 FAIL (2025-12-22 14:20:01 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.416321445s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..........X..................X...X..................X.......] duration: 6.025711972s trace: https://jaeger.dev.storj.io/trace/30697bf8e531908d FAIL (2025-12-22 14:19:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 22 14:16:07.103235978 (406h47m6.974319013s) [upload=357142]: Starting upload Dec 22 14:16:07.103295332 (406h47m6.974378377s) [upload=357142]: Done starting upload Dec 22 14:16:07.103399340 (406h47m6.974482375s) [upload=357142]: Uploading segments... Dec 22 14:16:07.103487287 (406h47m6.974570332s) [upload=357142]: Got next segment. Inline: false Dec 22 14:16:07.103515991 (406h47m6.974599026s) [upload=357142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 14:16:07.103520289 (406h47m6.974603325s) [upload=357142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 14:16:07.415289927 (406h47m7.286372962s) [seg_pos={0 0}, upload=357142]: Begin upload segment... Dec 22 14:16:07.415336014 (406h47m7.286419049s) [seg_pos={0 0}, upload=357142]: Done begin upload segment. Dec 22 14:16:07.415363607 (406h47m7.286446652s) [noise=true , address=145.79.227.17:4972 , piece_id=XOKLGQXCXZCOWYK5H4TPHRDHLLJXTII36XFZBX4XWYKHFE24WVOQ, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.415526457 (406h47m7.286609492s) [noise=true , address=145.79.227.18:4007 , piece_id=UMT4AQFVRTREIJDRAPCZTJY3URZYOHLX37QQTG5L53JFDW63ULQQ, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.416519017 (406h47m7.287602062s) [noise=true , address=145.79.227.48:4302 , piece_id=WYPWNL6UUKRVETGRAKJJMS5US3B4LQBJ6TRDYBUTRLXKGPFPPEEA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.417017777 (406h47m7.288100822s) [noise=true , address=145.79.227.50:4374 , piece_id=2XXNMOGMBDEQLIBGHJ6E25INEXYZR7JWYKA76WB5CKLK42IU5L4Q, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.417027014 (406h47m7.288110059s) [noise=true , address=145.79.227.43:4157 , piece_id=PLKGYWWJYEHSEMRCRFVQUTXTGYN6LYHFF4HPL2TOW6F7Q4K6QL7Q, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.417966864 (406h47m7.289049909s) [noise=true , address=45.91.116.8:4048 , piece_id=HQWNONHI5SJYLPULF2H75PVYLOPOPJTEC5ZPOZUT2HHLA6IC5ALA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.418744234 (406h47m7.289827269s) [noise=true , address=145.79.227.49:4339 , piece_id=ANLDLFM6BWPIJJIKHCY7CMWU2KIYTTNWA4ETR3DGNZG6GO7XM5QQ, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.419502227 (406h47m7.290585263s) [noise=true , address=77.90.31.10:4519 , piece_id=FMJHUEO445SYIHAV47SFERXWEFDSCCN36WPT4RJTXABWLJ6IO2VQ, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.419641743 (406h47m7.290724788s) [noise=true , address=145.79.227.10:4401 , piece_id=HVP7A7MFY7IM2S4P5QFS3T2BNYINUQRSF5TJ3OZLWH5REX2QCNJA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.420879160 (406h47m7.291962205s) [noise=true , address=145.79.227.29:4942 , piece_id=OMRNVS4BDDHYEFK5BEKTMKHTJUMJ4MY55FXPWOHZ3NVGHNST7KZA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.420914838 (406h47m7.291997883s) [noise=true , address=145.79.227.41:4085 , piece_id=OGRSMMM4JHNTZZZURJHFOBUAOPGDC6T2PCHDEX7YL7SADHG6AN6A, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.421340087 (406h47m7.292423132s) [noise=true , address=145.79.227.36:4517 , piece_id=DKVCILBC2JWXLG6G23ZMKEWQNDXYR4UNVD2XESJ57JIKIS5AXACA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.421402066 (406h47m7.292485101s) [noise=true , address=145.79.227.46:4603 , piece_id=5KQGOFJZAEFI5OGUVUIK2HEBRMMUVGXEOD47OQIALAIMVCPEMLFA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.422719935 (406h47m7.29380297s) [noise=true , address=145.79.227.45:4231 , piece_id=I7F2TC2A6E35C3LPHT4D62ZOAZRSMF6XZZWVKJEI6SUHWTRQ6DFA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.423220067 (406h47m7.294303112s) [noise=true , address=145.79.227.37:4566 , piece_id=4H2NM22RU4Y7H334JVNOTYO2FOUUXTKOCVYEHNJJ4HRVXVVXWOVA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.423702486 (406h47m7.294785531s) [noise=true , address=77.90.31.14:4663 , piece_id=X2NSIUKNVP2Y2MABYR3UPGZJSPF3CPS3EU6JYVNHX74RZTSGEQMA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:09.117304385 (406h47m8.988387421s) [upload=357142]: Next returned nil segment Dec 22 14:16:09.117321889 (406h47m8.988404924s) [upload=357142]: Waiting for error group managing segments... Dec 22 14:16:09.403723791 (406h47m9.274806826s) [noise=true , address=145.79.227.43:4157 , piece_id=PLKGYWWJYEHSEMRCRFVQUTXTGYN6LYHFF4HPL2TOW6F7Q4K6QL7Q, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:09.416402182 (406h47m9.287485247s) [noise=true , address=145.79.227.50:4374 , piece_id=2XXNMOGMBDEQLIBGHJ6E25INEXYZR7JWYKA76WB5CKLK42IU5L4Q, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:09.488678941 (406h47m9.359761976s) [noise=true , address=145.79.227.36:4517 , piece_id=DKVCILBC2JWXLG6G23ZMKEWQNDXYR4UNVD2XESJ57JIKIS5AXACA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:09.591529928 (406h47m9.462612963s) [noise=true , address=77.90.31.14:4663 , piece_id=X2NSIUKNVP2Y2MABYR3UPGZJSPF3CPS3EU6JYVNHX74RZTSGEQMA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:10.003825194 (406h47m9.874908229s) [noise=true , address=77.90.31.10:4519 , piece_id=FMJHUEO445SYIHAV47SFERXWEFDSCCN36WPT4RJTXABWLJ6IO2VQ, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.151289256 (406h47m11.022372301s) [noise=true , address=145.79.227.10:4401 , piece_id=HVP7A7MFY7IM2S4P5QFS3T2BNYINUQRSF5TJ3OZLWH5REX2QCNJA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.190801796 (406h47m11.061884841s) [noise=true , address=145.79.227.48:4302 , piece_id=WYPWNL6UUKRVETGRAKJJMS5US3B4LQBJ6TRDYBUTRLXKGPFPPEEA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.216210448 (406h47m11.087293483s) [noise=true , address=145.79.227.49:4339 , piece_id=ANLDLFM6BWPIJJIKHCY7CMWU2KIYTTNWA4ETR3DGNZG6GO7XM5QQ, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.447886943 (406h47m11.318969978s) [noise=true , address=145.79.227.17:4972 , piece_id=XOKLGQXCXZCOWYK5H4TPHRDHLLJXTII36XFZBX4XWYKHFE24WVOQ, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.513411914 (406h47m11.384494949s) [noise=true , address=145.79.227.41:4085 , piece_id=OGRSMMM4JHNTZZZURJHFOBUAOPGDC6T2PCHDEX7YL7SADHG6AN6A, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.768195260 (406h47m11.639278305s) [noise=true , address=145.79.227.29:4942 , piece_id=OMRNVS4BDDHYEFK5BEKTMKHTJUMJ4MY55FXPWOHZ3NVGHNST7KZA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.791646034 (406h47m11.66272908s) [noise=true , address=145.79.227.45:4231 , piece_id=I7F2TC2A6E35C3LPHT4D62ZOAZRSMF6XZZWVKJEI6SUHWTRQ6DFA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.807399862 (406h47m11.678482907s) [noise=true , address=145.79.227.46:4603 , piece_id=5KQGOFJZAEFI5OGUVUIK2HEBRMMUVGXEOD47OQIALAIMVCPEMLFA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.807424800 (406h47m11.678507845s) [seg_pos={0 0}, upload=357142]: Segment reached optimal threshold of 13 pieces. Dec 22 14:16:11.807688312 (406h47m11.678771357s) [noise=true , address=145.79.227.37:4566 , piece_id=4H2NM22RU4Y7H334JVNOTYO2FOUUXTKOCVYEHNJJ4HRVXVVXWOVA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F): protocol: expected piece hash; context canceled; EOF Dec 22 14:16:11.807698351 (406h47m11.678781396s) [noise=true , address=145.79.227.37:4566 , piece_id=4H2NM22RU4Y7H334JVNOTYO2FOUUXTKOCVYEHNJJ4HRVXVVXWOVA, seg_pos={0 0}, upload=357142]: Long tail cancellation due to optimal threshold Dec 22 14:16:11.807723459 (406h47m11.678806514s) [noise=true , address=145.79.227.18:4007 , piece_id=UMT4AQFVRTREIJDRAPCZTJY3URZYOHLX37QQTG5L53JFDW63ULQQ, seg_pos={0 0}, upload=357142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp): protocol: expected piece hash; context canceled; EOF Dec 22 14:16:11.807736343 (406h47m11.678819398s) [noise=true , address=145.79.227.18:4007 , piece_id=UMT4AQFVRTREIJDRAPCZTJY3URZYOHLX37QQTG5L53JFDW63ULQQ, seg_pos={0 0}, upload=357142]: Long tail cancellation due to optimal threshold Dec 22 14:16:11.807862974 (406h47m11.678946019s) [noise=true , address=45.91.116.8:4048 , piece_id=HQWNONHI5SJYLPULF2H75PVYLOPOPJTEC5ZPOZUT2HHLA6IC5ALA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 22 14:16:11.807871991 (406h47m11.678955026s) [noise=true , address=45.91.116.8:4048 , piece_id=HQWNONHI5SJYLPULF2H75PVYLOPOPJTEC5ZPOZUT2HHLA6IC5ALA, seg_pos={0 0}, upload=357142]: Long tail cancellation due to optimal threshold Dec 22 14:16:11.807896659 (406h47m11.678979704s) [seg_pos={0 0}, upload=357142]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 14:16:11.808114984 (406h47m11.679198019s) [upload=357142]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 14:16:11.808121817 (406h47m11.679204852s) [upload=357142]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 14:16:12.460702542 (406h47m12.331785577s) [upload=357142]: Done uploading segments... Dec 22 14:16:12.460721377 (406h47m12.331804412s) [upload=357142]: Upload finished. err: <nil> Duration: 6.025739855s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...........X..................X...X..................X......] duration: 5.920496924s trace: https://jaeger.dev.storj.io/trace/30697bf8e531908d FAIL (2025-12-22 14:19:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 22 14:16:07.103235978 (406h47m6.974319013s) [upload=357142]: Starting upload Dec 22 14:16:07.103295332 (406h47m6.974378377s) [upload=357142]: Done starting upload Dec 22 14:16:07.103399340 (406h47m6.974482375s) [upload=357142]: Uploading segments... Dec 22 14:16:07.103487287 (406h47m6.974570332s) [upload=357142]: Got next segment. Inline: false Dec 22 14:16:07.103515991 (406h47m6.974599026s) [upload=357142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 14:16:07.103520289 (406h47m6.974603325s) [upload=357142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 14:16:07.415289927 (406h47m7.286372962s) [seg_pos={0 0}, upload=357142]: Begin upload segment... Dec 22 14:16:07.415336014 (406h47m7.286419049s) [seg_pos={0 0}, upload=357142]: Done begin upload segment. Dec 22 14:16:07.415363607 (406h47m7.286446652s) [noise=true , address=145.79.227.17:4972 , piece_id=XOKLGQXCXZCOWYK5H4TPHRDHLLJXTII36XFZBX4XWYKHFE24WVOQ, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.415526457 (406h47m7.286609492s) [noise=true , address=145.79.227.18:4007 , piece_id=UMT4AQFVRTREIJDRAPCZTJY3URZYOHLX37QQTG5L53JFDW63ULQQ, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.416519017 (406h47m7.287602062s) [noise=true , address=145.79.227.48:4302 , piece_id=WYPWNL6UUKRVETGRAKJJMS5US3B4LQBJ6TRDYBUTRLXKGPFPPEEA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.417017777 (406h47m7.288100822s) [noise=true , address=145.79.227.50:4374 , piece_id=2XXNMOGMBDEQLIBGHJ6E25INEXYZR7JWYKA76WB5CKLK42IU5L4Q, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.417027014 (406h47m7.288110059s) [noise=true , address=145.79.227.43:4157 , piece_id=PLKGYWWJYEHSEMRCRFVQUTXTGYN6LYHFF4HPL2TOW6F7Q4K6QL7Q, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.417966864 (406h47m7.289049909s) [noise=true , address=45.91.116.8:4048 , piece_id=HQWNONHI5SJYLPULF2H75PVYLOPOPJTEC5ZPOZUT2HHLA6IC5ALA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.418744234 (406h47m7.289827269s) [noise=true , address=145.79.227.49:4339 , piece_id=ANLDLFM6BWPIJJIKHCY7CMWU2KIYTTNWA4ETR3DGNZG6GO7XM5QQ, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.419502227 (406h47m7.290585263s) [noise=true , address=77.90.31.10:4519 , piece_id=FMJHUEO445SYIHAV47SFERXWEFDSCCN36WPT4RJTXABWLJ6IO2VQ, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.419641743 (406h47m7.290724788s) [noise=true , address=145.79.227.10:4401 , piece_id=HVP7A7MFY7IM2S4P5QFS3T2BNYINUQRSF5TJ3OZLWH5REX2QCNJA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.420879160 (406h47m7.291962205s) [noise=true , address=145.79.227.29:4942 , piece_id=OMRNVS4BDDHYEFK5BEKTMKHTJUMJ4MY55FXPWOHZ3NVGHNST7KZA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.420914838 (406h47m7.291997883s) [noise=true , address=145.79.227.41:4085 , piece_id=OGRSMMM4JHNTZZZURJHFOBUAOPGDC6T2PCHDEX7YL7SADHG6AN6A, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.421340087 (406h47m7.292423132s) [noise=true , address=145.79.227.36:4517 , piece_id=DKVCILBC2JWXLG6G23ZMKEWQNDXYR4UNVD2XESJ57JIKIS5AXACA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.421402066 (406h47m7.292485101s) [noise=true , address=145.79.227.46:4603 , piece_id=5KQGOFJZAEFI5OGUVUIK2HEBRMMUVGXEOD47OQIALAIMVCPEMLFA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.422719935 (406h47m7.29380297s) [noise=true , address=145.79.227.45:4231 , piece_id=I7F2TC2A6E35C3LPHT4D62ZOAZRSMF6XZZWVKJEI6SUHWTRQ6DFA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.423220067 (406h47m7.294303112s) [noise=true , address=145.79.227.37:4566 , piece_id=4H2NM22RU4Y7H334JVNOTYO2FOUUXTKOCVYEHNJJ4HRVXVVXWOVA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.423702486 (406h47m7.294785531s) [noise=true , address=77.90.31.14:4663 , piece_id=X2NSIUKNVP2Y2MABYR3UPGZJSPF3CPS3EU6JYVNHX74RZTSGEQMA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:09.117304385 (406h47m8.988387421s) [upload=357142]: Next returned nil segment Dec 22 14:16:09.117321889 (406h47m8.988404924s) [upload=357142]: Waiting for error group managing segments... Dec 22 14:16:09.403723791 (406h47m9.274806826s) [noise=true , address=145.79.227.43:4157 , piece_id=PLKGYWWJYEHSEMRCRFVQUTXTGYN6LYHFF4HPL2TOW6F7Q4K6QL7Q, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:09.416402182 (406h47m9.287485247s) [noise=true , address=145.79.227.50:4374 , piece_id=2XXNMOGMBDEQLIBGHJ6E25INEXYZR7JWYKA76WB5CKLK42IU5L4Q, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:09.488678941 (406h47m9.359761976s) [noise=true , address=145.79.227.36:4517 , piece_id=DKVCILBC2JWXLG6G23ZMKEWQNDXYR4UNVD2XESJ57JIKIS5AXACA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:09.591529928 (406h47m9.462612963s) [noise=true , address=77.90.31.14:4663 , piece_id=X2NSIUKNVP2Y2MABYR3UPGZJSPF3CPS3EU6JYVNHX74RZTSGEQMA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:10.003825194 (406h47m9.874908229s) [noise=true , address=77.90.31.10:4519 , piece_id=FMJHUEO445SYIHAV47SFERXWEFDSCCN36WPT4RJTXABWLJ6IO2VQ, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.151289256 (406h47m11.022372301s) [noise=true , address=145.79.227.10:4401 , piece_id=HVP7A7MFY7IM2S4P5QFS3T2BNYINUQRSF5TJ3OZLWH5REX2QCNJA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.190801796 (406h47m11.061884841s) [noise=true , address=145.79.227.48:4302 , piece_id=WYPWNL6UUKRVETGRAKJJMS5US3B4LQBJ6TRDYBUTRLXKGPFPPEEA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.216210448 (406h47m11.087293483s) [noise=true , address=145.79.227.49:4339 , piece_id=ANLDLFM6BWPIJJIKHCY7CMWU2KIYTTNWA4ETR3DGNZG6GO7XM5QQ, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.447886943 (406h47m11.318969978s) [noise=true , address=145.79.227.17:4972 , piece_id=XOKLGQXCXZCOWYK5H4TPHRDHLLJXTII36XFZBX4XWYKHFE24WVOQ, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.513411914 (406h47m11.384494949s) [noise=true , address=145.79.227.41:4085 , piece_id=OGRSMMM4JHNTZZZURJHFOBUAOPGDC6T2PCHDEX7YL7SADHG6AN6A, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.768195260 (406h47m11.639278305s) [noise=true , address=145.79.227.29:4942 , piece_id=OMRNVS4BDDHYEFK5BEKTMKHTJUMJ4MY55FXPWOHZ3NVGHNST7KZA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.791646034 (406h47m11.66272908s) [noise=true , address=145.79.227.45:4231 , piece_id=I7F2TC2A6E35C3LPHT4D62ZOAZRSMF6XZZWVKJEI6SUHWTRQ6DFA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.807399862 (406h47m11.678482907s) [noise=true , address=145.79.227.46:4603 , piece_id=5KQGOFJZAEFI5OGUVUIK2HEBRMMUVGXEOD47OQIALAIMVCPEMLFA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.807424800 (406h47m11.678507845s) [seg_pos={0 0}, upload=357142]: Segment reached optimal threshold of 13 pieces. Dec 22 14:16:11.807688312 (406h47m11.678771357s) [noise=true , address=145.79.227.37:4566 , piece_id=4H2NM22RU4Y7H334JVNOTYO2FOUUXTKOCVYEHNJJ4HRVXVVXWOVA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F): protocol: expected piece hash; context canceled; EOF Dec 22 14:16:11.807698351 (406h47m11.678781396s) [noise=true , address=145.79.227.37:4566 , piece_id=4H2NM22RU4Y7H334JVNOTYO2FOUUXTKOCVYEHNJJ4HRVXVVXWOVA, seg_pos={0 0}, upload=357142]: Long tail cancellation due to optimal threshold Dec 22 14:16:11.807723459 (406h47m11.678806514s) [noise=true , address=145.79.227.18:4007 , piece_id=UMT4AQFVRTREIJDRAPCZTJY3URZYOHLX37QQTG5L53JFDW63ULQQ, seg_pos={0 0}, upload=357142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp): protocol: expected piece hash; context canceled; EOF Dec 22 14:16:11.807736343 (406h47m11.678819398s) [noise=true , address=145.79.227.18:4007 , piece_id=UMT4AQFVRTREIJDRAPCZTJY3URZYOHLX37QQTG5L53JFDW63ULQQ, seg_pos={0 0}, upload=357142]: Long tail cancellation due to optimal threshold Dec 22 14:16:11.807862974 (406h47m11.678946019s) [noise=true , address=45.91.116.8:4048 , piece_id=HQWNONHI5SJYLPULF2H75PVYLOPOPJTEC5ZPOZUT2HHLA6IC5ALA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 22 14:16:11.807871991 (406h47m11.678955026s) [noise=true , address=45.91.116.8:4048 , piece_id=HQWNONHI5SJYLPULF2H75PVYLOPOPJTEC5ZPOZUT2HHLA6IC5ALA, seg_pos={0 0}, upload=357142]: Long tail cancellation due to optimal threshold Dec 22 14:16:11.807896659 (406h47m11.678979704s) [seg_pos={0 0}, upload=357142]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 14:16:11.808114984 (406h47m11.679198019s) [upload=357142]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 14:16:11.808121817 (406h47m11.679204852s) [upload=357142]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 14:16:12.460702542 (406h47m12.331785577s) [upload=357142]: Done uploading segments... Dec 22 14:16:12.460721377 (406h47m12.331804412s) [upload=357142]: Upload finished. err: <nil> Duration: 5.920548232s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................................X.........X.] duration: 5.846547741s trace: https://jaeger.dev.storj.io/trace/7e72d81acd8b3600 FAIL (2025-12-22 14:19:26 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.846567489s core uplink eu1

Orbiter Check History

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)