Orbiter History

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................................X.........X.] duration: 5.846547741s trace: https://jaeger.dev.storj.io/trace/7e72d81acd8b3600 FAIL (2025-12-22 14:19:26 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.846567489s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [.............X....X.X....XX..X..............X.....X.X.......] duration: 3.553369919s trace: https://jaeger.dev.storj.io/trace/7fcb66a7d28c4377 FAIL (2025-12-22 14:19:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.553412631s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................................X.........X] duration: 1m13.045334302s trace: https://jaeger.dev.storj.io/trace/7e72d81acd8b3600 FAIL (2025-12-22 14:19:19 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1m13.045381351s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [............X..................X...X..................X.....] duration: 5.783733912s trace: https://jaeger.dev.storj.io/trace/30697bf8e531908d FAIL (2025-12-22 14:19:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 22 14:16:07.103235978 (406h47m6.974319013s) [upload=357142]: Starting upload Dec 22 14:16:07.103295332 (406h47m6.974378377s) [upload=357142]: Done starting upload Dec 22 14:16:07.103399340 (406h47m6.974482375s) [upload=357142]: Uploading segments... Dec 22 14:16:07.103487287 (406h47m6.974570332s) [upload=357142]: Got next segment. Inline: false Dec 22 14:16:07.103515991 (406h47m6.974599026s) [upload=357142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 14:16:07.103520289 (406h47m6.974603325s) [upload=357142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 14:16:07.415289927 (406h47m7.286372962s) [seg_pos={0 0}, upload=357142]: Begin upload segment... Dec 22 14:16:07.415336014 (406h47m7.286419049s) [seg_pos={0 0}, upload=357142]: Done begin upload segment. Dec 22 14:16:07.415363607 (406h47m7.286446652s) [noise=true , address=145.79.227.17:4972 , piece_id=XOKLGQXCXZCOWYK5H4TPHRDHLLJXTII36XFZBX4XWYKHFE24WVOQ, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.415526457 (406h47m7.286609492s) [noise=true , address=145.79.227.18:4007 , piece_id=UMT4AQFVRTREIJDRAPCZTJY3URZYOHLX37QQTG5L53JFDW63ULQQ, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.416519017 (406h47m7.287602062s) [noise=true , address=145.79.227.48:4302 , piece_id=WYPWNL6UUKRVETGRAKJJMS5US3B4LQBJ6TRDYBUTRLXKGPFPPEEA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.417017777 (406h47m7.288100822s) [noise=true , address=145.79.227.50:4374 , piece_id=2XXNMOGMBDEQLIBGHJ6E25INEXYZR7JWYKA76WB5CKLK42IU5L4Q, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.417027014 (406h47m7.288110059s) [noise=true , address=145.79.227.43:4157 , piece_id=PLKGYWWJYEHSEMRCRFVQUTXTGYN6LYHFF4HPL2TOW6F7Q4K6QL7Q, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.417966864 (406h47m7.289049909s) [noise=true , address=45.91.116.8:4048 , piece_id=HQWNONHI5SJYLPULF2H75PVYLOPOPJTEC5ZPOZUT2HHLA6IC5ALA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.418744234 (406h47m7.289827269s) [noise=true , address=145.79.227.49:4339 , piece_id=ANLDLFM6BWPIJJIKHCY7CMWU2KIYTTNWA4ETR3DGNZG6GO7XM5QQ, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.419502227 (406h47m7.290585263s) [noise=true , address=77.90.31.10:4519 , piece_id=FMJHUEO445SYIHAV47SFERXWEFDSCCN36WPT4RJTXABWLJ6IO2VQ, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.419641743 (406h47m7.290724788s) [noise=true , address=145.79.227.10:4401 , piece_id=HVP7A7MFY7IM2S4P5QFS3T2BNYINUQRSF5TJ3OZLWH5REX2QCNJA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.420879160 (406h47m7.291962205s) [noise=true , address=145.79.227.29:4942 , piece_id=OMRNVS4BDDHYEFK5BEKTMKHTJUMJ4MY55FXPWOHZ3NVGHNST7KZA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.420914838 (406h47m7.291997883s) [noise=true , address=145.79.227.41:4085 , piece_id=OGRSMMM4JHNTZZZURJHFOBUAOPGDC6T2PCHDEX7YL7SADHG6AN6A, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.421340087 (406h47m7.292423132s) [noise=true , address=145.79.227.36:4517 , piece_id=DKVCILBC2JWXLG6G23ZMKEWQNDXYR4UNVD2XESJ57JIKIS5AXACA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.421402066 (406h47m7.292485101s) [noise=true , address=145.79.227.46:4603 , piece_id=5KQGOFJZAEFI5OGUVUIK2HEBRMMUVGXEOD47OQIALAIMVCPEMLFA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.422719935 (406h47m7.29380297s) [noise=true , address=145.79.227.45:4231 , piece_id=I7F2TC2A6E35C3LPHT4D62ZOAZRSMF6XZZWVKJEI6SUHWTRQ6DFA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.423220067 (406h47m7.294303112s) [noise=true , address=145.79.227.37:4566 , piece_id=4H2NM22RU4Y7H334JVNOTYO2FOUUXTKOCVYEHNJJ4HRVXVVXWOVA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.423702486 (406h47m7.294785531s) [noise=true , address=77.90.31.14:4663 , piece_id=X2NSIUKNVP2Y2MABYR3UPGZJSPF3CPS3EU6JYVNHX74RZTSGEQMA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:09.117304385 (406h47m8.988387421s) [upload=357142]: Next returned nil segment Dec 22 14:16:09.117321889 (406h47m8.988404924s) [upload=357142]: Waiting for error group managing segments... Dec 22 14:16:09.403723791 (406h47m9.274806826s) [noise=true , address=145.79.227.43:4157 , piece_id=PLKGYWWJYEHSEMRCRFVQUTXTGYN6LYHFF4HPL2TOW6F7Q4K6QL7Q, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:09.416402182 (406h47m9.287485247s) [noise=true , address=145.79.227.50:4374 , piece_id=2XXNMOGMBDEQLIBGHJ6E25INEXYZR7JWYKA76WB5CKLK42IU5L4Q, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:09.488678941 (406h47m9.359761976s) [noise=true , address=145.79.227.36:4517 , piece_id=DKVCILBC2JWXLG6G23ZMKEWQNDXYR4UNVD2XESJ57JIKIS5AXACA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:09.591529928 (406h47m9.462612963s) [noise=true , address=77.90.31.14:4663 , piece_id=X2NSIUKNVP2Y2MABYR3UPGZJSPF3CPS3EU6JYVNHX74RZTSGEQMA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:10.003825194 (406h47m9.874908229s) [noise=true , address=77.90.31.10:4519 , piece_id=FMJHUEO445SYIHAV47SFERXWEFDSCCN36WPT4RJTXABWLJ6IO2VQ, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.151289256 (406h47m11.022372301s) [noise=true , address=145.79.227.10:4401 , piece_id=HVP7A7MFY7IM2S4P5QFS3T2BNYINUQRSF5TJ3OZLWH5REX2QCNJA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.190801796 (406h47m11.061884841s) [noise=true , address=145.79.227.48:4302 , piece_id=WYPWNL6UUKRVETGRAKJJMS5US3B4LQBJ6TRDYBUTRLXKGPFPPEEA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.216210448 (406h47m11.087293483s) [noise=true , address=145.79.227.49:4339 , piece_id=ANLDLFM6BWPIJJIKHCY7CMWU2KIYTTNWA4ETR3DGNZG6GO7XM5QQ, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.447886943 (406h47m11.318969978s) [noise=true , address=145.79.227.17:4972 , piece_id=XOKLGQXCXZCOWYK5H4TPHRDHLLJXTII36XFZBX4XWYKHFE24WVOQ, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.513411914 (406h47m11.384494949s) [noise=true , address=145.79.227.41:4085 , piece_id=OGRSMMM4JHNTZZZURJHFOBUAOPGDC6T2PCHDEX7YL7SADHG6AN6A, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.768195260 (406h47m11.639278305s) [noise=true , address=145.79.227.29:4942 , piece_id=OMRNVS4BDDHYEFK5BEKTMKHTJUMJ4MY55FXPWOHZ3NVGHNST7KZA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.791646034 (406h47m11.66272908s) [noise=true , address=145.79.227.45:4231 , piece_id=I7F2TC2A6E35C3LPHT4D62ZOAZRSMF6XZZWVKJEI6SUHWTRQ6DFA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.807399862 (406h47m11.678482907s) [noise=true , address=145.79.227.46:4603 , piece_id=5KQGOFJZAEFI5OGUVUIK2HEBRMMUVGXEOD47OQIALAIMVCPEMLFA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.807424800 (406h47m11.678507845s) [seg_pos={0 0}, upload=357142]: Segment reached optimal threshold of 13 pieces. Dec 22 14:16:11.807688312 (406h47m11.678771357s) [noise=true , address=145.79.227.37:4566 , piece_id=4H2NM22RU4Y7H334JVNOTYO2FOUUXTKOCVYEHNJJ4HRVXVVXWOVA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F): protocol: expected piece hash; context canceled; EOF Dec 22 14:16:11.807698351 (406h47m11.678781396s) [noise=true , address=145.79.227.37:4566 , piece_id=4H2NM22RU4Y7H334JVNOTYO2FOUUXTKOCVYEHNJJ4HRVXVVXWOVA, seg_pos={0 0}, upload=357142]: Long tail cancellation due to optimal threshold Dec 22 14:16:11.807723459 (406h47m11.678806514s) [noise=true , address=145.79.227.18:4007 , piece_id=UMT4AQFVRTREIJDRAPCZTJY3URZYOHLX37QQTG5L53JFDW63ULQQ, seg_pos={0 0}, upload=357142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp): protocol: expected piece hash; context canceled; EOF Dec 22 14:16:11.807736343 (406h47m11.678819398s) [noise=true , address=145.79.227.18:4007 , piece_id=UMT4AQFVRTREIJDRAPCZTJY3URZYOHLX37QQTG5L53JFDW63ULQQ, seg_pos={0 0}, upload=357142]: Long tail cancellation due to optimal threshold Dec 22 14:16:11.807862974 (406h47m11.678946019s) [noise=true , address=45.91.116.8:4048 , piece_id=HQWNONHI5SJYLPULF2H75PVYLOPOPJTEC5ZPOZUT2HHLA6IC5ALA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 22 14:16:11.807871991 (406h47m11.678955026s) [noise=true , address=45.91.116.8:4048 , piece_id=HQWNONHI5SJYLPULF2H75PVYLOPOPJTEC5ZPOZUT2HHLA6IC5ALA, seg_pos={0 0}, upload=357142]: Long tail cancellation due to optimal threshold Dec 22 14:16:11.807896659 (406h47m11.678979704s) [seg_pos={0 0}, upload=357142]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 14:16:11.808114984 (406h47m11.679198019s) [upload=357142]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 14:16:11.808121817 (406h47m11.679204852s) [upload=357142]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 14:16:12.460702542 (406h47m12.331785577s) [upload=357142]: Done uploading segments... Dec 22 14:16:12.460721377 (406h47m12.331804412s) [upload=357142]: Upload finished. err: <nil> Duration: 5.783766565s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 5.529593522s trace: https://jaeger.dev.storj.io/trace/1e7f77260788e649 FAIL (2025-12-22 14:18:59 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 5.529619503s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [X............X..................X...X..................X....] duration: 5.982714909s trace: https://jaeger.dev.storj.io/trace/30697bf8e531908d FAIL (2025-12-22 14:18:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 22 14:16:07.103235978 (406h47m6.974319013s) [upload=357142]: Starting upload Dec 22 14:16:07.103295332 (406h47m6.974378377s) [upload=357142]: Done starting upload Dec 22 14:16:07.103399340 (406h47m6.974482375s) [upload=357142]: Uploading segments... Dec 22 14:16:07.103487287 (406h47m6.974570332s) [upload=357142]: Got next segment. Inline: false Dec 22 14:16:07.103515991 (406h47m6.974599026s) [upload=357142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 14:16:07.103520289 (406h47m6.974603325s) [upload=357142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 14:16:07.415289927 (406h47m7.286372962s) [seg_pos={0 0}, upload=357142]: Begin upload segment... Dec 22 14:16:07.415336014 (406h47m7.286419049s) [seg_pos={0 0}, upload=357142]: Done begin upload segment. Dec 22 14:16:07.415363607 (406h47m7.286446652s) [noise=true , address=145.79.227.17:4972 , piece_id=XOKLGQXCXZCOWYK5H4TPHRDHLLJXTII36XFZBX4XWYKHFE24WVOQ, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.415526457 (406h47m7.286609492s) [noise=true , address=145.79.227.18:4007 , piece_id=UMT4AQFVRTREIJDRAPCZTJY3URZYOHLX37QQTG5L53JFDW63ULQQ, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.416519017 (406h47m7.287602062s) [noise=true , address=145.79.227.48:4302 , piece_id=WYPWNL6UUKRVETGRAKJJMS5US3B4LQBJ6TRDYBUTRLXKGPFPPEEA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.417017777 (406h47m7.288100822s) [noise=true , address=145.79.227.50:4374 , piece_id=2XXNMOGMBDEQLIBGHJ6E25INEXYZR7JWYKA76WB5CKLK42IU5L4Q, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.417027014 (406h47m7.288110059s) [noise=true , address=145.79.227.43:4157 , piece_id=PLKGYWWJYEHSEMRCRFVQUTXTGYN6LYHFF4HPL2TOW6F7Q4K6QL7Q, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.417966864 (406h47m7.289049909s) [noise=true , address=45.91.116.8:4048 , piece_id=HQWNONHI5SJYLPULF2H75PVYLOPOPJTEC5ZPOZUT2HHLA6IC5ALA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.418744234 (406h47m7.289827269s) [noise=true , address=145.79.227.49:4339 , piece_id=ANLDLFM6BWPIJJIKHCY7CMWU2KIYTTNWA4ETR3DGNZG6GO7XM5QQ, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.419502227 (406h47m7.290585263s) [noise=true , address=77.90.31.10:4519 , piece_id=FMJHUEO445SYIHAV47SFERXWEFDSCCN36WPT4RJTXABWLJ6IO2VQ, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.419641743 (406h47m7.290724788s) [noise=true , address=145.79.227.10:4401 , piece_id=HVP7A7MFY7IM2S4P5QFS3T2BNYINUQRSF5TJ3OZLWH5REX2QCNJA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.420879160 (406h47m7.291962205s) [noise=true , address=145.79.227.29:4942 , piece_id=OMRNVS4BDDHYEFK5BEKTMKHTJUMJ4MY55FXPWOHZ3NVGHNST7KZA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.420914838 (406h47m7.291997883s) [noise=true , address=145.79.227.41:4085 , piece_id=OGRSMMM4JHNTZZZURJHFOBUAOPGDC6T2PCHDEX7YL7SADHG6AN6A, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.421340087 (406h47m7.292423132s) [noise=true , address=145.79.227.36:4517 , piece_id=DKVCILBC2JWXLG6G23ZMKEWQNDXYR4UNVD2XESJ57JIKIS5AXACA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.421402066 (406h47m7.292485101s) [noise=true , address=145.79.227.46:4603 , piece_id=5KQGOFJZAEFI5OGUVUIK2HEBRMMUVGXEOD47OQIALAIMVCPEMLFA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.422719935 (406h47m7.29380297s) [noise=true , address=145.79.227.45:4231 , piece_id=I7F2TC2A6E35C3LPHT4D62ZOAZRSMF6XZZWVKJEI6SUHWTRQ6DFA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.423220067 (406h47m7.294303112s) [noise=true , address=145.79.227.37:4566 , piece_id=4H2NM22RU4Y7H334JVNOTYO2FOUUXTKOCVYEHNJJ4HRVXVVXWOVA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.423702486 (406h47m7.294785531s) [noise=true , address=77.90.31.14:4663 , piece_id=X2NSIUKNVP2Y2MABYR3UPGZJSPF3CPS3EU6JYVNHX74RZTSGEQMA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:09.117304385 (406h47m8.988387421s) [upload=357142]: Next returned nil segment Dec 22 14:16:09.117321889 (406h47m8.988404924s) [upload=357142]: Waiting for error group managing segments... Dec 22 14:16:09.403723791 (406h47m9.274806826s) [noise=true , address=145.79.227.43:4157 , piece_id=PLKGYWWJYEHSEMRCRFVQUTXTGYN6LYHFF4HPL2TOW6F7Q4K6QL7Q, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:09.416402182 (406h47m9.287485247s) [noise=true , address=145.79.227.50:4374 , piece_id=2XXNMOGMBDEQLIBGHJ6E25INEXYZR7JWYKA76WB5CKLK42IU5L4Q, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:09.488678941 (406h47m9.359761976s) [noise=true , address=145.79.227.36:4517 , piece_id=DKVCILBC2JWXLG6G23ZMKEWQNDXYR4UNVD2XESJ57JIKIS5AXACA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:09.591529928 (406h47m9.462612963s) [noise=true , address=77.90.31.14:4663 , piece_id=X2NSIUKNVP2Y2MABYR3UPGZJSPF3CPS3EU6JYVNHX74RZTSGEQMA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:10.003825194 (406h47m9.874908229s) [noise=true , address=77.90.31.10:4519 , piece_id=FMJHUEO445SYIHAV47SFERXWEFDSCCN36WPT4RJTXABWLJ6IO2VQ, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.151289256 (406h47m11.022372301s) [noise=true , address=145.79.227.10:4401 , piece_id=HVP7A7MFY7IM2S4P5QFS3T2BNYINUQRSF5TJ3OZLWH5REX2QCNJA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.190801796 (406h47m11.061884841s) [noise=true , address=145.79.227.48:4302 , piece_id=WYPWNL6UUKRVETGRAKJJMS5US3B4LQBJ6TRDYBUTRLXKGPFPPEEA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.216210448 (406h47m11.087293483s) [noise=true , address=145.79.227.49:4339 , piece_id=ANLDLFM6BWPIJJIKHCY7CMWU2KIYTTNWA4ETR3DGNZG6GO7XM5QQ, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.447886943 (406h47m11.318969978s) [noise=true , address=145.79.227.17:4972 , piece_id=XOKLGQXCXZCOWYK5H4TPHRDHLLJXTII36XFZBX4XWYKHFE24WVOQ, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.513411914 (406h47m11.384494949s) [noise=true , address=145.79.227.41:4085 , piece_id=OGRSMMM4JHNTZZZURJHFOBUAOPGDC6T2PCHDEX7YL7SADHG6AN6A, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.768195260 (406h47m11.639278305s) [noise=true , address=145.79.227.29:4942 , piece_id=OMRNVS4BDDHYEFK5BEKTMKHTJUMJ4MY55FXPWOHZ3NVGHNST7KZA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.791646034 (406h47m11.66272908s) [noise=true , address=145.79.227.45:4231 , piece_id=I7F2TC2A6E35C3LPHT4D62ZOAZRSMF6XZZWVKJEI6SUHWTRQ6DFA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.807399862 (406h47m11.678482907s) [noise=true , address=145.79.227.46:4603 , piece_id=5KQGOFJZAEFI5OGUVUIK2HEBRMMUVGXEOD47OQIALAIMVCPEMLFA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.807424800 (406h47m11.678507845s) [seg_pos={0 0}, upload=357142]: Segment reached optimal threshold of 13 pieces. Dec 22 14:16:11.807688312 (406h47m11.678771357s) [noise=true , address=145.79.227.37:4566 , piece_id=4H2NM22RU4Y7H334JVNOTYO2FOUUXTKOCVYEHNJJ4HRVXVVXWOVA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F): protocol: expected piece hash; context canceled; EOF Dec 22 14:16:11.807698351 (406h47m11.678781396s) [noise=true , address=145.79.227.37:4566 , piece_id=4H2NM22RU4Y7H334JVNOTYO2FOUUXTKOCVYEHNJJ4HRVXVVXWOVA, seg_pos={0 0}, upload=357142]: Long tail cancellation due to optimal threshold Dec 22 14:16:11.807723459 (406h47m11.678806514s) [noise=true , address=145.79.227.18:4007 , piece_id=UMT4AQFVRTREIJDRAPCZTJY3URZYOHLX37QQTG5L53JFDW63ULQQ, seg_pos={0 0}, upload=357142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp): protocol: expected piece hash; context canceled; EOF Dec 22 14:16:11.807736343 (406h47m11.678819398s) [noise=true , address=145.79.227.18:4007 , piece_id=UMT4AQFVRTREIJDRAPCZTJY3URZYOHLX37QQTG5L53JFDW63ULQQ, seg_pos={0 0}, upload=357142]: Long tail cancellation due to optimal threshold Dec 22 14:16:11.807862974 (406h47m11.678946019s) [noise=true , address=45.91.116.8:4048 , piece_id=HQWNONHI5SJYLPULF2H75PVYLOPOPJTEC5ZPOZUT2HHLA6IC5ALA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 22 14:16:11.807871991 (406h47m11.678955026s) [noise=true , address=45.91.116.8:4048 , piece_id=HQWNONHI5SJYLPULF2H75PVYLOPOPJTEC5ZPOZUT2HHLA6IC5ALA, seg_pos={0 0}, upload=357142]: Long tail cancellation due to optimal threshold Dec 22 14:16:11.807896659 (406h47m11.678979704s) [seg_pos={0 0}, upload=357142]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 14:16:11.808114984 (406h47m11.679198019s) [upload=357142]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 14:16:11.808121817 (406h47m11.679204852s) [upload=357142]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 14:16:12.460702542 (406h47m12.331785577s) [upload=357142]: Done uploading segments... Dec 22 14:16:12.460721377 (406h47m12.331804412s) [upload=357142]: Upload finished. err: <nil> Duration: 5.982737532s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [..............X....X.X....XX..X..............X.....X.X......] duration: 3.775421517s trace: https://jaeger.dev.storj.io/trace/7fcb66a7d28c4377 FAIL (2025-12-22 14:18:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.775452405s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.X............X..................X...X..................X...] duration: 6.1317502s trace: https://jaeger.dev.storj.io/trace/30697bf8e531908d FAIL (2025-12-22 14:17:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 22 14:16:07.103235978 (406h47m6.974319013s) [upload=357142]: Starting upload Dec 22 14:16:07.103295332 (406h47m6.974378377s) [upload=357142]: Done starting upload Dec 22 14:16:07.103399340 (406h47m6.974482375s) [upload=357142]: Uploading segments... Dec 22 14:16:07.103487287 (406h47m6.974570332s) [upload=357142]: Got next segment. Inline: false Dec 22 14:16:07.103515991 (406h47m6.974599026s) [upload=357142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 14:16:07.103520289 (406h47m6.974603325s) [upload=357142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 14:16:07.415289927 (406h47m7.286372962s) [seg_pos={0 0}, upload=357142]: Begin upload segment... Dec 22 14:16:07.415336014 (406h47m7.286419049s) [seg_pos={0 0}, upload=357142]: Done begin upload segment. Dec 22 14:16:07.415363607 (406h47m7.286446652s) [noise=true , address=145.79.227.17:4972 , piece_id=XOKLGQXCXZCOWYK5H4TPHRDHLLJXTII36XFZBX4XWYKHFE24WVOQ, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.415526457 (406h47m7.286609492s) [noise=true , address=145.79.227.18:4007 , piece_id=UMT4AQFVRTREIJDRAPCZTJY3URZYOHLX37QQTG5L53JFDW63ULQQ, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.416519017 (406h47m7.287602062s) [noise=true , address=145.79.227.48:4302 , piece_id=WYPWNL6UUKRVETGRAKJJMS5US3B4LQBJ6TRDYBUTRLXKGPFPPEEA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.417017777 (406h47m7.288100822s) [noise=true , address=145.79.227.50:4374 , piece_id=2XXNMOGMBDEQLIBGHJ6E25INEXYZR7JWYKA76WB5CKLK42IU5L4Q, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.417027014 (406h47m7.288110059s) [noise=true , address=145.79.227.43:4157 , piece_id=PLKGYWWJYEHSEMRCRFVQUTXTGYN6LYHFF4HPL2TOW6F7Q4K6QL7Q, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.417966864 (406h47m7.289049909s) [noise=true , address=45.91.116.8:4048 , piece_id=HQWNONHI5SJYLPULF2H75PVYLOPOPJTEC5ZPOZUT2HHLA6IC5ALA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.418744234 (406h47m7.289827269s) [noise=true , address=145.79.227.49:4339 , piece_id=ANLDLFM6BWPIJJIKHCY7CMWU2KIYTTNWA4ETR3DGNZG6GO7XM5QQ, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.419502227 (406h47m7.290585263s) [noise=true , address=77.90.31.10:4519 , piece_id=FMJHUEO445SYIHAV47SFERXWEFDSCCN36WPT4RJTXABWLJ6IO2VQ, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.419641743 (406h47m7.290724788s) [noise=true , address=145.79.227.10:4401 , piece_id=HVP7A7MFY7IM2S4P5QFS3T2BNYINUQRSF5TJ3OZLWH5REX2QCNJA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.420879160 (406h47m7.291962205s) [noise=true , address=145.79.227.29:4942 , piece_id=OMRNVS4BDDHYEFK5BEKTMKHTJUMJ4MY55FXPWOHZ3NVGHNST7KZA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.420914838 (406h47m7.291997883s) [noise=true , address=145.79.227.41:4085 , piece_id=OGRSMMM4JHNTZZZURJHFOBUAOPGDC6T2PCHDEX7YL7SADHG6AN6A, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.421340087 (406h47m7.292423132s) [noise=true , address=145.79.227.36:4517 , piece_id=DKVCILBC2JWXLG6G23ZMKEWQNDXYR4UNVD2XESJ57JIKIS5AXACA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.421402066 (406h47m7.292485101s) [noise=true , address=145.79.227.46:4603 , piece_id=5KQGOFJZAEFI5OGUVUIK2HEBRMMUVGXEOD47OQIALAIMVCPEMLFA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.422719935 (406h47m7.29380297s) [noise=true , address=145.79.227.45:4231 , piece_id=I7F2TC2A6E35C3LPHT4D62ZOAZRSMF6XZZWVKJEI6SUHWTRQ6DFA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.423220067 (406h47m7.294303112s) [noise=true , address=145.79.227.37:4566 , piece_id=4H2NM22RU4Y7H334JVNOTYO2FOUUXTKOCVYEHNJJ4HRVXVVXWOVA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.423702486 (406h47m7.294785531s) [noise=true , address=77.90.31.14:4663 , piece_id=X2NSIUKNVP2Y2MABYR3UPGZJSPF3CPS3EU6JYVNHX74RZTSGEQMA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:09.117304385 (406h47m8.988387421s) [upload=357142]: Next returned nil segment Dec 22 14:16:09.117321889 (406h47m8.988404924s) [upload=357142]: Waiting for error group managing segments... Dec 22 14:16:09.403723791 (406h47m9.274806826s) [noise=true , address=145.79.227.43:4157 , piece_id=PLKGYWWJYEHSEMRCRFVQUTXTGYN6LYHFF4HPL2TOW6F7Q4K6QL7Q, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:09.416402182 (406h47m9.287485247s) [noise=true , address=145.79.227.50:4374 , piece_id=2XXNMOGMBDEQLIBGHJ6E25INEXYZR7JWYKA76WB5CKLK42IU5L4Q, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:09.488678941 (406h47m9.359761976s) [noise=true , address=145.79.227.36:4517 , piece_id=DKVCILBC2JWXLG6G23ZMKEWQNDXYR4UNVD2XESJ57JIKIS5AXACA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:09.591529928 (406h47m9.462612963s) [noise=true , address=77.90.31.14:4663 , piece_id=X2NSIUKNVP2Y2MABYR3UPGZJSPF3CPS3EU6JYVNHX74RZTSGEQMA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:10.003825194 (406h47m9.874908229s) [noise=true , address=77.90.31.10:4519 , piece_id=FMJHUEO445SYIHAV47SFERXWEFDSCCN36WPT4RJTXABWLJ6IO2VQ, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.151289256 (406h47m11.022372301s) [noise=true , address=145.79.227.10:4401 , piece_id=HVP7A7MFY7IM2S4P5QFS3T2BNYINUQRSF5TJ3OZLWH5REX2QCNJA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.190801796 (406h47m11.061884841s) [noise=true , address=145.79.227.48:4302 , piece_id=WYPWNL6UUKRVETGRAKJJMS5US3B4LQBJ6TRDYBUTRLXKGPFPPEEA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.216210448 (406h47m11.087293483s) [noise=true , address=145.79.227.49:4339 , piece_id=ANLDLFM6BWPIJJIKHCY7CMWU2KIYTTNWA4ETR3DGNZG6GO7XM5QQ, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.447886943 (406h47m11.318969978s) [noise=true , address=145.79.227.17:4972 , piece_id=XOKLGQXCXZCOWYK5H4TPHRDHLLJXTII36XFZBX4XWYKHFE24WVOQ, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.513411914 (406h47m11.384494949s) [noise=true , address=145.79.227.41:4085 , piece_id=OGRSMMM4JHNTZZZURJHFOBUAOPGDC6T2PCHDEX7YL7SADHG6AN6A, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.768195260 (406h47m11.639278305s) [noise=true , address=145.79.227.29:4942 , piece_id=OMRNVS4BDDHYEFK5BEKTMKHTJUMJ4MY55FXPWOHZ3NVGHNST7KZA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.791646034 (406h47m11.66272908s) [noise=true , address=145.79.227.45:4231 , piece_id=I7F2TC2A6E35C3LPHT4D62ZOAZRSMF6XZZWVKJEI6SUHWTRQ6DFA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.807399862 (406h47m11.678482907s) [noise=true , address=145.79.227.46:4603 , piece_id=5KQGOFJZAEFI5OGUVUIK2HEBRMMUVGXEOD47OQIALAIMVCPEMLFA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.807424800 (406h47m11.678507845s) [seg_pos={0 0}, upload=357142]: Segment reached optimal threshold of 13 pieces. Dec 22 14:16:11.807688312 (406h47m11.678771357s) [noise=true , address=145.79.227.37:4566 , piece_id=4H2NM22RU4Y7H334JVNOTYO2FOUUXTKOCVYEHNJJ4HRVXVVXWOVA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F): protocol: expected piece hash; context canceled; EOF Dec 22 14:16:11.807698351 (406h47m11.678781396s) [noise=true , address=145.79.227.37:4566 , piece_id=4H2NM22RU4Y7H334JVNOTYO2FOUUXTKOCVYEHNJJ4HRVXVVXWOVA, seg_pos={0 0}, upload=357142]: Long tail cancellation due to optimal threshold Dec 22 14:16:11.807723459 (406h47m11.678806514s) [noise=true , address=145.79.227.18:4007 , piece_id=UMT4AQFVRTREIJDRAPCZTJY3URZYOHLX37QQTG5L53JFDW63ULQQ, seg_pos={0 0}, upload=357142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp): protocol: expected piece hash; context canceled; EOF Dec 22 14:16:11.807736343 (406h47m11.678819398s) [noise=true , address=145.79.227.18:4007 , piece_id=UMT4AQFVRTREIJDRAPCZTJY3URZYOHLX37QQTG5L53JFDW63ULQQ, seg_pos={0 0}, upload=357142]: Long tail cancellation due to optimal threshold Dec 22 14:16:11.807862974 (406h47m11.678946019s) [noise=true , address=45.91.116.8:4048 , piece_id=HQWNONHI5SJYLPULF2H75PVYLOPOPJTEC5ZPOZUT2HHLA6IC5ALA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 22 14:16:11.807871991 (406h47m11.678955026s) [noise=true , address=45.91.116.8:4048 , piece_id=HQWNONHI5SJYLPULF2H75PVYLOPOPJTEC5ZPOZUT2HHLA6IC5ALA, seg_pos={0 0}, upload=357142]: Long tail cancellation due to optimal threshold Dec 22 14:16:11.807896659 (406h47m11.678979704s) [seg_pos={0 0}, upload=357142]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 14:16:11.808114984 (406h47m11.679198019s) [upload=357142]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 14:16:11.808121817 (406h47m11.679204852s) [upload=357142]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 14:16:12.460702542 (406h47m12.331785577s) [upload=357142]: Done uploading segments... Dec 22 14:16:12.460721377 (406h47m12.331804412s) [upload=357142]: Upload finished. err: <nil> Duration: 6.131778103s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 1.497863889s trace: https://jaeger.dev.storj.io/trace/1e7f77260788e649 FAIL (2025-12-22 14:17:40 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.497893505s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [...............X....X.X....XX..X..............X.....X.X.....] duration: 3.59270386s trace: https://jaeger.dev.storj.io/trace/7fcb66a7d28c4377 FAIL (2025-12-22 14:17:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.592774314s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 5.222078048s trace: https://jaeger.dev.storj.io/trace/538cdd5d04469e38 FAIL (2025-12-22 14:17:03 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.22211087s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [................X....X.X....XX..X..............X.....X.X....] duration: 4.339172131s trace: https://jaeger.dev.storj.io/trace/7fcb66a7d28c4377 FAIL (2025-12-22 14:17:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.339199863s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..X............X..................X...X..................X..] duration: 6.068957682s trace: https://jaeger.dev.storj.io/trace/30697bf8e531908d FAIL (2025-12-22 14:16:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 22 14:16:07.103235978 (406h47m6.974319013s) [upload=357142]: Starting upload Dec 22 14:16:07.103295332 (406h47m6.974378377s) [upload=357142]: Done starting upload Dec 22 14:16:07.103399340 (406h47m6.974482375s) [upload=357142]: Uploading segments... Dec 22 14:16:07.103487287 (406h47m6.974570332s) [upload=357142]: Got next segment. Inline: false Dec 22 14:16:07.103515991 (406h47m6.974599026s) [upload=357142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 14:16:07.103520289 (406h47m6.974603325s) [upload=357142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 14:16:07.415289927 (406h47m7.286372962s) [seg_pos={0 0}, upload=357142]: Begin upload segment... Dec 22 14:16:07.415336014 (406h47m7.286419049s) [seg_pos={0 0}, upload=357142]: Done begin upload segment. Dec 22 14:16:07.415363607 (406h47m7.286446652s) [noise=true , address=145.79.227.17:4972 , piece_id=XOKLGQXCXZCOWYK5H4TPHRDHLLJXTII36XFZBX4XWYKHFE24WVOQ, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.415526457 (406h47m7.286609492s) [noise=true , address=145.79.227.18:4007 , piece_id=UMT4AQFVRTREIJDRAPCZTJY3URZYOHLX37QQTG5L53JFDW63ULQQ, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.416519017 (406h47m7.287602062s) [noise=true , address=145.79.227.48:4302 , piece_id=WYPWNL6UUKRVETGRAKJJMS5US3B4LQBJ6TRDYBUTRLXKGPFPPEEA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.417017777 (406h47m7.288100822s) [noise=true , address=145.79.227.50:4374 , piece_id=2XXNMOGMBDEQLIBGHJ6E25INEXYZR7JWYKA76WB5CKLK42IU5L4Q, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.417027014 (406h47m7.288110059s) [noise=true , address=145.79.227.43:4157 , piece_id=PLKGYWWJYEHSEMRCRFVQUTXTGYN6LYHFF4HPL2TOW6F7Q4K6QL7Q, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.417966864 (406h47m7.289049909s) [noise=true , address=45.91.116.8:4048 , piece_id=HQWNONHI5SJYLPULF2H75PVYLOPOPJTEC5ZPOZUT2HHLA6IC5ALA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.418744234 (406h47m7.289827269s) [noise=true , address=145.79.227.49:4339 , piece_id=ANLDLFM6BWPIJJIKHCY7CMWU2KIYTTNWA4ETR3DGNZG6GO7XM5QQ, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.419502227 (406h47m7.290585263s) [noise=true , address=77.90.31.10:4519 , piece_id=FMJHUEO445SYIHAV47SFERXWEFDSCCN36WPT4RJTXABWLJ6IO2VQ, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.419641743 (406h47m7.290724788s) [noise=true , address=145.79.227.10:4401 , piece_id=HVP7A7MFY7IM2S4P5QFS3T2BNYINUQRSF5TJ3OZLWH5REX2QCNJA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.420879160 (406h47m7.291962205s) [noise=true , address=145.79.227.29:4942 , piece_id=OMRNVS4BDDHYEFK5BEKTMKHTJUMJ4MY55FXPWOHZ3NVGHNST7KZA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.420914838 (406h47m7.291997883s) [noise=true , address=145.79.227.41:4085 , piece_id=OGRSMMM4JHNTZZZURJHFOBUAOPGDC6T2PCHDEX7YL7SADHG6AN6A, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.421340087 (406h47m7.292423132s) [noise=true , address=145.79.227.36:4517 , piece_id=DKVCILBC2JWXLG6G23ZMKEWQNDXYR4UNVD2XESJ57JIKIS5AXACA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.421402066 (406h47m7.292485101s) [noise=true , address=145.79.227.46:4603 , piece_id=5KQGOFJZAEFI5OGUVUIK2HEBRMMUVGXEOD47OQIALAIMVCPEMLFA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.422719935 (406h47m7.29380297s) [noise=true , address=145.79.227.45:4231 , piece_id=I7F2TC2A6E35C3LPHT4D62ZOAZRSMF6XZZWVKJEI6SUHWTRQ6DFA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.423220067 (406h47m7.294303112s) [noise=true , address=145.79.227.37:4566 , piece_id=4H2NM22RU4Y7H334JVNOTYO2FOUUXTKOCVYEHNJJ4HRVXVVXWOVA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.423702486 (406h47m7.294785531s) [noise=true , address=77.90.31.14:4663 , piece_id=X2NSIUKNVP2Y2MABYR3UPGZJSPF3CPS3EU6JYVNHX74RZTSGEQMA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:09.117304385 (406h47m8.988387421s) [upload=357142]: Next returned nil segment Dec 22 14:16:09.117321889 (406h47m8.988404924s) [upload=357142]: Waiting for error group managing segments... Dec 22 14:16:09.403723791 (406h47m9.274806826s) [noise=true , address=145.79.227.43:4157 , piece_id=PLKGYWWJYEHSEMRCRFVQUTXTGYN6LYHFF4HPL2TOW6F7Q4K6QL7Q, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:09.416402182 (406h47m9.287485247s) [noise=true , address=145.79.227.50:4374 , piece_id=2XXNMOGMBDEQLIBGHJ6E25INEXYZR7JWYKA76WB5CKLK42IU5L4Q, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:09.488678941 (406h47m9.359761976s) [noise=true , address=145.79.227.36:4517 , piece_id=DKVCILBC2JWXLG6G23ZMKEWQNDXYR4UNVD2XESJ57JIKIS5AXACA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:09.591529928 (406h47m9.462612963s) [noise=true , address=77.90.31.14:4663 , piece_id=X2NSIUKNVP2Y2MABYR3UPGZJSPF3CPS3EU6JYVNHX74RZTSGEQMA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:10.003825194 (406h47m9.874908229s) [noise=true , address=77.90.31.10:4519 , piece_id=FMJHUEO445SYIHAV47SFERXWEFDSCCN36WPT4RJTXABWLJ6IO2VQ, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.151289256 (406h47m11.022372301s) [noise=true , address=145.79.227.10:4401 , piece_id=HVP7A7MFY7IM2S4P5QFS3T2BNYINUQRSF5TJ3OZLWH5REX2QCNJA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.190801796 (406h47m11.061884841s) [noise=true , address=145.79.227.48:4302 , piece_id=WYPWNL6UUKRVETGRAKJJMS5US3B4LQBJ6TRDYBUTRLXKGPFPPEEA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.216210448 (406h47m11.087293483s) [noise=true , address=145.79.227.49:4339 , piece_id=ANLDLFM6BWPIJJIKHCY7CMWU2KIYTTNWA4ETR3DGNZG6GO7XM5QQ, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.447886943 (406h47m11.318969978s) [noise=true , address=145.79.227.17:4972 , piece_id=XOKLGQXCXZCOWYK5H4TPHRDHLLJXTII36XFZBX4XWYKHFE24WVOQ, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.513411914 (406h47m11.384494949s) [noise=true , address=145.79.227.41:4085 , piece_id=OGRSMMM4JHNTZZZURJHFOBUAOPGDC6T2PCHDEX7YL7SADHG6AN6A, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.768195260 (406h47m11.639278305s) [noise=true , address=145.79.227.29:4942 , piece_id=OMRNVS4BDDHYEFK5BEKTMKHTJUMJ4MY55FXPWOHZ3NVGHNST7KZA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.791646034 (406h47m11.66272908s) [noise=true , address=145.79.227.45:4231 , piece_id=I7F2TC2A6E35C3LPHT4D62ZOAZRSMF6XZZWVKJEI6SUHWTRQ6DFA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.807399862 (406h47m11.678482907s) [noise=true , address=145.79.227.46:4603 , piece_id=5KQGOFJZAEFI5OGUVUIK2HEBRMMUVGXEOD47OQIALAIMVCPEMLFA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.807424800 (406h47m11.678507845s) [seg_pos={0 0}, upload=357142]: Segment reached optimal threshold of 13 pieces. Dec 22 14:16:11.807688312 (406h47m11.678771357s) [noise=true , address=145.79.227.37:4566 , piece_id=4H2NM22RU4Y7H334JVNOTYO2FOUUXTKOCVYEHNJJ4HRVXVVXWOVA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F): protocol: expected piece hash; context canceled; EOF Dec 22 14:16:11.807698351 (406h47m11.678781396s) [noise=true , address=145.79.227.37:4566 , piece_id=4H2NM22RU4Y7H334JVNOTYO2FOUUXTKOCVYEHNJJ4HRVXVVXWOVA, seg_pos={0 0}, upload=357142]: Long tail cancellation due to optimal threshold Dec 22 14:16:11.807723459 (406h47m11.678806514s) [noise=true , address=145.79.227.18:4007 , piece_id=UMT4AQFVRTREIJDRAPCZTJY3URZYOHLX37QQTG5L53JFDW63ULQQ, seg_pos={0 0}, upload=357142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp): protocol: expected piece hash; context canceled; EOF Dec 22 14:16:11.807736343 (406h47m11.678819398s) [noise=true , address=145.79.227.18:4007 , piece_id=UMT4AQFVRTREIJDRAPCZTJY3URZYOHLX37QQTG5L53JFDW63ULQQ, seg_pos={0 0}, upload=357142]: Long tail cancellation due to optimal threshold Dec 22 14:16:11.807862974 (406h47m11.678946019s) [noise=true , address=45.91.116.8:4048 , piece_id=HQWNONHI5SJYLPULF2H75PVYLOPOPJTEC5ZPOZUT2HHLA6IC5ALA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 22 14:16:11.807871991 (406h47m11.678955026s) [noise=true , address=45.91.116.8:4048 , piece_id=HQWNONHI5SJYLPULF2H75PVYLOPOPJTEC5ZPOZUT2HHLA6IC5ALA, seg_pos={0 0}, upload=357142]: Long tail cancellation due to optimal threshold Dec 22 14:16:11.807896659 (406h47m11.678979704s) [seg_pos={0 0}, upload=357142]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 14:16:11.808114984 (406h47m11.679198019s) [upload=357142]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 14:16:11.808121817 (406h47m11.679204852s) [upload=357142]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 14:16:12.460702542 (406h47m12.331785577s) [upload=357142]: Done uploading segments... Dec 22 14:16:12.460721377 (406h47m12.331804412s) [upload=357142]: Upload finished. err: <nil> Duration: 6.068986037s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [.................X....X.X....XX..X..............X.....X.X...] duration: 3.692827473s trace: https://jaeger.dev.storj.io/trace/7fcb66a7d28c4377 FAIL (2025-12-22 14:16:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.693202457s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...X............X..................X...X..................X.] duration: 8.356616765s trace: https://jaeger.dev.storj.io/trace/30697bf8e531908d FAIL (2025-12-22 14:16:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 22 14:16:07.103235978 (406h47m6.974319013s) [upload=357142]: Starting upload Dec 22 14:16:07.103295332 (406h47m6.974378377s) [upload=357142]: Done starting upload Dec 22 14:16:07.103399340 (406h47m6.974482375s) [upload=357142]: Uploading segments... Dec 22 14:16:07.103487287 (406h47m6.974570332s) [upload=357142]: Got next segment. Inline: false Dec 22 14:16:07.103515991 (406h47m6.974599026s) [upload=357142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 14:16:07.103520289 (406h47m6.974603325s) [upload=357142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 14:16:07.415289927 (406h47m7.286372962s) [seg_pos={0 0}, upload=357142]: Begin upload segment... Dec 22 14:16:07.415336014 (406h47m7.286419049s) [seg_pos={0 0}, upload=357142]: Done begin upload segment. Dec 22 14:16:07.415363607 (406h47m7.286446652s) [noise=true , address=145.79.227.17:4972 , piece_id=XOKLGQXCXZCOWYK5H4TPHRDHLLJXTII36XFZBX4XWYKHFE24WVOQ, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.415526457 (406h47m7.286609492s) [noise=true , address=145.79.227.18:4007 , piece_id=UMT4AQFVRTREIJDRAPCZTJY3URZYOHLX37QQTG5L53JFDW63ULQQ, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.416519017 (406h47m7.287602062s) [noise=true , address=145.79.227.48:4302 , piece_id=WYPWNL6UUKRVETGRAKJJMS5US3B4LQBJ6TRDYBUTRLXKGPFPPEEA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.417017777 (406h47m7.288100822s) [noise=true , address=145.79.227.50:4374 , piece_id=2XXNMOGMBDEQLIBGHJ6E25INEXYZR7JWYKA76WB5CKLK42IU5L4Q, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.417027014 (406h47m7.288110059s) [noise=true , address=145.79.227.43:4157 , piece_id=PLKGYWWJYEHSEMRCRFVQUTXTGYN6LYHFF4HPL2TOW6F7Q4K6QL7Q, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.417966864 (406h47m7.289049909s) [noise=true , address=45.91.116.8:4048 , piece_id=HQWNONHI5SJYLPULF2H75PVYLOPOPJTEC5ZPOZUT2HHLA6IC5ALA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.418744234 (406h47m7.289827269s) [noise=true , address=145.79.227.49:4339 , piece_id=ANLDLFM6BWPIJJIKHCY7CMWU2KIYTTNWA4ETR3DGNZG6GO7XM5QQ, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.419502227 (406h47m7.290585263s) [noise=true , address=77.90.31.10:4519 , piece_id=FMJHUEO445SYIHAV47SFERXWEFDSCCN36WPT4RJTXABWLJ6IO2VQ, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.419641743 (406h47m7.290724788s) [noise=true , address=145.79.227.10:4401 , piece_id=HVP7A7MFY7IM2S4P5QFS3T2BNYINUQRSF5TJ3OZLWH5REX2QCNJA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.420879160 (406h47m7.291962205s) [noise=true , address=145.79.227.29:4942 , piece_id=OMRNVS4BDDHYEFK5BEKTMKHTJUMJ4MY55FXPWOHZ3NVGHNST7KZA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.420914838 (406h47m7.291997883s) [noise=true , address=145.79.227.41:4085 , piece_id=OGRSMMM4JHNTZZZURJHFOBUAOPGDC6T2PCHDEX7YL7SADHG6AN6A, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.421340087 (406h47m7.292423132s) [noise=true , address=145.79.227.36:4517 , piece_id=DKVCILBC2JWXLG6G23ZMKEWQNDXYR4UNVD2XESJ57JIKIS5AXACA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.421402066 (406h47m7.292485101s) [noise=true , address=145.79.227.46:4603 , piece_id=5KQGOFJZAEFI5OGUVUIK2HEBRMMUVGXEOD47OQIALAIMVCPEMLFA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.422719935 (406h47m7.29380297s) [noise=true , address=145.79.227.45:4231 , piece_id=I7F2TC2A6E35C3LPHT4D62ZOAZRSMF6XZZWVKJEI6SUHWTRQ6DFA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.423220067 (406h47m7.294303112s) [noise=true , address=145.79.227.37:4566 , piece_id=4H2NM22RU4Y7H334JVNOTYO2FOUUXTKOCVYEHNJJ4HRVXVVXWOVA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.423702486 (406h47m7.294785531s) [noise=true , address=77.90.31.14:4663 , piece_id=X2NSIUKNVP2Y2MABYR3UPGZJSPF3CPS3EU6JYVNHX74RZTSGEQMA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:09.117304385 (406h47m8.988387421s) [upload=357142]: Next returned nil segment Dec 22 14:16:09.117321889 (406h47m8.988404924s) [upload=357142]: Waiting for error group managing segments... Dec 22 14:16:09.403723791 (406h47m9.274806826s) [noise=true , address=145.79.227.43:4157 , piece_id=PLKGYWWJYEHSEMRCRFVQUTXTGYN6LYHFF4HPL2TOW6F7Q4K6QL7Q, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:09.416402182 (406h47m9.287485247s) [noise=true , address=145.79.227.50:4374 , piece_id=2XXNMOGMBDEQLIBGHJ6E25INEXYZR7JWYKA76WB5CKLK42IU5L4Q, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:09.488678941 (406h47m9.359761976s) [noise=true , address=145.79.227.36:4517 , piece_id=DKVCILBC2JWXLG6G23ZMKEWQNDXYR4UNVD2XESJ57JIKIS5AXACA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:09.591529928 (406h47m9.462612963s) [noise=true , address=77.90.31.14:4663 , piece_id=X2NSIUKNVP2Y2MABYR3UPGZJSPF3CPS3EU6JYVNHX74RZTSGEQMA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:10.003825194 (406h47m9.874908229s) [noise=true , address=77.90.31.10:4519 , piece_id=FMJHUEO445SYIHAV47SFERXWEFDSCCN36WPT4RJTXABWLJ6IO2VQ, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.151289256 (406h47m11.022372301s) [noise=true , address=145.79.227.10:4401 , piece_id=HVP7A7MFY7IM2S4P5QFS3T2BNYINUQRSF5TJ3OZLWH5REX2QCNJA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.190801796 (406h47m11.061884841s) [noise=true , address=145.79.227.48:4302 , piece_id=WYPWNL6UUKRVETGRAKJJMS5US3B4LQBJ6TRDYBUTRLXKGPFPPEEA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.216210448 (406h47m11.087293483s) [noise=true , address=145.79.227.49:4339 , piece_id=ANLDLFM6BWPIJJIKHCY7CMWU2KIYTTNWA4ETR3DGNZG6GO7XM5QQ, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.447886943 (406h47m11.318969978s) [noise=true , address=145.79.227.17:4972 , piece_id=XOKLGQXCXZCOWYK5H4TPHRDHLLJXTII36XFZBX4XWYKHFE24WVOQ, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.513411914 (406h47m11.384494949s) [noise=true , address=145.79.227.41:4085 , piece_id=OGRSMMM4JHNTZZZURJHFOBUAOPGDC6T2PCHDEX7YL7SADHG6AN6A, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.768195260 (406h47m11.639278305s) [noise=true , address=145.79.227.29:4942 , piece_id=OMRNVS4BDDHYEFK5BEKTMKHTJUMJ4MY55FXPWOHZ3NVGHNST7KZA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.791646034 (406h47m11.66272908s) [noise=true , address=145.79.227.45:4231 , piece_id=I7F2TC2A6E35C3LPHT4D62ZOAZRSMF6XZZWVKJEI6SUHWTRQ6DFA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.807399862 (406h47m11.678482907s) [noise=true , address=145.79.227.46:4603 , piece_id=5KQGOFJZAEFI5OGUVUIK2HEBRMMUVGXEOD47OQIALAIMVCPEMLFA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.807424800 (406h47m11.678507845s) [seg_pos={0 0}, upload=357142]: Segment reached optimal threshold of 13 pieces. Dec 22 14:16:11.807688312 (406h47m11.678771357s) [noise=true , address=145.79.227.37:4566 , piece_id=4H2NM22RU4Y7H334JVNOTYO2FOUUXTKOCVYEHNJJ4HRVXVVXWOVA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F): protocol: expected piece hash; context canceled; EOF Dec 22 14:16:11.807698351 (406h47m11.678781396s) [noise=true , address=145.79.227.37:4566 , piece_id=4H2NM22RU4Y7H334JVNOTYO2FOUUXTKOCVYEHNJJ4HRVXVVXWOVA, seg_pos={0 0}, upload=357142]: Long tail cancellation due to optimal threshold Dec 22 14:16:11.807723459 (406h47m11.678806514s) [noise=true , address=145.79.227.18:4007 , piece_id=UMT4AQFVRTREIJDRAPCZTJY3URZYOHLX37QQTG5L53JFDW63ULQQ, seg_pos={0 0}, upload=357142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp): protocol: expected piece hash; context canceled; EOF Dec 22 14:16:11.807736343 (406h47m11.678819398s) [noise=true , address=145.79.227.18:4007 , piece_id=UMT4AQFVRTREIJDRAPCZTJY3URZYOHLX37QQTG5L53JFDW63ULQQ, seg_pos={0 0}, upload=357142]: Long tail cancellation due to optimal threshold Dec 22 14:16:11.807862974 (406h47m11.678946019s) [noise=true , address=45.91.116.8:4048 , piece_id=HQWNONHI5SJYLPULF2H75PVYLOPOPJTEC5ZPOZUT2HHLA6IC5ALA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 22 14:16:11.807871991 (406h47m11.678955026s) [noise=true , address=45.91.116.8:4048 , piece_id=HQWNONHI5SJYLPULF2H75PVYLOPOPJTEC5ZPOZUT2HHLA6IC5ALA, seg_pos={0 0}, upload=357142]: Long tail cancellation due to optimal threshold Dec 22 14:16:11.807896659 (406h47m11.678979704s) [seg_pos={0 0}, upload=357142]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 14:16:11.808114984 (406h47m11.679198019s) [upload=357142]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 14:16:11.808121817 (406h47m11.679204852s) [upload=357142]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 14:16:12.460702542 (406h47m12.331785577s) [upload=357142]: Done uploading segments... Dec 22 14:16:12.460721377 (406h47m12.331804412s) [upload=357142]: Upload finished. err: <nil> Duration: 8.356636842s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [....X............X..................X...X..................X] duration: 30.003817595s trace: https://jaeger.dev.storj.io/trace/30697bf8e531908d FAIL (2025-12-22 14:16:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 22 14:16:07.103235978 (406h47m6.974319013s) [upload=357142]: Starting upload Dec 22 14:16:07.103295332 (406h47m6.974378377s) [upload=357142]: Done starting upload Dec 22 14:16:07.103399340 (406h47m6.974482375s) [upload=357142]: Uploading segments... Dec 22 14:16:07.103487287 (406h47m6.974570332s) [upload=357142]: Got next segment. Inline: false Dec 22 14:16:07.103515991 (406h47m6.974599026s) [upload=357142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 14:16:07.103520289 (406h47m6.974603325s) [upload=357142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 14:16:07.415289927 (406h47m7.286372962s) [seg_pos={0 0}, upload=357142]: Begin upload segment... Dec 22 14:16:07.415336014 (406h47m7.286419049s) [seg_pos={0 0}, upload=357142]: Done begin upload segment. Dec 22 14:16:07.415363607 (406h47m7.286446652s) [noise=true , address=145.79.227.17:4972 , piece_id=XOKLGQXCXZCOWYK5H4TPHRDHLLJXTII36XFZBX4XWYKHFE24WVOQ, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.415526457 (406h47m7.286609492s) [noise=true , address=145.79.227.18:4007 , piece_id=UMT4AQFVRTREIJDRAPCZTJY3URZYOHLX37QQTG5L53JFDW63ULQQ, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.416519017 (406h47m7.287602062s) [noise=true , address=145.79.227.48:4302 , piece_id=WYPWNL6UUKRVETGRAKJJMS5US3B4LQBJ6TRDYBUTRLXKGPFPPEEA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.417017777 (406h47m7.288100822s) [noise=true , address=145.79.227.50:4374 , piece_id=2XXNMOGMBDEQLIBGHJ6E25INEXYZR7JWYKA76WB5CKLK42IU5L4Q, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.417027014 (406h47m7.288110059s) [noise=true , address=145.79.227.43:4157 , piece_id=PLKGYWWJYEHSEMRCRFVQUTXTGYN6LYHFF4HPL2TOW6F7Q4K6QL7Q, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.417966864 (406h47m7.289049909s) [noise=true , address=45.91.116.8:4048 , piece_id=HQWNONHI5SJYLPULF2H75PVYLOPOPJTEC5ZPOZUT2HHLA6IC5ALA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.418744234 (406h47m7.289827269s) [noise=true , address=145.79.227.49:4339 , piece_id=ANLDLFM6BWPIJJIKHCY7CMWU2KIYTTNWA4ETR3DGNZG6GO7XM5QQ, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.419502227 (406h47m7.290585263s) [noise=true , address=77.90.31.10:4519 , piece_id=FMJHUEO445SYIHAV47SFERXWEFDSCCN36WPT4RJTXABWLJ6IO2VQ, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.419641743 (406h47m7.290724788s) [noise=true , address=145.79.227.10:4401 , piece_id=HVP7A7MFY7IM2S4P5QFS3T2BNYINUQRSF5TJ3OZLWH5REX2QCNJA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.420879160 (406h47m7.291962205s) [noise=true , address=145.79.227.29:4942 , piece_id=OMRNVS4BDDHYEFK5BEKTMKHTJUMJ4MY55FXPWOHZ3NVGHNST7KZA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.420914838 (406h47m7.291997883s) [noise=true , address=145.79.227.41:4085 , piece_id=OGRSMMM4JHNTZZZURJHFOBUAOPGDC6T2PCHDEX7YL7SADHG6AN6A, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.421340087 (406h47m7.292423132s) [noise=true , address=145.79.227.36:4517 , piece_id=DKVCILBC2JWXLG6G23ZMKEWQNDXYR4UNVD2XESJ57JIKIS5AXACA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.421402066 (406h47m7.292485101s) [noise=true , address=145.79.227.46:4603 , piece_id=5KQGOFJZAEFI5OGUVUIK2HEBRMMUVGXEOD47OQIALAIMVCPEMLFA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.422719935 (406h47m7.29380297s) [noise=true , address=145.79.227.45:4231 , piece_id=I7F2TC2A6E35C3LPHT4D62ZOAZRSMF6XZZWVKJEI6SUHWTRQ6DFA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.423220067 (406h47m7.294303112s) [noise=true , address=145.79.227.37:4566 , piece_id=4H2NM22RU4Y7H334JVNOTYO2FOUUXTKOCVYEHNJJ4HRVXVVXWOVA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:07.423702486 (406h47m7.294785531s) [noise=true , address=77.90.31.14:4663 , piece_id=X2NSIUKNVP2Y2MABYR3UPGZJSPF3CPS3EU6JYVNHX74RZTSGEQMA, seg_pos={0 0}, upload=357142]: Uploading piece... Dec 22 14:16:09.117304385 (406h47m8.988387421s) [upload=357142]: Next returned nil segment Dec 22 14:16:09.117321889 (406h47m8.988404924s) [upload=357142]: Waiting for error group managing segments... Dec 22 14:16:09.403723791 (406h47m9.274806826s) [noise=true , address=145.79.227.43:4157 , piece_id=PLKGYWWJYEHSEMRCRFVQUTXTGYN6LYHFF4HPL2TOW6F7Q4K6QL7Q, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:09.416402182 (406h47m9.287485247s) [noise=true , address=145.79.227.50:4374 , piece_id=2XXNMOGMBDEQLIBGHJ6E25INEXYZR7JWYKA76WB5CKLK42IU5L4Q, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:09.488678941 (406h47m9.359761976s) [noise=true , address=145.79.227.36:4517 , piece_id=DKVCILBC2JWXLG6G23ZMKEWQNDXYR4UNVD2XESJ57JIKIS5AXACA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:09.591529928 (406h47m9.462612963s) [noise=true , address=77.90.31.14:4663 , piece_id=X2NSIUKNVP2Y2MABYR3UPGZJSPF3CPS3EU6JYVNHX74RZTSGEQMA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:10.003825194 (406h47m9.874908229s) [noise=true , address=77.90.31.10:4519 , piece_id=FMJHUEO445SYIHAV47SFERXWEFDSCCN36WPT4RJTXABWLJ6IO2VQ, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.151289256 (406h47m11.022372301s) [noise=true , address=145.79.227.10:4401 , piece_id=HVP7A7MFY7IM2S4P5QFS3T2BNYINUQRSF5TJ3OZLWH5REX2QCNJA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.190801796 (406h47m11.061884841s) [noise=true , address=145.79.227.48:4302 , piece_id=WYPWNL6UUKRVETGRAKJJMS5US3B4LQBJ6TRDYBUTRLXKGPFPPEEA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.216210448 (406h47m11.087293483s) [noise=true , address=145.79.227.49:4339 , piece_id=ANLDLFM6BWPIJJIKHCY7CMWU2KIYTTNWA4ETR3DGNZG6GO7XM5QQ, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.447886943 (406h47m11.318969978s) [noise=true , address=145.79.227.17:4972 , piece_id=XOKLGQXCXZCOWYK5H4TPHRDHLLJXTII36XFZBX4XWYKHFE24WVOQ, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.513411914 (406h47m11.384494949s) [noise=true , address=145.79.227.41:4085 , piece_id=OGRSMMM4JHNTZZZURJHFOBUAOPGDC6T2PCHDEX7YL7SADHG6AN6A, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.768195260 (406h47m11.639278305s) [noise=true , address=145.79.227.29:4942 , piece_id=OMRNVS4BDDHYEFK5BEKTMKHTJUMJ4MY55FXPWOHZ3NVGHNST7KZA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.791646034 (406h47m11.66272908s) [noise=true , address=145.79.227.45:4231 , piece_id=I7F2TC2A6E35C3LPHT4D62ZOAZRSMF6XZZWVKJEI6SUHWTRQ6DFA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.807399862 (406h47m11.678482907s) [noise=true , address=145.79.227.46:4603 , piece_id=5KQGOFJZAEFI5OGUVUIK2HEBRMMUVGXEOD47OQIALAIMVCPEMLFA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: <nil> Dec 22 14:16:11.807424800 (406h47m11.678507845s) [seg_pos={0 0}, upload=357142]: Segment reached optimal threshold of 13 pieces. Dec 22 14:16:11.807688312 (406h47m11.678771357s) [noise=true , address=145.79.227.37:4566 , piece_id=4H2NM22RU4Y7H334JVNOTYO2FOUUXTKOCVYEHNJJ4HRVXVVXWOVA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F): protocol: expected piece hash; context canceled; EOF Dec 22 14:16:11.807698351 (406h47m11.678781396s) [noise=true , address=145.79.227.37:4566 , piece_id=4H2NM22RU4Y7H334JVNOTYO2FOUUXTKOCVYEHNJJ4HRVXVVXWOVA, seg_pos={0 0}, upload=357142]: Long tail cancellation due to optimal threshold Dec 22 14:16:11.807723459 (406h47m11.678806514s) [noise=true , address=145.79.227.18:4007 , piece_id=UMT4AQFVRTREIJDRAPCZTJY3URZYOHLX37QQTG5L53JFDW63ULQQ, seg_pos={0 0}, upload=357142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp): protocol: expected piece hash; context canceled; EOF Dec 22 14:16:11.807736343 (406h47m11.678819398s) [noise=true , address=145.79.227.18:4007 , piece_id=UMT4AQFVRTREIJDRAPCZTJY3URZYOHLX37QQTG5L53JFDW63ULQQ, seg_pos={0 0}, upload=357142]: Long tail cancellation due to optimal threshold Dec 22 14:16:11.807862974 (406h47m11.678946019s) [noise=true , address=45.91.116.8:4048 , piece_id=HQWNONHI5SJYLPULF2H75PVYLOPOPJTEC5ZPOZUT2HHLA6IC5ALA, seg_pos={0 0}, upload=357142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 22 14:16:11.807871991 (406h47m11.678955026s) [noise=true , address=45.91.116.8:4048 , piece_id=HQWNONHI5SJYLPULF2H75PVYLOPOPJTEC5ZPOZUT2HHLA6IC5ALA, seg_pos={0 0}, upload=357142]: Long tail cancellation due to optimal threshold Dec 22 14:16:11.807896659 (406h47m11.678979704s) [seg_pos={0 0}, upload=357142]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 14:16:11.808114984 (406h47m11.679198019s) [upload=357142]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 14:16:11.808121817 (406h47m11.679204852s) [upload=357142]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 14:16:12.460702542 (406h47m12.331785577s) [upload=357142]: Done uploading segments... Dec 22 14:16:12.460721377 (406h47m12.331804412s) [upload=357142]: Upload finished. err: <nil> Duration: 30.003843714s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 1.473894855s trace: https://jaeger.dev.storj.io/trace/1e7f77260788e649 FAIL (2025-12-22 14:16:31 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.473922578s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [..................X....X.X....XX..X..............X.....X.X..] duration: 3.54302481s trace: https://jaeger.dev.storj.io/trace/7fcb66a7d28c4377 FAIL (2025-12-22 14:16:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.543159547s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 5.710428635s trace: https://jaeger.dev.storj.io/trace/538cdd5d04469e38 FAIL (2025-12-22 14:15:54 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.710456197s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 1.478139649s trace: https://jaeger.dev.storj.io/trace/1e7f77260788e649 FAIL (2025-12-22 14:15:31 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.478161672s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....X............X..................X...X..................] duration: 6.014512588s trace: https://jaeger.dev.storj.io/trace/5192c8b80a992813 FAIL (2025-12-22 14:15:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: 6VPLAWTOFZGIO52L5V6QO3TXN4DZI65XAAYP674ZYQR3PHZGI7ZQ) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: AFZWKLQBUKO3CSBCLTFWQ7IKLDYVFWK6OQ6H2W4UDYQWTULV7LEQ) context deadline exceeded ======LOGS====== Dec 22 14:02:09.540319730 (406h33m9.411402765s) [upload=356939]: Starting upload Dec 22 14:02:09.540380797 (406h33m9.411463852s) [upload=356939]: Done starting upload Dec 22 14:02:09.540611786 (406h33m9.411694831s) [upload=356939]: Uploading segments... Dec 22 14:02:09.540757183 (406h33m9.411840228s) [upload=356939]: Got next segment. Inline: false Dec 22 14:02:09.540808471 (406h33m9.411891516s) [upload=356939]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 14:02:09.540816275 (406h33m9.41189932s) [upload=356939]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 14:02:09.787889539 (406h33m9.658972584s) [seg_pos={0 0}, upload=356939]: Begin upload segment... Dec 22 14:02:09.787961536 (406h33m9.659044571s) [seg_pos={0 0}, upload=356939]: Done begin upload segment. Dec 22 14:02:09.788026600 (406h33m9.659109645s) [noise=true , address=145.79.227.47:4266 , piece_id=GWA6QDHSVQPD7JN7FXRE53RQIUYC43HRMXN6HBZNJCNKCTVQIHDA, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:09.788051907 (406h33m9.659134973s) [noise=true , address=45.91.116.7:4009 , piece_id=DZGZKBYFKLID3CEBDK24SN6WA7BUMIZRZWEGR4UM77SGLRPZUFEA, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:09.788912206 (406h33m9.659995251s) [noise=true , address=77.90.31.16:4736 , piece_id=BPWDDWM5I5PPWTVLQS5UYVCZMUALMEGAZA5ID3AZFTFH22HKX3QA, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:09.789394484 (406h33m9.660477529s) [noise=true , address=145.79.227.43:4157 , piece_id=RJP5KBDP5I434PDQVDR6E6YEDBS5QH33GBIPSA5ETEELZRAFQTXQ, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:09.789783124 (406h33m9.660866169s) [noise=true , address=45.91.116.8:4048 , piece_id=EGUB527VKRNI3Z7RMOKKLKXAK2VBXOGEIVJ34JIMO3AUKFDCC2NA, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:09.789923622 (406h33m9.661006667s) [noise=true , address=145.79.227.50:4375 , piece_id=6IAOXZUAF6Q7MBTQKLMDMELI4Y7LICBFDZTZBWZAETSRTYFJ4JSQ, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:09.790372917 (406h33m9.661455952s) [noise=true , address=145.79.227.39:4013 , piece_id=BRVZ7WJRFCK7ZXEWF6TB4DZCKNZGB4IH5L44T3LKNVIXOL7PEUPA, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:09.790398216 (406h33m9.661481251s) [noise=true , address=145.79.227.46:4603 , piece_id=C22WFSY2CAFQM6SHZBC3BWAK56UI5O2YFXX3TFYCUL7F2SKQ5UDQ, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:09.790750667 (406h33m9.661833832s) [noise=true , address=145.79.227.28:4726 , piece_id=ADJKKC7WYM7VG4DOD6M3KPLY7CHWBVHGVP7WYULTMKO7247SZXYA, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:09.790810932 (406h33m9.661893977s) [noise=true , address=145.79.227.36:4518 , piece_id=ZDGONTP3IPQWJS7WZVAOUFJQWOP7VYPSMRXEBOUGMRWRGZDUCS6A, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:09.791152573 (406h33m9.662235608s) [noise=true , address=77.90.31.19:4842 , piece_id=TXDC477SJ73J7SGLRSKV5VJ2P76GPJ6V5D3SPY5FA2YMWARILDXQ, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:09.791214379 (406h33m9.662297425s) [noise=true , address=145.79.227.29:4942 , piece_id=F3JXRWHD4HBMDK3SQMF5BLWUHYNZYWGYEARIB22IWDODLICKO7QA, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:09.791438928 (406h33m9.662521963s) [noise=true , address=45.91.116.11:4117 , piece_id=6VPLAWTOFZGIO52L5V6QO3TXN4DZI65XAAYP674ZYQR3PHZGI7ZQ, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:09.791655159 (406h33m9.662738204s) [noise=true , address=145.79.227.48:4303 , piece_id=4LVJNSYFP3JY4VMQ3MYKB34EXSNHWYP55DJVANGNCN4FE5HGY4UA, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:09.792046454 (406h33m9.663129489s) [noise=true , address=45.91.116.5:4299 , piece_id=RDYBDEMBZ3BJ3OYNTGO3QXEJCMTVAX4OIOZX43U3AM4V4AZPYYGQ, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:09.793687288 (406h33m9.664770333s) [noise=true , address=45.91.116.6:4973 , piece_id=AFZWKLQBUKO3CSBCLTFWQ7IKLDYVFWK6OQ6H2W4UDYQWTULV7LEQ, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:11.517644048 (406h33m11.388727093s) [upload=356939]: Next returned nil segment Dec 22 14:02:11.517663625 (406h33m11.38874667s) [upload=356939]: Waiting for error group managing segments... Dec 22 14:02:11.806697598 (406h33m11.677780643s) [noise=true , address=145.79.227.50:4375 , piece_id=6IAOXZUAF6Q7MBTQKLMDMELI4Y7LICBFDZTZBWZAETSRTYFJ4JSQ, seg_pos={0 0}, upload=356939]: Done uploading piece. err: <nil> Dec 22 14:02:11.807456193 (406h33m11.678539238s) [noise=true , address=145.79.227.29:4942 , piece_id=F3JXRWHD4HBMDK3SQMF5BLWUHYNZYWGYEARIB22IWDODLICKO7QA, seg_pos={0 0}, upload=356939]: Done uploading piece. err: <nil> Dec 22 14:02:11.808354183 (406h33m11.679437228s) [noise=true , address=145.79.227.43:4157 , piece_id=RJP5KBDP5I434PDQVDR6E6YEDBS5QH33GBIPSA5ETEELZRAFQTXQ, seg_pos={0 0}, upload=356939]: Done uploading piece. err: <nil> Dec 22 14:02:11.808910823 (406h33m11.679993868s) [noise=true , address=145.79.227.36:4518 , piece_id=ZDGONTP3IPQWJS7WZVAOUFJQWOP7VYPSMRXEBOUGMRWRGZDUCS6A, seg_pos={0 0}, upload=356939]: Done uploading piece. err: <nil> Dec 22 14:02:11.812084746 (406h33m11.683167791s) [noise=true , address=145.79.227.48:4303 , piece_id=4LVJNSYFP3JY4VMQ3MYKB34EXSNHWYP55DJVANGNCN4FE5HGY4UA, seg_pos={0 0}, upload=356939]: Done uploading piece. err: <nil> Dec 22 14:02:11.827226038 (406h33m11.698309073s) [noise=true , address=145.79.227.28:4726 , piece_id=ADJKKC7WYM7VG4DOD6M3KPLY7CHWBVHGVP7WYULTMKO7247SZXYA, seg_pos={0 0}, upload=356939]: Done uploading piece. err: <nil> Dec 22 14:02:11.924134049 (406h33m11.795217084s) [noise=true , address=145.79.227.46:4603 , piece_id=C22WFSY2CAFQM6SHZBC3BWAK56UI5O2YFXX3TFYCUL7F2SKQ5UDQ, seg_pos={0 0}, upload=356939]: Done uploading piece. err: <nil> Dec 22 14:02:11.937124565 (406h33m11.80820761s) [noise=true , address=145.79.227.47:4266 , piece_id=GWA6QDHSVQPD7JN7FXRE53RQIUYC43HRMXN6HBZNJCNKCTVQIHDA, seg_pos={0 0}, upload=356939]: Done uploading piece. err: <nil> Dec 22 14:02:12.022864229 (406h33m11.893947274s) [noise=true , address=77.90.31.16:4736 , piece_id=BPWDDWM5I5PPWTVLQS5UYVCZMUALMEGAZA5ID3AZFTFH22HKX3QA, seg_pos={0 0}, upload=356939]: Done uploading piece. err: <nil> Dec 22 14:02:12.033942754 (406h33m11.905025799s) [noise=true , address=45.91.116.6:4973 , piece_id=AFZWKLQBUKO3CSBCLTFWQ7IKLDYVFWK6OQ6H2W4UDYQWTULV7LEQ, seg_pos={0 0}, upload=356939]: Done uploading piece. err: <nil> Dec 22 14:02:12.067209510 (406h33m11.938292575s) [noise=true , address=45.91.116.11:4117 , piece_id=6VPLAWTOFZGIO52L5V6QO3TXN4DZI65XAAYP674ZYQR3PHZGI7ZQ, seg_pos={0 0}, upload=356939]: Done uploading piece. err: <nil> Dec 22 14:02:12.145632713 (406h33m12.016715758s) [noise=true , address=45.91.116.5:4299 , piece_id=RDYBDEMBZ3BJ3OYNTGO3QXEJCMTVAX4OIOZX43U3AM4V4AZPYYGQ, seg_pos={0 0}, upload=356939]: Done uploading piece. err: <nil> Dec 22 14:02:12.209033870 (406h33m12.080116905s) [noise=true , address=77.90.31.19:4842 , piece_id=TXDC477SJ73J7SGLRSKV5VJ2P76GPJ6V5D3SPY5FA2YMWARILDXQ, seg_pos={0 0}, upload=356939]: Done uploading piece. err: <nil> Dec 22 14:02:12.209055591 (406h33m12.080138636s) [seg_pos={0 0}, upload=356939]: Segment reached optimal threshold of 13 pieces. Dec 22 14:02:12.209243389 (406h33m12.080326424s) [noise=true , address=45.91.116.8:4048 , piece_id=EGUB527VKRNI3Z7RMOKKLKXAK2VBXOGEIVJ34JIMO3AUKFDCC2NA, seg_pos={0 0}, upload=356939]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 22 14:02:12.209249490 (406h33m12.080332525s) [noise=true , address=45.91.116.8:4048 , piece_id=EGUB527VKRNI3Z7RMOKKLKXAK2VBXOGEIVJ34JIMO3AUKFDCC2NA, seg_pos={0 0}, upload=356939]: Long tail cancellation due to optimal threshold Dec 22 14:02:12.209432088 (406h33m12.080515123s) [noise=true , address=145.79.227.39:4013 , piece_id=BRVZ7WJRFCK7ZXEWF6TB4DZCKNZGB4IH5L44T3LKNVIXOL7PEUPA, seg_pos={0 0}, upload=356939]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b): protocol: expected piece hash; context canceled; EOF Dec 22 14:02:12.209449672 (406h33m12.080532717s) [noise=true , address=145.79.227.39:4013 , piece_id=BRVZ7WJRFCK7ZXEWF6TB4DZCKNZGB4IH5L44T3LKNVIXOL7PEUPA, seg_pos={0 0}, upload=356939]: Long tail cancellation due to optimal threshold Dec 22 14:02:12.209469569 (406h33m12.080552604s) [noise=true , address=45.91.116.7:4009 , piece_id=DZGZKBYFKLID3CEBDK24SN6WA7BUMIZRZWEGR4UM77SGLRPZUFEA, seg_pos={0 0}, upload=356939]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:49184->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:49184->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49184->45.91.116.7:4009: use of closed network connection Dec 22 14:02:12.209478777 (406h33m12.080561822s) [noise=true , address=45.91.116.7:4009 , piece_id=DZGZKBYFKLID3CEBDK24SN6WA7BUMIZRZWEGR4UM77SGLRPZUFEA, seg_pos={0 0}, upload=356939]: Long tail cancellation due to optimal threshold Dec 22 14:02:12.209497322 (406h33m12.080580357s) [seg_pos={0 0}, upload=356939]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 14:02:12.209586322 (406h33m12.080669367s) [upload=356939]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 14:02:12.209591762 (406h33m12.080674797s) [upload=356939]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 14:02:12.500528820 (406h33m12.371611875s) [upload=356939]: Done uploading segments... Dec 22 14:02:12.500556723 (406h33m12.371639768s) [upload=356939]: Upload finished. err: <nil> Duration: 6.01453984s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [...................X....X.X....XX..X..............X.....X.X.] duration: 4.952588317s trace: https://jaeger.dev.storj.io/trace/7fcb66a7d28c4377 FAIL (2025-12-22 14:15:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.952618304s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 4.902593379s trace: https://jaeger.dev.storj.io/trace/538cdd5d04469e38 FAIL (2025-12-22 14:15:21 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.902629739s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 1.462520137s trace: https://jaeger.dev.storj.io/trace/1e7f77260788e649 FAIL (2025-12-22 14:15:16 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.462551057s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 5.073703285s trace: https://jaeger.dev.storj.io/trace/538cdd5d04469e38 FAIL (2025-12-22 14:15:00 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.073733071s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [....................X....X.X....XX..X..............X.....X.X] duration: 5.001207869s trace: https://jaeger.dev.storj.io/trace/7fcb66a7d28c4377 FAIL (2025-12-22 14:14:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001236143s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 1.448657631s trace: https://jaeger.dev.storj.io/trace/1e7f77260788e649 FAIL (2025-12-22 14:14:49 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.448691026s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [......X............X..................X...X.................] duration: 5.931869398s trace: https://jaeger.dev.storj.io/trace/5192c8b80a992813 FAIL (2025-12-22 14:14:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: 6VPLAWTOFZGIO52L5V6QO3TXN4DZI65XAAYP674ZYQR3PHZGI7ZQ) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: AFZWKLQBUKO3CSBCLTFWQ7IKLDYVFWK6OQ6H2W4UDYQWTULV7LEQ) context deadline exceeded ======LOGS====== Dec 22 14:02:09.540319730 (406h33m9.411402765s) [upload=356939]: Starting upload Dec 22 14:02:09.540380797 (406h33m9.411463852s) [upload=356939]: Done starting upload Dec 22 14:02:09.540611786 (406h33m9.411694831s) [upload=356939]: Uploading segments... Dec 22 14:02:09.540757183 (406h33m9.411840228s) [upload=356939]: Got next segment. Inline: false Dec 22 14:02:09.540808471 (406h33m9.411891516s) [upload=356939]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 14:02:09.540816275 (406h33m9.41189932s) [upload=356939]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 14:02:09.787889539 (406h33m9.658972584s) [seg_pos={0 0}, upload=356939]: Begin upload segment... Dec 22 14:02:09.787961536 (406h33m9.659044571s) [seg_pos={0 0}, upload=356939]: Done begin upload segment. Dec 22 14:02:09.788026600 (406h33m9.659109645s) [noise=true , address=145.79.227.47:4266 , piece_id=GWA6QDHSVQPD7JN7FXRE53RQIUYC43HRMXN6HBZNJCNKCTVQIHDA, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:09.788051907 (406h33m9.659134973s) [noise=true , address=45.91.116.7:4009 , piece_id=DZGZKBYFKLID3CEBDK24SN6WA7BUMIZRZWEGR4UM77SGLRPZUFEA, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:09.788912206 (406h33m9.659995251s) [noise=true , address=77.90.31.16:4736 , piece_id=BPWDDWM5I5PPWTVLQS5UYVCZMUALMEGAZA5ID3AZFTFH22HKX3QA, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:09.789394484 (406h33m9.660477529s) [noise=true , address=145.79.227.43:4157 , piece_id=RJP5KBDP5I434PDQVDR6E6YEDBS5QH33GBIPSA5ETEELZRAFQTXQ, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:09.789783124 (406h33m9.660866169s) [noise=true , address=45.91.116.8:4048 , piece_id=EGUB527VKRNI3Z7RMOKKLKXAK2VBXOGEIVJ34JIMO3AUKFDCC2NA, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:09.789923622 (406h33m9.661006667s) [noise=true , address=145.79.227.50:4375 , piece_id=6IAOXZUAF6Q7MBTQKLMDMELI4Y7LICBFDZTZBWZAETSRTYFJ4JSQ, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:09.790372917 (406h33m9.661455952s) [noise=true , address=145.79.227.39:4013 , piece_id=BRVZ7WJRFCK7ZXEWF6TB4DZCKNZGB4IH5L44T3LKNVIXOL7PEUPA, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:09.790398216 (406h33m9.661481251s) [noise=true , address=145.79.227.46:4603 , piece_id=C22WFSY2CAFQM6SHZBC3BWAK56UI5O2YFXX3TFYCUL7F2SKQ5UDQ, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:09.790750667 (406h33m9.661833832s) [noise=true , address=145.79.227.28:4726 , piece_id=ADJKKC7WYM7VG4DOD6M3KPLY7CHWBVHGVP7WYULTMKO7247SZXYA, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:09.790810932 (406h33m9.661893977s) [noise=true , address=145.79.227.36:4518 , piece_id=ZDGONTP3IPQWJS7WZVAOUFJQWOP7VYPSMRXEBOUGMRWRGZDUCS6A, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:09.791152573 (406h33m9.662235608s) [noise=true , address=77.90.31.19:4842 , piece_id=TXDC477SJ73J7SGLRSKV5VJ2P76GPJ6V5D3SPY5FA2YMWARILDXQ, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:09.791214379 (406h33m9.662297425s) [noise=true , address=145.79.227.29:4942 , piece_id=F3JXRWHD4HBMDK3SQMF5BLWUHYNZYWGYEARIB22IWDODLICKO7QA, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:09.791438928 (406h33m9.662521963s) [noise=true , address=45.91.116.11:4117 , piece_id=6VPLAWTOFZGIO52L5V6QO3TXN4DZI65XAAYP674ZYQR3PHZGI7ZQ, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:09.791655159 (406h33m9.662738204s) [noise=true , address=145.79.227.48:4303 , piece_id=4LVJNSYFP3JY4VMQ3MYKB34EXSNHWYP55DJVANGNCN4FE5HGY4UA, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:09.792046454 (406h33m9.663129489s) [noise=true , address=45.91.116.5:4299 , piece_id=RDYBDEMBZ3BJ3OYNTGO3QXEJCMTVAX4OIOZX43U3AM4V4AZPYYGQ, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:09.793687288 (406h33m9.664770333s) [noise=true , address=45.91.116.6:4973 , piece_id=AFZWKLQBUKO3CSBCLTFWQ7IKLDYVFWK6OQ6H2W4UDYQWTULV7LEQ, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:11.517644048 (406h33m11.388727093s) [upload=356939]: Next returned nil segment Dec 22 14:02:11.517663625 (406h33m11.38874667s) [upload=356939]: Waiting for error group managing segments... Dec 22 14:02:11.806697598 (406h33m11.677780643s) [noise=true , address=145.79.227.50:4375 , piece_id=6IAOXZUAF6Q7MBTQKLMDMELI4Y7LICBFDZTZBWZAETSRTYFJ4JSQ, seg_pos={0 0}, upload=356939]: Done uploading piece. err: <nil> Dec 22 14:02:11.807456193 (406h33m11.678539238s) [noise=true , address=145.79.227.29:4942 , piece_id=F3JXRWHD4HBMDK3SQMF5BLWUHYNZYWGYEARIB22IWDODLICKO7QA, seg_pos={0 0}, upload=356939]: Done uploading piece. err: <nil> Dec 22 14:02:11.808354183 (406h33m11.679437228s) [noise=true , address=145.79.227.43:4157 , piece_id=RJP5KBDP5I434PDQVDR6E6YEDBS5QH33GBIPSA5ETEELZRAFQTXQ, seg_pos={0 0}, upload=356939]: Done uploading piece. err: <nil> Dec 22 14:02:11.808910823 (406h33m11.679993868s) [noise=true , address=145.79.227.36:4518 , piece_id=ZDGONTP3IPQWJS7WZVAOUFJQWOP7VYPSMRXEBOUGMRWRGZDUCS6A, seg_pos={0 0}, upload=356939]: Done uploading piece. err: <nil> Dec 22 14:02:11.812084746 (406h33m11.683167791s) [noise=true , address=145.79.227.48:4303 , piece_id=4LVJNSYFP3JY4VMQ3MYKB34EXSNHWYP55DJVANGNCN4FE5HGY4UA, seg_pos={0 0}, upload=356939]: Done uploading piece. err: <nil> Dec 22 14:02:11.827226038 (406h33m11.698309073s) [noise=true , address=145.79.227.28:4726 , piece_id=ADJKKC7WYM7VG4DOD6M3KPLY7CHWBVHGVP7WYULTMKO7247SZXYA, seg_pos={0 0}, upload=356939]: Done uploading piece. err: <nil> Dec 22 14:02:11.924134049 (406h33m11.795217084s) [noise=true , address=145.79.227.46:4603 , piece_id=C22WFSY2CAFQM6SHZBC3BWAK56UI5O2YFXX3TFYCUL7F2SKQ5UDQ, seg_pos={0 0}, upload=356939]: Done uploading piece. err: <nil> Dec 22 14:02:11.937124565 (406h33m11.80820761s) [noise=true , address=145.79.227.47:4266 , piece_id=GWA6QDHSVQPD7JN7FXRE53RQIUYC43HRMXN6HBZNJCNKCTVQIHDA, seg_pos={0 0}, upload=356939]: Done uploading piece. err: <nil> Dec 22 14:02:12.022864229 (406h33m11.893947274s) [noise=true , address=77.90.31.16:4736 , piece_id=BPWDDWM5I5PPWTVLQS5UYVCZMUALMEGAZA5ID3AZFTFH22HKX3QA, seg_pos={0 0}, upload=356939]: Done uploading piece. err: <nil> Dec 22 14:02:12.033942754 (406h33m11.905025799s) [noise=true , address=45.91.116.6:4973 , piece_id=AFZWKLQBUKO3CSBCLTFWQ7IKLDYVFWK6OQ6H2W4UDYQWTULV7LEQ, seg_pos={0 0}, upload=356939]: Done uploading piece. err: <nil> Dec 22 14:02:12.067209510 (406h33m11.938292575s) [noise=true , address=45.91.116.11:4117 , piece_id=6VPLAWTOFZGIO52L5V6QO3TXN4DZI65XAAYP674ZYQR3PHZGI7ZQ, seg_pos={0 0}, upload=356939]: Done uploading piece. err: <nil> Dec 22 14:02:12.145632713 (406h33m12.016715758s) [noise=true , address=45.91.116.5:4299 , piece_id=RDYBDEMBZ3BJ3OYNTGO3QXEJCMTVAX4OIOZX43U3AM4V4AZPYYGQ, seg_pos={0 0}, upload=356939]: Done uploading piece. err: <nil> Dec 22 14:02:12.209033870 (406h33m12.080116905s) [noise=true , address=77.90.31.19:4842 , piece_id=TXDC477SJ73J7SGLRSKV5VJ2P76GPJ6V5D3SPY5FA2YMWARILDXQ, seg_pos={0 0}, upload=356939]: Done uploading piece. err: <nil> Dec 22 14:02:12.209055591 (406h33m12.080138636s) [seg_pos={0 0}, upload=356939]: Segment reached optimal threshold of 13 pieces. Dec 22 14:02:12.209243389 (406h33m12.080326424s) [noise=true , address=45.91.116.8:4048 , piece_id=EGUB527VKRNI3Z7RMOKKLKXAK2VBXOGEIVJ34JIMO3AUKFDCC2NA, seg_pos={0 0}, upload=356939]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 22 14:02:12.209249490 (406h33m12.080332525s) [noise=true , address=45.91.116.8:4048 , piece_id=EGUB527VKRNI3Z7RMOKKLKXAK2VBXOGEIVJ34JIMO3AUKFDCC2NA, seg_pos={0 0}, upload=356939]: Long tail cancellation due to optimal threshold Dec 22 14:02:12.209432088 (406h33m12.080515123s) [noise=true , address=145.79.227.39:4013 , piece_id=BRVZ7WJRFCK7ZXEWF6TB4DZCKNZGB4IH5L44T3LKNVIXOL7PEUPA, seg_pos={0 0}, upload=356939]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b): protocol: expected piece hash; context canceled; EOF Dec 22 14:02:12.209449672 (406h33m12.080532717s) [noise=true , address=145.79.227.39:4013 , piece_id=BRVZ7WJRFCK7ZXEWF6TB4DZCKNZGB4IH5L44T3LKNVIXOL7PEUPA, seg_pos={0 0}, upload=356939]: Long tail cancellation due to optimal threshold Dec 22 14:02:12.209469569 (406h33m12.080552604s) [noise=true , address=45.91.116.7:4009 , piece_id=DZGZKBYFKLID3CEBDK24SN6WA7BUMIZRZWEGR4UM77SGLRPZUFEA, seg_pos={0 0}, upload=356939]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:49184->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:49184->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49184->45.91.116.7:4009: use of closed network connection Dec 22 14:02:12.209478777 (406h33m12.080561822s) [noise=true , address=45.91.116.7:4009 , piece_id=DZGZKBYFKLID3CEBDK24SN6WA7BUMIZRZWEGR4UM77SGLRPZUFEA, seg_pos={0 0}, upload=356939]: Long tail cancellation due to optimal threshold Dec 22 14:02:12.209497322 (406h33m12.080580357s) [seg_pos={0 0}, upload=356939]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 14:02:12.209586322 (406h33m12.080669367s) [upload=356939]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 14:02:12.209591762 (406h33m12.080674797s) [upload=356939]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 14:02:12.500528820 (406h33m12.371611875s) [upload=356939]: Done uploading segments... Dec 22 14:02:12.500556723 (406h33m12.371639768s) [upload=356939]: Upload finished. err: <nil> Duration: 5.931969449s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [.....................X....X.X....XX..X..............X.....X.] duration: 3.823365642s trace: https://jaeger.dev.storj.io/trace/669433dfad21eff6 FAIL (2025-12-22 14:14:03 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.823392754s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 4.931238023s trace: https://jaeger.dev.storj.io/trace/538cdd5d04469e38 FAIL (2025-12-22 14:13:58 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.931265574s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......X............X..................X...X................] duration: 5.788059067s trace: https://jaeger.dev.storj.io/trace/5192c8b80a992813 FAIL (2025-12-22 14:13:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: 6VPLAWTOFZGIO52L5V6QO3TXN4DZI65XAAYP674ZYQR3PHZGI7ZQ) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: AFZWKLQBUKO3CSBCLTFWQ7IKLDYVFWK6OQ6H2W4UDYQWTULV7LEQ) context deadline exceeded ======LOGS====== Dec 22 14:02:09.540319730 (406h33m9.411402765s) [upload=356939]: Starting upload Dec 22 14:02:09.540380797 (406h33m9.411463852s) [upload=356939]: Done starting upload Dec 22 14:02:09.540611786 (406h33m9.411694831s) [upload=356939]: Uploading segments... Dec 22 14:02:09.540757183 (406h33m9.411840228s) [upload=356939]: Got next segment. Inline: false Dec 22 14:02:09.540808471 (406h33m9.411891516s) [upload=356939]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 14:02:09.540816275 (406h33m9.41189932s) [upload=356939]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 14:02:09.787889539 (406h33m9.658972584s) [seg_pos={0 0}, upload=356939]: Begin upload segment... Dec 22 14:02:09.787961536 (406h33m9.659044571s) [seg_pos={0 0}, upload=356939]: Done begin upload segment. Dec 22 14:02:09.788026600 (406h33m9.659109645s) [noise=true , address=145.79.227.47:4266 , piece_id=GWA6QDHSVQPD7JN7FXRE53RQIUYC43HRMXN6HBZNJCNKCTVQIHDA, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:09.788051907 (406h33m9.659134973s) [noise=true , address=45.91.116.7:4009 , piece_id=DZGZKBYFKLID3CEBDK24SN6WA7BUMIZRZWEGR4UM77SGLRPZUFEA, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:09.788912206 (406h33m9.659995251s) [noise=true , address=77.90.31.16:4736 , piece_id=BPWDDWM5I5PPWTVLQS5UYVCZMUALMEGAZA5ID3AZFTFH22HKX3QA, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:09.789394484 (406h33m9.660477529s) [noise=true , address=145.79.227.43:4157 , piece_id=RJP5KBDP5I434PDQVDR6E6YEDBS5QH33GBIPSA5ETEELZRAFQTXQ, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:09.789783124 (406h33m9.660866169s) [noise=true , address=45.91.116.8:4048 , piece_id=EGUB527VKRNI3Z7RMOKKLKXAK2VBXOGEIVJ34JIMO3AUKFDCC2NA, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:09.789923622 (406h33m9.661006667s) [noise=true , address=145.79.227.50:4375 , piece_id=6IAOXZUAF6Q7MBTQKLMDMELI4Y7LICBFDZTZBWZAETSRTYFJ4JSQ, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:09.790372917 (406h33m9.661455952s) [noise=true , address=145.79.227.39:4013 , piece_id=BRVZ7WJRFCK7ZXEWF6TB4DZCKNZGB4IH5L44T3LKNVIXOL7PEUPA, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:09.790398216 (406h33m9.661481251s) [noise=true , address=145.79.227.46:4603 , piece_id=C22WFSY2CAFQM6SHZBC3BWAK56UI5O2YFXX3TFYCUL7F2SKQ5UDQ, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:09.790750667 (406h33m9.661833832s) [noise=true , address=145.79.227.28:4726 , piece_id=ADJKKC7WYM7VG4DOD6M3KPLY7CHWBVHGVP7WYULTMKO7247SZXYA, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:09.790810932 (406h33m9.661893977s) [noise=true , address=145.79.227.36:4518 , piece_id=ZDGONTP3IPQWJS7WZVAOUFJQWOP7VYPSMRXEBOUGMRWRGZDUCS6A, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:09.791152573 (406h33m9.662235608s) [noise=true , address=77.90.31.19:4842 , piece_id=TXDC477SJ73J7SGLRSKV5VJ2P76GPJ6V5D3SPY5FA2YMWARILDXQ, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:09.791214379 (406h33m9.662297425s) [noise=true , address=145.79.227.29:4942 , piece_id=F3JXRWHD4HBMDK3SQMF5BLWUHYNZYWGYEARIB22IWDODLICKO7QA, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:09.791438928 (406h33m9.662521963s) [noise=true , address=45.91.116.11:4117 , piece_id=6VPLAWTOFZGIO52L5V6QO3TXN4DZI65XAAYP674ZYQR3PHZGI7ZQ, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:09.791655159 (406h33m9.662738204s) [noise=true , address=145.79.227.48:4303 , piece_id=4LVJNSYFP3JY4VMQ3MYKB34EXSNHWYP55DJVANGNCN4FE5HGY4UA, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:09.792046454 (406h33m9.663129489s) [noise=true , address=45.91.116.5:4299 , piece_id=RDYBDEMBZ3BJ3OYNTGO3QXEJCMTVAX4OIOZX43U3AM4V4AZPYYGQ, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:09.793687288 (406h33m9.664770333s) [noise=true , address=45.91.116.6:4973 , piece_id=AFZWKLQBUKO3CSBCLTFWQ7IKLDYVFWK6OQ6H2W4UDYQWTULV7LEQ, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:11.517644048 (406h33m11.388727093s) [upload=356939]: Next returned nil segment Dec 22 14:02:11.517663625 (406h33m11.38874667s) [upload=356939]: Waiting for error group managing segments... Dec 22 14:02:11.806697598 (406h33m11.677780643s) [noise=true , address=145.79.227.50:4375 , piece_id=6IAOXZUAF6Q7MBTQKLMDMELI4Y7LICBFDZTZBWZAETSRTYFJ4JSQ, seg_pos={0 0}, upload=356939]: Done uploading piece. err: <nil> Dec 22 14:02:11.807456193 (406h33m11.678539238s) [noise=true , address=145.79.227.29:4942 , piece_id=F3JXRWHD4HBMDK3SQMF5BLWUHYNZYWGYEARIB22IWDODLICKO7QA, seg_pos={0 0}, upload=356939]: Done uploading piece. err: <nil> Dec 22 14:02:11.808354183 (406h33m11.679437228s) [noise=true , address=145.79.227.43:4157 , piece_id=RJP5KBDP5I434PDQVDR6E6YEDBS5QH33GBIPSA5ETEELZRAFQTXQ, seg_pos={0 0}, upload=356939]: Done uploading piece. err: <nil> Dec 22 14:02:11.808910823 (406h33m11.679993868s) [noise=true , address=145.79.227.36:4518 , piece_id=ZDGONTP3IPQWJS7WZVAOUFJQWOP7VYPSMRXEBOUGMRWRGZDUCS6A, seg_pos={0 0}, upload=356939]: Done uploading piece. err: <nil> Dec 22 14:02:11.812084746 (406h33m11.683167791s) [noise=true , address=145.79.227.48:4303 , piece_id=4LVJNSYFP3JY4VMQ3MYKB34EXSNHWYP55DJVANGNCN4FE5HGY4UA, seg_pos={0 0}, upload=356939]: Done uploading piece. err: <nil> Dec 22 14:02:11.827226038 (406h33m11.698309073s) [noise=true , address=145.79.227.28:4726 , piece_id=ADJKKC7WYM7VG4DOD6M3KPLY7CHWBVHGVP7WYULTMKO7247SZXYA, seg_pos={0 0}, upload=356939]: Done uploading piece. err: <nil> Dec 22 14:02:11.924134049 (406h33m11.795217084s) [noise=true , address=145.79.227.46:4603 , piece_id=C22WFSY2CAFQM6SHZBC3BWAK56UI5O2YFXX3TFYCUL7F2SKQ5UDQ, seg_pos={0 0}, upload=356939]: Done uploading piece. err: <nil> Dec 22 14:02:11.937124565 (406h33m11.80820761s) [noise=true , address=145.79.227.47:4266 , piece_id=GWA6QDHSVQPD7JN7FXRE53RQIUYC43HRMXN6HBZNJCNKCTVQIHDA, seg_pos={0 0}, upload=356939]: Done uploading piece. err: <nil> Dec 22 14:02:12.022864229 (406h33m11.893947274s) [noise=true , address=77.90.31.16:4736 , piece_id=BPWDDWM5I5PPWTVLQS5UYVCZMUALMEGAZA5ID3AZFTFH22HKX3QA, seg_pos={0 0}, upload=356939]: Done uploading piece. err: <nil> Dec 22 14:02:12.033942754 (406h33m11.905025799s) [noise=true , address=45.91.116.6:4973 , piece_id=AFZWKLQBUKO3CSBCLTFWQ7IKLDYVFWK6OQ6H2W4UDYQWTULV7LEQ, seg_pos={0 0}, upload=356939]: Done uploading piece. err: <nil> Dec 22 14:02:12.067209510 (406h33m11.938292575s) [noise=true , address=45.91.116.11:4117 , piece_id=6VPLAWTOFZGIO52L5V6QO3TXN4DZI65XAAYP674ZYQR3PHZGI7ZQ, seg_pos={0 0}, upload=356939]: Done uploading piece. err: <nil> Dec 22 14:02:12.145632713 (406h33m12.016715758s) [noise=true , address=45.91.116.5:4299 , piece_id=RDYBDEMBZ3BJ3OYNTGO3QXEJCMTVAX4OIOZX43U3AM4V4AZPYYGQ, seg_pos={0 0}, upload=356939]: Done uploading piece. err: <nil> Dec 22 14:02:12.209033870 (406h33m12.080116905s) [noise=true , address=77.90.31.19:4842 , piece_id=TXDC477SJ73J7SGLRSKV5VJ2P76GPJ6V5D3SPY5FA2YMWARILDXQ, seg_pos={0 0}, upload=356939]: Done uploading piece. err: <nil> Dec 22 14:02:12.209055591 (406h33m12.080138636s) [seg_pos={0 0}, upload=356939]: Segment reached optimal threshold of 13 pieces. Dec 22 14:02:12.209243389 (406h33m12.080326424s) [noise=true , address=45.91.116.8:4048 , piece_id=EGUB527VKRNI3Z7RMOKKLKXAK2VBXOGEIVJ34JIMO3AUKFDCC2NA, seg_pos={0 0}, upload=356939]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 22 14:02:12.209249490 (406h33m12.080332525s) [noise=true , address=45.91.116.8:4048 , piece_id=EGUB527VKRNI3Z7RMOKKLKXAK2VBXOGEIVJ34JIMO3AUKFDCC2NA, seg_pos={0 0}, upload=356939]: Long tail cancellation due to optimal threshold Dec 22 14:02:12.209432088 (406h33m12.080515123s) [noise=true , address=145.79.227.39:4013 , piece_id=BRVZ7WJRFCK7ZXEWF6TB4DZCKNZGB4IH5L44T3LKNVIXOL7PEUPA, seg_pos={0 0}, upload=356939]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b): protocol: expected piece hash; context canceled; EOF Dec 22 14:02:12.209449672 (406h33m12.080532717s) [noise=true , address=145.79.227.39:4013 , piece_id=BRVZ7WJRFCK7ZXEWF6TB4DZCKNZGB4IH5L44T3LKNVIXOL7PEUPA, seg_pos={0 0}, upload=356939]: Long tail cancellation due to optimal threshold Dec 22 14:02:12.209469569 (406h33m12.080552604s) [noise=true , address=45.91.116.7:4009 , piece_id=DZGZKBYFKLID3CEBDK24SN6WA7BUMIZRZWEGR4UM77SGLRPZUFEA, seg_pos={0 0}, upload=356939]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:49184->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:49184->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49184->45.91.116.7:4009: use of closed network connection Dec 22 14:02:12.209478777 (406h33m12.080561822s) [noise=true , address=45.91.116.7:4009 , piece_id=DZGZKBYFKLID3CEBDK24SN6WA7BUMIZRZWEGR4UM77SGLRPZUFEA, seg_pos={0 0}, upload=356939]: Long tail cancellation due to optimal threshold Dec 22 14:02:12.209497322 (406h33m12.080580357s) [seg_pos={0 0}, upload=356939]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 14:02:12.209586322 (406h33m12.080669367s) [upload=356939]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 14:02:12.209591762 (406h33m12.080674797s) [upload=356939]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 14:02:12.500528820 (406h33m12.371611875s) [upload=356939]: Done uploading segments... Dec 22 14:02:12.500556723 (406h33m12.371639768s) [upload=356939]: Upload finished. err: <nil> Duration: 5.788084868s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 1.400709809s trace: https://jaeger.dev.storj.io/trace/1e7f77260788e649 FAIL (2025-12-22 14:13:48 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.400726241s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 5.104826127s trace: https://jaeger.dev.storj.io/trace/538cdd5d04469e38 FAIL (2025-12-22 14:13:15 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.104849972s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [........X............X..................X...X...............] duration: 5.794941546s trace: https://jaeger.dev.storj.io/trace/5192c8b80a992813 FAIL (2025-12-22 14:13:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: 6VPLAWTOFZGIO52L5V6QO3TXN4DZI65XAAYP674ZYQR3PHZGI7ZQ) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: AFZWKLQBUKO3CSBCLTFWQ7IKLDYVFWK6OQ6H2W4UDYQWTULV7LEQ) context deadline exceeded ======LOGS====== Dec 22 14:02:09.540319730 (406h33m9.411402765s) [upload=356939]: Starting upload Dec 22 14:02:09.540380797 (406h33m9.411463852s) [upload=356939]: Done starting upload Dec 22 14:02:09.540611786 (406h33m9.411694831s) [upload=356939]: Uploading segments... Dec 22 14:02:09.540757183 (406h33m9.411840228s) [upload=356939]: Got next segment. Inline: false Dec 22 14:02:09.540808471 (406h33m9.411891516s) [upload=356939]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 14:02:09.540816275 (406h33m9.41189932s) [upload=356939]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 14:02:09.787889539 (406h33m9.658972584s) [seg_pos={0 0}, upload=356939]: Begin upload segment... Dec 22 14:02:09.787961536 (406h33m9.659044571s) [seg_pos={0 0}, upload=356939]: Done begin upload segment. Dec 22 14:02:09.788026600 (406h33m9.659109645s) [noise=true , address=145.79.227.47:4266 , piece_id=GWA6QDHSVQPD7JN7FXRE53RQIUYC43HRMXN6HBZNJCNKCTVQIHDA, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:09.788051907 (406h33m9.659134973s) [noise=true , address=45.91.116.7:4009 , piece_id=DZGZKBYFKLID3CEBDK24SN6WA7BUMIZRZWEGR4UM77SGLRPZUFEA, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:09.788912206 (406h33m9.659995251s) [noise=true , address=77.90.31.16:4736 , piece_id=BPWDDWM5I5PPWTVLQS5UYVCZMUALMEGAZA5ID3AZFTFH22HKX3QA, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:09.789394484 (406h33m9.660477529s) [noise=true , address=145.79.227.43:4157 , piece_id=RJP5KBDP5I434PDQVDR6E6YEDBS5QH33GBIPSA5ETEELZRAFQTXQ, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:09.789783124 (406h33m9.660866169s) [noise=true , address=45.91.116.8:4048 , piece_id=EGUB527VKRNI3Z7RMOKKLKXAK2VBXOGEIVJ34JIMO3AUKFDCC2NA, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:09.789923622 (406h33m9.661006667s) [noise=true , address=145.79.227.50:4375 , piece_id=6IAOXZUAF6Q7MBTQKLMDMELI4Y7LICBFDZTZBWZAETSRTYFJ4JSQ, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:09.790372917 (406h33m9.661455952s) [noise=true , address=145.79.227.39:4013 , piece_id=BRVZ7WJRFCK7ZXEWF6TB4DZCKNZGB4IH5L44T3LKNVIXOL7PEUPA, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:09.790398216 (406h33m9.661481251s) [noise=true , address=145.79.227.46:4603 , piece_id=C22WFSY2CAFQM6SHZBC3BWAK56UI5O2YFXX3TFYCUL7F2SKQ5UDQ, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:09.790750667 (406h33m9.661833832s) [noise=true , address=145.79.227.28:4726 , piece_id=ADJKKC7WYM7VG4DOD6M3KPLY7CHWBVHGVP7WYULTMKO7247SZXYA, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:09.790810932 (406h33m9.661893977s) [noise=true , address=145.79.227.36:4518 , piece_id=ZDGONTP3IPQWJS7WZVAOUFJQWOP7VYPSMRXEBOUGMRWRGZDUCS6A, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:09.791152573 (406h33m9.662235608s) [noise=true , address=77.90.31.19:4842 , piece_id=TXDC477SJ73J7SGLRSKV5VJ2P76GPJ6V5D3SPY5FA2YMWARILDXQ, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:09.791214379 (406h33m9.662297425s) [noise=true , address=145.79.227.29:4942 , piece_id=F3JXRWHD4HBMDK3SQMF5BLWUHYNZYWGYEARIB22IWDODLICKO7QA, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:09.791438928 (406h33m9.662521963s) [noise=true , address=45.91.116.11:4117 , piece_id=6VPLAWTOFZGIO52L5V6QO3TXN4DZI65XAAYP674ZYQR3PHZGI7ZQ, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:09.791655159 (406h33m9.662738204s) [noise=true , address=145.79.227.48:4303 , piece_id=4LVJNSYFP3JY4VMQ3MYKB34EXSNHWYP55DJVANGNCN4FE5HGY4UA, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:09.792046454 (406h33m9.663129489s) [noise=true , address=45.91.116.5:4299 , piece_id=RDYBDEMBZ3BJ3OYNTGO3QXEJCMTVAX4OIOZX43U3AM4V4AZPYYGQ, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:09.793687288 (406h33m9.664770333s) [noise=true , address=45.91.116.6:4973 , piece_id=AFZWKLQBUKO3CSBCLTFWQ7IKLDYVFWK6OQ6H2W4UDYQWTULV7LEQ, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:11.517644048 (406h33m11.388727093s) [upload=356939]: Next returned nil segment Dec 22 14:02:11.517663625 (406h33m11.38874667s) [upload=356939]: Waiting for error group managing segments... Dec 22 14:02:11.806697598 (406h33m11.677780643s) [noise=true , address=145.79.227.50:4375 , piece_id=6IAOXZUAF6Q7MBTQKLMDMELI4Y7LICBFDZTZBWZAETSRTYFJ4JSQ, seg_pos={0 0}, upload=356939]: Done uploading piece. err: <nil> Dec 22 14:02:11.807456193 (406h33m11.678539238s) [noise=true , address=145.79.227.29:4942 , piece_id=F3JXRWHD4HBMDK3SQMF5BLWUHYNZYWGYEARIB22IWDODLICKO7QA, seg_pos={0 0}, upload=356939]: Done uploading piece. err: <nil> Dec 22 14:02:11.808354183 (406h33m11.679437228s) [noise=true , address=145.79.227.43:4157 , piece_id=RJP5KBDP5I434PDQVDR6E6YEDBS5QH33GBIPSA5ETEELZRAFQTXQ, seg_pos={0 0}, upload=356939]: Done uploading piece. err: <nil> Dec 22 14:02:11.808910823 (406h33m11.679993868s) [noise=true , address=145.79.227.36:4518 , piece_id=ZDGONTP3IPQWJS7WZVAOUFJQWOP7VYPSMRXEBOUGMRWRGZDUCS6A, seg_pos={0 0}, upload=356939]: Done uploading piece. err: <nil> Dec 22 14:02:11.812084746 (406h33m11.683167791s) [noise=true , address=145.79.227.48:4303 , piece_id=4LVJNSYFP3JY4VMQ3MYKB34EXSNHWYP55DJVANGNCN4FE5HGY4UA, seg_pos={0 0}, upload=356939]: Done uploading piece. err: <nil> Dec 22 14:02:11.827226038 (406h33m11.698309073s) [noise=true , address=145.79.227.28:4726 , piece_id=ADJKKC7WYM7VG4DOD6M3KPLY7CHWBVHGVP7WYULTMKO7247SZXYA, seg_pos={0 0}, upload=356939]: Done uploading piece. err: <nil> Dec 22 14:02:11.924134049 (406h33m11.795217084s) [noise=true , address=145.79.227.46:4603 , piece_id=C22WFSY2CAFQM6SHZBC3BWAK56UI5O2YFXX3TFYCUL7F2SKQ5UDQ, seg_pos={0 0}, upload=356939]: Done uploading piece. err: <nil> Dec 22 14:02:11.937124565 (406h33m11.80820761s) [noise=true , address=145.79.227.47:4266 , piece_id=GWA6QDHSVQPD7JN7FXRE53RQIUYC43HRMXN6HBZNJCNKCTVQIHDA, seg_pos={0 0}, upload=356939]: Done uploading piece. err: <nil> Dec 22 14:02:12.022864229 (406h33m11.893947274s) [noise=true , address=77.90.31.16:4736 , piece_id=BPWDDWM5I5PPWTVLQS5UYVCZMUALMEGAZA5ID3AZFTFH22HKX3QA, seg_pos={0 0}, upload=356939]: Done uploading piece. err: <nil> Dec 22 14:02:12.033942754 (406h33m11.905025799s) [noise=true , address=45.91.116.6:4973 , piece_id=AFZWKLQBUKO3CSBCLTFWQ7IKLDYVFWK6OQ6H2W4UDYQWTULV7LEQ, seg_pos={0 0}, upload=356939]: Done uploading piece. err: <nil> Dec 22 14:02:12.067209510 (406h33m11.938292575s) [noise=true , address=45.91.116.11:4117 , piece_id=6VPLAWTOFZGIO52L5V6QO3TXN4DZI65XAAYP674ZYQR3PHZGI7ZQ, seg_pos={0 0}, upload=356939]: Done uploading piece. err: <nil> Dec 22 14:02:12.145632713 (406h33m12.016715758s) [noise=true , address=45.91.116.5:4299 , piece_id=RDYBDEMBZ3BJ3OYNTGO3QXEJCMTVAX4OIOZX43U3AM4V4AZPYYGQ, seg_pos={0 0}, upload=356939]: Done uploading piece. err: <nil> Dec 22 14:02:12.209033870 (406h33m12.080116905s) [noise=true , address=77.90.31.19:4842 , piece_id=TXDC477SJ73J7SGLRSKV5VJ2P76GPJ6V5D3SPY5FA2YMWARILDXQ, seg_pos={0 0}, upload=356939]: Done uploading piece. err: <nil> Dec 22 14:02:12.209055591 (406h33m12.080138636s) [seg_pos={0 0}, upload=356939]: Segment reached optimal threshold of 13 pieces. Dec 22 14:02:12.209243389 (406h33m12.080326424s) [noise=true , address=45.91.116.8:4048 , piece_id=EGUB527VKRNI3Z7RMOKKLKXAK2VBXOGEIVJ34JIMO3AUKFDCC2NA, seg_pos={0 0}, upload=356939]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 22 14:02:12.209249490 (406h33m12.080332525s) [noise=true , address=45.91.116.8:4048 , piece_id=EGUB527VKRNI3Z7RMOKKLKXAK2VBXOGEIVJ34JIMO3AUKFDCC2NA, seg_pos={0 0}, upload=356939]: Long tail cancellation due to optimal threshold Dec 22 14:02:12.209432088 (406h33m12.080515123s) [noise=true , address=145.79.227.39:4013 , piece_id=BRVZ7WJRFCK7ZXEWF6TB4DZCKNZGB4IH5L44T3LKNVIXOL7PEUPA, seg_pos={0 0}, upload=356939]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b): protocol: expected piece hash; context canceled; EOF Dec 22 14:02:12.209449672 (406h33m12.080532717s) [noise=true , address=145.79.227.39:4013 , piece_id=BRVZ7WJRFCK7ZXEWF6TB4DZCKNZGB4IH5L44T3LKNVIXOL7PEUPA, seg_pos={0 0}, upload=356939]: Long tail cancellation due to optimal threshold Dec 22 14:02:12.209469569 (406h33m12.080552604s) [noise=true , address=45.91.116.7:4009 , piece_id=DZGZKBYFKLID3CEBDK24SN6WA7BUMIZRZWEGR4UM77SGLRPZUFEA, seg_pos={0 0}, upload=356939]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:49184->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:49184->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49184->45.91.116.7:4009: use of closed network connection Dec 22 14:02:12.209478777 (406h33m12.080561822s) [noise=true , address=45.91.116.7:4009 , piece_id=DZGZKBYFKLID3CEBDK24SN6WA7BUMIZRZWEGR4UM77SGLRPZUFEA, seg_pos={0 0}, upload=356939]: Long tail cancellation due to optimal threshold Dec 22 14:02:12.209497322 (406h33m12.080580357s) [seg_pos={0 0}, upload=356939]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 14:02:12.209586322 (406h33m12.080669367s) [upload=356939]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 14:02:12.209591762 (406h33m12.080674797s) [upload=356939]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 14:02:12.500528820 (406h33m12.371611875s) [upload=356939]: Done uploading segments... Dec 22 14:02:12.500556723 (406h33m12.371639768s) [upload=356939]: Upload finished. err: <nil> Duration: 5.794970469s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [......................X....X.X....XX..X..............X.....X] duration: 5.001466663s trace: https://jaeger.dev.storj.io/trace/669433dfad21eff6 FAIL (2025-12-22 14:13:02 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001490979s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 5.656232517s trace: https://jaeger.dev.storj.io/trace/538cdd5d04469e38 FAIL (2025-12-22 14:12:57 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.656269737s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 1.421773898s trace: https://jaeger.dev.storj.io/trace/1e7f77260788e649 FAIL (2025-12-22 14:12:44 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.421801381s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 1.494300472s trace: https://jaeger.dev.storj.io/trace/1e7f77260788e649 FAIL (2025-12-22 14:12:12 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.494318837s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [.......................X....X.X....XX..X..............X.....] duration: 3.749043948s trace: https://jaeger.dev.storj.io/trace/61b2ab80bf876574 FAIL (2025-12-22 14:12:08 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.749097249s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [X........X............X..................X...X..............] duration: 5.948809876s trace: https://jaeger.dev.storj.io/trace/5192c8b80a992813 FAIL (2025-12-22 14:12:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: 6VPLAWTOFZGIO52L5V6QO3TXN4DZI65XAAYP674ZYQR3PHZGI7ZQ) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: AFZWKLQBUKO3CSBCLTFWQ7IKLDYVFWK6OQ6H2W4UDYQWTULV7LEQ) context deadline exceeded ======LOGS====== Dec 22 14:02:09.540319730 (406h33m9.411402765s) [upload=356939]: Starting upload Dec 22 14:02:09.540380797 (406h33m9.411463852s) [upload=356939]: Done starting upload Dec 22 14:02:09.540611786 (406h33m9.411694831s) [upload=356939]: Uploading segments... Dec 22 14:02:09.540757183 (406h33m9.411840228s) [upload=356939]: Got next segment. Inline: false Dec 22 14:02:09.540808471 (406h33m9.411891516s) [upload=356939]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 14:02:09.540816275 (406h33m9.41189932s) [upload=356939]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 14:02:09.787889539 (406h33m9.658972584s) [seg_pos={0 0}, upload=356939]: Begin upload segment... Dec 22 14:02:09.787961536 (406h33m9.659044571s) [seg_pos={0 0}, upload=356939]: Done begin upload segment. Dec 22 14:02:09.788026600 (406h33m9.659109645s) [noise=true , address=145.79.227.47:4266 , piece_id=GWA6QDHSVQPD7JN7FXRE53RQIUYC43HRMXN6HBZNJCNKCTVQIHDA, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:09.788051907 (406h33m9.659134973s) [noise=true , address=45.91.116.7:4009 , piece_id=DZGZKBYFKLID3CEBDK24SN6WA7BUMIZRZWEGR4UM77SGLRPZUFEA, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:09.788912206 (406h33m9.659995251s) [noise=true , address=77.90.31.16:4736 , piece_id=BPWDDWM5I5PPWTVLQS5UYVCZMUALMEGAZA5ID3AZFTFH22HKX3QA, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:09.789394484 (406h33m9.660477529s) [noise=true , address=145.79.227.43:4157 , piece_id=RJP5KBDP5I434PDQVDR6E6YEDBS5QH33GBIPSA5ETEELZRAFQTXQ, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:09.789783124 (406h33m9.660866169s) [noise=true , address=45.91.116.8:4048 , piece_id=EGUB527VKRNI3Z7RMOKKLKXAK2VBXOGEIVJ34JIMO3AUKFDCC2NA, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:09.789923622 (406h33m9.661006667s) [noise=true , address=145.79.227.50:4375 , piece_id=6IAOXZUAF6Q7MBTQKLMDMELI4Y7LICBFDZTZBWZAETSRTYFJ4JSQ, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:09.790372917 (406h33m9.661455952s) [noise=true , address=145.79.227.39:4013 , piece_id=BRVZ7WJRFCK7ZXEWF6TB4DZCKNZGB4IH5L44T3LKNVIXOL7PEUPA, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:09.790398216 (406h33m9.661481251s) [noise=true , address=145.79.227.46:4603 , piece_id=C22WFSY2CAFQM6SHZBC3BWAK56UI5O2YFXX3TFYCUL7F2SKQ5UDQ, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:09.790750667 (406h33m9.661833832s) [noise=true , address=145.79.227.28:4726 , piece_id=ADJKKC7WYM7VG4DOD6M3KPLY7CHWBVHGVP7WYULTMKO7247SZXYA, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:09.790810932 (406h33m9.661893977s) [noise=true , address=145.79.227.36:4518 , piece_id=ZDGONTP3IPQWJS7WZVAOUFJQWOP7VYPSMRXEBOUGMRWRGZDUCS6A, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:09.791152573 (406h33m9.662235608s) [noise=true , address=77.90.31.19:4842 , piece_id=TXDC477SJ73J7SGLRSKV5VJ2P76GPJ6V5D3SPY5FA2YMWARILDXQ, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:09.791214379 (406h33m9.662297425s) [noise=true , address=145.79.227.29:4942 , piece_id=F3JXRWHD4HBMDK3SQMF5BLWUHYNZYWGYEARIB22IWDODLICKO7QA, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:09.791438928 (406h33m9.662521963s) [noise=true , address=45.91.116.11:4117 , piece_id=6VPLAWTOFZGIO52L5V6QO3TXN4DZI65XAAYP674ZYQR3PHZGI7ZQ, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:09.791655159 (406h33m9.662738204s) [noise=true , address=145.79.227.48:4303 , piece_id=4LVJNSYFP3JY4VMQ3MYKB34EXSNHWYP55DJVANGNCN4FE5HGY4UA, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:09.792046454 (406h33m9.663129489s) [noise=true , address=45.91.116.5:4299 , piece_id=RDYBDEMBZ3BJ3OYNTGO3QXEJCMTVAX4OIOZX43U3AM4V4AZPYYGQ, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:09.793687288 (406h33m9.664770333s) [noise=true , address=45.91.116.6:4973 , piece_id=AFZWKLQBUKO3CSBCLTFWQ7IKLDYVFWK6OQ6H2W4UDYQWTULV7LEQ, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:11.517644048 (406h33m11.388727093s) [upload=356939]: Next returned nil segment Dec 22 14:02:11.517663625 (406h33m11.38874667s) [upload=356939]: Waiting for error group managing segments... Dec 22 14:02:11.806697598 (406h33m11.677780643s) [noise=true , address=145.79.227.50:4375 , piece_id=6IAOXZUAF6Q7MBTQKLMDMELI4Y7LICBFDZTZBWZAETSRTYFJ4JSQ, seg_pos={0 0}, upload=356939]: Done uploading piece. err: <nil> Dec 22 14:02:11.807456193 (406h33m11.678539238s) [noise=true , address=145.79.227.29:4942 , piece_id=F3JXRWHD4HBMDK3SQMF5BLWUHYNZYWGYEARIB22IWDODLICKO7QA, seg_pos={0 0}, upload=356939]: Done uploading piece. err: <nil> Dec 22 14:02:11.808354183 (406h33m11.679437228s) [noise=true , address=145.79.227.43:4157 , piece_id=RJP5KBDP5I434PDQVDR6E6YEDBS5QH33GBIPSA5ETEELZRAFQTXQ, seg_pos={0 0}, upload=356939]: Done uploading piece. err: <nil> Dec 22 14:02:11.808910823 (406h33m11.679993868s) [noise=true , address=145.79.227.36:4518 , piece_id=ZDGONTP3IPQWJS7WZVAOUFJQWOP7VYPSMRXEBOUGMRWRGZDUCS6A, seg_pos={0 0}, upload=356939]: Done uploading piece. err: <nil> Dec 22 14:02:11.812084746 (406h33m11.683167791s) [noise=true , address=145.79.227.48:4303 , piece_id=4LVJNSYFP3JY4VMQ3MYKB34EXSNHWYP55DJVANGNCN4FE5HGY4UA, seg_pos={0 0}, upload=356939]: Done uploading piece. err: <nil> Dec 22 14:02:11.827226038 (406h33m11.698309073s) [noise=true , address=145.79.227.28:4726 , piece_id=ADJKKC7WYM7VG4DOD6M3KPLY7CHWBVHGVP7WYULTMKO7247SZXYA, seg_pos={0 0}, upload=356939]: Done uploading piece. err: <nil> Dec 22 14:02:11.924134049 (406h33m11.795217084s) [noise=true , address=145.79.227.46:4603 , piece_id=C22WFSY2CAFQM6SHZBC3BWAK56UI5O2YFXX3TFYCUL7F2SKQ5UDQ, seg_pos={0 0}, upload=356939]: Done uploading piece. err: <nil> Dec 22 14:02:11.937124565 (406h33m11.80820761s) [noise=true , address=145.79.227.47:4266 , piece_id=GWA6QDHSVQPD7JN7FXRE53RQIUYC43HRMXN6HBZNJCNKCTVQIHDA, seg_pos={0 0}, upload=356939]: Done uploading piece. err: <nil> Dec 22 14:02:12.022864229 (406h33m11.893947274s) [noise=true , address=77.90.31.16:4736 , piece_id=BPWDDWM5I5PPWTVLQS5UYVCZMUALMEGAZA5ID3AZFTFH22HKX3QA, seg_pos={0 0}, upload=356939]: Done uploading piece. err: <nil> Dec 22 14:02:12.033942754 (406h33m11.905025799s) [noise=true , address=45.91.116.6:4973 , piece_id=AFZWKLQBUKO3CSBCLTFWQ7IKLDYVFWK6OQ6H2W4UDYQWTULV7LEQ, seg_pos={0 0}, upload=356939]: Done uploading piece. err: <nil> Dec 22 14:02:12.067209510 (406h33m11.938292575s) [noise=true , address=45.91.116.11:4117 , piece_id=6VPLAWTOFZGIO52L5V6QO3TXN4DZI65XAAYP674ZYQR3PHZGI7ZQ, seg_pos={0 0}, upload=356939]: Done uploading piece. err: <nil> Dec 22 14:02:12.145632713 (406h33m12.016715758s) [noise=true , address=45.91.116.5:4299 , piece_id=RDYBDEMBZ3BJ3OYNTGO3QXEJCMTVAX4OIOZX43U3AM4V4AZPYYGQ, seg_pos={0 0}, upload=356939]: Done uploading piece. err: <nil> Dec 22 14:02:12.209033870 (406h33m12.080116905s) [noise=true , address=77.90.31.19:4842 , piece_id=TXDC477SJ73J7SGLRSKV5VJ2P76GPJ6V5D3SPY5FA2YMWARILDXQ, seg_pos={0 0}, upload=356939]: Done uploading piece. err: <nil> Dec 22 14:02:12.209055591 (406h33m12.080138636s) [seg_pos={0 0}, upload=356939]: Segment reached optimal threshold of 13 pieces. Dec 22 14:02:12.209243389 (406h33m12.080326424s) [noise=true , address=45.91.116.8:4048 , piece_id=EGUB527VKRNI3Z7RMOKKLKXAK2VBXOGEIVJ34JIMO3AUKFDCC2NA, seg_pos={0 0}, upload=356939]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 22 14:02:12.209249490 (406h33m12.080332525s) [noise=true , address=45.91.116.8:4048 , piece_id=EGUB527VKRNI3Z7RMOKKLKXAK2VBXOGEIVJ34JIMO3AUKFDCC2NA, seg_pos={0 0}, upload=356939]: Long tail cancellation due to optimal threshold Dec 22 14:02:12.209432088 (406h33m12.080515123s) [noise=true , address=145.79.227.39:4013 , piece_id=BRVZ7WJRFCK7ZXEWF6TB4DZCKNZGB4IH5L44T3LKNVIXOL7PEUPA, seg_pos={0 0}, upload=356939]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b): protocol: expected piece hash; context canceled; EOF Dec 22 14:02:12.209449672 (406h33m12.080532717s) [noise=true , address=145.79.227.39:4013 , piece_id=BRVZ7WJRFCK7ZXEWF6TB4DZCKNZGB4IH5L44T3LKNVIXOL7PEUPA, seg_pos={0 0}, upload=356939]: Long tail cancellation due to optimal threshold Dec 22 14:02:12.209469569 (406h33m12.080552604s) [noise=true , address=45.91.116.7:4009 , piece_id=DZGZKBYFKLID3CEBDK24SN6WA7BUMIZRZWEGR4UM77SGLRPZUFEA, seg_pos={0 0}, upload=356939]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:49184->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:49184->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49184->45.91.116.7:4009: use of closed network connection Dec 22 14:02:12.209478777 (406h33m12.080561822s) [noise=true , address=45.91.116.7:4009 , piece_id=DZGZKBYFKLID3CEBDK24SN6WA7BUMIZRZWEGR4UM77SGLRPZUFEA, seg_pos={0 0}, upload=356939]: Long tail cancellation due to optimal threshold Dec 22 14:02:12.209497322 (406h33m12.080580357s) [seg_pos={0 0}, upload=356939]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 14:02:12.209586322 (406h33m12.080669367s) [upload=356939]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 14:02:12.209591762 (406h33m12.080674797s) [upload=356939]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 14:02:12.500528820 (406h33m12.371611875s) [upload=356939]: Done uploading segments... Dec 22 14:02:12.500556723 (406h33m12.371639768s) [upload=356939]: Upload finished. err: <nil> Duration: 5.948840295s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 4.943323144s trace: https://jaeger.dev.storj.io/trace/538cdd5d04469e38 FAIL (2025-12-22 14:11:55 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.943351908s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.X........X............X..................X...X.............] duration: 6.107254445s trace: https://jaeger.dev.storj.io/trace/5192c8b80a992813 FAIL (2025-12-22 14:11:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: 6VPLAWTOFZGIO52L5V6QO3TXN4DZI65XAAYP674ZYQR3PHZGI7ZQ) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: AFZWKLQBUKO3CSBCLTFWQ7IKLDYVFWK6OQ6H2W4UDYQWTULV7LEQ) context deadline exceeded ======LOGS====== Dec 22 14:02:09.540319730 (406h33m9.411402765s) [upload=356939]: Starting upload Dec 22 14:02:09.540380797 (406h33m9.411463852s) [upload=356939]: Done starting upload Dec 22 14:02:09.540611786 (406h33m9.411694831s) [upload=356939]: Uploading segments... Dec 22 14:02:09.540757183 (406h33m9.411840228s) [upload=356939]: Got next segment. Inline: false Dec 22 14:02:09.540808471 (406h33m9.411891516s) [upload=356939]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 14:02:09.540816275 (406h33m9.41189932s) [upload=356939]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 14:02:09.787889539 (406h33m9.658972584s) [seg_pos={0 0}, upload=356939]: Begin upload segment... Dec 22 14:02:09.787961536 (406h33m9.659044571s) [seg_pos={0 0}, upload=356939]: Done begin upload segment. Dec 22 14:02:09.788026600 (406h33m9.659109645s) [noise=true , address=145.79.227.47:4266 , piece_id=GWA6QDHSVQPD7JN7FXRE53RQIUYC43HRMXN6HBZNJCNKCTVQIHDA, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:09.788051907 (406h33m9.659134973s) [noise=true , address=45.91.116.7:4009 , piece_id=DZGZKBYFKLID3CEBDK24SN6WA7BUMIZRZWEGR4UM77SGLRPZUFEA, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:09.788912206 (406h33m9.659995251s) [noise=true , address=77.90.31.16:4736 , piece_id=BPWDDWM5I5PPWTVLQS5UYVCZMUALMEGAZA5ID3AZFTFH22HKX3QA, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:09.789394484 (406h33m9.660477529s) [noise=true , address=145.79.227.43:4157 , piece_id=RJP5KBDP5I434PDQVDR6E6YEDBS5QH33GBIPSA5ETEELZRAFQTXQ, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:09.789783124 (406h33m9.660866169s) [noise=true , address=45.91.116.8:4048 , piece_id=EGUB527VKRNI3Z7RMOKKLKXAK2VBXOGEIVJ34JIMO3AUKFDCC2NA, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:09.789923622 (406h33m9.661006667s) [noise=true , address=145.79.227.50:4375 , piece_id=6IAOXZUAF6Q7MBTQKLMDMELI4Y7LICBFDZTZBWZAETSRTYFJ4JSQ, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:09.790372917 (406h33m9.661455952s) [noise=true , address=145.79.227.39:4013 , piece_id=BRVZ7WJRFCK7ZXEWF6TB4DZCKNZGB4IH5L44T3LKNVIXOL7PEUPA, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:09.790398216 (406h33m9.661481251s) [noise=true , address=145.79.227.46:4603 , piece_id=C22WFSY2CAFQM6SHZBC3BWAK56UI5O2YFXX3TFYCUL7F2SKQ5UDQ, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:09.790750667 (406h33m9.661833832s) [noise=true , address=145.79.227.28:4726 , piece_id=ADJKKC7WYM7VG4DOD6M3KPLY7CHWBVHGVP7WYULTMKO7247SZXYA, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:09.790810932 (406h33m9.661893977s) [noise=true , address=145.79.227.36:4518 , piece_id=ZDGONTP3IPQWJS7WZVAOUFJQWOP7VYPSMRXEBOUGMRWRGZDUCS6A, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:09.791152573 (406h33m9.662235608s) [noise=true , address=77.90.31.19:4842 , piece_id=TXDC477SJ73J7SGLRSKV5VJ2P76GPJ6V5D3SPY5FA2YMWARILDXQ, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:09.791214379 (406h33m9.662297425s) [noise=true , address=145.79.227.29:4942 , piece_id=F3JXRWHD4HBMDK3SQMF5BLWUHYNZYWGYEARIB22IWDODLICKO7QA, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:09.791438928 (406h33m9.662521963s) [noise=true , address=45.91.116.11:4117 , piece_id=6VPLAWTOFZGIO52L5V6QO3TXN4DZI65XAAYP674ZYQR3PHZGI7ZQ, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:09.791655159 (406h33m9.662738204s) [noise=true , address=145.79.227.48:4303 , piece_id=4LVJNSYFP3JY4VMQ3MYKB34EXSNHWYP55DJVANGNCN4FE5HGY4UA, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:09.792046454 (406h33m9.663129489s) [noise=true , address=45.91.116.5:4299 , piece_id=RDYBDEMBZ3BJ3OYNTGO3QXEJCMTVAX4OIOZX43U3AM4V4AZPYYGQ, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:09.793687288 (406h33m9.664770333s) [noise=true , address=45.91.116.6:4973 , piece_id=AFZWKLQBUKO3CSBCLTFWQ7IKLDYVFWK6OQ6H2W4UDYQWTULV7LEQ, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:11.517644048 (406h33m11.388727093s) [upload=356939]: Next returned nil segment Dec 22 14:02:11.517663625 (406h33m11.38874667s) [upload=356939]: Waiting for error group managing segments... Dec 22 14:02:11.806697598 (406h33m11.677780643s) [noise=true , address=145.79.227.50:4375 , piece_id=6IAOXZUAF6Q7MBTQKLMDMELI4Y7LICBFDZTZBWZAETSRTYFJ4JSQ, seg_pos={0 0}, upload=356939]: Done uploading piece. err: <nil> Dec 22 14:02:11.807456193 (406h33m11.678539238s) [noise=true , address=145.79.227.29:4942 , piece_id=F3JXRWHD4HBMDK3SQMF5BLWUHYNZYWGYEARIB22IWDODLICKO7QA, seg_pos={0 0}, upload=356939]: Done uploading piece. err: <nil> Dec 22 14:02:11.808354183 (406h33m11.679437228s) [noise=true , address=145.79.227.43:4157 , piece_id=RJP5KBDP5I434PDQVDR6E6YEDBS5QH33GBIPSA5ETEELZRAFQTXQ, seg_pos={0 0}, upload=356939]: Done uploading piece. err: <nil> Dec 22 14:02:11.808910823 (406h33m11.679993868s) [noise=true , address=145.79.227.36:4518 , piece_id=ZDGONTP3IPQWJS7WZVAOUFJQWOP7VYPSMRXEBOUGMRWRGZDUCS6A, seg_pos={0 0}, upload=356939]: Done uploading piece. err: <nil> Dec 22 14:02:11.812084746 (406h33m11.683167791s) [noise=true , address=145.79.227.48:4303 , piece_id=4LVJNSYFP3JY4VMQ3MYKB34EXSNHWYP55DJVANGNCN4FE5HGY4UA, seg_pos={0 0}, upload=356939]: Done uploading piece. err: <nil> Dec 22 14:02:11.827226038 (406h33m11.698309073s) [noise=true , address=145.79.227.28:4726 , piece_id=ADJKKC7WYM7VG4DOD6M3KPLY7CHWBVHGVP7WYULTMKO7247SZXYA, seg_pos={0 0}, upload=356939]: Done uploading piece. err: <nil> Dec 22 14:02:11.924134049 (406h33m11.795217084s) [noise=true , address=145.79.227.46:4603 , piece_id=C22WFSY2CAFQM6SHZBC3BWAK56UI5O2YFXX3TFYCUL7F2SKQ5UDQ, seg_pos={0 0}, upload=356939]: Done uploading piece. err: <nil> Dec 22 14:02:11.937124565 (406h33m11.80820761s) [noise=true , address=145.79.227.47:4266 , piece_id=GWA6QDHSVQPD7JN7FXRE53RQIUYC43HRMXN6HBZNJCNKCTVQIHDA, seg_pos={0 0}, upload=356939]: Done uploading piece. err: <nil> Dec 22 14:02:12.022864229 (406h33m11.893947274s) [noise=true , address=77.90.31.16:4736 , piece_id=BPWDDWM5I5PPWTVLQS5UYVCZMUALMEGAZA5ID3AZFTFH22HKX3QA, seg_pos={0 0}, upload=356939]: Done uploading piece. err: <nil> Dec 22 14:02:12.033942754 (406h33m11.905025799s) [noise=true , address=45.91.116.6:4973 , piece_id=AFZWKLQBUKO3CSBCLTFWQ7IKLDYVFWK6OQ6H2W4UDYQWTULV7LEQ, seg_pos={0 0}, upload=356939]: Done uploading piece. err: <nil> Dec 22 14:02:12.067209510 (406h33m11.938292575s) [noise=true , address=45.91.116.11:4117 , piece_id=6VPLAWTOFZGIO52L5V6QO3TXN4DZI65XAAYP674ZYQR3PHZGI7ZQ, seg_pos={0 0}, upload=356939]: Done uploading piece. err: <nil> Dec 22 14:02:12.145632713 (406h33m12.016715758s) [noise=true , address=45.91.116.5:4299 , piece_id=RDYBDEMBZ3BJ3OYNTGO3QXEJCMTVAX4OIOZX43U3AM4V4AZPYYGQ, seg_pos={0 0}, upload=356939]: Done uploading piece. err: <nil> Dec 22 14:02:12.209033870 (406h33m12.080116905s) [noise=true , address=77.90.31.19:4842 , piece_id=TXDC477SJ73J7SGLRSKV5VJ2P76GPJ6V5D3SPY5FA2YMWARILDXQ, seg_pos={0 0}, upload=356939]: Done uploading piece. err: <nil> Dec 22 14:02:12.209055591 (406h33m12.080138636s) [seg_pos={0 0}, upload=356939]: Segment reached optimal threshold of 13 pieces. Dec 22 14:02:12.209243389 (406h33m12.080326424s) [noise=true , address=45.91.116.8:4048 , piece_id=EGUB527VKRNI3Z7RMOKKLKXAK2VBXOGEIVJ34JIMO3AUKFDCC2NA, seg_pos={0 0}, upload=356939]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 22 14:02:12.209249490 (406h33m12.080332525s) [noise=true , address=45.91.116.8:4048 , piece_id=EGUB527VKRNI3Z7RMOKKLKXAK2VBXOGEIVJ34JIMO3AUKFDCC2NA, seg_pos={0 0}, upload=356939]: Long tail cancellation due to optimal threshold Dec 22 14:02:12.209432088 (406h33m12.080515123s) [noise=true , address=145.79.227.39:4013 , piece_id=BRVZ7WJRFCK7ZXEWF6TB4DZCKNZGB4IH5L44T3LKNVIXOL7PEUPA, seg_pos={0 0}, upload=356939]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b): protocol: expected piece hash; context canceled; EOF Dec 22 14:02:12.209449672 (406h33m12.080532717s) [noise=true , address=145.79.227.39:4013 , piece_id=BRVZ7WJRFCK7ZXEWF6TB4DZCKNZGB4IH5L44T3LKNVIXOL7PEUPA, seg_pos={0 0}, upload=356939]: Long tail cancellation due to optimal threshold Dec 22 14:02:12.209469569 (406h33m12.080552604s) [noise=true , address=45.91.116.7:4009 , piece_id=DZGZKBYFKLID3CEBDK24SN6WA7BUMIZRZWEGR4UM77SGLRPZUFEA, seg_pos={0 0}, upload=356939]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:49184->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:49184->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49184->45.91.116.7:4009: use of closed network connection Dec 22 14:02:12.209478777 (406h33m12.080561822s) [noise=true , address=45.91.116.7:4009 , piece_id=DZGZKBYFKLID3CEBDK24SN6WA7BUMIZRZWEGR4UM77SGLRPZUFEA, seg_pos={0 0}, upload=356939]: Long tail cancellation due to optimal threshold Dec 22 14:02:12.209497322 (406h33m12.080580357s) [seg_pos={0 0}, upload=356939]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 14:02:12.209586322 (406h33m12.080669367s) [upload=356939]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 14:02:12.209591762 (406h33m12.080674797s) [upload=356939]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 14:02:12.500528820 (406h33m12.371611875s) [upload=356939]: Done uploading segments... Dec 22 14:02:12.500556723 (406h33m12.371639768s) [upload=356939]: Upload finished. err: <nil> Duration: 6.107286727s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [........................X....X.X....XX..X..............X....] duration: 3.922814068s trace: https://jaeger.dev.storj.io/trace/61b2ab80bf876574 FAIL (2025-12-22 14:11:23 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.922828316s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 1.629310986s trace: https://jaeger.dev.storj.io/trace/1e7f77260788e649 FAIL (2025-12-22 14:11:12 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.629340383s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [.........................X....X.X....XX..X..............X...] duration: 3.558572284s trace: https://jaeger.dev.storj.io/trace/61b2ab80bf876574 FAIL (2025-12-22 14:10:55 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.558601208s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 5.246079781s trace: https://jaeger.dev.storj.io/trace/538cdd5d04469e38 FAIL (2025-12-22 14:10:43 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.246099217s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 47.398237021s trace: https://jaeger.dev.storj.io/trace/538cdd5d04469e38 FAIL (2025-12-22 14:10:38 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 47.398259923s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.001266681s trace: https://jaeger.dev.storj.io/trace/1e7f77260788e649 FAIL (2025-12-22 14:10:27 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 30.001291227s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..X........X............X..................X...X............] duration: 5.831919276s trace: https://jaeger.dev.storj.io/trace/5192c8b80a992813 FAIL (2025-12-22 14:10:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: 6VPLAWTOFZGIO52L5V6QO3TXN4DZI65XAAYP674ZYQR3PHZGI7ZQ) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: AFZWKLQBUKO3CSBCLTFWQ7IKLDYVFWK6OQ6H2W4UDYQWTULV7LEQ) context deadline exceeded ======LOGS====== Dec 22 14:02:09.540319730 (406h33m9.411402765s) [upload=356939]: Starting upload Dec 22 14:02:09.540380797 (406h33m9.411463852s) [upload=356939]: Done starting upload Dec 22 14:02:09.540611786 (406h33m9.411694831s) [upload=356939]: Uploading segments... Dec 22 14:02:09.540757183 (406h33m9.411840228s) [upload=356939]: Got next segment. Inline: false Dec 22 14:02:09.540808471 (406h33m9.411891516s) [upload=356939]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 14:02:09.540816275 (406h33m9.41189932s) [upload=356939]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 14:02:09.787889539 (406h33m9.658972584s) [seg_pos={0 0}, upload=356939]: Begin upload segment... Dec 22 14:02:09.787961536 (406h33m9.659044571s) [seg_pos={0 0}, upload=356939]: Done begin upload segment. Dec 22 14:02:09.788026600 (406h33m9.659109645s) [noise=true , address=145.79.227.47:4266 , piece_id=GWA6QDHSVQPD7JN7FXRE53RQIUYC43HRMXN6HBZNJCNKCTVQIHDA, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:09.788051907 (406h33m9.659134973s) [noise=true , address=45.91.116.7:4009 , piece_id=DZGZKBYFKLID3CEBDK24SN6WA7BUMIZRZWEGR4UM77SGLRPZUFEA, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:09.788912206 (406h33m9.659995251s) [noise=true , address=77.90.31.16:4736 , piece_id=BPWDDWM5I5PPWTVLQS5UYVCZMUALMEGAZA5ID3AZFTFH22HKX3QA, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:09.789394484 (406h33m9.660477529s) [noise=true , address=145.79.227.43:4157 , piece_id=RJP5KBDP5I434PDQVDR6E6YEDBS5QH33GBIPSA5ETEELZRAFQTXQ, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:09.789783124 (406h33m9.660866169s) [noise=true , address=45.91.116.8:4048 , piece_id=EGUB527VKRNI3Z7RMOKKLKXAK2VBXOGEIVJ34JIMO3AUKFDCC2NA, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:09.789923622 (406h33m9.661006667s) [noise=true , address=145.79.227.50:4375 , piece_id=6IAOXZUAF6Q7MBTQKLMDMELI4Y7LICBFDZTZBWZAETSRTYFJ4JSQ, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:09.790372917 (406h33m9.661455952s) [noise=true , address=145.79.227.39:4013 , piece_id=BRVZ7WJRFCK7ZXEWF6TB4DZCKNZGB4IH5L44T3LKNVIXOL7PEUPA, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:09.790398216 (406h33m9.661481251s) [noise=true , address=145.79.227.46:4603 , piece_id=C22WFSY2CAFQM6SHZBC3BWAK56UI5O2YFXX3TFYCUL7F2SKQ5UDQ, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:09.790750667 (406h33m9.661833832s) [noise=true , address=145.79.227.28:4726 , piece_id=ADJKKC7WYM7VG4DOD6M3KPLY7CHWBVHGVP7WYULTMKO7247SZXYA, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:09.790810932 (406h33m9.661893977s) [noise=true , address=145.79.227.36:4518 , piece_id=ZDGONTP3IPQWJS7WZVAOUFJQWOP7VYPSMRXEBOUGMRWRGZDUCS6A, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:09.791152573 (406h33m9.662235608s) [noise=true , address=77.90.31.19:4842 , piece_id=TXDC477SJ73J7SGLRSKV5VJ2P76GPJ6V5D3SPY5FA2YMWARILDXQ, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:09.791214379 (406h33m9.662297425s) [noise=true , address=145.79.227.29:4942 , piece_id=F3JXRWHD4HBMDK3SQMF5BLWUHYNZYWGYEARIB22IWDODLICKO7QA, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:09.791438928 (406h33m9.662521963s) [noise=true , address=45.91.116.11:4117 , piece_id=6VPLAWTOFZGIO52L5V6QO3TXN4DZI65XAAYP674ZYQR3PHZGI7ZQ, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:09.791655159 (406h33m9.662738204s) [noise=true , address=145.79.227.48:4303 , piece_id=4LVJNSYFP3JY4VMQ3MYKB34EXSNHWYP55DJVANGNCN4FE5HGY4UA, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:09.792046454 (406h33m9.663129489s) [noise=true , address=45.91.116.5:4299 , piece_id=RDYBDEMBZ3BJ3OYNTGO3QXEJCMTVAX4OIOZX43U3AM4V4AZPYYGQ, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:09.793687288 (406h33m9.664770333s) [noise=true , address=45.91.116.6:4973 , piece_id=AFZWKLQBUKO3CSBCLTFWQ7IKLDYVFWK6OQ6H2W4UDYQWTULV7LEQ, seg_pos={0 0}, upload=356939]: Uploading piece... Dec 22 14:02:11.517644048 (406h33m11.388727093s) [upload=356939]: Next returned nil segment Dec 22 14:02:11.517663625 (406h33m11.38874667s) [upload=356939]: Waiting for error group managing segments... Dec 22 14:02:11.806697598 (406h33m11.677780643s) [noise=true , address=145.79.227.50:4375 , piece_id=6IAOXZUAF6Q7MBTQKLMDMELI4Y7LICBFDZTZBWZAETSRTYFJ4JSQ, seg_pos={0 0}, upload=356939]: Done uploading piece. err: <nil> Dec 22 14:02:11.807456193 (406h33m11.678539238s) [noise=true , address=145.79.227.29:4942 , piece_id=F3JXRWHD4HBMDK3SQMF5BLWUHYNZYWGYEARIB22IWDODLICKO7QA, seg_pos={0 0}, upload=356939]: Done uploading piece. err: <nil> Dec 22 14:02:11.808354183 (406h33m11.679437228s) [noise=true , address=145.79.227.43:4157 , piece_id=RJP5KBDP5I434PDQVDR6E6YEDBS5QH33GBIPSA5ETEELZRAFQTXQ, seg_pos={0 0}, upload=356939]: Done uploading piece. err: <nil> Dec 22 14:02:11.808910823 (406h33m11.679993868s) [noise=true , address=145.79.227.36:4518 , piece_id=ZDGONTP3IPQWJS7WZVAOUFJQWOP7VYPSMRXEBOUGMRWRGZDUCS6A, seg_pos={0 0}, upload=356939]: Done uploading piece. err: <nil> Dec 22 14:02:11.812084746 (406h33m11.683167791s) [noise=true , address=145.79.227.48:4303 , piece_id=4LVJNSYFP3JY4VMQ3MYKB34EXSNHWYP55DJVANGNCN4FE5HGY4UA, seg_pos={0 0}, upload=356939]: Done uploading piece. err: <nil> Dec 22 14:02:11.827226038 (406h33m11.698309073s) [noise=true , address=145.79.227.28:4726 , piece_id=ADJKKC7WYM7VG4DOD6M3KPLY7CHWBVHGVP7WYULTMKO7247SZXYA, seg_pos={0 0}, upload=356939]: Done uploading piece. err: <nil> Dec 22 14:02:11.924134049 (406h33m11.795217084s) [noise=true , address=145.79.227.46:4603 , piece_id=C22WFSY2CAFQM6SHZBC3BWAK56UI5O2YFXX3TFYCUL7F2SKQ5UDQ, seg_pos={0 0}, upload=356939]: Done uploading piece. err: <nil> Dec 22 14:02:11.937124565 (406h33m11.80820761s) [noise=true , address=145.79.227.47:4266 , piece_id=GWA6QDHSVQPD7JN7FXRE53RQIUYC43HRMXN6HBZNJCNKCTVQIHDA, seg_pos={0 0}, upload=356939]: Done uploading piece. err: <nil> Dec 22 14:02:12.022864229 (406h33m11.893947274s) [noise=true , address=77.90.31.16:4736 , piece_id=BPWDDWM5I5PPWTVLQS5UYVCZMUALMEGAZA5ID3AZFTFH22HKX3QA, seg_pos={0 0}, upload=356939]: Done uploading piece. err: <nil> Dec 22 14:02:12.033942754 (406h33m11.905025799s) [noise=true , address=45.91.116.6:4973 , piece_id=AFZWKLQBUKO3CSBCLTFWQ7IKLDYVFWK6OQ6H2W4UDYQWTULV7LEQ, seg_pos={0 0}, upload=356939]: Done uploading piece. err: <nil> Dec 22 14:02:12.067209510 (406h33m11.938292575s) [noise=true , address=45.91.116.11:4117 , piece_id=6VPLAWTOFZGIO52L5V6QO3TXN4DZI65XAAYP674ZYQR3PHZGI7ZQ, seg_pos={0 0}, upload=356939]: Done uploading piece. err: <nil> Dec 22 14:02:12.145632713 (406h33m12.016715758s) [noise=true , address=45.91.116.5:4299 , piece_id=RDYBDEMBZ3BJ3OYNTGO3QXEJCMTVAX4OIOZX43U3AM4V4AZPYYGQ, seg_pos={0 0}, upload=356939]: Done uploading piece. err: <nil> Dec 22 14:02:12.209033870 (406h33m12.080116905s) [noise=true , address=77.90.31.19:4842 , piece_id=TXDC477SJ73J7SGLRSKV5VJ2P76GPJ6V5D3SPY5FA2YMWARILDXQ, seg_pos={0 0}, upload=356939]: Done uploading piece. err: <nil> Dec 22 14:02:12.209055591 (406h33m12.080138636s) [seg_pos={0 0}, upload=356939]: Segment reached optimal threshold of 13 pieces. Dec 22 14:02:12.209243389 (406h33m12.080326424s) [noise=true , address=45.91.116.8:4048 , piece_id=EGUB527VKRNI3Z7RMOKKLKXAK2VBXOGEIVJ34JIMO3AUKFDCC2NA, seg_pos={0 0}, upload=356939]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 22 14:02:12.209249490 (406h33m12.080332525s) [noise=true , address=45.91.116.8:4048 , piece_id=EGUB527VKRNI3Z7RMOKKLKXAK2VBXOGEIVJ34JIMO3AUKFDCC2NA, seg_pos={0 0}, upload=356939]: Long tail cancellation due to optimal threshold Dec 22 14:02:12.209432088 (406h33m12.080515123s) [noise=true , address=145.79.227.39:4013 , piece_id=BRVZ7WJRFCK7ZXEWF6TB4DZCKNZGB4IH5L44T3LKNVIXOL7PEUPA, seg_pos={0 0}, upload=356939]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b): protocol: expected piece hash; context canceled; EOF Dec 22 14:02:12.209449672 (406h33m12.080532717s) [noise=true , address=145.79.227.39:4013 , piece_id=BRVZ7WJRFCK7ZXEWF6TB4DZCKNZGB4IH5L44T3LKNVIXOL7PEUPA, seg_pos={0 0}, upload=356939]: Long tail cancellation due to optimal threshold Dec 22 14:02:12.209469569 (406h33m12.080552604s) [noise=true , address=45.91.116.7:4009 , piece_id=DZGZKBYFKLID3CEBDK24SN6WA7BUMIZRZWEGR4UM77SGLRPZUFEA, seg_pos={0 0}, upload=356939]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:49184->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:49184->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49184->45.91.116.7:4009: use of closed network connection Dec 22 14:02:12.209478777 (406h33m12.080561822s) [noise=true , address=45.91.116.7:4009 , piece_id=DZGZKBYFKLID3CEBDK24SN6WA7BUMIZRZWEGR4UM77SGLRPZUFEA, seg_pos={0 0}, upload=356939]: Long tail cancellation due to optimal threshold Dec 22 14:02:12.209497322 (406h33m12.080580357s) [seg_pos={0 0}, upload=356939]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 14:02:12.209586322 (406h33m12.080669367s) [upload=356939]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 14:02:12.209591762 (406h33m12.080674797s) [upload=356939]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 14:02:12.500528820 (406h33m12.371611875s) [upload=356939]: Done uploading segments... Dec 22 14:02:12.500556723 (406h33m12.371639768s) [upload=356939]: Upload finished. err: <nil> Duration: 5.831935989s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [..........................X....X.X....XX..X..............X..] duration: 3.693565758s trace: https://jaeger.dev.storj.io/trace/61b2ab80bf876574 FAIL (2025-12-22 14:10:19 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.693598029s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)