Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
53 out of 60 is failed (threshold=6) [XXXXX.XXXXXXXXXXXX.XXXXXXX.XX..XXXXXXXXXXXXXXXXXXXXX.X.XXXXX] duration:
5.003297627s trace:
https://jaeger.dev.storj.io/trace/22346b7e9c38450
FAIL (2025-12-28 14:16:25 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.003324457s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [...........X....................X...X........X.............X] duration:
30.001439633s trace:
https://jaeger.dev.storj.io/trace/35a5d4335e2cde85
FAIL (2025-12-28 14:16:18 UTC)
orbiter warning: context deadline exceeded
Duration: 30.001465471s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
20 out of 60 is failed (threshold=3) [.X.....X..X.X...XX......X......XXX.XX....XX......XX.XX.X.X..] duration:
5.912746898s trace:
https://jaeger.dev.storj.io/trace/5be1acce1690abbe
FAIL (2025-12-28 14:16:10 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 28 14:14:05.707801185 (550h45m5.57888423s) [upload=482879]: Starting upload
Dec 28 14:14:05.707886898 (550h45m5.578969944s) [upload=482879]: Done starting upload
Dec 28 14:14:05.707951642 (550h45m5.579034687s) [upload=482879]: Uploading segments...
Dec 28 14:14:05.708074515 (550h45m5.57915756s) [upload=482879]: Got next segment. Inline: false
Dec 28 14:14:05.708106106 (550h45m5.579189151s) [upload=482879]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 14:14:05.708110434 (550h45m5.579193479s) [upload=482879]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 14:14:06.000535780 (550h45m5.871618845s) [seg_pos={0 0}, upload=482879]: Begin upload segment...
Dec 28 14:14:06.000598910 (550h45m5.871681945s) [seg_pos={0 0}, upload=482879]: Done begin upload segment.
Dec 28 14:14:06.000636912 (550h45m5.871719957s) [noise=true , address=145.79.227.40:4051 , piece_id=3QYIHWAQR7G5I57JDXJ4SJ6A2LARGCAP6XK7CVE6K2WVR4XGXXHQ, seg_pos={0 0}, upload=482879]: Uploading piece...
Dec 28 14:14:06.000757032 (550h45m5.871840097s) [noise=true , address=145.79.227.13:4804 , piece_id=PGTARYC3LNMTR4TAGR74FPBM7RHMTHRLDW2QUW6ARA2SI26DHLZQ, seg_pos={0 0}, upload=482879]: Uploading piece...
Dec 28 14:14:06.001400927 (550h45m5.872483972s) [noise=true , address=45.91.116.9:4262 , piece_id=54C5OVTRQ2FAS4X2EEPNRRAKFEWDO5JQEDQ5HOCOTPJW54PUK2YA, seg_pos={0 0}, upload=482879]: Uploading piece...
Dec 28 14:14:06.001895939 (550h45m5.872978975s) [noise=true , address=145.79.227.47:4266 , piece_id=FUBWPNHK7SKGUJSKNMTHO3RGUCTABSL4QOASCW23J6GCJ3H5LDMQ, seg_pos={0 0}, upload=482879]: Uploading piece...
Dec 28 14:14:06.002437291 (550h45m5.873520336s) [noise=true , address=45.91.116.14:4190 , piece_id=565ADKSRT363WQJIB2WFNFLNPS765AVU7R72A5LMHGACOYZNBYDQ, seg_pos={0 0}, upload=482879]: Uploading piece...
Dec 28 14:14:06.002952352 (550h45m5.874035397s) [noise=true , address=45.91.116.6:4975 , piece_id=IRL2CQSLRGQJIP3BHV7HYBS56R7HGUDQPQBNNFXEE6Q7IUF2CXHA, seg_pos={0 0}, upload=482879]: Uploading piece...
Dec 28 14:14:06.002965867 (550h45m5.874048912s) [noise=true , address=145.79.227.42:4123 , piece_id=7QUYG56BEL7FSD2A6AHQNONXE4PWH3QEHZSBKEGYKXYRH5DKIP5Q, seg_pos={0 0}, upload=482879]: Uploading piece...
Dec 28 14:14:06.003387630 (550h45m5.874470685s) [noise=true , address=45.91.116.11:4120 , piece_id=MRG7VL7KTA3T5FEUFJW5ISMKDWWLUGHHBGGL7RMWRSON4N2UVZUA, seg_pos={0 0}, upload=482879]: Uploading piece...
Dec 28 14:14:06.003838088 (550h45m5.874921133s) [noise=true , address=45.91.116.12:4228 , piece_id=VXDU5ZAC3NCP65MDUZ5LSVEIGRQDJIJDP2TIUGXPHGLBLGBMMCWA, seg_pos={0 0}, upload=482879]: Uploading piece...
Dec 28 14:14:06.004305238 (550h45m5.875388303s) [noise=true , address=145.79.227.50:4373 , piece_id=RAXJ6LT3KF7DOQRSV7DH5IAPJ4WMWDLIPIOFFVQUWRU5BF4SERNQ, seg_pos={0 0}, upload=482879]: Uploading piece...
Dec 28 14:14:06.004804198 (550h45m5.875887233s) [noise=true , address=145.79.227.46:4603 , piece_id=TWVG57KDN3OV4DU3QD4ONWNEEYOCKYXYQ63QBQPEZS676HDAMZYA, seg_pos={0 0}, upload=482879]: Uploading piece...
Dec 28 14:14:06.005276297 (550h45m5.876359342s) [noise=true , address=145.79.227.12:4767 , piece_id=VPWI5VMVMUCTRSYXD3QSZ6OUETWIAJQLGZQ7MRSUL2Y475DNCK3Q, seg_pos={0 0}, upload=482879]: Uploading piece...
Dec 28 14:14:06.005723668 (550h45m5.876806713s) [noise=true , address=145.79.227.52:4446 , piece_id=L2LK33S7SEF7NTBUT3AX7XMTKPGPICTAXSS6MYA2OJ3DOZWRJJHA, seg_pos={0 0}, upload=482879]: Uploading piece...
Dec 28 14:14:06.006312079 (550h45m5.877395124s) [noise=true , address=77.90.31.21:4880 , piece_id=2EP7PPWFUVNZLRRIGSC7GW53WGRS2XCZNP7O5BDDNHENMJBISNZA, seg_pos={0 0}, upload=482879]: Uploading piece...
Dec 28 14:14:06.006896762 (550h45m5.877979807s) [noise=true , address=145.79.227.37:4566 , piece_id=5ICU5SETRGGIIKRNDRCTUUC2VTSUGRD6PWQH7BTZYVYD7S3SQSQQ, seg_pos={0 0}, upload=482879]: Uploading piece...
Dec 28 14:14:06.007257900 (550h45m5.878340935s) [noise=true , address=145.79.227.20:4439 , piece_id=WC36XT6DAV27DTUCFKYSU3AWZLWZ5EM37JQH3SZPSWOUOE4PR5HQ, seg_pos={0 0}, upload=482879]: Uploading piece...
Dec 28 14:14:07.721543183 (550h45m7.592626218s) [upload=482879]: Next returned nil segment
Dec 28 14:14:07.721554704 (550h45m7.59263775s) [upload=482879]: Waiting for error group managing segments...
Dec 28 14:14:07.997311191 (550h45m7.868394236s) [noise=true , address=145.79.227.13:4804 , piece_id=PGTARYC3LNMTR4TAGR74FPBM7RHMTHRLDW2QUW6ARA2SI26DHLZQ, seg_pos={0 0}, upload=482879]: Done uploading piece. err: <nil>
Dec 28 14:14:08.005357442 (550h45m7.876440507s) [noise=true , address=145.79.227.12:4767 , piece_id=VPWI5VMVMUCTRSYXD3QSZ6OUETWIAJQLGZQ7MRSUL2Y475DNCK3Q, seg_pos={0 0}, upload=482879]: Done uploading piece. err: <nil>
Dec 28 14:14:08.008351083 (550h45m7.879434118s) [noise=true , address=145.79.227.52:4446 , piece_id=L2LK33S7SEF7NTBUT3AX7XMTKPGPICTAXSS6MYA2OJ3DOZWRJJHA, seg_pos={0 0}, upload=482879]: Done uploading piece. err: <nil>
Dec 28 14:14:08.010815325 (550h45m7.88189837s) [noise=true , address=145.79.227.50:4373 , piece_id=RAXJ6LT3KF7DOQRSV7DH5IAPJ4WMWDLIPIOFFVQUWRU5BF4SERNQ, seg_pos={0 0}, upload=482879]: Done uploading piece. err: <nil>
Dec 28 14:14:08.014508277 (550h45m7.885591352s) [noise=true , address=145.79.227.42:4123 , piece_id=7QUYG56BEL7FSD2A6AHQNONXE4PWH3QEHZSBKEGYKXYRH5DKIP5Q, seg_pos={0 0}, upload=482879]: Done uploading piece. err: <nil>
Dec 28 14:14:08.015107678 (550h45m7.886190723s) [noise=true , address=145.79.227.20:4439 , piece_id=WC36XT6DAV27DTUCFKYSU3AWZLWZ5EM37JQH3SZPSWOUOE4PR5HQ, seg_pos={0 0}, upload=482879]: Done uploading piece. err: <nil>
Dec 28 14:14:08.020768196 (550h45m7.891851231s) [noise=true , address=145.79.227.46:4603 , piece_id=TWVG57KDN3OV4DU3QD4ONWNEEYOCKYXYQ63QBQPEZS676HDAMZYA, seg_pos={0 0}, upload=482879]: Done uploading piece. err: <nil>
Dec 28 14:14:08.022418348 (550h45m7.893501383s) [noise=true , address=145.79.227.47:4266 , piece_id=FUBWPNHK7SKGUJSKNMTHO3RGUCTABSL4QOASCW23J6GCJ3H5LDMQ, seg_pos={0 0}, upload=482879]: Done uploading piece. err: <nil>
Dec 28 14:14:08.065573974 (550h45m7.936657019s) [noise=true , address=145.79.227.37:4566 , piece_id=5ICU5SETRGGIIKRNDRCTUUC2VTSUGRD6PWQH7BTZYVYD7S3SQSQQ, seg_pos={0 0}, upload=482879]: Done uploading piece. err: <nil>
Dec 28 14:14:08.252250557 (550h45m8.123333602s) [noise=true , address=45.91.116.6:4975 , piece_id=IRL2CQSLRGQJIP3BHV7HYBS56R7HGUDQPQBNNFXEE6Q7IUF2CXHA, seg_pos={0 0}, upload=482879]: Done uploading piece. err: <nil>
Dec 28 14:14:08.253942699 (550h45m8.125025734s) [noise=true , address=45.91.116.11:4120 , piece_id=MRG7VL7KTA3T5FEUFJW5ISMKDWWLUGHHBGGL7RMWRSON4N2UVZUA, seg_pos={0 0}, upload=482879]: Done uploading piece. err: <nil>
Dec 28 14:14:09.023479484 (550h45m8.894562519s) [noise=true , address=45.91.116.14:4190 , piece_id=565ADKSRT363WQJIB2WFNFLNPS765AVU7R72A5LMHGACOYZNBYDQ, seg_pos={0 0}, upload=482879]: Done uploading piece. err: <nil>
Dec 28 14:14:09.024215435 (550h45m8.89529848s) [noise=true , address=45.91.116.9:4262 , piece_id=54C5OVTRQ2FAS4X2EEPNRRAKFEWDO5JQEDQ5HOCOTPJW54PUK2YA, seg_pos={0 0}, upload=482879]: Done uploading piece. err: <nil>
Dec 28 14:14:09.024226005 (550h45m8.89530905s) [seg_pos={0 0}, upload=482879]: Segment reached optimal threshold of 13 pieces.
Dec 28 14:14:09.024424473 (550h45m8.895507508s) [noise=true , address=77.90.31.21:4880 , piece_id=2EP7PPWFUVNZLRRIGSC7GW53WGRS2XCZNP7O5BDDNHENMJBISNZA, seg_pos={0 0}, upload=482879]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF
Dec 28 14:14:09.024431356 (550h45m8.895514391s) [noise=true , address=77.90.31.21:4880 , piece_id=2EP7PPWFUVNZLRRIGSC7GW53WGRS2XCZNP7O5BDDNHENMJBISNZA, seg_pos={0 0}, upload=482879]: Long tail cancellation due to optimal threshold
Dec 28 14:14:09.024575721 (550h45m8.895658756s) [noise=true , address=145.79.227.40:4051 , piece_id=3QYIHWAQR7G5I57JDXJ4SJ6A2LARGCAP6XK7CVE6K2WVR4XGXXHQ, seg_pos={0 0}, upload=482879]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF
Dec 28 14:14:09.024582594 (550h45m8.895665639s) [noise=true , address=145.79.227.40:4051 , piece_id=3QYIHWAQR7G5I57JDXJ4SJ6A2LARGCAP6XK7CVE6K2WVR4XGXXHQ, seg_pos={0 0}, upload=482879]: Long tail cancellation due to optimal threshold
Dec 28 14:14:09.024595309 (550h45m8.895678354s) [noise=true , address=45.91.116.12:4228 , piece_id=VXDU5ZAC3NCP65MDUZ5LSVEIGRQDJIJDP2TIUGXPHGLBLGBMMCWA, seg_pos={0 0}, upload=482879]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:55860->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:55860->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55860->45.91.116.12:4228: use of closed network connection
Dec 28 14:14:09.024622490 (550h45m8.895705555s) [noise=true , address=45.91.116.12:4228 , piece_id=VXDU5ZAC3NCP65MDUZ5LSVEIGRQDJIJDP2TIUGXPHGLBLGBMMCWA, seg_pos={0 0}, upload=482879]: Long tail cancellation due to optimal threshold
Dec 28 14:14:09.024649401 (550h45m8.895732436s) [seg_pos={0 0}, upload=482879]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 14:14:09.024766164 (550h45m8.895849199s) [upload=482879]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 14:14:09.024773017 (550h45m8.895856062s) [upload=482879]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 14:14:09.321695907 (550h45m9.192778942s) [upload=482879]: Done uploading segments...
Dec 28 14:14:09.321715164 (550h45m9.192798199s) [upload=482879]: Upload finished. err: <nil>
Duration: 5.91277438s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [................................................X...X.......] duration:
15.865926112s trace:
https://jaeger.dev.storj.io/trace/102cf391e061ae77
FAIL (2025-12-28 14:15:44 UTC)
orbiter warning: context deadline exceeded
Duration: 15.86595679s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
53 out of 60 is failed (threshold=6) [XXXXXX.XXXXXXXXXXXX.XXXXXXX.XX..XXXXXXXXXXXXXXXXXXXXX.X.XXXX] duration:
5.00141358s trace:
https://jaeger.dev.storj.io/trace/8b2cfd73b75dea0
FAIL (2025-12-28 14:15:39 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001436373s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [..X....................................X........X...X.......] duration:
41.870873253s trace:
https://jaeger.dev.storj.io/trace/36d5af3c761e71b1
FAIL (2025-12-28 14:15:36 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
7.49:4360 , piece_id=QDT5N5LDL2HCVXPOOSXWD2TOEZDF2JZWFLDFYSIMN647FI3ZORUQ, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.372193736 (550h40m10.243276781s) [noise=true , address=77.90.31.12:4618 , piece_id=45WRZ26CPUODOAYF5B6RF37DNMDDQWWJUDK7BFKYS3K3LOCNYJCA, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.372518505 (550h40m10.24360155s) [noise=true , address=77.90.31.13:4646 , piece_id=5CDMQF7GG5PDPHIPMFZWDBKWH5ANO3UIX257LJPCWRITNOA7GY4A, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.372848052 (550h40m10.243931097s) [noise=true , address=64.49.13.170:4721 , piece_id=U76TI7IPAHPZAS6XLG66E4L2CGCYH43NH4ZHTYXMXKZR6SGE7SKQ, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.373163072 (550h40m10.244246117s) [noise=true , address=77.90.31.11:4565 , piece_id=4SBLRZK3K4ED73PJ64BI6ANKBWB2AAHQA4SBUNH6CJ63GN3FJR2Q, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.373445089 (550h40m10.244528134s) [noise=true , address=145.79.227.34:4911 , piece_id=24WE4EMZSL2JHQZ7PESCMLYCZOWGDTGJD2XNLUUJEFPSYLRI5E7A, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.373768855 (550h40m10.24485189s) [noise=true , address=77.90.31.18:4819 , piece_id=OYXPAN3EK4QRDEC6VHIDL3GSGEY27G72KII6HTGUMSPYLHCYBNEA, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.374285519 (550h40m10.245368564s) [noise=true , address=77.90.31.14:4678 , piece_id=VBGPSGMNDQZYY7UKRINAV6B3FVI5BPDPYS37FQIJCMLL65F3XOEQ, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.374710689 (550h40m10.245793724s) [noise=true , address=145.79.227.40:4053 , piece_id=RO4ZZQCKCC7T4T6SOT7GGTY626BJ6ESLQG6KATTPNIUM3ICUKHAQ, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.375099329 (550h40m10.246182364s) [noise=true , address=77.90.31.19:4859 , piece_id=A4HD3FPDPCCMHRFBTS3GNGB323RYD5C5AW4ZVSNOTXFI7FLIV37Q, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.376073433 (550h40m10.247156468s) [noise=true , address=77.90.31.15:4704 , piece_id=DXWLFOIKOPE33LMEQECUJVIGRG3LPADUMULLWI4ZBHEOQHD43ULQ, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.376542387 (550h40m10.247625422s) [noise=true , address=145.79.227.44:4221 , piece_id=H72XCWLKYJAXYGGDIWN67H42RUIEOJCPUSWT5IJMFPKETH7LK3KA, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.377379761 (550h40m10.248462796s) [noise=true , address=145.79.227.32:4863 , piece_id=27TOXK44RIF3DV3PDY4RBP6HEFRQJE6N7LNLRJKSU52YCB6A3JKA, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.378022765 (550h40m10.24910581s) [noise=true , address=145.79.227.30:4789 , piece_id=4QT5IUPIK7BOHJROEK2EBFTW434OKC75FZNENMYYCRMBZ7E5BUTA, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.378821687 (550h40m10.249904722s) [noise=true , address=145.79.227.2:4119 , piece_id=EQM3WMBQ7GXRWRBZK4ZWCI23K4LSLXW6QPEABHGMWRHEYUYANKSQ, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:11.676223695 (550h40m11.54730674s) [upload=482805]: Next returned nil segment
Dec 28 14:09:11.676238083 (550h40m11.547321128s) [upload=482805]: Waiting for error group managing segments...
Dec 28 14:09:12.006762275 (550h40m11.87784531s) [noise=true , address=64.49.13.170:4721 , piece_id=U76TI7IPAHPZAS6XLG66E4L2CGCYH43NH4ZHTYXMXKZR6SGE7SKQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.250307967 (550h40m12.121391012s) [noise=true , address=145.79.227.2:4119 , piece_id=EQM3WMBQ7GXRWRBZK4ZWCI23K4LSLXW6QPEABHGMWRHEYUYANKSQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.269380734 (550h40m12.140463769s) [noise=true , address=145.79.227.49:4360 , piece_id=QDT5N5LDL2HCVXPOOSXWD2TOEZDF2JZWFLDFYSIMN647FI3ZORUQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.290818335 (550h40m12.16190138s) [noise=true , address=145.79.227.32:4863 , piece_id=27TOXK44RIF3DV3PDY4RBP6HEFRQJE6N7LNLRJKSU52YCB6A3JKA, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.467679327 (550h40m12.338762362s) [noise=true , address=45.91.116.8:4072 , piece_id=GEVTX6LLENPLN3UQLSGTFTP3DO52IRYUIQFIWC4XMOC3ZF6V4GTQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.486615595 (550h40m12.35769863s) [noise=true , address=77.90.31.12:4618 , piece_id=45WRZ26CPUODOAYF5B6RF37DNMDDQWWJUDK7BFKYS3K3LOCNYJCA, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.566542810 (550h40m12.437625845s) [noise=true , address=77.90.31.19:4859 , piece_id=A4HD3FPDPCCMHRFBTS3GNGB323RYD5C5AW4ZVSNOTXFI7FLIV37Q, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.609337578 (550h40m12.480420623s) [noise=true , address=77.90.31.13:4646 , piece_id=5CDMQF7GG5PDPHIPMFZWDBKWH5ANO3UIX257LJPCWRITNOA7GY4A, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.640856565 (550h40m12.511939611s) [noise=true , address=45.91.116.5:4322 , piece_id=SS6DOCSJHY5Y3DPWMYX45F4VYYXR44JDLYQ2YPY5XIIAYKMAG7LQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.645490329 (550h40m12.516573364s) [noise=true , address=77.90.31.18:4819 , piece_id=OYXPAN3EK4QRDEC6VHIDL3GSGEY27G72KII6HTGUMSPYLHCYBNEA, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:13.103710286 (550h40m12.974793331s) [noise=true , address=145.79.227.34:4911 , piece_id=24WE4EMZSL2JHQZ7PESCMLYCZOWGDTGJD2XNLUUJEFPSYLRI5E7A, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:13.209917141 (550h40m13.081000176s) [noise=true , address=145.79.227.44:4221 , piece_id=H72XCWLKYJAXYGGDIWN67H42RUIEOJCPUSWT5IJMFPKETH7LK3KA, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:13.218355468 (550h40m13.089438514s) [noise=true , address=77.90.31.11:4565 , piece_id=4SBLRZK3K4ED73PJ64BI6ANKBWB2AAHQA4SBUNH6CJ63GN3FJR2Q, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:13.255968741 (550h40m13.127051806s) [noise=true , address=145.79.227.30:4789 , piece_id=4QT5IUPIK7BOHJROEK2EBFTW434OKC75FZNENMYYCRMBZ7E5BUTA, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:13.255996102 (550h40m13.127079137s) [seg_pos={0 0}, upload=482805]: Segment reached optimal threshold of 14 pieces.
Dec 28 14:09:13.256214589 (550h40m13.127297634s) [noise=true , address=145.79.227.40:4053 , piece_id=RO4ZZQCKCC7T4T6SOT7GGTY626BJ6ESLQG6KATTPNIUM3ICUKHAQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DqyTc4MHvxutFXs2WWRCk8MSuxF3Y1HayyQY4Y5sy8cqDLWuG): protocol: expected piece hash; context canceled; EOF
Dec 28 14:09:13.256223686 (550h40m13.127306731s) [noise=true , address=145.79.227.40:4053 , piece_id=RO4ZZQCKCC7T4T6SOT7GGTY626BJ6ESLQG6KATTPNIUM3ICUKHAQ, seg_pos={0 0}, upload=482805]: Long tail cancellation due to optimal threshold
Dec 28 14:09:13.256512506 (550h40m13.127595551s) [noise=true , address=77.90.31.15:4704 , piece_id=DXWLFOIKOPE33LMEQECUJVIGRG3LPADUMULLWI4ZBHEOQHD43ULQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14tJRdZ1guwwsYwPf3LtENRWqYx2jBQpi7uAMMPaB32Gks9zmj): protocol: expected piece hash; context canceled; EOF
Dec 28 14:09:13.256520060 (550h40m13.127603105s) [noise=true , address=77.90.31.15:4704 , piece_id=DXWLFOIKOPE33LMEQECUJVIGRG3LPADUMULLWI4ZBHEOQHD43ULQ, seg_pos={0 0}, upload=482805]: Long tail cancellation due to optimal threshold
Dec 28 14:09:13.256649106 (550h40m13.127732151s) [noise=true , address=77.90.31.14:4678 , piece_id=VBGPSGMNDQZYY7UKRINAV6B3FVI5BPDPYS37FQIJCMLL65F3XOEQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12beu3SUZMPNWfbeHV4tLtx9UVZoEHh4Kt7oBhT59w3bGBE1ovP): protocol: write tcp 188.166.225.170:57374->77.90.31.14:4678: use of closed network connection; write tcp 188.166.225.170:57374->77.90.31.14:4678: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57374->77.90.31.14:4678: use of closed network connection
Dec 28 14:09:13.256655819 (550h40m13.127738864s) [noise=true , address=77.90.31.14:4678 , piece_id=VBGPSGMNDQZYY7UKRINAV6B3FVI5BPDPYS37FQIJCMLL65F3XOEQ, seg_pos={0 0}, upload=482805]: Long tail cancellation due to optimal threshold
Dec 28 14:09:13.256752974 (550h40m13.127836009s) [noise=true , address=64.49.13.169:4710 , piece_id=SANL7TVJFES63XD46OKQL4KIJ6YVHSEWPZKMRISEKC3ZKWQBOMIA, seg_pos={0 0}, upload=482805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:35680->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:35680->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35680->64.49.13.169:4710: use of closed network connection
Dec 28 14:09:13.256758825 (550h40m13.12784186s) [noise=true , address=64.49.13.169:4710 , piece_id=SANL7TVJFES63XD46OKQL4KIJ6YVHSEWPZKMRISEKC3ZKWQBOMIA, seg_pos={0 0}, upload=482805]: Long tail cancellation due to optimal threshold
Dec 28 14:09:13.256776549 (550h40m13.127859594s) [seg_pos={0 0}, upload=482805]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 14:09:13.256856591 (550h40m13.127939626s) [upload=482805]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 14:09:13.256862432 (550h40m13.127945467s) [upload=482805]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 14:09:13.550363717 (550h40m13.421446782s) [upload=482805]: Done uploading segments...
Dec 28 14:09:13.550405126 (550h40m13.421488161s) [upload=482805]: Upload finished. err: <nil>
Duration: 41.870901758s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [..........X.................................................] duration:
7.606078296s trace:
https://jaeger.dev.storj.io/trace/5fe9d6dbb79fbe44
FAIL (2025-12-28 14:15:33 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
67.204.205:4027 , piece_id=PAPNMRJ7E47VOMOXPBBDWKZJHLRMXTWVR5VW3J7KNQSSMHOHWOAQ, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.967737953 (550h8m4.838820988s) [noise=true , address=185.167.204.206:4072 , piece_id=WXORBDMA3KW7WZ67A4FJT6URJRFADUJ7X2PAHIGNG47T3TGMXVTQ, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.968193240 (550h8m4.839276285s) [noise=true , address=185.167.204.205:4030 , piece_id=R5DLI4ZTLOVGO56GPLJL5ZNHYSXOSYNJQHFIO4MVVNHWWBMEOV2Q, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.968780638 (550h8m4.839863683s) [noise=true , address=89.149.201.147:4019 , piece_id=MK7ORCLHO3DOWIOVSOWIQMPJ447PHRQGXP3UKITIRTYJSU5MPCCQ, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.969308924 (550h8m4.840391959s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.970088349 (550h8m4.841171404s) [noise=true , address=81.171.14.70:4039 , piece_id=EDHPQWLQZTP7MKPGEVMP7JXNTW7LFWSXCTRTSXPCBIQPROX76WRA, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.970325381 (550h8m4.841408426s) [noise=true , address=81.171.14.70:4036 , piece_id=CICNNV7JE6TBNC4RAPXKCEUSJAADOUVGSVIVQISM3ZB4COAZQDDA, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.970783813 (550h8m4.841866858s) [noise=true , address=81.171.14.70:4044 , piece_id=DGN74IG6IWAHLEMO6NLL4TJBXZ6ONTN6QUB6HXY6K7SSO3LYOYIQ, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.971205406 (550h8m4.842288441s) [noise=true , address=81.171.14.70:4048 , piece_id=TLT47ORGUKGF3MIIEAX7D22VR2FO6W6NL3VFJNHKWD4L3LIMFDLA, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.971606299 (550h8m4.842689344s) [noise=true , address=185.167.204.206:4067 , piece_id=VSKTG5RH32N7OEVWJGQVB6WJ72TEVF6DDK25W7NOH4RAXUP52LTQ, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.972162538 (550h8m4.843245583s) [noise=true , address=89.149.201.147:4002 , piece_id=G2PABNTDWCK4MKOZF6PONBMRNIXI45JTHJMAZFPEKPEJ3PQ7MTBA, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:05.795167814 (550h8m5.666250849s) [upload=482337]: Next returned nil segment
Dec 28 13:37:05.795184566 (550h8m5.666267611s) [upload=482337]: Waiting for error group managing segments...
Dec 28 13:37:06.178573801 (550h8m6.049656836s) [noise=true , address=185.167.204.206:4049 , piece_id=6RG2DZV6FPTIRE63ZNNST42TSWEBQ456YXIPPVYHIXV55GWFVJ4Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.183060835 (550h8m6.05414391s) [noise=true , address=185.167.204.206:4072 , piece_id=WXORBDMA3KW7WZ67A4FJT6URJRFADUJ7X2PAHIGNG47T3TGMXVTQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.183411202 (550h8m6.054494237s) [noise=true , address=185.167.204.206:4064 , piece_id=OG3G74OU23CEMKA54WQE3DOJ6AKAXBLZ23QWOQC4UEGQQN4JSMXQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.183791146 (550h8m6.054874181s) [noise=true , address=185.167.204.205:4029 , piece_id=6JQB6ZSCQZKWH3JWCQ4DY6BBAX73KC4Y53PRSMEMLWURZ5QWWQQQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.183973743 (550h8m6.055056778s) [noise=true , address=185.167.204.206:4053 , piece_id=YDH77S6JLKJ6IYS6DRUCELP5EJHIC2O44QBO7MIOEUYBML4GIBHA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.184465369 (550h8m6.055548414s) [noise=true , address=185.167.204.205:4008 , piece_id=RTUZPEXBB6JNSMC5YOFQQGDM57YD233W4756ZBQVMFNOVX6YKB4A, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.184501989 (550h8m6.055585024s) [noise=true , address=185.167.204.205:4030 , piece_id=R5DLI4ZTLOVGO56GPLJL5ZNHYSXOSYNJQHFIO4MVVNHWWBMEOV2Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.184856224 (550h8m6.055939259s) [noise=true , address=185.167.204.205:4009 , piece_id=V6RA7Y7FKY4DCPTRRF3BYNBALWDY5VTPXMDUXFE3353KUXLSQJEA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.185063518 (550h8m6.056146563s) [noise=true , address=185.167.204.205:4016 , piece_id=JB7SC3J2VR4U33HNQS2A2CA75THT3XLQCOIFHAJJEK7MQGY2SYAQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.185288617 (550h8m6.056371652s) [noise=true , address=185.167.204.205:4027 , piece_id=PAPNMRJ7E47VOMOXPBBDWKZJHLRMXTWVR5VW3J7KNQSSMHOHWOAQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.186241401 (550h8m6.057324436s) [noise=true , address=185.167.204.205:4024 , piece_id=FM5IMWRSJJ4RY72IX2OOYC54BFKL4FWE3UOAN3GJ26MLH345COVA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.193113658 (550h8m6.064196703s) [noise=true , address=185.167.204.205:4005 , piece_id=R4DFQJZBPJ5WJWHPODV6ILHUDIRBUSAFOMVNCLXB4UNMI6EXTKHA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.193791408 (550h8m6.064874453s) [noise=true , address=185.167.204.206:4068 , piece_id=DAI37M67FWCSVQTQOZ6RSSOJDGUVZO6SNGAHJJ44G3PGS4OJZEMA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.194292262 (550h8m6.065375307s) [noise=true , address=185.167.204.206:4046 , piece_id=VWLVHMBMJ63DQEWYBIKH3Y3NTVWFNW5OXZKMYXXBHVI3VWYXLLPQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.194302652 (550h8m6.065385687s) [noise=true , address=185.167.204.206:4067 , piece_id=VSKTG5RH32N7OEVWJGQVB6WJ72TEVF6DDK25W7NOH4RAXUP52LTQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.195359193 (550h8m6.066442228s) [noise=true , address=185.167.204.205:4012 , piece_id=LV6RUA5BWXG4DMSUVZDBGUAF5IBWUGMLJYUKNIXHRVDEBIOW4Q2Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.198423448 (550h8m6.069506493s) [noise=true , address=185.167.204.206:4070 , piece_id=5A7YYQYREIZRZDR7TRSE7XFW765JVAV7FX6LKWUI2OXAA74XN4CA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.888968408 (550h8m6.760051453s) [noise=true , address=81.171.14.70:4044 , piece_id=DGN74IG6IWAHLEMO6NLL4TJBXZ6ONTN6QUB6HXY6K7SSO3LYOYIQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.892350789 (550h8m6.763433834s) [noise=true , address=81.171.14.70:4039 , piece_id=EDHPQWLQZTP7MKPGEVMP7JXNTW7LFWSXCTRTSXPCBIQPROX76WRA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.904477160 (550h8m6.775560195s) [noise=true , address=81.171.14.70:4040 , piece_id=DYF4GSZBAKQ3H63LTVCMEFWMABY3FXHOJCNVIFYRKLF2J6B52K5A, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.128088281 (550h8m6.999171326s) [noise=true , address=81.171.14.70:4036 , piece_id=CICNNV7JE6TBNC4RAPXKCEUSJAADOUVGSVIVQISM3ZB4COAZQDDA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.430242142 (550h8m7.301325177s) [noise=true , address=89.149.201.147:4009 , piece_id=EVW2NQFYU57PC2IIVXLYIEBMYXJNIC2CLCHW6PS6NYIPCFRUVHDA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.505304332 (550h8m7.376387367s) [noise=true , address=89.149.201.147:4019 , piece_id=MK7ORCLHO3DOWIOVSOWIQMPJ447PHRQGXP3UKITIRTYJSU5MPCCQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.575461543 (550h8m7.446544609s) [noise=true , address=81.171.14.70:4048 , piece_id=TLT47ORGUKGF3MIIEAX7D22VR2FO6W6NL3VFJNHKWD4L3LIMFDLA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.599667265 (550h8m7.4707503s) [noise=true , address=89.149.201.147:4023 , piece_id=5IINX2GVRFLRX2QH24OO2DDANOKOTXTDR6AGVJBGSJ7IAZQ3CASA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.928390728 (550h8m7.799473773s) [noise=true , address=89.149.201.147:4002 , piece_id=G2PABNTDWCK4MKOZF6PONBMRNIXI45JTHJMAZFPEKPEJ3PQ7MTBA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.928417148 (550h8m7.799500193s) [seg_pos={0 0}, upload=482337]: Segment reached optimal threshold of 26 pieces.
Dec 28 13:37:07.928656303 (550h8m7.799739348s) [noise=true , address=81.171.14.70:4030 , piece_id=74YDIKUST7X3JBXVQHVLFHIKH5A4DYH5TNL3I4XUA5A6AHQO2DRA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZoRwa8wLw66Pjumy1ybp6wrZ79RrSU55uYZpwaY3xi95iSYde): protocol: expected piece hash; context canceled; EOF
Dec 28 13:37:07.928664990 (550h8m7.799748035s) [noise=true , address=81.171.14.70:4030 , piece_id=74YDIKUST7X3JBXVQHVLFHIKH5A4DYH5TNL3I4XUA5A6AHQO2DRA, seg_pos={0 0}, upload=482337]: Long tail cancellation due to optimal threshold
Dec 28 13:37:07.928723351 (550h8m7.799806386s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wgtphHKaTejC1qrJ98sDysBwtj2Zeqw2c5UFk1HMqfY8Buvr): protocol: expected piece hash; context canceled; EOF
Dec 28 13:37:07.928733922 (550h8m7.799816957s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Long tail cancellation due to optimal threshold
Dec 28 13:37:07.928754921 (550h8m7.799837966s) [seg_pos={0 0}, upload=482337]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 28 13:37:07.928870170 (550h8m7.799953215s) [upload=482337]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 13:37:07.928878627 (550h8m7.799961672s) [upload=482337]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 13:37:08.166321536 (550h8m8.037404571s) [upload=482337]: Done uploading segments...
Dec 28 13:37:08.166341444 (550h8m8.037424479s) [upload=482337]: Upload finished. err: <nil>
Duration: 7.606105566s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [............X....................X...X........X.............] duration:
17.224535315s trace:
https://jaeger.dev.storj.io/trace/360e059013858fb1
FAIL (2025-12-28 14:15:29 UTC)
orbiter warning: context deadline exceeded
Duration: 17.224571374s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
20 out of 60 is failed (threshold=3) [..X.....X..X.X...XX......X......XXX.XX....XX......XX.XX.X.X.] duration:
6.044582566s trace:
https://jaeger.dev.storj.io/trace/5be1acce1690abbe
FAIL (2025-12-28 14:15:11 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 28 14:14:05.707801185 (550h45m5.57888423s) [upload=482879]: Starting upload
Dec 28 14:14:05.707886898 (550h45m5.578969944s) [upload=482879]: Done starting upload
Dec 28 14:14:05.707951642 (550h45m5.579034687s) [upload=482879]: Uploading segments...
Dec 28 14:14:05.708074515 (550h45m5.57915756s) [upload=482879]: Got next segment. Inline: false
Dec 28 14:14:05.708106106 (550h45m5.579189151s) [upload=482879]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 14:14:05.708110434 (550h45m5.579193479s) [upload=482879]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 14:14:06.000535780 (550h45m5.871618845s) [seg_pos={0 0}, upload=482879]: Begin upload segment...
Dec 28 14:14:06.000598910 (550h45m5.871681945s) [seg_pos={0 0}, upload=482879]: Done begin upload segment.
Dec 28 14:14:06.000636912 (550h45m5.871719957s) [noise=true , address=145.79.227.40:4051 , piece_id=3QYIHWAQR7G5I57JDXJ4SJ6A2LARGCAP6XK7CVE6K2WVR4XGXXHQ, seg_pos={0 0}, upload=482879]: Uploading piece...
Dec 28 14:14:06.000757032 (550h45m5.871840097s) [noise=true , address=145.79.227.13:4804 , piece_id=PGTARYC3LNMTR4TAGR74FPBM7RHMTHRLDW2QUW6ARA2SI26DHLZQ, seg_pos={0 0}, upload=482879]: Uploading piece...
Dec 28 14:14:06.001400927 (550h45m5.872483972s) [noise=true , address=45.91.116.9:4262 , piece_id=54C5OVTRQ2FAS4X2EEPNRRAKFEWDO5JQEDQ5HOCOTPJW54PUK2YA, seg_pos={0 0}, upload=482879]: Uploading piece...
Dec 28 14:14:06.001895939 (550h45m5.872978975s) [noise=true , address=145.79.227.47:4266 , piece_id=FUBWPNHK7SKGUJSKNMTHO3RGUCTABSL4QOASCW23J6GCJ3H5LDMQ, seg_pos={0 0}, upload=482879]: Uploading piece...
Dec 28 14:14:06.002437291 (550h45m5.873520336s) [noise=true , address=45.91.116.14:4190 , piece_id=565ADKSRT363WQJIB2WFNFLNPS765AVU7R72A5LMHGACOYZNBYDQ, seg_pos={0 0}, upload=482879]: Uploading piece...
Dec 28 14:14:06.002952352 (550h45m5.874035397s) [noise=true , address=45.91.116.6:4975 , piece_id=IRL2CQSLRGQJIP3BHV7HYBS56R7HGUDQPQBNNFXEE6Q7IUF2CXHA, seg_pos={0 0}, upload=482879]: Uploading piece...
Dec 28 14:14:06.002965867 (550h45m5.874048912s) [noise=true , address=145.79.227.42:4123 , piece_id=7QUYG56BEL7FSD2A6AHQNONXE4PWH3QEHZSBKEGYKXYRH5DKIP5Q, seg_pos={0 0}, upload=482879]: Uploading piece...
Dec 28 14:14:06.003387630 (550h45m5.874470685s) [noise=true , address=45.91.116.11:4120 , piece_id=MRG7VL7KTA3T5FEUFJW5ISMKDWWLUGHHBGGL7RMWRSON4N2UVZUA, seg_pos={0 0}, upload=482879]: Uploading piece...
Dec 28 14:14:06.003838088 (550h45m5.874921133s) [noise=true , address=45.91.116.12:4228 , piece_id=VXDU5ZAC3NCP65MDUZ5LSVEIGRQDJIJDP2TIUGXPHGLBLGBMMCWA, seg_pos={0 0}, upload=482879]: Uploading piece...
Dec 28 14:14:06.004305238 (550h45m5.875388303s) [noise=true , address=145.79.227.50:4373 , piece_id=RAXJ6LT3KF7DOQRSV7DH5IAPJ4WMWDLIPIOFFVQUWRU5BF4SERNQ, seg_pos={0 0}, upload=482879]: Uploading piece...
Dec 28 14:14:06.004804198 (550h45m5.875887233s) [noise=true , address=145.79.227.46:4603 , piece_id=TWVG57KDN3OV4DU3QD4ONWNEEYOCKYXYQ63QBQPEZS676HDAMZYA, seg_pos={0 0}, upload=482879]: Uploading piece...
Dec 28 14:14:06.005276297 (550h45m5.876359342s) [noise=true , address=145.79.227.12:4767 , piece_id=VPWI5VMVMUCTRSYXD3QSZ6OUETWIAJQLGZQ7MRSUL2Y475DNCK3Q, seg_pos={0 0}, upload=482879]: Uploading piece...
Dec 28 14:14:06.005723668 (550h45m5.876806713s) [noise=true , address=145.79.227.52:4446 , piece_id=L2LK33S7SEF7NTBUT3AX7XMTKPGPICTAXSS6MYA2OJ3DOZWRJJHA, seg_pos={0 0}, upload=482879]: Uploading piece...
Dec 28 14:14:06.006312079 (550h45m5.877395124s) [noise=true , address=77.90.31.21:4880 , piece_id=2EP7PPWFUVNZLRRIGSC7GW53WGRS2XCZNP7O5BDDNHENMJBISNZA, seg_pos={0 0}, upload=482879]: Uploading piece...
Dec 28 14:14:06.006896762 (550h45m5.877979807s) [noise=true , address=145.79.227.37:4566 , piece_id=5ICU5SETRGGIIKRNDRCTUUC2VTSUGRD6PWQH7BTZYVYD7S3SQSQQ, seg_pos={0 0}, upload=482879]: Uploading piece...
Dec 28 14:14:06.007257900 (550h45m5.878340935s) [noise=true , address=145.79.227.20:4439 , piece_id=WC36XT6DAV27DTUCFKYSU3AWZLWZ5EM37JQH3SZPSWOUOE4PR5HQ, seg_pos={0 0}, upload=482879]: Uploading piece...
Dec 28 14:14:07.721543183 (550h45m7.592626218s) [upload=482879]: Next returned nil segment
Dec 28 14:14:07.721554704 (550h45m7.59263775s) [upload=482879]: Waiting for error group managing segments...
Dec 28 14:14:07.997311191 (550h45m7.868394236s) [noise=true , address=145.79.227.13:4804 , piece_id=PGTARYC3LNMTR4TAGR74FPBM7RHMTHRLDW2QUW6ARA2SI26DHLZQ, seg_pos={0 0}, upload=482879]: Done uploading piece. err: <nil>
Dec 28 14:14:08.005357442 (550h45m7.876440507s) [noise=true , address=145.79.227.12:4767 , piece_id=VPWI5VMVMUCTRSYXD3QSZ6OUETWIAJQLGZQ7MRSUL2Y475DNCK3Q, seg_pos={0 0}, upload=482879]: Done uploading piece. err: <nil>
Dec 28 14:14:08.008351083 (550h45m7.879434118s) [noise=true , address=145.79.227.52:4446 , piece_id=L2LK33S7SEF7NTBUT3AX7XMTKPGPICTAXSS6MYA2OJ3DOZWRJJHA, seg_pos={0 0}, upload=482879]: Done uploading piece. err: <nil>
Dec 28 14:14:08.010815325 (550h45m7.88189837s) [noise=true , address=145.79.227.50:4373 , piece_id=RAXJ6LT3KF7DOQRSV7DH5IAPJ4WMWDLIPIOFFVQUWRU5BF4SERNQ, seg_pos={0 0}, upload=482879]: Done uploading piece. err: <nil>
Dec 28 14:14:08.014508277 (550h45m7.885591352s) [noise=true , address=145.79.227.42:4123 , piece_id=7QUYG56BEL7FSD2A6AHQNONXE4PWH3QEHZSBKEGYKXYRH5DKIP5Q, seg_pos={0 0}, upload=482879]: Done uploading piece. err: <nil>
Dec 28 14:14:08.015107678 (550h45m7.886190723s) [noise=true , address=145.79.227.20:4439 , piece_id=WC36XT6DAV27DTUCFKYSU3AWZLWZ5EM37JQH3SZPSWOUOE4PR5HQ, seg_pos={0 0}, upload=482879]: Done uploading piece. err: <nil>
Dec 28 14:14:08.020768196 (550h45m7.891851231s) [noise=true , address=145.79.227.46:4603 , piece_id=TWVG57KDN3OV4DU3QD4ONWNEEYOCKYXYQ63QBQPEZS676HDAMZYA, seg_pos={0 0}, upload=482879]: Done uploading piece. err: <nil>
Dec 28 14:14:08.022418348 (550h45m7.893501383s) [noise=true , address=145.79.227.47:4266 , piece_id=FUBWPNHK7SKGUJSKNMTHO3RGUCTABSL4QOASCW23J6GCJ3H5LDMQ, seg_pos={0 0}, upload=482879]: Done uploading piece. err: <nil>
Dec 28 14:14:08.065573974 (550h45m7.936657019s) [noise=true , address=145.79.227.37:4566 , piece_id=5ICU5SETRGGIIKRNDRCTUUC2VTSUGRD6PWQH7BTZYVYD7S3SQSQQ, seg_pos={0 0}, upload=482879]: Done uploading piece. err: <nil>
Dec 28 14:14:08.252250557 (550h45m8.123333602s) [noise=true , address=45.91.116.6:4975 , piece_id=IRL2CQSLRGQJIP3BHV7HYBS56R7HGUDQPQBNNFXEE6Q7IUF2CXHA, seg_pos={0 0}, upload=482879]: Done uploading piece. err: <nil>
Dec 28 14:14:08.253942699 (550h45m8.125025734s) [noise=true , address=45.91.116.11:4120 , piece_id=MRG7VL7KTA3T5FEUFJW5ISMKDWWLUGHHBGGL7RMWRSON4N2UVZUA, seg_pos={0 0}, upload=482879]: Done uploading piece. err: <nil>
Dec 28 14:14:09.023479484 (550h45m8.894562519s) [noise=true , address=45.91.116.14:4190 , piece_id=565ADKSRT363WQJIB2WFNFLNPS765AVU7R72A5LMHGACOYZNBYDQ, seg_pos={0 0}, upload=482879]: Done uploading piece. err: <nil>
Dec 28 14:14:09.024215435 (550h45m8.89529848s) [noise=true , address=45.91.116.9:4262 , piece_id=54C5OVTRQ2FAS4X2EEPNRRAKFEWDO5JQEDQ5HOCOTPJW54PUK2YA, seg_pos={0 0}, upload=482879]: Done uploading piece. err: <nil>
Dec 28 14:14:09.024226005 (550h45m8.89530905s) [seg_pos={0 0}, upload=482879]: Segment reached optimal threshold of 13 pieces.
Dec 28 14:14:09.024424473 (550h45m8.895507508s) [noise=true , address=77.90.31.21:4880 , piece_id=2EP7PPWFUVNZLRRIGSC7GW53WGRS2XCZNP7O5BDDNHENMJBISNZA, seg_pos={0 0}, upload=482879]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF
Dec 28 14:14:09.024431356 (550h45m8.895514391s) [noise=true , address=77.90.31.21:4880 , piece_id=2EP7PPWFUVNZLRRIGSC7GW53WGRS2XCZNP7O5BDDNHENMJBISNZA, seg_pos={0 0}, upload=482879]: Long tail cancellation due to optimal threshold
Dec 28 14:14:09.024575721 (550h45m8.895658756s) [noise=true , address=145.79.227.40:4051 , piece_id=3QYIHWAQR7G5I57JDXJ4SJ6A2LARGCAP6XK7CVE6K2WVR4XGXXHQ, seg_pos={0 0}, upload=482879]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF
Dec 28 14:14:09.024582594 (550h45m8.895665639s) [noise=true , address=145.79.227.40:4051 , piece_id=3QYIHWAQR7G5I57JDXJ4SJ6A2LARGCAP6XK7CVE6K2WVR4XGXXHQ, seg_pos={0 0}, upload=482879]: Long tail cancellation due to optimal threshold
Dec 28 14:14:09.024595309 (550h45m8.895678354s) [noise=true , address=45.91.116.12:4228 , piece_id=VXDU5ZAC3NCP65MDUZ5LSVEIGRQDJIJDP2TIUGXPHGLBLGBMMCWA, seg_pos={0 0}, upload=482879]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:55860->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:55860->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55860->45.91.116.12:4228: use of closed network connection
Dec 28 14:14:09.024622490 (550h45m8.895705555s) [noise=true , address=45.91.116.12:4228 , piece_id=VXDU5ZAC3NCP65MDUZ5LSVEIGRQDJIJDP2TIUGXPHGLBLGBMMCWA, seg_pos={0 0}, upload=482879]: Long tail cancellation due to optimal threshold
Dec 28 14:14:09.024649401 (550h45m8.895732436s) [seg_pos={0 0}, upload=482879]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 14:14:09.024766164 (550h45m8.895849199s) [upload=482879]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 14:14:09.024773017 (550h45m8.895856062s) [upload=482879]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 14:14:09.321695907 (550h45m9.192778942s) [upload=482879]: Done uploading segments...
Dec 28 14:14:09.321715164 (550h45m9.192798199s) [upload=482879]: Upload finished. err: <nil>
Duration: 6.044611181s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [X............X....................X...X........X............] duration:
18.082666691s trace:
https://jaeger.dev.storj.io/trace/360e059013858fb1
FAIL (2025-12-28 14:15:06 UTC)
orbiter warning: context deadline exceeded
Duration: 18.082754849s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [.................................................X...X......] duration:
20.586775326s trace:
https://jaeger.dev.storj.io/trace/102cf391e061ae77
FAIL (2025-12-28 14:14:58 UTC)
orbiter warning: context deadline exceeded
Duration: 20.58679805s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
53 out of 60 is failed (threshold=6) [XXXXXXX.XXXXXXXXXXXX.XXXXXXX.XX..XXXXXXXXXXXXXXXXXXXXX.X.XXX] duration:
5.000974634s trace:
https://jaeger.dev.storj.io/trace/46e96cb0f745b38d
FAIL (2025-12-28 14:14:40 UTC)
orbiter warning: context deadline exceeded
Duration: 5.000996795s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [...........X................................................] duration:
7.527342799s trace:
https://jaeger.dev.storj.io/trace/5fe9d6dbb79fbe44
FAIL (2025-12-28 14:14:38 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
67.204.205:4027 , piece_id=PAPNMRJ7E47VOMOXPBBDWKZJHLRMXTWVR5VW3J7KNQSSMHOHWOAQ, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.967737953 (550h8m4.838820988s) [noise=true , address=185.167.204.206:4072 , piece_id=WXORBDMA3KW7WZ67A4FJT6URJRFADUJ7X2PAHIGNG47T3TGMXVTQ, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.968193240 (550h8m4.839276285s) [noise=true , address=185.167.204.205:4030 , piece_id=R5DLI4ZTLOVGO56GPLJL5ZNHYSXOSYNJQHFIO4MVVNHWWBMEOV2Q, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.968780638 (550h8m4.839863683s) [noise=true , address=89.149.201.147:4019 , piece_id=MK7ORCLHO3DOWIOVSOWIQMPJ447PHRQGXP3UKITIRTYJSU5MPCCQ, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.969308924 (550h8m4.840391959s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.970088349 (550h8m4.841171404s) [noise=true , address=81.171.14.70:4039 , piece_id=EDHPQWLQZTP7MKPGEVMP7JXNTW7LFWSXCTRTSXPCBIQPROX76WRA, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.970325381 (550h8m4.841408426s) [noise=true , address=81.171.14.70:4036 , piece_id=CICNNV7JE6TBNC4RAPXKCEUSJAADOUVGSVIVQISM3ZB4COAZQDDA, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.970783813 (550h8m4.841866858s) [noise=true , address=81.171.14.70:4044 , piece_id=DGN74IG6IWAHLEMO6NLL4TJBXZ6ONTN6QUB6HXY6K7SSO3LYOYIQ, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.971205406 (550h8m4.842288441s) [noise=true , address=81.171.14.70:4048 , piece_id=TLT47ORGUKGF3MIIEAX7D22VR2FO6W6NL3VFJNHKWD4L3LIMFDLA, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.971606299 (550h8m4.842689344s) [noise=true , address=185.167.204.206:4067 , piece_id=VSKTG5RH32N7OEVWJGQVB6WJ72TEVF6DDK25W7NOH4RAXUP52LTQ, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.972162538 (550h8m4.843245583s) [noise=true , address=89.149.201.147:4002 , piece_id=G2PABNTDWCK4MKOZF6PONBMRNIXI45JTHJMAZFPEKPEJ3PQ7MTBA, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:05.795167814 (550h8m5.666250849s) [upload=482337]: Next returned nil segment
Dec 28 13:37:05.795184566 (550h8m5.666267611s) [upload=482337]: Waiting for error group managing segments...
Dec 28 13:37:06.178573801 (550h8m6.049656836s) [noise=true , address=185.167.204.206:4049 , piece_id=6RG2DZV6FPTIRE63ZNNST42TSWEBQ456YXIPPVYHIXV55GWFVJ4Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.183060835 (550h8m6.05414391s) [noise=true , address=185.167.204.206:4072 , piece_id=WXORBDMA3KW7WZ67A4FJT6URJRFADUJ7X2PAHIGNG47T3TGMXVTQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.183411202 (550h8m6.054494237s) [noise=true , address=185.167.204.206:4064 , piece_id=OG3G74OU23CEMKA54WQE3DOJ6AKAXBLZ23QWOQC4UEGQQN4JSMXQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.183791146 (550h8m6.054874181s) [noise=true , address=185.167.204.205:4029 , piece_id=6JQB6ZSCQZKWH3JWCQ4DY6BBAX73KC4Y53PRSMEMLWURZ5QWWQQQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.183973743 (550h8m6.055056778s) [noise=true , address=185.167.204.206:4053 , piece_id=YDH77S6JLKJ6IYS6DRUCELP5EJHIC2O44QBO7MIOEUYBML4GIBHA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.184465369 (550h8m6.055548414s) [noise=true , address=185.167.204.205:4008 , piece_id=RTUZPEXBB6JNSMC5YOFQQGDM57YD233W4756ZBQVMFNOVX6YKB4A, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.184501989 (550h8m6.055585024s) [noise=true , address=185.167.204.205:4030 , piece_id=R5DLI4ZTLOVGO56GPLJL5ZNHYSXOSYNJQHFIO4MVVNHWWBMEOV2Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.184856224 (550h8m6.055939259s) [noise=true , address=185.167.204.205:4009 , piece_id=V6RA7Y7FKY4DCPTRRF3BYNBALWDY5VTPXMDUXFE3353KUXLSQJEA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.185063518 (550h8m6.056146563s) [noise=true , address=185.167.204.205:4016 , piece_id=JB7SC3J2VR4U33HNQS2A2CA75THT3XLQCOIFHAJJEK7MQGY2SYAQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.185288617 (550h8m6.056371652s) [noise=true , address=185.167.204.205:4027 , piece_id=PAPNMRJ7E47VOMOXPBBDWKZJHLRMXTWVR5VW3J7KNQSSMHOHWOAQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.186241401 (550h8m6.057324436s) [noise=true , address=185.167.204.205:4024 , piece_id=FM5IMWRSJJ4RY72IX2OOYC54BFKL4FWE3UOAN3GJ26MLH345COVA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.193113658 (550h8m6.064196703s) [noise=true , address=185.167.204.205:4005 , piece_id=R4DFQJZBPJ5WJWHPODV6ILHUDIRBUSAFOMVNCLXB4UNMI6EXTKHA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.193791408 (550h8m6.064874453s) [noise=true , address=185.167.204.206:4068 , piece_id=DAI37M67FWCSVQTQOZ6RSSOJDGUVZO6SNGAHJJ44G3PGS4OJZEMA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.194292262 (550h8m6.065375307s) [noise=true , address=185.167.204.206:4046 , piece_id=VWLVHMBMJ63DQEWYBIKH3Y3NTVWFNW5OXZKMYXXBHVI3VWYXLLPQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.194302652 (550h8m6.065385687s) [noise=true , address=185.167.204.206:4067 , piece_id=VSKTG5RH32N7OEVWJGQVB6WJ72TEVF6DDK25W7NOH4RAXUP52LTQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.195359193 (550h8m6.066442228s) [noise=true , address=185.167.204.205:4012 , piece_id=LV6RUA5BWXG4DMSUVZDBGUAF5IBWUGMLJYUKNIXHRVDEBIOW4Q2Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.198423448 (550h8m6.069506493s) [noise=true , address=185.167.204.206:4070 , piece_id=5A7YYQYREIZRZDR7TRSE7XFW765JVAV7FX6LKWUI2OXAA74XN4CA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.888968408 (550h8m6.760051453s) [noise=true , address=81.171.14.70:4044 , piece_id=DGN74IG6IWAHLEMO6NLL4TJBXZ6ONTN6QUB6HXY6K7SSO3LYOYIQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.892350789 (550h8m6.763433834s) [noise=true , address=81.171.14.70:4039 , piece_id=EDHPQWLQZTP7MKPGEVMP7JXNTW7LFWSXCTRTSXPCBIQPROX76WRA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.904477160 (550h8m6.775560195s) [noise=true , address=81.171.14.70:4040 , piece_id=DYF4GSZBAKQ3H63LTVCMEFWMABY3FXHOJCNVIFYRKLF2J6B52K5A, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.128088281 (550h8m6.999171326s) [noise=true , address=81.171.14.70:4036 , piece_id=CICNNV7JE6TBNC4RAPXKCEUSJAADOUVGSVIVQISM3ZB4COAZQDDA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.430242142 (550h8m7.301325177s) [noise=true , address=89.149.201.147:4009 , piece_id=EVW2NQFYU57PC2IIVXLYIEBMYXJNIC2CLCHW6PS6NYIPCFRUVHDA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.505304332 (550h8m7.376387367s) [noise=true , address=89.149.201.147:4019 , piece_id=MK7ORCLHO3DOWIOVSOWIQMPJ447PHRQGXP3UKITIRTYJSU5MPCCQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.575461543 (550h8m7.446544609s) [noise=true , address=81.171.14.70:4048 , piece_id=TLT47ORGUKGF3MIIEAX7D22VR2FO6W6NL3VFJNHKWD4L3LIMFDLA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.599667265 (550h8m7.4707503s) [noise=true , address=89.149.201.147:4023 , piece_id=5IINX2GVRFLRX2QH24OO2DDANOKOTXTDR6AGVJBGSJ7IAZQ3CASA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.928390728 (550h8m7.799473773s) [noise=true , address=89.149.201.147:4002 , piece_id=G2PABNTDWCK4MKOZF6PONBMRNIXI45JTHJMAZFPEKPEJ3PQ7MTBA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.928417148 (550h8m7.799500193s) [seg_pos={0 0}, upload=482337]: Segment reached optimal threshold of 26 pieces.
Dec 28 13:37:07.928656303 (550h8m7.799739348s) [noise=true , address=81.171.14.70:4030 , piece_id=74YDIKUST7X3JBXVQHVLFHIKH5A4DYH5TNL3I4XUA5A6AHQO2DRA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZoRwa8wLw66Pjumy1ybp6wrZ79RrSU55uYZpwaY3xi95iSYde): protocol: expected piece hash; context canceled; EOF
Dec 28 13:37:07.928664990 (550h8m7.799748035s) [noise=true , address=81.171.14.70:4030 , piece_id=74YDIKUST7X3JBXVQHVLFHIKH5A4DYH5TNL3I4XUA5A6AHQO2DRA, seg_pos={0 0}, upload=482337]: Long tail cancellation due to optimal threshold
Dec 28 13:37:07.928723351 (550h8m7.799806386s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wgtphHKaTejC1qrJ98sDysBwtj2Zeqw2c5UFk1HMqfY8Buvr): protocol: expected piece hash; context canceled; EOF
Dec 28 13:37:07.928733922 (550h8m7.799816957s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Long tail cancellation due to optimal threshold
Dec 28 13:37:07.928754921 (550h8m7.799837966s) [seg_pos={0 0}, upload=482337]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 28 13:37:07.928870170 (550h8m7.799953215s) [upload=482337]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 13:37:07.928878627 (550h8m7.799961672s) [upload=482337]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 13:37:08.166321536 (550h8m8.037404571s) [upload=482337]: Done uploading segments...
Dec 28 13:37:08.166341444 (550h8m8.037424479s) [upload=482337]: Upload finished. err: <nil>
Duration: 7.5273705s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [..................................................X...X.....] duration:
23.241084707s trace:
https://jaeger.dev.storj.io/trace/102cf391e061ae77
FAIL (2025-12-28 14:14:37 UTC)
orbiter warning: context deadline exceeded
Duration: 23.241113641s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
21 out of 60 is failed (threshold=3) [X..X.....X..X.X...XX......X......XXX.XX....XX......XX.XX.X.X] duration:
30.002879072s trace:
https://jaeger.dev.storj.io/trace/5be1acce1690abbe
FAIL (2025-12-28 14:14:34 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 28 14:14:05.707801185 (550h45m5.57888423s) [upload=482879]: Starting upload
Dec 28 14:14:05.707886898 (550h45m5.578969944s) [upload=482879]: Done starting upload
Dec 28 14:14:05.707951642 (550h45m5.579034687s) [upload=482879]: Uploading segments...
Dec 28 14:14:05.708074515 (550h45m5.57915756s) [upload=482879]: Got next segment. Inline: false
Dec 28 14:14:05.708106106 (550h45m5.579189151s) [upload=482879]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 14:14:05.708110434 (550h45m5.579193479s) [upload=482879]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 14:14:06.000535780 (550h45m5.871618845s) [seg_pos={0 0}, upload=482879]: Begin upload segment...
Dec 28 14:14:06.000598910 (550h45m5.871681945s) [seg_pos={0 0}, upload=482879]: Done begin upload segment.
Dec 28 14:14:06.000636912 (550h45m5.871719957s) [noise=true , address=145.79.227.40:4051 , piece_id=3QYIHWAQR7G5I57JDXJ4SJ6A2LARGCAP6XK7CVE6K2WVR4XGXXHQ, seg_pos={0 0}, upload=482879]: Uploading piece...
Dec 28 14:14:06.000757032 (550h45m5.871840097s) [noise=true , address=145.79.227.13:4804 , piece_id=PGTARYC3LNMTR4TAGR74FPBM7RHMTHRLDW2QUW6ARA2SI26DHLZQ, seg_pos={0 0}, upload=482879]: Uploading piece...
Dec 28 14:14:06.001400927 (550h45m5.872483972s) [noise=true , address=45.91.116.9:4262 , piece_id=54C5OVTRQ2FAS4X2EEPNRRAKFEWDO5JQEDQ5HOCOTPJW54PUK2YA, seg_pos={0 0}, upload=482879]: Uploading piece...
Dec 28 14:14:06.001895939 (550h45m5.872978975s) [noise=true , address=145.79.227.47:4266 , piece_id=FUBWPNHK7SKGUJSKNMTHO3RGUCTABSL4QOASCW23J6GCJ3H5LDMQ, seg_pos={0 0}, upload=482879]: Uploading piece...
Dec 28 14:14:06.002437291 (550h45m5.873520336s) [noise=true , address=45.91.116.14:4190 , piece_id=565ADKSRT363WQJIB2WFNFLNPS765AVU7R72A5LMHGACOYZNBYDQ, seg_pos={0 0}, upload=482879]: Uploading piece...
Dec 28 14:14:06.002952352 (550h45m5.874035397s) [noise=true , address=45.91.116.6:4975 , piece_id=IRL2CQSLRGQJIP3BHV7HYBS56R7HGUDQPQBNNFXEE6Q7IUF2CXHA, seg_pos={0 0}, upload=482879]: Uploading piece...
Dec 28 14:14:06.002965867 (550h45m5.874048912s) [noise=true , address=145.79.227.42:4123 , piece_id=7QUYG56BEL7FSD2A6AHQNONXE4PWH3QEHZSBKEGYKXYRH5DKIP5Q, seg_pos={0 0}, upload=482879]: Uploading piece...
Dec 28 14:14:06.003387630 (550h45m5.874470685s) [noise=true , address=45.91.116.11:4120 , piece_id=MRG7VL7KTA3T5FEUFJW5ISMKDWWLUGHHBGGL7RMWRSON4N2UVZUA, seg_pos={0 0}, upload=482879]: Uploading piece...
Dec 28 14:14:06.003838088 (550h45m5.874921133s) [noise=true , address=45.91.116.12:4228 , piece_id=VXDU5ZAC3NCP65MDUZ5LSVEIGRQDJIJDP2TIUGXPHGLBLGBMMCWA, seg_pos={0 0}, upload=482879]: Uploading piece...
Dec 28 14:14:06.004305238 (550h45m5.875388303s) [noise=true , address=145.79.227.50:4373 , piece_id=RAXJ6LT3KF7DOQRSV7DH5IAPJ4WMWDLIPIOFFVQUWRU5BF4SERNQ, seg_pos={0 0}, upload=482879]: Uploading piece...
Dec 28 14:14:06.004804198 (550h45m5.875887233s) [noise=true , address=145.79.227.46:4603 , piece_id=TWVG57KDN3OV4DU3QD4ONWNEEYOCKYXYQ63QBQPEZS676HDAMZYA, seg_pos={0 0}, upload=482879]: Uploading piece...
Dec 28 14:14:06.005276297 (550h45m5.876359342s) [noise=true , address=145.79.227.12:4767 , piece_id=VPWI5VMVMUCTRSYXD3QSZ6OUETWIAJQLGZQ7MRSUL2Y475DNCK3Q, seg_pos={0 0}, upload=482879]: Uploading piece...
Dec 28 14:14:06.005723668 (550h45m5.876806713s) [noise=true , address=145.79.227.52:4446 , piece_id=L2LK33S7SEF7NTBUT3AX7XMTKPGPICTAXSS6MYA2OJ3DOZWRJJHA, seg_pos={0 0}, upload=482879]: Uploading piece...
Dec 28 14:14:06.006312079 (550h45m5.877395124s) [noise=true , address=77.90.31.21:4880 , piece_id=2EP7PPWFUVNZLRRIGSC7GW53WGRS2XCZNP7O5BDDNHENMJBISNZA, seg_pos={0 0}, upload=482879]: Uploading piece...
Dec 28 14:14:06.006896762 (550h45m5.877979807s) [noise=true , address=145.79.227.37:4566 , piece_id=5ICU5SETRGGIIKRNDRCTUUC2VTSUGRD6PWQH7BTZYVYD7S3SQSQQ, seg_pos={0 0}, upload=482879]: Uploading piece...
Dec 28 14:14:06.007257900 (550h45m5.878340935s) [noise=true , address=145.79.227.20:4439 , piece_id=WC36XT6DAV27DTUCFKYSU3AWZLWZ5EM37JQH3SZPSWOUOE4PR5HQ, seg_pos={0 0}, upload=482879]: Uploading piece...
Dec 28 14:14:07.721543183 (550h45m7.592626218s) [upload=482879]: Next returned nil segment
Dec 28 14:14:07.721554704 (550h45m7.59263775s) [upload=482879]: Waiting for error group managing segments...
Dec 28 14:14:07.997311191 (550h45m7.868394236s) [noise=true , address=145.79.227.13:4804 , piece_id=PGTARYC3LNMTR4TAGR74FPBM7RHMTHRLDW2QUW6ARA2SI26DHLZQ, seg_pos={0 0}, upload=482879]: Done uploading piece. err: <nil>
Dec 28 14:14:08.005357442 (550h45m7.876440507s) [noise=true , address=145.79.227.12:4767 , piece_id=VPWI5VMVMUCTRSYXD3QSZ6OUETWIAJQLGZQ7MRSUL2Y475DNCK3Q, seg_pos={0 0}, upload=482879]: Done uploading piece. err: <nil>
Dec 28 14:14:08.008351083 (550h45m7.879434118s) [noise=true , address=145.79.227.52:4446 , piece_id=L2LK33S7SEF7NTBUT3AX7XMTKPGPICTAXSS6MYA2OJ3DOZWRJJHA, seg_pos={0 0}, upload=482879]: Done uploading piece. err: <nil>
Dec 28 14:14:08.010815325 (550h45m7.88189837s) [noise=true , address=145.79.227.50:4373 , piece_id=RAXJ6LT3KF7DOQRSV7DH5IAPJ4WMWDLIPIOFFVQUWRU5BF4SERNQ, seg_pos={0 0}, upload=482879]: Done uploading piece. err: <nil>
Dec 28 14:14:08.014508277 (550h45m7.885591352s) [noise=true , address=145.79.227.42:4123 , piece_id=7QUYG56BEL7FSD2A6AHQNONXE4PWH3QEHZSBKEGYKXYRH5DKIP5Q, seg_pos={0 0}, upload=482879]: Done uploading piece. err: <nil>
Dec 28 14:14:08.015107678 (550h45m7.886190723s) [noise=true , address=145.79.227.20:4439 , piece_id=WC36XT6DAV27DTUCFKYSU3AWZLWZ5EM37JQH3SZPSWOUOE4PR5HQ, seg_pos={0 0}, upload=482879]: Done uploading piece. err: <nil>
Dec 28 14:14:08.020768196 (550h45m7.891851231s) [noise=true , address=145.79.227.46:4603 , piece_id=TWVG57KDN3OV4DU3QD4ONWNEEYOCKYXYQ63QBQPEZS676HDAMZYA, seg_pos={0 0}, upload=482879]: Done uploading piece. err: <nil>
Dec 28 14:14:08.022418348 (550h45m7.893501383s) [noise=true , address=145.79.227.47:4266 , piece_id=FUBWPNHK7SKGUJSKNMTHO3RGUCTABSL4QOASCW23J6GCJ3H5LDMQ, seg_pos={0 0}, upload=482879]: Done uploading piece. err: <nil>
Dec 28 14:14:08.065573974 (550h45m7.936657019s) [noise=true , address=145.79.227.37:4566 , piece_id=5ICU5SETRGGIIKRNDRCTUUC2VTSUGRD6PWQH7BTZYVYD7S3SQSQQ, seg_pos={0 0}, upload=482879]: Done uploading piece. err: <nil>
Dec 28 14:14:08.252250557 (550h45m8.123333602s) [noise=true , address=45.91.116.6:4975 , piece_id=IRL2CQSLRGQJIP3BHV7HYBS56R7HGUDQPQBNNFXEE6Q7IUF2CXHA, seg_pos={0 0}, upload=482879]: Done uploading piece. err: <nil>
Dec 28 14:14:08.253942699 (550h45m8.125025734s) [noise=true , address=45.91.116.11:4120 , piece_id=MRG7VL7KTA3T5FEUFJW5ISMKDWWLUGHHBGGL7RMWRSON4N2UVZUA, seg_pos={0 0}, upload=482879]: Done uploading piece. err: <nil>
Dec 28 14:14:09.023479484 (550h45m8.894562519s) [noise=true , address=45.91.116.14:4190 , piece_id=565ADKSRT363WQJIB2WFNFLNPS765AVU7R72A5LMHGACOYZNBYDQ, seg_pos={0 0}, upload=482879]: Done uploading piece. err: <nil>
Dec 28 14:14:09.024215435 (550h45m8.89529848s) [noise=true , address=45.91.116.9:4262 , piece_id=54C5OVTRQ2FAS4X2EEPNRRAKFEWDO5JQEDQ5HOCOTPJW54PUK2YA, seg_pos={0 0}, upload=482879]: Done uploading piece. err: <nil>
Dec 28 14:14:09.024226005 (550h45m8.89530905s) [seg_pos={0 0}, upload=482879]: Segment reached optimal threshold of 13 pieces.
Dec 28 14:14:09.024424473 (550h45m8.895507508s) [noise=true , address=77.90.31.21:4880 , piece_id=2EP7PPWFUVNZLRRIGSC7GW53WGRS2XCZNP7O5BDDNHENMJBISNZA, seg_pos={0 0}, upload=482879]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF
Dec 28 14:14:09.024431356 (550h45m8.895514391s) [noise=true , address=77.90.31.21:4880 , piece_id=2EP7PPWFUVNZLRRIGSC7GW53WGRS2XCZNP7O5BDDNHENMJBISNZA, seg_pos={0 0}, upload=482879]: Long tail cancellation due to optimal threshold
Dec 28 14:14:09.024575721 (550h45m8.895658756s) [noise=true , address=145.79.227.40:4051 , piece_id=3QYIHWAQR7G5I57JDXJ4SJ6A2LARGCAP6XK7CVE6K2WVR4XGXXHQ, seg_pos={0 0}, upload=482879]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF
Dec 28 14:14:09.024582594 (550h45m8.895665639s) [noise=true , address=145.79.227.40:4051 , piece_id=3QYIHWAQR7G5I57JDXJ4SJ6A2LARGCAP6XK7CVE6K2WVR4XGXXHQ, seg_pos={0 0}, upload=482879]: Long tail cancellation due to optimal threshold
Dec 28 14:14:09.024595309 (550h45m8.895678354s) [noise=true , address=45.91.116.12:4228 , piece_id=VXDU5ZAC3NCP65MDUZ5LSVEIGRQDJIJDP2TIUGXPHGLBLGBMMCWA, seg_pos={0 0}, upload=482879]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:55860->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:55860->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55860->45.91.116.12:4228: use of closed network connection
Dec 28 14:14:09.024622490 (550h45m8.895705555s) [noise=true , address=45.91.116.12:4228 , piece_id=VXDU5ZAC3NCP65MDUZ5LSVEIGRQDJIJDP2TIUGXPHGLBLGBMMCWA, seg_pos={0 0}, upload=482879]: Long tail cancellation due to optimal threshold
Dec 28 14:14:09.024649401 (550h45m8.895732436s) [seg_pos={0 0}, upload=482879]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 14:14:09.024766164 (550h45m8.895849199s) [upload=482879]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 14:14:09.024773017 (550h45m8.895856062s) [upload=482879]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 14:14:09.321695907 (550h45m9.192778942s) [upload=482879]: Done uploading segments...
Dec 28 14:14:09.321715164 (550h45m9.192798199s) [upload=482879]: Upload finished. err: <nil>
Duration: 30.002899861s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [...X....................................X........X...X......] duration:
33.554976921s trace:
https://jaeger.dev.storj.io/trace/36d5af3c761e71b1
FAIL (2025-12-28 14:14:21 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
7.49:4360 , piece_id=QDT5N5LDL2HCVXPOOSXWD2TOEZDF2JZWFLDFYSIMN647FI3ZORUQ, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.372193736 (550h40m10.243276781s) [noise=true , address=77.90.31.12:4618 , piece_id=45WRZ26CPUODOAYF5B6RF37DNMDDQWWJUDK7BFKYS3K3LOCNYJCA, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.372518505 (550h40m10.24360155s) [noise=true , address=77.90.31.13:4646 , piece_id=5CDMQF7GG5PDPHIPMFZWDBKWH5ANO3UIX257LJPCWRITNOA7GY4A, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.372848052 (550h40m10.243931097s) [noise=true , address=64.49.13.170:4721 , piece_id=U76TI7IPAHPZAS6XLG66E4L2CGCYH43NH4ZHTYXMXKZR6SGE7SKQ, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.373163072 (550h40m10.244246117s) [noise=true , address=77.90.31.11:4565 , piece_id=4SBLRZK3K4ED73PJ64BI6ANKBWB2AAHQA4SBUNH6CJ63GN3FJR2Q, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.373445089 (550h40m10.244528134s) [noise=true , address=145.79.227.34:4911 , piece_id=24WE4EMZSL2JHQZ7PESCMLYCZOWGDTGJD2XNLUUJEFPSYLRI5E7A, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.373768855 (550h40m10.24485189s) [noise=true , address=77.90.31.18:4819 , piece_id=OYXPAN3EK4QRDEC6VHIDL3GSGEY27G72KII6HTGUMSPYLHCYBNEA, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.374285519 (550h40m10.245368564s) [noise=true , address=77.90.31.14:4678 , piece_id=VBGPSGMNDQZYY7UKRINAV6B3FVI5BPDPYS37FQIJCMLL65F3XOEQ, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.374710689 (550h40m10.245793724s) [noise=true , address=145.79.227.40:4053 , piece_id=RO4ZZQCKCC7T4T6SOT7GGTY626BJ6ESLQG6KATTPNIUM3ICUKHAQ, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.375099329 (550h40m10.246182364s) [noise=true , address=77.90.31.19:4859 , piece_id=A4HD3FPDPCCMHRFBTS3GNGB323RYD5C5AW4ZVSNOTXFI7FLIV37Q, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.376073433 (550h40m10.247156468s) [noise=true , address=77.90.31.15:4704 , piece_id=DXWLFOIKOPE33LMEQECUJVIGRG3LPADUMULLWI4ZBHEOQHD43ULQ, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.376542387 (550h40m10.247625422s) [noise=true , address=145.79.227.44:4221 , piece_id=H72XCWLKYJAXYGGDIWN67H42RUIEOJCPUSWT5IJMFPKETH7LK3KA, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.377379761 (550h40m10.248462796s) [noise=true , address=145.79.227.32:4863 , piece_id=27TOXK44RIF3DV3PDY4RBP6HEFRQJE6N7LNLRJKSU52YCB6A3JKA, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.378022765 (550h40m10.24910581s) [noise=true , address=145.79.227.30:4789 , piece_id=4QT5IUPIK7BOHJROEK2EBFTW434OKC75FZNENMYYCRMBZ7E5BUTA, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.378821687 (550h40m10.249904722s) [noise=true , address=145.79.227.2:4119 , piece_id=EQM3WMBQ7GXRWRBZK4ZWCI23K4LSLXW6QPEABHGMWRHEYUYANKSQ, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:11.676223695 (550h40m11.54730674s) [upload=482805]: Next returned nil segment
Dec 28 14:09:11.676238083 (550h40m11.547321128s) [upload=482805]: Waiting for error group managing segments...
Dec 28 14:09:12.006762275 (550h40m11.87784531s) [noise=true , address=64.49.13.170:4721 , piece_id=U76TI7IPAHPZAS6XLG66E4L2CGCYH43NH4ZHTYXMXKZR6SGE7SKQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.250307967 (550h40m12.121391012s) [noise=true , address=145.79.227.2:4119 , piece_id=EQM3WMBQ7GXRWRBZK4ZWCI23K4LSLXW6QPEABHGMWRHEYUYANKSQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.269380734 (550h40m12.140463769s) [noise=true , address=145.79.227.49:4360 , piece_id=QDT5N5LDL2HCVXPOOSXWD2TOEZDF2JZWFLDFYSIMN647FI3ZORUQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.290818335 (550h40m12.16190138s) [noise=true , address=145.79.227.32:4863 , piece_id=27TOXK44RIF3DV3PDY4RBP6HEFRQJE6N7LNLRJKSU52YCB6A3JKA, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.467679327 (550h40m12.338762362s) [noise=true , address=45.91.116.8:4072 , piece_id=GEVTX6LLENPLN3UQLSGTFTP3DO52IRYUIQFIWC4XMOC3ZF6V4GTQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.486615595 (550h40m12.35769863s) [noise=true , address=77.90.31.12:4618 , piece_id=45WRZ26CPUODOAYF5B6RF37DNMDDQWWJUDK7BFKYS3K3LOCNYJCA, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.566542810 (550h40m12.437625845s) [noise=true , address=77.90.31.19:4859 , piece_id=A4HD3FPDPCCMHRFBTS3GNGB323RYD5C5AW4ZVSNOTXFI7FLIV37Q, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.609337578 (550h40m12.480420623s) [noise=true , address=77.90.31.13:4646 , piece_id=5CDMQF7GG5PDPHIPMFZWDBKWH5ANO3UIX257LJPCWRITNOA7GY4A, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.640856565 (550h40m12.511939611s) [noise=true , address=45.91.116.5:4322 , piece_id=SS6DOCSJHY5Y3DPWMYX45F4VYYXR44JDLYQ2YPY5XIIAYKMAG7LQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.645490329 (550h40m12.516573364s) [noise=true , address=77.90.31.18:4819 , piece_id=OYXPAN3EK4QRDEC6VHIDL3GSGEY27G72KII6HTGUMSPYLHCYBNEA, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:13.103710286 (550h40m12.974793331s) [noise=true , address=145.79.227.34:4911 , piece_id=24WE4EMZSL2JHQZ7PESCMLYCZOWGDTGJD2XNLUUJEFPSYLRI5E7A, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:13.209917141 (550h40m13.081000176s) [noise=true , address=145.79.227.44:4221 , piece_id=H72XCWLKYJAXYGGDIWN67H42RUIEOJCPUSWT5IJMFPKETH7LK3KA, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:13.218355468 (550h40m13.089438514s) [noise=true , address=77.90.31.11:4565 , piece_id=4SBLRZK3K4ED73PJ64BI6ANKBWB2AAHQA4SBUNH6CJ63GN3FJR2Q, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:13.255968741 (550h40m13.127051806s) [noise=true , address=145.79.227.30:4789 , piece_id=4QT5IUPIK7BOHJROEK2EBFTW434OKC75FZNENMYYCRMBZ7E5BUTA, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:13.255996102 (550h40m13.127079137s) [seg_pos={0 0}, upload=482805]: Segment reached optimal threshold of 14 pieces.
Dec 28 14:09:13.256214589 (550h40m13.127297634s) [noise=true , address=145.79.227.40:4053 , piece_id=RO4ZZQCKCC7T4T6SOT7GGTY626BJ6ESLQG6KATTPNIUM3ICUKHAQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DqyTc4MHvxutFXs2WWRCk8MSuxF3Y1HayyQY4Y5sy8cqDLWuG): protocol: expected piece hash; context canceled; EOF
Dec 28 14:09:13.256223686 (550h40m13.127306731s) [noise=true , address=145.79.227.40:4053 , piece_id=RO4ZZQCKCC7T4T6SOT7GGTY626BJ6ESLQG6KATTPNIUM3ICUKHAQ, seg_pos={0 0}, upload=482805]: Long tail cancellation due to optimal threshold
Dec 28 14:09:13.256512506 (550h40m13.127595551s) [noise=true , address=77.90.31.15:4704 , piece_id=DXWLFOIKOPE33LMEQECUJVIGRG3LPADUMULLWI4ZBHEOQHD43ULQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14tJRdZ1guwwsYwPf3LtENRWqYx2jBQpi7uAMMPaB32Gks9zmj): protocol: expected piece hash; context canceled; EOF
Dec 28 14:09:13.256520060 (550h40m13.127603105s) [noise=true , address=77.90.31.15:4704 , piece_id=DXWLFOIKOPE33LMEQECUJVIGRG3LPADUMULLWI4ZBHEOQHD43ULQ, seg_pos={0 0}, upload=482805]: Long tail cancellation due to optimal threshold
Dec 28 14:09:13.256649106 (550h40m13.127732151s) [noise=true , address=77.90.31.14:4678 , piece_id=VBGPSGMNDQZYY7UKRINAV6B3FVI5BPDPYS37FQIJCMLL65F3XOEQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12beu3SUZMPNWfbeHV4tLtx9UVZoEHh4Kt7oBhT59w3bGBE1ovP): protocol: write tcp 188.166.225.170:57374->77.90.31.14:4678: use of closed network connection; write tcp 188.166.225.170:57374->77.90.31.14:4678: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57374->77.90.31.14:4678: use of closed network connection
Dec 28 14:09:13.256655819 (550h40m13.127738864s) [noise=true , address=77.90.31.14:4678 , piece_id=VBGPSGMNDQZYY7UKRINAV6B3FVI5BPDPYS37FQIJCMLL65F3XOEQ, seg_pos={0 0}, upload=482805]: Long tail cancellation due to optimal threshold
Dec 28 14:09:13.256752974 (550h40m13.127836009s) [noise=true , address=64.49.13.169:4710 , piece_id=SANL7TVJFES63XD46OKQL4KIJ6YVHSEWPZKMRISEKC3ZKWQBOMIA, seg_pos={0 0}, upload=482805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:35680->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:35680->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35680->64.49.13.169:4710: use of closed network connection
Dec 28 14:09:13.256758825 (550h40m13.12784186s) [noise=true , address=64.49.13.169:4710 , piece_id=SANL7TVJFES63XD46OKQL4KIJ6YVHSEWPZKMRISEKC3ZKWQBOMIA, seg_pos={0 0}, upload=482805]: Long tail cancellation due to optimal threshold
Dec 28 14:09:13.256776549 (550h40m13.127859594s) [seg_pos={0 0}, upload=482805]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 14:09:13.256856591 (550h40m13.127939626s) [upload=482805]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 14:09:13.256862432 (550h40m13.127945467s) [upload=482805]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 14:09:13.550363717 (550h40m13.421446782s) [upload=482805]: Done uploading segments...
Dec 28 14:09:13.550405126 (550h40m13.421488161s) [upload=482805]: Upload finished. err: <nil>
Duration: 33.554993002s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
52 out of 60 is failed (threshold=6) [.XXXXXXX.XXXXXXXXXXXX.XXXXXXX.XX..XXXXXXXXXXXXXXXXXXXXX.X.XX] duration:
5.001425082s trace:
https://jaeger.dev.storj.io/trace/6de1a36fc7c72aa2
FAIL (2025-12-28 14:14:21 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001444249s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [.X............X....................X...X........X...........] duration:
16.828241938s trace:
https://jaeger.dev.storj.io/trace/360e059013858fb1
FAIL (2025-12-28 14:14:08 UTC)
orbiter warning: context deadline exceeded
Duration: 16.828258058s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [............X...............................................] duration:
7.589857725s trace:
https://jaeger.dev.storj.io/trace/5fe9d6dbb79fbe44
FAIL (2025-12-28 14:13:59 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
67.204.205:4027 , piece_id=PAPNMRJ7E47VOMOXPBBDWKZJHLRMXTWVR5VW3J7KNQSSMHOHWOAQ, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.967737953 (550h8m4.838820988s) [noise=true , address=185.167.204.206:4072 , piece_id=WXORBDMA3KW7WZ67A4FJT6URJRFADUJ7X2PAHIGNG47T3TGMXVTQ, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.968193240 (550h8m4.839276285s) [noise=true , address=185.167.204.205:4030 , piece_id=R5DLI4ZTLOVGO56GPLJL5ZNHYSXOSYNJQHFIO4MVVNHWWBMEOV2Q, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.968780638 (550h8m4.839863683s) [noise=true , address=89.149.201.147:4019 , piece_id=MK7ORCLHO3DOWIOVSOWIQMPJ447PHRQGXP3UKITIRTYJSU5MPCCQ, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.969308924 (550h8m4.840391959s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.970088349 (550h8m4.841171404s) [noise=true , address=81.171.14.70:4039 , piece_id=EDHPQWLQZTP7MKPGEVMP7JXNTW7LFWSXCTRTSXPCBIQPROX76WRA, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.970325381 (550h8m4.841408426s) [noise=true , address=81.171.14.70:4036 , piece_id=CICNNV7JE6TBNC4RAPXKCEUSJAADOUVGSVIVQISM3ZB4COAZQDDA, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.970783813 (550h8m4.841866858s) [noise=true , address=81.171.14.70:4044 , piece_id=DGN74IG6IWAHLEMO6NLL4TJBXZ6ONTN6QUB6HXY6K7SSO3LYOYIQ, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.971205406 (550h8m4.842288441s) [noise=true , address=81.171.14.70:4048 , piece_id=TLT47ORGUKGF3MIIEAX7D22VR2FO6W6NL3VFJNHKWD4L3LIMFDLA, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.971606299 (550h8m4.842689344s) [noise=true , address=185.167.204.206:4067 , piece_id=VSKTG5RH32N7OEVWJGQVB6WJ72TEVF6DDK25W7NOH4RAXUP52LTQ, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.972162538 (550h8m4.843245583s) [noise=true , address=89.149.201.147:4002 , piece_id=G2PABNTDWCK4MKOZF6PONBMRNIXI45JTHJMAZFPEKPEJ3PQ7MTBA, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:05.795167814 (550h8m5.666250849s) [upload=482337]: Next returned nil segment
Dec 28 13:37:05.795184566 (550h8m5.666267611s) [upload=482337]: Waiting for error group managing segments...
Dec 28 13:37:06.178573801 (550h8m6.049656836s) [noise=true , address=185.167.204.206:4049 , piece_id=6RG2DZV6FPTIRE63ZNNST42TSWEBQ456YXIPPVYHIXV55GWFVJ4Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.183060835 (550h8m6.05414391s) [noise=true , address=185.167.204.206:4072 , piece_id=WXORBDMA3KW7WZ67A4FJT6URJRFADUJ7X2PAHIGNG47T3TGMXVTQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.183411202 (550h8m6.054494237s) [noise=true , address=185.167.204.206:4064 , piece_id=OG3G74OU23CEMKA54WQE3DOJ6AKAXBLZ23QWOQC4UEGQQN4JSMXQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.183791146 (550h8m6.054874181s) [noise=true , address=185.167.204.205:4029 , piece_id=6JQB6ZSCQZKWH3JWCQ4DY6BBAX73KC4Y53PRSMEMLWURZ5QWWQQQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.183973743 (550h8m6.055056778s) [noise=true , address=185.167.204.206:4053 , piece_id=YDH77S6JLKJ6IYS6DRUCELP5EJHIC2O44QBO7MIOEUYBML4GIBHA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.184465369 (550h8m6.055548414s) [noise=true , address=185.167.204.205:4008 , piece_id=RTUZPEXBB6JNSMC5YOFQQGDM57YD233W4756ZBQVMFNOVX6YKB4A, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.184501989 (550h8m6.055585024s) [noise=true , address=185.167.204.205:4030 , piece_id=R5DLI4ZTLOVGO56GPLJL5ZNHYSXOSYNJQHFIO4MVVNHWWBMEOV2Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.184856224 (550h8m6.055939259s) [noise=true , address=185.167.204.205:4009 , piece_id=V6RA7Y7FKY4DCPTRRF3BYNBALWDY5VTPXMDUXFE3353KUXLSQJEA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.185063518 (550h8m6.056146563s) [noise=true , address=185.167.204.205:4016 , piece_id=JB7SC3J2VR4U33HNQS2A2CA75THT3XLQCOIFHAJJEK7MQGY2SYAQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.185288617 (550h8m6.056371652s) [noise=true , address=185.167.204.205:4027 , piece_id=PAPNMRJ7E47VOMOXPBBDWKZJHLRMXTWVR5VW3J7KNQSSMHOHWOAQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.186241401 (550h8m6.057324436s) [noise=true , address=185.167.204.205:4024 , piece_id=FM5IMWRSJJ4RY72IX2OOYC54BFKL4FWE3UOAN3GJ26MLH345COVA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.193113658 (550h8m6.064196703s) [noise=true , address=185.167.204.205:4005 , piece_id=R4DFQJZBPJ5WJWHPODV6ILHUDIRBUSAFOMVNCLXB4UNMI6EXTKHA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.193791408 (550h8m6.064874453s) [noise=true , address=185.167.204.206:4068 , piece_id=DAI37M67FWCSVQTQOZ6RSSOJDGUVZO6SNGAHJJ44G3PGS4OJZEMA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.194292262 (550h8m6.065375307s) [noise=true , address=185.167.204.206:4046 , piece_id=VWLVHMBMJ63DQEWYBIKH3Y3NTVWFNW5OXZKMYXXBHVI3VWYXLLPQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.194302652 (550h8m6.065385687s) [noise=true , address=185.167.204.206:4067 , piece_id=VSKTG5RH32N7OEVWJGQVB6WJ72TEVF6DDK25W7NOH4RAXUP52LTQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.195359193 (550h8m6.066442228s) [noise=true , address=185.167.204.205:4012 , piece_id=LV6RUA5BWXG4DMSUVZDBGUAF5IBWUGMLJYUKNIXHRVDEBIOW4Q2Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.198423448 (550h8m6.069506493s) [noise=true , address=185.167.204.206:4070 , piece_id=5A7YYQYREIZRZDR7TRSE7XFW765JVAV7FX6LKWUI2OXAA74XN4CA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.888968408 (550h8m6.760051453s) [noise=true , address=81.171.14.70:4044 , piece_id=DGN74IG6IWAHLEMO6NLL4TJBXZ6ONTN6QUB6HXY6K7SSO3LYOYIQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.892350789 (550h8m6.763433834s) [noise=true , address=81.171.14.70:4039 , piece_id=EDHPQWLQZTP7MKPGEVMP7JXNTW7LFWSXCTRTSXPCBIQPROX76WRA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.904477160 (550h8m6.775560195s) [noise=true , address=81.171.14.70:4040 , piece_id=DYF4GSZBAKQ3H63LTVCMEFWMABY3FXHOJCNVIFYRKLF2J6B52K5A, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.128088281 (550h8m6.999171326s) [noise=true , address=81.171.14.70:4036 , piece_id=CICNNV7JE6TBNC4RAPXKCEUSJAADOUVGSVIVQISM3ZB4COAZQDDA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.430242142 (550h8m7.301325177s) [noise=true , address=89.149.201.147:4009 , piece_id=EVW2NQFYU57PC2IIVXLYIEBMYXJNIC2CLCHW6PS6NYIPCFRUVHDA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.505304332 (550h8m7.376387367s) [noise=true , address=89.149.201.147:4019 , piece_id=MK7ORCLHO3DOWIOVSOWIQMPJ447PHRQGXP3UKITIRTYJSU5MPCCQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.575461543 (550h8m7.446544609s) [noise=true , address=81.171.14.70:4048 , piece_id=TLT47ORGUKGF3MIIEAX7D22VR2FO6W6NL3VFJNHKWD4L3LIMFDLA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.599667265 (550h8m7.4707503s) [noise=true , address=89.149.201.147:4023 , piece_id=5IINX2GVRFLRX2QH24OO2DDANOKOTXTDR6AGVJBGSJ7IAZQ3CASA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.928390728 (550h8m7.799473773s) [noise=true , address=89.149.201.147:4002 , piece_id=G2PABNTDWCK4MKOZF6PONBMRNIXI45JTHJMAZFPEKPEJ3PQ7MTBA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.928417148 (550h8m7.799500193s) [seg_pos={0 0}, upload=482337]: Segment reached optimal threshold of 26 pieces.
Dec 28 13:37:07.928656303 (550h8m7.799739348s) [noise=true , address=81.171.14.70:4030 , piece_id=74YDIKUST7X3JBXVQHVLFHIKH5A4DYH5TNL3I4XUA5A6AHQO2DRA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZoRwa8wLw66Pjumy1ybp6wrZ79RrSU55uYZpwaY3xi95iSYde): protocol: expected piece hash; context canceled; EOF
Dec 28 13:37:07.928664990 (550h8m7.799748035s) [noise=true , address=81.171.14.70:4030 , piece_id=74YDIKUST7X3JBXVQHVLFHIKH5A4DYH5TNL3I4XUA5A6AHQO2DRA, seg_pos={0 0}, upload=482337]: Long tail cancellation due to optimal threshold
Dec 28 13:37:07.928723351 (550h8m7.799806386s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wgtphHKaTejC1qrJ98sDysBwtj2Zeqw2c5UFk1HMqfY8Buvr): protocol: expected piece hash; context canceled; EOF
Dec 28 13:37:07.928733922 (550h8m7.799816957s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Long tail cancellation due to optimal threshold
Dec 28 13:37:07.928754921 (550h8m7.799837966s) [seg_pos={0 0}, upload=482337]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 28 13:37:07.928870170 (550h8m7.799953215s) [upload=482337]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 13:37:07.928878627 (550h8m7.799961672s) [upload=482337]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 13:37:08.166321536 (550h8m8.037404571s) [upload=482337]: Done uploading segments...
Dec 28 13:37:08.166341444 (550h8m8.037424479s) [upload=482337]: Upload finished. err: <nil>
Duration: 7.589879337s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [....X....................................X........X...X.....] duration:
38.76567092s trace:
https://jaeger.dev.storj.io/trace/36d5af3c761e71b1
FAIL (2025-12-28 14:13:48 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
7.49:4360 , piece_id=QDT5N5LDL2HCVXPOOSXWD2TOEZDF2JZWFLDFYSIMN647FI3ZORUQ, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.372193736 (550h40m10.243276781s) [noise=true , address=77.90.31.12:4618 , piece_id=45WRZ26CPUODOAYF5B6RF37DNMDDQWWJUDK7BFKYS3K3LOCNYJCA, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.372518505 (550h40m10.24360155s) [noise=true , address=77.90.31.13:4646 , piece_id=5CDMQF7GG5PDPHIPMFZWDBKWH5ANO3UIX257LJPCWRITNOA7GY4A, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.372848052 (550h40m10.243931097s) [noise=true , address=64.49.13.170:4721 , piece_id=U76TI7IPAHPZAS6XLG66E4L2CGCYH43NH4ZHTYXMXKZR6SGE7SKQ, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.373163072 (550h40m10.244246117s) [noise=true , address=77.90.31.11:4565 , piece_id=4SBLRZK3K4ED73PJ64BI6ANKBWB2AAHQA4SBUNH6CJ63GN3FJR2Q, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.373445089 (550h40m10.244528134s) [noise=true , address=145.79.227.34:4911 , piece_id=24WE4EMZSL2JHQZ7PESCMLYCZOWGDTGJD2XNLUUJEFPSYLRI5E7A, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.373768855 (550h40m10.24485189s) [noise=true , address=77.90.31.18:4819 , piece_id=OYXPAN3EK4QRDEC6VHIDL3GSGEY27G72KII6HTGUMSPYLHCYBNEA, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.374285519 (550h40m10.245368564s) [noise=true , address=77.90.31.14:4678 , piece_id=VBGPSGMNDQZYY7UKRINAV6B3FVI5BPDPYS37FQIJCMLL65F3XOEQ, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.374710689 (550h40m10.245793724s) [noise=true , address=145.79.227.40:4053 , piece_id=RO4ZZQCKCC7T4T6SOT7GGTY626BJ6ESLQG6KATTPNIUM3ICUKHAQ, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.375099329 (550h40m10.246182364s) [noise=true , address=77.90.31.19:4859 , piece_id=A4HD3FPDPCCMHRFBTS3GNGB323RYD5C5AW4ZVSNOTXFI7FLIV37Q, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.376073433 (550h40m10.247156468s) [noise=true , address=77.90.31.15:4704 , piece_id=DXWLFOIKOPE33LMEQECUJVIGRG3LPADUMULLWI4ZBHEOQHD43ULQ, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.376542387 (550h40m10.247625422s) [noise=true , address=145.79.227.44:4221 , piece_id=H72XCWLKYJAXYGGDIWN67H42RUIEOJCPUSWT5IJMFPKETH7LK3KA, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.377379761 (550h40m10.248462796s) [noise=true , address=145.79.227.32:4863 , piece_id=27TOXK44RIF3DV3PDY4RBP6HEFRQJE6N7LNLRJKSU52YCB6A3JKA, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.378022765 (550h40m10.24910581s) [noise=true , address=145.79.227.30:4789 , piece_id=4QT5IUPIK7BOHJROEK2EBFTW434OKC75FZNENMYYCRMBZ7E5BUTA, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.378821687 (550h40m10.249904722s) [noise=true , address=145.79.227.2:4119 , piece_id=EQM3WMBQ7GXRWRBZK4ZWCI23K4LSLXW6QPEABHGMWRHEYUYANKSQ, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:11.676223695 (550h40m11.54730674s) [upload=482805]: Next returned nil segment
Dec 28 14:09:11.676238083 (550h40m11.547321128s) [upload=482805]: Waiting for error group managing segments...
Dec 28 14:09:12.006762275 (550h40m11.87784531s) [noise=true , address=64.49.13.170:4721 , piece_id=U76TI7IPAHPZAS6XLG66E4L2CGCYH43NH4ZHTYXMXKZR6SGE7SKQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.250307967 (550h40m12.121391012s) [noise=true , address=145.79.227.2:4119 , piece_id=EQM3WMBQ7GXRWRBZK4ZWCI23K4LSLXW6QPEABHGMWRHEYUYANKSQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.269380734 (550h40m12.140463769s) [noise=true , address=145.79.227.49:4360 , piece_id=QDT5N5LDL2HCVXPOOSXWD2TOEZDF2JZWFLDFYSIMN647FI3ZORUQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.290818335 (550h40m12.16190138s) [noise=true , address=145.79.227.32:4863 , piece_id=27TOXK44RIF3DV3PDY4RBP6HEFRQJE6N7LNLRJKSU52YCB6A3JKA, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.467679327 (550h40m12.338762362s) [noise=true , address=45.91.116.8:4072 , piece_id=GEVTX6LLENPLN3UQLSGTFTP3DO52IRYUIQFIWC4XMOC3ZF6V4GTQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.486615595 (550h40m12.35769863s) [noise=true , address=77.90.31.12:4618 , piece_id=45WRZ26CPUODOAYF5B6RF37DNMDDQWWJUDK7BFKYS3K3LOCNYJCA, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.566542810 (550h40m12.437625845s) [noise=true , address=77.90.31.19:4859 , piece_id=A4HD3FPDPCCMHRFBTS3GNGB323RYD5C5AW4ZVSNOTXFI7FLIV37Q, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.609337578 (550h40m12.480420623s) [noise=true , address=77.90.31.13:4646 , piece_id=5CDMQF7GG5PDPHIPMFZWDBKWH5ANO3UIX257LJPCWRITNOA7GY4A, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.640856565 (550h40m12.511939611s) [noise=true , address=45.91.116.5:4322 , piece_id=SS6DOCSJHY5Y3DPWMYX45F4VYYXR44JDLYQ2YPY5XIIAYKMAG7LQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.645490329 (550h40m12.516573364s) [noise=true , address=77.90.31.18:4819 , piece_id=OYXPAN3EK4QRDEC6VHIDL3GSGEY27G72KII6HTGUMSPYLHCYBNEA, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:13.103710286 (550h40m12.974793331s) [noise=true , address=145.79.227.34:4911 , piece_id=24WE4EMZSL2JHQZ7PESCMLYCZOWGDTGJD2XNLUUJEFPSYLRI5E7A, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:13.209917141 (550h40m13.081000176s) [noise=true , address=145.79.227.44:4221 , piece_id=H72XCWLKYJAXYGGDIWN67H42RUIEOJCPUSWT5IJMFPKETH7LK3KA, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:13.218355468 (550h40m13.089438514s) [noise=true , address=77.90.31.11:4565 , piece_id=4SBLRZK3K4ED73PJ64BI6ANKBWB2AAHQA4SBUNH6CJ63GN3FJR2Q, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:13.255968741 (550h40m13.127051806s) [noise=true , address=145.79.227.30:4789 , piece_id=4QT5IUPIK7BOHJROEK2EBFTW434OKC75FZNENMYYCRMBZ7E5BUTA, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:13.255996102 (550h40m13.127079137s) [seg_pos={0 0}, upload=482805]: Segment reached optimal threshold of 14 pieces.
Dec 28 14:09:13.256214589 (550h40m13.127297634s) [noise=true , address=145.79.227.40:4053 , piece_id=RO4ZZQCKCC7T4T6SOT7GGTY626BJ6ESLQG6KATTPNIUM3ICUKHAQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DqyTc4MHvxutFXs2WWRCk8MSuxF3Y1HayyQY4Y5sy8cqDLWuG): protocol: expected piece hash; context canceled; EOF
Dec 28 14:09:13.256223686 (550h40m13.127306731s) [noise=true , address=145.79.227.40:4053 , piece_id=RO4ZZQCKCC7T4T6SOT7GGTY626BJ6ESLQG6KATTPNIUM3ICUKHAQ, seg_pos={0 0}, upload=482805]: Long tail cancellation due to optimal threshold
Dec 28 14:09:13.256512506 (550h40m13.127595551s) [noise=true , address=77.90.31.15:4704 , piece_id=DXWLFOIKOPE33LMEQECUJVIGRG3LPADUMULLWI4ZBHEOQHD43ULQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14tJRdZ1guwwsYwPf3LtENRWqYx2jBQpi7uAMMPaB32Gks9zmj): protocol: expected piece hash; context canceled; EOF
Dec 28 14:09:13.256520060 (550h40m13.127603105s) [noise=true , address=77.90.31.15:4704 , piece_id=DXWLFOIKOPE33LMEQECUJVIGRG3LPADUMULLWI4ZBHEOQHD43ULQ, seg_pos={0 0}, upload=482805]: Long tail cancellation due to optimal threshold
Dec 28 14:09:13.256649106 (550h40m13.127732151s) [noise=true , address=77.90.31.14:4678 , piece_id=VBGPSGMNDQZYY7UKRINAV6B3FVI5BPDPYS37FQIJCMLL65F3XOEQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12beu3SUZMPNWfbeHV4tLtx9UVZoEHh4Kt7oBhT59w3bGBE1ovP): protocol: write tcp 188.166.225.170:57374->77.90.31.14:4678: use of closed network connection; write tcp 188.166.225.170:57374->77.90.31.14:4678: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57374->77.90.31.14:4678: use of closed network connection
Dec 28 14:09:13.256655819 (550h40m13.127738864s) [noise=true , address=77.90.31.14:4678 , piece_id=VBGPSGMNDQZYY7UKRINAV6B3FVI5BPDPYS37FQIJCMLL65F3XOEQ, seg_pos={0 0}, upload=482805]: Long tail cancellation due to optimal threshold
Dec 28 14:09:13.256752974 (550h40m13.127836009s) [noise=true , address=64.49.13.169:4710 , piece_id=SANL7TVJFES63XD46OKQL4KIJ6YVHSEWPZKMRISEKC3ZKWQBOMIA, seg_pos={0 0}, upload=482805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:35680->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:35680->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35680->64.49.13.169:4710: use of closed network connection
Dec 28 14:09:13.256758825 (550h40m13.12784186s) [noise=true , address=64.49.13.169:4710 , piece_id=SANL7TVJFES63XD46OKQL4KIJ6YVHSEWPZKMRISEKC3ZKWQBOMIA, seg_pos={0 0}, upload=482805]: Long tail cancellation due to optimal threshold
Dec 28 14:09:13.256776549 (550h40m13.127859594s) [seg_pos={0 0}, upload=482805]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 14:09:13.256856591 (550h40m13.127939626s) [upload=482805]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 14:09:13.256862432 (550h40m13.127945467s) [upload=482805]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 14:09:13.550363717 (550h40m13.421446782s) [upload=482805]: Done uploading segments...
Dec 28 14:09:13.550405126 (550h40m13.421488161s) [upload=482805]: Upload finished. err: <nil>
Duration: 38.765730964s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [..X............X....................X...X........X..........] duration:
24.917566655s trace:
https://jaeger.dev.storj.io/trace/360e059013858fb1
FAIL (2025-12-28 14:13:46 UTC)
orbiter warning: context deadline exceeded
Duration: 24.917591002s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [...................................................X...X....] duration:
5.956791109s trace:
https://jaeger.dev.storj.io/trace/102cf391e061ae77
FAIL (2025-12-28 14:13:33 UTC)
orbiter warning: context deadline exceeded
Duration: 5.956822839s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [.............X..............................................] duration:
6.869523411s trace:
https://jaeger.dev.storj.io/trace/5fe9d6dbb79fbe44
FAIL (2025-12-28 14:13:12 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
67.204.205:4027 , piece_id=PAPNMRJ7E47VOMOXPBBDWKZJHLRMXTWVR5VW3J7KNQSSMHOHWOAQ, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.967737953 (550h8m4.838820988s) [noise=true , address=185.167.204.206:4072 , piece_id=WXORBDMA3KW7WZ67A4FJT6URJRFADUJ7X2PAHIGNG47T3TGMXVTQ, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.968193240 (550h8m4.839276285s) [noise=true , address=185.167.204.205:4030 , piece_id=R5DLI4ZTLOVGO56GPLJL5ZNHYSXOSYNJQHFIO4MVVNHWWBMEOV2Q, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.968780638 (550h8m4.839863683s) [noise=true , address=89.149.201.147:4019 , piece_id=MK7ORCLHO3DOWIOVSOWIQMPJ447PHRQGXP3UKITIRTYJSU5MPCCQ, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.969308924 (550h8m4.840391959s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.970088349 (550h8m4.841171404s) [noise=true , address=81.171.14.70:4039 , piece_id=EDHPQWLQZTP7MKPGEVMP7JXNTW7LFWSXCTRTSXPCBIQPROX76WRA, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.970325381 (550h8m4.841408426s) [noise=true , address=81.171.14.70:4036 , piece_id=CICNNV7JE6TBNC4RAPXKCEUSJAADOUVGSVIVQISM3ZB4COAZQDDA, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.970783813 (550h8m4.841866858s) [noise=true , address=81.171.14.70:4044 , piece_id=DGN74IG6IWAHLEMO6NLL4TJBXZ6ONTN6QUB6HXY6K7SSO3LYOYIQ, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.971205406 (550h8m4.842288441s) [noise=true , address=81.171.14.70:4048 , piece_id=TLT47ORGUKGF3MIIEAX7D22VR2FO6W6NL3VFJNHKWD4L3LIMFDLA, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.971606299 (550h8m4.842689344s) [noise=true , address=185.167.204.206:4067 , piece_id=VSKTG5RH32N7OEVWJGQVB6WJ72TEVF6DDK25W7NOH4RAXUP52LTQ, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.972162538 (550h8m4.843245583s) [noise=true , address=89.149.201.147:4002 , piece_id=G2PABNTDWCK4MKOZF6PONBMRNIXI45JTHJMAZFPEKPEJ3PQ7MTBA, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:05.795167814 (550h8m5.666250849s) [upload=482337]: Next returned nil segment
Dec 28 13:37:05.795184566 (550h8m5.666267611s) [upload=482337]: Waiting for error group managing segments...
Dec 28 13:37:06.178573801 (550h8m6.049656836s) [noise=true , address=185.167.204.206:4049 , piece_id=6RG2DZV6FPTIRE63ZNNST42TSWEBQ456YXIPPVYHIXV55GWFVJ4Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.183060835 (550h8m6.05414391s) [noise=true , address=185.167.204.206:4072 , piece_id=WXORBDMA3KW7WZ67A4FJT6URJRFADUJ7X2PAHIGNG47T3TGMXVTQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.183411202 (550h8m6.054494237s) [noise=true , address=185.167.204.206:4064 , piece_id=OG3G74OU23CEMKA54WQE3DOJ6AKAXBLZ23QWOQC4UEGQQN4JSMXQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.183791146 (550h8m6.054874181s) [noise=true , address=185.167.204.205:4029 , piece_id=6JQB6ZSCQZKWH3JWCQ4DY6BBAX73KC4Y53PRSMEMLWURZ5QWWQQQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.183973743 (550h8m6.055056778s) [noise=true , address=185.167.204.206:4053 , piece_id=YDH77S6JLKJ6IYS6DRUCELP5EJHIC2O44QBO7MIOEUYBML4GIBHA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.184465369 (550h8m6.055548414s) [noise=true , address=185.167.204.205:4008 , piece_id=RTUZPEXBB6JNSMC5YOFQQGDM57YD233W4756ZBQVMFNOVX6YKB4A, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.184501989 (550h8m6.055585024s) [noise=true , address=185.167.204.205:4030 , piece_id=R5DLI4ZTLOVGO56GPLJL5ZNHYSXOSYNJQHFIO4MVVNHWWBMEOV2Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.184856224 (550h8m6.055939259s) [noise=true , address=185.167.204.205:4009 , piece_id=V6RA7Y7FKY4DCPTRRF3BYNBALWDY5VTPXMDUXFE3353KUXLSQJEA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.185063518 (550h8m6.056146563s) [noise=true , address=185.167.204.205:4016 , piece_id=JB7SC3J2VR4U33HNQS2A2CA75THT3XLQCOIFHAJJEK7MQGY2SYAQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.185288617 (550h8m6.056371652s) [noise=true , address=185.167.204.205:4027 , piece_id=PAPNMRJ7E47VOMOXPBBDWKZJHLRMXTWVR5VW3J7KNQSSMHOHWOAQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.186241401 (550h8m6.057324436s) [noise=true , address=185.167.204.205:4024 , piece_id=FM5IMWRSJJ4RY72IX2OOYC54BFKL4FWE3UOAN3GJ26MLH345COVA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.193113658 (550h8m6.064196703s) [noise=true , address=185.167.204.205:4005 , piece_id=R4DFQJZBPJ5WJWHPODV6ILHUDIRBUSAFOMVNCLXB4UNMI6EXTKHA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.193791408 (550h8m6.064874453s) [noise=true , address=185.167.204.206:4068 , piece_id=DAI37M67FWCSVQTQOZ6RSSOJDGUVZO6SNGAHJJ44G3PGS4OJZEMA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.194292262 (550h8m6.065375307s) [noise=true , address=185.167.204.206:4046 , piece_id=VWLVHMBMJ63DQEWYBIKH3Y3NTVWFNW5OXZKMYXXBHVI3VWYXLLPQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.194302652 (550h8m6.065385687s) [noise=true , address=185.167.204.206:4067 , piece_id=VSKTG5RH32N7OEVWJGQVB6WJ72TEVF6DDK25W7NOH4RAXUP52LTQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.195359193 (550h8m6.066442228s) [noise=true , address=185.167.204.205:4012 , piece_id=LV6RUA5BWXG4DMSUVZDBGUAF5IBWUGMLJYUKNIXHRVDEBIOW4Q2Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.198423448 (550h8m6.069506493s) [noise=true , address=185.167.204.206:4070 , piece_id=5A7YYQYREIZRZDR7TRSE7XFW765JVAV7FX6LKWUI2OXAA74XN4CA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.888968408 (550h8m6.760051453s) [noise=true , address=81.171.14.70:4044 , piece_id=DGN74IG6IWAHLEMO6NLL4TJBXZ6ONTN6QUB6HXY6K7SSO3LYOYIQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.892350789 (550h8m6.763433834s) [noise=true , address=81.171.14.70:4039 , piece_id=EDHPQWLQZTP7MKPGEVMP7JXNTW7LFWSXCTRTSXPCBIQPROX76WRA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.904477160 (550h8m6.775560195s) [noise=true , address=81.171.14.70:4040 , piece_id=DYF4GSZBAKQ3H63LTVCMEFWMABY3FXHOJCNVIFYRKLF2J6B52K5A, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.128088281 (550h8m6.999171326s) [noise=true , address=81.171.14.70:4036 , piece_id=CICNNV7JE6TBNC4RAPXKCEUSJAADOUVGSVIVQISM3ZB4COAZQDDA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.430242142 (550h8m7.301325177s) [noise=true , address=89.149.201.147:4009 , piece_id=EVW2NQFYU57PC2IIVXLYIEBMYXJNIC2CLCHW6PS6NYIPCFRUVHDA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.505304332 (550h8m7.376387367s) [noise=true , address=89.149.201.147:4019 , piece_id=MK7ORCLHO3DOWIOVSOWIQMPJ447PHRQGXP3UKITIRTYJSU5MPCCQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.575461543 (550h8m7.446544609s) [noise=true , address=81.171.14.70:4048 , piece_id=TLT47ORGUKGF3MIIEAX7D22VR2FO6W6NL3VFJNHKWD4L3LIMFDLA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.599667265 (550h8m7.4707503s) [noise=true , address=89.149.201.147:4023 , piece_id=5IINX2GVRFLRX2QH24OO2DDANOKOTXTDR6AGVJBGSJ7IAZQ3CASA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.928390728 (550h8m7.799473773s) [noise=true , address=89.149.201.147:4002 , piece_id=G2PABNTDWCK4MKOZF6PONBMRNIXI45JTHJMAZFPEKPEJ3PQ7MTBA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.928417148 (550h8m7.799500193s) [seg_pos={0 0}, upload=482337]: Segment reached optimal threshold of 26 pieces.
Dec 28 13:37:07.928656303 (550h8m7.799739348s) [noise=true , address=81.171.14.70:4030 , piece_id=74YDIKUST7X3JBXVQHVLFHIKH5A4DYH5TNL3I4XUA5A6AHQO2DRA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZoRwa8wLw66Pjumy1ybp6wrZ79RrSU55uYZpwaY3xi95iSYde): protocol: expected piece hash; context canceled; EOF
Dec 28 13:37:07.928664990 (550h8m7.799748035s) [noise=true , address=81.171.14.70:4030 , piece_id=74YDIKUST7X3JBXVQHVLFHIKH5A4DYH5TNL3I4XUA5A6AHQO2DRA, seg_pos={0 0}, upload=482337]: Long tail cancellation due to optimal threshold
Dec 28 13:37:07.928723351 (550h8m7.799806386s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wgtphHKaTejC1qrJ98sDysBwtj2Zeqw2c5UFk1HMqfY8Buvr): protocol: expected piece hash; context canceled; EOF
Dec 28 13:37:07.928733922 (550h8m7.799816957s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Long tail cancellation due to optimal threshold
Dec 28 13:37:07.928754921 (550h8m7.799837966s) [seg_pos={0 0}, upload=482337]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 28 13:37:07.928870170 (550h8m7.799953215s) [upload=482337]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 13:37:07.928878627 (550h8m7.799961672s) [upload=482337]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 13:37:08.166321536 (550h8m8.037404571s) [upload=482337]: Done uploading segments...
Dec 28 13:37:08.166341444 (550h8m8.037424479s) [upload=482337]: Upload finished. err: <nil>
Duration: 6.869553407s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [....................................................X...X...] duration:
16.109682915s trace:
https://jaeger.dev.storj.io/trace/102cf391e061ae77
FAIL (2025-12-28 14:13:11 UTC)
orbiter warning: context deadline exceeded
Duration: 16.109716419s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
52 out of 60 is failed (threshold=6) [X.XXXXXXX.XXXXXXXXXXXX.XXXXXXX.XX..XXXXXXXXXXXXXXXXXXXXX.X.X] duration:
5.000682628s trace:
https://jaeger.dev.storj.io/trace/1a361e89699aa481
FAIL (2025-12-28 14:13:07 UTC)
orbiter warning: context deadline exceeded
Duration: 5.000707476s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [...X............X....................X...X........X.........] duration:
20.985729564s trace:
https://jaeger.dev.storj.io/trace/360e059013858fb1
FAIL (2025-12-28 14:13:06 UTC)
orbiter warning: context deadline exceeded
Duration: 20.985755394s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
20 out of 60 is failed (threshold=3) [.X..X.....X..X.X...XX......X......XXX.XX....XX......XX.XX.X.] duration:
5.730035979s trace:
https://jaeger.dev.storj.io/trace/7ed73ea0ef2fcf6e
FAIL (2025-12-28 14:13:02 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 28 14:12:26.955988030 (550h43m26.827071076s) [upload=482857]: Starting upload
Dec 28 14:12:26.956048295 (550h43m26.827131331s) [upload=482857]: Done starting upload
Dec 28 14:12:26.956220924 (550h43m26.827303959s) [upload=482857]: Uploading segments...
Dec 28 14:12:26.956304343 (550h43m26.827387378s) [upload=482857]: Got next segment. Inline: false
Dec 28 14:12:26.956356803 (550h43m26.827439838s) [upload=482857]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 14:12:26.956362854 (550h43m26.827445889s) [upload=482857]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 14:12:27.203346923 (550h43m27.074429968s) [seg_pos={0 0}, upload=482857]: Begin upload segment...
Dec 28 14:12:27.203420483 (550h43m27.074503528s) [seg_pos={0 0}, upload=482857]: Done begin upload segment.
Dec 28 14:12:27.203465909 (550h43m27.074548944s) [noise=true , address=45.91.116.10:4084 , piece_id=DODVFMCUC2SLGQBNAB2XJ7MUPDY346QOCHERDB2KLUXOZV47PMCQ, seg_pos={0 0}, upload=482857]: Uploading piece...
Dec 28 14:12:27.205011863 (550h43m27.076094898s) [noise=true , address=145.79.227.39:4015 , piece_id=BS3WL72BOIM5M5E3CIR6QAQTEBGAAMGL45KTBTNGFFRNQFGAOIGQ, seg_pos={0 0}, upload=482857]: Uploading piece...
Dec 28 14:12:27.205507086 (550h43m27.076590131s) [noise=true , address=145.79.227.36:4517 , piece_id=FSKK4MN5R5YCQPGEAUSFLGCKKSMQM6SHR3MU7XW4WGBBUPEHCDTA, seg_pos={0 0}, upload=482857]: Uploading piece...
Dec 28 14:12:27.206025684 (550h43m27.077108729s) [noise=true , address=145.79.227.7:4294 , piece_id=I5KWL6V3UGP2M75WIH2LD4DDFNIHCPKMB3LQ5AMFDINRXGWZTOGA, seg_pos={0 0}, upload=482857]: Uploading piece...
Dec 28 14:12:27.207686346 (550h43m27.078769401s) [noise=true , address=145.79.227.23:4546 , piece_id=PXCQYRS4XKGCI75GUD74NTZDS6B5PFDFWHP2UWHO6FFGZ7NYTJNA, seg_pos={0 0}, upload=482857]: Uploading piece...
Dec 28 14:12:27.209137098 (550h43m27.080220143s) [noise=true , address=45.91.116.13:4156 , piece_id=4JXOX4GL4OTZYYDTTEUYIYLKTYTVE6QUKSLAFICMS3J6MUNV5ZEQ, seg_pos={0 0}, upload=482857]: Uploading piece...
Dec 28 14:12:27.209241617 (550h43m27.080324662s) [noise=true , address=145.79.227.13:4802 , piece_id=KGJBMKAOESBSIRTFJASM4AVLIXTULAFQIDLC7BJOW4QNA7STADDQ, seg_pos={0 0}, upload=482857]: Uploading piece...
Dec 28 14:12:27.211032798 (550h43m27.082115833s) [noise=true , address=45.91.116.7:4009 , piece_id=BQEQBPOJ2DEA7SP52JUOBCE65O3CJIPNAOFRX6PCXZV5YYDCCNWQ, seg_pos={0 0}, upload=482857]: Uploading piece...
Dec 28 14:12:27.212335017 (550h43m27.083418062s) [noise=true , address=145.79.227.47:4267 , piece_id=U6AS4TE4KV4GSBTK6LNX6LJO6SVL6M3NN4G4XE6GWY4N3Y4REAZA, seg_pos={0 0}, upload=482857]: Uploading piece...
Dec 28 14:12:27.214760837 (550h43m27.085843872s) [noise=true , address=45.91.116.11:4119 , piece_id=DKKYFWRSICCYP6SD7AG4X5Y477QELMOQ7NPDQAW4QEUPOPW2XITQ, seg_pos={0 0}, upload=482857]: Uploading piece...
Dec 28 14:12:27.215345150 (550h43m27.086428195s) [noise=true , address=145.79.227.22:4511 , piece_id=CENRFEVSOLMISFMAKXW3HPKZA7Y6K72WDFUYJRTOYASV7JNWAFQQ, seg_pos={0 0}, upload=482857]: Uploading piece...
Dec 28 14:12:27.217163823 (550h43m27.088246868s) [noise=true , address=145.79.227.4:4185 , piece_id=WDD5X5TL57YJKVCAC5PVHPLKL7UBLQPRLXQL6AT5W4SGECLSRTWA, seg_pos={0 0}, upload=482857]: Uploading piece...
Dec 28 14:12:27.217605494 (550h43m27.088688539s) [noise=true , address=145.79.227.6:4258 , piece_id=TDXLR4XXCVIZRVNKRJAVTI2TV6YIWNC2MDLYKLNO254REB474J5Q, seg_pos={0 0}, upload=482857]: Uploading piece...
Dec 28 14:12:27.218542177 (550h43m27.089625222s) [noise=true , address=145.79.227.25:4618 , piece_id=PBB23LSOUMNTZNKGK43PCUVA4YADXDA4V6YZWJXD34XXGYINNGZA, seg_pos={0 0}, upload=482857]: Uploading piece...
Dec 28 14:12:27.220450450 (550h43m27.091533495s) [noise=true , address=145.79.227.37:4566 , piece_id=AESIVAA37WELHB4UNGKJVH73RW62TFZZYUCSEV6GSFZDRXM2XQGA, seg_pos={0 0}, upload=482857]: Uploading piece...
Dec 28 14:12:27.221721161 (550h43m27.092804206s) [noise=true , address=145.79.227.12:4765 , piece_id=PTT7UVJF7X6JUAGZSBNLGXCR43BNZPYMTZS26Y6INRMYLRVWW4SA, seg_pos={0 0}, upload=482857]: Uploading piece...
Dec 28 14:12:28.836487613 (550h43m28.707570658s) [upload=482857]: Next returned nil segment
Dec 28 14:12:28.836525865 (550h43m28.7076089s) [upload=482857]: Waiting for error group managing segments...
Dec 28 14:12:29.222239759 (550h43m29.093322814s) [noise=true , address=145.79.227.13:4802 , piece_id=KGJBMKAOESBSIRTFJASM4AVLIXTULAFQIDLC7BJOW4QNA7STADDQ, seg_pos={0 0}, upload=482857]: Done uploading piece. err: <nil>
Dec 28 14:12:29.235763600 (550h43m29.106846645s) [noise=true , address=145.79.227.25:4618 , piece_id=PBB23LSOUMNTZNKGK43PCUVA4YADXDA4V6YZWJXD34XXGYINNGZA, seg_pos={0 0}, upload=482857]: Done uploading piece. err: <nil>
Dec 28 14:12:29.238585754 (550h43m29.109668799s) [noise=true , address=145.79.227.39:4015 , piece_id=BS3WL72BOIM5M5E3CIR6QAQTEBGAAMGL45KTBTNGFFRNQFGAOIGQ, seg_pos={0 0}, upload=482857]: Done uploading piece. err: <nil>
Dec 28 14:12:29.249318180 (550h43m29.120401225s) [noise=true , address=145.79.227.22:4511 , piece_id=CENRFEVSOLMISFMAKXW3HPKZA7Y6K72WDFUYJRTOYASV7JNWAFQQ, seg_pos={0 0}, upload=482857]: Done uploading piece. err: <nil>
Dec 28 14:12:29.249642578 (550h43m29.120725613s) [noise=true , address=145.79.227.36:4517 , piece_id=FSKK4MN5R5YCQPGEAUSFLGCKKSMQM6SHR3MU7XW4WGBBUPEHCDTA, seg_pos={0 0}, upload=482857]: Done uploading piece. err: <nil>
Dec 28 14:12:29.268199995 (550h43m29.13928304s) [noise=true , address=145.79.227.47:4267 , piece_id=U6AS4TE4KV4GSBTK6LNX6LJO6SVL6M3NN4G4XE6GWY4N3Y4REAZA, seg_pos={0 0}, upload=482857]: Done uploading piece. err: <nil>
Dec 28 14:12:29.289329168 (550h43m29.160412204s) [noise=true , address=145.79.227.6:4258 , piece_id=TDXLR4XXCVIZRVNKRJAVTI2TV6YIWNC2MDLYKLNO254REB474J5Q, seg_pos={0 0}, upload=482857]: Done uploading piece. err: <nil>
Dec 28 14:12:29.301755379 (550h43m29.172838424s) [noise=true , address=145.79.227.4:4185 , piece_id=WDD5X5TL57YJKVCAC5PVHPLKL7UBLQPRLXQL6AT5W4SGECLSRTWA, seg_pos={0 0}, upload=482857]: Done uploading piece. err: <nil>
Dec 28 14:12:29.329926162 (550h43m29.201009197s) [noise=true , address=145.79.227.7:4294 , piece_id=I5KWL6V3UGP2M75WIH2LD4DDFNIHCPKMB3LQ5AMFDINRXGWZTOGA, seg_pos={0 0}, upload=482857]: Done uploading piece. err: <nil>
Dec 28 14:12:29.336949215 (550h43m29.20803225s) [noise=true , address=145.79.227.23:4546 , piece_id=PXCQYRS4XKGCI75GUD74NTZDS6B5PFDFWHP2UWHO6FFGZ7NYTJNA, seg_pos={0 0}, upload=482857]: Done uploading piece. err: <nil>
Dec 28 14:12:29.350750224 (550h43m29.221833269s) [noise=true , address=145.79.227.12:4765 , piece_id=PTT7UVJF7X6JUAGZSBNLGXCR43BNZPYMTZS26Y6INRMYLRVWW4SA, seg_pos={0 0}, upload=482857]: Done uploading piece. err: <nil>
Dec 28 14:12:29.440347803 (550h43m29.311430838s) [noise=true , address=45.91.116.13:4156 , piece_id=4JXOX4GL4OTZYYDTTEUYIYLKTYTVE6QUKSLAFICMS3J6MUNV5ZEQ, seg_pos={0 0}, upload=482857]: Done uploading piece. err: <nil>
Dec 28 14:12:29.465299544 (550h43m29.336382589s) [noise=true , address=45.91.116.11:4119 , piece_id=DKKYFWRSICCYP6SD7AG4X5Y477QELMOQ7NPDQAW4QEUPOPW2XITQ, seg_pos={0 0}, upload=482857]: Done uploading piece. err: <nil>
Dec 28 14:12:29.465323490 (550h43m29.336406525s) [seg_pos={0 0}, upload=482857]: Segment reached optimal threshold of 13 pieces.
Dec 28 14:12:29.465597592 (550h43m29.336680637s) [noise=true , address=145.79.227.37:4566 , piece_id=AESIVAA37WELHB4UNGKJVH73RW62TFZZYUCSEV6GSFZDRXM2XQGA, seg_pos={0 0}, upload=482857]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F): protocol: expected piece hash; context canceled; EOF
Dec 28 14:12:29.465605868 (550h43m29.336688903s) [noise=true , address=145.79.227.37:4566 , piece_id=AESIVAA37WELHB4UNGKJVH73RW62TFZZYUCSEV6GSFZDRXM2XQGA, seg_pos={0 0}, upload=482857]: Long tail cancellation due to optimal threshold
Dec 28 14:12:29.465747347 (550h43m29.336830392s) [noise=true , address=45.91.116.7:4009 , piece_id=BQEQBPOJ2DEA7SP52JUOBCE65O3CJIPNAOFRX6PCXZV5YYDCCNWQ, seg_pos={0 0}, upload=482857]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF
Dec 28 14:12:29.465757507 (550h43m29.336840542s) [noise=true , address=45.91.116.7:4009 , piece_id=BQEQBPOJ2DEA7SP52JUOBCE65O3CJIPNAOFRX6PCXZV5YYDCCNWQ, seg_pos={0 0}, upload=482857]: Long tail cancellation due to optimal threshold
Dec 28 14:12:29.465767195 (550h43m29.336850241s) [noise=true , address=45.91.116.10:4084 , piece_id=DODVFMCUC2SLGQBNAB2XJ7MUPDY346QOCHERDB2KLUXOZV47PMCQ, seg_pos={0 0}, upload=482857]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF
Dec 28 14:12:29.465775792 (550h43m29.336858827s) [noise=true , address=45.91.116.10:4084 , piece_id=DODVFMCUC2SLGQBNAB2XJ7MUPDY346QOCHERDB2KLUXOZV47PMCQ, seg_pos={0 0}, upload=482857]: Long tail cancellation due to optimal threshold
Dec 28 14:12:29.465793265 (550h43m29.33687631s) [seg_pos={0 0}, upload=482857]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 14:12:29.465897784 (550h43m29.336980829s) [upload=482857]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 14:12:29.465903876 (550h43m29.336986921s) [upload=482857]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 14:12:29.771625107 (550h43m29.642708153s) [upload=482857]: Done uploading segments...
Dec 28 14:12:29.771650276 (550h43m29.642733311s) [upload=482857]: Upload finished. err: <nil>
Duration: 5.730056266s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
20 out of 60 is failed (threshold=3) [..X..X.....X..X.X...XX......X......XXX.XX....XX......XX.XX.X] duration:
30.00351531s trace:
https://jaeger.dev.storj.io/trace/7ed73ea0ef2fcf6e
FAIL (2025-12-28 14:12:56 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 28 14:12:26.955988030 (550h43m26.827071076s) [upload=482857]: Starting upload
Dec 28 14:12:26.956048295 (550h43m26.827131331s) [upload=482857]: Done starting upload
Dec 28 14:12:26.956220924 (550h43m26.827303959s) [upload=482857]: Uploading segments...
Dec 28 14:12:26.956304343 (550h43m26.827387378s) [upload=482857]: Got next segment. Inline: false
Dec 28 14:12:26.956356803 (550h43m26.827439838s) [upload=482857]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 14:12:26.956362854 (550h43m26.827445889s) [upload=482857]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 14:12:27.203346923 (550h43m27.074429968s) [seg_pos={0 0}, upload=482857]: Begin upload segment...
Dec 28 14:12:27.203420483 (550h43m27.074503528s) [seg_pos={0 0}, upload=482857]: Done begin upload segment.
Dec 28 14:12:27.203465909 (550h43m27.074548944s) [noise=true , address=45.91.116.10:4084 , piece_id=DODVFMCUC2SLGQBNAB2XJ7MUPDY346QOCHERDB2KLUXOZV47PMCQ, seg_pos={0 0}, upload=482857]: Uploading piece...
Dec 28 14:12:27.205011863 (550h43m27.076094898s) [noise=true , address=145.79.227.39:4015 , piece_id=BS3WL72BOIM5M5E3CIR6QAQTEBGAAMGL45KTBTNGFFRNQFGAOIGQ, seg_pos={0 0}, upload=482857]: Uploading piece...
Dec 28 14:12:27.205507086 (550h43m27.076590131s) [noise=true , address=145.79.227.36:4517 , piece_id=FSKK4MN5R5YCQPGEAUSFLGCKKSMQM6SHR3MU7XW4WGBBUPEHCDTA, seg_pos={0 0}, upload=482857]: Uploading piece...
Dec 28 14:12:27.206025684 (550h43m27.077108729s) [noise=true , address=145.79.227.7:4294 , piece_id=I5KWL6V3UGP2M75WIH2LD4DDFNIHCPKMB3LQ5AMFDINRXGWZTOGA, seg_pos={0 0}, upload=482857]: Uploading piece...
Dec 28 14:12:27.207686346 (550h43m27.078769401s) [noise=true , address=145.79.227.23:4546 , piece_id=PXCQYRS4XKGCI75GUD74NTZDS6B5PFDFWHP2UWHO6FFGZ7NYTJNA, seg_pos={0 0}, upload=482857]: Uploading piece...
Dec 28 14:12:27.209137098 (550h43m27.080220143s) [noise=true , address=45.91.116.13:4156 , piece_id=4JXOX4GL4OTZYYDTTEUYIYLKTYTVE6QUKSLAFICMS3J6MUNV5ZEQ, seg_pos={0 0}, upload=482857]: Uploading piece...
Dec 28 14:12:27.209241617 (550h43m27.080324662s) [noise=true , address=145.79.227.13:4802 , piece_id=KGJBMKAOESBSIRTFJASM4AVLIXTULAFQIDLC7BJOW4QNA7STADDQ, seg_pos={0 0}, upload=482857]: Uploading piece...
Dec 28 14:12:27.211032798 (550h43m27.082115833s) [noise=true , address=45.91.116.7:4009 , piece_id=BQEQBPOJ2DEA7SP52JUOBCE65O3CJIPNAOFRX6PCXZV5YYDCCNWQ, seg_pos={0 0}, upload=482857]: Uploading piece...
Dec 28 14:12:27.212335017 (550h43m27.083418062s) [noise=true , address=145.79.227.47:4267 , piece_id=U6AS4TE4KV4GSBTK6LNX6LJO6SVL6M3NN4G4XE6GWY4N3Y4REAZA, seg_pos={0 0}, upload=482857]: Uploading piece...
Dec 28 14:12:27.214760837 (550h43m27.085843872s) [noise=true , address=45.91.116.11:4119 , piece_id=DKKYFWRSICCYP6SD7AG4X5Y477QELMOQ7NPDQAW4QEUPOPW2XITQ, seg_pos={0 0}, upload=482857]: Uploading piece...
Dec 28 14:12:27.215345150 (550h43m27.086428195s) [noise=true , address=145.79.227.22:4511 , piece_id=CENRFEVSOLMISFMAKXW3HPKZA7Y6K72WDFUYJRTOYASV7JNWAFQQ, seg_pos={0 0}, upload=482857]: Uploading piece...
Dec 28 14:12:27.217163823 (550h43m27.088246868s) [noise=true , address=145.79.227.4:4185 , piece_id=WDD5X5TL57YJKVCAC5PVHPLKL7UBLQPRLXQL6AT5W4SGECLSRTWA, seg_pos={0 0}, upload=482857]: Uploading piece...
Dec 28 14:12:27.217605494 (550h43m27.088688539s) [noise=true , address=145.79.227.6:4258 , piece_id=TDXLR4XXCVIZRVNKRJAVTI2TV6YIWNC2MDLYKLNO254REB474J5Q, seg_pos={0 0}, upload=482857]: Uploading piece...
Dec 28 14:12:27.218542177 (550h43m27.089625222s) [noise=true , address=145.79.227.25:4618 , piece_id=PBB23LSOUMNTZNKGK43PCUVA4YADXDA4V6YZWJXD34XXGYINNGZA, seg_pos={0 0}, upload=482857]: Uploading piece...
Dec 28 14:12:27.220450450 (550h43m27.091533495s) [noise=true , address=145.79.227.37:4566 , piece_id=AESIVAA37WELHB4UNGKJVH73RW62TFZZYUCSEV6GSFZDRXM2XQGA, seg_pos={0 0}, upload=482857]: Uploading piece...
Dec 28 14:12:27.221721161 (550h43m27.092804206s) [noise=true , address=145.79.227.12:4765 , piece_id=PTT7UVJF7X6JUAGZSBNLGXCR43BNZPYMTZS26Y6INRMYLRVWW4SA, seg_pos={0 0}, upload=482857]: Uploading piece...
Dec 28 14:12:28.836487613 (550h43m28.707570658s) [upload=482857]: Next returned nil segment
Dec 28 14:12:28.836525865 (550h43m28.7076089s) [upload=482857]: Waiting for error group managing segments...
Dec 28 14:12:29.222239759 (550h43m29.093322814s) [noise=true , address=145.79.227.13:4802 , piece_id=KGJBMKAOESBSIRTFJASM4AVLIXTULAFQIDLC7BJOW4QNA7STADDQ, seg_pos={0 0}, upload=482857]: Done uploading piece. err: <nil>
Dec 28 14:12:29.235763600 (550h43m29.106846645s) [noise=true , address=145.79.227.25:4618 , piece_id=PBB23LSOUMNTZNKGK43PCUVA4YADXDA4V6YZWJXD34XXGYINNGZA, seg_pos={0 0}, upload=482857]: Done uploading piece. err: <nil>
Dec 28 14:12:29.238585754 (550h43m29.109668799s) [noise=true , address=145.79.227.39:4015 , piece_id=BS3WL72BOIM5M5E3CIR6QAQTEBGAAMGL45KTBTNGFFRNQFGAOIGQ, seg_pos={0 0}, upload=482857]: Done uploading piece. err: <nil>
Dec 28 14:12:29.249318180 (550h43m29.120401225s) [noise=true , address=145.79.227.22:4511 , piece_id=CENRFEVSOLMISFMAKXW3HPKZA7Y6K72WDFUYJRTOYASV7JNWAFQQ, seg_pos={0 0}, upload=482857]: Done uploading piece. err: <nil>
Dec 28 14:12:29.249642578 (550h43m29.120725613s) [noise=true , address=145.79.227.36:4517 , piece_id=FSKK4MN5R5YCQPGEAUSFLGCKKSMQM6SHR3MU7XW4WGBBUPEHCDTA, seg_pos={0 0}, upload=482857]: Done uploading piece. err: <nil>
Dec 28 14:12:29.268199995 (550h43m29.13928304s) [noise=true , address=145.79.227.47:4267 , piece_id=U6AS4TE4KV4GSBTK6LNX6LJO6SVL6M3NN4G4XE6GWY4N3Y4REAZA, seg_pos={0 0}, upload=482857]: Done uploading piece. err: <nil>
Dec 28 14:12:29.289329168 (550h43m29.160412204s) [noise=true , address=145.79.227.6:4258 , piece_id=TDXLR4XXCVIZRVNKRJAVTI2TV6YIWNC2MDLYKLNO254REB474J5Q, seg_pos={0 0}, upload=482857]: Done uploading piece. err: <nil>
Dec 28 14:12:29.301755379 (550h43m29.172838424s) [noise=true , address=145.79.227.4:4185 , piece_id=WDD5X5TL57YJKVCAC5PVHPLKL7UBLQPRLXQL6AT5W4SGECLSRTWA, seg_pos={0 0}, upload=482857]: Done uploading piece. err: <nil>
Dec 28 14:12:29.329926162 (550h43m29.201009197s) [noise=true , address=145.79.227.7:4294 , piece_id=I5KWL6V3UGP2M75WIH2LD4DDFNIHCPKMB3LQ5AMFDINRXGWZTOGA, seg_pos={0 0}, upload=482857]: Done uploading piece. err: <nil>
Dec 28 14:12:29.336949215 (550h43m29.20803225s) [noise=true , address=145.79.227.23:4546 , piece_id=PXCQYRS4XKGCI75GUD74NTZDS6B5PFDFWHP2UWHO6FFGZ7NYTJNA, seg_pos={0 0}, upload=482857]: Done uploading piece. err: <nil>
Dec 28 14:12:29.350750224 (550h43m29.221833269s) [noise=true , address=145.79.227.12:4765 , piece_id=PTT7UVJF7X6JUAGZSBNLGXCR43BNZPYMTZS26Y6INRMYLRVWW4SA, seg_pos={0 0}, upload=482857]: Done uploading piece. err: <nil>
Dec 28 14:12:29.440347803 (550h43m29.311430838s) [noise=true , address=45.91.116.13:4156 , piece_id=4JXOX4GL4OTZYYDTTEUYIYLKTYTVE6QUKSLAFICMS3J6MUNV5ZEQ, seg_pos={0 0}, upload=482857]: Done uploading piece. err: <nil>
Dec 28 14:12:29.465299544 (550h43m29.336382589s) [noise=true , address=45.91.116.11:4119 , piece_id=DKKYFWRSICCYP6SD7AG4X5Y477QELMOQ7NPDQAW4QEUPOPW2XITQ, seg_pos={0 0}, upload=482857]: Done uploading piece. err: <nil>
Dec 28 14:12:29.465323490 (550h43m29.336406525s) [seg_pos={0 0}, upload=482857]: Segment reached optimal threshold of 13 pieces.
Dec 28 14:12:29.465597592 (550h43m29.336680637s) [noise=true , address=145.79.227.37:4566 , piece_id=AESIVAA37WELHB4UNGKJVH73RW62TFZZYUCSEV6GSFZDRXM2XQGA, seg_pos={0 0}, upload=482857]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F): protocol: expected piece hash; context canceled; EOF
Dec 28 14:12:29.465605868 (550h43m29.336688903s) [noise=true , address=145.79.227.37:4566 , piece_id=AESIVAA37WELHB4UNGKJVH73RW62TFZZYUCSEV6GSFZDRXM2XQGA, seg_pos={0 0}, upload=482857]: Long tail cancellation due to optimal threshold
Dec 28 14:12:29.465747347 (550h43m29.336830392s) [noise=true , address=45.91.116.7:4009 , piece_id=BQEQBPOJ2DEA7SP52JUOBCE65O3CJIPNAOFRX6PCXZV5YYDCCNWQ, seg_pos={0 0}, upload=482857]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF
Dec 28 14:12:29.465757507 (550h43m29.336840542s) [noise=true , address=45.91.116.7:4009 , piece_id=BQEQBPOJ2DEA7SP52JUOBCE65O3CJIPNAOFRX6PCXZV5YYDCCNWQ, seg_pos={0 0}, upload=482857]: Long tail cancellation due to optimal threshold
Dec 28 14:12:29.465767195 (550h43m29.336850241s) [noise=true , address=45.91.116.10:4084 , piece_id=DODVFMCUC2SLGQBNAB2XJ7MUPDY346QOCHERDB2KLUXOZV47PMCQ, seg_pos={0 0}, upload=482857]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF
Dec 28 14:12:29.465775792 (550h43m29.336858827s) [noise=true , address=45.91.116.10:4084 , piece_id=DODVFMCUC2SLGQBNAB2XJ7MUPDY346QOCHERDB2KLUXOZV47PMCQ, seg_pos={0 0}, upload=482857]: Long tail cancellation due to optimal threshold
Dec 28 14:12:29.465793265 (550h43m29.33687631s) [seg_pos={0 0}, upload=482857]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 14:12:29.465897784 (550h43m29.336980829s) [upload=482857]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 14:12:29.465903876 (550h43m29.336986921s) [upload=482857]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 14:12:29.771625107 (550h43m29.642708153s) [upload=482857]: Done uploading segments...
Dec 28 14:12:29.771650276 (550h43m29.642733311s) [upload=482857]: Upload finished. err: <nil>
Duration: 30.003539816s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [..............X.............................................] duration:
11.616631098s trace:
https://jaeger.dev.storj.io/trace/5fe9d6dbb79fbe44
FAIL (2025-12-28 14:12:30 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
67.204.205:4027 , piece_id=PAPNMRJ7E47VOMOXPBBDWKZJHLRMXTWVR5VW3J7KNQSSMHOHWOAQ, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.967737953 (550h8m4.838820988s) [noise=true , address=185.167.204.206:4072 , piece_id=WXORBDMA3KW7WZ67A4FJT6URJRFADUJ7X2PAHIGNG47T3TGMXVTQ, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.968193240 (550h8m4.839276285s) [noise=true , address=185.167.204.205:4030 , piece_id=R5DLI4ZTLOVGO56GPLJL5ZNHYSXOSYNJQHFIO4MVVNHWWBMEOV2Q, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.968780638 (550h8m4.839863683s) [noise=true , address=89.149.201.147:4019 , piece_id=MK7ORCLHO3DOWIOVSOWIQMPJ447PHRQGXP3UKITIRTYJSU5MPCCQ, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.969308924 (550h8m4.840391959s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.970088349 (550h8m4.841171404s) [noise=true , address=81.171.14.70:4039 , piece_id=EDHPQWLQZTP7MKPGEVMP7JXNTW7LFWSXCTRTSXPCBIQPROX76WRA, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.970325381 (550h8m4.841408426s) [noise=true , address=81.171.14.70:4036 , piece_id=CICNNV7JE6TBNC4RAPXKCEUSJAADOUVGSVIVQISM3ZB4COAZQDDA, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.970783813 (550h8m4.841866858s) [noise=true , address=81.171.14.70:4044 , piece_id=DGN74IG6IWAHLEMO6NLL4TJBXZ6ONTN6QUB6HXY6K7SSO3LYOYIQ, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.971205406 (550h8m4.842288441s) [noise=true , address=81.171.14.70:4048 , piece_id=TLT47ORGUKGF3MIIEAX7D22VR2FO6W6NL3VFJNHKWD4L3LIMFDLA, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.971606299 (550h8m4.842689344s) [noise=true , address=185.167.204.206:4067 , piece_id=VSKTG5RH32N7OEVWJGQVB6WJ72TEVF6DDK25W7NOH4RAXUP52LTQ, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.972162538 (550h8m4.843245583s) [noise=true , address=89.149.201.147:4002 , piece_id=G2PABNTDWCK4MKOZF6PONBMRNIXI45JTHJMAZFPEKPEJ3PQ7MTBA, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:05.795167814 (550h8m5.666250849s) [upload=482337]: Next returned nil segment
Dec 28 13:37:05.795184566 (550h8m5.666267611s) [upload=482337]: Waiting for error group managing segments...
Dec 28 13:37:06.178573801 (550h8m6.049656836s) [noise=true , address=185.167.204.206:4049 , piece_id=6RG2DZV6FPTIRE63ZNNST42TSWEBQ456YXIPPVYHIXV55GWFVJ4Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.183060835 (550h8m6.05414391s) [noise=true , address=185.167.204.206:4072 , piece_id=WXORBDMA3KW7WZ67A4FJT6URJRFADUJ7X2PAHIGNG47T3TGMXVTQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.183411202 (550h8m6.054494237s) [noise=true , address=185.167.204.206:4064 , piece_id=OG3G74OU23CEMKA54WQE3DOJ6AKAXBLZ23QWOQC4UEGQQN4JSMXQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.183791146 (550h8m6.054874181s) [noise=true , address=185.167.204.205:4029 , piece_id=6JQB6ZSCQZKWH3JWCQ4DY6BBAX73KC4Y53PRSMEMLWURZ5QWWQQQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.183973743 (550h8m6.055056778s) [noise=true , address=185.167.204.206:4053 , piece_id=YDH77S6JLKJ6IYS6DRUCELP5EJHIC2O44QBO7MIOEUYBML4GIBHA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.184465369 (550h8m6.055548414s) [noise=true , address=185.167.204.205:4008 , piece_id=RTUZPEXBB6JNSMC5YOFQQGDM57YD233W4756ZBQVMFNOVX6YKB4A, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.184501989 (550h8m6.055585024s) [noise=true , address=185.167.204.205:4030 , piece_id=R5DLI4ZTLOVGO56GPLJL5ZNHYSXOSYNJQHFIO4MVVNHWWBMEOV2Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.184856224 (550h8m6.055939259s) [noise=true , address=185.167.204.205:4009 , piece_id=V6RA7Y7FKY4DCPTRRF3BYNBALWDY5VTPXMDUXFE3353KUXLSQJEA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.185063518 (550h8m6.056146563s) [noise=true , address=185.167.204.205:4016 , piece_id=JB7SC3J2VR4U33HNQS2A2CA75THT3XLQCOIFHAJJEK7MQGY2SYAQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.185288617 (550h8m6.056371652s) [noise=true , address=185.167.204.205:4027 , piece_id=PAPNMRJ7E47VOMOXPBBDWKZJHLRMXTWVR5VW3J7KNQSSMHOHWOAQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.186241401 (550h8m6.057324436s) [noise=true , address=185.167.204.205:4024 , piece_id=FM5IMWRSJJ4RY72IX2OOYC54BFKL4FWE3UOAN3GJ26MLH345COVA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.193113658 (550h8m6.064196703s) [noise=true , address=185.167.204.205:4005 , piece_id=R4DFQJZBPJ5WJWHPODV6ILHUDIRBUSAFOMVNCLXB4UNMI6EXTKHA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.193791408 (550h8m6.064874453s) [noise=true , address=185.167.204.206:4068 , piece_id=DAI37M67FWCSVQTQOZ6RSSOJDGUVZO6SNGAHJJ44G3PGS4OJZEMA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.194292262 (550h8m6.065375307s) [noise=true , address=185.167.204.206:4046 , piece_id=VWLVHMBMJ63DQEWYBIKH3Y3NTVWFNW5OXZKMYXXBHVI3VWYXLLPQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.194302652 (550h8m6.065385687s) [noise=true , address=185.167.204.206:4067 , piece_id=VSKTG5RH32N7OEVWJGQVB6WJ72TEVF6DDK25W7NOH4RAXUP52LTQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.195359193 (550h8m6.066442228s) [noise=true , address=185.167.204.205:4012 , piece_id=LV6RUA5BWXG4DMSUVZDBGUAF5IBWUGMLJYUKNIXHRVDEBIOW4Q2Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.198423448 (550h8m6.069506493s) [noise=true , address=185.167.204.206:4070 , piece_id=5A7YYQYREIZRZDR7TRSE7XFW765JVAV7FX6LKWUI2OXAA74XN4CA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.888968408 (550h8m6.760051453s) [noise=true , address=81.171.14.70:4044 , piece_id=DGN74IG6IWAHLEMO6NLL4TJBXZ6ONTN6QUB6HXY6K7SSO3LYOYIQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.892350789 (550h8m6.763433834s) [noise=true , address=81.171.14.70:4039 , piece_id=EDHPQWLQZTP7MKPGEVMP7JXNTW7LFWSXCTRTSXPCBIQPROX76WRA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.904477160 (550h8m6.775560195s) [noise=true , address=81.171.14.70:4040 , piece_id=DYF4GSZBAKQ3H63LTVCMEFWMABY3FXHOJCNVIFYRKLF2J6B52K5A, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.128088281 (550h8m6.999171326s) [noise=true , address=81.171.14.70:4036 , piece_id=CICNNV7JE6TBNC4RAPXKCEUSJAADOUVGSVIVQISM3ZB4COAZQDDA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.430242142 (550h8m7.301325177s) [noise=true , address=89.149.201.147:4009 , piece_id=EVW2NQFYU57PC2IIVXLYIEBMYXJNIC2CLCHW6PS6NYIPCFRUVHDA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.505304332 (550h8m7.376387367s) [noise=true , address=89.149.201.147:4019 , piece_id=MK7ORCLHO3DOWIOVSOWIQMPJ447PHRQGXP3UKITIRTYJSU5MPCCQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.575461543 (550h8m7.446544609s) [noise=true , address=81.171.14.70:4048 , piece_id=TLT47ORGUKGF3MIIEAX7D22VR2FO6W6NL3VFJNHKWD4L3LIMFDLA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.599667265 (550h8m7.4707503s) [noise=true , address=89.149.201.147:4023 , piece_id=5IINX2GVRFLRX2QH24OO2DDANOKOTXTDR6AGVJBGSJ7IAZQ3CASA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.928390728 (550h8m7.799473773s) [noise=true , address=89.149.201.147:4002 , piece_id=G2PABNTDWCK4MKOZF6PONBMRNIXI45JTHJMAZFPEKPEJ3PQ7MTBA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.928417148 (550h8m7.799500193s) [seg_pos={0 0}, upload=482337]: Segment reached optimal threshold of 26 pieces.
Dec 28 13:37:07.928656303 (550h8m7.799739348s) [noise=true , address=81.171.14.70:4030 , piece_id=74YDIKUST7X3JBXVQHVLFHIKH5A4DYH5TNL3I4XUA5A6AHQO2DRA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZoRwa8wLw66Pjumy1ybp6wrZ79RrSU55uYZpwaY3xi95iSYde): protocol: expected piece hash; context canceled; EOF
Dec 28 13:37:07.928664990 (550h8m7.799748035s) [noise=true , address=81.171.14.70:4030 , piece_id=74YDIKUST7X3JBXVQHVLFHIKH5A4DYH5TNL3I4XUA5A6AHQO2DRA, seg_pos={0 0}, upload=482337]: Long tail cancellation due to optimal threshold
Dec 28 13:37:07.928723351 (550h8m7.799806386s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wgtphHKaTejC1qrJ98sDysBwtj2Zeqw2c5UFk1HMqfY8Buvr): protocol: expected piece hash; context canceled; EOF
Dec 28 13:37:07.928733922 (550h8m7.799816957s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Long tail cancellation due to optimal threshold
Dec 28 13:37:07.928754921 (550h8m7.799837966s) [seg_pos={0 0}, upload=482337]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 28 13:37:07.928870170 (550h8m7.799953215s) [upload=482337]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 13:37:07.928878627 (550h8m7.799961672s) [upload=482337]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 13:37:08.166321536 (550h8m8.037404571s) [upload=482337]: Done uploading segments...
Dec 28 13:37:08.166341444 (550h8m8.037424479s) [upload=482337]: Upload finished. err: <nil>
Duration: 11.616670783s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [.....X....................................X........X...X....] duration:
33.309863203s trace:
https://jaeger.dev.storj.io/trace/36d5af3c761e71b1
FAIL (2025-12-28 14:12:29 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
7.49:4360 , piece_id=QDT5N5LDL2HCVXPOOSXWD2TOEZDF2JZWFLDFYSIMN647FI3ZORUQ, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.372193736 (550h40m10.243276781s) [noise=true , address=77.90.31.12:4618 , piece_id=45WRZ26CPUODOAYF5B6RF37DNMDDQWWJUDK7BFKYS3K3LOCNYJCA, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.372518505 (550h40m10.24360155s) [noise=true , address=77.90.31.13:4646 , piece_id=5CDMQF7GG5PDPHIPMFZWDBKWH5ANO3UIX257LJPCWRITNOA7GY4A, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.372848052 (550h40m10.243931097s) [noise=true , address=64.49.13.170:4721 , piece_id=U76TI7IPAHPZAS6XLG66E4L2CGCYH43NH4ZHTYXMXKZR6SGE7SKQ, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.373163072 (550h40m10.244246117s) [noise=true , address=77.90.31.11:4565 , piece_id=4SBLRZK3K4ED73PJ64BI6ANKBWB2AAHQA4SBUNH6CJ63GN3FJR2Q, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.373445089 (550h40m10.244528134s) [noise=true , address=145.79.227.34:4911 , piece_id=24WE4EMZSL2JHQZ7PESCMLYCZOWGDTGJD2XNLUUJEFPSYLRI5E7A, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.373768855 (550h40m10.24485189s) [noise=true , address=77.90.31.18:4819 , piece_id=OYXPAN3EK4QRDEC6VHIDL3GSGEY27G72KII6HTGUMSPYLHCYBNEA, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.374285519 (550h40m10.245368564s) [noise=true , address=77.90.31.14:4678 , piece_id=VBGPSGMNDQZYY7UKRINAV6B3FVI5BPDPYS37FQIJCMLL65F3XOEQ, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.374710689 (550h40m10.245793724s) [noise=true , address=145.79.227.40:4053 , piece_id=RO4ZZQCKCC7T4T6SOT7GGTY626BJ6ESLQG6KATTPNIUM3ICUKHAQ, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.375099329 (550h40m10.246182364s) [noise=true , address=77.90.31.19:4859 , piece_id=A4HD3FPDPCCMHRFBTS3GNGB323RYD5C5AW4ZVSNOTXFI7FLIV37Q, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.376073433 (550h40m10.247156468s) [noise=true , address=77.90.31.15:4704 , piece_id=DXWLFOIKOPE33LMEQECUJVIGRG3LPADUMULLWI4ZBHEOQHD43ULQ, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.376542387 (550h40m10.247625422s) [noise=true , address=145.79.227.44:4221 , piece_id=H72XCWLKYJAXYGGDIWN67H42RUIEOJCPUSWT5IJMFPKETH7LK3KA, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.377379761 (550h40m10.248462796s) [noise=true , address=145.79.227.32:4863 , piece_id=27TOXK44RIF3DV3PDY4RBP6HEFRQJE6N7LNLRJKSU52YCB6A3JKA, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.378022765 (550h40m10.24910581s) [noise=true , address=145.79.227.30:4789 , piece_id=4QT5IUPIK7BOHJROEK2EBFTW434OKC75FZNENMYYCRMBZ7E5BUTA, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.378821687 (550h40m10.249904722s) [noise=true , address=145.79.227.2:4119 , piece_id=EQM3WMBQ7GXRWRBZK4ZWCI23K4LSLXW6QPEABHGMWRHEYUYANKSQ, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:11.676223695 (550h40m11.54730674s) [upload=482805]: Next returned nil segment
Dec 28 14:09:11.676238083 (550h40m11.547321128s) [upload=482805]: Waiting for error group managing segments...
Dec 28 14:09:12.006762275 (550h40m11.87784531s) [noise=true , address=64.49.13.170:4721 , piece_id=U76TI7IPAHPZAS6XLG66E4L2CGCYH43NH4ZHTYXMXKZR6SGE7SKQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.250307967 (550h40m12.121391012s) [noise=true , address=145.79.227.2:4119 , piece_id=EQM3WMBQ7GXRWRBZK4ZWCI23K4LSLXW6QPEABHGMWRHEYUYANKSQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.269380734 (550h40m12.140463769s) [noise=true , address=145.79.227.49:4360 , piece_id=QDT5N5LDL2HCVXPOOSXWD2TOEZDF2JZWFLDFYSIMN647FI3ZORUQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.290818335 (550h40m12.16190138s) [noise=true , address=145.79.227.32:4863 , piece_id=27TOXK44RIF3DV3PDY4RBP6HEFRQJE6N7LNLRJKSU52YCB6A3JKA, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.467679327 (550h40m12.338762362s) [noise=true , address=45.91.116.8:4072 , piece_id=GEVTX6LLENPLN3UQLSGTFTP3DO52IRYUIQFIWC4XMOC3ZF6V4GTQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.486615595 (550h40m12.35769863s) [noise=true , address=77.90.31.12:4618 , piece_id=45WRZ26CPUODOAYF5B6RF37DNMDDQWWJUDK7BFKYS3K3LOCNYJCA, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.566542810 (550h40m12.437625845s) [noise=true , address=77.90.31.19:4859 , piece_id=A4HD3FPDPCCMHRFBTS3GNGB323RYD5C5AW4ZVSNOTXFI7FLIV37Q, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.609337578 (550h40m12.480420623s) [noise=true , address=77.90.31.13:4646 , piece_id=5CDMQF7GG5PDPHIPMFZWDBKWH5ANO3UIX257LJPCWRITNOA7GY4A, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.640856565 (550h40m12.511939611s) [noise=true , address=45.91.116.5:4322 , piece_id=SS6DOCSJHY5Y3DPWMYX45F4VYYXR44JDLYQ2YPY5XIIAYKMAG7LQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.645490329 (550h40m12.516573364s) [noise=true , address=77.90.31.18:4819 , piece_id=OYXPAN3EK4QRDEC6VHIDL3GSGEY27G72KII6HTGUMSPYLHCYBNEA, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:13.103710286 (550h40m12.974793331s) [noise=true , address=145.79.227.34:4911 , piece_id=24WE4EMZSL2JHQZ7PESCMLYCZOWGDTGJD2XNLUUJEFPSYLRI5E7A, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:13.209917141 (550h40m13.081000176s) [noise=true , address=145.79.227.44:4221 , piece_id=H72XCWLKYJAXYGGDIWN67H42RUIEOJCPUSWT5IJMFPKETH7LK3KA, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:13.218355468 (550h40m13.089438514s) [noise=true , address=77.90.31.11:4565 , piece_id=4SBLRZK3K4ED73PJ64BI6ANKBWB2AAHQA4SBUNH6CJ63GN3FJR2Q, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:13.255968741 (550h40m13.127051806s) [noise=true , address=145.79.227.30:4789 , piece_id=4QT5IUPIK7BOHJROEK2EBFTW434OKC75FZNENMYYCRMBZ7E5BUTA, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:13.255996102 (550h40m13.127079137s) [seg_pos={0 0}, upload=482805]: Segment reached optimal threshold of 14 pieces.
Dec 28 14:09:13.256214589 (550h40m13.127297634s) [noise=true , address=145.79.227.40:4053 , piece_id=RO4ZZQCKCC7T4T6SOT7GGTY626BJ6ESLQG6KATTPNIUM3ICUKHAQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DqyTc4MHvxutFXs2WWRCk8MSuxF3Y1HayyQY4Y5sy8cqDLWuG): protocol: expected piece hash; context canceled; EOF
Dec 28 14:09:13.256223686 (550h40m13.127306731s) [noise=true , address=145.79.227.40:4053 , piece_id=RO4ZZQCKCC7T4T6SOT7GGTY626BJ6ESLQG6KATTPNIUM3ICUKHAQ, seg_pos={0 0}, upload=482805]: Long tail cancellation due to optimal threshold
Dec 28 14:09:13.256512506 (550h40m13.127595551s) [noise=true , address=77.90.31.15:4704 , piece_id=DXWLFOIKOPE33LMEQECUJVIGRG3LPADUMULLWI4ZBHEOQHD43ULQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14tJRdZ1guwwsYwPf3LtENRWqYx2jBQpi7uAMMPaB32Gks9zmj): protocol: expected piece hash; context canceled; EOF
Dec 28 14:09:13.256520060 (550h40m13.127603105s) [noise=true , address=77.90.31.15:4704 , piece_id=DXWLFOIKOPE33LMEQECUJVIGRG3LPADUMULLWI4ZBHEOQHD43ULQ, seg_pos={0 0}, upload=482805]: Long tail cancellation due to optimal threshold
Dec 28 14:09:13.256649106 (550h40m13.127732151s) [noise=true , address=77.90.31.14:4678 , piece_id=VBGPSGMNDQZYY7UKRINAV6B3FVI5BPDPYS37FQIJCMLL65F3XOEQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12beu3SUZMPNWfbeHV4tLtx9UVZoEHh4Kt7oBhT59w3bGBE1ovP): protocol: write tcp 188.166.225.170:57374->77.90.31.14:4678: use of closed network connection; write tcp 188.166.225.170:57374->77.90.31.14:4678: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57374->77.90.31.14:4678: use of closed network connection
Dec 28 14:09:13.256655819 (550h40m13.127738864s) [noise=true , address=77.90.31.14:4678 , piece_id=VBGPSGMNDQZYY7UKRINAV6B3FVI5BPDPYS37FQIJCMLL65F3XOEQ, seg_pos={0 0}, upload=482805]: Long tail cancellation due to optimal threshold
Dec 28 14:09:13.256752974 (550h40m13.127836009s) [noise=true , address=64.49.13.169:4710 , piece_id=SANL7TVJFES63XD46OKQL4KIJ6YVHSEWPZKMRISEKC3ZKWQBOMIA, seg_pos={0 0}, upload=482805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:35680->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:35680->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35680->64.49.13.169:4710: use of closed network connection
Dec 28 14:09:13.256758825 (550h40m13.12784186s) [noise=true , address=64.49.13.169:4710 , piece_id=SANL7TVJFES63XD46OKQL4KIJ6YVHSEWPZKMRISEKC3ZKWQBOMIA, seg_pos={0 0}, upload=482805]: Long tail cancellation due to optimal threshold
Dec 28 14:09:13.256776549 (550h40m13.127859594s) [seg_pos={0 0}, upload=482805]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 14:09:13.256856591 (550h40m13.127939626s) [upload=482805]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 14:09:13.256862432 (550h40m13.127945467s) [upload=482805]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 14:09:13.550363717 (550h40m13.421446782s) [upload=482805]: Done uploading segments...
Dec 28 14:09:13.550405126 (550h40m13.421488161s) [upload=482805]: Upload finished. err: <nil>
Duration: 33.309877752s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
52 out of 60 is failed (threshold=6) [XX.XXXXXXX.XXXXXXXXXXXX.XXXXXXX.XX..XXXXXXXXXXXXXXXXXXXXX.X.] duration:
3.544922299s trace:
https://jaeger.dev.storj.io/trace/92a90438d044faa
FAIL (2025-12-28 14:12:23 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.545077304s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [.....................................................X...X..] duration:
19.791847525s trace:
https://jaeger.dev.storj.io/trace/102cf391e061ae77
FAIL (2025-12-28 14:12:09 UTC)
orbiter warning: context deadline exceeded
Duration: 19.791912518s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [......X....................................X........X...X...] duration:
40.505540258s trace:
https://jaeger.dev.storj.io/trace/36d5af3c761e71b1
FAIL (2025-12-28 14:11:56 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
7.49:4360 , piece_id=QDT5N5LDL2HCVXPOOSXWD2TOEZDF2JZWFLDFYSIMN647FI3ZORUQ, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.372193736 (550h40m10.243276781s) [noise=true , address=77.90.31.12:4618 , piece_id=45WRZ26CPUODOAYF5B6RF37DNMDDQWWJUDK7BFKYS3K3LOCNYJCA, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.372518505 (550h40m10.24360155s) [noise=true , address=77.90.31.13:4646 , piece_id=5CDMQF7GG5PDPHIPMFZWDBKWH5ANO3UIX257LJPCWRITNOA7GY4A, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.372848052 (550h40m10.243931097s) [noise=true , address=64.49.13.170:4721 , piece_id=U76TI7IPAHPZAS6XLG66E4L2CGCYH43NH4ZHTYXMXKZR6SGE7SKQ, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.373163072 (550h40m10.244246117s) [noise=true , address=77.90.31.11:4565 , piece_id=4SBLRZK3K4ED73PJ64BI6ANKBWB2AAHQA4SBUNH6CJ63GN3FJR2Q, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.373445089 (550h40m10.244528134s) [noise=true , address=145.79.227.34:4911 , piece_id=24WE4EMZSL2JHQZ7PESCMLYCZOWGDTGJD2XNLUUJEFPSYLRI5E7A, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.373768855 (550h40m10.24485189s) [noise=true , address=77.90.31.18:4819 , piece_id=OYXPAN3EK4QRDEC6VHIDL3GSGEY27G72KII6HTGUMSPYLHCYBNEA, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.374285519 (550h40m10.245368564s) [noise=true , address=77.90.31.14:4678 , piece_id=VBGPSGMNDQZYY7UKRINAV6B3FVI5BPDPYS37FQIJCMLL65F3XOEQ, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.374710689 (550h40m10.245793724s) [noise=true , address=145.79.227.40:4053 , piece_id=RO4ZZQCKCC7T4T6SOT7GGTY626BJ6ESLQG6KATTPNIUM3ICUKHAQ, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.375099329 (550h40m10.246182364s) [noise=true , address=77.90.31.19:4859 , piece_id=A4HD3FPDPCCMHRFBTS3GNGB323RYD5C5AW4ZVSNOTXFI7FLIV37Q, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.376073433 (550h40m10.247156468s) [noise=true , address=77.90.31.15:4704 , piece_id=DXWLFOIKOPE33LMEQECUJVIGRG3LPADUMULLWI4ZBHEOQHD43ULQ, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.376542387 (550h40m10.247625422s) [noise=true , address=145.79.227.44:4221 , piece_id=H72XCWLKYJAXYGGDIWN67H42RUIEOJCPUSWT5IJMFPKETH7LK3KA, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.377379761 (550h40m10.248462796s) [noise=true , address=145.79.227.32:4863 , piece_id=27TOXK44RIF3DV3PDY4RBP6HEFRQJE6N7LNLRJKSU52YCB6A3JKA, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.378022765 (550h40m10.24910581s) [noise=true , address=145.79.227.30:4789 , piece_id=4QT5IUPIK7BOHJROEK2EBFTW434OKC75FZNENMYYCRMBZ7E5BUTA, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.378821687 (550h40m10.249904722s) [noise=true , address=145.79.227.2:4119 , piece_id=EQM3WMBQ7GXRWRBZK4ZWCI23K4LSLXW6QPEABHGMWRHEYUYANKSQ, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:11.676223695 (550h40m11.54730674s) [upload=482805]: Next returned nil segment
Dec 28 14:09:11.676238083 (550h40m11.547321128s) [upload=482805]: Waiting for error group managing segments...
Dec 28 14:09:12.006762275 (550h40m11.87784531s) [noise=true , address=64.49.13.170:4721 , piece_id=U76TI7IPAHPZAS6XLG66E4L2CGCYH43NH4ZHTYXMXKZR6SGE7SKQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.250307967 (550h40m12.121391012s) [noise=true , address=145.79.227.2:4119 , piece_id=EQM3WMBQ7GXRWRBZK4ZWCI23K4LSLXW6QPEABHGMWRHEYUYANKSQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.269380734 (550h40m12.140463769s) [noise=true , address=145.79.227.49:4360 , piece_id=QDT5N5LDL2HCVXPOOSXWD2TOEZDF2JZWFLDFYSIMN647FI3ZORUQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.290818335 (550h40m12.16190138s) [noise=true , address=145.79.227.32:4863 , piece_id=27TOXK44RIF3DV3PDY4RBP6HEFRQJE6N7LNLRJKSU52YCB6A3JKA, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.467679327 (550h40m12.338762362s) [noise=true , address=45.91.116.8:4072 , piece_id=GEVTX6LLENPLN3UQLSGTFTP3DO52IRYUIQFIWC4XMOC3ZF6V4GTQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.486615595 (550h40m12.35769863s) [noise=true , address=77.90.31.12:4618 , piece_id=45WRZ26CPUODOAYF5B6RF37DNMDDQWWJUDK7BFKYS3K3LOCNYJCA, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.566542810 (550h40m12.437625845s) [noise=true , address=77.90.31.19:4859 , piece_id=A4HD3FPDPCCMHRFBTS3GNGB323RYD5C5AW4ZVSNOTXFI7FLIV37Q, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.609337578 (550h40m12.480420623s) [noise=true , address=77.90.31.13:4646 , piece_id=5CDMQF7GG5PDPHIPMFZWDBKWH5ANO3UIX257LJPCWRITNOA7GY4A, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.640856565 (550h40m12.511939611s) [noise=true , address=45.91.116.5:4322 , piece_id=SS6DOCSJHY5Y3DPWMYX45F4VYYXR44JDLYQ2YPY5XIIAYKMAG7LQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.645490329 (550h40m12.516573364s) [noise=true , address=77.90.31.18:4819 , piece_id=OYXPAN3EK4QRDEC6VHIDL3GSGEY27G72KII6HTGUMSPYLHCYBNEA, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:13.103710286 (550h40m12.974793331s) [noise=true , address=145.79.227.34:4911 , piece_id=24WE4EMZSL2JHQZ7PESCMLYCZOWGDTGJD2XNLUUJEFPSYLRI5E7A, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:13.209917141 (550h40m13.081000176s) [noise=true , address=145.79.227.44:4221 , piece_id=H72XCWLKYJAXYGGDIWN67H42RUIEOJCPUSWT5IJMFPKETH7LK3KA, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:13.218355468 (550h40m13.089438514s) [noise=true , address=77.90.31.11:4565 , piece_id=4SBLRZK3K4ED73PJ64BI6ANKBWB2AAHQA4SBUNH6CJ63GN3FJR2Q, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:13.255968741 (550h40m13.127051806s) [noise=true , address=145.79.227.30:4789 , piece_id=4QT5IUPIK7BOHJROEK2EBFTW434OKC75FZNENMYYCRMBZ7E5BUTA, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:13.255996102 (550h40m13.127079137s) [seg_pos={0 0}, upload=482805]: Segment reached optimal threshold of 14 pieces.
Dec 28 14:09:13.256214589 (550h40m13.127297634s) [noise=true , address=145.79.227.40:4053 , piece_id=RO4ZZQCKCC7T4T6SOT7GGTY626BJ6ESLQG6KATTPNIUM3ICUKHAQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DqyTc4MHvxutFXs2WWRCk8MSuxF3Y1HayyQY4Y5sy8cqDLWuG): protocol: expected piece hash; context canceled; EOF
Dec 28 14:09:13.256223686 (550h40m13.127306731s) [noise=true , address=145.79.227.40:4053 , piece_id=RO4ZZQCKCC7T4T6SOT7GGTY626BJ6ESLQG6KATTPNIUM3ICUKHAQ, seg_pos={0 0}, upload=482805]: Long tail cancellation due to optimal threshold
Dec 28 14:09:13.256512506 (550h40m13.127595551s) [noise=true , address=77.90.31.15:4704 , piece_id=DXWLFOIKOPE33LMEQECUJVIGRG3LPADUMULLWI4ZBHEOQHD43ULQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14tJRdZ1guwwsYwPf3LtENRWqYx2jBQpi7uAMMPaB32Gks9zmj): protocol: expected piece hash; context canceled; EOF
Dec 28 14:09:13.256520060 (550h40m13.127603105s) [noise=true , address=77.90.31.15:4704 , piece_id=DXWLFOIKOPE33LMEQECUJVIGRG3LPADUMULLWI4ZBHEOQHD43ULQ, seg_pos={0 0}, upload=482805]: Long tail cancellation due to optimal threshold
Dec 28 14:09:13.256649106 (550h40m13.127732151s) [noise=true , address=77.90.31.14:4678 , piece_id=VBGPSGMNDQZYY7UKRINAV6B3FVI5BPDPYS37FQIJCMLL65F3XOEQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12beu3SUZMPNWfbeHV4tLtx9UVZoEHh4Kt7oBhT59w3bGBE1ovP): protocol: write tcp 188.166.225.170:57374->77.90.31.14:4678: use of closed network connection; write tcp 188.166.225.170:57374->77.90.31.14:4678: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57374->77.90.31.14:4678: use of closed network connection
Dec 28 14:09:13.256655819 (550h40m13.127738864s) [noise=true , address=77.90.31.14:4678 , piece_id=VBGPSGMNDQZYY7UKRINAV6B3FVI5BPDPYS37FQIJCMLL65F3XOEQ, seg_pos={0 0}, upload=482805]: Long tail cancellation due to optimal threshold
Dec 28 14:09:13.256752974 (550h40m13.127836009s) [noise=true , address=64.49.13.169:4710 , piece_id=SANL7TVJFES63XD46OKQL4KIJ6YVHSEWPZKMRISEKC3ZKWQBOMIA, seg_pos={0 0}, upload=482805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:35680->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:35680->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35680->64.49.13.169:4710: use of closed network connection
Dec 28 14:09:13.256758825 (550h40m13.12784186s) [noise=true , address=64.49.13.169:4710 , piece_id=SANL7TVJFES63XD46OKQL4KIJ6YVHSEWPZKMRISEKC3ZKWQBOMIA, seg_pos={0 0}, upload=482805]: Long tail cancellation due to optimal threshold
Dec 28 14:09:13.256776549 (550h40m13.127859594s) [seg_pos={0 0}, upload=482805]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 14:09:13.256856591 (550h40m13.127939626s) [upload=482805]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 14:09:13.256862432 (550h40m13.127945467s) [upload=482805]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 14:09:13.550363717 (550h40m13.421446782s) [upload=482805]: Done uploading segments...
Dec 28 14:09:13.550405126 (550h40m13.421488161s) [upload=482805]: Upload finished. err: <nil>
Duration: 40.505558552s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [....X............X....................X...X........X........] duration:
18.937812188s trace:
https://jaeger.dev.storj.io/trace/360e059013858fb1
FAIL (2025-12-28 14:11:56 UTC)
orbiter warning: context deadline exceeded
Duration: 18.937835332s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
19 out of 60 is failed (threshold=3) [...X..X.....X..X.X...XX......X......XXX.XX....XX......XX.XX.] duration:
5.748856094s trace:
https://jaeger.dev.storj.io/trace/7d5d0490f5dff817
FAIL (2025-12-28 14:11:51 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: M7YORSQQWQCXZKXVIDBXH2ZDVJ5V2UXQI2TVNJJQFLH2D3K3QV7A) context deadline exceeded; piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: 7MPOEIJEREEFT6BYEYOM7JG76OFL4QXGEYTAC6U5VZRMVMLZMROA) context deadline exceeded
======LOGS======
c 28 14:11:16.426053726 (550h42m16.297136761s) [upload=482833]: Starting upload
Dec 28 14:11:16.426128798 (550h42m16.297211834s) [upload=482833]: Done starting upload
Dec 28 14:11:16.426386770 (550h42m16.297469815s) [upload=482833]: Uploading segments...
Dec 28 14:11:16.426595868 (550h42m16.297678924s) [upload=482833]: Got next segment. Inline: false
Dec 28 14:11:16.426732469 (550h42m16.297815504s) [upload=482833]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 14:11:16.426773046 (550h42m16.297856081s) [upload=482833]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 14:11:16.676125223 (550h42m16.547208278s) [seg_pos={0 0}, upload=482833]: Begin upload segment...
Dec 28 14:11:16.676184687 (550h42m16.547267732s) [seg_pos={0 0}, upload=482833]: Done begin upload segment.
Dec 28 14:11:16.676217850 (550h42m16.547300895s) [noise=true , address=77.90.31.12:4589 , piece_id=JAOCLAFV4NNBJ5CIY5ZBNSX7VZM4P6WR3MJOSBXIUJ3O5ZJP2MHQ, seg_pos={0 0}, upload=482833]: Uploading piece...
Dec 28 14:11:16.676358137 (550h42m16.547441182s) [noise=true , address=45.91.116.12:4226 , piece_id=7MPOEIJEREEFT6BYEYOM7JG76OFL4QXGEYTAC6U5VZRMVMLZMROA, seg_pos={0 0}, upload=482833]: Uploading piece...
Dec 28 14:11:16.676913254 (550h42m16.547996299s) [noise=true , address=145.79.227.8:4330 , piece_id=MXWBEAJ7RXLLEGBN7K5IDP7KKW6SU35YURW6YA4UIYZNY5JSFZFQ, seg_pos={0 0}, upload=482833]: Uploading piece...
Dec 28 14:11:16.677078719 (550h42m16.548161765s) [noise=true , address=145.79.227.37:4565 , piece_id=RZE4XC4SXBKPBC4PDIJSJ7D6AZRODFR4TZDNNGN6ASDYVWHZ4EKQ, seg_pos={0 0}, upload=482833]: Uploading piece...
Dec 28 14:11:16.677540689 (550h42m16.548623734s) [noise=true , address=145.79.227.50:4374 , piece_id=2IJ7GDJBORTRLCXVF4EOYQITUAX4LARMSQBH5DDRB5LCGTBRYCVA, seg_pos={0 0}, upload=482833]: Uploading piece...
Dec 28 14:11:16.677878372 (550h42m16.548961417s) [noise=true , address=145.79.227.52:4446 , piece_id=ZLSC5T5PI7WOANXJXOROTZ4WSULWYQYGFDLAF4ERDKOZ6GUPMOMA, seg_pos={0 0}, upload=482833]: Uploading piece...
Dec 28 14:11:16.678228649 (550h42m16.549311704s) [noise=true , address=145.79.227.31:4798 , piece_id=BZSWK4RZJA2TEPJETROISZAKBJG3QSZDLQRNBONFFLTNZKDRYL3Q, seg_pos={0 0}, upload=482833]: Uploading piece...
Dec 28 14:11:16.678240501 (550h42m16.549323546s) [noise=true , address=45.91.116.8:4045 , piece_id=BIPE5UI7URFXHQ2C3EDRH5TLQDTN6CGE3UZJIEWX2OSEKD4Q2GLQ, seg_pos={0 0}, upload=482833]: Uploading piece...
Dec 28 14:11:16.678578706 (550h42m16.549661751s) [noise=true , address=145.79.227.39:4014 , piece_id=CJ2NROH7AON5OM5BDVAALKAQ6NQL3EBENXFUPTTDUQMBS7DPF6VA, seg_pos={0 0}, upload=482833]: Uploading piece...
Dec 28 14:11:16.678737918 (550h42m16.549820963s) [noise=true , address=145.79.227.47:4266 , piece_id=QGYCOMWTH6N372DX3HP67QPKF5NDVKXQGJBU3RMSKSJV24RZHV2A, seg_pos={0 0}, upload=482833]: Uploading piece...
Dec 28 14:11:16.678967957 (550h42m16.550051002s) [noise=true , address=45.91.116.5:4299 , piece_id=OYQ23GQ2UXFNVYW3IUOMMCQT5DGX6LE2LLIXS54ZUOOKV3W5IZ2Q, seg_pos={0 0}, upload=482833]: Uploading piece...
Dec 28 14:11:16.679216840 (550h42m16.550299875s) [noise=true , address=45.91.116.6:4975 , piece_id=M7YORSQQWQCXZKXVIDBXH2ZDVJ5V2UXQI2TVNJJQFLH2D3K3QV7A, seg_pos={0 0}, upload=482833]: Uploading piece...
Dec 28 14:11:16.679625689 (550h42m16.550708734s) [noise=true , address=145.79.227.12:4765 , piece_id=XIYLXZ6PYMM6V5HWW3ZAS2QIAAW6GRVYBWGODEGSNYVV3YWDOWAA, seg_pos={0 0}, upload=482833]: Uploading piece...
Dec 28 14:11:16.680093500 (550h42m16.551176545s) [noise=true , address=145.79.227.27:4691 , piece_id=QEPPAXCVK6YPTBCTEZEH4MEQ7CAZ2YFV2QHKUB3XZEOV32PWFUDA, seg_pos={0 0}, upload=482833]: Uploading piece...
Dec 28 14:11:16.680505695 (550h42m16.55158873s) [noise=true , address=145.79.227.38:4978 , piece_id=5JBU5OHM2PZ4V4XQGQ44PZN7346VSXRD5QTDWJWUE22QJ4GLULOA, seg_pos={0 0}, upload=482833]: Uploading piece...
Dec 28 14:11:16.681403655 (550h42m16.5524867s) [noise=true , address=45.91.116.14:4191 , piece_id=7TSAWJ62LGMWIQZ733RLGFFOAGBUBLV2G3UTXI7UGNMU6DF4HOHA, seg_pos={0 0}, upload=482833]: Uploading piece...
Dec 28 14:11:18.294800239 (550h42m18.165883284s) [upload=482833]: Next returned nil segment
Dec 28 14:11:18.294819686 (550h42m18.165902731s) [upload=482833]: Waiting for error group managing segments...
Dec 28 14:11:18.680627949 (550h42m18.551710994s) [noise=true , address=145.79.227.52:4446 , piece_id=ZLSC5T5PI7WOANXJXOROTZ4WSULWYQYGFDLAF4ERDKOZ6GUPMOMA, seg_pos={0 0}, upload=482833]: Done uploading piece. err: <nil>
Dec 28 14:11:18.681158509 (550h42m18.552241544s) [noise=true , address=145.79.227.8:4330 , piece_id=MXWBEAJ7RXLLEGBN7K5IDP7KKW6SU35YURW6YA4UIYZNY5JSFZFQ, seg_pos={0 0}, upload=482833]: Done uploading piece. err: <nil>
Dec 28 14:11:18.691046006 (550h42m18.562129051s) [noise=true , address=145.79.227.27:4691 , piece_id=QEPPAXCVK6YPTBCTEZEH4MEQ7CAZ2YFV2QHKUB3XZEOV32PWFUDA, seg_pos={0 0}, upload=482833]: Done uploading piece. err: <nil>
Dec 28 14:11:18.693001951 (550h42m18.564084986s) [noise=true , address=145.79.227.31:4798 , piece_id=BZSWK4RZJA2TEPJETROISZAKBJG3QSZDLQRNBONFFLTNZKDRYL3Q, seg_pos={0 0}, upload=482833]: Done uploading piece. err: <nil>
Dec 28 14:11:18.700095568 (550h42m18.571178603s) [noise=true , address=145.79.227.47:4266 , piece_id=QGYCOMWTH6N372DX3HP67QPKF5NDVKXQGJBU3RMSKSJV24RZHV2A, seg_pos={0 0}, upload=482833]: Done uploading piece. err: <nil>
Dec 28 14:11:18.702245151 (550h42m18.573328187s) [noise=true , address=145.79.227.39:4014 , piece_id=CJ2NROH7AON5OM5BDVAALKAQ6NQL3EBENXFUPTTDUQMBS7DPF6VA, seg_pos={0 0}, upload=482833]: Done uploading piece. err: <nil>
Dec 28 14:11:18.702904978 (550h42m18.573988023s) [noise=true , address=145.79.227.12:4765 , piece_id=XIYLXZ6PYMM6V5HWW3ZAS2QIAAW6GRVYBWGODEGSNYVV3YWDOWAA, seg_pos={0 0}, upload=482833]: Done uploading piece. err: <nil>
Dec 28 14:11:18.706012505 (550h42m18.57709555s) [noise=true , address=145.79.227.37:4565 , piece_id=RZE4XC4SXBKPBC4PDIJSJ7D6AZRODFR4TZDNNGN6ASDYVWHZ4EKQ, seg_pos={0 0}, upload=482833]: Done uploading piece. err: <nil>
Dec 28 14:11:18.706846794 (550h42m18.577929839s) [noise=true , address=145.79.227.38:4978 , piece_id=5JBU5OHM2PZ4V4XQGQ44PZN7346VSXRD5QTDWJWUE22QJ4GLULOA, seg_pos={0 0}, upload=482833]: Done uploading piece. err: <nil>
Dec 28 14:11:18.717068217 (550h42m18.588151252s) [noise=true , address=145.79.227.50:4374 , piece_id=2IJ7GDJBORTRLCXVF4EOYQITUAX4LARMSQBH5DDRB5LCGTBRYCVA, seg_pos={0 0}, upload=482833]: Done uploading piece. err: <nil>
Dec 28 14:11:18.905351469 (550h42m18.776434504s) [noise=true , address=45.91.116.6:4975 , piece_id=M7YORSQQWQCXZKXVIDBXH2ZDVJ5V2UXQI2TVNJJQFLH2D3K3QV7A, seg_pos={0 0}, upload=482833]: Done uploading piece. err: <nil>
Dec 28 14:11:18.954007870 (550h42m18.825090905s) [noise=true , address=45.91.116.12:4226 , piece_id=7MPOEIJEREEFT6BYEYOM7JG76OFL4QXGEYTAC6U5VZRMVMLZMROA, seg_pos={0 0}, upload=482833]: Done uploading piece. err: <nil>
Dec 28 14:11:18.957617743 (550h42m18.828700778s) [noise=true , address=45.91.116.14:4191 , piece_id=7TSAWJ62LGMWIQZ733RLGFFOAGBUBLV2G3UTXI7UGNMU6DF4HOHA, seg_pos={0 0}, upload=482833]: Done uploading piece. err: <nil>
Dec 28 14:11:18.957635297 (550h42m18.828718332s) [seg_pos={0 0}, upload=482833]: Segment reached optimal threshold of 13 pieces.
Dec 28 14:11:18.957837652 (550h42m18.828920697s) [noise=true , address=45.91.116.5:4299 , piece_id=OYQ23GQ2UXFNVYW3IUOMMCQT5DGX6LE2LLIXS54ZUOOKV3W5IZ2Q, seg_pos={0 0}, upload=482833]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF
Dec 28 14:11:18.957843984 (550h42m18.828927019s) [noise=true , address=45.91.116.5:4299 , piece_id=OYQ23GQ2UXFNVYW3IUOMMCQT5DGX6LE2LLIXS54ZUOOKV3W5IZ2Q, seg_pos={0 0}, upload=482833]: Long tail cancellation due to optimal threshold
Dec 28 14:11:18.957966427 (550h42m18.829049462s) [noise=true , address=45.91.116.8:4045 , piece_id=BIPE5UI7URFXHQ2C3EDRH5TLQDTN6CGE3UZJIEWX2OSEKD4Q2GLQ, seg_pos={0 0}, upload=482833]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: write tcp 188.166.225.170:55214->45.91.116.8:4045: use of closed network connection; write tcp 188.166.225.170:55214->45.91.116.8:4045: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55214->45.91.116.8:4045: use of closed network connection
Dec 28 14:11:18.957971316 (550h42m18.829054351s) [noise=true , address=45.91.116.8:4045 , piece_id=BIPE5UI7URFXHQ2C3EDRH5TLQDTN6CGE3UZJIEWX2OSEKD4Q2GLQ, seg_pos={0 0}, upload=482833]: Long tail cancellation due to optimal threshold
Dec 28 14:11:18.958205312 (550h42m18.829288357s) [noise=true , address=77.90.31.12:4589 , piece_id=JAOCLAFV4NNBJ5CIY5ZBNSX7VZM4P6WR3MJOSBXIUJ3O5ZJP2MHQ, seg_pos={0 0}, upload=482833]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF
Dec 28 14:11:18.958214289 (550h42m18.829297324s) [noise=true , address=77.90.31.12:4589 , piece_id=JAOCLAFV4NNBJ5CIY5ZBNSX7VZM4P6WR3MJOSBXIUJ3O5ZJP2MHQ, seg_pos={0 0}, upload=482833]: Long tail cancellation due to optimal threshold
Dec 28 14:11:18.958240759 (550h42m18.829323794s) [seg_pos={0 0}, upload=482833]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 14:11:18.958319619 (550h42m18.829402654s) [upload=482833]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 14:11:18.958325211 (550h42m18.829408256s) [upload=482833]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 14:11:19.259598824 (550h42m19.130681859s) [upload=482833]: Done uploading segments...
Dec 28 14:11:19.259676423 (550h42m19.130759458s) [upload=482833]: Upload finished. err: <nil>
Duration: 5.748878797s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
19 out of 60 is failed (threshold=3) [....X..X.....X..X.X...XX......X......XXX.XX....XX......XX.XX] duration:
30.002813718s trace:
https://jaeger.dev.storj.io/trace/7d5d0490f5dff817
FAIL (2025-12-28 14:11:45 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: M7YORSQQWQCXZKXVIDBXH2ZDVJ5V2UXQI2TVNJJQFLH2D3K3QV7A) context deadline exceeded; piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: 7MPOEIJEREEFT6BYEYOM7JG76OFL4QXGEYTAC6U5VZRMVMLZMROA) context deadline exceeded
======LOGS======
c 28 14:11:16.426053726 (550h42m16.297136761s) [upload=482833]: Starting upload
Dec 28 14:11:16.426128798 (550h42m16.297211834s) [upload=482833]: Done starting upload
Dec 28 14:11:16.426386770 (550h42m16.297469815s) [upload=482833]: Uploading segments...
Dec 28 14:11:16.426595868 (550h42m16.297678924s) [upload=482833]: Got next segment. Inline: false
Dec 28 14:11:16.426732469 (550h42m16.297815504s) [upload=482833]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 14:11:16.426773046 (550h42m16.297856081s) [upload=482833]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 14:11:16.676125223 (550h42m16.547208278s) [seg_pos={0 0}, upload=482833]: Begin upload segment...
Dec 28 14:11:16.676184687 (550h42m16.547267732s) [seg_pos={0 0}, upload=482833]: Done begin upload segment.
Dec 28 14:11:16.676217850 (550h42m16.547300895s) [noise=true , address=77.90.31.12:4589 , piece_id=JAOCLAFV4NNBJ5CIY5ZBNSX7VZM4P6WR3MJOSBXIUJ3O5ZJP2MHQ, seg_pos={0 0}, upload=482833]: Uploading piece...
Dec 28 14:11:16.676358137 (550h42m16.547441182s) [noise=true , address=45.91.116.12:4226 , piece_id=7MPOEIJEREEFT6BYEYOM7JG76OFL4QXGEYTAC6U5VZRMVMLZMROA, seg_pos={0 0}, upload=482833]: Uploading piece...
Dec 28 14:11:16.676913254 (550h42m16.547996299s) [noise=true , address=145.79.227.8:4330 , piece_id=MXWBEAJ7RXLLEGBN7K5IDP7KKW6SU35YURW6YA4UIYZNY5JSFZFQ, seg_pos={0 0}, upload=482833]: Uploading piece...
Dec 28 14:11:16.677078719 (550h42m16.548161765s) [noise=true , address=145.79.227.37:4565 , piece_id=RZE4XC4SXBKPBC4PDIJSJ7D6AZRODFR4TZDNNGN6ASDYVWHZ4EKQ, seg_pos={0 0}, upload=482833]: Uploading piece...
Dec 28 14:11:16.677540689 (550h42m16.548623734s) [noise=true , address=145.79.227.50:4374 , piece_id=2IJ7GDJBORTRLCXVF4EOYQITUAX4LARMSQBH5DDRB5LCGTBRYCVA, seg_pos={0 0}, upload=482833]: Uploading piece...
Dec 28 14:11:16.677878372 (550h42m16.548961417s) [noise=true , address=145.79.227.52:4446 , piece_id=ZLSC5T5PI7WOANXJXOROTZ4WSULWYQYGFDLAF4ERDKOZ6GUPMOMA, seg_pos={0 0}, upload=482833]: Uploading piece...
Dec 28 14:11:16.678228649 (550h42m16.549311704s) [noise=true , address=145.79.227.31:4798 , piece_id=BZSWK4RZJA2TEPJETROISZAKBJG3QSZDLQRNBONFFLTNZKDRYL3Q, seg_pos={0 0}, upload=482833]: Uploading piece...
Dec 28 14:11:16.678240501 (550h42m16.549323546s) [noise=true , address=45.91.116.8:4045 , piece_id=BIPE5UI7URFXHQ2C3EDRH5TLQDTN6CGE3UZJIEWX2OSEKD4Q2GLQ, seg_pos={0 0}, upload=482833]: Uploading piece...
Dec 28 14:11:16.678578706 (550h42m16.549661751s) [noise=true , address=145.79.227.39:4014 , piece_id=CJ2NROH7AON5OM5BDVAALKAQ6NQL3EBENXFUPTTDUQMBS7DPF6VA, seg_pos={0 0}, upload=482833]: Uploading piece...
Dec 28 14:11:16.678737918 (550h42m16.549820963s) [noise=true , address=145.79.227.47:4266 , piece_id=QGYCOMWTH6N372DX3HP67QPKF5NDVKXQGJBU3RMSKSJV24RZHV2A, seg_pos={0 0}, upload=482833]: Uploading piece...
Dec 28 14:11:16.678967957 (550h42m16.550051002s) [noise=true , address=45.91.116.5:4299 , piece_id=OYQ23GQ2UXFNVYW3IUOMMCQT5DGX6LE2LLIXS54ZUOOKV3W5IZ2Q, seg_pos={0 0}, upload=482833]: Uploading piece...
Dec 28 14:11:16.679216840 (550h42m16.550299875s) [noise=true , address=45.91.116.6:4975 , piece_id=M7YORSQQWQCXZKXVIDBXH2ZDVJ5V2UXQI2TVNJJQFLH2D3K3QV7A, seg_pos={0 0}, upload=482833]: Uploading piece...
Dec 28 14:11:16.679625689 (550h42m16.550708734s) [noise=true , address=145.79.227.12:4765 , piece_id=XIYLXZ6PYMM6V5HWW3ZAS2QIAAW6GRVYBWGODEGSNYVV3YWDOWAA, seg_pos={0 0}, upload=482833]: Uploading piece...
Dec 28 14:11:16.680093500 (550h42m16.551176545s) [noise=true , address=145.79.227.27:4691 , piece_id=QEPPAXCVK6YPTBCTEZEH4MEQ7CAZ2YFV2QHKUB3XZEOV32PWFUDA, seg_pos={0 0}, upload=482833]: Uploading piece...
Dec 28 14:11:16.680505695 (550h42m16.55158873s) [noise=true , address=145.79.227.38:4978 , piece_id=5JBU5OHM2PZ4V4XQGQ44PZN7346VSXRD5QTDWJWUE22QJ4GLULOA, seg_pos={0 0}, upload=482833]: Uploading piece...
Dec 28 14:11:16.681403655 (550h42m16.5524867s) [noise=true , address=45.91.116.14:4191 , piece_id=7TSAWJ62LGMWIQZ733RLGFFOAGBUBLV2G3UTXI7UGNMU6DF4HOHA, seg_pos={0 0}, upload=482833]: Uploading piece...
Dec 28 14:11:18.294800239 (550h42m18.165883284s) [upload=482833]: Next returned nil segment
Dec 28 14:11:18.294819686 (550h42m18.165902731s) [upload=482833]: Waiting for error group managing segments...
Dec 28 14:11:18.680627949 (550h42m18.551710994s) [noise=true , address=145.79.227.52:4446 , piece_id=ZLSC5T5PI7WOANXJXOROTZ4WSULWYQYGFDLAF4ERDKOZ6GUPMOMA, seg_pos={0 0}, upload=482833]: Done uploading piece. err: <nil>
Dec 28 14:11:18.681158509 (550h42m18.552241544s) [noise=true , address=145.79.227.8:4330 , piece_id=MXWBEAJ7RXLLEGBN7K5IDP7KKW6SU35YURW6YA4UIYZNY5JSFZFQ, seg_pos={0 0}, upload=482833]: Done uploading piece. err: <nil>
Dec 28 14:11:18.691046006 (550h42m18.562129051s) [noise=true , address=145.79.227.27:4691 , piece_id=QEPPAXCVK6YPTBCTEZEH4MEQ7CAZ2YFV2QHKUB3XZEOV32PWFUDA, seg_pos={0 0}, upload=482833]: Done uploading piece. err: <nil>
Dec 28 14:11:18.693001951 (550h42m18.564084986s) [noise=true , address=145.79.227.31:4798 , piece_id=BZSWK4RZJA2TEPJETROISZAKBJG3QSZDLQRNBONFFLTNZKDRYL3Q, seg_pos={0 0}, upload=482833]: Done uploading piece. err: <nil>
Dec 28 14:11:18.700095568 (550h42m18.571178603s) [noise=true , address=145.79.227.47:4266 , piece_id=QGYCOMWTH6N372DX3HP67QPKF5NDVKXQGJBU3RMSKSJV24RZHV2A, seg_pos={0 0}, upload=482833]: Done uploading piece. err: <nil>
Dec 28 14:11:18.702245151 (550h42m18.573328187s) [noise=true , address=145.79.227.39:4014 , piece_id=CJ2NROH7AON5OM5BDVAALKAQ6NQL3EBENXFUPTTDUQMBS7DPF6VA, seg_pos={0 0}, upload=482833]: Done uploading piece. err: <nil>
Dec 28 14:11:18.702904978 (550h42m18.573988023s) [noise=true , address=145.79.227.12:4765 , piece_id=XIYLXZ6PYMM6V5HWW3ZAS2QIAAW6GRVYBWGODEGSNYVV3YWDOWAA, seg_pos={0 0}, upload=482833]: Done uploading piece. err: <nil>
Dec 28 14:11:18.706012505 (550h42m18.57709555s) [noise=true , address=145.79.227.37:4565 , piece_id=RZE4XC4SXBKPBC4PDIJSJ7D6AZRODFR4TZDNNGN6ASDYVWHZ4EKQ, seg_pos={0 0}, upload=482833]: Done uploading piece. err: <nil>
Dec 28 14:11:18.706846794 (550h42m18.577929839s) [noise=true , address=145.79.227.38:4978 , piece_id=5JBU5OHM2PZ4V4XQGQ44PZN7346VSXRD5QTDWJWUE22QJ4GLULOA, seg_pos={0 0}, upload=482833]: Done uploading piece. err: <nil>
Dec 28 14:11:18.717068217 (550h42m18.588151252s) [noise=true , address=145.79.227.50:4374 , piece_id=2IJ7GDJBORTRLCXVF4EOYQITUAX4LARMSQBH5DDRB5LCGTBRYCVA, seg_pos={0 0}, upload=482833]: Done uploading piece. err: <nil>
Dec 28 14:11:18.905351469 (550h42m18.776434504s) [noise=true , address=45.91.116.6:4975 , piece_id=M7YORSQQWQCXZKXVIDBXH2ZDVJ5V2UXQI2TVNJJQFLH2D3K3QV7A, seg_pos={0 0}, upload=482833]: Done uploading piece. err: <nil>
Dec 28 14:11:18.954007870 (550h42m18.825090905s) [noise=true , address=45.91.116.12:4226 , piece_id=7MPOEIJEREEFT6BYEYOM7JG76OFL4QXGEYTAC6U5VZRMVMLZMROA, seg_pos={0 0}, upload=482833]: Done uploading piece. err: <nil>
Dec 28 14:11:18.957617743 (550h42m18.828700778s) [noise=true , address=45.91.116.14:4191 , piece_id=7TSAWJ62LGMWIQZ733RLGFFOAGBUBLV2G3UTXI7UGNMU6DF4HOHA, seg_pos={0 0}, upload=482833]: Done uploading piece. err: <nil>
Dec 28 14:11:18.957635297 (550h42m18.828718332s) [seg_pos={0 0}, upload=482833]: Segment reached optimal threshold of 13 pieces.
Dec 28 14:11:18.957837652 (550h42m18.828920697s) [noise=true , address=45.91.116.5:4299 , piece_id=OYQ23GQ2UXFNVYW3IUOMMCQT5DGX6LE2LLIXS54ZUOOKV3W5IZ2Q, seg_pos={0 0}, upload=482833]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF
Dec 28 14:11:18.957843984 (550h42m18.828927019s) [noise=true , address=45.91.116.5:4299 , piece_id=OYQ23GQ2UXFNVYW3IUOMMCQT5DGX6LE2LLIXS54ZUOOKV3W5IZ2Q, seg_pos={0 0}, upload=482833]: Long tail cancellation due to optimal threshold
Dec 28 14:11:18.957966427 (550h42m18.829049462s) [noise=true , address=45.91.116.8:4045 , piece_id=BIPE5UI7URFXHQ2C3EDRH5TLQDTN6CGE3UZJIEWX2OSEKD4Q2GLQ, seg_pos={0 0}, upload=482833]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: write tcp 188.166.225.170:55214->45.91.116.8:4045: use of closed network connection; write tcp 188.166.225.170:55214->45.91.116.8:4045: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55214->45.91.116.8:4045: use of closed network connection
Dec 28 14:11:18.957971316 (550h42m18.829054351s) [noise=true , address=45.91.116.8:4045 , piece_id=BIPE5UI7URFXHQ2C3EDRH5TLQDTN6CGE3UZJIEWX2OSEKD4Q2GLQ, seg_pos={0 0}, upload=482833]: Long tail cancellation due to optimal threshold
Dec 28 14:11:18.958205312 (550h42m18.829288357s) [noise=true , address=77.90.31.12:4589 , piece_id=JAOCLAFV4NNBJ5CIY5ZBNSX7VZM4P6WR3MJOSBXIUJ3O5ZJP2MHQ, seg_pos={0 0}, upload=482833]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF
Dec 28 14:11:18.958214289 (550h42m18.829297324s) [noise=true , address=77.90.31.12:4589 , piece_id=JAOCLAFV4NNBJ5CIY5ZBNSX7VZM4P6WR3MJOSBXIUJ3O5ZJP2MHQ, seg_pos={0 0}, upload=482833]: Long tail cancellation due to optimal threshold
Dec 28 14:11:18.958240759 (550h42m18.829323794s) [seg_pos={0 0}, upload=482833]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 14:11:18.958319619 (550h42m18.829402654s) [upload=482833]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 14:11:18.958325211 (550h42m18.829408256s) [upload=482833]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 14:11:19.259598824 (550h42m19.130681859s) [upload=482833]: Done uploading segments...
Dec 28 14:11:19.259676423 (550h42m19.130759458s) [upload=482833]: Upload finished. err: <nil>
Duration: 30.002839648s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
53 out of 60 is failed (threshold=6) [XXX.XXXXXXX.XXXXXXXXXXXX.XXXXXXX.XX..XXXXXXXXXXXXXXXXXXXXX.X] duration:
5.001028466s trace:
https://jaeger.dev.storj.io/trace/92a90438d044faa
FAIL (2025-12-28 14:11:27 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001055628s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [...............X............................................] duration:
6.958141113s trace:
https://jaeger.dev.storj.io/trace/5fe9d6dbb79fbe44
FAIL (2025-12-28 14:11:23 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
67.204.205:4027 , piece_id=PAPNMRJ7E47VOMOXPBBDWKZJHLRMXTWVR5VW3J7KNQSSMHOHWOAQ, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.967737953 (550h8m4.838820988s) [noise=true , address=185.167.204.206:4072 , piece_id=WXORBDMA3KW7WZ67A4FJT6URJRFADUJ7X2PAHIGNG47T3TGMXVTQ, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.968193240 (550h8m4.839276285s) [noise=true , address=185.167.204.205:4030 , piece_id=R5DLI4ZTLOVGO56GPLJL5ZNHYSXOSYNJQHFIO4MVVNHWWBMEOV2Q, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.968780638 (550h8m4.839863683s) [noise=true , address=89.149.201.147:4019 , piece_id=MK7ORCLHO3DOWIOVSOWIQMPJ447PHRQGXP3UKITIRTYJSU5MPCCQ, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.969308924 (550h8m4.840391959s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.970088349 (550h8m4.841171404s) [noise=true , address=81.171.14.70:4039 , piece_id=EDHPQWLQZTP7MKPGEVMP7JXNTW7LFWSXCTRTSXPCBIQPROX76WRA, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.970325381 (550h8m4.841408426s) [noise=true , address=81.171.14.70:4036 , piece_id=CICNNV7JE6TBNC4RAPXKCEUSJAADOUVGSVIVQISM3ZB4COAZQDDA, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.970783813 (550h8m4.841866858s) [noise=true , address=81.171.14.70:4044 , piece_id=DGN74IG6IWAHLEMO6NLL4TJBXZ6ONTN6QUB6HXY6K7SSO3LYOYIQ, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.971205406 (550h8m4.842288441s) [noise=true , address=81.171.14.70:4048 , piece_id=TLT47ORGUKGF3MIIEAX7D22VR2FO6W6NL3VFJNHKWD4L3LIMFDLA, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.971606299 (550h8m4.842689344s) [noise=true , address=185.167.204.206:4067 , piece_id=VSKTG5RH32N7OEVWJGQVB6WJ72TEVF6DDK25W7NOH4RAXUP52LTQ, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.972162538 (550h8m4.843245583s) [noise=true , address=89.149.201.147:4002 , piece_id=G2PABNTDWCK4MKOZF6PONBMRNIXI45JTHJMAZFPEKPEJ3PQ7MTBA, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:05.795167814 (550h8m5.666250849s) [upload=482337]: Next returned nil segment
Dec 28 13:37:05.795184566 (550h8m5.666267611s) [upload=482337]: Waiting for error group managing segments...
Dec 28 13:37:06.178573801 (550h8m6.049656836s) [noise=true , address=185.167.204.206:4049 , piece_id=6RG2DZV6FPTIRE63ZNNST42TSWEBQ456YXIPPVYHIXV55GWFVJ4Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.183060835 (550h8m6.05414391s) [noise=true , address=185.167.204.206:4072 , piece_id=WXORBDMA3KW7WZ67A4FJT6URJRFADUJ7X2PAHIGNG47T3TGMXVTQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.183411202 (550h8m6.054494237s) [noise=true , address=185.167.204.206:4064 , piece_id=OG3G74OU23CEMKA54WQE3DOJ6AKAXBLZ23QWOQC4UEGQQN4JSMXQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.183791146 (550h8m6.054874181s) [noise=true , address=185.167.204.205:4029 , piece_id=6JQB6ZSCQZKWH3JWCQ4DY6BBAX73KC4Y53PRSMEMLWURZ5QWWQQQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.183973743 (550h8m6.055056778s) [noise=true , address=185.167.204.206:4053 , piece_id=YDH77S6JLKJ6IYS6DRUCELP5EJHIC2O44QBO7MIOEUYBML4GIBHA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.184465369 (550h8m6.055548414s) [noise=true , address=185.167.204.205:4008 , piece_id=RTUZPEXBB6JNSMC5YOFQQGDM57YD233W4756ZBQVMFNOVX6YKB4A, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.184501989 (550h8m6.055585024s) [noise=true , address=185.167.204.205:4030 , piece_id=R5DLI4ZTLOVGO56GPLJL5ZNHYSXOSYNJQHFIO4MVVNHWWBMEOV2Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.184856224 (550h8m6.055939259s) [noise=true , address=185.167.204.205:4009 , piece_id=V6RA7Y7FKY4DCPTRRF3BYNBALWDY5VTPXMDUXFE3353KUXLSQJEA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.185063518 (550h8m6.056146563s) [noise=true , address=185.167.204.205:4016 , piece_id=JB7SC3J2VR4U33HNQS2A2CA75THT3XLQCOIFHAJJEK7MQGY2SYAQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.185288617 (550h8m6.056371652s) [noise=true , address=185.167.204.205:4027 , piece_id=PAPNMRJ7E47VOMOXPBBDWKZJHLRMXTWVR5VW3J7KNQSSMHOHWOAQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.186241401 (550h8m6.057324436s) [noise=true , address=185.167.204.205:4024 , piece_id=FM5IMWRSJJ4RY72IX2OOYC54BFKL4FWE3UOAN3GJ26MLH345COVA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.193113658 (550h8m6.064196703s) [noise=true , address=185.167.204.205:4005 , piece_id=R4DFQJZBPJ5WJWHPODV6ILHUDIRBUSAFOMVNCLXB4UNMI6EXTKHA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.193791408 (550h8m6.064874453s) [noise=true , address=185.167.204.206:4068 , piece_id=DAI37M67FWCSVQTQOZ6RSSOJDGUVZO6SNGAHJJ44G3PGS4OJZEMA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.194292262 (550h8m6.065375307s) [noise=true , address=185.167.204.206:4046 , piece_id=VWLVHMBMJ63DQEWYBIKH3Y3NTVWFNW5OXZKMYXXBHVI3VWYXLLPQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.194302652 (550h8m6.065385687s) [noise=true , address=185.167.204.206:4067 , piece_id=VSKTG5RH32N7OEVWJGQVB6WJ72TEVF6DDK25W7NOH4RAXUP52LTQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.195359193 (550h8m6.066442228s) [noise=true , address=185.167.204.205:4012 , piece_id=LV6RUA5BWXG4DMSUVZDBGUAF5IBWUGMLJYUKNIXHRVDEBIOW4Q2Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.198423448 (550h8m6.069506493s) [noise=true , address=185.167.204.206:4070 , piece_id=5A7YYQYREIZRZDR7TRSE7XFW765JVAV7FX6LKWUI2OXAA74XN4CA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.888968408 (550h8m6.760051453s) [noise=true , address=81.171.14.70:4044 , piece_id=DGN74IG6IWAHLEMO6NLL4TJBXZ6ONTN6QUB6HXY6K7SSO3LYOYIQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.892350789 (550h8m6.763433834s) [noise=true , address=81.171.14.70:4039 , piece_id=EDHPQWLQZTP7MKPGEVMP7JXNTW7LFWSXCTRTSXPCBIQPROX76WRA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.904477160 (550h8m6.775560195s) [noise=true , address=81.171.14.70:4040 , piece_id=DYF4GSZBAKQ3H63LTVCMEFWMABY3FXHOJCNVIFYRKLF2J6B52K5A, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.128088281 (550h8m6.999171326s) [noise=true , address=81.171.14.70:4036 , piece_id=CICNNV7JE6TBNC4RAPXKCEUSJAADOUVGSVIVQISM3ZB4COAZQDDA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.430242142 (550h8m7.301325177s) [noise=true , address=89.149.201.147:4009 , piece_id=EVW2NQFYU57PC2IIVXLYIEBMYXJNIC2CLCHW6PS6NYIPCFRUVHDA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.505304332 (550h8m7.376387367s) [noise=true , address=89.149.201.147:4019 , piece_id=MK7ORCLHO3DOWIOVSOWIQMPJ447PHRQGXP3UKITIRTYJSU5MPCCQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.575461543 (550h8m7.446544609s) [noise=true , address=81.171.14.70:4048 , piece_id=TLT47ORGUKGF3MIIEAX7D22VR2FO6W6NL3VFJNHKWD4L3LIMFDLA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.599667265 (550h8m7.4707503s) [noise=true , address=89.149.201.147:4023 , piece_id=5IINX2GVRFLRX2QH24OO2DDANOKOTXTDR6AGVJBGSJ7IAZQ3CASA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.928390728 (550h8m7.799473773s) [noise=true , address=89.149.201.147:4002 , piece_id=G2PABNTDWCK4MKOZF6PONBMRNIXI45JTHJMAZFPEKPEJ3PQ7MTBA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.928417148 (550h8m7.799500193s) [seg_pos={0 0}, upload=482337]: Segment reached optimal threshold of 26 pieces.
Dec 28 13:37:07.928656303 (550h8m7.799739348s) [noise=true , address=81.171.14.70:4030 , piece_id=74YDIKUST7X3JBXVQHVLFHIKH5A4DYH5TNL3I4XUA5A6AHQO2DRA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZoRwa8wLw66Pjumy1ybp6wrZ79RrSU55uYZpwaY3xi95iSYde): protocol: expected piece hash; context canceled; EOF
Dec 28 13:37:07.928664990 (550h8m7.799748035s) [noise=true , address=81.171.14.70:4030 , piece_id=74YDIKUST7X3JBXVQHVLFHIKH5A4DYH5TNL3I4XUA5A6AHQO2DRA, seg_pos={0 0}, upload=482337]: Long tail cancellation due to optimal threshold
Dec 28 13:37:07.928723351 (550h8m7.799806386s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wgtphHKaTejC1qrJ98sDysBwtj2Zeqw2c5UFk1HMqfY8Buvr): protocol: expected piece hash; context canceled; EOF
Dec 28 13:37:07.928733922 (550h8m7.799816957s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Long tail cancellation due to optimal threshold
Dec 28 13:37:07.928754921 (550h8m7.799837966s) [seg_pos={0 0}, upload=482337]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 28 13:37:07.928870170 (550h8m7.799953215s) [upload=482337]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 13:37:07.928878627 (550h8m7.799961672s) [upload=482337]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 13:37:08.166321536 (550h8m8.037404571s) [upload=482337]: Done uploading segments...
Dec 28 13:37:08.166341444 (550h8m8.037424479s) [upload=482337]: Upload finished. err: <nil>
Duration: 6.958165188s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [......................................................X...X.] duration:
25.172353661s trace:
https://jaeger.dev.storj.io/trace/102cf391e061ae77
FAIL (2025-12-28 14:11:06 UTC)
orbiter warning: context deadline exceeded
Duration: 25.172365553s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [................X...........................................] duration:
17.83426808s trace:
https://jaeger.dev.storj.io/trace/5fe9d6dbb79fbe44
FAIL (2025-12-28 14:11:05 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
67.204.205:4027 , piece_id=PAPNMRJ7E47VOMOXPBBDWKZJHLRMXTWVR5VW3J7KNQSSMHOHWOAQ, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.967737953 (550h8m4.838820988s) [noise=true , address=185.167.204.206:4072 , piece_id=WXORBDMA3KW7WZ67A4FJT6URJRFADUJ7X2PAHIGNG47T3TGMXVTQ, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.968193240 (550h8m4.839276285s) [noise=true , address=185.167.204.205:4030 , piece_id=R5DLI4ZTLOVGO56GPLJL5ZNHYSXOSYNJQHFIO4MVVNHWWBMEOV2Q, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.968780638 (550h8m4.839863683s) [noise=true , address=89.149.201.147:4019 , piece_id=MK7ORCLHO3DOWIOVSOWIQMPJ447PHRQGXP3UKITIRTYJSU5MPCCQ, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.969308924 (550h8m4.840391959s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.970088349 (550h8m4.841171404s) [noise=true , address=81.171.14.70:4039 , piece_id=EDHPQWLQZTP7MKPGEVMP7JXNTW7LFWSXCTRTSXPCBIQPROX76WRA, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.970325381 (550h8m4.841408426s) [noise=true , address=81.171.14.70:4036 , piece_id=CICNNV7JE6TBNC4RAPXKCEUSJAADOUVGSVIVQISM3ZB4COAZQDDA, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.970783813 (550h8m4.841866858s) [noise=true , address=81.171.14.70:4044 , piece_id=DGN74IG6IWAHLEMO6NLL4TJBXZ6ONTN6QUB6HXY6K7SSO3LYOYIQ, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.971205406 (550h8m4.842288441s) [noise=true , address=81.171.14.70:4048 , piece_id=TLT47ORGUKGF3MIIEAX7D22VR2FO6W6NL3VFJNHKWD4L3LIMFDLA, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.971606299 (550h8m4.842689344s) [noise=true , address=185.167.204.206:4067 , piece_id=VSKTG5RH32N7OEVWJGQVB6WJ72TEVF6DDK25W7NOH4RAXUP52LTQ, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.972162538 (550h8m4.843245583s) [noise=true , address=89.149.201.147:4002 , piece_id=G2PABNTDWCK4MKOZF6PONBMRNIXI45JTHJMAZFPEKPEJ3PQ7MTBA, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:05.795167814 (550h8m5.666250849s) [upload=482337]: Next returned nil segment
Dec 28 13:37:05.795184566 (550h8m5.666267611s) [upload=482337]: Waiting for error group managing segments...
Dec 28 13:37:06.178573801 (550h8m6.049656836s) [noise=true , address=185.167.204.206:4049 , piece_id=6RG2DZV6FPTIRE63ZNNST42TSWEBQ456YXIPPVYHIXV55GWFVJ4Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.183060835 (550h8m6.05414391s) [noise=true , address=185.167.204.206:4072 , piece_id=WXORBDMA3KW7WZ67A4FJT6URJRFADUJ7X2PAHIGNG47T3TGMXVTQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.183411202 (550h8m6.054494237s) [noise=true , address=185.167.204.206:4064 , piece_id=OG3G74OU23CEMKA54WQE3DOJ6AKAXBLZ23QWOQC4UEGQQN4JSMXQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.183791146 (550h8m6.054874181s) [noise=true , address=185.167.204.205:4029 , piece_id=6JQB6ZSCQZKWH3JWCQ4DY6BBAX73KC4Y53PRSMEMLWURZ5QWWQQQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.183973743 (550h8m6.055056778s) [noise=true , address=185.167.204.206:4053 , piece_id=YDH77S6JLKJ6IYS6DRUCELP5EJHIC2O44QBO7MIOEUYBML4GIBHA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.184465369 (550h8m6.055548414s) [noise=true , address=185.167.204.205:4008 , piece_id=RTUZPEXBB6JNSMC5YOFQQGDM57YD233W4756ZBQVMFNOVX6YKB4A, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.184501989 (550h8m6.055585024s) [noise=true , address=185.167.204.205:4030 , piece_id=R5DLI4ZTLOVGO56GPLJL5ZNHYSXOSYNJQHFIO4MVVNHWWBMEOV2Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.184856224 (550h8m6.055939259s) [noise=true , address=185.167.204.205:4009 , piece_id=V6RA7Y7FKY4DCPTRRF3BYNBALWDY5VTPXMDUXFE3353KUXLSQJEA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.185063518 (550h8m6.056146563s) [noise=true , address=185.167.204.205:4016 , piece_id=JB7SC3J2VR4U33HNQS2A2CA75THT3XLQCOIFHAJJEK7MQGY2SYAQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.185288617 (550h8m6.056371652s) [noise=true , address=185.167.204.205:4027 , piece_id=PAPNMRJ7E47VOMOXPBBDWKZJHLRMXTWVR5VW3J7KNQSSMHOHWOAQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.186241401 (550h8m6.057324436s) [noise=true , address=185.167.204.205:4024 , piece_id=FM5IMWRSJJ4RY72IX2OOYC54BFKL4FWE3UOAN3GJ26MLH345COVA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.193113658 (550h8m6.064196703s) [noise=true , address=185.167.204.205:4005 , piece_id=R4DFQJZBPJ5WJWHPODV6ILHUDIRBUSAFOMVNCLXB4UNMI6EXTKHA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.193791408 (550h8m6.064874453s) [noise=true , address=185.167.204.206:4068 , piece_id=DAI37M67FWCSVQTQOZ6RSSOJDGUVZO6SNGAHJJ44G3PGS4OJZEMA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.194292262 (550h8m6.065375307s) [noise=true , address=185.167.204.206:4046 , piece_id=VWLVHMBMJ63DQEWYBIKH3Y3NTVWFNW5OXZKMYXXBHVI3VWYXLLPQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.194302652 (550h8m6.065385687s) [noise=true , address=185.167.204.206:4067 , piece_id=VSKTG5RH32N7OEVWJGQVB6WJ72TEVF6DDK25W7NOH4RAXUP52LTQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.195359193 (550h8m6.066442228s) [noise=true , address=185.167.204.205:4012 , piece_id=LV6RUA5BWXG4DMSUVZDBGUAF5IBWUGMLJYUKNIXHRVDEBIOW4Q2Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.198423448 (550h8m6.069506493s) [noise=true , address=185.167.204.206:4070 , piece_id=5A7YYQYREIZRZDR7TRSE7XFW765JVAV7FX6LKWUI2OXAA74XN4CA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.888968408 (550h8m6.760051453s) [noise=true , address=81.171.14.70:4044 , piece_id=DGN74IG6IWAHLEMO6NLL4TJBXZ6ONTN6QUB6HXY6K7SSO3LYOYIQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.892350789 (550h8m6.763433834s) [noise=true , address=81.171.14.70:4039 , piece_id=EDHPQWLQZTP7MKPGEVMP7JXNTW7LFWSXCTRTSXPCBIQPROX76WRA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.904477160 (550h8m6.775560195s) [noise=true , address=81.171.14.70:4040 , piece_id=DYF4GSZBAKQ3H63LTVCMEFWMABY3FXHOJCNVIFYRKLF2J6B52K5A, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.128088281 (550h8m6.999171326s) [noise=true , address=81.171.14.70:4036 , piece_id=CICNNV7JE6TBNC4RAPXKCEUSJAADOUVGSVIVQISM3ZB4COAZQDDA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.430242142 (550h8m7.301325177s) [noise=true , address=89.149.201.147:4009 , piece_id=EVW2NQFYU57PC2IIVXLYIEBMYXJNIC2CLCHW6PS6NYIPCFRUVHDA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.505304332 (550h8m7.376387367s) [noise=true , address=89.149.201.147:4019 , piece_id=MK7ORCLHO3DOWIOVSOWIQMPJ447PHRQGXP3UKITIRTYJSU5MPCCQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.575461543 (550h8m7.446544609s) [noise=true , address=81.171.14.70:4048 , piece_id=TLT47ORGUKGF3MIIEAX7D22VR2FO6W6NL3VFJNHKWD4L3LIMFDLA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.599667265 (550h8m7.4707503s) [noise=true , address=89.149.201.147:4023 , piece_id=5IINX2GVRFLRX2QH24OO2DDANOKOTXTDR6AGVJBGSJ7IAZQ3CASA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.928390728 (550h8m7.799473773s) [noise=true , address=89.149.201.147:4002 , piece_id=G2PABNTDWCK4MKOZF6PONBMRNIXI45JTHJMAZFPEKPEJ3PQ7MTBA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.928417148 (550h8m7.799500193s) [seg_pos={0 0}, upload=482337]: Segment reached optimal threshold of 26 pieces.
Dec 28 13:37:07.928656303 (550h8m7.799739348s) [noise=true , address=81.171.14.70:4030 , piece_id=74YDIKUST7X3JBXVQHVLFHIKH5A4DYH5TNL3I4XUA5A6AHQO2DRA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZoRwa8wLw66Pjumy1ybp6wrZ79RrSU55uYZpwaY3xi95iSYde): protocol: expected piece hash; context canceled; EOF
Dec 28 13:37:07.928664990 (550h8m7.799748035s) [noise=true , address=81.171.14.70:4030 , piece_id=74YDIKUST7X3JBXVQHVLFHIKH5A4DYH5TNL3I4XUA5A6AHQO2DRA, seg_pos={0 0}, upload=482337]: Long tail cancellation due to optimal threshold
Dec 28 13:37:07.928723351 (550h8m7.799806386s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wgtphHKaTejC1qrJ98sDysBwtj2Zeqw2c5UFk1HMqfY8Buvr): protocol: expected piece hash; context canceled; EOF
Dec 28 13:37:07.928733922 (550h8m7.799816957s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Long tail cancellation due to optimal threshold
Dec 28 13:37:07.928754921 (550h8m7.799837966s) [seg_pos={0 0}, upload=482337]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 28 13:37:07.928870170 (550h8m7.799953215s) [upload=482337]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 13:37:07.928878627 (550h8m7.799961672s) [upload=482337]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 13:37:08.166321536 (550h8m8.037404571s) [upload=482337]: Done uploading segments...
Dec 28 13:37:08.166341444 (550h8m8.037424479s) [upload=482337]: Upload finished. err: <nil>
Duration: 17.834295743s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [.......X....................................X........X...X..] duration:
16.318693213s trace:
https://jaeger.dev.storj.io/trace/36d5af3c761e71b1
FAIL (2025-12-28 14:11:03 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
7.49:4360 , piece_id=QDT5N5LDL2HCVXPOOSXWD2TOEZDF2JZWFLDFYSIMN647FI3ZORUQ, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.372193736 (550h40m10.243276781s) [noise=true , address=77.90.31.12:4618 , piece_id=45WRZ26CPUODOAYF5B6RF37DNMDDQWWJUDK7BFKYS3K3LOCNYJCA, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.372518505 (550h40m10.24360155s) [noise=true , address=77.90.31.13:4646 , piece_id=5CDMQF7GG5PDPHIPMFZWDBKWH5ANO3UIX257LJPCWRITNOA7GY4A, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.372848052 (550h40m10.243931097s) [noise=true , address=64.49.13.170:4721 , piece_id=U76TI7IPAHPZAS6XLG66E4L2CGCYH43NH4ZHTYXMXKZR6SGE7SKQ, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.373163072 (550h40m10.244246117s) [noise=true , address=77.90.31.11:4565 , piece_id=4SBLRZK3K4ED73PJ64BI6ANKBWB2AAHQA4SBUNH6CJ63GN3FJR2Q, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.373445089 (550h40m10.244528134s) [noise=true , address=145.79.227.34:4911 , piece_id=24WE4EMZSL2JHQZ7PESCMLYCZOWGDTGJD2XNLUUJEFPSYLRI5E7A, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.373768855 (550h40m10.24485189s) [noise=true , address=77.90.31.18:4819 , piece_id=OYXPAN3EK4QRDEC6VHIDL3GSGEY27G72KII6HTGUMSPYLHCYBNEA, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.374285519 (550h40m10.245368564s) [noise=true , address=77.90.31.14:4678 , piece_id=VBGPSGMNDQZYY7UKRINAV6B3FVI5BPDPYS37FQIJCMLL65F3XOEQ, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.374710689 (550h40m10.245793724s) [noise=true , address=145.79.227.40:4053 , piece_id=RO4ZZQCKCC7T4T6SOT7GGTY626BJ6ESLQG6KATTPNIUM3ICUKHAQ, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.375099329 (550h40m10.246182364s) [noise=true , address=77.90.31.19:4859 , piece_id=A4HD3FPDPCCMHRFBTS3GNGB323RYD5C5AW4ZVSNOTXFI7FLIV37Q, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.376073433 (550h40m10.247156468s) [noise=true , address=77.90.31.15:4704 , piece_id=DXWLFOIKOPE33LMEQECUJVIGRG3LPADUMULLWI4ZBHEOQHD43ULQ, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.376542387 (550h40m10.247625422s) [noise=true , address=145.79.227.44:4221 , piece_id=H72XCWLKYJAXYGGDIWN67H42RUIEOJCPUSWT5IJMFPKETH7LK3KA, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.377379761 (550h40m10.248462796s) [noise=true , address=145.79.227.32:4863 , piece_id=27TOXK44RIF3DV3PDY4RBP6HEFRQJE6N7LNLRJKSU52YCB6A3JKA, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.378022765 (550h40m10.24910581s) [noise=true , address=145.79.227.30:4789 , piece_id=4QT5IUPIK7BOHJROEK2EBFTW434OKC75FZNENMYYCRMBZ7E5BUTA, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.378821687 (550h40m10.249904722s) [noise=true , address=145.79.227.2:4119 , piece_id=EQM3WMBQ7GXRWRBZK4ZWCI23K4LSLXW6QPEABHGMWRHEYUYANKSQ, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:11.676223695 (550h40m11.54730674s) [upload=482805]: Next returned nil segment
Dec 28 14:09:11.676238083 (550h40m11.547321128s) [upload=482805]: Waiting for error group managing segments...
Dec 28 14:09:12.006762275 (550h40m11.87784531s) [noise=true , address=64.49.13.170:4721 , piece_id=U76TI7IPAHPZAS6XLG66E4L2CGCYH43NH4ZHTYXMXKZR6SGE7SKQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.250307967 (550h40m12.121391012s) [noise=true , address=145.79.227.2:4119 , piece_id=EQM3WMBQ7GXRWRBZK4ZWCI23K4LSLXW6QPEABHGMWRHEYUYANKSQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.269380734 (550h40m12.140463769s) [noise=true , address=145.79.227.49:4360 , piece_id=QDT5N5LDL2HCVXPOOSXWD2TOEZDF2JZWFLDFYSIMN647FI3ZORUQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.290818335 (550h40m12.16190138s) [noise=true , address=145.79.227.32:4863 , piece_id=27TOXK44RIF3DV3PDY4RBP6HEFRQJE6N7LNLRJKSU52YCB6A3JKA, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.467679327 (550h40m12.338762362s) [noise=true , address=45.91.116.8:4072 , piece_id=GEVTX6LLENPLN3UQLSGTFTP3DO52IRYUIQFIWC4XMOC3ZF6V4GTQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.486615595 (550h40m12.35769863s) [noise=true , address=77.90.31.12:4618 , piece_id=45WRZ26CPUODOAYF5B6RF37DNMDDQWWJUDK7BFKYS3K3LOCNYJCA, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.566542810 (550h40m12.437625845s) [noise=true , address=77.90.31.19:4859 , piece_id=A4HD3FPDPCCMHRFBTS3GNGB323RYD5C5AW4ZVSNOTXFI7FLIV37Q, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.609337578 (550h40m12.480420623s) [noise=true , address=77.90.31.13:4646 , piece_id=5CDMQF7GG5PDPHIPMFZWDBKWH5ANO3UIX257LJPCWRITNOA7GY4A, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.640856565 (550h40m12.511939611s) [noise=true , address=45.91.116.5:4322 , piece_id=SS6DOCSJHY5Y3DPWMYX45F4VYYXR44JDLYQ2YPY5XIIAYKMAG7LQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.645490329 (550h40m12.516573364s) [noise=true , address=77.90.31.18:4819 , piece_id=OYXPAN3EK4QRDEC6VHIDL3GSGEY27G72KII6HTGUMSPYLHCYBNEA, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:13.103710286 (550h40m12.974793331s) [noise=true , address=145.79.227.34:4911 , piece_id=24WE4EMZSL2JHQZ7PESCMLYCZOWGDTGJD2XNLUUJEFPSYLRI5E7A, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:13.209917141 (550h40m13.081000176s) [noise=true , address=145.79.227.44:4221 , piece_id=H72XCWLKYJAXYGGDIWN67H42RUIEOJCPUSWT5IJMFPKETH7LK3KA, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:13.218355468 (550h40m13.089438514s) [noise=true , address=77.90.31.11:4565 , piece_id=4SBLRZK3K4ED73PJ64BI6ANKBWB2AAHQA4SBUNH6CJ63GN3FJR2Q, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:13.255968741 (550h40m13.127051806s) [noise=true , address=145.79.227.30:4789 , piece_id=4QT5IUPIK7BOHJROEK2EBFTW434OKC75FZNENMYYCRMBZ7E5BUTA, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:13.255996102 (550h40m13.127079137s) [seg_pos={0 0}, upload=482805]: Segment reached optimal threshold of 14 pieces.
Dec 28 14:09:13.256214589 (550h40m13.127297634s) [noise=true , address=145.79.227.40:4053 , piece_id=RO4ZZQCKCC7T4T6SOT7GGTY626BJ6ESLQG6KATTPNIUM3ICUKHAQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DqyTc4MHvxutFXs2WWRCk8MSuxF3Y1HayyQY4Y5sy8cqDLWuG): protocol: expected piece hash; context canceled; EOF
Dec 28 14:09:13.256223686 (550h40m13.127306731s) [noise=true , address=145.79.227.40:4053 , piece_id=RO4ZZQCKCC7T4T6SOT7GGTY626BJ6ESLQG6KATTPNIUM3ICUKHAQ, seg_pos={0 0}, upload=482805]: Long tail cancellation due to optimal threshold
Dec 28 14:09:13.256512506 (550h40m13.127595551s) [noise=true , address=77.90.31.15:4704 , piece_id=DXWLFOIKOPE33LMEQECUJVIGRG3LPADUMULLWI4ZBHEOQHD43ULQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14tJRdZ1guwwsYwPf3LtENRWqYx2jBQpi7uAMMPaB32Gks9zmj): protocol: expected piece hash; context canceled; EOF
Dec 28 14:09:13.256520060 (550h40m13.127603105s) [noise=true , address=77.90.31.15:4704 , piece_id=DXWLFOIKOPE33LMEQECUJVIGRG3LPADUMULLWI4ZBHEOQHD43ULQ, seg_pos={0 0}, upload=482805]: Long tail cancellation due to optimal threshold
Dec 28 14:09:13.256649106 (550h40m13.127732151s) [noise=true , address=77.90.31.14:4678 , piece_id=VBGPSGMNDQZYY7UKRINAV6B3FVI5BPDPYS37FQIJCMLL65F3XOEQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12beu3SUZMPNWfbeHV4tLtx9UVZoEHh4Kt7oBhT59w3bGBE1ovP): protocol: write tcp 188.166.225.170:57374->77.90.31.14:4678: use of closed network connection; write tcp 188.166.225.170:57374->77.90.31.14:4678: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57374->77.90.31.14:4678: use of closed network connection
Dec 28 14:09:13.256655819 (550h40m13.127738864s) [noise=true , address=77.90.31.14:4678 , piece_id=VBGPSGMNDQZYY7UKRINAV6B3FVI5BPDPYS37FQIJCMLL65F3XOEQ, seg_pos={0 0}, upload=482805]: Long tail cancellation due to optimal threshold
Dec 28 14:09:13.256752974 (550h40m13.127836009s) [noise=true , address=64.49.13.169:4710 , piece_id=SANL7TVJFES63XD46OKQL4KIJ6YVHSEWPZKMRISEKC3ZKWQBOMIA, seg_pos={0 0}, upload=482805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:35680->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:35680->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35680->64.49.13.169:4710: use of closed network connection
Dec 28 14:09:13.256758825 (550h40m13.12784186s) [noise=true , address=64.49.13.169:4710 , piece_id=SANL7TVJFES63XD46OKQL4KIJ6YVHSEWPZKMRISEKC3ZKWQBOMIA, seg_pos={0 0}, upload=482805]: Long tail cancellation due to optimal threshold
Dec 28 14:09:13.256776549 (550h40m13.127859594s) [seg_pos={0 0}, upload=482805]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 14:09:13.256856591 (550h40m13.127939626s) [upload=482805]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 14:09:13.256862432 (550h40m13.127945467s) [upload=482805]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 14:09:13.550363717 (550h40m13.421446782s) [upload=482805]: Done uploading segments...
Dec 28 14:09:13.550405126 (550h40m13.421488161s) [upload=482805]: Upload finished. err: <nil>
Duration: 16.318727288s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [.....X............X....................X...X........X.......] duration:
22.14793751s trace:
https://jaeger.dev.storj.io/trace/360e059013858fb1
FAIL (2025-12-28 14:10:55 UTC)
orbiter warning: context deadline exceeded
Duration: 22.147960626s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [.......................................................X...X] duration:
1m0.001655465s trace:
https://jaeger.dev.storj.io/trace/102cf391e061ae77
FAIL (2025-12-28 14:10:41 UTC)
orbiter warning: context deadline exceeded
Duration: 1m0.00167956s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
18 out of 60 is failed (threshold=3) [.....X..X.....X..X.X...XX......X......XXX.XX....XX......XX.X] duration:
30.003043654s trace:
https://jaeger.dev.storj.io/trace/70c16b2ce32ace56
FAIL (2025-12-28 14:10:40 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: PZA2UND5WNXFWJVEGZLVU7HCK7TJ7TUAG3ZUYCPXRATINSWXQHPQ) context deadline exceeded; piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: A3M4L3U6CKNUFMVJVDTLRVCXYGV6SMYKSI3GFN3C2UIPUVZSYTXQ) context deadline exceeded
======LOGS======
Dec 28 14:10:11.360009179 (550h41m11.231092214s) [upload=482820]: Starting upload
Dec 28 14:10:11.360106004 (550h41m11.231189049s) [upload=482820]: Done starting upload
Dec 28 14:10:11.360377411 (550h41m11.231460456s) [upload=482820]: Uploading segments...
Dec 28 14:10:11.360432275 (550h41m11.23151532s) [upload=482820]: Got next segment. Inline: false
Dec 28 14:10:11.360471700 (550h41m11.231554745s) [upload=482820]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 14:10:11.360478302 (550h41m11.231561347s) [upload=482820]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 14:10:11.609170885 (550h41m11.48025393s) [seg_pos={0 0}, upload=482820]: Begin upload segment...
Dec 28 14:10:11.609256888 (550h41m11.480339923s) [seg_pos={0 0}, upload=482820]: Done begin upload segment.
Dec 28 14:10:11.609298126 (550h41m11.480381161s) [noise=true , address=45.91.116.4:4939 , piece_id=ZEJB74D3MTKECKG4KY6U34ZYJYC5LXSN6RUVLWGF2FFCQZIXDYWA, seg_pos={0 0}, upload=482820]: Uploading piece...
Dec 28 14:10:11.609390663 (550h41m11.480473708s) [noise=true , address=145.79.227.22:4511 , piece_id=R4HLC4QP65352TPEBKQ6ZZK67LDPNZXLXL3PQWTJRJZF7WB2KYFA, seg_pos={0 0}, upload=482820]: Uploading piece...
Dec 28 14:10:11.610474597 (550h41m11.481557642s) [noise=true , address=45.91.116.13:4153 , piece_id=CSDMGEJBPHQ23QOT5B4UHPQ2XFSZW6PFJCF2EI5KO64ZGFBERW3A, seg_pos={0 0}, upload=482820]: Uploading piece...
Dec 28 14:10:11.611363760 (550h41m11.482446795s) [noise=true , address=145.79.227.44:4193 , piece_id=XBJVSHRFA3FBSIJMINYHVYAVR5WY7DUMH4APYUYJRKFUAKM7W2LQ, seg_pos={0 0}, upload=482820]: Uploading piece...
Dec 28 14:10:11.611866257 (550h41m11.482949312s) [noise=true , address=145.79.227.50:4375 , piece_id=KMTUDFHTXTX7FUXBHW55TWUMKADOMOTKRKEHI6OFMWQ7P6IJHIWQ, seg_pos={0 0}, upload=482820]: Uploading piece...
Dec 28 14:10:11.612258143 (550h41m11.483341188s) [noise=true , address=145.79.227.7:4295 , piece_id=EATYYPNZEY5EVSQKOMROO766WHP45TZW5UZWAWILLGUJHABF5DPQ, seg_pos={0 0}, upload=482820]: Uploading piece...
Dec 28 14:10:11.612697289 (550h41m11.483780334s) [noise=true , address=45.91.116.5:4297 , piece_id=PZA2UND5WNXFWJVEGZLVU7HCK7TJ7TUAG3ZUYCPXRATINSWXQHPQ, seg_pos={0 0}, upload=482820]: Uploading piece...
Dec 28 14:10:11.613159419 (550h41m11.484242464s) [noise=true , address=145.79.227.39:4013 , piece_id=WHA3GP6UK2LJXR3YYBGHZCONMR3NWOCBDIXAFSKEXXZZBIVOQVVA, seg_pos={0 0}, upload=482820]: Uploading piece...
Dec 28 14:10:11.613660253 (550h41m11.484743298s) [noise=true , address=45.91.116.6:4974 , piece_id=AK2EG3QNLBNLNSSYWOSDOQX3LT47OEI5UOHKYCISCH4UONSGNJRA, seg_pos={0 0}, upload=482820]: Uploading piece...
Dec 28 14:10:11.616281745 (550h41m11.48736478s) [noise=true , address=145.79.227.11:4731 , piece_id=MCGPYILPENBUI2QNXR5KUY6AM4QC57NS6RLA5QB2XTY74RWTDLOA, seg_pos={0 0}, upload=482820]: Uploading piece...
Dec 28 14:10:11.619467571 (550h41m11.490550606s) [noise=true , address=45.91.116.10:4084 , piece_id=GBQNBS4GBOBEUOSOKJIJPKQMOZS7YXKBRNC2IQ4DWZ23ZYBXVA7A, seg_pos={0 0}, upload=482820]: Uploading piece...
Dec 28 14:10:11.621658543 (550h41m11.492741618s) [noise=true , address=145.79.227.16:4909 , piece_id=C73U2HJTBPHJY56D5T2JQFR63CNSCNGVWBZVRH4UMRXBMROUDHAA, seg_pos={0 0}, upload=482820]: Uploading piece...
Dec 28 14:10:11.623986395 (550h41m11.49506944s) [noise=true , address=145.79.227.51:4410 , piece_id=ICDBQH3SDDWCVMGCYBHFKFKVBJBDZ5VERQXEO7BJCL35VMQJYWJQ, seg_pos={0 0}, upload=482820]: Uploading piece...
Dec 28 14:10:11.626432614 (550h41m11.497515659s) [noise=true , address=77.90.31.21:4878 , piece_id=A3M4L3U6CKNUFMVJVDTLRVCXYGV6SMYKSI3GFN3C2UIPUVZSYTXQ, seg_pos={0 0}, upload=482820]: Uploading piece...
Dec 28 14:10:11.631509541 (550h41m11.502592586s) [noise=true , address=77.90.31.10:4519 , piece_id=5N64HDULTS65PIZEHQ7OOJO3WAT6KLYKF7E4S72VE6U73SGTKNZQ, seg_pos={0 0}, upload=482820]: Uploading piece...
Dec 28 14:10:11.632239190 (550h41m11.503322245s) [noise=true , address=145.79.227.48:4301 , piece_id=EYWIOGZRNJAWR7LRJBG6MHZVWQTJFMRYONLCLCEKT2IHT3VMF2IA, seg_pos={0 0}, upload=482820]: Uploading piece...
Dec 28 14:10:13.235793624 (550h41m13.106876659s) [upload=482820]: Next returned nil segment
Dec 28 14:10:13.235808111 (550h41m13.106891146s) [upload=482820]: Waiting for error group managing segments...
Dec 28 14:10:13.617627498 (550h41m13.488710543s) [noise=true , address=145.79.227.50:4375 , piece_id=KMTUDFHTXTX7FUXBHW55TWUMKADOMOTKRKEHI6OFMWQ7P6IJHIWQ, seg_pos={0 0}, upload=482820]: Done uploading piece. err: <nil>
Dec 28 14:10:13.628132322 (550h41m13.499215367s) [noise=true , address=145.79.227.39:4013 , piece_id=WHA3GP6UK2LJXR3YYBGHZCONMR3NWOCBDIXAFSKEXXZZBIVOQVVA, seg_pos={0 0}, upload=482820]: Done uploading piece. err: <nil>
Dec 28 14:10:13.630463911 (550h41m13.501546956s) [noise=true , address=145.79.227.16:4909 , piece_id=C73U2HJTBPHJY56D5T2JQFR63CNSCNGVWBZVRH4UMRXBMROUDHAA, seg_pos={0 0}, upload=482820]: Done uploading piece. err: <nil>
Dec 28 14:10:13.631499583 (550h41m13.502582628s) [noise=true , address=145.79.227.7:4295 , piece_id=EATYYPNZEY5EVSQKOMROO766WHP45TZW5UZWAWILLGUJHABF5DPQ, seg_pos={0 0}, upload=482820]: Done uploading piece. err: <nil>
Dec 28 14:10:13.642151315 (550h41m13.513234371s) [noise=true , address=145.79.227.48:4301 , piece_id=EYWIOGZRNJAWR7LRJBG6MHZVWQTJFMRYONLCLCEKT2IHT3VMF2IA, seg_pos={0 0}, upload=482820]: Done uploading piece. err: <nil>
Dec 28 14:10:13.643527867 (550h41m13.514610912s) [noise=true , address=145.79.227.22:4511 , piece_id=R4HLC4QP65352TPEBKQ6ZZK67LDPNZXLXL3PQWTJRJZF7WB2KYFA, seg_pos={0 0}, upload=482820]: Done uploading piece. err: <nil>
Dec 28 14:10:13.644981555 (550h41m13.516064591s) [noise=true , address=145.79.227.51:4410 , piece_id=ICDBQH3SDDWCVMGCYBHFKFKVBJBDZ5VERQXEO7BJCL35VMQJYWJQ, seg_pos={0 0}, upload=482820]: Done uploading piece. err: <nil>
Dec 28 14:10:13.652449425 (550h41m13.52353246s) [noise=true , address=145.79.227.11:4731 , piece_id=MCGPYILPENBUI2QNXR5KUY6AM4QC57NS6RLA5QB2XTY74RWTDLOA, seg_pos={0 0}, upload=482820]: Done uploading piece. err: <nil>
Dec 28 14:10:13.886561964 (550h41m13.75764504s) [noise=true , address=45.91.116.5:4297 , piece_id=PZA2UND5WNXFWJVEGZLVU7HCK7TJ7TUAG3ZUYCPXRATINSWXQHPQ, seg_pos={0 0}, upload=482820]: Done uploading piece. err: <nil>
Dec 28 14:10:13.970472053 (550h41m13.841555098s) [noise=true , address=45.91.116.13:4153 , piece_id=CSDMGEJBPHQ23QOT5B4UHPQ2XFSZW6PFJCF2EI5KO64ZGFBERW3A, seg_pos={0 0}, upload=482820]: Done uploading piece. err: <nil>
Dec 28 14:10:13.988603158 (550h41m13.859686193s) [noise=true , address=45.91.116.10:4084 , piece_id=GBQNBS4GBOBEUOSOKJIJPKQMOZS7YXKBRNC2IQ4DWZ23ZYBXVA7A, seg_pos={0 0}, upload=482820]: Done uploading piece. err: <nil>
Dec 28 14:10:14.165726920 (550h41m14.036809965s) [noise=true , address=45.91.116.4:4939 , piece_id=ZEJB74D3MTKECKG4KY6U34ZYJYC5LXSN6RUVLWGF2FFCQZIXDYWA, seg_pos={0 0}, upload=482820]: Done uploading piece. err: <nil>
Dec 28 14:10:14.533320460 (550h41m14.404403495s) [noise=true , address=77.90.31.21:4878 , piece_id=A3M4L3U6CKNUFMVJVDTLRVCXYGV6SMYKSI3GFN3C2UIPUVZSYTXQ, seg_pos={0 0}, upload=482820]: Done uploading piece. err: <nil>
Dec 28 14:10:14.533344054 (550h41m14.404427099s) [seg_pos={0 0}, upload=482820]: Segment reached optimal threshold of 13 pieces.
Dec 28 14:10:14.533578160 (550h41m14.404661205s) [noise=true , address=145.79.227.44:4193 , piece_id=XBJVSHRFA3FBSIJMINYHVYAVR5WY7DUMH4APYUYJRKFUAKM7W2LQ, seg_pos={0 0}, upload=482820]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF
Dec 28 14:10:14.533586646 (550h41m14.404669681s) [noise=true , address=145.79.227.44:4193 , piece_id=XBJVSHRFA3FBSIJMINYHVYAVR5WY7DUMH4APYUYJRKFUAKM7W2LQ, seg_pos={0 0}, upload=482820]: Long tail cancellation due to optimal threshold
Dec 28 14:10:14.533635800 (550h41m14.404718845s) [noise=true , address=77.90.31.10:4519 , piece_id=5N64HDULTS65PIZEHQ7OOJO3WAT6KLYKF7E4S72VE6U73SGTKNZQ, seg_pos={0 0}, upload=482820]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF
Dec 28 14:10:14.533645909 (550h41m14.404728944s) [noise=true , address=77.90.31.10:4519 , piece_id=5N64HDULTS65PIZEHQ7OOJO3WAT6KLYKF7E4S72VE6U73SGTKNZQ, seg_pos={0 0}, upload=482820]: Long tail cancellation due to optimal threshold
Dec 28 14:10:14.533888391 (550h41m14.404971436s) [noise=true , address=45.91.116.6:4974 , piece_id=AK2EG3QNLBNLNSSYWOSDOQX3LT47OEI5UOHKYCISCH4UONSGNJRA, seg_pos={0 0}, upload=482820]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: read tcp 188.166.225.170:53800->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 28 14:10:14.533901756 (550h41m14.404984801s) [noise=true , address=45.91.116.6:4974 , piece_id=AK2EG3QNLBNLNSSYWOSDOQX3LT47OEI5UOHKYCISCH4UONSGNJRA, seg_pos={0 0}, upload=482820]: Long tail cancellation due to optimal threshold
Dec 28 14:10:14.533916906 (550h41m14.404999941s) [seg_pos={0 0}, upload=482820]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 14:10:14.533983082 (550h41m14.405066127s) [upload=482820]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 14:10:14.533987550 (550h41m14.405070595s) [upload=482820]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 14:10:14.828877259 (550h41m14.699960304s) [upload=482820]: Done uploading segments...
Dec 28 14:10:14.828898408 (550h41m14.699981454s) [upload=482820]: Upload finished. err: <nil>
Duration: 30.00305708s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [........X....................................X........X...X.] duration:
24.781747209s trace:
https://jaeger.dev.storj.io/trace/36d5af3c761e71b1
FAIL (2025-12-28 14:10:34 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
7.49:4360 , piece_id=QDT5N5LDL2HCVXPOOSXWD2TOEZDF2JZWFLDFYSIMN647FI3ZORUQ, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.372193736 (550h40m10.243276781s) [noise=true , address=77.90.31.12:4618 , piece_id=45WRZ26CPUODOAYF5B6RF37DNMDDQWWJUDK7BFKYS3K3LOCNYJCA, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.372518505 (550h40m10.24360155s) [noise=true , address=77.90.31.13:4646 , piece_id=5CDMQF7GG5PDPHIPMFZWDBKWH5ANO3UIX257LJPCWRITNOA7GY4A, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.372848052 (550h40m10.243931097s) [noise=true , address=64.49.13.170:4721 , piece_id=U76TI7IPAHPZAS6XLG66E4L2CGCYH43NH4ZHTYXMXKZR6SGE7SKQ, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.373163072 (550h40m10.244246117s) [noise=true , address=77.90.31.11:4565 , piece_id=4SBLRZK3K4ED73PJ64BI6ANKBWB2AAHQA4SBUNH6CJ63GN3FJR2Q, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.373445089 (550h40m10.244528134s) [noise=true , address=145.79.227.34:4911 , piece_id=24WE4EMZSL2JHQZ7PESCMLYCZOWGDTGJD2XNLUUJEFPSYLRI5E7A, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.373768855 (550h40m10.24485189s) [noise=true , address=77.90.31.18:4819 , piece_id=OYXPAN3EK4QRDEC6VHIDL3GSGEY27G72KII6HTGUMSPYLHCYBNEA, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.374285519 (550h40m10.245368564s) [noise=true , address=77.90.31.14:4678 , piece_id=VBGPSGMNDQZYY7UKRINAV6B3FVI5BPDPYS37FQIJCMLL65F3XOEQ, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.374710689 (550h40m10.245793724s) [noise=true , address=145.79.227.40:4053 , piece_id=RO4ZZQCKCC7T4T6SOT7GGTY626BJ6ESLQG6KATTPNIUM3ICUKHAQ, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.375099329 (550h40m10.246182364s) [noise=true , address=77.90.31.19:4859 , piece_id=A4HD3FPDPCCMHRFBTS3GNGB323RYD5C5AW4ZVSNOTXFI7FLIV37Q, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.376073433 (550h40m10.247156468s) [noise=true , address=77.90.31.15:4704 , piece_id=DXWLFOIKOPE33LMEQECUJVIGRG3LPADUMULLWI4ZBHEOQHD43ULQ, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.376542387 (550h40m10.247625422s) [noise=true , address=145.79.227.44:4221 , piece_id=H72XCWLKYJAXYGGDIWN67H42RUIEOJCPUSWT5IJMFPKETH7LK3KA, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.377379761 (550h40m10.248462796s) [noise=true , address=145.79.227.32:4863 , piece_id=27TOXK44RIF3DV3PDY4RBP6HEFRQJE6N7LNLRJKSU52YCB6A3JKA, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.378022765 (550h40m10.24910581s) [noise=true , address=145.79.227.30:4789 , piece_id=4QT5IUPIK7BOHJROEK2EBFTW434OKC75FZNENMYYCRMBZ7E5BUTA, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.378821687 (550h40m10.249904722s) [noise=true , address=145.79.227.2:4119 , piece_id=EQM3WMBQ7GXRWRBZK4ZWCI23K4LSLXW6QPEABHGMWRHEYUYANKSQ, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:11.676223695 (550h40m11.54730674s) [upload=482805]: Next returned nil segment
Dec 28 14:09:11.676238083 (550h40m11.547321128s) [upload=482805]: Waiting for error group managing segments...
Dec 28 14:09:12.006762275 (550h40m11.87784531s) [noise=true , address=64.49.13.170:4721 , piece_id=U76TI7IPAHPZAS6XLG66E4L2CGCYH43NH4ZHTYXMXKZR6SGE7SKQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.250307967 (550h40m12.121391012s) [noise=true , address=145.79.227.2:4119 , piece_id=EQM3WMBQ7GXRWRBZK4ZWCI23K4LSLXW6QPEABHGMWRHEYUYANKSQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.269380734 (550h40m12.140463769s) [noise=true , address=145.79.227.49:4360 , piece_id=QDT5N5LDL2HCVXPOOSXWD2TOEZDF2JZWFLDFYSIMN647FI3ZORUQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.290818335 (550h40m12.16190138s) [noise=true , address=145.79.227.32:4863 , piece_id=27TOXK44RIF3DV3PDY4RBP6HEFRQJE6N7LNLRJKSU52YCB6A3JKA, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.467679327 (550h40m12.338762362s) [noise=true , address=45.91.116.8:4072 , piece_id=GEVTX6LLENPLN3UQLSGTFTP3DO52IRYUIQFIWC4XMOC3ZF6V4GTQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.486615595 (550h40m12.35769863s) [noise=true , address=77.90.31.12:4618 , piece_id=45WRZ26CPUODOAYF5B6RF37DNMDDQWWJUDK7BFKYS3K3LOCNYJCA, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.566542810 (550h40m12.437625845s) [noise=true , address=77.90.31.19:4859 , piece_id=A4HD3FPDPCCMHRFBTS3GNGB323RYD5C5AW4ZVSNOTXFI7FLIV37Q, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.609337578 (550h40m12.480420623s) [noise=true , address=77.90.31.13:4646 , piece_id=5CDMQF7GG5PDPHIPMFZWDBKWH5ANO3UIX257LJPCWRITNOA7GY4A, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.640856565 (550h40m12.511939611s) [noise=true , address=45.91.116.5:4322 , piece_id=SS6DOCSJHY5Y3DPWMYX45F4VYYXR44JDLYQ2YPY5XIIAYKMAG7LQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.645490329 (550h40m12.516573364s) [noise=true , address=77.90.31.18:4819 , piece_id=OYXPAN3EK4QRDEC6VHIDL3GSGEY27G72KII6HTGUMSPYLHCYBNEA, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:13.103710286 (550h40m12.974793331s) [noise=true , address=145.79.227.34:4911 , piece_id=24WE4EMZSL2JHQZ7PESCMLYCZOWGDTGJD2XNLUUJEFPSYLRI5E7A, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:13.209917141 (550h40m13.081000176s) [noise=true , address=145.79.227.44:4221 , piece_id=H72XCWLKYJAXYGGDIWN67H42RUIEOJCPUSWT5IJMFPKETH7LK3KA, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:13.218355468 (550h40m13.089438514s) [noise=true , address=77.90.31.11:4565 , piece_id=4SBLRZK3K4ED73PJ64BI6ANKBWB2AAHQA4SBUNH6CJ63GN3FJR2Q, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:13.255968741 (550h40m13.127051806s) [noise=true , address=145.79.227.30:4789 , piece_id=4QT5IUPIK7BOHJROEK2EBFTW434OKC75FZNENMYYCRMBZ7E5BUTA, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:13.255996102 (550h40m13.127079137s) [seg_pos={0 0}, upload=482805]: Segment reached optimal threshold of 14 pieces.
Dec 28 14:09:13.256214589 (550h40m13.127297634s) [noise=true , address=145.79.227.40:4053 , piece_id=RO4ZZQCKCC7T4T6SOT7GGTY626BJ6ESLQG6KATTPNIUM3ICUKHAQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DqyTc4MHvxutFXs2WWRCk8MSuxF3Y1HayyQY4Y5sy8cqDLWuG): protocol: expected piece hash; context canceled; EOF
Dec 28 14:09:13.256223686 (550h40m13.127306731s) [noise=true , address=145.79.227.40:4053 , piece_id=RO4ZZQCKCC7T4T6SOT7GGTY626BJ6ESLQG6KATTPNIUM3ICUKHAQ, seg_pos={0 0}, upload=482805]: Long tail cancellation due to optimal threshold
Dec 28 14:09:13.256512506 (550h40m13.127595551s) [noise=true , address=77.90.31.15:4704 , piece_id=DXWLFOIKOPE33LMEQECUJVIGRG3LPADUMULLWI4ZBHEOQHD43ULQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14tJRdZ1guwwsYwPf3LtENRWqYx2jBQpi7uAMMPaB32Gks9zmj): protocol: expected piece hash; context canceled; EOF
Dec 28 14:09:13.256520060 (550h40m13.127603105s) [noise=true , address=77.90.31.15:4704 , piece_id=DXWLFOIKOPE33LMEQECUJVIGRG3LPADUMULLWI4ZBHEOQHD43ULQ, seg_pos={0 0}, upload=482805]: Long tail cancellation due to optimal threshold
Dec 28 14:09:13.256649106 (550h40m13.127732151s) [noise=true , address=77.90.31.14:4678 , piece_id=VBGPSGMNDQZYY7UKRINAV6B3FVI5BPDPYS37FQIJCMLL65F3XOEQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12beu3SUZMPNWfbeHV4tLtx9UVZoEHh4Kt7oBhT59w3bGBE1ovP): protocol: write tcp 188.166.225.170:57374->77.90.31.14:4678: use of closed network connection; write tcp 188.166.225.170:57374->77.90.31.14:4678: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57374->77.90.31.14:4678: use of closed network connection
Dec 28 14:09:13.256655819 (550h40m13.127738864s) [noise=true , address=77.90.31.14:4678 , piece_id=VBGPSGMNDQZYY7UKRINAV6B3FVI5BPDPYS37FQIJCMLL65F3XOEQ, seg_pos={0 0}, upload=482805]: Long tail cancellation due to optimal threshold
Dec 28 14:09:13.256752974 (550h40m13.127836009s) [noise=true , address=64.49.13.169:4710 , piece_id=SANL7TVJFES63XD46OKQL4KIJ6YVHSEWPZKMRISEKC3ZKWQBOMIA, seg_pos={0 0}, upload=482805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:35680->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:35680->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35680->64.49.13.169:4710: use of closed network connection
Dec 28 14:09:13.256758825 (550h40m13.12784186s) [noise=true , address=64.49.13.169:4710 , piece_id=SANL7TVJFES63XD46OKQL4KIJ6YVHSEWPZKMRISEKC3ZKWQBOMIA, seg_pos={0 0}, upload=482805]: Long tail cancellation due to optimal threshold
Dec 28 14:09:13.256776549 (550h40m13.127859594s) [seg_pos={0 0}, upload=482805]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 14:09:13.256856591 (550h40m13.127939626s) [upload=482805]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 14:09:13.256862432 (550h40m13.127945467s) [upload=482805]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 14:09:13.550363717 (550h40m13.421446782s) [upload=482805]: Done uploading segments...
Dec 28 14:09:13.550405126 (550h40m13.421488161s) [upload=482805]: Upload finished. err: <nil>
Duration: 24.781767399s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
53 out of 60 is failed (threshold=6) [XXXX.XXXXXXX.XXXXXXXXXXXX.XXXXXXX.XX..XXXXXXXXXXXXXXXXXXXXX.] duration:
3.456737531s trace:
https://jaeger.dev.storj.io/trace/48d01a086303dbc
FAIL (2025-12-28 14:10:29 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.456767659s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [......X............X....................X...X........X......] duration:
24.185047083s trace:
https://jaeger.dev.storj.io/trace/360e059013858fb1
FAIL (2025-12-28 14:10:26 UTC)
orbiter warning: context deadline exceeded
Duration: 24.185107178s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
54 out of 60 is failed (threshold=6) [XXXXX.XXXXXXX.XXXXXXXXXXXX.XXXXXXX.XX..XXXXXXXXXXXXXXXXXXXXX] duration:
5.002063789s trace:
https://jaeger.dev.storj.io/trace/48d01a086303dbc
FAIL (2025-12-28 14:10:14 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.002090669s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [.................X..........................................] duration:
13.637095507s trace:
https://jaeger.dev.storj.io/trace/5fe9d6dbb79fbe44
FAIL (2025-12-28 14:10:13 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
67.204.205:4027 , piece_id=PAPNMRJ7E47VOMOXPBBDWKZJHLRMXTWVR5VW3J7KNQSSMHOHWOAQ, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.967737953 (550h8m4.838820988s) [noise=true , address=185.167.204.206:4072 , piece_id=WXORBDMA3KW7WZ67A4FJT6URJRFADUJ7X2PAHIGNG47T3TGMXVTQ, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.968193240 (550h8m4.839276285s) [noise=true , address=185.167.204.205:4030 , piece_id=R5DLI4ZTLOVGO56GPLJL5ZNHYSXOSYNJQHFIO4MVVNHWWBMEOV2Q, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.968780638 (550h8m4.839863683s) [noise=true , address=89.149.201.147:4019 , piece_id=MK7ORCLHO3DOWIOVSOWIQMPJ447PHRQGXP3UKITIRTYJSU5MPCCQ, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.969308924 (550h8m4.840391959s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.970088349 (550h8m4.841171404s) [noise=true , address=81.171.14.70:4039 , piece_id=EDHPQWLQZTP7MKPGEVMP7JXNTW7LFWSXCTRTSXPCBIQPROX76WRA, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.970325381 (550h8m4.841408426s) [noise=true , address=81.171.14.70:4036 , piece_id=CICNNV7JE6TBNC4RAPXKCEUSJAADOUVGSVIVQISM3ZB4COAZQDDA, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.970783813 (550h8m4.841866858s) [noise=true , address=81.171.14.70:4044 , piece_id=DGN74IG6IWAHLEMO6NLL4TJBXZ6ONTN6QUB6HXY6K7SSO3LYOYIQ, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.971205406 (550h8m4.842288441s) [noise=true , address=81.171.14.70:4048 , piece_id=TLT47ORGUKGF3MIIEAX7D22VR2FO6W6NL3VFJNHKWD4L3LIMFDLA, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.971606299 (550h8m4.842689344s) [noise=true , address=185.167.204.206:4067 , piece_id=VSKTG5RH32N7OEVWJGQVB6WJ72TEVF6DDK25W7NOH4RAXUP52LTQ, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.972162538 (550h8m4.843245583s) [noise=true , address=89.149.201.147:4002 , piece_id=G2PABNTDWCK4MKOZF6PONBMRNIXI45JTHJMAZFPEKPEJ3PQ7MTBA, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:05.795167814 (550h8m5.666250849s) [upload=482337]: Next returned nil segment
Dec 28 13:37:05.795184566 (550h8m5.666267611s) [upload=482337]: Waiting for error group managing segments...
Dec 28 13:37:06.178573801 (550h8m6.049656836s) [noise=true , address=185.167.204.206:4049 , piece_id=6RG2DZV6FPTIRE63ZNNST42TSWEBQ456YXIPPVYHIXV55GWFVJ4Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.183060835 (550h8m6.05414391s) [noise=true , address=185.167.204.206:4072 , piece_id=WXORBDMA3KW7WZ67A4FJT6URJRFADUJ7X2PAHIGNG47T3TGMXVTQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.183411202 (550h8m6.054494237s) [noise=true , address=185.167.204.206:4064 , piece_id=OG3G74OU23CEMKA54WQE3DOJ6AKAXBLZ23QWOQC4UEGQQN4JSMXQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.183791146 (550h8m6.054874181s) [noise=true , address=185.167.204.205:4029 , piece_id=6JQB6ZSCQZKWH3JWCQ4DY6BBAX73KC4Y53PRSMEMLWURZ5QWWQQQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.183973743 (550h8m6.055056778s) [noise=true , address=185.167.204.206:4053 , piece_id=YDH77S6JLKJ6IYS6DRUCELP5EJHIC2O44QBO7MIOEUYBML4GIBHA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.184465369 (550h8m6.055548414s) [noise=true , address=185.167.204.205:4008 , piece_id=RTUZPEXBB6JNSMC5YOFQQGDM57YD233W4756ZBQVMFNOVX6YKB4A, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.184501989 (550h8m6.055585024s) [noise=true , address=185.167.204.205:4030 , piece_id=R5DLI4ZTLOVGO56GPLJL5ZNHYSXOSYNJQHFIO4MVVNHWWBMEOV2Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.184856224 (550h8m6.055939259s) [noise=true , address=185.167.204.205:4009 , piece_id=V6RA7Y7FKY4DCPTRRF3BYNBALWDY5VTPXMDUXFE3353KUXLSQJEA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.185063518 (550h8m6.056146563s) [noise=true , address=185.167.204.205:4016 , piece_id=JB7SC3J2VR4U33HNQS2A2CA75THT3XLQCOIFHAJJEK7MQGY2SYAQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.185288617 (550h8m6.056371652s) [noise=true , address=185.167.204.205:4027 , piece_id=PAPNMRJ7E47VOMOXPBBDWKZJHLRMXTWVR5VW3J7KNQSSMHOHWOAQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.186241401 (550h8m6.057324436s) [noise=true , address=185.167.204.205:4024 , piece_id=FM5IMWRSJJ4RY72IX2OOYC54BFKL4FWE3UOAN3GJ26MLH345COVA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.193113658 (550h8m6.064196703s) [noise=true , address=185.167.204.205:4005 , piece_id=R4DFQJZBPJ5WJWHPODV6ILHUDIRBUSAFOMVNCLXB4UNMI6EXTKHA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.193791408 (550h8m6.064874453s) [noise=true , address=185.167.204.206:4068 , piece_id=DAI37M67FWCSVQTQOZ6RSSOJDGUVZO6SNGAHJJ44G3PGS4OJZEMA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.194292262 (550h8m6.065375307s) [noise=true , address=185.167.204.206:4046 , piece_id=VWLVHMBMJ63DQEWYBIKH3Y3NTVWFNW5OXZKMYXXBHVI3VWYXLLPQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.194302652 (550h8m6.065385687s) [noise=true , address=185.167.204.206:4067 , piece_id=VSKTG5RH32N7OEVWJGQVB6WJ72TEVF6DDK25W7NOH4RAXUP52LTQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.195359193 (550h8m6.066442228s) [noise=true , address=185.167.204.205:4012 , piece_id=LV6RUA5BWXG4DMSUVZDBGUAF5IBWUGMLJYUKNIXHRVDEBIOW4Q2Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.198423448 (550h8m6.069506493s) [noise=true , address=185.167.204.206:4070 , piece_id=5A7YYQYREIZRZDR7TRSE7XFW765JVAV7FX6LKWUI2OXAA74XN4CA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.888968408 (550h8m6.760051453s) [noise=true , address=81.171.14.70:4044 , piece_id=DGN74IG6IWAHLEMO6NLL4TJBXZ6ONTN6QUB6HXY6K7SSO3LYOYIQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.892350789 (550h8m6.763433834s) [noise=true , address=81.171.14.70:4039 , piece_id=EDHPQWLQZTP7MKPGEVMP7JXNTW7LFWSXCTRTSXPCBIQPROX76WRA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.904477160 (550h8m6.775560195s) [noise=true , address=81.171.14.70:4040 , piece_id=DYF4GSZBAKQ3H63LTVCMEFWMABY3FXHOJCNVIFYRKLF2J6B52K5A, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.128088281 (550h8m6.999171326s) [noise=true , address=81.171.14.70:4036 , piece_id=CICNNV7JE6TBNC4RAPXKCEUSJAADOUVGSVIVQISM3ZB4COAZQDDA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.430242142 (550h8m7.301325177s) [noise=true , address=89.149.201.147:4009 , piece_id=EVW2NQFYU57PC2IIVXLYIEBMYXJNIC2CLCHW6PS6NYIPCFRUVHDA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.505304332 (550h8m7.376387367s) [noise=true , address=89.149.201.147:4019 , piece_id=MK7ORCLHO3DOWIOVSOWIQMPJ447PHRQGXP3UKITIRTYJSU5MPCCQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.575461543 (550h8m7.446544609s) [noise=true , address=81.171.14.70:4048 , piece_id=TLT47ORGUKGF3MIIEAX7D22VR2FO6W6NL3VFJNHKWD4L3LIMFDLA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.599667265 (550h8m7.4707503s) [noise=true , address=89.149.201.147:4023 , piece_id=5IINX2GVRFLRX2QH24OO2DDANOKOTXTDR6AGVJBGSJ7IAZQ3CASA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.928390728 (550h8m7.799473773s) [noise=true , address=89.149.201.147:4002 , piece_id=G2PABNTDWCK4MKOZF6PONBMRNIXI45JTHJMAZFPEKPEJ3PQ7MTBA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.928417148 (550h8m7.799500193s) [seg_pos={0 0}, upload=482337]: Segment reached optimal threshold of 26 pieces.
Dec 28 13:37:07.928656303 (550h8m7.799739348s) [noise=true , address=81.171.14.70:4030 , piece_id=74YDIKUST7X3JBXVQHVLFHIKH5A4DYH5TNL3I4XUA5A6AHQO2DRA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZoRwa8wLw66Pjumy1ybp6wrZ79RrSU55uYZpwaY3xi95iSYde): protocol: expected piece hash; context canceled; EOF
Dec 28 13:37:07.928664990 (550h8m7.799748035s) [noise=true , address=81.171.14.70:4030 , piece_id=74YDIKUST7X3JBXVQHVLFHIKH5A4DYH5TNL3I4XUA5A6AHQO2DRA, seg_pos={0 0}, upload=482337]: Long tail cancellation due to optimal threshold
Dec 28 13:37:07.928723351 (550h8m7.799806386s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wgtphHKaTejC1qrJ98sDysBwtj2Zeqw2c5UFk1HMqfY8Buvr): protocol: expected piece hash; context canceled; EOF
Dec 28 13:37:07.928733922 (550h8m7.799816957s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Long tail cancellation due to optimal threshold
Dec 28 13:37:07.928754921 (550h8m7.799837966s) [seg_pos={0 0}, upload=482337]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 28 13:37:07.928870170 (550h8m7.799953215s) [upload=482337]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 13:37:07.928878627 (550h8m7.799961672s) [upload=482337]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 13:37:08.166321536 (550h8m8.037404571s) [upload=482337]: Done uploading segments...
Dec 28 13:37:08.166341444 (550h8m8.037424479s) [upload=482337]: Upload finished. err: <nil>
Duration: 13.637125515s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [X........X....................................X........X...X] duration:
1m0.002745847s trace:
https://jaeger.dev.storj.io/trace/36d5af3c761e71b1
FAIL (2025-12-28 14:10:09 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
7.49:4360 , piece_id=QDT5N5LDL2HCVXPOOSXWD2TOEZDF2JZWFLDFYSIMN647FI3ZORUQ, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.372193736 (550h40m10.243276781s) [noise=true , address=77.90.31.12:4618 , piece_id=45WRZ26CPUODOAYF5B6RF37DNMDDQWWJUDK7BFKYS3K3LOCNYJCA, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.372518505 (550h40m10.24360155s) [noise=true , address=77.90.31.13:4646 , piece_id=5CDMQF7GG5PDPHIPMFZWDBKWH5ANO3UIX257LJPCWRITNOA7GY4A, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.372848052 (550h40m10.243931097s) [noise=true , address=64.49.13.170:4721 , piece_id=U76TI7IPAHPZAS6XLG66E4L2CGCYH43NH4ZHTYXMXKZR6SGE7SKQ, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.373163072 (550h40m10.244246117s) [noise=true , address=77.90.31.11:4565 , piece_id=4SBLRZK3K4ED73PJ64BI6ANKBWB2AAHQA4SBUNH6CJ63GN3FJR2Q, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.373445089 (550h40m10.244528134s) [noise=true , address=145.79.227.34:4911 , piece_id=24WE4EMZSL2JHQZ7PESCMLYCZOWGDTGJD2XNLUUJEFPSYLRI5E7A, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.373768855 (550h40m10.24485189s) [noise=true , address=77.90.31.18:4819 , piece_id=OYXPAN3EK4QRDEC6VHIDL3GSGEY27G72KII6HTGUMSPYLHCYBNEA, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.374285519 (550h40m10.245368564s) [noise=true , address=77.90.31.14:4678 , piece_id=VBGPSGMNDQZYY7UKRINAV6B3FVI5BPDPYS37FQIJCMLL65F3XOEQ, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.374710689 (550h40m10.245793724s) [noise=true , address=145.79.227.40:4053 , piece_id=RO4ZZQCKCC7T4T6SOT7GGTY626BJ6ESLQG6KATTPNIUM3ICUKHAQ, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.375099329 (550h40m10.246182364s) [noise=true , address=77.90.31.19:4859 , piece_id=A4HD3FPDPCCMHRFBTS3GNGB323RYD5C5AW4ZVSNOTXFI7FLIV37Q, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.376073433 (550h40m10.247156468s) [noise=true , address=77.90.31.15:4704 , piece_id=DXWLFOIKOPE33LMEQECUJVIGRG3LPADUMULLWI4ZBHEOQHD43ULQ, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.376542387 (550h40m10.247625422s) [noise=true , address=145.79.227.44:4221 , piece_id=H72XCWLKYJAXYGGDIWN67H42RUIEOJCPUSWT5IJMFPKETH7LK3KA, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.377379761 (550h40m10.248462796s) [noise=true , address=145.79.227.32:4863 , piece_id=27TOXK44RIF3DV3PDY4RBP6HEFRQJE6N7LNLRJKSU52YCB6A3JKA, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.378022765 (550h40m10.24910581s) [noise=true , address=145.79.227.30:4789 , piece_id=4QT5IUPIK7BOHJROEK2EBFTW434OKC75FZNENMYYCRMBZ7E5BUTA, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.378821687 (550h40m10.249904722s) [noise=true , address=145.79.227.2:4119 , piece_id=EQM3WMBQ7GXRWRBZK4ZWCI23K4LSLXW6QPEABHGMWRHEYUYANKSQ, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:11.676223695 (550h40m11.54730674s) [upload=482805]: Next returned nil segment
Dec 28 14:09:11.676238083 (550h40m11.547321128s) [upload=482805]: Waiting for error group managing segments...
Dec 28 14:09:12.006762275 (550h40m11.87784531s) [noise=true , address=64.49.13.170:4721 , piece_id=U76TI7IPAHPZAS6XLG66E4L2CGCYH43NH4ZHTYXMXKZR6SGE7SKQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.250307967 (550h40m12.121391012s) [noise=true , address=145.79.227.2:4119 , piece_id=EQM3WMBQ7GXRWRBZK4ZWCI23K4LSLXW6QPEABHGMWRHEYUYANKSQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.269380734 (550h40m12.140463769s) [noise=true , address=145.79.227.49:4360 , piece_id=QDT5N5LDL2HCVXPOOSXWD2TOEZDF2JZWFLDFYSIMN647FI3ZORUQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.290818335 (550h40m12.16190138s) [noise=true , address=145.79.227.32:4863 , piece_id=27TOXK44RIF3DV3PDY4RBP6HEFRQJE6N7LNLRJKSU52YCB6A3JKA, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.467679327 (550h40m12.338762362s) [noise=true , address=45.91.116.8:4072 , piece_id=GEVTX6LLENPLN3UQLSGTFTP3DO52IRYUIQFIWC4XMOC3ZF6V4GTQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.486615595 (550h40m12.35769863s) [noise=true , address=77.90.31.12:4618 , piece_id=45WRZ26CPUODOAYF5B6RF37DNMDDQWWJUDK7BFKYS3K3LOCNYJCA, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.566542810 (550h40m12.437625845s) [noise=true , address=77.90.31.19:4859 , piece_id=A4HD3FPDPCCMHRFBTS3GNGB323RYD5C5AW4ZVSNOTXFI7FLIV37Q, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.609337578 (550h40m12.480420623s) [noise=true , address=77.90.31.13:4646 , piece_id=5CDMQF7GG5PDPHIPMFZWDBKWH5ANO3UIX257LJPCWRITNOA7GY4A, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.640856565 (550h40m12.511939611s) [noise=true , address=45.91.116.5:4322 , piece_id=SS6DOCSJHY5Y3DPWMYX45F4VYYXR44JDLYQ2YPY5XIIAYKMAG7LQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.645490329 (550h40m12.516573364s) [noise=true , address=77.90.31.18:4819 , piece_id=OYXPAN3EK4QRDEC6VHIDL3GSGEY27G72KII6HTGUMSPYLHCYBNEA, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:13.103710286 (550h40m12.974793331s) [noise=true , address=145.79.227.34:4911 , piece_id=24WE4EMZSL2JHQZ7PESCMLYCZOWGDTGJD2XNLUUJEFPSYLRI5E7A, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:13.209917141 (550h40m13.081000176s) [noise=true , address=145.79.227.44:4221 , piece_id=H72XCWLKYJAXYGGDIWN67H42RUIEOJCPUSWT5IJMFPKETH7LK3KA, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:13.218355468 (550h40m13.089438514s) [noise=true , address=77.90.31.11:4565 , piece_id=4SBLRZK3K4ED73PJ64BI6ANKBWB2AAHQA4SBUNH6CJ63GN3FJR2Q, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:13.255968741 (550h40m13.127051806s) [noise=true , address=145.79.227.30:4789 , piece_id=4QT5IUPIK7BOHJROEK2EBFTW434OKC75FZNENMYYCRMBZ7E5BUTA, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:13.255996102 (550h40m13.127079137s) [seg_pos={0 0}, upload=482805]: Segment reached optimal threshold of 14 pieces.
Dec 28 14:09:13.256214589 (550h40m13.127297634s) [noise=true , address=145.79.227.40:4053 , piece_id=RO4ZZQCKCC7T4T6SOT7GGTY626BJ6ESLQG6KATTPNIUM3ICUKHAQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DqyTc4MHvxutFXs2WWRCk8MSuxF3Y1HayyQY4Y5sy8cqDLWuG): protocol: expected piece hash; context canceled; EOF
Dec 28 14:09:13.256223686 (550h40m13.127306731s) [noise=true , address=145.79.227.40:4053 , piece_id=RO4ZZQCKCC7T4T6SOT7GGTY626BJ6ESLQG6KATTPNIUM3ICUKHAQ, seg_pos={0 0}, upload=482805]: Long tail cancellation due to optimal threshold
Dec 28 14:09:13.256512506 (550h40m13.127595551s) [noise=true , address=77.90.31.15:4704 , piece_id=DXWLFOIKOPE33LMEQECUJVIGRG3LPADUMULLWI4ZBHEOQHD43ULQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14tJRdZ1guwwsYwPf3LtENRWqYx2jBQpi7uAMMPaB32Gks9zmj): protocol: expected piece hash; context canceled; EOF
Dec 28 14:09:13.256520060 (550h40m13.127603105s) [noise=true , address=77.90.31.15:4704 , piece_id=DXWLFOIKOPE33LMEQECUJVIGRG3LPADUMULLWI4ZBHEOQHD43ULQ, seg_pos={0 0}, upload=482805]: Long tail cancellation due to optimal threshold
Dec 28 14:09:13.256649106 (550h40m13.127732151s) [noise=true , address=77.90.31.14:4678 , piece_id=VBGPSGMNDQZYY7UKRINAV6B3FVI5BPDPYS37FQIJCMLL65F3XOEQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12beu3SUZMPNWfbeHV4tLtx9UVZoEHh4Kt7oBhT59w3bGBE1ovP): protocol: write tcp 188.166.225.170:57374->77.90.31.14:4678: use of closed network connection; write tcp 188.166.225.170:57374->77.90.31.14:4678: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57374->77.90.31.14:4678: use of closed network connection
Dec 28 14:09:13.256655819 (550h40m13.127738864s) [noise=true , address=77.90.31.14:4678 , piece_id=VBGPSGMNDQZYY7UKRINAV6B3FVI5BPDPYS37FQIJCMLL65F3XOEQ, seg_pos={0 0}, upload=482805]: Long tail cancellation due to optimal threshold
Dec 28 14:09:13.256752974 (550h40m13.127836009s) [noise=true , address=64.49.13.169:4710 , piece_id=SANL7TVJFES63XD46OKQL4KIJ6YVHSEWPZKMRISEKC3ZKWQBOMIA, seg_pos={0 0}, upload=482805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:35680->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:35680->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35680->64.49.13.169:4710: use of closed network connection
Dec 28 14:09:13.256758825 (550h40m13.12784186s) [noise=true , address=64.49.13.169:4710 , piece_id=SANL7TVJFES63XD46OKQL4KIJ6YVHSEWPZKMRISEKC3ZKWQBOMIA, seg_pos={0 0}, upload=482805]: Long tail cancellation due to optimal threshold
Dec 28 14:09:13.256776549 (550h40m13.127859594s) [seg_pos={0 0}, upload=482805]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 14:09:13.256856591 (550h40m13.127939626s) [upload=482805]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 14:09:13.256862432 (550h40m13.127945467s) [upload=482805]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 14:09:13.550363717 (550h40m13.421446782s) [upload=482805]: Done uploading segments...
Dec 28 14:09:13.550405126 (550h40m13.421488161s) [upload=482805]: Upload finished. err: <nil>
Duration: 1m0.002758823s
core
uplink
us-select-1
us1-placement-20
|