Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
53 out of 60 is failed (threshold=6) [XXX.XXXXXXXXXXXX.XXXXXXX.XX..XXXXXXXXXXXXXXXXXXXXX.X.XXXXXXX] duration:
5.002006087s trace:
https://jaeger.dev.storj.io/trace/44598cee0b4ca564
FAIL (2025-12-28 14:17:52 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.002041475s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [.........X....................X...X........X.............X..] duration:
17.908805333s trace:
https://jaeger.dev.storj.io/trace/35a5d4335e2cde85
FAIL (2025-12-28 14:17:51 UTC)
orbiter warning: context deadline exceeded
Duration: 17.908882249s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
21 out of 60 is failed (threshold=3) [.....X..X.X...XX......X......XXX.XX....XX......XX.XX.X.X..XX] duration:
30.002969404s trace:
https://jaeger.dev.storj.io/trace/30c2dbb6a657315
FAIL (2025-12-28 14:17:42 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
ments...
Dec 28 14:17:13.393515120 (550h48m13.264598155s) [upload=482923]: Got next segment. Inline: false
Dec 28 14:17:13.393566056 (550h48m13.264649101s) [upload=482923]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 14:17:13.393573550 (550h48m13.264656596s) [upload=482923]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 14:17:13.639277692 (550h48m13.510360737s) [seg_pos={0 0}, upload=482923]: Begin upload segment...
Dec 28 14:17:13.639338889 (550h48m13.510421924s) [seg_pos={0 0}, upload=482923]: Done begin upload segment.
Dec 28 14:17:13.639375168 (550h48m13.510458203s) [noise=true , address=145.79.227.8:4330 , piece_id=T66I4YVNXRZMIJIA3IYJM55VX2QFUWQKCSU3VK5BZBWR66GEAYPA, seg_pos={0 0}, upload=482923]: Uploading piece...
Dec 28 14:17:13.639392901 (550h48m13.510475946s) [noise=true , address=145.79.227.50:4375 , piece_id=FAJJPUQFK2RWVXEBQKSRM72IENELA2SVIQKFQOBA5D65IPPPKSYA, seg_pos={0 0}, upload=482923]: Uploading piece...
Dec 28 14:17:13.639826888 (550h48m13.510909933s) [noise=true , address=145.79.227.12:4765 , piece_id=YVKZYWIDXMX6OK73AHEJ7QMVV3I2QVN5GZZFGQDTODLKL5VUM5CA, seg_pos={0 0}, upload=482923]: Uploading piece...
Dec 28 14:17:13.640879352 (550h48m13.511962397s) [noise=true , address=145.79.227.48:4303 , piece_id=XFRDLB76H44IEMS7PZZKKAX6XH7HE2CLXAP2DQGYIIR7SW6457WQ, seg_pos={0 0}, upload=482923]: Uploading piece...
Dec 28 14:17:13.641329549 (550h48m13.512412584s) [noise=true , address=145.79.227.3:4150 , piece_id=BD3DCAWHIK33S4NKGNNHYQS7AITNANQWT6AC6NWREACDYGZXIZIQ, seg_pos={0 0}, upload=482923]: Uploading piece...
Dec 28 14:17:13.641762874 (550h48m13.512845919s) [noise=true , address=77.90.31.21:4878 , piece_id=LUNYHIFTK5RHGQUYFPKBF63GTYRBWD6YXAY7EX2HI62LQUF6FR7Q, seg_pos={0 0}, upload=482923]: Uploading piece...
Dec 28 14:17:13.643155316 (550h48m13.514238361s) [noise=true , address=145.79.227.38:4977 , piece_id=GSGTKGIDMAXMGAGHDWLNFXPV6BMVZN6JSIZLNQNBLAOOBY3XW3FA, seg_pos={0 0}, upload=482923]: Uploading piece...
Dec 28 14:17:13.643316433 (550h48m13.514399478s) [noise=true , address=145.79.227.36:4519 , piece_id=CROFFYUSC5XY66WIENHWOFQHHICY3C7OF7HTVAWPV2JG7EEDXR2A, seg_pos={0 0}, upload=482923]: Uploading piece...
Dec 28 14:17:13.644658267 (550h48m13.515741302s) [noise=true , address=145.79.227.52:4445 , piece_id=F7FPRICFWCMVPLTO4KPUE6EMZISWBV77OCF4YAWN6ZQVA3WHOJ2Q, seg_pos={0 0}, upload=482923]: Uploading piece...
Dec 28 14:17:13.645037149 (550h48m13.516120184s) [noise=true , address=145.79.227.51:4409 , piece_id=4ZKPLDJ4ELJCVVPESJGGT4H365AEM3M6DO4XQK5PDCN3GQCLVNUA, seg_pos={0 0}, upload=482923]: Uploading piece...
Dec 28 14:17:13.646306957 (550h48m13.517389992s) [noise=true , address=45.91.116.6:4975 , piece_id=JEFDFQJ2EB3DMV6ZL7YZUNV3EPDX5TG4TPM55OIEJOMYTW2P35QA, seg_pos={0 0}, upload=482923]: Uploading piece...
Dec 28 14:17:13.647001009 (550h48m13.518084044s) [noise=true , address=45.91.116.9:4262 , piece_id=5O3AAP2JGSJ3EMCKXMYPYLZOEK3GI7NWGRUETRYGITAQ6R6EAHGQ, seg_pos={0 0}, upload=482923]: Uploading piece...
Dec 28 14:17:13.648779195 (550h48m13.51986224s) [noise=true , address=77.90.31.17:4770 , piece_id=6T56M4XZ3FQ2YQAXX52QVPSHDEZPRONZ5RJYHSVRJG4YFHXE4X4A, seg_pos={0 0}, upload=482923]: Uploading piece...
Dec 28 14:17:13.650291644 (550h48m13.521374689s) [noise=true , address=145.79.227.2:4114 , piece_id=TCXOQRFG7N36UTV7XYDIM5NWW6TMHT57BKDRFX74JWPNYUDNKJAQ, seg_pos={0 0}, upload=482923]: Uploading piece...
Dec 28 14:17:13.651926387 (550h48m13.523009432s) [noise=true , address=145.79.227.15:4874 , piece_id=CX3YZNCGM3XIF22TWAF6J5GH3PZZIAMDTLTHQ5PDAFDJXHF7XPAQ, seg_pos={0 0}, upload=482923]: Uploading piece...
Dec 28 14:17:13.652697646 (550h48m13.523780691s) [noise=true , address=145.79.227.40:4050 , piece_id=BWYEEXQWUXZR6BLEWW2735ZAUZ542RWCTRONSVGJQ3MYQCBPBODA, seg_pos={0 0}, upload=482923]: Uploading piece...
Dec 28 14:17:15.266871601 (550h48m15.137954636s) [upload=482923]: Next returned nil segment
Dec 28 14:17:15.266886259 (550h48m15.137969304s) [upload=482923]: Waiting for error group managing segments...
Dec 28 14:17:15.644608575 (550h48m15.51569164s) [noise=true , address=145.79.227.48:4303 , piece_id=XFRDLB76H44IEMS7PZZKKAX6XH7HE2CLXAP2DQGYIIR7SW6457WQ, seg_pos={0 0}, upload=482923]: Done uploading piece. err: <nil>
Dec 28 14:17:15.648960161 (550h48m15.520043206s) [noise=true , address=145.79.227.52:4445 , piece_id=F7FPRICFWCMVPLTO4KPUE6EMZISWBV77OCF4YAWN6ZQVA3WHOJ2Q, seg_pos={0 0}, upload=482923]: Done uploading piece. err: <nil>
Dec 28 14:17:15.654324596 (550h48m15.525407641s) [noise=true , address=145.79.227.12:4765 , piece_id=YVKZYWIDXMX6OK73AHEJ7QMVV3I2QVN5GZZFGQDTODLKL5VUM5CA, seg_pos={0 0}, upload=482923]: Done uploading piece. err: <nil>
Dec 28 14:17:15.654740367 (550h48m15.525823402s) [noise=true , address=145.79.227.38:4977 , piece_id=GSGTKGIDMAXMGAGHDWLNFXPV6BMVZN6JSIZLNQNBLAOOBY3XW3FA, seg_pos={0 0}, upload=482923]: Done uploading piece. err: <nil>
Dec 28 14:17:15.655797159 (550h48m15.526880204s) [noise=true , address=145.79.227.36:4519 , piece_id=CROFFYUSC5XY66WIENHWOFQHHICY3C7OF7HTVAWPV2JG7EEDXR2A, seg_pos={0 0}, upload=482923]: Done uploading piece. err: <nil>
Dec 28 14:17:15.656037197 (550h48m15.527120232s) [noise=true , address=145.79.227.15:4874 , piece_id=CX3YZNCGM3XIF22TWAF6J5GH3PZZIAMDTLTHQ5PDAFDJXHF7XPAQ, seg_pos={0 0}, upload=482923]: Done uploading piece. err: <nil>
Dec 28 14:17:15.659493428 (550h48m15.530576463s) [noise=true , address=145.79.227.8:4330 , piece_id=T66I4YVNXRZMIJIA3IYJM55VX2QFUWQKCSU3VK5BZBWR66GEAYPA, seg_pos={0 0}, upload=482923]: Done uploading piece. err: <nil>
Dec 28 14:17:15.664005630 (550h48m15.535088675s) [noise=true , address=145.79.227.51:4409 , piece_id=4ZKPLDJ4ELJCVVPESJGGT4H365AEM3M6DO4XQK5PDCN3GQCLVNUA, seg_pos={0 0}, upload=482923]: Done uploading piece. err: <nil>
Dec 28 14:17:15.669418638 (550h48m15.540501683s) [noise=true , address=145.79.227.50:4375 , piece_id=FAJJPUQFK2RWVXEBQKSRM72IENELA2SVIQKFQOBA5D65IPPPKSYA, seg_pos={0 0}, upload=482923]: Done uploading piece. err: <nil>
Dec 28 14:17:15.673674622 (550h48m15.544757687s) [noise=true , address=145.79.227.2:4114 , piece_id=TCXOQRFG7N36UTV7XYDIM5NWW6TMHT57BKDRFX74JWPNYUDNKJAQ, seg_pos={0 0}, upload=482923]: Done uploading piece. err: <nil>
Dec 28 14:17:15.675925087 (550h48m15.547008122s) [noise=true , address=145.79.227.3:4150 , piece_id=BD3DCAWHIK33S4NKGNNHYQS7AITNANQWT6AC6NWREACDYGZXIZIQ, seg_pos={0 0}, upload=482923]: Done uploading piece. err: <nil>
Dec 28 14:17:15.820606514 (550h48m15.691689549s) [noise=true , address=77.90.31.21:4878 , piece_id=LUNYHIFTK5RHGQUYFPKBF63GTYRBWD6YXAY7EX2HI62LQUF6FR7Q, seg_pos={0 0}, upload=482923]: Done uploading piece. err: <nil>
Dec 28 14:17:15.884303534 (550h48m15.755386569s) [noise=true , address=45.91.116.9:4262 , piece_id=5O3AAP2JGSJ3EMCKXMYPYLZOEK3GI7NWGRUETRYGITAQ6R6EAHGQ, seg_pos={0 0}, upload=482923]: Done uploading piece. err: <nil>
Dec 28 14:17:15.884325366 (550h48m15.755408401s) [seg_pos={0 0}, upload=482923]: Segment reached optimal threshold of 13 pieces.
Dec 28 14:17:15.884559230 (550h48m15.755642265s) [noise=true , address=77.90.31.17:4770 , piece_id=6T56M4XZ3FQ2YQAXX52QVPSHDEZPRONZ5RJYHSVRJG4YFHXE4X4A, seg_pos={0 0}, upload=482923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: write tcp 188.166.225.170:45054->77.90.31.17:4770: use of closed network connection; write tcp 188.166.225.170:45054->77.90.31.17:4770: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45054->77.90.31.17:4770: use of closed network connection
Dec 28 14:17:15.884565754 (550h48m15.755648789s) [noise=true , address=77.90.31.17:4770 , piece_id=6T56M4XZ3FQ2YQAXX52QVPSHDEZPRONZ5RJYHSVRJG4YFHXE4X4A, seg_pos={0 0}, upload=482923]: Long tail cancellation due to optimal threshold
Dec 28 14:17:15.884760343 (550h48m15.755843389s) [noise=true , address=145.79.227.40:4050 , piece_id=BWYEEXQWUXZR6BLEWW2735ZAUZ542RWCTRONSVGJQ3MYQCBPBODA, seg_pos={0 0}, upload=482923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF
Dec 28 14:17:15.884775894 (550h48m15.755858939s) [noise=true , address=145.79.227.40:4050 , piece_id=BWYEEXQWUXZR6BLEWW2735ZAUZ542RWCTRONSVGJQ3MYQCBPBODA, seg_pos={0 0}, upload=482923]: Long tail cancellation due to optimal threshold
Dec 28 14:17:15.884940687 (550h48m15.756023732s) [noise=true , address=45.91.116.6:4975 , piece_id=JEFDFQJ2EB3DMV6ZL7YZUNV3EPDX5TG4TPM55OIEJOMYTW2P35QA, seg_pos={0 0}, upload=482923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: write tcp 188.166.225.170:43900->45.91.116.6:4975: use of closed network connection; write tcp 188.166.225.170:43900->45.91.116.6:4975: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43900->45.91.116.6:4975: use of closed network connection
Dec 28 14:17:15.884953231 (550h48m15.756036276s) [noise=true , address=45.91.116.6:4975 , piece_id=JEFDFQJ2EB3DMV6ZL7YZUNV3EPDX5TG4TPM55OIEJOMYTW2P35QA, seg_pos={0 0}, upload=482923]: Long tail cancellation due to optimal threshold
Dec 28 14:17:15.884973018 (550h48m15.756056053s) [seg_pos={0 0}, upload=482923]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 14:17:15.885039024 (550h48m15.756122069s) [upload=482923]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 14:17:15.885043823 (550h48m15.756126868s) [upload=482923]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 14:17:16.180095501 (550h48m16.051178546s) [upload=482923]: Done uploading segments...
Dec 28 14:17:16.180128083 (550h48m16.051211118s) [upload=482923]: Upload finished. err: <nil>
Duration: 30.002990414s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [........X...................................................] duration:
10.988507058s trace:
https://jaeger.dev.storj.io/trace/5fe9d6dbb79fbe44
FAIL (2025-12-28 14:17:24 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
67.204.205:4027 , piece_id=PAPNMRJ7E47VOMOXPBBDWKZJHLRMXTWVR5VW3J7KNQSSMHOHWOAQ, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.967737953 (550h8m4.838820988s) [noise=true , address=185.167.204.206:4072 , piece_id=WXORBDMA3KW7WZ67A4FJT6URJRFADUJ7X2PAHIGNG47T3TGMXVTQ, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.968193240 (550h8m4.839276285s) [noise=true , address=185.167.204.205:4030 , piece_id=R5DLI4ZTLOVGO56GPLJL5ZNHYSXOSYNJQHFIO4MVVNHWWBMEOV2Q, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.968780638 (550h8m4.839863683s) [noise=true , address=89.149.201.147:4019 , piece_id=MK7ORCLHO3DOWIOVSOWIQMPJ447PHRQGXP3UKITIRTYJSU5MPCCQ, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.969308924 (550h8m4.840391959s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.970088349 (550h8m4.841171404s) [noise=true , address=81.171.14.70:4039 , piece_id=EDHPQWLQZTP7MKPGEVMP7JXNTW7LFWSXCTRTSXPCBIQPROX76WRA, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.970325381 (550h8m4.841408426s) [noise=true , address=81.171.14.70:4036 , piece_id=CICNNV7JE6TBNC4RAPXKCEUSJAADOUVGSVIVQISM3ZB4COAZQDDA, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.970783813 (550h8m4.841866858s) [noise=true , address=81.171.14.70:4044 , piece_id=DGN74IG6IWAHLEMO6NLL4TJBXZ6ONTN6QUB6HXY6K7SSO3LYOYIQ, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.971205406 (550h8m4.842288441s) [noise=true , address=81.171.14.70:4048 , piece_id=TLT47ORGUKGF3MIIEAX7D22VR2FO6W6NL3VFJNHKWD4L3LIMFDLA, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.971606299 (550h8m4.842689344s) [noise=true , address=185.167.204.206:4067 , piece_id=VSKTG5RH32N7OEVWJGQVB6WJ72TEVF6DDK25W7NOH4RAXUP52LTQ, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.972162538 (550h8m4.843245583s) [noise=true , address=89.149.201.147:4002 , piece_id=G2PABNTDWCK4MKOZF6PONBMRNIXI45JTHJMAZFPEKPEJ3PQ7MTBA, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:05.795167814 (550h8m5.666250849s) [upload=482337]: Next returned nil segment
Dec 28 13:37:05.795184566 (550h8m5.666267611s) [upload=482337]: Waiting for error group managing segments...
Dec 28 13:37:06.178573801 (550h8m6.049656836s) [noise=true , address=185.167.204.206:4049 , piece_id=6RG2DZV6FPTIRE63ZNNST42TSWEBQ456YXIPPVYHIXV55GWFVJ4Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.183060835 (550h8m6.05414391s) [noise=true , address=185.167.204.206:4072 , piece_id=WXORBDMA3KW7WZ67A4FJT6URJRFADUJ7X2PAHIGNG47T3TGMXVTQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.183411202 (550h8m6.054494237s) [noise=true , address=185.167.204.206:4064 , piece_id=OG3G74OU23CEMKA54WQE3DOJ6AKAXBLZ23QWOQC4UEGQQN4JSMXQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.183791146 (550h8m6.054874181s) [noise=true , address=185.167.204.205:4029 , piece_id=6JQB6ZSCQZKWH3JWCQ4DY6BBAX73KC4Y53PRSMEMLWURZ5QWWQQQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.183973743 (550h8m6.055056778s) [noise=true , address=185.167.204.206:4053 , piece_id=YDH77S6JLKJ6IYS6DRUCELP5EJHIC2O44QBO7MIOEUYBML4GIBHA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.184465369 (550h8m6.055548414s) [noise=true , address=185.167.204.205:4008 , piece_id=RTUZPEXBB6JNSMC5YOFQQGDM57YD233W4756ZBQVMFNOVX6YKB4A, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.184501989 (550h8m6.055585024s) [noise=true , address=185.167.204.205:4030 , piece_id=R5DLI4ZTLOVGO56GPLJL5ZNHYSXOSYNJQHFIO4MVVNHWWBMEOV2Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.184856224 (550h8m6.055939259s) [noise=true , address=185.167.204.205:4009 , piece_id=V6RA7Y7FKY4DCPTRRF3BYNBALWDY5VTPXMDUXFE3353KUXLSQJEA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.185063518 (550h8m6.056146563s) [noise=true , address=185.167.204.205:4016 , piece_id=JB7SC3J2VR4U33HNQS2A2CA75THT3XLQCOIFHAJJEK7MQGY2SYAQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.185288617 (550h8m6.056371652s) [noise=true , address=185.167.204.205:4027 , piece_id=PAPNMRJ7E47VOMOXPBBDWKZJHLRMXTWVR5VW3J7KNQSSMHOHWOAQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.186241401 (550h8m6.057324436s) [noise=true , address=185.167.204.205:4024 , piece_id=FM5IMWRSJJ4RY72IX2OOYC54BFKL4FWE3UOAN3GJ26MLH345COVA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.193113658 (550h8m6.064196703s) [noise=true , address=185.167.204.205:4005 , piece_id=R4DFQJZBPJ5WJWHPODV6ILHUDIRBUSAFOMVNCLXB4UNMI6EXTKHA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.193791408 (550h8m6.064874453s) [noise=true , address=185.167.204.206:4068 , piece_id=DAI37M67FWCSVQTQOZ6RSSOJDGUVZO6SNGAHJJ44G3PGS4OJZEMA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.194292262 (550h8m6.065375307s) [noise=true , address=185.167.204.206:4046 , piece_id=VWLVHMBMJ63DQEWYBIKH3Y3NTVWFNW5OXZKMYXXBHVI3VWYXLLPQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.194302652 (550h8m6.065385687s) [noise=true , address=185.167.204.206:4067 , piece_id=VSKTG5RH32N7OEVWJGQVB6WJ72TEVF6DDK25W7NOH4RAXUP52LTQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.195359193 (550h8m6.066442228s) [noise=true , address=185.167.204.205:4012 , piece_id=LV6RUA5BWXG4DMSUVZDBGUAF5IBWUGMLJYUKNIXHRVDEBIOW4Q2Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.198423448 (550h8m6.069506493s) [noise=true , address=185.167.204.206:4070 , piece_id=5A7YYQYREIZRZDR7TRSE7XFW765JVAV7FX6LKWUI2OXAA74XN4CA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.888968408 (550h8m6.760051453s) [noise=true , address=81.171.14.70:4044 , piece_id=DGN74IG6IWAHLEMO6NLL4TJBXZ6ONTN6QUB6HXY6K7SSO3LYOYIQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.892350789 (550h8m6.763433834s) [noise=true , address=81.171.14.70:4039 , piece_id=EDHPQWLQZTP7MKPGEVMP7JXNTW7LFWSXCTRTSXPCBIQPROX76WRA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.904477160 (550h8m6.775560195s) [noise=true , address=81.171.14.70:4040 , piece_id=DYF4GSZBAKQ3H63LTVCMEFWMABY3FXHOJCNVIFYRKLF2J6B52K5A, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.128088281 (550h8m6.999171326s) [noise=true , address=81.171.14.70:4036 , piece_id=CICNNV7JE6TBNC4RAPXKCEUSJAADOUVGSVIVQISM3ZB4COAZQDDA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.430242142 (550h8m7.301325177s) [noise=true , address=89.149.201.147:4009 , piece_id=EVW2NQFYU57PC2IIVXLYIEBMYXJNIC2CLCHW6PS6NYIPCFRUVHDA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.505304332 (550h8m7.376387367s) [noise=true , address=89.149.201.147:4019 , piece_id=MK7ORCLHO3DOWIOVSOWIQMPJ447PHRQGXP3UKITIRTYJSU5MPCCQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.575461543 (550h8m7.446544609s) [noise=true , address=81.171.14.70:4048 , piece_id=TLT47ORGUKGF3MIIEAX7D22VR2FO6W6NL3VFJNHKWD4L3LIMFDLA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.599667265 (550h8m7.4707503s) [noise=true , address=89.149.201.147:4023 , piece_id=5IINX2GVRFLRX2QH24OO2DDANOKOTXTDR6AGVJBGSJ7IAZQ3CASA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.928390728 (550h8m7.799473773s) [noise=true , address=89.149.201.147:4002 , piece_id=G2PABNTDWCK4MKOZF6PONBMRNIXI45JTHJMAZFPEKPEJ3PQ7MTBA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.928417148 (550h8m7.799500193s) [seg_pos={0 0}, upload=482337]: Segment reached optimal threshold of 26 pieces.
Dec 28 13:37:07.928656303 (550h8m7.799739348s) [noise=true , address=81.171.14.70:4030 , piece_id=74YDIKUST7X3JBXVQHVLFHIKH5A4DYH5TNL3I4XUA5A6AHQO2DRA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZoRwa8wLw66Pjumy1ybp6wrZ79RrSU55uYZpwaY3xi95iSYde): protocol: expected piece hash; context canceled; EOF
Dec 28 13:37:07.928664990 (550h8m7.799748035s) [noise=true , address=81.171.14.70:4030 , piece_id=74YDIKUST7X3JBXVQHVLFHIKH5A4DYH5TNL3I4XUA5A6AHQO2DRA, seg_pos={0 0}, upload=482337]: Long tail cancellation due to optimal threshold
Dec 28 13:37:07.928723351 (550h8m7.799806386s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wgtphHKaTejC1qrJ98sDysBwtj2Zeqw2c5UFk1HMqfY8Buvr): protocol: expected piece hash; context canceled; EOF
Dec 28 13:37:07.928733922 (550h8m7.799816957s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Long tail cancellation due to optimal threshold
Dec 28 13:37:07.928754921 (550h8m7.799837966s) [seg_pos={0 0}, upload=482337]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 28 13:37:07.928870170 (550h8m7.799953215s) [upload=482337]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 13:37:07.928878627 (550h8m7.799961672s) [upload=482337]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 13:37:08.166321536 (550h8m8.037404571s) [upload=482337]: Done uploading segments...
Dec 28 13:37:08.166341444 (550h8m8.037424479s) [upload=482337]: Upload finished. err: <nil>
Duration: 10.988527335s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [X....................................X........X...X.........] duration:
33.927382104s trace:
https://jaeger.dev.storj.io/trace/36d5af3c761e71b1
FAIL (2025-12-28 14:17:22 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
7.49:4360 , piece_id=QDT5N5LDL2HCVXPOOSXWD2TOEZDF2JZWFLDFYSIMN647FI3ZORUQ, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.372193736 (550h40m10.243276781s) [noise=true , address=77.90.31.12:4618 , piece_id=45WRZ26CPUODOAYF5B6RF37DNMDDQWWJUDK7BFKYS3K3LOCNYJCA, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.372518505 (550h40m10.24360155s) [noise=true , address=77.90.31.13:4646 , piece_id=5CDMQF7GG5PDPHIPMFZWDBKWH5ANO3UIX257LJPCWRITNOA7GY4A, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.372848052 (550h40m10.243931097s) [noise=true , address=64.49.13.170:4721 , piece_id=U76TI7IPAHPZAS6XLG66E4L2CGCYH43NH4ZHTYXMXKZR6SGE7SKQ, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.373163072 (550h40m10.244246117s) [noise=true , address=77.90.31.11:4565 , piece_id=4SBLRZK3K4ED73PJ64BI6ANKBWB2AAHQA4SBUNH6CJ63GN3FJR2Q, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.373445089 (550h40m10.244528134s) [noise=true , address=145.79.227.34:4911 , piece_id=24WE4EMZSL2JHQZ7PESCMLYCZOWGDTGJD2XNLUUJEFPSYLRI5E7A, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.373768855 (550h40m10.24485189s) [noise=true , address=77.90.31.18:4819 , piece_id=OYXPAN3EK4QRDEC6VHIDL3GSGEY27G72KII6HTGUMSPYLHCYBNEA, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.374285519 (550h40m10.245368564s) [noise=true , address=77.90.31.14:4678 , piece_id=VBGPSGMNDQZYY7UKRINAV6B3FVI5BPDPYS37FQIJCMLL65F3XOEQ, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.374710689 (550h40m10.245793724s) [noise=true , address=145.79.227.40:4053 , piece_id=RO4ZZQCKCC7T4T6SOT7GGTY626BJ6ESLQG6KATTPNIUM3ICUKHAQ, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.375099329 (550h40m10.246182364s) [noise=true , address=77.90.31.19:4859 , piece_id=A4HD3FPDPCCMHRFBTS3GNGB323RYD5C5AW4ZVSNOTXFI7FLIV37Q, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.376073433 (550h40m10.247156468s) [noise=true , address=77.90.31.15:4704 , piece_id=DXWLFOIKOPE33LMEQECUJVIGRG3LPADUMULLWI4ZBHEOQHD43ULQ, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.376542387 (550h40m10.247625422s) [noise=true , address=145.79.227.44:4221 , piece_id=H72XCWLKYJAXYGGDIWN67H42RUIEOJCPUSWT5IJMFPKETH7LK3KA, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.377379761 (550h40m10.248462796s) [noise=true , address=145.79.227.32:4863 , piece_id=27TOXK44RIF3DV3PDY4RBP6HEFRQJE6N7LNLRJKSU52YCB6A3JKA, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.378022765 (550h40m10.24910581s) [noise=true , address=145.79.227.30:4789 , piece_id=4QT5IUPIK7BOHJROEK2EBFTW434OKC75FZNENMYYCRMBZ7E5BUTA, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.378821687 (550h40m10.249904722s) [noise=true , address=145.79.227.2:4119 , piece_id=EQM3WMBQ7GXRWRBZK4ZWCI23K4LSLXW6QPEABHGMWRHEYUYANKSQ, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:11.676223695 (550h40m11.54730674s) [upload=482805]: Next returned nil segment
Dec 28 14:09:11.676238083 (550h40m11.547321128s) [upload=482805]: Waiting for error group managing segments...
Dec 28 14:09:12.006762275 (550h40m11.87784531s) [noise=true , address=64.49.13.170:4721 , piece_id=U76TI7IPAHPZAS6XLG66E4L2CGCYH43NH4ZHTYXMXKZR6SGE7SKQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.250307967 (550h40m12.121391012s) [noise=true , address=145.79.227.2:4119 , piece_id=EQM3WMBQ7GXRWRBZK4ZWCI23K4LSLXW6QPEABHGMWRHEYUYANKSQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.269380734 (550h40m12.140463769s) [noise=true , address=145.79.227.49:4360 , piece_id=QDT5N5LDL2HCVXPOOSXWD2TOEZDF2JZWFLDFYSIMN647FI3ZORUQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.290818335 (550h40m12.16190138s) [noise=true , address=145.79.227.32:4863 , piece_id=27TOXK44RIF3DV3PDY4RBP6HEFRQJE6N7LNLRJKSU52YCB6A3JKA, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.467679327 (550h40m12.338762362s) [noise=true , address=45.91.116.8:4072 , piece_id=GEVTX6LLENPLN3UQLSGTFTP3DO52IRYUIQFIWC4XMOC3ZF6V4GTQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.486615595 (550h40m12.35769863s) [noise=true , address=77.90.31.12:4618 , piece_id=45WRZ26CPUODOAYF5B6RF37DNMDDQWWJUDK7BFKYS3K3LOCNYJCA, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.566542810 (550h40m12.437625845s) [noise=true , address=77.90.31.19:4859 , piece_id=A4HD3FPDPCCMHRFBTS3GNGB323RYD5C5AW4ZVSNOTXFI7FLIV37Q, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.609337578 (550h40m12.480420623s) [noise=true , address=77.90.31.13:4646 , piece_id=5CDMQF7GG5PDPHIPMFZWDBKWH5ANO3UIX257LJPCWRITNOA7GY4A, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.640856565 (550h40m12.511939611s) [noise=true , address=45.91.116.5:4322 , piece_id=SS6DOCSJHY5Y3DPWMYX45F4VYYXR44JDLYQ2YPY5XIIAYKMAG7LQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.645490329 (550h40m12.516573364s) [noise=true , address=77.90.31.18:4819 , piece_id=OYXPAN3EK4QRDEC6VHIDL3GSGEY27G72KII6HTGUMSPYLHCYBNEA, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:13.103710286 (550h40m12.974793331s) [noise=true , address=145.79.227.34:4911 , piece_id=24WE4EMZSL2JHQZ7PESCMLYCZOWGDTGJD2XNLUUJEFPSYLRI5E7A, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:13.209917141 (550h40m13.081000176s) [noise=true , address=145.79.227.44:4221 , piece_id=H72XCWLKYJAXYGGDIWN67H42RUIEOJCPUSWT5IJMFPKETH7LK3KA, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:13.218355468 (550h40m13.089438514s) [noise=true , address=77.90.31.11:4565 , piece_id=4SBLRZK3K4ED73PJ64BI6ANKBWB2AAHQA4SBUNH6CJ63GN3FJR2Q, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:13.255968741 (550h40m13.127051806s) [noise=true , address=145.79.227.30:4789 , piece_id=4QT5IUPIK7BOHJROEK2EBFTW434OKC75FZNENMYYCRMBZ7E5BUTA, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:13.255996102 (550h40m13.127079137s) [seg_pos={0 0}, upload=482805]: Segment reached optimal threshold of 14 pieces.
Dec 28 14:09:13.256214589 (550h40m13.127297634s) [noise=true , address=145.79.227.40:4053 , piece_id=RO4ZZQCKCC7T4T6SOT7GGTY626BJ6ESLQG6KATTPNIUM3ICUKHAQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DqyTc4MHvxutFXs2WWRCk8MSuxF3Y1HayyQY4Y5sy8cqDLWuG): protocol: expected piece hash; context canceled; EOF
Dec 28 14:09:13.256223686 (550h40m13.127306731s) [noise=true , address=145.79.227.40:4053 , piece_id=RO4ZZQCKCC7T4T6SOT7GGTY626BJ6ESLQG6KATTPNIUM3ICUKHAQ, seg_pos={0 0}, upload=482805]: Long tail cancellation due to optimal threshold
Dec 28 14:09:13.256512506 (550h40m13.127595551s) [noise=true , address=77.90.31.15:4704 , piece_id=DXWLFOIKOPE33LMEQECUJVIGRG3LPADUMULLWI4ZBHEOQHD43ULQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14tJRdZ1guwwsYwPf3LtENRWqYx2jBQpi7uAMMPaB32Gks9zmj): protocol: expected piece hash; context canceled; EOF
Dec 28 14:09:13.256520060 (550h40m13.127603105s) [noise=true , address=77.90.31.15:4704 , piece_id=DXWLFOIKOPE33LMEQECUJVIGRG3LPADUMULLWI4ZBHEOQHD43ULQ, seg_pos={0 0}, upload=482805]: Long tail cancellation due to optimal threshold
Dec 28 14:09:13.256649106 (550h40m13.127732151s) [noise=true , address=77.90.31.14:4678 , piece_id=VBGPSGMNDQZYY7UKRINAV6B3FVI5BPDPYS37FQIJCMLL65F3XOEQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12beu3SUZMPNWfbeHV4tLtx9UVZoEHh4Kt7oBhT59w3bGBE1ovP): protocol: write tcp 188.166.225.170:57374->77.90.31.14:4678: use of closed network connection; write tcp 188.166.225.170:57374->77.90.31.14:4678: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57374->77.90.31.14:4678: use of closed network connection
Dec 28 14:09:13.256655819 (550h40m13.127738864s) [noise=true , address=77.90.31.14:4678 , piece_id=VBGPSGMNDQZYY7UKRINAV6B3FVI5BPDPYS37FQIJCMLL65F3XOEQ, seg_pos={0 0}, upload=482805]: Long tail cancellation due to optimal threshold
Dec 28 14:09:13.256752974 (550h40m13.127836009s) [noise=true , address=64.49.13.169:4710 , piece_id=SANL7TVJFES63XD46OKQL4KIJ6YVHSEWPZKMRISEKC3ZKWQBOMIA, seg_pos={0 0}, upload=482805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:35680->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:35680->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35680->64.49.13.169:4710: use of closed network connection
Dec 28 14:09:13.256758825 (550h40m13.12784186s) [noise=true , address=64.49.13.169:4710 , piece_id=SANL7TVJFES63XD46OKQL4KIJ6YVHSEWPZKMRISEKC3ZKWQBOMIA, seg_pos={0 0}, upload=482805]: Long tail cancellation due to optimal threshold
Dec 28 14:09:13.256776549 (550h40m13.127859594s) [seg_pos={0 0}, upload=482805]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 14:09:13.256856591 (550h40m13.127939626s) [upload=482805]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 14:09:13.256862432 (550h40m13.127945467s) [upload=482805]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 14:09:13.550363717 (550h40m13.421446782s) [upload=482805]: Done uploading segments...
Dec 28 14:09:13.550405126 (550h40m13.421488161s) [upload=482805]: Upload finished. err: <nil>
Duration: 33.927408684s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
53 out of 60 is failed (threshold=6) [XXXX.XXXXXXXXXXXX.XXXXXXX.XX..XXXXXXXXXXXXXXXXXXXXX.X.XXXXXX] duration:
5.000976658s trace:
https://jaeger.dev.storj.io/trace/193a4146a793e859
FAIL (2025-12-28 14:17:21 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001000464s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
21 out of 60 is failed (threshold=3) [X.....X..X.X...XX......X......XXX.XX....XX......XX.XX.X.X..X] duration:
30.003398658s trace:
https://jaeger.dev.storj.io/trace/3ccfda23ca2b15ae
FAIL (2025-12-28 14:17:12 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: JOGZR3JAVHJ5KSXG524QVRNKGZOQN2TJJGQA6EC7UP67YWW67PYA) context deadline exceeded; piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: QKCA26Z3VAQ3SKTFZP4RTNTE26ODUCRT5UOSN73DPHHAXHWN35DA) context deadline exceeded
======LOGS======
Dec 28 14:16:43.169067837 (550h47m43.040150872s) [upload=482915]: Starting upload
Dec 28 14:16:43.169121118 (550h47m43.040204153s) [upload=482915]: Done starting upload
Dec 28 14:16:43.169223152 (550h47m43.040306187s) [upload=482915]: Uploading segments...
Dec 28 14:16:43.169360514 (550h47m43.040443549s) [upload=482915]: Got next segment. Inline: false
Dec 28 14:16:43.169397394 (550h47m43.040480439s) [upload=482915]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 14:16:43.169402724 (550h47m43.040485759s) [upload=482915]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 14:16:43.417091676 (550h47m43.288174721s) [seg_pos={0 0}, upload=482915]: Begin upload segment...
Dec 28 14:16:43.417155969 (550h47m43.288239004s) [seg_pos={0 0}, upload=482915]: Done begin upload segment.
Dec 28 14:16:43.417183892 (550h47m43.288266937s) [noise=true , address=45.91.116.9:4261 , piece_id=JOGZR3JAVHJ5KSXG524QVRNKGZOQN2TJJGQA6EC7UP67YWW67PYA, seg_pos={0 0}, upload=482915]: Uploading piece...
Dec 28 14:16:43.417255468 (550h47m43.288338503s) [noise=true , address=145.79.227.51:4410 , piece_id=D54CNAJW42WNUOVADLLG4CXSAAWQJVU5QQY5L5RYIPV7UW5CCDNQ, seg_pos={0 0}, upload=482915]: Uploading piece...
Dec 28 14:16:43.417757444 (550h47m43.288840479s) [noise=true , address=145.79.227.32:4836 , piece_id=KU3KMONL7TRKK3RD6FDSC4IJGQUQRON5LRKHPMS7ITTCZZ36RWEQ, seg_pos={0 0}, upload=482915]: Uploading piece...
Dec 28 14:16:43.417920585 (550h47m43.28900362s) [noise=true , address=145.79.227.52:4446 , piece_id=7LMULWAFSMMFNMAI3SY2FGPC4LKXHICDHPDJ4PDSNFS2CVVIHIHQ, seg_pos={0 0}, upload=482915]: Uploading piece...
Dec 28 14:16:43.418375671 (550h47m43.289458706s) [noise=true , address=145.79.227.4:4186 , piece_id=F2QAT7WBDPU4BVVOVLFCFZJP4YYEZL4X23UCXSLMTZ63PMABWPKA, seg_pos={0 0}, upload=482915]: Uploading piece...
Dec 28 14:16:43.418859492 (550h47m43.289942537s) [noise=true , address=77.90.31.12:4589 , piece_id=QKCA26Z3VAQ3SKTFZP4RTNTE26ODUCRT5UOSN73DPHHAXHWN35DA, seg_pos={0 0}, upload=482915]: Uploading piece...
Dec 28 14:16:43.419266217 (550h47m43.290349252s) [noise=true , address=145.79.227.12:4765 , piece_id=REZVMN72Y37UCHHOATYR3SGKYZE7ZT5IETPNIGGJZ6YXBZARLTYQ, seg_pos={0 0}, upload=482915]: Uploading piece...
Dec 28 14:16:43.419510412 (550h47m43.290593447s) [noise=true , address=77.90.31.16:4736 , piece_id=HNPPZ7EZ36NIOVKU7WBDPKQFTC4UFGMOZR567IHMGIGUXJ7G5CQA, seg_pos={0 0}, upload=482915]: Uploading piece...
Dec 28 14:16:43.420103612 (550h47m43.291186647s) [noise=true , address=145.79.227.2:4115 , piece_id=M4HN5C5BNM5FHCKVKVDRYCSTHCKULR2WAQBQ5GT3MNAWMYW2LWBA, seg_pos={0 0}, upload=482915]: Uploading piece...
Dec 28 14:16:43.420528060 (550h47m43.291611095s) [noise=true , address=145.79.227.14:4839 , piece_id=WPULRJZEINPHM3DIM2QKAAZFW2ILLTATMSPUY6T4OF66YZE6SI5A, seg_pos={0 0}, upload=482915]: Uploading piece...
Dec 28 14:16:43.420919675 (550h47m43.29200272s) [noise=true , address=145.79.227.25:4618 , piece_id=MFO22WITIXSLXUMZH3HYKQ5MSRC63SANUULAVZQKPARMOOQRRA6Q, seg_pos={0 0}, upload=482915]: Uploading piece...
Dec 28 14:16:43.421175221 (550h47m43.292258256s) [noise=true , address=45.91.116.12:4228 , piece_id=73Z626ZAGADQJZF5R25PESOPUDS5GACL6ITK5ANDIX3X3IGL5SWQ, seg_pos={0 0}, upload=482915]: Uploading piece...
Dec 28 14:16:43.421749996 (550h47m43.292833031s) [noise=true , address=45.91.116.7:4010 , piece_id=WPUSCFE6P5B7VO2SFTA5MUWLHHTZWICL22Y6NDXPFR3O3AK3Q3CQ, seg_pos={0 0}, upload=482915]: Uploading piece...
Dec 28 14:16:43.422020832 (550h47m43.293103867s) [noise=true , address=145.79.227.41:4086 , piece_id=LM7JQTVB4UBXDPKSRGQHMPCHVIFTBBZMZBYF6SCCHVR324FA6DWA, seg_pos={0 0}, upload=482915]: Uploading piece...
Dec 28 14:16:43.422937217 (550h47m43.294020262s) [noise=true , address=145.79.227.50:4373 , piece_id=MSPPIR5CJB2RDUO6AME3GZXZIRUSGAOVVPQI6FKAHQBNX52NZEOQ, seg_pos={0 0}, upload=482915]: Uploading piece...
Dec 28 14:16:43.423361765 (550h47m43.2944448s) [noise=true , address=145.79.227.47:4265 , piece_id=K4T2EXISJVRLN2P5O3SRNMP64EFCU544JS36CMFLHZWPXDJIIE5Q, seg_pos={0 0}, upload=482915]: Uploading piece...
Dec 28 14:16:45.043655343 (550h47m44.914738378s) [upload=482915]: Next returned nil segment
Dec 28 14:16:45.043687795 (550h47m44.91477083s) [upload=482915]: Waiting for error group managing segments...
Dec 28 14:16:45.430477838 (550h47m45.301560883s) [noise=true , address=145.79.227.14:4839 , piece_id=WPULRJZEINPHM3DIM2QKAAZFW2ILLTATMSPUY6T4OF66YZE6SI5A, seg_pos={0 0}, upload=482915]: Done uploading piece. err: <nil>
Dec 28 14:16:45.431223748 (550h47m45.302306793s) [noise=true , address=145.79.227.41:4086 , piece_id=LM7JQTVB4UBXDPKSRGQHMPCHVIFTBBZMZBYF6SCCHVR324FA6DWA, seg_pos={0 0}, upload=482915]: Done uploading piece. err: <nil>
Dec 28 14:16:45.431471129 (550h47m45.302554174s) [noise=true , address=145.79.227.51:4410 , piece_id=D54CNAJW42WNUOVADLLG4CXSAAWQJVU5QQY5L5RYIPV7UW5CCDNQ, seg_pos={0 0}, upload=482915]: Done uploading piece. err: <nil>
Dec 28 14:16:45.432101900 (550h47m45.303184945s) [noise=true , address=145.79.227.50:4373 , piece_id=MSPPIR5CJB2RDUO6AME3GZXZIRUSGAOVVPQI6FKAHQBNX52NZEOQ, seg_pos={0 0}, upload=482915]: Done uploading piece. err: <nil>
Dec 28 14:16:45.432474099 (550h47m45.303557144s) [noise=true , address=145.79.227.52:4446 , piece_id=7LMULWAFSMMFNMAI3SY2FGPC4LKXHICDHPDJ4PDSNFS2CVVIHIHQ, seg_pos={0 0}, upload=482915]: Done uploading piece. err: <nil>
Dec 28 14:16:45.433244125 (550h47m45.30432716s) [noise=true , address=145.79.227.25:4618 , piece_id=MFO22WITIXSLXUMZH3HYKQ5MSRC63SANUULAVZQKPARMOOQRRA6Q, seg_pos={0 0}, upload=482915]: Done uploading piece. err: <nil>
Dec 28 14:16:45.435245826 (550h47m45.306328861s) [noise=true , address=145.79.227.12:4765 , piece_id=REZVMN72Y37UCHHOATYR3SGKYZE7ZT5IETPNIGGJZ6YXBZARLTYQ, seg_pos={0 0}, upload=482915]: Done uploading piece. err: <nil>
Dec 28 14:16:45.441369777 (550h47m45.312452822s) [noise=true , address=145.79.227.47:4265 , piece_id=K4T2EXISJVRLN2P5O3SRNMP64EFCU544JS36CMFLHZWPXDJIIE5Q, seg_pos={0 0}, upload=482915]: Done uploading piece. err: <nil>
Dec 28 14:16:45.447304649 (550h47m45.318387684s) [noise=true , address=145.79.227.4:4186 , piece_id=F2QAT7WBDPU4BVVOVLFCFZJP4YYEZL4X23UCXSLMTZ63PMABWPKA, seg_pos={0 0}, upload=482915]: Done uploading piece. err: <nil>
Dec 28 14:16:45.451226816 (550h47m45.322309851s) [noise=true , address=145.79.227.2:4115 , piece_id=M4HN5C5BNM5FHCKVKVDRYCSTHCKULR2WAQBQ5GT3MNAWMYW2LWBA, seg_pos={0 0}, upload=482915]: Done uploading piece. err: <nil>
Dec 28 14:16:45.451984159 (550h47m45.323067204s) [noise=true , address=145.79.227.32:4836 , piece_id=KU3KMONL7TRKK3RD6FDSC4IJGQUQRON5LRKHPMS7ITTCZZ36RWEQ, seg_pos={0 0}, upload=482915]: Done uploading piece. err: <nil>
Dec 28 14:16:45.624008943 (550h47m45.495091988s) [noise=true , address=77.90.31.12:4589 , piece_id=QKCA26Z3VAQ3SKTFZP4RTNTE26ODUCRT5UOSN73DPHHAXHWN35DA, seg_pos={0 0}, upload=482915]: Done uploading piece. err: <nil>
Dec 28 14:16:45.673846592 (550h47m45.544929637s) [noise=true , address=45.91.116.9:4261 , piece_id=JOGZR3JAVHJ5KSXG524QVRNKGZOQN2TJJGQA6EC7UP67YWW67PYA, seg_pos={0 0}, upload=482915]: Done uploading piece. err: <nil>
Dec 28 14:16:45.673872401 (550h47m45.544955436s) [seg_pos={0 0}, upload=482915]: Segment reached optimal threshold of 13 pieces.
Dec 28 14:16:45.674298632 (550h47m45.545381678s) [noise=true , address=45.91.116.7:4010 , piece_id=WPUSCFE6P5B7VO2SFTA5MUWLHHTZWICL22Y6NDXPFR3O3AK3Q3CQ, seg_pos={0 0}, upload=482915]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF
Dec 28 14:16:45.674309263 (550h47m45.545392308s) [noise=true , address=45.91.116.7:4010 , piece_id=WPUSCFE6P5B7VO2SFTA5MUWLHHTZWICL22Y6NDXPFR3O3AK3Q3CQ, seg_pos={0 0}, upload=482915]: Long tail cancellation due to optimal threshold
Dec 28 14:16:45.674637468 (550h47m45.545720513s) [noise=true , address=77.90.31.16:4736 , piece_id=HNPPZ7EZ36NIOVKU7WBDPKQFTC4UFGMOZR567IHMGIGUXJ7G5CQA, seg_pos={0 0}, upload=482915]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF
Dec 28 14:16:45.674656655 (550h47m45.5457397s) [noise=true , address=77.90.31.16:4736 , piece_id=HNPPZ7EZ36NIOVKU7WBDPKQFTC4UFGMOZR567IHMGIGUXJ7G5CQA, seg_pos={0 0}, upload=482915]: Long tail cancellation due to optimal threshold
Dec 28 14:16:45.674664960 (550h47m45.545748026s) [noise=true , address=45.91.116.12:4228 , piece_id=73Z626ZAGADQJZF5R25PESOPUDS5GACL6ITK5ANDIX3X3IGL5SWQ, seg_pos={0 0}, upload=482915]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF
Dec 28 14:16:45.674673086 (550h47m45.545756131s) [noise=true , address=45.91.116.12:4228 , piece_id=73Z626ZAGADQJZF5R25PESOPUDS5GACL6ITK5ANDIX3X3IGL5SWQ, seg_pos={0 0}, upload=482915]: Long tail cancellation due to optimal threshold
Dec 28 14:16:45.674691962 (550h47m45.545774997s) [seg_pos={0 0}, upload=482915]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 14:16:45.674776873 (550h47m45.545859928s) [upload=482915]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 14:16:45.674783957 (550h47m45.545867002s) [upload=482915]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 14:16:45.960697644 (550h47m45.831780689s) [upload=482915]: Done uploading segments...
Dec 28 14:16:45.960736558 (550h47m45.831819593s) [upload=482915]: Upload finished. err: <nil>
Duration: 30.003427664s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [..........X....................X...X........X.............X.] duration:
22.821548615s trace:
https://jaeger.dev.storj.io/trace/35a5d4335e2cde85
FAIL (2025-12-28 14:17:10 UTC)
orbiter warning: context deadline exceeded
Duration: 22.82157732s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [...............................................X...X........] duration:
14.029828076s trace:
https://jaeger.dev.storj.io/trace/102cf391e061ae77
FAIL (2025-12-28 14:16:54 UTC)
orbiter warning: context deadline exceeded
Duration: 14.029854517s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [.X....................................X........X...X........] duration:
51.669042805s trace:
https://jaeger.dev.storj.io/trace/36d5af3c761e71b1
FAIL (2025-12-28 14:16:39 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
7.49:4360 , piece_id=QDT5N5LDL2HCVXPOOSXWD2TOEZDF2JZWFLDFYSIMN647FI3ZORUQ, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.372193736 (550h40m10.243276781s) [noise=true , address=77.90.31.12:4618 , piece_id=45WRZ26CPUODOAYF5B6RF37DNMDDQWWJUDK7BFKYS3K3LOCNYJCA, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.372518505 (550h40m10.24360155s) [noise=true , address=77.90.31.13:4646 , piece_id=5CDMQF7GG5PDPHIPMFZWDBKWH5ANO3UIX257LJPCWRITNOA7GY4A, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.372848052 (550h40m10.243931097s) [noise=true , address=64.49.13.170:4721 , piece_id=U76TI7IPAHPZAS6XLG66E4L2CGCYH43NH4ZHTYXMXKZR6SGE7SKQ, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.373163072 (550h40m10.244246117s) [noise=true , address=77.90.31.11:4565 , piece_id=4SBLRZK3K4ED73PJ64BI6ANKBWB2AAHQA4SBUNH6CJ63GN3FJR2Q, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.373445089 (550h40m10.244528134s) [noise=true , address=145.79.227.34:4911 , piece_id=24WE4EMZSL2JHQZ7PESCMLYCZOWGDTGJD2XNLUUJEFPSYLRI5E7A, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.373768855 (550h40m10.24485189s) [noise=true , address=77.90.31.18:4819 , piece_id=OYXPAN3EK4QRDEC6VHIDL3GSGEY27G72KII6HTGUMSPYLHCYBNEA, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.374285519 (550h40m10.245368564s) [noise=true , address=77.90.31.14:4678 , piece_id=VBGPSGMNDQZYY7UKRINAV6B3FVI5BPDPYS37FQIJCMLL65F3XOEQ, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.374710689 (550h40m10.245793724s) [noise=true , address=145.79.227.40:4053 , piece_id=RO4ZZQCKCC7T4T6SOT7GGTY626BJ6ESLQG6KATTPNIUM3ICUKHAQ, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.375099329 (550h40m10.246182364s) [noise=true , address=77.90.31.19:4859 , piece_id=A4HD3FPDPCCMHRFBTS3GNGB323RYD5C5AW4ZVSNOTXFI7FLIV37Q, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.376073433 (550h40m10.247156468s) [noise=true , address=77.90.31.15:4704 , piece_id=DXWLFOIKOPE33LMEQECUJVIGRG3LPADUMULLWI4ZBHEOQHD43ULQ, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.376542387 (550h40m10.247625422s) [noise=true , address=145.79.227.44:4221 , piece_id=H72XCWLKYJAXYGGDIWN67H42RUIEOJCPUSWT5IJMFPKETH7LK3KA, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.377379761 (550h40m10.248462796s) [noise=true , address=145.79.227.32:4863 , piece_id=27TOXK44RIF3DV3PDY4RBP6HEFRQJE6N7LNLRJKSU52YCB6A3JKA, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.378022765 (550h40m10.24910581s) [noise=true , address=145.79.227.30:4789 , piece_id=4QT5IUPIK7BOHJROEK2EBFTW434OKC75FZNENMYYCRMBZ7E5BUTA, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.378821687 (550h40m10.249904722s) [noise=true , address=145.79.227.2:4119 , piece_id=EQM3WMBQ7GXRWRBZK4ZWCI23K4LSLXW6QPEABHGMWRHEYUYANKSQ, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:11.676223695 (550h40m11.54730674s) [upload=482805]: Next returned nil segment
Dec 28 14:09:11.676238083 (550h40m11.547321128s) [upload=482805]: Waiting for error group managing segments...
Dec 28 14:09:12.006762275 (550h40m11.87784531s) [noise=true , address=64.49.13.170:4721 , piece_id=U76TI7IPAHPZAS6XLG66E4L2CGCYH43NH4ZHTYXMXKZR6SGE7SKQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.250307967 (550h40m12.121391012s) [noise=true , address=145.79.227.2:4119 , piece_id=EQM3WMBQ7GXRWRBZK4ZWCI23K4LSLXW6QPEABHGMWRHEYUYANKSQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.269380734 (550h40m12.140463769s) [noise=true , address=145.79.227.49:4360 , piece_id=QDT5N5LDL2HCVXPOOSXWD2TOEZDF2JZWFLDFYSIMN647FI3ZORUQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.290818335 (550h40m12.16190138s) [noise=true , address=145.79.227.32:4863 , piece_id=27TOXK44RIF3DV3PDY4RBP6HEFRQJE6N7LNLRJKSU52YCB6A3JKA, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.467679327 (550h40m12.338762362s) [noise=true , address=45.91.116.8:4072 , piece_id=GEVTX6LLENPLN3UQLSGTFTP3DO52IRYUIQFIWC4XMOC3ZF6V4GTQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.486615595 (550h40m12.35769863s) [noise=true , address=77.90.31.12:4618 , piece_id=45WRZ26CPUODOAYF5B6RF37DNMDDQWWJUDK7BFKYS3K3LOCNYJCA, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.566542810 (550h40m12.437625845s) [noise=true , address=77.90.31.19:4859 , piece_id=A4HD3FPDPCCMHRFBTS3GNGB323RYD5C5AW4ZVSNOTXFI7FLIV37Q, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.609337578 (550h40m12.480420623s) [noise=true , address=77.90.31.13:4646 , piece_id=5CDMQF7GG5PDPHIPMFZWDBKWH5ANO3UIX257LJPCWRITNOA7GY4A, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.640856565 (550h40m12.511939611s) [noise=true , address=45.91.116.5:4322 , piece_id=SS6DOCSJHY5Y3DPWMYX45F4VYYXR44JDLYQ2YPY5XIIAYKMAG7LQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.645490329 (550h40m12.516573364s) [noise=true , address=77.90.31.18:4819 , piece_id=OYXPAN3EK4QRDEC6VHIDL3GSGEY27G72KII6HTGUMSPYLHCYBNEA, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:13.103710286 (550h40m12.974793331s) [noise=true , address=145.79.227.34:4911 , piece_id=24WE4EMZSL2JHQZ7PESCMLYCZOWGDTGJD2XNLUUJEFPSYLRI5E7A, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:13.209917141 (550h40m13.081000176s) [noise=true , address=145.79.227.44:4221 , piece_id=H72XCWLKYJAXYGGDIWN67H42RUIEOJCPUSWT5IJMFPKETH7LK3KA, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:13.218355468 (550h40m13.089438514s) [noise=true , address=77.90.31.11:4565 , piece_id=4SBLRZK3K4ED73PJ64BI6ANKBWB2AAHQA4SBUNH6CJ63GN3FJR2Q, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:13.255968741 (550h40m13.127051806s) [noise=true , address=145.79.227.30:4789 , piece_id=4QT5IUPIK7BOHJROEK2EBFTW434OKC75FZNENMYYCRMBZ7E5BUTA, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:13.255996102 (550h40m13.127079137s) [seg_pos={0 0}, upload=482805]: Segment reached optimal threshold of 14 pieces.
Dec 28 14:09:13.256214589 (550h40m13.127297634s) [noise=true , address=145.79.227.40:4053 , piece_id=RO4ZZQCKCC7T4T6SOT7GGTY626BJ6ESLQG6KATTPNIUM3ICUKHAQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DqyTc4MHvxutFXs2WWRCk8MSuxF3Y1HayyQY4Y5sy8cqDLWuG): protocol: expected piece hash; context canceled; EOF
Dec 28 14:09:13.256223686 (550h40m13.127306731s) [noise=true , address=145.79.227.40:4053 , piece_id=RO4ZZQCKCC7T4T6SOT7GGTY626BJ6ESLQG6KATTPNIUM3ICUKHAQ, seg_pos={0 0}, upload=482805]: Long tail cancellation due to optimal threshold
Dec 28 14:09:13.256512506 (550h40m13.127595551s) [noise=true , address=77.90.31.15:4704 , piece_id=DXWLFOIKOPE33LMEQECUJVIGRG3LPADUMULLWI4ZBHEOQHD43ULQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14tJRdZ1guwwsYwPf3LtENRWqYx2jBQpi7uAMMPaB32Gks9zmj): protocol: expected piece hash; context canceled; EOF
Dec 28 14:09:13.256520060 (550h40m13.127603105s) [noise=true , address=77.90.31.15:4704 , piece_id=DXWLFOIKOPE33LMEQECUJVIGRG3LPADUMULLWI4ZBHEOQHD43ULQ, seg_pos={0 0}, upload=482805]: Long tail cancellation due to optimal threshold
Dec 28 14:09:13.256649106 (550h40m13.127732151s) [noise=true , address=77.90.31.14:4678 , piece_id=VBGPSGMNDQZYY7UKRINAV6B3FVI5BPDPYS37FQIJCMLL65F3XOEQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12beu3SUZMPNWfbeHV4tLtx9UVZoEHh4Kt7oBhT59w3bGBE1ovP): protocol: write tcp 188.166.225.170:57374->77.90.31.14:4678: use of closed network connection; write tcp 188.166.225.170:57374->77.90.31.14:4678: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57374->77.90.31.14:4678: use of closed network connection
Dec 28 14:09:13.256655819 (550h40m13.127738864s) [noise=true , address=77.90.31.14:4678 , piece_id=VBGPSGMNDQZYY7UKRINAV6B3FVI5BPDPYS37FQIJCMLL65F3XOEQ, seg_pos={0 0}, upload=482805]: Long tail cancellation due to optimal threshold
Dec 28 14:09:13.256752974 (550h40m13.127836009s) [noise=true , address=64.49.13.169:4710 , piece_id=SANL7TVJFES63XD46OKQL4KIJ6YVHSEWPZKMRISEKC3ZKWQBOMIA, seg_pos={0 0}, upload=482805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:35680->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:35680->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35680->64.49.13.169:4710: use of closed network connection
Dec 28 14:09:13.256758825 (550h40m13.12784186s) [noise=true , address=64.49.13.169:4710 , piece_id=SANL7TVJFES63XD46OKQL4KIJ6YVHSEWPZKMRISEKC3ZKWQBOMIA, seg_pos={0 0}, upload=482805]: Long tail cancellation due to optimal threshold
Dec 28 14:09:13.256776549 (550h40m13.127859594s) [seg_pos={0 0}, upload=482805]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 14:09:13.256856591 (550h40m13.127939626s) [upload=482805]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 14:09:13.256862432 (550h40m13.127945467s) [upload=482805]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 14:09:13.550363717 (550h40m13.421446782s) [upload=482805]: Done uploading segments...
Dec 28 14:09:13.550405126 (550h40m13.421488161s) [upload=482805]: Upload finished. err: <nil>
Duration: 51.66907237s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [.........X..................................................] duration:
9.692092631s trace:
https://jaeger.dev.storj.io/trace/5fe9d6dbb79fbe44
FAIL (2025-12-28 14:16:33 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
67.204.205:4027 , piece_id=PAPNMRJ7E47VOMOXPBBDWKZJHLRMXTWVR5VW3J7KNQSSMHOHWOAQ, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.967737953 (550h8m4.838820988s) [noise=true , address=185.167.204.206:4072 , piece_id=WXORBDMA3KW7WZ67A4FJT6URJRFADUJ7X2PAHIGNG47T3TGMXVTQ, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.968193240 (550h8m4.839276285s) [noise=true , address=185.167.204.205:4030 , piece_id=R5DLI4ZTLOVGO56GPLJL5ZNHYSXOSYNJQHFIO4MVVNHWWBMEOV2Q, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.968780638 (550h8m4.839863683s) [noise=true , address=89.149.201.147:4019 , piece_id=MK7ORCLHO3DOWIOVSOWIQMPJ447PHRQGXP3UKITIRTYJSU5MPCCQ, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.969308924 (550h8m4.840391959s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.970088349 (550h8m4.841171404s) [noise=true , address=81.171.14.70:4039 , piece_id=EDHPQWLQZTP7MKPGEVMP7JXNTW7LFWSXCTRTSXPCBIQPROX76WRA, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.970325381 (550h8m4.841408426s) [noise=true , address=81.171.14.70:4036 , piece_id=CICNNV7JE6TBNC4RAPXKCEUSJAADOUVGSVIVQISM3ZB4COAZQDDA, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.970783813 (550h8m4.841866858s) [noise=true , address=81.171.14.70:4044 , piece_id=DGN74IG6IWAHLEMO6NLL4TJBXZ6ONTN6QUB6HXY6K7SSO3LYOYIQ, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.971205406 (550h8m4.842288441s) [noise=true , address=81.171.14.70:4048 , piece_id=TLT47ORGUKGF3MIIEAX7D22VR2FO6W6NL3VFJNHKWD4L3LIMFDLA, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.971606299 (550h8m4.842689344s) [noise=true , address=185.167.204.206:4067 , piece_id=VSKTG5RH32N7OEVWJGQVB6WJ72TEVF6DDK25W7NOH4RAXUP52LTQ, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.972162538 (550h8m4.843245583s) [noise=true , address=89.149.201.147:4002 , piece_id=G2PABNTDWCK4MKOZF6PONBMRNIXI45JTHJMAZFPEKPEJ3PQ7MTBA, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:05.795167814 (550h8m5.666250849s) [upload=482337]: Next returned nil segment
Dec 28 13:37:05.795184566 (550h8m5.666267611s) [upload=482337]: Waiting for error group managing segments...
Dec 28 13:37:06.178573801 (550h8m6.049656836s) [noise=true , address=185.167.204.206:4049 , piece_id=6RG2DZV6FPTIRE63ZNNST42TSWEBQ456YXIPPVYHIXV55GWFVJ4Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.183060835 (550h8m6.05414391s) [noise=true , address=185.167.204.206:4072 , piece_id=WXORBDMA3KW7WZ67A4FJT6URJRFADUJ7X2PAHIGNG47T3TGMXVTQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.183411202 (550h8m6.054494237s) [noise=true , address=185.167.204.206:4064 , piece_id=OG3G74OU23CEMKA54WQE3DOJ6AKAXBLZ23QWOQC4UEGQQN4JSMXQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.183791146 (550h8m6.054874181s) [noise=true , address=185.167.204.205:4029 , piece_id=6JQB6ZSCQZKWH3JWCQ4DY6BBAX73KC4Y53PRSMEMLWURZ5QWWQQQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.183973743 (550h8m6.055056778s) [noise=true , address=185.167.204.206:4053 , piece_id=YDH77S6JLKJ6IYS6DRUCELP5EJHIC2O44QBO7MIOEUYBML4GIBHA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.184465369 (550h8m6.055548414s) [noise=true , address=185.167.204.205:4008 , piece_id=RTUZPEXBB6JNSMC5YOFQQGDM57YD233W4756ZBQVMFNOVX6YKB4A, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.184501989 (550h8m6.055585024s) [noise=true , address=185.167.204.205:4030 , piece_id=R5DLI4ZTLOVGO56GPLJL5ZNHYSXOSYNJQHFIO4MVVNHWWBMEOV2Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.184856224 (550h8m6.055939259s) [noise=true , address=185.167.204.205:4009 , piece_id=V6RA7Y7FKY4DCPTRRF3BYNBALWDY5VTPXMDUXFE3353KUXLSQJEA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.185063518 (550h8m6.056146563s) [noise=true , address=185.167.204.205:4016 , piece_id=JB7SC3J2VR4U33HNQS2A2CA75THT3XLQCOIFHAJJEK7MQGY2SYAQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.185288617 (550h8m6.056371652s) [noise=true , address=185.167.204.205:4027 , piece_id=PAPNMRJ7E47VOMOXPBBDWKZJHLRMXTWVR5VW3J7KNQSSMHOHWOAQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.186241401 (550h8m6.057324436s) [noise=true , address=185.167.204.205:4024 , piece_id=FM5IMWRSJJ4RY72IX2OOYC54BFKL4FWE3UOAN3GJ26MLH345COVA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.193113658 (550h8m6.064196703s) [noise=true , address=185.167.204.205:4005 , piece_id=R4DFQJZBPJ5WJWHPODV6ILHUDIRBUSAFOMVNCLXB4UNMI6EXTKHA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.193791408 (550h8m6.064874453s) [noise=true , address=185.167.204.206:4068 , piece_id=DAI37M67FWCSVQTQOZ6RSSOJDGUVZO6SNGAHJJ44G3PGS4OJZEMA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.194292262 (550h8m6.065375307s) [noise=true , address=185.167.204.206:4046 , piece_id=VWLVHMBMJ63DQEWYBIKH3Y3NTVWFNW5OXZKMYXXBHVI3VWYXLLPQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.194302652 (550h8m6.065385687s) [noise=true , address=185.167.204.206:4067 , piece_id=VSKTG5RH32N7OEVWJGQVB6WJ72TEVF6DDK25W7NOH4RAXUP52LTQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.195359193 (550h8m6.066442228s) [noise=true , address=185.167.204.205:4012 , piece_id=LV6RUA5BWXG4DMSUVZDBGUAF5IBWUGMLJYUKNIXHRVDEBIOW4Q2Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.198423448 (550h8m6.069506493s) [noise=true , address=185.167.204.206:4070 , piece_id=5A7YYQYREIZRZDR7TRSE7XFW765JVAV7FX6LKWUI2OXAA74XN4CA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.888968408 (550h8m6.760051453s) [noise=true , address=81.171.14.70:4044 , piece_id=DGN74IG6IWAHLEMO6NLL4TJBXZ6ONTN6QUB6HXY6K7SSO3LYOYIQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.892350789 (550h8m6.763433834s) [noise=true , address=81.171.14.70:4039 , piece_id=EDHPQWLQZTP7MKPGEVMP7JXNTW7LFWSXCTRTSXPCBIQPROX76WRA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.904477160 (550h8m6.775560195s) [noise=true , address=81.171.14.70:4040 , piece_id=DYF4GSZBAKQ3H63LTVCMEFWMABY3FXHOJCNVIFYRKLF2J6B52K5A, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.128088281 (550h8m6.999171326s) [noise=true , address=81.171.14.70:4036 , piece_id=CICNNV7JE6TBNC4RAPXKCEUSJAADOUVGSVIVQISM3ZB4COAZQDDA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.430242142 (550h8m7.301325177s) [noise=true , address=89.149.201.147:4009 , piece_id=EVW2NQFYU57PC2IIVXLYIEBMYXJNIC2CLCHW6PS6NYIPCFRUVHDA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.505304332 (550h8m7.376387367s) [noise=true , address=89.149.201.147:4019 , piece_id=MK7ORCLHO3DOWIOVSOWIQMPJ447PHRQGXP3UKITIRTYJSU5MPCCQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.575461543 (550h8m7.446544609s) [noise=true , address=81.171.14.70:4048 , piece_id=TLT47ORGUKGF3MIIEAX7D22VR2FO6W6NL3VFJNHKWD4L3LIMFDLA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.599667265 (550h8m7.4707503s) [noise=true , address=89.149.201.147:4023 , piece_id=5IINX2GVRFLRX2QH24OO2DDANOKOTXTDR6AGVJBGSJ7IAZQ3CASA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.928390728 (550h8m7.799473773s) [noise=true , address=89.149.201.147:4002 , piece_id=G2PABNTDWCK4MKOZF6PONBMRNIXI45JTHJMAZFPEKPEJ3PQ7MTBA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.928417148 (550h8m7.799500193s) [seg_pos={0 0}, upload=482337]: Segment reached optimal threshold of 26 pieces.
Dec 28 13:37:07.928656303 (550h8m7.799739348s) [noise=true , address=81.171.14.70:4030 , piece_id=74YDIKUST7X3JBXVQHVLFHIKH5A4DYH5TNL3I4XUA5A6AHQO2DRA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZoRwa8wLw66Pjumy1ybp6wrZ79RrSU55uYZpwaY3xi95iSYde): protocol: expected piece hash; context canceled; EOF
Dec 28 13:37:07.928664990 (550h8m7.799748035s) [noise=true , address=81.171.14.70:4030 , piece_id=74YDIKUST7X3JBXVQHVLFHIKH5A4DYH5TNL3I4XUA5A6AHQO2DRA, seg_pos={0 0}, upload=482337]: Long tail cancellation due to optimal threshold
Dec 28 13:37:07.928723351 (550h8m7.799806386s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wgtphHKaTejC1qrJ98sDysBwtj2Zeqw2c5UFk1HMqfY8Buvr): protocol: expected piece hash; context canceled; EOF
Dec 28 13:37:07.928733922 (550h8m7.799816957s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Long tail cancellation due to optimal threshold
Dec 28 13:37:07.928754921 (550h8m7.799837966s) [seg_pos={0 0}, upload=482337]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 28 13:37:07.928870170 (550h8m7.799953215s) [upload=482337]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 13:37:07.928878627 (550h8m7.799961672s) [upload=482337]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 13:37:08.166321536 (550h8m8.037404571s) [upload=482337]: Done uploading segments...
Dec 28 13:37:08.166341444 (550h8m8.037424479s) [upload=482337]: Upload finished. err: <nil>
Duration: 9.692122316s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
53 out of 60 is failed (threshold=6) [XXXXX.XXXXXXXXXXXX.XXXXXXX.XX..XXXXXXXXXXXXXXXXXXXXX.X.XXXXX] duration:
5.003297627s trace:
https://jaeger.dev.storj.io/trace/22346b7e9c38450
FAIL (2025-12-28 14:16:25 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.003324457s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [...........X....................X...X........X.............X] duration:
30.001439633s trace:
https://jaeger.dev.storj.io/trace/35a5d4335e2cde85
FAIL (2025-12-28 14:16:18 UTC)
orbiter warning: context deadline exceeded
Duration: 30.001465471s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
20 out of 60 is failed (threshold=3) [.X.....X..X.X...XX......X......XXX.XX....XX......XX.XX.X.X..] duration:
5.912746898s trace:
https://jaeger.dev.storj.io/trace/5be1acce1690abbe
FAIL (2025-12-28 14:16:10 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 28 14:14:05.707801185 (550h45m5.57888423s) [upload=482879]: Starting upload
Dec 28 14:14:05.707886898 (550h45m5.578969944s) [upload=482879]: Done starting upload
Dec 28 14:14:05.707951642 (550h45m5.579034687s) [upload=482879]: Uploading segments...
Dec 28 14:14:05.708074515 (550h45m5.57915756s) [upload=482879]: Got next segment. Inline: false
Dec 28 14:14:05.708106106 (550h45m5.579189151s) [upload=482879]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 14:14:05.708110434 (550h45m5.579193479s) [upload=482879]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 14:14:06.000535780 (550h45m5.871618845s) [seg_pos={0 0}, upload=482879]: Begin upload segment...
Dec 28 14:14:06.000598910 (550h45m5.871681945s) [seg_pos={0 0}, upload=482879]: Done begin upload segment.
Dec 28 14:14:06.000636912 (550h45m5.871719957s) [noise=true , address=145.79.227.40:4051 , piece_id=3QYIHWAQR7G5I57JDXJ4SJ6A2LARGCAP6XK7CVE6K2WVR4XGXXHQ, seg_pos={0 0}, upload=482879]: Uploading piece...
Dec 28 14:14:06.000757032 (550h45m5.871840097s) [noise=true , address=145.79.227.13:4804 , piece_id=PGTARYC3LNMTR4TAGR74FPBM7RHMTHRLDW2QUW6ARA2SI26DHLZQ, seg_pos={0 0}, upload=482879]: Uploading piece...
Dec 28 14:14:06.001400927 (550h45m5.872483972s) [noise=true , address=45.91.116.9:4262 , piece_id=54C5OVTRQ2FAS4X2EEPNRRAKFEWDO5JQEDQ5HOCOTPJW54PUK2YA, seg_pos={0 0}, upload=482879]: Uploading piece...
Dec 28 14:14:06.001895939 (550h45m5.872978975s) [noise=true , address=145.79.227.47:4266 , piece_id=FUBWPNHK7SKGUJSKNMTHO3RGUCTABSL4QOASCW23J6GCJ3H5LDMQ, seg_pos={0 0}, upload=482879]: Uploading piece...
Dec 28 14:14:06.002437291 (550h45m5.873520336s) [noise=true , address=45.91.116.14:4190 , piece_id=565ADKSRT363WQJIB2WFNFLNPS765AVU7R72A5LMHGACOYZNBYDQ, seg_pos={0 0}, upload=482879]: Uploading piece...
Dec 28 14:14:06.002952352 (550h45m5.874035397s) [noise=true , address=45.91.116.6:4975 , piece_id=IRL2CQSLRGQJIP3BHV7HYBS56R7HGUDQPQBNNFXEE6Q7IUF2CXHA, seg_pos={0 0}, upload=482879]: Uploading piece...
Dec 28 14:14:06.002965867 (550h45m5.874048912s) [noise=true , address=145.79.227.42:4123 , piece_id=7QUYG56BEL7FSD2A6AHQNONXE4PWH3QEHZSBKEGYKXYRH5DKIP5Q, seg_pos={0 0}, upload=482879]: Uploading piece...
Dec 28 14:14:06.003387630 (550h45m5.874470685s) [noise=true , address=45.91.116.11:4120 , piece_id=MRG7VL7KTA3T5FEUFJW5ISMKDWWLUGHHBGGL7RMWRSON4N2UVZUA, seg_pos={0 0}, upload=482879]: Uploading piece...
Dec 28 14:14:06.003838088 (550h45m5.874921133s) [noise=true , address=45.91.116.12:4228 , piece_id=VXDU5ZAC3NCP65MDUZ5LSVEIGRQDJIJDP2TIUGXPHGLBLGBMMCWA, seg_pos={0 0}, upload=482879]: Uploading piece...
Dec 28 14:14:06.004305238 (550h45m5.875388303s) [noise=true , address=145.79.227.50:4373 , piece_id=RAXJ6LT3KF7DOQRSV7DH5IAPJ4WMWDLIPIOFFVQUWRU5BF4SERNQ, seg_pos={0 0}, upload=482879]: Uploading piece...
Dec 28 14:14:06.004804198 (550h45m5.875887233s) [noise=true , address=145.79.227.46:4603 , piece_id=TWVG57KDN3OV4DU3QD4ONWNEEYOCKYXYQ63QBQPEZS676HDAMZYA, seg_pos={0 0}, upload=482879]: Uploading piece...
Dec 28 14:14:06.005276297 (550h45m5.876359342s) [noise=true , address=145.79.227.12:4767 , piece_id=VPWI5VMVMUCTRSYXD3QSZ6OUETWIAJQLGZQ7MRSUL2Y475DNCK3Q, seg_pos={0 0}, upload=482879]: Uploading piece...
Dec 28 14:14:06.005723668 (550h45m5.876806713s) [noise=true , address=145.79.227.52:4446 , piece_id=L2LK33S7SEF7NTBUT3AX7XMTKPGPICTAXSS6MYA2OJ3DOZWRJJHA, seg_pos={0 0}, upload=482879]: Uploading piece...
Dec 28 14:14:06.006312079 (550h45m5.877395124s) [noise=true , address=77.90.31.21:4880 , piece_id=2EP7PPWFUVNZLRRIGSC7GW53WGRS2XCZNP7O5BDDNHENMJBISNZA, seg_pos={0 0}, upload=482879]: Uploading piece...
Dec 28 14:14:06.006896762 (550h45m5.877979807s) [noise=true , address=145.79.227.37:4566 , piece_id=5ICU5SETRGGIIKRNDRCTUUC2VTSUGRD6PWQH7BTZYVYD7S3SQSQQ, seg_pos={0 0}, upload=482879]: Uploading piece...
Dec 28 14:14:06.007257900 (550h45m5.878340935s) [noise=true , address=145.79.227.20:4439 , piece_id=WC36XT6DAV27DTUCFKYSU3AWZLWZ5EM37JQH3SZPSWOUOE4PR5HQ, seg_pos={0 0}, upload=482879]: Uploading piece...
Dec 28 14:14:07.721543183 (550h45m7.592626218s) [upload=482879]: Next returned nil segment
Dec 28 14:14:07.721554704 (550h45m7.59263775s) [upload=482879]: Waiting for error group managing segments...
Dec 28 14:14:07.997311191 (550h45m7.868394236s) [noise=true , address=145.79.227.13:4804 , piece_id=PGTARYC3LNMTR4TAGR74FPBM7RHMTHRLDW2QUW6ARA2SI26DHLZQ, seg_pos={0 0}, upload=482879]: Done uploading piece. err: <nil>
Dec 28 14:14:08.005357442 (550h45m7.876440507s) [noise=true , address=145.79.227.12:4767 , piece_id=VPWI5VMVMUCTRSYXD3QSZ6OUETWIAJQLGZQ7MRSUL2Y475DNCK3Q, seg_pos={0 0}, upload=482879]: Done uploading piece. err: <nil>
Dec 28 14:14:08.008351083 (550h45m7.879434118s) [noise=true , address=145.79.227.52:4446 , piece_id=L2LK33S7SEF7NTBUT3AX7XMTKPGPICTAXSS6MYA2OJ3DOZWRJJHA, seg_pos={0 0}, upload=482879]: Done uploading piece. err: <nil>
Dec 28 14:14:08.010815325 (550h45m7.88189837s) [noise=true , address=145.79.227.50:4373 , piece_id=RAXJ6LT3KF7DOQRSV7DH5IAPJ4WMWDLIPIOFFVQUWRU5BF4SERNQ, seg_pos={0 0}, upload=482879]: Done uploading piece. err: <nil>
Dec 28 14:14:08.014508277 (550h45m7.885591352s) [noise=true , address=145.79.227.42:4123 , piece_id=7QUYG56BEL7FSD2A6AHQNONXE4PWH3QEHZSBKEGYKXYRH5DKIP5Q, seg_pos={0 0}, upload=482879]: Done uploading piece. err: <nil>
Dec 28 14:14:08.015107678 (550h45m7.886190723s) [noise=true , address=145.79.227.20:4439 , piece_id=WC36XT6DAV27DTUCFKYSU3AWZLWZ5EM37JQH3SZPSWOUOE4PR5HQ, seg_pos={0 0}, upload=482879]: Done uploading piece. err: <nil>
Dec 28 14:14:08.020768196 (550h45m7.891851231s) [noise=true , address=145.79.227.46:4603 , piece_id=TWVG57KDN3OV4DU3QD4ONWNEEYOCKYXYQ63QBQPEZS676HDAMZYA, seg_pos={0 0}, upload=482879]: Done uploading piece. err: <nil>
Dec 28 14:14:08.022418348 (550h45m7.893501383s) [noise=true , address=145.79.227.47:4266 , piece_id=FUBWPNHK7SKGUJSKNMTHO3RGUCTABSL4QOASCW23J6GCJ3H5LDMQ, seg_pos={0 0}, upload=482879]: Done uploading piece. err: <nil>
Dec 28 14:14:08.065573974 (550h45m7.936657019s) [noise=true , address=145.79.227.37:4566 , piece_id=5ICU5SETRGGIIKRNDRCTUUC2VTSUGRD6PWQH7BTZYVYD7S3SQSQQ, seg_pos={0 0}, upload=482879]: Done uploading piece. err: <nil>
Dec 28 14:14:08.252250557 (550h45m8.123333602s) [noise=true , address=45.91.116.6:4975 , piece_id=IRL2CQSLRGQJIP3BHV7HYBS56R7HGUDQPQBNNFXEE6Q7IUF2CXHA, seg_pos={0 0}, upload=482879]: Done uploading piece. err: <nil>
Dec 28 14:14:08.253942699 (550h45m8.125025734s) [noise=true , address=45.91.116.11:4120 , piece_id=MRG7VL7KTA3T5FEUFJW5ISMKDWWLUGHHBGGL7RMWRSON4N2UVZUA, seg_pos={0 0}, upload=482879]: Done uploading piece. err: <nil>
Dec 28 14:14:09.023479484 (550h45m8.894562519s) [noise=true , address=45.91.116.14:4190 , piece_id=565ADKSRT363WQJIB2WFNFLNPS765AVU7R72A5LMHGACOYZNBYDQ, seg_pos={0 0}, upload=482879]: Done uploading piece. err: <nil>
Dec 28 14:14:09.024215435 (550h45m8.89529848s) [noise=true , address=45.91.116.9:4262 , piece_id=54C5OVTRQ2FAS4X2EEPNRRAKFEWDO5JQEDQ5HOCOTPJW54PUK2YA, seg_pos={0 0}, upload=482879]: Done uploading piece. err: <nil>
Dec 28 14:14:09.024226005 (550h45m8.89530905s) [seg_pos={0 0}, upload=482879]: Segment reached optimal threshold of 13 pieces.
Dec 28 14:14:09.024424473 (550h45m8.895507508s) [noise=true , address=77.90.31.21:4880 , piece_id=2EP7PPWFUVNZLRRIGSC7GW53WGRS2XCZNP7O5BDDNHENMJBISNZA, seg_pos={0 0}, upload=482879]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF
Dec 28 14:14:09.024431356 (550h45m8.895514391s) [noise=true , address=77.90.31.21:4880 , piece_id=2EP7PPWFUVNZLRRIGSC7GW53WGRS2XCZNP7O5BDDNHENMJBISNZA, seg_pos={0 0}, upload=482879]: Long tail cancellation due to optimal threshold
Dec 28 14:14:09.024575721 (550h45m8.895658756s) [noise=true , address=145.79.227.40:4051 , piece_id=3QYIHWAQR7G5I57JDXJ4SJ6A2LARGCAP6XK7CVE6K2WVR4XGXXHQ, seg_pos={0 0}, upload=482879]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF
Dec 28 14:14:09.024582594 (550h45m8.895665639s) [noise=true , address=145.79.227.40:4051 , piece_id=3QYIHWAQR7G5I57JDXJ4SJ6A2LARGCAP6XK7CVE6K2WVR4XGXXHQ, seg_pos={0 0}, upload=482879]: Long tail cancellation due to optimal threshold
Dec 28 14:14:09.024595309 (550h45m8.895678354s) [noise=true , address=45.91.116.12:4228 , piece_id=VXDU5ZAC3NCP65MDUZ5LSVEIGRQDJIJDP2TIUGXPHGLBLGBMMCWA, seg_pos={0 0}, upload=482879]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:55860->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:55860->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55860->45.91.116.12:4228: use of closed network connection
Dec 28 14:14:09.024622490 (550h45m8.895705555s) [noise=true , address=45.91.116.12:4228 , piece_id=VXDU5ZAC3NCP65MDUZ5LSVEIGRQDJIJDP2TIUGXPHGLBLGBMMCWA, seg_pos={0 0}, upload=482879]: Long tail cancellation due to optimal threshold
Dec 28 14:14:09.024649401 (550h45m8.895732436s) [seg_pos={0 0}, upload=482879]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 14:14:09.024766164 (550h45m8.895849199s) [upload=482879]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 14:14:09.024773017 (550h45m8.895856062s) [upload=482879]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 14:14:09.321695907 (550h45m9.192778942s) [upload=482879]: Done uploading segments...
Dec 28 14:14:09.321715164 (550h45m9.192798199s) [upload=482879]: Upload finished. err: <nil>
Duration: 5.91277438s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [................................................X...X.......] duration:
15.865926112s trace:
https://jaeger.dev.storj.io/trace/102cf391e061ae77
FAIL (2025-12-28 14:15:44 UTC)
orbiter warning: context deadline exceeded
Duration: 15.86595679s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
53 out of 60 is failed (threshold=6) [XXXXXX.XXXXXXXXXXXX.XXXXXXX.XX..XXXXXXXXXXXXXXXXXXXXX.X.XXXX] duration:
5.00141358s trace:
https://jaeger.dev.storj.io/trace/8b2cfd73b75dea0
FAIL (2025-12-28 14:15:39 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001436373s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [..X....................................X........X...X.......] duration:
41.870873253s trace:
https://jaeger.dev.storj.io/trace/36d5af3c761e71b1
FAIL (2025-12-28 14:15:36 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
7.49:4360 , piece_id=QDT5N5LDL2HCVXPOOSXWD2TOEZDF2JZWFLDFYSIMN647FI3ZORUQ, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.372193736 (550h40m10.243276781s) [noise=true , address=77.90.31.12:4618 , piece_id=45WRZ26CPUODOAYF5B6RF37DNMDDQWWJUDK7BFKYS3K3LOCNYJCA, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.372518505 (550h40m10.24360155s) [noise=true , address=77.90.31.13:4646 , piece_id=5CDMQF7GG5PDPHIPMFZWDBKWH5ANO3UIX257LJPCWRITNOA7GY4A, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.372848052 (550h40m10.243931097s) [noise=true , address=64.49.13.170:4721 , piece_id=U76TI7IPAHPZAS6XLG66E4L2CGCYH43NH4ZHTYXMXKZR6SGE7SKQ, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.373163072 (550h40m10.244246117s) [noise=true , address=77.90.31.11:4565 , piece_id=4SBLRZK3K4ED73PJ64BI6ANKBWB2AAHQA4SBUNH6CJ63GN3FJR2Q, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.373445089 (550h40m10.244528134s) [noise=true , address=145.79.227.34:4911 , piece_id=24WE4EMZSL2JHQZ7PESCMLYCZOWGDTGJD2XNLUUJEFPSYLRI5E7A, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.373768855 (550h40m10.24485189s) [noise=true , address=77.90.31.18:4819 , piece_id=OYXPAN3EK4QRDEC6VHIDL3GSGEY27G72KII6HTGUMSPYLHCYBNEA, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.374285519 (550h40m10.245368564s) [noise=true , address=77.90.31.14:4678 , piece_id=VBGPSGMNDQZYY7UKRINAV6B3FVI5BPDPYS37FQIJCMLL65F3XOEQ, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.374710689 (550h40m10.245793724s) [noise=true , address=145.79.227.40:4053 , piece_id=RO4ZZQCKCC7T4T6SOT7GGTY626BJ6ESLQG6KATTPNIUM3ICUKHAQ, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.375099329 (550h40m10.246182364s) [noise=true , address=77.90.31.19:4859 , piece_id=A4HD3FPDPCCMHRFBTS3GNGB323RYD5C5AW4ZVSNOTXFI7FLIV37Q, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.376073433 (550h40m10.247156468s) [noise=true , address=77.90.31.15:4704 , piece_id=DXWLFOIKOPE33LMEQECUJVIGRG3LPADUMULLWI4ZBHEOQHD43ULQ, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.376542387 (550h40m10.247625422s) [noise=true , address=145.79.227.44:4221 , piece_id=H72XCWLKYJAXYGGDIWN67H42RUIEOJCPUSWT5IJMFPKETH7LK3KA, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.377379761 (550h40m10.248462796s) [noise=true , address=145.79.227.32:4863 , piece_id=27TOXK44RIF3DV3PDY4RBP6HEFRQJE6N7LNLRJKSU52YCB6A3JKA, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.378022765 (550h40m10.24910581s) [noise=true , address=145.79.227.30:4789 , piece_id=4QT5IUPIK7BOHJROEK2EBFTW434OKC75FZNENMYYCRMBZ7E5BUTA, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.378821687 (550h40m10.249904722s) [noise=true , address=145.79.227.2:4119 , piece_id=EQM3WMBQ7GXRWRBZK4ZWCI23K4LSLXW6QPEABHGMWRHEYUYANKSQ, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:11.676223695 (550h40m11.54730674s) [upload=482805]: Next returned nil segment
Dec 28 14:09:11.676238083 (550h40m11.547321128s) [upload=482805]: Waiting for error group managing segments...
Dec 28 14:09:12.006762275 (550h40m11.87784531s) [noise=true , address=64.49.13.170:4721 , piece_id=U76TI7IPAHPZAS6XLG66E4L2CGCYH43NH4ZHTYXMXKZR6SGE7SKQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.250307967 (550h40m12.121391012s) [noise=true , address=145.79.227.2:4119 , piece_id=EQM3WMBQ7GXRWRBZK4ZWCI23K4LSLXW6QPEABHGMWRHEYUYANKSQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.269380734 (550h40m12.140463769s) [noise=true , address=145.79.227.49:4360 , piece_id=QDT5N5LDL2HCVXPOOSXWD2TOEZDF2JZWFLDFYSIMN647FI3ZORUQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.290818335 (550h40m12.16190138s) [noise=true , address=145.79.227.32:4863 , piece_id=27TOXK44RIF3DV3PDY4RBP6HEFRQJE6N7LNLRJKSU52YCB6A3JKA, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.467679327 (550h40m12.338762362s) [noise=true , address=45.91.116.8:4072 , piece_id=GEVTX6LLENPLN3UQLSGTFTP3DO52IRYUIQFIWC4XMOC3ZF6V4GTQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.486615595 (550h40m12.35769863s) [noise=true , address=77.90.31.12:4618 , piece_id=45WRZ26CPUODOAYF5B6RF37DNMDDQWWJUDK7BFKYS3K3LOCNYJCA, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.566542810 (550h40m12.437625845s) [noise=true , address=77.90.31.19:4859 , piece_id=A4HD3FPDPCCMHRFBTS3GNGB323RYD5C5AW4ZVSNOTXFI7FLIV37Q, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.609337578 (550h40m12.480420623s) [noise=true , address=77.90.31.13:4646 , piece_id=5CDMQF7GG5PDPHIPMFZWDBKWH5ANO3UIX257LJPCWRITNOA7GY4A, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.640856565 (550h40m12.511939611s) [noise=true , address=45.91.116.5:4322 , piece_id=SS6DOCSJHY5Y3DPWMYX45F4VYYXR44JDLYQ2YPY5XIIAYKMAG7LQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.645490329 (550h40m12.516573364s) [noise=true , address=77.90.31.18:4819 , piece_id=OYXPAN3EK4QRDEC6VHIDL3GSGEY27G72KII6HTGUMSPYLHCYBNEA, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:13.103710286 (550h40m12.974793331s) [noise=true , address=145.79.227.34:4911 , piece_id=24WE4EMZSL2JHQZ7PESCMLYCZOWGDTGJD2XNLUUJEFPSYLRI5E7A, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:13.209917141 (550h40m13.081000176s) [noise=true , address=145.79.227.44:4221 , piece_id=H72XCWLKYJAXYGGDIWN67H42RUIEOJCPUSWT5IJMFPKETH7LK3KA, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:13.218355468 (550h40m13.089438514s) [noise=true , address=77.90.31.11:4565 , piece_id=4SBLRZK3K4ED73PJ64BI6ANKBWB2AAHQA4SBUNH6CJ63GN3FJR2Q, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:13.255968741 (550h40m13.127051806s) [noise=true , address=145.79.227.30:4789 , piece_id=4QT5IUPIK7BOHJROEK2EBFTW434OKC75FZNENMYYCRMBZ7E5BUTA, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:13.255996102 (550h40m13.127079137s) [seg_pos={0 0}, upload=482805]: Segment reached optimal threshold of 14 pieces.
Dec 28 14:09:13.256214589 (550h40m13.127297634s) [noise=true , address=145.79.227.40:4053 , piece_id=RO4ZZQCKCC7T4T6SOT7GGTY626BJ6ESLQG6KATTPNIUM3ICUKHAQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DqyTc4MHvxutFXs2WWRCk8MSuxF3Y1HayyQY4Y5sy8cqDLWuG): protocol: expected piece hash; context canceled; EOF
Dec 28 14:09:13.256223686 (550h40m13.127306731s) [noise=true , address=145.79.227.40:4053 , piece_id=RO4ZZQCKCC7T4T6SOT7GGTY626BJ6ESLQG6KATTPNIUM3ICUKHAQ, seg_pos={0 0}, upload=482805]: Long tail cancellation due to optimal threshold
Dec 28 14:09:13.256512506 (550h40m13.127595551s) [noise=true , address=77.90.31.15:4704 , piece_id=DXWLFOIKOPE33LMEQECUJVIGRG3LPADUMULLWI4ZBHEOQHD43ULQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14tJRdZ1guwwsYwPf3LtENRWqYx2jBQpi7uAMMPaB32Gks9zmj): protocol: expected piece hash; context canceled; EOF
Dec 28 14:09:13.256520060 (550h40m13.127603105s) [noise=true , address=77.90.31.15:4704 , piece_id=DXWLFOIKOPE33LMEQECUJVIGRG3LPADUMULLWI4ZBHEOQHD43ULQ, seg_pos={0 0}, upload=482805]: Long tail cancellation due to optimal threshold
Dec 28 14:09:13.256649106 (550h40m13.127732151s) [noise=true , address=77.90.31.14:4678 , piece_id=VBGPSGMNDQZYY7UKRINAV6B3FVI5BPDPYS37FQIJCMLL65F3XOEQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12beu3SUZMPNWfbeHV4tLtx9UVZoEHh4Kt7oBhT59w3bGBE1ovP): protocol: write tcp 188.166.225.170:57374->77.90.31.14:4678: use of closed network connection; write tcp 188.166.225.170:57374->77.90.31.14:4678: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57374->77.90.31.14:4678: use of closed network connection
Dec 28 14:09:13.256655819 (550h40m13.127738864s) [noise=true , address=77.90.31.14:4678 , piece_id=VBGPSGMNDQZYY7UKRINAV6B3FVI5BPDPYS37FQIJCMLL65F3XOEQ, seg_pos={0 0}, upload=482805]: Long tail cancellation due to optimal threshold
Dec 28 14:09:13.256752974 (550h40m13.127836009s) [noise=true , address=64.49.13.169:4710 , piece_id=SANL7TVJFES63XD46OKQL4KIJ6YVHSEWPZKMRISEKC3ZKWQBOMIA, seg_pos={0 0}, upload=482805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:35680->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:35680->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35680->64.49.13.169:4710: use of closed network connection
Dec 28 14:09:13.256758825 (550h40m13.12784186s) [noise=true , address=64.49.13.169:4710 , piece_id=SANL7TVJFES63XD46OKQL4KIJ6YVHSEWPZKMRISEKC3ZKWQBOMIA, seg_pos={0 0}, upload=482805]: Long tail cancellation due to optimal threshold
Dec 28 14:09:13.256776549 (550h40m13.127859594s) [seg_pos={0 0}, upload=482805]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 14:09:13.256856591 (550h40m13.127939626s) [upload=482805]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 14:09:13.256862432 (550h40m13.127945467s) [upload=482805]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 14:09:13.550363717 (550h40m13.421446782s) [upload=482805]: Done uploading segments...
Dec 28 14:09:13.550405126 (550h40m13.421488161s) [upload=482805]: Upload finished. err: <nil>
Duration: 41.870901758s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [..........X.................................................] duration:
7.606078296s trace:
https://jaeger.dev.storj.io/trace/5fe9d6dbb79fbe44
FAIL (2025-12-28 14:15:33 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
67.204.205:4027 , piece_id=PAPNMRJ7E47VOMOXPBBDWKZJHLRMXTWVR5VW3J7KNQSSMHOHWOAQ, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.967737953 (550h8m4.838820988s) [noise=true , address=185.167.204.206:4072 , piece_id=WXORBDMA3KW7WZ67A4FJT6URJRFADUJ7X2PAHIGNG47T3TGMXVTQ, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.968193240 (550h8m4.839276285s) [noise=true , address=185.167.204.205:4030 , piece_id=R5DLI4ZTLOVGO56GPLJL5ZNHYSXOSYNJQHFIO4MVVNHWWBMEOV2Q, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.968780638 (550h8m4.839863683s) [noise=true , address=89.149.201.147:4019 , piece_id=MK7ORCLHO3DOWIOVSOWIQMPJ447PHRQGXP3UKITIRTYJSU5MPCCQ, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.969308924 (550h8m4.840391959s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.970088349 (550h8m4.841171404s) [noise=true , address=81.171.14.70:4039 , piece_id=EDHPQWLQZTP7MKPGEVMP7JXNTW7LFWSXCTRTSXPCBIQPROX76WRA, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.970325381 (550h8m4.841408426s) [noise=true , address=81.171.14.70:4036 , piece_id=CICNNV7JE6TBNC4RAPXKCEUSJAADOUVGSVIVQISM3ZB4COAZQDDA, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.970783813 (550h8m4.841866858s) [noise=true , address=81.171.14.70:4044 , piece_id=DGN74IG6IWAHLEMO6NLL4TJBXZ6ONTN6QUB6HXY6K7SSO3LYOYIQ, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.971205406 (550h8m4.842288441s) [noise=true , address=81.171.14.70:4048 , piece_id=TLT47ORGUKGF3MIIEAX7D22VR2FO6W6NL3VFJNHKWD4L3LIMFDLA, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.971606299 (550h8m4.842689344s) [noise=true , address=185.167.204.206:4067 , piece_id=VSKTG5RH32N7OEVWJGQVB6WJ72TEVF6DDK25W7NOH4RAXUP52LTQ, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.972162538 (550h8m4.843245583s) [noise=true , address=89.149.201.147:4002 , piece_id=G2PABNTDWCK4MKOZF6PONBMRNIXI45JTHJMAZFPEKPEJ3PQ7MTBA, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:05.795167814 (550h8m5.666250849s) [upload=482337]: Next returned nil segment
Dec 28 13:37:05.795184566 (550h8m5.666267611s) [upload=482337]: Waiting for error group managing segments...
Dec 28 13:37:06.178573801 (550h8m6.049656836s) [noise=true , address=185.167.204.206:4049 , piece_id=6RG2DZV6FPTIRE63ZNNST42TSWEBQ456YXIPPVYHIXV55GWFVJ4Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.183060835 (550h8m6.05414391s) [noise=true , address=185.167.204.206:4072 , piece_id=WXORBDMA3KW7WZ67A4FJT6URJRFADUJ7X2PAHIGNG47T3TGMXVTQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.183411202 (550h8m6.054494237s) [noise=true , address=185.167.204.206:4064 , piece_id=OG3G74OU23CEMKA54WQE3DOJ6AKAXBLZ23QWOQC4UEGQQN4JSMXQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.183791146 (550h8m6.054874181s) [noise=true , address=185.167.204.205:4029 , piece_id=6JQB6ZSCQZKWH3JWCQ4DY6BBAX73KC4Y53PRSMEMLWURZ5QWWQQQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.183973743 (550h8m6.055056778s) [noise=true , address=185.167.204.206:4053 , piece_id=YDH77S6JLKJ6IYS6DRUCELP5EJHIC2O44QBO7MIOEUYBML4GIBHA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.184465369 (550h8m6.055548414s) [noise=true , address=185.167.204.205:4008 , piece_id=RTUZPEXBB6JNSMC5YOFQQGDM57YD233W4756ZBQVMFNOVX6YKB4A, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.184501989 (550h8m6.055585024s) [noise=true , address=185.167.204.205:4030 , piece_id=R5DLI4ZTLOVGO56GPLJL5ZNHYSXOSYNJQHFIO4MVVNHWWBMEOV2Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.184856224 (550h8m6.055939259s) [noise=true , address=185.167.204.205:4009 , piece_id=V6RA7Y7FKY4DCPTRRF3BYNBALWDY5VTPXMDUXFE3353KUXLSQJEA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.185063518 (550h8m6.056146563s) [noise=true , address=185.167.204.205:4016 , piece_id=JB7SC3J2VR4U33HNQS2A2CA75THT3XLQCOIFHAJJEK7MQGY2SYAQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.185288617 (550h8m6.056371652s) [noise=true , address=185.167.204.205:4027 , piece_id=PAPNMRJ7E47VOMOXPBBDWKZJHLRMXTWVR5VW3J7KNQSSMHOHWOAQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.186241401 (550h8m6.057324436s) [noise=true , address=185.167.204.205:4024 , piece_id=FM5IMWRSJJ4RY72IX2OOYC54BFKL4FWE3UOAN3GJ26MLH345COVA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.193113658 (550h8m6.064196703s) [noise=true , address=185.167.204.205:4005 , piece_id=R4DFQJZBPJ5WJWHPODV6ILHUDIRBUSAFOMVNCLXB4UNMI6EXTKHA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.193791408 (550h8m6.064874453s) [noise=true , address=185.167.204.206:4068 , piece_id=DAI37M67FWCSVQTQOZ6RSSOJDGUVZO6SNGAHJJ44G3PGS4OJZEMA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.194292262 (550h8m6.065375307s) [noise=true , address=185.167.204.206:4046 , piece_id=VWLVHMBMJ63DQEWYBIKH3Y3NTVWFNW5OXZKMYXXBHVI3VWYXLLPQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.194302652 (550h8m6.065385687s) [noise=true , address=185.167.204.206:4067 , piece_id=VSKTG5RH32N7OEVWJGQVB6WJ72TEVF6DDK25W7NOH4RAXUP52LTQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.195359193 (550h8m6.066442228s) [noise=true , address=185.167.204.205:4012 , piece_id=LV6RUA5BWXG4DMSUVZDBGUAF5IBWUGMLJYUKNIXHRVDEBIOW4Q2Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.198423448 (550h8m6.069506493s) [noise=true , address=185.167.204.206:4070 , piece_id=5A7YYQYREIZRZDR7TRSE7XFW765JVAV7FX6LKWUI2OXAA74XN4CA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.888968408 (550h8m6.760051453s) [noise=true , address=81.171.14.70:4044 , piece_id=DGN74IG6IWAHLEMO6NLL4TJBXZ6ONTN6QUB6HXY6K7SSO3LYOYIQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.892350789 (550h8m6.763433834s) [noise=true , address=81.171.14.70:4039 , piece_id=EDHPQWLQZTP7MKPGEVMP7JXNTW7LFWSXCTRTSXPCBIQPROX76WRA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.904477160 (550h8m6.775560195s) [noise=true , address=81.171.14.70:4040 , piece_id=DYF4GSZBAKQ3H63LTVCMEFWMABY3FXHOJCNVIFYRKLF2J6B52K5A, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.128088281 (550h8m6.999171326s) [noise=true , address=81.171.14.70:4036 , piece_id=CICNNV7JE6TBNC4RAPXKCEUSJAADOUVGSVIVQISM3ZB4COAZQDDA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.430242142 (550h8m7.301325177s) [noise=true , address=89.149.201.147:4009 , piece_id=EVW2NQFYU57PC2IIVXLYIEBMYXJNIC2CLCHW6PS6NYIPCFRUVHDA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.505304332 (550h8m7.376387367s) [noise=true , address=89.149.201.147:4019 , piece_id=MK7ORCLHO3DOWIOVSOWIQMPJ447PHRQGXP3UKITIRTYJSU5MPCCQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.575461543 (550h8m7.446544609s) [noise=true , address=81.171.14.70:4048 , piece_id=TLT47ORGUKGF3MIIEAX7D22VR2FO6W6NL3VFJNHKWD4L3LIMFDLA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.599667265 (550h8m7.4707503s) [noise=true , address=89.149.201.147:4023 , piece_id=5IINX2GVRFLRX2QH24OO2DDANOKOTXTDR6AGVJBGSJ7IAZQ3CASA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.928390728 (550h8m7.799473773s) [noise=true , address=89.149.201.147:4002 , piece_id=G2PABNTDWCK4MKOZF6PONBMRNIXI45JTHJMAZFPEKPEJ3PQ7MTBA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.928417148 (550h8m7.799500193s) [seg_pos={0 0}, upload=482337]: Segment reached optimal threshold of 26 pieces.
Dec 28 13:37:07.928656303 (550h8m7.799739348s) [noise=true , address=81.171.14.70:4030 , piece_id=74YDIKUST7X3JBXVQHVLFHIKH5A4DYH5TNL3I4XUA5A6AHQO2DRA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZoRwa8wLw66Pjumy1ybp6wrZ79RrSU55uYZpwaY3xi95iSYde): protocol: expected piece hash; context canceled; EOF
Dec 28 13:37:07.928664990 (550h8m7.799748035s) [noise=true , address=81.171.14.70:4030 , piece_id=74YDIKUST7X3JBXVQHVLFHIKH5A4DYH5TNL3I4XUA5A6AHQO2DRA, seg_pos={0 0}, upload=482337]: Long tail cancellation due to optimal threshold
Dec 28 13:37:07.928723351 (550h8m7.799806386s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wgtphHKaTejC1qrJ98sDysBwtj2Zeqw2c5UFk1HMqfY8Buvr): protocol: expected piece hash; context canceled; EOF
Dec 28 13:37:07.928733922 (550h8m7.799816957s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Long tail cancellation due to optimal threshold
Dec 28 13:37:07.928754921 (550h8m7.799837966s) [seg_pos={0 0}, upload=482337]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 28 13:37:07.928870170 (550h8m7.799953215s) [upload=482337]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 13:37:07.928878627 (550h8m7.799961672s) [upload=482337]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 13:37:08.166321536 (550h8m8.037404571s) [upload=482337]: Done uploading segments...
Dec 28 13:37:08.166341444 (550h8m8.037424479s) [upload=482337]: Upload finished. err: <nil>
Duration: 7.606105566s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [............X....................X...X........X.............] duration:
17.224535315s trace:
https://jaeger.dev.storj.io/trace/360e059013858fb1
FAIL (2025-12-28 14:15:29 UTC)
orbiter warning: context deadline exceeded
Duration: 17.224571374s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
20 out of 60 is failed (threshold=3) [..X.....X..X.X...XX......X......XXX.XX....XX......XX.XX.X.X.] duration:
6.044582566s trace:
https://jaeger.dev.storj.io/trace/5be1acce1690abbe
FAIL (2025-12-28 14:15:11 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 28 14:14:05.707801185 (550h45m5.57888423s) [upload=482879]: Starting upload
Dec 28 14:14:05.707886898 (550h45m5.578969944s) [upload=482879]: Done starting upload
Dec 28 14:14:05.707951642 (550h45m5.579034687s) [upload=482879]: Uploading segments...
Dec 28 14:14:05.708074515 (550h45m5.57915756s) [upload=482879]: Got next segment. Inline: false
Dec 28 14:14:05.708106106 (550h45m5.579189151s) [upload=482879]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 14:14:05.708110434 (550h45m5.579193479s) [upload=482879]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 14:14:06.000535780 (550h45m5.871618845s) [seg_pos={0 0}, upload=482879]: Begin upload segment...
Dec 28 14:14:06.000598910 (550h45m5.871681945s) [seg_pos={0 0}, upload=482879]: Done begin upload segment.
Dec 28 14:14:06.000636912 (550h45m5.871719957s) [noise=true , address=145.79.227.40:4051 , piece_id=3QYIHWAQR7G5I57JDXJ4SJ6A2LARGCAP6XK7CVE6K2WVR4XGXXHQ, seg_pos={0 0}, upload=482879]: Uploading piece...
Dec 28 14:14:06.000757032 (550h45m5.871840097s) [noise=true , address=145.79.227.13:4804 , piece_id=PGTARYC3LNMTR4TAGR74FPBM7RHMTHRLDW2QUW6ARA2SI26DHLZQ, seg_pos={0 0}, upload=482879]: Uploading piece...
Dec 28 14:14:06.001400927 (550h45m5.872483972s) [noise=true , address=45.91.116.9:4262 , piece_id=54C5OVTRQ2FAS4X2EEPNRRAKFEWDO5JQEDQ5HOCOTPJW54PUK2YA, seg_pos={0 0}, upload=482879]: Uploading piece...
Dec 28 14:14:06.001895939 (550h45m5.872978975s) [noise=true , address=145.79.227.47:4266 , piece_id=FUBWPNHK7SKGUJSKNMTHO3RGUCTABSL4QOASCW23J6GCJ3H5LDMQ, seg_pos={0 0}, upload=482879]: Uploading piece...
Dec 28 14:14:06.002437291 (550h45m5.873520336s) [noise=true , address=45.91.116.14:4190 , piece_id=565ADKSRT363WQJIB2WFNFLNPS765AVU7R72A5LMHGACOYZNBYDQ, seg_pos={0 0}, upload=482879]: Uploading piece...
Dec 28 14:14:06.002952352 (550h45m5.874035397s) [noise=true , address=45.91.116.6:4975 , piece_id=IRL2CQSLRGQJIP3BHV7HYBS56R7HGUDQPQBNNFXEE6Q7IUF2CXHA, seg_pos={0 0}, upload=482879]: Uploading piece...
Dec 28 14:14:06.002965867 (550h45m5.874048912s) [noise=true , address=145.79.227.42:4123 , piece_id=7QUYG56BEL7FSD2A6AHQNONXE4PWH3QEHZSBKEGYKXYRH5DKIP5Q, seg_pos={0 0}, upload=482879]: Uploading piece...
Dec 28 14:14:06.003387630 (550h45m5.874470685s) [noise=true , address=45.91.116.11:4120 , piece_id=MRG7VL7KTA3T5FEUFJW5ISMKDWWLUGHHBGGL7RMWRSON4N2UVZUA, seg_pos={0 0}, upload=482879]: Uploading piece...
Dec 28 14:14:06.003838088 (550h45m5.874921133s) [noise=true , address=45.91.116.12:4228 , piece_id=VXDU5ZAC3NCP65MDUZ5LSVEIGRQDJIJDP2TIUGXPHGLBLGBMMCWA, seg_pos={0 0}, upload=482879]: Uploading piece...
Dec 28 14:14:06.004305238 (550h45m5.875388303s) [noise=true , address=145.79.227.50:4373 , piece_id=RAXJ6LT3KF7DOQRSV7DH5IAPJ4WMWDLIPIOFFVQUWRU5BF4SERNQ, seg_pos={0 0}, upload=482879]: Uploading piece...
Dec 28 14:14:06.004804198 (550h45m5.875887233s) [noise=true , address=145.79.227.46:4603 , piece_id=TWVG57KDN3OV4DU3QD4ONWNEEYOCKYXYQ63QBQPEZS676HDAMZYA, seg_pos={0 0}, upload=482879]: Uploading piece...
Dec 28 14:14:06.005276297 (550h45m5.876359342s) [noise=true , address=145.79.227.12:4767 , piece_id=VPWI5VMVMUCTRSYXD3QSZ6OUETWIAJQLGZQ7MRSUL2Y475DNCK3Q, seg_pos={0 0}, upload=482879]: Uploading piece...
Dec 28 14:14:06.005723668 (550h45m5.876806713s) [noise=true , address=145.79.227.52:4446 , piece_id=L2LK33S7SEF7NTBUT3AX7XMTKPGPICTAXSS6MYA2OJ3DOZWRJJHA, seg_pos={0 0}, upload=482879]: Uploading piece...
Dec 28 14:14:06.006312079 (550h45m5.877395124s) [noise=true , address=77.90.31.21:4880 , piece_id=2EP7PPWFUVNZLRRIGSC7GW53WGRS2XCZNP7O5BDDNHENMJBISNZA, seg_pos={0 0}, upload=482879]: Uploading piece...
Dec 28 14:14:06.006896762 (550h45m5.877979807s) [noise=true , address=145.79.227.37:4566 , piece_id=5ICU5SETRGGIIKRNDRCTUUC2VTSUGRD6PWQH7BTZYVYD7S3SQSQQ, seg_pos={0 0}, upload=482879]: Uploading piece...
Dec 28 14:14:06.007257900 (550h45m5.878340935s) [noise=true , address=145.79.227.20:4439 , piece_id=WC36XT6DAV27DTUCFKYSU3AWZLWZ5EM37JQH3SZPSWOUOE4PR5HQ, seg_pos={0 0}, upload=482879]: Uploading piece...
Dec 28 14:14:07.721543183 (550h45m7.592626218s) [upload=482879]: Next returned nil segment
Dec 28 14:14:07.721554704 (550h45m7.59263775s) [upload=482879]: Waiting for error group managing segments...
Dec 28 14:14:07.997311191 (550h45m7.868394236s) [noise=true , address=145.79.227.13:4804 , piece_id=PGTARYC3LNMTR4TAGR74FPBM7RHMTHRLDW2QUW6ARA2SI26DHLZQ, seg_pos={0 0}, upload=482879]: Done uploading piece. err: <nil>
Dec 28 14:14:08.005357442 (550h45m7.876440507s) [noise=true , address=145.79.227.12:4767 , piece_id=VPWI5VMVMUCTRSYXD3QSZ6OUETWIAJQLGZQ7MRSUL2Y475DNCK3Q, seg_pos={0 0}, upload=482879]: Done uploading piece. err: <nil>
Dec 28 14:14:08.008351083 (550h45m7.879434118s) [noise=true , address=145.79.227.52:4446 , piece_id=L2LK33S7SEF7NTBUT3AX7XMTKPGPICTAXSS6MYA2OJ3DOZWRJJHA, seg_pos={0 0}, upload=482879]: Done uploading piece. err: <nil>
Dec 28 14:14:08.010815325 (550h45m7.88189837s) [noise=true , address=145.79.227.50:4373 , piece_id=RAXJ6LT3KF7DOQRSV7DH5IAPJ4WMWDLIPIOFFVQUWRU5BF4SERNQ, seg_pos={0 0}, upload=482879]: Done uploading piece. err: <nil>
Dec 28 14:14:08.014508277 (550h45m7.885591352s) [noise=true , address=145.79.227.42:4123 , piece_id=7QUYG56BEL7FSD2A6AHQNONXE4PWH3QEHZSBKEGYKXYRH5DKIP5Q, seg_pos={0 0}, upload=482879]: Done uploading piece. err: <nil>
Dec 28 14:14:08.015107678 (550h45m7.886190723s) [noise=true , address=145.79.227.20:4439 , piece_id=WC36XT6DAV27DTUCFKYSU3AWZLWZ5EM37JQH3SZPSWOUOE4PR5HQ, seg_pos={0 0}, upload=482879]: Done uploading piece. err: <nil>
Dec 28 14:14:08.020768196 (550h45m7.891851231s) [noise=true , address=145.79.227.46:4603 , piece_id=TWVG57KDN3OV4DU3QD4ONWNEEYOCKYXYQ63QBQPEZS676HDAMZYA, seg_pos={0 0}, upload=482879]: Done uploading piece. err: <nil>
Dec 28 14:14:08.022418348 (550h45m7.893501383s) [noise=true , address=145.79.227.47:4266 , piece_id=FUBWPNHK7SKGUJSKNMTHO3RGUCTABSL4QOASCW23J6GCJ3H5LDMQ, seg_pos={0 0}, upload=482879]: Done uploading piece. err: <nil>
Dec 28 14:14:08.065573974 (550h45m7.936657019s) [noise=true , address=145.79.227.37:4566 , piece_id=5ICU5SETRGGIIKRNDRCTUUC2VTSUGRD6PWQH7BTZYVYD7S3SQSQQ, seg_pos={0 0}, upload=482879]: Done uploading piece. err: <nil>
Dec 28 14:14:08.252250557 (550h45m8.123333602s) [noise=true , address=45.91.116.6:4975 , piece_id=IRL2CQSLRGQJIP3BHV7HYBS56R7HGUDQPQBNNFXEE6Q7IUF2CXHA, seg_pos={0 0}, upload=482879]: Done uploading piece. err: <nil>
Dec 28 14:14:08.253942699 (550h45m8.125025734s) [noise=true , address=45.91.116.11:4120 , piece_id=MRG7VL7KTA3T5FEUFJW5ISMKDWWLUGHHBGGL7RMWRSON4N2UVZUA, seg_pos={0 0}, upload=482879]: Done uploading piece. err: <nil>
Dec 28 14:14:09.023479484 (550h45m8.894562519s) [noise=true , address=45.91.116.14:4190 , piece_id=565ADKSRT363WQJIB2WFNFLNPS765AVU7R72A5LMHGACOYZNBYDQ, seg_pos={0 0}, upload=482879]: Done uploading piece. err: <nil>
Dec 28 14:14:09.024215435 (550h45m8.89529848s) [noise=true , address=45.91.116.9:4262 , piece_id=54C5OVTRQ2FAS4X2EEPNRRAKFEWDO5JQEDQ5HOCOTPJW54PUK2YA, seg_pos={0 0}, upload=482879]: Done uploading piece. err: <nil>
Dec 28 14:14:09.024226005 (550h45m8.89530905s) [seg_pos={0 0}, upload=482879]: Segment reached optimal threshold of 13 pieces.
Dec 28 14:14:09.024424473 (550h45m8.895507508s) [noise=true , address=77.90.31.21:4880 , piece_id=2EP7PPWFUVNZLRRIGSC7GW53WGRS2XCZNP7O5BDDNHENMJBISNZA, seg_pos={0 0}, upload=482879]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF
Dec 28 14:14:09.024431356 (550h45m8.895514391s) [noise=true , address=77.90.31.21:4880 , piece_id=2EP7PPWFUVNZLRRIGSC7GW53WGRS2XCZNP7O5BDDNHENMJBISNZA, seg_pos={0 0}, upload=482879]: Long tail cancellation due to optimal threshold
Dec 28 14:14:09.024575721 (550h45m8.895658756s) [noise=true , address=145.79.227.40:4051 , piece_id=3QYIHWAQR7G5I57JDXJ4SJ6A2LARGCAP6XK7CVE6K2WVR4XGXXHQ, seg_pos={0 0}, upload=482879]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF
Dec 28 14:14:09.024582594 (550h45m8.895665639s) [noise=true , address=145.79.227.40:4051 , piece_id=3QYIHWAQR7G5I57JDXJ4SJ6A2LARGCAP6XK7CVE6K2WVR4XGXXHQ, seg_pos={0 0}, upload=482879]: Long tail cancellation due to optimal threshold
Dec 28 14:14:09.024595309 (550h45m8.895678354s) [noise=true , address=45.91.116.12:4228 , piece_id=VXDU5ZAC3NCP65MDUZ5LSVEIGRQDJIJDP2TIUGXPHGLBLGBMMCWA, seg_pos={0 0}, upload=482879]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:55860->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:55860->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55860->45.91.116.12:4228: use of closed network connection
Dec 28 14:14:09.024622490 (550h45m8.895705555s) [noise=true , address=45.91.116.12:4228 , piece_id=VXDU5ZAC3NCP65MDUZ5LSVEIGRQDJIJDP2TIUGXPHGLBLGBMMCWA, seg_pos={0 0}, upload=482879]: Long tail cancellation due to optimal threshold
Dec 28 14:14:09.024649401 (550h45m8.895732436s) [seg_pos={0 0}, upload=482879]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 14:14:09.024766164 (550h45m8.895849199s) [upload=482879]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 14:14:09.024773017 (550h45m8.895856062s) [upload=482879]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 14:14:09.321695907 (550h45m9.192778942s) [upload=482879]: Done uploading segments...
Dec 28 14:14:09.321715164 (550h45m9.192798199s) [upload=482879]: Upload finished. err: <nil>
Duration: 6.044611181s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [X............X....................X...X........X............] duration:
18.082666691s trace:
https://jaeger.dev.storj.io/trace/360e059013858fb1
FAIL (2025-12-28 14:15:06 UTC)
orbiter warning: context deadline exceeded
Duration: 18.082754849s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [.................................................X...X......] duration:
20.586775326s trace:
https://jaeger.dev.storj.io/trace/102cf391e061ae77
FAIL (2025-12-28 14:14:58 UTC)
orbiter warning: context deadline exceeded
Duration: 20.58679805s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
53 out of 60 is failed (threshold=6) [XXXXXXX.XXXXXXXXXXXX.XXXXXXX.XX..XXXXXXXXXXXXXXXXXXXXX.X.XXX] duration:
5.000974634s trace:
https://jaeger.dev.storj.io/trace/46e96cb0f745b38d
FAIL (2025-12-28 14:14:40 UTC)
orbiter warning: context deadline exceeded
Duration: 5.000996795s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [...........X................................................] duration:
7.527342799s trace:
https://jaeger.dev.storj.io/trace/5fe9d6dbb79fbe44
FAIL (2025-12-28 14:14:38 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
67.204.205:4027 , piece_id=PAPNMRJ7E47VOMOXPBBDWKZJHLRMXTWVR5VW3J7KNQSSMHOHWOAQ, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.967737953 (550h8m4.838820988s) [noise=true , address=185.167.204.206:4072 , piece_id=WXORBDMA3KW7WZ67A4FJT6URJRFADUJ7X2PAHIGNG47T3TGMXVTQ, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.968193240 (550h8m4.839276285s) [noise=true , address=185.167.204.205:4030 , piece_id=R5DLI4ZTLOVGO56GPLJL5ZNHYSXOSYNJQHFIO4MVVNHWWBMEOV2Q, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.968780638 (550h8m4.839863683s) [noise=true , address=89.149.201.147:4019 , piece_id=MK7ORCLHO3DOWIOVSOWIQMPJ447PHRQGXP3UKITIRTYJSU5MPCCQ, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.969308924 (550h8m4.840391959s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.970088349 (550h8m4.841171404s) [noise=true , address=81.171.14.70:4039 , piece_id=EDHPQWLQZTP7MKPGEVMP7JXNTW7LFWSXCTRTSXPCBIQPROX76WRA, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.970325381 (550h8m4.841408426s) [noise=true , address=81.171.14.70:4036 , piece_id=CICNNV7JE6TBNC4RAPXKCEUSJAADOUVGSVIVQISM3ZB4COAZQDDA, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.970783813 (550h8m4.841866858s) [noise=true , address=81.171.14.70:4044 , piece_id=DGN74IG6IWAHLEMO6NLL4TJBXZ6ONTN6QUB6HXY6K7SSO3LYOYIQ, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.971205406 (550h8m4.842288441s) [noise=true , address=81.171.14.70:4048 , piece_id=TLT47ORGUKGF3MIIEAX7D22VR2FO6W6NL3VFJNHKWD4L3LIMFDLA, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.971606299 (550h8m4.842689344s) [noise=true , address=185.167.204.206:4067 , piece_id=VSKTG5RH32N7OEVWJGQVB6WJ72TEVF6DDK25W7NOH4RAXUP52LTQ, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.972162538 (550h8m4.843245583s) [noise=true , address=89.149.201.147:4002 , piece_id=G2PABNTDWCK4MKOZF6PONBMRNIXI45JTHJMAZFPEKPEJ3PQ7MTBA, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:05.795167814 (550h8m5.666250849s) [upload=482337]: Next returned nil segment
Dec 28 13:37:05.795184566 (550h8m5.666267611s) [upload=482337]: Waiting for error group managing segments...
Dec 28 13:37:06.178573801 (550h8m6.049656836s) [noise=true , address=185.167.204.206:4049 , piece_id=6RG2DZV6FPTIRE63ZNNST42TSWEBQ456YXIPPVYHIXV55GWFVJ4Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.183060835 (550h8m6.05414391s) [noise=true , address=185.167.204.206:4072 , piece_id=WXORBDMA3KW7WZ67A4FJT6URJRFADUJ7X2PAHIGNG47T3TGMXVTQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.183411202 (550h8m6.054494237s) [noise=true , address=185.167.204.206:4064 , piece_id=OG3G74OU23CEMKA54WQE3DOJ6AKAXBLZ23QWOQC4UEGQQN4JSMXQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.183791146 (550h8m6.054874181s) [noise=true , address=185.167.204.205:4029 , piece_id=6JQB6ZSCQZKWH3JWCQ4DY6BBAX73KC4Y53PRSMEMLWURZ5QWWQQQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.183973743 (550h8m6.055056778s) [noise=true , address=185.167.204.206:4053 , piece_id=YDH77S6JLKJ6IYS6DRUCELP5EJHIC2O44QBO7MIOEUYBML4GIBHA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.184465369 (550h8m6.055548414s) [noise=true , address=185.167.204.205:4008 , piece_id=RTUZPEXBB6JNSMC5YOFQQGDM57YD233W4756ZBQVMFNOVX6YKB4A, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.184501989 (550h8m6.055585024s) [noise=true , address=185.167.204.205:4030 , piece_id=R5DLI4ZTLOVGO56GPLJL5ZNHYSXOSYNJQHFIO4MVVNHWWBMEOV2Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.184856224 (550h8m6.055939259s) [noise=true , address=185.167.204.205:4009 , piece_id=V6RA7Y7FKY4DCPTRRF3BYNBALWDY5VTPXMDUXFE3353KUXLSQJEA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.185063518 (550h8m6.056146563s) [noise=true , address=185.167.204.205:4016 , piece_id=JB7SC3J2VR4U33HNQS2A2CA75THT3XLQCOIFHAJJEK7MQGY2SYAQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.185288617 (550h8m6.056371652s) [noise=true , address=185.167.204.205:4027 , piece_id=PAPNMRJ7E47VOMOXPBBDWKZJHLRMXTWVR5VW3J7KNQSSMHOHWOAQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.186241401 (550h8m6.057324436s) [noise=true , address=185.167.204.205:4024 , piece_id=FM5IMWRSJJ4RY72IX2OOYC54BFKL4FWE3UOAN3GJ26MLH345COVA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.193113658 (550h8m6.064196703s) [noise=true , address=185.167.204.205:4005 , piece_id=R4DFQJZBPJ5WJWHPODV6ILHUDIRBUSAFOMVNCLXB4UNMI6EXTKHA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.193791408 (550h8m6.064874453s) [noise=true , address=185.167.204.206:4068 , piece_id=DAI37M67FWCSVQTQOZ6RSSOJDGUVZO6SNGAHJJ44G3PGS4OJZEMA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.194292262 (550h8m6.065375307s) [noise=true , address=185.167.204.206:4046 , piece_id=VWLVHMBMJ63DQEWYBIKH3Y3NTVWFNW5OXZKMYXXBHVI3VWYXLLPQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.194302652 (550h8m6.065385687s) [noise=true , address=185.167.204.206:4067 , piece_id=VSKTG5RH32N7OEVWJGQVB6WJ72TEVF6DDK25W7NOH4RAXUP52LTQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.195359193 (550h8m6.066442228s) [noise=true , address=185.167.204.205:4012 , piece_id=LV6RUA5BWXG4DMSUVZDBGUAF5IBWUGMLJYUKNIXHRVDEBIOW4Q2Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.198423448 (550h8m6.069506493s) [noise=true , address=185.167.204.206:4070 , piece_id=5A7YYQYREIZRZDR7TRSE7XFW765JVAV7FX6LKWUI2OXAA74XN4CA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.888968408 (550h8m6.760051453s) [noise=true , address=81.171.14.70:4044 , piece_id=DGN74IG6IWAHLEMO6NLL4TJBXZ6ONTN6QUB6HXY6K7SSO3LYOYIQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.892350789 (550h8m6.763433834s) [noise=true , address=81.171.14.70:4039 , piece_id=EDHPQWLQZTP7MKPGEVMP7JXNTW7LFWSXCTRTSXPCBIQPROX76WRA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.904477160 (550h8m6.775560195s) [noise=true , address=81.171.14.70:4040 , piece_id=DYF4GSZBAKQ3H63LTVCMEFWMABY3FXHOJCNVIFYRKLF2J6B52K5A, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.128088281 (550h8m6.999171326s) [noise=true , address=81.171.14.70:4036 , piece_id=CICNNV7JE6TBNC4RAPXKCEUSJAADOUVGSVIVQISM3ZB4COAZQDDA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.430242142 (550h8m7.301325177s) [noise=true , address=89.149.201.147:4009 , piece_id=EVW2NQFYU57PC2IIVXLYIEBMYXJNIC2CLCHW6PS6NYIPCFRUVHDA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.505304332 (550h8m7.376387367s) [noise=true , address=89.149.201.147:4019 , piece_id=MK7ORCLHO3DOWIOVSOWIQMPJ447PHRQGXP3UKITIRTYJSU5MPCCQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.575461543 (550h8m7.446544609s) [noise=true , address=81.171.14.70:4048 , piece_id=TLT47ORGUKGF3MIIEAX7D22VR2FO6W6NL3VFJNHKWD4L3LIMFDLA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.599667265 (550h8m7.4707503s) [noise=true , address=89.149.201.147:4023 , piece_id=5IINX2GVRFLRX2QH24OO2DDANOKOTXTDR6AGVJBGSJ7IAZQ3CASA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.928390728 (550h8m7.799473773s) [noise=true , address=89.149.201.147:4002 , piece_id=G2PABNTDWCK4MKOZF6PONBMRNIXI45JTHJMAZFPEKPEJ3PQ7MTBA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.928417148 (550h8m7.799500193s) [seg_pos={0 0}, upload=482337]: Segment reached optimal threshold of 26 pieces.
Dec 28 13:37:07.928656303 (550h8m7.799739348s) [noise=true , address=81.171.14.70:4030 , piece_id=74YDIKUST7X3JBXVQHVLFHIKH5A4DYH5TNL3I4XUA5A6AHQO2DRA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZoRwa8wLw66Pjumy1ybp6wrZ79RrSU55uYZpwaY3xi95iSYde): protocol: expected piece hash; context canceled; EOF
Dec 28 13:37:07.928664990 (550h8m7.799748035s) [noise=true , address=81.171.14.70:4030 , piece_id=74YDIKUST7X3JBXVQHVLFHIKH5A4DYH5TNL3I4XUA5A6AHQO2DRA, seg_pos={0 0}, upload=482337]: Long tail cancellation due to optimal threshold
Dec 28 13:37:07.928723351 (550h8m7.799806386s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wgtphHKaTejC1qrJ98sDysBwtj2Zeqw2c5UFk1HMqfY8Buvr): protocol: expected piece hash; context canceled; EOF
Dec 28 13:37:07.928733922 (550h8m7.799816957s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Long tail cancellation due to optimal threshold
Dec 28 13:37:07.928754921 (550h8m7.799837966s) [seg_pos={0 0}, upload=482337]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 28 13:37:07.928870170 (550h8m7.799953215s) [upload=482337]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 13:37:07.928878627 (550h8m7.799961672s) [upload=482337]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 13:37:08.166321536 (550h8m8.037404571s) [upload=482337]: Done uploading segments...
Dec 28 13:37:08.166341444 (550h8m8.037424479s) [upload=482337]: Upload finished. err: <nil>
Duration: 7.5273705s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [..................................................X...X.....] duration:
23.241084707s trace:
https://jaeger.dev.storj.io/trace/102cf391e061ae77
FAIL (2025-12-28 14:14:37 UTC)
orbiter warning: context deadline exceeded
Duration: 23.241113641s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
21 out of 60 is failed (threshold=3) [X..X.....X..X.X...XX......X......XXX.XX....XX......XX.XX.X.X] duration:
30.002879072s trace:
https://jaeger.dev.storj.io/trace/5be1acce1690abbe
FAIL (2025-12-28 14:14:34 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 28 14:14:05.707801185 (550h45m5.57888423s) [upload=482879]: Starting upload
Dec 28 14:14:05.707886898 (550h45m5.578969944s) [upload=482879]: Done starting upload
Dec 28 14:14:05.707951642 (550h45m5.579034687s) [upload=482879]: Uploading segments...
Dec 28 14:14:05.708074515 (550h45m5.57915756s) [upload=482879]: Got next segment. Inline: false
Dec 28 14:14:05.708106106 (550h45m5.579189151s) [upload=482879]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 14:14:05.708110434 (550h45m5.579193479s) [upload=482879]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 14:14:06.000535780 (550h45m5.871618845s) [seg_pos={0 0}, upload=482879]: Begin upload segment...
Dec 28 14:14:06.000598910 (550h45m5.871681945s) [seg_pos={0 0}, upload=482879]: Done begin upload segment.
Dec 28 14:14:06.000636912 (550h45m5.871719957s) [noise=true , address=145.79.227.40:4051 , piece_id=3QYIHWAQR7G5I57JDXJ4SJ6A2LARGCAP6XK7CVE6K2WVR4XGXXHQ, seg_pos={0 0}, upload=482879]: Uploading piece...
Dec 28 14:14:06.000757032 (550h45m5.871840097s) [noise=true , address=145.79.227.13:4804 , piece_id=PGTARYC3LNMTR4TAGR74FPBM7RHMTHRLDW2QUW6ARA2SI26DHLZQ, seg_pos={0 0}, upload=482879]: Uploading piece...
Dec 28 14:14:06.001400927 (550h45m5.872483972s) [noise=true , address=45.91.116.9:4262 , piece_id=54C5OVTRQ2FAS4X2EEPNRRAKFEWDO5JQEDQ5HOCOTPJW54PUK2YA, seg_pos={0 0}, upload=482879]: Uploading piece...
Dec 28 14:14:06.001895939 (550h45m5.872978975s) [noise=true , address=145.79.227.47:4266 , piece_id=FUBWPNHK7SKGUJSKNMTHO3RGUCTABSL4QOASCW23J6GCJ3H5LDMQ, seg_pos={0 0}, upload=482879]: Uploading piece...
Dec 28 14:14:06.002437291 (550h45m5.873520336s) [noise=true , address=45.91.116.14:4190 , piece_id=565ADKSRT363WQJIB2WFNFLNPS765AVU7R72A5LMHGACOYZNBYDQ, seg_pos={0 0}, upload=482879]: Uploading piece...
Dec 28 14:14:06.002952352 (550h45m5.874035397s) [noise=true , address=45.91.116.6:4975 , piece_id=IRL2CQSLRGQJIP3BHV7HYBS56R7HGUDQPQBNNFXEE6Q7IUF2CXHA, seg_pos={0 0}, upload=482879]: Uploading piece...
Dec 28 14:14:06.002965867 (550h45m5.874048912s) [noise=true , address=145.79.227.42:4123 , piece_id=7QUYG56BEL7FSD2A6AHQNONXE4PWH3QEHZSBKEGYKXYRH5DKIP5Q, seg_pos={0 0}, upload=482879]: Uploading piece...
Dec 28 14:14:06.003387630 (550h45m5.874470685s) [noise=true , address=45.91.116.11:4120 , piece_id=MRG7VL7KTA3T5FEUFJW5ISMKDWWLUGHHBGGL7RMWRSON4N2UVZUA, seg_pos={0 0}, upload=482879]: Uploading piece...
Dec 28 14:14:06.003838088 (550h45m5.874921133s) [noise=true , address=45.91.116.12:4228 , piece_id=VXDU5ZAC3NCP65MDUZ5LSVEIGRQDJIJDP2TIUGXPHGLBLGBMMCWA, seg_pos={0 0}, upload=482879]: Uploading piece...
Dec 28 14:14:06.004305238 (550h45m5.875388303s) [noise=true , address=145.79.227.50:4373 , piece_id=RAXJ6LT3KF7DOQRSV7DH5IAPJ4WMWDLIPIOFFVQUWRU5BF4SERNQ, seg_pos={0 0}, upload=482879]: Uploading piece...
Dec 28 14:14:06.004804198 (550h45m5.875887233s) [noise=true , address=145.79.227.46:4603 , piece_id=TWVG57KDN3OV4DU3QD4ONWNEEYOCKYXYQ63QBQPEZS676HDAMZYA, seg_pos={0 0}, upload=482879]: Uploading piece...
Dec 28 14:14:06.005276297 (550h45m5.876359342s) [noise=true , address=145.79.227.12:4767 , piece_id=VPWI5VMVMUCTRSYXD3QSZ6OUETWIAJQLGZQ7MRSUL2Y475DNCK3Q, seg_pos={0 0}, upload=482879]: Uploading piece...
Dec 28 14:14:06.005723668 (550h45m5.876806713s) [noise=true , address=145.79.227.52:4446 , piece_id=L2LK33S7SEF7NTBUT3AX7XMTKPGPICTAXSS6MYA2OJ3DOZWRJJHA, seg_pos={0 0}, upload=482879]: Uploading piece...
Dec 28 14:14:06.006312079 (550h45m5.877395124s) [noise=true , address=77.90.31.21:4880 , piece_id=2EP7PPWFUVNZLRRIGSC7GW53WGRS2XCZNP7O5BDDNHENMJBISNZA, seg_pos={0 0}, upload=482879]: Uploading piece...
Dec 28 14:14:06.006896762 (550h45m5.877979807s) [noise=true , address=145.79.227.37:4566 , piece_id=5ICU5SETRGGIIKRNDRCTUUC2VTSUGRD6PWQH7BTZYVYD7S3SQSQQ, seg_pos={0 0}, upload=482879]: Uploading piece...
Dec 28 14:14:06.007257900 (550h45m5.878340935s) [noise=true , address=145.79.227.20:4439 , piece_id=WC36XT6DAV27DTUCFKYSU3AWZLWZ5EM37JQH3SZPSWOUOE4PR5HQ, seg_pos={0 0}, upload=482879]: Uploading piece...
Dec 28 14:14:07.721543183 (550h45m7.592626218s) [upload=482879]: Next returned nil segment
Dec 28 14:14:07.721554704 (550h45m7.59263775s) [upload=482879]: Waiting for error group managing segments...
Dec 28 14:14:07.997311191 (550h45m7.868394236s) [noise=true , address=145.79.227.13:4804 , piece_id=PGTARYC3LNMTR4TAGR74FPBM7RHMTHRLDW2QUW6ARA2SI26DHLZQ, seg_pos={0 0}, upload=482879]: Done uploading piece. err: <nil>
Dec 28 14:14:08.005357442 (550h45m7.876440507s) [noise=true , address=145.79.227.12:4767 , piece_id=VPWI5VMVMUCTRSYXD3QSZ6OUETWIAJQLGZQ7MRSUL2Y475DNCK3Q, seg_pos={0 0}, upload=482879]: Done uploading piece. err: <nil>
Dec 28 14:14:08.008351083 (550h45m7.879434118s) [noise=true , address=145.79.227.52:4446 , piece_id=L2LK33S7SEF7NTBUT3AX7XMTKPGPICTAXSS6MYA2OJ3DOZWRJJHA, seg_pos={0 0}, upload=482879]: Done uploading piece. err: <nil>
Dec 28 14:14:08.010815325 (550h45m7.88189837s) [noise=true , address=145.79.227.50:4373 , piece_id=RAXJ6LT3KF7DOQRSV7DH5IAPJ4WMWDLIPIOFFVQUWRU5BF4SERNQ, seg_pos={0 0}, upload=482879]: Done uploading piece. err: <nil>
Dec 28 14:14:08.014508277 (550h45m7.885591352s) [noise=true , address=145.79.227.42:4123 , piece_id=7QUYG56BEL7FSD2A6AHQNONXE4PWH3QEHZSBKEGYKXYRH5DKIP5Q, seg_pos={0 0}, upload=482879]: Done uploading piece. err: <nil>
Dec 28 14:14:08.015107678 (550h45m7.886190723s) [noise=true , address=145.79.227.20:4439 , piece_id=WC36XT6DAV27DTUCFKYSU3AWZLWZ5EM37JQH3SZPSWOUOE4PR5HQ, seg_pos={0 0}, upload=482879]: Done uploading piece. err: <nil>
Dec 28 14:14:08.020768196 (550h45m7.891851231s) [noise=true , address=145.79.227.46:4603 , piece_id=TWVG57KDN3OV4DU3QD4ONWNEEYOCKYXYQ63QBQPEZS676HDAMZYA, seg_pos={0 0}, upload=482879]: Done uploading piece. err: <nil>
Dec 28 14:14:08.022418348 (550h45m7.893501383s) [noise=true , address=145.79.227.47:4266 , piece_id=FUBWPNHK7SKGUJSKNMTHO3RGUCTABSL4QOASCW23J6GCJ3H5LDMQ, seg_pos={0 0}, upload=482879]: Done uploading piece. err: <nil>
Dec 28 14:14:08.065573974 (550h45m7.936657019s) [noise=true , address=145.79.227.37:4566 , piece_id=5ICU5SETRGGIIKRNDRCTUUC2VTSUGRD6PWQH7BTZYVYD7S3SQSQQ, seg_pos={0 0}, upload=482879]: Done uploading piece. err: <nil>
Dec 28 14:14:08.252250557 (550h45m8.123333602s) [noise=true , address=45.91.116.6:4975 , piece_id=IRL2CQSLRGQJIP3BHV7HYBS56R7HGUDQPQBNNFXEE6Q7IUF2CXHA, seg_pos={0 0}, upload=482879]: Done uploading piece. err: <nil>
Dec 28 14:14:08.253942699 (550h45m8.125025734s) [noise=true , address=45.91.116.11:4120 , piece_id=MRG7VL7KTA3T5FEUFJW5ISMKDWWLUGHHBGGL7RMWRSON4N2UVZUA, seg_pos={0 0}, upload=482879]: Done uploading piece. err: <nil>
Dec 28 14:14:09.023479484 (550h45m8.894562519s) [noise=true , address=45.91.116.14:4190 , piece_id=565ADKSRT363WQJIB2WFNFLNPS765AVU7R72A5LMHGACOYZNBYDQ, seg_pos={0 0}, upload=482879]: Done uploading piece. err: <nil>
Dec 28 14:14:09.024215435 (550h45m8.89529848s) [noise=true , address=45.91.116.9:4262 , piece_id=54C5OVTRQ2FAS4X2EEPNRRAKFEWDO5JQEDQ5HOCOTPJW54PUK2YA, seg_pos={0 0}, upload=482879]: Done uploading piece. err: <nil>
Dec 28 14:14:09.024226005 (550h45m8.89530905s) [seg_pos={0 0}, upload=482879]: Segment reached optimal threshold of 13 pieces.
Dec 28 14:14:09.024424473 (550h45m8.895507508s) [noise=true , address=77.90.31.21:4880 , piece_id=2EP7PPWFUVNZLRRIGSC7GW53WGRS2XCZNP7O5BDDNHENMJBISNZA, seg_pos={0 0}, upload=482879]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF
Dec 28 14:14:09.024431356 (550h45m8.895514391s) [noise=true , address=77.90.31.21:4880 , piece_id=2EP7PPWFUVNZLRRIGSC7GW53WGRS2XCZNP7O5BDDNHENMJBISNZA, seg_pos={0 0}, upload=482879]: Long tail cancellation due to optimal threshold
Dec 28 14:14:09.024575721 (550h45m8.895658756s) [noise=true , address=145.79.227.40:4051 , piece_id=3QYIHWAQR7G5I57JDXJ4SJ6A2LARGCAP6XK7CVE6K2WVR4XGXXHQ, seg_pos={0 0}, upload=482879]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF
Dec 28 14:14:09.024582594 (550h45m8.895665639s) [noise=true , address=145.79.227.40:4051 , piece_id=3QYIHWAQR7G5I57JDXJ4SJ6A2LARGCAP6XK7CVE6K2WVR4XGXXHQ, seg_pos={0 0}, upload=482879]: Long tail cancellation due to optimal threshold
Dec 28 14:14:09.024595309 (550h45m8.895678354s) [noise=true , address=45.91.116.12:4228 , piece_id=VXDU5ZAC3NCP65MDUZ5LSVEIGRQDJIJDP2TIUGXPHGLBLGBMMCWA, seg_pos={0 0}, upload=482879]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:55860->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:55860->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55860->45.91.116.12:4228: use of closed network connection
Dec 28 14:14:09.024622490 (550h45m8.895705555s) [noise=true , address=45.91.116.12:4228 , piece_id=VXDU5ZAC3NCP65MDUZ5LSVEIGRQDJIJDP2TIUGXPHGLBLGBMMCWA, seg_pos={0 0}, upload=482879]: Long tail cancellation due to optimal threshold
Dec 28 14:14:09.024649401 (550h45m8.895732436s) [seg_pos={0 0}, upload=482879]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 14:14:09.024766164 (550h45m8.895849199s) [upload=482879]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 14:14:09.024773017 (550h45m8.895856062s) [upload=482879]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 14:14:09.321695907 (550h45m9.192778942s) [upload=482879]: Done uploading segments...
Dec 28 14:14:09.321715164 (550h45m9.192798199s) [upload=482879]: Upload finished. err: <nil>
Duration: 30.002899861s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [...X....................................X........X...X......] duration:
33.554976921s trace:
https://jaeger.dev.storj.io/trace/36d5af3c761e71b1
FAIL (2025-12-28 14:14:21 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
7.49:4360 , piece_id=QDT5N5LDL2HCVXPOOSXWD2TOEZDF2JZWFLDFYSIMN647FI3ZORUQ, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.372193736 (550h40m10.243276781s) [noise=true , address=77.90.31.12:4618 , piece_id=45WRZ26CPUODOAYF5B6RF37DNMDDQWWJUDK7BFKYS3K3LOCNYJCA, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.372518505 (550h40m10.24360155s) [noise=true , address=77.90.31.13:4646 , piece_id=5CDMQF7GG5PDPHIPMFZWDBKWH5ANO3UIX257LJPCWRITNOA7GY4A, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.372848052 (550h40m10.243931097s) [noise=true , address=64.49.13.170:4721 , piece_id=U76TI7IPAHPZAS6XLG66E4L2CGCYH43NH4ZHTYXMXKZR6SGE7SKQ, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.373163072 (550h40m10.244246117s) [noise=true , address=77.90.31.11:4565 , piece_id=4SBLRZK3K4ED73PJ64BI6ANKBWB2AAHQA4SBUNH6CJ63GN3FJR2Q, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.373445089 (550h40m10.244528134s) [noise=true , address=145.79.227.34:4911 , piece_id=24WE4EMZSL2JHQZ7PESCMLYCZOWGDTGJD2XNLUUJEFPSYLRI5E7A, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.373768855 (550h40m10.24485189s) [noise=true , address=77.90.31.18:4819 , piece_id=OYXPAN3EK4QRDEC6VHIDL3GSGEY27G72KII6HTGUMSPYLHCYBNEA, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.374285519 (550h40m10.245368564s) [noise=true , address=77.90.31.14:4678 , piece_id=VBGPSGMNDQZYY7UKRINAV6B3FVI5BPDPYS37FQIJCMLL65F3XOEQ, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.374710689 (550h40m10.245793724s) [noise=true , address=145.79.227.40:4053 , piece_id=RO4ZZQCKCC7T4T6SOT7GGTY626BJ6ESLQG6KATTPNIUM3ICUKHAQ, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.375099329 (550h40m10.246182364s) [noise=true , address=77.90.31.19:4859 , piece_id=A4HD3FPDPCCMHRFBTS3GNGB323RYD5C5AW4ZVSNOTXFI7FLIV37Q, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.376073433 (550h40m10.247156468s) [noise=true , address=77.90.31.15:4704 , piece_id=DXWLFOIKOPE33LMEQECUJVIGRG3LPADUMULLWI4ZBHEOQHD43ULQ, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.376542387 (550h40m10.247625422s) [noise=true , address=145.79.227.44:4221 , piece_id=H72XCWLKYJAXYGGDIWN67H42RUIEOJCPUSWT5IJMFPKETH7LK3KA, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.377379761 (550h40m10.248462796s) [noise=true , address=145.79.227.32:4863 , piece_id=27TOXK44RIF3DV3PDY4RBP6HEFRQJE6N7LNLRJKSU52YCB6A3JKA, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.378022765 (550h40m10.24910581s) [noise=true , address=145.79.227.30:4789 , piece_id=4QT5IUPIK7BOHJROEK2EBFTW434OKC75FZNENMYYCRMBZ7E5BUTA, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.378821687 (550h40m10.249904722s) [noise=true , address=145.79.227.2:4119 , piece_id=EQM3WMBQ7GXRWRBZK4ZWCI23K4LSLXW6QPEABHGMWRHEYUYANKSQ, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:11.676223695 (550h40m11.54730674s) [upload=482805]: Next returned nil segment
Dec 28 14:09:11.676238083 (550h40m11.547321128s) [upload=482805]: Waiting for error group managing segments...
Dec 28 14:09:12.006762275 (550h40m11.87784531s) [noise=true , address=64.49.13.170:4721 , piece_id=U76TI7IPAHPZAS6XLG66E4L2CGCYH43NH4ZHTYXMXKZR6SGE7SKQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.250307967 (550h40m12.121391012s) [noise=true , address=145.79.227.2:4119 , piece_id=EQM3WMBQ7GXRWRBZK4ZWCI23K4LSLXW6QPEABHGMWRHEYUYANKSQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.269380734 (550h40m12.140463769s) [noise=true , address=145.79.227.49:4360 , piece_id=QDT5N5LDL2HCVXPOOSXWD2TOEZDF2JZWFLDFYSIMN647FI3ZORUQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.290818335 (550h40m12.16190138s) [noise=true , address=145.79.227.32:4863 , piece_id=27TOXK44RIF3DV3PDY4RBP6HEFRQJE6N7LNLRJKSU52YCB6A3JKA, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.467679327 (550h40m12.338762362s) [noise=true , address=45.91.116.8:4072 , piece_id=GEVTX6LLENPLN3UQLSGTFTP3DO52IRYUIQFIWC4XMOC3ZF6V4GTQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.486615595 (550h40m12.35769863s) [noise=true , address=77.90.31.12:4618 , piece_id=45WRZ26CPUODOAYF5B6RF37DNMDDQWWJUDK7BFKYS3K3LOCNYJCA, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.566542810 (550h40m12.437625845s) [noise=true , address=77.90.31.19:4859 , piece_id=A4HD3FPDPCCMHRFBTS3GNGB323RYD5C5AW4ZVSNOTXFI7FLIV37Q, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.609337578 (550h40m12.480420623s) [noise=true , address=77.90.31.13:4646 , piece_id=5CDMQF7GG5PDPHIPMFZWDBKWH5ANO3UIX257LJPCWRITNOA7GY4A, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.640856565 (550h40m12.511939611s) [noise=true , address=45.91.116.5:4322 , piece_id=SS6DOCSJHY5Y3DPWMYX45F4VYYXR44JDLYQ2YPY5XIIAYKMAG7LQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.645490329 (550h40m12.516573364s) [noise=true , address=77.90.31.18:4819 , piece_id=OYXPAN3EK4QRDEC6VHIDL3GSGEY27G72KII6HTGUMSPYLHCYBNEA, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:13.103710286 (550h40m12.974793331s) [noise=true , address=145.79.227.34:4911 , piece_id=24WE4EMZSL2JHQZ7PESCMLYCZOWGDTGJD2XNLUUJEFPSYLRI5E7A, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:13.209917141 (550h40m13.081000176s) [noise=true , address=145.79.227.44:4221 , piece_id=H72XCWLKYJAXYGGDIWN67H42RUIEOJCPUSWT5IJMFPKETH7LK3KA, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:13.218355468 (550h40m13.089438514s) [noise=true , address=77.90.31.11:4565 , piece_id=4SBLRZK3K4ED73PJ64BI6ANKBWB2AAHQA4SBUNH6CJ63GN3FJR2Q, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:13.255968741 (550h40m13.127051806s) [noise=true , address=145.79.227.30:4789 , piece_id=4QT5IUPIK7BOHJROEK2EBFTW434OKC75FZNENMYYCRMBZ7E5BUTA, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:13.255996102 (550h40m13.127079137s) [seg_pos={0 0}, upload=482805]: Segment reached optimal threshold of 14 pieces.
Dec 28 14:09:13.256214589 (550h40m13.127297634s) [noise=true , address=145.79.227.40:4053 , piece_id=RO4ZZQCKCC7T4T6SOT7GGTY626BJ6ESLQG6KATTPNIUM3ICUKHAQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DqyTc4MHvxutFXs2WWRCk8MSuxF3Y1HayyQY4Y5sy8cqDLWuG): protocol: expected piece hash; context canceled; EOF
Dec 28 14:09:13.256223686 (550h40m13.127306731s) [noise=true , address=145.79.227.40:4053 , piece_id=RO4ZZQCKCC7T4T6SOT7GGTY626BJ6ESLQG6KATTPNIUM3ICUKHAQ, seg_pos={0 0}, upload=482805]: Long tail cancellation due to optimal threshold
Dec 28 14:09:13.256512506 (550h40m13.127595551s) [noise=true , address=77.90.31.15:4704 , piece_id=DXWLFOIKOPE33LMEQECUJVIGRG3LPADUMULLWI4ZBHEOQHD43ULQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14tJRdZ1guwwsYwPf3LtENRWqYx2jBQpi7uAMMPaB32Gks9zmj): protocol: expected piece hash; context canceled; EOF
Dec 28 14:09:13.256520060 (550h40m13.127603105s) [noise=true , address=77.90.31.15:4704 , piece_id=DXWLFOIKOPE33LMEQECUJVIGRG3LPADUMULLWI4ZBHEOQHD43ULQ, seg_pos={0 0}, upload=482805]: Long tail cancellation due to optimal threshold
Dec 28 14:09:13.256649106 (550h40m13.127732151s) [noise=true , address=77.90.31.14:4678 , piece_id=VBGPSGMNDQZYY7UKRINAV6B3FVI5BPDPYS37FQIJCMLL65F3XOEQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12beu3SUZMPNWfbeHV4tLtx9UVZoEHh4Kt7oBhT59w3bGBE1ovP): protocol: write tcp 188.166.225.170:57374->77.90.31.14:4678: use of closed network connection; write tcp 188.166.225.170:57374->77.90.31.14:4678: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57374->77.90.31.14:4678: use of closed network connection
Dec 28 14:09:13.256655819 (550h40m13.127738864s) [noise=true , address=77.90.31.14:4678 , piece_id=VBGPSGMNDQZYY7UKRINAV6B3FVI5BPDPYS37FQIJCMLL65F3XOEQ, seg_pos={0 0}, upload=482805]: Long tail cancellation due to optimal threshold
Dec 28 14:09:13.256752974 (550h40m13.127836009s) [noise=true , address=64.49.13.169:4710 , piece_id=SANL7TVJFES63XD46OKQL4KIJ6YVHSEWPZKMRISEKC3ZKWQBOMIA, seg_pos={0 0}, upload=482805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:35680->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:35680->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35680->64.49.13.169:4710: use of closed network connection
Dec 28 14:09:13.256758825 (550h40m13.12784186s) [noise=true , address=64.49.13.169:4710 , piece_id=SANL7TVJFES63XD46OKQL4KIJ6YVHSEWPZKMRISEKC3ZKWQBOMIA, seg_pos={0 0}, upload=482805]: Long tail cancellation due to optimal threshold
Dec 28 14:09:13.256776549 (550h40m13.127859594s) [seg_pos={0 0}, upload=482805]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 14:09:13.256856591 (550h40m13.127939626s) [upload=482805]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 14:09:13.256862432 (550h40m13.127945467s) [upload=482805]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 14:09:13.550363717 (550h40m13.421446782s) [upload=482805]: Done uploading segments...
Dec 28 14:09:13.550405126 (550h40m13.421488161s) [upload=482805]: Upload finished. err: <nil>
Duration: 33.554993002s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
52 out of 60 is failed (threshold=6) [.XXXXXXX.XXXXXXXXXXXX.XXXXXXX.XX..XXXXXXXXXXXXXXXXXXXXX.X.XX] duration:
5.001425082s trace:
https://jaeger.dev.storj.io/trace/6de1a36fc7c72aa2
FAIL (2025-12-28 14:14:21 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001444249s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [.X............X....................X...X........X...........] duration:
16.828241938s trace:
https://jaeger.dev.storj.io/trace/360e059013858fb1
FAIL (2025-12-28 14:14:08 UTC)
orbiter warning: context deadline exceeded
Duration: 16.828258058s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [............X...............................................] duration:
7.589857725s trace:
https://jaeger.dev.storj.io/trace/5fe9d6dbb79fbe44
FAIL (2025-12-28 14:13:59 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
67.204.205:4027 , piece_id=PAPNMRJ7E47VOMOXPBBDWKZJHLRMXTWVR5VW3J7KNQSSMHOHWOAQ, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.967737953 (550h8m4.838820988s) [noise=true , address=185.167.204.206:4072 , piece_id=WXORBDMA3KW7WZ67A4FJT6URJRFADUJ7X2PAHIGNG47T3TGMXVTQ, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.968193240 (550h8m4.839276285s) [noise=true , address=185.167.204.205:4030 , piece_id=R5DLI4ZTLOVGO56GPLJL5ZNHYSXOSYNJQHFIO4MVVNHWWBMEOV2Q, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.968780638 (550h8m4.839863683s) [noise=true , address=89.149.201.147:4019 , piece_id=MK7ORCLHO3DOWIOVSOWIQMPJ447PHRQGXP3UKITIRTYJSU5MPCCQ, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.969308924 (550h8m4.840391959s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.970088349 (550h8m4.841171404s) [noise=true , address=81.171.14.70:4039 , piece_id=EDHPQWLQZTP7MKPGEVMP7JXNTW7LFWSXCTRTSXPCBIQPROX76WRA, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.970325381 (550h8m4.841408426s) [noise=true , address=81.171.14.70:4036 , piece_id=CICNNV7JE6TBNC4RAPXKCEUSJAADOUVGSVIVQISM3ZB4COAZQDDA, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.970783813 (550h8m4.841866858s) [noise=true , address=81.171.14.70:4044 , piece_id=DGN74IG6IWAHLEMO6NLL4TJBXZ6ONTN6QUB6HXY6K7SSO3LYOYIQ, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.971205406 (550h8m4.842288441s) [noise=true , address=81.171.14.70:4048 , piece_id=TLT47ORGUKGF3MIIEAX7D22VR2FO6W6NL3VFJNHKWD4L3LIMFDLA, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.971606299 (550h8m4.842689344s) [noise=true , address=185.167.204.206:4067 , piece_id=VSKTG5RH32N7OEVWJGQVB6WJ72TEVF6DDK25W7NOH4RAXUP52LTQ, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.972162538 (550h8m4.843245583s) [noise=true , address=89.149.201.147:4002 , piece_id=G2PABNTDWCK4MKOZF6PONBMRNIXI45JTHJMAZFPEKPEJ3PQ7MTBA, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:05.795167814 (550h8m5.666250849s) [upload=482337]: Next returned nil segment
Dec 28 13:37:05.795184566 (550h8m5.666267611s) [upload=482337]: Waiting for error group managing segments...
Dec 28 13:37:06.178573801 (550h8m6.049656836s) [noise=true , address=185.167.204.206:4049 , piece_id=6RG2DZV6FPTIRE63ZNNST42TSWEBQ456YXIPPVYHIXV55GWFVJ4Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.183060835 (550h8m6.05414391s) [noise=true , address=185.167.204.206:4072 , piece_id=WXORBDMA3KW7WZ67A4FJT6URJRFADUJ7X2PAHIGNG47T3TGMXVTQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.183411202 (550h8m6.054494237s) [noise=true , address=185.167.204.206:4064 , piece_id=OG3G74OU23CEMKA54WQE3DOJ6AKAXBLZ23QWOQC4UEGQQN4JSMXQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.183791146 (550h8m6.054874181s) [noise=true , address=185.167.204.205:4029 , piece_id=6JQB6ZSCQZKWH3JWCQ4DY6BBAX73KC4Y53PRSMEMLWURZ5QWWQQQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.183973743 (550h8m6.055056778s) [noise=true , address=185.167.204.206:4053 , piece_id=YDH77S6JLKJ6IYS6DRUCELP5EJHIC2O44QBO7MIOEUYBML4GIBHA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.184465369 (550h8m6.055548414s) [noise=true , address=185.167.204.205:4008 , piece_id=RTUZPEXBB6JNSMC5YOFQQGDM57YD233W4756ZBQVMFNOVX6YKB4A, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.184501989 (550h8m6.055585024s) [noise=true , address=185.167.204.205:4030 , piece_id=R5DLI4ZTLOVGO56GPLJL5ZNHYSXOSYNJQHFIO4MVVNHWWBMEOV2Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.184856224 (550h8m6.055939259s) [noise=true , address=185.167.204.205:4009 , piece_id=V6RA7Y7FKY4DCPTRRF3BYNBALWDY5VTPXMDUXFE3353KUXLSQJEA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.185063518 (550h8m6.056146563s) [noise=true , address=185.167.204.205:4016 , piece_id=JB7SC3J2VR4U33HNQS2A2CA75THT3XLQCOIFHAJJEK7MQGY2SYAQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.185288617 (550h8m6.056371652s) [noise=true , address=185.167.204.205:4027 , piece_id=PAPNMRJ7E47VOMOXPBBDWKZJHLRMXTWVR5VW3J7KNQSSMHOHWOAQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.186241401 (550h8m6.057324436s) [noise=true , address=185.167.204.205:4024 , piece_id=FM5IMWRSJJ4RY72IX2OOYC54BFKL4FWE3UOAN3GJ26MLH345COVA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.193113658 (550h8m6.064196703s) [noise=true , address=185.167.204.205:4005 , piece_id=R4DFQJZBPJ5WJWHPODV6ILHUDIRBUSAFOMVNCLXB4UNMI6EXTKHA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.193791408 (550h8m6.064874453s) [noise=true , address=185.167.204.206:4068 , piece_id=DAI37M67FWCSVQTQOZ6RSSOJDGUVZO6SNGAHJJ44G3PGS4OJZEMA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.194292262 (550h8m6.065375307s) [noise=true , address=185.167.204.206:4046 , piece_id=VWLVHMBMJ63DQEWYBIKH3Y3NTVWFNW5OXZKMYXXBHVI3VWYXLLPQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.194302652 (550h8m6.065385687s) [noise=true , address=185.167.204.206:4067 , piece_id=VSKTG5RH32N7OEVWJGQVB6WJ72TEVF6DDK25W7NOH4RAXUP52LTQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.195359193 (550h8m6.066442228s) [noise=true , address=185.167.204.205:4012 , piece_id=LV6RUA5BWXG4DMSUVZDBGUAF5IBWUGMLJYUKNIXHRVDEBIOW4Q2Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.198423448 (550h8m6.069506493s) [noise=true , address=185.167.204.206:4070 , piece_id=5A7YYQYREIZRZDR7TRSE7XFW765JVAV7FX6LKWUI2OXAA74XN4CA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.888968408 (550h8m6.760051453s) [noise=true , address=81.171.14.70:4044 , piece_id=DGN74IG6IWAHLEMO6NLL4TJBXZ6ONTN6QUB6HXY6K7SSO3LYOYIQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.892350789 (550h8m6.763433834s) [noise=true , address=81.171.14.70:4039 , piece_id=EDHPQWLQZTP7MKPGEVMP7JXNTW7LFWSXCTRTSXPCBIQPROX76WRA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.904477160 (550h8m6.775560195s) [noise=true , address=81.171.14.70:4040 , piece_id=DYF4GSZBAKQ3H63LTVCMEFWMABY3FXHOJCNVIFYRKLF2J6B52K5A, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.128088281 (550h8m6.999171326s) [noise=true , address=81.171.14.70:4036 , piece_id=CICNNV7JE6TBNC4RAPXKCEUSJAADOUVGSVIVQISM3ZB4COAZQDDA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.430242142 (550h8m7.301325177s) [noise=true , address=89.149.201.147:4009 , piece_id=EVW2NQFYU57PC2IIVXLYIEBMYXJNIC2CLCHW6PS6NYIPCFRUVHDA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.505304332 (550h8m7.376387367s) [noise=true , address=89.149.201.147:4019 , piece_id=MK7ORCLHO3DOWIOVSOWIQMPJ447PHRQGXP3UKITIRTYJSU5MPCCQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.575461543 (550h8m7.446544609s) [noise=true , address=81.171.14.70:4048 , piece_id=TLT47ORGUKGF3MIIEAX7D22VR2FO6W6NL3VFJNHKWD4L3LIMFDLA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.599667265 (550h8m7.4707503s) [noise=true , address=89.149.201.147:4023 , piece_id=5IINX2GVRFLRX2QH24OO2DDANOKOTXTDR6AGVJBGSJ7IAZQ3CASA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.928390728 (550h8m7.799473773s) [noise=true , address=89.149.201.147:4002 , piece_id=G2PABNTDWCK4MKOZF6PONBMRNIXI45JTHJMAZFPEKPEJ3PQ7MTBA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.928417148 (550h8m7.799500193s) [seg_pos={0 0}, upload=482337]: Segment reached optimal threshold of 26 pieces.
Dec 28 13:37:07.928656303 (550h8m7.799739348s) [noise=true , address=81.171.14.70:4030 , piece_id=74YDIKUST7X3JBXVQHVLFHIKH5A4DYH5TNL3I4XUA5A6AHQO2DRA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZoRwa8wLw66Pjumy1ybp6wrZ79RrSU55uYZpwaY3xi95iSYde): protocol: expected piece hash; context canceled; EOF
Dec 28 13:37:07.928664990 (550h8m7.799748035s) [noise=true , address=81.171.14.70:4030 , piece_id=74YDIKUST7X3JBXVQHVLFHIKH5A4DYH5TNL3I4XUA5A6AHQO2DRA, seg_pos={0 0}, upload=482337]: Long tail cancellation due to optimal threshold
Dec 28 13:37:07.928723351 (550h8m7.799806386s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wgtphHKaTejC1qrJ98sDysBwtj2Zeqw2c5UFk1HMqfY8Buvr): protocol: expected piece hash; context canceled; EOF
Dec 28 13:37:07.928733922 (550h8m7.799816957s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Long tail cancellation due to optimal threshold
Dec 28 13:37:07.928754921 (550h8m7.799837966s) [seg_pos={0 0}, upload=482337]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 28 13:37:07.928870170 (550h8m7.799953215s) [upload=482337]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 13:37:07.928878627 (550h8m7.799961672s) [upload=482337]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 13:37:08.166321536 (550h8m8.037404571s) [upload=482337]: Done uploading segments...
Dec 28 13:37:08.166341444 (550h8m8.037424479s) [upload=482337]: Upload finished. err: <nil>
Duration: 7.589879337s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [....X....................................X........X...X.....] duration:
38.76567092s trace:
https://jaeger.dev.storj.io/trace/36d5af3c761e71b1
FAIL (2025-12-28 14:13:48 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
7.49:4360 , piece_id=QDT5N5LDL2HCVXPOOSXWD2TOEZDF2JZWFLDFYSIMN647FI3ZORUQ, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.372193736 (550h40m10.243276781s) [noise=true , address=77.90.31.12:4618 , piece_id=45WRZ26CPUODOAYF5B6RF37DNMDDQWWJUDK7BFKYS3K3LOCNYJCA, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.372518505 (550h40m10.24360155s) [noise=true , address=77.90.31.13:4646 , piece_id=5CDMQF7GG5PDPHIPMFZWDBKWH5ANO3UIX257LJPCWRITNOA7GY4A, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.372848052 (550h40m10.243931097s) [noise=true , address=64.49.13.170:4721 , piece_id=U76TI7IPAHPZAS6XLG66E4L2CGCYH43NH4ZHTYXMXKZR6SGE7SKQ, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.373163072 (550h40m10.244246117s) [noise=true , address=77.90.31.11:4565 , piece_id=4SBLRZK3K4ED73PJ64BI6ANKBWB2AAHQA4SBUNH6CJ63GN3FJR2Q, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.373445089 (550h40m10.244528134s) [noise=true , address=145.79.227.34:4911 , piece_id=24WE4EMZSL2JHQZ7PESCMLYCZOWGDTGJD2XNLUUJEFPSYLRI5E7A, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.373768855 (550h40m10.24485189s) [noise=true , address=77.90.31.18:4819 , piece_id=OYXPAN3EK4QRDEC6VHIDL3GSGEY27G72KII6HTGUMSPYLHCYBNEA, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.374285519 (550h40m10.245368564s) [noise=true , address=77.90.31.14:4678 , piece_id=VBGPSGMNDQZYY7UKRINAV6B3FVI5BPDPYS37FQIJCMLL65F3XOEQ, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.374710689 (550h40m10.245793724s) [noise=true , address=145.79.227.40:4053 , piece_id=RO4ZZQCKCC7T4T6SOT7GGTY626BJ6ESLQG6KATTPNIUM3ICUKHAQ, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.375099329 (550h40m10.246182364s) [noise=true , address=77.90.31.19:4859 , piece_id=A4HD3FPDPCCMHRFBTS3GNGB323RYD5C5AW4ZVSNOTXFI7FLIV37Q, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.376073433 (550h40m10.247156468s) [noise=true , address=77.90.31.15:4704 , piece_id=DXWLFOIKOPE33LMEQECUJVIGRG3LPADUMULLWI4ZBHEOQHD43ULQ, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.376542387 (550h40m10.247625422s) [noise=true , address=145.79.227.44:4221 , piece_id=H72XCWLKYJAXYGGDIWN67H42RUIEOJCPUSWT5IJMFPKETH7LK3KA, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.377379761 (550h40m10.248462796s) [noise=true , address=145.79.227.32:4863 , piece_id=27TOXK44RIF3DV3PDY4RBP6HEFRQJE6N7LNLRJKSU52YCB6A3JKA, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.378022765 (550h40m10.24910581s) [noise=true , address=145.79.227.30:4789 , piece_id=4QT5IUPIK7BOHJROEK2EBFTW434OKC75FZNENMYYCRMBZ7E5BUTA, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.378821687 (550h40m10.249904722s) [noise=true , address=145.79.227.2:4119 , piece_id=EQM3WMBQ7GXRWRBZK4ZWCI23K4LSLXW6QPEABHGMWRHEYUYANKSQ, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:11.676223695 (550h40m11.54730674s) [upload=482805]: Next returned nil segment
Dec 28 14:09:11.676238083 (550h40m11.547321128s) [upload=482805]: Waiting for error group managing segments...
Dec 28 14:09:12.006762275 (550h40m11.87784531s) [noise=true , address=64.49.13.170:4721 , piece_id=U76TI7IPAHPZAS6XLG66E4L2CGCYH43NH4ZHTYXMXKZR6SGE7SKQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.250307967 (550h40m12.121391012s) [noise=true , address=145.79.227.2:4119 , piece_id=EQM3WMBQ7GXRWRBZK4ZWCI23K4LSLXW6QPEABHGMWRHEYUYANKSQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.269380734 (550h40m12.140463769s) [noise=true , address=145.79.227.49:4360 , piece_id=QDT5N5LDL2HCVXPOOSXWD2TOEZDF2JZWFLDFYSIMN647FI3ZORUQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.290818335 (550h40m12.16190138s) [noise=true , address=145.79.227.32:4863 , piece_id=27TOXK44RIF3DV3PDY4RBP6HEFRQJE6N7LNLRJKSU52YCB6A3JKA, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.467679327 (550h40m12.338762362s) [noise=true , address=45.91.116.8:4072 , piece_id=GEVTX6LLENPLN3UQLSGTFTP3DO52IRYUIQFIWC4XMOC3ZF6V4GTQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.486615595 (550h40m12.35769863s) [noise=true , address=77.90.31.12:4618 , piece_id=45WRZ26CPUODOAYF5B6RF37DNMDDQWWJUDK7BFKYS3K3LOCNYJCA, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.566542810 (550h40m12.437625845s) [noise=true , address=77.90.31.19:4859 , piece_id=A4HD3FPDPCCMHRFBTS3GNGB323RYD5C5AW4ZVSNOTXFI7FLIV37Q, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.609337578 (550h40m12.480420623s) [noise=true , address=77.90.31.13:4646 , piece_id=5CDMQF7GG5PDPHIPMFZWDBKWH5ANO3UIX257LJPCWRITNOA7GY4A, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.640856565 (550h40m12.511939611s) [noise=true , address=45.91.116.5:4322 , piece_id=SS6DOCSJHY5Y3DPWMYX45F4VYYXR44JDLYQ2YPY5XIIAYKMAG7LQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.645490329 (550h40m12.516573364s) [noise=true , address=77.90.31.18:4819 , piece_id=OYXPAN3EK4QRDEC6VHIDL3GSGEY27G72KII6HTGUMSPYLHCYBNEA, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:13.103710286 (550h40m12.974793331s) [noise=true , address=145.79.227.34:4911 , piece_id=24WE4EMZSL2JHQZ7PESCMLYCZOWGDTGJD2XNLUUJEFPSYLRI5E7A, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:13.209917141 (550h40m13.081000176s) [noise=true , address=145.79.227.44:4221 , piece_id=H72XCWLKYJAXYGGDIWN67H42RUIEOJCPUSWT5IJMFPKETH7LK3KA, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:13.218355468 (550h40m13.089438514s) [noise=true , address=77.90.31.11:4565 , piece_id=4SBLRZK3K4ED73PJ64BI6ANKBWB2AAHQA4SBUNH6CJ63GN3FJR2Q, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:13.255968741 (550h40m13.127051806s) [noise=true , address=145.79.227.30:4789 , piece_id=4QT5IUPIK7BOHJROEK2EBFTW434OKC75FZNENMYYCRMBZ7E5BUTA, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:13.255996102 (550h40m13.127079137s) [seg_pos={0 0}, upload=482805]: Segment reached optimal threshold of 14 pieces.
Dec 28 14:09:13.256214589 (550h40m13.127297634s) [noise=true , address=145.79.227.40:4053 , piece_id=RO4ZZQCKCC7T4T6SOT7GGTY626BJ6ESLQG6KATTPNIUM3ICUKHAQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DqyTc4MHvxutFXs2WWRCk8MSuxF3Y1HayyQY4Y5sy8cqDLWuG): protocol: expected piece hash; context canceled; EOF
Dec 28 14:09:13.256223686 (550h40m13.127306731s) [noise=true , address=145.79.227.40:4053 , piece_id=RO4ZZQCKCC7T4T6SOT7GGTY626BJ6ESLQG6KATTPNIUM3ICUKHAQ, seg_pos={0 0}, upload=482805]: Long tail cancellation due to optimal threshold
Dec 28 14:09:13.256512506 (550h40m13.127595551s) [noise=true , address=77.90.31.15:4704 , piece_id=DXWLFOIKOPE33LMEQECUJVIGRG3LPADUMULLWI4ZBHEOQHD43ULQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14tJRdZ1guwwsYwPf3LtENRWqYx2jBQpi7uAMMPaB32Gks9zmj): protocol: expected piece hash; context canceled; EOF
Dec 28 14:09:13.256520060 (550h40m13.127603105s) [noise=true , address=77.90.31.15:4704 , piece_id=DXWLFOIKOPE33LMEQECUJVIGRG3LPADUMULLWI4ZBHEOQHD43ULQ, seg_pos={0 0}, upload=482805]: Long tail cancellation due to optimal threshold
Dec 28 14:09:13.256649106 (550h40m13.127732151s) [noise=true , address=77.90.31.14:4678 , piece_id=VBGPSGMNDQZYY7UKRINAV6B3FVI5BPDPYS37FQIJCMLL65F3XOEQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12beu3SUZMPNWfbeHV4tLtx9UVZoEHh4Kt7oBhT59w3bGBE1ovP): protocol: write tcp 188.166.225.170:57374->77.90.31.14:4678: use of closed network connection; write tcp 188.166.225.170:57374->77.90.31.14:4678: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57374->77.90.31.14:4678: use of closed network connection
Dec 28 14:09:13.256655819 (550h40m13.127738864s) [noise=true , address=77.90.31.14:4678 , piece_id=VBGPSGMNDQZYY7UKRINAV6B3FVI5BPDPYS37FQIJCMLL65F3XOEQ, seg_pos={0 0}, upload=482805]: Long tail cancellation due to optimal threshold
Dec 28 14:09:13.256752974 (550h40m13.127836009s) [noise=true , address=64.49.13.169:4710 , piece_id=SANL7TVJFES63XD46OKQL4KIJ6YVHSEWPZKMRISEKC3ZKWQBOMIA, seg_pos={0 0}, upload=482805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:35680->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:35680->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35680->64.49.13.169:4710: use of closed network connection
Dec 28 14:09:13.256758825 (550h40m13.12784186s) [noise=true , address=64.49.13.169:4710 , piece_id=SANL7TVJFES63XD46OKQL4KIJ6YVHSEWPZKMRISEKC3ZKWQBOMIA, seg_pos={0 0}, upload=482805]: Long tail cancellation due to optimal threshold
Dec 28 14:09:13.256776549 (550h40m13.127859594s) [seg_pos={0 0}, upload=482805]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 14:09:13.256856591 (550h40m13.127939626s) [upload=482805]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 14:09:13.256862432 (550h40m13.127945467s) [upload=482805]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 14:09:13.550363717 (550h40m13.421446782s) [upload=482805]: Done uploading segments...
Dec 28 14:09:13.550405126 (550h40m13.421488161s) [upload=482805]: Upload finished. err: <nil>
Duration: 38.765730964s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [..X............X....................X...X........X..........] duration:
24.917566655s trace:
https://jaeger.dev.storj.io/trace/360e059013858fb1
FAIL (2025-12-28 14:13:46 UTC)
orbiter warning: context deadline exceeded
Duration: 24.917591002s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [...................................................X...X....] duration:
5.956791109s trace:
https://jaeger.dev.storj.io/trace/102cf391e061ae77
FAIL (2025-12-28 14:13:33 UTC)
orbiter warning: context deadline exceeded
Duration: 5.956822839s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [.............X..............................................] duration:
6.869523411s trace:
https://jaeger.dev.storj.io/trace/5fe9d6dbb79fbe44
FAIL (2025-12-28 14:13:12 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
67.204.205:4027 , piece_id=PAPNMRJ7E47VOMOXPBBDWKZJHLRMXTWVR5VW3J7KNQSSMHOHWOAQ, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.967737953 (550h8m4.838820988s) [noise=true , address=185.167.204.206:4072 , piece_id=WXORBDMA3KW7WZ67A4FJT6URJRFADUJ7X2PAHIGNG47T3TGMXVTQ, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.968193240 (550h8m4.839276285s) [noise=true , address=185.167.204.205:4030 , piece_id=R5DLI4ZTLOVGO56GPLJL5ZNHYSXOSYNJQHFIO4MVVNHWWBMEOV2Q, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.968780638 (550h8m4.839863683s) [noise=true , address=89.149.201.147:4019 , piece_id=MK7ORCLHO3DOWIOVSOWIQMPJ447PHRQGXP3UKITIRTYJSU5MPCCQ, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.969308924 (550h8m4.840391959s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.970088349 (550h8m4.841171404s) [noise=true , address=81.171.14.70:4039 , piece_id=EDHPQWLQZTP7MKPGEVMP7JXNTW7LFWSXCTRTSXPCBIQPROX76WRA, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.970325381 (550h8m4.841408426s) [noise=true , address=81.171.14.70:4036 , piece_id=CICNNV7JE6TBNC4RAPXKCEUSJAADOUVGSVIVQISM3ZB4COAZQDDA, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.970783813 (550h8m4.841866858s) [noise=true , address=81.171.14.70:4044 , piece_id=DGN74IG6IWAHLEMO6NLL4TJBXZ6ONTN6QUB6HXY6K7SSO3LYOYIQ, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.971205406 (550h8m4.842288441s) [noise=true , address=81.171.14.70:4048 , piece_id=TLT47ORGUKGF3MIIEAX7D22VR2FO6W6NL3VFJNHKWD4L3LIMFDLA, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.971606299 (550h8m4.842689344s) [noise=true , address=185.167.204.206:4067 , piece_id=VSKTG5RH32N7OEVWJGQVB6WJ72TEVF6DDK25W7NOH4RAXUP52LTQ, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.972162538 (550h8m4.843245583s) [noise=true , address=89.149.201.147:4002 , piece_id=G2PABNTDWCK4MKOZF6PONBMRNIXI45JTHJMAZFPEKPEJ3PQ7MTBA, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:05.795167814 (550h8m5.666250849s) [upload=482337]: Next returned nil segment
Dec 28 13:37:05.795184566 (550h8m5.666267611s) [upload=482337]: Waiting for error group managing segments...
Dec 28 13:37:06.178573801 (550h8m6.049656836s) [noise=true , address=185.167.204.206:4049 , piece_id=6RG2DZV6FPTIRE63ZNNST42TSWEBQ456YXIPPVYHIXV55GWFVJ4Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.183060835 (550h8m6.05414391s) [noise=true , address=185.167.204.206:4072 , piece_id=WXORBDMA3KW7WZ67A4FJT6URJRFADUJ7X2PAHIGNG47T3TGMXVTQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.183411202 (550h8m6.054494237s) [noise=true , address=185.167.204.206:4064 , piece_id=OG3G74OU23CEMKA54WQE3DOJ6AKAXBLZ23QWOQC4UEGQQN4JSMXQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.183791146 (550h8m6.054874181s) [noise=true , address=185.167.204.205:4029 , piece_id=6JQB6ZSCQZKWH3JWCQ4DY6BBAX73KC4Y53PRSMEMLWURZ5QWWQQQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.183973743 (550h8m6.055056778s) [noise=true , address=185.167.204.206:4053 , piece_id=YDH77S6JLKJ6IYS6DRUCELP5EJHIC2O44QBO7MIOEUYBML4GIBHA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.184465369 (550h8m6.055548414s) [noise=true , address=185.167.204.205:4008 , piece_id=RTUZPEXBB6JNSMC5YOFQQGDM57YD233W4756ZBQVMFNOVX6YKB4A, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.184501989 (550h8m6.055585024s) [noise=true , address=185.167.204.205:4030 , piece_id=R5DLI4ZTLOVGO56GPLJL5ZNHYSXOSYNJQHFIO4MVVNHWWBMEOV2Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.184856224 (550h8m6.055939259s) [noise=true , address=185.167.204.205:4009 , piece_id=V6RA7Y7FKY4DCPTRRF3BYNBALWDY5VTPXMDUXFE3353KUXLSQJEA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.185063518 (550h8m6.056146563s) [noise=true , address=185.167.204.205:4016 , piece_id=JB7SC3J2VR4U33HNQS2A2CA75THT3XLQCOIFHAJJEK7MQGY2SYAQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.185288617 (550h8m6.056371652s) [noise=true , address=185.167.204.205:4027 , piece_id=PAPNMRJ7E47VOMOXPBBDWKZJHLRMXTWVR5VW3J7KNQSSMHOHWOAQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.186241401 (550h8m6.057324436s) [noise=true , address=185.167.204.205:4024 , piece_id=FM5IMWRSJJ4RY72IX2OOYC54BFKL4FWE3UOAN3GJ26MLH345COVA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.193113658 (550h8m6.064196703s) [noise=true , address=185.167.204.205:4005 , piece_id=R4DFQJZBPJ5WJWHPODV6ILHUDIRBUSAFOMVNCLXB4UNMI6EXTKHA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.193791408 (550h8m6.064874453s) [noise=true , address=185.167.204.206:4068 , piece_id=DAI37M67FWCSVQTQOZ6RSSOJDGUVZO6SNGAHJJ44G3PGS4OJZEMA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.194292262 (550h8m6.065375307s) [noise=true , address=185.167.204.206:4046 , piece_id=VWLVHMBMJ63DQEWYBIKH3Y3NTVWFNW5OXZKMYXXBHVI3VWYXLLPQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.194302652 (550h8m6.065385687s) [noise=true , address=185.167.204.206:4067 , piece_id=VSKTG5RH32N7OEVWJGQVB6WJ72TEVF6DDK25W7NOH4RAXUP52LTQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.195359193 (550h8m6.066442228s) [noise=true , address=185.167.204.205:4012 , piece_id=LV6RUA5BWXG4DMSUVZDBGUAF5IBWUGMLJYUKNIXHRVDEBIOW4Q2Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.198423448 (550h8m6.069506493s) [noise=true , address=185.167.204.206:4070 , piece_id=5A7YYQYREIZRZDR7TRSE7XFW765JVAV7FX6LKWUI2OXAA74XN4CA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.888968408 (550h8m6.760051453s) [noise=true , address=81.171.14.70:4044 , piece_id=DGN74IG6IWAHLEMO6NLL4TJBXZ6ONTN6QUB6HXY6K7SSO3LYOYIQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.892350789 (550h8m6.763433834s) [noise=true , address=81.171.14.70:4039 , piece_id=EDHPQWLQZTP7MKPGEVMP7JXNTW7LFWSXCTRTSXPCBIQPROX76WRA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.904477160 (550h8m6.775560195s) [noise=true , address=81.171.14.70:4040 , piece_id=DYF4GSZBAKQ3H63LTVCMEFWMABY3FXHOJCNVIFYRKLF2J6B52K5A, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.128088281 (550h8m6.999171326s) [noise=true , address=81.171.14.70:4036 , piece_id=CICNNV7JE6TBNC4RAPXKCEUSJAADOUVGSVIVQISM3ZB4COAZQDDA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.430242142 (550h8m7.301325177s) [noise=true , address=89.149.201.147:4009 , piece_id=EVW2NQFYU57PC2IIVXLYIEBMYXJNIC2CLCHW6PS6NYIPCFRUVHDA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.505304332 (550h8m7.376387367s) [noise=true , address=89.149.201.147:4019 , piece_id=MK7ORCLHO3DOWIOVSOWIQMPJ447PHRQGXP3UKITIRTYJSU5MPCCQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.575461543 (550h8m7.446544609s) [noise=true , address=81.171.14.70:4048 , piece_id=TLT47ORGUKGF3MIIEAX7D22VR2FO6W6NL3VFJNHKWD4L3LIMFDLA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.599667265 (550h8m7.4707503s) [noise=true , address=89.149.201.147:4023 , piece_id=5IINX2GVRFLRX2QH24OO2DDANOKOTXTDR6AGVJBGSJ7IAZQ3CASA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.928390728 (550h8m7.799473773s) [noise=true , address=89.149.201.147:4002 , piece_id=G2PABNTDWCK4MKOZF6PONBMRNIXI45JTHJMAZFPEKPEJ3PQ7MTBA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.928417148 (550h8m7.799500193s) [seg_pos={0 0}, upload=482337]: Segment reached optimal threshold of 26 pieces.
Dec 28 13:37:07.928656303 (550h8m7.799739348s) [noise=true , address=81.171.14.70:4030 , piece_id=74YDIKUST7X3JBXVQHVLFHIKH5A4DYH5TNL3I4XUA5A6AHQO2DRA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZoRwa8wLw66Pjumy1ybp6wrZ79RrSU55uYZpwaY3xi95iSYde): protocol: expected piece hash; context canceled; EOF
Dec 28 13:37:07.928664990 (550h8m7.799748035s) [noise=true , address=81.171.14.70:4030 , piece_id=74YDIKUST7X3JBXVQHVLFHIKH5A4DYH5TNL3I4XUA5A6AHQO2DRA, seg_pos={0 0}, upload=482337]: Long tail cancellation due to optimal threshold
Dec 28 13:37:07.928723351 (550h8m7.799806386s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wgtphHKaTejC1qrJ98sDysBwtj2Zeqw2c5UFk1HMqfY8Buvr): protocol: expected piece hash; context canceled; EOF
Dec 28 13:37:07.928733922 (550h8m7.799816957s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Long tail cancellation due to optimal threshold
Dec 28 13:37:07.928754921 (550h8m7.799837966s) [seg_pos={0 0}, upload=482337]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 28 13:37:07.928870170 (550h8m7.799953215s) [upload=482337]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 13:37:07.928878627 (550h8m7.799961672s) [upload=482337]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 13:37:08.166321536 (550h8m8.037404571s) [upload=482337]: Done uploading segments...
Dec 28 13:37:08.166341444 (550h8m8.037424479s) [upload=482337]: Upload finished. err: <nil>
Duration: 6.869553407s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [....................................................X...X...] duration:
16.109682915s trace:
https://jaeger.dev.storj.io/trace/102cf391e061ae77
FAIL (2025-12-28 14:13:11 UTC)
orbiter warning: context deadline exceeded
Duration: 16.109716419s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
52 out of 60 is failed (threshold=6) [X.XXXXXXX.XXXXXXXXXXXX.XXXXXXX.XX..XXXXXXXXXXXXXXXXXXXXX.X.X] duration:
5.000682628s trace:
https://jaeger.dev.storj.io/trace/1a361e89699aa481
FAIL (2025-12-28 14:13:07 UTC)
orbiter warning: context deadline exceeded
Duration: 5.000707476s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [...X............X....................X...X........X.........] duration:
20.985729564s trace:
https://jaeger.dev.storj.io/trace/360e059013858fb1
FAIL (2025-12-28 14:13:06 UTC)
orbiter warning: context deadline exceeded
Duration: 20.985755394s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
20 out of 60 is failed (threshold=3) [.X..X.....X..X.X...XX......X......XXX.XX....XX......XX.XX.X.] duration:
5.730035979s trace:
https://jaeger.dev.storj.io/trace/7ed73ea0ef2fcf6e
FAIL (2025-12-28 14:13:02 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 28 14:12:26.955988030 (550h43m26.827071076s) [upload=482857]: Starting upload
Dec 28 14:12:26.956048295 (550h43m26.827131331s) [upload=482857]: Done starting upload
Dec 28 14:12:26.956220924 (550h43m26.827303959s) [upload=482857]: Uploading segments...
Dec 28 14:12:26.956304343 (550h43m26.827387378s) [upload=482857]: Got next segment. Inline: false
Dec 28 14:12:26.956356803 (550h43m26.827439838s) [upload=482857]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 14:12:26.956362854 (550h43m26.827445889s) [upload=482857]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 14:12:27.203346923 (550h43m27.074429968s) [seg_pos={0 0}, upload=482857]: Begin upload segment...
Dec 28 14:12:27.203420483 (550h43m27.074503528s) [seg_pos={0 0}, upload=482857]: Done begin upload segment.
Dec 28 14:12:27.203465909 (550h43m27.074548944s) [noise=true , address=45.91.116.10:4084 , piece_id=DODVFMCUC2SLGQBNAB2XJ7MUPDY346QOCHERDB2KLUXOZV47PMCQ, seg_pos={0 0}, upload=482857]: Uploading piece...
Dec 28 14:12:27.205011863 (550h43m27.076094898s) [noise=true , address=145.79.227.39:4015 , piece_id=BS3WL72BOIM5M5E3CIR6QAQTEBGAAMGL45KTBTNGFFRNQFGAOIGQ, seg_pos={0 0}, upload=482857]: Uploading piece...
Dec 28 14:12:27.205507086 (550h43m27.076590131s) [noise=true , address=145.79.227.36:4517 , piece_id=FSKK4MN5R5YCQPGEAUSFLGCKKSMQM6SHR3MU7XW4WGBBUPEHCDTA, seg_pos={0 0}, upload=482857]: Uploading piece...
Dec 28 14:12:27.206025684 (550h43m27.077108729s) [noise=true , address=145.79.227.7:4294 , piece_id=I5KWL6V3UGP2M75WIH2LD4DDFNIHCPKMB3LQ5AMFDINRXGWZTOGA, seg_pos={0 0}, upload=482857]: Uploading piece...
Dec 28 14:12:27.207686346 (550h43m27.078769401s) [noise=true , address=145.79.227.23:4546 , piece_id=PXCQYRS4XKGCI75GUD74NTZDS6B5PFDFWHP2UWHO6FFGZ7NYTJNA, seg_pos={0 0}, upload=482857]: Uploading piece...
Dec 28 14:12:27.209137098 (550h43m27.080220143s) [noise=true , address=45.91.116.13:4156 , piece_id=4JXOX4GL4OTZYYDTTEUYIYLKTYTVE6QUKSLAFICMS3J6MUNV5ZEQ, seg_pos={0 0}, upload=482857]: Uploading piece...
Dec 28 14:12:27.209241617 (550h43m27.080324662s) [noise=true , address=145.79.227.13:4802 , piece_id=KGJBMKAOESBSIRTFJASM4AVLIXTULAFQIDLC7BJOW4QNA7STADDQ, seg_pos={0 0}, upload=482857]: Uploading piece...
Dec 28 14:12:27.211032798 (550h43m27.082115833s) [noise=true , address=45.91.116.7:4009 , piece_id=BQEQBPOJ2DEA7SP52JUOBCE65O3CJIPNAOFRX6PCXZV5YYDCCNWQ, seg_pos={0 0}, upload=482857]: Uploading piece...
Dec 28 14:12:27.212335017 (550h43m27.083418062s) [noise=true , address=145.79.227.47:4267 , piece_id=U6AS4TE4KV4GSBTK6LNX6LJO6SVL6M3NN4G4XE6GWY4N3Y4REAZA, seg_pos={0 0}, upload=482857]: Uploading piece...
Dec 28 14:12:27.214760837 (550h43m27.085843872s) [noise=true , address=45.91.116.11:4119 , piece_id=DKKYFWRSICCYP6SD7AG4X5Y477QELMOQ7NPDQAW4QEUPOPW2XITQ, seg_pos={0 0}, upload=482857]: Uploading piece...
Dec 28 14:12:27.215345150 (550h43m27.086428195s) [noise=true , address=145.79.227.22:4511 , piece_id=CENRFEVSOLMISFMAKXW3HPKZA7Y6K72WDFUYJRTOYASV7JNWAFQQ, seg_pos={0 0}, upload=482857]: Uploading piece...
Dec 28 14:12:27.217163823 (550h43m27.088246868s) [noise=true , address=145.79.227.4:4185 , piece_id=WDD5X5TL57YJKVCAC5PVHPLKL7UBLQPRLXQL6AT5W4SGECLSRTWA, seg_pos={0 0}, upload=482857]: Uploading piece...
Dec 28 14:12:27.217605494 (550h43m27.088688539s) [noise=true , address=145.79.227.6:4258 , piece_id=TDXLR4XXCVIZRVNKRJAVTI2TV6YIWNC2MDLYKLNO254REB474J5Q, seg_pos={0 0}, upload=482857]: Uploading piece...
Dec 28 14:12:27.218542177 (550h43m27.089625222s) [noise=true , address=145.79.227.25:4618 , piece_id=PBB23LSOUMNTZNKGK43PCUVA4YADXDA4V6YZWJXD34XXGYINNGZA, seg_pos={0 0}, upload=482857]: Uploading piece...
Dec 28 14:12:27.220450450 (550h43m27.091533495s) [noise=true , address=145.79.227.37:4566 , piece_id=AESIVAA37WELHB4UNGKJVH73RW62TFZZYUCSEV6GSFZDRXM2XQGA, seg_pos={0 0}, upload=482857]: Uploading piece...
Dec 28 14:12:27.221721161 (550h43m27.092804206s) [noise=true , address=145.79.227.12:4765 , piece_id=PTT7UVJF7X6JUAGZSBNLGXCR43BNZPYMTZS26Y6INRMYLRVWW4SA, seg_pos={0 0}, upload=482857]: Uploading piece...
Dec 28 14:12:28.836487613 (550h43m28.707570658s) [upload=482857]: Next returned nil segment
Dec 28 14:12:28.836525865 (550h43m28.7076089s) [upload=482857]: Waiting for error group managing segments...
Dec 28 14:12:29.222239759 (550h43m29.093322814s) [noise=true , address=145.79.227.13:4802 , piece_id=KGJBMKAOESBSIRTFJASM4AVLIXTULAFQIDLC7BJOW4QNA7STADDQ, seg_pos={0 0}, upload=482857]: Done uploading piece. err: <nil>
Dec 28 14:12:29.235763600 (550h43m29.106846645s) [noise=true , address=145.79.227.25:4618 , piece_id=PBB23LSOUMNTZNKGK43PCUVA4YADXDA4V6YZWJXD34XXGYINNGZA, seg_pos={0 0}, upload=482857]: Done uploading piece. err: <nil>
Dec 28 14:12:29.238585754 (550h43m29.109668799s) [noise=true , address=145.79.227.39:4015 , piece_id=BS3WL72BOIM5M5E3CIR6QAQTEBGAAMGL45KTBTNGFFRNQFGAOIGQ, seg_pos={0 0}, upload=482857]: Done uploading piece. err: <nil>
Dec 28 14:12:29.249318180 (550h43m29.120401225s) [noise=true , address=145.79.227.22:4511 , piece_id=CENRFEVSOLMISFMAKXW3HPKZA7Y6K72WDFUYJRTOYASV7JNWAFQQ, seg_pos={0 0}, upload=482857]: Done uploading piece. err: <nil>
Dec 28 14:12:29.249642578 (550h43m29.120725613s) [noise=true , address=145.79.227.36:4517 , piece_id=FSKK4MN5R5YCQPGEAUSFLGCKKSMQM6SHR3MU7XW4WGBBUPEHCDTA, seg_pos={0 0}, upload=482857]: Done uploading piece. err: <nil>
Dec 28 14:12:29.268199995 (550h43m29.13928304s) [noise=true , address=145.79.227.47:4267 , piece_id=U6AS4TE4KV4GSBTK6LNX6LJO6SVL6M3NN4G4XE6GWY4N3Y4REAZA, seg_pos={0 0}, upload=482857]: Done uploading piece. err: <nil>
Dec 28 14:12:29.289329168 (550h43m29.160412204s) [noise=true , address=145.79.227.6:4258 , piece_id=TDXLR4XXCVIZRVNKRJAVTI2TV6YIWNC2MDLYKLNO254REB474J5Q, seg_pos={0 0}, upload=482857]: Done uploading piece. err: <nil>
Dec 28 14:12:29.301755379 (550h43m29.172838424s) [noise=true , address=145.79.227.4:4185 , piece_id=WDD5X5TL57YJKVCAC5PVHPLKL7UBLQPRLXQL6AT5W4SGECLSRTWA, seg_pos={0 0}, upload=482857]: Done uploading piece. err: <nil>
Dec 28 14:12:29.329926162 (550h43m29.201009197s) [noise=true , address=145.79.227.7:4294 , piece_id=I5KWL6V3UGP2M75WIH2LD4DDFNIHCPKMB3LQ5AMFDINRXGWZTOGA, seg_pos={0 0}, upload=482857]: Done uploading piece. err: <nil>
Dec 28 14:12:29.336949215 (550h43m29.20803225s) [noise=true , address=145.79.227.23:4546 , piece_id=PXCQYRS4XKGCI75GUD74NTZDS6B5PFDFWHP2UWHO6FFGZ7NYTJNA, seg_pos={0 0}, upload=482857]: Done uploading piece. err: <nil>
Dec 28 14:12:29.350750224 (550h43m29.221833269s) [noise=true , address=145.79.227.12:4765 , piece_id=PTT7UVJF7X6JUAGZSBNLGXCR43BNZPYMTZS26Y6INRMYLRVWW4SA, seg_pos={0 0}, upload=482857]: Done uploading piece. err: <nil>
Dec 28 14:12:29.440347803 (550h43m29.311430838s) [noise=true , address=45.91.116.13:4156 , piece_id=4JXOX4GL4OTZYYDTTEUYIYLKTYTVE6QUKSLAFICMS3J6MUNV5ZEQ, seg_pos={0 0}, upload=482857]: Done uploading piece. err: <nil>
Dec 28 14:12:29.465299544 (550h43m29.336382589s) [noise=true , address=45.91.116.11:4119 , piece_id=DKKYFWRSICCYP6SD7AG4X5Y477QELMOQ7NPDQAW4QEUPOPW2XITQ, seg_pos={0 0}, upload=482857]: Done uploading piece. err: <nil>
Dec 28 14:12:29.465323490 (550h43m29.336406525s) [seg_pos={0 0}, upload=482857]: Segment reached optimal threshold of 13 pieces.
Dec 28 14:12:29.465597592 (550h43m29.336680637s) [noise=true , address=145.79.227.37:4566 , piece_id=AESIVAA37WELHB4UNGKJVH73RW62TFZZYUCSEV6GSFZDRXM2XQGA, seg_pos={0 0}, upload=482857]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F): protocol: expected piece hash; context canceled; EOF
Dec 28 14:12:29.465605868 (550h43m29.336688903s) [noise=true , address=145.79.227.37:4566 , piece_id=AESIVAA37WELHB4UNGKJVH73RW62TFZZYUCSEV6GSFZDRXM2XQGA, seg_pos={0 0}, upload=482857]: Long tail cancellation due to optimal threshold
Dec 28 14:12:29.465747347 (550h43m29.336830392s) [noise=true , address=45.91.116.7:4009 , piece_id=BQEQBPOJ2DEA7SP52JUOBCE65O3CJIPNAOFRX6PCXZV5YYDCCNWQ, seg_pos={0 0}, upload=482857]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF
Dec 28 14:12:29.465757507 (550h43m29.336840542s) [noise=true , address=45.91.116.7:4009 , piece_id=BQEQBPOJ2DEA7SP52JUOBCE65O3CJIPNAOFRX6PCXZV5YYDCCNWQ, seg_pos={0 0}, upload=482857]: Long tail cancellation due to optimal threshold
Dec 28 14:12:29.465767195 (550h43m29.336850241s) [noise=true , address=45.91.116.10:4084 , piece_id=DODVFMCUC2SLGQBNAB2XJ7MUPDY346QOCHERDB2KLUXOZV47PMCQ, seg_pos={0 0}, upload=482857]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF
Dec 28 14:12:29.465775792 (550h43m29.336858827s) [noise=true , address=45.91.116.10:4084 , piece_id=DODVFMCUC2SLGQBNAB2XJ7MUPDY346QOCHERDB2KLUXOZV47PMCQ, seg_pos={0 0}, upload=482857]: Long tail cancellation due to optimal threshold
Dec 28 14:12:29.465793265 (550h43m29.33687631s) [seg_pos={0 0}, upload=482857]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 14:12:29.465897784 (550h43m29.336980829s) [upload=482857]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 14:12:29.465903876 (550h43m29.336986921s) [upload=482857]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 14:12:29.771625107 (550h43m29.642708153s) [upload=482857]: Done uploading segments...
Dec 28 14:12:29.771650276 (550h43m29.642733311s) [upload=482857]: Upload finished. err: <nil>
Duration: 5.730056266s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
20 out of 60 is failed (threshold=3) [..X..X.....X..X.X...XX......X......XXX.XX....XX......XX.XX.X] duration:
30.00351531s trace:
https://jaeger.dev.storj.io/trace/7ed73ea0ef2fcf6e
FAIL (2025-12-28 14:12:56 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 28 14:12:26.955988030 (550h43m26.827071076s) [upload=482857]: Starting upload
Dec 28 14:12:26.956048295 (550h43m26.827131331s) [upload=482857]: Done starting upload
Dec 28 14:12:26.956220924 (550h43m26.827303959s) [upload=482857]: Uploading segments...
Dec 28 14:12:26.956304343 (550h43m26.827387378s) [upload=482857]: Got next segment. Inline: false
Dec 28 14:12:26.956356803 (550h43m26.827439838s) [upload=482857]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 14:12:26.956362854 (550h43m26.827445889s) [upload=482857]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 14:12:27.203346923 (550h43m27.074429968s) [seg_pos={0 0}, upload=482857]: Begin upload segment...
Dec 28 14:12:27.203420483 (550h43m27.074503528s) [seg_pos={0 0}, upload=482857]: Done begin upload segment.
Dec 28 14:12:27.203465909 (550h43m27.074548944s) [noise=true , address=45.91.116.10:4084 , piece_id=DODVFMCUC2SLGQBNAB2XJ7MUPDY346QOCHERDB2KLUXOZV47PMCQ, seg_pos={0 0}, upload=482857]: Uploading piece...
Dec 28 14:12:27.205011863 (550h43m27.076094898s) [noise=true , address=145.79.227.39:4015 , piece_id=BS3WL72BOIM5M5E3CIR6QAQTEBGAAMGL45KTBTNGFFRNQFGAOIGQ, seg_pos={0 0}, upload=482857]: Uploading piece...
Dec 28 14:12:27.205507086 (550h43m27.076590131s) [noise=true , address=145.79.227.36:4517 , piece_id=FSKK4MN5R5YCQPGEAUSFLGCKKSMQM6SHR3MU7XW4WGBBUPEHCDTA, seg_pos={0 0}, upload=482857]: Uploading piece...
Dec 28 14:12:27.206025684 (550h43m27.077108729s) [noise=true , address=145.79.227.7:4294 , piece_id=I5KWL6V3UGP2M75WIH2LD4DDFNIHCPKMB3LQ5AMFDINRXGWZTOGA, seg_pos={0 0}, upload=482857]: Uploading piece...
Dec 28 14:12:27.207686346 (550h43m27.078769401s) [noise=true , address=145.79.227.23:4546 , piece_id=PXCQYRS4XKGCI75GUD74NTZDS6B5PFDFWHP2UWHO6FFGZ7NYTJNA, seg_pos={0 0}, upload=482857]: Uploading piece...
Dec 28 14:12:27.209137098 (550h43m27.080220143s) [noise=true , address=45.91.116.13:4156 , piece_id=4JXOX4GL4OTZYYDTTEUYIYLKTYTVE6QUKSLAFICMS3J6MUNV5ZEQ, seg_pos={0 0}, upload=482857]: Uploading piece...
Dec 28 14:12:27.209241617 (550h43m27.080324662s) [noise=true , address=145.79.227.13:4802 , piece_id=KGJBMKAOESBSIRTFJASM4AVLIXTULAFQIDLC7BJOW4QNA7STADDQ, seg_pos={0 0}, upload=482857]: Uploading piece...
Dec 28 14:12:27.211032798 (550h43m27.082115833s) [noise=true , address=45.91.116.7:4009 , piece_id=BQEQBPOJ2DEA7SP52JUOBCE65O3CJIPNAOFRX6PCXZV5YYDCCNWQ, seg_pos={0 0}, upload=482857]: Uploading piece...
Dec 28 14:12:27.212335017 (550h43m27.083418062s) [noise=true , address=145.79.227.47:4267 , piece_id=U6AS4TE4KV4GSBTK6LNX6LJO6SVL6M3NN4G4XE6GWY4N3Y4REAZA, seg_pos={0 0}, upload=482857]: Uploading piece...
Dec 28 14:12:27.214760837 (550h43m27.085843872s) [noise=true , address=45.91.116.11:4119 , piece_id=DKKYFWRSICCYP6SD7AG4X5Y477QELMOQ7NPDQAW4QEUPOPW2XITQ, seg_pos={0 0}, upload=482857]: Uploading piece...
Dec 28 14:12:27.215345150 (550h43m27.086428195s) [noise=true , address=145.79.227.22:4511 , piece_id=CENRFEVSOLMISFMAKXW3HPKZA7Y6K72WDFUYJRTOYASV7JNWAFQQ, seg_pos={0 0}, upload=482857]: Uploading piece...
Dec 28 14:12:27.217163823 (550h43m27.088246868s) [noise=true , address=145.79.227.4:4185 , piece_id=WDD5X5TL57YJKVCAC5PVHPLKL7UBLQPRLXQL6AT5W4SGECLSRTWA, seg_pos={0 0}, upload=482857]: Uploading piece...
Dec 28 14:12:27.217605494 (550h43m27.088688539s) [noise=true , address=145.79.227.6:4258 , piece_id=TDXLR4XXCVIZRVNKRJAVTI2TV6YIWNC2MDLYKLNO254REB474J5Q, seg_pos={0 0}, upload=482857]: Uploading piece...
Dec 28 14:12:27.218542177 (550h43m27.089625222s) [noise=true , address=145.79.227.25:4618 , piece_id=PBB23LSOUMNTZNKGK43PCUVA4YADXDA4V6YZWJXD34XXGYINNGZA, seg_pos={0 0}, upload=482857]: Uploading piece...
Dec 28 14:12:27.220450450 (550h43m27.091533495s) [noise=true , address=145.79.227.37:4566 , piece_id=AESIVAA37WELHB4UNGKJVH73RW62TFZZYUCSEV6GSFZDRXM2XQGA, seg_pos={0 0}, upload=482857]: Uploading piece...
Dec 28 14:12:27.221721161 (550h43m27.092804206s) [noise=true , address=145.79.227.12:4765 , piece_id=PTT7UVJF7X6JUAGZSBNLGXCR43BNZPYMTZS26Y6INRMYLRVWW4SA, seg_pos={0 0}, upload=482857]: Uploading piece...
Dec 28 14:12:28.836487613 (550h43m28.707570658s) [upload=482857]: Next returned nil segment
Dec 28 14:12:28.836525865 (550h43m28.7076089s) [upload=482857]: Waiting for error group managing segments...
Dec 28 14:12:29.222239759 (550h43m29.093322814s) [noise=true , address=145.79.227.13:4802 , piece_id=KGJBMKAOESBSIRTFJASM4AVLIXTULAFQIDLC7BJOW4QNA7STADDQ, seg_pos={0 0}, upload=482857]: Done uploading piece. err: <nil>
Dec 28 14:12:29.235763600 (550h43m29.106846645s) [noise=true , address=145.79.227.25:4618 , piece_id=PBB23LSOUMNTZNKGK43PCUVA4YADXDA4V6YZWJXD34XXGYINNGZA, seg_pos={0 0}, upload=482857]: Done uploading piece. err: <nil>
Dec 28 14:12:29.238585754 (550h43m29.109668799s) [noise=true , address=145.79.227.39:4015 , piece_id=BS3WL72BOIM5M5E3CIR6QAQTEBGAAMGL45KTBTNGFFRNQFGAOIGQ, seg_pos={0 0}, upload=482857]: Done uploading piece. err: <nil>
Dec 28 14:12:29.249318180 (550h43m29.120401225s) [noise=true , address=145.79.227.22:4511 , piece_id=CENRFEVSOLMISFMAKXW3HPKZA7Y6K72WDFUYJRTOYASV7JNWAFQQ, seg_pos={0 0}, upload=482857]: Done uploading piece. err: <nil>
Dec 28 14:12:29.249642578 (550h43m29.120725613s) [noise=true , address=145.79.227.36:4517 , piece_id=FSKK4MN5R5YCQPGEAUSFLGCKKSMQM6SHR3MU7XW4WGBBUPEHCDTA, seg_pos={0 0}, upload=482857]: Done uploading piece. err: <nil>
Dec 28 14:12:29.268199995 (550h43m29.13928304s) [noise=true , address=145.79.227.47:4267 , piece_id=U6AS4TE4KV4GSBTK6LNX6LJO6SVL6M3NN4G4XE6GWY4N3Y4REAZA, seg_pos={0 0}, upload=482857]: Done uploading piece. err: <nil>
Dec 28 14:12:29.289329168 (550h43m29.160412204s) [noise=true , address=145.79.227.6:4258 , piece_id=TDXLR4XXCVIZRVNKRJAVTI2TV6YIWNC2MDLYKLNO254REB474J5Q, seg_pos={0 0}, upload=482857]: Done uploading piece. err: <nil>
Dec 28 14:12:29.301755379 (550h43m29.172838424s) [noise=true , address=145.79.227.4:4185 , piece_id=WDD5X5TL57YJKVCAC5PVHPLKL7UBLQPRLXQL6AT5W4SGECLSRTWA, seg_pos={0 0}, upload=482857]: Done uploading piece. err: <nil>
Dec 28 14:12:29.329926162 (550h43m29.201009197s) [noise=true , address=145.79.227.7:4294 , piece_id=I5KWL6V3UGP2M75WIH2LD4DDFNIHCPKMB3LQ5AMFDINRXGWZTOGA, seg_pos={0 0}, upload=482857]: Done uploading piece. err: <nil>
Dec 28 14:12:29.336949215 (550h43m29.20803225s) [noise=true , address=145.79.227.23:4546 , piece_id=PXCQYRS4XKGCI75GUD74NTZDS6B5PFDFWHP2UWHO6FFGZ7NYTJNA, seg_pos={0 0}, upload=482857]: Done uploading piece. err: <nil>
Dec 28 14:12:29.350750224 (550h43m29.221833269s) [noise=true , address=145.79.227.12:4765 , piece_id=PTT7UVJF7X6JUAGZSBNLGXCR43BNZPYMTZS26Y6INRMYLRVWW4SA, seg_pos={0 0}, upload=482857]: Done uploading piece. err: <nil>
Dec 28 14:12:29.440347803 (550h43m29.311430838s) [noise=true , address=45.91.116.13:4156 , piece_id=4JXOX4GL4OTZYYDTTEUYIYLKTYTVE6QUKSLAFICMS3J6MUNV5ZEQ, seg_pos={0 0}, upload=482857]: Done uploading piece. err: <nil>
Dec 28 14:12:29.465299544 (550h43m29.336382589s) [noise=true , address=45.91.116.11:4119 , piece_id=DKKYFWRSICCYP6SD7AG4X5Y477QELMOQ7NPDQAW4QEUPOPW2XITQ, seg_pos={0 0}, upload=482857]: Done uploading piece. err: <nil>
Dec 28 14:12:29.465323490 (550h43m29.336406525s) [seg_pos={0 0}, upload=482857]: Segment reached optimal threshold of 13 pieces.
Dec 28 14:12:29.465597592 (550h43m29.336680637s) [noise=true , address=145.79.227.37:4566 , piece_id=AESIVAA37WELHB4UNGKJVH73RW62TFZZYUCSEV6GSFZDRXM2XQGA, seg_pos={0 0}, upload=482857]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F): protocol: expected piece hash; context canceled; EOF
Dec 28 14:12:29.465605868 (550h43m29.336688903s) [noise=true , address=145.79.227.37:4566 , piece_id=AESIVAA37WELHB4UNGKJVH73RW62TFZZYUCSEV6GSFZDRXM2XQGA, seg_pos={0 0}, upload=482857]: Long tail cancellation due to optimal threshold
Dec 28 14:12:29.465747347 (550h43m29.336830392s) [noise=true , address=45.91.116.7:4009 , piece_id=BQEQBPOJ2DEA7SP52JUOBCE65O3CJIPNAOFRX6PCXZV5YYDCCNWQ, seg_pos={0 0}, upload=482857]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF
Dec 28 14:12:29.465757507 (550h43m29.336840542s) [noise=true , address=45.91.116.7:4009 , piece_id=BQEQBPOJ2DEA7SP52JUOBCE65O3CJIPNAOFRX6PCXZV5YYDCCNWQ, seg_pos={0 0}, upload=482857]: Long tail cancellation due to optimal threshold
Dec 28 14:12:29.465767195 (550h43m29.336850241s) [noise=true , address=45.91.116.10:4084 , piece_id=DODVFMCUC2SLGQBNAB2XJ7MUPDY346QOCHERDB2KLUXOZV47PMCQ, seg_pos={0 0}, upload=482857]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF
Dec 28 14:12:29.465775792 (550h43m29.336858827s) [noise=true , address=45.91.116.10:4084 , piece_id=DODVFMCUC2SLGQBNAB2XJ7MUPDY346QOCHERDB2KLUXOZV47PMCQ, seg_pos={0 0}, upload=482857]: Long tail cancellation due to optimal threshold
Dec 28 14:12:29.465793265 (550h43m29.33687631s) [seg_pos={0 0}, upload=482857]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 14:12:29.465897784 (550h43m29.336980829s) [upload=482857]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 14:12:29.465903876 (550h43m29.336986921s) [upload=482857]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 14:12:29.771625107 (550h43m29.642708153s) [upload=482857]: Done uploading segments...
Dec 28 14:12:29.771650276 (550h43m29.642733311s) [upload=482857]: Upload finished. err: <nil>
Duration: 30.003539816s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [..............X.............................................] duration:
11.616631098s trace:
https://jaeger.dev.storj.io/trace/5fe9d6dbb79fbe44
FAIL (2025-12-28 14:12:30 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
67.204.205:4027 , piece_id=PAPNMRJ7E47VOMOXPBBDWKZJHLRMXTWVR5VW3J7KNQSSMHOHWOAQ, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.967737953 (550h8m4.838820988s) [noise=true , address=185.167.204.206:4072 , piece_id=WXORBDMA3KW7WZ67A4FJT6URJRFADUJ7X2PAHIGNG47T3TGMXVTQ, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.968193240 (550h8m4.839276285s) [noise=true , address=185.167.204.205:4030 , piece_id=R5DLI4ZTLOVGO56GPLJL5ZNHYSXOSYNJQHFIO4MVVNHWWBMEOV2Q, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.968780638 (550h8m4.839863683s) [noise=true , address=89.149.201.147:4019 , piece_id=MK7ORCLHO3DOWIOVSOWIQMPJ447PHRQGXP3UKITIRTYJSU5MPCCQ, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.969308924 (550h8m4.840391959s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.970088349 (550h8m4.841171404s) [noise=true , address=81.171.14.70:4039 , piece_id=EDHPQWLQZTP7MKPGEVMP7JXNTW7LFWSXCTRTSXPCBIQPROX76WRA, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.970325381 (550h8m4.841408426s) [noise=true , address=81.171.14.70:4036 , piece_id=CICNNV7JE6TBNC4RAPXKCEUSJAADOUVGSVIVQISM3ZB4COAZQDDA, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.970783813 (550h8m4.841866858s) [noise=true , address=81.171.14.70:4044 , piece_id=DGN74IG6IWAHLEMO6NLL4TJBXZ6ONTN6QUB6HXY6K7SSO3LYOYIQ, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.971205406 (550h8m4.842288441s) [noise=true , address=81.171.14.70:4048 , piece_id=TLT47ORGUKGF3MIIEAX7D22VR2FO6W6NL3VFJNHKWD4L3LIMFDLA, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.971606299 (550h8m4.842689344s) [noise=true , address=185.167.204.206:4067 , piece_id=VSKTG5RH32N7OEVWJGQVB6WJ72TEVF6DDK25W7NOH4RAXUP52LTQ, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.972162538 (550h8m4.843245583s) [noise=true , address=89.149.201.147:4002 , piece_id=G2PABNTDWCK4MKOZF6PONBMRNIXI45JTHJMAZFPEKPEJ3PQ7MTBA, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:05.795167814 (550h8m5.666250849s) [upload=482337]: Next returned nil segment
Dec 28 13:37:05.795184566 (550h8m5.666267611s) [upload=482337]: Waiting for error group managing segments...
Dec 28 13:37:06.178573801 (550h8m6.049656836s) [noise=true , address=185.167.204.206:4049 , piece_id=6RG2DZV6FPTIRE63ZNNST42TSWEBQ456YXIPPVYHIXV55GWFVJ4Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.183060835 (550h8m6.05414391s) [noise=true , address=185.167.204.206:4072 , piece_id=WXORBDMA3KW7WZ67A4FJT6URJRFADUJ7X2PAHIGNG47T3TGMXVTQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.183411202 (550h8m6.054494237s) [noise=true , address=185.167.204.206:4064 , piece_id=OG3G74OU23CEMKA54WQE3DOJ6AKAXBLZ23QWOQC4UEGQQN4JSMXQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.183791146 (550h8m6.054874181s) [noise=true , address=185.167.204.205:4029 , piece_id=6JQB6ZSCQZKWH3JWCQ4DY6BBAX73KC4Y53PRSMEMLWURZ5QWWQQQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.183973743 (550h8m6.055056778s) [noise=true , address=185.167.204.206:4053 , piece_id=YDH77S6JLKJ6IYS6DRUCELP5EJHIC2O44QBO7MIOEUYBML4GIBHA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.184465369 (550h8m6.055548414s) [noise=true , address=185.167.204.205:4008 , piece_id=RTUZPEXBB6JNSMC5YOFQQGDM57YD233W4756ZBQVMFNOVX6YKB4A, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.184501989 (550h8m6.055585024s) [noise=true , address=185.167.204.205:4030 , piece_id=R5DLI4ZTLOVGO56GPLJL5ZNHYSXOSYNJQHFIO4MVVNHWWBMEOV2Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.184856224 (550h8m6.055939259s) [noise=true , address=185.167.204.205:4009 , piece_id=V6RA7Y7FKY4DCPTRRF3BYNBALWDY5VTPXMDUXFE3353KUXLSQJEA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.185063518 (550h8m6.056146563s) [noise=true , address=185.167.204.205:4016 , piece_id=JB7SC3J2VR4U33HNQS2A2CA75THT3XLQCOIFHAJJEK7MQGY2SYAQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.185288617 (550h8m6.056371652s) [noise=true , address=185.167.204.205:4027 , piece_id=PAPNMRJ7E47VOMOXPBBDWKZJHLRMXTWVR5VW3J7KNQSSMHOHWOAQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.186241401 (550h8m6.057324436s) [noise=true , address=185.167.204.205:4024 , piece_id=FM5IMWRSJJ4RY72IX2OOYC54BFKL4FWE3UOAN3GJ26MLH345COVA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.193113658 (550h8m6.064196703s) [noise=true , address=185.167.204.205:4005 , piece_id=R4DFQJZBPJ5WJWHPODV6ILHUDIRBUSAFOMVNCLXB4UNMI6EXTKHA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.193791408 (550h8m6.064874453s) [noise=true , address=185.167.204.206:4068 , piece_id=DAI37M67FWCSVQTQOZ6RSSOJDGUVZO6SNGAHJJ44G3PGS4OJZEMA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.194292262 (550h8m6.065375307s) [noise=true , address=185.167.204.206:4046 , piece_id=VWLVHMBMJ63DQEWYBIKH3Y3NTVWFNW5OXZKMYXXBHVI3VWYXLLPQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.194302652 (550h8m6.065385687s) [noise=true , address=185.167.204.206:4067 , piece_id=VSKTG5RH32N7OEVWJGQVB6WJ72TEVF6DDK25W7NOH4RAXUP52LTQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.195359193 (550h8m6.066442228s) [noise=true , address=185.167.204.205:4012 , piece_id=LV6RUA5BWXG4DMSUVZDBGUAF5IBWUGMLJYUKNIXHRVDEBIOW4Q2Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.198423448 (550h8m6.069506493s) [noise=true , address=185.167.204.206:4070 , piece_id=5A7YYQYREIZRZDR7TRSE7XFW765JVAV7FX6LKWUI2OXAA74XN4CA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.888968408 (550h8m6.760051453s) [noise=true , address=81.171.14.70:4044 , piece_id=DGN74IG6IWAHLEMO6NLL4TJBXZ6ONTN6QUB6HXY6K7SSO3LYOYIQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.892350789 (550h8m6.763433834s) [noise=true , address=81.171.14.70:4039 , piece_id=EDHPQWLQZTP7MKPGEVMP7JXNTW7LFWSXCTRTSXPCBIQPROX76WRA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.904477160 (550h8m6.775560195s) [noise=true , address=81.171.14.70:4040 , piece_id=DYF4GSZBAKQ3H63LTVCMEFWMABY3FXHOJCNVIFYRKLF2J6B52K5A, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.128088281 (550h8m6.999171326s) [noise=true , address=81.171.14.70:4036 , piece_id=CICNNV7JE6TBNC4RAPXKCEUSJAADOUVGSVIVQISM3ZB4COAZQDDA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.430242142 (550h8m7.301325177s) [noise=true , address=89.149.201.147:4009 , piece_id=EVW2NQFYU57PC2IIVXLYIEBMYXJNIC2CLCHW6PS6NYIPCFRUVHDA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.505304332 (550h8m7.376387367s) [noise=true , address=89.149.201.147:4019 , piece_id=MK7ORCLHO3DOWIOVSOWIQMPJ447PHRQGXP3UKITIRTYJSU5MPCCQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.575461543 (550h8m7.446544609s) [noise=true , address=81.171.14.70:4048 , piece_id=TLT47ORGUKGF3MIIEAX7D22VR2FO6W6NL3VFJNHKWD4L3LIMFDLA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.599667265 (550h8m7.4707503s) [noise=true , address=89.149.201.147:4023 , piece_id=5IINX2GVRFLRX2QH24OO2DDANOKOTXTDR6AGVJBGSJ7IAZQ3CASA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.928390728 (550h8m7.799473773s) [noise=true , address=89.149.201.147:4002 , piece_id=G2PABNTDWCK4MKOZF6PONBMRNIXI45JTHJMAZFPEKPEJ3PQ7MTBA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.928417148 (550h8m7.799500193s) [seg_pos={0 0}, upload=482337]: Segment reached optimal threshold of 26 pieces.
Dec 28 13:37:07.928656303 (550h8m7.799739348s) [noise=true , address=81.171.14.70:4030 , piece_id=74YDIKUST7X3JBXVQHVLFHIKH5A4DYH5TNL3I4XUA5A6AHQO2DRA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZoRwa8wLw66Pjumy1ybp6wrZ79RrSU55uYZpwaY3xi95iSYde): protocol: expected piece hash; context canceled; EOF
Dec 28 13:37:07.928664990 (550h8m7.799748035s) [noise=true , address=81.171.14.70:4030 , piece_id=74YDIKUST7X3JBXVQHVLFHIKH5A4DYH5TNL3I4XUA5A6AHQO2DRA, seg_pos={0 0}, upload=482337]: Long tail cancellation due to optimal threshold
Dec 28 13:37:07.928723351 (550h8m7.799806386s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wgtphHKaTejC1qrJ98sDysBwtj2Zeqw2c5UFk1HMqfY8Buvr): protocol: expected piece hash; context canceled; EOF
Dec 28 13:37:07.928733922 (550h8m7.799816957s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Long tail cancellation due to optimal threshold
Dec 28 13:37:07.928754921 (550h8m7.799837966s) [seg_pos={0 0}, upload=482337]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 28 13:37:07.928870170 (550h8m7.799953215s) [upload=482337]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 13:37:07.928878627 (550h8m7.799961672s) [upload=482337]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 13:37:08.166321536 (550h8m8.037404571s) [upload=482337]: Done uploading segments...
Dec 28 13:37:08.166341444 (550h8m8.037424479s) [upload=482337]: Upload finished. err: <nil>
Duration: 11.616670783s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [.....X....................................X........X...X....] duration:
33.309863203s trace:
https://jaeger.dev.storj.io/trace/36d5af3c761e71b1
FAIL (2025-12-28 14:12:29 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
7.49:4360 , piece_id=QDT5N5LDL2HCVXPOOSXWD2TOEZDF2JZWFLDFYSIMN647FI3ZORUQ, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.372193736 (550h40m10.243276781s) [noise=true , address=77.90.31.12:4618 , piece_id=45WRZ26CPUODOAYF5B6RF37DNMDDQWWJUDK7BFKYS3K3LOCNYJCA, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.372518505 (550h40m10.24360155s) [noise=true , address=77.90.31.13:4646 , piece_id=5CDMQF7GG5PDPHIPMFZWDBKWH5ANO3UIX257LJPCWRITNOA7GY4A, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.372848052 (550h40m10.243931097s) [noise=true , address=64.49.13.170:4721 , piece_id=U76TI7IPAHPZAS6XLG66E4L2CGCYH43NH4ZHTYXMXKZR6SGE7SKQ, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.373163072 (550h40m10.244246117s) [noise=true , address=77.90.31.11:4565 , piece_id=4SBLRZK3K4ED73PJ64BI6ANKBWB2AAHQA4SBUNH6CJ63GN3FJR2Q, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.373445089 (550h40m10.244528134s) [noise=true , address=145.79.227.34:4911 , piece_id=24WE4EMZSL2JHQZ7PESCMLYCZOWGDTGJD2XNLUUJEFPSYLRI5E7A, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.373768855 (550h40m10.24485189s) [noise=true , address=77.90.31.18:4819 , piece_id=OYXPAN3EK4QRDEC6VHIDL3GSGEY27G72KII6HTGUMSPYLHCYBNEA, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.374285519 (550h40m10.245368564s) [noise=true , address=77.90.31.14:4678 , piece_id=VBGPSGMNDQZYY7UKRINAV6B3FVI5BPDPYS37FQIJCMLL65F3XOEQ, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.374710689 (550h40m10.245793724s) [noise=true , address=145.79.227.40:4053 , piece_id=RO4ZZQCKCC7T4T6SOT7GGTY626BJ6ESLQG6KATTPNIUM3ICUKHAQ, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.375099329 (550h40m10.246182364s) [noise=true , address=77.90.31.19:4859 , piece_id=A4HD3FPDPCCMHRFBTS3GNGB323RYD5C5AW4ZVSNOTXFI7FLIV37Q, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.376073433 (550h40m10.247156468s) [noise=true , address=77.90.31.15:4704 , piece_id=DXWLFOIKOPE33LMEQECUJVIGRG3LPADUMULLWI4ZBHEOQHD43ULQ, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.376542387 (550h40m10.247625422s) [noise=true , address=145.79.227.44:4221 , piece_id=H72XCWLKYJAXYGGDIWN67H42RUIEOJCPUSWT5IJMFPKETH7LK3KA, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.377379761 (550h40m10.248462796s) [noise=true , address=145.79.227.32:4863 , piece_id=27TOXK44RIF3DV3PDY4RBP6HEFRQJE6N7LNLRJKSU52YCB6A3JKA, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.378022765 (550h40m10.24910581s) [noise=true , address=145.79.227.30:4789 , piece_id=4QT5IUPIK7BOHJROEK2EBFTW434OKC75FZNENMYYCRMBZ7E5BUTA, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.378821687 (550h40m10.249904722s) [noise=true , address=145.79.227.2:4119 , piece_id=EQM3WMBQ7GXRWRBZK4ZWCI23K4LSLXW6QPEABHGMWRHEYUYANKSQ, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:11.676223695 (550h40m11.54730674s) [upload=482805]: Next returned nil segment
Dec 28 14:09:11.676238083 (550h40m11.547321128s) [upload=482805]: Waiting for error group managing segments...
Dec 28 14:09:12.006762275 (550h40m11.87784531s) [noise=true , address=64.49.13.170:4721 , piece_id=U76TI7IPAHPZAS6XLG66E4L2CGCYH43NH4ZHTYXMXKZR6SGE7SKQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.250307967 (550h40m12.121391012s) [noise=true , address=145.79.227.2:4119 , piece_id=EQM3WMBQ7GXRWRBZK4ZWCI23K4LSLXW6QPEABHGMWRHEYUYANKSQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.269380734 (550h40m12.140463769s) [noise=true , address=145.79.227.49:4360 , piece_id=QDT5N5LDL2HCVXPOOSXWD2TOEZDF2JZWFLDFYSIMN647FI3ZORUQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.290818335 (550h40m12.16190138s) [noise=true , address=145.79.227.32:4863 , piece_id=27TOXK44RIF3DV3PDY4RBP6HEFRQJE6N7LNLRJKSU52YCB6A3JKA, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.467679327 (550h40m12.338762362s) [noise=true , address=45.91.116.8:4072 , piece_id=GEVTX6LLENPLN3UQLSGTFTP3DO52IRYUIQFIWC4XMOC3ZF6V4GTQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.486615595 (550h40m12.35769863s) [noise=true , address=77.90.31.12:4618 , piece_id=45WRZ26CPUODOAYF5B6RF37DNMDDQWWJUDK7BFKYS3K3LOCNYJCA, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.566542810 (550h40m12.437625845s) [noise=true , address=77.90.31.19:4859 , piece_id=A4HD3FPDPCCMHRFBTS3GNGB323RYD5C5AW4ZVSNOTXFI7FLIV37Q, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.609337578 (550h40m12.480420623s) [noise=true , address=77.90.31.13:4646 , piece_id=5CDMQF7GG5PDPHIPMFZWDBKWH5ANO3UIX257LJPCWRITNOA7GY4A, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.640856565 (550h40m12.511939611s) [noise=true , address=45.91.116.5:4322 , piece_id=SS6DOCSJHY5Y3DPWMYX45F4VYYXR44JDLYQ2YPY5XIIAYKMAG7LQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.645490329 (550h40m12.516573364s) [noise=true , address=77.90.31.18:4819 , piece_id=OYXPAN3EK4QRDEC6VHIDL3GSGEY27G72KII6HTGUMSPYLHCYBNEA, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:13.103710286 (550h40m12.974793331s) [noise=true , address=145.79.227.34:4911 , piece_id=24WE4EMZSL2JHQZ7PESCMLYCZOWGDTGJD2XNLUUJEFPSYLRI5E7A, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:13.209917141 (550h40m13.081000176s) [noise=true , address=145.79.227.44:4221 , piece_id=H72XCWLKYJAXYGGDIWN67H42RUIEOJCPUSWT5IJMFPKETH7LK3KA, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:13.218355468 (550h40m13.089438514s) [noise=true , address=77.90.31.11:4565 , piece_id=4SBLRZK3K4ED73PJ64BI6ANKBWB2AAHQA4SBUNH6CJ63GN3FJR2Q, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:13.255968741 (550h40m13.127051806s) [noise=true , address=145.79.227.30:4789 , piece_id=4QT5IUPIK7BOHJROEK2EBFTW434OKC75FZNENMYYCRMBZ7E5BUTA, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:13.255996102 (550h40m13.127079137s) [seg_pos={0 0}, upload=482805]: Segment reached optimal threshold of 14 pieces.
Dec 28 14:09:13.256214589 (550h40m13.127297634s) [noise=true , address=145.79.227.40:4053 , piece_id=RO4ZZQCKCC7T4T6SOT7GGTY626BJ6ESLQG6KATTPNIUM3ICUKHAQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DqyTc4MHvxutFXs2WWRCk8MSuxF3Y1HayyQY4Y5sy8cqDLWuG): protocol: expected piece hash; context canceled; EOF
Dec 28 14:09:13.256223686 (550h40m13.127306731s) [noise=true , address=145.79.227.40:4053 , piece_id=RO4ZZQCKCC7T4T6SOT7GGTY626BJ6ESLQG6KATTPNIUM3ICUKHAQ, seg_pos={0 0}, upload=482805]: Long tail cancellation due to optimal threshold
Dec 28 14:09:13.256512506 (550h40m13.127595551s) [noise=true , address=77.90.31.15:4704 , piece_id=DXWLFOIKOPE33LMEQECUJVIGRG3LPADUMULLWI4ZBHEOQHD43ULQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14tJRdZ1guwwsYwPf3LtENRWqYx2jBQpi7uAMMPaB32Gks9zmj): protocol: expected piece hash; context canceled; EOF
Dec 28 14:09:13.256520060 (550h40m13.127603105s) [noise=true , address=77.90.31.15:4704 , piece_id=DXWLFOIKOPE33LMEQECUJVIGRG3LPADUMULLWI4ZBHEOQHD43ULQ, seg_pos={0 0}, upload=482805]: Long tail cancellation due to optimal threshold
Dec 28 14:09:13.256649106 (550h40m13.127732151s) [noise=true , address=77.90.31.14:4678 , piece_id=VBGPSGMNDQZYY7UKRINAV6B3FVI5BPDPYS37FQIJCMLL65F3XOEQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12beu3SUZMPNWfbeHV4tLtx9UVZoEHh4Kt7oBhT59w3bGBE1ovP): protocol: write tcp 188.166.225.170:57374->77.90.31.14:4678: use of closed network connection; write tcp 188.166.225.170:57374->77.90.31.14:4678: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57374->77.90.31.14:4678: use of closed network connection
Dec 28 14:09:13.256655819 (550h40m13.127738864s) [noise=true , address=77.90.31.14:4678 , piece_id=VBGPSGMNDQZYY7UKRINAV6B3FVI5BPDPYS37FQIJCMLL65F3XOEQ, seg_pos={0 0}, upload=482805]: Long tail cancellation due to optimal threshold
Dec 28 14:09:13.256752974 (550h40m13.127836009s) [noise=true , address=64.49.13.169:4710 , piece_id=SANL7TVJFES63XD46OKQL4KIJ6YVHSEWPZKMRISEKC3ZKWQBOMIA, seg_pos={0 0}, upload=482805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:35680->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:35680->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35680->64.49.13.169:4710: use of closed network connection
Dec 28 14:09:13.256758825 (550h40m13.12784186s) [noise=true , address=64.49.13.169:4710 , piece_id=SANL7TVJFES63XD46OKQL4KIJ6YVHSEWPZKMRISEKC3ZKWQBOMIA, seg_pos={0 0}, upload=482805]: Long tail cancellation due to optimal threshold
Dec 28 14:09:13.256776549 (550h40m13.127859594s) [seg_pos={0 0}, upload=482805]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 14:09:13.256856591 (550h40m13.127939626s) [upload=482805]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 14:09:13.256862432 (550h40m13.127945467s) [upload=482805]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 14:09:13.550363717 (550h40m13.421446782s) [upload=482805]: Done uploading segments...
Dec 28 14:09:13.550405126 (550h40m13.421488161s) [upload=482805]: Upload finished. err: <nil>
Duration: 33.309877752s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
52 out of 60 is failed (threshold=6) [XX.XXXXXXX.XXXXXXXXXXXX.XXXXXXX.XX..XXXXXXXXXXXXXXXXXXXXX.X.] duration:
3.544922299s trace:
https://jaeger.dev.storj.io/trace/92a90438d044faa
FAIL (2025-12-28 14:12:23 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.545077304s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [.....................................................X...X..] duration:
19.791847525s trace:
https://jaeger.dev.storj.io/trace/102cf391e061ae77
FAIL (2025-12-28 14:12:09 UTC)
orbiter warning: context deadline exceeded
Duration: 19.791912518s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [......X....................................X........X...X...] duration:
40.505540258s trace:
https://jaeger.dev.storj.io/trace/36d5af3c761e71b1
FAIL (2025-12-28 14:11:56 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
7.49:4360 , piece_id=QDT5N5LDL2HCVXPOOSXWD2TOEZDF2JZWFLDFYSIMN647FI3ZORUQ, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.372193736 (550h40m10.243276781s) [noise=true , address=77.90.31.12:4618 , piece_id=45WRZ26CPUODOAYF5B6RF37DNMDDQWWJUDK7BFKYS3K3LOCNYJCA, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.372518505 (550h40m10.24360155s) [noise=true , address=77.90.31.13:4646 , piece_id=5CDMQF7GG5PDPHIPMFZWDBKWH5ANO3UIX257LJPCWRITNOA7GY4A, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.372848052 (550h40m10.243931097s) [noise=true , address=64.49.13.170:4721 , piece_id=U76TI7IPAHPZAS6XLG66E4L2CGCYH43NH4ZHTYXMXKZR6SGE7SKQ, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.373163072 (550h40m10.244246117s) [noise=true , address=77.90.31.11:4565 , piece_id=4SBLRZK3K4ED73PJ64BI6ANKBWB2AAHQA4SBUNH6CJ63GN3FJR2Q, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.373445089 (550h40m10.244528134s) [noise=true , address=145.79.227.34:4911 , piece_id=24WE4EMZSL2JHQZ7PESCMLYCZOWGDTGJD2XNLUUJEFPSYLRI5E7A, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.373768855 (550h40m10.24485189s) [noise=true , address=77.90.31.18:4819 , piece_id=OYXPAN3EK4QRDEC6VHIDL3GSGEY27G72KII6HTGUMSPYLHCYBNEA, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.374285519 (550h40m10.245368564s) [noise=true , address=77.90.31.14:4678 , piece_id=VBGPSGMNDQZYY7UKRINAV6B3FVI5BPDPYS37FQIJCMLL65F3XOEQ, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.374710689 (550h40m10.245793724s) [noise=true , address=145.79.227.40:4053 , piece_id=RO4ZZQCKCC7T4T6SOT7GGTY626BJ6ESLQG6KATTPNIUM3ICUKHAQ, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.375099329 (550h40m10.246182364s) [noise=true , address=77.90.31.19:4859 , piece_id=A4HD3FPDPCCMHRFBTS3GNGB323RYD5C5AW4ZVSNOTXFI7FLIV37Q, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.376073433 (550h40m10.247156468s) [noise=true , address=77.90.31.15:4704 , piece_id=DXWLFOIKOPE33LMEQECUJVIGRG3LPADUMULLWI4ZBHEOQHD43ULQ, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.376542387 (550h40m10.247625422s) [noise=true , address=145.79.227.44:4221 , piece_id=H72XCWLKYJAXYGGDIWN67H42RUIEOJCPUSWT5IJMFPKETH7LK3KA, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.377379761 (550h40m10.248462796s) [noise=true , address=145.79.227.32:4863 , piece_id=27TOXK44RIF3DV3PDY4RBP6HEFRQJE6N7LNLRJKSU52YCB6A3JKA, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.378022765 (550h40m10.24910581s) [noise=true , address=145.79.227.30:4789 , piece_id=4QT5IUPIK7BOHJROEK2EBFTW434OKC75FZNENMYYCRMBZ7E5BUTA, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:10.378821687 (550h40m10.249904722s) [noise=true , address=145.79.227.2:4119 , piece_id=EQM3WMBQ7GXRWRBZK4ZWCI23K4LSLXW6QPEABHGMWRHEYUYANKSQ, seg_pos={0 0}, upload=482805]: Uploading piece...
Dec 28 14:09:11.676223695 (550h40m11.54730674s) [upload=482805]: Next returned nil segment
Dec 28 14:09:11.676238083 (550h40m11.547321128s) [upload=482805]: Waiting for error group managing segments...
Dec 28 14:09:12.006762275 (550h40m11.87784531s) [noise=true , address=64.49.13.170:4721 , piece_id=U76TI7IPAHPZAS6XLG66E4L2CGCYH43NH4ZHTYXMXKZR6SGE7SKQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.250307967 (550h40m12.121391012s) [noise=true , address=145.79.227.2:4119 , piece_id=EQM3WMBQ7GXRWRBZK4ZWCI23K4LSLXW6QPEABHGMWRHEYUYANKSQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.269380734 (550h40m12.140463769s) [noise=true , address=145.79.227.49:4360 , piece_id=QDT5N5LDL2HCVXPOOSXWD2TOEZDF2JZWFLDFYSIMN647FI3ZORUQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.290818335 (550h40m12.16190138s) [noise=true , address=145.79.227.32:4863 , piece_id=27TOXK44RIF3DV3PDY4RBP6HEFRQJE6N7LNLRJKSU52YCB6A3JKA, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.467679327 (550h40m12.338762362s) [noise=true , address=45.91.116.8:4072 , piece_id=GEVTX6LLENPLN3UQLSGTFTP3DO52IRYUIQFIWC4XMOC3ZF6V4GTQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.486615595 (550h40m12.35769863s) [noise=true , address=77.90.31.12:4618 , piece_id=45WRZ26CPUODOAYF5B6RF37DNMDDQWWJUDK7BFKYS3K3LOCNYJCA, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.566542810 (550h40m12.437625845s) [noise=true , address=77.90.31.19:4859 , piece_id=A4HD3FPDPCCMHRFBTS3GNGB323RYD5C5AW4ZVSNOTXFI7FLIV37Q, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.609337578 (550h40m12.480420623s) [noise=true , address=77.90.31.13:4646 , piece_id=5CDMQF7GG5PDPHIPMFZWDBKWH5ANO3UIX257LJPCWRITNOA7GY4A, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.640856565 (550h40m12.511939611s) [noise=true , address=45.91.116.5:4322 , piece_id=SS6DOCSJHY5Y3DPWMYX45F4VYYXR44JDLYQ2YPY5XIIAYKMAG7LQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:12.645490329 (550h40m12.516573364s) [noise=true , address=77.90.31.18:4819 , piece_id=OYXPAN3EK4QRDEC6VHIDL3GSGEY27G72KII6HTGUMSPYLHCYBNEA, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:13.103710286 (550h40m12.974793331s) [noise=true , address=145.79.227.34:4911 , piece_id=24WE4EMZSL2JHQZ7PESCMLYCZOWGDTGJD2XNLUUJEFPSYLRI5E7A, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:13.209917141 (550h40m13.081000176s) [noise=true , address=145.79.227.44:4221 , piece_id=H72XCWLKYJAXYGGDIWN67H42RUIEOJCPUSWT5IJMFPKETH7LK3KA, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:13.218355468 (550h40m13.089438514s) [noise=true , address=77.90.31.11:4565 , piece_id=4SBLRZK3K4ED73PJ64BI6ANKBWB2AAHQA4SBUNH6CJ63GN3FJR2Q, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:13.255968741 (550h40m13.127051806s) [noise=true , address=145.79.227.30:4789 , piece_id=4QT5IUPIK7BOHJROEK2EBFTW434OKC75FZNENMYYCRMBZ7E5BUTA, seg_pos={0 0}, upload=482805]: Done uploading piece. err: <nil>
Dec 28 14:09:13.255996102 (550h40m13.127079137s) [seg_pos={0 0}, upload=482805]: Segment reached optimal threshold of 14 pieces.
Dec 28 14:09:13.256214589 (550h40m13.127297634s) [noise=true , address=145.79.227.40:4053 , piece_id=RO4ZZQCKCC7T4T6SOT7GGTY626BJ6ESLQG6KATTPNIUM3ICUKHAQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DqyTc4MHvxutFXs2WWRCk8MSuxF3Y1HayyQY4Y5sy8cqDLWuG): protocol: expected piece hash; context canceled; EOF
Dec 28 14:09:13.256223686 (550h40m13.127306731s) [noise=true , address=145.79.227.40:4053 , piece_id=RO4ZZQCKCC7T4T6SOT7GGTY626BJ6ESLQG6KATTPNIUM3ICUKHAQ, seg_pos={0 0}, upload=482805]: Long tail cancellation due to optimal threshold
Dec 28 14:09:13.256512506 (550h40m13.127595551s) [noise=true , address=77.90.31.15:4704 , piece_id=DXWLFOIKOPE33LMEQECUJVIGRG3LPADUMULLWI4ZBHEOQHD43ULQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14tJRdZ1guwwsYwPf3LtENRWqYx2jBQpi7uAMMPaB32Gks9zmj): protocol: expected piece hash; context canceled; EOF
Dec 28 14:09:13.256520060 (550h40m13.127603105s) [noise=true , address=77.90.31.15:4704 , piece_id=DXWLFOIKOPE33LMEQECUJVIGRG3LPADUMULLWI4ZBHEOQHD43ULQ, seg_pos={0 0}, upload=482805]: Long tail cancellation due to optimal threshold
Dec 28 14:09:13.256649106 (550h40m13.127732151s) [noise=true , address=77.90.31.14:4678 , piece_id=VBGPSGMNDQZYY7UKRINAV6B3FVI5BPDPYS37FQIJCMLL65F3XOEQ, seg_pos={0 0}, upload=482805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12beu3SUZMPNWfbeHV4tLtx9UVZoEHh4Kt7oBhT59w3bGBE1ovP): protocol: write tcp 188.166.225.170:57374->77.90.31.14:4678: use of closed network connection; write tcp 188.166.225.170:57374->77.90.31.14:4678: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57374->77.90.31.14:4678: use of closed network connection
Dec 28 14:09:13.256655819 (550h40m13.127738864s) [noise=true , address=77.90.31.14:4678 , piece_id=VBGPSGMNDQZYY7UKRINAV6B3FVI5BPDPYS37FQIJCMLL65F3XOEQ, seg_pos={0 0}, upload=482805]: Long tail cancellation due to optimal threshold
Dec 28 14:09:13.256752974 (550h40m13.127836009s) [noise=true , address=64.49.13.169:4710 , piece_id=SANL7TVJFES63XD46OKQL4KIJ6YVHSEWPZKMRISEKC3ZKWQBOMIA, seg_pos={0 0}, upload=482805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:35680->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:35680->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35680->64.49.13.169:4710: use of closed network connection
Dec 28 14:09:13.256758825 (550h40m13.12784186s) [noise=true , address=64.49.13.169:4710 , piece_id=SANL7TVJFES63XD46OKQL4KIJ6YVHSEWPZKMRISEKC3ZKWQBOMIA, seg_pos={0 0}, upload=482805]: Long tail cancellation due to optimal threshold
Dec 28 14:09:13.256776549 (550h40m13.127859594s) [seg_pos={0 0}, upload=482805]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 14:09:13.256856591 (550h40m13.127939626s) [upload=482805]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 14:09:13.256862432 (550h40m13.127945467s) [upload=482805]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 14:09:13.550363717 (550h40m13.421446782s) [upload=482805]: Done uploading segments...
Dec 28 14:09:13.550405126 (550h40m13.421488161s) [upload=482805]: Upload finished. err: <nil>
Duration: 40.505558552s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [....X............X....................X...X........X........] duration:
18.937812188s trace:
https://jaeger.dev.storj.io/trace/360e059013858fb1
FAIL (2025-12-28 14:11:56 UTC)
orbiter warning: context deadline exceeded
Duration: 18.937835332s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
19 out of 60 is failed (threshold=3) [...X..X.....X..X.X...XX......X......XXX.XX....XX......XX.XX.] duration:
5.748856094s trace:
https://jaeger.dev.storj.io/trace/7d5d0490f5dff817
FAIL (2025-12-28 14:11:51 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: M7YORSQQWQCXZKXVIDBXH2ZDVJ5V2UXQI2TVNJJQFLH2D3K3QV7A) context deadline exceeded; piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: 7MPOEIJEREEFT6BYEYOM7JG76OFL4QXGEYTAC6U5VZRMVMLZMROA) context deadline exceeded
======LOGS======
c 28 14:11:16.426053726 (550h42m16.297136761s) [upload=482833]: Starting upload
Dec 28 14:11:16.426128798 (550h42m16.297211834s) [upload=482833]: Done starting upload
Dec 28 14:11:16.426386770 (550h42m16.297469815s) [upload=482833]: Uploading segments...
Dec 28 14:11:16.426595868 (550h42m16.297678924s) [upload=482833]: Got next segment. Inline: false
Dec 28 14:11:16.426732469 (550h42m16.297815504s) [upload=482833]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 14:11:16.426773046 (550h42m16.297856081s) [upload=482833]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 14:11:16.676125223 (550h42m16.547208278s) [seg_pos={0 0}, upload=482833]: Begin upload segment...
Dec 28 14:11:16.676184687 (550h42m16.547267732s) [seg_pos={0 0}, upload=482833]: Done begin upload segment.
Dec 28 14:11:16.676217850 (550h42m16.547300895s) [noise=true , address=77.90.31.12:4589 , piece_id=JAOCLAFV4NNBJ5CIY5ZBNSX7VZM4P6WR3MJOSBXIUJ3O5ZJP2MHQ, seg_pos={0 0}, upload=482833]: Uploading piece...
Dec 28 14:11:16.676358137 (550h42m16.547441182s) [noise=true , address=45.91.116.12:4226 , piece_id=7MPOEIJEREEFT6BYEYOM7JG76OFL4QXGEYTAC6U5VZRMVMLZMROA, seg_pos={0 0}, upload=482833]: Uploading piece...
Dec 28 14:11:16.676913254 (550h42m16.547996299s) [noise=true , address=145.79.227.8:4330 , piece_id=MXWBEAJ7RXLLEGBN7K5IDP7KKW6SU35YURW6YA4UIYZNY5JSFZFQ, seg_pos={0 0}, upload=482833]: Uploading piece...
Dec 28 14:11:16.677078719 (550h42m16.548161765s) [noise=true , address=145.79.227.37:4565 , piece_id=RZE4XC4SXBKPBC4PDIJSJ7D6AZRODFR4TZDNNGN6ASDYVWHZ4EKQ, seg_pos={0 0}, upload=482833]: Uploading piece...
Dec 28 14:11:16.677540689 (550h42m16.548623734s) [noise=true , address=145.79.227.50:4374 , piece_id=2IJ7GDJBORTRLCXVF4EOYQITUAX4LARMSQBH5DDRB5LCGTBRYCVA, seg_pos={0 0}, upload=482833]: Uploading piece...
Dec 28 14:11:16.677878372 (550h42m16.548961417s) [noise=true , address=145.79.227.52:4446 , piece_id=ZLSC5T5PI7WOANXJXOROTZ4WSULWYQYGFDLAF4ERDKOZ6GUPMOMA, seg_pos={0 0}, upload=482833]: Uploading piece...
Dec 28 14:11:16.678228649 (550h42m16.549311704s) [noise=true , address=145.79.227.31:4798 , piece_id=BZSWK4RZJA2TEPJETROISZAKBJG3QSZDLQRNBONFFLTNZKDRYL3Q, seg_pos={0 0}, upload=482833]: Uploading piece...
Dec 28 14:11:16.678240501 (550h42m16.549323546s) [noise=true , address=45.91.116.8:4045 , piece_id=BIPE5UI7URFXHQ2C3EDRH5TLQDTN6CGE3UZJIEWX2OSEKD4Q2GLQ, seg_pos={0 0}, upload=482833]: Uploading piece...
Dec 28 14:11:16.678578706 (550h42m16.549661751s) [noise=true , address=145.79.227.39:4014 , piece_id=CJ2NROH7AON5OM5BDVAALKAQ6NQL3EBENXFUPTTDUQMBS7DPF6VA, seg_pos={0 0}, upload=482833]: Uploading piece...
Dec 28 14:11:16.678737918 (550h42m16.549820963s) [noise=true , address=145.79.227.47:4266 , piece_id=QGYCOMWTH6N372DX3HP67QPKF5NDVKXQGJBU3RMSKSJV24RZHV2A, seg_pos={0 0}, upload=482833]: Uploading piece...
Dec 28 14:11:16.678967957 (550h42m16.550051002s) [noise=true , address=45.91.116.5:4299 , piece_id=OYQ23GQ2UXFNVYW3IUOMMCQT5DGX6LE2LLIXS54ZUOOKV3W5IZ2Q, seg_pos={0 0}, upload=482833]: Uploading piece...
Dec 28 14:11:16.679216840 (550h42m16.550299875s) [noise=true , address=45.91.116.6:4975 , piece_id=M7YORSQQWQCXZKXVIDBXH2ZDVJ5V2UXQI2TVNJJQFLH2D3K3QV7A, seg_pos={0 0}, upload=482833]: Uploading piece...
Dec 28 14:11:16.679625689 (550h42m16.550708734s) [noise=true , address=145.79.227.12:4765 , piece_id=XIYLXZ6PYMM6V5HWW3ZAS2QIAAW6GRVYBWGODEGSNYVV3YWDOWAA, seg_pos={0 0}, upload=482833]: Uploading piece...
Dec 28 14:11:16.680093500 (550h42m16.551176545s) [noise=true , address=145.79.227.27:4691 , piece_id=QEPPAXCVK6YPTBCTEZEH4MEQ7CAZ2YFV2QHKUB3XZEOV32PWFUDA, seg_pos={0 0}, upload=482833]: Uploading piece...
Dec 28 14:11:16.680505695 (550h42m16.55158873s) [noise=true , address=145.79.227.38:4978 , piece_id=5JBU5OHM2PZ4V4XQGQ44PZN7346VSXRD5QTDWJWUE22QJ4GLULOA, seg_pos={0 0}, upload=482833]: Uploading piece...
Dec 28 14:11:16.681403655 (550h42m16.5524867s) [noise=true , address=45.91.116.14:4191 , piece_id=7TSAWJ62LGMWIQZ733RLGFFOAGBUBLV2G3UTXI7UGNMU6DF4HOHA, seg_pos={0 0}, upload=482833]: Uploading piece...
Dec 28 14:11:18.294800239 (550h42m18.165883284s) [upload=482833]: Next returned nil segment
Dec 28 14:11:18.294819686 (550h42m18.165902731s) [upload=482833]: Waiting for error group managing segments...
Dec 28 14:11:18.680627949 (550h42m18.551710994s) [noise=true , address=145.79.227.52:4446 , piece_id=ZLSC5T5PI7WOANXJXOROTZ4WSULWYQYGFDLAF4ERDKOZ6GUPMOMA, seg_pos={0 0}, upload=482833]: Done uploading piece. err: <nil>
Dec 28 14:11:18.681158509 (550h42m18.552241544s) [noise=true , address=145.79.227.8:4330 , piece_id=MXWBEAJ7RXLLEGBN7K5IDP7KKW6SU35YURW6YA4UIYZNY5JSFZFQ, seg_pos={0 0}, upload=482833]: Done uploading piece. err: <nil>
Dec 28 14:11:18.691046006 (550h42m18.562129051s) [noise=true , address=145.79.227.27:4691 , piece_id=QEPPAXCVK6YPTBCTEZEH4MEQ7CAZ2YFV2QHKUB3XZEOV32PWFUDA, seg_pos={0 0}, upload=482833]: Done uploading piece. err: <nil>
Dec 28 14:11:18.693001951 (550h42m18.564084986s) [noise=true , address=145.79.227.31:4798 , piece_id=BZSWK4RZJA2TEPJETROISZAKBJG3QSZDLQRNBONFFLTNZKDRYL3Q, seg_pos={0 0}, upload=482833]: Done uploading piece. err: <nil>
Dec 28 14:11:18.700095568 (550h42m18.571178603s) [noise=true , address=145.79.227.47:4266 , piece_id=QGYCOMWTH6N372DX3HP67QPKF5NDVKXQGJBU3RMSKSJV24RZHV2A, seg_pos={0 0}, upload=482833]: Done uploading piece. err: <nil>
Dec 28 14:11:18.702245151 (550h42m18.573328187s) [noise=true , address=145.79.227.39:4014 , piece_id=CJ2NROH7AON5OM5BDVAALKAQ6NQL3EBENXFUPTTDUQMBS7DPF6VA, seg_pos={0 0}, upload=482833]: Done uploading piece. err: <nil>
Dec 28 14:11:18.702904978 (550h42m18.573988023s) [noise=true , address=145.79.227.12:4765 , piece_id=XIYLXZ6PYMM6V5HWW3ZAS2QIAAW6GRVYBWGODEGSNYVV3YWDOWAA, seg_pos={0 0}, upload=482833]: Done uploading piece. err: <nil>
Dec 28 14:11:18.706012505 (550h42m18.57709555s) [noise=true , address=145.79.227.37:4565 , piece_id=RZE4XC4SXBKPBC4PDIJSJ7D6AZRODFR4TZDNNGN6ASDYVWHZ4EKQ, seg_pos={0 0}, upload=482833]: Done uploading piece. err: <nil>
Dec 28 14:11:18.706846794 (550h42m18.577929839s) [noise=true , address=145.79.227.38:4978 , piece_id=5JBU5OHM2PZ4V4XQGQ44PZN7346VSXRD5QTDWJWUE22QJ4GLULOA, seg_pos={0 0}, upload=482833]: Done uploading piece. err: <nil>
Dec 28 14:11:18.717068217 (550h42m18.588151252s) [noise=true , address=145.79.227.50:4374 , piece_id=2IJ7GDJBORTRLCXVF4EOYQITUAX4LARMSQBH5DDRB5LCGTBRYCVA, seg_pos={0 0}, upload=482833]: Done uploading piece. err: <nil>
Dec 28 14:11:18.905351469 (550h42m18.776434504s) [noise=true , address=45.91.116.6:4975 , piece_id=M7YORSQQWQCXZKXVIDBXH2ZDVJ5V2UXQI2TVNJJQFLH2D3K3QV7A, seg_pos={0 0}, upload=482833]: Done uploading piece. err: <nil>
Dec 28 14:11:18.954007870 (550h42m18.825090905s) [noise=true , address=45.91.116.12:4226 , piece_id=7MPOEIJEREEFT6BYEYOM7JG76OFL4QXGEYTAC6U5VZRMVMLZMROA, seg_pos={0 0}, upload=482833]: Done uploading piece. err: <nil>
Dec 28 14:11:18.957617743 (550h42m18.828700778s) [noise=true , address=45.91.116.14:4191 , piece_id=7TSAWJ62LGMWIQZ733RLGFFOAGBUBLV2G3UTXI7UGNMU6DF4HOHA, seg_pos={0 0}, upload=482833]: Done uploading piece. err: <nil>
Dec 28 14:11:18.957635297 (550h42m18.828718332s) [seg_pos={0 0}, upload=482833]: Segment reached optimal threshold of 13 pieces.
Dec 28 14:11:18.957837652 (550h42m18.828920697s) [noise=true , address=45.91.116.5:4299 , piece_id=OYQ23GQ2UXFNVYW3IUOMMCQT5DGX6LE2LLIXS54ZUOOKV3W5IZ2Q, seg_pos={0 0}, upload=482833]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF
Dec 28 14:11:18.957843984 (550h42m18.828927019s) [noise=true , address=45.91.116.5:4299 , piece_id=OYQ23GQ2UXFNVYW3IUOMMCQT5DGX6LE2LLIXS54ZUOOKV3W5IZ2Q, seg_pos={0 0}, upload=482833]: Long tail cancellation due to optimal threshold
Dec 28 14:11:18.957966427 (550h42m18.829049462s) [noise=true , address=45.91.116.8:4045 , piece_id=BIPE5UI7URFXHQ2C3EDRH5TLQDTN6CGE3UZJIEWX2OSEKD4Q2GLQ, seg_pos={0 0}, upload=482833]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: write tcp 188.166.225.170:55214->45.91.116.8:4045: use of closed network connection; write tcp 188.166.225.170:55214->45.91.116.8:4045: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55214->45.91.116.8:4045: use of closed network connection
Dec 28 14:11:18.957971316 (550h42m18.829054351s) [noise=true , address=45.91.116.8:4045 , piece_id=BIPE5UI7URFXHQ2C3EDRH5TLQDTN6CGE3UZJIEWX2OSEKD4Q2GLQ, seg_pos={0 0}, upload=482833]: Long tail cancellation due to optimal threshold
Dec 28 14:11:18.958205312 (550h42m18.829288357s) [noise=true , address=77.90.31.12:4589 , piece_id=JAOCLAFV4NNBJ5CIY5ZBNSX7VZM4P6WR3MJOSBXIUJ3O5ZJP2MHQ, seg_pos={0 0}, upload=482833]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF
Dec 28 14:11:18.958214289 (550h42m18.829297324s) [noise=true , address=77.90.31.12:4589 , piece_id=JAOCLAFV4NNBJ5CIY5ZBNSX7VZM4P6WR3MJOSBXIUJ3O5ZJP2MHQ, seg_pos={0 0}, upload=482833]: Long tail cancellation due to optimal threshold
Dec 28 14:11:18.958240759 (550h42m18.829323794s) [seg_pos={0 0}, upload=482833]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 14:11:18.958319619 (550h42m18.829402654s) [upload=482833]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 14:11:18.958325211 (550h42m18.829408256s) [upload=482833]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 14:11:19.259598824 (550h42m19.130681859s) [upload=482833]: Done uploading segments...
Dec 28 14:11:19.259676423 (550h42m19.130759458s) [upload=482833]: Upload finished. err: <nil>
Duration: 5.748878797s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
19 out of 60 is failed (threshold=3) [....X..X.....X..X.X...XX......X......XXX.XX....XX......XX.XX] duration:
30.002813718s trace:
https://jaeger.dev.storj.io/trace/7d5d0490f5dff817
FAIL (2025-12-28 14:11:45 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: M7YORSQQWQCXZKXVIDBXH2ZDVJ5V2UXQI2TVNJJQFLH2D3K3QV7A) context deadline exceeded; piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: 7MPOEIJEREEFT6BYEYOM7JG76OFL4QXGEYTAC6U5VZRMVMLZMROA) context deadline exceeded
======LOGS======
c 28 14:11:16.426053726 (550h42m16.297136761s) [upload=482833]: Starting upload
Dec 28 14:11:16.426128798 (550h42m16.297211834s) [upload=482833]: Done starting upload
Dec 28 14:11:16.426386770 (550h42m16.297469815s) [upload=482833]: Uploading segments...
Dec 28 14:11:16.426595868 (550h42m16.297678924s) [upload=482833]: Got next segment. Inline: false
Dec 28 14:11:16.426732469 (550h42m16.297815504s) [upload=482833]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 14:11:16.426773046 (550h42m16.297856081s) [upload=482833]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 14:11:16.676125223 (550h42m16.547208278s) [seg_pos={0 0}, upload=482833]: Begin upload segment...
Dec 28 14:11:16.676184687 (550h42m16.547267732s) [seg_pos={0 0}, upload=482833]: Done begin upload segment.
Dec 28 14:11:16.676217850 (550h42m16.547300895s) [noise=true , address=77.90.31.12:4589 , piece_id=JAOCLAFV4NNBJ5CIY5ZBNSX7VZM4P6WR3MJOSBXIUJ3O5ZJP2MHQ, seg_pos={0 0}, upload=482833]: Uploading piece...
Dec 28 14:11:16.676358137 (550h42m16.547441182s) [noise=true , address=45.91.116.12:4226 , piece_id=7MPOEIJEREEFT6BYEYOM7JG76OFL4QXGEYTAC6U5VZRMVMLZMROA, seg_pos={0 0}, upload=482833]: Uploading piece...
Dec 28 14:11:16.676913254 (550h42m16.547996299s) [noise=true , address=145.79.227.8:4330 , piece_id=MXWBEAJ7RXLLEGBN7K5IDP7KKW6SU35YURW6YA4UIYZNY5JSFZFQ, seg_pos={0 0}, upload=482833]: Uploading piece...
Dec 28 14:11:16.677078719 (550h42m16.548161765s) [noise=true , address=145.79.227.37:4565 , piece_id=RZE4XC4SXBKPBC4PDIJSJ7D6AZRODFR4TZDNNGN6ASDYVWHZ4EKQ, seg_pos={0 0}, upload=482833]: Uploading piece...
Dec 28 14:11:16.677540689 (550h42m16.548623734s) [noise=true , address=145.79.227.50:4374 , piece_id=2IJ7GDJBORTRLCXVF4EOYQITUAX4LARMSQBH5DDRB5LCGTBRYCVA, seg_pos={0 0}, upload=482833]: Uploading piece...
Dec 28 14:11:16.677878372 (550h42m16.548961417s) [noise=true , address=145.79.227.52:4446 , piece_id=ZLSC5T5PI7WOANXJXOROTZ4WSULWYQYGFDLAF4ERDKOZ6GUPMOMA, seg_pos={0 0}, upload=482833]: Uploading piece...
Dec 28 14:11:16.678228649 (550h42m16.549311704s) [noise=true , address=145.79.227.31:4798 , piece_id=BZSWK4RZJA2TEPJETROISZAKBJG3QSZDLQRNBONFFLTNZKDRYL3Q, seg_pos={0 0}, upload=482833]: Uploading piece...
Dec 28 14:11:16.678240501 (550h42m16.549323546s) [noise=true , address=45.91.116.8:4045 , piece_id=BIPE5UI7URFXHQ2C3EDRH5TLQDTN6CGE3UZJIEWX2OSEKD4Q2GLQ, seg_pos={0 0}, upload=482833]: Uploading piece...
Dec 28 14:11:16.678578706 (550h42m16.549661751s) [noise=true , address=145.79.227.39:4014 , piece_id=CJ2NROH7AON5OM5BDVAALKAQ6NQL3EBENXFUPTTDUQMBS7DPF6VA, seg_pos={0 0}, upload=482833]: Uploading piece...
Dec 28 14:11:16.678737918 (550h42m16.549820963s) [noise=true , address=145.79.227.47:4266 , piece_id=QGYCOMWTH6N372DX3HP67QPKF5NDVKXQGJBU3RMSKSJV24RZHV2A, seg_pos={0 0}, upload=482833]: Uploading piece...
Dec 28 14:11:16.678967957 (550h42m16.550051002s) [noise=true , address=45.91.116.5:4299 , piece_id=OYQ23GQ2UXFNVYW3IUOMMCQT5DGX6LE2LLIXS54ZUOOKV3W5IZ2Q, seg_pos={0 0}, upload=482833]: Uploading piece...
Dec 28 14:11:16.679216840 (550h42m16.550299875s) [noise=true , address=45.91.116.6:4975 , piece_id=M7YORSQQWQCXZKXVIDBXH2ZDVJ5V2UXQI2TVNJJQFLH2D3K3QV7A, seg_pos={0 0}, upload=482833]: Uploading piece...
Dec 28 14:11:16.679625689 (550h42m16.550708734s) [noise=true , address=145.79.227.12:4765 , piece_id=XIYLXZ6PYMM6V5HWW3ZAS2QIAAW6GRVYBWGODEGSNYVV3YWDOWAA, seg_pos={0 0}, upload=482833]: Uploading piece...
Dec 28 14:11:16.680093500 (550h42m16.551176545s) [noise=true , address=145.79.227.27:4691 , piece_id=QEPPAXCVK6YPTBCTEZEH4MEQ7CAZ2YFV2QHKUB3XZEOV32PWFUDA, seg_pos={0 0}, upload=482833]: Uploading piece...
Dec 28 14:11:16.680505695 (550h42m16.55158873s) [noise=true , address=145.79.227.38:4978 , piece_id=5JBU5OHM2PZ4V4XQGQ44PZN7346VSXRD5QTDWJWUE22QJ4GLULOA, seg_pos={0 0}, upload=482833]: Uploading piece...
Dec 28 14:11:16.681403655 (550h42m16.5524867s) [noise=true , address=45.91.116.14:4191 , piece_id=7TSAWJ62LGMWIQZ733RLGFFOAGBUBLV2G3UTXI7UGNMU6DF4HOHA, seg_pos={0 0}, upload=482833]: Uploading piece...
Dec 28 14:11:18.294800239 (550h42m18.165883284s) [upload=482833]: Next returned nil segment
Dec 28 14:11:18.294819686 (550h42m18.165902731s) [upload=482833]: Waiting for error group managing segments...
Dec 28 14:11:18.680627949 (550h42m18.551710994s) [noise=true , address=145.79.227.52:4446 , piece_id=ZLSC5T5PI7WOANXJXOROTZ4WSULWYQYGFDLAF4ERDKOZ6GUPMOMA, seg_pos={0 0}, upload=482833]: Done uploading piece. err: <nil>
Dec 28 14:11:18.681158509 (550h42m18.552241544s) [noise=true , address=145.79.227.8:4330 , piece_id=MXWBEAJ7RXLLEGBN7K5IDP7KKW6SU35YURW6YA4UIYZNY5JSFZFQ, seg_pos={0 0}, upload=482833]: Done uploading piece. err: <nil>
Dec 28 14:11:18.691046006 (550h42m18.562129051s) [noise=true , address=145.79.227.27:4691 , piece_id=QEPPAXCVK6YPTBCTEZEH4MEQ7CAZ2YFV2QHKUB3XZEOV32PWFUDA, seg_pos={0 0}, upload=482833]: Done uploading piece. err: <nil>
Dec 28 14:11:18.693001951 (550h42m18.564084986s) [noise=true , address=145.79.227.31:4798 , piece_id=BZSWK4RZJA2TEPJETROISZAKBJG3QSZDLQRNBONFFLTNZKDRYL3Q, seg_pos={0 0}, upload=482833]: Done uploading piece. err: <nil>
Dec 28 14:11:18.700095568 (550h42m18.571178603s) [noise=true , address=145.79.227.47:4266 , piece_id=QGYCOMWTH6N372DX3HP67QPKF5NDVKXQGJBU3RMSKSJV24RZHV2A, seg_pos={0 0}, upload=482833]: Done uploading piece. err: <nil>
Dec 28 14:11:18.702245151 (550h42m18.573328187s) [noise=true , address=145.79.227.39:4014 , piece_id=CJ2NROH7AON5OM5BDVAALKAQ6NQL3EBENXFUPTTDUQMBS7DPF6VA, seg_pos={0 0}, upload=482833]: Done uploading piece. err: <nil>
Dec 28 14:11:18.702904978 (550h42m18.573988023s) [noise=true , address=145.79.227.12:4765 , piece_id=XIYLXZ6PYMM6V5HWW3ZAS2QIAAW6GRVYBWGODEGSNYVV3YWDOWAA, seg_pos={0 0}, upload=482833]: Done uploading piece. err: <nil>
Dec 28 14:11:18.706012505 (550h42m18.57709555s) [noise=true , address=145.79.227.37:4565 , piece_id=RZE4XC4SXBKPBC4PDIJSJ7D6AZRODFR4TZDNNGN6ASDYVWHZ4EKQ, seg_pos={0 0}, upload=482833]: Done uploading piece. err: <nil>
Dec 28 14:11:18.706846794 (550h42m18.577929839s) [noise=true , address=145.79.227.38:4978 , piece_id=5JBU5OHM2PZ4V4XQGQ44PZN7346VSXRD5QTDWJWUE22QJ4GLULOA, seg_pos={0 0}, upload=482833]: Done uploading piece. err: <nil>
Dec 28 14:11:18.717068217 (550h42m18.588151252s) [noise=true , address=145.79.227.50:4374 , piece_id=2IJ7GDJBORTRLCXVF4EOYQITUAX4LARMSQBH5DDRB5LCGTBRYCVA, seg_pos={0 0}, upload=482833]: Done uploading piece. err: <nil>
Dec 28 14:11:18.905351469 (550h42m18.776434504s) [noise=true , address=45.91.116.6:4975 , piece_id=M7YORSQQWQCXZKXVIDBXH2ZDVJ5V2UXQI2TVNJJQFLH2D3K3QV7A, seg_pos={0 0}, upload=482833]: Done uploading piece. err: <nil>
Dec 28 14:11:18.954007870 (550h42m18.825090905s) [noise=true , address=45.91.116.12:4226 , piece_id=7MPOEIJEREEFT6BYEYOM7JG76OFL4QXGEYTAC6U5VZRMVMLZMROA, seg_pos={0 0}, upload=482833]: Done uploading piece. err: <nil>
Dec 28 14:11:18.957617743 (550h42m18.828700778s) [noise=true , address=45.91.116.14:4191 , piece_id=7TSAWJ62LGMWIQZ733RLGFFOAGBUBLV2G3UTXI7UGNMU6DF4HOHA, seg_pos={0 0}, upload=482833]: Done uploading piece. err: <nil>
Dec 28 14:11:18.957635297 (550h42m18.828718332s) [seg_pos={0 0}, upload=482833]: Segment reached optimal threshold of 13 pieces.
Dec 28 14:11:18.957837652 (550h42m18.828920697s) [noise=true , address=45.91.116.5:4299 , piece_id=OYQ23GQ2UXFNVYW3IUOMMCQT5DGX6LE2LLIXS54ZUOOKV3W5IZ2Q, seg_pos={0 0}, upload=482833]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF
Dec 28 14:11:18.957843984 (550h42m18.828927019s) [noise=true , address=45.91.116.5:4299 , piece_id=OYQ23GQ2UXFNVYW3IUOMMCQT5DGX6LE2LLIXS54ZUOOKV3W5IZ2Q, seg_pos={0 0}, upload=482833]: Long tail cancellation due to optimal threshold
Dec 28 14:11:18.957966427 (550h42m18.829049462s) [noise=true , address=45.91.116.8:4045 , piece_id=BIPE5UI7URFXHQ2C3EDRH5TLQDTN6CGE3UZJIEWX2OSEKD4Q2GLQ, seg_pos={0 0}, upload=482833]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: write tcp 188.166.225.170:55214->45.91.116.8:4045: use of closed network connection; write tcp 188.166.225.170:55214->45.91.116.8:4045: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55214->45.91.116.8:4045: use of closed network connection
Dec 28 14:11:18.957971316 (550h42m18.829054351s) [noise=true , address=45.91.116.8:4045 , piece_id=BIPE5UI7URFXHQ2C3EDRH5TLQDTN6CGE3UZJIEWX2OSEKD4Q2GLQ, seg_pos={0 0}, upload=482833]: Long tail cancellation due to optimal threshold
Dec 28 14:11:18.958205312 (550h42m18.829288357s) [noise=true , address=77.90.31.12:4589 , piece_id=JAOCLAFV4NNBJ5CIY5ZBNSX7VZM4P6WR3MJOSBXIUJ3O5ZJP2MHQ, seg_pos={0 0}, upload=482833]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF
Dec 28 14:11:18.958214289 (550h42m18.829297324s) [noise=true , address=77.90.31.12:4589 , piece_id=JAOCLAFV4NNBJ5CIY5ZBNSX7VZM4P6WR3MJOSBXIUJ3O5ZJP2MHQ, seg_pos={0 0}, upload=482833]: Long tail cancellation due to optimal threshold
Dec 28 14:11:18.958240759 (550h42m18.829323794s) [seg_pos={0 0}, upload=482833]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 14:11:18.958319619 (550h42m18.829402654s) [upload=482833]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 14:11:18.958325211 (550h42m18.829408256s) [upload=482833]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 14:11:19.259598824 (550h42m19.130681859s) [upload=482833]: Done uploading segments...
Dec 28 14:11:19.259676423 (550h42m19.130759458s) [upload=482833]: Upload finished. err: <nil>
Duration: 30.002839648s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
53 out of 60 is failed (threshold=6) [XXX.XXXXXXX.XXXXXXXXXXXX.XXXXXXX.XX..XXXXXXXXXXXXXXXXXXXXX.X] duration:
5.001028466s trace:
https://jaeger.dev.storj.io/trace/92a90438d044faa
FAIL (2025-12-28 14:11:27 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001055628s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [...............X............................................] duration:
6.958141113s trace:
https://jaeger.dev.storj.io/trace/5fe9d6dbb79fbe44
FAIL (2025-12-28 14:11:23 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
67.204.205:4027 , piece_id=PAPNMRJ7E47VOMOXPBBDWKZJHLRMXTWVR5VW3J7KNQSSMHOHWOAQ, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.967737953 (550h8m4.838820988s) [noise=true , address=185.167.204.206:4072 , piece_id=WXORBDMA3KW7WZ67A4FJT6URJRFADUJ7X2PAHIGNG47T3TGMXVTQ, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.968193240 (550h8m4.839276285s) [noise=true , address=185.167.204.205:4030 , piece_id=R5DLI4ZTLOVGO56GPLJL5ZNHYSXOSYNJQHFIO4MVVNHWWBMEOV2Q, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.968780638 (550h8m4.839863683s) [noise=true , address=89.149.201.147:4019 , piece_id=MK7ORCLHO3DOWIOVSOWIQMPJ447PHRQGXP3UKITIRTYJSU5MPCCQ, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.969308924 (550h8m4.840391959s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.970088349 (550h8m4.841171404s) [noise=true , address=81.171.14.70:4039 , piece_id=EDHPQWLQZTP7MKPGEVMP7JXNTW7LFWSXCTRTSXPCBIQPROX76WRA, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.970325381 (550h8m4.841408426s) [noise=true , address=81.171.14.70:4036 , piece_id=CICNNV7JE6TBNC4RAPXKCEUSJAADOUVGSVIVQISM3ZB4COAZQDDA, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.970783813 (550h8m4.841866858s) [noise=true , address=81.171.14.70:4044 , piece_id=DGN74IG6IWAHLEMO6NLL4TJBXZ6ONTN6QUB6HXY6K7SSO3LYOYIQ, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.971205406 (550h8m4.842288441s) [noise=true , address=81.171.14.70:4048 , piece_id=TLT47ORGUKGF3MIIEAX7D22VR2FO6W6NL3VFJNHKWD4L3LIMFDLA, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.971606299 (550h8m4.842689344s) [noise=true , address=185.167.204.206:4067 , piece_id=VSKTG5RH32N7OEVWJGQVB6WJ72TEVF6DDK25W7NOH4RAXUP52LTQ, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:04.972162538 (550h8m4.843245583s) [noise=true , address=89.149.201.147:4002 , piece_id=G2PABNTDWCK4MKOZF6PONBMRNIXI45JTHJMAZFPEKPEJ3PQ7MTBA, seg_pos={0 0}, upload=482337]: Uploading piece...
Dec 28 13:37:05.795167814 (550h8m5.666250849s) [upload=482337]: Next returned nil segment
Dec 28 13:37:05.795184566 (550h8m5.666267611s) [upload=482337]: Waiting for error group managing segments...
Dec 28 13:37:06.178573801 (550h8m6.049656836s) [noise=true , address=185.167.204.206:4049 , piece_id=6RG2DZV6FPTIRE63ZNNST42TSWEBQ456YXIPPVYHIXV55GWFVJ4Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.183060835 (550h8m6.05414391s) [noise=true , address=185.167.204.206:4072 , piece_id=WXORBDMA3KW7WZ67A4FJT6URJRFADUJ7X2PAHIGNG47T3TGMXVTQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.183411202 (550h8m6.054494237s) [noise=true , address=185.167.204.206:4064 , piece_id=OG3G74OU23CEMKA54WQE3DOJ6AKAXBLZ23QWOQC4UEGQQN4JSMXQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.183791146 (550h8m6.054874181s) [noise=true , address=185.167.204.205:4029 , piece_id=6JQB6ZSCQZKWH3JWCQ4DY6BBAX73KC4Y53PRSMEMLWURZ5QWWQQQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.183973743 (550h8m6.055056778s) [noise=true , address=185.167.204.206:4053 , piece_id=YDH77S6JLKJ6IYS6DRUCELP5EJHIC2O44QBO7MIOEUYBML4GIBHA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.184465369 (550h8m6.055548414s) [noise=true , address=185.167.204.205:4008 , piece_id=RTUZPEXBB6JNSMC5YOFQQGDM57YD233W4756ZBQVMFNOVX6YKB4A, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.184501989 (550h8m6.055585024s) [noise=true , address=185.167.204.205:4030 , piece_id=R5DLI4ZTLOVGO56GPLJL5ZNHYSXOSYNJQHFIO4MVVNHWWBMEOV2Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.184856224 (550h8m6.055939259s) [noise=true , address=185.167.204.205:4009 , piece_id=V6RA7Y7FKY4DCPTRRF3BYNBALWDY5VTPXMDUXFE3353KUXLSQJEA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.185063518 (550h8m6.056146563s) [noise=true , address=185.167.204.205:4016 , piece_id=JB7SC3J2VR4U33HNQS2A2CA75THT3XLQCOIFHAJJEK7MQGY2SYAQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.185288617 (550h8m6.056371652s) [noise=true , address=185.167.204.205:4027 , piece_id=PAPNMRJ7E47VOMOXPBBDWKZJHLRMXTWVR5VW3J7KNQSSMHOHWOAQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.186241401 (550h8m6.057324436s) [noise=true , address=185.167.204.205:4024 , piece_id=FM5IMWRSJJ4RY72IX2OOYC54BFKL4FWE3UOAN3GJ26MLH345COVA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.193113658 (550h8m6.064196703s) [noise=true , address=185.167.204.205:4005 , piece_id=R4DFQJZBPJ5WJWHPODV6ILHUDIRBUSAFOMVNCLXB4UNMI6EXTKHA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.193791408 (550h8m6.064874453s) [noise=true , address=185.167.204.206:4068 , piece_id=DAI37M67FWCSVQTQOZ6RSSOJDGUVZO6SNGAHJJ44G3PGS4OJZEMA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.194292262 (550h8m6.065375307s) [noise=true , address=185.167.204.206:4046 , piece_id=VWLVHMBMJ63DQEWYBIKH3Y3NTVWFNW5OXZKMYXXBHVI3VWYXLLPQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.194302652 (550h8m6.065385687s) [noise=true , address=185.167.204.206:4067 , piece_id=VSKTG5RH32N7OEVWJGQVB6WJ72TEVF6DDK25W7NOH4RAXUP52LTQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.195359193 (550h8m6.066442228s) [noise=true , address=185.167.204.205:4012 , piece_id=LV6RUA5BWXG4DMSUVZDBGUAF5IBWUGMLJYUKNIXHRVDEBIOW4Q2Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.198423448 (550h8m6.069506493s) [noise=true , address=185.167.204.206:4070 , piece_id=5A7YYQYREIZRZDR7TRSE7XFW765JVAV7FX6LKWUI2OXAA74XN4CA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.888968408 (550h8m6.760051453s) [noise=true , address=81.171.14.70:4044 , piece_id=DGN74IG6IWAHLEMO6NLL4TJBXZ6ONTN6QUB6HXY6K7SSO3LYOYIQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.892350789 (550h8m6.763433834s) [noise=true , address=81.171.14.70:4039 , piece_id=EDHPQWLQZTP7MKPGEVMP7JXNTW7LFWSXCTRTSXPCBIQPROX76WRA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:06.904477160 (550h8m6.775560195s) [noise=true , address=81.171.14.70:4040 , piece_id=DYF4GSZBAKQ3H63LTVCMEFWMABY3FXHOJCNVIFYRKLF2J6B52K5A, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.128088281 (550h8m6.999171326s) [noise=true , address=81.171.14.70:4036 , piece_id=CICNNV7JE6TBNC4RAPXKCEUSJAADOUVGSVIVQISM3ZB4COAZQDDA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.430242142 (550h8m7.301325177s) [noise=true , address=89.149.201.147:4009 , piece_id=EVW2NQFYU57PC2IIVXLYIEBMYXJNIC2CLCHW6PS6NYIPCFRUVHDA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.505304332 (550h8m7.376387367s) [noise=true , address=89.149.201.147:4019 , piece_id=MK7ORCLHO3DOWIOVSOWIQMPJ447PHRQGXP3UKITIRTYJSU5MPCCQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.575461543 (550h8m7.446544609s) [noise=true , address=81.171.14.70:4048 , piece_id=TLT47ORGUKGF3MIIEAX7D22VR2FO6W6NL3VFJNHKWD4L3LIMFDLA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.599667265 (550h8m7.4707503s) [noise=true , address=89.149.201.147:4023 , piece_id=5IINX2GVRFLRX2QH24OO2DDANOKOTXTDR6AGVJBGSJ7IAZQ3CASA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.928390728 (550h8m7.799473773s) [noise=true , address=89.149.201.147:4002 , piece_id=G2PABNTDWCK4MKOZF6PONBMRNIXI45JTHJMAZFPEKPEJ3PQ7MTBA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil>
Dec 28 13:37:07.928417148 (550h8m7.799500193s) [seg_pos={0 0}, upload=482337]: Segment reached optimal threshold of 26 pieces.
Dec 28 13:37:07.928656303 (550h8m7.799739348s) [noise=true , address=81.171.14.70:4030 , piece_id=74YDIKUST7X3JBXVQHVLFHIKH5A4DYH5TNL3I4XUA5A6AHQO2DRA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZoRwa8wLw66Pjumy1ybp6wrZ79RrSU55uYZpwaY3xi95iSYde): protocol: expected piece hash; context canceled; EOF
Dec 28 13:37:07.928664990 (550h8m7.799748035s) [noise=true , address=81.171.14.70:4030 , piece_id=74YDIKUST7X3JBXVQHVLFHIKH5A4DYH5TNL3I4XUA5A6AHQO2DRA, seg_pos={0 0}, upload=482337]: Long tail cancellation due to optimal threshold
Dec 28 13:37:07.928723351 (550h8m7.799806386s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wgtphHKaTejC1qrJ98sDysBwtj2Zeqw2c5UFk1HMqfY8Buvr): protocol: expected piece hash; context canceled; EOF
Dec 28 13:37:07.928733922 (550h8m7.799816957s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Long tail cancellation due to optimal threshold
Dec 28 13:37:07.928754921 (550h8m7.799837966s) [seg_pos={0 0}, upload=482337]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 28 13:37:07.928870170 (550h8m7.799953215s) [upload=482337]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 13:37:07.928878627 (550h8m7.799961672s) [upload=482337]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 13:37:08.166321536 (550h8m8.037404571s) [upload=482337]: Done uploading segments...
Dec 28 13:37:08.166341444 (550h8m8.037424479s) [upload=482337]: Upload finished. err: <nil>
Duration: 6.958165188s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [......................................................X...X.] duration:
25.172353661s trace:
https://jaeger.dev.storj.io/trace/102cf391e061ae77
FAIL (2025-12-28 14:11:06 UTC)
orbiter warning: context deadline exceeded
Duration: 25.172365553s
core
uplink
us-select-1
us1-placement-20
|