Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.............................................X..........X...] duration: 3.42354924s trace: https://jaeger.dev.storj.io/trace/405976c68d606443 FAIL (2025-12-05 12:06:15 UTC) orbiter warning: context deadline exceeded Duration: 3.423584456s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..............................................X..........X..] duration: 4.809363871s trace: https://jaeger.dev.storj.io/trace/405976c68d606443 FAIL (2025-12-05 12:05:42 UTC) orbiter warning: context deadline exceeded Duration: 4.809380764s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...............................................X..........X.] duration: 5.001380248s trace: https://jaeger.dev.storj.io/trace/405976c68d606443 FAIL (2025-12-05 12:05:07 UTC) orbiter warning: context deadline exceeded Duration: 5.001406669s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [................................................X..........X] duration: 5.001284177s trace: https://jaeger.dev.storj.io/trace/405976c68d606443 FAIL (2025-12-05 12:04:28 UTC) orbiter warning: context deadline exceeded Duration: 5.001306699s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................................................X..........] duration: 4.015332375s trace: https://jaeger.dev.storj.io/trace/165d449f6f4b7198 FAIL (2025-12-05 12:03:48 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.015366159s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................................................X.........] duration: 3.909423633s trace: https://jaeger.dev.storj.io/trace/165d449f6f4b7198 FAIL (2025-12-05 12:03:27 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.909452196s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................................................X........] duration: 4.846661895s trace: https://jaeger.dev.storj.io/trace/165d449f6f4b7198 FAIL (2025-12-05 12:02:40 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.846685981s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................................................X.......] duration: 5.001472978s trace: https://jaeger.dev.storj.io/trace/165d449f6f4b7198 FAIL (2025-12-05 12:01:40 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001498668s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................................................X......] duration: 3.598946795s trace: https://jaeger.dev.storj.io/trace/165d449f6f4b7198 FAIL (2025-12-05 12:01:08 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.598992021s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................................................X.....] duration: 3.338941407s trace: https://jaeger.dev.storj.io/trace/165d449f6f4b7198 FAIL (2025-12-05 12:00:25 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.338970873s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................................................X....] duration: 5.001466902s trace: https://jaeger.dev.storj.io/trace/165d449f6f4b7198 FAIL (2025-12-05 12:00:11 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001524793s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................................................X...] duration: 3.533873223s trace: https://jaeger.dev.storj.io/trace/165d449f6f4b7198 FAIL (2025-12-05 11:59:04 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.533905947s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................................................X..] duration: 4.7887611s trace: https://jaeger.dev.storj.io/trace/165d449f6f4b7198 FAIL (2025-12-05 11:58:13 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.78878688s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................................................X.] duration: 3.652188328s trace: https://jaeger.dev.storj.io/trace/165d449f6f4b7198 FAIL (2025-12-05 11:57:20 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.65222116s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................................................X] duration: 5.001550299s trace: https://jaeger.dev.storj.io/trace/165d449f6f4b7198 FAIL (2025-12-05 11:56:17 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001709463s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 8.108688741s trace: https://jaeger.dev.storj.io/trace/55b376cc76cfe920 FAIL (2025-12-05 10:42:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 9 (41h1m29.913335078s) [upload=36107]: Uploading segments... Dec 5 09:56:53.442159225 (41h1m29.913369955s) [upload=36107]: Got next segment. Inline: false Dec 5 09:56:53.442188120 (41h1m29.913398849s) [upload=36107]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 5 09:56:53.442194372 (41h1m29.913405101s) [upload=36107]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 5 09:56:53.701976567 (41h1m30.173187296s) [seg_pos={0 0}, upload=36107]: Begin upload segment... Dec 5 09:56:53.702020460 (41h1m30.173231189s) [seg_pos={0 0}, upload=36107]: Done begin upload segment. Dec 5 09:56:53.702056649 (41h1m30.173267388s) [noise=true , address=77.90.31.21:4878 , piece_id=RCFTETF7SGR6FTUBJWWBJDKOZIHZKBMSUBZYVWYKF2MP56PMIWMQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702064965 (41h1m30.173275694s) [noise=true , address=145.79.227.28:4725 , piece_id=ZE642TJU4ZJKBOHQ6K6GPLPTN2YKC4BOP325LN7QXALZSDPG66HQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702111533 (41h1m30.173322272s) [noise=true , address=145.79.227.42:4121 , piece_id=6D47MY4GTQ3YQOGHS2V2OTO2VFDL6XKKS2R4VA6VZZ557WYFY34A, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702548986 (41h1m30.173759766s) [noise=true , address=145.79.227.27:4691 , piece_id=Z53YBOXTW4L23P4M3EOW6ADUHDWHIJVRDT7IL4I5FUPPZOW2MRLQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702931455 (41h1m30.174142194s) [noise=true , address=77.90.31.16:4735 , piece_id=V4YJIHVMXYTBK7IHU272RYCKUXI44B2MXRLETMWMO2WAI2XRU4SQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.703301079 (41h1m30.174511808s) [noise=true , address=145.79.227.10:4401 , piece_id=XAVSV3SGPDDLZWGKQLHA2CVYTGOPO7J3YVEO5C3MYIN5SJEUZ2OQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.703633000 (41h1m30.174843729s) [noise=true , address=145.79.227.24:4583 , piece_id=IB6T3J3CF3P3A54JEHHBK6KEPMUEI32B4QYSSA2G2BUMIVUC7DGQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.703957648 (41h1m30.175168377s) [noise=true , address=145.79.227.26:4654 , piece_id=23NNREG47WSZWJNPZWP275L62YBBSTB6RQWTINIRTTAYU66DRT7Q, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.704788621 (41h1m30.17599936s) [noise=true , address=145.79.227.43:4159 , piece_id=S5T27VFA6UNNONPZBZILNTQUUNSRLWPREKOJORMLKOXWNL3GMBSQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.705457525 (41h1m30.176668264s) [noise=true , address=145.79.227.32:4835 , piece_id=PJURUANT2XJY5PNYTSO5W2EDT3HYFAC2BFNCANKOAZ3I6P2GRC3Q, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.706486825 (41h1m30.177697554s) [noise=true , address=145.79.227.24:4582 , piece_id=MCQWT2LFD472TQDOAHLGQ5L2X7S654P6TL4J5MSP3CML5ZCURHMA, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.707626675 (41h1m30.178837414s) [noise=true , address=145.79.227.44:4193 , piece_id=327UUYIASLKFGUIQVY4P5UML4R2R4ODLUAYXR3GISKIT6CFDUGAQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.708631078 (41h1m30.179841817s) [noise=true , address=145.79.227.40:4050 , piece_id=QFSFC3WFER7VBVZNB3XLRIEFWGM4Q5VTAAJIWG4HAERKXSHZGZ4Q, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.709908630 (41h1m30.181119359s) [noise=true , address=145.79.227.22:4510 , piece_id=HCJIUH3VECRBR744KIQ65UKSMTEEKUPHJWXSDBM5SEMUVBJ5YX6A, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.710331386 (41h1m30.181542125s) [noise=true , address=145.79.227.41:4087 , piece_id=6GY2NC7F26VLU5NEN65OYU7YIHJYSTI5GNVRTNXM5HO5XQMZKEPA, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.710803685 (41h1m30.182014424s) [noise=true , address=45.91.116.4:4937 , piece_id=VFAPMIAB2G2BBOMNSM5EXONUL5BNORMR7KXIYSTQ7UFRVASVTQHQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:55.513663514 (41h1m31.984874243s) [upload=36107]: Next returned nil segment Dec 5 09:56:55.513682680 (41h1m31.98489341s) [upload=36107]: Waiting for error group managing segments... Dec 5 09:56:55.808504186 (41h1m32.279714915s) [noise=true , address=77.90.31.21:4878 , piece_id=RCFTETF7SGR6FTUBJWWBJDKOZIHZKBMSUBZYVWYKF2MP56PMIWMQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.866933347 (41h1m32.338144086s) [noise=true , address=145.79.227.10:4401 , piece_id=XAVSV3SGPDDLZWGKQLHA2CVYTGOPO7J3YVEO5C3MYIN5SJEUZ2OQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.869510454 (41h1m32.340721183s) [noise=true , address=145.79.227.28:4725 , piece_id=ZE642TJU4ZJKBOHQ6K6GPLPTN2YKC4BOP325LN7QXALZSDPG66HQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.875082795 (41h1m32.346293524s) [noise=true , address=145.79.227.43:4159 , piece_id=S5T27VFA6UNNONPZBZILNTQUUNSRLWPREKOJORMLKOXWNL3GMBSQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.884542648 (41h1m32.355753387s) [noise=true , address=145.79.227.24:4583 , piece_id=IB6T3J3CF3P3A54JEHHBK6KEPMUEI32B4QYSSA2G2BUMIVUC7DGQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.888876321 (41h1m32.36008707s) [noise=true , address=145.79.227.40:4050 , piece_id=QFSFC3WFER7VBVZNB3XLRIEFWGM4Q5VTAAJIWG4HAERKXSHZGZ4Q, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.889468719 (41h1m32.360679448s) [noise=true , address=145.79.227.26:4654 , piece_id=23NNREG47WSZWJNPZWP275L62YBBSTB6RQWTINIRTTAYU66DRT7Q, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.900234388 (41h1m32.371445117s) [noise=true , address=45.91.116.4:4937 , piece_id=VFAPMIAB2G2BBOMNSM5EXONUL5BNORMR7KXIYSTQ7UFRVASVTQHQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.923941432 (41h1m32.395152161s) [noise=true , address=145.79.227.42:4121 , piece_id=6D47MY4GTQ3YQOGHS2V2OTO2VFDL6XKKS2R4VA6VZZ557WYFY34A, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.929131404 (41h1m32.400342133s) [noise=true , address=145.79.227.22:4510 , piece_id=HCJIUH3VECRBR744KIQ65UKSMTEEKUPHJWXSDBM5SEMUVBJ5YX6A, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.180555093 (41h1m32.651765822s) [noise=true , address=145.79.227.32:4835 , piece_id=PJURUANT2XJY5PNYTSO5W2EDT3HYFAC2BFNCANKOAZ3I6P2GRC3Q, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.587529048 (41h1m33.058739787s) [noise=true , address=145.79.227.41:4087 , piece_id=6GY2NC7F26VLU5NEN65OYU7YIHJYSTI5GNVRTNXM5HO5XQMZKEPA, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.671173874 (41h1m33.142384603s) [noise=true , address=145.79.227.27:4691 , piece_id=Z53YBOXTW4L23P4M3EOW6ADUHDWHIJVRDT7IL4I5FUPPZOW2MRLQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.671209982 (41h1m33.142420711s) [seg_pos={0 0}, upload=36107]: Segment reached optimal threshold of 13 pieces. Dec 5 09:56:56.671423689 (41h1m33.142634428s) [noise=true , address=145.79.227.24:4582 , piece_id=MCQWT2LFD472TQDOAHLGQ5L2X7S654P6TL4J5MSP3CML5ZCURHMA, seg_pos={0 0}, upload=36107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa): protocol: write tcp 188.166.225.170:54136->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:54136->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54136->145.79.227.24:4582: use of closed network connection Dec 5 09:56:56.671432325 (41h1m33.142643064s) [noise=true , address=145.79.227.24:4582 , piece_id=MCQWT2LFD472TQDOAHLGQ5L2X7S654P6TL4J5MSP3CML5ZCURHMA, seg_pos={0 0}, upload=36107]: Long tail cancellation due to optimal threshold Dec 5 09:56:56.671496377 (41h1m33.142707106s) [noise=true , address=145.79.227.44:4193 , piece_id=327UUYIASLKFGUIQVY4P5UML4R2R4ODLUAYXR3GISKIT6CFDUGAQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: write tcp 188.166.225.170:39276->145.79.227.44:4193: use of closed network connection; write tcp 188.166.225.170:39276->145.79.227.44:4193: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39276->145.79.227.44:4193: use of closed network connection Dec 5 09:56:56.671501627 (41h1m33.142712366s) [noise=true , address=145.79.227.44:4193 , piece_id=327UUYIASLKFGUIQVY4P5UML4R2R4ODLUAYXR3GISKIT6CFDUGAQ, seg_pos={0 0}, upload=36107]: Long tail cancellation due to optimal threshold Dec 5 09:56:56.671516466 (41h1m33.142727195s) [noise=true , address=77.90.31.16:4735 , piece_id=V4YJIHVMXYTBK7IHU272RYCKUXI44B2MXRLETMWMO2WAI2XRU4SQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: write tcp 188.166.225.170:41866->77.90.31.16:4735: use of closed network connection; storage node overloaded, request limit: 2000 Dec 5 09:56:56.671525944 (41h1m33.142736673s) [noise=true , address=77.90.31.16:4735 , piece_id=V4YJIHVMXYTBK7IHU272RYCKUXI44B2MXRLETMWMO2WAI2XRU4SQ, seg_pos={0 0}, upload=36107]: Long tail cancellation due to optimal threshold Dec 5 09:56:56.671548016 (41h1m33.142758745s) [seg_pos={0 0}, upload=36107]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 5 09:56:56.671623500 (41h1m33.142834229s) [upload=36107]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 5 09:56:56.671631204 (41h1m33.142841933s) [upload=36107]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 5 09:56:57.112987571 (41h1m33.5841983s) [upload=36107]: Done uploading segments... Dec 5 09:56:57.113007228 (41h1m33.584217967s) [upload=36107]: Upload finished. err: <nil> Duration: 8.108720912s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 7.969943607s trace: https://jaeger.dev.storj.io/trace/55b376cc76cfe920 FAIL (2025-12-05 10:41:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 9 (41h1m29.913335078s) [upload=36107]: Uploading segments... Dec 5 09:56:53.442159225 (41h1m29.913369955s) [upload=36107]: Got next segment. Inline: false Dec 5 09:56:53.442188120 (41h1m29.913398849s) [upload=36107]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 5 09:56:53.442194372 (41h1m29.913405101s) [upload=36107]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 5 09:56:53.701976567 (41h1m30.173187296s) [seg_pos={0 0}, upload=36107]: Begin upload segment... Dec 5 09:56:53.702020460 (41h1m30.173231189s) [seg_pos={0 0}, upload=36107]: Done begin upload segment. Dec 5 09:56:53.702056649 (41h1m30.173267388s) [noise=true , address=77.90.31.21:4878 , piece_id=RCFTETF7SGR6FTUBJWWBJDKOZIHZKBMSUBZYVWYKF2MP56PMIWMQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702064965 (41h1m30.173275694s) [noise=true , address=145.79.227.28:4725 , piece_id=ZE642TJU4ZJKBOHQ6K6GPLPTN2YKC4BOP325LN7QXALZSDPG66HQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702111533 (41h1m30.173322272s) [noise=true , address=145.79.227.42:4121 , piece_id=6D47MY4GTQ3YQOGHS2V2OTO2VFDL6XKKS2R4VA6VZZ557WYFY34A, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702548986 (41h1m30.173759766s) [noise=true , address=145.79.227.27:4691 , piece_id=Z53YBOXTW4L23P4M3EOW6ADUHDWHIJVRDT7IL4I5FUPPZOW2MRLQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702931455 (41h1m30.174142194s) [noise=true , address=77.90.31.16:4735 , piece_id=V4YJIHVMXYTBK7IHU272RYCKUXI44B2MXRLETMWMO2WAI2XRU4SQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.703301079 (41h1m30.174511808s) [noise=true , address=145.79.227.10:4401 , piece_id=XAVSV3SGPDDLZWGKQLHA2CVYTGOPO7J3YVEO5C3MYIN5SJEUZ2OQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.703633000 (41h1m30.174843729s) [noise=true , address=145.79.227.24:4583 , piece_id=IB6T3J3CF3P3A54JEHHBK6KEPMUEI32B4QYSSA2G2BUMIVUC7DGQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.703957648 (41h1m30.175168377s) [noise=true , address=145.79.227.26:4654 , piece_id=23NNREG47WSZWJNPZWP275L62YBBSTB6RQWTINIRTTAYU66DRT7Q, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.704788621 (41h1m30.17599936s) [noise=true , address=145.79.227.43:4159 , piece_id=S5T27VFA6UNNONPZBZILNTQUUNSRLWPREKOJORMLKOXWNL3GMBSQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.705457525 (41h1m30.176668264s) [noise=true , address=145.79.227.32:4835 , piece_id=PJURUANT2XJY5PNYTSO5W2EDT3HYFAC2BFNCANKOAZ3I6P2GRC3Q, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.706486825 (41h1m30.177697554s) [noise=true , address=145.79.227.24:4582 , piece_id=MCQWT2LFD472TQDOAHLGQ5L2X7S654P6TL4J5MSP3CML5ZCURHMA, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.707626675 (41h1m30.178837414s) [noise=true , address=145.79.227.44:4193 , piece_id=327UUYIASLKFGUIQVY4P5UML4R2R4ODLUAYXR3GISKIT6CFDUGAQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.708631078 (41h1m30.179841817s) [noise=true , address=145.79.227.40:4050 , piece_id=QFSFC3WFER7VBVZNB3XLRIEFWGM4Q5VTAAJIWG4HAERKXSHZGZ4Q, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.709908630 (41h1m30.181119359s) [noise=true , address=145.79.227.22:4510 , piece_id=HCJIUH3VECRBR744KIQ65UKSMTEEKUPHJWXSDBM5SEMUVBJ5YX6A, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.710331386 (41h1m30.181542125s) [noise=true , address=145.79.227.41:4087 , piece_id=6GY2NC7F26VLU5NEN65OYU7YIHJYSTI5GNVRTNXM5HO5XQMZKEPA, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.710803685 (41h1m30.182014424s) [noise=true , address=45.91.116.4:4937 , piece_id=VFAPMIAB2G2BBOMNSM5EXONUL5BNORMR7KXIYSTQ7UFRVASVTQHQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:55.513663514 (41h1m31.984874243s) [upload=36107]: Next returned nil segment Dec 5 09:56:55.513682680 (41h1m31.98489341s) [upload=36107]: Waiting for error group managing segments... Dec 5 09:56:55.808504186 (41h1m32.279714915s) [noise=true , address=77.90.31.21:4878 , piece_id=RCFTETF7SGR6FTUBJWWBJDKOZIHZKBMSUBZYVWYKF2MP56PMIWMQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.866933347 (41h1m32.338144086s) [noise=true , address=145.79.227.10:4401 , piece_id=XAVSV3SGPDDLZWGKQLHA2CVYTGOPO7J3YVEO5C3MYIN5SJEUZ2OQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.869510454 (41h1m32.340721183s) [noise=true , address=145.79.227.28:4725 , piece_id=ZE642TJU4ZJKBOHQ6K6GPLPTN2YKC4BOP325LN7QXALZSDPG66HQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.875082795 (41h1m32.346293524s) [noise=true , address=145.79.227.43:4159 , piece_id=S5T27VFA6UNNONPZBZILNTQUUNSRLWPREKOJORMLKOXWNL3GMBSQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.884542648 (41h1m32.355753387s) [noise=true , address=145.79.227.24:4583 , piece_id=IB6T3J3CF3P3A54JEHHBK6KEPMUEI32B4QYSSA2G2BUMIVUC7DGQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.888876321 (41h1m32.36008707s) [noise=true , address=145.79.227.40:4050 , piece_id=QFSFC3WFER7VBVZNB3XLRIEFWGM4Q5VTAAJIWG4HAERKXSHZGZ4Q, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.889468719 (41h1m32.360679448s) [noise=true , address=145.79.227.26:4654 , piece_id=23NNREG47WSZWJNPZWP275L62YBBSTB6RQWTINIRTTAYU66DRT7Q, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.900234388 (41h1m32.371445117s) [noise=true , address=45.91.116.4:4937 , piece_id=VFAPMIAB2G2BBOMNSM5EXONUL5BNORMR7KXIYSTQ7UFRVASVTQHQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.923941432 (41h1m32.395152161s) [noise=true , address=145.79.227.42:4121 , piece_id=6D47MY4GTQ3YQOGHS2V2OTO2VFDL6XKKS2R4VA6VZZ557WYFY34A, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.929131404 (41h1m32.400342133s) [noise=true , address=145.79.227.22:4510 , piece_id=HCJIUH3VECRBR744KIQ65UKSMTEEKUPHJWXSDBM5SEMUVBJ5YX6A, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.180555093 (41h1m32.651765822s) [noise=true , address=145.79.227.32:4835 , piece_id=PJURUANT2XJY5PNYTSO5W2EDT3HYFAC2BFNCANKOAZ3I6P2GRC3Q, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.587529048 (41h1m33.058739787s) [noise=true , address=145.79.227.41:4087 , piece_id=6GY2NC7F26VLU5NEN65OYU7YIHJYSTI5GNVRTNXM5HO5XQMZKEPA, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.671173874 (41h1m33.142384603s) [noise=true , address=145.79.227.27:4691 , piece_id=Z53YBOXTW4L23P4M3EOW6ADUHDWHIJVRDT7IL4I5FUPPZOW2MRLQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.671209982 (41h1m33.142420711s) [seg_pos={0 0}, upload=36107]: Segment reached optimal threshold of 13 pieces. Dec 5 09:56:56.671423689 (41h1m33.142634428s) [noise=true , address=145.79.227.24:4582 , piece_id=MCQWT2LFD472TQDOAHLGQ5L2X7S654P6TL4J5MSP3CML5ZCURHMA, seg_pos={0 0}, upload=36107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa): protocol: write tcp 188.166.225.170:54136->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:54136->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54136->145.79.227.24:4582: use of closed network connection Dec 5 09:56:56.671432325 (41h1m33.142643064s) [noise=true , address=145.79.227.24:4582 , piece_id=MCQWT2LFD472TQDOAHLGQ5L2X7S654P6TL4J5MSP3CML5ZCURHMA, seg_pos={0 0}, upload=36107]: Long tail cancellation due to optimal threshold Dec 5 09:56:56.671496377 (41h1m33.142707106s) [noise=true , address=145.79.227.44:4193 , piece_id=327UUYIASLKFGUIQVY4P5UML4R2R4ODLUAYXR3GISKIT6CFDUGAQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: write tcp 188.166.225.170:39276->145.79.227.44:4193: use of closed network connection; write tcp 188.166.225.170:39276->145.79.227.44:4193: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39276->145.79.227.44:4193: use of closed network connection Dec 5 09:56:56.671501627 (41h1m33.142712366s) [noise=true , address=145.79.227.44:4193 , piece_id=327UUYIASLKFGUIQVY4P5UML4R2R4ODLUAYXR3GISKIT6CFDUGAQ, seg_pos={0 0}, upload=36107]: Long tail cancellation due to optimal threshold Dec 5 09:56:56.671516466 (41h1m33.142727195s) [noise=true , address=77.90.31.16:4735 , piece_id=V4YJIHVMXYTBK7IHU272RYCKUXI44B2MXRLETMWMO2WAI2XRU4SQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: write tcp 188.166.225.170:41866->77.90.31.16:4735: use of closed network connection; storage node overloaded, request limit: 2000 Dec 5 09:56:56.671525944 (41h1m33.142736673s) [noise=true , address=77.90.31.16:4735 , piece_id=V4YJIHVMXYTBK7IHU272RYCKUXI44B2MXRLETMWMO2WAI2XRU4SQ, seg_pos={0 0}, upload=36107]: Long tail cancellation due to optimal threshold Dec 5 09:56:56.671548016 (41h1m33.142758745s) [seg_pos={0 0}, upload=36107]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 5 09:56:56.671623500 (41h1m33.142834229s) [upload=36107]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 5 09:56:56.671631204 (41h1m33.142841933s) [upload=36107]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 5 09:56:57.112987571 (41h1m33.5841983s) [upload=36107]: Done uploading segments... Dec 5 09:56:57.113007228 (41h1m33.584217967s) [upload=36107]: Upload finished. err: <nil> Duration: 7.9699767s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 6.133094954s trace: https://jaeger.dev.storj.io/trace/55b376cc76cfe920 FAIL (2025-12-05 10:40:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 9 (41h1m29.913335078s) [upload=36107]: Uploading segments... Dec 5 09:56:53.442159225 (41h1m29.913369955s) [upload=36107]: Got next segment. Inline: false Dec 5 09:56:53.442188120 (41h1m29.913398849s) [upload=36107]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 5 09:56:53.442194372 (41h1m29.913405101s) [upload=36107]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 5 09:56:53.701976567 (41h1m30.173187296s) [seg_pos={0 0}, upload=36107]: Begin upload segment... Dec 5 09:56:53.702020460 (41h1m30.173231189s) [seg_pos={0 0}, upload=36107]: Done begin upload segment. Dec 5 09:56:53.702056649 (41h1m30.173267388s) [noise=true , address=77.90.31.21:4878 , piece_id=RCFTETF7SGR6FTUBJWWBJDKOZIHZKBMSUBZYVWYKF2MP56PMIWMQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702064965 (41h1m30.173275694s) [noise=true , address=145.79.227.28:4725 , piece_id=ZE642TJU4ZJKBOHQ6K6GPLPTN2YKC4BOP325LN7QXALZSDPG66HQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702111533 (41h1m30.173322272s) [noise=true , address=145.79.227.42:4121 , piece_id=6D47MY4GTQ3YQOGHS2V2OTO2VFDL6XKKS2R4VA6VZZ557WYFY34A, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702548986 (41h1m30.173759766s) [noise=true , address=145.79.227.27:4691 , piece_id=Z53YBOXTW4L23P4M3EOW6ADUHDWHIJVRDT7IL4I5FUPPZOW2MRLQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702931455 (41h1m30.174142194s) [noise=true , address=77.90.31.16:4735 , piece_id=V4YJIHVMXYTBK7IHU272RYCKUXI44B2MXRLETMWMO2WAI2XRU4SQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.703301079 (41h1m30.174511808s) [noise=true , address=145.79.227.10:4401 , piece_id=XAVSV3SGPDDLZWGKQLHA2CVYTGOPO7J3YVEO5C3MYIN5SJEUZ2OQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.703633000 (41h1m30.174843729s) [noise=true , address=145.79.227.24:4583 , piece_id=IB6T3J3CF3P3A54JEHHBK6KEPMUEI32B4QYSSA2G2BUMIVUC7DGQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.703957648 (41h1m30.175168377s) [noise=true , address=145.79.227.26:4654 , piece_id=23NNREG47WSZWJNPZWP275L62YBBSTB6RQWTINIRTTAYU66DRT7Q, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.704788621 (41h1m30.17599936s) [noise=true , address=145.79.227.43:4159 , piece_id=S5T27VFA6UNNONPZBZILNTQUUNSRLWPREKOJORMLKOXWNL3GMBSQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.705457525 (41h1m30.176668264s) [noise=true , address=145.79.227.32:4835 , piece_id=PJURUANT2XJY5PNYTSO5W2EDT3HYFAC2BFNCANKOAZ3I6P2GRC3Q, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.706486825 (41h1m30.177697554s) [noise=true , address=145.79.227.24:4582 , piece_id=MCQWT2LFD472TQDOAHLGQ5L2X7S654P6TL4J5MSP3CML5ZCURHMA, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.707626675 (41h1m30.178837414s) [noise=true , address=145.79.227.44:4193 , piece_id=327UUYIASLKFGUIQVY4P5UML4R2R4ODLUAYXR3GISKIT6CFDUGAQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.708631078 (41h1m30.179841817s) [noise=true , address=145.79.227.40:4050 , piece_id=QFSFC3WFER7VBVZNB3XLRIEFWGM4Q5VTAAJIWG4HAERKXSHZGZ4Q, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.709908630 (41h1m30.181119359s) [noise=true , address=145.79.227.22:4510 , piece_id=HCJIUH3VECRBR744KIQ65UKSMTEEKUPHJWXSDBM5SEMUVBJ5YX6A, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.710331386 (41h1m30.181542125s) [noise=true , address=145.79.227.41:4087 , piece_id=6GY2NC7F26VLU5NEN65OYU7YIHJYSTI5GNVRTNXM5HO5XQMZKEPA, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.710803685 (41h1m30.182014424s) [noise=true , address=45.91.116.4:4937 , piece_id=VFAPMIAB2G2BBOMNSM5EXONUL5BNORMR7KXIYSTQ7UFRVASVTQHQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:55.513663514 (41h1m31.984874243s) [upload=36107]: Next returned nil segment Dec 5 09:56:55.513682680 (41h1m31.98489341s) [upload=36107]: Waiting for error group managing segments... Dec 5 09:56:55.808504186 (41h1m32.279714915s) [noise=true , address=77.90.31.21:4878 , piece_id=RCFTETF7SGR6FTUBJWWBJDKOZIHZKBMSUBZYVWYKF2MP56PMIWMQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.866933347 (41h1m32.338144086s) [noise=true , address=145.79.227.10:4401 , piece_id=XAVSV3SGPDDLZWGKQLHA2CVYTGOPO7J3YVEO5C3MYIN5SJEUZ2OQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.869510454 (41h1m32.340721183s) [noise=true , address=145.79.227.28:4725 , piece_id=ZE642TJU4ZJKBOHQ6K6GPLPTN2YKC4BOP325LN7QXALZSDPG66HQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.875082795 (41h1m32.346293524s) [noise=true , address=145.79.227.43:4159 , piece_id=S5T27VFA6UNNONPZBZILNTQUUNSRLWPREKOJORMLKOXWNL3GMBSQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.884542648 (41h1m32.355753387s) [noise=true , address=145.79.227.24:4583 , piece_id=IB6T3J3CF3P3A54JEHHBK6KEPMUEI32B4QYSSA2G2BUMIVUC7DGQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.888876321 (41h1m32.36008707s) [noise=true , address=145.79.227.40:4050 , piece_id=QFSFC3WFER7VBVZNB3XLRIEFWGM4Q5VTAAJIWG4HAERKXSHZGZ4Q, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.889468719 (41h1m32.360679448s) [noise=true , address=145.79.227.26:4654 , piece_id=23NNREG47WSZWJNPZWP275L62YBBSTB6RQWTINIRTTAYU66DRT7Q, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.900234388 (41h1m32.371445117s) [noise=true , address=45.91.116.4:4937 , piece_id=VFAPMIAB2G2BBOMNSM5EXONUL5BNORMR7KXIYSTQ7UFRVASVTQHQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.923941432 (41h1m32.395152161s) [noise=true , address=145.79.227.42:4121 , piece_id=6D47MY4GTQ3YQOGHS2V2OTO2VFDL6XKKS2R4VA6VZZ557WYFY34A, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.929131404 (41h1m32.400342133s) [noise=true , address=145.79.227.22:4510 , piece_id=HCJIUH3VECRBR744KIQ65UKSMTEEKUPHJWXSDBM5SEMUVBJ5YX6A, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.180555093 (41h1m32.651765822s) [noise=true , address=145.79.227.32:4835 , piece_id=PJURUANT2XJY5PNYTSO5W2EDT3HYFAC2BFNCANKOAZ3I6P2GRC3Q, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.587529048 (41h1m33.058739787s) [noise=true , address=145.79.227.41:4087 , piece_id=6GY2NC7F26VLU5NEN65OYU7YIHJYSTI5GNVRTNXM5HO5XQMZKEPA, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.671173874 (41h1m33.142384603s) [noise=true , address=145.79.227.27:4691 , piece_id=Z53YBOXTW4L23P4M3EOW6ADUHDWHIJVRDT7IL4I5FUPPZOW2MRLQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.671209982 (41h1m33.142420711s) [seg_pos={0 0}, upload=36107]: Segment reached optimal threshold of 13 pieces. Dec 5 09:56:56.671423689 (41h1m33.142634428s) [noise=true , address=145.79.227.24:4582 , piece_id=MCQWT2LFD472TQDOAHLGQ5L2X7S654P6TL4J5MSP3CML5ZCURHMA, seg_pos={0 0}, upload=36107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa): protocol: write tcp 188.166.225.170:54136->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:54136->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54136->145.79.227.24:4582: use of closed network connection Dec 5 09:56:56.671432325 (41h1m33.142643064s) [noise=true , address=145.79.227.24:4582 , piece_id=MCQWT2LFD472TQDOAHLGQ5L2X7S654P6TL4J5MSP3CML5ZCURHMA, seg_pos={0 0}, upload=36107]: Long tail cancellation due to optimal threshold Dec 5 09:56:56.671496377 (41h1m33.142707106s) [noise=true , address=145.79.227.44:4193 , piece_id=327UUYIASLKFGUIQVY4P5UML4R2R4ODLUAYXR3GISKIT6CFDUGAQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: write tcp 188.166.225.170:39276->145.79.227.44:4193: use of closed network connection; write tcp 188.166.225.170:39276->145.79.227.44:4193: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39276->145.79.227.44:4193: use of closed network connection Dec 5 09:56:56.671501627 (41h1m33.142712366s) [noise=true , address=145.79.227.44:4193 , piece_id=327UUYIASLKFGUIQVY4P5UML4R2R4ODLUAYXR3GISKIT6CFDUGAQ, seg_pos={0 0}, upload=36107]: Long tail cancellation due to optimal threshold Dec 5 09:56:56.671516466 (41h1m33.142727195s) [noise=true , address=77.90.31.16:4735 , piece_id=V4YJIHVMXYTBK7IHU272RYCKUXI44B2MXRLETMWMO2WAI2XRU4SQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: write tcp 188.166.225.170:41866->77.90.31.16:4735: use of closed network connection; storage node overloaded, request limit: 2000 Dec 5 09:56:56.671525944 (41h1m33.142736673s) [noise=true , address=77.90.31.16:4735 , piece_id=V4YJIHVMXYTBK7IHU272RYCKUXI44B2MXRLETMWMO2WAI2XRU4SQ, seg_pos={0 0}, upload=36107]: Long tail cancellation due to optimal threshold Dec 5 09:56:56.671548016 (41h1m33.142758745s) [seg_pos={0 0}, upload=36107]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 5 09:56:56.671623500 (41h1m33.142834229s) [upload=36107]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 5 09:56:56.671631204 (41h1m33.142841933s) [upload=36107]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 5 09:56:57.112987571 (41h1m33.5841983s) [upload=36107]: Done uploading segments... Dec 5 09:56:57.113007228 (41h1m33.584217967s) [upload=36107]: Upload finished. err: <nil> Duration: 6.133122075s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 7.02149783s trace: https://jaeger.dev.storj.io/trace/55b376cc76cfe920 FAIL (2025-12-05 10:39:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 9 (41h1m29.913335078s) [upload=36107]: Uploading segments... Dec 5 09:56:53.442159225 (41h1m29.913369955s) [upload=36107]: Got next segment. Inline: false Dec 5 09:56:53.442188120 (41h1m29.913398849s) [upload=36107]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 5 09:56:53.442194372 (41h1m29.913405101s) [upload=36107]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 5 09:56:53.701976567 (41h1m30.173187296s) [seg_pos={0 0}, upload=36107]: Begin upload segment... Dec 5 09:56:53.702020460 (41h1m30.173231189s) [seg_pos={0 0}, upload=36107]: Done begin upload segment. Dec 5 09:56:53.702056649 (41h1m30.173267388s) [noise=true , address=77.90.31.21:4878 , piece_id=RCFTETF7SGR6FTUBJWWBJDKOZIHZKBMSUBZYVWYKF2MP56PMIWMQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702064965 (41h1m30.173275694s) [noise=true , address=145.79.227.28:4725 , piece_id=ZE642TJU4ZJKBOHQ6K6GPLPTN2YKC4BOP325LN7QXALZSDPG66HQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702111533 (41h1m30.173322272s) [noise=true , address=145.79.227.42:4121 , piece_id=6D47MY4GTQ3YQOGHS2V2OTO2VFDL6XKKS2R4VA6VZZ557WYFY34A, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702548986 (41h1m30.173759766s) [noise=true , address=145.79.227.27:4691 , piece_id=Z53YBOXTW4L23P4M3EOW6ADUHDWHIJVRDT7IL4I5FUPPZOW2MRLQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702931455 (41h1m30.174142194s) [noise=true , address=77.90.31.16:4735 , piece_id=V4YJIHVMXYTBK7IHU272RYCKUXI44B2MXRLETMWMO2WAI2XRU4SQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.703301079 (41h1m30.174511808s) [noise=true , address=145.79.227.10:4401 , piece_id=XAVSV3SGPDDLZWGKQLHA2CVYTGOPO7J3YVEO5C3MYIN5SJEUZ2OQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.703633000 (41h1m30.174843729s) [noise=true , address=145.79.227.24:4583 , piece_id=IB6T3J3CF3P3A54JEHHBK6KEPMUEI32B4QYSSA2G2BUMIVUC7DGQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.703957648 (41h1m30.175168377s) [noise=true , address=145.79.227.26:4654 , piece_id=23NNREG47WSZWJNPZWP275L62YBBSTB6RQWTINIRTTAYU66DRT7Q, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.704788621 (41h1m30.17599936s) [noise=true , address=145.79.227.43:4159 , piece_id=S5T27VFA6UNNONPZBZILNTQUUNSRLWPREKOJORMLKOXWNL3GMBSQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.705457525 (41h1m30.176668264s) [noise=true , address=145.79.227.32:4835 , piece_id=PJURUANT2XJY5PNYTSO5W2EDT3HYFAC2BFNCANKOAZ3I6P2GRC3Q, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.706486825 (41h1m30.177697554s) [noise=true , address=145.79.227.24:4582 , piece_id=MCQWT2LFD472TQDOAHLGQ5L2X7S654P6TL4J5MSP3CML5ZCURHMA, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.707626675 (41h1m30.178837414s) [noise=true , address=145.79.227.44:4193 , piece_id=327UUYIASLKFGUIQVY4P5UML4R2R4ODLUAYXR3GISKIT6CFDUGAQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.708631078 (41h1m30.179841817s) [noise=true , address=145.79.227.40:4050 , piece_id=QFSFC3WFER7VBVZNB3XLRIEFWGM4Q5VTAAJIWG4HAERKXSHZGZ4Q, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.709908630 (41h1m30.181119359s) [noise=true , address=145.79.227.22:4510 , piece_id=HCJIUH3VECRBR744KIQ65UKSMTEEKUPHJWXSDBM5SEMUVBJ5YX6A, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.710331386 (41h1m30.181542125s) [noise=true , address=145.79.227.41:4087 , piece_id=6GY2NC7F26VLU5NEN65OYU7YIHJYSTI5GNVRTNXM5HO5XQMZKEPA, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.710803685 (41h1m30.182014424s) [noise=true , address=45.91.116.4:4937 , piece_id=VFAPMIAB2G2BBOMNSM5EXONUL5BNORMR7KXIYSTQ7UFRVASVTQHQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:55.513663514 (41h1m31.984874243s) [upload=36107]: Next returned nil segment Dec 5 09:56:55.513682680 (41h1m31.98489341s) [upload=36107]: Waiting for error group managing segments... Dec 5 09:56:55.808504186 (41h1m32.279714915s) [noise=true , address=77.90.31.21:4878 , piece_id=RCFTETF7SGR6FTUBJWWBJDKOZIHZKBMSUBZYVWYKF2MP56PMIWMQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.866933347 (41h1m32.338144086s) [noise=true , address=145.79.227.10:4401 , piece_id=XAVSV3SGPDDLZWGKQLHA2CVYTGOPO7J3YVEO5C3MYIN5SJEUZ2OQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.869510454 (41h1m32.340721183s) [noise=true , address=145.79.227.28:4725 , piece_id=ZE642TJU4ZJKBOHQ6K6GPLPTN2YKC4BOP325LN7QXALZSDPG66HQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.875082795 (41h1m32.346293524s) [noise=true , address=145.79.227.43:4159 , piece_id=S5T27VFA6UNNONPZBZILNTQUUNSRLWPREKOJORMLKOXWNL3GMBSQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.884542648 (41h1m32.355753387s) [noise=true , address=145.79.227.24:4583 , piece_id=IB6T3J3CF3P3A54JEHHBK6KEPMUEI32B4QYSSA2G2BUMIVUC7DGQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.888876321 (41h1m32.36008707s) [noise=true , address=145.79.227.40:4050 , piece_id=QFSFC3WFER7VBVZNB3XLRIEFWGM4Q5VTAAJIWG4HAERKXSHZGZ4Q, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.889468719 (41h1m32.360679448s) [noise=true , address=145.79.227.26:4654 , piece_id=23NNREG47WSZWJNPZWP275L62YBBSTB6RQWTINIRTTAYU66DRT7Q, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.900234388 (41h1m32.371445117s) [noise=true , address=45.91.116.4:4937 , piece_id=VFAPMIAB2G2BBOMNSM5EXONUL5BNORMR7KXIYSTQ7UFRVASVTQHQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.923941432 (41h1m32.395152161s) [noise=true , address=145.79.227.42:4121 , piece_id=6D47MY4GTQ3YQOGHS2V2OTO2VFDL6XKKS2R4VA6VZZ557WYFY34A, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.929131404 (41h1m32.400342133s) [noise=true , address=145.79.227.22:4510 , piece_id=HCJIUH3VECRBR744KIQ65UKSMTEEKUPHJWXSDBM5SEMUVBJ5YX6A, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.180555093 (41h1m32.651765822s) [noise=true , address=145.79.227.32:4835 , piece_id=PJURUANT2XJY5PNYTSO5W2EDT3HYFAC2BFNCANKOAZ3I6P2GRC3Q, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.587529048 (41h1m33.058739787s) [noise=true , address=145.79.227.41:4087 , piece_id=6GY2NC7F26VLU5NEN65OYU7YIHJYSTI5GNVRTNXM5HO5XQMZKEPA, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.671173874 (41h1m33.142384603s) [noise=true , address=145.79.227.27:4691 , piece_id=Z53YBOXTW4L23P4M3EOW6ADUHDWHIJVRDT7IL4I5FUPPZOW2MRLQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.671209982 (41h1m33.142420711s) [seg_pos={0 0}, upload=36107]: Segment reached optimal threshold of 13 pieces. Dec 5 09:56:56.671423689 (41h1m33.142634428s) [noise=true , address=145.79.227.24:4582 , piece_id=MCQWT2LFD472TQDOAHLGQ5L2X7S654P6TL4J5MSP3CML5ZCURHMA, seg_pos={0 0}, upload=36107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa): protocol: write tcp 188.166.225.170:54136->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:54136->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54136->145.79.227.24:4582: use of closed network connection Dec 5 09:56:56.671432325 (41h1m33.142643064s) [noise=true , address=145.79.227.24:4582 , piece_id=MCQWT2LFD472TQDOAHLGQ5L2X7S654P6TL4J5MSP3CML5ZCURHMA, seg_pos={0 0}, upload=36107]: Long tail cancellation due to optimal threshold Dec 5 09:56:56.671496377 (41h1m33.142707106s) [noise=true , address=145.79.227.44:4193 , piece_id=327UUYIASLKFGUIQVY4P5UML4R2R4ODLUAYXR3GISKIT6CFDUGAQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: write tcp 188.166.225.170:39276->145.79.227.44:4193: use of closed network connection; write tcp 188.166.225.170:39276->145.79.227.44:4193: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39276->145.79.227.44:4193: use of closed network connection Dec 5 09:56:56.671501627 (41h1m33.142712366s) [noise=true , address=145.79.227.44:4193 , piece_id=327UUYIASLKFGUIQVY4P5UML4R2R4ODLUAYXR3GISKIT6CFDUGAQ, seg_pos={0 0}, upload=36107]: Long tail cancellation due to optimal threshold Dec 5 09:56:56.671516466 (41h1m33.142727195s) [noise=true , address=77.90.31.16:4735 , piece_id=V4YJIHVMXYTBK7IHU272RYCKUXI44B2MXRLETMWMO2WAI2XRU4SQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: write tcp 188.166.225.170:41866->77.90.31.16:4735: use of closed network connection; storage node overloaded, request limit: 2000 Dec 5 09:56:56.671525944 (41h1m33.142736673s) [noise=true , address=77.90.31.16:4735 , piece_id=V4YJIHVMXYTBK7IHU272RYCKUXI44B2MXRLETMWMO2WAI2XRU4SQ, seg_pos={0 0}, upload=36107]: Long tail cancellation due to optimal threshold Dec 5 09:56:56.671548016 (41h1m33.142758745s) [seg_pos={0 0}, upload=36107]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 5 09:56:56.671623500 (41h1m33.142834229s) [upload=36107]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 5 09:56:56.671631204 (41h1m33.142841933s) [upload=36107]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 5 09:56:57.112987571 (41h1m33.5841983s) [upload=36107]: Done uploading segments... Dec 5 09:56:57.113007228 (41h1m33.584217967s) [upload=36107]: Upload finished. err: <nil> Duration: 7.021527196s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 8.296918583s trace: https://jaeger.dev.storj.io/trace/55b376cc76cfe920 FAIL (2025-12-05 10:38:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 9 (41h1m29.913335078s) [upload=36107]: Uploading segments... Dec 5 09:56:53.442159225 (41h1m29.913369955s) [upload=36107]: Got next segment. Inline: false Dec 5 09:56:53.442188120 (41h1m29.913398849s) [upload=36107]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 5 09:56:53.442194372 (41h1m29.913405101s) [upload=36107]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 5 09:56:53.701976567 (41h1m30.173187296s) [seg_pos={0 0}, upload=36107]: Begin upload segment... Dec 5 09:56:53.702020460 (41h1m30.173231189s) [seg_pos={0 0}, upload=36107]: Done begin upload segment. Dec 5 09:56:53.702056649 (41h1m30.173267388s) [noise=true , address=77.90.31.21:4878 , piece_id=RCFTETF7SGR6FTUBJWWBJDKOZIHZKBMSUBZYVWYKF2MP56PMIWMQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702064965 (41h1m30.173275694s) [noise=true , address=145.79.227.28:4725 , piece_id=ZE642TJU4ZJKBOHQ6K6GPLPTN2YKC4BOP325LN7QXALZSDPG66HQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702111533 (41h1m30.173322272s) [noise=true , address=145.79.227.42:4121 , piece_id=6D47MY4GTQ3YQOGHS2V2OTO2VFDL6XKKS2R4VA6VZZ557WYFY34A, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702548986 (41h1m30.173759766s) [noise=true , address=145.79.227.27:4691 , piece_id=Z53YBOXTW4L23P4M3EOW6ADUHDWHIJVRDT7IL4I5FUPPZOW2MRLQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702931455 (41h1m30.174142194s) [noise=true , address=77.90.31.16:4735 , piece_id=V4YJIHVMXYTBK7IHU272RYCKUXI44B2MXRLETMWMO2WAI2XRU4SQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.703301079 (41h1m30.174511808s) [noise=true , address=145.79.227.10:4401 , piece_id=XAVSV3SGPDDLZWGKQLHA2CVYTGOPO7J3YVEO5C3MYIN5SJEUZ2OQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.703633000 (41h1m30.174843729s) [noise=true , address=145.79.227.24:4583 , piece_id=IB6T3J3CF3P3A54JEHHBK6KEPMUEI32B4QYSSA2G2BUMIVUC7DGQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.703957648 (41h1m30.175168377s) [noise=true , address=145.79.227.26:4654 , piece_id=23NNREG47WSZWJNPZWP275L62YBBSTB6RQWTINIRTTAYU66DRT7Q, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.704788621 (41h1m30.17599936s) [noise=true , address=145.79.227.43:4159 , piece_id=S5T27VFA6UNNONPZBZILNTQUUNSRLWPREKOJORMLKOXWNL3GMBSQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.705457525 (41h1m30.176668264s) [noise=true , address=145.79.227.32:4835 , piece_id=PJURUANT2XJY5PNYTSO5W2EDT3HYFAC2BFNCANKOAZ3I6P2GRC3Q, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.706486825 (41h1m30.177697554s) [noise=true , address=145.79.227.24:4582 , piece_id=MCQWT2LFD472TQDOAHLGQ5L2X7S654P6TL4J5MSP3CML5ZCURHMA, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.707626675 (41h1m30.178837414s) [noise=true , address=145.79.227.44:4193 , piece_id=327UUYIASLKFGUIQVY4P5UML4R2R4ODLUAYXR3GISKIT6CFDUGAQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.708631078 (41h1m30.179841817s) [noise=true , address=145.79.227.40:4050 , piece_id=QFSFC3WFER7VBVZNB3XLRIEFWGM4Q5VTAAJIWG4HAERKXSHZGZ4Q, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.709908630 (41h1m30.181119359s) [noise=true , address=145.79.227.22:4510 , piece_id=HCJIUH3VECRBR744KIQ65UKSMTEEKUPHJWXSDBM5SEMUVBJ5YX6A, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.710331386 (41h1m30.181542125s) [noise=true , address=145.79.227.41:4087 , piece_id=6GY2NC7F26VLU5NEN65OYU7YIHJYSTI5GNVRTNXM5HO5XQMZKEPA, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.710803685 (41h1m30.182014424s) [noise=true , address=45.91.116.4:4937 , piece_id=VFAPMIAB2G2BBOMNSM5EXONUL5BNORMR7KXIYSTQ7UFRVASVTQHQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:55.513663514 (41h1m31.984874243s) [upload=36107]: Next returned nil segment Dec 5 09:56:55.513682680 (41h1m31.98489341s) [upload=36107]: Waiting for error group managing segments... Dec 5 09:56:55.808504186 (41h1m32.279714915s) [noise=true , address=77.90.31.21:4878 , piece_id=RCFTETF7SGR6FTUBJWWBJDKOZIHZKBMSUBZYVWYKF2MP56PMIWMQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.866933347 (41h1m32.338144086s) [noise=true , address=145.79.227.10:4401 , piece_id=XAVSV3SGPDDLZWGKQLHA2CVYTGOPO7J3YVEO5C3MYIN5SJEUZ2OQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.869510454 (41h1m32.340721183s) [noise=true , address=145.79.227.28:4725 , piece_id=ZE642TJU4ZJKBOHQ6K6GPLPTN2YKC4BOP325LN7QXALZSDPG66HQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.875082795 (41h1m32.346293524s) [noise=true , address=145.79.227.43:4159 , piece_id=S5T27VFA6UNNONPZBZILNTQUUNSRLWPREKOJORMLKOXWNL3GMBSQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.884542648 (41h1m32.355753387s) [noise=true , address=145.79.227.24:4583 , piece_id=IB6T3J3CF3P3A54JEHHBK6KEPMUEI32B4QYSSA2G2BUMIVUC7DGQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.888876321 (41h1m32.36008707s) [noise=true , address=145.79.227.40:4050 , piece_id=QFSFC3WFER7VBVZNB3XLRIEFWGM4Q5VTAAJIWG4HAERKXSHZGZ4Q, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.889468719 (41h1m32.360679448s) [noise=true , address=145.79.227.26:4654 , piece_id=23NNREG47WSZWJNPZWP275L62YBBSTB6RQWTINIRTTAYU66DRT7Q, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.900234388 (41h1m32.371445117s) [noise=true , address=45.91.116.4:4937 , piece_id=VFAPMIAB2G2BBOMNSM5EXONUL5BNORMR7KXIYSTQ7UFRVASVTQHQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.923941432 (41h1m32.395152161s) [noise=true , address=145.79.227.42:4121 , piece_id=6D47MY4GTQ3YQOGHS2V2OTO2VFDL6XKKS2R4VA6VZZ557WYFY34A, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.929131404 (41h1m32.400342133s) [noise=true , address=145.79.227.22:4510 , piece_id=HCJIUH3VECRBR744KIQ65UKSMTEEKUPHJWXSDBM5SEMUVBJ5YX6A, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.180555093 (41h1m32.651765822s) [noise=true , address=145.79.227.32:4835 , piece_id=PJURUANT2XJY5PNYTSO5W2EDT3HYFAC2BFNCANKOAZ3I6P2GRC3Q, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.587529048 (41h1m33.058739787s) [noise=true , address=145.79.227.41:4087 , piece_id=6GY2NC7F26VLU5NEN65OYU7YIHJYSTI5GNVRTNXM5HO5XQMZKEPA, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.671173874 (41h1m33.142384603s) [noise=true , address=145.79.227.27:4691 , piece_id=Z53YBOXTW4L23P4M3EOW6ADUHDWHIJVRDT7IL4I5FUPPZOW2MRLQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.671209982 (41h1m33.142420711s) [seg_pos={0 0}, upload=36107]: Segment reached optimal threshold of 13 pieces. Dec 5 09:56:56.671423689 (41h1m33.142634428s) [noise=true , address=145.79.227.24:4582 , piece_id=MCQWT2LFD472TQDOAHLGQ5L2X7S654P6TL4J5MSP3CML5ZCURHMA, seg_pos={0 0}, upload=36107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa): protocol: write tcp 188.166.225.170:54136->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:54136->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54136->145.79.227.24:4582: use of closed network connection Dec 5 09:56:56.671432325 (41h1m33.142643064s) [noise=true , address=145.79.227.24:4582 , piece_id=MCQWT2LFD472TQDOAHLGQ5L2X7S654P6TL4J5MSP3CML5ZCURHMA, seg_pos={0 0}, upload=36107]: Long tail cancellation due to optimal threshold Dec 5 09:56:56.671496377 (41h1m33.142707106s) [noise=true , address=145.79.227.44:4193 , piece_id=327UUYIASLKFGUIQVY4P5UML4R2R4ODLUAYXR3GISKIT6CFDUGAQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: write tcp 188.166.225.170:39276->145.79.227.44:4193: use of closed network connection; write tcp 188.166.225.170:39276->145.79.227.44:4193: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39276->145.79.227.44:4193: use of closed network connection Dec 5 09:56:56.671501627 (41h1m33.142712366s) [noise=true , address=145.79.227.44:4193 , piece_id=327UUYIASLKFGUIQVY4P5UML4R2R4ODLUAYXR3GISKIT6CFDUGAQ, seg_pos={0 0}, upload=36107]: Long tail cancellation due to optimal threshold Dec 5 09:56:56.671516466 (41h1m33.142727195s) [noise=true , address=77.90.31.16:4735 , piece_id=V4YJIHVMXYTBK7IHU272RYCKUXI44B2MXRLETMWMO2WAI2XRU4SQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: write tcp 188.166.225.170:41866->77.90.31.16:4735: use of closed network connection; storage node overloaded, request limit: 2000 Dec 5 09:56:56.671525944 (41h1m33.142736673s) [noise=true , address=77.90.31.16:4735 , piece_id=V4YJIHVMXYTBK7IHU272RYCKUXI44B2MXRLETMWMO2WAI2XRU4SQ, seg_pos={0 0}, upload=36107]: Long tail cancellation due to optimal threshold Dec 5 09:56:56.671548016 (41h1m33.142758745s) [seg_pos={0 0}, upload=36107]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 5 09:56:56.671623500 (41h1m33.142834229s) [upload=36107]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 5 09:56:56.671631204 (41h1m33.142841933s) [upload=36107]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 5 09:56:57.112987571 (41h1m33.5841983s) [upload=36107]: Done uploading segments... Dec 5 09:56:57.113007228 (41h1m33.584217967s) [upload=36107]: Upload finished. err: <nil> Duration: 8.296949923s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 7.904372605s trace: https://jaeger.dev.storj.io/trace/55b376cc76cfe920 FAIL (2025-12-05 10:37:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 9 (41h1m29.913335078s) [upload=36107]: Uploading segments... Dec 5 09:56:53.442159225 (41h1m29.913369955s) [upload=36107]: Got next segment. Inline: false Dec 5 09:56:53.442188120 (41h1m29.913398849s) [upload=36107]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 5 09:56:53.442194372 (41h1m29.913405101s) [upload=36107]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 5 09:56:53.701976567 (41h1m30.173187296s) [seg_pos={0 0}, upload=36107]: Begin upload segment... Dec 5 09:56:53.702020460 (41h1m30.173231189s) [seg_pos={0 0}, upload=36107]: Done begin upload segment. Dec 5 09:56:53.702056649 (41h1m30.173267388s) [noise=true , address=77.90.31.21:4878 , piece_id=RCFTETF7SGR6FTUBJWWBJDKOZIHZKBMSUBZYVWYKF2MP56PMIWMQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702064965 (41h1m30.173275694s) [noise=true , address=145.79.227.28:4725 , piece_id=ZE642TJU4ZJKBOHQ6K6GPLPTN2YKC4BOP325LN7QXALZSDPG66HQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702111533 (41h1m30.173322272s) [noise=true , address=145.79.227.42:4121 , piece_id=6D47MY4GTQ3YQOGHS2V2OTO2VFDL6XKKS2R4VA6VZZ557WYFY34A, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702548986 (41h1m30.173759766s) [noise=true , address=145.79.227.27:4691 , piece_id=Z53YBOXTW4L23P4M3EOW6ADUHDWHIJVRDT7IL4I5FUPPZOW2MRLQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702931455 (41h1m30.174142194s) [noise=true , address=77.90.31.16:4735 , piece_id=V4YJIHVMXYTBK7IHU272RYCKUXI44B2MXRLETMWMO2WAI2XRU4SQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.703301079 (41h1m30.174511808s) [noise=true , address=145.79.227.10:4401 , piece_id=XAVSV3SGPDDLZWGKQLHA2CVYTGOPO7J3YVEO5C3MYIN5SJEUZ2OQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.703633000 (41h1m30.174843729s) [noise=true , address=145.79.227.24:4583 , piece_id=IB6T3J3CF3P3A54JEHHBK6KEPMUEI32B4QYSSA2G2BUMIVUC7DGQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.703957648 (41h1m30.175168377s) [noise=true , address=145.79.227.26:4654 , piece_id=23NNREG47WSZWJNPZWP275L62YBBSTB6RQWTINIRTTAYU66DRT7Q, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.704788621 (41h1m30.17599936s) [noise=true , address=145.79.227.43:4159 , piece_id=S5T27VFA6UNNONPZBZILNTQUUNSRLWPREKOJORMLKOXWNL3GMBSQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.705457525 (41h1m30.176668264s) [noise=true , address=145.79.227.32:4835 , piece_id=PJURUANT2XJY5PNYTSO5W2EDT3HYFAC2BFNCANKOAZ3I6P2GRC3Q, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.706486825 (41h1m30.177697554s) [noise=true , address=145.79.227.24:4582 , piece_id=MCQWT2LFD472TQDOAHLGQ5L2X7S654P6TL4J5MSP3CML5ZCURHMA, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.707626675 (41h1m30.178837414s) [noise=true , address=145.79.227.44:4193 , piece_id=327UUYIASLKFGUIQVY4P5UML4R2R4ODLUAYXR3GISKIT6CFDUGAQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.708631078 (41h1m30.179841817s) [noise=true , address=145.79.227.40:4050 , piece_id=QFSFC3WFER7VBVZNB3XLRIEFWGM4Q5VTAAJIWG4HAERKXSHZGZ4Q, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.709908630 (41h1m30.181119359s) [noise=true , address=145.79.227.22:4510 , piece_id=HCJIUH3VECRBR744KIQ65UKSMTEEKUPHJWXSDBM5SEMUVBJ5YX6A, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.710331386 (41h1m30.181542125s) [noise=true , address=145.79.227.41:4087 , piece_id=6GY2NC7F26VLU5NEN65OYU7YIHJYSTI5GNVRTNXM5HO5XQMZKEPA, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.710803685 (41h1m30.182014424s) [noise=true , address=45.91.116.4:4937 , piece_id=VFAPMIAB2G2BBOMNSM5EXONUL5BNORMR7KXIYSTQ7UFRVASVTQHQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:55.513663514 (41h1m31.984874243s) [upload=36107]: Next returned nil segment Dec 5 09:56:55.513682680 (41h1m31.98489341s) [upload=36107]: Waiting for error group managing segments... Dec 5 09:56:55.808504186 (41h1m32.279714915s) [noise=true , address=77.90.31.21:4878 , piece_id=RCFTETF7SGR6FTUBJWWBJDKOZIHZKBMSUBZYVWYKF2MP56PMIWMQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.866933347 (41h1m32.338144086s) [noise=true , address=145.79.227.10:4401 , piece_id=XAVSV3SGPDDLZWGKQLHA2CVYTGOPO7J3YVEO5C3MYIN5SJEUZ2OQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.869510454 (41h1m32.340721183s) [noise=true , address=145.79.227.28:4725 , piece_id=ZE642TJU4ZJKBOHQ6K6GPLPTN2YKC4BOP325LN7QXALZSDPG66HQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.875082795 (41h1m32.346293524s) [noise=true , address=145.79.227.43:4159 , piece_id=S5T27VFA6UNNONPZBZILNTQUUNSRLWPREKOJORMLKOXWNL3GMBSQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.884542648 (41h1m32.355753387s) [noise=true , address=145.79.227.24:4583 , piece_id=IB6T3J3CF3P3A54JEHHBK6KEPMUEI32B4QYSSA2G2BUMIVUC7DGQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.888876321 (41h1m32.36008707s) [noise=true , address=145.79.227.40:4050 , piece_id=QFSFC3WFER7VBVZNB3XLRIEFWGM4Q5VTAAJIWG4HAERKXSHZGZ4Q, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.889468719 (41h1m32.360679448s) [noise=true , address=145.79.227.26:4654 , piece_id=23NNREG47WSZWJNPZWP275L62YBBSTB6RQWTINIRTTAYU66DRT7Q, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.900234388 (41h1m32.371445117s) [noise=true , address=45.91.116.4:4937 , piece_id=VFAPMIAB2G2BBOMNSM5EXONUL5BNORMR7KXIYSTQ7UFRVASVTQHQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.923941432 (41h1m32.395152161s) [noise=true , address=145.79.227.42:4121 , piece_id=6D47MY4GTQ3YQOGHS2V2OTO2VFDL6XKKS2R4VA6VZZ557WYFY34A, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.929131404 (41h1m32.400342133s) [noise=true , address=145.79.227.22:4510 , piece_id=HCJIUH3VECRBR744KIQ65UKSMTEEKUPHJWXSDBM5SEMUVBJ5YX6A, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.180555093 (41h1m32.651765822s) [noise=true , address=145.79.227.32:4835 , piece_id=PJURUANT2XJY5PNYTSO5W2EDT3HYFAC2BFNCANKOAZ3I6P2GRC3Q, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.587529048 (41h1m33.058739787s) [noise=true , address=145.79.227.41:4087 , piece_id=6GY2NC7F26VLU5NEN65OYU7YIHJYSTI5GNVRTNXM5HO5XQMZKEPA, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.671173874 (41h1m33.142384603s) [noise=true , address=145.79.227.27:4691 , piece_id=Z53YBOXTW4L23P4M3EOW6ADUHDWHIJVRDT7IL4I5FUPPZOW2MRLQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.671209982 (41h1m33.142420711s) [seg_pos={0 0}, upload=36107]: Segment reached optimal threshold of 13 pieces. Dec 5 09:56:56.671423689 (41h1m33.142634428s) [noise=true , address=145.79.227.24:4582 , piece_id=MCQWT2LFD472TQDOAHLGQ5L2X7S654P6TL4J5MSP3CML5ZCURHMA, seg_pos={0 0}, upload=36107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa): protocol: write tcp 188.166.225.170:54136->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:54136->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54136->145.79.227.24:4582: use of closed network connection Dec 5 09:56:56.671432325 (41h1m33.142643064s) [noise=true , address=145.79.227.24:4582 , piece_id=MCQWT2LFD472TQDOAHLGQ5L2X7S654P6TL4J5MSP3CML5ZCURHMA, seg_pos={0 0}, upload=36107]: Long tail cancellation due to optimal threshold Dec 5 09:56:56.671496377 (41h1m33.142707106s) [noise=true , address=145.79.227.44:4193 , piece_id=327UUYIASLKFGUIQVY4P5UML4R2R4ODLUAYXR3GISKIT6CFDUGAQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: write tcp 188.166.225.170:39276->145.79.227.44:4193: use of closed network connection; write tcp 188.166.225.170:39276->145.79.227.44:4193: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39276->145.79.227.44:4193: use of closed network connection Dec 5 09:56:56.671501627 (41h1m33.142712366s) [noise=true , address=145.79.227.44:4193 , piece_id=327UUYIASLKFGUIQVY4P5UML4R2R4ODLUAYXR3GISKIT6CFDUGAQ, seg_pos={0 0}, upload=36107]: Long tail cancellation due to optimal threshold Dec 5 09:56:56.671516466 (41h1m33.142727195s) [noise=true , address=77.90.31.16:4735 , piece_id=V4YJIHVMXYTBK7IHU272RYCKUXI44B2MXRLETMWMO2WAI2XRU4SQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: write tcp 188.166.225.170:41866->77.90.31.16:4735: use of closed network connection; storage node overloaded, request limit: 2000 Dec 5 09:56:56.671525944 (41h1m33.142736673s) [noise=true , address=77.90.31.16:4735 , piece_id=V4YJIHVMXYTBK7IHU272RYCKUXI44B2MXRLETMWMO2WAI2XRU4SQ, seg_pos={0 0}, upload=36107]: Long tail cancellation due to optimal threshold Dec 5 09:56:56.671548016 (41h1m33.142758745s) [seg_pos={0 0}, upload=36107]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 5 09:56:56.671623500 (41h1m33.142834229s) [upload=36107]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 5 09:56:56.671631204 (41h1m33.142841933s) [upload=36107]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 5 09:56:57.112987571 (41h1m33.5841983s) [upload=36107]: Done uploading segments... Dec 5 09:56:57.113007228 (41h1m33.584217967s) [upload=36107]: Upload finished. err: <nil> Duration: 7.90439608s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 6.46166286s trace: https://jaeger.dev.storj.io/trace/55b376cc76cfe920 FAIL (2025-12-05 10:36:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 9 (41h1m29.913335078s) [upload=36107]: Uploading segments... Dec 5 09:56:53.442159225 (41h1m29.913369955s) [upload=36107]: Got next segment. Inline: false Dec 5 09:56:53.442188120 (41h1m29.913398849s) [upload=36107]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 5 09:56:53.442194372 (41h1m29.913405101s) [upload=36107]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 5 09:56:53.701976567 (41h1m30.173187296s) [seg_pos={0 0}, upload=36107]: Begin upload segment... Dec 5 09:56:53.702020460 (41h1m30.173231189s) [seg_pos={0 0}, upload=36107]: Done begin upload segment. Dec 5 09:56:53.702056649 (41h1m30.173267388s) [noise=true , address=77.90.31.21:4878 , piece_id=RCFTETF7SGR6FTUBJWWBJDKOZIHZKBMSUBZYVWYKF2MP56PMIWMQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702064965 (41h1m30.173275694s) [noise=true , address=145.79.227.28:4725 , piece_id=ZE642TJU4ZJKBOHQ6K6GPLPTN2YKC4BOP325LN7QXALZSDPG66HQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702111533 (41h1m30.173322272s) [noise=true , address=145.79.227.42:4121 , piece_id=6D47MY4GTQ3YQOGHS2V2OTO2VFDL6XKKS2R4VA6VZZ557WYFY34A, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702548986 (41h1m30.173759766s) [noise=true , address=145.79.227.27:4691 , piece_id=Z53YBOXTW4L23P4M3EOW6ADUHDWHIJVRDT7IL4I5FUPPZOW2MRLQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702931455 (41h1m30.174142194s) [noise=true , address=77.90.31.16:4735 , piece_id=V4YJIHVMXYTBK7IHU272RYCKUXI44B2MXRLETMWMO2WAI2XRU4SQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.703301079 (41h1m30.174511808s) [noise=true , address=145.79.227.10:4401 , piece_id=XAVSV3SGPDDLZWGKQLHA2CVYTGOPO7J3YVEO5C3MYIN5SJEUZ2OQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.703633000 (41h1m30.174843729s) [noise=true , address=145.79.227.24:4583 , piece_id=IB6T3J3CF3P3A54JEHHBK6KEPMUEI32B4QYSSA2G2BUMIVUC7DGQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.703957648 (41h1m30.175168377s) [noise=true , address=145.79.227.26:4654 , piece_id=23NNREG47WSZWJNPZWP275L62YBBSTB6RQWTINIRTTAYU66DRT7Q, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.704788621 (41h1m30.17599936s) [noise=true , address=145.79.227.43:4159 , piece_id=S5T27VFA6UNNONPZBZILNTQUUNSRLWPREKOJORMLKOXWNL3GMBSQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.705457525 (41h1m30.176668264s) [noise=true , address=145.79.227.32:4835 , piece_id=PJURUANT2XJY5PNYTSO5W2EDT3HYFAC2BFNCANKOAZ3I6P2GRC3Q, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.706486825 (41h1m30.177697554s) [noise=true , address=145.79.227.24:4582 , piece_id=MCQWT2LFD472TQDOAHLGQ5L2X7S654P6TL4J5MSP3CML5ZCURHMA, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.707626675 (41h1m30.178837414s) [noise=true , address=145.79.227.44:4193 , piece_id=327UUYIASLKFGUIQVY4P5UML4R2R4ODLUAYXR3GISKIT6CFDUGAQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.708631078 (41h1m30.179841817s) [noise=true , address=145.79.227.40:4050 , piece_id=QFSFC3WFER7VBVZNB3XLRIEFWGM4Q5VTAAJIWG4HAERKXSHZGZ4Q, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.709908630 (41h1m30.181119359s) [noise=true , address=145.79.227.22:4510 , piece_id=HCJIUH3VECRBR744KIQ65UKSMTEEKUPHJWXSDBM5SEMUVBJ5YX6A, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.710331386 (41h1m30.181542125s) [noise=true , address=145.79.227.41:4087 , piece_id=6GY2NC7F26VLU5NEN65OYU7YIHJYSTI5GNVRTNXM5HO5XQMZKEPA, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.710803685 (41h1m30.182014424s) [noise=true , address=45.91.116.4:4937 , piece_id=VFAPMIAB2G2BBOMNSM5EXONUL5BNORMR7KXIYSTQ7UFRVASVTQHQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:55.513663514 (41h1m31.984874243s) [upload=36107]: Next returned nil segment Dec 5 09:56:55.513682680 (41h1m31.98489341s) [upload=36107]: Waiting for error group managing segments... Dec 5 09:56:55.808504186 (41h1m32.279714915s) [noise=true , address=77.90.31.21:4878 , piece_id=RCFTETF7SGR6FTUBJWWBJDKOZIHZKBMSUBZYVWYKF2MP56PMIWMQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.866933347 (41h1m32.338144086s) [noise=true , address=145.79.227.10:4401 , piece_id=XAVSV3SGPDDLZWGKQLHA2CVYTGOPO7J3YVEO5C3MYIN5SJEUZ2OQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.869510454 (41h1m32.340721183s) [noise=true , address=145.79.227.28:4725 , piece_id=ZE642TJU4ZJKBOHQ6K6GPLPTN2YKC4BOP325LN7QXALZSDPG66HQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.875082795 (41h1m32.346293524s) [noise=true , address=145.79.227.43:4159 , piece_id=S5T27VFA6UNNONPZBZILNTQUUNSRLWPREKOJORMLKOXWNL3GMBSQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.884542648 (41h1m32.355753387s) [noise=true , address=145.79.227.24:4583 , piece_id=IB6T3J3CF3P3A54JEHHBK6KEPMUEI32B4QYSSA2G2BUMIVUC7DGQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.888876321 (41h1m32.36008707s) [noise=true , address=145.79.227.40:4050 , piece_id=QFSFC3WFER7VBVZNB3XLRIEFWGM4Q5VTAAJIWG4HAERKXSHZGZ4Q, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.889468719 (41h1m32.360679448s) [noise=true , address=145.79.227.26:4654 , piece_id=23NNREG47WSZWJNPZWP275L62YBBSTB6RQWTINIRTTAYU66DRT7Q, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.900234388 (41h1m32.371445117s) [noise=true , address=45.91.116.4:4937 , piece_id=VFAPMIAB2G2BBOMNSM5EXONUL5BNORMR7KXIYSTQ7UFRVASVTQHQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.923941432 (41h1m32.395152161s) [noise=true , address=145.79.227.42:4121 , piece_id=6D47MY4GTQ3YQOGHS2V2OTO2VFDL6XKKS2R4VA6VZZ557WYFY34A, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.929131404 (41h1m32.400342133s) [noise=true , address=145.79.227.22:4510 , piece_id=HCJIUH3VECRBR744KIQ65UKSMTEEKUPHJWXSDBM5SEMUVBJ5YX6A, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.180555093 (41h1m32.651765822s) [noise=true , address=145.79.227.32:4835 , piece_id=PJURUANT2XJY5PNYTSO5W2EDT3HYFAC2BFNCANKOAZ3I6P2GRC3Q, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.587529048 (41h1m33.058739787s) [noise=true , address=145.79.227.41:4087 , piece_id=6GY2NC7F26VLU5NEN65OYU7YIHJYSTI5GNVRTNXM5HO5XQMZKEPA, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.671173874 (41h1m33.142384603s) [noise=true , address=145.79.227.27:4691 , piece_id=Z53YBOXTW4L23P4M3EOW6ADUHDWHIJVRDT7IL4I5FUPPZOW2MRLQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.671209982 (41h1m33.142420711s) [seg_pos={0 0}, upload=36107]: Segment reached optimal threshold of 13 pieces. Dec 5 09:56:56.671423689 (41h1m33.142634428s) [noise=true , address=145.79.227.24:4582 , piece_id=MCQWT2LFD472TQDOAHLGQ5L2X7S654P6TL4J5MSP3CML5ZCURHMA, seg_pos={0 0}, upload=36107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa): protocol: write tcp 188.166.225.170:54136->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:54136->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54136->145.79.227.24:4582: use of closed network connection Dec 5 09:56:56.671432325 (41h1m33.142643064s) [noise=true , address=145.79.227.24:4582 , piece_id=MCQWT2LFD472TQDOAHLGQ5L2X7S654P6TL4J5MSP3CML5ZCURHMA, seg_pos={0 0}, upload=36107]: Long tail cancellation due to optimal threshold Dec 5 09:56:56.671496377 (41h1m33.142707106s) [noise=true , address=145.79.227.44:4193 , piece_id=327UUYIASLKFGUIQVY4P5UML4R2R4ODLUAYXR3GISKIT6CFDUGAQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: write tcp 188.166.225.170:39276->145.79.227.44:4193: use of closed network connection; write tcp 188.166.225.170:39276->145.79.227.44:4193: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39276->145.79.227.44:4193: use of closed network connection Dec 5 09:56:56.671501627 (41h1m33.142712366s) [noise=true , address=145.79.227.44:4193 , piece_id=327UUYIASLKFGUIQVY4P5UML4R2R4ODLUAYXR3GISKIT6CFDUGAQ, seg_pos={0 0}, upload=36107]: Long tail cancellation due to optimal threshold Dec 5 09:56:56.671516466 (41h1m33.142727195s) [noise=true , address=77.90.31.16:4735 , piece_id=V4YJIHVMXYTBK7IHU272RYCKUXI44B2MXRLETMWMO2WAI2XRU4SQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: write tcp 188.166.225.170:41866->77.90.31.16:4735: use of closed network connection; storage node overloaded, request limit: 2000 Dec 5 09:56:56.671525944 (41h1m33.142736673s) [noise=true , address=77.90.31.16:4735 , piece_id=V4YJIHVMXYTBK7IHU272RYCKUXI44B2MXRLETMWMO2WAI2XRU4SQ, seg_pos={0 0}, upload=36107]: Long tail cancellation due to optimal threshold Dec 5 09:56:56.671548016 (41h1m33.142758745s) [seg_pos={0 0}, upload=36107]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 5 09:56:56.671623500 (41h1m33.142834229s) [upload=36107]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 5 09:56:56.671631204 (41h1m33.142841933s) [upload=36107]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 5 09:56:57.112987571 (41h1m33.5841983s) [upload=36107]: Done uploading segments... Dec 5 09:56:57.113007228 (41h1m33.584217967s) [upload=36107]: Upload finished. err: <nil> Duration: 6.461692818s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 7.887854283s trace: https://jaeger.dev.storj.io/trace/55b376cc76cfe920 FAIL (2025-12-05 10:36:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 9 (41h1m29.913335078s) [upload=36107]: Uploading segments... Dec 5 09:56:53.442159225 (41h1m29.913369955s) [upload=36107]: Got next segment. Inline: false Dec 5 09:56:53.442188120 (41h1m29.913398849s) [upload=36107]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 5 09:56:53.442194372 (41h1m29.913405101s) [upload=36107]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 5 09:56:53.701976567 (41h1m30.173187296s) [seg_pos={0 0}, upload=36107]: Begin upload segment... Dec 5 09:56:53.702020460 (41h1m30.173231189s) [seg_pos={0 0}, upload=36107]: Done begin upload segment. Dec 5 09:56:53.702056649 (41h1m30.173267388s) [noise=true , address=77.90.31.21:4878 , piece_id=RCFTETF7SGR6FTUBJWWBJDKOZIHZKBMSUBZYVWYKF2MP56PMIWMQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702064965 (41h1m30.173275694s) [noise=true , address=145.79.227.28:4725 , piece_id=ZE642TJU4ZJKBOHQ6K6GPLPTN2YKC4BOP325LN7QXALZSDPG66HQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702111533 (41h1m30.173322272s) [noise=true , address=145.79.227.42:4121 , piece_id=6D47MY4GTQ3YQOGHS2V2OTO2VFDL6XKKS2R4VA6VZZ557WYFY34A, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702548986 (41h1m30.173759766s) [noise=true , address=145.79.227.27:4691 , piece_id=Z53YBOXTW4L23P4M3EOW6ADUHDWHIJVRDT7IL4I5FUPPZOW2MRLQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702931455 (41h1m30.174142194s) [noise=true , address=77.90.31.16:4735 , piece_id=V4YJIHVMXYTBK7IHU272RYCKUXI44B2MXRLETMWMO2WAI2XRU4SQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.703301079 (41h1m30.174511808s) [noise=true , address=145.79.227.10:4401 , piece_id=XAVSV3SGPDDLZWGKQLHA2CVYTGOPO7J3YVEO5C3MYIN5SJEUZ2OQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.703633000 (41h1m30.174843729s) [noise=true , address=145.79.227.24:4583 , piece_id=IB6T3J3CF3P3A54JEHHBK6KEPMUEI32B4QYSSA2G2BUMIVUC7DGQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.703957648 (41h1m30.175168377s) [noise=true , address=145.79.227.26:4654 , piece_id=23NNREG47WSZWJNPZWP275L62YBBSTB6RQWTINIRTTAYU66DRT7Q, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.704788621 (41h1m30.17599936s) [noise=true , address=145.79.227.43:4159 , piece_id=S5T27VFA6UNNONPZBZILNTQUUNSRLWPREKOJORMLKOXWNL3GMBSQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.705457525 (41h1m30.176668264s) [noise=true , address=145.79.227.32:4835 , piece_id=PJURUANT2XJY5PNYTSO5W2EDT3HYFAC2BFNCANKOAZ3I6P2GRC3Q, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.706486825 (41h1m30.177697554s) [noise=true , address=145.79.227.24:4582 , piece_id=MCQWT2LFD472TQDOAHLGQ5L2X7S654P6TL4J5MSP3CML5ZCURHMA, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.707626675 (41h1m30.178837414s) [noise=true , address=145.79.227.44:4193 , piece_id=327UUYIASLKFGUIQVY4P5UML4R2R4ODLUAYXR3GISKIT6CFDUGAQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.708631078 (41h1m30.179841817s) [noise=true , address=145.79.227.40:4050 , piece_id=QFSFC3WFER7VBVZNB3XLRIEFWGM4Q5VTAAJIWG4HAERKXSHZGZ4Q, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.709908630 (41h1m30.181119359s) [noise=true , address=145.79.227.22:4510 , piece_id=HCJIUH3VECRBR744KIQ65UKSMTEEKUPHJWXSDBM5SEMUVBJ5YX6A, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.710331386 (41h1m30.181542125s) [noise=true , address=145.79.227.41:4087 , piece_id=6GY2NC7F26VLU5NEN65OYU7YIHJYSTI5GNVRTNXM5HO5XQMZKEPA, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.710803685 (41h1m30.182014424s) [noise=true , address=45.91.116.4:4937 , piece_id=VFAPMIAB2G2BBOMNSM5EXONUL5BNORMR7KXIYSTQ7UFRVASVTQHQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:55.513663514 (41h1m31.984874243s) [upload=36107]: Next returned nil segment Dec 5 09:56:55.513682680 (41h1m31.98489341s) [upload=36107]: Waiting for error group managing segments... Dec 5 09:56:55.808504186 (41h1m32.279714915s) [noise=true , address=77.90.31.21:4878 , piece_id=RCFTETF7SGR6FTUBJWWBJDKOZIHZKBMSUBZYVWYKF2MP56PMIWMQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.866933347 (41h1m32.338144086s) [noise=true , address=145.79.227.10:4401 , piece_id=XAVSV3SGPDDLZWGKQLHA2CVYTGOPO7J3YVEO5C3MYIN5SJEUZ2OQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.869510454 (41h1m32.340721183s) [noise=true , address=145.79.227.28:4725 , piece_id=ZE642TJU4ZJKBOHQ6K6GPLPTN2YKC4BOP325LN7QXALZSDPG66HQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.875082795 (41h1m32.346293524s) [noise=true , address=145.79.227.43:4159 , piece_id=S5T27VFA6UNNONPZBZILNTQUUNSRLWPREKOJORMLKOXWNL3GMBSQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.884542648 (41h1m32.355753387s) [noise=true , address=145.79.227.24:4583 , piece_id=IB6T3J3CF3P3A54JEHHBK6KEPMUEI32B4QYSSA2G2BUMIVUC7DGQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.888876321 (41h1m32.36008707s) [noise=true , address=145.79.227.40:4050 , piece_id=QFSFC3WFER7VBVZNB3XLRIEFWGM4Q5VTAAJIWG4HAERKXSHZGZ4Q, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.889468719 (41h1m32.360679448s) [noise=true , address=145.79.227.26:4654 , piece_id=23NNREG47WSZWJNPZWP275L62YBBSTB6RQWTINIRTTAYU66DRT7Q, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.900234388 (41h1m32.371445117s) [noise=true , address=45.91.116.4:4937 , piece_id=VFAPMIAB2G2BBOMNSM5EXONUL5BNORMR7KXIYSTQ7UFRVASVTQHQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.923941432 (41h1m32.395152161s) [noise=true , address=145.79.227.42:4121 , piece_id=6D47MY4GTQ3YQOGHS2V2OTO2VFDL6XKKS2R4VA6VZZ557WYFY34A, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.929131404 (41h1m32.400342133s) [noise=true , address=145.79.227.22:4510 , piece_id=HCJIUH3VECRBR744KIQ65UKSMTEEKUPHJWXSDBM5SEMUVBJ5YX6A, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.180555093 (41h1m32.651765822s) [noise=true , address=145.79.227.32:4835 , piece_id=PJURUANT2XJY5PNYTSO5W2EDT3HYFAC2BFNCANKOAZ3I6P2GRC3Q, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.587529048 (41h1m33.058739787s) [noise=true , address=145.79.227.41:4087 , piece_id=6GY2NC7F26VLU5NEN65OYU7YIHJYSTI5GNVRTNXM5HO5XQMZKEPA, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.671173874 (41h1m33.142384603s) [noise=true , address=145.79.227.27:4691 , piece_id=Z53YBOXTW4L23P4M3EOW6ADUHDWHIJVRDT7IL4I5FUPPZOW2MRLQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.671209982 (41h1m33.142420711s) [seg_pos={0 0}, upload=36107]: Segment reached optimal threshold of 13 pieces. Dec 5 09:56:56.671423689 (41h1m33.142634428s) [noise=true , address=145.79.227.24:4582 , piece_id=MCQWT2LFD472TQDOAHLGQ5L2X7S654P6TL4J5MSP3CML5ZCURHMA, seg_pos={0 0}, upload=36107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa): protocol: write tcp 188.166.225.170:54136->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:54136->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54136->145.79.227.24:4582: use of closed network connection Dec 5 09:56:56.671432325 (41h1m33.142643064s) [noise=true , address=145.79.227.24:4582 , piece_id=MCQWT2LFD472TQDOAHLGQ5L2X7S654P6TL4J5MSP3CML5ZCURHMA, seg_pos={0 0}, upload=36107]: Long tail cancellation due to optimal threshold Dec 5 09:56:56.671496377 (41h1m33.142707106s) [noise=true , address=145.79.227.44:4193 , piece_id=327UUYIASLKFGUIQVY4P5UML4R2R4ODLUAYXR3GISKIT6CFDUGAQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: write tcp 188.166.225.170:39276->145.79.227.44:4193: use of closed network connection; write tcp 188.166.225.170:39276->145.79.227.44:4193: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39276->145.79.227.44:4193: use of closed network connection Dec 5 09:56:56.671501627 (41h1m33.142712366s) [noise=true , address=145.79.227.44:4193 , piece_id=327UUYIASLKFGUIQVY4P5UML4R2R4ODLUAYXR3GISKIT6CFDUGAQ, seg_pos={0 0}, upload=36107]: Long tail cancellation due to optimal threshold Dec 5 09:56:56.671516466 (41h1m33.142727195s) [noise=true , address=77.90.31.16:4735 , piece_id=V4YJIHVMXYTBK7IHU272RYCKUXI44B2MXRLETMWMO2WAI2XRU4SQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: write tcp 188.166.225.170:41866->77.90.31.16:4735: use of closed network connection; storage node overloaded, request limit: 2000 Dec 5 09:56:56.671525944 (41h1m33.142736673s) [noise=true , address=77.90.31.16:4735 , piece_id=V4YJIHVMXYTBK7IHU272RYCKUXI44B2MXRLETMWMO2WAI2XRU4SQ, seg_pos={0 0}, upload=36107]: Long tail cancellation due to optimal threshold Dec 5 09:56:56.671548016 (41h1m33.142758745s) [seg_pos={0 0}, upload=36107]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 5 09:56:56.671623500 (41h1m33.142834229s) [upload=36107]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 5 09:56:56.671631204 (41h1m33.142841933s) [upload=36107]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 5 09:56:57.112987571 (41h1m33.5841983s) [upload=36107]: Done uploading segments... Dec 5 09:56:57.113007228 (41h1m33.584217967s) [upload=36107]: Upload finished. err: <nil> Duration: 7.88788435s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 10.523325009s trace: https://jaeger.dev.storj.io/trace/55b376cc76cfe920 FAIL (2025-12-05 10:35:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 9 (41h1m29.913335078s) [upload=36107]: Uploading segments... Dec 5 09:56:53.442159225 (41h1m29.913369955s) [upload=36107]: Got next segment. Inline: false Dec 5 09:56:53.442188120 (41h1m29.913398849s) [upload=36107]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 5 09:56:53.442194372 (41h1m29.913405101s) [upload=36107]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 5 09:56:53.701976567 (41h1m30.173187296s) [seg_pos={0 0}, upload=36107]: Begin upload segment... Dec 5 09:56:53.702020460 (41h1m30.173231189s) [seg_pos={0 0}, upload=36107]: Done begin upload segment. Dec 5 09:56:53.702056649 (41h1m30.173267388s) [noise=true , address=77.90.31.21:4878 , piece_id=RCFTETF7SGR6FTUBJWWBJDKOZIHZKBMSUBZYVWYKF2MP56PMIWMQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702064965 (41h1m30.173275694s) [noise=true , address=145.79.227.28:4725 , piece_id=ZE642TJU4ZJKBOHQ6K6GPLPTN2YKC4BOP325LN7QXALZSDPG66HQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702111533 (41h1m30.173322272s) [noise=true , address=145.79.227.42:4121 , piece_id=6D47MY4GTQ3YQOGHS2V2OTO2VFDL6XKKS2R4VA6VZZ557WYFY34A, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702548986 (41h1m30.173759766s) [noise=true , address=145.79.227.27:4691 , piece_id=Z53YBOXTW4L23P4M3EOW6ADUHDWHIJVRDT7IL4I5FUPPZOW2MRLQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702931455 (41h1m30.174142194s) [noise=true , address=77.90.31.16:4735 , piece_id=V4YJIHVMXYTBK7IHU272RYCKUXI44B2MXRLETMWMO2WAI2XRU4SQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.703301079 (41h1m30.174511808s) [noise=true , address=145.79.227.10:4401 , piece_id=XAVSV3SGPDDLZWGKQLHA2CVYTGOPO7J3YVEO5C3MYIN5SJEUZ2OQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.703633000 (41h1m30.174843729s) [noise=true , address=145.79.227.24:4583 , piece_id=IB6T3J3CF3P3A54JEHHBK6KEPMUEI32B4QYSSA2G2BUMIVUC7DGQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.703957648 (41h1m30.175168377s) [noise=true , address=145.79.227.26:4654 , piece_id=23NNREG47WSZWJNPZWP275L62YBBSTB6RQWTINIRTTAYU66DRT7Q, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.704788621 (41h1m30.17599936s) [noise=true , address=145.79.227.43:4159 , piece_id=S5T27VFA6UNNONPZBZILNTQUUNSRLWPREKOJORMLKOXWNL3GMBSQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.705457525 (41h1m30.176668264s) [noise=true , address=145.79.227.32:4835 , piece_id=PJURUANT2XJY5PNYTSO5W2EDT3HYFAC2BFNCANKOAZ3I6P2GRC3Q, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.706486825 (41h1m30.177697554s) [noise=true , address=145.79.227.24:4582 , piece_id=MCQWT2LFD472TQDOAHLGQ5L2X7S654P6TL4J5MSP3CML5ZCURHMA, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.707626675 (41h1m30.178837414s) [noise=true , address=145.79.227.44:4193 , piece_id=327UUYIASLKFGUIQVY4P5UML4R2R4ODLUAYXR3GISKIT6CFDUGAQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.708631078 (41h1m30.179841817s) [noise=true , address=145.79.227.40:4050 , piece_id=QFSFC3WFER7VBVZNB3XLRIEFWGM4Q5VTAAJIWG4HAERKXSHZGZ4Q, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.709908630 (41h1m30.181119359s) [noise=true , address=145.79.227.22:4510 , piece_id=HCJIUH3VECRBR744KIQ65UKSMTEEKUPHJWXSDBM5SEMUVBJ5YX6A, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.710331386 (41h1m30.181542125s) [noise=true , address=145.79.227.41:4087 , piece_id=6GY2NC7F26VLU5NEN65OYU7YIHJYSTI5GNVRTNXM5HO5XQMZKEPA, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.710803685 (41h1m30.182014424s) [noise=true , address=45.91.116.4:4937 , piece_id=VFAPMIAB2G2BBOMNSM5EXONUL5BNORMR7KXIYSTQ7UFRVASVTQHQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:55.513663514 (41h1m31.984874243s) [upload=36107]: Next returned nil segment Dec 5 09:56:55.513682680 (41h1m31.98489341s) [upload=36107]: Waiting for error group managing segments... Dec 5 09:56:55.808504186 (41h1m32.279714915s) [noise=true , address=77.90.31.21:4878 , piece_id=RCFTETF7SGR6FTUBJWWBJDKOZIHZKBMSUBZYVWYKF2MP56PMIWMQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.866933347 (41h1m32.338144086s) [noise=true , address=145.79.227.10:4401 , piece_id=XAVSV3SGPDDLZWGKQLHA2CVYTGOPO7J3YVEO5C3MYIN5SJEUZ2OQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.869510454 (41h1m32.340721183s) [noise=true , address=145.79.227.28:4725 , piece_id=ZE642TJU4ZJKBOHQ6K6GPLPTN2YKC4BOP325LN7QXALZSDPG66HQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.875082795 (41h1m32.346293524s) [noise=true , address=145.79.227.43:4159 , piece_id=S5T27VFA6UNNONPZBZILNTQUUNSRLWPREKOJORMLKOXWNL3GMBSQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.884542648 (41h1m32.355753387s) [noise=true , address=145.79.227.24:4583 , piece_id=IB6T3J3CF3P3A54JEHHBK6KEPMUEI32B4QYSSA2G2BUMIVUC7DGQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.888876321 (41h1m32.36008707s) [noise=true , address=145.79.227.40:4050 , piece_id=QFSFC3WFER7VBVZNB3XLRIEFWGM4Q5VTAAJIWG4HAERKXSHZGZ4Q, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.889468719 (41h1m32.360679448s) [noise=true , address=145.79.227.26:4654 , piece_id=23NNREG47WSZWJNPZWP275L62YBBSTB6RQWTINIRTTAYU66DRT7Q, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.900234388 (41h1m32.371445117s) [noise=true , address=45.91.116.4:4937 , piece_id=VFAPMIAB2G2BBOMNSM5EXONUL5BNORMR7KXIYSTQ7UFRVASVTQHQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.923941432 (41h1m32.395152161s) [noise=true , address=145.79.227.42:4121 , piece_id=6D47MY4GTQ3YQOGHS2V2OTO2VFDL6XKKS2R4VA6VZZ557WYFY34A, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.929131404 (41h1m32.400342133s) [noise=true , address=145.79.227.22:4510 , piece_id=HCJIUH3VECRBR744KIQ65UKSMTEEKUPHJWXSDBM5SEMUVBJ5YX6A, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.180555093 (41h1m32.651765822s) [noise=true , address=145.79.227.32:4835 , piece_id=PJURUANT2XJY5PNYTSO5W2EDT3HYFAC2BFNCANKOAZ3I6P2GRC3Q, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.587529048 (41h1m33.058739787s) [noise=true , address=145.79.227.41:4087 , piece_id=6GY2NC7F26VLU5NEN65OYU7YIHJYSTI5GNVRTNXM5HO5XQMZKEPA, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.671173874 (41h1m33.142384603s) [noise=true , address=145.79.227.27:4691 , piece_id=Z53YBOXTW4L23P4M3EOW6ADUHDWHIJVRDT7IL4I5FUPPZOW2MRLQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.671209982 (41h1m33.142420711s) [seg_pos={0 0}, upload=36107]: Segment reached optimal threshold of 13 pieces. Dec 5 09:56:56.671423689 (41h1m33.142634428s) [noise=true , address=145.79.227.24:4582 , piece_id=MCQWT2LFD472TQDOAHLGQ5L2X7S654P6TL4J5MSP3CML5ZCURHMA, seg_pos={0 0}, upload=36107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa): protocol: write tcp 188.166.225.170:54136->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:54136->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54136->145.79.227.24:4582: use of closed network connection Dec 5 09:56:56.671432325 (41h1m33.142643064s) [noise=true , address=145.79.227.24:4582 , piece_id=MCQWT2LFD472TQDOAHLGQ5L2X7S654P6TL4J5MSP3CML5ZCURHMA, seg_pos={0 0}, upload=36107]: Long tail cancellation due to optimal threshold Dec 5 09:56:56.671496377 (41h1m33.142707106s) [noise=true , address=145.79.227.44:4193 , piece_id=327UUYIASLKFGUIQVY4P5UML4R2R4ODLUAYXR3GISKIT6CFDUGAQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: write tcp 188.166.225.170:39276->145.79.227.44:4193: use of closed network connection; write tcp 188.166.225.170:39276->145.79.227.44:4193: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39276->145.79.227.44:4193: use of closed network connection Dec 5 09:56:56.671501627 (41h1m33.142712366s) [noise=true , address=145.79.227.44:4193 , piece_id=327UUYIASLKFGUIQVY4P5UML4R2R4ODLUAYXR3GISKIT6CFDUGAQ, seg_pos={0 0}, upload=36107]: Long tail cancellation due to optimal threshold Dec 5 09:56:56.671516466 (41h1m33.142727195s) [noise=true , address=77.90.31.16:4735 , piece_id=V4YJIHVMXYTBK7IHU272RYCKUXI44B2MXRLETMWMO2WAI2XRU4SQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: write tcp 188.166.225.170:41866->77.90.31.16:4735: use of closed network connection; storage node overloaded, request limit: 2000 Dec 5 09:56:56.671525944 (41h1m33.142736673s) [noise=true , address=77.90.31.16:4735 , piece_id=V4YJIHVMXYTBK7IHU272RYCKUXI44B2MXRLETMWMO2WAI2XRU4SQ, seg_pos={0 0}, upload=36107]: Long tail cancellation due to optimal threshold Dec 5 09:56:56.671548016 (41h1m33.142758745s) [seg_pos={0 0}, upload=36107]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 5 09:56:56.671623500 (41h1m33.142834229s) [upload=36107]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 5 09:56:56.671631204 (41h1m33.142841933s) [upload=36107]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 5 09:56:57.112987571 (41h1m33.5841983s) [upload=36107]: Done uploading segments... Dec 5 09:56:57.113007228 (41h1m33.584217967s) [upload=36107]: Upload finished. err: <nil> Duration: 10.523370106s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 8.296945585s trace: https://jaeger.dev.storj.io/trace/55b376cc76cfe920 FAIL (2025-12-05 10:34:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 9 (41h1m29.913335078s) [upload=36107]: Uploading segments... Dec 5 09:56:53.442159225 (41h1m29.913369955s) [upload=36107]: Got next segment. Inline: false Dec 5 09:56:53.442188120 (41h1m29.913398849s) [upload=36107]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 5 09:56:53.442194372 (41h1m29.913405101s) [upload=36107]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 5 09:56:53.701976567 (41h1m30.173187296s) [seg_pos={0 0}, upload=36107]: Begin upload segment... Dec 5 09:56:53.702020460 (41h1m30.173231189s) [seg_pos={0 0}, upload=36107]: Done begin upload segment. Dec 5 09:56:53.702056649 (41h1m30.173267388s) [noise=true , address=77.90.31.21:4878 , piece_id=RCFTETF7SGR6FTUBJWWBJDKOZIHZKBMSUBZYVWYKF2MP56PMIWMQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702064965 (41h1m30.173275694s) [noise=true , address=145.79.227.28:4725 , piece_id=ZE642TJU4ZJKBOHQ6K6GPLPTN2YKC4BOP325LN7QXALZSDPG66HQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702111533 (41h1m30.173322272s) [noise=true , address=145.79.227.42:4121 , piece_id=6D47MY4GTQ3YQOGHS2V2OTO2VFDL6XKKS2R4VA6VZZ557WYFY34A, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702548986 (41h1m30.173759766s) [noise=true , address=145.79.227.27:4691 , piece_id=Z53YBOXTW4L23P4M3EOW6ADUHDWHIJVRDT7IL4I5FUPPZOW2MRLQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702931455 (41h1m30.174142194s) [noise=true , address=77.90.31.16:4735 , piece_id=V4YJIHVMXYTBK7IHU272RYCKUXI44B2MXRLETMWMO2WAI2XRU4SQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.703301079 (41h1m30.174511808s) [noise=true , address=145.79.227.10:4401 , piece_id=XAVSV3SGPDDLZWGKQLHA2CVYTGOPO7J3YVEO5C3MYIN5SJEUZ2OQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.703633000 (41h1m30.174843729s) [noise=true , address=145.79.227.24:4583 , piece_id=IB6T3J3CF3P3A54JEHHBK6KEPMUEI32B4QYSSA2G2BUMIVUC7DGQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.703957648 (41h1m30.175168377s) [noise=true , address=145.79.227.26:4654 , piece_id=23NNREG47WSZWJNPZWP275L62YBBSTB6RQWTINIRTTAYU66DRT7Q, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.704788621 (41h1m30.17599936s) [noise=true , address=145.79.227.43:4159 , piece_id=S5T27VFA6UNNONPZBZILNTQUUNSRLWPREKOJORMLKOXWNL3GMBSQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.705457525 (41h1m30.176668264s) [noise=true , address=145.79.227.32:4835 , piece_id=PJURUANT2XJY5PNYTSO5W2EDT3HYFAC2BFNCANKOAZ3I6P2GRC3Q, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.706486825 (41h1m30.177697554s) [noise=true , address=145.79.227.24:4582 , piece_id=MCQWT2LFD472TQDOAHLGQ5L2X7S654P6TL4J5MSP3CML5ZCURHMA, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.707626675 (41h1m30.178837414s) [noise=true , address=145.79.227.44:4193 , piece_id=327UUYIASLKFGUIQVY4P5UML4R2R4ODLUAYXR3GISKIT6CFDUGAQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.708631078 (41h1m30.179841817s) [noise=true , address=145.79.227.40:4050 , piece_id=QFSFC3WFER7VBVZNB3XLRIEFWGM4Q5VTAAJIWG4HAERKXSHZGZ4Q, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.709908630 (41h1m30.181119359s) [noise=true , address=145.79.227.22:4510 , piece_id=HCJIUH3VECRBR744KIQ65UKSMTEEKUPHJWXSDBM5SEMUVBJ5YX6A, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.710331386 (41h1m30.181542125s) [noise=true , address=145.79.227.41:4087 , piece_id=6GY2NC7F26VLU5NEN65OYU7YIHJYSTI5GNVRTNXM5HO5XQMZKEPA, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.710803685 (41h1m30.182014424s) [noise=true , address=45.91.116.4:4937 , piece_id=VFAPMIAB2G2BBOMNSM5EXONUL5BNORMR7KXIYSTQ7UFRVASVTQHQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:55.513663514 (41h1m31.984874243s) [upload=36107]: Next returned nil segment Dec 5 09:56:55.513682680 (41h1m31.98489341s) [upload=36107]: Waiting for error group managing segments... Dec 5 09:56:55.808504186 (41h1m32.279714915s) [noise=true , address=77.90.31.21:4878 , piece_id=RCFTETF7SGR6FTUBJWWBJDKOZIHZKBMSUBZYVWYKF2MP56PMIWMQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.866933347 (41h1m32.338144086s) [noise=true , address=145.79.227.10:4401 , piece_id=XAVSV3SGPDDLZWGKQLHA2CVYTGOPO7J3YVEO5C3MYIN5SJEUZ2OQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.869510454 (41h1m32.340721183s) [noise=true , address=145.79.227.28:4725 , piece_id=ZE642TJU4ZJKBOHQ6K6GPLPTN2YKC4BOP325LN7QXALZSDPG66HQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.875082795 (41h1m32.346293524s) [noise=true , address=145.79.227.43:4159 , piece_id=S5T27VFA6UNNONPZBZILNTQUUNSRLWPREKOJORMLKOXWNL3GMBSQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.884542648 (41h1m32.355753387s) [noise=true , address=145.79.227.24:4583 , piece_id=IB6T3J3CF3P3A54JEHHBK6KEPMUEI32B4QYSSA2G2BUMIVUC7DGQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.888876321 (41h1m32.36008707s) [noise=true , address=145.79.227.40:4050 , piece_id=QFSFC3WFER7VBVZNB3XLRIEFWGM4Q5VTAAJIWG4HAERKXSHZGZ4Q, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.889468719 (41h1m32.360679448s) [noise=true , address=145.79.227.26:4654 , piece_id=23NNREG47WSZWJNPZWP275L62YBBSTB6RQWTINIRTTAYU66DRT7Q, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.900234388 (41h1m32.371445117s) [noise=true , address=45.91.116.4:4937 , piece_id=VFAPMIAB2G2BBOMNSM5EXONUL5BNORMR7KXIYSTQ7UFRVASVTQHQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.923941432 (41h1m32.395152161s) [noise=true , address=145.79.227.42:4121 , piece_id=6D47MY4GTQ3YQOGHS2V2OTO2VFDL6XKKS2R4VA6VZZ557WYFY34A, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.929131404 (41h1m32.400342133s) [noise=true , address=145.79.227.22:4510 , piece_id=HCJIUH3VECRBR744KIQ65UKSMTEEKUPHJWXSDBM5SEMUVBJ5YX6A, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.180555093 (41h1m32.651765822s) [noise=true , address=145.79.227.32:4835 , piece_id=PJURUANT2XJY5PNYTSO5W2EDT3HYFAC2BFNCANKOAZ3I6P2GRC3Q, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.587529048 (41h1m33.058739787s) [noise=true , address=145.79.227.41:4087 , piece_id=6GY2NC7F26VLU5NEN65OYU7YIHJYSTI5GNVRTNXM5HO5XQMZKEPA, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.671173874 (41h1m33.142384603s) [noise=true , address=145.79.227.27:4691 , piece_id=Z53YBOXTW4L23P4M3EOW6ADUHDWHIJVRDT7IL4I5FUPPZOW2MRLQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.671209982 (41h1m33.142420711s) [seg_pos={0 0}, upload=36107]: Segment reached optimal threshold of 13 pieces. Dec 5 09:56:56.671423689 (41h1m33.142634428s) [noise=true , address=145.79.227.24:4582 , piece_id=MCQWT2LFD472TQDOAHLGQ5L2X7S654P6TL4J5MSP3CML5ZCURHMA, seg_pos={0 0}, upload=36107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa): protocol: write tcp 188.166.225.170:54136->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:54136->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54136->145.79.227.24:4582: use of closed network connection Dec 5 09:56:56.671432325 (41h1m33.142643064s) [noise=true , address=145.79.227.24:4582 , piece_id=MCQWT2LFD472TQDOAHLGQ5L2X7S654P6TL4J5MSP3CML5ZCURHMA, seg_pos={0 0}, upload=36107]: Long tail cancellation due to optimal threshold Dec 5 09:56:56.671496377 (41h1m33.142707106s) [noise=true , address=145.79.227.44:4193 , piece_id=327UUYIASLKFGUIQVY4P5UML4R2R4ODLUAYXR3GISKIT6CFDUGAQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: write tcp 188.166.225.170:39276->145.79.227.44:4193: use of closed network connection; write tcp 188.166.225.170:39276->145.79.227.44:4193: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39276->145.79.227.44:4193: use of closed network connection Dec 5 09:56:56.671501627 (41h1m33.142712366s) [noise=true , address=145.79.227.44:4193 , piece_id=327UUYIASLKFGUIQVY4P5UML4R2R4ODLUAYXR3GISKIT6CFDUGAQ, seg_pos={0 0}, upload=36107]: Long tail cancellation due to optimal threshold Dec 5 09:56:56.671516466 (41h1m33.142727195s) [noise=true , address=77.90.31.16:4735 , piece_id=V4YJIHVMXYTBK7IHU272RYCKUXI44B2MXRLETMWMO2WAI2XRU4SQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: write tcp 188.166.225.170:41866->77.90.31.16:4735: use of closed network connection; storage node overloaded, request limit: 2000 Dec 5 09:56:56.671525944 (41h1m33.142736673s) [noise=true , address=77.90.31.16:4735 , piece_id=V4YJIHVMXYTBK7IHU272RYCKUXI44B2MXRLETMWMO2WAI2XRU4SQ, seg_pos={0 0}, upload=36107]: Long tail cancellation due to optimal threshold Dec 5 09:56:56.671548016 (41h1m33.142758745s) [seg_pos={0 0}, upload=36107]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 5 09:56:56.671623500 (41h1m33.142834229s) [upload=36107]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 5 09:56:56.671631204 (41h1m33.142841933s) [upload=36107]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 5 09:56:57.112987571 (41h1m33.5841983s) [upload=36107]: Done uploading segments... Dec 5 09:56:57.113007228 (41h1m33.584217967s) [upload=36107]: Upload finished. err: <nil> Duration: 8.296971855s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 9.062179976s trace: https://jaeger.dev.storj.io/trace/55b376cc76cfe920 FAIL (2025-12-05 10:33:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 9 (41h1m29.913335078s) [upload=36107]: Uploading segments... Dec 5 09:56:53.442159225 (41h1m29.913369955s) [upload=36107]: Got next segment. Inline: false Dec 5 09:56:53.442188120 (41h1m29.913398849s) [upload=36107]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 5 09:56:53.442194372 (41h1m29.913405101s) [upload=36107]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 5 09:56:53.701976567 (41h1m30.173187296s) [seg_pos={0 0}, upload=36107]: Begin upload segment... Dec 5 09:56:53.702020460 (41h1m30.173231189s) [seg_pos={0 0}, upload=36107]: Done begin upload segment. Dec 5 09:56:53.702056649 (41h1m30.173267388s) [noise=true , address=77.90.31.21:4878 , piece_id=RCFTETF7SGR6FTUBJWWBJDKOZIHZKBMSUBZYVWYKF2MP56PMIWMQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702064965 (41h1m30.173275694s) [noise=true , address=145.79.227.28:4725 , piece_id=ZE642TJU4ZJKBOHQ6K6GPLPTN2YKC4BOP325LN7QXALZSDPG66HQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702111533 (41h1m30.173322272s) [noise=true , address=145.79.227.42:4121 , piece_id=6D47MY4GTQ3YQOGHS2V2OTO2VFDL6XKKS2R4VA6VZZ557WYFY34A, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702548986 (41h1m30.173759766s) [noise=true , address=145.79.227.27:4691 , piece_id=Z53YBOXTW4L23P4M3EOW6ADUHDWHIJVRDT7IL4I5FUPPZOW2MRLQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702931455 (41h1m30.174142194s) [noise=true , address=77.90.31.16:4735 , piece_id=V4YJIHVMXYTBK7IHU272RYCKUXI44B2MXRLETMWMO2WAI2XRU4SQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.703301079 (41h1m30.174511808s) [noise=true , address=145.79.227.10:4401 , piece_id=XAVSV3SGPDDLZWGKQLHA2CVYTGOPO7J3YVEO5C3MYIN5SJEUZ2OQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.703633000 (41h1m30.174843729s) [noise=true , address=145.79.227.24:4583 , piece_id=IB6T3J3CF3P3A54JEHHBK6KEPMUEI32B4QYSSA2G2BUMIVUC7DGQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.703957648 (41h1m30.175168377s) [noise=true , address=145.79.227.26:4654 , piece_id=23NNREG47WSZWJNPZWP275L62YBBSTB6RQWTINIRTTAYU66DRT7Q, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.704788621 (41h1m30.17599936s) [noise=true , address=145.79.227.43:4159 , piece_id=S5T27VFA6UNNONPZBZILNTQUUNSRLWPREKOJORMLKOXWNL3GMBSQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.705457525 (41h1m30.176668264s) [noise=true , address=145.79.227.32:4835 , piece_id=PJURUANT2XJY5PNYTSO5W2EDT3HYFAC2BFNCANKOAZ3I6P2GRC3Q, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.706486825 (41h1m30.177697554s) [noise=true , address=145.79.227.24:4582 , piece_id=MCQWT2LFD472TQDOAHLGQ5L2X7S654P6TL4J5MSP3CML5ZCURHMA, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.707626675 (41h1m30.178837414s) [noise=true , address=145.79.227.44:4193 , piece_id=327UUYIASLKFGUIQVY4P5UML4R2R4ODLUAYXR3GISKIT6CFDUGAQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.708631078 (41h1m30.179841817s) [noise=true , address=145.79.227.40:4050 , piece_id=QFSFC3WFER7VBVZNB3XLRIEFWGM4Q5VTAAJIWG4HAERKXSHZGZ4Q, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.709908630 (41h1m30.181119359s) [noise=true , address=145.79.227.22:4510 , piece_id=HCJIUH3VECRBR744KIQ65UKSMTEEKUPHJWXSDBM5SEMUVBJ5YX6A, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.710331386 (41h1m30.181542125s) [noise=true , address=145.79.227.41:4087 , piece_id=6GY2NC7F26VLU5NEN65OYU7YIHJYSTI5GNVRTNXM5HO5XQMZKEPA, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.710803685 (41h1m30.182014424s) [noise=true , address=45.91.116.4:4937 , piece_id=VFAPMIAB2G2BBOMNSM5EXONUL5BNORMR7KXIYSTQ7UFRVASVTQHQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:55.513663514 (41h1m31.984874243s) [upload=36107]: Next returned nil segment Dec 5 09:56:55.513682680 (41h1m31.98489341s) [upload=36107]: Waiting for error group managing segments... Dec 5 09:56:55.808504186 (41h1m32.279714915s) [noise=true , address=77.90.31.21:4878 , piece_id=RCFTETF7SGR6FTUBJWWBJDKOZIHZKBMSUBZYVWYKF2MP56PMIWMQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.866933347 (41h1m32.338144086s) [noise=true , address=145.79.227.10:4401 , piece_id=XAVSV3SGPDDLZWGKQLHA2CVYTGOPO7J3YVEO5C3MYIN5SJEUZ2OQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.869510454 (41h1m32.340721183s) [noise=true , address=145.79.227.28:4725 , piece_id=ZE642TJU4ZJKBOHQ6K6GPLPTN2YKC4BOP325LN7QXALZSDPG66HQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.875082795 (41h1m32.346293524s) [noise=true , address=145.79.227.43:4159 , piece_id=S5T27VFA6UNNONPZBZILNTQUUNSRLWPREKOJORMLKOXWNL3GMBSQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.884542648 (41h1m32.355753387s) [noise=true , address=145.79.227.24:4583 , piece_id=IB6T3J3CF3P3A54JEHHBK6KEPMUEI32B4QYSSA2G2BUMIVUC7DGQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.888876321 (41h1m32.36008707s) [noise=true , address=145.79.227.40:4050 , piece_id=QFSFC3WFER7VBVZNB3XLRIEFWGM4Q5VTAAJIWG4HAERKXSHZGZ4Q, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.889468719 (41h1m32.360679448s) [noise=true , address=145.79.227.26:4654 , piece_id=23NNREG47WSZWJNPZWP275L62YBBSTB6RQWTINIRTTAYU66DRT7Q, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.900234388 (41h1m32.371445117s) [noise=true , address=45.91.116.4:4937 , piece_id=VFAPMIAB2G2BBOMNSM5EXONUL5BNORMR7KXIYSTQ7UFRVASVTQHQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.923941432 (41h1m32.395152161s) [noise=true , address=145.79.227.42:4121 , piece_id=6D47MY4GTQ3YQOGHS2V2OTO2VFDL6XKKS2R4VA6VZZ557WYFY34A, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.929131404 (41h1m32.400342133s) [noise=true , address=145.79.227.22:4510 , piece_id=HCJIUH3VECRBR744KIQ65UKSMTEEKUPHJWXSDBM5SEMUVBJ5YX6A, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.180555093 (41h1m32.651765822s) [noise=true , address=145.79.227.32:4835 , piece_id=PJURUANT2XJY5PNYTSO5W2EDT3HYFAC2BFNCANKOAZ3I6P2GRC3Q, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.587529048 (41h1m33.058739787s) [noise=true , address=145.79.227.41:4087 , piece_id=6GY2NC7F26VLU5NEN65OYU7YIHJYSTI5GNVRTNXM5HO5XQMZKEPA, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.671173874 (41h1m33.142384603s) [noise=true , address=145.79.227.27:4691 , piece_id=Z53YBOXTW4L23P4M3EOW6ADUHDWHIJVRDT7IL4I5FUPPZOW2MRLQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.671209982 (41h1m33.142420711s) [seg_pos={0 0}, upload=36107]: Segment reached optimal threshold of 13 pieces. Dec 5 09:56:56.671423689 (41h1m33.142634428s) [noise=true , address=145.79.227.24:4582 , piece_id=MCQWT2LFD472TQDOAHLGQ5L2X7S654P6TL4J5MSP3CML5ZCURHMA, seg_pos={0 0}, upload=36107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa): protocol: write tcp 188.166.225.170:54136->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:54136->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54136->145.79.227.24:4582: use of closed network connection Dec 5 09:56:56.671432325 (41h1m33.142643064s) [noise=true , address=145.79.227.24:4582 , piece_id=MCQWT2LFD472TQDOAHLGQ5L2X7S654P6TL4J5MSP3CML5ZCURHMA, seg_pos={0 0}, upload=36107]: Long tail cancellation due to optimal threshold Dec 5 09:56:56.671496377 (41h1m33.142707106s) [noise=true , address=145.79.227.44:4193 , piece_id=327UUYIASLKFGUIQVY4P5UML4R2R4ODLUAYXR3GISKIT6CFDUGAQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: write tcp 188.166.225.170:39276->145.79.227.44:4193: use of closed network connection; write tcp 188.166.225.170:39276->145.79.227.44:4193: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39276->145.79.227.44:4193: use of closed network connection Dec 5 09:56:56.671501627 (41h1m33.142712366s) [noise=true , address=145.79.227.44:4193 , piece_id=327UUYIASLKFGUIQVY4P5UML4R2R4ODLUAYXR3GISKIT6CFDUGAQ, seg_pos={0 0}, upload=36107]: Long tail cancellation due to optimal threshold Dec 5 09:56:56.671516466 (41h1m33.142727195s) [noise=true , address=77.90.31.16:4735 , piece_id=V4YJIHVMXYTBK7IHU272RYCKUXI44B2MXRLETMWMO2WAI2XRU4SQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: write tcp 188.166.225.170:41866->77.90.31.16:4735: use of closed network connection; storage node overloaded, request limit: 2000 Dec 5 09:56:56.671525944 (41h1m33.142736673s) [noise=true , address=77.90.31.16:4735 , piece_id=V4YJIHVMXYTBK7IHU272RYCKUXI44B2MXRLETMWMO2WAI2XRU4SQ, seg_pos={0 0}, upload=36107]: Long tail cancellation due to optimal threshold Dec 5 09:56:56.671548016 (41h1m33.142758745s) [seg_pos={0 0}, upload=36107]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 5 09:56:56.671623500 (41h1m33.142834229s) [upload=36107]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 5 09:56:56.671631204 (41h1m33.142841933s) [upload=36107]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 5 09:56:57.112987571 (41h1m33.5841983s) [upload=36107]: Done uploading segments... Dec 5 09:56:57.113007228 (41h1m33.584217967s) [upload=36107]: Upload finished. err: <nil> Duration: 9.062294985s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 8.289933623s trace: https://jaeger.dev.storj.io/trace/55b376cc76cfe920 FAIL (2025-12-05 10:33:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 9 (41h1m29.913335078s) [upload=36107]: Uploading segments... Dec 5 09:56:53.442159225 (41h1m29.913369955s) [upload=36107]: Got next segment. Inline: false Dec 5 09:56:53.442188120 (41h1m29.913398849s) [upload=36107]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 5 09:56:53.442194372 (41h1m29.913405101s) [upload=36107]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 5 09:56:53.701976567 (41h1m30.173187296s) [seg_pos={0 0}, upload=36107]: Begin upload segment... Dec 5 09:56:53.702020460 (41h1m30.173231189s) [seg_pos={0 0}, upload=36107]: Done begin upload segment. Dec 5 09:56:53.702056649 (41h1m30.173267388s) [noise=true , address=77.90.31.21:4878 , piece_id=RCFTETF7SGR6FTUBJWWBJDKOZIHZKBMSUBZYVWYKF2MP56PMIWMQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702064965 (41h1m30.173275694s) [noise=true , address=145.79.227.28:4725 , piece_id=ZE642TJU4ZJKBOHQ6K6GPLPTN2YKC4BOP325LN7QXALZSDPG66HQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702111533 (41h1m30.173322272s) [noise=true , address=145.79.227.42:4121 , piece_id=6D47MY4GTQ3YQOGHS2V2OTO2VFDL6XKKS2R4VA6VZZ557WYFY34A, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702548986 (41h1m30.173759766s) [noise=true , address=145.79.227.27:4691 , piece_id=Z53YBOXTW4L23P4M3EOW6ADUHDWHIJVRDT7IL4I5FUPPZOW2MRLQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702931455 (41h1m30.174142194s) [noise=true , address=77.90.31.16:4735 , piece_id=V4YJIHVMXYTBK7IHU272RYCKUXI44B2MXRLETMWMO2WAI2XRU4SQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.703301079 (41h1m30.174511808s) [noise=true , address=145.79.227.10:4401 , piece_id=XAVSV3SGPDDLZWGKQLHA2CVYTGOPO7J3YVEO5C3MYIN5SJEUZ2OQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.703633000 (41h1m30.174843729s) [noise=true , address=145.79.227.24:4583 , piece_id=IB6T3J3CF3P3A54JEHHBK6KEPMUEI32B4QYSSA2G2BUMIVUC7DGQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.703957648 (41h1m30.175168377s) [noise=true , address=145.79.227.26:4654 , piece_id=23NNREG47WSZWJNPZWP275L62YBBSTB6RQWTINIRTTAYU66DRT7Q, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.704788621 (41h1m30.17599936s) [noise=true , address=145.79.227.43:4159 , piece_id=S5T27VFA6UNNONPZBZILNTQUUNSRLWPREKOJORMLKOXWNL3GMBSQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.705457525 (41h1m30.176668264s) [noise=true , address=145.79.227.32:4835 , piece_id=PJURUANT2XJY5PNYTSO5W2EDT3HYFAC2BFNCANKOAZ3I6P2GRC3Q, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.706486825 (41h1m30.177697554s) [noise=true , address=145.79.227.24:4582 , piece_id=MCQWT2LFD472TQDOAHLGQ5L2X7S654P6TL4J5MSP3CML5ZCURHMA, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.707626675 (41h1m30.178837414s) [noise=true , address=145.79.227.44:4193 , piece_id=327UUYIASLKFGUIQVY4P5UML4R2R4ODLUAYXR3GISKIT6CFDUGAQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.708631078 (41h1m30.179841817s) [noise=true , address=145.79.227.40:4050 , piece_id=QFSFC3WFER7VBVZNB3XLRIEFWGM4Q5VTAAJIWG4HAERKXSHZGZ4Q, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.709908630 (41h1m30.181119359s) [noise=true , address=145.79.227.22:4510 , piece_id=HCJIUH3VECRBR744KIQ65UKSMTEEKUPHJWXSDBM5SEMUVBJ5YX6A, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.710331386 (41h1m30.181542125s) [noise=true , address=145.79.227.41:4087 , piece_id=6GY2NC7F26VLU5NEN65OYU7YIHJYSTI5GNVRTNXM5HO5XQMZKEPA, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.710803685 (41h1m30.182014424s) [noise=true , address=45.91.116.4:4937 , piece_id=VFAPMIAB2G2BBOMNSM5EXONUL5BNORMR7KXIYSTQ7UFRVASVTQHQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:55.513663514 (41h1m31.984874243s) [upload=36107]: Next returned nil segment Dec 5 09:56:55.513682680 (41h1m31.98489341s) [upload=36107]: Waiting for error group managing segments... Dec 5 09:56:55.808504186 (41h1m32.279714915s) [noise=true , address=77.90.31.21:4878 , piece_id=RCFTETF7SGR6FTUBJWWBJDKOZIHZKBMSUBZYVWYKF2MP56PMIWMQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.866933347 (41h1m32.338144086s) [noise=true , address=145.79.227.10:4401 , piece_id=XAVSV3SGPDDLZWGKQLHA2CVYTGOPO7J3YVEO5C3MYIN5SJEUZ2OQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.869510454 (41h1m32.340721183s) [noise=true , address=145.79.227.28:4725 , piece_id=ZE642TJU4ZJKBOHQ6K6GPLPTN2YKC4BOP325LN7QXALZSDPG66HQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.875082795 (41h1m32.346293524s) [noise=true , address=145.79.227.43:4159 , piece_id=S5T27VFA6UNNONPZBZILNTQUUNSRLWPREKOJORMLKOXWNL3GMBSQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.884542648 (41h1m32.355753387s) [noise=true , address=145.79.227.24:4583 , piece_id=IB6T3J3CF3P3A54JEHHBK6KEPMUEI32B4QYSSA2G2BUMIVUC7DGQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.888876321 (41h1m32.36008707s) [noise=true , address=145.79.227.40:4050 , piece_id=QFSFC3WFER7VBVZNB3XLRIEFWGM4Q5VTAAJIWG4HAERKXSHZGZ4Q, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.889468719 (41h1m32.360679448s) [noise=true , address=145.79.227.26:4654 , piece_id=23NNREG47WSZWJNPZWP275L62YBBSTB6RQWTINIRTTAYU66DRT7Q, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.900234388 (41h1m32.371445117s) [noise=true , address=45.91.116.4:4937 , piece_id=VFAPMIAB2G2BBOMNSM5EXONUL5BNORMR7KXIYSTQ7UFRVASVTQHQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.923941432 (41h1m32.395152161s) [noise=true , address=145.79.227.42:4121 , piece_id=6D47MY4GTQ3YQOGHS2V2OTO2VFDL6XKKS2R4VA6VZZ557WYFY34A, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.929131404 (41h1m32.400342133s) [noise=true , address=145.79.227.22:4510 , piece_id=HCJIUH3VECRBR744KIQ65UKSMTEEKUPHJWXSDBM5SEMUVBJ5YX6A, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.180555093 (41h1m32.651765822s) [noise=true , address=145.79.227.32:4835 , piece_id=PJURUANT2XJY5PNYTSO5W2EDT3HYFAC2BFNCANKOAZ3I6P2GRC3Q, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.587529048 (41h1m33.058739787s) [noise=true , address=145.79.227.41:4087 , piece_id=6GY2NC7F26VLU5NEN65OYU7YIHJYSTI5GNVRTNXM5HO5XQMZKEPA, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.671173874 (41h1m33.142384603s) [noise=true , address=145.79.227.27:4691 , piece_id=Z53YBOXTW4L23P4M3EOW6ADUHDWHIJVRDT7IL4I5FUPPZOW2MRLQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.671209982 (41h1m33.142420711s) [seg_pos={0 0}, upload=36107]: Segment reached optimal threshold of 13 pieces. Dec 5 09:56:56.671423689 (41h1m33.142634428s) [noise=true , address=145.79.227.24:4582 , piece_id=MCQWT2LFD472TQDOAHLGQ5L2X7S654P6TL4J5MSP3CML5ZCURHMA, seg_pos={0 0}, upload=36107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa): protocol: write tcp 188.166.225.170:54136->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:54136->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54136->145.79.227.24:4582: use of closed network connection Dec 5 09:56:56.671432325 (41h1m33.142643064s) [noise=true , address=145.79.227.24:4582 , piece_id=MCQWT2LFD472TQDOAHLGQ5L2X7S654P6TL4J5MSP3CML5ZCURHMA, seg_pos={0 0}, upload=36107]: Long tail cancellation due to optimal threshold Dec 5 09:56:56.671496377 (41h1m33.142707106s) [noise=true , address=145.79.227.44:4193 , piece_id=327UUYIASLKFGUIQVY4P5UML4R2R4ODLUAYXR3GISKIT6CFDUGAQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: write tcp 188.166.225.170:39276->145.79.227.44:4193: use of closed network connection; write tcp 188.166.225.170:39276->145.79.227.44:4193: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39276->145.79.227.44:4193: use of closed network connection Dec 5 09:56:56.671501627 (41h1m33.142712366s) [noise=true , address=145.79.227.44:4193 , piece_id=327UUYIASLKFGUIQVY4P5UML4R2R4ODLUAYXR3GISKIT6CFDUGAQ, seg_pos={0 0}, upload=36107]: Long tail cancellation due to optimal threshold Dec 5 09:56:56.671516466 (41h1m33.142727195s) [noise=true , address=77.90.31.16:4735 , piece_id=V4YJIHVMXYTBK7IHU272RYCKUXI44B2MXRLETMWMO2WAI2XRU4SQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: write tcp 188.166.225.170:41866->77.90.31.16:4735: use of closed network connection; storage node overloaded, request limit: 2000 Dec 5 09:56:56.671525944 (41h1m33.142736673s) [noise=true , address=77.90.31.16:4735 , piece_id=V4YJIHVMXYTBK7IHU272RYCKUXI44B2MXRLETMWMO2WAI2XRU4SQ, seg_pos={0 0}, upload=36107]: Long tail cancellation due to optimal threshold Dec 5 09:56:56.671548016 (41h1m33.142758745s) [seg_pos={0 0}, upload=36107]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 5 09:56:56.671623500 (41h1m33.142834229s) [upload=36107]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 5 09:56:56.671631204 (41h1m33.142841933s) [upload=36107]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 5 09:56:57.112987571 (41h1m33.5841983s) [upload=36107]: Done uploading segments... Dec 5 09:56:57.113007228 (41h1m33.584217967s) [upload=36107]: Upload finished. err: <nil> Duration: 8.28996431s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 8.986779079s trace: https://jaeger.dev.storj.io/trace/55b376cc76cfe920 FAIL (2025-12-05 10:32:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 9 (41h1m29.913335078s) [upload=36107]: Uploading segments... Dec 5 09:56:53.442159225 (41h1m29.913369955s) [upload=36107]: Got next segment. Inline: false Dec 5 09:56:53.442188120 (41h1m29.913398849s) [upload=36107]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 5 09:56:53.442194372 (41h1m29.913405101s) [upload=36107]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 5 09:56:53.701976567 (41h1m30.173187296s) [seg_pos={0 0}, upload=36107]: Begin upload segment... Dec 5 09:56:53.702020460 (41h1m30.173231189s) [seg_pos={0 0}, upload=36107]: Done begin upload segment. Dec 5 09:56:53.702056649 (41h1m30.173267388s) [noise=true , address=77.90.31.21:4878 , piece_id=RCFTETF7SGR6FTUBJWWBJDKOZIHZKBMSUBZYVWYKF2MP56PMIWMQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702064965 (41h1m30.173275694s) [noise=true , address=145.79.227.28:4725 , piece_id=ZE642TJU4ZJKBOHQ6K6GPLPTN2YKC4BOP325LN7QXALZSDPG66HQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702111533 (41h1m30.173322272s) [noise=true , address=145.79.227.42:4121 , piece_id=6D47MY4GTQ3YQOGHS2V2OTO2VFDL6XKKS2R4VA6VZZ557WYFY34A, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702548986 (41h1m30.173759766s) [noise=true , address=145.79.227.27:4691 , piece_id=Z53YBOXTW4L23P4M3EOW6ADUHDWHIJVRDT7IL4I5FUPPZOW2MRLQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702931455 (41h1m30.174142194s) [noise=true , address=77.90.31.16:4735 , piece_id=V4YJIHVMXYTBK7IHU272RYCKUXI44B2MXRLETMWMO2WAI2XRU4SQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.703301079 (41h1m30.174511808s) [noise=true , address=145.79.227.10:4401 , piece_id=XAVSV3SGPDDLZWGKQLHA2CVYTGOPO7J3YVEO5C3MYIN5SJEUZ2OQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.703633000 (41h1m30.174843729s) [noise=true , address=145.79.227.24:4583 , piece_id=IB6T3J3CF3P3A54JEHHBK6KEPMUEI32B4QYSSA2G2BUMIVUC7DGQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.703957648 (41h1m30.175168377s) [noise=true , address=145.79.227.26:4654 , piece_id=23NNREG47WSZWJNPZWP275L62YBBSTB6RQWTINIRTTAYU66DRT7Q, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.704788621 (41h1m30.17599936s) [noise=true , address=145.79.227.43:4159 , piece_id=S5T27VFA6UNNONPZBZILNTQUUNSRLWPREKOJORMLKOXWNL3GMBSQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.705457525 (41h1m30.176668264s) [noise=true , address=145.79.227.32:4835 , piece_id=PJURUANT2XJY5PNYTSO5W2EDT3HYFAC2BFNCANKOAZ3I6P2GRC3Q, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.706486825 (41h1m30.177697554s) [noise=true , address=145.79.227.24:4582 , piece_id=MCQWT2LFD472TQDOAHLGQ5L2X7S654P6TL4J5MSP3CML5ZCURHMA, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.707626675 (41h1m30.178837414s) [noise=true , address=145.79.227.44:4193 , piece_id=327UUYIASLKFGUIQVY4P5UML4R2R4ODLUAYXR3GISKIT6CFDUGAQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.708631078 (41h1m30.179841817s) [noise=true , address=145.79.227.40:4050 , piece_id=QFSFC3WFER7VBVZNB3XLRIEFWGM4Q5VTAAJIWG4HAERKXSHZGZ4Q, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.709908630 (41h1m30.181119359s) [noise=true , address=145.79.227.22:4510 , piece_id=HCJIUH3VECRBR744KIQ65UKSMTEEKUPHJWXSDBM5SEMUVBJ5YX6A, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.710331386 (41h1m30.181542125s) [noise=true , address=145.79.227.41:4087 , piece_id=6GY2NC7F26VLU5NEN65OYU7YIHJYSTI5GNVRTNXM5HO5XQMZKEPA, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.710803685 (41h1m30.182014424s) [noise=true , address=45.91.116.4:4937 , piece_id=VFAPMIAB2G2BBOMNSM5EXONUL5BNORMR7KXIYSTQ7UFRVASVTQHQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:55.513663514 (41h1m31.984874243s) [upload=36107]: Next returned nil segment Dec 5 09:56:55.513682680 (41h1m31.98489341s) [upload=36107]: Waiting for error group managing segments... Dec 5 09:56:55.808504186 (41h1m32.279714915s) [noise=true , address=77.90.31.21:4878 , piece_id=RCFTETF7SGR6FTUBJWWBJDKOZIHZKBMSUBZYVWYKF2MP56PMIWMQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.866933347 (41h1m32.338144086s) [noise=true , address=145.79.227.10:4401 , piece_id=XAVSV3SGPDDLZWGKQLHA2CVYTGOPO7J3YVEO5C3MYIN5SJEUZ2OQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.869510454 (41h1m32.340721183s) [noise=true , address=145.79.227.28:4725 , piece_id=ZE642TJU4ZJKBOHQ6K6GPLPTN2YKC4BOP325LN7QXALZSDPG66HQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.875082795 (41h1m32.346293524s) [noise=true , address=145.79.227.43:4159 , piece_id=S5T27VFA6UNNONPZBZILNTQUUNSRLWPREKOJORMLKOXWNL3GMBSQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.884542648 (41h1m32.355753387s) [noise=true , address=145.79.227.24:4583 , piece_id=IB6T3J3CF3P3A54JEHHBK6KEPMUEI32B4QYSSA2G2BUMIVUC7DGQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.888876321 (41h1m32.36008707s) [noise=true , address=145.79.227.40:4050 , piece_id=QFSFC3WFER7VBVZNB3XLRIEFWGM4Q5VTAAJIWG4HAERKXSHZGZ4Q, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.889468719 (41h1m32.360679448s) [noise=true , address=145.79.227.26:4654 , piece_id=23NNREG47WSZWJNPZWP275L62YBBSTB6RQWTINIRTTAYU66DRT7Q, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.900234388 (41h1m32.371445117s) [noise=true , address=45.91.116.4:4937 , piece_id=VFAPMIAB2G2BBOMNSM5EXONUL5BNORMR7KXIYSTQ7UFRVASVTQHQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.923941432 (41h1m32.395152161s) [noise=true , address=145.79.227.42:4121 , piece_id=6D47MY4GTQ3YQOGHS2V2OTO2VFDL6XKKS2R4VA6VZZ557WYFY34A, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.929131404 (41h1m32.400342133s) [noise=true , address=145.79.227.22:4510 , piece_id=HCJIUH3VECRBR744KIQ65UKSMTEEKUPHJWXSDBM5SEMUVBJ5YX6A, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.180555093 (41h1m32.651765822s) [noise=true , address=145.79.227.32:4835 , piece_id=PJURUANT2XJY5PNYTSO5W2EDT3HYFAC2BFNCANKOAZ3I6P2GRC3Q, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.587529048 (41h1m33.058739787s) [noise=true , address=145.79.227.41:4087 , piece_id=6GY2NC7F26VLU5NEN65OYU7YIHJYSTI5GNVRTNXM5HO5XQMZKEPA, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.671173874 (41h1m33.142384603s) [noise=true , address=145.79.227.27:4691 , piece_id=Z53YBOXTW4L23P4M3EOW6ADUHDWHIJVRDT7IL4I5FUPPZOW2MRLQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.671209982 (41h1m33.142420711s) [seg_pos={0 0}, upload=36107]: Segment reached optimal threshold of 13 pieces. Dec 5 09:56:56.671423689 (41h1m33.142634428s) [noise=true , address=145.79.227.24:4582 , piece_id=MCQWT2LFD472TQDOAHLGQ5L2X7S654P6TL4J5MSP3CML5ZCURHMA, seg_pos={0 0}, upload=36107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa): protocol: write tcp 188.166.225.170:54136->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:54136->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54136->145.79.227.24:4582: use of closed network connection Dec 5 09:56:56.671432325 (41h1m33.142643064s) [noise=true , address=145.79.227.24:4582 , piece_id=MCQWT2LFD472TQDOAHLGQ5L2X7S654P6TL4J5MSP3CML5ZCURHMA, seg_pos={0 0}, upload=36107]: Long tail cancellation due to optimal threshold Dec 5 09:56:56.671496377 (41h1m33.142707106s) [noise=true , address=145.79.227.44:4193 , piece_id=327UUYIASLKFGUIQVY4P5UML4R2R4ODLUAYXR3GISKIT6CFDUGAQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: write tcp 188.166.225.170:39276->145.79.227.44:4193: use of closed network connection; write tcp 188.166.225.170:39276->145.79.227.44:4193: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39276->145.79.227.44:4193: use of closed network connection Dec 5 09:56:56.671501627 (41h1m33.142712366s) [noise=true , address=145.79.227.44:4193 , piece_id=327UUYIASLKFGUIQVY4P5UML4R2R4ODLUAYXR3GISKIT6CFDUGAQ, seg_pos={0 0}, upload=36107]: Long tail cancellation due to optimal threshold Dec 5 09:56:56.671516466 (41h1m33.142727195s) [noise=true , address=77.90.31.16:4735 , piece_id=V4YJIHVMXYTBK7IHU272RYCKUXI44B2MXRLETMWMO2WAI2XRU4SQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: write tcp 188.166.225.170:41866->77.90.31.16:4735: use of closed network connection; storage node overloaded, request limit: 2000 Dec 5 09:56:56.671525944 (41h1m33.142736673s) [noise=true , address=77.90.31.16:4735 , piece_id=V4YJIHVMXYTBK7IHU272RYCKUXI44B2MXRLETMWMO2WAI2XRU4SQ, seg_pos={0 0}, upload=36107]: Long tail cancellation due to optimal threshold Dec 5 09:56:56.671548016 (41h1m33.142758745s) [seg_pos={0 0}, upload=36107]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 5 09:56:56.671623500 (41h1m33.142834229s) [upload=36107]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 5 09:56:56.671631204 (41h1m33.142841933s) [upload=36107]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 5 09:56:57.112987571 (41h1m33.5841983s) [upload=36107]: Done uploading segments... Dec 5 09:56:57.113007228 (41h1m33.584217967s) [upload=36107]: Upload finished. err: <nil> Duration: 8.986807303s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 8.54243924s trace: https://jaeger.dev.storj.io/trace/55b376cc76cfe920 FAIL (2025-12-05 10:31:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 9 (41h1m29.913335078s) [upload=36107]: Uploading segments... Dec 5 09:56:53.442159225 (41h1m29.913369955s) [upload=36107]: Got next segment. Inline: false Dec 5 09:56:53.442188120 (41h1m29.913398849s) [upload=36107]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 5 09:56:53.442194372 (41h1m29.913405101s) [upload=36107]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 5 09:56:53.701976567 (41h1m30.173187296s) [seg_pos={0 0}, upload=36107]: Begin upload segment... Dec 5 09:56:53.702020460 (41h1m30.173231189s) [seg_pos={0 0}, upload=36107]: Done begin upload segment. Dec 5 09:56:53.702056649 (41h1m30.173267388s) [noise=true , address=77.90.31.21:4878 , piece_id=RCFTETF7SGR6FTUBJWWBJDKOZIHZKBMSUBZYVWYKF2MP56PMIWMQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702064965 (41h1m30.173275694s) [noise=true , address=145.79.227.28:4725 , piece_id=ZE642TJU4ZJKBOHQ6K6GPLPTN2YKC4BOP325LN7QXALZSDPG66HQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702111533 (41h1m30.173322272s) [noise=true , address=145.79.227.42:4121 , piece_id=6D47MY4GTQ3YQOGHS2V2OTO2VFDL6XKKS2R4VA6VZZ557WYFY34A, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702548986 (41h1m30.173759766s) [noise=true , address=145.79.227.27:4691 , piece_id=Z53YBOXTW4L23P4M3EOW6ADUHDWHIJVRDT7IL4I5FUPPZOW2MRLQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702931455 (41h1m30.174142194s) [noise=true , address=77.90.31.16:4735 , piece_id=V4YJIHVMXYTBK7IHU272RYCKUXI44B2MXRLETMWMO2WAI2XRU4SQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.703301079 (41h1m30.174511808s) [noise=true , address=145.79.227.10:4401 , piece_id=XAVSV3SGPDDLZWGKQLHA2CVYTGOPO7J3YVEO5C3MYIN5SJEUZ2OQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.703633000 (41h1m30.174843729s) [noise=true , address=145.79.227.24:4583 , piece_id=IB6T3J3CF3P3A54JEHHBK6KEPMUEI32B4QYSSA2G2BUMIVUC7DGQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.703957648 (41h1m30.175168377s) [noise=true , address=145.79.227.26:4654 , piece_id=23NNREG47WSZWJNPZWP275L62YBBSTB6RQWTINIRTTAYU66DRT7Q, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.704788621 (41h1m30.17599936s) [noise=true , address=145.79.227.43:4159 , piece_id=S5T27VFA6UNNONPZBZILNTQUUNSRLWPREKOJORMLKOXWNL3GMBSQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.705457525 (41h1m30.176668264s) [noise=true , address=145.79.227.32:4835 , piece_id=PJURUANT2XJY5PNYTSO5W2EDT3HYFAC2BFNCANKOAZ3I6P2GRC3Q, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.706486825 (41h1m30.177697554s) [noise=true , address=145.79.227.24:4582 , piece_id=MCQWT2LFD472TQDOAHLGQ5L2X7S654P6TL4J5MSP3CML5ZCURHMA, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.707626675 (41h1m30.178837414s) [noise=true , address=145.79.227.44:4193 , piece_id=327UUYIASLKFGUIQVY4P5UML4R2R4ODLUAYXR3GISKIT6CFDUGAQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.708631078 (41h1m30.179841817s) [noise=true , address=145.79.227.40:4050 , piece_id=QFSFC3WFER7VBVZNB3XLRIEFWGM4Q5VTAAJIWG4HAERKXSHZGZ4Q, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.709908630 (41h1m30.181119359s) [noise=true , address=145.79.227.22:4510 , piece_id=HCJIUH3VECRBR744KIQ65UKSMTEEKUPHJWXSDBM5SEMUVBJ5YX6A, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.710331386 (41h1m30.181542125s) [noise=true , address=145.79.227.41:4087 , piece_id=6GY2NC7F26VLU5NEN65OYU7YIHJYSTI5GNVRTNXM5HO5XQMZKEPA, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.710803685 (41h1m30.182014424s) [noise=true , address=45.91.116.4:4937 , piece_id=VFAPMIAB2G2BBOMNSM5EXONUL5BNORMR7KXIYSTQ7UFRVASVTQHQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:55.513663514 (41h1m31.984874243s) [upload=36107]: Next returned nil segment Dec 5 09:56:55.513682680 (41h1m31.98489341s) [upload=36107]: Waiting for error group managing segments... Dec 5 09:56:55.808504186 (41h1m32.279714915s) [noise=true , address=77.90.31.21:4878 , piece_id=RCFTETF7SGR6FTUBJWWBJDKOZIHZKBMSUBZYVWYKF2MP56PMIWMQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.866933347 (41h1m32.338144086s) [noise=true , address=145.79.227.10:4401 , piece_id=XAVSV3SGPDDLZWGKQLHA2CVYTGOPO7J3YVEO5C3MYIN5SJEUZ2OQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.869510454 (41h1m32.340721183s) [noise=true , address=145.79.227.28:4725 , piece_id=ZE642TJU4ZJKBOHQ6K6GPLPTN2YKC4BOP325LN7QXALZSDPG66HQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.875082795 (41h1m32.346293524s) [noise=true , address=145.79.227.43:4159 , piece_id=S5T27VFA6UNNONPZBZILNTQUUNSRLWPREKOJORMLKOXWNL3GMBSQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.884542648 (41h1m32.355753387s) [noise=true , address=145.79.227.24:4583 , piece_id=IB6T3J3CF3P3A54JEHHBK6KEPMUEI32B4QYSSA2G2BUMIVUC7DGQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.888876321 (41h1m32.36008707s) [noise=true , address=145.79.227.40:4050 , piece_id=QFSFC3WFER7VBVZNB3XLRIEFWGM4Q5VTAAJIWG4HAERKXSHZGZ4Q, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.889468719 (41h1m32.360679448s) [noise=true , address=145.79.227.26:4654 , piece_id=23NNREG47WSZWJNPZWP275L62YBBSTB6RQWTINIRTTAYU66DRT7Q, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.900234388 (41h1m32.371445117s) [noise=true , address=45.91.116.4:4937 , piece_id=VFAPMIAB2G2BBOMNSM5EXONUL5BNORMR7KXIYSTQ7UFRVASVTQHQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.923941432 (41h1m32.395152161s) [noise=true , address=145.79.227.42:4121 , piece_id=6D47MY4GTQ3YQOGHS2V2OTO2VFDL6XKKS2R4VA6VZZ557WYFY34A, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.929131404 (41h1m32.400342133s) [noise=true , address=145.79.227.22:4510 , piece_id=HCJIUH3VECRBR744KIQ65UKSMTEEKUPHJWXSDBM5SEMUVBJ5YX6A, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.180555093 (41h1m32.651765822s) [noise=true , address=145.79.227.32:4835 , piece_id=PJURUANT2XJY5PNYTSO5W2EDT3HYFAC2BFNCANKOAZ3I6P2GRC3Q, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.587529048 (41h1m33.058739787s) [noise=true , address=145.79.227.41:4087 , piece_id=6GY2NC7F26VLU5NEN65OYU7YIHJYSTI5GNVRTNXM5HO5XQMZKEPA, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.671173874 (41h1m33.142384603s) [noise=true , address=145.79.227.27:4691 , piece_id=Z53YBOXTW4L23P4M3EOW6ADUHDWHIJVRDT7IL4I5FUPPZOW2MRLQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.671209982 (41h1m33.142420711s) [seg_pos={0 0}, upload=36107]: Segment reached optimal threshold of 13 pieces. Dec 5 09:56:56.671423689 (41h1m33.142634428s) [noise=true , address=145.79.227.24:4582 , piece_id=MCQWT2LFD472TQDOAHLGQ5L2X7S654P6TL4J5MSP3CML5ZCURHMA, seg_pos={0 0}, upload=36107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa): protocol: write tcp 188.166.225.170:54136->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:54136->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54136->145.79.227.24:4582: use of closed network connection Dec 5 09:56:56.671432325 (41h1m33.142643064s) [noise=true , address=145.79.227.24:4582 , piece_id=MCQWT2LFD472TQDOAHLGQ5L2X7S654P6TL4J5MSP3CML5ZCURHMA, seg_pos={0 0}, upload=36107]: Long tail cancellation due to optimal threshold Dec 5 09:56:56.671496377 (41h1m33.142707106s) [noise=true , address=145.79.227.44:4193 , piece_id=327UUYIASLKFGUIQVY4P5UML4R2R4ODLUAYXR3GISKIT6CFDUGAQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: write tcp 188.166.225.170:39276->145.79.227.44:4193: use of closed network connection; write tcp 188.166.225.170:39276->145.79.227.44:4193: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39276->145.79.227.44:4193: use of closed network connection Dec 5 09:56:56.671501627 (41h1m33.142712366s) [noise=true , address=145.79.227.44:4193 , piece_id=327UUYIASLKFGUIQVY4P5UML4R2R4ODLUAYXR3GISKIT6CFDUGAQ, seg_pos={0 0}, upload=36107]: Long tail cancellation due to optimal threshold Dec 5 09:56:56.671516466 (41h1m33.142727195s) [noise=true , address=77.90.31.16:4735 , piece_id=V4YJIHVMXYTBK7IHU272RYCKUXI44B2MXRLETMWMO2WAI2XRU4SQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: write tcp 188.166.225.170:41866->77.90.31.16:4735: use of closed network connection; storage node overloaded, request limit: 2000 Dec 5 09:56:56.671525944 (41h1m33.142736673s) [noise=true , address=77.90.31.16:4735 , piece_id=V4YJIHVMXYTBK7IHU272RYCKUXI44B2MXRLETMWMO2WAI2XRU4SQ, seg_pos={0 0}, upload=36107]: Long tail cancellation due to optimal threshold Dec 5 09:56:56.671548016 (41h1m33.142758745s) [seg_pos={0 0}, upload=36107]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 5 09:56:56.671623500 (41h1m33.142834229s) [upload=36107]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 5 09:56:56.671631204 (41h1m33.142841933s) [upload=36107]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 5 09:56:57.112987571 (41h1m33.5841983s) [upload=36107]: Done uploading segments... Dec 5 09:56:57.113007228 (41h1m33.584217967s) [upload=36107]: Upload finished. err: <nil> Duration: 8.542491609s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 9.989724688s trace: https://jaeger.dev.storj.io/trace/55b376cc76cfe920 FAIL (2025-12-05 10:31:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 9 (41h1m29.913335078s) [upload=36107]: Uploading segments... Dec 5 09:56:53.442159225 (41h1m29.913369955s) [upload=36107]: Got next segment. Inline: false Dec 5 09:56:53.442188120 (41h1m29.913398849s) [upload=36107]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 5 09:56:53.442194372 (41h1m29.913405101s) [upload=36107]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 5 09:56:53.701976567 (41h1m30.173187296s) [seg_pos={0 0}, upload=36107]: Begin upload segment... Dec 5 09:56:53.702020460 (41h1m30.173231189s) [seg_pos={0 0}, upload=36107]: Done begin upload segment. Dec 5 09:56:53.702056649 (41h1m30.173267388s) [noise=true , address=77.90.31.21:4878 , piece_id=RCFTETF7SGR6FTUBJWWBJDKOZIHZKBMSUBZYVWYKF2MP56PMIWMQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702064965 (41h1m30.173275694s) [noise=true , address=145.79.227.28:4725 , piece_id=ZE642TJU4ZJKBOHQ6K6GPLPTN2YKC4BOP325LN7QXALZSDPG66HQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702111533 (41h1m30.173322272s) [noise=true , address=145.79.227.42:4121 , piece_id=6D47MY4GTQ3YQOGHS2V2OTO2VFDL6XKKS2R4VA6VZZ557WYFY34A, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702548986 (41h1m30.173759766s) [noise=true , address=145.79.227.27:4691 , piece_id=Z53YBOXTW4L23P4M3EOW6ADUHDWHIJVRDT7IL4I5FUPPZOW2MRLQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702931455 (41h1m30.174142194s) [noise=true , address=77.90.31.16:4735 , piece_id=V4YJIHVMXYTBK7IHU272RYCKUXI44B2MXRLETMWMO2WAI2XRU4SQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.703301079 (41h1m30.174511808s) [noise=true , address=145.79.227.10:4401 , piece_id=XAVSV3SGPDDLZWGKQLHA2CVYTGOPO7J3YVEO5C3MYIN5SJEUZ2OQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.703633000 (41h1m30.174843729s) [noise=true , address=145.79.227.24:4583 , piece_id=IB6T3J3CF3P3A54JEHHBK6KEPMUEI32B4QYSSA2G2BUMIVUC7DGQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.703957648 (41h1m30.175168377s) [noise=true , address=145.79.227.26:4654 , piece_id=23NNREG47WSZWJNPZWP275L62YBBSTB6RQWTINIRTTAYU66DRT7Q, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.704788621 (41h1m30.17599936s) [noise=true , address=145.79.227.43:4159 , piece_id=S5T27VFA6UNNONPZBZILNTQUUNSRLWPREKOJORMLKOXWNL3GMBSQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.705457525 (41h1m30.176668264s) [noise=true , address=145.79.227.32:4835 , piece_id=PJURUANT2XJY5PNYTSO5W2EDT3HYFAC2BFNCANKOAZ3I6P2GRC3Q, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.706486825 (41h1m30.177697554s) [noise=true , address=145.79.227.24:4582 , piece_id=MCQWT2LFD472TQDOAHLGQ5L2X7S654P6TL4J5MSP3CML5ZCURHMA, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.707626675 (41h1m30.178837414s) [noise=true , address=145.79.227.44:4193 , piece_id=327UUYIASLKFGUIQVY4P5UML4R2R4ODLUAYXR3GISKIT6CFDUGAQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.708631078 (41h1m30.179841817s) [noise=true , address=145.79.227.40:4050 , piece_id=QFSFC3WFER7VBVZNB3XLRIEFWGM4Q5VTAAJIWG4HAERKXSHZGZ4Q, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.709908630 (41h1m30.181119359s) [noise=true , address=145.79.227.22:4510 , piece_id=HCJIUH3VECRBR744KIQ65UKSMTEEKUPHJWXSDBM5SEMUVBJ5YX6A, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.710331386 (41h1m30.181542125s) [noise=true , address=145.79.227.41:4087 , piece_id=6GY2NC7F26VLU5NEN65OYU7YIHJYSTI5GNVRTNXM5HO5XQMZKEPA, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.710803685 (41h1m30.182014424s) [noise=true , address=45.91.116.4:4937 , piece_id=VFAPMIAB2G2BBOMNSM5EXONUL5BNORMR7KXIYSTQ7UFRVASVTQHQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:55.513663514 (41h1m31.984874243s) [upload=36107]: Next returned nil segment Dec 5 09:56:55.513682680 (41h1m31.98489341s) [upload=36107]: Waiting for error group managing segments... Dec 5 09:56:55.808504186 (41h1m32.279714915s) [noise=true , address=77.90.31.21:4878 , piece_id=RCFTETF7SGR6FTUBJWWBJDKOZIHZKBMSUBZYVWYKF2MP56PMIWMQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.866933347 (41h1m32.338144086s) [noise=true , address=145.79.227.10:4401 , piece_id=XAVSV3SGPDDLZWGKQLHA2CVYTGOPO7J3YVEO5C3MYIN5SJEUZ2OQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.869510454 (41h1m32.340721183s) [noise=true , address=145.79.227.28:4725 , piece_id=ZE642TJU4ZJKBOHQ6K6GPLPTN2YKC4BOP325LN7QXALZSDPG66HQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.875082795 (41h1m32.346293524s) [noise=true , address=145.79.227.43:4159 , piece_id=S5T27VFA6UNNONPZBZILNTQUUNSRLWPREKOJORMLKOXWNL3GMBSQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.884542648 (41h1m32.355753387s) [noise=true , address=145.79.227.24:4583 , piece_id=IB6T3J3CF3P3A54JEHHBK6KEPMUEI32B4QYSSA2G2BUMIVUC7DGQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.888876321 (41h1m32.36008707s) [noise=true , address=145.79.227.40:4050 , piece_id=QFSFC3WFER7VBVZNB3XLRIEFWGM4Q5VTAAJIWG4HAERKXSHZGZ4Q, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.889468719 (41h1m32.360679448s) [noise=true , address=145.79.227.26:4654 , piece_id=23NNREG47WSZWJNPZWP275L62YBBSTB6RQWTINIRTTAYU66DRT7Q, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.900234388 (41h1m32.371445117s) [noise=true , address=45.91.116.4:4937 , piece_id=VFAPMIAB2G2BBOMNSM5EXONUL5BNORMR7KXIYSTQ7UFRVASVTQHQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.923941432 (41h1m32.395152161s) [noise=true , address=145.79.227.42:4121 , piece_id=6D47MY4GTQ3YQOGHS2V2OTO2VFDL6XKKS2R4VA6VZZ557WYFY34A, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.929131404 (41h1m32.400342133s) [noise=true , address=145.79.227.22:4510 , piece_id=HCJIUH3VECRBR744KIQ65UKSMTEEKUPHJWXSDBM5SEMUVBJ5YX6A, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.180555093 (41h1m32.651765822s) [noise=true , address=145.79.227.32:4835 , piece_id=PJURUANT2XJY5PNYTSO5W2EDT3HYFAC2BFNCANKOAZ3I6P2GRC3Q, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.587529048 (41h1m33.058739787s) [noise=true , address=145.79.227.41:4087 , piece_id=6GY2NC7F26VLU5NEN65OYU7YIHJYSTI5GNVRTNXM5HO5XQMZKEPA, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.671173874 (41h1m33.142384603s) [noise=true , address=145.79.227.27:4691 , piece_id=Z53YBOXTW4L23P4M3EOW6ADUHDWHIJVRDT7IL4I5FUPPZOW2MRLQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.671209982 (41h1m33.142420711s) [seg_pos={0 0}, upload=36107]: Segment reached optimal threshold of 13 pieces. Dec 5 09:56:56.671423689 (41h1m33.142634428s) [noise=true , address=145.79.227.24:4582 , piece_id=MCQWT2LFD472TQDOAHLGQ5L2X7S654P6TL4J5MSP3CML5ZCURHMA, seg_pos={0 0}, upload=36107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa): protocol: write tcp 188.166.225.170:54136->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:54136->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54136->145.79.227.24:4582: use of closed network connection Dec 5 09:56:56.671432325 (41h1m33.142643064s) [noise=true , address=145.79.227.24:4582 , piece_id=MCQWT2LFD472TQDOAHLGQ5L2X7S654P6TL4J5MSP3CML5ZCURHMA, seg_pos={0 0}, upload=36107]: Long tail cancellation due to optimal threshold Dec 5 09:56:56.671496377 (41h1m33.142707106s) [noise=true , address=145.79.227.44:4193 , piece_id=327UUYIASLKFGUIQVY4P5UML4R2R4ODLUAYXR3GISKIT6CFDUGAQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: write tcp 188.166.225.170:39276->145.79.227.44:4193: use of closed network connection; write tcp 188.166.225.170:39276->145.79.227.44:4193: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39276->145.79.227.44:4193: use of closed network connection Dec 5 09:56:56.671501627 (41h1m33.142712366s) [noise=true , address=145.79.227.44:4193 , piece_id=327UUYIASLKFGUIQVY4P5UML4R2R4ODLUAYXR3GISKIT6CFDUGAQ, seg_pos={0 0}, upload=36107]: Long tail cancellation due to optimal threshold Dec 5 09:56:56.671516466 (41h1m33.142727195s) [noise=true , address=77.90.31.16:4735 , piece_id=V4YJIHVMXYTBK7IHU272RYCKUXI44B2MXRLETMWMO2WAI2XRU4SQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: write tcp 188.166.225.170:41866->77.90.31.16:4735: use of closed network connection; storage node overloaded, request limit: 2000 Dec 5 09:56:56.671525944 (41h1m33.142736673s) [noise=true , address=77.90.31.16:4735 , piece_id=V4YJIHVMXYTBK7IHU272RYCKUXI44B2MXRLETMWMO2WAI2XRU4SQ, seg_pos={0 0}, upload=36107]: Long tail cancellation due to optimal threshold Dec 5 09:56:56.671548016 (41h1m33.142758745s) [seg_pos={0 0}, upload=36107]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 5 09:56:56.671623500 (41h1m33.142834229s) [upload=36107]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 5 09:56:56.671631204 (41h1m33.142841933s) [upload=36107]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 5 09:56:57.112987571 (41h1m33.5841983s) [upload=36107]: Done uploading segments... Dec 5 09:56:57.113007228 (41h1m33.584217967s) [upload=36107]: Upload finished. err: <nil> Duration: 9.989750549s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 8.605957729s trace: https://jaeger.dev.storj.io/trace/55b376cc76cfe920 FAIL (2025-12-05 10:30:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 9 (41h1m29.913335078s) [upload=36107]: Uploading segments... Dec 5 09:56:53.442159225 (41h1m29.913369955s) [upload=36107]: Got next segment. Inline: false Dec 5 09:56:53.442188120 (41h1m29.913398849s) [upload=36107]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 5 09:56:53.442194372 (41h1m29.913405101s) [upload=36107]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 5 09:56:53.701976567 (41h1m30.173187296s) [seg_pos={0 0}, upload=36107]: Begin upload segment... Dec 5 09:56:53.702020460 (41h1m30.173231189s) [seg_pos={0 0}, upload=36107]: Done begin upload segment. Dec 5 09:56:53.702056649 (41h1m30.173267388s) [noise=true , address=77.90.31.21:4878 , piece_id=RCFTETF7SGR6FTUBJWWBJDKOZIHZKBMSUBZYVWYKF2MP56PMIWMQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702064965 (41h1m30.173275694s) [noise=true , address=145.79.227.28:4725 , piece_id=ZE642TJU4ZJKBOHQ6K6GPLPTN2YKC4BOP325LN7QXALZSDPG66HQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702111533 (41h1m30.173322272s) [noise=true , address=145.79.227.42:4121 , piece_id=6D47MY4GTQ3YQOGHS2V2OTO2VFDL6XKKS2R4VA6VZZ557WYFY34A, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702548986 (41h1m30.173759766s) [noise=true , address=145.79.227.27:4691 , piece_id=Z53YBOXTW4L23P4M3EOW6ADUHDWHIJVRDT7IL4I5FUPPZOW2MRLQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702931455 (41h1m30.174142194s) [noise=true , address=77.90.31.16:4735 , piece_id=V4YJIHVMXYTBK7IHU272RYCKUXI44B2MXRLETMWMO2WAI2XRU4SQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.703301079 (41h1m30.174511808s) [noise=true , address=145.79.227.10:4401 , piece_id=XAVSV3SGPDDLZWGKQLHA2CVYTGOPO7J3YVEO5C3MYIN5SJEUZ2OQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.703633000 (41h1m30.174843729s) [noise=true , address=145.79.227.24:4583 , piece_id=IB6T3J3CF3P3A54JEHHBK6KEPMUEI32B4QYSSA2G2BUMIVUC7DGQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.703957648 (41h1m30.175168377s) [noise=true , address=145.79.227.26:4654 , piece_id=23NNREG47WSZWJNPZWP275L62YBBSTB6RQWTINIRTTAYU66DRT7Q, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.704788621 (41h1m30.17599936s) [noise=true , address=145.79.227.43:4159 , piece_id=S5T27VFA6UNNONPZBZILNTQUUNSRLWPREKOJORMLKOXWNL3GMBSQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.705457525 (41h1m30.176668264s) [noise=true , address=145.79.227.32:4835 , piece_id=PJURUANT2XJY5PNYTSO5W2EDT3HYFAC2BFNCANKOAZ3I6P2GRC3Q, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.706486825 (41h1m30.177697554s) [noise=true , address=145.79.227.24:4582 , piece_id=MCQWT2LFD472TQDOAHLGQ5L2X7S654P6TL4J5MSP3CML5ZCURHMA, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.707626675 (41h1m30.178837414s) [noise=true , address=145.79.227.44:4193 , piece_id=327UUYIASLKFGUIQVY4P5UML4R2R4ODLUAYXR3GISKIT6CFDUGAQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.708631078 (41h1m30.179841817s) [noise=true , address=145.79.227.40:4050 , piece_id=QFSFC3WFER7VBVZNB3XLRIEFWGM4Q5VTAAJIWG4HAERKXSHZGZ4Q, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.709908630 (41h1m30.181119359s) [noise=true , address=145.79.227.22:4510 , piece_id=HCJIUH3VECRBR744KIQ65UKSMTEEKUPHJWXSDBM5SEMUVBJ5YX6A, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.710331386 (41h1m30.181542125s) [noise=true , address=145.79.227.41:4087 , piece_id=6GY2NC7F26VLU5NEN65OYU7YIHJYSTI5GNVRTNXM5HO5XQMZKEPA, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.710803685 (41h1m30.182014424s) [noise=true , address=45.91.116.4:4937 , piece_id=VFAPMIAB2G2BBOMNSM5EXONUL5BNORMR7KXIYSTQ7UFRVASVTQHQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:55.513663514 (41h1m31.984874243s) [upload=36107]: Next returned nil segment Dec 5 09:56:55.513682680 (41h1m31.98489341s) [upload=36107]: Waiting for error group managing segments... Dec 5 09:56:55.808504186 (41h1m32.279714915s) [noise=true , address=77.90.31.21:4878 , piece_id=RCFTETF7SGR6FTUBJWWBJDKOZIHZKBMSUBZYVWYKF2MP56PMIWMQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.866933347 (41h1m32.338144086s) [noise=true , address=145.79.227.10:4401 , piece_id=XAVSV3SGPDDLZWGKQLHA2CVYTGOPO7J3YVEO5C3MYIN5SJEUZ2OQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.869510454 (41h1m32.340721183s) [noise=true , address=145.79.227.28:4725 , piece_id=ZE642TJU4ZJKBOHQ6K6GPLPTN2YKC4BOP325LN7QXALZSDPG66HQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.875082795 (41h1m32.346293524s) [noise=true , address=145.79.227.43:4159 , piece_id=S5T27VFA6UNNONPZBZILNTQUUNSRLWPREKOJORMLKOXWNL3GMBSQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.884542648 (41h1m32.355753387s) [noise=true , address=145.79.227.24:4583 , piece_id=IB6T3J3CF3P3A54JEHHBK6KEPMUEI32B4QYSSA2G2BUMIVUC7DGQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.888876321 (41h1m32.36008707s) [noise=true , address=145.79.227.40:4050 , piece_id=QFSFC3WFER7VBVZNB3XLRIEFWGM4Q5VTAAJIWG4HAERKXSHZGZ4Q, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.889468719 (41h1m32.360679448s) [noise=true , address=145.79.227.26:4654 , piece_id=23NNREG47WSZWJNPZWP275L62YBBSTB6RQWTINIRTTAYU66DRT7Q, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.900234388 (41h1m32.371445117s) [noise=true , address=45.91.116.4:4937 , piece_id=VFAPMIAB2G2BBOMNSM5EXONUL5BNORMR7KXIYSTQ7UFRVASVTQHQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.923941432 (41h1m32.395152161s) [noise=true , address=145.79.227.42:4121 , piece_id=6D47MY4GTQ3YQOGHS2V2OTO2VFDL6XKKS2R4VA6VZZ557WYFY34A, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.929131404 (41h1m32.400342133s) [noise=true , address=145.79.227.22:4510 , piece_id=HCJIUH3VECRBR744KIQ65UKSMTEEKUPHJWXSDBM5SEMUVBJ5YX6A, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.180555093 (41h1m32.651765822s) [noise=true , address=145.79.227.32:4835 , piece_id=PJURUANT2XJY5PNYTSO5W2EDT3HYFAC2BFNCANKOAZ3I6P2GRC3Q, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.587529048 (41h1m33.058739787s) [noise=true , address=145.79.227.41:4087 , piece_id=6GY2NC7F26VLU5NEN65OYU7YIHJYSTI5GNVRTNXM5HO5XQMZKEPA, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.671173874 (41h1m33.142384603s) [noise=true , address=145.79.227.27:4691 , piece_id=Z53YBOXTW4L23P4M3EOW6ADUHDWHIJVRDT7IL4I5FUPPZOW2MRLQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.671209982 (41h1m33.142420711s) [seg_pos={0 0}, upload=36107]: Segment reached optimal threshold of 13 pieces. Dec 5 09:56:56.671423689 (41h1m33.142634428s) [noise=true , address=145.79.227.24:4582 , piece_id=MCQWT2LFD472TQDOAHLGQ5L2X7S654P6TL4J5MSP3CML5ZCURHMA, seg_pos={0 0}, upload=36107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa): protocol: write tcp 188.166.225.170:54136->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:54136->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54136->145.79.227.24:4582: use of closed network connection Dec 5 09:56:56.671432325 (41h1m33.142643064s) [noise=true , address=145.79.227.24:4582 , piece_id=MCQWT2LFD472TQDOAHLGQ5L2X7S654P6TL4J5MSP3CML5ZCURHMA, seg_pos={0 0}, upload=36107]: Long tail cancellation due to optimal threshold Dec 5 09:56:56.671496377 (41h1m33.142707106s) [noise=true , address=145.79.227.44:4193 , piece_id=327UUYIASLKFGUIQVY4P5UML4R2R4ODLUAYXR3GISKIT6CFDUGAQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: write tcp 188.166.225.170:39276->145.79.227.44:4193: use of closed network connection; write tcp 188.166.225.170:39276->145.79.227.44:4193: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39276->145.79.227.44:4193: use of closed network connection Dec 5 09:56:56.671501627 (41h1m33.142712366s) [noise=true , address=145.79.227.44:4193 , piece_id=327UUYIASLKFGUIQVY4P5UML4R2R4ODLUAYXR3GISKIT6CFDUGAQ, seg_pos={0 0}, upload=36107]: Long tail cancellation due to optimal threshold Dec 5 09:56:56.671516466 (41h1m33.142727195s) [noise=true , address=77.90.31.16:4735 , piece_id=V4YJIHVMXYTBK7IHU272RYCKUXI44B2MXRLETMWMO2WAI2XRU4SQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: write tcp 188.166.225.170:41866->77.90.31.16:4735: use of closed network connection; storage node overloaded, request limit: 2000 Dec 5 09:56:56.671525944 (41h1m33.142736673s) [noise=true , address=77.90.31.16:4735 , piece_id=V4YJIHVMXYTBK7IHU272RYCKUXI44B2MXRLETMWMO2WAI2XRU4SQ, seg_pos={0 0}, upload=36107]: Long tail cancellation due to optimal threshold Dec 5 09:56:56.671548016 (41h1m33.142758745s) [seg_pos={0 0}, upload=36107]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 5 09:56:56.671623500 (41h1m33.142834229s) [upload=36107]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 5 09:56:56.671631204 (41h1m33.142841933s) [upload=36107]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 5 09:56:57.112987571 (41h1m33.5841983s) [upload=36107]: Done uploading segments... Dec 5 09:56:57.113007228 (41h1m33.584217967s) [upload=36107]: Upload finished. err: <nil> Duration: 8.605992715s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 9.1528076s trace: https://jaeger.dev.storj.io/trace/55b376cc76cfe920 FAIL (2025-12-05 10:29:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 9 (41h1m29.913335078s) [upload=36107]: Uploading segments... Dec 5 09:56:53.442159225 (41h1m29.913369955s) [upload=36107]: Got next segment. Inline: false Dec 5 09:56:53.442188120 (41h1m29.913398849s) [upload=36107]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 5 09:56:53.442194372 (41h1m29.913405101s) [upload=36107]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 5 09:56:53.701976567 (41h1m30.173187296s) [seg_pos={0 0}, upload=36107]: Begin upload segment... Dec 5 09:56:53.702020460 (41h1m30.173231189s) [seg_pos={0 0}, upload=36107]: Done begin upload segment. Dec 5 09:56:53.702056649 (41h1m30.173267388s) [noise=true , address=77.90.31.21:4878 , piece_id=RCFTETF7SGR6FTUBJWWBJDKOZIHZKBMSUBZYVWYKF2MP56PMIWMQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702064965 (41h1m30.173275694s) [noise=true , address=145.79.227.28:4725 , piece_id=ZE642TJU4ZJKBOHQ6K6GPLPTN2YKC4BOP325LN7QXALZSDPG66HQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702111533 (41h1m30.173322272s) [noise=true , address=145.79.227.42:4121 , piece_id=6D47MY4GTQ3YQOGHS2V2OTO2VFDL6XKKS2R4VA6VZZ557WYFY34A, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702548986 (41h1m30.173759766s) [noise=true , address=145.79.227.27:4691 , piece_id=Z53YBOXTW4L23P4M3EOW6ADUHDWHIJVRDT7IL4I5FUPPZOW2MRLQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702931455 (41h1m30.174142194s) [noise=true , address=77.90.31.16:4735 , piece_id=V4YJIHVMXYTBK7IHU272RYCKUXI44B2MXRLETMWMO2WAI2XRU4SQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.703301079 (41h1m30.174511808s) [noise=true , address=145.79.227.10:4401 , piece_id=XAVSV3SGPDDLZWGKQLHA2CVYTGOPO7J3YVEO5C3MYIN5SJEUZ2OQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.703633000 (41h1m30.174843729s) [noise=true , address=145.79.227.24:4583 , piece_id=IB6T3J3CF3P3A54JEHHBK6KEPMUEI32B4QYSSA2G2BUMIVUC7DGQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.703957648 (41h1m30.175168377s) [noise=true , address=145.79.227.26:4654 , piece_id=23NNREG47WSZWJNPZWP275L62YBBSTB6RQWTINIRTTAYU66DRT7Q, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.704788621 (41h1m30.17599936s) [noise=true , address=145.79.227.43:4159 , piece_id=S5T27VFA6UNNONPZBZILNTQUUNSRLWPREKOJORMLKOXWNL3GMBSQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.705457525 (41h1m30.176668264s) [noise=true , address=145.79.227.32:4835 , piece_id=PJURUANT2XJY5PNYTSO5W2EDT3HYFAC2BFNCANKOAZ3I6P2GRC3Q, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.706486825 (41h1m30.177697554s) [noise=true , address=145.79.227.24:4582 , piece_id=MCQWT2LFD472TQDOAHLGQ5L2X7S654P6TL4J5MSP3CML5ZCURHMA, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.707626675 (41h1m30.178837414s) [noise=true , address=145.79.227.44:4193 , piece_id=327UUYIASLKFGUIQVY4P5UML4R2R4ODLUAYXR3GISKIT6CFDUGAQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.708631078 (41h1m30.179841817s) [noise=true , address=145.79.227.40:4050 , piece_id=QFSFC3WFER7VBVZNB3XLRIEFWGM4Q5VTAAJIWG4HAERKXSHZGZ4Q, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.709908630 (41h1m30.181119359s) [noise=true , address=145.79.227.22:4510 , piece_id=HCJIUH3VECRBR744KIQ65UKSMTEEKUPHJWXSDBM5SEMUVBJ5YX6A, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.710331386 (41h1m30.181542125s) [noise=true , address=145.79.227.41:4087 , piece_id=6GY2NC7F26VLU5NEN65OYU7YIHJYSTI5GNVRTNXM5HO5XQMZKEPA, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.710803685 (41h1m30.182014424s) [noise=true , address=45.91.116.4:4937 , piece_id=VFAPMIAB2G2BBOMNSM5EXONUL5BNORMR7KXIYSTQ7UFRVASVTQHQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:55.513663514 (41h1m31.984874243s) [upload=36107]: Next returned nil segment Dec 5 09:56:55.513682680 (41h1m31.98489341s) [upload=36107]: Waiting for error group managing segments... Dec 5 09:56:55.808504186 (41h1m32.279714915s) [noise=true , address=77.90.31.21:4878 , piece_id=RCFTETF7SGR6FTUBJWWBJDKOZIHZKBMSUBZYVWYKF2MP56PMIWMQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.866933347 (41h1m32.338144086s) [noise=true , address=145.79.227.10:4401 , piece_id=XAVSV3SGPDDLZWGKQLHA2CVYTGOPO7J3YVEO5C3MYIN5SJEUZ2OQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.869510454 (41h1m32.340721183s) [noise=true , address=145.79.227.28:4725 , piece_id=ZE642TJU4ZJKBOHQ6K6GPLPTN2YKC4BOP325LN7QXALZSDPG66HQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.875082795 (41h1m32.346293524s) [noise=true , address=145.79.227.43:4159 , piece_id=S5T27VFA6UNNONPZBZILNTQUUNSRLWPREKOJORMLKOXWNL3GMBSQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.884542648 (41h1m32.355753387s) [noise=true , address=145.79.227.24:4583 , piece_id=IB6T3J3CF3P3A54JEHHBK6KEPMUEI32B4QYSSA2G2BUMIVUC7DGQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.888876321 (41h1m32.36008707s) [noise=true , address=145.79.227.40:4050 , piece_id=QFSFC3WFER7VBVZNB3XLRIEFWGM4Q5VTAAJIWG4HAERKXSHZGZ4Q, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.889468719 (41h1m32.360679448s) [noise=true , address=145.79.227.26:4654 , piece_id=23NNREG47WSZWJNPZWP275L62YBBSTB6RQWTINIRTTAYU66DRT7Q, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.900234388 (41h1m32.371445117s) [noise=true , address=45.91.116.4:4937 , piece_id=VFAPMIAB2G2BBOMNSM5EXONUL5BNORMR7KXIYSTQ7UFRVASVTQHQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.923941432 (41h1m32.395152161s) [noise=true , address=145.79.227.42:4121 , piece_id=6D47MY4GTQ3YQOGHS2V2OTO2VFDL6XKKS2R4VA6VZZ557WYFY34A, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.929131404 (41h1m32.400342133s) [noise=true , address=145.79.227.22:4510 , piece_id=HCJIUH3VECRBR744KIQ65UKSMTEEKUPHJWXSDBM5SEMUVBJ5YX6A, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.180555093 (41h1m32.651765822s) [noise=true , address=145.79.227.32:4835 , piece_id=PJURUANT2XJY5PNYTSO5W2EDT3HYFAC2BFNCANKOAZ3I6P2GRC3Q, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.587529048 (41h1m33.058739787s) [noise=true , address=145.79.227.41:4087 , piece_id=6GY2NC7F26VLU5NEN65OYU7YIHJYSTI5GNVRTNXM5HO5XQMZKEPA, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.671173874 (41h1m33.142384603s) [noise=true , address=145.79.227.27:4691 , piece_id=Z53YBOXTW4L23P4M3EOW6ADUHDWHIJVRDT7IL4I5FUPPZOW2MRLQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.671209982 (41h1m33.142420711s) [seg_pos={0 0}, upload=36107]: Segment reached optimal threshold of 13 pieces. Dec 5 09:56:56.671423689 (41h1m33.142634428s) [noise=true , address=145.79.227.24:4582 , piece_id=MCQWT2LFD472TQDOAHLGQ5L2X7S654P6TL4J5MSP3CML5ZCURHMA, seg_pos={0 0}, upload=36107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa): protocol: write tcp 188.166.225.170:54136->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:54136->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54136->145.79.227.24:4582: use of closed network connection Dec 5 09:56:56.671432325 (41h1m33.142643064s) [noise=true , address=145.79.227.24:4582 , piece_id=MCQWT2LFD472TQDOAHLGQ5L2X7S654P6TL4J5MSP3CML5ZCURHMA, seg_pos={0 0}, upload=36107]: Long tail cancellation due to optimal threshold Dec 5 09:56:56.671496377 (41h1m33.142707106s) [noise=true , address=145.79.227.44:4193 , piece_id=327UUYIASLKFGUIQVY4P5UML4R2R4ODLUAYXR3GISKIT6CFDUGAQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: write tcp 188.166.225.170:39276->145.79.227.44:4193: use of closed network connection; write tcp 188.166.225.170:39276->145.79.227.44:4193: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39276->145.79.227.44:4193: use of closed network connection Dec 5 09:56:56.671501627 (41h1m33.142712366s) [noise=true , address=145.79.227.44:4193 , piece_id=327UUYIASLKFGUIQVY4P5UML4R2R4ODLUAYXR3GISKIT6CFDUGAQ, seg_pos={0 0}, upload=36107]: Long tail cancellation due to optimal threshold Dec 5 09:56:56.671516466 (41h1m33.142727195s) [noise=true , address=77.90.31.16:4735 , piece_id=V4YJIHVMXYTBK7IHU272RYCKUXI44B2MXRLETMWMO2WAI2XRU4SQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: write tcp 188.166.225.170:41866->77.90.31.16:4735: use of closed network connection; storage node overloaded, request limit: 2000 Dec 5 09:56:56.671525944 (41h1m33.142736673s) [noise=true , address=77.90.31.16:4735 , piece_id=V4YJIHVMXYTBK7IHU272RYCKUXI44B2MXRLETMWMO2WAI2XRU4SQ, seg_pos={0 0}, upload=36107]: Long tail cancellation due to optimal threshold Dec 5 09:56:56.671548016 (41h1m33.142758745s) [seg_pos={0 0}, upload=36107]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 5 09:56:56.671623500 (41h1m33.142834229s) [upload=36107]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 5 09:56:56.671631204 (41h1m33.142841933s) [upload=36107]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 5 09:56:57.112987571 (41h1m33.5841983s) [upload=36107]: Done uploading segments... Dec 5 09:56:57.113007228 (41h1m33.584217967s) [upload=36107]: Upload finished. err: <nil> Duration: 9.152948108s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 10.495303702s trace: https://jaeger.dev.storj.io/trace/55b376cc76cfe920 FAIL (2025-12-05 10:29:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 9 (41h1m29.913335078s) [upload=36107]: Uploading segments... Dec 5 09:56:53.442159225 (41h1m29.913369955s) [upload=36107]: Got next segment. Inline: false Dec 5 09:56:53.442188120 (41h1m29.913398849s) [upload=36107]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 5 09:56:53.442194372 (41h1m29.913405101s) [upload=36107]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 5 09:56:53.701976567 (41h1m30.173187296s) [seg_pos={0 0}, upload=36107]: Begin upload segment... Dec 5 09:56:53.702020460 (41h1m30.173231189s) [seg_pos={0 0}, upload=36107]: Done begin upload segment. Dec 5 09:56:53.702056649 (41h1m30.173267388s) [noise=true , address=77.90.31.21:4878 , piece_id=RCFTETF7SGR6FTUBJWWBJDKOZIHZKBMSUBZYVWYKF2MP56PMIWMQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702064965 (41h1m30.173275694s) [noise=true , address=145.79.227.28:4725 , piece_id=ZE642TJU4ZJKBOHQ6K6GPLPTN2YKC4BOP325LN7QXALZSDPG66HQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702111533 (41h1m30.173322272s) [noise=true , address=145.79.227.42:4121 , piece_id=6D47MY4GTQ3YQOGHS2V2OTO2VFDL6XKKS2R4VA6VZZ557WYFY34A, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702548986 (41h1m30.173759766s) [noise=true , address=145.79.227.27:4691 , piece_id=Z53YBOXTW4L23P4M3EOW6ADUHDWHIJVRDT7IL4I5FUPPZOW2MRLQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702931455 (41h1m30.174142194s) [noise=true , address=77.90.31.16:4735 , piece_id=V4YJIHVMXYTBK7IHU272RYCKUXI44B2MXRLETMWMO2WAI2XRU4SQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.703301079 (41h1m30.174511808s) [noise=true , address=145.79.227.10:4401 , piece_id=XAVSV3SGPDDLZWGKQLHA2CVYTGOPO7J3YVEO5C3MYIN5SJEUZ2OQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.703633000 (41h1m30.174843729s) [noise=true , address=145.79.227.24:4583 , piece_id=IB6T3J3CF3P3A54JEHHBK6KEPMUEI32B4QYSSA2G2BUMIVUC7DGQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.703957648 (41h1m30.175168377s) [noise=true , address=145.79.227.26:4654 , piece_id=23NNREG47WSZWJNPZWP275L62YBBSTB6RQWTINIRTTAYU66DRT7Q, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.704788621 (41h1m30.17599936s) [noise=true , address=145.79.227.43:4159 , piece_id=S5T27VFA6UNNONPZBZILNTQUUNSRLWPREKOJORMLKOXWNL3GMBSQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.705457525 (41h1m30.176668264s) [noise=true , address=145.79.227.32:4835 , piece_id=PJURUANT2XJY5PNYTSO5W2EDT3HYFAC2BFNCANKOAZ3I6P2GRC3Q, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.706486825 (41h1m30.177697554s) [noise=true , address=145.79.227.24:4582 , piece_id=MCQWT2LFD472TQDOAHLGQ5L2X7S654P6TL4J5MSP3CML5ZCURHMA, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.707626675 (41h1m30.178837414s) [noise=true , address=145.79.227.44:4193 , piece_id=327UUYIASLKFGUIQVY4P5UML4R2R4ODLUAYXR3GISKIT6CFDUGAQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.708631078 (41h1m30.179841817s) [noise=true , address=145.79.227.40:4050 , piece_id=QFSFC3WFER7VBVZNB3XLRIEFWGM4Q5VTAAJIWG4HAERKXSHZGZ4Q, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.709908630 (41h1m30.181119359s) [noise=true , address=145.79.227.22:4510 , piece_id=HCJIUH3VECRBR744KIQ65UKSMTEEKUPHJWXSDBM5SEMUVBJ5YX6A, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.710331386 (41h1m30.181542125s) [noise=true , address=145.79.227.41:4087 , piece_id=6GY2NC7F26VLU5NEN65OYU7YIHJYSTI5GNVRTNXM5HO5XQMZKEPA, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.710803685 (41h1m30.182014424s) [noise=true , address=45.91.116.4:4937 , piece_id=VFAPMIAB2G2BBOMNSM5EXONUL5BNORMR7KXIYSTQ7UFRVASVTQHQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:55.513663514 (41h1m31.984874243s) [upload=36107]: Next returned nil segment Dec 5 09:56:55.513682680 (41h1m31.98489341s) [upload=36107]: Waiting for error group managing segments... Dec 5 09:56:55.808504186 (41h1m32.279714915s) [noise=true , address=77.90.31.21:4878 , piece_id=RCFTETF7SGR6FTUBJWWBJDKOZIHZKBMSUBZYVWYKF2MP56PMIWMQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.866933347 (41h1m32.338144086s) [noise=true , address=145.79.227.10:4401 , piece_id=XAVSV3SGPDDLZWGKQLHA2CVYTGOPO7J3YVEO5C3MYIN5SJEUZ2OQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.869510454 (41h1m32.340721183s) [noise=true , address=145.79.227.28:4725 , piece_id=ZE642TJU4ZJKBOHQ6K6GPLPTN2YKC4BOP325LN7QXALZSDPG66HQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.875082795 (41h1m32.346293524s) [noise=true , address=145.79.227.43:4159 , piece_id=S5T27VFA6UNNONPZBZILNTQUUNSRLWPREKOJORMLKOXWNL3GMBSQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.884542648 (41h1m32.355753387s) [noise=true , address=145.79.227.24:4583 , piece_id=IB6T3J3CF3P3A54JEHHBK6KEPMUEI32B4QYSSA2G2BUMIVUC7DGQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.888876321 (41h1m32.36008707s) [noise=true , address=145.79.227.40:4050 , piece_id=QFSFC3WFER7VBVZNB3XLRIEFWGM4Q5VTAAJIWG4HAERKXSHZGZ4Q, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.889468719 (41h1m32.360679448s) [noise=true , address=145.79.227.26:4654 , piece_id=23NNREG47WSZWJNPZWP275L62YBBSTB6RQWTINIRTTAYU66DRT7Q, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.900234388 (41h1m32.371445117s) [noise=true , address=45.91.116.4:4937 , piece_id=VFAPMIAB2G2BBOMNSM5EXONUL5BNORMR7KXIYSTQ7UFRVASVTQHQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.923941432 (41h1m32.395152161s) [noise=true , address=145.79.227.42:4121 , piece_id=6D47MY4GTQ3YQOGHS2V2OTO2VFDL6XKKS2R4VA6VZZ557WYFY34A, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.929131404 (41h1m32.400342133s) [noise=true , address=145.79.227.22:4510 , piece_id=HCJIUH3VECRBR744KIQ65UKSMTEEKUPHJWXSDBM5SEMUVBJ5YX6A, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.180555093 (41h1m32.651765822s) [noise=true , address=145.79.227.32:4835 , piece_id=PJURUANT2XJY5PNYTSO5W2EDT3HYFAC2BFNCANKOAZ3I6P2GRC3Q, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.587529048 (41h1m33.058739787s) [noise=true , address=145.79.227.41:4087 , piece_id=6GY2NC7F26VLU5NEN65OYU7YIHJYSTI5GNVRTNXM5HO5XQMZKEPA, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.671173874 (41h1m33.142384603s) [noise=true , address=145.79.227.27:4691 , piece_id=Z53YBOXTW4L23P4M3EOW6ADUHDWHIJVRDT7IL4I5FUPPZOW2MRLQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.671209982 (41h1m33.142420711s) [seg_pos={0 0}, upload=36107]: Segment reached optimal threshold of 13 pieces. Dec 5 09:56:56.671423689 (41h1m33.142634428s) [noise=true , address=145.79.227.24:4582 , piece_id=MCQWT2LFD472TQDOAHLGQ5L2X7S654P6TL4J5MSP3CML5ZCURHMA, seg_pos={0 0}, upload=36107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa): protocol: write tcp 188.166.225.170:54136->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:54136->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54136->145.79.227.24:4582: use of closed network connection Dec 5 09:56:56.671432325 (41h1m33.142643064s) [noise=true , address=145.79.227.24:4582 , piece_id=MCQWT2LFD472TQDOAHLGQ5L2X7S654P6TL4J5MSP3CML5ZCURHMA, seg_pos={0 0}, upload=36107]: Long tail cancellation due to optimal threshold Dec 5 09:56:56.671496377 (41h1m33.142707106s) [noise=true , address=145.79.227.44:4193 , piece_id=327UUYIASLKFGUIQVY4P5UML4R2R4ODLUAYXR3GISKIT6CFDUGAQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: write tcp 188.166.225.170:39276->145.79.227.44:4193: use of closed network connection; write tcp 188.166.225.170:39276->145.79.227.44:4193: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39276->145.79.227.44:4193: use of closed network connection Dec 5 09:56:56.671501627 (41h1m33.142712366s) [noise=true , address=145.79.227.44:4193 , piece_id=327UUYIASLKFGUIQVY4P5UML4R2R4ODLUAYXR3GISKIT6CFDUGAQ, seg_pos={0 0}, upload=36107]: Long tail cancellation due to optimal threshold Dec 5 09:56:56.671516466 (41h1m33.142727195s) [noise=true , address=77.90.31.16:4735 , piece_id=V4YJIHVMXYTBK7IHU272RYCKUXI44B2MXRLETMWMO2WAI2XRU4SQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: write tcp 188.166.225.170:41866->77.90.31.16:4735: use of closed network connection; storage node overloaded, request limit: 2000 Dec 5 09:56:56.671525944 (41h1m33.142736673s) [noise=true , address=77.90.31.16:4735 , piece_id=V4YJIHVMXYTBK7IHU272RYCKUXI44B2MXRLETMWMO2WAI2XRU4SQ, seg_pos={0 0}, upload=36107]: Long tail cancellation due to optimal threshold Dec 5 09:56:56.671548016 (41h1m33.142758745s) [seg_pos={0 0}, upload=36107]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 5 09:56:56.671623500 (41h1m33.142834229s) [upload=36107]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 5 09:56:56.671631204 (41h1m33.142841933s) [upload=36107]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 5 09:56:57.112987571 (41h1m33.5841983s) [upload=36107]: Done uploading segments... Dec 5 09:56:57.113007228 (41h1m33.584217967s) [upload=36107]: Upload finished. err: <nil> Duration: 10.495333958s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 8.468361993s trace: https://jaeger.dev.storj.io/trace/55b376cc76cfe920 FAIL (2025-12-05 10:28:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 9 (41h1m29.913335078s) [upload=36107]: Uploading segments... Dec 5 09:56:53.442159225 (41h1m29.913369955s) [upload=36107]: Got next segment. Inline: false Dec 5 09:56:53.442188120 (41h1m29.913398849s) [upload=36107]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 5 09:56:53.442194372 (41h1m29.913405101s) [upload=36107]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 5 09:56:53.701976567 (41h1m30.173187296s) [seg_pos={0 0}, upload=36107]: Begin upload segment... Dec 5 09:56:53.702020460 (41h1m30.173231189s) [seg_pos={0 0}, upload=36107]: Done begin upload segment. Dec 5 09:56:53.702056649 (41h1m30.173267388s) [noise=true , address=77.90.31.21:4878 , piece_id=RCFTETF7SGR6FTUBJWWBJDKOZIHZKBMSUBZYVWYKF2MP56PMIWMQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702064965 (41h1m30.173275694s) [noise=true , address=145.79.227.28:4725 , piece_id=ZE642TJU4ZJKBOHQ6K6GPLPTN2YKC4BOP325LN7QXALZSDPG66HQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702111533 (41h1m30.173322272s) [noise=true , address=145.79.227.42:4121 , piece_id=6D47MY4GTQ3YQOGHS2V2OTO2VFDL6XKKS2R4VA6VZZ557WYFY34A, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702548986 (41h1m30.173759766s) [noise=true , address=145.79.227.27:4691 , piece_id=Z53YBOXTW4L23P4M3EOW6ADUHDWHIJVRDT7IL4I5FUPPZOW2MRLQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702931455 (41h1m30.174142194s) [noise=true , address=77.90.31.16:4735 , piece_id=V4YJIHVMXYTBK7IHU272RYCKUXI44B2MXRLETMWMO2WAI2XRU4SQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.703301079 (41h1m30.174511808s) [noise=true , address=145.79.227.10:4401 , piece_id=XAVSV3SGPDDLZWGKQLHA2CVYTGOPO7J3YVEO5C3MYIN5SJEUZ2OQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.703633000 (41h1m30.174843729s) [noise=true , address=145.79.227.24:4583 , piece_id=IB6T3J3CF3P3A54JEHHBK6KEPMUEI32B4QYSSA2G2BUMIVUC7DGQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.703957648 (41h1m30.175168377s) [noise=true , address=145.79.227.26:4654 , piece_id=23NNREG47WSZWJNPZWP275L62YBBSTB6RQWTINIRTTAYU66DRT7Q, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.704788621 (41h1m30.17599936s) [noise=true , address=145.79.227.43:4159 , piece_id=S5T27VFA6UNNONPZBZILNTQUUNSRLWPREKOJORMLKOXWNL3GMBSQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.705457525 (41h1m30.176668264s) [noise=true , address=145.79.227.32:4835 , piece_id=PJURUANT2XJY5PNYTSO5W2EDT3HYFAC2BFNCANKOAZ3I6P2GRC3Q, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.706486825 (41h1m30.177697554s) [noise=true , address=145.79.227.24:4582 , piece_id=MCQWT2LFD472TQDOAHLGQ5L2X7S654P6TL4J5MSP3CML5ZCURHMA, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.707626675 (41h1m30.178837414s) [noise=true , address=145.79.227.44:4193 , piece_id=327UUYIASLKFGUIQVY4P5UML4R2R4ODLUAYXR3GISKIT6CFDUGAQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.708631078 (41h1m30.179841817s) [noise=true , address=145.79.227.40:4050 , piece_id=QFSFC3WFER7VBVZNB3XLRIEFWGM4Q5VTAAJIWG4HAERKXSHZGZ4Q, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.709908630 (41h1m30.181119359s) [noise=true , address=145.79.227.22:4510 , piece_id=HCJIUH3VECRBR744KIQ65UKSMTEEKUPHJWXSDBM5SEMUVBJ5YX6A, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.710331386 (41h1m30.181542125s) [noise=true , address=145.79.227.41:4087 , piece_id=6GY2NC7F26VLU5NEN65OYU7YIHJYSTI5GNVRTNXM5HO5XQMZKEPA, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.710803685 (41h1m30.182014424s) [noise=true , address=45.91.116.4:4937 , piece_id=VFAPMIAB2G2BBOMNSM5EXONUL5BNORMR7KXIYSTQ7UFRVASVTQHQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:55.513663514 (41h1m31.984874243s) [upload=36107]: Next returned nil segment Dec 5 09:56:55.513682680 (41h1m31.98489341s) [upload=36107]: Waiting for error group managing segments... Dec 5 09:56:55.808504186 (41h1m32.279714915s) [noise=true , address=77.90.31.21:4878 , piece_id=RCFTETF7SGR6FTUBJWWBJDKOZIHZKBMSUBZYVWYKF2MP56PMIWMQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.866933347 (41h1m32.338144086s) [noise=true , address=145.79.227.10:4401 , piece_id=XAVSV3SGPDDLZWGKQLHA2CVYTGOPO7J3YVEO5C3MYIN5SJEUZ2OQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.869510454 (41h1m32.340721183s) [noise=true , address=145.79.227.28:4725 , piece_id=ZE642TJU4ZJKBOHQ6K6GPLPTN2YKC4BOP325LN7QXALZSDPG66HQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.875082795 (41h1m32.346293524s) [noise=true , address=145.79.227.43:4159 , piece_id=S5T27VFA6UNNONPZBZILNTQUUNSRLWPREKOJORMLKOXWNL3GMBSQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.884542648 (41h1m32.355753387s) [noise=true , address=145.79.227.24:4583 , piece_id=IB6T3J3CF3P3A54JEHHBK6KEPMUEI32B4QYSSA2G2BUMIVUC7DGQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.888876321 (41h1m32.36008707s) [noise=true , address=145.79.227.40:4050 , piece_id=QFSFC3WFER7VBVZNB3XLRIEFWGM4Q5VTAAJIWG4HAERKXSHZGZ4Q, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.889468719 (41h1m32.360679448s) [noise=true , address=145.79.227.26:4654 , piece_id=23NNREG47WSZWJNPZWP275L62YBBSTB6RQWTINIRTTAYU66DRT7Q, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.900234388 (41h1m32.371445117s) [noise=true , address=45.91.116.4:4937 , piece_id=VFAPMIAB2G2BBOMNSM5EXONUL5BNORMR7KXIYSTQ7UFRVASVTQHQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.923941432 (41h1m32.395152161s) [noise=true , address=145.79.227.42:4121 , piece_id=6D47MY4GTQ3YQOGHS2V2OTO2VFDL6XKKS2R4VA6VZZ557WYFY34A, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.929131404 (41h1m32.400342133s) [noise=true , address=145.79.227.22:4510 , piece_id=HCJIUH3VECRBR744KIQ65UKSMTEEKUPHJWXSDBM5SEMUVBJ5YX6A, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.180555093 (41h1m32.651765822s) [noise=true , address=145.79.227.32:4835 , piece_id=PJURUANT2XJY5PNYTSO5W2EDT3HYFAC2BFNCANKOAZ3I6P2GRC3Q, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.587529048 (41h1m33.058739787s) [noise=true , address=145.79.227.41:4087 , piece_id=6GY2NC7F26VLU5NEN65OYU7YIHJYSTI5GNVRTNXM5HO5XQMZKEPA, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.671173874 (41h1m33.142384603s) [noise=true , address=145.79.227.27:4691 , piece_id=Z53YBOXTW4L23P4M3EOW6ADUHDWHIJVRDT7IL4I5FUPPZOW2MRLQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.671209982 (41h1m33.142420711s) [seg_pos={0 0}, upload=36107]: Segment reached optimal threshold of 13 pieces. Dec 5 09:56:56.671423689 (41h1m33.142634428s) [noise=true , address=145.79.227.24:4582 , piece_id=MCQWT2LFD472TQDOAHLGQ5L2X7S654P6TL4J5MSP3CML5ZCURHMA, seg_pos={0 0}, upload=36107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa): protocol: write tcp 188.166.225.170:54136->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:54136->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54136->145.79.227.24:4582: use of closed network connection Dec 5 09:56:56.671432325 (41h1m33.142643064s) [noise=true , address=145.79.227.24:4582 , piece_id=MCQWT2LFD472TQDOAHLGQ5L2X7S654P6TL4J5MSP3CML5ZCURHMA, seg_pos={0 0}, upload=36107]: Long tail cancellation due to optimal threshold Dec 5 09:56:56.671496377 (41h1m33.142707106s) [noise=true , address=145.79.227.44:4193 , piece_id=327UUYIASLKFGUIQVY4P5UML4R2R4ODLUAYXR3GISKIT6CFDUGAQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: write tcp 188.166.225.170:39276->145.79.227.44:4193: use of closed network connection; write tcp 188.166.225.170:39276->145.79.227.44:4193: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39276->145.79.227.44:4193: use of closed network connection Dec 5 09:56:56.671501627 (41h1m33.142712366s) [noise=true , address=145.79.227.44:4193 , piece_id=327UUYIASLKFGUIQVY4P5UML4R2R4ODLUAYXR3GISKIT6CFDUGAQ, seg_pos={0 0}, upload=36107]: Long tail cancellation due to optimal threshold Dec 5 09:56:56.671516466 (41h1m33.142727195s) [noise=true , address=77.90.31.16:4735 , piece_id=V4YJIHVMXYTBK7IHU272RYCKUXI44B2MXRLETMWMO2WAI2XRU4SQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: write tcp 188.166.225.170:41866->77.90.31.16:4735: use of closed network connection; storage node overloaded, request limit: 2000 Dec 5 09:56:56.671525944 (41h1m33.142736673s) [noise=true , address=77.90.31.16:4735 , piece_id=V4YJIHVMXYTBK7IHU272RYCKUXI44B2MXRLETMWMO2WAI2XRU4SQ, seg_pos={0 0}, upload=36107]: Long tail cancellation due to optimal threshold Dec 5 09:56:56.671548016 (41h1m33.142758745s) [seg_pos={0 0}, upload=36107]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 5 09:56:56.671623500 (41h1m33.142834229s) [upload=36107]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 5 09:56:56.671631204 (41h1m33.142841933s) [upload=36107]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 5 09:56:57.112987571 (41h1m33.5841983s) [upload=36107]: Done uploading segments... Dec 5 09:56:57.113007228 (41h1m33.584217967s) [upload=36107]: Upload finished. err: <nil> Duration: 8.468520584s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 8.232134715s trace: https://jaeger.dev.storj.io/trace/55b376cc76cfe920 FAIL (2025-12-05 10:27:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 9 (41h1m29.913335078s) [upload=36107]: Uploading segments... Dec 5 09:56:53.442159225 (41h1m29.913369955s) [upload=36107]: Got next segment. Inline: false Dec 5 09:56:53.442188120 (41h1m29.913398849s) [upload=36107]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 5 09:56:53.442194372 (41h1m29.913405101s) [upload=36107]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 5 09:56:53.701976567 (41h1m30.173187296s) [seg_pos={0 0}, upload=36107]: Begin upload segment... Dec 5 09:56:53.702020460 (41h1m30.173231189s) [seg_pos={0 0}, upload=36107]: Done begin upload segment. Dec 5 09:56:53.702056649 (41h1m30.173267388s) [noise=true , address=77.90.31.21:4878 , piece_id=RCFTETF7SGR6FTUBJWWBJDKOZIHZKBMSUBZYVWYKF2MP56PMIWMQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702064965 (41h1m30.173275694s) [noise=true , address=145.79.227.28:4725 , piece_id=ZE642TJU4ZJKBOHQ6K6GPLPTN2YKC4BOP325LN7QXALZSDPG66HQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702111533 (41h1m30.173322272s) [noise=true , address=145.79.227.42:4121 , piece_id=6D47MY4GTQ3YQOGHS2V2OTO2VFDL6XKKS2R4VA6VZZ557WYFY34A, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702548986 (41h1m30.173759766s) [noise=true , address=145.79.227.27:4691 , piece_id=Z53YBOXTW4L23P4M3EOW6ADUHDWHIJVRDT7IL4I5FUPPZOW2MRLQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702931455 (41h1m30.174142194s) [noise=true , address=77.90.31.16:4735 , piece_id=V4YJIHVMXYTBK7IHU272RYCKUXI44B2MXRLETMWMO2WAI2XRU4SQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.703301079 (41h1m30.174511808s) [noise=true , address=145.79.227.10:4401 , piece_id=XAVSV3SGPDDLZWGKQLHA2CVYTGOPO7J3YVEO5C3MYIN5SJEUZ2OQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.703633000 (41h1m30.174843729s) [noise=true , address=145.79.227.24:4583 , piece_id=IB6T3J3CF3P3A54JEHHBK6KEPMUEI32B4QYSSA2G2BUMIVUC7DGQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.703957648 (41h1m30.175168377s) [noise=true , address=145.79.227.26:4654 , piece_id=23NNREG47WSZWJNPZWP275L62YBBSTB6RQWTINIRTTAYU66DRT7Q, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.704788621 (41h1m30.17599936s) [noise=true , address=145.79.227.43:4159 , piece_id=S5T27VFA6UNNONPZBZILNTQUUNSRLWPREKOJORMLKOXWNL3GMBSQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.705457525 (41h1m30.176668264s) [noise=true , address=145.79.227.32:4835 , piece_id=PJURUANT2XJY5PNYTSO5W2EDT3HYFAC2BFNCANKOAZ3I6P2GRC3Q, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.706486825 (41h1m30.177697554s) [noise=true , address=145.79.227.24:4582 , piece_id=MCQWT2LFD472TQDOAHLGQ5L2X7S654P6TL4J5MSP3CML5ZCURHMA, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.707626675 (41h1m30.178837414s) [noise=true , address=145.79.227.44:4193 , piece_id=327UUYIASLKFGUIQVY4P5UML4R2R4ODLUAYXR3GISKIT6CFDUGAQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.708631078 (41h1m30.179841817s) [noise=true , address=145.79.227.40:4050 , piece_id=QFSFC3WFER7VBVZNB3XLRIEFWGM4Q5VTAAJIWG4HAERKXSHZGZ4Q, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.709908630 (41h1m30.181119359s) [noise=true , address=145.79.227.22:4510 , piece_id=HCJIUH3VECRBR744KIQ65UKSMTEEKUPHJWXSDBM5SEMUVBJ5YX6A, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.710331386 (41h1m30.181542125s) [noise=true , address=145.79.227.41:4087 , piece_id=6GY2NC7F26VLU5NEN65OYU7YIHJYSTI5GNVRTNXM5HO5XQMZKEPA, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.710803685 (41h1m30.182014424s) [noise=true , address=45.91.116.4:4937 , piece_id=VFAPMIAB2G2BBOMNSM5EXONUL5BNORMR7KXIYSTQ7UFRVASVTQHQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:55.513663514 (41h1m31.984874243s) [upload=36107]: Next returned nil segment Dec 5 09:56:55.513682680 (41h1m31.98489341s) [upload=36107]: Waiting for error group managing segments... Dec 5 09:56:55.808504186 (41h1m32.279714915s) [noise=true , address=77.90.31.21:4878 , piece_id=RCFTETF7SGR6FTUBJWWBJDKOZIHZKBMSUBZYVWYKF2MP56PMIWMQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.866933347 (41h1m32.338144086s) [noise=true , address=145.79.227.10:4401 , piece_id=XAVSV3SGPDDLZWGKQLHA2CVYTGOPO7J3YVEO5C3MYIN5SJEUZ2OQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.869510454 (41h1m32.340721183s) [noise=true , address=145.79.227.28:4725 , piece_id=ZE642TJU4ZJKBOHQ6K6GPLPTN2YKC4BOP325LN7QXALZSDPG66HQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.875082795 (41h1m32.346293524s) [noise=true , address=145.79.227.43:4159 , piece_id=S5T27VFA6UNNONPZBZILNTQUUNSRLWPREKOJORMLKOXWNL3GMBSQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.884542648 (41h1m32.355753387s) [noise=true , address=145.79.227.24:4583 , piece_id=IB6T3J3CF3P3A54JEHHBK6KEPMUEI32B4QYSSA2G2BUMIVUC7DGQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.888876321 (41h1m32.36008707s) [noise=true , address=145.79.227.40:4050 , piece_id=QFSFC3WFER7VBVZNB3XLRIEFWGM4Q5VTAAJIWG4HAERKXSHZGZ4Q, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.889468719 (41h1m32.360679448s) [noise=true , address=145.79.227.26:4654 , piece_id=23NNREG47WSZWJNPZWP275L62YBBSTB6RQWTINIRTTAYU66DRT7Q, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.900234388 (41h1m32.371445117s) [noise=true , address=45.91.116.4:4937 , piece_id=VFAPMIAB2G2BBOMNSM5EXONUL5BNORMR7KXIYSTQ7UFRVASVTQHQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.923941432 (41h1m32.395152161s) [noise=true , address=145.79.227.42:4121 , piece_id=6D47MY4GTQ3YQOGHS2V2OTO2VFDL6XKKS2R4VA6VZZ557WYFY34A, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.929131404 (41h1m32.400342133s) [noise=true , address=145.79.227.22:4510 , piece_id=HCJIUH3VECRBR744KIQ65UKSMTEEKUPHJWXSDBM5SEMUVBJ5YX6A, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.180555093 (41h1m32.651765822s) [noise=true , address=145.79.227.32:4835 , piece_id=PJURUANT2XJY5PNYTSO5W2EDT3HYFAC2BFNCANKOAZ3I6P2GRC3Q, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.587529048 (41h1m33.058739787s) [noise=true , address=145.79.227.41:4087 , piece_id=6GY2NC7F26VLU5NEN65OYU7YIHJYSTI5GNVRTNXM5HO5XQMZKEPA, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.671173874 (41h1m33.142384603s) [noise=true , address=145.79.227.27:4691 , piece_id=Z53YBOXTW4L23P4M3EOW6ADUHDWHIJVRDT7IL4I5FUPPZOW2MRLQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.671209982 (41h1m33.142420711s) [seg_pos={0 0}, upload=36107]: Segment reached optimal threshold of 13 pieces. Dec 5 09:56:56.671423689 (41h1m33.142634428s) [noise=true , address=145.79.227.24:4582 , piece_id=MCQWT2LFD472TQDOAHLGQ5L2X7S654P6TL4J5MSP3CML5ZCURHMA, seg_pos={0 0}, upload=36107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa): protocol: write tcp 188.166.225.170:54136->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:54136->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54136->145.79.227.24:4582: use of closed network connection Dec 5 09:56:56.671432325 (41h1m33.142643064s) [noise=true , address=145.79.227.24:4582 , piece_id=MCQWT2LFD472TQDOAHLGQ5L2X7S654P6TL4J5MSP3CML5ZCURHMA, seg_pos={0 0}, upload=36107]: Long tail cancellation due to optimal threshold Dec 5 09:56:56.671496377 (41h1m33.142707106s) [noise=true , address=145.79.227.44:4193 , piece_id=327UUYIASLKFGUIQVY4P5UML4R2R4ODLUAYXR3GISKIT6CFDUGAQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: write tcp 188.166.225.170:39276->145.79.227.44:4193: use of closed network connection; write tcp 188.166.225.170:39276->145.79.227.44:4193: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39276->145.79.227.44:4193: use of closed network connection Dec 5 09:56:56.671501627 (41h1m33.142712366s) [noise=true , address=145.79.227.44:4193 , piece_id=327UUYIASLKFGUIQVY4P5UML4R2R4ODLUAYXR3GISKIT6CFDUGAQ, seg_pos={0 0}, upload=36107]: Long tail cancellation due to optimal threshold Dec 5 09:56:56.671516466 (41h1m33.142727195s) [noise=true , address=77.90.31.16:4735 , piece_id=V4YJIHVMXYTBK7IHU272RYCKUXI44B2MXRLETMWMO2WAI2XRU4SQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: write tcp 188.166.225.170:41866->77.90.31.16:4735: use of closed network connection; storage node overloaded, request limit: 2000 Dec 5 09:56:56.671525944 (41h1m33.142736673s) [noise=true , address=77.90.31.16:4735 , piece_id=V4YJIHVMXYTBK7IHU272RYCKUXI44B2MXRLETMWMO2WAI2XRU4SQ, seg_pos={0 0}, upload=36107]: Long tail cancellation due to optimal threshold Dec 5 09:56:56.671548016 (41h1m33.142758745s) [seg_pos={0 0}, upload=36107]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 5 09:56:56.671623500 (41h1m33.142834229s) [upload=36107]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 5 09:56:56.671631204 (41h1m33.142841933s) [upload=36107]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 5 09:56:57.112987571 (41h1m33.5841983s) [upload=36107]: Done uploading segments... Dec 5 09:56:57.113007228 (41h1m33.584217967s) [upload=36107]: Upload finished. err: <nil> Duration: 8.232171635s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 9.429247511s trace: https://jaeger.dev.storj.io/trace/55b376cc76cfe920 FAIL (2025-12-05 10:26:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 9 (41h1m29.913335078s) [upload=36107]: Uploading segments... Dec 5 09:56:53.442159225 (41h1m29.913369955s) [upload=36107]: Got next segment. Inline: false Dec 5 09:56:53.442188120 (41h1m29.913398849s) [upload=36107]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 5 09:56:53.442194372 (41h1m29.913405101s) [upload=36107]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 5 09:56:53.701976567 (41h1m30.173187296s) [seg_pos={0 0}, upload=36107]: Begin upload segment... Dec 5 09:56:53.702020460 (41h1m30.173231189s) [seg_pos={0 0}, upload=36107]: Done begin upload segment. Dec 5 09:56:53.702056649 (41h1m30.173267388s) [noise=true , address=77.90.31.21:4878 , piece_id=RCFTETF7SGR6FTUBJWWBJDKOZIHZKBMSUBZYVWYKF2MP56PMIWMQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702064965 (41h1m30.173275694s) [noise=true , address=145.79.227.28:4725 , piece_id=ZE642TJU4ZJKBOHQ6K6GPLPTN2YKC4BOP325LN7QXALZSDPG66HQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702111533 (41h1m30.173322272s) [noise=true , address=145.79.227.42:4121 , piece_id=6D47MY4GTQ3YQOGHS2V2OTO2VFDL6XKKS2R4VA6VZZ557WYFY34A, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702548986 (41h1m30.173759766s) [noise=true , address=145.79.227.27:4691 , piece_id=Z53YBOXTW4L23P4M3EOW6ADUHDWHIJVRDT7IL4I5FUPPZOW2MRLQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702931455 (41h1m30.174142194s) [noise=true , address=77.90.31.16:4735 , piece_id=V4YJIHVMXYTBK7IHU272RYCKUXI44B2MXRLETMWMO2WAI2XRU4SQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.703301079 (41h1m30.174511808s) [noise=true , address=145.79.227.10:4401 , piece_id=XAVSV3SGPDDLZWGKQLHA2CVYTGOPO7J3YVEO5C3MYIN5SJEUZ2OQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.703633000 (41h1m30.174843729s) [noise=true , address=145.79.227.24:4583 , piece_id=IB6T3J3CF3P3A54JEHHBK6KEPMUEI32B4QYSSA2G2BUMIVUC7DGQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.703957648 (41h1m30.175168377s) [noise=true , address=145.79.227.26:4654 , piece_id=23NNREG47WSZWJNPZWP275L62YBBSTB6RQWTINIRTTAYU66DRT7Q, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.704788621 (41h1m30.17599936s) [noise=true , address=145.79.227.43:4159 , piece_id=S5T27VFA6UNNONPZBZILNTQUUNSRLWPREKOJORMLKOXWNL3GMBSQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.705457525 (41h1m30.176668264s) [noise=true , address=145.79.227.32:4835 , piece_id=PJURUANT2XJY5PNYTSO5W2EDT3HYFAC2BFNCANKOAZ3I6P2GRC3Q, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.706486825 (41h1m30.177697554s) [noise=true , address=145.79.227.24:4582 , piece_id=MCQWT2LFD472TQDOAHLGQ5L2X7S654P6TL4J5MSP3CML5ZCURHMA, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.707626675 (41h1m30.178837414s) [noise=true , address=145.79.227.44:4193 , piece_id=327UUYIASLKFGUIQVY4P5UML4R2R4ODLUAYXR3GISKIT6CFDUGAQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.708631078 (41h1m30.179841817s) [noise=true , address=145.79.227.40:4050 , piece_id=QFSFC3WFER7VBVZNB3XLRIEFWGM4Q5VTAAJIWG4HAERKXSHZGZ4Q, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.709908630 (41h1m30.181119359s) [noise=true , address=145.79.227.22:4510 , piece_id=HCJIUH3VECRBR744KIQ65UKSMTEEKUPHJWXSDBM5SEMUVBJ5YX6A, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.710331386 (41h1m30.181542125s) [noise=true , address=145.79.227.41:4087 , piece_id=6GY2NC7F26VLU5NEN65OYU7YIHJYSTI5GNVRTNXM5HO5XQMZKEPA, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.710803685 (41h1m30.182014424s) [noise=true , address=45.91.116.4:4937 , piece_id=VFAPMIAB2G2BBOMNSM5EXONUL5BNORMR7KXIYSTQ7UFRVASVTQHQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:55.513663514 (41h1m31.984874243s) [upload=36107]: Next returned nil segment Dec 5 09:56:55.513682680 (41h1m31.98489341s) [upload=36107]: Waiting for error group managing segments... Dec 5 09:56:55.808504186 (41h1m32.279714915s) [noise=true , address=77.90.31.21:4878 , piece_id=RCFTETF7SGR6FTUBJWWBJDKOZIHZKBMSUBZYVWYKF2MP56PMIWMQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.866933347 (41h1m32.338144086s) [noise=true , address=145.79.227.10:4401 , piece_id=XAVSV3SGPDDLZWGKQLHA2CVYTGOPO7J3YVEO5C3MYIN5SJEUZ2OQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.869510454 (41h1m32.340721183s) [noise=true , address=145.79.227.28:4725 , piece_id=ZE642TJU4ZJKBOHQ6K6GPLPTN2YKC4BOP325LN7QXALZSDPG66HQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.875082795 (41h1m32.346293524s) [noise=true , address=145.79.227.43:4159 , piece_id=S5T27VFA6UNNONPZBZILNTQUUNSRLWPREKOJORMLKOXWNL3GMBSQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.884542648 (41h1m32.355753387s) [noise=true , address=145.79.227.24:4583 , piece_id=IB6T3J3CF3P3A54JEHHBK6KEPMUEI32B4QYSSA2G2BUMIVUC7DGQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.888876321 (41h1m32.36008707s) [noise=true , address=145.79.227.40:4050 , piece_id=QFSFC3WFER7VBVZNB3XLRIEFWGM4Q5VTAAJIWG4HAERKXSHZGZ4Q, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.889468719 (41h1m32.360679448s) [noise=true , address=145.79.227.26:4654 , piece_id=23NNREG47WSZWJNPZWP275L62YBBSTB6RQWTINIRTTAYU66DRT7Q, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.900234388 (41h1m32.371445117s) [noise=true , address=45.91.116.4:4937 , piece_id=VFAPMIAB2G2BBOMNSM5EXONUL5BNORMR7KXIYSTQ7UFRVASVTQHQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.923941432 (41h1m32.395152161s) [noise=true , address=145.79.227.42:4121 , piece_id=6D47MY4GTQ3YQOGHS2V2OTO2VFDL6XKKS2R4VA6VZZ557WYFY34A, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.929131404 (41h1m32.400342133s) [noise=true , address=145.79.227.22:4510 , piece_id=HCJIUH3VECRBR744KIQ65UKSMTEEKUPHJWXSDBM5SEMUVBJ5YX6A, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.180555093 (41h1m32.651765822s) [noise=true , address=145.79.227.32:4835 , piece_id=PJURUANT2XJY5PNYTSO5W2EDT3HYFAC2BFNCANKOAZ3I6P2GRC3Q, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.587529048 (41h1m33.058739787s) [noise=true , address=145.79.227.41:4087 , piece_id=6GY2NC7F26VLU5NEN65OYU7YIHJYSTI5GNVRTNXM5HO5XQMZKEPA, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.671173874 (41h1m33.142384603s) [noise=true , address=145.79.227.27:4691 , piece_id=Z53YBOXTW4L23P4M3EOW6ADUHDWHIJVRDT7IL4I5FUPPZOW2MRLQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.671209982 (41h1m33.142420711s) [seg_pos={0 0}, upload=36107]: Segment reached optimal threshold of 13 pieces. Dec 5 09:56:56.671423689 (41h1m33.142634428s) [noise=true , address=145.79.227.24:4582 , piece_id=MCQWT2LFD472TQDOAHLGQ5L2X7S654P6TL4J5MSP3CML5ZCURHMA, seg_pos={0 0}, upload=36107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa): protocol: write tcp 188.166.225.170:54136->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:54136->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54136->145.79.227.24:4582: use of closed network connection Dec 5 09:56:56.671432325 (41h1m33.142643064s) [noise=true , address=145.79.227.24:4582 , piece_id=MCQWT2LFD472TQDOAHLGQ5L2X7S654P6TL4J5MSP3CML5ZCURHMA, seg_pos={0 0}, upload=36107]: Long tail cancellation due to optimal threshold Dec 5 09:56:56.671496377 (41h1m33.142707106s) [noise=true , address=145.79.227.44:4193 , piece_id=327UUYIASLKFGUIQVY4P5UML4R2R4ODLUAYXR3GISKIT6CFDUGAQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: write tcp 188.166.225.170:39276->145.79.227.44:4193: use of closed network connection; write tcp 188.166.225.170:39276->145.79.227.44:4193: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39276->145.79.227.44:4193: use of closed network connection Dec 5 09:56:56.671501627 (41h1m33.142712366s) [noise=true , address=145.79.227.44:4193 , piece_id=327UUYIASLKFGUIQVY4P5UML4R2R4ODLUAYXR3GISKIT6CFDUGAQ, seg_pos={0 0}, upload=36107]: Long tail cancellation due to optimal threshold Dec 5 09:56:56.671516466 (41h1m33.142727195s) [noise=true , address=77.90.31.16:4735 , piece_id=V4YJIHVMXYTBK7IHU272RYCKUXI44B2MXRLETMWMO2WAI2XRU4SQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: write tcp 188.166.225.170:41866->77.90.31.16:4735: use of closed network connection; storage node overloaded, request limit: 2000 Dec 5 09:56:56.671525944 (41h1m33.142736673s) [noise=true , address=77.90.31.16:4735 , piece_id=V4YJIHVMXYTBK7IHU272RYCKUXI44B2MXRLETMWMO2WAI2XRU4SQ, seg_pos={0 0}, upload=36107]: Long tail cancellation due to optimal threshold Dec 5 09:56:56.671548016 (41h1m33.142758745s) [seg_pos={0 0}, upload=36107]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 5 09:56:56.671623500 (41h1m33.142834229s) [upload=36107]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 5 09:56:56.671631204 (41h1m33.142841933s) [upload=36107]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 5 09:56:57.112987571 (41h1m33.5841983s) [upload=36107]: Done uploading segments... Dec 5 09:56:57.113007228 (41h1m33.584217967s) [upload=36107]: Upload finished. err: <nil> Duration: 9.429290302s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 6.728711969s trace: https://jaeger.dev.storj.io/trace/55b376cc76cfe920 FAIL (2025-12-05 10:26:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 9 (41h1m29.913335078s) [upload=36107]: Uploading segments... Dec 5 09:56:53.442159225 (41h1m29.913369955s) [upload=36107]: Got next segment. Inline: false Dec 5 09:56:53.442188120 (41h1m29.913398849s) [upload=36107]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 5 09:56:53.442194372 (41h1m29.913405101s) [upload=36107]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 5 09:56:53.701976567 (41h1m30.173187296s) [seg_pos={0 0}, upload=36107]: Begin upload segment... Dec 5 09:56:53.702020460 (41h1m30.173231189s) [seg_pos={0 0}, upload=36107]: Done begin upload segment. Dec 5 09:56:53.702056649 (41h1m30.173267388s) [noise=true , address=77.90.31.21:4878 , piece_id=RCFTETF7SGR6FTUBJWWBJDKOZIHZKBMSUBZYVWYKF2MP56PMIWMQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702064965 (41h1m30.173275694s) [noise=true , address=145.79.227.28:4725 , piece_id=ZE642TJU4ZJKBOHQ6K6GPLPTN2YKC4BOP325LN7QXALZSDPG66HQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702111533 (41h1m30.173322272s) [noise=true , address=145.79.227.42:4121 , piece_id=6D47MY4GTQ3YQOGHS2V2OTO2VFDL6XKKS2R4VA6VZZ557WYFY34A, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702548986 (41h1m30.173759766s) [noise=true , address=145.79.227.27:4691 , piece_id=Z53YBOXTW4L23P4M3EOW6ADUHDWHIJVRDT7IL4I5FUPPZOW2MRLQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702931455 (41h1m30.174142194s) [noise=true , address=77.90.31.16:4735 , piece_id=V4YJIHVMXYTBK7IHU272RYCKUXI44B2MXRLETMWMO2WAI2XRU4SQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.703301079 (41h1m30.174511808s) [noise=true , address=145.79.227.10:4401 , piece_id=XAVSV3SGPDDLZWGKQLHA2CVYTGOPO7J3YVEO5C3MYIN5SJEUZ2OQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.703633000 (41h1m30.174843729s) [noise=true , address=145.79.227.24:4583 , piece_id=IB6T3J3CF3P3A54JEHHBK6KEPMUEI32B4QYSSA2G2BUMIVUC7DGQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.703957648 (41h1m30.175168377s) [noise=true , address=145.79.227.26:4654 , piece_id=23NNREG47WSZWJNPZWP275L62YBBSTB6RQWTINIRTTAYU66DRT7Q, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.704788621 (41h1m30.17599936s) [noise=true , address=145.79.227.43:4159 , piece_id=S5T27VFA6UNNONPZBZILNTQUUNSRLWPREKOJORMLKOXWNL3GMBSQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.705457525 (41h1m30.176668264s) [noise=true , address=145.79.227.32:4835 , piece_id=PJURUANT2XJY5PNYTSO5W2EDT3HYFAC2BFNCANKOAZ3I6P2GRC3Q, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.706486825 (41h1m30.177697554s) [noise=true , address=145.79.227.24:4582 , piece_id=MCQWT2LFD472TQDOAHLGQ5L2X7S654P6TL4J5MSP3CML5ZCURHMA, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.707626675 (41h1m30.178837414s) [noise=true , address=145.79.227.44:4193 , piece_id=327UUYIASLKFGUIQVY4P5UML4R2R4ODLUAYXR3GISKIT6CFDUGAQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.708631078 (41h1m30.179841817s) [noise=true , address=145.79.227.40:4050 , piece_id=QFSFC3WFER7VBVZNB3XLRIEFWGM4Q5VTAAJIWG4HAERKXSHZGZ4Q, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.709908630 (41h1m30.181119359s) [noise=true , address=145.79.227.22:4510 , piece_id=HCJIUH3VECRBR744KIQ65UKSMTEEKUPHJWXSDBM5SEMUVBJ5YX6A, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.710331386 (41h1m30.181542125s) [noise=true , address=145.79.227.41:4087 , piece_id=6GY2NC7F26VLU5NEN65OYU7YIHJYSTI5GNVRTNXM5HO5XQMZKEPA, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.710803685 (41h1m30.182014424s) [noise=true , address=45.91.116.4:4937 , piece_id=VFAPMIAB2G2BBOMNSM5EXONUL5BNORMR7KXIYSTQ7UFRVASVTQHQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:55.513663514 (41h1m31.984874243s) [upload=36107]: Next returned nil segment Dec 5 09:56:55.513682680 (41h1m31.98489341s) [upload=36107]: Waiting for error group managing segments... Dec 5 09:56:55.808504186 (41h1m32.279714915s) [noise=true , address=77.90.31.21:4878 , piece_id=RCFTETF7SGR6FTUBJWWBJDKOZIHZKBMSUBZYVWYKF2MP56PMIWMQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.866933347 (41h1m32.338144086s) [noise=true , address=145.79.227.10:4401 , piece_id=XAVSV3SGPDDLZWGKQLHA2CVYTGOPO7J3YVEO5C3MYIN5SJEUZ2OQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.869510454 (41h1m32.340721183s) [noise=true , address=145.79.227.28:4725 , piece_id=ZE642TJU4ZJKBOHQ6K6GPLPTN2YKC4BOP325LN7QXALZSDPG66HQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.875082795 (41h1m32.346293524s) [noise=true , address=145.79.227.43:4159 , piece_id=S5T27VFA6UNNONPZBZILNTQUUNSRLWPREKOJORMLKOXWNL3GMBSQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.884542648 (41h1m32.355753387s) [noise=true , address=145.79.227.24:4583 , piece_id=IB6T3J3CF3P3A54JEHHBK6KEPMUEI32B4QYSSA2G2BUMIVUC7DGQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.888876321 (41h1m32.36008707s) [noise=true , address=145.79.227.40:4050 , piece_id=QFSFC3WFER7VBVZNB3XLRIEFWGM4Q5VTAAJIWG4HAERKXSHZGZ4Q, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.889468719 (41h1m32.360679448s) [noise=true , address=145.79.227.26:4654 , piece_id=23NNREG47WSZWJNPZWP275L62YBBSTB6RQWTINIRTTAYU66DRT7Q, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.900234388 (41h1m32.371445117s) [noise=true , address=45.91.116.4:4937 , piece_id=VFAPMIAB2G2BBOMNSM5EXONUL5BNORMR7KXIYSTQ7UFRVASVTQHQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.923941432 (41h1m32.395152161s) [noise=true , address=145.79.227.42:4121 , piece_id=6D47MY4GTQ3YQOGHS2V2OTO2VFDL6XKKS2R4VA6VZZ557WYFY34A, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.929131404 (41h1m32.400342133s) [noise=true , address=145.79.227.22:4510 , piece_id=HCJIUH3VECRBR744KIQ65UKSMTEEKUPHJWXSDBM5SEMUVBJ5YX6A, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.180555093 (41h1m32.651765822s) [noise=true , address=145.79.227.32:4835 , piece_id=PJURUANT2XJY5PNYTSO5W2EDT3HYFAC2BFNCANKOAZ3I6P2GRC3Q, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.587529048 (41h1m33.058739787s) [noise=true , address=145.79.227.41:4087 , piece_id=6GY2NC7F26VLU5NEN65OYU7YIHJYSTI5GNVRTNXM5HO5XQMZKEPA, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.671173874 (41h1m33.142384603s) [noise=true , address=145.79.227.27:4691 , piece_id=Z53YBOXTW4L23P4M3EOW6ADUHDWHIJVRDT7IL4I5FUPPZOW2MRLQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.671209982 (41h1m33.142420711s) [seg_pos={0 0}, upload=36107]: Segment reached optimal threshold of 13 pieces. Dec 5 09:56:56.671423689 (41h1m33.142634428s) [noise=true , address=145.79.227.24:4582 , piece_id=MCQWT2LFD472TQDOAHLGQ5L2X7S654P6TL4J5MSP3CML5ZCURHMA, seg_pos={0 0}, upload=36107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa): protocol: write tcp 188.166.225.170:54136->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:54136->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54136->145.79.227.24:4582: use of closed network connection Dec 5 09:56:56.671432325 (41h1m33.142643064s) [noise=true , address=145.79.227.24:4582 , piece_id=MCQWT2LFD472TQDOAHLGQ5L2X7S654P6TL4J5MSP3CML5ZCURHMA, seg_pos={0 0}, upload=36107]: Long tail cancellation due to optimal threshold Dec 5 09:56:56.671496377 (41h1m33.142707106s) [noise=true , address=145.79.227.44:4193 , piece_id=327UUYIASLKFGUIQVY4P5UML4R2R4ODLUAYXR3GISKIT6CFDUGAQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: write tcp 188.166.225.170:39276->145.79.227.44:4193: use of closed network connection; write tcp 188.166.225.170:39276->145.79.227.44:4193: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39276->145.79.227.44:4193: use of closed network connection Dec 5 09:56:56.671501627 (41h1m33.142712366s) [noise=true , address=145.79.227.44:4193 , piece_id=327UUYIASLKFGUIQVY4P5UML4R2R4ODLUAYXR3GISKIT6CFDUGAQ, seg_pos={0 0}, upload=36107]: Long tail cancellation due to optimal threshold Dec 5 09:56:56.671516466 (41h1m33.142727195s) [noise=true , address=77.90.31.16:4735 , piece_id=V4YJIHVMXYTBK7IHU272RYCKUXI44B2MXRLETMWMO2WAI2XRU4SQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: write tcp 188.166.225.170:41866->77.90.31.16:4735: use of closed network connection; storage node overloaded, request limit: 2000 Dec 5 09:56:56.671525944 (41h1m33.142736673s) [noise=true , address=77.90.31.16:4735 , piece_id=V4YJIHVMXYTBK7IHU272RYCKUXI44B2MXRLETMWMO2WAI2XRU4SQ, seg_pos={0 0}, upload=36107]: Long tail cancellation due to optimal threshold Dec 5 09:56:56.671548016 (41h1m33.142758745s) [seg_pos={0 0}, upload=36107]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 5 09:56:56.671623500 (41h1m33.142834229s) [upload=36107]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 5 09:56:56.671631204 (41h1m33.142841933s) [upload=36107]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 5 09:56:57.112987571 (41h1m33.5841983s) [upload=36107]: Done uploading segments... Dec 5 09:56:57.113007228 (41h1m33.584217967s) [upload=36107]: Upload finished. err: <nil> Duration: 6.728792663s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 8.903617147s trace: https://jaeger.dev.storj.io/trace/55b376cc76cfe920 FAIL (2025-12-05 10:25:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 9 (41h1m29.913335078s) [upload=36107]: Uploading segments... Dec 5 09:56:53.442159225 (41h1m29.913369955s) [upload=36107]: Got next segment. Inline: false Dec 5 09:56:53.442188120 (41h1m29.913398849s) [upload=36107]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 5 09:56:53.442194372 (41h1m29.913405101s) [upload=36107]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 5 09:56:53.701976567 (41h1m30.173187296s) [seg_pos={0 0}, upload=36107]: Begin upload segment... Dec 5 09:56:53.702020460 (41h1m30.173231189s) [seg_pos={0 0}, upload=36107]: Done begin upload segment. Dec 5 09:56:53.702056649 (41h1m30.173267388s) [noise=true , address=77.90.31.21:4878 , piece_id=RCFTETF7SGR6FTUBJWWBJDKOZIHZKBMSUBZYVWYKF2MP56PMIWMQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702064965 (41h1m30.173275694s) [noise=true , address=145.79.227.28:4725 , piece_id=ZE642TJU4ZJKBOHQ6K6GPLPTN2YKC4BOP325LN7QXALZSDPG66HQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702111533 (41h1m30.173322272s) [noise=true , address=145.79.227.42:4121 , piece_id=6D47MY4GTQ3YQOGHS2V2OTO2VFDL6XKKS2R4VA6VZZ557WYFY34A, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702548986 (41h1m30.173759766s) [noise=true , address=145.79.227.27:4691 , piece_id=Z53YBOXTW4L23P4M3EOW6ADUHDWHIJVRDT7IL4I5FUPPZOW2MRLQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702931455 (41h1m30.174142194s) [noise=true , address=77.90.31.16:4735 , piece_id=V4YJIHVMXYTBK7IHU272RYCKUXI44B2MXRLETMWMO2WAI2XRU4SQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.703301079 (41h1m30.174511808s) [noise=true , address=145.79.227.10:4401 , piece_id=XAVSV3SGPDDLZWGKQLHA2CVYTGOPO7J3YVEO5C3MYIN5SJEUZ2OQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.703633000 (41h1m30.174843729s) [noise=true , address=145.79.227.24:4583 , piece_id=IB6T3J3CF3P3A54JEHHBK6KEPMUEI32B4QYSSA2G2BUMIVUC7DGQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.703957648 (41h1m30.175168377s) [noise=true , address=145.79.227.26:4654 , piece_id=23NNREG47WSZWJNPZWP275L62YBBSTB6RQWTINIRTTAYU66DRT7Q, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.704788621 (41h1m30.17599936s) [noise=true , address=145.79.227.43:4159 , piece_id=S5T27VFA6UNNONPZBZILNTQUUNSRLWPREKOJORMLKOXWNL3GMBSQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.705457525 (41h1m30.176668264s) [noise=true , address=145.79.227.32:4835 , piece_id=PJURUANT2XJY5PNYTSO5W2EDT3HYFAC2BFNCANKOAZ3I6P2GRC3Q, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.706486825 (41h1m30.177697554s) [noise=true , address=145.79.227.24:4582 , piece_id=MCQWT2LFD472TQDOAHLGQ5L2X7S654P6TL4J5MSP3CML5ZCURHMA, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.707626675 (41h1m30.178837414s) [noise=true , address=145.79.227.44:4193 , piece_id=327UUYIASLKFGUIQVY4P5UML4R2R4ODLUAYXR3GISKIT6CFDUGAQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.708631078 (41h1m30.179841817s) [noise=true , address=145.79.227.40:4050 , piece_id=QFSFC3WFER7VBVZNB3XLRIEFWGM4Q5VTAAJIWG4HAERKXSHZGZ4Q, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.709908630 (41h1m30.181119359s) [noise=true , address=145.79.227.22:4510 , piece_id=HCJIUH3VECRBR744KIQ65UKSMTEEKUPHJWXSDBM5SEMUVBJ5YX6A, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.710331386 (41h1m30.181542125s) [noise=true , address=145.79.227.41:4087 , piece_id=6GY2NC7F26VLU5NEN65OYU7YIHJYSTI5GNVRTNXM5HO5XQMZKEPA, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.710803685 (41h1m30.182014424s) [noise=true , address=45.91.116.4:4937 , piece_id=VFAPMIAB2G2BBOMNSM5EXONUL5BNORMR7KXIYSTQ7UFRVASVTQHQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:55.513663514 (41h1m31.984874243s) [upload=36107]: Next returned nil segment Dec 5 09:56:55.513682680 (41h1m31.98489341s) [upload=36107]: Waiting for error group managing segments... Dec 5 09:56:55.808504186 (41h1m32.279714915s) [noise=true , address=77.90.31.21:4878 , piece_id=RCFTETF7SGR6FTUBJWWBJDKOZIHZKBMSUBZYVWYKF2MP56PMIWMQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.866933347 (41h1m32.338144086s) [noise=true , address=145.79.227.10:4401 , piece_id=XAVSV3SGPDDLZWGKQLHA2CVYTGOPO7J3YVEO5C3MYIN5SJEUZ2OQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.869510454 (41h1m32.340721183s) [noise=true , address=145.79.227.28:4725 , piece_id=ZE642TJU4ZJKBOHQ6K6GPLPTN2YKC4BOP325LN7QXALZSDPG66HQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.875082795 (41h1m32.346293524s) [noise=true , address=145.79.227.43:4159 , piece_id=S5T27VFA6UNNONPZBZILNTQUUNSRLWPREKOJORMLKOXWNL3GMBSQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.884542648 (41h1m32.355753387s) [noise=true , address=145.79.227.24:4583 , piece_id=IB6T3J3CF3P3A54JEHHBK6KEPMUEI32B4QYSSA2G2BUMIVUC7DGQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.888876321 (41h1m32.36008707s) [noise=true , address=145.79.227.40:4050 , piece_id=QFSFC3WFER7VBVZNB3XLRIEFWGM4Q5VTAAJIWG4HAERKXSHZGZ4Q, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.889468719 (41h1m32.360679448s) [noise=true , address=145.79.227.26:4654 , piece_id=23NNREG47WSZWJNPZWP275L62YBBSTB6RQWTINIRTTAYU66DRT7Q, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.900234388 (41h1m32.371445117s) [noise=true , address=45.91.116.4:4937 , piece_id=VFAPMIAB2G2BBOMNSM5EXONUL5BNORMR7KXIYSTQ7UFRVASVTQHQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.923941432 (41h1m32.395152161s) [noise=true , address=145.79.227.42:4121 , piece_id=6D47MY4GTQ3YQOGHS2V2OTO2VFDL6XKKS2R4VA6VZZ557WYFY34A, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.929131404 (41h1m32.400342133s) [noise=true , address=145.79.227.22:4510 , piece_id=HCJIUH3VECRBR744KIQ65UKSMTEEKUPHJWXSDBM5SEMUVBJ5YX6A, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.180555093 (41h1m32.651765822s) [noise=true , address=145.79.227.32:4835 , piece_id=PJURUANT2XJY5PNYTSO5W2EDT3HYFAC2BFNCANKOAZ3I6P2GRC3Q, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.587529048 (41h1m33.058739787s) [noise=true , address=145.79.227.41:4087 , piece_id=6GY2NC7F26VLU5NEN65OYU7YIHJYSTI5GNVRTNXM5HO5XQMZKEPA, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.671173874 (41h1m33.142384603s) [noise=true , address=145.79.227.27:4691 , piece_id=Z53YBOXTW4L23P4M3EOW6ADUHDWHIJVRDT7IL4I5FUPPZOW2MRLQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.671209982 (41h1m33.142420711s) [seg_pos={0 0}, upload=36107]: Segment reached optimal threshold of 13 pieces. Dec 5 09:56:56.671423689 (41h1m33.142634428s) [noise=true , address=145.79.227.24:4582 , piece_id=MCQWT2LFD472TQDOAHLGQ5L2X7S654P6TL4J5MSP3CML5ZCURHMA, seg_pos={0 0}, upload=36107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa): protocol: write tcp 188.166.225.170:54136->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:54136->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54136->145.79.227.24:4582: use of closed network connection Dec 5 09:56:56.671432325 (41h1m33.142643064s) [noise=true , address=145.79.227.24:4582 , piece_id=MCQWT2LFD472TQDOAHLGQ5L2X7S654P6TL4J5MSP3CML5ZCURHMA, seg_pos={0 0}, upload=36107]: Long tail cancellation due to optimal threshold Dec 5 09:56:56.671496377 (41h1m33.142707106s) [noise=true , address=145.79.227.44:4193 , piece_id=327UUYIASLKFGUIQVY4P5UML4R2R4ODLUAYXR3GISKIT6CFDUGAQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: write tcp 188.166.225.170:39276->145.79.227.44:4193: use of closed network connection; write tcp 188.166.225.170:39276->145.79.227.44:4193: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39276->145.79.227.44:4193: use of closed network connection Dec 5 09:56:56.671501627 (41h1m33.142712366s) [noise=true , address=145.79.227.44:4193 , piece_id=327UUYIASLKFGUIQVY4P5UML4R2R4ODLUAYXR3GISKIT6CFDUGAQ, seg_pos={0 0}, upload=36107]: Long tail cancellation due to optimal threshold Dec 5 09:56:56.671516466 (41h1m33.142727195s) [noise=true , address=77.90.31.16:4735 , piece_id=V4YJIHVMXYTBK7IHU272RYCKUXI44B2MXRLETMWMO2WAI2XRU4SQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: write tcp 188.166.225.170:41866->77.90.31.16:4735: use of closed network connection; storage node overloaded, request limit: 2000 Dec 5 09:56:56.671525944 (41h1m33.142736673s) [noise=true , address=77.90.31.16:4735 , piece_id=V4YJIHVMXYTBK7IHU272RYCKUXI44B2MXRLETMWMO2WAI2XRU4SQ, seg_pos={0 0}, upload=36107]: Long tail cancellation due to optimal threshold Dec 5 09:56:56.671548016 (41h1m33.142758745s) [seg_pos={0 0}, upload=36107]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 5 09:56:56.671623500 (41h1m33.142834229s) [upload=36107]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 5 09:56:56.671631204 (41h1m33.142841933s) [upload=36107]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 5 09:56:57.112987571 (41h1m33.5841983s) [upload=36107]: Done uploading segments... Dec 5 09:56:57.113007228 (41h1m33.584217967s) [upload=36107]: Upload finished. err: <nil> Duration: 8.903648937s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 7.107699957s trace: https://jaeger.dev.storj.io/trace/55b376cc76cfe920 FAIL (2025-12-05 10:24:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 9 (41h1m29.913335078s) [upload=36107]: Uploading segments... Dec 5 09:56:53.442159225 (41h1m29.913369955s) [upload=36107]: Got next segment. Inline: false Dec 5 09:56:53.442188120 (41h1m29.913398849s) [upload=36107]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 5 09:56:53.442194372 (41h1m29.913405101s) [upload=36107]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 5 09:56:53.701976567 (41h1m30.173187296s) [seg_pos={0 0}, upload=36107]: Begin upload segment... Dec 5 09:56:53.702020460 (41h1m30.173231189s) [seg_pos={0 0}, upload=36107]: Done begin upload segment. Dec 5 09:56:53.702056649 (41h1m30.173267388s) [noise=true , address=77.90.31.21:4878 , piece_id=RCFTETF7SGR6FTUBJWWBJDKOZIHZKBMSUBZYVWYKF2MP56PMIWMQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702064965 (41h1m30.173275694s) [noise=true , address=145.79.227.28:4725 , piece_id=ZE642TJU4ZJKBOHQ6K6GPLPTN2YKC4BOP325LN7QXALZSDPG66HQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702111533 (41h1m30.173322272s) [noise=true , address=145.79.227.42:4121 , piece_id=6D47MY4GTQ3YQOGHS2V2OTO2VFDL6XKKS2R4VA6VZZ557WYFY34A, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702548986 (41h1m30.173759766s) [noise=true , address=145.79.227.27:4691 , piece_id=Z53YBOXTW4L23P4M3EOW6ADUHDWHIJVRDT7IL4I5FUPPZOW2MRLQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702931455 (41h1m30.174142194s) [noise=true , address=77.90.31.16:4735 , piece_id=V4YJIHVMXYTBK7IHU272RYCKUXI44B2MXRLETMWMO2WAI2XRU4SQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.703301079 (41h1m30.174511808s) [noise=true , address=145.79.227.10:4401 , piece_id=XAVSV3SGPDDLZWGKQLHA2CVYTGOPO7J3YVEO5C3MYIN5SJEUZ2OQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.703633000 (41h1m30.174843729s) [noise=true , address=145.79.227.24:4583 , piece_id=IB6T3J3CF3P3A54JEHHBK6KEPMUEI32B4QYSSA2G2BUMIVUC7DGQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.703957648 (41h1m30.175168377s) [noise=true , address=145.79.227.26:4654 , piece_id=23NNREG47WSZWJNPZWP275L62YBBSTB6RQWTINIRTTAYU66DRT7Q, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.704788621 (41h1m30.17599936s) [noise=true , address=145.79.227.43:4159 , piece_id=S5T27VFA6UNNONPZBZILNTQUUNSRLWPREKOJORMLKOXWNL3GMBSQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.705457525 (41h1m30.176668264s) [noise=true , address=145.79.227.32:4835 , piece_id=PJURUANT2XJY5PNYTSO5W2EDT3HYFAC2BFNCANKOAZ3I6P2GRC3Q, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.706486825 (41h1m30.177697554s) [noise=true , address=145.79.227.24:4582 , piece_id=MCQWT2LFD472TQDOAHLGQ5L2X7S654P6TL4J5MSP3CML5ZCURHMA, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.707626675 (41h1m30.178837414s) [noise=true , address=145.79.227.44:4193 , piece_id=327UUYIASLKFGUIQVY4P5UML4R2R4ODLUAYXR3GISKIT6CFDUGAQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.708631078 (41h1m30.179841817s) [noise=true , address=145.79.227.40:4050 , piece_id=QFSFC3WFER7VBVZNB3XLRIEFWGM4Q5VTAAJIWG4HAERKXSHZGZ4Q, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.709908630 (41h1m30.181119359s) [noise=true , address=145.79.227.22:4510 , piece_id=HCJIUH3VECRBR744KIQ65UKSMTEEKUPHJWXSDBM5SEMUVBJ5YX6A, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.710331386 (41h1m30.181542125s) [noise=true , address=145.79.227.41:4087 , piece_id=6GY2NC7F26VLU5NEN65OYU7YIHJYSTI5GNVRTNXM5HO5XQMZKEPA, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.710803685 (41h1m30.182014424s) [noise=true , address=45.91.116.4:4937 , piece_id=VFAPMIAB2G2BBOMNSM5EXONUL5BNORMR7KXIYSTQ7UFRVASVTQHQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:55.513663514 (41h1m31.984874243s) [upload=36107]: Next returned nil segment Dec 5 09:56:55.513682680 (41h1m31.98489341s) [upload=36107]: Waiting for error group managing segments... Dec 5 09:56:55.808504186 (41h1m32.279714915s) [noise=true , address=77.90.31.21:4878 , piece_id=RCFTETF7SGR6FTUBJWWBJDKOZIHZKBMSUBZYVWYKF2MP56PMIWMQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.866933347 (41h1m32.338144086s) [noise=true , address=145.79.227.10:4401 , piece_id=XAVSV3SGPDDLZWGKQLHA2CVYTGOPO7J3YVEO5C3MYIN5SJEUZ2OQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.869510454 (41h1m32.340721183s) [noise=true , address=145.79.227.28:4725 , piece_id=ZE642TJU4ZJKBOHQ6K6GPLPTN2YKC4BOP325LN7QXALZSDPG66HQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.875082795 (41h1m32.346293524s) [noise=true , address=145.79.227.43:4159 , piece_id=S5T27VFA6UNNONPZBZILNTQUUNSRLWPREKOJORMLKOXWNL3GMBSQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.884542648 (41h1m32.355753387s) [noise=true , address=145.79.227.24:4583 , piece_id=IB6T3J3CF3P3A54JEHHBK6KEPMUEI32B4QYSSA2G2BUMIVUC7DGQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.888876321 (41h1m32.36008707s) [noise=true , address=145.79.227.40:4050 , piece_id=QFSFC3WFER7VBVZNB3XLRIEFWGM4Q5VTAAJIWG4HAERKXSHZGZ4Q, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.889468719 (41h1m32.360679448s) [noise=true , address=145.79.227.26:4654 , piece_id=23NNREG47WSZWJNPZWP275L62YBBSTB6RQWTINIRTTAYU66DRT7Q, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.900234388 (41h1m32.371445117s) [noise=true , address=45.91.116.4:4937 , piece_id=VFAPMIAB2G2BBOMNSM5EXONUL5BNORMR7KXIYSTQ7UFRVASVTQHQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.923941432 (41h1m32.395152161s) [noise=true , address=145.79.227.42:4121 , piece_id=6D47MY4GTQ3YQOGHS2V2OTO2VFDL6XKKS2R4VA6VZZ557WYFY34A, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.929131404 (41h1m32.400342133s) [noise=true , address=145.79.227.22:4510 , piece_id=HCJIUH3VECRBR744KIQ65UKSMTEEKUPHJWXSDBM5SEMUVBJ5YX6A, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.180555093 (41h1m32.651765822s) [noise=true , address=145.79.227.32:4835 , piece_id=PJURUANT2XJY5PNYTSO5W2EDT3HYFAC2BFNCANKOAZ3I6P2GRC3Q, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.587529048 (41h1m33.058739787s) [noise=true , address=145.79.227.41:4087 , piece_id=6GY2NC7F26VLU5NEN65OYU7YIHJYSTI5GNVRTNXM5HO5XQMZKEPA, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.671173874 (41h1m33.142384603s) [noise=true , address=145.79.227.27:4691 , piece_id=Z53YBOXTW4L23P4M3EOW6ADUHDWHIJVRDT7IL4I5FUPPZOW2MRLQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.671209982 (41h1m33.142420711s) [seg_pos={0 0}, upload=36107]: Segment reached optimal threshold of 13 pieces. Dec 5 09:56:56.671423689 (41h1m33.142634428s) [noise=true , address=145.79.227.24:4582 , piece_id=MCQWT2LFD472TQDOAHLGQ5L2X7S654P6TL4J5MSP3CML5ZCURHMA, seg_pos={0 0}, upload=36107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa): protocol: write tcp 188.166.225.170:54136->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:54136->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54136->145.79.227.24:4582: use of closed network connection Dec 5 09:56:56.671432325 (41h1m33.142643064s) [noise=true , address=145.79.227.24:4582 , piece_id=MCQWT2LFD472TQDOAHLGQ5L2X7S654P6TL4J5MSP3CML5ZCURHMA, seg_pos={0 0}, upload=36107]: Long tail cancellation due to optimal threshold Dec 5 09:56:56.671496377 (41h1m33.142707106s) [noise=true , address=145.79.227.44:4193 , piece_id=327UUYIASLKFGUIQVY4P5UML4R2R4ODLUAYXR3GISKIT6CFDUGAQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: write tcp 188.166.225.170:39276->145.79.227.44:4193: use of closed network connection; write tcp 188.166.225.170:39276->145.79.227.44:4193: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39276->145.79.227.44:4193: use of closed network connection Dec 5 09:56:56.671501627 (41h1m33.142712366s) [noise=true , address=145.79.227.44:4193 , piece_id=327UUYIASLKFGUIQVY4P5UML4R2R4ODLUAYXR3GISKIT6CFDUGAQ, seg_pos={0 0}, upload=36107]: Long tail cancellation due to optimal threshold Dec 5 09:56:56.671516466 (41h1m33.142727195s) [noise=true , address=77.90.31.16:4735 , piece_id=V4YJIHVMXYTBK7IHU272RYCKUXI44B2MXRLETMWMO2WAI2XRU4SQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: write tcp 188.166.225.170:41866->77.90.31.16:4735: use of closed network connection; storage node overloaded, request limit: 2000 Dec 5 09:56:56.671525944 (41h1m33.142736673s) [noise=true , address=77.90.31.16:4735 , piece_id=V4YJIHVMXYTBK7IHU272RYCKUXI44B2MXRLETMWMO2WAI2XRU4SQ, seg_pos={0 0}, upload=36107]: Long tail cancellation due to optimal threshold Dec 5 09:56:56.671548016 (41h1m33.142758745s) [seg_pos={0 0}, upload=36107]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 5 09:56:56.671623500 (41h1m33.142834229s) [upload=36107]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 5 09:56:56.671631204 (41h1m33.142841933s) [upload=36107]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 5 09:56:57.112987571 (41h1m33.5841983s) [upload=36107]: Done uploading segments... Dec 5 09:56:57.113007228 (41h1m33.584217967s) [upload=36107]: Upload finished. err: <nil> Duration: 7.107726798s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 5.932792417s trace: https://jaeger.dev.storj.io/trace/55b376cc76cfe920 FAIL (2025-12-05 10:24:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 9 (41h1m29.913335078s) [upload=36107]: Uploading segments... Dec 5 09:56:53.442159225 (41h1m29.913369955s) [upload=36107]: Got next segment. Inline: false Dec 5 09:56:53.442188120 (41h1m29.913398849s) [upload=36107]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 5 09:56:53.442194372 (41h1m29.913405101s) [upload=36107]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 5 09:56:53.701976567 (41h1m30.173187296s) [seg_pos={0 0}, upload=36107]: Begin upload segment... Dec 5 09:56:53.702020460 (41h1m30.173231189s) [seg_pos={0 0}, upload=36107]: Done begin upload segment. Dec 5 09:56:53.702056649 (41h1m30.173267388s) [noise=true , address=77.90.31.21:4878 , piece_id=RCFTETF7SGR6FTUBJWWBJDKOZIHZKBMSUBZYVWYKF2MP56PMIWMQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702064965 (41h1m30.173275694s) [noise=true , address=145.79.227.28:4725 , piece_id=ZE642TJU4ZJKBOHQ6K6GPLPTN2YKC4BOP325LN7QXALZSDPG66HQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702111533 (41h1m30.173322272s) [noise=true , address=145.79.227.42:4121 , piece_id=6D47MY4GTQ3YQOGHS2V2OTO2VFDL6XKKS2R4VA6VZZ557WYFY34A, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702548986 (41h1m30.173759766s) [noise=true , address=145.79.227.27:4691 , piece_id=Z53YBOXTW4L23P4M3EOW6ADUHDWHIJVRDT7IL4I5FUPPZOW2MRLQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702931455 (41h1m30.174142194s) [noise=true , address=77.90.31.16:4735 , piece_id=V4YJIHVMXYTBK7IHU272RYCKUXI44B2MXRLETMWMO2WAI2XRU4SQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.703301079 (41h1m30.174511808s) [noise=true , address=145.79.227.10:4401 , piece_id=XAVSV3SGPDDLZWGKQLHA2CVYTGOPO7J3YVEO5C3MYIN5SJEUZ2OQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.703633000 (41h1m30.174843729s) [noise=true , address=145.79.227.24:4583 , piece_id=IB6T3J3CF3P3A54JEHHBK6KEPMUEI32B4QYSSA2G2BUMIVUC7DGQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.703957648 (41h1m30.175168377s) [noise=true , address=145.79.227.26:4654 , piece_id=23NNREG47WSZWJNPZWP275L62YBBSTB6RQWTINIRTTAYU66DRT7Q, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.704788621 (41h1m30.17599936s) [noise=true , address=145.79.227.43:4159 , piece_id=S5T27VFA6UNNONPZBZILNTQUUNSRLWPREKOJORMLKOXWNL3GMBSQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.705457525 (41h1m30.176668264s) [noise=true , address=145.79.227.32:4835 , piece_id=PJURUANT2XJY5PNYTSO5W2EDT3HYFAC2BFNCANKOAZ3I6P2GRC3Q, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.706486825 (41h1m30.177697554s) [noise=true , address=145.79.227.24:4582 , piece_id=MCQWT2LFD472TQDOAHLGQ5L2X7S654P6TL4J5MSP3CML5ZCURHMA, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.707626675 (41h1m30.178837414s) [noise=true , address=145.79.227.44:4193 , piece_id=327UUYIASLKFGUIQVY4P5UML4R2R4ODLUAYXR3GISKIT6CFDUGAQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.708631078 (41h1m30.179841817s) [noise=true , address=145.79.227.40:4050 , piece_id=QFSFC3WFER7VBVZNB3XLRIEFWGM4Q5VTAAJIWG4HAERKXSHZGZ4Q, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.709908630 (41h1m30.181119359s) [noise=true , address=145.79.227.22:4510 , piece_id=HCJIUH3VECRBR744KIQ65UKSMTEEKUPHJWXSDBM5SEMUVBJ5YX6A, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.710331386 (41h1m30.181542125s) [noise=true , address=145.79.227.41:4087 , piece_id=6GY2NC7F26VLU5NEN65OYU7YIHJYSTI5GNVRTNXM5HO5XQMZKEPA, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.710803685 (41h1m30.182014424s) [noise=true , address=45.91.116.4:4937 , piece_id=VFAPMIAB2G2BBOMNSM5EXONUL5BNORMR7KXIYSTQ7UFRVASVTQHQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:55.513663514 (41h1m31.984874243s) [upload=36107]: Next returned nil segment Dec 5 09:56:55.513682680 (41h1m31.98489341s) [upload=36107]: Waiting for error group managing segments... Dec 5 09:56:55.808504186 (41h1m32.279714915s) [noise=true , address=77.90.31.21:4878 , piece_id=RCFTETF7SGR6FTUBJWWBJDKOZIHZKBMSUBZYVWYKF2MP56PMIWMQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.866933347 (41h1m32.338144086s) [noise=true , address=145.79.227.10:4401 , piece_id=XAVSV3SGPDDLZWGKQLHA2CVYTGOPO7J3YVEO5C3MYIN5SJEUZ2OQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.869510454 (41h1m32.340721183s) [noise=true , address=145.79.227.28:4725 , piece_id=ZE642TJU4ZJKBOHQ6K6GPLPTN2YKC4BOP325LN7QXALZSDPG66HQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.875082795 (41h1m32.346293524s) [noise=true , address=145.79.227.43:4159 , piece_id=S5T27VFA6UNNONPZBZILNTQUUNSRLWPREKOJORMLKOXWNL3GMBSQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.884542648 (41h1m32.355753387s) [noise=true , address=145.79.227.24:4583 , piece_id=IB6T3J3CF3P3A54JEHHBK6KEPMUEI32B4QYSSA2G2BUMIVUC7DGQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.888876321 (41h1m32.36008707s) [noise=true , address=145.79.227.40:4050 , piece_id=QFSFC3WFER7VBVZNB3XLRIEFWGM4Q5VTAAJIWG4HAERKXSHZGZ4Q, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.889468719 (41h1m32.360679448s) [noise=true , address=145.79.227.26:4654 , piece_id=23NNREG47WSZWJNPZWP275L62YBBSTB6RQWTINIRTTAYU66DRT7Q, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.900234388 (41h1m32.371445117s) [noise=true , address=45.91.116.4:4937 , piece_id=VFAPMIAB2G2BBOMNSM5EXONUL5BNORMR7KXIYSTQ7UFRVASVTQHQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.923941432 (41h1m32.395152161s) [noise=true , address=145.79.227.42:4121 , piece_id=6D47MY4GTQ3YQOGHS2V2OTO2VFDL6XKKS2R4VA6VZZ557WYFY34A, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.929131404 (41h1m32.400342133s) [noise=true , address=145.79.227.22:4510 , piece_id=HCJIUH3VECRBR744KIQ65UKSMTEEKUPHJWXSDBM5SEMUVBJ5YX6A, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.180555093 (41h1m32.651765822s) [noise=true , address=145.79.227.32:4835 , piece_id=PJURUANT2XJY5PNYTSO5W2EDT3HYFAC2BFNCANKOAZ3I6P2GRC3Q, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.587529048 (41h1m33.058739787s) [noise=true , address=145.79.227.41:4087 , piece_id=6GY2NC7F26VLU5NEN65OYU7YIHJYSTI5GNVRTNXM5HO5XQMZKEPA, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.671173874 (41h1m33.142384603s) [noise=true , address=145.79.227.27:4691 , piece_id=Z53YBOXTW4L23P4M3EOW6ADUHDWHIJVRDT7IL4I5FUPPZOW2MRLQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.671209982 (41h1m33.142420711s) [seg_pos={0 0}, upload=36107]: Segment reached optimal threshold of 13 pieces. Dec 5 09:56:56.671423689 (41h1m33.142634428s) [noise=true , address=145.79.227.24:4582 , piece_id=MCQWT2LFD472TQDOAHLGQ5L2X7S654P6TL4J5MSP3CML5ZCURHMA, seg_pos={0 0}, upload=36107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa): protocol: write tcp 188.166.225.170:54136->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:54136->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54136->145.79.227.24:4582: use of closed network connection Dec 5 09:56:56.671432325 (41h1m33.142643064s) [noise=true , address=145.79.227.24:4582 , piece_id=MCQWT2LFD472TQDOAHLGQ5L2X7S654P6TL4J5MSP3CML5ZCURHMA, seg_pos={0 0}, upload=36107]: Long tail cancellation due to optimal threshold Dec 5 09:56:56.671496377 (41h1m33.142707106s) [noise=true , address=145.79.227.44:4193 , piece_id=327UUYIASLKFGUIQVY4P5UML4R2R4ODLUAYXR3GISKIT6CFDUGAQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: write tcp 188.166.225.170:39276->145.79.227.44:4193: use of closed network connection; write tcp 188.166.225.170:39276->145.79.227.44:4193: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39276->145.79.227.44:4193: use of closed network connection Dec 5 09:56:56.671501627 (41h1m33.142712366s) [noise=true , address=145.79.227.44:4193 , piece_id=327UUYIASLKFGUIQVY4P5UML4R2R4ODLUAYXR3GISKIT6CFDUGAQ, seg_pos={0 0}, upload=36107]: Long tail cancellation due to optimal threshold Dec 5 09:56:56.671516466 (41h1m33.142727195s) [noise=true , address=77.90.31.16:4735 , piece_id=V4YJIHVMXYTBK7IHU272RYCKUXI44B2MXRLETMWMO2WAI2XRU4SQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: write tcp 188.166.225.170:41866->77.90.31.16:4735: use of closed network connection; storage node overloaded, request limit: 2000 Dec 5 09:56:56.671525944 (41h1m33.142736673s) [noise=true , address=77.90.31.16:4735 , piece_id=V4YJIHVMXYTBK7IHU272RYCKUXI44B2MXRLETMWMO2WAI2XRU4SQ, seg_pos={0 0}, upload=36107]: Long tail cancellation due to optimal threshold Dec 5 09:56:56.671548016 (41h1m33.142758745s) [seg_pos={0 0}, upload=36107]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 5 09:56:56.671623500 (41h1m33.142834229s) [upload=36107]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 5 09:56:56.671631204 (41h1m33.142841933s) [upload=36107]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 5 09:56:57.112987571 (41h1m33.5841983s) [upload=36107]: Done uploading segments... Dec 5 09:56:57.113007228 (41h1m33.584217967s) [upload=36107]: Upload finished. err: <nil> Duration: 5.932830149s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 6.454735148s trace: https://jaeger.dev.storj.io/trace/55b376cc76cfe920 FAIL (2025-12-05 10:23:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 9 (41h1m29.913335078s) [upload=36107]: Uploading segments... Dec 5 09:56:53.442159225 (41h1m29.913369955s) [upload=36107]: Got next segment. Inline: false Dec 5 09:56:53.442188120 (41h1m29.913398849s) [upload=36107]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 5 09:56:53.442194372 (41h1m29.913405101s) [upload=36107]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 5 09:56:53.701976567 (41h1m30.173187296s) [seg_pos={0 0}, upload=36107]: Begin upload segment... Dec 5 09:56:53.702020460 (41h1m30.173231189s) [seg_pos={0 0}, upload=36107]: Done begin upload segment. Dec 5 09:56:53.702056649 (41h1m30.173267388s) [noise=true , address=77.90.31.21:4878 , piece_id=RCFTETF7SGR6FTUBJWWBJDKOZIHZKBMSUBZYVWYKF2MP56PMIWMQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702064965 (41h1m30.173275694s) [noise=true , address=145.79.227.28:4725 , piece_id=ZE642TJU4ZJKBOHQ6K6GPLPTN2YKC4BOP325LN7QXALZSDPG66HQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702111533 (41h1m30.173322272s) [noise=true , address=145.79.227.42:4121 , piece_id=6D47MY4GTQ3YQOGHS2V2OTO2VFDL6XKKS2R4VA6VZZ557WYFY34A, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702548986 (41h1m30.173759766s) [noise=true , address=145.79.227.27:4691 , piece_id=Z53YBOXTW4L23P4M3EOW6ADUHDWHIJVRDT7IL4I5FUPPZOW2MRLQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702931455 (41h1m30.174142194s) [noise=true , address=77.90.31.16:4735 , piece_id=V4YJIHVMXYTBK7IHU272RYCKUXI44B2MXRLETMWMO2WAI2XRU4SQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.703301079 (41h1m30.174511808s) [noise=true , address=145.79.227.10:4401 , piece_id=XAVSV3SGPDDLZWGKQLHA2CVYTGOPO7J3YVEO5C3MYIN5SJEUZ2OQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.703633000 (41h1m30.174843729s) [noise=true , address=145.79.227.24:4583 , piece_id=IB6T3J3CF3P3A54JEHHBK6KEPMUEI32B4QYSSA2G2BUMIVUC7DGQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.703957648 (41h1m30.175168377s) [noise=true , address=145.79.227.26:4654 , piece_id=23NNREG47WSZWJNPZWP275L62YBBSTB6RQWTINIRTTAYU66DRT7Q, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.704788621 (41h1m30.17599936s) [noise=true , address=145.79.227.43:4159 , piece_id=S5T27VFA6UNNONPZBZILNTQUUNSRLWPREKOJORMLKOXWNL3GMBSQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.705457525 (41h1m30.176668264s) [noise=true , address=145.79.227.32:4835 , piece_id=PJURUANT2XJY5PNYTSO5W2EDT3HYFAC2BFNCANKOAZ3I6P2GRC3Q, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.706486825 (41h1m30.177697554s) [noise=true , address=145.79.227.24:4582 , piece_id=MCQWT2LFD472TQDOAHLGQ5L2X7S654P6TL4J5MSP3CML5ZCURHMA, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.707626675 (41h1m30.178837414s) [noise=true , address=145.79.227.44:4193 , piece_id=327UUYIASLKFGUIQVY4P5UML4R2R4ODLUAYXR3GISKIT6CFDUGAQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.708631078 (41h1m30.179841817s) [noise=true , address=145.79.227.40:4050 , piece_id=QFSFC3WFER7VBVZNB3XLRIEFWGM4Q5VTAAJIWG4HAERKXSHZGZ4Q, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.709908630 (41h1m30.181119359s) [noise=true , address=145.79.227.22:4510 , piece_id=HCJIUH3VECRBR744KIQ65UKSMTEEKUPHJWXSDBM5SEMUVBJ5YX6A, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.710331386 (41h1m30.181542125s) [noise=true , address=145.79.227.41:4087 , piece_id=6GY2NC7F26VLU5NEN65OYU7YIHJYSTI5GNVRTNXM5HO5XQMZKEPA, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.710803685 (41h1m30.182014424s) [noise=true , address=45.91.116.4:4937 , piece_id=VFAPMIAB2G2BBOMNSM5EXONUL5BNORMR7KXIYSTQ7UFRVASVTQHQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:55.513663514 (41h1m31.984874243s) [upload=36107]: Next returned nil segment Dec 5 09:56:55.513682680 (41h1m31.98489341s) [upload=36107]: Waiting for error group managing segments... Dec 5 09:56:55.808504186 (41h1m32.279714915s) [noise=true , address=77.90.31.21:4878 , piece_id=RCFTETF7SGR6FTUBJWWBJDKOZIHZKBMSUBZYVWYKF2MP56PMIWMQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.866933347 (41h1m32.338144086s) [noise=true , address=145.79.227.10:4401 , piece_id=XAVSV3SGPDDLZWGKQLHA2CVYTGOPO7J3YVEO5C3MYIN5SJEUZ2OQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.869510454 (41h1m32.340721183s) [noise=true , address=145.79.227.28:4725 , piece_id=ZE642TJU4ZJKBOHQ6K6GPLPTN2YKC4BOP325LN7QXALZSDPG66HQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.875082795 (41h1m32.346293524s) [noise=true , address=145.79.227.43:4159 , piece_id=S5T27VFA6UNNONPZBZILNTQUUNSRLWPREKOJORMLKOXWNL3GMBSQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.884542648 (41h1m32.355753387s) [noise=true , address=145.79.227.24:4583 , piece_id=IB6T3J3CF3P3A54JEHHBK6KEPMUEI32B4QYSSA2G2BUMIVUC7DGQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.888876321 (41h1m32.36008707s) [noise=true , address=145.79.227.40:4050 , piece_id=QFSFC3WFER7VBVZNB3XLRIEFWGM4Q5VTAAJIWG4HAERKXSHZGZ4Q, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.889468719 (41h1m32.360679448s) [noise=true , address=145.79.227.26:4654 , piece_id=23NNREG47WSZWJNPZWP275L62YBBSTB6RQWTINIRTTAYU66DRT7Q, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.900234388 (41h1m32.371445117s) [noise=true , address=45.91.116.4:4937 , piece_id=VFAPMIAB2G2BBOMNSM5EXONUL5BNORMR7KXIYSTQ7UFRVASVTQHQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.923941432 (41h1m32.395152161s) [noise=true , address=145.79.227.42:4121 , piece_id=6D47MY4GTQ3YQOGHS2V2OTO2VFDL6XKKS2R4VA6VZZ557WYFY34A, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.929131404 (41h1m32.400342133s) [noise=true , address=145.79.227.22:4510 , piece_id=HCJIUH3VECRBR744KIQ65UKSMTEEKUPHJWXSDBM5SEMUVBJ5YX6A, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.180555093 (41h1m32.651765822s) [noise=true , address=145.79.227.32:4835 , piece_id=PJURUANT2XJY5PNYTSO5W2EDT3HYFAC2BFNCANKOAZ3I6P2GRC3Q, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.587529048 (41h1m33.058739787s) [noise=true , address=145.79.227.41:4087 , piece_id=6GY2NC7F26VLU5NEN65OYU7YIHJYSTI5GNVRTNXM5HO5XQMZKEPA, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.671173874 (41h1m33.142384603s) [noise=true , address=145.79.227.27:4691 , piece_id=Z53YBOXTW4L23P4M3EOW6ADUHDWHIJVRDT7IL4I5FUPPZOW2MRLQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.671209982 (41h1m33.142420711s) [seg_pos={0 0}, upload=36107]: Segment reached optimal threshold of 13 pieces. Dec 5 09:56:56.671423689 (41h1m33.142634428s) [noise=true , address=145.79.227.24:4582 , piece_id=MCQWT2LFD472TQDOAHLGQ5L2X7S654P6TL4J5MSP3CML5ZCURHMA, seg_pos={0 0}, upload=36107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa): protocol: write tcp 188.166.225.170:54136->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:54136->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54136->145.79.227.24:4582: use of closed network connection Dec 5 09:56:56.671432325 (41h1m33.142643064s) [noise=true , address=145.79.227.24:4582 , piece_id=MCQWT2LFD472TQDOAHLGQ5L2X7S654P6TL4J5MSP3CML5ZCURHMA, seg_pos={0 0}, upload=36107]: Long tail cancellation due to optimal threshold Dec 5 09:56:56.671496377 (41h1m33.142707106s) [noise=true , address=145.79.227.44:4193 , piece_id=327UUYIASLKFGUIQVY4P5UML4R2R4ODLUAYXR3GISKIT6CFDUGAQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: write tcp 188.166.225.170:39276->145.79.227.44:4193: use of closed network connection; write tcp 188.166.225.170:39276->145.79.227.44:4193: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39276->145.79.227.44:4193: use of closed network connection Dec 5 09:56:56.671501627 (41h1m33.142712366s) [noise=true , address=145.79.227.44:4193 , piece_id=327UUYIASLKFGUIQVY4P5UML4R2R4ODLUAYXR3GISKIT6CFDUGAQ, seg_pos={0 0}, upload=36107]: Long tail cancellation due to optimal threshold Dec 5 09:56:56.671516466 (41h1m33.142727195s) [noise=true , address=77.90.31.16:4735 , piece_id=V4YJIHVMXYTBK7IHU272RYCKUXI44B2MXRLETMWMO2WAI2XRU4SQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: write tcp 188.166.225.170:41866->77.90.31.16:4735: use of closed network connection; storage node overloaded, request limit: 2000 Dec 5 09:56:56.671525944 (41h1m33.142736673s) [noise=true , address=77.90.31.16:4735 , piece_id=V4YJIHVMXYTBK7IHU272RYCKUXI44B2MXRLETMWMO2WAI2XRU4SQ, seg_pos={0 0}, upload=36107]: Long tail cancellation due to optimal threshold Dec 5 09:56:56.671548016 (41h1m33.142758745s) [seg_pos={0 0}, upload=36107]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 5 09:56:56.671623500 (41h1m33.142834229s) [upload=36107]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 5 09:56:56.671631204 (41h1m33.142841933s) [upload=36107]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 5 09:56:57.112987571 (41h1m33.5841983s) [upload=36107]: Done uploading segments... Dec 5 09:56:57.113007228 (41h1m33.584217967s) [upload=36107]: Upload finished. err: <nil> Duration: 6.454822193s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 9.427809963s trace: https://jaeger.dev.storj.io/trace/55b376cc76cfe920 FAIL (2025-12-05 10:22:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 9 (41h1m29.913335078s) [upload=36107]: Uploading segments... Dec 5 09:56:53.442159225 (41h1m29.913369955s) [upload=36107]: Got next segment. Inline: false Dec 5 09:56:53.442188120 (41h1m29.913398849s) [upload=36107]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 5 09:56:53.442194372 (41h1m29.913405101s) [upload=36107]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 5 09:56:53.701976567 (41h1m30.173187296s) [seg_pos={0 0}, upload=36107]: Begin upload segment... Dec 5 09:56:53.702020460 (41h1m30.173231189s) [seg_pos={0 0}, upload=36107]: Done begin upload segment. Dec 5 09:56:53.702056649 (41h1m30.173267388s) [noise=true , address=77.90.31.21:4878 , piece_id=RCFTETF7SGR6FTUBJWWBJDKOZIHZKBMSUBZYVWYKF2MP56PMIWMQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702064965 (41h1m30.173275694s) [noise=true , address=145.79.227.28:4725 , piece_id=ZE642TJU4ZJKBOHQ6K6GPLPTN2YKC4BOP325LN7QXALZSDPG66HQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702111533 (41h1m30.173322272s) [noise=true , address=145.79.227.42:4121 , piece_id=6D47MY4GTQ3YQOGHS2V2OTO2VFDL6XKKS2R4VA6VZZ557WYFY34A, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702548986 (41h1m30.173759766s) [noise=true , address=145.79.227.27:4691 , piece_id=Z53YBOXTW4L23P4M3EOW6ADUHDWHIJVRDT7IL4I5FUPPZOW2MRLQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702931455 (41h1m30.174142194s) [noise=true , address=77.90.31.16:4735 , piece_id=V4YJIHVMXYTBK7IHU272RYCKUXI44B2MXRLETMWMO2WAI2XRU4SQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.703301079 (41h1m30.174511808s) [noise=true , address=145.79.227.10:4401 , piece_id=XAVSV3SGPDDLZWGKQLHA2CVYTGOPO7J3YVEO5C3MYIN5SJEUZ2OQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.703633000 (41h1m30.174843729s) [noise=true , address=145.79.227.24:4583 , piece_id=IB6T3J3CF3P3A54JEHHBK6KEPMUEI32B4QYSSA2G2BUMIVUC7DGQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.703957648 (41h1m30.175168377s) [noise=true , address=145.79.227.26:4654 , piece_id=23NNREG47WSZWJNPZWP275L62YBBSTB6RQWTINIRTTAYU66DRT7Q, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.704788621 (41h1m30.17599936s) [noise=true , address=145.79.227.43:4159 , piece_id=S5T27VFA6UNNONPZBZILNTQUUNSRLWPREKOJORMLKOXWNL3GMBSQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.705457525 (41h1m30.176668264s) [noise=true , address=145.79.227.32:4835 , piece_id=PJURUANT2XJY5PNYTSO5W2EDT3HYFAC2BFNCANKOAZ3I6P2GRC3Q, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.706486825 (41h1m30.177697554s) [noise=true , address=145.79.227.24:4582 , piece_id=MCQWT2LFD472TQDOAHLGQ5L2X7S654P6TL4J5MSP3CML5ZCURHMA, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.707626675 (41h1m30.178837414s) [noise=true , address=145.79.227.44:4193 , piece_id=327UUYIASLKFGUIQVY4P5UML4R2R4ODLUAYXR3GISKIT6CFDUGAQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.708631078 (41h1m30.179841817s) [noise=true , address=145.79.227.40:4050 , piece_id=QFSFC3WFER7VBVZNB3XLRIEFWGM4Q5VTAAJIWG4HAERKXSHZGZ4Q, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.709908630 (41h1m30.181119359s) [noise=true , address=145.79.227.22:4510 , piece_id=HCJIUH3VECRBR744KIQ65UKSMTEEKUPHJWXSDBM5SEMUVBJ5YX6A, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.710331386 (41h1m30.181542125s) [noise=true , address=145.79.227.41:4087 , piece_id=6GY2NC7F26VLU5NEN65OYU7YIHJYSTI5GNVRTNXM5HO5XQMZKEPA, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.710803685 (41h1m30.182014424s) [noise=true , address=45.91.116.4:4937 , piece_id=VFAPMIAB2G2BBOMNSM5EXONUL5BNORMR7KXIYSTQ7UFRVASVTQHQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:55.513663514 (41h1m31.984874243s) [upload=36107]: Next returned nil segment Dec 5 09:56:55.513682680 (41h1m31.98489341s) [upload=36107]: Waiting for error group managing segments... Dec 5 09:56:55.808504186 (41h1m32.279714915s) [noise=true , address=77.90.31.21:4878 , piece_id=RCFTETF7SGR6FTUBJWWBJDKOZIHZKBMSUBZYVWYKF2MP56PMIWMQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.866933347 (41h1m32.338144086s) [noise=true , address=145.79.227.10:4401 , piece_id=XAVSV3SGPDDLZWGKQLHA2CVYTGOPO7J3YVEO5C3MYIN5SJEUZ2OQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.869510454 (41h1m32.340721183s) [noise=true , address=145.79.227.28:4725 , piece_id=ZE642TJU4ZJKBOHQ6K6GPLPTN2YKC4BOP325LN7QXALZSDPG66HQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.875082795 (41h1m32.346293524s) [noise=true , address=145.79.227.43:4159 , piece_id=S5T27VFA6UNNONPZBZILNTQUUNSRLWPREKOJORMLKOXWNL3GMBSQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.884542648 (41h1m32.355753387s) [noise=true , address=145.79.227.24:4583 , piece_id=IB6T3J3CF3P3A54JEHHBK6KEPMUEI32B4QYSSA2G2BUMIVUC7DGQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.888876321 (41h1m32.36008707s) [noise=true , address=145.79.227.40:4050 , piece_id=QFSFC3WFER7VBVZNB3XLRIEFWGM4Q5VTAAJIWG4HAERKXSHZGZ4Q, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.889468719 (41h1m32.360679448s) [noise=true , address=145.79.227.26:4654 , piece_id=23NNREG47WSZWJNPZWP275L62YBBSTB6RQWTINIRTTAYU66DRT7Q, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.900234388 (41h1m32.371445117s) [noise=true , address=45.91.116.4:4937 , piece_id=VFAPMIAB2G2BBOMNSM5EXONUL5BNORMR7KXIYSTQ7UFRVASVTQHQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.923941432 (41h1m32.395152161s) [noise=true , address=145.79.227.42:4121 , piece_id=6D47MY4GTQ3YQOGHS2V2OTO2VFDL6XKKS2R4VA6VZZ557WYFY34A, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.929131404 (41h1m32.400342133s) [noise=true , address=145.79.227.22:4510 , piece_id=HCJIUH3VECRBR744KIQ65UKSMTEEKUPHJWXSDBM5SEMUVBJ5YX6A, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.180555093 (41h1m32.651765822s) [noise=true , address=145.79.227.32:4835 , piece_id=PJURUANT2XJY5PNYTSO5W2EDT3HYFAC2BFNCANKOAZ3I6P2GRC3Q, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.587529048 (41h1m33.058739787s) [noise=true , address=145.79.227.41:4087 , piece_id=6GY2NC7F26VLU5NEN65OYU7YIHJYSTI5GNVRTNXM5HO5XQMZKEPA, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.671173874 (41h1m33.142384603s) [noise=true , address=145.79.227.27:4691 , piece_id=Z53YBOXTW4L23P4M3EOW6ADUHDWHIJVRDT7IL4I5FUPPZOW2MRLQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.671209982 (41h1m33.142420711s) [seg_pos={0 0}, upload=36107]: Segment reached optimal threshold of 13 pieces. Dec 5 09:56:56.671423689 (41h1m33.142634428s) [noise=true , address=145.79.227.24:4582 , piece_id=MCQWT2LFD472TQDOAHLGQ5L2X7S654P6TL4J5MSP3CML5ZCURHMA, seg_pos={0 0}, upload=36107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa): protocol: write tcp 188.166.225.170:54136->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:54136->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54136->145.79.227.24:4582: use of closed network connection Dec 5 09:56:56.671432325 (41h1m33.142643064s) [noise=true , address=145.79.227.24:4582 , piece_id=MCQWT2LFD472TQDOAHLGQ5L2X7S654P6TL4J5MSP3CML5ZCURHMA, seg_pos={0 0}, upload=36107]: Long tail cancellation due to optimal threshold Dec 5 09:56:56.671496377 (41h1m33.142707106s) [noise=true , address=145.79.227.44:4193 , piece_id=327UUYIASLKFGUIQVY4P5UML4R2R4ODLUAYXR3GISKIT6CFDUGAQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: write tcp 188.166.225.170:39276->145.79.227.44:4193: use of closed network connection; write tcp 188.166.225.170:39276->145.79.227.44:4193: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39276->145.79.227.44:4193: use of closed network connection Dec 5 09:56:56.671501627 (41h1m33.142712366s) [noise=true , address=145.79.227.44:4193 , piece_id=327UUYIASLKFGUIQVY4P5UML4R2R4ODLUAYXR3GISKIT6CFDUGAQ, seg_pos={0 0}, upload=36107]: Long tail cancellation due to optimal threshold Dec 5 09:56:56.671516466 (41h1m33.142727195s) [noise=true , address=77.90.31.16:4735 , piece_id=V4YJIHVMXYTBK7IHU272RYCKUXI44B2MXRLETMWMO2WAI2XRU4SQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: write tcp 188.166.225.170:41866->77.90.31.16:4735: use of closed network connection; storage node overloaded, request limit: 2000 Dec 5 09:56:56.671525944 (41h1m33.142736673s) [noise=true , address=77.90.31.16:4735 , piece_id=V4YJIHVMXYTBK7IHU272RYCKUXI44B2MXRLETMWMO2WAI2XRU4SQ, seg_pos={0 0}, upload=36107]: Long tail cancellation due to optimal threshold Dec 5 09:56:56.671548016 (41h1m33.142758745s) [seg_pos={0 0}, upload=36107]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 5 09:56:56.671623500 (41h1m33.142834229s) [upload=36107]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 5 09:56:56.671631204 (41h1m33.142841933s) [upload=36107]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 5 09:56:57.112987571 (41h1m33.5841983s) [upload=36107]: Done uploading segments... Dec 5 09:56:57.113007228 (41h1m33.584217967s) [upload=36107]: Upload finished. err: <nil> Duration: 9.427836383s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 6.303070353s trace: https://jaeger.dev.storj.io/trace/55b376cc76cfe920 FAIL (2025-12-05 10:21:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 9 (41h1m29.913335078s) [upload=36107]: Uploading segments... Dec 5 09:56:53.442159225 (41h1m29.913369955s) [upload=36107]: Got next segment. Inline: false Dec 5 09:56:53.442188120 (41h1m29.913398849s) [upload=36107]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 5 09:56:53.442194372 (41h1m29.913405101s) [upload=36107]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 5 09:56:53.701976567 (41h1m30.173187296s) [seg_pos={0 0}, upload=36107]: Begin upload segment... Dec 5 09:56:53.702020460 (41h1m30.173231189s) [seg_pos={0 0}, upload=36107]: Done begin upload segment. Dec 5 09:56:53.702056649 (41h1m30.173267388s) [noise=true , address=77.90.31.21:4878 , piece_id=RCFTETF7SGR6FTUBJWWBJDKOZIHZKBMSUBZYVWYKF2MP56PMIWMQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702064965 (41h1m30.173275694s) [noise=true , address=145.79.227.28:4725 , piece_id=ZE642TJU4ZJKBOHQ6K6GPLPTN2YKC4BOP325LN7QXALZSDPG66HQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702111533 (41h1m30.173322272s) [noise=true , address=145.79.227.42:4121 , piece_id=6D47MY4GTQ3YQOGHS2V2OTO2VFDL6XKKS2R4VA6VZZ557WYFY34A, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702548986 (41h1m30.173759766s) [noise=true , address=145.79.227.27:4691 , piece_id=Z53YBOXTW4L23P4M3EOW6ADUHDWHIJVRDT7IL4I5FUPPZOW2MRLQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702931455 (41h1m30.174142194s) [noise=true , address=77.90.31.16:4735 , piece_id=V4YJIHVMXYTBK7IHU272RYCKUXI44B2MXRLETMWMO2WAI2XRU4SQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.703301079 (41h1m30.174511808s) [noise=true , address=145.79.227.10:4401 , piece_id=XAVSV3SGPDDLZWGKQLHA2CVYTGOPO7J3YVEO5C3MYIN5SJEUZ2OQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.703633000 (41h1m30.174843729s) [noise=true , address=145.79.227.24:4583 , piece_id=IB6T3J3CF3P3A54JEHHBK6KEPMUEI32B4QYSSA2G2BUMIVUC7DGQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.703957648 (41h1m30.175168377s) [noise=true , address=145.79.227.26:4654 , piece_id=23NNREG47WSZWJNPZWP275L62YBBSTB6RQWTINIRTTAYU66DRT7Q, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.704788621 (41h1m30.17599936s) [noise=true , address=145.79.227.43:4159 , piece_id=S5T27VFA6UNNONPZBZILNTQUUNSRLWPREKOJORMLKOXWNL3GMBSQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.705457525 (41h1m30.176668264s) [noise=true , address=145.79.227.32:4835 , piece_id=PJURUANT2XJY5PNYTSO5W2EDT3HYFAC2BFNCANKOAZ3I6P2GRC3Q, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.706486825 (41h1m30.177697554s) [noise=true , address=145.79.227.24:4582 , piece_id=MCQWT2LFD472TQDOAHLGQ5L2X7S654P6TL4J5MSP3CML5ZCURHMA, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.707626675 (41h1m30.178837414s) [noise=true , address=145.79.227.44:4193 , piece_id=327UUYIASLKFGUIQVY4P5UML4R2R4ODLUAYXR3GISKIT6CFDUGAQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.708631078 (41h1m30.179841817s) [noise=true , address=145.79.227.40:4050 , piece_id=QFSFC3WFER7VBVZNB3XLRIEFWGM4Q5VTAAJIWG4HAERKXSHZGZ4Q, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.709908630 (41h1m30.181119359s) [noise=true , address=145.79.227.22:4510 , piece_id=HCJIUH3VECRBR744KIQ65UKSMTEEKUPHJWXSDBM5SEMUVBJ5YX6A, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.710331386 (41h1m30.181542125s) [noise=true , address=145.79.227.41:4087 , piece_id=6GY2NC7F26VLU5NEN65OYU7YIHJYSTI5GNVRTNXM5HO5XQMZKEPA, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.710803685 (41h1m30.182014424s) [noise=true , address=45.91.116.4:4937 , piece_id=VFAPMIAB2G2BBOMNSM5EXONUL5BNORMR7KXIYSTQ7UFRVASVTQHQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:55.513663514 (41h1m31.984874243s) [upload=36107]: Next returned nil segment Dec 5 09:56:55.513682680 (41h1m31.98489341s) [upload=36107]: Waiting for error group managing segments... Dec 5 09:56:55.808504186 (41h1m32.279714915s) [noise=true , address=77.90.31.21:4878 , piece_id=RCFTETF7SGR6FTUBJWWBJDKOZIHZKBMSUBZYVWYKF2MP56PMIWMQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.866933347 (41h1m32.338144086s) [noise=true , address=145.79.227.10:4401 , piece_id=XAVSV3SGPDDLZWGKQLHA2CVYTGOPO7J3YVEO5C3MYIN5SJEUZ2OQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.869510454 (41h1m32.340721183s) [noise=true , address=145.79.227.28:4725 , piece_id=ZE642TJU4ZJKBOHQ6K6GPLPTN2YKC4BOP325LN7QXALZSDPG66HQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.875082795 (41h1m32.346293524s) [noise=true , address=145.79.227.43:4159 , piece_id=S5T27VFA6UNNONPZBZILNTQUUNSRLWPREKOJORMLKOXWNL3GMBSQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.884542648 (41h1m32.355753387s) [noise=true , address=145.79.227.24:4583 , piece_id=IB6T3J3CF3P3A54JEHHBK6KEPMUEI32B4QYSSA2G2BUMIVUC7DGQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.888876321 (41h1m32.36008707s) [noise=true , address=145.79.227.40:4050 , piece_id=QFSFC3WFER7VBVZNB3XLRIEFWGM4Q5VTAAJIWG4HAERKXSHZGZ4Q, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.889468719 (41h1m32.360679448s) [noise=true , address=145.79.227.26:4654 , piece_id=23NNREG47WSZWJNPZWP275L62YBBSTB6RQWTINIRTTAYU66DRT7Q, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.900234388 (41h1m32.371445117s) [noise=true , address=45.91.116.4:4937 , piece_id=VFAPMIAB2G2BBOMNSM5EXONUL5BNORMR7KXIYSTQ7UFRVASVTQHQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.923941432 (41h1m32.395152161s) [noise=true , address=145.79.227.42:4121 , piece_id=6D47MY4GTQ3YQOGHS2V2OTO2VFDL6XKKS2R4VA6VZZ557WYFY34A, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.929131404 (41h1m32.400342133s) [noise=true , address=145.79.227.22:4510 , piece_id=HCJIUH3VECRBR744KIQ65UKSMTEEKUPHJWXSDBM5SEMUVBJ5YX6A, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.180555093 (41h1m32.651765822s) [noise=true , address=145.79.227.32:4835 , piece_id=PJURUANT2XJY5PNYTSO5W2EDT3HYFAC2BFNCANKOAZ3I6P2GRC3Q, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.587529048 (41h1m33.058739787s) [noise=true , address=145.79.227.41:4087 , piece_id=6GY2NC7F26VLU5NEN65OYU7YIHJYSTI5GNVRTNXM5HO5XQMZKEPA, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.671173874 (41h1m33.142384603s) [noise=true , address=145.79.227.27:4691 , piece_id=Z53YBOXTW4L23P4M3EOW6ADUHDWHIJVRDT7IL4I5FUPPZOW2MRLQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.671209982 (41h1m33.142420711s) [seg_pos={0 0}, upload=36107]: Segment reached optimal threshold of 13 pieces. Dec 5 09:56:56.671423689 (41h1m33.142634428s) [noise=true , address=145.79.227.24:4582 , piece_id=MCQWT2LFD472TQDOAHLGQ5L2X7S654P6TL4J5MSP3CML5ZCURHMA, seg_pos={0 0}, upload=36107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa): protocol: write tcp 188.166.225.170:54136->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:54136->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54136->145.79.227.24:4582: use of closed network connection Dec 5 09:56:56.671432325 (41h1m33.142643064s) [noise=true , address=145.79.227.24:4582 , piece_id=MCQWT2LFD472TQDOAHLGQ5L2X7S654P6TL4J5MSP3CML5ZCURHMA, seg_pos={0 0}, upload=36107]: Long tail cancellation due to optimal threshold Dec 5 09:56:56.671496377 (41h1m33.142707106s) [noise=true , address=145.79.227.44:4193 , piece_id=327UUYIASLKFGUIQVY4P5UML4R2R4ODLUAYXR3GISKIT6CFDUGAQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: write tcp 188.166.225.170:39276->145.79.227.44:4193: use of closed network connection; write tcp 188.166.225.170:39276->145.79.227.44:4193: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39276->145.79.227.44:4193: use of closed network connection Dec 5 09:56:56.671501627 (41h1m33.142712366s) [noise=true , address=145.79.227.44:4193 , piece_id=327UUYIASLKFGUIQVY4P5UML4R2R4ODLUAYXR3GISKIT6CFDUGAQ, seg_pos={0 0}, upload=36107]: Long tail cancellation due to optimal threshold Dec 5 09:56:56.671516466 (41h1m33.142727195s) [noise=true , address=77.90.31.16:4735 , piece_id=V4YJIHVMXYTBK7IHU272RYCKUXI44B2MXRLETMWMO2WAI2XRU4SQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: write tcp 188.166.225.170:41866->77.90.31.16:4735: use of closed network connection; storage node overloaded, request limit: 2000 Dec 5 09:56:56.671525944 (41h1m33.142736673s) [noise=true , address=77.90.31.16:4735 , piece_id=V4YJIHVMXYTBK7IHU272RYCKUXI44B2MXRLETMWMO2WAI2XRU4SQ, seg_pos={0 0}, upload=36107]: Long tail cancellation due to optimal threshold Dec 5 09:56:56.671548016 (41h1m33.142758745s) [seg_pos={0 0}, upload=36107]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 5 09:56:56.671623500 (41h1m33.142834229s) [upload=36107]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 5 09:56:56.671631204 (41h1m33.142841933s) [upload=36107]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 5 09:56:57.112987571 (41h1m33.5841983s) [upload=36107]: Done uploading segments... Dec 5 09:56:57.113007228 (41h1m33.584217967s) [upload=36107]: Upload finished. err: <nil> Duration: 6.303096392s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 9.59584953s trace: https://jaeger.dev.storj.io/trace/55b376cc76cfe920 FAIL (2025-12-05 10:20:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 9 (41h1m29.913335078s) [upload=36107]: Uploading segments... Dec 5 09:56:53.442159225 (41h1m29.913369955s) [upload=36107]: Got next segment. Inline: false Dec 5 09:56:53.442188120 (41h1m29.913398849s) [upload=36107]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 5 09:56:53.442194372 (41h1m29.913405101s) [upload=36107]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 5 09:56:53.701976567 (41h1m30.173187296s) [seg_pos={0 0}, upload=36107]: Begin upload segment... Dec 5 09:56:53.702020460 (41h1m30.173231189s) [seg_pos={0 0}, upload=36107]: Done begin upload segment. Dec 5 09:56:53.702056649 (41h1m30.173267388s) [noise=true , address=77.90.31.21:4878 , piece_id=RCFTETF7SGR6FTUBJWWBJDKOZIHZKBMSUBZYVWYKF2MP56PMIWMQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702064965 (41h1m30.173275694s) [noise=true , address=145.79.227.28:4725 , piece_id=ZE642TJU4ZJKBOHQ6K6GPLPTN2YKC4BOP325LN7QXALZSDPG66HQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702111533 (41h1m30.173322272s) [noise=true , address=145.79.227.42:4121 , piece_id=6D47MY4GTQ3YQOGHS2V2OTO2VFDL6XKKS2R4VA6VZZ557WYFY34A, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702548986 (41h1m30.173759766s) [noise=true , address=145.79.227.27:4691 , piece_id=Z53YBOXTW4L23P4M3EOW6ADUHDWHIJVRDT7IL4I5FUPPZOW2MRLQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702931455 (41h1m30.174142194s) [noise=true , address=77.90.31.16:4735 , piece_id=V4YJIHVMXYTBK7IHU272RYCKUXI44B2MXRLETMWMO2WAI2XRU4SQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.703301079 (41h1m30.174511808s) [noise=true , address=145.79.227.10:4401 , piece_id=XAVSV3SGPDDLZWGKQLHA2CVYTGOPO7J3YVEO5C3MYIN5SJEUZ2OQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.703633000 (41h1m30.174843729s) [noise=true , address=145.79.227.24:4583 , piece_id=IB6T3J3CF3P3A54JEHHBK6KEPMUEI32B4QYSSA2G2BUMIVUC7DGQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.703957648 (41h1m30.175168377s) [noise=true , address=145.79.227.26:4654 , piece_id=23NNREG47WSZWJNPZWP275L62YBBSTB6RQWTINIRTTAYU66DRT7Q, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.704788621 (41h1m30.17599936s) [noise=true , address=145.79.227.43:4159 , piece_id=S5T27VFA6UNNONPZBZILNTQUUNSRLWPREKOJORMLKOXWNL3GMBSQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.705457525 (41h1m30.176668264s) [noise=true , address=145.79.227.32:4835 , piece_id=PJURUANT2XJY5PNYTSO5W2EDT3HYFAC2BFNCANKOAZ3I6P2GRC3Q, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.706486825 (41h1m30.177697554s) [noise=true , address=145.79.227.24:4582 , piece_id=MCQWT2LFD472TQDOAHLGQ5L2X7S654P6TL4J5MSP3CML5ZCURHMA, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.707626675 (41h1m30.178837414s) [noise=true , address=145.79.227.44:4193 , piece_id=327UUYIASLKFGUIQVY4P5UML4R2R4ODLUAYXR3GISKIT6CFDUGAQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.708631078 (41h1m30.179841817s) [noise=true , address=145.79.227.40:4050 , piece_id=QFSFC3WFER7VBVZNB3XLRIEFWGM4Q5VTAAJIWG4HAERKXSHZGZ4Q, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.709908630 (41h1m30.181119359s) [noise=true , address=145.79.227.22:4510 , piece_id=HCJIUH3VECRBR744KIQ65UKSMTEEKUPHJWXSDBM5SEMUVBJ5YX6A, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.710331386 (41h1m30.181542125s) [noise=true , address=145.79.227.41:4087 , piece_id=6GY2NC7F26VLU5NEN65OYU7YIHJYSTI5GNVRTNXM5HO5XQMZKEPA, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.710803685 (41h1m30.182014424s) [noise=true , address=45.91.116.4:4937 , piece_id=VFAPMIAB2G2BBOMNSM5EXONUL5BNORMR7KXIYSTQ7UFRVASVTQHQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:55.513663514 (41h1m31.984874243s) [upload=36107]: Next returned nil segment Dec 5 09:56:55.513682680 (41h1m31.98489341s) [upload=36107]: Waiting for error group managing segments... Dec 5 09:56:55.808504186 (41h1m32.279714915s) [noise=true , address=77.90.31.21:4878 , piece_id=RCFTETF7SGR6FTUBJWWBJDKOZIHZKBMSUBZYVWYKF2MP56PMIWMQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.866933347 (41h1m32.338144086s) [noise=true , address=145.79.227.10:4401 , piece_id=XAVSV3SGPDDLZWGKQLHA2CVYTGOPO7J3YVEO5C3MYIN5SJEUZ2OQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.869510454 (41h1m32.340721183s) [noise=true , address=145.79.227.28:4725 , piece_id=ZE642TJU4ZJKBOHQ6K6GPLPTN2YKC4BOP325LN7QXALZSDPG66HQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.875082795 (41h1m32.346293524s) [noise=true , address=145.79.227.43:4159 , piece_id=S5T27VFA6UNNONPZBZILNTQUUNSRLWPREKOJORMLKOXWNL3GMBSQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.884542648 (41h1m32.355753387s) [noise=true , address=145.79.227.24:4583 , piece_id=IB6T3J3CF3P3A54JEHHBK6KEPMUEI32B4QYSSA2G2BUMIVUC7DGQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.888876321 (41h1m32.36008707s) [noise=true , address=145.79.227.40:4050 , piece_id=QFSFC3WFER7VBVZNB3XLRIEFWGM4Q5VTAAJIWG4HAERKXSHZGZ4Q, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.889468719 (41h1m32.360679448s) [noise=true , address=145.79.227.26:4654 , piece_id=23NNREG47WSZWJNPZWP275L62YBBSTB6RQWTINIRTTAYU66DRT7Q, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.900234388 (41h1m32.371445117s) [noise=true , address=45.91.116.4:4937 , piece_id=VFAPMIAB2G2BBOMNSM5EXONUL5BNORMR7KXIYSTQ7UFRVASVTQHQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.923941432 (41h1m32.395152161s) [noise=true , address=145.79.227.42:4121 , piece_id=6D47MY4GTQ3YQOGHS2V2OTO2VFDL6XKKS2R4VA6VZZ557WYFY34A, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.929131404 (41h1m32.400342133s) [noise=true , address=145.79.227.22:4510 , piece_id=HCJIUH3VECRBR744KIQ65UKSMTEEKUPHJWXSDBM5SEMUVBJ5YX6A, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.180555093 (41h1m32.651765822s) [noise=true , address=145.79.227.32:4835 , piece_id=PJURUANT2XJY5PNYTSO5W2EDT3HYFAC2BFNCANKOAZ3I6P2GRC3Q, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.587529048 (41h1m33.058739787s) [noise=true , address=145.79.227.41:4087 , piece_id=6GY2NC7F26VLU5NEN65OYU7YIHJYSTI5GNVRTNXM5HO5XQMZKEPA, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.671173874 (41h1m33.142384603s) [noise=true , address=145.79.227.27:4691 , piece_id=Z53YBOXTW4L23P4M3EOW6ADUHDWHIJVRDT7IL4I5FUPPZOW2MRLQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.671209982 (41h1m33.142420711s) [seg_pos={0 0}, upload=36107]: Segment reached optimal threshold of 13 pieces. Dec 5 09:56:56.671423689 (41h1m33.142634428s) [noise=true , address=145.79.227.24:4582 , piece_id=MCQWT2LFD472TQDOAHLGQ5L2X7S654P6TL4J5MSP3CML5ZCURHMA, seg_pos={0 0}, upload=36107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa): protocol: write tcp 188.166.225.170:54136->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:54136->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54136->145.79.227.24:4582: use of closed network connection Dec 5 09:56:56.671432325 (41h1m33.142643064s) [noise=true , address=145.79.227.24:4582 , piece_id=MCQWT2LFD472TQDOAHLGQ5L2X7S654P6TL4J5MSP3CML5ZCURHMA, seg_pos={0 0}, upload=36107]: Long tail cancellation due to optimal threshold Dec 5 09:56:56.671496377 (41h1m33.142707106s) [noise=true , address=145.79.227.44:4193 , piece_id=327UUYIASLKFGUIQVY4P5UML4R2R4ODLUAYXR3GISKIT6CFDUGAQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: write tcp 188.166.225.170:39276->145.79.227.44:4193: use of closed network connection; write tcp 188.166.225.170:39276->145.79.227.44:4193: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39276->145.79.227.44:4193: use of closed network connection Dec 5 09:56:56.671501627 (41h1m33.142712366s) [noise=true , address=145.79.227.44:4193 , piece_id=327UUYIASLKFGUIQVY4P5UML4R2R4ODLUAYXR3GISKIT6CFDUGAQ, seg_pos={0 0}, upload=36107]: Long tail cancellation due to optimal threshold Dec 5 09:56:56.671516466 (41h1m33.142727195s) [noise=true , address=77.90.31.16:4735 , piece_id=V4YJIHVMXYTBK7IHU272RYCKUXI44B2MXRLETMWMO2WAI2XRU4SQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: write tcp 188.166.225.170:41866->77.90.31.16:4735: use of closed network connection; storage node overloaded, request limit: 2000 Dec 5 09:56:56.671525944 (41h1m33.142736673s) [noise=true , address=77.90.31.16:4735 , piece_id=V4YJIHVMXYTBK7IHU272RYCKUXI44B2MXRLETMWMO2WAI2XRU4SQ, seg_pos={0 0}, upload=36107]: Long tail cancellation due to optimal threshold Dec 5 09:56:56.671548016 (41h1m33.142758745s) [seg_pos={0 0}, upload=36107]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 5 09:56:56.671623500 (41h1m33.142834229s) [upload=36107]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 5 09:56:56.671631204 (41h1m33.142841933s) [upload=36107]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 5 09:56:57.112987571 (41h1m33.5841983s) [upload=36107]: Done uploading segments... Dec 5 09:56:57.113007228 (41h1m33.584217967s) [upload=36107]: Upload finished. err: <nil> Duration: 9.595866563s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 8.99877104s trace: https://jaeger.dev.storj.io/trace/55b376cc76cfe920 FAIL (2025-12-05 10:20:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 9 (41h1m29.913335078s) [upload=36107]: Uploading segments... Dec 5 09:56:53.442159225 (41h1m29.913369955s) [upload=36107]: Got next segment. Inline: false Dec 5 09:56:53.442188120 (41h1m29.913398849s) [upload=36107]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 5 09:56:53.442194372 (41h1m29.913405101s) [upload=36107]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 5 09:56:53.701976567 (41h1m30.173187296s) [seg_pos={0 0}, upload=36107]: Begin upload segment... Dec 5 09:56:53.702020460 (41h1m30.173231189s) [seg_pos={0 0}, upload=36107]: Done begin upload segment. Dec 5 09:56:53.702056649 (41h1m30.173267388s) [noise=true , address=77.90.31.21:4878 , piece_id=RCFTETF7SGR6FTUBJWWBJDKOZIHZKBMSUBZYVWYKF2MP56PMIWMQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702064965 (41h1m30.173275694s) [noise=true , address=145.79.227.28:4725 , piece_id=ZE642TJU4ZJKBOHQ6K6GPLPTN2YKC4BOP325LN7QXALZSDPG66HQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702111533 (41h1m30.173322272s) [noise=true , address=145.79.227.42:4121 , piece_id=6D47MY4GTQ3YQOGHS2V2OTO2VFDL6XKKS2R4VA6VZZ557WYFY34A, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702548986 (41h1m30.173759766s) [noise=true , address=145.79.227.27:4691 , piece_id=Z53YBOXTW4L23P4M3EOW6ADUHDWHIJVRDT7IL4I5FUPPZOW2MRLQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702931455 (41h1m30.174142194s) [noise=true , address=77.90.31.16:4735 , piece_id=V4YJIHVMXYTBK7IHU272RYCKUXI44B2MXRLETMWMO2WAI2XRU4SQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.703301079 (41h1m30.174511808s) [noise=true , address=145.79.227.10:4401 , piece_id=XAVSV3SGPDDLZWGKQLHA2CVYTGOPO7J3YVEO5C3MYIN5SJEUZ2OQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.703633000 (41h1m30.174843729s) [noise=true , address=145.79.227.24:4583 , piece_id=IB6T3J3CF3P3A54JEHHBK6KEPMUEI32B4QYSSA2G2BUMIVUC7DGQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.703957648 (41h1m30.175168377s) [noise=true , address=145.79.227.26:4654 , piece_id=23NNREG47WSZWJNPZWP275L62YBBSTB6RQWTINIRTTAYU66DRT7Q, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.704788621 (41h1m30.17599936s) [noise=true , address=145.79.227.43:4159 , piece_id=S5T27VFA6UNNONPZBZILNTQUUNSRLWPREKOJORMLKOXWNL3GMBSQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.705457525 (41h1m30.176668264s) [noise=true , address=145.79.227.32:4835 , piece_id=PJURUANT2XJY5PNYTSO5W2EDT3HYFAC2BFNCANKOAZ3I6P2GRC3Q, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.706486825 (41h1m30.177697554s) [noise=true , address=145.79.227.24:4582 , piece_id=MCQWT2LFD472TQDOAHLGQ5L2X7S654P6TL4J5MSP3CML5ZCURHMA, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.707626675 (41h1m30.178837414s) [noise=true , address=145.79.227.44:4193 , piece_id=327UUYIASLKFGUIQVY4P5UML4R2R4ODLUAYXR3GISKIT6CFDUGAQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.708631078 (41h1m30.179841817s) [noise=true , address=145.79.227.40:4050 , piece_id=QFSFC3WFER7VBVZNB3XLRIEFWGM4Q5VTAAJIWG4HAERKXSHZGZ4Q, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.709908630 (41h1m30.181119359s) [noise=true , address=145.79.227.22:4510 , piece_id=HCJIUH3VECRBR744KIQ65UKSMTEEKUPHJWXSDBM5SEMUVBJ5YX6A, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.710331386 (41h1m30.181542125s) [noise=true , address=145.79.227.41:4087 , piece_id=6GY2NC7F26VLU5NEN65OYU7YIHJYSTI5GNVRTNXM5HO5XQMZKEPA, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.710803685 (41h1m30.182014424s) [noise=true , address=45.91.116.4:4937 , piece_id=VFAPMIAB2G2BBOMNSM5EXONUL5BNORMR7KXIYSTQ7UFRVASVTQHQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:55.513663514 (41h1m31.984874243s) [upload=36107]: Next returned nil segment Dec 5 09:56:55.513682680 (41h1m31.98489341s) [upload=36107]: Waiting for error group managing segments... Dec 5 09:56:55.808504186 (41h1m32.279714915s) [noise=true , address=77.90.31.21:4878 , piece_id=RCFTETF7SGR6FTUBJWWBJDKOZIHZKBMSUBZYVWYKF2MP56PMIWMQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.866933347 (41h1m32.338144086s) [noise=true , address=145.79.227.10:4401 , piece_id=XAVSV3SGPDDLZWGKQLHA2CVYTGOPO7J3YVEO5C3MYIN5SJEUZ2OQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.869510454 (41h1m32.340721183s) [noise=true , address=145.79.227.28:4725 , piece_id=ZE642TJU4ZJKBOHQ6K6GPLPTN2YKC4BOP325LN7QXALZSDPG66HQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.875082795 (41h1m32.346293524s) [noise=true , address=145.79.227.43:4159 , piece_id=S5T27VFA6UNNONPZBZILNTQUUNSRLWPREKOJORMLKOXWNL3GMBSQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.884542648 (41h1m32.355753387s) [noise=true , address=145.79.227.24:4583 , piece_id=IB6T3J3CF3P3A54JEHHBK6KEPMUEI32B4QYSSA2G2BUMIVUC7DGQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.888876321 (41h1m32.36008707s) [noise=true , address=145.79.227.40:4050 , piece_id=QFSFC3WFER7VBVZNB3XLRIEFWGM4Q5VTAAJIWG4HAERKXSHZGZ4Q, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.889468719 (41h1m32.360679448s) [noise=true , address=145.79.227.26:4654 , piece_id=23NNREG47WSZWJNPZWP275L62YBBSTB6RQWTINIRTTAYU66DRT7Q, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.900234388 (41h1m32.371445117s) [noise=true , address=45.91.116.4:4937 , piece_id=VFAPMIAB2G2BBOMNSM5EXONUL5BNORMR7KXIYSTQ7UFRVASVTQHQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.923941432 (41h1m32.395152161s) [noise=true , address=145.79.227.42:4121 , piece_id=6D47MY4GTQ3YQOGHS2V2OTO2VFDL6XKKS2R4VA6VZZ557WYFY34A, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.929131404 (41h1m32.400342133s) [noise=true , address=145.79.227.22:4510 , piece_id=HCJIUH3VECRBR744KIQ65UKSMTEEKUPHJWXSDBM5SEMUVBJ5YX6A, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.180555093 (41h1m32.651765822s) [noise=true , address=145.79.227.32:4835 , piece_id=PJURUANT2XJY5PNYTSO5W2EDT3HYFAC2BFNCANKOAZ3I6P2GRC3Q, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.587529048 (41h1m33.058739787s) [noise=true , address=145.79.227.41:4087 , piece_id=6GY2NC7F26VLU5NEN65OYU7YIHJYSTI5GNVRTNXM5HO5XQMZKEPA, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.671173874 (41h1m33.142384603s) [noise=true , address=145.79.227.27:4691 , piece_id=Z53YBOXTW4L23P4M3EOW6ADUHDWHIJVRDT7IL4I5FUPPZOW2MRLQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.671209982 (41h1m33.142420711s) [seg_pos={0 0}, upload=36107]: Segment reached optimal threshold of 13 pieces. Dec 5 09:56:56.671423689 (41h1m33.142634428s) [noise=true , address=145.79.227.24:4582 , piece_id=MCQWT2LFD472TQDOAHLGQ5L2X7S654P6TL4J5MSP3CML5ZCURHMA, seg_pos={0 0}, upload=36107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa): protocol: write tcp 188.166.225.170:54136->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:54136->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54136->145.79.227.24:4582: use of closed network connection Dec 5 09:56:56.671432325 (41h1m33.142643064s) [noise=true , address=145.79.227.24:4582 , piece_id=MCQWT2LFD472TQDOAHLGQ5L2X7S654P6TL4J5MSP3CML5ZCURHMA, seg_pos={0 0}, upload=36107]: Long tail cancellation due to optimal threshold Dec 5 09:56:56.671496377 (41h1m33.142707106s) [noise=true , address=145.79.227.44:4193 , piece_id=327UUYIASLKFGUIQVY4P5UML4R2R4ODLUAYXR3GISKIT6CFDUGAQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: write tcp 188.166.225.170:39276->145.79.227.44:4193: use of closed network connection; write tcp 188.166.225.170:39276->145.79.227.44:4193: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39276->145.79.227.44:4193: use of closed network connection Dec 5 09:56:56.671501627 (41h1m33.142712366s) [noise=true , address=145.79.227.44:4193 , piece_id=327UUYIASLKFGUIQVY4P5UML4R2R4ODLUAYXR3GISKIT6CFDUGAQ, seg_pos={0 0}, upload=36107]: Long tail cancellation due to optimal threshold Dec 5 09:56:56.671516466 (41h1m33.142727195s) [noise=true , address=77.90.31.16:4735 , piece_id=V4YJIHVMXYTBK7IHU272RYCKUXI44B2MXRLETMWMO2WAI2XRU4SQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: write tcp 188.166.225.170:41866->77.90.31.16:4735: use of closed network connection; storage node overloaded, request limit: 2000 Dec 5 09:56:56.671525944 (41h1m33.142736673s) [noise=true , address=77.90.31.16:4735 , piece_id=V4YJIHVMXYTBK7IHU272RYCKUXI44B2MXRLETMWMO2WAI2XRU4SQ, seg_pos={0 0}, upload=36107]: Long tail cancellation due to optimal threshold Dec 5 09:56:56.671548016 (41h1m33.142758745s) [seg_pos={0 0}, upload=36107]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 5 09:56:56.671623500 (41h1m33.142834229s) [upload=36107]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 5 09:56:56.671631204 (41h1m33.142841933s) [upload=36107]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 5 09:56:57.112987571 (41h1m33.5841983s) [upload=36107]: Done uploading segments... Dec 5 09:56:57.113007228 (41h1m33.584217967s) [upload=36107]: Upload finished. err: <nil> Duration: 8.998807148s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 11.163889329s trace: https://jaeger.dev.storj.io/trace/55b376cc76cfe920 FAIL (2025-12-05 10:19:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 9 (41h1m29.913335078s) [upload=36107]: Uploading segments... Dec 5 09:56:53.442159225 (41h1m29.913369955s) [upload=36107]: Got next segment. Inline: false Dec 5 09:56:53.442188120 (41h1m29.913398849s) [upload=36107]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 5 09:56:53.442194372 (41h1m29.913405101s) [upload=36107]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 5 09:56:53.701976567 (41h1m30.173187296s) [seg_pos={0 0}, upload=36107]: Begin upload segment... Dec 5 09:56:53.702020460 (41h1m30.173231189s) [seg_pos={0 0}, upload=36107]: Done begin upload segment. Dec 5 09:56:53.702056649 (41h1m30.173267388s) [noise=true , address=77.90.31.21:4878 , piece_id=RCFTETF7SGR6FTUBJWWBJDKOZIHZKBMSUBZYVWYKF2MP56PMIWMQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702064965 (41h1m30.173275694s) [noise=true , address=145.79.227.28:4725 , piece_id=ZE642TJU4ZJKBOHQ6K6GPLPTN2YKC4BOP325LN7QXALZSDPG66HQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702111533 (41h1m30.173322272s) [noise=true , address=145.79.227.42:4121 , piece_id=6D47MY4GTQ3YQOGHS2V2OTO2VFDL6XKKS2R4VA6VZZ557WYFY34A, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702548986 (41h1m30.173759766s) [noise=true , address=145.79.227.27:4691 , piece_id=Z53YBOXTW4L23P4M3EOW6ADUHDWHIJVRDT7IL4I5FUPPZOW2MRLQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702931455 (41h1m30.174142194s) [noise=true , address=77.90.31.16:4735 , piece_id=V4YJIHVMXYTBK7IHU272RYCKUXI44B2MXRLETMWMO2WAI2XRU4SQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.703301079 (41h1m30.174511808s) [noise=true , address=145.79.227.10:4401 , piece_id=XAVSV3SGPDDLZWGKQLHA2CVYTGOPO7J3YVEO5C3MYIN5SJEUZ2OQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.703633000 (41h1m30.174843729s) [noise=true , address=145.79.227.24:4583 , piece_id=IB6T3J3CF3P3A54JEHHBK6KEPMUEI32B4QYSSA2G2BUMIVUC7DGQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.703957648 (41h1m30.175168377s) [noise=true , address=145.79.227.26:4654 , piece_id=23NNREG47WSZWJNPZWP275L62YBBSTB6RQWTINIRTTAYU66DRT7Q, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.704788621 (41h1m30.17599936s) [noise=true , address=145.79.227.43:4159 , piece_id=S5T27VFA6UNNONPZBZILNTQUUNSRLWPREKOJORMLKOXWNL3GMBSQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.705457525 (41h1m30.176668264s) [noise=true , address=145.79.227.32:4835 , piece_id=PJURUANT2XJY5PNYTSO5W2EDT3HYFAC2BFNCANKOAZ3I6P2GRC3Q, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.706486825 (41h1m30.177697554s) [noise=true , address=145.79.227.24:4582 , piece_id=MCQWT2LFD472TQDOAHLGQ5L2X7S654P6TL4J5MSP3CML5ZCURHMA, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.707626675 (41h1m30.178837414s) [noise=true , address=145.79.227.44:4193 , piece_id=327UUYIASLKFGUIQVY4P5UML4R2R4ODLUAYXR3GISKIT6CFDUGAQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.708631078 (41h1m30.179841817s) [noise=true , address=145.79.227.40:4050 , piece_id=QFSFC3WFER7VBVZNB3XLRIEFWGM4Q5VTAAJIWG4HAERKXSHZGZ4Q, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.709908630 (41h1m30.181119359s) [noise=true , address=145.79.227.22:4510 , piece_id=HCJIUH3VECRBR744KIQ65UKSMTEEKUPHJWXSDBM5SEMUVBJ5YX6A, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.710331386 (41h1m30.181542125s) [noise=true , address=145.79.227.41:4087 , piece_id=6GY2NC7F26VLU5NEN65OYU7YIHJYSTI5GNVRTNXM5HO5XQMZKEPA, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.710803685 (41h1m30.182014424s) [noise=true , address=45.91.116.4:4937 , piece_id=VFAPMIAB2G2BBOMNSM5EXONUL5BNORMR7KXIYSTQ7UFRVASVTQHQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:55.513663514 (41h1m31.984874243s) [upload=36107]: Next returned nil segment Dec 5 09:56:55.513682680 (41h1m31.98489341s) [upload=36107]: Waiting for error group managing segments... Dec 5 09:56:55.808504186 (41h1m32.279714915s) [noise=true , address=77.90.31.21:4878 , piece_id=RCFTETF7SGR6FTUBJWWBJDKOZIHZKBMSUBZYVWYKF2MP56PMIWMQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.866933347 (41h1m32.338144086s) [noise=true , address=145.79.227.10:4401 , piece_id=XAVSV3SGPDDLZWGKQLHA2CVYTGOPO7J3YVEO5C3MYIN5SJEUZ2OQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.869510454 (41h1m32.340721183s) [noise=true , address=145.79.227.28:4725 , piece_id=ZE642TJU4ZJKBOHQ6K6GPLPTN2YKC4BOP325LN7QXALZSDPG66HQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.875082795 (41h1m32.346293524s) [noise=true , address=145.79.227.43:4159 , piece_id=S5T27VFA6UNNONPZBZILNTQUUNSRLWPREKOJORMLKOXWNL3GMBSQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.884542648 (41h1m32.355753387s) [noise=true , address=145.79.227.24:4583 , piece_id=IB6T3J3CF3P3A54JEHHBK6KEPMUEI32B4QYSSA2G2BUMIVUC7DGQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.888876321 (41h1m32.36008707s) [noise=true , address=145.79.227.40:4050 , piece_id=QFSFC3WFER7VBVZNB3XLRIEFWGM4Q5VTAAJIWG4HAERKXSHZGZ4Q, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.889468719 (41h1m32.360679448s) [noise=true , address=145.79.227.26:4654 , piece_id=23NNREG47WSZWJNPZWP275L62YBBSTB6RQWTINIRTTAYU66DRT7Q, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.900234388 (41h1m32.371445117s) [noise=true , address=45.91.116.4:4937 , piece_id=VFAPMIAB2G2BBOMNSM5EXONUL5BNORMR7KXIYSTQ7UFRVASVTQHQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.923941432 (41h1m32.395152161s) [noise=true , address=145.79.227.42:4121 , piece_id=6D47MY4GTQ3YQOGHS2V2OTO2VFDL6XKKS2R4VA6VZZ557WYFY34A, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.929131404 (41h1m32.400342133s) [noise=true , address=145.79.227.22:4510 , piece_id=HCJIUH3VECRBR744KIQ65UKSMTEEKUPHJWXSDBM5SEMUVBJ5YX6A, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.180555093 (41h1m32.651765822s) [noise=true , address=145.79.227.32:4835 , piece_id=PJURUANT2XJY5PNYTSO5W2EDT3HYFAC2BFNCANKOAZ3I6P2GRC3Q, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.587529048 (41h1m33.058739787s) [noise=true , address=145.79.227.41:4087 , piece_id=6GY2NC7F26VLU5NEN65OYU7YIHJYSTI5GNVRTNXM5HO5XQMZKEPA, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.671173874 (41h1m33.142384603s) [noise=true , address=145.79.227.27:4691 , piece_id=Z53YBOXTW4L23P4M3EOW6ADUHDWHIJVRDT7IL4I5FUPPZOW2MRLQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.671209982 (41h1m33.142420711s) [seg_pos={0 0}, upload=36107]: Segment reached optimal threshold of 13 pieces. Dec 5 09:56:56.671423689 (41h1m33.142634428s) [noise=true , address=145.79.227.24:4582 , piece_id=MCQWT2LFD472TQDOAHLGQ5L2X7S654P6TL4J5MSP3CML5ZCURHMA, seg_pos={0 0}, upload=36107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa): protocol: write tcp 188.166.225.170:54136->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:54136->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54136->145.79.227.24:4582: use of closed network connection Dec 5 09:56:56.671432325 (41h1m33.142643064s) [noise=true , address=145.79.227.24:4582 , piece_id=MCQWT2LFD472TQDOAHLGQ5L2X7S654P6TL4J5MSP3CML5ZCURHMA, seg_pos={0 0}, upload=36107]: Long tail cancellation due to optimal threshold Dec 5 09:56:56.671496377 (41h1m33.142707106s) [noise=true , address=145.79.227.44:4193 , piece_id=327UUYIASLKFGUIQVY4P5UML4R2R4ODLUAYXR3GISKIT6CFDUGAQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: write tcp 188.166.225.170:39276->145.79.227.44:4193: use of closed network connection; write tcp 188.166.225.170:39276->145.79.227.44:4193: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39276->145.79.227.44:4193: use of closed network connection Dec 5 09:56:56.671501627 (41h1m33.142712366s) [noise=true , address=145.79.227.44:4193 , piece_id=327UUYIASLKFGUIQVY4P5UML4R2R4ODLUAYXR3GISKIT6CFDUGAQ, seg_pos={0 0}, upload=36107]: Long tail cancellation due to optimal threshold Dec 5 09:56:56.671516466 (41h1m33.142727195s) [noise=true , address=77.90.31.16:4735 , piece_id=V4YJIHVMXYTBK7IHU272RYCKUXI44B2MXRLETMWMO2WAI2XRU4SQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: write tcp 188.166.225.170:41866->77.90.31.16:4735: use of closed network connection; storage node overloaded, request limit: 2000 Dec 5 09:56:56.671525944 (41h1m33.142736673s) [noise=true , address=77.90.31.16:4735 , piece_id=V4YJIHVMXYTBK7IHU272RYCKUXI44B2MXRLETMWMO2WAI2XRU4SQ, seg_pos={0 0}, upload=36107]: Long tail cancellation due to optimal threshold Dec 5 09:56:56.671548016 (41h1m33.142758745s) [seg_pos={0 0}, upload=36107]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 5 09:56:56.671623500 (41h1m33.142834229s) [upload=36107]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 5 09:56:56.671631204 (41h1m33.142841933s) [upload=36107]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 5 09:56:57.112987571 (41h1m33.5841983s) [upload=36107]: Done uploading segments... Dec 5 09:56:57.113007228 (41h1m33.584217967s) [upload=36107]: Upload finished. err: <nil> Duration: 11.163919987s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 8.995890862s trace: https://jaeger.dev.storj.io/trace/55b376cc76cfe920 FAIL (2025-12-05 10:19:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 9 (41h1m29.913335078s) [upload=36107]: Uploading segments... Dec 5 09:56:53.442159225 (41h1m29.913369955s) [upload=36107]: Got next segment. Inline: false Dec 5 09:56:53.442188120 (41h1m29.913398849s) [upload=36107]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 5 09:56:53.442194372 (41h1m29.913405101s) [upload=36107]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 5 09:56:53.701976567 (41h1m30.173187296s) [seg_pos={0 0}, upload=36107]: Begin upload segment... Dec 5 09:56:53.702020460 (41h1m30.173231189s) [seg_pos={0 0}, upload=36107]: Done begin upload segment. Dec 5 09:56:53.702056649 (41h1m30.173267388s) [noise=true , address=77.90.31.21:4878 , piece_id=RCFTETF7SGR6FTUBJWWBJDKOZIHZKBMSUBZYVWYKF2MP56PMIWMQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702064965 (41h1m30.173275694s) [noise=true , address=145.79.227.28:4725 , piece_id=ZE642TJU4ZJKBOHQ6K6GPLPTN2YKC4BOP325LN7QXALZSDPG66HQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702111533 (41h1m30.173322272s) [noise=true , address=145.79.227.42:4121 , piece_id=6D47MY4GTQ3YQOGHS2V2OTO2VFDL6XKKS2R4VA6VZZ557WYFY34A, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702548986 (41h1m30.173759766s) [noise=true , address=145.79.227.27:4691 , piece_id=Z53YBOXTW4L23P4M3EOW6ADUHDWHIJVRDT7IL4I5FUPPZOW2MRLQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702931455 (41h1m30.174142194s) [noise=true , address=77.90.31.16:4735 , piece_id=V4YJIHVMXYTBK7IHU272RYCKUXI44B2MXRLETMWMO2WAI2XRU4SQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.703301079 (41h1m30.174511808s) [noise=true , address=145.79.227.10:4401 , piece_id=XAVSV3SGPDDLZWGKQLHA2CVYTGOPO7J3YVEO5C3MYIN5SJEUZ2OQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.703633000 (41h1m30.174843729s) [noise=true , address=145.79.227.24:4583 , piece_id=IB6T3J3CF3P3A54JEHHBK6KEPMUEI32B4QYSSA2G2BUMIVUC7DGQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.703957648 (41h1m30.175168377s) [noise=true , address=145.79.227.26:4654 , piece_id=23NNREG47WSZWJNPZWP275L62YBBSTB6RQWTINIRTTAYU66DRT7Q, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.704788621 (41h1m30.17599936s) [noise=true , address=145.79.227.43:4159 , piece_id=S5T27VFA6UNNONPZBZILNTQUUNSRLWPREKOJORMLKOXWNL3GMBSQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.705457525 (41h1m30.176668264s) [noise=true , address=145.79.227.32:4835 , piece_id=PJURUANT2XJY5PNYTSO5W2EDT3HYFAC2BFNCANKOAZ3I6P2GRC3Q, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.706486825 (41h1m30.177697554s) [noise=true , address=145.79.227.24:4582 , piece_id=MCQWT2LFD472TQDOAHLGQ5L2X7S654P6TL4J5MSP3CML5ZCURHMA, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.707626675 (41h1m30.178837414s) [noise=true , address=145.79.227.44:4193 , piece_id=327UUYIASLKFGUIQVY4P5UML4R2R4ODLUAYXR3GISKIT6CFDUGAQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.708631078 (41h1m30.179841817s) [noise=true , address=145.79.227.40:4050 , piece_id=QFSFC3WFER7VBVZNB3XLRIEFWGM4Q5VTAAJIWG4HAERKXSHZGZ4Q, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.709908630 (41h1m30.181119359s) [noise=true , address=145.79.227.22:4510 , piece_id=HCJIUH3VECRBR744KIQ65UKSMTEEKUPHJWXSDBM5SEMUVBJ5YX6A, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.710331386 (41h1m30.181542125s) [noise=true , address=145.79.227.41:4087 , piece_id=6GY2NC7F26VLU5NEN65OYU7YIHJYSTI5GNVRTNXM5HO5XQMZKEPA, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.710803685 (41h1m30.182014424s) [noise=true , address=45.91.116.4:4937 , piece_id=VFAPMIAB2G2BBOMNSM5EXONUL5BNORMR7KXIYSTQ7UFRVASVTQHQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:55.513663514 (41h1m31.984874243s) [upload=36107]: Next returned nil segment Dec 5 09:56:55.513682680 (41h1m31.98489341s) [upload=36107]: Waiting for error group managing segments... Dec 5 09:56:55.808504186 (41h1m32.279714915s) [noise=true , address=77.90.31.21:4878 , piece_id=RCFTETF7SGR6FTUBJWWBJDKOZIHZKBMSUBZYVWYKF2MP56PMIWMQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.866933347 (41h1m32.338144086s) [noise=true , address=145.79.227.10:4401 , piece_id=XAVSV3SGPDDLZWGKQLHA2CVYTGOPO7J3YVEO5C3MYIN5SJEUZ2OQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.869510454 (41h1m32.340721183s) [noise=true , address=145.79.227.28:4725 , piece_id=ZE642TJU4ZJKBOHQ6K6GPLPTN2YKC4BOP325LN7QXALZSDPG66HQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.875082795 (41h1m32.346293524s) [noise=true , address=145.79.227.43:4159 , piece_id=S5T27VFA6UNNONPZBZILNTQUUNSRLWPREKOJORMLKOXWNL3GMBSQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.884542648 (41h1m32.355753387s) [noise=true , address=145.79.227.24:4583 , piece_id=IB6T3J3CF3P3A54JEHHBK6KEPMUEI32B4QYSSA2G2BUMIVUC7DGQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.888876321 (41h1m32.36008707s) [noise=true , address=145.79.227.40:4050 , piece_id=QFSFC3WFER7VBVZNB3XLRIEFWGM4Q5VTAAJIWG4HAERKXSHZGZ4Q, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.889468719 (41h1m32.360679448s) [noise=true , address=145.79.227.26:4654 , piece_id=23NNREG47WSZWJNPZWP275L62YBBSTB6RQWTINIRTTAYU66DRT7Q, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.900234388 (41h1m32.371445117s) [noise=true , address=45.91.116.4:4937 , piece_id=VFAPMIAB2G2BBOMNSM5EXONUL5BNORMR7KXIYSTQ7UFRVASVTQHQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.923941432 (41h1m32.395152161s) [noise=true , address=145.79.227.42:4121 , piece_id=6D47MY4GTQ3YQOGHS2V2OTO2VFDL6XKKS2R4VA6VZZ557WYFY34A, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.929131404 (41h1m32.400342133s) [noise=true , address=145.79.227.22:4510 , piece_id=HCJIUH3VECRBR744KIQ65UKSMTEEKUPHJWXSDBM5SEMUVBJ5YX6A, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.180555093 (41h1m32.651765822s) [noise=true , address=145.79.227.32:4835 , piece_id=PJURUANT2XJY5PNYTSO5W2EDT3HYFAC2BFNCANKOAZ3I6P2GRC3Q, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.587529048 (41h1m33.058739787s) [noise=true , address=145.79.227.41:4087 , piece_id=6GY2NC7F26VLU5NEN65OYU7YIHJYSTI5GNVRTNXM5HO5XQMZKEPA, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.671173874 (41h1m33.142384603s) [noise=true , address=145.79.227.27:4691 , piece_id=Z53YBOXTW4L23P4M3EOW6ADUHDWHIJVRDT7IL4I5FUPPZOW2MRLQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.671209982 (41h1m33.142420711s) [seg_pos={0 0}, upload=36107]: Segment reached optimal threshold of 13 pieces. Dec 5 09:56:56.671423689 (41h1m33.142634428s) [noise=true , address=145.79.227.24:4582 , piece_id=MCQWT2LFD472TQDOAHLGQ5L2X7S654P6TL4J5MSP3CML5ZCURHMA, seg_pos={0 0}, upload=36107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa): protocol: write tcp 188.166.225.170:54136->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:54136->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54136->145.79.227.24:4582: use of closed network connection Dec 5 09:56:56.671432325 (41h1m33.142643064s) [noise=true , address=145.79.227.24:4582 , piece_id=MCQWT2LFD472TQDOAHLGQ5L2X7S654P6TL4J5MSP3CML5ZCURHMA, seg_pos={0 0}, upload=36107]: Long tail cancellation due to optimal threshold Dec 5 09:56:56.671496377 (41h1m33.142707106s) [noise=true , address=145.79.227.44:4193 , piece_id=327UUYIASLKFGUIQVY4P5UML4R2R4ODLUAYXR3GISKIT6CFDUGAQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: write tcp 188.166.225.170:39276->145.79.227.44:4193: use of closed network connection; write tcp 188.166.225.170:39276->145.79.227.44:4193: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39276->145.79.227.44:4193: use of closed network connection Dec 5 09:56:56.671501627 (41h1m33.142712366s) [noise=true , address=145.79.227.44:4193 , piece_id=327UUYIASLKFGUIQVY4P5UML4R2R4ODLUAYXR3GISKIT6CFDUGAQ, seg_pos={0 0}, upload=36107]: Long tail cancellation due to optimal threshold Dec 5 09:56:56.671516466 (41h1m33.142727195s) [noise=true , address=77.90.31.16:4735 , piece_id=V4YJIHVMXYTBK7IHU272RYCKUXI44B2MXRLETMWMO2WAI2XRU4SQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: write tcp 188.166.225.170:41866->77.90.31.16:4735: use of closed network connection; storage node overloaded, request limit: 2000 Dec 5 09:56:56.671525944 (41h1m33.142736673s) [noise=true , address=77.90.31.16:4735 , piece_id=V4YJIHVMXYTBK7IHU272RYCKUXI44B2MXRLETMWMO2WAI2XRU4SQ, seg_pos={0 0}, upload=36107]: Long tail cancellation due to optimal threshold Dec 5 09:56:56.671548016 (41h1m33.142758745s) [seg_pos={0 0}, upload=36107]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 5 09:56:56.671623500 (41h1m33.142834229s) [upload=36107]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 5 09:56:56.671631204 (41h1m33.142841933s) [upload=36107]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 5 09:56:57.112987571 (41h1m33.5841983s) [upload=36107]: Done uploading segments... Dec 5 09:56:57.113007228 (41h1m33.584217967s) [upload=36107]: Upload finished. err: <nil> Duration: 8.995914838s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 9.206467104s trace: https://jaeger.dev.storj.io/trace/55b376cc76cfe920 FAIL (2025-12-05 10:18:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 9 (41h1m29.913335078s) [upload=36107]: Uploading segments... Dec 5 09:56:53.442159225 (41h1m29.913369955s) [upload=36107]: Got next segment. Inline: false Dec 5 09:56:53.442188120 (41h1m29.913398849s) [upload=36107]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 5 09:56:53.442194372 (41h1m29.913405101s) [upload=36107]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 5 09:56:53.701976567 (41h1m30.173187296s) [seg_pos={0 0}, upload=36107]: Begin upload segment... Dec 5 09:56:53.702020460 (41h1m30.173231189s) [seg_pos={0 0}, upload=36107]: Done begin upload segment. Dec 5 09:56:53.702056649 (41h1m30.173267388s) [noise=true , address=77.90.31.21:4878 , piece_id=RCFTETF7SGR6FTUBJWWBJDKOZIHZKBMSUBZYVWYKF2MP56PMIWMQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702064965 (41h1m30.173275694s) [noise=true , address=145.79.227.28:4725 , piece_id=ZE642TJU4ZJKBOHQ6K6GPLPTN2YKC4BOP325LN7QXALZSDPG66HQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702111533 (41h1m30.173322272s) [noise=true , address=145.79.227.42:4121 , piece_id=6D47MY4GTQ3YQOGHS2V2OTO2VFDL6XKKS2R4VA6VZZ557WYFY34A, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702548986 (41h1m30.173759766s) [noise=true , address=145.79.227.27:4691 , piece_id=Z53YBOXTW4L23P4M3EOW6ADUHDWHIJVRDT7IL4I5FUPPZOW2MRLQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702931455 (41h1m30.174142194s) [noise=true , address=77.90.31.16:4735 , piece_id=V4YJIHVMXYTBK7IHU272RYCKUXI44B2MXRLETMWMO2WAI2XRU4SQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.703301079 (41h1m30.174511808s) [noise=true , address=145.79.227.10:4401 , piece_id=XAVSV3SGPDDLZWGKQLHA2CVYTGOPO7J3YVEO5C3MYIN5SJEUZ2OQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.703633000 (41h1m30.174843729s) [noise=true , address=145.79.227.24:4583 , piece_id=IB6T3J3CF3P3A54JEHHBK6KEPMUEI32B4QYSSA2G2BUMIVUC7DGQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.703957648 (41h1m30.175168377s) [noise=true , address=145.79.227.26:4654 , piece_id=23NNREG47WSZWJNPZWP275L62YBBSTB6RQWTINIRTTAYU66DRT7Q, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.704788621 (41h1m30.17599936s) [noise=true , address=145.79.227.43:4159 , piece_id=S5T27VFA6UNNONPZBZILNTQUUNSRLWPREKOJORMLKOXWNL3GMBSQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.705457525 (41h1m30.176668264s) [noise=true , address=145.79.227.32:4835 , piece_id=PJURUANT2XJY5PNYTSO5W2EDT3HYFAC2BFNCANKOAZ3I6P2GRC3Q, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.706486825 (41h1m30.177697554s) [noise=true , address=145.79.227.24:4582 , piece_id=MCQWT2LFD472TQDOAHLGQ5L2X7S654P6TL4J5MSP3CML5ZCURHMA, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.707626675 (41h1m30.178837414s) [noise=true , address=145.79.227.44:4193 , piece_id=327UUYIASLKFGUIQVY4P5UML4R2R4ODLUAYXR3GISKIT6CFDUGAQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.708631078 (41h1m30.179841817s) [noise=true , address=145.79.227.40:4050 , piece_id=QFSFC3WFER7VBVZNB3XLRIEFWGM4Q5VTAAJIWG4HAERKXSHZGZ4Q, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.709908630 (41h1m30.181119359s) [noise=true , address=145.79.227.22:4510 , piece_id=HCJIUH3VECRBR744KIQ65UKSMTEEKUPHJWXSDBM5SEMUVBJ5YX6A, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.710331386 (41h1m30.181542125s) [noise=true , address=145.79.227.41:4087 , piece_id=6GY2NC7F26VLU5NEN65OYU7YIHJYSTI5GNVRTNXM5HO5XQMZKEPA, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.710803685 (41h1m30.182014424s) [noise=true , address=45.91.116.4:4937 , piece_id=VFAPMIAB2G2BBOMNSM5EXONUL5BNORMR7KXIYSTQ7UFRVASVTQHQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:55.513663514 (41h1m31.984874243s) [upload=36107]: Next returned nil segment Dec 5 09:56:55.513682680 (41h1m31.98489341s) [upload=36107]: Waiting for error group managing segments... Dec 5 09:56:55.808504186 (41h1m32.279714915s) [noise=true , address=77.90.31.21:4878 , piece_id=RCFTETF7SGR6FTUBJWWBJDKOZIHZKBMSUBZYVWYKF2MP56PMIWMQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.866933347 (41h1m32.338144086s) [noise=true , address=145.79.227.10:4401 , piece_id=XAVSV3SGPDDLZWGKQLHA2CVYTGOPO7J3YVEO5C3MYIN5SJEUZ2OQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.869510454 (41h1m32.340721183s) [noise=true , address=145.79.227.28:4725 , piece_id=ZE642TJU4ZJKBOHQ6K6GPLPTN2YKC4BOP325LN7QXALZSDPG66HQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.875082795 (41h1m32.346293524s) [noise=true , address=145.79.227.43:4159 , piece_id=S5T27VFA6UNNONPZBZILNTQUUNSRLWPREKOJORMLKOXWNL3GMBSQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.884542648 (41h1m32.355753387s) [noise=true , address=145.79.227.24:4583 , piece_id=IB6T3J3CF3P3A54JEHHBK6KEPMUEI32B4QYSSA2G2BUMIVUC7DGQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.888876321 (41h1m32.36008707s) [noise=true , address=145.79.227.40:4050 , piece_id=QFSFC3WFER7VBVZNB3XLRIEFWGM4Q5VTAAJIWG4HAERKXSHZGZ4Q, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.889468719 (41h1m32.360679448s) [noise=true , address=145.79.227.26:4654 , piece_id=23NNREG47WSZWJNPZWP275L62YBBSTB6RQWTINIRTTAYU66DRT7Q, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.900234388 (41h1m32.371445117s) [noise=true , address=45.91.116.4:4937 , piece_id=VFAPMIAB2G2BBOMNSM5EXONUL5BNORMR7KXIYSTQ7UFRVASVTQHQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.923941432 (41h1m32.395152161s) [noise=true , address=145.79.227.42:4121 , piece_id=6D47MY4GTQ3YQOGHS2V2OTO2VFDL6XKKS2R4VA6VZZ557WYFY34A, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.929131404 (41h1m32.400342133s) [noise=true , address=145.79.227.22:4510 , piece_id=HCJIUH3VECRBR744KIQ65UKSMTEEKUPHJWXSDBM5SEMUVBJ5YX6A, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.180555093 (41h1m32.651765822s) [noise=true , address=145.79.227.32:4835 , piece_id=PJURUANT2XJY5PNYTSO5W2EDT3HYFAC2BFNCANKOAZ3I6P2GRC3Q, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.587529048 (41h1m33.058739787s) [noise=true , address=145.79.227.41:4087 , piece_id=6GY2NC7F26VLU5NEN65OYU7YIHJYSTI5GNVRTNXM5HO5XQMZKEPA, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.671173874 (41h1m33.142384603s) [noise=true , address=145.79.227.27:4691 , piece_id=Z53YBOXTW4L23P4M3EOW6ADUHDWHIJVRDT7IL4I5FUPPZOW2MRLQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.671209982 (41h1m33.142420711s) [seg_pos={0 0}, upload=36107]: Segment reached optimal threshold of 13 pieces. Dec 5 09:56:56.671423689 (41h1m33.142634428s) [noise=true , address=145.79.227.24:4582 , piece_id=MCQWT2LFD472TQDOAHLGQ5L2X7S654P6TL4J5MSP3CML5ZCURHMA, seg_pos={0 0}, upload=36107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa): protocol: write tcp 188.166.225.170:54136->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:54136->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54136->145.79.227.24:4582: use of closed network connection Dec 5 09:56:56.671432325 (41h1m33.142643064s) [noise=true , address=145.79.227.24:4582 , piece_id=MCQWT2LFD472TQDOAHLGQ5L2X7S654P6TL4J5MSP3CML5ZCURHMA, seg_pos={0 0}, upload=36107]: Long tail cancellation due to optimal threshold Dec 5 09:56:56.671496377 (41h1m33.142707106s) [noise=true , address=145.79.227.44:4193 , piece_id=327UUYIASLKFGUIQVY4P5UML4R2R4ODLUAYXR3GISKIT6CFDUGAQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: write tcp 188.166.225.170:39276->145.79.227.44:4193: use of closed network connection; write tcp 188.166.225.170:39276->145.79.227.44:4193: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39276->145.79.227.44:4193: use of closed network connection Dec 5 09:56:56.671501627 (41h1m33.142712366s) [noise=true , address=145.79.227.44:4193 , piece_id=327UUYIASLKFGUIQVY4P5UML4R2R4ODLUAYXR3GISKIT6CFDUGAQ, seg_pos={0 0}, upload=36107]: Long tail cancellation due to optimal threshold Dec 5 09:56:56.671516466 (41h1m33.142727195s) [noise=true , address=77.90.31.16:4735 , piece_id=V4YJIHVMXYTBK7IHU272RYCKUXI44B2MXRLETMWMO2WAI2XRU4SQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: write tcp 188.166.225.170:41866->77.90.31.16:4735: use of closed network connection; storage node overloaded, request limit: 2000 Dec 5 09:56:56.671525944 (41h1m33.142736673s) [noise=true , address=77.90.31.16:4735 , piece_id=V4YJIHVMXYTBK7IHU272RYCKUXI44B2MXRLETMWMO2WAI2XRU4SQ, seg_pos={0 0}, upload=36107]: Long tail cancellation due to optimal threshold Dec 5 09:56:56.671548016 (41h1m33.142758745s) [seg_pos={0 0}, upload=36107]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 5 09:56:56.671623500 (41h1m33.142834229s) [upload=36107]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 5 09:56:56.671631204 (41h1m33.142841933s) [upload=36107]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 5 09:56:57.112987571 (41h1m33.5841983s) [upload=36107]: Done uploading segments... Dec 5 09:56:57.113007228 (41h1m33.584217967s) [upload=36107]: Upload finished. err: <nil> Duration: 9.206490419s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 10.360804373s trace: https://jaeger.dev.storj.io/trace/55b376cc76cfe920 FAIL (2025-12-05 10:17:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 9 (41h1m29.913335078s) [upload=36107]: Uploading segments... Dec 5 09:56:53.442159225 (41h1m29.913369955s) [upload=36107]: Got next segment. Inline: false Dec 5 09:56:53.442188120 (41h1m29.913398849s) [upload=36107]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 5 09:56:53.442194372 (41h1m29.913405101s) [upload=36107]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 5 09:56:53.701976567 (41h1m30.173187296s) [seg_pos={0 0}, upload=36107]: Begin upload segment... Dec 5 09:56:53.702020460 (41h1m30.173231189s) [seg_pos={0 0}, upload=36107]: Done begin upload segment. Dec 5 09:56:53.702056649 (41h1m30.173267388s) [noise=true , address=77.90.31.21:4878 , piece_id=RCFTETF7SGR6FTUBJWWBJDKOZIHZKBMSUBZYVWYKF2MP56PMIWMQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702064965 (41h1m30.173275694s) [noise=true , address=145.79.227.28:4725 , piece_id=ZE642TJU4ZJKBOHQ6K6GPLPTN2YKC4BOP325LN7QXALZSDPG66HQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702111533 (41h1m30.173322272s) [noise=true , address=145.79.227.42:4121 , piece_id=6D47MY4GTQ3YQOGHS2V2OTO2VFDL6XKKS2R4VA6VZZ557WYFY34A, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702548986 (41h1m30.173759766s) [noise=true , address=145.79.227.27:4691 , piece_id=Z53YBOXTW4L23P4M3EOW6ADUHDWHIJVRDT7IL4I5FUPPZOW2MRLQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702931455 (41h1m30.174142194s) [noise=true , address=77.90.31.16:4735 , piece_id=V4YJIHVMXYTBK7IHU272RYCKUXI44B2MXRLETMWMO2WAI2XRU4SQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.703301079 (41h1m30.174511808s) [noise=true , address=145.79.227.10:4401 , piece_id=XAVSV3SGPDDLZWGKQLHA2CVYTGOPO7J3YVEO5C3MYIN5SJEUZ2OQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.703633000 (41h1m30.174843729s) [noise=true , address=145.79.227.24:4583 , piece_id=IB6T3J3CF3P3A54JEHHBK6KEPMUEI32B4QYSSA2G2BUMIVUC7DGQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.703957648 (41h1m30.175168377s) [noise=true , address=145.79.227.26:4654 , piece_id=23NNREG47WSZWJNPZWP275L62YBBSTB6RQWTINIRTTAYU66DRT7Q, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.704788621 (41h1m30.17599936s) [noise=true , address=145.79.227.43:4159 , piece_id=S5T27VFA6UNNONPZBZILNTQUUNSRLWPREKOJORMLKOXWNL3GMBSQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.705457525 (41h1m30.176668264s) [noise=true , address=145.79.227.32:4835 , piece_id=PJURUANT2XJY5PNYTSO5W2EDT3HYFAC2BFNCANKOAZ3I6P2GRC3Q, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.706486825 (41h1m30.177697554s) [noise=true , address=145.79.227.24:4582 , piece_id=MCQWT2LFD472TQDOAHLGQ5L2X7S654P6TL4J5MSP3CML5ZCURHMA, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.707626675 (41h1m30.178837414s) [noise=true , address=145.79.227.44:4193 , piece_id=327UUYIASLKFGUIQVY4P5UML4R2R4ODLUAYXR3GISKIT6CFDUGAQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.708631078 (41h1m30.179841817s) [noise=true , address=145.79.227.40:4050 , piece_id=QFSFC3WFER7VBVZNB3XLRIEFWGM4Q5VTAAJIWG4HAERKXSHZGZ4Q, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.709908630 (41h1m30.181119359s) [noise=true , address=145.79.227.22:4510 , piece_id=HCJIUH3VECRBR744KIQ65UKSMTEEKUPHJWXSDBM5SEMUVBJ5YX6A, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.710331386 (41h1m30.181542125s) [noise=true , address=145.79.227.41:4087 , piece_id=6GY2NC7F26VLU5NEN65OYU7YIHJYSTI5GNVRTNXM5HO5XQMZKEPA, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.710803685 (41h1m30.182014424s) [noise=true , address=45.91.116.4:4937 , piece_id=VFAPMIAB2G2BBOMNSM5EXONUL5BNORMR7KXIYSTQ7UFRVASVTQHQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:55.513663514 (41h1m31.984874243s) [upload=36107]: Next returned nil segment Dec 5 09:56:55.513682680 (41h1m31.98489341s) [upload=36107]: Waiting for error group managing segments... Dec 5 09:56:55.808504186 (41h1m32.279714915s) [noise=true , address=77.90.31.21:4878 , piece_id=RCFTETF7SGR6FTUBJWWBJDKOZIHZKBMSUBZYVWYKF2MP56PMIWMQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.866933347 (41h1m32.338144086s) [noise=true , address=145.79.227.10:4401 , piece_id=XAVSV3SGPDDLZWGKQLHA2CVYTGOPO7J3YVEO5C3MYIN5SJEUZ2OQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.869510454 (41h1m32.340721183s) [noise=true , address=145.79.227.28:4725 , piece_id=ZE642TJU4ZJKBOHQ6K6GPLPTN2YKC4BOP325LN7QXALZSDPG66HQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.875082795 (41h1m32.346293524s) [noise=true , address=145.79.227.43:4159 , piece_id=S5T27VFA6UNNONPZBZILNTQUUNSRLWPREKOJORMLKOXWNL3GMBSQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.884542648 (41h1m32.355753387s) [noise=true , address=145.79.227.24:4583 , piece_id=IB6T3J3CF3P3A54JEHHBK6KEPMUEI32B4QYSSA2G2BUMIVUC7DGQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.888876321 (41h1m32.36008707s) [noise=true , address=145.79.227.40:4050 , piece_id=QFSFC3WFER7VBVZNB3XLRIEFWGM4Q5VTAAJIWG4HAERKXSHZGZ4Q, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.889468719 (41h1m32.360679448s) [noise=true , address=145.79.227.26:4654 , piece_id=23NNREG47WSZWJNPZWP275L62YBBSTB6RQWTINIRTTAYU66DRT7Q, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.900234388 (41h1m32.371445117s) [noise=true , address=45.91.116.4:4937 , piece_id=VFAPMIAB2G2BBOMNSM5EXONUL5BNORMR7KXIYSTQ7UFRVASVTQHQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.923941432 (41h1m32.395152161s) [noise=true , address=145.79.227.42:4121 , piece_id=6D47MY4GTQ3YQOGHS2V2OTO2VFDL6XKKS2R4VA6VZZ557WYFY34A, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.929131404 (41h1m32.400342133s) [noise=true , address=145.79.227.22:4510 , piece_id=HCJIUH3VECRBR744KIQ65UKSMTEEKUPHJWXSDBM5SEMUVBJ5YX6A, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.180555093 (41h1m32.651765822s) [noise=true , address=145.79.227.32:4835 , piece_id=PJURUANT2XJY5PNYTSO5W2EDT3HYFAC2BFNCANKOAZ3I6P2GRC3Q, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.587529048 (41h1m33.058739787s) [noise=true , address=145.79.227.41:4087 , piece_id=6GY2NC7F26VLU5NEN65OYU7YIHJYSTI5GNVRTNXM5HO5XQMZKEPA, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.671173874 (41h1m33.142384603s) [noise=true , address=145.79.227.27:4691 , piece_id=Z53YBOXTW4L23P4M3EOW6ADUHDWHIJVRDT7IL4I5FUPPZOW2MRLQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.671209982 (41h1m33.142420711s) [seg_pos={0 0}, upload=36107]: Segment reached optimal threshold of 13 pieces. Dec 5 09:56:56.671423689 (41h1m33.142634428s) [noise=true , address=145.79.227.24:4582 , piece_id=MCQWT2LFD472TQDOAHLGQ5L2X7S654P6TL4J5MSP3CML5ZCURHMA, seg_pos={0 0}, upload=36107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa): protocol: write tcp 188.166.225.170:54136->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:54136->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54136->145.79.227.24:4582: use of closed network connection Dec 5 09:56:56.671432325 (41h1m33.142643064s) [noise=true , address=145.79.227.24:4582 , piece_id=MCQWT2LFD472TQDOAHLGQ5L2X7S654P6TL4J5MSP3CML5ZCURHMA, seg_pos={0 0}, upload=36107]: Long tail cancellation due to optimal threshold Dec 5 09:56:56.671496377 (41h1m33.142707106s) [noise=true , address=145.79.227.44:4193 , piece_id=327UUYIASLKFGUIQVY4P5UML4R2R4ODLUAYXR3GISKIT6CFDUGAQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: write tcp 188.166.225.170:39276->145.79.227.44:4193: use of closed network connection; write tcp 188.166.225.170:39276->145.79.227.44:4193: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39276->145.79.227.44:4193: use of closed network connection Dec 5 09:56:56.671501627 (41h1m33.142712366s) [noise=true , address=145.79.227.44:4193 , piece_id=327UUYIASLKFGUIQVY4P5UML4R2R4ODLUAYXR3GISKIT6CFDUGAQ, seg_pos={0 0}, upload=36107]: Long tail cancellation due to optimal threshold Dec 5 09:56:56.671516466 (41h1m33.142727195s) [noise=true , address=77.90.31.16:4735 , piece_id=V4YJIHVMXYTBK7IHU272RYCKUXI44B2MXRLETMWMO2WAI2XRU4SQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: write tcp 188.166.225.170:41866->77.90.31.16:4735: use of closed network connection; storage node overloaded, request limit: 2000 Dec 5 09:56:56.671525944 (41h1m33.142736673s) [noise=true , address=77.90.31.16:4735 , piece_id=V4YJIHVMXYTBK7IHU272RYCKUXI44B2MXRLETMWMO2WAI2XRU4SQ, seg_pos={0 0}, upload=36107]: Long tail cancellation due to optimal threshold Dec 5 09:56:56.671548016 (41h1m33.142758745s) [seg_pos={0 0}, upload=36107]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 5 09:56:56.671623500 (41h1m33.142834229s) [upload=36107]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 5 09:56:56.671631204 (41h1m33.142841933s) [upload=36107]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 5 09:56:57.112987571 (41h1m33.5841983s) [upload=36107]: Done uploading segments... Dec 5 09:56:57.113007228 (41h1m33.584217967s) [upload=36107]: Upload finished. err: <nil> Duration: 10.360833288s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 9.028454034s trace: https://jaeger.dev.storj.io/trace/55b376cc76cfe920 FAIL (2025-12-05 10:17:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 9 (41h1m29.913335078s) [upload=36107]: Uploading segments... Dec 5 09:56:53.442159225 (41h1m29.913369955s) [upload=36107]: Got next segment. Inline: false Dec 5 09:56:53.442188120 (41h1m29.913398849s) [upload=36107]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 5 09:56:53.442194372 (41h1m29.913405101s) [upload=36107]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 5 09:56:53.701976567 (41h1m30.173187296s) [seg_pos={0 0}, upload=36107]: Begin upload segment... Dec 5 09:56:53.702020460 (41h1m30.173231189s) [seg_pos={0 0}, upload=36107]: Done begin upload segment. Dec 5 09:56:53.702056649 (41h1m30.173267388s) [noise=true , address=77.90.31.21:4878 , piece_id=RCFTETF7SGR6FTUBJWWBJDKOZIHZKBMSUBZYVWYKF2MP56PMIWMQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702064965 (41h1m30.173275694s) [noise=true , address=145.79.227.28:4725 , piece_id=ZE642TJU4ZJKBOHQ6K6GPLPTN2YKC4BOP325LN7QXALZSDPG66HQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702111533 (41h1m30.173322272s) [noise=true , address=145.79.227.42:4121 , piece_id=6D47MY4GTQ3YQOGHS2V2OTO2VFDL6XKKS2R4VA6VZZ557WYFY34A, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702548986 (41h1m30.173759766s) [noise=true , address=145.79.227.27:4691 , piece_id=Z53YBOXTW4L23P4M3EOW6ADUHDWHIJVRDT7IL4I5FUPPZOW2MRLQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.702931455 (41h1m30.174142194s) [noise=true , address=77.90.31.16:4735 , piece_id=V4YJIHVMXYTBK7IHU272RYCKUXI44B2MXRLETMWMO2WAI2XRU4SQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.703301079 (41h1m30.174511808s) [noise=true , address=145.79.227.10:4401 , piece_id=XAVSV3SGPDDLZWGKQLHA2CVYTGOPO7J3YVEO5C3MYIN5SJEUZ2OQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.703633000 (41h1m30.174843729s) [noise=true , address=145.79.227.24:4583 , piece_id=IB6T3J3CF3P3A54JEHHBK6KEPMUEI32B4QYSSA2G2BUMIVUC7DGQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.703957648 (41h1m30.175168377s) [noise=true , address=145.79.227.26:4654 , piece_id=23NNREG47WSZWJNPZWP275L62YBBSTB6RQWTINIRTTAYU66DRT7Q, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.704788621 (41h1m30.17599936s) [noise=true , address=145.79.227.43:4159 , piece_id=S5T27VFA6UNNONPZBZILNTQUUNSRLWPREKOJORMLKOXWNL3GMBSQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.705457525 (41h1m30.176668264s) [noise=true , address=145.79.227.32:4835 , piece_id=PJURUANT2XJY5PNYTSO5W2EDT3HYFAC2BFNCANKOAZ3I6P2GRC3Q, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.706486825 (41h1m30.177697554s) [noise=true , address=145.79.227.24:4582 , piece_id=MCQWT2LFD472TQDOAHLGQ5L2X7S654P6TL4J5MSP3CML5ZCURHMA, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.707626675 (41h1m30.178837414s) [noise=true , address=145.79.227.44:4193 , piece_id=327UUYIASLKFGUIQVY4P5UML4R2R4ODLUAYXR3GISKIT6CFDUGAQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.708631078 (41h1m30.179841817s) [noise=true , address=145.79.227.40:4050 , piece_id=QFSFC3WFER7VBVZNB3XLRIEFWGM4Q5VTAAJIWG4HAERKXSHZGZ4Q, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.709908630 (41h1m30.181119359s) [noise=true , address=145.79.227.22:4510 , piece_id=HCJIUH3VECRBR744KIQ65UKSMTEEKUPHJWXSDBM5SEMUVBJ5YX6A, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.710331386 (41h1m30.181542125s) [noise=true , address=145.79.227.41:4087 , piece_id=6GY2NC7F26VLU5NEN65OYU7YIHJYSTI5GNVRTNXM5HO5XQMZKEPA, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:53.710803685 (41h1m30.182014424s) [noise=true , address=45.91.116.4:4937 , piece_id=VFAPMIAB2G2BBOMNSM5EXONUL5BNORMR7KXIYSTQ7UFRVASVTQHQ, seg_pos={0 0}, upload=36107]: Uploading piece... Dec 5 09:56:55.513663514 (41h1m31.984874243s) [upload=36107]: Next returned nil segment Dec 5 09:56:55.513682680 (41h1m31.98489341s) [upload=36107]: Waiting for error group managing segments... Dec 5 09:56:55.808504186 (41h1m32.279714915s) [noise=true , address=77.90.31.21:4878 , piece_id=RCFTETF7SGR6FTUBJWWBJDKOZIHZKBMSUBZYVWYKF2MP56PMIWMQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.866933347 (41h1m32.338144086s) [noise=true , address=145.79.227.10:4401 , piece_id=XAVSV3SGPDDLZWGKQLHA2CVYTGOPO7J3YVEO5C3MYIN5SJEUZ2OQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.869510454 (41h1m32.340721183s) [noise=true , address=145.79.227.28:4725 , piece_id=ZE642TJU4ZJKBOHQ6K6GPLPTN2YKC4BOP325LN7QXALZSDPG66HQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.875082795 (41h1m32.346293524s) [noise=true , address=145.79.227.43:4159 , piece_id=S5T27VFA6UNNONPZBZILNTQUUNSRLWPREKOJORMLKOXWNL3GMBSQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.884542648 (41h1m32.355753387s) [noise=true , address=145.79.227.24:4583 , piece_id=IB6T3J3CF3P3A54JEHHBK6KEPMUEI32B4QYSSA2G2BUMIVUC7DGQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.888876321 (41h1m32.36008707s) [noise=true , address=145.79.227.40:4050 , piece_id=QFSFC3WFER7VBVZNB3XLRIEFWGM4Q5VTAAJIWG4HAERKXSHZGZ4Q, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.889468719 (41h1m32.360679448s) [noise=true , address=145.79.227.26:4654 , piece_id=23NNREG47WSZWJNPZWP275L62YBBSTB6RQWTINIRTTAYU66DRT7Q, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.900234388 (41h1m32.371445117s) [noise=true , address=45.91.116.4:4937 , piece_id=VFAPMIAB2G2BBOMNSM5EXONUL5BNORMR7KXIYSTQ7UFRVASVTQHQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.923941432 (41h1m32.395152161s) [noise=true , address=145.79.227.42:4121 , piece_id=6D47MY4GTQ3YQOGHS2V2OTO2VFDL6XKKS2R4VA6VZZ557WYFY34A, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:55.929131404 (41h1m32.400342133s) [noise=true , address=145.79.227.22:4510 , piece_id=HCJIUH3VECRBR744KIQ65UKSMTEEKUPHJWXSDBM5SEMUVBJ5YX6A, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.180555093 (41h1m32.651765822s) [noise=true , address=145.79.227.32:4835 , piece_id=PJURUANT2XJY5PNYTSO5W2EDT3HYFAC2BFNCANKOAZ3I6P2GRC3Q, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.587529048 (41h1m33.058739787s) [noise=true , address=145.79.227.41:4087 , piece_id=6GY2NC7F26VLU5NEN65OYU7YIHJYSTI5GNVRTNXM5HO5XQMZKEPA, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.671173874 (41h1m33.142384603s) [noise=true , address=145.79.227.27:4691 , piece_id=Z53YBOXTW4L23P4M3EOW6ADUHDWHIJVRDT7IL4I5FUPPZOW2MRLQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: <nil> Dec 5 09:56:56.671209982 (41h1m33.142420711s) [seg_pos={0 0}, upload=36107]: Segment reached optimal threshold of 13 pieces. Dec 5 09:56:56.671423689 (41h1m33.142634428s) [noise=true , address=145.79.227.24:4582 , piece_id=MCQWT2LFD472TQDOAHLGQ5L2X7S654P6TL4J5MSP3CML5ZCURHMA, seg_pos={0 0}, upload=36107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa): protocol: write tcp 188.166.225.170:54136->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:54136->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54136->145.79.227.24:4582: use of closed network connection Dec 5 09:56:56.671432325 (41h1m33.142643064s) [noise=true , address=145.79.227.24:4582 , piece_id=MCQWT2LFD472TQDOAHLGQ5L2X7S654P6TL4J5MSP3CML5ZCURHMA, seg_pos={0 0}, upload=36107]: Long tail cancellation due to optimal threshold Dec 5 09:56:56.671496377 (41h1m33.142707106s) [noise=true , address=145.79.227.44:4193 , piece_id=327UUYIASLKFGUIQVY4P5UML4R2R4ODLUAYXR3GISKIT6CFDUGAQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: write tcp 188.166.225.170:39276->145.79.227.44:4193: use of closed network connection; write tcp 188.166.225.170:39276->145.79.227.44:4193: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39276->145.79.227.44:4193: use of closed network connection Dec 5 09:56:56.671501627 (41h1m33.142712366s) [noise=true , address=145.79.227.44:4193 , piece_id=327UUYIASLKFGUIQVY4P5UML4R2R4ODLUAYXR3GISKIT6CFDUGAQ, seg_pos={0 0}, upload=36107]: Long tail cancellation due to optimal threshold Dec 5 09:56:56.671516466 (41h1m33.142727195s) [noise=true , address=77.90.31.16:4735 , piece_id=V4YJIHVMXYTBK7IHU272RYCKUXI44B2MXRLETMWMO2WAI2XRU4SQ, seg_pos={0 0}, upload=36107]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: write tcp 188.166.225.170:41866->77.90.31.16:4735: use of closed network connection; storage node overloaded, request limit: 2000 Dec 5 09:56:56.671525944 (41h1m33.142736673s) [noise=true , address=77.90.31.16:4735 , piece_id=V4YJIHVMXYTBK7IHU272RYCKUXI44B2MXRLETMWMO2WAI2XRU4SQ, seg_pos={0 0}, upload=36107]: Long tail cancellation due to optimal threshold Dec 5 09:56:56.671548016 (41h1m33.142758745s) [seg_pos={0 0}, upload=36107]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 5 09:56:56.671623500 (41h1m33.142834229s) [upload=36107]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 5 09:56:56.671631204 (41h1m33.142841933s) [upload=36107]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 5 09:56:57.112987571 (41h1m33.5841983s) [upload=36107]: Done uploading segments... Dec 5 09:56:57.113007228 (41h1m33.584217967s) [upload=36107]: Upload finished. err: <nil> Duration: 9.028482229s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)