Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............................................X............] duration: 3.566825247s trace: https://jaeger.dev.storj.io/trace/7f7fb60dfbab7793 FAIL (2025-12-23 02:25:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.566910199s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 6.567260753s trace: https://jaeger.dev.storj.io/trace/61a0842c4fdd78ca FAIL (2025-12-23 02:25:51 UTC) orbiter warning: context deadline exceeded Duration: 6.567286402s edge gateway us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 4.914368721s trace: https://jaeger.dev.storj.io/trace/14f27c56f52ae4e9 FAIL (2025-12-23 02:25:50 UTC) orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 45) Duration: 4.914392368s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 5.993885673s trace: https://jaeger.dev.storj.io/trace/70da446b2b2ff016 FAIL (2025-12-23 02:25:48 UTC) orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 46) Duration: 5.993909498s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................................................X...........] duration: 3.633877266s trace: https://jaeger.dev.storj.io/trace/7f7fb60dfbab7793 FAIL (2025-12-23 02:25:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.633898387s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 6.937299448s trace: https://jaeger.dev.storj.io/trace/61a0842c4fdd78ca FAIL (2025-12-23 02:25:29 UTC) orbiter warning: context deadline exceeded Duration: 6.937330457s edge gateway us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 3.359199786s trace: https://jaeger.dev.storj.io/trace/20c185d15563c01c FAIL (2025-12-23 02:25:22 UTC) orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 39) Duration: 3.35923932s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 8.199031702s trace: https://jaeger.dev.storj.io/trace/2c98a82e82cbeda7 FAIL (2025-12-23 02:25:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== KL7BLGSD3TCZNAHNK7STV2RKAZQWCHN6PROBXAPTA5ZBCPT6A, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.700896105 (418h48m56.57197914s) [noise=true , address=145.79.227.30:4796 , piece_id=IF6NPIC5DTTMJ4DZMIDD6SFN7TGDBN6RD3H7KT7A4OC7VJVCEZOA, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.701329429 (418h48m56.572412474s) [noise=true , address=145.79.227.32:4848 , piece_id=NMEVAZ6HHKE2Z4Q5G4VLNPOIURXE5TXXUMLFJLURZUI4Z62W7VMA, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.701714032 (418h48m56.572797077s) [noise=true , address=145.79.227.28:4742 , piece_id=3IMQS3J42XXGHCZFMBYPDCPCJYRY3I3EUIQ2Z2JOK4TPXQQH75OQ, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.702231197 (418h48m56.573314232s) [noise=true , address=64.49.13.137:4500 , piece_id=RFGLMWE4OHC7RL2NBPX6VRKG4YGRVCIMBHQFW2W5XAEE376UTVVA, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.702751507 (418h48m56.573834552s) [noise=true , address=64.49.13.169:4715 , piece_id=6RCBEKXRJ56IGXWT4PVSTDYRIH3UCAD22C4Z5BU33U5YIPWBEPDQ, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.703271658 (418h48m56.574354693s) [noise=true , address=45.91.116.12:4257 , piece_id=SPTNFMB3LXNY74LCASNSAZIPCOAQHO5V6SOFP27Y4IY25CHWBO2A, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.703948437 (418h48m56.575031482s) [noise=true , address=145.79.227.23:4549 , piece_id=XH7BPLOCXKVTJQDYKIRKFXVLHIXXNJFAXCFZPSSASWLOG3GM5PEQ, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.703969256 (418h48m56.575052291s) [noise=true , address=5.231.73.4:4030 , piece_id=M6FJSRGUNCEIBMNKLBK7HHFL4DA3PDD2MZ2GN2MJKAFSH3XRI7XQ, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.704457947 (418h48m56.575540982s) [noise=true , address=145.79.227.17:4984 , piece_id=DAMTMQIEBM3I2QZTWVEGISFJ5T7I3LMZ3CTXUFXO2CS775GY4F7Q, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.704834724 (418h48m56.575917769s) [noise=true , address=77.90.31.18:4815 , piece_id=XQDHXOY2ZQQHZZHMGLO5GKZOUJZFKKWGLDJRHJLWCFEX6PKNNHWA, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.706591609 (418h48m56.577674654s) [noise=true , address=77.90.31.13:4651 , piece_id=H6MCF4TXI5WFQ2BBCPXL3OX6OA4NCRBIH6IBLMPYPIJBETEN6UGA, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.707118754 (418h48m56.578201789s) [noise=true , address=145.79.227.13:4811 , piece_id=KVHW3GEE4KWLOOUJBRVOOLDA3EGPRUU7VHPH7TQXLURUCCU5J77Q, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.707431179 (418h48m56.578514224s) [noise=true , address=77.90.31.16:4740 , piece_id=KYCKJ7F7CNIXIVAZZ7CQPWAYNCMIM5RKO4IKXETP5EZNOAGKGONA, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.707910952 (418h48m56.578993997s) [noise=true , address=45.91.116.5:4330 , piece_id=EMSR7LFH5Q6R7YAJNH45TDQS2E4KX25Q335IFFUORN5AJJWS7W5Q, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.708492299 (418h48m56.579575344s) [noise=true , address=64.49.13.139:4478 , piece_id=QUBJUJHQ6AYU6L67666PA7BHCRL2IIFOJDISOISTEK4JWABGMXIQ, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.708774406 (418h48m56.579857441s) [noise=true , address=45.91.116.14:4213 , piece_id=VLD3RNN2RLOQI2AKMHNY3G57QJPDNJ3BA5JDRRJHCHNHS6YE52JA, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:18:00.137346621 (418h49m0.008429666s) [upload=367702]: Next returned nil segment Dec 23 02:18:00.137360507 (418h49m0.008443552s) [upload=367702]: Waiting for error group managing segments... Dec 23 02:18:01.750516581 (418h49m1.621599616s) [noise=true , address=64.49.13.137:4500 , piece_id=RFGLMWE4OHC7RL2NBPX6VRKG4YGRVCIMBHQFW2W5XAEE376UTVVA, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:02.125124405 (418h49m1.99620745s) [noise=true , address=77.90.31.18:4815 , piece_id=XQDHXOY2ZQQHZZHMGLO5GKZOUJZFKKWGLDJRHJLWCFEX6PKNNHWA, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:02.226869841 (418h49m2.097952876s) [noise=true , address=45.91.116.8:4060 , piece_id=TRNAVEFSFXMIN3DFMQ2DUAKKDTSZH36TCSYZXFKALUM7TCKM3Y6A, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:05.457222504 (418h49m5.328305539s) [noise=true , address=77.90.31.13:4651 , piece_id=H6MCF4TXI5WFQ2BBCPXL3OX6OA4NCRBIH6IBLMPYPIJBETEN6UGA, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:06.079641082 (418h49m5.950724128s) [noise=true , address=45.91.116.12:4257 , piece_id=SPTNFMB3LXNY74LCASNSAZIPCOAQHO5V6SOFP27Y4IY25CHWBO2A, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:09.265548070 (418h49m9.136631105s) [noise=true , address=64.49.13.139:4478 , piece_id=QUBJUJHQ6AYU6L67666PA7BHCRL2IIFOJDISOISTEK4JWABGMXIQ, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:10.880784395 (418h49m10.751867441s) [noise=true , address=45.91.116.14:4213 , piece_id=VLD3RNN2RLOQI2AKMHNY3G57QJPDNJ3BA5JDRRJHCHNHS6YE52JA, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:12.686956874 (418h49m12.558039929s) [noise=true , address=145.79.227.28:4742 , piece_id=3IMQS3J42XXGHCZFMBYPDCPCJYRY3I3EUIQ2Z2JOK4TPXQQH75OQ, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:12.832920510 (418h49m12.704003545s) [noise=true , address=45.91.116.5:4330 , piece_id=EMSR7LFH5Q6R7YAJNH45TDQS2E4KX25Q335IFFUORN5AJJWS7W5Q, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:22.466644063 (418h49m22.337727098s) [noise=true , address=77.90.31.16:4740 , piece_id=KYCKJ7F7CNIXIVAZZ7CQPWAYNCMIM5RKO4IKXETP5EZNOAGKGONA, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:34.649740548 (418h49m34.520823583s) [noise=true , address=64.49.13.169:4715 , piece_id=6RCBEKXRJ56IGXWT4PVSTDYRIH3UCAD22C4Z5BU33U5YIPWBEPDQ, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:38.250548484 (418h49m38.121631529s) [noise=true , address=145.79.227.17:4984 , piece_id=DAMTMQIEBM3I2QZTWVEGISFJ5T7I3LMZ3CTXUFXO2CS775GY4F7Q, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:43.329219400 (418h49m43.200302445s) [noise=true , address=145.79.227.23:4549 , piece_id=XH7BPLOCXKVTJQDYKIRKFXVLHIXXNJFAXCFZPSSASWLOG3GM5PEQ, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:43.400788824 (418h49m43.271871859s) [noise=true , address=145.79.227.32:4848 , piece_id=NMEVAZ6HHKE2Z4Q5G4VLNPOIURXE5TXXUMLFJLURZUI4Z62W7VMA, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:43.400819082 (418h49m43.271902117s) [seg_pos={0 0}, upload=367702]: Segment reached optimal threshold of 14 pieces. Dec 23 02:18:43.401090247 (418h49m43.272173302s) [noise=true , address=5.231.73.4:4030 , piece_id=M6FJSRGUNCEIBMNKLBK7HHFL4DA3PDD2MZ2GN2MJKAFSH3XRI7XQ, seg_pos={0 0}, upload=367702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rWJM3vC8XrUAfVaoqVZm1qzqcG8AqxLzncExPHEak5oo3QuPA): protocol: expected piece hash; context canceled; EOF Dec 23 02:18:43.401100788 (418h49m43.272183823s) [noise=true , address=5.231.73.4:4030 , piece_id=M6FJSRGUNCEIBMNKLBK7HHFL4DA3PDD2MZ2GN2MJKAFSH3XRI7XQ, seg_pos={0 0}, upload=367702]: Long tail cancellation due to optimal threshold Dec 23 02:18:43.401281792 (418h49m43.272364837s) [noise=true , address=77.90.31.14:4667 , piece_id=5ZXKL7BLGSD3TCZNAHNK7STV2RKAZQWCHN6PROBXAPTA5ZBCPT6A, seg_pos={0 0}, upload=367702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S): protocol: expected piece hash; context canceled; EOF Dec 23 02:18:43.401290889 (418h49m43.272373925s) [noise=true , address=77.90.31.14:4667 , piece_id=5ZXKL7BLGSD3TCZNAHNK7STV2RKAZQWCHN6PROBXAPTA5ZBCPT6A, seg_pos={0 0}, upload=367702]: Long tail cancellation due to optimal threshold Dec 23 02:18:43.401412151 (418h49m43.272495186s) [noise=true , address=145.79.227.13:4811 , piece_id=KVHW3GEE4KWLOOUJBRVOOLDA3EGPRUU7VHPH7TQXLURUCCU5J77Q, seg_pos={0 0}, upload=367702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TxjP4CnxyU4t2DTNBjBMBxnizcHCD42fMrB5Da6Qiqg1Eg17V): protocol: context canceled; context canceled Dec 23 02:18:43.401418663 (418h49m43.272501698s) [noise=true , address=145.79.227.13:4811 , piece_id=KVHW3GEE4KWLOOUJBRVOOLDA3EGPRUU7VHPH7TQXLURUCCU5J77Q, seg_pos={0 0}, upload=367702]: Long tail cancellation due to optimal threshold Dec 23 02:18:43.401630526 (418h49m43.272713561s) [noise=true , address=145.79.227.30:4796 , piece_id=IF6NPIC5DTTMJ4DZMIDD6SFN7TGDBN6RD3H7KT7A4OC7VJVCEZOA, seg_pos={0 0}, upload=367702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: read tcp 188.166.225.170:44484->145.79.227.30:4796: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Dec 23 02:18:43.401638241 (418h49m43.272721276s) [noise=true , address=145.79.227.30:4796 , piece_id=IF6NPIC5DTTMJ4DZMIDD6SFN7TGDBN6RD3H7KT7A4OC7VJVCEZOA, seg_pos={0 0}, upload=367702]: Long tail cancellation due to optimal threshold Dec 23 02:18:43.401657227 (418h49m43.272740273s) [seg_pos={0 0}, upload=367702]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 23 02:18:43.401864953 (418h49m43.272947988s) [upload=367702]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 02:18:43.401870614 (418h49m43.272953649s) [upload=367702]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 02:18:43.703068278 (418h49m43.574151323s) [upload=367702]: Done uploading segments... Dec 23 02:18:43.703144954 (418h49m43.574227999s) [upload=367702]: Upload finished. err: <nil> Duration: 8.199066829s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 6.733562916s trace: https://jaeger.dev.storj.io/trace/463d97d26edafa8a FAIL (2025-12-23 02:25:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=367700]: Uploading piece... Dec 23 02:17:52.581508683 (418h48m52.452591718s) [noise=true , address=185.167.204.206:4040 , piece_id=Y6HVWGUESE2BSEIKEMDSLNHWDTOEKGBR76NIFRRQU6KFTVHD4Q7Q, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.581711800 (418h48m52.452794835s) [noise=true , address=185.167.204.206:4058 , piece_id=4CZUKD7KF6H3JH6DLWT3KM2PQC72NRYW4YPN7NL445VBER4QR6SA, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.581958349 (418h48m52.453041384s) [noise=true , address=89.149.201.147:4016 , piece_id=MFNURTZ2HJFACIAHZUO7YZJXRJZB26AVF24JLU3I7IYKDPZ2JJIQ, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.582064401 (418h48m52.453147436s) [noise=true , address=89.149.201.147:4009 , piece_id=WGVNGAHNZIVLBR665VB7GO77EZ7SSJOC5P7ML2UAK6MKV4BAF27A, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.582336870 (418h48m52.453419905s) [noise=true , address=81.171.14.70:4025 , piece_id=PFRRLSNJYBUCYSOVOE3VHDWAKTEPMPQRTE4U2VTPBT7KGYVNQHIQ, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.582391083 (418h48m52.453474118s) [noise=true , address=81.171.14.70:4029 , piece_id=JKKAJ3PXL7WDZ7XNKM752C6NA4B4FUN2AZJCGAZPDVNAGKTAOK7Q, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.582715682 (418h48m52.453798717s) [noise=true , address=89.149.201.147:4002 , piece_id=QCDLIQPY7COKADVRRRUN7UBEIV2AARIPA64LMFCCYPFNA54TA7VQ, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.582847693 (418h48m52.453930738s) [noise=true , address=185.167.204.205:4033 , piece_id=42CCWQIIXKFWWCLUSLGSFQHZMA2UPGZJMOSWD7Z562DV6ABPDVRQ, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.583274976 (418h48m52.454358022s) [noise=true , address=185.167.204.206:4066 , piece_id=WAVNL7LBCOQEW3CFA3DWRITRTBGCWYVASH43KC4RFHE5KFQUQ45A, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.583662655 (418h48m52.45474569s) [noise=true , address=185.167.204.206:4070 , piece_id=GA3GZFHS36RXW4PWKSHZ2QOAXQJVNIWVMED3XAAGG5GQPA27ST7A, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:53.405090439 (418h48m53.276173484s) [upload=367700]: Next returned nil segment Dec 23 02:17:53.405105007 (418h48m53.276188042s) [upload=367700]: Waiting for error group managing segments... Dec 23 02:17:53.798794305 (418h48m53.66987735s) [noise=true , address=185.167.204.205:4025 , piece_id=XY5XNAAQZFPKWJP7V5LWBBH5ZQ7XFUTQ657QEF5KXFTP2EKCSIBA, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.799500561 (418h48m53.670583596s) [noise=true , address=185.167.204.205:4016 , piece_id=523RGOAK7OI6VYQY5X4XT5Y6F5C7NMB2WDXPN74Q27P3BCH2Q37A, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.799686735 (418h48m53.67076977s) [noise=true , address=185.167.204.205:4036 , piece_id=IQ6I5X4HC2ANQSM72N7X4OJWVTUMOWH5YPSSRHUFTGYSXW5ZK4WQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.800348164 (418h48m53.671431209s) [noise=true , address=185.167.204.205:4033 , piece_id=42CCWQIIXKFWWCLUSLGSFQHZMA2UPGZJMOSWD7Z562DV6ABPDVRQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.802008737 (418h48m53.673091782s) [noise=true , address=185.167.204.206:4057 , piece_id=UNUF4NZ5CL74EWLPZLVX44MAYYUQFIAZ54N7FGBLCQHEKLGX6RSQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.802484733 (418h48m53.673567778s) [noise=true , address=185.167.204.206:4040 , piece_id=Y6HVWGUESE2BSEIKEMDSLNHWDTOEKGBR76NIFRRQU6KFTVHD4Q7Q, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.803136594 (418h48m53.674219639s) [noise=true , address=185.167.204.206:4070 , piece_id=GA3GZFHS36RXW4PWKSHZ2QOAXQJVNIWVMED3XAAGG5GQPA27ST7A, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.803750904 (418h48m53.674833939s) [noise=true , address=185.167.204.206:4053 , piece_id=GG5TA7D4KK2IGSKRO4T3APKSZ43IWLND3L54EXYD2LRFIKGI5MRQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.804347520 (418h48m53.675430565s) [noise=true , address=185.167.204.206:4058 , piece_id=4CZUKD7KF6H3JH6DLWT3KM2PQC72NRYW4YPN7NL445VBER4QR6SA, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.805167522 (418h48m53.676250567s) [noise=true , address=185.167.204.206:4054 , piece_id=3QWOKGEC5JODMZCL5R5ULU7EUAKGI2W5QWONT5H2I7EOBH7PVODA, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.805898293 (418h48m53.676981328s) [noise=true , address=185.167.204.206:4066 , piece_id=WAVNL7LBCOQEW3CFA3DWRITRTBGCWYVASH43KC4RFHE5KFQUQ45A, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.808200908 (418h48m53.679283943s) [noise=true , address=185.167.204.206:4064 , piece_id=4K4KD2ZMFXUZT6W3OV7E6BC5UGQHSVZXR67B2QPXIPKGAM6CTVSA, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.808393705 (418h48m53.67947675s) [noise=true , address=185.167.204.205:4004 , piece_id=G5Q7QERF2O3WXIDK5ZAFGSZIN6VXJOFYHAPEC647NIXIJQCSNLKA, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.808844694 (418h48m53.679927739s) [noise=true , address=185.167.204.205:4028 , piece_id=DINID3XD5734Z3RWWS3WPIFLGRUXZMZUNXC2BC3NMX7LWTSDVN3Q, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.810924914 (418h48m53.682007959s) [noise=true , address=185.167.204.206:4047 , piece_id=JSWKSSXVNEFRZPWP56EHG26DRYA7V4K2LPEBCKUOBUG2WET5ZQ7A, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.813330996 (418h48m53.684414031s) [noise=true , address=185.167.204.206:4069 , piece_id=OLTP357X7TH2ABIRWBMAI445KR5SGJVY77PPIQFLRD2KFCBOML2Q, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.813884790 (418h48m53.684967826s) [noise=true , address=185.167.204.205:4001 , piece_id=LMIOF4TXW2JJBYGBNU36M46PXZUNKBKFK56XVNMWRQVR7LSZ3QCQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.815627059 (418h48m53.686710094s) [noise=true , address=185.167.204.206:4067 , piece_id=D24D7H4IJKMKHRJUCQ2ZAYDIVRUYHK2YHEYLVCIL3VA4P5FQJB5A, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:54.448896803 (418h48m54.319979838s) [noise=true , address=81.171.14.70:4029 , piece_id=JKKAJ3PXL7WDZ7XNKM752C6NA4B4FUN2AZJCGAZPDVNAGKTAOK7Q, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:54.835287845 (418h48m54.70637088s) [noise=true , address=81.171.14.70:4025 , piece_id=PFRRLSNJYBUCYSOVOE3VHDWAKTEPMPQRTE4U2VTPBT7KGYVNQHIQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:54.946890399 (418h48m54.817973434s) [noise=true , address=81.171.14.70:4040 , piece_id=6ZIPFOQCGRNK3TDTODGI34TL2WM65BQSRLXXAFX3XQ3VCPAVT2YA, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:54.960812961 (418h48m54.831895996s) [noise=true , address=81.171.14.70:4034 , piece_id=2EA6SNLDV3ZVTNQ5UXEXIC2SJTPH3D6VCVZJI2IAGS22U2J6Y6WQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:55.069208920 (418h48m54.940291965s) [noise=true , address=89.149.201.147:4009 , piece_id=WGVNGAHNZIVLBR665VB7GO77EZ7SSJOC5P7ML2UAK6MKV4BAF27A, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:55.073463732 (418h48m54.944546768s) [noise=true , address=89.149.201.147:4002 , piece_id=QCDLIQPY7COKADVRRRUN7UBEIV2AARIPA64LMFCCYPFNA54TA7VQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:55.132812454 (418h48m55.003895489s) [noise=true , address=89.149.201.147:4010 , piece_id=LZXMVADI5CHNTPWZNXNBEFI3SMGAWQGNUIWDNPM4JMRBLZ62HO4A, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:55.185887030 (418h48m55.056970065s) [noise=true , address=89.149.201.147:4012 , piece_id=2Z64OIS2BBYH6ZCKBXHW4XQZY4FFASHDIHH3C3YVY7MGTTBI6QNA, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:55.185911748 (418h48m55.056994783s) [seg_pos={0 0}, upload=367700]: Segment reached optimal threshold of 26 pieces. Dec 23 02:17:55.186137848 (418h48m55.057220893s) [noise=true , address=89.149.201.147:4015 , piece_id=WENNEH2N2S7NV3TQILPUZLFRBV5CO6VMP52NKDIQE23NO2X7GDWQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RbD6M2MHJfxcJXogZG8nUGFmeZjwUQ1auN6KGWJMPTffdK42K): protocol: expected piece hash; context canceled; EOF Dec 23 02:17:55.186148097 (418h48m55.057231132s) [noise=true , address=89.149.201.147:4015 , piece_id=WENNEH2N2S7NV3TQILPUZLFRBV5CO6VMP52NKDIQE23NO2X7GDWQ, seg_pos={0 0}, upload=367700]: Long tail cancellation due to optimal threshold Dec 23 02:17:55.186212209 (418h48m55.057295244s) [noise=true , address=89.149.201.147:4016 , piece_id=MFNURTZ2HJFACIAHZUO7YZJXRJZB26AVF24JLU3I7IYKDPZ2JJIQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XsvrVhDGkU8BkYsBGmTtax47E43dZAYuGwD8WBMpp3n4GgDM5): protocol: expected piece hash; context canceled; EOF Dec 23 02:17:55.186228150 (418h48m55.057311185s) [noise=true , address=89.149.201.147:4016 , piece_id=MFNURTZ2HJFACIAHZUO7YZJXRJZB26AVF24JLU3I7IYKDPZ2JJIQ, seg_pos={0 0}, upload=367700]: Long tail cancellation due to optimal threshold Dec 23 02:17:55.186247767 (418h48m55.057330802s) [seg_pos={0 0}, upload=367700]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 23 02:17:55.186337368 (418h48m55.057420403s) [upload=367700]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 02:17:55.186342387 (418h48m55.057425422s) [upload=367700]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 02:17:55.430535735 (418h48m55.30161878s) [upload=367700]: Done uploading segments... Dec 23 02:17:55.430557656 (418h48m55.301640701s) [upload=367700]: Upload finished. err: <nil> Duration: 6.733610017s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 15.440509145s trace: https://jaeger.dev.storj.io/trace/5b87e1ad666dcc24 FAIL (2025-12-23 02:25:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 2) ======LOGS====== 7Y7UWG6NFFSQ, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:05.480444685 (418h49m5.35152773s) [noise=true , address=45.91.116.4:4937 , piece_id=IIV6VFBKACIVI3M6A6L2WSDU54D3MRGZ36LG6E3E4SXKQVRX74LQ, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:05.480943264 (418h49m5.352026309s) [noise=true , address=145.79.227.9:4366 , piece_id=73FK36X6U4CYXVKSGCTLK4YCL2QB3CQKUST2IY4OQGG4JWMLTV6A, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:05.482473688 (418h49m5.353556733s) [noise=true , address=145.79.227.41:4086 , piece_id=KXPKPLKHLFQ5GU7Z7LM3NXGWGXSGISKBOHAWPJ7QGG4ESA64YYWA, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:05.482598135 (418h49m5.35368118s) [noise=true , address=145.79.227.43:4159 , piece_id=WUJFSGFHRK4PHRLS4H3MNT5TVXYR6MSTUQXRN6QBXI47WRPKKN4A, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:05.485571146 (418h49m5.356654182s) [noise=true , address=77.90.31.12:4589 , piece_id=RD5KILZQDADOR2UPRRAO3XM7YWI7AQ57HBR37VC44DNZPOS555SA, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:05.488437175 (418h49m5.35952021s) [noise=true , address=145.79.227.35:4481 , piece_id=EVLROBIPMCRCZYYUSUKKCHCMTJH6G2H56QJR6ZPIF7R7HVVCP6CA, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:05.491243618 (418h49m5.362326653s) [noise=true , address=145.79.227.39:4015 , piece_id=UAGG4FHO2SDQKDF6T4QIGVN7IMNANYWTD3SHYRMBVGCPS3Z4VIYA, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:05.492801755 (418h49m5.36388479s) [noise=true , address=77.90.31.16:4735 , piece_id=XEZKWPQDBCT7YRGYTH4TFR4BDCP34BPJB6OVFTYGMQ2NQI34NVZA, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:05.495283631 (418h49m5.366366666s) [noise=true , address=145.79.227.37:4565 , piece_id=PAKR2BG4S3KYFXOLS7BL5AANJMIHMN477KGMJJ4XIHMTF7UOTCHA, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:05.495702669 (418h49m5.366785704s) [noise=true , address=145.79.227.42:4121 , piece_id=ICOCUSNLYIVKOJTONNBAEPWGSRTB44H2KAHESVBGGBEDOEHROYGA, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:05.499769763 (418h49m5.370852808s) [noise=true , address=145.79.227.51:4411 , piece_id=2JQUVX7JG3GYCRYJMYE7AQNIX7XG7XEYRXAC6JTTYRM343CVP3IA, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:08.589264332 (418h49m8.460347367s) [upload=367705]: Next returned nil segment Dec 23 02:18:08.589286223 (418h49m8.460369258s) [upload=367705]: Waiting for error group managing segments... Dec 23 02:18:12.569357328 (418h49m12.440440364s) [noise=true , address=45.91.116.4:4937 , piece_id=IIV6VFBKACIVI3M6A6L2WSDU54D3MRGZ36LG6E3E4SXKQVRX74LQ, seg_pos={0 0}, upload=367705]: Done uploading piece. err: <nil> Dec 23 02:18:15.106396805 (418h49m14.977479851s) [noise=true , address=77.90.31.12:4589 , piece_id=RD5KILZQDADOR2UPRRAO3XM7YWI7AQ57HBR37VC44DNZPOS555SA, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 23 02:18:17.598569766 (418h49m17.469652801s) [noise=true , address=77.90.31.16:4735 , piece_id=XEZKWPQDBCT7YRGYTH4TFR4BDCP34BPJB6OVFTYGMQ2NQI34NVZA, seg_pos={0 0}, upload=367705]: Done uploading piece. err: <nil> Dec 23 02:18:33.434728164 (418h49m33.305811199s) [noise=true , address=145.79.227.42:4121 , piece_id=ICOCUSNLYIVKOJTONNBAEPWGSRTB44H2KAHESVBGGBEDOEHROYGA, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: read tcp 188.166.225.170:50058->145.79.227.42:4121: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.434850206 (418h49m33.305933241s) [noise=true , address=145.79.227.15:4874 , piece_id=PWRV5UFUDENFBPJFK3KYNXMYLQZBHFG6KBJFZXA6P32NT7ZNOWDA, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: read tcp 188.166.225.170:50240->145.79.227.15:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.434955458 (418h49m33.306038493s) [noise=true , address=145.79.227.50:4373 , piece_id=PZRIRRQQETAX56EVN7LY3Z5DUPUYUE7N7H2ZTYWK7Y7UWG6NFFSQ, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, address:145.79.227.50:4373): protocol: read tcp 188.166.225.170:54252->145.79.227.50:4373: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.435015211 (418h49m33.306098246s) [noise=true , address=145.79.227.9:4366 , piece_id=73FK36X6U4CYXVKSGCTLK4YCL2QB3CQKUST2IY4OQGG4JWMLTV6A, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: rpc: dial tcp 145.79.227.9:4366: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.435058384 (418h49m33.306141429s) [noise=true , address=145.79.227.43:4159 , piece_id=WUJFSGFHRK4PHRLS4H3MNT5TVXYR6MSTUQXRN6QBXI47WRPKKN4A, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA, address:145.79.227.43:4159): protocol: read tcp 188.166.225.170:49750->145.79.227.43:4159: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.435156329 (418h49m33.306239374s) [noise=true , address=145.79.227.47:4267 , piece_id=CFW6HHUK6IP232DVR64FPSBPULEIVPVGCS2HR2523RAIP5N77POQ, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8, address:145.79.227.47:4267): protocol: read tcp 188.166.225.170:33996->145.79.227.47:4267: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.435495826 (418h49m33.306578871s) [noise=true , address=77.90.31.21:4877 , piece_id=J3XPMHNIGDTRPYW2ITOOW4FRJ4SUKMCCGRP22WKD55A7OA7EOV3A, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 188.166.225.170:35802->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.435680898 (418h49m33.306763943s) [noise=true , address=145.79.227.41:4086 , piece_id=KXPKPLKHLFQ5GU7Z7LM3NXGWGXSGISKBOHAWPJ7QGG4ESA64YYWA, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, address:145.79.227.41:4086): protocol: read tcp 188.166.225.170:33906->145.79.227.41:4086: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.435701508 (418h49m33.306784543s) [noise=true , address=145.79.227.39:4015 , piece_id=UAGG4FHO2SDQKDF6T4QIGVN7IMNANYWTD3SHYRMBVGCPS3Z4VIYA, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, address:145.79.227.39:4015): protocol: read tcp 188.166.225.170:45508->145.79.227.39:4015: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.435883414 (418h49m33.306966449s) [noise=true , address=145.79.227.37:4565 , piece_id=PAKR2BG4S3KYFXOLS7BL5AANJMIHMN477KGMJJ4XIHMTF7UOTCHA, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:1nvvNczHtQrSEJuJbErnjiAWa4BRDzBrQmBNbB7RrrqN1JWNHT, address:145.79.227.37:4565): protocol: read tcp 188.166.225.170:43228->145.79.227.37:4565: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.435913952 (418h49m33.306996987s) [noise=true , address=145.79.227.38:4978 , piece_id=5VMIQGZ32DI7IE6Z6HMVCJUXIMBFVN6BWONEBE452PG5WDWZXDZQ, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: read tcp 188.166.225.170:57482->145.79.227.38:4978: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.436073857 (418h49m33.307156892s) [noise=true , address=145.79.227.35:4481 , piece_id=EVLROBIPMCRCZYYUSUKKCHCMTJH6G2H56QJR6ZPIF7R7HVVCP6CA, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp, address:145.79.227.35:4481): protocol: read tcp 188.166.225.170:34156->145.79.227.35:4481: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.436167987 (418h49m33.307251022s) [noise=true , address=145.79.227.51:4411 , piece_id=2JQUVX7JG3GYCRYJMYE7AQNIX7XG7XEYRXAC6JTTYRM343CVP3IA, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR, address:145.79.227.51:4411): protocol: read tcp 188.166.225.170:54092->145.79.227.51:4411: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.436191531 (418h49m33.307274566s) [seg_pos={0 0}, upload=367705]: Done waiting for segment. successful: 2 optimal: 13 errs: <nil> Dec 23 02:18:33.726571837 (418h49m33.597654882s) [upload=367705]: Done uploading segments... Dec 23 02:18:33.726612455 (418h49m33.59769549s) [upload=367705]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 2) Duration: 15.440561916s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................................................X..........] duration: 3.600552973s trace: https://jaeger.dev.storj.io/trace/7f7fb60dfbab7793 FAIL (2025-12-23 02:25:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.600603669s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 3.301898932s trace: https://jaeger.dev.storj.io/trace/20c185d15563c01c FAIL (2025-12-23 02:24:59 UTC) orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 39) Duration: 3.301927948s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 5.81512835s trace: https://jaeger.dev.storj.io/trace/70da446b2b2ff016 FAIL (2025-12-23 02:24:40 UTC) orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 46) Duration: 5.815157966s core uplink us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 4.900090719s trace: https://jaeger.dev.storj.io/trace/14f27c56f52ae4e9 FAIL (2025-12-23 02:24:40 UTC) orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 45) Duration: 4.900114716s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 11.402041765s trace: https://jaeger.dev.storj.io/trace/2c98a82e82cbeda7 FAIL (2025-12-23 02:24:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== KL7BLGSD3TCZNAHNK7STV2RKAZQWCHN6PROBXAPTA5ZBCPT6A, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.700896105 (418h48m56.57197914s) [noise=true , address=145.79.227.30:4796 , piece_id=IF6NPIC5DTTMJ4DZMIDD6SFN7TGDBN6RD3H7KT7A4OC7VJVCEZOA, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.701329429 (418h48m56.572412474s) [noise=true , address=145.79.227.32:4848 , piece_id=NMEVAZ6HHKE2Z4Q5G4VLNPOIURXE5TXXUMLFJLURZUI4Z62W7VMA, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.701714032 (418h48m56.572797077s) [noise=true , address=145.79.227.28:4742 , piece_id=3IMQS3J42XXGHCZFMBYPDCPCJYRY3I3EUIQ2Z2JOK4TPXQQH75OQ, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.702231197 (418h48m56.573314232s) [noise=true , address=64.49.13.137:4500 , piece_id=RFGLMWE4OHC7RL2NBPX6VRKG4YGRVCIMBHQFW2W5XAEE376UTVVA, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.702751507 (418h48m56.573834552s) [noise=true , address=64.49.13.169:4715 , piece_id=6RCBEKXRJ56IGXWT4PVSTDYRIH3UCAD22C4Z5BU33U5YIPWBEPDQ, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.703271658 (418h48m56.574354693s) [noise=true , address=45.91.116.12:4257 , piece_id=SPTNFMB3LXNY74LCASNSAZIPCOAQHO5V6SOFP27Y4IY25CHWBO2A, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.703948437 (418h48m56.575031482s) [noise=true , address=145.79.227.23:4549 , piece_id=XH7BPLOCXKVTJQDYKIRKFXVLHIXXNJFAXCFZPSSASWLOG3GM5PEQ, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.703969256 (418h48m56.575052291s) [noise=true , address=5.231.73.4:4030 , piece_id=M6FJSRGUNCEIBMNKLBK7HHFL4DA3PDD2MZ2GN2MJKAFSH3XRI7XQ, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.704457947 (418h48m56.575540982s) [noise=true , address=145.79.227.17:4984 , piece_id=DAMTMQIEBM3I2QZTWVEGISFJ5T7I3LMZ3CTXUFXO2CS775GY4F7Q, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.704834724 (418h48m56.575917769s) [noise=true , address=77.90.31.18:4815 , piece_id=XQDHXOY2ZQQHZZHMGLO5GKZOUJZFKKWGLDJRHJLWCFEX6PKNNHWA, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.706591609 (418h48m56.577674654s) [noise=true , address=77.90.31.13:4651 , piece_id=H6MCF4TXI5WFQ2BBCPXL3OX6OA4NCRBIH6IBLMPYPIJBETEN6UGA, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.707118754 (418h48m56.578201789s) [noise=true , address=145.79.227.13:4811 , piece_id=KVHW3GEE4KWLOOUJBRVOOLDA3EGPRUU7VHPH7TQXLURUCCU5J77Q, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.707431179 (418h48m56.578514224s) [noise=true , address=77.90.31.16:4740 , piece_id=KYCKJ7F7CNIXIVAZZ7CQPWAYNCMIM5RKO4IKXETP5EZNOAGKGONA, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.707910952 (418h48m56.578993997s) [noise=true , address=45.91.116.5:4330 , piece_id=EMSR7LFH5Q6R7YAJNH45TDQS2E4KX25Q335IFFUORN5AJJWS7W5Q, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.708492299 (418h48m56.579575344s) [noise=true , address=64.49.13.139:4478 , piece_id=QUBJUJHQ6AYU6L67666PA7BHCRL2IIFOJDISOISTEK4JWABGMXIQ, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.708774406 (418h48m56.579857441s) [noise=true , address=45.91.116.14:4213 , piece_id=VLD3RNN2RLOQI2AKMHNY3G57QJPDNJ3BA5JDRRJHCHNHS6YE52JA, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:18:00.137346621 (418h49m0.008429666s) [upload=367702]: Next returned nil segment Dec 23 02:18:00.137360507 (418h49m0.008443552s) [upload=367702]: Waiting for error group managing segments... Dec 23 02:18:01.750516581 (418h49m1.621599616s) [noise=true , address=64.49.13.137:4500 , piece_id=RFGLMWE4OHC7RL2NBPX6VRKG4YGRVCIMBHQFW2W5XAEE376UTVVA, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:02.125124405 (418h49m1.99620745s) [noise=true , address=77.90.31.18:4815 , piece_id=XQDHXOY2ZQQHZZHMGLO5GKZOUJZFKKWGLDJRHJLWCFEX6PKNNHWA, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:02.226869841 (418h49m2.097952876s) [noise=true , address=45.91.116.8:4060 , piece_id=TRNAVEFSFXMIN3DFMQ2DUAKKDTSZH36TCSYZXFKALUM7TCKM3Y6A, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:05.457222504 (418h49m5.328305539s) [noise=true , address=77.90.31.13:4651 , piece_id=H6MCF4TXI5WFQ2BBCPXL3OX6OA4NCRBIH6IBLMPYPIJBETEN6UGA, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:06.079641082 (418h49m5.950724128s) [noise=true , address=45.91.116.12:4257 , piece_id=SPTNFMB3LXNY74LCASNSAZIPCOAQHO5V6SOFP27Y4IY25CHWBO2A, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:09.265548070 (418h49m9.136631105s) [noise=true , address=64.49.13.139:4478 , piece_id=QUBJUJHQ6AYU6L67666PA7BHCRL2IIFOJDISOISTEK4JWABGMXIQ, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:10.880784395 (418h49m10.751867441s) [noise=true , address=45.91.116.14:4213 , piece_id=VLD3RNN2RLOQI2AKMHNY3G57QJPDNJ3BA5JDRRJHCHNHS6YE52JA, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:12.686956874 (418h49m12.558039929s) [noise=true , address=145.79.227.28:4742 , piece_id=3IMQS3J42XXGHCZFMBYPDCPCJYRY3I3EUIQ2Z2JOK4TPXQQH75OQ, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:12.832920510 (418h49m12.704003545s) [noise=true , address=45.91.116.5:4330 , piece_id=EMSR7LFH5Q6R7YAJNH45TDQS2E4KX25Q335IFFUORN5AJJWS7W5Q, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:22.466644063 (418h49m22.337727098s) [noise=true , address=77.90.31.16:4740 , piece_id=KYCKJ7F7CNIXIVAZZ7CQPWAYNCMIM5RKO4IKXETP5EZNOAGKGONA, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:34.649740548 (418h49m34.520823583s) [noise=true , address=64.49.13.169:4715 , piece_id=6RCBEKXRJ56IGXWT4PVSTDYRIH3UCAD22C4Z5BU33U5YIPWBEPDQ, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:38.250548484 (418h49m38.121631529s) [noise=true , address=145.79.227.17:4984 , piece_id=DAMTMQIEBM3I2QZTWVEGISFJ5T7I3LMZ3CTXUFXO2CS775GY4F7Q, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:43.329219400 (418h49m43.200302445s) [noise=true , address=145.79.227.23:4549 , piece_id=XH7BPLOCXKVTJQDYKIRKFXVLHIXXNJFAXCFZPSSASWLOG3GM5PEQ, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:43.400788824 (418h49m43.271871859s) [noise=true , address=145.79.227.32:4848 , piece_id=NMEVAZ6HHKE2Z4Q5G4VLNPOIURXE5TXXUMLFJLURZUI4Z62W7VMA, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:43.400819082 (418h49m43.271902117s) [seg_pos={0 0}, upload=367702]: Segment reached optimal threshold of 14 pieces. Dec 23 02:18:43.401090247 (418h49m43.272173302s) [noise=true , address=5.231.73.4:4030 , piece_id=M6FJSRGUNCEIBMNKLBK7HHFL4DA3PDD2MZ2GN2MJKAFSH3XRI7XQ, seg_pos={0 0}, upload=367702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rWJM3vC8XrUAfVaoqVZm1qzqcG8AqxLzncExPHEak5oo3QuPA): protocol: expected piece hash; context canceled; EOF Dec 23 02:18:43.401100788 (418h49m43.272183823s) [noise=true , address=5.231.73.4:4030 , piece_id=M6FJSRGUNCEIBMNKLBK7HHFL4DA3PDD2MZ2GN2MJKAFSH3XRI7XQ, seg_pos={0 0}, upload=367702]: Long tail cancellation due to optimal threshold Dec 23 02:18:43.401281792 (418h49m43.272364837s) [noise=true , address=77.90.31.14:4667 , piece_id=5ZXKL7BLGSD3TCZNAHNK7STV2RKAZQWCHN6PROBXAPTA5ZBCPT6A, seg_pos={0 0}, upload=367702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S): protocol: expected piece hash; context canceled; EOF Dec 23 02:18:43.401290889 (418h49m43.272373925s) [noise=true , address=77.90.31.14:4667 , piece_id=5ZXKL7BLGSD3TCZNAHNK7STV2RKAZQWCHN6PROBXAPTA5ZBCPT6A, seg_pos={0 0}, upload=367702]: Long tail cancellation due to optimal threshold Dec 23 02:18:43.401412151 (418h49m43.272495186s) [noise=true , address=145.79.227.13:4811 , piece_id=KVHW3GEE4KWLOOUJBRVOOLDA3EGPRUU7VHPH7TQXLURUCCU5J77Q, seg_pos={0 0}, upload=367702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TxjP4CnxyU4t2DTNBjBMBxnizcHCD42fMrB5Da6Qiqg1Eg17V): protocol: context canceled; context canceled Dec 23 02:18:43.401418663 (418h49m43.272501698s) [noise=true , address=145.79.227.13:4811 , piece_id=KVHW3GEE4KWLOOUJBRVOOLDA3EGPRUU7VHPH7TQXLURUCCU5J77Q, seg_pos={0 0}, upload=367702]: Long tail cancellation due to optimal threshold Dec 23 02:18:43.401630526 (418h49m43.272713561s) [noise=true , address=145.79.227.30:4796 , piece_id=IF6NPIC5DTTMJ4DZMIDD6SFN7TGDBN6RD3H7KT7A4OC7VJVCEZOA, seg_pos={0 0}, upload=367702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: read tcp 188.166.225.170:44484->145.79.227.30:4796: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Dec 23 02:18:43.401638241 (418h49m43.272721276s) [noise=true , address=145.79.227.30:4796 , piece_id=IF6NPIC5DTTMJ4DZMIDD6SFN7TGDBN6RD3H7KT7A4OC7VJVCEZOA, seg_pos={0 0}, upload=367702]: Long tail cancellation due to optimal threshold Dec 23 02:18:43.401657227 (418h49m43.272740273s) [seg_pos={0 0}, upload=367702]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 23 02:18:43.401864953 (418h49m43.272947988s) [upload=367702]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 02:18:43.401870614 (418h49m43.272953649s) [upload=367702]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 02:18:43.703068278 (418h49m43.574151323s) [upload=367702]: Done uploading segments... Dec 23 02:18:43.703144954 (418h49m43.574227999s) [upload=367702]: Upload finished. err: <nil> Duration: 11.402064007s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 5.733923623s trace: https://jaeger.dev.storj.io/trace/463d97d26edafa8a FAIL (2025-12-23 02:24:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=367700]: Uploading piece... Dec 23 02:17:52.581508683 (418h48m52.452591718s) [noise=true , address=185.167.204.206:4040 , piece_id=Y6HVWGUESE2BSEIKEMDSLNHWDTOEKGBR76NIFRRQU6KFTVHD4Q7Q, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.581711800 (418h48m52.452794835s) [noise=true , address=185.167.204.206:4058 , piece_id=4CZUKD7KF6H3JH6DLWT3KM2PQC72NRYW4YPN7NL445VBER4QR6SA, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.581958349 (418h48m52.453041384s) [noise=true , address=89.149.201.147:4016 , piece_id=MFNURTZ2HJFACIAHZUO7YZJXRJZB26AVF24JLU3I7IYKDPZ2JJIQ, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.582064401 (418h48m52.453147436s) [noise=true , address=89.149.201.147:4009 , piece_id=WGVNGAHNZIVLBR665VB7GO77EZ7SSJOC5P7ML2UAK6MKV4BAF27A, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.582336870 (418h48m52.453419905s) [noise=true , address=81.171.14.70:4025 , piece_id=PFRRLSNJYBUCYSOVOE3VHDWAKTEPMPQRTE4U2VTPBT7KGYVNQHIQ, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.582391083 (418h48m52.453474118s) [noise=true , address=81.171.14.70:4029 , piece_id=JKKAJ3PXL7WDZ7XNKM752C6NA4B4FUN2AZJCGAZPDVNAGKTAOK7Q, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.582715682 (418h48m52.453798717s) [noise=true , address=89.149.201.147:4002 , piece_id=QCDLIQPY7COKADVRRRUN7UBEIV2AARIPA64LMFCCYPFNA54TA7VQ, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.582847693 (418h48m52.453930738s) [noise=true , address=185.167.204.205:4033 , piece_id=42CCWQIIXKFWWCLUSLGSFQHZMA2UPGZJMOSWD7Z562DV6ABPDVRQ, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.583274976 (418h48m52.454358022s) [noise=true , address=185.167.204.206:4066 , piece_id=WAVNL7LBCOQEW3CFA3DWRITRTBGCWYVASH43KC4RFHE5KFQUQ45A, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.583662655 (418h48m52.45474569s) [noise=true , address=185.167.204.206:4070 , piece_id=GA3GZFHS36RXW4PWKSHZ2QOAXQJVNIWVMED3XAAGG5GQPA27ST7A, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:53.405090439 (418h48m53.276173484s) [upload=367700]: Next returned nil segment Dec 23 02:17:53.405105007 (418h48m53.276188042s) [upload=367700]: Waiting for error group managing segments... Dec 23 02:17:53.798794305 (418h48m53.66987735s) [noise=true , address=185.167.204.205:4025 , piece_id=XY5XNAAQZFPKWJP7V5LWBBH5ZQ7XFUTQ657QEF5KXFTP2EKCSIBA, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.799500561 (418h48m53.670583596s) [noise=true , address=185.167.204.205:4016 , piece_id=523RGOAK7OI6VYQY5X4XT5Y6F5C7NMB2WDXPN74Q27P3BCH2Q37A, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.799686735 (418h48m53.67076977s) [noise=true , address=185.167.204.205:4036 , piece_id=IQ6I5X4HC2ANQSM72N7X4OJWVTUMOWH5YPSSRHUFTGYSXW5ZK4WQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.800348164 (418h48m53.671431209s) [noise=true , address=185.167.204.205:4033 , piece_id=42CCWQIIXKFWWCLUSLGSFQHZMA2UPGZJMOSWD7Z562DV6ABPDVRQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.802008737 (418h48m53.673091782s) [noise=true , address=185.167.204.206:4057 , piece_id=UNUF4NZ5CL74EWLPZLVX44MAYYUQFIAZ54N7FGBLCQHEKLGX6RSQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.802484733 (418h48m53.673567778s) [noise=true , address=185.167.204.206:4040 , piece_id=Y6HVWGUESE2BSEIKEMDSLNHWDTOEKGBR76NIFRRQU6KFTVHD4Q7Q, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.803136594 (418h48m53.674219639s) [noise=true , address=185.167.204.206:4070 , piece_id=GA3GZFHS36RXW4PWKSHZ2QOAXQJVNIWVMED3XAAGG5GQPA27ST7A, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.803750904 (418h48m53.674833939s) [noise=true , address=185.167.204.206:4053 , piece_id=GG5TA7D4KK2IGSKRO4T3APKSZ43IWLND3L54EXYD2LRFIKGI5MRQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.804347520 (418h48m53.675430565s) [noise=true , address=185.167.204.206:4058 , piece_id=4CZUKD7KF6H3JH6DLWT3KM2PQC72NRYW4YPN7NL445VBER4QR6SA, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.805167522 (418h48m53.676250567s) [noise=true , address=185.167.204.206:4054 , piece_id=3QWOKGEC5JODMZCL5R5ULU7EUAKGI2W5QWONT5H2I7EOBH7PVODA, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.805898293 (418h48m53.676981328s) [noise=true , address=185.167.204.206:4066 , piece_id=WAVNL7LBCOQEW3CFA3DWRITRTBGCWYVASH43KC4RFHE5KFQUQ45A, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.808200908 (418h48m53.679283943s) [noise=true , address=185.167.204.206:4064 , piece_id=4K4KD2ZMFXUZT6W3OV7E6BC5UGQHSVZXR67B2QPXIPKGAM6CTVSA, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.808393705 (418h48m53.67947675s) [noise=true , address=185.167.204.205:4004 , piece_id=G5Q7QERF2O3WXIDK5ZAFGSZIN6VXJOFYHAPEC647NIXIJQCSNLKA, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.808844694 (418h48m53.679927739s) [noise=true , address=185.167.204.205:4028 , piece_id=DINID3XD5734Z3RWWS3WPIFLGRUXZMZUNXC2BC3NMX7LWTSDVN3Q, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.810924914 (418h48m53.682007959s) [noise=true , address=185.167.204.206:4047 , piece_id=JSWKSSXVNEFRZPWP56EHG26DRYA7V4K2LPEBCKUOBUG2WET5ZQ7A, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.813330996 (418h48m53.684414031s) [noise=true , address=185.167.204.206:4069 , piece_id=OLTP357X7TH2ABIRWBMAI445KR5SGJVY77PPIQFLRD2KFCBOML2Q, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.813884790 (418h48m53.684967826s) [noise=true , address=185.167.204.205:4001 , piece_id=LMIOF4TXW2JJBYGBNU36M46PXZUNKBKFK56XVNMWRQVR7LSZ3QCQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.815627059 (418h48m53.686710094s) [noise=true , address=185.167.204.206:4067 , piece_id=D24D7H4IJKMKHRJUCQ2ZAYDIVRUYHK2YHEYLVCIL3VA4P5FQJB5A, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:54.448896803 (418h48m54.319979838s) [noise=true , address=81.171.14.70:4029 , piece_id=JKKAJ3PXL7WDZ7XNKM752C6NA4B4FUN2AZJCGAZPDVNAGKTAOK7Q, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:54.835287845 (418h48m54.70637088s) [noise=true , address=81.171.14.70:4025 , piece_id=PFRRLSNJYBUCYSOVOE3VHDWAKTEPMPQRTE4U2VTPBT7KGYVNQHIQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:54.946890399 (418h48m54.817973434s) [noise=true , address=81.171.14.70:4040 , piece_id=6ZIPFOQCGRNK3TDTODGI34TL2WM65BQSRLXXAFX3XQ3VCPAVT2YA, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:54.960812961 (418h48m54.831895996s) [noise=true , address=81.171.14.70:4034 , piece_id=2EA6SNLDV3ZVTNQ5UXEXIC2SJTPH3D6VCVZJI2IAGS22U2J6Y6WQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:55.069208920 (418h48m54.940291965s) [noise=true , address=89.149.201.147:4009 , piece_id=WGVNGAHNZIVLBR665VB7GO77EZ7SSJOC5P7ML2UAK6MKV4BAF27A, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:55.073463732 (418h48m54.944546768s) [noise=true , address=89.149.201.147:4002 , piece_id=QCDLIQPY7COKADVRRRUN7UBEIV2AARIPA64LMFCCYPFNA54TA7VQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:55.132812454 (418h48m55.003895489s) [noise=true , address=89.149.201.147:4010 , piece_id=LZXMVADI5CHNTPWZNXNBEFI3SMGAWQGNUIWDNPM4JMRBLZ62HO4A, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:55.185887030 (418h48m55.056970065s) [noise=true , address=89.149.201.147:4012 , piece_id=2Z64OIS2BBYH6ZCKBXHW4XQZY4FFASHDIHH3C3YVY7MGTTBI6QNA, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:55.185911748 (418h48m55.056994783s) [seg_pos={0 0}, upload=367700]: Segment reached optimal threshold of 26 pieces. Dec 23 02:17:55.186137848 (418h48m55.057220893s) [noise=true , address=89.149.201.147:4015 , piece_id=WENNEH2N2S7NV3TQILPUZLFRBV5CO6VMP52NKDIQE23NO2X7GDWQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RbD6M2MHJfxcJXogZG8nUGFmeZjwUQ1auN6KGWJMPTffdK42K): protocol: expected piece hash; context canceled; EOF Dec 23 02:17:55.186148097 (418h48m55.057231132s) [noise=true , address=89.149.201.147:4015 , piece_id=WENNEH2N2S7NV3TQILPUZLFRBV5CO6VMP52NKDIQE23NO2X7GDWQ, seg_pos={0 0}, upload=367700]: Long tail cancellation due to optimal threshold Dec 23 02:17:55.186212209 (418h48m55.057295244s) [noise=true , address=89.149.201.147:4016 , piece_id=MFNURTZ2HJFACIAHZUO7YZJXRJZB26AVF24JLU3I7IYKDPZ2JJIQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XsvrVhDGkU8BkYsBGmTtax47E43dZAYuGwD8WBMpp3n4GgDM5): protocol: expected piece hash; context canceled; EOF Dec 23 02:17:55.186228150 (418h48m55.057311185s) [noise=true , address=89.149.201.147:4016 , piece_id=MFNURTZ2HJFACIAHZUO7YZJXRJZB26AVF24JLU3I7IYKDPZ2JJIQ, seg_pos={0 0}, upload=367700]: Long tail cancellation due to optimal threshold Dec 23 02:17:55.186247767 (418h48m55.057330802s) [seg_pos={0 0}, upload=367700]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 23 02:17:55.186337368 (418h48m55.057420403s) [upload=367700]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 02:17:55.186342387 (418h48m55.057425422s) [upload=367700]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 02:17:55.430535735 (418h48m55.30161878s) [upload=367700]: Done uploading segments... Dec 23 02:17:55.430557656 (418h48m55.301640701s) [upload=367700]: Upload finished. err: <nil> Duration: 5.733949241s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 5.710075111s trace: https://jaeger.dev.storj.io/trace/70da446b2b2ff016 FAIL (2025-12-23 02:24:24 UTC) orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 46) Duration: 5.710101469s core uplink us1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 3.125250166s trace: https://jaeger.dev.storj.io/trace/20c185d15563c01c FAIL (2025-12-23 02:24:23 UTC) orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 39) Duration: 3.125282237s core uplink ap1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................................................X.........] duration: 3.475682971s trace: https://jaeger.dev.storj.io/trace/7f7fb60dfbab7793 FAIL (2025-12-23 02:24:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.47570936s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 8.584164243s trace: https://jaeger.dev.storj.io/trace/2c98a82e82cbeda7 FAIL (2025-12-23 02:24:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== KL7BLGSD3TCZNAHNK7STV2RKAZQWCHN6PROBXAPTA5ZBCPT6A, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.700896105 (418h48m56.57197914s) [noise=true , address=145.79.227.30:4796 , piece_id=IF6NPIC5DTTMJ4DZMIDD6SFN7TGDBN6RD3H7KT7A4OC7VJVCEZOA, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.701329429 (418h48m56.572412474s) [noise=true , address=145.79.227.32:4848 , piece_id=NMEVAZ6HHKE2Z4Q5G4VLNPOIURXE5TXXUMLFJLURZUI4Z62W7VMA, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.701714032 (418h48m56.572797077s) [noise=true , address=145.79.227.28:4742 , piece_id=3IMQS3J42XXGHCZFMBYPDCPCJYRY3I3EUIQ2Z2JOK4TPXQQH75OQ, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.702231197 (418h48m56.573314232s) [noise=true , address=64.49.13.137:4500 , piece_id=RFGLMWE4OHC7RL2NBPX6VRKG4YGRVCIMBHQFW2W5XAEE376UTVVA, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.702751507 (418h48m56.573834552s) [noise=true , address=64.49.13.169:4715 , piece_id=6RCBEKXRJ56IGXWT4PVSTDYRIH3UCAD22C4Z5BU33U5YIPWBEPDQ, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.703271658 (418h48m56.574354693s) [noise=true , address=45.91.116.12:4257 , piece_id=SPTNFMB3LXNY74LCASNSAZIPCOAQHO5V6SOFP27Y4IY25CHWBO2A, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.703948437 (418h48m56.575031482s) [noise=true , address=145.79.227.23:4549 , piece_id=XH7BPLOCXKVTJQDYKIRKFXVLHIXXNJFAXCFZPSSASWLOG3GM5PEQ, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.703969256 (418h48m56.575052291s) [noise=true , address=5.231.73.4:4030 , piece_id=M6FJSRGUNCEIBMNKLBK7HHFL4DA3PDD2MZ2GN2MJKAFSH3XRI7XQ, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.704457947 (418h48m56.575540982s) [noise=true , address=145.79.227.17:4984 , piece_id=DAMTMQIEBM3I2QZTWVEGISFJ5T7I3LMZ3CTXUFXO2CS775GY4F7Q, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.704834724 (418h48m56.575917769s) [noise=true , address=77.90.31.18:4815 , piece_id=XQDHXOY2ZQQHZZHMGLO5GKZOUJZFKKWGLDJRHJLWCFEX6PKNNHWA, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.706591609 (418h48m56.577674654s) [noise=true , address=77.90.31.13:4651 , piece_id=H6MCF4TXI5WFQ2BBCPXL3OX6OA4NCRBIH6IBLMPYPIJBETEN6UGA, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.707118754 (418h48m56.578201789s) [noise=true , address=145.79.227.13:4811 , piece_id=KVHW3GEE4KWLOOUJBRVOOLDA3EGPRUU7VHPH7TQXLURUCCU5J77Q, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.707431179 (418h48m56.578514224s) [noise=true , address=77.90.31.16:4740 , piece_id=KYCKJ7F7CNIXIVAZZ7CQPWAYNCMIM5RKO4IKXETP5EZNOAGKGONA, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.707910952 (418h48m56.578993997s) [noise=true , address=45.91.116.5:4330 , piece_id=EMSR7LFH5Q6R7YAJNH45TDQS2E4KX25Q335IFFUORN5AJJWS7W5Q, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.708492299 (418h48m56.579575344s) [noise=true , address=64.49.13.139:4478 , piece_id=QUBJUJHQ6AYU6L67666PA7BHCRL2IIFOJDISOISTEK4JWABGMXIQ, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.708774406 (418h48m56.579857441s) [noise=true , address=45.91.116.14:4213 , piece_id=VLD3RNN2RLOQI2AKMHNY3G57QJPDNJ3BA5JDRRJHCHNHS6YE52JA, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:18:00.137346621 (418h49m0.008429666s) [upload=367702]: Next returned nil segment Dec 23 02:18:00.137360507 (418h49m0.008443552s) [upload=367702]: Waiting for error group managing segments... Dec 23 02:18:01.750516581 (418h49m1.621599616s) [noise=true , address=64.49.13.137:4500 , piece_id=RFGLMWE4OHC7RL2NBPX6VRKG4YGRVCIMBHQFW2W5XAEE376UTVVA, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:02.125124405 (418h49m1.99620745s) [noise=true , address=77.90.31.18:4815 , piece_id=XQDHXOY2ZQQHZZHMGLO5GKZOUJZFKKWGLDJRHJLWCFEX6PKNNHWA, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:02.226869841 (418h49m2.097952876s) [noise=true , address=45.91.116.8:4060 , piece_id=TRNAVEFSFXMIN3DFMQ2DUAKKDTSZH36TCSYZXFKALUM7TCKM3Y6A, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:05.457222504 (418h49m5.328305539s) [noise=true , address=77.90.31.13:4651 , piece_id=H6MCF4TXI5WFQ2BBCPXL3OX6OA4NCRBIH6IBLMPYPIJBETEN6UGA, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:06.079641082 (418h49m5.950724128s) [noise=true , address=45.91.116.12:4257 , piece_id=SPTNFMB3LXNY74LCASNSAZIPCOAQHO5V6SOFP27Y4IY25CHWBO2A, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:09.265548070 (418h49m9.136631105s) [noise=true , address=64.49.13.139:4478 , piece_id=QUBJUJHQ6AYU6L67666PA7BHCRL2IIFOJDISOISTEK4JWABGMXIQ, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:10.880784395 (418h49m10.751867441s) [noise=true , address=45.91.116.14:4213 , piece_id=VLD3RNN2RLOQI2AKMHNY3G57QJPDNJ3BA5JDRRJHCHNHS6YE52JA, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:12.686956874 (418h49m12.558039929s) [noise=true , address=145.79.227.28:4742 , piece_id=3IMQS3J42XXGHCZFMBYPDCPCJYRY3I3EUIQ2Z2JOK4TPXQQH75OQ, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:12.832920510 (418h49m12.704003545s) [noise=true , address=45.91.116.5:4330 , piece_id=EMSR7LFH5Q6R7YAJNH45TDQS2E4KX25Q335IFFUORN5AJJWS7W5Q, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:22.466644063 (418h49m22.337727098s) [noise=true , address=77.90.31.16:4740 , piece_id=KYCKJ7F7CNIXIVAZZ7CQPWAYNCMIM5RKO4IKXETP5EZNOAGKGONA, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:34.649740548 (418h49m34.520823583s) [noise=true , address=64.49.13.169:4715 , piece_id=6RCBEKXRJ56IGXWT4PVSTDYRIH3UCAD22C4Z5BU33U5YIPWBEPDQ, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:38.250548484 (418h49m38.121631529s) [noise=true , address=145.79.227.17:4984 , piece_id=DAMTMQIEBM3I2QZTWVEGISFJ5T7I3LMZ3CTXUFXO2CS775GY4F7Q, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:43.329219400 (418h49m43.200302445s) [noise=true , address=145.79.227.23:4549 , piece_id=XH7BPLOCXKVTJQDYKIRKFXVLHIXXNJFAXCFZPSSASWLOG3GM5PEQ, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:43.400788824 (418h49m43.271871859s) [noise=true , address=145.79.227.32:4848 , piece_id=NMEVAZ6HHKE2Z4Q5G4VLNPOIURXE5TXXUMLFJLURZUI4Z62W7VMA, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:43.400819082 (418h49m43.271902117s) [seg_pos={0 0}, upload=367702]: Segment reached optimal threshold of 14 pieces. Dec 23 02:18:43.401090247 (418h49m43.272173302s) [noise=true , address=5.231.73.4:4030 , piece_id=M6FJSRGUNCEIBMNKLBK7HHFL4DA3PDD2MZ2GN2MJKAFSH3XRI7XQ, seg_pos={0 0}, upload=367702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rWJM3vC8XrUAfVaoqVZm1qzqcG8AqxLzncExPHEak5oo3QuPA): protocol: expected piece hash; context canceled; EOF Dec 23 02:18:43.401100788 (418h49m43.272183823s) [noise=true , address=5.231.73.4:4030 , piece_id=M6FJSRGUNCEIBMNKLBK7HHFL4DA3PDD2MZ2GN2MJKAFSH3XRI7XQ, seg_pos={0 0}, upload=367702]: Long tail cancellation due to optimal threshold Dec 23 02:18:43.401281792 (418h49m43.272364837s) [noise=true , address=77.90.31.14:4667 , piece_id=5ZXKL7BLGSD3TCZNAHNK7STV2RKAZQWCHN6PROBXAPTA5ZBCPT6A, seg_pos={0 0}, upload=367702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S): protocol: expected piece hash; context canceled; EOF Dec 23 02:18:43.401290889 (418h49m43.272373925s) [noise=true , address=77.90.31.14:4667 , piece_id=5ZXKL7BLGSD3TCZNAHNK7STV2RKAZQWCHN6PROBXAPTA5ZBCPT6A, seg_pos={0 0}, upload=367702]: Long tail cancellation due to optimal threshold Dec 23 02:18:43.401412151 (418h49m43.272495186s) [noise=true , address=145.79.227.13:4811 , piece_id=KVHW3GEE4KWLOOUJBRVOOLDA3EGPRUU7VHPH7TQXLURUCCU5J77Q, seg_pos={0 0}, upload=367702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TxjP4CnxyU4t2DTNBjBMBxnizcHCD42fMrB5Da6Qiqg1Eg17V): protocol: context canceled; context canceled Dec 23 02:18:43.401418663 (418h49m43.272501698s) [noise=true , address=145.79.227.13:4811 , piece_id=KVHW3GEE4KWLOOUJBRVOOLDA3EGPRUU7VHPH7TQXLURUCCU5J77Q, seg_pos={0 0}, upload=367702]: Long tail cancellation due to optimal threshold Dec 23 02:18:43.401630526 (418h49m43.272713561s) [noise=true , address=145.79.227.30:4796 , piece_id=IF6NPIC5DTTMJ4DZMIDD6SFN7TGDBN6RD3H7KT7A4OC7VJVCEZOA, seg_pos={0 0}, upload=367702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: read tcp 188.166.225.170:44484->145.79.227.30:4796: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Dec 23 02:18:43.401638241 (418h49m43.272721276s) [noise=true , address=145.79.227.30:4796 , piece_id=IF6NPIC5DTTMJ4DZMIDD6SFN7TGDBN6RD3H7KT7A4OC7VJVCEZOA, seg_pos={0 0}, upload=367702]: Long tail cancellation due to optimal threshold Dec 23 02:18:43.401657227 (418h49m43.272740273s) [seg_pos={0 0}, upload=367702]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 23 02:18:43.401864953 (418h49m43.272947988s) [upload=367702]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 02:18:43.401870614 (418h49m43.272953649s) [upload=367702]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 02:18:43.703068278 (418h49m43.574151323s) [upload=367702]: Done uploading segments... Dec 23 02:18:43.703144954 (418h49m43.574227999s) [upload=367702]: Upload finished. err: <nil> Duration: 8.584192056s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 6.375277672s trace: https://jaeger.dev.storj.io/trace/61a0842c4fdd78ca FAIL (2025-12-23 02:24:14 UTC) orbiter warning: context deadline exceeded Duration: 6.375322557s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 5.759429015s trace: https://jaeger.dev.storj.io/trace/5b87e1ad666dcc24 FAIL (2025-12-23 02:24:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 2) ======LOGS====== 7Y7UWG6NFFSQ, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:05.480444685 (418h49m5.35152773s) [noise=true , address=45.91.116.4:4937 , piece_id=IIV6VFBKACIVI3M6A6L2WSDU54D3MRGZ36LG6E3E4SXKQVRX74LQ, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:05.480943264 (418h49m5.352026309s) [noise=true , address=145.79.227.9:4366 , piece_id=73FK36X6U4CYXVKSGCTLK4YCL2QB3CQKUST2IY4OQGG4JWMLTV6A, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:05.482473688 (418h49m5.353556733s) [noise=true , address=145.79.227.41:4086 , piece_id=KXPKPLKHLFQ5GU7Z7LM3NXGWGXSGISKBOHAWPJ7QGG4ESA64YYWA, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:05.482598135 (418h49m5.35368118s) [noise=true , address=145.79.227.43:4159 , piece_id=WUJFSGFHRK4PHRLS4H3MNT5TVXYR6MSTUQXRN6QBXI47WRPKKN4A, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:05.485571146 (418h49m5.356654182s) [noise=true , address=77.90.31.12:4589 , piece_id=RD5KILZQDADOR2UPRRAO3XM7YWI7AQ57HBR37VC44DNZPOS555SA, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:05.488437175 (418h49m5.35952021s) [noise=true , address=145.79.227.35:4481 , piece_id=EVLROBIPMCRCZYYUSUKKCHCMTJH6G2H56QJR6ZPIF7R7HVVCP6CA, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:05.491243618 (418h49m5.362326653s) [noise=true , address=145.79.227.39:4015 , piece_id=UAGG4FHO2SDQKDF6T4QIGVN7IMNANYWTD3SHYRMBVGCPS3Z4VIYA, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:05.492801755 (418h49m5.36388479s) [noise=true , address=77.90.31.16:4735 , piece_id=XEZKWPQDBCT7YRGYTH4TFR4BDCP34BPJB6OVFTYGMQ2NQI34NVZA, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:05.495283631 (418h49m5.366366666s) [noise=true , address=145.79.227.37:4565 , piece_id=PAKR2BG4S3KYFXOLS7BL5AANJMIHMN477KGMJJ4XIHMTF7UOTCHA, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:05.495702669 (418h49m5.366785704s) [noise=true , address=145.79.227.42:4121 , piece_id=ICOCUSNLYIVKOJTONNBAEPWGSRTB44H2KAHESVBGGBEDOEHROYGA, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:05.499769763 (418h49m5.370852808s) [noise=true , address=145.79.227.51:4411 , piece_id=2JQUVX7JG3GYCRYJMYE7AQNIX7XG7XEYRXAC6JTTYRM343CVP3IA, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:08.589264332 (418h49m8.460347367s) [upload=367705]: Next returned nil segment Dec 23 02:18:08.589286223 (418h49m8.460369258s) [upload=367705]: Waiting for error group managing segments... Dec 23 02:18:12.569357328 (418h49m12.440440364s) [noise=true , address=45.91.116.4:4937 , piece_id=IIV6VFBKACIVI3M6A6L2WSDU54D3MRGZ36LG6E3E4SXKQVRX74LQ, seg_pos={0 0}, upload=367705]: Done uploading piece. err: <nil> Dec 23 02:18:15.106396805 (418h49m14.977479851s) [noise=true , address=77.90.31.12:4589 , piece_id=RD5KILZQDADOR2UPRRAO3XM7YWI7AQ57HBR37VC44DNZPOS555SA, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 23 02:18:17.598569766 (418h49m17.469652801s) [noise=true , address=77.90.31.16:4735 , piece_id=XEZKWPQDBCT7YRGYTH4TFR4BDCP34BPJB6OVFTYGMQ2NQI34NVZA, seg_pos={0 0}, upload=367705]: Done uploading piece. err: <nil> Dec 23 02:18:33.434728164 (418h49m33.305811199s) [noise=true , address=145.79.227.42:4121 , piece_id=ICOCUSNLYIVKOJTONNBAEPWGSRTB44H2KAHESVBGGBEDOEHROYGA, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: read tcp 188.166.225.170:50058->145.79.227.42:4121: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.434850206 (418h49m33.305933241s) [noise=true , address=145.79.227.15:4874 , piece_id=PWRV5UFUDENFBPJFK3KYNXMYLQZBHFG6KBJFZXA6P32NT7ZNOWDA, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: read tcp 188.166.225.170:50240->145.79.227.15:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.434955458 (418h49m33.306038493s) [noise=true , address=145.79.227.50:4373 , piece_id=PZRIRRQQETAX56EVN7LY3Z5DUPUYUE7N7H2ZTYWK7Y7UWG6NFFSQ, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, address:145.79.227.50:4373): protocol: read tcp 188.166.225.170:54252->145.79.227.50:4373: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.435015211 (418h49m33.306098246s) [noise=true , address=145.79.227.9:4366 , piece_id=73FK36X6U4CYXVKSGCTLK4YCL2QB3CQKUST2IY4OQGG4JWMLTV6A, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: rpc: dial tcp 145.79.227.9:4366: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.435058384 (418h49m33.306141429s) [noise=true , address=145.79.227.43:4159 , piece_id=WUJFSGFHRK4PHRLS4H3MNT5TVXYR6MSTUQXRN6QBXI47WRPKKN4A, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA, address:145.79.227.43:4159): protocol: read tcp 188.166.225.170:49750->145.79.227.43:4159: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.435156329 (418h49m33.306239374s) [noise=true , address=145.79.227.47:4267 , piece_id=CFW6HHUK6IP232DVR64FPSBPULEIVPVGCS2HR2523RAIP5N77POQ, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8, address:145.79.227.47:4267): protocol: read tcp 188.166.225.170:33996->145.79.227.47:4267: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.435495826 (418h49m33.306578871s) [noise=true , address=77.90.31.21:4877 , piece_id=J3XPMHNIGDTRPYW2ITOOW4FRJ4SUKMCCGRP22WKD55A7OA7EOV3A, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 188.166.225.170:35802->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.435680898 (418h49m33.306763943s) [noise=true , address=145.79.227.41:4086 , piece_id=KXPKPLKHLFQ5GU7Z7LM3NXGWGXSGISKBOHAWPJ7QGG4ESA64YYWA, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, address:145.79.227.41:4086): protocol: read tcp 188.166.225.170:33906->145.79.227.41:4086: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.435701508 (418h49m33.306784543s) [noise=true , address=145.79.227.39:4015 , piece_id=UAGG4FHO2SDQKDF6T4QIGVN7IMNANYWTD3SHYRMBVGCPS3Z4VIYA, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, address:145.79.227.39:4015): protocol: read tcp 188.166.225.170:45508->145.79.227.39:4015: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.435883414 (418h49m33.306966449s) [noise=true , address=145.79.227.37:4565 , piece_id=PAKR2BG4S3KYFXOLS7BL5AANJMIHMN477KGMJJ4XIHMTF7UOTCHA, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:1nvvNczHtQrSEJuJbErnjiAWa4BRDzBrQmBNbB7RrrqN1JWNHT, address:145.79.227.37:4565): protocol: read tcp 188.166.225.170:43228->145.79.227.37:4565: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.435913952 (418h49m33.306996987s) [noise=true , address=145.79.227.38:4978 , piece_id=5VMIQGZ32DI7IE6Z6HMVCJUXIMBFVN6BWONEBE452PG5WDWZXDZQ, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: read tcp 188.166.225.170:57482->145.79.227.38:4978: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.436073857 (418h49m33.307156892s) [noise=true , address=145.79.227.35:4481 , piece_id=EVLROBIPMCRCZYYUSUKKCHCMTJH6G2H56QJR6ZPIF7R7HVVCP6CA, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp, address:145.79.227.35:4481): protocol: read tcp 188.166.225.170:34156->145.79.227.35:4481: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.436167987 (418h49m33.307251022s) [noise=true , address=145.79.227.51:4411 , piece_id=2JQUVX7JG3GYCRYJMYE7AQNIX7XG7XEYRXAC6JTTYRM343CVP3IA, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR, address:145.79.227.51:4411): protocol: read tcp 188.166.225.170:54092->145.79.227.51:4411: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.436191531 (418h49m33.307274566s) [seg_pos={0 0}, upload=367705]: Done waiting for segment. successful: 2 optimal: 13 errs: <nil> Dec 23 02:18:33.726571837 (418h49m33.597654882s) [upload=367705]: Done uploading segments... Dec 23 02:18:33.726612455 (418h49m33.59769549s) [upload=367705]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 2) Duration: 5.759458893s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 5.123470394s trace: https://jaeger.dev.storj.io/trace/14f27c56f52ae4e9 FAIL (2025-12-23 02:24:06 UTC) orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 45) Duration: 5.123506231s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 5.916716762s trace: https://jaeger.dev.storj.io/trace/463d97d26edafa8a FAIL (2025-12-23 02:24:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=367700]: Uploading piece... Dec 23 02:17:52.581508683 (418h48m52.452591718s) [noise=true , address=185.167.204.206:4040 , piece_id=Y6HVWGUESE2BSEIKEMDSLNHWDTOEKGBR76NIFRRQU6KFTVHD4Q7Q, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.581711800 (418h48m52.452794835s) [noise=true , address=185.167.204.206:4058 , piece_id=4CZUKD7KF6H3JH6DLWT3KM2PQC72NRYW4YPN7NL445VBER4QR6SA, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.581958349 (418h48m52.453041384s) [noise=true , address=89.149.201.147:4016 , piece_id=MFNURTZ2HJFACIAHZUO7YZJXRJZB26AVF24JLU3I7IYKDPZ2JJIQ, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.582064401 (418h48m52.453147436s) [noise=true , address=89.149.201.147:4009 , piece_id=WGVNGAHNZIVLBR665VB7GO77EZ7SSJOC5P7ML2UAK6MKV4BAF27A, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.582336870 (418h48m52.453419905s) [noise=true , address=81.171.14.70:4025 , piece_id=PFRRLSNJYBUCYSOVOE3VHDWAKTEPMPQRTE4U2VTPBT7KGYVNQHIQ, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.582391083 (418h48m52.453474118s) [noise=true , address=81.171.14.70:4029 , piece_id=JKKAJ3PXL7WDZ7XNKM752C6NA4B4FUN2AZJCGAZPDVNAGKTAOK7Q, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.582715682 (418h48m52.453798717s) [noise=true , address=89.149.201.147:4002 , piece_id=QCDLIQPY7COKADVRRRUN7UBEIV2AARIPA64LMFCCYPFNA54TA7VQ, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.582847693 (418h48m52.453930738s) [noise=true , address=185.167.204.205:4033 , piece_id=42CCWQIIXKFWWCLUSLGSFQHZMA2UPGZJMOSWD7Z562DV6ABPDVRQ, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.583274976 (418h48m52.454358022s) [noise=true , address=185.167.204.206:4066 , piece_id=WAVNL7LBCOQEW3CFA3DWRITRTBGCWYVASH43KC4RFHE5KFQUQ45A, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.583662655 (418h48m52.45474569s) [noise=true , address=185.167.204.206:4070 , piece_id=GA3GZFHS36RXW4PWKSHZ2QOAXQJVNIWVMED3XAAGG5GQPA27ST7A, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:53.405090439 (418h48m53.276173484s) [upload=367700]: Next returned nil segment Dec 23 02:17:53.405105007 (418h48m53.276188042s) [upload=367700]: Waiting for error group managing segments... Dec 23 02:17:53.798794305 (418h48m53.66987735s) [noise=true , address=185.167.204.205:4025 , piece_id=XY5XNAAQZFPKWJP7V5LWBBH5ZQ7XFUTQ657QEF5KXFTP2EKCSIBA, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.799500561 (418h48m53.670583596s) [noise=true , address=185.167.204.205:4016 , piece_id=523RGOAK7OI6VYQY5X4XT5Y6F5C7NMB2WDXPN74Q27P3BCH2Q37A, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.799686735 (418h48m53.67076977s) [noise=true , address=185.167.204.205:4036 , piece_id=IQ6I5X4HC2ANQSM72N7X4OJWVTUMOWH5YPSSRHUFTGYSXW5ZK4WQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.800348164 (418h48m53.671431209s) [noise=true , address=185.167.204.205:4033 , piece_id=42CCWQIIXKFWWCLUSLGSFQHZMA2UPGZJMOSWD7Z562DV6ABPDVRQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.802008737 (418h48m53.673091782s) [noise=true , address=185.167.204.206:4057 , piece_id=UNUF4NZ5CL74EWLPZLVX44MAYYUQFIAZ54N7FGBLCQHEKLGX6RSQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.802484733 (418h48m53.673567778s) [noise=true , address=185.167.204.206:4040 , piece_id=Y6HVWGUESE2BSEIKEMDSLNHWDTOEKGBR76NIFRRQU6KFTVHD4Q7Q, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.803136594 (418h48m53.674219639s) [noise=true , address=185.167.204.206:4070 , piece_id=GA3GZFHS36RXW4PWKSHZ2QOAXQJVNIWVMED3XAAGG5GQPA27ST7A, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.803750904 (418h48m53.674833939s) [noise=true , address=185.167.204.206:4053 , piece_id=GG5TA7D4KK2IGSKRO4T3APKSZ43IWLND3L54EXYD2LRFIKGI5MRQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.804347520 (418h48m53.675430565s) [noise=true , address=185.167.204.206:4058 , piece_id=4CZUKD7KF6H3JH6DLWT3KM2PQC72NRYW4YPN7NL445VBER4QR6SA, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.805167522 (418h48m53.676250567s) [noise=true , address=185.167.204.206:4054 , piece_id=3QWOKGEC5JODMZCL5R5ULU7EUAKGI2W5QWONT5H2I7EOBH7PVODA, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.805898293 (418h48m53.676981328s) [noise=true , address=185.167.204.206:4066 , piece_id=WAVNL7LBCOQEW3CFA3DWRITRTBGCWYVASH43KC4RFHE5KFQUQ45A, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.808200908 (418h48m53.679283943s) [noise=true , address=185.167.204.206:4064 , piece_id=4K4KD2ZMFXUZT6W3OV7E6BC5UGQHSVZXR67B2QPXIPKGAM6CTVSA, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.808393705 (418h48m53.67947675s) [noise=true , address=185.167.204.205:4004 , piece_id=G5Q7QERF2O3WXIDK5ZAFGSZIN6VXJOFYHAPEC647NIXIJQCSNLKA, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.808844694 (418h48m53.679927739s) [noise=true , address=185.167.204.205:4028 , piece_id=DINID3XD5734Z3RWWS3WPIFLGRUXZMZUNXC2BC3NMX7LWTSDVN3Q, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.810924914 (418h48m53.682007959s) [noise=true , address=185.167.204.206:4047 , piece_id=JSWKSSXVNEFRZPWP56EHG26DRYA7V4K2LPEBCKUOBUG2WET5ZQ7A, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.813330996 (418h48m53.684414031s) [noise=true , address=185.167.204.206:4069 , piece_id=OLTP357X7TH2ABIRWBMAI445KR5SGJVY77PPIQFLRD2KFCBOML2Q, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.813884790 (418h48m53.684967826s) [noise=true , address=185.167.204.205:4001 , piece_id=LMIOF4TXW2JJBYGBNU36M46PXZUNKBKFK56XVNMWRQVR7LSZ3QCQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.815627059 (418h48m53.686710094s) [noise=true , address=185.167.204.206:4067 , piece_id=D24D7H4IJKMKHRJUCQ2ZAYDIVRUYHK2YHEYLVCIL3VA4P5FQJB5A, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:54.448896803 (418h48m54.319979838s) [noise=true , address=81.171.14.70:4029 , piece_id=JKKAJ3PXL7WDZ7XNKM752C6NA4B4FUN2AZJCGAZPDVNAGKTAOK7Q, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:54.835287845 (418h48m54.70637088s) [noise=true , address=81.171.14.70:4025 , piece_id=PFRRLSNJYBUCYSOVOE3VHDWAKTEPMPQRTE4U2VTPBT7KGYVNQHIQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:54.946890399 (418h48m54.817973434s) [noise=true , address=81.171.14.70:4040 , piece_id=6ZIPFOQCGRNK3TDTODGI34TL2WM65BQSRLXXAFX3XQ3VCPAVT2YA, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:54.960812961 (418h48m54.831895996s) [noise=true , address=81.171.14.70:4034 , piece_id=2EA6SNLDV3ZVTNQ5UXEXIC2SJTPH3D6VCVZJI2IAGS22U2J6Y6WQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:55.069208920 (418h48m54.940291965s) [noise=true , address=89.149.201.147:4009 , piece_id=WGVNGAHNZIVLBR665VB7GO77EZ7SSJOC5P7ML2UAK6MKV4BAF27A, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:55.073463732 (418h48m54.944546768s) [noise=true , address=89.149.201.147:4002 , piece_id=QCDLIQPY7COKADVRRRUN7UBEIV2AARIPA64LMFCCYPFNA54TA7VQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:55.132812454 (418h48m55.003895489s) [noise=true , address=89.149.201.147:4010 , piece_id=LZXMVADI5CHNTPWZNXNBEFI3SMGAWQGNUIWDNPM4JMRBLZ62HO4A, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:55.185887030 (418h48m55.056970065s) [noise=true , address=89.149.201.147:4012 , piece_id=2Z64OIS2BBYH6ZCKBXHW4XQZY4FFASHDIHH3C3YVY7MGTTBI6QNA, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:55.185911748 (418h48m55.056994783s) [seg_pos={0 0}, upload=367700]: Segment reached optimal threshold of 26 pieces. Dec 23 02:17:55.186137848 (418h48m55.057220893s) [noise=true , address=89.149.201.147:4015 , piece_id=WENNEH2N2S7NV3TQILPUZLFRBV5CO6VMP52NKDIQE23NO2X7GDWQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RbD6M2MHJfxcJXogZG8nUGFmeZjwUQ1auN6KGWJMPTffdK42K): protocol: expected piece hash; context canceled; EOF Dec 23 02:17:55.186148097 (418h48m55.057231132s) [noise=true , address=89.149.201.147:4015 , piece_id=WENNEH2N2S7NV3TQILPUZLFRBV5CO6VMP52NKDIQE23NO2X7GDWQ, seg_pos={0 0}, upload=367700]: Long tail cancellation due to optimal threshold Dec 23 02:17:55.186212209 (418h48m55.057295244s) [noise=true , address=89.149.201.147:4016 , piece_id=MFNURTZ2HJFACIAHZUO7YZJXRJZB26AVF24JLU3I7IYKDPZ2JJIQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XsvrVhDGkU8BkYsBGmTtax47E43dZAYuGwD8WBMpp3n4GgDM5): protocol: expected piece hash; context canceled; EOF Dec 23 02:17:55.186228150 (418h48m55.057311185s) [noise=true , address=89.149.201.147:4016 , piece_id=MFNURTZ2HJFACIAHZUO7YZJXRJZB26AVF24JLU3I7IYKDPZ2JJIQ, seg_pos={0 0}, upload=367700]: Long tail cancellation due to optimal threshold Dec 23 02:17:55.186247767 (418h48m55.057330802s) [seg_pos={0 0}, upload=367700]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 23 02:17:55.186337368 (418h48m55.057420403s) [upload=367700]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 02:17:55.186342387 (418h48m55.057425422s) [upload=367700]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 02:17:55.430535735 (418h48m55.30161878s) [upload=367700]: Done uploading segments... Dec 23 02:17:55.430557656 (418h48m55.301640701s) [upload=367700]: Upload finished. err: <nil> Duration: 5.916742682s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 3.595618577s trace: https://jaeger.dev.storj.io/trace/20c185d15563c01c FAIL (2025-12-23 02:23:56 UTC) orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 39) Duration: 3.595652171s core uplink ap1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 5.233804002s trace: https://jaeger.dev.storj.io/trace/14f27c56f52ae4e9 FAIL (2025-12-23 02:23:45 UTC) orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 45) Duration: 5.2338296s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................................................X........] duration: 3.374795715s trace: https://jaeger.dev.storj.io/trace/7f7fb60dfbab7793 FAIL (2025-12-23 02:23:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.374848646s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 5.940484072s trace: https://jaeger.dev.storj.io/trace/70da446b2b2ff016 FAIL (2025-12-23 02:23:21 UTC) orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 46) Duration: 5.94050323s core uplink us1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 3.345211629s trace: https://jaeger.dev.storj.io/trace/20c185d15563c01c FAIL (2025-12-23 02:23:17 UTC) orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 39) Duration: 3.345304886s core uplink ap1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 5.980800092s trace: https://jaeger.dev.storj.io/trace/61a0842c4fdd78ca FAIL (2025-12-23 02:23:17 UTC) orbiter warning: context deadline exceeded Duration: 5.980826802s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 8.900472691s trace: https://jaeger.dev.storj.io/trace/5b87e1ad666dcc24 FAIL (2025-12-23 02:23:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 2) ======LOGS====== 7Y7UWG6NFFSQ, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:05.480444685 (418h49m5.35152773s) [noise=true , address=45.91.116.4:4937 , piece_id=IIV6VFBKACIVI3M6A6L2WSDU54D3MRGZ36LG6E3E4SXKQVRX74LQ, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:05.480943264 (418h49m5.352026309s) [noise=true , address=145.79.227.9:4366 , piece_id=73FK36X6U4CYXVKSGCTLK4YCL2QB3CQKUST2IY4OQGG4JWMLTV6A, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:05.482473688 (418h49m5.353556733s) [noise=true , address=145.79.227.41:4086 , piece_id=KXPKPLKHLFQ5GU7Z7LM3NXGWGXSGISKBOHAWPJ7QGG4ESA64YYWA, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:05.482598135 (418h49m5.35368118s) [noise=true , address=145.79.227.43:4159 , piece_id=WUJFSGFHRK4PHRLS4H3MNT5TVXYR6MSTUQXRN6QBXI47WRPKKN4A, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:05.485571146 (418h49m5.356654182s) [noise=true , address=77.90.31.12:4589 , piece_id=RD5KILZQDADOR2UPRRAO3XM7YWI7AQ57HBR37VC44DNZPOS555SA, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:05.488437175 (418h49m5.35952021s) [noise=true , address=145.79.227.35:4481 , piece_id=EVLROBIPMCRCZYYUSUKKCHCMTJH6G2H56QJR6ZPIF7R7HVVCP6CA, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:05.491243618 (418h49m5.362326653s) [noise=true , address=145.79.227.39:4015 , piece_id=UAGG4FHO2SDQKDF6T4QIGVN7IMNANYWTD3SHYRMBVGCPS3Z4VIYA, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:05.492801755 (418h49m5.36388479s) [noise=true , address=77.90.31.16:4735 , piece_id=XEZKWPQDBCT7YRGYTH4TFR4BDCP34BPJB6OVFTYGMQ2NQI34NVZA, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:05.495283631 (418h49m5.366366666s) [noise=true , address=145.79.227.37:4565 , piece_id=PAKR2BG4S3KYFXOLS7BL5AANJMIHMN477KGMJJ4XIHMTF7UOTCHA, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:05.495702669 (418h49m5.366785704s) [noise=true , address=145.79.227.42:4121 , piece_id=ICOCUSNLYIVKOJTONNBAEPWGSRTB44H2KAHESVBGGBEDOEHROYGA, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:05.499769763 (418h49m5.370852808s) [noise=true , address=145.79.227.51:4411 , piece_id=2JQUVX7JG3GYCRYJMYE7AQNIX7XG7XEYRXAC6JTTYRM343CVP3IA, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:08.589264332 (418h49m8.460347367s) [upload=367705]: Next returned nil segment Dec 23 02:18:08.589286223 (418h49m8.460369258s) [upload=367705]: Waiting for error group managing segments... Dec 23 02:18:12.569357328 (418h49m12.440440364s) [noise=true , address=45.91.116.4:4937 , piece_id=IIV6VFBKACIVI3M6A6L2WSDU54D3MRGZ36LG6E3E4SXKQVRX74LQ, seg_pos={0 0}, upload=367705]: Done uploading piece. err: <nil> Dec 23 02:18:15.106396805 (418h49m14.977479851s) [noise=true , address=77.90.31.12:4589 , piece_id=RD5KILZQDADOR2UPRRAO3XM7YWI7AQ57HBR37VC44DNZPOS555SA, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 23 02:18:17.598569766 (418h49m17.469652801s) [noise=true , address=77.90.31.16:4735 , piece_id=XEZKWPQDBCT7YRGYTH4TFR4BDCP34BPJB6OVFTYGMQ2NQI34NVZA, seg_pos={0 0}, upload=367705]: Done uploading piece. err: <nil> Dec 23 02:18:33.434728164 (418h49m33.305811199s) [noise=true , address=145.79.227.42:4121 , piece_id=ICOCUSNLYIVKOJTONNBAEPWGSRTB44H2KAHESVBGGBEDOEHROYGA, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: read tcp 188.166.225.170:50058->145.79.227.42:4121: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.434850206 (418h49m33.305933241s) [noise=true , address=145.79.227.15:4874 , piece_id=PWRV5UFUDENFBPJFK3KYNXMYLQZBHFG6KBJFZXA6P32NT7ZNOWDA, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: read tcp 188.166.225.170:50240->145.79.227.15:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.434955458 (418h49m33.306038493s) [noise=true , address=145.79.227.50:4373 , piece_id=PZRIRRQQETAX56EVN7LY3Z5DUPUYUE7N7H2ZTYWK7Y7UWG6NFFSQ, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, address:145.79.227.50:4373): protocol: read tcp 188.166.225.170:54252->145.79.227.50:4373: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.435015211 (418h49m33.306098246s) [noise=true , address=145.79.227.9:4366 , piece_id=73FK36X6U4CYXVKSGCTLK4YCL2QB3CQKUST2IY4OQGG4JWMLTV6A, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: rpc: dial tcp 145.79.227.9:4366: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.435058384 (418h49m33.306141429s) [noise=true , address=145.79.227.43:4159 , piece_id=WUJFSGFHRK4PHRLS4H3MNT5TVXYR6MSTUQXRN6QBXI47WRPKKN4A, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA, address:145.79.227.43:4159): protocol: read tcp 188.166.225.170:49750->145.79.227.43:4159: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.435156329 (418h49m33.306239374s) [noise=true , address=145.79.227.47:4267 , piece_id=CFW6HHUK6IP232DVR64FPSBPULEIVPVGCS2HR2523RAIP5N77POQ, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8, address:145.79.227.47:4267): protocol: read tcp 188.166.225.170:33996->145.79.227.47:4267: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.435495826 (418h49m33.306578871s) [noise=true , address=77.90.31.21:4877 , piece_id=J3XPMHNIGDTRPYW2ITOOW4FRJ4SUKMCCGRP22WKD55A7OA7EOV3A, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 188.166.225.170:35802->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.435680898 (418h49m33.306763943s) [noise=true , address=145.79.227.41:4086 , piece_id=KXPKPLKHLFQ5GU7Z7LM3NXGWGXSGISKBOHAWPJ7QGG4ESA64YYWA, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, address:145.79.227.41:4086): protocol: read tcp 188.166.225.170:33906->145.79.227.41:4086: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.435701508 (418h49m33.306784543s) [noise=true , address=145.79.227.39:4015 , piece_id=UAGG4FHO2SDQKDF6T4QIGVN7IMNANYWTD3SHYRMBVGCPS3Z4VIYA, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, address:145.79.227.39:4015): protocol: read tcp 188.166.225.170:45508->145.79.227.39:4015: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.435883414 (418h49m33.306966449s) [noise=true , address=145.79.227.37:4565 , piece_id=PAKR2BG4S3KYFXOLS7BL5AANJMIHMN477KGMJJ4XIHMTF7UOTCHA, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:1nvvNczHtQrSEJuJbErnjiAWa4BRDzBrQmBNbB7RrrqN1JWNHT, address:145.79.227.37:4565): protocol: read tcp 188.166.225.170:43228->145.79.227.37:4565: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.435913952 (418h49m33.306996987s) [noise=true , address=145.79.227.38:4978 , piece_id=5VMIQGZ32DI7IE6Z6HMVCJUXIMBFVN6BWONEBE452PG5WDWZXDZQ, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: read tcp 188.166.225.170:57482->145.79.227.38:4978: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.436073857 (418h49m33.307156892s) [noise=true , address=145.79.227.35:4481 , piece_id=EVLROBIPMCRCZYYUSUKKCHCMTJH6G2H56QJR6ZPIF7R7HVVCP6CA, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp, address:145.79.227.35:4481): protocol: read tcp 188.166.225.170:34156->145.79.227.35:4481: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.436167987 (418h49m33.307251022s) [noise=true , address=145.79.227.51:4411 , piece_id=2JQUVX7JG3GYCRYJMYE7AQNIX7XG7XEYRXAC6JTTYRM343CVP3IA, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR, address:145.79.227.51:4411): protocol: read tcp 188.166.225.170:54092->145.79.227.51:4411: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.436191531 (418h49m33.307274566s) [seg_pos={0 0}, upload=367705]: Done waiting for segment. successful: 2 optimal: 13 errs: <nil> Dec 23 02:18:33.726571837 (418h49m33.597654882s) [upload=367705]: Done uploading segments... Dec 23 02:18:33.726612455 (418h49m33.59769549s) [upload=367705]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 2) Duration: 8.900495734s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 7.567566614s trace: https://jaeger.dev.storj.io/trace/2c98a82e82cbeda7 FAIL (2025-12-23 02:23:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== KL7BLGSD3TCZNAHNK7STV2RKAZQWCHN6PROBXAPTA5ZBCPT6A, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.700896105 (418h48m56.57197914s) [noise=true , address=145.79.227.30:4796 , piece_id=IF6NPIC5DTTMJ4DZMIDD6SFN7TGDBN6RD3H7KT7A4OC7VJVCEZOA, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.701329429 (418h48m56.572412474s) [noise=true , address=145.79.227.32:4848 , piece_id=NMEVAZ6HHKE2Z4Q5G4VLNPOIURXE5TXXUMLFJLURZUI4Z62W7VMA, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.701714032 (418h48m56.572797077s) [noise=true , address=145.79.227.28:4742 , piece_id=3IMQS3J42XXGHCZFMBYPDCPCJYRY3I3EUIQ2Z2JOK4TPXQQH75OQ, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.702231197 (418h48m56.573314232s) [noise=true , address=64.49.13.137:4500 , piece_id=RFGLMWE4OHC7RL2NBPX6VRKG4YGRVCIMBHQFW2W5XAEE376UTVVA, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.702751507 (418h48m56.573834552s) [noise=true , address=64.49.13.169:4715 , piece_id=6RCBEKXRJ56IGXWT4PVSTDYRIH3UCAD22C4Z5BU33U5YIPWBEPDQ, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.703271658 (418h48m56.574354693s) [noise=true , address=45.91.116.12:4257 , piece_id=SPTNFMB3LXNY74LCASNSAZIPCOAQHO5V6SOFP27Y4IY25CHWBO2A, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.703948437 (418h48m56.575031482s) [noise=true , address=145.79.227.23:4549 , piece_id=XH7BPLOCXKVTJQDYKIRKFXVLHIXXNJFAXCFZPSSASWLOG3GM5PEQ, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.703969256 (418h48m56.575052291s) [noise=true , address=5.231.73.4:4030 , piece_id=M6FJSRGUNCEIBMNKLBK7HHFL4DA3PDD2MZ2GN2MJKAFSH3XRI7XQ, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.704457947 (418h48m56.575540982s) [noise=true , address=145.79.227.17:4984 , piece_id=DAMTMQIEBM3I2QZTWVEGISFJ5T7I3LMZ3CTXUFXO2CS775GY4F7Q, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.704834724 (418h48m56.575917769s) [noise=true , address=77.90.31.18:4815 , piece_id=XQDHXOY2ZQQHZZHMGLO5GKZOUJZFKKWGLDJRHJLWCFEX6PKNNHWA, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.706591609 (418h48m56.577674654s) [noise=true , address=77.90.31.13:4651 , piece_id=H6MCF4TXI5WFQ2BBCPXL3OX6OA4NCRBIH6IBLMPYPIJBETEN6UGA, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.707118754 (418h48m56.578201789s) [noise=true , address=145.79.227.13:4811 , piece_id=KVHW3GEE4KWLOOUJBRVOOLDA3EGPRUU7VHPH7TQXLURUCCU5J77Q, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.707431179 (418h48m56.578514224s) [noise=true , address=77.90.31.16:4740 , piece_id=KYCKJ7F7CNIXIVAZZ7CQPWAYNCMIM5RKO4IKXETP5EZNOAGKGONA, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.707910952 (418h48m56.578993997s) [noise=true , address=45.91.116.5:4330 , piece_id=EMSR7LFH5Q6R7YAJNH45TDQS2E4KX25Q335IFFUORN5AJJWS7W5Q, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.708492299 (418h48m56.579575344s) [noise=true , address=64.49.13.139:4478 , piece_id=QUBJUJHQ6AYU6L67666PA7BHCRL2IIFOJDISOISTEK4JWABGMXIQ, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.708774406 (418h48m56.579857441s) [noise=true , address=45.91.116.14:4213 , piece_id=VLD3RNN2RLOQI2AKMHNY3G57QJPDNJ3BA5JDRRJHCHNHS6YE52JA, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:18:00.137346621 (418h49m0.008429666s) [upload=367702]: Next returned nil segment Dec 23 02:18:00.137360507 (418h49m0.008443552s) [upload=367702]: Waiting for error group managing segments... Dec 23 02:18:01.750516581 (418h49m1.621599616s) [noise=true , address=64.49.13.137:4500 , piece_id=RFGLMWE4OHC7RL2NBPX6VRKG4YGRVCIMBHQFW2W5XAEE376UTVVA, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:02.125124405 (418h49m1.99620745s) [noise=true , address=77.90.31.18:4815 , piece_id=XQDHXOY2ZQQHZZHMGLO5GKZOUJZFKKWGLDJRHJLWCFEX6PKNNHWA, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:02.226869841 (418h49m2.097952876s) [noise=true , address=45.91.116.8:4060 , piece_id=TRNAVEFSFXMIN3DFMQ2DUAKKDTSZH36TCSYZXFKALUM7TCKM3Y6A, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:05.457222504 (418h49m5.328305539s) [noise=true , address=77.90.31.13:4651 , piece_id=H6MCF4TXI5WFQ2BBCPXL3OX6OA4NCRBIH6IBLMPYPIJBETEN6UGA, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:06.079641082 (418h49m5.950724128s) [noise=true , address=45.91.116.12:4257 , piece_id=SPTNFMB3LXNY74LCASNSAZIPCOAQHO5V6SOFP27Y4IY25CHWBO2A, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:09.265548070 (418h49m9.136631105s) [noise=true , address=64.49.13.139:4478 , piece_id=QUBJUJHQ6AYU6L67666PA7BHCRL2IIFOJDISOISTEK4JWABGMXIQ, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:10.880784395 (418h49m10.751867441s) [noise=true , address=45.91.116.14:4213 , piece_id=VLD3RNN2RLOQI2AKMHNY3G57QJPDNJ3BA5JDRRJHCHNHS6YE52JA, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:12.686956874 (418h49m12.558039929s) [noise=true , address=145.79.227.28:4742 , piece_id=3IMQS3J42XXGHCZFMBYPDCPCJYRY3I3EUIQ2Z2JOK4TPXQQH75OQ, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:12.832920510 (418h49m12.704003545s) [noise=true , address=45.91.116.5:4330 , piece_id=EMSR7LFH5Q6R7YAJNH45TDQS2E4KX25Q335IFFUORN5AJJWS7W5Q, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:22.466644063 (418h49m22.337727098s) [noise=true , address=77.90.31.16:4740 , piece_id=KYCKJ7F7CNIXIVAZZ7CQPWAYNCMIM5RKO4IKXETP5EZNOAGKGONA, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:34.649740548 (418h49m34.520823583s) [noise=true , address=64.49.13.169:4715 , piece_id=6RCBEKXRJ56IGXWT4PVSTDYRIH3UCAD22C4Z5BU33U5YIPWBEPDQ, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:38.250548484 (418h49m38.121631529s) [noise=true , address=145.79.227.17:4984 , piece_id=DAMTMQIEBM3I2QZTWVEGISFJ5T7I3LMZ3CTXUFXO2CS775GY4F7Q, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:43.329219400 (418h49m43.200302445s) [noise=true , address=145.79.227.23:4549 , piece_id=XH7BPLOCXKVTJQDYKIRKFXVLHIXXNJFAXCFZPSSASWLOG3GM5PEQ, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:43.400788824 (418h49m43.271871859s) [noise=true , address=145.79.227.32:4848 , piece_id=NMEVAZ6HHKE2Z4Q5G4VLNPOIURXE5TXXUMLFJLURZUI4Z62W7VMA, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:43.400819082 (418h49m43.271902117s) [seg_pos={0 0}, upload=367702]: Segment reached optimal threshold of 14 pieces. Dec 23 02:18:43.401090247 (418h49m43.272173302s) [noise=true , address=5.231.73.4:4030 , piece_id=M6FJSRGUNCEIBMNKLBK7HHFL4DA3PDD2MZ2GN2MJKAFSH3XRI7XQ, seg_pos={0 0}, upload=367702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rWJM3vC8XrUAfVaoqVZm1qzqcG8AqxLzncExPHEak5oo3QuPA): protocol: expected piece hash; context canceled; EOF Dec 23 02:18:43.401100788 (418h49m43.272183823s) [noise=true , address=5.231.73.4:4030 , piece_id=M6FJSRGUNCEIBMNKLBK7HHFL4DA3PDD2MZ2GN2MJKAFSH3XRI7XQ, seg_pos={0 0}, upload=367702]: Long tail cancellation due to optimal threshold Dec 23 02:18:43.401281792 (418h49m43.272364837s) [noise=true , address=77.90.31.14:4667 , piece_id=5ZXKL7BLGSD3TCZNAHNK7STV2RKAZQWCHN6PROBXAPTA5ZBCPT6A, seg_pos={0 0}, upload=367702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S): protocol: expected piece hash; context canceled; EOF Dec 23 02:18:43.401290889 (418h49m43.272373925s) [noise=true , address=77.90.31.14:4667 , piece_id=5ZXKL7BLGSD3TCZNAHNK7STV2RKAZQWCHN6PROBXAPTA5ZBCPT6A, seg_pos={0 0}, upload=367702]: Long tail cancellation due to optimal threshold Dec 23 02:18:43.401412151 (418h49m43.272495186s) [noise=true , address=145.79.227.13:4811 , piece_id=KVHW3GEE4KWLOOUJBRVOOLDA3EGPRUU7VHPH7TQXLURUCCU5J77Q, seg_pos={0 0}, upload=367702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TxjP4CnxyU4t2DTNBjBMBxnizcHCD42fMrB5Da6Qiqg1Eg17V): protocol: context canceled; context canceled Dec 23 02:18:43.401418663 (418h49m43.272501698s) [noise=true , address=145.79.227.13:4811 , piece_id=KVHW3GEE4KWLOOUJBRVOOLDA3EGPRUU7VHPH7TQXLURUCCU5J77Q, seg_pos={0 0}, upload=367702]: Long tail cancellation due to optimal threshold Dec 23 02:18:43.401630526 (418h49m43.272713561s) [noise=true , address=145.79.227.30:4796 , piece_id=IF6NPIC5DTTMJ4DZMIDD6SFN7TGDBN6RD3H7KT7A4OC7VJVCEZOA, seg_pos={0 0}, upload=367702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: read tcp 188.166.225.170:44484->145.79.227.30:4796: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Dec 23 02:18:43.401638241 (418h49m43.272721276s) [noise=true , address=145.79.227.30:4796 , piece_id=IF6NPIC5DTTMJ4DZMIDD6SFN7TGDBN6RD3H7KT7A4OC7VJVCEZOA, seg_pos={0 0}, upload=367702]: Long tail cancellation due to optimal threshold Dec 23 02:18:43.401657227 (418h49m43.272740273s) [seg_pos={0 0}, upload=367702]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 23 02:18:43.401864953 (418h49m43.272947988s) [upload=367702]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 02:18:43.401870614 (418h49m43.272953649s) [upload=367702]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 02:18:43.703068278 (418h49m43.574151323s) [upload=367702]: Done uploading segments... Dec 23 02:18:43.703144954 (418h49m43.574227999s) [upload=367702]: Upload finished. err: <nil> Duration: 7.567591161s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 5.350905199s trace: https://jaeger.dev.storj.io/trace/14f27c56f52ae4e9 FAIL (2025-12-23 02:23:00 UTC) orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 45) Duration: 5.350934846s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 7.180270794s trace: https://jaeger.dev.storj.io/trace/463d97d26edafa8a FAIL (2025-12-23 02:23:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=367700]: Uploading piece... Dec 23 02:17:52.581508683 (418h48m52.452591718s) [noise=true , address=185.167.204.206:4040 , piece_id=Y6HVWGUESE2BSEIKEMDSLNHWDTOEKGBR76NIFRRQU6KFTVHD4Q7Q, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.581711800 (418h48m52.452794835s) [noise=true , address=185.167.204.206:4058 , piece_id=4CZUKD7KF6H3JH6DLWT3KM2PQC72NRYW4YPN7NL445VBER4QR6SA, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.581958349 (418h48m52.453041384s) [noise=true , address=89.149.201.147:4016 , piece_id=MFNURTZ2HJFACIAHZUO7YZJXRJZB26AVF24JLU3I7IYKDPZ2JJIQ, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.582064401 (418h48m52.453147436s) [noise=true , address=89.149.201.147:4009 , piece_id=WGVNGAHNZIVLBR665VB7GO77EZ7SSJOC5P7ML2UAK6MKV4BAF27A, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.582336870 (418h48m52.453419905s) [noise=true , address=81.171.14.70:4025 , piece_id=PFRRLSNJYBUCYSOVOE3VHDWAKTEPMPQRTE4U2VTPBT7KGYVNQHIQ, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.582391083 (418h48m52.453474118s) [noise=true , address=81.171.14.70:4029 , piece_id=JKKAJ3PXL7WDZ7XNKM752C6NA4B4FUN2AZJCGAZPDVNAGKTAOK7Q, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.582715682 (418h48m52.453798717s) [noise=true , address=89.149.201.147:4002 , piece_id=QCDLIQPY7COKADVRRRUN7UBEIV2AARIPA64LMFCCYPFNA54TA7VQ, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.582847693 (418h48m52.453930738s) [noise=true , address=185.167.204.205:4033 , piece_id=42CCWQIIXKFWWCLUSLGSFQHZMA2UPGZJMOSWD7Z562DV6ABPDVRQ, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.583274976 (418h48m52.454358022s) [noise=true , address=185.167.204.206:4066 , piece_id=WAVNL7LBCOQEW3CFA3DWRITRTBGCWYVASH43KC4RFHE5KFQUQ45A, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.583662655 (418h48m52.45474569s) [noise=true , address=185.167.204.206:4070 , piece_id=GA3GZFHS36RXW4PWKSHZ2QOAXQJVNIWVMED3XAAGG5GQPA27ST7A, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:53.405090439 (418h48m53.276173484s) [upload=367700]: Next returned nil segment Dec 23 02:17:53.405105007 (418h48m53.276188042s) [upload=367700]: Waiting for error group managing segments... Dec 23 02:17:53.798794305 (418h48m53.66987735s) [noise=true , address=185.167.204.205:4025 , piece_id=XY5XNAAQZFPKWJP7V5LWBBH5ZQ7XFUTQ657QEF5KXFTP2EKCSIBA, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.799500561 (418h48m53.670583596s) [noise=true , address=185.167.204.205:4016 , piece_id=523RGOAK7OI6VYQY5X4XT5Y6F5C7NMB2WDXPN74Q27P3BCH2Q37A, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.799686735 (418h48m53.67076977s) [noise=true , address=185.167.204.205:4036 , piece_id=IQ6I5X4HC2ANQSM72N7X4OJWVTUMOWH5YPSSRHUFTGYSXW5ZK4WQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.800348164 (418h48m53.671431209s) [noise=true , address=185.167.204.205:4033 , piece_id=42CCWQIIXKFWWCLUSLGSFQHZMA2UPGZJMOSWD7Z562DV6ABPDVRQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.802008737 (418h48m53.673091782s) [noise=true , address=185.167.204.206:4057 , piece_id=UNUF4NZ5CL74EWLPZLVX44MAYYUQFIAZ54N7FGBLCQHEKLGX6RSQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.802484733 (418h48m53.673567778s) [noise=true , address=185.167.204.206:4040 , piece_id=Y6HVWGUESE2BSEIKEMDSLNHWDTOEKGBR76NIFRRQU6KFTVHD4Q7Q, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.803136594 (418h48m53.674219639s) [noise=true , address=185.167.204.206:4070 , piece_id=GA3GZFHS36RXW4PWKSHZ2QOAXQJVNIWVMED3XAAGG5GQPA27ST7A, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.803750904 (418h48m53.674833939s) [noise=true , address=185.167.204.206:4053 , piece_id=GG5TA7D4KK2IGSKRO4T3APKSZ43IWLND3L54EXYD2LRFIKGI5MRQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.804347520 (418h48m53.675430565s) [noise=true , address=185.167.204.206:4058 , piece_id=4CZUKD7KF6H3JH6DLWT3KM2PQC72NRYW4YPN7NL445VBER4QR6SA, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.805167522 (418h48m53.676250567s) [noise=true , address=185.167.204.206:4054 , piece_id=3QWOKGEC5JODMZCL5R5ULU7EUAKGI2W5QWONT5H2I7EOBH7PVODA, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.805898293 (418h48m53.676981328s) [noise=true , address=185.167.204.206:4066 , piece_id=WAVNL7LBCOQEW3CFA3DWRITRTBGCWYVASH43KC4RFHE5KFQUQ45A, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.808200908 (418h48m53.679283943s) [noise=true , address=185.167.204.206:4064 , piece_id=4K4KD2ZMFXUZT6W3OV7E6BC5UGQHSVZXR67B2QPXIPKGAM6CTVSA, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.808393705 (418h48m53.67947675s) [noise=true , address=185.167.204.205:4004 , piece_id=G5Q7QERF2O3WXIDK5ZAFGSZIN6VXJOFYHAPEC647NIXIJQCSNLKA, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.808844694 (418h48m53.679927739s) [noise=true , address=185.167.204.205:4028 , piece_id=DINID3XD5734Z3RWWS3WPIFLGRUXZMZUNXC2BC3NMX7LWTSDVN3Q, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.810924914 (418h48m53.682007959s) [noise=true , address=185.167.204.206:4047 , piece_id=JSWKSSXVNEFRZPWP56EHG26DRYA7V4K2LPEBCKUOBUG2WET5ZQ7A, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.813330996 (418h48m53.684414031s) [noise=true , address=185.167.204.206:4069 , piece_id=OLTP357X7TH2ABIRWBMAI445KR5SGJVY77PPIQFLRD2KFCBOML2Q, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.813884790 (418h48m53.684967826s) [noise=true , address=185.167.204.205:4001 , piece_id=LMIOF4TXW2JJBYGBNU36M46PXZUNKBKFK56XVNMWRQVR7LSZ3QCQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.815627059 (418h48m53.686710094s) [noise=true , address=185.167.204.206:4067 , piece_id=D24D7H4IJKMKHRJUCQ2ZAYDIVRUYHK2YHEYLVCIL3VA4P5FQJB5A, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:54.448896803 (418h48m54.319979838s) [noise=true , address=81.171.14.70:4029 , piece_id=JKKAJ3PXL7WDZ7XNKM752C6NA4B4FUN2AZJCGAZPDVNAGKTAOK7Q, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:54.835287845 (418h48m54.70637088s) [noise=true , address=81.171.14.70:4025 , piece_id=PFRRLSNJYBUCYSOVOE3VHDWAKTEPMPQRTE4U2VTPBT7KGYVNQHIQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:54.946890399 (418h48m54.817973434s) [noise=true , address=81.171.14.70:4040 , piece_id=6ZIPFOQCGRNK3TDTODGI34TL2WM65BQSRLXXAFX3XQ3VCPAVT2YA, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:54.960812961 (418h48m54.831895996s) [noise=true , address=81.171.14.70:4034 , piece_id=2EA6SNLDV3ZVTNQ5UXEXIC2SJTPH3D6VCVZJI2IAGS22U2J6Y6WQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:55.069208920 (418h48m54.940291965s) [noise=true , address=89.149.201.147:4009 , piece_id=WGVNGAHNZIVLBR665VB7GO77EZ7SSJOC5P7ML2UAK6MKV4BAF27A, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:55.073463732 (418h48m54.944546768s) [noise=true , address=89.149.201.147:4002 , piece_id=QCDLIQPY7COKADVRRRUN7UBEIV2AARIPA64LMFCCYPFNA54TA7VQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:55.132812454 (418h48m55.003895489s) [noise=true , address=89.149.201.147:4010 , piece_id=LZXMVADI5CHNTPWZNXNBEFI3SMGAWQGNUIWDNPM4JMRBLZ62HO4A, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:55.185887030 (418h48m55.056970065s) [noise=true , address=89.149.201.147:4012 , piece_id=2Z64OIS2BBYH6ZCKBXHW4XQZY4FFASHDIHH3C3YVY7MGTTBI6QNA, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:55.185911748 (418h48m55.056994783s) [seg_pos={0 0}, upload=367700]: Segment reached optimal threshold of 26 pieces. Dec 23 02:17:55.186137848 (418h48m55.057220893s) [noise=true , address=89.149.201.147:4015 , piece_id=WENNEH2N2S7NV3TQILPUZLFRBV5CO6VMP52NKDIQE23NO2X7GDWQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RbD6M2MHJfxcJXogZG8nUGFmeZjwUQ1auN6KGWJMPTffdK42K): protocol: expected piece hash; context canceled; EOF Dec 23 02:17:55.186148097 (418h48m55.057231132s) [noise=true , address=89.149.201.147:4015 , piece_id=WENNEH2N2S7NV3TQILPUZLFRBV5CO6VMP52NKDIQE23NO2X7GDWQ, seg_pos={0 0}, upload=367700]: Long tail cancellation due to optimal threshold Dec 23 02:17:55.186212209 (418h48m55.057295244s) [noise=true , address=89.149.201.147:4016 , piece_id=MFNURTZ2HJFACIAHZUO7YZJXRJZB26AVF24JLU3I7IYKDPZ2JJIQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XsvrVhDGkU8BkYsBGmTtax47E43dZAYuGwD8WBMpp3n4GgDM5): protocol: expected piece hash; context canceled; EOF Dec 23 02:17:55.186228150 (418h48m55.057311185s) [noise=true , address=89.149.201.147:4016 , piece_id=MFNURTZ2HJFACIAHZUO7YZJXRJZB26AVF24JLU3I7IYKDPZ2JJIQ, seg_pos={0 0}, upload=367700]: Long tail cancellation due to optimal threshold Dec 23 02:17:55.186247767 (418h48m55.057330802s) [seg_pos={0 0}, upload=367700]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 23 02:17:55.186337368 (418h48m55.057420403s) [upload=367700]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 02:17:55.186342387 (418h48m55.057425422s) [upload=367700]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 02:17:55.430535735 (418h48m55.30161878s) [upload=367700]: Done uploading segments... Dec 23 02:17:55.430557656 (418h48m55.301640701s) [upload=367700]: Upload finished. err: <nil> Duration: 7.180302194s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 3.398303588s trace: https://jaeger.dev.storj.io/trace/20c185d15563c01c FAIL (2025-12-23 02:23:00 UTC) orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 39) Duration: 3.398326562s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 5.886765892s trace: https://jaeger.dev.storj.io/trace/70da446b2b2ff016 FAIL (2025-12-23 02:22:50 UTC) orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 46) Duration: 5.886793246s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................................................X.......] duration: 3.63079091s trace: https://jaeger.dev.storj.io/trace/7f7fb60dfbab7793 FAIL (2025-12-23 02:22:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.630871484s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 3.254507842s trace: https://jaeger.dev.storj.io/trace/20c185d15563c01c FAIL (2025-12-23 02:22:39 UTC) orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 39) Duration: 3.25453762s core uplink ap1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 5.400955648s trace: https://jaeger.dev.storj.io/trace/463d97d26edafa8a FAIL (2025-12-23 02:22:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=367700]: Uploading piece... Dec 23 02:17:52.581508683 (418h48m52.452591718s) [noise=true , address=185.167.204.206:4040 , piece_id=Y6HVWGUESE2BSEIKEMDSLNHWDTOEKGBR76NIFRRQU6KFTVHD4Q7Q, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.581711800 (418h48m52.452794835s) [noise=true , address=185.167.204.206:4058 , piece_id=4CZUKD7KF6H3JH6DLWT3KM2PQC72NRYW4YPN7NL445VBER4QR6SA, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.581958349 (418h48m52.453041384s) [noise=true , address=89.149.201.147:4016 , piece_id=MFNURTZ2HJFACIAHZUO7YZJXRJZB26AVF24JLU3I7IYKDPZ2JJIQ, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.582064401 (418h48m52.453147436s) [noise=true , address=89.149.201.147:4009 , piece_id=WGVNGAHNZIVLBR665VB7GO77EZ7SSJOC5P7ML2UAK6MKV4BAF27A, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.582336870 (418h48m52.453419905s) [noise=true , address=81.171.14.70:4025 , piece_id=PFRRLSNJYBUCYSOVOE3VHDWAKTEPMPQRTE4U2VTPBT7KGYVNQHIQ, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.582391083 (418h48m52.453474118s) [noise=true , address=81.171.14.70:4029 , piece_id=JKKAJ3PXL7WDZ7XNKM752C6NA4B4FUN2AZJCGAZPDVNAGKTAOK7Q, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.582715682 (418h48m52.453798717s) [noise=true , address=89.149.201.147:4002 , piece_id=QCDLIQPY7COKADVRRRUN7UBEIV2AARIPA64LMFCCYPFNA54TA7VQ, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.582847693 (418h48m52.453930738s) [noise=true , address=185.167.204.205:4033 , piece_id=42CCWQIIXKFWWCLUSLGSFQHZMA2UPGZJMOSWD7Z562DV6ABPDVRQ, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.583274976 (418h48m52.454358022s) [noise=true , address=185.167.204.206:4066 , piece_id=WAVNL7LBCOQEW3CFA3DWRITRTBGCWYVASH43KC4RFHE5KFQUQ45A, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.583662655 (418h48m52.45474569s) [noise=true , address=185.167.204.206:4070 , piece_id=GA3GZFHS36RXW4PWKSHZ2QOAXQJVNIWVMED3XAAGG5GQPA27ST7A, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:53.405090439 (418h48m53.276173484s) [upload=367700]: Next returned nil segment Dec 23 02:17:53.405105007 (418h48m53.276188042s) [upload=367700]: Waiting for error group managing segments... Dec 23 02:17:53.798794305 (418h48m53.66987735s) [noise=true , address=185.167.204.205:4025 , piece_id=XY5XNAAQZFPKWJP7V5LWBBH5ZQ7XFUTQ657QEF5KXFTP2EKCSIBA, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.799500561 (418h48m53.670583596s) [noise=true , address=185.167.204.205:4016 , piece_id=523RGOAK7OI6VYQY5X4XT5Y6F5C7NMB2WDXPN74Q27P3BCH2Q37A, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.799686735 (418h48m53.67076977s) [noise=true , address=185.167.204.205:4036 , piece_id=IQ6I5X4HC2ANQSM72N7X4OJWVTUMOWH5YPSSRHUFTGYSXW5ZK4WQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.800348164 (418h48m53.671431209s) [noise=true , address=185.167.204.205:4033 , piece_id=42CCWQIIXKFWWCLUSLGSFQHZMA2UPGZJMOSWD7Z562DV6ABPDVRQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.802008737 (418h48m53.673091782s) [noise=true , address=185.167.204.206:4057 , piece_id=UNUF4NZ5CL74EWLPZLVX44MAYYUQFIAZ54N7FGBLCQHEKLGX6RSQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.802484733 (418h48m53.673567778s) [noise=true , address=185.167.204.206:4040 , piece_id=Y6HVWGUESE2BSEIKEMDSLNHWDTOEKGBR76NIFRRQU6KFTVHD4Q7Q, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.803136594 (418h48m53.674219639s) [noise=true , address=185.167.204.206:4070 , piece_id=GA3GZFHS36RXW4PWKSHZ2QOAXQJVNIWVMED3XAAGG5GQPA27ST7A, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.803750904 (418h48m53.674833939s) [noise=true , address=185.167.204.206:4053 , piece_id=GG5TA7D4KK2IGSKRO4T3APKSZ43IWLND3L54EXYD2LRFIKGI5MRQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.804347520 (418h48m53.675430565s) [noise=true , address=185.167.204.206:4058 , piece_id=4CZUKD7KF6H3JH6DLWT3KM2PQC72NRYW4YPN7NL445VBER4QR6SA, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.805167522 (418h48m53.676250567s) [noise=true , address=185.167.204.206:4054 , piece_id=3QWOKGEC5JODMZCL5R5ULU7EUAKGI2W5QWONT5H2I7EOBH7PVODA, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.805898293 (418h48m53.676981328s) [noise=true , address=185.167.204.206:4066 , piece_id=WAVNL7LBCOQEW3CFA3DWRITRTBGCWYVASH43KC4RFHE5KFQUQ45A, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.808200908 (418h48m53.679283943s) [noise=true , address=185.167.204.206:4064 , piece_id=4K4KD2ZMFXUZT6W3OV7E6BC5UGQHSVZXR67B2QPXIPKGAM6CTVSA, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.808393705 (418h48m53.67947675s) [noise=true , address=185.167.204.205:4004 , piece_id=G5Q7QERF2O3WXIDK5ZAFGSZIN6VXJOFYHAPEC647NIXIJQCSNLKA, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.808844694 (418h48m53.679927739s) [noise=true , address=185.167.204.205:4028 , piece_id=DINID3XD5734Z3RWWS3WPIFLGRUXZMZUNXC2BC3NMX7LWTSDVN3Q, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.810924914 (418h48m53.682007959s) [noise=true , address=185.167.204.206:4047 , piece_id=JSWKSSXVNEFRZPWP56EHG26DRYA7V4K2LPEBCKUOBUG2WET5ZQ7A, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.813330996 (418h48m53.684414031s) [noise=true , address=185.167.204.206:4069 , piece_id=OLTP357X7TH2ABIRWBMAI445KR5SGJVY77PPIQFLRD2KFCBOML2Q, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.813884790 (418h48m53.684967826s) [noise=true , address=185.167.204.205:4001 , piece_id=LMIOF4TXW2JJBYGBNU36M46PXZUNKBKFK56XVNMWRQVR7LSZ3QCQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.815627059 (418h48m53.686710094s) [noise=true , address=185.167.204.206:4067 , piece_id=D24D7H4IJKMKHRJUCQ2ZAYDIVRUYHK2YHEYLVCIL3VA4P5FQJB5A, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:54.448896803 (418h48m54.319979838s) [noise=true , address=81.171.14.70:4029 , piece_id=JKKAJ3PXL7WDZ7XNKM752C6NA4B4FUN2AZJCGAZPDVNAGKTAOK7Q, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:54.835287845 (418h48m54.70637088s) [noise=true , address=81.171.14.70:4025 , piece_id=PFRRLSNJYBUCYSOVOE3VHDWAKTEPMPQRTE4U2VTPBT7KGYVNQHIQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:54.946890399 (418h48m54.817973434s) [noise=true , address=81.171.14.70:4040 , piece_id=6ZIPFOQCGRNK3TDTODGI34TL2WM65BQSRLXXAFX3XQ3VCPAVT2YA, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:54.960812961 (418h48m54.831895996s) [noise=true , address=81.171.14.70:4034 , piece_id=2EA6SNLDV3ZVTNQ5UXEXIC2SJTPH3D6VCVZJI2IAGS22U2J6Y6WQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:55.069208920 (418h48m54.940291965s) [noise=true , address=89.149.201.147:4009 , piece_id=WGVNGAHNZIVLBR665VB7GO77EZ7SSJOC5P7ML2UAK6MKV4BAF27A, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:55.073463732 (418h48m54.944546768s) [noise=true , address=89.149.201.147:4002 , piece_id=QCDLIQPY7COKADVRRRUN7UBEIV2AARIPA64LMFCCYPFNA54TA7VQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:55.132812454 (418h48m55.003895489s) [noise=true , address=89.149.201.147:4010 , piece_id=LZXMVADI5CHNTPWZNXNBEFI3SMGAWQGNUIWDNPM4JMRBLZ62HO4A, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:55.185887030 (418h48m55.056970065s) [noise=true , address=89.149.201.147:4012 , piece_id=2Z64OIS2BBYH6ZCKBXHW4XQZY4FFASHDIHH3C3YVY7MGTTBI6QNA, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:55.185911748 (418h48m55.056994783s) [seg_pos={0 0}, upload=367700]: Segment reached optimal threshold of 26 pieces. Dec 23 02:17:55.186137848 (418h48m55.057220893s) [noise=true , address=89.149.201.147:4015 , piece_id=WENNEH2N2S7NV3TQILPUZLFRBV5CO6VMP52NKDIQE23NO2X7GDWQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RbD6M2MHJfxcJXogZG8nUGFmeZjwUQ1auN6KGWJMPTffdK42K): protocol: expected piece hash; context canceled; EOF Dec 23 02:17:55.186148097 (418h48m55.057231132s) [noise=true , address=89.149.201.147:4015 , piece_id=WENNEH2N2S7NV3TQILPUZLFRBV5CO6VMP52NKDIQE23NO2X7GDWQ, seg_pos={0 0}, upload=367700]: Long tail cancellation due to optimal threshold Dec 23 02:17:55.186212209 (418h48m55.057295244s) [noise=true , address=89.149.201.147:4016 , piece_id=MFNURTZ2HJFACIAHZUO7YZJXRJZB26AVF24JLU3I7IYKDPZ2JJIQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XsvrVhDGkU8BkYsBGmTtax47E43dZAYuGwD8WBMpp3n4GgDM5): protocol: expected piece hash; context canceled; EOF Dec 23 02:17:55.186228150 (418h48m55.057311185s) [noise=true , address=89.149.201.147:4016 , piece_id=MFNURTZ2HJFACIAHZUO7YZJXRJZB26AVF24JLU3I7IYKDPZ2JJIQ, seg_pos={0 0}, upload=367700]: Long tail cancellation due to optimal threshold Dec 23 02:17:55.186247767 (418h48m55.057330802s) [seg_pos={0 0}, upload=367700]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 23 02:17:55.186337368 (418h48m55.057420403s) [upload=367700]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 02:17:55.186342387 (418h48m55.057425422s) [upload=367700]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 02:17:55.430535735 (418h48m55.30161878s) [upload=367700]: Done uploading segments... Dec 23 02:17:55.430557656 (418h48m55.301640701s) [upload=367700]: Upload finished. err: <nil> Duration: 5.400986395s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 6.005244955s trace: https://jaeger.dev.storj.io/trace/5b87e1ad666dcc24 FAIL (2025-12-23 02:22:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 2) ======LOGS====== 7Y7UWG6NFFSQ, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:05.480444685 (418h49m5.35152773s) [noise=true , address=45.91.116.4:4937 , piece_id=IIV6VFBKACIVI3M6A6L2WSDU54D3MRGZ36LG6E3E4SXKQVRX74LQ, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:05.480943264 (418h49m5.352026309s) [noise=true , address=145.79.227.9:4366 , piece_id=73FK36X6U4CYXVKSGCTLK4YCL2QB3CQKUST2IY4OQGG4JWMLTV6A, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:05.482473688 (418h49m5.353556733s) [noise=true , address=145.79.227.41:4086 , piece_id=KXPKPLKHLFQ5GU7Z7LM3NXGWGXSGISKBOHAWPJ7QGG4ESA64YYWA, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:05.482598135 (418h49m5.35368118s) [noise=true , address=145.79.227.43:4159 , piece_id=WUJFSGFHRK4PHRLS4H3MNT5TVXYR6MSTUQXRN6QBXI47WRPKKN4A, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:05.485571146 (418h49m5.356654182s) [noise=true , address=77.90.31.12:4589 , piece_id=RD5KILZQDADOR2UPRRAO3XM7YWI7AQ57HBR37VC44DNZPOS555SA, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:05.488437175 (418h49m5.35952021s) [noise=true , address=145.79.227.35:4481 , piece_id=EVLROBIPMCRCZYYUSUKKCHCMTJH6G2H56QJR6ZPIF7R7HVVCP6CA, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:05.491243618 (418h49m5.362326653s) [noise=true , address=145.79.227.39:4015 , piece_id=UAGG4FHO2SDQKDF6T4QIGVN7IMNANYWTD3SHYRMBVGCPS3Z4VIYA, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:05.492801755 (418h49m5.36388479s) [noise=true , address=77.90.31.16:4735 , piece_id=XEZKWPQDBCT7YRGYTH4TFR4BDCP34BPJB6OVFTYGMQ2NQI34NVZA, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:05.495283631 (418h49m5.366366666s) [noise=true , address=145.79.227.37:4565 , piece_id=PAKR2BG4S3KYFXOLS7BL5AANJMIHMN477KGMJJ4XIHMTF7UOTCHA, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:05.495702669 (418h49m5.366785704s) [noise=true , address=145.79.227.42:4121 , piece_id=ICOCUSNLYIVKOJTONNBAEPWGSRTB44H2KAHESVBGGBEDOEHROYGA, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:05.499769763 (418h49m5.370852808s) [noise=true , address=145.79.227.51:4411 , piece_id=2JQUVX7JG3GYCRYJMYE7AQNIX7XG7XEYRXAC6JTTYRM343CVP3IA, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:08.589264332 (418h49m8.460347367s) [upload=367705]: Next returned nil segment Dec 23 02:18:08.589286223 (418h49m8.460369258s) [upload=367705]: Waiting for error group managing segments... Dec 23 02:18:12.569357328 (418h49m12.440440364s) [noise=true , address=45.91.116.4:4937 , piece_id=IIV6VFBKACIVI3M6A6L2WSDU54D3MRGZ36LG6E3E4SXKQVRX74LQ, seg_pos={0 0}, upload=367705]: Done uploading piece. err: <nil> Dec 23 02:18:15.106396805 (418h49m14.977479851s) [noise=true , address=77.90.31.12:4589 , piece_id=RD5KILZQDADOR2UPRRAO3XM7YWI7AQ57HBR37VC44DNZPOS555SA, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 23 02:18:17.598569766 (418h49m17.469652801s) [noise=true , address=77.90.31.16:4735 , piece_id=XEZKWPQDBCT7YRGYTH4TFR4BDCP34BPJB6OVFTYGMQ2NQI34NVZA, seg_pos={0 0}, upload=367705]: Done uploading piece. err: <nil> Dec 23 02:18:33.434728164 (418h49m33.305811199s) [noise=true , address=145.79.227.42:4121 , piece_id=ICOCUSNLYIVKOJTONNBAEPWGSRTB44H2KAHESVBGGBEDOEHROYGA, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: read tcp 188.166.225.170:50058->145.79.227.42:4121: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.434850206 (418h49m33.305933241s) [noise=true , address=145.79.227.15:4874 , piece_id=PWRV5UFUDENFBPJFK3KYNXMYLQZBHFG6KBJFZXA6P32NT7ZNOWDA, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: read tcp 188.166.225.170:50240->145.79.227.15:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.434955458 (418h49m33.306038493s) [noise=true , address=145.79.227.50:4373 , piece_id=PZRIRRQQETAX56EVN7LY3Z5DUPUYUE7N7H2ZTYWK7Y7UWG6NFFSQ, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, address:145.79.227.50:4373): protocol: read tcp 188.166.225.170:54252->145.79.227.50:4373: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.435015211 (418h49m33.306098246s) [noise=true , address=145.79.227.9:4366 , piece_id=73FK36X6U4CYXVKSGCTLK4YCL2QB3CQKUST2IY4OQGG4JWMLTV6A, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: rpc: dial tcp 145.79.227.9:4366: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.435058384 (418h49m33.306141429s) [noise=true , address=145.79.227.43:4159 , piece_id=WUJFSGFHRK4PHRLS4H3MNT5TVXYR6MSTUQXRN6QBXI47WRPKKN4A, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA, address:145.79.227.43:4159): protocol: read tcp 188.166.225.170:49750->145.79.227.43:4159: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.435156329 (418h49m33.306239374s) [noise=true , address=145.79.227.47:4267 , piece_id=CFW6HHUK6IP232DVR64FPSBPULEIVPVGCS2HR2523RAIP5N77POQ, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8, address:145.79.227.47:4267): protocol: read tcp 188.166.225.170:33996->145.79.227.47:4267: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.435495826 (418h49m33.306578871s) [noise=true , address=77.90.31.21:4877 , piece_id=J3XPMHNIGDTRPYW2ITOOW4FRJ4SUKMCCGRP22WKD55A7OA7EOV3A, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 188.166.225.170:35802->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.435680898 (418h49m33.306763943s) [noise=true , address=145.79.227.41:4086 , piece_id=KXPKPLKHLFQ5GU7Z7LM3NXGWGXSGISKBOHAWPJ7QGG4ESA64YYWA, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, address:145.79.227.41:4086): protocol: read tcp 188.166.225.170:33906->145.79.227.41:4086: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.435701508 (418h49m33.306784543s) [noise=true , address=145.79.227.39:4015 , piece_id=UAGG4FHO2SDQKDF6T4QIGVN7IMNANYWTD3SHYRMBVGCPS3Z4VIYA, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, address:145.79.227.39:4015): protocol: read tcp 188.166.225.170:45508->145.79.227.39:4015: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.435883414 (418h49m33.306966449s) [noise=true , address=145.79.227.37:4565 , piece_id=PAKR2BG4S3KYFXOLS7BL5AANJMIHMN477KGMJJ4XIHMTF7UOTCHA, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:1nvvNczHtQrSEJuJbErnjiAWa4BRDzBrQmBNbB7RrrqN1JWNHT, address:145.79.227.37:4565): protocol: read tcp 188.166.225.170:43228->145.79.227.37:4565: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.435913952 (418h49m33.306996987s) [noise=true , address=145.79.227.38:4978 , piece_id=5VMIQGZ32DI7IE6Z6HMVCJUXIMBFVN6BWONEBE452PG5WDWZXDZQ, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: read tcp 188.166.225.170:57482->145.79.227.38:4978: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.436073857 (418h49m33.307156892s) [noise=true , address=145.79.227.35:4481 , piece_id=EVLROBIPMCRCZYYUSUKKCHCMTJH6G2H56QJR6ZPIF7R7HVVCP6CA, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp, address:145.79.227.35:4481): protocol: read tcp 188.166.225.170:34156->145.79.227.35:4481: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.436167987 (418h49m33.307251022s) [noise=true , address=145.79.227.51:4411 , piece_id=2JQUVX7JG3GYCRYJMYE7AQNIX7XG7XEYRXAC6JTTYRM343CVP3IA, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR, address:145.79.227.51:4411): protocol: read tcp 188.166.225.170:54092->145.79.227.51:4411: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.436191531 (418h49m33.307274566s) [seg_pos={0 0}, upload=367705]: Done waiting for segment. successful: 2 optimal: 13 errs: <nil> Dec 23 02:18:33.726571837 (418h49m33.597654882s) [upload=367705]: Done uploading segments... Dec 23 02:18:33.726612455 (418h49m33.59769549s) [upload=367705]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 2) Duration: 6.005275863s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 5.02795793s trace: https://jaeger.dev.storj.io/trace/14f27c56f52ae4e9 FAIL (2025-12-23 02:22:26 UTC) orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 45) Duration: 5.027982908s core uplink eu1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 5.95691461s trace: https://jaeger.dev.storj.io/trace/61a0842c4fdd78ca FAIL (2025-12-23 02:22:18 UTC) orbiter warning: context deadline exceeded Duration: 5.956941813s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 6.10645727s trace: https://jaeger.dev.storj.io/trace/70da446b2b2ff016 FAIL (2025-12-23 02:22:17 UTC) orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 46) Duration: 6.1065101s core uplink us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 5.858918594s trace: https://jaeger.dev.storj.io/trace/463d97d26edafa8a FAIL (2025-12-23 02:22:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=367700]: Uploading piece... Dec 23 02:17:52.581508683 (418h48m52.452591718s) [noise=true , address=185.167.204.206:4040 , piece_id=Y6HVWGUESE2BSEIKEMDSLNHWDTOEKGBR76NIFRRQU6KFTVHD4Q7Q, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.581711800 (418h48m52.452794835s) [noise=true , address=185.167.204.206:4058 , piece_id=4CZUKD7KF6H3JH6DLWT3KM2PQC72NRYW4YPN7NL445VBER4QR6SA, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.581958349 (418h48m52.453041384s) [noise=true , address=89.149.201.147:4016 , piece_id=MFNURTZ2HJFACIAHZUO7YZJXRJZB26AVF24JLU3I7IYKDPZ2JJIQ, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.582064401 (418h48m52.453147436s) [noise=true , address=89.149.201.147:4009 , piece_id=WGVNGAHNZIVLBR665VB7GO77EZ7SSJOC5P7ML2UAK6MKV4BAF27A, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.582336870 (418h48m52.453419905s) [noise=true , address=81.171.14.70:4025 , piece_id=PFRRLSNJYBUCYSOVOE3VHDWAKTEPMPQRTE4U2VTPBT7KGYVNQHIQ, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.582391083 (418h48m52.453474118s) [noise=true , address=81.171.14.70:4029 , piece_id=JKKAJ3PXL7WDZ7XNKM752C6NA4B4FUN2AZJCGAZPDVNAGKTAOK7Q, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.582715682 (418h48m52.453798717s) [noise=true , address=89.149.201.147:4002 , piece_id=QCDLIQPY7COKADVRRRUN7UBEIV2AARIPA64LMFCCYPFNA54TA7VQ, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.582847693 (418h48m52.453930738s) [noise=true , address=185.167.204.205:4033 , piece_id=42CCWQIIXKFWWCLUSLGSFQHZMA2UPGZJMOSWD7Z562DV6ABPDVRQ, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.583274976 (418h48m52.454358022s) [noise=true , address=185.167.204.206:4066 , piece_id=WAVNL7LBCOQEW3CFA3DWRITRTBGCWYVASH43KC4RFHE5KFQUQ45A, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.583662655 (418h48m52.45474569s) [noise=true , address=185.167.204.206:4070 , piece_id=GA3GZFHS36RXW4PWKSHZ2QOAXQJVNIWVMED3XAAGG5GQPA27ST7A, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:53.405090439 (418h48m53.276173484s) [upload=367700]: Next returned nil segment Dec 23 02:17:53.405105007 (418h48m53.276188042s) [upload=367700]: Waiting for error group managing segments... Dec 23 02:17:53.798794305 (418h48m53.66987735s) [noise=true , address=185.167.204.205:4025 , piece_id=XY5XNAAQZFPKWJP7V5LWBBH5ZQ7XFUTQ657QEF5KXFTP2EKCSIBA, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.799500561 (418h48m53.670583596s) [noise=true , address=185.167.204.205:4016 , piece_id=523RGOAK7OI6VYQY5X4XT5Y6F5C7NMB2WDXPN74Q27P3BCH2Q37A, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.799686735 (418h48m53.67076977s) [noise=true , address=185.167.204.205:4036 , piece_id=IQ6I5X4HC2ANQSM72N7X4OJWVTUMOWH5YPSSRHUFTGYSXW5ZK4WQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.800348164 (418h48m53.671431209s) [noise=true , address=185.167.204.205:4033 , piece_id=42CCWQIIXKFWWCLUSLGSFQHZMA2UPGZJMOSWD7Z562DV6ABPDVRQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.802008737 (418h48m53.673091782s) [noise=true , address=185.167.204.206:4057 , piece_id=UNUF4NZ5CL74EWLPZLVX44MAYYUQFIAZ54N7FGBLCQHEKLGX6RSQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.802484733 (418h48m53.673567778s) [noise=true , address=185.167.204.206:4040 , piece_id=Y6HVWGUESE2BSEIKEMDSLNHWDTOEKGBR76NIFRRQU6KFTVHD4Q7Q, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.803136594 (418h48m53.674219639s) [noise=true , address=185.167.204.206:4070 , piece_id=GA3GZFHS36RXW4PWKSHZ2QOAXQJVNIWVMED3XAAGG5GQPA27ST7A, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.803750904 (418h48m53.674833939s) [noise=true , address=185.167.204.206:4053 , piece_id=GG5TA7D4KK2IGSKRO4T3APKSZ43IWLND3L54EXYD2LRFIKGI5MRQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.804347520 (418h48m53.675430565s) [noise=true , address=185.167.204.206:4058 , piece_id=4CZUKD7KF6H3JH6DLWT3KM2PQC72NRYW4YPN7NL445VBER4QR6SA, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.805167522 (418h48m53.676250567s) [noise=true , address=185.167.204.206:4054 , piece_id=3QWOKGEC5JODMZCL5R5ULU7EUAKGI2W5QWONT5H2I7EOBH7PVODA, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.805898293 (418h48m53.676981328s) [noise=true , address=185.167.204.206:4066 , piece_id=WAVNL7LBCOQEW3CFA3DWRITRTBGCWYVASH43KC4RFHE5KFQUQ45A, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.808200908 (418h48m53.679283943s) [noise=true , address=185.167.204.206:4064 , piece_id=4K4KD2ZMFXUZT6W3OV7E6BC5UGQHSVZXR67B2QPXIPKGAM6CTVSA, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.808393705 (418h48m53.67947675s) [noise=true , address=185.167.204.205:4004 , piece_id=G5Q7QERF2O3WXIDK5ZAFGSZIN6VXJOFYHAPEC647NIXIJQCSNLKA, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.808844694 (418h48m53.679927739s) [noise=true , address=185.167.204.205:4028 , piece_id=DINID3XD5734Z3RWWS3WPIFLGRUXZMZUNXC2BC3NMX7LWTSDVN3Q, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.810924914 (418h48m53.682007959s) [noise=true , address=185.167.204.206:4047 , piece_id=JSWKSSXVNEFRZPWP56EHG26DRYA7V4K2LPEBCKUOBUG2WET5ZQ7A, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.813330996 (418h48m53.684414031s) [noise=true , address=185.167.204.206:4069 , piece_id=OLTP357X7TH2ABIRWBMAI445KR5SGJVY77PPIQFLRD2KFCBOML2Q, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.813884790 (418h48m53.684967826s) [noise=true , address=185.167.204.205:4001 , piece_id=LMIOF4TXW2JJBYGBNU36M46PXZUNKBKFK56XVNMWRQVR7LSZ3QCQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.815627059 (418h48m53.686710094s) [noise=true , address=185.167.204.206:4067 , piece_id=D24D7H4IJKMKHRJUCQ2ZAYDIVRUYHK2YHEYLVCIL3VA4P5FQJB5A, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:54.448896803 (418h48m54.319979838s) [noise=true , address=81.171.14.70:4029 , piece_id=JKKAJ3PXL7WDZ7XNKM752C6NA4B4FUN2AZJCGAZPDVNAGKTAOK7Q, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:54.835287845 (418h48m54.70637088s) [noise=true , address=81.171.14.70:4025 , piece_id=PFRRLSNJYBUCYSOVOE3VHDWAKTEPMPQRTE4U2VTPBT7KGYVNQHIQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:54.946890399 (418h48m54.817973434s) [noise=true , address=81.171.14.70:4040 , piece_id=6ZIPFOQCGRNK3TDTODGI34TL2WM65BQSRLXXAFX3XQ3VCPAVT2YA, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:54.960812961 (418h48m54.831895996s) [noise=true , address=81.171.14.70:4034 , piece_id=2EA6SNLDV3ZVTNQ5UXEXIC2SJTPH3D6VCVZJI2IAGS22U2J6Y6WQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:55.069208920 (418h48m54.940291965s) [noise=true , address=89.149.201.147:4009 , piece_id=WGVNGAHNZIVLBR665VB7GO77EZ7SSJOC5P7ML2UAK6MKV4BAF27A, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:55.073463732 (418h48m54.944546768s) [noise=true , address=89.149.201.147:4002 , piece_id=QCDLIQPY7COKADVRRRUN7UBEIV2AARIPA64LMFCCYPFNA54TA7VQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:55.132812454 (418h48m55.003895489s) [noise=true , address=89.149.201.147:4010 , piece_id=LZXMVADI5CHNTPWZNXNBEFI3SMGAWQGNUIWDNPM4JMRBLZ62HO4A, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:55.185887030 (418h48m55.056970065s) [noise=true , address=89.149.201.147:4012 , piece_id=2Z64OIS2BBYH6ZCKBXHW4XQZY4FFASHDIHH3C3YVY7MGTTBI6QNA, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:55.185911748 (418h48m55.056994783s) [seg_pos={0 0}, upload=367700]: Segment reached optimal threshold of 26 pieces. Dec 23 02:17:55.186137848 (418h48m55.057220893s) [noise=true , address=89.149.201.147:4015 , piece_id=WENNEH2N2S7NV3TQILPUZLFRBV5CO6VMP52NKDIQE23NO2X7GDWQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RbD6M2MHJfxcJXogZG8nUGFmeZjwUQ1auN6KGWJMPTffdK42K): protocol: expected piece hash; context canceled; EOF Dec 23 02:17:55.186148097 (418h48m55.057231132s) [noise=true , address=89.149.201.147:4015 , piece_id=WENNEH2N2S7NV3TQILPUZLFRBV5CO6VMP52NKDIQE23NO2X7GDWQ, seg_pos={0 0}, upload=367700]: Long tail cancellation due to optimal threshold Dec 23 02:17:55.186212209 (418h48m55.057295244s) [noise=true , address=89.149.201.147:4016 , piece_id=MFNURTZ2HJFACIAHZUO7YZJXRJZB26AVF24JLU3I7IYKDPZ2JJIQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XsvrVhDGkU8BkYsBGmTtax47E43dZAYuGwD8WBMpp3n4GgDM5): protocol: expected piece hash; context canceled; EOF Dec 23 02:17:55.186228150 (418h48m55.057311185s) [noise=true , address=89.149.201.147:4016 , piece_id=MFNURTZ2HJFACIAHZUO7YZJXRJZB26AVF24JLU3I7IYKDPZ2JJIQ, seg_pos={0 0}, upload=367700]: Long tail cancellation due to optimal threshold Dec 23 02:17:55.186247767 (418h48m55.057330802s) [seg_pos={0 0}, upload=367700]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 23 02:17:55.186337368 (418h48m55.057420403s) [upload=367700]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 02:17:55.186342387 (418h48m55.057425422s) [upload=367700]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 02:17:55.430535735 (418h48m55.30161878s) [upload=367700]: Done uploading segments... Dec 23 02:17:55.430557656 (418h48m55.301640701s) [upload=367700]: Upload finished. err: <nil> Duration: 5.858943704s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................................................X......] duration: 3.611657638s trace: https://jaeger.dev.storj.io/trace/7f7fb60dfbab7793 FAIL (2025-12-23 02:22:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.611776977s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 8.124478522s trace: https://jaeger.dev.storj.io/trace/2c98a82e82cbeda7 FAIL (2025-12-23 02:21:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== KL7BLGSD3TCZNAHNK7STV2RKAZQWCHN6PROBXAPTA5ZBCPT6A, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.700896105 (418h48m56.57197914s) [noise=true , address=145.79.227.30:4796 , piece_id=IF6NPIC5DTTMJ4DZMIDD6SFN7TGDBN6RD3H7KT7A4OC7VJVCEZOA, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.701329429 (418h48m56.572412474s) [noise=true , address=145.79.227.32:4848 , piece_id=NMEVAZ6HHKE2Z4Q5G4VLNPOIURXE5TXXUMLFJLURZUI4Z62W7VMA, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.701714032 (418h48m56.572797077s) [noise=true , address=145.79.227.28:4742 , piece_id=3IMQS3J42XXGHCZFMBYPDCPCJYRY3I3EUIQ2Z2JOK4TPXQQH75OQ, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.702231197 (418h48m56.573314232s) [noise=true , address=64.49.13.137:4500 , piece_id=RFGLMWE4OHC7RL2NBPX6VRKG4YGRVCIMBHQFW2W5XAEE376UTVVA, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.702751507 (418h48m56.573834552s) [noise=true , address=64.49.13.169:4715 , piece_id=6RCBEKXRJ56IGXWT4PVSTDYRIH3UCAD22C4Z5BU33U5YIPWBEPDQ, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.703271658 (418h48m56.574354693s) [noise=true , address=45.91.116.12:4257 , piece_id=SPTNFMB3LXNY74LCASNSAZIPCOAQHO5V6SOFP27Y4IY25CHWBO2A, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.703948437 (418h48m56.575031482s) [noise=true , address=145.79.227.23:4549 , piece_id=XH7BPLOCXKVTJQDYKIRKFXVLHIXXNJFAXCFZPSSASWLOG3GM5PEQ, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.703969256 (418h48m56.575052291s) [noise=true , address=5.231.73.4:4030 , piece_id=M6FJSRGUNCEIBMNKLBK7HHFL4DA3PDD2MZ2GN2MJKAFSH3XRI7XQ, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.704457947 (418h48m56.575540982s) [noise=true , address=145.79.227.17:4984 , piece_id=DAMTMQIEBM3I2QZTWVEGISFJ5T7I3LMZ3CTXUFXO2CS775GY4F7Q, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.704834724 (418h48m56.575917769s) [noise=true , address=77.90.31.18:4815 , piece_id=XQDHXOY2ZQQHZZHMGLO5GKZOUJZFKKWGLDJRHJLWCFEX6PKNNHWA, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.706591609 (418h48m56.577674654s) [noise=true , address=77.90.31.13:4651 , piece_id=H6MCF4TXI5WFQ2BBCPXL3OX6OA4NCRBIH6IBLMPYPIJBETEN6UGA, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.707118754 (418h48m56.578201789s) [noise=true , address=145.79.227.13:4811 , piece_id=KVHW3GEE4KWLOOUJBRVOOLDA3EGPRUU7VHPH7TQXLURUCCU5J77Q, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.707431179 (418h48m56.578514224s) [noise=true , address=77.90.31.16:4740 , piece_id=KYCKJ7F7CNIXIVAZZ7CQPWAYNCMIM5RKO4IKXETP5EZNOAGKGONA, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.707910952 (418h48m56.578993997s) [noise=true , address=45.91.116.5:4330 , piece_id=EMSR7LFH5Q6R7YAJNH45TDQS2E4KX25Q335IFFUORN5AJJWS7W5Q, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.708492299 (418h48m56.579575344s) [noise=true , address=64.49.13.139:4478 , piece_id=QUBJUJHQ6AYU6L67666PA7BHCRL2IIFOJDISOISTEK4JWABGMXIQ, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.708774406 (418h48m56.579857441s) [noise=true , address=45.91.116.14:4213 , piece_id=VLD3RNN2RLOQI2AKMHNY3G57QJPDNJ3BA5JDRRJHCHNHS6YE52JA, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:18:00.137346621 (418h49m0.008429666s) [upload=367702]: Next returned nil segment Dec 23 02:18:00.137360507 (418h49m0.008443552s) [upload=367702]: Waiting for error group managing segments... Dec 23 02:18:01.750516581 (418h49m1.621599616s) [noise=true , address=64.49.13.137:4500 , piece_id=RFGLMWE4OHC7RL2NBPX6VRKG4YGRVCIMBHQFW2W5XAEE376UTVVA, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:02.125124405 (418h49m1.99620745s) [noise=true , address=77.90.31.18:4815 , piece_id=XQDHXOY2ZQQHZZHMGLO5GKZOUJZFKKWGLDJRHJLWCFEX6PKNNHWA, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:02.226869841 (418h49m2.097952876s) [noise=true , address=45.91.116.8:4060 , piece_id=TRNAVEFSFXMIN3DFMQ2DUAKKDTSZH36TCSYZXFKALUM7TCKM3Y6A, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:05.457222504 (418h49m5.328305539s) [noise=true , address=77.90.31.13:4651 , piece_id=H6MCF4TXI5WFQ2BBCPXL3OX6OA4NCRBIH6IBLMPYPIJBETEN6UGA, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:06.079641082 (418h49m5.950724128s) [noise=true , address=45.91.116.12:4257 , piece_id=SPTNFMB3LXNY74LCASNSAZIPCOAQHO5V6SOFP27Y4IY25CHWBO2A, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:09.265548070 (418h49m9.136631105s) [noise=true , address=64.49.13.139:4478 , piece_id=QUBJUJHQ6AYU6L67666PA7BHCRL2IIFOJDISOISTEK4JWABGMXIQ, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:10.880784395 (418h49m10.751867441s) [noise=true , address=45.91.116.14:4213 , piece_id=VLD3RNN2RLOQI2AKMHNY3G57QJPDNJ3BA5JDRRJHCHNHS6YE52JA, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:12.686956874 (418h49m12.558039929s) [noise=true , address=145.79.227.28:4742 , piece_id=3IMQS3J42XXGHCZFMBYPDCPCJYRY3I3EUIQ2Z2JOK4TPXQQH75OQ, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:12.832920510 (418h49m12.704003545s) [noise=true , address=45.91.116.5:4330 , piece_id=EMSR7LFH5Q6R7YAJNH45TDQS2E4KX25Q335IFFUORN5AJJWS7W5Q, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:22.466644063 (418h49m22.337727098s) [noise=true , address=77.90.31.16:4740 , piece_id=KYCKJ7F7CNIXIVAZZ7CQPWAYNCMIM5RKO4IKXETP5EZNOAGKGONA, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:34.649740548 (418h49m34.520823583s) [noise=true , address=64.49.13.169:4715 , piece_id=6RCBEKXRJ56IGXWT4PVSTDYRIH3UCAD22C4Z5BU33U5YIPWBEPDQ, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:38.250548484 (418h49m38.121631529s) [noise=true , address=145.79.227.17:4984 , piece_id=DAMTMQIEBM3I2QZTWVEGISFJ5T7I3LMZ3CTXUFXO2CS775GY4F7Q, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:43.329219400 (418h49m43.200302445s) [noise=true , address=145.79.227.23:4549 , piece_id=XH7BPLOCXKVTJQDYKIRKFXVLHIXXNJFAXCFZPSSASWLOG3GM5PEQ, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:43.400788824 (418h49m43.271871859s) [noise=true , address=145.79.227.32:4848 , piece_id=NMEVAZ6HHKE2Z4Q5G4VLNPOIURXE5TXXUMLFJLURZUI4Z62W7VMA, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:43.400819082 (418h49m43.271902117s) [seg_pos={0 0}, upload=367702]: Segment reached optimal threshold of 14 pieces. Dec 23 02:18:43.401090247 (418h49m43.272173302s) [noise=true , address=5.231.73.4:4030 , piece_id=M6FJSRGUNCEIBMNKLBK7HHFL4DA3PDD2MZ2GN2MJKAFSH3XRI7XQ, seg_pos={0 0}, upload=367702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rWJM3vC8XrUAfVaoqVZm1qzqcG8AqxLzncExPHEak5oo3QuPA): protocol: expected piece hash; context canceled; EOF Dec 23 02:18:43.401100788 (418h49m43.272183823s) [noise=true , address=5.231.73.4:4030 , piece_id=M6FJSRGUNCEIBMNKLBK7HHFL4DA3PDD2MZ2GN2MJKAFSH3XRI7XQ, seg_pos={0 0}, upload=367702]: Long tail cancellation due to optimal threshold Dec 23 02:18:43.401281792 (418h49m43.272364837s) [noise=true , address=77.90.31.14:4667 , piece_id=5ZXKL7BLGSD3TCZNAHNK7STV2RKAZQWCHN6PROBXAPTA5ZBCPT6A, seg_pos={0 0}, upload=367702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S): protocol: expected piece hash; context canceled; EOF Dec 23 02:18:43.401290889 (418h49m43.272373925s) [noise=true , address=77.90.31.14:4667 , piece_id=5ZXKL7BLGSD3TCZNAHNK7STV2RKAZQWCHN6PROBXAPTA5ZBCPT6A, seg_pos={0 0}, upload=367702]: Long tail cancellation due to optimal threshold Dec 23 02:18:43.401412151 (418h49m43.272495186s) [noise=true , address=145.79.227.13:4811 , piece_id=KVHW3GEE4KWLOOUJBRVOOLDA3EGPRUU7VHPH7TQXLURUCCU5J77Q, seg_pos={0 0}, upload=367702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TxjP4CnxyU4t2DTNBjBMBxnizcHCD42fMrB5Da6Qiqg1Eg17V): protocol: context canceled; context canceled Dec 23 02:18:43.401418663 (418h49m43.272501698s) [noise=true , address=145.79.227.13:4811 , piece_id=KVHW3GEE4KWLOOUJBRVOOLDA3EGPRUU7VHPH7TQXLURUCCU5J77Q, seg_pos={0 0}, upload=367702]: Long tail cancellation due to optimal threshold Dec 23 02:18:43.401630526 (418h49m43.272713561s) [noise=true , address=145.79.227.30:4796 , piece_id=IF6NPIC5DTTMJ4DZMIDD6SFN7TGDBN6RD3H7KT7A4OC7VJVCEZOA, seg_pos={0 0}, upload=367702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: read tcp 188.166.225.170:44484->145.79.227.30:4796: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Dec 23 02:18:43.401638241 (418h49m43.272721276s) [noise=true , address=145.79.227.30:4796 , piece_id=IF6NPIC5DTTMJ4DZMIDD6SFN7TGDBN6RD3H7KT7A4OC7VJVCEZOA, seg_pos={0 0}, upload=367702]: Long tail cancellation due to optimal threshold Dec 23 02:18:43.401657227 (418h49m43.272740273s) [seg_pos={0 0}, upload=367702]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 23 02:18:43.401864953 (418h49m43.272947988s) [upload=367702]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 02:18:43.401870614 (418h49m43.272953649s) [upload=367702]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 02:18:43.703068278 (418h49m43.574151323s) [upload=367702]: Done uploading segments... Dec 23 02:18:43.703144954 (418h49m43.574227999s) [upload=367702]: Upload finished. err: <nil> Duration: 8.124507006s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................................................X.....] duration: 3.474430101s trace: https://jaeger.dev.storj.io/trace/7f7fb60dfbab7793 FAIL (2025-12-23 02:21:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.474459818s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 5.832757148s trace: https://jaeger.dev.storj.io/trace/463d97d26edafa8a FAIL (2025-12-23 02:21:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=367700]: Uploading piece... Dec 23 02:17:52.581508683 (418h48m52.452591718s) [noise=true , address=185.167.204.206:4040 , piece_id=Y6HVWGUESE2BSEIKEMDSLNHWDTOEKGBR76NIFRRQU6KFTVHD4Q7Q, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.581711800 (418h48m52.452794835s) [noise=true , address=185.167.204.206:4058 , piece_id=4CZUKD7KF6H3JH6DLWT3KM2PQC72NRYW4YPN7NL445VBER4QR6SA, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.581958349 (418h48m52.453041384s) [noise=true , address=89.149.201.147:4016 , piece_id=MFNURTZ2HJFACIAHZUO7YZJXRJZB26AVF24JLU3I7IYKDPZ2JJIQ, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.582064401 (418h48m52.453147436s) [noise=true , address=89.149.201.147:4009 , piece_id=WGVNGAHNZIVLBR665VB7GO77EZ7SSJOC5P7ML2UAK6MKV4BAF27A, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.582336870 (418h48m52.453419905s) [noise=true , address=81.171.14.70:4025 , piece_id=PFRRLSNJYBUCYSOVOE3VHDWAKTEPMPQRTE4U2VTPBT7KGYVNQHIQ, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.582391083 (418h48m52.453474118s) [noise=true , address=81.171.14.70:4029 , piece_id=JKKAJ3PXL7WDZ7XNKM752C6NA4B4FUN2AZJCGAZPDVNAGKTAOK7Q, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.582715682 (418h48m52.453798717s) [noise=true , address=89.149.201.147:4002 , piece_id=QCDLIQPY7COKADVRRRUN7UBEIV2AARIPA64LMFCCYPFNA54TA7VQ, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.582847693 (418h48m52.453930738s) [noise=true , address=185.167.204.205:4033 , piece_id=42CCWQIIXKFWWCLUSLGSFQHZMA2UPGZJMOSWD7Z562DV6ABPDVRQ, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.583274976 (418h48m52.454358022s) [noise=true , address=185.167.204.206:4066 , piece_id=WAVNL7LBCOQEW3CFA3DWRITRTBGCWYVASH43KC4RFHE5KFQUQ45A, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.583662655 (418h48m52.45474569s) [noise=true , address=185.167.204.206:4070 , piece_id=GA3GZFHS36RXW4PWKSHZ2QOAXQJVNIWVMED3XAAGG5GQPA27ST7A, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:53.405090439 (418h48m53.276173484s) [upload=367700]: Next returned nil segment Dec 23 02:17:53.405105007 (418h48m53.276188042s) [upload=367700]: Waiting for error group managing segments... Dec 23 02:17:53.798794305 (418h48m53.66987735s) [noise=true , address=185.167.204.205:4025 , piece_id=XY5XNAAQZFPKWJP7V5LWBBH5ZQ7XFUTQ657QEF5KXFTP2EKCSIBA, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.799500561 (418h48m53.670583596s) [noise=true , address=185.167.204.205:4016 , piece_id=523RGOAK7OI6VYQY5X4XT5Y6F5C7NMB2WDXPN74Q27P3BCH2Q37A, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.799686735 (418h48m53.67076977s) [noise=true , address=185.167.204.205:4036 , piece_id=IQ6I5X4HC2ANQSM72N7X4OJWVTUMOWH5YPSSRHUFTGYSXW5ZK4WQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.800348164 (418h48m53.671431209s) [noise=true , address=185.167.204.205:4033 , piece_id=42CCWQIIXKFWWCLUSLGSFQHZMA2UPGZJMOSWD7Z562DV6ABPDVRQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.802008737 (418h48m53.673091782s) [noise=true , address=185.167.204.206:4057 , piece_id=UNUF4NZ5CL74EWLPZLVX44MAYYUQFIAZ54N7FGBLCQHEKLGX6RSQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.802484733 (418h48m53.673567778s) [noise=true , address=185.167.204.206:4040 , piece_id=Y6HVWGUESE2BSEIKEMDSLNHWDTOEKGBR76NIFRRQU6KFTVHD4Q7Q, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.803136594 (418h48m53.674219639s) [noise=true , address=185.167.204.206:4070 , piece_id=GA3GZFHS36RXW4PWKSHZ2QOAXQJVNIWVMED3XAAGG5GQPA27ST7A, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.803750904 (418h48m53.674833939s) [noise=true , address=185.167.204.206:4053 , piece_id=GG5TA7D4KK2IGSKRO4T3APKSZ43IWLND3L54EXYD2LRFIKGI5MRQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.804347520 (418h48m53.675430565s) [noise=true , address=185.167.204.206:4058 , piece_id=4CZUKD7KF6H3JH6DLWT3KM2PQC72NRYW4YPN7NL445VBER4QR6SA, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.805167522 (418h48m53.676250567s) [noise=true , address=185.167.204.206:4054 , piece_id=3QWOKGEC5JODMZCL5R5ULU7EUAKGI2W5QWONT5H2I7EOBH7PVODA, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.805898293 (418h48m53.676981328s) [noise=true , address=185.167.204.206:4066 , piece_id=WAVNL7LBCOQEW3CFA3DWRITRTBGCWYVASH43KC4RFHE5KFQUQ45A, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.808200908 (418h48m53.679283943s) [noise=true , address=185.167.204.206:4064 , piece_id=4K4KD2ZMFXUZT6W3OV7E6BC5UGQHSVZXR67B2QPXIPKGAM6CTVSA, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.808393705 (418h48m53.67947675s) [noise=true , address=185.167.204.205:4004 , piece_id=G5Q7QERF2O3WXIDK5ZAFGSZIN6VXJOFYHAPEC647NIXIJQCSNLKA, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.808844694 (418h48m53.679927739s) [noise=true , address=185.167.204.205:4028 , piece_id=DINID3XD5734Z3RWWS3WPIFLGRUXZMZUNXC2BC3NMX7LWTSDVN3Q, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.810924914 (418h48m53.682007959s) [noise=true , address=185.167.204.206:4047 , piece_id=JSWKSSXVNEFRZPWP56EHG26DRYA7V4K2LPEBCKUOBUG2WET5ZQ7A, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.813330996 (418h48m53.684414031s) [noise=true , address=185.167.204.206:4069 , piece_id=OLTP357X7TH2ABIRWBMAI445KR5SGJVY77PPIQFLRD2KFCBOML2Q, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.813884790 (418h48m53.684967826s) [noise=true , address=185.167.204.205:4001 , piece_id=LMIOF4TXW2JJBYGBNU36M46PXZUNKBKFK56XVNMWRQVR7LSZ3QCQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.815627059 (418h48m53.686710094s) [noise=true , address=185.167.204.206:4067 , piece_id=D24D7H4IJKMKHRJUCQ2ZAYDIVRUYHK2YHEYLVCIL3VA4P5FQJB5A, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:54.448896803 (418h48m54.319979838s) [noise=true , address=81.171.14.70:4029 , piece_id=JKKAJ3PXL7WDZ7XNKM752C6NA4B4FUN2AZJCGAZPDVNAGKTAOK7Q, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:54.835287845 (418h48m54.70637088s) [noise=true , address=81.171.14.70:4025 , piece_id=PFRRLSNJYBUCYSOVOE3VHDWAKTEPMPQRTE4U2VTPBT7KGYVNQHIQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:54.946890399 (418h48m54.817973434s) [noise=true , address=81.171.14.70:4040 , piece_id=6ZIPFOQCGRNK3TDTODGI34TL2WM65BQSRLXXAFX3XQ3VCPAVT2YA, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:54.960812961 (418h48m54.831895996s) [noise=true , address=81.171.14.70:4034 , piece_id=2EA6SNLDV3ZVTNQ5UXEXIC2SJTPH3D6VCVZJI2IAGS22U2J6Y6WQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:55.069208920 (418h48m54.940291965s) [noise=true , address=89.149.201.147:4009 , piece_id=WGVNGAHNZIVLBR665VB7GO77EZ7SSJOC5P7ML2UAK6MKV4BAF27A, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:55.073463732 (418h48m54.944546768s) [noise=true , address=89.149.201.147:4002 , piece_id=QCDLIQPY7COKADVRRRUN7UBEIV2AARIPA64LMFCCYPFNA54TA7VQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:55.132812454 (418h48m55.003895489s) [noise=true , address=89.149.201.147:4010 , piece_id=LZXMVADI5CHNTPWZNXNBEFI3SMGAWQGNUIWDNPM4JMRBLZ62HO4A, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:55.185887030 (418h48m55.056970065s) [noise=true , address=89.149.201.147:4012 , piece_id=2Z64OIS2BBYH6ZCKBXHW4XQZY4FFASHDIHH3C3YVY7MGTTBI6QNA, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:55.185911748 (418h48m55.056994783s) [seg_pos={0 0}, upload=367700]: Segment reached optimal threshold of 26 pieces. Dec 23 02:17:55.186137848 (418h48m55.057220893s) [noise=true , address=89.149.201.147:4015 , piece_id=WENNEH2N2S7NV3TQILPUZLFRBV5CO6VMP52NKDIQE23NO2X7GDWQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RbD6M2MHJfxcJXogZG8nUGFmeZjwUQ1auN6KGWJMPTffdK42K): protocol: expected piece hash; context canceled; EOF Dec 23 02:17:55.186148097 (418h48m55.057231132s) [noise=true , address=89.149.201.147:4015 , piece_id=WENNEH2N2S7NV3TQILPUZLFRBV5CO6VMP52NKDIQE23NO2X7GDWQ, seg_pos={0 0}, upload=367700]: Long tail cancellation due to optimal threshold Dec 23 02:17:55.186212209 (418h48m55.057295244s) [noise=true , address=89.149.201.147:4016 , piece_id=MFNURTZ2HJFACIAHZUO7YZJXRJZB26AVF24JLU3I7IYKDPZ2JJIQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XsvrVhDGkU8BkYsBGmTtax47E43dZAYuGwD8WBMpp3n4GgDM5): protocol: expected piece hash; context canceled; EOF Dec 23 02:17:55.186228150 (418h48m55.057311185s) [noise=true , address=89.149.201.147:4016 , piece_id=MFNURTZ2HJFACIAHZUO7YZJXRJZB26AVF24JLU3I7IYKDPZ2JJIQ, seg_pos={0 0}, upload=367700]: Long tail cancellation due to optimal threshold Dec 23 02:17:55.186247767 (418h48m55.057330802s) [seg_pos={0 0}, upload=367700]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 23 02:17:55.186337368 (418h48m55.057420403s) [upload=367700]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 02:17:55.186342387 (418h48m55.057425422s) [upload=367700]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 02:17:55.430535735 (418h48m55.30161878s) [upload=367700]: Done uploading segments... Dec 23 02:17:55.430557656 (418h48m55.301640701s) [upload=367700]: Upload finished. err: <nil> Duration: 5.832787545s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 5.960303535s trace: https://jaeger.dev.storj.io/trace/61a0842c4fdd78ca FAIL (2025-12-23 02:21:43 UTC) orbiter warning: context deadline exceeded Duration: 5.96032747s edge gateway us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 3.380901048s trace: https://jaeger.dev.storj.io/trace/20c185d15563c01c FAIL (2025-12-23 02:21:42 UTC) orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 39) Duration: 3.380935183s core uplink ap1

Orbiter Check History

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)