Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................................X.X...........] duration: 5.688993027s trace: https://jaeger.dev.storj.io/trace/335d60b5e3cb0611 FAIL (2025-12-23 13:11:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== gments... Dec 23 13:03:41.146773219 (429h34m41.017856254s) [upload=377134]: Got next segment. Inline: false Dec 23 13:03:41.146815550 (429h34m41.017898585s) [upload=377134]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 13:03:41.146820509 (429h34m41.017903544s) [upload=377134]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 13:03:41.395177081 (429h34m41.266260127s) [seg_pos={0 0}, upload=377134]: Begin upload segment... Dec 23 13:03:41.395218049 (429h34m41.266301084s) [seg_pos={0 0}, upload=377134]: Done begin upload segment. Dec 23 13:03:41.395250882 (429h34m41.266333927s) [noise=true , address=45.91.116.12:4226 , piece_id=BN76ETRXTTAEDWXZWR3A4KEOHHVKV4OFMG2GE6UOV7DIBE2WHIZA, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.395268705 (429h34m41.26635175s) [noise=true , address=45.91.116.11:4120 , piece_id=FODNYYBM7M4TD5KQZYKWGSQXULEIPEBPE6BTUPF5AZZUZSLQ7ACQ, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.395658838 (429h34m41.266741873s) [noise=true , address=45.91.116.5:4297 , piece_id=TQXE3Q6QFA7F63DGJTQKNWEHIP2TOVRCPRVGCX6XC57BY7PJB3UQ, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.395663367 (429h34m41.266746413s) [noise=true , address=145.79.227.20:4438 , piece_id=TAOBXUAGOSBITQLEPKPRC7CUW6VMUM3JVR7VJ4Y4V4EFZ4YKQ7HQ, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.395962717 (429h34m41.267045762s) [noise=true , address=77.90.31.13:4629 , piece_id=VX2CVI5FXKWL3W6ZFTRXG5RREY37EBUEQAHSQVPGBTJ63SC7XA4Q, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.396003695 (429h34m41.26708673s) [noise=true , address=145.79.227.52:4446 , piece_id=PU54PSDQBMMYEONFTQT5IDII5K3PU4RAFZ5LMHOOM7YNAJS47OXA, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.396288427 (429h34m41.267371462s) [noise=true , address=145.79.227.11:4731 , piece_id=3N5KV6DMUF4QI2ZD3WB55AMP6X2YAFWCWP7VM4OBFPFB5UZEPGZQ, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.396312643 (429h34m41.267395678s) [noise=true , address=145.79.227.34:4905 , piece_id=D54AOI2BG6TPMWQSOJIQFFPQUQRMNYVFDY4PI7JK2NMFHJZAI23Q, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.396575905 (429h34m41.26765894s) [noise=true , address=145.79.227.29:4942 , piece_id=2HHIYW2ZIIAKDV4CZCO74BAVPK457IVGTE4EJVGANHF65D5W7PNA, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.396676416 (429h34m41.267759461s) [noise=true , address=145.79.227.37:4565 , piece_id=HBJ366WIOAHI6ADVVEEDQQWXUS2E5KPUSFEP2KL4XO2X5LOTFKYQ, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.396865416 (429h34m41.267948451s) [noise=true , address=145.79.227.21:4473 , piece_id=7BR47RIYUFP5FWPABRZROLAAMKGHEPZPL3IDLGQ3XQNKXAMJ6URA, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.397008699 (429h34m41.268091734s) [noise=true , address=45.91.116.6:4976 , piece_id=Y72PN4WXYYPJ2ZNRBR2XA6QMHHTZ6OYOGNCWDBVLZKTO3I4S5KZQ, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.397165558 (429h34m41.268248593s) [noise=true , address=145.79.227.12:4765 , piece_id=NOLXPWRRNQXRZ4OA2KMMX5M6FYJR2S3AOGPDIPVHXHQB5HN4GMZA, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.397266479 (429h34m41.268349514s) [noise=true , address=45.91.116.14:4191 , piece_id=J4OQFX6XPN6GB6RJIVPBZUIBDDPLCWMM6GYGJGT7BAZAUM4DBNCA, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.397521615 (429h34m41.268604661s) [noise=true , address=145.79.227.38:4977 , piece_id=O6QO5Q3IFUTUCDOK7HXTJUYKTDJ5RBWDYB4NGEAMZOHDDR3IIC4A, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.398961146 (429h34m41.270044191s) [noise=true , address=145.79.227.32:4835 , piece_id=ULJPGLR3BQ6H2CGZHO7C5XO2EVN7Q7EY5L4NHRXZ3MQ6U7S64MVA, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:43.108995170 (429h34m42.980078205s) [upload=377134]: Next returned nil segment Dec 23 13:03:43.109012081 (429h34m42.980095146s) [upload=377134]: Waiting for error group managing segments... Dec 23 13:03:43.392567427 (429h34m43.263650462s) [noise=true , address=145.79.227.37:4565 , piece_id=HBJ366WIOAHI6ADVVEEDQQWXUS2E5KPUSFEP2KL4XO2X5LOTFKYQ, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.402738524 (429h34m43.273821559s) [noise=true , address=145.79.227.11:4731 , piece_id=3N5KV6DMUF4QI2ZD3WB55AMP6X2YAFWCWP7VM4OBFPFB5UZEPGZQ, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.403357101 (429h34m43.274440156s) [noise=true , address=145.79.227.38:4977 , piece_id=O6QO5Q3IFUTUCDOK7HXTJUYKTDJ5RBWDYB4NGEAMZOHDDR3IIC4A, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.407110529 (429h34m43.278193564s) [noise=true , address=145.79.227.21:4473 , piece_id=7BR47RIYUFP5FWPABRZROLAAMKGHEPZPL3IDLGQ3XQNKXAMJ6URA, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.407617795 (429h34m43.27870083s) [noise=true , address=145.79.227.29:4942 , piece_id=2HHIYW2ZIIAKDV4CZCO74BAVPK457IVGTE4EJVGANHF65D5W7PNA, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.408029740 (429h34m43.279112775s) [noise=true , address=145.79.227.20:4438 , piece_id=TAOBXUAGOSBITQLEPKPRC7CUW6VMUM3JVR7VJ4Y4V4EFZ4YKQ7HQ, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.412549485 (429h34m43.28363253s) [noise=true , address=145.79.227.52:4446 , piece_id=PU54PSDQBMMYEONFTQT5IDII5K3PU4RAFZ5LMHOOM7YNAJS47OXA, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.432973536 (429h34m43.304056571s) [noise=true , address=145.79.227.34:4905 , piece_id=D54AOI2BG6TPMWQSOJIQFFPQUQRMNYVFDY4PI7JK2NMFHJZAI23Q, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.445500129 (429h34m43.316583164s) [noise=true , address=145.79.227.32:4835 , piece_id=ULJPGLR3BQ6H2CGZHO7C5XO2EVN7Q7EY5L4NHRXZ3MQ6U7S64MVA, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.456746244 (429h34m43.327829279s) [noise=true , address=145.79.227.12:4765 , piece_id=NOLXPWRRNQXRZ4OA2KMMX5M6FYJR2S3AOGPDIPVHXHQB5HN4GMZA, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.644894828 (429h34m43.515977873s) [noise=true , address=45.91.116.5:4297 , piece_id=TQXE3Q6QFA7F63DGJTQKNWEHIP2TOVRCPRVGCX6XC57BY7PJB3UQ, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.713352172 (429h34m43.584435217s) [noise=true , address=45.91.116.6:4976 , piece_id=Y72PN4WXYYPJ2ZNRBR2XA6QMHHTZ6OYOGNCWDBVLZKTO3I4S5KZQ, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.731354943 (429h34m43.602437988s) [noise=true , address=45.91.116.14:4191 , piece_id=J4OQFX6XPN6GB6RJIVPBZUIBDDPLCWMM6GYGJGT7BAZAUM4DBNCA, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.731377275 (429h34m43.602460311s) [seg_pos={0 0}, upload=377134]: Segment reached optimal threshold of 13 pieces. Dec 23 13:03:43.731577968 (429h34m43.602661013s) [noise=true , address=77.90.31.13:4629 , piece_id=VX2CVI5FXKWL3W6ZFTRXG5RREY37EBUEQAHSQVPGBTJ63SC7XA4Q, seg_pos={0 0}, upload=377134]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: write tcp 188.166.225.170:54770->77.90.31.13:4629: use of closed network connection; write tcp 188.166.225.170:54770->77.90.31.13:4629: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54770->77.90.31.13:4629: use of closed network connection Dec 23 13:03:43.731585332 (429h34m43.602668377s) [noise=true , address=77.90.31.13:4629 , piece_id=VX2CVI5FXKWL3W6ZFTRXG5RREY37EBUEQAHSQVPGBTJ63SC7XA4Q, seg_pos={0 0}, upload=377134]: Long tail cancellation due to optimal threshold Dec 23 13:03:43.731671216 (429h34m43.602754261s) [noise=true , address=45.91.116.12:4226 , piece_id=BN76ETRXTTAEDWXZWR3A4KEOHHVKV4OFMG2GE6UOV7DIBE2WHIZA, seg_pos={0 0}, upload=377134]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 23 13:03:43.731682317 (429h34m43.602765362s) [noise=true , address=45.91.116.12:4226 , piece_id=BN76ETRXTTAEDWXZWR3A4KEOHHVKV4OFMG2GE6UOV7DIBE2WHIZA, seg_pos={0 0}, upload=377134]: Long tail cancellation due to optimal threshold Dec 23 13:03:43.731838263 (429h34m43.602921308s) [noise=true , address=45.91.116.11:4120 , piece_id=FODNYYBM7M4TD5KQZYKWGSQXULEIPEBPE6BTUPF5AZZUZSLQ7ACQ, seg_pos={0 0}, upload=377134]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 188.166.225.170:43872->45.91.116.11:4120: use of closed network connection; write tcp 188.166.225.170:43872->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43872->45.91.116.11:4120: use of closed network connection Dec 23 13:03:43.731847792 (429h34m43.602930837s) [noise=true , address=45.91.116.11:4120 , piece_id=FODNYYBM7M4TD5KQZYKWGSQXULEIPEBPE6BTUPF5AZZUZSLQ7ACQ, seg_pos={0 0}, upload=377134]: Long tail cancellation due to optimal threshold Dec 23 13:03:43.731869032 (429h34m43.602952067s) [seg_pos={0 0}, upload=377134]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 13:03:43.731968902 (429h34m43.603051947s) [upload=377134]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 13:03:43.731975024 (429h34m43.603058069s) [upload=377134]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 13:03:44.027725380 (429h34m43.898808415s) [upload=377134]: Done uploading segments... Dec 23 13:03:44.027744758 (429h34m43.898827793s) [upload=377134]: Upload finished. err: <nil> Duration: 5.689027934s core uplink us-select-1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [...................................X........................] duration: 236.289594ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/73921c1a3ca0437b FAIL (2025-12-23 13:10:48 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 236.340702ms edge linksharing bgp eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................................X.X..........] duration: 5.716601578s trace: https://jaeger.dev.storj.io/trace/335d60b5e3cb0611 FAIL (2025-12-23 13:10:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== gments... Dec 23 13:03:41.146773219 (429h34m41.017856254s) [upload=377134]: Got next segment. Inline: false Dec 23 13:03:41.146815550 (429h34m41.017898585s) [upload=377134]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 13:03:41.146820509 (429h34m41.017903544s) [upload=377134]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 13:03:41.395177081 (429h34m41.266260127s) [seg_pos={0 0}, upload=377134]: Begin upload segment... Dec 23 13:03:41.395218049 (429h34m41.266301084s) [seg_pos={0 0}, upload=377134]: Done begin upload segment. Dec 23 13:03:41.395250882 (429h34m41.266333927s) [noise=true , address=45.91.116.12:4226 , piece_id=BN76ETRXTTAEDWXZWR3A4KEOHHVKV4OFMG2GE6UOV7DIBE2WHIZA, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.395268705 (429h34m41.26635175s) [noise=true , address=45.91.116.11:4120 , piece_id=FODNYYBM7M4TD5KQZYKWGSQXULEIPEBPE6BTUPF5AZZUZSLQ7ACQ, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.395658838 (429h34m41.266741873s) [noise=true , address=45.91.116.5:4297 , piece_id=TQXE3Q6QFA7F63DGJTQKNWEHIP2TOVRCPRVGCX6XC57BY7PJB3UQ, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.395663367 (429h34m41.266746413s) [noise=true , address=145.79.227.20:4438 , piece_id=TAOBXUAGOSBITQLEPKPRC7CUW6VMUM3JVR7VJ4Y4V4EFZ4YKQ7HQ, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.395962717 (429h34m41.267045762s) [noise=true , address=77.90.31.13:4629 , piece_id=VX2CVI5FXKWL3W6ZFTRXG5RREY37EBUEQAHSQVPGBTJ63SC7XA4Q, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.396003695 (429h34m41.26708673s) [noise=true , address=145.79.227.52:4446 , piece_id=PU54PSDQBMMYEONFTQT5IDII5K3PU4RAFZ5LMHOOM7YNAJS47OXA, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.396288427 (429h34m41.267371462s) [noise=true , address=145.79.227.11:4731 , piece_id=3N5KV6DMUF4QI2ZD3WB55AMP6X2YAFWCWP7VM4OBFPFB5UZEPGZQ, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.396312643 (429h34m41.267395678s) [noise=true , address=145.79.227.34:4905 , piece_id=D54AOI2BG6TPMWQSOJIQFFPQUQRMNYVFDY4PI7JK2NMFHJZAI23Q, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.396575905 (429h34m41.26765894s) [noise=true , address=145.79.227.29:4942 , piece_id=2HHIYW2ZIIAKDV4CZCO74BAVPK457IVGTE4EJVGANHF65D5W7PNA, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.396676416 (429h34m41.267759461s) [noise=true , address=145.79.227.37:4565 , piece_id=HBJ366WIOAHI6ADVVEEDQQWXUS2E5KPUSFEP2KL4XO2X5LOTFKYQ, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.396865416 (429h34m41.267948451s) [noise=true , address=145.79.227.21:4473 , piece_id=7BR47RIYUFP5FWPABRZROLAAMKGHEPZPL3IDLGQ3XQNKXAMJ6URA, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.397008699 (429h34m41.268091734s) [noise=true , address=45.91.116.6:4976 , piece_id=Y72PN4WXYYPJ2ZNRBR2XA6QMHHTZ6OYOGNCWDBVLZKTO3I4S5KZQ, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.397165558 (429h34m41.268248593s) [noise=true , address=145.79.227.12:4765 , piece_id=NOLXPWRRNQXRZ4OA2KMMX5M6FYJR2S3AOGPDIPVHXHQB5HN4GMZA, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.397266479 (429h34m41.268349514s) [noise=true , address=45.91.116.14:4191 , piece_id=J4OQFX6XPN6GB6RJIVPBZUIBDDPLCWMM6GYGJGT7BAZAUM4DBNCA, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.397521615 (429h34m41.268604661s) [noise=true , address=145.79.227.38:4977 , piece_id=O6QO5Q3IFUTUCDOK7HXTJUYKTDJ5RBWDYB4NGEAMZOHDDR3IIC4A, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.398961146 (429h34m41.270044191s) [noise=true , address=145.79.227.32:4835 , piece_id=ULJPGLR3BQ6H2CGZHO7C5XO2EVN7Q7EY5L4NHRXZ3MQ6U7S64MVA, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:43.108995170 (429h34m42.980078205s) [upload=377134]: Next returned nil segment Dec 23 13:03:43.109012081 (429h34m42.980095146s) [upload=377134]: Waiting for error group managing segments... Dec 23 13:03:43.392567427 (429h34m43.263650462s) [noise=true , address=145.79.227.37:4565 , piece_id=HBJ366WIOAHI6ADVVEEDQQWXUS2E5KPUSFEP2KL4XO2X5LOTFKYQ, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.402738524 (429h34m43.273821559s) [noise=true , address=145.79.227.11:4731 , piece_id=3N5KV6DMUF4QI2ZD3WB55AMP6X2YAFWCWP7VM4OBFPFB5UZEPGZQ, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.403357101 (429h34m43.274440156s) [noise=true , address=145.79.227.38:4977 , piece_id=O6QO5Q3IFUTUCDOK7HXTJUYKTDJ5RBWDYB4NGEAMZOHDDR3IIC4A, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.407110529 (429h34m43.278193564s) [noise=true , address=145.79.227.21:4473 , piece_id=7BR47RIYUFP5FWPABRZROLAAMKGHEPZPL3IDLGQ3XQNKXAMJ6URA, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.407617795 (429h34m43.27870083s) [noise=true , address=145.79.227.29:4942 , piece_id=2HHIYW2ZIIAKDV4CZCO74BAVPK457IVGTE4EJVGANHF65D5W7PNA, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.408029740 (429h34m43.279112775s) [noise=true , address=145.79.227.20:4438 , piece_id=TAOBXUAGOSBITQLEPKPRC7CUW6VMUM3JVR7VJ4Y4V4EFZ4YKQ7HQ, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.412549485 (429h34m43.28363253s) [noise=true , address=145.79.227.52:4446 , piece_id=PU54PSDQBMMYEONFTQT5IDII5K3PU4RAFZ5LMHOOM7YNAJS47OXA, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.432973536 (429h34m43.304056571s) [noise=true , address=145.79.227.34:4905 , piece_id=D54AOI2BG6TPMWQSOJIQFFPQUQRMNYVFDY4PI7JK2NMFHJZAI23Q, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.445500129 (429h34m43.316583164s) [noise=true , address=145.79.227.32:4835 , piece_id=ULJPGLR3BQ6H2CGZHO7C5XO2EVN7Q7EY5L4NHRXZ3MQ6U7S64MVA, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.456746244 (429h34m43.327829279s) [noise=true , address=145.79.227.12:4765 , piece_id=NOLXPWRRNQXRZ4OA2KMMX5M6FYJR2S3AOGPDIPVHXHQB5HN4GMZA, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.644894828 (429h34m43.515977873s) [noise=true , address=45.91.116.5:4297 , piece_id=TQXE3Q6QFA7F63DGJTQKNWEHIP2TOVRCPRVGCX6XC57BY7PJB3UQ, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.713352172 (429h34m43.584435217s) [noise=true , address=45.91.116.6:4976 , piece_id=Y72PN4WXYYPJ2ZNRBR2XA6QMHHTZ6OYOGNCWDBVLZKTO3I4S5KZQ, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.731354943 (429h34m43.602437988s) [noise=true , address=45.91.116.14:4191 , piece_id=J4OQFX6XPN6GB6RJIVPBZUIBDDPLCWMM6GYGJGT7BAZAUM4DBNCA, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.731377275 (429h34m43.602460311s) [seg_pos={0 0}, upload=377134]: Segment reached optimal threshold of 13 pieces. Dec 23 13:03:43.731577968 (429h34m43.602661013s) [noise=true , address=77.90.31.13:4629 , piece_id=VX2CVI5FXKWL3W6ZFTRXG5RREY37EBUEQAHSQVPGBTJ63SC7XA4Q, seg_pos={0 0}, upload=377134]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: write tcp 188.166.225.170:54770->77.90.31.13:4629: use of closed network connection; write tcp 188.166.225.170:54770->77.90.31.13:4629: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54770->77.90.31.13:4629: use of closed network connection Dec 23 13:03:43.731585332 (429h34m43.602668377s) [noise=true , address=77.90.31.13:4629 , piece_id=VX2CVI5FXKWL3W6ZFTRXG5RREY37EBUEQAHSQVPGBTJ63SC7XA4Q, seg_pos={0 0}, upload=377134]: Long tail cancellation due to optimal threshold Dec 23 13:03:43.731671216 (429h34m43.602754261s) [noise=true , address=45.91.116.12:4226 , piece_id=BN76ETRXTTAEDWXZWR3A4KEOHHVKV4OFMG2GE6UOV7DIBE2WHIZA, seg_pos={0 0}, upload=377134]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 23 13:03:43.731682317 (429h34m43.602765362s) [noise=true , address=45.91.116.12:4226 , piece_id=BN76ETRXTTAEDWXZWR3A4KEOHHVKV4OFMG2GE6UOV7DIBE2WHIZA, seg_pos={0 0}, upload=377134]: Long tail cancellation due to optimal threshold Dec 23 13:03:43.731838263 (429h34m43.602921308s) [noise=true , address=45.91.116.11:4120 , piece_id=FODNYYBM7M4TD5KQZYKWGSQXULEIPEBPE6BTUPF5AZZUZSLQ7ACQ, seg_pos={0 0}, upload=377134]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 188.166.225.170:43872->45.91.116.11:4120: use of closed network connection; write tcp 188.166.225.170:43872->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43872->45.91.116.11:4120: use of closed network connection Dec 23 13:03:43.731847792 (429h34m43.602930837s) [noise=true , address=45.91.116.11:4120 , piece_id=FODNYYBM7M4TD5KQZYKWGSQXULEIPEBPE6BTUPF5AZZUZSLQ7ACQ, seg_pos={0 0}, upload=377134]: Long tail cancellation due to optimal threshold Dec 23 13:03:43.731869032 (429h34m43.602952067s) [seg_pos={0 0}, upload=377134]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 13:03:43.731968902 (429h34m43.603051947s) [upload=377134]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 13:03:43.731975024 (429h34m43.603058069s) [upload=377134]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 13:03:44.027725380 (429h34m43.898808415s) [upload=377134]: Done uploading segments... Dec 23 13:03:44.027744758 (429h34m43.898827793s) [upload=377134]: Upload finished. err: <nil> Duration: 5.716630525s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................................X.X.........] duration: 6.140981496s trace: https://jaeger.dev.storj.io/trace/335d60b5e3cb0611 FAIL (2025-12-23 13:10:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== gments... Dec 23 13:03:41.146773219 (429h34m41.017856254s) [upload=377134]: Got next segment. Inline: false Dec 23 13:03:41.146815550 (429h34m41.017898585s) [upload=377134]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 13:03:41.146820509 (429h34m41.017903544s) [upload=377134]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 13:03:41.395177081 (429h34m41.266260127s) [seg_pos={0 0}, upload=377134]: Begin upload segment... Dec 23 13:03:41.395218049 (429h34m41.266301084s) [seg_pos={0 0}, upload=377134]: Done begin upload segment. Dec 23 13:03:41.395250882 (429h34m41.266333927s) [noise=true , address=45.91.116.12:4226 , piece_id=BN76ETRXTTAEDWXZWR3A4KEOHHVKV4OFMG2GE6UOV7DIBE2WHIZA, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.395268705 (429h34m41.26635175s) [noise=true , address=45.91.116.11:4120 , piece_id=FODNYYBM7M4TD5KQZYKWGSQXULEIPEBPE6BTUPF5AZZUZSLQ7ACQ, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.395658838 (429h34m41.266741873s) [noise=true , address=45.91.116.5:4297 , piece_id=TQXE3Q6QFA7F63DGJTQKNWEHIP2TOVRCPRVGCX6XC57BY7PJB3UQ, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.395663367 (429h34m41.266746413s) [noise=true , address=145.79.227.20:4438 , piece_id=TAOBXUAGOSBITQLEPKPRC7CUW6VMUM3JVR7VJ4Y4V4EFZ4YKQ7HQ, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.395962717 (429h34m41.267045762s) [noise=true , address=77.90.31.13:4629 , piece_id=VX2CVI5FXKWL3W6ZFTRXG5RREY37EBUEQAHSQVPGBTJ63SC7XA4Q, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.396003695 (429h34m41.26708673s) [noise=true , address=145.79.227.52:4446 , piece_id=PU54PSDQBMMYEONFTQT5IDII5K3PU4RAFZ5LMHOOM7YNAJS47OXA, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.396288427 (429h34m41.267371462s) [noise=true , address=145.79.227.11:4731 , piece_id=3N5KV6DMUF4QI2ZD3WB55AMP6X2YAFWCWP7VM4OBFPFB5UZEPGZQ, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.396312643 (429h34m41.267395678s) [noise=true , address=145.79.227.34:4905 , piece_id=D54AOI2BG6TPMWQSOJIQFFPQUQRMNYVFDY4PI7JK2NMFHJZAI23Q, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.396575905 (429h34m41.26765894s) [noise=true , address=145.79.227.29:4942 , piece_id=2HHIYW2ZIIAKDV4CZCO74BAVPK457IVGTE4EJVGANHF65D5W7PNA, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.396676416 (429h34m41.267759461s) [noise=true , address=145.79.227.37:4565 , piece_id=HBJ366WIOAHI6ADVVEEDQQWXUS2E5KPUSFEP2KL4XO2X5LOTFKYQ, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.396865416 (429h34m41.267948451s) [noise=true , address=145.79.227.21:4473 , piece_id=7BR47RIYUFP5FWPABRZROLAAMKGHEPZPL3IDLGQ3XQNKXAMJ6URA, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.397008699 (429h34m41.268091734s) [noise=true , address=45.91.116.6:4976 , piece_id=Y72PN4WXYYPJ2ZNRBR2XA6QMHHTZ6OYOGNCWDBVLZKTO3I4S5KZQ, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.397165558 (429h34m41.268248593s) [noise=true , address=145.79.227.12:4765 , piece_id=NOLXPWRRNQXRZ4OA2KMMX5M6FYJR2S3AOGPDIPVHXHQB5HN4GMZA, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.397266479 (429h34m41.268349514s) [noise=true , address=45.91.116.14:4191 , piece_id=J4OQFX6XPN6GB6RJIVPBZUIBDDPLCWMM6GYGJGT7BAZAUM4DBNCA, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.397521615 (429h34m41.268604661s) [noise=true , address=145.79.227.38:4977 , piece_id=O6QO5Q3IFUTUCDOK7HXTJUYKTDJ5RBWDYB4NGEAMZOHDDR3IIC4A, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.398961146 (429h34m41.270044191s) [noise=true , address=145.79.227.32:4835 , piece_id=ULJPGLR3BQ6H2CGZHO7C5XO2EVN7Q7EY5L4NHRXZ3MQ6U7S64MVA, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:43.108995170 (429h34m42.980078205s) [upload=377134]: Next returned nil segment Dec 23 13:03:43.109012081 (429h34m42.980095146s) [upload=377134]: Waiting for error group managing segments... Dec 23 13:03:43.392567427 (429h34m43.263650462s) [noise=true , address=145.79.227.37:4565 , piece_id=HBJ366WIOAHI6ADVVEEDQQWXUS2E5KPUSFEP2KL4XO2X5LOTFKYQ, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.402738524 (429h34m43.273821559s) [noise=true , address=145.79.227.11:4731 , piece_id=3N5KV6DMUF4QI2ZD3WB55AMP6X2YAFWCWP7VM4OBFPFB5UZEPGZQ, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.403357101 (429h34m43.274440156s) [noise=true , address=145.79.227.38:4977 , piece_id=O6QO5Q3IFUTUCDOK7HXTJUYKTDJ5RBWDYB4NGEAMZOHDDR3IIC4A, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.407110529 (429h34m43.278193564s) [noise=true , address=145.79.227.21:4473 , piece_id=7BR47RIYUFP5FWPABRZROLAAMKGHEPZPL3IDLGQ3XQNKXAMJ6URA, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.407617795 (429h34m43.27870083s) [noise=true , address=145.79.227.29:4942 , piece_id=2HHIYW2ZIIAKDV4CZCO74BAVPK457IVGTE4EJVGANHF65D5W7PNA, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.408029740 (429h34m43.279112775s) [noise=true , address=145.79.227.20:4438 , piece_id=TAOBXUAGOSBITQLEPKPRC7CUW6VMUM3JVR7VJ4Y4V4EFZ4YKQ7HQ, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.412549485 (429h34m43.28363253s) [noise=true , address=145.79.227.52:4446 , piece_id=PU54PSDQBMMYEONFTQT5IDII5K3PU4RAFZ5LMHOOM7YNAJS47OXA, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.432973536 (429h34m43.304056571s) [noise=true , address=145.79.227.34:4905 , piece_id=D54AOI2BG6TPMWQSOJIQFFPQUQRMNYVFDY4PI7JK2NMFHJZAI23Q, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.445500129 (429h34m43.316583164s) [noise=true , address=145.79.227.32:4835 , piece_id=ULJPGLR3BQ6H2CGZHO7C5XO2EVN7Q7EY5L4NHRXZ3MQ6U7S64MVA, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.456746244 (429h34m43.327829279s) [noise=true , address=145.79.227.12:4765 , piece_id=NOLXPWRRNQXRZ4OA2KMMX5M6FYJR2S3AOGPDIPVHXHQB5HN4GMZA, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.644894828 (429h34m43.515977873s) [noise=true , address=45.91.116.5:4297 , piece_id=TQXE3Q6QFA7F63DGJTQKNWEHIP2TOVRCPRVGCX6XC57BY7PJB3UQ, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.713352172 (429h34m43.584435217s) [noise=true , address=45.91.116.6:4976 , piece_id=Y72PN4WXYYPJ2ZNRBR2XA6QMHHTZ6OYOGNCWDBVLZKTO3I4S5KZQ, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.731354943 (429h34m43.602437988s) [noise=true , address=45.91.116.14:4191 , piece_id=J4OQFX6XPN6GB6RJIVPBZUIBDDPLCWMM6GYGJGT7BAZAUM4DBNCA, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.731377275 (429h34m43.602460311s) [seg_pos={0 0}, upload=377134]: Segment reached optimal threshold of 13 pieces. Dec 23 13:03:43.731577968 (429h34m43.602661013s) [noise=true , address=77.90.31.13:4629 , piece_id=VX2CVI5FXKWL3W6ZFTRXG5RREY37EBUEQAHSQVPGBTJ63SC7XA4Q, seg_pos={0 0}, upload=377134]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: write tcp 188.166.225.170:54770->77.90.31.13:4629: use of closed network connection; write tcp 188.166.225.170:54770->77.90.31.13:4629: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54770->77.90.31.13:4629: use of closed network connection Dec 23 13:03:43.731585332 (429h34m43.602668377s) [noise=true , address=77.90.31.13:4629 , piece_id=VX2CVI5FXKWL3W6ZFTRXG5RREY37EBUEQAHSQVPGBTJ63SC7XA4Q, seg_pos={0 0}, upload=377134]: Long tail cancellation due to optimal threshold Dec 23 13:03:43.731671216 (429h34m43.602754261s) [noise=true , address=45.91.116.12:4226 , piece_id=BN76ETRXTTAEDWXZWR3A4KEOHHVKV4OFMG2GE6UOV7DIBE2WHIZA, seg_pos={0 0}, upload=377134]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 23 13:03:43.731682317 (429h34m43.602765362s) [noise=true , address=45.91.116.12:4226 , piece_id=BN76ETRXTTAEDWXZWR3A4KEOHHVKV4OFMG2GE6UOV7DIBE2WHIZA, seg_pos={0 0}, upload=377134]: Long tail cancellation due to optimal threshold Dec 23 13:03:43.731838263 (429h34m43.602921308s) [noise=true , address=45.91.116.11:4120 , piece_id=FODNYYBM7M4TD5KQZYKWGSQXULEIPEBPE6BTUPF5AZZUZSLQ7ACQ, seg_pos={0 0}, upload=377134]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 188.166.225.170:43872->45.91.116.11:4120: use of closed network connection; write tcp 188.166.225.170:43872->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43872->45.91.116.11:4120: use of closed network connection Dec 23 13:03:43.731847792 (429h34m43.602930837s) [noise=true , address=45.91.116.11:4120 , piece_id=FODNYYBM7M4TD5KQZYKWGSQXULEIPEBPE6BTUPF5AZZUZSLQ7ACQ, seg_pos={0 0}, upload=377134]: Long tail cancellation due to optimal threshold Dec 23 13:03:43.731869032 (429h34m43.602952067s) [seg_pos={0 0}, upload=377134]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 13:03:43.731968902 (429h34m43.603051947s) [upload=377134]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 13:03:43.731975024 (429h34m43.603058069s) [upload=377134]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 13:03:44.027725380 (429h34m43.898808415s) [upload=377134]: Done uploading segments... Dec 23 13:03:44.027744758 (429h34m43.898827793s) [upload=377134]: Upload finished. err: <nil> Duration: 6.141040268s core uplink us-select-1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [....................................X.......................] duration: 256.794909ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/73921c1a3ca0437b FAIL (2025-12-23 13:09:48 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 256.817102ms edge linksharing bgp eu1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [.....................................X......................] duration: 260.586298ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/73921c1a3ca0437b FAIL (2025-12-23 13:09:03 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 260.638226ms edge linksharing bgp eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................................X.X........] duration: 6.719165123s trace: https://jaeger.dev.storj.io/trace/335d60b5e3cb0611 FAIL (2025-12-23 13:09:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== gments... Dec 23 13:03:41.146773219 (429h34m41.017856254s) [upload=377134]: Got next segment. Inline: false Dec 23 13:03:41.146815550 (429h34m41.017898585s) [upload=377134]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 13:03:41.146820509 (429h34m41.017903544s) [upload=377134]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 13:03:41.395177081 (429h34m41.266260127s) [seg_pos={0 0}, upload=377134]: Begin upload segment... Dec 23 13:03:41.395218049 (429h34m41.266301084s) [seg_pos={0 0}, upload=377134]: Done begin upload segment. Dec 23 13:03:41.395250882 (429h34m41.266333927s) [noise=true , address=45.91.116.12:4226 , piece_id=BN76ETRXTTAEDWXZWR3A4KEOHHVKV4OFMG2GE6UOV7DIBE2WHIZA, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.395268705 (429h34m41.26635175s) [noise=true , address=45.91.116.11:4120 , piece_id=FODNYYBM7M4TD5KQZYKWGSQXULEIPEBPE6BTUPF5AZZUZSLQ7ACQ, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.395658838 (429h34m41.266741873s) [noise=true , address=45.91.116.5:4297 , piece_id=TQXE3Q6QFA7F63DGJTQKNWEHIP2TOVRCPRVGCX6XC57BY7PJB3UQ, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.395663367 (429h34m41.266746413s) [noise=true , address=145.79.227.20:4438 , piece_id=TAOBXUAGOSBITQLEPKPRC7CUW6VMUM3JVR7VJ4Y4V4EFZ4YKQ7HQ, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.395962717 (429h34m41.267045762s) [noise=true , address=77.90.31.13:4629 , piece_id=VX2CVI5FXKWL3W6ZFTRXG5RREY37EBUEQAHSQVPGBTJ63SC7XA4Q, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.396003695 (429h34m41.26708673s) [noise=true , address=145.79.227.52:4446 , piece_id=PU54PSDQBMMYEONFTQT5IDII5K3PU4RAFZ5LMHOOM7YNAJS47OXA, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.396288427 (429h34m41.267371462s) [noise=true , address=145.79.227.11:4731 , piece_id=3N5KV6DMUF4QI2ZD3WB55AMP6X2YAFWCWP7VM4OBFPFB5UZEPGZQ, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.396312643 (429h34m41.267395678s) [noise=true , address=145.79.227.34:4905 , piece_id=D54AOI2BG6TPMWQSOJIQFFPQUQRMNYVFDY4PI7JK2NMFHJZAI23Q, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.396575905 (429h34m41.26765894s) [noise=true , address=145.79.227.29:4942 , piece_id=2HHIYW2ZIIAKDV4CZCO74BAVPK457IVGTE4EJVGANHF65D5W7PNA, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.396676416 (429h34m41.267759461s) [noise=true , address=145.79.227.37:4565 , piece_id=HBJ366WIOAHI6ADVVEEDQQWXUS2E5KPUSFEP2KL4XO2X5LOTFKYQ, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.396865416 (429h34m41.267948451s) [noise=true , address=145.79.227.21:4473 , piece_id=7BR47RIYUFP5FWPABRZROLAAMKGHEPZPL3IDLGQ3XQNKXAMJ6URA, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.397008699 (429h34m41.268091734s) [noise=true , address=45.91.116.6:4976 , piece_id=Y72PN4WXYYPJ2ZNRBR2XA6QMHHTZ6OYOGNCWDBVLZKTO3I4S5KZQ, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.397165558 (429h34m41.268248593s) [noise=true , address=145.79.227.12:4765 , piece_id=NOLXPWRRNQXRZ4OA2KMMX5M6FYJR2S3AOGPDIPVHXHQB5HN4GMZA, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.397266479 (429h34m41.268349514s) [noise=true , address=45.91.116.14:4191 , piece_id=J4OQFX6XPN6GB6RJIVPBZUIBDDPLCWMM6GYGJGT7BAZAUM4DBNCA, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.397521615 (429h34m41.268604661s) [noise=true , address=145.79.227.38:4977 , piece_id=O6QO5Q3IFUTUCDOK7HXTJUYKTDJ5RBWDYB4NGEAMZOHDDR3IIC4A, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.398961146 (429h34m41.270044191s) [noise=true , address=145.79.227.32:4835 , piece_id=ULJPGLR3BQ6H2CGZHO7C5XO2EVN7Q7EY5L4NHRXZ3MQ6U7S64MVA, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:43.108995170 (429h34m42.980078205s) [upload=377134]: Next returned nil segment Dec 23 13:03:43.109012081 (429h34m42.980095146s) [upload=377134]: Waiting for error group managing segments... Dec 23 13:03:43.392567427 (429h34m43.263650462s) [noise=true , address=145.79.227.37:4565 , piece_id=HBJ366WIOAHI6ADVVEEDQQWXUS2E5KPUSFEP2KL4XO2X5LOTFKYQ, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.402738524 (429h34m43.273821559s) [noise=true , address=145.79.227.11:4731 , piece_id=3N5KV6DMUF4QI2ZD3WB55AMP6X2YAFWCWP7VM4OBFPFB5UZEPGZQ, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.403357101 (429h34m43.274440156s) [noise=true , address=145.79.227.38:4977 , piece_id=O6QO5Q3IFUTUCDOK7HXTJUYKTDJ5RBWDYB4NGEAMZOHDDR3IIC4A, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.407110529 (429h34m43.278193564s) [noise=true , address=145.79.227.21:4473 , piece_id=7BR47RIYUFP5FWPABRZROLAAMKGHEPZPL3IDLGQ3XQNKXAMJ6URA, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.407617795 (429h34m43.27870083s) [noise=true , address=145.79.227.29:4942 , piece_id=2HHIYW2ZIIAKDV4CZCO74BAVPK457IVGTE4EJVGANHF65D5W7PNA, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.408029740 (429h34m43.279112775s) [noise=true , address=145.79.227.20:4438 , piece_id=TAOBXUAGOSBITQLEPKPRC7CUW6VMUM3JVR7VJ4Y4V4EFZ4YKQ7HQ, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.412549485 (429h34m43.28363253s) [noise=true , address=145.79.227.52:4446 , piece_id=PU54PSDQBMMYEONFTQT5IDII5K3PU4RAFZ5LMHOOM7YNAJS47OXA, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.432973536 (429h34m43.304056571s) [noise=true , address=145.79.227.34:4905 , piece_id=D54AOI2BG6TPMWQSOJIQFFPQUQRMNYVFDY4PI7JK2NMFHJZAI23Q, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.445500129 (429h34m43.316583164s) [noise=true , address=145.79.227.32:4835 , piece_id=ULJPGLR3BQ6H2CGZHO7C5XO2EVN7Q7EY5L4NHRXZ3MQ6U7S64MVA, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.456746244 (429h34m43.327829279s) [noise=true , address=145.79.227.12:4765 , piece_id=NOLXPWRRNQXRZ4OA2KMMX5M6FYJR2S3AOGPDIPVHXHQB5HN4GMZA, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.644894828 (429h34m43.515977873s) [noise=true , address=45.91.116.5:4297 , piece_id=TQXE3Q6QFA7F63DGJTQKNWEHIP2TOVRCPRVGCX6XC57BY7PJB3UQ, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.713352172 (429h34m43.584435217s) [noise=true , address=45.91.116.6:4976 , piece_id=Y72PN4WXYYPJ2ZNRBR2XA6QMHHTZ6OYOGNCWDBVLZKTO3I4S5KZQ, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.731354943 (429h34m43.602437988s) [noise=true , address=45.91.116.14:4191 , piece_id=J4OQFX6XPN6GB6RJIVPBZUIBDDPLCWMM6GYGJGT7BAZAUM4DBNCA, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.731377275 (429h34m43.602460311s) [seg_pos={0 0}, upload=377134]: Segment reached optimal threshold of 13 pieces. Dec 23 13:03:43.731577968 (429h34m43.602661013s) [noise=true , address=77.90.31.13:4629 , piece_id=VX2CVI5FXKWL3W6ZFTRXG5RREY37EBUEQAHSQVPGBTJ63SC7XA4Q, seg_pos={0 0}, upload=377134]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: write tcp 188.166.225.170:54770->77.90.31.13:4629: use of closed network connection; write tcp 188.166.225.170:54770->77.90.31.13:4629: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54770->77.90.31.13:4629: use of closed network connection Dec 23 13:03:43.731585332 (429h34m43.602668377s) [noise=true , address=77.90.31.13:4629 , piece_id=VX2CVI5FXKWL3W6ZFTRXG5RREY37EBUEQAHSQVPGBTJ63SC7XA4Q, seg_pos={0 0}, upload=377134]: Long tail cancellation due to optimal threshold Dec 23 13:03:43.731671216 (429h34m43.602754261s) [noise=true , address=45.91.116.12:4226 , piece_id=BN76ETRXTTAEDWXZWR3A4KEOHHVKV4OFMG2GE6UOV7DIBE2WHIZA, seg_pos={0 0}, upload=377134]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 23 13:03:43.731682317 (429h34m43.602765362s) [noise=true , address=45.91.116.12:4226 , piece_id=BN76ETRXTTAEDWXZWR3A4KEOHHVKV4OFMG2GE6UOV7DIBE2WHIZA, seg_pos={0 0}, upload=377134]: Long tail cancellation due to optimal threshold Dec 23 13:03:43.731838263 (429h34m43.602921308s) [noise=true , address=45.91.116.11:4120 , piece_id=FODNYYBM7M4TD5KQZYKWGSQXULEIPEBPE6BTUPF5AZZUZSLQ7ACQ, seg_pos={0 0}, upload=377134]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 188.166.225.170:43872->45.91.116.11:4120: use of closed network connection; write tcp 188.166.225.170:43872->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43872->45.91.116.11:4120: use of closed network connection Dec 23 13:03:43.731847792 (429h34m43.602930837s) [noise=true , address=45.91.116.11:4120 , piece_id=FODNYYBM7M4TD5KQZYKWGSQXULEIPEBPE6BTUPF5AZZUZSLQ7ACQ, seg_pos={0 0}, upload=377134]: Long tail cancellation due to optimal threshold Dec 23 13:03:43.731869032 (429h34m43.602952067s) [seg_pos={0 0}, upload=377134]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 13:03:43.731968902 (429h34m43.603051947s) [upload=377134]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 13:03:43.731975024 (429h34m43.603058069s) [upload=377134]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 13:03:44.027725380 (429h34m43.898808415s) [upload=377134]: Done uploading segments... Dec 23 13:03:44.027744758 (429h34m43.898827793s) [upload=377134]: Upload finished. err: <nil> Duration: 6.719200891s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................................X.X.......] duration: 5.699084051s trace: https://jaeger.dev.storj.io/trace/335d60b5e3cb0611 FAIL (2025-12-23 13:08:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== gments... Dec 23 13:03:41.146773219 (429h34m41.017856254s) [upload=377134]: Got next segment. Inline: false Dec 23 13:03:41.146815550 (429h34m41.017898585s) [upload=377134]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 13:03:41.146820509 (429h34m41.017903544s) [upload=377134]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 13:03:41.395177081 (429h34m41.266260127s) [seg_pos={0 0}, upload=377134]: Begin upload segment... Dec 23 13:03:41.395218049 (429h34m41.266301084s) [seg_pos={0 0}, upload=377134]: Done begin upload segment. Dec 23 13:03:41.395250882 (429h34m41.266333927s) [noise=true , address=45.91.116.12:4226 , piece_id=BN76ETRXTTAEDWXZWR3A4KEOHHVKV4OFMG2GE6UOV7DIBE2WHIZA, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.395268705 (429h34m41.26635175s) [noise=true , address=45.91.116.11:4120 , piece_id=FODNYYBM7M4TD5KQZYKWGSQXULEIPEBPE6BTUPF5AZZUZSLQ7ACQ, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.395658838 (429h34m41.266741873s) [noise=true , address=45.91.116.5:4297 , piece_id=TQXE3Q6QFA7F63DGJTQKNWEHIP2TOVRCPRVGCX6XC57BY7PJB3UQ, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.395663367 (429h34m41.266746413s) [noise=true , address=145.79.227.20:4438 , piece_id=TAOBXUAGOSBITQLEPKPRC7CUW6VMUM3JVR7VJ4Y4V4EFZ4YKQ7HQ, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.395962717 (429h34m41.267045762s) [noise=true , address=77.90.31.13:4629 , piece_id=VX2CVI5FXKWL3W6ZFTRXG5RREY37EBUEQAHSQVPGBTJ63SC7XA4Q, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.396003695 (429h34m41.26708673s) [noise=true , address=145.79.227.52:4446 , piece_id=PU54PSDQBMMYEONFTQT5IDII5K3PU4RAFZ5LMHOOM7YNAJS47OXA, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.396288427 (429h34m41.267371462s) [noise=true , address=145.79.227.11:4731 , piece_id=3N5KV6DMUF4QI2ZD3WB55AMP6X2YAFWCWP7VM4OBFPFB5UZEPGZQ, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.396312643 (429h34m41.267395678s) [noise=true , address=145.79.227.34:4905 , piece_id=D54AOI2BG6TPMWQSOJIQFFPQUQRMNYVFDY4PI7JK2NMFHJZAI23Q, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.396575905 (429h34m41.26765894s) [noise=true , address=145.79.227.29:4942 , piece_id=2HHIYW2ZIIAKDV4CZCO74BAVPK457IVGTE4EJVGANHF65D5W7PNA, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.396676416 (429h34m41.267759461s) [noise=true , address=145.79.227.37:4565 , piece_id=HBJ366WIOAHI6ADVVEEDQQWXUS2E5KPUSFEP2KL4XO2X5LOTFKYQ, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.396865416 (429h34m41.267948451s) [noise=true , address=145.79.227.21:4473 , piece_id=7BR47RIYUFP5FWPABRZROLAAMKGHEPZPL3IDLGQ3XQNKXAMJ6URA, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.397008699 (429h34m41.268091734s) [noise=true , address=45.91.116.6:4976 , piece_id=Y72PN4WXYYPJ2ZNRBR2XA6QMHHTZ6OYOGNCWDBVLZKTO3I4S5KZQ, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.397165558 (429h34m41.268248593s) [noise=true , address=145.79.227.12:4765 , piece_id=NOLXPWRRNQXRZ4OA2KMMX5M6FYJR2S3AOGPDIPVHXHQB5HN4GMZA, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.397266479 (429h34m41.268349514s) [noise=true , address=45.91.116.14:4191 , piece_id=J4OQFX6XPN6GB6RJIVPBZUIBDDPLCWMM6GYGJGT7BAZAUM4DBNCA, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.397521615 (429h34m41.268604661s) [noise=true , address=145.79.227.38:4977 , piece_id=O6QO5Q3IFUTUCDOK7HXTJUYKTDJ5RBWDYB4NGEAMZOHDDR3IIC4A, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.398961146 (429h34m41.270044191s) [noise=true , address=145.79.227.32:4835 , piece_id=ULJPGLR3BQ6H2CGZHO7C5XO2EVN7Q7EY5L4NHRXZ3MQ6U7S64MVA, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:43.108995170 (429h34m42.980078205s) [upload=377134]: Next returned nil segment Dec 23 13:03:43.109012081 (429h34m42.980095146s) [upload=377134]: Waiting for error group managing segments... Dec 23 13:03:43.392567427 (429h34m43.263650462s) [noise=true , address=145.79.227.37:4565 , piece_id=HBJ366WIOAHI6ADVVEEDQQWXUS2E5KPUSFEP2KL4XO2X5LOTFKYQ, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.402738524 (429h34m43.273821559s) [noise=true , address=145.79.227.11:4731 , piece_id=3N5KV6DMUF4QI2ZD3WB55AMP6X2YAFWCWP7VM4OBFPFB5UZEPGZQ, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.403357101 (429h34m43.274440156s) [noise=true , address=145.79.227.38:4977 , piece_id=O6QO5Q3IFUTUCDOK7HXTJUYKTDJ5RBWDYB4NGEAMZOHDDR3IIC4A, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.407110529 (429h34m43.278193564s) [noise=true , address=145.79.227.21:4473 , piece_id=7BR47RIYUFP5FWPABRZROLAAMKGHEPZPL3IDLGQ3XQNKXAMJ6URA, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.407617795 (429h34m43.27870083s) [noise=true , address=145.79.227.29:4942 , piece_id=2HHIYW2ZIIAKDV4CZCO74BAVPK457IVGTE4EJVGANHF65D5W7PNA, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.408029740 (429h34m43.279112775s) [noise=true , address=145.79.227.20:4438 , piece_id=TAOBXUAGOSBITQLEPKPRC7CUW6VMUM3JVR7VJ4Y4V4EFZ4YKQ7HQ, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.412549485 (429h34m43.28363253s) [noise=true , address=145.79.227.52:4446 , piece_id=PU54PSDQBMMYEONFTQT5IDII5K3PU4RAFZ5LMHOOM7YNAJS47OXA, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.432973536 (429h34m43.304056571s) [noise=true , address=145.79.227.34:4905 , piece_id=D54AOI2BG6TPMWQSOJIQFFPQUQRMNYVFDY4PI7JK2NMFHJZAI23Q, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.445500129 (429h34m43.316583164s) [noise=true , address=145.79.227.32:4835 , piece_id=ULJPGLR3BQ6H2CGZHO7C5XO2EVN7Q7EY5L4NHRXZ3MQ6U7S64MVA, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.456746244 (429h34m43.327829279s) [noise=true , address=145.79.227.12:4765 , piece_id=NOLXPWRRNQXRZ4OA2KMMX5M6FYJR2S3AOGPDIPVHXHQB5HN4GMZA, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.644894828 (429h34m43.515977873s) [noise=true , address=45.91.116.5:4297 , piece_id=TQXE3Q6QFA7F63DGJTQKNWEHIP2TOVRCPRVGCX6XC57BY7PJB3UQ, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.713352172 (429h34m43.584435217s) [noise=true , address=45.91.116.6:4976 , piece_id=Y72PN4WXYYPJ2ZNRBR2XA6QMHHTZ6OYOGNCWDBVLZKTO3I4S5KZQ, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.731354943 (429h34m43.602437988s) [noise=true , address=45.91.116.14:4191 , piece_id=J4OQFX6XPN6GB6RJIVPBZUIBDDPLCWMM6GYGJGT7BAZAUM4DBNCA, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.731377275 (429h34m43.602460311s) [seg_pos={0 0}, upload=377134]: Segment reached optimal threshold of 13 pieces. Dec 23 13:03:43.731577968 (429h34m43.602661013s) [noise=true , address=77.90.31.13:4629 , piece_id=VX2CVI5FXKWL3W6ZFTRXG5RREY37EBUEQAHSQVPGBTJ63SC7XA4Q, seg_pos={0 0}, upload=377134]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: write tcp 188.166.225.170:54770->77.90.31.13:4629: use of closed network connection; write tcp 188.166.225.170:54770->77.90.31.13:4629: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54770->77.90.31.13:4629: use of closed network connection Dec 23 13:03:43.731585332 (429h34m43.602668377s) [noise=true , address=77.90.31.13:4629 , piece_id=VX2CVI5FXKWL3W6ZFTRXG5RREY37EBUEQAHSQVPGBTJ63SC7XA4Q, seg_pos={0 0}, upload=377134]: Long tail cancellation due to optimal threshold Dec 23 13:03:43.731671216 (429h34m43.602754261s) [noise=true , address=45.91.116.12:4226 , piece_id=BN76ETRXTTAEDWXZWR3A4KEOHHVKV4OFMG2GE6UOV7DIBE2WHIZA, seg_pos={0 0}, upload=377134]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 23 13:03:43.731682317 (429h34m43.602765362s) [noise=true , address=45.91.116.12:4226 , piece_id=BN76ETRXTTAEDWXZWR3A4KEOHHVKV4OFMG2GE6UOV7DIBE2WHIZA, seg_pos={0 0}, upload=377134]: Long tail cancellation due to optimal threshold Dec 23 13:03:43.731838263 (429h34m43.602921308s) [noise=true , address=45.91.116.11:4120 , piece_id=FODNYYBM7M4TD5KQZYKWGSQXULEIPEBPE6BTUPF5AZZUZSLQ7ACQ, seg_pos={0 0}, upload=377134]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 188.166.225.170:43872->45.91.116.11:4120: use of closed network connection; write tcp 188.166.225.170:43872->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43872->45.91.116.11:4120: use of closed network connection Dec 23 13:03:43.731847792 (429h34m43.602930837s) [noise=true , address=45.91.116.11:4120 , piece_id=FODNYYBM7M4TD5KQZYKWGSQXULEIPEBPE6BTUPF5AZZUZSLQ7ACQ, seg_pos={0 0}, upload=377134]: Long tail cancellation due to optimal threshold Dec 23 13:03:43.731869032 (429h34m43.602952067s) [seg_pos={0 0}, upload=377134]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 13:03:43.731968902 (429h34m43.603051947s) [upload=377134]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 13:03:43.731975024 (429h34m43.603058069s) [upload=377134]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 13:03:44.027725380 (429h34m43.898808415s) [upload=377134]: Done uploading segments... Dec 23 13:03:44.027744758 (429h34m43.898827793s) [upload=377134]: Upload finished. err: <nil> Duration: 5.699145127s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................................X.X......] duration: 5.782899791s trace: https://jaeger.dev.storj.io/trace/335d60b5e3cb0611 FAIL (2025-12-23 13:07:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== gments... Dec 23 13:03:41.146773219 (429h34m41.017856254s) [upload=377134]: Got next segment. Inline: false Dec 23 13:03:41.146815550 (429h34m41.017898585s) [upload=377134]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 13:03:41.146820509 (429h34m41.017903544s) [upload=377134]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 13:03:41.395177081 (429h34m41.266260127s) [seg_pos={0 0}, upload=377134]: Begin upload segment... Dec 23 13:03:41.395218049 (429h34m41.266301084s) [seg_pos={0 0}, upload=377134]: Done begin upload segment. Dec 23 13:03:41.395250882 (429h34m41.266333927s) [noise=true , address=45.91.116.12:4226 , piece_id=BN76ETRXTTAEDWXZWR3A4KEOHHVKV4OFMG2GE6UOV7DIBE2WHIZA, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.395268705 (429h34m41.26635175s) [noise=true , address=45.91.116.11:4120 , piece_id=FODNYYBM7M4TD5KQZYKWGSQXULEIPEBPE6BTUPF5AZZUZSLQ7ACQ, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.395658838 (429h34m41.266741873s) [noise=true , address=45.91.116.5:4297 , piece_id=TQXE3Q6QFA7F63DGJTQKNWEHIP2TOVRCPRVGCX6XC57BY7PJB3UQ, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.395663367 (429h34m41.266746413s) [noise=true , address=145.79.227.20:4438 , piece_id=TAOBXUAGOSBITQLEPKPRC7CUW6VMUM3JVR7VJ4Y4V4EFZ4YKQ7HQ, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.395962717 (429h34m41.267045762s) [noise=true , address=77.90.31.13:4629 , piece_id=VX2CVI5FXKWL3W6ZFTRXG5RREY37EBUEQAHSQVPGBTJ63SC7XA4Q, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.396003695 (429h34m41.26708673s) [noise=true , address=145.79.227.52:4446 , piece_id=PU54PSDQBMMYEONFTQT5IDII5K3PU4RAFZ5LMHOOM7YNAJS47OXA, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.396288427 (429h34m41.267371462s) [noise=true , address=145.79.227.11:4731 , piece_id=3N5KV6DMUF4QI2ZD3WB55AMP6X2YAFWCWP7VM4OBFPFB5UZEPGZQ, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.396312643 (429h34m41.267395678s) [noise=true , address=145.79.227.34:4905 , piece_id=D54AOI2BG6TPMWQSOJIQFFPQUQRMNYVFDY4PI7JK2NMFHJZAI23Q, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.396575905 (429h34m41.26765894s) [noise=true , address=145.79.227.29:4942 , piece_id=2HHIYW2ZIIAKDV4CZCO74BAVPK457IVGTE4EJVGANHF65D5W7PNA, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.396676416 (429h34m41.267759461s) [noise=true , address=145.79.227.37:4565 , piece_id=HBJ366WIOAHI6ADVVEEDQQWXUS2E5KPUSFEP2KL4XO2X5LOTFKYQ, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.396865416 (429h34m41.267948451s) [noise=true , address=145.79.227.21:4473 , piece_id=7BR47RIYUFP5FWPABRZROLAAMKGHEPZPL3IDLGQ3XQNKXAMJ6URA, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.397008699 (429h34m41.268091734s) [noise=true , address=45.91.116.6:4976 , piece_id=Y72PN4WXYYPJ2ZNRBR2XA6QMHHTZ6OYOGNCWDBVLZKTO3I4S5KZQ, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.397165558 (429h34m41.268248593s) [noise=true , address=145.79.227.12:4765 , piece_id=NOLXPWRRNQXRZ4OA2KMMX5M6FYJR2S3AOGPDIPVHXHQB5HN4GMZA, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.397266479 (429h34m41.268349514s) [noise=true , address=45.91.116.14:4191 , piece_id=J4OQFX6XPN6GB6RJIVPBZUIBDDPLCWMM6GYGJGT7BAZAUM4DBNCA, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.397521615 (429h34m41.268604661s) [noise=true , address=145.79.227.38:4977 , piece_id=O6QO5Q3IFUTUCDOK7HXTJUYKTDJ5RBWDYB4NGEAMZOHDDR3IIC4A, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.398961146 (429h34m41.270044191s) [noise=true , address=145.79.227.32:4835 , piece_id=ULJPGLR3BQ6H2CGZHO7C5XO2EVN7Q7EY5L4NHRXZ3MQ6U7S64MVA, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:43.108995170 (429h34m42.980078205s) [upload=377134]: Next returned nil segment Dec 23 13:03:43.109012081 (429h34m42.980095146s) [upload=377134]: Waiting for error group managing segments... Dec 23 13:03:43.392567427 (429h34m43.263650462s) [noise=true , address=145.79.227.37:4565 , piece_id=HBJ366WIOAHI6ADVVEEDQQWXUS2E5KPUSFEP2KL4XO2X5LOTFKYQ, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.402738524 (429h34m43.273821559s) [noise=true , address=145.79.227.11:4731 , piece_id=3N5KV6DMUF4QI2ZD3WB55AMP6X2YAFWCWP7VM4OBFPFB5UZEPGZQ, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.403357101 (429h34m43.274440156s) [noise=true , address=145.79.227.38:4977 , piece_id=O6QO5Q3IFUTUCDOK7HXTJUYKTDJ5RBWDYB4NGEAMZOHDDR3IIC4A, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.407110529 (429h34m43.278193564s) [noise=true , address=145.79.227.21:4473 , piece_id=7BR47RIYUFP5FWPABRZROLAAMKGHEPZPL3IDLGQ3XQNKXAMJ6URA, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.407617795 (429h34m43.27870083s) [noise=true , address=145.79.227.29:4942 , piece_id=2HHIYW2ZIIAKDV4CZCO74BAVPK457IVGTE4EJVGANHF65D5W7PNA, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.408029740 (429h34m43.279112775s) [noise=true , address=145.79.227.20:4438 , piece_id=TAOBXUAGOSBITQLEPKPRC7CUW6VMUM3JVR7VJ4Y4V4EFZ4YKQ7HQ, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.412549485 (429h34m43.28363253s) [noise=true , address=145.79.227.52:4446 , piece_id=PU54PSDQBMMYEONFTQT5IDII5K3PU4RAFZ5LMHOOM7YNAJS47OXA, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.432973536 (429h34m43.304056571s) [noise=true , address=145.79.227.34:4905 , piece_id=D54AOI2BG6TPMWQSOJIQFFPQUQRMNYVFDY4PI7JK2NMFHJZAI23Q, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.445500129 (429h34m43.316583164s) [noise=true , address=145.79.227.32:4835 , piece_id=ULJPGLR3BQ6H2CGZHO7C5XO2EVN7Q7EY5L4NHRXZ3MQ6U7S64MVA, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.456746244 (429h34m43.327829279s) [noise=true , address=145.79.227.12:4765 , piece_id=NOLXPWRRNQXRZ4OA2KMMX5M6FYJR2S3AOGPDIPVHXHQB5HN4GMZA, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.644894828 (429h34m43.515977873s) [noise=true , address=45.91.116.5:4297 , piece_id=TQXE3Q6QFA7F63DGJTQKNWEHIP2TOVRCPRVGCX6XC57BY7PJB3UQ, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.713352172 (429h34m43.584435217s) [noise=true , address=45.91.116.6:4976 , piece_id=Y72PN4WXYYPJ2ZNRBR2XA6QMHHTZ6OYOGNCWDBVLZKTO3I4S5KZQ, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.731354943 (429h34m43.602437988s) [noise=true , address=45.91.116.14:4191 , piece_id=J4OQFX6XPN6GB6RJIVPBZUIBDDPLCWMM6GYGJGT7BAZAUM4DBNCA, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.731377275 (429h34m43.602460311s) [seg_pos={0 0}, upload=377134]: Segment reached optimal threshold of 13 pieces. Dec 23 13:03:43.731577968 (429h34m43.602661013s) [noise=true , address=77.90.31.13:4629 , piece_id=VX2CVI5FXKWL3W6ZFTRXG5RREY37EBUEQAHSQVPGBTJ63SC7XA4Q, seg_pos={0 0}, upload=377134]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: write tcp 188.166.225.170:54770->77.90.31.13:4629: use of closed network connection; write tcp 188.166.225.170:54770->77.90.31.13:4629: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54770->77.90.31.13:4629: use of closed network connection Dec 23 13:03:43.731585332 (429h34m43.602668377s) [noise=true , address=77.90.31.13:4629 , piece_id=VX2CVI5FXKWL3W6ZFTRXG5RREY37EBUEQAHSQVPGBTJ63SC7XA4Q, seg_pos={0 0}, upload=377134]: Long tail cancellation due to optimal threshold Dec 23 13:03:43.731671216 (429h34m43.602754261s) [noise=true , address=45.91.116.12:4226 , piece_id=BN76ETRXTTAEDWXZWR3A4KEOHHVKV4OFMG2GE6UOV7DIBE2WHIZA, seg_pos={0 0}, upload=377134]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 23 13:03:43.731682317 (429h34m43.602765362s) [noise=true , address=45.91.116.12:4226 , piece_id=BN76ETRXTTAEDWXZWR3A4KEOHHVKV4OFMG2GE6UOV7DIBE2WHIZA, seg_pos={0 0}, upload=377134]: Long tail cancellation due to optimal threshold Dec 23 13:03:43.731838263 (429h34m43.602921308s) [noise=true , address=45.91.116.11:4120 , piece_id=FODNYYBM7M4TD5KQZYKWGSQXULEIPEBPE6BTUPF5AZZUZSLQ7ACQ, seg_pos={0 0}, upload=377134]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 188.166.225.170:43872->45.91.116.11:4120: use of closed network connection; write tcp 188.166.225.170:43872->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43872->45.91.116.11:4120: use of closed network connection Dec 23 13:03:43.731847792 (429h34m43.602930837s) [noise=true , address=45.91.116.11:4120 , piece_id=FODNYYBM7M4TD5KQZYKWGSQXULEIPEBPE6BTUPF5AZZUZSLQ7ACQ, seg_pos={0 0}, upload=377134]: Long tail cancellation due to optimal threshold Dec 23 13:03:43.731869032 (429h34m43.602952067s) [seg_pos={0 0}, upload=377134]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 13:03:43.731968902 (429h34m43.603051947s) [upload=377134]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 13:03:43.731975024 (429h34m43.603058069s) [upload=377134]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 13:03:44.027725380 (429h34m43.898808415s) [upload=377134]: Done uploading segments... Dec 23 13:03:44.027744758 (429h34m43.898827793s) [upload=377134]: Upload finished. err: <nil> Duration: 5.782917804s core uplink us-select-1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [......................................X.....................] duration: 251.989008ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/73921c1a3ca0437b FAIL (2025-12-23 13:07:52 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 252.013133ms edge linksharing bgp eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................................X.X.....] duration: 5.881995072s trace: https://jaeger.dev.storj.io/trace/335d60b5e3cb0611 FAIL (2025-12-23 13:07:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== gments... Dec 23 13:03:41.146773219 (429h34m41.017856254s) [upload=377134]: Got next segment. Inline: false Dec 23 13:03:41.146815550 (429h34m41.017898585s) [upload=377134]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 13:03:41.146820509 (429h34m41.017903544s) [upload=377134]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 13:03:41.395177081 (429h34m41.266260127s) [seg_pos={0 0}, upload=377134]: Begin upload segment... Dec 23 13:03:41.395218049 (429h34m41.266301084s) [seg_pos={0 0}, upload=377134]: Done begin upload segment. Dec 23 13:03:41.395250882 (429h34m41.266333927s) [noise=true , address=45.91.116.12:4226 , piece_id=BN76ETRXTTAEDWXZWR3A4KEOHHVKV4OFMG2GE6UOV7DIBE2WHIZA, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.395268705 (429h34m41.26635175s) [noise=true , address=45.91.116.11:4120 , piece_id=FODNYYBM7M4TD5KQZYKWGSQXULEIPEBPE6BTUPF5AZZUZSLQ7ACQ, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.395658838 (429h34m41.266741873s) [noise=true , address=45.91.116.5:4297 , piece_id=TQXE3Q6QFA7F63DGJTQKNWEHIP2TOVRCPRVGCX6XC57BY7PJB3UQ, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.395663367 (429h34m41.266746413s) [noise=true , address=145.79.227.20:4438 , piece_id=TAOBXUAGOSBITQLEPKPRC7CUW6VMUM3JVR7VJ4Y4V4EFZ4YKQ7HQ, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.395962717 (429h34m41.267045762s) [noise=true , address=77.90.31.13:4629 , piece_id=VX2CVI5FXKWL3W6ZFTRXG5RREY37EBUEQAHSQVPGBTJ63SC7XA4Q, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.396003695 (429h34m41.26708673s) [noise=true , address=145.79.227.52:4446 , piece_id=PU54PSDQBMMYEONFTQT5IDII5K3PU4RAFZ5LMHOOM7YNAJS47OXA, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.396288427 (429h34m41.267371462s) [noise=true , address=145.79.227.11:4731 , piece_id=3N5KV6DMUF4QI2ZD3WB55AMP6X2YAFWCWP7VM4OBFPFB5UZEPGZQ, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.396312643 (429h34m41.267395678s) [noise=true , address=145.79.227.34:4905 , piece_id=D54AOI2BG6TPMWQSOJIQFFPQUQRMNYVFDY4PI7JK2NMFHJZAI23Q, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.396575905 (429h34m41.26765894s) [noise=true , address=145.79.227.29:4942 , piece_id=2HHIYW2ZIIAKDV4CZCO74BAVPK457IVGTE4EJVGANHF65D5W7PNA, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.396676416 (429h34m41.267759461s) [noise=true , address=145.79.227.37:4565 , piece_id=HBJ366WIOAHI6ADVVEEDQQWXUS2E5KPUSFEP2KL4XO2X5LOTFKYQ, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.396865416 (429h34m41.267948451s) [noise=true , address=145.79.227.21:4473 , piece_id=7BR47RIYUFP5FWPABRZROLAAMKGHEPZPL3IDLGQ3XQNKXAMJ6URA, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.397008699 (429h34m41.268091734s) [noise=true , address=45.91.116.6:4976 , piece_id=Y72PN4WXYYPJ2ZNRBR2XA6QMHHTZ6OYOGNCWDBVLZKTO3I4S5KZQ, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.397165558 (429h34m41.268248593s) [noise=true , address=145.79.227.12:4765 , piece_id=NOLXPWRRNQXRZ4OA2KMMX5M6FYJR2S3AOGPDIPVHXHQB5HN4GMZA, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.397266479 (429h34m41.268349514s) [noise=true , address=45.91.116.14:4191 , piece_id=J4OQFX6XPN6GB6RJIVPBZUIBDDPLCWMM6GYGJGT7BAZAUM4DBNCA, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.397521615 (429h34m41.268604661s) [noise=true , address=145.79.227.38:4977 , piece_id=O6QO5Q3IFUTUCDOK7HXTJUYKTDJ5RBWDYB4NGEAMZOHDDR3IIC4A, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.398961146 (429h34m41.270044191s) [noise=true , address=145.79.227.32:4835 , piece_id=ULJPGLR3BQ6H2CGZHO7C5XO2EVN7Q7EY5L4NHRXZ3MQ6U7S64MVA, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:43.108995170 (429h34m42.980078205s) [upload=377134]: Next returned nil segment Dec 23 13:03:43.109012081 (429h34m42.980095146s) [upload=377134]: Waiting for error group managing segments... Dec 23 13:03:43.392567427 (429h34m43.263650462s) [noise=true , address=145.79.227.37:4565 , piece_id=HBJ366WIOAHI6ADVVEEDQQWXUS2E5KPUSFEP2KL4XO2X5LOTFKYQ, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.402738524 (429h34m43.273821559s) [noise=true , address=145.79.227.11:4731 , piece_id=3N5KV6DMUF4QI2ZD3WB55AMP6X2YAFWCWP7VM4OBFPFB5UZEPGZQ, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.403357101 (429h34m43.274440156s) [noise=true , address=145.79.227.38:4977 , piece_id=O6QO5Q3IFUTUCDOK7HXTJUYKTDJ5RBWDYB4NGEAMZOHDDR3IIC4A, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.407110529 (429h34m43.278193564s) [noise=true , address=145.79.227.21:4473 , piece_id=7BR47RIYUFP5FWPABRZROLAAMKGHEPZPL3IDLGQ3XQNKXAMJ6URA, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.407617795 (429h34m43.27870083s) [noise=true , address=145.79.227.29:4942 , piece_id=2HHIYW2ZIIAKDV4CZCO74BAVPK457IVGTE4EJVGANHF65D5W7PNA, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.408029740 (429h34m43.279112775s) [noise=true , address=145.79.227.20:4438 , piece_id=TAOBXUAGOSBITQLEPKPRC7CUW6VMUM3JVR7VJ4Y4V4EFZ4YKQ7HQ, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.412549485 (429h34m43.28363253s) [noise=true , address=145.79.227.52:4446 , piece_id=PU54PSDQBMMYEONFTQT5IDII5K3PU4RAFZ5LMHOOM7YNAJS47OXA, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.432973536 (429h34m43.304056571s) [noise=true , address=145.79.227.34:4905 , piece_id=D54AOI2BG6TPMWQSOJIQFFPQUQRMNYVFDY4PI7JK2NMFHJZAI23Q, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.445500129 (429h34m43.316583164s) [noise=true , address=145.79.227.32:4835 , piece_id=ULJPGLR3BQ6H2CGZHO7C5XO2EVN7Q7EY5L4NHRXZ3MQ6U7S64MVA, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.456746244 (429h34m43.327829279s) [noise=true , address=145.79.227.12:4765 , piece_id=NOLXPWRRNQXRZ4OA2KMMX5M6FYJR2S3AOGPDIPVHXHQB5HN4GMZA, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.644894828 (429h34m43.515977873s) [noise=true , address=45.91.116.5:4297 , piece_id=TQXE3Q6QFA7F63DGJTQKNWEHIP2TOVRCPRVGCX6XC57BY7PJB3UQ, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.713352172 (429h34m43.584435217s) [noise=true , address=45.91.116.6:4976 , piece_id=Y72PN4WXYYPJ2ZNRBR2XA6QMHHTZ6OYOGNCWDBVLZKTO3I4S5KZQ, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.731354943 (429h34m43.602437988s) [noise=true , address=45.91.116.14:4191 , piece_id=J4OQFX6XPN6GB6RJIVPBZUIBDDPLCWMM6GYGJGT7BAZAUM4DBNCA, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.731377275 (429h34m43.602460311s) [seg_pos={0 0}, upload=377134]: Segment reached optimal threshold of 13 pieces. Dec 23 13:03:43.731577968 (429h34m43.602661013s) [noise=true , address=77.90.31.13:4629 , piece_id=VX2CVI5FXKWL3W6ZFTRXG5RREY37EBUEQAHSQVPGBTJ63SC7XA4Q, seg_pos={0 0}, upload=377134]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: write tcp 188.166.225.170:54770->77.90.31.13:4629: use of closed network connection; write tcp 188.166.225.170:54770->77.90.31.13:4629: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54770->77.90.31.13:4629: use of closed network connection Dec 23 13:03:43.731585332 (429h34m43.602668377s) [noise=true , address=77.90.31.13:4629 , piece_id=VX2CVI5FXKWL3W6ZFTRXG5RREY37EBUEQAHSQVPGBTJ63SC7XA4Q, seg_pos={0 0}, upload=377134]: Long tail cancellation due to optimal threshold Dec 23 13:03:43.731671216 (429h34m43.602754261s) [noise=true , address=45.91.116.12:4226 , piece_id=BN76ETRXTTAEDWXZWR3A4KEOHHVKV4OFMG2GE6UOV7DIBE2WHIZA, seg_pos={0 0}, upload=377134]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 23 13:03:43.731682317 (429h34m43.602765362s) [noise=true , address=45.91.116.12:4226 , piece_id=BN76ETRXTTAEDWXZWR3A4KEOHHVKV4OFMG2GE6UOV7DIBE2WHIZA, seg_pos={0 0}, upload=377134]: Long tail cancellation due to optimal threshold Dec 23 13:03:43.731838263 (429h34m43.602921308s) [noise=true , address=45.91.116.11:4120 , piece_id=FODNYYBM7M4TD5KQZYKWGSQXULEIPEBPE6BTUPF5AZZUZSLQ7ACQ, seg_pos={0 0}, upload=377134]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 188.166.225.170:43872->45.91.116.11:4120: use of closed network connection; write tcp 188.166.225.170:43872->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43872->45.91.116.11:4120: use of closed network connection Dec 23 13:03:43.731847792 (429h34m43.602930837s) [noise=true , address=45.91.116.11:4120 , piece_id=FODNYYBM7M4TD5KQZYKWGSQXULEIPEBPE6BTUPF5AZZUZSLQ7ACQ, seg_pos={0 0}, upload=377134]: Long tail cancellation due to optimal threshold Dec 23 13:03:43.731869032 (429h34m43.602952067s) [seg_pos={0 0}, upload=377134]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 13:03:43.731968902 (429h34m43.603051947s) [upload=377134]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 13:03:43.731975024 (429h34m43.603058069s) [upload=377134]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 13:03:44.027725380 (429h34m43.898808415s) [upload=377134]: Done uploading segments... Dec 23 13:03:44.027744758 (429h34m43.898827793s) [upload=377134]: Upload finished. err: <nil> Duration: 5.882013095s core uplink us-select-1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [.......................................X....................] duration: 270.469947ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/73921c1a3ca0437b FAIL (2025-12-23 13:07:32 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 270.495557ms edge linksharing bgp eu1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [........................................X...................] duration: 260.758437ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/73921c1a3ca0437b FAIL (2025-12-23 13:06:45 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 260.788764ms edge linksharing bgp eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................................X.X....] duration: 5.718216133s trace: https://jaeger.dev.storj.io/trace/335d60b5e3cb0611 FAIL (2025-12-23 13:06:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== gments... Dec 23 13:03:41.146773219 (429h34m41.017856254s) [upload=377134]: Got next segment. Inline: false Dec 23 13:03:41.146815550 (429h34m41.017898585s) [upload=377134]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 13:03:41.146820509 (429h34m41.017903544s) [upload=377134]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 13:03:41.395177081 (429h34m41.266260127s) [seg_pos={0 0}, upload=377134]: Begin upload segment... Dec 23 13:03:41.395218049 (429h34m41.266301084s) [seg_pos={0 0}, upload=377134]: Done begin upload segment. Dec 23 13:03:41.395250882 (429h34m41.266333927s) [noise=true , address=45.91.116.12:4226 , piece_id=BN76ETRXTTAEDWXZWR3A4KEOHHVKV4OFMG2GE6UOV7DIBE2WHIZA, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.395268705 (429h34m41.26635175s) [noise=true , address=45.91.116.11:4120 , piece_id=FODNYYBM7M4TD5KQZYKWGSQXULEIPEBPE6BTUPF5AZZUZSLQ7ACQ, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.395658838 (429h34m41.266741873s) [noise=true , address=45.91.116.5:4297 , piece_id=TQXE3Q6QFA7F63DGJTQKNWEHIP2TOVRCPRVGCX6XC57BY7PJB3UQ, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.395663367 (429h34m41.266746413s) [noise=true , address=145.79.227.20:4438 , piece_id=TAOBXUAGOSBITQLEPKPRC7CUW6VMUM3JVR7VJ4Y4V4EFZ4YKQ7HQ, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.395962717 (429h34m41.267045762s) [noise=true , address=77.90.31.13:4629 , piece_id=VX2CVI5FXKWL3W6ZFTRXG5RREY37EBUEQAHSQVPGBTJ63SC7XA4Q, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.396003695 (429h34m41.26708673s) [noise=true , address=145.79.227.52:4446 , piece_id=PU54PSDQBMMYEONFTQT5IDII5K3PU4RAFZ5LMHOOM7YNAJS47OXA, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.396288427 (429h34m41.267371462s) [noise=true , address=145.79.227.11:4731 , piece_id=3N5KV6DMUF4QI2ZD3WB55AMP6X2YAFWCWP7VM4OBFPFB5UZEPGZQ, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.396312643 (429h34m41.267395678s) [noise=true , address=145.79.227.34:4905 , piece_id=D54AOI2BG6TPMWQSOJIQFFPQUQRMNYVFDY4PI7JK2NMFHJZAI23Q, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.396575905 (429h34m41.26765894s) [noise=true , address=145.79.227.29:4942 , piece_id=2HHIYW2ZIIAKDV4CZCO74BAVPK457IVGTE4EJVGANHF65D5W7PNA, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.396676416 (429h34m41.267759461s) [noise=true , address=145.79.227.37:4565 , piece_id=HBJ366WIOAHI6ADVVEEDQQWXUS2E5KPUSFEP2KL4XO2X5LOTFKYQ, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.396865416 (429h34m41.267948451s) [noise=true , address=145.79.227.21:4473 , piece_id=7BR47RIYUFP5FWPABRZROLAAMKGHEPZPL3IDLGQ3XQNKXAMJ6URA, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.397008699 (429h34m41.268091734s) [noise=true , address=45.91.116.6:4976 , piece_id=Y72PN4WXYYPJ2ZNRBR2XA6QMHHTZ6OYOGNCWDBVLZKTO3I4S5KZQ, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.397165558 (429h34m41.268248593s) [noise=true , address=145.79.227.12:4765 , piece_id=NOLXPWRRNQXRZ4OA2KMMX5M6FYJR2S3AOGPDIPVHXHQB5HN4GMZA, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.397266479 (429h34m41.268349514s) [noise=true , address=45.91.116.14:4191 , piece_id=J4OQFX6XPN6GB6RJIVPBZUIBDDPLCWMM6GYGJGT7BAZAUM4DBNCA, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.397521615 (429h34m41.268604661s) [noise=true , address=145.79.227.38:4977 , piece_id=O6QO5Q3IFUTUCDOK7HXTJUYKTDJ5RBWDYB4NGEAMZOHDDR3IIC4A, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.398961146 (429h34m41.270044191s) [noise=true , address=145.79.227.32:4835 , piece_id=ULJPGLR3BQ6H2CGZHO7C5XO2EVN7Q7EY5L4NHRXZ3MQ6U7S64MVA, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:43.108995170 (429h34m42.980078205s) [upload=377134]: Next returned nil segment Dec 23 13:03:43.109012081 (429h34m42.980095146s) [upload=377134]: Waiting for error group managing segments... Dec 23 13:03:43.392567427 (429h34m43.263650462s) [noise=true , address=145.79.227.37:4565 , piece_id=HBJ366WIOAHI6ADVVEEDQQWXUS2E5KPUSFEP2KL4XO2X5LOTFKYQ, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.402738524 (429h34m43.273821559s) [noise=true , address=145.79.227.11:4731 , piece_id=3N5KV6DMUF4QI2ZD3WB55AMP6X2YAFWCWP7VM4OBFPFB5UZEPGZQ, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.403357101 (429h34m43.274440156s) [noise=true , address=145.79.227.38:4977 , piece_id=O6QO5Q3IFUTUCDOK7HXTJUYKTDJ5RBWDYB4NGEAMZOHDDR3IIC4A, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.407110529 (429h34m43.278193564s) [noise=true , address=145.79.227.21:4473 , piece_id=7BR47RIYUFP5FWPABRZROLAAMKGHEPZPL3IDLGQ3XQNKXAMJ6URA, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.407617795 (429h34m43.27870083s) [noise=true , address=145.79.227.29:4942 , piece_id=2HHIYW2ZIIAKDV4CZCO74BAVPK457IVGTE4EJVGANHF65D5W7PNA, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.408029740 (429h34m43.279112775s) [noise=true , address=145.79.227.20:4438 , piece_id=TAOBXUAGOSBITQLEPKPRC7CUW6VMUM3JVR7VJ4Y4V4EFZ4YKQ7HQ, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.412549485 (429h34m43.28363253s) [noise=true , address=145.79.227.52:4446 , piece_id=PU54PSDQBMMYEONFTQT5IDII5K3PU4RAFZ5LMHOOM7YNAJS47OXA, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.432973536 (429h34m43.304056571s) [noise=true , address=145.79.227.34:4905 , piece_id=D54AOI2BG6TPMWQSOJIQFFPQUQRMNYVFDY4PI7JK2NMFHJZAI23Q, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.445500129 (429h34m43.316583164s) [noise=true , address=145.79.227.32:4835 , piece_id=ULJPGLR3BQ6H2CGZHO7C5XO2EVN7Q7EY5L4NHRXZ3MQ6U7S64MVA, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.456746244 (429h34m43.327829279s) [noise=true , address=145.79.227.12:4765 , piece_id=NOLXPWRRNQXRZ4OA2KMMX5M6FYJR2S3AOGPDIPVHXHQB5HN4GMZA, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.644894828 (429h34m43.515977873s) [noise=true , address=45.91.116.5:4297 , piece_id=TQXE3Q6QFA7F63DGJTQKNWEHIP2TOVRCPRVGCX6XC57BY7PJB3UQ, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.713352172 (429h34m43.584435217s) [noise=true , address=45.91.116.6:4976 , piece_id=Y72PN4WXYYPJ2ZNRBR2XA6QMHHTZ6OYOGNCWDBVLZKTO3I4S5KZQ, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.731354943 (429h34m43.602437988s) [noise=true , address=45.91.116.14:4191 , piece_id=J4OQFX6XPN6GB6RJIVPBZUIBDDPLCWMM6GYGJGT7BAZAUM4DBNCA, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.731377275 (429h34m43.602460311s) [seg_pos={0 0}, upload=377134]: Segment reached optimal threshold of 13 pieces. Dec 23 13:03:43.731577968 (429h34m43.602661013s) [noise=true , address=77.90.31.13:4629 , piece_id=VX2CVI5FXKWL3W6ZFTRXG5RREY37EBUEQAHSQVPGBTJ63SC7XA4Q, seg_pos={0 0}, upload=377134]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: write tcp 188.166.225.170:54770->77.90.31.13:4629: use of closed network connection; write tcp 188.166.225.170:54770->77.90.31.13:4629: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54770->77.90.31.13:4629: use of closed network connection Dec 23 13:03:43.731585332 (429h34m43.602668377s) [noise=true , address=77.90.31.13:4629 , piece_id=VX2CVI5FXKWL3W6ZFTRXG5RREY37EBUEQAHSQVPGBTJ63SC7XA4Q, seg_pos={0 0}, upload=377134]: Long tail cancellation due to optimal threshold Dec 23 13:03:43.731671216 (429h34m43.602754261s) [noise=true , address=45.91.116.12:4226 , piece_id=BN76ETRXTTAEDWXZWR3A4KEOHHVKV4OFMG2GE6UOV7DIBE2WHIZA, seg_pos={0 0}, upload=377134]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 23 13:03:43.731682317 (429h34m43.602765362s) [noise=true , address=45.91.116.12:4226 , piece_id=BN76ETRXTTAEDWXZWR3A4KEOHHVKV4OFMG2GE6UOV7DIBE2WHIZA, seg_pos={0 0}, upload=377134]: Long tail cancellation due to optimal threshold Dec 23 13:03:43.731838263 (429h34m43.602921308s) [noise=true , address=45.91.116.11:4120 , piece_id=FODNYYBM7M4TD5KQZYKWGSQXULEIPEBPE6BTUPF5AZZUZSLQ7ACQ, seg_pos={0 0}, upload=377134]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 188.166.225.170:43872->45.91.116.11:4120: use of closed network connection; write tcp 188.166.225.170:43872->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43872->45.91.116.11:4120: use of closed network connection Dec 23 13:03:43.731847792 (429h34m43.602930837s) [noise=true , address=45.91.116.11:4120 , piece_id=FODNYYBM7M4TD5KQZYKWGSQXULEIPEBPE6BTUPF5AZZUZSLQ7ACQ, seg_pos={0 0}, upload=377134]: Long tail cancellation due to optimal threshold Dec 23 13:03:43.731869032 (429h34m43.602952067s) [seg_pos={0 0}, upload=377134]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 13:03:43.731968902 (429h34m43.603051947s) [upload=377134]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 13:03:43.731975024 (429h34m43.603058069s) [upload=377134]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 13:03:44.027725380 (429h34m43.898808415s) [upload=377134]: Done uploading segments... Dec 23 13:03:44.027744758 (429h34m43.898827793s) [upload=377134]: Upload finished. err: <nil> Duration: 5.718240951s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................................X.X...] duration: 6.083555919s trace: https://jaeger.dev.storj.io/trace/335d60b5e3cb0611 FAIL (2025-12-23 13:05:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== gments... Dec 23 13:03:41.146773219 (429h34m41.017856254s) [upload=377134]: Got next segment. Inline: false Dec 23 13:03:41.146815550 (429h34m41.017898585s) [upload=377134]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 13:03:41.146820509 (429h34m41.017903544s) [upload=377134]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 13:03:41.395177081 (429h34m41.266260127s) [seg_pos={0 0}, upload=377134]: Begin upload segment... Dec 23 13:03:41.395218049 (429h34m41.266301084s) [seg_pos={0 0}, upload=377134]: Done begin upload segment. Dec 23 13:03:41.395250882 (429h34m41.266333927s) [noise=true , address=45.91.116.12:4226 , piece_id=BN76ETRXTTAEDWXZWR3A4KEOHHVKV4OFMG2GE6UOV7DIBE2WHIZA, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.395268705 (429h34m41.26635175s) [noise=true , address=45.91.116.11:4120 , piece_id=FODNYYBM7M4TD5KQZYKWGSQXULEIPEBPE6BTUPF5AZZUZSLQ7ACQ, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.395658838 (429h34m41.266741873s) [noise=true , address=45.91.116.5:4297 , piece_id=TQXE3Q6QFA7F63DGJTQKNWEHIP2TOVRCPRVGCX6XC57BY7PJB3UQ, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.395663367 (429h34m41.266746413s) [noise=true , address=145.79.227.20:4438 , piece_id=TAOBXUAGOSBITQLEPKPRC7CUW6VMUM3JVR7VJ4Y4V4EFZ4YKQ7HQ, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.395962717 (429h34m41.267045762s) [noise=true , address=77.90.31.13:4629 , piece_id=VX2CVI5FXKWL3W6ZFTRXG5RREY37EBUEQAHSQVPGBTJ63SC7XA4Q, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.396003695 (429h34m41.26708673s) [noise=true , address=145.79.227.52:4446 , piece_id=PU54PSDQBMMYEONFTQT5IDII5K3PU4RAFZ5LMHOOM7YNAJS47OXA, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.396288427 (429h34m41.267371462s) [noise=true , address=145.79.227.11:4731 , piece_id=3N5KV6DMUF4QI2ZD3WB55AMP6X2YAFWCWP7VM4OBFPFB5UZEPGZQ, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.396312643 (429h34m41.267395678s) [noise=true , address=145.79.227.34:4905 , piece_id=D54AOI2BG6TPMWQSOJIQFFPQUQRMNYVFDY4PI7JK2NMFHJZAI23Q, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.396575905 (429h34m41.26765894s) [noise=true , address=145.79.227.29:4942 , piece_id=2HHIYW2ZIIAKDV4CZCO74BAVPK457IVGTE4EJVGANHF65D5W7PNA, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.396676416 (429h34m41.267759461s) [noise=true , address=145.79.227.37:4565 , piece_id=HBJ366WIOAHI6ADVVEEDQQWXUS2E5KPUSFEP2KL4XO2X5LOTFKYQ, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.396865416 (429h34m41.267948451s) [noise=true , address=145.79.227.21:4473 , piece_id=7BR47RIYUFP5FWPABRZROLAAMKGHEPZPL3IDLGQ3XQNKXAMJ6URA, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.397008699 (429h34m41.268091734s) [noise=true , address=45.91.116.6:4976 , piece_id=Y72PN4WXYYPJ2ZNRBR2XA6QMHHTZ6OYOGNCWDBVLZKTO3I4S5KZQ, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.397165558 (429h34m41.268248593s) [noise=true , address=145.79.227.12:4765 , piece_id=NOLXPWRRNQXRZ4OA2KMMX5M6FYJR2S3AOGPDIPVHXHQB5HN4GMZA, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.397266479 (429h34m41.268349514s) [noise=true , address=45.91.116.14:4191 , piece_id=J4OQFX6XPN6GB6RJIVPBZUIBDDPLCWMM6GYGJGT7BAZAUM4DBNCA, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.397521615 (429h34m41.268604661s) [noise=true , address=145.79.227.38:4977 , piece_id=O6QO5Q3IFUTUCDOK7HXTJUYKTDJ5RBWDYB4NGEAMZOHDDR3IIC4A, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.398961146 (429h34m41.270044191s) [noise=true , address=145.79.227.32:4835 , piece_id=ULJPGLR3BQ6H2CGZHO7C5XO2EVN7Q7EY5L4NHRXZ3MQ6U7S64MVA, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:43.108995170 (429h34m42.980078205s) [upload=377134]: Next returned nil segment Dec 23 13:03:43.109012081 (429h34m42.980095146s) [upload=377134]: Waiting for error group managing segments... Dec 23 13:03:43.392567427 (429h34m43.263650462s) [noise=true , address=145.79.227.37:4565 , piece_id=HBJ366WIOAHI6ADVVEEDQQWXUS2E5KPUSFEP2KL4XO2X5LOTFKYQ, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.402738524 (429h34m43.273821559s) [noise=true , address=145.79.227.11:4731 , piece_id=3N5KV6DMUF4QI2ZD3WB55AMP6X2YAFWCWP7VM4OBFPFB5UZEPGZQ, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.403357101 (429h34m43.274440156s) [noise=true , address=145.79.227.38:4977 , piece_id=O6QO5Q3IFUTUCDOK7HXTJUYKTDJ5RBWDYB4NGEAMZOHDDR3IIC4A, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.407110529 (429h34m43.278193564s) [noise=true , address=145.79.227.21:4473 , piece_id=7BR47RIYUFP5FWPABRZROLAAMKGHEPZPL3IDLGQ3XQNKXAMJ6URA, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.407617795 (429h34m43.27870083s) [noise=true , address=145.79.227.29:4942 , piece_id=2HHIYW2ZIIAKDV4CZCO74BAVPK457IVGTE4EJVGANHF65D5W7PNA, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.408029740 (429h34m43.279112775s) [noise=true , address=145.79.227.20:4438 , piece_id=TAOBXUAGOSBITQLEPKPRC7CUW6VMUM3JVR7VJ4Y4V4EFZ4YKQ7HQ, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.412549485 (429h34m43.28363253s) [noise=true , address=145.79.227.52:4446 , piece_id=PU54PSDQBMMYEONFTQT5IDII5K3PU4RAFZ5LMHOOM7YNAJS47OXA, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.432973536 (429h34m43.304056571s) [noise=true , address=145.79.227.34:4905 , piece_id=D54AOI2BG6TPMWQSOJIQFFPQUQRMNYVFDY4PI7JK2NMFHJZAI23Q, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.445500129 (429h34m43.316583164s) [noise=true , address=145.79.227.32:4835 , piece_id=ULJPGLR3BQ6H2CGZHO7C5XO2EVN7Q7EY5L4NHRXZ3MQ6U7S64MVA, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.456746244 (429h34m43.327829279s) [noise=true , address=145.79.227.12:4765 , piece_id=NOLXPWRRNQXRZ4OA2KMMX5M6FYJR2S3AOGPDIPVHXHQB5HN4GMZA, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.644894828 (429h34m43.515977873s) [noise=true , address=45.91.116.5:4297 , piece_id=TQXE3Q6QFA7F63DGJTQKNWEHIP2TOVRCPRVGCX6XC57BY7PJB3UQ, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.713352172 (429h34m43.584435217s) [noise=true , address=45.91.116.6:4976 , piece_id=Y72PN4WXYYPJ2ZNRBR2XA6QMHHTZ6OYOGNCWDBVLZKTO3I4S5KZQ, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.731354943 (429h34m43.602437988s) [noise=true , address=45.91.116.14:4191 , piece_id=J4OQFX6XPN6GB6RJIVPBZUIBDDPLCWMM6GYGJGT7BAZAUM4DBNCA, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.731377275 (429h34m43.602460311s) [seg_pos={0 0}, upload=377134]: Segment reached optimal threshold of 13 pieces. Dec 23 13:03:43.731577968 (429h34m43.602661013s) [noise=true , address=77.90.31.13:4629 , piece_id=VX2CVI5FXKWL3W6ZFTRXG5RREY37EBUEQAHSQVPGBTJ63SC7XA4Q, seg_pos={0 0}, upload=377134]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: write tcp 188.166.225.170:54770->77.90.31.13:4629: use of closed network connection; write tcp 188.166.225.170:54770->77.90.31.13:4629: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54770->77.90.31.13:4629: use of closed network connection Dec 23 13:03:43.731585332 (429h34m43.602668377s) [noise=true , address=77.90.31.13:4629 , piece_id=VX2CVI5FXKWL3W6ZFTRXG5RREY37EBUEQAHSQVPGBTJ63SC7XA4Q, seg_pos={0 0}, upload=377134]: Long tail cancellation due to optimal threshold Dec 23 13:03:43.731671216 (429h34m43.602754261s) [noise=true , address=45.91.116.12:4226 , piece_id=BN76ETRXTTAEDWXZWR3A4KEOHHVKV4OFMG2GE6UOV7DIBE2WHIZA, seg_pos={0 0}, upload=377134]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 23 13:03:43.731682317 (429h34m43.602765362s) [noise=true , address=45.91.116.12:4226 , piece_id=BN76ETRXTTAEDWXZWR3A4KEOHHVKV4OFMG2GE6UOV7DIBE2WHIZA, seg_pos={0 0}, upload=377134]: Long tail cancellation due to optimal threshold Dec 23 13:03:43.731838263 (429h34m43.602921308s) [noise=true , address=45.91.116.11:4120 , piece_id=FODNYYBM7M4TD5KQZYKWGSQXULEIPEBPE6BTUPF5AZZUZSLQ7ACQ, seg_pos={0 0}, upload=377134]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 188.166.225.170:43872->45.91.116.11:4120: use of closed network connection; write tcp 188.166.225.170:43872->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43872->45.91.116.11:4120: use of closed network connection Dec 23 13:03:43.731847792 (429h34m43.602930837s) [noise=true , address=45.91.116.11:4120 , piece_id=FODNYYBM7M4TD5KQZYKWGSQXULEIPEBPE6BTUPF5AZZUZSLQ7ACQ, seg_pos={0 0}, upload=377134]: Long tail cancellation due to optimal threshold Dec 23 13:03:43.731869032 (429h34m43.602952067s) [seg_pos={0 0}, upload=377134]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 13:03:43.731968902 (429h34m43.603051947s) [upload=377134]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 13:03:43.731975024 (429h34m43.603058069s) [upload=377134]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 13:03:44.027725380 (429h34m43.898808415s) [upload=377134]: Done uploading segments... Dec 23 13:03:44.027744758 (429h34m43.898827793s) [upload=377134]: Upload finished. err: <nil> Duration: 6.083580987s core uplink us-select-1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [.........................................X..................] duration: 236.681991ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/73921c1a3ca0437b FAIL (2025-12-23 13:05:35 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 236.710395ms edge linksharing bgp eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................................X.X..] duration: 5.73520449s trace: https://jaeger.dev.storj.io/trace/335d60b5e3cb0611 FAIL (2025-12-23 13:05:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== gments... Dec 23 13:03:41.146773219 (429h34m41.017856254s) [upload=377134]: Got next segment. Inline: false Dec 23 13:03:41.146815550 (429h34m41.017898585s) [upload=377134]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 13:03:41.146820509 (429h34m41.017903544s) [upload=377134]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 13:03:41.395177081 (429h34m41.266260127s) [seg_pos={0 0}, upload=377134]: Begin upload segment... Dec 23 13:03:41.395218049 (429h34m41.266301084s) [seg_pos={0 0}, upload=377134]: Done begin upload segment. Dec 23 13:03:41.395250882 (429h34m41.266333927s) [noise=true , address=45.91.116.12:4226 , piece_id=BN76ETRXTTAEDWXZWR3A4KEOHHVKV4OFMG2GE6UOV7DIBE2WHIZA, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.395268705 (429h34m41.26635175s) [noise=true , address=45.91.116.11:4120 , piece_id=FODNYYBM7M4TD5KQZYKWGSQXULEIPEBPE6BTUPF5AZZUZSLQ7ACQ, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.395658838 (429h34m41.266741873s) [noise=true , address=45.91.116.5:4297 , piece_id=TQXE3Q6QFA7F63DGJTQKNWEHIP2TOVRCPRVGCX6XC57BY7PJB3UQ, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.395663367 (429h34m41.266746413s) [noise=true , address=145.79.227.20:4438 , piece_id=TAOBXUAGOSBITQLEPKPRC7CUW6VMUM3JVR7VJ4Y4V4EFZ4YKQ7HQ, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.395962717 (429h34m41.267045762s) [noise=true , address=77.90.31.13:4629 , piece_id=VX2CVI5FXKWL3W6ZFTRXG5RREY37EBUEQAHSQVPGBTJ63SC7XA4Q, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.396003695 (429h34m41.26708673s) [noise=true , address=145.79.227.52:4446 , piece_id=PU54PSDQBMMYEONFTQT5IDII5K3PU4RAFZ5LMHOOM7YNAJS47OXA, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.396288427 (429h34m41.267371462s) [noise=true , address=145.79.227.11:4731 , piece_id=3N5KV6DMUF4QI2ZD3WB55AMP6X2YAFWCWP7VM4OBFPFB5UZEPGZQ, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.396312643 (429h34m41.267395678s) [noise=true , address=145.79.227.34:4905 , piece_id=D54AOI2BG6TPMWQSOJIQFFPQUQRMNYVFDY4PI7JK2NMFHJZAI23Q, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.396575905 (429h34m41.26765894s) [noise=true , address=145.79.227.29:4942 , piece_id=2HHIYW2ZIIAKDV4CZCO74BAVPK457IVGTE4EJVGANHF65D5W7PNA, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.396676416 (429h34m41.267759461s) [noise=true , address=145.79.227.37:4565 , piece_id=HBJ366WIOAHI6ADVVEEDQQWXUS2E5KPUSFEP2KL4XO2X5LOTFKYQ, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.396865416 (429h34m41.267948451s) [noise=true , address=145.79.227.21:4473 , piece_id=7BR47RIYUFP5FWPABRZROLAAMKGHEPZPL3IDLGQ3XQNKXAMJ6URA, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.397008699 (429h34m41.268091734s) [noise=true , address=45.91.116.6:4976 , piece_id=Y72PN4WXYYPJ2ZNRBR2XA6QMHHTZ6OYOGNCWDBVLZKTO3I4S5KZQ, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.397165558 (429h34m41.268248593s) [noise=true , address=145.79.227.12:4765 , piece_id=NOLXPWRRNQXRZ4OA2KMMX5M6FYJR2S3AOGPDIPVHXHQB5HN4GMZA, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.397266479 (429h34m41.268349514s) [noise=true , address=45.91.116.14:4191 , piece_id=J4OQFX6XPN6GB6RJIVPBZUIBDDPLCWMM6GYGJGT7BAZAUM4DBNCA, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.397521615 (429h34m41.268604661s) [noise=true , address=145.79.227.38:4977 , piece_id=O6QO5Q3IFUTUCDOK7HXTJUYKTDJ5RBWDYB4NGEAMZOHDDR3IIC4A, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.398961146 (429h34m41.270044191s) [noise=true , address=145.79.227.32:4835 , piece_id=ULJPGLR3BQ6H2CGZHO7C5XO2EVN7Q7EY5L4NHRXZ3MQ6U7S64MVA, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:43.108995170 (429h34m42.980078205s) [upload=377134]: Next returned nil segment Dec 23 13:03:43.109012081 (429h34m42.980095146s) [upload=377134]: Waiting for error group managing segments... Dec 23 13:03:43.392567427 (429h34m43.263650462s) [noise=true , address=145.79.227.37:4565 , piece_id=HBJ366WIOAHI6ADVVEEDQQWXUS2E5KPUSFEP2KL4XO2X5LOTFKYQ, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.402738524 (429h34m43.273821559s) [noise=true , address=145.79.227.11:4731 , piece_id=3N5KV6DMUF4QI2ZD3WB55AMP6X2YAFWCWP7VM4OBFPFB5UZEPGZQ, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.403357101 (429h34m43.274440156s) [noise=true , address=145.79.227.38:4977 , piece_id=O6QO5Q3IFUTUCDOK7HXTJUYKTDJ5RBWDYB4NGEAMZOHDDR3IIC4A, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.407110529 (429h34m43.278193564s) [noise=true , address=145.79.227.21:4473 , piece_id=7BR47RIYUFP5FWPABRZROLAAMKGHEPZPL3IDLGQ3XQNKXAMJ6URA, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.407617795 (429h34m43.27870083s) [noise=true , address=145.79.227.29:4942 , piece_id=2HHIYW2ZIIAKDV4CZCO74BAVPK457IVGTE4EJVGANHF65D5W7PNA, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.408029740 (429h34m43.279112775s) [noise=true , address=145.79.227.20:4438 , piece_id=TAOBXUAGOSBITQLEPKPRC7CUW6VMUM3JVR7VJ4Y4V4EFZ4YKQ7HQ, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.412549485 (429h34m43.28363253s) [noise=true , address=145.79.227.52:4446 , piece_id=PU54PSDQBMMYEONFTQT5IDII5K3PU4RAFZ5LMHOOM7YNAJS47OXA, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.432973536 (429h34m43.304056571s) [noise=true , address=145.79.227.34:4905 , piece_id=D54AOI2BG6TPMWQSOJIQFFPQUQRMNYVFDY4PI7JK2NMFHJZAI23Q, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.445500129 (429h34m43.316583164s) [noise=true , address=145.79.227.32:4835 , piece_id=ULJPGLR3BQ6H2CGZHO7C5XO2EVN7Q7EY5L4NHRXZ3MQ6U7S64MVA, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.456746244 (429h34m43.327829279s) [noise=true , address=145.79.227.12:4765 , piece_id=NOLXPWRRNQXRZ4OA2KMMX5M6FYJR2S3AOGPDIPVHXHQB5HN4GMZA, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.644894828 (429h34m43.515977873s) [noise=true , address=45.91.116.5:4297 , piece_id=TQXE3Q6QFA7F63DGJTQKNWEHIP2TOVRCPRVGCX6XC57BY7PJB3UQ, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.713352172 (429h34m43.584435217s) [noise=true , address=45.91.116.6:4976 , piece_id=Y72PN4WXYYPJ2ZNRBR2XA6QMHHTZ6OYOGNCWDBVLZKTO3I4S5KZQ, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.731354943 (429h34m43.602437988s) [noise=true , address=45.91.116.14:4191 , piece_id=J4OQFX6XPN6GB6RJIVPBZUIBDDPLCWMM6GYGJGT7BAZAUM4DBNCA, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.731377275 (429h34m43.602460311s) [seg_pos={0 0}, upload=377134]: Segment reached optimal threshold of 13 pieces. Dec 23 13:03:43.731577968 (429h34m43.602661013s) [noise=true , address=77.90.31.13:4629 , piece_id=VX2CVI5FXKWL3W6ZFTRXG5RREY37EBUEQAHSQVPGBTJ63SC7XA4Q, seg_pos={0 0}, upload=377134]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: write tcp 188.166.225.170:54770->77.90.31.13:4629: use of closed network connection; write tcp 188.166.225.170:54770->77.90.31.13:4629: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54770->77.90.31.13:4629: use of closed network connection Dec 23 13:03:43.731585332 (429h34m43.602668377s) [noise=true , address=77.90.31.13:4629 , piece_id=VX2CVI5FXKWL3W6ZFTRXG5RREY37EBUEQAHSQVPGBTJ63SC7XA4Q, seg_pos={0 0}, upload=377134]: Long tail cancellation due to optimal threshold Dec 23 13:03:43.731671216 (429h34m43.602754261s) [noise=true , address=45.91.116.12:4226 , piece_id=BN76ETRXTTAEDWXZWR3A4KEOHHVKV4OFMG2GE6UOV7DIBE2WHIZA, seg_pos={0 0}, upload=377134]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 23 13:03:43.731682317 (429h34m43.602765362s) [noise=true , address=45.91.116.12:4226 , piece_id=BN76ETRXTTAEDWXZWR3A4KEOHHVKV4OFMG2GE6UOV7DIBE2WHIZA, seg_pos={0 0}, upload=377134]: Long tail cancellation due to optimal threshold Dec 23 13:03:43.731838263 (429h34m43.602921308s) [noise=true , address=45.91.116.11:4120 , piece_id=FODNYYBM7M4TD5KQZYKWGSQXULEIPEBPE6BTUPF5AZZUZSLQ7ACQ, seg_pos={0 0}, upload=377134]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 188.166.225.170:43872->45.91.116.11:4120: use of closed network connection; write tcp 188.166.225.170:43872->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43872->45.91.116.11:4120: use of closed network connection Dec 23 13:03:43.731847792 (429h34m43.602930837s) [noise=true , address=45.91.116.11:4120 , piece_id=FODNYYBM7M4TD5KQZYKWGSQXULEIPEBPE6BTUPF5AZZUZSLQ7ACQ, seg_pos={0 0}, upload=377134]: Long tail cancellation due to optimal threshold Dec 23 13:03:43.731869032 (429h34m43.602952067s) [seg_pos={0 0}, upload=377134]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 13:03:43.731968902 (429h34m43.603051947s) [upload=377134]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 13:03:43.731975024 (429h34m43.603058069s) [upload=377134]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 13:03:44.027725380 (429h34m43.898808415s) [upload=377134]: Done uploading segments... Dec 23 13:03:44.027744758 (429h34m43.898827793s) [upload=377134]: Upload finished. err: <nil> Duration: 5.735249526s core uplink us-select-1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [..........................................X.................] duration: 236.921847ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/73921c1a3ca0437b FAIL (2025-12-23 13:04:45 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 236.951785ms edge linksharing bgp eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................................................X.X.] duration: 6.021807942s trace: https://jaeger.dev.storj.io/trace/335d60b5e3cb0611 FAIL (2025-12-23 13:04:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== gments... Dec 23 13:03:41.146773219 (429h34m41.017856254s) [upload=377134]: Got next segment. Inline: false Dec 23 13:03:41.146815550 (429h34m41.017898585s) [upload=377134]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 13:03:41.146820509 (429h34m41.017903544s) [upload=377134]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 13:03:41.395177081 (429h34m41.266260127s) [seg_pos={0 0}, upload=377134]: Begin upload segment... Dec 23 13:03:41.395218049 (429h34m41.266301084s) [seg_pos={0 0}, upload=377134]: Done begin upload segment. Dec 23 13:03:41.395250882 (429h34m41.266333927s) [noise=true , address=45.91.116.12:4226 , piece_id=BN76ETRXTTAEDWXZWR3A4KEOHHVKV4OFMG2GE6UOV7DIBE2WHIZA, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.395268705 (429h34m41.26635175s) [noise=true , address=45.91.116.11:4120 , piece_id=FODNYYBM7M4TD5KQZYKWGSQXULEIPEBPE6BTUPF5AZZUZSLQ7ACQ, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.395658838 (429h34m41.266741873s) [noise=true , address=45.91.116.5:4297 , piece_id=TQXE3Q6QFA7F63DGJTQKNWEHIP2TOVRCPRVGCX6XC57BY7PJB3UQ, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.395663367 (429h34m41.266746413s) [noise=true , address=145.79.227.20:4438 , piece_id=TAOBXUAGOSBITQLEPKPRC7CUW6VMUM3JVR7VJ4Y4V4EFZ4YKQ7HQ, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.395962717 (429h34m41.267045762s) [noise=true , address=77.90.31.13:4629 , piece_id=VX2CVI5FXKWL3W6ZFTRXG5RREY37EBUEQAHSQVPGBTJ63SC7XA4Q, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.396003695 (429h34m41.26708673s) [noise=true , address=145.79.227.52:4446 , piece_id=PU54PSDQBMMYEONFTQT5IDII5K3PU4RAFZ5LMHOOM7YNAJS47OXA, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.396288427 (429h34m41.267371462s) [noise=true , address=145.79.227.11:4731 , piece_id=3N5KV6DMUF4QI2ZD3WB55AMP6X2YAFWCWP7VM4OBFPFB5UZEPGZQ, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.396312643 (429h34m41.267395678s) [noise=true , address=145.79.227.34:4905 , piece_id=D54AOI2BG6TPMWQSOJIQFFPQUQRMNYVFDY4PI7JK2NMFHJZAI23Q, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.396575905 (429h34m41.26765894s) [noise=true , address=145.79.227.29:4942 , piece_id=2HHIYW2ZIIAKDV4CZCO74BAVPK457IVGTE4EJVGANHF65D5W7PNA, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.396676416 (429h34m41.267759461s) [noise=true , address=145.79.227.37:4565 , piece_id=HBJ366WIOAHI6ADVVEEDQQWXUS2E5KPUSFEP2KL4XO2X5LOTFKYQ, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.396865416 (429h34m41.267948451s) [noise=true , address=145.79.227.21:4473 , piece_id=7BR47RIYUFP5FWPABRZROLAAMKGHEPZPL3IDLGQ3XQNKXAMJ6URA, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.397008699 (429h34m41.268091734s) [noise=true , address=45.91.116.6:4976 , piece_id=Y72PN4WXYYPJ2ZNRBR2XA6QMHHTZ6OYOGNCWDBVLZKTO3I4S5KZQ, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.397165558 (429h34m41.268248593s) [noise=true , address=145.79.227.12:4765 , piece_id=NOLXPWRRNQXRZ4OA2KMMX5M6FYJR2S3AOGPDIPVHXHQB5HN4GMZA, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.397266479 (429h34m41.268349514s) [noise=true , address=45.91.116.14:4191 , piece_id=J4OQFX6XPN6GB6RJIVPBZUIBDDPLCWMM6GYGJGT7BAZAUM4DBNCA, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.397521615 (429h34m41.268604661s) [noise=true , address=145.79.227.38:4977 , piece_id=O6QO5Q3IFUTUCDOK7HXTJUYKTDJ5RBWDYB4NGEAMZOHDDR3IIC4A, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.398961146 (429h34m41.270044191s) [noise=true , address=145.79.227.32:4835 , piece_id=ULJPGLR3BQ6H2CGZHO7C5XO2EVN7Q7EY5L4NHRXZ3MQ6U7S64MVA, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:43.108995170 (429h34m42.980078205s) [upload=377134]: Next returned nil segment Dec 23 13:03:43.109012081 (429h34m42.980095146s) [upload=377134]: Waiting for error group managing segments... Dec 23 13:03:43.392567427 (429h34m43.263650462s) [noise=true , address=145.79.227.37:4565 , piece_id=HBJ366WIOAHI6ADVVEEDQQWXUS2E5KPUSFEP2KL4XO2X5LOTFKYQ, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.402738524 (429h34m43.273821559s) [noise=true , address=145.79.227.11:4731 , piece_id=3N5KV6DMUF4QI2ZD3WB55AMP6X2YAFWCWP7VM4OBFPFB5UZEPGZQ, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.403357101 (429h34m43.274440156s) [noise=true , address=145.79.227.38:4977 , piece_id=O6QO5Q3IFUTUCDOK7HXTJUYKTDJ5RBWDYB4NGEAMZOHDDR3IIC4A, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.407110529 (429h34m43.278193564s) [noise=true , address=145.79.227.21:4473 , piece_id=7BR47RIYUFP5FWPABRZROLAAMKGHEPZPL3IDLGQ3XQNKXAMJ6URA, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.407617795 (429h34m43.27870083s) [noise=true , address=145.79.227.29:4942 , piece_id=2HHIYW2ZIIAKDV4CZCO74BAVPK457IVGTE4EJVGANHF65D5W7PNA, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.408029740 (429h34m43.279112775s) [noise=true , address=145.79.227.20:4438 , piece_id=TAOBXUAGOSBITQLEPKPRC7CUW6VMUM3JVR7VJ4Y4V4EFZ4YKQ7HQ, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.412549485 (429h34m43.28363253s) [noise=true , address=145.79.227.52:4446 , piece_id=PU54PSDQBMMYEONFTQT5IDII5K3PU4RAFZ5LMHOOM7YNAJS47OXA, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.432973536 (429h34m43.304056571s) [noise=true , address=145.79.227.34:4905 , piece_id=D54AOI2BG6TPMWQSOJIQFFPQUQRMNYVFDY4PI7JK2NMFHJZAI23Q, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.445500129 (429h34m43.316583164s) [noise=true , address=145.79.227.32:4835 , piece_id=ULJPGLR3BQ6H2CGZHO7C5XO2EVN7Q7EY5L4NHRXZ3MQ6U7S64MVA, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.456746244 (429h34m43.327829279s) [noise=true , address=145.79.227.12:4765 , piece_id=NOLXPWRRNQXRZ4OA2KMMX5M6FYJR2S3AOGPDIPVHXHQB5HN4GMZA, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.644894828 (429h34m43.515977873s) [noise=true , address=45.91.116.5:4297 , piece_id=TQXE3Q6QFA7F63DGJTQKNWEHIP2TOVRCPRVGCX6XC57BY7PJB3UQ, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.713352172 (429h34m43.584435217s) [noise=true , address=45.91.116.6:4976 , piece_id=Y72PN4WXYYPJ2ZNRBR2XA6QMHHTZ6OYOGNCWDBVLZKTO3I4S5KZQ, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.731354943 (429h34m43.602437988s) [noise=true , address=45.91.116.14:4191 , piece_id=J4OQFX6XPN6GB6RJIVPBZUIBDDPLCWMM6GYGJGT7BAZAUM4DBNCA, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.731377275 (429h34m43.602460311s) [seg_pos={0 0}, upload=377134]: Segment reached optimal threshold of 13 pieces. Dec 23 13:03:43.731577968 (429h34m43.602661013s) [noise=true , address=77.90.31.13:4629 , piece_id=VX2CVI5FXKWL3W6ZFTRXG5RREY37EBUEQAHSQVPGBTJ63SC7XA4Q, seg_pos={0 0}, upload=377134]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: write tcp 188.166.225.170:54770->77.90.31.13:4629: use of closed network connection; write tcp 188.166.225.170:54770->77.90.31.13:4629: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54770->77.90.31.13:4629: use of closed network connection Dec 23 13:03:43.731585332 (429h34m43.602668377s) [noise=true , address=77.90.31.13:4629 , piece_id=VX2CVI5FXKWL3W6ZFTRXG5RREY37EBUEQAHSQVPGBTJ63SC7XA4Q, seg_pos={0 0}, upload=377134]: Long tail cancellation due to optimal threshold Dec 23 13:03:43.731671216 (429h34m43.602754261s) [noise=true , address=45.91.116.12:4226 , piece_id=BN76ETRXTTAEDWXZWR3A4KEOHHVKV4OFMG2GE6UOV7DIBE2WHIZA, seg_pos={0 0}, upload=377134]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 23 13:03:43.731682317 (429h34m43.602765362s) [noise=true , address=45.91.116.12:4226 , piece_id=BN76ETRXTTAEDWXZWR3A4KEOHHVKV4OFMG2GE6UOV7DIBE2WHIZA, seg_pos={0 0}, upload=377134]: Long tail cancellation due to optimal threshold Dec 23 13:03:43.731838263 (429h34m43.602921308s) [noise=true , address=45.91.116.11:4120 , piece_id=FODNYYBM7M4TD5KQZYKWGSQXULEIPEBPE6BTUPF5AZZUZSLQ7ACQ, seg_pos={0 0}, upload=377134]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 188.166.225.170:43872->45.91.116.11:4120: use of closed network connection; write tcp 188.166.225.170:43872->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43872->45.91.116.11:4120: use of closed network connection Dec 23 13:03:43.731847792 (429h34m43.602930837s) [noise=true , address=45.91.116.11:4120 , piece_id=FODNYYBM7M4TD5KQZYKWGSQXULEIPEBPE6BTUPF5AZZUZSLQ7ACQ, seg_pos={0 0}, upload=377134]: Long tail cancellation due to optimal threshold Dec 23 13:03:43.731869032 (429h34m43.602952067s) [seg_pos={0 0}, upload=377134]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 13:03:43.731968902 (429h34m43.603051947s) [upload=377134]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 13:03:43.731975024 (429h34m43.603058069s) [upload=377134]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 13:03:44.027725380 (429h34m43.898808415s) [upload=377134]: Done uploading segments... Dec 23 13:03:44.027744758 (429h34m43.898827793s) [upload=377134]: Upload finished. err: <nil> Duration: 6.021834152s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................................................X.X] duration: 30.002161024s trace: https://jaeger.dev.storj.io/trace/335d60b5e3cb0611 FAIL (2025-12-23 13:04:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== gments... Dec 23 13:03:41.146773219 (429h34m41.017856254s) [upload=377134]: Got next segment. Inline: false Dec 23 13:03:41.146815550 (429h34m41.017898585s) [upload=377134]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 13:03:41.146820509 (429h34m41.017903544s) [upload=377134]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 13:03:41.395177081 (429h34m41.266260127s) [seg_pos={0 0}, upload=377134]: Begin upload segment... Dec 23 13:03:41.395218049 (429h34m41.266301084s) [seg_pos={0 0}, upload=377134]: Done begin upload segment. Dec 23 13:03:41.395250882 (429h34m41.266333927s) [noise=true , address=45.91.116.12:4226 , piece_id=BN76ETRXTTAEDWXZWR3A4KEOHHVKV4OFMG2GE6UOV7DIBE2WHIZA, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.395268705 (429h34m41.26635175s) [noise=true , address=45.91.116.11:4120 , piece_id=FODNYYBM7M4TD5KQZYKWGSQXULEIPEBPE6BTUPF5AZZUZSLQ7ACQ, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.395658838 (429h34m41.266741873s) [noise=true , address=45.91.116.5:4297 , piece_id=TQXE3Q6QFA7F63DGJTQKNWEHIP2TOVRCPRVGCX6XC57BY7PJB3UQ, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.395663367 (429h34m41.266746413s) [noise=true , address=145.79.227.20:4438 , piece_id=TAOBXUAGOSBITQLEPKPRC7CUW6VMUM3JVR7VJ4Y4V4EFZ4YKQ7HQ, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.395962717 (429h34m41.267045762s) [noise=true , address=77.90.31.13:4629 , piece_id=VX2CVI5FXKWL3W6ZFTRXG5RREY37EBUEQAHSQVPGBTJ63SC7XA4Q, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.396003695 (429h34m41.26708673s) [noise=true , address=145.79.227.52:4446 , piece_id=PU54PSDQBMMYEONFTQT5IDII5K3PU4RAFZ5LMHOOM7YNAJS47OXA, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.396288427 (429h34m41.267371462s) [noise=true , address=145.79.227.11:4731 , piece_id=3N5KV6DMUF4QI2ZD3WB55AMP6X2YAFWCWP7VM4OBFPFB5UZEPGZQ, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.396312643 (429h34m41.267395678s) [noise=true , address=145.79.227.34:4905 , piece_id=D54AOI2BG6TPMWQSOJIQFFPQUQRMNYVFDY4PI7JK2NMFHJZAI23Q, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.396575905 (429h34m41.26765894s) [noise=true , address=145.79.227.29:4942 , piece_id=2HHIYW2ZIIAKDV4CZCO74BAVPK457IVGTE4EJVGANHF65D5W7PNA, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.396676416 (429h34m41.267759461s) [noise=true , address=145.79.227.37:4565 , piece_id=HBJ366WIOAHI6ADVVEEDQQWXUS2E5KPUSFEP2KL4XO2X5LOTFKYQ, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.396865416 (429h34m41.267948451s) [noise=true , address=145.79.227.21:4473 , piece_id=7BR47RIYUFP5FWPABRZROLAAMKGHEPZPL3IDLGQ3XQNKXAMJ6URA, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.397008699 (429h34m41.268091734s) [noise=true , address=45.91.116.6:4976 , piece_id=Y72PN4WXYYPJ2ZNRBR2XA6QMHHTZ6OYOGNCWDBVLZKTO3I4S5KZQ, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.397165558 (429h34m41.268248593s) [noise=true , address=145.79.227.12:4765 , piece_id=NOLXPWRRNQXRZ4OA2KMMX5M6FYJR2S3AOGPDIPVHXHQB5HN4GMZA, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.397266479 (429h34m41.268349514s) [noise=true , address=45.91.116.14:4191 , piece_id=J4OQFX6XPN6GB6RJIVPBZUIBDDPLCWMM6GYGJGT7BAZAUM4DBNCA, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.397521615 (429h34m41.268604661s) [noise=true , address=145.79.227.38:4977 , piece_id=O6QO5Q3IFUTUCDOK7HXTJUYKTDJ5RBWDYB4NGEAMZOHDDR3IIC4A, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:41.398961146 (429h34m41.270044191s) [noise=true , address=145.79.227.32:4835 , piece_id=ULJPGLR3BQ6H2CGZHO7C5XO2EVN7Q7EY5L4NHRXZ3MQ6U7S64MVA, seg_pos={0 0}, upload=377134]: Uploading piece... Dec 23 13:03:43.108995170 (429h34m42.980078205s) [upload=377134]: Next returned nil segment Dec 23 13:03:43.109012081 (429h34m42.980095146s) [upload=377134]: Waiting for error group managing segments... Dec 23 13:03:43.392567427 (429h34m43.263650462s) [noise=true , address=145.79.227.37:4565 , piece_id=HBJ366WIOAHI6ADVVEEDQQWXUS2E5KPUSFEP2KL4XO2X5LOTFKYQ, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.402738524 (429h34m43.273821559s) [noise=true , address=145.79.227.11:4731 , piece_id=3N5KV6DMUF4QI2ZD3WB55AMP6X2YAFWCWP7VM4OBFPFB5UZEPGZQ, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.403357101 (429h34m43.274440156s) [noise=true , address=145.79.227.38:4977 , piece_id=O6QO5Q3IFUTUCDOK7HXTJUYKTDJ5RBWDYB4NGEAMZOHDDR3IIC4A, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.407110529 (429h34m43.278193564s) [noise=true , address=145.79.227.21:4473 , piece_id=7BR47RIYUFP5FWPABRZROLAAMKGHEPZPL3IDLGQ3XQNKXAMJ6URA, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.407617795 (429h34m43.27870083s) [noise=true , address=145.79.227.29:4942 , piece_id=2HHIYW2ZIIAKDV4CZCO74BAVPK457IVGTE4EJVGANHF65D5W7PNA, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.408029740 (429h34m43.279112775s) [noise=true , address=145.79.227.20:4438 , piece_id=TAOBXUAGOSBITQLEPKPRC7CUW6VMUM3JVR7VJ4Y4V4EFZ4YKQ7HQ, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.412549485 (429h34m43.28363253s) [noise=true , address=145.79.227.52:4446 , piece_id=PU54PSDQBMMYEONFTQT5IDII5K3PU4RAFZ5LMHOOM7YNAJS47OXA, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.432973536 (429h34m43.304056571s) [noise=true , address=145.79.227.34:4905 , piece_id=D54AOI2BG6TPMWQSOJIQFFPQUQRMNYVFDY4PI7JK2NMFHJZAI23Q, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.445500129 (429h34m43.316583164s) [noise=true , address=145.79.227.32:4835 , piece_id=ULJPGLR3BQ6H2CGZHO7C5XO2EVN7Q7EY5L4NHRXZ3MQ6U7S64MVA, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.456746244 (429h34m43.327829279s) [noise=true , address=145.79.227.12:4765 , piece_id=NOLXPWRRNQXRZ4OA2KMMX5M6FYJR2S3AOGPDIPVHXHQB5HN4GMZA, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.644894828 (429h34m43.515977873s) [noise=true , address=45.91.116.5:4297 , piece_id=TQXE3Q6QFA7F63DGJTQKNWEHIP2TOVRCPRVGCX6XC57BY7PJB3UQ, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.713352172 (429h34m43.584435217s) [noise=true , address=45.91.116.6:4976 , piece_id=Y72PN4WXYYPJ2ZNRBR2XA6QMHHTZ6OYOGNCWDBVLZKTO3I4S5KZQ, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.731354943 (429h34m43.602437988s) [noise=true , address=45.91.116.14:4191 , piece_id=J4OQFX6XPN6GB6RJIVPBZUIBDDPLCWMM6GYGJGT7BAZAUM4DBNCA, seg_pos={0 0}, upload=377134]: Done uploading piece. err: <nil> Dec 23 13:03:43.731377275 (429h34m43.602460311s) [seg_pos={0 0}, upload=377134]: Segment reached optimal threshold of 13 pieces. Dec 23 13:03:43.731577968 (429h34m43.602661013s) [noise=true , address=77.90.31.13:4629 , piece_id=VX2CVI5FXKWL3W6ZFTRXG5RREY37EBUEQAHSQVPGBTJ63SC7XA4Q, seg_pos={0 0}, upload=377134]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: write tcp 188.166.225.170:54770->77.90.31.13:4629: use of closed network connection; write tcp 188.166.225.170:54770->77.90.31.13:4629: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54770->77.90.31.13:4629: use of closed network connection Dec 23 13:03:43.731585332 (429h34m43.602668377s) [noise=true , address=77.90.31.13:4629 , piece_id=VX2CVI5FXKWL3W6ZFTRXG5RREY37EBUEQAHSQVPGBTJ63SC7XA4Q, seg_pos={0 0}, upload=377134]: Long tail cancellation due to optimal threshold Dec 23 13:03:43.731671216 (429h34m43.602754261s) [noise=true , address=45.91.116.12:4226 , piece_id=BN76ETRXTTAEDWXZWR3A4KEOHHVKV4OFMG2GE6UOV7DIBE2WHIZA, seg_pos={0 0}, upload=377134]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 23 13:03:43.731682317 (429h34m43.602765362s) [noise=true , address=45.91.116.12:4226 , piece_id=BN76ETRXTTAEDWXZWR3A4KEOHHVKV4OFMG2GE6UOV7DIBE2WHIZA, seg_pos={0 0}, upload=377134]: Long tail cancellation due to optimal threshold Dec 23 13:03:43.731838263 (429h34m43.602921308s) [noise=true , address=45.91.116.11:4120 , piece_id=FODNYYBM7M4TD5KQZYKWGSQXULEIPEBPE6BTUPF5AZZUZSLQ7ACQ, seg_pos={0 0}, upload=377134]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 188.166.225.170:43872->45.91.116.11:4120: use of closed network connection; write tcp 188.166.225.170:43872->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43872->45.91.116.11:4120: use of closed network connection Dec 23 13:03:43.731847792 (429h34m43.602930837s) [noise=true , address=45.91.116.11:4120 , piece_id=FODNYYBM7M4TD5KQZYKWGSQXULEIPEBPE6BTUPF5AZZUZSLQ7ACQ, seg_pos={0 0}, upload=377134]: Long tail cancellation due to optimal threshold Dec 23 13:03:43.731869032 (429h34m43.602952067s) [seg_pos={0 0}, upload=377134]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 13:03:43.731968902 (429h34m43.603051947s) [upload=377134]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 13:03:43.731975024 (429h34m43.603058069s) [upload=377134]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 13:03:44.027725380 (429h34m43.898808415s) [upload=377134]: Done uploading segments... Dec 23 13:03:44.027744758 (429h34m43.898827793s) [upload=377134]: Upload finished. err: <nil> Duration: 30.002180441s core uplink us-select-1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [...........................................X................] duration: 255.068302ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/73921c1a3ca0437b FAIL (2025-12-23 13:03:41 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 255.097036ms edge linksharing bgp eu1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [............................................X...............] duration: 739.030375ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/73921c1a3ca0437b FAIL (2025-12-23 13:02:40 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 739.063146ms edge linksharing bgp eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 5.916113694s trace: https://jaeger.dev.storj.io/trace/3be401a4bb61f3de FAIL (2025-12-23 13:02:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 23 13:01:50.209856776 (429h32m50.080939811s) [upload=377105]: Starting upload Dec 23 13:01:50.209920386 (429h32m50.081003431s) [upload=377105]: Done starting upload Dec 23 13:01:50.210036468 (429h32m50.081119503s) [upload=377105]: Uploading segments... Dec 23 13:01:50.210133573 (429h32m50.081216628s) [upload=377105]: Got next segment. Inline: false Dec 23 13:01:50.210170113 (429h32m50.081253148s) [upload=377105]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 13:01:50.210175563 (429h32m50.081258608s) [upload=377105]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 13:01:50.472723519 (429h32m50.343806565s) [seg_pos={0 0}, upload=377105]: Begin upload segment... Dec 23 13:01:50.472775920 (429h32m50.343858955s) [seg_pos={0 0}, upload=377105]: Done begin upload segment. Dec 23 13:01:50.472814092 (429h32m50.343897137s) [noise=true , address=45.91.116.14:4190 , piece_id=YOUFVOVGFK6BXESHHXYY2DOEH4E3YIJY3QAEUFTJHKS65QTQJNKQ, seg_pos={0 0}, upload=377105]: Uploading piece... Dec 23 13:01:50.472921236 (429h32m50.344004271s) [noise=true , address=145.79.227.34:4905 , piece_id=NXTPONXXJUZFPRGBMGSKLDZPMT5UOBNKCGDVAUF6NSUOTWJRHURA, seg_pos={0 0}, upload=377105]: Uploading piece... Dec 23 13:01:50.473366224 (429h32m50.344449259s) [noise=true , address=145.79.227.11:4731 , piece_id=QTFACZA3XAEVSNFK45GYZCYWOXWJ4NYWEXZAKEC5PK5B7HWEFA4A, seg_pos={0 0}, upload=377105]: Uploading piece... Dec 23 13:01:50.473633814 (429h32m50.344716859s) [noise=true , address=145.79.227.49:4339 , piece_id=4DUBWY5NTO4XGJAIDZCHZJLBJWW2Y3KPLIDZ3WPVQFZDXVFDMYUQ, seg_pos={0 0}, upload=377105]: Uploading piece... Dec 23 13:01:50.474052280 (429h32m50.345135326s) [noise=true , address=45.91.116.6:4976 , piece_id=LJDYCGUIKJMIZOSYTFLXK2OXRTOA34P7FAPSZFLTAXJ6OBKSTYXA, seg_pos={0 0}, upload=377105]: Uploading piece... Dec 23 13:01:50.474494532 (429h32m50.345577607s) [noise=true , address=145.79.227.42:4121 , piece_id=A4NA3V7BEWENLDQHGRTQ7HQ3NPN3GQ5URTNE7NUBVI5J2JL6X5UQ, seg_pos={0 0}, upload=377105]: Uploading piece... Dec 23 13:01:50.474919070 (429h32m50.346002105s) [noise=true , address=145.79.227.37:4566 , piece_id=NFDUVZWXADEFCJ3RSQGUG3NX27MY4B3DFCDZ43VR6U6XUKSALDVQ, seg_pos={0 0}, upload=377105]: Uploading piece... Dec 23 13:01:50.475282463 (429h32m50.346365508s) [noise=true , address=45.91.116.13:4155 , piece_id=AJ5OPEXWZSA24RN7QZOIFPANNTQSZIDVHBUVUMSSSH565DTS34EA, seg_pos={0 0}, upload=377105]: Uploading piece... Dec 23 13:01:50.475612861 (429h32m50.346695906s) [noise=true , address=145.79.227.46:4602 , piece_id=QHT4HMOHUETZLBGJKFCH3MQID7OP3J32S42UNPYUJQXQH6ETLUUQ, seg_pos={0 0}, upload=377105]: Uploading piece... Dec 23 13:01:50.477151171 (429h32m50.348234216s) [noise=true , address=145.79.227.45:4230 , piece_id=V5XP4N6OE6Q4QEOB5W7D2J7U3SP67SRPAK2IJMVPXICFQJHWUJAA, seg_pos={0 0}, upload=377105]: Uploading piece... Dec 23 13:01:50.477276479 (429h32m50.348359514s) [noise=true , address=145.79.227.48:4302 , piece_id=OBV2AHJTPXQ7JN7Q4AFLOGJU3PX6C7ROSAOFPHIYIUFZG4LQHUOQ, seg_pos={0 0}, upload=377105]: Uploading piece... Dec 23 13:01:50.478712724 (429h32m50.349795759s) [noise=true , address=145.79.227.13:4804 , piece_id=5WNZNWL557DHGBDAELUNQWVQMHC52T5NISQ7XESK7N7AB7SM4XEQ, seg_pos={0 0}, upload=377105]: Uploading piece... Dec 23 13:01:50.480160231 (429h32m50.351243266s) [noise=true , address=45.91.116.11:4118 , piece_id=NC472UBH2DMEFMPNEWMVRWQYMOV7GUGPLATVFKPGZAIUKIEVW6SQ, seg_pos={0 0}, upload=377105]: Uploading piece... Dec 23 13:01:50.481616634 (429h32m50.352699669s) [noise=true , address=145.79.227.4:4187 , piece_id=33LHWNAE3TLAKMCXYQYMCVWOE4WOHXLUU52RJ4UCVIX5QOVUTRCA, seg_pos={0 0}, upload=377105]: Uploading piece... Dec 23 13:01:50.482751314 (429h32m50.353834359s) [noise=true , address=45.91.116.5:4300 , piece_id=2GII6PLQAGYWPV3NM3WM7BP5CKLPH5FGN76QZVDOL5SMWS3U2V7Q, seg_pos={0 0}, upload=377105]: Uploading piece... Dec 23 13:01:50.483909200 (429h32m50.354992235s) [noise=true , address=145.79.227.29:4942 , piece_id=PZTO6TJE3NI4APPCBG6PUP7HBXKFGZMCBKU5X2Z573WYGZQLIDWQ, seg_pos={0 0}, upload=377105]: Uploading piece... Dec 23 13:01:52.190684160 (429h32m52.061767205s) [upload=377105]: Next returned nil segment Dec 23 13:01:52.190700431 (429h32m52.061783476s) [upload=377105]: Waiting for error group managing segments... Dec 23 13:01:52.390085625 (429h32m52.26116866s) [noise=true , address=45.91.116.14:4190 , piece_id=YOUFVOVGFK6BXESHHXYY2DOEH4E3YIJY3QAEUFTJHKS65QTQJNKQ, seg_pos={0 0}, upload=377105]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: storage node overloaded, request limit: 2000 Dec 23 13:01:52.465012610 (429h32m52.336095655s) [noise=true , address=145.79.227.34:4905 , piece_id=NXTPONXXJUZFPRGBMGSKLDZPMT5UOBNKCGDVAUF6NSUOTWJRHURA, seg_pos={0 0}, upload=377105]: Done uploading piece. err: <nil> Dec 23 13:01:52.473133103 (429h32m52.344216138s) [noise=true , address=145.79.227.45:4230 , piece_id=V5XP4N6OE6Q4QEOB5W7D2J7U3SP67SRPAK2IJMVPXICFQJHWUJAA, seg_pos={0 0}, upload=377105]: Done uploading piece. err: <nil> Dec 23 13:01:52.482395581 (429h32m52.353478616s) [noise=true , address=145.79.227.37:4566 , piece_id=NFDUVZWXADEFCJ3RSQGUG3NX27MY4B3DFCDZ43VR6U6XUKSALDVQ, seg_pos={0 0}, upload=377105]: Done uploading piece. err: <nil> Dec 23 13:01:52.482940177 (429h32m52.354023212s) [noise=true , address=145.79.227.11:4731 , piece_id=QTFACZA3XAEVSNFK45GYZCYWOXWJ4NYWEXZAKEC5PK5B7HWEFA4A, seg_pos={0 0}, upload=377105]: Done uploading piece. err: <nil> Dec 23 13:01:52.482961087 (429h32m52.354044122s) [noise=true , address=145.79.227.49:4339 , piece_id=4DUBWY5NTO4XGJAIDZCHZJLBJWW2Y3KPLIDZ3WPVQFZDXVFDMYUQ, seg_pos={0 0}, upload=377105]: Done uploading piece. err: <nil> Dec 23 13:01:52.483342393 (429h32m52.354425428s) [noise=true , address=145.79.227.46:4602 , piece_id=QHT4HMOHUETZLBGJKFCH3MQID7OP3J32S42UNPYUJQXQH6ETLUUQ, seg_pos={0 0}, upload=377105]: Done uploading piece. err: <nil> Dec 23 13:01:52.489414255 (429h32m52.36049729s) [noise=true , address=145.79.227.4:4187 , piece_id=33LHWNAE3TLAKMCXYQYMCVWOE4WOHXLUU52RJ4UCVIX5QOVUTRCA, seg_pos={0 0}, upload=377105]: Done uploading piece. err: <nil> Dec 23 13:01:52.490255778 (429h32m52.361338813s) [noise=true , address=145.79.227.42:4121 , piece_id=A4NA3V7BEWENLDQHGRTQ7HQ3NPN3GQ5URTNE7NUBVI5J2JL6X5UQ, seg_pos={0 0}, upload=377105]: Done uploading piece. err: <nil> Dec 23 13:01:52.503142076 (429h32m52.374225111s) [noise=true , address=145.79.227.13:4804 , piece_id=5WNZNWL557DHGBDAELUNQWVQMHC52T5NISQ7XESK7N7AB7SM4XEQ, seg_pos={0 0}, upload=377105]: Done uploading piece. err: <nil> Dec 23 13:01:52.505736105 (429h32m52.37681914s) [noise=true , address=145.79.227.29:4942 , piece_id=PZTO6TJE3NI4APPCBG6PUP7HBXKFGZMCBKU5X2Z573WYGZQLIDWQ, seg_pos={0 0}, upload=377105]: Done uploading piece. err: <nil> Dec 23 13:01:52.745151584 (429h32m52.616234619s) [noise=true , address=145.79.227.48:4302 , piece_id=OBV2AHJTPXQ7JN7Q4AFLOGJU3PX6C7ROSAOFPHIYIUFZG4LQHUOQ, seg_pos={0 0}, upload=377105]: Done uploading piece. err: <nil> Dec 23 13:01:53.049982346 (429h32m52.921065401s) [noise=true , address=45.91.116.5:4300 , piece_id=2GII6PLQAGYWPV3NM3WM7BP5CKLPH5FGN76QZVDOL5SMWS3U2V7Q, seg_pos={0 0}, upload=377105]: Done uploading piece. err: <nil> Dec 23 13:01:53.065313249 (429h32m52.936396294s) [noise=true , address=45.91.116.6:4976 , piece_id=LJDYCGUIKJMIZOSYTFLXK2OXRTOA34P7FAPSZFLTAXJ6OBKSTYXA, seg_pos={0 0}, upload=377105]: Done uploading piece. err: <nil> Dec 23 13:01:53.065341903 (429h32m52.936424938s) [seg_pos={0 0}, upload=377105]: Segment reached optimal threshold of 13 pieces. Dec 23 13:01:53.065589845 (429h32m52.93667288s) [noise=true , address=45.91.116.11:4118 , piece_id=NC472UBH2DMEFMPNEWMVRWQYMOV7GUGPLATVFKPGZAIUKIEVW6SQ, seg_pos={0 0}, upload=377105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF Dec 23 13:01:53.065598843 (429h32m52.936681878s) [noise=true , address=45.91.116.11:4118 , piece_id=NC472UBH2DMEFMPNEWMVRWQYMOV7GUGPLATVFKPGZAIUKIEVW6SQ, seg_pos={0 0}, upload=377105]: Long tail cancellation due to optimal threshold Dec 23 13:01:53.065869357 (429h32m52.936952392s) [noise=true , address=45.91.116.13:4155 , piece_id=AJ5OPEXWZSA24RN7QZOIFPANNTQSZIDVHBUVUMSSSH565DTS34EA, seg_pos={0 0}, upload=377105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: write tcp 188.166.225.170:40332->45.91.116.13:4155: use of closed network connection; write tcp 188.166.225.170:40332->45.91.116.13:4155: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40332->45.91.116.13:4155: use of closed network connection Dec 23 13:01:53.065880539 (429h32m52.936963574s) [noise=true , address=45.91.116.13:4155 , piece_id=AJ5OPEXWZSA24RN7QZOIFPANNTQSZIDVHBUVUMSSSH565DTS34EA, seg_pos={0 0}, upload=377105]: Long tail cancellation due to optimal threshold Dec 23 13:01:53.065925935 (429h32m52.93700899s) [seg_pos={0 0}, upload=377105]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 13:01:53.066048309 (429h32m52.937131364s) [upload=377105]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 13:01:53.066056264 (429h32m52.937139309s) [upload=377105]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 13:01:53.356946671 (429h32m53.228029736s) [upload=377105]: Done uploading segments... Dec 23 13:01:53.356975245 (429h32m53.22805829s) [upload=377105]: Upload finished. err: <nil> Duration: 5.916145655s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.002846142s trace: https://jaeger.dev.storj.io/trace/3be401a4bb61f3de FAIL (2025-12-23 13:02:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 23 13:01:50.209856776 (429h32m50.080939811s) [upload=377105]: Starting upload Dec 23 13:01:50.209920386 (429h32m50.081003431s) [upload=377105]: Done starting upload Dec 23 13:01:50.210036468 (429h32m50.081119503s) [upload=377105]: Uploading segments... Dec 23 13:01:50.210133573 (429h32m50.081216628s) [upload=377105]: Got next segment. Inline: false Dec 23 13:01:50.210170113 (429h32m50.081253148s) [upload=377105]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 13:01:50.210175563 (429h32m50.081258608s) [upload=377105]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 13:01:50.472723519 (429h32m50.343806565s) [seg_pos={0 0}, upload=377105]: Begin upload segment... Dec 23 13:01:50.472775920 (429h32m50.343858955s) [seg_pos={0 0}, upload=377105]: Done begin upload segment. Dec 23 13:01:50.472814092 (429h32m50.343897137s) [noise=true , address=45.91.116.14:4190 , piece_id=YOUFVOVGFK6BXESHHXYY2DOEH4E3YIJY3QAEUFTJHKS65QTQJNKQ, seg_pos={0 0}, upload=377105]: Uploading piece... Dec 23 13:01:50.472921236 (429h32m50.344004271s) [noise=true , address=145.79.227.34:4905 , piece_id=NXTPONXXJUZFPRGBMGSKLDZPMT5UOBNKCGDVAUF6NSUOTWJRHURA, seg_pos={0 0}, upload=377105]: Uploading piece... Dec 23 13:01:50.473366224 (429h32m50.344449259s) [noise=true , address=145.79.227.11:4731 , piece_id=QTFACZA3XAEVSNFK45GYZCYWOXWJ4NYWEXZAKEC5PK5B7HWEFA4A, seg_pos={0 0}, upload=377105]: Uploading piece... Dec 23 13:01:50.473633814 (429h32m50.344716859s) [noise=true , address=145.79.227.49:4339 , piece_id=4DUBWY5NTO4XGJAIDZCHZJLBJWW2Y3KPLIDZ3WPVQFZDXVFDMYUQ, seg_pos={0 0}, upload=377105]: Uploading piece... Dec 23 13:01:50.474052280 (429h32m50.345135326s) [noise=true , address=45.91.116.6:4976 , piece_id=LJDYCGUIKJMIZOSYTFLXK2OXRTOA34P7FAPSZFLTAXJ6OBKSTYXA, seg_pos={0 0}, upload=377105]: Uploading piece... Dec 23 13:01:50.474494532 (429h32m50.345577607s) [noise=true , address=145.79.227.42:4121 , piece_id=A4NA3V7BEWENLDQHGRTQ7HQ3NPN3GQ5URTNE7NUBVI5J2JL6X5UQ, seg_pos={0 0}, upload=377105]: Uploading piece... Dec 23 13:01:50.474919070 (429h32m50.346002105s) [noise=true , address=145.79.227.37:4566 , piece_id=NFDUVZWXADEFCJ3RSQGUG3NX27MY4B3DFCDZ43VR6U6XUKSALDVQ, seg_pos={0 0}, upload=377105]: Uploading piece... Dec 23 13:01:50.475282463 (429h32m50.346365508s) [noise=true , address=45.91.116.13:4155 , piece_id=AJ5OPEXWZSA24RN7QZOIFPANNTQSZIDVHBUVUMSSSH565DTS34EA, seg_pos={0 0}, upload=377105]: Uploading piece... Dec 23 13:01:50.475612861 (429h32m50.346695906s) [noise=true , address=145.79.227.46:4602 , piece_id=QHT4HMOHUETZLBGJKFCH3MQID7OP3J32S42UNPYUJQXQH6ETLUUQ, seg_pos={0 0}, upload=377105]: Uploading piece... Dec 23 13:01:50.477151171 (429h32m50.348234216s) [noise=true , address=145.79.227.45:4230 , piece_id=V5XP4N6OE6Q4QEOB5W7D2J7U3SP67SRPAK2IJMVPXICFQJHWUJAA, seg_pos={0 0}, upload=377105]: Uploading piece... Dec 23 13:01:50.477276479 (429h32m50.348359514s) [noise=true , address=145.79.227.48:4302 , piece_id=OBV2AHJTPXQ7JN7Q4AFLOGJU3PX6C7ROSAOFPHIYIUFZG4LQHUOQ, seg_pos={0 0}, upload=377105]: Uploading piece... Dec 23 13:01:50.478712724 (429h32m50.349795759s) [noise=true , address=145.79.227.13:4804 , piece_id=5WNZNWL557DHGBDAELUNQWVQMHC52T5NISQ7XESK7N7AB7SM4XEQ, seg_pos={0 0}, upload=377105]: Uploading piece... Dec 23 13:01:50.480160231 (429h32m50.351243266s) [noise=true , address=45.91.116.11:4118 , piece_id=NC472UBH2DMEFMPNEWMVRWQYMOV7GUGPLATVFKPGZAIUKIEVW6SQ, seg_pos={0 0}, upload=377105]: Uploading piece... Dec 23 13:01:50.481616634 (429h32m50.352699669s) [noise=true , address=145.79.227.4:4187 , piece_id=33LHWNAE3TLAKMCXYQYMCVWOE4WOHXLUU52RJ4UCVIX5QOVUTRCA, seg_pos={0 0}, upload=377105]: Uploading piece... Dec 23 13:01:50.482751314 (429h32m50.353834359s) [noise=true , address=45.91.116.5:4300 , piece_id=2GII6PLQAGYWPV3NM3WM7BP5CKLPH5FGN76QZVDOL5SMWS3U2V7Q, seg_pos={0 0}, upload=377105]: Uploading piece... Dec 23 13:01:50.483909200 (429h32m50.354992235s) [noise=true , address=145.79.227.29:4942 , piece_id=PZTO6TJE3NI4APPCBG6PUP7HBXKFGZMCBKU5X2Z573WYGZQLIDWQ, seg_pos={0 0}, upload=377105]: Uploading piece... Dec 23 13:01:52.190684160 (429h32m52.061767205s) [upload=377105]: Next returned nil segment Dec 23 13:01:52.190700431 (429h32m52.061783476s) [upload=377105]: Waiting for error group managing segments... Dec 23 13:01:52.390085625 (429h32m52.26116866s) [noise=true , address=45.91.116.14:4190 , piece_id=YOUFVOVGFK6BXESHHXYY2DOEH4E3YIJY3QAEUFTJHKS65QTQJNKQ, seg_pos={0 0}, upload=377105]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: storage node overloaded, request limit: 2000 Dec 23 13:01:52.465012610 (429h32m52.336095655s) [noise=true , address=145.79.227.34:4905 , piece_id=NXTPONXXJUZFPRGBMGSKLDZPMT5UOBNKCGDVAUF6NSUOTWJRHURA, seg_pos={0 0}, upload=377105]: Done uploading piece. err: <nil> Dec 23 13:01:52.473133103 (429h32m52.344216138s) [noise=true , address=145.79.227.45:4230 , piece_id=V5XP4N6OE6Q4QEOB5W7D2J7U3SP67SRPAK2IJMVPXICFQJHWUJAA, seg_pos={0 0}, upload=377105]: Done uploading piece. err: <nil> Dec 23 13:01:52.482395581 (429h32m52.353478616s) [noise=true , address=145.79.227.37:4566 , piece_id=NFDUVZWXADEFCJ3RSQGUG3NX27MY4B3DFCDZ43VR6U6XUKSALDVQ, seg_pos={0 0}, upload=377105]: Done uploading piece. err: <nil> Dec 23 13:01:52.482940177 (429h32m52.354023212s) [noise=true , address=145.79.227.11:4731 , piece_id=QTFACZA3XAEVSNFK45GYZCYWOXWJ4NYWEXZAKEC5PK5B7HWEFA4A, seg_pos={0 0}, upload=377105]: Done uploading piece. err: <nil> Dec 23 13:01:52.482961087 (429h32m52.354044122s) [noise=true , address=145.79.227.49:4339 , piece_id=4DUBWY5NTO4XGJAIDZCHZJLBJWW2Y3KPLIDZ3WPVQFZDXVFDMYUQ, seg_pos={0 0}, upload=377105]: Done uploading piece. err: <nil> Dec 23 13:01:52.483342393 (429h32m52.354425428s) [noise=true , address=145.79.227.46:4602 , piece_id=QHT4HMOHUETZLBGJKFCH3MQID7OP3J32S42UNPYUJQXQH6ETLUUQ, seg_pos={0 0}, upload=377105]: Done uploading piece. err: <nil> Dec 23 13:01:52.489414255 (429h32m52.36049729s) [noise=true , address=145.79.227.4:4187 , piece_id=33LHWNAE3TLAKMCXYQYMCVWOE4WOHXLUU52RJ4UCVIX5QOVUTRCA, seg_pos={0 0}, upload=377105]: Done uploading piece. err: <nil> Dec 23 13:01:52.490255778 (429h32m52.361338813s) [noise=true , address=145.79.227.42:4121 , piece_id=A4NA3V7BEWENLDQHGRTQ7HQ3NPN3GQ5URTNE7NUBVI5J2JL6X5UQ, seg_pos={0 0}, upload=377105]: Done uploading piece. err: <nil> Dec 23 13:01:52.503142076 (429h32m52.374225111s) [noise=true , address=145.79.227.13:4804 , piece_id=5WNZNWL557DHGBDAELUNQWVQMHC52T5NISQ7XESK7N7AB7SM4XEQ, seg_pos={0 0}, upload=377105]: Done uploading piece. err: <nil> Dec 23 13:01:52.505736105 (429h32m52.37681914s) [noise=true , address=145.79.227.29:4942 , piece_id=PZTO6TJE3NI4APPCBG6PUP7HBXKFGZMCBKU5X2Z573WYGZQLIDWQ, seg_pos={0 0}, upload=377105]: Done uploading piece. err: <nil> Dec 23 13:01:52.745151584 (429h32m52.616234619s) [noise=true , address=145.79.227.48:4302 , piece_id=OBV2AHJTPXQ7JN7Q4AFLOGJU3PX6C7ROSAOFPHIYIUFZG4LQHUOQ, seg_pos={0 0}, upload=377105]: Done uploading piece. err: <nil> Dec 23 13:01:53.049982346 (429h32m52.921065401s) [noise=true , address=45.91.116.5:4300 , piece_id=2GII6PLQAGYWPV3NM3WM7BP5CKLPH5FGN76QZVDOL5SMWS3U2V7Q, seg_pos={0 0}, upload=377105]: Done uploading piece. err: <nil> Dec 23 13:01:53.065313249 (429h32m52.936396294s) [noise=true , address=45.91.116.6:4976 , piece_id=LJDYCGUIKJMIZOSYTFLXK2OXRTOA34P7FAPSZFLTAXJ6OBKSTYXA, seg_pos={0 0}, upload=377105]: Done uploading piece. err: <nil> Dec 23 13:01:53.065341903 (429h32m52.936424938s) [seg_pos={0 0}, upload=377105]: Segment reached optimal threshold of 13 pieces. Dec 23 13:01:53.065589845 (429h32m52.93667288s) [noise=true , address=45.91.116.11:4118 , piece_id=NC472UBH2DMEFMPNEWMVRWQYMOV7GUGPLATVFKPGZAIUKIEVW6SQ, seg_pos={0 0}, upload=377105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF Dec 23 13:01:53.065598843 (429h32m52.936681878s) [noise=true , address=45.91.116.11:4118 , piece_id=NC472UBH2DMEFMPNEWMVRWQYMOV7GUGPLATVFKPGZAIUKIEVW6SQ, seg_pos={0 0}, upload=377105]: Long tail cancellation due to optimal threshold Dec 23 13:01:53.065869357 (429h32m52.936952392s) [noise=true , address=45.91.116.13:4155 , piece_id=AJ5OPEXWZSA24RN7QZOIFPANNTQSZIDVHBUVUMSSSH565DTS34EA, seg_pos={0 0}, upload=377105]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: write tcp 188.166.225.170:40332->45.91.116.13:4155: use of closed network connection; write tcp 188.166.225.170:40332->45.91.116.13:4155: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40332->45.91.116.13:4155: use of closed network connection Dec 23 13:01:53.065880539 (429h32m52.936963574s) [noise=true , address=45.91.116.13:4155 , piece_id=AJ5OPEXWZSA24RN7QZOIFPANNTQSZIDVHBUVUMSSSH565DTS34EA, seg_pos={0 0}, upload=377105]: Long tail cancellation due to optimal threshold Dec 23 13:01:53.065925935 (429h32m52.93700899s) [seg_pos={0 0}, upload=377105]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 13:01:53.066048309 (429h32m52.937131364s) [upload=377105]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 13:01:53.066056264 (429h32m52.937139309s) [upload=377105]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 13:01:53.356946671 (429h32m53.228029736s) [upload=377105]: Done uploading segments... Dec 23 13:01:53.356975245 (429h32m53.22805829s) [upload=377105]: Upload finished. err: <nil> Duration: 30.002867904s core uplink us-select-1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [.............................................X..............] duration: 260.259049ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/73921c1a3ca0437b FAIL (2025-12-23 13:01:46 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 260.276843ms edge linksharing bgp eu1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [..............................................X.............] duration: 257.254108ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/73921c1a3ca0437b FAIL (2025-12-23 13:00:41 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 257.279196ms edge linksharing bgp eu1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [...............................................X............] duration: 581.617042ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/73921c1a3ca0437b FAIL (2025-12-23 13:00:02 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 581.890483ms edge linksharing bgp eu1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [................................................X...........] duration: 268.979273ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/73921c1a3ca0437b FAIL (2025-12-23 12:59:08 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 269.00403ms edge linksharing bgp eu1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [.................................................X..........] duration: 268.170544ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/73921c1a3ca0437b FAIL (2025-12-23 12:58:51 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 268.201353ms edge linksharing bgp eu1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [..................................................X.........] duration: 252.897672ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/73921c1a3ca0437b FAIL (2025-12-23 12:57:52 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 252.926726ms edge linksharing bgp eu1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [...................................................X........] duration: 254.64569ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/73921c1a3ca0437b FAIL (2025-12-23 12:57:27 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 254.671619ms edge linksharing bgp eu1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [....................................................X.......] duration: 250.535323ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/73921c1a3ca0437b FAIL (2025-12-23 12:57:12 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 250.557866ms edge linksharing bgp eu1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [.....................................................X......] duration: 247.990326ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/73921c1a3ca0437b FAIL (2025-12-23 12:56:33 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 248.006548ms edge linksharing bgp eu1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [......................................................X.....] duration: 251.081884ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/73921c1a3ca0437b FAIL (2025-12-23 12:55:45 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 251.119666ms edge linksharing bgp eu1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [.......................................................X....] duration: 250.429463ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/73921c1a3ca0437b FAIL (2025-12-23 12:55:12 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 250.47543ms edge linksharing bgp eu1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [........................................................X...] duration: 270.90994ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/73921c1a3ca0437b FAIL (2025-12-23 12:54:18 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 270.943555ms edge linksharing bgp eu1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [.........................................................X..] duration: 244.463262ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/73921c1a3ca0437b FAIL (2025-12-23 12:53:06 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 244.492397ms edge linksharing bgp eu1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [..........................................................X.] duration: 376.883111ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/73921c1a3ca0437b FAIL (2025-12-23 12:52:40 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 376.896857ms edge linksharing bgp eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 5.890021789s trace: https://jaeger.dev.storj.io/trace/587ca7f719cacadf FAIL (2025-12-23 10:43:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== ploading segments... Dec 23 09:56:07.712403417 (426h27m7.583486462s) [upload=374371]: Got next segment. Inline: false Dec 23 09:56:07.712445147 (426h27m7.583528182s) [upload=374371]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 09:56:07.712452090 (426h27m7.583535125s) [upload=374371]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 09:56:08.088205172 (426h27m7.959288207s) [seg_pos={0 0}, upload=374371]: Begin upload segment... Dec 23 09:56:08.088258283 (426h27m7.959341328s) [seg_pos={0 0}, upload=374371]: Done begin upload segment. Dec 23 09:56:08.088290625 (426h27m7.95937367s) [noise=true , address=145.79.227.36:4518 , piece_id=RG4EIL5AGFVP2Z3J3Z7W6NJSA4LPDTFEXSLHZW722X3RSNG7H3LQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.088349738 (426h27m7.959432783s) [noise=true , address=145.79.227.38:4978 , piece_id=UAHAMPWHYQUY7YYS7VJPGTXBCPD6KWZADF2TVZIVU7ECDBF36NWQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.089542198 (426h27m7.960625233s) [noise=true , address=45.91.116.14:4189 , piece_id=NOUKCZQ5LQT7H62V7OBWQXNFVCOETFGAFHM6LIQSZYYX5JOCX4RQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.089679400 (426h27m7.960762445s) [noise=true , address=145.79.227.46:4602 , piece_id=UUIEHKP4PL7OS7IICNOB27C5SS5ZOIJPHC35K3R4IVTEXXV5SPZA, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.090450538 (426h27m7.961533583s) [noise=true , address=77.90.31.19:4842 , piece_id=FWDKIRAA4726YXGKXSB7PHUN7IZMNPV5ZSWKZAKZKJMQ45HDPLTQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.091327698 (426h27m7.962410743s) [noise=true , address=145.79.227.47:4267 , piece_id=QNH4GZCBVRW6FIBGILDQOHOZSBAYN2KR42RP4G77EL2RODLXRWZQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.092179080 (426h27m7.963262115s) [noise=true , address=77.90.31.17:4773 , piece_id=MU4UUWSSJNXVEOHAV2BWR2H4EWYFOEA2YNIQVQU6XQZIWWFDJXRQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.092774453 (426h27m7.963857498s) [noise=true , address=145.79.227.41:4085 , piece_id=BKPJP6TQTF7G6RMNI3K3S23WZWXSGOJCDBIWSHMXULK4OQQYQCZA, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.093537637 (426h27m7.964620682s) [noise=true , address=145.79.227.48:4301 , piece_id=5IV2WAC77MWBYSQ4YRNZH2DH5BGG4XU3SVAZE5DLQJS7A2J4O3BA, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.094292854 (426h27m7.965375919s) [noise=true , address=145.79.227.28:4725 , piece_id=EHBGDM7PYS2BZ2A7HKASKRHDNE3FGBX6W3GEEX7QHKPW5ZDZCNWQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.094608194 (426h27m7.965691239s) [noise=true , address=145.79.227.7:4293 , piece_id=YTBS6YPNK5BYOZLVYF5BX43SDZNNTTZJ473HE2XQPPZ4T3K6WMXQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.095183480 (426h27m7.966266525s) [noise=true , address=45.91.116.11:4118 , piece_id=VYLGLLGMTK2JZWHGLLTMUEYFVUH3AYU7P4UEYU4EMU5N5FVZHOMQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.095350008 (426h27m7.966433053s) [noise=true , address=145.79.227.35:4481 , piece_id=4TK6V52M3EHIBHFWCZJ5SLH6LZGKSRHPFTVL5LARU5JUVO7MGCKA, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.096434292 (426h27m7.967517337s) [noise=true , address=145.79.227.14:4839 , piece_id=YNIFPV6FLHOIEEPWL34RW7CLDTXTYTG7RKGAIMFZYT6OLIB5J7TQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.096712121 (426h27m7.967795166s) [noise=true , address=45.91.116.8:4046 , piece_id=BMFCPFPRSK2LFMCRWPCLJPVETOD5QL3VTHS6POMRVETSJOB72OOA, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.097512405 (426h27m7.96859546s) [noise=true , address=145.79.227.31:4798 , piece_id=7SNRYHTEAWMXXH6KQ4UPURGY7PYJ72HOOCVA7YCIPLFG75NPNJYQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:09.969534065 (426h27m9.8406171s) [upload=374371]: Next returned nil segment Dec 23 09:56:09.969568862 (426h27m9.840651897s) [upload=374371]: Waiting for error group managing segments... Dec 23 09:56:10.267890059 (426h27m10.138973104s) [noise=true , address=77.90.31.17:4773 , piece_id=MU4UUWSSJNXVEOHAV2BWR2H4EWYFOEA2YNIQVQU6XQZIWWFDJXRQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:10.424500182 (426h27m10.295583227s) [noise=true , address=77.90.31.19:4842 , piece_id=FWDKIRAA4726YXGKXSB7PHUN7IZMNPV5ZSWKZAKZKJMQ45HDPLTQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:10.842613724 (426h27m10.713696769s) [noise=true , address=145.79.227.41:4085 , piece_id=BKPJP6TQTF7G6RMNI3K3S23WZWXSGOJCDBIWSHMXULK4OQQYQCZA, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:11.071019640 (426h27m10.942102685s) [noise=true , address=45.91.116.14:4189 , piece_id=NOUKCZQ5LQT7H62V7OBWQXNFVCOETFGAFHM6LIQSZYYX5JOCX4RQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:11.141880171 (426h27m11.012963216s) [noise=true , address=145.79.227.48:4301 , piece_id=5IV2WAC77MWBYSQ4YRNZH2DH5BGG4XU3SVAZE5DLQJS7A2J4O3BA, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:11.149149884 (426h27m11.020232919s) [noise=true , address=145.79.227.38:4978 , piece_id=UAHAMPWHYQUY7YYS7VJPGTXBCPD6KWZADF2TVZIVU7ECDBF36NWQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:11.160413952 (426h27m11.031496997s) [noise=true , address=145.79.227.46:4602 , piece_id=UUIEHKP4PL7OS7IICNOB27C5SS5ZOIJPHC35K3R4IVTEXXV5SPZA, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:12.884195178 (426h27m12.755278223s) [noise=true , address=45.91.116.8:4046 , piece_id=BMFCPFPRSK2LFMCRWPCLJPVETOD5QL3VTHS6POMRVETSJOB72OOA, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:31.365089909 (426h27m31.236172944s) [noise=true , address=145.79.227.36:4518 , piece_id=RG4EIL5AGFVP2Z3J3Z7W6NJSA4LPDTFEXSLHZW722X3RSNG7H3LQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:31.715116052 (426h27m31.586199087s) [noise=true , address=145.79.227.31:4798 , piece_id=7SNRYHTEAWMXXH6KQ4UPURGY7PYJ72HOOCVA7YCIPLFG75NPNJYQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:32.549036955 (426h27m32.420119991s) [noise=true , address=145.79.227.35:4481 , piece_id=4TK6V52M3EHIBHFWCZJ5SLH6LZGKSRHPFTVL5LARU5JUVO7MGCKA, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:36.707023852 (426h27m36.578106887s) [noise=true , address=145.79.227.28:4725 , piece_id=EHBGDM7PYS2BZ2A7HKASKRHDNE3FGBX6W3GEEX7QHKPW5ZDZCNWQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:45348->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:45348->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45348->145.79.227.28:4725: use of closed network connection Dec 23 09:56:36.707046335 (426h27m36.57812937s) [noise=true , address=145.79.227.14:4839 , piece_id=YNIFPV6FLHOIEEPWL34RW7CLDTXTYTG7RKGAIMFZYT6OLIB5J7TQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:37586->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:37586->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37586->145.79.227.14:4839: use of closed network connection Dec 23 09:56:36.707164720 (426h27m36.578247765s) [noise=true , address=145.79.227.47:4267 , piece_id=QNH4GZCBVRW6FIBGILDQOHOZSBAYN2KR42RP4G77EL2RODLXRWZQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: ecclient: upload failed (node:1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8, address:145.79.227.47:4267): protocol: write tcp 188.166.225.170:42832->145.79.227.47:4267: use of closed network connection; write tcp 188.166.225.170:42832->145.79.227.47:4267: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42832->145.79.227.47:4267: use of closed network connection Dec 23 09:56:36.707206169 (426h27m36.578289214s) [noise=true , address=145.79.227.7:4293 , piece_id=YTBS6YPNK5BYOZLVYF5BX43SDZNNTTZJ473HE2XQPPZ4T3K6WMXQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: write tcp 188.166.225.170:46776->145.79.227.7:4293: use of closed network connection; write tcp 188.166.225.170:46776->145.79.227.7:4293: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46776->145.79.227.7:4293: use of closed network connection Dec 23 09:56:36.707318803 (426h27m36.578401838s) [noise=true , address=45.91.116.11:4118 , piece_id=VYLGLLGMTK2JZWHGLLTMUEYFVUH3AYU7P4UEYU4EMU5N5FVZHOMQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: ecclient: upload failed (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, address:45.91.116.11:4118): protocol: write tcp 188.166.225.170:34940->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:34940->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34940->45.91.116.11:4118: use of closed network connection Dec 23 09:56:36.707356044 (426h27m36.578439079s) [seg_pos={0 0}, upload=374371]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Dec 23 09:56:36.958957688 (426h27m36.830040733s) [upload=374371]: Done uploading segments... Dec 23 09:56:36.958995501 (426h27m36.830078546s) [upload=374371]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.890050744s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 5.964619727s trace: https://jaeger.dev.storj.io/trace/587ca7f719cacadf FAIL (2025-12-23 10:42:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== ploading segments... Dec 23 09:56:07.712403417 (426h27m7.583486462s) [upload=374371]: Got next segment. Inline: false Dec 23 09:56:07.712445147 (426h27m7.583528182s) [upload=374371]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 09:56:07.712452090 (426h27m7.583535125s) [upload=374371]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 09:56:08.088205172 (426h27m7.959288207s) [seg_pos={0 0}, upload=374371]: Begin upload segment... Dec 23 09:56:08.088258283 (426h27m7.959341328s) [seg_pos={0 0}, upload=374371]: Done begin upload segment. Dec 23 09:56:08.088290625 (426h27m7.95937367s) [noise=true , address=145.79.227.36:4518 , piece_id=RG4EIL5AGFVP2Z3J3Z7W6NJSA4LPDTFEXSLHZW722X3RSNG7H3LQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.088349738 (426h27m7.959432783s) [noise=true , address=145.79.227.38:4978 , piece_id=UAHAMPWHYQUY7YYS7VJPGTXBCPD6KWZADF2TVZIVU7ECDBF36NWQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.089542198 (426h27m7.960625233s) [noise=true , address=45.91.116.14:4189 , piece_id=NOUKCZQ5LQT7H62V7OBWQXNFVCOETFGAFHM6LIQSZYYX5JOCX4RQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.089679400 (426h27m7.960762445s) [noise=true , address=145.79.227.46:4602 , piece_id=UUIEHKP4PL7OS7IICNOB27C5SS5ZOIJPHC35K3R4IVTEXXV5SPZA, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.090450538 (426h27m7.961533583s) [noise=true , address=77.90.31.19:4842 , piece_id=FWDKIRAA4726YXGKXSB7PHUN7IZMNPV5ZSWKZAKZKJMQ45HDPLTQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.091327698 (426h27m7.962410743s) [noise=true , address=145.79.227.47:4267 , piece_id=QNH4GZCBVRW6FIBGILDQOHOZSBAYN2KR42RP4G77EL2RODLXRWZQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.092179080 (426h27m7.963262115s) [noise=true , address=77.90.31.17:4773 , piece_id=MU4UUWSSJNXVEOHAV2BWR2H4EWYFOEA2YNIQVQU6XQZIWWFDJXRQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.092774453 (426h27m7.963857498s) [noise=true , address=145.79.227.41:4085 , piece_id=BKPJP6TQTF7G6RMNI3K3S23WZWXSGOJCDBIWSHMXULK4OQQYQCZA, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.093537637 (426h27m7.964620682s) [noise=true , address=145.79.227.48:4301 , piece_id=5IV2WAC77MWBYSQ4YRNZH2DH5BGG4XU3SVAZE5DLQJS7A2J4O3BA, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.094292854 (426h27m7.965375919s) [noise=true , address=145.79.227.28:4725 , piece_id=EHBGDM7PYS2BZ2A7HKASKRHDNE3FGBX6W3GEEX7QHKPW5ZDZCNWQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.094608194 (426h27m7.965691239s) [noise=true , address=145.79.227.7:4293 , piece_id=YTBS6YPNK5BYOZLVYF5BX43SDZNNTTZJ473HE2XQPPZ4T3K6WMXQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.095183480 (426h27m7.966266525s) [noise=true , address=45.91.116.11:4118 , piece_id=VYLGLLGMTK2JZWHGLLTMUEYFVUH3AYU7P4UEYU4EMU5N5FVZHOMQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.095350008 (426h27m7.966433053s) [noise=true , address=145.79.227.35:4481 , piece_id=4TK6V52M3EHIBHFWCZJ5SLH6LZGKSRHPFTVL5LARU5JUVO7MGCKA, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.096434292 (426h27m7.967517337s) [noise=true , address=145.79.227.14:4839 , piece_id=YNIFPV6FLHOIEEPWL34RW7CLDTXTYTG7RKGAIMFZYT6OLIB5J7TQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.096712121 (426h27m7.967795166s) [noise=true , address=45.91.116.8:4046 , piece_id=BMFCPFPRSK2LFMCRWPCLJPVETOD5QL3VTHS6POMRVETSJOB72OOA, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.097512405 (426h27m7.96859546s) [noise=true , address=145.79.227.31:4798 , piece_id=7SNRYHTEAWMXXH6KQ4UPURGY7PYJ72HOOCVA7YCIPLFG75NPNJYQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:09.969534065 (426h27m9.8406171s) [upload=374371]: Next returned nil segment Dec 23 09:56:09.969568862 (426h27m9.840651897s) [upload=374371]: Waiting for error group managing segments... Dec 23 09:56:10.267890059 (426h27m10.138973104s) [noise=true , address=77.90.31.17:4773 , piece_id=MU4UUWSSJNXVEOHAV2BWR2H4EWYFOEA2YNIQVQU6XQZIWWFDJXRQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:10.424500182 (426h27m10.295583227s) [noise=true , address=77.90.31.19:4842 , piece_id=FWDKIRAA4726YXGKXSB7PHUN7IZMNPV5ZSWKZAKZKJMQ45HDPLTQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:10.842613724 (426h27m10.713696769s) [noise=true , address=145.79.227.41:4085 , piece_id=BKPJP6TQTF7G6RMNI3K3S23WZWXSGOJCDBIWSHMXULK4OQQYQCZA, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:11.071019640 (426h27m10.942102685s) [noise=true , address=45.91.116.14:4189 , piece_id=NOUKCZQ5LQT7H62V7OBWQXNFVCOETFGAFHM6LIQSZYYX5JOCX4RQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:11.141880171 (426h27m11.012963216s) [noise=true , address=145.79.227.48:4301 , piece_id=5IV2WAC77MWBYSQ4YRNZH2DH5BGG4XU3SVAZE5DLQJS7A2J4O3BA, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:11.149149884 (426h27m11.020232919s) [noise=true , address=145.79.227.38:4978 , piece_id=UAHAMPWHYQUY7YYS7VJPGTXBCPD6KWZADF2TVZIVU7ECDBF36NWQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:11.160413952 (426h27m11.031496997s) [noise=true , address=145.79.227.46:4602 , piece_id=UUIEHKP4PL7OS7IICNOB27C5SS5ZOIJPHC35K3R4IVTEXXV5SPZA, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:12.884195178 (426h27m12.755278223s) [noise=true , address=45.91.116.8:4046 , piece_id=BMFCPFPRSK2LFMCRWPCLJPVETOD5QL3VTHS6POMRVETSJOB72OOA, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:31.365089909 (426h27m31.236172944s) [noise=true , address=145.79.227.36:4518 , piece_id=RG4EIL5AGFVP2Z3J3Z7W6NJSA4LPDTFEXSLHZW722X3RSNG7H3LQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:31.715116052 (426h27m31.586199087s) [noise=true , address=145.79.227.31:4798 , piece_id=7SNRYHTEAWMXXH6KQ4UPURGY7PYJ72HOOCVA7YCIPLFG75NPNJYQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:32.549036955 (426h27m32.420119991s) [noise=true , address=145.79.227.35:4481 , piece_id=4TK6V52M3EHIBHFWCZJ5SLH6LZGKSRHPFTVL5LARU5JUVO7MGCKA, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:36.707023852 (426h27m36.578106887s) [noise=true , address=145.79.227.28:4725 , piece_id=EHBGDM7PYS2BZ2A7HKASKRHDNE3FGBX6W3GEEX7QHKPW5ZDZCNWQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:45348->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:45348->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45348->145.79.227.28:4725: use of closed network connection Dec 23 09:56:36.707046335 (426h27m36.57812937s) [noise=true , address=145.79.227.14:4839 , piece_id=YNIFPV6FLHOIEEPWL34RW7CLDTXTYTG7RKGAIMFZYT6OLIB5J7TQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:37586->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:37586->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37586->145.79.227.14:4839: use of closed network connection Dec 23 09:56:36.707164720 (426h27m36.578247765s) [noise=true , address=145.79.227.47:4267 , piece_id=QNH4GZCBVRW6FIBGILDQOHOZSBAYN2KR42RP4G77EL2RODLXRWZQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: ecclient: upload failed (node:1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8, address:145.79.227.47:4267): protocol: write tcp 188.166.225.170:42832->145.79.227.47:4267: use of closed network connection; write tcp 188.166.225.170:42832->145.79.227.47:4267: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42832->145.79.227.47:4267: use of closed network connection Dec 23 09:56:36.707206169 (426h27m36.578289214s) [noise=true , address=145.79.227.7:4293 , piece_id=YTBS6YPNK5BYOZLVYF5BX43SDZNNTTZJ473HE2XQPPZ4T3K6WMXQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: write tcp 188.166.225.170:46776->145.79.227.7:4293: use of closed network connection; write tcp 188.166.225.170:46776->145.79.227.7:4293: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46776->145.79.227.7:4293: use of closed network connection Dec 23 09:56:36.707318803 (426h27m36.578401838s) [noise=true , address=45.91.116.11:4118 , piece_id=VYLGLLGMTK2JZWHGLLTMUEYFVUH3AYU7P4UEYU4EMU5N5FVZHOMQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: ecclient: upload failed (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, address:45.91.116.11:4118): protocol: write tcp 188.166.225.170:34940->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:34940->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34940->45.91.116.11:4118: use of closed network connection Dec 23 09:56:36.707356044 (426h27m36.578439079s) [seg_pos={0 0}, upload=374371]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Dec 23 09:56:36.958957688 (426h27m36.830040733s) [upload=374371]: Done uploading segments... Dec 23 09:56:36.958995501 (426h27m36.830078546s) [upload=374371]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.964651437s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 5.627972348s trace: https://jaeger.dev.storj.io/trace/587ca7f719cacadf FAIL (2025-12-23 10:41:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== ploading segments... Dec 23 09:56:07.712403417 (426h27m7.583486462s) [upload=374371]: Got next segment. Inline: false Dec 23 09:56:07.712445147 (426h27m7.583528182s) [upload=374371]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 09:56:07.712452090 (426h27m7.583535125s) [upload=374371]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 09:56:08.088205172 (426h27m7.959288207s) [seg_pos={0 0}, upload=374371]: Begin upload segment... Dec 23 09:56:08.088258283 (426h27m7.959341328s) [seg_pos={0 0}, upload=374371]: Done begin upload segment. Dec 23 09:56:08.088290625 (426h27m7.95937367s) [noise=true , address=145.79.227.36:4518 , piece_id=RG4EIL5AGFVP2Z3J3Z7W6NJSA4LPDTFEXSLHZW722X3RSNG7H3LQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.088349738 (426h27m7.959432783s) [noise=true , address=145.79.227.38:4978 , piece_id=UAHAMPWHYQUY7YYS7VJPGTXBCPD6KWZADF2TVZIVU7ECDBF36NWQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.089542198 (426h27m7.960625233s) [noise=true , address=45.91.116.14:4189 , piece_id=NOUKCZQ5LQT7H62V7OBWQXNFVCOETFGAFHM6LIQSZYYX5JOCX4RQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.089679400 (426h27m7.960762445s) [noise=true , address=145.79.227.46:4602 , piece_id=UUIEHKP4PL7OS7IICNOB27C5SS5ZOIJPHC35K3R4IVTEXXV5SPZA, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.090450538 (426h27m7.961533583s) [noise=true , address=77.90.31.19:4842 , piece_id=FWDKIRAA4726YXGKXSB7PHUN7IZMNPV5ZSWKZAKZKJMQ45HDPLTQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.091327698 (426h27m7.962410743s) [noise=true , address=145.79.227.47:4267 , piece_id=QNH4GZCBVRW6FIBGILDQOHOZSBAYN2KR42RP4G77EL2RODLXRWZQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.092179080 (426h27m7.963262115s) [noise=true , address=77.90.31.17:4773 , piece_id=MU4UUWSSJNXVEOHAV2BWR2H4EWYFOEA2YNIQVQU6XQZIWWFDJXRQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.092774453 (426h27m7.963857498s) [noise=true , address=145.79.227.41:4085 , piece_id=BKPJP6TQTF7G6RMNI3K3S23WZWXSGOJCDBIWSHMXULK4OQQYQCZA, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.093537637 (426h27m7.964620682s) [noise=true , address=145.79.227.48:4301 , piece_id=5IV2WAC77MWBYSQ4YRNZH2DH5BGG4XU3SVAZE5DLQJS7A2J4O3BA, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.094292854 (426h27m7.965375919s) [noise=true , address=145.79.227.28:4725 , piece_id=EHBGDM7PYS2BZ2A7HKASKRHDNE3FGBX6W3GEEX7QHKPW5ZDZCNWQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.094608194 (426h27m7.965691239s) [noise=true , address=145.79.227.7:4293 , piece_id=YTBS6YPNK5BYOZLVYF5BX43SDZNNTTZJ473HE2XQPPZ4T3K6WMXQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.095183480 (426h27m7.966266525s) [noise=true , address=45.91.116.11:4118 , piece_id=VYLGLLGMTK2JZWHGLLTMUEYFVUH3AYU7P4UEYU4EMU5N5FVZHOMQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.095350008 (426h27m7.966433053s) [noise=true , address=145.79.227.35:4481 , piece_id=4TK6V52M3EHIBHFWCZJ5SLH6LZGKSRHPFTVL5LARU5JUVO7MGCKA, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.096434292 (426h27m7.967517337s) [noise=true , address=145.79.227.14:4839 , piece_id=YNIFPV6FLHOIEEPWL34RW7CLDTXTYTG7RKGAIMFZYT6OLIB5J7TQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.096712121 (426h27m7.967795166s) [noise=true , address=45.91.116.8:4046 , piece_id=BMFCPFPRSK2LFMCRWPCLJPVETOD5QL3VTHS6POMRVETSJOB72OOA, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.097512405 (426h27m7.96859546s) [noise=true , address=145.79.227.31:4798 , piece_id=7SNRYHTEAWMXXH6KQ4UPURGY7PYJ72HOOCVA7YCIPLFG75NPNJYQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:09.969534065 (426h27m9.8406171s) [upload=374371]: Next returned nil segment Dec 23 09:56:09.969568862 (426h27m9.840651897s) [upload=374371]: Waiting for error group managing segments... Dec 23 09:56:10.267890059 (426h27m10.138973104s) [noise=true , address=77.90.31.17:4773 , piece_id=MU4UUWSSJNXVEOHAV2BWR2H4EWYFOEA2YNIQVQU6XQZIWWFDJXRQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:10.424500182 (426h27m10.295583227s) [noise=true , address=77.90.31.19:4842 , piece_id=FWDKIRAA4726YXGKXSB7PHUN7IZMNPV5ZSWKZAKZKJMQ45HDPLTQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:10.842613724 (426h27m10.713696769s) [noise=true , address=145.79.227.41:4085 , piece_id=BKPJP6TQTF7G6RMNI3K3S23WZWXSGOJCDBIWSHMXULK4OQQYQCZA, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:11.071019640 (426h27m10.942102685s) [noise=true , address=45.91.116.14:4189 , piece_id=NOUKCZQ5LQT7H62V7OBWQXNFVCOETFGAFHM6LIQSZYYX5JOCX4RQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:11.141880171 (426h27m11.012963216s) [noise=true , address=145.79.227.48:4301 , piece_id=5IV2WAC77MWBYSQ4YRNZH2DH5BGG4XU3SVAZE5DLQJS7A2J4O3BA, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:11.149149884 (426h27m11.020232919s) [noise=true , address=145.79.227.38:4978 , piece_id=UAHAMPWHYQUY7YYS7VJPGTXBCPD6KWZADF2TVZIVU7ECDBF36NWQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:11.160413952 (426h27m11.031496997s) [noise=true , address=145.79.227.46:4602 , piece_id=UUIEHKP4PL7OS7IICNOB27C5SS5ZOIJPHC35K3R4IVTEXXV5SPZA, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:12.884195178 (426h27m12.755278223s) [noise=true , address=45.91.116.8:4046 , piece_id=BMFCPFPRSK2LFMCRWPCLJPVETOD5QL3VTHS6POMRVETSJOB72OOA, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:31.365089909 (426h27m31.236172944s) [noise=true , address=145.79.227.36:4518 , piece_id=RG4EIL5AGFVP2Z3J3Z7W6NJSA4LPDTFEXSLHZW722X3RSNG7H3LQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:31.715116052 (426h27m31.586199087s) [noise=true , address=145.79.227.31:4798 , piece_id=7SNRYHTEAWMXXH6KQ4UPURGY7PYJ72HOOCVA7YCIPLFG75NPNJYQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:32.549036955 (426h27m32.420119991s) [noise=true , address=145.79.227.35:4481 , piece_id=4TK6V52M3EHIBHFWCZJ5SLH6LZGKSRHPFTVL5LARU5JUVO7MGCKA, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:36.707023852 (426h27m36.578106887s) [noise=true , address=145.79.227.28:4725 , piece_id=EHBGDM7PYS2BZ2A7HKASKRHDNE3FGBX6W3GEEX7QHKPW5ZDZCNWQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:45348->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:45348->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45348->145.79.227.28:4725: use of closed network connection Dec 23 09:56:36.707046335 (426h27m36.57812937s) [noise=true , address=145.79.227.14:4839 , piece_id=YNIFPV6FLHOIEEPWL34RW7CLDTXTYTG7RKGAIMFZYT6OLIB5J7TQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:37586->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:37586->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37586->145.79.227.14:4839: use of closed network connection Dec 23 09:56:36.707164720 (426h27m36.578247765s) [noise=true , address=145.79.227.47:4267 , piece_id=QNH4GZCBVRW6FIBGILDQOHOZSBAYN2KR42RP4G77EL2RODLXRWZQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: ecclient: upload failed (node:1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8, address:145.79.227.47:4267): protocol: write tcp 188.166.225.170:42832->145.79.227.47:4267: use of closed network connection; write tcp 188.166.225.170:42832->145.79.227.47:4267: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42832->145.79.227.47:4267: use of closed network connection Dec 23 09:56:36.707206169 (426h27m36.578289214s) [noise=true , address=145.79.227.7:4293 , piece_id=YTBS6YPNK5BYOZLVYF5BX43SDZNNTTZJ473HE2XQPPZ4T3K6WMXQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: write tcp 188.166.225.170:46776->145.79.227.7:4293: use of closed network connection; write tcp 188.166.225.170:46776->145.79.227.7:4293: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46776->145.79.227.7:4293: use of closed network connection Dec 23 09:56:36.707318803 (426h27m36.578401838s) [noise=true , address=45.91.116.11:4118 , piece_id=VYLGLLGMTK2JZWHGLLTMUEYFVUH3AYU7P4UEYU4EMU5N5FVZHOMQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: ecclient: upload failed (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, address:45.91.116.11:4118): protocol: write tcp 188.166.225.170:34940->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:34940->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34940->45.91.116.11:4118: use of closed network connection Dec 23 09:56:36.707356044 (426h27m36.578439079s) [seg_pos={0 0}, upload=374371]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Dec 23 09:56:36.958957688 (426h27m36.830040733s) [upload=374371]: Done uploading segments... Dec 23 09:56:36.958995501 (426h27m36.830078546s) [upload=374371]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.627988649s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 5.852922564s trace: https://jaeger.dev.storj.io/trace/587ca7f719cacadf FAIL (2025-12-23 10:40:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== ploading segments... Dec 23 09:56:07.712403417 (426h27m7.583486462s) [upload=374371]: Got next segment. Inline: false Dec 23 09:56:07.712445147 (426h27m7.583528182s) [upload=374371]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 09:56:07.712452090 (426h27m7.583535125s) [upload=374371]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 09:56:08.088205172 (426h27m7.959288207s) [seg_pos={0 0}, upload=374371]: Begin upload segment... Dec 23 09:56:08.088258283 (426h27m7.959341328s) [seg_pos={0 0}, upload=374371]: Done begin upload segment. Dec 23 09:56:08.088290625 (426h27m7.95937367s) [noise=true , address=145.79.227.36:4518 , piece_id=RG4EIL5AGFVP2Z3J3Z7W6NJSA4LPDTFEXSLHZW722X3RSNG7H3LQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.088349738 (426h27m7.959432783s) [noise=true , address=145.79.227.38:4978 , piece_id=UAHAMPWHYQUY7YYS7VJPGTXBCPD6KWZADF2TVZIVU7ECDBF36NWQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.089542198 (426h27m7.960625233s) [noise=true , address=45.91.116.14:4189 , piece_id=NOUKCZQ5LQT7H62V7OBWQXNFVCOETFGAFHM6LIQSZYYX5JOCX4RQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.089679400 (426h27m7.960762445s) [noise=true , address=145.79.227.46:4602 , piece_id=UUIEHKP4PL7OS7IICNOB27C5SS5ZOIJPHC35K3R4IVTEXXV5SPZA, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.090450538 (426h27m7.961533583s) [noise=true , address=77.90.31.19:4842 , piece_id=FWDKIRAA4726YXGKXSB7PHUN7IZMNPV5ZSWKZAKZKJMQ45HDPLTQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.091327698 (426h27m7.962410743s) [noise=true , address=145.79.227.47:4267 , piece_id=QNH4GZCBVRW6FIBGILDQOHOZSBAYN2KR42RP4G77EL2RODLXRWZQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.092179080 (426h27m7.963262115s) [noise=true , address=77.90.31.17:4773 , piece_id=MU4UUWSSJNXVEOHAV2BWR2H4EWYFOEA2YNIQVQU6XQZIWWFDJXRQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.092774453 (426h27m7.963857498s) [noise=true , address=145.79.227.41:4085 , piece_id=BKPJP6TQTF7G6RMNI3K3S23WZWXSGOJCDBIWSHMXULK4OQQYQCZA, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.093537637 (426h27m7.964620682s) [noise=true , address=145.79.227.48:4301 , piece_id=5IV2WAC77MWBYSQ4YRNZH2DH5BGG4XU3SVAZE5DLQJS7A2J4O3BA, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.094292854 (426h27m7.965375919s) [noise=true , address=145.79.227.28:4725 , piece_id=EHBGDM7PYS2BZ2A7HKASKRHDNE3FGBX6W3GEEX7QHKPW5ZDZCNWQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.094608194 (426h27m7.965691239s) [noise=true , address=145.79.227.7:4293 , piece_id=YTBS6YPNK5BYOZLVYF5BX43SDZNNTTZJ473HE2XQPPZ4T3K6WMXQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.095183480 (426h27m7.966266525s) [noise=true , address=45.91.116.11:4118 , piece_id=VYLGLLGMTK2JZWHGLLTMUEYFVUH3AYU7P4UEYU4EMU5N5FVZHOMQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.095350008 (426h27m7.966433053s) [noise=true , address=145.79.227.35:4481 , piece_id=4TK6V52M3EHIBHFWCZJ5SLH6LZGKSRHPFTVL5LARU5JUVO7MGCKA, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.096434292 (426h27m7.967517337s) [noise=true , address=145.79.227.14:4839 , piece_id=YNIFPV6FLHOIEEPWL34RW7CLDTXTYTG7RKGAIMFZYT6OLIB5J7TQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.096712121 (426h27m7.967795166s) [noise=true , address=45.91.116.8:4046 , piece_id=BMFCPFPRSK2LFMCRWPCLJPVETOD5QL3VTHS6POMRVETSJOB72OOA, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.097512405 (426h27m7.96859546s) [noise=true , address=145.79.227.31:4798 , piece_id=7SNRYHTEAWMXXH6KQ4UPURGY7PYJ72HOOCVA7YCIPLFG75NPNJYQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:09.969534065 (426h27m9.8406171s) [upload=374371]: Next returned nil segment Dec 23 09:56:09.969568862 (426h27m9.840651897s) [upload=374371]: Waiting for error group managing segments... Dec 23 09:56:10.267890059 (426h27m10.138973104s) [noise=true , address=77.90.31.17:4773 , piece_id=MU4UUWSSJNXVEOHAV2BWR2H4EWYFOEA2YNIQVQU6XQZIWWFDJXRQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:10.424500182 (426h27m10.295583227s) [noise=true , address=77.90.31.19:4842 , piece_id=FWDKIRAA4726YXGKXSB7PHUN7IZMNPV5ZSWKZAKZKJMQ45HDPLTQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:10.842613724 (426h27m10.713696769s) [noise=true , address=145.79.227.41:4085 , piece_id=BKPJP6TQTF7G6RMNI3K3S23WZWXSGOJCDBIWSHMXULK4OQQYQCZA, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:11.071019640 (426h27m10.942102685s) [noise=true , address=45.91.116.14:4189 , piece_id=NOUKCZQ5LQT7H62V7OBWQXNFVCOETFGAFHM6LIQSZYYX5JOCX4RQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:11.141880171 (426h27m11.012963216s) [noise=true , address=145.79.227.48:4301 , piece_id=5IV2WAC77MWBYSQ4YRNZH2DH5BGG4XU3SVAZE5DLQJS7A2J4O3BA, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:11.149149884 (426h27m11.020232919s) [noise=true , address=145.79.227.38:4978 , piece_id=UAHAMPWHYQUY7YYS7VJPGTXBCPD6KWZADF2TVZIVU7ECDBF36NWQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:11.160413952 (426h27m11.031496997s) [noise=true , address=145.79.227.46:4602 , piece_id=UUIEHKP4PL7OS7IICNOB27C5SS5ZOIJPHC35K3R4IVTEXXV5SPZA, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:12.884195178 (426h27m12.755278223s) [noise=true , address=45.91.116.8:4046 , piece_id=BMFCPFPRSK2LFMCRWPCLJPVETOD5QL3VTHS6POMRVETSJOB72OOA, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:31.365089909 (426h27m31.236172944s) [noise=true , address=145.79.227.36:4518 , piece_id=RG4EIL5AGFVP2Z3J3Z7W6NJSA4LPDTFEXSLHZW722X3RSNG7H3LQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:31.715116052 (426h27m31.586199087s) [noise=true , address=145.79.227.31:4798 , piece_id=7SNRYHTEAWMXXH6KQ4UPURGY7PYJ72HOOCVA7YCIPLFG75NPNJYQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:32.549036955 (426h27m32.420119991s) [noise=true , address=145.79.227.35:4481 , piece_id=4TK6V52M3EHIBHFWCZJ5SLH6LZGKSRHPFTVL5LARU5JUVO7MGCKA, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:36.707023852 (426h27m36.578106887s) [noise=true , address=145.79.227.28:4725 , piece_id=EHBGDM7PYS2BZ2A7HKASKRHDNE3FGBX6W3GEEX7QHKPW5ZDZCNWQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:45348->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:45348->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45348->145.79.227.28:4725: use of closed network connection Dec 23 09:56:36.707046335 (426h27m36.57812937s) [noise=true , address=145.79.227.14:4839 , piece_id=YNIFPV6FLHOIEEPWL34RW7CLDTXTYTG7RKGAIMFZYT6OLIB5J7TQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:37586->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:37586->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37586->145.79.227.14:4839: use of closed network connection Dec 23 09:56:36.707164720 (426h27m36.578247765s) [noise=true , address=145.79.227.47:4267 , piece_id=QNH4GZCBVRW6FIBGILDQOHOZSBAYN2KR42RP4G77EL2RODLXRWZQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: ecclient: upload failed (node:1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8, address:145.79.227.47:4267): protocol: write tcp 188.166.225.170:42832->145.79.227.47:4267: use of closed network connection; write tcp 188.166.225.170:42832->145.79.227.47:4267: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42832->145.79.227.47:4267: use of closed network connection Dec 23 09:56:36.707206169 (426h27m36.578289214s) [noise=true , address=145.79.227.7:4293 , piece_id=YTBS6YPNK5BYOZLVYF5BX43SDZNNTTZJ473HE2XQPPZ4T3K6WMXQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: write tcp 188.166.225.170:46776->145.79.227.7:4293: use of closed network connection; write tcp 188.166.225.170:46776->145.79.227.7:4293: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46776->145.79.227.7:4293: use of closed network connection Dec 23 09:56:36.707318803 (426h27m36.578401838s) [noise=true , address=45.91.116.11:4118 , piece_id=VYLGLLGMTK2JZWHGLLTMUEYFVUH3AYU7P4UEYU4EMU5N5FVZHOMQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: ecclient: upload failed (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, address:45.91.116.11:4118): protocol: write tcp 188.166.225.170:34940->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:34940->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34940->45.91.116.11:4118: use of closed network connection Dec 23 09:56:36.707356044 (426h27m36.578439079s) [seg_pos={0 0}, upload=374371]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Dec 23 09:56:36.958957688 (426h27m36.830040733s) [upload=374371]: Done uploading segments... Dec 23 09:56:36.958995501 (426h27m36.830078546s) [upload=374371]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.852938705s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 29.969920666s trace: https://jaeger.dev.storj.io/trace/587ca7f719cacadf FAIL (2025-12-23 10:40:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== ploading segments... Dec 23 09:56:07.712403417 (426h27m7.583486462s) [upload=374371]: Got next segment. Inline: false Dec 23 09:56:07.712445147 (426h27m7.583528182s) [upload=374371]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 09:56:07.712452090 (426h27m7.583535125s) [upload=374371]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 09:56:08.088205172 (426h27m7.959288207s) [seg_pos={0 0}, upload=374371]: Begin upload segment... Dec 23 09:56:08.088258283 (426h27m7.959341328s) [seg_pos={0 0}, upload=374371]: Done begin upload segment. Dec 23 09:56:08.088290625 (426h27m7.95937367s) [noise=true , address=145.79.227.36:4518 , piece_id=RG4EIL5AGFVP2Z3J3Z7W6NJSA4LPDTFEXSLHZW722X3RSNG7H3LQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.088349738 (426h27m7.959432783s) [noise=true , address=145.79.227.38:4978 , piece_id=UAHAMPWHYQUY7YYS7VJPGTXBCPD6KWZADF2TVZIVU7ECDBF36NWQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.089542198 (426h27m7.960625233s) [noise=true , address=45.91.116.14:4189 , piece_id=NOUKCZQ5LQT7H62V7OBWQXNFVCOETFGAFHM6LIQSZYYX5JOCX4RQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.089679400 (426h27m7.960762445s) [noise=true , address=145.79.227.46:4602 , piece_id=UUIEHKP4PL7OS7IICNOB27C5SS5ZOIJPHC35K3R4IVTEXXV5SPZA, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.090450538 (426h27m7.961533583s) [noise=true , address=77.90.31.19:4842 , piece_id=FWDKIRAA4726YXGKXSB7PHUN7IZMNPV5ZSWKZAKZKJMQ45HDPLTQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.091327698 (426h27m7.962410743s) [noise=true , address=145.79.227.47:4267 , piece_id=QNH4GZCBVRW6FIBGILDQOHOZSBAYN2KR42RP4G77EL2RODLXRWZQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.092179080 (426h27m7.963262115s) [noise=true , address=77.90.31.17:4773 , piece_id=MU4UUWSSJNXVEOHAV2BWR2H4EWYFOEA2YNIQVQU6XQZIWWFDJXRQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.092774453 (426h27m7.963857498s) [noise=true , address=145.79.227.41:4085 , piece_id=BKPJP6TQTF7G6RMNI3K3S23WZWXSGOJCDBIWSHMXULK4OQQYQCZA, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.093537637 (426h27m7.964620682s) [noise=true , address=145.79.227.48:4301 , piece_id=5IV2WAC77MWBYSQ4YRNZH2DH5BGG4XU3SVAZE5DLQJS7A2J4O3BA, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.094292854 (426h27m7.965375919s) [noise=true , address=145.79.227.28:4725 , piece_id=EHBGDM7PYS2BZ2A7HKASKRHDNE3FGBX6W3GEEX7QHKPW5ZDZCNWQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.094608194 (426h27m7.965691239s) [noise=true , address=145.79.227.7:4293 , piece_id=YTBS6YPNK5BYOZLVYF5BX43SDZNNTTZJ473HE2XQPPZ4T3K6WMXQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.095183480 (426h27m7.966266525s) [noise=true , address=45.91.116.11:4118 , piece_id=VYLGLLGMTK2JZWHGLLTMUEYFVUH3AYU7P4UEYU4EMU5N5FVZHOMQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.095350008 (426h27m7.966433053s) [noise=true , address=145.79.227.35:4481 , piece_id=4TK6V52M3EHIBHFWCZJ5SLH6LZGKSRHPFTVL5LARU5JUVO7MGCKA, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.096434292 (426h27m7.967517337s) [noise=true , address=145.79.227.14:4839 , piece_id=YNIFPV6FLHOIEEPWL34RW7CLDTXTYTG7RKGAIMFZYT6OLIB5J7TQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.096712121 (426h27m7.967795166s) [noise=true , address=45.91.116.8:4046 , piece_id=BMFCPFPRSK2LFMCRWPCLJPVETOD5QL3VTHS6POMRVETSJOB72OOA, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.097512405 (426h27m7.96859546s) [noise=true , address=145.79.227.31:4798 , piece_id=7SNRYHTEAWMXXH6KQ4UPURGY7PYJ72HOOCVA7YCIPLFG75NPNJYQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:09.969534065 (426h27m9.8406171s) [upload=374371]: Next returned nil segment Dec 23 09:56:09.969568862 (426h27m9.840651897s) [upload=374371]: Waiting for error group managing segments... Dec 23 09:56:10.267890059 (426h27m10.138973104s) [noise=true , address=77.90.31.17:4773 , piece_id=MU4UUWSSJNXVEOHAV2BWR2H4EWYFOEA2YNIQVQU6XQZIWWFDJXRQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:10.424500182 (426h27m10.295583227s) [noise=true , address=77.90.31.19:4842 , piece_id=FWDKIRAA4726YXGKXSB7PHUN7IZMNPV5ZSWKZAKZKJMQ45HDPLTQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:10.842613724 (426h27m10.713696769s) [noise=true , address=145.79.227.41:4085 , piece_id=BKPJP6TQTF7G6RMNI3K3S23WZWXSGOJCDBIWSHMXULK4OQQYQCZA, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:11.071019640 (426h27m10.942102685s) [noise=true , address=45.91.116.14:4189 , piece_id=NOUKCZQ5LQT7H62V7OBWQXNFVCOETFGAFHM6LIQSZYYX5JOCX4RQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:11.141880171 (426h27m11.012963216s) [noise=true , address=145.79.227.48:4301 , piece_id=5IV2WAC77MWBYSQ4YRNZH2DH5BGG4XU3SVAZE5DLQJS7A2J4O3BA, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:11.149149884 (426h27m11.020232919s) [noise=true , address=145.79.227.38:4978 , piece_id=UAHAMPWHYQUY7YYS7VJPGTXBCPD6KWZADF2TVZIVU7ECDBF36NWQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:11.160413952 (426h27m11.031496997s) [noise=true , address=145.79.227.46:4602 , piece_id=UUIEHKP4PL7OS7IICNOB27C5SS5ZOIJPHC35K3R4IVTEXXV5SPZA, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:12.884195178 (426h27m12.755278223s) [noise=true , address=45.91.116.8:4046 , piece_id=BMFCPFPRSK2LFMCRWPCLJPVETOD5QL3VTHS6POMRVETSJOB72OOA, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:31.365089909 (426h27m31.236172944s) [noise=true , address=145.79.227.36:4518 , piece_id=RG4EIL5AGFVP2Z3J3Z7W6NJSA4LPDTFEXSLHZW722X3RSNG7H3LQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:31.715116052 (426h27m31.586199087s) [noise=true , address=145.79.227.31:4798 , piece_id=7SNRYHTEAWMXXH6KQ4UPURGY7PYJ72HOOCVA7YCIPLFG75NPNJYQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:32.549036955 (426h27m32.420119991s) [noise=true , address=145.79.227.35:4481 , piece_id=4TK6V52M3EHIBHFWCZJ5SLH6LZGKSRHPFTVL5LARU5JUVO7MGCKA, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:36.707023852 (426h27m36.578106887s) [noise=true , address=145.79.227.28:4725 , piece_id=EHBGDM7PYS2BZ2A7HKASKRHDNE3FGBX6W3GEEX7QHKPW5ZDZCNWQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:45348->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:45348->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45348->145.79.227.28:4725: use of closed network connection Dec 23 09:56:36.707046335 (426h27m36.57812937s) [noise=true , address=145.79.227.14:4839 , piece_id=YNIFPV6FLHOIEEPWL34RW7CLDTXTYTG7RKGAIMFZYT6OLIB5J7TQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:37586->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:37586->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37586->145.79.227.14:4839: use of closed network connection Dec 23 09:56:36.707164720 (426h27m36.578247765s) [noise=true , address=145.79.227.47:4267 , piece_id=QNH4GZCBVRW6FIBGILDQOHOZSBAYN2KR42RP4G77EL2RODLXRWZQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: ecclient: upload failed (node:1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8, address:145.79.227.47:4267): protocol: write tcp 188.166.225.170:42832->145.79.227.47:4267: use of closed network connection; write tcp 188.166.225.170:42832->145.79.227.47:4267: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42832->145.79.227.47:4267: use of closed network connection Dec 23 09:56:36.707206169 (426h27m36.578289214s) [noise=true , address=145.79.227.7:4293 , piece_id=YTBS6YPNK5BYOZLVYF5BX43SDZNNTTZJ473HE2XQPPZ4T3K6WMXQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: write tcp 188.166.225.170:46776->145.79.227.7:4293: use of closed network connection; write tcp 188.166.225.170:46776->145.79.227.7:4293: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46776->145.79.227.7:4293: use of closed network connection Dec 23 09:56:36.707318803 (426h27m36.578401838s) [noise=true , address=45.91.116.11:4118 , piece_id=VYLGLLGMTK2JZWHGLLTMUEYFVUH3AYU7P4UEYU4EMU5N5FVZHOMQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: ecclient: upload failed (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, address:45.91.116.11:4118): protocol: write tcp 188.166.225.170:34940->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:34940->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34940->45.91.116.11:4118: use of closed network connection Dec 23 09:56:36.707356044 (426h27m36.578439079s) [seg_pos={0 0}, upload=374371]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Dec 23 09:56:36.958957688 (426h27m36.830040733s) [upload=374371]: Done uploading segments... Dec 23 09:56:36.958995501 (426h27m36.830078546s) [upload=374371]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 29.96994355s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 6.113100619s trace: https://jaeger.dev.storj.io/trace/587ca7f719cacadf FAIL (2025-12-23 10:40:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== ploading segments... Dec 23 09:56:07.712403417 (426h27m7.583486462s) [upload=374371]: Got next segment. Inline: false Dec 23 09:56:07.712445147 (426h27m7.583528182s) [upload=374371]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 09:56:07.712452090 (426h27m7.583535125s) [upload=374371]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 09:56:08.088205172 (426h27m7.959288207s) [seg_pos={0 0}, upload=374371]: Begin upload segment... Dec 23 09:56:08.088258283 (426h27m7.959341328s) [seg_pos={0 0}, upload=374371]: Done begin upload segment. Dec 23 09:56:08.088290625 (426h27m7.95937367s) [noise=true , address=145.79.227.36:4518 , piece_id=RG4EIL5AGFVP2Z3J3Z7W6NJSA4LPDTFEXSLHZW722X3RSNG7H3LQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.088349738 (426h27m7.959432783s) [noise=true , address=145.79.227.38:4978 , piece_id=UAHAMPWHYQUY7YYS7VJPGTXBCPD6KWZADF2TVZIVU7ECDBF36NWQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.089542198 (426h27m7.960625233s) [noise=true , address=45.91.116.14:4189 , piece_id=NOUKCZQ5LQT7H62V7OBWQXNFVCOETFGAFHM6LIQSZYYX5JOCX4RQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.089679400 (426h27m7.960762445s) [noise=true , address=145.79.227.46:4602 , piece_id=UUIEHKP4PL7OS7IICNOB27C5SS5ZOIJPHC35K3R4IVTEXXV5SPZA, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.090450538 (426h27m7.961533583s) [noise=true , address=77.90.31.19:4842 , piece_id=FWDKIRAA4726YXGKXSB7PHUN7IZMNPV5ZSWKZAKZKJMQ45HDPLTQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.091327698 (426h27m7.962410743s) [noise=true , address=145.79.227.47:4267 , piece_id=QNH4GZCBVRW6FIBGILDQOHOZSBAYN2KR42RP4G77EL2RODLXRWZQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.092179080 (426h27m7.963262115s) [noise=true , address=77.90.31.17:4773 , piece_id=MU4UUWSSJNXVEOHAV2BWR2H4EWYFOEA2YNIQVQU6XQZIWWFDJXRQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.092774453 (426h27m7.963857498s) [noise=true , address=145.79.227.41:4085 , piece_id=BKPJP6TQTF7G6RMNI3K3S23WZWXSGOJCDBIWSHMXULK4OQQYQCZA, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.093537637 (426h27m7.964620682s) [noise=true , address=145.79.227.48:4301 , piece_id=5IV2WAC77MWBYSQ4YRNZH2DH5BGG4XU3SVAZE5DLQJS7A2J4O3BA, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.094292854 (426h27m7.965375919s) [noise=true , address=145.79.227.28:4725 , piece_id=EHBGDM7PYS2BZ2A7HKASKRHDNE3FGBX6W3GEEX7QHKPW5ZDZCNWQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.094608194 (426h27m7.965691239s) [noise=true , address=145.79.227.7:4293 , piece_id=YTBS6YPNK5BYOZLVYF5BX43SDZNNTTZJ473HE2XQPPZ4T3K6WMXQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.095183480 (426h27m7.966266525s) [noise=true , address=45.91.116.11:4118 , piece_id=VYLGLLGMTK2JZWHGLLTMUEYFVUH3AYU7P4UEYU4EMU5N5FVZHOMQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.095350008 (426h27m7.966433053s) [noise=true , address=145.79.227.35:4481 , piece_id=4TK6V52M3EHIBHFWCZJ5SLH6LZGKSRHPFTVL5LARU5JUVO7MGCKA, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.096434292 (426h27m7.967517337s) [noise=true , address=145.79.227.14:4839 , piece_id=YNIFPV6FLHOIEEPWL34RW7CLDTXTYTG7RKGAIMFZYT6OLIB5J7TQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.096712121 (426h27m7.967795166s) [noise=true , address=45.91.116.8:4046 , piece_id=BMFCPFPRSK2LFMCRWPCLJPVETOD5QL3VTHS6POMRVETSJOB72OOA, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.097512405 (426h27m7.96859546s) [noise=true , address=145.79.227.31:4798 , piece_id=7SNRYHTEAWMXXH6KQ4UPURGY7PYJ72HOOCVA7YCIPLFG75NPNJYQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:09.969534065 (426h27m9.8406171s) [upload=374371]: Next returned nil segment Dec 23 09:56:09.969568862 (426h27m9.840651897s) [upload=374371]: Waiting for error group managing segments... Dec 23 09:56:10.267890059 (426h27m10.138973104s) [noise=true , address=77.90.31.17:4773 , piece_id=MU4UUWSSJNXVEOHAV2BWR2H4EWYFOEA2YNIQVQU6XQZIWWFDJXRQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:10.424500182 (426h27m10.295583227s) [noise=true , address=77.90.31.19:4842 , piece_id=FWDKIRAA4726YXGKXSB7PHUN7IZMNPV5ZSWKZAKZKJMQ45HDPLTQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:10.842613724 (426h27m10.713696769s) [noise=true , address=145.79.227.41:4085 , piece_id=BKPJP6TQTF7G6RMNI3K3S23WZWXSGOJCDBIWSHMXULK4OQQYQCZA, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:11.071019640 (426h27m10.942102685s) [noise=true , address=45.91.116.14:4189 , piece_id=NOUKCZQ5LQT7H62V7OBWQXNFVCOETFGAFHM6LIQSZYYX5JOCX4RQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:11.141880171 (426h27m11.012963216s) [noise=true , address=145.79.227.48:4301 , piece_id=5IV2WAC77MWBYSQ4YRNZH2DH5BGG4XU3SVAZE5DLQJS7A2J4O3BA, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:11.149149884 (426h27m11.020232919s) [noise=true , address=145.79.227.38:4978 , piece_id=UAHAMPWHYQUY7YYS7VJPGTXBCPD6KWZADF2TVZIVU7ECDBF36NWQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:11.160413952 (426h27m11.031496997s) [noise=true , address=145.79.227.46:4602 , piece_id=UUIEHKP4PL7OS7IICNOB27C5SS5ZOIJPHC35K3R4IVTEXXV5SPZA, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:12.884195178 (426h27m12.755278223s) [noise=true , address=45.91.116.8:4046 , piece_id=BMFCPFPRSK2LFMCRWPCLJPVETOD5QL3VTHS6POMRVETSJOB72OOA, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:31.365089909 (426h27m31.236172944s) [noise=true , address=145.79.227.36:4518 , piece_id=RG4EIL5AGFVP2Z3J3Z7W6NJSA4LPDTFEXSLHZW722X3RSNG7H3LQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:31.715116052 (426h27m31.586199087s) [noise=true , address=145.79.227.31:4798 , piece_id=7SNRYHTEAWMXXH6KQ4UPURGY7PYJ72HOOCVA7YCIPLFG75NPNJYQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:32.549036955 (426h27m32.420119991s) [noise=true , address=145.79.227.35:4481 , piece_id=4TK6V52M3EHIBHFWCZJ5SLH6LZGKSRHPFTVL5LARU5JUVO7MGCKA, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:36.707023852 (426h27m36.578106887s) [noise=true , address=145.79.227.28:4725 , piece_id=EHBGDM7PYS2BZ2A7HKASKRHDNE3FGBX6W3GEEX7QHKPW5ZDZCNWQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:45348->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:45348->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45348->145.79.227.28:4725: use of closed network connection Dec 23 09:56:36.707046335 (426h27m36.57812937s) [noise=true , address=145.79.227.14:4839 , piece_id=YNIFPV6FLHOIEEPWL34RW7CLDTXTYTG7RKGAIMFZYT6OLIB5J7TQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:37586->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:37586->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37586->145.79.227.14:4839: use of closed network connection Dec 23 09:56:36.707164720 (426h27m36.578247765s) [noise=true , address=145.79.227.47:4267 , piece_id=QNH4GZCBVRW6FIBGILDQOHOZSBAYN2KR42RP4G77EL2RODLXRWZQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: ecclient: upload failed (node:1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8, address:145.79.227.47:4267): protocol: write tcp 188.166.225.170:42832->145.79.227.47:4267: use of closed network connection; write tcp 188.166.225.170:42832->145.79.227.47:4267: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42832->145.79.227.47:4267: use of closed network connection Dec 23 09:56:36.707206169 (426h27m36.578289214s) [noise=true , address=145.79.227.7:4293 , piece_id=YTBS6YPNK5BYOZLVYF5BX43SDZNNTTZJ473HE2XQPPZ4T3K6WMXQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: write tcp 188.166.225.170:46776->145.79.227.7:4293: use of closed network connection; write tcp 188.166.225.170:46776->145.79.227.7:4293: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46776->145.79.227.7:4293: use of closed network connection Dec 23 09:56:36.707318803 (426h27m36.578401838s) [noise=true , address=45.91.116.11:4118 , piece_id=VYLGLLGMTK2JZWHGLLTMUEYFVUH3AYU7P4UEYU4EMU5N5FVZHOMQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: ecclient: upload failed (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, address:45.91.116.11:4118): protocol: write tcp 188.166.225.170:34940->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:34940->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34940->45.91.116.11:4118: use of closed network connection Dec 23 09:56:36.707356044 (426h27m36.578439079s) [seg_pos={0 0}, upload=374371]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Dec 23 09:56:36.958957688 (426h27m36.830040733s) [upload=374371]: Done uploading segments... Dec 23 09:56:36.958995501 (426h27m36.830078546s) [upload=374371]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.113130677s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 6.31903128s trace: https://jaeger.dev.storj.io/trace/587ca7f719cacadf FAIL (2025-12-23 10:39:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== ploading segments... Dec 23 09:56:07.712403417 (426h27m7.583486462s) [upload=374371]: Got next segment. Inline: false Dec 23 09:56:07.712445147 (426h27m7.583528182s) [upload=374371]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 09:56:07.712452090 (426h27m7.583535125s) [upload=374371]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 09:56:08.088205172 (426h27m7.959288207s) [seg_pos={0 0}, upload=374371]: Begin upload segment... Dec 23 09:56:08.088258283 (426h27m7.959341328s) [seg_pos={0 0}, upload=374371]: Done begin upload segment. Dec 23 09:56:08.088290625 (426h27m7.95937367s) [noise=true , address=145.79.227.36:4518 , piece_id=RG4EIL5AGFVP2Z3J3Z7W6NJSA4LPDTFEXSLHZW722X3RSNG7H3LQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.088349738 (426h27m7.959432783s) [noise=true , address=145.79.227.38:4978 , piece_id=UAHAMPWHYQUY7YYS7VJPGTXBCPD6KWZADF2TVZIVU7ECDBF36NWQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.089542198 (426h27m7.960625233s) [noise=true , address=45.91.116.14:4189 , piece_id=NOUKCZQ5LQT7H62V7OBWQXNFVCOETFGAFHM6LIQSZYYX5JOCX4RQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.089679400 (426h27m7.960762445s) [noise=true , address=145.79.227.46:4602 , piece_id=UUIEHKP4PL7OS7IICNOB27C5SS5ZOIJPHC35K3R4IVTEXXV5SPZA, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.090450538 (426h27m7.961533583s) [noise=true , address=77.90.31.19:4842 , piece_id=FWDKIRAA4726YXGKXSB7PHUN7IZMNPV5ZSWKZAKZKJMQ45HDPLTQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.091327698 (426h27m7.962410743s) [noise=true , address=145.79.227.47:4267 , piece_id=QNH4GZCBVRW6FIBGILDQOHOZSBAYN2KR42RP4G77EL2RODLXRWZQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.092179080 (426h27m7.963262115s) [noise=true , address=77.90.31.17:4773 , piece_id=MU4UUWSSJNXVEOHAV2BWR2H4EWYFOEA2YNIQVQU6XQZIWWFDJXRQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.092774453 (426h27m7.963857498s) [noise=true , address=145.79.227.41:4085 , piece_id=BKPJP6TQTF7G6RMNI3K3S23WZWXSGOJCDBIWSHMXULK4OQQYQCZA, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.093537637 (426h27m7.964620682s) [noise=true , address=145.79.227.48:4301 , piece_id=5IV2WAC77MWBYSQ4YRNZH2DH5BGG4XU3SVAZE5DLQJS7A2J4O3BA, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.094292854 (426h27m7.965375919s) [noise=true , address=145.79.227.28:4725 , piece_id=EHBGDM7PYS2BZ2A7HKASKRHDNE3FGBX6W3GEEX7QHKPW5ZDZCNWQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.094608194 (426h27m7.965691239s) [noise=true , address=145.79.227.7:4293 , piece_id=YTBS6YPNK5BYOZLVYF5BX43SDZNNTTZJ473HE2XQPPZ4T3K6WMXQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.095183480 (426h27m7.966266525s) [noise=true , address=45.91.116.11:4118 , piece_id=VYLGLLGMTK2JZWHGLLTMUEYFVUH3AYU7P4UEYU4EMU5N5FVZHOMQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.095350008 (426h27m7.966433053s) [noise=true , address=145.79.227.35:4481 , piece_id=4TK6V52M3EHIBHFWCZJ5SLH6LZGKSRHPFTVL5LARU5JUVO7MGCKA, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.096434292 (426h27m7.967517337s) [noise=true , address=145.79.227.14:4839 , piece_id=YNIFPV6FLHOIEEPWL34RW7CLDTXTYTG7RKGAIMFZYT6OLIB5J7TQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.096712121 (426h27m7.967795166s) [noise=true , address=45.91.116.8:4046 , piece_id=BMFCPFPRSK2LFMCRWPCLJPVETOD5QL3VTHS6POMRVETSJOB72OOA, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.097512405 (426h27m7.96859546s) [noise=true , address=145.79.227.31:4798 , piece_id=7SNRYHTEAWMXXH6KQ4UPURGY7PYJ72HOOCVA7YCIPLFG75NPNJYQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:09.969534065 (426h27m9.8406171s) [upload=374371]: Next returned nil segment Dec 23 09:56:09.969568862 (426h27m9.840651897s) [upload=374371]: Waiting for error group managing segments... Dec 23 09:56:10.267890059 (426h27m10.138973104s) [noise=true , address=77.90.31.17:4773 , piece_id=MU4UUWSSJNXVEOHAV2BWR2H4EWYFOEA2YNIQVQU6XQZIWWFDJXRQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:10.424500182 (426h27m10.295583227s) [noise=true , address=77.90.31.19:4842 , piece_id=FWDKIRAA4726YXGKXSB7PHUN7IZMNPV5ZSWKZAKZKJMQ45HDPLTQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:10.842613724 (426h27m10.713696769s) [noise=true , address=145.79.227.41:4085 , piece_id=BKPJP6TQTF7G6RMNI3K3S23WZWXSGOJCDBIWSHMXULK4OQQYQCZA, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:11.071019640 (426h27m10.942102685s) [noise=true , address=45.91.116.14:4189 , piece_id=NOUKCZQ5LQT7H62V7OBWQXNFVCOETFGAFHM6LIQSZYYX5JOCX4RQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:11.141880171 (426h27m11.012963216s) [noise=true , address=145.79.227.48:4301 , piece_id=5IV2WAC77MWBYSQ4YRNZH2DH5BGG4XU3SVAZE5DLQJS7A2J4O3BA, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:11.149149884 (426h27m11.020232919s) [noise=true , address=145.79.227.38:4978 , piece_id=UAHAMPWHYQUY7YYS7VJPGTXBCPD6KWZADF2TVZIVU7ECDBF36NWQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:11.160413952 (426h27m11.031496997s) [noise=true , address=145.79.227.46:4602 , piece_id=UUIEHKP4PL7OS7IICNOB27C5SS5ZOIJPHC35K3R4IVTEXXV5SPZA, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:12.884195178 (426h27m12.755278223s) [noise=true , address=45.91.116.8:4046 , piece_id=BMFCPFPRSK2LFMCRWPCLJPVETOD5QL3VTHS6POMRVETSJOB72OOA, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:31.365089909 (426h27m31.236172944s) [noise=true , address=145.79.227.36:4518 , piece_id=RG4EIL5AGFVP2Z3J3Z7W6NJSA4LPDTFEXSLHZW722X3RSNG7H3LQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:31.715116052 (426h27m31.586199087s) [noise=true , address=145.79.227.31:4798 , piece_id=7SNRYHTEAWMXXH6KQ4UPURGY7PYJ72HOOCVA7YCIPLFG75NPNJYQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:32.549036955 (426h27m32.420119991s) [noise=true , address=145.79.227.35:4481 , piece_id=4TK6V52M3EHIBHFWCZJ5SLH6LZGKSRHPFTVL5LARU5JUVO7MGCKA, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:36.707023852 (426h27m36.578106887s) [noise=true , address=145.79.227.28:4725 , piece_id=EHBGDM7PYS2BZ2A7HKASKRHDNE3FGBX6W3GEEX7QHKPW5ZDZCNWQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:45348->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:45348->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45348->145.79.227.28:4725: use of closed network connection Dec 23 09:56:36.707046335 (426h27m36.57812937s) [noise=true , address=145.79.227.14:4839 , piece_id=YNIFPV6FLHOIEEPWL34RW7CLDTXTYTG7RKGAIMFZYT6OLIB5J7TQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:37586->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:37586->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37586->145.79.227.14:4839: use of closed network connection Dec 23 09:56:36.707164720 (426h27m36.578247765s) [noise=true , address=145.79.227.47:4267 , piece_id=QNH4GZCBVRW6FIBGILDQOHOZSBAYN2KR42RP4G77EL2RODLXRWZQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: ecclient: upload failed (node:1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8, address:145.79.227.47:4267): protocol: write tcp 188.166.225.170:42832->145.79.227.47:4267: use of closed network connection; write tcp 188.166.225.170:42832->145.79.227.47:4267: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42832->145.79.227.47:4267: use of closed network connection Dec 23 09:56:36.707206169 (426h27m36.578289214s) [noise=true , address=145.79.227.7:4293 , piece_id=YTBS6YPNK5BYOZLVYF5BX43SDZNNTTZJ473HE2XQPPZ4T3K6WMXQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: write tcp 188.166.225.170:46776->145.79.227.7:4293: use of closed network connection; write tcp 188.166.225.170:46776->145.79.227.7:4293: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46776->145.79.227.7:4293: use of closed network connection Dec 23 09:56:36.707318803 (426h27m36.578401838s) [noise=true , address=45.91.116.11:4118 , piece_id=VYLGLLGMTK2JZWHGLLTMUEYFVUH3AYU7P4UEYU4EMU5N5FVZHOMQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: ecclient: upload failed (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, address:45.91.116.11:4118): protocol: write tcp 188.166.225.170:34940->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:34940->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34940->45.91.116.11:4118: use of closed network connection Dec 23 09:56:36.707356044 (426h27m36.578439079s) [seg_pos={0 0}, upload=374371]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Dec 23 09:56:36.958957688 (426h27m36.830040733s) [upload=374371]: Done uploading segments... Dec 23 09:56:36.958995501 (426h27m36.830078546s) [upload=374371]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.319066166s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 5.967355947s trace: https://jaeger.dev.storj.io/trace/587ca7f719cacadf FAIL (2025-12-23 10:38:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== ploading segments... Dec 23 09:56:07.712403417 (426h27m7.583486462s) [upload=374371]: Got next segment. Inline: false Dec 23 09:56:07.712445147 (426h27m7.583528182s) [upload=374371]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 09:56:07.712452090 (426h27m7.583535125s) [upload=374371]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 09:56:08.088205172 (426h27m7.959288207s) [seg_pos={0 0}, upload=374371]: Begin upload segment... Dec 23 09:56:08.088258283 (426h27m7.959341328s) [seg_pos={0 0}, upload=374371]: Done begin upload segment. Dec 23 09:56:08.088290625 (426h27m7.95937367s) [noise=true , address=145.79.227.36:4518 , piece_id=RG4EIL5AGFVP2Z3J3Z7W6NJSA4LPDTFEXSLHZW722X3RSNG7H3LQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.088349738 (426h27m7.959432783s) [noise=true , address=145.79.227.38:4978 , piece_id=UAHAMPWHYQUY7YYS7VJPGTXBCPD6KWZADF2TVZIVU7ECDBF36NWQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.089542198 (426h27m7.960625233s) [noise=true , address=45.91.116.14:4189 , piece_id=NOUKCZQ5LQT7H62V7OBWQXNFVCOETFGAFHM6LIQSZYYX5JOCX4RQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.089679400 (426h27m7.960762445s) [noise=true , address=145.79.227.46:4602 , piece_id=UUIEHKP4PL7OS7IICNOB27C5SS5ZOIJPHC35K3R4IVTEXXV5SPZA, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.090450538 (426h27m7.961533583s) [noise=true , address=77.90.31.19:4842 , piece_id=FWDKIRAA4726YXGKXSB7PHUN7IZMNPV5ZSWKZAKZKJMQ45HDPLTQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.091327698 (426h27m7.962410743s) [noise=true , address=145.79.227.47:4267 , piece_id=QNH4GZCBVRW6FIBGILDQOHOZSBAYN2KR42RP4G77EL2RODLXRWZQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.092179080 (426h27m7.963262115s) [noise=true , address=77.90.31.17:4773 , piece_id=MU4UUWSSJNXVEOHAV2BWR2H4EWYFOEA2YNIQVQU6XQZIWWFDJXRQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.092774453 (426h27m7.963857498s) [noise=true , address=145.79.227.41:4085 , piece_id=BKPJP6TQTF7G6RMNI3K3S23WZWXSGOJCDBIWSHMXULK4OQQYQCZA, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.093537637 (426h27m7.964620682s) [noise=true , address=145.79.227.48:4301 , piece_id=5IV2WAC77MWBYSQ4YRNZH2DH5BGG4XU3SVAZE5DLQJS7A2J4O3BA, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.094292854 (426h27m7.965375919s) [noise=true , address=145.79.227.28:4725 , piece_id=EHBGDM7PYS2BZ2A7HKASKRHDNE3FGBX6W3GEEX7QHKPW5ZDZCNWQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.094608194 (426h27m7.965691239s) [noise=true , address=145.79.227.7:4293 , piece_id=YTBS6YPNK5BYOZLVYF5BX43SDZNNTTZJ473HE2XQPPZ4T3K6WMXQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.095183480 (426h27m7.966266525s) [noise=true , address=45.91.116.11:4118 , piece_id=VYLGLLGMTK2JZWHGLLTMUEYFVUH3AYU7P4UEYU4EMU5N5FVZHOMQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.095350008 (426h27m7.966433053s) [noise=true , address=145.79.227.35:4481 , piece_id=4TK6V52M3EHIBHFWCZJ5SLH6LZGKSRHPFTVL5LARU5JUVO7MGCKA, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.096434292 (426h27m7.967517337s) [noise=true , address=145.79.227.14:4839 , piece_id=YNIFPV6FLHOIEEPWL34RW7CLDTXTYTG7RKGAIMFZYT6OLIB5J7TQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.096712121 (426h27m7.967795166s) [noise=true , address=45.91.116.8:4046 , piece_id=BMFCPFPRSK2LFMCRWPCLJPVETOD5QL3VTHS6POMRVETSJOB72OOA, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.097512405 (426h27m7.96859546s) [noise=true , address=145.79.227.31:4798 , piece_id=7SNRYHTEAWMXXH6KQ4UPURGY7PYJ72HOOCVA7YCIPLFG75NPNJYQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:09.969534065 (426h27m9.8406171s) [upload=374371]: Next returned nil segment Dec 23 09:56:09.969568862 (426h27m9.840651897s) [upload=374371]: Waiting for error group managing segments... Dec 23 09:56:10.267890059 (426h27m10.138973104s) [noise=true , address=77.90.31.17:4773 , piece_id=MU4UUWSSJNXVEOHAV2BWR2H4EWYFOEA2YNIQVQU6XQZIWWFDJXRQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:10.424500182 (426h27m10.295583227s) [noise=true , address=77.90.31.19:4842 , piece_id=FWDKIRAA4726YXGKXSB7PHUN7IZMNPV5ZSWKZAKZKJMQ45HDPLTQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:10.842613724 (426h27m10.713696769s) [noise=true , address=145.79.227.41:4085 , piece_id=BKPJP6TQTF7G6RMNI3K3S23WZWXSGOJCDBIWSHMXULK4OQQYQCZA, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:11.071019640 (426h27m10.942102685s) [noise=true , address=45.91.116.14:4189 , piece_id=NOUKCZQ5LQT7H62V7OBWQXNFVCOETFGAFHM6LIQSZYYX5JOCX4RQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:11.141880171 (426h27m11.012963216s) [noise=true , address=145.79.227.48:4301 , piece_id=5IV2WAC77MWBYSQ4YRNZH2DH5BGG4XU3SVAZE5DLQJS7A2J4O3BA, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:11.149149884 (426h27m11.020232919s) [noise=true , address=145.79.227.38:4978 , piece_id=UAHAMPWHYQUY7YYS7VJPGTXBCPD6KWZADF2TVZIVU7ECDBF36NWQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:11.160413952 (426h27m11.031496997s) [noise=true , address=145.79.227.46:4602 , piece_id=UUIEHKP4PL7OS7IICNOB27C5SS5ZOIJPHC35K3R4IVTEXXV5SPZA, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:12.884195178 (426h27m12.755278223s) [noise=true , address=45.91.116.8:4046 , piece_id=BMFCPFPRSK2LFMCRWPCLJPVETOD5QL3VTHS6POMRVETSJOB72OOA, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:31.365089909 (426h27m31.236172944s) [noise=true , address=145.79.227.36:4518 , piece_id=RG4EIL5AGFVP2Z3J3Z7W6NJSA4LPDTFEXSLHZW722X3RSNG7H3LQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:31.715116052 (426h27m31.586199087s) [noise=true , address=145.79.227.31:4798 , piece_id=7SNRYHTEAWMXXH6KQ4UPURGY7PYJ72HOOCVA7YCIPLFG75NPNJYQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:32.549036955 (426h27m32.420119991s) [noise=true , address=145.79.227.35:4481 , piece_id=4TK6V52M3EHIBHFWCZJ5SLH6LZGKSRHPFTVL5LARU5JUVO7MGCKA, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:36.707023852 (426h27m36.578106887s) [noise=true , address=145.79.227.28:4725 , piece_id=EHBGDM7PYS2BZ2A7HKASKRHDNE3FGBX6W3GEEX7QHKPW5ZDZCNWQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:45348->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:45348->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45348->145.79.227.28:4725: use of closed network connection Dec 23 09:56:36.707046335 (426h27m36.57812937s) [noise=true , address=145.79.227.14:4839 , piece_id=YNIFPV6FLHOIEEPWL34RW7CLDTXTYTG7RKGAIMFZYT6OLIB5J7TQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:37586->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:37586->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37586->145.79.227.14:4839: use of closed network connection Dec 23 09:56:36.707164720 (426h27m36.578247765s) [noise=true , address=145.79.227.47:4267 , piece_id=QNH4GZCBVRW6FIBGILDQOHOZSBAYN2KR42RP4G77EL2RODLXRWZQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: ecclient: upload failed (node:1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8, address:145.79.227.47:4267): protocol: write tcp 188.166.225.170:42832->145.79.227.47:4267: use of closed network connection; write tcp 188.166.225.170:42832->145.79.227.47:4267: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42832->145.79.227.47:4267: use of closed network connection Dec 23 09:56:36.707206169 (426h27m36.578289214s) [noise=true , address=145.79.227.7:4293 , piece_id=YTBS6YPNK5BYOZLVYF5BX43SDZNNTTZJ473HE2XQPPZ4T3K6WMXQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: write tcp 188.166.225.170:46776->145.79.227.7:4293: use of closed network connection; write tcp 188.166.225.170:46776->145.79.227.7:4293: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46776->145.79.227.7:4293: use of closed network connection Dec 23 09:56:36.707318803 (426h27m36.578401838s) [noise=true , address=45.91.116.11:4118 , piece_id=VYLGLLGMTK2JZWHGLLTMUEYFVUH3AYU7P4UEYU4EMU5N5FVZHOMQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: ecclient: upload failed (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, address:45.91.116.11:4118): protocol: write tcp 188.166.225.170:34940->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:34940->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34940->45.91.116.11:4118: use of closed network connection Dec 23 09:56:36.707356044 (426h27m36.578439079s) [seg_pos={0 0}, upload=374371]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Dec 23 09:56:36.958957688 (426h27m36.830040733s) [upload=374371]: Done uploading segments... Dec 23 09:56:36.958995501 (426h27m36.830078546s) [upload=374371]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.967379801s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 5.976415145s trace: https://jaeger.dev.storj.io/trace/587ca7f719cacadf FAIL (2025-12-23 10:37:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== ploading segments... Dec 23 09:56:07.712403417 (426h27m7.583486462s) [upload=374371]: Got next segment. Inline: false Dec 23 09:56:07.712445147 (426h27m7.583528182s) [upload=374371]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 09:56:07.712452090 (426h27m7.583535125s) [upload=374371]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 09:56:08.088205172 (426h27m7.959288207s) [seg_pos={0 0}, upload=374371]: Begin upload segment... Dec 23 09:56:08.088258283 (426h27m7.959341328s) [seg_pos={0 0}, upload=374371]: Done begin upload segment. Dec 23 09:56:08.088290625 (426h27m7.95937367s) [noise=true , address=145.79.227.36:4518 , piece_id=RG4EIL5AGFVP2Z3J3Z7W6NJSA4LPDTFEXSLHZW722X3RSNG7H3LQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.088349738 (426h27m7.959432783s) [noise=true , address=145.79.227.38:4978 , piece_id=UAHAMPWHYQUY7YYS7VJPGTXBCPD6KWZADF2TVZIVU7ECDBF36NWQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.089542198 (426h27m7.960625233s) [noise=true , address=45.91.116.14:4189 , piece_id=NOUKCZQ5LQT7H62V7OBWQXNFVCOETFGAFHM6LIQSZYYX5JOCX4RQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.089679400 (426h27m7.960762445s) [noise=true , address=145.79.227.46:4602 , piece_id=UUIEHKP4PL7OS7IICNOB27C5SS5ZOIJPHC35K3R4IVTEXXV5SPZA, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.090450538 (426h27m7.961533583s) [noise=true , address=77.90.31.19:4842 , piece_id=FWDKIRAA4726YXGKXSB7PHUN7IZMNPV5ZSWKZAKZKJMQ45HDPLTQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.091327698 (426h27m7.962410743s) [noise=true , address=145.79.227.47:4267 , piece_id=QNH4GZCBVRW6FIBGILDQOHOZSBAYN2KR42RP4G77EL2RODLXRWZQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.092179080 (426h27m7.963262115s) [noise=true , address=77.90.31.17:4773 , piece_id=MU4UUWSSJNXVEOHAV2BWR2H4EWYFOEA2YNIQVQU6XQZIWWFDJXRQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.092774453 (426h27m7.963857498s) [noise=true , address=145.79.227.41:4085 , piece_id=BKPJP6TQTF7G6RMNI3K3S23WZWXSGOJCDBIWSHMXULK4OQQYQCZA, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.093537637 (426h27m7.964620682s) [noise=true , address=145.79.227.48:4301 , piece_id=5IV2WAC77MWBYSQ4YRNZH2DH5BGG4XU3SVAZE5DLQJS7A2J4O3BA, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.094292854 (426h27m7.965375919s) [noise=true , address=145.79.227.28:4725 , piece_id=EHBGDM7PYS2BZ2A7HKASKRHDNE3FGBX6W3GEEX7QHKPW5ZDZCNWQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.094608194 (426h27m7.965691239s) [noise=true , address=145.79.227.7:4293 , piece_id=YTBS6YPNK5BYOZLVYF5BX43SDZNNTTZJ473HE2XQPPZ4T3K6WMXQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.095183480 (426h27m7.966266525s) [noise=true , address=45.91.116.11:4118 , piece_id=VYLGLLGMTK2JZWHGLLTMUEYFVUH3AYU7P4UEYU4EMU5N5FVZHOMQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.095350008 (426h27m7.966433053s) [noise=true , address=145.79.227.35:4481 , piece_id=4TK6V52M3EHIBHFWCZJ5SLH6LZGKSRHPFTVL5LARU5JUVO7MGCKA, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.096434292 (426h27m7.967517337s) [noise=true , address=145.79.227.14:4839 , piece_id=YNIFPV6FLHOIEEPWL34RW7CLDTXTYTG7RKGAIMFZYT6OLIB5J7TQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.096712121 (426h27m7.967795166s) [noise=true , address=45.91.116.8:4046 , piece_id=BMFCPFPRSK2LFMCRWPCLJPVETOD5QL3VTHS6POMRVETSJOB72OOA, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.097512405 (426h27m7.96859546s) [noise=true , address=145.79.227.31:4798 , piece_id=7SNRYHTEAWMXXH6KQ4UPURGY7PYJ72HOOCVA7YCIPLFG75NPNJYQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:09.969534065 (426h27m9.8406171s) [upload=374371]: Next returned nil segment Dec 23 09:56:09.969568862 (426h27m9.840651897s) [upload=374371]: Waiting for error group managing segments... Dec 23 09:56:10.267890059 (426h27m10.138973104s) [noise=true , address=77.90.31.17:4773 , piece_id=MU4UUWSSJNXVEOHAV2BWR2H4EWYFOEA2YNIQVQU6XQZIWWFDJXRQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:10.424500182 (426h27m10.295583227s) [noise=true , address=77.90.31.19:4842 , piece_id=FWDKIRAA4726YXGKXSB7PHUN7IZMNPV5ZSWKZAKZKJMQ45HDPLTQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:10.842613724 (426h27m10.713696769s) [noise=true , address=145.79.227.41:4085 , piece_id=BKPJP6TQTF7G6RMNI3K3S23WZWXSGOJCDBIWSHMXULK4OQQYQCZA, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:11.071019640 (426h27m10.942102685s) [noise=true , address=45.91.116.14:4189 , piece_id=NOUKCZQ5LQT7H62V7OBWQXNFVCOETFGAFHM6LIQSZYYX5JOCX4RQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:11.141880171 (426h27m11.012963216s) [noise=true , address=145.79.227.48:4301 , piece_id=5IV2WAC77MWBYSQ4YRNZH2DH5BGG4XU3SVAZE5DLQJS7A2J4O3BA, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:11.149149884 (426h27m11.020232919s) [noise=true , address=145.79.227.38:4978 , piece_id=UAHAMPWHYQUY7YYS7VJPGTXBCPD6KWZADF2TVZIVU7ECDBF36NWQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:11.160413952 (426h27m11.031496997s) [noise=true , address=145.79.227.46:4602 , piece_id=UUIEHKP4PL7OS7IICNOB27C5SS5ZOIJPHC35K3R4IVTEXXV5SPZA, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:12.884195178 (426h27m12.755278223s) [noise=true , address=45.91.116.8:4046 , piece_id=BMFCPFPRSK2LFMCRWPCLJPVETOD5QL3VTHS6POMRVETSJOB72OOA, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:31.365089909 (426h27m31.236172944s) [noise=true , address=145.79.227.36:4518 , piece_id=RG4EIL5AGFVP2Z3J3Z7W6NJSA4LPDTFEXSLHZW722X3RSNG7H3LQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:31.715116052 (426h27m31.586199087s) [noise=true , address=145.79.227.31:4798 , piece_id=7SNRYHTEAWMXXH6KQ4UPURGY7PYJ72HOOCVA7YCIPLFG75NPNJYQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:32.549036955 (426h27m32.420119991s) [noise=true , address=145.79.227.35:4481 , piece_id=4TK6V52M3EHIBHFWCZJ5SLH6LZGKSRHPFTVL5LARU5JUVO7MGCKA, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:36.707023852 (426h27m36.578106887s) [noise=true , address=145.79.227.28:4725 , piece_id=EHBGDM7PYS2BZ2A7HKASKRHDNE3FGBX6W3GEEX7QHKPW5ZDZCNWQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:45348->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:45348->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45348->145.79.227.28:4725: use of closed network connection Dec 23 09:56:36.707046335 (426h27m36.57812937s) [noise=true , address=145.79.227.14:4839 , piece_id=YNIFPV6FLHOIEEPWL34RW7CLDTXTYTG7RKGAIMFZYT6OLIB5J7TQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:37586->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:37586->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37586->145.79.227.14:4839: use of closed network connection Dec 23 09:56:36.707164720 (426h27m36.578247765s) [noise=true , address=145.79.227.47:4267 , piece_id=QNH4GZCBVRW6FIBGILDQOHOZSBAYN2KR42RP4G77EL2RODLXRWZQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: ecclient: upload failed (node:1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8, address:145.79.227.47:4267): protocol: write tcp 188.166.225.170:42832->145.79.227.47:4267: use of closed network connection; write tcp 188.166.225.170:42832->145.79.227.47:4267: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42832->145.79.227.47:4267: use of closed network connection Dec 23 09:56:36.707206169 (426h27m36.578289214s) [noise=true , address=145.79.227.7:4293 , piece_id=YTBS6YPNK5BYOZLVYF5BX43SDZNNTTZJ473HE2XQPPZ4T3K6WMXQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: write tcp 188.166.225.170:46776->145.79.227.7:4293: use of closed network connection; write tcp 188.166.225.170:46776->145.79.227.7:4293: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46776->145.79.227.7:4293: use of closed network connection Dec 23 09:56:36.707318803 (426h27m36.578401838s) [noise=true , address=45.91.116.11:4118 , piece_id=VYLGLLGMTK2JZWHGLLTMUEYFVUH3AYU7P4UEYU4EMU5N5FVZHOMQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: ecclient: upload failed (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, address:45.91.116.11:4118): protocol: write tcp 188.166.225.170:34940->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:34940->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34940->45.91.116.11:4118: use of closed network connection Dec 23 09:56:36.707356044 (426h27m36.578439079s) [seg_pos={0 0}, upload=374371]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Dec 23 09:56:36.958957688 (426h27m36.830040733s) [upload=374371]: Done uploading segments... Dec 23 09:56:36.958995501 (426h27m36.830078546s) [upload=374371]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.976448198s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 5.681462064s trace: https://jaeger.dev.storj.io/trace/587ca7f719cacadf FAIL (2025-12-23 10:36:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== ploading segments... Dec 23 09:56:07.712403417 (426h27m7.583486462s) [upload=374371]: Got next segment. Inline: false Dec 23 09:56:07.712445147 (426h27m7.583528182s) [upload=374371]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 09:56:07.712452090 (426h27m7.583535125s) [upload=374371]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 09:56:08.088205172 (426h27m7.959288207s) [seg_pos={0 0}, upload=374371]: Begin upload segment... Dec 23 09:56:08.088258283 (426h27m7.959341328s) [seg_pos={0 0}, upload=374371]: Done begin upload segment. Dec 23 09:56:08.088290625 (426h27m7.95937367s) [noise=true , address=145.79.227.36:4518 , piece_id=RG4EIL5AGFVP2Z3J3Z7W6NJSA4LPDTFEXSLHZW722X3RSNG7H3LQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.088349738 (426h27m7.959432783s) [noise=true , address=145.79.227.38:4978 , piece_id=UAHAMPWHYQUY7YYS7VJPGTXBCPD6KWZADF2TVZIVU7ECDBF36NWQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.089542198 (426h27m7.960625233s) [noise=true , address=45.91.116.14:4189 , piece_id=NOUKCZQ5LQT7H62V7OBWQXNFVCOETFGAFHM6LIQSZYYX5JOCX4RQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.089679400 (426h27m7.960762445s) [noise=true , address=145.79.227.46:4602 , piece_id=UUIEHKP4PL7OS7IICNOB27C5SS5ZOIJPHC35K3R4IVTEXXV5SPZA, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.090450538 (426h27m7.961533583s) [noise=true , address=77.90.31.19:4842 , piece_id=FWDKIRAA4726YXGKXSB7PHUN7IZMNPV5ZSWKZAKZKJMQ45HDPLTQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.091327698 (426h27m7.962410743s) [noise=true , address=145.79.227.47:4267 , piece_id=QNH4GZCBVRW6FIBGILDQOHOZSBAYN2KR42RP4G77EL2RODLXRWZQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.092179080 (426h27m7.963262115s) [noise=true , address=77.90.31.17:4773 , piece_id=MU4UUWSSJNXVEOHAV2BWR2H4EWYFOEA2YNIQVQU6XQZIWWFDJXRQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.092774453 (426h27m7.963857498s) [noise=true , address=145.79.227.41:4085 , piece_id=BKPJP6TQTF7G6RMNI3K3S23WZWXSGOJCDBIWSHMXULK4OQQYQCZA, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.093537637 (426h27m7.964620682s) [noise=true , address=145.79.227.48:4301 , piece_id=5IV2WAC77MWBYSQ4YRNZH2DH5BGG4XU3SVAZE5DLQJS7A2J4O3BA, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.094292854 (426h27m7.965375919s) [noise=true , address=145.79.227.28:4725 , piece_id=EHBGDM7PYS2BZ2A7HKASKRHDNE3FGBX6W3GEEX7QHKPW5ZDZCNWQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.094608194 (426h27m7.965691239s) [noise=true , address=145.79.227.7:4293 , piece_id=YTBS6YPNK5BYOZLVYF5BX43SDZNNTTZJ473HE2XQPPZ4T3K6WMXQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.095183480 (426h27m7.966266525s) [noise=true , address=45.91.116.11:4118 , piece_id=VYLGLLGMTK2JZWHGLLTMUEYFVUH3AYU7P4UEYU4EMU5N5FVZHOMQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.095350008 (426h27m7.966433053s) [noise=true , address=145.79.227.35:4481 , piece_id=4TK6V52M3EHIBHFWCZJ5SLH6LZGKSRHPFTVL5LARU5JUVO7MGCKA, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.096434292 (426h27m7.967517337s) [noise=true , address=145.79.227.14:4839 , piece_id=YNIFPV6FLHOIEEPWL34RW7CLDTXTYTG7RKGAIMFZYT6OLIB5J7TQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.096712121 (426h27m7.967795166s) [noise=true , address=45.91.116.8:4046 , piece_id=BMFCPFPRSK2LFMCRWPCLJPVETOD5QL3VTHS6POMRVETSJOB72OOA, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.097512405 (426h27m7.96859546s) [noise=true , address=145.79.227.31:4798 , piece_id=7SNRYHTEAWMXXH6KQ4UPURGY7PYJ72HOOCVA7YCIPLFG75NPNJYQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:09.969534065 (426h27m9.8406171s) [upload=374371]: Next returned nil segment Dec 23 09:56:09.969568862 (426h27m9.840651897s) [upload=374371]: Waiting for error group managing segments... Dec 23 09:56:10.267890059 (426h27m10.138973104s) [noise=true , address=77.90.31.17:4773 , piece_id=MU4UUWSSJNXVEOHAV2BWR2H4EWYFOEA2YNIQVQU6XQZIWWFDJXRQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:10.424500182 (426h27m10.295583227s) [noise=true , address=77.90.31.19:4842 , piece_id=FWDKIRAA4726YXGKXSB7PHUN7IZMNPV5ZSWKZAKZKJMQ45HDPLTQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:10.842613724 (426h27m10.713696769s) [noise=true , address=145.79.227.41:4085 , piece_id=BKPJP6TQTF7G6RMNI3K3S23WZWXSGOJCDBIWSHMXULK4OQQYQCZA, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:11.071019640 (426h27m10.942102685s) [noise=true , address=45.91.116.14:4189 , piece_id=NOUKCZQ5LQT7H62V7OBWQXNFVCOETFGAFHM6LIQSZYYX5JOCX4RQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:11.141880171 (426h27m11.012963216s) [noise=true , address=145.79.227.48:4301 , piece_id=5IV2WAC77MWBYSQ4YRNZH2DH5BGG4XU3SVAZE5DLQJS7A2J4O3BA, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:11.149149884 (426h27m11.020232919s) [noise=true , address=145.79.227.38:4978 , piece_id=UAHAMPWHYQUY7YYS7VJPGTXBCPD6KWZADF2TVZIVU7ECDBF36NWQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:11.160413952 (426h27m11.031496997s) [noise=true , address=145.79.227.46:4602 , piece_id=UUIEHKP4PL7OS7IICNOB27C5SS5ZOIJPHC35K3R4IVTEXXV5SPZA, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:12.884195178 (426h27m12.755278223s) [noise=true , address=45.91.116.8:4046 , piece_id=BMFCPFPRSK2LFMCRWPCLJPVETOD5QL3VTHS6POMRVETSJOB72OOA, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:31.365089909 (426h27m31.236172944s) [noise=true , address=145.79.227.36:4518 , piece_id=RG4EIL5AGFVP2Z3J3Z7W6NJSA4LPDTFEXSLHZW722X3RSNG7H3LQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:31.715116052 (426h27m31.586199087s) [noise=true , address=145.79.227.31:4798 , piece_id=7SNRYHTEAWMXXH6KQ4UPURGY7PYJ72HOOCVA7YCIPLFG75NPNJYQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:32.549036955 (426h27m32.420119991s) [noise=true , address=145.79.227.35:4481 , piece_id=4TK6V52M3EHIBHFWCZJ5SLH6LZGKSRHPFTVL5LARU5JUVO7MGCKA, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:36.707023852 (426h27m36.578106887s) [noise=true , address=145.79.227.28:4725 , piece_id=EHBGDM7PYS2BZ2A7HKASKRHDNE3FGBX6W3GEEX7QHKPW5ZDZCNWQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:45348->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:45348->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45348->145.79.227.28:4725: use of closed network connection Dec 23 09:56:36.707046335 (426h27m36.57812937s) [noise=true , address=145.79.227.14:4839 , piece_id=YNIFPV6FLHOIEEPWL34RW7CLDTXTYTG7RKGAIMFZYT6OLIB5J7TQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:37586->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:37586->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37586->145.79.227.14:4839: use of closed network connection Dec 23 09:56:36.707164720 (426h27m36.578247765s) [noise=true , address=145.79.227.47:4267 , piece_id=QNH4GZCBVRW6FIBGILDQOHOZSBAYN2KR42RP4G77EL2RODLXRWZQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: ecclient: upload failed (node:1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8, address:145.79.227.47:4267): protocol: write tcp 188.166.225.170:42832->145.79.227.47:4267: use of closed network connection; write tcp 188.166.225.170:42832->145.79.227.47:4267: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42832->145.79.227.47:4267: use of closed network connection Dec 23 09:56:36.707206169 (426h27m36.578289214s) [noise=true , address=145.79.227.7:4293 , piece_id=YTBS6YPNK5BYOZLVYF5BX43SDZNNTTZJ473HE2XQPPZ4T3K6WMXQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: write tcp 188.166.225.170:46776->145.79.227.7:4293: use of closed network connection; write tcp 188.166.225.170:46776->145.79.227.7:4293: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46776->145.79.227.7:4293: use of closed network connection Dec 23 09:56:36.707318803 (426h27m36.578401838s) [noise=true , address=45.91.116.11:4118 , piece_id=VYLGLLGMTK2JZWHGLLTMUEYFVUH3AYU7P4UEYU4EMU5N5FVZHOMQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: ecclient: upload failed (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, address:45.91.116.11:4118): protocol: write tcp 188.166.225.170:34940->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:34940->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34940->45.91.116.11:4118: use of closed network connection Dec 23 09:56:36.707356044 (426h27m36.578439079s) [seg_pos={0 0}, upload=374371]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Dec 23 09:56:36.958957688 (426h27m36.830040733s) [upload=374371]: Done uploading segments... Dec 23 09:56:36.958995501 (426h27m36.830078546s) [upload=374371]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.681489546s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 5.940040822s trace: https://jaeger.dev.storj.io/trace/587ca7f719cacadf FAIL (2025-12-23 10:36:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== ploading segments... Dec 23 09:56:07.712403417 (426h27m7.583486462s) [upload=374371]: Got next segment. Inline: false Dec 23 09:56:07.712445147 (426h27m7.583528182s) [upload=374371]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 09:56:07.712452090 (426h27m7.583535125s) [upload=374371]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 09:56:08.088205172 (426h27m7.959288207s) [seg_pos={0 0}, upload=374371]: Begin upload segment... Dec 23 09:56:08.088258283 (426h27m7.959341328s) [seg_pos={0 0}, upload=374371]: Done begin upload segment. Dec 23 09:56:08.088290625 (426h27m7.95937367s) [noise=true , address=145.79.227.36:4518 , piece_id=RG4EIL5AGFVP2Z3J3Z7W6NJSA4LPDTFEXSLHZW722X3RSNG7H3LQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.088349738 (426h27m7.959432783s) [noise=true , address=145.79.227.38:4978 , piece_id=UAHAMPWHYQUY7YYS7VJPGTXBCPD6KWZADF2TVZIVU7ECDBF36NWQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.089542198 (426h27m7.960625233s) [noise=true , address=45.91.116.14:4189 , piece_id=NOUKCZQ5LQT7H62V7OBWQXNFVCOETFGAFHM6LIQSZYYX5JOCX4RQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.089679400 (426h27m7.960762445s) [noise=true , address=145.79.227.46:4602 , piece_id=UUIEHKP4PL7OS7IICNOB27C5SS5ZOIJPHC35K3R4IVTEXXV5SPZA, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.090450538 (426h27m7.961533583s) [noise=true , address=77.90.31.19:4842 , piece_id=FWDKIRAA4726YXGKXSB7PHUN7IZMNPV5ZSWKZAKZKJMQ45HDPLTQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.091327698 (426h27m7.962410743s) [noise=true , address=145.79.227.47:4267 , piece_id=QNH4GZCBVRW6FIBGILDQOHOZSBAYN2KR42RP4G77EL2RODLXRWZQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.092179080 (426h27m7.963262115s) [noise=true , address=77.90.31.17:4773 , piece_id=MU4UUWSSJNXVEOHAV2BWR2H4EWYFOEA2YNIQVQU6XQZIWWFDJXRQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.092774453 (426h27m7.963857498s) [noise=true , address=145.79.227.41:4085 , piece_id=BKPJP6TQTF7G6RMNI3K3S23WZWXSGOJCDBIWSHMXULK4OQQYQCZA, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.093537637 (426h27m7.964620682s) [noise=true , address=145.79.227.48:4301 , piece_id=5IV2WAC77MWBYSQ4YRNZH2DH5BGG4XU3SVAZE5DLQJS7A2J4O3BA, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.094292854 (426h27m7.965375919s) [noise=true , address=145.79.227.28:4725 , piece_id=EHBGDM7PYS2BZ2A7HKASKRHDNE3FGBX6W3GEEX7QHKPW5ZDZCNWQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.094608194 (426h27m7.965691239s) [noise=true , address=145.79.227.7:4293 , piece_id=YTBS6YPNK5BYOZLVYF5BX43SDZNNTTZJ473HE2XQPPZ4T3K6WMXQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.095183480 (426h27m7.966266525s) [noise=true , address=45.91.116.11:4118 , piece_id=VYLGLLGMTK2JZWHGLLTMUEYFVUH3AYU7P4UEYU4EMU5N5FVZHOMQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.095350008 (426h27m7.966433053s) [noise=true , address=145.79.227.35:4481 , piece_id=4TK6V52M3EHIBHFWCZJ5SLH6LZGKSRHPFTVL5LARU5JUVO7MGCKA, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.096434292 (426h27m7.967517337s) [noise=true , address=145.79.227.14:4839 , piece_id=YNIFPV6FLHOIEEPWL34RW7CLDTXTYTG7RKGAIMFZYT6OLIB5J7TQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.096712121 (426h27m7.967795166s) [noise=true , address=45.91.116.8:4046 , piece_id=BMFCPFPRSK2LFMCRWPCLJPVETOD5QL3VTHS6POMRVETSJOB72OOA, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.097512405 (426h27m7.96859546s) [noise=true , address=145.79.227.31:4798 , piece_id=7SNRYHTEAWMXXH6KQ4UPURGY7PYJ72HOOCVA7YCIPLFG75NPNJYQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:09.969534065 (426h27m9.8406171s) [upload=374371]: Next returned nil segment Dec 23 09:56:09.969568862 (426h27m9.840651897s) [upload=374371]: Waiting for error group managing segments... Dec 23 09:56:10.267890059 (426h27m10.138973104s) [noise=true , address=77.90.31.17:4773 , piece_id=MU4UUWSSJNXVEOHAV2BWR2H4EWYFOEA2YNIQVQU6XQZIWWFDJXRQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:10.424500182 (426h27m10.295583227s) [noise=true , address=77.90.31.19:4842 , piece_id=FWDKIRAA4726YXGKXSB7PHUN7IZMNPV5ZSWKZAKZKJMQ45HDPLTQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:10.842613724 (426h27m10.713696769s) [noise=true , address=145.79.227.41:4085 , piece_id=BKPJP6TQTF7G6RMNI3K3S23WZWXSGOJCDBIWSHMXULK4OQQYQCZA, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:11.071019640 (426h27m10.942102685s) [noise=true , address=45.91.116.14:4189 , piece_id=NOUKCZQ5LQT7H62V7OBWQXNFVCOETFGAFHM6LIQSZYYX5JOCX4RQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:11.141880171 (426h27m11.012963216s) [noise=true , address=145.79.227.48:4301 , piece_id=5IV2WAC77MWBYSQ4YRNZH2DH5BGG4XU3SVAZE5DLQJS7A2J4O3BA, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:11.149149884 (426h27m11.020232919s) [noise=true , address=145.79.227.38:4978 , piece_id=UAHAMPWHYQUY7YYS7VJPGTXBCPD6KWZADF2TVZIVU7ECDBF36NWQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:11.160413952 (426h27m11.031496997s) [noise=true , address=145.79.227.46:4602 , piece_id=UUIEHKP4PL7OS7IICNOB27C5SS5ZOIJPHC35K3R4IVTEXXV5SPZA, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:12.884195178 (426h27m12.755278223s) [noise=true , address=45.91.116.8:4046 , piece_id=BMFCPFPRSK2LFMCRWPCLJPVETOD5QL3VTHS6POMRVETSJOB72OOA, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:31.365089909 (426h27m31.236172944s) [noise=true , address=145.79.227.36:4518 , piece_id=RG4EIL5AGFVP2Z3J3Z7W6NJSA4LPDTFEXSLHZW722X3RSNG7H3LQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:31.715116052 (426h27m31.586199087s) [noise=true , address=145.79.227.31:4798 , piece_id=7SNRYHTEAWMXXH6KQ4UPURGY7PYJ72HOOCVA7YCIPLFG75NPNJYQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:32.549036955 (426h27m32.420119991s) [noise=true , address=145.79.227.35:4481 , piece_id=4TK6V52M3EHIBHFWCZJ5SLH6LZGKSRHPFTVL5LARU5JUVO7MGCKA, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:36.707023852 (426h27m36.578106887s) [noise=true , address=145.79.227.28:4725 , piece_id=EHBGDM7PYS2BZ2A7HKASKRHDNE3FGBX6W3GEEX7QHKPW5ZDZCNWQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:45348->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:45348->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45348->145.79.227.28:4725: use of closed network connection Dec 23 09:56:36.707046335 (426h27m36.57812937s) [noise=true , address=145.79.227.14:4839 , piece_id=YNIFPV6FLHOIEEPWL34RW7CLDTXTYTG7RKGAIMFZYT6OLIB5J7TQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:37586->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:37586->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37586->145.79.227.14:4839: use of closed network connection Dec 23 09:56:36.707164720 (426h27m36.578247765s) [noise=true , address=145.79.227.47:4267 , piece_id=QNH4GZCBVRW6FIBGILDQOHOZSBAYN2KR42RP4G77EL2RODLXRWZQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: ecclient: upload failed (node:1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8, address:145.79.227.47:4267): protocol: write tcp 188.166.225.170:42832->145.79.227.47:4267: use of closed network connection; write tcp 188.166.225.170:42832->145.79.227.47:4267: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42832->145.79.227.47:4267: use of closed network connection Dec 23 09:56:36.707206169 (426h27m36.578289214s) [noise=true , address=145.79.227.7:4293 , piece_id=YTBS6YPNK5BYOZLVYF5BX43SDZNNTTZJ473HE2XQPPZ4T3K6WMXQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: write tcp 188.166.225.170:46776->145.79.227.7:4293: use of closed network connection; write tcp 188.166.225.170:46776->145.79.227.7:4293: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46776->145.79.227.7:4293: use of closed network connection Dec 23 09:56:36.707318803 (426h27m36.578401838s) [noise=true , address=45.91.116.11:4118 , piece_id=VYLGLLGMTK2JZWHGLLTMUEYFVUH3AYU7P4UEYU4EMU5N5FVZHOMQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: ecclient: upload failed (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, address:45.91.116.11:4118): protocol: write tcp 188.166.225.170:34940->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:34940->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34940->45.91.116.11:4118: use of closed network connection Dec 23 09:56:36.707356044 (426h27m36.578439079s) [seg_pos={0 0}, upload=374371]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Dec 23 09:56:36.958957688 (426h27m36.830040733s) [upload=374371]: Done uploading segments... Dec 23 09:56:36.958995501 (426h27m36.830078546s) [upload=374371]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.94007136s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 5.679516755s trace: https://jaeger.dev.storj.io/trace/587ca7f719cacadf FAIL (2025-12-23 10:35:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== ploading segments... Dec 23 09:56:07.712403417 (426h27m7.583486462s) [upload=374371]: Got next segment. Inline: false Dec 23 09:56:07.712445147 (426h27m7.583528182s) [upload=374371]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 09:56:07.712452090 (426h27m7.583535125s) [upload=374371]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 09:56:08.088205172 (426h27m7.959288207s) [seg_pos={0 0}, upload=374371]: Begin upload segment... Dec 23 09:56:08.088258283 (426h27m7.959341328s) [seg_pos={0 0}, upload=374371]: Done begin upload segment. Dec 23 09:56:08.088290625 (426h27m7.95937367s) [noise=true , address=145.79.227.36:4518 , piece_id=RG4EIL5AGFVP2Z3J3Z7W6NJSA4LPDTFEXSLHZW722X3RSNG7H3LQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.088349738 (426h27m7.959432783s) [noise=true , address=145.79.227.38:4978 , piece_id=UAHAMPWHYQUY7YYS7VJPGTXBCPD6KWZADF2TVZIVU7ECDBF36NWQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.089542198 (426h27m7.960625233s) [noise=true , address=45.91.116.14:4189 , piece_id=NOUKCZQ5LQT7H62V7OBWQXNFVCOETFGAFHM6LIQSZYYX5JOCX4RQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.089679400 (426h27m7.960762445s) [noise=true , address=145.79.227.46:4602 , piece_id=UUIEHKP4PL7OS7IICNOB27C5SS5ZOIJPHC35K3R4IVTEXXV5SPZA, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.090450538 (426h27m7.961533583s) [noise=true , address=77.90.31.19:4842 , piece_id=FWDKIRAA4726YXGKXSB7PHUN7IZMNPV5ZSWKZAKZKJMQ45HDPLTQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.091327698 (426h27m7.962410743s) [noise=true , address=145.79.227.47:4267 , piece_id=QNH4GZCBVRW6FIBGILDQOHOZSBAYN2KR42RP4G77EL2RODLXRWZQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.092179080 (426h27m7.963262115s) [noise=true , address=77.90.31.17:4773 , piece_id=MU4UUWSSJNXVEOHAV2BWR2H4EWYFOEA2YNIQVQU6XQZIWWFDJXRQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.092774453 (426h27m7.963857498s) [noise=true , address=145.79.227.41:4085 , piece_id=BKPJP6TQTF7G6RMNI3K3S23WZWXSGOJCDBIWSHMXULK4OQQYQCZA, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.093537637 (426h27m7.964620682s) [noise=true , address=145.79.227.48:4301 , piece_id=5IV2WAC77MWBYSQ4YRNZH2DH5BGG4XU3SVAZE5DLQJS7A2J4O3BA, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.094292854 (426h27m7.965375919s) [noise=true , address=145.79.227.28:4725 , piece_id=EHBGDM7PYS2BZ2A7HKASKRHDNE3FGBX6W3GEEX7QHKPW5ZDZCNWQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.094608194 (426h27m7.965691239s) [noise=true , address=145.79.227.7:4293 , piece_id=YTBS6YPNK5BYOZLVYF5BX43SDZNNTTZJ473HE2XQPPZ4T3K6WMXQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.095183480 (426h27m7.966266525s) [noise=true , address=45.91.116.11:4118 , piece_id=VYLGLLGMTK2JZWHGLLTMUEYFVUH3AYU7P4UEYU4EMU5N5FVZHOMQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.095350008 (426h27m7.966433053s) [noise=true , address=145.79.227.35:4481 , piece_id=4TK6V52M3EHIBHFWCZJ5SLH6LZGKSRHPFTVL5LARU5JUVO7MGCKA, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.096434292 (426h27m7.967517337s) [noise=true , address=145.79.227.14:4839 , piece_id=YNIFPV6FLHOIEEPWL34RW7CLDTXTYTG7RKGAIMFZYT6OLIB5J7TQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.096712121 (426h27m7.967795166s) [noise=true , address=45.91.116.8:4046 , piece_id=BMFCPFPRSK2LFMCRWPCLJPVETOD5QL3VTHS6POMRVETSJOB72OOA, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:08.097512405 (426h27m7.96859546s) [noise=true , address=145.79.227.31:4798 , piece_id=7SNRYHTEAWMXXH6KQ4UPURGY7PYJ72HOOCVA7YCIPLFG75NPNJYQ, seg_pos={0 0}, upload=374371]: Uploading piece... Dec 23 09:56:09.969534065 (426h27m9.8406171s) [upload=374371]: Next returned nil segment Dec 23 09:56:09.969568862 (426h27m9.840651897s) [upload=374371]: Waiting for error group managing segments... Dec 23 09:56:10.267890059 (426h27m10.138973104s) [noise=true , address=77.90.31.17:4773 , piece_id=MU4UUWSSJNXVEOHAV2BWR2H4EWYFOEA2YNIQVQU6XQZIWWFDJXRQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:10.424500182 (426h27m10.295583227s) [noise=true , address=77.90.31.19:4842 , piece_id=FWDKIRAA4726YXGKXSB7PHUN7IZMNPV5ZSWKZAKZKJMQ45HDPLTQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:10.842613724 (426h27m10.713696769s) [noise=true , address=145.79.227.41:4085 , piece_id=BKPJP6TQTF7G6RMNI3K3S23WZWXSGOJCDBIWSHMXULK4OQQYQCZA, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:11.071019640 (426h27m10.942102685s) [noise=true , address=45.91.116.14:4189 , piece_id=NOUKCZQ5LQT7H62V7OBWQXNFVCOETFGAFHM6LIQSZYYX5JOCX4RQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:11.141880171 (426h27m11.012963216s) [noise=true , address=145.79.227.48:4301 , piece_id=5IV2WAC77MWBYSQ4YRNZH2DH5BGG4XU3SVAZE5DLQJS7A2J4O3BA, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:11.149149884 (426h27m11.020232919s) [noise=true , address=145.79.227.38:4978 , piece_id=UAHAMPWHYQUY7YYS7VJPGTXBCPD6KWZADF2TVZIVU7ECDBF36NWQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:11.160413952 (426h27m11.031496997s) [noise=true , address=145.79.227.46:4602 , piece_id=UUIEHKP4PL7OS7IICNOB27C5SS5ZOIJPHC35K3R4IVTEXXV5SPZA, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:12.884195178 (426h27m12.755278223s) [noise=true , address=45.91.116.8:4046 , piece_id=BMFCPFPRSK2LFMCRWPCLJPVETOD5QL3VTHS6POMRVETSJOB72OOA, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:31.365089909 (426h27m31.236172944s) [noise=true , address=145.79.227.36:4518 , piece_id=RG4EIL5AGFVP2Z3J3Z7W6NJSA4LPDTFEXSLHZW722X3RSNG7H3LQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:31.715116052 (426h27m31.586199087s) [noise=true , address=145.79.227.31:4798 , piece_id=7SNRYHTEAWMXXH6KQ4UPURGY7PYJ72HOOCVA7YCIPLFG75NPNJYQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:32.549036955 (426h27m32.420119991s) [noise=true , address=145.79.227.35:4481 , piece_id=4TK6V52M3EHIBHFWCZJ5SLH6LZGKSRHPFTVL5LARU5JUVO7MGCKA, seg_pos={0 0}, upload=374371]: Done uploading piece. err: <nil> Dec 23 09:56:36.707023852 (426h27m36.578106887s) [noise=true , address=145.79.227.28:4725 , piece_id=EHBGDM7PYS2BZ2A7HKASKRHDNE3FGBX6W3GEEX7QHKPW5ZDZCNWQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:45348->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:45348->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45348->145.79.227.28:4725: use of closed network connection Dec 23 09:56:36.707046335 (426h27m36.57812937s) [noise=true , address=145.79.227.14:4839 , piece_id=YNIFPV6FLHOIEEPWL34RW7CLDTXTYTG7RKGAIMFZYT6OLIB5J7TQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:37586->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:37586->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37586->145.79.227.14:4839: use of closed network connection Dec 23 09:56:36.707164720 (426h27m36.578247765s) [noise=true , address=145.79.227.47:4267 , piece_id=QNH4GZCBVRW6FIBGILDQOHOZSBAYN2KR42RP4G77EL2RODLXRWZQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: ecclient: upload failed (node:1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8, address:145.79.227.47:4267): protocol: write tcp 188.166.225.170:42832->145.79.227.47:4267: use of closed network connection; write tcp 188.166.225.170:42832->145.79.227.47:4267: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42832->145.79.227.47:4267: use of closed network connection Dec 23 09:56:36.707206169 (426h27m36.578289214s) [noise=true , address=145.79.227.7:4293 , piece_id=YTBS6YPNK5BYOZLVYF5BX43SDZNNTTZJ473HE2XQPPZ4T3K6WMXQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: write tcp 188.166.225.170:46776->145.79.227.7:4293: use of closed network connection; write tcp 188.166.225.170:46776->145.79.227.7:4293: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46776->145.79.227.7:4293: use of closed network connection Dec 23 09:56:36.707318803 (426h27m36.578401838s) [noise=true , address=45.91.116.11:4118 , piece_id=VYLGLLGMTK2JZWHGLLTMUEYFVUH3AYU7P4UEYU4EMU5N5FVZHOMQ, seg_pos={0 0}, upload=374371]: Done uploading piece. err: ecclient: upload failed (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, address:45.91.116.11:4118): protocol: write tcp 188.166.225.170:34940->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:34940->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34940->45.91.116.11:4118: use of closed network connection Dec 23 09:56:36.707356044 (426h27m36.578439079s) [seg_pos={0 0}, upload=374371]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Dec 23 09:56:36.958957688 (426h27m36.830040733s) [upload=374371]: Done uploading segments... Dec 23 09:56:36.958995501 (426h27m36.830078546s) [upload=374371]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.679543226s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)