Orbiter History

Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure

Orbiter Check History

Showing 50 of 869079 reports (Page 11621 of 17382)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 12.205211008s trace: https://jaeger.dev.storj.io/trace/781235978226387b FAIL (2025-12-09 10:33:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ... Dec 9 10:09:19.659155420 (90h40m19.530238455s) [noise=true , address=77.90.31.19:4870 , piece_id=P6W2QVXDH57FK755LD67JDSC23BZPHVQHBVWZLJDC457JPG3ZQLQ, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.659573456 (90h40m19.530656491s) [noise=true , address=77.90.31.15:4703 , piece_id=3FIJGK2XHNRTDF4Q3DN3AX2P42AUR5CNUI5QPXQ256MPMCQB5QRQ, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.660082366 (90h40m19.531165401s) [noise=true , address=77.90.31.16:4760 , piece_id=LJPEX76BQ4ICDJ3I4AWHE7UE7D2WIZCVGQS6XU6ZZ2II2NBYDKHQ, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.660519568 (90h40m19.531602613s) [noise=true , address=45.91.116.9:4264 , piece_id=7UODMOFUXQZ3KYCY4RFA7OUVYXQTWLKNSWR3QIVMIT5KHEQMGT2Q, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.660971178 (90h40m19.532054223s) [noise=true , address=77.90.31.21:4882 , piece_id=CJSACAOIT5IXGJXTL4YDNRUK4H5KD6QCFE6VINQGMQJ2TT2TAU5A, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.661377291 (90h40m19.532460326s) [noise=true , address=77.90.31.13:4639 , piece_id=I2GHDHJTAHCDYP7BAG6QAHGPAFIKQZOLUPD5V7AICS4CATG3XNDA, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.661793374 (90h40m19.532876409s) [noise=true , address=45.91.116.11:4140 , piece_id=UJVOUUVBOSA4I4YZJIAPP2QGWBUV322ZSSWINU52U5RIVJ4CEZXA, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.661991682 (90h40m19.533074717s) [noise=true , address=77.90.31.14:4684 , piece_id=ARECBM73CDZDYSVWPLH2XCQC6A45SGW57TPWX5URGCAUW355COOA, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.662431088 (90h40m19.533514133s) [noise=true , address=77.90.31.12:4615 , piece_id=BLNHDHE34WZMROFSL5SNTHOITFWI6JOR7VKQBYZB77LTINEGLS5Q, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.662848232 (90h40m19.533931277s) [noise=true , address=145.79.227.13:4824 , piece_id=M6P4NFRFOU5LGE2HRGUXRUZXJ2HJYA6BWC3TW4MWUWIYYSUTR76Q, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.663338626 (90h40m19.534421671s) [noise=true , address=145.79.227.24:4587 , piece_id=EVUNUBMSMMIERRCDR67LDFSGY6S2EF6IRV6EGEX4IOFX4PCTYIDQ, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.664502282 (90h40m19.535585327s) [noise=true , address=45.91.116.12:4254 , piece_id=WOGMNSLO5FOGUSOWXRTLPZNASJB2ZMPUEKC4WRQNROTKQ74JV6KQ, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.666711839 (90h40m19.537794884s) [noise=true , address=77.90.31.10:4529 , piece_id=YIT3CCLUOLSQ3NGOUVF7PYVV6FIDT7WKTRQVU4VTU4LRMPYL4H3A, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.669766776 (90h40m19.540849821s) [noise=true , address=77.90.31.17:4783 , piece_id=LSTC3TY566OQBGL6ZXBLL5IFNPORZRWOEGFKGT3EXYOB3IPWI6JQ, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.677090573 (90h40m19.548173618s) [noise=true , address=45.91.116.13:4173 , piece_id=7PIVANLRS7SL6ERBMUC4LT4P3C3JRHRY6Z4I7336EVMXC3IHFVDQ, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:21.186498404 (90h40m21.057581499s) [upload=79096]: Next returned nil segment Dec 9 10:09:21.186518313 (90h40m21.057601348s) [upload=79096]: Waiting for error group managing segments... Dec 9 10:09:21.574704328 (90h40m21.445787374s) [noise=true , address=77.90.31.18:4829 , piece_id=CDRLGWU6VF6TU6E4E5B6R7NOF35WOQOA2TSDPASZP424U23QORNA, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:21.587662915 (90h40m21.45874595s) [noise=true , address=77.90.31.10:4529 , piece_id=YIT3CCLUOLSQ3NGOUVF7PYVV6FIDT7WKTRQVU4VTU4LRMPYL4H3A, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:21.624695792 (90h40m21.495778827s) [noise=true , address=77.90.31.15:4703 , piece_id=3FIJGK2XHNRTDF4Q3DN3AX2P42AUR5CNUI5QPXQ256MPMCQB5QRQ, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:21.659658729 (90h40m21.530741774s) [noise=true , address=77.90.31.12:4615 , piece_id=BLNHDHE34WZMROFSL5SNTHOITFWI6JOR7VKQBYZB77LTINEGLS5Q, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:21.667474853 (90h40m21.538557898s) [noise=true , address=145.79.227.24:4587 , piece_id=EVUNUBMSMMIERRCDR67LDFSGY6S2EF6IRV6EGEX4IOFX4PCTYIDQ, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:21.669766096 (90h40m21.540849131s) [noise=true , address=77.90.31.14:4684 , piece_id=ARECBM73CDZDYSVWPLH2XCQC6A45SGW57TPWX5URGCAUW355COOA, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:21.670505664 (90h40m21.541588709s) [noise=true , address=77.90.31.17:4783 , piece_id=LSTC3TY566OQBGL6ZXBLL5IFNPORZRWOEGFKGT3EXYOB3IPWI6JQ, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:21.680637627 (90h40m21.551720662s) [noise=true , address=77.90.31.13:4639 , piece_id=I2GHDHJTAHCDYP7BAG6QAHGPAFIKQZOLUPD5V7AICS4CATG3XNDA, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:21.685855022 (90h40m21.556938057s) [noise=true , address=77.90.31.21:4882 , piece_id=CJSACAOIT5IXGJXTL4YDNRUK4H5KD6QCFE6VINQGMQJ2TT2TAU5A, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:21.741439516 (90h40m21.612522561s) [noise=true , address=77.90.31.19:4870 , piece_id=P6W2QVXDH57FK755LD67JDSC23BZPHVQHBVWZLJDC457JPG3ZQLQ, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:22.306045351 (90h40m22.177128396s) [noise=true , address=77.90.31.16:4760 , piece_id=LJPEX76BQ4ICDJ3I4AWHE7UE7D2WIZCVGQS6XU6ZZ2II2NBYDKHQ, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:22.498367742 (90h40m22.369450787s) [noise=true , address=145.79.227.13:4824 , piece_id=M6P4NFRFOU5LGE2HRGUXRUZXJ2HJYA6BWC3TW4MWUWIYYSUTR76Q, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:10:18.740403422 (90h41m18.611486467s) [noise=true , address=45.91.116.13:4173 , piece_id=7PIVANLRS7SL6ERBMUC4LT4P3C3JRHRY6Z4I7336EVMXC3IHFVDQ, seg_pos={0 0}, upload=79096]: Done uploading piece. err: ecclient: upload failed (node:1J3HZY416Dm9F5zRXtoDdt8s8tGoU6GpkwS14CXJUzd3zpcYa1, address:45.91.116.13:4173): protocol: write tcp 188.166.225.170:53476->45.91.116.13:4173: use of closed network connection; write tcp 188.166.225.170:53476->45.91.116.13:4173: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53476->45.91.116.13:4173: use of closed network connection Dec 9 10:10:18.740531145 (90h41m18.61161419s) [noise=true , address=64.49.13.169:4710 , piece_id=XEKNBKQ4WTECDQD6YYP5RUU2KOSFPRGKXRWDN5THYLE4HXO65BEA, seg_pos={0 0}, upload=79096]: Done uploading piece. err: ecclient: upload failed (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf, address:64.49.13.169:4710): protocol: write tcp 188.166.225.170:37916->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:37916->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37916->64.49.13.169:4710: use of closed network connection Dec 9 10:10:18.740657375 (90h41m18.61174042s) [noise=true , address=45.91.116.12:4254 , piece_id=WOGMNSLO5FOGUSOWXRTLPZNASJB2ZMPUEKC4WRQNROTKQ74JV6KQ, seg_pos={0 0}, upload=79096]: Done uploading piece. err: ecclient: upload failed (node:12wAKmaLHwjBKN8UozGFjGvNMYG57fCxTH8wuDZDRnMdyGBRsTv, address:45.91.116.12:4254): protocol: write tcp 188.166.225.170:35550->45.91.116.12:4254: use of closed network connection; write tcp 188.166.225.170:35550->45.91.116.12:4254: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35550->45.91.116.12:4254: use of closed network connection Dec 9 10:10:18.740778986 (90h41m18.611862021s) [noise=true , address=45.91.116.5:4324 , piece_id=XUK3P6OWTGEJMVLR5B3ZR667WEMCEYIAIVW3PTE6ZWTREKSNBTBA, seg_pos={0 0}, upload=79096]: Done uploading piece. err: ecclient: upload failed (node:12btBm53zb1MCTy6b7CUeujUruMR9nxBoRgqhHmG5ffPf7rL4F, address:45.91.116.5:4324): protocol: write tcp 188.166.225.170:41676->45.91.116.5:4324: use of closed network connection; write tcp 188.166.225.170:41676->45.91.116.5:4324: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41676->45.91.116.5:4324: use of closed network connection Dec 9 10:10:18.740879748 (90h41m18.611962793s) [noise=true , address=45.91.116.11:4140 , piece_id=UJVOUUVBOSA4I4YZJIAPP2QGWBUV322ZSSWINU52U5RIVJ4CEZXA, seg_pos={0 0}, upload=79096]: Done uploading piece. err: ecclient: upload failed (node:14DNa2ysTGtfy3Zz72Dq7S3NSGXqnTHZRkuXorDvuuBw1abwMT, address:45.91.116.11:4140): protocol: write tcp 188.166.225.170:34050->45.91.116.11:4140: use of closed network connection; write tcp 188.166.225.170:34050->45.91.116.11:4140: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34050->45.91.116.11:4140: use of closed network connection Dec 9 10:10:18.740962677 (90h41m18.612045712s) [noise=true , address=45.91.116.9:4264 , piece_id=7UODMOFUXQZ3KYCY4RFA7OUVYXQTWLKNSWR3QIVMIT5KHEQMGT2Q, seg_pos={0 0}, upload=79096]: Done uploading piece. err: ecclient: upload failed (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, address:45.91.116.9:4264): protocol: write tcp 188.166.225.170:50170->45.91.116.9:4264: use of closed network connection; write tcp 188.166.225.170:50170->45.91.116.9:4264: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50170->45.91.116.9:4264: use of closed network connection Dec 9 10:10:18.740989428 (90h41m18.612072463s) [seg_pos={0 0}, upload=79096]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Dec 9 10:10:18.978016295 (90h41m18.84909934s) [upload=79096]: Done uploading segments... Dec 9 10:10:18.978064968 (90h41m18.849148013s) [upload=79096]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 12.205241065s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [........X......X..................X.....................X...] duration: 5.863828819s trace: https://jaeger.dev.storj.io/trace/5231fff0fb8dd178 FAIL (2025-12-09 10:33:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.863854359s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [X........X......X..................X.....................X..] duration: 5.500060055s trace: https://jaeger.dev.storj.io/trace/5231fff0fb8dd178 FAIL (2025-12-09 10:33:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.500088599s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.........................X..X..............X................] duration: 3.396065925s trace: https://jaeger.dev.storj.io/trace/9e23c2a25a964f2 FAIL (2025-12-09 10:33:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.39609408s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 2.119917746s trace: https://jaeger.dev.storj.io/trace/6cb2657ad006bedf FAIL (2025-12-09 10:33:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 09:57:44.389846478 (90h28m44.260929523s) [upload=78917]: Starting upload Dec 9 09:57:44.390051578 (90h28m44.261134613s) [upload=78917]: Done starting upload Dec 9 09:57:44.390125740 (90h28m44.261208775s) [upload=78917]: Uploading segments... Dec 9 09:57:44.390224027 (90h28m44.261307072s) [upload=78917]: Got next segment. Inline: true Dec 9 09:57:44.390230800 (90h28m44.261313835s) [upload=78917]: Waiting for error group managing segments... Dec 9 09:57:44.390312034 (90h28m44.261395069s) [upload=78917]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 09:57:44.390318787 (90h28m44.261401822s) [upload=78917]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 9 09:57:44.390323356 (90h28m44.261406391s) [upload=78917]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 09:58:21.934924620 (90h29m21.806007665s) [upload=78917]: Done uploading segments... Dec 9 09:58:21.934974756 (90h29m21.806057801s) [upload=78917]: Upload finished. err: metaclient: context deadline exceeded Duration:* 2.119979104s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 5.755345136s trace: https://jaeger.dev.storj.io/trace/47335bbbc9460f72 FAIL (2025-12-09 10:33:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 10:27:57.242720755 (90h58m57.1138038s) [upload=79359]: Starting upload Dec 9 10:27:57.242779226 (90h58m57.113862261s) [upload=79359]: Done starting upload Dec 9 10:27:57.242849329 (90h58m57.113932365s) [upload=79359]: Uploading segments... Dec 9 10:27:57.242961895 (90h58m57.11404493s) [upload=79359]: Got next segment. Inline: false Dec 9 10:27:57.242992913 (90h58m57.114075948s) [upload=79359]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 10:27:57.242997352 (90h58m57.114080397s) [upload=79359]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 10:27:57.490478536 (90h58m57.361561581s) [seg_pos={0 0}, upload=79359]: Begin upload segment... Dec 9 10:27:57.490533320 (90h58m57.361616365s) [seg_pos={0 0}, upload=79359]: Done begin upload segment. Dec 9 10:27:57.490560913 (90h58m57.361643948s) [noise=true , address=145.79.227.22:4511 , piece_id=R5QAVASG7T7UEXDTFXZOK5BPGO7LJARFI7A77MASBGBQJ4DBP4MA, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.490634854 (90h58m57.361717889s) [noise=true , address=145.79.227.40:4051 , piece_id=GBINALPVGCPMYGIYXJOXKZFKQK7ERXVAWTEZ6XMPEQMVTRMRYRWQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.491225929 (90h58m57.362308994s) [noise=true , address=145.79.227.38:4977 , piece_id=5TTHLFLJJXTFEHA2CSV4DCCOMKT5OH7TQPR63TTB57S7O6L4Y73A, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.491642873 (90h58m57.362725918s) [noise=true , address=145.79.227.39:4014 , piece_id=EW67DYNN54WK4Y7YP7HAZUN227LYDM3LAGCFH7OBET7YIEAUTF3A, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.492140761 (90h58m57.363223796s) [noise=true , address=45.91.116.8:4047 , piece_id=XOO3P7YG5DW5SYP4OVSAGLKCZ6W3KSVXIUK24GDYOXX2H3HMS5GQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.492602540 (90h58m57.363685585s) [noise=true , address=145.79.227.20:4439 , piece_id=JE2X5VYQRHXJDWQFUT2KWCE6UWMD6VLUEHMBZSLL3O6FYMGEY7UQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.493057557 (90h58m57.364140603s) [noise=true , address=145.79.227.29:4941 , piece_id=RJUQZUTVF3KJEGZ4MDOP5RR4DEQH5X4OHHXAK3W64CGQOUY4HZWA, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.493071273 (90h58m57.364154318s) [noise=true , address=145.79.227.42:4123 , piece_id=6JHXIPA352SVNLEWKKPMANX5MQRHXFURZR753T74MMFTPPK4LHJA, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.493431789 (90h58m57.364514854s) [noise=true , address=145.79.227.31:4799 , piece_id=AN4NPL2MIY3NEKKR4TZ7STWCUIZVY6MUQ36D5PJROGD2UG722P2A, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.493540356 (90h58m57.364623401s) [noise=true , address=145.79.227.24:4583 , piece_id=XSC6J57R4UDC2AL3I7ZK73FLNQG7EIIHIJGR34WAK3LT3U6UZJAQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.493892386 (90h58m57.364975431s) [noise=true , address=145.79.227.32:4835 , piece_id=HVGEKOXP5H2HHK3IG2VTIGXX6UTQKWBX63VKL5VQGQMFIRXQV6BQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.494083991 (90h58m57.365167027s) [noise=true , address=145.79.227.23:4547 , piece_id=SQG4BHO3ZZUORDL6YOPFQNGCKVF7TDJEONIEJWNYGCCFXDWX6KIQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.494255959 (90h58m57.365339004s) [noise=true , address=145.79.227.25:4618 , piece_id=UMDOX46LAA6YKCG74OXMIUAVVFG7MNE3XHL2PBERUOASXEBYMGSQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.494606567 (90h58m57.365689602s) [noise=true , address=145.79.227.27:4691 , piece_id=7GAKOVJNXFEN5CDKVGK46EIZ6OU4JI247M6Q35WVF4JOWQN72JPA, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.494965920 (90h58m57.366048956s) [noise=true , address=145.79.227.34:4907 , piece_id=PLUYUJWUQXB6BOP6H6YKCRQ4NNOHJ3M4JHNB7FVKQHO75OJU5HYQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.495345513 (90h58m57.366428558s) [noise=true , address=145.79.227.42:4121 , piece_id=ERCQT36FQSO22HNFY5QZOJJMERRASCVHJW5G4OKCEJB32UNYUQAQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:59.193560364 (90h58m59.064643409s) [upload=79359]: Next returned nil segment Dec 9 10:27:59.193597184 (90h58m59.064680219s) [upload=79359]: Waiting for error group managing segments... Dec 9 10:27:59.477920084 (90h58m59.349003119s) [noise=true , address=145.79.227.42:4121 , piece_id=ERCQT36FQSO22HNFY5QZOJJMERRASCVHJW5G4OKCEJB32UNYUQAQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.480823943 (90h58m59.351906988s) [noise=true , address=145.79.227.31:4799 , piece_id=AN4NPL2MIY3NEKKR4TZ7STWCUIZVY6MUQ36D5PJROGD2UG722P2A, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.487502670 (90h58m59.358585715s) [noise=true , address=145.79.227.27:4691 , piece_id=7GAKOVJNXFEN5CDKVGK46EIZ6OU4JI247M6Q35WVF4JOWQN72JPA, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.491620402 (90h58m59.362703437s) [noise=true , address=145.79.227.29:4941 , piece_id=RJUQZUTVF3KJEGZ4MDOP5RR4DEQH5X4OHHXAK3W64CGQOUY4HZWA, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.491964066 (90h58m59.363047101s) [noise=true , address=145.79.227.42:4123 , piece_id=6JHXIPA352SVNLEWKKPMANX5MQRHXFURZR753T74MMFTPPK4LHJA, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.500675514 (90h58m59.371758559s) [noise=true , address=145.79.227.25:4618 , piece_id=UMDOX46LAA6YKCG74OXMIUAVVFG7MNE3XHL2PBERUOASXEBYMGSQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.501834542 (90h58m59.372917577s) [noise=true , address=145.79.227.39:4014 , piece_id=EW67DYNN54WK4Y7YP7HAZUN227LYDM3LAGCFH7OBET7YIEAUTF3A, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.530610947 (90h58m59.401693992s) [noise=true , address=145.79.227.20:4439 , piece_id=JE2X5VYQRHXJDWQFUT2KWCE6UWMD6VLUEHMBZSLL3O6FYMGEY7UQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.554079847 (90h58m59.425162892s) [noise=true , address=145.79.227.22:4511 , piece_id=R5QAVASG7T7UEXDTFXZOK5BPGO7LJARFI7A77MASBGBQJ4DBP4MA, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.558634540 (90h58m59.429717585s) [noise=true , address=145.79.227.24:4583 , piece_id=XSC6J57R4UDC2AL3I7ZK73FLNQG7EIIHIJGR34WAK3LT3U6UZJAQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.571829987 (90h58m59.442913023s) [noise=true , address=145.79.227.32:4835 , piece_id=HVGEKOXP5H2HHK3IG2VTIGXX6UTQKWBX63VKL5VQGQMFIRXQV6BQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.574149323 (90h58m59.445232368s) [noise=true , address=145.79.227.34:4907 , piece_id=PLUYUJWUQXB6BOP6H6YKCRQ4NNOHJ3M4JHNB7FVKQHO75OJU5HYQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:28:26.550754779 (90h59m26.421837824s) [noise=true , address=45.91.116.8:4047 , piece_id=XOO3P7YG5DW5SYP4OVSAGLKCZ6W3KSVXIUK24GDYOXX2H3HMS5GQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: write tcp 188.166.225.170:58652->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:58652->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58652->45.91.116.8:4047: use of closed network connection Dec 9 10:28:26.550891860 (90h59m26.421974905s) [noise=true , address=145.79.227.38:4977 , piece_id=5TTHLFLJJXTFEHA2CSV4DCCOMKT5OH7TQPR63TTB57S7O6L4Y73A, seg_pos={0 0}, upload=79359]: Done uploading piece. err: ecclient: upload failed (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, address:145.79.227.38:4977): protocol: write tcp 188.166.225.170:48912->145.79.227.38:4977: use of closed network connection; write tcp 188.166.225.170:48912->145.79.227.38:4977: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48912->145.79.227.38:4977: use of closed network connection Dec 9 10:28:26.551013401 (90h59m26.422096446s) [noise=true , address=145.79.227.40:4051 , piece_id=GBINALPVGCPMYGIYXJOXKZFKQK7ERXVAWTEZ6XMPEQMVTRMRYRWQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: write tcp 188.166.225.170:49228->145.79.227.40:4051: use of closed network connection; write tcp 188.166.225.170:49228->145.79.227.40:4051: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49228->145.79.227.40:4051: use of closed network connection Dec 9 10:28:26.551092152 (90h59m26.422175187s) [noise=true , address=145.79.227.23:4547 , piece_id=SQG4BHO3ZZUORDL6YOPFQNGCKVF7TDJEONIEJWNYGCCFXDWX6KIQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: write tcp 188.166.225.170:60386->145.79.227.23:4547: use of closed network connection; write tcp 188.166.225.170:60386->145.79.227.23:4547: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60386->145.79.227.23:4547: use of closed network connection Dec 9 10:28:26.551125365 (90h59m26.42220841s) [seg_pos={0 0}, upload=79359]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 10:28:26.791381862 (90h59m26.662464897s) [upload=79359]: Done uploading segments... Dec 9 10:28:26.791420656 (90h59m26.662503691s) [upload=79359]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.755380865s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 12.442511743s trace: https://jaeger.dev.storj.io/trace/781235978226387b FAIL (2025-12-09 10:33:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ... Dec 9 10:09:19.659155420 (90h40m19.530238455s) [noise=true , address=77.90.31.19:4870 , piece_id=P6W2QVXDH57FK755LD67JDSC23BZPHVQHBVWZLJDC457JPG3ZQLQ, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.659573456 (90h40m19.530656491s) [noise=true , address=77.90.31.15:4703 , piece_id=3FIJGK2XHNRTDF4Q3DN3AX2P42AUR5CNUI5QPXQ256MPMCQB5QRQ, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.660082366 (90h40m19.531165401s) [noise=true , address=77.90.31.16:4760 , piece_id=LJPEX76BQ4ICDJ3I4AWHE7UE7D2WIZCVGQS6XU6ZZ2II2NBYDKHQ, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.660519568 (90h40m19.531602613s) [noise=true , address=45.91.116.9:4264 , piece_id=7UODMOFUXQZ3KYCY4RFA7OUVYXQTWLKNSWR3QIVMIT5KHEQMGT2Q, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.660971178 (90h40m19.532054223s) [noise=true , address=77.90.31.21:4882 , piece_id=CJSACAOIT5IXGJXTL4YDNRUK4H5KD6QCFE6VINQGMQJ2TT2TAU5A, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.661377291 (90h40m19.532460326s) [noise=true , address=77.90.31.13:4639 , piece_id=I2GHDHJTAHCDYP7BAG6QAHGPAFIKQZOLUPD5V7AICS4CATG3XNDA, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.661793374 (90h40m19.532876409s) [noise=true , address=45.91.116.11:4140 , piece_id=UJVOUUVBOSA4I4YZJIAPP2QGWBUV322ZSSWINU52U5RIVJ4CEZXA, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.661991682 (90h40m19.533074717s) [noise=true , address=77.90.31.14:4684 , piece_id=ARECBM73CDZDYSVWPLH2XCQC6A45SGW57TPWX5URGCAUW355COOA, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.662431088 (90h40m19.533514133s) [noise=true , address=77.90.31.12:4615 , piece_id=BLNHDHE34WZMROFSL5SNTHOITFWI6JOR7VKQBYZB77LTINEGLS5Q, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.662848232 (90h40m19.533931277s) [noise=true , address=145.79.227.13:4824 , piece_id=M6P4NFRFOU5LGE2HRGUXRUZXJ2HJYA6BWC3TW4MWUWIYYSUTR76Q, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.663338626 (90h40m19.534421671s) [noise=true , address=145.79.227.24:4587 , piece_id=EVUNUBMSMMIERRCDR67LDFSGY6S2EF6IRV6EGEX4IOFX4PCTYIDQ, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.664502282 (90h40m19.535585327s) [noise=true , address=45.91.116.12:4254 , piece_id=WOGMNSLO5FOGUSOWXRTLPZNASJB2ZMPUEKC4WRQNROTKQ74JV6KQ, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.666711839 (90h40m19.537794884s) [noise=true , address=77.90.31.10:4529 , piece_id=YIT3CCLUOLSQ3NGOUVF7PYVV6FIDT7WKTRQVU4VTU4LRMPYL4H3A, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.669766776 (90h40m19.540849821s) [noise=true , address=77.90.31.17:4783 , piece_id=LSTC3TY566OQBGL6ZXBLL5IFNPORZRWOEGFKGT3EXYOB3IPWI6JQ, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.677090573 (90h40m19.548173618s) [noise=true , address=45.91.116.13:4173 , piece_id=7PIVANLRS7SL6ERBMUC4LT4P3C3JRHRY6Z4I7336EVMXC3IHFVDQ, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:21.186498404 (90h40m21.057581499s) [upload=79096]: Next returned nil segment Dec 9 10:09:21.186518313 (90h40m21.057601348s) [upload=79096]: Waiting for error group managing segments... Dec 9 10:09:21.574704328 (90h40m21.445787374s) [noise=true , address=77.90.31.18:4829 , piece_id=CDRLGWU6VF6TU6E4E5B6R7NOF35WOQOA2TSDPASZP424U23QORNA, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:21.587662915 (90h40m21.45874595s) [noise=true , address=77.90.31.10:4529 , piece_id=YIT3CCLUOLSQ3NGOUVF7PYVV6FIDT7WKTRQVU4VTU4LRMPYL4H3A, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:21.624695792 (90h40m21.495778827s) [noise=true , address=77.90.31.15:4703 , piece_id=3FIJGK2XHNRTDF4Q3DN3AX2P42AUR5CNUI5QPXQ256MPMCQB5QRQ, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:21.659658729 (90h40m21.530741774s) [noise=true , address=77.90.31.12:4615 , piece_id=BLNHDHE34WZMROFSL5SNTHOITFWI6JOR7VKQBYZB77LTINEGLS5Q, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:21.667474853 (90h40m21.538557898s) [noise=true , address=145.79.227.24:4587 , piece_id=EVUNUBMSMMIERRCDR67LDFSGY6S2EF6IRV6EGEX4IOFX4PCTYIDQ, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:21.669766096 (90h40m21.540849131s) [noise=true , address=77.90.31.14:4684 , piece_id=ARECBM73CDZDYSVWPLH2XCQC6A45SGW57TPWX5URGCAUW355COOA, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:21.670505664 (90h40m21.541588709s) [noise=true , address=77.90.31.17:4783 , piece_id=LSTC3TY566OQBGL6ZXBLL5IFNPORZRWOEGFKGT3EXYOB3IPWI6JQ, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:21.680637627 (90h40m21.551720662s) [noise=true , address=77.90.31.13:4639 , piece_id=I2GHDHJTAHCDYP7BAG6QAHGPAFIKQZOLUPD5V7AICS4CATG3XNDA, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:21.685855022 (90h40m21.556938057s) [noise=true , address=77.90.31.21:4882 , piece_id=CJSACAOIT5IXGJXTL4YDNRUK4H5KD6QCFE6VINQGMQJ2TT2TAU5A, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:21.741439516 (90h40m21.612522561s) [noise=true , address=77.90.31.19:4870 , piece_id=P6W2QVXDH57FK755LD67JDSC23BZPHVQHBVWZLJDC457JPG3ZQLQ, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:22.306045351 (90h40m22.177128396s) [noise=true , address=77.90.31.16:4760 , piece_id=LJPEX76BQ4ICDJ3I4AWHE7UE7D2WIZCVGQS6XU6ZZ2II2NBYDKHQ, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:22.498367742 (90h40m22.369450787s) [noise=true , address=145.79.227.13:4824 , piece_id=M6P4NFRFOU5LGE2HRGUXRUZXJ2HJYA6BWC3TW4MWUWIYYSUTR76Q, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:10:18.740403422 (90h41m18.611486467s) [noise=true , address=45.91.116.13:4173 , piece_id=7PIVANLRS7SL6ERBMUC4LT4P3C3JRHRY6Z4I7336EVMXC3IHFVDQ, seg_pos={0 0}, upload=79096]: Done uploading piece. err: ecclient: upload failed (node:1J3HZY416Dm9F5zRXtoDdt8s8tGoU6GpkwS14CXJUzd3zpcYa1, address:45.91.116.13:4173): protocol: write tcp 188.166.225.170:53476->45.91.116.13:4173: use of closed network connection; write tcp 188.166.225.170:53476->45.91.116.13:4173: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53476->45.91.116.13:4173: use of closed network connection Dec 9 10:10:18.740531145 (90h41m18.61161419s) [noise=true , address=64.49.13.169:4710 , piece_id=XEKNBKQ4WTECDQD6YYP5RUU2KOSFPRGKXRWDN5THYLE4HXO65BEA, seg_pos={0 0}, upload=79096]: Done uploading piece. err: ecclient: upload failed (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf, address:64.49.13.169:4710): protocol: write tcp 188.166.225.170:37916->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:37916->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37916->64.49.13.169:4710: use of closed network connection Dec 9 10:10:18.740657375 (90h41m18.61174042s) [noise=true , address=45.91.116.12:4254 , piece_id=WOGMNSLO5FOGUSOWXRTLPZNASJB2ZMPUEKC4WRQNROTKQ74JV6KQ, seg_pos={0 0}, upload=79096]: Done uploading piece. err: ecclient: upload failed (node:12wAKmaLHwjBKN8UozGFjGvNMYG57fCxTH8wuDZDRnMdyGBRsTv, address:45.91.116.12:4254): protocol: write tcp 188.166.225.170:35550->45.91.116.12:4254: use of closed network connection; write tcp 188.166.225.170:35550->45.91.116.12:4254: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35550->45.91.116.12:4254: use of closed network connection Dec 9 10:10:18.740778986 (90h41m18.611862021s) [noise=true , address=45.91.116.5:4324 , piece_id=XUK3P6OWTGEJMVLR5B3ZR667WEMCEYIAIVW3PTE6ZWTREKSNBTBA, seg_pos={0 0}, upload=79096]: Done uploading piece. err: ecclient: upload failed (node:12btBm53zb1MCTy6b7CUeujUruMR9nxBoRgqhHmG5ffPf7rL4F, address:45.91.116.5:4324): protocol: write tcp 188.166.225.170:41676->45.91.116.5:4324: use of closed network connection; write tcp 188.166.225.170:41676->45.91.116.5:4324: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41676->45.91.116.5:4324: use of closed network connection Dec 9 10:10:18.740879748 (90h41m18.611962793s) [noise=true , address=45.91.116.11:4140 , piece_id=UJVOUUVBOSA4I4YZJIAPP2QGWBUV322ZSSWINU52U5RIVJ4CEZXA, seg_pos={0 0}, upload=79096]: Done uploading piece. err: ecclient: upload failed (node:14DNa2ysTGtfy3Zz72Dq7S3NSGXqnTHZRkuXorDvuuBw1abwMT, address:45.91.116.11:4140): protocol: write tcp 188.166.225.170:34050->45.91.116.11:4140: use of closed network connection; write tcp 188.166.225.170:34050->45.91.116.11:4140: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34050->45.91.116.11:4140: use of closed network connection Dec 9 10:10:18.740962677 (90h41m18.612045712s) [noise=true , address=45.91.116.9:4264 , piece_id=7UODMOFUXQZ3KYCY4RFA7OUVYXQTWLKNSWR3QIVMIT5KHEQMGT2Q, seg_pos={0 0}, upload=79096]: Done uploading piece. err: ecclient: upload failed (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, address:45.91.116.9:4264): protocol: write tcp 188.166.225.170:50170->45.91.116.9:4264: use of closed network connection; write tcp 188.166.225.170:50170->45.91.116.9:4264: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50170->45.91.116.9:4264: use of closed network connection Dec 9 10:10:18.740989428 (90h41m18.612072463s) [seg_pos={0 0}, upload=79096]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Dec 9 10:10:18.978016295 (90h41m18.84909934s) [upload=79096]: Done uploading segments... Dec 9 10:10:18.978064968 (90h41m18.849148013s) [upload=79096]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 12.442534667s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 1.990278998s trace: https://jaeger.dev.storj.io/trace/6cb2657ad006bedf FAIL (2025-12-09 10:32:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 09:57:44.389846478 (90h28m44.260929523s) [upload=78917]: Starting upload Dec 9 09:57:44.390051578 (90h28m44.261134613s) [upload=78917]: Done starting upload Dec 9 09:57:44.390125740 (90h28m44.261208775s) [upload=78917]: Uploading segments... Dec 9 09:57:44.390224027 (90h28m44.261307072s) [upload=78917]: Got next segment. Inline: true Dec 9 09:57:44.390230800 (90h28m44.261313835s) [upload=78917]: Waiting for error group managing segments... Dec 9 09:57:44.390312034 (90h28m44.261395069s) [upload=78917]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 09:57:44.390318787 (90h28m44.261401822s) [upload=78917]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 9 09:57:44.390323356 (90h28m44.261406391s) [upload=78917]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 09:58:21.934924620 (90h29m21.806007665s) [upload=78917]: Done uploading segments... Dec 9 09:58:21.934974756 (90h29m21.806057801s) [upload=78917]: Upload finished. err: metaclient: context deadline exceeded Duration:* 1.990318263s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 11.453122808s trace: https://jaeger.dev.storj.io/trace/781235978226387b FAIL (2025-12-09 10:32:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ... Dec 9 10:09:19.659155420 (90h40m19.530238455s) [noise=true , address=77.90.31.19:4870 , piece_id=P6W2QVXDH57FK755LD67JDSC23BZPHVQHBVWZLJDC457JPG3ZQLQ, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.659573456 (90h40m19.530656491s) [noise=true , address=77.90.31.15:4703 , piece_id=3FIJGK2XHNRTDF4Q3DN3AX2P42AUR5CNUI5QPXQ256MPMCQB5QRQ, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.660082366 (90h40m19.531165401s) [noise=true , address=77.90.31.16:4760 , piece_id=LJPEX76BQ4ICDJ3I4AWHE7UE7D2WIZCVGQS6XU6ZZ2II2NBYDKHQ, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.660519568 (90h40m19.531602613s) [noise=true , address=45.91.116.9:4264 , piece_id=7UODMOFUXQZ3KYCY4RFA7OUVYXQTWLKNSWR3QIVMIT5KHEQMGT2Q, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.660971178 (90h40m19.532054223s) [noise=true , address=77.90.31.21:4882 , piece_id=CJSACAOIT5IXGJXTL4YDNRUK4H5KD6QCFE6VINQGMQJ2TT2TAU5A, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.661377291 (90h40m19.532460326s) [noise=true , address=77.90.31.13:4639 , piece_id=I2GHDHJTAHCDYP7BAG6QAHGPAFIKQZOLUPD5V7AICS4CATG3XNDA, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.661793374 (90h40m19.532876409s) [noise=true , address=45.91.116.11:4140 , piece_id=UJVOUUVBOSA4I4YZJIAPP2QGWBUV322ZSSWINU52U5RIVJ4CEZXA, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.661991682 (90h40m19.533074717s) [noise=true , address=77.90.31.14:4684 , piece_id=ARECBM73CDZDYSVWPLH2XCQC6A45SGW57TPWX5URGCAUW355COOA, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.662431088 (90h40m19.533514133s) [noise=true , address=77.90.31.12:4615 , piece_id=BLNHDHE34WZMROFSL5SNTHOITFWI6JOR7VKQBYZB77LTINEGLS5Q, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.662848232 (90h40m19.533931277s) [noise=true , address=145.79.227.13:4824 , piece_id=M6P4NFRFOU5LGE2HRGUXRUZXJ2HJYA6BWC3TW4MWUWIYYSUTR76Q, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.663338626 (90h40m19.534421671s) [noise=true , address=145.79.227.24:4587 , piece_id=EVUNUBMSMMIERRCDR67LDFSGY6S2EF6IRV6EGEX4IOFX4PCTYIDQ, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.664502282 (90h40m19.535585327s) [noise=true , address=45.91.116.12:4254 , piece_id=WOGMNSLO5FOGUSOWXRTLPZNASJB2ZMPUEKC4WRQNROTKQ74JV6KQ, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.666711839 (90h40m19.537794884s) [noise=true , address=77.90.31.10:4529 , piece_id=YIT3CCLUOLSQ3NGOUVF7PYVV6FIDT7WKTRQVU4VTU4LRMPYL4H3A, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.669766776 (90h40m19.540849821s) [noise=true , address=77.90.31.17:4783 , piece_id=LSTC3TY566OQBGL6ZXBLL5IFNPORZRWOEGFKGT3EXYOB3IPWI6JQ, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.677090573 (90h40m19.548173618s) [noise=true , address=45.91.116.13:4173 , piece_id=7PIVANLRS7SL6ERBMUC4LT4P3C3JRHRY6Z4I7336EVMXC3IHFVDQ, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:21.186498404 (90h40m21.057581499s) [upload=79096]: Next returned nil segment Dec 9 10:09:21.186518313 (90h40m21.057601348s) [upload=79096]: Waiting for error group managing segments... Dec 9 10:09:21.574704328 (90h40m21.445787374s) [noise=true , address=77.90.31.18:4829 , piece_id=CDRLGWU6VF6TU6E4E5B6R7NOF35WOQOA2TSDPASZP424U23QORNA, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:21.587662915 (90h40m21.45874595s) [noise=true , address=77.90.31.10:4529 , piece_id=YIT3CCLUOLSQ3NGOUVF7PYVV6FIDT7WKTRQVU4VTU4LRMPYL4H3A, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:21.624695792 (90h40m21.495778827s) [noise=true , address=77.90.31.15:4703 , piece_id=3FIJGK2XHNRTDF4Q3DN3AX2P42AUR5CNUI5QPXQ256MPMCQB5QRQ, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:21.659658729 (90h40m21.530741774s) [noise=true , address=77.90.31.12:4615 , piece_id=BLNHDHE34WZMROFSL5SNTHOITFWI6JOR7VKQBYZB77LTINEGLS5Q, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:21.667474853 (90h40m21.538557898s) [noise=true , address=145.79.227.24:4587 , piece_id=EVUNUBMSMMIERRCDR67LDFSGY6S2EF6IRV6EGEX4IOFX4PCTYIDQ, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:21.669766096 (90h40m21.540849131s) [noise=true , address=77.90.31.14:4684 , piece_id=ARECBM73CDZDYSVWPLH2XCQC6A45SGW57TPWX5URGCAUW355COOA, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:21.670505664 (90h40m21.541588709s) [noise=true , address=77.90.31.17:4783 , piece_id=LSTC3TY566OQBGL6ZXBLL5IFNPORZRWOEGFKGT3EXYOB3IPWI6JQ, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:21.680637627 (90h40m21.551720662s) [noise=true , address=77.90.31.13:4639 , piece_id=I2GHDHJTAHCDYP7BAG6QAHGPAFIKQZOLUPD5V7AICS4CATG3XNDA, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:21.685855022 (90h40m21.556938057s) [noise=true , address=77.90.31.21:4882 , piece_id=CJSACAOIT5IXGJXTL4YDNRUK4H5KD6QCFE6VINQGMQJ2TT2TAU5A, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:21.741439516 (90h40m21.612522561s) [noise=true , address=77.90.31.19:4870 , piece_id=P6W2QVXDH57FK755LD67JDSC23BZPHVQHBVWZLJDC457JPG3ZQLQ, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:22.306045351 (90h40m22.177128396s) [noise=true , address=77.90.31.16:4760 , piece_id=LJPEX76BQ4ICDJ3I4AWHE7UE7D2WIZCVGQS6XU6ZZ2II2NBYDKHQ, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:22.498367742 (90h40m22.369450787s) [noise=true , address=145.79.227.13:4824 , piece_id=M6P4NFRFOU5LGE2HRGUXRUZXJ2HJYA6BWC3TW4MWUWIYYSUTR76Q, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:10:18.740403422 (90h41m18.611486467s) [noise=true , address=45.91.116.13:4173 , piece_id=7PIVANLRS7SL6ERBMUC4LT4P3C3JRHRY6Z4I7336EVMXC3IHFVDQ, seg_pos={0 0}, upload=79096]: Done uploading piece. err: ecclient: upload failed (node:1J3HZY416Dm9F5zRXtoDdt8s8tGoU6GpkwS14CXJUzd3zpcYa1, address:45.91.116.13:4173): protocol: write tcp 188.166.225.170:53476->45.91.116.13:4173: use of closed network connection; write tcp 188.166.225.170:53476->45.91.116.13:4173: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53476->45.91.116.13:4173: use of closed network connection Dec 9 10:10:18.740531145 (90h41m18.61161419s) [noise=true , address=64.49.13.169:4710 , piece_id=XEKNBKQ4WTECDQD6YYP5RUU2KOSFPRGKXRWDN5THYLE4HXO65BEA, seg_pos={0 0}, upload=79096]: Done uploading piece. err: ecclient: upload failed (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf, address:64.49.13.169:4710): protocol: write tcp 188.166.225.170:37916->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:37916->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37916->64.49.13.169:4710: use of closed network connection Dec 9 10:10:18.740657375 (90h41m18.61174042s) [noise=true , address=45.91.116.12:4254 , piece_id=WOGMNSLO5FOGUSOWXRTLPZNASJB2ZMPUEKC4WRQNROTKQ74JV6KQ, seg_pos={0 0}, upload=79096]: Done uploading piece. err: ecclient: upload failed (node:12wAKmaLHwjBKN8UozGFjGvNMYG57fCxTH8wuDZDRnMdyGBRsTv, address:45.91.116.12:4254): protocol: write tcp 188.166.225.170:35550->45.91.116.12:4254: use of closed network connection; write tcp 188.166.225.170:35550->45.91.116.12:4254: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35550->45.91.116.12:4254: use of closed network connection Dec 9 10:10:18.740778986 (90h41m18.611862021s) [noise=true , address=45.91.116.5:4324 , piece_id=XUK3P6OWTGEJMVLR5B3ZR667WEMCEYIAIVW3PTE6ZWTREKSNBTBA, seg_pos={0 0}, upload=79096]: Done uploading piece. err: ecclient: upload failed (node:12btBm53zb1MCTy6b7CUeujUruMR9nxBoRgqhHmG5ffPf7rL4F, address:45.91.116.5:4324): protocol: write tcp 188.166.225.170:41676->45.91.116.5:4324: use of closed network connection; write tcp 188.166.225.170:41676->45.91.116.5:4324: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41676->45.91.116.5:4324: use of closed network connection Dec 9 10:10:18.740879748 (90h41m18.611962793s) [noise=true , address=45.91.116.11:4140 , piece_id=UJVOUUVBOSA4I4YZJIAPP2QGWBUV322ZSSWINU52U5RIVJ4CEZXA, seg_pos={0 0}, upload=79096]: Done uploading piece. err: ecclient: upload failed (node:14DNa2ysTGtfy3Zz72Dq7S3NSGXqnTHZRkuXorDvuuBw1abwMT, address:45.91.116.11:4140): protocol: write tcp 188.166.225.170:34050->45.91.116.11:4140: use of closed network connection; write tcp 188.166.225.170:34050->45.91.116.11:4140: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34050->45.91.116.11:4140: use of closed network connection Dec 9 10:10:18.740962677 (90h41m18.612045712s) [noise=true , address=45.91.116.9:4264 , piece_id=7UODMOFUXQZ3KYCY4RFA7OUVYXQTWLKNSWR3QIVMIT5KHEQMGT2Q, seg_pos={0 0}, upload=79096]: Done uploading piece. err: ecclient: upload failed (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, address:45.91.116.9:4264): protocol: write tcp 188.166.225.170:50170->45.91.116.9:4264: use of closed network connection; write tcp 188.166.225.170:50170->45.91.116.9:4264: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50170->45.91.116.9:4264: use of closed network connection Dec 9 10:10:18.740989428 (90h41m18.612072463s) [seg_pos={0 0}, upload=79096]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Dec 9 10:10:18.978016295 (90h41m18.84909934s) [upload=79096]: Done uploading segments... Dec 9 10:10:18.978064968 (90h41m18.849148013s) [upload=79096]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 11.453152235s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.X........X......X..................X.....................X.] duration: 5.615233504s trace: https://jaeger.dev.storj.io/trace/5231fff0fb8dd178 FAIL (2025-12-09 10:32:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.615255636s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [..X........X......X..................X.....................X] duration: 1m0.001487551s trace: https://jaeger.dev.storj.io/trace/5231fff0fb8dd178 FAIL (2025-12-09 10:32:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001501387s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..........................X..X..............X...............] duration: 3.534201363s trace: https://jaeger.dev.storj.io/trace/9e23c2a25a964f2 FAIL (2025-12-09 10:32:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.534227943s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 5.877769188s trace: https://jaeger.dev.storj.io/trace/47335bbbc9460f72 FAIL (2025-12-09 10:32:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 10:27:57.242720755 (90h58m57.1138038s) [upload=79359]: Starting upload Dec 9 10:27:57.242779226 (90h58m57.113862261s) [upload=79359]: Done starting upload Dec 9 10:27:57.242849329 (90h58m57.113932365s) [upload=79359]: Uploading segments... Dec 9 10:27:57.242961895 (90h58m57.11404493s) [upload=79359]: Got next segment. Inline: false Dec 9 10:27:57.242992913 (90h58m57.114075948s) [upload=79359]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 10:27:57.242997352 (90h58m57.114080397s) [upload=79359]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 10:27:57.490478536 (90h58m57.361561581s) [seg_pos={0 0}, upload=79359]: Begin upload segment... Dec 9 10:27:57.490533320 (90h58m57.361616365s) [seg_pos={0 0}, upload=79359]: Done begin upload segment. Dec 9 10:27:57.490560913 (90h58m57.361643948s) [noise=true , address=145.79.227.22:4511 , piece_id=R5QAVASG7T7UEXDTFXZOK5BPGO7LJARFI7A77MASBGBQJ4DBP4MA, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.490634854 (90h58m57.361717889s) [noise=true , address=145.79.227.40:4051 , piece_id=GBINALPVGCPMYGIYXJOXKZFKQK7ERXVAWTEZ6XMPEQMVTRMRYRWQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.491225929 (90h58m57.362308994s) [noise=true , address=145.79.227.38:4977 , piece_id=5TTHLFLJJXTFEHA2CSV4DCCOMKT5OH7TQPR63TTB57S7O6L4Y73A, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.491642873 (90h58m57.362725918s) [noise=true , address=145.79.227.39:4014 , piece_id=EW67DYNN54WK4Y7YP7HAZUN227LYDM3LAGCFH7OBET7YIEAUTF3A, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.492140761 (90h58m57.363223796s) [noise=true , address=45.91.116.8:4047 , piece_id=XOO3P7YG5DW5SYP4OVSAGLKCZ6W3KSVXIUK24GDYOXX2H3HMS5GQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.492602540 (90h58m57.363685585s) [noise=true , address=145.79.227.20:4439 , piece_id=JE2X5VYQRHXJDWQFUT2KWCE6UWMD6VLUEHMBZSLL3O6FYMGEY7UQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.493057557 (90h58m57.364140603s) [noise=true , address=145.79.227.29:4941 , piece_id=RJUQZUTVF3KJEGZ4MDOP5RR4DEQH5X4OHHXAK3W64CGQOUY4HZWA, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.493071273 (90h58m57.364154318s) [noise=true , address=145.79.227.42:4123 , piece_id=6JHXIPA352SVNLEWKKPMANX5MQRHXFURZR753T74MMFTPPK4LHJA, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.493431789 (90h58m57.364514854s) [noise=true , address=145.79.227.31:4799 , piece_id=AN4NPL2MIY3NEKKR4TZ7STWCUIZVY6MUQ36D5PJROGD2UG722P2A, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.493540356 (90h58m57.364623401s) [noise=true , address=145.79.227.24:4583 , piece_id=XSC6J57R4UDC2AL3I7ZK73FLNQG7EIIHIJGR34WAK3LT3U6UZJAQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.493892386 (90h58m57.364975431s) [noise=true , address=145.79.227.32:4835 , piece_id=HVGEKOXP5H2HHK3IG2VTIGXX6UTQKWBX63VKL5VQGQMFIRXQV6BQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.494083991 (90h58m57.365167027s) [noise=true , address=145.79.227.23:4547 , piece_id=SQG4BHO3ZZUORDL6YOPFQNGCKVF7TDJEONIEJWNYGCCFXDWX6KIQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.494255959 (90h58m57.365339004s) [noise=true , address=145.79.227.25:4618 , piece_id=UMDOX46LAA6YKCG74OXMIUAVVFG7MNE3XHL2PBERUOASXEBYMGSQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.494606567 (90h58m57.365689602s) [noise=true , address=145.79.227.27:4691 , piece_id=7GAKOVJNXFEN5CDKVGK46EIZ6OU4JI247M6Q35WVF4JOWQN72JPA, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.494965920 (90h58m57.366048956s) [noise=true , address=145.79.227.34:4907 , piece_id=PLUYUJWUQXB6BOP6H6YKCRQ4NNOHJ3M4JHNB7FVKQHO75OJU5HYQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.495345513 (90h58m57.366428558s) [noise=true , address=145.79.227.42:4121 , piece_id=ERCQT36FQSO22HNFY5QZOJJMERRASCVHJW5G4OKCEJB32UNYUQAQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:59.193560364 (90h58m59.064643409s) [upload=79359]: Next returned nil segment Dec 9 10:27:59.193597184 (90h58m59.064680219s) [upload=79359]: Waiting for error group managing segments... Dec 9 10:27:59.477920084 (90h58m59.349003119s) [noise=true , address=145.79.227.42:4121 , piece_id=ERCQT36FQSO22HNFY5QZOJJMERRASCVHJW5G4OKCEJB32UNYUQAQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.480823943 (90h58m59.351906988s) [noise=true , address=145.79.227.31:4799 , piece_id=AN4NPL2MIY3NEKKR4TZ7STWCUIZVY6MUQ36D5PJROGD2UG722P2A, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.487502670 (90h58m59.358585715s) [noise=true , address=145.79.227.27:4691 , piece_id=7GAKOVJNXFEN5CDKVGK46EIZ6OU4JI247M6Q35WVF4JOWQN72JPA, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.491620402 (90h58m59.362703437s) [noise=true , address=145.79.227.29:4941 , piece_id=RJUQZUTVF3KJEGZ4MDOP5RR4DEQH5X4OHHXAK3W64CGQOUY4HZWA, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.491964066 (90h58m59.363047101s) [noise=true , address=145.79.227.42:4123 , piece_id=6JHXIPA352SVNLEWKKPMANX5MQRHXFURZR753T74MMFTPPK4LHJA, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.500675514 (90h58m59.371758559s) [noise=true , address=145.79.227.25:4618 , piece_id=UMDOX46LAA6YKCG74OXMIUAVVFG7MNE3XHL2PBERUOASXEBYMGSQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.501834542 (90h58m59.372917577s) [noise=true , address=145.79.227.39:4014 , piece_id=EW67DYNN54WK4Y7YP7HAZUN227LYDM3LAGCFH7OBET7YIEAUTF3A, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.530610947 (90h58m59.401693992s) [noise=true , address=145.79.227.20:4439 , piece_id=JE2X5VYQRHXJDWQFUT2KWCE6UWMD6VLUEHMBZSLL3O6FYMGEY7UQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.554079847 (90h58m59.425162892s) [noise=true , address=145.79.227.22:4511 , piece_id=R5QAVASG7T7UEXDTFXZOK5BPGO7LJARFI7A77MASBGBQJ4DBP4MA, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.558634540 (90h58m59.429717585s) [noise=true , address=145.79.227.24:4583 , piece_id=XSC6J57R4UDC2AL3I7ZK73FLNQG7EIIHIJGR34WAK3LT3U6UZJAQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.571829987 (90h58m59.442913023s) [noise=true , address=145.79.227.32:4835 , piece_id=HVGEKOXP5H2HHK3IG2VTIGXX6UTQKWBX63VKL5VQGQMFIRXQV6BQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.574149323 (90h58m59.445232368s) [noise=true , address=145.79.227.34:4907 , piece_id=PLUYUJWUQXB6BOP6H6YKCRQ4NNOHJ3M4JHNB7FVKQHO75OJU5HYQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:28:26.550754779 (90h59m26.421837824s) [noise=true , address=45.91.116.8:4047 , piece_id=XOO3P7YG5DW5SYP4OVSAGLKCZ6W3KSVXIUK24GDYOXX2H3HMS5GQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: write tcp 188.166.225.170:58652->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:58652->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58652->45.91.116.8:4047: use of closed network connection Dec 9 10:28:26.550891860 (90h59m26.421974905s) [noise=true , address=145.79.227.38:4977 , piece_id=5TTHLFLJJXTFEHA2CSV4DCCOMKT5OH7TQPR63TTB57S7O6L4Y73A, seg_pos={0 0}, upload=79359]: Done uploading piece. err: ecclient: upload failed (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, address:145.79.227.38:4977): protocol: write tcp 188.166.225.170:48912->145.79.227.38:4977: use of closed network connection; write tcp 188.166.225.170:48912->145.79.227.38:4977: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48912->145.79.227.38:4977: use of closed network connection Dec 9 10:28:26.551013401 (90h59m26.422096446s) [noise=true , address=145.79.227.40:4051 , piece_id=GBINALPVGCPMYGIYXJOXKZFKQK7ERXVAWTEZ6XMPEQMVTRMRYRWQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: write tcp 188.166.225.170:49228->145.79.227.40:4051: use of closed network connection; write tcp 188.166.225.170:49228->145.79.227.40:4051: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49228->145.79.227.40:4051: use of closed network connection Dec 9 10:28:26.551092152 (90h59m26.422175187s) [noise=true , address=145.79.227.23:4547 , piece_id=SQG4BHO3ZZUORDL6YOPFQNGCKVF7TDJEONIEJWNYGCCFXDWX6KIQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: write tcp 188.166.225.170:60386->145.79.227.23:4547: use of closed network connection; write tcp 188.166.225.170:60386->145.79.227.23:4547: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60386->145.79.227.23:4547: use of closed network connection Dec 9 10:28:26.551125365 (90h59m26.42220841s) [seg_pos={0 0}, upload=79359]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 10:28:26.791381862 (90h59m26.662464897s) [upload=79359]: Done uploading segments... Dec 9 10:28:26.791420656 (90h59m26.662503691s) [upload=79359]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.877864591s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 11.941214424s trace: https://jaeger.dev.storj.io/trace/781235978226387b FAIL (2025-12-09 10:31:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ... Dec 9 10:09:19.659155420 (90h40m19.530238455s) [noise=true , address=77.90.31.19:4870 , piece_id=P6W2QVXDH57FK755LD67JDSC23BZPHVQHBVWZLJDC457JPG3ZQLQ, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.659573456 (90h40m19.530656491s) [noise=true , address=77.90.31.15:4703 , piece_id=3FIJGK2XHNRTDF4Q3DN3AX2P42AUR5CNUI5QPXQ256MPMCQB5QRQ, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.660082366 (90h40m19.531165401s) [noise=true , address=77.90.31.16:4760 , piece_id=LJPEX76BQ4ICDJ3I4AWHE7UE7D2WIZCVGQS6XU6ZZ2II2NBYDKHQ, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.660519568 (90h40m19.531602613s) [noise=true , address=45.91.116.9:4264 , piece_id=7UODMOFUXQZ3KYCY4RFA7OUVYXQTWLKNSWR3QIVMIT5KHEQMGT2Q, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.660971178 (90h40m19.532054223s) [noise=true , address=77.90.31.21:4882 , piece_id=CJSACAOIT5IXGJXTL4YDNRUK4H5KD6QCFE6VINQGMQJ2TT2TAU5A, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.661377291 (90h40m19.532460326s) [noise=true , address=77.90.31.13:4639 , piece_id=I2GHDHJTAHCDYP7BAG6QAHGPAFIKQZOLUPD5V7AICS4CATG3XNDA, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.661793374 (90h40m19.532876409s) [noise=true , address=45.91.116.11:4140 , piece_id=UJVOUUVBOSA4I4YZJIAPP2QGWBUV322ZSSWINU52U5RIVJ4CEZXA, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.661991682 (90h40m19.533074717s) [noise=true , address=77.90.31.14:4684 , piece_id=ARECBM73CDZDYSVWPLH2XCQC6A45SGW57TPWX5URGCAUW355COOA, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.662431088 (90h40m19.533514133s) [noise=true , address=77.90.31.12:4615 , piece_id=BLNHDHE34WZMROFSL5SNTHOITFWI6JOR7VKQBYZB77LTINEGLS5Q, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.662848232 (90h40m19.533931277s) [noise=true , address=145.79.227.13:4824 , piece_id=M6P4NFRFOU5LGE2HRGUXRUZXJ2HJYA6BWC3TW4MWUWIYYSUTR76Q, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.663338626 (90h40m19.534421671s) [noise=true , address=145.79.227.24:4587 , piece_id=EVUNUBMSMMIERRCDR67LDFSGY6S2EF6IRV6EGEX4IOFX4PCTYIDQ, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.664502282 (90h40m19.535585327s) [noise=true , address=45.91.116.12:4254 , piece_id=WOGMNSLO5FOGUSOWXRTLPZNASJB2ZMPUEKC4WRQNROTKQ74JV6KQ, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.666711839 (90h40m19.537794884s) [noise=true , address=77.90.31.10:4529 , piece_id=YIT3CCLUOLSQ3NGOUVF7PYVV6FIDT7WKTRQVU4VTU4LRMPYL4H3A, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.669766776 (90h40m19.540849821s) [noise=true , address=77.90.31.17:4783 , piece_id=LSTC3TY566OQBGL6ZXBLL5IFNPORZRWOEGFKGT3EXYOB3IPWI6JQ, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.677090573 (90h40m19.548173618s) [noise=true , address=45.91.116.13:4173 , piece_id=7PIVANLRS7SL6ERBMUC4LT4P3C3JRHRY6Z4I7336EVMXC3IHFVDQ, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:21.186498404 (90h40m21.057581499s) [upload=79096]: Next returned nil segment Dec 9 10:09:21.186518313 (90h40m21.057601348s) [upload=79096]: Waiting for error group managing segments... Dec 9 10:09:21.574704328 (90h40m21.445787374s) [noise=true , address=77.90.31.18:4829 , piece_id=CDRLGWU6VF6TU6E4E5B6R7NOF35WOQOA2TSDPASZP424U23QORNA, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:21.587662915 (90h40m21.45874595s) [noise=true , address=77.90.31.10:4529 , piece_id=YIT3CCLUOLSQ3NGOUVF7PYVV6FIDT7WKTRQVU4VTU4LRMPYL4H3A, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:21.624695792 (90h40m21.495778827s) [noise=true , address=77.90.31.15:4703 , piece_id=3FIJGK2XHNRTDF4Q3DN3AX2P42AUR5CNUI5QPXQ256MPMCQB5QRQ, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:21.659658729 (90h40m21.530741774s) [noise=true , address=77.90.31.12:4615 , piece_id=BLNHDHE34WZMROFSL5SNTHOITFWI6JOR7VKQBYZB77LTINEGLS5Q, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:21.667474853 (90h40m21.538557898s) [noise=true , address=145.79.227.24:4587 , piece_id=EVUNUBMSMMIERRCDR67LDFSGY6S2EF6IRV6EGEX4IOFX4PCTYIDQ, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:21.669766096 (90h40m21.540849131s) [noise=true , address=77.90.31.14:4684 , piece_id=ARECBM73CDZDYSVWPLH2XCQC6A45SGW57TPWX5URGCAUW355COOA, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:21.670505664 (90h40m21.541588709s) [noise=true , address=77.90.31.17:4783 , piece_id=LSTC3TY566OQBGL6ZXBLL5IFNPORZRWOEGFKGT3EXYOB3IPWI6JQ, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:21.680637627 (90h40m21.551720662s) [noise=true , address=77.90.31.13:4639 , piece_id=I2GHDHJTAHCDYP7BAG6QAHGPAFIKQZOLUPD5V7AICS4CATG3XNDA, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:21.685855022 (90h40m21.556938057s) [noise=true , address=77.90.31.21:4882 , piece_id=CJSACAOIT5IXGJXTL4YDNRUK4H5KD6QCFE6VINQGMQJ2TT2TAU5A, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:21.741439516 (90h40m21.612522561s) [noise=true , address=77.90.31.19:4870 , piece_id=P6W2QVXDH57FK755LD67JDSC23BZPHVQHBVWZLJDC457JPG3ZQLQ, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:22.306045351 (90h40m22.177128396s) [noise=true , address=77.90.31.16:4760 , piece_id=LJPEX76BQ4ICDJ3I4AWHE7UE7D2WIZCVGQS6XU6ZZ2II2NBYDKHQ, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:22.498367742 (90h40m22.369450787s) [noise=true , address=145.79.227.13:4824 , piece_id=M6P4NFRFOU5LGE2HRGUXRUZXJ2HJYA6BWC3TW4MWUWIYYSUTR76Q, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:10:18.740403422 (90h41m18.611486467s) [noise=true , address=45.91.116.13:4173 , piece_id=7PIVANLRS7SL6ERBMUC4LT4P3C3JRHRY6Z4I7336EVMXC3IHFVDQ, seg_pos={0 0}, upload=79096]: Done uploading piece. err: ecclient: upload failed (node:1J3HZY416Dm9F5zRXtoDdt8s8tGoU6GpkwS14CXJUzd3zpcYa1, address:45.91.116.13:4173): protocol: write tcp 188.166.225.170:53476->45.91.116.13:4173: use of closed network connection; write tcp 188.166.225.170:53476->45.91.116.13:4173: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53476->45.91.116.13:4173: use of closed network connection Dec 9 10:10:18.740531145 (90h41m18.61161419s) [noise=true , address=64.49.13.169:4710 , piece_id=XEKNBKQ4WTECDQD6YYP5RUU2KOSFPRGKXRWDN5THYLE4HXO65BEA, seg_pos={0 0}, upload=79096]: Done uploading piece. err: ecclient: upload failed (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf, address:64.49.13.169:4710): protocol: write tcp 188.166.225.170:37916->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:37916->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37916->64.49.13.169:4710: use of closed network connection Dec 9 10:10:18.740657375 (90h41m18.61174042s) [noise=true , address=45.91.116.12:4254 , piece_id=WOGMNSLO5FOGUSOWXRTLPZNASJB2ZMPUEKC4WRQNROTKQ74JV6KQ, seg_pos={0 0}, upload=79096]: Done uploading piece. err: ecclient: upload failed (node:12wAKmaLHwjBKN8UozGFjGvNMYG57fCxTH8wuDZDRnMdyGBRsTv, address:45.91.116.12:4254): protocol: write tcp 188.166.225.170:35550->45.91.116.12:4254: use of closed network connection; write tcp 188.166.225.170:35550->45.91.116.12:4254: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35550->45.91.116.12:4254: use of closed network connection Dec 9 10:10:18.740778986 (90h41m18.611862021s) [noise=true , address=45.91.116.5:4324 , piece_id=XUK3P6OWTGEJMVLR5B3ZR667WEMCEYIAIVW3PTE6ZWTREKSNBTBA, seg_pos={0 0}, upload=79096]: Done uploading piece. err: ecclient: upload failed (node:12btBm53zb1MCTy6b7CUeujUruMR9nxBoRgqhHmG5ffPf7rL4F, address:45.91.116.5:4324): protocol: write tcp 188.166.225.170:41676->45.91.116.5:4324: use of closed network connection; write tcp 188.166.225.170:41676->45.91.116.5:4324: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41676->45.91.116.5:4324: use of closed network connection Dec 9 10:10:18.740879748 (90h41m18.611962793s) [noise=true , address=45.91.116.11:4140 , piece_id=UJVOUUVBOSA4I4YZJIAPP2QGWBUV322ZSSWINU52U5RIVJ4CEZXA, seg_pos={0 0}, upload=79096]: Done uploading piece. err: ecclient: upload failed (node:14DNa2ysTGtfy3Zz72Dq7S3NSGXqnTHZRkuXorDvuuBw1abwMT, address:45.91.116.11:4140): protocol: write tcp 188.166.225.170:34050->45.91.116.11:4140: use of closed network connection; write tcp 188.166.225.170:34050->45.91.116.11:4140: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34050->45.91.116.11:4140: use of closed network connection Dec 9 10:10:18.740962677 (90h41m18.612045712s) [noise=true , address=45.91.116.9:4264 , piece_id=7UODMOFUXQZ3KYCY4RFA7OUVYXQTWLKNSWR3QIVMIT5KHEQMGT2Q, seg_pos={0 0}, upload=79096]: Done uploading piece. err: ecclient: upload failed (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, address:45.91.116.9:4264): protocol: write tcp 188.166.225.170:50170->45.91.116.9:4264: use of closed network connection; write tcp 188.166.225.170:50170->45.91.116.9:4264: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50170->45.91.116.9:4264: use of closed network connection Dec 9 10:10:18.740989428 (90h41m18.612072463s) [seg_pos={0 0}, upload=79096]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Dec 9 10:10:18.978016295 (90h41m18.84909934s) [upload=79096]: Done uploading segments... Dec 9 10:10:18.978064968 (90h41m18.849148013s) [upload=79096]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 11.94123914s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 1.980667317s trace: https://jaeger.dev.storj.io/trace/6cb2657ad006bedf FAIL (2025-12-09 10:31:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 09:57:44.389846478 (90h28m44.260929523s) [upload=78917]: Starting upload Dec 9 09:57:44.390051578 (90h28m44.261134613s) [upload=78917]: Done starting upload Dec 9 09:57:44.390125740 (90h28m44.261208775s) [upload=78917]: Uploading segments... Dec 9 09:57:44.390224027 (90h28m44.261307072s) [upload=78917]: Got next segment. Inline: true Dec 9 09:57:44.390230800 (90h28m44.261313835s) [upload=78917]: Waiting for error group managing segments... Dec 9 09:57:44.390312034 (90h28m44.261395069s) [upload=78917]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 09:57:44.390318787 (90h28m44.261401822s) [upload=78917]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 9 09:57:44.390323356 (90h28m44.261406391s) [upload=78917]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 09:58:21.934924620 (90h29m21.806007665s) [upload=78917]: Done uploading segments... Dec 9 09:58:21.934974756 (90h29m21.806057801s) [upload=78917]: Upload finished. err: metaclient: context deadline exceeded Duration:* 1.980698585s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...........................X..X..............X..............] duration: 3.503257483s trace: https://jaeger.dev.storj.io/trace/9e23c2a25a964f2 FAIL (2025-12-09 10:31:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.503288772s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 1.904116065s trace: https://jaeger.dev.storj.io/trace/6cb2657ad006bedf FAIL (2025-12-09 10:31:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 09:57:44.389846478 (90h28m44.260929523s) [upload=78917]: Starting upload Dec 9 09:57:44.390051578 (90h28m44.261134613s) [upload=78917]: Done starting upload Dec 9 09:57:44.390125740 (90h28m44.261208775s) [upload=78917]: Uploading segments... Dec 9 09:57:44.390224027 (90h28m44.261307072s) [upload=78917]: Got next segment. Inline: true Dec 9 09:57:44.390230800 (90h28m44.261313835s) [upload=78917]: Waiting for error group managing segments... Dec 9 09:57:44.390312034 (90h28m44.261395069s) [upload=78917]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 09:57:44.390318787 (90h28m44.261401822s) [upload=78917]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 9 09:57:44.390323356 (90h28m44.261406391s) [upload=78917]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 09:58:21.934924620 (90h29m21.806007665s) [upload=78917]: Done uploading segments... Dec 9 09:58:21.934974756 (90h29m21.806057801s) [upload=78917]: Upload finished. err: metaclient: context deadline exceeded Duration:* 1.904146282s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [...X........X......X..................X.....................] duration: 5.805858623s trace: https://jaeger.dev.storj.io/trace/5b4024e5e8cc17e6 FAIL (2025-12-09 10:31:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.805885665s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 6.084857272s trace: https://jaeger.dev.storj.io/trace/47335bbbc9460f72 FAIL (2025-12-09 10:30:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 10:27:57.242720755 (90h58m57.1138038s) [upload=79359]: Starting upload Dec 9 10:27:57.242779226 (90h58m57.113862261s) [upload=79359]: Done starting upload Dec 9 10:27:57.242849329 (90h58m57.113932365s) [upload=79359]: Uploading segments... Dec 9 10:27:57.242961895 (90h58m57.11404493s) [upload=79359]: Got next segment. Inline: false Dec 9 10:27:57.242992913 (90h58m57.114075948s) [upload=79359]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 10:27:57.242997352 (90h58m57.114080397s) [upload=79359]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 10:27:57.490478536 (90h58m57.361561581s) [seg_pos={0 0}, upload=79359]: Begin upload segment... Dec 9 10:27:57.490533320 (90h58m57.361616365s) [seg_pos={0 0}, upload=79359]: Done begin upload segment. Dec 9 10:27:57.490560913 (90h58m57.361643948s) [noise=true , address=145.79.227.22:4511 , piece_id=R5QAVASG7T7UEXDTFXZOK5BPGO7LJARFI7A77MASBGBQJ4DBP4MA, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.490634854 (90h58m57.361717889s) [noise=true , address=145.79.227.40:4051 , piece_id=GBINALPVGCPMYGIYXJOXKZFKQK7ERXVAWTEZ6XMPEQMVTRMRYRWQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.491225929 (90h58m57.362308994s) [noise=true , address=145.79.227.38:4977 , piece_id=5TTHLFLJJXTFEHA2CSV4DCCOMKT5OH7TQPR63TTB57S7O6L4Y73A, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.491642873 (90h58m57.362725918s) [noise=true , address=145.79.227.39:4014 , piece_id=EW67DYNN54WK4Y7YP7HAZUN227LYDM3LAGCFH7OBET7YIEAUTF3A, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.492140761 (90h58m57.363223796s) [noise=true , address=45.91.116.8:4047 , piece_id=XOO3P7YG5DW5SYP4OVSAGLKCZ6W3KSVXIUK24GDYOXX2H3HMS5GQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.492602540 (90h58m57.363685585s) [noise=true , address=145.79.227.20:4439 , piece_id=JE2X5VYQRHXJDWQFUT2KWCE6UWMD6VLUEHMBZSLL3O6FYMGEY7UQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.493057557 (90h58m57.364140603s) [noise=true , address=145.79.227.29:4941 , piece_id=RJUQZUTVF3KJEGZ4MDOP5RR4DEQH5X4OHHXAK3W64CGQOUY4HZWA, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.493071273 (90h58m57.364154318s) [noise=true , address=145.79.227.42:4123 , piece_id=6JHXIPA352SVNLEWKKPMANX5MQRHXFURZR753T74MMFTPPK4LHJA, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.493431789 (90h58m57.364514854s) [noise=true , address=145.79.227.31:4799 , piece_id=AN4NPL2MIY3NEKKR4TZ7STWCUIZVY6MUQ36D5PJROGD2UG722P2A, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.493540356 (90h58m57.364623401s) [noise=true , address=145.79.227.24:4583 , piece_id=XSC6J57R4UDC2AL3I7ZK73FLNQG7EIIHIJGR34WAK3LT3U6UZJAQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.493892386 (90h58m57.364975431s) [noise=true , address=145.79.227.32:4835 , piece_id=HVGEKOXP5H2HHK3IG2VTIGXX6UTQKWBX63VKL5VQGQMFIRXQV6BQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.494083991 (90h58m57.365167027s) [noise=true , address=145.79.227.23:4547 , piece_id=SQG4BHO3ZZUORDL6YOPFQNGCKVF7TDJEONIEJWNYGCCFXDWX6KIQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.494255959 (90h58m57.365339004s) [noise=true , address=145.79.227.25:4618 , piece_id=UMDOX46LAA6YKCG74OXMIUAVVFG7MNE3XHL2PBERUOASXEBYMGSQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.494606567 (90h58m57.365689602s) [noise=true , address=145.79.227.27:4691 , piece_id=7GAKOVJNXFEN5CDKVGK46EIZ6OU4JI247M6Q35WVF4JOWQN72JPA, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.494965920 (90h58m57.366048956s) [noise=true , address=145.79.227.34:4907 , piece_id=PLUYUJWUQXB6BOP6H6YKCRQ4NNOHJ3M4JHNB7FVKQHO75OJU5HYQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.495345513 (90h58m57.366428558s) [noise=true , address=145.79.227.42:4121 , piece_id=ERCQT36FQSO22HNFY5QZOJJMERRASCVHJW5G4OKCEJB32UNYUQAQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:59.193560364 (90h58m59.064643409s) [upload=79359]: Next returned nil segment Dec 9 10:27:59.193597184 (90h58m59.064680219s) [upload=79359]: Waiting for error group managing segments... Dec 9 10:27:59.477920084 (90h58m59.349003119s) [noise=true , address=145.79.227.42:4121 , piece_id=ERCQT36FQSO22HNFY5QZOJJMERRASCVHJW5G4OKCEJB32UNYUQAQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.480823943 (90h58m59.351906988s) [noise=true , address=145.79.227.31:4799 , piece_id=AN4NPL2MIY3NEKKR4TZ7STWCUIZVY6MUQ36D5PJROGD2UG722P2A, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.487502670 (90h58m59.358585715s) [noise=true , address=145.79.227.27:4691 , piece_id=7GAKOVJNXFEN5CDKVGK46EIZ6OU4JI247M6Q35WVF4JOWQN72JPA, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.491620402 (90h58m59.362703437s) [noise=true , address=145.79.227.29:4941 , piece_id=RJUQZUTVF3KJEGZ4MDOP5RR4DEQH5X4OHHXAK3W64CGQOUY4HZWA, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.491964066 (90h58m59.363047101s) [noise=true , address=145.79.227.42:4123 , piece_id=6JHXIPA352SVNLEWKKPMANX5MQRHXFURZR753T74MMFTPPK4LHJA, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.500675514 (90h58m59.371758559s) [noise=true , address=145.79.227.25:4618 , piece_id=UMDOX46LAA6YKCG74OXMIUAVVFG7MNE3XHL2PBERUOASXEBYMGSQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.501834542 (90h58m59.372917577s) [noise=true , address=145.79.227.39:4014 , piece_id=EW67DYNN54WK4Y7YP7HAZUN227LYDM3LAGCFH7OBET7YIEAUTF3A, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.530610947 (90h58m59.401693992s) [noise=true , address=145.79.227.20:4439 , piece_id=JE2X5VYQRHXJDWQFUT2KWCE6UWMD6VLUEHMBZSLL3O6FYMGEY7UQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.554079847 (90h58m59.425162892s) [noise=true , address=145.79.227.22:4511 , piece_id=R5QAVASG7T7UEXDTFXZOK5BPGO7LJARFI7A77MASBGBQJ4DBP4MA, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.558634540 (90h58m59.429717585s) [noise=true , address=145.79.227.24:4583 , piece_id=XSC6J57R4UDC2AL3I7ZK73FLNQG7EIIHIJGR34WAK3LT3U6UZJAQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.571829987 (90h58m59.442913023s) [noise=true , address=145.79.227.32:4835 , piece_id=HVGEKOXP5H2HHK3IG2VTIGXX6UTQKWBX63VKL5VQGQMFIRXQV6BQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.574149323 (90h58m59.445232368s) [noise=true , address=145.79.227.34:4907 , piece_id=PLUYUJWUQXB6BOP6H6YKCRQ4NNOHJ3M4JHNB7FVKQHO75OJU5HYQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:28:26.550754779 (90h59m26.421837824s) [noise=true , address=45.91.116.8:4047 , piece_id=XOO3P7YG5DW5SYP4OVSAGLKCZ6W3KSVXIUK24GDYOXX2H3HMS5GQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: write tcp 188.166.225.170:58652->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:58652->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58652->45.91.116.8:4047: use of closed network connection Dec 9 10:28:26.550891860 (90h59m26.421974905s) [noise=true , address=145.79.227.38:4977 , piece_id=5TTHLFLJJXTFEHA2CSV4DCCOMKT5OH7TQPR63TTB57S7O6L4Y73A, seg_pos={0 0}, upload=79359]: Done uploading piece. err: ecclient: upload failed (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, address:145.79.227.38:4977): protocol: write tcp 188.166.225.170:48912->145.79.227.38:4977: use of closed network connection; write tcp 188.166.225.170:48912->145.79.227.38:4977: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48912->145.79.227.38:4977: use of closed network connection Dec 9 10:28:26.551013401 (90h59m26.422096446s) [noise=true , address=145.79.227.40:4051 , piece_id=GBINALPVGCPMYGIYXJOXKZFKQK7ERXVAWTEZ6XMPEQMVTRMRYRWQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: write tcp 188.166.225.170:49228->145.79.227.40:4051: use of closed network connection; write tcp 188.166.225.170:49228->145.79.227.40:4051: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49228->145.79.227.40:4051: use of closed network connection Dec 9 10:28:26.551092152 (90h59m26.422175187s) [noise=true , address=145.79.227.23:4547 , piece_id=SQG4BHO3ZZUORDL6YOPFQNGCKVF7TDJEONIEJWNYGCCFXDWX6KIQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: write tcp 188.166.225.170:60386->145.79.227.23:4547: use of closed network connection; write tcp 188.166.225.170:60386->145.79.227.23:4547: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60386->145.79.227.23:4547: use of closed network connection Dec 9 10:28:26.551125365 (90h59m26.42220841s) [seg_pos={0 0}, upload=79359]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 10:28:26.791381862 (90h59m26.662464897s) [upload=79359]: Done uploading segments... Dec 9 10:28:26.791420656 (90h59m26.662503691s) [upload=79359]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.084880787s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [....X........X......X..................X....................] duration: 5.971463208s trace: https://jaeger.dev.storj.io/trace/5b4024e5e8cc17e6 FAIL (2025-12-09 10:30:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.97149119s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 11.330469331s trace: https://jaeger.dev.storj.io/trace/781235978226387b FAIL (2025-12-09 10:30:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ... Dec 9 10:09:19.659155420 (90h40m19.530238455s) [noise=true , address=77.90.31.19:4870 , piece_id=P6W2QVXDH57FK755LD67JDSC23BZPHVQHBVWZLJDC457JPG3ZQLQ, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.659573456 (90h40m19.530656491s) [noise=true , address=77.90.31.15:4703 , piece_id=3FIJGK2XHNRTDF4Q3DN3AX2P42AUR5CNUI5QPXQ256MPMCQB5QRQ, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.660082366 (90h40m19.531165401s) [noise=true , address=77.90.31.16:4760 , piece_id=LJPEX76BQ4ICDJ3I4AWHE7UE7D2WIZCVGQS6XU6ZZ2II2NBYDKHQ, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.660519568 (90h40m19.531602613s) [noise=true , address=45.91.116.9:4264 , piece_id=7UODMOFUXQZ3KYCY4RFA7OUVYXQTWLKNSWR3QIVMIT5KHEQMGT2Q, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.660971178 (90h40m19.532054223s) [noise=true , address=77.90.31.21:4882 , piece_id=CJSACAOIT5IXGJXTL4YDNRUK4H5KD6QCFE6VINQGMQJ2TT2TAU5A, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.661377291 (90h40m19.532460326s) [noise=true , address=77.90.31.13:4639 , piece_id=I2GHDHJTAHCDYP7BAG6QAHGPAFIKQZOLUPD5V7AICS4CATG3XNDA, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.661793374 (90h40m19.532876409s) [noise=true , address=45.91.116.11:4140 , piece_id=UJVOUUVBOSA4I4YZJIAPP2QGWBUV322ZSSWINU52U5RIVJ4CEZXA, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.661991682 (90h40m19.533074717s) [noise=true , address=77.90.31.14:4684 , piece_id=ARECBM73CDZDYSVWPLH2XCQC6A45SGW57TPWX5URGCAUW355COOA, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.662431088 (90h40m19.533514133s) [noise=true , address=77.90.31.12:4615 , piece_id=BLNHDHE34WZMROFSL5SNTHOITFWI6JOR7VKQBYZB77LTINEGLS5Q, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.662848232 (90h40m19.533931277s) [noise=true , address=145.79.227.13:4824 , piece_id=M6P4NFRFOU5LGE2HRGUXRUZXJ2HJYA6BWC3TW4MWUWIYYSUTR76Q, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.663338626 (90h40m19.534421671s) [noise=true , address=145.79.227.24:4587 , piece_id=EVUNUBMSMMIERRCDR67LDFSGY6S2EF6IRV6EGEX4IOFX4PCTYIDQ, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.664502282 (90h40m19.535585327s) [noise=true , address=45.91.116.12:4254 , piece_id=WOGMNSLO5FOGUSOWXRTLPZNASJB2ZMPUEKC4WRQNROTKQ74JV6KQ, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.666711839 (90h40m19.537794884s) [noise=true , address=77.90.31.10:4529 , piece_id=YIT3CCLUOLSQ3NGOUVF7PYVV6FIDT7WKTRQVU4VTU4LRMPYL4H3A, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.669766776 (90h40m19.540849821s) [noise=true , address=77.90.31.17:4783 , piece_id=LSTC3TY566OQBGL6ZXBLL5IFNPORZRWOEGFKGT3EXYOB3IPWI6JQ, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.677090573 (90h40m19.548173618s) [noise=true , address=45.91.116.13:4173 , piece_id=7PIVANLRS7SL6ERBMUC4LT4P3C3JRHRY6Z4I7336EVMXC3IHFVDQ, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:21.186498404 (90h40m21.057581499s) [upload=79096]: Next returned nil segment Dec 9 10:09:21.186518313 (90h40m21.057601348s) [upload=79096]: Waiting for error group managing segments... Dec 9 10:09:21.574704328 (90h40m21.445787374s) [noise=true , address=77.90.31.18:4829 , piece_id=CDRLGWU6VF6TU6E4E5B6R7NOF35WOQOA2TSDPASZP424U23QORNA, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:21.587662915 (90h40m21.45874595s) [noise=true , address=77.90.31.10:4529 , piece_id=YIT3CCLUOLSQ3NGOUVF7PYVV6FIDT7WKTRQVU4VTU4LRMPYL4H3A, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:21.624695792 (90h40m21.495778827s) [noise=true , address=77.90.31.15:4703 , piece_id=3FIJGK2XHNRTDF4Q3DN3AX2P42AUR5CNUI5QPXQ256MPMCQB5QRQ, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:21.659658729 (90h40m21.530741774s) [noise=true , address=77.90.31.12:4615 , piece_id=BLNHDHE34WZMROFSL5SNTHOITFWI6JOR7VKQBYZB77LTINEGLS5Q, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:21.667474853 (90h40m21.538557898s) [noise=true , address=145.79.227.24:4587 , piece_id=EVUNUBMSMMIERRCDR67LDFSGY6S2EF6IRV6EGEX4IOFX4PCTYIDQ, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:21.669766096 (90h40m21.540849131s) [noise=true , address=77.90.31.14:4684 , piece_id=ARECBM73CDZDYSVWPLH2XCQC6A45SGW57TPWX5URGCAUW355COOA, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:21.670505664 (90h40m21.541588709s) [noise=true , address=77.90.31.17:4783 , piece_id=LSTC3TY566OQBGL6ZXBLL5IFNPORZRWOEGFKGT3EXYOB3IPWI6JQ, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:21.680637627 (90h40m21.551720662s) [noise=true , address=77.90.31.13:4639 , piece_id=I2GHDHJTAHCDYP7BAG6QAHGPAFIKQZOLUPD5V7AICS4CATG3XNDA, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:21.685855022 (90h40m21.556938057s) [noise=true , address=77.90.31.21:4882 , piece_id=CJSACAOIT5IXGJXTL4YDNRUK4H5KD6QCFE6VINQGMQJ2TT2TAU5A, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:21.741439516 (90h40m21.612522561s) [noise=true , address=77.90.31.19:4870 , piece_id=P6W2QVXDH57FK755LD67JDSC23BZPHVQHBVWZLJDC457JPG3ZQLQ, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:22.306045351 (90h40m22.177128396s) [noise=true , address=77.90.31.16:4760 , piece_id=LJPEX76BQ4ICDJ3I4AWHE7UE7D2WIZCVGQS6XU6ZZ2II2NBYDKHQ, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:22.498367742 (90h40m22.369450787s) [noise=true , address=145.79.227.13:4824 , piece_id=M6P4NFRFOU5LGE2HRGUXRUZXJ2HJYA6BWC3TW4MWUWIYYSUTR76Q, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:10:18.740403422 (90h41m18.611486467s) [noise=true , address=45.91.116.13:4173 , piece_id=7PIVANLRS7SL6ERBMUC4LT4P3C3JRHRY6Z4I7336EVMXC3IHFVDQ, seg_pos={0 0}, upload=79096]: Done uploading piece. err: ecclient: upload failed (node:1J3HZY416Dm9F5zRXtoDdt8s8tGoU6GpkwS14CXJUzd3zpcYa1, address:45.91.116.13:4173): protocol: write tcp 188.166.225.170:53476->45.91.116.13:4173: use of closed network connection; write tcp 188.166.225.170:53476->45.91.116.13:4173: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53476->45.91.116.13:4173: use of closed network connection Dec 9 10:10:18.740531145 (90h41m18.61161419s) [noise=true , address=64.49.13.169:4710 , piece_id=XEKNBKQ4WTECDQD6YYP5RUU2KOSFPRGKXRWDN5THYLE4HXO65BEA, seg_pos={0 0}, upload=79096]: Done uploading piece. err: ecclient: upload failed (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf, address:64.49.13.169:4710): protocol: write tcp 188.166.225.170:37916->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:37916->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37916->64.49.13.169:4710: use of closed network connection Dec 9 10:10:18.740657375 (90h41m18.61174042s) [noise=true , address=45.91.116.12:4254 , piece_id=WOGMNSLO5FOGUSOWXRTLPZNASJB2ZMPUEKC4WRQNROTKQ74JV6KQ, seg_pos={0 0}, upload=79096]: Done uploading piece. err: ecclient: upload failed (node:12wAKmaLHwjBKN8UozGFjGvNMYG57fCxTH8wuDZDRnMdyGBRsTv, address:45.91.116.12:4254): protocol: write tcp 188.166.225.170:35550->45.91.116.12:4254: use of closed network connection; write tcp 188.166.225.170:35550->45.91.116.12:4254: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35550->45.91.116.12:4254: use of closed network connection Dec 9 10:10:18.740778986 (90h41m18.611862021s) [noise=true , address=45.91.116.5:4324 , piece_id=XUK3P6OWTGEJMVLR5B3ZR667WEMCEYIAIVW3PTE6ZWTREKSNBTBA, seg_pos={0 0}, upload=79096]: Done uploading piece. err: ecclient: upload failed (node:12btBm53zb1MCTy6b7CUeujUruMR9nxBoRgqhHmG5ffPf7rL4F, address:45.91.116.5:4324): protocol: write tcp 188.166.225.170:41676->45.91.116.5:4324: use of closed network connection; write tcp 188.166.225.170:41676->45.91.116.5:4324: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41676->45.91.116.5:4324: use of closed network connection Dec 9 10:10:18.740879748 (90h41m18.611962793s) [noise=true , address=45.91.116.11:4140 , piece_id=UJVOUUVBOSA4I4YZJIAPP2QGWBUV322ZSSWINU52U5RIVJ4CEZXA, seg_pos={0 0}, upload=79096]: Done uploading piece. err: ecclient: upload failed (node:14DNa2ysTGtfy3Zz72Dq7S3NSGXqnTHZRkuXorDvuuBw1abwMT, address:45.91.116.11:4140): protocol: write tcp 188.166.225.170:34050->45.91.116.11:4140: use of closed network connection; write tcp 188.166.225.170:34050->45.91.116.11:4140: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34050->45.91.116.11:4140: use of closed network connection Dec 9 10:10:18.740962677 (90h41m18.612045712s) [noise=true , address=45.91.116.9:4264 , piece_id=7UODMOFUXQZ3KYCY4RFA7OUVYXQTWLKNSWR3QIVMIT5KHEQMGT2Q, seg_pos={0 0}, upload=79096]: Done uploading piece. err: ecclient: upload failed (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, address:45.91.116.9:4264): protocol: write tcp 188.166.225.170:50170->45.91.116.9:4264: use of closed network connection; write tcp 188.166.225.170:50170->45.91.116.9:4264: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50170->45.91.116.9:4264: use of closed network connection Dec 9 10:10:18.740989428 (90h41m18.612072463s) [seg_pos={0 0}, upload=79096]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Dec 9 10:10:18.978016295 (90h41m18.84909934s) [upload=79096]: Done uploading segments... Dec 9 10:10:18.978064968 (90h41m18.849148013s) [upload=79096]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 11.330500902s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 5.744025659s trace: https://jaeger.dev.storj.io/trace/47335bbbc9460f72 FAIL (2025-12-09 10:30:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 10:27:57.242720755 (90h58m57.1138038s) [upload=79359]: Starting upload Dec 9 10:27:57.242779226 (90h58m57.113862261s) [upload=79359]: Done starting upload Dec 9 10:27:57.242849329 (90h58m57.113932365s) [upload=79359]: Uploading segments... Dec 9 10:27:57.242961895 (90h58m57.11404493s) [upload=79359]: Got next segment. Inline: false Dec 9 10:27:57.242992913 (90h58m57.114075948s) [upload=79359]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 10:27:57.242997352 (90h58m57.114080397s) [upload=79359]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 10:27:57.490478536 (90h58m57.361561581s) [seg_pos={0 0}, upload=79359]: Begin upload segment... Dec 9 10:27:57.490533320 (90h58m57.361616365s) [seg_pos={0 0}, upload=79359]: Done begin upload segment. Dec 9 10:27:57.490560913 (90h58m57.361643948s) [noise=true , address=145.79.227.22:4511 , piece_id=R5QAVASG7T7UEXDTFXZOK5BPGO7LJARFI7A77MASBGBQJ4DBP4MA, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.490634854 (90h58m57.361717889s) [noise=true , address=145.79.227.40:4051 , piece_id=GBINALPVGCPMYGIYXJOXKZFKQK7ERXVAWTEZ6XMPEQMVTRMRYRWQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.491225929 (90h58m57.362308994s) [noise=true , address=145.79.227.38:4977 , piece_id=5TTHLFLJJXTFEHA2CSV4DCCOMKT5OH7TQPR63TTB57S7O6L4Y73A, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.491642873 (90h58m57.362725918s) [noise=true , address=145.79.227.39:4014 , piece_id=EW67DYNN54WK4Y7YP7HAZUN227LYDM3LAGCFH7OBET7YIEAUTF3A, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.492140761 (90h58m57.363223796s) [noise=true , address=45.91.116.8:4047 , piece_id=XOO3P7YG5DW5SYP4OVSAGLKCZ6W3KSVXIUK24GDYOXX2H3HMS5GQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.492602540 (90h58m57.363685585s) [noise=true , address=145.79.227.20:4439 , piece_id=JE2X5VYQRHXJDWQFUT2KWCE6UWMD6VLUEHMBZSLL3O6FYMGEY7UQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.493057557 (90h58m57.364140603s) [noise=true , address=145.79.227.29:4941 , piece_id=RJUQZUTVF3KJEGZ4MDOP5RR4DEQH5X4OHHXAK3W64CGQOUY4HZWA, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.493071273 (90h58m57.364154318s) [noise=true , address=145.79.227.42:4123 , piece_id=6JHXIPA352SVNLEWKKPMANX5MQRHXFURZR753T74MMFTPPK4LHJA, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.493431789 (90h58m57.364514854s) [noise=true , address=145.79.227.31:4799 , piece_id=AN4NPL2MIY3NEKKR4TZ7STWCUIZVY6MUQ36D5PJROGD2UG722P2A, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.493540356 (90h58m57.364623401s) [noise=true , address=145.79.227.24:4583 , piece_id=XSC6J57R4UDC2AL3I7ZK73FLNQG7EIIHIJGR34WAK3LT3U6UZJAQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.493892386 (90h58m57.364975431s) [noise=true , address=145.79.227.32:4835 , piece_id=HVGEKOXP5H2HHK3IG2VTIGXX6UTQKWBX63VKL5VQGQMFIRXQV6BQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.494083991 (90h58m57.365167027s) [noise=true , address=145.79.227.23:4547 , piece_id=SQG4BHO3ZZUORDL6YOPFQNGCKVF7TDJEONIEJWNYGCCFXDWX6KIQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.494255959 (90h58m57.365339004s) [noise=true , address=145.79.227.25:4618 , piece_id=UMDOX46LAA6YKCG74OXMIUAVVFG7MNE3XHL2PBERUOASXEBYMGSQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.494606567 (90h58m57.365689602s) [noise=true , address=145.79.227.27:4691 , piece_id=7GAKOVJNXFEN5CDKVGK46EIZ6OU4JI247M6Q35WVF4JOWQN72JPA, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.494965920 (90h58m57.366048956s) [noise=true , address=145.79.227.34:4907 , piece_id=PLUYUJWUQXB6BOP6H6YKCRQ4NNOHJ3M4JHNB7FVKQHO75OJU5HYQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.495345513 (90h58m57.366428558s) [noise=true , address=145.79.227.42:4121 , piece_id=ERCQT36FQSO22HNFY5QZOJJMERRASCVHJW5G4OKCEJB32UNYUQAQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:59.193560364 (90h58m59.064643409s) [upload=79359]: Next returned nil segment Dec 9 10:27:59.193597184 (90h58m59.064680219s) [upload=79359]: Waiting for error group managing segments... Dec 9 10:27:59.477920084 (90h58m59.349003119s) [noise=true , address=145.79.227.42:4121 , piece_id=ERCQT36FQSO22HNFY5QZOJJMERRASCVHJW5G4OKCEJB32UNYUQAQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.480823943 (90h58m59.351906988s) [noise=true , address=145.79.227.31:4799 , piece_id=AN4NPL2MIY3NEKKR4TZ7STWCUIZVY6MUQ36D5PJROGD2UG722P2A, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.487502670 (90h58m59.358585715s) [noise=true , address=145.79.227.27:4691 , piece_id=7GAKOVJNXFEN5CDKVGK46EIZ6OU4JI247M6Q35WVF4JOWQN72JPA, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.491620402 (90h58m59.362703437s) [noise=true , address=145.79.227.29:4941 , piece_id=RJUQZUTVF3KJEGZ4MDOP5RR4DEQH5X4OHHXAK3W64CGQOUY4HZWA, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.491964066 (90h58m59.363047101s) [noise=true , address=145.79.227.42:4123 , piece_id=6JHXIPA352SVNLEWKKPMANX5MQRHXFURZR753T74MMFTPPK4LHJA, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.500675514 (90h58m59.371758559s) [noise=true , address=145.79.227.25:4618 , piece_id=UMDOX46LAA6YKCG74OXMIUAVVFG7MNE3XHL2PBERUOASXEBYMGSQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.501834542 (90h58m59.372917577s) [noise=true , address=145.79.227.39:4014 , piece_id=EW67DYNN54WK4Y7YP7HAZUN227LYDM3LAGCFH7OBET7YIEAUTF3A, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.530610947 (90h58m59.401693992s) [noise=true , address=145.79.227.20:4439 , piece_id=JE2X5VYQRHXJDWQFUT2KWCE6UWMD6VLUEHMBZSLL3O6FYMGEY7UQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.554079847 (90h58m59.425162892s) [noise=true , address=145.79.227.22:4511 , piece_id=R5QAVASG7T7UEXDTFXZOK5BPGO7LJARFI7A77MASBGBQJ4DBP4MA, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.558634540 (90h58m59.429717585s) [noise=true , address=145.79.227.24:4583 , piece_id=XSC6J57R4UDC2AL3I7ZK73FLNQG7EIIHIJGR34WAK3LT3U6UZJAQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.571829987 (90h58m59.442913023s) [noise=true , address=145.79.227.32:4835 , piece_id=HVGEKOXP5H2HHK3IG2VTIGXX6UTQKWBX63VKL5VQGQMFIRXQV6BQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.574149323 (90h58m59.445232368s) [noise=true , address=145.79.227.34:4907 , piece_id=PLUYUJWUQXB6BOP6H6YKCRQ4NNOHJ3M4JHNB7FVKQHO75OJU5HYQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:28:26.550754779 (90h59m26.421837824s) [noise=true , address=45.91.116.8:4047 , piece_id=XOO3P7YG5DW5SYP4OVSAGLKCZ6W3KSVXIUK24GDYOXX2H3HMS5GQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: write tcp 188.166.225.170:58652->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:58652->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58652->45.91.116.8:4047: use of closed network connection Dec 9 10:28:26.550891860 (90h59m26.421974905s) [noise=true , address=145.79.227.38:4977 , piece_id=5TTHLFLJJXTFEHA2CSV4DCCOMKT5OH7TQPR63TTB57S7O6L4Y73A, seg_pos={0 0}, upload=79359]: Done uploading piece. err: ecclient: upload failed (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, address:145.79.227.38:4977): protocol: write tcp 188.166.225.170:48912->145.79.227.38:4977: use of closed network connection; write tcp 188.166.225.170:48912->145.79.227.38:4977: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48912->145.79.227.38:4977: use of closed network connection Dec 9 10:28:26.551013401 (90h59m26.422096446s) [noise=true , address=145.79.227.40:4051 , piece_id=GBINALPVGCPMYGIYXJOXKZFKQK7ERXVAWTEZ6XMPEQMVTRMRYRWQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: write tcp 188.166.225.170:49228->145.79.227.40:4051: use of closed network connection; write tcp 188.166.225.170:49228->145.79.227.40:4051: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49228->145.79.227.40:4051: use of closed network connection Dec 9 10:28:26.551092152 (90h59m26.422175187s) [noise=true , address=145.79.227.23:4547 , piece_id=SQG4BHO3ZZUORDL6YOPFQNGCKVF7TDJEONIEJWNYGCCFXDWX6KIQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: write tcp 188.166.225.170:60386->145.79.227.23:4547: use of closed network connection; write tcp 188.166.225.170:60386->145.79.227.23:4547: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60386->145.79.227.23:4547: use of closed network connection Dec 9 10:28:26.551125365 (90h59m26.42220841s) [seg_pos={0 0}, upload=79359]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 10:28:26.791381862 (90h59m26.662464897s) [upload=79359]: Done uploading segments... Dec 9 10:28:26.791420656 (90h59m26.662503691s) [upload=79359]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.744055576s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 1.931091243s trace: https://jaeger.dev.storj.io/trace/6cb2657ad006bedf FAIL (2025-12-09 10:30:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 09:57:44.389846478 (90h28m44.260929523s) [upload=78917]: Starting upload Dec 9 09:57:44.390051578 (90h28m44.261134613s) [upload=78917]: Done starting upload Dec 9 09:57:44.390125740 (90h28m44.261208775s) [upload=78917]: Uploading segments... Dec 9 09:57:44.390224027 (90h28m44.261307072s) [upload=78917]: Got next segment. Inline: true Dec 9 09:57:44.390230800 (90h28m44.261313835s) [upload=78917]: Waiting for error group managing segments... Dec 9 09:57:44.390312034 (90h28m44.261395069s) [upload=78917]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 09:57:44.390318787 (90h28m44.261401822s) [upload=78917]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 9 09:57:44.390323356 (90h28m44.261406391s) [upload=78917]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 09:58:21.934924620 (90h29m21.806007665s) [upload=78917]: Done uploading segments... Dec 9 09:58:21.934974756 (90h29m21.806057801s) [upload=78917]: Upload finished. err: metaclient: context deadline exceeded Duration:* 1.931113615s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.....X........X......X..................X...................] duration: 6.124116962s trace: https://jaeger.dev.storj.io/trace/5b4024e5e8cc17e6 FAIL (2025-12-09 10:30:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.124150385s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [............................X..X..............X.............] duration: 3.555179138s trace: https://jaeger.dev.storj.io/trace/9e23c2a25a964f2 FAIL (2025-12-09 10:29:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.555204757s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 2.06506618s trace: https://jaeger.dev.storj.io/trace/6cb2657ad006bedf FAIL (2025-12-09 10:29:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 09:57:44.389846478 (90h28m44.260929523s) [upload=78917]: Starting upload Dec 9 09:57:44.390051578 (90h28m44.261134613s) [upload=78917]: Done starting upload Dec 9 09:57:44.390125740 (90h28m44.261208775s) [upload=78917]: Uploading segments... Dec 9 09:57:44.390224027 (90h28m44.261307072s) [upload=78917]: Got next segment. Inline: true Dec 9 09:57:44.390230800 (90h28m44.261313835s) [upload=78917]: Waiting for error group managing segments... Dec 9 09:57:44.390312034 (90h28m44.261395069s) [upload=78917]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 09:57:44.390318787 (90h28m44.261401822s) [upload=78917]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 9 09:57:44.390323356 (90h28m44.261406391s) [upload=78917]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 09:58:21.934924620 (90h29m21.806007665s) [upload=78917]: Done uploading segments... Dec 9 09:58:21.934974756 (90h29m21.806057801s) [upload=78917]: Upload finished. err: metaclient: context deadline exceeded Duration:* 2.065099994s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [......X........X......X..................X..................] duration: 5.815391887s trace: https://jaeger.dev.storj.io/trace/5b4024e5e8cc17e6 FAIL (2025-12-09 10:29:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.815422735s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.............................X..X..............X............] duration: 3.439716124s trace: https://jaeger.dev.storj.io/trace/9e23c2a25a964f2 FAIL (2025-12-09 10:29:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.439743927s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 11.203788683s trace: https://jaeger.dev.storj.io/trace/781235978226387b FAIL (2025-12-09 10:29:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ... Dec 9 10:09:19.659155420 (90h40m19.530238455s) [noise=true , address=77.90.31.19:4870 , piece_id=P6W2QVXDH57FK755LD67JDSC23BZPHVQHBVWZLJDC457JPG3ZQLQ, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.659573456 (90h40m19.530656491s) [noise=true , address=77.90.31.15:4703 , piece_id=3FIJGK2XHNRTDF4Q3DN3AX2P42AUR5CNUI5QPXQ256MPMCQB5QRQ, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.660082366 (90h40m19.531165401s) [noise=true , address=77.90.31.16:4760 , piece_id=LJPEX76BQ4ICDJ3I4AWHE7UE7D2WIZCVGQS6XU6ZZ2II2NBYDKHQ, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.660519568 (90h40m19.531602613s) [noise=true , address=45.91.116.9:4264 , piece_id=7UODMOFUXQZ3KYCY4RFA7OUVYXQTWLKNSWR3QIVMIT5KHEQMGT2Q, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.660971178 (90h40m19.532054223s) [noise=true , address=77.90.31.21:4882 , piece_id=CJSACAOIT5IXGJXTL4YDNRUK4H5KD6QCFE6VINQGMQJ2TT2TAU5A, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.661377291 (90h40m19.532460326s) [noise=true , address=77.90.31.13:4639 , piece_id=I2GHDHJTAHCDYP7BAG6QAHGPAFIKQZOLUPD5V7AICS4CATG3XNDA, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.661793374 (90h40m19.532876409s) [noise=true , address=45.91.116.11:4140 , piece_id=UJVOUUVBOSA4I4YZJIAPP2QGWBUV322ZSSWINU52U5RIVJ4CEZXA, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.661991682 (90h40m19.533074717s) [noise=true , address=77.90.31.14:4684 , piece_id=ARECBM73CDZDYSVWPLH2XCQC6A45SGW57TPWX5URGCAUW355COOA, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.662431088 (90h40m19.533514133s) [noise=true , address=77.90.31.12:4615 , piece_id=BLNHDHE34WZMROFSL5SNTHOITFWI6JOR7VKQBYZB77LTINEGLS5Q, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.662848232 (90h40m19.533931277s) [noise=true , address=145.79.227.13:4824 , piece_id=M6P4NFRFOU5LGE2HRGUXRUZXJ2HJYA6BWC3TW4MWUWIYYSUTR76Q, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.663338626 (90h40m19.534421671s) [noise=true , address=145.79.227.24:4587 , piece_id=EVUNUBMSMMIERRCDR67LDFSGY6S2EF6IRV6EGEX4IOFX4PCTYIDQ, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.664502282 (90h40m19.535585327s) [noise=true , address=45.91.116.12:4254 , piece_id=WOGMNSLO5FOGUSOWXRTLPZNASJB2ZMPUEKC4WRQNROTKQ74JV6KQ, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.666711839 (90h40m19.537794884s) [noise=true , address=77.90.31.10:4529 , piece_id=YIT3CCLUOLSQ3NGOUVF7PYVV6FIDT7WKTRQVU4VTU4LRMPYL4H3A, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.669766776 (90h40m19.540849821s) [noise=true , address=77.90.31.17:4783 , piece_id=LSTC3TY566OQBGL6ZXBLL5IFNPORZRWOEGFKGT3EXYOB3IPWI6JQ, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.677090573 (90h40m19.548173618s) [noise=true , address=45.91.116.13:4173 , piece_id=7PIVANLRS7SL6ERBMUC4LT4P3C3JRHRY6Z4I7336EVMXC3IHFVDQ, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:21.186498404 (90h40m21.057581499s) [upload=79096]: Next returned nil segment Dec 9 10:09:21.186518313 (90h40m21.057601348s) [upload=79096]: Waiting for error group managing segments... Dec 9 10:09:21.574704328 (90h40m21.445787374s) [noise=true , address=77.90.31.18:4829 , piece_id=CDRLGWU6VF6TU6E4E5B6R7NOF35WOQOA2TSDPASZP424U23QORNA, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:21.587662915 (90h40m21.45874595s) [noise=true , address=77.90.31.10:4529 , piece_id=YIT3CCLUOLSQ3NGOUVF7PYVV6FIDT7WKTRQVU4VTU4LRMPYL4H3A, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:21.624695792 (90h40m21.495778827s) [noise=true , address=77.90.31.15:4703 , piece_id=3FIJGK2XHNRTDF4Q3DN3AX2P42AUR5CNUI5QPXQ256MPMCQB5QRQ, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:21.659658729 (90h40m21.530741774s) [noise=true , address=77.90.31.12:4615 , piece_id=BLNHDHE34WZMROFSL5SNTHOITFWI6JOR7VKQBYZB77LTINEGLS5Q, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:21.667474853 (90h40m21.538557898s) [noise=true , address=145.79.227.24:4587 , piece_id=EVUNUBMSMMIERRCDR67LDFSGY6S2EF6IRV6EGEX4IOFX4PCTYIDQ, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:21.669766096 (90h40m21.540849131s) [noise=true , address=77.90.31.14:4684 , piece_id=ARECBM73CDZDYSVWPLH2XCQC6A45SGW57TPWX5URGCAUW355COOA, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:21.670505664 (90h40m21.541588709s) [noise=true , address=77.90.31.17:4783 , piece_id=LSTC3TY566OQBGL6ZXBLL5IFNPORZRWOEGFKGT3EXYOB3IPWI6JQ, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:21.680637627 (90h40m21.551720662s) [noise=true , address=77.90.31.13:4639 , piece_id=I2GHDHJTAHCDYP7BAG6QAHGPAFIKQZOLUPD5V7AICS4CATG3XNDA, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:21.685855022 (90h40m21.556938057s) [noise=true , address=77.90.31.21:4882 , piece_id=CJSACAOIT5IXGJXTL4YDNRUK4H5KD6QCFE6VINQGMQJ2TT2TAU5A, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:21.741439516 (90h40m21.612522561s) [noise=true , address=77.90.31.19:4870 , piece_id=P6W2QVXDH57FK755LD67JDSC23BZPHVQHBVWZLJDC457JPG3ZQLQ, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:22.306045351 (90h40m22.177128396s) [noise=true , address=77.90.31.16:4760 , piece_id=LJPEX76BQ4ICDJ3I4AWHE7UE7D2WIZCVGQS6XU6ZZ2II2NBYDKHQ, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:22.498367742 (90h40m22.369450787s) [noise=true , address=145.79.227.13:4824 , piece_id=M6P4NFRFOU5LGE2HRGUXRUZXJ2HJYA6BWC3TW4MWUWIYYSUTR76Q, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:10:18.740403422 (90h41m18.611486467s) [noise=true , address=45.91.116.13:4173 , piece_id=7PIVANLRS7SL6ERBMUC4LT4P3C3JRHRY6Z4I7336EVMXC3IHFVDQ, seg_pos={0 0}, upload=79096]: Done uploading piece. err: ecclient: upload failed (node:1J3HZY416Dm9F5zRXtoDdt8s8tGoU6GpkwS14CXJUzd3zpcYa1, address:45.91.116.13:4173): protocol: write tcp 188.166.225.170:53476->45.91.116.13:4173: use of closed network connection; write tcp 188.166.225.170:53476->45.91.116.13:4173: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53476->45.91.116.13:4173: use of closed network connection Dec 9 10:10:18.740531145 (90h41m18.61161419s) [noise=true , address=64.49.13.169:4710 , piece_id=XEKNBKQ4WTECDQD6YYP5RUU2KOSFPRGKXRWDN5THYLE4HXO65BEA, seg_pos={0 0}, upload=79096]: Done uploading piece. err: ecclient: upload failed (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf, address:64.49.13.169:4710): protocol: write tcp 188.166.225.170:37916->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:37916->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37916->64.49.13.169:4710: use of closed network connection Dec 9 10:10:18.740657375 (90h41m18.61174042s) [noise=true , address=45.91.116.12:4254 , piece_id=WOGMNSLO5FOGUSOWXRTLPZNASJB2ZMPUEKC4WRQNROTKQ74JV6KQ, seg_pos={0 0}, upload=79096]: Done uploading piece. err: ecclient: upload failed (node:12wAKmaLHwjBKN8UozGFjGvNMYG57fCxTH8wuDZDRnMdyGBRsTv, address:45.91.116.12:4254): protocol: write tcp 188.166.225.170:35550->45.91.116.12:4254: use of closed network connection; write tcp 188.166.225.170:35550->45.91.116.12:4254: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35550->45.91.116.12:4254: use of closed network connection Dec 9 10:10:18.740778986 (90h41m18.611862021s) [noise=true , address=45.91.116.5:4324 , piece_id=XUK3P6OWTGEJMVLR5B3ZR667WEMCEYIAIVW3PTE6ZWTREKSNBTBA, seg_pos={0 0}, upload=79096]: Done uploading piece. err: ecclient: upload failed (node:12btBm53zb1MCTy6b7CUeujUruMR9nxBoRgqhHmG5ffPf7rL4F, address:45.91.116.5:4324): protocol: write tcp 188.166.225.170:41676->45.91.116.5:4324: use of closed network connection; write tcp 188.166.225.170:41676->45.91.116.5:4324: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41676->45.91.116.5:4324: use of closed network connection Dec 9 10:10:18.740879748 (90h41m18.611962793s) [noise=true , address=45.91.116.11:4140 , piece_id=UJVOUUVBOSA4I4YZJIAPP2QGWBUV322ZSSWINU52U5RIVJ4CEZXA, seg_pos={0 0}, upload=79096]: Done uploading piece. err: ecclient: upload failed (node:14DNa2ysTGtfy3Zz72Dq7S3NSGXqnTHZRkuXorDvuuBw1abwMT, address:45.91.116.11:4140): protocol: write tcp 188.166.225.170:34050->45.91.116.11:4140: use of closed network connection; write tcp 188.166.225.170:34050->45.91.116.11:4140: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34050->45.91.116.11:4140: use of closed network connection Dec 9 10:10:18.740962677 (90h41m18.612045712s) [noise=true , address=45.91.116.9:4264 , piece_id=7UODMOFUXQZ3KYCY4RFA7OUVYXQTWLKNSWR3QIVMIT5KHEQMGT2Q, seg_pos={0 0}, upload=79096]: Done uploading piece. err: ecclient: upload failed (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, address:45.91.116.9:4264): protocol: write tcp 188.166.225.170:50170->45.91.116.9:4264: use of closed network connection; write tcp 188.166.225.170:50170->45.91.116.9:4264: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50170->45.91.116.9:4264: use of closed network connection Dec 9 10:10:18.740989428 (90h41m18.612072463s) [seg_pos={0 0}, upload=79096]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Dec 9 10:10:18.978016295 (90h41m18.84909934s) [upload=79096]: Done uploading segments... Dec 9 10:10:18.978064968 (90h41m18.849148013s) [upload=79096]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 11.203815864s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 5.830889382s trace: https://jaeger.dev.storj.io/trace/47335bbbc9460f72 FAIL (2025-12-09 10:29:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 10:27:57.242720755 (90h58m57.1138038s) [upload=79359]: Starting upload Dec 9 10:27:57.242779226 (90h58m57.113862261s) [upload=79359]: Done starting upload Dec 9 10:27:57.242849329 (90h58m57.113932365s) [upload=79359]: Uploading segments... Dec 9 10:27:57.242961895 (90h58m57.11404493s) [upload=79359]: Got next segment. Inline: false Dec 9 10:27:57.242992913 (90h58m57.114075948s) [upload=79359]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 10:27:57.242997352 (90h58m57.114080397s) [upload=79359]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 10:27:57.490478536 (90h58m57.361561581s) [seg_pos={0 0}, upload=79359]: Begin upload segment... Dec 9 10:27:57.490533320 (90h58m57.361616365s) [seg_pos={0 0}, upload=79359]: Done begin upload segment. Dec 9 10:27:57.490560913 (90h58m57.361643948s) [noise=true , address=145.79.227.22:4511 , piece_id=R5QAVASG7T7UEXDTFXZOK5BPGO7LJARFI7A77MASBGBQJ4DBP4MA, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.490634854 (90h58m57.361717889s) [noise=true , address=145.79.227.40:4051 , piece_id=GBINALPVGCPMYGIYXJOXKZFKQK7ERXVAWTEZ6XMPEQMVTRMRYRWQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.491225929 (90h58m57.362308994s) [noise=true , address=145.79.227.38:4977 , piece_id=5TTHLFLJJXTFEHA2CSV4DCCOMKT5OH7TQPR63TTB57S7O6L4Y73A, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.491642873 (90h58m57.362725918s) [noise=true , address=145.79.227.39:4014 , piece_id=EW67DYNN54WK4Y7YP7HAZUN227LYDM3LAGCFH7OBET7YIEAUTF3A, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.492140761 (90h58m57.363223796s) [noise=true , address=45.91.116.8:4047 , piece_id=XOO3P7YG5DW5SYP4OVSAGLKCZ6W3KSVXIUK24GDYOXX2H3HMS5GQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.492602540 (90h58m57.363685585s) [noise=true , address=145.79.227.20:4439 , piece_id=JE2X5VYQRHXJDWQFUT2KWCE6UWMD6VLUEHMBZSLL3O6FYMGEY7UQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.493057557 (90h58m57.364140603s) [noise=true , address=145.79.227.29:4941 , piece_id=RJUQZUTVF3KJEGZ4MDOP5RR4DEQH5X4OHHXAK3W64CGQOUY4HZWA, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.493071273 (90h58m57.364154318s) [noise=true , address=145.79.227.42:4123 , piece_id=6JHXIPA352SVNLEWKKPMANX5MQRHXFURZR753T74MMFTPPK4LHJA, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.493431789 (90h58m57.364514854s) [noise=true , address=145.79.227.31:4799 , piece_id=AN4NPL2MIY3NEKKR4TZ7STWCUIZVY6MUQ36D5PJROGD2UG722P2A, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.493540356 (90h58m57.364623401s) [noise=true , address=145.79.227.24:4583 , piece_id=XSC6J57R4UDC2AL3I7ZK73FLNQG7EIIHIJGR34WAK3LT3U6UZJAQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.493892386 (90h58m57.364975431s) [noise=true , address=145.79.227.32:4835 , piece_id=HVGEKOXP5H2HHK3IG2VTIGXX6UTQKWBX63VKL5VQGQMFIRXQV6BQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.494083991 (90h58m57.365167027s) [noise=true , address=145.79.227.23:4547 , piece_id=SQG4BHO3ZZUORDL6YOPFQNGCKVF7TDJEONIEJWNYGCCFXDWX6KIQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.494255959 (90h58m57.365339004s) [noise=true , address=145.79.227.25:4618 , piece_id=UMDOX46LAA6YKCG74OXMIUAVVFG7MNE3XHL2PBERUOASXEBYMGSQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.494606567 (90h58m57.365689602s) [noise=true , address=145.79.227.27:4691 , piece_id=7GAKOVJNXFEN5CDKVGK46EIZ6OU4JI247M6Q35WVF4JOWQN72JPA, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.494965920 (90h58m57.366048956s) [noise=true , address=145.79.227.34:4907 , piece_id=PLUYUJWUQXB6BOP6H6YKCRQ4NNOHJ3M4JHNB7FVKQHO75OJU5HYQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.495345513 (90h58m57.366428558s) [noise=true , address=145.79.227.42:4121 , piece_id=ERCQT36FQSO22HNFY5QZOJJMERRASCVHJW5G4OKCEJB32UNYUQAQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:59.193560364 (90h58m59.064643409s) [upload=79359]: Next returned nil segment Dec 9 10:27:59.193597184 (90h58m59.064680219s) [upload=79359]: Waiting for error group managing segments... Dec 9 10:27:59.477920084 (90h58m59.349003119s) [noise=true , address=145.79.227.42:4121 , piece_id=ERCQT36FQSO22HNFY5QZOJJMERRASCVHJW5G4OKCEJB32UNYUQAQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.480823943 (90h58m59.351906988s) [noise=true , address=145.79.227.31:4799 , piece_id=AN4NPL2MIY3NEKKR4TZ7STWCUIZVY6MUQ36D5PJROGD2UG722P2A, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.487502670 (90h58m59.358585715s) [noise=true , address=145.79.227.27:4691 , piece_id=7GAKOVJNXFEN5CDKVGK46EIZ6OU4JI247M6Q35WVF4JOWQN72JPA, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.491620402 (90h58m59.362703437s) [noise=true , address=145.79.227.29:4941 , piece_id=RJUQZUTVF3KJEGZ4MDOP5RR4DEQH5X4OHHXAK3W64CGQOUY4HZWA, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.491964066 (90h58m59.363047101s) [noise=true , address=145.79.227.42:4123 , piece_id=6JHXIPA352SVNLEWKKPMANX5MQRHXFURZR753T74MMFTPPK4LHJA, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.500675514 (90h58m59.371758559s) [noise=true , address=145.79.227.25:4618 , piece_id=UMDOX46LAA6YKCG74OXMIUAVVFG7MNE3XHL2PBERUOASXEBYMGSQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.501834542 (90h58m59.372917577s) [noise=true , address=145.79.227.39:4014 , piece_id=EW67DYNN54WK4Y7YP7HAZUN227LYDM3LAGCFH7OBET7YIEAUTF3A, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.530610947 (90h58m59.401693992s) [noise=true , address=145.79.227.20:4439 , piece_id=JE2X5VYQRHXJDWQFUT2KWCE6UWMD6VLUEHMBZSLL3O6FYMGEY7UQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.554079847 (90h58m59.425162892s) [noise=true , address=145.79.227.22:4511 , piece_id=R5QAVASG7T7UEXDTFXZOK5BPGO7LJARFI7A77MASBGBQJ4DBP4MA, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.558634540 (90h58m59.429717585s) [noise=true , address=145.79.227.24:4583 , piece_id=XSC6J57R4UDC2AL3I7ZK73FLNQG7EIIHIJGR34WAK3LT3U6UZJAQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.571829987 (90h58m59.442913023s) [noise=true , address=145.79.227.32:4835 , piece_id=HVGEKOXP5H2HHK3IG2VTIGXX6UTQKWBX63VKL5VQGQMFIRXQV6BQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.574149323 (90h58m59.445232368s) [noise=true , address=145.79.227.34:4907 , piece_id=PLUYUJWUQXB6BOP6H6YKCRQ4NNOHJ3M4JHNB7FVKQHO75OJU5HYQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:28:26.550754779 (90h59m26.421837824s) [noise=true , address=45.91.116.8:4047 , piece_id=XOO3P7YG5DW5SYP4OVSAGLKCZ6W3KSVXIUK24GDYOXX2H3HMS5GQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: write tcp 188.166.225.170:58652->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:58652->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58652->45.91.116.8:4047: use of closed network connection Dec 9 10:28:26.550891860 (90h59m26.421974905s) [noise=true , address=145.79.227.38:4977 , piece_id=5TTHLFLJJXTFEHA2CSV4DCCOMKT5OH7TQPR63TTB57S7O6L4Y73A, seg_pos={0 0}, upload=79359]: Done uploading piece. err: ecclient: upload failed (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, address:145.79.227.38:4977): protocol: write tcp 188.166.225.170:48912->145.79.227.38:4977: use of closed network connection; write tcp 188.166.225.170:48912->145.79.227.38:4977: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48912->145.79.227.38:4977: use of closed network connection Dec 9 10:28:26.551013401 (90h59m26.422096446s) [noise=true , address=145.79.227.40:4051 , piece_id=GBINALPVGCPMYGIYXJOXKZFKQK7ERXVAWTEZ6XMPEQMVTRMRYRWQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: write tcp 188.166.225.170:49228->145.79.227.40:4051: use of closed network connection; write tcp 188.166.225.170:49228->145.79.227.40:4051: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49228->145.79.227.40:4051: use of closed network connection Dec 9 10:28:26.551092152 (90h59m26.422175187s) [noise=true , address=145.79.227.23:4547 , piece_id=SQG4BHO3ZZUORDL6YOPFQNGCKVF7TDJEONIEJWNYGCCFXDWX6KIQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: write tcp 188.166.225.170:60386->145.79.227.23:4547: use of closed network connection; write tcp 188.166.225.170:60386->145.79.227.23:4547: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60386->145.79.227.23:4547: use of closed network connection Dec 9 10:28:26.551125365 (90h59m26.42220841s) [seg_pos={0 0}, upload=79359]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 10:28:26.791381862 (90h59m26.662464897s) [upload=79359]: Done uploading segments... Dec 9 10:28:26.791420656 (90h59m26.662503691s) [upload=79359]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.830917305s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 1.988438815s trace: https://jaeger.dev.storj.io/trace/6cb2657ad006bedf FAIL (2025-12-09 10:29:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 09:57:44.389846478 (90h28m44.260929523s) [upload=78917]: Starting upload Dec 9 09:57:44.390051578 (90h28m44.261134613s) [upload=78917]: Done starting upload Dec 9 09:57:44.390125740 (90h28m44.261208775s) [upload=78917]: Uploading segments... Dec 9 09:57:44.390224027 (90h28m44.261307072s) [upload=78917]: Got next segment. Inline: true Dec 9 09:57:44.390230800 (90h28m44.261313835s) [upload=78917]: Waiting for error group managing segments... Dec 9 09:57:44.390312034 (90h28m44.261395069s) [upload=78917]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 09:57:44.390318787 (90h28m44.261401822s) [upload=78917]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 9 09:57:44.390323356 (90h28m44.261406391s) [upload=78917]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 09:58:21.934924620 (90h29m21.806007665s) [upload=78917]: Done uploading segments... Dec 9 09:58:21.934974756 (90h29m21.806057801s) [upload=78917]: Upload finished. err: metaclient: context deadline exceeded Duration:* 1.988469655s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 9.141162849s trace: https://jaeger.dev.storj.io/trace/47335bbbc9460f72 FAIL (2025-12-09 10:28:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 10:27:57.242720755 (90h58m57.1138038s) [upload=79359]: Starting upload Dec 9 10:27:57.242779226 (90h58m57.113862261s) [upload=79359]: Done starting upload Dec 9 10:27:57.242849329 (90h58m57.113932365s) [upload=79359]: Uploading segments... Dec 9 10:27:57.242961895 (90h58m57.11404493s) [upload=79359]: Got next segment. Inline: false Dec 9 10:27:57.242992913 (90h58m57.114075948s) [upload=79359]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 10:27:57.242997352 (90h58m57.114080397s) [upload=79359]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 10:27:57.490478536 (90h58m57.361561581s) [seg_pos={0 0}, upload=79359]: Begin upload segment... Dec 9 10:27:57.490533320 (90h58m57.361616365s) [seg_pos={0 0}, upload=79359]: Done begin upload segment. Dec 9 10:27:57.490560913 (90h58m57.361643948s) [noise=true , address=145.79.227.22:4511 , piece_id=R5QAVASG7T7UEXDTFXZOK5BPGO7LJARFI7A77MASBGBQJ4DBP4MA, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.490634854 (90h58m57.361717889s) [noise=true , address=145.79.227.40:4051 , piece_id=GBINALPVGCPMYGIYXJOXKZFKQK7ERXVAWTEZ6XMPEQMVTRMRYRWQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.491225929 (90h58m57.362308994s) [noise=true , address=145.79.227.38:4977 , piece_id=5TTHLFLJJXTFEHA2CSV4DCCOMKT5OH7TQPR63TTB57S7O6L4Y73A, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.491642873 (90h58m57.362725918s) [noise=true , address=145.79.227.39:4014 , piece_id=EW67DYNN54WK4Y7YP7HAZUN227LYDM3LAGCFH7OBET7YIEAUTF3A, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.492140761 (90h58m57.363223796s) [noise=true , address=45.91.116.8:4047 , piece_id=XOO3P7YG5DW5SYP4OVSAGLKCZ6W3KSVXIUK24GDYOXX2H3HMS5GQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.492602540 (90h58m57.363685585s) [noise=true , address=145.79.227.20:4439 , piece_id=JE2X5VYQRHXJDWQFUT2KWCE6UWMD6VLUEHMBZSLL3O6FYMGEY7UQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.493057557 (90h58m57.364140603s) [noise=true , address=145.79.227.29:4941 , piece_id=RJUQZUTVF3KJEGZ4MDOP5RR4DEQH5X4OHHXAK3W64CGQOUY4HZWA, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.493071273 (90h58m57.364154318s) [noise=true , address=145.79.227.42:4123 , piece_id=6JHXIPA352SVNLEWKKPMANX5MQRHXFURZR753T74MMFTPPK4LHJA, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.493431789 (90h58m57.364514854s) [noise=true , address=145.79.227.31:4799 , piece_id=AN4NPL2MIY3NEKKR4TZ7STWCUIZVY6MUQ36D5PJROGD2UG722P2A, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.493540356 (90h58m57.364623401s) [noise=true , address=145.79.227.24:4583 , piece_id=XSC6J57R4UDC2AL3I7ZK73FLNQG7EIIHIJGR34WAK3LT3U6UZJAQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.493892386 (90h58m57.364975431s) [noise=true , address=145.79.227.32:4835 , piece_id=HVGEKOXP5H2HHK3IG2VTIGXX6UTQKWBX63VKL5VQGQMFIRXQV6BQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.494083991 (90h58m57.365167027s) [noise=true , address=145.79.227.23:4547 , piece_id=SQG4BHO3ZZUORDL6YOPFQNGCKVF7TDJEONIEJWNYGCCFXDWX6KIQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.494255959 (90h58m57.365339004s) [noise=true , address=145.79.227.25:4618 , piece_id=UMDOX46LAA6YKCG74OXMIUAVVFG7MNE3XHL2PBERUOASXEBYMGSQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.494606567 (90h58m57.365689602s) [noise=true , address=145.79.227.27:4691 , piece_id=7GAKOVJNXFEN5CDKVGK46EIZ6OU4JI247M6Q35WVF4JOWQN72JPA, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.494965920 (90h58m57.366048956s) [noise=true , address=145.79.227.34:4907 , piece_id=PLUYUJWUQXB6BOP6H6YKCRQ4NNOHJ3M4JHNB7FVKQHO75OJU5HYQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.495345513 (90h58m57.366428558s) [noise=true , address=145.79.227.42:4121 , piece_id=ERCQT36FQSO22HNFY5QZOJJMERRASCVHJW5G4OKCEJB32UNYUQAQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:59.193560364 (90h58m59.064643409s) [upload=79359]: Next returned nil segment Dec 9 10:27:59.193597184 (90h58m59.064680219s) [upload=79359]: Waiting for error group managing segments... Dec 9 10:27:59.477920084 (90h58m59.349003119s) [noise=true , address=145.79.227.42:4121 , piece_id=ERCQT36FQSO22HNFY5QZOJJMERRASCVHJW5G4OKCEJB32UNYUQAQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.480823943 (90h58m59.351906988s) [noise=true , address=145.79.227.31:4799 , piece_id=AN4NPL2MIY3NEKKR4TZ7STWCUIZVY6MUQ36D5PJROGD2UG722P2A, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.487502670 (90h58m59.358585715s) [noise=true , address=145.79.227.27:4691 , piece_id=7GAKOVJNXFEN5CDKVGK46EIZ6OU4JI247M6Q35WVF4JOWQN72JPA, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.491620402 (90h58m59.362703437s) [noise=true , address=145.79.227.29:4941 , piece_id=RJUQZUTVF3KJEGZ4MDOP5RR4DEQH5X4OHHXAK3W64CGQOUY4HZWA, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.491964066 (90h58m59.363047101s) [noise=true , address=145.79.227.42:4123 , piece_id=6JHXIPA352SVNLEWKKPMANX5MQRHXFURZR753T74MMFTPPK4LHJA, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.500675514 (90h58m59.371758559s) [noise=true , address=145.79.227.25:4618 , piece_id=UMDOX46LAA6YKCG74OXMIUAVVFG7MNE3XHL2PBERUOASXEBYMGSQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.501834542 (90h58m59.372917577s) [noise=true , address=145.79.227.39:4014 , piece_id=EW67DYNN54WK4Y7YP7HAZUN227LYDM3LAGCFH7OBET7YIEAUTF3A, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.530610947 (90h58m59.401693992s) [noise=true , address=145.79.227.20:4439 , piece_id=JE2X5VYQRHXJDWQFUT2KWCE6UWMD6VLUEHMBZSLL3O6FYMGEY7UQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.554079847 (90h58m59.425162892s) [noise=true , address=145.79.227.22:4511 , piece_id=R5QAVASG7T7UEXDTFXZOK5BPGO7LJARFI7A77MASBGBQJ4DBP4MA, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.558634540 (90h58m59.429717585s) [noise=true , address=145.79.227.24:4583 , piece_id=XSC6J57R4UDC2AL3I7ZK73FLNQG7EIIHIJGR34WAK3LT3U6UZJAQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.571829987 (90h58m59.442913023s) [noise=true , address=145.79.227.32:4835 , piece_id=HVGEKOXP5H2HHK3IG2VTIGXX6UTQKWBX63VKL5VQGQMFIRXQV6BQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.574149323 (90h58m59.445232368s) [noise=true , address=145.79.227.34:4907 , piece_id=PLUYUJWUQXB6BOP6H6YKCRQ4NNOHJ3M4JHNB7FVKQHO75OJU5HYQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:28:26.550754779 (90h59m26.421837824s) [noise=true , address=45.91.116.8:4047 , piece_id=XOO3P7YG5DW5SYP4OVSAGLKCZ6W3KSVXIUK24GDYOXX2H3HMS5GQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: write tcp 188.166.225.170:58652->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:58652->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58652->45.91.116.8:4047: use of closed network connection Dec 9 10:28:26.550891860 (90h59m26.421974905s) [noise=true , address=145.79.227.38:4977 , piece_id=5TTHLFLJJXTFEHA2CSV4DCCOMKT5OH7TQPR63TTB57S7O6L4Y73A, seg_pos={0 0}, upload=79359]: Done uploading piece. err: ecclient: upload failed (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, address:145.79.227.38:4977): protocol: write tcp 188.166.225.170:48912->145.79.227.38:4977: use of closed network connection; write tcp 188.166.225.170:48912->145.79.227.38:4977: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48912->145.79.227.38:4977: use of closed network connection Dec 9 10:28:26.551013401 (90h59m26.422096446s) [noise=true , address=145.79.227.40:4051 , piece_id=GBINALPVGCPMYGIYXJOXKZFKQK7ERXVAWTEZ6XMPEQMVTRMRYRWQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: write tcp 188.166.225.170:49228->145.79.227.40:4051: use of closed network connection; write tcp 188.166.225.170:49228->145.79.227.40:4051: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49228->145.79.227.40:4051: use of closed network connection Dec 9 10:28:26.551092152 (90h59m26.422175187s) [noise=true , address=145.79.227.23:4547 , piece_id=SQG4BHO3ZZUORDL6YOPFQNGCKVF7TDJEONIEJWNYGCCFXDWX6KIQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: write tcp 188.166.225.170:60386->145.79.227.23:4547: use of closed network connection; write tcp 188.166.225.170:60386->145.79.227.23:4547: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60386->145.79.227.23:4547: use of closed network connection Dec 9 10:28:26.551125365 (90h59m26.42220841s) [seg_pos={0 0}, upload=79359]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 10:28:26.791381862 (90h59m26.662464897s) [upload=79359]: Done uploading segments... Dec 9 10:28:26.791420656 (90h59m26.662503691s) [upload=79359]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 9.141215257s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..............................X..X..............X...........] duration: 3.568485622s trace: https://jaeger.dev.storj.io/trace/9e23c2a25a964f2 FAIL (2025-12-09 10:28:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.568506572s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 10.737283021s trace: https://jaeger.dev.storj.io/trace/781235978226387b FAIL (2025-12-09 10:28:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ... Dec 9 10:09:19.659155420 (90h40m19.530238455s) [noise=true , address=77.90.31.19:4870 , piece_id=P6W2QVXDH57FK755LD67JDSC23BZPHVQHBVWZLJDC457JPG3ZQLQ, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.659573456 (90h40m19.530656491s) [noise=true , address=77.90.31.15:4703 , piece_id=3FIJGK2XHNRTDF4Q3DN3AX2P42AUR5CNUI5QPXQ256MPMCQB5QRQ, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.660082366 (90h40m19.531165401s) [noise=true , address=77.90.31.16:4760 , piece_id=LJPEX76BQ4ICDJ3I4AWHE7UE7D2WIZCVGQS6XU6ZZ2II2NBYDKHQ, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.660519568 (90h40m19.531602613s) [noise=true , address=45.91.116.9:4264 , piece_id=7UODMOFUXQZ3KYCY4RFA7OUVYXQTWLKNSWR3QIVMIT5KHEQMGT2Q, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.660971178 (90h40m19.532054223s) [noise=true , address=77.90.31.21:4882 , piece_id=CJSACAOIT5IXGJXTL4YDNRUK4H5KD6QCFE6VINQGMQJ2TT2TAU5A, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.661377291 (90h40m19.532460326s) [noise=true , address=77.90.31.13:4639 , piece_id=I2GHDHJTAHCDYP7BAG6QAHGPAFIKQZOLUPD5V7AICS4CATG3XNDA, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.661793374 (90h40m19.532876409s) [noise=true , address=45.91.116.11:4140 , piece_id=UJVOUUVBOSA4I4YZJIAPP2QGWBUV322ZSSWINU52U5RIVJ4CEZXA, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.661991682 (90h40m19.533074717s) [noise=true , address=77.90.31.14:4684 , piece_id=ARECBM73CDZDYSVWPLH2XCQC6A45SGW57TPWX5URGCAUW355COOA, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.662431088 (90h40m19.533514133s) [noise=true , address=77.90.31.12:4615 , piece_id=BLNHDHE34WZMROFSL5SNTHOITFWI6JOR7VKQBYZB77LTINEGLS5Q, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.662848232 (90h40m19.533931277s) [noise=true , address=145.79.227.13:4824 , piece_id=M6P4NFRFOU5LGE2HRGUXRUZXJ2HJYA6BWC3TW4MWUWIYYSUTR76Q, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.663338626 (90h40m19.534421671s) [noise=true , address=145.79.227.24:4587 , piece_id=EVUNUBMSMMIERRCDR67LDFSGY6S2EF6IRV6EGEX4IOFX4PCTYIDQ, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.664502282 (90h40m19.535585327s) [noise=true , address=45.91.116.12:4254 , piece_id=WOGMNSLO5FOGUSOWXRTLPZNASJB2ZMPUEKC4WRQNROTKQ74JV6KQ, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.666711839 (90h40m19.537794884s) [noise=true , address=77.90.31.10:4529 , piece_id=YIT3CCLUOLSQ3NGOUVF7PYVV6FIDT7WKTRQVU4VTU4LRMPYL4H3A, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.669766776 (90h40m19.540849821s) [noise=true , address=77.90.31.17:4783 , piece_id=LSTC3TY566OQBGL6ZXBLL5IFNPORZRWOEGFKGT3EXYOB3IPWI6JQ, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.677090573 (90h40m19.548173618s) [noise=true , address=45.91.116.13:4173 , piece_id=7PIVANLRS7SL6ERBMUC4LT4P3C3JRHRY6Z4I7336EVMXC3IHFVDQ, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:21.186498404 (90h40m21.057581499s) [upload=79096]: Next returned nil segment Dec 9 10:09:21.186518313 (90h40m21.057601348s) [upload=79096]: Waiting for error group managing segments... Dec 9 10:09:21.574704328 (90h40m21.445787374s) [noise=true , address=77.90.31.18:4829 , piece_id=CDRLGWU6VF6TU6E4E5B6R7NOF35WOQOA2TSDPASZP424U23QORNA, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:21.587662915 (90h40m21.45874595s) [noise=true , address=77.90.31.10:4529 , piece_id=YIT3CCLUOLSQ3NGOUVF7PYVV6FIDT7WKTRQVU4VTU4LRMPYL4H3A, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:21.624695792 (90h40m21.495778827s) [noise=true , address=77.90.31.15:4703 , piece_id=3FIJGK2XHNRTDF4Q3DN3AX2P42AUR5CNUI5QPXQ256MPMCQB5QRQ, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:21.659658729 (90h40m21.530741774s) [noise=true , address=77.90.31.12:4615 , piece_id=BLNHDHE34WZMROFSL5SNTHOITFWI6JOR7VKQBYZB77LTINEGLS5Q, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:21.667474853 (90h40m21.538557898s) [noise=true , address=145.79.227.24:4587 , piece_id=EVUNUBMSMMIERRCDR67LDFSGY6S2EF6IRV6EGEX4IOFX4PCTYIDQ, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:21.669766096 (90h40m21.540849131s) [noise=true , address=77.90.31.14:4684 , piece_id=ARECBM73CDZDYSVWPLH2XCQC6A45SGW57TPWX5URGCAUW355COOA, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:21.670505664 (90h40m21.541588709s) [noise=true , address=77.90.31.17:4783 , piece_id=LSTC3TY566OQBGL6ZXBLL5IFNPORZRWOEGFKGT3EXYOB3IPWI6JQ, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:21.680637627 (90h40m21.551720662s) [noise=true , address=77.90.31.13:4639 , piece_id=I2GHDHJTAHCDYP7BAG6QAHGPAFIKQZOLUPD5V7AICS4CATG3XNDA, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:21.685855022 (90h40m21.556938057s) [noise=true , address=77.90.31.21:4882 , piece_id=CJSACAOIT5IXGJXTL4YDNRUK4H5KD6QCFE6VINQGMQJ2TT2TAU5A, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:21.741439516 (90h40m21.612522561s) [noise=true , address=77.90.31.19:4870 , piece_id=P6W2QVXDH57FK755LD67JDSC23BZPHVQHBVWZLJDC457JPG3ZQLQ, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:22.306045351 (90h40m22.177128396s) [noise=true , address=77.90.31.16:4760 , piece_id=LJPEX76BQ4ICDJ3I4AWHE7UE7D2WIZCVGQS6XU6ZZ2II2NBYDKHQ, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:22.498367742 (90h40m22.369450787s) [noise=true , address=145.79.227.13:4824 , piece_id=M6P4NFRFOU5LGE2HRGUXRUZXJ2HJYA6BWC3TW4MWUWIYYSUTR76Q, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:10:18.740403422 (90h41m18.611486467s) [noise=true , address=45.91.116.13:4173 , piece_id=7PIVANLRS7SL6ERBMUC4LT4P3C3JRHRY6Z4I7336EVMXC3IHFVDQ, seg_pos={0 0}, upload=79096]: Done uploading piece. err: ecclient: upload failed (node:1J3HZY416Dm9F5zRXtoDdt8s8tGoU6GpkwS14CXJUzd3zpcYa1, address:45.91.116.13:4173): protocol: write tcp 188.166.225.170:53476->45.91.116.13:4173: use of closed network connection; write tcp 188.166.225.170:53476->45.91.116.13:4173: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53476->45.91.116.13:4173: use of closed network connection Dec 9 10:10:18.740531145 (90h41m18.61161419s) [noise=true , address=64.49.13.169:4710 , piece_id=XEKNBKQ4WTECDQD6YYP5RUU2KOSFPRGKXRWDN5THYLE4HXO65BEA, seg_pos={0 0}, upload=79096]: Done uploading piece. err: ecclient: upload failed (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf, address:64.49.13.169:4710): protocol: write tcp 188.166.225.170:37916->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:37916->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37916->64.49.13.169:4710: use of closed network connection Dec 9 10:10:18.740657375 (90h41m18.61174042s) [noise=true , address=45.91.116.12:4254 , piece_id=WOGMNSLO5FOGUSOWXRTLPZNASJB2ZMPUEKC4WRQNROTKQ74JV6KQ, seg_pos={0 0}, upload=79096]: Done uploading piece. err: ecclient: upload failed (node:12wAKmaLHwjBKN8UozGFjGvNMYG57fCxTH8wuDZDRnMdyGBRsTv, address:45.91.116.12:4254): protocol: write tcp 188.166.225.170:35550->45.91.116.12:4254: use of closed network connection; write tcp 188.166.225.170:35550->45.91.116.12:4254: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35550->45.91.116.12:4254: use of closed network connection Dec 9 10:10:18.740778986 (90h41m18.611862021s) [noise=true , address=45.91.116.5:4324 , piece_id=XUK3P6OWTGEJMVLR5B3ZR667WEMCEYIAIVW3PTE6ZWTREKSNBTBA, seg_pos={0 0}, upload=79096]: Done uploading piece. err: ecclient: upload failed (node:12btBm53zb1MCTy6b7CUeujUruMR9nxBoRgqhHmG5ffPf7rL4F, address:45.91.116.5:4324): protocol: write tcp 188.166.225.170:41676->45.91.116.5:4324: use of closed network connection; write tcp 188.166.225.170:41676->45.91.116.5:4324: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41676->45.91.116.5:4324: use of closed network connection Dec 9 10:10:18.740879748 (90h41m18.611962793s) [noise=true , address=45.91.116.11:4140 , piece_id=UJVOUUVBOSA4I4YZJIAPP2QGWBUV322ZSSWINU52U5RIVJ4CEZXA, seg_pos={0 0}, upload=79096]: Done uploading piece. err: ecclient: upload failed (node:14DNa2ysTGtfy3Zz72Dq7S3NSGXqnTHZRkuXorDvuuBw1abwMT, address:45.91.116.11:4140): protocol: write tcp 188.166.225.170:34050->45.91.116.11:4140: use of closed network connection; write tcp 188.166.225.170:34050->45.91.116.11:4140: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34050->45.91.116.11:4140: use of closed network connection Dec 9 10:10:18.740962677 (90h41m18.612045712s) [noise=true , address=45.91.116.9:4264 , piece_id=7UODMOFUXQZ3KYCY4RFA7OUVYXQTWLKNSWR3QIVMIT5KHEQMGT2Q, seg_pos={0 0}, upload=79096]: Done uploading piece. err: ecclient: upload failed (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, address:45.91.116.9:4264): protocol: write tcp 188.166.225.170:50170->45.91.116.9:4264: use of closed network connection; write tcp 188.166.225.170:50170->45.91.116.9:4264: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50170->45.91.116.9:4264: use of closed network connection Dec 9 10:10:18.740989428 (90h41m18.612072463s) [seg_pos={0 0}, upload=79096]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Dec 9 10:10:18.978016295 (90h41m18.84909934s) [upload=79096]: Done uploading segments... Dec 9 10:10:18.978064968 (90h41m18.849148013s) [upload=79096]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 10.737313158s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.......X........X......X..................X.................] duration: 5.932575027s trace: https://jaeger.dev.storj.io/trace/5b4024e5e8cc17e6 FAIL (2025-12-09 10:28:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.932602568s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.242996964s trace: https://jaeger.dev.storj.io/trace/47335bbbc9460f72 FAIL (2025-12-09 10:28:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 10:27:57.242720755 (90h58m57.1138038s) [upload=79359]: Starting upload Dec 9 10:27:57.242779226 (90h58m57.113862261s) [upload=79359]: Done starting upload Dec 9 10:27:57.242849329 (90h58m57.113932365s) [upload=79359]: Uploading segments... Dec 9 10:27:57.242961895 (90h58m57.11404493s) [upload=79359]: Got next segment. Inline: false Dec 9 10:27:57.242992913 (90h58m57.114075948s) [upload=79359]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 10:27:57.242997352 (90h58m57.114080397s) [upload=79359]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 10:27:57.490478536 (90h58m57.361561581s) [seg_pos={0 0}, upload=79359]: Begin upload segment... Dec 9 10:27:57.490533320 (90h58m57.361616365s) [seg_pos={0 0}, upload=79359]: Done begin upload segment. Dec 9 10:27:57.490560913 (90h58m57.361643948s) [noise=true , address=145.79.227.22:4511 , piece_id=R5QAVASG7T7UEXDTFXZOK5BPGO7LJARFI7A77MASBGBQJ4DBP4MA, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.490634854 (90h58m57.361717889s) [noise=true , address=145.79.227.40:4051 , piece_id=GBINALPVGCPMYGIYXJOXKZFKQK7ERXVAWTEZ6XMPEQMVTRMRYRWQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.491225929 (90h58m57.362308994s) [noise=true , address=145.79.227.38:4977 , piece_id=5TTHLFLJJXTFEHA2CSV4DCCOMKT5OH7TQPR63TTB57S7O6L4Y73A, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.491642873 (90h58m57.362725918s) [noise=true , address=145.79.227.39:4014 , piece_id=EW67DYNN54WK4Y7YP7HAZUN227LYDM3LAGCFH7OBET7YIEAUTF3A, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.492140761 (90h58m57.363223796s) [noise=true , address=45.91.116.8:4047 , piece_id=XOO3P7YG5DW5SYP4OVSAGLKCZ6W3KSVXIUK24GDYOXX2H3HMS5GQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.492602540 (90h58m57.363685585s) [noise=true , address=145.79.227.20:4439 , piece_id=JE2X5VYQRHXJDWQFUT2KWCE6UWMD6VLUEHMBZSLL3O6FYMGEY7UQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.493057557 (90h58m57.364140603s) [noise=true , address=145.79.227.29:4941 , piece_id=RJUQZUTVF3KJEGZ4MDOP5RR4DEQH5X4OHHXAK3W64CGQOUY4HZWA, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.493071273 (90h58m57.364154318s) [noise=true , address=145.79.227.42:4123 , piece_id=6JHXIPA352SVNLEWKKPMANX5MQRHXFURZR753T74MMFTPPK4LHJA, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.493431789 (90h58m57.364514854s) [noise=true , address=145.79.227.31:4799 , piece_id=AN4NPL2MIY3NEKKR4TZ7STWCUIZVY6MUQ36D5PJROGD2UG722P2A, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.493540356 (90h58m57.364623401s) [noise=true , address=145.79.227.24:4583 , piece_id=XSC6J57R4UDC2AL3I7ZK73FLNQG7EIIHIJGR34WAK3LT3U6UZJAQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.493892386 (90h58m57.364975431s) [noise=true , address=145.79.227.32:4835 , piece_id=HVGEKOXP5H2HHK3IG2VTIGXX6UTQKWBX63VKL5VQGQMFIRXQV6BQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.494083991 (90h58m57.365167027s) [noise=true , address=145.79.227.23:4547 , piece_id=SQG4BHO3ZZUORDL6YOPFQNGCKVF7TDJEONIEJWNYGCCFXDWX6KIQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.494255959 (90h58m57.365339004s) [noise=true , address=145.79.227.25:4618 , piece_id=UMDOX46LAA6YKCG74OXMIUAVVFG7MNE3XHL2PBERUOASXEBYMGSQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.494606567 (90h58m57.365689602s) [noise=true , address=145.79.227.27:4691 , piece_id=7GAKOVJNXFEN5CDKVGK46EIZ6OU4JI247M6Q35WVF4JOWQN72JPA, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.494965920 (90h58m57.366048956s) [noise=true , address=145.79.227.34:4907 , piece_id=PLUYUJWUQXB6BOP6H6YKCRQ4NNOHJ3M4JHNB7FVKQHO75OJU5HYQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:57.495345513 (90h58m57.366428558s) [noise=true , address=145.79.227.42:4121 , piece_id=ERCQT36FQSO22HNFY5QZOJJMERRASCVHJW5G4OKCEJB32UNYUQAQ, seg_pos={0 0}, upload=79359]: Uploading piece... Dec 9 10:27:59.193560364 (90h58m59.064643409s) [upload=79359]: Next returned nil segment Dec 9 10:27:59.193597184 (90h58m59.064680219s) [upload=79359]: Waiting for error group managing segments... Dec 9 10:27:59.477920084 (90h58m59.349003119s) [noise=true , address=145.79.227.42:4121 , piece_id=ERCQT36FQSO22HNFY5QZOJJMERRASCVHJW5G4OKCEJB32UNYUQAQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.480823943 (90h58m59.351906988s) [noise=true , address=145.79.227.31:4799 , piece_id=AN4NPL2MIY3NEKKR4TZ7STWCUIZVY6MUQ36D5PJROGD2UG722P2A, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.487502670 (90h58m59.358585715s) [noise=true , address=145.79.227.27:4691 , piece_id=7GAKOVJNXFEN5CDKVGK46EIZ6OU4JI247M6Q35WVF4JOWQN72JPA, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.491620402 (90h58m59.362703437s) [noise=true , address=145.79.227.29:4941 , piece_id=RJUQZUTVF3KJEGZ4MDOP5RR4DEQH5X4OHHXAK3W64CGQOUY4HZWA, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.491964066 (90h58m59.363047101s) [noise=true , address=145.79.227.42:4123 , piece_id=6JHXIPA352SVNLEWKKPMANX5MQRHXFURZR753T74MMFTPPK4LHJA, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.500675514 (90h58m59.371758559s) [noise=true , address=145.79.227.25:4618 , piece_id=UMDOX46LAA6YKCG74OXMIUAVVFG7MNE3XHL2PBERUOASXEBYMGSQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.501834542 (90h58m59.372917577s) [noise=true , address=145.79.227.39:4014 , piece_id=EW67DYNN54WK4Y7YP7HAZUN227LYDM3LAGCFH7OBET7YIEAUTF3A, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.530610947 (90h58m59.401693992s) [noise=true , address=145.79.227.20:4439 , piece_id=JE2X5VYQRHXJDWQFUT2KWCE6UWMD6VLUEHMBZSLL3O6FYMGEY7UQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.554079847 (90h58m59.425162892s) [noise=true , address=145.79.227.22:4511 , piece_id=R5QAVASG7T7UEXDTFXZOK5BPGO7LJARFI7A77MASBGBQJ4DBP4MA, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.558634540 (90h58m59.429717585s) [noise=true , address=145.79.227.24:4583 , piece_id=XSC6J57R4UDC2AL3I7ZK73FLNQG7EIIHIJGR34WAK3LT3U6UZJAQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.571829987 (90h58m59.442913023s) [noise=true , address=145.79.227.32:4835 , piece_id=HVGEKOXP5H2HHK3IG2VTIGXX6UTQKWBX63VKL5VQGQMFIRXQV6BQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:27:59.574149323 (90h58m59.445232368s) [noise=true , address=145.79.227.34:4907 , piece_id=PLUYUJWUQXB6BOP6H6YKCRQ4NNOHJ3M4JHNB7FVKQHO75OJU5HYQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: <nil> Dec 9 10:28:26.550754779 (90h59m26.421837824s) [noise=true , address=45.91.116.8:4047 , piece_id=XOO3P7YG5DW5SYP4OVSAGLKCZ6W3KSVXIUK24GDYOXX2H3HMS5GQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: write tcp 188.166.225.170:58652->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:58652->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58652->45.91.116.8:4047: use of closed network connection Dec 9 10:28:26.550891860 (90h59m26.421974905s) [noise=true , address=145.79.227.38:4977 , piece_id=5TTHLFLJJXTFEHA2CSV4DCCOMKT5OH7TQPR63TTB57S7O6L4Y73A, seg_pos={0 0}, upload=79359]: Done uploading piece. err: ecclient: upload failed (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, address:145.79.227.38:4977): protocol: write tcp 188.166.225.170:48912->145.79.227.38:4977: use of closed network connection; write tcp 188.166.225.170:48912->145.79.227.38:4977: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48912->145.79.227.38:4977: use of closed network connection Dec 9 10:28:26.551013401 (90h59m26.422096446s) [noise=true , address=145.79.227.40:4051 , piece_id=GBINALPVGCPMYGIYXJOXKZFKQK7ERXVAWTEZ6XMPEQMVTRMRYRWQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: write tcp 188.166.225.170:49228->145.79.227.40:4051: use of closed network connection; write tcp 188.166.225.170:49228->145.79.227.40:4051: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49228->145.79.227.40:4051: use of closed network connection Dec 9 10:28:26.551092152 (90h59m26.422175187s) [noise=true , address=145.79.227.23:4547 , piece_id=SQG4BHO3ZZUORDL6YOPFQNGCKVF7TDJEONIEJWNYGCCFXDWX6KIQ, seg_pos={0 0}, upload=79359]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: write tcp 188.166.225.170:60386->145.79.227.23:4547: use of closed network connection; write tcp 188.166.225.170:60386->145.79.227.23:4547: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60386->145.79.227.23:4547: use of closed network connection Dec 9 10:28:26.551125365 (90h59m26.42220841s) [seg_pos={0 0}, upload=79359]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 10:28:26.791381862 (90h59m26.662464897s) [upload=79359]: Done uploading segments... Dec 9 10:28:26.791420656 (90h59m26.662503691s) [upload=79359]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.243017092s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 1.977006355s trace: https://jaeger.dev.storj.io/trace/6cb2657ad006bedf FAIL (2025-12-09 10:28:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 09:57:44.389846478 (90h28m44.260929523s) [upload=78917]: Starting upload Dec 9 09:57:44.390051578 (90h28m44.261134613s) [upload=78917]: Done starting upload Dec 9 09:57:44.390125740 (90h28m44.261208775s) [upload=78917]: Uploading segments... Dec 9 09:57:44.390224027 (90h28m44.261307072s) [upload=78917]: Got next segment. Inline: true Dec 9 09:57:44.390230800 (90h28m44.261313835s) [upload=78917]: Waiting for error group managing segments... Dec 9 09:57:44.390312034 (90h28m44.261395069s) [upload=78917]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 09:57:44.390318787 (90h28m44.261401822s) [upload=78917]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 9 09:57:44.390323356 (90h28m44.261406391s) [upload=78917]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 09:58:21.934924620 (90h29m21.806007665s) [upload=78917]: Done uploading segments... Dec 9 09:58:21.934974756 (90h29m21.806057801s) [upload=78917]: Upload finished. err: metaclient: context deadline exceeded Duration:* 1.977108649s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [........X........X......X..................X................] duration: 5.581352966s trace: https://jaeger.dev.storj.io/trace/5b4024e5e8cc17e6 FAIL (2025-12-09 10:27:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.581384006s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...............................X..X..............X..........] duration: 3.367971446s trace: https://jaeger.dev.storj.io/trace/9e23c2a25a964f2 FAIL (2025-12-09 10:27:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.36799998s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [................................X..X..............X.........] duration: 3.545656998s trace: https://jaeger.dev.storj.io/trace/9e23c2a25a964f2 FAIL (2025-12-09 10:27:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.545848733s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 12.654075465s trace: https://jaeger.dev.storj.io/trace/781235978226387b FAIL (2025-12-09 10:27:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ... Dec 9 10:09:19.659155420 (90h40m19.530238455s) [noise=true , address=77.90.31.19:4870 , piece_id=P6W2QVXDH57FK755LD67JDSC23BZPHVQHBVWZLJDC457JPG3ZQLQ, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.659573456 (90h40m19.530656491s) [noise=true , address=77.90.31.15:4703 , piece_id=3FIJGK2XHNRTDF4Q3DN3AX2P42AUR5CNUI5QPXQ256MPMCQB5QRQ, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.660082366 (90h40m19.531165401s) [noise=true , address=77.90.31.16:4760 , piece_id=LJPEX76BQ4ICDJ3I4AWHE7UE7D2WIZCVGQS6XU6ZZ2II2NBYDKHQ, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.660519568 (90h40m19.531602613s) [noise=true , address=45.91.116.9:4264 , piece_id=7UODMOFUXQZ3KYCY4RFA7OUVYXQTWLKNSWR3QIVMIT5KHEQMGT2Q, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.660971178 (90h40m19.532054223s) [noise=true , address=77.90.31.21:4882 , piece_id=CJSACAOIT5IXGJXTL4YDNRUK4H5KD6QCFE6VINQGMQJ2TT2TAU5A, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.661377291 (90h40m19.532460326s) [noise=true , address=77.90.31.13:4639 , piece_id=I2GHDHJTAHCDYP7BAG6QAHGPAFIKQZOLUPD5V7AICS4CATG3XNDA, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.661793374 (90h40m19.532876409s) [noise=true , address=45.91.116.11:4140 , piece_id=UJVOUUVBOSA4I4YZJIAPP2QGWBUV322ZSSWINU52U5RIVJ4CEZXA, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.661991682 (90h40m19.533074717s) [noise=true , address=77.90.31.14:4684 , piece_id=ARECBM73CDZDYSVWPLH2XCQC6A45SGW57TPWX5URGCAUW355COOA, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.662431088 (90h40m19.533514133s) [noise=true , address=77.90.31.12:4615 , piece_id=BLNHDHE34WZMROFSL5SNTHOITFWI6JOR7VKQBYZB77LTINEGLS5Q, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.662848232 (90h40m19.533931277s) [noise=true , address=145.79.227.13:4824 , piece_id=M6P4NFRFOU5LGE2HRGUXRUZXJ2HJYA6BWC3TW4MWUWIYYSUTR76Q, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.663338626 (90h40m19.534421671s) [noise=true , address=145.79.227.24:4587 , piece_id=EVUNUBMSMMIERRCDR67LDFSGY6S2EF6IRV6EGEX4IOFX4PCTYIDQ, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.664502282 (90h40m19.535585327s) [noise=true , address=45.91.116.12:4254 , piece_id=WOGMNSLO5FOGUSOWXRTLPZNASJB2ZMPUEKC4WRQNROTKQ74JV6KQ, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.666711839 (90h40m19.537794884s) [noise=true , address=77.90.31.10:4529 , piece_id=YIT3CCLUOLSQ3NGOUVF7PYVV6FIDT7WKTRQVU4VTU4LRMPYL4H3A, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.669766776 (90h40m19.540849821s) [noise=true , address=77.90.31.17:4783 , piece_id=LSTC3TY566OQBGL6ZXBLL5IFNPORZRWOEGFKGT3EXYOB3IPWI6JQ, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.677090573 (90h40m19.548173618s) [noise=true , address=45.91.116.13:4173 , piece_id=7PIVANLRS7SL6ERBMUC4LT4P3C3JRHRY6Z4I7336EVMXC3IHFVDQ, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:21.186498404 (90h40m21.057581499s) [upload=79096]: Next returned nil segment Dec 9 10:09:21.186518313 (90h40m21.057601348s) [upload=79096]: Waiting for error group managing segments... Dec 9 10:09:21.574704328 (90h40m21.445787374s) [noise=true , address=77.90.31.18:4829 , piece_id=CDRLGWU6VF6TU6E4E5B6R7NOF35WOQOA2TSDPASZP424U23QORNA, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:21.587662915 (90h40m21.45874595s) [noise=true , address=77.90.31.10:4529 , piece_id=YIT3CCLUOLSQ3NGOUVF7PYVV6FIDT7WKTRQVU4VTU4LRMPYL4H3A, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:21.624695792 (90h40m21.495778827s) [noise=true , address=77.90.31.15:4703 , piece_id=3FIJGK2XHNRTDF4Q3DN3AX2P42AUR5CNUI5QPXQ256MPMCQB5QRQ, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:21.659658729 (90h40m21.530741774s) [noise=true , address=77.90.31.12:4615 , piece_id=BLNHDHE34WZMROFSL5SNTHOITFWI6JOR7VKQBYZB77LTINEGLS5Q, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:21.667474853 (90h40m21.538557898s) [noise=true , address=145.79.227.24:4587 , piece_id=EVUNUBMSMMIERRCDR67LDFSGY6S2EF6IRV6EGEX4IOFX4PCTYIDQ, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:21.669766096 (90h40m21.540849131s) [noise=true , address=77.90.31.14:4684 , piece_id=ARECBM73CDZDYSVWPLH2XCQC6A45SGW57TPWX5URGCAUW355COOA, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:21.670505664 (90h40m21.541588709s) [noise=true , address=77.90.31.17:4783 , piece_id=LSTC3TY566OQBGL6ZXBLL5IFNPORZRWOEGFKGT3EXYOB3IPWI6JQ, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:21.680637627 (90h40m21.551720662s) [noise=true , address=77.90.31.13:4639 , piece_id=I2GHDHJTAHCDYP7BAG6QAHGPAFIKQZOLUPD5V7AICS4CATG3XNDA, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:21.685855022 (90h40m21.556938057s) [noise=true , address=77.90.31.21:4882 , piece_id=CJSACAOIT5IXGJXTL4YDNRUK4H5KD6QCFE6VINQGMQJ2TT2TAU5A, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:21.741439516 (90h40m21.612522561s) [noise=true , address=77.90.31.19:4870 , piece_id=P6W2QVXDH57FK755LD67JDSC23BZPHVQHBVWZLJDC457JPG3ZQLQ, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:22.306045351 (90h40m22.177128396s) [noise=true , address=77.90.31.16:4760 , piece_id=LJPEX76BQ4ICDJ3I4AWHE7UE7D2WIZCVGQS6XU6ZZ2II2NBYDKHQ, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:22.498367742 (90h40m22.369450787s) [noise=true , address=145.79.227.13:4824 , piece_id=M6P4NFRFOU5LGE2HRGUXRUZXJ2HJYA6BWC3TW4MWUWIYYSUTR76Q, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:10:18.740403422 (90h41m18.611486467s) [noise=true , address=45.91.116.13:4173 , piece_id=7PIVANLRS7SL6ERBMUC4LT4P3C3JRHRY6Z4I7336EVMXC3IHFVDQ, seg_pos={0 0}, upload=79096]: Done uploading piece. err: ecclient: upload failed (node:1J3HZY416Dm9F5zRXtoDdt8s8tGoU6GpkwS14CXJUzd3zpcYa1, address:45.91.116.13:4173): protocol: write tcp 188.166.225.170:53476->45.91.116.13:4173: use of closed network connection; write tcp 188.166.225.170:53476->45.91.116.13:4173: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53476->45.91.116.13:4173: use of closed network connection Dec 9 10:10:18.740531145 (90h41m18.61161419s) [noise=true , address=64.49.13.169:4710 , piece_id=XEKNBKQ4WTECDQD6YYP5RUU2KOSFPRGKXRWDN5THYLE4HXO65BEA, seg_pos={0 0}, upload=79096]: Done uploading piece. err: ecclient: upload failed (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf, address:64.49.13.169:4710): protocol: write tcp 188.166.225.170:37916->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:37916->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37916->64.49.13.169:4710: use of closed network connection Dec 9 10:10:18.740657375 (90h41m18.61174042s) [noise=true , address=45.91.116.12:4254 , piece_id=WOGMNSLO5FOGUSOWXRTLPZNASJB2ZMPUEKC4WRQNROTKQ74JV6KQ, seg_pos={0 0}, upload=79096]: Done uploading piece. err: ecclient: upload failed (node:12wAKmaLHwjBKN8UozGFjGvNMYG57fCxTH8wuDZDRnMdyGBRsTv, address:45.91.116.12:4254): protocol: write tcp 188.166.225.170:35550->45.91.116.12:4254: use of closed network connection; write tcp 188.166.225.170:35550->45.91.116.12:4254: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35550->45.91.116.12:4254: use of closed network connection Dec 9 10:10:18.740778986 (90h41m18.611862021s) [noise=true , address=45.91.116.5:4324 , piece_id=XUK3P6OWTGEJMVLR5B3ZR667WEMCEYIAIVW3PTE6ZWTREKSNBTBA, seg_pos={0 0}, upload=79096]: Done uploading piece. err: ecclient: upload failed (node:12btBm53zb1MCTy6b7CUeujUruMR9nxBoRgqhHmG5ffPf7rL4F, address:45.91.116.5:4324): protocol: write tcp 188.166.225.170:41676->45.91.116.5:4324: use of closed network connection; write tcp 188.166.225.170:41676->45.91.116.5:4324: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41676->45.91.116.5:4324: use of closed network connection Dec 9 10:10:18.740879748 (90h41m18.611962793s) [noise=true , address=45.91.116.11:4140 , piece_id=UJVOUUVBOSA4I4YZJIAPP2QGWBUV322ZSSWINU52U5RIVJ4CEZXA, seg_pos={0 0}, upload=79096]: Done uploading piece. err: ecclient: upload failed (node:14DNa2ysTGtfy3Zz72Dq7S3NSGXqnTHZRkuXorDvuuBw1abwMT, address:45.91.116.11:4140): protocol: write tcp 188.166.225.170:34050->45.91.116.11:4140: use of closed network connection; write tcp 188.166.225.170:34050->45.91.116.11:4140: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34050->45.91.116.11:4140: use of closed network connection Dec 9 10:10:18.740962677 (90h41m18.612045712s) [noise=true , address=45.91.116.9:4264 , piece_id=7UODMOFUXQZ3KYCY4RFA7OUVYXQTWLKNSWR3QIVMIT5KHEQMGT2Q, seg_pos={0 0}, upload=79096]: Done uploading piece. err: ecclient: upload failed (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, address:45.91.116.9:4264): protocol: write tcp 188.166.225.170:50170->45.91.116.9:4264: use of closed network connection; write tcp 188.166.225.170:50170->45.91.116.9:4264: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50170->45.91.116.9:4264: use of closed network connection Dec 9 10:10:18.740989428 (90h41m18.612072463s) [seg_pos={0 0}, upload=79096]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Dec 9 10:10:18.978016295 (90h41m18.84909934s) [upload=79096]: Done uploading segments... Dec 9 10:10:18.978064968 (90h41m18.849148013s) [upload=79096]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 12.654103979s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 1.971838354s trace: https://jaeger.dev.storj.io/trace/6cb2657ad006bedf FAIL (2025-12-09 10:27:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 09:57:44.389846478 (90h28m44.260929523s) [upload=78917]: Starting upload Dec 9 09:57:44.390051578 (90h28m44.261134613s) [upload=78917]: Done starting upload Dec 9 09:57:44.390125740 (90h28m44.261208775s) [upload=78917]: Uploading segments... Dec 9 09:57:44.390224027 (90h28m44.261307072s) [upload=78917]: Got next segment. Inline: true Dec 9 09:57:44.390230800 (90h28m44.261313835s) [upload=78917]: Waiting for error group managing segments... Dec 9 09:57:44.390312034 (90h28m44.261395069s) [upload=78917]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 09:57:44.390318787 (90h28m44.261401822s) [upload=78917]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 9 09:57:44.390323356 (90h28m44.261406391s) [upload=78917]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 09:58:21.934924620 (90h29m21.806007665s) [upload=78917]: Done uploading segments... Dec 9 09:58:21.934974756 (90h29m21.806057801s) [upload=78917]: Upload finished. err: metaclient: context deadline exceeded Duration:* 1.971868893s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 12.348744138s trace: https://jaeger.dev.storj.io/trace/781235978226387b FAIL (2025-12-09 10:27:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ... Dec 9 10:09:19.659155420 (90h40m19.530238455s) [noise=true , address=77.90.31.19:4870 , piece_id=P6W2QVXDH57FK755LD67JDSC23BZPHVQHBVWZLJDC457JPG3ZQLQ, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.659573456 (90h40m19.530656491s) [noise=true , address=77.90.31.15:4703 , piece_id=3FIJGK2XHNRTDF4Q3DN3AX2P42AUR5CNUI5QPXQ256MPMCQB5QRQ, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.660082366 (90h40m19.531165401s) [noise=true , address=77.90.31.16:4760 , piece_id=LJPEX76BQ4ICDJ3I4AWHE7UE7D2WIZCVGQS6XU6ZZ2II2NBYDKHQ, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.660519568 (90h40m19.531602613s) [noise=true , address=45.91.116.9:4264 , piece_id=7UODMOFUXQZ3KYCY4RFA7OUVYXQTWLKNSWR3QIVMIT5KHEQMGT2Q, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.660971178 (90h40m19.532054223s) [noise=true , address=77.90.31.21:4882 , piece_id=CJSACAOIT5IXGJXTL4YDNRUK4H5KD6QCFE6VINQGMQJ2TT2TAU5A, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.661377291 (90h40m19.532460326s) [noise=true , address=77.90.31.13:4639 , piece_id=I2GHDHJTAHCDYP7BAG6QAHGPAFIKQZOLUPD5V7AICS4CATG3XNDA, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.661793374 (90h40m19.532876409s) [noise=true , address=45.91.116.11:4140 , piece_id=UJVOUUVBOSA4I4YZJIAPP2QGWBUV322ZSSWINU52U5RIVJ4CEZXA, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.661991682 (90h40m19.533074717s) [noise=true , address=77.90.31.14:4684 , piece_id=ARECBM73CDZDYSVWPLH2XCQC6A45SGW57TPWX5URGCAUW355COOA, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.662431088 (90h40m19.533514133s) [noise=true , address=77.90.31.12:4615 , piece_id=BLNHDHE34WZMROFSL5SNTHOITFWI6JOR7VKQBYZB77LTINEGLS5Q, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.662848232 (90h40m19.533931277s) [noise=true , address=145.79.227.13:4824 , piece_id=M6P4NFRFOU5LGE2HRGUXRUZXJ2HJYA6BWC3TW4MWUWIYYSUTR76Q, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.663338626 (90h40m19.534421671s) [noise=true , address=145.79.227.24:4587 , piece_id=EVUNUBMSMMIERRCDR67LDFSGY6S2EF6IRV6EGEX4IOFX4PCTYIDQ, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.664502282 (90h40m19.535585327s) [noise=true , address=45.91.116.12:4254 , piece_id=WOGMNSLO5FOGUSOWXRTLPZNASJB2ZMPUEKC4WRQNROTKQ74JV6KQ, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.666711839 (90h40m19.537794884s) [noise=true , address=77.90.31.10:4529 , piece_id=YIT3CCLUOLSQ3NGOUVF7PYVV6FIDT7WKTRQVU4VTU4LRMPYL4H3A, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.669766776 (90h40m19.540849821s) [noise=true , address=77.90.31.17:4783 , piece_id=LSTC3TY566OQBGL6ZXBLL5IFNPORZRWOEGFKGT3EXYOB3IPWI6JQ, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.677090573 (90h40m19.548173618s) [noise=true , address=45.91.116.13:4173 , piece_id=7PIVANLRS7SL6ERBMUC4LT4P3C3JRHRY6Z4I7336EVMXC3IHFVDQ, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:21.186498404 (90h40m21.057581499s) [upload=79096]: Next returned nil segment Dec 9 10:09:21.186518313 (90h40m21.057601348s) [upload=79096]: Waiting for error group managing segments... Dec 9 10:09:21.574704328 (90h40m21.445787374s) [noise=true , address=77.90.31.18:4829 , piece_id=CDRLGWU6VF6TU6E4E5B6R7NOF35WOQOA2TSDPASZP424U23QORNA, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:21.587662915 (90h40m21.45874595s) [noise=true , address=77.90.31.10:4529 , piece_id=YIT3CCLUOLSQ3NGOUVF7PYVV6FIDT7WKTRQVU4VTU4LRMPYL4H3A, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:21.624695792 (90h40m21.495778827s) [noise=true , address=77.90.31.15:4703 , piece_id=3FIJGK2XHNRTDF4Q3DN3AX2P42AUR5CNUI5QPXQ256MPMCQB5QRQ, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:21.659658729 (90h40m21.530741774s) [noise=true , address=77.90.31.12:4615 , piece_id=BLNHDHE34WZMROFSL5SNTHOITFWI6JOR7VKQBYZB77LTINEGLS5Q, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:21.667474853 (90h40m21.538557898s) [noise=true , address=145.79.227.24:4587 , piece_id=EVUNUBMSMMIERRCDR67LDFSGY6S2EF6IRV6EGEX4IOFX4PCTYIDQ, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:21.669766096 (90h40m21.540849131s) [noise=true , address=77.90.31.14:4684 , piece_id=ARECBM73CDZDYSVWPLH2XCQC6A45SGW57TPWX5URGCAUW355COOA, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:21.670505664 (90h40m21.541588709s) [noise=true , address=77.90.31.17:4783 , piece_id=LSTC3TY566OQBGL6ZXBLL5IFNPORZRWOEGFKGT3EXYOB3IPWI6JQ, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:21.680637627 (90h40m21.551720662s) [noise=true , address=77.90.31.13:4639 , piece_id=I2GHDHJTAHCDYP7BAG6QAHGPAFIKQZOLUPD5V7AICS4CATG3XNDA, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:21.685855022 (90h40m21.556938057s) [noise=true , address=77.90.31.21:4882 , piece_id=CJSACAOIT5IXGJXTL4YDNRUK4H5KD6QCFE6VINQGMQJ2TT2TAU5A, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:21.741439516 (90h40m21.612522561s) [noise=true , address=77.90.31.19:4870 , piece_id=P6W2QVXDH57FK755LD67JDSC23BZPHVQHBVWZLJDC457JPG3ZQLQ, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:22.306045351 (90h40m22.177128396s) [noise=true , address=77.90.31.16:4760 , piece_id=LJPEX76BQ4ICDJ3I4AWHE7UE7D2WIZCVGQS6XU6ZZ2II2NBYDKHQ, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:22.498367742 (90h40m22.369450787s) [noise=true , address=145.79.227.13:4824 , piece_id=M6P4NFRFOU5LGE2HRGUXRUZXJ2HJYA6BWC3TW4MWUWIYYSUTR76Q, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:10:18.740403422 (90h41m18.611486467s) [noise=true , address=45.91.116.13:4173 , piece_id=7PIVANLRS7SL6ERBMUC4LT4P3C3JRHRY6Z4I7336EVMXC3IHFVDQ, seg_pos={0 0}, upload=79096]: Done uploading piece. err: ecclient: upload failed (node:1J3HZY416Dm9F5zRXtoDdt8s8tGoU6GpkwS14CXJUzd3zpcYa1, address:45.91.116.13:4173): protocol: write tcp 188.166.225.170:53476->45.91.116.13:4173: use of closed network connection; write tcp 188.166.225.170:53476->45.91.116.13:4173: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53476->45.91.116.13:4173: use of closed network connection Dec 9 10:10:18.740531145 (90h41m18.61161419s) [noise=true , address=64.49.13.169:4710 , piece_id=XEKNBKQ4WTECDQD6YYP5RUU2KOSFPRGKXRWDN5THYLE4HXO65BEA, seg_pos={0 0}, upload=79096]: Done uploading piece. err: ecclient: upload failed (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf, address:64.49.13.169:4710): protocol: write tcp 188.166.225.170:37916->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:37916->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37916->64.49.13.169:4710: use of closed network connection Dec 9 10:10:18.740657375 (90h41m18.61174042s) [noise=true , address=45.91.116.12:4254 , piece_id=WOGMNSLO5FOGUSOWXRTLPZNASJB2ZMPUEKC4WRQNROTKQ74JV6KQ, seg_pos={0 0}, upload=79096]: Done uploading piece. err: ecclient: upload failed (node:12wAKmaLHwjBKN8UozGFjGvNMYG57fCxTH8wuDZDRnMdyGBRsTv, address:45.91.116.12:4254): protocol: write tcp 188.166.225.170:35550->45.91.116.12:4254: use of closed network connection; write tcp 188.166.225.170:35550->45.91.116.12:4254: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35550->45.91.116.12:4254: use of closed network connection Dec 9 10:10:18.740778986 (90h41m18.611862021s) [noise=true , address=45.91.116.5:4324 , piece_id=XUK3P6OWTGEJMVLR5B3ZR667WEMCEYIAIVW3PTE6ZWTREKSNBTBA, seg_pos={0 0}, upload=79096]: Done uploading piece. err: ecclient: upload failed (node:12btBm53zb1MCTy6b7CUeujUruMR9nxBoRgqhHmG5ffPf7rL4F, address:45.91.116.5:4324): protocol: write tcp 188.166.225.170:41676->45.91.116.5:4324: use of closed network connection; write tcp 188.166.225.170:41676->45.91.116.5:4324: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41676->45.91.116.5:4324: use of closed network connection Dec 9 10:10:18.740879748 (90h41m18.611962793s) [noise=true , address=45.91.116.11:4140 , piece_id=UJVOUUVBOSA4I4YZJIAPP2QGWBUV322ZSSWINU52U5RIVJ4CEZXA, seg_pos={0 0}, upload=79096]: Done uploading piece. err: ecclient: upload failed (node:14DNa2ysTGtfy3Zz72Dq7S3NSGXqnTHZRkuXorDvuuBw1abwMT, address:45.91.116.11:4140): protocol: write tcp 188.166.225.170:34050->45.91.116.11:4140: use of closed network connection; write tcp 188.166.225.170:34050->45.91.116.11:4140: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34050->45.91.116.11:4140: use of closed network connection Dec 9 10:10:18.740962677 (90h41m18.612045712s) [noise=true , address=45.91.116.9:4264 , piece_id=7UODMOFUXQZ3KYCY4RFA7OUVYXQTWLKNSWR3QIVMIT5KHEQMGT2Q, seg_pos={0 0}, upload=79096]: Done uploading piece. err: ecclient: upload failed (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, address:45.91.116.9:4264): protocol: write tcp 188.166.225.170:50170->45.91.116.9:4264: use of closed network connection; write tcp 188.166.225.170:50170->45.91.116.9:4264: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50170->45.91.116.9:4264: use of closed network connection Dec 9 10:10:18.740989428 (90h41m18.612072463s) [seg_pos={0 0}, upload=79096]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Dec 9 10:10:18.978016295 (90h41m18.84909934s) [upload=79096]: Done uploading segments... Dec 9 10:10:18.978064968 (90h41m18.849148013s) [upload=79096]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 12.348773545s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.........X........X......X..................X...............] duration: 5.731039502s trace: https://jaeger.dev.storj.io/trace/5b4024e5e8cc17e6 FAIL (2025-12-09 10:27:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.731069048s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.................................X..X..............X........] duration: 3.515761981s trace: https://jaeger.dev.storj.io/trace/9e23c2a25a964f2 FAIL (2025-12-09 10:26:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.515787379s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 10.283124208s trace: https://jaeger.dev.storj.io/trace/781235978226387b FAIL (2025-12-09 10:26:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ... Dec 9 10:09:19.659155420 (90h40m19.530238455s) [noise=true , address=77.90.31.19:4870 , piece_id=P6W2QVXDH57FK755LD67JDSC23BZPHVQHBVWZLJDC457JPG3ZQLQ, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.659573456 (90h40m19.530656491s) [noise=true , address=77.90.31.15:4703 , piece_id=3FIJGK2XHNRTDF4Q3DN3AX2P42AUR5CNUI5QPXQ256MPMCQB5QRQ, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.660082366 (90h40m19.531165401s) [noise=true , address=77.90.31.16:4760 , piece_id=LJPEX76BQ4ICDJ3I4AWHE7UE7D2WIZCVGQS6XU6ZZ2II2NBYDKHQ, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.660519568 (90h40m19.531602613s) [noise=true , address=45.91.116.9:4264 , piece_id=7UODMOFUXQZ3KYCY4RFA7OUVYXQTWLKNSWR3QIVMIT5KHEQMGT2Q, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.660971178 (90h40m19.532054223s) [noise=true , address=77.90.31.21:4882 , piece_id=CJSACAOIT5IXGJXTL4YDNRUK4H5KD6QCFE6VINQGMQJ2TT2TAU5A, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.661377291 (90h40m19.532460326s) [noise=true , address=77.90.31.13:4639 , piece_id=I2GHDHJTAHCDYP7BAG6QAHGPAFIKQZOLUPD5V7AICS4CATG3XNDA, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.661793374 (90h40m19.532876409s) [noise=true , address=45.91.116.11:4140 , piece_id=UJVOUUVBOSA4I4YZJIAPP2QGWBUV322ZSSWINU52U5RIVJ4CEZXA, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.661991682 (90h40m19.533074717s) [noise=true , address=77.90.31.14:4684 , piece_id=ARECBM73CDZDYSVWPLH2XCQC6A45SGW57TPWX5URGCAUW355COOA, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.662431088 (90h40m19.533514133s) [noise=true , address=77.90.31.12:4615 , piece_id=BLNHDHE34WZMROFSL5SNTHOITFWI6JOR7VKQBYZB77LTINEGLS5Q, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.662848232 (90h40m19.533931277s) [noise=true , address=145.79.227.13:4824 , piece_id=M6P4NFRFOU5LGE2HRGUXRUZXJ2HJYA6BWC3TW4MWUWIYYSUTR76Q, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.663338626 (90h40m19.534421671s) [noise=true , address=145.79.227.24:4587 , piece_id=EVUNUBMSMMIERRCDR67LDFSGY6S2EF6IRV6EGEX4IOFX4PCTYIDQ, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.664502282 (90h40m19.535585327s) [noise=true , address=45.91.116.12:4254 , piece_id=WOGMNSLO5FOGUSOWXRTLPZNASJB2ZMPUEKC4WRQNROTKQ74JV6KQ, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.666711839 (90h40m19.537794884s) [noise=true , address=77.90.31.10:4529 , piece_id=YIT3CCLUOLSQ3NGOUVF7PYVV6FIDT7WKTRQVU4VTU4LRMPYL4H3A, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.669766776 (90h40m19.540849821s) [noise=true , address=77.90.31.17:4783 , piece_id=LSTC3TY566OQBGL6ZXBLL5IFNPORZRWOEGFKGT3EXYOB3IPWI6JQ, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.677090573 (90h40m19.548173618s) [noise=true , address=45.91.116.13:4173 , piece_id=7PIVANLRS7SL6ERBMUC4LT4P3C3JRHRY6Z4I7336EVMXC3IHFVDQ, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:21.186498404 (90h40m21.057581499s) [upload=79096]: Next returned nil segment Dec 9 10:09:21.186518313 (90h40m21.057601348s) [upload=79096]: Waiting for error group managing segments... Dec 9 10:09:21.574704328 (90h40m21.445787374s) [noise=true , address=77.90.31.18:4829 , piece_id=CDRLGWU6VF6TU6E4E5B6R7NOF35WOQOA2TSDPASZP424U23QORNA, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:21.587662915 (90h40m21.45874595s) [noise=true , address=77.90.31.10:4529 , piece_id=YIT3CCLUOLSQ3NGOUVF7PYVV6FIDT7WKTRQVU4VTU4LRMPYL4H3A, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:21.624695792 (90h40m21.495778827s) [noise=true , address=77.90.31.15:4703 , piece_id=3FIJGK2XHNRTDF4Q3DN3AX2P42AUR5CNUI5QPXQ256MPMCQB5QRQ, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:21.659658729 (90h40m21.530741774s) [noise=true , address=77.90.31.12:4615 , piece_id=BLNHDHE34WZMROFSL5SNTHOITFWI6JOR7VKQBYZB77LTINEGLS5Q, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:21.667474853 (90h40m21.538557898s) [noise=true , address=145.79.227.24:4587 , piece_id=EVUNUBMSMMIERRCDR67LDFSGY6S2EF6IRV6EGEX4IOFX4PCTYIDQ, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:21.669766096 (90h40m21.540849131s) [noise=true , address=77.90.31.14:4684 , piece_id=ARECBM73CDZDYSVWPLH2XCQC6A45SGW57TPWX5URGCAUW355COOA, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:21.670505664 (90h40m21.541588709s) [noise=true , address=77.90.31.17:4783 , piece_id=LSTC3TY566OQBGL6ZXBLL5IFNPORZRWOEGFKGT3EXYOB3IPWI6JQ, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:21.680637627 (90h40m21.551720662s) [noise=true , address=77.90.31.13:4639 , piece_id=I2GHDHJTAHCDYP7BAG6QAHGPAFIKQZOLUPD5V7AICS4CATG3XNDA, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:21.685855022 (90h40m21.556938057s) [noise=true , address=77.90.31.21:4882 , piece_id=CJSACAOIT5IXGJXTL4YDNRUK4H5KD6QCFE6VINQGMQJ2TT2TAU5A, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:21.741439516 (90h40m21.612522561s) [noise=true , address=77.90.31.19:4870 , piece_id=P6W2QVXDH57FK755LD67JDSC23BZPHVQHBVWZLJDC457JPG3ZQLQ, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:22.306045351 (90h40m22.177128396s) [noise=true , address=77.90.31.16:4760 , piece_id=LJPEX76BQ4ICDJ3I4AWHE7UE7D2WIZCVGQS6XU6ZZ2II2NBYDKHQ, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:22.498367742 (90h40m22.369450787s) [noise=true , address=145.79.227.13:4824 , piece_id=M6P4NFRFOU5LGE2HRGUXRUZXJ2HJYA6BWC3TW4MWUWIYYSUTR76Q, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:10:18.740403422 (90h41m18.611486467s) [noise=true , address=45.91.116.13:4173 , piece_id=7PIVANLRS7SL6ERBMUC4LT4P3C3JRHRY6Z4I7336EVMXC3IHFVDQ, seg_pos={0 0}, upload=79096]: Done uploading piece. err: ecclient: upload failed (node:1J3HZY416Dm9F5zRXtoDdt8s8tGoU6GpkwS14CXJUzd3zpcYa1, address:45.91.116.13:4173): protocol: write tcp 188.166.225.170:53476->45.91.116.13:4173: use of closed network connection; write tcp 188.166.225.170:53476->45.91.116.13:4173: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53476->45.91.116.13:4173: use of closed network connection Dec 9 10:10:18.740531145 (90h41m18.61161419s) [noise=true , address=64.49.13.169:4710 , piece_id=XEKNBKQ4WTECDQD6YYP5RUU2KOSFPRGKXRWDN5THYLE4HXO65BEA, seg_pos={0 0}, upload=79096]: Done uploading piece. err: ecclient: upload failed (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf, address:64.49.13.169:4710): protocol: write tcp 188.166.225.170:37916->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:37916->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37916->64.49.13.169:4710: use of closed network connection Dec 9 10:10:18.740657375 (90h41m18.61174042s) [noise=true , address=45.91.116.12:4254 , piece_id=WOGMNSLO5FOGUSOWXRTLPZNASJB2ZMPUEKC4WRQNROTKQ74JV6KQ, seg_pos={0 0}, upload=79096]: Done uploading piece. err: ecclient: upload failed (node:12wAKmaLHwjBKN8UozGFjGvNMYG57fCxTH8wuDZDRnMdyGBRsTv, address:45.91.116.12:4254): protocol: write tcp 188.166.225.170:35550->45.91.116.12:4254: use of closed network connection; write tcp 188.166.225.170:35550->45.91.116.12:4254: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35550->45.91.116.12:4254: use of closed network connection Dec 9 10:10:18.740778986 (90h41m18.611862021s) [noise=true , address=45.91.116.5:4324 , piece_id=XUK3P6OWTGEJMVLR5B3ZR667WEMCEYIAIVW3PTE6ZWTREKSNBTBA, seg_pos={0 0}, upload=79096]: Done uploading piece. err: ecclient: upload failed (node:12btBm53zb1MCTy6b7CUeujUruMR9nxBoRgqhHmG5ffPf7rL4F, address:45.91.116.5:4324): protocol: write tcp 188.166.225.170:41676->45.91.116.5:4324: use of closed network connection; write tcp 188.166.225.170:41676->45.91.116.5:4324: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41676->45.91.116.5:4324: use of closed network connection Dec 9 10:10:18.740879748 (90h41m18.611962793s) [noise=true , address=45.91.116.11:4140 , piece_id=UJVOUUVBOSA4I4YZJIAPP2QGWBUV322ZSSWINU52U5RIVJ4CEZXA, seg_pos={0 0}, upload=79096]: Done uploading piece. err: ecclient: upload failed (node:14DNa2ysTGtfy3Zz72Dq7S3NSGXqnTHZRkuXorDvuuBw1abwMT, address:45.91.116.11:4140): protocol: write tcp 188.166.225.170:34050->45.91.116.11:4140: use of closed network connection; write tcp 188.166.225.170:34050->45.91.116.11:4140: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34050->45.91.116.11:4140: use of closed network connection Dec 9 10:10:18.740962677 (90h41m18.612045712s) [noise=true , address=45.91.116.9:4264 , piece_id=7UODMOFUXQZ3KYCY4RFA7OUVYXQTWLKNSWR3QIVMIT5KHEQMGT2Q, seg_pos={0 0}, upload=79096]: Done uploading piece. err: ecclient: upload failed (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, address:45.91.116.9:4264): protocol: write tcp 188.166.225.170:50170->45.91.116.9:4264: use of closed network connection; write tcp 188.166.225.170:50170->45.91.116.9:4264: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50170->45.91.116.9:4264: use of closed network connection Dec 9 10:10:18.740989428 (90h41m18.612072463s) [seg_pos={0 0}, upload=79096]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Dec 9 10:10:18.978016295 (90h41m18.84909934s) [upload=79096]: Done uploading segments... Dec 9 10:10:18.978064968 (90h41m18.849148013s) [upload=79096]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 10.283142443s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [..........X........X......X..................X..............] duration: 5.645314418s trace: https://jaeger.dev.storj.io/trace/5b4024e5e8cc17e6 FAIL (2025-12-09 10:26:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.645342402s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 14.763171034s trace: https://jaeger.dev.storj.io/trace/781235978226387b FAIL (2025-12-09 10:26:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ... Dec 9 10:09:19.659155420 (90h40m19.530238455s) [noise=true , address=77.90.31.19:4870 , piece_id=P6W2QVXDH57FK755LD67JDSC23BZPHVQHBVWZLJDC457JPG3ZQLQ, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.659573456 (90h40m19.530656491s) [noise=true , address=77.90.31.15:4703 , piece_id=3FIJGK2XHNRTDF4Q3DN3AX2P42AUR5CNUI5QPXQ256MPMCQB5QRQ, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.660082366 (90h40m19.531165401s) [noise=true , address=77.90.31.16:4760 , piece_id=LJPEX76BQ4ICDJ3I4AWHE7UE7D2WIZCVGQS6XU6ZZ2II2NBYDKHQ, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.660519568 (90h40m19.531602613s) [noise=true , address=45.91.116.9:4264 , piece_id=7UODMOFUXQZ3KYCY4RFA7OUVYXQTWLKNSWR3QIVMIT5KHEQMGT2Q, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.660971178 (90h40m19.532054223s) [noise=true , address=77.90.31.21:4882 , piece_id=CJSACAOIT5IXGJXTL4YDNRUK4H5KD6QCFE6VINQGMQJ2TT2TAU5A, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.661377291 (90h40m19.532460326s) [noise=true , address=77.90.31.13:4639 , piece_id=I2GHDHJTAHCDYP7BAG6QAHGPAFIKQZOLUPD5V7AICS4CATG3XNDA, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.661793374 (90h40m19.532876409s) [noise=true , address=45.91.116.11:4140 , piece_id=UJVOUUVBOSA4I4YZJIAPP2QGWBUV322ZSSWINU52U5RIVJ4CEZXA, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.661991682 (90h40m19.533074717s) [noise=true , address=77.90.31.14:4684 , piece_id=ARECBM73CDZDYSVWPLH2XCQC6A45SGW57TPWX5URGCAUW355COOA, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.662431088 (90h40m19.533514133s) [noise=true , address=77.90.31.12:4615 , piece_id=BLNHDHE34WZMROFSL5SNTHOITFWI6JOR7VKQBYZB77LTINEGLS5Q, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.662848232 (90h40m19.533931277s) [noise=true , address=145.79.227.13:4824 , piece_id=M6P4NFRFOU5LGE2HRGUXRUZXJ2HJYA6BWC3TW4MWUWIYYSUTR76Q, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.663338626 (90h40m19.534421671s) [noise=true , address=145.79.227.24:4587 , piece_id=EVUNUBMSMMIERRCDR67LDFSGY6S2EF6IRV6EGEX4IOFX4PCTYIDQ, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.664502282 (90h40m19.535585327s) [noise=true , address=45.91.116.12:4254 , piece_id=WOGMNSLO5FOGUSOWXRTLPZNASJB2ZMPUEKC4WRQNROTKQ74JV6KQ, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.666711839 (90h40m19.537794884s) [noise=true , address=77.90.31.10:4529 , piece_id=YIT3CCLUOLSQ3NGOUVF7PYVV6FIDT7WKTRQVU4VTU4LRMPYL4H3A, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.669766776 (90h40m19.540849821s) [noise=true , address=77.90.31.17:4783 , piece_id=LSTC3TY566OQBGL6ZXBLL5IFNPORZRWOEGFKGT3EXYOB3IPWI6JQ, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:19.677090573 (90h40m19.548173618s) [noise=true , address=45.91.116.13:4173 , piece_id=7PIVANLRS7SL6ERBMUC4LT4P3C3JRHRY6Z4I7336EVMXC3IHFVDQ, seg_pos={0 0}, upload=79096]: Uploading piece... Dec 9 10:09:21.186498404 (90h40m21.057581499s) [upload=79096]: Next returned nil segment Dec 9 10:09:21.186518313 (90h40m21.057601348s) [upload=79096]: Waiting for error group managing segments... Dec 9 10:09:21.574704328 (90h40m21.445787374s) [noise=true , address=77.90.31.18:4829 , piece_id=CDRLGWU6VF6TU6E4E5B6R7NOF35WOQOA2TSDPASZP424U23QORNA, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:21.587662915 (90h40m21.45874595s) [noise=true , address=77.90.31.10:4529 , piece_id=YIT3CCLUOLSQ3NGOUVF7PYVV6FIDT7WKTRQVU4VTU4LRMPYL4H3A, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:21.624695792 (90h40m21.495778827s) [noise=true , address=77.90.31.15:4703 , piece_id=3FIJGK2XHNRTDF4Q3DN3AX2P42AUR5CNUI5QPXQ256MPMCQB5QRQ, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:21.659658729 (90h40m21.530741774s) [noise=true , address=77.90.31.12:4615 , piece_id=BLNHDHE34WZMROFSL5SNTHOITFWI6JOR7VKQBYZB77LTINEGLS5Q, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:21.667474853 (90h40m21.538557898s) [noise=true , address=145.79.227.24:4587 , piece_id=EVUNUBMSMMIERRCDR67LDFSGY6S2EF6IRV6EGEX4IOFX4PCTYIDQ, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:21.669766096 (90h40m21.540849131s) [noise=true , address=77.90.31.14:4684 , piece_id=ARECBM73CDZDYSVWPLH2XCQC6A45SGW57TPWX5URGCAUW355COOA, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:21.670505664 (90h40m21.541588709s) [noise=true , address=77.90.31.17:4783 , piece_id=LSTC3TY566OQBGL6ZXBLL5IFNPORZRWOEGFKGT3EXYOB3IPWI6JQ, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:21.680637627 (90h40m21.551720662s) [noise=true , address=77.90.31.13:4639 , piece_id=I2GHDHJTAHCDYP7BAG6QAHGPAFIKQZOLUPD5V7AICS4CATG3XNDA, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:21.685855022 (90h40m21.556938057s) [noise=true , address=77.90.31.21:4882 , piece_id=CJSACAOIT5IXGJXTL4YDNRUK4H5KD6QCFE6VINQGMQJ2TT2TAU5A, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:21.741439516 (90h40m21.612522561s) [noise=true , address=77.90.31.19:4870 , piece_id=P6W2QVXDH57FK755LD67JDSC23BZPHVQHBVWZLJDC457JPG3ZQLQ, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:22.306045351 (90h40m22.177128396s) [noise=true , address=77.90.31.16:4760 , piece_id=LJPEX76BQ4ICDJ3I4AWHE7UE7D2WIZCVGQS6XU6ZZ2II2NBYDKHQ, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:09:22.498367742 (90h40m22.369450787s) [noise=true , address=145.79.227.13:4824 , piece_id=M6P4NFRFOU5LGE2HRGUXRUZXJ2HJYA6BWC3TW4MWUWIYYSUTR76Q, seg_pos={0 0}, upload=79096]: Done uploading piece. err: <nil> Dec 9 10:10:18.740403422 (90h41m18.611486467s) [noise=true , address=45.91.116.13:4173 , piece_id=7PIVANLRS7SL6ERBMUC4LT4P3C3JRHRY6Z4I7336EVMXC3IHFVDQ, seg_pos={0 0}, upload=79096]: Done uploading piece. err: ecclient: upload failed (node:1J3HZY416Dm9F5zRXtoDdt8s8tGoU6GpkwS14CXJUzd3zpcYa1, address:45.91.116.13:4173): protocol: write tcp 188.166.225.170:53476->45.91.116.13:4173: use of closed network connection; write tcp 188.166.225.170:53476->45.91.116.13:4173: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53476->45.91.116.13:4173: use of closed network connection Dec 9 10:10:18.740531145 (90h41m18.61161419s) [noise=true , address=64.49.13.169:4710 , piece_id=XEKNBKQ4WTECDQD6YYP5RUU2KOSFPRGKXRWDN5THYLE4HXO65BEA, seg_pos={0 0}, upload=79096]: Done uploading piece. err: ecclient: upload failed (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf, address:64.49.13.169:4710): protocol: write tcp 188.166.225.170:37916->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:37916->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37916->64.49.13.169:4710: use of closed network connection Dec 9 10:10:18.740657375 (90h41m18.61174042s) [noise=true , address=45.91.116.12:4254 , piece_id=WOGMNSLO5FOGUSOWXRTLPZNASJB2ZMPUEKC4WRQNROTKQ74JV6KQ, seg_pos={0 0}, upload=79096]: Done uploading piece. err: ecclient: upload failed (node:12wAKmaLHwjBKN8UozGFjGvNMYG57fCxTH8wuDZDRnMdyGBRsTv, address:45.91.116.12:4254): protocol: write tcp 188.166.225.170:35550->45.91.116.12:4254: use of closed network connection; write tcp 188.166.225.170:35550->45.91.116.12:4254: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35550->45.91.116.12:4254: use of closed network connection Dec 9 10:10:18.740778986 (90h41m18.611862021s) [noise=true , address=45.91.116.5:4324 , piece_id=XUK3P6OWTGEJMVLR5B3ZR667WEMCEYIAIVW3PTE6ZWTREKSNBTBA, seg_pos={0 0}, upload=79096]: Done uploading piece. err: ecclient: upload failed (node:12btBm53zb1MCTy6b7CUeujUruMR9nxBoRgqhHmG5ffPf7rL4F, address:45.91.116.5:4324): protocol: write tcp 188.166.225.170:41676->45.91.116.5:4324: use of closed network connection; write tcp 188.166.225.170:41676->45.91.116.5:4324: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41676->45.91.116.5:4324: use of closed network connection Dec 9 10:10:18.740879748 (90h41m18.611962793s) [noise=true , address=45.91.116.11:4140 , piece_id=UJVOUUVBOSA4I4YZJIAPP2QGWBUV322ZSSWINU52U5RIVJ4CEZXA, seg_pos={0 0}, upload=79096]: Done uploading piece. err: ecclient: upload failed (node:14DNa2ysTGtfy3Zz72Dq7S3NSGXqnTHZRkuXorDvuuBw1abwMT, address:45.91.116.11:4140): protocol: write tcp 188.166.225.170:34050->45.91.116.11:4140: use of closed network connection; write tcp 188.166.225.170:34050->45.91.116.11:4140: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34050->45.91.116.11:4140: use of closed network connection Dec 9 10:10:18.740962677 (90h41m18.612045712s) [noise=true , address=45.91.116.9:4264 , piece_id=7UODMOFUXQZ3KYCY4RFA7OUVYXQTWLKNSWR3QIVMIT5KHEQMGT2Q, seg_pos={0 0}, upload=79096]: Done uploading piece. err: ecclient: upload failed (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, address:45.91.116.9:4264): protocol: write tcp 188.166.225.170:50170->45.91.116.9:4264: use of closed network connection; write tcp 188.166.225.170:50170->45.91.116.9:4264: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50170->45.91.116.9:4264: use of closed network connection Dec 9 10:10:18.740989428 (90h41m18.612072463s) [seg_pos={0 0}, upload=79096]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Dec 9 10:10:18.978016295 (90h41m18.84909934s) [upload=79096]: Done uploading segments... Dec 9 10:10:18.978064968 (90h41m18.849148013s) [upload=79096]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 14.763199177s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..................................X..X..............X.......] duration: 3.444307925s trace: https://jaeger.dev.storj.io/trace/9e23c2a25a964f2 FAIL (2025-12-09 10:26:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.444335198s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 2.017868344s trace: https://jaeger.dev.storj.io/trace/6cb2657ad006bedf FAIL (2025-12-09 10:26:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 09:57:44.389846478 (90h28m44.260929523s) [upload=78917]: Starting upload Dec 9 09:57:44.390051578 (90h28m44.261134613s) [upload=78917]: Done starting upload Dec 9 09:57:44.390125740 (90h28m44.261208775s) [upload=78917]: Uploading segments... Dec 9 09:57:44.390224027 (90h28m44.261307072s) [upload=78917]: Got next segment. Inline: true Dec 9 09:57:44.390230800 (90h28m44.261313835s) [upload=78917]: Waiting for error group managing segments... Dec 9 09:57:44.390312034 (90h28m44.261395069s) [upload=78917]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 09:57:44.390318787 (90h28m44.261401822s) [upload=78917]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 9 09:57:44.390323356 (90h28m44.261406391s) [upload=78917]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 09:58:21.934924620 (90h29m21.806007665s) [upload=78917]: Done uploading segments... Dec 9 09:58:21.934974756 (90h29m21.806057801s) [upload=78917]: Upload finished. err: metaclient: context deadline exceeded Duration:* 2.017895265s core uplink us-select-1 us1-placement-20

1
...
11619
11620
11621
11622
11623
...
17382