Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.......................X.........X...........X.......XXXXXXX] duration: 30.248953872s FAIL (2025-07-26 23:11:51 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== :11:21.659185414 (59h35m31.381748454s) [upload=57174]: Uploading segments... Jul 26 23:11:21.659357833 (59h35m31.381920883s) [upload=57174]: Got next segment. Inline: false Jul 26 23:11:21.659485127 (59h35m31.382048177s) [upload=57174]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:11:21.659493102 (59h35m31.382056162s) [upload=57174]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:11:21.894149121 (59h35m31.616712161s) [seg_pos={0 0}, upload=57174]: Begin upload segment... Jul 26 23:11:21.894187665 (59h35m31.616750705s) [seg_pos={0 0}, upload=57174]: Done begin upload segment. Jul 26 23:11:21.894209346 (59h35m31.616772386s) [noise=true , address=104.36.82.16:4293 , piece_id=IDLIQYUAE7RZ7FP5UM7XEFS4R57LRSCABCE5GAGG4IZA36NUEYFQ, seg_pos={0 0}, upload=57174]: Uploading piece... Jul 26 23:11:21.894249493 (59h35m31.616812533s) [noise=true , address=104.36.82.2:4729 , piece_id=4VYHVFZY2DHSW5J47APEREXY5G4GUPPX5FHS3DK7LZQJGVKHCHQA, seg_pos={0 0}, upload=57174]: Uploading piece... Jul 26 23:11:21.894557011 (59h35m31.617120041s) [noise=true , address=45.91.116.8:4047 , piece_id=2F5SP5PAKG4CJWOI5OJJSGNM3EOJRQOTUNQUKJL57BHLHGUI6Y4Q, seg_pos={0 0}, upload=57174]: Uploading piece... Jul 26 23:11:21.894815345 (59h35m31.617378385s) [noise=true , address=104.36.82.11:4115 , piece_id=PLT37RR5RPL24KNP7UNAK736KDPNHJFVGYGEGKKEBVPCF3F3Y2FQ, seg_pos={0 0}, upload=57174]: Uploading piece... Jul 26 23:11:21.894947929 (59h35m31.617510969s) [noise=true , address=104.36.82.2:4731 , piece_id=XIO3C6XJ76ELPR5DRKLYOZWS4NBAOHIIGJWRIPBWPBXS7F44WLGA, seg_pos={0 0}, upload=57174]: Uploading piece... Jul 26 23:11:21.895025497 (59h35m31.617588537s) [noise=true , address=45.91.116.11:4119 , piece_id=HA3WURU75MWEG2NZZLTID7FJIIR4UQTI72UP65YIGXB2C2WETSKA, seg_pos={0 0}, upload=57174]: Uploading piece... Jul 26 23:11:21.895170334 (59h35m31.617733374s) [noise=true , address=104.36.82.6:4876 , piece_id=O2LDPB2FMQEALJ3C7EA2WKOXENIVJX2IM67B2BRUEOBDMZSUWO6A, seg_pos={0 0}, upload=57174]: Uploading piece... Jul 26 23:11:21.895338275 (59h35m31.617901316s) [noise=true , address=104.36.82.9:4007 , piece_id=ZFU6E35MKEH74XRPJ32RICXEZ6JMG656QASWLQFLTUBSUYF4RY2Q, seg_pos={0 0}, upload=57174]: Uploading piece... Jul 26 23:11:21.895395887 (59h35m31.617958927s) [noise=true , address=104.36.82.16:4295 , piece_id=CR7ALFL2DJKLQE36B5H4FS5M2QYH3KXSVYVHBNVW3BBLNPLEAPEA, seg_pos={0 0}, upload=57174]: Uploading piece... Jul 26 23:11:21.895641395 (59h35m31.618204435s) [noise=true , address=104.36.82.6:4874 , piece_id=EQJGONMMZ2IUDP3CUXH3KO3Q5RTUMQMMDJ3APKMM7FIGR3FQEXIQ, seg_pos={0 0}, upload=57174]: Uploading piece... Jul 26 23:11:21.895776223 (59h35m31.618339254s) [noise=true , address=104.36.82.3:4767 , piece_id=M5OBU5ZPHW753QN2ZHDMH6TLMEQACY5RY4N5JWLJANDIFKPMYHZQ, seg_pos={0 0}, upload=57174]: Uploading piece... Jul 26 23:11:21.895845466 (59h35m31.618408507s) [noise=true , address=45.91.116.4:4940 , piece_id=IJHZ6LCHNQ7HFVH4SXMSGY7PUP77JCUCTBUIPA4LNDK6TM7ZPFGA, seg_pos={0 0}, upload=57174]: Uploading piece... Jul 26 23:11:21.895989963 (59h35m31.618553003s) [noise=true , address=104.36.82.15:4259 , piece_id=VJ5YBQNCTMKR7APXWEJTRR3AIWOGCSNX4FQEWOKCNSB6IJ6WOL2A, seg_pos={0 0}, upload=57174]: Uploading piece... Jul 26 23:11:21.896035470 (59h35m31.6185985s) [noise=true , address=45.91.116.7:4009 , piece_id=WL6NDLXFKTXPW4X4BK7ZJ55APPQGDGI7B4TC6TEICYZLFYGXW3RA, seg_pos={0 0}, upload=57174]: Uploading piece... Jul 26 23:11:21.896186308 (59h35m31.618749338s) [noise=true , address=45.91.116.6:4973 , piece_id=MY5N2L5QHIYLCUTVO2RTIGTHX3FSPM7UCONVOZH3375IWXNWCPAQ, seg_pos={0 0}, upload=57174]: Uploading piece... Jul 26 23:11:21.896447198 (59h35m31.619010228s) [noise=true , address=104.36.82.14:4223 , piece_id=4BI2IRYVHWDLOEJ6G24KNCBMWREWVQQNSODC5PP2SFKDYG64GJNQ, seg_pos={0 0}, upload=57174]: Uploading piece... Jul 26 23:11:23.655849980 (59h35m33.37841302s) [upload=57174]: Next returned nil segment Jul 26 23:11:23.655867724 (59h35m33.378430764s) [upload=57174]: Waiting for error group managing segments... Jul 26 23:11:23.953631217 (59h35m33.676194247s) [noise=true , address=104.36.82.16:4293 , piece_id=IDLIQYUAE7RZ7FP5UM7XEFS4R57LRSCABCE5GAGG4IZA36NUEYFQ, seg_pos={0 0}, upload=57174]: Done uploading piece. err: <nil> Jul 26 23:11:23.986343134 (59h35m33.708906174s) [noise=true , address=45.91.116.8:4047 , piece_id=2F5SP5PAKG4CJWOI5OJJSGNM3EOJRQOTUNQUKJL57BHLHGUI6Y4Q, seg_pos={0 0}, upload=57174]: Done uploading piece. err: <nil> Jul 26 23:11:23.990852642 (59h35m33.713415672s) [noise=true , address=45.91.116.6:4973 , piece_id=MY5N2L5QHIYLCUTVO2RTIGTHX3FSPM7UCONVOZH3375IWXNWCPAQ, seg_pos={0 0}, upload=57174]: Done uploading piece. err: <nil> Jul 26 23:11:23.998638429 (59h35m33.721201469s) [noise=true , address=45.91.116.4:4940 , piece_id=IJHZ6LCHNQ7HFVH4SXMSGY7PUP77JCUCTBUIPA4LNDK6TM7ZPFGA, seg_pos={0 0}, upload=57174]: Done uploading piece. err: <nil> Jul 26 23:11:24.001997466 (59h35m33.724560516s) [noise=true , address=45.91.116.7:4009 , piece_id=WL6NDLXFKTXPW4X4BK7ZJ55APPQGDGI7B4TC6TEICYZLFYGXW3RA, seg_pos={0 0}, upload=57174]: Done uploading piece. err: <nil> Jul 26 23:11:24.010224727 (59h35m33.732787767s) [noise=true , address=45.91.116.11:4119 , piece_id=HA3WURU75MWEG2NZZLTID7FJIIR4UQTI72UP65YIGXB2C2WETSKA, seg_pos={0 0}, upload=57174]: Done uploading piece. err: <nil> Jul 26 23:11:24.026829207 (59h35m33.749392247s) [noise=true , address=104.36.82.16:4295 , piece_id=CR7ALFL2DJKLQE36B5H4FS5M2QYH3KXSVYVHBNVW3BBLNPLEAPEA, seg_pos={0 0}, upload=57174]: Done uploading piece. err: <nil> Jul 26 23:11:50.992334832 (59h36m0.714897862s) [noise=true , address=104.36.82.2:4729 , piece_id=4VYHVFZY2DHSW5J47APEREXY5G4GUPPX5FHS3DK7LZQJGVKHCHQA, seg_pos={0 0}, upload=57174]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:104.36.82.2:4729): protocol: rpc: dial tcp 104.36.82.2:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:11:50.992384207 (59h36m0.714947247s) [noise=true , address=104.36.82.11:4115 , piece_id=PLT37RR5RPL24KNP7UNAK736KDPNHJFVGYGEGKKEBVPCF3F3Y2FQ, seg_pos={0 0}, upload=57174]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: rpc: dial tcp 104.36.82.11:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:11:50.992422590 (59h36m0.71498565s) [noise=true , address=104.36.82.9:4007 , piece_id=ZFU6E35MKEH74XRPJ32RICXEZ6JMG656QASWLQFLTUBSUYF4RY2Q, seg_pos={0 0}, upload=57174]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:11:50.992446225 (59h36m0.715009256s) [noise=true , address=104.36.82.6:4874 , piece_id=EQJGONMMZ2IUDP3CUXH3KO3Q5RTUMQMMDJ3APKMM7FIGR3FQEXIQ, seg_pos={0 0}, upload=57174]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:11:50.992470572 (59h36m0.715033612s) [noise=true , address=104.36.82.3:4767 , piece_id=M5OBU5ZPHW753QN2ZHDMH6TLMEQACY5RY4N5JWLJANDIFKPMYHZQ, seg_pos={0 0}, upload=57174]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: rpc: dial tcp 104.36.82.3:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:11:50.992489918 (59h36m0.715052959s) [noise=true , address=104.36.82.15:4259 , piece_id=VJ5YBQNCTMKR7APXWEJTRR3AIWOGCSNX4FQEWOKCNSB6IJ6WOL2A, seg_pos={0 0}, upload=57174]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: rpc: dial tcp 104.36.82.15:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:11:50.992515297 (59h36m0.715078327s) [noise=true , address=104.36.82.6:4876 , piece_id=O2LDPB2FMQEALJ3C7EA2WKOXENIVJX2IM67B2BRUEOBDMZSUWO6A, seg_pos={0 0}, upload=57174]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: rpc: dial tcp 104.36.82.6:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:11:50.992524235 (59h36m0.715087275s) [noise=true , address=104.36.82.14:4223 , piece_id=4BI2IRYVHWDLOEJ6G24KNCBMWREWVQQNSODC5PP2SFKDYG64GJNQ, seg_pos={0 0}, upload=57174]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: rpc: dial tcp 104.36.82.14:4223: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:11:50.992553771 (59h36m0.715116811s) [noise=true , address=104.36.82.2:4731 , piece_id=XIO3C6XJ76ELPR5DRKLYOZWS4NBAOHIIGJWRIPBWPBXS7F44WLGA, seg_pos={0 0}, upload=57174]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: write tcp 188.166.225.170:57358->104.36.82.2:4731: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:11:50.992581434 (59h36m0.715144474s) [seg_pos={0 0}, upload=57174]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jul 26 23:11:51.239414501 (59h36m0.961977541s) [upload=57174]: Done uploading segments... Jul 26 23:11:51.239439549 (59h36m0.962002589s) [upload=57174]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 30.248978958s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [X...X...............X..XX...X.X.X.X.......XX..XX.XX..XXX....] duration: 21.244579172s FAIL (2025-07-26 23:11:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 21.244605783s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 11 out of 60 is failed (threshold=3) [..X..........X.X........X.X...........................XXXXXX] duration: 30.235402032s FAIL (2025-07-26 23:11:49 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.235415027s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 1.909196664s FAIL (2025-07-26 23:11:38 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.909242692s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 52 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXX.X.XX.XX.X.XXXXXXXXX...XXXXXXXXXXXXXXXXX] duration: 1m0.22991813s FAIL (2025-07-26 23:11:26 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 9.127.219.38:4006 , piece_id=RAN6PN3GEGKKJ7ABJVM4RCYARRM5VBFOKSXAAKPWTUNBEQ3VM34Q, seg_pos={0 0}, upload=57157]: Uploading piece... Jul 26 23:10:27.725268569 (59h34m37.447831609s) [noise=true , address=104.36.82.54:4173 , piece_id=HJWAQ6WNNIIUU2ULF7S5GRAA7UXJAJ2YOF7AQJPCDXYDXBCGZA7Q, seg_pos={0 0}, upload=57157]: Uploading piece... Jul 26 23:10:27.725315097 (59h34m37.447878137s) [noise=true , address=79.127.205.246:4272 , piece_id=5DZDDAMZUZI63SMFYXD4PSVXV7ONIXMBKK45WSWVFRLGOP5YCABA, seg_pos={0 0}, upload=57157]: Uploading piece... Jul 26 23:10:27.725588550 (59h34m37.44815159s) [noise=true , address=79.127.201.215:4344 , piece_id=RD7Y4F5XUCLUDVRVARA7CLGOXPVH633DZHNBM6HXI4ODYNMJHXKQ, seg_pos={0 0}, upload=57157]: Uploading piece... Jul 26 23:10:27.725596276 (59h34m37.448159306s) [noise=true , address=77.90.31.16:4754 , piece_id=FQP4KCQRQ32MAXBDDR63ZLN6YGQT7SBZRSLAKVOET2OCWEAPYTYA, seg_pos={0 0}, upload=57157]: Uploading piece... Jul 26 23:10:27.725850592 (59h34m37.448413632s) [noise=true , address=79.127.219.47:4048 , piece_id=COZVG55BDVSOP4LFRPFHCKYPNGY5XFTIRK2UNBZ7MFS3WLM2TKVQ, seg_pos={0 0}, upload=57157]: Uploading piece... Jul 26 23:10:27.725862835 (59h34m37.448425875s) [noise=true , address=77.90.31.11:4577 , piece_id=KCGI6KCFPBGN4B2XU2HMQ66LKOKV6FPCMAHWKMUKUXFP5KKZVGKA, seg_pos={0 0}, upload=57157]: Uploading piece... Jul 26 23:10:27.726102384 (59h34m37.448665424s) [noise=true , address=79.127.205.235:4690 , piece_id=UPBYMUENHM7YOHO6H3ZTAUW424YRA3452MX4XMVITTRECB7STCTA, seg_pos={0 0}, upload=57157]: Uploading piece... Jul 26 23:10:27.726219598 (59h34m37.448782638s) [noise=true , address=77.90.31.21:4897 , piece_id=NKHVNW75EJZIEXD4WSJPLIMZ26SAPX3CUPIH3I3FOF3IPGPAFWTQ, seg_pos={0 0}, upload=57157]: Uploading piece... Jul 26 23:10:27.726837641 (59h34m37.449400681s) [noise=true , address=156.146.43.225:4448 , piece_id=JRBYHQ6MA7JWG6MRL6CLUSLWAYOQ5QLJTJGKALCWDZ7SR72GYYGQ, seg_pos={0 0}, upload=57157]: Uploading piece... Jul 26 23:10:27.727046069 (59h34m37.449609109s) [noise=true , address=104.36.82.46:4498 , piece_id=2KDKCQJNNVQTAQULPNNZOP3LZ6HS5STSC4K7WLMQVXFZ524JOBFQ, seg_pos={0 0}, upload=57157]: Uploading piece... Jul 26 23:10:27.727269527 (59h34m37.449832567s) [noise=true , address=64.49.13.137:4502 , piece_id=PPABPXMDGUKAMFSC7NM23K74HBB2EJAPPAWPLQBIPGMCTV6EQGNQ, seg_pos={0 0}, upload=57157]: Uploading piece... Jul 26 23:10:27.727459600 (59h34m37.45002264s) [noise=true , address=79.127.205.234:4609 , piece_id=EDDRPGM2HSDKNTAAWTCMY2MTZEJLAMQ4T6QK2J546BDED56QJ5EA, seg_pos={0 0}, upload=57157]: Uploading piece... Jul 26 23:10:27.727650886 (59h34m37.450213916s) [noise=true , address=104.36.82.56:4239 , piece_id=56LTUTYLA47HQIZI3BTJZEWTU6KIET3HV7DPCH4DNHZQ6FEIFR5A, seg_pos={0 0}, upload=57157]: Uploading piece... Jul 26 23:10:27.727830951 (59h34m37.450393981s) [noise=true , address=104.36.82.47:4533 , piece_id=OIEN24ZB7ZUKSW76D7JF4E5MSOVMXG5HBNWFBTFPR3DUU4M7N6NA, seg_pos={0 0}, upload=57157]: Uploading piece... Jul 26 23:10:27.728008400 (59h34m37.45057144s) [noise=true , address=77.90.31.12:4619 , piece_id=S3L7NFL65JEXVVF5JLBB2SXOUBS6O3KAXDWP6LYUGSBOUXRHBKHA, seg_pos={0 0}, upload=57157]: Uploading piece... Jul 26 23:10:27.728914945 (59h34m37.451477985s) [noise=true , address=104.36.82.55:4228 , piece_id=JWS5HBCBUN4I2XV5UQI7FTQLBUXJW2264FGJ4AEV3WL4TFBBU4KQ, seg_pos={0 0}, upload=57157]: Uploading piece... Jul 26 23:10:27.729117272 (59h34m37.451680312s) [noise=true , address=77.90.31.17:4799 , piece_id=CLYXP6WV4AH3WLLQ4FSQEWM6BWYG7YX2GNCOQY2VH7ZJDQMRZULQ, seg_pos={0 0}, upload=57157]: Uploading piece... Jul 26 23:10:28.813503573 (59h34m38.536066623s) [upload=57157]: Next returned nil segment Jul 26 23:10:28.813526156 (59h34m38.536089206s) [upload=57157]: Waiting for error group managing segments... Jul 26 23:10:29.105923411 (59h34m38.828486451s) [noise=true , address=79.127.205.246:4272 , piece_id=5DZDDAMZUZI63SMFYXD4PSVXV7ONIXMBKK45WSWVFRLGOP5YCABA, seg_pos={0 0}, upload=57157]: Done uploading piece. err: <nil> Jul 26 23:10:29.123889836 (59h34m38.846452876s) [noise=true , address=79.127.205.234:4609 , piece_id=EDDRPGM2HSDKNTAAWTCMY2MTZEJLAMQ4T6QK2J546BDED56QJ5EA, seg_pos={0 0}, upload=57157]: Done uploading piece. err: <nil> Jul 26 23:10:29.136541782 (59h34m38.859104822s) [noise=true , address=79.127.205.235:4690 , piece_id=UPBYMUENHM7YOHO6H3ZTAUW424YRA3452MX4XMVITTRECB7STCTA, seg_pos={0 0}, upload=57157]: Done uploading piece. err: <nil> Jul 26 23:10:29.694645936 (59h34m39.417208976s) [noise=true , address=77.90.31.21:4897 , piece_id=NKHVNW75EJZIEXD4WSJPLIMZ26SAPX3CUPIH3I3FOF3IPGPAFWTQ, seg_pos={0 0}, upload=57157]: Done uploading piece. err: <nil> Jul 26 23:10:29.731616352 (59h34m39.454179392s) [noise=true , address=79.127.219.38:4006 , piece_id=RAN6PN3GEGKKJ7ABJVM4RCYARRM5VBFOKSXAAKPWTUNBEQ3VM34Q, seg_pos={0 0}, upload=57157]: Done uploading piece. err: <nil> Jul 26 23:10:29.750532334 (59h34m39.473095374s) [noise=true , address=77.90.31.11:4577 , piece_id=KCGI6KCFPBGN4B2XU2HMQ66LKOKV6FPCMAHWKMUKUXFP5KKZVGKA, seg_pos={0 0}, upload=57157]: Done uploading piece. err: <nil> Jul 26 23:10:31.573921475 (59h34m41.296484515s) [noise=true , address=77.90.31.16:4754 , piece_id=FQP4KCQRQ32MAXBDDR63ZLN6YGQT7SBZRSLAKVOET2OCWEAPYTYA, seg_pos={0 0}, upload=57157]: Done uploading piece. err: <nil> Jul 26 23:10:33.957310786 (59h34m43.679873816s) [noise=true , address=79.127.219.47:4048 , piece_id=COZVG55BDVSOP4LFRPFHCKYPNGY5XFTIRK2UNBZ7MFS3WLM2TKVQ, seg_pos={0 0}, upload=57157]: Done uploading piece. err: <nil> Jul 26 23:10:36.042396793 (59h34m45.764959863s) [noise=true , address=156.146.43.225:4448 , piece_id=JRBYHQ6MA7JWG6MRL6CLUSLWAYOQ5QLJTJGKALCWDZ7SR72GYYGQ, seg_pos={0 0}, upload=57157]: Done uploading piece. err: <nil> Jul 26 23:10:39.479044569 (59h34m49.201607609s) [noise=true , address=79.127.201.215:4344 , piece_id=RD7Y4F5XUCLUDVRVARA7CLGOXPVH633DZHNBM6HXI4ODYNMJHXKQ, seg_pos={0 0}, upload=57157]: Done uploading piece. err: <nil> Jul 26 23:10:42.984704478 (59h34m52.707267518s) [noise=true , address=64.49.13.137:4502 , piece_id=PPABPXMDGUKAMFSC7NM23K74HBB2EJAPPAWPLQBIPGMCTV6EQGNQ, seg_pos={0 0}, upload=57157]: Done uploading piece. err: <nil> Jul 26 23:11:26.630508677 (59h35m36.353071717s) [noise=true , address=104.36.82.54:4173 , piece_id=HJWAQ6WNNIIUU2ULF7S5GRAA7UXJAJ2YOF7AQJPCDXYDXBCGZA7Q, seg_pos={0 0}, upload=57157]: Done uploading piece. err: ecclient: upload failed (node:1w2v4iyYtJtcdz7shYhG8CjCDQ1dWjeJVwY5Ubi7Ex5Y38e2i1, address:104.36.82.54:4173): protocol: rpc: dial tcp 104.36.82.54:4173: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:11:26.630560496 (59h35m36.353123536s) [noise=true , address=77.90.31.17:4799 , piece_id=CLYXP6WV4AH3WLLQ4FSQEWM6BWYG7YX2GNCOQY2VH7ZJDQMRZULQ, seg_pos={0 0}, upload=57157]: Done uploading piece. err: ecclient: upload failed (node:12UhAksgXZuFGAVheaYzP7D4kwVyAjUAq3BtrvdSxszjNCFCTWT, address:77.90.31.17:4799): protocol: rpc: dial tcp 77.90.31.17:4799: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:11:26.630606485 (59h35m36.353169525s) [noise=true , address=104.36.82.56:4239 , piece_id=56LTUTYLA47HQIZI3BTJZEWTU6KIET3HV7DPCH4DNHZQ6FEIFR5A, seg_pos={0 0}, upload=57157]: Done uploading piece. err: ecclient: upload failed (node:126Feu3gWgog12YavTHZkkUKK8wq6RynV8zsWd3WUAUYjvQwBnc, address:104.36.82.56:4239): protocol: write tcp 188.166.225.170:50296->104.36.82.56:4239: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:11:26.630612536 (59h35m36.353175576s) [noise=true , address=104.36.82.46:4498 , piece_id=2KDKCQJNNVQTAQULPNNZOP3LZ6HS5STSC4K7WLMQVXFZ524JOBFQ, seg_pos={0 0}, upload=57157]: Done uploading piece. err: ecclient: upload failed (node:1jyjxdWkBRBkDHSKknNmPdcnJ3HRcSCUQNnmpb1CaiEdMPEGk4, address:104.36.82.46:4498): protocol: write tcp 188.166.225.170:41190->104.36.82.46:4498: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:11:26.630617806 (59h35m36.353180846s) [noise=true , address=104.36.82.47:4533 , piece_id=OIEN24ZB7ZUKSW76D7JF4E5MSOVMXG5HBNWFBTFPR3DUU4M7N6NA, seg_pos={0 0}, upload=57157]: Done uploading piece. err: ecclient: upload failed (node:1JUUdAnaUPAE8tkzPvVCokFbZMwmhqZvwKHXReK9VU2QeDqavR, address:104.36.82.47:4533): protocol: write tcp 188.166.225.170:48288->104.36.82.47:4533: use of closed network connection; write tcp 188.166.225.170:48288->104.36.82.47:4533: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48288->104.36.82.47:4533: use of closed network connection Jul 26 23:11:26.630622616 (59h35m36.353185656s) [noise=true , address=77.90.31.12:4619 , piece_id=S3L7NFL65JEXVVF5JLBB2SXOUBS6O3KAXDWP6LYUGSBOUXRHBKHA, seg_pos={0 0}, upload=57157]: Done uploading piece. err: ecclient: upload failed (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX, address:77.90.31.12:4619): protocol: rpc: dial tcp 77.90.31.12:4619: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:11:26.630638135 (59h35m36.353201165s) [noise=true , address=104.36.82.55:4228 , piece_id=JWS5HBCBUN4I2XV5UQI7FTQLBUXJW2264FGJ4AEV3WL4TFBBU4KQ, seg_pos={0 0}, upload=57157]: Done uploading piece. err: ecclient: upload failed (node:12NBsd33JtAwPdcbD8vBzZUYpjiqZpRzs8y2QrYJFE245pd8T5X, address:104.36.82.55:4228): protocol: write tcp 188.166.225.170:47846->104.36.82.55:4228: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:11:26.630663924 (59h35m36.353226965s) [seg_pos={0 0}, upload=57157]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 26 23:11:26.858862147 (59h35m36.581425187s) [upload=57157]: Done uploading segments... Jul 26 23:11:26.858888137 (59h35m36.581451177s) [upload=57157]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1m0.229930344s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 10 out of 60 is failed (threshold=3) [...X..........X.X........X.X...........................XXXXX] duration: 30.242398351s FAIL (2025-07-26 23:11:19 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.242417818s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 1.867915572s FAIL (2025-07-26 23:11:18 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.867946009s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [......................................................XXX.X.] duration: 6.628724982s FAIL (2025-07-26 23:11:10 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.628747485s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [.X...X...............X..XX...X.X.X.X.......XX..XX.XX..XXX...] duration: 47.793636319s FAIL (2025-07-26 23:11:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 47.793659373s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [........................X.........X...........X.......XXXXXX] duration: 30.231291203s FAIL (2025-07-26 23:10:59 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 9.844625234s) [upload=57159]: Starting upload Jul 26 23:10:30.122147727 (59h34m39.844710757s) [upload=57159]: Done starting upload Jul 26 23:10:30.122270512 (59h34m39.844833552s) [upload=57159]: Uploading segments... Jul 26 23:10:30.122370884 (59h34m39.844933924s) [upload=57159]: Got next segment. Inline: false Jul 26 23:10:30.122414146 (59h34m39.844977186s) [upload=57159]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:10:30.122419577 (59h34m39.844982607s) [upload=57159]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:10:30.357136201 (59h34m40.079699231s) [seg_pos={0 0}, upload=57159]: Begin upload segment... Jul 26 23:10:30.357187289 (59h34m40.079750329s) [seg_pos={0 0}, upload=57159]: Done begin upload segment. Jul 26 23:10:30.357208590 (59h34m40.07977163s) [noise=true , address=104.36.82.16:4293 , piece_id=24VWIYDK5FMIRSOOOHGXCN5EKEN3JNII2BS6N6CIV4FQY4TQ5HQA, seg_pos={0 0}, upload=57159]: Uploading piece... Jul 26 23:10:30.357246562 (59h34m40.079809603s) [noise=true , address=45.91.116.6:4973 , piece_id=K4DR5I676LMIGOOIBA63MU6QJ543BCICP5RXMNT4Y3D77YQ4PXXA, seg_pos={0 0}, upload=57159]: Uploading piece... Jul 26 23:10:30.357714828 (59h34m40.080277858s) [noise=true , address=45.91.116.4:4938 , piece_id=UIPFHT4VPGEUBREKC6VB34EP6QBQNWVYDO7NP2QY4SIASMKUSIRA, seg_pos={0 0}, upload=57159]: Uploading piece... Jul 26 23:10:30.357721811 (59h34m40.080284852s) [noise=true , address=104.36.82.2:4729 , piece_id=LLTZV4IIDCB6LZ5CWNUPY7JXBYKFRT2M2X4VGTNFHAYGPBYAVU3Q, seg_pos={0 0}, upload=57159]: Uploading piece... Jul 26 23:10:30.358028117 (59h34m40.080591157s) [noise=true , address=104.36.82.15:4259 , piece_id=BGIAXLIRPECU36HOIX3SQXXENFX5SED3OW7VJQB5KZXDGG57V7HQ, seg_pos={0 0}, upload=57159]: Uploading piece... Jul 26 23:10:30.358281171 (59h34m40.080844212s) [noise=true , address=104.36.82.8:4972 , piece_id=ZYOABU7WYVXGADWHJ6KRSIJVY2XPZRTF6OJ62I4FTC6WW4MX62NQ, seg_pos={0 0}, upload=57159]: Uploading piece... Jul 26 23:10:30.362397788 (59h34m40.084960828s) [noise=true , address=45.91.116.12:4226 , piece_id=YQ226CJZDTOROZ57LB5RAMSNW5K7R24UHE6VY5572OKZRJCWSKIQ, seg_pos={0 0}, upload=57159]: Uploading piece... Jul 26 23:10:30.362704845 (59h34m40.085267885s) [noise=true , address=104.36.82.6:4875 , piece_id=2LLDHIA2WZQHUR2TPBVDG3O75BXXGHOLQXR6OYPCC4VT7VTJN4FA, seg_pos={0 0}, upload=57159]: Uploading piece... Jul 26 23:10:30.362930798 (59h34m40.085493838s) [noise=true , address=45.91.116.7:4011 , piece_id=IZCYIOT6LDUZGQ3VTBZGAQNWRGIOIGQJVXBOLYBKRBLZ43FGT6TA, seg_pos={0 0}, upload=57159]: Uploading piece... Jul 26 23:10:30.363119709 (59h34m40.085682739s) [noise=true , address=104.36.82.2:4730 , piece_id=3JRRNAB6LSEDHQ4KWK26V6UU2ERVEVUK5MWY7A4OERHSISIFN3QA, seg_pos={0 0}, upload=57159]: Uploading piece... Jul 26 23:10:30.363319782 (59h34m40.085882823s) [noise=true , address=104.36.82.11:4114 , piece_id=KO7QSJE4EJAXV2MOSPNQHXYBNJK5AIFJA3KPUO6YWRHGIPSS5YGQ, seg_pos={0 0}, upload=57159]: Uploading piece... Jul 26 23:10:30.363583496 (59h34m40.086146537s) [noise=true , address=45.91.116.6:4974 , piece_id=CQZ54M5UD43GSQLSOEH4DKBAFGAZRM3HLO2Q523SWXRCQUKMKM5Q, seg_pos={0 0}, upload=57159]: Uploading piece... Jul 26 23:10:30.363766587 (59h34m40.086329627s) [noise=true , address=45.91.116.13:4154 , piece_id=QHSFIIJ4EVESY3AHENPDOLK4EBR6M4CKJG7WXRN75L2SOU7FY3GA, seg_pos={0 0}, upload=57159]: Uploading piece... Jul 26 23:10:30.363988622 (59h34m40.086551662s) [noise=true , address=77.90.31.15:4698 , piece_id=CMTLQH2QL4C7Y2OLQMW7CYL6V2MZSOF2U6J23HIX7JGDHBDHIWNA, seg_pos={0 0}, upload=57159]: Uploading piece... Jul 26 23:10:30.364306641 (59h34m40.086869681s) [noise=true , address=45.91.116.7:4010 , piece_id=YDAKQDRK2HXBWEWIWJ7LKN4HGOKRBZJHY4XL7LRIWTGPL5YAFWCQ, seg_pos={0 0}, upload=57159]: Uploading piece... Jul 26 23:10:30.364550647 (59h34m40.087113687s) [noise=true , address=104.36.82.4:4802 , piece_id=SP2MRSG6KS3B5IZEAE7XRADQ4ZPYUWWBV5FZ7MZAFC3Q3CPGQWZQ, seg_pos={0 0}, upload=57159]: Uploading piece... Jul 26 23:10:32.074067015 (59h34m41.796630055s) [upload=57159]: Next returned nil segment Jul 26 23:10:32.074092844 (59h34m41.796655874s) [upload=57159]: Waiting for error group managing segments... Jul 26 23:10:32.386261320 (59h34m42.10882436s) [noise=true , address=45.91.116.4:4938 , piece_id=UIPFHT4VPGEUBREKC6VB34EP6QBQNWVYDO7NP2QY4SIASMKUSIRA, seg_pos={0 0}, upload=57159]: Done uploading piece. err: <nil> Jul 26 23:10:32.386445211 (59h34m42.109008251s) [noise=true , address=45.91.116.7:4010 , piece_id=YDAKQDRK2HXBWEWIWJ7LKN4HGOKRBZJHY4XL7LRIWTGPL5YAFWCQ, seg_pos={0 0}, upload=57159]: Done uploading piece. err: <nil> Jul 26 23:10:32.388309739 (59h34m42.110872779s) [noise=true , address=45.91.116.7:4011 , piece_id=IZCYIOT6LDUZGQ3VTBZGAQNWRGIOIGQJVXBOLYBKRBLZ43FGT6TA, seg_pos={0 0}, upload=57159]: Done uploading piece. err: <nil> Jul 26 23:10:32.460036263 (59h34m42.182599353s) [noise=true , address=45.91.116.12:4226 , piece_id=YQ226CJZDTOROZ57LB5RAMSNW5K7R24UHE6VY5572OKZRJCWSKIQ, seg_pos={0 0}, upload=57159]: Done uploading piece. err: <nil> Jul 26 23:10:32.472645098 (59h34m42.195208128s) [noise=true , address=45.91.116.13:4154 , piece_id=QHSFIIJ4EVESY3AHENPDOLK4EBR6M4CKJG7WXRN75L2SOU7FY3GA, seg_pos={0 0}, upload=57159]: Done uploading piece. err: <nil> Jul 26 23:10:32.473069570 (59h34m42.19563261s) [noise=true , address=45.91.116.6:4974 , piece_id=CQZ54M5UD43GSQLSOEH4DKBAFGAZRM3HLO2Q523SWXRCQUKMKM5Q, seg_pos={0 0}, upload=57159]: Done uploading piece. err: <nil> Jul 26 23:10:32.595435483 (59h34m42.317998704s) [noise=true , address=45.91.116.6:4973 , piece_id=K4DR5I676LMIGOOIBA63MU6QJ543BCICP5RXMNT4Y3D77YQ4PXXA, seg_pos={0 0}, upload=57159]: Done uploading piece. err: <nil> Jul 26 23:10:33.442396859 (59h34m43.164959899s) [noise=true , address=77.90.31.15:4698 , piece_id=CMTLQH2QL4C7Y2OLQMW7CYL6V2MZSOF2U6J23HIX7JGDHBDHIWNA, seg_pos={0 0}, upload=57159]: Done uploading piece. err: <nil> Jul 26 23:10:59.477235607 (59h35m9.199798647s) [noise=true , address=104.36.82.4:4802 , piece_id=SP2MRSG6KS3B5IZEAE7XRADQ4ZPYUWWBV5FZ7MZAFC3Q3CPGQWZQ, seg_pos={0 0}, upload=57159]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 188.166.225.170:34324->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:10:59.477325007 (59h35m9.199888048s) [noise=true , address=104.36.82.16:4293 , piece_id=24VWIYDK5FMIRSOOOHGXCN5EKEN3JNII2BS6N6CIV4FQY4TQ5HQA, seg_pos={0 0}, upload=57159]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: rpc: dial tcp 104.36.82.16:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:10:59.477361247 (59h35m9.199924278s) [noise=true , address=104.36.82.11:4114 , piece_id=KO7QSJE4EJAXV2MOSPNQHXYBNJK5AIFJA3KPUO6YWRHGIPSS5YGQ, seg_pos={0 0}, upload=57159]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 188.166.225.170:50958->104.36.82.11:4114: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:10:59.477374633 (59h35m9.199937673s) [noise=true , address=104.36.82.15:4259 , piece_id=BGIAXLIRPECU36HOIX3SQXXENFX5SED3OW7VJQB5KZXDGG57V7HQ, seg_pos={0 0}, upload=57159]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: rpc: dial tcp 104.36.82.15:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:10:59.477412746 (59h35m9.199975786s) [noise=true , address=104.36.82.2:4730 , piece_id=3JRRNAB6LSEDHQ4KWK26V6UU2ERVEVUK5MWY7A4OERHSISIFN3QA, seg_pos={0 0}, upload=57159]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 188.166.225.170:57184->104.36.82.2:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:10:59.477432012 (59h35m9.199995042s) [noise=true , address=104.36.82.6:4875 , piece_id=2LLDHIA2WZQHUR2TPBVDG3O75BXXGHOLQXR6OYPCC4VT7VTJN4FA, seg_pos={0 0}, upload=57159]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: rpc: dial tcp 104.36.82.6:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:10:59.477437263 (59h35m9.200000303s) [noise=true , address=104.36.82.2:4729 , piece_id=LLTZV4IIDCB6LZ5CWNUPY7JXBYKFRT2M2X4VGTNFHAYGPBYAVU3Q, seg_pos={0 0}, upload=57159]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:104.36.82.2:4729): protocol: rpc: dial tcp 104.36.82.2:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:10:59.477442313 (59h35m9.200005353s) [noise=true , address=104.36.82.8:4972 , piece_id=ZYOABU7WYVXGADWHJ6KRSIJVY2XPZRTF6OJ62I4FTC6WW4MX62NQ, seg_pos={0 0}, upload=57159]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: rpc: dial tcp 104.36.82.8:4972: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:10:59.477477520 (59h35m9.20004056s) [seg_pos={0 0}, upload=57159]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 26 23:10:59.707152488 (59h35m9.429715528s) [upload=57159]: Done uploading segments... Jul 26 23:10:59.707182856 (59h35m9.429745886s) [upload=57159]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.231309429s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......................................................XXX.X] duration: 30.00050145s FAIL (2025-07-26 23:10:48 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.00052268s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 1.875307128s FAIL (2025-07-26 23:10:42 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.875352405s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 52 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXX.X.XX.XX.X.XXXXXXXXX...XXXXXXXXXXXXXXXX] duration: 1m0.237348594s FAIL (2025-07-26 23:10:26 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== Uploading piece... Jul 26 23:09:27.267997766 (59h33m36.990560796s) [noise=true , address=104.36.82.54:4174 , piece_id=UHJU5O227UBIHLGEZ575LPBK42LZQRTH2FPOWC4I6S2WS3XGDDCA, seg_pos={0 0}, upload=57142]: Uploading piece... Jul 26 23:09:27.268325824 (59h33m36.990888864s) [noise=true , address=45.91.116.9:4277 , piece_id=QACZNEFXAMJL7TY3DWQSQLZXPL2OX35VK43NKSGVXGNWTBIXC2FA, seg_pos={0 0}, upload=57142]: Uploading piece... Jul 26 23:09:27.268406318 (59h33m36.990969358s) [noise=true , address=79.127.205.233:4623 , piece_id=2WXB6AFH54XZ5V25HVRWIFIZ6FNBL3QY3PHDVVMLSSKQULZ5XREQ, seg_pos={0 0}, upload=57142]: Uploading piece... Jul 26 23:09:27.268568749 (59h33m36.991131789s) [noise=true , address=104.36.82.53:4156 , piece_id=SBHXE7AABK2U2NTD4JPNTPRZ7ZRFU3M2PMUA2BAUKPV2DNVMIBCQ, seg_pos={0 0}, upload=57142]: Uploading piece... Jul 26 23:09:27.268714238 (59h33m36.991277278s) [noise=true , address=79.127.205.246:4270 , piece_id=WMREQRQ7GZNZMUJPY52D7MRI3U5TDWILHWW5FPT5ESSDPARRJZVQ, seg_pos={0 0}, upload=57142]: Uploading piece... Jul 26 23:09:27.268803929 (59h33m36.991366969s) [noise=true , address=104.36.82.56:4255 , piece_id=WFAQ2KJWPAOJ4PV43OJMX2I5NKPHCEI4BKEH6VZKOHOEBEPCK3TQ, seg_pos={0 0}, upload=57142]: Uploading piece... Jul 26 23:09:27.269002409 (59h33m36.991565449s) [noise=true , address=104.36.82.46:4486 , piece_id=PF5UMCDXFU5SERHVCMRWNLZRJHG2PXKZ3SQWKFZTMGZC2K5BCCFQ, seg_pos={0 0}, upload=57142]: Uploading piece... Jul 26 23:09:27.269047876 (59h33m36.991610906s) [noise=true , address=45.91.116.14:4194 , piece_id=VSM3Y44E3QCASVVCE5QT6GYOFQ3SF5OTLIFSEVM6577YJTH4WWIA, seg_pos={0 0}, upload=57142]: Uploading piece... Jul 26 23:09:27.269287454 (59h33m36.991850494s) [noise=true , address=79.127.205.235:4659 , piece_id=VFHSMNWC6XRFIHT4ZCOX742RR7L6O4STCSDRH6JJSOQ3MFVBBZMQ, seg_pos={0 0}, upload=57142]: Uploading piece... Jul 26 23:09:27.269290981 (59h33m36.991854011s) [noise=true , address=77.90.31.21:4889 , piece_id=2VWXDM4LFONCJTEI2F3E3777TX5BQCNWF3HNTACHDQYLVIVVNBLA, seg_pos={0 0}, upload=57142]: Uploading piece... Jul 26 23:09:27.269520099 (59h33m36.992083139s) [noise=true , address=104.36.82.55:4195 , piece_id=R5PHE7ABWAIPT5T5ZIE3RG3KXQRNZLKUU5UUL5B5JLNL62RW7FZA, seg_pos={0 0}, upload=57142]: Uploading piece... Jul 26 23:09:27.269739419 (59h33m36.992302459s) [noise=true , address=156.146.43.225:4448 , piece_id=R6JUTB4R344J4AHJWPAI5EYIEEB6CAUGMU4MXDIMSAJ5FVVSB5QQ, seg_pos={0 0}, upload=57142]: Uploading piece... Jul 26 23:09:27.271248436 (59h33m36.993811476s) [noise=true , address=104.36.82.52:4115 , piece_id=CQTBKIPBKIMXZN25ZNV6XQCLZIQJW6C77BYAQDUCNIEA4XZSDLLQ, seg_pos={0 0}, upload=57142]: Uploading piece... Jul 26 23:09:27.271635897 (59h33m36.994198938s) [noise=true , address=64.49.13.155:4923 , piece_id=W7FRJ2ZIPEECOG3PP66TPD47FTSBQAC2GR5XWJWOYBTIBXFVGJXA, seg_pos={0 0}, upload=57142]: Uploading piece... Jul 26 23:09:27.277236182 (59h33m36.999799222s) [noise=true , address=77.90.31.11:4557 , piece_id=REC65VLTLVHCEFEPKNJU4OSDF6EEBSZHGFVGJEW7AMCOOB2KIXZA, seg_pos={0 0}, upload=57142]: Uploading piece... Jul 26 23:09:28.356742997 (59h33m38.079306037s) [upload=57142]: Next returned nil segment Jul 26 23:09:28.356760731 (59h33m38.079323761s) [upload=57142]: Waiting for error group managing segments... Jul 26 23:09:28.573652030 (59h33m38.29621507s) [noise=true , address=45.91.116.9:4277 , piece_id=QACZNEFXAMJL7TY3DWQSQLZXPL2OX35VK43NKSGVXGNWTBIXC2FA, seg_pos={0 0}, upload=57142]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: storage node overloaded, request limit: 2000 Jul 26 23:09:28.646956514 (59h33m38.369519554s) [noise=true , address=79.127.205.233:4623 , piece_id=2WXB6AFH54XZ5V25HVRWIFIZ6FNBL3QY3PHDVVMLSSKQULZ5XREQ, seg_pos={0 0}, upload=57142]: Done uploading piece. err: <nil> Jul 26 23:09:28.658011866 (59h33m38.380574906s) [noise=true , address=79.127.205.235:4659 , piece_id=VFHSMNWC6XRFIHT4ZCOX742RR7L6O4STCSDRH6JJSOQ3MFVBBZMQ, seg_pos={0 0}, upload=57142]: Done uploading piece. err: <nil> Jul 26 23:09:28.682372176 (59h33m38.404935216s) [noise=true , address=79.127.205.246:4270 , piece_id=WMREQRQ7GZNZMUJPY52D7MRI3U5TDWILHWW5FPT5ESSDPARRJZVQ, seg_pos={0 0}, upload=57142]: Done uploading piece. err: <nil> Jul 26 23:09:28.688806486 (59h33m38.411369556s) [noise=true , address=64.49.13.155:4923 , piece_id=W7FRJ2ZIPEECOG3PP66TPD47FTSBQAC2GR5XWJWOYBTIBXFVGJXA, seg_pos={0 0}, upload=57142]: Done uploading piece. err: <nil> Jul 26 23:09:28.821391611 (59h33m38.543954642s) [noise=true , address=156.146.43.225:4448 , piece_id=R6JUTB4R344J4AHJWPAI5EYIEEB6CAUGMU4MXDIMSAJ5FVVSB5QQ, seg_pos={0 0}, upload=57142]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jul 26 23:09:29.205843575 (59h33m38.928406615s) [noise=true , address=77.90.31.11:4557 , piece_id=REC65VLTLVHCEFEPKNJU4OSDF6EEBSZHGFVGJEW7AMCOOB2KIXZA, seg_pos={0 0}, upload=57142]: Done uploading piece. err: <nil> Jul 26 23:09:29.271847401 (59h33m38.994410431s) [noise=true , address=45.91.116.14:4194 , piece_id=VSM3Y44E3QCASVVCE5QT6GYOFQ3SF5OTLIFSEVM6577YJTH4WWIA, seg_pos={0 0}, upload=57142]: Done uploading piece. err: ecclient: upload failed (node:19mdk9cXDR79ZXC4n6rMXFUhpwxMPXFpkPA14ymY24RK1jKkmd, address:45.91.116.14:4194): protocol: storage node overloaded, request limit: 2000 Jul 26 23:09:29.316980104 (59h33m39.039543144s) [noise=true , address=77.90.31.21:4889 , piece_id=2VWXDM4LFONCJTEI2F3E3777TX5BQCNWF3HNTACHDQYLVIVVNBLA, seg_pos={0 0}, upload=57142]: Done uploading piece. err: <nil> Jul 26 23:09:29.500832930 (59h33m39.22339597s) [noise=true , address=77.90.31.19:4871 , piece_id=GPVWV67BZEVDOVXRUEHBMCELJFL2ZAQWKCNILM7HYLSX3S66OWLA, seg_pos={0 0}, upload=57142]: Done uploading piece. err: <nil> Jul 26 23:10:26.154954881 (59h34m35.877517921s) [noise=true , address=104.36.82.54:4174 , piece_id=UHJU5O227UBIHLGEZ575LPBK42LZQRTH2FPOWC4I6S2WS3XGDDCA, seg_pos={0 0}, upload=57142]: Done uploading piece. err: ecclient: upload failed (node:1293SNDGTKNrYrKDuB5p2VpHWo7ANpQsVKVGa97jXVaGP83Pw51, address:104.36.82.54:4174): protocol: rpc: dial tcp 104.36.82.54:4174: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:10:26.154966733 (59h34m35.877529773s) [noise=true , address=104.36.82.47:4545 , piece_id=AKFGF57WCKTFML7SOJZFZLO6WJQO6HQCMILSZHB7N7OQMXLHCTRA, seg_pos={0 0}, upload=57142]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: write tcp 188.166.225.170:57736->104.36.82.47:4545: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:10:26.154992452 (59h34m35.877555482s) [noise=true , address=104.36.82.55:4195 , piece_id=R5PHE7ABWAIPT5T5ZIE3RG3KXQRNZLKUU5UUL5B5JLNL62RW7FZA, seg_pos={0 0}, upload=57142]: Done uploading piece. err: ecclient: upload failed (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa, address:104.36.82.55:4195): protocol: rpc: dial tcp 104.36.82.55:4195: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:10:26.155029864 (59h34m35.877592904s) [noise=true , address=77.90.31.17:4780 , piece_id=ONWKN2QNZZ24IZLAKMJBNKCHX4THV6YIOBLWN7ATXYQSAR66UD6A, seg_pos={0 0}, upload=57142]: Done uploading piece. err: ecclient: upload failed (node:1XfvmVdBudtFnKwNxCxuGbQMS6HSSkVwPkYmffa5X6e6uYNcuF, address:77.90.31.17:4780): protocol: write tcp 188.166.225.170:36090->77.90.31.17:4780: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:10:26.155035645 (59h34m35.877598675s) [noise=true , address=104.36.82.56:4255 , piece_id=WFAQ2KJWPAOJ4PV43OJMX2I5NKPHCEI4BKEH6VZKOHOEBEPCK3TQ, seg_pos={0 0}, upload=57142]: Done uploading piece. err: ecclient: upload failed (node:12S6rwimXBvaEb4KneAwL5H3t5zTmvGVhyJSWCTU1u17r4RWJta, address:104.36.82.56:4255): protocol: rpc: dial tcp 104.36.82.56:4255: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:10:26.155059391 (59h34m35.877622431s) [noise=true , address=104.36.82.46:4486 , piece_id=PF5UMCDXFU5SERHVCMRWNLZRJHG2PXKZ3SQWKFZTMGZC2K5BCCFQ, seg_pos={0 0}, upload=57142]: Done uploading piece. err: ecclient: upload failed (node:1fbs2btcbiu2yYGrvxNZe8sNSqDnNPG1Bmfyhcew86FCFLKAic, address:104.36.82.46:4486): protocol: rpc: dial tcp 104.36.82.46:4486: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:10:26.155077897 (59h34m35.877640927s) [noise=true , address=104.36.82.52:4115 , piece_id=CQTBKIPBKIMXZN25ZNV6XQCLZIQJW6C77BYAQDUCNIEA4XZSDLLQ, seg_pos={0 0}, upload=57142]: Done uploading piece. err: ecclient: upload failed (node:1ikQ2b343QztY3vBFQ3cmEwEEFszpB2TPMtstufHdFfMEkJkBV, address:104.36.82.52:4115): protocol: write tcp 188.166.225.170:41764->104.36.82.52:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:10:26.155109427 (59h34m35.877672467s) [noise=true , address=104.36.82.53:4156 , piece_id=SBHXE7AABK2U2NTD4JPNTPRZ7ZRFU3M2PMUA2BAUKPV2DNVMIBCQ, seg_pos={0 0}, upload=57142]: Done uploading piece. err: ecclient: upload failed (node:12cFTycUosmf2ynRZmHYT1axN7VcS31wjqv74CxUpb5nThvmW5, address:104.36.82.53:4156): protocol: rpc: dial tcp 104.36.82.53:4156: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:10:26.155143852 (59h34m35.877706952s) [seg_pos={0 0}, upload=57142]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Jul 26 23:10:26.389788628 (59h34m36.112351668s) [upload=57142]: Done uploading segments... Jul 26 23:10:26.389829066 (59h34m36.112392106s) [upload=57142]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 1m0.23736283s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 2.074175118s FAIL (2025-07-26 23:10:17 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 2.074201811s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 9 out of 60 is failed (threshold=3) [....X..........X.X........X.X...........................XXXX] duration: 30.306239886s FAIL (2025-07-26 23:10:09 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.306259674s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [..X...X...............X..XX...X.X.X.X.......XX..XX.XX..XXX..] duration: 13.285103132s FAIL (2025-07-26 23:10:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 13.285130222s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.........................X.........X...........X.......XXXXX] duration: 30.240864204s FAIL (2025-07-26 23:10:07 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 9351347s) [upload=57146]: Done starting upload Jul 26 23:09:37.771163227 (59h33m47.493726267s) [upload=57146]: Uploading segments... Jul 26 23:09:37.771194487 (59h33m47.493757527s) [upload=57146]: Got next segment. Inline: false Jul 26 23:09:37.771216689 (59h33m47.493779719s) [upload=57146]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:09:37.771221318 (59h33m47.493784358s) [upload=57146]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:09:38.010069086 (59h33m47.732632126s) [seg_pos={0 0}, upload=57146]: Begin upload segment... Jul 26 23:09:38.010132468 (59h33m47.732695508s) [seg_pos={0 0}, upload=57146]: Done begin upload segment. Jul 26 23:09:38.010161213 (59h33m47.732724253s) [noise=true , address=104.36.82.2:4729 , piece_id=HNY3KDSUNKISBFL2D4HROSG2PGD4A6OVSERM34SR6ZKODRXFPOGQ, seg_pos={0 0}, upload=57146]: Uploading piece... Jul 26 23:09:38.010726724 (59h33m47.733289754s) [noise=true , address=45.91.116.12:4226 , piece_id=DTJBSQPBL6LC5YPU3FD2WNDU7ECZS5DNDSMOY472E2BFEWYBEZBQ, seg_pos={0 0}, upload=57146]: Uploading piece... Jul 26 23:09:38.010999817 (59h33m47.733562857s) [noise=true , address=45.91.116.4:4939 , piece_id=QJC5D3GOSKOFIN4TBBBEBRMXNAC5MSLC5B3FV3CPIHKOYEPS7B4A, seg_pos={0 0}, upload=57146]: Uploading piece... Jul 26 23:09:38.011272429 (59h33m47.733835459s) [noise=true , address=45.91.116.5:4299 , piece_id=DLQKPAMZCAI22MWLIYNL5YPKPXYW56HGB5RN5JFD7YT324PG3ESQ, seg_pos={0 0}, upload=57146]: Uploading piece... Jul 26 23:09:38.012813607 (59h33m47.735376647s) [noise=true , address=45.91.116.7:4009 , piece_id=UOJPEKV4ITWEESGPPUXAMBS3LL2CZKBAWJVK3DEQ2ZEZRB3GXQ3A, seg_pos={0 0}, upload=57146]: Uploading piece... Jul 26 23:09:38.013201690 (59h33m47.73576473s) [noise=true , address=77.90.31.17:4770 , piece_id=HOIOPLZEA76HJNSTHHSAGQ7C2Z7NPTCV345XBGMO4RAFCXCOKO6Q, seg_pos={0 0}, upload=57146]: Uploading piece... Jul 26 23:09:38.013432422 (59h33m47.735995452s) [noise=true , address=77.90.31.14:4661 , piece_id=K6P6F44MMGESBHC6L7QYP7NYAO3O2X756Q3225ZIGZKXOZFKJS2Q, seg_pos={0 0}, upload=57146]: Uploading piece... Jul 26 23:09:38.013711856 (59h33m47.736274896s) [noise=true , address=104.36.82.14:4222 , piece_id=52VIW7Y7CYMMFFTZ5TNEXFP65ANKOMBOEPGMENJLWJ2ZLWIKO6XA, seg_pos={0 0}, upload=57146]: Uploading piece... Jul 26 23:09:38.014004827 (59h33m47.736567867s) [noise=true , address=45.91.116.13:4154 , piece_id=I2YH3FWRZBJN6Y4CEJXWNEJ7QQL4JT7HRJT6BPFQGII3MNDIRXFA, seg_pos={0 0}, upload=57146]: Uploading piece... Jul 26 23:09:38.014259013 (59h33m47.736822053s) [noise=true , address=45.91.116.7:4010 , piece_id=AEQV4RIFYB2XTM3SWG4VNQHOJ6BXHYBF4REQKPUO5ES57ITICW3A, seg_pos={0 0}, upload=57146]: Uploading piece... Jul 26 23:09:38.014462382 (59h33m47.737025422s) [noise=true , address=104.36.82.4:4804 , piece_id=7FUM567RERS2MQVCTE7OF3YKNHKRALO2WGXOWKNXBGEAMLRDHSEQ, seg_pos={0 0}, upload=57146]: Uploading piece... Jul 26 23:09:38.014812993 (59h33m47.737376033s) [noise=true , address=104.36.82.3:4765 , piece_id=G3PGBEJPHJYPPYUX37EHG266LO64PJQ2ECEXBTKVABQXRYYZWWGQ, seg_pos={0 0}, upload=57146]: Uploading piece... Jul 26 23:09:38.015169064 (59h33m47.737732094s) [noise=true , address=104.36.82.6:4876 , piece_id=6Y5WOZ3LWWJFKUNNGSDGGGFZONLSFG2FHCYFBCE3NPNVRZ2FORVQ, seg_pos={0 0}, upload=57146]: Uploading piece... Jul 26 23:09:38.015449761 (59h33m47.738012801s) [noise=true , address=104.36.82.6:4875 , piece_id=BO4D3PYLGGAXGWS3R4OYRHVAL5LLMQCTWX45YDAHBCRDYXFKFA7A, seg_pos={0 0}, upload=57146]: Uploading piece... Jul 26 23:09:38.015718435 (59h33m47.738281475s) [noise=true , address=104.36.82.4:4802 , piece_id=NGF5M3T7AIW3J4Z57HWPBF2S7AQ2QB2CYKBTKRF2WNL7QSU36JSA, seg_pos={0 0}, upload=57146]: Uploading piece... Jul 26 23:09:38.016008330 (59h33m47.73857137s) [noise=true , address=77.90.31.10:4519 , piece_id=NFH4H5ESEJYZRS4UBGHEZ5BI4NKWHJFGKSO5NBZYC3AK7RPJBBLQ, seg_pos={0 0}, upload=57146]: Uploading piece... Jul 26 23:09:39.740835258 (59h33m49.463398288s) [upload=57146]: Next returned nil segment Jul 26 23:09:39.740850137 (59h33m49.463413177s) [upload=57146]: Waiting for error group managing segments... Jul 26 23:09:40.019803960 (59h33m49.742367s) [noise=true , address=45.91.116.12:4226 , piece_id=DTJBSQPBL6LC5YPU3FD2WNDU7ECZS5DNDSMOY472E2BFEWYBEZBQ, seg_pos={0 0}, upload=57146]: Done uploading piece. err: <nil> Jul 26 23:09:40.038647995 (59h33m49.761211035s) [noise=true , address=45.91.116.4:4939 , piece_id=QJC5D3GOSKOFIN4TBBBEBRMXNAC5MSLC5B3FV3CPIHKOYEPS7B4A, seg_pos={0 0}, upload=57146]: Done uploading piece. err: <nil> Jul 26 23:09:40.113136203 (59h33m49.835699244s) [noise=true , address=45.91.116.7:4009 , piece_id=UOJPEKV4ITWEESGPPUXAMBS3LL2CZKBAWJVK3DEQ2ZEZRB3GXQ3A, seg_pos={0 0}, upload=57146]: Done uploading piece. err: <nil> Jul 26 23:09:40.119887812 (59h33m49.842450852s) [noise=true , address=77.90.31.14:4661 , piece_id=K6P6F44MMGESBHC6L7QYP7NYAO3O2X756Q3225ZIGZKXOZFKJS2Q, seg_pos={0 0}, upload=57146]: Done uploading piece. err: <nil> Jul 26 23:09:40.125225995 (59h33m49.847789035s) [noise=true , address=45.91.116.7:4010 , piece_id=AEQV4RIFYB2XTM3SWG4VNQHOJ6BXHYBF4REQKPUO5ES57ITICW3A, seg_pos={0 0}, upload=57146]: Done uploading piece. err: <nil> Jul 26 23:09:40.137042714 (59h33m49.859605754s) [noise=true , address=45.91.116.13:4154 , piece_id=I2YH3FWRZBJN6Y4CEJXWNEJ7QQL4JT7HRJT6BPFQGII3MNDIRXFA, seg_pos={0 0}, upload=57146]: Done uploading piece. err: <nil> Jul 26 23:09:40.241954613 (59h33m49.964517653s) [noise=true , address=77.90.31.10:4519 , piece_id=NFH4H5ESEJYZRS4UBGHEZ5BI4NKWHJFGKSO5NBZYC3AK7RPJBBLQ, seg_pos={0 0}, upload=57146]: Done uploading piece. err: <nil> Jul 26 23:09:40.269870965 (59h33m49.992434005s) [noise=true , address=45.91.116.5:4299 , piece_id=DLQKPAMZCAI22MWLIYNL5YPKPXYW56HGB5RN5JFD7YT324PG3ESQ, seg_pos={0 0}, upload=57146]: Done uploading piece. err: <nil> Jul 26 23:10:07.131163678 (59h34m16.853726708s) [noise=true , address=104.36.82.6:4875 , piece_id=BO4D3PYLGGAXGWS3R4OYRHVAL5LLMQCTWX45YDAHBCRDYXFKFA7A, seg_pos={0 0}, upload=57146]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: rpc: dial tcp 104.36.82.6:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:10:07.131188696 (59h34m16.853751736s) [noise=true , address=104.36.82.4:4802 , piece_id=NGF5M3T7AIW3J4Z57HWPBF2S7AQ2QB2CYKBTKRF2WNL7QSU36JSA, seg_pos={0 0}, upload=57146]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 188.166.225.170:55528->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:10:07.131224074 (59h34m16.853787104s) [noise=true , address=104.36.82.14:4222 , piece_id=52VIW7Y7CYMMFFTZ5TNEXFP65ANKOMBOEPGMENJLWJ2ZLWIKO6XA, seg_pos={0 0}, upload=57146]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: rpc: dial tcp 104.36.82.14:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:10:07.131248490 (59h34m16.85381153s) [noise=true , address=77.90.31.17:4770 , piece_id=HOIOPLZEA76HJNSTHHSAGQ7C2Z7NPTCV345XBGMO4RAFCXCOKO6Q, seg_pos={0 0}, upload=57146]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: write tcp 188.166.225.170:35040->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:10:07.131275252 (59h34m16.853838292s) [noise=true , address=104.36.82.4:4804 , piece_id=7FUM567RERS2MQVCTE7OF3YKNHKRALO2WGXOWKNXBGEAMLRDHSEQ, seg_pos={0 0}, upload=57146]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: rpc: dial tcp 104.36.82.4:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:10:07.131283588 (59h34m16.853846628s) [noise=true , address=104.36.82.6:4876 , piece_id=6Y5WOZ3LWWJFKUNNGSDGGGFZONLSFG2FHCYFBCE3NPNVRZ2FORVQ, seg_pos={0 0}, upload=57146]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 188.166.225.170:56910->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:10:07.131288678 (59h34m16.853851718s) [noise=true , address=104.36.82.2:4729 , piece_id=HNY3KDSUNKISBFL2D4HROSG2PGD4A6OVSERM34SR6ZKODRXFPOGQ, seg_pos={0 0}, upload=57146]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:104.36.82.2:4729): protocol: write tcp 188.166.225.170:41712->104.36.82.2:4729: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:10:07.131313505 (59h34m16.853876556s) [noise=true , address=104.36.82.3:4765 , piece_id=G3PGBEJPHJYPPYUX37EHG266LO64PJQ2ECEXBTKVABQXRYYZWWGQ, seg_pos={0 0}, upload=57146]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: write tcp 188.166.225.170:44958->104.36.82.3:4765: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:10:07.131348001 (59h34m16.853911041s) [seg_pos={0 0}, upload=57146]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 26 23:10:07.370645873 (59h34m17.093208903s) [upload=57146]: Done uploading segments... Jul 26 23:10:07.370672163 (59h34m17.093235203s) [upload=57146]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.240887951s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 2.120944011s FAIL (2025-07-26 23:09:42 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 2.120973317s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 52 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXX.X.XX.XX.X.XXXXXXXXX...XXXXXXXXXXXXXXX] duration: 1m0.230791517s FAIL (2025-07-26 23:09:25 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 4) ======LOGS====== Q4EVS6CGJFYQBDFSFARNVMSYHAZRKBUGC7BKSN2RRXA, seg_pos={0 0}, upload=57125]: Uploading piece... Jul 26 23:08:26.859206362 (59h32m36.581769392s) [noise=true , address=104.36.82.51:4059 , piece_id=UHJEVJ53P6CZUMCFEPIBK4BA7VLVELQISFQPL3KC4TDY2LGY7YAQ, seg_pos={0 0}, upload=57125]: Uploading piece... Jul 26 23:08:26.859240166 (59h32m36.581803206s) [noise=true , address=104.36.82.53:4155 , piece_id=5BJUPF3M7SDPIVEK6AFCGACRRFSOBTTW4SAM4MFWB5URCMEXFIYA, seg_pos={0 0}, upload=57125]: Uploading piece... Jul 26 23:08:26.859453595 (59h32m36.582016635s) [noise=true , address=104.36.82.11:4138 , piece_id=N7EAQB5ZCHJNLMAXVS42ZBNRKO32RXSSCYUHBU555JMPUUBB4QPA, seg_pos={0 0}, upload=57125]: Uploading piece... Jul 26 23:08:26.859473022 (59h32m36.582036062s) [noise=true , address=104.36.82.40:4972 , piece_id=YYQJVPAEZNBK54OO65DNEA73BQMA6HTL5YXYHSQ6KOLKRZND3QBA, seg_pos={0 0}, upload=57125]: Uploading piece... Jul 26 23:08:26.859651784 (59h32m36.582214824s) [noise=true , address=79.127.205.246:4277 , piece_id=TFV6FDC55MAXH5CUMAABDOUDNDPY2K7X7WWBTJIXUHT6KJJPFYIQ, seg_pos={0 0}, upload=57125]: Uploading piece... Jul 26 23:08:26.859839724 (59h32m36.582402754s) [noise=true , address=79.127.205.245:4247 , piece_id=ED622SNOH4HPTPZLL5CJJDKAJQ2EI7DGNJFIQFZ2YJRTY7WAQZWA, seg_pos={0 0}, upload=57125]: Uploading piece... Jul 26 23:08:26.860029747 (59h32m36.582592787s) [noise=true , address=104.36.82.43:4848 , piece_id=3HDQWUHIY4SBFXXSDARAPVTAO7JGSSCJ3BMH6W77LE2WVSVWY64A, seg_pos={0 0}, upload=57125]: Uploading piece... Jul 26 23:08:26.860248145 (59h32m36.582811185s) [noise=true , address=79.127.205.235:4684 , piece_id=UEV7D5ABDW2QG2JB6TY3DU3QCFKNKQ6E75LOJK3HUZUZ7P6GQMIQ, seg_pos={0 0}, upload=57125]: Uploading piece... Jul 26 23:08:26.860447647 (59h32m36.583010687s) [noise=true , address=104.36.82.39:4754 , piece_id=N6EX6LRNMZTYKTXLK3FESEAYVJJDNZ34DK2DFOJGU26MLE7XPRLQ, seg_pos={0 0}, upload=57125]: Uploading piece... Jul 26 23:08:26.863250870 (59h32m36.5858139s) [noise=true , address=104.36.82.38:4703 , piece_id=APJXGEB24EMGZYAH74GYM55XIKOS3MCBP6OAJRU25J7O6YA5PIAA, seg_pos={0 0}, upload=57125]: Uploading piece... Jul 26 23:08:27.962761720 (59h32m37.68532476s) [upload=57125]: Next returned nil segment Jul 26 23:08:27.962785314 (59h32m37.685348354s) [upload=57125]: Waiting for error group managing segments... Jul 26 23:08:28.254929866 (59h32m37.977492906s) [noise=true , address=79.127.205.245:4247 , piece_id=ED622SNOH4HPTPZLL5CJJDKAJQ2EI7DGNJFIQFZ2YJRTY7WAQZWA, seg_pos={0 0}, upload=57125]: Done uploading piece. err: <nil> Jul 26 23:08:28.255546586 (59h32m37.978109616s) [noise=true , address=79.127.205.235:4684 , piece_id=UEV7D5ABDW2QG2JB6TY3DU3QCFKNKQ6E75LOJK3HUZUZ7P6GQMIQ, seg_pos={0 0}, upload=57125]: Done uploading piece. err: <nil> Jul 26 23:08:28.285894491 (59h32m38.008457531s) [noise=true , address=79.127.205.233:4642 , piece_id=FXGPULQLUPHK4OVBYCHDJXGZGQDRGNFCOQVOWV2CCGRZR6IAD3OQ, seg_pos={0 0}, upload=57125]: Done uploading piece. err: <nil> Jul 26 23:08:28.312935878 (59h32m38.035498918s) [noise=true , address=79.127.205.246:4277 , piece_id=TFV6FDC55MAXH5CUMAABDOUDNDPY2K7X7WWBTJIXUHT6KJJPFYIQ, seg_pos={0 0}, upload=57125]: Done uploading piece. err: <nil> Jul 26 23:09:25.695762883 (59h33m35.418325923s) [noise=true , address=104.36.82.47:4545 , piece_id=QDDNJ6FG7D2NFYZBGC37N3ECCXTEVCVU3NOJS26EA2FQEUC7JUHA, seg_pos={0 0}, upload=57125]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: write tcp 188.166.225.170:44716->104.36.82.47:4545: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:09:25.695819672 (59h33m35.418382712s) [noise=true , address=104.36.82.52:4095 , piece_id=XB7KJKNNTSBAIFUQ5WBWYQJEKDML5AL5JSBL53K4NDBSFBWDRR2Q, seg_pos={0 0}, upload=57125]: Done uploading piece. err: ecclient: upload failed (node:1idvNc8SGRP7tkAckr2TLv7LnvftiRUTGcwy5GCrH5rVyDEK5N, address:104.36.82.52:4095): protocol: write tcp 188.166.225.170:57306->104.36.82.52:4095: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:09:25.695868856 (59h33m35.418431896s) [noise=true , address=104.36.82.53:4155 , piece_id=5BJUPF3M7SDPIVEK6AFCGACRRFSOBTTW4SAM4MFWB5URCMEXFIYA, seg_pos={0 0}, upload=57125]: Done uploading piece. err: ecclient: upload failed (node:17wRBzmgMvpNyXy9SZEwEjwc4W5TfKzTj8vaa9c2HXnPNLMMYi, address:104.36.82.53:4155): protocol: rpc: dial tcp 104.36.82.53:4155: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:09:25.695980279 (59h33m35.418543319s) [noise=true , address=104.36.82.11:4138 , piece_id=N7EAQB5ZCHJNLMAXVS42ZBNRKO32RXSSCYUHBU555JMPUUBB4QPA, seg_pos={0 0}, upload=57125]: Done uploading piece. err: ecclient: upload failed (node:1H28XSTvFGa3XhavfhythHWs95cqjazu2RuqpsjNCmdSFvRLoh, address:104.36.82.11:4138): protocol: rpc: dial tcp 104.36.82.11:4138: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:09:25.696016718 (59h33m35.418579759s) [noise=true , address=104.36.82.40:4972 , piece_id=YYQJVPAEZNBK54OO65DNEA73BQMA6HTL5YXYHSQ6KOLKRZND3QBA, seg_pos={0 0}, upload=57125]: Done uploading piece. err: ecclient: upload failed (node:1XSuvCvq716GAa9WWmM3n2tqnfh8paaFUhjBzVzpiqwJtNifnr, address:104.36.82.40:4972): protocol: rpc: dial tcp 104.36.82.40:4972: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:09:25.696101621 (59h33m35.418664661s) [noise=true , address=104.36.82.19:4425 , piece_id=NWOHUCRYIFLCOLMZBKNBXSOG37PTMCQ5QW24XLRJVBO6WF56W6ZQ, seg_pos={0 0}, upload=57125]: Done uploading piece. err: ecclient: upload failed (node:13sAcGFjAXuesAkUn7NuLaaV8TAah3Mr1kqd33wwh9MTELqmpD, address:104.36.82.19:4425): protocol: rpc: dial tcp 104.36.82.19:4425: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:09:25.696161074 (59h33m35.418724114s) [noise=true , address=104.36.82.51:4059 , piece_id=UHJEVJ53P6CZUMCFEPIBK4BA7VLVELQISFQPL3KC4TDY2LGY7YAQ, seg_pos={0 0}, upload=57125]: Done uploading piece. err: ecclient: upload failed (node:12fQsGGr5K6TVMU7XC2v1mQBFwHznj5ugYvYLmmAUnP81AyHAjY, address:104.36.82.51:4059): protocol: rpc: dial tcp 104.36.82.51:4059: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:09:25.696241298 (59h33m35.418804338s) [noise=true , address=104.36.82.41:4775 , piece_id=KILTSLANAM3O5GVULL3SBNIO53NAEX4RVRRNDBWGIEE36LJU676Q, seg_pos={0 0}, upload=57125]: Done uploading piece. err: ecclient: upload failed (node:166yRbNnnDeZRifM5q7vUVLx7iV5cZ9GqQnmvD9AbUUJsaCjmj, address:104.36.82.41:4775): protocol: rpc: dial tcp 104.36.82.41:4775: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 26 23:09:25.696289290 (59h33m35.41885233s) [noise=true , address=104.36.82.46:4502 , piece_id=4727FQI67Q4EVS6CGJFYQBDFSFARNVMSYHAZRKBUGC7BKSN2RRXA, seg_pos={0 0}, upload=57125]: Done uploading piece. err: ecclient: upload failed (node:12HP5xuRAGgTsL11UQh7VRgfuFzwNdEMXgc8zPVv4QknStyv6Ta, address:104.36.82.46:4502): protocol: write tcp 188.166.225.170:57910->104.36.82.46:4502: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:09:25.696357240 (59h33m35.41892028s) [noise=true , address=104.36.82.39:4754 , piece_id=N6EX6LRNMZTYKTXLK3FESEAYVJJDNZ34DK2DFOJGU26MLE7XPRLQ, seg_pos={0 0}, upload=57125]: Done uploading piece. err: ecclient: upload failed (node:1HB1v3ZURPMSYsbHBjJs1neE7HQtLS1KmeLwbKK2sTKAZcXPhi, address:104.36.82.39:4754): protocol: write tcp 188.166.225.170:33796->104.36.82.39:4754: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:09:25.696422786 (59h33m35.418985826s) [noise=true , address=104.36.82.43:4848 , piece_id=3HDQWUHIY4SBFXXSDARAPVTAO7JGSSCJ3BMH6W77LE2WVSVWY64A, seg_pos={0 0}, upload=57125]: Done uploading piece. err: ecclient: upload failed (node:1DPLv5Kfsx9RLVWeRrxbdkaLGJc4dfrzGnPLwc8us9hT5uiPbj, address:104.36.82.43:4848): protocol: write tcp 188.166.225.170:48724->104.36.82.43:4848: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:09:25.696563906 (59h33m35.419126936s) [noise=true , address=104.36.82.55:4228 , piece_id=NJ5AJEFHLFX2YB7TRZXJTGQEG4DIV4WH6UPU7JRZJIOPYRDLIYAQ, seg_pos={0 0}, upload=57125]: Done uploading piece. err: ecclient: upload failed (node:12NBsd33JtAwPdcbD8vBzZUYpjiqZpRzs8y2QrYJFE245pd8T5X, address:104.36.82.55:4228): protocol: rpc: dial tcp 104.36.82.55:4228: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:09:25.696601207 (59h33m35.419164247s) [noise=true , address=104.36.82.38:4703 , piece_id=APJXGEB24EMGZYAH74GYM55XIKOS3MCBP6OAJRU25J7O6YA5PIAA, seg_pos={0 0}, upload=57125]: Done uploading piece. err: ecclient: upload failed (node:12nHGPE4LDH4wwTXZXA8vHqn4Tm735RvfC7mKx1jEMBYNbA1jNd, address:104.36.82.38:4703): protocol: rpc: dial tcp 104.36.82.38:4703: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:09:25.696648357 (59h33m35.419211397s) [noise=true , address=104.36.82.18:4394 , piece_id=JVJVVN7PP5HTIPCGQXL264RP6VPPEPPXJYLF4WAJQAFTDELFSD6A, seg_pos={0 0}, upload=57125]: Done uploading piece. err: ecclient: upload failed (node:12Q6GgLmLeGugk1rQu21KNV2pEjXTmgAfiYcV45ibvDixBW41K2, address:104.36.82.18:4394): protocol: rpc: dial tcp 104.36.82.18:4394: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:09:25.696663276 (59h33m35.419226316s) [seg_pos={0 0}, upload=57125]: Done waiting for segment. successful: 4 optimal: 14 errs: <nil> Jul 26 23:09:25.925048987 (59h33m35.647612027s) [upload=57125]: Done uploading segments... Jul 26 23:09:25.925069947 (59h33m35.647632987s) [upload=57125]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 4) Duration: 1m0.230812999s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [...X...X...............X..XX...X.X.X.X.......XX..XX.XX..XXX.] duration: 31.312542888s FAIL (2025-07-26 23:09:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 31.312565671s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [..........................X.........X...........X.......XXXX] duration: 30.232181687s FAIL (2025-07-26 23:09:14 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== iecestore: piecestore close: conn closed: op Jul 26 23:08:38.968744705 (59h32m48.691307775s) [seg_pos={0 0}, upload=57121]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jul 26 23:08:39.198190344 (59h32m48.920753384s) [upload=57121]: Done uploading segments... Jul 26 23:08:39.198217035 (59h32m48.920780065s) [upload=57121]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Jul 26 23:08:45.416861909 (59h32m55.139424949s) [upload=57132]: Starting upload Jul 26 23:08:45.416927263 (59h32m55.139490303s) [upload=57132]: Done starting upload Jul 26 23:08:45.417027075 (59h32m55.139590115s) [upload=57132]: Uploading segments... Jul 26 23:08:45.417141343 (59h32m55.139704383s) [upload=57132]: Got next segment. Inline: false Jul 26 23:08:45.417177221 (59h32m55.139740262s) [upload=57132]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:08:45.417181950 (59h32m55.13974499s) [upload=57132]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:08:45.652969433 (59h32m55.375532523s) [seg_pos={0 0}, upload=57132]: Begin upload segment... Jul 26 23:08:45.653033106 (59h32m55.375596176s) [seg_pos={0 0}, upload=57132]: Done begin upload segment. Jul 26 23:08:45.653059176 (59h32m55.375622216s) [noise=true , address=104.36.82.11:4114 , piece_id=EUFZGF3QPKFVEPEPTLKMBT7EJHGLH4BEYC2ILII4T4BLGSNZQOVA, seg_pos={0 0}, upload=57132]: Uploading piece... Jul 26 23:08:45.653066690 (59h32m55.37562976s) [noise=true , address=45.91.116.8:4045 , piece_id=MTXHE4WEASM5XQ2SDXN7SCHRPMJEVSKMSM2VJU5A2UUDO3C6GDJA, seg_pos={0 0}, upload=57132]: Uploading piece... Jul 26 23:08:45.653449482 (59h32m55.376012522s) [noise=true , address=104.36.82.15:4258 , piece_id=RURAMKZQBLP35SUQG37ZB2K4LTAQHTJC27TD7UC2APZFRNVG24FQ, seg_pos={0 0}, upload=57132]: Uploading piece... Jul 26 23:08:45.653664604 (59h32m55.376227644s) [noise=true , address=45.91.116.13:4154 , piece_id=KY7TFYFBCM2QA6CBS4PEQGMCYWIMGUAWP66NO6P5KQGUXP7225ZQ, seg_pos={0 0}, upload=57132]: Uploading piece... Jul 26 23:08:45.653723156 (59h32m55.376286196s) [noise=true , address=104.36.82.4:4801 , piece_id=GWD3T4KTBXY76VT74A2HYZHRGNGLIJJXOT2LV62B2B4IWOP5OY3A, seg_pos={0 0}, upload=57132]: Uploading piece... Jul 26 23:08:45.654025705 (59h32m55.376588745s) [noise=true , address=45.91.116.10:4082 , piece_id=2KBPGXCUZINGSFY2WKPCGVH5C7DNKKELUVGRV3TMC2HTTIFRX63Q, seg_pos={0 0}, upload=57132]: Uploading piece... Jul 26 23:08:45.654153629 (59h32m55.376716669s) [noise=true , address=45.91.116.12:4225 , piece_id=3RUBE2M53QMIWL2ACBXFLS23QUP7YBNBHHGO5KLRZV6CHKHGQYXA, seg_pos={0 0}, upload=57132]: Uploading piece... Jul 26 23:08:45.654306592 (59h32m55.376869632s) [noise=true , address=45.91.116.4:4939 , piece_id=RWRHBEHTKBHE3RSJR22NU5D6PTC52VRXRQLXZPPP7EGMCKOBVPWA, seg_pos={0 0}, upload=57132]: Uploading piece... Jul 26 23:08:45.654430949 (59h32m55.376993989s) [noise=true , address=45.91.116.7:4011 , piece_id=4CKVEH6TLSVDINSBKMKVHTL5WU4UCNQEFTT4JC4A4N6OGT4FAGWQ, seg_pos={0 0}, upload=57132]: Uploading piece... Jul 26 23:08:45.654571238 (59h32m55.377134278s) [noise=true , address=104.36.82.4:4804 , piece_id=OTHXF5A65FAYOC3EEAELLKQQLFDBCPKNOW3JBJRHKHXNEVKUMOXA, seg_pos={0 0}, upload=57132]: Uploading piece... Jul 26 23:08:45.654639268 (59h32m55.377202308s) [noise=true , address=77.90.31.10:4520 , piece_id=MW46VPXIITXMJQOHWND5DMAC22HLT4BCBGDI3KOLBX6JNL2EWD6Q, seg_pos={0 0}, upload=57132]: Uploading piece... Jul 26 23:08:45.654793414 (59h32m55.377356454s) [noise=true , address=104.36.82.6:4875 , piece_id=MX55JOGV4AMFRN7JCAJLPXNDGC2RSSRWGJFQWQP4R5IDWEIHAB6A, seg_pos={0 0}, upload=57132]: Uploading piece... Jul 26 23:08:45.654855773 (59h32m55.377418803s) [noise=true , address=45.91.116.7:4010 , piece_id=KCJK355UTKJLZ57HLVSJC6OUYP5TZZLARFVLE4WOW4WZQYPJ5BOQ, seg_pos={0 0}, upload=57132]: Uploading piece... Jul 26 23:08:45.655051667 (59h32m55.377614708s) [noise=true , address=104.36.82.9:4007 , piece_id=LMYTZBCIUE6A5CQD3I6DE4ASMKYPVJPJTOCLWTZJRUZSR7MHD7ZA, seg_pos={0 0}, upload=57132]: Uploading piece... Jul 26 23:08:45.655057939 (59h32m55.377620979s) [noise=true , address=45.91.116.5:4299 , piece_id=TFRWMFFIURW6ATEUHKY44MO6K25HPT2IWL2DKF7BV4WILDCQNFMA, seg_pos={0 0}, upload=57132]: Uploading piece... Jul 26 23:08:45.655253453 (59h32m55.377816493s) [noise=true , address=77.90.31.14:4662 , piece_id=EU4UOXWS67OEABJLU2ZJEJZTHLOZKWSJNLYHO52ZV27ZU5HKKKAA, seg_pos={0 0}, upload=57132]: Uploading piece... Jul 26 23:08:47.374699869 (59h32m57.097262899s) [upload=57132]: Next returned nil segment Jul 26 23:08:47.374716200 (59h32m57.09727924s) [upload=57132]: Waiting for error group managing segments... Jul 26 23:08:47.658746035 (59h32m57.381309065s) [noise=true , address=45.91.116.7:4010 , piece_id=KCJK355UTKJLZ57HLVSJC6OUYP5TZZLARFVLE4WOW4WZQYPJ5BOQ, seg_pos={0 0}, upload=57132]: Done uploading piece. err: <nil> Jul 26 23:08:47.660286352 (59h32m57.382849382s) [noise=true , address=77.90.31.14:4662 , piece_id=EU4UOXWS67OEABJLU2ZJEJZTHLOZKWSJNLYHO52ZV27ZU5HKKKAA, seg_pos={0 0}, upload=57132]: Done uploading piece. err: <nil> Jul 26 23:08:47.669315578 (59h32m57.391878619s) [noise=true , address=45.91.116.5:4299 , piece_id=TFRWMFFIURW6ATEUHKY44MO6K25HPT2IWL2DKF7BV4WILDCQNFMA, seg_pos={0 0}, upload=57132]: Done uploading piece. err: <nil> Jul 26 23:08:47.748186562 (59h32m57.470749602s) [noise=true , address=45.91.116.10:4082 , piece_id=2KBPGXCUZINGSFY2WKPCGVH5C7DNKKELUVGRV3TMC2HTTIFRX63Q, seg_pos={0 0}, upload=57132]: Done uploading piece. err: <nil> Jul 26 23:08:47.754857135 (59h32m57.477420165s) [noise=true , address=45.91.116.4:4939 , piece_id=RWRHBEHTKBHE3RSJR22NU5D6PTC52VRXRQLXZPPP7EGMCKOBVPWA, seg_pos={0 0}, upload=57132]: Done uploading piece. err: <nil> Jul 26 23:08:47.759651559 (59h32m57.482214589s) [noise=true , address=45.91.116.13:4154 , piece_id=KY7TFYFBCM2QA6CBS4PEQGMCYWIMGUAWP66NO6P5KQGUXP7225ZQ, seg_pos={0 0}, upload=57132]: Done uploading piece. err: <nil> Jul 26 23:08:47.761409322 (59h32m57.483972362s) [noise=true , address=45.91.116.12:4225 , piece_id=3RUBE2M53QMIWL2ACBXFLS23QUP7YBNBHHGO5KLRZV6CHKHGQYXA, seg_pos={0 0}, upload=57132]: Done uploading piece. err: <nil> Jul 26 23:08:47.770619013 (59h32m57.493182053s) [noise=true , address=77.90.31.10:4520 , piece_id=MW46VPXIITXMJQOHWND5DMAC22HLT4BCBGDI3KOLBX6JNL2EWD6Q, seg_pos={0 0}, upload=57132]: Done uploading piece. err: <nil> Jul 26 23:08:47.772062114 (59h32m57.494625154s) [noise=true , address=45.91.116.7:4011 , piece_id=4CKVEH6TLSVDINSBKMKVHTL5WU4UCNQEFTT4JC4A4N6OGT4FAGWQ, seg_pos={0 0}, upload=57132]: Done uploading piece. err: <nil> Jul 26 23:08:47.830112743 (59h32m57.552675773s) [noise=true , address=45.91.116.8:4045 , piece_id=MTXHE4WEASM5XQ2SDXN7SCHRPMJEVSKMSM2VJU5A2UUDO3C6GDJA, seg_pos={0 0}, upload=57132]: Done uploading piece. err: <nil> Jul 26 23:09:14.767106933 (59h33m24.489669973s) [noise=true , address=104.36.82.15:4258 , piece_id=RURAMKZQBLP35SUQG37ZB2K4LTAQHTJC27TD7UC2APZFRNVG24FQ, seg_pos={0 0}, upload=57132]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: rpc: dial tcp 104.36.82.15:4258: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:09:14.767156327 (59h33m24.489719357s) [noise=true , address=104.36.82.4:4801 , piece_id=GWD3T4KTBXY76VT74A2HYZHRGNGLIJJXOT2LV62B2B4IWOP5OY3A, seg_pos={0 0}, upload=57132]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 188.166.225.170:35126->104.36.82.4:4801: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:09:14.767211543 (59h33m24.489774573s) [noise=true , address=104.36.82.4:4804 , piece_id=OTHXF5A65FAYOC3EEAELLKQQLFDBCPKNOW3JBJRHKHXNEVKUMOXA, seg_pos={0 0}, upload=57132]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: rpc: dial tcp 104.36.82.4:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:09:14.767297948 (59h33m24.489860988s) [noise=true , address=104.36.82.11:4114 , piece_id=EUFZGF3QPKFVEPEPTLKMBT7EJHGLH4BEYC2ILII4T4BLGSNZQOVA, seg_pos={0 0}, upload=57132]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:09:14.767360639 (59h33m24.489923679s) [noise=true , address=104.36.82.6:4875 , piece_id=MX55JOGV4AMFRN7JCAJLPXNDGC2RSSRWGJFQWQP4R5IDWEIHAB6A, seg_pos={0 0}, upload=57132]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: rpc: dial tcp 104.36.82.6:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:09:14.767453185 (59h33m24.490016215s) [noise=true , address=104.36.82.9:4007 , piece_id=LMYTZBCIUE6A5CQD3I6DE4ASMKYPVJPJTOCLWTZJRUZSR7MHD7ZA, seg_pos={0 0}, upload=57132]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 188.166.225.170:43866->104.36.82.9:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 26 23:09:14.767481620 (59h33m24.49004466s) [seg_pos={0 0}, upload=57132]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 26 23:09:14.998326145 (59h33m24.720889185s) [upload=57132]: Done uploading segments... Jul 26 23:09:14.998352706 (59h33m24.720915746s) [upload=57132]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.232201155s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........................................................XXX.] duration: 5.873204647s FAIL (2025-07-26 23:09:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.873221359s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........................................................XXX] duration: 30.00087478s FAIL (2025-07-26 23:09:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000891642s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [.....X..........X.X........X.X...........................XXX] duration: 30.250596161s FAIL (2025-07-26 23:09:01 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 6) Duration: 30.25061689s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [....X...X...............X..XX...X.X.X.X.......XX..XX.XX..XXX] duration: 1m0.00065835s FAIL (2025-07-26 23:08:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000677977s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...........................X.........X...........X.......XXX] duration: 30.231556285s FAIL (2025-07-26 23:08:39 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== Item_SegmentBegin Jul 26 23:08:09.841941379 (59h32m19.564504409s) [seg_pos={0 0}, upload=57121]: Begin upload segment... Jul 26 23:08:09.841981756 (59h32m19.564544796s) [seg_pos={0 0}, upload=57121]: Done begin upload segment. Jul 26 23:08:09.842006183 (59h32m19.564569223s) [noise=true , address=77.90.31.16:4735 , piece_id=SVFUQGMXTOLLVKLFS7HKHV6QV2BBAYZ5XHE2467Z3X7F2S23KPOQ, seg_pos={0 0}, upload=57121]: Uploading piece... Jul 26 23:08:09.842039937 (59h32m19.564602967s) [noise=true , address=104.36.82.4:4801 , piece_id=SBWD4FBMZCX5QIF6NKGYOEUY7YVZQEC74REI6IMYZIMEX4IUGSMQ, seg_pos={0 0}, upload=57121]: Uploading piece... Jul 26 23:08:09.842359098 (59h32m19.564922138s) [noise=true , address=104.36.82.15:4257 , piece_id=CEPG3S4VCGM4WSUZBEFUGCM6MK6I3ZV4DLTXAEWFY5YVIDII6MUA, seg_pos={0 0}, upload=57121]: Uploading piece... Jul 26 23:08:09.842619817 (59h32m19.565182857s) [noise=true , address=104.36.82.6:4876 , piece_id=WTINWNBROVXVVAT5C42MQ5TAIUVSFNVT3DAYS6XIWDDLT5PNOJPA, seg_pos={0 0}, upload=57121]: Uploading piece... Jul 26 23:08:09.842689571 (59h32m19.565252601s) [noise=true , address=77.90.31.12:4590 , piece_id=3FQZXR2WMUFYRTXA2TK2PAT7S7GUW5OLYQU4Z6CQFAACGJWBSUTQ, seg_pos={0 0}, upload=57121]: Uploading piece... Jul 26 23:08:09.842921965 (59h32m19.565485005s) [noise=true , address=45.91.116.7:4009 , piece_id=H3Y2BUDRPE5SCJRBKYWKSGK4NGN42ANVZW6SVIVR6AQ7GGGVQ4ZQ, seg_pos={0 0}, upload=57121]: Uploading piece... Jul 26 23:08:09.842970608 (59h32m19.565533648s) [noise=true , address=45.91.116.5:4298 , piece_id=6VTK4L3MUUD524HAEJ4EV3T6YRM7DI2OT7UKYUC4OMDZ4N7IOWBQ, seg_pos={0 0}, upload=57121]: Uploading piece... Jul 26 23:08:09.843216529 (59h32m19.565779569s) [noise=true , address=104.36.82.2:4729 , piece_id=SR6OV5YTPTJZDZ5DU7LPRD56I37WSQHAJRJ4TJ2S4DGIIBDYBIYA, seg_pos={0 0}, upload=57121]: Uploading piece... Jul 26 23:08:09.843413896 (59h32m19.565976936s) [noise=true , address=45.91.116.10:4082 , piece_id=WK7TT3TGMRON3ELPQEY7VPQMG5KD3PCGPAPQK23YPY2VP3H7HMWQ, seg_pos={0 0}, upload=57121]: Uploading piece... Jul 26 23:08:09.843496985 (59h32m19.566060025s) [noise=true , address=45.91.116.6:4975 , piece_id=TDJ3ED44SRP6IUL4YGDRQZHJEGPGFGZKLRBTIOJO27SNOKOP6CPA, seg_pos={0 0}, upload=57121]: Uploading piece... Jul 26 23:08:09.844100570 (59h32m19.5666636s) [noise=true , address=104.36.82.16:4295 , piece_id=ZUEFZPKRETZLRCLKLRWCN7LRAMSM3YFXABUMMCQ5HKKK66KM5DAQ, seg_pos={0 0}, upload=57121]: Uploading piece... Jul 26 23:08:09.844192557 (59h32m19.566755597s) [noise=true , address=104.36.82.9:4005 , piece_id=RID6366CMFOZRL76FQT6MY3J37HMXGI66PKYKTUVXDT5U37AW5UA, seg_pos={0 0}, upload=57121]: Uploading piece... Jul 26 23:08:09.844549368 (59h32m19.567112408s) [noise=true , address=45.91.116.13:4155 , piece_id=WBRY5TII7CIJAPXKMLRZGAAS6PD2MXZ67SLCTTXBMILTARPIEKHA, seg_pos={0 0}, upload=57121]: Uploading piece... Jul 26 23:08:09.845062470 (59h32m19.567625521s) [noise=true , address=104.36.82.6:4873 , piece_id=333ONMIQQFRZJ547CDXWV6COHJWLUYEVRWHCE7AH5TJQC7H6E6YQ, seg_pos={0 0}, upload=57121]: Uploading piece... Jul 26 23:08:09.846274870 (59h32m19.56883793s) [noise=true , address=45.91.116.8:4047 , piece_id=S4JB6GF2IGEDWQL7EIMY343EJBCDLBIP2YD3RNBTDFMUG623O2FQ, seg_pos={0 0}, upload=57121]: Uploading piece... Jul 26 23:08:09.847118815 (59h32m19.569681855s) [noise=true , address=45.91.116.12:4227 , piece_id=IGDBX2ELO323N62CRZ6BEPIO6V6Q5PXXF54WDZ5RI7XTUV53KJ7Q, seg_pos={0 0}, upload=57121]: Uploading piece... Jul 26 23:08:11.586226045 (59h32m21.308789075s) [upload=57121]: Next returned nil segment Jul 26 23:08:11.586241305 (59h32m21.308804335s) [upload=57121]: Waiting for error group managing segments... Jul 26 23:08:11.860931901 (59h32m21.583494941s) [noise=true , address=45.91.116.13:4155 , piece_id=WBRY5TII7CIJAPXKMLRZGAAS6PD2MXZ67SLCTTXBMILTARPIEKHA, seg_pos={0 0}, upload=57121]: Done uploading piece. err: <nil> Jul 26 23:08:11.866390746 (59h32m21.588953816s) [noise=true , address=45.91.116.12:4227 , piece_id=IGDBX2ELO323N62CRZ6BEPIO6V6Q5PXXF54WDZ5RI7XTUV53KJ7Q, seg_pos={0 0}, upload=57121]: Done uploading piece. err: <nil> Jul 26 23:08:11.931503787 (59h32m21.654066827s) [noise=true , address=45.91.116.5:4298 , piece_id=6VTK4L3MUUD524HAEJ4EV3T6YRM7DI2OT7UKYUC4OMDZ4N7IOWBQ, seg_pos={0 0}, upload=57121]: Done uploading piece. err: <nil> Jul 26 23:08:11.941528477 (59h32m21.664091517s) [noise=true , address=45.91.116.7:4009 , piece_id=H3Y2BUDRPE5SCJRBKYWKSGK4NGN42ANVZW6SVIVR6AQ7GGGVQ4ZQ, seg_pos={0 0}, upload=57121]: Done uploading piece. err: <nil> Jul 26 23:08:11.944608660 (59h32m21.667171701s) [noise=true , address=45.91.116.10:4082 , piece_id=WK7TT3TGMRON3ELPQEY7VPQMG5KD3PCGPAPQK23YPY2VP3H7HMWQ, seg_pos={0 0}, upload=57121]: Done uploading piece. err: <nil> Jul 26 23:08:11.945284984 (59h32m21.667848024s) [noise=true , address=45.91.116.6:4975 , piece_id=TDJ3ED44SRP6IUL4YGDRQZHJEGPGFGZKLRBTIOJO27SNOKOP6CPA, seg_pos={0 0}, upload=57121]: Done uploading piece. err: <nil> Jul 26 23:08:12.001014883 (59h32m21.723577923s) [noise=true , address=45.91.116.8:4047 , piece_id=S4JB6GF2IGEDWQL7EIMY343EJBCDLBIP2YD3RNBTDFMUG623O2FQ, seg_pos={0 0}, upload=57121]: Done uploading piece. err: <nil> Jul 26 23:08:38.968073952 (59h32m48.690637002s) [noise=true , address=104.36.82.4:4801 , piece_id=SBWD4FBMZCX5QIF6NKGYOEUY7YVZQEC74REI6IMYZIMEX4IUGSMQ, seg_pos={0 0}, upload=57121]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 188.166.225.170:45118->104.36.82.4:4801: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:08:38.968166609 (59h32m48.690729659s) [noise=true , address=104.36.82.6:4876 , piece_id=WTINWNBROVXVVAT5C42MQ5TAIUVSFNVT3DAYS6XIWDDLT5PNOJPA, seg_pos={0 0}, upload=57121]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 188.166.225.170:44012->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:08:38.968275297 (59h32m48.690838337s) [noise=true , address=104.36.82.9:4005 , piece_id=RID6366CMFOZRL76FQT6MY3J37HMXGI66PKYKTUVXDT5U37AW5UA, seg_pos={0 0}, upload=57121]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: write tcp 188.166.225.170:44546->104.36.82.9:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:08:38.968364998 (59h32m48.690928078s) [noise=true , address=104.36.82.2:4729 , piece_id=SR6OV5YTPTJZDZ5DU7LPRD56I37WSQHAJRJ4TJ2S4DGIIBDYBIYA, seg_pos={0 0}, upload=57121]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:104.36.82.2:4729): protocol: write tcp 188.166.225.170:35232->104.36.82.2:4729: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:08:38.968471271 (59h32m48.691034311s) [noise=true , address=77.90.31.16:4735 , piece_id=SVFUQGMXTOLLVKLFS7HKHV6QV2BBAYZ5XHE2467Z3X7F2S23KPOQ, seg_pos={0 0}, upload=57121]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:51162->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:51162->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51162->77.90.31.16:4735: use of closed network connection Jul 26 23:08:38.968565262 (59h32m48.691128342s) [noise=true , address=104.36.82.15:4257 , piece_id=CEPG3S4VCGM4WSUZBEFUGCM6MK6I3ZV4DLTXAEWFY5YVIDII6MUA, seg_pos={0 0}, upload=57121]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: rpc: dial tcp 104.36.82.15:4257: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:08:38.968654262 (59h32m48.691217302s) [noise=true , address=77.90.31.12:4590 , piece_id=3FQZXR2WMUFYRTXA2TK2PAT7S7GUW5OLYQU4Z6CQFAACGJWBSUTQ, seg_pos={0 0}, upload=57121]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:08:38.968668800 (59h32m48.69123184s) [noise=true , address=104.36.82.6:4873 , piece_id=333ONMIQQFRZJ547CDXWV6COHJWLUYEVRWHCE7AH5TJQC7H6E6YQ, seg_pos={0 0}, upload=57121]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:104.36.82.6:4873): protocol: write tcp 188.166.225.170:53664->104.36.82.6:4873: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:08:38.968716150 (59h32m48.69127919s) [noise=true , address=104.36.82.16:4295 , piece_id=ZUEFZPKRETZLRCLKLRWCN7LRAMSM3YFXABUMMCQ5HKKK66KM5DAQ, seg_pos={0 0}, upload=57121]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 188.166.225.170:49172->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:08:38.968744705 (59h32m48.691307775s) [seg_pos={0 0}, upload=57121]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jul 26 23:08:39.198190344 (59h32m48.920753384s) [upload=57121]: Done uploading segments... Jul 26 23:08:39.198217035 (59h32m48.920780065s) [upload=57121]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 30.231569862s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 1.923549084s FAIL (2025-07-26 23:08:28 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.923577477s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 52 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXX.X.XX.XX.X.XXXXXXXXX...XXXXXXXXXXXXXX] duration: 1m0.233588805s FAIL (2025-07-26 23:08:13 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== Uploading piece... Jul 26 23:07:14.866152584 (59h31m24.588715624s) [noise=true , address=104.36.82.21:4495 , piece_id=LK3TA2RQWAY5DAQHLX4QOAK2EBZOD4FZ6FJMURYJ6YLTD2WFZ4GQ, seg_pos={0 0}, upload=57107]: Uploading piece... Jul 26 23:07:14.866976010 (59h31m24.58953905s) [noise=true , address=104.36.82.42:4824 , piece_id=UMQLXDHZFLZZKIX4LCY6KARUSNL2WDZIB7STHSIJTIV3S4DS5AOQ, seg_pos={0 0}, upload=57107]: Uploading piece... Jul 26 23:07:14.867592649 (59h31m24.590155689s) [noise=true , address=104.36.82.44:4887 , piece_id=TIEB4HHQIBTUMDD3LLTZ42RUMWGPATKVVJJZVHP2UODQBPFREUPQ, seg_pos={0 0}, upload=57107]: Uploading piece... Jul 26 23:07:14.868026640 (59h31m24.59058969s) [noise=true , address=104.36.82.7:4923 , piece_id=NDN3RDPB7U6FUAU3BJK4EKGTZQ5JGMPDBYQ26KNPBPE4IMFOML6A, seg_pos={0 0}, upload=57107]: Uploading piece... Jul 26 23:07:14.868495346 (59h31m24.591058386s) [noise=true , address=104.36.82.12:4175 , piece_id=DTKKBL6MNTT3JBCADEEIG6RB7ZM6STSHQDJFUJH7XEYHLRZSAB7A, seg_pos={0 0}, upload=57107]: Uploading piece... Jul 26 23:07:14.868977890 (59h31m24.59154093s) [noise=true , address=104.36.82.53:4135 , piece_id=YSYQ7FXVZTOWVL3OE237WGDDJWJKXXKKXUIBQCMZJWUV4STU5GJA, seg_pos={0 0}, upload=57107]: Uploading piece... Jul 26 23:07:14.869574451 (59h31m24.592137491s) [noise=true , address=156.146.43.225:4449 , piece_id=VIJ2LPKMS75UJWXG77LBXLGBDW2PIJYKGI2PW5Y54RRINUN7EZAQ, seg_pos={0 0}, upload=57107]: Uploading piece... Jul 26 23:07:14.869912448 (59h31m24.592475488s) [noise=true , address=104.36.82.12:4164 , piece_id=A47HMZEE7H5TRROTIESOODM4BFYHV2PPLN7O2UB2PRE6W4ZRGKTQ, seg_pos={0 0}, upload=57107]: Uploading piece... Jul 26 23:07:14.870131026 (59h31m24.592694056s) [noise=true , address=79.127.219.38:4006 , piece_id=AYS4STACOC5LZMDBKYW6Q5MGBQCE3ZSTV54O6BXYWVBR6MOX2IXQ, seg_pos={0 0}, upload=57107]: Uploading piece... Jul 26 23:07:14.870344724 (59h31m24.592907764s) [noise=true , address=104.36.82.46:4507 , piece_id=576PYZ2DUVRNOYWALSHDA3IY72O4RKSKDYSWNMMI6HUUCXKFGZHQ, seg_pos={0 0}, upload=57107]: Uploading piece... Jul 26 23:07:14.870566739 (59h31m24.593129789s) [noise=true , address=104.36.82.50:4020 , piece_id=5TCI6DPNEZ3FGU75JFDYVOUIXI36QHPZSDI4FLYALO2CKUJQGN3A, seg_pos={0 0}, upload=57107]: Uploading piece... Jul 26 23:07:14.870877695 (59h31m24.593440735s) [noise=true , address=104.36.82.4:4805 , piece_id=MV7EHWT7O2ZX2WW2RY2Z4PBD22PUT6BGRVLXZWGQUEMZRSZN723Q, seg_pos={0 0}, upload=57107]: Uploading piece... Jul 26 23:07:14.871161918 (59h31m24.593724958s) [noise=true , address=104.36.82.54:4188 , piece_id=S3ULLIMTUOSQF4FWWNQ6NU7DNPWDUDH75BDHK7RL74NPCPHUX33A, seg_pos={0 0}, upload=57107]: Uploading piece... Jul 26 23:07:16.395362087 (59h31m26.117925127s) [upload=57107]: Next returned nil segment Jul 26 23:07:16.395381353 (59h31m26.117944393s) [upload=57107]: Waiting for error group managing segments... Jul 26 23:07:16.798283890 (59h31m26.52084697s) [noise=true , address=156.146.43.225:4449 , piece_id=VIJ2LPKMS75UJWXG77LBXLGBDW2PIJYKGI2PW5Y54RRINUN7EZAQ, seg_pos={0 0}, upload=57107]: Done uploading piece. err: <nil> Jul 26 23:07:16.824348710 (59h31m26.54691177s) [noise=true , address=79.127.219.38:4006 , piece_id=AYS4STACOC5LZMDBKYW6Q5MGBQCE3ZSTV54O6BXYWVBR6MOX2IXQ, seg_pos={0 0}, upload=57107]: Done uploading piece. err: <nil> Jul 26 23:07:16.910206261 (59h31m26.632769301s) [noise=true , address=104.36.82.55:4193 , piece_id=HJ2HQS7REFB6KX2PPIRSPED6X3NIERE35EB5XFWLZOO3HAI3TEAQ, seg_pos={0 0}, upload=57107]: Done uploading piece. err: <nil> Jul 26 23:07:16.928322974 (59h31m26.650886014s) [noise=true , address=104.36.82.55:4194 , piece_id=FSQ4MZ5NYGU5GYYUQK7CKORSYZJS6XY6XVNIZYHB5IORIXRAG3ZQ, seg_pos={0 0}, upload=57107]: Done uploading piece. err: <nil> Jul 26 23:07:48.983128039 (59h31m58.705691079s) [noise=true , address=104.36.82.54:4188 , piece_id=S3ULLIMTUOSQF4FWWNQ6NU7DNPWDUDH75BDHK7RL74NPCPHUX33A, seg_pos={0 0}, upload=57107]: Done uploading piece. err: <nil> Jul 26 23:07:49.049779574 (59h31m58.772342614s) [noise=true , address=104.36.82.53:4135 , piece_id=YSYQ7FXVZTOWVL3OE237WGDDJWJKXXKKXUIBQCMZJWUV4STU5GJA, seg_pos={0 0}, upload=57107]: Done uploading piece. err: <nil> Jul 26 23:08:13.738564954 (59h32m23.461127994s) [noise=true , address=104.36.82.43:4859 , piece_id=VFWPRTYYTJJIJNU5TAOC65LTYXXMREPCWAZF2CZG3IDCTVYHP4RA, seg_pos={0 0}, upload=57107]: Done uploading piece. err: ecclient: upload failed (node:126SqCtRGPziEeopRjA4nNkjkCZWvUwkt5XaMgcqZKJwoxuYWY7, address:104.36.82.43:4859): protocol: rpc: dial tcp 104.36.82.43:4859: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:08:13.738608497 (59h32m23.461171537s) [noise=true , address=104.36.82.21:4495 , piece_id=LK3TA2RQWAY5DAQHLX4QOAK2EBZOD4FZ6FJMURYJ6YLTD2WFZ4GQ, seg_pos={0 0}, upload=57107]: Done uploading piece. err: ecclient: upload failed (node:1W4NyaW26WRURmwd6dBbD6aQqqVAMSPsfzKKHfbjtM4sWDMUP3, address:104.36.82.21:4495): protocol: rpc: dial tcp 104.36.82.21:4495: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:08:13.738639105 (59h32m23.461202145s) [noise=true , address=104.36.82.46:4507 , piece_id=576PYZ2DUVRNOYWALSHDA3IY72O4RKSKDYSWNMMI6HUUCXKFGZHQ, seg_pos={0 0}, upload=57107]: Done uploading piece. err: ecclient: upload failed (node:14jurWWPpmxwiBS3XmZysEUwugsceYJqVXFj7BbYpx8UFcTR8R, address:104.36.82.46:4507): protocol: rpc: dial tcp 104.36.82.46:4507: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:08:13.738699771 (59h32m23.461262821s) [noise=true , address=104.36.82.47:4518 , piece_id=SWN25POI54U66UXQCAOXPHEWKJ2QARY7V3XVA5ZPJ5G2BE3IOWMQ, seg_pos={0 0}, upload=57107]: Done uploading piece. err: ecclient: upload failed (node:17TZ9JLdCivFF46guhSTW6hcoUmTz5uo5wvLkDqC4BGmpm9eLb, address:104.36.82.47:4518): protocol: rpc: dial tcp 104.36.82.47:4518: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:08:13.738735190 (59h32m23.46129822s) [noise=true , address=104.36.82.44:4887 , piece_id=TIEB4HHQIBTUMDD3LLTZ42RUMWGPATKVVJJZVHP2UODQBPFREUPQ, seg_pos={0 0}, upload=57107]: Done uploading piece. err: ecclient: upload failed (node:1orvhuWfkUGCykWsS2E5sHtMcEW3fFcCNzTQ4xSxqdhUrBza8H, address:104.36.82.44:4887): protocol: write tcp 188.166.225.170:59614->104.36.82.44:4887: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:08:13.738978175 (59h32m23.461541225s) [noise=true , address=104.36.82.12:4164 , piece_id=A47HMZEE7H5TRROTIESOODM4BFYHV2PPLN7O2UB2PRE6W4ZRGKTQ, seg_pos={0 0}, upload=57107]: Done uploading piece. err: ecclient: upload failed (node:12iQgyDiz9vu1TbCRuATGmnrfbgBaia72b77ZjQJ6gpnEFbQzTJ, address:104.36.82.12:4164): protocol: rpc: dial tcp 104.36.82.12:4164: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:08:13.739009875 (59h32m23.461572915s) [noise=true , address=104.36.82.12:4175 , piece_id=DTKKBL6MNTT3JBCADEEIG6RB7ZM6STSHQDJFUJH7XEYHLRZSAB7A, seg_pos={0 0}, upload=57107]: Done uploading piece. err: ecclient: upload failed (node:12JEJYMUS93YZ9Vxx6LT6psJmt2gn569s7dV7mBJtLsjh2e9zKE, address:104.36.82.12:4175): protocol: rpc: dial tcp 104.36.82.12:4175: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:08:13.739041145 (59h32m23.461604185s) [noise=true , address=104.36.82.42:4824 , piece_id=UMQLXDHZFLZZKIX4LCY6KARUSNL2WDZIB7STHSIJTIV3S4DS5AOQ, seg_pos={0 0}, upload=57107]: Done uploading piece. err: ecclient: upload failed (node:12QTjnomQYFETHbaJBWxWEPo5yweArk68bKPeooZXpRZrpw1ACq, address:104.36.82.42:4824): protocol: write tcp 188.166.225.170:58958->104.36.82.42:4824: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:08:13.739071804 (59h32m23.461634844s) [noise=true , address=104.36.82.7:4923 , piece_id=NDN3RDPB7U6FUAU3BJK4EKGTZQ5JGMPDBYQ26KNPBPE4IMFOML6A, seg_pos={0 0}, upload=57107]: Done uploading piece. err: ecclient: upload failed (node:12Ee9ZfH81oy4SAUxozjQNWJ7ALmWDXQZYPLFwuAiJ8V6E7jT6r, address:104.36.82.7:4923): protocol: rpc: dial tcp 104.36.82.7:4923: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:08:13.739099336 (59h32m23.461662376s) [noise=true , address=104.36.82.22:4537 , piece_id=PVCNQEI434YKFKE4MRL7LA4SRGXALK6CFZXNNHKMFYD7KPMGCOCQ, seg_pos={0 0}, upload=57107]: Done uploading piece. err: ecclient: upload failed (node:12U3QSKhAmoaakTJRReBtoRpzbvzYdibd1xjsEVuU8LD29Fcu1y, address:104.36.82.22:4537): protocol: write tcp 188.166.225.170:42310->104.36.82.22:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:08:13.739111539 (59h32m23.461674589s) [noise=true , address=104.36.82.50:4020 , piece_id=5TCI6DPNEZ3FGU75JFDYVOUIXI36QHPZSDI4FLYALO2CKUJQGN3A, seg_pos={0 0}, upload=57107]: Done uploading piece. err: ecclient: upload failed (node:129LAxYoAkRhCwZLWqeTHSUUMnAQZTotqKyBr3vMTDKF9cEKNCk, address:104.36.82.50:4020): protocol: rpc: dial tcp 104.36.82.50:4020: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:08:13.739127620 (59h32m23.46169066s) [noise=true , address=104.36.82.4:4805 , piece_id=MV7EHWT7O2ZX2WW2RY2Z4PBD22PUT6BGRVLXZWGQUEMZRSZN723Q, seg_pos={0 0}, upload=57107]: Done uploading piece. err: ecclient: upload failed (node:12YEVsEwZwfWsnZFyaSkd9Dn53Tb9EFUyWEVDwTQeMJkcjwrj7x, address:104.36.82.4:4805): protocol: rpc: dial tcp 104.36.82.4:4805: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:08:13.739142629 (59h32m23.46170567s) [seg_pos={0 0}, upload=57107]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Jul 26 23:08:13.970746607 (59h32m23.693309647s) [upload=57107]: Done uploading segments... Jul 26 23:08:13.970773679 (59h32m23.693336729s) [upload=57107]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 1m0.233603283s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................................................XX] duration: 30.000910655s FAIL (2025-07-26 23:08:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000934981s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [............................X.........X...........X.......XX] duration: 30.240972829s FAIL (2025-07-26 23:08:08 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 5609 (59h31m30.80637865s) [upload=57101]: Done uploading segments... Jul 26 23:07:21.083841930 (59h31m30.80640496s) [upload=57101]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Jul 26 23:07:39.140761135 (59h31m48.863324175s) [upload=57115]: Starting upload Jul 26 23:07:39.140817423 (59h31m48.863380463s) [upload=57115]: Done starting upload Jul 26 23:07:39.140910821 (59h31m48.863473861s) [upload=57115]: Uploading segments... Jul 26 23:07:39.141034448 (59h31m48.863597488s) [upload=57115]: Got next segment. Inline: false Jul 26 23:07:39.141069535 (59h31m48.863632575s) [upload=57115]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:07:39.141074694 (59h31m48.863637734s) [upload=57115]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:07:39.378124703 (59h31m49.100687743s) [seg_pos={0 0}, upload=57115]: Begin upload segment... Jul 26 23:07:39.378164179 (59h31m49.100727219s) [seg_pos={0 0}, upload=57115]: Done begin upload segment. Jul 26 23:07:39.378184267 (59h31m49.100747297s) [noise=true , address=77.90.31.14:4662 , piece_id=JMFROLNVFPDMFT6IR7NGTTXHA7MZ7C2EY2QQ7D2V4TE5UFFNAMOQ, seg_pos={0 0}, upload=57115]: Uploading piece... Jul 26 23:07:39.378250554 (59h31m49.100813594s) [noise=true , address=104.36.82.4:4804 , piece_id=PR3PUS4OMX2G63WIVMY7UYSOXQFOMUIZDDJVUWYTCG4ZDJMRIOWA, seg_pos={0 0}, upload=57115]: Uploading piece... Jul 26 23:07:39.378521733 (59h31m49.101084773s) [noise=true , address=45.91.116.13:4155 , piece_id=QJCRJEYVCBBGIGHLLXAYBDOQZNBIVJ6NT2ZNSZ6NXWDBOR7SF6PA, seg_pos={0 0}, upload=57115]: Uploading piece... Jul 26 23:07:39.378683422 (59h31m49.101246462s) [noise=true , address=45.91.116.10:4082 , piece_id=H2TEE66SDC7X5XRRLOG7EB6BICYFDVVWXVWXIDZ774FFZVLE53CA, seg_pos={0 0}, upload=57115]: Uploading piece... Jul 26 23:07:39.378827268 (59h31m49.101390308s) [noise=true , address=45.91.116.4:4940 , piece_id=V5ZZOYC7BKMGJ6JRN7IO7UU6DKUAO4PSGLYCBALQPRM5OC2QIFHQ, seg_pos={0 0}, upload=57115]: Uploading piece... Jul 26 23:07:39.378949221 (59h31m49.101512251s) [noise=true , address=45.91.116.10:4083 , piece_id=QBLOXM3GOS2PLNZM4L2KK2G3ZMKHNXQNJ5TPJJ3IBEO6PJB54A7Q, seg_pos={0 0}, upload=57115]: Uploading piece... Jul 26 23:07:39.379069310 (59h31m49.10163235s) [noise=true , address=45.91.116.5:4298 , piece_id=SI35OO4YD224PCYVGGSHTI6GA2D3HHBFQEPAAS37PHW6DRO33CIA, seg_pos={0 0}, upload=57115]: Uploading piece... Jul 26 23:07:39.379209128 (59h31m49.101772168s) [noise=true , address=104.36.82.2:4731 , piece_id=W7W55QROKNJI2V74A53XX5YTNDYRJNMAKYDZB2SO5PF5EVXRQTZQ, seg_pos={0 0}, upload=57115]: Uploading piece... Jul 26 23:07:39.379338926 (59h31m49.101901956s) [noise=true , address=45.91.116.6:4975 , piece_id=GXBBO3TVDX4JQ7QFQ6ZQANKZRAE7LDZ4PHUCTGSNIBJ3REOONO3A, seg_pos={0 0}, upload=57115]: Uploading piece... Jul 26 23:07:39.379567344 (59h31m49.102130384s) [noise=true , address=104.36.82.12:4149 , piece_id=H6H6WCF5TBQQ6LTRSBUWKSOVOTLULHS2KYQAWIH25BDMUP4Z55WQ, seg_pos={0 0}, upload=57115]: Uploading piece... Jul 26 23:07:39.379737459 (59h31m49.102300499s) [noise=true , address=104.36.82.4:4801 , piece_id=UTCS4FFWCEGEX6VLWYI2CKMZQUEUT4ZOES7HFLY5ETAUBQJXFQSA, seg_pos={0 0}, upload=57115]: Uploading piece... Jul 26 23:07:39.379856717 (59h31m49.102419757s) [noise=true , address=104.36.82.15:4258 , piece_id=6JUSWVK5WTMLMAOLWT3MPG6UZ2DV7T2WONFS6HSJ4CZ6RQFWR7FQ, seg_pos={0 0}, upload=57115]: Uploading piece... Jul 26 23:07:39.380083581 (59h31m49.102646621s) [noise=true , address=104.36.82.8:4971 , piece_id=GVFRGYOCQFBSO2JWVURBKH2PX4XNAOL3SFHCGOPDLW2NHR4BHSTQ, seg_pos={0 0}, upload=57115]: Uploading piece... Jul 26 23:07:39.380103910 (59h31m49.10266695s) [noise=true , address=45.91.116.8:4047 , piece_id=NARV65RR4DZGBGGJLAYUP55J6AL7RJEBCWNYSTXACSJVNDKIE74A, seg_pos={0 0}, upload=57115]: Uploading piece... Jul 26 23:07:39.380420967 (59h31m49.102984007s) [noise=true , address=104.36.82.3:4767 , piece_id=G5S24DLMB75JVINVSTIKV7J24BMLTSPOL76UR4WGQPXTJ66AW22A, seg_pos={0 0}, upload=57115]: Uploading piece... Jul 26 23:07:39.380637540 (59h31m49.10320057s) [noise=true , address=45.91.116.11:4119 , piece_id=XE645ZQWDHAJVKL2NTFOHBML55PCM6EX23LMUEGMLEIPEUKJXQWQ, seg_pos={0 0}, upload=57115]: Uploading piece... Jul 26 23:07:41.105109341 (59h31m50.827672381s) [upload=57115]: Next returned nil segment Jul 26 23:07:41.105145249 (59h31m50.827708289s) [upload=57115]: Waiting for error group managing segments... Jul 26 23:07:41.386743873 (59h31m51.109306913s) [noise=true , address=77.90.31.14:4662 , piece_id=JMFROLNVFPDMFT6IR7NGTTXHA7MZ7C2EY2QQ7D2V4TE5UFFNAMOQ, seg_pos={0 0}, upload=57115]: Done uploading piece. err: <nil> Jul 26 23:07:41.394545530 (59h31m51.11710857s) [noise=true , address=45.91.116.11:4119 , piece_id=XE645ZQWDHAJVKL2NTFOHBML55PCM6EX23LMUEGMLEIPEUKJXQWQ, seg_pos={0 0}, upload=57115]: Done uploading piece. err: <nil> Jul 26 23:07:41.395028545 (59h31m51.117591585s) [noise=true , address=45.91.116.8:4047 , piece_id=NARV65RR4DZGBGGJLAYUP55J6AL7RJEBCWNYSTXACSJVNDKIE74A, seg_pos={0 0}, upload=57115]: Done uploading piece. err: <nil> Jul 26 23:07:41.401075243 (59h31m51.123638283s) [noise=true , address=45.91.116.5:4298 , piece_id=SI35OO4YD224PCYVGGSHTI6GA2D3HHBFQEPAAS37PHW6DRO33CIA, seg_pos={0 0}, upload=57115]: Done uploading piece. err: <nil> Jul 26 23:07:41.401362894 (59h31m51.123925934s) [noise=true , address=45.91.116.4:4940 , piece_id=V5ZZOYC7BKMGJ6JRN7IO7UU6DKUAO4PSGLYCBALQPRM5OC2QIFHQ, seg_pos={0 0}, upload=57115]: Done uploading piece. err: <nil> Jul 26 23:07:41.424038450 (59h31m51.14660148s) [noise=true , address=45.91.116.6:4975 , piece_id=GXBBO3TVDX4JQ7QFQ6ZQANKZRAE7LDZ4PHUCTGSNIBJ3REOONO3A, seg_pos={0 0}, upload=57115]: Done uploading piece. err: <nil> Jul 26 23:07:41.484360774 (59h31m51.206923825s) [noise=true , address=45.91.116.10:4083 , piece_id=QBLOXM3GOS2PLNZM4L2KK2G3ZMKHNXQNJ5TPJJ3IBEO6PJB54A7Q, seg_pos={0 0}, upload=57115]: Done uploading piece. err: <nil> Jul 26 23:07:41.502340385 (59h31m51.224903425s) [noise=true , address=45.91.116.13:4155 , piece_id=QJCRJEYVCBBGIGHLLXAYBDOQZNBIVJ6NT2ZNSZ6NXWDBOR7SF6PA, seg_pos={0 0}, upload=57115]: Done uploading piece. err: <nil> Jul 26 23:07:41.502582177 (59h31m51.225145217s) [noise=true , address=45.91.116.10:4082 , piece_id=H2TEE66SDC7X5XRRLOG7EB6BICYFDVVWXVWXIDZ774FFZVLE53CA, seg_pos={0 0}, upload=57115]: Done uploading piece. err: <nil> Jul 26 23:08:08.492057128 (59h32m18.214620168s) [noise=true , address=104.36.82.15:4258 , piece_id=6JUSWVK5WTMLMAOLWT3MPG6UZ2DV7T2WONFS6HSJ4CZ6RQFWR7FQ, seg_pos={0 0}, upload=57115]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: rpc: dial tcp 104.36.82.15:4258: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:08:08.492086324 (59h32m18.214649364s) [noise=true , address=104.36.82.3:4767 , piece_id=G5S24DLMB75JVINVSTIKV7J24BMLTSPOL76UR4WGQPXTJ66AW22A, seg_pos={0 0}, upload=57115]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: rpc: dial tcp 104.36.82.3:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:08:08.492100161 (59h32m18.214663201s) [noise=true , address=104.36.82.4:4804 , piece_id=PR3PUS4OMX2G63WIVMY7UYSOXQFOMUIZDDJVUWYTCG4ZDJMRIOWA, seg_pos={0 0}, upload=57115]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: rpc: dial tcp 104.36.82.4:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:08:08.492123827 (59h32m18.214686867s) [noise=true , address=104.36.82.12:4149 , piece_id=H6H6WCF5TBQQ6LTRSBUWKSOVOTLULHS2KYQAWIH25BDMUP4Z55WQ, seg_pos={0 0}, upload=57115]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 188.166.225.170:35378->104.36.82.12:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:08:08.492132603 (59h32m18.214695643s) [noise=true , address=104.36.82.2:4731 , piece_id=W7W55QROKNJI2V74A53XX5YTNDYRJNMAKYDZB2SO5PF5EVXRQTZQ, seg_pos={0 0}, upload=57115]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: rpc: dial tcp 104.36.82.2:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:08:08.492154846 (59h32m18.214717886s) [noise=true , address=104.36.82.4:4801 , piece_id=UTCS4FFWCEGEX6VLWYI2CKMZQUEUT4ZOES7HFLY5ETAUBQJXFQSA, seg_pos={0 0}, upload=57115]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:08:08.492160967 (59h32m18.214723997s) [noise=true , address=104.36.82.8:4971 , piece_id=GVFRGYOCQFBSO2JWVURBKH2PX4XNAOL3SFHCGOPDLW2NHR4BHSTQ, seg_pos={0 0}, upload=57115]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: write tcp 188.166.225.170:42866->104.36.82.8:4971: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:08:08.492184372 (59h32m18.214747412s) [seg_pos={0 0}, upload=57115]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Jul 26 23:08:08.731022324 (59h32m18.453585374s) [upload=57115]: Done uploading segments... Jul 26 23:08:08.731052411 (59h32m18.453615451s) [upload=57115]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.24099393s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 1.940193131s FAIL (2025-07-26 23:08:07 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.940224781s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [......X..........X.X........X.X...........................XX] duration: 30.235372537s FAIL (2025-07-26 23:07:50 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.235392524s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 17 out of 60 is failed (threshold=3) [.....X...X...............X..XX...X.X.X.X.......XX..XX.XX..XX] duration: 1m0.001061971s FAIL (2025-07-26 23:07:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001082089s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.001362203s FAIL (2025-07-26 23:07:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.00138099s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.............................X.........X...........X.......X] duration: 30.239615023s FAIL (2025-07-26 23:07:21 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 43s) [seg_pos={0 0}, upload=57093]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 23:06:21.524718264 (59h30m31.247281304s) [upload=57093]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 23:06:21.524722242 (59h30m31.247285282s) [upload=57093]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 23:06:21.858504729 (59h30m31.581067769s) [upload=57093]: Done uploading segments... Jul 26 23:06:21.858521230 (59h30m31.58108426s) [upload=57093]: Upload finished. err: <nil> Jul 26 23:06:51.505853761 (59h31m1.228416801s) [upload=57101]: Starting upload Jul 26 23:06:51.505936219 (59h31m1.228499259s) [upload=57101]: Done starting upload Jul 26 23:06:51.506035629 (59h31m1.228598669s) [upload=57101]: Uploading segments... Jul 26 23:06:51.506175457 (59h31m1.228738497s) [upload=57101]: Got next segment. Inline: false Jul 26 23:06:51.506209081 (59h31m1.228772121s) [upload=57101]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:06:51.506216555 (59h31m1.228779595s) [upload=57101]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:06:51.745042582 (59h31m1.467605612s) [seg_pos={0 0}, upload=57101]: Begin upload segment... Jul 26 23:06:51.745097186 (59h31m1.467660226s) [seg_pos={0 0}, upload=57101]: Done begin upload segment. Jul 26 23:06:51.745122034 (59h31m1.467685074s) [noise=true , address=45.91.116.7:4010 , piece_id=55T2Y4I6LJMSBGLQ25K5GWYEEFS2BAU5AIGN43ZJQFW5OVUSBVUQ, seg_pos={0 0}, upload=57101]: Uploading piece... Jul 26 23:06:51.745163593 (59h31m1.467726633s) [noise=true , address=104.36.82.2:4730 , piece_id=ZOCGTQO3CCR4QT4S3Q3UPE2I23UJQNQR4UPNVIVPXTGDONXS4HDQ, seg_pos={0 0}, upload=57101]: Uploading piece... Jul 26 23:06:51.745515496 (59h31m1.468078526s) [noise=true , address=45.91.116.10:4081 , piece_id=DKHDZBCFC7TNLUARYIDJHEADQR2EHDDBFZWWZLMTYOMBGQFBOG5A, seg_pos={0 0}, upload=57101]: Uploading piece... Jul 26 23:06:51.745642620 (59h31m1.46820565s) [noise=true , address=104.36.82.6:4873 , piece_id=HOT2MEFCBA6WTIZYQ5B7I4ZM3AEZLNV3OSIEHMU4HFYHUJ2EPRKA, seg_pos={0 0}, upload=57101]: Uploading piece... Jul 26 23:06:51.745856019 (59h31m1.468419059s) [noise=true , address=77.90.31.21:4878 , piece_id=LMTCQKNMHU2Y7Z74AHRYNTQ7VOEJ66JXEF6GMMZLG67GHGSZ2MLA, seg_pos={0 0}, upload=57101]: Uploading piece... Jul 26 23:06:51.745974365 (59h31m1.468537405s) [noise=true , address=104.36.82.15:4259 , piece_id=UW2KT4TYX7QNDV3G56AOZL5VAWTNQXMVNRIDQN3K65MOA3U4NT2A, seg_pos={0 0}, upload=57101]: Uploading piece... Jul 26 23:06:51.746134611 (59h31m1.468697651s) [noise=true , address=77.90.31.13:4628 , piece_id=UH375J7WZV4TWO572DI5XKLHVR33M2FAG72XG24HCXCDU2XILOQA, seg_pos={0 0}, upload=57101]: Uploading piece... Jul 26 23:06:51.746254861 (59h31m1.468817891s) [noise=true , address=45.91.116.10:4083 , piece_id=PWRMIWU7FSUJUIV4KAIECOSTHKPBDMHZSQCHYCJLYRHP4VQG5OUA, seg_pos={0 0}, upload=57101]: Uploading piece... Jul 26 23:06:51.746425527 (59h31m1.468988568s) [noise=true , address=45.91.116.6:4975 , piece_id=MMAHUUVK7ZF5DYJ4AJSOGJSOVMJSRC7LFS4X7TZPAIHNVDNM6TLA, seg_pos={0 0}, upload=57101]: Uploading piece... Jul 26 23:06:51.746528344 (59h31m1.469091384s) [noise=true , address=45.91.116.13:4154 , piece_id=3VPMGA7FTKH7D5GGFNTOUHXISTOLFAH7HJ3QY54VW5DYE2AQK6JA, seg_pos={0 0}, upload=57101]: Uploading piece... Jul 26 23:06:51.746692509 (59h31m1.469255549s) [noise=true , address=104.36.82.2:4731 , piece_id=3VAQDLUIYTHT7QBHWVCA3V6TAWFMOWKBWKTMYAUPRQKIZJYTAJVA, seg_pos={0 0}, upload=57101]: Uploading piece... Jul 26 23:06:51.746800495 (59h31m1.469363535s) [noise=true , address=45.91.116.8:4045 , piece_id=25PYBFVXOHWKCT2QBN4S5DI57XPNRUKNLYIE46Q3PREQNZLYOXDA, seg_pos={0 0}, upload=57101]: Uploading piece... Jul 26 23:06:51.746935744 (59h31m1.469498784s) [noise=true , address=45.91.116.4:4939 , piece_id=3LGXVHQI2PCOK4ZJ5IBUIHAMTGBQC4PT5BW7FJBJY7BYWZQ4R2AA, seg_pos={0 0}, upload=57101]: Uploading piece... Jul 26 23:06:51.747073017 (59h31m1.469636057s) [noise=true , address=45.91.116.12:4227 , piece_id=OG2DJS4WKRRVZOAJHVZF5WUEE2LQBJKQHSMAWUPVFYLNK7EX5UJQ, seg_pos={0 0}, upload=57101]: Uploading piece... Jul 26 23:06:51.747198446 (59h31m1.469761476s) [noise=true , address=104.36.82.4:4801 , piece_id=IFHRSSKB52W2PJQQ3HMLAYBD266JLC2ZGFF6EJNSHOM4MBBGP55Q, seg_pos={0 0}, upload=57101]: Uploading piece... Jul 26 23:06:51.747312515 (59h31m1.469875555s) [noise=true , address=77.90.31.15:4698 , piece_id=ACDMNCXI7QY5VTOZTF44UUEVRYZKVKAFGPPCITQ5PAIMDC63AGIA, seg_pos={0 0}, upload=57101]: Uploading piece... Jul 26 23:06:53.500544911 (59h31m3.223107951s) [upload=57101]: Next returned nil segment Jul 26 23:06:53.500560932 (59h31m3.223123962s) [upload=57101]: Waiting for error group managing segments... Jul 26 23:06:53.777052385 (59h31m3.499615425s) [noise=true , address=45.91.116.10:4081 , piece_id=DKHDZBCFC7TNLUARYIDJHEADQR2EHDDBFZWWZLMTYOMBGQFBOG5A, seg_pos={0 0}, upload=57101]: Done uploading piece. err: <nil> Jul 26 23:06:53.779757010 (59h31m3.50232005s) [noise=true , address=45.91.116.12:4227 , piece_id=OG2DJS4WKRRVZOAJHVZF5WUEE2LQBJKQHSMAWUPVFYLNK7EX5UJQ, seg_pos={0 0}, upload=57101]: Done uploading piece. err: <nil> Jul 26 23:06:53.785660265 (59h31m3.508223305s) [noise=true , address=45.91.116.13:4154 , piece_id=3VPMGA7FTKH7D5GGFNTOUHXISTOLFAH7HJ3QY54VW5DYE2AQK6JA, seg_pos={0 0}, upload=57101]: Done uploading piece. err: <nil> Jul 26 23:06:53.785716633 (59h31m3.508279673s) [noise=true , address=77.90.31.21:4878 , piece_id=LMTCQKNMHU2Y7Z74AHRYNTQ7VOEJ66JXEF6GMMZLG67GHGSZ2MLA, seg_pos={0 0}, upload=57101]: Done uploading piece. err: <nil> Jul 26 23:06:53.845351792 (59h31m3.567914832s) [noise=true , address=45.91.116.4:4939 , piece_id=3LGXVHQI2PCOK4ZJ5IBUIHAMTGBQC4PT5BW7FJBJY7BYWZQ4R2AA, seg_pos={0 0}, upload=57101]: Done uploading piece. err: <nil> Jul 26 23:06:53.845933245 (59h31m3.568496285s) [noise=true , address=45.91.116.10:4083 , piece_id=PWRMIWU7FSUJUIV4KAIECOSTHKPBDMHZSQCHYCJLYRHP4VQG5OUA, seg_pos={0 0}, upload=57101]: Done uploading piece. err: <nil> Jul 26 23:06:53.884572433 (59h31m3.607135463s) [noise=true , address=77.90.31.15:4698 , piece_id=ACDMNCXI7QY5VTOZTF44UUEVRYZKVKAFGPPCITQ5PAIMDC63AGIA, seg_pos={0 0}, upload=57101]: Done uploading piece. err: <nil> Jul 26 23:06:53.914750604 (59h31m3.637313644s) [noise=true , address=45.91.116.8:4045 , piece_id=25PYBFVXOHWKCT2QBN4S5DI57XPNRUKNLYIE46Q3PREQNZLYOXDA, seg_pos={0 0}, upload=57101]: Done uploading piece. err: <nil> Jul 26 23:06:53.924084913 (59h31m3.646647953s) [noise=true , address=45.91.116.7:4010 , piece_id=55T2Y4I6LJMSBGLQ25K5GWYEEFS2BAU5AIGN43ZJQFW5OVUSBVUQ, seg_pos={0 0}, upload=57101]: Done uploading piece. err: <nil> Jul 26 23:06:54.062689285 (59h31m3.785252325s) [noise=true , address=45.91.116.6:4975 , piece_id=MMAHUUVK7ZF5DYJ4AJSOGJSOVMJSRC7LFS4X7TZPAIHNVDNM6TLA, seg_pos={0 0}, upload=57101]: Done uploading piece. err: <nil> Jul 26 23:07:20.846086451 (59h31m30.568649491s) [noise=true , address=104.36.82.15:4259 , piece_id=UW2KT4TYX7QNDV3G56AOZL5VAWTNQXMVNRIDQN3K65MOA3U4NT2A, seg_pos={0 0}, upload=57101]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: rpc: dial tcp 104.36.82.15:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:07:20.846148229 (59h31m30.568711259s) [noise=true , address=104.36.82.2:4730 , piece_id=ZOCGTQO3CCR4QT4S3Q3UPE2I23UJQNQR4UPNVIVPXTGDONXS4HDQ, seg_pos={0 0}, upload=57101]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: rpc: dial tcp 104.36.82.2:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:07:20.846184097 (59h31m30.568747137s) [noise=true , address=77.90.31.13:4628 , piece_id=UH375J7WZV4TWO572DI5XKLHVR33M2FAG72XG24HCXCDU2XILOQA, seg_pos={0 0}, upload=57101]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: rpc: dial tcp 77.90.31.13:4628: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:07:20.846214906 (59h31m30.568777946s) [noise=true , address=104.36.82.4:4801 , piece_id=IFHRSSKB52W2PJQQ3HMLAYBD266JLC2ZGFF6EJNSHOM4MBBGP55Q, seg_pos={0 0}, upload=57101]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 188.166.225.170:52640->104.36.82.4:4801: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:07:20.846232380 (59h31m30.56879542s) [noise=true , address=104.36.82.6:4873 , piece_id=HOT2MEFCBA6WTIZYQ5B7I4ZM3AEZLNV3OSIEHMU4HFYHUJ2EPRKA, seg_pos={0 0}, upload=57101]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:104.36.82.6:4873): protocol: rpc: dial tcp 104.36.82.6:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:07:20.846266054 (59h31m30.568829094s) [noise=true , address=104.36.82.2:4731 , piece_id=3VAQDLUIYTHT7QBHWVCA3V6TAWFMOWKBWKTMYAUPRQKIZJYTAJVA, seg_pos={0 0}, upload=57101]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: rpc: dial tcp 104.36.82.2:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:07:20.846293787 (59h31m30.568856827s) [seg_pos={0 0}, upload=57101]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 26 23:07:21.083815609 (59h31m30.80637865s) [upload=57101]: Done uploading segments... Jul 26 23:07:21.083841930 (59h31m30.80640496s) [upload=57101]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.239635943s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 1.893560059s FAIL (2025-07-26 23:07:14 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.893589274s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 52 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXX.X.XX.XX.X.XXXXXXXXX...XXXXXXXXXXXXX] duration: 1m0.233220124s FAIL (2025-07-26 23:07:13 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== 80 (59h30m24.15342651s) [noise=true , address=104.36.82.51:4055 , piece_id=TT7HOCCK4KJLSATEZC234Z7EIHXGIMJ4IG3RDDZZEUN6GEJ3DWOQ, seg_pos={0 0}, upload=57091]: Uploading piece... Jul 26 23:06:14.431109330 (59h30m24.15367237s) [noise=true , address=104.36.82.14:4256 , piece_id=LPZYMI44SWDQDKTZFZL2U3GM7KRZNWGX7UWMSSIUKI6YUBWRMY6Q, seg_pos={0 0}, upload=57091]: Uploading piece... Jul 26 23:06:14.431362455 (59h30m24.153925495s) [noise=true , address=104.36.82.13:4196 , piece_id=6AP54CKEFIT4VB5FSVFVGBSG5Y53NLVK4HZW5ADNL47FGAPR4D5Q, seg_pos={0 0}, upload=57091]: Uploading piece... Jul 26 23:06:14.431679521 (59h30m24.154242551s) [noise=true , address=104.36.82.21:4477 , piece_id=6W2NDYX27VIJFWKH2PYNN7DCNLSLU7K2WHH4IYXVD57BRJ4KBVHQ, seg_pos={0 0}, upload=57091]: Uploading piece... Jul 26 23:06:14.432063286 (59h30m24.154626326s) [noise=true , address=104.36.82.13:4215 , piece_id=ILLAHTYKYVGXTZRPVSY5MU2G25N4XC3QCVB7UDJOFVEQNLSSSXHQ, seg_pos={0 0}, upload=57091]: Uploading piece... Jul 26 23:06:14.432334966 (59h30m24.154898006s) [noise=true , address=45.91.116.9:4267 , piece_id=Z7DWTKBOOBRQDWKHNAKVMAILLO6K5BVTZJPYMTO5ZUCGWGJTAB5A, seg_pos={0 0}, upload=57091]: Uploading piece... Jul 26 23:06:14.432538124 (59h30m24.155101164s) [noise=true , address=104.36.82.54:4183 , piece_id=GE5LJETPLXZPIGDAADI64OQL35YQFJY3UDWAZPJKSCCZYFKTRKXQ, seg_pos={0 0}, upload=57091]: Uploading piece... Jul 26 23:06:14.432729881 (59h30m24.155292921s) [noise=true , address=104.36.82.11:4119 , piece_id=YTNPFZ5M4CJPRXMSWL57CP7RBDKJVCCSZWDC3S6L7PIKQNWLLQYQ, seg_pos={0 0}, upload=57091]: Uploading piece... Jul 26 23:06:14.432964039 (59h30m24.155527079s) [noise=true , address=104.36.82.47:4546 , piece_id=EIQC2HCMFZWGNYLX2O7JLQPEQB3SWKMEPE5W7LDTWIYZTO6SYS4A, seg_pos={0 0}, upload=57091]: Uploading piece... Jul 26 23:06:14.433164082 (59h30m24.155727122s) [noise=true , address=104.36.82.42:4801 , piece_id=62DFPZGDKEMYULRJC5V46WTSB7CRW7NYNP7WWR3RCGP3XQ274TEA, seg_pos={0 0}, upload=57091]: Uploading piece... Jul 26 23:06:14.433355728 (59h30m24.155918758s) [noise=true , address=104.36.82.46:4498 , piece_id=UNYPUUFNUTR6NLTPA7AYSLRSZCNA65375B2VYIABCM7PYOUF5P5A, seg_pos={0 0}, upload=57091]: Uploading piece... Jul 26 23:06:14.433536635 (59h30m24.156099675s) [noise=true , address=104.36.82.52:4112 , piece_id=6RVY5BSWONDLRFFXA3IACSCWENJLWGCX5NAX5ZRDHFU5JFZAVMXQ, seg_pos={0 0}, upload=57091]: Uploading piece... Jul 26 23:06:14.433715066 (59h30m24.156278106s) [noise=true , address=104.36.82.4:4825 , piece_id=NQVZZUVDLF6ZBKGDKATMPZRJZ2LJR2JT2DZHXRMZ3ZSWFQPGOUBQ, seg_pos={0 0}, upload=57091]: Uploading piece... Jul 26 23:06:14.433914347 (59h30m24.156477387s) [noise=true , address=104.36.82.12:4161 , piece_id=BNQZS5Y2H6PT4YR36ZQI6SUQHP4JFPFOA73LTJEJUW76AYKKFEWA, seg_pos={0 0}, upload=57091]: Uploading piece... Jul 26 23:06:15.920626449 (59h30m25.643189489s) [upload=57091]: Next returned nil segment Jul 26 23:06:15.920642691 (59h30m25.643205731s) [upload=57091]: Waiting for error group managing segments... Jul 26 23:06:16.343669871 (59h30m26.066232921s) [noise=true , address=104.36.82.52:4112 , piece_id=6RVY5BSWONDLRFFXA3IACSCWENJLWGCX5NAX5ZRDHFU5JFZAVMXQ, seg_pos={0 0}, upload=57091]: Done uploading piece. err: <nil> Jul 26 23:06:16.348335358 (59h30m26.070898408s) [noise=true , address=104.36.82.46:4498 , piece_id=UNYPUUFNUTR6NLTPA7AYSLRSZCNA65375B2VYIABCM7PYOUF5P5A, seg_pos={0 0}, upload=57091]: Done uploading piece. err: <nil> Jul 26 23:06:16.417836615 (59h30m26.140399655s) [noise=true , address=104.36.82.53:4141 , piece_id=PBVOQB4A2OJD53BSGJUGBWPHHUA6TYSAFQZPDUS7VKK7JRWNB7LQ, seg_pos={0 0}, upload=57091]: Done uploading piece. err: <nil> Jul 26 23:06:16.423213032 (59h30m26.145776072s) [noise=true , address=104.36.82.47:4546 , piece_id=EIQC2HCMFZWGNYLX2O7JLQPEQB3SWKMEPE5W7LDTWIYZTO6SYS4A, seg_pos={0 0}, upload=57091]: Done uploading piece. err: <nil> Jul 26 23:06:16.428993752 (59h30m26.151556792s) [noise=true , address=104.36.82.56:4255 , piece_id=YGPR4F255A7CEL5QRDLK4W6TD3AVRGMIR6CNPKZGILUABXPGQ32Q, seg_pos={0 0}, upload=57091]: Done uploading piece. err: <nil> Jul 26 23:06:16.453509248 (59h30m26.176072288s) [noise=true , address=104.36.82.54:4183 , piece_id=GE5LJETPLXZPIGDAADI64OQL35YQFJY3UDWAZPJKSCCZYFKTRKXQ, seg_pos={0 0}, upload=57091]: Done uploading piece. err: <nil> Jul 26 23:06:20.889361183 (59h30m30.611924213s) [noise=true , address=45.91.116.9:4267 , piece_id=Z7DWTKBOOBRQDWKHNAKVMAILLO6K5BVTZJPYMTO5ZUCGWGJTAB5A, seg_pos={0 0}, upload=57091]: Done uploading piece. err: <nil> Jul 26 23:06:39.414415451 (59h30m49.136978521s) [noise=true , address=77.90.31.11:4565 , piece_id=4MHJBYLOV5HWWUVACSJWLWGDGTPW7YLXLQ35Z2VQGGIHVTBLS63Q, seg_pos={0 0}, upload=57091]: Done uploading piece. err: <nil> Jul 26 23:07:13.272952035 (59h31m22.995515075s) [noise=true , address=104.36.82.13:4215 , piece_id=ILLAHTYKYVGXTZRPVSY5MU2G25N4XC3QCVB7UDJOFVEQNLSSSXHQ, seg_pos={0 0}, upload=57091]: Done uploading piece. err: ecclient: upload failed (node:1vV1hGhZqeEoh5ei7yRu49oyAvTZ2T7S1x3ooufJPtFH2TGBkQ, address:104.36.82.13:4215): protocol: write tcp 188.166.225.170:42332->104.36.82.13:4215: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:07:13.273039763 (59h31m22.995602793s) [noise=true , address=104.36.82.11:4119 , piece_id=YTNPFZ5M4CJPRXMSWL57CP7RBDKJVCCSZWDC3S6L7PIKQNWLLQYQ, seg_pos={0 0}, upload=57091]: Done uploading piece. err: ecclient: upload failed (node:121xNAnkm9HZXUszp26UmWTsuhp3igyeDVea6gSCdtrQUQDVHpW, address:104.36.82.11:4119): protocol: write tcp 188.166.225.170:52682->104.36.82.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:07:13.273111941 (59h31m22.995674981s) [noise=true , address=104.36.82.4:4825 , piece_id=NQVZZUVDLF6ZBKGDKATMPZRJZ2LJR2JT2DZHXRMZ3ZSWFQPGOUBQ, seg_pos={0 0}, upload=57091]: Done uploading piece. err: ecclient: upload failed (node:121Bec7R14ujGEdoUqFTikngLVLb6jmLAcHkeBAaCkhZ5dr6APb, address:104.36.82.4:4825): protocol: write tcp 188.166.225.170:50500->104.36.82.4:4825: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:07:13.273185081 (59h31m22.995748132s) [noise=true , address=104.36.82.12:4161 , piece_id=BNQZS5Y2H6PT4YR36ZQI6SUQHP4JFPFOA73LTJEJUW76AYKKFEWA, seg_pos={0 0}, upload=57091]: Done uploading piece. err: ecclient: upload failed (node:1SdSwg6tZCKSturNytQmWUmSWQ2G5dUSrb2RA8hxLLpBtBE6tf, address:104.36.82.12:4161): protocol: write tcp 188.166.225.170:47824->104.36.82.12:4161: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:07:13.273235358 (59h31m22.995798398s) [noise=true , address=104.36.82.51:4055 , piece_id=TT7HOCCK4KJLSATEZC234Z7EIHXGIMJ4IG3RDDZZEUN6GEJ3DWOQ, seg_pos={0 0}, upload=57091]: Done uploading piece. err: ecclient: upload failed (node:1SAWKJT2JEb9ReJUGDZ545Rp7LeQoCJKYsQgmrDA9csufPf8us, address:104.36.82.51:4055): protocol: write tcp 188.166.225.170:57050->104.36.82.51:4055: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:07:13.273306553 (59h31m22.995869604s) [noise=true , address=104.36.82.13:4196 , piece_id=6AP54CKEFIT4VB5FSVFVGBSG5Y53NLVK4HZW5ADNL47FGAPR4D5Q, seg_pos={0 0}, upload=57091]: Done uploading piece. err: ecclient: upload failed (node:1aLZa6kv5VhDgqvUZp2HkzpkbA81KU13iGTTsiVfSeUDzLu8Lu, address:104.36.82.13:4196): protocol: write tcp 188.166.225.170:42682->104.36.82.13:4196: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:07:13.273373743 (59h31m22.995936783s) [noise=true , address=104.36.82.14:4256 , piece_id=LPZYMI44SWDQDKTZFZL2U3GM7KRZNWGX7UWMSSIUKI6YUBWRMY6Q, seg_pos={0 0}, upload=57091]: Done uploading piece. err: ecclient: upload failed (node:12rxUCiyihymapVWXRYTgyrg2VMHxMe1HnYiV9JCKgaRyiFLiPm, address:104.36.82.14:4256): protocol: write tcp 188.166.225.170:58044->104.36.82.14:4256: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:07:13.273416734 (59h31m22.995979774s) [noise=true , address=104.36.82.21:4477 , piece_id=6W2NDYX27VIJFWKH2PYNN7DCNLSLU7K2WHH4IYXVD57BRJ4KBVHQ, seg_pos={0 0}, upload=57091]: Done uploading piece. err: ecclient: upload failed (node:1QNztqPaSGyZVsQihaj9YjG9C1nzKi1crfSDjDHimYboYAKWZY, address:104.36.82.21:4477): protocol: write tcp 188.166.225.170:33672->104.36.82.21:4477: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:07:13.273484304 (59h31m22.996047344s) [noise=true , address=104.36.82.42:4801 , piece_id=62DFPZGDKEMYULRJC5V46WTSB7CRW7NYNP7WWR3RCGP3XQ274TEA, seg_pos={0 0}, upload=57091]: Done uploading piece. err: ecclient: upload failed (node:12pdMen2HsTBypaNt4cUu12TWjhbKnk3V287JJus1273x28xDXM, address:104.36.82.42:4801): protocol: write tcp 188.166.225.170:40208->104.36.82.42:4801: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:07:13.273556071 (59h31m22.996119111s) [noise=true , address=104.36.82.20:4440 , piece_id=BSHU5AMI5KSRKDT3VKQI6NMDYR2QM52QAJSXKUCKBJ3CJOONTPLA, seg_pos={0 0}, upload=57091]: Done uploading piece. err: ecclient: upload failed (node:12GfuKp6kJEy1NTHpo9i21UDm62hrAtKq7124Z6FUehNyyLQ33i, address:104.36.82.20:4440): protocol: write tcp 188.166.225.170:44478->104.36.82.20:4440: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:07:13.273575067 (59h31m22.996138118s) [seg_pos={0 0}, upload=57091]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Jul 26 23:07:13.503221030 (59h31m23.22578407s) [upload=57091]: Done uploading segments... Jul 26 23:07:13.503259224 (59h31m23.225822264s) [upload=57091]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 1m0.233230804s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1.509600607s FAIL (2025-07-26 23:06:57 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.509630233s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [.......X..........X.X........X.X...........................X] duration: 30.279478015s FAIL (2025-07-26 23:06:52 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.279500139s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 16 out of 60 is failed (threshold=3) [......X...X...............X..XX...X.X.X.X.......XX..XX.XX..X] duration: 1m0.000331185s FAIL (2025-07-26 23:06:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000351565s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............................X.........X...........X.......] duration: 5.989529603s FAIL (2025-07-26 23:06:24 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== UPht7vBRdA3BcqRek): protocol: rpc: dial tcp 77.90.31.11:4554: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:00:36.538079013 (59h24m46.260642053s) [seg_pos={0 0}, upload=57008]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 23:00:36.538172713 (59h24m46.260735753s) [upload=57008]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 23:00:36.538178714 (59h24m46.260741784s) [upload=57008]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 23:00:36.813206693 (59h24m46.535769733s) [upload=57008]: Done uploading segments... Jul 26 23:00:36.813224688 (59h24m46.535787728s) [upload=57008]: Upload finished. err: <nil> Jul 26 23:01:26.980982718 (59h25m36.703545748s) [upload=57021]: Starting upload Jul 26 23:01:26.981034367 (59h25m36.703597407s) [upload=57021]: Done starting upload Jul 26 23:01:26.981136591 (59h25m36.703699632s) [upload=57021]: Uploading segments... Jul 26 23:01:26.981370199 (59h25m36.703933229s) [upload=57021]: Got next segment. Inline: false Jul 26 23:01:26.981405947 (59h25m36.703968987s) [upload=57021]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:01:26.981412309 (59h25m36.703975349s) [upload=57021]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:01:27.212088890 (59h25m36.93465193s) [seg_pos={0 0}, upload=57021]: Begin upload segment... Jul 26 23:01:27.212149156 (59h25m36.934712196s) [seg_pos={0 0}, upload=57021]: Done begin upload segment. Jul 26 23:01:27.212191998 (59h25m36.934755038s) [noise=true , address=45.91.116.8:4048 , piece_id=XPYI4RBD4CPXZDEQQDNOHSOFIGFIN5FOLGTCAEUDJ54NECUNIEAA, seg_pos={0 0}, upload=57021]: Uploading piece... Jul 26 23:01:27.212264527 (59h25m36.934827577s) [noise=true , address=45.91.116.5:4298 , piece_id=K7332VRJDC2KTKZ3LVD3IRVGXT6HOYKD4MPER454SXOE4GFW67LQ, seg_pos={0 0}, upload=57021]: Uploading piece... Jul 26 23:01:27.212718034 (59h25m36.935281074s) [noise=true , address=77.90.31.11:4555 , piece_id=GED5M25FMD6GAM42AVXKR7U7XONLKFRQPPLYVHGKCHTTCRC6AFOA, seg_pos={0 0}, upload=57021]: Uploading piece... Jul 26 23:01:27.212787557 (59h25m36.935350588s) [noise=true , address=45.91.116.7:4009 , piece_id=3ORNCRFYD5OI5SPXYZEMSVMZ5LDIEEK7YOUA7JERUKAKZ7QDMMKA, seg_pos={0 0}, upload=57021]: Uploading piece... Jul 26 23:01:27.213064708 (59h25m36.935627748s) [noise=true , address=77.90.31.16:4734 , piece_id=ESJWP6PQ7TWJY24SAYQUGVYF7CJP6WUXAKLLH6BXHXOB6KOTQ4AQ, seg_pos={0 0}, upload=57021]: Uploading piece... Jul 26 23:01:27.213128920 (59h25m36.93569196s) [noise=true , address=77.90.31.21:4879 , piece_id=TERNE4P5KQB5WZHFHY5WJEDQH4JNPBCHN7DPMTZCT6GOY6RMH53Q, seg_pos={0 0}, upload=57021]: Uploading piece... Jul 26 23:01:27.213497365 (59h25m36.936060405s) [noise=true , address=45.91.116.5:4299 , piece_id=SBE7NRD4OHRHSGDZ6JLXVXKHNXR2MHQZDNJ5JCFLI7N73S6YOLAA, seg_pos={0 0}, upload=57021]: Uploading piece... Jul 26 23:01:27.213883113 (59h25m36.936446153s) [noise=true , address=45.91.116.10:4081 , piece_id=PX4GKYJPT5IPYLSCUAPDFU5YUZQ5AUG6L7SU5EG2T5YRH6OPFBIQ, seg_pos={0 0}, upload=57021]: Uploading piece... Jul 26 23:01:27.214727529 (59h25m36.937290569s) [noise=true , address=77.90.31.21:4878 , piece_id=LN7SPTAD7D67WVOSS4CWSUTBTHZM3O5B47XJHVMRKLUSH6NSJZOQ, seg_pos={0 0}, upload=57021]: Uploading piece... Jul 26 23:01:27.218515485 (59h25m36.941078525s) [noise=true , address=45.91.116.7:4011 , piece_id=6NVHAO3YBTOAVDC5SS6ZKUMSWIYWALL6T7HJCCB5NOU3UDB2MOLA, seg_pos={0 0}, upload=57021]: Uploading piece... Jul 26 23:01:27.219396701 (59h25m36.941959741s) [noise=true , address=45.91.116.10:4083 , piece_id=BUKUHO4PYDFYZI7JVLT7TZ34PHOIZVTLFPVCVKNFE53YULYY7TMA, seg_pos={0 0}, upload=57021]: Uploading piece... Jul 26 23:01:27.219692617 (59h25m36.942255657s) [noise=true , address=77.90.31.13:4628 , piece_id=FIFB6R2VTRLXOLDRDKI52HCONBTFCCTVXDPHQ3EEKZXXUBVX7QIA, seg_pos={0 0}, upload=57021]: Uploading piece... Jul 26 23:01:27.219909352 (59h25m36.942472382s) [noise=true , address=45.91.116.12:4225 , piece_id=UVWLV3DIL25HOU6PEGNVLHB3RJAHJ2U72CQG2LFICLB6NXMZEKIA, seg_pos={0 0}, upload=57021]: Uploading piece... Jul 26 23:01:27.220107901 (59h25m36.942670941s) [noise=true , address=45.91.116.4:4940 , piece_id=7553SGZFOIK7GNACZ7VSUSGJWXSLPZGSY6K6E2JGB4GPUUBGQYUQ, seg_pos={0 0}, upload=57021]: Uploading piece... Jul 26 23:01:27.220340697 (59h25m36.942903737s) [noise=true , address=45.91.116.4:4937 , piece_id=EBF5J3XDUYMJVGFWIMYRZ6YNSUETMATVVOJIH2E3NMNWDBD6JEDQ, seg_pos={0 0}, upload=57021]: Uploading piece... Jul 26 23:01:27.220536642 (59h25m36.943099682s) [noise=true , address=45.91.116.6:4973 , piece_id=WCZZVRBQWYMSZ366CIB3ILVH4GR77KOGRSEZUFH22YMZLYYVJ64Q, seg_pos={0 0}, upload=57021]: Uploading piece... Jul 26 23:01:28.909687093 (59h25m38.632250133s) [upload=57021]: Next returned nil segment Jul 26 23:01:28.909721920 (59h25m38.63228496s) [upload=57021]: Waiting for error group managing segments... Jul 26 23:01:29.241265540 (59h25m38.96382859s) [noise=true , address=45.91.116.10:4081 , piece_id=PX4GKYJPT5IPYLSCUAPDFU5YUZQ5AUG6L7SU5EG2T5YRH6OPFBIQ, seg_pos={0 0}, upload=57021]: Done uploading piece. err: <nil> Jul 26 23:01:29.245035954 (59h25m38.967599004s) [noise=true , address=45.91.116.5:4299 , piece_id=SBE7NRD4OHRHSGDZ6JLXVXKHNXR2MHQZDNJ5JCFLI7N73S6YOLAA, seg_pos={0 0}, upload=57021]: Done uploading piece. err: <nil> Jul 26 23:01:29.248616093 (59h25m38.971179143s) [noise=true , address=45.91.116.7:4009 , piece_id=3ORNCRFYD5OI5SPXYZEMSVMZ5LDIEEK7YOUA7JERUKAKZ7QDMMKA, seg_pos={0 0}, upload=57021]: Done uploading piece. err: <nil> Jul 26 23:01:29.251100826 (59h25m38.973663886s) [noise=true , address=45.91.116.6:4973 , piece_id=WCZZVRBQWYMSZ366CIB3ILVH4GR77KOGRSEZUFH22YMZLYYVJ64Q, seg_pos={0 0}, upload=57021]: Done uploading piece. err: <nil> Jul 26 23:01:29.252327092 (59h25m38.974890132s) [noise=true , address=45.91.116.7:4011 , piece_id=6NVHAO3YBTOAVDC5SS6ZKUMSWIYWALL6T7HJCCB5NOU3UDB2MOLA, seg_pos={0 0}, upload=57021]: Done uploading piece. err: <nil> Jul 26 23:01:29.252808223 (59h25m38.975371263s) [noise=true , address=45.91.116.5:4298 , piece_id=K7332VRJDC2KTKZ3LVD3IRVGXT6HOYKD4MPER454SXOE4GFW67LQ, seg_pos={0 0}, upload=57021]: Done uploading piece. err: <nil> Jul 26 23:01:29.256185123 (59h25m38.978748173s) [noise=true , address=45.91.116.4:4940 , piece_id=7553SGZFOIK7GNACZ7VSUSGJWXSLPZGSY6K6E2JGB4GPUUBGQYUQ, seg_pos={0 0}, upload=57021]: Done uploading piece. err: <nil> Jul 26 23:01:29.257135561 (59h25m38.979698611s) [noise=true , address=45.91.116.4:4937 , piece_id=EBF5J3XDUYMJVGFWIMYRZ6YNSUETMATVVOJIH2E3NMNWDBD6JEDQ, seg_pos={0 0}, upload=57021]: Done uploading piece. err: <nil> Jul 26 23:01:29.325606113 (59h25m39.048169153s) [noise=true , address=45.91.116.8:4048 , piece_id=XPYI4RBD4CPXZDEQQDNOHSOFIGFIN5FOLGTCAEUDJ54NECUNIEAA, seg_pos={0 0}, upload=57021]: Done uploading piece. err: <nil> Jul 26 23:01:29.331150530 (59h25m39.05371356s) [noise=true , address=45.91.116.12:4225 , piece_id=UVWLV3DIL25HOU6PEGNVLHB3RJAHJ2U72CQG2LFICLB6NXMZEKIA, seg_pos={0 0}, upload=57021]: Done uploading piece. err: <nil> Jul 26 23:01:29.392913809 (59h25m39.115476879s) [noise=true , address=45.91.116.10:4083 , piece_id=BUKUHO4PYDFYZI7JVLT7TZ34PHOIZVTLFPVCVKNFE53YULYY7TMA, seg_pos={0 0}, upload=57021]: Done uploading piece. err: <nil> Jul 26 23:01:56.339456448 (59h26m6.062019488s) [noise=true , address=77.90.31.21:4878 , piece_id=LN7SPTAD7D67WVOSS4CWSUTBTHZM3O5B47XJHVMRKLUSH6NSJZOQ, seg_pos={0 0}, upload=57021]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: rpc: dial tcp 77.90.31.21:4878: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:01:56.339508978 (59h26m6.062072019s) [noise=true , address=77.90.31.13:4628 , piece_id=FIFB6R2VTRLXOLDRDKI52HCONBTFCCTVXDPHQ3EEKZXXUBVX7QIA, seg_pos={0 0}, upload=57021]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:35758->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:01:56.339539306 (59h26m6.062102346s) [noise=true , address=77.90.31.11:4555 , piece_id=GED5M25FMD6GAM42AVXKR7U7XONLKFRQPPLYVHGKCHTTCRC6AFOA, seg_pos={0 0}, upload=57021]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: rpc: dial tcp 77.90.31.11:4555: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:01:56.339599221 (59h26m6.062162261s) [noise=true , address=77.90.31.16:4734 , piece_id=ESJWP6PQ7TWJY24SAYQUGVYF7CJP6WUXAKLLH6BXHXOB6KOTQ4AQ, seg_pos={0 0}, upload=57021]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: rpc: dial tcp 77.90.31.16:4734: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:01:56.339728408 (59h26m6.062291438s) [noise=true , address=77.90.31.21:4879 , piece_id=TERNE4P5KQB5WZHFHY5WJEDQH4JNPBCHN7DPMTZCT6GOY6RMH53Q, seg_pos={0 0}, upload=57021]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: write tcp 188.166.225.170:50280->77.90.31.21:4879: use of closed network connection; context deadline exceeded Jul 26 23:01:56.339777903 (59h26m6.062340943s) [seg_pos={0 0}, upload=57021]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 26 23:01:56.572323148 (59h26m6.294886188s) [upload=57021]: Done uploading segments... Jul 26 23:01:56.572342274 (59h26m6.294905314s) [upload=57021]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.989699337s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 52 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXX.X.XX.XX.X.XXXXXXXXX...XXXXXXXXXXXX] duration: 1m0.247720241s FAIL (2025-07-26 23:06:13 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== ading segments... Jul 26 23:05:13.446403697 (59h29m23.168966737s) [upload=57076]: Got next segment. Inline: false Jul 26 23:05:13.446437281 (59h29m23.169000321s) [upload=57076]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:05:13.446441339 (59h29m23.169004379s) [upload=57076]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:05:14.025733605 (59h29m23.748296645s) [seg_pos={0 0}, upload=57076]: Begin upload segment... Jul 26 23:05:14.025788189 (59h29m23.748351229s) [seg_pos={0 0}, upload=57076]: Done begin upload segment. Jul 26 23:05:14.025847432 (59h29m23.748410482s) [noise=true , address=64.49.13.155:4923 , piece_id=WFQCL2LGV6Z3SLGWIXF4IGTX3BK4MH4I7N2TTPJFUFFLV34B3NUA, seg_pos={0 0}, upload=57076]: Uploading piece... Jul 26 23:05:14.025858584 (59h29m23.748421624s) [noise=true , address=79.127.201.214:4321 , piece_id=CCE5KZAL3AGRSXZQ6BREBLKVDMLQ5BUSYDT4A47Y25RKYRWLLUFQ, seg_pos={0 0}, upload=57076]: Uploading piece... Jul 26 23:05:14.025896035 (59h29m23.748459075s) [noise=true , address=79.127.205.235:4688 , piece_id=76DQBZD2V65KQXFFI5S2NFEVWR6TSNSEXGRSAG3JVETCHM36JNEQ, seg_pos={0 0}, upload=57076]: Uploading piece... Jul 26 23:05:14.025922426 (59h29m23.748485466s) [noise=true , address=64.49.13.138:4482 , piece_id=MGTEFFJXTONDIZDQ7EVMWMVPXPW6KRNIF3Z3E3BELBRYQ2S2T3QQ, seg_pos={0 0}, upload=57076]: Uploading piece... Jul 26 23:05:14.026197482 (59h29m23.748760522s) [noise=true , address=77.90.31.11:4578 , piece_id=Y7G7AA2VHEQR44X42J3KKX74SNHZYUTV4LFKYAAYPDVKJZ6PF3JA, seg_pos={0 0}, upload=57076]: Uploading piece... Jul 26 23:05:14.026457730 (59h29m23.74902077s) [noise=true , address=79.127.205.246:4286 , piece_id=Y32RULXBRIEXLGPCBZVXWUUBKQE4W5WC5FTIFJ3CYLCAF73VI2WQ, seg_pos={0 0}, upload=57076]: Uploading piece... Jul 26 23:05:14.026595183 (59h29m23.749158233s) [noise=true , address=45.91.116.13:4175 , piece_id=N2CPWKDUFA27MWLZOF46GL2EWUKHRDNWDQCVRCCOWJ6MIPOKEFMA, seg_pos={0 0}, upload=57076]: Uploading piece... Jul 26 23:05:14.026703951 (59h29m23.749266991s) [noise=true , address=77.90.31.21:4880 , piece_id=VZT265HOVZNSRYSWL7Q3HXCJ5UV56O5G3OVHRRNSYJ6RNMXNLEKA, seg_pos={0 0}, upload=57076]: Uploading piece... Jul 26 23:05:14.026863657 (59h29m23.749426697s) [noise=true , address=45.91.116.14:4222 , piece_id=INEZYZE3EOWBPD77SL6GLXHEKHC2IP5AUOOPIR3X445PILXBWP4A, seg_pos={0 0}, upload=57076]: Uploading piece... Jul 26 23:05:14.027112964 (59h29m23.749676004s) [noise=true , address=79.127.219.38:4006 , piece_id=HJSVN7OWZWKAIAKJ77JZ7LK3CTUF65AUIFJXZHOCPE6ARYMDQYYQ, seg_pos={0 0}, upload=57076]: Uploading piece... Jul 26 23:05:14.027325581 (59h29m23.749888621s) [noise=true , address=79.127.205.245:4236 , piece_id=PRWPTD2FXEBBHQIGC5G6VWYBOW4ECWZZ3V35GO7CRPABRSKTU76A, seg_pos={0 0}, upload=57076]: Uploading piece... Jul 26 23:05:14.027518510 (59h29m23.75008155s) [noise=true , address=79.127.201.215:4346 , piece_id=RURNL6PZXJXVOMKRWZVPAPPEKFNFE7EANO6YGS5IJLXSXJIG7HKA, seg_pos={0 0}, upload=57076]: Uploading piece... Jul 26 23:05:14.027705377 (59h29m23.750268417s) [noise=true , address=156.146.43.225:4434 , piece_id=MAMSBV6EY3PS3QSJWP5B3JVHQTLO6KR7IQ7OUJUKAYW6CWY7BVRA, seg_pos={0 0}, upload=57076]: Uploading piece... Jul 26 23:05:14.027919748 (59h29m23.750482788s) [noise=true , address=77.90.31.11:4573 , piece_id=3RE7Y4L5ZCET4JANCBK65MSIVMF7LONIB2UDHPYNEH74ULISL5HA, seg_pos={0 0}, upload=57076]: Uploading piece... Jul 26 23:05:14.028117526 (59h29m23.750680556s) [noise=true , address=79.127.205.246:4275 , piece_id=QW2HQUAUEGKMFWSWOKETFHZTNUEISLVXEGUZ3UWALOWO2BEHSO6A, seg_pos={0 0}, upload=57076]: Uploading piece... Jul 26 23:05:14.028401559 (59h29m23.750964599s) [noise=true , address=45.91.116.14:4206 , piece_id=YRPOA6QCCZZ4KCPGDGGWZOKLFXEV6TU6IWFLGHJKJQOASGOTI4YA, seg_pos={0 0}, upload=57076]: Uploading piece... Jul 26 23:05:14.028637461 (59h29m23.751200501s) [noise=true , address=77.90.31.13:4648 , piece_id=5EEHOLABMDA4BEQT6XLC7OXEVGYQCCNDT5Y4GRQOHT35X4JQLWLA, seg_pos={0 0}, upload=57076]: Uploading piece... Jul 26 23:05:14.028694550 (59h29m23.75125759s) [noise=true , address=79.127.219.38:4004 , piece_id=QJIKCD3LYRVIONMIOM2OWOK2U2WLLMSF5ERREVKAEQAQCLPFM2LQ, seg_pos={0 0}, upload=57076]: Uploading piece... Jul 26 23:05:15.049266579 (59h29m24.771829619s) [noise=true , address=156.146.43.225:4434 , piece_id=MAMSBV6EY3PS3QSJWP5B3JVHQTLO6KR7IQ7OUJUKAYW6CWY7BVRA, seg_pos={0 0}, upload=57076]: Done uploading piece. err: ecclient: upload failed (node:12WJ1b6sKJG4NcqmNnvSLVw1e8Us6phnN5UpJY2tvo38pdmhFLp, address:156.146.43.225:4434): protocol: storage node overloaded, request limit: 2000 Jul 26 23:05:15.115739954 (59h29m24.838302994s) [upload=57076]: Next returned nil segment Jul 26 23:05:15.115761394 (59h29m24.838324434s) [upload=57076]: Waiting for error group managing segments... Jul 26 23:05:15.352336024 (59h29m25.074899064s) [noise=true , address=79.127.201.214:4321 , piece_id=CCE5KZAL3AGRSXZQ6BREBLKVDMLQ5BUSYDT4A47Y25RKYRWLLUFQ, seg_pos={0 0}, upload=57076]: Done uploading piece. err: ecclient: upload failed (node:127EgxjVRhAGcz1DJjE539rWnmDjsVTRFQc1esEZUyQ1yxGiCD5, address:79.127.201.214:4321): protocol: storage node overloaded, request limit: 2000 Jul 26 23:05:15.406702765 (59h29m25.129265805s) [noise=true , address=79.127.205.246:4286 , piece_id=Y32RULXBRIEXLGPCBZVXWUUBKQE4W5WC5FTIFJ3CYLCAF73VI2WQ, seg_pos={0 0}, upload=57076]: Done uploading piece. err: <nil> Jul 26 23:05:15.425199355 (59h29m25.147762385s) [noise=true , address=79.127.205.245:4236 , piece_id=PRWPTD2FXEBBHQIGC5G6VWYBOW4ECWZZ3V35GO7CRPABRSKTU76A, seg_pos={0 0}, upload=57076]: Done uploading piece. err: <nil> Jul 26 23:05:15.425751250 (59h29m25.148314291s) [noise=true , address=79.127.205.235:4688 , piece_id=76DQBZD2V65KQXFFI5S2NFEVWR6TSNSEXGRSAG3JVETCHM36JNEQ, seg_pos={0 0}, upload=57076]: Done uploading piece. err: <nil> Jul 26 23:05:15.443845311 (59h29m25.166408351s) [noise=true , address=79.127.205.246:4275 , piece_id=QW2HQUAUEGKMFWSWOKETFHZTNUEISLVXEGUZ3UWALOWO2BEHSO6A, seg_pos={0 0}, upload=57076]: Done uploading piece. err: <nil> Jul 26 23:05:15.925710917 (59h29m25.648273947s) [noise=true , address=79.127.219.38:4006 , piece_id=HJSVN7OWZWKAIAKJ77JZ7LK3CTUF65AUIFJXZHOCPE6ARYMDQYYQ, seg_pos={0 0}, upload=57076]: Done uploading piece. err: <nil> Jul 26 23:05:16.022692063 (59h29m25.745255134s) [noise=true , address=45.91.116.13:4175 , piece_id=N2CPWKDUFA27MWLZOF46GL2EWUKHRDNWDQCVRCCOWJ6MIPOKEFMA, seg_pos={0 0}, upload=57076]: Done uploading piece. err: <nil> Jul 26 23:05:16.025696251 (59h29m25.748259291s) [noise=true , address=79.127.219.38:4004 , piece_id=QJIKCD3LYRVIONMIOM2OWOK2U2WLLMSF5ERREVKAEQAQCLPFM2LQ, seg_pos={0 0}, upload=57076]: Done uploading piece. err: <nil> Jul 26 23:05:16.885942399 (59h29m26.608505439s) [noise=true , address=45.91.116.14:4222 , piece_id=INEZYZE3EOWBPD77SL6GLXHEKHC2IP5AUOOPIR3X445PILXBWP4A, seg_pos={0 0}, upload=57076]: Done uploading piece. err: <nil> Jul 26 23:05:17.106452413 (59h29m26.829015453s) [noise=true , address=64.49.13.138:4482 , piece_id=MGTEFFJXTONDIZDQ7EVMWMVPXPW6KRNIF3Z3E3BELBRYQ2S2T3QQ, seg_pos={0 0}, upload=57076]: Done uploading piece. err: <nil> Jul 26 23:05:20.293580567 (59h29m30.016143607s) [noise=true , address=45.91.116.14:4206 , piece_id=YRPOA6QCCZZ4KCPGDGGWZOKLFXEV6TU6IWFLGHJKJQOASGOTI4YA, seg_pos={0 0}, upload=57076]: Done uploading piece. err: <nil> Jul 26 23:05:22.927982715 (59h29m32.650545755s) [noise=true , address=64.49.13.155:4923 , piece_id=WFQCL2LGV6Z3SLGWIXF4IGTX3BK4MH4I7N2TTPJFUFFLV34B3NUA, seg_pos={0 0}, upload=57076]: Done uploading piece. err: <nil> Jul 26 23:06:00.742732230 (59h30m10.46529527s) [noise=true , address=77.90.31.21:4880 , piece_id=VZT265HOVZNSRYSWL7Q3HXCJ5UV56O5G3OVHRRNSYJ6RNMXNLEKA, seg_pos={0 0}, upload=57076]: Done uploading piece. err: <nil> Jul 26 23:06:12.783101880 (59h30m22.50566492s) [noise=true , address=77.90.31.11:4578 , piece_id=Y7G7AA2VHEQR44X42J3KKX74SNHZYUTV4LFKYAAYPDVKJZ6PF3JA, seg_pos={0 0}, upload=57076]: Done uploading piece. err: ecclient: upload failed (node:1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, address:77.90.31.11:4578): protocol: write tcp 188.166.225.170:58220->77.90.31.11:4578: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:06:12.783210969 (59h30m22.505774009s) [noise=true , address=79.127.201.215:4346 , piece_id=RURNL6PZXJXVOMKRWZVPAPPEKFNFE7EANO6YGS5IJLXSXJIG7HKA, seg_pos={0 0}, upload=57076]: Done uploading piece. err: ecclient: upload failed (node:1233Rr1BSqzZQFufL1FJor8kQyMNY22A9kpcKwvWe6SMniRBYAT, address:79.127.201.215:4346): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:06:12.783281264 (59h30m22.505844304s) [noise=true , address=77.90.31.13:4648 , piece_id=5EEHOLABMDA4BEQT6XLC7OXEVGYQCCNDT5Y4GRQOHT35X4JQLWLA, seg_pos={0 0}, upload=57076]: Done uploading piece. err: ecclient: upload failed (node:1niTsTaHjA6LoMMrkEsSnLcRDhZtVAjxsbXUXVeXzhUp3FPbWh, address:77.90.31.13:4648): protocol: rpc: dial tcp 77.90.31.13:4648: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:06:12.783309648 (59h30m22.505872688s) [noise=true , address=77.90.31.11:4573 , piece_id=3RE7Y4L5ZCET4JANCBK65MSIVMF7LONIB2UDHPYNEH74ULISL5HA, seg_pos={0 0}, upload=57076]: Done uploading piece. err: ecclient: upload failed (node:1Q3atbNPC93bXM7pr4KCeea5La9sX4xs2qNwGxNWSrNQs1ftrV, address:77.90.31.11:4573): protocol: write tcp 188.166.225.170:34248->77.90.31.11:4573: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:06:12.783337251 (59h30m22.505900291s) [seg_pos={0 0}, upload=57076]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 26 23:06:13.028947308 (59h30m22.751510348s) [upload=57076]: Done uploading segments... Jul 26 23:06:13.028972226 (59h30m22.751535266s) [upload=57076]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1m0.247741421s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............................X.........X...........X......] duration: 6.197647297s FAIL (2025-07-26 23:05:51 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== UPht7vBRdA3BcqRek): protocol: rpc: dial tcp 77.90.31.11:4554: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:00:36.538079013 (59h24m46.260642053s) [seg_pos={0 0}, upload=57008]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 23:00:36.538172713 (59h24m46.260735753s) [upload=57008]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 23:00:36.538178714 (59h24m46.260741784s) [upload=57008]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 23:00:36.813206693 (59h24m46.535769733s) [upload=57008]: Done uploading segments... Jul 26 23:00:36.813224688 (59h24m46.535787728s) [upload=57008]: Upload finished. err: <nil> Jul 26 23:01:26.980982718 (59h25m36.703545748s) [upload=57021]: Starting upload Jul 26 23:01:26.981034367 (59h25m36.703597407s) [upload=57021]: Done starting upload Jul 26 23:01:26.981136591 (59h25m36.703699632s) [upload=57021]: Uploading segments... Jul 26 23:01:26.981370199 (59h25m36.703933229s) [upload=57021]: Got next segment. Inline: false Jul 26 23:01:26.981405947 (59h25m36.703968987s) [upload=57021]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:01:26.981412309 (59h25m36.703975349s) [upload=57021]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:01:27.212088890 (59h25m36.93465193s) [seg_pos={0 0}, upload=57021]: Begin upload segment... Jul 26 23:01:27.212149156 (59h25m36.934712196s) [seg_pos={0 0}, upload=57021]: Done begin upload segment. Jul 26 23:01:27.212191998 (59h25m36.934755038s) [noise=true , address=45.91.116.8:4048 , piece_id=XPYI4RBD4CPXZDEQQDNOHSOFIGFIN5FOLGTCAEUDJ54NECUNIEAA, seg_pos={0 0}, upload=57021]: Uploading piece... Jul 26 23:01:27.212264527 (59h25m36.934827577s) [noise=true , address=45.91.116.5:4298 , piece_id=K7332VRJDC2KTKZ3LVD3IRVGXT6HOYKD4MPER454SXOE4GFW67LQ, seg_pos={0 0}, upload=57021]: Uploading piece... Jul 26 23:01:27.212718034 (59h25m36.935281074s) [noise=true , address=77.90.31.11:4555 , piece_id=GED5M25FMD6GAM42AVXKR7U7XONLKFRQPPLYVHGKCHTTCRC6AFOA, seg_pos={0 0}, upload=57021]: Uploading piece... Jul 26 23:01:27.212787557 (59h25m36.935350588s) [noise=true , address=45.91.116.7:4009 , piece_id=3ORNCRFYD5OI5SPXYZEMSVMZ5LDIEEK7YOUA7JERUKAKZ7QDMMKA, seg_pos={0 0}, upload=57021]: Uploading piece... Jul 26 23:01:27.213064708 (59h25m36.935627748s) [noise=true , address=77.90.31.16:4734 , piece_id=ESJWP6PQ7TWJY24SAYQUGVYF7CJP6WUXAKLLH6BXHXOB6KOTQ4AQ, seg_pos={0 0}, upload=57021]: Uploading piece... Jul 26 23:01:27.213128920 (59h25m36.93569196s) [noise=true , address=77.90.31.21:4879 , piece_id=TERNE4P5KQB5WZHFHY5WJEDQH4JNPBCHN7DPMTZCT6GOY6RMH53Q, seg_pos={0 0}, upload=57021]: Uploading piece... Jul 26 23:01:27.213497365 (59h25m36.936060405s) [noise=true , address=45.91.116.5:4299 , piece_id=SBE7NRD4OHRHSGDZ6JLXVXKHNXR2MHQZDNJ5JCFLI7N73S6YOLAA, seg_pos={0 0}, upload=57021]: Uploading piece... Jul 26 23:01:27.213883113 (59h25m36.936446153s) [noise=true , address=45.91.116.10:4081 , piece_id=PX4GKYJPT5IPYLSCUAPDFU5YUZQ5AUG6L7SU5EG2T5YRH6OPFBIQ, seg_pos={0 0}, upload=57021]: Uploading piece... Jul 26 23:01:27.214727529 (59h25m36.937290569s) [noise=true , address=77.90.31.21:4878 , piece_id=LN7SPTAD7D67WVOSS4CWSUTBTHZM3O5B47XJHVMRKLUSH6NSJZOQ, seg_pos={0 0}, upload=57021]: Uploading piece... Jul 26 23:01:27.218515485 (59h25m36.941078525s) [noise=true , address=45.91.116.7:4011 , piece_id=6NVHAO3YBTOAVDC5SS6ZKUMSWIYWALL6T7HJCCB5NOU3UDB2MOLA, seg_pos={0 0}, upload=57021]: Uploading piece... Jul 26 23:01:27.219396701 (59h25m36.941959741s) [noise=true , address=45.91.116.10:4083 , piece_id=BUKUHO4PYDFYZI7JVLT7TZ34PHOIZVTLFPVCVKNFE53YULYY7TMA, seg_pos={0 0}, upload=57021]: Uploading piece... Jul 26 23:01:27.219692617 (59h25m36.942255657s) [noise=true , address=77.90.31.13:4628 , piece_id=FIFB6R2VTRLXOLDRDKI52HCONBTFCCTVXDPHQ3EEKZXXUBVX7QIA, seg_pos={0 0}, upload=57021]: Uploading piece... Jul 26 23:01:27.219909352 (59h25m36.942472382s) [noise=true , address=45.91.116.12:4225 , piece_id=UVWLV3DIL25HOU6PEGNVLHB3RJAHJ2U72CQG2LFICLB6NXMZEKIA, seg_pos={0 0}, upload=57021]: Uploading piece... Jul 26 23:01:27.220107901 (59h25m36.942670941s) [noise=true , address=45.91.116.4:4940 , piece_id=7553SGZFOIK7GNACZ7VSUSGJWXSLPZGSY6K6E2JGB4GPUUBGQYUQ, seg_pos={0 0}, upload=57021]: Uploading piece... Jul 26 23:01:27.220340697 (59h25m36.942903737s) [noise=true , address=45.91.116.4:4937 , piece_id=EBF5J3XDUYMJVGFWIMYRZ6YNSUETMATVVOJIH2E3NMNWDBD6JEDQ, seg_pos={0 0}, upload=57021]: Uploading piece... Jul 26 23:01:27.220536642 (59h25m36.943099682s) [noise=true , address=45.91.116.6:4973 , piece_id=WCZZVRBQWYMSZ366CIB3ILVH4GR77KOGRSEZUFH22YMZLYYVJ64Q, seg_pos={0 0}, upload=57021]: Uploading piece... Jul 26 23:01:28.909687093 (59h25m38.632250133s) [upload=57021]: Next returned nil segment Jul 26 23:01:28.909721920 (59h25m38.63228496s) [upload=57021]: Waiting for error group managing segments... Jul 26 23:01:29.241265540 (59h25m38.96382859s) [noise=true , address=45.91.116.10:4081 , piece_id=PX4GKYJPT5IPYLSCUAPDFU5YUZQ5AUG6L7SU5EG2T5YRH6OPFBIQ, seg_pos={0 0}, upload=57021]: Done uploading piece. err: <nil> Jul 26 23:01:29.245035954 (59h25m38.967599004s) [noise=true , address=45.91.116.5:4299 , piece_id=SBE7NRD4OHRHSGDZ6JLXVXKHNXR2MHQZDNJ5JCFLI7N73S6YOLAA, seg_pos={0 0}, upload=57021]: Done uploading piece. err: <nil> Jul 26 23:01:29.248616093 (59h25m38.971179143s) [noise=true , address=45.91.116.7:4009 , piece_id=3ORNCRFYD5OI5SPXYZEMSVMZ5LDIEEK7YOUA7JERUKAKZ7QDMMKA, seg_pos={0 0}, upload=57021]: Done uploading piece. err: <nil> Jul 26 23:01:29.251100826 (59h25m38.973663886s) [noise=true , address=45.91.116.6:4973 , piece_id=WCZZVRBQWYMSZ366CIB3ILVH4GR77KOGRSEZUFH22YMZLYYVJ64Q, seg_pos={0 0}, upload=57021]: Done uploading piece. err: <nil> Jul 26 23:01:29.252327092 (59h25m38.974890132s) [noise=true , address=45.91.116.7:4011 , piece_id=6NVHAO3YBTOAVDC5SS6ZKUMSWIYWALL6T7HJCCB5NOU3UDB2MOLA, seg_pos={0 0}, upload=57021]: Done uploading piece. err: <nil> Jul 26 23:01:29.252808223 (59h25m38.975371263s) [noise=true , address=45.91.116.5:4298 , piece_id=K7332VRJDC2KTKZ3LVD3IRVGXT6HOYKD4MPER454SXOE4GFW67LQ, seg_pos={0 0}, upload=57021]: Done uploading piece. err: <nil> Jul 26 23:01:29.256185123 (59h25m38.978748173s) [noise=true , address=45.91.116.4:4940 , piece_id=7553SGZFOIK7GNACZ7VSUSGJWXSLPZGSY6K6E2JGB4GPUUBGQYUQ, seg_pos={0 0}, upload=57021]: Done uploading piece. err: <nil> Jul 26 23:01:29.257135561 (59h25m38.979698611s) [noise=true , address=45.91.116.4:4937 , piece_id=EBF5J3XDUYMJVGFWIMYRZ6YNSUETMATVVOJIH2E3NMNWDBD6JEDQ, seg_pos={0 0}, upload=57021]: Done uploading piece. err: <nil> Jul 26 23:01:29.325606113 (59h25m39.048169153s) [noise=true , address=45.91.116.8:4048 , piece_id=XPYI4RBD4CPXZDEQQDNOHSOFIGFIN5FOLGTCAEUDJ54NECUNIEAA, seg_pos={0 0}, upload=57021]: Done uploading piece. err: <nil> Jul 26 23:01:29.331150530 (59h25m39.05371356s) [noise=true , address=45.91.116.12:4225 , piece_id=UVWLV3DIL25HOU6PEGNVLHB3RJAHJ2U72CQG2LFICLB6NXMZEKIA, seg_pos={0 0}, upload=57021]: Done uploading piece. err: <nil> Jul 26 23:01:29.392913809 (59h25m39.115476879s) [noise=true , address=45.91.116.10:4083 , piece_id=BUKUHO4PYDFYZI7JVLT7TZ34PHOIZVTLFPVCVKNFE53YULYY7TMA, seg_pos={0 0}, upload=57021]: Done uploading piece. err: <nil> Jul 26 23:01:56.339456448 (59h26m6.062019488s) [noise=true , address=77.90.31.21:4878 , piece_id=LN7SPTAD7D67WVOSS4CWSUTBTHZM3O5B47XJHVMRKLUSH6NSJZOQ, seg_pos={0 0}, upload=57021]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: rpc: dial tcp 77.90.31.21:4878: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:01:56.339508978 (59h26m6.062072019s) [noise=true , address=77.90.31.13:4628 , piece_id=FIFB6R2VTRLXOLDRDKI52HCONBTFCCTVXDPHQ3EEKZXXUBVX7QIA, seg_pos={0 0}, upload=57021]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:35758->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:01:56.339539306 (59h26m6.062102346s) [noise=true , address=77.90.31.11:4555 , piece_id=GED5M25FMD6GAM42AVXKR7U7XONLKFRQPPLYVHGKCHTTCRC6AFOA, seg_pos={0 0}, upload=57021]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: rpc: dial tcp 77.90.31.11:4555: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:01:56.339599221 (59h26m6.062162261s) [noise=true , address=77.90.31.16:4734 , piece_id=ESJWP6PQ7TWJY24SAYQUGVYF7CJP6WUXAKLLH6BXHXOB6KOTQ4AQ, seg_pos={0 0}, upload=57021]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: rpc: dial tcp 77.90.31.16:4734: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:01:56.339728408 (59h26m6.062291438s) [noise=true , address=77.90.31.21:4879 , piece_id=TERNE4P5KQB5WZHFHY5WJEDQH4JNPBCHN7DPMTZCT6GOY6RMH53Q, seg_pos={0 0}, upload=57021]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: write tcp 188.166.225.170:50280->77.90.31.21:4879: use of closed network connection; context deadline exceeded Jul 26 23:01:56.339777903 (59h26m6.062340943s) [seg_pos={0 0}, upload=57021]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 26 23:01:56.572323148 (59h26m6.294886188s) [upload=57021]: Done uploading segments... Jul 26 23:01:56.572342274 (59h26m6.294905314s) [upload=57021]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.197676613s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [........X..........X.X........X.X...........................] duration: 5.880433259s FAIL (2025-07-26 23:05:31 UTC) orbiter warning: uplink: too many requests Duration: 5.880453899s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [................................X.........X...........X.....] duration: 6.506969246s FAIL (2025-07-26 23:05:14 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== UPht7vBRdA3BcqRek): protocol: rpc: dial tcp 77.90.31.11:4554: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:00:36.538079013 (59h24m46.260642053s) [seg_pos={0 0}, upload=57008]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 23:00:36.538172713 (59h24m46.260735753s) [upload=57008]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 23:00:36.538178714 (59h24m46.260741784s) [upload=57008]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 23:00:36.813206693 (59h24m46.535769733s) [upload=57008]: Done uploading segments... Jul 26 23:00:36.813224688 (59h24m46.535787728s) [upload=57008]: Upload finished. err: <nil> Jul 26 23:01:26.980982718 (59h25m36.703545748s) [upload=57021]: Starting upload Jul 26 23:01:26.981034367 (59h25m36.703597407s) [upload=57021]: Done starting upload Jul 26 23:01:26.981136591 (59h25m36.703699632s) [upload=57021]: Uploading segments... Jul 26 23:01:26.981370199 (59h25m36.703933229s) [upload=57021]: Got next segment. Inline: false Jul 26 23:01:26.981405947 (59h25m36.703968987s) [upload=57021]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:01:26.981412309 (59h25m36.703975349s) [upload=57021]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:01:27.212088890 (59h25m36.93465193s) [seg_pos={0 0}, upload=57021]: Begin upload segment... Jul 26 23:01:27.212149156 (59h25m36.934712196s) [seg_pos={0 0}, upload=57021]: Done begin upload segment. Jul 26 23:01:27.212191998 (59h25m36.934755038s) [noise=true , address=45.91.116.8:4048 , piece_id=XPYI4RBD4CPXZDEQQDNOHSOFIGFIN5FOLGTCAEUDJ54NECUNIEAA, seg_pos={0 0}, upload=57021]: Uploading piece... Jul 26 23:01:27.212264527 (59h25m36.934827577s) [noise=true , address=45.91.116.5:4298 , piece_id=K7332VRJDC2KTKZ3LVD3IRVGXT6HOYKD4MPER454SXOE4GFW67LQ, seg_pos={0 0}, upload=57021]: Uploading piece... Jul 26 23:01:27.212718034 (59h25m36.935281074s) [noise=true , address=77.90.31.11:4555 , piece_id=GED5M25FMD6GAM42AVXKR7U7XONLKFRQPPLYVHGKCHTTCRC6AFOA, seg_pos={0 0}, upload=57021]: Uploading piece... Jul 26 23:01:27.212787557 (59h25m36.935350588s) [noise=true , address=45.91.116.7:4009 , piece_id=3ORNCRFYD5OI5SPXYZEMSVMZ5LDIEEK7YOUA7JERUKAKZ7QDMMKA, seg_pos={0 0}, upload=57021]: Uploading piece... Jul 26 23:01:27.213064708 (59h25m36.935627748s) [noise=true , address=77.90.31.16:4734 , piece_id=ESJWP6PQ7TWJY24SAYQUGVYF7CJP6WUXAKLLH6BXHXOB6KOTQ4AQ, seg_pos={0 0}, upload=57021]: Uploading piece... Jul 26 23:01:27.213128920 (59h25m36.93569196s) [noise=true , address=77.90.31.21:4879 , piece_id=TERNE4P5KQB5WZHFHY5WJEDQH4JNPBCHN7DPMTZCT6GOY6RMH53Q, seg_pos={0 0}, upload=57021]: Uploading piece... Jul 26 23:01:27.213497365 (59h25m36.936060405s) [noise=true , address=45.91.116.5:4299 , piece_id=SBE7NRD4OHRHSGDZ6JLXVXKHNXR2MHQZDNJ5JCFLI7N73S6YOLAA, seg_pos={0 0}, upload=57021]: Uploading piece... Jul 26 23:01:27.213883113 (59h25m36.936446153s) [noise=true , address=45.91.116.10:4081 , piece_id=PX4GKYJPT5IPYLSCUAPDFU5YUZQ5AUG6L7SU5EG2T5YRH6OPFBIQ, seg_pos={0 0}, upload=57021]: Uploading piece... Jul 26 23:01:27.214727529 (59h25m36.937290569s) [noise=true , address=77.90.31.21:4878 , piece_id=LN7SPTAD7D67WVOSS4CWSUTBTHZM3O5B47XJHVMRKLUSH6NSJZOQ, seg_pos={0 0}, upload=57021]: Uploading piece... Jul 26 23:01:27.218515485 (59h25m36.941078525s) [noise=true , address=45.91.116.7:4011 , piece_id=6NVHAO3YBTOAVDC5SS6ZKUMSWIYWALL6T7HJCCB5NOU3UDB2MOLA, seg_pos={0 0}, upload=57021]: Uploading piece... Jul 26 23:01:27.219396701 (59h25m36.941959741s) [noise=true , address=45.91.116.10:4083 , piece_id=BUKUHO4PYDFYZI7JVLT7TZ34PHOIZVTLFPVCVKNFE53YULYY7TMA, seg_pos={0 0}, upload=57021]: Uploading piece... Jul 26 23:01:27.219692617 (59h25m36.942255657s) [noise=true , address=77.90.31.13:4628 , piece_id=FIFB6R2VTRLXOLDRDKI52HCONBTFCCTVXDPHQ3EEKZXXUBVX7QIA, seg_pos={0 0}, upload=57021]: Uploading piece... Jul 26 23:01:27.219909352 (59h25m36.942472382s) [noise=true , address=45.91.116.12:4225 , piece_id=UVWLV3DIL25HOU6PEGNVLHB3RJAHJ2U72CQG2LFICLB6NXMZEKIA, seg_pos={0 0}, upload=57021]: Uploading piece... Jul 26 23:01:27.220107901 (59h25m36.942670941s) [noise=true , address=45.91.116.4:4940 , piece_id=7553SGZFOIK7GNACZ7VSUSGJWXSLPZGSY6K6E2JGB4GPUUBGQYUQ, seg_pos={0 0}, upload=57021]: Uploading piece... Jul 26 23:01:27.220340697 (59h25m36.942903737s) [noise=true , address=45.91.116.4:4937 , piece_id=EBF5J3XDUYMJVGFWIMYRZ6YNSUETMATVVOJIH2E3NMNWDBD6JEDQ, seg_pos={0 0}, upload=57021]: Uploading piece... Jul 26 23:01:27.220536642 (59h25m36.943099682s) [noise=true , address=45.91.116.6:4973 , piece_id=WCZZVRBQWYMSZ366CIB3ILVH4GR77KOGRSEZUFH22YMZLYYVJ64Q, seg_pos={0 0}, upload=57021]: Uploading piece... Jul 26 23:01:28.909687093 (59h25m38.632250133s) [upload=57021]: Next returned nil segment Jul 26 23:01:28.909721920 (59h25m38.63228496s) [upload=57021]: Waiting for error group managing segments... Jul 26 23:01:29.241265540 (59h25m38.96382859s) [noise=true , address=45.91.116.10:4081 , piece_id=PX4GKYJPT5IPYLSCUAPDFU5YUZQ5AUG6L7SU5EG2T5YRH6OPFBIQ, seg_pos={0 0}, upload=57021]: Done uploading piece. err: <nil> Jul 26 23:01:29.245035954 (59h25m38.967599004s) [noise=true , address=45.91.116.5:4299 , piece_id=SBE7NRD4OHRHSGDZ6JLXVXKHNXR2MHQZDNJ5JCFLI7N73S6YOLAA, seg_pos={0 0}, upload=57021]: Done uploading piece. err: <nil> Jul 26 23:01:29.248616093 (59h25m38.971179143s) [noise=true , address=45.91.116.7:4009 , piece_id=3ORNCRFYD5OI5SPXYZEMSVMZ5LDIEEK7YOUA7JERUKAKZ7QDMMKA, seg_pos={0 0}, upload=57021]: Done uploading piece. err: <nil> Jul 26 23:01:29.251100826 (59h25m38.973663886s) [noise=true , address=45.91.116.6:4973 , piece_id=WCZZVRBQWYMSZ366CIB3ILVH4GR77KOGRSEZUFH22YMZLYYVJ64Q, seg_pos={0 0}, upload=57021]: Done uploading piece. err: <nil> Jul 26 23:01:29.252327092 (59h25m38.974890132s) [noise=true , address=45.91.116.7:4011 , piece_id=6NVHAO3YBTOAVDC5SS6ZKUMSWIYWALL6T7HJCCB5NOU3UDB2MOLA, seg_pos={0 0}, upload=57021]: Done uploading piece. err: <nil> Jul 26 23:01:29.252808223 (59h25m38.975371263s) [noise=true , address=45.91.116.5:4298 , piece_id=K7332VRJDC2KTKZ3LVD3IRVGXT6HOYKD4MPER454SXOE4GFW67LQ, seg_pos={0 0}, upload=57021]: Done uploading piece. err: <nil> Jul 26 23:01:29.256185123 (59h25m38.978748173s) [noise=true , address=45.91.116.4:4940 , piece_id=7553SGZFOIK7GNACZ7VSUSGJWXSLPZGSY6K6E2JGB4GPUUBGQYUQ, seg_pos={0 0}, upload=57021]: Done uploading piece. err: <nil> Jul 26 23:01:29.257135561 (59h25m38.979698611s) [noise=true , address=45.91.116.4:4937 , piece_id=EBF5J3XDUYMJVGFWIMYRZ6YNSUETMATVVOJIH2E3NMNWDBD6JEDQ, seg_pos={0 0}, upload=57021]: Done uploading piece. err: <nil> Jul 26 23:01:29.325606113 (59h25m39.048169153s) [noise=true , address=45.91.116.8:4048 , piece_id=XPYI4RBD4CPXZDEQQDNOHSOFIGFIN5FOLGTCAEUDJ54NECUNIEAA, seg_pos={0 0}, upload=57021]: Done uploading piece. err: <nil> Jul 26 23:01:29.331150530 (59h25m39.05371356s) [noise=true , address=45.91.116.12:4225 , piece_id=UVWLV3DIL25HOU6PEGNVLHB3RJAHJ2U72CQG2LFICLB6NXMZEKIA, seg_pos={0 0}, upload=57021]: Done uploading piece. err: <nil> Jul 26 23:01:29.392913809 (59h25m39.115476879s) [noise=true , address=45.91.116.10:4083 , piece_id=BUKUHO4PYDFYZI7JVLT7TZ34PHOIZVTLFPVCVKNFE53YULYY7TMA, seg_pos={0 0}, upload=57021]: Done uploading piece. err: <nil> Jul 26 23:01:56.339456448 (59h26m6.062019488s) [noise=true , address=77.90.31.21:4878 , piece_id=LN7SPTAD7D67WVOSS4CWSUTBTHZM3O5B47XJHVMRKLUSH6NSJZOQ, seg_pos={0 0}, upload=57021]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: rpc: dial tcp 77.90.31.21:4878: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:01:56.339508978 (59h26m6.062072019s) [noise=true , address=77.90.31.13:4628 , piece_id=FIFB6R2VTRLXOLDRDKI52HCONBTFCCTVXDPHQ3EEKZXXUBVX7QIA, seg_pos={0 0}, upload=57021]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:35758->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:01:56.339539306 (59h26m6.062102346s) [noise=true , address=77.90.31.11:4555 , piece_id=GED5M25FMD6GAM42AVXKR7U7XONLKFRQPPLYVHGKCHTTCRC6AFOA, seg_pos={0 0}, upload=57021]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: rpc: dial tcp 77.90.31.11:4555: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:01:56.339599221 (59h26m6.062162261s) [noise=true , address=77.90.31.16:4734 , piece_id=ESJWP6PQ7TWJY24SAYQUGVYF7CJP6WUXAKLLH6BXHXOB6KOTQ4AQ, seg_pos={0 0}, upload=57021]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: rpc: dial tcp 77.90.31.16:4734: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:01:56.339728408 (59h26m6.062291438s) [noise=true , address=77.90.31.21:4879 , piece_id=TERNE4P5KQB5WZHFHY5WJEDQH4JNPBCHN7DPMTZCT6GOY6RMH53Q, seg_pos={0 0}, upload=57021]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: write tcp 188.166.225.170:50280->77.90.31.21:4879: use of closed network connection; context deadline exceeded Jul 26 23:01:56.339777903 (59h26m6.062340943s) [seg_pos={0 0}, upload=57021]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 26 23:01:56.572323148 (59h26m6.294886188s) [upload=57021]: Done uploading segments... Jul 26 23:01:56.572342274 (59h26m6.294905314s) [upload=57021]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.506993983s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 52 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXX.X.XX.XX.X.XXXXXXXXX...XXXXXXXXXXX] duration: 1m0.235446336s FAIL (2025-07-26 23:05:05 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== es (needed at least 14 but got 10) Jul 26 23:04:06.362285842 (59h28m16.084848913s) [upload=57060]: Starting upload Jul 26 23:04:06.362442332 (59h28m16.085005372s) [upload=57060]: Done starting upload Jul 26 23:04:06.362532765 (59h28m16.085095845s) [upload=57060]: Uploading segments... Jul 26 23:04:06.362892814 (59h28m16.085455854s) [upload=57060]: Got next segment. Inline: false Jul 26 23:04:06.362974580 (59h28m16.08553762s) [upload=57060]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:04:06.362983076 (59h28m16.085546146s) [upload=57060]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:04:06.806938350 (59h28m16.52950139s) [seg_pos={0 0}, upload=57060]: Begin upload segment... Jul 26 23:04:06.806999918 (59h28m16.529562948s) [seg_pos={0 0}, upload=57060]: Done begin upload segment. Jul 26 23:04:06.807026559 (59h28m16.529589599s) [noise=true , address=79.127.220.97:4361 , piece_id=SEIM7SPP6ZFV4327HPIQIW7NTEKC6VHFQOSJNNMQKUSRAYTPBANQ, seg_pos={0 0}, upload=57060]: Uploading piece... Jul 26 23:04:06.807417507 (59h28m16.529980537s) [noise=true , address=79.127.219.47:4061 , piece_id=SL6FSCMAX4CSBR3QEHLAJM5XVNFOFCGU6NWG6UERMTXE327MXVCQ, seg_pos={0 0}, upload=57060]: Uploading piece... Jul 26 23:04:06.807520514 (59h28m16.530083554s) [noise=true , address=77.90.31.19:4875 , piece_id=JCSE64XIUP5SNEOSQOEMYVNBXOM6GOWBFDKN544FINLRARR6OILA, seg_pos={0 0}, upload=57060]: Uploading piece... Jul 26 23:04:06.807869742 (59h28m16.530432782s) [noise=true , address=77.90.31.19:4850 , piece_id=Q3STRX3WDECFMF3X2ZYTFYXMTHK736HTI45QJCPBBQN6AO7TQAQA, seg_pos={0 0}, upload=57060]: Uploading piece... Jul 26 23:04:06.807949404 (59h28m16.530512434s) [noise=true , address=77.90.31.12:4601 , piece_id=L7VLEU4J3K4XGUIHPKU77U3MM7E4XUDHPNZNPQO7M7UTM7TUBYRA, seg_pos={0 0}, upload=57060]: Uploading piece... Jul 26 23:04:06.808109351 (59h28m16.530672391s) [noise=true , address=79.127.201.215:4348 , piece_id=U6QH4RGLE6SJI6KGZ46OGWZEDDZ3WBPTJP35CFABBYXF4TPNNJIQ, seg_pos={0 0}, upload=57060]: Uploading piece... Jul 26 23:04:06.808326025 (59h28m16.530889075s) [noise=true , address=79.127.205.245:4247 , piece_id=4NHIH6EMTFNHMTBJVHB6S3YBXALY726STCICQLIYPAZUQW7NWMWQ, seg_pos={0 0}, upload=57060]: Uploading piece... Jul 26 23:04:06.808412761 (59h28m16.530975791s) [noise=true , address=79.127.219.51:4210 , piece_id=U2VV733BQ5OST2ZXUZVS4CXBA74KQ56WE4MP5UBVEZY2AFDT6RWQ, seg_pos={0 0}, upload=57060]: Uploading piece... Jul 26 23:04:06.808702987 (59h28m16.531266027s) [noise=true , address=156.146.43.225:4449 , piece_id=IXR2VSJ5L5CF3DHATJ5YUJJTKYTSJHTGDJFBINHTJP3IUTNIEIUA, seg_pos={0 0}, upload=57060]: Uploading piece... Jul 26 23:04:06.808839418 (59h28m16.531402448s) [noise=true , address=79.127.205.234:4616 , piece_id=UWRFH2UD5FL6H757C4BXNQRPH3NEHB3OKYRKG2YUBZ4R6P5TKYGQ, seg_pos={0 0}, upload=57060]: Uploading piece... Jul 26 23:04:06.808963334 (59h28m16.531526375s) [noise=true , address=79.127.220.101:4048 , piece_id=JRQ6SPUVQ4VCAE37CAUTHROMAKNYRP5QWF5MLODYMRJRSXCFDMGA, seg_pos={0 0}, upload=57060]: Uploading piece... Jul 26 23:04:06.809131737 (59h28m16.531694777s) [noise=true , address=79.127.201.214:4321 , piece_id=34NGGJDL7X6CSPXFC5ELTEXJXYLCNKPMHFN6WELYMNTRH5WAMLOQ, seg_pos={0 0}, upload=57060]: Uploading piece... Jul 26 23:04:06.809235205 (59h28m16.531798245s) [noise=true , address=156.146.43.225:4434 , piece_id=A7SUZAMMUPCSWMICP3YOFDCGUQPKNE2LABPDPPKWD523WYWTJK5A, seg_pos={0 0}, upload=57060]: Uploading piece... Jul 26 23:04:06.809413415 (59h28m16.531976455s) [noise=true , address=45.91.116.14:4222 , piece_id=P6CT2WESKZPMQ2C66PQWJ2VKJVIZJT453LSWTCIROHWO5RNBFOSA, seg_pos={0 0}, upload=57060]: Uploading piece... Jul 26 23:04:06.809438393 (59h28m16.532001423s) [noise=true , address=79.127.219.38:4005 , piece_id=OMPH5ELBD42ZEHK74NY6CLWHT2ZCI6QRXKGG7FA4HPJZVPZVF2NQ, seg_pos={0 0}, upload=57060]: Uploading piece... Jul 26 23:04:06.809663765 (59h28m16.532226805s) [noise=true , address=64.49.13.138:4482 , piece_id=2DTKNFFXBNTI4BFJAIH6EZQZGRPCCPTUQ7IB65UH2ZID5V5VCICQ, seg_pos={0 0}, upload=57060]: Uploading piece... Jul 26 23:04:06.809967596 (59h28m16.532530626s) [noise=true , address=45.91.116.11:4121 , piece_id=ICVFMI7J7IDANDKTKUD3OF4JXQD2VS5WEHZRJVAQJ7LW7ZOLWDHQ, seg_pos={0 0}, upload=57060]: Uploading piece... Jul 26 23:04:06.810190863 (59h28m16.532753904s) [noise=true , address=79.127.219.38:4018 , piece_id=6YOYUWQ5IN7KQT2PPPF73X63GUGASOI3CHRJHT4DPLH54YJYVK4A, seg_pos={0 0}, upload=57060]: Uploading piece... Jul 26 23:04:07.741168321 (59h28m17.463731361s) [noise=true , address=79.127.220.101:4048 , piece_id=JRQ6SPUVQ4VCAE37CAUTHROMAKNYRP5QWF5MLODYMRJRSXCFDMGA, seg_pos={0 0}, upload=57060]: Done uploading piece. err: ecclient: upload failed (node:1csdxFUHCWKuGfrZnVfpRJ3R6DYCzxx3voY1UR5kWwSXaxZaPg, address:79.127.220.101:4048): protocol: storage node overloaded, request limit: 2000 Jul 26 23:04:07.843515683 (59h28m17.566078723s) [noise=true , address=79.127.219.51:4210 , piece_id=U2VV733BQ5OST2ZXUZVS4CXBA74KQ56WE4MP5UBVEZY2AFDT6RWQ, seg_pos={0 0}, upload=57060]: Done uploading piece. err: ecclient: upload failed (node:1yCKELQfksAs1CLfacoYZgpHGzRM3UTGak3eWPu9ds3qn1a9G2, address:79.127.219.51:4210): protocol: storage node overloaded, request limit: 2000 Jul 26 23:04:07.885374262 (59h28m17.607937352s) [noise=true , address=79.127.219.38:4018 , piece_id=6YOYUWQ5IN7KQT2PPPF73X63GUGASOI3CHRJHT4DPLH54YJYVK4A, seg_pos={0 0}, upload=57060]: Done uploading piece. err: ecclient: upload failed (node:12waxNyWj1kBeWE3W1VtF94zw1berZJHeERXuBQYq7UMxERry2f, address:79.127.219.38:4018): protocol: storage node overloaded, request limit: 2000 Jul 26 23:04:07.903753618 (59h28m17.626316648s) [upload=57060]: Next returned nil segment Jul 26 23:04:07.903770810 (59h28m17.62633385s) [upload=57060]: Waiting for error group managing segments... Jul 26 23:04:07.964059912 (59h28m17.686622952s) [noise=true , address=79.127.220.97:4361 , piece_id=SEIM7SPP6ZFV4327HPIQIW7NTEKC6VHFQOSJNNMQKUSRAYTPBANQ, seg_pos={0 0}, upload=57060]: Done uploading piece. err: ecclient: upload failed (node:1ojv5zf5QshKyMqGypaEaYmGvzt96UM9kZgyGeFVPMFk8Twzyn, address:79.127.220.97:4361): protocol: storage node overloaded, request limit: 2000 Jul 26 23:04:08.194813986 (59h28m17.917377016s) [noise=true , address=79.127.205.234:4616 , piece_id=UWRFH2UD5FL6H757C4BXNQRPH3NEHB3OKYRKG2YUBZ4R6P5TKYGQ, seg_pos={0 0}, upload=57060]: Done uploading piece. err: <nil> Jul 26 23:04:08.245099397 (59h28m17.967662437s) [noise=true , address=79.127.205.245:4247 , piece_id=4NHIH6EMTFNHMTBJVHB6S3YBXALY726STCICQLIYPAZUQW7NWMWQ, seg_pos={0 0}, upload=57060]: Done uploading piece. err: <nil> Jul 26 23:04:08.501951826 (59h28m18.224514856s) [noise=true , address=64.49.13.138:4482 , piece_id=2DTKNFFXBNTI4BFJAIH6EZQZGRPCCPTUQ7IB65UH2ZID5V5VCICQ, seg_pos={0 0}, upload=57060]: Done uploading piece. err: <nil> Jul 26 23:04:08.699122404 (59h28m18.421685444s) [noise=true , address=79.127.219.38:4005 , piece_id=OMPH5ELBD42ZEHK74NY6CLWHT2ZCI6QRXKGG7FA4HPJZVPZVF2NQ, seg_pos={0 0}, upload=57060]: Done uploading piece. err: <nil> Jul 26 23:04:10.843739892 (59h28m20.566302932s) [noise=true , address=79.127.219.47:4061 , piece_id=SL6FSCMAX4CSBR3QEHLAJM5XVNFOFCGU6NWG6UERMTXE327MXVCQ, seg_pos={0 0}, upload=57060]: Done uploading piece. err: <nil> Jul 26 23:04:11.785943139 (59h28m21.508506179s) [noise=true , address=79.127.201.214:4321 , piece_id=34NGGJDL7X6CSPXFC5ELTEXJXYLCNKPMHFN6WELYMNTRH5WAMLOQ, seg_pos={0 0}, upload=57060]: Done uploading piece. err: <nil> Jul 26 23:04:11.806556990 (59h28m21.52912003s) [noise=true , address=156.146.43.225:4434 , piece_id=A7SUZAMMUPCSWMICP3YOFDCGUQPKNE2LABPDPPKWD523WYWTJK5A, seg_pos={0 0}, upload=57060]: Done uploading piece. err: <nil> Jul 26 23:04:14.122333398 (59h28m23.844896438s) [noise=true , address=156.146.43.225:4449 , piece_id=IXR2VSJ5L5CF3DHATJ5YUJJTKYTSJHTGDJFBINHTJP3IUTNIEIUA, seg_pos={0 0}, upload=57060]: Done uploading piece. err: <nil> Jul 26 23:04:14.163382567 (59h28m23.885945607s) [noise=true , address=45.91.116.11:4121 , piece_id=ICVFMI7J7IDANDKTKUD3OF4JXQD2VS5WEHZRJVAQJ7LW7ZOLWDHQ, seg_pos={0 0}, upload=57060]: Done uploading piece. err: ecclient: upload failed (node:1wkujwGr7G6yGYFLiYcxfBnN1pmKY6mHXoAzZWAjGCCiWWZNdh, address:45.91.116.11:4121): protocol: storage node overloaded, request limit: 2000 Jul 26 23:04:17.042675687 (59h28m26.765238727s) [noise=true , address=77.90.31.19:4875 , piece_id=JCSE64XIUP5SNEOSQOEMYVNBXOM6GOWBFDKN544FINLRARR6OILA, seg_pos={0 0}, upload=57060]: Done uploading piece. err: <nil> Jul 26 23:04:32.718582027 (59h28m42.441145067s) [noise=true , address=45.91.116.14:4222 , piece_id=P6CT2WESKZPMQ2C66PQWJ2VKJVIZJT453LSWTCIROHWO5RNBFOSA, seg_pos={0 0}, upload=57060]: Done uploading piece. err: <nil> Jul 26 23:04:36.575015937 (59h28m46.297578977s) [noise=true , address=77.90.31.12:4601 , piece_id=L7VLEU4J3K4XGUIHPKU77U3MM7E4XUDHPNZNPQO7M7UTM7TUBYRA, seg_pos={0 0}, upload=57060]: Done uploading piece. err: <nil> Jul 26 23:04:46.541082188 (59h28m56.263645228s) [noise=true , address=79.127.201.215:4348 , piece_id=U6QH4RGLE6SJI6KGZ46OGWZEDDZ3WBPTJP35CFABBYXF4TPNNJIQ, seg_pos={0 0}, upload=57060]: Done uploading piece. err: <nil> Jul 26 23:05:05.718284653 (59h29m15.440847693s) [noise=true , address=77.90.31.19:4850 , piece_id=Q3STRX3WDECFMF3X2ZYTFYXMTHK736HTI45QJCPBBQN6AO7TQAQA, seg_pos={0 0}, upload=57060]: Done uploading piece. err: ecclient: upload failed (node:12oGsNdVAKAabvVuWFZ2mpydVYjzhJycig4cqdcE5oAS2YsHmYv, address:77.90.31.19:4850): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:05:05.718336933 (59h29m15.440899973s) [seg_pos={0 0}, upload=57060]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 26 23:05:05.951810039 (59h29m15.674373069s) [upload=57060]: Done uploading segments... Jul 26 23:05:05.951834506 (59h29m15.674397546s) [upload=57060]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1m0.235463699s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [.........X..........X.X........X.X..........................] duration: 5.714129429s FAIL (2025-07-26 23:04:48 UTC) orbiter warning: uplink: too many requests Duration: 5.714154909s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [.......X...X...............X..XX...X.X.X.X.......XX..XX.XX..] duration: 12.962351499s FAIL (2025-07-26 23:04:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 12.962380435s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................................X.........X...........X....] duration: 5.78343344s FAIL (2025-07-26 23:04:17 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== UPht7vBRdA3BcqRek): protocol: rpc: dial tcp 77.90.31.11:4554: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:00:36.538079013 (59h24m46.260642053s) [seg_pos={0 0}, upload=57008]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 23:00:36.538172713 (59h24m46.260735753s) [upload=57008]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 23:00:36.538178714 (59h24m46.260741784s) [upload=57008]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 23:00:36.813206693 (59h24m46.535769733s) [upload=57008]: Done uploading segments... Jul 26 23:00:36.813224688 (59h24m46.535787728s) [upload=57008]: Upload finished. err: <nil> Jul 26 23:01:26.980982718 (59h25m36.703545748s) [upload=57021]: Starting upload Jul 26 23:01:26.981034367 (59h25m36.703597407s) [upload=57021]: Done starting upload Jul 26 23:01:26.981136591 (59h25m36.703699632s) [upload=57021]: Uploading segments... Jul 26 23:01:26.981370199 (59h25m36.703933229s) [upload=57021]: Got next segment. Inline: false Jul 26 23:01:26.981405947 (59h25m36.703968987s) [upload=57021]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:01:26.981412309 (59h25m36.703975349s) [upload=57021]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:01:27.212088890 (59h25m36.93465193s) [seg_pos={0 0}, upload=57021]: Begin upload segment... Jul 26 23:01:27.212149156 (59h25m36.934712196s) [seg_pos={0 0}, upload=57021]: Done begin upload segment. Jul 26 23:01:27.212191998 (59h25m36.934755038s) [noise=true , address=45.91.116.8:4048 , piece_id=XPYI4RBD4CPXZDEQQDNOHSOFIGFIN5FOLGTCAEUDJ54NECUNIEAA, seg_pos={0 0}, upload=57021]: Uploading piece... Jul 26 23:01:27.212264527 (59h25m36.934827577s) [noise=true , address=45.91.116.5:4298 , piece_id=K7332VRJDC2KTKZ3LVD3IRVGXT6HOYKD4MPER454SXOE4GFW67LQ, seg_pos={0 0}, upload=57021]: Uploading piece... Jul 26 23:01:27.212718034 (59h25m36.935281074s) [noise=true , address=77.90.31.11:4555 , piece_id=GED5M25FMD6GAM42AVXKR7U7XONLKFRQPPLYVHGKCHTTCRC6AFOA, seg_pos={0 0}, upload=57021]: Uploading piece... Jul 26 23:01:27.212787557 (59h25m36.935350588s) [noise=true , address=45.91.116.7:4009 , piece_id=3ORNCRFYD5OI5SPXYZEMSVMZ5LDIEEK7YOUA7JERUKAKZ7QDMMKA, seg_pos={0 0}, upload=57021]: Uploading piece... Jul 26 23:01:27.213064708 (59h25m36.935627748s) [noise=true , address=77.90.31.16:4734 , piece_id=ESJWP6PQ7TWJY24SAYQUGVYF7CJP6WUXAKLLH6BXHXOB6KOTQ4AQ, seg_pos={0 0}, upload=57021]: Uploading piece... Jul 26 23:01:27.213128920 (59h25m36.93569196s) [noise=true , address=77.90.31.21:4879 , piece_id=TERNE4P5KQB5WZHFHY5WJEDQH4JNPBCHN7DPMTZCT6GOY6RMH53Q, seg_pos={0 0}, upload=57021]: Uploading piece... Jul 26 23:01:27.213497365 (59h25m36.936060405s) [noise=true , address=45.91.116.5:4299 , piece_id=SBE7NRD4OHRHSGDZ6JLXVXKHNXR2MHQZDNJ5JCFLI7N73S6YOLAA, seg_pos={0 0}, upload=57021]: Uploading piece... Jul 26 23:01:27.213883113 (59h25m36.936446153s) [noise=true , address=45.91.116.10:4081 , piece_id=PX4GKYJPT5IPYLSCUAPDFU5YUZQ5AUG6L7SU5EG2T5YRH6OPFBIQ, seg_pos={0 0}, upload=57021]: Uploading piece... Jul 26 23:01:27.214727529 (59h25m36.937290569s) [noise=true , address=77.90.31.21:4878 , piece_id=LN7SPTAD7D67WVOSS4CWSUTBTHZM3O5B47XJHVMRKLUSH6NSJZOQ, seg_pos={0 0}, upload=57021]: Uploading piece... Jul 26 23:01:27.218515485 (59h25m36.941078525s) [noise=true , address=45.91.116.7:4011 , piece_id=6NVHAO3YBTOAVDC5SS6ZKUMSWIYWALL6T7HJCCB5NOU3UDB2MOLA, seg_pos={0 0}, upload=57021]: Uploading piece... Jul 26 23:01:27.219396701 (59h25m36.941959741s) [noise=true , address=45.91.116.10:4083 , piece_id=BUKUHO4PYDFYZI7JVLT7TZ34PHOIZVTLFPVCVKNFE53YULYY7TMA, seg_pos={0 0}, upload=57021]: Uploading piece... Jul 26 23:01:27.219692617 (59h25m36.942255657s) [noise=true , address=77.90.31.13:4628 , piece_id=FIFB6R2VTRLXOLDRDKI52HCONBTFCCTVXDPHQ3EEKZXXUBVX7QIA, seg_pos={0 0}, upload=57021]: Uploading piece... Jul 26 23:01:27.219909352 (59h25m36.942472382s) [noise=true , address=45.91.116.12:4225 , piece_id=UVWLV3DIL25HOU6PEGNVLHB3RJAHJ2U72CQG2LFICLB6NXMZEKIA, seg_pos={0 0}, upload=57021]: Uploading piece... Jul 26 23:01:27.220107901 (59h25m36.942670941s) [noise=true , address=45.91.116.4:4940 , piece_id=7553SGZFOIK7GNACZ7VSUSGJWXSLPZGSY6K6E2JGB4GPUUBGQYUQ, seg_pos={0 0}, upload=57021]: Uploading piece... Jul 26 23:01:27.220340697 (59h25m36.942903737s) [noise=true , address=45.91.116.4:4937 , piece_id=EBF5J3XDUYMJVGFWIMYRZ6YNSUETMATVVOJIH2E3NMNWDBD6JEDQ, seg_pos={0 0}, upload=57021]: Uploading piece... Jul 26 23:01:27.220536642 (59h25m36.943099682s) [noise=true , address=45.91.116.6:4973 , piece_id=WCZZVRBQWYMSZ366CIB3ILVH4GR77KOGRSEZUFH22YMZLYYVJ64Q, seg_pos={0 0}, upload=57021]: Uploading piece... Jul 26 23:01:28.909687093 (59h25m38.632250133s) [upload=57021]: Next returned nil segment Jul 26 23:01:28.909721920 (59h25m38.63228496s) [upload=57021]: Waiting for error group managing segments... Jul 26 23:01:29.241265540 (59h25m38.96382859s) [noise=true , address=45.91.116.10:4081 , piece_id=PX4GKYJPT5IPYLSCUAPDFU5YUZQ5AUG6L7SU5EG2T5YRH6OPFBIQ, seg_pos={0 0}, upload=57021]: Done uploading piece. err: <nil> Jul 26 23:01:29.245035954 (59h25m38.967599004s) [noise=true , address=45.91.116.5:4299 , piece_id=SBE7NRD4OHRHSGDZ6JLXVXKHNXR2MHQZDNJ5JCFLI7N73S6YOLAA, seg_pos={0 0}, upload=57021]: Done uploading piece. err: <nil> Jul 26 23:01:29.248616093 (59h25m38.971179143s) [noise=true , address=45.91.116.7:4009 , piece_id=3ORNCRFYD5OI5SPXYZEMSVMZ5LDIEEK7YOUA7JERUKAKZ7QDMMKA, seg_pos={0 0}, upload=57021]: Done uploading piece. err: <nil> Jul 26 23:01:29.251100826 (59h25m38.973663886s) [noise=true , address=45.91.116.6:4973 , piece_id=WCZZVRBQWYMSZ366CIB3ILVH4GR77KOGRSEZUFH22YMZLYYVJ64Q, seg_pos={0 0}, upload=57021]: Done uploading piece. err: <nil> Jul 26 23:01:29.252327092 (59h25m38.974890132s) [noise=true , address=45.91.116.7:4011 , piece_id=6NVHAO3YBTOAVDC5SS6ZKUMSWIYWALL6T7HJCCB5NOU3UDB2MOLA, seg_pos={0 0}, upload=57021]: Done uploading piece. err: <nil> Jul 26 23:01:29.252808223 (59h25m38.975371263s) [noise=true , address=45.91.116.5:4298 , piece_id=K7332VRJDC2KTKZ3LVD3IRVGXT6HOYKD4MPER454SXOE4GFW67LQ, seg_pos={0 0}, upload=57021]: Done uploading piece. err: <nil> Jul 26 23:01:29.256185123 (59h25m38.978748173s) [noise=true , address=45.91.116.4:4940 , piece_id=7553SGZFOIK7GNACZ7VSUSGJWXSLPZGSY6K6E2JGB4GPUUBGQYUQ, seg_pos={0 0}, upload=57021]: Done uploading piece. err: <nil> Jul 26 23:01:29.257135561 (59h25m38.979698611s) [noise=true , address=45.91.116.4:4937 , piece_id=EBF5J3XDUYMJVGFWIMYRZ6YNSUETMATVVOJIH2E3NMNWDBD6JEDQ, seg_pos={0 0}, upload=57021]: Done uploading piece. err: <nil> Jul 26 23:01:29.325606113 (59h25m39.048169153s) [noise=true , address=45.91.116.8:4048 , piece_id=XPYI4RBD4CPXZDEQQDNOHSOFIGFIN5FOLGTCAEUDJ54NECUNIEAA, seg_pos={0 0}, upload=57021]: Done uploading piece. err: <nil> Jul 26 23:01:29.331150530 (59h25m39.05371356s) [noise=true , address=45.91.116.12:4225 , piece_id=UVWLV3DIL25HOU6PEGNVLHB3RJAHJ2U72CQG2LFICLB6NXMZEKIA, seg_pos={0 0}, upload=57021]: Done uploading piece. err: <nil> Jul 26 23:01:29.392913809 (59h25m39.115476879s) [noise=true , address=45.91.116.10:4083 , piece_id=BUKUHO4PYDFYZI7JVLT7TZ34PHOIZVTLFPVCVKNFE53YULYY7TMA, seg_pos={0 0}, upload=57021]: Done uploading piece. err: <nil> Jul 26 23:01:56.339456448 (59h26m6.062019488s) [noise=true , address=77.90.31.21:4878 , piece_id=LN7SPTAD7D67WVOSS4CWSUTBTHZM3O5B47XJHVMRKLUSH6NSJZOQ, seg_pos={0 0}, upload=57021]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: rpc: dial tcp 77.90.31.21:4878: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:01:56.339508978 (59h26m6.062072019s) [noise=true , address=77.90.31.13:4628 , piece_id=FIFB6R2VTRLXOLDRDKI52HCONBTFCCTVXDPHQ3EEKZXXUBVX7QIA, seg_pos={0 0}, upload=57021]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: write tcp 188.166.225.170:35758->77.90.31.13:4628: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:01:56.339539306 (59h26m6.062102346s) [noise=true , address=77.90.31.11:4555 , piece_id=GED5M25FMD6GAM42AVXKR7U7XONLKFRQPPLYVHGKCHTTCRC6AFOA, seg_pos={0 0}, upload=57021]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: rpc: dial tcp 77.90.31.11:4555: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:01:56.339599221 (59h26m6.062162261s) [noise=true , address=77.90.31.16:4734 , piece_id=ESJWP6PQ7TWJY24SAYQUGVYF7CJP6WUXAKLLH6BXHXOB6KOTQ4AQ, seg_pos={0 0}, upload=57021]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: rpc: dial tcp 77.90.31.16:4734: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:01:56.339728408 (59h26m6.062291438s) [noise=true , address=77.90.31.21:4879 , piece_id=TERNE4P5KQB5WZHFHY5WJEDQH4JNPBCHN7DPMTZCT6GOY6RMH53Q, seg_pos={0 0}, upload=57021]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: write tcp 188.166.225.170:50280->77.90.31.21:4879: use of closed network connection; context deadline exceeded Jul 26 23:01:56.339777903 (59h26m6.062340943s) [seg_pos={0 0}, upload=57021]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 26 23:01:56.572323148 (59h26m6.294886188s) [upload=57021]: Done uploading segments... Jul 26 23:01:56.572342274 (59h26m6.294905314s) [upload=57021]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.783457996s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [........X...X...............X..XX...X.X.X.X.......XX..XX.XX.] duration: 25.396680938s FAIL (2025-07-26 23:04:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 25.396703539s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)