Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 22 out of 60 is failed (threshold=3) [XXXX.XX.XXXXX.X.XXXXX.XXX.....X.........................X...] duration: 5.791578449s FAIL (2025-07-26 22:31:17 UTC) orbiter warning: digest mismatch: after b80432bf250cf8c010a58d5a57ae8eb2e733e016512b73b27f8bbe5c05d36821 != before 1ae5bfd23ae4b196a480bbedcce3ef7b0078f8ffef79859a6873ad6df52a9a98 Duration: 5.791608136s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.237142639s FAIL (2025-07-26 22:30:56 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 6 22:29:56.977419260 (58h54m6.6999823s) [upload=56520]: Done starting upload Jul 26 22:29:56.977499442 (58h54m6.700062482s) [upload=56520]: Uploading segments... Jul 26 22:29:56.977603632 (58h54m6.700166662s) [upload=56520]: Got next segment. Inline: false Jul 26 22:29:56.977643058 (58h54m6.700206098s) [upload=56520]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:29:56.977649159 (58h54m6.700212199s) [upload=56520]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:29:57.422543561 (58h54m7.145106601s) [seg_pos={0 0}, upload=56520]: Begin upload segment... Jul 26 22:29:57.422590712 (58h54m7.145153742s) [seg_pos={0 0}, upload=56520]: Done begin upload segment. Jul 26 22:29:57.422613165 (58h54m7.145176195s) [noise=true , address=77.90.31.21:4880 , piece_id=NVZHDGIIDNB3Z4OTVMWTFQZV5PROW5OBI2V5ASCDEFID4SXDX3PA, seg_pos={0 0}, upload=56520]: Uploading piece... Jul 26 22:29:57.422666758 (58h54m7.145229798s) [noise=true , address=64.49.13.140:4514 , piece_id=PBFMXEI246DE7FOOHU2IKHN7LCWKRUIPLKMFFGGQFOHETTFB4PNA, seg_pos={0 0}, upload=56520]: Uploading piece... Jul 26 22:29:57.423035032 (58h54m7.145598072s) [noise=true , address=45.91.116.9:4274 , piece_id=QO4D5UDPSXU75JJT7RU7O7DEPHOXSOVGR3SIYY2C4G3S4RLKQDGQ, seg_pos={0 0}, upload=56520]: Uploading piece... Jul 26 22:29:57.423105597 (58h54m7.145668627s) [noise=true , address=79.127.220.101:4048 , piece_id=X6QNXZLS2H2B6VXWERS3LVKKEYOGH4H7JMRO3JVUN56USUYJ43KA, seg_pos={0 0}, upload=56520]: Uploading piece... Jul 26 22:29:57.423374382 (58h54m7.145937422s) [noise=true , address=79.127.205.234:4610 , piece_id=NV74ZIH2DVJ2YWOQA6SVLT3EBRITLL4FU555OHHWGDN77EOO367A, seg_pos={0 0}, upload=56520]: Uploading piece... Jul 26 22:29:57.423488270 (58h54m7.14605131s) [noise=true , address=45.91.116.8:4065 , piece_id=TXPNX2IAFMFZKBAAR5VG53UPGHI6DUTR3FNV2TIJVR7CQ6EVBMWA, seg_pos={0 0}, upload=56520]: Uploading piece... Jul 26 22:29:57.423644539 (58h54m7.146207579s) [noise=true , address=156.146.43.225:4448 , piece_id=XTNGF2FAVKQRFVANNXJQRLUFXHYP3WG4SIUCAF4RTDZQ2W4FEBOA, seg_pos={0 0}, upload=56520]: Uploading piece... Jul 26 22:29:57.423881862 (58h54m7.146444892s) [noise=true , address=79.127.201.215:4359 , piece_id=CO35TK5PFR3IQM5OA2VR3W46LB7Q6QNCIYVWODPL3FLDQAJZB6VA, seg_pos={0 0}, upload=56520]: Uploading piece... Jul 26 22:29:57.423962276 (58h54m7.146525316s) [noise=true , address=79.127.205.235:4688 , piece_id=HREWQEQUX6PS7VV6Y2E7JGVN2SAHPG2IML6QTT2LE37BW3HMO5MA, seg_pos={0 0}, upload=56520]: Uploading piece... Jul 26 22:29:57.424137351 (58h54m7.146700391s) [noise=true , address=77.90.31.18:4816 , piece_id=KYSUASIXMJWDGYELCFV27MY5CEP2GVBOVHEF4HAWERJQJJEKAR3A, seg_pos={0 0}, upload=56520]: Uploading piece... Jul 26 22:29:57.424308239 (58h54m7.146871279s) [noise=true , address=77.90.31.19:4846 , piece_id=GPSC7BU7PBJAQ664EDHWBE4NQRDVSYIGNKBEJPCDVONH2AGEVZDA, seg_pos={0 0}, upload=56520]: Uploading piece... Jul 26 22:29:57.424415324 (58h54m7.146978364s) [noise=true , address=79.127.219.47:4061 , piece_id=BIYP433L5VSR2ZFQTXVAZT7DJNSQ424VQVAH3IIZOG3BLUTDO3SA, seg_pos={0 0}, upload=56520]: Uploading piece... Jul 26 22:29:57.424565020 (58h54m7.14712806s) [noise=true , address=45.91.116.12:4229 , piece_id=2ZFUUMC66AO63X23GQ5VCCXTMVLLAWODR6ZHJCFIFEORVOFSG4JA, seg_pos={0 0}, upload=56520]: Uploading piece... Jul 26 22:29:57.424677034 (58h54m7.147240074s) [noise=true , address=79.127.205.246:4253 , piece_id=MOS5LTHPIMRU2S2HQOTF6P4SDIVBBF2TZ3SKZEYE7756YRJATK3A, seg_pos={0 0}, upload=56520]: Uploading piece... Jul 26 22:29:57.424778338 (58h54m7.147341378s) [noise=true , address=64.49.13.138:4483 , piece_id=JJ5PUVKLU567L3DDIA57OVMG7AC7ESGNWLX7FKPBLS5LZEDVLHQA, seg_pos={0 0}, upload=56520]: Uploading piece... Jul 26 22:29:57.424972760 (58h54m7.1475358s) [noise=true , address=79.127.205.245:4242 , piece_id=VCESOP4DKJ4SLODZUH3HV33AIHY2ISJ47ZAN6LXTXCWM2IUNRTDA, seg_pos={0 0}, upload=56520]: Uploading piece... Jul 26 22:29:57.425159718 (58h54m7.147722758s) [noise=true , address=79.127.219.51:4210 , piece_id=3EFZHTZSJC2V2VFWIPRCRGGTCZZ4C47ESGV247XFSPUGPPL6DMVA, seg_pos={0 0}, upload=56520]: Uploading piece... Jul 26 22:29:57.425344781 (58h54m7.147907821s) [noise=true , address=45.91.116.14:4206 , piece_id=UHZK52GIQ7AESOOBGCX7DWJ5ZFWU5MXQZP6AMUVQQZTV4Q5TS5JQ, seg_pos={0 0}, upload=56520]: Uploading piece... Jul 26 22:29:58.345614291 (58h54m8.068177321s) [noise=true , address=79.127.220.101:4048 , piece_id=X6QNXZLS2H2B6VXWERS3LVKKEYOGH4H7JMRO3JVUN56USUYJ43KA, seg_pos={0 0}, upload=56520]: Done uploading piece. err: ecclient: upload failed (node:1csdxFUHCWKuGfrZnVfpRJ3R6DYCzxx3voY1UR5kWwSXaxZaPg, address:79.127.220.101:4048): protocol: storage node overloaded, request limit: 2000 Jul 26 22:29:58.447540042 (58h54m8.170103083s) [noise=true , address=156.146.43.225:4448 , piece_id=XTNGF2FAVKQRFVANNXJQRLUFXHYP3WG4SIUCAF4RTDZQ2W4FEBOA, seg_pos={0 0}, upload=56520]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jul 26 22:29:58.513681368 (58h54m8.236244408s) [noise=true , address=79.127.219.51:4210 , piece_id=3EFZHTZSJC2V2VFWIPRCRGGTCZZ4C47ESGV247XFSPUGPPL6DMVA, seg_pos={0 0}, upload=56520]: Done uploading piece. err: ecclient: upload failed (node:1yCKELQfksAs1CLfacoYZgpHGzRM3UTGak3eWPu9ds3qn1a9G2, address:79.127.219.51:4210): protocol: storage node overloaded, request limit: 2000 Jul 26 22:29:58.514680189 (58h54m8.237243239s) [upload=56520]: Next returned nil segment Jul 26 22:29:58.514690539 (58h54m8.237253579s) [upload=56520]: Waiting for error group managing segments... Jul 26 22:29:58.532158489 (58h54m8.25472154s) [noise=true , address=79.127.219.47:4061 , piece_id=BIYP433L5VSR2ZFQTXVAZT7DJNSQ424VQVAH3IIZOG3BLUTDO3SA, seg_pos={0 0}, upload=56520]: Done uploading piece. err: ecclient: upload failed (node:12f4AfVs2T4a8Yhd456WKdse7TZSjHBDULMR8myvo1ua9jiUrNu, address:79.127.219.47:4061): protocol: storage node overloaded, request limit: 2000 Jul 26 22:29:58.799542497 (58h54m8.522105537s) [noise=true , address=79.127.205.245:4242 , piece_id=VCESOP4DKJ4SLODZUH3HV33AIHY2ISJ47ZAN6LXTXCWM2IUNRTDA, seg_pos={0 0}, upload=56520]: Done uploading piece. err: <nil> Jul 26 22:29:58.816919113 (58h54m8.539482143s) [noise=true , address=79.127.205.234:4610 , piece_id=NV74ZIH2DVJ2YWOQA6SVLT3EBRITLL4FU555OHHWGDN77EOO367A, seg_pos={0 0}, upload=56520]: Done uploading piece. err: <nil> Jul 26 22:29:58.817901092 (58h54m8.540464132s) [noise=true , address=79.127.205.246:4253 , piece_id=MOS5LTHPIMRU2S2HQOTF6P4SDIVBBF2TZ3SKZEYE7756YRJATK3A, seg_pos={0 0}, upload=56520]: Done uploading piece. err: <nil> Jul 26 22:29:59.392780238 (58h54m9.115343279s) [noise=true , address=64.49.13.138:4483 , piece_id=JJ5PUVKLU567L3DDIA57OVMG7AC7ESGNWLX7FKPBLS5LZEDVLHQA, seg_pos={0 0}, upload=56520]: Done uploading piece. err: <nil> Jul 26 22:29:59.421651306 (58h54m9.144214347s) [noise=true , address=77.90.31.21:4880 , piece_id=NVZHDGIIDNB3Z4OTVMWTFQZV5PROW5OBI2V5ASCDEFID4SXDX3PA, seg_pos={0 0}, upload=56520]: Done uploading piece. err: <nil> Jul 26 22:29:59.593493475 (58h54m9.316056505s) [noise=true , address=64.49.13.140:4514 , piece_id=PBFMXEI246DE7FOOHU2IKHN7LCWKRUIPLKMFFGGQFOHETTFB4PNA, seg_pos={0 0}, upload=56520]: Done uploading piece. err: <nil> Jul 26 22:29:59.841941222 (58h54m9.564504262s) [noise=true , address=79.127.205.235:4688 , piece_id=HREWQEQUX6PS7VV6Y2E7JGVN2SAHPG2IML6QTT2LE37BW3HMO5MA, seg_pos={0 0}, upload=56520]: Done uploading piece. err: <nil> Jul 26 22:30:00.091771804 (58h54m9.814334834s) [noise=true , address=45.91.116.9:4274 , piece_id=QO4D5UDPSXU75JJT7RU7O7DEPHOXSOVGR3SIYY2C4G3S4RLKQDGQ, seg_pos={0 0}, upload=56520]: Done uploading piece. err: <nil> Jul 26 22:30:01.473265338 (58h54m11.195828388s) [noise=true , address=45.91.116.14:4206 , piece_id=UHZK52GIQ7AESOOBGCX7DWJ5ZFWU5MXQZP6AMUVQQZTV4Q5TS5JQ, seg_pos={0 0}, upload=56520]: Done uploading piece. err: <nil> Jul 26 22:30:03.285952370 (58h54m13.00851541s) [noise=true , address=45.91.116.8:4065 , piece_id=TXPNX2IAFMFZKBAAR5VG53UPGHI6DUTR3FNV2TIJVR7CQ6EVBMWA, seg_pos={0 0}, upload=56520]: Done uploading piece. err: <nil> Jul 26 22:30:18.684172553 (58h54m28.406735593s) [noise=true , address=45.91.116.12:4229 , piece_id=2ZFUUMC66AO63X23GQ5VCCXTMVLLAWODR6ZHJCFIFEORVOFSG4JA, seg_pos={0 0}, upload=56520]: Done uploading piece. err: ecclient: upload failed (node:12rfuAG41swAn82EM5ZTAN31NhGp4UKMHFiZiT57NXmSAi3MHe2, address:45.91.116.12:4229): protocol: storage node overloaded, request limit: 2000 Jul 26 22:30:42.327137798 (58h54m52.049700838s) [noise=true , address=79.127.201.215:4359 , piece_id=CO35TK5PFR3IQM5OA2VR3W46LB7Q6QNCIYVWODPL3FLDQAJZB6VA, seg_pos={0 0}, upload=56520]: Done uploading piece. err: <nil> Jul 26 22:30:56.333445359 (58h55m6.056008399s) [noise=true , address=77.90.31.19:4846 , piece_id=GPSC7BU7PBJAQ664EDHWBE4NQRDVSYIGNKBEJPCDVONH2AGEVZDA, seg_pos={0 0}, upload=56520]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:30:56.333491207 (58h55m6.056054247s) [noise=true , address=77.90.31.18:4816 , piece_id=KYSUASIXMJWDGYELCFV27MY5CEP2GVBOVHEF4HAWERJQJJEKAR3A, seg_pos={0 0}, upload=56520]: Done uploading piece. err: ecclient: upload failed (node:12Vno3GkFXR7KdTUEAhvesqif7KBF6DZoeMTv8n9MpRbYZNktaT, address:77.90.31.18:4816): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:30:56.333519381 (58h55m6.056082421s) [seg_pos={0 0}, upload=56520]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 26 22:30:56.569284821 (58h55m6.291847851s) [upload=56520]: Done uploading segments... Jul 26 22:30:56.569317734 (58h55m6.291880774s) [upload=56520]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1m0.237155833s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 22 out of 60 is failed (threshold=3) [.XXXX.XX.XXXXX.X.XXXXX.XXX.....X.........................X..] duration: 5.860902162s FAIL (2025-07-26 22:30:56 UTC) orbiter warning: digest mismatch: after b80432bf250cf8c010a58d5a57ae8eb2e733e016512b73b27f8bbe5c05d36821 != before 1ae5bfd23ae4b196a480bbedcce3ef7b0078f8ffef79859a6873ad6df52a9a98 Duration: 5.860929726s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 23 out of 60 is failed (threshold=3) [X.XXXX.XX.XXXXX.X.XXXXX.XXX.....X.........................X.] duration: 5.742804602s FAIL (2025-07-26 22:30:40 UTC) orbiter warning: digest mismatch: after b80432bf250cf8c010a58d5a57ae8eb2e733e016512b73b27f8bbe5c05d36821 != before 1ae5bfd23ae4b196a480bbedcce3ef7b0078f8ffef79859a6873ad6df52a9a98 Duration: 5.742847645s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [................................................X...X.......] duration: 15.477371209s FAIL (2025-07-26 22:30:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 15.477387161s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [............X..X...........X................................] duration: 5.992793727s FAIL (2025-07-26 22:30:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.992822183s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.XXX.X.XX.XXX.XXXXXX..XXXX..X.X...X.........................] duration: 5.736765699s FAIL (2025-07-26 22:30:24 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== rite tcp 188.166.225.170:53646->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:53646->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53646->77.90.31.10:4519: use of closed network connection Jul 26 22:11:05.345059537 (58h35m15.067622577s) [seg_pos={0 0}, upload=56223]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:11:05.345121094 (58h35m15.067684124s) [upload=56223]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:11:05.345124551 (58h35m15.067687591s) [upload=56223]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:11:05.638665486 (58h35m15.361228526s) [upload=56223]: Done uploading segments... Jul 26 22:11:05.638683710 (58h35m15.36124674s) [upload=56223]: Upload finished. err: <nil> Jul 26 22:12:00.122995475 (58h36m9.845558505s) [upload=56238]: Starting upload Jul 26 22:12:00.123061241 (58h36m9.845624351s) [upload=56238]: Done starting upload Jul 26 22:12:00.123134752 (58h36m9.845697782s) [upload=56238]: Uploading segments... Jul 26 22:12:00.123493779 (58h36m9.846056819s) [upload=56238]: Got next segment. Inline: false Jul 26 22:12:00.123543474 (58h36m9.846106514s) [upload=56238]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:12:00.123548433 (58h36m9.846111473s) [upload=56238]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:12:00.355422881 (58h36m10.077985911s) [seg_pos={0 0}, upload=56238]: Begin upload segment... Jul 26 22:12:00.355465902 (58h36m10.078028942s) [seg_pos={0 0}, upload=56238]: Done begin upload segment. Jul 26 22:12:00.355487364 (58h36m10.078050404s) [noise=true , address=104.36.82.5:4837 , piece_id=WTBNXCZ3PWC5TQM4UWPTKT5QRXHHWQF6JVQQ7VQ4BXKLPEIZMIFQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.355513444 (58h36m10.078076474s) [noise=true , address=45.91.116.9:4263 , piece_id=H67DGXYB4V7UFZAIE4X4VRVWU53RIIQQOWOANN52TAZHFJNQT4IA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.355843715 (58h36m10.078406746s) [noise=true , address=45.91.116.8:4045 , piece_id=PRKLVJLWB37FDN52QBLNATG2SIWCXJCN7OR6EFLRZL74JZLC57VA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.355985106 (58h36m10.078548146s) [noise=true , address=77.90.31.19:4843 , piece_id=P3UAQQ2AO27TVMESOSBZUUXGX6QOBWZPXOMG4LIAUHTSBDN3YBJQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356118291 (58h36m10.078681331s) [noise=true , address=45.91.116.4:4939 , piece_id=XJUAUMJSYPFXTK7JIIJECLKWJOTWJQHYJHQ7TYPHMPBXRBS3HFPA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356442141 (58h36m10.079005181s) [noise=true , address=104.36.82.4:4802 , piece_id=KQ4ABYCX4T6VM477B56ETRPXTVFWDZXUTZEGPNAWIQ7ZV2XPYEIQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356714532 (58h36m10.079277572s) [noise=true , address=45.91.116.12:4225 , piece_id=E2D6FTGWZ22OFNFPRWBE335BANIUZIWVS433SMUC2LPMD6PTI6IA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356937989 (58h36m10.079501029s) [noise=true , address=45.91.116.13:4155 , piece_id=OXPGOXKCI4YOIKSIMPXVV6D74BBVMRPXXZKBBX2P7D4YQVR73VXQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357235789 (58h36m10.079798819s) [noise=true , address=77.90.31.12:4592 , piece_id=O3UI7JY3LQTABYKN3JKO5O4Z4RCLAX6HJZS4HQXJPGKDU3JNECTQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357472842 (58h36m10.080035882s) [noise=true , address=45.91.116.10:4082 , piece_id=J6PNAFSZBQCDVTAIQBHAMMN23AS7X3WLGV3YHVAKDU26RMWGKBLQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357542045 (58h36m10.080105085s) [noise=true , address=45.91.116.9:4262 , piece_id=TPC2GAHLX2R2B3ZZBTFYGWNMPYCX7KMZLNMNVM4GKQDOY6ADGUVQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357791683 (58h36m10.080354723s) [noise=true , address=45.91.116.5:4298 , piece_id=HURPSTTY5SXKPKJIHWPQ6VGKWQPIGSQONUMRQM6BREDGEA5V5ALA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358046780 (58h36m10.080609831s) [noise=true , address=104.36.82.15:4258 , piece_id=MJO75LPAAYCYPVPZYKBASP7ZN7Q3FI44W3XRN3CS5JAARSD32LXA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358143686 (58h36m10.080706716s) [noise=true , address=45.91.116.6:4973 , piece_id=E4B6O574VP2Y3J6PNPKTWAN7BFGSHIVQEPMTBDXQCDIORGOFMU6A, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358493164 (58h36m10.081056204s) [noise=true , address=104.36.82.12:4149 , piece_id=6ZXU4V2YOW3IXGT5XYX3Q7FT4U6YW7IVA72UPFHYUIUBWNPBDOIQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358579300 (58h36m10.08114233s) [noise=true , address=45.91.116.14:4190 , piece_id=PVZ2OCKX3JTUYT6KNUGIRHIYH5YVBSDMIZUOYSYSYBYB6AUNMOOA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:02.081979808 (58h36m11.804542868s) [upload=56238]: Next returned nil segment Jul 26 22:12:02.082008563 (58h36m11.804571643s) [upload=56238]: Waiting for error group managing segments... Jul 26 22:12:02.373463127 (58h36m12.096026167s) [noise=true , address=45.91.116.5:4298 , piece_id=HURPSTTY5SXKPKJIHWPQ6VGKWQPIGSQONUMRQM6BREDGEA5V5ALA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.391294504 (58h36m12.113857544s) [noise=true , address=77.90.31.12:4592 , piece_id=O3UI7JY3LQTABYKN3JKO5O4Z4RCLAX6HJZS4HQXJPGKDU3JNECTQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.392103922 (58h36m12.114666972s) [noise=true , address=45.91.116.9:4263 , piece_id=H67DGXYB4V7UFZAIE4X4VRVWU53RIIQQOWOANN52TAZHFJNQT4IA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.395004321 (58h36m12.117567361s) [noise=true , address=45.91.116.12:4225 , piece_id=E2D6FTGWZ22OFNFPRWBE335BANIUZIWVS433SMUC2LPMD6PTI6IA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.395744818 (58h36m12.118307858s) [noise=true , address=45.91.116.9:4262 , piece_id=TPC2GAHLX2R2B3ZZBTFYGWNMPYCX7KMZLNMNVM4GKQDOY6ADGUVQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.451177047 (58h36m12.173740097s) [noise=true , address=45.91.116.8:4045 , piece_id=PRKLVJLWB37FDN52QBLNATG2SIWCXJCN7OR6EFLRZL74JZLC57VA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.462789285 (58h36m12.185352325s) [noise=true , address=45.91.116.4:4939 , piece_id=XJUAUMJSYPFXTK7JIIJECLKWJOTWJQHYJHQ7TYPHMPBXRBS3HFPA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.477161623 (58h36m12.199724674s) [noise=true , address=45.91.116.13:4155 , piece_id=OXPGOXKCI4YOIKSIMPXVV6D74BBVMRPXXZKBBX2P7D4YQVR73VXQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.477354983 (58h36m12.199918023s) [noise=true , address=45.91.116.10:4082 , piece_id=J6PNAFSZBQCDVTAIQBHAMMN23AS7X3WLGV3YHVAKDU26RMWGKBLQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.482097287 (58h36m12.204660327s) [noise=true , address=45.91.116.6:4973 , piece_id=E4B6O574VP2Y3J6PNPKTWAN7BFGSHIVQEPMTBDXQCDIORGOFMU6A, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.522569814 (58h36m12.245132854s) [noise=true , address=77.90.31.19:4843 , piece_id=P3UAQQ2AO27TVMESOSBZUUXGX6QOBWZPXOMG4LIAUHTSBDN3YBJQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.537542521 (58h36m12.260105571s) [noise=true , address=45.91.116.14:4190 , piece_id=PVZ2OCKX3JTUYT6KNUGIRHIYH5YVBSDMIZUOYSYSYBYB6AUNMOOA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:29.482251815 (58h36m39.204814855s) [noise=true , address=104.36.82.5:4837 , piece_id=WTBNXCZ3PWC5TQM4UWPTKT5QRXHHWQF6JVQQ7VQ4BXKLPEIZMIFQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 188.166.225.170:38046->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482276953 (58h36m39.204839993s) [noise=true , address=104.36.82.4:4802 , piece_id=KQ4ABYCX4T6VM477B56ETRPXTVFWDZXUTZEGPNAWIQ7ZV2XPYEIQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482340875 (58h36m39.204903915s) [noise=true , address=104.36.82.15:4258 , piece_id=MJO75LPAAYCYPVPZYKBASP7ZN7Q3FI44W3XRN3CS5JAARSD32LXA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: write tcp 188.166.225.170:41258->104.36.82.15:4258: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482356565 (58h36m39.204919605s) [noise=true , address=104.36.82.12:4149 , piece_id=6ZXU4V2YOW3IXGT5XYX3Q7FT4U6YW7IVA72UPFHYUIUBWNPBDOIQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 188.166.225.170:39256->104.36.82.12:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482383747 (58h36m39.204946787s) [seg_pos={0 0}, upload=56238]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:12:29.709832457 (58h36m39.432395497s) [upload=56238]: Done uploading segments... Jul 26 22:12:29.709874738 (58h36m39.432437768s) [upload=56238]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.736791076s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 59 out of 60 is failed (threshold=3) [.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.227127217s FAIL (2025-07-26 22:29:56 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== JJOVBSLYFC2YZO4EOHLS36F2XE2VCEBRA, seg_pos={0 0}, upload=56504]: Uploading piece... Jul 26 22:28:56.961766368 (58h53m6.684329428s) [noise=true , address=77.90.31.16:4752 , piece_id=VYROPZXWSN2GOISQLAHDS3DGDDA2VK4IE6R7EVQGW6HS3QG4CW7Q, seg_pos={0 0}, upload=56504]: Uploading piece... Jul 26 22:28:56.961862352 (58h53m6.684425402s) [noise=true , address=104.36.82.40:4964 , piece_id=QTYX4F7KX6A3K4PABTILJJHDLP7O23SBW6PRW4ENMN2HBUI7XWYQ, seg_pos={0 0}, upload=56504]: Uploading piece... Jul 26 22:28:56.962571068 (58h53m6.685134108s) [noise=true , address=77.90.31.15:4717 , piece_id=PQQYSPZUMDOLKRULVRN7VHSGYIVNEK4DRYIHFTYZP3N6NIHYT3PA, seg_pos={0 0}, upload=56504]: Uploading piece... Jul 26 22:28:56.962709152 (58h53m6.685272222s) [noise=true , address=79.127.205.235:4670 , piece_id=SJNVNCJUEZDT6W5GRGMKSTSF2GROFBBMMQHKPJRFVXICE2SPX5GQ, seg_pos={0 0}, upload=56504]: Uploading piece... Jul 26 22:28:56.963224157 (58h53m6.685787227s) [noise=true , address=64.49.13.138:4483 , piece_id=JIDRCH237ZLFDCJ5PO6LWXOBWMV7IKG2T4GWRHO4JRRIXNMVGIQQ, seg_pos={0 0}, upload=56504]: Uploading piece... Jul 26 22:28:56.963688225 (58h53m6.686251265s) [noise=true , address=104.36.82.2:4752 , piece_id=A7SOQHOA6Z7QBFQEKEGJISEALDNUUOJRZ6ZWQW3UFYK2SK2E2UPQ, seg_pos={0 0}, upload=56504]: Uploading piece... Jul 26 22:28:56.964873603 (58h53m6.687436643s) [noise=true , address=79.127.219.47:4069 , piece_id=KKQ4QHTKUEYCTCCUNMVVYERB42LZFO76PO2O3MXD6OWQ7FBBUPPA, seg_pos={0 0}, upload=56504]: Uploading piece... Jul 26 22:28:56.965300390 (58h53m6.68786343s) [noise=true , address=104.36.82.21:4495 , piece_id=C3W2IHXNWFVPSBMJEMITK3DGTLUI4FA3RNPOVLWU63VIARIUVBEQ, seg_pos={0 0}, upload=56504]: Uploading piece... Jul 26 22:28:56.965626534 (58h53m6.688189585s) [noise=true , address=79.127.205.246:4279 , piece_id=B7RQL2GX5X3QIFKGCLVILJ4UWRVGZZ4O2RATXOSA2IAHJQAMPRJA, seg_pos={0 0}, upload=56504]: Uploading piece... Jul 26 22:28:56.965966725 (58h53m6.688529775s) [noise=true , address=104.36.82.11:4127 , piece_id=KTQ7YDWYZU6V2MCK7KGHXPORZOTIMIK4T6MZG3TRNG3RAXPYM33A, seg_pos={0 0}, upload=56504]: Uploading piece... Jul 26 22:28:56.966296155 (58h53m6.688859195s) [noise=true , address=156.146.43.225:4448 , piece_id=OFJMXJZTE4YGTUDUJRKTJHCU55SVPD3NJ3TASBFJSB53MSTAFCCA, seg_pos={0 0}, upload=56504]: Uploading piece... Jul 26 22:28:56.966595999 (58h53m6.689159039s) [noise=true , address=45.91.116.5:4310 , piece_id=WRTZX3B3GVJDJFSLKEWTFZ7PS7COBFNLDCYC7LWDXW6NOOPWWUPQ, seg_pos={0 0}, upload=56504]: Uploading piece... Jul 26 22:28:56.966885022 (58h53m6.689448062s) [noise=true , address=64.49.13.140:4508 , piece_id=YL3ZLS4ICEFQJ4BTFNCS3SICPSURRYYVXEIDT2MS2LJ6ZCOTBBDQ, seg_pos={0 0}, upload=56504]: Uploading piece... Jul 26 22:28:56.967184024 (58h53m6.689747064s) [noise=true , address=79.127.205.245:4246 , piece_id=R6XDRBQNTBINAU4EPOEBEZB5HZQ7PKIAI4GC7FGVLTMH4XXAGU6A, seg_pos={0 0}, upload=56504]: Uploading piece... Jul 26 22:28:56.967467446 (58h53m6.690030486s) [noise=true , address=104.36.82.52:4086 , piece_id=TWQJ3O6ZO4WHCCQGC3HPEG6B3KPUGNXR66V4EZ57RTFG2ICYM3UQ, seg_pos={0 0}, upload=56504]: Uploading piece... Jul 26 22:28:56.967761479 (58h53m6.690324519s) [noise=true , address=104.36.82.12:4163 , piece_id=RB3PZABPFP7PEL6BOMD2CERMT663BKNKBJHENYN7X67RZMWSYW3A, seg_pos={0 0}, upload=56504]: Uploading piece... Jul 26 22:28:56.968043237 (58h53m6.690606267s) [noise=true , address=104.36.82.56:4237 , piece_id=BNJLYY2FYVXAVXYIEEJ3FCALQYCAGXJW22IEGDRIDXH44PM3M4GA, seg_pos={0 0}, upload=56504]: Uploading piece... Jul 26 22:28:57.993381822 (58h53m7.715944862s) [noise=true , address=156.146.43.225:4448 , piece_id=OFJMXJZTE4YGTUDUJRKTJHCU55SVPD3NJ3TASBFJSB53MSTAFCCA, seg_pos={0 0}, upload=56504]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jul 26 22:28:58.076784574 (58h53m7.799347614s) [upload=56504]: Next returned nil segment Jul 26 22:28:58.076805704 (58h53m7.799368744s) [upload=56504]: Waiting for error group managing segments... Jul 26 22:28:58.101272545 (58h53m7.823835595s) [noise=true , address=79.127.219.47:4069 , piece_id=KKQ4QHTKUEYCTCCUNMVVYERB42LZFO76PO2O3MXD6OWQ7FBBUPPA, seg_pos={0 0}, upload=56504]: Done uploading piece. err: ecclient: upload failed (node:18dK6RxbMxU4a46pW4YPiQW87xgpeoTb1ouqE9fzTdzTAFAa3W, address:79.127.219.47:4069): protocol: storage node overloaded, request limit: 2000 Jul 26 22:28:58.400836842 (58h53m8.123399882s) [noise=true , address=79.127.205.235:4670 , piece_id=SJNVNCJUEZDT6W5GRGMKSTSF2GROFBBMMQHKPJRFVXICE2SPX5GQ, seg_pos={0 0}, upload=56504]: Done uploading piece. err: <nil> Jul 26 22:28:58.547227485 (58h53m8.269790525s) [noise=true , address=64.49.13.138:4483 , piece_id=JIDRCH237ZLFDCJ5PO6LWXOBWMV7IKG2T4GWRHO4JRRIXNMVGIQQ, seg_pos={0 0}, upload=56504]: Done uploading piece. err: <nil> Jul 26 22:28:58.705941828 (58h53m8.428504868s) [noise=true , address=79.127.205.245:4246 , piece_id=R6XDRBQNTBINAU4EPOEBEZB5HZQ7PKIAI4GC7FGVLTMH4XXAGU6A, seg_pos={0 0}, upload=56504]: Done uploading piece. err: <nil> Jul 26 22:28:58.843172817 (58h53m8.565735857s) [noise=true , address=79.127.205.246:4279 , piece_id=B7RQL2GX5X3QIFKGCLVILJ4UWRVGZZ4O2RATXOSA2IAHJQAMPRJA, seg_pos={0 0}, upload=56504]: Done uploading piece. err: <nil> Jul 26 22:28:58.889924255 (58h53m8.612487295s) [noise=true , address=77.90.31.15:4717 , piece_id=PQQYSPZUMDOLKRULVRN7VHSGYIVNEK4DRYIHFTYZP3N6NIHYT3PA, seg_pos={0 0}, upload=56504]: Done uploading piece. err: <nil> Jul 26 22:28:58.959881641 (58h53m8.682444682s) [noise=true , address=45.91.116.5:4310 , piece_id=WRTZX3B3GVJDJFSLKEWTFZ7PS7COBFNLDCYC7LWDXW6NOOPWWUPQ, seg_pos={0 0}, upload=56504]: Done uploading piece. err: <nil> Jul 26 22:28:59.096226396 (58h53m8.818789436s) [noise=true , address=77.90.31.16:4752 , piece_id=VYROPZXWSN2GOISQLAHDS3DGDDA2VK4IE6R7EVQGW6HS3QG4CW7Q, seg_pos={0 0}, upload=56504]: Done uploading piece. err: <nil> Jul 26 22:28:59.170922129 (58h53m8.893485169s) [noise=true , address=104.36.82.55:4228 , piece_id=BYCVXH7OTUA2TG4UKRWJJOVBSLYFC2YZO4EOHLS36F2XE2VCEBRA, seg_pos={0 0}, upload=56504]: Done uploading piece. err: <nil> Jul 26 22:29:02.808475677 (58h53m12.531038727s) [noise=true , address=64.49.13.140:4508 , piece_id=YL3ZLS4ICEFQJ4BTFNCS3SICPSURRYYVXEIDT2MS2LJ6ZCOTBBDQ, seg_pos={0 0}, upload=56504]: Done uploading piece. err: <nil> Jul 26 22:29:55.877423284 (58h54m5.599986324s) [noise=true , address=104.36.82.56:4237 , piece_id=BNJLYY2FYVXAVXYIEEJ3FCALQYCAGXJW22IEGDRIDXH44PM3M4GA, seg_pos={0 0}, upload=56504]: Done uploading piece. err: ecclient: upload failed (node:1VzewCRbNpJv7kNwp9wruQuoVSaZFjA44qN33Ct7qm1GnfhtDj, address:104.36.82.56:4237): protocol: rpc: dial tcp 104.36.82.56:4237: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:29:55.877465234 (58h54m5.600028274s) [noise=true , address=104.36.82.40:4964 , piece_id=QTYX4F7KX6A3K4PABTILJJHDLP7O23SBW6PRW4ENMN2HBUI7XWYQ, seg_pos={0 0}, upload=56504]: Done uploading piece. err: ecclient: upload failed (node:12rFNeSLW6puVr6pd3M3cBtVE9uSkvbc2QVYERVZ5kGPvT3BvFR, address:104.36.82.40:4964): protocol: rpc: dial tcp 104.36.82.40:4964: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:29:55.877497536 (58h54m5.600060576s) [noise=true , address=104.36.82.2:4752 , piece_id=A7SOQHOA6Z7QBFQEKEGJISEALDNUUOJRZ6ZWQW3UFYK2SK2E2UPQ, seg_pos={0 0}, upload=56504]: Done uploading piece. err: ecclient: upload failed (node:12oRYWQwoe5JtDCqfSShqtyRWVQLjL4UtbokBgNpDAocBKv1vqL, address:104.36.82.2:4752): protocol: rpc: dial tcp 104.36.82.2:4752: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:29:55.877545969 (58h54m5.600109009s) [noise=true , address=104.36.82.52:4086 , piece_id=TWQJ3O6ZO4WHCCQGC3HPEG6B3KPUGNXR66V4EZ57RTFG2ICYM3UQ, seg_pos={0 0}, upload=56504]: Done uploading piece. err: ecclient: upload failed (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, address:104.36.82.52:4086): protocol: context deadline exceeded; context deadline exceeded Jul 26 22:29:55.877596345 (58h54m5.600159386s) [noise=true , address=104.36.82.11:4127 , piece_id=KTQ7YDWYZU6V2MCK7KGHXPORZOTIMIK4T6MZG3TRNG3RAXPYM33A, seg_pos={0 0}, upload=56504]: Done uploading piece. err: ecclient: upload failed (node:12Pxos2xcjHwcYyQnXeRL3PL72G9pXaAnBs8XnkVFkxegDRPrqp, address:104.36.82.11:4127): protocol: write tcp 188.166.225.170:57008->104.36.82.11:4127: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:29:55.877633616 (58h54m5.600196646s) [noise=true , address=104.36.82.21:4495 , piece_id=C3W2IHXNWFVPSBMJEMITK3DGTLUI4FA3RNPOVLWU63VIARIUVBEQ, seg_pos={0 0}, upload=56504]: Done uploading piece. err: ecclient: upload failed (node:1W4NyaW26WRURmwd6dBbD6aQqqVAMSPsfzKKHfbjtM4sWDMUP3, address:104.36.82.21:4495): protocol: write tcp 188.166.225.170:35670->104.36.82.21:4495: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:29:55.877673422 (58h54m5.600236502s) [noise=true , address=104.36.82.12:4163 , piece_id=RB3PZABPFP7PEL6BOMD2CERMT663BKNKBJHENYN7X67RZMWSYW3A, seg_pos={0 0}, upload=56504]: Done uploading piece. err: ecclient: upload failed (node:12w88UkGu5duYivJ6L6C9iJmTbnB63fDKiUUQprhwM3JshpXTdC, address:104.36.82.12:4163): protocol: write tcp 188.166.225.170:39606->104.36.82.12:4163: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:29:55.877709442 (58h54m5.600272522s) [seg_pos={0 0}, upload=56504]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 26 22:29:56.103141743 (58h54m5.825704783s) [upload=56504]: Done uploading segments... Jul 26 22:29:56.103175227 (58h54m5.825738267s) [upload=56504]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.227150441s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [X.XXX.X.XX.XXX.XXXXXX..XXXX..X.X...X........................] duration: 5.806602126s FAIL (2025-07-26 22:29:53 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== rite tcp 188.166.225.170:53646->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:53646->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53646->77.90.31.10:4519: use of closed network connection Jul 26 22:11:05.345059537 (58h35m15.067622577s) [seg_pos={0 0}, upload=56223]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:11:05.345121094 (58h35m15.067684124s) [upload=56223]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:11:05.345124551 (58h35m15.067687591s) [upload=56223]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:11:05.638665486 (58h35m15.361228526s) [upload=56223]: Done uploading segments... Jul 26 22:11:05.638683710 (58h35m15.36124674s) [upload=56223]: Upload finished. err: <nil> Jul 26 22:12:00.122995475 (58h36m9.845558505s) [upload=56238]: Starting upload Jul 26 22:12:00.123061241 (58h36m9.845624351s) [upload=56238]: Done starting upload Jul 26 22:12:00.123134752 (58h36m9.845697782s) [upload=56238]: Uploading segments... Jul 26 22:12:00.123493779 (58h36m9.846056819s) [upload=56238]: Got next segment. Inline: false Jul 26 22:12:00.123543474 (58h36m9.846106514s) [upload=56238]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:12:00.123548433 (58h36m9.846111473s) [upload=56238]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:12:00.355422881 (58h36m10.077985911s) [seg_pos={0 0}, upload=56238]: Begin upload segment... Jul 26 22:12:00.355465902 (58h36m10.078028942s) [seg_pos={0 0}, upload=56238]: Done begin upload segment. Jul 26 22:12:00.355487364 (58h36m10.078050404s) [noise=true , address=104.36.82.5:4837 , piece_id=WTBNXCZ3PWC5TQM4UWPTKT5QRXHHWQF6JVQQ7VQ4BXKLPEIZMIFQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.355513444 (58h36m10.078076474s) [noise=true , address=45.91.116.9:4263 , piece_id=H67DGXYB4V7UFZAIE4X4VRVWU53RIIQQOWOANN52TAZHFJNQT4IA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.355843715 (58h36m10.078406746s) [noise=true , address=45.91.116.8:4045 , piece_id=PRKLVJLWB37FDN52QBLNATG2SIWCXJCN7OR6EFLRZL74JZLC57VA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.355985106 (58h36m10.078548146s) [noise=true , address=77.90.31.19:4843 , piece_id=P3UAQQ2AO27TVMESOSBZUUXGX6QOBWZPXOMG4LIAUHTSBDN3YBJQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356118291 (58h36m10.078681331s) [noise=true , address=45.91.116.4:4939 , piece_id=XJUAUMJSYPFXTK7JIIJECLKWJOTWJQHYJHQ7TYPHMPBXRBS3HFPA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356442141 (58h36m10.079005181s) [noise=true , address=104.36.82.4:4802 , piece_id=KQ4ABYCX4T6VM477B56ETRPXTVFWDZXUTZEGPNAWIQ7ZV2XPYEIQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356714532 (58h36m10.079277572s) [noise=true , address=45.91.116.12:4225 , piece_id=E2D6FTGWZ22OFNFPRWBE335BANIUZIWVS433SMUC2LPMD6PTI6IA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356937989 (58h36m10.079501029s) [noise=true , address=45.91.116.13:4155 , piece_id=OXPGOXKCI4YOIKSIMPXVV6D74BBVMRPXXZKBBX2P7D4YQVR73VXQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357235789 (58h36m10.079798819s) [noise=true , address=77.90.31.12:4592 , piece_id=O3UI7JY3LQTABYKN3JKO5O4Z4RCLAX6HJZS4HQXJPGKDU3JNECTQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357472842 (58h36m10.080035882s) [noise=true , address=45.91.116.10:4082 , piece_id=J6PNAFSZBQCDVTAIQBHAMMN23AS7X3WLGV3YHVAKDU26RMWGKBLQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357542045 (58h36m10.080105085s) [noise=true , address=45.91.116.9:4262 , piece_id=TPC2GAHLX2R2B3ZZBTFYGWNMPYCX7KMZLNMNVM4GKQDOY6ADGUVQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357791683 (58h36m10.080354723s) [noise=true , address=45.91.116.5:4298 , piece_id=HURPSTTY5SXKPKJIHWPQ6VGKWQPIGSQONUMRQM6BREDGEA5V5ALA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358046780 (58h36m10.080609831s) [noise=true , address=104.36.82.15:4258 , piece_id=MJO75LPAAYCYPVPZYKBASP7ZN7Q3FI44W3XRN3CS5JAARSD32LXA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358143686 (58h36m10.080706716s) [noise=true , address=45.91.116.6:4973 , piece_id=E4B6O574VP2Y3J6PNPKTWAN7BFGSHIVQEPMTBDXQCDIORGOFMU6A, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358493164 (58h36m10.081056204s) [noise=true , address=104.36.82.12:4149 , piece_id=6ZXU4V2YOW3IXGT5XYX3Q7FT4U6YW7IVA72UPFHYUIUBWNPBDOIQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358579300 (58h36m10.08114233s) [noise=true , address=45.91.116.14:4190 , piece_id=PVZ2OCKX3JTUYT6KNUGIRHIYH5YVBSDMIZUOYSYSYBYB6AUNMOOA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:02.081979808 (58h36m11.804542868s) [upload=56238]: Next returned nil segment Jul 26 22:12:02.082008563 (58h36m11.804571643s) [upload=56238]: Waiting for error group managing segments... Jul 26 22:12:02.373463127 (58h36m12.096026167s) [noise=true , address=45.91.116.5:4298 , piece_id=HURPSTTY5SXKPKJIHWPQ6VGKWQPIGSQONUMRQM6BREDGEA5V5ALA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.391294504 (58h36m12.113857544s) [noise=true , address=77.90.31.12:4592 , piece_id=O3UI7JY3LQTABYKN3JKO5O4Z4RCLAX6HJZS4HQXJPGKDU3JNECTQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.392103922 (58h36m12.114666972s) [noise=true , address=45.91.116.9:4263 , piece_id=H67DGXYB4V7UFZAIE4X4VRVWU53RIIQQOWOANN52TAZHFJNQT4IA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.395004321 (58h36m12.117567361s) [noise=true , address=45.91.116.12:4225 , piece_id=E2D6FTGWZ22OFNFPRWBE335BANIUZIWVS433SMUC2LPMD6PTI6IA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.395744818 (58h36m12.118307858s) [noise=true , address=45.91.116.9:4262 , piece_id=TPC2GAHLX2R2B3ZZBTFYGWNMPYCX7KMZLNMNVM4GKQDOY6ADGUVQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.451177047 (58h36m12.173740097s) [noise=true , address=45.91.116.8:4045 , piece_id=PRKLVJLWB37FDN52QBLNATG2SIWCXJCN7OR6EFLRZL74JZLC57VA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.462789285 (58h36m12.185352325s) [noise=true , address=45.91.116.4:4939 , piece_id=XJUAUMJSYPFXTK7JIIJECLKWJOTWJQHYJHQ7TYPHMPBXRBS3HFPA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.477161623 (58h36m12.199724674s) [noise=true , address=45.91.116.13:4155 , piece_id=OXPGOXKCI4YOIKSIMPXVV6D74BBVMRPXXZKBBX2P7D4YQVR73VXQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.477354983 (58h36m12.199918023s) [noise=true , address=45.91.116.10:4082 , piece_id=J6PNAFSZBQCDVTAIQBHAMMN23AS7X3WLGV3YHVAKDU26RMWGKBLQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.482097287 (58h36m12.204660327s) [noise=true , address=45.91.116.6:4973 , piece_id=E4B6O574VP2Y3J6PNPKTWAN7BFGSHIVQEPMTBDXQCDIORGOFMU6A, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.522569814 (58h36m12.245132854s) [noise=true , address=77.90.31.19:4843 , piece_id=P3UAQQ2AO27TVMESOSBZUUXGX6QOBWZPXOMG4LIAUHTSBDN3YBJQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.537542521 (58h36m12.260105571s) [noise=true , address=45.91.116.14:4190 , piece_id=PVZ2OCKX3JTUYT6KNUGIRHIYH5YVBSDMIZUOYSYSYBYB6AUNMOOA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:29.482251815 (58h36m39.204814855s) [noise=true , address=104.36.82.5:4837 , piece_id=WTBNXCZ3PWC5TQM4UWPTKT5QRXHHWQF6JVQQ7VQ4BXKLPEIZMIFQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 188.166.225.170:38046->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482276953 (58h36m39.204839993s) [noise=true , address=104.36.82.4:4802 , piece_id=KQ4ABYCX4T6VM477B56ETRPXTVFWDZXUTZEGPNAWIQ7ZV2XPYEIQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482340875 (58h36m39.204903915s) [noise=true , address=104.36.82.15:4258 , piece_id=MJO75LPAAYCYPVPZYKBASP7ZN7Q3FI44W3XRN3CS5JAARSD32LXA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: write tcp 188.166.225.170:41258->104.36.82.15:4258: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482356565 (58h36m39.204919605s) [noise=true , address=104.36.82.12:4149 , piece_id=6ZXU4V2YOW3IXGT5XYX3Q7FT4U6YW7IVA72UPFHYUIUBWNPBDOIQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 188.166.225.170:39256->104.36.82.12:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482383747 (58h36m39.204946787s) [seg_pos={0 0}, upload=56238]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:12:29.709832457 (58h36m39.432395497s) [upload=56238]: Done uploading segments... Jul 26 22:12:29.709874738 (58h36m39.432437768s) [upload=56238]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.806622356s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.................................................X...X......] duration: 19.203415405s FAIL (2025-07-26 22:29:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 19.203436574s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............X..X...........X...............................] duration: 6.124318691s FAIL (2025-07-26 22:29:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.124335092s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 24 out of 60 is failed (threshold=3) [XX.XXXX.XX.XXXXX.X.XXXXX.XXX.....X.........................X] duration: 5.836356815s FAIL (2025-07-26 22:29:40 UTC) orbiter warning: digest mismatch: after b80432bf250cf8c010a58d5a57ae8eb2e733e016512b73b27f8bbe5c05d36821 != before 1ae5bfd23ae4b196a480bbedcce3ef7b0078f8ffef79859a6873ad6df52a9a98 Duration: 5.836383025s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..................................................X...X.....] duration: 54.205649251s FAIL (2025-07-26 22:29:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 54.205678989s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............X..X...........X..............................] duration: 5.947722716s FAIL (2025-07-26 22:29:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.947742954s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [XX.XXX.X.XX.XXX.XXXXXX..XXXX..X.X...X.......................] duration: 5.766731379s FAIL (2025-07-26 22:29:15 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== rite tcp 188.166.225.170:53646->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:53646->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53646->77.90.31.10:4519: use of closed network connection Jul 26 22:11:05.345059537 (58h35m15.067622577s) [seg_pos={0 0}, upload=56223]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:11:05.345121094 (58h35m15.067684124s) [upload=56223]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:11:05.345124551 (58h35m15.067687591s) [upload=56223]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:11:05.638665486 (58h35m15.361228526s) [upload=56223]: Done uploading segments... Jul 26 22:11:05.638683710 (58h35m15.36124674s) [upload=56223]: Upload finished. err: <nil> Jul 26 22:12:00.122995475 (58h36m9.845558505s) [upload=56238]: Starting upload Jul 26 22:12:00.123061241 (58h36m9.845624351s) [upload=56238]: Done starting upload Jul 26 22:12:00.123134752 (58h36m9.845697782s) [upload=56238]: Uploading segments... Jul 26 22:12:00.123493779 (58h36m9.846056819s) [upload=56238]: Got next segment. Inline: false Jul 26 22:12:00.123543474 (58h36m9.846106514s) [upload=56238]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:12:00.123548433 (58h36m9.846111473s) [upload=56238]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:12:00.355422881 (58h36m10.077985911s) [seg_pos={0 0}, upload=56238]: Begin upload segment... Jul 26 22:12:00.355465902 (58h36m10.078028942s) [seg_pos={0 0}, upload=56238]: Done begin upload segment. Jul 26 22:12:00.355487364 (58h36m10.078050404s) [noise=true , address=104.36.82.5:4837 , piece_id=WTBNXCZ3PWC5TQM4UWPTKT5QRXHHWQF6JVQQ7VQ4BXKLPEIZMIFQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.355513444 (58h36m10.078076474s) [noise=true , address=45.91.116.9:4263 , piece_id=H67DGXYB4V7UFZAIE4X4VRVWU53RIIQQOWOANN52TAZHFJNQT4IA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.355843715 (58h36m10.078406746s) [noise=true , address=45.91.116.8:4045 , piece_id=PRKLVJLWB37FDN52QBLNATG2SIWCXJCN7OR6EFLRZL74JZLC57VA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.355985106 (58h36m10.078548146s) [noise=true , address=77.90.31.19:4843 , piece_id=P3UAQQ2AO27TVMESOSBZUUXGX6QOBWZPXOMG4LIAUHTSBDN3YBJQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356118291 (58h36m10.078681331s) [noise=true , address=45.91.116.4:4939 , piece_id=XJUAUMJSYPFXTK7JIIJECLKWJOTWJQHYJHQ7TYPHMPBXRBS3HFPA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356442141 (58h36m10.079005181s) [noise=true , address=104.36.82.4:4802 , piece_id=KQ4ABYCX4T6VM477B56ETRPXTVFWDZXUTZEGPNAWIQ7ZV2XPYEIQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356714532 (58h36m10.079277572s) [noise=true , address=45.91.116.12:4225 , piece_id=E2D6FTGWZ22OFNFPRWBE335BANIUZIWVS433SMUC2LPMD6PTI6IA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356937989 (58h36m10.079501029s) [noise=true , address=45.91.116.13:4155 , piece_id=OXPGOXKCI4YOIKSIMPXVV6D74BBVMRPXXZKBBX2P7D4YQVR73VXQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357235789 (58h36m10.079798819s) [noise=true , address=77.90.31.12:4592 , piece_id=O3UI7JY3LQTABYKN3JKO5O4Z4RCLAX6HJZS4HQXJPGKDU3JNECTQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357472842 (58h36m10.080035882s) [noise=true , address=45.91.116.10:4082 , piece_id=J6PNAFSZBQCDVTAIQBHAMMN23AS7X3WLGV3YHVAKDU26RMWGKBLQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357542045 (58h36m10.080105085s) [noise=true , address=45.91.116.9:4262 , piece_id=TPC2GAHLX2R2B3ZZBTFYGWNMPYCX7KMZLNMNVM4GKQDOY6ADGUVQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357791683 (58h36m10.080354723s) [noise=true , address=45.91.116.5:4298 , piece_id=HURPSTTY5SXKPKJIHWPQ6VGKWQPIGSQONUMRQM6BREDGEA5V5ALA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358046780 (58h36m10.080609831s) [noise=true , address=104.36.82.15:4258 , piece_id=MJO75LPAAYCYPVPZYKBASP7ZN7Q3FI44W3XRN3CS5JAARSD32LXA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358143686 (58h36m10.080706716s) [noise=true , address=45.91.116.6:4973 , piece_id=E4B6O574VP2Y3J6PNPKTWAN7BFGSHIVQEPMTBDXQCDIORGOFMU6A, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358493164 (58h36m10.081056204s) [noise=true , address=104.36.82.12:4149 , piece_id=6ZXU4V2YOW3IXGT5XYX3Q7FT4U6YW7IVA72UPFHYUIUBWNPBDOIQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358579300 (58h36m10.08114233s) [noise=true , address=45.91.116.14:4190 , piece_id=PVZ2OCKX3JTUYT6KNUGIRHIYH5YVBSDMIZUOYSYSYBYB6AUNMOOA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:02.081979808 (58h36m11.804542868s) [upload=56238]: Next returned nil segment Jul 26 22:12:02.082008563 (58h36m11.804571643s) [upload=56238]: Waiting for error group managing segments... Jul 26 22:12:02.373463127 (58h36m12.096026167s) [noise=true , address=45.91.116.5:4298 , piece_id=HURPSTTY5SXKPKJIHWPQ6VGKWQPIGSQONUMRQM6BREDGEA5V5ALA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.391294504 (58h36m12.113857544s) [noise=true , address=77.90.31.12:4592 , piece_id=O3UI7JY3LQTABYKN3JKO5O4Z4RCLAX6HJZS4HQXJPGKDU3JNECTQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.392103922 (58h36m12.114666972s) [noise=true , address=45.91.116.9:4263 , piece_id=H67DGXYB4V7UFZAIE4X4VRVWU53RIIQQOWOANN52TAZHFJNQT4IA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.395004321 (58h36m12.117567361s) [noise=true , address=45.91.116.12:4225 , piece_id=E2D6FTGWZ22OFNFPRWBE335BANIUZIWVS433SMUC2LPMD6PTI6IA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.395744818 (58h36m12.118307858s) [noise=true , address=45.91.116.9:4262 , piece_id=TPC2GAHLX2R2B3ZZBTFYGWNMPYCX7KMZLNMNVM4GKQDOY6ADGUVQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.451177047 (58h36m12.173740097s) [noise=true , address=45.91.116.8:4045 , piece_id=PRKLVJLWB37FDN52QBLNATG2SIWCXJCN7OR6EFLRZL74JZLC57VA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.462789285 (58h36m12.185352325s) [noise=true , address=45.91.116.4:4939 , piece_id=XJUAUMJSYPFXTK7JIIJECLKWJOTWJQHYJHQ7TYPHMPBXRBS3HFPA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.477161623 (58h36m12.199724674s) [noise=true , address=45.91.116.13:4155 , piece_id=OXPGOXKCI4YOIKSIMPXVV6D74BBVMRPXXZKBBX2P7D4YQVR73VXQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.477354983 (58h36m12.199918023s) [noise=true , address=45.91.116.10:4082 , piece_id=J6PNAFSZBQCDVTAIQBHAMMN23AS7X3WLGV3YHVAKDU26RMWGKBLQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.482097287 (58h36m12.204660327s) [noise=true , address=45.91.116.6:4973 , piece_id=E4B6O574VP2Y3J6PNPKTWAN7BFGSHIVQEPMTBDXQCDIORGOFMU6A, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.522569814 (58h36m12.245132854s) [noise=true , address=77.90.31.19:4843 , piece_id=P3UAQQ2AO27TVMESOSBZUUXGX6QOBWZPXOMG4LIAUHTSBDN3YBJQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.537542521 (58h36m12.260105571s) [noise=true , address=45.91.116.14:4190 , piece_id=PVZ2OCKX3JTUYT6KNUGIRHIYH5YVBSDMIZUOYSYSYBYB6AUNMOOA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:29.482251815 (58h36m39.204814855s) [noise=true , address=104.36.82.5:4837 , piece_id=WTBNXCZ3PWC5TQM4UWPTKT5QRXHHWQF6JVQQ7VQ4BXKLPEIZMIFQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 188.166.225.170:38046->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482276953 (58h36m39.204839993s) [noise=true , address=104.36.82.4:4802 , piece_id=KQ4ABYCX4T6VM477B56ETRPXTVFWDZXUTZEGPNAWIQ7ZV2XPYEIQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482340875 (58h36m39.204903915s) [noise=true , address=104.36.82.15:4258 , piece_id=MJO75LPAAYCYPVPZYKBASP7ZN7Q3FI44W3XRN3CS5JAARSD32LXA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: write tcp 188.166.225.170:41258->104.36.82.15:4258: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482356565 (58h36m39.204919605s) [noise=true , address=104.36.82.12:4149 , piece_id=6ZXU4V2YOW3IXGT5XYX3Q7FT4U6YW7IVA72UPFHYUIUBWNPBDOIQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 188.166.225.170:39256->104.36.82.12:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482383747 (58h36m39.204946787s) [seg_pos={0 0}, upload=56238]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:12:29.709832457 (58h36m39.432395497s) [upload=56238]: Done uploading segments... Jul 26 22:12:29.709874738 (58h36m39.432437768s) [upload=56238]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.766758312s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 23 out of 60 is failed (threshold=3) [.XX.XXXX.XX.XXXXX.X.XXXXX.XXX.....X.........................] duration: 5.760468687s FAIL (2025-07-26 22:29:12 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.760498092s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [.XX.XXX.X.XX.XXX.XXXXXX..XXXX..X.X...X......................] duration: 5.800011836s FAIL (2025-07-26 22:28:49 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== rite tcp 188.166.225.170:53646->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:53646->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53646->77.90.31.10:4519: use of closed network connection Jul 26 22:11:05.345059537 (58h35m15.067622577s) [seg_pos={0 0}, upload=56223]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:11:05.345121094 (58h35m15.067684124s) [upload=56223]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:11:05.345124551 (58h35m15.067687591s) [upload=56223]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:11:05.638665486 (58h35m15.361228526s) [upload=56223]: Done uploading segments... Jul 26 22:11:05.638683710 (58h35m15.36124674s) [upload=56223]: Upload finished. err: <nil> Jul 26 22:12:00.122995475 (58h36m9.845558505s) [upload=56238]: Starting upload Jul 26 22:12:00.123061241 (58h36m9.845624351s) [upload=56238]: Done starting upload Jul 26 22:12:00.123134752 (58h36m9.845697782s) [upload=56238]: Uploading segments... Jul 26 22:12:00.123493779 (58h36m9.846056819s) [upload=56238]: Got next segment. Inline: false Jul 26 22:12:00.123543474 (58h36m9.846106514s) [upload=56238]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:12:00.123548433 (58h36m9.846111473s) [upload=56238]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:12:00.355422881 (58h36m10.077985911s) [seg_pos={0 0}, upload=56238]: Begin upload segment... Jul 26 22:12:00.355465902 (58h36m10.078028942s) [seg_pos={0 0}, upload=56238]: Done begin upload segment. Jul 26 22:12:00.355487364 (58h36m10.078050404s) [noise=true , address=104.36.82.5:4837 , piece_id=WTBNXCZ3PWC5TQM4UWPTKT5QRXHHWQF6JVQQ7VQ4BXKLPEIZMIFQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.355513444 (58h36m10.078076474s) [noise=true , address=45.91.116.9:4263 , piece_id=H67DGXYB4V7UFZAIE4X4VRVWU53RIIQQOWOANN52TAZHFJNQT4IA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.355843715 (58h36m10.078406746s) [noise=true , address=45.91.116.8:4045 , piece_id=PRKLVJLWB37FDN52QBLNATG2SIWCXJCN7OR6EFLRZL74JZLC57VA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.355985106 (58h36m10.078548146s) [noise=true , address=77.90.31.19:4843 , piece_id=P3UAQQ2AO27TVMESOSBZUUXGX6QOBWZPXOMG4LIAUHTSBDN3YBJQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356118291 (58h36m10.078681331s) [noise=true , address=45.91.116.4:4939 , piece_id=XJUAUMJSYPFXTK7JIIJECLKWJOTWJQHYJHQ7TYPHMPBXRBS3HFPA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356442141 (58h36m10.079005181s) [noise=true , address=104.36.82.4:4802 , piece_id=KQ4ABYCX4T6VM477B56ETRPXTVFWDZXUTZEGPNAWIQ7ZV2XPYEIQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356714532 (58h36m10.079277572s) [noise=true , address=45.91.116.12:4225 , piece_id=E2D6FTGWZ22OFNFPRWBE335BANIUZIWVS433SMUC2LPMD6PTI6IA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356937989 (58h36m10.079501029s) [noise=true , address=45.91.116.13:4155 , piece_id=OXPGOXKCI4YOIKSIMPXVV6D74BBVMRPXXZKBBX2P7D4YQVR73VXQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357235789 (58h36m10.079798819s) [noise=true , address=77.90.31.12:4592 , piece_id=O3UI7JY3LQTABYKN3JKO5O4Z4RCLAX6HJZS4HQXJPGKDU3JNECTQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357472842 (58h36m10.080035882s) [noise=true , address=45.91.116.10:4082 , piece_id=J6PNAFSZBQCDVTAIQBHAMMN23AS7X3WLGV3YHVAKDU26RMWGKBLQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357542045 (58h36m10.080105085s) [noise=true , address=45.91.116.9:4262 , piece_id=TPC2GAHLX2R2B3ZZBTFYGWNMPYCX7KMZLNMNVM4GKQDOY6ADGUVQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357791683 (58h36m10.080354723s) [noise=true , address=45.91.116.5:4298 , piece_id=HURPSTTY5SXKPKJIHWPQ6VGKWQPIGSQONUMRQM6BREDGEA5V5ALA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358046780 (58h36m10.080609831s) [noise=true , address=104.36.82.15:4258 , piece_id=MJO75LPAAYCYPVPZYKBASP7ZN7Q3FI44W3XRN3CS5JAARSD32LXA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358143686 (58h36m10.080706716s) [noise=true , address=45.91.116.6:4973 , piece_id=E4B6O574VP2Y3J6PNPKTWAN7BFGSHIVQEPMTBDXQCDIORGOFMU6A, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358493164 (58h36m10.081056204s) [noise=true , address=104.36.82.12:4149 , piece_id=6ZXU4V2YOW3IXGT5XYX3Q7FT4U6YW7IVA72UPFHYUIUBWNPBDOIQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358579300 (58h36m10.08114233s) [noise=true , address=45.91.116.14:4190 , piece_id=PVZ2OCKX3JTUYT6KNUGIRHIYH5YVBSDMIZUOYSYSYBYB6AUNMOOA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:02.081979808 (58h36m11.804542868s) [upload=56238]: Next returned nil segment Jul 26 22:12:02.082008563 (58h36m11.804571643s) [upload=56238]: Waiting for error group managing segments... Jul 26 22:12:02.373463127 (58h36m12.096026167s) [noise=true , address=45.91.116.5:4298 , piece_id=HURPSTTY5SXKPKJIHWPQ6VGKWQPIGSQONUMRQM6BREDGEA5V5ALA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.391294504 (58h36m12.113857544s) [noise=true , address=77.90.31.12:4592 , piece_id=O3UI7JY3LQTABYKN3JKO5O4Z4RCLAX6HJZS4HQXJPGKDU3JNECTQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.392103922 (58h36m12.114666972s) [noise=true , address=45.91.116.9:4263 , piece_id=H67DGXYB4V7UFZAIE4X4VRVWU53RIIQQOWOANN52TAZHFJNQT4IA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.395004321 (58h36m12.117567361s) [noise=true , address=45.91.116.12:4225 , piece_id=E2D6FTGWZ22OFNFPRWBE335BANIUZIWVS433SMUC2LPMD6PTI6IA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.395744818 (58h36m12.118307858s) [noise=true , address=45.91.116.9:4262 , piece_id=TPC2GAHLX2R2B3ZZBTFYGWNMPYCX7KMZLNMNVM4GKQDOY6ADGUVQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.451177047 (58h36m12.173740097s) [noise=true , address=45.91.116.8:4045 , piece_id=PRKLVJLWB37FDN52QBLNATG2SIWCXJCN7OR6EFLRZL74JZLC57VA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.462789285 (58h36m12.185352325s) [noise=true , address=45.91.116.4:4939 , piece_id=XJUAUMJSYPFXTK7JIIJECLKWJOTWJQHYJHQ7TYPHMPBXRBS3HFPA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.477161623 (58h36m12.199724674s) [noise=true , address=45.91.116.13:4155 , piece_id=OXPGOXKCI4YOIKSIMPXVV6D74BBVMRPXXZKBBX2P7D4YQVR73VXQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.477354983 (58h36m12.199918023s) [noise=true , address=45.91.116.10:4082 , piece_id=J6PNAFSZBQCDVTAIQBHAMMN23AS7X3WLGV3YHVAKDU26RMWGKBLQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.482097287 (58h36m12.204660327s) [noise=true , address=45.91.116.6:4973 , piece_id=E4B6O574VP2Y3J6PNPKTWAN7BFGSHIVQEPMTBDXQCDIORGOFMU6A, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.522569814 (58h36m12.245132854s) [noise=true , address=77.90.31.19:4843 , piece_id=P3UAQQ2AO27TVMESOSBZUUXGX6QOBWZPXOMG4LIAUHTSBDN3YBJQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.537542521 (58h36m12.260105571s) [noise=true , address=45.91.116.14:4190 , piece_id=PVZ2OCKX3JTUYT6KNUGIRHIYH5YVBSDMIZUOYSYSYBYB6AUNMOOA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:29.482251815 (58h36m39.204814855s) [noise=true , address=104.36.82.5:4837 , piece_id=WTBNXCZ3PWC5TQM4UWPTKT5QRXHHWQF6JVQQ7VQ4BXKLPEIZMIFQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 188.166.225.170:38046->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482276953 (58h36m39.204839993s) [noise=true , address=104.36.82.4:4802 , piece_id=KQ4ABYCX4T6VM477B56ETRPXTVFWDZXUTZEGPNAWIQ7ZV2XPYEIQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482340875 (58h36m39.204903915s) [noise=true , address=104.36.82.15:4258 , piece_id=MJO75LPAAYCYPVPZYKBASP7ZN7Q3FI44W3XRN3CS5JAARSD32LXA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: write tcp 188.166.225.170:41258->104.36.82.15:4258: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482356565 (58h36m39.204919605s) [noise=true , address=104.36.82.12:4149 , piece_id=6ZXU4V2YOW3IXGT5XYX3Q7FT4U6YW7IVA72UPFHYUIUBWNPBDOIQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 188.166.225.170:39256->104.36.82.12:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482383747 (58h36m39.204946787s) [seg_pos={0 0}, upload=56238]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:12:29.709832457 (58h36m39.432395497s) [upload=56238]: Done uploading segments... Jul 26 22:12:29.709874738 (58h36m39.432437768s) [upload=56238]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.80003542s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 58 out of 60 is failed (threshold=3) [..XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.236257654s FAIL (2025-07-26 22:28:49 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== ul 26 22:27:50.869867015 (58h52m0.592430085s) [noise=true , address=77.90.31.13:4658 , piece_id=2AU3RZH5WF2Z3JMMMH2JOPKWCVOH4A7JVQNFRCXOYBPP5DPWCLQQ, seg_pos={0 0}, upload=56488]: Uploading piece... Jul 26 22:27:50.869886833 (58h52m0.592449863s) [noise=true , address=45.91.116.5:4310 , piece_id=XPARGFNF4YRMT754GW2JECXZVGHPTEJBOHRB5MRQZY6CYQ3D5T2A, seg_pos={0 0}, upload=56488]: Uploading piece... Jul 26 22:27:50.870223958 (58h52m0.592786998s) [noise=true , address=64.49.13.140:4514 , piece_id=AHQNNCTIZUPG735Z3BFLOPWYADXVWNE37IW6YYIBCTVB6SBLSWHQ, seg_pos={0 0}, upload=56488]: Uploading piece... Jul 26 22:27:50.870353887 (58h52m0.592916927s) [noise=true , address=79.127.219.51:4212 , piece_id=IBNJFMBQOJRYKV2X7Q7USUHA3RRQ3QVTTVE52ZKLOCMEJJCK7ZNA, seg_pos={0 0}, upload=56488]: Uploading piece... Jul 26 22:27:50.870487162 (58h52m0.593050202s) [noise=true , address=104.36.82.52:4092 , piece_id=HPHC3LEWOX3DZJIKSTMO6GIJTBEC3QXQ4NUN4WCWMDQQPECHH5JA, seg_pos={0 0}, upload=56488]: Uploading piece... Jul 26 22:27:50.870663018 (58h52m0.593226048s) [noise=true , address=79.127.205.245:4239 , piece_id=T26WPX5IQG43ZZPARCSD5BBX3C6HJ2BNOGTMWZMV54MLBOCVOC2Q, seg_pos={0 0}, upload=56488]: Uploading piece... Jul 26 22:27:50.870743612 (58h52m0.593306652s) [noise=true , address=79.127.205.233:4633 , piece_id=XGU3SX6NJA3TQYM2IOZYR4HRJGOJRV77QKBIQKVTICR6JVWGTL2A, seg_pos={0 0}, upload=56488]: Uploading piece... Jul 26 22:27:50.870970006 (58h52m0.593533046s) [noise=true , address=104.36.82.41:4764 , piece_id=CTEFZGCU5OKYRRKHRKO7B4RV6UZNUONLZV5EJ4VBHDZBISD6BRPA, seg_pos={0 0}, upload=56488]: Uploading piece... Jul 26 22:27:50.871241846 (58h52m0.593804886s) [noise=true , address=104.36.82.44:4877 , piece_id=HHN7QY6FPUHUSCS3OU2YNCUPII2JX7LV2OQWASKFJ77FYTJCC33A, seg_pos={0 0}, upload=56488]: Uploading piece... Jul 26 22:27:50.871269038 (58h52m0.593832078s) [noise=true , address=45.91.116.14:4206 , piece_id=VDM3BYMJISBBMBOE2LB5IAQDX4NTORVV6Z3RGK2JLO52X32AL5NQ, seg_pos={0 0}, upload=56488]: Uploading piece... Jul 26 22:27:50.871465244 (58h52m0.594028274s) [noise=true , address=79.127.201.215:4348 , piece_id=ZXTYONFYCS5NBQAYUWNV67UJBHW2A3EE5RU2E3W4NLIJCYED4MIA, seg_pos={0 0}, upload=56488]: Uploading piece... Jul 26 22:27:50.871523555 (58h52m0.594086585s) [noise=true , address=45.91.116.9:4268 , piece_id=MJ2JUCHVVRF6NWP5XHWM5HD3MMWZTD3EAKWY4UPJQUK633ZHRWGQ, seg_pos={0 0}, upload=56488]: Uploading piece... Jul 26 22:27:50.871745780 (58h52m0.59430882s) [noise=true , address=79.127.205.246:4272 , piece_id=BC2PHVSSJ7L4FF5VAO2PSS5ODJLHDF2IK6BEPI4RDHDMWFL7L32A, seg_pos={0 0}, upload=56488]: Uploading piece... Jul 26 22:27:50.872499973 (58h52m0.595063013s) [noise=true , address=104.36.82.56:4236 , piece_id=AQCLTX3V2DWW2PDWFBT5FUAOUOF26L55776GLYJYYIHBG774EY2A, seg_pos={0 0}, upload=56488]: Uploading piece... Jul 26 22:27:50.872832289 (58h52m0.595395329s) [noise=true , address=104.36.82.11:4121 , piece_id=7VCBYQTPBPKANEDK6UOVCWRTTR46MTB6RSVAJDVLPMZVWIX4PPFA, seg_pos={0 0}, upload=56488]: Uploading piece... Jul 26 22:27:50.873084341 (58h52m0.595647381s) [noise=true , address=79.127.220.101:4048 , piece_id=B2LJBNIUIYSFSSY6PR3D3H2LF4MRDEDNCNXY3WZL7CPIBTGBUVBA, seg_pos={0 0}, upload=56488]: Uploading piece... Jul 26 22:27:50.873975466 (58h52m0.596538506s) [noise=true , address=79.127.205.234:4601 , piece_id=OMZARB6S6MPPMQDAQEPDCPLPLB7EV2JHT75D4KYYHYB3UNHD7L6A, seg_pos={0 0}, upload=56488]: Uploading piece... Jul 26 22:27:50.874218261 (58h52m0.596781301s) [noise=true , address=45.91.116.13:4175 , piece_id=KKMM673PDBXY7PNXSJ7Q4VR2AKC6IR7OWNXEAOHDEZLHU372QKWA, seg_pos={0 0}, upload=56488]: Uploading piece... Jul 26 22:27:51.889928304 (58h52m1.612491344s) [noise=true , address=79.127.219.51:4212 , piece_id=IBNJFMBQOJRYKV2X7Q7USUHA3RRQ3QVTTVE52ZKLOCMEJJCK7ZNA, seg_pos={0 0}, upload=56488]: Done uploading piece. err: ecclient: upload failed (node:17d1EWYAhx1xHZYRMKakqNkNpUAEGFG7HWcsfqGYvb3m1jyGFL, address:79.127.219.51:4212): protocol: storage node overloaded, request limit: 2000 Jul 26 22:27:51.959301834 (58h52m1.681864875s) [upload=56488]: Next returned nil segment Jul 26 22:27:51.959320330 (58h52m1.68188337s) [upload=56488]: Waiting for error group managing segments... Jul 26 22:27:52.270906427 (58h52m1.993469477s) [noise=true , address=79.127.205.234:4601 , piece_id=OMZARB6S6MPPMQDAQEPDCPLPLB7EV2JHT75D4KYYHYB3UNHD7L6A, seg_pos={0 0}, upload=56488]: Done uploading piece. err: <nil> Jul 26 22:27:52.289906810 (58h52m2.01246987s) [noise=true , address=79.127.205.245:4239 , piece_id=T26WPX5IQG43ZZPARCSD5BBX3C6HJ2BNOGTMWZMV54MLBOCVOC2Q, seg_pos={0 0}, upload=56488]: Done uploading piece. err: <nil> Jul 26 22:27:52.303764523 (58h52m2.026327563s) [noise=true , address=79.127.205.233:4633 , piece_id=XGU3SX6NJA3TQYM2IOZYR4HRJGOJRV77QKBIQKVTICR6JVWGTL2A, seg_pos={0 0}, upload=56488]: Done uploading piece. err: <nil> Jul 26 22:27:52.437401116 (58h52m2.159964156s) [noise=true , address=45.91.116.5:4310 , piece_id=XPARGFNF4YRMT754GW2JECXZVGHPTEJBOHRB5MRQZY6CYQ3D5T2A, seg_pos={0 0}, upload=56488]: Done uploading piece. err: ecclient: upload failed (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY, address:45.91.116.5:4310): protocol: storage node overloaded, request limit: 2000 Jul 26 22:27:52.927149726 (58h52m2.649712766s) [noise=true , address=45.91.116.13:4175 , piece_id=KKMM673PDBXY7PNXSJ7Q4VR2AKC6IR7OWNXEAOHDEZLHU372QKWA, seg_pos={0 0}, upload=56488]: Done uploading piece. err: <nil> Jul 26 22:27:53.676230777 (58h52m3.398793817s) [noise=true , address=79.127.205.246:4272 , piece_id=BC2PHVSSJ7L4FF5VAO2PSS5ODJLHDF2IK6BEPI4RDHDMWFL7L32A, seg_pos={0 0}, upload=56488]: Done uploading piece. err: <nil> Jul 26 22:27:56.067237589 (58h52m5.789800629s) [noise=true , address=45.91.116.9:4268 , piece_id=MJ2JUCHVVRF6NWP5XHWM5HD3MMWZTD3EAKWY4UPJQUK633ZHRWGQ, seg_pos={0 0}, upload=56488]: Done uploading piece. err: <nil> Jul 26 22:27:56.264261668 (58h52m5.986824708s) [noise=true , address=79.127.220.101:4048 , piece_id=B2LJBNIUIYSFSSY6PR3D3H2LF4MRDEDNCNXY3WZL7CPIBTGBUVBA, seg_pos={0 0}, upload=56488]: Done uploading piece. err: <nil> Jul 26 22:28:01.571030953 (58h52m11.293593993s) [noise=true , address=64.49.13.140:4514 , piece_id=AHQNNCTIZUPG735Z3BFLOPWYADXVWNE37IW6YYIBCTVB6SBLSWHQ, seg_pos={0 0}, upload=56488]: Done uploading piece. err: <nil> Jul 26 22:28:04.257033199 (58h52m13.979596239s) [noise=true , address=79.127.201.215:4348 , piece_id=ZXTYONFYCS5NBQAYUWNV67UJBHW2A3EE5RU2E3W4NLIJCYED4MIA, seg_pos={0 0}, upload=56488]: Done uploading piece. err: <nil> Jul 26 22:28:08.022112619 (58h52m17.744675679s) [noise=true , address=45.91.116.14:4206 , piece_id=VDM3BYMJISBBMBOE2LB5IAQDX4NTORVV6Z3RGK2JLO52X32AL5NQ, seg_pos={0 0}, upload=56488]: Done uploading piece. err: <nil> Jul 26 22:28:49.732005785 (58h52m59.454568855s) [noise=true , address=104.36.82.56:4236 , piece_id=AQCLTX3V2DWW2PDWFBT5FUAOUOF26L55776GLYJYYIHBG774EY2A, seg_pos={0 0}, upload=56488]: Done uploading piece. err: ecclient: upload failed (node:1pRLRqP5JEepqEKV9R3BUUhztQXnakq8QKMNjBZ1KR5TMMAXAb, address:104.36.82.56:4236): protocol: rpc: dial tcp 104.36.82.56:4236: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:28:49.732131465 (58h52m59.454694565s) [noise=true , address=77.90.31.13:4658 , piece_id=2AU3RZH5WF2Z3JMMMH2JOPKWCVOH4A7JVQNFRCXOYBPP5DPWCLQQ, seg_pos={0 0}, upload=56488]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: write tcp 188.166.225.170:59172->77.90.31.13:4658: use of closed network connection; write tcp 188.166.225.170:59172->77.90.31.13:4658: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59172->77.90.31.13:4658: use of closed network connection Jul 26 22:28:49.732231025 (58h52m59.454794106s) [noise=true , address=104.36.82.41:4764 , piece_id=CTEFZGCU5OKYRRKHRKO7B4RV6UZNUONLZV5EJ4VBHDZBISD6BRPA, seg_pos={0 0}, upload=56488]: Done uploading piece. err: ecclient: upload failed (node:127hafVDbun7FoG4JdcogBmRNT6rhL2QQbLDLstFnx7HCWyiC9N, address:104.36.82.41:4764): protocol: context deadline exceeded; context deadline exceeded Jul 26 22:28:49.732325456 (58h52m59.454888496s) [noise=true , address=104.36.82.52:4092 , piece_id=HPHC3LEWOX3DZJIKSTMO6GIJTBEC3QXQ4NUN4WCWMDQQPECHH5JA, seg_pos={0 0}, upload=56488]: Done uploading piece. err: ecclient: upload failed (node:12wYaUu5QpFU8T4mZt1iqeHqVJKJ5CyVW5vkeTWf1a8G3pLsx3P, address:104.36.82.52:4092): protocol: rpc: dial tcp 104.36.82.52:4092: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:28:49.732380761 (58h52m59.454943802s) [noise=true , address=104.36.82.44:4877 , piece_id=HHN7QY6FPUHUSCS3OU2YNCUPII2JX7LV2OQWASKFJ77FYTJCC33A, seg_pos={0 0}, upload=56488]: Done uploading piece. err: ecclient: upload failed (node:17TuauVrjKmehRpXqmK7bemU9V33GDZu9RGBs6fanXvJkV1gdH, address:104.36.82.44:4877): protocol: rpc: dial tcp 104.36.82.44:4877: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:28:49.732472778 (58h52m59.455035818s) [noise=true , address=104.36.82.11:4121 , piece_id=7VCBYQTPBPKANEDK6UOVCWRTTR46MTB6RSVAJDVLPMZVWIX4PPFA, seg_pos={0 0}, upload=56488]: Done uploading piece. err: ecclient: upload failed (node:1Fq1JNvrZLTFTQLVF8jzyavJ6dU1xX96FTGcyxfaHiPdCLu4pB, address:104.36.82.11:4121): protocol: write tcp 188.166.225.170:51614->104.36.82.11:4121: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:28:49.732500591 (58h52m59.455063661s) [seg_pos={0 0}, upload=56488]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 26 22:28:49.966073688 (58h52m59.688636738s) [upload=56488]: Done uploading segments... Jul 26 22:28:49.966108324 (58h52m59.688671364s) [upload=56488]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.236273114s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............X..X...........X.............................] duration: 6.235533482s FAIL (2025-07-26 22:28:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.235555054s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 24 out of 60 is failed (threshold=3) [X.XX.XXXX.XX.XXXXX.X.XXXXX.XXX.....X........................] duration: 5.797973162s FAIL (2025-07-26 22:28:44 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.797999773s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [................X..X...........X............................] duration: 5.840410707s FAIL (2025-07-26 22:28:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.840432116s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 25 out of 60 is failed (threshold=3) [XX.XX.XXXX.XX.XXXXX.X.XXXXX.XXX.....X.......................] duration: 6.000029091s FAIL (2025-07-26 22:28:18 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.000057606s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 57 out of 60 is failed (threshold=3) [...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.248864825s FAIL (2025-07-26 22:27:49 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== 184s) [noise=true , address=45.91.116.12:4229 , piece_id=U7RGYWB5663SUTYH26OUTFP6DN2FV63IMHAI7PTOAS3Z67UOFFPA, seg_pos={0 0}, upload=56472]: Uploading piece... Jul 26 22:26:50.351800216 (58h51m0.074363246s) [noise=true , address=104.36.82.56:4236 , piece_id=4ICDFMNAWC2BD6NROBVOUW4DXCJOVV5SJFGJ4IDKWATJWHT5S2HA, seg_pos={0 0}, upload=56472]: Uploading piece... Jul 26 22:26:50.352008165 (58h51m0.074571195s) [noise=true , address=104.36.82.40:4959 , piece_id=JBWBXH3HQVUL7EM6XAJDTWUOY6CH5NANCLX4GZK2O4JY5YHTLGHQ, seg_pos={0 0}, upload=56472]: Uploading piece... Jul 26 22:26:50.352194742 (58h51m0.074757772s) [noise=true , address=79.127.219.51:4201 , piece_id=QBCUGQYJHAXYJ3ATZQBYV523NPUOK3KX53FNATHAXKMC67A2V6SQ, seg_pos={0 0}, upload=56472]: Uploading piece... Jul 26 22:26:50.352401777 (58h51m0.074964817s) [noise=true , address=104.36.82.45:4909 , piece_id=APPBLGBM4EF6TRMXOQIK7YY6S6G3G3GLLZQJ2SK5LJELGHRAX5DQ, seg_pos={0 0}, upload=56472]: Uploading piece... Jul 26 22:26:50.352588365 (58h51m0.075151405s) [noise=true , address=79.127.220.101:4048 , piece_id=IRBI6RRNBO3VSZZPRK5YQTG7GPXIL3U633VD3ACUFXZ5QXBQN5IA, seg_pos={0 0}, upload=56472]: Uploading piece... Jul 26 22:26:50.352778729 (58h51m0.075341769s) [noise=true , address=104.36.82.52:4092 , piece_id=SST23QUQM4U3JOEPB5DIRWVZASBNVR2SWEBU5MWMQYDO6RBZTXOQ, seg_pos={0 0}, upload=56472]: Uploading piece... Jul 26 22:26:50.352960086 (58h51m0.075523126s) [noise=true , address=104.36.82.41:4791 , piece_id=HU4DPYQQL7MBMG2XDJD5BYPRIXATOIR3BHWU5TT4KOHJABPNOKUQ, seg_pos={0 0}, upload=56472]: Uploading piece... Jul 26 22:26:50.353148376 (58h51m0.075711416s) [noise=true , address=77.90.31.16:4760 , piece_id=BHED4ADOORXRJNTVBRQOQXTVMVJOGQ45AT2NEOT5CVICEK3EPOSQ, seg_pos={0 0}, upload=56472]: Uploading piece... Jul 26 22:26:50.353323581 (58h51m0.075886622s) [noise=true , address=104.36.82.4:4828 , piece_id=7XPURFL3MMWMNY5QLDYSJH6IXGYASW7PWFPOIN6WC3JRW23BHNGA, seg_pos={0 0}, upload=56472]: Uploading piece... Jul 26 22:26:50.353499137 (58h51m0.076062177s) [noise=true , address=79.127.201.215:4349 , piece_id=XVTFJVFHSW5JXNBX2QOJS7SZDV7VAPUTKDFC6LPDLEDMYNQQTKMA, seg_pos={0 0}, upload=56472]: Uploading piece... Jul 26 22:26:50.353682839 (58h51m0.076245879s) [noise=true , address=79.127.219.38:4013 , piece_id=UQSTLF2N7PI4XLYPJEAW2JRLBJYXG34WSRLH5ZYYBYBYRCWE66GQ, seg_pos={0 0}, upload=56472]: Uploading piece... Jul 26 22:26:50.353975027 (58h51m0.076538057s) [noise=true , address=45.91.116.14:4206 , piece_id=OORHO2ZN4EUVQNJWOETXVTG4WYRN4AMMSDYATXS6TSRJCNIGN4OQ, seg_pos={0 0}, upload=56472]: Uploading piece... Jul 26 22:26:50.354261135 (58h51m0.076824175s) [noise=true , address=104.36.82.41:4783 , piece_id=CA54TIKUKSLPSQTCFQTZEFVDFGPVZ5U45FJPBJHCI2ZI2ELTSLIA, seg_pos={0 0}, upload=56472]: Uploading piece... Jul 26 22:26:50.354515612 (58h51m0.077078652s) [noise=true , address=79.127.205.234:4620 , piece_id=PFQMYYDDZK5GKCYWJWNUUZHLZSJYMHJGOVNU2PX5H2IQKJ4FGFWQ, seg_pos={0 0}, upload=56472]: Uploading piece... Jul 26 22:26:50.354856704 (58h51m0.077419744s) [noise=true , address=64.49.13.138:4483 , piece_id=DBBO6SCFT33CSP46PVLPWHQDMHOEBFMBTFZNJ4U3WT676V5L7IZA, seg_pos={0 0}, upload=56472]: Uploading piece... Jul 26 22:26:50.355118075 (58h51m0.077681115s) [noise=true , address=104.36.82.5:4871 , piece_id=JAA4KOL57SW5PBYV2EQEASJ4534UVE7EVIW7QHOCHZN2IZAPEGFA, seg_pos={0 0}, upload=56472]: Uploading piece... Jul 26 22:26:51.457576369 (58h51m1.180139409s) [upload=56472]: Next returned nil segment Jul 26 22:26:51.457595976 (58h51m1.180159016s) [upload=56472]: Waiting for error group managing segments... Jul 26 22:26:51.753520679 (58h51m1.476083719s) [noise=true , address=79.127.205.234:4620 , piece_id=PFQMYYDDZK5GKCYWJWNUUZHLZSJYMHJGOVNU2PX5H2IQKJ4FGFWQ, seg_pos={0 0}, upload=56472]: Done uploading piece. err: <nil> Jul 26 22:26:52.352664007 (58h51m2.075227047s) [noise=true , address=79.127.219.38:4013 , piece_id=UQSTLF2N7PI4XLYPJEAW2JRLBJYXG34WSRLH5ZYYBYBYRCWE66GQ, seg_pos={0 0}, upload=56472]: Done uploading piece. err: <nil> Jul 26 22:26:52.361512838 (58h51m2.084075878s) [noise=true , address=77.90.31.18:4816 , piece_id=QBKSRTPIZY3KQEPEK5EALQ6JL4TDVK4DA4DM7YMFCMZWGOL3BXCQ, seg_pos={0 0}, upload=56472]: Done uploading piece. err: <nil> Jul 26 22:26:52.424609359 (58h51m2.147172389s) [noise=true , address=77.90.31.16:4760 , piece_id=BHED4ADOORXRJNTVBRQOQXTVMVJOGQ45AT2NEOT5CVICEK3EPOSQ, seg_pos={0 0}, upload=56472]: Done uploading piece. err: <nil> Jul 26 22:26:54.085384441 (58h51m3.807947481s) [noise=true , address=79.127.219.51:4201 , piece_id=QBCUGQYJHAXYJ3ATZQBYV523NPUOK3KX53FNATHAXKMC67A2V6SQ, seg_pos={0 0}, upload=56472]: Done uploading piece. err: <nil> Jul 26 22:26:54.616944478 (58h51m4.339507508s) [noise=true , address=45.91.116.14:4206 , piece_id=OORHO2ZN4EUVQNJWOETXVTG4WYRN4AMMSDYATXS6TSRJCNIGN4OQ, seg_pos={0 0}, upload=56472]: Done uploading piece. err: ecclient: upload failed (node:12d12pAMGhxjtLBE1tp9yEE7NMdfh8kcvsA8abhTnM8nbBB5LEK, address:45.91.116.14:4206): protocol: storage node overloaded, request limit: 2000 Jul 26 22:26:55.394030260 (58h51m5.1165933s) [noise=true , address=79.127.220.101:4048 , piece_id=IRBI6RRNBO3VSZZPRK5YQTG7GPXIL3U633VD3ACUFXZ5QXBQN5IA, seg_pos={0 0}, upload=56472]: Done uploading piece. err: <nil> Jul 26 22:26:58.714311691 (58h51m8.436874732s) [noise=true , address=64.49.13.138:4483 , piece_id=DBBO6SCFT33CSP46PVLPWHQDMHOEBFMBTFZNJ4U3WT676V5L7IZA, seg_pos={0 0}, upload=56472]: Done uploading piece. err: <nil> Jul 26 22:27:02.555231302 (58h51m12.277794352s) [noise=true , address=45.91.116.12:4229 , piece_id=U7RGYWB5663SUTYH26OUTFP6DN2FV63IMHAI7PTOAS3Z67UOFFPA, seg_pos={0 0}, upload=56472]: Done uploading piece. err: <nil> Jul 26 22:27:07.581951306 (58h51m17.304514346s) [noise=true , address=79.127.201.215:4349 , piece_id=XVTFJVFHSW5JXNBX2QOJS7SZDV7VAPUTKDFC6LPDLEDMYNQQTKMA, seg_pos={0 0}, upload=56472]: Done uploading piece. err: <nil> Jul 26 22:27:49.262026179 (58h51m58.984589209s) [noise=true , address=104.36.82.40:4959 , piece_id=JBWBXH3HQVUL7EM6XAJDTWUOY6CH5NANCLX4GZK2O4JY5YHTLGHQ, seg_pos={0 0}, upload=56472]: Done uploading piece. err: ecclient: upload failed (node:1NCN7sTw12KVDpD8zLjUxQki2MdMuDJyEnPSirG8ni5L2xi7aa, address:104.36.82.40:4959): protocol: rpc: dial tcp 104.36.82.40:4959: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:27:49.262067317 (58h51m58.984630357s) [noise=true , address=104.36.82.45:4909 , piece_id=APPBLGBM4EF6TRMXOQIK7YY6S6G3G3GLLZQJ2SK5LJELGHRAX5DQ, seg_pos={0 0}, upload=56472]: Done uploading piece. err: ecclient: upload failed (node:14Pqp5cbhmd44QnikAuD5TgsKFDR3gB2qtLuVSroaWVv4U2cDh, address:104.36.82.45:4909): protocol: rpc: dial tcp 104.36.82.45:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:27:49.262116201 (58h51m58.984679241s) [noise=true , address=104.36.82.5:4871 , piece_id=JAA4KOL57SW5PBYV2EQEASJ4534UVE7EVIW7QHOCHZN2IZAPEGFA, seg_pos={0 0}, upload=56472]: Done uploading piece. err: ecclient: upload failed (node:1NEVGFZtdgMqREnYBPuBbYqwGRxExenRwncwCBLDFhuwdFyWdb, address:104.36.82.5:4871): protocol: rpc: dial tcp 104.36.82.5:4871: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:27:49.262155285 (58h51m58.984718326s) [noise=true , address=104.36.82.52:4092 , piece_id=SST23QUQM4U3JOEPB5DIRWVZASBNVR2SWEBU5MWMQYDO6RBZTXOQ, seg_pos={0 0}, upload=56472]: Done uploading piece. err: ecclient: upload failed (node:12wYaUu5QpFU8T4mZt1iqeHqVJKJ5CyVW5vkeTWf1a8G3pLsx3P, address:104.36.82.52:4092): protocol: write tcp 188.166.225.170:42638->104.36.82.52:4092: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:27:49.262170425 (58h51m58.984733465s) [noise=true , address=104.36.82.4:4828 , piece_id=7XPURFL3MMWMNY5QLDYSJH6IXGYASW7PWFPOIN6WC3JRW23BHNGA, seg_pos={0 0}, upload=56472]: Done uploading piece. err: ecclient: upload failed (node:12MV49LDqJJn8S7nLwgVSpaYiGKa3BZJYbbge8oRuiXQfbNecaJ, address:104.36.82.4:4828): protocol: write tcp 188.166.225.170:44032->104.36.82.4:4828: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:27:49.262176326 (58h51m58.984739366s) [noise=true , address=104.36.82.41:4791 , piece_id=HU4DPYQQL7MBMG2XDJD5BYPRIXATOIR3BHWU5TT4KOHJABPNOKUQ, seg_pos={0 0}, upload=56472]: Done uploading piece. err: ecclient: upload failed (node:12LpFW1mfotWPqgQxyBSimChkttujx21L6rU2GRSwvqbZ7CVeLn, address:104.36.82.41:4791): protocol: write tcp 188.166.225.170:59620->104.36.82.41:4791: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:27:49.262182988 (58h51m58.984746029s) [noise=true , address=104.36.82.56:4236 , piece_id=4ICDFMNAWC2BD6NROBVOUW4DXCJOVV5SJFGJ4IDKWATJWHT5S2HA, seg_pos={0 0}, upload=56472]: Done uploading piece. err: ecclient: upload failed (node:1pRLRqP5JEepqEKV9R3BUUhztQXnakq8QKMNjBZ1KR5TMMAXAb, address:104.36.82.56:4236): protocol: rpc: dial tcp 104.36.82.56:4236: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:27:49.262192367 (58h51m58.984755407s) [noise=true , address=104.36.82.41:4783 , piece_id=CA54TIKUKSLPSQTCFQTZEFVDFGPVZ5U45FJPBJHCI2ZI2ELTSLIA, seg_pos={0 0}, upload=56472]: Done uploading piece. err: ecclient: upload failed (node:1sNMd8YrgaagKDoWivhbTmRc5TPM6BNiHscECBF2yhncKgkvz, address:104.36.82.41:4783): protocol: rpc: dial tcp 104.36.82.41:4783: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:27:49.262213958 (58h51m58.984776998s) [seg_pos={0 0}, upload=56472]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 26 22:27:49.509294169 (58h51m59.231857229s) [upload=56472]: Done uploading segments... Jul 26 22:27:49.509326341 (58h51m59.231889421s) [upload=56472]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.248891126s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...................................................X...X....] duration: 15.762332937s FAIL (2025-07-26 22:27:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 15.762346664s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [X.XX.XXX.X.XX.XXX.XXXXXX..XXXX..X.X...X.....................] duration: 5.872171758s FAIL (2025-07-26 22:27:35 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== rite tcp 188.166.225.170:53646->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:53646->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53646->77.90.31.10:4519: use of closed network connection Jul 26 22:11:05.345059537 (58h35m15.067622577s) [seg_pos={0 0}, upload=56223]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:11:05.345121094 (58h35m15.067684124s) [upload=56223]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:11:05.345124551 (58h35m15.067687591s) [upload=56223]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:11:05.638665486 (58h35m15.361228526s) [upload=56223]: Done uploading segments... Jul 26 22:11:05.638683710 (58h35m15.36124674s) [upload=56223]: Upload finished. err: <nil> Jul 26 22:12:00.122995475 (58h36m9.845558505s) [upload=56238]: Starting upload Jul 26 22:12:00.123061241 (58h36m9.845624351s) [upload=56238]: Done starting upload Jul 26 22:12:00.123134752 (58h36m9.845697782s) [upload=56238]: Uploading segments... Jul 26 22:12:00.123493779 (58h36m9.846056819s) [upload=56238]: Got next segment. Inline: false Jul 26 22:12:00.123543474 (58h36m9.846106514s) [upload=56238]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:12:00.123548433 (58h36m9.846111473s) [upload=56238]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:12:00.355422881 (58h36m10.077985911s) [seg_pos={0 0}, upload=56238]: Begin upload segment... Jul 26 22:12:00.355465902 (58h36m10.078028942s) [seg_pos={0 0}, upload=56238]: Done begin upload segment. Jul 26 22:12:00.355487364 (58h36m10.078050404s) [noise=true , address=104.36.82.5:4837 , piece_id=WTBNXCZ3PWC5TQM4UWPTKT5QRXHHWQF6JVQQ7VQ4BXKLPEIZMIFQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.355513444 (58h36m10.078076474s) [noise=true , address=45.91.116.9:4263 , piece_id=H67DGXYB4V7UFZAIE4X4VRVWU53RIIQQOWOANN52TAZHFJNQT4IA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.355843715 (58h36m10.078406746s) [noise=true , address=45.91.116.8:4045 , piece_id=PRKLVJLWB37FDN52QBLNATG2SIWCXJCN7OR6EFLRZL74JZLC57VA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.355985106 (58h36m10.078548146s) [noise=true , address=77.90.31.19:4843 , piece_id=P3UAQQ2AO27TVMESOSBZUUXGX6QOBWZPXOMG4LIAUHTSBDN3YBJQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356118291 (58h36m10.078681331s) [noise=true , address=45.91.116.4:4939 , piece_id=XJUAUMJSYPFXTK7JIIJECLKWJOTWJQHYJHQ7TYPHMPBXRBS3HFPA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356442141 (58h36m10.079005181s) [noise=true , address=104.36.82.4:4802 , piece_id=KQ4ABYCX4T6VM477B56ETRPXTVFWDZXUTZEGPNAWIQ7ZV2XPYEIQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356714532 (58h36m10.079277572s) [noise=true , address=45.91.116.12:4225 , piece_id=E2D6FTGWZ22OFNFPRWBE335BANIUZIWVS433SMUC2LPMD6PTI6IA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356937989 (58h36m10.079501029s) [noise=true , address=45.91.116.13:4155 , piece_id=OXPGOXKCI4YOIKSIMPXVV6D74BBVMRPXXZKBBX2P7D4YQVR73VXQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357235789 (58h36m10.079798819s) [noise=true , address=77.90.31.12:4592 , piece_id=O3UI7JY3LQTABYKN3JKO5O4Z4RCLAX6HJZS4HQXJPGKDU3JNECTQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357472842 (58h36m10.080035882s) [noise=true , address=45.91.116.10:4082 , piece_id=J6PNAFSZBQCDVTAIQBHAMMN23AS7X3WLGV3YHVAKDU26RMWGKBLQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357542045 (58h36m10.080105085s) [noise=true , address=45.91.116.9:4262 , piece_id=TPC2GAHLX2R2B3ZZBTFYGWNMPYCX7KMZLNMNVM4GKQDOY6ADGUVQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357791683 (58h36m10.080354723s) [noise=true , address=45.91.116.5:4298 , piece_id=HURPSTTY5SXKPKJIHWPQ6VGKWQPIGSQONUMRQM6BREDGEA5V5ALA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358046780 (58h36m10.080609831s) [noise=true , address=104.36.82.15:4258 , piece_id=MJO75LPAAYCYPVPZYKBASP7ZN7Q3FI44W3XRN3CS5JAARSD32LXA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358143686 (58h36m10.080706716s) [noise=true , address=45.91.116.6:4973 , piece_id=E4B6O574VP2Y3J6PNPKTWAN7BFGSHIVQEPMTBDXQCDIORGOFMU6A, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358493164 (58h36m10.081056204s) [noise=true , address=104.36.82.12:4149 , piece_id=6ZXU4V2YOW3IXGT5XYX3Q7FT4U6YW7IVA72UPFHYUIUBWNPBDOIQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358579300 (58h36m10.08114233s) [noise=true , address=45.91.116.14:4190 , piece_id=PVZ2OCKX3JTUYT6KNUGIRHIYH5YVBSDMIZUOYSYSYBYB6AUNMOOA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:02.081979808 (58h36m11.804542868s) [upload=56238]: Next returned nil segment Jul 26 22:12:02.082008563 (58h36m11.804571643s) [upload=56238]: Waiting for error group managing segments... Jul 26 22:12:02.373463127 (58h36m12.096026167s) [noise=true , address=45.91.116.5:4298 , piece_id=HURPSTTY5SXKPKJIHWPQ6VGKWQPIGSQONUMRQM6BREDGEA5V5ALA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.391294504 (58h36m12.113857544s) [noise=true , address=77.90.31.12:4592 , piece_id=O3UI7JY3LQTABYKN3JKO5O4Z4RCLAX6HJZS4HQXJPGKDU3JNECTQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.392103922 (58h36m12.114666972s) [noise=true , address=45.91.116.9:4263 , piece_id=H67DGXYB4V7UFZAIE4X4VRVWU53RIIQQOWOANN52TAZHFJNQT4IA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.395004321 (58h36m12.117567361s) [noise=true , address=45.91.116.12:4225 , piece_id=E2D6FTGWZ22OFNFPRWBE335BANIUZIWVS433SMUC2LPMD6PTI6IA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.395744818 (58h36m12.118307858s) [noise=true , address=45.91.116.9:4262 , piece_id=TPC2GAHLX2R2B3ZZBTFYGWNMPYCX7KMZLNMNVM4GKQDOY6ADGUVQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.451177047 (58h36m12.173740097s) [noise=true , address=45.91.116.8:4045 , piece_id=PRKLVJLWB37FDN52QBLNATG2SIWCXJCN7OR6EFLRZL74JZLC57VA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.462789285 (58h36m12.185352325s) [noise=true , address=45.91.116.4:4939 , piece_id=XJUAUMJSYPFXTK7JIIJECLKWJOTWJQHYJHQ7TYPHMPBXRBS3HFPA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.477161623 (58h36m12.199724674s) [noise=true , address=45.91.116.13:4155 , piece_id=OXPGOXKCI4YOIKSIMPXVV6D74BBVMRPXXZKBBX2P7D4YQVR73VXQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.477354983 (58h36m12.199918023s) [noise=true , address=45.91.116.10:4082 , piece_id=J6PNAFSZBQCDVTAIQBHAMMN23AS7X3WLGV3YHVAKDU26RMWGKBLQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.482097287 (58h36m12.204660327s) [noise=true , address=45.91.116.6:4973 , piece_id=E4B6O574VP2Y3J6PNPKTWAN7BFGSHIVQEPMTBDXQCDIORGOFMU6A, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.522569814 (58h36m12.245132854s) [noise=true , address=77.90.31.19:4843 , piece_id=P3UAQQ2AO27TVMESOSBZUUXGX6QOBWZPXOMG4LIAUHTSBDN3YBJQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.537542521 (58h36m12.260105571s) [noise=true , address=45.91.116.14:4190 , piece_id=PVZ2OCKX3JTUYT6KNUGIRHIYH5YVBSDMIZUOYSYSYBYB6AUNMOOA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:29.482251815 (58h36m39.204814855s) [noise=true , address=104.36.82.5:4837 , piece_id=WTBNXCZ3PWC5TQM4UWPTKT5QRXHHWQF6JVQQ7VQ4BXKLPEIZMIFQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 188.166.225.170:38046->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482276953 (58h36m39.204839993s) [noise=true , address=104.36.82.4:4802 , piece_id=KQ4ABYCX4T6VM477B56ETRPXTVFWDZXUTZEGPNAWIQ7ZV2XPYEIQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482340875 (58h36m39.204903915s) [noise=true , address=104.36.82.15:4258 , piece_id=MJO75LPAAYCYPVPZYKBASP7ZN7Q3FI44W3XRN3CS5JAARSD32LXA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: write tcp 188.166.225.170:41258->104.36.82.15:4258: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482356565 (58h36m39.204919605s) [noise=true , address=104.36.82.12:4149 , piece_id=6ZXU4V2YOW3IXGT5XYX3Q7FT4U6YW7IVA72UPFHYUIUBWNPBDOIQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 188.166.225.170:39256->104.36.82.12:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482383747 (58h36m39.204946787s) [seg_pos={0 0}, upload=56238]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:12:29.709832457 (58h36m39.432395497s) [upload=56238]: Done uploading segments... Jul 26 22:12:29.709874738 (58h36m39.432437768s) [upload=56238]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.872198801s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [....................................................X...X...] duration: 52.376389879s FAIL (2025-07-26 22:27:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 52.376410398s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 25 out of 60 is failed (threshold=3) [.XX.XX.XXXX.XX.XXXXX.X.XXXXX.XXX.....X......................] duration: 5.731170774s FAIL (2025-07-26 22:27:18 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.731200912s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................X..X...........X...........................] duration: 6.017649915s FAIL (2025-07-26 22:27:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.017664853s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 56 out of 60 is failed (threshold=3) [....XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.237755229s FAIL (2025-07-26 22:26:35 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== {0 0}, upload=56455]: Done begin upload segment. Jul 26 22:25:36.684315636 (58h49m46.406878687s) [noise=true , address=104.36.82.13:4197 , piece_id=NOIXGZXAX2LNCWS2KQJX6H62QCJHKU2CTNA2BVL3COALLY5H6GYQ, seg_pos={0 0}, upload=56455]: Uploading piece... Jul 26 22:25:36.684397213 (58h49m46.406960263s) [noise=true , address=104.36.82.55:4228 , piece_id=CY2JTITPEAS4JJ2NICELG4R2KQPWFT2TWZV4QMD6OQITPPTA72HQ, seg_pos={0 0}, upload=56455]: Uploading piece... Jul 26 22:25:36.684501501 (58h49m46.407064612s) [noise=true , address=79.127.201.215:4345 , piece_id=ZTPLIZYFOWLH7V3PZLZTFV6S6IRS3VBW33MDKSN2GMJMMSWAFXGA, seg_pos={0 0}, upload=56455]: Uploading piece... Jul 26 22:25:36.685326099 (58h49m46.407889149s) [noise=true , address=104.36.82.44:4894 , piece_id=TRVF6RC2HQDGQHVU3ZNYEVOI4YYR6W7VZOUR53LZKHCRJAKM4N7Q, seg_pos={0 0}, upload=56455]: Uploading piece... Jul 26 22:25:36.685338403 (58h49m46.407901463s) [noise=true , address=79.127.219.38:4007 , piece_id=DFUCQX2IRACG7NXOJKFVSWWFFS36WH5LHWFIXQRDGQE2NYHLHPNQ, seg_pos={0 0}, upload=56455]: Uploading piece... Jul 26 22:25:36.685976073 (58h49m46.408539143s) [noise=true , address=79.127.219.51:4210 , piece_id=FCEFCE32PKZQLMHCZDYFKDRODJE2ZG7VL5C5HKG5DSPS36IGVWJQ, seg_pos={0 0}, upload=56455]: Uploading piece... Jul 26 22:25:36.685994298 (58h49m46.408557348s) [noise=true , address=104.36.82.16:4296 , piece_id=Q3FEWC3EM267MZQJL3Y26UXJNFKQ55VANKAJUJ6KWEZIECUEVOCQ, seg_pos={0 0}, upload=56455]: Uploading piece... Jul 26 22:25:36.686545012 (58h49m46.409108042s) [noise=true , address=77.90.31.15:4723 , piece_id=Q3LLTPYHQ7BBBGJWSHAWGSSLD2UUJWQSFTZVXUJPS54IV7W67PZA, seg_pos={0 0}, upload=56455]: Uploading piece... Jul 26 22:25:36.686561282 (58h49m46.409124322s) [noise=true , address=79.127.205.235:4670 , piece_id=KJ63UANGT5GRZGFVS27YCNDR5PIWFZN2GAXUGPQDYR2UAPBZLVGA, seg_pos={0 0}, upload=56455]: Uploading piece... Jul 26 22:25:36.687096798 (58h49m46.409659868s) [noise=true , address=79.127.205.234:4614 , piece_id=7OHKUYF4QYLGH4ZNIVDOJCEI4MWIIOSVPW6KYEFEOYN4CHLPFCXA, seg_pos={0 0}, upload=56455]: Uploading piece... Jul 26 22:25:36.687107257 (58h49m46.409670307s) [noise=true , address=156.146.43.225:4448 , piece_id=2HKLPFSEHB2KU5LNEIUW5O4P7IRXWRXBCXBITBQN5WBDPCWYVVAA, seg_pos={0 0}, upload=56455]: Uploading piece... Jul 26 22:25:36.687173595 (58h49m46.409736635s) [noise=true , address=79.127.205.245:4247 , piece_id=AZD4Q7EL6L7GTPPY64IJWAA4SWEG5ONAIDLFGFQVB7DODGGE7JKA, seg_pos={0 0}, upload=56455]: Uploading piece... Jul 26 22:25:36.687617844 (58h49m46.410180885s) [noise=true , address=45.91.116.10:4100 , piece_id=4K4TSBYK55LIBXK2OOVZTFQLNKGKQLUTYBNSBXKVLXKPT3FGWFRA, seg_pos={0 0}, upload=56455]: Uploading piece... Jul 26 22:25:36.688034642 (58h49m46.410597692s) [noise=true , address=64.49.13.169:4716 , piece_id=23B6D4S6XQULQNMBTGOR47YFS6JNDVYXRT3RXZ4BXQ43VJLONNIQ, seg_pos={0 0}, upload=56455]: Uploading piece... Jul 26 22:25:36.688431200 (58h49m46.41099424s) [noise=true , address=45.91.116.5:4310 , piece_id=IOCFK2W6LDLWHAZMS3DZA3S4RJGKB3JO3PXWOC6FSFGFNTNXZ2VQ, seg_pos={0 0}, upload=56455]: Uploading piece... Jul 26 22:25:36.689594236 (58h49m46.412157276s) [noise=true , address=104.36.82.56:4229 , piece_id=7BH4O7G3W3UALGQI5SOGGVTYCNFCR2NOBFS7Q6DSTK33GDCDADYA, seg_pos={0 0}, upload=56455]: Uploading piece... Jul 26 22:25:36.691811808 (58h49m46.414374848s) [noise=true , address=104.36.82.52:4089 , piece_id=WWPUNHODX46YLDFHTMXOKDFVZZMAH2PANNPPAAG23UBIBSSBWKGA, seg_pos={0 0}, upload=56455]: Uploading piece... Jul 26 22:25:36.692925589 (58h49m46.415488639s) [noise=true , address=45.91.116.11:4146 , piece_id=AOWRR6ICJRRKWGZX7ZDIJHMDKFW27NTGX2ID2HGW2NJHIAPBAZYQ, seg_pos={0 0}, upload=56455]: Uploading piece... Jul 26 22:25:37.697618886 (58h49m47.420181916s) [noise=true , address=64.49.13.169:4716 , piece_id=23B6D4S6XQULQNMBTGOR47YFS6JNDVYXRT3RXZ4BXQ43VJLONNIQ, seg_pos={0 0}, upload=56455]: Done uploading piece. err: ecclient: upload failed (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN, address:64.49.13.169:4716): protocol: storage node overloaded, request limit: 2000 Jul 26 22:25:37.795507083 (58h49m47.518070123s) [upload=56455]: Next returned nil segment Jul 26 22:25:37.795529676 (58h49m47.518092716s) [upload=56455]: Waiting for error group managing segments... Jul 26 22:25:38.086980268 (58h49m47.809543308s) [noise=true , address=79.127.205.235:4670 , piece_id=KJ63UANGT5GRZGFVS27YCNDR5PIWFZN2GAXUGPQDYR2UAPBZLVGA, seg_pos={0 0}, upload=56455]: Done uploading piece. err: <nil> Jul 26 22:25:38.116698296 (58h49m47.839261326s) [noise=true , address=79.127.205.245:4247 , piece_id=AZD4Q7EL6L7GTPPY64IJWAA4SWEG5ONAIDLFGFQVB7DODGGE7JKA, seg_pos={0 0}, upload=56455]: Done uploading piece. err: <nil> Jul 26 22:25:38.638920417 (58h49m48.361483447s) [noise=true , address=79.127.219.38:4007 , piece_id=DFUCQX2IRACG7NXOJKFVSWWFFS36WH5LHWFIXQRDGQE2NYHLHPNQ, seg_pos={0 0}, upload=56455]: Done uploading piece. err: <nil> Jul 26 22:25:38.646342226 (58h49m48.368905256s) [noise=true , address=77.90.31.15:4723 , piece_id=Q3LLTPYHQ7BBBGJWSHAWGSSLD2UUJWQSFTZVXUJPS54IV7W67PZA, seg_pos={0 0}, upload=56455]: Done uploading piece. err: <nil> Jul 26 22:25:38.724635782 (58h49m48.447198822s) [noise=true , address=45.91.116.11:4146 , piece_id=AOWRR6ICJRRKWGZX7ZDIJHMDKFW27NTGX2ID2HGW2NJHIAPBAZYQ, seg_pos={0 0}, upload=56455]: Done uploading piece. err: <nil> Jul 26 22:25:38.746986645 (58h49m48.469549675s) [noise=true , address=79.127.205.234:4614 , piece_id=7OHKUYF4QYLGH4ZNIVDOJCEI4MWIIOSVPW6KYEFEOYN4CHLPFCXA, seg_pos={0 0}, upload=56455]: Done uploading piece. err: <nil> Jul 26 22:25:38.907754367 (58h49m48.630317407s) [noise=true , address=156.146.43.225:4448 , piece_id=2HKLPFSEHB2KU5LNEIUW5O4P7IRXWRXBCXBITBQN5WBDPCWYVVAA, seg_pos={0 0}, upload=56455]: Done uploading piece. err: <nil> Jul 26 22:25:39.177249513 (58h49m48.899812553s) [noise=true , address=79.127.219.51:4210 , piece_id=FCEFCE32PKZQLMHCZDYFKDRODJE2ZG7VL5C5HKG5DSPS36IGVWJQ, seg_pos={0 0}, upload=56455]: Done uploading piece. err: <nil> Jul 26 22:25:39.899951776 (58h49m49.622514816s) [noise=true , address=45.91.116.10:4100 , piece_id=4K4TSBYK55LIBXK2OOVZTFQLNKGKQLUTYBNSBXKVLXKPT3FGWFRA, seg_pos={0 0}, upload=56455]: Done uploading piece. err: ecclient: upload failed (node:12PQzzS3ZUKXCpiPHqTokRAC9MUuU8QQp7fJTDydmYehhs21YBn, address:45.91.116.10:4100): protocol: storage node overloaded, request limit: 2000 Jul 26 22:25:40.449403955 (58h49m50.171966995s) [noise=true , address=104.36.82.55:4228 , piece_id=CY2JTITPEAS4JJ2NICELG4R2KQPWFT2TWZV4QMD6OQITPPTA72HQ, seg_pos={0 0}, upload=56455]: Done uploading piece. err: ecclient: upload failed (node:12NBsd33JtAwPdcbD8vBzZUYpjiqZpRzs8y2QrYJFE245pd8T5X, address:104.36.82.55:4228): protocol: storage node overloaded, request limit: 2000 Jul 26 22:25:45.097929801 (58h49m54.820492841s) [noise=true , address=45.91.116.5:4310 , piece_id=IOCFK2W6LDLWHAZMS3DZA3S4RJGKB3JO3PXWOC6FSFGFNTNXZ2VQ, seg_pos={0 0}, upload=56455]: Done uploading piece. err: ecclient: upload failed (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY, address:45.91.116.5:4310): protocol: storage node overloaded, request limit: 2000 Jul 26 22:25:48.171091747 (58h49m57.893654787s) [noise=true , address=79.127.201.215:4345 , piece_id=ZTPLIZYFOWLH7V3PZLZTFV6S6IRS3VBW33MDKSN2GMJMMSWAFXGA, seg_pos={0 0}, upload=56455]: Done uploading piece. err: <nil> Jul 26 22:26:35.583781523 (58h50m45.306344564s) [noise=true , address=104.36.82.13:4197 , piece_id=NOIXGZXAX2LNCWS2KQJX6H62QCJHKU2CTNA2BVL3COALLY5H6GYQ, seg_pos={0 0}, upload=56455]: Done uploading piece. err: ecclient: upload failed (node:1cB89C5xnya9WGD9dGBsSouqkZH3nLqytrg9Qo4UH2kdFonTjM, address:104.36.82.13:4197): protocol: rpc: dial tcp 104.36.82.13:4197: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:26:35.583831199 (58h50m45.306394239s) [noise=true , address=104.36.82.44:4894 , piece_id=TRVF6RC2HQDGQHVU3ZNYEVOI4YYR6W7VZOUR53LZKHCRJAKM4N7Q, seg_pos={0 0}, upload=56455]: Done uploading piece. err: ecclient: upload failed (node:12KWNyruYhmSJVvgUF7fBpGf4H7eYmtAeEXToYjd2dcEPcj1E44, address:104.36.82.44:4894): protocol: rpc: dial tcp 104.36.82.44:4894: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:26:35.583864011 (58h50m45.306427051s) [noise=true , address=104.36.82.16:4296 , piece_id=Q3FEWC3EM267MZQJL3Y26UXJNFKQ55VANKAJUJ6KWEZIECUEVOCQ, seg_pos={0 0}, upload=56455]: Done uploading piece. err: ecclient: upload failed (node:12cKhDbA4ptwux2LYNPNAJ7Qh8jECsGWd1KWaK6T6pD38bJWdt1, address:104.36.82.16:4296): protocol: rpc: dial tcp 104.36.82.16:4296: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:26:35.584004841 (58h50m45.306567871s) [noise=true , address=104.36.82.52:4089 , piece_id=WWPUNHODX46YLDFHTMXOKDFVZZMAH2PANNPPAAG23UBIBSSBWKGA, seg_pos={0 0}, upload=56455]: Done uploading piece. err: ecclient: upload failed (node:12L2DTyJJ3v7KRjA424PVwQn9mpeq5Pq4ZTCerhrStgqgc3cVFk, address:104.36.82.52:4089): protocol: write tcp 188.166.225.170:35622->104.36.82.52:4089: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:26:35.584265409 (58h50m45.306828449s) [noise=true , address=104.36.82.56:4229 , piece_id=7BH4O7G3W3UALGQI5SOGGVTYCNFCR2NOBFS7Q6DSTK33GDCDADYA, seg_pos={0 0}, upload=56455]: Done uploading piece. err: ecclient: upload failed (node:12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka, address:104.36.82.56:4229): protocol: context deadline exceeded; context deadline exceeded Jul 26 22:26:35.584307740 (58h50m45.30687077s) [seg_pos={0 0}, upload=56455]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 26 22:26:35.819675320 (58h50m45.54223836s) [upload=56455]: Done uploading segments... Jul 26 22:26:35.819709315 (58h50m45.542272355s) [upload=56455]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.237773456s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 25 out of 60 is failed (threshold=3) [..XX.XX.XXXX.XX.XXXXX.X.XXXXX.XXX.....X.....................] duration: 5.769977885s FAIL (2025-07-26 22:26:34 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.770004969s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................X..X...........X..........................] duration: 6.609440886s FAIL (2025-07-26 22:26:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.609462067s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [XX.XX.XXX.X.XX.XXX.XXXXXX..XXXX..X.X...X....................] duration: 6.724751872s FAIL (2025-07-26 22:26:26 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== rite tcp 188.166.225.170:53646->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:53646->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53646->77.90.31.10:4519: use of closed network connection Jul 26 22:11:05.345059537 (58h35m15.067622577s) [seg_pos={0 0}, upload=56223]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:11:05.345121094 (58h35m15.067684124s) [upload=56223]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:11:05.345124551 (58h35m15.067687591s) [upload=56223]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:11:05.638665486 (58h35m15.361228526s) [upload=56223]: Done uploading segments... Jul 26 22:11:05.638683710 (58h35m15.36124674s) [upload=56223]: Upload finished. err: <nil> Jul 26 22:12:00.122995475 (58h36m9.845558505s) [upload=56238]: Starting upload Jul 26 22:12:00.123061241 (58h36m9.845624351s) [upload=56238]: Done starting upload Jul 26 22:12:00.123134752 (58h36m9.845697782s) [upload=56238]: Uploading segments... Jul 26 22:12:00.123493779 (58h36m9.846056819s) [upload=56238]: Got next segment. Inline: false Jul 26 22:12:00.123543474 (58h36m9.846106514s) [upload=56238]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:12:00.123548433 (58h36m9.846111473s) [upload=56238]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:12:00.355422881 (58h36m10.077985911s) [seg_pos={0 0}, upload=56238]: Begin upload segment... Jul 26 22:12:00.355465902 (58h36m10.078028942s) [seg_pos={0 0}, upload=56238]: Done begin upload segment. Jul 26 22:12:00.355487364 (58h36m10.078050404s) [noise=true , address=104.36.82.5:4837 , piece_id=WTBNXCZ3PWC5TQM4UWPTKT5QRXHHWQF6JVQQ7VQ4BXKLPEIZMIFQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.355513444 (58h36m10.078076474s) [noise=true , address=45.91.116.9:4263 , piece_id=H67DGXYB4V7UFZAIE4X4VRVWU53RIIQQOWOANN52TAZHFJNQT4IA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.355843715 (58h36m10.078406746s) [noise=true , address=45.91.116.8:4045 , piece_id=PRKLVJLWB37FDN52QBLNATG2SIWCXJCN7OR6EFLRZL74JZLC57VA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.355985106 (58h36m10.078548146s) [noise=true , address=77.90.31.19:4843 , piece_id=P3UAQQ2AO27TVMESOSBZUUXGX6QOBWZPXOMG4LIAUHTSBDN3YBJQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356118291 (58h36m10.078681331s) [noise=true , address=45.91.116.4:4939 , piece_id=XJUAUMJSYPFXTK7JIIJECLKWJOTWJQHYJHQ7TYPHMPBXRBS3HFPA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356442141 (58h36m10.079005181s) [noise=true , address=104.36.82.4:4802 , piece_id=KQ4ABYCX4T6VM477B56ETRPXTVFWDZXUTZEGPNAWIQ7ZV2XPYEIQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356714532 (58h36m10.079277572s) [noise=true , address=45.91.116.12:4225 , piece_id=E2D6FTGWZ22OFNFPRWBE335BANIUZIWVS433SMUC2LPMD6PTI6IA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356937989 (58h36m10.079501029s) [noise=true , address=45.91.116.13:4155 , piece_id=OXPGOXKCI4YOIKSIMPXVV6D74BBVMRPXXZKBBX2P7D4YQVR73VXQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357235789 (58h36m10.079798819s) [noise=true , address=77.90.31.12:4592 , piece_id=O3UI7JY3LQTABYKN3JKO5O4Z4RCLAX6HJZS4HQXJPGKDU3JNECTQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357472842 (58h36m10.080035882s) [noise=true , address=45.91.116.10:4082 , piece_id=J6PNAFSZBQCDVTAIQBHAMMN23AS7X3WLGV3YHVAKDU26RMWGKBLQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357542045 (58h36m10.080105085s) [noise=true , address=45.91.116.9:4262 , piece_id=TPC2GAHLX2R2B3ZZBTFYGWNMPYCX7KMZLNMNVM4GKQDOY6ADGUVQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357791683 (58h36m10.080354723s) [noise=true , address=45.91.116.5:4298 , piece_id=HURPSTTY5SXKPKJIHWPQ6VGKWQPIGSQONUMRQM6BREDGEA5V5ALA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358046780 (58h36m10.080609831s) [noise=true , address=104.36.82.15:4258 , piece_id=MJO75LPAAYCYPVPZYKBASP7ZN7Q3FI44W3XRN3CS5JAARSD32LXA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358143686 (58h36m10.080706716s) [noise=true , address=45.91.116.6:4973 , piece_id=E4B6O574VP2Y3J6PNPKTWAN7BFGSHIVQEPMTBDXQCDIORGOFMU6A, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358493164 (58h36m10.081056204s) [noise=true , address=104.36.82.12:4149 , piece_id=6ZXU4V2YOW3IXGT5XYX3Q7FT4U6YW7IVA72UPFHYUIUBWNPBDOIQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358579300 (58h36m10.08114233s) [noise=true , address=45.91.116.14:4190 , piece_id=PVZ2OCKX3JTUYT6KNUGIRHIYH5YVBSDMIZUOYSYSYBYB6AUNMOOA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:02.081979808 (58h36m11.804542868s) [upload=56238]: Next returned nil segment Jul 26 22:12:02.082008563 (58h36m11.804571643s) [upload=56238]: Waiting for error group managing segments... Jul 26 22:12:02.373463127 (58h36m12.096026167s) [noise=true , address=45.91.116.5:4298 , piece_id=HURPSTTY5SXKPKJIHWPQ6VGKWQPIGSQONUMRQM6BREDGEA5V5ALA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.391294504 (58h36m12.113857544s) [noise=true , address=77.90.31.12:4592 , piece_id=O3UI7JY3LQTABYKN3JKO5O4Z4RCLAX6HJZS4HQXJPGKDU3JNECTQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.392103922 (58h36m12.114666972s) [noise=true , address=45.91.116.9:4263 , piece_id=H67DGXYB4V7UFZAIE4X4VRVWU53RIIQQOWOANN52TAZHFJNQT4IA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.395004321 (58h36m12.117567361s) [noise=true , address=45.91.116.12:4225 , piece_id=E2D6FTGWZ22OFNFPRWBE335BANIUZIWVS433SMUC2LPMD6PTI6IA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.395744818 (58h36m12.118307858s) [noise=true , address=45.91.116.9:4262 , piece_id=TPC2GAHLX2R2B3ZZBTFYGWNMPYCX7KMZLNMNVM4GKQDOY6ADGUVQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.451177047 (58h36m12.173740097s) [noise=true , address=45.91.116.8:4045 , piece_id=PRKLVJLWB37FDN52QBLNATG2SIWCXJCN7OR6EFLRZL74JZLC57VA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.462789285 (58h36m12.185352325s) [noise=true , address=45.91.116.4:4939 , piece_id=XJUAUMJSYPFXTK7JIIJECLKWJOTWJQHYJHQ7TYPHMPBXRBS3HFPA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.477161623 (58h36m12.199724674s) [noise=true , address=45.91.116.13:4155 , piece_id=OXPGOXKCI4YOIKSIMPXVV6D74BBVMRPXXZKBBX2P7D4YQVR73VXQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.477354983 (58h36m12.199918023s) [noise=true , address=45.91.116.10:4082 , piece_id=J6PNAFSZBQCDVTAIQBHAMMN23AS7X3WLGV3YHVAKDU26RMWGKBLQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.482097287 (58h36m12.204660327s) [noise=true , address=45.91.116.6:4973 , piece_id=E4B6O574VP2Y3J6PNPKTWAN7BFGSHIVQEPMTBDXQCDIORGOFMU6A, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.522569814 (58h36m12.245132854s) [noise=true , address=77.90.31.19:4843 , piece_id=P3UAQQ2AO27TVMESOSBZUUXGX6QOBWZPXOMG4LIAUHTSBDN3YBJQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.537542521 (58h36m12.260105571s) [noise=true , address=45.91.116.14:4190 , piece_id=PVZ2OCKX3JTUYT6KNUGIRHIYH5YVBSDMIZUOYSYSYBYB6AUNMOOA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:29.482251815 (58h36m39.204814855s) [noise=true , address=104.36.82.5:4837 , piece_id=WTBNXCZ3PWC5TQM4UWPTKT5QRXHHWQF6JVQQ7VQ4BXKLPEIZMIFQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 188.166.225.170:38046->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482276953 (58h36m39.204839993s) [noise=true , address=104.36.82.4:4802 , piece_id=KQ4ABYCX4T6VM477B56ETRPXTVFWDZXUTZEGPNAWIQ7ZV2XPYEIQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482340875 (58h36m39.204903915s) [noise=true , address=104.36.82.15:4258 , piece_id=MJO75LPAAYCYPVPZYKBASP7ZN7Q3FI44W3XRN3CS5JAARSD32LXA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: write tcp 188.166.225.170:41258->104.36.82.15:4258: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482356565 (58h36m39.204919605s) [noise=true , address=104.36.82.12:4149 , piece_id=6ZXU4V2YOW3IXGT5XYX3Q7FT4U6YW7IVA72UPFHYUIUBWNPBDOIQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 188.166.225.170:39256->104.36.82.12:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482383747 (58h36m39.204946787s) [seg_pos={0 0}, upload=56238]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:12:29.709832457 (58h36m39.432395497s) [upload=56238]: Done uploading segments... Jul 26 22:12:29.709874738 (58h36m39.432437768s) [upload=56238]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.724780648s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.....................................................X...X..] duration: 40.89467834s FAIL (2025-07-26 22:26:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 40.894690202s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 25 out of 60 is failed (threshold=3) [...XX.XX.XXXX.XX.XXXXX.X.XXXXX.XXX.....X....................] duration: 5.845822335s FAIL (2025-07-26 22:26:15 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.845848675s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [......................................................X...X.] duration: 40.451048218s FAIL (2025-07-26 22:25:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 40.451066374s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................X..X...........X.........................] duration: 5.873959568s FAIL (2025-07-26 22:25:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.87398175s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 55 out of 60 is failed (threshold=3) [.....XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.230768566s FAIL (2025-07-26 22:25:25 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== SXUIA, seg_pos={0 0}, upload=56436]: Uploading piece... Jul 26 22:24:26.448198714 (58h48m36.170761754s) [noise=true , address=104.36.82.11:4147 , piece_id=LPQGSZ7LRTXAXJLY2L6VDOPS2IK444X6ONVQWXJ7DBLZKG6EPRVQ, seg_pos={0 0}, upload=56436]: Uploading piece... Jul 26 22:24:26.448290720 (58h48m36.17085376s) [noise=true , address=79.127.219.47:4061 , piece_id=X2BBKPPKZQMMO4AIYVGVEOIXXLPQPQEAUGSQLAB2V7OXYNQAUZTQ, seg_pos={0 0}, upload=56436]: Uploading piece... Jul 26 22:24:26.448594321 (58h48m36.171157351s) [noise=true , address=45.91.116.13:4187 , piece_id=5X7V2LQJTUH3634ICFYXJ6PEWZLILIYTSRM32TT7FIMA2DR7KGLQ, seg_pos={0 0}, upload=56436]: Uploading piece... Jul 26 22:24:26.448601334 (58h48m36.171164374s) [noise=true , address=104.36.82.45:4935 , piece_id=WGG7PN5GI37NNFRV3P65ELHJOIYGPKRBUF4LZSIWV6BBOKNLBC6Q, seg_pos={0 0}, upload=56436]: Uploading piece... Jul 26 22:24:26.448850802 (58h48m36.171413832s) [noise=true , address=104.36.82.13:4199 , piece_id=SY5ME7XZKVEAH6B2YFU4WTZQ5E7XZJKSTQWDIOPTWOVE5BH7QFBQ, seg_pos={0 0}, upload=56436]: Uploading piece... Jul 26 22:24:26.448867283 (58h48m36.171430313s) [noise=true , address=79.127.201.215:4356 , piece_id=RLNVN4WI5EQS45GPG75BCHRVS4ZVM3NAXX3TUET2HSQ6KNTCIHTQ, seg_pos={0 0}, upload=56436]: Uploading piece... Jul 26 22:24:26.449104988 (58h48m36.171668028s) [noise=true , address=77.90.31.19:4849 , piece_id=XHCQBMA5OJGMAUTVTGZ4QN6XWKBW5ZIS5QN25NOIFMUO6ZZOO5SA, seg_pos={0 0}, upload=56436]: Uploading piece... Jul 26 22:24:26.449124104 (58h48m36.171687144s) [noise=true , address=45.91.116.11:4146 , piece_id=ELI46CVPKV4MCTAVD5KVWF35ZSDIQYOAGRV4RQCBT4DLHOHHCX5A, seg_pos={0 0}, upload=56436]: Uploading piece... Jul 26 22:24:26.449341891 (58h48m36.171904931s) [noise=true , address=79.127.219.51:4210 , piece_id=IC4PRZ477VK6NPRRBJWJI62ATSRKMFUZREAVFVY7WIMJFK2QUTJQ, seg_pos={0 0}, upload=56436]: Uploading piece... Jul 26 22:24:26.449373713 (58h48m36.171936743s) [noise=true , address=104.36.82.37:4682 , piece_id=5JLKRWQUIAESFJF7OPCLKD3X4GZSNRG2EJ7MSAQILYWNETCTT4VQ, seg_pos={0 0}, upload=56436]: Uploading piece... Jul 26 22:24:26.449586108 (58h48m36.172149148s) [noise=true , address=77.90.31.11:4581 , piece_id=Y44T62FTM45JRKSRQ4ISH5ND4NNCNOYTPGH4LMY5H5FNXKYQBF3A, seg_pos={0 0}, upload=56436]: Uploading piece... Jul 26 22:24:26.449640603 (58h48m36.172203643s) [noise=true , address=104.36.82.2:4752 , piece_id=A4Z3YM4IQPRKVADKRRA6L4VEBJ52I6E22J2WRTFDXMDJXRAVBQYA, seg_pos={0 0}, upload=56436]: Uploading piece... Jul 26 22:24:26.449904708 (58h48m36.172467748s) [noise=true , address=104.36.82.55:4228 , piece_id=H2PLDQ33257IJHOYGOOF3FIPGYYEIPS3RKHATZ6K7JG27BYA6UVA, seg_pos={0 0}, upload=56436]: Uploading piece... Jul 26 22:24:26.450115853 (58h48m36.172678883s) [noise=true , address=77.90.31.10:4540 , piece_id=3E274R33HLMT3AVVQ5ALKNCD3T4OAX6BATZKFE67J2WXDV3CZYBA, seg_pos={0 0}, upload=56436]: Uploading piece... Jul 26 22:24:26.451201139 (58h48m36.173764179s) [noise=true , address=79.127.219.38:4004 , piece_id=G4MN4PR4OKOZO3DEJIU2U7KF2BCBBYGIG5Y5UZ4NRQHOVE5EO4PQ, seg_pos={0 0}, upload=56436]: Uploading piece... Jul 26 22:24:26.452137260 (58h48m36.1747003s) [noise=true , address=104.36.82.56:4232 , piece_id=SACCGKZOOZQDPK5JCFIRSJE7BSCJMDKJPWTBWVNVELA3Q2CXIPYA, seg_pos={0 0}, upload=56436]: Uploading piece... Jul 26 22:24:26.452740213 (58h48m36.175303253s) [noise=true , address=79.127.205.234:4617 , piece_id=PTERMLNJN3VUPKQ7K52GHNJQCAPJA52RM7GM7UFMI2ZPAMQDPFUQ, seg_pos={0 0}, upload=56436]: Uploading piece... Jul 26 22:24:27.584309718 (58h48m37.306872758s) [upload=56436]: Next returned nil segment Jul 26 22:24:27.584331149 (58h48m37.306894189s) [upload=56436]: Waiting for error group managing segments... Jul 26 22:24:27.591066525 (58h48m37.313629555s) [noise=true , address=79.127.219.47:4061 , piece_id=X2BBKPPKZQMMO4AIYVGVEOIXXLPQPQEAUGSQLAB2V7OXYNQAUZTQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: ecclient: upload failed (node:12f4AfVs2T4a8Yhd456WKdse7TZSjHBDULMR8myvo1ua9jiUrNu, address:79.127.219.47:4061): protocol: storage node overloaded, request limit: 2000 Jul 26 22:24:27.882992688 (58h48m37.605555728s) [noise=true , address=79.127.205.234:4617 , piece_id=PTERMLNJN3VUPKQ7K52GHNJQCAPJA52RM7GM7UFMI2ZPAMQDPFUQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Jul 26 22:24:28.334487017 (58h48m38.057050057s) [noise=true , address=79.127.219.38:4004 , piece_id=G4MN4PR4OKOZO3DEJIU2U7KF2BCBBYGIG5Y5UZ4NRQHOVE5EO4PQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Jul 26 22:24:28.462993043 (58h48m38.185556083s) [noise=true , address=77.90.31.10:4540 , piece_id=3E274R33HLMT3AVVQ5ALKNCD3T4OAX6BATZKFE67J2WXDV3CZYBA, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Jul 26 22:24:28.568151239 (58h48m38.290714269s) [noise=true , address=77.90.31.11:4581 , piece_id=Y44T62FTM45JRKSRQ4ISH5ND4NNCNOYTPGH4LMY5H5FNXKYQBF3A, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Jul 26 22:24:30.703353129 (58h48m40.425916169s) [noise=true , address=156.146.43.225:4434 , piece_id=ZM53OWACKUYJF6UAEGE5TF5QILFY75LPSD73GDTU7YSNUE3SXUIA, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Jul 26 22:24:31.013092473 (58h48m40.735655513s) [noise=true , address=79.127.219.51:4210 , piece_id=IC4PRZ477VK6NPRRBJWJI62ATSRKMFUZREAVFVY7WIMJFK2QUTJQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Jul 26 22:24:31.491463614 (58h48m41.214026655s) [noise=true , address=45.91.116.11:4146 , piece_id=ELI46CVPKV4MCTAVD5KVWF35ZSDIQYOAGRV4RQCBT4DLHOHHCX5A, seg_pos={0 0}, upload=56436]: Done uploading piece. err: ecclient: upload failed (node:12VJf4TrDJ7So1W3XuGbfnbKMWC8ftHReLgqfWGqjpRWAbwCjrp, address:45.91.116.11:4146): protocol: storage node overloaded, request limit: 2000 Jul 26 22:24:31.864075009 (58h48m41.586638049s) [noise=true , address=104.36.82.55:4228 , piece_id=H2PLDQ33257IJHOYGOOF3FIPGYYEIPS3RKHATZ6K7JG27BYA6UVA, seg_pos={0 0}, upload=56436]: Done uploading piece. err: ecclient: upload failed (node:12NBsd33JtAwPdcbD8vBzZUYpjiqZpRzs8y2QrYJFE245pd8T5X, address:104.36.82.55:4228): protocol: storage node overloaded, request limit: 2000 Jul 26 22:24:40.722041844 (58h48m50.444604885s) [noise=true , address=79.127.201.215:4356 , piece_id=RLNVN4WI5EQS45GPG75BCHRVS4ZVM3NAXX3TUET2HSQ6KNTCIHTQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Jul 26 22:24:40.910783237 (58h48m50.633346277s) [noise=true , address=45.91.116.13:4187 , piece_id=5X7V2LQJTUH3634ICFYXJ6PEWZLILIYTSRM32TT7FIMA2DR7KGLQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Jul 26 22:24:46.762035915 (58h48m56.484598955s) [noise=true , address=77.90.31.19:4849 , piece_id=XHCQBMA5OJGMAUTVTGZ4QN6XWKBW5ZIS5QN25NOIFMUO6ZZOO5SA, seg_pos={0 0}, upload=56436]: Done uploading piece. err: <nil> Jul 26 22:25:25.357944454 (58h49m35.080507494s) [noise=true , address=104.36.82.37:4682 , piece_id=5JLKRWQUIAESFJF7OPCLKD3X4GZSNRG2EJ7MSAQILYWNETCTT4VQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: ecclient: upload failed (node:1ZNetQr9AyAnVMiPj6apAfCEJ5Zm9HjMiFp9KhHtSsxJaY6pN8, address:104.36.82.37:4682): protocol: rpc: dial tcp 104.36.82.37:4682: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:25.357995201 (58h49m35.080558241s) [noise=true , address=104.36.82.2:4752 , piece_id=A4Z3YM4IQPRKVADKRRA6L4VEBJ52I6E22J2WRTFDXMDJXRAVBQYA, seg_pos={0 0}, upload=56436]: Done uploading piece. err: ecclient: upload failed (node:12oRYWQwoe5JtDCqfSShqtyRWVQLjL4UtbokBgNpDAocBKv1vqL, address:104.36.82.2:4752): protocol: write tcp 188.166.225.170:33046->104.36.82.2:4752: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:25.358027663 (58h49m35.080590693s) [noise=true , address=104.36.82.13:4199 , piece_id=SY5ME7XZKVEAH6B2YFU4WTZQ5E7XZJKSTQWDIOPTWOVE5BH7QFBQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: ecclient: upload failed (node:1DagQMPpe9F6w5xewXXCfzVBPcmgg6EKCauFsNCWHoHdP56bs7, address:104.36.82.13:4199): protocol: write tcp 188.166.225.170:59642->104.36.82.13:4199: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:25.358160538 (58h49m35.080723598s) [noise=true , address=104.36.82.45:4935 , piece_id=WGG7PN5GI37NNFRV3P65ELHJOIYGPKRBUF4LZSIWV6BBOKNLBC6Q, seg_pos={0 0}, upload=56436]: Done uploading piece. err: ecclient: upload failed (node:12Ks2kQWUmV377HgGbyoTt96CTyK1P1nH8QtZQzh3xvnFCYaD8X, address:104.36.82.45:4935): protocol: write tcp 188.166.225.170:34834->104.36.82.45:4935: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:25.358209491 (58h49m35.080772541s) [noise=true , address=104.36.82.56:4232 , piece_id=SACCGKZOOZQDPK5JCFIRSJE7BSCJMDKJPWTBWVNVELA3Q2CXIPYA, seg_pos={0 0}, upload=56436]: Done uploading piece. err: ecclient: upload failed (node:1knYU6KSZJCmdpXTRHXZmCsRJYfNGGQCLPEPkWkBgUdyvtSgJe, address:104.36.82.56:4232): protocol: rpc: dial tcp 104.36.82.56:4232: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:25.358219019 (58h49m35.080782059s) [noise=true , address=104.36.82.11:4147 , piece_id=LPQGSZ7LRTXAXJLY2L6VDOPS2IK444X6ONVQWXJ7DBLZKG6EPRVQ, seg_pos={0 0}, upload=56436]: Done uploading piece. err: ecclient: upload failed (node:12nNFCM4wDSvGMeqF8MF6csgKDUV6QyozoC5fZwUaj7MN1Woz4P, address:104.36.82.11:4147): protocol: rpc: dial tcp 104.36.82.11:4147: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:25.358241713 (58h49m35.080804753s) [seg_pos={0 0}, upload=56436]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 26 22:25:25.587184691 (58h49m35.309747731s) [upload=56436]: Done uploading segments... Jul 26 22:25:25.587210400 (58h49m35.30977343s) [upload=56436]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.230784098s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 25 out of 60 is failed (threshold=3) [....XX.XX.XXXX.XX.XXXXX.X.XXXXX.XXX.....X...................] duration: 5.712507826s FAIL (2025-07-26 22:25:23 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.712537032s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [.XX.XX.XXX.X.XX.XXX.XXXXXX..XXXX..X.X...X...................] duration: 5.784182901s FAIL (2025-07-26 22:25:20 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== rite tcp 188.166.225.170:53646->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:53646->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53646->77.90.31.10:4519: use of closed network connection Jul 26 22:11:05.345059537 (58h35m15.067622577s) [seg_pos={0 0}, upload=56223]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:11:05.345121094 (58h35m15.067684124s) [upload=56223]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:11:05.345124551 (58h35m15.067687591s) [upload=56223]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:11:05.638665486 (58h35m15.361228526s) [upload=56223]: Done uploading segments... Jul 26 22:11:05.638683710 (58h35m15.36124674s) [upload=56223]: Upload finished. err: <nil> Jul 26 22:12:00.122995475 (58h36m9.845558505s) [upload=56238]: Starting upload Jul 26 22:12:00.123061241 (58h36m9.845624351s) [upload=56238]: Done starting upload Jul 26 22:12:00.123134752 (58h36m9.845697782s) [upload=56238]: Uploading segments... Jul 26 22:12:00.123493779 (58h36m9.846056819s) [upload=56238]: Got next segment. Inline: false Jul 26 22:12:00.123543474 (58h36m9.846106514s) [upload=56238]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:12:00.123548433 (58h36m9.846111473s) [upload=56238]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:12:00.355422881 (58h36m10.077985911s) [seg_pos={0 0}, upload=56238]: Begin upload segment... Jul 26 22:12:00.355465902 (58h36m10.078028942s) [seg_pos={0 0}, upload=56238]: Done begin upload segment. Jul 26 22:12:00.355487364 (58h36m10.078050404s) [noise=true , address=104.36.82.5:4837 , piece_id=WTBNXCZ3PWC5TQM4UWPTKT5QRXHHWQF6JVQQ7VQ4BXKLPEIZMIFQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.355513444 (58h36m10.078076474s) [noise=true , address=45.91.116.9:4263 , piece_id=H67DGXYB4V7UFZAIE4X4VRVWU53RIIQQOWOANN52TAZHFJNQT4IA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.355843715 (58h36m10.078406746s) [noise=true , address=45.91.116.8:4045 , piece_id=PRKLVJLWB37FDN52QBLNATG2SIWCXJCN7OR6EFLRZL74JZLC57VA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.355985106 (58h36m10.078548146s) [noise=true , address=77.90.31.19:4843 , piece_id=P3UAQQ2AO27TVMESOSBZUUXGX6QOBWZPXOMG4LIAUHTSBDN3YBJQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356118291 (58h36m10.078681331s) [noise=true , address=45.91.116.4:4939 , piece_id=XJUAUMJSYPFXTK7JIIJECLKWJOTWJQHYJHQ7TYPHMPBXRBS3HFPA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356442141 (58h36m10.079005181s) [noise=true , address=104.36.82.4:4802 , piece_id=KQ4ABYCX4T6VM477B56ETRPXTVFWDZXUTZEGPNAWIQ7ZV2XPYEIQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356714532 (58h36m10.079277572s) [noise=true , address=45.91.116.12:4225 , piece_id=E2D6FTGWZ22OFNFPRWBE335BANIUZIWVS433SMUC2LPMD6PTI6IA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356937989 (58h36m10.079501029s) [noise=true , address=45.91.116.13:4155 , piece_id=OXPGOXKCI4YOIKSIMPXVV6D74BBVMRPXXZKBBX2P7D4YQVR73VXQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357235789 (58h36m10.079798819s) [noise=true , address=77.90.31.12:4592 , piece_id=O3UI7JY3LQTABYKN3JKO5O4Z4RCLAX6HJZS4HQXJPGKDU3JNECTQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357472842 (58h36m10.080035882s) [noise=true , address=45.91.116.10:4082 , piece_id=J6PNAFSZBQCDVTAIQBHAMMN23AS7X3WLGV3YHVAKDU26RMWGKBLQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357542045 (58h36m10.080105085s) [noise=true , address=45.91.116.9:4262 , piece_id=TPC2GAHLX2R2B3ZZBTFYGWNMPYCX7KMZLNMNVM4GKQDOY6ADGUVQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357791683 (58h36m10.080354723s) [noise=true , address=45.91.116.5:4298 , piece_id=HURPSTTY5SXKPKJIHWPQ6VGKWQPIGSQONUMRQM6BREDGEA5V5ALA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358046780 (58h36m10.080609831s) [noise=true , address=104.36.82.15:4258 , piece_id=MJO75LPAAYCYPVPZYKBASP7ZN7Q3FI44W3XRN3CS5JAARSD32LXA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358143686 (58h36m10.080706716s) [noise=true , address=45.91.116.6:4973 , piece_id=E4B6O574VP2Y3J6PNPKTWAN7BFGSHIVQEPMTBDXQCDIORGOFMU6A, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358493164 (58h36m10.081056204s) [noise=true , address=104.36.82.12:4149 , piece_id=6ZXU4V2YOW3IXGT5XYX3Q7FT4U6YW7IVA72UPFHYUIUBWNPBDOIQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358579300 (58h36m10.08114233s) [noise=true , address=45.91.116.14:4190 , piece_id=PVZ2OCKX3JTUYT6KNUGIRHIYH5YVBSDMIZUOYSYSYBYB6AUNMOOA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:02.081979808 (58h36m11.804542868s) [upload=56238]: Next returned nil segment Jul 26 22:12:02.082008563 (58h36m11.804571643s) [upload=56238]: Waiting for error group managing segments... Jul 26 22:12:02.373463127 (58h36m12.096026167s) [noise=true , address=45.91.116.5:4298 , piece_id=HURPSTTY5SXKPKJIHWPQ6VGKWQPIGSQONUMRQM6BREDGEA5V5ALA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.391294504 (58h36m12.113857544s) [noise=true , address=77.90.31.12:4592 , piece_id=O3UI7JY3LQTABYKN3JKO5O4Z4RCLAX6HJZS4HQXJPGKDU3JNECTQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.392103922 (58h36m12.114666972s) [noise=true , address=45.91.116.9:4263 , piece_id=H67DGXYB4V7UFZAIE4X4VRVWU53RIIQQOWOANN52TAZHFJNQT4IA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.395004321 (58h36m12.117567361s) [noise=true , address=45.91.116.12:4225 , piece_id=E2D6FTGWZ22OFNFPRWBE335BANIUZIWVS433SMUC2LPMD6PTI6IA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.395744818 (58h36m12.118307858s) [noise=true , address=45.91.116.9:4262 , piece_id=TPC2GAHLX2R2B3ZZBTFYGWNMPYCX7KMZLNMNVM4GKQDOY6ADGUVQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.451177047 (58h36m12.173740097s) [noise=true , address=45.91.116.8:4045 , piece_id=PRKLVJLWB37FDN52QBLNATG2SIWCXJCN7OR6EFLRZL74JZLC57VA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.462789285 (58h36m12.185352325s) [noise=true , address=45.91.116.4:4939 , piece_id=XJUAUMJSYPFXTK7JIIJECLKWJOTWJQHYJHQ7TYPHMPBXRBS3HFPA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.477161623 (58h36m12.199724674s) [noise=true , address=45.91.116.13:4155 , piece_id=OXPGOXKCI4YOIKSIMPXVV6D74BBVMRPXXZKBBX2P7D4YQVR73VXQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.477354983 (58h36m12.199918023s) [noise=true , address=45.91.116.10:4082 , piece_id=J6PNAFSZBQCDVTAIQBHAMMN23AS7X3WLGV3YHVAKDU26RMWGKBLQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.482097287 (58h36m12.204660327s) [noise=true , address=45.91.116.6:4973 , piece_id=E4B6O574VP2Y3J6PNPKTWAN7BFGSHIVQEPMTBDXQCDIORGOFMU6A, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.522569814 (58h36m12.245132854s) [noise=true , address=77.90.31.19:4843 , piece_id=P3UAQQ2AO27TVMESOSBZUUXGX6QOBWZPXOMG4LIAUHTSBDN3YBJQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.537542521 (58h36m12.260105571s) [noise=true , address=45.91.116.14:4190 , piece_id=PVZ2OCKX3JTUYT6KNUGIRHIYH5YVBSDMIZUOYSYSYBYB6AUNMOOA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:29.482251815 (58h36m39.204814855s) [noise=true , address=104.36.82.5:4837 , piece_id=WTBNXCZ3PWC5TQM4UWPTKT5QRXHHWQF6JVQQ7VQ4BXKLPEIZMIFQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 188.166.225.170:38046->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482276953 (58h36m39.204839993s) [noise=true , address=104.36.82.4:4802 , piece_id=KQ4ABYCX4T6VM477B56ETRPXTVFWDZXUTZEGPNAWIQ7ZV2XPYEIQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482340875 (58h36m39.204903915s) [noise=true , address=104.36.82.15:4258 , piece_id=MJO75LPAAYCYPVPZYKBASP7ZN7Q3FI44W3XRN3CS5JAARSD32LXA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: write tcp 188.166.225.170:41258->104.36.82.15:4258: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482356565 (58h36m39.204919605s) [noise=true , address=104.36.82.12:4149 , piece_id=6ZXU4V2YOW3IXGT5XYX3Q7FT4U6YW7IVA72UPFHYUIUBWNPBDOIQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 188.166.225.170:39256->104.36.82.12:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482383747 (58h36m39.204946787s) [seg_pos={0 0}, upload=56238]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:12:29.709832457 (58h36m39.432395497s) [upload=56238]: Done uploading segments... Jul 26 22:12:29.709874738 (58h36m39.432437768s) [upload=56238]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.784210254s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 25 out of 60 is failed (threshold=3) [.....XX.XX.XXXX.XX.XXXXX.X.XXXXX.XXX.....X..................] duration: 5.706362356s FAIL (2025-07-26 22:25:00 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.706388597s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.......................................................X...X] duration: 1m0.000817693s FAIL (2025-07-26 22:24:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.00082701s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................X..X...........X........................] duration: 5.890114637s FAIL (2025-07-26 22:24:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.890138111s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 54 out of 60 is failed (threshold=3) [......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.227968285s FAIL (2025-07-26 22:24:25 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== 0}, upload=56420]: Done begin upload segment. Jul 26 22:23:25.972398552 (58h47m35.694961592s) [noise=true , address=104.36.82.41:4777 , piece_id=ZJNTVVF4GHHZRD4FVV4JK54L2LCXZM23RTO772NZDERHVLJ76M3A, seg_pos={0 0}, upload=56420]: Uploading piece... Jul 26 22:23:25.972444349 (58h47m35.695007389s) [noise=true , address=79.127.201.215:4343 , piece_id=XIU6EYPQPAX56W5Q2LRQYTOGHTD2OCCCBBZUKEUW5HXER4TE3A5A, seg_pos={0 0}, upload=56420]: Uploading piece... Jul 26 22:23:25.972756186 (58h47m35.695319216s) [noise=true , address=104.36.82.11:4141 , piece_id=4BFXJWB2U6AIMQPGOOVHIHXRGJXFQ7U7DQMPOFGZU2YEPZNCEDXQ, seg_pos={0 0}, upload=56420]: Uploading piece... Jul 26 22:23:25.972979143 (58h47m35.695542183s) [noise=true , address=79.127.205.235:4681 , piece_id=YUBQM2UVOERNKISW245NL4VVCSMJOA7NG4Y5Z5AEGES4LVJV6VBA, seg_pos={0 0}, upload=56420]: Uploading piece... Jul 26 22:23:25.973212639 (58h47m35.695775689s) [noise=true , address=104.36.82.56:4232 , piece_id=QRNYYOAT4ROWBIWLSVNJBX6ILNETOH7ZWC6NT4ZXMVKUZG55PAEA, seg_pos={0 0}, upload=56420]: Uploading piece... Jul 26 22:23:25.973244049 (58h47m35.695807079s) [noise=true , address=77.90.31.21:4897 , piece_id=5UW5IFKFZSLZ43RALU6HAG27O5PNDMROINANBJGP6SXJHMWKLCOQ, seg_pos={0 0}, upload=56420]: Uploading piece... Jul 26 22:23:25.973465202 (58h47m35.696028242s) [noise=true , address=104.36.82.40:4961 , piece_id=IXV2DTBXEFMRBXYAOBNJ6DMWGW5UTMGCLT2T2ZF63YK5ZXQHNCSA, seg_pos={0 0}, upload=56420]: Uploading piece... Jul 26 22:23:25.973697587 (58h47m35.696260627s) [noise=true , address=104.36.82.21:4477 , piece_id=MRTHYSAXYPUA24FNBAIFZBF6RHQRSIAQMOUDKJFELXDMZDEDBMNQ, seg_pos={0 0}, upload=56420]: Uploading piece... Jul 26 22:23:25.973829670 (58h47m35.69639271s) [noise=true , address=104.36.82.55:4228 , piece_id=SJKIDFGXTGRU2NVAKQOQLZI5UTR23N3FL4C4BEJJOWGWKV5SKGSA, seg_pos={0 0}, upload=56420]: Uploading piece... Jul 26 22:23:25.973934961 (58h47m35.696497991s) [noise=true , address=79.127.205.245:4236 , piece_id=6VYHJKM5QWRINFV3UPZA5XF2APU5PTETTGEB7SAZ7JGYC4MR4Y6A, seg_pos={0 0}, upload=56420]: Uploading piece... Jul 26 22:23:25.974136507 (58h47m35.696699547s) [noise=true , address=79.127.219.38:4006 , piece_id=JQKKKTRJJ74YCW5IHRHHSI2GHOJ6OMRHKOLU5KMA7PACW3QUGPKQ, seg_pos={0 0}, upload=56420]: Uploading piece... Jul 26 22:23:25.974262087 (58h47m35.696825127s) [noise=true , address=79.127.220.101:4048 , piece_id=2QIFNH54UYSCNPWBRA7PQW37VQUJFBPFQNLGV7QEEKBEZIC3YSKA, seg_pos={0 0}, upload=56420]: Uploading piece... Jul 26 22:23:25.974329597 (58h47m35.696892637s) [noise=true , address=45.91.116.14:4206 , piece_id=RJTL5VWYHRAQFCS52V2TXX2YI6PCS4ICDFPXFLE7YY2JHFNLU2RA, seg_pos={0 0}, upload=56420]: Uploading piece... Jul 26 22:23:25.974528126 (58h47m35.697091166s) [noise=true , address=79.127.219.51:4212 , piece_id=IBFWKL45TRM4Z4NJZP5JAKUIXAUF56QISVZG4IFSAQ3GTYYBK6DQ, seg_pos={0 0}, upload=56420]: Uploading piece... Jul 26 22:23:25.974670969 (58h47m35.697234009s) [noise=true , address=79.127.205.233:4645 , piece_id=JJV7Y4NIPOODJZSBCAWRY2DOJ7OAOME7NGG5U6OSSMKU4K6B5GCQ, seg_pos={0 0}, upload=56420]: Uploading piece... Jul 26 22:23:25.974725443 (58h47m35.697288484s) [noise=true , address=79.127.205.234:4620 , piece_id=EZEJUEN54TU4NRODEFHGR7RMICQWWWQOZJKNFQCT3SVPTSNMJDQA, seg_pos={0 0}, upload=56420]: Uploading piece... Jul 26 22:23:25.974998145 (58h47m35.697561175s) [noise=true , address=45.91.116.10:4100 , piece_id=MIKZKGUB45NZMUXOIQPR4WTDIJA3F3MY7PMLXK67ZM6A77OBMUKA, seg_pos={0 0}, upload=56420]: Uploading piece... Jul 26 22:23:25.976674393 (58h47m35.699237433s) [noise=true , address=104.36.82.36:4626 , piece_id=TGCZ7USQJ4SSKA7MPLXQ43I3RCALKY2QREL3H6F6ES3LHGYG26PA, seg_pos={0 0}, upload=56420]: Uploading piece... Jul 26 22:23:26.968452678 (58h47m36.691015718s) [noise=true , address=79.127.220.101:4048 , piece_id=2QIFNH54UYSCNPWBRA7PQW37VQUJFBPFQNLGV7QEEKBEZIC3YSKA, seg_pos={0 0}, upload=56420]: Done uploading piece. err: ecclient: upload failed (node:1csdxFUHCWKuGfrZnVfpRJ3R6DYCzxx3voY1UR5kWwSXaxZaPg, address:79.127.220.101:4048): protocol: storage node overloaded, request limit: 2000 Jul 26 22:23:27.068659100 (58h47m36.79122214s) [upload=56420]: Next returned nil segment Jul 26 22:23:27.068676964 (58h47m36.791240004s) [upload=56420]: Waiting for error group managing segments... Jul 26 22:23:27.358757576 (58h47m37.081320616s) [noise=true , address=79.127.205.245:4236 , piece_id=6VYHJKM5QWRINFV3UPZA5XF2APU5PTETTGEB7SAZ7JGYC4MR4Y6A, seg_pos={0 0}, upload=56420]: Done uploading piece. err: <nil> Jul 26 22:23:27.378592004 (58h47m37.101155034s) [noise=true , address=79.127.205.235:4681 , piece_id=YUBQM2UVOERNKISW245NL4VVCSMJOA7NG4Y5Z5AEGES4LVJV6VBA, seg_pos={0 0}, upload=56420]: Done uploading piece. err: <nil> Jul 26 22:23:27.392488451 (58h47m37.115051481s) [noise=true , address=79.127.205.234:4620 , piece_id=EZEJUEN54TU4NRODEFHGR7RMICQWWWQOZJKNFQCT3SVPTSNMJDQA, seg_pos={0 0}, upload=56420]: Done uploading piece. err: <nil> Jul 26 22:23:27.473674272 (58h47m37.196237312s) [noise=true , address=79.127.205.233:4645 , piece_id=JJV7Y4NIPOODJZSBCAWRY2DOJ7OAOME7NGG5U6OSSMKU4K6B5GCQ, seg_pos={0 0}, upload=56420]: Done uploading piece. err: <nil> Jul 26 22:23:27.968271128 (58h47m37.690834168s) [noise=true , address=79.127.219.38:4006 , piece_id=JQKKKTRJJ74YCW5IHRHHSI2GHOJ6OMRHKOLU5KMA7PACW3QUGPKQ, seg_pos={0 0}, upload=56420]: Done uploading piece. err: <nil> Jul 26 22:23:27.989648718 (58h47m37.712211748s) [noise=true , address=77.90.31.21:4897 , piece_id=5UW5IFKFZSLZ43RALU6HAG27O5PNDMROINANBJGP6SXJHMWKLCOQ, seg_pos={0 0}, upload=56420]: Done uploading piece. err: <nil> Jul 26 22:23:28.374687590 (58h47m38.09725063s) [noise=true , address=45.91.116.14:4206 , piece_id=RJTL5VWYHRAQFCS52V2TXX2YI6PCS4ICDFPXFLE7YY2JHFNLU2RA, seg_pos={0 0}, upload=56420]: Done uploading piece. err: <nil> Jul 26 22:23:28.794267918 (58h47m38.516830958s) [noise=true , address=79.127.219.51:4212 , piece_id=IBFWKL45TRM4Z4NJZP5JAKUIXAUF56QISVZG4IFSAQ3GTYYBK6DQ, seg_pos={0 0}, upload=56420]: Done uploading piece. err: <nil> Jul 26 22:23:29.132833164 (58h47m38.855396194s) [noise=true , address=104.36.82.55:4228 , piece_id=SJKIDFGXTGRU2NVAKQOQLZI5UTR23N3FL4C4BEJJOWGWKV5SKGSA, seg_pos={0 0}, upload=56420]: Done uploading piece. err: ecclient: upload failed (node:12NBsd33JtAwPdcbD8vBzZUYpjiqZpRzs8y2QrYJFE245pd8T5X, address:104.36.82.55:4228): protocol: storage node overloaded, request limit: 2000 Jul 26 22:23:33.270854901 (58h47m42.993417941s) [noise=true , address=45.91.116.10:4100 , piece_id=MIKZKGUB45NZMUXOIQPR4WTDIJA3F3MY7PMLXK67ZM6A77OBMUKA, seg_pos={0 0}, upload=56420]: Done uploading piece. err: <nil> Jul 26 22:23:40.548138829 (58h47m50.270701869s) [noise=true , address=79.127.201.215:4343 , piece_id=XIU6EYPQPAX56W5Q2LRQYTOGHTD2OCCCBBZUKEUW5HXER4TE3A5A, seg_pos={0 0}, upload=56420]: Done uploading piece. err: <nil> Jul 26 22:24:24.890240005 (58h48m34.612803045s) [noise=true , address=104.36.82.11:4141 , piece_id=4BFXJWB2U6AIMQPGOOVHIHXRGJXFQ7U7DQMPOFGZU2YEPZNCEDXQ, seg_pos={0 0}, upload=56420]: Done uploading piece. err: ecclient: upload failed (node:129zfpWSguxcUH9723ivdoL39WgpjiTxA189mYbzsGAGVjFcXLL, address:104.36.82.11:4141): protocol: write tcp 188.166.225.170:56062->104.36.82.11:4141: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:24.890415782 (58h48m34.612978822s) [noise=true , address=104.36.82.56:4232 , piece_id=QRNYYOAT4ROWBIWLSVNJBX6ILNETOH7ZWC6NT4ZXMVKUZG55PAEA, seg_pos={0 0}, upload=56420]: Done uploading piece. err: ecclient: upload failed (node:1knYU6KSZJCmdpXTRHXZmCsRJYfNGGQCLPEPkWkBgUdyvtSgJe, address:104.36.82.56:4232): protocol: rpc: dial tcp 104.36.82.56:4232: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:24.890459865 (58h48m34.613022895s) [noise=true , address=104.36.82.41:4777 , piece_id=ZJNTVVF4GHHZRD4FVV4JK54L2LCXZM23RTO772NZDERHVLJ76M3A, seg_pos={0 0}, upload=56420]: Done uploading piece. err: ecclient: upload failed (node:1Pvm1SftEF3tnCEBqUn1ZmyVcLDBkR6pyegEb7mtzjdQiiqYqt, address:104.36.82.41:4777): protocol: rpc: dial tcp 104.36.82.41:4777: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:24.890465687 (58h48m34.613028727s) [noise=true , address=104.36.82.21:4477 , piece_id=MRTHYSAXYPUA24FNBAIFZBF6RHQRSIAQMOUDKJFELXDMZDEDBMNQ, seg_pos={0 0}, upload=56420]: Done uploading piece. err: ecclient: upload failed (node:1QNztqPaSGyZVsQihaj9YjG9C1nzKi1crfSDjDHimYboYAKWZY, address:104.36.82.21:4477): protocol: rpc: dial tcp 104.36.82.21:4477: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:24.890493641 (58h48m34.613056671s) [noise=true , address=104.36.82.40:4961 , piece_id=IXV2DTBXEFMRBXYAOBNJ6DMWGW5UTMGCLT2T2ZF63YK5ZXQHNCSA, seg_pos={0 0}, upload=56420]: Done uploading piece. err: ecclient: upload failed (node:12ZGPLhM6NsdjKLkLCCLSeaNS8ikfnHPZxc3CCw9mLzSznbEoUa, address:104.36.82.40:4961): protocol: rpc: dial tcp 104.36.82.40:4961: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:24.890502517 (58h48m34.613065558s) [noise=true , address=104.36.82.36:4626 , piece_id=TGCZ7USQJ4SSKA7MPLXQ43I3RCALKY2QREL3H6F6ES3LHGYG26PA, seg_pos={0 0}, upload=56420]: Done uploading piece. err: ecclient: upload failed (node:12kx49CN28mLjnGhQBsdwH5zQyWypePDe3Ss15LyVEYmTczhHan, address:104.36.82.36:4626): protocol: write tcp 188.166.225.170:59550->104.36.82.36:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:24.890523838 (58h48m34.613086879s) [seg_pos={0 0}, upload=56420]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 26 22:24:25.117057758 (58h48m34.839620798s) [upload=56420]: Done uploading segments... Jul 26 22:24:25.117085902 (58h48m34.839648932s) [upload=56420]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.22798132s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 25 out of 60 is failed (threshold=3) [......XX.XX.XXXX.XX.XXXXX.X.XXXXX.XXX.....X.................] duration: 5.703097345s FAIL (2025-07-26 22:24:14 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.703127112s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................X..X...........X.......................] duration: 6.127793513s FAIL (2025-07-26 22:24:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.127814914s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [..XX.XX.XXX.X.XX.XXX.XXXXXX..XXXX..X.X...X..................] duration: 5.718213786s FAIL (2025-07-26 22:24:09 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== rite tcp 188.166.225.170:53646->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:53646->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53646->77.90.31.10:4519: use of closed network connection Jul 26 22:11:05.345059537 (58h35m15.067622577s) [seg_pos={0 0}, upload=56223]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:11:05.345121094 (58h35m15.067684124s) [upload=56223]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:11:05.345124551 (58h35m15.067687591s) [upload=56223]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:11:05.638665486 (58h35m15.361228526s) [upload=56223]: Done uploading segments... Jul 26 22:11:05.638683710 (58h35m15.36124674s) [upload=56223]: Upload finished. err: <nil> Jul 26 22:12:00.122995475 (58h36m9.845558505s) [upload=56238]: Starting upload Jul 26 22:12:00.123061241 (58h36m9.845624351s) [upload=56238]: Done starting upload Jul 26 22:12:00.123134752 (58h36m9.845697782s) [upload=56238]: Uploading segments... Jul 26 22:12:00.123493779 (58h36m9.846056819s) [upload=56238]: Got next segment. Inline: false Jul 26 22:12:00.123543474 (58h36m9.846106514s) [upload=56238]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:12:00.123548433 (58h36m9.846111473s) [upload=56238]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:12:00.355422881 (58h36m10.077985911s) [seg_pos={0 0}, upload=56238]: Begin upload segment... Jul 26 22:12:00.355465902 (58h36m10.078028942s) [seg_pos={0 0}, upload=56238]: Done begin upload segment. Jul 26 22:12:00.355487364 (58h36m10.078050404s) [noise=true , address=104.36.82.5:4837 , piece_id=WTBNXCZ3PWC5TQM4UWPTKT5QRXHHWQF6JVQQ7VQ4BXKLPEIZMIFQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.355513444 (58h36m10.078076474s) [noise=true , address=45.91.116.9:4263 , piece_id=H67DGXYB4V7UFZAIE4X4VRVWU53RIIQQOWOANN52TAZHFJNQT4IA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.355843715 (58h36m10.078406746s) [noise=true , address=45.91.116.8:4045 , piece_id=PRKLVJLWB37FDN52QBLNATG2SIWCXJCN7OR6EFLRZL74JZLC57VA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.355985106 (58h36m10.078548146s) [noise=true , address=77.90.31.19:4843 , piece_id=P3UAQQ2AO27TVMESOSBZUUXGX6QOBWZPXOMG4LIAUHTSBDN3YBJQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356118291 (58h36m10.078681331s) [noise=true , address=45.91.116.4:4939 , piece_id=XJUAUMJSYPFXTK7JIIJECLKWJOTWJQHYJHQ7TYPHMPBXRBS3HFPA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356442141 (58h36m10.079005181s) [noise=true , address=104.36.82.4:4802 , piece_id=KQ4ABYCX4T6VM477B56ETRPXTVFWDZXUTZEGPNAWIQ7ZV2XPYEIQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356714532 (58h36m10.079277572s) [noise=true , address=45.91.116.12:4225 , piece_id=E2D6FTGWZ22OFNFPRWBE335BANIUZIWVS433SMUC2LPMD6PTI6IA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.356937989 (58h36m10.079501029s) [noise=true , address=45.91.116.13:4155 , piece_id=OXPGOXKCI4YOIKSIMPXVV6D74BBVMRPXXZKBBX2P7D4YQVR73VXQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357235789 (58h36m10.079798819s) [noise=true , address=77.90.31.12:4592 , piece_id=O3UI7JY3LQTABYKN3JKO5O4Z4RCLAX6HJZS4HQXJPGKDU3JNECTQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357472842 (58h36m10.080035882s) [noise=true , address=45.91.116.10:4082 , piece_id=J6PNAFSZBQCDVTAIQBHAMMN23AS7X3WLGV3YHVAKDU26RMWGKBLQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357542045 (58h36m10.080105085s) [noise=true , address=45.91.116.9:4262 , piece_id=TPC2GAHLX2R2B3ZZBTFYGWNMPYCX7KMZLNMNVM4GKQDOY6ADGUVQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.357791683 (58h36m10.080354723s) [noise=true , address=45.91.116.5:4298 , piece_id=HURPSTTY5SXKPKJIHWPQ6VGKWQPIGSQONUMRQM6BREDGEA5V5ALA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358046780 (58h36m10.080609831s) [noise=true , address=104.36.82.15:4258 , piece_id=MJO75LPAAYCYPVPZYKBASP7ZN7Q3FI44W3XRN3CS5JAARSD32LXA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358143686 (58h36m10.080706716s) [noise=true , address=45.91.116.6:4973 , piece_id=E4B6O574VP2Y3J6PNPKTWAN7BFGSHIVQEPMTBDXQCDIORGOFMU6A, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358493164 (58h36m10.081056204s) [noise=true , address=104.36.82.12:4149 , piece_id=6ZXU4V2YOW3IXGT5XYX3Q7FT4U6YW7IVA72UPFHYUIUBWNPBDOIQ, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:00.358579300 (58h36m10.08114233s) [noise=true , address=45.91.116.14:4190 , piece_id=PVZ2OCKX3JTUYT6KNUGIRHIYH5YVBSDMIZUOYSYSYBYB6AUNMOOA, seg_pos={0 0}, upload=56238]: Uploading piece... Jul 26 22:12:02.081979808 (58h36m11.804542868s) [upload=56238]: Next returned nil segment Jul 26 22:12:02.082008563 (58h36m11.804571643s) [upload=56238]: Waiting for error group managing segments... Jul 26 22:12:02.373463127 (58h36m12.096026167s) [noise=true , address=45.91.116.5:4298 , piece_id=HURPSTTY5SXKPKJIHWPQ6VGKWQPIGSQONUMRQM6BREDGEA5V5ALA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.391294504 (58h36m12.113857544s) [noise=true , address=77.90.31.12:4592 , piece_id=O3UI7JY3LQTABYKN3JKO5O4Z4RCLAX6HJZS4HQXJPGKDU3JNECTQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.392103922 (58h36m12.114666972s) [noise=true , address=45.91.116.9:4263 , piece_id=H67DGXYB4V7UFZAIE4X4VRVWU53RIIQQOWOANN52TAZHFJNQT4IA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.395004321 (58h36m12.117567361s) [noise=true , address=45.91.116.12:4225 , piece_id=E2D6FTGWZ22OFNFPRWBE335BANIUZIWVS433SMUC2LPMD6PTI6IA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.395744818 (58h36m12.118307858s) [noise=true , address=45.91.116.9:4262 , piece_id=TPC2GAHLX2R2B3ZZBTFYGWNMPYCX7KMZLNMNVM4GKQDOY6ADGUVQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.451177047 (58h36m12.173740097s) [noise=true , address=45.91.116.8:4045 , piece_id=PRKLVJLWB37FDN52QBLNATG2SIWCXJCN7OR6EFLRZL74JZLC57VA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.462789285 (58h36m12.185352325s) [noise=true , address=45.91.116.4:4939 , piece_id=XJUAUMJSYPFXTK7JIIJECLKWJOTWJQHYJHQ7TYPHMPBXRBS3HFPA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.477161623 (58h36m12.199724674s) [noise=true , address=45.91.116.13:4155 , piece_id=OXPGOXKCI4YOIKSIMPXVV6D74BBVMRPXXZKBBX2P7D4YQVR73VXQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.477354983 (58h36m12.199918023s) [noise=true , address=45.91.116.10:4082 , piece_id=J6PNAFSZBQCDVTAIQBHAMMN23AS7X3WLGV3YHVAKDU26RMWGKBLQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.482097287 (58h36m12.204660327s) [noise=true , address=45.91.116.6:4973 , piece_id=E4B6O574VP2Y3J6PNPKTWAN7BFGSHIVQEPMTBDXQCDIORGOFMU6A, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.522569814 (58h36m12.245132854s) [noise=true , address=77.90.31.19:4843 , piece_id=P3UAQQ2AO27TVMESOSBZUUXGX6QOBWZPXOMG4LIAUHTSBDN3YBJQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:02.537542521 (58h36m12.260105571s) [noise=true , address=45.91.116.14:4190 , piece_id=PVZ2OCKX3JTUYT6KNUGIRHIYH5YVBSDMIZUOYSYSYBYB6AUNMOOA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: <nil> Jul 26 22:12:29.482251815 (58h36m39.204814855s) [noise=true , address=104.36.82.5:4837 , piece_id=WTBNXCZ3PWC5TQM4UWPTKT5QRXHHWQF6JVQQ7VQ4BXKLPEIZMIFQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 188.166.225.170:38046->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482276953 (58h36m39.204839993s) [noise=true , address=104.36.82.4:4802 , piece_id=KQ4ABYCX4T6VM477B56ETRPXTVFWDZXUTZEGPNAWIQ7ZV2XPYEIQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482340875 (58h36m39.204903915s) [noise=true , address=104.36.82.15:4258 , piece_id=MJO75LPAAYCYPVPZYKBASP7ZN7Q3FI44W3XRN3CS5JAARSD32LXA, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: write tcp 188.166.225.170:41258->104.36.82.15:4258: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482356565 (58h36m39.204919605s) [noise=true , address=104.36.82.12:4149 , piece_id=6ZXU4V2YOW3IXGT5XYX3Q7FT4U6YW7IVA72UPFHYUIUBWNPBDOIQ, seg_pos={0 0}, upload=56238]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 188.166.225.170:39256->104.36.82.12:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:12:29.482383747 (58h36m39.204946787s) [seg_pos={0 0}, upload=56238]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:12:29.709832457 (58h36m39.432395497s) [upload=56238]: Done uploading segments... Jul 26 22:12:29.709874738 (58h36m39.432437768s) [upload=56238]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.718245947s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 52.870611015s FAIL (2025-07-26 22:23:55 UTC) orbiter warning: operation error S3: GetObject, exceeded maximum number of attempts, 3, https response error StatusCode: 500, RequestID: 1855ED318630ED12, HostID: , api error InternalError: We encountered an internal error, please try again.: cause(uplink: metaclient: unable to create order limits) Duration: 52.870632677s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......................X..X...........X......................] duration: 5.994241639s FAIL (2025-07-26 22:23:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.994264023s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 53 out of 60 is failed (threshold=3) [.......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.232693437s FAIL (2025-07-26 22:23:24 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== ece... Jul 26 22:22:25.537397386 (58h46m35.259960416s) [noise=true , address=104.36.82.5:4853 , piece_id=O7IZCW4OQS3MUYEIF4WSTO3BQMUMB5GE4VZXM5UJPVWALIANWZGA, seg_pos={0 0}, upload=56405]: Uploading piece... Jul 26 22:22:25.537441770 (58h46m35.2600048s) [noise=true , address=104.36.82.10:4090 , piece_id=PBR5CVYTAS26E3UINTYNBMZVYMTH2RI7CCJ5BMNLARD2RYHDHAAA, seg_pos={0 0}, upload=56405]: Uploading piece... Jul 26 22:22:25.537643467 (58h46m35.260206507s) [noise=true , address=79.127.219.51:4212 , piece_id=MA3DJ6ORXGWD2TWD6KWAWPUWVI272SP2RONGKO72YOS7WTLRX2CA, seg_pos={0 0}, upload=56405]: Uploading piece... Jul 26 22:22:25.537745431 (58h46m35.260308471s) [noise=true , address=45.91.116.9:4267 , piece_id=BXJGR6J2WRGNVIUWWYUPHT74JP2VCS5Z65IFBEUFBHUHWELJQQRA, seg_pos={0 0}, upload=56405]: Uploading piece... Jul 26 22:22:25.537917170 (58h46m35.26048021s) [noise=true , address=79.127.219.38:4001 , piece_id=25V3MZW5X5TX722LPWN3YSBQ6YVJQOSLJ2C5B3E5CLFB2VP5AX4Q, seg_pos={0 0}, upload=56405]: Uploading piece... Jul 26 22:22:25.537997574 (58h46m35.260560614s) [noise=true , address=156.146.43.225:4434 , piece_id=ZHPR5WUPHHYOHYUSRBD6JSJA7EPHLIA4JG5YORE4FAAVKZ5JBIEA, seg_pos={0 0}, upload=56405]: Uploading piece... Jul 26 22:22:25.538240909 (58h46m35.260803949s) [noise=true , address=79.127.219.47:4069 , piece_id=KBJKWO6IYEJ73N4QSAPAUXXI2QDDNRLLEOGBQTJWCWBNPXEVEVMA, seg_pos={0 0}, upload=56405]: Uploading piece... Jul 26 22:22:25.538516717 (58h46m35.261079757s) [noise=true , address=104.36.82.11:4147 , piece_id=DRDND4FEV2JRCUWXLP2T27QC7XPSI3CIIU6UPT2ULQA7B2R5XJYQ, seg_pos={0 0}, upload=56405]: Uploading piece... Jul 26 22:22:25.538739574 (58h46m35.261302614s) [noise=true , address=104.36.82.12:4157 , piece_id=PO2M57BMNDE7EF6IJAZL7MKRXZH6T5SE4FF467246QEZZ7WDARAA, seg_pos={0 0}, upload=56405]: Uploading piece... Jul 26 22:22:25.538966408 (58h46m35.261529448s) [noise=true , address=79.127.201.215:4348 , piece_id=Z4HWXKKGEMXSBJCSUTQNXTNI2RQXETSGOT44GQKESS5IWIPMOL5Q, seg_pos={0 0}, upload=56405]: Uploading piece... Jul 26 22:22:25.539204122 (58h46m35.261767152s) [noise=true , address=104.36.82.7:4916 , piece_id=WNIN4YLBIRVVRBI7IEHVEK746GAOZQTFUQUF35ZQ6BGWCONH3SKQ, seg_pos={0 0}, upload=56405]: Uploading piece... Jul 26 22:22:25.539412511 (58h46m35.261975551s) [noise=true , address=77.90.31.17:4801 , piece_id=K4GQKHZW22LPAGOJLBYU5LWNCBENH4U6XBKWVKATVRYTOOP5VSFQ, seg_pos={0 0}, upload=56405]: Uploading piece... Jul 26 22:22:25.539673731 (58h46m35.262236761s) [noise=true , address=104.36.82.42:4809 , piece_id=IC3LJZFPUS67I4NXXRMIPLHCPZ4JL3YFKJS2GQS6NRUA2Y2XOFLA, seg_pos={0 0}, upload=56405]: Uploading piece... Jul 26 22:22:27.057813139 (58h46m36.780376179s) [upload=56405]: Next returned nil segment Jul 26 22:22:27.057833196 (58h46m36.780396236s) [upload=56405]: Waiting for error group managing segments... Jul 26 22:22:27.540530195 (58h46m37.263093235s) [noise=true , address=77.90.31.16:4759 , piece_id=FVGECBP3ZH7RT5IRD3B3TPZDW3OOTPEWMEEOBI2P2LQBTVDY3HRA, seg_pos={0 0}, upload=56405]: Done uploading piece. err: <nil> Jul 26 22:22:27.734198634 (58h46m37.456761674s) [noise=true , address=79.127.219.38:4001 , piece_id=25V3MZW5X5TX722LPWN3YSBQ6YVJQOSLJ2C5B3E5CLFB2VP5AX4Q, seg_pos={0 0}, upload=56405]: Done uploading piece. err: <nil> Jul 26 22:22:28.594129421 (58h46m38.316692561s) [noise=true , address=45.91.116.9:4267 , piece_id=BXJGR6J2WRGNVIUWWYUPHT74JP2VCS5Z65IFBEUFBHUHWELJQQRA, seg_pos={0 0}, upload=56405]: Done uploading piece. err: ecclient: upload failed (node:1p6M5TgGqUwjXyM25uqv5hsh67mS81ByzMM4kWFUNaFvisDN9d, address:45.91.116.9:4267): protocol: storage node overloaded, request limit: 2000 Jul 26 22:22:29.983982629 (58h46m39.706545669s) [noise=true , address=156.146.43.225:4434 , piece_id=ZHPR5WUPHHYOHYUSRBD6JSJA7EPHLIA4JG5YORE4FAAVKZ5JBIEA, seg_pos={0 0}, upload=56405]: Done uploading piece. err: <nil> Jul 26 22:22:30.754303760 (58h46m40.47686685s) [noise=true , address=79.127.219.51:4212 , piece_id=MA3DJ6ORXGWD2TWD6KWAWPUWVI272SP2RONGKO72YOS7WTLRX2CA, seg_pos={0 0}, upload=56405]: Done uploading piece. err: <nil> Jul 26 22:22:31.106372341 (58h46m40.828935411s) [noise=true , address=79.127.219.47:4069 , piece_id=KBJKWO6IYEJ73N4QSAPAUXXI2QDDNRLLEOGBQTJWCWBNPXEVEVMA, seg_pos={0 0}, upload=56405]: Done uploading piece. err: <nil> Jul 26 22:22:33.021379208 (58h46m42.743942248s) [noise=true , address=79.127.201.215:4348 , piece_id=Z4HWXKKGEMXSBJCSUTQNXTNI2RQXETSGOT44GQKESS5IWIPMOL5Q, seg_pos={0 0}, upload=56405]: Done uploading piece. err: <nil> Jul 26 22:23:24.431394761 (58h47m34.153957801s) [noise=true , address=77.90.31.17:4801 , piece_id=K4GQKHZW22LPAGOJLBYU5LWNCBENH4U6XBKWVKATVRYTOOP5VSFQ, seg_pos={0 0}, upload=56405]: Done uploading piece. err: ecclient: upload failed (node:127QRLNwZLcR5gnAgZrCjSUsH5T4jESmZ4kWeAnpj9YxaTSjoYe, address:77.90.31.17:4801): protocol: write tcp 188.166.225.170:48086->77.90.31.17:4801: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:23:24.431460417 (58h47m34.154023457s) [noise=true , address=104.36.82.42:4809 , piece_id=IC3LJZFPUS67I4NXXRMIPLHCPZ4JL3YFKJS2GQS6NRUA2Y2XOFLA, seg_pos={0 0}, upload=56405]: Done uploading piece. err: ecclient: upload failed (node:1VZgucaa2gRuFhNxARMb7BDS6EGZXKChGPppbFksbPc5we7fTQ, address:104.36.82.42:4809): protocol: write tcp 188.166.225.170:53788->104.36.82.42:4809: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:23:24.431521283 (58h47m34.154084353s) [noise=true , address=104.36.82.10:4090 , piece_id=PBR5CVYTAS26E3UINTYNBMZVYMTH2RI7CCJ5BMNLARD2RYHDHAAA, seg_pos={0 0}, upload=56405]: Done uploading piece. err: ecclient: upload failed (node:12qSkJk9HUyQon3UP1AA8mRcqJbNdkKJh6cEUZoHLGNniqDvNqB, address:104.36.82.10:4090): protocol: rpc: dial tcp 104.36.82.10:4090: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:23:24.431557733 (58h47m34.154120773s) [noise=true , address=104.36.82.11:4147 , piece_id=DRDND4FEV2JRCUWXLP2T27QC7XPSI3CIIU6UPT2ULQA7B2R5XJYQ, seg_pos={0 0}, upload=56405]: Done uploading piece. err: ecclient: upload failed (node:12nNFCM4wDSvGMeqF8MF6csgKDUV6QyozoC5fZwUaj7MN1Woz4P, address:104.36.82.11:4147): protocol: rpc: dial tcp 104.36.82.11:4147: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:23:24.431604593 (58h47m34.154167623s) [noise=true , address=104.36.82.7:4916 , piece_id=WNIN4YLBIRVVRBI7IEHVEK746GAOZQTFUQUF35ZQ6BGWCONH3SKQ, seg_pos={0 0}, upload=56405]: Done uploading piece. err: ecclient: upload failed (node:12fifEPABGWmVzxBHNzhjh3r4ue3RRSeNdydifpXruXtH4FELBK, address:104.36.82.7:4916): protocol: write tcp 188.166.225.170:57742->104.36.82.7:4916: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:23:24.431639609 (58h47m34.154202649s) [noise=true , address=104.36.82.4:4823 , piece_id=XHXVZHJYWWYHMD7DHWYBUC7XD6MUA2FCU5Z5AX4GGBHEEAWZTMUQ, seg_pos={0 0}, upload=56405]: Done uploading piece. err: ecclient: upload failed (node:1JWhB5QxbZsTD863tEfkEN1CUbKBMQvMJ9ZaiMAwKvxxXyDkEq, address:104.36.82.4:4823): protocol: rpc: dial tcp 104.36.82.4:4823: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:23:24.431686670 (58h47m34.15424977s) [noise=true , address=104.36.82.12:4172 , piece_id=QG6U6GZZM4LBGIDQMUD3V5H5URA3KY2UJ6CV52I74WYLMKV52I6A, seg_pos={0 0}, upload=56405]: Done uploading piece. err: ecclient: upload failed (node:1xfh8U6X8B8bpXvyX7JHtAGY6zsApfjQtZghScuo7BerzDj4yj, address:104.36.82.12:4172): protocol: rpc: dial tcp 104.36.82.12:4172: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:23:24.431743017 (58h47m34.154306057s) [noise=true , address=104.36.82.5:4853 , piece_id=O7IZCW4OQS3MUYEIF4WSTO3BQMUMB5GE4VZXM5UJPVWALIANWZGA, seg_pos={0 0}, upload=56405]: Done uploading piece. err: ecclient: upload failed (node:1ycsU8APac3LBkfY1xCsc8b9KJQVAx5kFJnYebWhKSKSbKyf5u, address:104.36.82.5:4853): protocol: rpc: dial tcp 104.36.82.5:4853: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:23:24.431780328 (58h47m34.154343368s) [noise=true , address=104.36.82.35:4614 , piece_id=T652T54FRTWGCMDBKIC2N5FB6ORHUVOF7O35KJMPTSBO4T2OBRRA, seg_pos={0 0}, upload=56405]: Done uploading piece. err: ecclient: upload failed (node:12Pua5x8rAsuqYWCgoAY6fgy2SXNoDjzt7WwgMhzv1HN2KG5NY4, address:104.36.82.35:4614): protocol: rpc: dial tcp 104.36.82.35:4614: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:23:24.431823321 (58h47m34.154386351s) [noise=true , address=104.36.82.10:4089 , piece_id=EEUWHIRJ4CPXUMXQW4YRVV7SGTHAIPK77WDSJ54B4EGQEZ5K6UMQ, seg_pos={0 0}, upload=56405]: Done uploading piece. err: ecclient: upload failed (node:12bFi2opJ6FejAMwSL2qbqWoCk24QzSjT6xQYE8BytLgVJikZMT, address:104.36.82.10:4089): protocol: write tcp 188.166.225.170:56840->104.36.82.10:4089: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:23:24.431868418 (58h47m34.154431448s) [noise=true , address=104.36.82.12:4157 , piece_id=PO2M57BMNDE7EF6IJAZL7MKRXZH6T5SE4FF467246QEZZ7WDARAA, seg_pos={0 0}, upload=56405]: Done uploading piece. err: ecclient: upload failed (node:12vPyYh3yrhc3zt4nd5C6TjHGaMAHGcpVwLTcgYUqCKGhAvicsX, address:104.36.82.12:4157): protocol: write tcp 188.166.225.170:35432->104.36.82.12:4157: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:23:24.431901711 (58h47m34.154464741s) [seg_pos={0 0}, upload=56405]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Jul 26 22:23:24.662008447 (58h47m34.384571487s) [upload=56405]: Done uploading segments... Jul 26 22:23:24.662037873 (58h47m34.384600923s) [upload=56405]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 1m0.23272148s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 25 out of 60 is failed (threshold=3) [.......XX.XX.XXXX.XX.XXXXX.X.XXXXX.XXX.....X................] duration: 5.724627013s FAIL (2025-07-26 22:23:18 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.724652851s core uplink us1 vivint

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)