Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [...X...............X........X...X..XX.......................] duration: 6.019206965s FAIL (2025-07-29 13:21:08 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 14 but got 7) Duration: 6.019238175s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [....................................X.X.....................] duration: 6.075555909s FAIL (2025-07-29 13:21:01 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ding segments... Jul 29 13:03:41.091320246 (20h17m53.567258701s) [upload=19386]: Got next segment. Inline: false Jul 29 13:03:41.091349191 (20h17m53.567287656s) [upload=19386]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 29 13:03:41.091354331 (20h17m53.567292796s) [upload=19386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 29 13:03:41.616444845 (20h17m54.09238331s) [seg_pos={0 0}, upload=19386]: Begin upload segment... Jul 29 13:03:41.616487748 (20h17m54.092426213s) [seg_pos={0 0}, upload=19386]: Done begin upload segment. Jul 29 13:03:41.616507376 (20h17m54.092445841s) [noise=true , address=104.36.82.15:4263 , piece_id=3GCLYN4UBCLM5BPLXDQT5SSWPJBTKIOZM4UIUNO3OZIXW6OANUBQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.616564315 (20h17m54.09250278s) [noise=true , address=104.36.82.11:4119 , piece_id=6KWLABSUMX6YVKD6LGKR6ONLQKFFTSTKFHHRVFIN7U4KDALBNCBQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.616866433 (20h17m54.092804888s) [noise=true , address=104.36.82.54:4175 , piece_id=5PAE3BM5AFSLZ5TVUQC3QK3RVWCVGFDLOK452BDPWQ7ARTL7EYWQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617072557 (20h17m54.093011022s) [noise=true , address=104.36.82.56:4242 , piece_id=BHKC5MH4G65BK2YLJJANIYZ4MLAUTMBG32XSRISARHCO26AZUXRA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617110629 (20h17m54.093049084s) [noise=true , address=104.36.82.5:4866 , piece_id=EFRLC4LA232NZZJU3GDSP2EEBZO3W2LGJRD3L2A6SPSRBTLUTIFA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617355037 (20h17m54.093293492s) [noise=true , address=104.36.82.61:4391 , piece_id=TI5YDGDSTWKDX5XAWL36F3TJDQNHKNTREZVTUGIS2EVZYP6DWSDA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617481549 (20h17m54.093420024s) [noise=true , address=104.36.82.20:4448 , piece_id=CSHYRDGTS4ROFDTKETIZ36TWJE3OEMAKLB4RGFAU557G725EANXQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617583044 (20h17m54.093521509s) [noise=true , address=104.36.82.47:4545 , piece_id=PZWVIOWLBZ72IOFRJHPAD4OZZPA2G4XYCDLGYEY7THSOR7XDHMUQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617828073 (20h17m54.093766528s) [noise=true , address=104.36.82.42:4802 , piece_id=6HORHYYKIKDX4IWWKUQPVGZQZ2YYIGC47OCZHRVBC4BZFRJX5TQQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617987597 (20h17m54.093926062s) [noise=true , address=104.36.82.13:4214 , piece_id=SLX3YJCEWCM3553VZHILH337FKSPPFNKVFTPXO4KIBOS6IXCLXHA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618055948 (20h17m54.093994403s) [noise=true , address=104.36.82.46:4500 , piece_id=VCYKKDSLEKTTL3KO2FJ4VA5JGIVPA4GFBBHH65HPFZTP4REUTHYQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618285828 (20h17m54.094224293s) [noise=true , address=104.36.82.52:4092 , piece_id=BMJCAQYXJCUVJTEXH52UXYLD4MHWLDGQTJIPDOFRF4LPLTT36ESA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618485871 (20h17m54.094424336s) [noise=true , address=104.36.82.10:4088 , piece_id=6IV2Z7FXDNEOBJMNPRG7WSPZWZTJ5CGDF6JHBSF7WCIIL2QQWFUA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618663881 (20h17m54.094602346s) [noise=true , address=104.36.82.51:4075 , piece_id=P5NU32M3V7AMYFZVJ6XZY5K2YMQWF4EULQGX6RRYWG6SWDWWAFMQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618840449 (20h17m54.094778914s) [noise=true , address=104.36.82.14:4227 , piece_id=O2FXJBOV3CJCKLZHRPBTLXX2QRT2OWJQ5HPNHZI5IVIFDVUUICVQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.619028879 (20h17m54.094967344s) [noise=true , address=104.36.82.9:4028 , piece_id=XCRNGUPBFYCZOYT23L6NHJ6WQR55TQMTWZF4SNKSRLQGC5TYO3LA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.619215837 (20h17m54.095154302s) [noise=true , address=104.36.82.53:4126 , piece_id=GZ5Z7AGMYVGYVUJAWFAIS6SQLRHFB772RLQY736QUGOXA5OWQ34Q, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.619408406 (20h17m54.095346871s) [noise=true , address=104.36.82.55:4219 , piece_id=IURHWONYUYYJWIL5BDF6HLCEPA76O6QAZAJ22NCI6LQYIGO6K4IA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:43.299790904 (20h17m55.775729369s) [upload=19386]: Next returned nil segment Jul 29 13:03:43.299806024 (20h17m55.775744479s) [upload=19386]: Waiting for error group managing segments... Jul 29 13:03:43.847040928 (20h17m56.322979393s) [noise=true , address=104.36.82.9:4028 , piece_id=XCRNGUPBFYCZOYT23L6NHJ6WQR55TQMTWZF4SNKSRLQGC5TYO3LA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:43.879495404 (20h17m56.355433869s) [noise=true , address=104.36.82.61:4391 , piece_id=TI5YDGDSTWKDX5XAWL36F3TJDQNHKNTREZVTUGIS2EVZYP6DWSDA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:46.350727192 (20h17m58.826665657s) [noise=true , address=104.36.82.5:4866 , piece_id=EFRLC4LA232NZZJU3GDSP2EEBZO3W2LGJRD3L2A6SPSRBTLUTIFA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:51.702944756 (20h18m4.178883221s) [noise=true , address=104.36.82.42:4802 , piece_id=6HORHYYKIKDX4IWWKUQPVGZQZ2YYIGC47OCZHRVBC4BZFRJX5TQQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:55.250827413 (20h18m7.726765878s) [noise=true , address=104.36.82.55:4219 , piece_id=IURHWONYUYYJWIL5BDF6HLCEPA76O6QAZAJ22NCI6LQYIGO6K4IA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:57.858469692 (20h18m10.334408157s) [noise=true , address=104.36.82.53:4126 , piece_id=GZ5Z7AGMYVGYVUJAWFAIS6SQLRHFB772RLQY736QUGOXA5OWQ34Q, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:59.265385056 (20h18m11.741323511s) [noise=true , address=104.36.82.10:4088 , piece_id=6IV2Z7FXDNEOBJMNPRG7WSPZWZTJ5CGDF6JHBSF7WCIIL2QQWFUA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:59.973580263 (20h18m12.449518729s) [noise=true , address=104.36.82.11:4119 , piece_id=6KWLABSUMX6YVKD6LGKR6ONLQKFFTSTKFHHRVFIN7U4KDALBNCBQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:16.002200978 (20h18m28.478139453s) [noise=true , address=104.36.82.52:4092 , piece_id=BMJCAQYXJCUVJTEXH52UXYLD4MHWLDGQTJIPDOFRF4LPLTT36ESA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:30.906580908 (20h18m43.382519373s) [noise=true , address=104.36.82.20:4448 , piece_id=CSHYRDGTS4ROFDTKETIZ36TWJE3OEMAKLB4RGFAU557G725EANXQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:31.164952501 (20h18m43.640890966s) [noise=true , address=104.36.82.14:4227 , piece_id=O2FXJBOV3CJCKLZHRPBTLXX2QRT2OWJQ5HPNHZI5IVIFDVUUICVQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:33.000876204 (20h18m45.476814679s) [noise=true , address=104.36.82.56:4242 , piece_id=BHKC5MH4G65BK2YLJJANIYZ4MLAUTMBG32XSRISARHCO26AZUXRA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:34.725342452 (20h18m47.201280907s) [noise=true , address=104.36.82.47:4545 , piece_id=PZWVIOWLBZ72IOFRJHPAD4OZZPA2G4XYCDLGYEY7THSOR7XDHMUQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:40.451024021 (20h18m52.926962486s) [noise=true , address=104.36.82.46:4500 , piece_id=VCYKKDSLEKTTL3KO2FJ4VA5JGIVPA4GFBBHH65HPFZTP4REUTHYQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:12B7ox4ehqxJ2mwqeXj8C9N9GuSU5etNC23iT1pJnTgMLGBAGQK, address:104.36.82.46:4500): protocol: write tcp 188.166.225.170:46700->104.36.82.46:4500: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451060500 (20h18m52.926998965s) [noise=true , address=104.36.82.51:4075 , piece_id=P5NU32M3V7AMYFZVJ6XZY5K2YMQWF4EULQGX6RRYWG6SWDWWAFMQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK, address:104.36.82.51:4075): protocol: write tcp 188.166.225.170:41162->104.36.82.51:4075: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451086179 (20h18m52.927024644s) [noise=true , address=104.36.82.54:4175 , piece_id=5PAE3BM5AFSLZ5TVUQC3QK3RVWCVGFDLOK452BDPWQ7ARTL7EYWQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:12ou7jwFyJP3mNoALddniUTDT9duVyQKZjeq4xHCyx6sRWpwEPJ, address:104.36.82.54:4175): protocol: rpc: dial tcp 104.36.82.54:4175: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451120024 (20h18m52.927058489s) [noise=true , address=104.36.82.13:4214 , piece_id=SLX3YJCEWCM3553VZHILH337FKSPPFNKVFTPXO4KIBOS6IXCLXHA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:1nJ8uqwxf6x4hX52AjAdCbMKi1BKGGbMvrQbiazJdLpn6j69ay, address:104.36.82.13:4214): protocol: rpc: dial tcp 104.36.82.13:4214: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451147116 (20h18m52.927085581s) [noise=true , address=104.36.82.15:4263 , piece_id=3GCLYN4UBCLM5BPLXDQT5SSWPJBTKIOZM4UIUNO3OZIXW6OANUBQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:1f97yBEvcrYHqMq7iovVoju77jzhUYgoQCsUGULeTagugFXrCu, address:104.36.82.15:4263): protocol: write tcp 188.166.225.170:39380->104.36.82.15:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451167805 (20h18m52.927106271s) [seg_pos={0 0}, upload=19386]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 29 13:04:40.696628655 (20h18m53.17256714s) [upload=19386]: Done uploading segments... Jul 29 13:04:40.696658391 (20h18m53.172596856s) [upload=19386]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.075580136s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.........X...........X......................................] duration: 1.832413733s FAIL (2025-07-29 13:21:00 UTC) orbiter warning: unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K Duration: 1.832440472s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [....X...............X........X...X..XX......................] duration: 5.852658313s FAIL (2025-07-29 13:20:51 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 14 but got 7) Duration: 5.852684262s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [.....X...............X........X...X..XX.....................] duration: 5.949629788s FAIL (2025-07-29 13:19:59 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 14 but got 7) Duration: 5.949658703s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [..........X...........X.....................................] duration: 2.042369378s FAIL (2025-07-29 13:19:54 UTC) orbiter warning: unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K Duration: 2.04239646s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.....................................X.X....................] duration: 5.997608961s FAIL (2025-07-29 13:19:46 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ding segments... Jul 29 13:03:41.091320246 (20h17m53.567258701s) [upload=19386]: Got next segment. Inline: false Jul 29 13:03:41.091349191 (20h17m53.567287656s) [upload=19386]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 29 13:03:41.091354331 (20h17m53.567292796s) [upload=19386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 29 13:03:41.616444845 (20h17m54.09238331s) [seg_pos={0 0}, upload=19386]: Begin upload segment... Jul 29 13:03:41.616487748 (20h17m54.092426213s) [seg_pos={0 0}, upload=19386]: Done begin upload segment. Jul 29 13:03:41.616507376 (20h17m54.092445841s) [noise=true , address=104.36.82.15:4263 , piece_id=3GCLYN4UBCLM5BPLXDQT5SSWPJBTKIOZM4UIUNO3OZIXW6OANUBQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.616564315 (20h17m54.09250278s) [noise=true , address=104.36.82.11:4119 , piece_id=6KWLABSUMX6YVKD6LGKR6ONLQKFFTSTKFHHRVFIN7U4KDALBNCBQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.616866433 (20h17m54.092804888s) [noise=true , address=104.36.82.54:4175 , piece_id=5PAE3BM5AFSLZ5TVUQC3QK3RVWCVGFDLOK452BDPWQ7ARTL7EYWQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617072557 (20h17m54.093011022s) [noise=true , address=104.36.82.56:4242 , piece_id=BHKC5MH4G65BK2YLJJANIYZ4MLAUTMBG32XSRISARHCO26AZUXRA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617110629 (20h17m54.093049084s) [noise=true , address=104.36.82.5:4866 , piece_id=EFRLC4LA232NZZJU3GDSP2EEBZO3W2LGJRD3L2A6SPSRBTLUTIFA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617355037 (20h17m54.093293492s) [noise=true , address=104.36.82.61:4391 , piece_id=TI5YDGDSTWKDX5XAWL36F3TJDQNHKNTREZVTUGIS2EVZYP6DWSDA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617481549 (20h17m54.093420024s) [noise=true , address=104.36.82.20:4448 , piece_id=CSHYRDGTS4ROFDTKETIZ36TWJE3OEMAKLB4RGFAU557G725EANXQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617583044 (20h17m54.093521509s) [noise=true , address=104.36.82.47:4545 , piece_id=PZWVIOWLBZ72IOFRJHPAD4OZZPA2G4XYCDLGYEY7THSOR7XDHMUQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617828073 (20h17m54.093766528s) [noise=true , address=104.36.82.42:4802 , piece_id=6HORHYYKIKDX4IWWKUQPVGZQZ2YYIGC47OCZHRVBC4BZFRJX5TQQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617987597 (20h17m54.093926062s) [noise=true , address=104.36.82.13:4214 , piece_id=SLX3YJCEWCM3553VZHILH337FKSPPFNKVFTPXO4KIBOS6IXCLXHA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618055948 (20h17m54.093994403s) [noise=true , address=104.36.82.46:4500 , piece_id=VCYKKDSLEKTTL3KO2FJ4VA5JGIVPA4GFBBHH65HPFZTP4REUTHYQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618285828 (20h17m54.094224293s) [noise=true , address=104.36.82.52:4092 , piece_id=BMJCAQYXJCUVJTEXH52UXYLD4MHWLDGQTJIPDOFRF4LPLTT36ESA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618485871 (20h17m54.094424336s) [noise=true , address=104.36.82.10:4088 , piece_id=6IV2Z7FXDNEOBJMNPRG7WSPZWZTJ5CGDF6JHBSF7WCIIL2QQWFUA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618663881 (20h17m54.094602346s) [noise=true , address=104.36.82.51:4075 , piece_id=P5NU32M3V7AMYFZVJ6XZY5K2YMQWF4EULQGX6RRYWG6SWDWWAFMQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618840449 (20h17m54.094778914s) [noise=true , address=104.36.82.14:4227 , piece_id=O2FXJBOV3CJCKLZHRPBTLXX2QRT2OWJQ5HPNHZI5IVIFDVUUICVQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.619028879 (20h17m54.094967344s) [noise=true , address=104.36.82.9:4028 , piece_id=XCRNGUPBFYCZOYT23L6NHJ6WQR55TQMTWZF4SNKSRLQGC5TYO3LA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.619215837 (20h17m54.095154302s) [noise=true , address=104.36.82.53:4126 , piece_id=GZ5Z7AGMYVGYVUJAWFAIS6SQLRHFB772RLQY736QUGOXA5OWQ34Q, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.619408406 (20h17m54.095346871s) [noise=true , address=104.36.82.55:4219 , piece_id=IURHWONYUYYJWIL5BDF6HLCEPA76O6QAZAJ22NCI6LQYIGO6K4IA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:43.299790904 (20h17m55.775729369s) [upload=19386]: Next returned nil segment Jul 29 13:03:43.299806024 (20h17m55.775744479s) [upload=19386]: Waiting for error group managing segments... Jul 29 13:03:43.847040928 (20h17m56.322979393s) [noise=true , address=104.36.82.9:4028 , piece_id=XCRNGUPBFYCZOYT23L6NHJ6WQR55TQMTWZF4SNKSRLQGC5TYO3LA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:43.879495404 (20h17m56.355433869s) [noise=true , address=104.36.82.61:4391 , piece_id=TI5YDGDSTWKDX5XAWL36F3TJDQNHKNTREZVTUGIS2EVZYP6DWSDA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:46.350727192 (20h17m58.826665657s) [noise=true , address=104.36.82.5:4866 , piece_id=EFRLC4LA232NZZJU3GDSP2EEBZO3W2LGJRD3L2A6SPSRBTLUTIFA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:51.702944756 (20h18m4.178883221s) [noise=true , address=104.36.82.42:4802 , piece_id=6HORHYYKIKDX4IWWKUQPVGZQZ2YYIGC47OCZHRVBC4BZFRJX5TQQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:55.250827413 (20h18m7.726765878s) [noise=true , address=104.36.82.55:4219 , piece_id=IURHWONYUYYJWIL5BDF6HLCEPA76O6QAZAJ22NCI6LQYIGO6K4IA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:57.858469692 (20h18m10.334408157s) [noise=true , address=104.36.82.53:4126 , piece_id=GZ5Z7AGMYVGYVUJAWFAIS6SQLRHFB772RLQY736QUGOXA5OWQ34Q, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:59.265385056 (20h18m11.741323511s) [noise=true , address=104.36.82.10:4088 , piece_id=6IV2Z7FXDNEOBJMNPRG7WSPZWZTJ5CGDF6JHBSF7WCIIL2QQWFUA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:59.973580263 (20h18m12.449518729s) [noise=true , address=104.36.82.11:4119 , piece_id=6KWLABSUMX6YVKD6LGKR6ONLQKFFTSTKFHHRVFIN7U4KDALBNCBQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:16.002200978 (20h18m28.478139453s) [noise=true , address=104.36.82.52:4092 , piece_id=BMJCAQYXJCUVJTEXH52UXYLD4MHWLDGQTJIPDOFRF4LPLTT36ESA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:30.906580908 (20h18m43.382519373s) [noise=true , address=104.36.82.20:4448 , piece_id=CSHYRDGTS4ROFDTKETIZ36TWJE3OEMAKLB4RGFAU557G725EANXQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:31.164952501 (20h18m43.640890966s) [noise=true , address=104.36.82.14:4227 , piece_id=O2FXJBOV3CJCKLZHRPBTLXX2QRT2OWJQ5HPNHZI5IVIFDVUUICVQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:33.000876204 (20h18m45.476814679s) [noise=true , address=104.36.82.56:4242 , piece_id=BHKC5MH4G65BK2YLJJANIYZ4MLAUTMBG32XSRISARHCO26AZUXRA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:34.725342452 (20h18m47.201280907s) [noise=true , address=104.36.82.47:4545 , piece_id=PZWVIOWLBZ72IOFRJHPAD4OZZPA2G4XYCDLGYEY7THSOR7XDHMUQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:40.451024021 (20h18m52.926962486s) [noise=true , address=104.36.82.46:4500 , piece_id=VCYKKDSLEKTTL3KO2FJ4VA5JGIVPA4GFBBHH65HPFZTP4REUTHYQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:12B7ox4ehqxJ2mwqeXj8C9N9GuSU5etNC23iT1pJnTgMLGBAGQK, address:104.36.82.46:4500): protocol: write tcp 188.166.225.170:46700->104.36.82.46:4500: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451060500 (20h18m52.926998965s) [noise=true , address=104.36.82.51:4075 , piece_id=P5NU32M3V7AMYFZVJ6XZY5K2YMQWF4EULQGX6RRYWG6SWDWWAFMQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK, address:104.36.82.51:4075): protocol: write tcp 188.166.225.170:41162->104.36.82.51:4075: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451086179 (20h18m52.927024644s) [noise=true , address=104.36.82.54:4175 , piece_id=5PAE3BM5AFSLZ5TVUQC3QK3RVWCVGFDLOK452BDPWQ7ARTL7EYWQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:12ou7jwFyJP3mNoALddniUTDT9duVyQKZjeq4xHCyx6sRWpwEPJ, address:104.36.82.54:4175): protocol: rpc: dial tcp 104.36.82.54:4175: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451120024 (20h18m52.927058489s) [noise=true , address=104.36.82.13:4214 , piece_id=SLX3YJCEWCM3553VZHILH337FKSPPFNKVFTPXO4KIBOS6IXCLXHA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:1nJ8uqwxf6x4hX52AjAdCbMKi1BKGGbMvrQbiazJdLpn6j69ay, address:104.36.82.13:4214): protocol: rpc: dial tcp 104.36.82.13:4214: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451147116 (20h18m52.927085581s) [noise=true , address=104.36.82.15:4263 , piece_id=3GCLYN4UBCLM5BPLXDQT5SSWPJBTKIOZM4UIUNO3OZIXW6OANUBQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:1f97yBEvcrYHqMq7iovVoju77jzhUYgoQCsUGULeTagugFXrCu, address:104.36.82.15:4263): protocol: write tcp 188.166.225.170:39380->104.36.82.15:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451167805 (20h18m52.927106271s) [seg_pos={0 0}, upload=19386]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 29 13:04:40.696628655 (20h18m53.17256714s) [upload=19386]: Done uploading segments... Jul 29 13:04:40.696658391 (20h18m53.172596856s) [upload=19386]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.99764542s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [...........X...........X....................................] duration: 1.844388175s FAIL (2025-07-29 13:19:31 UTC) orbiter warning: unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K Duration: 1.844431498s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [......X...............X........X...X..XX....................] duration: 6.026664166s FAIL (2025-07-29 13:18:55 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 14 but got 7) Duration: 6.026689185s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [......................................X.X...................] duration: 5.844493866s FAIL (2025-07-29 13:18:32 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ding segments... Jul 29 13:03:41.091320246 (20h17m53.567258701s) [upload=19386]: Got next segment. Inline: false Jul 29 13:03:41.091349191 (20h17m53.567287656s) [upload=19386]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 29 13:03:41.091354331 (20h17m53.567292796s) [upload=19386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 29 13:03:41.616444845 (20h17m54.09238331s) [seg_pos={0 0}, upload=19386]: Begin upload segment... Jul 29 13:03:41.616487748 (20h17m54.092426213s) [seg_pos={0 0}, upload=19386]: Done begin upload segment. Jul 29 13:03:41.616507376 (20h17m54.092445841s) [noise=true , address=104.36.82.15:4263 , piece_id=3GCLYN4UBCLM5BPLXDQT5SSWPJBTKIOZM4UIUNO3OZIXW6OANUBQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.616564315 (20h17m54.09250278s) [noise=true , address=104.36.82.11:4119 , piece_id=6KWLABSUMX6YVKD6LGKR6ONLQKFFTSTKFHHRVFIN7U4KDALBNCBQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.616866433 (20h17m54.092804888s) [noise=true , address=104.36.82.54:4175 , piece_id=5PAE3BM5AFSLZ5TVUQC3QK3RVWCVGFDLOK452BDPWQ7ARTL7EYWQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617072557 (20h17m54.093011022s) [noise=true , address=104.36.82.56:4242 , piece_id=BHKC5MH4G65BK2YLJJANIYZ4MLAUTMBG32XSRISARHCO26AZUXRA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617110629 (20h17m54.093049084s) [noise=true , address=104.36.82.5:4866 , piece_id=EFRLC4LA232NZZJU3GDSP2EEBZO3W2LGJRD3L2A6SPSRBTLUTIFA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617355037 (20h17m54.093293492s) [noise=true , address=104.36.82.61:4391 , piece_id=TI5YDGDSTWKDX5XAWL36F3TJDQNHKNTREZVTUGIS2EVZYP6DWSDA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617481549 (20h17m54.093420024s) [noise=true , address=104.36.82.20:4448 , piece_id=CSHYRDGTS4ROFDTKETIZ36TWJE3OEMAKLB4RGFAU557G725EANXQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617583044 (20h17m54.093521509s) [noise=true , address=104.36.82.47:4545 , piece_id=PZWVIOWLBZ72IOFRJHPAD4OZZPA2G4XYCDLGYEY7THSOR7XDHMUQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617828073 (20h17m54.093766528s) [noise=true , address=104.36.82.42:4802 , piece_id=6HORHYYKIKDX4IWWKUQPVGZQZ2YYIGC47OCZHRVBC4BZFRJX5TQQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617987597 (20h17m54.093926062s) [noise=true , address=104.36.82.13:4214 , piece_id=SLX3YJCEWCM3553VZHILH337FKSPPFNKVFTPXO4KIBOS6IXCLXHA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618055948 (20h17m54.093994403s) [noise=true , address=104.36.82.46:4500 , piece_id=VCYKKDSLEKTTL3KO2FJ4VA5JGIVPA4GFBBHH65HPFZTP4REUTHYQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618285828 (20h17m54.094224293s) [noise=true , address=104.36.82.52:4092 , piece_id=BMJCAQYXJCUVJTEXH52UXYLD4MHWLDGQTJIPDOFRF4LPLTT36ESA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618485871 (20h17m54.094424336s) [noise=true , address=104.36.82.10:4088 , piece_id=6IV2Z7FXDNEOBJMNPRG7WSPZWZTJ5CGDF6JHBSF7WCIIL2QQWFUA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618663881 (20h17m54.094602346s) [noise=true , address=104.36.82.51:4075 , piece_id=P5NU32M3V7AMYFZVJ6XZY5K2YMQWF4EULQGX6RRYWG6SWDWWAFMQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618840449 (20h17m54.094778914s) [noise=true , address=104.36.82.14:4227 , piece_id=O2FXJBOV3CJCKLZHRPBTLXX2QRT2OWJQ5HPNHZI5IVIFDVUUICVQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.619028879 (20h17m54.094967344s) [noise=true , address=104.36.82.9:4028 , piece_id=XCRNGUPBFYCZOYT23L6NHJ6WQR55TQMTWZF4SNKSRLQGC5TYO3LA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.619215837 (20h17m54.095154302s) [noise=true , address=104.36.82.53:4126 , piece_id=GZ5Z7AGMYVGYVUJAWFAIS6SQLRHFB772RLQY736QUGOXA5OWQ34Q, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.619408406 (20h17m54.095346871s) [noise=true , address=104.36.82.55:4219 , piece_id=IURHWONYUYYJWIL5BDF6HLCEPA76O6QAZAJ22NCI6LQYIGO6K4IA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:43.299790904 (20h17m55.775729369s) [upload=19386]: Next returned nil segment Jul 29 13:03:43.299806024 (20h17m55.775744479s) [upload=19386]: Waiting for error group managing segments... Jul 29 13:03:43.847040928 (20h17m56.322979393s) [noise=true , address=104.36.82.9:4028 , piece_id=XCRNGUPBFYCZOYT23L6NHJ6WQR55TQMTWZF4SNKSRLQGC5TYO3LA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:43.879495404 (20h17m56.355433869s) [noise=true , address=104.36.82.61:4391 , piece_id=TI5YDGDSTWKDX5XAWL36F3TJDQNHKNTREZVTUGIS2EVZYP6DWSDA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:46.350727192 (20h17m58.826665657s) [noise=true , address=104.36.82.5:4866 , piece_id=EFRLC4LA232NZZJU3GDSP2EEBZO3W2LGJRD3L2A6SPSRBTLUTIFA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:51.702944756 (20h18m4.178883221s) [noise=true , address=104.36.82.42:4802 , piece_id=6HORHYYKIKDX4IWWKUQPVGZQZ2YYIGC47OCZHRVBC4BZFRJX5TQQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:55.250827413 (20h18m7.726765878s) [noise=true , address=104.36.82.55:4219 , piece_id=IURHWONYUYYJWIL5BDF6HLCEPA76O6QAZAJ22NCI6LQYIGO6K4IA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:57.858469692 (20h18m10.334408157s) [noise=true , address=104.36.82.53:4126 , piece_id=GZ5Z7AGMYVGYVUJAWFAIS6SQLRHFB772RLQY736QUGOXA5OWQ34Q, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:59.265385056 (20h18m11.741323511s) [noise=true , address=104.36.82.10:4088 , piece_id=6IV2Z7FXDNEOBJMNPRG7WSPZWZTJ5CGDF6JHBSF7WCIIL2QQWFUA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:59.973580263 (20h18m12.449518729s) [noise=true , address=104.36.82.11:4119 , piece_id=6KWLABSUMX6YVKD6LGKR6ONLQKFFTSTKFHHRVFIN7U4KDALBNCBQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:16.002200978 (20h18m28.478139453s) [noise=true , address=104.36.82.52:4092 , piece_id=BMJCAQYXJCUVJTEXH52UXYLD4MHWLDGQTJIPDOFRF4LPLTT36ESA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:30.906580908 (20h18m43.382519373s) [noise=true , address=104.36.82.20:4448 , piece_id=CSHYRDGTS4ROFDTKETIZ36TWJE3OEMAKLB4RGFAU557G725EANXQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:31.164952501 (20h18m43.640890966s) [noise=true , address=104.36.82.14:4227 , piece_id=O2FXJBOV3CJCKLZHRPBTLXX2QRT2OWJQ5HPNHZI5IVIFDVUUICVQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:33.000876204 (20h18m45.476814679s) [noise=true , address=104.36.82.56:4242 , piece_id=BHKC5MH4G65BK2YLJJANIYZ4MLAUTMBG32XSRISARHCO26AZUXRA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:34.725342452 (20h18m47.201280907s) [noise=true , address=104.36.82.47:4545 , piece_id=PZWVIOWLBZ72IOFRJHPAD4OZZPA2G4XYCDLGYEY7THSOR7XDHMUQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:40.451024021 (20h18m52.926962486s) [noise=true , address=104.36.82.46:4500 , piece_id=VCYKKDSLEKTTL3KO2FJ4VA5JGIVPA4GFBBHH65HPFZTP4REUTHYQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:12B7ox4ehqxJ2mwqeXj8C9N9GuSU5etNC23iT1pJnTgMLGBAGQK, address:104.36.82.46:4500): protocol: write tcp 188.166.225.170:46700->104.36.82.46:4500: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451060500 (20h18m52.926998965s) [noise=true , address=104.36.82.51:4075 , piece_id=P5NU32M3V7AMYFZVJ6XZY5K2YMQWF4EULQGX6RRYWG6SWDWWAFMQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK, address:104.36.82.51:4075): protocol: write tcp 188.166.225.170:41162->104.36.82.51:4075: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451086179 (20h18m52.927024644s) [noise=true , address=104.36.82.54:4175 , piece_id=5PAE3BM5AFSLZ5TVUQC3QK3RVWCVGFDLOK452BDPWQ7ARTL7EYWQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:12ou7jwFyJP3mNoALddniUTDT9duVyQKZjeq4xHCyx6sRWpwEPJ, address:104.36.82.54:4175): protocol: rpc: dial tcp 104.36.82.54:4175: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451120024 (20h18m52.927058489s) [noise=true , address=104.36.82.13:4214 , piece_id=SLX3YJCEWCM3553VZHILH337FKSPPFNKVFTPXO4KIBOS6IXCLXHA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:1nJ8uqwxf6x4hX52AjAdCbMKi1BKGGbMvrQbiazJdLpn6j69ay, address:104.36.82.13:4214): protocol: rpc: dial tcp 104.36.82.13:4214: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451147116 (20h18m52.927085581s) [noise=true , address=104.36.82.15:4263 , piece_id=3GCLYN4UBCLM5BPLXDQT5SSWPJBTKIOZM4UIUNO3OZIXW6OANUBQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:1f97yBEvcrYHqMq7iovVoju77jzhUYgoQCsUGULeTagugFXrCu, address:104.36.82.15:4263): protocol: write tcp 188.166.225.170:39380->104.36.82.15:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451167805 (20h18m52.927106271s) [seg_pos={0 0}, upload=19386]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 29 13:04:40.696628655 (20h18m53.17256714s) [upload=19386]: Done uploading segments... Jul 29 13:04:40.696658391 (20h18m53.172596856s) [upload=19386]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.844524063s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [............X...........X...................................] duration: 1.897046588s FAIL (2025-07-29 13:18:30 UTC) orbiter warning: unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K Duration: 1.897076574s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [.......X...............X........X...X..XX...................] duration: 5.920664268s FAIL (2025-07-29 13:18:19 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 14 but got 7) Duration: 5.920687612s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.......................................X.X..................] duration: 5.919757513s FAIL (2025-07-29 13:18:02 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ding segments... Jul 29 13:03:41.091320246 (20h17m53.567258701s) [upload=19386]: Got next segment. Inline: false Jul 29 13:03:41.091349191 (20h17m53.567287656s) [upload=19386]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 29 13:03:41.091354331 (20h17m53.567292796s) [upload=19386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 29 13:03:41.616444845 (20h17m54.09238331s) [seg_pos={0 0}, upload=19386]: Begin upload segment... Jul 29 13:03:41.616487748 (20h17m54.092426213s) [seg_pos={0 0}, upload=19386]: Done begin upload segment. Jul 29 13:03:41.616507376 (20h17m54.092445841s) [noise=true , address=104.36.82.15:4263 , piece_id=3GCLYN4UBCLM5BPLXDQT5SSWPJBTKIOZM4UIUNO3OZIXW6OANUBQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.616564315 (20h17m54.09250278s) [noise=true , address=104.36.82.11:4119 , piece_id=6KWLABSUMX6YVKD6LGKR6ONLQKFFTSTKFHHRVFIN7U4KDALBNCBQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.616866433 (20h17m54.092804888s) [noise=true , address=104.36.82.54:4175 , piece_id=5PAE3BM5AFSLZ5TVUQC3QK3RVWCVGFDLOK452BDPWQ7ARTL7EYWQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617072557 (20h17m54.093011022s) [noise=true , address=104.36.82.56:4242 , piece_id=BHKC5MH4G65BK2YLJJANIYZ4MLAUTMBG32XSRISARHCO26AZUXRA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617110629 (20h17m54.093049084s) [noise=true , address=104.36.82.5:4866 , piece_id=EFRLC4LA232NZZJU3GDSP2EEBZO3W2LGJRD3L2A6SPSRBTLUTIFA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617355037 (20h17m54.093293492s) [noise=true , address=104.36.82.61:4391 , piece_id=TI5YDGDSTWKDX5XAWL36F3TJDQNHKNTREZVTUGIS2EVZYP6DWSDA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617481549 (20h17m54.093420024s) [noise=true , address=104.36.82.20:4448 , piece_id=CSHYRDGTS4ROFDTKETIZ36TWJE3OEMAKLB4RGFAU557G725EANXQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617583044 (20h17m54.093521509s) [noise=true , address=104.36.82.47:4545 , piece_id=PZWVIOWLBZ72IOFRJHPAD4OZZPA2G4XYCDLGYEY7THSOR7XDHMUQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617828073 (20h17m54.093766528s) [noise=true , address=104.36.82.42:4802 , piece_id=6HORHYYKIKDX4IWWKUQPVGZQZ2YYIGC47OCZHRVBC4BZFRJX5TQQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617987597 (20h17m54.093926062s) [noise=true , address=104.36.82.13:4214 , piece_id=SLX3YJCEWCM3553VZHILH337FKSPPFNKVFTPXO4KIBOS6IXCLXHA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618055948 (20h17m54.093994403s) [noise=true , address=104.36.82.46:4500 , piece_id=VCYKKDSLEKTTL3KO2FJ4VA5JGIVPA4GFBBHH65HPFZTP4REUTHYQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618285828 (20h17m54.094224293s) [noise=true , address=104.36.82.52:4092 , piece_id=BMJCAQYXJCUVJTEXH52UXYLD4MHWLDGQTJIPDOFRF4LPLTT36ESA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618485871 (20h17m54.094424336s) [noise=true , address=104.36.82.10:4088 , piece_id=6IV2Z7FXDNEOBJMNPRG7WSPZWZTJ5CGDF6JHBSF7WCIIL2QQWFUA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618663881 (20h17m54.094602346s) [noise=true , address=104.36.82.51:4075 , piece_id=P5NU32M3V7AMYFZVJ6XZY5K2YMQWF4EULQGX6RRYWG6SWDWWAFMQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618840449 (20h17m54.094778914s) [noise=true , address=104.36.82.14:4227 , piece_id=O2FXJBOV3CJCKLZHRPBTLXX2QRT2OWJQ5HPNHZI5IVIFDVUUICVQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.619028879 (20h17m54.094967344s) [noise=true , address=104.36.82.9:4028 , piece_id=XCRNGUPBFYCZOYT23L6NHJ6WQR55TQMTWZF4SNKSRLQGC5TYO3LA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.619215837 (20h17m54.095154302s) [noise=true , address=104.36.82.53:4126 , piece_id=GZ5Z7AGMYVGYVUJAWFAIS6SQLRHFB772RLQY736QUGOXA5OWQ34Q, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.619408406 (20h17m54.095346871s) [noise=true , address=104.36.82.55:4219 , piece_id=IURHWONYUYYJWIL5BDF6HLCEPA76O6QAZAJ22NCI6LQYIGO6K4IA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:43.299790904 (20h17m55.775729369s) [upload=19386]: Next returned nil segment Jul 29 13:03:43.299806024 (20h17m55.775744479s) [upload=19386]: Waiting for error group managing segments... Jul 29 13:03:43.847040928 (20h17m56.322979393s) [noise=true , address=104.36.82.9:4028 , piece_id=XCRNGUPBFYCZOYT23L6NHJ6WQR55TQMTWZF4SNKSRLQGC5TYO3LA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:43.879495404 (20h17m56.355433869s) [noise=true , address=104.36.82.61:4391 , piece_id=TI5YDGDSTWKDX5XAWL36F3TJDQNHKNTREZVTUGIS2EVZYP6DWSDA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:46.350727192 (20h17m58.826665657s) [noise=true , address=104.36.82.5:4866 , piece_id=EFRLC4LA232NZZJU3GDSP2EEBZO3W2LGJRD3L2A6SPSRBTLUTIFA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:51.702944756 (20h18m4.178883221s) [noise=true , address=104.36.82.42:4802 , piece_id=6HORHYYKIKDX4IWWKUQPVGZQZ2YYIGC47OCZHRVBC4BZFRJX5TQQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:55.250827413 (20h18m7.726765878s) [noise=true , address=104.36.82.55:4219 , piece_id=IURHWONYUYYJWIL5BDF6HLCEPA76O6QAZAJ22NCI6LQYIGO6K4IA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:57.858469692 (20h18m10.334408157s) [noise=true , address=104.36.82.53:4126 , piece_id=GZ5Z7AGMYVGYVUJAWFAIS6SQLRHFB772RLQY736QUGOXA5OWQ34Q, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:59.265385056 (20h18m11.741323511s) [noise=true , address=104.36.82.10:4088 , piece_id=6IV2Z7FXDNEOBJMNPRG7WSPZWZTJ5CGDF6JHBSF7WCIIL2QQWFUA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:59.973580263 (20h18m12.449518729s) [noise=true , address=104.36.82.11:4119 , piece_id=6KWLABSUMX6YVKD6LGKR6ONLQKFFTSTKFHHRVFIN7U4KDALBNCBQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:16.002200978 (20h18m28.478139453s) [noise=true , address=104.36.82.52:4092 , piece_id=BMJCAQYXJCUVJTEXH52UXYLD4MHWLDGQTJIPDOFRF4LPLTT36ESA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:30.906580908 (20h18m43.382519373s) [noise=true , address=104.36.82.20:4448 , piece_id=CSHYRDGTS4ROFDTKETIZ36TWJE3OEMAKLB4RGFAU557G725EANXQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:31.164952501 (20h18m43.640890966s) [noise=true , address=104.36.82.14:4227 , piece_id=O2FXJBOV3CJCKLZHRPBTLXX2QRT2OWJQ5HPNHZI5IVIFDVUUICVQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:33.000876204 (20h18m45.476814679s) [noise=true , address=104.36.82.56:4242 , piece_id=BHKC5MH4G65BK2YLJJANIYZ4MLAUTMBG32XSRISARHCO26AZUXRA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:34.725342452 (20h18m47.201280907s) [noise=true , address=104.36.82.47:4545 , piece_id=PZWVIOWLBZ72IOFRJHPAD4OZZPA2G4XYCDLGYEY7THSOR7XDHMUQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:40.451024021 (20h18m52.926962486s) [noise=true , address=104.36.82.46:4500 , piece_id=VCYKKDSLEKTTL3KO2FJ4VA5JGIVPA4GFBBHH65HPFZTP4REUTHYQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:12B7ox4ehqxJ2mwqeXj8C9N9GuSU5etNC23iT1pJnTgMLGBAGQK, address:104.36.82.46:4500): protocol: write tcp 188.166.225.170:46700->104.36.82.46:4500: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451060500 (20h18m52.926998965s) [noise=true , address=104.36.82.51:4075 , piece_id=P5NU32M3V7AMYFZVJ6XZY5K2YMQWF4EULQGX6RRYWG6SWDWWAFMQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK, address:104.36.82.51:4075): protocol: write tcp 188.166.225.170:41162->104.36.82.51:4075: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451086179 (20h18m52.927024644s) [noise=true , address=104.36.82.54:4175 , piece_id=5PAE3BM5AFSLZ5TVUQC3QK3RVWCVGFDLOK452BDPWQ7ARTL7EYWQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:12ou7jwFyJP3mNoALddniUTDT9duVyQKZjeq4xHCyx6sRWpwEPJ, address:104.36.82.54:4175): protocol: rpc: dial tcp 104.36.82.54:4175: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451120024 (20h18m52.927058489s) [noise=true , address=104.36.82.13:4214 , piece_id=SLX3YJCEWCM3553VZHILH337FKSPPFNKVFTPXO4KIBOS6IXCLXHA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:1nJ8uqwxf6x4hX52AjAdCbMKi1BKGGbMvrQbiazJdLpn6j69ay, address:104.36.82.13:4214): protocol: rpc: dial tcp 104.36.82.13:4214: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451147116 (20h18m52.927085581s) [noise=true , address=104.36.82.15:4263 , piece_id=3GCLYN4UBCLM5BPLXDQT5SSWPJBTKIOZM4UIUNO3OZIXW6OANUBQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:1f97yBEvcrYHqMq7iovVoju77jzhUYgoQCsUGULeTagugFXrCu, address:104.36.82.15:4263): protocol: write tcp 188.166.225.170:39380->104.36.82.15:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451167805 (20h18m52.927106271s) [seg_pos={0 0}, upload=19386]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 29 13:04:40.696628655 (20h18m53.17256714s) [upload=19386]: Done uploading segments... Jul 29 13:04:40.696658391 (20h18m53.172596856s) [upload=19386]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.919788733s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [X............X...........X..................................] duration: 1.868035639s FAIL (2025-07-29 13:17:57 UTC) orbiter warning: unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K Duration: 1.868065436s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [........................................X.X.................] duration: 5.922287827s FAIL (2025-07-29 13:17:32 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ding segments... Jul 29 13:03:41.091320246 (20h17m53.567258701s) [upload=19386]: Got next segment. Inline: false Jul 29 13:03:41.091349191 (20h17m53.567287656s) [upload=19386]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 29 13:03:41.091354331 (20h17m53.567292796s) [upload=19386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 29 13:03:41.616444845 (20h17m54.09238331s) [seg_pos={0 0}, upload=19386]: Begin upload segment... Jul 29 13:03:41.616487748 (20h17m54.092426213s) [seg_pos={0 0}, upload=19386]: Done begin upload segment. Jul 29 13:03:41.616507376 (20h17m54.092445841s) [noise=true , address=104.36.82.15:4263 , piece_id=3GCLYN4UBCLM5BPLXDQT5SSWPJBTKIOZM4UIUNO3OZIXW6OANUBQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.616564315 (20h17m54.09250278s) [noise=true , address=104.36.82.11:4119 , piece_id=6KWLABSUMX6YVKD6LGKR6ONLQKFFTSTKFHHRVFIN7U4KDALBNCBQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.616866433 (20h17m54.092804888s) [noise=true , address=104.36.82.54:4175 , piece_id=5PAE3BM5AFSLZ5TVUQC3QK3RVWCVGFDLOK452BDPWQ7ARTL7EYWQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617072557 (20h17m54.093011022s) [noise=true , address=104.36.82.56:4242 , piece_id=BHKC5MH4G65BK2YLJJANIYZ4MLAUTMBG32XSRISARHCO26AZUXRA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617110629 (20h17m54.093049084s) [noise=true , address=104.36.82.5:4866 , piece_id=EFRLC4LA232NZZJU3GDSP2EEBZO3W2LGJRD3L2A6SPSRBTLUTIFA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617355037 (20h17m54.093293492s) [noise=true , address=104.36.82.61:4391 , piece_id=TI5YDGDSTWKDX5XAWL36F3TJDQNHKNTREZVTUGIS2EVZYP6DWSDA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617481549 (20h17m54.093420024s) [noise=true , address=104.36.82.20:4448 , piece_id=CSHYRDGTS4ROFDTKETIZ36TWJE3OEMAKLB4RGFAU557G725EANXQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617583044 (20h17m54.093521509s) [noise=true , address=104.36.82.47:4545 , piece_id=PZWVIOWLBZ72IOFRJHPAD4OZZPA2G4XYCDLGYEY7THSOR7XDHMUQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617828073 (20h17m54.093766528s) [noise=true , address=104.36.82.42:4802 , piece_id=6HORHYYKIKDX4IWWKUQPVGZQZ2YYIGC47OCZHRVBC4BZFRJX5TQQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617987597 (20h17m54.093926062s) [noise=true , address=104.36.82.13:4214 , piece_id=SLX3YJCEWCM3553VZHILH337FKSPPFNKVFTPXO4KIBOS6IXCLXHA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618055948 (20h17m54.093994403s) [noise=true , address=104.36.82.46:4500 , piece_id=VCYKKDSLEKTTL3KO2FJ4VA5JGIVPA4GFBBHH65HPFZTP4REUTHYQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618285828 (20h17m54.094224293s) [noise=true , address=104.36.82.52:4092 , piece_id=BMJCAQYXJCUVJTEXH52UXYLD4MHWLDGQTJIPDOFRF4LPLTT36ESA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618485871 (20h17m54.094424336s) [noise=true , address=104.36.82.10:4088 , piece_id=6IV2Z7FXDNEOBJMNPRG7WSPZWZTJ5CGDF6JHBSF7WCIIL2QQWFUA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618663881 (20h17m54.094602346s) [noise=true , address=104.36.82.51:4075 , piece_id=P5NU32M3V7AMYFZVJ6XZY5K2YMQWF4EULQGX6RRYWG6SWDWWAFMQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618840449 (20h17m54.094778914s) [noise=true , address=104.36.82.14:4227 , piece_id=O2FXJBOV3CJCKLZHRPBTLXX2QRT2OWJQ5HPNHZI5IVIFDVUUICVQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.619028879 (20h17m54.094967344s) [noise=true , address=104.36.82.9:4028 , piece_id=XCRNGUPBFYCZOYT23L6NHJ6WQR55TQMTWZF4SNKSRLQGC5TYO3LA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.619215837 (20h17m54.095154302s) [noise=true , address=104.36.82.53:4126 , piece_id=GZ5Z7AGMYVGYVUJAWFAIS6SQLRHFB772RLQY736QUGOXA5OWQ34Q, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.619408406 (20h17m54.095346871s) [noise=true , address=104.36.82.55:4219 , piece_id=IURHWONYUYYJWIL5BDF6HLCEPA76O6QAZAJ22NCI6LQYIGO6K4IA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:43.299790904 (20h17m55.775729369s) [upload=19386]: Next returned nil segment Jul 29 13:03:43.299806024 (20h17m55.775744479s) [upload=19386]: Waiting for error group managing segments... Jul 29 13:03:43.847040928 (20h17m56.322979393s) [noise=true , address=104.36.82.9:4028 , piece_id=XCRNGUPBFYCZOYT23L6NHJ6WQR55TQMTWZF4SNKSRLQGC5TYO3LA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:43.879495404 (20h17m56.355433869s) [noise=true , address=104.36.82.61:4391 , piece_id=TI5YDGDSTWKDX5XAWL36F3TJDQNHKNTREZVTUGIS2EVZYP6DWSDA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:46.350727192 (20h17m58.826665657s) [noise=true , address=104.36.82.5:4866 , piece_id=EFRLC4LA232NZZJU3GDSP2EEBZO3W2LGJRD3L2A6SPSRBTLUTIFA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:51.702944756 (20h18m4.178883221s) [noise=true , address=104.36.82.42:4802 , piece_id=6HORHYYKIKDX4IWWKUQPVGZQZ2YYIGC47OCZHRVBC4BZFRJX5TQQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:55.250827413 (20h18m7.726765878s) [noise=true , address=104.36.82.55:4219 , piece_id=IURHWONYUYYJWIL5BDF6HLCEPA76O6QAZAJ22NCI6LQYIGO6K4IA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:57.858469692 (20h18m10.334408157s) [noise=true , address=104.36.82.53:4126 , piece_id=GZ5Z7AGMYVGYVUJAWFAIS6SQLRHFB772RLQY736QUGOXA5OWQ34Q, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:59.265385056 (20h18m11.741323511s) [noise=true , address=104.36.82.10:4088 , piece_id=6IV2Z7FXDNEOBJMNPRG7WSPZWZTJ5CGDF6JHBSF7WCIIL2QQWFUA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:59.973580263 (20h18m12.449518729s) [noise=true , address=104.36.82.11:4119 , piece_id=6KWLABSUMX6YVKD6LGKR6ONLQKFFTSTKFHHRVFIN7U4KDALBNCBQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:16.002200978 (20h18m28.478139453s) [noise=true , address=104.36.82.52:4092 , piece_id=BMJCAQYXJCUVJTEXH52UXYLD4MHWLDGQTJIPDOFRF4LPLTT36ESA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:30.906580908 (20h18m43.382519373s) [noise=true , address=104.36.82.20:4448 , piece_id=CSHYRDGTS4ROFDTKETIZ36TWJE3OEMAKLB4RGFAU557G725EANXQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:31.164952501 (20h18m43.640890966s) [noise=true , address=104.36.82.14:4227 , piece_id=O2FXJBOV3CJCKLZHRPBTLXX2QRT2OWJQ5HPNHZI5IVIFDVUUICVQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:33.000876204 (20h18m45.476814679s) [noise=true , address=104.36.82.56:4242 , piece_id=BHKC5MH4G65BK2YLJJANIYZ4MLAUTMBG32XSRISARHCO26AZUXRA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:34.725342452 (20h18m47.201280907s) [noise=true , address=104.36.82.47:4545 , piece_id=PZWVIOWLBZ72IOFRJHPAD4OZZPA2G4XYCDLGYEY7THSOR7XDHMUQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:40.451024021 (20h18m52.926962486s) [noise=true , address=104.36.82.46:4500 , piece_id=VCYKKDSLEKTTL3KO2FJ4VA5JGIVPA4GFBBHH65HPFZTP4REUTHYQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:12B7ox4ehqxJ2mwqeXj8C9N9GuSU5etNC23iT1pJnTgMLGBAGQK, address:104.36.82.46:4500): protocol: write tcp 188.166.225.170:46700->104.36.82.46:4500: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451060500 (20h18m52.926998965s) [noise=true , address=104.36.82.51:4075 , piece_id=P5NU32M3V7AMYFZVJ6XZY5K2YMQWF4EULQGX6RRYWG6SWDWWAFMQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK, address:104.36.82.51:4075): protocol: write tcp 188.166.225.170:41162->104.36.82.51:4075: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451086179 (20h18m52.927024644s) [noise=true , address=104.36.82.54:4175 , piece_id=5PAE3BM5AFSLZ5TVUQC3QK3RVWCVGFDLOK452BDPWQ7ARTL7EYWQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:12ou7jwFyJP3mNoALddniUTDT9duVyQKZjeq4xHCyx6sRWpwEPJ, address:104.36.82.54:4175): protocol: rpc: dial tcp 104.36.82.54:4175: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451120024 (20h18m52.927058489s) [noise=true , address=104.36.82.13:4214 , piece_id=SLX3YJCEWCM3553VZHILH337FKSPPFNKVFTPXO4KIBOS6IXCLXHA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:1nJ8uqwxf6x4hX52AjAdCbMKi1BKGGbMvrQbiazJdLpn6j69ay, address:104.36.82.13:4214): protocol: rpc: dial tcp 104.36.82.13:4214: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451147116 (20h18m52.927085581s) [noise=true , address=104.36.82.15:4263 , piece_id=3GCLYN4UBCLM5BPLXDQT5SSWPJBTKIOZM4UIUNO3OZIXW6OANUBQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:1f97yBEvcrYHqMq7iovVoju77jzhUYgoQCsUGULeTagugFXrCu, address:104.36.82.15:4263): protocol: write tcp 188.166.225.170:39380->104.36.82.15:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451167805 (20h18m52.927106271s) [seg_pos={0 0}, upload=19386]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 29 13:04:40.696628655 (20h18m53.17256714s) [upload=19386]: Done uploading segments... Jul 29 13:04:40.696658391 (20h18m53.172596856s) [upload=19386]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.922312223s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [XX............X...........X.................................] duration: 1.933488594s FAIL (2025-07-29 13:17:17 UTC) orbiter warning: unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K Duration: 1.933552146s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [........X...............X........X...X..XX..................] duration: 5.865951114s FAIL (2025-07-29 13:17:09 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 14 but got 7) Duration: 5.866051947s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.........................................X.X................] duration: 5.907976124s FAIL (2025-07-29 13:16:56 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ding segments... Jul 29 13:03:41.091320246 (20h17m53.567258701s) [upload=19386]: Got next segment. Inline: false Jul 29 13:03:41.091349191 (20h17m53.567287656s) [upload=19386]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 29 13:03:41.091354331 (20h17m53.567292796s) [upload=19386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 29 13:03:41.616444845 (20h17m54.09238331s) [seg_pos={0 0}, upload=19386]: Begin upload segment... Jul 29 13:03:41.616487748 (20h17m54.092426213s) [seg_pos={0 0}, upload=19386]: Done begin upload segment. Jul 29 13:03:41.616507376 (20h17m54.092445841s) [noise=true , address=104.36.82.15:4263 , piece_id=3GCLYN4UBCLM5BPLXDQT5SSWPJBTKIOZM4UIUNO3OZIXW6OANUBQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.616564315 (20h17m54.09250278s) [noise=true , address=104.36.82.11:4119 , piece_id=6KWLABSUMX6YVKD6LGKR6ONLQKFFTSTKFHHRVFIN7U4KDALBNCBQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.616866433 (20h17m54.092804888s) [noise=true , address=104.36.82.54:4175 , piece_id=5PAE3BM5AFSLZ5TVUQC3QK3RVWCVGFDLOK452BDPWQ7ARTL7EYWQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617072557 (20h17m54.093011022s) [noise=true , address=104.36.82.56:4242 , piece_id=BHKC5MH4G65BK2YLJJANIYZ4MLAUTMBG32XSRISARHCO26AZUXRA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617110629 (20h17m54.093049084s) [noise=true , address=104.36.82.5:4866 , piece_id=EFRLC4LA232NZZJU3GDSP2EEBZO3W2LGJRD3L2A6SPSRBTLUTIFA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617355037 (20h17m54.093293492s) [noise=true , address=104.36.82.61:4391 , piece_id=TI5YDGDSTWKDX5XAWL36F3TJDQNHKNTREZVTUGIS2EVZYP6DWSDA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617481549 (20h17m54.093420024s) [noise=true , address=104.36.82.20:4448 , piece_id=CSHYRDGTS4ROFDTKETIZ36TWJE3OEMAKLB4RGFAU557G725EANXQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617583044 (20h17m54.093521509s) [noise=true , address=104.36.82.47:4545 , piece_id=PZWVIOWLBZ72IOFRJHPAD4OZZPA2G4XYCDLGYEY7THSOR7XDHMUQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617828073 (20h17m54.093766528s) [noise=true , address=104.36.82.42:4802 , piece_id=6HORHYYKIKDX4IWWKUQPVGZQZ2YYIGC47OCZHRVBC4BZFRJX5TQQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617987597 (20h17m54.093926062s) [noise=true , address=104.36.82.13:4214 , piece_id=SLX3YJCEWCM3553VZHILH337FKSPPFNKVFTPXO4KIBOS6IXCLXHA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618055948 (20h17m54.093994403s) [noise=true , address=104.36.82.46:4500 , piece_id=VCYKKDSLEKTTL3KO2FJ4VA5JGIVPA4GFBBHH65HPFZTP4REUTHYQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618285828 (20h17m54.094224293s) [noise=true , address=104.36.82.52:4092 , piece_id=BMJCAQYXJCUVJTEXH52UXYLD4MHWLDGQTJIPDOFRF4LPLTT36ESA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618485871 (20h17m54.094424336s) [noise=true , address=104.36.82.10:4088 , piece_id=6IV2Z7FXDNEOBJMNPRG7WSPZWZTJ5CGDF6JHBSF7WCIIL2QQWFUA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618663881 (20h17m54.094602346s) [noise=true , address=104.36.82.51:4075 , piece_id=P5NU32M3V7AMYFZVJ6XZY5K2YMQWF4EULQGX6RRYWG6SWDWWAFMQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618840449 (20h17m54.094778914s) [noise=true , address=104.36.82.14:4227 , piece_id=O2FXJBOV3CJCKLZHRPBTLXX2QRT2OWJQ5HPNHZI5IVIFDVUUICVQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.619028879 (20h17m54.094967344s) [noise=true , address=104.36.82.9:4028 , piece_id=XCRNGUPBFYCZOYT23L6NHJ6WQR55TQMTWZF4SNKSRLQGC5TYO3LA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.619215837 (20h17m54.095154302s) [noise=true , address=104.36.82.53:4126 , piece_id=GZ5Z7AGMYVGYVUJAWFAIS6SQLRHFB772RLQY736QUGOXA5OWQ34Q, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.619408406 (20h17m54.095346871s) [noise=true , address=104.36.82.55:4219 , piece_id=IURHWONYUYYJWIL5BDF6HLCEPA76O6QAZAJ22NCI6LQYIGO6K4IA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:43.299790904 (20h17m55.775729369s) [upload=19386]: Next returned nil segment Jul 29 13:03:43.299806024 (20h17m55.775744479s) [upload=19386]: Waiting for error group managing segments... Jul 29 13:03:43.847040928 (20h17m56.322979393s) [noise=true , address=104.36.82.9:4028 , piece_id=XCRNGUPBFYCZOYT23L6NHJ6WQR55TQMTWZF4SNKSRLQGC5TYO3LA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:43.879495404 (20h17m56.355433869s) [noise=true , address=104.36.82.61:4391 , piece_id=TI5YDGDSTWKDX5XAWL36F3TJDQNHKNTREZVTUGIS2EVZYP6DWSDA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:46.350727192 (20h17m58.826665657s) [noise=true , address=104.36.82.5:4866 , piece_id=EFRLC4LA232NZZJU3GDSP2EEBZO3W2LGJRD3L2A6SPSRBTLUTIFA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:51.702944756 (20h18m4.178883221s) [noise=true , address=104.36.82.42:4802 , piece_id=6HORHYYKIKDX4IWWKUQPVGZQZ2YYIGC47OCZHRVBC4BZFRJX5TQQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:55.250827413 (20h18m7.726765878s) [noise=true , address=104.36.82.55:4219 , piece_id=IURHWONYUYYJWIL5BDF6HLCEPA76O6QAZAJ22NCI6LQYIGO6K4IA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:57.858469692 (20h18m10.334408157s) [noise=true , address=104.36.82.53:4126 , piece_id=GZ5Z7AGMYVGYVUJAWFAIS6SQLRHFB772RLQY736QUGOXA5OWQ34Q, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:59.265385056 (20h18m11.741323511s) [noise=true , address=104.36.82.10:4088 , piece_id=6IV2Z7FXDNEOBJMNPRG7WSPZWZTJ5CGDF6JHBSF7WCIIL2QQWFUA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:59.973580263 (20h18m12.449518729s) [noise=true , address=104.36.82.11:4119 , piece_id=6KWLABSUMX6YVKD6LGKR6ONLQKFFTSTKFHHRVFIN7U4KDALBNCBQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:16.002200978 (20h18m28.478139453s) [noise=true , address=104.36.82.52:4092 , piece_id=BMJCAQYXJCUVJTEXH52UXYLD4MHWLDGQTJIPDOFRF4LPLTT36ESA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:30.906580908 (20h18m43.382519373s) [noise=true , address=104.36.82.20:4448 , piece_id=CSHYRDGTS4ROFDTKETIZ36TWJE3OEMAKLB4RGFAU557G725EANXQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:31.164952501 (20h18m43.640890966s) [noise=true , address=104.36.82.14:4227 , piece_id=O2FXJBOV3CJCKLZHRPBTLXX2QRT2OWJQ5HPNHZI5IVIFDVUUICVQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:33.000876204 (20h18m45.476814679s) [noise=true , address=104.36.82.56:4242 , piece_id=BHKC5MH4G65BK2YLJJANIYZ4MLAUTMBG32XSRISARHCO26AZUXRA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:34.725342452 (20h18m47.201280907s) [noise=true , address=104.36.82.47:4545 , piece_id=PZWVIOWLBZ72IOFRJHPAD4OZZPA2G4XYCDLGYEY7THSOR7XDHMUQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:40.451024021 (20h18m52.926962486s) [noise=true , address=104.36.82.46:4500 , piece_id=VCYKKDSLEKTTL3KO2FJ4VA5JGIVPA4GFBBHH65HPFZTP4REUTHYQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:12B7ox4ehqxJ2mwqeXj8C9N9GuSU5etNC23iT1pJnTgMLGBAGQK, address:104.36.82.46:4500): protocol: write tcp 188.166.225.170:46700->104.36.82.46:4500: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451060500 (20h18m52.926998965s) [noise=true , address=104.36.82.51:4075 , piece_id=P5NU32M3V7AMYFZVJ6XZY5K2YMQWF4EULQGX6RRYWG6SWDWWAFMQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK, address:104.36.82.51:4075): protocol: write tcp 188.166.225.170:41162->104.36.82.51:4075: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451086179 (20h18m52.927024644s) [noise=true , address=104.36.82.54:4175 , piece_id=5PAE3BM5AFSLZ5TVUQC3QK3RVWCVGFDLOK452BDPWQ7ARTL7EYWQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:12ou7jwFyJP3mNoALddniUTDT9duVyQKZjeq4xHCyx6sRWpwEPJ, address:104.36.82.54:4175): protocol: rpc: dial tcp 104.36.82.54:4175: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451120024 (20h18m52.927058489s) [noise=true , address=104.36.82.13:4214 , piece_id=SLX3YJCEWCM3553VZHILH337FKSPPFNKVFTPXO4KIBOS6IXCLXHA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:1nJ8uqwxf6x4hX52AjAdCbMKi1BKGGbMvrQbiazJdLpn6j69ay, address:104.36.82.13:4214): protocol: rpc: dial tcp 104.36.82.13:4214: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451147116 (20h18m52.927085581s) [noise=true , address=104.36.82.15:4263 , piece_id=3GCLYN4UBCLM5BPLXDQT5SSWPJBTKIOZM4UIUNO3OZIXW6OANUBQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:1f97yBEvcrYHqMq7iovVoju77jzhUYgoQCsUGULeTagugFXrCu, address:104.36.82.15:4263): protocol: write tcp 188.166.225.170:39380->104.36.82.15:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451167805 (20h18m52.927106271s) [seg_pos={0 0}, upload=19386]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 29 13:04:40.696628655 (20h18m53.17256714s) [upload=19386]: Done uploading segments... Jul 29 13:04:40.696658391 (20h18m53.172596856s) [upload=19386]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.908003456s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [.........X...............X........X...X..XX.................] duration: 5.888660217s FAIL (2025-07-29 13:16:04 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 14 but got 7) Duration: 5.888685645s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..........................................X.X...............] duration: 5.755349218s FAIL (2025-07-29 13:16:03 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ding segments... Jul 29 13:03:41.091320246 (20h17m53.567258701s) [upload=19386]: Got next segment. Inline: false Jul 29 13:03:41.091349191 (20h17m53.567287656s) [upload=19386]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 29 13:03:41.091354331 (20h17m53.567292796s) [upload=19386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 29 13:03:41.616444845 (20h17m54.09238331s) [seg_pos={0 0}, upload=19386]: Begin upload segment... Jul 29 13:03:41.616487748 (20h17m54.092426213s) [seg_pos={0 0}, upload=19386]: Done begin upload segment. Jul 29 13:03:41.616507376 (20h17m54.092445841s) [noise=true , address=104.36.82.15:4263 , piece_id=3GCLYN4UBCLM5BPLXDQT5SSWPJBTKIOZM4UIUNO3OZIXW6OANUBQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.616564315 (20h17m54.09250278s) [noise=true , address=104.36.82.11:4119 , piece_id=6KWLABSUMX6YVKD6LGKR6ONLQKFFTSTKFHHRVFIN7U4KDALBNCBQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.616866433 (20h17m54.092804888s) [noise=true , address=104.36.82.54:4175 , piece_id=5PAE3BM5AFSLZ5TVUQC3QK3RVWCVGFDLOK452BDPWQ7ARTL7EYWQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617072557 (20h17m54.093011022s) [noise=true , address=104.36.82.56:4242 , piece_id=BHKC5MH4G65BK2YLJJANIYZ4MLAUTMBG32XSRISARHCO26AZUXRA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617110629 (20h17m54.093049084s) [noise=true , address=104.36.82.5:4866 , piece_id=EFRLC4LA232NZZJU3GDSP2EEBZO3W2LGJRD3L2A6SPSRBTLUTIFA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617355037 (20h17m54.093293492s) [noise=true , address=104.36.82.61:4391 , piece_id=TI5YDGDSTWKDX5XAWL36F3TJDQNHKNTREZVTUGIS2EVZYP6DWSDA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617481549 (20h17m54.093420024s) [noise=true , address=104.36.82.20:4448 , piece_id=CSHYRDGTS4ROFDTKETIZ36TWJE3OEMAKLB4RGFAU557G725EANXQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617583044 (20h17m54.093521509s) [noise=true , address=104.36.82.47:4545 , piece_id=PZWVIOWLBZ72IOFRJHPAD4OZZPA2G4XYCDLGYEY7THSOR7XDHMUQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617828073 (20h17m54.093766528s) [noise=true , address=104.36.82.42:4802 , piece_id=6HORHYYKIKDX4IWWKUQPVGZQZ2YYIGC47OCZHRVBC4BZFRJX5TQQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617987597 (20h17m54.093926062s) [noise=true , address=104.36.82.13:4214 , piece_id=SLX3YJCEWCM3553VZHILH337FKSPPFNKVFTPXO4KIBOS6IXCLXHA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618055948 (20h17m54.093994403s) [noise=true , address=104.36.82.46:4500 , piece_id=VCYKKDSLEKTTL3KO2FJ4VA5JGIVPA4GFBBHH65HPFZTP4REUTHYQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618285828 (20h17m54.094224293s) [noise=true , address=104.36.82.52:4092 , piece_id=BMJCAQYXJCUVJTEXH52UXYLD4MHWLDGQTJIPDOFRF4LPLTT36ESA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618485871 (20h17m54.094424336s) [noise=true , address=104.36.82.10:4088 , piece_id=6IV2Z7FXDNEOBJMNPRG7WSPZWZTJ5CGDF6JHBSF7WCIIL2QQWFUA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618663881 (20h17m54.094602346s) [noise=true , address=104.36.82.51:4075 , piece_id=P5NU32M3V7AMYFZVJ6XZY5K2YMQWF4EULQGX6RRYWG6SWDWWAFMQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618840449 (20h17m54.094778914s) [noise=true , address=104.36.82.14:4227 , piece_id=O2FXJBOV3CJCKLZHRPBTLXX2QRT2OWJQ5HPNHZI5IVIFDVUUICVQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.619028879 (20h17m54.094967344s) [noise=true , address=104.36.82.9:4028 , piece_id=XCRNGUPBFYCZOYT23L6NHJ6WQR55TQMTWZF4SNKSRLQGC5TYO3LA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.619215837 (20h17m54.095154302s) [noise=true , address=104.36.82.53:4126 , piece_id=GZ5Z7AGMYVGYVUJAWFAIS6SQLRHFB772RLQY736QUGOXA5OWQ34Q, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.619408406 (20h17m54.095346871s) [noise=true , address=104.36.82.55:4219 , piece_id=IURHWONYUYYJWIL5BDF6HLCEPA76O6QAZAJ22NCI6LQYIGO6K4IA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:43.299790904 (20h17m55.775729369s) [upload=19386]: Next returned nil segment Jul 29 13:03:43.299806024 (20h17m55.775744479s) [upload=19386]: Waiting for error group managing segments... Jul 29 13:03:43.847040928 (20h17m56.322979393s) [noise=true , address=104.36.82.9:4028 , piece_id=XCRNGUPBFYCZOYT23L6NHJ6WQR55TQMTWZF4SNKSRLQGC5TYO3LA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:43.879495404 (20h17m56.355433869s) [noise=true , address=104.36.82.61:4391 , piece_id=TI5YDGDSTWKDX5XAWL36F3TJDQNHKNTREZVTUGIS2EVZYP6DWSDA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:46.350727192 (20h17m58.826665657s) [noise=true , address=104.36.82.5:4866 , piece_id=EFRLC4LA232NZZJU3GDSP2EEBZO3W2LGJRD3L2A6SPSRBTLUTIFA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:51.702944756 (20h18m4.178883221s) [noise=true , address=104.36.82.42:4802 , piece_id=6HORHYYKIKDX4IWWKUQPVGZQZ2YYIGC47OCZHRVBC4BZFRJX5TQQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:55.250827413 (20h18m7.726765878s) [noise=true , address=104.36.82.55:4219 , piece_id=IURHWONYUYYJWIL5BDF6HLCEPA76O6QAZAJ22NCI6LQYIGO6K4IA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:57.858469692 (20h18m10.334408157s) [noise=true , address=104.36.82.53:4126 , piece_id=GZ5Z7AGMYVGYVUJAWFAIS6SQLRHFB772RLQY736QUGOXA5OWQ34Q, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:59.265385056 (20h18m11.741323511s) [noise=true , address=104.36.82.10:4088 , piece_id=6IV2Z7FXDNEOBJMNPRG7WSPZWZTJ5CGDF6JHBSF7WCIIL2QQWFUA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:59.973580263 (20h18m12.449518729s) [noise=true , address=104.36.82.11:4119 , piece_id=6KWLABSUMX6YVKD6LGKR6ONLQKFFTSTKFHHRVFIN7U4KDALBNCBQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:16.002200978 (20h18m28.478139453s) [noise=true , address=104.36.82.52:4092 , piece_id=BMJCAQYXJCUVJTEXH52UXYLD4MHWLDGQTJIPDOFRF4LPLTT36ESA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:30.906580908 (20h18m43.382519373s) [noise=true , address=104.36.82.20:4448 , piece_id=CSHYRDGTS4ROFDTKETIZ36TWJE3OEMAKLB4RGFAU557G725EANXQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:31.164952501 (20h18m43.640890966s) [noise=true , address=104.36.82.14:4227 , piece_id=O2FXJBOV3CJCKLZHRPBTLXX2QRT2OWJQ5HPNHZI5IVIFDVUUICVQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:33.000876204 (20h18m45.476814679s) [noise=true , address=104.36.82.56:4242 , piece_id=BHKC5MH4G65BK2YLJJANIYZ4MLAUTMBG32XSRISARHCO26AZUXRA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:34.725342452 (20h18m47.201280907s) [noise=true , address=104.36.82.47:4545 , piece_id=PZWVIOWLBZ72IOFRJHPAD4OZZPA2G4XYCDLGYEY7THSOR7XDHMUQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:40.451024021 (20h18m52.926962486s) [noise=true , address=104.36.82.46:4500 , piece_id=VCYKKDSLEKTTL3KO2FJ4VA5JGIVPA4GFBBHH65HPFZTP4REUTHYQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:12B7ox4ehqxJ2mwqeXj8C9N9GuSU5etNC23iT1pJnTgMLGBAGQK, address:104.36.82.46:4500): protocol: write tcp 188.166.225.170:46700->104.36.82.46:4500: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451060500 (20h18m52.926998965s) [noise=true , address=104.36.82.51:4075 , piece_id=P5NU32M3V7AMYFZVJ6XZY5K2YMQWF4EULQGX6RRYWG6SWDWWAFMQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK, address:104.36.82.51:4075): protocol: write tcp 188.166.225.170:41162->104.36.82.51:4075: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451086179 (20h18m52.927024644s) [noise=true , address=104.36.82.54:4175 , piece_id=5PAE3BM5AFSLZ5TVUQC3QK3RVWCVGFDLOK452BDPWQ7ARTL7EYWQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:12ou7jwFyJP3mNoALddniUTDT9duVyQKZjeq4xHCyx6sRWpwEPJ, address:104.36.82.54:4175): protocol: rpc: dial tcp 104.36.82.54:4175: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451120024 (20h18m52.927058489s) [noise=true , address=104.36.82.13:4214 , piece_id=SLX3YJCEWCM3553VZHILH337FKSPPFNKVFTPXO4KIBOS6IXCLXHA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:1nJ8uqwxf6x4hX52AjAdCbMKi1BKGGbMvrQbiazJdLpn6j69ay, address:104.36.82.13:4214): protocol: rpc: dial tcp 104.36.82.13:4214: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451147116 (20h18m52.927085581s) [noise=true , address=104.36.82.15:4263 , piece_id=3GCLYN4UBCLM5BPLXDQT5SSWPJBTKIOZM4UIUNO3OZIXW6OANUBQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:1f97yBEvcrYHqMq7iovVoju77jzhUYgoQCsUGULeTagugFXrCu, address:104.36.82.15:4263): protocol: write tcp 188.166.225.170:39380->104.36.82.15:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451167805 (20h18m52.927106271s) [seg_pos={0 0}, upload=19386]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 29 13:04:40.696628655 (20h18m53.17256714s) [upload=19386]: Done uploading segments... Jul 29 13:04:40.696658391 (20h18m53.172596856s) [upload=19386]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.75538188s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [.XX............X...........X................................] duration: 1.856332078s FAIL (2025-07-29 13:16:02 UTC) orbiter warning: unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K Duration: 1.856355724s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [..XX............X...........X...............................] duration: 1.984942362s FAIL (2025-07-29 13:15:32 UTC) orbiter warning: unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K Duration: 1.984977921s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [..........X...............X........X...X..XX................] duration: 6.038739717s FAIL (2025-07-29 13:15:13 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 14 but got 7) Duration: 6.03876658s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...........................................X.X..............] duration: 5.896267609s FAIL (2025-07-29 13:14:56 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ding segments... Jul 29 13:03:41.091320246 (20h17m53.567258701s) [upload=19386]: Got next segment. Inline: false Jul 29 13:03:41.091349191 (20h17m53.567287656s) [upload=19386]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 29 13:03:41.091354331 (20h17m53.567292796s) [upload=19386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 29 13:03:41.616444845 (20h17m54.09238331s) [seg_pos={0 0}, upload=19386]: Begin upload segment... Jul 29 13:03:41.616487748 (20h17m54.092426213s) [seg_pos={0 0}, upload=19386]: Done begin upload segment. Jul 29 13:03:41.616507376 (20h17m54.092445841s) [noise=true , address=104.36.82.15:4263 , piece_id=3GCLYN4UBCLM5BPLXDQT5SSWPJBTKIOZM4UIUNO3OZIXW6OANUBQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.616564315 (20h17m54.09250278s) [noise=true , address=104.36.82.11:4119 , piece_id=6KWLABSUMX6YVKD6LGKR6ONLQKFFTSTKFHHRVFIN7U4KDALBNCBQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.616866433 (20h17m54.092804888s) [noise=true , address=104.36.82.54:4175 , piece_id=5PAE3BM5AFSLZ5TVUQC3QK3RVWCVGFDLOK452BDPWQ7ARTL7EYWQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617072557 (20h17m54.093011022s) [noise=true , address=104.36.82.56:4242 , piece_id=BHKC5MH4G65BK2YLJJANIYZ4MLAUTMBG32XSRISARHCO26AZUXRA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617110629 (20h17m54.093049084s) [noise=true , address=104.36.82.5:4866 , piece_id=EFRLC4LA232NZZJU3GDSP2EEBZO3W2LGJRD3L2A6SPSRBTLUTIFA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617355037 (20h17m54.093293492s) [noise=true , address=104.36.82.61:4391 , piece_id=TI5YDGDSTWKDX5XAWL36F3TJDQNHKNTREZVTUGIS2EVZYP6DWSDA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617481549 (20h17m54.093420024s) [noise=true , address=104.36.82.20:4448 , piece_id=CSHYRDGTS4ROFDTKETIZ36TWJE3OEMAKLB4RGFAU557G725EANXQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617583044 (20h17m54.093521509s) [noise=true , address=104.36.82.47:4545 , piece_id=PZWVIOWLBZ72IOFRJHPAD4OZZPA2G4XYCDLGYEY7THSOR7XDHMUQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617828073 (20h17m54.093766528s) [noise=true , address=104.36.82.42:4802 , piece_id=6HORHYYKIKDX4IWWKUQPVGZQZ2YYIGC47OCZHRVBC4BZFRJX5TQQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617987597 (20h17m54.093926062s) [noise=true , address=104.36.82.13:4214 , piece_id=SLX3YJCEWCM3553VZHILH337FKSPPFNKVFTPXO4KIBOS6IXCLXHA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618055948 (20h17m54.093994403s) [noise=true , address=104.36.82.46:4500 , piece_id=VCYKKDSLEKTTL3KO2FJ4VA5JGIVPA4GFBBHH65HPFZTP4REUTHYQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618285828 (20h17m54.094224293s) [noise=true , address=104.36.82.52:4092 , piece_id=BMJCAQYXJCUVJTEXH52UXYLD4MHWLDGQTJIPDOFRF4LPLTT36ESA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618485871 (20h17m54.094424336s) [noise=true , address=104.36.82.10:4088 , piece_id=6IV2Z7FXDNEOBJMNPRG7WSPZWZTJ5CGDF6JHBSF7WCIIL2QQWFUA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618663881 (20h17m54.094602346s) [noise=true , address=104.36.82.51:4075 , piece_id=P5NU32M3V7AMYFZVJ6XZY5K2YMQWF4EULQGX6RRYWG6SWDWWAFMQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618840449 (20h17m54.094778914s) [noise=true , address=104.36.82.14:4227 , piece_id=O2FXJBOV3CJCKLZHRPBTLXX2QRT2OWJQ5HPNHZI5IVIFDVUUICVQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.619028879 (20h17m54.094967344s) [noise=true , address=104.36.82.9:4028 , piece_id=XCRNGUPBFYCZOYT23L6NHJ6WQR55TQMTWZF4SNKSRLQGC5TYO3LA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.619215837 (20h17m54.095154302s) [noise=true , address=104.36.82.53:4126 , piece_id=GZ5Z7AGMYVGYVUJAWFAIS6SQLRHFB772RLQY736QUGOXA5OWQ34Q, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.619408406 (20h17m54.095346871s) [noise=true , address=104.36.82.55:4219 , piece_id=IURHWONYUYYJWIL5BDF6HLCEPA76O6QAZAJ22NCI6LQYIGO6K4IA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:43.299790904 (20h17m55.775729369s) [upload=19386]: Next returned nil segment Jul 29 13:03:43.299806024 (20h17m55.775744479s) [upload=19386]: Waiting for error group managing segments... Jul 29 13:03:43.847040928 (20h17m56.322979393s) [noise=true , address=104.36.82.9:4028 , piece_id=XCRNGUPBFYCZOYT23L6NHJ6WQR55TQMTWZF4SNKSRLQGC5TYO3LA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:43.879495404 (20h17m56.355433869s) [noise=true , address=104.36.82.61:4391 , piece_id=TI5YDGDSTWKDX5XAWL36F3TJDQNHKNTREZVTUGIS2EVZYP6DWSDA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:46.350727192 (20h17m58.826665657s) [noise=true , address=104.36.82.5:4866 , piece_id=EFRLC4LA232NZZJU3GDSP2EEBZO3W2LGJRD3L2A6SPSRBTLUTIFA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:51.702944756 (20h18m4.178883221s) [noise=true , address=104.36.82.42:4802 , piece_id=6HORHYYKIKDX4IWWKUQPVGZQZ2YYIGC47OCZHRVBC4BZFRJX5TQQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:55.250827413 (20h18m7.726765878s) [noise=true , address=104.36.82.55:4219 , piece_id=IURHWONYUYYJWIL5BDF6HLCEPA76O6QAZAJ22NCI6LQYIGO6K4IA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:57.858469692 (20h18m10.334408157s) [noise=true , address=104.36.82.53:4126 , piece_id=GZ5Z7AGMYVGYVUJAWFAIS6SQLRHFB772RLQY736QUGOXA5OWQ34Q, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:59.265385056 (20h18m11.741323511s) [noise=true , address=104.36.82.10:4088 , piece_id=6IV2Z7FXDNEOBJMNPRG7WSPZWZTJ5CGDF6JHBSF7WCIIL2QQWFUA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:59.973580263 (20h18m12.449518729s) [noise=true , address=104.36.82.11:4119 , piece_id=6KWLABSUMX6YVKD6LGKR6ONLQKFFTSTKFHHRVFIN7U4KDALBNCBQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:16.002200978 (20h18m28.478139453s) [noise=true , address=104.36.82.52:4092 , piece_id=BMJCAQYXJCUVJTEXH52UXYLD4MHWLDGQTJIPDOFRF4LPLTT36ESA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:30.906580908 (20h18m43.382519373s) [noise=true , address=104.36.82.20:4448 , piece_id=CSHYRDGTS4ROFDTKETIZ36TWJE3OEMAKLB4RGFAU557G725EANXQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:31.164952501 (20h18m43.640890966s) [noise=true , address=104.36.82.14:4227 , piece_id=O2FXJBOV3CJCKLZHRPBTLXX2QRT2OWJQ5HPNHZI5IVIFDVUUICVQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:33.000876204 (20h18m45.476814679s) [noise=true , address=104.36.82.56:4242 , piece_id=BHKC5MH4G65BK2YLJJANIYZ4MLAUTMBG32XSRISARHCO26AZUXRA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:34.725342452 (20h18m47.201280907s) [noise=true , address=104.36.82.47:4545 , piece_id=PZWVIOWLBZ72IOFRJHPAD4OZZPA2G4XYCDLGYEY7THSOR7XDHMUQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:40.451024021 (20h18m52.926962486s) [noise=true , address=104.36.82.46:4500 , piece_id=VCYKKDSLEKTTL3KO2FJ4VA5JGIVPA4GFBBHH65HPFZTP4REUTHYQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:12B7ox4ehqxJ2mwqeXj8C9N9GuSU5etNC23iT1pJnTgMLGBAGQK, address:104.36.82.46:4500): protocol: write tcp 188.166.225.170:46700->104.36.82.46:4500: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451060500 (20h18m52.926998965s) [noise=true , address=104.36.82.51:4075 , piece_id=P5NU32M3V7AMYFZVJ6XZY5K2YMQWF4EULQGX6RRYWG6SWDWWAFMQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK, address:104.36.82.51:4075): protocol: write tcp 188.166.225.170:41162->104.36.82.51:4075: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451086179 (20h18m52.927024644s) [noise=true , address=104.36.82.54:4175 , piece_id=5PAE3BM5AFSLZ5TVUQC3QK3RVWCVGFDLOK452BDPWQ7ARTL7EYWQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:12ou7jwFyJP3mNoALddniUTDT9duVyQKZjeq4xHCyx6sRWpwEPJ, address:104.36.82.54:4175): protocol: rpc: dial tcp 104.36.82.54:4175: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451120024 (20h18m52.927058489s) [noise=true , address=104.36.82.13:4214 , piece_id=SLX3YJCEWCM3553VZHILH337FKSPPFNKVFTPXO4KIBOS6IXCLXHA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:1nJ8uqwxf6x4hX52AjAdCbMKi1BKGGbMvrQbiazJdLpn6j69ay, address:104.36.82.13:4214): protocol: rpc: dial tcp 104.36.82.13:4214: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451147116 (20h18m52.927085581s) [noise=true , address=104.36.82.15:4263 , piece_id=3GCLYN4UBCLM5BPLXDQT5SSWPJBTKIOZM4UIUNO3OZIXW6OANUBQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:1f97yBEvcrYHqMq7iovVoju77jzhUYgoQCsUGULeTagugFXrCu, address:104.36.82.15:4263): protocol: write tcp 188.166.225.170:39380->104.36.82.15:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451167805 (20h18m52.927106271s) [seg_pos={0 0}, upload=19386]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 29 13:04:40.696628655 (20h18m53.17256714s) [upload=19386]: Done uploading segments... Jul 29 13:04:40.696658391 (20h18m53.172596856s) [upload=19386]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.896298598s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [...XX............X...........X..............................] duration: 1.853646908s FAIL (2025-07-29 13:14:30 UTC) orbiter warning: unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K Duration: 1.853676595s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [...........X...............X........X...X..XX...............] duration: 5.879845467s FAIL (2025-07-29 13:14:25 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 14 but got 7) Duration: 5.879870735s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [............X...............X........X...X..XX..............] duration: 6.007549908s FAIL (2025-07-29 13:14:01 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 14 but got 7) Duration: 6.007580316s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [....XX............X...........X.............................] duration: 1.928493948s FAIL (2025-07-29 13:13:53 UTC) orbiter warning: unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K Duration: 1.928528624s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [............................................X.X.............] duration: 5.919130376s FAIL (2025-07-29 13:13:44 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ding segments... Jul 29 13:03:41.091320246 (20h17m53.567258701s) [upload=19386]: Got next segment. Inline: false Jul 29 13:03:41.091349191 (20h17m53.567287656s) [upload=19386]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 29 13:03:41.091354331 (20h17m53.567292796s) [upload=19386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 29 13:03:41.616444845 (20h17m54.09238331s) [seg_pos={0 0}, upload=19386]: Begin upload segment... Jul 29 13:03:41.616487748 (20h17m54.092426213s) [seg_pos={0 0}, upload=19386]: Done begin upload segment. Jul 29 13:03:41.616507376 (20h17m54.092445841s) [noise=true , address=104.36.82.15:4263 , piece_id=3GCLYN4UBCLM5BPLXDQT5SSWPJBTKIOZM4UIUNO3OZIXW6OANUBQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.616564315 (20h17m54.09250278s) [noise=true , address=104.36.82.11:4119 , piece_id=6KWLABSUMX6YVKD6LGKR6ONLQKFFTSTKFHHRVFIN7U4KDALBNCBQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.616866433 (20h17m54.092804888s) [noise=true , address=104.36.82.54:4175 , piece_id=5PAE3BM5AFSLZ5TVUQC3QK3RVWCVGFDLOK452BDPWQ7ARTL7EYWQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617072557 (20h17m54.093011022s) [noise=true , address=104.36.82.56:4242 , piece_id=BHKC5MH4G65BK2YLJJANIYZ4MLAUTMBG32XSRISARHCO26AZUXRA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617110629 (20h17m54.093049084s) [noise=true , address=104.36.82.5:4866 , piece_id=EFRLC4LA232NZZJU3GDSP2EEBZO3W2LGJRD3L2A6SPSRBTLUTIFA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617355037 (20h17m54.093293492s) [noise=true , address=104.36.82.61:4391 , piece_id=TI5YDGDSTWKDX5XAWL36F3TJDQNHKNTREZVTUGIS2EVZYP6DWSDA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617481549 (20h17m54.093420024s) [noise=true , address=104.36.82.20:4448 , piece_id=CSHYRDGTS4ROFDTKETIZ36TWJE3OEMAKLB4RGFAU557G725EANXQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617583044 (20h17m54.093521509s) [noise=true , address=104.36.82.47:4545 , piece_id=PZWVIOWLBZ72IOFRJHPAD4OZZPA2G4XYCDLGYEY7THSOR7XDHMUQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617828073 (20h17m54.093766528s) [noise=true , address=104.36.82.42:4802 , piece_id=6HORHYYKIKDX4IWWKUQPVGZQZ2YYIGC47OCZHRVBC4BZFRJX5TQQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617987597 (20h17m54.093926062s) [noise=true , address=104.36.82.13:4214 , piece_id=SLX3YJCEWCM3553VZHILH337FKSPPFNKVFTPXO4KIBOS6IXCLXHA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618055948 (20h17m54.093994403s) [noise=true , address=104.36.82.46:4500 , piece_id=VCYKKDSLEKTTL3KO2FJ4VA5JGIVPA4GFBBHH65HPFZTP4REUTHYQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618285828 (20h17m54.094224293s) [noise=true , address=104.36.82.52:4092 , piece_id=BMJCAQYXJCUVJTEXH52UXYLD4MHWLDGQTJIPDOFRF4LPLTT36ESA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618485871 (20h17m54.094424336s) [noise=true , address=104.36.82.10:4088 , piece_id=6IV2Z7FXDNEOBJMNPRG7WSPZWZTJ5CGDF6JHBSF7WCIIL2QQWFUA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618663881 (20h17m54.094602346s) [noise=true , address=104.36.82.51:4075 , piece_id=P5NU32M3V7AMYFZVJ6XZY5K2YMQWF4EULQGX6RRYWG6SWDWWAFMQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618840449 (20h17m54.094778914s) [noise=true , address=104.36.82.14:4227 , piece_id=O2FXJBOV3CJCKLZHRPBTLXX2QRT2OWJQ5HPNHZI5IVIFDVUUICVQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.619028879 (20h17m54.094967344s) [noise=true , address=104.36.82.9:4028 , piece_id=XCRNGUPBFYCZOYT23L6NHJ6WQR55TQMTWZF4SNKSRLQGC5TYO3LA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.619215837 (20h17m54.095154302s) [noise=true , address=104.36.82.53:4126 , piece_id=GZ5Z7AGMYVGYVUJAWFAIS6SQLRHFB772RLQY736QUGOXA5OWQ34Q, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.619408406 (20h17m54.095346871s) [noise=true , address=104.36.82.55:4219 , piece_id=IURHWONYUYYJWIL5BDF6HLCEPA76O6QAZAJ22NCI6LQYIGO6K4IA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:43.299790904 (20h17m55.775729369s) [upload=19386]: Next returned nil segment Jul 29 13:03:43.299806024 (20h17m55.775744479s) [upload=19386]: Waiting for error group managing segments... Jul 29 13:03:43.847040928 (20h17m56.322979393s) [noise=true , address=104.36.82.9:4028 , piece_id=XCRNGUPBFYCZOYT23L6NHJ6WQR55TQMTWZF4SNKSRLQGC5TYO3LA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:43.879495404 (20h17m56.355433869s) [noise=true , address=104.36.82.61:4391 , piece_id=TI5YDGDSTWKDX5XAWL36F3TJDQNHKNTREZVTUGIS2EVZYP6DWSDA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:46.350727192 (20h17m58.826665657s) [noise=true , address=104.36.82.5:4866 , piece_id=EFRLC4LA232NZZJU3GDSP2EEBZO3W2LGJRD3L2A6SPSRBTLUTIFA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:51.702944756 (20h18m4.178883221s) [noise=true , address=104.36.82.42:4802 , piece_id=6HORHYYKIKDX4IWWKUQPVGZQZ2YYIGC47OCZHRVBC4BZFRJX5TQQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:55.250827413 (20h18m7.726765878s) [noise=true , address=104.36.82.55:4219 , piece_id=IURHWONYUYYJWIL5BDF6HLCEPA76O6QAZAJ22NCI6LQYIGO6K4IA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:57.858469692 (20h18m10.334408157s) [noise=true , address=104.36.82.53:4126 , piece_id=GZ5Z7AGMYVGYVUJAWFAIS6SQLRHFB772RLQY736QUGOXA5OWQ34Q, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:59.265385056 (20h18m11.741323511s) [noise=true , address=104.36.82.10:4088 , piece_id=6IV2Z7FXDNEOBJMNPRG7WSPZWZTJ5CGDF6JHBSF7WCIIL2QQWFUA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:59.973580263 (20h18m12.449518729s) [noise=true , address=104.36.82.11:4119 , piece_id=6KWLABSUMX6YVKD6LGKR6ONLQKFFTSTKFHHRVFIN7U4KDALBNCBQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:16.002200978 (20h18m28.478139453s) [noise=true , address=104.36.82.52:4092 , piece_id=BMJCAQYXJCUVJTEXH52UXYLD4MHWLDGQTJIPDOFRF4LPLTT36ESA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:30.906580908 (20h18m43.382519373s) [noise=true , address=104.36.82.20:4448 , piece_id=CSHYRDGTS4ROFDTKETIZ36TWJE3OEMAKLB4RGFAU557G725EANXQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:31.164952501 (20h18m43.640890966s) [noise=true , address=104.36.82.14:4227 , piece_id=O2FXJBOV3CJCKLZHRPBTLXX2QRT2OWJQ5HPNHZI5IVIFDVUUICVQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:33.000876204 (20h18m45.476814679s) [noise=true , address=104.36.82.56:4242 , piece_id=BHKC5MH4G65BK2YLJJANIYZ4MLAUTMBG32XSRISARHCO26AZUXRA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:34.725342452 (20h18m47.201280907s) [noise=true , address=104.36.82.47:4545 , piece_id=PZWVIOWLBZ72IOFRJHPAD4OZZPA2G4XYCDLGYEY7THSOR7XDHMUQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:40.451024021 (20h18m52.926962486s) [noise=true , address=104.36.82.46:4500 , piece_id=VCYKKDSLEKTTL3KO2FJ4VA5JGIVPA4GFBBHH65HPFZTP4REUTHYQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:12B7ox4ehqxJ2mwqeXj8C9N9GuSU5etNC23iT1pJnTgMLGBAGQK, address:104.36.82.46:4500): protocol: write tcp 188.166.225.170:46700->104.36.82.46:4500: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451060500 (20h18m52.926998965s) [noise=true , address=104.36.82.51:4075 , piece_id=P5NU32M3V7AMYFZVJ6XZY5K2YMQWF4EULQGX6RRYWG6SWDWWAFMQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK, address:104.36.82.51:4075): protocol: write tcp 188.166.225.170:41162->104.36.82.51:4075: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451086179 (20h18m52.927024644s) [noise=true , address=104.36.82.54:4175 , piece_id=5PAE3BM5AFSLZ5TVUQC3QK3RVWCVGFDLOK452BDPWQ7ARTL7EYWQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:12ou7jwFyJP3mNoALddniUTDT9duVyQKZjeq4xHCyx6sRWpwEPJ, address:104.36.82.54:4175): protocol: rpc: dial tcp 104.36.82.54:4175: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451120024 (20h18m52.927058489s) [noise=true , address=104.36.82.13:4214 , piece_id=SLX3YJCEWCM3553VZHILH337FKSPPFNKVFTPXO4KIBOS6IXCLXHA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:1nJ8uqwxf6x4hX52AjAdCbMKi1BKGGbMvrQbiazJdLpn6j69ay, address:104.36.82.13:4214): protocol: rpc: dial tcp 104.36.82.13:4214: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451147116 (20h18m52.927085581s) [noise=true , address=104.36.82.15:4263 , piece_id=3GCLYN4UBCLM5BPLXDQT5SSWPJBTKIOZM4UIUNO3OZIXW6OANUBQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:1f97yBEvcrYHqMq7iovVoju77jzhUYgoQCsUGULeTagugFXrCu, address:104.36.82.15:4263): protocol: write tcp 188.166.225.170:39380->104.36.82.15:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451167805 (20h18m52.927106271s) [seg_pos={0 0}, upload=19386]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 29 13:04:40.696628655 (20h18m53.17256714s) [upload=19386]: Done uploading segments... Jul 29 13:04:40.696658391 (20h18m53.172596856s) [upload=19386]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.919159301s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [X............X...............X........X...X..XX.............] duration: 6.031926465s FAIL (2025-07-29 13:13:32 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 14 but got 7) Duration: 6.031961061s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [.X............X...............X........X...X..XX............] duration: 6.022738383s FAIL (2025-07-29 13:13:08 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 14 but got 7) Duration: 6.02276842s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.............................................X.X............] duration: 5.810736901s FAIL (2025-07-29 13:12:54 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ding segments... Jul 29 13:03:41.091320246 (20h17m53.567258701s) [upload=19386]: Got next segment. Inline: false Jul 29 13:03:41.091349191 (20h17m53.567287656s) [upload=19386]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 29 13:03:41.091354331 (20h17m53.567292796s) [upload=19386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 29 13:03:41.616444845 (20h17m54.09238331s) [seg_pos={0 0}, upload=19386]: Begin upload segment... Jul 29 13:03:41.616487748 (20h17m54.092426213s) [seg_pos={0 0}, upload=19386]: Done begin upload segment. Jul 29 13:03:41.616507376 (20h17m54.092445841s) [noise=true , address=104.36.82.15:4263 , piece_id=3GCLYN4UBCLM5BPLXDQT5SSWPJBTKIOZM4UIUNO3OZIXW6OANUBQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.616564315 (20h17m54.09250278s) [noise=true , address=104.36.82.11:4119 , piece_id=6KWLABSUMX6YVKD6LGKR6ONLQKFFTSTKFHHRVFIN7U4KDALBNCBQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.616866433 (20h17m54.092804888s) [noise=true , address=104.36.82.54:4175 , piece_id=5PAE3BM5AFSLZ5TVUQC3QK3RVWCVGFDLOK452BDPWQ7ARTL7EYWQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617072557 (20h17m54.093011022s) [noise=true , address=104.36.82.56:4242 , piece_id=BHKC5MH4G65BK2YLJJANIYZ4MLAUTMBG32XSRISARHCO26AZUXRA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617110629 (20h17m54.093049084s) [noise=true , address=104.36.82.5:4866 , piece_id=EFRLC4LA232NZZJU3GDSP2EEBZO3W2LGJRD3L2A6SPSRBTLUTIFA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617355037 (20h17m54.093293492s) [noise=true , address=104.36.82.61:4391 , piece_id=TI5YDGDSTWKDX5XAWL36F3TJDQNHKNTREZVTUGIS2EVZYP6DWSDA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617481549 (20h17m54.093420024s) [noise=true , address=104.36.82.20:4448 , piece_id=CSHYRDGTS4ROFDTKETIZ36TWJE3OEMAKLB4RGFAU557G725EANXQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617583044 (20h17m54.093521509s) [noise=true , address=104.36.82.47:4545 , piece_id=PZWVIOWLBZ72IOFRJHPAD4OZZPA2G4XYCDLGYEY7THSOR7XDHMUQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617828073 (20h17m54.093766528s) [noise=true , address=104.36.82.42:4802 , piece_id=6HORHYYKIKDX4IWWKUQPVGZQZ2YYIGC47OCZHRVBC4BZFRJX5TQQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617987597 (20h17m54.093926062s) [noise=true , address=104.36.82.13:4214 , piece_id=SLX3YJCEWCM3553VZHILH337FKSPPFNKVFTPXO4KIBOS6IXCLXHA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618055948 (20h17m54.093994403s) [noise=true , address=104.36.82.46:4500 , piece_id=VCYKKDSLEKTTL3KO2FJ4VA5JGIVPA4GFBBHH65HPFZTP4REUTHYQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618285828 (20h17m54.094224293s) [noise=true , address=104.36.82.52:4092 , piece_id=BMJCAQYXJCUVJTEXH52UXYLD4MHWLDGQTJIPDOFRF4LPLTT36ESA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618485871 (20h17m54.094424336s) [noise=true , address=104.36.82.10:4088 , piece_id=6IV2Z7FXDNEOBJMNPRG7WSPZWZTJ5CGDF6JHBSF7WCIIL2QQWFUA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618663881 (20h17m54.094602346s) [noise=true , address=104.36.82.51:4075 , piece_id=P5NU32M3V7AMYFZVJ6XZY5K2YMQWF4EULQGX6RRYWG6SWDWWAFMQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618840449 (20h17m54.094778914s) [noise=true , address=104.36.82.14:4227 , piece_id=O2FXJBOV3CJCKLZHRPBTLXX2QRT2OWJQ5HPNHZI5IVIFDVUUICVQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.619028879 (20h17m54.094967344s) [noise=true , address=104.36.82.9:4028 , piece_id=XCRNGUPBFYCZOYT23L6NHJ6WQR55TQMTWZF4SNKSRLQGC5TYO3LA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.619215837 (20h17m54.095154302s) [noise=true , address=104.36.82.53:4126 , piece_id=GZ5Z7AGMYVGYVUJAWFAIS6SQLRHFB772RLQY736QUGOXA5OWQ34Q, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.619408406 (20h17m54.095346871s) [noise=true , address=104.36.82.55:4219 , piece_id=IURHWONYUYYJWIL5BDF6HLCEPA76O6QAZAJ22NCI6LQYIGO6K4IA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:43.299790904 (20h17m55.775729369s) [upload=19386]: Next returned nil segment Jul 29 13:03:43.299806024 (20h17m55.775744479s) [upload=19386]: Waiting for error group managing segments... Jul 29 13:03:43.847040928 (20h17m56.322979393s) [noise=true , address=104.36.82.9:4028 , piece_id=XCRNGUPBFYCZOYT23L6NHJ6WQR55TQMTWZF4SNKSRLQGC5TYO3LA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:43.879495404 (20h17m56.355433869s) [noise=true , address=104.36.82.61:4391 , piece_id=TI5YDGDSTWKDX5XAWL36F3TJDQNHKNTREZVTUGIS2EVZYP6DWSDA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:46.350727192 (20h17m58.826665657s) [noise=true , address=104.36.82.5:4866 , piece_id=EFRLC4LA232NZZJU3GDSP2EEBZO3W2LGJRD3L2A6SPSRBTLUTIFA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:51.702944756 (20h18m4.178883221s) [noise=true , address=104.36.82.42:4802 , piece_id=6HORHYYKIKDX4IWWKUQPVGZQZ2YYIGC47OCZHRVBC4BZFRJX5TQQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:55.250827413 (20h18m7.726765878s) [noise=true , address=104.36.82.55:4219 , piece_id=IURHWONYUYYJWIL5BDF6HLCEPA76O6QAZAJ22NCI6LQYIGO6K4IA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:57.858469692 (20h18m10.334408157s) [noise=true , address=104.36.82.53:4126 , piece_id=GZ5Z7AGMYVGYVUJAWFAIS6SQLRHFB772RLQY736QUGOXA5OWQ34Q, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:59.265385056 (20h18m11.741323511s) [noise=true , address=104.36.82.10:4088 , piece_id=6IV2Z7FXDNEOBJMNPRG7WSPZWZTJ5CGDF6JHBSF7WCIIL2QQWFUA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:59.973580263 (20h18m12.449518729s) [noise=true , address=104.36.82.11:4119 , piece_id=6KWLABSUMX6YVKD6LGKR6ONLQKFFTSTKFHHRVFIN7U4KDALBNCBQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:16.002200978 (20h18m28.478139453s) [noise=true , address=104.36.82.52:4092 , piece_id=BMJCAQYXJCUVJTEXH52UXYLD4MHWLDGQTJIPDOFRF4LPLTT36ESA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:30.906580908 (20h18m43.382519373s) [noise=true , address=104.36.82.20:4448 , piece_id=CSHYRDGTS4ROFDTKETIZ36TWJE3OEMAKLB4RGFAU557G725EANXQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:31.164952501 (20h18m43.640890966s) [noise=true , address=104.36.82.14:4227 , piece_id=O2FXJBOV3CJCKLZHRPBTLXX2QRT2OWJQ5HPNHZI5IVIFDVUUICVQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:33.000876204 (20h18m45.476814679s) [noise=true , address=104.36.82.56:4242 , piece_id=BHKC5MH4G65BK2YLJJANIYZ4MLAUTMBG32XSRISARHCO26AZUXRA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:34.725342452 (20h18m47.201280907s) [noise=true , address=104.36.82.47:4545 , piece_id=PZWVIOWLBZ72IOFRJHPAD4OZZPA2G4XYCDLGYEY7THSOR7XDHMUQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:40.451024021 (20h18m52.926962486s) [noise=true , address=104.36.82.46:4500 , piece_id=VCYKKDSLEKTTL3KO2FJ4VA5JGIVPA4GFBBHH65HPFZTP4REUTHYQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:12B7ox4ehqxJ2mwqeXj8C9N9GuSU5etNC23iT1pJnTgMLGBAGQK, address:104.36.82.46:4500): protocol: write tcp 188.166.225.170:46700->104.36.82.46:4500: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451060500 (20h18m52.926998965s) [noise=true , address=104.36.82.51:4075 , piece_id=P5NU32M3V7AMYFZVJ6XZY5K2YMQWF4EULQGX6RRYWG6SWDWWAFMQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK, address:104.36.82.51:4075): protocol: write tcp 188.166.225.170:41162->104.36.82.51:4075: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451086179 (20h18m52.927024644s) [noise=true , address=104.36.82.54:4175 , piece_id=5PAE3BM5AFSLZ5TVUQC3QK3RVWCVGFDLOK452BDPWQ7ARTL7EYWQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:12ou7jwFyJP3mNoALddniUTDT9duVyQKZjeq4xHCyx6sRWpwEPJ, address:104.36.82.54:4175): protocol: rpc: dial tcp 104.36.82.54:4175: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451120024 (20h18m52.927058489s) [noise=true , address=104.36.82.13:4214 , piece_id=SLX3YJCEWCM3553VZHILH337FKSPPFNKVFTPXO4KIBOS6IXCLXHA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:1nJ8uqwxf6x4hX52AjAdCbMKi1BKGGbMvrQbiazJdLpn6j69ay, address:104.36.82.13:4214): protocol: rpc: dial tcp 104.36.82.13:4214: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451147116 (20h18m52.927085581s) [noise=true , address=104.36.82.15:4263 , piece_id=3GCLYN4UBCLM5BPLXDQT5SSWPJBTKIOZM4UIUNO3OZIXW6OANUBQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:1f97yBEvcrYHqMq7iovVoju77jzhUYgoQCsUGULeTagugFXrCu, address:104.36.82.15:4263): protocol: write tcp 188.166.225.170:39380->104.36.82.15:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451167805 (20h18m52.927106271s) [seg_pos={0 0}, upload=19386]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 29 13:04:40.696628655 (20h18m53.17256714s) [upload=19386]: Done uploading segments... Jul 29 13:04:40.696658391 (20h18m53.172596856s) [upload=19386]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.810765885s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [.....XX............X...........X............................] duration: 1.853576725s FAIL (2025-07-29 13:12:47 UTC) orbiter warning: unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K Duration: 1.853606212s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..............................................X.X...........] duration: 5.844678001s FAIL (2025-07-29 13:12:22 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ding segments... Jul 29 13:03:41.091320246 (20h17m53.567258701s) [upload=19386]: Got next segment. Inline: false Jul 29 13:03:41.091349191 (20h17m53.567287656s) [upload=19386]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 29 13:03:41.091354331 (20h17m53.567292796s) [upload=19386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 29 13:03:41.616444845 (20h17m54.09238331s) [seg_pos={0 0}, upload=19386]: Begin upload segment... Jul 29 13:03:41.616487748 (20h17m54.092426213s) [seg_pos={0 0}, upload=19386]: Done begin upload segment. Jul 29 13:03:41.616507376 (20h17m54.092445841s) [noise=true , address=104.36.82.15:4263 , piece_id=3GCLYN4UBCLM5BPLXDQT5SSWPJBTKIOZM4UIUNO3OZIXW6OANUBQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.616564315 (20h17m54.09250278s) [noise=true , address=104.36.82.11:4119 , piece_id=6KWLABSUMX6YVKD6LGKR6ONLQKFFTSTKFHHRVFIN7U4KDALBNCBQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.616866433 (20h17m54.092804888s) [noise=true , address=104.36.82.54:4175 , piece_id=5PAE3BM5AFSLZ5TVUQC3QK3RVWCVGFDLOK452BDPWQ7ARTL7EYWQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617072557 (20h17m54.093011022s) [noise=true , address=104.36.82.56:4242 , piece_id=BHKC5MH4G65BK2YLJJANIYZ4MLAUTMBG32XSRISARHCO26AZUXRA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617110629 (20h17m54.093049084s) [noise=true , address=104.36.82.5:4866 , piece_id=EFRLC4LA232NZZJU3GDSP2EEBZO3W2LGJRD3L2A6SPSRBTLUTIFA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617355037 (20h17m54.093293492s) [noise=true , address=104.36.82.61:4391 , piece_id=TI5YDGDSTWKDX5XAWL36F3TJDQNHKNTREZVTUGIS2EVZYP6DWSDA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617481549 (20h17m54.093420024s) [noise=true , address=104.36.82.20:4448 , piece_id=CSHYRDGTS4ROFDTKETIZ36TWJE3OEMAKLB4RGFAU557G725EANXQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617583044 (20h17m54.093521509s) [noise=true , address=104.36.82.47:4545 , piece_id=PZWVIOWLBZ72IOFRJHPAD4OZZPA2G4XYCDLGYEY7THSOR7XDHMUQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617828073 (20h17m54.093766528s) [noise=true , address=104.36.82.42:4802 , piece_id=6HORHYYKIKDX4IWWKUQPVGZQZ2YYIGC47OCZHRVBC4BZFRJX5TQQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617987597 (20h17m54.093926062s) [noise=true , address=104.36.82.13:4214 , piece_id=SLX3YJCEWCM3553VZHILH337FKSPPFNKVFTPXO4KIBOS6IXCLXHA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618055948 (20h17m54.093994403s) [noise=true , address=104.36.82.46:4500 , piece_id=VCYKKDSLEKTTL3KO2FJ4VA5JGIVPA4GFBBHH65HPFZTP4REUTHYQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618285828 (20h17m54.094224293s) [noise=true , address=104.36.82.52:4092 , piece_id=BMJCAQYXJCUVJTEXH52UXYLD4MHWLDGQTJIPDOFRF4LPLTT36ESA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618485871 (20h17m54.094424336s) [noise=true , address=104.36.82.10:4088 , piece_id=6IV2Z7FXDNEOBJMNPRG7WSPZWZTJ5CGDF6JHBSF7WCIIL2QQWFUA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618663881 (20h17m54.094602346s) [noise=true , address=104.36.82.51:4075 , piece_id=P5NU32M3V7AMYFZVJ6XZY5K2YMQWF4EULQGX6RRYWG6SWDWWAFMQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618840449 (20h17m54.094778914s) [noise=true , address=104.36.82.14:4227 , piece_id=O2FXJBOV3CJCKLZHRPBTLXX2QRT2OWJQ5HPNHZI5IVIFDVUUICVQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.619028879 (20h17m54.094967344s) [noise=true , address=104.36.82.9:4028 , piece_id=XCRNGUPBFYCZOYT23L6NHJ6WQR55TQMTWZF4SNKSRLQGC5TYO3LA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.619215837 (20h17m54.095154302s) [noise=true , address=104.36.82.53:4126 , piece_id=GZ5Z7AGMYVGYVUJAWFAIS6SQLRHFB772RLQY736QUGOXA5OWQ34Q, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.619408406 (20h17m54.095346871s) [noise=true , address=104.36.82.55:4219 , piece_id=IURHWONYUYYJWIL5BDF6HLCEPA76O6QAZAJ22NCI6LQYIGO6K4IA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:43.299790904 (20h17m55.775729369s) [upload=19386]: Next returned nil segment Jul 29 13:03:43.299806024 (20h17m55.775744479s) [upload=19386]: Waiting for error group managing segments... Jul 29 13:03:43.847040928 (20h17m56.322979393s) [noise=true , address=104.36.82.9:4028 , piece_id=XCRNGUPBFYCZOYT23L6NHJ6WQR55TQMTWZF4SNKSRLQGC5TYO3LA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:43.879495404 (20h17m56.355433869s) [noise=true , address=104.36.82.61:4391 , piece_id=TI5YDGDSTWKDX5XAWL36F3TJDQNHKNTREZVTUGIS2EVZYP6DWSDA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:46.350727192 (20h17m58.826665657s) [noise=true , address=104.36.82.5:4866 , piece_id=EFRLC4LA232NZZJU3GDSP2EEBZO3W2LGJRD3L2A6SPSRBTLUTIFA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:51.702944756 (20h18m4.178883221s) [noise=true , address=104.36.82.42:4802 , piece_id=6HORHYYKIKDX4IWWKUQPVGZQZ2YYIGC47OCZHRVBC4BZFRJX5TQQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:55.250827413 (20h18m7.726765878s) [noise=true , address=104.36.82.55:4219 , piece_id=IURHWONYUYYJWIL5BDF6HLCEPA76O6QAZAJ22NCI6LQYIGO6K4IA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:57.858469692 (20h18m10.334408157s) [noise=true , address=104.36.82.53:4126 , piece_id=GZ5Z7AGMYVGYVUJAWFAIS6SQLRHFB772RLQY736QUGOXA5OWQ34Q, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:59.265385056 (20h18m11.741323511s) [noise=true , address=104.36.82.10:4088 , piece_id=6IV2Z7FXDNEOBJMNPRG7WSPZWZTJ5CGDF6JHBSF7WCIIL2QQWFUA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:59.973580263 (20h18m12.449518729s) [noise=true , address=104.36.82.11:4119 , piece_id=6KWLABSUMX6YVKD6LGKR6ONLQKFFTSTKFHHRVFIN7U4KDALBNCBQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:16.002200978 (20h18m28.478139453s) [noise=true , address=104.36.82.52:4092 , piece_id=BMJCAQYXJCUVJTEXH52UXYLD4MHWLDGQTJIPDOFRF4LPLTT36ESA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:30.906580908 (20h18m43.382519373s) [noise=true , address=104.36.82.20:4448 , piece_id=CSHYRDGTS4ROFDTKETIZ36TWJE3OEMAKLB4RGFAU557G725EANXQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:31.164952501 (20h18m43.640890966s) [noise=true , address=104.36.82.14:4227 , piece_id=O2FXJBOV3CJCKLZHRPBTLXX2QRT2OWJQ5HPNHZI5IVIFDVUUICVQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:33.000876204 (20h18m45.476814679s) [noise=true , address=104.36.82.56:4242 , piece_id=BHKC5MH4G65BK2YLJJANIYZ4MLAUTMBG32XSRISARHCO26AZUXRA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:34.725342452 (20h18m47.201280907s) [noise=true , address=104.36.82.47:4545 , piece_id=PZWVIOWLBZ72IOFRJHPAD4OZZPA2G4XYCDLGYEY7THSOR7XDHMUQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:40.451024021 (20h18m52.926962486s) [noise=true , address=104.36.82.46:4500 , piece_id=VCYKKDSLEKTTL3KO2FJ4VA5JGIVPA4GFBBHH65HPFZTP4REUTHYQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:12B7ox4ehqxJ2mwqeXj8C9N9GuSU5etNC23iT1pJnTgMLGBAGQK, address:104.36.82.46:4500): protocol: write tcp 188.166.225.170:46700->104.36.82.46:4500: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451060500 (20h18m52.926998965s) [noise=true , address=104.36.82.51:4075 , piece_id=P5NU32M3V7AMYFZVJ6XZY5K2YMQWF4EULQGX6RRYWG6SWDWWAFMQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK, address:104.36.82.51:4075): protocol: write tcp 188.166.225.170:41162->104.36.82.51:4075: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451086179 (20h18m52.927024644s) [noise=true , address=104.36.82.54:4175 , piece_id=5PAE3BM5AFSLZ5TVUQC3QK3RVWCVGFDLOK452BDPWQ7ARTL7EYWQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:12ou7jwFyJP3mNoALddniUTDT9duVyQKZjeq4xHCyx6sRWpwEPJ, address:104.36.82.54:4175): protocol: rpc: dial tcp 104.36.82.54:4175: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451120024 (20h18m52.927058489s) [noise=true , address=104.36.82.13:4214 , piece_id=SLX3YJCEWCM3553VZHILH337FKSPPFNKVFTPXO4KIBOS6IXCLXHA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:1nJ8uqwxf6x4hX52AjAdCbMKi1BKGGbMvrQbiazJdLpn6j69ay, address:104.36.82.13:4214): protocol: rpc: dial tcp 104.36.82.13:4214: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451147116 (20h18m52.927085581s) [noise=true , address=104.36.82.15:4263 , piece_id=3GCLYN4UBCLM5BPLXDQT5SSWPJBTKIOZM4UIUNO3OZIXW6OANUBQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:1f97yBEvcrYHqMq7iovVoju77jzhUYgoQCsUGULeTagugFXrCu, address:104.36.82.15:4263): protocol: write tcp 188.166.225.170:39380->104.36.82.15:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451167805 (20h18m52.927106271s) [seg_pos={0 0}, upload=19386]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 29 13:04:40.696628655 (20h18m53.17256714s) [upload=19386]: Done uploading segments... Jul 29 13:04:40.696658391 (20h18m53.172596856s) [upload=19386]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.844699642s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [..X............X...............X........X...X..XX...........] duration: 6.058839944s FAIL (2025-07-29 13:12:11 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 14 but got 7) Duration: 6.058864951s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [...X............X...............X........X...X..XX..........] duration: 6.004734655s FAIL (2025-07-29 13:11:55 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 14 but got 7) Duration: 6.004766174s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [......XX............X...........X...........................] duration: 2.018366294s FAIL (2025-07-29 13:11:49 UTC) orbiter warning: unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K Duration: 2.018396532s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...............................................X.X..........] duration: 5.983506325s FAIL (2025-07-29 13:11:27 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ding segments... Jul 29 13:03:41.091320246 (20h17m53.567258701s) [upload=19386]: Got next segment. Inline: false Jul 29 13:03:41.091349191 (20h17m53.567287656s) [upload=19386]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 29 13:03:41.091354331 (20h17m53.567292796s) [upload=19386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 29 13:03:41.616444845 (20h17m54.09238331s) [seg_pos={0 0}, upload=19386]: Begin upload segment... Jul 29 13:03:41.616487748 (20h17m54.092426213s) [seg_pos={0 0}, upload=19386]: Done begin upload segment. Jul 29 13:03:41.616507376 (20h17m54.092445841s) [noise=true , address=104.36.82.15:4263 , piece_id=3GCLYN4UBCLM5BPLXDQT5SSWPJBTKIOZM4UIUNO3OZIXW6OANUBQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.616564315 (20h17m54.09250278s) [noise=true , address=104.36.82.11:4119 , piece_id=6KWLABSUMX6YVKD6LGKR6ONLQKFFTSTKFHHRVFIN7U4KDALBNCBQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.616866433 (20h17m54.092804888s) [noise=true , address=104.36.82.54:4175 , piece_id=5PAE3BM5AFSLZ5TVUQC3QK3RVWCVGFDLOK452BDPWQ7ARTL7EYWQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617072557 (20h17m54.093011022s) [noise=true , address=104.36.82.56:4242 , piece_id=BHKC5MH4G65BK2YLJJANIYZ4MLAUTMBG32XSRISARHCO26AZUXRA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617110629 (20h17m54.093049084s) [noise=true , address=104.36.82.5:4866 , piece_id=EFRLC4LA232NZZJU3GDSP2EEBZO3W2LGJRD3L2A6SPSRBTLUTIFA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617355037 (20h17m54.093293492s) [noise=true , address=104.36.82.61:4391 , piece_id=TI5YDGDSTWKDX5XAWL36F3TJDQNHKNTREZVTUGIS2EVZYP6DWSDA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617481549 (20h17m54.093420024s) [noise=true , address=104.36.82.20:4448 , piece_id=CSHYRDGTS4ROFDTKETIZ36TWJE3OEMAKLB4RGFAU557G725EANXQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617583044 (20h17m54.093521509s) [noise=true , address=104.36.82.47:4545 , piece_id=PZWVIOWLBZ72IOFRJHPAD4OZZPA2G4XYCDLGYEY7THSOR7XDHMUQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617828073 (20h17m54.093766528s) [noise=true , address=104.36.82.42:4802 , piece_id=6HORHYYKIKDX4IWWKUQPVGZQZ2YYIGC47OCZHRVBC4BZFRJX5TQQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617987597 (20h17m54.093926062s) [noise=true , address=104.36.82.13:4214 , piece_id=SLX3YJCEWCM3553VZHILH337FKSPPFNKVFTPXO4KIBOS6IXCLXHA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618055948 (20h17m54.093994403s) [noise=true , address=104.36.82.46:4500 , piece_id=VCYKKDSLEKTTL3KO2FJ4VA5JGIVPA4GFBBHH65HPFZTP4REUTHYQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618285828 (20h17m54.094224293s) [noise=true , address=104.36.82.52:4092 , piece_id=BMJCAQYXJCUVJTEXH52UXYLD4MHWLDGQTJIPDOFRF4LPLTT36ESA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618485871 (20h17m54.094424336s) [noise=true , address=104.36.82.10:4088 , piece_id=6IV2Z7FXDNEOBJMNPRG7WSPZWZTJ5CGDF6JHBSF7WCIIL2QQWFUA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618663881 (20h17m54.094602346s) [noise=true , address=104.36.82.51:4075 , piece_id=P5NU32M3V7AMYFZVJ6XZY5K2YMQWF4EULQGX6RRYWG6SWDWWAFMQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618840449 (20h17m54.094778914s) [noise=true , address=104.36.82.14:4227 , piece_id=O2FXJBOV3CJCKLZHRPBTLXX2QRT2OWJQ5HPNHZI5IVIFDVUUICVQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.619028879 (20h17m54.094967344s) [noise=true , address=104.36.82.9:4028 , piece_id=XCRNGUPBFYCZOYT23L6NHJ6WQR55TQMTWZF4SNKSRLQGC5TYO3LA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.619215837 (20h17m54.095154302s) [noise=true , address=104.36.82.53:4126 , piece_id=GZ5Z7AGMYVGYVUJAWFAIS6SQLRHFB772RLQY736QUGOXA5OWQ34Q, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.619408406 (20h17m54.095346871s) [noise=true , address=104.36.82.55:4219 , piece_id=IURHWONYUYYJWIL5BDF6HLCEPA76O6QAZAJ22NCI6LQYIGO6K4IA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:43.299790904 (20h17m55.775729369s) [upload=19386]: Next returned nil segment Jul 29 13:03:43.299806024 (20h17m55.775744479s) [upload=19386]: Waiting for error group managing segments... Jul 29 13:03:43.847040928 (20h17m56.322979393s) [noise=true , address=104.36.82.9:4028 , piece_id=XCRNGUPBFYCZOYT23L6NHJ6WQR55TQMTWZF4SNKSRLQGC5TYO3LA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:43.879495404 (20h17m56.355433869s) [noise=true , address=104.36.82.61:4391 , piece_id=TI5YDGDSTWKDX5XAWL36F3TJDQNHKNTREZVTUGIS2EVZYP6DWSDA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:46.350727192 (20h17m58.826665657s) [noise=true , address=104.36.82.5:4866 , piece_id=EFRLC4LA232NZZJU3GDSP2EEBZO3W2LGJRD3L2A6SPSRBTLUTIFA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:51.702944756 (20h18m4.178883221s) [noise=true , address=104.36.82.42:4802 , piece_id=6HORHYYKIKDX4IWWKUQPVGZQZ2YYIGC47OCZHRVBC4BZFRJX5TQQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:55.250827413 (20h18m7.726765878s) [noise=true , address=104.36.82.55:4219 , piece_id=IURHWONYUYYJWIL5BDF6HLCEPA76O6QAZAJ22NCI6LQYIGO6K4IA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:57.858469692 (20h18m10.334408157s) [noise=true , address=104.36.82.53:4126 , piece_id=GZ5Z7AGMYVGYVUJAWFAIS6SQLRHFB772RLQY736QUGOXA5OWQ34Q, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:59.265385056 (20h18m11.741323511s) [noise=true , address=104.36.82.10:4088 , piece_id=6IV2Z7FXDNEOBJMNPRG7WSPZWZTJ5CGDF6JHBSF7WCIIL2QQWFUA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:59.973580263 (20h18m12.449518729s) [noise=true , address=104.36.82.11:4119 , piece_id=6KWLABSUMX6YVKD6LGKR6ONLQKFFTSTKFHHRVFIN7U4KDALBNCBQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:16.002200978 (20h18m28.478139453s) [noise=true , address=104.36.82.52:4092 , piece_id=BMJCAQYXJCUVJTEXH52UXYLD4MHWLDGQTJIPDOFRF4LPLTT36ESA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:30.906580908 (20h18m43.382519373s) [noise=true , address=104.36.82.20:4448 , piece_id=CSHYRDGTS4ROFDTKETIZ36TWJE3OEMAKLB4RGFAU557G725EANXQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:31.164952501 (20h18m43.640890966s) [noise=true , address=104.36.82.14:4227 , piece_id=O2FXJBOV3CJCKLZHRPBTLXX2QRT2OWJQ5HPNHZI5IVIFDVUUICVQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:33.000876204 (20h18m45.476814679s) [noise=true , address=104.36.82.56:4242 , piece_id=BHKC5MH4G65BK2YLJJANIYZ4MLAUTMBG32XSRISARHCO26AZUXRA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:34.725342452 (20h18m47.201280907s) [noise=true , address=104.36.82.47:4545 , piece_id=PZWVIOWLBZ72IOFRJHPAD4OZZPA2G4XYCDLGYEY7THSOR7XDHMUQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:40.451024021 (20h18m52.926962486s) [noise=true , address=104.36.82.46:4500 , piece_id=VCYKKDSLEKTTL3KO2FJ4VA5JGIVPA4GFBBHH65HPFZTP4REUTHYQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:12B7ox4ehqxJ2mwqeXj8C9N9GuSU5etNC23iT1pJnTgMLGBAGQK, address:104.36.82.46:4500): protocol: write tcp 188.166.225.170:46700->104.36.82.46:4500: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451060500 (20h18m52.926998965s) [noise=true , address=104.36.82.51:4075 , piece_id=P5NU32M3V7AMYFZVJ6XZY5K2YMQWF4EULQGX6RRYWG6SWDWWAFMQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK, address:104.36.82.51:4075): protocol: write tcp 188.166.225.170:41162->104.36.82.51:4075: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451086179 (20h18m52.927024644s) [noise=true , address=104.36.82.54:4175 , piece_id=5PAE3BM5AFSLZ5TVUQC3QK3RVWCVGFDLOK452BDPWQ7ARTL7EYWQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:12ou7jwFyJP3mNoALddniUTDT9duVyQKZjeq4xHCyx6sRWpwEPJ, address:104.36.82.54:4175): protocol: rpc: dial tcp 104.36.82.54:4175: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451120024 (20h18m52.927058489s) [noise=true , address=104.36.82.13:4214 , piece_id=SLX3YJCEWCM3553VZHILH337FKSPPFNKVFTPXO4KIBOS6IXCLXHA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:1nJ8uqwxf6x4hX52AjAdCbMKi1BKGGbMvrQbiazJdLpn6j69ay, address:104.36.82.13:4214): protocol: rpc: dial tcp 104.36.82.13:4214: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451147116 (20h18m52.927085581s) [noise=true , address=104.36.82.15:4263 , piece_id=3GCLYN4UBCLM5BPLXDQT5SSWPJBTKIOZM4UIUNO3OZIXW6OANUBQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:1f97yBEvcrYHqMq7iovVoju77jzhUYgoQCsUGULeTagugFXrCu, address:104.36.82.15:4263): protocol: write tcp 188.166.225.170:39380->104.36.82.15:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451167805 (20h18m52.927106271s) [seg_pos={0 0}, upload=19386]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 29 13:04:40.696628655 (20h18m53.17256714s) [upload=19386]: Done uploading segments... Jul 29 13:04:40.696658391 (20h18m53.172596856s) [upload=19386]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.983530392s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [....X............X...............X........X...X..XX.........] duration: 6.047447894s FAIL (2025-07-29 13:10:55 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 14 but got 7) Duration: 6.047486519s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [................................................X.X.........] duration: 6.013573615s FAIL (2025-07-29 13:10:48 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ding segments... Jul 29 13:03:41.091320246 (20h17m53.567258701s) [upload=19386]: Got next segment. Inline: false Jul 29 13:03:41.091349191 (20h17m53.567287656s) [upload=19386]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 29 13:03:41.091354331 (20h17m53.567292796s) [upload=19386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 29 13:03:41.616444845 (20h17m54.09238331s) [seg_pos={0 0}, upload=19386]: Begin upload segment... Jul 29 13:03:41.616487748 (20h17m54.092426213s) [seg_pos={0 0}, upload=19386]: Done begin upload segment. Jul 29 13:03:41.616507376 (20h17m54.092445841s) [noise=true , address=104.36.82.15:4263 , piece_id=3GCLYN4UBCLM5BPLXDQT5SSWPJBTKIOZM4UIUNO3OZIXW6OANUBQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.616564315 (20h17m54.09250278s) [noise=true , address=104.36.82.11:4119 , piece_id=6KWLABSUMX6YVKD6LGKR6ONLQKFFTSTKFHHRVFIN7U4KDALBNCBQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.616866433 (20h17m54.092804888s) [noise=true , address=104.36.82.54:4175 , piece_id=5PAE3BM5AFSLZ5TVUQC3QK3RVWCVGFDLOK452BDPWQ7ARTL7EYWQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617072557 (20h17m54.093011022s) [noise=true , address=104.36.82.56:4242 , piece_id=BHKC5MH4G65BK2YLJJANIYZ4MLAUTMBG32XSRISARHCO26AZUXRA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617110629 (20h17m54.093049084s) [noise=true , address=104.36.82.5:4866 , piece_id=EFRLC4LA232NZZJU3GDSP2EEBZO3W2LGJRD3L2A6SPSRBTLUTIFA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617355037 (20h17m54.093293492s) [noise=true , address=104.36.82.61:4391 , piece_id=TI5YDGDSTWKDX5XAWL36F3TJDQNHKNTREZVTUGIS2EVZYP6DWSDA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617481549 (20h17m54.093420024s) [noise=true , address=104.36.82.20:4448 , piece_id=CSHYRDGTS4ROFDTKETIZ36TWJE3OEMAKLB4RGFAU557G725EANXQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617583044 (20h17m54.093521509s) [noise=true , address=104.36.82.47:4545 , piece_id=PZWVIOWLBZ72IOFRJHPAD4OZZPA2G4XYCDLGYEY7THSOR7XDHMUQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617828073 (20h17m54.093766528s) [noise=true , address=104.36.82.42:4802 , piece_id=6HORHYYKIKDX4IWWKUQPVGZQZ2YYIGC47OCZHRVBC4BZFRJX5TQQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617987597 (20h17m54.093926062s) [noise=true , address=104.36.82.13:4214 , piece_id=SLX3YJCEWCM3553VZHILH337FKSPPFNKVFTPXO4KIBOS6IXCLXHA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618055948 (20h17m54.093994403s) [noise=true , address=104.36.82.46:4500 , piece_id=VCYKKDSLEKTTL3KO2FJ4VA5JGIVPA4GFBBHH65HPFZTP4REUTHYQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618285828 (20h17m54.094224293s) [noise=true , address=104.36.82.52:4092 , piece_id=BMJCAQYXJCUVJTEXH52UXYLD4MHWLDGQTJIPDOFRF4LPLTT36ESA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618485871 (20h17m54.094424336s) [noise=true , address=104.36.82.10:4088 , piece_id=6IV2Z7FXDNEOBJMNPRG7WSPZWZTJ5CGDF6JHBSF7WCIIL2QQWFUA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618663881 (20h17m54.094602346s) [noise=true , address=104.36.82.51:4075 , piece_id=P5NU32M3V7AMYFZVJ6XZY5K2YMQWF4EULQGX6RRYWG6SWDWWAFMQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618840449 (20h17m54.094778914s) [noise=true , address=104.36.82.14:4227 , piece_id=O2FXJBOV3CJCKLZHRPBTLXX2QRT2OWJQ5HPNHZI5IVIFDVUUICVQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.619028879 (20h17m54.094967344s) [noise=true , address=104.36.82.9:4028 , piece_id=XCRNGUPBFYCZOYT23L6NHJ6WQR55TQMTWZF4SNKSRLQGC5TYO3LA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.619215837 (20h17m54.095154302s) [noise=true , address=104.36.82.53:4126 , piece_id=GZ5Z7AGMYVGYVUJAWFAIS6SQLRHFB772RLQY736QUGOXA5OWQ34Q, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.619408406 (20h17m54.095346871s) [noise=true , address=104.36.82.55:4219 , piece_id=IURHWONYUYYJWIL5BDF6HLCEPA76O6QAZAJ22NCI6LQYIGO6K4IA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:43.299790904 (20h17m55.775729369s) [upload=19386]: Next returned nil segment Jul 29 13:03:43.299806024 (20h17m55.775744479s) [upload=19386]: Waiting for error group managing segments... Jul 29 13:03:43.847040928 (20h17m56.322979393s) [noise=true , address=104.36.82.9:4028 , piece_id=XCRNGUPBFYCZOYT23L6NHJ6WQR55TQMTWZF4SNKSRLQGC5TYO3LA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:43.879495404 (20h17m56.355433869s) [noise=true , address=104.36.82.61:4391 , piece_id=TI5YDGDSTWKDX5XAWL36F3TJDQNHKNTREZVTUGIS2EVZYP6DWSDA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:46.350727192 (20h17m58.826665657s) [noise=true , address=104.36.82.5:4866 , piece_id=EFRLC4LA232NZZJU3GDSP2EEBZO3W2LGJRD3L2A6SPSRBTLUTIFA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:51.702944756 (20h18m4.178883221s) [noise=true , address=104.36.82.42:4802 , piece_id=6HORHYYKIKDX4IWWKUQPVGZQZ2YYIGC47OCZHRVBC4BZFRJX5TQQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:55.250827413 (20h18m7.726765878s) [noise=true , address=104.36.82.55:4219 , piece_id=IURHWONYUYYJWIL5BDF6HLCEPA76O6QAZAJ22NCI6LQYIGO6K4IA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:57.858469692 (20h18m10.334408157s) [noise=true , address=104.36.82.53:4126 , piece_id=GZ5Z7AGMYVGYVUJAWFAIS6SQLRHFB772RLQY736QUGOXA5OWQ34Q, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:59.265385056 (20h18m11.741323511s) [noise=true , address=104.36.82.10:4088 , piece_id=6IV2Z7FXDNEOBJMNPRG7WSPZWZTJ5CGDF6JHBSF7WCIIL2QQWFUA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:59.973580263 (20h18m12.449518729s) [noise=true , address=104.36.82.11:4119 , piece_id=6KWLABSUMX6YVKD6LGKR6ONLQKFFTSTKFHHRVFIN7U4KDALBNCBQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:16.002200978 (20h18m28.478139453s) [noise=true , address=104.36.82.52:4092 , piece_id=BMJCAQYXJCUVJTEXH52UXYLD4MHWLDGQTJIPDOFRF4LPLTT36ESA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:30.906580908 (20h18m43.382519373s) [noise=true , address=104.36.82.20:4448 , piece_id=CSHYRDGTS4ROFDTKETIZ36TWJE3OEMAKLB4RGFAU557G725EANXQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:31.164952501 (20h18m43.640890966s) [noise=true , address=104.36.82.14:4227 , piece_id=O2FXJBOV3CJCKLZHRPBTLXX2QRT2OWJQ5HPNHZI5IVIFDVUUICVQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:33.000876204 (20h18m45.476814679s) [noise=true , address=104.36.82.56:4242 , piece_id=BHKC5MH4G65BK2YLJJANIYZ4MLAUTMBG32XSRISARHCO26AZUXRA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:34.725342452 (20h18m47.201280907s) [noise=true , address=104.36.82.47:4545 , piece_id=PZWVIOWLBZ72IOFRJHPAD4OZZPA2G4XYCDLGYEY7THSOR7XDHMUQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:40.451024021 (20h18m52.926962486s) [noise=true , address=104.36.82.46:4500 , piece_id=VCYKKDSLEKTTL3KO2FJ4VA5JGIVPA4GFBBHH65HPFZTP4REUTHYQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:12B7ox4ehqxJ2mwqeXj8C9N9GuSU5etNC23iT1pJnTgMLGBAGQK, address:104.36.82.46:4500): protocol: write tcp 188.166.225.170:46700->104.36.82.46:4500: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451060500 (20h18m52.926998965s) [noise=true , address=104.36.82.51:4075 , piece_id=P5NU32M3V7AMYFZVJ6XZY5K2YMQWF4EULQGX6RRYWG6SWDWWAFMQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK, address:104.36.82.51:4075): protocol: write tcp 188.166.225.170:41162->104.36.82.51:4075: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451086179 (20h18m52.927024644s) [noise=true , address=104.36.82.54:4175 , piece_id=5PAE3BM5AFSLZ5TVUQC3QK3RVWCVGFDLOK452BDPWQ7ARTL7EYWQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:12ou7jwFyJP3mNoALddniUTDT9duVyQKZjeq4xHCyx6sRWpwEPJ, address:104.36.82.54:4175): protocol: rpc: dial tcp 104.36.82.54:4175: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451120024 (20h18m52.927058489s) [noise=true , address=104.36.82.13:4214 , piece_id=SLX3YJCEWCM3553VZHILH337FKSPPFNKVFTPXO4KIBOS6IXCLXHA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:1nJ8uqwxf6x4hX52AjAdCbMKi1BKGGbMvrQbiazJdLpn6j69ay, address:104.36.82.13:4214): protocol: rpc: dial tcp 104.36.82.13:4214: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451147116 (20h18m52.927085581s) [noise=true , address=104.36.82.15:4263 , piece_id=3GCLYN4UBCLM5BPLXDQT5SSWPJBTKIOZM4UIUNO3OZIXW6OANUBQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:1f97yBEvcrYHqMq7iovVoju77jzhUYgoQCsUGULeTagugFXrCu, address:104.36.82.15:4263): protocol: write tcp 188.166.225.170:39380->104.36.82.15:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451167805 (20h18m52.927106271s) [seg_pos={0 0}, upload=19386]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 29 13:04:40.696628655 (20h18m53.17256714s) [upload=19386]: Done uploading segments... Jul 29 13:04:40.696658391 (20h18m53.172596856s) [upload=19386]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.013598071s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [.......XX............X...........X..........................] duration: 1.95495889s FAIL (2025-07-29 13:10:39 UTC) orbiter warning: unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K Duration: 1.954991583s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [.....X............X...............X........X...X..XX........] duration: 5.960770214s FAIL (2025-07-29 13:09:52 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 14 but got 7) Duration: 5.960796876s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [........XX............X...........X.........................] duration: 1.858109227s FAIL (2025-07-29 13:09:44 UTC) orbiter warning: unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K Duration: 1.858134806s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.................................................X.X........] duration: 5.931730842s FAIL (2025-07-29 13:09:33 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ding segments... Jul 29 13:03:41.091320246 (20h17m53.567258701s) [upload=19386]: Got next segment. Inline: false Jul 29 13:03:41.091349191 (20h17m53.567287656s) [upload=19386]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 29 13:03:41.091354331 (20h17m53.567292796s) [upload=19386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 29 13:03:41.616444845 (20h17m54.09238331s) [seg_pos={0 0}, upload=19386]: Begin upload segment... Jul 29 13:03:41.616487748 (20h17m54.092426213s) [seg_pos={0 0}, upload=19386]: Done begin upload segment. Jul 29 13:03:41.616507376 (20h17m54.092445841s) [noise=true , address=104.36.82.15:4263 , piece_id=3GCLYN4UBCLM5BPLXDQT5SSWPJBTKIOZM4UIUNO3OZIXW6OANUBQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.616564315 (20h17m54.09250278s) [noise=true , address=104.36.82.11:4119 , piece_id=6KWLABSUMX6YVKD6LGKR6ONLQKFFTSTKFHHRVFIN7U4KDALBNCBQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.616866433 (20h17m54.092804888s) [noise=true , address=104.36.82.54:4175 , piece_id=5PAE3BM5AFSLZ5TVUQC3QK3RVWCVGFDLOK452BDPWQ7ARTL7EYWQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617072557 (20h17m54.093011022s) [noise=true , address=104.36.82.56:4242 , piece_id=BHKC5MH4G65BK2YLJJANIYZ4MLAUTMBG32XSRISARHCO26AZUXRA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617110629 (20h17m54.093049084s) [noise=true , address=104.36.82.5:4866 , piece_id=EFRLC4LA232NZZJU3GDSP2EEBZO3W2LGJRD3L2A6SPSRBTLUTIFA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617355037 (20h17m54.093293492s) [noise=true , address=104.36.82.61:4391 , piece_id=TI5YDGDSTWKDX5XAWL36F3TJDQNHKNTREZVTUGIS2EVZYP6DWSDA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617481549 (20h17m54.093420024s) [noise=true , address=104.36.82.20:4448 , piece_id=CSHYRDGTS4ROFDTKETIZ36TWJE3OEMAKLB4RGFAU557G725EANXQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617583044 (20h17m54.093521509s) [noise=true , address=104.36.82.47:4545 , piece_id=PZWVIOWLBZ72IOFRJHPAD4OZZPA2G4XYCDLGYEY7THSOR7XDHMUQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617828073 (20h17m54.093766528s) [noise=true , address=104.36.82.42:4802 , piece_id=6HORHYYKIKDX4IWWKUQPVGZQZ2YYIGC47OCZHRVBC4BZFRJX5TQQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617987597 (20h17m54.093926062s) [noise=true , address=104.36.82.13:4214 , piece_id=SLX3YJCEWCM3553VZHILH337FKSPPFNKVFTPXO4KIBOS6IXCLXHA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618055948 (20h17m54.093994403s) [noise=true , address=104.36.82.46:4500 , piece_id=VCYKKDSLEKTTL3KO2FJ4VA5JGIVPA4GFBBHH65HPFZTP4REUTHYQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618285828 (20h17m54.094224293s) [noise=true , address=104.36.82.52:4092 , piece_id=BMJCAQYXJCUVJTEXH52UXYLD4MHWLDGQTJIPDOFRF4LPLTT36ESA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618485871 (20h17m54.094424336s) [noise=true , address=104.36.82.10:4088 , piece_id=6IV2Z7FXDNEOBJMNPRG7WSPZWZTJ5CGDF6JHBSF7WCIIL2QQWFUA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618663881 (20h17m54.094602346s) [noise=true , address=104.36.82.51:4075 , piece_id=P5NU32M3V7AMYFZVJ6XZY5K2YMQWF4EULQGX6RRYWG6SWDWWAFMQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618840449 (20h17m54.094778914s) [noise=true , address=104.36.82.14:4227 , piece_id=O2FXJBOV3CJCKLZHRPBTLXX2QRT2OWJQ5HPNHZI5IVIFDVUUICVQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.619028879 (20h17m54.094967344s) [noise=true , address=104.36.82.9:4028 , piece_id=XCRNGUPBFYCZOYT23L6NHJ6WQR55TQMTWZF4SNKSRLQGC5TYO3LA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.619215837 (20h17m54.095154302s) [noise=true , address=104.36.82.53:4126 , piece_id=GZ5Z7AGMYVGYVUJAWFAIS6SQLRHFB772RLQY736QUGOXA5OWQ34Q, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.619408406 (20h17m54.095346871s) [noise=true , address=104.36.82.55:4219 , piece_id=IURHWONYUYYJWIL5BDF6HLCEPA76O6QAZAJ22NCI6LQYIGO6K4IA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:43.299790904 (20h17m55.775729369s) [upload=19386]: Next returned nil segment Jul 29 13:03:43.299806024 (20h17m55.775744479s) [upload=19386]: Waiting for error group managing segments... Jul 29 13:03:43.847040928 (20h17m56.322979393s) [noise=true , address=104.36.82.9:4028 , piece_id=XCRNGUPBFYCZOYT23L6NHJ6WQR55TQMTWZF4SNKSRLQGC5TYO3LA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:43.879495404 (20h17m56.355433869s) [noise=true , address=104.36.82.61:4391 , piece_id=TI5YDGDSTWKDX5XAWL36F3TJDQNHKNTREZVTUGIS2EVZYP6DWSDA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:46.350727192 (20h17m58.826665657s) [noise=true , address=104.36.82.5:4866 , piece_id=EFRLC4LA232NZZJU3GDSP2EEBZO3W2LGJRD3L2A6SPSRBTLUTIFA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:51.702944756 (20h18m4.178883221s) [noise=true , address=104.36.82.42:4802 , piece_id=6HORHYYKIKDX4IWWKUQPVGZQZ2YYIGC47OCZHRVBC4BZFRJX5TQQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:55.250827413 (20h18m7.726765878s) [noise=true , address=104.36.82.55:4219 , piece_id=IURHWONYUYYJWIL5BDF6HLCEPA76O6QAZAJ22NCI6LQYIGO6K4IA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:57.858469692 (20h18m10.334408157s) [noise=true , address=104.36.82.53:4126 , piece_id=GZ5Z7AGMYVGYVUJAWFAIS6SQLRHFB772RLQY736QUGOXA5OWQ34Q, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:59.265385056 (20h18m11.741323511s) [noise=true , address=104.36.82.10:4088 , piece_id=6IV2Z7FXDNEOBJMNPRG7WSPZWZTJ5CGDF6JHBSF7WCIIL2QQWFUA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:59.973580263 (20h18m12.449518729s) [noise=true , address=104.36.82.11:4119 , piece_id=6KWLABSUMX6YVKD6LGKR6ONLQKFFTSTKFHHRVFIN7U4KDALBNCBQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:16.002200978 (20h18m28.478139453s) [noise=true , address=104.36.82.52:4092 , piece_id=BMJCAQYXJCUVJTEXH52UXYLD4MHWLDGQTJIPDOFRF4LPLTT36ESA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:30.906580908 (20h18m43.382519373s) [noise=true , address=104.36.82.20:4448 , piece_id=CSHYRDGTS4ROFDTKETIZ36TWJE3OEMAKLB4RGFAU557G725EANXQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:31.164952501 (20h18m43.640890966s) [noise=true , address=104.36.82.14:4227 , piece_id=O2FXJBOV3CJCKLZHRPBTLXX2QRT2OWJQ5HPNHZI5IVIFDVUUICVQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:33.000876204 (20h18m45.476814679s) [noise=true , address=104.36.82.56:4242 , piece_id=BHKC5MH4G65BK2YLJJANIYZ4MLAUTMBG32XSRISARHCO26AZUXRA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:34.725342452 (20h18m47.201280907s) [noise=true , address=104.36.82.47:4545 , piece_id=PZWVIOWLBZ72IOFRJHPAD4OZZPA2G4XYCDLGYEY7THSOR7XDHMUQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:40.451024021 (20h18m52.926962486s) [noise=true , address=104.36.82.46:4500 , piece_id=VCYKKDSLEKTTL3KO2FJ4VA5JGIVPA4GFBBHH65HPFZTP4REUTHYQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:12B7ox4ehqxJ2mwqeXj8C9N9GuSU5etNC23iT1pJnTgMLGBAGQK, address:104.36.82.46:4500): protocol: write tcp 188.166.225.170:46700->104.36.82.46:4500: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451060500 (20h18m52.926998965s) [noise=true , address=104.36.82.51:4075 , piece_id=P5NU32M3V7AMYFZVJ6XZY5K2YMQWF4EULQGX6RRYWG6SWDWWAFMQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK, address:104.36.82.51:4075): protocol: write tcp 188.166.225.170:41162->104.36.82.51:4075: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451086179 (20h18m52.927024644s) [noise=true , address=104.36.82.54:4175 , piece_id=5PAE3BM5AFSLZ5TVUQC3QK3RVWCVGFDLOK452BDPWQ7ARTL7EYWQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:12ou7jwFyJP3mNoALddniUTDT9duVyQKZjeq4xHCyx6sRWpwEPJ, address:104.36.82.54:4175): protocol: rpc: dial tcp 104.36.82.54:4175: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451120024 (20h18m52.927058489s) [noise=true , address=104.36.82.13:4214 , piece_id=SLX3YJCEWCM3553VZHILH337FKSPPFNKVFTPXO4KIBOS6IXCLXHA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:1nJ8uqwxf6x4hX52AjAdCbMKi1BKGGbMvrQbiazJdLpn6j69ay, address:104.36.82.13:4214): protocol: rpc: dial tcp 104.36.82.13:4214: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451147116 (20h18m52.927085581s) [noise=true , address=104.36.82.15:4263 , piece_id=3GCLYN4UBCLM5BPLXDQT5SSWPJBTKIOZM4UIUNO3OZIXW6OANUBQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:1f97yBEvcrYHqMq7iovVoju77jzhUYgoQCsUGULeTagugFXrCu, address:104.36.82.15:4263): protocol: write tcp 188.166.225.170:39380->104.36.82.15:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451167805 (20h18m52.927106271s) [seg_pos={0 0}, upload=19386]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 29 13:04:40.696628655 (20h18m53.17256714s) [upload=19386]: Done uploading segments... Jul 29 13:04:40.696658391 (20h18m53.172596856s) [upload=19386]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.931757724s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [......X............X...............X........X...X..XX.......] duration: 5.866724872s FAIL (2025-07-29 13:09:19 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 14 but got 7) Duration: 5.866756151s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [.........XX............X...........X........................] duration: 1.998899927s FAIL (2025-07-29 13:08:49 UTC) orbiter warning: unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K Duration: 1.998923111s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..................................................X.X.......] duration: 5.995657012s FAIL (2025-07-29 13:08:29 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ding segments... Jul 29 13:03:41.091320246 (20h17m53.567258701s) [upload=19386]: Got next segment. Inline: false Jul 29 13:03:41.091349191 (20h17m53.567287656s) [upload=19386]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 29 13:03:41.091354331 (20h17m53.567292796s) [upload=19386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 29 13:03:41.616444845 (20h17m54.09238331s) [seg_pos={0 0}, upload=19386]: Begin upload segment... Jul 29 13:03:41.616487748 (20h17m54.092426213s) [seg_pos={0 0}, upload=19386]: Done begin upload segment. Jul 29 13:03:41.616507376 (20h17m54.092445841s) [noise=true , address=104.36.82.15:4263 , piece_id=3GCLYN4UBCLM5BPLXDQT5SSWPJBTKIOZM4UIUNO3OZIXW6OANUBQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.616564315 (20h17m54.09250278s) [noise=true , address=104.36.82.11:4119 , piece_id=6KWLABSUMX6YVKD6LGKR6ONLQKFFTSTKFHHRVFIN7U4KDALBNCBQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.616866433 (20h17m54.092804888s) [noise=true , address=104.36.82.54:4175 , piece_id=5PAE3BM5AFSLZ5TVUQC3QK3RVWCVGFDLOK452BDPWQ7ARTL7EYWQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617072557 (20h17m54.093011022s) [noise=true , address=104.36.82.56:4242 , piece_id=BHKC5MH4G65BK2YLJJANIYZ4MLAUTMBG32XSRISARHCO26AZUXRA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617110629 (20h17m54.093049084s) [noise=true , address=104.36.82.5:4866 , piece_id=EFRLC4LA232NZZJU3GDSP2EEBZO3W2LGJRD3L2A6SPSRBTLUTIFA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617355037 (20h17m54.093293492s) [noise=true , address=104.36.82.61:4391 , piece_id=TI5YDGDSTWKDX5XAWL36F3TJDQNHKNTREZVTUGIS2EVZYP6DWSDA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617481549 (20h17m54.093420024s) [noise=true , address=104.36.82.20:4448 , piece_id=CSHYRDGTS4ROFDTKETIZ36TWJE3OEMAKLB4RGFAU557G725EANXQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617583044 (20h17m54.093521509s) [noise=true , address=104.36.82.47:4545 , piece_id=PZWVIOWLBZ72IOFRJHPAD4OZZPA2G4XYCDLGYEY7THSOR7XDHMUQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617828073 (20h17m54.093766528s) [noise=true , address=104.36.82.42:4802 , piece_id=6HORHYYKIKDX4IWWKUQPVGZQZ2YYIGC47OCZHRVBC4BZFRJX5TQQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617987597 (20h17m54.093926062s) [noise=true , address=104.36.82.13:4214 , piece_id=SLX3YJCEWCM3553VZHILH337FKSPPFNKVFTPXO4KIBOS6IXCLXHA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618055948 (20h17m54.093994403s) [noise=true , address=104.36.82.46:4500 , piece_id=VCYKKDSLEKTTL3KO2FJ4VA5JGIVPA4GFBBHH65HPFZTP4REUTHYQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618285828 (20h17m54.094224293s) [noise=true , address=104.36.82.52:4092 , piece_id=BMJCAQYXJCUVJTEXH52UXYLD4MHWLDGQTJIPDOFRF4LPLTT36ESA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618485871 (20h17m54.094424336s) [noise=true , address=104.36.82.10:4088 , piece_id=6IV2Z7FXDNEOBJMNPRG7WSPZWZTJ5CGDF6JHBSF7WCIIL2QQWFUA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618663881 (20h17m54.094602346s) [noise=true , address=104.36.82.51:4075 , piece_id=P5NU32M3V7AMYFZVJ6XZY5K2YMQWF4EULQGX6RRYWG6SWDWWAFMQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618840449 (20h17m54.094778914s) [noise=true , address=104.36.82.14:4227 , piece_id=O2FXJBOV3CJCKLZHRPBTLXX2QRT2OWJQ5HPNHZI5IVIFDVUUICVQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.619028879 (20h17m54.094967344s) [noise=true , address=104.36.82.9:4028 , piece_id=XCRNGUPBFYCZOYT23L6NHJ6WQR55TQMTWZF4SNKSRLQGC5TYO3LA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.619215837 (20h17m54.095154302s) [noise=true , address=104.36.82.53:4126 , piece_id=GZ5Z7AGMYVGYVUJAWFAIS6SQLRHFB772RLQY736QUGOXA5OWQ34Q, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.619408406 (20h17m54.095346871s) [noise=true , address=104.36.82.55:4219 , piece_id=IURHWONYUYYJWIL5BDF6HLCEPA76O6QAZAJ22NCI6LQYIGO6K4IA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:43.299790904 (20h17m55.775729369s) [upload=19386]: Next returned nil segment Jul 29 13:03:43.299806024 (20h17m55.775744479s) [upload=19386]: Waiting for error group managing segments... Jul 29 13:03:43.847040928 (20h17m56.322979393s) [noise=true , address=104.36.82.9:4028 , piece_id=XCRNGUPBFYCZOYT23L6NHJ6WQR55TQMTWZF4SNKSRLQGC5TYO3LA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:43.879495404 (20h17m56.355433869s) [noise=true , address=104.36.82.61:4391 , piece_id=TI5YDGDSTWKDX5XAWL36F3TJDQNHKNTREZVTUGIS2EVZYP6DWSDA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:46.350727192 (20h17m58.826665657s) [noise=true , address=104.36.82.5:4866 , piece_id=EFRLC4LA232NZZJU3GDSP2EEBZO3W2LGJRD3L2A6SPSRBTLUTIFA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:51.702944756 (20h18m4.178883221s) [noise=true , address=104.36.82.42:4802 , piece_id=6HORHYYKIKDX4IWWKUQPVGZQZ2YYIGC47OCZHRVBC4BZFRJX5TQQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:55.250827413 (20h18m7.726765878s) [noise=true , address=104.36.82.55:4219 , piece_id=IURHWONYUYYJWIL5BDF6HLCEPA76O6QAZAJ22NCI6LQYIGO6K4IA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:57.858469692 (20h18m10.334408157s) [noise=true , address=104.36.82.53:4126 , piece_id=GZ5Z7AGMYVGYVUJAWFAIS6SQLRHFB772RLQY736QUGOXA5OWQ34Q, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:59.265385056 (20h18m11.741323511s) [noise=true , address=104.36.82.10:4088 , piece_id=6IV2Z7FXDNEOBJMNPRG7WSPZWZTJ5CGDF6JHBSF7WCIIL2QQWFUA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:59.973580263 (20h18m12.449518729s) [noise=true , address=104.36.82.11:4119 , piece_id=6KWLABSUMX6YVKD6LGKR6ONLQKFFTSTKFHHRVFIN7U4KDALBNCBQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:16.002200978 (20h18m28.478139453s) [noise=true , address=104.36.82.52:4092 , piece_id=BMJCAQYXJCUVJTEXH52UXYLD4MHWLDGQTJIPDOFRF4LPLTT36ESA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:30.906580908 (20h18m43.382519373s) [noise=true , address=104.36.82.20:4448 , piece_id=CSHYRDGTS4ROFDTKETIZ36TWJE3OEMAKLB4RGFAU557G725EANXQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:31.164952501 (20h18m43.640890966s) [noise=true , address=104.36.82.14:4227 , piece_id=O2FXJBOV3CJCKLZHRPBTLXX2QRT2OWJQ5HPNHZI5IVIFDVUUICVQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:33.000876204 (20h18m45.476814679s) [noise=true , address=104.36.82.56:4242 , piece_id=BHKC5MH4G65BK2YLJJANIYZ4MLAUTMBG32XSRISARHCO26AZUXRA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:34.725342452 (20h18m47.201280907s) [noise=true , address=104.36.82.47:4545 , piece_id=PZWVIOWLBZ72IOFRJHPAD4OZZPA2G4XYCDLGYEY7THSOR7XDHMUQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:40.451024021 (20h18m52.926962486s) [noise=true , address=104.36.82.46:4500 , piece_id=VCYKKDSLEKTTL3KO2FJ4VA5JGIVPA4GFBBHH65HPFZTP4REUTHYQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:12B7ox4ehqxJ2mwqeXj8C9N9GuSU5etNC23iT1pJnTgMLGBAGQK, address:104.36.82.46:4500): protocol: write tcp 188.166.225.170:46700->104.36.82.46:4500: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451060500 (20h18m52.926998965s) [noise=true , address=104.36.82.51:4075 , piece_id=P5NU32M3V7AMYFZVJ6XZY5K2YMQWF4EULQGX6RRYWG6SWDWWAFMQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK, address:104.36.82.51:4075): protocol: write tcp 188.166.225.170:41162->104.36.82.51:4075: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451086179 (20h18m52.927024644s) [noise=true , address=104.36.82.54:4175 , piece_id=5PAE3BM5AFSLZ5TVUQC3QK3RVWCVGFDLOK452BDPWQ7ARTL7EYWQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:12ou7jwFyJP3mNoALddniUTDT9duVyQKZjeq4xHCyx6sRWpwEPJ, address:104.36.82.54:4175): protocol: rpc: dial tcp 104.36.82.54:4175: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451120024 (20h18m52.927058489s) [noise=true , address=104.36.82.13:4214 , piece_id=SLX3YJCEWCM3553VZHILH337FKSPPFNKVFTPXO4KIBOS6IXCLXHA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:1nJ8uqwxf6x4hX52AjAdCbMKi1BKGGbMvrQbiazJdLpn6j69ay, address:104.36.82.13:4214): protocol: rpc: dial tcp 104.36.82.13:4214: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451147116 (20h18m52.927085581s) [noise=true , address=104.36.82.15:4263 , piece_id=3GCLYN4UBCLM5BPLXDQT5SSWPJBTKIOZM4UIUNO3OZIXW6OANUBQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:1f97yBEvcrYHqMq7iovVoju77jzhUYgoQCsUGULeTagugFXrCu, address:104.36.82.15:4263): protocol: write tcp 188.166.225.170:39380->104.36.82.15:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451167805 (20h18m52.927106271s) [seg_pos={0 0}, upload=19386]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 29 13:04:40.696628655 (20h18m53.17256714s) [upload=19386]: Done uploading segments... Jul 29 13:04:40.696658391 (20h18m53.172596856s) [upload=19386]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.99568178s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [.......X............X...............X........X...X..XX......] duration: 6.099800856s FAIL (2025-07-29 13:08:05 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 14 but got 7) Duration: 6.099817698s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...................................................X.X......] duration: 6.147260689s FAIL (2025-07-29 13:07:41 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ding segments... Jul 29 13:03:41.091320246 (20h17m53.567258701s) [upload=19386]: Got next segment. Inline: false Jul 29 13:03:41.091349191 (20h17m53.567287656s) [upload=19386]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 29 13:03:41.091354331 (20h17m53.567292796s) [upload=19386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 29 13:03:41.616444845 (20h17m54.09238331s) [seg_pos={0 0}, upload=19386]: Begin upload segment... Jul 29 13:03:41.616487748 (20h17m54.092426213s) [seg_pos={0 0}, upload=19386]: Done begin upload segment. Jul 29 13:03:41.616507376 (20h17m54.092445841s) [noise=true , address=104.36.82.15:4263 , piece_id=3GCLYN4UBCLM5BPLXDQT5SSWPJBTKIOZM4UIUNO3OZIXW6OANUBQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.616564315 (20h17m54.09250278s) [noise=true , address=104.36.82.11:4119 , piece_id=6KWLABSUMX6YVKD6LGKR6ONLQKFFTSTKFHHRVFIN7U4KDALBNCBQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.616866433 (20h17m54.092804888s) [noise=true , address=104.36.82.54:4175 , piece_id=5PAE3BM5AFSLZ5TVUQC3QK3RVWCVGFDLOK452BDPWQ7ARTL7EYWQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617072557 (20h17m54.093011022s) [noise=true , address=104.36.82.56:4242 , piece_id=BHKC5MH4G65BK2YLJJANIYZ4MLAUTMBG32XSRISARHCO26AZUXRA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617110629 (20h17m54.093049084s) [noise=true , address=104.36.82.5:4866 , piece_id=EFRLC4LA232NZZJU3GDSP2EEBZO3W2LGJRD3L2A6SPSRBTLUTIFA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617355037 (20h17m54.093293492s) [noise=true , address=104.36.82.61:4391 , piece_id=TI5YDGDSTWKDX5XAWL36F3TJDQNHKNTREZVTUGIS2EVZYP6DWSDA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617481549 (20h17m54.093420024s) [noise=true , address=104.36.82.20:4448 , piece_id=CSHYRDGTS4ROFDTKETIZ36TWJE3OEMAKLB4RGFAU557G725EANXQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617583044 (20h17m54.093521509s) [noise=true , address=104.36.82.47:4545 , piece_id=PZWVIOWLBZ72IOFRJHPAD4OZZPA2G4XYCDLGYEY7THSOR7XDHMUQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617828073 (20h17m54.093766528s) [noise=true , address=104.36.82.42:4802 , piece_id=6HORHYYKIKDX4IWWKUQPVGZQZ2YYIGC47OCZHRVBC4BZFRJX5TQQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617987597 (20h17m54.093926062s) [noise=true , address=104.36.82.13:4214 , piece_id=SLX3YJCEWCM3553VZHILH337FKSPPFNKVFTPXO4KIBOS6IXCLXHA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618055948 (20h17m54.093994403s) [noise=true , address=104.36.82.46:4500 , piece_id=VCYKKDSLEKTTL3KO2FJ4VA5JGIVPA4GFBBHH65HPFZTP4REUTHYQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618285828 (20h17m54.094224293s) [noise=true , address=104.36.82.52:4092 , piece_id=BMJCAQYXJCUVJTEXH52UXYLD4MHWLDGQTJIPDOFRF4LPLTT36ESA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618485871 (20h17m54.094424336s) [noise=true , address=104.36.82.10:4088 , piece_id=6IV2Z7FXDNEOBJMNPRG7WSPZWZTJ5CGDF6JHBSF7WCIIL2QQWFUA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618663881 (20h17m54.094602346s) [noise=true , address=104.36.82.51:4075 , piece_id=P5NU32M3V7AMYFZVJ6XZY5K2YMQWF4EULQGX6RRYWG6SWDWWAFMQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618840449 (20h17m54.094778914s) [noise=true , address=104.36.82.14:4227 , piece_id=O2FXJBOV3CJCKLZHRPBTLXX2QRT2OWJQ5HPNHZI5IVIFDVUUICVQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.619028879 (20h17m54.094967344s) [noise=true , address=104.36.82.9:4028 , piece_id=XCRNGUPBFYCZOYT23L6NHJ6WQR55TQMTWZF4SNKSRLQGC5TYO3LA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.619215837 (20h17m54.095154302s) [noise=true , address=104.36.82.53:4126 , piece_id=GZ5Z7AGMYVGYVUJAWFAIS6SQLRHFB772RLQY736QUGOXA5OWQ34Q, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.619408406 (20h17m54.095346871s) [noise=true , address=104.36.82.55:4219 , piece_id=IURHWONYUYYJWIL5BDF6HLCEPA76O6QAZAJ22NCI6LQYIGO6K4IA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:43.299790904 (20h17m55.775729369s) [upload=19386]: Next returned nil segment Jul 29 13:03:43.299806024 (20h17m55.775744479s) [upload=19386]: Waiting for error group managing segments... Jul 29 13:03:43.847040928 (20h17m56.322979393s) [noise=true , address=104.36.82.9:4028 , piece_id=XCRNGUPBFYCZOYT23L6NHJ6WQR55TQMTWZF4SNKSRLQGC5TYO3LA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:43.879495404 (20h17m56.355433869s) [noise=true , address=104.36.82.61:4391 , piece_id=TI5YDGDSTWKDX5XAWL36F3TJDQNHKNTREZVTUGIS2EVZYP6DWSDA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:46.350727192 (20h17m58.826665657s) [noise=true , address=104.36.82.5:4866 , piece_id=EFRLC4LA232NZZJU3GDSP2EEBZO3W2LGJRD3L2A6SPSRBTLUTIFA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:51.702944756 (20h18m4.178883221s) [noise=true , address=104.36.82.42:4802 , piece_id=6HORHYYKIKDX4IWWKUQPVGZQZ2YYIGC47OCZHRVBC4BZFRJX5TQQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:55.250827413 (20h18m7.726765878s) [noise=true , address=104.36.82.55:4219 , piece_id=IURHWONYUYYJWIL5BDF6HLCEPA76O6QAZAJ22NCI6LQYIGO6K4IA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:57.858469692 (20h18m10.334408157s) [noise=true , address=104.36.82.53:4126 , piece_id=GZ5Z7AGMYVGYVUJAWFAIS6SQLRHFB772RLQY736QUGOXA5OWQ34Q, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:59.265385056 (20h18m11.741323511s) [noise=true , address=104.36.82.10:4088 , piece_id=6IV2Z7FXDNEOBJMNPRG7WSPZWZTJ5CGDF6JHBSF7WCIIL2QQWFUA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:59.973580263 (20h18m12.449518729s) [noise=true , address=104.36.82.11:4119 , piece_id=6KWLABSUMX6YVKD6LGKR6ONLQKFFTSTKFHHRVFIN7U4KDALBNCBQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:16.002200978 (20h18m28.478139453s) [noise=true , address=104.36.82.52:4092 , piece_id=BMJCAQYXJCUVJTEXH52UXYLD4MHWLDGQTJIPDOFRF4LPLTT36ESA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:30.906580908 (20h18m43.382519373s) [noise=true , address=104.36.82.20:4448 , piece_id=CSHYRDGTS4ROFDTKETIZ36TWJE3OEMAKLB4RGFAU557G725EANXQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:31.164952501 (20h18m43.640890966s) [noise=true , address=104.36.82.14:4227 , piece_id=O2FXJBOV3CJCKLZHRPBTLXX2QRT2OWJQ5HPNHZI5IVIFDVUUICVQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:33.000876204 (20h18m45.476814679s) [noise=true , address=104.36.82.56:4242 , piece_id=BHKC5MH4G65BK2YLJJANIYZ4MLAUTMBG32XSRISARHCO26AZUXRA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:34.725342452 (20h18m47.201280907s) [noise=true , address=104.36.82.47:4545 , piece_id=PZWVIOWLBZ72IOFRJHPAD4OZZPA2G4XYCDLGYEY7THSOR7XDHMUQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:40.451024021 (20h18m52.926962486s) [noise=true , address=104.36.82.46:4500 , piece_id=VCYKKDSLEKTTL3KO2FJ4VA5JGIVPA4GFBBHH65HPFZTP4REUTHYQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:12B7ox4ehqxJ2mwqeXj8C9N9GuSU5etNC23iT1pJnTgMLGBAGQK, address:104.36.82.46:4500): protocol: write tcp 188.166.225.170:46700->104.36.82.46:4500: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451060500 (20h18m52.926998965s) [noise=true , address=104.36.82.51:4075 , piece_id=P5NU32M3V7AMYFZVJ6XZY5K2YMQWF4EULQGX6RRYWG6SWDWWAFMQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK, address:104.36.82.51:4075): protocol: write tcp 188.166.225.170:41162->104.36.82.51:4075: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451086179 (20h18m52.927024644s) [noise=true , address=104.36.82.54:4175 , piece_id=5PAE3BM5AFSLZ5TVUQC3QK3RVWCVGFDLOK452BDPWQ7ARTL7EYWQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:12ou7jwFyJP3mNoALddniUTDT9duVyQKZjeq4xHCyx6sRWpwEPJ, address:104.36.82.54:4175): protocol: rpc: dial tcp 104.36.82.54:4175: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451120024 (20h18m52.927058489s) [noise=true , address=104.36.82.13:4214 , piece_id=SLX3YJCEWCM3553VZHILH337FKSPPFNKVFTPXO4KIBOS6IXCLXHA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:1nJ8uqwxf6x4hX52AjAdCbMKi1BKGGbMvrQbiazJdLpn6j69ay, address:104.36.82.13:4214): protocol: rpc: dial tcp 104.36.82.13:4214: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451147116 (20h18m52.927085581s) [noise=true , address=104.36.82.15:4263 , piece_id=3GCLYN4UBCLM5BPLXDQT5SSWPJBTKIOZM4UIUNO3OZIXW6OANUBQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:1f97yBEvcrYHqMq7iovVoju77jzhUYgoQCsUGULeTagugFXrCu, address:104.36.82.15:4263): protocol: write tcp 188.166.225.170:39380->104.36.82.15:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451167805 (20h18m52.927106271s) [seg_pos={0 0}, upload=19386]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 29 13:04:40.696628655 (20h18m53.17256714s) [upload=19386]: Done uploading segments... Jul 29 13:04:40.696658391 (20h18m53.172596856s) [upload=19386]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.147281658s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 4 out of 60 is failed (threshold=3) [..........XX............X...........X.......................] duration: 1.938172024s FAIL (2025-07-29 13:07:36 UTC) orbiter warning: unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K Duration: 1.93820123s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)