Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 1.873687021s FAIL (2025-07-29 13:50:49 UTC) orbiter warning: uplink: too many requests Duration: 1.873711569s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................X.............X................] duration: 5.764317135s FAIL (2025-07-29 13:50:39 UTC) orbiter warning: digest mismatch: after ee3c39c295e32d2453f46eedbd5a023ecb901c93b585366a6221da2ecebc35b3 != before f1d4b9d3eca6a9d63854d84cb67ac0fa26d296bc93761368409da19470c1a42f Duration: 5.764357702s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................X.............X...............] duration: 5.877396812s FAIL (2025-07-29 13:50:18 UTC) orbiter warning: digest mismatch: after ee3c39c295e32d2453f46eedbd5a023ecb901c93b585366a6221da2ecebc35b3 != before f1d4b9d3eca6a9d63854d84cb67ac0fa26d296bc93761368409da19470c1a42f Duration: 5.877424496s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 5.740265075s FAIL (2025-07-29 13:50:14 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ding segments... Jul 29 13:03:41.091320246 (20h17m53.567258701s) [upload=19386]: Got next segment. Inline: false Jul 29 13:03:41.091349191 (20h17m53.567287656s) [upload=19386]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 29 13:03:41.091354331 (20h17m53.567292796s) [upload=19386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 29 13:03:41.616444845 (20h17m54.09238331s) [seg_pos={0 0}, upload=19386]: Begin upload segment... Jul 29 13:03:41.616487748 (20h17m54.092426213s) [seg_pos={0 0}, upload=19386]: Done begin upload segment. Jul 29 13:03:41.616507376 (20h17m54.092445841s) [noise=true , address=104.36.82.15:4263 , piece_id=3GCLYN4UBCLM5BPLXDQT5SSWPJBTKIOZM4UIUNO3OZIXW6OANUBQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.616564315 (20h17m54.09250278s) [noise=true , address=104.36.82.11:4119 , piece_id=6KWLABSUMX6YVKD6LGKR6ONLQKFFTSTKFHHRVFIN7U4KDALBNCBQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.616866433 (20h17m54.092804888s) [noise=true , address=104.36.82.54:4175 , piece_id=5PAE3BM5AFSLZ5TVUQC3QK3RVWCVGFDLOK452BDPWQ7ARTL7EYWQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617072557 (20h17m54.093011022s) [noise=true , address=104.36.82.56:4242 , piece_id=BHKC5MH4G65BK2YLJJANIYZ4MLAUTMBG32XSRISARHCO26AZUXRA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617110629 (20h17m54.093049084s) [noise=true , address=104.36.82.5:4866 , piece_id=EFRLC4LA232NZZJU3GDSP2EEBZO3W2LGJRD3L2A6SPSRBTLUTIFA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617355037 (20h17m54.093293492s) [noise=true , address=104.36.82.61:4391 , piece_id=TI5YDGDSTWKDX5XAWL36F3TJDQNHKNTREZVTUGIS2EVZYP6DWSDA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617481549 (20h17m54.093420024s) [noise=true , address=104.36.82.20:4448 , piece_id=CSHYRDGTS4ROFDTKETIZ36TWJE3OEMAKLB4RGFAU557G725EANXQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617583044 (20h17m54.093521509s) [noise=true , address=104.36.82.47:4545 , piece_id=PZWVIOWLBZ72IOFRJHPAD4OZZPA2G4XYCDLGYEY7THSOR7XDHMUQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617828073 (20h17m54.093766528s) [noise=true , address=104.36.82.42:4802 , piece_id=6HORHYYKIKDX4IWWKUQPVGZQZ2YYIGC47OCZHRVBC4BZFRJX5TQQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617987597 (20h17m54.093926062s) [noise=true , address=104.36.82.13:4214 , piece_id=SLX3YJCEWCM3553VZHILH337FKSPPFNKVFTPXO4KIBOS6IXCLXHA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618055948 (20h17m54.093994403s) [noise=true , address=104.36.82.46:4500 , piece_id=VCYKKDSLEKTTL3KO2FJ4VA5JGIVPA4GFBBHH65HPFZTP4REUTHYQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618285828 (20h17m54.094224293s) [noise=true , address=104.36.82.52:4092 , piece_id=BMJCAQYXJCUVJTEXH52UXYLD4MHWLDGQTJIPDOFRF4LPLTT36ESA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618485871 (20h17m54.094424336s) [noise=true , address=104.36.82.10:4088 , piece_id=6IV2Z7FXDNEOBJMNPRG7WSPZWZTJ5CGDF6JHBSF7WCIIL2QQWFUA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618663881 (20h17m54.094602346s) [noise=true , address=104.36.82.51:4075 , piece_id=P5NU32M3V7AMYFZVJ6XZY5K2YMQWF4EULQGX6RRYWG6SWDWWAFMQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618840449 (20h17m54.094778914s) [noise=true , address=104.36.82.14:4227 , piece_id=O2FXJBOV3CJCKLZHRPBTLXX2QRT2OWJQ5HPNHZI5IVIFDVUUICVQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.619028879 (20h17m54.094967344s) [noise=true , address=104.36.82.9:4028 , piece_id=XCRNGUPBFYCZOYT23L6NHJ6WQR55TQMTWZF4SNKSRLQGC5TYO3LA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.619215837 (20h17m54.095154302s) [noise=true , address=104.36.82.53:4126 , piece_id=GZ5Z7AGMYVGYVUJAWFAIS6SQLRHFB772RLQY736QUGOXA5OWQ34Q, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.619408406 (20h17m54.095346871s) [noise=true , address=104.36.82.55:4219 , piece_id=IURHWONYUYYJWIL5BDF6HLCEPA76O6QAZAJ22NCI6LQYIGO6K4IA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:43.299790904 (20h17m55.775729369s) [upload=19386]: Next returned nil segment Jul 29 13:03:43.299806024 (20h17m55.775744479s) [upload=19386]: Waiting for error group managing segments... Jul 29 13:03:43.847040928 (20h17m56.322979393s) [noise=true , address=104.36.82.9:4028 , piece_id=XCRNGUPBFYCZOYT23L6NHJ6WQR55TQMTWZF4SNKSRLQGC5TYO3LA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:43.879495404 (20h17m56.355433869s) [noise=true , address=104.36.82.61:4391 , piece_id=TI5YDGDSTWKDX5XAWL36F3TJDQNHKNTREZVTUGIS2EVZYP6DWSDA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:46.350727192 (20h17m58.826665657s) [noise=true , address=104.36.82.5:4866 , piece_id=EFRLC4LA232NZZJU3GDSP2EEBZO3W2LGJRD3L2A6SPSRBTLUTIFA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:51.702944756 (20h18m4.178883221s) [noise=true , address=104.36.82.42:4802 , piece_id=6HORHYYKIKDX4IWWKUQPVGZQZ2YYIGC47OCZHRVBC4BZFRJX5TQQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:55.250827413 (20h18m7.726765878s) [noise=true , address=104.36.82.55:4219 , piece_id=IURHWONYUYYJWIL5BDF6HLCEPA76O6QAZAJ22NCI6LQYIGO6K4IA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:57.858469692 (20h18m10.334408157s) [noise=true , address=104.36.82.53:4126 , piece_id=GZ5Z7AGMYVGYVUJAWFAIS6SQLRHFB772RLQY736QUGOXA5OWQ34Q, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:59.265385056 (20h18m11.741323511s) [noise=true , address=104.36.82.10:4088 , piece_id=6IV2Z7FXDNEOBJMNPRG7WSPZWZTJ5CGDF6JHBSF7WCIIL2QQWFUA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:59.973580263 (20h18m12.449518729s) [noise=true , address=104.36.82.11:4119 , piece_id=6KWLABSUMX6YVKD6LGKR6ONLQKFFTSTKFHHRVFIN7U4KDALBNCBQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:16.002200978 (20h18m28.478139453s) [noise=true , address=104.36.82.52:4092 , piece_id=BMJCAQYXJCUVJTEXH52UXYLD4MHWLDGQTJIPDOFRF4LPLTT36ESA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:30.906580908 (20h18m43.382519373s) [noise=true , address=104.36.82.20:4448 , piece_id=CSHYRDGTS4ROFDTKETIZ36TWJE3OEMAKLB4RGFAU557G725EANXQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:31.164952501 (20h18m43.640890966s) [noise=true , address=104.36.82.14:4227 , piece_id=O2FXJBOV3CJCKLZHRPBTLXX2QRT2OWJQ5HPNHZI5IVIFDVUUICVQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:33.000876204 (20h18m45.476814679s) [noise=true , address=104.36.82.56:4242 , piece_id=BHKC5MH4G65BK2YLJJANIYZ4MLAUTMBG32XSRISARHCO26AZUXRA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:34.725342452 (20h18m47.201280907s) [noise=true , address=104.36.82.47:4545 , piece_id=PZWVIOWLBZ72IOFRJHPAD4OZZPA2G4XYCDLGYEY7THSOR7XDHMUQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:40.451024021 (20h18m52.926962486s) [noise=true , address=104.36.82.46:4500 , piece_id=VCYKKDSLEKTTL3KO2FJ4VA5JGIVPA4GFBBHH65HPFZTP4REUTHYQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:12B7ox4ehqxJ2mwqeXj8C9N9GuSU5etNC23iT1pJnTgMLGBAGQK, address:104.36.82.46:4500): protocol: write tcp 188.166.225.170:46700->104.36.82.46:4500: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451060500 (20h18m52.926998965s) [noise=true , address=104.36.82.51:4075 , piece_id=P5NU32M3V7AMYFZVJ6XZY5K2YMQWF4EULQGX6RRYWG6SWDWWAFMQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK, address:104.36.82.51:4075): protocol: write tcp 188.166.225.170:41162->104.36.82.51:4075: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451086179 (20h18m52.927024644s) [noise=true , address=104.36.82.54:4175 , piece_id=5PAE3BM5AFSLZ5TVUQC3QK3RVWCVGFDLOK452BDPWQ7ARTL7EYWQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:12ou7jwFyJP3mNoALddniUTDT9duVyQKZjeq4xHCyx6sRWpwEPJ, address:104.36.82.54:4175): protocol: rpc: dial tcp 104.36.82.54:4175: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451120024 (20h18m52.927058489s) [noise=true , address=104.36.82.13:4214 , piece_id=SLX3YJCEWCM3553VZHILH337FKSPPFNKVFTPXO4KIBOS6IXCLXHA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:1nJ8uqwxf6x4hX52AjAdCbMKi1BKGGbMvrQbiazJdLpn6j69ay, address:104.36.82.13:4214): protocol: rpc: dial tcp 104.36.82.13:4214: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451147116 (20h18m52.927085581s) [noise=true , address=104.36.82.15:4263 , piece_id=3GCLYN4UBCLM5BPLXDQT5SSWPJBTKIOZM4UIUNO3OZIXW6OANUBQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:1f97yBEvcrYHqMq7iovVoju77jzhUYgoQCsUGULeTagugFXrCu, address:104.36.82.15:4263): protocol: write tcp 188.166.225.170:39380->104.36.82.15:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451167805 (20h18m52.927106271s) [seg_pos={0 0}, upload=19386]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 29 13:04:40.696628655 (20h18m53.17256714s) [upload=19386]: Done uploading segments... Jul 29 13:04:40.696658391 (20h18m53.172596856s) [upload=19386]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.740306455s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 5.824538207s FAIL (2025-07-29 13:49:45 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ding segments... Jul 29 13:03:41.091320246 (20h17m53.567258701s) [upload=19386]: Got next segment. Inline: false Jul 29 13:03:41.091349191 (20h17m53.567287656s) [upload=19386]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 29 13:03:41.091354331 (20h17m53.567292796s) [upload=19386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 29 13:03:41.616444845 (20h17m54.09238331s) [seg_pos={0 0}, upload=19386]: Begin upload segment... Jul 29 13:03:41.616487748 (20h17m54.092426213s) [seg_pos={0 0}, upload=19386]: Done begin upload segment. Jul 29 13:03:41.616507376 (20h17m54.092445841s) [noise=true , address=104.36.82.15:4263 , piece_id=3GCLYN4UBCLM5BPLXDQT5SSWPJBTKIOZM4UIUNO3OZIXW6OANUBQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.616564315 (20h17m54.09250278s) [noise=true , address=104.36.82.11:4119 , piece_id=6KWLABSUMX6YVKD6LGKR6ONLQKFFTSTKFHHRVFIN7U4KDALBNCBQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.616866433 (20h17m54.092804888s) [noise=true , address=104.36.82.54:4175 , piece_id=5PAE3BM5AFSLZ5TVUQC3QK3RVWCVGFDLOK452BDPWQ7ARTL7EYWQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617072557 (20h17m54.093011022s) [noise=true , address=104.36.82.56:4242 , piece_id=BHKC5MH4G65BK2YLJJANIYZ4MLAUTMBG32XSRISARHCO26AZUXRA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617110629 (20h17m54.093049084s) [noise=true , address=104.36.82.5:4866 , piece_id=EFRLC4LA232NZZJU3GDSP2EEBZO3W2LGJRD3L2A6SPSRBTLUTIFA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617355037 (20h17m54.093293492s) [noise=true , address=104.36.82.61:4391 , piece_id=TI5YDGDSTWKDX5XAWL36F3TJDQNHKNTREZVTUGIS2EVZYP6DWSDA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617481549 (20h17m54.093420024s) [noise=true , address=104.36.82.20:4448 , piece_id=CSHYRDGTS4ROFDTKETIZ36TWJE3OEMAKLB4RGFAU557G725EANXQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617583044 (20h17m54.093521509s) [noise=true , address=104.36.82.47:4545 , piece_id=PZWVIOWLBZ72IOFRJHPAD4OZZPA2G4XYCDLGYEY7THSOR7XDHMUQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617828073 (20h17m54.093766528s) [noise=true , address=104.36.82.42:4802 , piece_id=6HORHYYKIKDX4IWWKUQPVGZQZ2YYIGC47OCZHRVBC4BZFRJX5TQQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617987597 (20h17m54.093926062s) [noise=true , address=104.36.82.13:4214 , piece_id=SLX3YJCEWCM3553VZHILH337FKSPPFNKVFTPXO4KIBOS6IXCLXHA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618055948 (20h17m54.093994403s) [noise=true , address=104.36.82.46:4500 , piece_id=VCYKKDSLEKTTL3KO2FJ4VA5JGIVPA4GFBBHH65HPFZTP4REUTHYQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618285828 (20h17m54.094224293s) [noise=true , address=104.36.82.52:4092 , piece_id=BMJCAQYXJCUVJTEXH52UXYLD4MHWLDGQTJIPDOFRF4LPLTT36ESA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618485871 (20h17m54.094424336s) [noise=true , address=104.36.82.10:4088 , piece_id=6IV2Z7FXDNEOBJMNPRG7WSPZWZTJ5CGDF6JHBSF7WCIIL2QQWFUA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618663881 (20h17m54.094602346s) [noise=true , address=104.36.82.51:4075 , piece_id=P5NU32M3V7AMYFZVJ6XZY5K2YMQWF4EULQGX6RRYWG6SWDWWAFMQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618840449 (20h17m54.094778914s) [noise=true , address=104.36.82.14:4227 , piece_id=O2FXJBOV3CJCKLZHRPBTLXX2QRT2OWJQ5HPNHZI5IVIFDVUUICVQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.619028879 (20h17m54.094967344s) [noise=true , address=104.36.82.9:4028 , piece_id=XCRNGUPBFYCZOYT23L6NHJ6WQR55TQMTWZF4SNKSRLQGC5TYO3LA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.619215837 (20h17m54.095154302s) [noise=true , address=104.36.82.53:4126 , piece_id=GZ5Z7AGMYVGYVUJAWFAIS6SQLRHFB772RLQY736QUGOXA5OWQ34Q, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.619408406 (20h17m54.095346871s) [noise=true , address=104.36.82.55:4219 , piece_id=IURHWONYUYYJWIL5BDF6HLCEPA76O6QAZAJ22NCI6LQYIGO6K4IA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:43.299790904 (20h17m55.775729369s) [upload=19386]: Next returned nil segment Jul 29 13:03:43.299806024 (20h17m55.775744479s) [upload=19386]: Waiting for error group managing segments... Jul 29 13:03:43.847040928 (20h17m56.322979393s) [noise=true , address=104.36.82.9:4028 , piece_id=XCRNGUPBFYCZOYT23L6NHJ6WQR55TQMTWZF4SNKSRLQGC5TYO3LA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:43.879495404 (20h17m56.355433869s) [noise=true , address=104.36.82.61:4391 , piece_id=TI5YDGDSTWKDX5XAWL36F3TJDQNHKNTREZVTUGIS2EVZYP6DWSDA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:46.350727192 (20h17m58.826665657s) [noise=true , address=104.36.82.5:4866 , piece_id=EFRLC4LA232NZZJU3GDSP2EEBZO3W2LGJRD3L2A6SPSRBTLUTIFA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:51.702944756 (20h18m4.178883221s) [noise=true , address=104.36.82.42:4802 , piece_id=6HORHYYKIKDX4IWWKUQPVGZQZ2YYIGC47OCZHRVBC4BZFRJX5TQQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:55.250827413 (20h18m7.726765878s) [noise=true , address=104.36.82.55:4219 , piece_id=IURHWONYUYYJWIL5BDF6HLCEPA76O6QAZAJ22NCI6LQYIGO6K4IA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:57.858469692 (20h18m10.334408157s) [noise=true , address=104.36.82.53:4126 , piece_id=GZ5Z7AGMYVGYVUJAWFAIS6SQLRHFB772RLQY736QUGOXA5OWQ34Q, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:59.265385056 (20h18m11.741323511s) [noise=true , address=104.36.82.10:4088 , piece_id=6IV2Z7FXDNEOBJMNPRG7WSPZWZTJ5CGDF6JHBSF7WCIIL2QQWFUA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:59.973580263 (20h18m12.449518729s) [noise=true , address=104.36.82.11:4119 , piece_id=6KWLABSUMX6YVKD6LGKR6ONLQKFFTSTKFHHRVFIN7U4KDALBNCBQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:16.002200978 (20h18m28.478139453s) [noise=true , address=104.36.82.52:4092 , piece_id=BMJCAQYXJCUVJTEXH52UXYLD4MHWLDGQTJIPDOFRF4LPLTT36ESA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:30.906580908 (20h18m43.382519373s) [noise=true , address=104.36.82.20:4448 , piece_id=CSHYRDGTS4ROFDTKETIZ36TWJE3OEMAKLB4RGFAU557G725EANXQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:31.164952501 (20h18m43.640890966s) [noise=true , address=104.36.82.14:4227 , piece_id=O2FXJBOV3CJCKLZHRPBTLXX2QRT2OWJQ5HPNHZI5IVIFDVUUICVQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:33.000876204 (20h18m45.476814679s) [noise=true , address=104.36.82.56:4242 , piece_id=BHKC5MH4G65BK2YLJJANIYZ4MLAUTMBG32XSRISARHCO26AZUXRA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:34.725342452 (20h18m47.201280907s) [noise=true , address=104.36.82.47:4545 , piece_id=PZWVIOWLBZ72IOFRJHPAD4OZZPA2G4XYCDLGYEY7THSOR7XDHMUQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:40.451024021 (20h18m52.926962486s) [noise=true , address=104.36.82.46:4500 , piece_id=VCYKKDSLEKTTL3KO2FJ4VA5JGIVPA4GFBBHH65HPFZTP4REUTHYQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:12B7ox4ehqxJ2mwqeXj8C9N9GuSU5etNC23iT1pJnTgMLGBAGQK, address:104.36.82.46:4500): protocol: write tcp 188.166.225.170:46700->104.36.82.46:4500: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451060500 (20h18m52.926998965s) [noise=true , address=104.36.82.51:4075 , piece_id=P5NU32M3V7AMYFZVJ6XZY5K2YMQWF4EULQGX6RRYWG6SWDWWAFMQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK, address:104.36.82.51:4075): protocol: write tcp 188.166.225.170:41162->104.36.82.51:4075: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451086179 (20h18m52.927024644s) [noise=true , address=104.36.82.54:4175 , piece_id=5PAE3BM5AFSLZ5TVUQC3QK3RVWCVGFDLOK452BDPWQ7ARTL7EYWQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:12ou7jwFyJP3mNoALddniUTDT9duVyQKZjeq4xHCyx6sRWpwEPJ, address:104.36.82.54:4175): protocol: rpc: dial tcp 104.36.82.54:4175: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451120024 (20h18m52.927058489s) [noise=true , address=104.36.82.13:4214 , piece_id=SLX3YJCEWCM3553VZHILH337FKSPPFNKVFTPXO4KIBOS6IXCLXHA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:1nJ8uqwxf6x4hX52AjAdCbMKi1BKGGbMvrQbiazJdLpn6j69ay, address:104.36.82.13:4214): protocol: rpc: dial tcp 104.36.82.13:4214: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451147116 (20h18m52.927085581s) [noise=true , address=104.36.82.15:4263 , piece_id=3GCLYN4UBCLM5BPLXDQT5SSWPJBTKIOZM4UIUNO3OZIXW6OANUBQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:1f97yBEvcrYHqMq7iovVoju77jzhUYgoQCsUGULeTagugFXrCu, address:104.36.82.15:4263): protocol: write tcp 188.166.225.170:39380->104.36.82.15:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451167805 (20h18m52.927106271s) [seg_pos={0 0}, upload=19386]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 29 13:04:40.696628655 (20h18m53.17256714s) [upload=19386]: Done uploading segments... Jul 29 13:04:40.696658391 (20h18m53.172596856s) [upload=19386]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.824562213s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 1.868935399s FAIL (2025-07-29 13:49:40 UTC) orbiter warning: uplink: too many requests Duration: 1.86895141s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................X.............X..............] duration: 5.937685918s FAIL (2025-07-29 13:49:29 UTC) orbiter warning: digest mismatch: after ee3c39c295e32d2453f46eedbd5a023ecb901c93b585366a6221da2ecebc35b3 != before f1d4b9d3eca6a9d63854d84cb67ac0fa26d296bc93761368409da19470c1a42f Duration: 5.937715736s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [X.X.........................................................] duration: 5.760972329s FAIL (2025-07-29 13:48:58 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ding segments... Jul 29 13:03:41.091320246 (20h17m53.567258701s) [upload=19386]: Got next segment. Inline: false Jul 29 13:03:41.091349191 (20h17m53.567287656s) [upload=19386]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 29 13:03:41.091354331 (20h17m53.567292796s) [upload=19386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 29 13:03:41.616444845 (20h17m54.09238331s) [seg_pos={0 0}, upload=19386]: Begin upload segment... Jul 29 13:03:41.616487748 (20h17m54.092426213s) [seg_pos={0 0}, upload=19386]: Done begin upload segment. Jul 29 13:03:41.616507376 (20h17m54.092445841s) [noise=true , address=104.36.82.15:4263 , piece_id=3GCLYN4UBCLM5BPLXDQT5SSWPJBTKIOZM4UIUNO3OZIXW6OANUBQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.616564315 (20h17m54.09250278s) [noise=true , address=104.36.82.11:4119 , piece_id=6KWLABSUMX6YVKD6LGKR6ONLQKFFTSTKFHHRVFIN7U4KDALBNCBQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.616866433 (20h17m54.092804888s) [noise=true , address=104.36.82.54:4175 , piece_id=5PAE3BM5AFSLZ5TVUQC3QK3RVWCVGFDLOK452BDPWQ7ARTL7EYWQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617072557 (20h17m54.093011022s) [noise=true , address=104.36.82.56:4242 , piece_id=BHKC5MH4G65BK2YLJJANIYZ4MLAUTMBG32XSRISARHCO26AZUXRA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617110629 (20h17m54.093049084s) [noise=true , address=104.36.82.5:4866 , piece_id=EFRLC4LA232NZZJU3GDSP2EEBZO3W2LGJRD3L2A6SPSRBTLUTIFA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617355037 (20h17m54.093293492s) [noise=true , address=104.36.82.61:4391 , piece_id=TI5YDGDSTWKDX5XAWL36F3TJDQNHKNTREZVTUGIS2EVZYP6DWSDA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617481549 (20h17m54.093420024s) [noise=true , address=104.36.82.20:4448 , piece_id=CSHYRDGTS4ROFDTKETIZ36TWJE3OEMAKLB4RGFAU557G725EANXQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617583044 (20h17m54.093521509s) [noise=true , address=104.36.82.47:4545 , piece_id=PZWVIOWLBZ72IOFRJHPAD4OZZPA2G4XYCDLGYEY7THSOR7XDHMUQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617828073 (20h17m54.093766528s) [noise=true , address=104.36.82.42:4802 , piece_id=6HORHYYKIKDX4IWWKUQPVGZQZ2YYIGC47OCZHRVBC4BZFRJX5TQQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617987597 (20h17m54.093926062s) [noise=true , address=104.36.82.13:4214 , piece_id=SLX3YJCEWCM3553VZHILH337FKSPPFNKVFTPXO4KIBOS6IXCLXHA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618055948 (20h17m54.093994403s) [noise=true , address=104.36.82.46:4500 , piece_id=VCYKKDSLEKTTL3KO2FJ4VA5JGIVPA4GFBBHH65HPFZTP4REUTHYQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618285828 (20h17m54.094224293s) [noise=true , address=104.36.82.52:4092 , piece_id=BMJCAQYXJCUVJTEXH52UXYLD4MHWLDGQTJIPDOFRF4LPLTT36ESA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618485871 (20h17m54.094424336s) [noise=true , address=104.36.82.10:4088 , piece_id=6IV2Z7FXDNEOBJMNPRG7WSPZWZTJ5CGDF6JHBSF7WCIIL2QQWFUA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618663881 (20h17m54.094602346s) [noise=true , address=104.36.82.51:4075 , piece_id=P5NU32M3V7AMYFZVJ6XZY5K2YMQWF4EULQGX6RRYWG6SWDWWAFMQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618840449 (20h17m54.094778914s) [noise=true , address=104.36.82.14:4227 , piece_id=O2FXJBOV3CJCKLZHRPBTLXX2QRT2OWJQ5HPNHZI5IVIFDVUUICVQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.619028879 (20h17m54.094967344s) [noise=true , address=104.36.82.9:4028 , piece_id=XCRNGUPBFYCZOYT23L6NHJ6WQR55TQMTWZF4SNKSRLQGC5TYO3LA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.619215837 (20h17m54.095154302s) [noise=true , address=104.36.82.53:4126 , piece_id=GZ5Z7AGMYVGYVUJAWFAIS6SQLRHFB772RLQY736QUGOXA5OWQ34Q, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.619408406 (20h17m54.095346871s) [noise=true , address=104.36.82.55:4219 , piece_id=IURHWONYUYYJWIL5BDF6HLCEPA76O6QAZAJ22NCI6LQYIGO6K4IA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:43.299790904 (20h17m55.775729369s) [upload=19386]: Next returned nil segment Jul 29 13:03:43.299806024 (20h17m55.775744479s) [upload=19386]: Waiting for error group managing segments... Jul 29 13:03:43.847040928 (20h17m56.322979393s) [noise=true , address=104.36.82.9:4028 , piece_id=XCRNGUPBFYCZOYT23L6NHJ6WQR55TQMTWZF4SNKSRLQGC5TYO3LA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:43.879495404 (20h17m56.355433869s) [noise=true , address=104.36.82.61:4391 , piece_id=TI5YDGDSTWKDX5XAWL36F3TJDQNHKNTREZVTUGIS2EVZYP6DWSDA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:46.350727192 (20h17m58.826665657s) [noise=true , address=104.36.82.5:4866 , piece_id=EFRLC4LA232NZZJU3GDSP2EEBZO3W2LGJRD3L2A6SPSRBTLUTIFA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:51.702944756 (20h18m4.178883221s) [noise=true , address=104.36.82.42:4802 , piece_id=6HORHYYKIKDX4IWWKUQPVGZQZ2YYIGC47OCZHRVBC4BZFRJX5TQQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:55.250827413 (20h18m7.726765878s) [noise=true , address=104.36.82.55:4219 , piece_id=IURHWONYUYYJWIL5BDF6HLCEPA76O6QAZAJ22NCI6LQYIGO6K4IA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:57.858469692 (20h18m10.334408157s) [noise=true , address=104.36.82.53:4126 , piece_id=GZ5Z7AGMYVGYVUJAWFAIS6SQLRHFB772RLQY736QUGOXA5OWQ34Q, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:59.265385056 (20h18m11.741323511s) [noise=true , address=104.36.82.10:4088 , piece_id=6IV2Z7FXDNEOBJMNPRG7WSPZWZTJ5CGDF6JHBSF7WCIIL2QQWFUA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:59.973580263 (20h18m12.449518729s) [noise=true , address=104.36.82.11:4119 , piece_id=6KWLABSUMX6YVKD6LGKR6ONLQKFFTSTKFHHRVFIN7U4KDALBNCBQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:16.002200978 (20h18m28.478139453s) [noise=true , address=104.36.82.52:4092 , piece_id=BMJCAQYXJCUVJTEXH52UXYLD4MHWLDGQTJIPDOFRF4LPLTT36ESA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:30.906580908 (20h18m43.382519373s) [noise=true , address=104.36.82.20:4448 , piece_id=CSHYRDGTS4ROFDTKETIZ36TWJE3OEMAKLB4RGFAU557G725EANXQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:31.164952501 (20h18m43.640890966s) [noise=true , address=104.36.82.14:4227 , piece_id=O2FXJBOV3CJCKLZHRPBTLXX2QRT2OWJQ5HPNHZI5IVIFDVUUICVQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:33.000876204 (20h18m45.476814679s) [noise=true , address=104.36.82.56:4242 , piece_id=BHKC5MH4G65BK2YLJJANIYZ4MLAUTMBG32XSRISARHCO26AZUXRA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:34.725342452 (20h18m47.201280907s) [noise=true , address=104.36.82.47:4545 , piece_id=PZWVIOWLBZ72IOFRJHPAD4OZZPA2G4XYCDLGYEY7THSOR7XDHMUQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:40.451024021 (20h18m52.926962486s) [noise=true , address=104.36.82.46:4500 , piece_id=VCYKKDSLEKTTL3KO2FJ4VA5JGIVPA4GFBBHH65HPFZTP4REUTHYQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:12B7ox4ehqxJ2mwqeXj8C9N9GuSU5etNC23iT1pJnTgMLGBAGQK, address:104.36.82.46:4500): protocol: write tcp 188.166.225.170:46700->104.36.82.46:4500: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451060500 (20h18m52.926998965s) [noise=true , address=104.36.82.51:4075 , piece_id=P5NU32M3V7AMYFZVJ6XZY5K2YMQWF4EULQGX6RRYWG6SWDWWAFMQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK, address:104.36.82.51:4075): protocol: write tcp 188.166.225.170:41162->104.36.82.51:4075: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451086179 (20h18m52.927024644s) [noise=true , address=104.36.82.54:4175 , piece_id=5PAE3BM5AFSLZ5TVUQC3QK3RVWCVGFDLOK452BDPWQ7ARTL7EYWQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:12ou7jwFyJP3mNoALddniUTDT9duVyQKZjeq4xHCyx6sRWpwEPJ, address:104.36.82.54:4175): protocol: rpc: dial tcp 104.36.82.54:4175: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451120024 (20h18m52.927058489s) [noise=true , address=104.36.82.13:4214 , piece_id=SLX3YJCEWCM3553VZHILH337FKSPPFNKVFTPXO4KIBOS6IXCLXHA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:1nJ8uqwxf6x4hX52AjAdCbMKi1BKGGbMvrQbiazJdLpn6j69ay, address:104.36.82.13:4214): protocol: rpc: dial tcp 104.36.82.13:4214: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451147116 (20h18m52.927085581s) [noise=true , address=104.36.82.15:4263 , piece_id=3GCLYN4UBCLM5BPLXDQT5SSWPJBTKIOZM4UIUNO3OZIXW6OANUBQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:1f97yBEvcrYHqMq7iovVoju77jzhUYgoQCsUGULeTagugFXrCu, address:104.36.82.15:4263): protocol: write tcp 188.166.225.170:39380->104.36.82.15:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451167805 (20h18m52.927106271s) [seg_pos={0 0}, upload=19386]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 29 13:04:40.696628655 (20h18m53.17256714s) [upload=19386]: Done uploading segments... Jul 29 13:04:40.696658391 (20h18m53.172596856s) [upload=19386]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.760996245s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................X.............X.............] duration: 6.025537599s FAIL (2025-07-29 13:48:58 UTC) orbiter warning: digest mismatch: after ee3c39c295e32d2453f46eedbd5a023ecb901c93b585366a6221da2ecebc35b3 != before f1d4b9d3eca6a9d63854d84cb67ac0fa26d296bc93761368409da19470c1a42f Duration: 6.025562807s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 1.845854357s FAIL (2025-07-29 13:48:50 UTC) orbiter warning: uplink: too many requests Duration: 1.845878584s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.X.X........................................................] duration: 5.546402817s FAIL (2025-07-29 13:48:39 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ding segments... Jul 29 13:03:41.091320246 (20h17m53.567258701s) [upload=19386]: Got next segment. Inline: false Jul 29 13:03:41.091349191 (20h17m53.567287656s) [upload=19386]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 29 13:03:41.091354331 (20h17m53.567292796s) [upload=19386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 29 13:03:41.616444845 (20h17m54.09238331s) [seg_pos={0 0}, upload=19386]: Begin upload segment... Jul 29 13:03:41.616487748 (20h17m54.092426213s) [seg_pos={0 0}, upload=19386]: Done begin upload segment. Jul 29 13:03:41.616507376 (20h17m54.092445841s) [noise=true , address=104.36.82.15:4263 , piece_id=3GCLYN4UBCLM5BPLXDQT5SSWPJBTKIOZM4UIUNO3OZIXW6OANUBQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.616564315 (20h17m54.09250278s) [noise=true , address=104.36.82.11:4119 , piece_id=6KWLABSUMX6YVKD6LGKR6ONLQKFFTSTKFHHRVFIN7U4KDALBNCBQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.616866433 (20h17m54.092804888s) [noise=true , address=104.36.82.54:4175 , piece_id=5PAE3BM5AFSLZ5TVUQC3QK3RVWCVGFDLOK452BDPWQ7ARTL7EYWQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617072557 (20h17m54.093011022s) [noise=true , address=104.36.82.56:4242 , piece_id=BHKC5MH4G65BK2YLJJANIYZ4MLAUTMBG32XSRISARHCO26AZUXRA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617110629 (20h17m54.093049084s) [noise=true , address=104.36.82.5:4866 , piece_id=EFRLC4LA232NZZJU3GDSP2EEBZO3W2LGJRD3L2A6SPSRBTLUTIFA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617355037 (20h17m54.093293492s) [noise=true , address=104.36.82.61:4391 , piece_id=TI5YDGDSTWKDX5XAWL36F3TJDQNHKNTREZVTUGIS2EVZYP6DWSDA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617481549 (20h17m54.093420024s) [noise=true , address=104.36.82.20:4448 , piece_id=CSHYRDGTS4ROFDTKETIZ36TWJE3OEMAKLB4RGFAU557G725EANXQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617583044 (20h17m54.093521509s) [noise=true , address=104.36.82.47:4545 , piece_id=PZWVIOWLBZ72IOFRJHPAD4OZZPA2G4XYCDLGYEY7THSOR7XDHMUQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617828073 (20h17m54.093766528s) [noise=true , address=104.36.82.42:4802 , piece_id=6HORHYYKIKDX4IWWKUQPVGZQZ2YYIGC47OCZHRVBC4BZFRJX5TQQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617987597 (20h17m54.093926062s) [noise=true , address=104.36.82.13:4214 , piece_id=SLX3YJCEWCM3553VZHILH337FKSPPFNKVFTPXO4KIBOS6IXCLXHA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618055948 (20h17m54.093994403s) [noise=true , address=104.36.82.46:4500 , piece_id=VCYKKDSLEKTTL3KO2FJ4VA5JGIVPA4GFBBHH65HPFZTP4REUTHYQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618285828 (20h17m54.094224293s) [noise=true , address=104.36.82.52:4092 , piece_id=BMJCAQYXJCUVJTEXH52UXYLD4MHWLDGQTJIPDOFRF4LPLTT36ESA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618485871 (20h17m54.094424336s) [noise=true , address=104.36.82.10:4088 , piece_id=6IV2Z7FXDNEOBJMNPRG7WSPZWZTJ5CGDF6JHBSF7WCIIL2QQWFUA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618663881 (20h17m54.094602346s) [noise=true , address=104.36.82.51:4075 , piece_id=P5NU32M3V7AMYFZVJ6XZY5K2YMQWF4EULQGX6RRYWG6SWDWWAFMQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618840449 (20h17m54.094778914s) [noise=true , address=104.36.82.14:4227 , piece_id=O2FXJBOV3CJCKLZHRPBTLXX2QRT2OWJQ5HPNHZI5IVIFDVUUICVQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.619028879 (20h17m54.094967344s) [noise=true , address=104.36.82.9:4028 , piece_id=XCRNGUPBFYCZOYT23L6NHJ6WQR55TQMTWZF4SNKSRLQGC5TYO3LA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.619215837 (20h17m54.095154302s) [noise=true , address=104.36.82.53:4126 , piece_id=GZ5Z7AGMYVGYVUJAWFAIS6SQLRHFB772RLQY736QUGOXA5OWQ34Q, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.619408406 (20h17m54.095346871s) [noise=true , address=104.36.82.55:4219 , piece_id=IURHWONYUYYJWIL5BDF6HLCEPA76O6QAZAJ22NCI6LQYIGO6K4IA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:43.299790904 (20h17m55.775729369s) [upload=19386]: Next returned nil segment Jul 29 13:03:43.299806024 (20h17m55.775744479s) [upload=19386]: Waiting for error group managing segments... Jul 29 13:03:43.847040928 (20h17m56.322979393s) [noise=true , address=104.36.82.9:4028 , piece_id=XCRNGUPBFYCZOYT23L6NHJ6WQR55TQMTWZF4SNKSRLQGC5TYO3LA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:43.879495404 (20h17m56.355433869s) [noise=true , address=104.36.82.61:4391 , piece_id=TI5YDGDSTWKDX5XAWL36F3TJDQNHKNTREZVTUGIS2EVZYP6DWSDA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:46.350727192 (20h17m58.826665657s) [noise=true , address=104.36.82.5:4866 , piece_id=EFRLC4LA232NZZJU3GDSP2EEBZO3W2LGJRD3L2A6SPSRBTLUTIFA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:51.702944756 (20h18m4.178883221s) [noise=true , address=104.36.82.42:4802 , piece_id=6HORHYYKIKDX4IWWKUQPVGZQZ2YYIGC47OCZHRVBC4BZFRJX5TQQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:55.250827413 (20h18m7.726765878s) [noise=true , address=104.36.82.55:4219 , piece_id=IURHWONYUYYJWIL5BDF6HLCEPA76O6QAZAJ22NCI6LQYIGO6K4IA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:57.858469692 (20h18m10.334408157s) [noise=true , address=104.36.82.53:4126 , piece_id=GZ5Z7AGMYVGYVUJAWFAIS6SQLRHFB772RLQY736QUGOXA5OWQ34Q, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:59.265385056 (20h18m11.741323511s) [noise=true , address=104.36.82.10:4088 , piece_id=6IV2Z7FXDNEOBJMNPRG7WSPZWZTJ5CGDF6JHBSF7WCIIL2QQWFUA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:59.973580263 (20h18m12.449518729s) [noise=true , address=104.36.82.11:4119 , piece_id=6KWLABSUMX6YVKD6LGKR6ONLQKFFTSTKFHHRVFIN7U4KDALBNCBQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:16.002200978 (20h18m28.478139453s) [noise=true , address=104.36.82.52:4092 , piece_id=BMJCAQYXJCUVJTEXH52UXYLD4MHWLDGQTJIPDOFRF4LPLTT36ESA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:30.906580908 (20h18m43.382519373s) [noise=true , address=104.36.82.20:4448 , piece_id=CSHYRDGTS4ROFDTKETIZ36TWJE3OEMAKLB4RGFAU557G725EANXQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:31.164952501 (20h18m43.640890966s) [noise=true , address=104.36.82.14:4227 , piece_id=O2FXJBOV3CJCKLZHRPBTLXX2QRT2OWJQ5HPNHZI5IVIFDVUUICVQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:33.000876204 (20h18m45.476814679s) [noise=true , address=104.36.82.56:4242 , piece_id=BHKC5MH4G65BK2YLJJANIYZ4MLAUTMBG32XSRISARHCO26AZUXRA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:34.725342452 (20h18m47.201280907s) [noise=true , address=104.36.82.47:4545 , piece_id=PZWVIOWLBZ72IOFRJHPAD4OZZPA2G4XYCDLGYEY7THSOR7XDHMUQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:40.451024021 (20h18m52.926962486s) [noise=true , address=104.36.82.46:4500 , piece_id=VCYKKDSLEKTTL3KO2FJ4VA5JGIVPA4GFBBHH65HPFZTP4REUTHYQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:12B7ox4ehqxJ2mwqeXj8C9N9GuSU5etNC23iT1pJnTgMLGBAGQK, address:104.36.82.46:4500): protocol: write tcp 188.166.225.170:46700->104.36.82.46:4500: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451060500 (20h18m52.926998965s) [noise=true , address=104.36.82.51:4075 , piece_id=P5NU32M3V7AMYFZVJ6XZY5K2YMQWF4EULQGX6RRYWG6SWDWWAFMQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK, address:104.36.82.51:4075): protocol: write tcp 188.166.225.170:41162->104.36.82.51:4075: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451086179 (20h18m52.927024644s) [noise=true , address=104.36.82.54:4175 , piece_id=5PAE3BM5AFSLZ5TVUQC3QK3RVWCVGFDLOK452BDPWQ7ARTL7EYWQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:12ou7jwFyJP3mNoALddniUTDT9duVyQKZjeq4xHCyx6sRWpwEPJ, address:104.36.82.54:4175): protocol: rpc: dial tcp 104.36.82.54:4175: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451120024 (20h18m52.927058489s) [noise=true , address=104.36.82.13:4214 , piece_id=SLX3YJCEWCM3553VZHILH337FKSPPFNKVFTPXO4KIBOS6IXCLXHA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:1nJ8uqwxf6x4hX52AjAdCbMKi1BKGGbMvrQbiazJdLpn6j69ay, address:104.36.82.13:4214): protocol: rpc: dial tcp 104.36.82.13:4214: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451147116 (20h18m52.927085581s) [noise=true , address=104.36.82.15:4263 , piece_id=3GCLYN4UBCLM5BPLXDQT5SSWPJBTKIOZM4UIUNO3OZIXW6OANUBQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:1f97yBEvcrYHqMq7iovVoju77jzhUYgoQCsUGULeTagugFXrCu, address:104.36.82.15:4263): protocol: write tcp 188.166.225.170:39380->104.36.82.15:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451167805 (20h18m52.927106271s) [seg_pos={0 0}, upload=19386]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 29 13:04:40.696628655 (20h18m53.17256714s) [upload=19386]: Done uploading segments... Jul 29 13:04:40.696658391 (20h18m53.172596856s) [upload=19386]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.546426822s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................X.............X............] duration: 6.041674329s FAIL (2025-07-29 13:48:38 UTC) orbiter warning: digest mismatch: after ee3c39c295e32d2453f46eedbd5a023ecb901c93b585366a6221da2ecebc35b3 != before f1d4b9d3eca6a9d63854d84cb67ac0fa26d296bc93761368409da19470c1a42f Duration: 6.041697315s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 1.976180054s FAIL (2025-07-29 13:48:23 UTC) orbiter warning: uplink: too many requests Duration: 1.976219149s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [X.................................X.............X...........] duration: 5.69481119s FAIL (2025-07-29 13:47:51 UTC) orbiter warning: digest mismatch: after ee3c39c295e32d2453f46eedbd5a023ecb901c93b585366a6221da2ecebc35b3 != before f1d4b9d3eca6a9d63854d84cb67ac0fa26d296bc93761368409da19470c1a42f Duration: 5.694835568s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..X.X.......................................................] duration: 5.949131254s FAIL (2025-07-29 13:47:37 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ding segments... Jul 29 13:03:41.091320246 (20h17m53.567258701s) [upload=19386]: Got next segment. Inline: false Jul 29 13:03:41.091349191 (20h17m53.567287656s) [upload=19386]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 29 13:03:41.091354331 (20h17m53.567292796s) [upload=19386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 29 13:03:41.616444845 (20h17m54.09238331s) [seg_pos={0 0}, upload=19386]: Begin upload segment... Jul 29 13:03:41.616487748 (20h17m54.092426213s) [seg_pos={0 0}, upload=19386]: Done begin upload segment. Jul 29 13:03:41.616507376 (20h17m54.092445841s) [noise=true , address=104.36.82.15:4263 , piece_id=3GCLYN4UBCLM5BPLXDQT5SSWPJBTKIOZM4UIUNO3OZIXW6OANUBQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.616564315 (20h17m54.09250278s) [noise=true , address=104.36.82.11:4119 , piece_id=6KWLABSUMX6YVKD6LGKR6ONLQKFFTSTKFHHRVFIN7U4KDALBNCBQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.616866433 (20h17m54.092804888s) [noise=true , address=104.36.82.54:4175 , piece_id=5PAE3BM5AFSLZ5TVUQC3QK3RVWCVGFDLOK452BDPWQ7ARTL7EYWQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617072557 (20h17m54.093011022s) [noise=true , address=104.36.82.56:4242 , piece_id=BHKC5MH4G65BK2YLJJANIYZ4MLAUTMBG32XSRISARHCO26AZUXRA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617110629 (20h17m54.093049084s) [noise=true , address=104.36.82.5:4866 , piece_id=EFRLC4LA232NZZJU3GDSP2EEBZO3W2LGJRD3L2A6SPSRBTLUTIFA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617355037 (20h17m54.093293492s) [noise=true , address=104.36.82.61:4391 , piece_id=TI5YDGDSTWKDX5XAWL36F3TJDQNHKNTREZVTUGIS2EVZYP6DWSDA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617481549 (20h17m54.093420024s) [noise=true , address=104.36.82.20:4448 , piece_id=CSHYRDGTS4ROFDTKETIZ36TWJE3OEMAKLB4RGFAU557G725EANXQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617583044 (20h17m54.093521509s) [noise=true , address=104.36.82.47:4545 , piece_id=PZWVIOWLBZ72IOFRJHPAD4OZZPA2G4XYCDLGYEY7THSOR7XDHMUQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617828073 (20h17m54.093766528s) [noise=true , address=104.36.82.42:4802 , piece_id=6HORHYYKIKDX4IWWKUQPVGZQZ2YYIGC47OCZHRVBC4BZFRJX5TQQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617987597 (20h17m54.093926062s) [noise=true , address=104.36.82.13:4214 , piece_id=SLX3YJCEWCM3553VZHILH337FKSPPFNKVFTPXO4KIBOS6IXCLXHA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618055948 (20h17m54.093994403s) [noise=true , address=104.36.82.46:4500 , piece_id=VCYKKDSLEKTTL3KO2FJ4VA5JGIVPA4GFBBHH65HPFZTP4REUTHYQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618285828 (20h17m54.094224293s) [noise=true , address=104.36.82.52:4092 , piece_id=BMJCAQYXJCUVJTEXH52UXYLD4MHWLDGQTJIPDOFRF4LPLTT36ESA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618485871 (20h17m54.094424336s) [noise=true , address=104.36.82.10:4088 , piece_id=6IV2Z7FXDNEOBJMNPRG7WSPZWZTJ5CGDF6JHBSF7WCIIL2QQWFUA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618663881 (20h17m54.094602346s) [noise=true , address=104.36.82.51:4075 , piece_id=P5NU32M3V7AMYFZVJ6XZY5K2YMQWF4EULQGX6RRYWG6SWDWWAFMQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618840449 (20h17m54.094778914s) [noise=true , address=104.36.82.14:4227 , piece_id=O2FXJBOV3CJCKLZHRPBTLXX2QRT2OWJQ5HPNHZI5IVIFDVUUICVQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.619028879 (20h17m54.094967344s) [noise=true , address=104.36.82.9:4028 , piece_id=XCRNGUPBFYCZOYT23L6NHJ6WQR55TQMTWZF4SNKSRLQGC5TYO3LA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.619215837 (20h17m54.095154302s) [noise=true , address=104.36.82.53:4126 , piece_id=GZ5Z7AGMYVGYVUJAWFAIS6SQLRHFB772RLQY736QUGOXA5OWQ34Q, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.619408406 (20h17m54.095346871s) [noise=true , address=104.36.82.55:4219 , piece_id=IURHWONYUYYJWIL5BDF6HLCEPA76O6QAZAJ22NCI6LQYIGO6K4IA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:43.299790904 (20h17m55.775729369s) [upload=19386]: Next returned nil segment Jul 29 13:03:43.299806024 (20h17m55.775744479s) [upload=19386]: Waiting for error group managing segments... Jul 29 13:03:43.847040928 (20h17m56.322979393s) [noise=true , address=104.36.82.9:4028 , piece_id=XCRNGUPBFYCZOYT23L6NHJ6WQR55TQMTWZF4SNKSRLQGC5TYO3LA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:43.879495404 (20h17m56.355433869s) [noise=true , address=104.36.82.61:4391 , piece_id=TI5YDGDSTWKDX5XAWL36F3TJDQNHKNTREZVTUGIS2EVZYP6DWSDA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:46.350727192 (20h17m58.826665657s) [noise=true , address=104.36.82.5:4866 , piece_id=EFRLC4LA232NZZJU3GDSP2EEBZO3W2LGJRD3L2A6SPSRBTLUTIFA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:51.702944756 (20h18m4.178883221s) [noise=true , address=104.36.82.42:4802 , piece_id=6HORHYYKIKDX4IWWKUQPVGZQZ2YYIGC47OCZHRVBC4BZFRJX5TQQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:55.250827413 (20h18m7.726765878s) [noise=true , address=104.36.82.55:4219 , piece_id=IURHWONYUYYJWIL5BDF6HLCEPA76O6QAZAJ22NCI6LQYIGO6K4IA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:57.858469692 (20h18m10.334408157s) [noise=true , address=104.36.82.53:4126 , piece_id=GZ5Z7AGMYVGYVUJAWFAIS6SQLRHFB772RLQY736QUGOXA5OWQ34Q, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:59.265385056 (20h18m11.741323511s) [noise=true , address=104.36.82.10:4088 , piece_id=6IV2Z7FXDNEOBJMNPRG7WSPZWZTJ5CGDF6JHBSF7WCIIL2QQWFUA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:59.973580263 (20h18m12.449518729s) [noise=true , address=104.36.82.11:4119 , piece_id=6KWLABSUMX6YVKD6LGKR6ONLQKFFTSTKFHHRVFIN7U4KDALBNCBQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:16.002200978 (20h18m28.478139453s) [noise=true , address=104.36.82.52:4092 , piece_id=BMJCAQYXJCUVJTEXH52UXYLD4MHWLDGQTJIPDOFRF4LPLTT36ESA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:30.906580908 (20h18m43.382519373s) [noise=true , address=104.36.82.20:4448 , piece_id=CSHYRDGTS4ROFDTKETIZ36TWJE3OEMAKLB4RGFAU557G725EANXQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:31.164952501 (20h18m43.640890966s) [noise=true , address=104.36.82.14:4227 , piece_id=O2FXJBOV3CJCKLZHRPBTLXX2QRT2OWJQ5HPNHZI5IVIFDVUUICVQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:33.000876204 (20h18m45.476814679s) [noise=true , address=104.36.82.56:4242 , piece_id=BHKC5MH4G65BK2YLJJANIYZ4MLAUTMBG32XSRISARHCO26AZUXRA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:34.725342452 (20h18m47.201280907s) [noise=true , address=104.36.82.47:4545 , piece_id=PZWVIOWLBZ72IOFRJHPAD4OZZPA2G4XYCDLGYEY7THSOR7XDHMUQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:40.451024021 (20h18m52.926962486s) [noise=true , address=104.36.82.46:4500 , piece_id=VCYKKDSLEKTTL3KO2FJ4VA5JGIVPA4GFBBHH65HPFZTP4REUTHYQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:12B7ox4ehqxJ2mwqeXj8C9N9GuSU5etNC23iT1pJnTgMLGBAGQK, address:104.36.82.46:4500): protocol: write tcp 188.166.225.170:46700->104.36.82.46:4500: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451060500 (20h18m52.926998965s) [noise=true , address=104.36.82.51:4075 , piece_id=P5NU32M3V7AMYFZVJ6XZY5K2YMQWF4EULQGX6RRYWG6SWDWWAFMQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK, address:104.36.82.51:4075): protocol: write tcp 188.166.225.170:41162->104.36.82.51:4075: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451086179 (20h18m52.927024644s) [noise=true , address=104.36.82.54:4175 , piece_id=5PAE3BM5AFSLZ5TVUQC3QK3RVWCVGFDLOK452BDPWQ7ARTL7EYWQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:12ou7jwFyJP3mNoALddniUTDT9duVyQKZjeq4xHCyx6sRWpwEPJ, address:104.36.82.54:4175): protocol: rpc: dial tcp 104.36.82.54:4175: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451120024 (20h18m52.927058489s) [noise=true , address=104.36.82.13:4214 , piece_id=SLX3YJCEWCM3553VZHILH337FKSPPFNKVFTPXO4KIBOS6IXCLXHA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:1nJ8uqwxf6x4hX52AjAdCbMKi1BKGGbMvrQbiazJdLpn6j69ay, address:104.36.82.13:4214): protocol: rpc: dial tcp 104.36.82.13:4214: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451147116 (20h18m52.927085581s) [noise=true , address=104.36.82.15:4263 , piece_id=3GCLYN4UBCLM5BPLXDQT5SSWPJBTKIOZM4UIUNO3OZIXW6OANUBQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:1f97yBEvcrYHqMq7iovVoju77jzhUYgoQCsUGULeTagugFXrCu, address:104.36.82.15:4263): protocol: write tcp 188.166.225.170:39380->104.36.82.15:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451167805 (20h18m52.927106271s) [seg_pos={0 0}, upload=19386]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 29 13:04:40.696628655 (20h18m53.17256714s) [upload=19386]: Done uploading segments... Jul 29 13:04:40.696658391 (20h18m53.172596856s) [upload=19386]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.949155202s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [XX.................................X.............X..........] duration: 5.906036058s FAIL (2025-07-29 13:47:35 UTC) orbiter warning: digest mismatch: after ee3c39c295e32d2453f46eedbd5a023ecb901c93b585366a6221da2ecebc35b3 != before f1d4b9d3eca6a9d63854d84cb67ac0fa26d296bc93761368409da19470c1a42f Duration: 5.906065063s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 1.826091354s FAIL (2025-07-29 13:47:23 UTC) orbiter warning: uplink: too many requests Duration: 1.826116873s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [.XX.................................X.............X.........] duration: 6.06444627s FAIL (2025-07-29 13:47:15 UTC) orbiter warning: digest mismatch: after ee3c39c295e32d2453f46eedbd5a023ecb901c93b585366a6221da2ecebc35b3 != before f1d4b9d3eca6a9d63854d84cb67ac0fa26d296bc93761368409da19470c1a42f Duration: 6.064475164s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [..XX.................................X.............X........] duration: 5.87568407s FAIL (2025-07-29 13:46:49 UTC) orbiter warning: digest mismatch: after ee3c39c295e32d2453f46eedbd5a023ecb901c93b585366a6221da2ecebc35b3 != before f1d4b9d3eca6a9d63854d84cb67ac0fa26d296bc93761368409da19470c1a42f Duration: 5.875712764s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...X.X......................................................] duration: 5.798712565s FAIL (2025-07-29 13:46:30 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ding segments... Jul 29 13:03:41.091320246 (20h17m53.567258701s) [upload=19386]: Got next segment. Inline: false Jul 29 13:03:41.091349191 (20h17m53.567287656s) [upload=19386]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 29 13:03:41.091354331 (20h17m53.567292796s) [upload=19386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 29 13:03:41.616444845 (20h17m54.09238331s) [seg_pos={0 0}, upload=19386]: Begin upload segment... Jul 29 13:03:41.616487748 (20h17m54.092426213s) [seg_pos={0 0}, upload=19386]: Done begin upload segment. Jul 29 13:03:41.616507376 (20h17m54.092445841s) [noise=true , address=104.36.82.15:4263 , piece_id=3GCLYN4UBCLM5BPLXDQT5SSWPJBTKIOZM4UIUNO3OZIXW6OANUBQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.616564315 (20h17m54.09250278s) [noise=true , address=104.36.82.11:4119 , piece_id=6KWLABSUMX6YVKD6LGKR6ONLQKFFTSTKFHHRVFIN7U4KDALBNCBQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.616866433 (20h17m54.092804888s) [noise=true , address=104.36.82.54:4175 , piece_id=5PAE3BM5AFSLZ5TVUQC3QK3RVWCVGFDLOK452BDPWQ7ARTL7EYWQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617072557 (20h17m54.093011022s) [noise=true , address=104.36.82.56:4242 , piece_id=BHKC5MH4G65BK2YLJJANIYZ4MLAUTMBG32XSRISARHCO26AZUXRA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617110629 (20h17m54.093049084s) [noise=true , address=104.36.82.5:4866 , piece_id=EFRLC4LA232NZZJU3GDSP2EEBZO3W2LGJRD3L2A6SPSRBTLUTIFA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617355037 (20h17m54.093293492s) [noise=true , address=104.36.82.61:4391 , piece_id=TI5YDGDSTWKDX5XAWL36F3TJDQNHKNTREZVTUGIS2EVZYP6DWSDA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617481549 (20h17m54.093420024s) [noise=true , address=104.36.82.20:4448 , piece_id=CSHYRDGTS4ROFDTKETIZ36TWJE3OEMAKLB4RGFAU557G725EANXQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617583044 (20h17m54.093521509s) [noise=true , address=104.36.82.47:4545 , piece_id=PZWVIOWLBZ72IOFRJHPAD4OZZPA2G4XYCDLGYEY7THSOR7XDHMUQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617828073 (20h17m54.093766528s) [noise=true , address=104.36.82.42:4802 , piece_id=6HORHYYKIKDX4IWWKUQPVGZQZ2YYIGC47OCZHRVBC4BZFRJX5TQQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617987597 (20h17m54.093926062s) [noise=true , address=104.36.82.13:4214 , piece_id=SLX3YJCEWCM3553VZHILH337FKSPPFNKVFTPXO4KIBOS6IXCLXHA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618055948 (20h17m54.093994403s) [noise=true , address=104.36.82.46:4500 , piece_id=VCYKKDSLEKTTL3KO2FJ4VA5JGIVPA4GFBBHH65HPFZTP4REUTHYQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618285828 (20h17m54.094224293s) [noise=true , address=104.36.82.52:4092 , piece_id=BMJCAQYXJCUVJTEXH52UXYLD4MHWLDGQTJIPDOFRF4LPLTT36ESA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618485871 (20h17m54.094424336s) [noise=true , address=104.36.82.10:4088 , piece_id=6IV2Z7FXDNEOBJMNPRG7WSPZWZTJ5CGDF6JHBSF7WCIIL2QQWFUA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618663881 (20h17m54.094602346s) [noise=true , address=104.36.82.51:4075 , piece_id=P5NU32M3V7AMYFZVJ6XZY5K2YMQWF4EULQGX6RRYWG6SWDWWAFMQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618840449 (20h17m54.094778914s) [noise=true , address=104.36.82.14:4227 , piece_id=O2FXJBOV3CJCKLZHRPBTLXX2QRT2OWJQ5HPNHZI5IVIFDVUUICVQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.619028879 (20h17m54.094967344s) [noise=true , address=104.36.82.9:4028 , piece_id=XCRNGUPBFYCZOYT23L6NHJ6WQR55TQMTWZF4SNKSRLQGC5TYO3LA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.619215837 (20h17m54.095154302s) [noise=true , address=104.36.82.53:4126 , piece_id=GZ5Z7AGMYVGYVUJAWFAIS6SQLRHFB772RLQY736QUGOXA5OWQ34Q, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.619408406 (20h17m54.095346871s) [noise=true , address=104.36.82.55:4219 , piece_id=IURHWONYUYYJWIL5BDF6HLCEPA76O6QAZAJ22NCI6LQYIGO6K4IA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:43.299790904 (20h17m55.775729369s) [upload=19386]: Next returned nil segment Jul 29 13:03:43.299806024 (20h17m55.775744479s) [upload=19386]: Waiting for error group managing segments... Jul 29 13:03:43.847040928 (20h17m56.322979393s) [noise=true , address=104.36.82.9:4028 , piece_id=XCRNGUPBFYCZOYT23L6NHJ6WQR55TQMTWZF4SNKSRLQGC5TYO3LA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:43.879495404 (20h17m56.355433869s) [noise=true , address=104.36.82.61:4391 , piece_id=TI5YDGDSTWKDX5XAWL36F3TJDQNHKNTREZVTUGIS2EVZYP6DWSDA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:46.350727192 (20h17m58.826665657s) [noise=true , address=104.36.82.5:4866 , piece_id=EFRLC4LA232NZZJU3GDSP2EEBZO3W2LGJRD3L2A6SPSRBTLUTIFA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:51.702944756 (20h18m4.178883221s) [noise=true , address=104.36.82.42:4802 , piece_id=6HORHYYKIKDX4IWWKUQPVGZQZ2YYIGC47OCZHRVBC4BZFRJX5TQQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:55.250827413 (20h18m7.726765878s) [noise=true , address=104.36.82.55:4219 , piece_id=IURHWONYUYYJWIL5BDF6HLCEPA76O6QAZAJ22NCI6LQYIGO6K4IA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:57.858469692 (20h18m10.334408157s) [noise=true , address=104.36.82.53:4126 , piece_id=GZ5Z7AGMYVGYVUJAWFAIS6SQLRHFB772RLQY736QUGOXA5OWQ34Q, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:59.265385056 (20h18m11.741323511s) [noise=true , address=104.36.82.10:4088 , piece_id=6IV2Z7FXDNEOBJMNPRG7WSPZWZTJ5CGDF6JHBSF7WCIIL2QQWFUA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:59.973580263 (20h18m12.449518729s) [noise=true , address=104.36.82.11:4119 , piece_id=6KWLABSUMX6YVKD6LGKR6ONLQKFFTSTKFHHRVFIN7U4KDALBNCBQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:16.002200978 (20h18m28.478139453s) [noise=true , address=104.36.82.52:4092 , piece_id=BMJCAQYXJCUVJTEXH52UXYLD4MHWLDGQTJIPDOFRF4LPLTT36ESA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:30.906580908 (20h18m43.382519373s) [noise=true , address=104.36.82.20:4448 , piece_id=CSHYRDGTS4ROFDTKETIZ36TWJE3OEMAKLB4RGFAU557G725EANXQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:31.164952501 (20h18m43.640890966s) [noise=true , address=104.36.82.14:4227 , piece_id=O2FXJBOV3CJCKLZHRPBTLXX2QRT2OWJQ5HPNHZI5IVIFDVUUICVQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:33.000876204 (20h18m45.476814679s) [noise=true , address=104.36.82.56:4242 , piece_id=BHKC5MH4G65BK2YLJJANIYZ4MLAUTMBG32XSRISARHCO26AZUXRA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:34.725342452 (20h18m47.201280907s) [noise=true , address=104.36.82.47:4545 , piece_id=PZWVIOWLBZ72IOFRJHPAD4OZZPA2G4XYCDLGYEY7THSOR7XDHMUQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:40.451024021 (20h18m52.926962486s) [noise=true , address=104.36.82.46:4500 , piece_id=VCYKKDSLEKTTL3KO2FJ4VA5JGIVPA4GFBBHH65HPFZTP4REUTHYQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:12B7ox4ehqxJ2mwqeXj8C9N9GuSU5etNC23iT1pJnTgMLGBAGQK, address:104.36.82.46:4500): protocol: write tcp 188.166.225.170:46700->104.36.82.46:4500: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451060500 (20h18m52.926998965s) [noise=true , address=104.36.82.51:4075 , piece_id=P5NU32M3V7AMYFZVJ6XZY5K2YMQWF4EULQGX6RRYWG6SWDWWAFMQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK, address:104.36.82.51:4075): protocol: write tcp 188.166.225.170:41162->104.36.82.51:4075: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451086179 (20h18m52.927024644s) [noise=true , address=104.36.82.54:4175 , piece_id=5PAE3BM5AFSLZ5TVUQC3QK3RVWCVGFDLOK452BDPWQ7ARTL7EYWQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:12ou7jwFyJP3mNoALddniUTDT9duVyQKZjeq4xHCyx6sRWpwEPJ, address:104.36.82.54:4175): protocol: rpc: dial tcp 104.36.82.54:4175: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451120024 (20h18m52.927058489s) [noise=true , address=104.36.82.13:4214 , piece_id=SLX3YJCEWCM3553VZHILH337FKSPPFNKVFTPXO4KIBOS6IXCLXHA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:1nJ8uqwxf6x4hX52AjAdCbMKi1BKGGbMvrQbiazJdLpn6j69ay, address:104.36.82.13:4214): protocol: rpc: dial tcp 104.36.82.13:4214: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451147116 (20h18m52.927085581s) [noise=true , address=104.36.82.15:4263 , piece_id=3GCLYN4UBCLM5BPLXDQT5SSWPJBTKIOZM4UIUNO3OZIXW6OANUBQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:1f97yBEvcrYHqMq7iovVoju77jzhUYgoQCsUGULeTagugFXrCu, address:104.36.82.15:4263): protocol: write tcp 188.166.225.170:39380->104.36.82.15:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451167805 (20h18m52.927106271s) [seg_pos={0 0}, upload=19386]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 29 13:04:40.696628655 (20h18m53.17256714s) [upload=19386]: Done uploading segments... Jul 29 13:04:40.696658391 (20h18m53.172596856s) [upload=19386]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.798745048s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 1.833802471s FAIL (2025-07-29 13:46:20 UTC) orbiter warning: uplink: too many requests Duration: 1.833827428s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [X..XX.................................X.............X.......] duration: 5.838463401s FAIL (2025-07-29 13:45:57 UTC) orbiter warning: digest mismatch: after ee3c39c295e32d2453f46eedbd5a023ecb901c93b585366a6221da2ecebc35b3 != before f1d4b9d3eca6a9d63854d84cb67ac0fa26d296bc93761368409da19470c1a42f Duration: 5.838487026s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [....X.X.....................................................] duration: 5.906451826s FAIL (2025-07-29 13:45:49 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ding segments... Jul 29 13:03:41.091320246 (20h17m53.567258701s) [upload=19386]: Got next segment. Inline: false Jul 29 13:03:41.091349191 (20h17m53.567287656s) [upload=19386]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 29 13:03:41.091354331 (20h17m53.567292796s) [upload=19386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 29 13:03:41.616444845 (20h17m54.09238331s) [seg_pos={0 0}, upload=19386]: Begin upload segment... Jul 29 13:03:41.616487748 (20h17m54.092426213s) [seg_pos={0 0}, upload=19386]: Done begin upload segment. Jul 29 13:03:41.616507376 (20h17m54.092445841s) [noise=true , address=104.36.82.15:4263 , piece_id=3GCLYN4UBCLM5BPLXDQT5SSWPJBTKIOZM4UIUNO3OZIXW6OANUBQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.616564315 (20h17m54.09250278s) [noise=true , address=104.36.82.11:4119 , piece_id=6KWLABSUMX6YVKD6LGKR6ONLQKFFTSTKFHHRVFIN7U4KDALBNCBQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.616866433 (20h17m54.092804888s) [noise=true , address=104.36.82.54:4175 , piece_id=5PAE3BM5AFSLZ5TVUQC3QK3RVWCVGFDLOK452BDPWQ7ARTL7EYWQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617072557 (20h17m54.093011022s) [noise=true , address=104.36.82.56:4242 , piece_id=BHKC5MH4G65BK2YLJJANIYZ4MLAUTMBG32XSRISARHCO26AZUXRA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617110629 (20h17m54.093049084s) [noise=true , address=104.36.82.5:4866 , piece_id=EFRLC4LA232NZZJU3GDSP2EEBZO3W2LGJRD3L2A6SPSRBTLUTIFA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617355037 (20h17m54.093293492s) [noise=true , address=104.36.82.61:4391 , piece_id=TI5YDGDSTWKDX5XAWL36F3TJDQNHKNTREZVTUGIS2EVZYP6DWSDA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617481549 (20h17m54.093420024s) [noise=true , address=104.36.82.20:4448 , piece_id=CSHYRDGTS4ROFDTKETIZ36TWJE3OEMAKLB4RGFAU557G725EANXQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617583044 (20h17m54.093521509s) [noise=true , address=104.36.82.47:4545 , piece_id=PZWVIOWLBZ72IOFRJHPAD4OZZPA2G4XYCDLGYEY7THSOR7XDHMUQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617828073 (20h17m54.093766528s) [noise=true , address=104.36.82.42:4802 , piece_id=6HORHYYKIKDX4IWWKUQPVGZQZ2YYIGC47OCZHRVBC4BZFRJX5TQQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617987597 (20h17m54.093926062s) [noise=true , address=104.36.82.13:4214 , piece_id=SLX3YJCEWCM3553VZHILH337FKSPPFNKVFTPXO4KIBOS6IXCLXHA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618055948 (20h17m54.093994403s) [noise=true , address=104.36.82.46:4500 , piece_id=VCYKKDSLEKTTL3KO2FJ4VA5JGIVPA4GFBBHH65HPFZTP4REUTHYQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618285828 (20h17m54.094224293s) [noise=true , address=104.36.82.52:4092 , piece_id=BMJCAQYXJCUVJTEXH52UXYLD4MHWLDGQTJIPDOFRF4LPLTT36ESA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618485871 (20h17m54.094424336s) [noise=true , address=104.36.82.10:4088 , piece_id=6IV2Z7FXDNEOBJMNPRG7WSPZWZTJ5CGDF6JHBSF7WCIIL2QQWFUA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618663881 (20h17m54.094602346s) [noise=true , address=104.36.82.51:4075 , piece_id=P5NU32M3V7AMYFZVJ6XZY5K2YMQWF4EULQGX6RRYWG6SWDWWAFMQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618840449 (20h17m54.094778914s) [noise=true , address=104.36.82.14:4227 , piece_id=O2FXJBOV3CJCKLZHRPBTLXX2QRT2OWJQ5HPNHZI5IVIFDVUUICVQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.619028879 (20h17m54.094967344s) [noise=true , address=104.36.82.9:4028 , piece_id=XCRNGUPBFYCZOYT23L6NHJ6WQR55TQMTWZF4SNKSRLQGC5TYO3LA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.619215837 (20h17m54.095154302s) [noise=true , address=104.36.82.53:4126 , piece_id=GZ5Z7AGMYVGYVUJAWFAIS6SQLRHFB772RLQY736QUGOXA5OWQ34Q, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.619408406 (20h17m54.095346871s) [noise=true , address=104.36.82.55:4219 , piece_id=IURHWONYUYYJWIL5BDF6HLCEPA76O6QAZAJ22NCI6LQYIGO6K4IA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:43.299790904 (20h17m55.775729369s) [upload=19386]: Next returned nil segment Jul 29 13:03:43.299806024 (20h17m55.775744479s) [upload=19386]: Waiting for error group managing segments... Jul 29 13:03:43.847040928 (20h17m56.322979393s) [noise=true , address=104.36.82.9:4028 , piece_id=XCRNGUPBFYCZOYT23L6NHJ6WQR55TQMTWZF4SNKSRLQGC5TYO3LA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:43.879495404 (20h17m56.355433869s) [noise=true , address=104.36.82.61:4391 , piece_id=TI5YDGDSTWKDX5XAWL36F3TJDQNHKNTREZVTUGIS2EVZYP6DWSDA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:46.350727192 (20h17m58.826665657s) [noise=true , address=104.36.82.5:4866 , piece_id=EFRLC4LA232NZZJU3GDSP2EEBZO3W2LGJRD3L2A6SPSRBTLUTIFA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:51.702944756 (20h18m4.178883221s) [noise=true , address=104.36.82.42:4802 , piece_id=6HORHYYKIKDX4IWWKUQPVGZQZ2YYIGC47OCZHRVBC4BZFRJX5TQQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:55.250827413 (20h18m7.726765878s) [noise=true , address=104.36.82.55:4219 , piece_id=IURHWONYUYYJWIL5BDF6HLCEPA76O6QAZAJ22NCI6LQYIGO6K4IA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:57.858469692 (20h18m10.334408157s) [noise=true , address=104.36.82.53:4126 , piece_id=GZ5Z7AGMYVGYVUJAWFAIS6SQLRHFB772RLQY736QUGOXA5OWQ34Q, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:59.265385056 (20h18m11.741323511s) [noise=true , address=104.36.82.10:4088 , piece_id=6IV2Z7FXDNEOBJMNPRG7WSPZWZTJ5CGDF6JHBSF7WCIIL2QQWFUA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:59.973580263 (20h18m12.449518729s) [noise=true , address=104.36.82.11:4119 , piece_id=6KWLABSUMX6YVKD6LGKR6ONLQKFFTSTKFHHRVFIN7U4KDALBNCBQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:16.002200978 (20h18m28.478139453s) [noise=true , address=104.36.82.52:4092 , piece_id=BMJCAQYXJCUVJTEXH52UXYLD4MHWLDGQTJIPDOFRF4LPLTT36ESA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:30.906580908 (20h18m43.382519373s) [noise=true , address=104.36.82.20:4448 , piece_id=CSHYRDGTS4ROFDTKETIZ36TWJE3OEMAKLB4RGFAU557G725EANXQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:31.164952501 (20h18m43.640890966s) [noise=true , address=104.36.82.14:4227 , piece_id=O2FXJBOV3CJCKLZHRPBTLXX2QRT2OWJQ5HPNHZI5IVIFDVUUICVQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:33.000876204 (20h18m45.476814679s) [noise=true , address=104.36.82.56:4242 , piece_id=BHKC5MH4G65BK2YLJJANIYZ4MLAUTMBG32XSRISARHCO26AZUXRA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:34.725342452 (20h18m47.201280907s) [noise=true , address=104.36.82.47:4545 , piece_id=PZWVIOWLBZ72IOFRJHPAD4OZZPA2G4XYCDLGYEY7THSOR7XDHMUQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:40.451024021 (20h18m52.926962486s) [noise=true , address=104.36.82.46:4500 , piece_id=VCYKKDSLEKTTL3KO2FJ4VA5JGIVPA4GFBBHH65HPFZTP4REUTHYQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:12B7ox4ehqxJ2mwqeXj8C9N9GuSU5etNC23iT1pJnTgMLGBAGQK, address:104.36.82.46:4500): protocol: write tcp 188.166.225.170:46700->104.36.82.46:4500: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451060500 (20h18m52.926998965s) [noise=true , address=104.36.82.51:4075 , piece_id=P5NU32M3V7AMYFZVJ6XZY5K2YMQWF4EULQGX6RRYWG6SWDWWAFMQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK, address:104.36.82.51:4075): protocol: write tcp 188.166.225.170:41162->104.36.82.51:4075: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451086179 (20h18m52.927024644s) [noise=true , address=104.36.82.54:4175 , piece_id=5PAE3BM5AFSLZ5TVUQC3QK3RVWCVGFDLOK452BDPWQ7ARTL7EYWQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:12ou7jwFyJP3mNoALddniUTDT9duVyQKZjeq4xHCyx6sRWpwEPJ, address:104.36.82.54:4175): protocol: rpc: dial tcp 104.36.82.54:4175: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451120024 (20h18m52.927058489s) [noise=true , address=104.36.82.13:4214 , piece_id=SLX3YJCEWCM3553VZHILH337FKSPPFNKVFTPXO4KIBOS6IXCLXHA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:1nJ8uqwxf6x4hX52AjAdCbMKi1BKGGbMvrQbiazJdLpn6j69ay, address:104.36.82.13:4214): protocol: rpc: dial tcp 104.36.82.13:4214: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451147116 (20h18m52.927085581s) [noise=true , address=104.36.82.15:4263 , piece_id=3GCLYN4UBCLM5BPLXDQT5SSWPJBTKIOZM4UIUNO3OZIXW6OANUBQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:1f97yBEvcrYHqMq7iovVoju77jzhUYgoQCsUGULeTagugFXrCu, address:104.36.82.15:4263): protocol: write tcp 188.166.225.170:39380->104.36.82.15:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451167805 (20h18m52.927106271s) [seg_pos={0 0}, upload=19386]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 29 13:04:40.696628655 (20h18m53.17256714s) [upload=19386]: Done uploading segments... Jul 29 13:04:40.696658391 (20h18m53.172596856s) [upload=19386]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.906477524s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [.X..XX.................................X.............X......] duration: 5.792931723s FAIL (2025-07-29 13:45:32 UTC) orbiter warning: digest mismatch: after ee3c39c295e32d2453f46eedbd5a023ecb901c93b585366a6221da2ecebc35b3 != before f1d4b9d3eca6a9d63854d84cb67ac0fa26d296bc93761368409da19470c1a42f Duration: 5.792955108s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 1.815334824s FAIL (2025-07-29 13:45:27 UTC) orbiter warning: uplink: too many requests Duration: 1.815412742s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 1.849146166s FAIL (2025-07-29 13:45:03 UTC) orbiter warning: uplink: too many requests Duration: 1.849180913s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.....X.X....................................................] duration: 5.699062575s FAIL (2025-07-29 13:44:40 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ding segments... Jul 29 13:03:41.091320246 (20h17m53.567258701s) [upload=19386]: Got next segment. Inline: false Jul 29 13:03:41.091349191 (20h17m53.567287656s) [upload=19386]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 29 13:03:41.091354331 (20h17m53.567292796s) [upload=19386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 29 13:03:41.616444845 (20h17m54.09238331s) [seg_pos={0 0}, upload=19386]: Begin upload segment... Jul 29 13:03:41.616487748 (20h17m54.092426213s) [seg_pos={0 0}, upload=19386]: Done begin upload segment. Jul 29 13:03:41.616507376 (20h17m54.092445841s) [noise=true , address=104.36.82.15:4263 , piece_id=3GCLYN4UBCLM5BPLXDQT5SSWPJBTKIOZM4UIUNO3OZIXW6OANUBQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.616564315 (20h17m54.09250278s) [noise=true , address=104.36.82.11:4119 , piece_id=6KWLABSUMX6YVKD6LGKR6ONLQKFFTSTKFHHRVFIN7U4KDALBNCBQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.616866433 (20h17m54.092804888s) [noise=true , address=104.36.82.54:4175 , piece_id=5PAE3BM5AFSLZ5TVUQC3QK3RVWCVGFDLOK452BDPWQ7ARTL7EYWQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617072557 (20h17m54.093011022s) [noise=true , address=104.36.82.56:4242 , piece_id=BHKC5MH4G65BK2YLJJANIYZ4MLAUTMBG32XSRISARHCO26AZUXRA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617110629 (20h17m54.093049084s) [noise=true , address=104.36.82.5:4866 , piece_id=EFRLC4LA232NZZJU3GDSP2EEBZO3W2LGJRD3L2A6SPSRBTLUTIFA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617355037 (20h17m54.093293492s) [noise=true , address=104.36.82.61:4391 , piece_id=TI5YDGDSTWKDX5XAWL36F3TJDQNHKNTREZVTUGIS2EVZYP6DWSDA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617481549 (20h17m54.093420024s) [noise=true , address=104.36.82.20:4448 , piece_id=CSHYRDGTS4ROFDTKETIZ36TWJE3OEMAKLB4RGFAU557G725EANXQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617583044 (20h17m54.093521509s) [noise=true , address=104.36.82.47:4545 , piece_id=PZWVIOWLBZ72IOFRJHPAD4OZZPA2G4XYCDLGYEY7THSOR7XDHMUQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617828073 (20h17m54.093766528s) [noise=true , address=104.36.82.42:4802 , piece_id=6HORHYYKIKDX4IWWKUQPVGZQZ2YYIGC47OCZHRVBC4BZFRJX5TQQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617987597 (20h17m54.093926062s) [noise=true , address=104.36.82.13:4214 , piece_id=SLX3YJCEWCM3553VZHILH337FKSPPFNKVFTPXO4KIBOS6IXCLXHA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618055948 (20h17m54.093994403s) [noise=true , address=104.36.82.46:4500 , piece_id=VCYKKDSLEKTTL3KO2FJ4VA5JGIVPA4GFBBHH65HPFZTP4REUTHYQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618285828 (20h17m54.094224293s) [noise=true , address=104.36.82.52:4092 , piece_id=BMJCAQYXJCUVJTEXH52UXYLD4MHWLDGQTJIPDOFRF4LPLTT36ESA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618485871 (20h17m54.094424336s) [noise=true , address=104.36.82.10:4088 , piece_id=6IV2Z7FXDNEOBJMNPRG7WSPZWZTJ5CGDF6JHBSF7WCIIL2QQWFUA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618663881 (20h17m54.094602346s) [noise=true , address=104.36.82.51:4075 , piece_id=P5NU32M3V7AMYFZVJ6XZY5K2YMQWF4EULQGX6RRYWG6SWDWWAFMQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618840449 (20h17m54.094778914s) [noise=true , address=104.36.82.14:4227 , piece_id=O2FXJBOV3CJCKLZHRPBTLXX2QRT2OWJQ5HPNHZI5IVIFDVUUICVQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.619028879 (20h17m54.094967344s) [noise=true , address=104.36.82.9:4028 , piece_id=XCRNGUPBFYCZOYT23L6NHJ6WQR55TQMTWZF4SNKSRLQGC5TYO3LA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.619215837 (20h17m54.095154302s) [noise=true , address=104.36.82.53:4126 , piece_id=GZ5Z7AGMYVGYVUJAWFAIS6SQLRHFB772RLQY736QUGOXA5OWQ34Q, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.619408406 (20h17m54.095346871s) [noise=true , address=104.36.82.55:4219 , piece_id=IURHWONYUYYJWIL5BDF6HLCEPA76O6QAZAJ22NCI6LQYIGO6K4IA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:43.299790904 (20h17m55.775729369s) [upload=19386]: Next returned nil segment Jul 29 13:03:43.299806024 (20h17m55.775744479s) [upload=19386]: Waiting for error group managing segments... Jul 29 13:03:43.847040928 (20h17m56.322979393s) [noise=true , address=104.36.82.9:4028 , piece_id=XCRNGUPBFYCZOYT23L6NHJ6WQR55TQMTWZF4SNKSRLQGC5TYO3LA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:43.879495404 (20h17m56.355433869s) [noise=true , address=104.36.82.61:4391 , piece_id=TI5YDGDSTWKDX5XAWL36F3TJDQNHKNTREZVTUGIS2EVZYP6DWSDA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:46.350727192 (20h17m58.826665657s) [noise=true , address=104.36.82.5:4866 , piece_id=EFRLC4LA232NZZJU3GDSP2EEBZO3W2LGJRD3L2A6SPSRBTLUTIFA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:51.702944756 (20h18m4.178883221s) [noise=true , address=104.36.82.42:4802 , piece_id=6HORHYYKIKDX4IWWKUQPVGZQZ2YYIGC47OCZHRVBC4BZFRJX5TQQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:55.250827413 (20h18m7.726765878s) [noise=true , address=104.36.82.55:4219 , piece_id=IURHWONYUYYJWIL5BDF6HLCEPA76O6QAZAJ22NCI6LQYIGO6K4IA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:57.858469692 (20h18m10.334408157s) [noise=true , address=104.36.82.53:4126 , piece_id=GZ5Z7AGMYVGYVUJAWFAIS6SQLRHFB772RLQY736QUGOXA5OWQ34Q, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:59.265385056 (20h18m11.741323511s) [noise=true , address=104.36.82.10:4088 , piece_id=6IV2Z7FXDNEOBJMNPRG7WSPZWZTJ5CGDF6JHBSF7WCIIL2QQWFUA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:59.973580263 (20h18m12.449518729s) [noise=true , address=104.36.82.11:4119 , piece_id=6KWLABSUMX6YVKD6LGKR6ONLQKFFTSTKFHHRVFIN7U4KDALBNCBQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:16.002200978 (20h18m28.478139453s) [noise=true , address=104.36.82.52:4092 , piece_id=BMJCAQYXJCUVJTEXH52UXYLD4MHWLDGQTJIPDOFRF4LPLTT36ESA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:30.906580908 (20h18m43.382519373s) [noise=true , address=104.36.82.20:4448 , piece_id=CSHYRDGTS4ROFDTKETIZ36TWJE3OEMAKLB4RGFAU557G725EANXQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:31.164952501 (20h18m43.640890966s) [noise=true , address=104.36.82.14:4227 , piece_id=O2FXJBOV3CJCKLZHRPBTLXX2QRT2OWJQ5HPNHZI5IVIFDVUUICVQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:33.000876204 (20h18m45.476814679s) [noise=true , address=104.36.82.56:4242 , piece_id=BHKC5MH4G65BK2YLJJANIYZ4MLAUTMBG32XSRISARHCO26AZUXRA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:34.725342452 (20h18m47.201280907s) [noise=true , address=104.36.82.47:4545 , piece_id=PZWVIOWLBZ72IOFRJHPAD4OZZPA2G4XYCDLGYEY7THSOR7XDHMUQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:40.451024021 (20h18m52.926962486s) [noise=true , address=104.36.82.46:4500 , piece_id=VCYKKDSLEKTTL3KO2FJ4VA5JGIVPA4GFBBHH65HPFZTP4REUTHYQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:12B7ox4ehqxJ2mwqeXj8C9N9GuSU5etNC23iT1pJnTgMLGBAGQK, address:104.36.82.46:4500): protocol: write tcp 188.166.225.170:46700->104.36.82.46:4500: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451060500 (20h18m52.926998965s) [noise=true , address=104.36.82.51:4075 , piece_id=P5NU32M3V7AMYFZVJ6XZY5K2YMQWF4EULQGX6RRYWG6SWDWWAFMQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK, address:104.36.82.51:4075): protocol: write tcp 188.166.225.170:41162->104.36.82.51:4075: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451086179 (20h18m52.927024644s) [noise=true , address=104.36.82.54:4175 , piece_id=5PAE3BM5AFSLZ5TVUQC3QK3RVWCVGFDLOK452BDPWQ7ARTL7EYWQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:12ou7jwFyJP3mNoALddniUTDT9duVyQKZjeq4xHCyx6sRWpwEPJ, address:104.36.82.54:4175): protocol: rpc: dial tcp 104.36.82.54:4175: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451120024 (20h18m52.927058489s) [noise=true , address=104.36.82.13:4214 , piece_id=SLX3YJCEWCM3553VZHILH337FKSPPFNKVFTPXO4KIBOS6IXCLXHA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:1nJ8uqwxf6x4hX52AjAdCbMKi1BKGGbMvrQbiazJdLpn6j69ay, address:104.36.82.13:4214): protocol: rpc: dial tcp 104.36.82.13:4214: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451147116 (20h18m52.927085581s) [noise=true , address=104.36.82.15:4263 , piece_id=3GCLYN4UBCLM5BPLXDQT5SSWPJBTKIOZM4UIUNO3OZIXW6OANUBQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:1f97yBEvcrYHqMq7iovVoju77jzhUYgoQCsUGULeTagugFXrCu, address:104.36.82.15:4263): protocol: write tcp 188.166.225.170:39380->104.36.82.15:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451167805 (20h18m52.927106271s) [seg_pos={0 0}, upload=19386]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 29 13:04:40.696628655 (20h18m53.17256714s) [upload=19386]: Done uploading segments... Jul 29 13:04:40.696658391 (20h18m53.172596856s) [upload=19386]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.699090457s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [..X..XX.................................X.............X.....] duration: 5.811800079s FAIL (2025-07-29 13:44:18 UTC) orbiter warning: digest mismatch: after ee3c39c295e32d2453f46eedbd5a023ecb901c93b585366a6221da2ecebc35b3 != before f1d4b9d3eca6a9d63854d84cb67ac0fa26d296bc93761368409da19470c1a42f Duration: 5.811830948s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [......X.X...................................................] duration: 6.049637364s FAIL (2025-07-29 13:44:13 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ding segments... Jul 29 13:03:41.091320246 (20h17m53.567258701s) [upload=19386]: Got next segment. Inline: false Jul 29 13:03:41.091349191 (20h17m53.567287656s) [upload=19386]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 29 13:03:41.091354331 (20h17m53.567292796s) [upload=19386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 29 13:03:41.616444845 (20h17m54.09238331s) [seg_pos={0 0}, upload=19386]: Begin upload segment... Jul 29 13:03:41.616487748 (20h17m54.092426213s) [seg_pos={0 0}, upload=19386]: Done begin upload segment. Jul 29 13:03:41.616507376 (20h17m54.092445841s) [noise=true , address=104.36.82.15:4263 , piece_id=3GCLYN4UBCLM5BPLXDQT5SSWPJBTKIOZM4UIUNO3OZIXW6OANUBQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.616564315 (20h17m54.09250278s) [noise=true , address=104.36.82.11:4119 , piece_id=6KWLABSUMX6YVKD6LGKR6ONLQKFFTSTKFHHRVFIN7U4KDALBNCBQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.616866433 (20h17m54.092804888s) [noise=true , address=104.36.82.54:4175 , piece_id=5PAE3BM5AFSLZ5TVUQC3QK3RVWCVGFDLOK452BDPWQ7ARTL7EYWQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617072557 (20h17m54.093011022s) [noise=true , address=104.36.82.56:4242 , piece_id=BHKC5MH4G65BK2YLJJANIYZ4MLAUTMBG32XSRISARHCO26AZUXRA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617110629 (20h17m54.093049084s) [noise=true , address=104.36.82.5:4866 , piece_id=EFRLC4LA232NZZJU3GDSP2EEBZO3W2LGJRD3L2A6SPSRBTLUTIFA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617355037 (20h17m54.093293492s) [noise=true , address=104.36.82.61:4391 , piece_id=TI5YDGDSTWKDX5XAWL36F3TJDQNHKNTREZVTUGIS2EVZYP6DWSDA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617481549 (20h17m54.093420024s) [noise=true , address=104.36.82.20:4448 , piece_id=CSHYRDGTS4ROFDTKETIZ36TWJE3OEMAKLB4RGFAU557G725EANXQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617583044 (20h17m54.093521509s) [noise=true , address=104.36.82.47:4545 , piece_id=PZWVIOWLBZ72IOFRJHPAD4OZZPA2G4XYCDLGYEY7THSOR7XDHMUQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617828073 (20h17m54.093766528s) [noise=true , address=104.36.82.42:4802 , piece_id=6HORHYYKIKDX4IWWKUQPVGZQZ2YYIGC47OCZHRVBC4BZFRJX5TQQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617987597 (20h17m54.093926062s) [noise=true , address=104.36.82.13:4214 , piece_id=SLX3YJCEWCM3553VZHILH337FKSPPFNKVFTPXO4KIBOS6IXCLXHA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618055948 (20h17m54.093994403s) [noise=true , address=104.36.82.46:4500 , piece_id=VCYKKDSLEKTTL3KO2FJ4VA5JGIVPA4GFBBHH65HPFZTP4REUTHYQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618285828 (20h17m54.094224293s) [noise=true , address=104.36.82.52:4092 , piece_id=BMJCAQYXJCUVJTEXH52UXYLD4MHWLDGQTJIPDOFRF4LPLTT36ESA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618485871 (20h17m54.094424336s) [noise=true , address=104.36.82.10:4088 , piece_id=6IV2Z7FXDNEOBJMNPRG7WSPZWZTJ5CGDF6JHBSF7WCIIL2QQWFUA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618663881 (20h17m54.094602346s) [noise=true , address=104.36.82.51:4075 , piece_id=P5NU32M3V7AMYFZVJ6XZY5K2YMQWF4EULQGX6RRYWG6SWDWWAFMQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618840449 (20h17m54.094778914s) [noise=true , address=104.36.82.14:4227 , piece_id=O2FXJBOV3CJCKLZHRPBTLXX2QRT2OWJQ5HPNHZI5IVIFDVUUICVQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.619028879 (20h17m54.094967344s) [noise=true , address=104.36.82.9:4028 , piece_id=XCRNGUPBFYCZOYT23L6NHJ6WQR55TQMTWZF4SNKSRLQGC5TYO3LA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.619215837 (20h17m54.095154302s) [noise=true , address=104.36.82.53:4126 , piece_id=GZ5Z7AGMYVGYVUJAWFAIS6SQLRHFB772RLQY736QUGOXA5OWQ34Q, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.619408406 (20h17m54.095346871s) [noise=true , address=104.36.82.55:4219 , piece_id=IURHWONYUYYJWIL5BDF6HLCEPA76O6QAZAJ22NCI6LQYIGO6K4IA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:43.299790904 (20h17m55.775729369s) [upload=19386]: Next returned nil segment Jul 29 13:03:43.299806024 (20h17m55.775744479s) [upload=19386]: Waiting for error group managing segments... Jul 29 13:03:43.847040928 (20h17m56.322979393s) [noise=true , address=104.36.82.9:4028 , piece_id=XCRNGUPBFYCZOYT23L6NHJ6WQR55TQMTWZF4SNKSRLQGC5TYO3LA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:43.879495404 (20h17m56.355433869s) [noise=true , address=104.36.82.61:4391 , piece_id=TI5YDGDSTWKDX5XAWL36F3TJDQNHKNTREZVTUGIS2EVZYP6DWSDA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:46.350727192 (20h17m58.826665657s) [noise=true , address=104.36.82.5:4866 , piece_id=EFRLC4LA232NZZJU3GDSP2EEBZO3W2LGJRD3L2A6SPSRBTLUTIFA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:51.702944756 (20h18m4.178883221s) [noise=true , address=104.36.82.42:4802 , piece_id=6HORHYYKIKDX4IWWKUQPVGZQZ2YYIGC47OCZHRVBC4BZFRJX5TQQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:55.250827413 (20h18m7.726765878s) [noise=true , address=104.36.82.55:4219 , piece_id=IURHWONYUYYJWIL5BDF6HLCEPA76O6QAZAJ22NCI6LQYIGO6K4IA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:57.858469692 (20h18m10.334408157s) [noise=true , address=104.36.82.53:4126 , piece_id=GZ5Z7AGMYVGYVUJAWFAIS6SQLRHFB772RLQY736QUGOXA5OWQ34Q, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:59.265385056 (20h18m11.741323511s) [noise=true , address=104.36.82.10:4088 , piece_id=6IV2Z7FXDNEOBJMNPRG7WSPZWZTJ5CGDF6JHBSF7WCIIL2QQWFUA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:59.973580263 (20h18m12.449518729s) [noise=true , address=104.36.82.11:4119 , piece_id=6KWLABSUMX6YVKD6LGKR6ONLQKFFTSTKFHHRVFIN7U4KDALBNCBQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:16.002200978 (20h18m28.478139453s) [noise=true , address=104.36.82.52:4092 , piece_id=BMJCAQYXJCUVJTEXH52UXYLD4MHWLDGQTJIPDOFRF4LPLTT36ESA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:30.906580908 (20h18m43.382519373s) [noise=true , address=104.36.82.20:4448 , piece_id=CSHYRDGTS4ROFDTKETIZ36TWJE3OEMAKLB4RGFAU557G725EANXQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:31.164952501 (20h18m43.640890966s) [noise=true , address=104.36.82.14:4227 , piece_id=O2FXJBOV3CJCKLZHRPBTLXX2QRT2OWJQ5HPNHZI5IVIFDVUUICVQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:33.000876204 (20h18m45.476814679s) [noise=true , address=104.36.82.56:4242 , piece_id=BHKC5MH4G65BK2YLJJANIYZ4MLAUTMBG32XSRISARHCO26AZUXRA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:34.725342452 (20h18m47.201280907s) [noise=true , address=104.36.82.47:4545 , piece_id=PZWVIOWLBZ72IOFRJHPAD4OZZPA2G4XYCDLGYEY7THSOR7XDHMUQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:40.451024021 (20h18m52.926962486s) [noise=true , address=104.36.82.46:4500 , piece_id=VCYKKDSLEKTTL3KO2FJ4VA5JGIVPA4GFBBHH65HPFZTP4REUTHYQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:12B7ox4ehqxJ2mwqeXj8C9N9GuSU5etNC23iT1pJnTgMLGBAGQK, address:104.36.82.46:4500): protocol: write tcp 188.166.225.170:46700->104.36.82.46:4500: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451060500 (20h18m52.926998965s) [noise=true , address=104.36.82.51:4075 , piece_id=P5NU32M3V7AMYFZVJ6XZY5K2YMQWF4EULQGX6RRYWG6SWDWWAFMQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK, address:104.36.82.51:4075): protocol: write tcp 188.166.225.170:41162->104.36.82.51:4075: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451086179 (20h18m52.927024644s) [noise=true , address=104.36.82.54:4175 , piece_id=5PAE3BM5AFSLZ5TVUQC3QK3RVWCVGFDLOK452BDPWQ7ARTL7EYWQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:12ou7jwFyJP3mNoALddniUTDT9duVyQKZjeq4xHCyx6sRWpwEPJ, address:104.36.82.54:4175): protocol: rpc: dial tcp 104.36.82.54:4175: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451120024 (20h18m52.927058489s) [noise=true , address=104.36.82.13:4214 , piece_id=SLX3YJCEWCM3553VZHILH337FKSPPFNKVFTPXO4KIBOS6IXCLXHA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:1nJ8uqwxf6x4hX52AjAdCbMKi1BKGGbMvrQbiazJdLpn6j69ay, address:104.36.82.13:4214): protocol: rpc: dial tcp 104.36.82.13:4214: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451147116 (20h18m52.927085581s) [noise=true , address=104.36.82.15:4263 , piece_id=3GCLYN4UBCLM5BPLXDQT5SSWPJBTKIOZM4UIUNO3OZIXW6OANUBQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:1f97yBEvcrYHqMq7iovVoju77jzhUYgoQCsUGULeTagugFXrCu, address:104.36.82.15:4263): protocol: write tcp 188.166.225.170:39380->104.36.82.15:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451167805 (20h18m52.927106271s) [seg_pos={0 0}, upload=19386]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 29 13:04:40.696628655 (20h18m53.17256714s) [upload=19386]: Done uploading segments... Jul 29 13:04:40.696658391 (20h18m53.172596856s) [upload=19386]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.049663514s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 856.368997ms FAIL (2025-07-29 13:44:07 UTC) orbiter warning: uplink: too many requests Duration: 856.398484ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [...X..XX.................................X.............X....] duration: 5.635494929s FAIL (2025-07-29 13:43:32 UTC) orbiter warning: digest mismatch: after ee3c39c295e32d2453f46eedbd5a023ecb901c93b585366a6221da2ecebc35b3 != before f1d4b9d3eca6a9d63854d84cb67ac0fa26d296bc93761368409da19470c1a42f Duration: 5.635516981s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.......X.X..................................................] duration: 5.916074404s FAIL (2025-07-29 13:43:17 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ding segments... Jul 29 13:03:41.091320246 (20h17m53.567258701s) [upload=19386]: Got next segment. Inline: false Jul 29 13:03:41.091349191 (20h17m53.567287656s) [upload=19386]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 29 13:03:41.091354331 (20h17m53.567292796s) [upload=19386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 29 13:03:41.616444845 (20h17m54.09238331s) [seg_pos={0 0}, upload=19386]: Begin upload segment... Jul 29 13:03:41.616487748 (20h17m54.092426213s) [seg_pos={0 0}, upload=19386]: Done begin upload segment. Jul 29 13:03:41.616507376 (20h17m54.092445841s) [noise=true , address=104.36.82.15:4263 , piece_id=3GCLYN4UBCLM5BPLXDQT5SSWPJBTKIOZM4UIUNO3OZIXW6OANUBQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.616564315 (20h17m54.09250278s) [noise=true , address=104.36.82.11:4119 , piece_id=6KWLABSUMX6YVKD6LGKR6ONLQKFFTSTKFHHRVFIN7U4KDALBNCBQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.616866433 (20h17m54.092804888s) [noise=true , address=104.36.82.54:4175 , piece_id=5PAE3BM5AFSLZ5TVUQC3QK3RVWCVGFDLOK452BDPWQ7ARTL7EYWQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617072557 (20h17m54.093011022s) [noise=true , address=104.36.82.56:4242 , piece_id=BHKC5MH4G65BK2YLJJANIYZ4MLAUTMBG32XSRISARHCO26AZUXRA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617110629 (20h17m54.093049084s) [noise=true , address=104.36.82.5:4866 , piece_id=EFRLC4LA232NZZJU3GDSP2EEBZO3W2LGJRD3L2A6SPSRBTLUTIFA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617355037 (20h17m54.093293492s) [noise=true , address=104.36.82.61:4391 , piece_id=TI5YDGDSTWKDX5XAWL36F3TJDQNHKNTREZVTUGIS2EVZYP6DWSDA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617481549 (20h17m54.093420024s) [noise=true , address=104.36.82.20:4448 , piece_id=CSHYRDGTS4ROFDTKETIZ36TWJE3OEMAKLB4RGFAU557G725EANXQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617583044 (20h17m54.093521509s) [noise=true , address=104.36.82.47:4545 , piece_id=PZWVIOWLBZ72IOFRJHPAD4OZZPA2G4XYCDLGYEY7THSOR7XDHMUQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617828073 (20h17m54.093766528s) [noise=true , address=104.36.82.42:4802 , piece_id=6HORHYYKIKDX4IWWKUQPVGZQZ2YYIGC47OCZHRVBC4BZFRJX5TQQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617987597 (20h17m54.093926062s) [noise=true , address=104.36.82.13:4214 , piece_id=SLX3YJCEWCM3553VZHILH337FKSPPFNKVFTPXO4KIBOS6IXCLXHA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618055948 (20h17m54.093994403s) [noise=true , address=104.36.82.46:4500 , piece_id=VCYKKDSLEKTTL3KO2FJ4VA5JGIVPA4GFBBHH65HPFZTP4REUTHYQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618285828 (20h17m54.094224293s) [noise=true , address=104.36.82.52:4092 , piece_id=BMJCAQYXJCUVJTEXH52UXYLD4MHWLDGQTJIPDOFRF4LPLTT36ESA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618485871 (20h17m54.094424336s) [noise=true , address=104.36.82.10:4088 , piece_id=6IV2Z7FXDNEOBJMNPRG7WSPZWZTJ5CGDF6JHBSF7WCIIL2QQWFUA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618663881 (20h17m54.094602346s) [noise=true , address=104.36.82.51:4075 , piece_id=P5NU32M3V7AMYFZVJ6XZY5K2YMQWF4EULQGX6RRYWG6SWDWWAFMQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618840449 (20h17m54.094778914s) [noise=true , address=104.36.82.14:4227 , piece_id=O2FXJBOV3CJCKLZHRPBTLXX2QRT2OWJQ5HPNHZI5IVIFDVUUICVQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.619028879 (20h17m54.094967344s) [noise=true , address=104.36.82.9:4028 , piece_id=XCRNGUPBFYCZOYT23L6NHJ6WQR55TQMTWZF4SNKSRLQGC5TYO3LA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.619215837 (20h17m54.095154302s) [noise=true , address=104.36.82.53:4126 , piece_id=GZ5Z7AGMYVGYVUJAWFAIS6SQLRHFB772RLQY736QUGOXA5OWQ34Q, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.619408406 (20h17m54.095346871s) [noise=true , address=104.36.82.55:4219 , piece_id=IURHWONYUYYJWIL5BDF6HLCEPA76O6QAZAJ22NCI6LQYIGO6K4IA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:43.299790904 (20h17m55.775729369s) [upload=19386]: Next returned nil segment Jul 29 13:03:43.299806024 (20h17m55.775744479s) [upload=19386]: Waiting for error group managing segments... Jul 29 13:03:43.847040928 (20h17m56.322979393s) [noise=true , address=104.36.82.9:4028 , piece_id=XCRNGUPBFYCZOYT23L6NHJ6WQR55TQMTWZF4SNKSRLQGC5TYO3LA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:43.879495404 (20h17m56.355433869s) [noise=true , address=104.36.82.61:4391 , piece_id=TI5YDGDSTWKDX5XAWL36F3TJDQNHKNTREZVTUGIS2EVZYP6DWSDA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:46.350727192 (20h17m58.826665657s) [noise=true , address=104.36.82.5:4866 , piece_id=EFRLC4LA232NZZJU3GDSP2EEBZO3W2LGJRD3L2A6SPSRBTLUTIFA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:51.702944756 (20h18m4.178883221s) [noise=true , address=104.36.82.42:4802 , piece_id=6HORHYYKIKDX4IWWKUQPVGZQZ2YYIGC47OCZHRVBC4BZFRJX5TQQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:55.250827413 (20h18m7.726765878s) [noise=true , address=104.36.82.55:4219 , piece_id=IURHWONYUYYJWIL5BDF6HLCEPA76O6QAZAJ22NCI6LQYIGO6K4IA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:57.858469692 (20h18m10.334408157s) [noise=true , address=104.36.82.53:4126 , piece_id=GZ5Z7AGMYVGYVUJAWFAIS6SQLRHFB772RLQY736QUGOXA5OWQ34Q, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:59.265385056 (20h18m11.741323511s) [noise=true , address=104.36.82.10:4088 , piece_id=6IV2Z7FXDNEOBJMNPRG7WSPZWZTJ5CGDF6JHBSF7WCIIL2QQWFUA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:59.973580263 (20h18m12.449518729s) [noise=true , address=104.36.82.11:4119 , piece_id=6KWLABSUMX6YVKD6LGKR6ONLQKFFTSTKFHHRVFIN7U4KDALBNCBQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:16.002200978 (20h18m28.478139453s) [noise=true , address=104.36.82.52:4092 , piece_id=BMJCAQYXJCUVJTEXH52UXYLD4MHWLDGQTJIPDOFRF4LPLTT36ESA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:30.906580908 (20h18m43.382519373s) [noise=true , address=104.36.82.20:4448 , piece_id=CSHYRDGTS4ROFDTKETIZ36TWJE3OEMAKLB4RGFAU557G725EANXQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:31.164952501 (20h18m43.640890966s) [noise=true , address=104.36.82.14:4227 , piece_id=O2FXJBOV3CJCKLZHRPBTLXX2QRT2OWJQ5HPNHZI5IVIFDVUUICVQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:33.000876204 (20h18m45.476814679s) [noise=true , address=104.36.82.56:4242 , piece_id=BHKC5MH4G65BK2YLJJANIYZ4MLAUTMBG32XSRISARHCO26AZUXRA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:34.725342452 (20h18m47.201280907s) [noise=true , address=104.36.82.47:4545 , piece_id=PZWVIOWLBZ72IOFRJHPAD4OZZPA2G4XYCDLGYEY7THSOR7XDHMUQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:40.451024021 (20h18m52.926962486s) [noise=true , address=104.36.82.46:4500 , piece_id=VCYKKDSLEKTTL3KO2FJ4VA5JGIVPA4GFBBHH65HPFZTP4REUTHYQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:12B7ox4ehqxJ2mwqeXj8C9N9GuSU5etNC23iT1pJnTgMLGBAGQK, address:104.36.82.46:4500): protocol: write tcp 188.166.225.170:46700->104.36.82.46:4500: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451060500 (20h18m52.926998965s) [noise=true , address=104.36.82.51:4075 , piece_id=P5NU32M3V7AMYFZVJ6XZY5K2YMQWF4EULQGX6RRYWG6SWDWWAFMQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK, address:104.36.82.51:4075): protocol: write tcp 188.166.225.170:41162->104.36.82.51:4075: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451086179 (20h18m52.927024644s) [noise=true , address=104.36.82.54:4175 , piece_id=5PAE3BM5AFSLZ5TVUQC3QK3RVWCVGFDLOK452BDPWQ7ARTL7EYWQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:12ou7jwFyJP3mNoALddniUTDT9duVyQKZjeq4xHCyx6sRWpwEPJ, address:104.36.82.54:4175): protocol: rpc: dial tcp 104.36.82.54:4175: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451120024 (20h18m52.927058489s) [noise=true , address=104.36.82.13:4214 , piece_id=SLX3YJCEWCM3553VZHILH337FKSPPFNKVFTPXO4KIBOS6IXCLXHA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:1nJ8uqwxf6x4hX52AjAdCbMKi1BKGGbMvrQbiazJdLpn6j69ay, address:104.36.82.13:4214): protocol: rpc: dial tcp 104.36.82.13:4214: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451147116 (20h18m52.927085581s) [noise=true , address=104.36.82.15:4263 , piece_id=3GCLYN4UBCLM5BPLXDQT5SSWPJBTKIOZM4UIUNO3OZIXW6OANUBQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:1f97yBEvcrYHqMq7iovVoju77jzhUYgoQCsUGULeTagugFXrCu, address:104.36.82.15:4263): protocol: write tcp 188.166.225.170:39380->104.36.82.15:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451167805 (20h18m52.927106271s) [seg_pos={0 0}, upload=19386]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 29 13:04:40.696628655 (20h18m53.17256714s) [upload=19386]: Done uploading segments... Jul 29 13:04:40.696658391 (20h18m53.172596856s) [upload=19386]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.916101426s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [........X.X.................................................] duration: 5.943782566s FAIL (2025-07-29 13:42:30 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ding segments... Jul 29 13:03:41.091320246 (20h17m53.567258701s) [upload=19386]: Got next segment. Inline: false Jul 29 13:03:41.091349191 (20h17m53.567287656s) [upload=19386]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 29 13:03:41.091354331 (20h17m53.567292796s) [upload=19386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 29 13:03:41.616444845 (20h17m54.09238331s) [seg_pos={0 0}, upload=19386]: Begin upload segment... Jul 29 13:03:41.616487748 (20h17m54.092426213s) [seg_pos={0 0}, upload=19386]: Done begin upload segment. Jul 29 13:03:41.616507376 (20h17m54.092445841s) [noise=true , address=104.36.82.15:4263 , piece_id=3GCLYN4UBCLM5BPLXDQT5SSWPJBTKIOZM4UIUNO3OZIXW6OANUBQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.616564315 (20h17m54.09250278s) [noise=true , address=104.36.82.11:4119 , piece_id=6KWLABSUMX6YVKD6LGKR6ONLQKFFTSTKFHHRVFIN7U4KDALBNCBQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.616866433 (20h17m54.092804888s) [noise=true , address=104.36.82.54:4175 , piece_id=5PAE3BM5AFSLZ5TVUQC3QK3RVWCVGFDLOK452BDPWQ7ARTL7EYWQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617072557 (20h17m54.093011022s) [noise=true , address=104.36.82.56:4242 , piece_id=BHKC5MH4G65BK2YLJJANIYZ4MLAUTMBG32XSRISARHCO26AZUXRA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617110629 (20h17m54.093049084s) [noise=true , address=104.36.82.5:4866 , piece_id=EFRLC4LA232NZZJU3GDSP2EEBZO3W2LGJRD3L2A6SPSRBTLUTIFA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617355037 (20h17m54.093293492s) [noise=true , address=104.36.82.61:4391 , piece_id=TI5YDGDSTWKDX5XAWL36F3TJDQNHKNTREZVTUGIS2EVZYP6DWSDA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617481549 (20h17m54.093420024s) [noise=true , address=104.36.82.20:4448 , piece_id=CSHYRDGTS4ROFDTKETIZ36TWJE3OEMAKLB4RGFAU557G725EANXQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617583044 (20h17m54.093521509s) [noise=true , address=104.36.82.47:4545 , piece_id=PZWVIOWLBZ72IOFRJHPAD4OZZPA2G4XYCDLGYEY7THSOR7XDHMUQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617828073 (20h17m54.093766528s) [noise=true , address=104.36.82.42:4802 , piece_id=6HORHYYKIKDX4IWWKUQPVGZQZ2YYIGC47OCZHRVBC4BZFRJX5TQQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617987597 (20h17m54.093926062s) [noise=true , address=104.36.82.13:4214 , piece_id=SLX3YJCEWCM3553VZHILH337FKSPPFNKVFTPXO4KIBOS6IXCLXHA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618055948 (20h17m54.093994403s) [noise=true , address=104.36.82.46:4500 , piece_id=VCYKKDSLEKTTL3KO2FJ4VA5JGIVPA4GFBBHH65HPFZTP4REUTHYQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618285828 (20h17m54.094224293s) [noise=true , address=104.36.82.52:4092 , piece_id=BMJCAQYXJCUVJTEXH52UXYLD4MHWLDGQTJIPDOFRF4LPLTT36ESA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618485871 (20h17m54.094424336s) [noise=true , address=104.36.82.10:4088 , piece_id=6IV2Z7FXDNEOBJMNPRG7WSPZWZTJ5CGDF6JHBSF7WCIIL2QQWFUA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618663881 (20h17m54.094602346s) [noise=true , address=104.36.82.51:4075 , piece_id=P5NU32M3V7AMYFZVJ6XZY5K2YMQWF4EULQGX6RRYWG6SWDWWAFMQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618840449 (20h17m54.094778914s) [noise=true , address=104.36.82.14:4227 , piece_id=O2FXJBOV3CJCKLZHRPBTLXX2QRT2OWJQ5HPNHZI5IVIFDVUUICVQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.619028879 (20h17m54.094967344s) [noise=true , address=104.36.82.9:4028 , piece_id=XCRNGUPBFYCZOYT23L6NHJ6WQR55TQMTWZF4SNKSRLQGC5TYO3LA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.619215837 (20h17m54.095154302s) [noise=true , address=104.36.82.53:4126 , piece_id=GZ5Z7AGMYVGYVUJAWFAIS6SQLRHFB772RLQY736QUGOXA5OWQ34Q, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.619408406 (20h17m54.095346871s) [noise=true , address=104.36.82.55:4219 , piece_id=IURHWONYUYYJWIL5BDF6HLCEPA76O6QAZAJ22NCI6LQYIGO6K4IA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:43.299790904 (20h17m55.775729369s) [upload=19386]: Next returned nil segment Jul 29 13:03:43.299806024 (20h17m55.775744479s) [upload=19386]: Waiting for error group managing segments... Jul 29 13:03:43.847040928 (20h17m56.322979393s) [noise=true , address=104.36.82.9:4028 , piece_id=XCRNGUPBFYCZOYT23L6NHJ6WQR55TQMTWZF4SNKSRLQGC5TYO3LA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:43.879495404 (20h17m56.355433869s) [noise=true , address=104.36.82.61:4391 , piece_id=TI5YDGDSTWKDX5XAWL36F3TJDQNHKNTREZVTUGIS2EVZYP6DWSDA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:46.350727192 (20h17m58.826665657s) [noise=true , address=104.36.82.5:4866 , piece_id=EFRLC4LA232NZZJU3GDSP2EEBZO3W2LGJRD3L2A6SPSRBTLUTIFA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:51.702944756 (20h18m4.178883221s) [noise=true , address=104.36.82.42:4802 , piece_id=6HORHYYKIKDX4IWWKUQPVGZQZ2YYIGC47OCZHRVBC4BZFRJX5TQQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:55.250827413 (20h18m7.726765878s) [noise=true , address=104.36.82.55:4219 , piece_id=IURHWONYUYYJWIL5BDF6HLCEPA76O6QAZAJ22NCI6LQYIGO6K4IA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:57.858469692 (20h18m10.334408157s) [noise=true , address=104.36.82.53:4126 , piece_id=GZ5Z7AGMYVGYVUJAWFAIS6SQLRHFB772RLQY736QUGOXA5OWQ34Q, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:59.265385056 (20h18m11.741323511s) [noise=true , address=104.36.82.10:4088 , piece_id=6IV2Z7FXDNEOBJMNPRG7WSPZWZTJ5CGDF6JHBSF7WCIIL2QQWFUA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:59.973580263 (20h18m12.449518729s) [noise=true , address=104.36.82.11:4119 , piece_id=6KWLABSUMX6YVKD6LGKR6ONLQKFFTSTKFHHRVFIN7U4KDALBNCBQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:16.002200978 (20h18m28.478139453s) [noise=true , address=104.36.82.52:4092 , piece_id=BMJCAQYXJCUVJTEXH52UXYLD4MHWLDGQTJIPDOFRF4LPLTT36ESA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:30.906580908 (20h18m43.382519373s) [noise=true , address=104.36.82.20:4448 , piece_id=CSHYRDGTS4ROFDTKETIZ36TWJE3OEMAKLB4RGFAU557G725EANXQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:31.164952501 (20h18m43.640890966s) [noise=true , address=104.36.82.14:4227 , piece_id=O2FXJBOV3CJCKLZHRPBTLXX2QRT2OWJQ5HPNHZI5IVIFDVUUICVQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:33.000876204 (20h18m45.476814679s) [noise=true , address=104.36.82.56:4242 , piece_id=BHKC5MH4G65BK2YLJJANIYZ4MLAUTMBG32XSRISARHCO26AZUXRA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:34.725342452 (20h18m47.201280907s) [noise=true , address=104.36.82.47:4545 , piece_id=PZWVIOWLBZ72IOFRJHPAD4OZZPA2G4XYCDLGYEY7THSOR7XDHMUQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:40.451024021 (20h18m52.926962486s) [noise=true , address=104.36.82.46:4500 , piece_id=VCYKKDSLEKTTL3KO2FJ4VA5JGIVPA4GFBBHH65HPFZTP4REUTHYQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:12B7ox4ehqxJ2mwqeXj8C9N9GuSU5etNC23iT1pJnTgMLGBAGQK, address:104.36.82.46:4500): protocol: write tcp 188.166.225.170:46700->104.36.82.46:4500: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451060500 (20h18m52.926998965s) [noise=true , address=104.36.82.51:4075 , piece_id=P5NU32M3V7AMYFZVJ6XZY5K2YMQWF4EULQGX6RRYWG6SWDWWAFMQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK, address:104.36.82.51:4075): protocol: write tcp 188.166.225.170:41162->104.36.82.51:4075: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451086179 (20h18m52.927024644s) [noise=true , address=104.36.82.54:4175 , piece_id=5PAE3BM5AFSLZ5TVUQC3QK3RVWCVGFDLOK452BDPWQ7ARTL7EYWQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:12ou7jwFyJP3mNoALddniUTDT9duVyQKZjeq4xHCyx6sRWpwEPJ, address:104.36.82.54:4175): protocol: rpc: dial tcp 104.36.82.54:4175: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451120024 (20h18m52.927058489s) [noise=true , address=104.36.82.13:4214 , piece_id=SLX3YJCEWCM3553VZHILH337FKSPPFNKVFTPXO4KIBOS6IXCLXHA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:1nJ8uqwxf6x4hX52AjAdCbMKi1BKGGbMvrQbiazJdLpn6j69ay, address:104.36.82.13:4214): protocol: rpc: dial tcp 104.36.82.13:4214: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451147116 (20h18m52.927085581s) [noise=true , address=104.36.82.15:4263 , piece_id=3GCLYN4UBCLM5BPLXDQT5SSWPJBTKIOZM4UIUNO3OZIXW6OANUBQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:1f97yBEvcrYHqMq7iovVoju77jzhUYgoQCsUGULeTagugFXrCu, address:104.36.82.15:4263): protocol: write tcp 188.166.225.170:39380->104.36.82.15:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451167805 (20h18m52.927106271s) [seg_pos={0 0}, upload=19386]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 29 13:04:40.696628655 (20h18m53.17256714s) [upload=19386]: Done uploading segments... Jul 29 13:04:40.696658391 (20h18m53.172596856s) [upload=19386]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.943807243s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [X...X..XX.................................X.............X...] duration: 5.844894751s FAIL (2025-07-29 13:42:23 UTC) orbiter warning: digest mismatch: after ee3c39c295e32d2453f46eedbd5a023ecb901c93b585366a6221da2ecebc35b3 != before f1d4b9d3eca6a9d63854d84cb67ac0fa26d296bc93761368409da19470c1a42f Duration: 5.844919679s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.........X.X................................................] duration: 5.839863152s FAIL (2025-07-29 13:42:05 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ding segments... Jul 29 13:03:41.091320246 (20h17m53.567258701s) [upload=19386]: Got next segment. Inline: false Jul 29 13:03:41.091349191 (20h17m53.567287656s) [upload=19386]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 29 13:03:41.091354331 (20h17m53.567292796s) [upload=19386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 29 13:03:41.616444845 (20h17m54.09238331s) [seg_pos={0 0}, upload=19386]: Begin upload segment... Jul 29 13:03:41.616487748 (20h17m54.092426213s) [seg_pos={0 0}, upload=19386]: Done begin upload segment. Jul 29 13:03:41.616507376 (20h17m54.092445841s) [noise=true , address=104.36.82.15:4263 , piece_id=3GCLYN4UBCLM5BPLXDQT5SSWPJBTKIOZM4UIUNO3OZIXW6OANUBQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.616564315 (20h17m54.09250278s) [noise=true , address=104.36.82.11:4119 , piece_id=6KWLABSUMX6YVKD6LGKR6ONLQKFFTSTKFHHRVFIN7U4KDALBNCBQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.616866433 (20h17m54.092804888s) [noise=true , address=104.36.82.54:4175 , piece_id=5PAE3BM5AFSLZ5TVUQC3QK3RVWCVGFDLOK452BDPWQ7ARTL7EYWQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617072557 (20h17m54.093011022s) [noise=true , address=104.36.82.56:4242 , piece_id=BHKC5MH4G65BK2YLJJANIYZ4MLAUTMBG32XSRISARHCO26AZUXRA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617110629 (20h17m54.093049084s) [noise=true , address=104.36.82.5:4866 , piece_id=EFRLC4LA232NZZJU3GDSP2EEBZO3W2LGJRD3L2A6SPSRBTLUTIFA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617355037 (20h17m54.093293492s) [noise=true , address=104.36.82.61:4391 , piece_id=TI5YDGDSTWKDX5XAWL36F3TJDQNHKNTREZVTUGIS2EVZYP6DWSDA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617481549 (20h17m54.093420024s) [noise=true , address=104.36.82.20:4448 , piece_id=CSHYRDGTS4ROFDTKETIZ36TWJE3OEMAKLB4RGFAU557G725EANXQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617583044 (20h17m54.093521509s) [noise=true , address=104.36.82.47:4545 , piece_id=PZWVIOWLBZ72IOFRJHPAD4OZZPA2G4XYCDLGYEY7THSOR7XDHMUQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617828073 (20h17m54.093766528s) [noise=true , address=104.36.82.42:4802 , piece_id=6HORHYYKIKDX4IWWKUQPVGZQZ2YYIGC47OCZHRVBC4BZFRJX5TQQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617987597 (20h17m54.093926062s) [noise=true , address=104.36.82.13:4214 , piece_id=SLX3YJCEWCM3553VZHILH337FKSPPFNKVFTPXO4KIBOS6IXCLXHA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618055948 (20h17m54.093994403s) [noise=true , address=104.36.82.46:4500 , piece_id=VCYKKDSLEKTTL3KO2FJ4VA5JGIVPA4GFBBHH65HPFZTP4REUTHYQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618285828 (20h17m54.094224293s) [noise=true , address=104.36.82.52:4092 , piece_id=BMJCAQYXJCUVJTEXH52UXYLD4MHWLDGQTJIPDOFRF4LPLTT36ESA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618485871 (20h17m54.094424336s) [noise=true , address=104.36.82.10:4088 , piece_id=6IV2Z7FXDNEOBJMNPRG7WSPZWZTJ5CGDF6JHBSF7WCIIL2QQWFUA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618663881 (20h17m54.094602346s) [noise=true , address=104.36.82.51:4075 , piece_id=P5NU32M3V7AMYFZVJ6XZY5K2YMQWF4EULQGX6RRYWG6SWDWWAFMQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618840449 (20h17m54.094778914s) [noise=true , address=104.36.82.14:4227 , piece_id=O2FXJBOV3CJCKLZHRPBTLXX2QRT2OWJQ5HPNHZI5IVIFDVUUICVQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.619028879 (20h17m54.094967344s) [noise=true , address=104.36.82.9:4028 , piece_id=XCRNGUPBFYCZOYT23L6NHJ6WQR55TQMTWZF4SNKSRLQGC5TYO3LA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.619215837 (20h17m54.095154302s) [noise=true , address=104.36.82.53:4126 , piece_id=GZ5Z7AGMYVGYVUJAWFAIS6SQLRHFB772RLQY736QUGOXA5OWQ34Q, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.619408406 (20h17m54.095346871s) [noise=true , address=104.36.82.55:4219 , piece_id=IURHWONYUYYJWIL5BDF6HLCEPA76O6QAZAJ22NCI6LQYIGO6K4IA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:43.299790904 (20h17m55.775729369s) [upload=19386]: Next returned nil segment Jul 29 13:03:43.299806024 (20h17m55.775744479s) [upload=19386]: Waiting for error group managing segments... Jul 29 13:03:43.847040928 (20h17m56.322979393s) [noise=true , address=104.36.82.9:4028 , piece_id=XCRNGUPBFYCZOYT23L6NHJ6WQR55TQMTWZF4SNKSRLQGC5TYO3LA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:43.879495404 (20h17m56.355433869s) [noise=true , address=104.36.82.61:4391 , piece_id=TI5YDGDSTWKDX5XAWL36F3TJDQNHKNTREZVTUGIS2EVZYP6DWSDA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:46.350727192 (20h17m58.826665657s) [noise=true , address=104.36.82.5:4866 , piece_id=EFRLC4LA232NZZJU3GDSP2EEBZO3W2LGJRD3L2A6SPSRBTLUTIFA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:51.702944756 (20h18m4.178883221s) [noise=true , address=104.36.82.42:4802 , piece_id=6HORHYYKIKDX4IWWKUQPVGZQZ2YYIGC47OCZHRVBC4BZFRJX5TQQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:55.250827413 (20h18m7.726765878s) [noise=true , address=104.36.82.55:4219 , piece_id=IURHWONYUYYJWIL5BDF6HLCEPA76O6QAZAJ22NCI6LQYIGO6K4IA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:57.858469692 (20h18m10.334408157s) [noise=true , address=104.36.82.53:4126 , piece_id=GZ5Z7AGMYVGYVUJAWFAIS6SQLRHFB772RLQY736QUGOXA5OWQ34Q, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:59.265385056 (20h18m11.741323511s) [noise=true , address=104.36.82.10:4088 , piece_id=6IV2Z7FXDNEOBJMNPRG7WSPZWZTJ5CGDF6JHBSF7WCIIL2QQWFUA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:59.973580263 (20h18m12.449518729s) [noise=true , address=104.36.82.11:4119 , piece_id=6KWLABSUMX6YVKD6LGKR6ONLQKFFTSTKFHHRVFIN7U4KDALBNCBQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:16.002200978 (20h18m28.478139453s) [noise=true , address=104.36.82.52:4092 , piece_id=BMJCAQYXJCUVJTEXH52UXYLD4MHWLDGQTJIPDOFRF4LPLTT36ESA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:30.906580908 (20h18m43.382519373s) [noise=true , address=104.36.82.20:4448 , piece_id=CSHYRDGTS4ROFDTKETIZ36TWJE3OEMAKLB4RGFAU557G725EANXQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:31.164952501 (20h18m43.640890966s) [noise=true , address=104.36.82.14:4227 , piece_id=O2FXJBOV3CJCKLZHRPBTLXX2QRT2OWJQ5HPNHZI5IVIFDVUUICVQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:33.000876204 (20h18m45.476814679s) [noise=true , address=104.36.82.56:4242 , piece_id=BHKC5MH4G65BK2YLJJANIYZ4MLAUTMBG32XSRISARHCO26AZUXRA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:34.725342452 (20h18m47.201280907s) [noise=true , address=104.36.82.47:4545 , piece_id=PZWVIOWLBZ72IOFRJHPAD4OZZPA2G4XYCDLGYEY7THSOR7XDHMUQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:40.451024021 (20h18m52.926962486s) [noise=true , address=104.36.82.46:4500 , piece_id=VCYKKDSLEKTTL3KO2FJ4VA5JGIVPA4GFBBHH65HPFZTP4REUTHYQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:12B7ox4ehqxJ2mwqeXj8C9N9GuSU5etNC23iT1pJnTgMLGBAGQK, address:104.36.82.46:4500): protocol: write tcp 188.166.225.170:46700->104.36.82.46:4500: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451060500 (20h18m52.926998965s) [noise=true , address=104.36.82.51:4075 , piece_id=P5NU32M3V7AMYFZVJ6XZY5K2YMQWF4EULQGX6RRYWG6SWDWWAFMQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK, address:104.36.82.51:4075): protocol: write tcp 188.166.225.170:41162->104.36.82.51:4075: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451086179 (20h18m52.927024644s) [noise=true , address=104.36.82.54:4175 , piece_id=5PAE3BM5AFSLZ5TVUQC3QK3RVWCVGFDLOK452BDPWQ7ARTL7EYWQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:12ou7jwFyJP3mNoALddniUTDT9duVyQKZjeq4xHCyx6sRWpwEPJ, address:104.36.82.54:4175): protocol: rpc: dial tcp 104.36.82.54:4175: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451120024 (20h18m52.927058489s) [noise=true , address=104.36.82.13:4214 , piece_id=SLX3YJCEWCM3553VZHILH337FKSPPFNKVFTPXO4KIBOS6IXCLXHA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:1nJ8uqwxf6x4hX52AjAdCbMKi1BKGGbMvrQbiazJdLpn6j69ay, address:104.36.82.13:4214): protocol: rpc: dial tcp 104.36.82.13:4214: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451147116 (20h18m52.927085581s) [noise=true , address=104.36.82.15:4263 , piece_id=3GCLYN4UBCLM5BPLXDQT5SSWPJBTKIOZM4UIUNO3OZIXW6OANUBQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:1f97yBEvcrYHqMq7iovVoju77jzhUYgoQCsUGULeTagugFXrCu, address:104.36.82.15:4263): protocol: write tcp 188.166.225.170:39380->104.36.82.15:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451167805 (20h18m52.927106271s) [seg_pos={0 0}, upload=19386]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 29 13:04:40.696628655 (20h18m53.17256714s) [upload=19386]: Done uploading segments... Jul 29 13:04:40.696658391 (20h18m53.172596856s) [upload=19386]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.839895846s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [.X...X..XX.................................X.............X..] duration: 5.984982046s FAIL (2025-07-29 13:41:12 UTC) orbiter warning: digest mismatch: after ee3c39c295e32d2453f46eedbd5a023ecb901c93b585366a6221da2ecebc35b3 != before f1d4b9d3eca6a9d63854d84cb67ac0fa26d296bc93761368409da19470c1a42f Duration: 5.985014378s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..........X.X...............................................] duration: 5.947835518s FAIL (2025-07-29 13:40:57 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ding segments... Jul 29 13:03:41.091320246 (20h17m53.567258701s) [upload=19386]: Got next segment. Inline: false Jul 29 13:03:41.091349191 (20h17m53.567287656s) [upload=19386]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 29 13:03:41.091354331 (20h17m53.567292796s) [upload=19386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 29 13:03:41.616444845 (20h17m54.09238331s) [seg_pos={0 0}, upload=19386]: Begin upload segment... Jul 29 13:03:41.616487748 (20h17m54.092426213s) [seg_pos={0 0}, upload=19386]: Done begin upload segment. Jul 29 13:03:41.616507376 (20h17m54.092445841s) [noise=true , address=104.36.82.15:4263 , piece_id=3GCLYN4UBCLM5BPLXDQT5SSWPJBTKIOZM4UIUNO3OZIXW6OANUBQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.616564315 (20h17m54.09250278s) [noise=true , address=104.36.82.11:4119 , piece_id=6KWLABSUMX6YVKD6LGKR6ONLQKFFTSTKFHHRVFIN7U4KDALBNCBQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.616866433 (20h17m54.092804888s) [noise=true , address=104.36.82.54:4175 , piece_id=5PAE3BM5AFSLZ5TVUQC3QK3RVWCVGFDLOK452BDPWQ7ARTL7EYWQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617072557 (20h17m54.093011022s) [noise=true , address=104.36.82.56:4242 , piece_id=BHKC5MH4G65BK2YLJJANIYZ4MLAUTMBG32XSRISARHCO26AZUXRA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617110629 (20h17m54.093049084s) [noise=true , address=104.36.82.5:4866 , piece_id=EFRLC4LA232NZZJU3GDSP2EEBZO3W2LGJRD3L2A6SPSRBTLUTIFA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617355037 (20h17m54.093293492s) [noise=true , address=104.36.82.61:4391 , piece_id=TI5YDGDSTWKDX5XAWL36F3TJDQNHKNTREZVTUGIS2EVZYP6DWSDA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617481549 (20h17m54.093420024s) [noise=true , address=104.36.82.20:4448 , piece_id=CSHYRDGTS4ROFDTKETIZ36TWJE3OEMAKLB4RGFAU557G725EANXQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617583044 (20h17m54.093521509s) [noise=true , address=104.36.82.47:4545 , piece_id=PZWVIOWLBZ72IOFRJHPAD4OZZPA2G4XYCDLGYEY7THSOR7XDHMUQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617828073 (20h17m54.093766528s) [noise=true , address=104.36.82.42:4802 , piece_id=6HORHYYKIKDX4IWWKUQPVGZQZ2YYIGC47OCZHRVBC4BZFRJX5TQQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617987597 (20h17m54.093926062s) [noise=true , address=104.36.82.13:4214 , piece_id=SLX3YJCEWCM3553VZHILH337FKSPPFNKVFTPXO4KIBOS6IXCLXHA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618055948 (20h17m54.093994403s) [noise=true , address=104.36.82.46:4500 , piece_id=VCYKKDSLEKTTL3KO2FJ4VA5JGIVPA4GFBBHH65HPFZTP4REUTHYQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618285828 (20h17m54.094224293s) [noise=true , address=104.36.82.52:4092 , piece_id=BMJCAQYXJCUVJTEXH52UXYLD4MHWLDGQTJIPDOFRF4LPLTT36ESA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618485871 (20h17m54.094424336s) [noise=true , address=104.36.82.10:4088 , piece_id=6IV2Z7FXDNEOBJMNPRG7WSPZWZTJ5CGDF6JHBSF7WCIIL2QQWFUA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618663881 (20h17m54.094602346s) [noise=true , address=104.36.82.51:4075 , piece_id=P5NU32M3V7AMYFZVJ6XZY5K2YMQWF4EULQGX6RRYWG6SWDWWAFMQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618840449 (20h17m54.094778914s) [noise=true , address=104.36.82.14:4227 , piece_id=O2FXJBOV3CJCKLZHRPBTLXX2QRT2OWJQ5HPNHZI5IVIFDVUUICVQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.619028879 (20h17m54.094967344s) [noise=true , address=104.36.82.9:4028 , piece_id=XCRNGUPBFYCZOYT23L6NHJ6WQR55TQMTWZF4SNKSRLQGC5TYO3LA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.619215837 (20h17m54.095154302s) [noise=true , address=104.36.82.53:4126 , piece_id=GZ5Z7AGMYVGYVUJAWFAIS6SQLRHFB772RLQY736QUGOXA5OWQ34Q, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.619408406 (20h17m54.095346871s) [noise=true , address=104.36.82.55:4219 , piece_id=IURHWONYUYYJWIL5BDF6HLCEPA76O6QAZAJ22NCI6LQYIGO6K4IA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:43.299790904 (20h17m55.775729369s) [upload=19386]: Next returned nil segment Jul 29 13:03:43.299806024 (20h17m55.775744479s) [upload=19386]: Waiting for error group managing segments... Jul 29 13:03:43.847040928 (20h17m56.322979393s) [noise=true , address=104.36.82.9:4028 , piece_id=XCRNGUPBFYCZOYT23L6NHJ6WQR55TQMTWZF4SNKSRLQGC5TYO3LA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:43.879495404 (20h17m56.355433869s) [noise=true , address=104.36.82.61:4391 , piece_id=TI5YDGDSTWKDX5XAWL36F3TJDQNHKNTREZVTUGIS2EVZYP6DWSDA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:46.350727192 (20h17m58.826665657s) [noise=true , address=104.36.82.5:4866 , piece_id=EFRLC4LA232NZZJU3GDSP2EEBZO3W2LGJRD3L2A6SPSRBTLUTIFA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:51.702944756 (20h18m4.178883221s) [noise=true , address=104.36.82.42:4802 , piece_id=6HORHYYKIKDX4IWWKUQPVGZQZ2YYIGC47OCZHRVBC4BZFRJX5TQQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:55.250827413 (20h18m7.726765878s) [noise=true , address=104.36.82.55:4219 , piece_id=IURHWONYUYYJWIL5BDF6HLCEPA76O6QAZAJ22NCI6LQYIGO6K4IA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:57.858469692 (20h18m10.334408157s) [noise=true , address=104.36.82.53:4126 , piece_id=GZ5Z7AGMYVGYVUJAWFAIS6SQLRHFB772RLQY736QUGOXA5OWQ34Q, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:59.265385056 (20h18m11.741323511s) [noise=true , address=104.36.82.10:4088 , piece_id=6IV2Z7FXDNEOBJMNPRG7WSPZWZTJ5CGDF6JHBSF7WCIIL2QQWFUA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:59.973580263 (20h18m12.449518729s) [noise=true , address=104.36.82.11:4119 , piece_id=6KWLABSUMX6YVKD6LGKR6ONLQKFFTSTKFHHRVFIN7U4KDALBNCBQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:16.002200978 (20h18m28.478139453s) [noise=true , address=104.36.82.52:4092 , piece_id=BMJCAQYXJCUVJTEXH52UXYLD4MHWLDGQTJIPDOFRF4LPLTT36ESA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:30.906580908 (20h18m43.382519373s) [noise=true , address=104.36.82.20:4448 , piece_id=CSHYRDGTS4ROFDTKETIZ36TWJE3OEMAKLB4RGFAU557G725EANXQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:31.164952501 (20h18m43.640890966s) [noise=true , address=104.36.82.14:4227 , piece_id=O2FXJBOV3CJCKLZHRPBTLXX2QRT2OWJQ5HPNHZI5IVIFDVUUICVQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:33.000876204 (20h18m45.476814679s) [noise=true , address=104.36.82.56:4242 , piece_id=BHKC5MH4G65BK2YLJJANIYZ4MLAUTMBG32XSRISARHCO26AZUXRA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:34.725342452 (20h18m47.201280907s) [noise=true , address=104.36.82.47:4545 , piece_id=PZWVIOWLBZ72IOFRJHPAD4OZZPA2G4XYCDLGYEY7THSOR7XDHMUQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:40.451024021 (20h18m52.926962486s) [noise=true , address=104.36.82.46:4500 , piece_id=VCYKKDSLEKTTL3KO2FJ4VA5JGIVPA4GFBBHH65HPFZTP4REUTHYQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:12B7ox4ehqxJ2mwqeXj8C9N9GuSU5etNC23iT1pJnTgMLGBAGQK, address:104.36.82.46:4500): protocol: write tcp 188.166.225.170:46700->104.36.82.46:4500: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451060500 (20h18m52.926998965s) [noise=true , address=104.36.82.51:4075 , piece_id=P5NU32M3V7AMYFZVJ6XZY5K2YMQWF4EULQGX6RRYWG6SWDWWAFMQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK, address:104.36.82.51:4075): protocol: write tcp 188.166.225.170:41162->104.36.82.51:4075: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451086179 (20h18m52.927024644s) [noise=true , address=104.36.82.54:4175 , piece_id=5PAE3BM5AFSLZ5TVUQC3QK3RVWCVGFDLOK452BDPWQ7ARTL7EYWQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:12ou7jwFyJP3mNoALddniUTDT9duVyQKZjeq4xHCyx6sRWpwEPJ, address:104.36.82.54:4175): protocol: rpc: dial tcp 104.36.82.54:4175: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451120024 (20h18m52.927058489s) [noise=true , address=104.36.82.13:4214 , piece_id=SLX3YJCEWCM3553VZHILH337FKSPPFNKVFTPXO4KIBOS6IXCLXHA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:1nJ8uqwxf6x4hX52AjAdCbMKi1BKGGbMvrQbiazJdLpn6j69ay, address:104.36.82.13:4214): protocol: rpc: dial tcp 104.36.82.13:4214: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451147116 (20h18m52.927085581s) [noise=true , address=104.36.82.15:4263 , piece_id=3GCLYN4UBCLM5BPLXDQT5SSWPJBTKIOZM4UIUNO3OZIXW6OANUBQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:1f97yBEvcrYHqMq7iovVoju77jzhUYgoQCsUGULeTagugFXrCu, address:104.36.82.15:4263): protocol: write tcp 188.166.225.170:39380->104.36.82.15:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451167805 (20h18m52.927106271s) [seg_pos={0 0}, upload=19386]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 29 13:04:40.696628655 (20h18m53.17256714s) [upload=19386]: Done uploading segments... Jul 29 13:04:40.696658391 (20h18m53.172596856s) [upload=19386]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.947865515s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...........X.X..............................................] duration: 5.924752354s FAIL (2025-07-29 13:40:28 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ding segments... Jul 29 13:03:41.091320246 (20h17m53.567258701s) [upload=19386]: Got next segment. Inline: false Jul 29 13:03:41.091349191 (20h17m53.567287656s) [upload=19386]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 29 13:03:41.091354331 (20h17m53.567292796s) [upload=19386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 29 13:03:41.616444845 (20h17m54.09238331s) [seg_pos={0 0}, upload=19386]: Begin upload segment... Jul 29 13:03:41.616487748 (20h17m54.092426213s) [seg_pos={0 0}, upload=19386]: Done begin upload segment. Jul 29 13:03:41.616507376 (20h17m54.092445841s) [noise=true , address=104.36.82.15:4263 , piece_id=3GCLYN4UBCLM5BPLXDQT5SSWPJBTKIOZM4UIUNO3OZIXW6OANUBQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.616564315 (20h17m54.09250278s) [noise=true , address=104.36.82.11:4119 , piece_id=6KWLABSUMX6YVKD6LGKR6ONLQKFFTSTKFHHRVFIN7U4KDALBNCBQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.616866433 (20h17m54.092804888s) [noise=true , address=104.36.82.54:4175 , piece_id=5PAE3BM5AFSLZ5TVUQC3QK3RVWCVGFDLOK452BDPWQ7ARTL7EYWQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617072557 (20h17m54.093011022s) [noise=true , address=104.36.82.56:4242 , piece_id=BHKC5MH4G65BK2YLJJANIYZ4MLAUTMBG32XSRISARHCO26AZUXRA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617110629 (20h17m54.093049084s) [noise=true , address=104.36.82.5:4866 , piece_id=EFRLC4LA232NZZJU3GDSP2EEBZO3W2LGJRD3L2A6SPSRBTLUTIFA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617355037 (20h17m54.093293492s) [noise=true , address=104.36.82.61:4391 , piece_id=TI5YDGDSTWKDX5XAWL36F3TJDQNHKNTREZVTUGIS2EVZYP6DWSDA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617481549 (20h17m54.093420024s) [noise=true , address=104.36.82.20:4448 , piece_id=CSHYRDGTS4ROFDTKETIZ36TWJE3OEMAKLB4RGFAU557G725EANXQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617583044 (20h17m54.093521509s) [noise=true , address=104.36.82.47:4545 , piece_id=PZWVIOWLBZ72IOFRJHPAD4OZZPA2G4XYCDLGYEY7THSOR7XDHMUQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617828073 (20h17m54.093766528s) [noise=true , address=104.36.82.42:4802 , piece_id=6HORHYYKIKDX4IWWKUQPVGZQZ2YYIGC47OCZHRVBC4BZFRJX5TQQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617987597 (20h17m54.093926062s) [noise=true , address=104.36.82.13:4214 , piece_id=SLX3YJCEWCM3553VZHILH337FKSPPFNKVFTPXO4KIBOS6IXCLXHA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618055948 (20h17m54.093994403s) [noise=true , address=104.36.82.46:4500 , piece_id=VCYKKDSLEKTTL3KO2FJ4VA5JGIVPA4GFBBHH65HPFZTP4REUTHYQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618285828 (20h17m54.094224293s) [noise=true , address=104.36.82.52:4092 , piece_id=BMJCAQYXJCUVJTEXH52UXYLD4MHWLDGQTJIPDOFRF4LPLTT36ESA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618485871 (20h17m54.094424336s) [noise=true , address=104.36.82.10:4088 , piece_id=6IV2Z7FXDNEOBJMNPRG7WSPZWZTJ5CGDF6JHBSF7WCIIL2QQWFUA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618663881 (20h17m54.094602346s) [noise=true , address=104.36.82.51:4075 , piece_id=P5NU32M3V7AMYFZVJ6XZY5K2YMQWF4EULQGX6RRYWG6SWDWWAFMQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618840449 (20h17m54.094778914s) [noise=true , address=104.36.82.14:4227 , piece_id=O2FXJBOV3CJCKLZHRPBTLXX2QRT2OWJQ5HPNHZI5IVIFDVUUICVQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.619028879 (20h17m54.094967344s) [noise=true , address=104.36.82.9:4028 , piece_id=XCRNGUPBFYCZOYT23L6NHJ6WQR55TQMTWZF4SNKSRLQGC5TYO3LA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.619215837 (20h17m54.095154302s) [noise=true , address=104.36.82.53:4126 , piece_id=GZ5Z7AGMYVGYVUJAWFAIS6SQLRHFB772RLQY736QUGOXA5OWQ34Q, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.619408406 (20h17m54.095346871s) [noise=true , address=104.36.82.55:4219 , piece_id=IURHWONYUYYJWIL5BDF6HLCEPA76O6QAZAJ22NCI6LQYIGO6K4IA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:43.299790904 (20h17m55.775729369s) [upload=19386]: Next returned nil segment Jul 29 13:03:43.299806024 (20h17m55.775744479s) [upload=19386]: Waiting for error group managing segments... Jul 29 13:03:43.847040928 (20h17m56.322979393s) [noise=true , address=104.36.82.9:4028 , piece_id=XCRNGUPBFYCZOYT23L6NHJ6WQR55TQMTWZF4SNKSRLQGC5TYO3LA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:43.879495404 (20h17m56.355433869s) [noise=true , address=104.36.82.61:4391 , piece_id=TI5YDGDSTWKDX5XAWL36F3TJDQNHKNTREZVTUGIS2EVZYP6DWSDA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:46.350727192 (20h17m58.826665657s) [noise=true , address=104.36.82.5:4866 , piece_id=EFRLC4LA232NZZJU3GDSP2EEBZO3W2LGJRD3L2A6SPSRBTLUTIFA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:51.702944756 (20h18m4.178883221s) [noise=true , address=104.36.82.42:4802 , piece_id=6HORHYYKIKDX4IWWKUQPVGZQZ2YYIGC47OCZHRVBC4BZFRJX5TQQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:55.250827413 (20h18m7.726765878s) [noise=true , address=104.36.82.55:4219 , piece_id=IURHWONYUYYJWIL5BDF6HLCEPA76O6QAZAJ22NCI6LQYIGO6K4IA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:57.858469692 (20h18m10.334408157s) [noise=true , address=104.36.82.53:4126 , piece_id=GZ5Z7AGMYVGYVUJAWFAIS6SQLRHFB772RLQY736QUGOXA5OWQ34Q, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:59.265385056 (20h18m11.741323511s) [noise=true , address=104.36.82.10:4088 , piece_id=6IV2Z7FXDNEOBJMNPRG7WSPZWZTJ5CGDF6JHBSF7WCIIL2QQWFUA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:59.973580263 (20h18m12.449518729s) [noise=true , address=104.36.82.11:4119 , piece_id=6KWLABSUMX6YVKD6LGKR6ONLQKFFTSTKFHHRVFIN7U4KDALBNCBQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:16.002200978 (20h18m28.478139453s) [noise=true , address=104.36.82.52:4092 , piece_id=BMJCAQYXJCUVJTEXH52UXYLD4MHWLDGQTJIPDOFRF4LPLTT36ESA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:30.906580908 (20h18m43.382519373s) [noise=true , address=104.36.82.20:4448 , piece_id=CSHYRDGTS4ROFDTKETIZ36TWJE3OEMAKLB4RGFAU557G725EANXQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:31.164952501 (20h18m43.640890966s) [noise=true , address=104.36.82.14:4227 , piece_id=O2FXJBOV3CJCKLZHRPBTLXX2QRT2OWJQ5HPNHZI5IVIFDVUUICVQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:33.000876204 (20h18m45.476814679s) [noise=true , address=104.36.82.56:4242 , piece_id=BHKC5MH4G65BK2YLJJANIYZ4MLAUTMBG32XSRISARHCO26AZUXRA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:34.725342452 (20h18m47.201280907s) [noise=true , address=104.36.82.47:4545 , piece_id=PZWVIOWLBZ72IOFRJHPAD4OZZPA2G4XYCDLGYEY7THSOR7XDHMUQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:40.451024021 (20h18m52.926962486s) [noise=true , address=104.36.82.46:4500 , piece_id=VCYKKDSLEKTTL3KO2FJ4VA5JGIVPA4GFBBHH65HPFZTP4REUTHYQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:12B7ox4ehqxJ2mwqeXj8C9N9GuSU5etNC23iT1pJnTgMLGBAGQK, address:104.36.82.46:4500): protocol: write tcp 188.166.225.170:46700->104.36.82.46:4500: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451060500 (20h18m52.926998965s) [noise=true , address=104.36.82.51:4075 , piece_id=P5NU32M3V7AMYFZVJ6XZY5K2YMQWF4EULQGX6RRYWG6SWDWWAFMQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK, address:104.36.82.51:4075): protocol: write tcp 188.166.225.170:41162->104.36.82.51:4075: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451086179 (20h18m52.927024644s) [noise=true , address=104.36.82.54:4175 , piece_id=5PAE3BM5AFSLZ5TVUQC3QK3RVWCVGFDLOK452BDPWQ7ARTL7EYWQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:12ou7jwFyJP3mNoALddniUTDT9duVyQKZjeq4xHCyx6sRWpwEPJ, address:104.36.82.54:4175): protocol: rpc: dial tcp 104.36.82.54:4175: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451120024 (20h18m52.927058489s) [noise=true , address=104.36.82.13:4214 , piece_id=SLX3YJCEWCM3553VZHILH337FKSPPFNKVFTPXO4KIBOS6IXCLXHA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:1nJ8uqwxf6x4hX52AjAdCbMKi1BKGGbMvrQbiazJdLpn6j69ay, address:104.36.82.13:4214): protocol: rpc: dial tcp 104.36.82.13:4214: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451147116 (20h18m52.927085581s) [noise=true , address=104.36.82.15:4263 , piece_id=3GCLYN4UBCLM5BPLXDQT5SSWPJBTKIOZM4UIUNO3OZIXW6OANUBQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:1f97yBEvcrYHqMq7iovVoju77jzhUYgoQCsUGULeTagugFXrCu, address:104.36.82.15:4263): protocol: write tcp 188.166.225.170:39380->104.36.82.15:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451167805 (20h18m52.927106271s) [seg_pos={0 0}, upload=19386]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 29 13:04:40.696628655 (20h18m53.17256714s) [upload=19386]: Done uploading segments... Jul 29 13:04:40.696658391 (20h18m53.172596856s) [upload=19386]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.924775718s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [..X...X..XX.................................X.............X.] duration: 5.773391464s FAIL (2025-07-29 13:40:01 UTC) orbiter warning: digest mismatch: after ee3c39c295e32d2453f46eedbd5a023ecb901c93b585366a6221da2ecebc35b3 != before f1d4b9d3eca6a9d63854d84cb67ac0fa26d296bc93761368409da19470c1a42f Duration: 5.77342592s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [............X.X.............................................] duration: 5.825787518s FAIL (2025-07-29 13:39:17 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ding segments... Jul 29 13:03:41.091320246 (20h17m53.567258701s) [upload=19386]: Got next segment. Inline: false Jul 29 13:03:41.091349191 (20h17m53.567287656s) [upload=19386]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 29 13:03:41.091354331 (20h17m53.567292796s) [upload=19386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 29 13:03:41.616444845 (20h17m54.09238331s) [seg_pos={0 0}, upload=19386]: Begin upload segment... Jul 29 13:03:41.616487748 (20h17m54.092426213s) [seg_pos={0 0}, upload=19386]: Done begin upload segment. Jul 29 13:03:41.616507376 (20h17m54.092445841s) [noise=true , address=104.36.82.15:4263 , piece_id=3GCLYN4UBCLM5BPLXDQT5SSWPJBTKIOZM4UIUNO3OZIXW6OANUBQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.616564315 (20h17m54.09250278s) [noise=true , address=104.36.82.11:4119 , piece_id=6KWLABSUMX6YVKD6LGKR6ONLQKFFTSTKFHHRVFIN7U4KDALBNCBQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.616866433 (20h17m54.092804888s) [noise=true , address=104.36.82.54:4175 , piece_id=5PAE3BM5AFSLZ5TVUQC3QK3RVWCVGFDLOK452BDPWQ7ARTL7EYWQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617072557 (20h17m54.093011022s) [noise=true , address=104.36.82.56:4242 , piece_id=BHKC5MH4G65BK2YLJJANIYZ4MLAUTMBG32XSRISARHCO26AZUXRA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617110629 (20h17m54.093049084s) [noise=true , address=104.36.82.5:4866 , piece_id=EFRLC4LA232NZZJU3GDSP2EEBZO3W2LGJRD3L2A6SPSRBTLUTIFA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617355037 (20h17m54.093293492s) [noise=true , address=104.36.82.61:4391 , piece_id=TI5YDGDSTWKDX5XAWL36F3TJDQNHKNTREZVTUGIS2EVZYP6DWSDA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617481549 (20h17m54.093420024s) [noise=true , address=104.36.82.20:4448 , piece_id=CSHYRDGTS4ROFDTKETIZ36TWJE3OEMAKLB4RGFAU557G725EANXQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617583044 (20h17m54.093521509s) [noise=true , address=104.36.82.47:4545 , piece_id=PZWVIOWLBZ72IOFRJHPAD4OZZPA2G4XYCDLGYEY7THSOR7XDHMUQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617828073 (20h17m54.093766528s) [noise=true , address=104.36.82.42:4802 , piece_id=6HORHYYKIKDX4IWWKUQPVGZQZ2YYIGC47OCZHRVBC4BZFRJX5TQQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617987597 (20h17m54.093926062s) [noise=true , address=104.36.82.13:4214 , piece_id=SLX3YJCEWCM3553VZHILH337FKSPPFNKVFTPXO4KIBOS6IXCLXHA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618055948 (20h17m54.093994403s) [noise=true , address=104.36.82.46:4500 , piece_id=VCYKKDSLEKTTL3KO2FJ4VA5JGIVPA4GFBBHH65HPFZTP4REUTHYQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618285828 (20h17m54.094224293s) [noise=true , address=104.36.82.52:4092 , piece_id=BMJCAQYXJCUVJTEXH52UXYLD4MHWLDGQTJIPDOFRF4LPLTT36ESA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618485871 (20h17m54.094424336s) [noise=true , address=104.36.82.10:4088 , piece_id=6IV2Z7FXDNEOBJMNPRG7WSPZWZTJ5CGDF6JHBSF7WCIIL2QQWFUA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618663881 (20h17m54.094602346s) [noise=true , address=104.36.82.51:4075 , piece_id=P5NU32M3V7AMYFZVJ6XZY5K2YMQWF4EULQGX6RRYWG6SWDWWAFMQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618840449 (20h17m54.094778914s) [noise=true , address=104.36.82.14:4227 , piece_id=O2FXJBOV3CJCKLZHRPBTLXX2QRT2OWJQ5HPNHZI5IVIFDVUUICVQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.619028879 (20h17m54.094967344s) [noise=true , address=104.36.82.9:4028 , piece_id=XCRNGUPBFYCZOYT23L6NHJ6WQR55TQMTWZF4SNKSRLQGC5TYO3LA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.619215837 (20h17m54.095154302s) [noise=true , address=104.36.82.53:4126 , piece_id=GZ5Z7AGMYVGYVUJAWFAIS6SQLRHFB772RLQY736QUGOXA5OWQ34Q, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.619408406 (20h17m54.095346871s) [noise=true , address=104.36.82.55:4219 , piece_id=IURHWONYUYYJWIL5BDF6HLCEPA76O6QAZAJ22NCI6LQYIGO6K4IA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:43.299790904 (20h17m55.775729369s) [upload=19386]: Next returned nil segment Jul 29 13:03:43.299806024 (20h17m55.775744479s) [upload=19386]: Waiting for error group managing segments... Jul 29 13:03:43.847040928 (20h17m56.322979393s) [noise=true , address=104.36.82.9:4028 , piece_id=XCRNGUPBFYCZOYT23L6NHJ6WQR55TQMTWZF4SNKSRLQGC5TYO3LA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:43.879495404 (20h17m56.355433869s) [noise=true , address=104.36.82.61:4391 , piece_id=TI5YDGDSTWKDX5XAWL36F3TJDQNHKNTREZVTUGIS2EVZYP6DWSDA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:46.350727192 (20h17m58.826665657s) [noise=true , address=104.36.82.5:4866 , piece_id=EFRLC4LA232NZZJU3GDSP2EEBZO3W2LGJRD3L2A6SPSRBTLUTIFA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:51.702944756 (20h18m4.178883221s) [noise=true , address=104.36.82.42:4802 , piece_id=6HORHYYKIKDX4IWWKUQPVGZQZ2YYIGC47OCZHRVBC4BZFRJX5TQQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:55.250827413 (20h18m7.726765878s) [noise=true , address=104.36.82.55:4219 , piece_id=IURHWONYUYYJWIL5BDF6HLCEPA76O6QAZAJ22NCI6LQYIGO6K4IA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:57.858469692 (20h18m10.334408157s) [noise=true , address=104.36.82.53:4126 , piece_id=GZ5Z7AGMYVGYVUJAWFAIS6SQLRHFB772RLQY736QUGOXA5OWQ34Q, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:59.265385056 (20h18m11.741323511s) [noise=true , address=104.36.82.10:4088 , piece_id=6IV2Z7FXDNEOBJMNPRG7WSPZWZTJ5CGDF6JHBSF7WCIIL2QQWFUA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:59.973580263 (20h18m12.449518729s) [noise=true , address=104.36.82.11:4119 , piece_id=6KWLABSUMX6YVKD6LGKR6ONLQKFFTSTKFHHRVFIN7U4KDALBNCBQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:16.002200978 (20h18m28.478139453s) [noise=true , address=104.36.82.52:4092 , piece_id=BMJCAQYXJCUVJTEXH52UXYLD4MHWLDGQTJIPDOFRF4LPLTT36ESA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:30.906580908 (20h18m43.382519373s) [noise=true , address=104.36.82.20:4448 , piece_id=CSHYRDGTS4ROFDTKETIZ36TWJE3OEMAKLB4RGFAU557G725EANXQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:31.164952501 (20h18m43.640890966s) [noise=true , address=104.36.82.14:4227 , piece_id=O2FXJBOV3CJCKLZHRPBTLXX2QRT2OWJQ5HPNHZI5IVIFDVUUICVQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:33.000876204 (20h18m45.476814679s) [noise=true , address=104.36.82.56:4242 , piece_id=BHKC5MH4G65BK2YLJJANIYZ4MLAUTMBG32XSRISARHCO26AZUXRA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:34.725342452 (20h18m47.201280907s) [noise=true , address=104.36.82.47:4545 , piece_id=PZWVIOWLBZ72IOFRJHPAD4OZZPA2G4XYCDLGYEY7THSOR7XDHMUQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:40.451024021 (20h18m52.926962486s) [noise=true , address=104.36.82.46:4500 , piece_id=VCYKKDSLEKTTL3KO2FJ4VA5JGIVPA4GFBBHH65HPFZTP4REUTHYQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:12B7ox4ehqxJ2mwqeXj8C9N9GuSU5etNC23iT1pJnTgMLGBAGQK, address:104.36.82.46:4500): protocol: write tcp 188.166.225.170:46700->104.36.82.46:4500: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451060500 (20h18m52.926998965s) [noise=true , address=104.36.82.51:4075 , piece_id=P5NU32M3V7AMYFZVJ6XZY5K2YMQWF4EULQGX6RRYWG6SWDWWAFMQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK, address:104.36.82.51:4075): protocol: write tcp 188.166.225.170:41162->104.36.82.51:4075: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451086179 (20h18m52.927024644s) [noise=true , address=104.36.82.54:4175 , piece_id=5PAE3BM5AFSLZ5TVUQC3QK3RVWCVGFDLOK452BDPWQ7ARTL7EYWQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:12ou7jwFyJP3mNoALddniUTDT9duVyQKZjeq4xHCyx6sRWpwEPJ, address:104.36.82.54:4175): protocol: rpc: dial tcp 104.36.82.54:4175: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451120024 (20h18m52.927058489s) [noise=true , address=104.36.82.13:4214 , piece_id=SLX3YJCEWCM3553VZHILH337FKSPPFNKVFTPXO4KIBOS6IXCLXHA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:1nJ8uqwxf6x4hX52AjAdCbMKi1BKGGbMvrQbiazJdLpn6j69ay, address:104.36.82.13:4214): protocol: rpc: dial tcp 104.36.82.13:4214: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451147116 (20h18m52.927085581s) [noise=true , address=104.36.82.15:4263 , piece_id=3GCLYN4UBCLM5BPLXDQT5SSWPJBTKIOZM4UIUNO3OZIXW6OANUBQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:1f97yBEvcrYHqMq7iovVoju77jzhUYgoQCsUGULeTagugFXrCu, address:104.36.82.15:4263): protocol: write tcp 188.166.225.170:39380->104.36.82.15:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451167805 (20h18m52.927106271s) [seg_pos={0 0}, upload=19386]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 29 13:04:40.696628655 (20h18m53.17256714s) [upload=19386]: Done uploading segments... Jul 29 13:04:40.696658391 (20h18m53.172596856s) [upload=19386]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.825826011s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.............X.X............................................] duration: 5.797625256s FAIL (2025-07-29 13:38:56 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ding segments... Jul 29 13:03:41.091320246 (20h17m53.567258701s) [upload=19386]: Got next segment. Inline: false Jul 29 13:03:41.091349191 (20h17m53.567287656s) [upload=19386]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 29 13:03:41.091354331 (20h17m53.567292796s) [upload=19386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 29 13:03:41.616444845 (20h17m54.09238331s) [seg_pos={0 0}, upload=19386]: Begin upload segment... Jul 29 13:03:41.616487748 (20h17m54.092426213s) [seg_pos={0 0}, upload=19386]: Done begin upload segment. Jul 29 13:03:41.616507376 (20h17m54.092445841s) [noise=true , address=104.36.82.15:4263 , piece_id=3GCLYN4UBCLM5BPLXDQT5SSWPJBTKIOZM4UIUNO3OZIXW6OANUBQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.616564315 (20h17m54.09250278s) [noise=true , address=104.36.82.11:4119 , piece_id=6KWLABSUMX6YVKD6LGKR6ONLQKFFTSTKFHHRVFIN7U4KDALBNCBQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.616866433 (20h17m54.092804888s) [noise=true , address=104.36.82.54:4175 , piece_id=5PAE3BM5AFSLZ5TVUQC3QK3RVWCVGFDLOK452BDPWQ7ARTL7EYWQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617072557 (20h17m54.093011022s) [noise=true , address=104.36.82.56:4242 , piece_id=BHKC5MH4G65BK2YLJJANIYZ4MLAUTMBG32XSRISARHCO26AZUXRA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617110629 (20h17m54.093049084s) [noise=true , address=104.36.82.5:4866 , piece_id=EFRLC4LA232NZZJU3GDSP2EEBZO3W2LGJRD3L2A6SPSRBTLUTIFA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617355037 (20h17m54.093293492s) [noise=true , address=104.36.82.61:4391 , piece_id=TI5YDGDSTWKDX5XAWL36F3TJDQNHKNTREZVTUGIS2EVZYP6DWSDA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617481549 (20h17m54.093420024s) [noise=true , address=104.36.82.20:4448 , piece_id=CSHYRDGTS4ROFDTKETIZ36TWJE3OEMAKLB4RGFAU557G725EANXQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617583044 (20h17m54.093521509s) [noise=true , address=104.36.82.47:4545 , piece_id=PZWVIOWLBZ72IOFRJHPAD4OZZPA2G4XYCDLGYEY7THSOR7XDHMUQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617828073 (20h17m54.093766528s) [noise=true , address=104.36.82.42:4802 , piece_id=6HORHYYKIKDX4IWWKUQPVGZQZ2YYIGC47OCZHRVBC4BZFRJX5TQQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617987597 (20h17m54.093926062s) [noise=true , address=104.36.82.13:4214 , piece_id=SLX3YJCEWCM3553VZHILH337FKSPPFNKVFTPXO4KIBOS6IXCLXHA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618055948 (20h17m54.093994403s) [noise=true , address=104.36.82.46:4500 , piece_id=VCYKKDSLEKTTL3KO2FJ4VA5JGIVPA4GFBBHH65HPFZTP4REUTHYQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618285828 (20h17m54.094224293s) [noise=true , address=104.36.82.52:4092 , piece_id=BMJCAQYXJCUVJTEXH52UXYLD4MHWLDGQTJIPDOFRF4LPLTT36ESA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618485871 (20h17m54.094424336s) [noise=true , address=104.36.82.10:4088 , piece_id=6IV2Z7FXDNEOBJMNPRG7WSPZWZTJ5CGDF6JHBSF7WCIIL2QQWFUA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618663881 (20h17m54.094602346s) [noise=true , address=104.36.82.51:4075 , piece_id=P5NU32M3V7AMYFZVJ6XZY5K2YMQWF4EULQGX6RRYWG6SWDWWAFMQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618840449 (20h17m54.094778914s) [noise=true , address=104.36.82.14:4227 , piece_id=O2FXJBOV3CJCKLZHRPBTLXX2QRT2OWJQ5HPNHZI5IVIFDVUUICVQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.619028879 (20h17m54.094967344s) [noise=true , address=104.36.82.9:4028 , piece_id=XCRNGUPBFYCZOYT23L6NHJ6WQR55TQMTWZF4SNKSRLQGC5TYO3LA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.619215837 (20h17m54.095154302s) [noise=true , address=104.36.82.53:4126 , piece_id=GZ5Z7AGMYVGYVUJAWFAIS6SQLRHFB772RLQY736QUGOXA5OWQ34Q, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.619408406 (20h17m54.095346871s) [noise=true , address=104.36.82.55:4219 , piece_id=IURHWONYUYYJWIL5BDF6HLCEPA76O6QAZAJ22NCI6LQYIGO6K4IA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:43.299790904 (20h17m55.775729369s) [upload=19386]: Next returned nil segment Jul 29 13:03:43.299806024 (20h17m55.775744479s) [upload=19386]: Waiting for error group managing segments... Jul 29 13:03:43.847040928 (20h17m56.322979393s) [noise=true , address=104.36.82.9:4028 , piece_id=XCRNGUPBFYCZOYT23L6NHJ6WQR55TQMTWZF4SNKSRLQGC5TYO3LA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:43.879495404 (20h17m56.355433869s) [noise=true , address=104.36.82.61:4391 , piece_id=TI5YDGDSTWKDX5XAWL36F3TJDQNHKNTREZVTUGIS2EVZYP6DWSDA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:46.350727192 (20h17m58.826665657s) [noise=true , address=104.36.82.5:4866 , piece_id=EFRLC4LA232NZZJU3GDSP2EEBZO3W2LGJRD3L2A6SPSRBTLUTIFA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:51.702944756 (20h18m4.178883221s) [noise=true , address=104.36.82.42:4802 , piece_id=6HORHYYKIKDX4IWWKUQPVGZQZ2YYIGC47OCZHRVBC4BZFRJX5TQQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:55.250827413 (20h18m7.726765878s) [noise=true , address=104.36.82.55:4219 , piece_id=IURHWONYUYYJWIL5BDF6HLCEPA76O6QAZAJ22NCI6LQYIGO6K4IA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:57.858469692 (20h18m10.334408157s) [noise=true , address=104.36.82.53:4126 , piece_id=GZ5Z7AGMYVGYVUJAWFAIS6SQLRHFB772RLQY736QUGOXA5OWQ34Q, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:59.265385056 (20h18m11.741323511s) [noise=true , address=104.36.82.10:4088 , piece_id=6IV2Z7FXDNEOBJMNPRG7WSPZWZTJ5CGDF6JHBSF7WCIIL2QQWFUA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:59.973580263 (20h18m12.449518729s) [noise=true , address=104.36.82.11:4119 , piece_id=6KWLABSUMX6YVKD6LGKR6ONLQKFFTSTKFHHRVFIN7U4KDALBNCBQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:16.002200978 (20h18m28.478139453s) [noise=true , address=104.36.82.52:4092 , piece_id=BMJCAQYXJCUVJTEXH52UXYLD4MHWLDGQTJIPDOFRF4LPLTT36ESA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:30.906580908 (20h18m43.382519373s) [noise=true , address=104.36.82.20:4448 , piece_id=CSHYRDGTS4ROFDTKETIZ36TWJE3OEMAKLB4RGFAU557G725EANXQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:31.164952501 (20h18m43.640890966s) [noise=true , address=104.36.82.14:4227 , piece_id=O2FXJBOV3CJCKLZHRPBTLXX2QRT2OWJQ5HPNHZI5IVIFDVUUICVQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:33.000876204 (20h18m45.476814679s) [noise=true , address=104.36.82.56:4242 , piece_id=BHKC5MH4G65BK2YLJJANIYZ4MLAUTMBG32XSRISARHCO26AZUXRA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:34.725342452 (20h18m47.201280907s) [noise=true , address=104.36.82.47:4545 , piece_id=PZWVIOWLBZ72IOFRJHPAD4OZZPA2G4XYCDLGYEY7THSOR7XDHMUQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:40.451024021 (20h18m52.926962486s) [noise=true , address=104.36.82.46:4500 , piece_id=VCYKKDSLEKTTL3KO2FJ4VA5JGIVPA4GFBBHH65HPFZTP4REUTHYQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:12B7ox4ehqxJ2mwqeXj8C9N9GuSU5etNC23iT1pJnTgMLGBAGQK, address:104.36.82.46:4500): protocol: write tcp 188.166.225.170:46700->104.36.82.46:4500: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451060500 (20h18m52.926998965s) [noise=true , address=104.36.82.51:4075 , piece_id=P5NU32M3V7AMYFZVJ6XZY5K2YMQWF4EULQGX6RRYWG6SWDWWAFMQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK, address:104.36.82.51:4075): protocol: write tcp 188.166.225.170:41162->104.36.82.51:4075: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451086179 (20h18m52.927024644s) [noise=true , address=104.36.82.54:4175 , piece_id=5PAE3BM5AFSLZ5TVUQC3QK3RVWCVGFDLOK452BDPWQ7ARTL7EYWQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:12ou7jwFyJP3mNoALddniUTDT9duVyQKZjeq4xHCyx6sRWpwEPJ, address:104.36.82.54:4175): protocol: rpc: dial tcp 104.36.82.54:4175: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451120024 (20h18m52.927058489s) [noise=true , address=104.36.82.13:4214 , piece_id=SLX3YJCEWCM3553VZHILH337FKSPPFNKVFTPXO4KIBOS6IXCLXHA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:1nJ8uqwxf6x4hX52AjAdCbMKi1BKGGbMvrQbiazJdLpn6j69ay, address:104.36.82.13:4214): protocol: rpc: dial tcp 104.36.82.13:4214: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451147116 (20h18m52.927085581s) [noise=true , address=104.36.82.15:4263 , piece_id=3GCLYN4UBCLM5BPLXDQT5SSWPJBTKIOZM4UIUNO3OZIXW6OANUBQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:1f97yBEvcrYHqMq7iovVoju77jzhUYgoQCsUGULeTagugFXrCu, address:104.36.82.15:4263): protocol: write tcp 188.166.225.170:39380->104.36.82.15:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451167805 (20h18m52.927106271s) [seg_pos={0 0}, upload=19386]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 29 13:04:40.696628655 (20h18m53.17256714s) [upload=19386]: Done uploading segments... Jul 29 13:04:40.696658391 (20h18m53.172596856s) [upload=19386]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.797651987s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [...X...X..XX.................................X.............X] duration: 5.646443972s FAIL (2025-07-29 13:38:46 UTC) orbiter warning: digest mismatch: after ee3c39c295e32d2453f46eedbd5a023ecb901c93b585366a6221da2ecebc35b3 != before f1d4b9d3eca6a9d63854d84cb67ac0fa26d296bc93761368409da19470c1a42f Duration: 5.646469732s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [....X...X..XX.................................X.............] duration: 5.921261586s FAIL (2025-07-29 13:38:16 UTC) orbiter warning: digest mismatch: after 786d2c4be8d438b2c616b59df6d4f93d4922ed144e0f79b484237567514766eb != before 754c097f166f520f3e7f168fcc989b028ff6249f3b26903ffdb38389ae26cd4d Duration: 5.921284149s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..............X.X...........................................] duration: 5.897074547s FAIL (2025-07-29 13:38:02 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ding segments... Jul 29 13:03:41.091320246 (20h17m53.567258701s) [upload=19386]: Got next segment. Inline: false Jul 29 13:03:41.091349191 (20h17m53.567287656s) [upload=19386]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 29 13:03:41.091354331 (20h17m53.567292796s) [upload=19386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 29 13:03:41.616444845 (20h17m54.09238331s) [seg_pos={0 0}, upload=19386]: Begin upload segment... Jul 29 13:03:41.616487748 (20h17m54.092426213s) [seg_pos={0 0}, upload=19386]: Done begin upload segment. Jul 29 13:03:41.616507376 (20h17m54.092445841s) [noise=true , address=104.36.82.15:4263 , piece_id=3GCLYN4UBCLM5BPLXDQT5SSWPJBTKIOZM4UIUNO3OZIXW6OANUBQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.616564315 (20h17m54.09250278s) [noise=true , address=104.36.82.11:4119 , piece_id=6KWLABSUMX6YVKD6LGKR6ONLQKFFTSTKFHHRVFIN7U4KDALBNCBQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.616866433 (20h17m54.092804888s) [noise=true , address=104.36.82.54:4175 , piece_id=5PAE3BM5AFSLZ5TVUQC3QK3RVWCVGFDLOK452BDPWQ7ARTL7EYWQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617072557 (20h17m54.093011022s) [noise=true , address=104.36.82.56:4242 , piece_id=BHKC5MH4G65BK2YLJJANIYZ4MLAUTMBG32XSRISARHCO26AZUXRA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617110629 (20h17m54.093049084s) [noise=true , address=104.36.82.5:4866 , piece_id=EFRLC4LA232NZZJU3GDSP2EEBZO3W2LGJRD3L2A6SPSRBTLUTIFA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617355037 (20h17m54.093293492s) [noise=true , address=104.36.82.61:4391 , piece_id=TI5YDGDSTWKDX5XAWL36F3TJDQNHKNTREZVTUGIS2EVZYP6DWSDA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617481549 (20h17m54.093420024s) [noise=true , address=104.36.82.20:4448 , piece_id=CSHYRDGTS4ROFDTKETIZ36TWJE3OEMAKLB4RGFAU557G725EANXQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617583044 (20h17m54.093521509s) [noise=true , address=104.36.82.47:4545 , piece_id=PZWVIOWLBZ72IOFRJHPAD4OZZPA2G4XYCDLGYEY7THSOR7XDHMUQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617828073 (20h17m54.093766528s) [noise=true , address=104.36.82.42:4802 , piece_id=6HORHYYKIKDX4IWWKUQPVGZQZ2YYIGC47OCZHRVBC4BZFRJX5TQQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617987597 (20h17m54.093926062s) [noise=true , address=104.36.82.13:4214 , piece_id=SLX3YJCEWCM3553VZHILH337FKSPPFNKVFTPXO4KIBOS6IXCLXHA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618055948 (20h17m54.093994403s) [noise=true , address=104.36.82.46:4500 , piece_id=VCYKKDSLEKTTL3KO2FJ4VA5JGIVPA4GFBBHH65HPFZTP4REUTHYQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618285828 (20h17m54.094224293s) [noise=true , address=104.36.82.52:4092 , piece_id=BMJCAQYXJCUVJTEXH52UXYLD4MHWLDGQTJIPDOFRF4LPLTT36ESA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618485871 (20h17m54.094424336s) [noise=true , address=104.36.82.10:4088 , piece_id=6IV2Z7FXDNEOBJMNPRG7WSPZWZTJ5CGDF6JHBSF7WCIIL2QQWFUA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618663881 (20h17m54.094602346s) [noise=true , address=104.36.82.51:4075 , piece_id=P5NU32M3V7AMYFZVJ6XZY5K2YMQWF4EULQGX6RRYWG6SWDWWAFMQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618840449 (20h17m54.094778914s) [noise=true , address=104.36.82.14:4227 , piece_id=O2FXJBOV3CJCKLZHRPBTLXX2QRT2OWJQ5HPNHZI5IVIFDVUUICVQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.619028879 (20h17m54.094967344s) [noise=true , address=104.36.82.9:4028 , piece_id=XCRNGUPBFYCZOYT23L6NHJ6WQR55TQMTWZF4SNKSRLQGC5TYO3LA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.619215837 (20h17m54.095154302s) [noise=true , address=104.36.82.53:4126 , piece_id=GZ5Z7AGMYVGYVUJAWFAIS6SQLRHFB772RLQY736QUGOXA5OWQ34Q, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.619408406 (20h17m54.095346871s) [noise=true , address=104.36.82.55:4219 , piece_id=IURHWONYUYYJWIL5BDF6HLCEPA76O6QAZAJ22NCI6LQYIGO6K4IA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:43.299790904 (20h17m55.775729369s) [upload=19386]: Next returned nil segment Jul 29 13:03:43.299806024 (20h17m55.775744479s) [upload=19386]: Waiting for error group managing segments... Jul 29 13:03:43.847040928 (20h17m56.322979393s) [noise=true , address=104.36.82.9:4028 , piece_id=XCRNGUPBFYCZOYT23L6NHJ6WQR55TQMTWZF4SNKSRLQGC5TYO3LA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:43.879495404 (20h17m56.355433869s) [noise=true , address=104.36.82.61:4391 , piece_id=TI5YDGDSTWKDX5XAWL36F3TJDQNHKNTREZVTUGIS2EVZYP6DWSDA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:46.350727192 (20h17m58.826665657s) [noise=true , address=104.36.82.5:4866 , piece_id=EFRLC4LA232NZZJU3GDSP2EEBZO3W2LGJRD3L2A6SPSRBTLUTIFA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:51.702944756 (20h18m4.178883221s) [noise=true , address=104.36.82.42:4802 , piece_id=6HORHYYKIKDX4IWWKUQPVGZQZ2YYIGC47OCZHRVBC4BZFRJX5TQQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:55.250827413 (20h18m7.726765878s) [noise=true , address=104.36.82.55:4219 , piece_id=IURHWONYUYYJWIL5BDF6HLCEPA76O6QAZAJ22NCI6LQYIGO6K4IA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:57.858469692 (20h18m10.334408157s) [noise=true , address=104.36.82.53:4126 , piece_id=GZ5Z7AGMYVGYVUJAWFAIS6SQLRHFB772RLQY736QUGOXA5OWQ34Q, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:59.265385056 (20h18m11.741323511s) [noise=true , address=104.36.82.10:4088 , piece_id=6IV2Z7FXDNEOBJMNPRG7WSPZWZTJ5CGDF6JHBSF7WCIIL2QQWFUA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:59.973580263 (20h18m12.449518729s) [noise=true , address=104.36.82.11:4119 , piece_id=6KWLABSUMX6YVKD6LGKR6ONLQKFFTSTKFHHRVFIN7U4KDALBNCBQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:16.002200978 (20h18m28.478139453s) [noise=true , address=104.36.82.52:4092 , piece_id=BMJCAQYXJCUVJTEXH52UXYLD4MHWLDGQTJIPDOFRF4LPLTT36ESA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:30.906580908 (20h18m43.382519373s) [noise=true , address=104.36.82.20:4448 , piece_id=CSHYRDGTS4ROFDTKETIZ36TWJE3OEMAKLB4RGFAU557G725EANXQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:31.164952501 (20h18m43.640890966s) [noise=true , address=104.36.82.14:4227 , piece_id=O2FXJBOV3CJCKLZHRPBTLXX2QRT2OWJQ5HPNHZI5IVIFDVUUICVQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:33.000876204 (20h18m45.476814679s) [noise=true , address=104.36.82.56:4242 , piece_id=BHKC5MH4G65BK2YLJJANIYZ4MLAUTMBG32XSRISARHCO26AZUXRA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:34.725342452 (20h18m47.201280907s) [noise=true , address=104.36.82.47:4545 , piece_id=PZWVIOWLBZ72IOFRJHPAD4OZZPA2G4XYCDLGYEY7THSOR7XDHMUQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:40.451024021 (20h18m52.926962486s) [noise=true , address=104.36.82.46:4500 , piece_id=VCYKKDSLEKTTL3KO2FJ4VA5JGIVPA4GFBBHH65HPFZTP4REUTHYQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:12B7ox4ehqxJ2mwqeXj8C9N9GuSU5etNC23iT1pJnTgMLGBAGQK, address:104.36.82.46:4500): protocol: write tcp 188.166.225.170:46700->104.36.82.46:4500: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451060500 (20h18m52.926998965s) [noise=true , address=104.36.82.51:4075 , piece_id=P5NU32M3V7AMYFZVJ6XZY5K2YMQWF4EULQGX6RRYWG6SWDWWAFMQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK, address:104.36.82.51:4075): protocol: write tcp 188.166.225.170:41162->104.36.82.51:4075: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451086179 (20h18m52.927024644s) [noise=true , address=104.36.82.54:4175 , piece_id=5PAE3BM5AFSLZ5TVUQC3QK3RVWCVGFDLOK452BDPWQ7ARTL7EYWQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:12ou7jwFyJP3mNoALddniUTDT9duVyQKZjeq4xHCyx6sRWpwEPJ, address:104.36.82.54:4175): protocol: rpc: dial tcp 104.36.82.54:4175: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451120024 (20h18m52.927058489s) [noise=true , address=104.36.82.13:4214 , piece_id=SLX3YJCEWCM3553VZHILH337FKSPPFNKVFTPXO4KIBOS6IXCLXHA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:1nJ8uqwxf6x4hX52AjAdCbMKi1BKGGbMvrQbiazJdLpn6j69ay, address:104.36.82.13:4214): protocol: rpc: dial tcp 104.36.82.13:4214: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451147116 (20h18m52.927085581s) [noise=true , address=104.36.82.15:4263 , piece_id=3GCLYN4UBCLM5BPLXDQT5SSWPJBTKIOZM4UIUNO3OZIXW6OANUBQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:1f97yBEvcrYHqMq7iovVoju77jzhUYgoQCsUGULeTagugFXrCu, address:104.36.82.15:4263): protocol: write tcp 188.166.225.170:39380->104.36.82.15:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451167805 (20h18m52.927106271s) [seg_pos={0 0}, upload=19386]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 29 13:04:40.696628655 (20h18m53.17256714s) [upload=19386]: Done uploading segments... Jul 29 13:04:40.696658391 (20h18m53.172596856s) [upload=19386]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.897107913s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 1.816230964s FAIL (2025-07-29 13:37:28 UTC) orbiter warning: unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K Duration: 1.816258076s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....X...X..XX.................................X............] duration: 5.960952693s FAIL (2025-07-29 13:37:18 UTC) orbiter warning: digest mismatch: after 786d2c4be8d438b2c616b59df6d4f93d4922ed144e0f79b484237567514766eb != before 754c097f166f520f3e7f168fcc989b028ff6249f3b26903ffdb38389ae26cd4d Duration: 5.960970737s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...............X.X..........................................] duration: 5.707678221s FAIL (2025-07-29 13:36:53 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ding segments... Jul 29 13:03:41.091320246 (20h17m53.567258701s) [upload=19386]: Got next segment. Inline: false Jul 29 13:03:41.091349191 (20h17m53.567287656s) [upload=19386]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 29 13:03:41.091354331 (20h17m53.567292796s) [upload=19386]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 29 13:03:41.616444845 (20h17m54.09238331s) [seg_pos={0 0}, upload=19386]: Begin upload segment... Jul 29 13:03:41.616487748 (20h17m54.092426213s) [seg_pos={0 0}, upload=19386]: Done begin upload segment. Jul 29 13:03:41.616507376 (20h17m54.092445841s) [noise=true , address=104.36.82.15:4263 , piece_id=3GCLYN4UBCLM5BPLXDQT5SSWPJBTKIOZM4UIUNO3OZIXW6OANUBQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.616564315 (20h17m54.09250278s) [noise=true , address=104.36.82.11:4119 , piece_id=6KWLABSUMX6YVKD6LGKR6ONLQKFFTSTKFHHRVFIN7U4KDALBNCBQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.616866433 (20h17m54.092804888s) [noise=true , address=104.36.82.54:4175 , piece_id=5PAE3BM5AFSLZ5TVUQC3QK3RVWCVGFDLOK452BDPWQ7ARTL7EYWQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617072557 (20h17m54.093011022s) [noise=true , address=104.36.82.56:4242 , piece_id=BHKC5MH4G65BK2YLJJANIYZ4MLAUTMBG32XSRISARHCO26AZUXRA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617110629 (20h17m54.093049084s) [noise=true , address=104.36.82.5:4866 , piece_id=EFRLC4LA232NZZJU3GDSP2EEBZO3W2LGJRD3L2A6SPSRBTLUTIFA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617355037 (20h17m54.093293492s) [noise=true , address=104.36.82.61:4391 , piece_id=TI5YDGDSTWKDX5XAWL36F3TJDQNHKNTREZVTUGIS2EVZYP6DWSDA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617481549 (20h17m54.093420024s) [noise=true , address=104.36.82.20:4448 , piece_id=CSHYRDGTS4ROFDTKETIZ36TWJE3OEMAKLB4RGFAU557G725EANXQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617583044 (20h17m54.093521509s) [noise=true , address=104.36.82.47:4545 , piece_id=PZWVIOWLBZ72IOFRJHPAD4OZZPA2G4XYCDLGYEY7THSOR7XDHMUQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617828073 (20h17m54.093766528s) [noise=true , address=104.36.82.42:4802 , piece_id=6HORHYYKIKDX4IWWKUQPVGZQZ2YYIGC47OCZHRVBC4BZFRJX5TQQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.617987597 (20h17m54.093926062s) [noise=true , address=104.36.82.13:4214 , piece_id=SLX3YJCEWCM3553VZHILH337FKSPPFNKVFTPXO4KIBOS6IXCLXHA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618055948 (20h17m54.093994403s) [noise=true , address=104.36.82.46:4500 , piece_id=VCYKKDSLEKTTL3KO2FJ4VA5JGIVPA4GFBBHH65HPFZTP4REUTHYQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618285828 (20h17m54.094224293s) [noise=true , address=104.36.82.52:4092 , piece_id=BMJCAQYXJCUVJTEXH52UXYLD4MHWLDGQTJIPDOFRF4LPLTT36ESA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618485871 (20h17m54.094424336s) [noise=true , address=104.36.82.10:4088 , piece_id=6IV2Z7FXDNEOBJMNPRG7WSPZWZTJ5CGDF6JHBSF7WCIIL2QQWFUA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618663881 (20h17m54.094602346s) [noise=true , address=104.36.82.51:4075 , piece_id=P5NU32M3V7AMYFZVJ6XZY5K2YMQWF4EULQGX6RRYWG6SWDWWAFMQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.618840449 (20h17m54.094778914s) [noise=true , address=104.36.82.14:4227 , piece_id=O2FXJBOV3CJCKLZHRPBTLXX2QRT2OWJQ5HPNHZI5IVIFDVUUICVQ, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.619028879 (20h17m54.094967344s) [noise=true , address=104.36.82.9:4028 , piece_id=XCRNGUPBFYCZOYT23L6NHJ6WQR55TQMTWZF4SNKSRLQGC5TYO3LA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.619215837 (20h17m54.095154302s) [noise=true , address=104.36.82.53:4126 , piece_id=GZ5Z7AGMYVGYVUJAWFAIS6SQLRHFB772RLQY736QUGOXA5OWQ34Q, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:41.619408406 (20h17m54.095346871s) [noise=true , address=104.36.82.55:4219 , piece_id=IURHWONYUYYJWIL5BDF6HLCEPA76O6QAZAJ22NCI6LQYIGO6K4IA, seg_pos={0 0}, upload=19386]: Uploading piece... Jul 29 13:03:43.299790904 (20h17m55.775729369s) [upload=19386]: Next returned nil segment Jul 29 13:03:43.299806024 (20h17m55.775744479s) [upload=19386]: Waiting for error group managing segments... Jul 29 13:03:43.847040928 (20h17m56.322979393s) [noise=true , address=104.36.82.9:4028 , piece_id=XCRNGUPBFYCZOYT23L6NHJ6WQR55TQMTWZF4SNKSRLQGC5TYO3LA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:43.879495404 (20h17m56.355433869s) [noise=true , address=104.36.82.61:4391 , piece_id=TI5YDGDSTWKDX5XAWL36F3TJDQNHKNTREZVTUGIS2EVZYP6DWSDA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:46.350727192 (20h17m58.826665657s) [noise=true , address=104.36.82.5:4866 , piece_id=EFRLC4LA232NZZJU3GDSP2EEBZO3W2LGJRD3L2A6SPSRBTLUTIFA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:51.702944756 (20h18m4.178883221s) [noise=true , address=104.36.82.42:4802 , piece_id=6HORHYYKIKDX4IWWKUQPVGZQZ2YYIGC47OCZHRVBC4BZFRJX5TQQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:55.250827413 (20h18m7.726765878s) [noise=true , address=104.36.82.55:4219 , piece_id=IURHWONYUYYJWIL5BDF6HLCEPA76O6QAZAJ22NCI6LQYIGO6K4IA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:57.858469692 (20h18m10.334408157s) [noise=true , address=104.36.82.53:4126 , piece_id=GZ5Z7AGMYVGYVUJAWFAIS6SQLRHFB772RLQY736QUGOXA5OWQ34Q, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:59.265385056 (20h18m11.741323511s) [noise=true , address=104.36.82.10:4088 , piece_id=6IV2Z7FXDNEOBJMNPRG7WSPZWZTJ5CGDF6JHBSF7WCIIL2QQWFUA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:03:59.973580263 (20h18m12.449518729s) [noise=true , address=104.36.82.11:4119 , piece_id=6KWLABSUMX6YVKD6LGKR6ONLQKFFTSTKFHHRVFIN7U4KDALBNCBQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:16.002200978 (20h18m28.478139453s) [noise=true , address=104.36.82.52:4092 , piece_id=BMJCAQYXJCUVJTEXH52UXYLD4MHWLDGQTJIPDOFRF4LPLTT36ESA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:30.906580908 (20h18m43.382519373s) [noise=true , address=104.36.82.20:4448 , piece_id=CSHYRDGTS4ROFDTKETIZ36TWJE3OEMAKLB4RGFAU557G725EANXQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:31.164952501 (20h18m43.640890966s) [noise=true , address=104.36.82.14:4227 , piece_id=O2FXJBOV3CJCKLZHRPBTLXX2QRT2OWJQ5HPNHZI5IVIFDVUUICVQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:33.000876204 (20h18m45.476814679s) [noise=true , address=104.36.82.56:4242 , piece_id=BHKC5MH4G65BK2YLJJANIYZ4MLAUTMBG32XSRISARHCO26AZUXRA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:34.725342452 (20h18m47.201280907s) [noise=true , address=104.36.82.47:4545 , piece_id=PZWVIOWLBZ72IOFRJHPAD4OZZPA2G4XYCDLGYEY7THSOR7XDHMUQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: <nil> Jul 29 13:04:40.451024021 (20h18m52.926962486s) [noise=true , address=104.36.82.46:4500 , piece_id=VCYKKDSLEKTTL3KO2FJ4VA5JGIVPA4GFBBHH65HPFZTP4REUTHYQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:12B7ox4ehqxJ2mwqeXj8C9N9GuSU5etNC23iT1pJnTgMLGBAGQK, address:104.36.82.46:4500): protocol: write tcp 188.166.225.170:46700->104.36.82.46:4500: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451060500 (20h18m52.926998965s) [noise=true , address=104.36.82.51:4075 , piece_id=P5NU32M3V7AMYFZVJ6XZY5K2YMQWF4EULQGX6RRYWG6SWDWWAFMQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK, address:104.36.82.51:4075): protocol: write tcp 188.166.225.170:41162->104.36.82.51:4075: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451086179 (20h18m52.927024644s) [noise=true , address=104.36.82.54:4175 , piece_id=5PAE3BM5AFSLZ5TVUQC3QK3RVWCVGFDLOK452BDPWQ7ARTL7EYWQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:12ou7jwFyJP3mNoALddniUTDT9duVyQKZjeq4xHCyx6sRWpwEPJ, address:104.36.82.54:4175): protocol: rpc: dial tcp 104.36.82.54:4175: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451120024 (20h18m52.927058489s) [noise=true , address=104.36.82.13:4214 , piece_id=SLX3YJCEWCM3553VZHILH337FKSPPFNKVFTPXO4KIBOS6IXCLXHA, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:1nJ8uqwxf6x4hX52AjAdCbMKi1BKGGbMvrQbiazJdLpn6j69ay, address:104.36.82.13:4214): protocol: rpc: dial tcp 104.36.82.13:4214: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451147116 (20h18m52.927085581s) [noise=true , address=104.36.82.15:4263 , piece_id=3GCLYN4UBCLM5BPLXDQT5SSWPJBTKIOZM4UIUNO3OZIXW6OANUBQ, seg_pos={0 0}, upload=19386]: Done uploading piece. err: ecclient: upload failed (node:1f97yBEvcrYHqMq7iovVoju77jzhUYgoQCsUGULeTagugFXrCu, address:104.36.82.15:4263): protocol: write tcp 188.166.225.170:39380->104.36.82.15:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 29 13:04:40.451167805 (20h18m52.927106271s) [seg_pos={0 0}, upload=19386]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 29 13:04:40.696628655 (20h18m53.17256714s) [upload=19386]: Done uploading segments... Jul 29 13:04:40.696658391 (20h18m53.172596856s) [upload=19386]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.707704622s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 1.828886388s FAIL (2025-07-29 13:36:53 UTC) orbiter warning: unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K Duration: 1.828910786s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [......X...X..XX.................................X...........] duration: 5.880155919s FAIL (2025-07-29 13:36:32 UTC) orbiter warning: digest mismatch: after 786d2c4be8d438b2c616b59df6d4f93d4922ed144e0f79b484237567514766eb != before 754c097f166f520f3e7f168fcc989b028ff6249f3b26903ffdb38389ae26cd4d Duration: 5.880182308s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......X...X..XX.................................X..........] duration: 6.307032529s FAIL (2025-07-29 13:36:16 UTC) orbiter warning: digest mismatch: after 786d2c4be8d438b2c616b59df6d4f93d4922ed144e0f79b484237567514766eb != before 754c097f166f520f3e7f168fcc989b028ff6249f3b26903ffdb38389ae26cd4d Duration: 6.307074007s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)