Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 51 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..XX.X.XXX.XXXXXX...XX.X] duration: 30.261240024s FAIL (2025-07-27 00:15:20 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.261283407s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 1.864476784s FAIL (2025-07-27 00:15:08 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.864496091s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 52 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXX.XXXXXXXXXX.XXXX..X..XX.] duration: 5.786423553s FAIL (2025-07-27 00:14:45 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 3DZRCZIGKKAFDENB3KZPRECDXW2EDTLA, seg_pos={0 0}, upload=58125]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: rpc: dial tcp 104.36.82.14:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:12:44.483229939 (60h36m54.205792979s) [seg_pos={0 0}, upload=58125]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 27 00:12:44.713369970 (60h36m54.43593301s) [upload=58125]: Done uploading segments... Jul 27 00:12:44.713400107 (60h36m54.435963178s) [upload=58125]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Jul 27 00:13:29.432607376 (60h37m39.155170477s) [upload=58146]: Starting upload Jul 27 00:13:29.432661901 (60h37m39.155224931s) [upload=58146]: Done starting upload Jul 27 00:13:29.432724671 (60h37m39.155287741s) [upload=58146]: Uploading segments... Jul 27 00:13:29.432835032 (60h37m39.155398072s) [upload=58146]: Got next segment. Inline: false Jul 27 00:13:29.432862134 (60h37m39.155425174s) [upload=58146]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:13:29.432866161 (60h37m39.155429201s) [upload=58146]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:13:29.882545201 (60h37m39.605108231s) [seg_pos={0 0}, upload=58146]: Begin upload segment... Jul 27 00:13:29.882588954 (60h37m39.605151994s) [seg_pos={0 0}, upload=58146]: Done begin upload segment. Jul 27 00:13:29.882609453 (60h37m39.605172493s) [noise=true , address=104.36.82.13:4186 , piece_id=HAS3YDK5DG5URSE3Z2QRNX6HO3H3RYW7CBHBAG7J2UMTWYFIK4DA, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:29.882655722 (60h37m39.605218762s) [noise=true , address=77.90.31.16:4733 , piece_id=6GNJSP4EN72D24BIWOD7PGXHFKZK6ZXBIH4TRKUY5RFCZMDOGBJA, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:29.882981065 (60h37m39.605544105s) [noise=true , address=45.91.116.12:4225 , piece_id=FNVXPLUIBJRZJDMUZIQFYOSO2IJ34NBVDOPNTWRIJGCDFCG7PCMQ, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:29.883123006 (60h37m39.605686036s) [noise=true , address=45.91.116.11:4117 , piece_id=KQL7E6TQ3MYQYIL3B36ZOF67SDKBVR6ABWKSA3JKMLWKYG46GAUQ, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:29.883299494 (60h37m39.605862534s) [noise=true , address=104.36.82.4:4802 , piece_id=2ZTVFJQ2PR4NQMD4BYR6ZCXKPTBY5OGLSF2UHR4YILEVBWMLVDDQ, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:29.883512962 (60h37m39.606076002s) [noise=true , address=45.91.116.11:4119 , piece_id=V7U4FATH7XZ23LS5GT55ICBMHPE66IOLQJXIQLG73P5PK2SEJTHQ, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:29.883691324 (60h37m39.606254364s) [noise=true , address=77.90.31.18:4807 , piece_id=P2Y4AZHOAMKSE7XAXTZKBBMZRIQL2X275MA2WQ3QL747GN7I3BAQ, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:29.883927075 (60h37m39.606490115s) [noise=true , address=77.90.31.12:4592 , piece_id=CNXHVVT66TTFMDZDTTBP5UYIMVJKQUAOACPVALNWC5D6ZZBQAHGQ, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:29.884177864 (60h37m39.606740904s) [noise=true , address=104.36.82.2:4732 , piece_id=5ARCTJ6Y77YFVLIO6CIZKKJAW42YMYJPM6C42U4A7WVYRCYMAOCA, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:29.884390321 (60h37m39.606953351s) [noise=true , address=45.91.116.7:4011 , piece_id=2FPMK6CEORJE2I6NVGWUOZZJGGJAGO72TQZA5HIUWJ2AHKLVZLFA, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:29.884591375 (60h37m39.607154415s) [noise=true , address=45.91.116.10:4082 , piece_id=ZSIXJUW6D7KXY2JX33TK2F3JQVIRQXPQ754FAJDZVEN6JJ46XZSA, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:29.884736734 (60h37m39.607299774s) [noise=true , address=45.91.116.9:4262 , piece_id=OWJD7LGSVXFSUKHNUSDFRQ235P2XIBJOJG54FN7INRFSNG376C5Q, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:29.885020567 (60h37m39.607583607s) [noise=true , address=77.90.31.13:4626 , piece_id=T34BL7CW2T6NRA2IC74TLNAUBXCRO6LFDCVRQWYCHSU3YV2ECAUQ, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:29.885295232 (60h37m39.607858262s) [noise=true , address=104.36.82.14:4221 , piece_id=25ZT65VYYWQNVEOD4TIXA3P6QH6UPTN3PCQSOSXDZ3COYIGK34UA, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:29.885414510 (60h37m39.60797755s) [noise=true , address=77.90.31.17:4771 , piece_id=BXGHM4HAHAB2I43I4COYUTB7CSJPESSSLFPWG3SY4ZF3FDXXZQQA, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:29.885627789 (60h37m39.608190839s) [noise=true , address=77.90.31.14:4661 , piece_id=56HNJRJ6YUT3WYHB6GPEOIFPIOF4XUQTXBLOC4DYLYFRZHEUA2PA, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:31.629985152 (60h37m41.352548182s) [upload=58146]: Next returned nil segment Jul 27 00:13:31.630002535 (60h37m41.352565575s) [upload=58146]: Waiting for error group managing segments... Jul 27 00:13:31.908463617 (60h37m41.631026647s) [noise=true , address=45.91.116.12:4225 , piece_id=FNVXPLUIBJRZJDMUZIQFYOSO2IJ34NBVDOPNTWRIJGCDFCG7PCMQ, seg_pos={0 0}, upload=58146]: Done uploading piece. err: <nil> Jul 27 00:13:31.977161876 (60h37m41.699724916s) [noise=true , address=77.90.31.17:4771 , piece_id=BXGHM4HAHAB2I43I4COYUTB7CSJPESSSLFPWG3SY4ZF3FDXXZQQA, seg_pos={0 0}, upload=58146]: Done uploading piece. err: <nil> Jul 27 00:13:31.994378346 (60h37m41.716941386s) [noise=true , address=77.90.31.16:4733 , piece_id=6GNJSP4EN72D24BIWOD7PGXHFKZK6ZXBIH4TRKUY5RFCZMDOGBJA, seg_pos={0 0}, upload=58146]: Done uploading piece. err: <nil> Jul 27 00:13:31.995792631 (60h37m41.718355661s) [noise=true , address=45.91.116.9:4262 , piece_id=OWJD7LGSVXFSUKHNUSDFRQ235P2XIBJOJG54FN7INRFSNG376C5Q, seg_pos={0 0}, upload=58146]: Done uploading piece. err: <nil> Jul 27 00:13:31.998736975 (60h37m41.721300015s) [noise=true , address=45.91.116.10:4082 , piece_id=ZSIXJUW6D7KXY2JX33TK2F3JQVIRQXPQ754FAJDZVEN6JJ46XZSA, seg_pos={0 0}, upload=58146]: Done uploading piece. err: <nil> Jul 27 00:13:32.045466081 (60h37m41.768029121s) [noise=true , address=45.91.116.7:4011 , piece_id=2FPMK6CEORJE2I6NVGWUOZZJGGJAGO72TQZA5HIUWJ2AHKLVZLFA, seg_pos={0 0}, upload=58146]: Done uploading piece. err: <nil> Jul 27 00:13:32.052034557 (60h37m41.774597598s) [noise=true , address=45.91.116.11:4119 , piece_id=V7U4FATH7XZ23LS5GT55ICBMHPE66IOLQJXIQLG73P5PK2SEJTHQ, seg_pos={0 0}, upload=58146]: Done uploading piece. err: <nil> Jul 27 00:13:32.067646859 (60h37m41.790209899s) [noise=true , address=77.90.31.14:4661 , piece_id=56HNJRJ6YUT3WYHB6GPEOIFPIOF4XUQTXBLOC4DYLYFRZHEUA2PA, seg_pos={0 0}, upload=58146]: Done uploading piece. err: <nil> Jul 27 00:13:32.075787915 (60h37m41.798350955s) [noise=true , address=45.91.116.11:4117 , piece_id=KQL7E6TQ3MYQYIL3B36ZOF67SDKBVR6ABWKSA3JKMLWKYG46GAUQ, seg_pos={0 0}, upload=58146]: Done uploading piece. err: <nil> Jul 27 00:13:33.403628758 (60h37m43.126191798s) [noise=true , address=77.90.31.12:4592 , piece_id=CNXHVVT66TTFMDZDTTBP5UYIMVJKQUAOACPVALNWC5D6ZZBQAHGQ, seg_pos={0 0}, upload=58146]: Done uploading piece. err: <nil> Jul 27 00:13:33.460620549 (60h37m43.183183579s) [noise=true , address=77.90.31.13:4626 , piece_id=T34BL7CW2T6NRA2IC74TLNAUBXCRO6LFDCVRQWYCHSU3YV2ECAUQ, seg_pos={0 0}, upload=58146]: Done uploading piece. err: <nil> Jul 27 00:13:58.786001067 (60h38m8.508564107s) [noise=true , address=104.36.82.13:4186 , piece_id=HAS3YDK5DG5URSE3Z2QRNX6HO3H3RYW7CBHBAG7J2UMTWYFIK4DA, seg_pos={0 0}, upload=58146]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:13:58.786085258 (60h38m8.508648288s) [noise=true , address=104.36.82.4:4802 , piece_id=2ZTVFJQ2PR4NQMD4BYR6ZCXKPTBY5OGLSF2UHR4YILEVBWMLVDDQ, seg_pos={0 0}, upload=58146]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:13:58.786147937 (60h38m8.508710978s) [noise=true , address=104.36.82.2:4732 , piece_id=5ARCTJ6Y77YFVLIO6CIZKKJAW42YMYJPM6C42U4A7WVYRCYMAOCA, seg_pos={0 0}, upload=58146]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: rpc: dial tcp 104.36.82.2:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:13:58.786210106 (60h38m8.508773146s) [noise=true , address=77.90.31.18:4807 , piece_id=P2Y4AZHOAMKSE7XAXTZKBBMZRIQL2X275MA2WQ3QL747GN7I3BAQ, seg_pos={0 0}, upload=58146]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:13:58.786391444 (60h38m8.508954484s) [noise=true , address=104.36.82.14:4221 , piece_id=25ZT65VYYWQNVEOD4TIXA3P6QH6UPTN3PCQSOSXDZ3COYIGK34UA, seg_pos={0 0}, upload=58146]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: write tcp 188.166.225.170:48452->104.36.82.14:4221: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:13:58.786437472 (60h38m8.509000502s) [seg_pos={0 0}, upload=58146]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 27 00:13:59.022092002 (60h38m8.744655042s) [upload=58146]: Done uploading segments... Jul 27 00:13:59.022115117 (60h38m8.744678157s) [upload=58146]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.786453059s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.235673409s FAIL (2025-07-27 00:14:31 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 28416903 (60h37m41.650979943s) [seg_pos={0 0}, upload=58147]: Done begin upload segment. Jul 27 00:13:31.928440568 (60h37m41.651003598s) [noise=true , address=104.36.82.56:4236 , piece_id=SU2VYBUWD7AEGLINFQYY5QNIGOBYM5WV5S7AKTAKVDEJQBHT3GIA, seg_pos={0 0}, upload=58147]: Uploading piece... Jul 27 00:13:31.928472249 (60h37m41.65103528s) [noise=true , address=104.36.82.54:4178 , piece_id=YF7XKEKKYSTCCGCONGSPIAKW56ZETJX35KP2YIFUKQJRJ3OPEDGA, seg_pos={0 0}, upload=58147]: Uploading piece... Jul 27 00:13:31.928832559 (60h37m41.651395589s) [noise=true , address=45.91.116.14:4206 , piece_id=JCUIKJVZTKHHM2BEUVCNQPSMJQZAY2Z4O256UVYIX2BOM4VPAKSQ, seg_pos={0 0}, upload=58147]: Uploading piece... Jul 27 00:13:31.928888486 (60h37m41.651451526s) [noise=true , address=156.146.43.225:4448 , piece_id=R3EP7X762SBMUQXNYGBWGHN4YUO3NW5HJPB35ILDNBHM5N36FDPA, seg_pos={0 0}, upload=58147]: Uploading piece... Jul 27 00:13:31.929108517 (60h37m41.651671557s) [noise=true , address=77.90.31.13:4643 , piece_id=WBG7EOI56MVA5LSKQIOMFUOTG4Q6TYO5NOHLCUUEWTOF3QTSCJEA, seg_pos={0 0}, upload=58147]: Uploading piece... Jul 27 00:13:31.929371089 (60h37m41.651934129s) [noise=true , address=79.127.219.48:4074 , piece_id=3NPXRUXYUN2PSC2VPRAGCZXGNJ6K2VYQMH2KACVFDEQCJLTTDTPQ, seg_pos={0 0}, upload=58147]: Uploading piece... Jul 27 00:13:31.929644582 (60h37m41.652207622s) [noise=true , address=104.36.82.55:4221 , piece_id=IUFDFPVN3WPSDFGXQ77XFLRYQ3MO6Y4NJPQ5AK5OUV7V73ZWZBYQ, seg_pos={0 0}, upload=58147]: Uploading piece... Jul 27 00:13:31.930034808 (60h37m41.652597848s) [noise=true , address=104.36.82.52:4085 , piece_id=NQM7ZAJBUA3P2DYREEQTFWBBIRQOFASAF7MWUDSVG24TFEYY3X5A, seg_pos={0 0}, upload=58147]: Uploading piece... Jul 27 00:13:31.930361453 (60h37m41.652924493s) [noise=true , address=79.127.205.235:4685 , piece_id=OVNY3HICKYNJYZ3GW4HWO4PZDWVOXGKPAO6AW5M6FKSMCMXGBFOQ, seg_pos={0 0}, upload=58147]: Uploading piece... Jul 27 00:13:31.930653773 (60h37m41.653216813s) [noise=true , address=77.90.31.15:4717 , piece_id=BBLZR3D5I5DBKWLPIUGZXLN4CKO2P2YTKAPNJMAXURSNFOV6FHSQ, seg_pos={0 0}, upload=58147]: Uploading piece... Jul 27 00:13:31.930894213 (60h37m41.653457253s) [noise=true , address=77.90.31.16:4737 , piece_id=KBJHI2NZFQOOJBLIZTOBXZ2YWYHTRYD3XT4R32ZKANSGKZXCCP2Q, seg_pos={0 0}, upload=58147]: Uploading piece... Jul 27 00:13:31.931108022 (60h37m41.653671062s) [noise=true , address=79.127.201.215:4353 , piece_id=U4VO7FBOQVXCF3AKCLZWKRGF3F6DFTTOCWLG2EGWKVXKPZ2AOXAQ, seg_pos={0 0}, upload=58147]: Uploading piece... Jul 27 00:13:31.931359473 (60h37m41.653922513s) [noise=true , address=79.127.219.38:4004 , piece_id=HCMN2IFXDW3DMTSJWYUWZQKNQFHARZZAFEGH36JBFY6FUAKK7IIQ, seg_pos={0 0}, upload=58147]: Uploading piece... Jul 27 00:13:31.931576077 (60h37m41.654139117s) [noise=true , address=64.49.13.138:4482 , piece_id=YA2UF5GDSI5RJQVQK4OEIP6MHOEVCDCNVPGATKPXXIDIM3CPTZYA, seg_pos={0 0}, upload=58147]: Uploading piece... Jul 27 00:13:31.931788273 (60h37m41.654351314s) [noise=true , address=79.127.205.233:4639 , piece_id=ENQHRZAA2PTHZQGCFT6UMSWN3XDT3AIKCNRYPAC4L2J6XRN3MABQ, seg_pos={0 0}, upload=58147]: Uploading piece... Jul 27 00:13:31.932060815 (60h37m41.654623855s) [noise=true , address=104.36.82.44:4886 , piece_id=7PJECFYRA2F44D4QT2CKSLYQQQ23EIYQ6GBUVUJWJDJBK5DYVIRQ, seg_pos={0 0}, upload=58147]: Uploading piece... Jul 27 00:13:31.932312446 (60h37m41.654875486s) [noise=true , address=104.36.82.53:4150 , piece_id=NRJTT22H7ZUNQIZLFRSLLWWQR3S3BANFSYWSYVRC5SIALFX6OJ4A, seg_pos={0 0}, upload=58147]: Uploading piece... Jul 27 00:13:31.932543339 (60h37m41.655106379s) [noise=true , address=77.90.31.10:4525 , piece_id=6TPS42S3CQQHF4N3LZ4345MTJATCT7FBJLPMUOUQQMXGBBZZHOPQ, seg_pos={0 0}, upload=58147]: Uploading piece... Jul 27 00:13:32.958467822 (60h37m42.681030863s) [noise=true , address=156.146.43.225:4448 , piece_id=R3EP7X762SBMUQXNYGBWGHN4YUO3NW5HJPB35ILDNBHM5N36FDPA, seg_pos={0 0}, upload=58147]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jul 27 00:13:33.047452184 (60h37m42.770015224s) [upload=58147]: Next returned nil segment Jul 27 00:13:33.047490968 (60h37m42.770054008s) [upload=58147]: Waiting for error group managing segments... Jul 27 00:13:33.330630772 (60h37m43.053193842s) [noise=true , address=79.127.205.233:4639 , piece_id=ENQHRZAA2PTHZQGCFT6UMSWN3XDT3AIKCNRYPAC4L2J6XRN3MABQ, seg_pos={0 0}, upload=58147]: Done uploading piece. err: <nil> Jul 27 00:13:33.348175059 (60h37m43.070738099s) [noise=true , address=79.127.205.235:4685 , piece_id=OVNY3HICKYNJYZ3GW4HWO4PZDWVOXGKPAO6AW5M6FKSMCMXGBFOQ, seg_pos={0 0}, upload=58147]: Done uploading piece. err: <nil> Jul 27 00:13:33.369863495 (60h37m43.092426565s) [noise=true , address=64.49.13.138:4482 , piece_id=YA2UF5GDSI5RJQVQK4OEIP6MHOEVCDCNVPGATKPXXIDIM3CPTZYA, seg_pos={0 0}, upload=58147]: Done uploading piece. err: <nil> Jul 27 00:13:33.951196995 (60h37m43.673760025s) [noise=true , address=77.90.31.15:4717 , piece_id=BBLZR3D5I5DBKWLPIUGZXLN4CKO2P2YTKAPNJMAXURSNFOV6FHSQ, seg_pos={0 0}, upload=58147]: Done uploading piece. err: <nil> Jul 27 00:13:33.996608721 (60h37m43.719171751s) [noise=true , address=79.127.219.38:4004 , piece_id=HCMN2IFXDW3DMTSJWYUWZQKNQFHARZZAFEGH36JBFY6FUAKK7IIQ, seg_pos={0 0}, upload=58147]: Done uploading piece. err: <nil> Jul 27 00:13:34.063162086 (60h37m43.785725126s) [noise=true , address=77.90.31.16:4737 , piece_id=KBJHI2NZFQOOJBLIZTOBXZ2YWYHTRYD3XT4R32ZKANSGKZXCCP2Q, seg_pos={0 0}, upload=58147]: Done uploading piece. err: <nil> Jul 27 00:13:34.255635539 (60h37m43.978198599s) [noise=true , address=79.127.219.48:4074 , piece_id=3NPXRUXYUN2PSC2VPRAGCZXGNJ6K2VYQMH2KACVFDEQCJLTTDTPQ, seg_pos={0 0}, upload=58147]: Done uploading piece. err: <nil> Jul 27 00:13:35.905447101 (60h37m45.628010131s) [noise=true , address=45.91.116.14:4206 , piece_id=JCUIKJVZTKHHM2BEUVCNQPSMJQZAY2Z4O256UVYIX2BOM4VPAKSQ, seg_pos={0 0}, upload=58147]: Done uploading piece. err: <nil> Jul 27 00:13:47.894801826 (60h37m57.617364866s) [noise=true , address=79.127.201.215:4353 , piece_id=U4VO7FBOQVXCF3AKCLZWKRGF3F6DFTTOCWLG2EGWKVXKPZ2AOXAQ, seg_pos={0 0}, upload=58147]: Done uploading piece. err: <nil> Jul 27 00:13:48.375089258 (60h37m58.097652298s) [noise=true , address=77.90.31.13:4643 , piece_id=WBG7EOI56MVA5LSKQIOMFUOTG4Q6TYO5NOHLCUUEWTOF3QTSCJEA, seg_pos={0 0}, upload=58147]: Done uploading piece. err: <nil> Jul 27 00:13:58.109318546 (60h38m7.831881576s) [noise=true , address=77.90.31.10:4525 , piece_id=6TPS42S3CQQHF4N3LZ4345MTJATCT7FBJLPMUOUQQMXGBBZZHOPQ, seg_pos={0 0}, upload=58147]: Done uploading piece. err: <nil> Jul 27 00:14:30.836823416 (60h38m40.559386466s) [noise=true , address=104.36.82.52:4085 , piece_id=NQM7ZAJBUA3P2DYREEQTFWBBIRQOFASAF7MWUDSVG24TFEYY3X5A, seg_pos={0 0}, upload=58147]: Done uploading piece. err: ecclient: upload failed (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C, address:104.36.82.52:4085): protocol: write tcp 188.166.225.170:37954->104.36.82.52:4085: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:14:30.836916514 (60h38m40.559479575s) [noise=true , address=104.36.82.56:4236 , piece_id=SU2VYBUWD7AEGLINFQYY5QNIGOBYM5WV5S7AKTAKVDEJQBHT3GIA, seg_pos={0 0}, upload=58147]: Done uploading piece. err: ecclient: upload failed (node:1pRLRqP5JEepqEKV9R3BUUhztQXnakq8QKMNjBZ1KR5TMMAXAb, address:104.36.82.56:4236): protocol: write tcp 188.166.225.170:51754->104.36.82.56:4236: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:14:30.837056552 (60h38m40.559619612s) [noise=true , address=104.36.82.44:4886 , piece_id=7PJECFYRA2F44D4QT2CKSLYQQQ23EIYQ6GBUVUJWJDJBK5DYVIRQ, seg_pos={0 0}, upload=58147]: Done uploading piece. err: ecclient: upload failed (node:1n2cXhqd5EWbNPTXem83jQUjYq5ZaGKBidbVCQzxYfR3rWddGR, address:104.36.82.44:4886): protocol: rpc: dial tcp 104.36.82.44:4886: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:14:30.837158728 (60h38m40.559721808s) [noise=true , address=104.36.82.55:4221 , piece_id=IUFDFPVN3WPSDFGXQ77XFLRYQ3MO6Y4NJPQ5AK5OUV7V73ZWZBYQ, seg_pos={0 0}, upload=58147]: Done uploading piece. err: ecclient: upload failed (node:1ssFiXk5oKTgnVMJ5TBJuSXuD1D2C46gT2hcY6yaDxf8fL1cjm, address:104.36.82.55:4221): protocol: write tcp 188.166.225.170:58344->104.36.82.55:4221: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:14:30.837184107 (60h38m40.559747187s) [noise=true , address=104.36.82.53:4150 , piece_id=NRJTT22H7ZUNQIZLFRSLLWWQR3S3BANFSYWSYVRC5SIALFX6OJ4A, seg_pos={0 0}, upload=58147]: Done uploading piece. err: ecclient: upload failed (node:1jDz53H9KVPn5dBA1gqj66ZQEKbo9dNLucS9c3u8WvY1Xn7jH, address:104.36.82.53:4150): protocol: write tcp 188.166.225.170:55702->104.36.82.53:4150: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:14:30.837234262 (60h38m40.559797302s) [noise=true , address=104.36.82.54:4178 , piece_id=YF7XKEKKYSTCCGCONGSPIAKW56ZETJX35KP2YIFUKQJRJ3OPEDGA, seg_pos={0 0}, upload=58147]: Done uploading piece. err: ecclient: upload failed (node:1jyRCtxSCu2bkU41Kgb1aeGppQLgX5iEo322Zoi1cWFkmGeYgr, address:104.36.82.54:4178): protocol: write tcp 188.166.225.170:48602->104.36.82.54:4178: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:14:30.837267857 (60h38m40.559830927s) [seg_pos={0 0}, upload=58147]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 27 00:14:31.070880202 (60h38m40.793443242s) [upload=58147]: Done uploading segments... Jul 27 00:14:31.070906612 (60h38m40.793469642s) [upload=58147]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1m0.235686033s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [.X.....X.X..X..X.X....X.X....XXX.X...X......X.XX...XXXX.....] duration: 7.305369763s FAIL (2025-07-27 00:14:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.305406684s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 1.831017177s FAIL (2025-07-27 00:14:26 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.831049789s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [X.X.....X.X..X..X.X....X.X....XXX.X...X......X.XX...XXXX....] duration: 7.252766366s FAIL (2025-07-27 00:14:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.252984685s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 16 out of 60 is failed (threshold=3) [.....XXX..X.....X......X...XX..XX.X.XX...X...............XX.] duration: 29.825325554s FAIL (2025-07-27 00:14:03 UTC) orbiter warning: context deadline exceeded Duration: 29.825338108s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 53 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXX.XXXXXXXXXX.XXXX..X..XX] duration: 30.237561816s FAIL (2025-07-27 00:13:59 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 3DZRCZIGKKAFDENB3KZPRECDXW2EDTLA, seg_pos={0 0}, upload=58125]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: rpc: dial tcp 104.36.82.14:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:12:44.483229939 (60h36m54.205792979s) [seg_pos={0 0}, upload=58125]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 27 00:12:44.713369970 (60h36m54.43593301s) [upload=58125]: Done uploading segments... Jul 27 00:12:44.713400107 (60h36m54.435963178s) [upload=58125]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Jul 27 00:13:29.432607376 (60h37m39.155170477s) [upload=58146]: Starting upload Jul 27 00:13:29.432661901 (60h37m39.155224931s) [upload=58146]: Done starting upload Jul 27 00:13:29.432724671 (60h37m39.155287741s) [upload=58146]: Uploading segments... Jul 27 00:13:29.432835032 (60h37m39.155398072s) [upload=58146]: Got next segment. Inline: false Jul 27 00:13:29.432862134 (60h37m39.155425174s) [upload=58146]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:13:29.432866161 (60h37m39.155429201s) [upload=58146]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:13:29.882545201 (60h37m39.605108231s) [seg_pos={0 0}, upload=58146]: Begin upload segment... Jul 27 00:13:29.882588954 (60h37m39.605151994s) [seg_pos={0 0}, upload=58146]: Done begin upload segment. Jul 27 00:13:29.882609453 (60h37m39.605172493s) [noise=true , address=104.36.82.13:4186 , piece_id=HAS3YDK5DG5URSE3Z2QRNX6HO3H3RYW7CBHBAG7J2UMTWYFIK4DA, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:29.882655722 (60h37m39.605218762s) [noise=true , address=77.90.31.16:4733 , piece_id=6GNJSP4EN72D24BIWOD7PGXHFKZK6ZXBIH4TRKUY5RFCZMDOGBJA, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:29.882981065 (60h37m39.605544105s) [noise=true , address=45.91.116.12:4225 , piece_id=FNVXPLUIBJRZJDMUZIQFYOSO2IJ34NBVDOPNTWRIJGCDFCG7PCMQ, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:29.883123006 (60h37m39.605686036s) [noise=true , address=45.91.116.11:4117 , piece_id=KQL7E6TQ3MYQYIL3B36ZOF67SDKBVR6ABWKSA3JKMLWKYG46GAUQ, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:29.883299494 (60h37m39.605862534s) [noise=true , address=104.36.82.4:4802 , piece_id=2ZTVFJQ2PR4NQMD4BYR6ZCXKPTBY5OGLSF2UHR4YILEVBWMLVDDQ, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:29.883512962 (60h37m39.606076002s) [noise=true , address=45.91.116.11:4119 , piece_id=V7U4FATH7XZ23LS5GT55ICBMHPE66IOLQJXIQLG73P5PK2SEJTHQ, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:29.883691324 (60h37m39.606254364s) [noise=true , address=77.90.31.18:4807 , piece_id=P2Y4AZHOAMKSE7XAXTZKBBMZRIQL2X275MA2WQ3QL747GN7I3BAQ, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:29.883927075 (60h37m39.606490115s) [noise=true , address=77.90.31.12:4592 , piece_id=CNXHVVT66TTFMDZDTTBP5UYIMVJKQUAOACPVALNWC5D6ZZBQAHGQ, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:29.884177864 (60h37m39.606740904s) [noise=true , address=104.36.82.2:4732 , piece_id=5ARCTJ6Y77YFVLIO6CIZKKJAW42YMYJPM6C42U4A7WVYRCYMAOCA, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:29.884390321 (60h37m39.606953351s) [noise=true , address=45.91.116.7:4011 , piece_id=2FPMK6CEORJE2I6NVGWUOZZJGGJAGO72TQZA5HIUWJ2AHKLVZLFA, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:29.884591375 (60h37m39.607154415s) [noise=true , address=45.91.116.10:4082 , piece_id=ZSIXJUW6D7KXY2JX33TK2F3JQVIRQXPQ754FAJDZVEN6JJ46XZSA, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:29.884736734 (60h37m39.607299774s) [noise=true , address=45.91.116.9:4262 , piece_id=OWJD7LGSVXFSUKHNUSDFRQ235P2XIBJOJG54FN7INRFSNG376C5Q, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:29.885020567 (60h37m39.607583607s) [noise=true , address=77.90.31.13:4626 , piece_id=T34BL7CW2T6NRA2IC74TLNAUBXCRO6LFDCVRQWYCHSU3YV2ECAUQ, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:29.885295232 (60h37m39.607858262s) [noise=true , address=104.36.82.14:4221 , piece_id=25ZT65VYYWQNVEOD4TIXA3P6QH6UPTN3PCQSOSXDZ3COYIGK34UA, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:29.885414510 (60h37m39.60797755s) [noise=true , address=77.90.31.17:4771 , piece_id=BXGHM4HAHAB2I43I4COYUTB7CSJPESSSLFPWG3SY4ZF3FDXXZQQA, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:29.885627789 (60h37m39.608190839s) [noise=true , address=77.90.31.14:4661 , piece_id=56HNJRJ6YUT3WYHB6GPEOIFPIOF4XUQTXBLOC4DYLYFRZHEUA2PA, seg_pos={0 0}, upload=58146]: Uploading piece... Jul 27 00:13:31.629985152 (60h37m41.352548182s) [upload=58146]: Next returned nil segment Jul 27 00:13:31.630002535 (60h37m41.352565575s) [upload=58146]: Waiting for error group managing segments... Jul 27 00:13:31.908463617 (60h37m41.631026647s) [noise=true , address=45.91.116.12:4225 , piece_id=FNVXPLUIBJRZJDMUZIQFYOSO2IJ34NBVDOPNTWRIJGCDFCG7PCMQ, seg_pos={0 0}, upload=58146]: Done uploading piece. err: <nil> Jul 27 00:13:31.977161876 (60h37m41.699724916s) [noise=true , address=77.90.31.17:4771 , piece_id=BXGHM4HAHAB2I43I4COYUTB7CSJPESSSLFPWG3SY4ZF3FDXXZQQA, seg_pos={0 0}, upload=58146]: Done uploading piece. err: <nil> Jul 27 00:13:31.994378346 (60h37m41.716941386s) [noise=true , address=77.90.31.16:4733 , piece_id=6GNJSP4EN72D24BIWOD7PGXHFKZK6ZXBIH4TRKUY5RFCZMDOGBJA, seg_pos={0 0}, upload=58146]: Done uploading piece. err: <nil> Jul 27 00:13:31.995792631 (60h37m41.718355661s) [noise=true , address=45.91.116.9:4262 , piece_id=OWJD7LGSVXFSUKHNUSDFRQ235P2XIBJOJG54FN7INRFSNG376C5Q, seg_pos={0 0}, upload=58146]: Done uploading piece. err: <nil> Jul 27 00:13:31.998736975 (60h37m41.721300015s) [noise=true , address=45.91.116.10:4082 , piece_id=ZSIXJUW6D7KXY2JX33TK2F3JQVIRQXPQ754FAJDZVEN6JJ46XZSA, seg_pos={0 0}, upload=58146]: Done uploading piece. err: <nil> Jul 27 00:13:32.045466081 (60h37m41.768029121s) [noise=true , address=45.91.116.7:4011 , piece_id=2FPMK6CEORJE2I6NVGWUOZZJGGJAGO72TQZA5HIUWJ2AHKLVZLFA, seg_pos={0 0}, upload=58146]: Done uploading piece. err: <nil> Jul 27 00:13:32.052034557 (60h37m41.774597598s) [noise=true , address=45.91.116.11:4119 , piece_id=V7U4FATH7XZ23LS5GT55ICBMHPE66IOLQJXIQLG73P5PK2SEJTHQ, seg_pos={0 0}, upload=58146]: Done uploading piece. err: <nil> Jul 27 00:13:32.067646859 (60h37m41.790209899s) [noise=true , address=77.90.31.14:4661 , piece_id=56HNJRJ6YUT3WYHB6GPEOIFPIOF4XUQTXBLOC4DYLYFRZHEUA2PA, seg_pos={0 0}, upload=58146]: Done uploading piece. err: <nil> Jul 27 00:13:32.075787915 (60h37m41.798350955s) [noise=true , address=45.91.116.11:4117 , piece_id=KQL7E6TQ3MYQYIL3B36ZOF67SDKBVR6ABWKSA3JKMLWKYG46GAUQ, seg_pos={0 0}, upload=58146]: Done uploading piece. err: <nil> Jul 27 00:13:33.403628758 (60h37m43.126191798s) [noise=true , address=77.90.31.12:4592 , piece_id=CNXHVVT66TTFMDZDTTBP5UYIMVJKQUAOACPVALNWC5D6ZZBQAHGQ, seg_pos={0 0}, upload=58146]: Done uploading piece. err: <nil> Jul 27 00:13:33.460620549 (60h37m43.183183579s) [noise=true , address=77.90.31.13:4626 , piece_id=T34BL7CW2T6NRA2IC74TLNAUBXCRO6LFDCVRQWYCHSU3YV2ECAUQ, seg_pos={0 0}, upload=58146]: Done uploading piece. err: <nil> Jul 27 00:13:58.786001067 (60h38m8.508564107s) [noise=true , address=104.36.82.13:4186 , piece_id=HAS3YDK5DG5URSE3Z2QRNX6HO3H3RYW7CBHBAG7J2UMTWYFIK4DA, seg_pos={0 0}, upload=58146]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:13:58.786085258 (60h38m8.508648288s) [noise=true , address=104.36.82.4:4802 , piece_id=2ZTVFJQ2PR4NQMD4BYR6ZCXKPTBY5OGLSF2UHR4YILEVBWMLVDDQ, seg_pos={0 0}, upload=58146]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:13:58.786147937 (60h38m8.508710978s) [noise=true , address=104.36.82.2:4732 , piece_id=5ARCTJ6Y77YFVLIO6CIZKKJAW42YMYJPM6C42U4A7WVYRCYMAOCA, seg_pos={0 0}, upload=58146]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: rpc: dial tcp 104.36.82.2:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:13:58.786210106 (60h38m8.508773146s) [noise=true , address=77.90.31.18:4807 , piece_id=P2Y4AZHOAMKSE7XAXTZKBBMZRIQL2X275MA2WQ3QL747GN7I3BAQ, seg_pos={0 0}, upload=58146]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:13:58.786391444 (60h38m8.508954484s) [noise=true , address=104.36.82.14:4221 , piece_id=25ZT65VYYWQNVEOD4TIXA3P6QH6UPTN3PCQSOSXDZ3COYIGK34UA, seg_pos={0 0}, upload=58146]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: write tcp 188.166.225.170:48452->104.36.82.14:4221: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:13:58.786437472 (60h38m8.509000502s) [seg_pos={0 0}, upload=58146]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 27 00:13:59.022092002 (60h38m8.744655042s) [upload=58146]: Done uploading segments... Jul 27 00:13:59.022115117 (60h38m8.744678157s) [upload=58146]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.237579049s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 51 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..XX.X.XXX.XXXXXX...XX.] duration: 5.931110717s FAIL (2025-07-27 00:13:42 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.931135334s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 17 out of 60 is failed (threshold=3) [X.....XXX..X.....X......X...XX..XX.X.XX...X...............XX] duration: 1m0.001316119s FAIL (2025-07-27 00:13:33 UTC) orbiter warning: context deadline exceeded Duration: 1m0.001327992s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 52 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..XX.X.XXX.XXXXXX...XX] duration: 30.235165588s FAIL (2025-07-27 00:13:32 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.235191346s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.233159098s FAIL (2025-07-27 00:13:30 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== pb.BatchRequestItem_ObjectBegin Jul 27 00:12:31.026903870 (60h36m40.74946691s) [upload=58129]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:12:31.478225380 (60h36m41.20078842s) [seg_pos={0 0}, upload=58129]: Begin upload segment... Jul 27 00:12:31.478278161 (60h36m41.200841201s) [seg_pos={0 0}, upload=58129]: Done begin upload segment. Jul 27 00:12:31.478301195 (60h36m41.200864235s) [noise=true , address=79.127.201.215:4350 , piece_id=AP7PIMDUSN6LTBE34B3TKHGWYSNRWCQAKPWLH44N2NHKAKDXOK3A, seg_pos={0 0}, upload=58129]: Uploading piece... Jul 27 00:12:31.478354848 (60h36m41.200917888s) [noise=true , address=79.127.220.101:4041 , piece_id=6DVKN65Q7VVWS7GHJ3ZGYD4XQTYT5W6YF47WPXSMJOYSHVU4T35A, seg_pos={0 0}, upload=58129]: Uploading piece... Jul 27 00:12:31.478654591 (60h36m41.201217632s) [noise=true , address=104.36.82.47:4534 , piece_id=43HSPMEYXZYDMCRQTWKOJX6SSIOUCWSWCC3PIVBCIEJPUO3GA3XA, seg_pos={0 0}, upload=58129]: Uploading piece... Jul 27 00:12:31.478873220 (60h36m41.20143626s) [noise=true , address=104.36.82.46:4484 , piece_id=CRFU7MWRA7YIOMOTX32Q26YROGJN5MRF4IVMX4VQYX25Z2S36UEQ, seg_pos={0 0}, upload=58129]: Uploading piece... Jul 27 00:12:31.479186629 (60h36m41.201749669s) [noise=true , address=77.90.31.10:4531 , piece_id=BJCL4MHMZPPF24ESZRGTR2PFMFMAU3F3W7PJTIP2TSGSZJZASBFA, seg_pos={0 0}, upload=58129]: Uploading piece... Jul 27 00:12:31.479239400 (60h36m41.20180244s) [noise=true , address=79.127.205.234:4614 , piece_id=QY5DPWNHPRH6RK3SW3IQQ2BNK2PEOM5D6U3FJAU5E5A3VVUE4B7A, seg_pos={0 0}, upload=58129]: Uploading piece... Jul 27 00:12:31.479496552 (60h36m41.202059592s) [noise=true , address=79.127.205.235:4682 , piece_id=T2NH7OPDTYKY5G54TRIG5CP4AXX3PTWWM6OAMTBN6QYC3T2BSU5Q, seg_pos={0 0}, upload=58129]: Uploading piece... Jul 27 00:12:31.479544854 (60h36m41.202107894s) [noise=true , address=79.127.219.47:4048 , piece_id=6MV33SFVBT5QMIRGEL7JVQ3YL4F2URONBGJVJTMPMASGHMMIJMUA, seg_pos={0 0}, upload=58129]: Uploading piece... Jul 27 00:12:31.479733475 (60h36m41.202296516s) [noise=true , address=79.127.220.98:4422 , piece_id=DICLWS5NHPUEPN2ZVYVMTN4S654A43QDLTJ2IQWAIY5HJZ4EIX3A, seg_pos={0 0}, upload=58129]: Uploading piece... Jul 27 00:12:31.479872942 (60h36m41.202435982s) [noise=true , address=104.36.82.55:4211 , piece_id=F3OBX672E4BWQ2YU7NOR4U3QY4EYW5OOYQLUMLT4UPDDNRJ4W7IA, seg_pos={0 0}, upload=58129]: Uploading piece... Jul 27 00:12:31.479971742 (60h36m41.202534782s) [noise=true , address=79.127.219.38:4004 , piece_id=CA4FYHIK3F5YQ5V5KX7IV3QBAYVBHGNZAGD5TDKM5V54IQGWNH7A, seg_pos={0 0}, upload=58129]: Uploading piece... Jul 27 00:12:31.480208815 (60h36m41.202771855s) [noise=true , address=79.127.205.233:4624 , piece_id=JSUZMMEKY2RUVRJ2XCOICYE3OK33IQP33NBOCG5Z7ULBYFLF7TKQ, seg_pos={0 0}, upload=58129]: Uploading piece... Jul 27 00:12:31.480278288 (60h36m41.202841328s) [noise=true , address=64.49.13.138:4483 , piece_id=QFIH742V62JXFP5KLTLNA32PAFW5QVJF5TGZNVC7OBYJ7HLLXITA, seg_pos={0 0}, upload=58129]: Uploading piece... Jul 27 00:12:31.480506044 (60h36m41.203069084s) [noise=true , address=104.36.82.53:4149 , piece_id=Q4S7UJRDCIS3473MSSXFPAR6WJMF76OWDD7SMA5MGUKJTOEXM36Q, seg_pos={0 0}, upload=58129]: Uploading piece... Jul 27 00:12:31.480705956 (60h36m41.203268986s) [noise=true , address=45.91.116.9:4277 , piece_id=YLHBIM36PKVKK2FKBYNE4QTXHIGQRXHNFSDUZSPP74AJUWZEUAJA, seg_pos={0 0}, upload=58129]: Uploading piece... Jul 27 00:12:31.480910428 (60h36m41.203473458s) [noise=true , address=104.36.82.52:4109 , piece_id=62HBDBLDIFA4NIIJK6FKSYTHLVYHHXE7RTW4VZFQM73NE53BU7DA, seg_pos={0 0}, upload=58129]: Uploading piece... Jul 27 00:12:31.481107134 (60h36m41.203670174s) [noise=true , address=45.91.116.14:4206 , piece_id=2MFVCUVABQSY7B6SQK3NKFR26UGQZKZYSTTGTUZVX4HNO35EDTDA, seg_pos={0 0}, upload=58129]: Uploading piece... Jul 27 00:12:31.481308018 (60h36m41.203871068s) [noise=true , address=79.127.220.101:4048 , piece_id=EVV7G4MS43G6RTQZMTB4TZBWPN52HJB5CFZHZXECYNAMA3EGRG6A, seg_pos={0 0}, upload=58129]: Uploading piece... Jul 27 00:12:32.575385570 (60h36m42.29794861s) [upload=58129]: Next returned nil segment Jul 27 00:12:32.575402632 (60h36m42.297965672s) [upload=58129]: Waiting for error group managing segments... Jul 27 00:12:32.877416878 (60h36m42.599979918s) [noise=true , address=79.127.205.233:4624 , piece_id=JSUZMMEKY2RUVRJ2XCOICYE3OK33IQP33NBOCG5Z7ULBYFLF7TKQ, seg_pos={0 0}, upload=58129]: Done uploading piece. err: <nil> Jul 27 00:12:32.882944154 (60h36m42.605507194s) [noise=true , address=79.127.205.234:4614 , piece_id=QY5DPWNHPRH6RK3SW3IQQ2BNK2PEOM5D6U3FJAU5E5A3VVUE4B7A, seg_pos={0 0}, upload=58129]: Done uploading piece. err: <nil> Jul 27 00:12:32.883372023 (60h36m42.605935063s) [noise=true , address=79.127.205.235:4682 , piece_id=T2NH7OPDTYKY5G54TRIG5CP4AXX3PTWWM6OAMTBN6QYC3T2BSU5Q, seg_pos={0 0}, upload=58129]: Done uploading piece. err: <nil> Jul 27 00:12:33.354169718 (60h36m43.076732758s) [noise=true , address=79.127.219.38:4004 , piece_id=CA4FYHIK3F5YQ5V5KX7IV3QBAYVBHGNZAGD5TDKM5V54IQGWNH7A, seg_pos={0 0}, upload=58129]: Done uploading piece. err: <nil> Jul 27 00:12:33.390453408 (60h36m43.113016438s) [noise=true , address=79.127.201.215:4350 , piece_id=AP7PIMDUSN6LTBE34B3TKHGWYSNRWCQAKPWLH44N2NHKAKDXOK3A, seg_pos={0 0}, upload=58129]: Done uploading piece. err: <nil> Jul 27 00:12:35.208209173 (60h36m44.930772214s) [noise=true , address=77.90.31.10:4531 , piece_id=BJCL4MHMZPPF24ESZRGTR2PFMFMAU3F3W7PJTIP2TSGSZJZASBFA, seg_pos={0 0}, upload=58129]: Done uploading piece. err: <nil> Jul 27 00:12:35.560171254 (60h36m45.282734444s) [noise=true , address=79.127.219.47:4048 , piece_id=6MV33SFVBT5QMIRGEL7JVQ3YL4F2URONBGJVJTMPMASGHMMIJMUA, seg_pos={0 0}, upload=58129]: Done uploading piece. err: <nil> Jul 27 00:12:35.817925048 (60h36m45.540488088s) [noise=true , address=79.127.220.101:4048 , piece_id=EVV7G4MS43G6RTQZMTB4TZBWPN52HJB5CFZHZXECYNAMA3EGRG6A, seg_pos={0 0}, upload=58129]: Done uploading piece. err: <nil> Jul 27 00:12:36.023331778 (60h36m45.745894808s) [noise=true , address=64.49.13.138:4483 , piece_id=QFIH742V62JXFP5KLTLNA32PAFW5QVJF5TGZNVC7OBYJ7HLLXITA, seg_pos={0 0}, upload=58129]: Done uploading piece. err: <nil> Jul 27 00:12:36.431969173 (60h36m46.154532213s) [noise=true , address=45.91.116.9:4277 , piece_id=YLHBIM36PKVKK2FKBYNE4QTXHIGQRXHNFSDUZSPP74AJUWZEUAJA, seg_pos={0 0}, upload=58129]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: storage node overloaded, request limit: 2000 Jul 27 00:12:37.019430645 (60h36m46.741993675s) [noise=true , address=79.127.220.98:4422 , piece_id=DICLWS5NHPUEPN2ZVYVMTN4S654A43QDLTJ2IQWAIY5HJZ4EIX3A, seg_pos={0 0}, upload=58129]: Done uploading piece. err: <nil> Jul 27 00:12:37.140244517 (60h36m46.862807547s) [noise=true , address=79.127.220.101:4041 , piece_id=6DVKN65Q7VVWS7GHJ3ZGYD4XQTYT5W6YF47WPXSMJOYSHVU4T35A, seg_pos={0 0}, upload=58129]: Done uploading piece. err: <nil> Jul 27 00:12:37.869550775 (60h36m47.592113815s) [noise=true , address=45.91.116.14:4206 , piece_id=2MFVCUVABQSY7B6SQK3NKFR26UGQZKZYSTTGTUZVX4HNO35EDTDA, seg_pos={0 0}, upload=58129]: Done uploading piece. err: <nil> Jul 27 00:13:30.379652793 (60h37m40.102215843s) [noise=true , address=104.36.82.46:4484 , piece_id=CRFU7MWRA7YIOMOTX32Q26YROGJN5MRF4IVMX4VQYX25Z2S36UEQ, seg_pos={0 0}, upload=58129]: Done uploading piece. err: ecclient: upload failed (node:1a5S5Ar7jNbK9MESghbWfmEgBPNSMN2tEnZeeG1kKzeX2dnSX3, address:104.36.82.46:4484): protocol: write tcp 188.166.225.170:56326->104.36.82.46:4484: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:13:30.379718449 (60h37m40.102281489s) [noise=true , address=104.36.82.55:4211 , piece_id=F3OBX672E4BWQ2YU7NOR4U3QY4EYW5OOYQLUMLT4UPDDNRJ4W7IA, seg_pos={0 0}, upload=58129]: Done uploading piece. err: ecclient: upload failed (node:1269tDiDZTJSwmka4HLKMEaHErdCaUqXEYTFimGWhtjai9o4dAB, address:104.36.82.55:4211): protocol: rpc: dial tcp 104.36.82.55:4211: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:13:30.379795475 (60h37m40.102358516s) [noise=true , address=104.36.82.47:4534 , piece_id=43HSPMEYXZYDMCRQTWKOJX6SSIOUCWSWCC3PIVBCIEJPUO3GA3XA, seg_pos={0 0}, upload=58129]: Done uploading piece. err: ecclient: upload failed (node:1VDrNGcCnaKEmG5NN4nex1JoZAgdCEfX5CdsKbxytfLfKzPbr, address:104.36.82.47:4534): protocol: write tcp 188.166.225.170:52358->104.36.82.47:4534: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:13:30.379802490 (60h37m40.10236554s) [noise=true , address=104.36.82.53:4149 , piece_id=Q4S7UJRDCIS3473MSSXFPAR6WJMF76OWDD7SMA5MGUKJTOEXM36Q, seg_pos={0 0}, upload=58129]: Done uploading piece. err: ecclient: upload failed (node:1nDCYjyLUqt1zkfotDNVcor73Cdpq3fnyMiygujdc9V4tGNHLY, address:104.36.82.53:4149): protocol: write tcp 188.166.225.170:37754->104.36.82.53:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:13:30.379870640 (60h37m40.10243369s) [noise=true , address=104.36.82.52:4109 , piece_id=62HBDBLDIFA4NIIJK6FKSYTHLVYHHXE7RTW4VZFQM73NE53BU7DA, seg_pos={0 0}, upload=58129]: Done uploading piece. err: ecclient: upload failed (node:1hBqAk4HrRmUKRw3RS7xUHZ4rdUMiF25jKWF7jqNrffZG8pEcj, address:104.36.82.52:4109): protocol: write tcp 188.166.225.170:41724->104.36.82.52:4109: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:13:30.379910366 (60h37m40.102473406s) [seg_pos={0 0}, upload=58129]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 27 00:13:30.611392384 (60h37m40.333955424s) [upload=58129]: Done uploading segments... Jul 27 00:13:30.611412573 (60h37m40.333975613s) [upload=58129]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1m0.233169979s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 1.991619004s FAIL (2025-07-27 00:13:25 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.991647769s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [XX.X.....X.X..X..X.X....X.X....XXX.X...X......X.XX...XXXX...] duration: 6.20457161s FAIL (2025-07-27 00:13:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.204584865s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 53 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXX.XXXXXXXXXX.XXXX..X..X] duration: 30.232647615s FAIL (2025-07-27 00:12:44 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== timal: 13 errs: <nil> Jul 27 00:11:50.257632003 (60h35m59.980195043s) [upload=58119]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:11:50.257636792 (60h35m59.980199832s) [upload=58119]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:11:50.547920805 (60h36m0.270483835s) [upload=58119]: Done uploading segments... Jul 27 00:11:50.547935042 (60h36m0.270498072s) [upload=58119]: Upload finished. err: <nil> Jul 27 00:12:15.130189924 (60h36m24.852752964s) [upload=58125]: Starting upload Jul 27 00:12:15.130249278 (60h36m24.852812318s) [upload=58125]: Done starting upload Jul 27 00:12:15.130350221 (60h36m24.852913261s) [upload=58125]: Uploading segments... Jul 27 00:12:15.130485049 (60h36m24.853048089s) [upload=58125]: Got next segment. Inline: false Jul 27 00:12:15.130517591 (60h36m24.853080631s) [upload=58125]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:12:15.130522711 (60h36m24.853085751s) [upload=58125]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:12:15.364289469 (60h36m25.086852509s) [seg_pos={0 0}, upload=58125]: Begin upload segment... Jul 27 00:12:15.364335027 (60h36m25.086898067s) [seg_pos={0 0}, upload=58125]: Done begin upload segment. Jul 27 00:12:15.364358562 (60h36m25.086921602s) [noise=true , address=77.90.31.18:4805 , piece_id=J3J3BGYQ3P56A5XHXTKV76LNTDYH7TVICAWH347OTNCBJGQWM32A, seg_pos={0 0}, upload=58125]: Uploading piece... Jul 27 00:12:15.364383348 (60h36m25.086946378s) [noise=true , address=77.90.31.11:4555 , piece_id=WGHTIXXMTXXHPWHOMD6LBBLM6FH7ZVKD6EVFEVKW4SR36VTUB4UA, seg_pos={0 0}, upload=58125]: Uploading piece... Jul 27 00:12:15.364686328 (60h36m25.087249358s) [noise=true , address=45.91.116.6:4975 , piece_id=NONSFAX777AESHHYK5GMZQSWACODPXTFDE5NYVWINAFKP2GXBNMA, seg_pos={0 0}, upload=58125]: Uploading piece... Jul 27 00:12:15.364796078 (60h36m25.087359118s) [noise=true , address=104.36.82.5:4840 , piece_id=6NZXAQI2PHYYDGMB2SNK376D2ZPAHGAMDUC2L22IV5OEHILT4M2A, seg_pos={0 0}, upload=58125]: Uploading piece... Jul 27 00:12:15.364930625 (60h36m25.087493656s) [noise=true , address=45.91.116.9:4263 , piece_id=4EXFMBDDM6Q3NR2S3OZZ2XZVXA23MNFCWYYR466GLWRD5KJGAWCA, seg_pos={0 0}, upload=58125]: Uploading piece... Jul 27 00:12:15.365169252 (60h36m25.087732292s) [noise=true , address=77.90.31.10:4520 , piece_id=TFGJVDUPYPHIAUPQ3ANCDWFNRJPGFNJ74O7YCA4WCLRZYJUIBAYQ, seg_pos={0 0}, upload=58125]: Uploading piece... Jul 27 00:12:15.365396016 (60h36m25.087959056s) [noise=true , address=45.91.116.10:4081 , piece_id=BGYHTRQB7OVMGD3WKD7SFT6E75A47JKKDEWOT7GI2VPFOCZ6JQLQ, seg_pos={0 0}, upload=58125]: Uploading piece... Jul 27 00:12:15.365604796 (60h36m25.088167836s) [noise=true , address=104.36.82.5:4838 , piece_id=WRYCJUJ3X7ZSGW5RJ6EZBLCCV5XOEWO6XXMDPRCFBUNUABYHSAAA, seg_pos={0 0}, upload=58125]: Uploading piece... Jul 27 00:12:15.365864532 (60h36m25.088427572s) [noise=true , address=77.90.31.15:4698 , piece_id=LO6AHA6D2JVB6RVNSPOO3DOIGGSK6JC3WGC2USRTP7J5WM5WGXLQ, seg_pos={0 0}, upload=58125]: Uploading piece... Jul 27 00:12:15.366087559 (60h36m25.088650619s) [noise=true , address=104.36.82.8:4972 , piece_id=K44LZEJKLHTSLVRZJRCOYGPDAZHH4MQBY7IVZZVY2DY4GVVBXAGA, seg_pos={0 0}, upload=58125]: Uploading piece... Jul 27 00:12:15.366300998 (60h36m25.088864038s) [noise=true , address=45.91.116.6:4974 , piece_id=4TIJOOEAJ3Q7474QI75O56GOCW3XTTI75BS3N72V2ILWPIJTUUHQ, seg_pos={0 0}, upload=58125]: Uploading piece... Jul 27 00:12:15.366504057 (60h36m25.089067097s) [noise=true , address=45.91.116.4:4940 , piece_id=SZCTGPH42EQB3ZOK4D5R667Q64S3WIWXFUCSWLZMEEHRCYFQJH6A, seg_pos={0 0}, upload=58125]: Uploading piece... Jul 27 00:12:15.366686165 (60h36m25.089249205s) [noise=true , address=45.91.116.14:4191 , piece_id=LMQLC4GCAVBCBVHCHJYDIWF7CBSEXOVN5WF3HJAJP7QPON6QMNKQ, seg_pos={0 0}, upload=58125]: Uploading piece... Jul 27 00:12:15.366874625 (60h36m25.089437665s) [noise=true , address=104.36.82.14:4222 , piece_id=EGTCKWAJISTAI3KJLN3G3DZRCZIGKKAFDENB3KZPRECDXW2EDTLA, seg_pos={0 0}, upload=58125]: Uploading piece... Jul 27 00:12:15.367061613 (60h36m25.089624653s) [noise=true , address=45.91.116.10:4083 , piece_id=BC5POQB34FGQ2S2YDGAXBPYOXWXH6JLQYYGQ5JOFIKC4ICUIWR2Q, seg_pos={0 0}, upload=58125]: Uploading piece... Jul 27 00:12:15.367246868 (60h36m25.089809908s) [noise=true , address=104.36.82.12:4149 , piece_id=YAKK7MXZG3YCVGOAX5L7VWVFHGGSHX6YABNRFLNAMX62QTW426XQ, seg_pos={0 0}, upload=58125]: Uploading piece... Jul 27 00:12:17.118387739 (60h36m26.840950779s) [upload=58125]: Next returned nil segment Jul 27 00:12:17.118424640 (60h36m26.84098768s) [upload=58125]: Waiting for error group managing segments... Jul 27 00:12:17.395594322 (60h36m27.118157362s) [noise=true , address=77.90.31.15:4698 , piece_id=LO6AHA6D2JVB6RVNSPOO3DOIGGSK6JC3WGC2USRTP7J5WM5WGXLQ, seg_pos={0 0}, upload=58125]: Done uploading piece. err: <nil> Jul 27 00:12:17.403140179 (60h36m27.125703219s) [noise=true , address=45.91.116.10:4081 , piece_id=BGYHTRQB7OVMGD3WKD7SFT6E75A47JKKDEWOT7GI2VPFOCZ6JQLQ, seg_pos={0 0}, upload=58125]: Done uploading piece. err: <nil> Jul 27 00:12:17.456923140 (60h36m27.17948618s) [noise=true , address=45.91.116.10:4083 , piece_id=BC5POQB34FGQ2S2YDGAXBPYOXWXH6JLQYYGQ5JOFIKC4ICUIWR2Q, seg_pos={0 0}, upload=58125]: Done uploading piece. err: <nil> Jul 27 00:12:17.470870104 (60h36m27.193433134s) [noise=true , address=45.91.116.6:4974 , piece_id=4TIJOOEAJ3Q7474QI75O56GOCW3XTTI75BS3N72V2ILWPIJTUUHQ, seg_pos={0 0}, upload=58125]: Done uploading piece. err: <nil> Jul 27 00:12:17.485776194 (60h36m27.208339224s) [noise=true , address=45.91.116.6:4975 , piece_id=NONSFAX777AESHHYK5GMZQSWACODPXTFDE5NYVWINAFKP2GXBNMA, seg_pos={0 0}, upload=58125]: Done uploading piece. err: <nil> Jul 27 00:12:17.529157194 (60h36m27.251720234s) [noise=true , address=45.91.116.4:4940 , piece_id=SZCTGPH42EQB3ZOK4D5R667Q64S3WIWXFUCSWLZMEEHRCYFQJH6A, seg_pos={0 0}, upload=58125]: Done uploading piece. err: <nil> Jul 27 00:12:17.566592398 (60h36m27.289155438s) [noise=true , address=77.90.31.11:4555 , piece_id=WGHTIXXMTXXHPWHOMD6LBBLM6FH7ZVKD6EVFEVKW4SR36VTUB4UA, seg_pos={0 0}, upload=58125]: Done uploading piece. err: <nil> Jul 27 00:12:17.581888173 (60h36m27.304451213s) [noise=true , address=45.91.116.9:4263 , piece_id=4EXFMBDDM6Q3NR2S3OZZ2XZVXA23MNFCWYYR466GLWRD5KJGAWCA, seg_pos={0 0}, upload=58125]: Done uploading piece. err: <nil> Jul 27 00:12:17.596212489 (60h36m27.318775529s) [noise=true , address=45.91.116.14:4191 , piece_id=LMQLC4GCAVBCBVHCHJYDIWF7CBSEXOVN5WF3HJAJP7QPON6QMNKQ, seg_pos={0 0}, upload=58125]: Done uploading piece. err: <nil> Jul 27 00:12:18.935772373 (60h36m28.658335413s) [noise=true , address=77.90.31.10:4520 , piece_id=TFGJVDUPYPHIAUPQ3ANCDWFNRJPGFNJ74O7YCA4WCLRZYJUIBAYQ, seg_pos={0 0}, upload=58125]: Done uploading piece. err: <nil> Jul 27 00:12:44.482920968 (60h36m54.205484008s) [noise=true , address=104.36.82.5:4840 , piece_id=6NZXAQI2PHYYDGMB2SNK376D2ZPAHGAMDUC2L22IV5OEHILT4M2A, seg_pos={0 0}, upload=58125]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:104.36.82.5:4840): protocol: rpc: dial tcp 104.36.82.5:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:12:44.482972567 (60h36m54.205535597s) [noise=true , address=104.36.82.5:4838 , piece_id=WRYCJUJ3X7ZSGW5RJ6EZBLCCV5XOEWO6XXMDPRCFBUNUABYHSAAA, seg_pos={0 0}, upload=58125]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: write tcp 188.166.225.170:44666->104.36.82.5:4838: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:12:44.483121992 (60h36m54.205685022s) [noise=true , address=77.90.31.18:4805 , piece_id=J3J3BGYQ3P56A5XHXTKV76LNTDYH7TVICAWH347OTNCBJGQWM32A, seg_pos={0 0}, upload=58125]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: rpc: dial tcp 77.90.31.18:4805: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:12:44.483161168 (60h36m54.205724208s) [noise=true , address=104.36.82.12:4149 , piece_id=YAKK7MXZG3YCVGOAX5L7VWVFHGGSHX6YABNRFLNAMX62QTW426XQ, seg_pos={0 0}, upload=58125]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 188.166.225.170:49192->104.36.82.12:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:12:44.483192738 (60h36m54.205755808s) [noise=true , address=104.36.82.8:4972 , piece_id=K44LZEJKLHTSLVRZJRCOYGPDAZHH4MQBY7IVZZVY2DY4GVVBXAGA, seg_pos={0 0}, upload=58125]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: rpc: dial tcp 104.36.82.8:4972: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:12:44.483211874 (60h36m54.205774904s) [noise=true , address=104.36.82.14:4222 , piece_id=EGTCKWAJISTAI3KJLN3G3DZRCZIGKKAFDENB3KZPRECDXW2EDTLA, seg_pos={0 0}, upload=58125]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: rpc: dial tcp 104.36.82.14:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:12:44.483229939 (60h36m54.205792979s) [seg_pos={0 0}, upload=58125]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 27 00:12:44.713369970 (60h36m54.43593301s) [upload=58125]: Done uploading segments... Jul 27 00:12:44.713400107 (60h36m54.435963178s) [upload=58125]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.23267066s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 52 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..XX.X.XXX.XXXXXX...X] duration: 30.235676933s FAIL (2025-07-27 00:12:41 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 6) Duration: 30.235719543s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 1.874476376s FAIL (2025-07-27 00:12:40 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.874508218s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 17 out of 60 is failed (threshold=3) [XX.....XXX..X.....X......X...XX..XX.X.XX...X...............X] duration: 1m0.000491142s FAIL (2025-07-27 00:12:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000520688s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.23181715s FAIL (2025-07-27 00:12:30 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== piece... Jul 27 00:11:31.017033996 (60h35m40.739597036s) [noise=true , address=79.127.205.245:4240 , piece_id=WJ255QQBQWPBQS5DMJ72MJ6IM2JVTVFD6WMKHYT7HR46NZP6EO3Q, seg_pos={0 0}, upload=58114]: Uploading piece... Jul 27 00:11:31.017246372 (60h35m40.739809412s) [noise=true , address=77.90.31.18:4817 , piece_id=4ZXCWQQ2WLGENBRCFI3B7IIT5AWH6X6IIJ3KVAHKLOWO2CAJRDZQ, seg_pos={0 0}, upload=58114]: Uploading piece... Jul 27 00:11:31.017307839 (60h35m40.739870879s) [noise=true , address=79.127.205.234:4604 , piece_id=T2HKHTZDWBVRGMMU5WMSNXURMHQWKSOXWJW5BH75RULVWTIIDCSA, seg_pos={0 0}, upload=58114]: Uploading piece... Jul 27 00:11:31.017570702 (60h35m40.740133733s) [noise=true , address=104.36.82.15:4284 , piece_id=CKPI3HIJO6IOLQAQXM3MJSL3KIOI5Z5WOZMWDEDGIKA2GJUQCDNQ, seg_pos={0 0}, upload=58114]: Uploading piece... Jul 27 00:11:31.017637410 (60h35m40.74020045s) [noise=true , address=64.49.13.26:4953 , piece_id=OF23F7MJ4V3RNCCWIEBC2QPNYFD4VAYX6MDMRI4MBD3QENNPB5OA, seg_pos={0 0}, upload=58114]: Uploading piece... Jul 27 00:11:31.017870216 (60h35m40.740433256s) [noise=true , address=79.127.220.101:4041 , piece_id=AXPE5CYLHBMO5YVDR2R6EZXE4FO73EOUDYTAAH7QRR5L5XCD6QEQ, seg_pos={0 0}, upload=58114]: Uploading piece... Jul 27 00:11:31.017953896 (60h35m40.740516936s) [noise=true , address=77.90.31.13:4658 , piece_id=TNKDY7OTTOSQVHSSV7MIKRWK4WSL3OWVTIL7UICXX4OPMDHJA6IA, seg_pos={0 0}, upload=58114]: Uploading piece... Jul 27 00:11:31.018152706 (60h35m40.740715746s) [noise=true , address=64.49.13.140:4508 , piece_id=253UHTR2WJEXZLBBUVZSZPXTNB5BCJCJESX5GNJM3U5LAJYJZ7YA, seg_pos={0 0}, upload=58114]: Uploading piece... Jul 27 00:11:31.018210096 (60h35m40.740773136s) [noise=true , address=79.127.205.235:4685 , piece_id=SQF2VPXAXGKKCR5A672XHF7LG3TA6T6OZ7BGJG563OZYF5A5AXZQ, seg_pos={0 0}, upload=58114]: Uploading piece... Jul 27 00:11:31.018371675 (60h35m40.740934705s) [noise=true , address=104.36.82.46:4512 , piece_id=MYVSMBZ3MKSFUZ3R6PYLTPMGOMBFQ5J5L2MCTDUTMSHER4VX4GTQ, seg_pos={0 0}, upload=58114]: Uploading piece... Jul 27 00:11:31.018440778 (60h35m40.741003818s) [noise=true , address=77.90.31.16:4748 , piece_id=QI4KIP6JGETWMIFSIOWA366AYQ72MVTG44AP24CC5WPN76IVPCQQ, seg_pos={0 0}, upload=58114]: Uploading piece... Jul 27 00:11:31.018729870 (60h35m40.74129291s) [noise=true , address=104.36.82.51:4062 , piece_id=2V3OXLJEHQN424CALZ34TSBMGHYVCTS7ESDADE53YMZYLFBJP7IQ, seg_pos={0 0}, upload=58114]: Uploading piece... Jul 27 00:11:31.018981191 (60h35m40.741544221s) [noise=true , address=79.127.205.246:4274 , piece_id=BLXPNB57JPUTAXHLBOG3NCDP7VWSP25CV3LRGQJOG2LXCSUPBW6Q, seg_pos={0 0}, upload=58114]: Uploading piece... Jul 27 00:11:31.019211282 (60h35m40.741774322s) [noise=true , address=104.36.82.12:4168 , piece_id=CDFBMTTRNE7JCWHNTHBM2QS4S2R6NOLFL3HHWECJJJ43OJXAJZXQ, seg_pos={0 0}, upload=58114]: Uploading piece... Jul 27 00:11:31.019456681 (60h35m40.742019721s) [noise=true , address=104.36.82.20:4461 , piece_id=LEQQH3P2GD6K3ILMZRSZOCEIUFG6MU5R25X4MHZORL3URWIMSWMQ, seg_pos={0 0}, upload=58114]: Uploading piece... Jul 27 00:11:31.019673756 (60h35m40.742236796s) [noise=true , address=79.127.219.51:4210 , piece_id=5KLO3A4G555YCTZ7QCZVX3JWCMP4EESBUVUDU24LZFCKWZTB2EYA, seg_pos={0 0}, upload=58114]: Uploading piece... Jul 27 00:11:31.019901743 (60h35m40.742464783s) [noise=true , address=104.36.82.47:4537 , piece_id=WJNHUEACCBGUZQ4MHHKCSK7AEDX34G5Z5K2FRZHVAFKWNJFG3JXA, seg_pos={0 0}, upload=58114]: Uploading piece... Jul 27 00:11:32.116889070 (60h35m41.83945211s) [upload=58114]: Next returned nil segment Jul 27 00:11:32.116907094 (60h35m41.839470135s) [upload=58114]: Waiting for error group managing segments... Jul 27 00:11:32.400438007 (60h35m42.123001047s) [noise=true , address=79.127.205.245:4240 , piece_id=WJ255QQBQWPBQS5DMJ72MJ6IM2JVTVFD6WMKHYT7HR46NZP6EO3Q, seg_pos={0 0}, upload=58114]: Done uploading piece. err: <nil> Jul 27 00:11:32.419754955 (60h35m42.142317995s) [noise=true , address=79.127.205.235:4685 , piece_id=SQF2VPXAXGKKCR5A672XHF7LG3TA6T6OZ7BGJG563OZYF5A5AXZQ, seg_pos={0 0}, upload=58114]: Done uploading piece. err: <nil> Jul 27 00:11:32.454796929 (60h35m42.17735997s) [noise=true , address=79.127.205.246:4274 , piece_id=BLXPNB57JPUTAXHLBOG3NCDP7VWSP25CV3LRGQJOG2LXCSUPBW6Q, seg_pos={0 0}, upload=58114]: Done uploading piece. err: <nil> Jul 27 00:11:32.508361473 (60h35m42.230924514s) [noise=true , address=79.127.205.234:4604 , piece_id=T2HKHTZDWBVRGMMU5WMSNXURMHQWKSOXWJW5BH75RULVWTIIDCSA, seg_pos={0 0}, upload=58114]: Done uploading piece. err: <nil> Jul 27 00:11:33.092872603 (60h35m42.815435643s) [noise=true , address=77.90.31.16:4748 , piece_id=QI4KIP6JGETWMIFSIOWA366AYQ72MVTG44AP24CC5WPN76IVPCQQ, seg_pos={0 0}, upload=58114]: Done uploading piece. err: <nil> Jul 27 00:11:34.345847555 (60h35m44.068410595s) [noise=true , address=79.127.219.51:4210 , piece_id=5KLO3A4G555YCTZ7QCZVX3JWCMP4EESBUVUDU24LZFCKWZTB2EYA, seg_pos={0 0}, upload=58114]: Done uploading piece. err: <nil> Jul 27 00:11:36.620976689 (60h35m46.343539729s) [noise=true , address=79.127.220.101:4041 , piece_id=AXPE5CYLHBMO5YVDR2R6EZXE4FO73EOUDYTAAH7QRR5L5XCD6QEQ, seg_pos={0 0}, upload=58114]: Done uploading piece. err: <nil> Jul 27 00:11:36.896156473 (60h35m46.618719513s) [noise=true , address=77.90.31.13:4658 , piece_id=TNKDY7OTTOSQVHSSV7MIKRWK4WSL3OWVTIL7UICXX4OPMDHJA6IA, seg_pos={0 0}, upload=58114]: Done uploading piece. err: <nil> Jul 27 00:11:40.115868474 (60h35m49.838431514s) [noise=true , address=64.49.13.26:4953 , piece_id=OF23F7MJ4V3RNCCWIEBC2QPNYFD4VAYX6MDMRI4MBD3QENNPB5OA, seg_pos={0 0}, upload=58114]: Done uploading piece. err: <nil> Jul 27 00:11:49.826606684 (60h35m59.549169724s) [noise=true , address=64.49.13.140:4508 , piece_id=253UHTR2WJEXZLBBUVZSZPXTNB5BCJCJESX5GNJM3U5LAJYJZ7YA, seg_pos={0 0}, upload=58114]: Done uploading piece. err: <nil> Jul 27 00:12:29.931183938 (60h36m39.653746978s) [noise=true , address=104.36.82.46:4512 , piece_id=MYVSMBZ3MKSFUZ3R6PYLTPMGOMBFQ5J5L2MCTDUTMSHER4VX4GTQ, seg_pos={0 0}, upload=58114]: Done uploading piece. err: ecclient: upload failed (node:1rUdrYUJgCF8aa3MdNe82dT765nqW46by7mJ2phMSpfHAhzgU2, address:104.36.82.46:4512): protocol: write tcp 188.166.225.170:40114->104.36.82.46:4512: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:12:29.931219377 (60h36m39.653782487s) [noise=true , address=104.36.82.51:4062 , piece_id=2V3OXLJEHQN424CALZ34TSBMGHYVCTS7ESDADE53YMZYLFBJP7IQ, seg_pos={0 0}, upload=58114]: Done uploading piece. err: ecclient: upload failed (node:12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, address:104.36.82.51:4062): protocol: write tcp 188.166.225.170:48958->104.36.82.51:4062: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:12:29.931255435 (60h36m39.653818476s) [noise=true , address=104.36.82.54:4173 , piece_id=DN3POBT5YQ5G27ZN2TTHITQD47ZH5M5CK5X5CRWIVP5Y7QKOS2HA, seg_pos={0 0}, upload=58114]: Done uploading piece. err: ecclient: upload failed (node:1w2v4iyYtJtcdz7shYhG8CjCDQ1dWjeJVwY5Ubi7Ex5Y38e2i1, address:104.36.82.54:4173): protocol: write tcp 188.166.225.170:55092->104.36.82.54:4173: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:12:29.931289260 (60h36m39.6538523s) [noise=true , address=77.90.31.18:4817 , piece_id=4ZXCWQQ2WLGENBRCFI3B7IIT5AWH6X6IIJ3KVAHKLOWO2CAJRDZQ, seg_pos={0 0}, upload=58114]: Done uploading piece. err: ecclient: upload failed (node:12Qv5LBuANP9vS2vbsVNrJ7Mkc3SqXwTNCBzLvN955RHFKQVv5x, address:77.90.31.18:4817): protocol: write tcp 188.166.225.170:33636->77.90.31.18:4817: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:12:29.931319177 (60h36m39.653882207s) [noise=true , address=104.36.82.15:4284 , piece_id=CKPI3HIJO6IOLQAQXM3MJSL3KIOI5Z5WOZMWDEDGIKA2GJUQCDNQ, seg_pos={0 0}, upload=58114]: Done uploading piece. err: ecclient: upload failed (node:1YAnLu8onh19Mx524VikQzCHEFW4U8dgtuLx8amtQAiReSCisF, address:104.36.82.15:4284): protocol: write tcp 188.166.225.170:38452->104.36.82.15:4284: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:12:29.931322904 (60h36m39.653886004s) [noise=true , address=104.36.82.12:4168 , piece_id=CDFBMTTRNE7JCWHNTHBM2QS4S2R6NOLFL3HHWECJJJ43OJXAJZXQ, seg_pos={0 0}, upload=58114]: Done uploading piece. err: ecclient: upload failed (node:12XJ5kTKY8pLq3uDJZfFVkTgS56Xx8pG2R5i2A9VRNDEckdKnTF, address:104.36.82.12:4168): protocol: rpc: dial tcp 104.36.82.12:4168: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:12:29.931339937 (60h36m39.653903017s) [noise=true , address=104.36.82.47:4537 , piece_id=WJNHUEACCBGUZQ4MHHKCSK7AEDX34G5Z5K2FRZHVAFKWNJFG3JXA, seg_pos={0 0}, upload=58114]: Done uploading piece. err: ecclient: upload failed (node:12LTTCDNikuwf94KeKsQGFSu6ABXVYt1ombRsEyrZN5JC8d4Eot, address:104.36.82.47:4537): protocol: rpc: dial tcp 104.36.82.47:4537: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:12:29.931349455 (60h36m39.653912485s) [noise=true , address=104.36.82.20:4461 , piece_id=LEQQH3P2GD6K3ILMZRSZOCEIUFG6MU5R25X4MHZORL3URWIMSWMQ, seg_pos={0 0}, upload=58114]: Done uploading piece. err: ecclient: upload failed (node:12nhwT2THUu6ptumqU9Bz1Nd34FHLewx9vWjGFC34j3Lz3RWfkw, address:104.36.82.20:4461): protocol: rpc: dial tcp 104.36.82.20:4461: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:12:29.931371597 (60h36m39.653934648s) [seg_pos={0 0}, upload=58114]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 27 00:12:30.160452502 (60h36m39.883015582s) [upload=58114]: Done uploading segments... Jul 27 00:12:30.160484543 (60h36m39.883047583s) [upload=58114]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.231844842s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [XXX.X.....X.X..X..X.X....X.X....XXX.X...X......X.XX...XXXX..] duration: 6.231432698s FAIL (2025-07-27 00:12:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.231454349s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [.XXX.X.....X.X..X..X.X....X.X....XXX.X...X......X.XX...XXXX.] duration: 14.163498248s FAIL (2025-07-27 00:11:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 14.163549446s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 53 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXX.XXXXXXXXXX.XXXX..X..] duration: 7.096051247s FAIL (2025-07-27 00:11:52 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== }, upload=58077]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV): protocol: rpc: dial tcp 104.36.82.3:4765: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:08:58.749291092 (60h33m8.471854132s) [seg_pos={0 0}, upload=58077]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:08:58.749388238 (60h33m8.471951278s) [upload=58077]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:08:58.749395502 (60h33m8.471958542s) [upload=58077]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:08:59.071572826 (60h33m8.794135866s) [upload=58077]: Done uploading segments... Jul 27 00:08:59.071590289 (60h33m8.794153319s) [upload=58077]: Upload finished. err: <nil> Jul 27 00:09:23.191431943 (60h33m32.913994983s) [upload=58085]: Starting upload Jul 27 00:09:23.191484463 (60h33m32.914047503s) [upload=58085]: Done starting upload Jul 27 00:09:23.191558114 (60h33m32.914121154s) [upload=58085]: Uploading segments... Jul 27 00:09:23.191660350 (60h33m32.91422345s) [upload=58085]: Got next segment. Inline: false Jul 27 00:09:23.191695326 (60h33m32.914258356s) [upload=58085]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:09:23.191699464 (60h33m32.914262504s) [upload=58085]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:09:23.424088299 (60h33m33.146651339s) [seg_pos={0 0}, upload=58085]: Begin upload segment... Jul 27 00:09:23.424132493 (60h33m33.146695533s) [seg_pos={0 0}, upload=58085]: Done begin upload segment. Jul 27 00:09:23.424154705 (60h33m33.146717746s) [noise=true , address=104.36.82.8:4969 , piece_id=RGXUMB5UELRCP3JJNMDPNAUSKXAIB3H5R22KISQULBYM5JFTSRGQ, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:23.424193240 (60h33m33.14675628s) [noise=true , address=104.36.82.2:4730 , piece_id=AF6N5Z3FL5YDYGDLZJFTVORZSKPGCT5335GJSWDDZNUFBNHNQKGQ, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:23.424508462 (60h33m33.147071503s) [noise=true , address=45.91.116.8:4048 , piece_id=ML22FPHXORK7G6QYGZXIRKMHYQVBXWQT4FR3UIPCABJV674DO7UQ, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:23.424731339 (60h33m33.147294389s) [noise=true , address=77.90.31.18:4807 , piece_id=6QQ33YWILLVQCU4GOAOI6FOOPSXARZFG2YYPGY7BHESDNEGT57YQ, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:23.424777367 (60h33m33.147340397s) [noise=true , address=45.91.116.8:4047 , piece_id=SUJWMYZOYMDNH6UOOJUYWNEWXIR7CDAKGSM6VX4FJCBMV7JNFEKQ, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:23.425039278 (60h33m33.147602318s) [noise=true , address=45.91.116.7:4009 , piece_id=6XQK3DTIL3YS23DJKQFRNIZUMUY3PHF7A4H4C5OEJ3NHGZ7SQBFA, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:23.425091207 (60h33m33.147654247s) [noise=true , address=45.91.116.12:4227 , piece_id=CIROC6VSFJQWW372P2NHHG2ZVJCOJAXOKMNAFXNY5PW324WDD5XA, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:23.425286531 (60h33m33.147849571s) [noise=true , address=45.91.116.11:4119 , piece_id=G2KMTBCV2665ZX2O7GPCKBRFSSIHFF6SDY6TZWCDNZREWPZNQ24A, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:23.425540938 (60h33m33.148103978s) [noise=true , address=77.90.31.15:4698 , piece_id=JVSARDGWDK5B5QOARWEGBRMSQYKYOEXATS2FNVNAJXRJXNN2U5DA, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:23.425552049 (60h33m33.148115089s) [noise=true , address=45.91.116.13:4154 , piece_id=LYFRMKYHVOCECGME4GJRAUGOWNBAOGGDVU7AS7GN6UBBHHZTTLHQ, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:23.425763755 (60h33m33.148326785s) [noise=true , address=77.90.31.11:4555 , piece_id=MJR6LHPXWOVMEFF6Q66MJ2FHCK6GA5M6F334RNC5J7JIZGJ7NJ4Q, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:23.425836293 (60h33m33.148399333s) [noise=true , address=104.36.82.16:4295 , piece_id=ZG5AHIG3CBQF2SKTYOWYG7Y53O3NBC36VU2I3CWIGIQZ5IUPT6UQ, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:23.426064771 (60h33m33.148627811s) [noise=true , address=45.91.116.13:4153 , piece_id=WF6LCBKFOXJF3DNCNZI4E4LXGXSYIWXWGDDEZHN4CBUFY6YPWJQA, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:23.426081131 (60h33m33.148644171s) [noise=true , address=77.90.31.13:4628 , piece_id=OQHKCDILQA7TEDQP27C3SM7NZC3NSQTNTREJIQWFFPYZN6CHJDBQ, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:23.426305661 (60h33m33.148868701s) [noise=true , address=77.90.31.12:4589 , piece_id=IHNWWRBL2E5VV77ANZHNWCCGC3L4JC5Y6F4DJWVGWQ7BESVK26MQ, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:23.426336901 (60h33m33.148899941s) [noise=true , address=45.91.116.5:4297 , piece_id=EQEYGBRJY6O2IYYC344ZHXPZQTNOSBFR7S33LE7HJPFD5GSMFRBQ, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:25.166110004 (60h33m34.888673044s) [upload=58085]: Next returned nil segment Jul 27 00:09:25.166138388 (60h33m34.888701428s) [upload=58085]: Waiting for error group managing segments... Jul 27 00:09:25.453532572 (60h33m35.176095613s) [noise=true , address=45.91.116.13:4153 , piece_id=WF6LCBKFOXJF3DNCNZI4E4LXGXSYIWXWGDDEZHN4CBUFY6YPWJQA, seg_pos={0 0}, upload=58085]: Done uploading piece. err: <nil> Jul 27 00:09:25.457229505 (60h33m35.179792545s) [noise=true , address=45.91.116.8:4047 , piece_id=SUJWMYZOYMDNH6UOOJUYWNEWXIR7CDAKGSM6VX4FJCBMV7JNFEKQ, seg_pos={0 0}, upload=58085]: Done uploading piece. err: <nil> Jul 27 00:09:25.526191879 (60h33m35.248754919s) [noise=true , address=45.91.116.12:4227 , piece_id=CIROC6VSFJQWW372P2NHHG2ZVJCOJAXOKMNAFXNY5PW324WDD5XA, seg_pos={0 0}, upload=58085]: Done uploading piece. err: <nil> Jul 27 00:09:25.527573101 (60h33m35.250136142s) [noise=true , address=45.91.116.13:4154 , piece_id=LYFRMKYHVOCECGME4GJRAUGOWNBAOGGDVU7AS7GN6UBBHHZTTLHQ, seg_pos={0 0}, upload=58085]: Done uploading piece. err: <nil> Jul 27 00:09:25.531417748 (60h33m35.253980788s) [noise=true , address=45.91.116.8:4048 , piece_id=ML22FPHXORK7G6QYGZXIRKMHYQVBXWQT4FR3UIPCABJV674DO7UQ, seg_pos={0 0}, upload=58085]: Done uploading piece. err: <nil> Jul 27 00:09:25.542134953 (60h33m35.264697993s) [noise=true , address=45.91.116.7:4009 , piece_id=6XQK3DTIL3YS23DJKQFRNIZUMUY3PHF7A4H4C5OEJ3NHGZ7SQBFA, seg_pos={0 0}, upload=58085]: Done uploading piece. err: <nil> Jul 27 00:09:25.550081446 (60h33m35.272644486s) [noise=true , address=45.91.116.11:4119 , piece_id=G2KMTBCV2665ZX2O7GPCKBRFSSIHFF6SDY6TZWCDNZREWPZNQ24A, seg_pos={0 0}, upload=58085]: Done uploading piece. err: <nil> Jul 27 00:09:25.608111033 (60h33m35.330674073s) [noise=true , address=45.91.116.5:4297 , piece_id=EQEYGBRJY6O2IYYC344ZHXPZQTNOSBFR7S33LE7HJPFD5GSMFRBQ, seg_pos={0 0}, upload=58085]: Done uploading piece. err: <nil> Jul 27 00:09:25.614659251 (60h33m35.337222291s) [noise=true , address=77.90.31.11:4555 , piece_id=MJR6LHPXWOVMEFF6Q66MJ2FHCK6GA5M6F334RNC5J7JIZGJ7NJ4Q, seg_pos={0 0}, upload=58085]: Done uploading piece. err: <nil> Jul 27 00:09:25.731328030 (60h33m35.453891081s) [noise=true , address=77.90.31.12:4589 , piece_id=IHNWWRBL2E5VV77ANZHNWCCGC3L4JC5Y6F4DJWVGWQ7BESVK26MQ, seg_pos={0 0}, upload=58085]: Done uploading piece. err: <nil> Jul 27 00:09:26.884837420 (60h33m36.60740046s) [noise=true , address=77.90.31.18:4807 , piece_id=6QQ33YWILLVQCU4GOAOI6FOOPSXARZFG2YYPGY7BHESDNEGT57YQ, seg_pos={0 0}, upload=58085]: Done uploading piece. err: <nil> Jul 27 00:09:26.902326182 (60h33m36.624889222s) [noise=true , address=77.90.31.13:4628 , piece_id=OQHKCDILQA7TEDQP27C3SM7NZC3NSQTNTREJIQWFFPYZN6CHJDBQ, seg_pos={0 0}, upload=58085]: Done uploading piece. err: <nil> Jul 27 00:09:52.543670387 (60h34m2.266233467s) [noise=true , address=104.36.82.2:4730 , piece_id=AF6N5Z3FL5YDYGDLZJFTVORZSKPGCT5335GJSWDDZNUFBNHNQKGQ, seg_pos={0 0}, upload=58085]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 188.166.225.170:41546->104.36.82.2:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:09:52.543696337 (60h34m2.266259427s) [noise=true , address=104.36.82.16:4295 , piece_id=ZG5AHIG3CBQF2SKTYOWYG7Y53O3NBC36VU2I3CWIGIQZ5IUPT6UQ, seg_pos={0 0}, upload=58085]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: rpc: dial tcp 104.36.82.16:4295: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:09:52.543714811 (60h34m2.266277881s) [noise=true , address=104.36.82.8:4969 , piece_id=RGXUMB5UELRCP3JJNMDPNAUSKXAIB3H5R22KISQULBYM5JFTSRGQ, seg_pos={0 0}, upload=58085]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: write tcp 188.166.225.170:41130->104.36.82.8:4969: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:09:52.543776821 (60h34m2.266339921s) [noise=true , address=77.90.31.15:4698 , piece_id=JVSARDGWDK5B5QOARWEGBRMSQYKYOEXATS2FNVNAJXRJXNN2U5DA, seg_pos={0 0}, upload=58085]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: write tcp 188.166.225.170:49080->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:09:52.543813270 (60h34m2.26637632s) [seg_pos={0 0}, upload=58085]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:09:52.770059330 (60h34m2.49262237s) [upload=58085]: Done uploading segments... Jul 27 00:09:52.770089268 (60h34m2.492652308s) [upload=58085]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 7.096076656s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 52 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..XX.X.XXX.XXXXXX...] duration: 7.473576427s FAIL (2025-07-27 00:11:51 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 7.473602526s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [..XXX.X.....X.X..X..X.X....X.X....XXX.X...X......X.XX...XXXX] duration: 30.000885718s FAIL (2025-07-27 00:11:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.00090808s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 1.953156002s FAIL (2025-07-27 00:11:40 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.953188435s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.230953081s FAIL (2025-07-27 00:11:29 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== piece_id=ENGGRHWRI3OMM4BOAOK7JGHXFPHAG34DO6FQ75I2BIFMAAJ2HL3Q, seg_pos={0 0}, upload=58101]: Uploading piece... Jul 27 00:10:29.960984242 (60h34m39.683547312s) [noise=true , address=77.90.31.16:4747 , piece_id=XKGI6ZN7EGFFQVDN6R4C65VYA2XI45Q5FES2WG542YIR2KLAL3YA, seg_pos={0 0}, upload=58101]: Uploading piece... Jul 27 00:10:29.961282062 (60h34m39.683845102s) [noise=true , address=77.90.31.11:4581 , piece_id=MEVZS3JHH2HI46Y2AZ4RLKJK3I5QD432KBUYI6DFUX632X7XLUPA, seg_pos={0 0}, upload=58101]: Uploading piece... Jul 27 00:10:29.961544423 (60h34m39.684107464s) [noise=true , address=104.36.82.52:4115 , piece_id=S6KY6ZLJYXIPQ2YQ4P7EMDGCYJZXNKMDNEGM2YZTWY75HJ2ZB4NQ, seg_pos={0 0}, upload=58101]: Uploading piece... Jul 27 00:10:29.961578839 (60h34m39.684141879s) [noise=true , address=104.36.82.46:4488 , piece_id=UTRSLBGHKRDZY2LX3OSRCSAV663ZMJ3WZT2V3SIPBHY3D2WG7FKQ, seg_pos={0 0}, upload=58101]: Uploading piece... Jul 27 00:10:29.961829630 (60h34m39.68439267s) [noise=true , address=79.127.205.246:4278 , piece_id=7LSR475EINMZ2LGKN6DMZKPRCSW2LSEDOAWPYUMRWOXCHIIQCL7A, seg_pos={0 0}, upload=58101]: Uploading piece... Jul 27 00:10:29.961851902 (60h34m39.684414932s) [noise=true , address=77.90.31.18:4808 , piece_id=POFPOA5RP3BZPLCEZFQFGCN54DVAVFJUY4BF7HVIJ6LZHI7OTISQ, seg_pos={0 0}, upload=58101]: Uploading piece... Jul 27 00:10:29.962092733 (60h34m39.684655773s) [noise=true , address=45.91.116.8:4062 , piece_id=PCM46AIXPQRYH3IFIGCAMAGDVBXRQVAIISVHXFARWRVRQ4QSZPUQ, seg_pos={0 0}, upload=58101]: Uploading piece... Jul 27 00:10:29.962165031 (60h34m39.684728081s) [noise=true , address=79.127.220.97:4361 , piece_id=2ZLXHTXDVRB4FZ5UAJJIGHAYYLOS3AUCYKBJHLF2ZZX7QVVDIRRQ, seg_pos={0 0}, upload=58101]: Uploading piece... Jul 27 00:10:29.962360525 (60h34m39.684923565s) [noise=true , address=104.36.82.47:4533 , piece_id=PMYE3EDT3DHWUD67JYXTVW4EGQ5KB7NOWQJ2YAKNIBO5X42WTPRQ, seg_pos={0 0}, upload=58101]: Uploading piece... Jul 27 00:10:29.962406674 (60h34m39.684969704s) [noise=true , address=104.36.82.38:4689 , piece_id=EB6WFS7RTJAYPH2OC4BOBVD3JXLVTBZTTK3YDWB77YJP6PEZJFHA, seg_pos={0 0}, upload=58101]: Uploading piece... Jul 27 00:10:29.962613269 (60h34m39.685176299s) [noise=true , address=45.91.116.9:4277 , piece_id=LNGQZ2L7URAJDOGWPQNF554JJ3GAL53UMHKRZQVOW4YQPMCKLYUA, seg_pos={0 0}, upload=58101]: Uploading piece... Jul 27 00:10:29.962675848 (60h34m39.685238888s) [noise=true , address=79.127.205.245:4244 , piece_id=HU7Z5WKSQSKUD5ACVBDX5DXAZJD6LGLNOI33ZC6DW7D7D72U5DLA, seg_pos={0 0}, upload=58101]: Uploading piece... Jul 27 00:10:29.962830334 (60h34m39.685393374s) [noise=true , address=79.127.220.98:4422 , piece_id=624SSD7F2OZP5UTM26LTX3O2H2URNRMSEQN6Z2II7XQDVSSIS2DA, seg_pos={0 0}, upload=58101]: Uploading piece... Jul 27 00:10:29.962928943 (60h34m39.685491983s) [noise=true , address=77.90.31.19:4846 , piece_id=JBGDSLLASW2MIIRRUXCW7PYK24TWBRLLEBDCNMNAPRMNSJ7RD2WA, seg_pos={0 0}, upload=58101]: Uploading piece... Jul 27 00:10:29.963106011 (60h34m39.685669052s) [noise=true , address=104.36.82.44:4870 , piece_id=IPTBCOOX6CY2LVIBJQAXZGZLQREVQZOEZRJHAQUHHBGNUFVDQ4FQ, seg_pos={0 0}, upload=58101]: Uploading piece... Jul 27 00:10:29.963146589 (60h34m39.685709629s) [noise=true , address=79.127.219.47:4048 , piece_id=3AIR5DQG3EWOTVCP4OOXOBMGG2OWF4KDYZORUVB6DIVROL4MD6HA, seg_pos={0 0}, upload=58101]: Uploading piece... Jul 27 00:10:29.965030644 (60h34m39.687593684s) [noise=true , address=79.127.205.233:4640 , piece_id=TBAPYHJTOSXFY4R5CMZNHVP5VIBFYUQ7DQZYUZCE34ZKX66H26ZA, seg_pos={0 0}, upload=58101]: Uploading piece... Jul 27 00:10:31.057425385 (60h34m40.779988425s) [upload=58101]: Next returned nil segment Jul 27 00:10:31.057442587 (60h34m40.780005617s) [upload=58101]: Waiting for error group managing segments... Jul 27 00:10:31.358062366 (60h34m41.080625406s) [noise=true , address=79.127.205.246:4278 , piece_id=7LSR475EINMZ2LGKN6DMZKPRCSW2LSEDOAWPYUMRWOXCHIIQCL7A, seg_pos={0 0}, upload=58101]: Done uploading piece. err: <nil> Jul 27 00:10:31.365146409 (60h34m41.087709439s) [noise=true , address=79.127.205.245:4244 , piece_id=HU7Z5WKSQSKUD5ACVBDX5DXAZJD6LGLNOI33ZC6DW7D7D72U5DLA, seg_pos={0 0}, upload=58101]: Done uploading piece. err: <nil> Jul 27 00:10:31.394481485 (60h34m41.117044525s) [noise=true , address=79.127.205.233:4640 , piece_id=TBAPYHJTOSXFY4R5CMZNHVP5VIBFYUQ7DQZYUZCE34ZKX66H26ZA, seg_pos={0 0}, upload=58101]: Done uploading piece. err: <nil> Jul 27 00:10:31.451086155 (60h34m41.173649195s) [noise=true , address=45.91.116.8:4062 , piece_id=PCM46AIXPQRYH3IFIGCAMAGDVBXRQVAIISVHXFARWRVRQ4QSZPUQ, seg_pos={0 0}, upload=58101]: Done uploading piece. err: ecclient: upload failed (node:1ZgtJJK3pt2NqSfgFnXjZiYhDafhRdh6WkEf12AFTTuiku3Uhc, address:45.91.116.8:4062): protocol: storage node overloaded, request limit: 2000 Jul 27 00:10:31.944066471 (60h34m41.666629501s) [noise=true , address=77.90.31.16:4747 , piece_id=XKGI6ZN7EGFFQVDN6R4C65VYA2XI45Q5FES2WG542YIR2KLAL3YA, seg_pos={0 0}, upload=58101]: Done uploading piece. err: <nil> Jul 27 00:10:32.067400052 (60h34m41.789963092s) [noise=true , address=77.90.31.11:4581 , piece_id=MEVZS3JHH2HI46Y2AZ4RLKJK3I5QD432KBUYI6DFUX632X7XLUPA, seg_pos={0 0}, upload=58101]: Done uploading piece. err: <nil> Jul 27 00:10:34.098541683 (60h34m43.821104723s) [noise=true , address=79.127.220.97:4361 , piece_id=2ZLXHTXDVRB4FZ5UAJJIGHAYYLOS3AUCYKBJHLF2ZZX7QVVDIRRQ, seg_pos={0 0}, upload=58101]: Done uploading piece. err: <nil> Jul 27 00:10:35.030914207 (60h34m44.753477247s) [noise=true , address=79.127.220.98:4422 , piece_id=624SSD7F2OZP5UTM26LTX3O2H2URNRMSEQN6Z2II7XQDVSSIS2DA, seg_pos={0 0}, upload=58101]: Done uploading piece. err: <nil> Jul 27 00:10:35.422624445 (60h34m45.145187485s) [noise=true , address=79.127.219.47:4048 , piece_id=3AIR5DQG3EWOTVCP4OOXOBMGG2OWF4KDYZORUVB6DIVROL4MD6HA, seg_pos={0 0}, upload=58101]: Done uploading piece. err: <nil> Jul 27 00:10:35.728360292 (60h34m45.450923322s) [noise=true , address=77.90.31.19:4846 , piece_id=JBGDSLLASW2MIIRRUXCW7PYK24TWBRLLEBDCNMNAPRMNSJ7RD2WA, seg_pos={0 0}, upload=58101]: Done uploading piece. err: <nil> Jul 27 00:10:38.214886788 (60h34m47.937449888s) [noise=true , address=45.91.116.9:4277 , piece_id=LNGQZ2L7URAJDOGWPQNF554JJ3GAL53UMHKRZQVOW4YQPMCKLYUA, seg_pos={0 0}, upload=58101]: Done uploading piece. err: <nil> Jul 27 00:11:28.856710878 (60h35m38.579273908s) [noise=true , address=104.36.82.52:4115 , piece_id=S6KY6ZLJYXIPQ2YQ4P7EMDGCYJZXNKMDNEGM2YZTWY75HJ2ZB4NQ, seg_pos={0 0}, upload=58101]: Done uploading piece. err: ecclient: upload failed (node:1ikQ2b343QztY3vBFQ3cmEwEEFszpB2TPMtstufHdFfMEkJkBV, address:104.36.82.52:4115): protocol: rpc: dial tcp 104.36.82.52:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:11:28.856757427 (60h35m38.579320457s) [noise=true , address=104.36.82.46:4488 , piece_id=UTRSLBGHKRDZY2LX3OSRCSAV663ZMJ3WZT2V3SIPBHY3D2WG7FKQ, seg_pos={0 0}, upload=58101]: Done uploading piece. err: ecclient: upload failed (node:1MU3fdv9cTkq4pUZ3h989STAG1wxrbhFgzmr6tcrfumZKHP9BZ, address:104.36.82.46:4488): protocol: rpc: dial tcp 104.36.82.46:4488: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:11:28.856831208 (60h35m38.579394238s) [noise=true , address=77.90.31.18:4808 , piece_id=POFPOA5RP3BZPLCEZFQFGCN54DVAVFJUY4BF7HVIJ6LZHI7OTISQ, seg_pos={0 0}, upload=58101]: Done uploading piece. err: ecclient: upload failed (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, address:77.90.31.18:4808): protocol: write tcp 188.166.225.170:50928->77.90.31.18:4808: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:11:28.856960185 (60h35m38.579523225s) [noise=true , address=104.36.82.44:4870 , piece_id=IPTBCOOX6CY2LVIBJQAXZGZLQREVQZOEZRJHAQUHHBGNUFVDQ4FQ, seg_pos={0 0}, upload=58101]: Done uploading piece. err: ecclient: upload failed (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt, address:104.36.82.44:4870): protocol: rpc: dial tcp 104.36.82.44:4870: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:11:28.857005111 (60h35m38.579568191s) [noise=true , address=104.36.82.54:4166 , piece_id=ENGGRHWRI3OMM4BOAOK7JGHXFPHAG34DO6FQ75I2BIFMAAJ2HL3Q, seg_pos={0 0}, upload=58101]: Done uploading piece. err: ecclient: upload failed (node:1BV7mWGAwKMXzmZBc3qAQFUXYyWtgQC7QEfG54YJgyJ4U3tFiv, address:104.36.82.54:4166): protocol: write tcp 188.166.225.170:48336->104.36.82.54:4166: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:11:28.857048403 (60h35m38.579611443s) [noise=true , address=104.36.82.38:4689 , piece_id=EB6WFS7RTJAYPH2OC4BOBVD3JXLVTBZTTK3YDWB77YJP6PEZJFHA, seg_pos={0 0}, upload=58101]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:104.36.82.38:4689): protocol: write tcp 188.166.225.170:52234->104.36.82.38:4689: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:11:28.857114791 (60h35m38.579677831s) [noise=true , address=104.36.82.47:4533 , piece_id=PMYE3EDT3DHWUD67JYXTVW4EGQ5KB7NOWQJ2YAKNIBO5X42WTPRQ, seg_pos={0 0}, upload=58101]: Done uploading piece. err: ecclient: upload failed (node:1JUUdAnaUPAE8tkzPvVCokFbZMwmhqZvwKHXReK9VU2QeDqavR, address:104.36.82.47:4533): protocol: write tcp 188.166.225.170:44362->104.36.82.47:4533: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:11:28.857138366 (60h35m38.579701406s) [seg_pos={0 0}, upload=58101]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 27 00:11:29.085947751 (60h35m38.808510791s) [upload=58101]: Done uploading segments... Jul 27 00:11:29.085980072 (60h35m38.808543133s) [upload=58101]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.230977779s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 17 out of 60 is failed (threshold=3) [XXX.....XXX..X.....X......X...XX..XX.X.XX...X...............] duration: 18.044760268s FAIL (2025-07-27 00:11:03 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 18.044788652s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 1.874451351s FAIL (2025-07-27 00:10:58 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.874501938s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 53 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..XX.X.XXX.XXXXXX..] duration: 7.088718198s FAIL (2025-07-27 00:10:45 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 7.088741593s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 54 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXX.XXXXXXXXXX.XXXX..X.] duration: 7.624144686s FAIL (2025-07-27 00:10:39 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== }, upload=58077]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV): protocol: rpc: dial tcp 104.36.82.3:4765: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:08:58.749291092 (60h33m8.471854132s) [seg_pos={0 0}, upload=58077]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:08:58.749388238 (60h33m8.471951278s) [upload=58077]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:08:58.749395502 (60h33m8.471958542s) [upload=58077]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:08:59.071572826 (60h33m8.794135866s) [upload=58077]: Done uploading segments... Jul 27 00:08:59.071590289 (60h33m8.794153319s) [upload=58077]: Upload finished. err: <nil> Jul 27 00:09:23.191431943 (60h33m32.913994983s) [upload=58085]: Starting upload Jul 27 00:09:23.191484463 (60h33m32.914047503s) [upload=58085]: Done starting upload Jul 27 00:09:23.191558114 (60h33m32.914121154s) [upload=58085]: Uploading segments... Jul 27 00:09:23.191660350 (60h33m32.91422345s) [upload=58085]: Got next segment. Inline: false Jul 27 00:09:23.191695326 (60h33m32.914258356s) [upload=58085]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:09:23.191699464 (60h33m32.914262504s) [upload=58085]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:09:23.424088299 (60h33m33.146651339s) [seg_pos={0 0}, upload=58085]: Begin upload segment... Jul 27 00:09:23.424132493 (60h33m33.146695533s) [seg_pos={0 0}, upload=58085]: Done begin upload segment. Jul 27 00:09:23.424154705 (60h33m33.146717746s) [noise=true , address=104.36.82.8:4969 , piece_id=RGXUMB5UELRCP3JJNMDPNAUSKXAIB3H5R22KISQULBYM5JFTSRGQ, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:23.424193240 (60h33m33.14675628s) [noise=true , address=104.36.82.2:4730 , piece_id=AF6N5Z3FL5YDYGDLZJFTVORZSKPGCT5335GJSWDDZNUFBNHNQKGQ, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:23.424508462 (60h33m33.147071503s) [noise=true , address=45.91.116.8:4048 , piece_id=ML22FPHXORK7G6QYGZXIRKMHYQVBXWQT4FR3UIPCABJV674DO7UQ, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:23.424731339 (60h33m33.147294389s) [noise=true , address=77.90.31.18:4807 , piece_id=6QQ33YWILLVQCU4GOAOI6FOOPSXARZFG2YYPGY7BHESDNEGT57YQ, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:23.424777367 (60h33m33.147340397s) [noise=true , address=45.91.116.8:4047 , piece_id=SUJWMYZOYMDNH6UOOJUYWNEWXIR7CDAKGSM6VX4FJCBMV7JNFEKQ, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:23.425039278 (60h33m33.147602318s) [noise=true , address=45.91.116.7:4009 , piece_id=6XQK3DTIL3YS23DJKQFRNIZUMUY3PHF7A4H4C5OEJ3NHGZ7SQBFA, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:23.425091207 (60h33m33.147654247s) [noise=true , address=45.91.116.12:4227 , piece_id=CIROC6VSFJQWW372P2NHHG2ZVJCOJAXOKMNAFXNY5PW324WDD5XA, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:23.425286531 (60h33m33.147849571s) [noise=true , address=45.91.116.11:4119 , piece_id=G2KMTBCV2665ZX2O7GPCKBRFSSIHFF6SDY6TZWCDNZREWPZNQ24A, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:23.425540938 (60h33m33.148103978s) [noise=true , address=77.90.31.15:4698 , piece_id=JVSARDGWDK5B5QOARWEGBRMSQYKYOEXATS2FNVNAJXRJXNN2U5DA, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:23.425552049 (60h33m33.148115089s) [noise=true , address=45.91.116.13:4154 , piece_id=LYFRMKYHVOCECGME4GJRAUGOWNBAOGGDVU7AS7GN6UBBHHZTTLHQ, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:23.425763755 (60h33m33.148326785s) [noise=true , address=77.90.31.11:4555 , piece_id=MJR6LHPXWOVMEFF6Q66MJ2FHCK6GA5M6F334RNC5J7JIZGJ7NJ4Q, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:23.425836293 (60h33m33.148399333s) [noise=true , address=104.36.82.16:4295 , piece_id=ZG5AHIG3CBQF2SKTYOWYG7Y53O3NBC36VU2I3CWIGIQZ5IUPT6UQ, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:23.426064771 (60h33m33.148627811s) [noise=true , address=45.91.116.13:4153 , piece_id=WF6LCBKFOXJF3DNCNZI4E4LXGXSYIWXWGDDEZHN4CBUFY6YPWJQA, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:23.426081131 (60h33m33.148644171s) [noise=true , address=77.90.31.13:4628 , piece_id=OQHKCDILQA7TEDQP27C3SM7NZC3NSQTNTREJIQWFFPYZN6CHJDBQ, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:23.426305661 (60h33m33.148868701s) [noise=true , address=77.90.31.12:4589 , piece_id=IHNWWRBL2E5VV77ANZHNWCCGC3L4JC5Y6F4DJWVGWQ7BESVK26MQ, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:23.426336901 (60h33m33.148899941s) [noise=true , address=45.91.116.5:4297 , piece_id=EQEYGBRJY6O2IYYC344ZHXPZQTNOSBFR7S33LE7HJPFD5GSMFRBQ, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:25.166110004 (60h33m34.888673044s) [upload=58085]: Next returned nil segment Jul 27 00:09:25.166138388 (60h33m34.888701428s) [upload=58085]: Waiting for error group managing segments... Jul 27 00:09:25.453532572 (60h33m35.176095613s) [noise=true , address=45.91.116.13:4153 , piece_id=WF6LCBKFOXJF3DNCNZI4E4LXGXSYIWXWGDDEZHN4CBUFY6YPWJQA, seg_pos={0 0}, upload=58085]: Done uploading piece. err: <nil> Jul 27 00:09:25.457229505 (60h33m35.179792545s) [noise=true , address=45.91.116.8:4047 , piece_id=SUJWMYZOYMDNH6UOOJUYWNEWXIR7CDAKGSM6VX4FJCBMV7JNFEKQ, seg_pos={0 0}, upload=58085]: Done uploading piece. err: <nil> Jul 27 00:09:25.526191879 (60h33m35.248754919s) [noise=true , address=45.91.116.12:4227 , piece_id=CIROC6VSFJQWW372P2NHHG2ZVJCOJAXOKMNAFXNY5PW324WDD5XA, seg_pos={0 0}, upload=58085]: Done uploading piece. err: <nil> Jul 27 00:09:25.527573101 (60h33m35.250136142s) [noise=true , address=45.91.116.13:4154 , piece_id=LYFRMKYHVOCECGME4GJRAUGOWNBAOGGDVU7AS7GN6UBBHHZTTLHQ, seg_pos={0 0}, upload=58085]: Done uploading piece. err: <nil> Jul 27 00:09:25.531417748 (60h33m35.253980788s) [noise=true , address=45.91.116.8:4048 , piece_id=ML22FPHXORK7G6QYGZXIRKMHYQVBXWQT4FR3UIPCABJV674DO7UQ, seg_pos={0 0}, upload=58085]: Done uploading piece. err: <nil> Jul 27 00:09:25.542134953 (60h33m35.264697993s) [noise=true , address=45.91.116.7:4009 , piece_id=6XQK3DTIL3YS23DJKQFRNIZUMUY3PHF7A4H4C5OEJ3NHGZ7SQBFA, seg_pos={0 0}, upload=58085]: Done uploading piece. err: <nil> Jul 27 00:09:25.550081446 (60h33m35.272644486s) [noise=true , address=45.91.116.11:4119 , piece_id=G2KMTBCV2665ZX2O7GPCKBRFSSIHFF6SDY6TZWCDNZREWPZNQ24A, seg_pos={0 0}, upload=58085]: Done uploading piece. err: <nil> Jul 27 00:09:25.608111033 (60h33m35.330674073s) [noise=true , address=45.91.116.5:4297 , piece_id=EQEYGBRJY6O2IYYC344ZHXPZQTNOSBFR7S33LE7HJPFD5GSMFRBQ, seg_pos={0 0}, upload=58085]: Done uploading piece. err: <nil> Jul 27 00:09:25.614659251 (60h33m35.337222291s) [noise=true , address=77.90.31.11:4555 , piece_id=MJR6LHPXWOVMEFF6Q66MJ2FHCK6GA5M6F334RNC5J7JIZGJ7NJ4Q, seg_pos={0 0}, upload=58085]: Done uploading piece. err: <nil> Jul 27 00:09:25.731328030 (60h33m35.453891081s) [noise=true , address=77.90.31.12:4589 , piece_id=IHNWWRBL2E5VV77ANZHNWCCGC3L4JC5Y6F4DJWVGWQ7BESVK26MQ, seg_pos={0 0}, upload=58085]: Done uploading piece. err: <nil> Jul 27 00:09:26.884837420 (60h33m36.60740046s) [noise=true , address=77.90.31.18:4807 , piece_id=6QQ33YWILLVQCU4GOAOI6FOOPSXARZFG2YYPGY7BHESDNEGT57YQ, seg_pos={0 0}, upload=58085]: Done uploading piece. err: <nil> Jul 27 00:09:26.902326182 (60h33m36.624889222s) [noise=true , address=77.90.31.13:4628 , piece_id=OQHKCDILQA7TEDQP27C3SM7NZC3NSQTNTREJIQWFFPYZN6CHJDBQ, seg_pos={0 0}, upload=58085]: Done uploading piece. err: <nil> Jul 27 00:09:52.543670387 (60h34m2.266233467s) [noise=true , address=104.36.82.2:4730 , piece_id=AF6N5Z3FL5YDYGDLZJFTVORZSKPGCT5335GJSWDDZNUFBNHNQKGQ, seg_pos={0 0}, upload=58085]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 188.166.225.170:41546->104.36.82.2:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:09:52.543696337 (60h34m2.266259427s) [noise=true , address=104.36.82.16:4295 , piece_id=ZG5AHIG3CBQF2SKTYOWYG7Y53O3NBC36VU2I3CWIGIQZ5IUPT6UQ, seg_pos={0 0}, upload=58085]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: rpc: dial tcp 104.36.82.16:4295: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:09:52.543714811 (60h34m2.266277881s) [noise=true , address=104.36.82.8:4969 , piece_id=RGXUMB5UELRCP3JJNMDPNAUSKXAIB3H5R22KISQULBYM5JFTSRGQ, seg_pos={0 0}, upload=58085]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: write tcp 188.166.225.170:41130->104.36.82.8:4969: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:09:52.543776821 (60h34m2.266339921s) [noise=true , address=77.90.31.15:4698 , piece_id=JVSARDGWDK5B5QOARWEGBRMSQYKYOEXATS2FNVNAJXRJXNN2U5DA, seg_pos={0 0}, upload=58085]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: write tcp 188.166.225.170:49080->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:09:52.543813270 (60h34m2.26637632s) [seg_pos={0 0}, upload=58085]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:09:52.770059330 (60h34m2.49262237s) [upload=58085]: Done uploading segments... Jul 27 00:09:52.770089268 (60h34m2.492652308s) [upload=58085]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 7.624172981s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [...XXX.X.....X.X..X..X.X....X.X....XXX.X...X......X.XX...XXX] duration: 30.000458881s FAIL (2025-07-27 00:10:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000469311s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.25568657s FAIL (2025-07-27 00:10:23 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 15762s) [upload=58086]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:09:24.686101576 (60h33m34.408664616s) [seg_pos={0 0}, upload=58086]: Begin upload segment... Jul 27 00:09:24.686160258 (60h33m34.408723288s) [seg_pos={0 0}, upload=58086]: Done begin upload segment. Jul 27 00:09:24.686189354 (60h33m34.408752394s) [noise=true , address=79.127.205.246:4260 , piece_id=QEI2FBYJJGMY6FHYU25QOFTXIOQRWTVXNVKCNPVIQHVMON4UKFFQ, seg_pos={0 0}, upload=58086]: Uploading piece... Jul 27 00:09:24.686197971 (60h33m34.408761001s) [noise=true , address=156.146.43.225:4448 , piece_id=YLGP74SY5PSZETOZDSDR6KNKU2XQFQ5OQ6NUT465E2PPRTL6VLFA, seg_pos={0 0}, upload=58086]: Uploading piece... Jul 27 00:09:24.686587656 (60h33m34.409150697s) [noise=true , address=77.90.31.15:4717 , piece_id=YUL6OKG4DOZ5LL5SAB3XLZT3IWBAJDBTWDBVYLOC2ALMZPL55HGQ, seg_pos={0 0}, upload=58086]: Uploading piece... Jul 27 00:09:24.686610470 (60h33m34.40917351s) [noise=true , address=104.36.82.14:4249 , piece_id=BSJHZEKSG33WSWXWVM5AHWJDEIVI5D7GMF4M2FQBIJ4IE3JLTO5Q, seg_pos={0 0}, upload=58086]: Uploading piece... Jul 27 00:09:24.686910794 (60h33m34.409473824s) [noise=true , address=79.127.201.215:4357 , piece_id=ZKIJASB57WEYXHB46PT2UHHBGTSFFWJZHAXZFUTLDRFHZUW4YVFQ, seg_pos={0 0}, upload=58086]: Uploading piece... Jul 27 00:09:24.686971751 (60h33m34.409534791s) [noise=true , address=77.90.31.19:4847 , piece_id=C4X7BU57535CR7YSGN73C3YF4SX6A6RUEI2H2YVXHDIXRUZMQNCA, seg_pos={0 0}, upload=58086]: Uploading piece... Jul 27 00:09:24.687216329 (60h33m34.40977936s) [noise=true , address=104.36.82.55:4217 , piece_id=5DY4Q3OU7BT7LEQIPIA22SPCIKLMH2CAJSBU366DXOGGANCV5GZQ, seg_pos={0 0}, upload=58086]: Uploading piece... Jul 27 00:09:24.687286513 (60h33m34.409849553s) [noise=true , address=79.127.219.47:4048 , piece_id=ZMCODSORUV6Y37UWYC7PUPVTURVVCP4MZ74HSAFALKQWOLJPPN3A, seg_pos={0 0}, upload=58086]: Uploading piece... Jul 27 00:09:24.687496445 (60h33m34.410059485s) [noise=true , address=77.90.31.10:4528 , piece_id=XMNFWOI55V7CIKQFBIBSTTSVWW4UAN5GMOQRDTUMZIAHRYEVDBDA, seg_pos={0 0}, upload=58086]: Uploading piece... Jul 27 00:09:24.687753146 (60h33m34.410316186s) [noise=true , address=79.127.219.38:4004 , piece_id=TG7AXUQ7XTLFENB4RFNFAMYZUY2DCR7MSGSMOZK5VEJA5ZUCXDGA, seg_pos={0 0}, upload=58086]: Uploading piece... Jul 27 00:09:24.687991753 (60h33m34.410554783s) [noise=true , address=79.127.205.234:4597 , piece_id=KMQAEF663QK2V5LR2ZML6EE7AAUNH72PW2HQXLPWKYG5C4WFPVAA, seg_pos={0 0}, upload=58086]: Uploading piece... Jul 27 00:09:24.688301886 (60h33m34.410864926s) [noise=true , address=79.127.219.48:4074 , piece_id=5GU4SGJYKYULA53USXR2CJFLBGPMVAZ3AD3ROEKQDXOMMINN7GQA, seg_pos={0 0}, upload=58086]: Uploading piece... Jul 27 00:09:24.688518040 (60h33m34.41108108s) [noise=true , address=104.36.82.46:4488 , piece_id=6MQUXWLTRNTKO6XWRF2YC5G4X4BUIBZ75FMK72XJ34KHV3EQLCAA, seg_pos={0 0}, upload=58086]: Uploading piece... Jul 27 00:09:24.688777977 (60h33m34.411341007s) [noise=true , address=64.49.13.138:4488 , piece_id=TV2WT4RXACQVFVAIS3EIR5V3KNBA5ZWQQ7M2WWOKYC5WYQ2DCZFA, seg_pos={0 0}, upload=58086]: Uploading piece... Jul 27 00:09:24.688995473 (60h33m34.411558513s) [noise=true , address=79.127.205.245:4242 , piece_id=VPKPS44T6WPAS6ITZKVHWSLGNE6VO5SUL6XDMQY4H4SHTM7JAEKA, seg_pos={0 0}, upload=58086]: Uploading piece... Jul 27 00:09:24.689256663 (60h33m34.411819703s) [noise=true , address=79.127.205.235:4689 , piece_id=X747T7GUJFOCFAF2RAIDQVE6XEZXU753JF6YM5474RJKBS5TITVA, seg_pos={0 0}, upload=58086]: Uploading piece... Jul 27 00:09:24.689466354 (60h33m34.412029394s) [noise=true , address=104.36.82.56:4259 , piece_id=GUP64TUWZ4DCOZAA6FN2ASOH7QF2AZVFNVN2TJMVO7KI7XJ4CKIQ, seg_pos={0 0}, upload=58086]: Uploading piece... Jul 27 00:09:24.689707966 (60h33m34.412270996s) [noise=true , address=104.36.82.19:4427 , piece_id=VKTF65IPBPUCSNCL5HLEFSLBWC3OSSAOEVTJU2RI7T26DX6UAYQA, seg_pos={0 0}, upload=58086]: Uploading piece... Jul 27 00:09:25.792629708 (60h33m35.515192748s) [upload=58086]: Next returned nil segment Jul 27 00:09:25.793502126 (60h33m35.516065156s) [upload=58086]: Waiting for error group managing segments... Jul 27 00:09:25.805481586 (60h33m35.528044626s) [noise=true , address=79.127.219.48:4074 , piece_id=5GU4SGJYKYULA53USXR2CJFLBGPMVAZ3AD3ROEKQDXOMMINN7GQA, seg_pos={0 0}, upload=58086]: Done uploading piece. err: ecclient: upload failed (node:127YpFQAmSFbgY2cmAEwcCPDUhg9AnRHS3SkcafEaQeSGVUwVdj, address:79.127.219.48:4074): protocol: storage node overloaded, request limit: 2000 Jul 27 00:09:26.102507057 (60h33m35.825070097s) [noise=true , address=79.127.205.245:4242 , piece_id=VPKPS44T6WPAS6ITZKVHWSLGNE6VO5SUL6XDMQY4H4SHTM7JAEKA, seg_pos={0 0}, upload=58086]: Done uploading piece. err: <nil> Jul 27 00:09:26.108799727 (60h33m35.831362777s) [noise=true , address=79.127.205.235:4689 , piece_id=X747T7GUJFOCFAF2RAIDQVE6XEZXU753JF6YM5474RJKBS5TITVA, seg_pos={0 0}, upload=58086]: Done uploading piece. err: <nil> Jul 27 00:09:26.123596598 (60h33m35.846159638s) [noise=true , address=79.127.205.246:4260 , piece_id=QEI2FBYJJGMY6FHYU25QOFTXIOQRWTVXNVKCNPVIQHVMON4UKFFQ, seg_pos={0 0}, upload=58086]: Done uploading piece. err: <nil> Jul 27 00:09:26.160404912 (60h33m35.882967952s) [noise=true , address=79.127.205.234:4597 , piece_id=KMQAEF663QK2V5LR2ZML6EE7AAUNH72PW2HQXLPWKYG5C4WFPVAA, seg_pos={0 0}, upload=58086]: Done uploading piece. err: <nil> Jul 27 00:09:26.692108297 (60h33m36.414671337s) [noise=true , address=79.127.219.38:4004 , piece_id=TG7AXUQ7XTLFENB4RFNFAMYZUY2DCR7MSGSMOZK5VEJA5ZUCXDGA, seg_pos={0 0}, upload=58086]: Done uploading piece. err: <nil> Jul 27 00:09:26.874541202 (60h33m36.597104232s) [noise=true , address=77.90.31.10:4528 , piece_id=XMNFWOI55V7CIKQFBIBSTTSVWW4UAN5GMOQRDTUMZIAHRYEVDBDA, seg_pos={0 0}, upload=58086]: Done uploading piece. err: <nil> Jul 27 00:09:27.819559289 (60h33m37.542122329s) [noise=true , address=79.127.219.47:4048 , piece_id=ZMCODSORUV6Y37UWYC7PUPVTURVVCP4MZ74HSAFALKQWOLJPPN3A, seg_pos={0 0}, upload=58086]: Done uploading piece. err: <nil> Jul 27 00:09:28.856775588 (60h33m38.579338628s) [noise=true , address=64.49.13.138:4488 , piece_id=TV2WT4RXACQVFVAIS3EIR5V3KNBA5ZWQQ7M2WWOKYC5WYQ2DCZFA, seg_pos={0 0}, upload=58086]: Done uploading piece. err: <nil> Jul 27 00:09:29.980372795 (60h33m39.702935835s) [noise=true , address=156.146.43.225:4448 , piece_id=YLGP74SY5PSZETOZDSDR6KNKU2XQFQ5OQ6NUT465E2PPRTL6VLFA, seg_pos={0 0}, upload=58086]: Done uploading piece. err: <nil> Jul 27 00:09:43.674892227 (60h33m53.397455267s) [noise=true , address=77.90.31.19:4847 , piece_id=C4X7BU57535CR7YSGN73C3YF4SX6A6RUEI2H2YVXHDIXRUZMQNCA, seg_pos={0 0}, upload=58086]: Done uploading piece. err: <nil> Jul 27 00:10:01.216970223 (60h34m10.939533264s) [noise=true , address=79.127.201.215:4357 , piece_id=ZKIJASB57WEYXHB46PT2UHHBGTSFFWJZHAXZFUTLDRFHZUW4YVFQ, seg_pos={0 0}, upload=58086]: Done uploading piece. err: <nil> Jul 27 00:10:23.570404312 (60h34m33.292967352s) [noise=true , address=104.36.82.56:4259 , piece_id=GUP64TUWZ4DCOZAA6FN2ASOH7QF2AZVFNVN2TJMVO7KI7XJ4CKIQ, seg_pos={0 0}, upload=58086]: Done uploading piece. err: ecclient: upload failed (node:12BCJsPfe1NNUxVUfQWKbvkzviQWYgMxkRZjDhKJ7KFZc1GmnjV, address:104.36.82.56:4259): protocol: rpc: dial tcp 104.36.82.56:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:10:23.570447816 (60h34m33.293010856s) [noise=true , address=104.36.82.55:4217 , piece_id=5DY4Q3OU7BT7LEQIPIA22SPCIKLMH2CAJSBU366DXOGGANCV5GZQ, seg_pos={0 0}, upload=58086]: Done uploading piece. err: ecclient: upload failed (node:12bw9V8UMi7pFxqJ8fh22x5ZPmtJUJyJGw8hD9NhQcfHdfx4cZj, address:104.36.82.55:4217): protocol: rpc: dial tcp 104.36.82.55:4217: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:10:23.570480628 (60h34m33.293043668s) [noise=true , address=104.36.82.46:4488 , piece_id=6MQUXWLTRNTKO6XWRF2YC5G4X4BUIBZ75FMK72XJ34KHV3EQLCAA, seg_pos={0 0}, upload=58086]: Done uploading piece. err: ecclient: upload failed (node:1MU3fdv9cTkq4pUZ3h989STAG1wxrbhFgzmr6tcrfumZKHP9BZ, address:104.36.82.46:4488): protocol: rpc: dial tcp 104.36.82.46:4488: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:10:23.570492442 (60h34m33.293055482s) [noise=true , address=104.36.82.14:4249 , piece_id=BSJHZEKSG33WSWXWVM5AHWJDEIVI5D7GMF4M2FQBIJ4IE3JLTO5Q, seg_pos={0 0}, upload=58086]: Done uploading piece. err: ecclient: upload failed (node:1GnfZTc54L9eRyjCUwQuaFcG6FjsUDjDwT3Gx4zoNTQrvP158L, address:104.36.82.14:4249): protocol: rpc: dial tcp 104.36.82.14:4249: i/o timeout; context deadline exceeded Jul 27 00:10:23.570511227 (60h34m33.293074267s) [noise=true , address=104.36.82.19:4427 , piece_id=VKTF65IPBPUCSNCL5HLEFSLBWC3OSSAOEVTJU2RI7T26DX6UAYQA, seg_pos={0 0}, upload=58086]: Done uploading piece. err: ecclient: upload failed (node:1NdaXHU5WMvCtca7R6XRycKQGBQio4cikujGTu8TwnyPcGf7uG, address:104.36.82.19:4427): protocol: rpc: dial tcp 104.36.82.19:4427: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:10:23.570546214 (60h34m33.293109255s) [noise=true , address=77.90.31.15:4717 , piece_id=YUL6OKG4DOZ5LL5SAB3XLZT3IWBAJDBTWDBVYLOC2ALMZPL55HGQ, seg_pos={0 0}, upload=58086]: Done uploading piece. err: ecclient: upload failed (node:12ULANtxKNaJviAZFsmHjDQWjG4x2wfmjCwbQtHmn9LpnELmkPc, address:77.90.31.15:4717): protocol: write tcp 188.166.225.170:53434->77.90.31.15:4717: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:10:23.570566783 (60h34m33.293129823s) [seg_pos={0 0}, upload=58086]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 27 00:10:23.823831128 (60h34m33.546394168s) [upload=58086]: Done uploading segments... Jul 27 00:10:23.823856045 (60h34m33.546419085s) [upload=58086]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration:* 1m0.255703863s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [XXXX.....XXX..X.....X......X...XX..XX.X.XX...X..............] duration: 30.211471675s FAIL (2025-07-27 00:10:16 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.211485181s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 54 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..XX.X.XXX.XXXXXX.] duration: 6.993072152s FAIL (2025-07-27 00:10:07 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.993097601s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [X...XXX.X.....X.X..X..X.X....X.X....XXX.X...X......X.XX...XX] duration: 30.000893536s FAIL (2025-07-27 00:09:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000932181s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 55 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXX.XXXXXXXXXX.XXXX..X] duration: 30.227924289s FAIL (2025-07-27 00:09:52 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== }, upload=58077]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV): protocol: rpc: dial tcp 104.36.82.3:4765: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:08:58.749291092 (60h33m8.471854132s) [seg_pos={0 0}, upload=58077]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:08:58.749388238 (60h33m8.471951278s) [upload=58077]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:08:58.749395502 (60h33m8.471958542s) [upload=58077]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:08:59.071572826 (60h33m8.794135866s) [upload=58077]: Done uploading segments... Jul 27 00:08:59.071590289 (60h33m8.794153319s) [upload=58077]: Upload finished. err: <nil> Jul 27 00:09:23.191431943 (60h33m32.913994983s) [upload=58085]: Starting upload Jul 27 00:09:23.191484463 (60h33m32.914047503s) [upload=58085]: Done starting upload Jul 27 00:09:23.191558114 (60h33m32.914121154s) [upload=58085]: Uploading segments... Jul 27 00:09:23.191660350 (60h33m32.91422345s) [upload=58085]: Got next segment. Inline: false Jul 27 00:09:23.191695326 (60h33m32.914258356s) [upload=58085]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:09:23.191699464 (60h33m32.914262504s) [upload=58085]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:09:23.424088299 (60h33m33.146651339s) [seg_pos={0 0}, upload=58085]: Begin upload segment... Jul 27 00:09:23.424132493 (60h33m33.146695533s) [seg_pos={0 0}, upload=58085]: Done begin upload segment. Jul 27 00:09:23.424154705 (60h33m33.146717746s) [noise=true , address=104.36.82.8:4969 , piece_id=RGXUMB5UELRCP3JJNMDPNAUSKXAIB3H5R22KISQULBYM5JFTSRGQ, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:23.424193240 (60h33m33.14675628s) [noise=true , address=104.36.82.2:4730 , piece_id=AF6N5Z3FL5YDYGDLZJFTVORZSKPGCT5335GJSWDDZNUFBNHNQKGQ, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:23.424508462 (60h33m33.147071503s) [noise=true , address=45.91.116.8:4048 , piece_id=ML22FPHXORK7G6QYGZXIRKMHYQVBXWQT4FR3UIPCABJV674DO7UQ, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:23.424731339 (60h33m33.147294389s) [noise=true , address=77.90.31.18:4807 , piece_id=6QQ33YWILLVQCU4GOAOI6FOOPSXARZFG2YYPGY7BHESDNEGT57YQ, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:23.424777367 (60h33m33.147340397s) [noise=true , address=45.91.116.8:4047 , piece_id=SUJWMYZOYMDNH6UOOJUYWNEWXIR7CDAKGSM6VX4FJCBMV7JNFEKQ, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:23.425039278 (60h33m33.147602318s) [noise=true , address=45.91.116.7:4009 , piece_id=6XQK3DTIL3YS23DJKQFRNIZUMUY3PHF7A4H4C5OEJ3NHGZ7SQBFA, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:23.425091207 (60h33m33.147654247s) [noise=true , address=45.91.116.12:4227 , piece_id=CIROC6VSFJQWW372P2NHHG2ZVJCOJAXOKMNAFXNY5PW324WDD5XA, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:23.425286531 (60h33m33.147849571s) [noise=true , address=45.91.116.11:4119 , piece_id=G2KMTBCV2665ZX2O7GPCKBRFSSIHFF6SDY6TZWCDNZREWPZNQ24A, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:23.425540938 (60h33m33.148103978s) [noise=true , address=77.90.31.15:4698 , piece_id=JVSARDGWDK5B5QOARWEGBRMSQYKYOEXATS2FNVNAJXRJXNN2U5DA, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:23.425552049 (60h33m33.148115089s) [noise=true , address=45.91.116.13:4154 , piece_id=LYFRMKYHVOCECGME4GJRAUGOWNBAOGGDVU7AS7GN6UBBHHZTTLHQ, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:23.425763755 (60h33m33.148326785s) [noise=true , address=77.90.31.11:4555 , piece_id=MJR6LHPXWOVMEFF6Q66MJ2FHCK6GA5M6F334RNC5J7JIZGJ7NJ4Q, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:23.425836293 (60h33m33.148399333s) [noise=true , address=104.36.82.16:4295 , piece_id=ZG5AHIG3CBQF2SKTYOWYG7Y53O3NBC36VU2I3CWIGIQZ5IUPT6UQ, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:23.426064771 (60h33m33.148627811s) [noise=true , address=45.91.116.13:4153 , piece_id=WF6LCBKFOXJF3DNCNZI4E4LXGXSYIWXWGDDEZHN4CBUFY6YPWJQA, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:23.426081131 (60h33m33.148644171s) [noise=true , address=77.90.31.13:4628 , piece_id=OQHKCDILQA7TEDQP27C3SM7NZC3NSQTNTREJIQWFFPYZN6CHJDBQ, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:23.426305661 (60h33m33.148868701s) [noise=true , address=77.90.31.12:4589 , piece_id=IHNWWRBL2E5VV77ANZHNWCCGC3L4JC5Y6F4DJWVGWQ7BESVK26MQ, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:23.426336901 (60h33m33.148899941s) [noise=true , address=45.91.116.5:4297 , piece_id=EQEYGBRJY6O2IYYC344ZHXPZQTNOSBFR7S33LE7HJPFD5GSMFRBQ, seg_pos={0 0}, upload=58085]: Uploading piece... Jul 27 00:09:25.166110004 (60h33m34.888673044s) [upload=58085]: Next returned nil segment Jul 27 00:09:25.166138388 (60h33m34.888701428s) [upload=58085]: Waiting for error group managing segments... Jul 27 00:09:25.453532572 (60h33m35.176095613s) [noise=true , address=45.91.116.13:4153 , piece_id=WF6LCBKFOXJF3DNCNZI4E4LXGXSYIWXWGDDEZHN4CBUFY6YPWJQA, seg_pos={0 0}, upload=58085]: Done uploading piece. err: <nil> Jul 27 00:09:25.457229505 (60h33m35.179792545s) [noise=true , address=45.91.116.8:4047 , piece_id=SUJWMYZOYMDNH6UOOJUYWNEWXIR7CDAKGSM6VX4FJCBMV7JNFEKQ, seg_pos={0 0}, upload=58085]: Done uploading piece. err: <nil> Jul 27 00:09:25.526191879 (60h33m35.248754919s) [noise=true , address=45.91.116.12:4227 , piece_id=CIROC6VSFJQWW372P2NHHG2ZVJCOJAXOKMNAFXNY5PW324WDD5XA, seg_pos={0 0}, upload=58085]: Done uploading piece. err: <nil> Jul 27 00:09:25.527573101 (60h33m35.250136142s) [noise=true , address=45.91.116.13:4154 , piece_id=LYFRMKYHVOCECGME4GJRAUGOWNBAOGGDVU7AS7GN6UBBHHZTTLHQ, seg_pos={0 0}, upload=58085]: Done uploading piece. err: <nil> Jul 27 00:09:25.531417748 (60h33m35.253980788s) [noise=true , address=45.91.116.8:4048 , piece_id=ML22FPHXORK7G6QYGZXIRKMHYQVBXWQT4FR3UIPCABJV674DO7UQ, seg_pos={0 0}, upload=58085]: Done uploading piece. err: <nil> Jul 27 00:09:25.542134953 (60h33m35.264697993s) [noise=true , address=45.91.116.7:4009 , piece_id=6XQK3DTIL3YS23DJKQFRNIZUMUY3PHF7A4H4C5OEJ3NHGZ7SQBFA, seg_pos={0 0}, upload=58085]: Done uploading piece. err: <nil> Jul 27 00:09:25.550081446 (60h33m35.272644486s) [noise=true , address=45.91.116.11:4119 , piece_id=G2KMTBCV2665ZX2O7GPCKBRFSSIHFF6SDY6TZWCDNZREWPZNQ24A, seg_pos={0 0}, upload=58085]: Done uploading piece. err: <nil> Jul 27 00:09:25.608111033 (60h33m35.330674073s) [noise=true , address=45.91.116.5:4297 , piece_id=EQEYGBRJY6O2IYYC344ZHXPZQTNOSBFR7S33LE7HJPFD5GSMFRBQ, seg_pos={0 0}, upload=58085]: Done uploading piece. err: <nil> Jul 27 00:09:25.614659251 (60h33m35.337222291s) [noise=true , address=77.90.31.11:4555 , piece_id=MJR6LHPXWOVMEFF6Q66MJ2FHCK6GA5M6F334RNC5J7JIZGJ7NJ4Q, seg_pos={0 0}, upload=58085]: Done uploading piece. err: <nil> Jul 27 00:09:25.731328030 (60h33m35.453891081s) [noise=true , address=77.90.31.12:4589 , piece_id=IHNWWRBL2E5VV77ANZHNWCCGC3L4JC5Y6F4DJWVGWQ7BESVK26MQ, seg_pos={0 0}, upload=58085]: Done uploading piece. err: <nil> Jul 27 00:09:26.884837420 (60h33m36.60740046s) [noise=true , address=77.90.31.18:4807 , piece_id=6QQ33YWILLVQCU4GOAOI6FOOPSXARZFG2YYPGY7BHESDNEGT57YQ, seg_pos={0 0}, upload=58085]: Done uploading piece. err: <nil> Jul 27 00:09:26.902326182 (60h33m36.624889222s) [noise=true , address=77.90.31.13:4628 , piece_id=OQHKCDILQA7TEDQP27C3SM7NZC3NSQTNTREJIQWFFPYZN6CHJDBQ, seg_pos={0 0}, upload=58085]: Done uploading piece. err: <nil> Jul 27 00:09:52.543670387 (60h34m2.266233467s) [noise=true , address=104.36.82.2:4730 , piece_id=AF6N5Z3FL5YDYGDLZJFTVORZSKPGCT5335GJSWDDZNUFBNHNQKGQ, seg_pos={0 0}, upload=58085]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 188.166.225.170:41546->104.36.82.2:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:09:52.543696337 (60h34m2.266259427s) [noise=true , address=104.36.82.16:4295 , piece_id=ZG5AHIG3CBQF2SKTYOWYG7Y53O3NBC36VU2I3CWIGIQZ5IUPT6UQ, seg_pos={0 0}, upload=58085]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: rpc: dial tcp 104.36.82.16:4295: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:09:52.543714811 (60h34m2.266277881s) [noise=true , address=104.36.82.8:4969 , piece_id=RGXUMB5UELRCP3JJNMDPNAUSKXAIB3H5R22KISQULBYM5JFTSRGQ, seg_pos={0 0}, upload=58085]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: write tcp 188.166.225.170:41130->104.36.82.8:4969: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:09:52.543776821 (60h34m2.266339921s) [noise=true , address=77.90.31.15:4698 , piece_id=JVSARDGWDK5B5QOARWEGBRMSQYKYOEXATS2FNVNAJXRJXNN2U5DA, seg_pos={0 0}, upload=58085]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: write tcp 188.166.225.170:49080->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:09:52.543813270 (60h34m2.26637632s) [seg_pos={0 0}, upload=58085]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:09:52.770059330 (60h34m2.49262237s) [upload=58085]: Done uploading segments... Jul 27 00:09:52.770089268 (60h34m2.492652308s) [upload=58085]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.227946281s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 1.847118933s FAIL (2025-07-27 00:09:47 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.847148731s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [.XXXX.....XXX..X.....X......X...XX..XX.X.XX...X.............] duration: 52.860733119s FAIL (2025-07-27 00:09:46 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 52.860746493s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 55 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..XX.X.XXX.XXXXXX] duration: 30.264421826s FAIL (2025-07-27 00:09:40 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.264443898s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.234980967s FAIL (2025-07-27 00:09:23 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== , piece_id=HUUZUMK2MXON23DYDTI5SAVL2PDGQCZJBFOSBBSVY3I5IXCRBXAA, seg_pos={0 0}, upload=58070]: Uploading piece... Jul 27 00:08:24.191468653 (60h32m33.914031693s) [noise=true , address=79.127.219.38:4003 , piece_id=HGV4JIAWMIF52L6BMBE2JTBDSZXRKDPBRCIBFS72SDIKUL5X4QCQ, seg_pos={0 0}, upload=58070]: Uploading piece... Jul 27 00:08:24.191592921 (60h32m33.914155961s) [noise=true , address=45.91.116.9:4277 , piece_id=VI4FPX34RBSQBZVYJG6JEHWH3VDIZFD36TSEYKUWBC5MZTSCFD4A, seg_pos={0 0}, upload=58070]: Uploading piece... Jul 27 00:08:24.191758949 (60h32m33.914322019s) [noise=true , address=77.90.31.12:4608 , piece_id=PNSNCD462QGF44F57B5ABKWNAFXNXRRCIJOUPLQWWYR5UIYC3LQA, seg_pos={0 0}, upload=58070]: Uploading piece... Jul 27 00:08:24.191896712 (60h32m33.914459752s) [noise=true , address=104.36.82.50:4016 , piece_id=WDZYETH3W6LW2T6UWYTQMN6SFLGHIAJWEDRAD2P3AE4A2UMNCPOA, seg_pos={0 0}, upload=58070]: Uploading piece... Jul 27 00:08:24.192077898 (60h32m33.914640939s) [noise=true , address=79.127.220.101:4041 , piece_id=EMIQWCARLYTXACBD6O6DZH77RLYWX7EKA5PMFVNCJVQLCFW7WLFA, seg_pos={0 0}, upload=58070]: Uploading piece... Jul 27 00:08:24.192227135 (60h32m33.914790175s) [noise=true , address=104.36.82.47:4532 , piece_id=BV2SJ3WCFJVTTO3A5QMITXSZVAMCQTKHJZOQQEQWVE5B4FUCHCJQ, seg_pos={0 0}, upload=58070]: Uploading piece... Jul 27 00:08:24.192366661 (60h32m33.914929702s) [noise=true , address=79.127.205.245:4250 , piece_id=KVMWKUPRKG2AD5YD2DLDBASPNKQWX3DVK4HPGJRTQE4HZEF36SZA, seg_pos={0 0}, upload=58070]: Uploading piece... Jul 27 00:08:24.192522870 (60h32m33.91508591s) [noise=true , address=79.127.219.47:4048 , piece_id=LWIVS27EL4PHHBJ7W44KW3LNXSLK3VLXLGACZPIQZMIQVRLZ5HCA, seg_pos={0 0}, upload=58070]: Uploading piece... Jul 27 00:08:24.192645184 (60h32m33.915208244s) [noise=true , address=79.127.205.235:4680 , piece_id=5SEJDTYH3PQAZSRZ3BNCILT4SPTDBN6F2I5CBM7XCF5HDFPYUU2A, seg_pos={0 0}, upload=58070]: Uploading piece... Jul 27 00:08:24.192798868 (60h32m33.915361908s) [noise=true , address=104.36.82.54:4192 , piece_id=ERPUZFYA2ND4TU5NBQPX22UW426C4VSOW2Z7TJBNYIMTAJUVVP4Q, seg_pos={0 0}, upload=58070]: Uploading piece... Jul 27 00:08:24.192909720 (60h32m33.91547276s) [noise=true , address=104.36.82.41:4783 , piece_id=FSSYD2VORTCAASQAK62RV4RFU5XFRJXNT4LTSRFQDCC5HJCNOUKQ, seg_pos={0 0}, upload=58070]: Uploading piece... Jul 27 00:08:24.193028888 (60h32m33.915591928s) [noise=true , address=104.36.82.46:4491 , piece_id=6ACSWEXX2EOI6Y2JHZIDTJ22AKUTUSZZUR7MCY4WE5XFEOWKHMQA, seg_pos={0 0}, upload=58070]: Uploading piece... Jul 27 00:08:24.193189866 (60h32m33.915752906s) [noise=true , address=79.127.201.215:4356 , piece_id=DQHKW2AN3YEYLHOGW6JPUVLP4JWLCK5NFZ5AKIQRDQ3ZQZFOZYFQ, seg_pos={0 0}, upload=58070]: Uploading piece... Jul 27 00:08:24.193269258 (60h32m33.915832298s) [noise=true , address=45.91.116.10:4093 , piece_id=3AHXMLVONPDEVJLF7M7FKS4TWOU5HD3GJJIBHB5K5YQKC7O4A6HQ, seg_pos={0 0}, upload=58070]: Uploading piece... Jul 27 00:08:24.193446668 (60h32m33.916009708s) [noise=true , address=79.127.219.48:4087 , piece_id=RDDEOGNCM5DDUBZS7HPVJPKBK6GWKOHX7NCCS5GXROE3MXDUY2SA, seg_pos={0 0}, upload=58070]: Uploading piece... Jul 27 00:08:24.193675024 (60h32m33.916238064s) [noise=true , address=156.146.43.225:4434 , piece_id=M4DFDGN23AS7FY326EBM57YNL3Y552CWGHOEDVUYLB5L25IECD7Q, seg_pos={0 0}, upload=58070]: Uploading piece... Jul 27 00:08:25.135671862 (60h32m34.858234902s) [noise=true , address=79.127.220.101:4041 , piece_id=EMIQWCARLYTXACBD6O6DZH77RLYWX7EKA5PMFVNCJVQLCFW7WLFA, seg_pos={0 0}, upload=58070]: Done uploading piece. err: ecclient: upload failed (node:123G35C5bhcQpSSGRtKGo78ZfxNj6bZatY3kCrdEu1kMyjV5Yiw, address:79.127.220.101:4041): protocol: storage node overloaded, request limit: 2000 Jul 27 00:08:25.229683572 (60h32m34.952246612s) [noise=true , address=79.127.219.47:4048 , piece_id=LWIVS27EL4PHHBJ7W44KW3LNXSLK3VLXLGACZPIQZMIQVRLZ5HCA, seg_pos={0 0}, upload=58070]: Done uploading piece. err: ecclient: upload failed (node:12LWLin5hV2KsxHKkUPMx7p3PDcFGb15CJsvbaGdjSy6wwcgRZL, address:79.127.219.47:4048): protocol: storage node overloaded, request limit: 2000 Jul 27 00:08:25.308928970 (60h32m35.03149201s) [upload=58070]: Next returned nil segment Jul 27 00:08:25.308950642 (60h32m35.031513682s) [upload=58070]: Waiting for error group managing segments... Jul 27 00:08:25.596587471 (60h32m35.319150501s) [noise=true , address=79.127.205.235:4680 , piece_id=5SEJDTYH3PQAZSRZ3BNCILT4SPTDBN6F2I5CBM7XCF5HDFPYUU2A, seg_pos={0 0}, upload=58070]: Done uploading piece. err: <nil> Jul 27 00:08:25.596694865 (60h32m35.319257905s) [noise=true , address=79.127.205.245:4250 , piece_id=KVMWKUPRKG2AD5YD2DLDBASPNKQWX3DVK4HPGJRTQE4HZEF36SZA, seg_pos={0 0}, upload=58070]: Done uploading piece. err: <nil> Jul 27 00:08:26.179102912 (60h32m35.901665952s) [noise=true , address=79.127.219.38:4003 , piece_id=HGV4JIAWMIF52L6BMBE2JTBDSZXRKDPBRCIBFS72SDIKUL5X4QCQ, seg_pos={0 0}, upload=58070]: Done uploading piece. err: <nil> Jul 27 00:08:26.303026391 (60h32m36.025589431s) [noise=true , address=79.127.219.48:4087 , piece_id=RDDEOGNCM5DDUBZS7HPVJPKBK6GWKOHX7NCCS5GXROE3MXDUY2SA, seg_pos={0 0}, upload=58070]: Done uploading piece. err: <nil> Jul 27 00:08:29.930039220 (60h32m39.65260226s) [noise=true , address=45.91.116.10:4093 , piece_id=3AHXMLVONPDEVJLF7M7FKS4TWOU5HD3GJJIBHB5K5YQKC7O4A6HQ, seg_pos={0 0}, upload=58070]: Done uploading piece. err: <nil> Jul 27 00:08:29.939541440 (60h32m39.66210449s) [noise=true , address=77.90.31.12:4608 , piece_id=PNSNCD462QGF44F57B5ABKWNAFXNXRRCIJOUPLQWWYR5UIYC3LQA, seg_pos={0 0}, upload=58070]: Done uploading piece. err: <nil> Jul 27 00:08:30.410036577 (60h32m40.132599657s) [noise=true , address=156.146.43.225:4434 , piece_id=M4DFDGN23AS7FY326EBM57YNL3Y552CWGHOEDVUYLB5L25IECD7Q, seg_pos={0 0}, upload=58070]: Done uploading piece. err: <nil> Jul 27 00:08:35.944972414 (60h32m45.667535484s) [noise=true , address=45.91.116.9:4277 , piece_id=VI4FPX34RBSQBZVYJG6JEHWH3VDIZFD36TSEYKUWBC5MZTSCFD4A, seg_pos={0 0}, upload=58070]: Done uploading piece. err: <nil> Jul 27 00:08:59.456991076 (60h33m9.179554116s) [noise=true , address=79.127.201.215:4356 , piece_id=DQHKW2AN3YEYLHOGW6JPUVLP4JWLCK5NFZ5AKIQRDQ3ZQZFOZYFQ, seg_pos={0 0}, upload=58070]: Done uploading piece. err: <nil> Jul 27 00:09:23.094493363 (60h33m32.817056393s) [noise=true , address=104.36.82.54:4192 , piece_id=ERPUZFYA2ND4TU5NBQPX22UW426C4VSOW2Z7TJBNYIMTAJUVVP4Q, seg_pos={0 0}, upload=58070]: Done uploading piece. err: ecclient: upload failed (node:12peSG85BzWEjq3FZ5xMqLE2gStDvy2dViRdYmDJz7Lq8yBfaKY, address:104.36.82.54:4192): protocol: rpc: dial tcp 104.36.82.54:4192: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:09:23.094541304 (60h33m32.817104344s) [noise=true , address=104.36.82.50:4016 , piece_id=WDZYETH3W6LW2T6UWYTQMN6SFLGHIAJWEDRAD2P3AE4A2UMNCPOA, seg_pos={0 0}, upload=58070]: Done uploading piece. err: ecclient: upload failed (node:1jC98jEWn6K8jFHPA9knJeHAkuNEmcS7ThxUvbySDfbq6FmKVR, address:104.36.82.50:4016): protocol: write tcp 188.166.225.170:59734->104.36.82.50:4016: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:09:23.094859924 (60h33m32.817422994s) [noise=true , address=104.36.82.47:4532 , piece_id=BV2SJ3WCFJVTTO3A5QMITXSZVAMCQTKHJZOQQEQWVE5B4FUCHCJQ, seg_pos={0 0}, upload=58070]: Done uploading piece. err: ecclient: upload failed (node:15buPXDh9xXPzcL5Z7q2XZTQujajVRBp97VnPp5ngUDQL26AbL, address:104.36.82.47:4532): protocol: write tcp 188.166.225.170:41088->104.36.82.47:4532: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:09:23.094920129 (60h33m32.817483219s) [noise=true , address=104.36.82.46:4491 , piece_id=6ACSWEXX2EOI6Y2JHZIDTJ22AKUTUSZZUR7MCY4WE5XFEOWKHMQA, seg_pos={0 0}, upload=58070]: Done uploading piece. err: ecclient: upload failed (node:1CspvuKb3CWXrAW8FtFEoCVUNhLzWsbpwpW5uUfgyWy71R3qYX, address:104.36.82.46:4491): protocol: write tcp 188.166.225.170:51990->104.36.82.46:4491: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:09:23.095027074 (60h33m32.817590154s) [noise=true , address=104.36.82.55:4213 , piece_id=XAY5SRUIMYDFPKPWZTBSVHCNFSAVMOPKOIB7ZZ6AJEPN7WNB4AAA, seg_pos={0 0}, upload=58070]: Done uploading piece. err: ecclient: upload failed (node:129LRnDm7b4Fo5db7pto8GXYEtxHz9qdPAtQvaYH6gqdkMatQ6g, address:104.36.82.55:4213): protocol: rpc: dial tcp 104.36.82.55:4213: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:09:23.095099132 (60h33m32.817662172s) [noise=true , address=104.36.82.41:4783 , piece_id=FSSYD2VORTCAASQAK62RV4RFU5XFRJXNT4LTSRFQDCC5HJCNOUKQ, seg_pos={0 0}, upload=58070]: Done uploading piece. err: ecclient: upload failed (node:1sNMd8YrgaagKDoWivhbTmRc5TPM6BNiHscECBF2yhncKgkvz, address:104.36.82.41:4783): protocol: write tcp 188.166.225.170:54126->104.36.82.41:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:09:23.095202749 (60h33m32.817765819s) [noise=true , address=104.36.82.10:4094 , piece_id=HUUZUMK2MXON23DYDTI5SAVL2PDGQCZJBFOSBBSVY3I5IXCRBXAA, seg_pos={0 0}, upload=58070]: Done uploading piece. err: ecclient: upload failed (node:124zBCfZBTW69BSaGRzUyjWUBjSJhaLMZXQYnqMPGSKL4EX6HVV, address:104.36.82.10:4094): protocol: write tcp 188.166.225.170:48722->104.36.82.10:4094: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:09:23.095250731 (60h33m32.817813791s) [seg_pos={0 0}, upload=58070]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 27 00:09:23.327131253 (60h33m33.049694293s) [upload=58070]: Done uploading segments... Jul 27 00:09:23.327157522 (60h33m33.049720562s) [upload=58070]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.234997437s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [XX...XXX.X.....X.X..X..X.X....X.X....XXX.X...X......X.XX...X] duration: 30.000883423s FAIL (2025-07-27 00:09:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000905465s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 55 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXX.XXXXXXXXXX.XXXX..] duration: 7.289167177s FAIL (2025-07-27 00:09:01 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ol: rpc: dial tcp 104.36.82.7:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:06:26.796770452 (60h30m36.519333492s) [noise=true , address=104.36.82.11:4115 , piece_id=QSSAJ6XJOKGFA53IBRRLX2EQWTTG53AETOW777RKGE6HIEWXX7XA, seg_pos={0 0}, upload=58029]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: rpc: dial tcp 104.36.82.11:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:06:26.796848422 (60h30m36.519411462s) [seg_pos={0 0}, upload=58029]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 27 00:06:27.026095616 (60h30m36.748658666s) [upload=58029]: Done uploading segments... Jul 27 00:06:27.026124742 (60h30m36.748687782s) [upload=58029]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Jul 27 00:07:11.755040247 (60h31m21.477603287s) [upload=58048]: Starting upload Jul 27 00:07:11.755105291 (60h31m21.477668331s) [upload=58048]: Done starting upload Jul 27 00:07:11.755198160 (60h31m21.47776122s) [upload=58048]: Uploading segments... Jul 27 00:07:11.755291858 (60h31m21.477854969s) [upload=58048]: Got next segment. Inline: false Jul 27 00:07:11.755341433 (60h31m21.477904474s) [upload=58048]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:07:11.755345661 (60h31m21.477908701s) [upload=58048]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:07:11.990165825 (60h31m21.712728865s) [seg_pos={0 0}, upload=58048]: Begin upload segment... Jul 27 00:07:11.990211012 (60h31m21.712774052s) [seg_pos={0 0}, upload=58048]: Done begin upload segment. Jul 27 00:07:11.990221621 (60h31m21.712784652s) [noise=true , address=45.91.116.4:4939 , piece_id=X4ZR2OA2GHU5KTT35GJ7D5UFGXMZEB7MBAISVH4U3TS4O7WMBC7Q, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:11.990230568 (60h31m21.712793608s) [noise=true , address=77.90.31.15:4699 , piece_id=A5YPLNO2MB2GAH3C2NWSE6J6RW6QBWIPYB6E7TRPPXJXRPNUYIEQ, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:11.990633510 (60h31m21.71319655s) [noise=true , address=104.36.82.14:4221 , piece_id=Z2DNSMAKDDK3QZTOBLL3LQTD7FJ3NBWVC42YQZPS6JNKI5MERT2A, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:11.990967869 (60h31m21.713530899s) [noise=true , address=104.36.82.7:4910 , piece_id=ZWVWT3HWL3XXA4YCFG4FHT5KHWKHDD3ZAMN7GZVDKJRBLAH343YA, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:11.991721360 (60h31m21.7142844s) [noise=true , address=45.91.116.7:4011 , piece_id=ELDIMN6K3EQBIYGID3FUTBYCPDZVITKGJUJLLDRD45PM4GHNV5EA, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:11.992179327 (60h31m21.714742367s) [noise=true , address=77.90.31.16:4733 , piece_id=QYNXVUFGMKRQJYFKPKMX3KBMGYBZW4QNH34VNZWGKCGNM66ZLTHQ, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:11.992587718 (60h31m21.715150759s) [noise=true , address=45.91.116.9:4263 , piece_id=ZT7FKCQ2NGKF4IDJJPNAT3SOKUOHW3FD2WT2KJXTTYRS56LBVOFA, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:11.992994487 (60h31m21.715557527s) [noise=true , address=77.90.31.11:4553 , piece_id=FS4FQPMX4O65KSI4GHO5T64SNI54WWBOTQ2NTNIL43CBWBJLDHSA, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:11.993531444 (60h31m21.716094484s) [noise=true , address=45.91.116.4:4940 , piece_id=6DWR56CZLRWUA6HWY3KPBO7EC7KGCPJLMPI6LVLCOOQCRM73F5FQ, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:11.993948862 (60h31m21.716511902s) [noise=true , address=45.91.116.11:4118 , piece_id=T7UKLFOQFQL6GO2GRPBS6ZLIV6BYSLA3ELP2BDIZBC7VIK6FDSMQ, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:11.994373135 (60h31m21.716936175s) [noise=true , address=77.90.31.12:4592 , piece_id=SRDQKJK5GKGZOXTWVLDLHBIA2MHZL7YBAXTPYY6GIYWHTQ4W3VXA, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:11.995537432 (60h31m21.718100472s) [noise=true , address=77.90.31.14:4663 , piece_id=KCHQK7BDYYL62QC347PZDCA3EJSXB567TANZUTUJZWQAB6S6KMJQ, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:11.995812709 (60h31m21.718375749s) [noise=true , address=45.91.116.10:4081 , piece_id=T2LCDXM244PIE7GMDABAN574XMC2DH3N7P35YYOAZVB2CJG4EN5A, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:11.996213716 (60h31m21.718776756s) [noise=true , address=104.36.82.16:4293 , piece_id=5V2QJDJBFL7EEDYXGM3SHBJX7UZKQG2NN43X5VB5RFVX4QWKZK7Q, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:11.996482962 (60h31m21.719046002s) [noise=true , address=45.91.116.7:4010 , piece_id=TDOHBHE3PXZ7TMKDXC2EQVXJBU5AQ2Y5UEJOARRK6XU73VJ4ALOA, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:11.996721988 (60h31m21.719285028s) [noise=true , address=45.91.116.6:4973 , piece_id=4UAQHXEH55JQODDX4Y4SSBOAUJUJR3GNF3M3BX7Y5UX2TXEBHHLA, seg_pos={0 0}, upload=58048]: Uploading piece... Jul 27 00:07:13.734955056 (60h31m23.457518096s) [upload=58048]: Next returned nil segment Jul 27 00:07:13.734972579 (60h31m23.457535619s) [upload=58048]: Waiting for error group managing segments... Jul 27 00:07:14.013610340 (60h31m23.73617338s) [noise=true , address=77.90.31.16:4733 , piece_id=QYNXVUFGMKRQJYFKPKMX3KBMGYBZW4QNH34VNZWGKCGNM66ZLTHQ, seg_pos={0 0}, upload=58048]: Done uploading piece. err: <nil> Jul 27 00:07:14.018322977 (60h31m23.740886027s) [noise=true , address=45.91.116.11:4118 , piece_id=T7UKLFOQFQL6GO2GRPBS6ZLIV6BYSLA3ELP2BDIZBC7VIK6FDSMQ, seg_pos={0 0}, upload=58048]: Done uploading piece. err: <nil> Jul 27 00:07:14.018799488 (60h31m23.741362528s) [noise=true , address=45.91.116.7:4010 , piece_id=TDOHBHE3PXZ7TMKDXC2EQVXJBU5AQ2Y5UEJOARRK6XU73VJ4ALOA, seg_pos={0 0}, upload=58048]: Done uploading piece. err: <nil> Jul 27 00:07:14.089681915 (60h31m23.812244955s) [noise=true , address=45.91.116.4:4939 , piece_id=X4ZR2OA2GHU5KTT35GJ7D5UFGXMZEB7MBAISVH4U3TS4O7WMBC7Q, seg_pos={0 0}, upload=58048]: Done uploading piece. err: <nil> Jul 27 00:07:14.098404755 (60h31m23.820967795s) [noise=true , address=45.91.116.4:4940 , piece_id=6DWR56CZLRWUA6HWY3KPBO7EC7KGCPJLMPI6LVLCOOQCRM73F5FQ, seg_pos={0 0}, upload=58048]: Done uploading piece. err: <nil> Jul 27 00:07:14.106109907 (60h31m23.828672947s) [noise=true , address=45.91.116.6:4973 , piece_id=4UAQHXEH55JQODDX4Y4SSBOAUJUJR3GNF3M3BX7Y5UX2TXEBHHLA, seg_pos={0 0}, upload=58048]: Done uploading piece. err: <nil> Jul 27 00:07:14.112275553 (60h31m23.834838583s) [noise=true , address=45.91.116.7:4011 , piece_id=ELDIMN6K3EQBIYGID3FUTBYCPDZVITKGJUJLLDRD45PM4GHNV5EA, seg_pos={0 0}, upload=58048]: Done uploading piece. err: <nil> Jul 27 00:07:14.113721300 (60h31m23.83628433s) [noise=true , address=77.90.31.11:4553 , piece_id=FS4FQPMX4O65KSI4GHO5T64SNI54WWBOTQ2NTNIL43CBWBJLDHSA, seg_pos={0 0}, upload=58048]: Done uploading piece. err: <nil> Jul 27 00:07:14.117463299 (60h31m23.840026339s) [noise=true , address=77.90.31.14:4663 , piece_id=KCHQK7BDYYL62QC347PZDCA3EJSXB567TANZUTUJZWQAB6S6KMJQ, seg_pos={0 0}, upload=58048]: Done uploading piece. err: <nil> Jul 27 00:07:14.178263677 (60h31m23.900826717s) [noise=true , address=45.91.116.10:4081 , piece_id=T2LCDXM244PIE7GMDABAN574XMC2DH3N7P35YYOAZVB2CJG4EN5A, seg_pos={0 0}, upload=58048]: Done uploading piece. err: <nil> Jul 27 00:07:14.464873981 (60h31m24.187437021s) [noise=true , address=45.91.116.9:4263 , piece_id=ZT7FKCQ2NGKF4IDJJPNAT3SOKUOHW3FD2WT2KJXTTYRS56LBVOFA, seg_pos={0 0}, upload=58048]: Done uploading piece. err: <nil> Jul 27 00:07:15.435585766 (60h31m25.158148796s) [noise=true , address=77.90.31.12:4592 , piece_id=SRDQKJK5GKGZOXTWVLDLHBIA2MHZL7YBAXTPYY6GIYWHTQ4W3VXA, seg_pos={0 0}, upload=58048]: Done uploading piece. err: <nil> Jul 27 00:07:41.107251487 (60h31m50.829814527s) [noise=true , address=104.36.82.14:4221 , piece_id=Z2DNSMAKDDK3QZTOBLL3LQTD7FJ3NBWVC42YQZPS6JNKI5MERT2A, seg_pos={0 0}, upload=58048]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: rpc: dial tcp 104.36.82.14:4221: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:07:41.107464594 (60h31m50.830027664s) [noise=true , address=104.36.82.7:4910 , piece_id=ZWVWT3HWL3XXA4YCFG4FHT5KHWKHDD3ZAMN7GZVDKJRBLAH343YA, seg_pos={0 0}, upload=58048]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: write tcp 188.166.225.170:57142->104.36.82.7:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:07:41.107538756 (60h31m50.830101827s) [noise=true , address=77.90.31.15:4699 , piece_id=A5YPLNO2MB2GAH3C2NWSE6J6RW6QBWIPYB6E7TRPPXJXRPNUYIEQ, seg_pos={0 0}, upload=58048]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:38974->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:07:41.107575747 (60h31m50.830138787s) [noise=true , address=104.36.82.16:4293 , piece_id=5V2QJDJBFL7EEDYXGM3SHBJX7UZKQG2NN43X5VB5RFVX4QWKZK7Q, seg_pos={0 0}, upload=58048]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:07:41.107609272 (60h31m50.830172302s) [seg_pos={0 0}, upload=58048]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:07:41.340882517 (60h31m51.063445547s) [upload=58048]: Done uploading segments... Jul 27 00:07:41.340925319 (60h31m51.063488369s) [upload=58048]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 7.289196683s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [..XXXX.....XXX..X.....X......X...XX..XX.X.XX...X............] duration: 47.107254625s FAIL (2025-07-27 00:08:52 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 47.107266408s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 1.872802594s FAIL (2025-07-27 00:08:49 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.872831339s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 55 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..XX.X.XXX.XXXXX] duration: 30.239811509s FAIL (2025-07-27 00:08:39 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.239842867s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 1.817189051s FAIL (2025-07-27 00:08:27 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.81722028s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.244427209s FAIL (2025-07-27 00:08:22 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== ddress=64.49.13.139:4478 , piece_id=6UJX4IIE6CYNLDMRUJ6BHLD65WT3UXZFVNRPD6MEVIUHPR6GXSAA, seg_pos={0 0}, upload=58052]: Uploading piece... Jul 27 00:07:23.727061052 (60h31m33.449624122s) [noise=true , address=79.127.220.97:4361 , piece_id=TYJP6IPDMMHMERFCFJNL4NXJZZCG2UYNS2KB3JPGXWXEPINXJ2PA, seg_pos={0 0}, upload=58052]: Uploading piece... Jul 27 00:07:23.727196460 (60h31m33.44975955s) [noise=true , address=77.90.31.15:4726 , piece_id=7KKKQMUN7BQHWDMCJTPGNKDKPFGUR33HY4CBCHH343ICI467ROXA, seg_pos={0 0}, upload=58052]: Uploading piece... Jul 27 00:07:23.727682550 (60h31m33.45024561s) [noise=true , address=104.36.82.54:4191 , piece_id=A7MNSY6RGTLUT6V46SLEEAQOCPXDRY5QFXBCQOAZ2KHUBQY73SWA, seg_pos={0 0}, upload=58052]: Uploading piece... Jul 27 00:07:23.728099728 (60h31m33.450662758s) [noise=true , address=104.36.82.46:4493 , piece_id=KTQPHH3XGIMXINOQRHSVA7ZY2VBOHTFI6LLHGUDBO5X5JBKVJE6A, seg_pos={0 0}, upload=58052]: Uploading piece... Jul 27 00:07:23.728106522 (60h31m33.450669562s) [noise=true , address=79.127.205.235:4687 , piece_id=3J6XO5R7RAKTZZVUSTCQ5S545NDCFCLXO3PBFSHXFWSZANLNDLAA, seg_pos={0 0}, upload=58052]: Uploading piece... Jul 27 00:07:23.728548447 (60h31m33.451111487s) [noise=true , address=104.36.82.55:4204 , piece_id=VYOPSSZNCYIO2NE5GOSKVWOEDXSNCMXDLIEHMHSJYXF6KQHAE6QA, seg_pos={0 0}, upload=58052]: Uploading piece... Jul 27 00:07:23.729766277 (60h31m33.452329317s) [noise=true , address=79.127.219.38:4003 , piece_id=N5R7ILGBGWODMYVA72UKNSBLHO2ZCZOHPVGIKSSRILX2TDNAK6QA, seg_pos={0 0}, upload=58052]: Uploading piece... Jul 27 00:07:23.730872413 (60h31m33.453435453s) [noise=true , address=104.36.82.16:4300 , piece_id=SW3L2SBYEEUMIEPT6BNUUBGYLBGZW4CB3KZUWVER2MNHZ3GH6HEA, seg_pos={0 0}, upload=58052]: Uploading piece... Jul 27 00:07:23.731313136 (60h31m33.453876177s) [noise=true , address=77.90.31.16:4737 , piece_id=H3GQJUAD3HGZLR4UFSVWNQDWWYFQGABYWJYWQPZULNFETEN2V6MA, seg_pos={0 0}, upload=58052]: Uploading piece... Jul 27 00:07:23.731703463 (60h31m33.454266503s) [noise=true , address=104.36.82.53:4142 , piece_id=WMZCU4FYC4KXVD5TO7PQ5S45NSSJL4IN45UNRBZTUZ5N6MBA35WA, seg_pos={0 0}, upload=58052]: Uploading piece... Jul 27 00:07:23.732105703 (60h31m33.454668743s) [noise=true , address=79.127.205.234:4610 , piece_id=GBFRWZZDAX2SUH55K7SHR4MBTV5ADHFH26JOR2F7D5R5VWVFKVTA, seg_pos={0 0}, upload=58052]: Uploading piece... Jul 27 00:07:23.732370520 (60h31m33.45493357s) [noise=true , address=79.127.219.47:4048 , piece_id=DS26KSX25RBT3DEEKXDFNXCROHU2AINQKMMS5TN6DJMTNVVUA56Q, seg_pos={0 0}, upload=58052]: Uploading piece... Jul 27 00:07:23.732647630 (60h31m33.45521067s) [noise=true , address=79.127.205.246:4270 , piece_id=AY4DVMJGAVSKY5C2ZP4WAAQZPRA2KNHP6HBSTJHY6OFL3MT5NCNA, seg_pos={0 0}, upload=58052]: Uploading piece... Jul 27 00:07:23.732921374 (60h31m33.455484414s) [noise=true , address=45.91.116.10:4093 , piece_id=USC6TAFNA35EBKBQLDEJRSIYUESRZWSEI6SMICMWISZY7U7R7RHA, seg_pos={0 0}, upload=58052]: Uploading piece... Jul 27 00:07:23.733277195 (60h31m33.455840225s) [noise=true , address=79.127.201.215:4355 , piece_id=RV2YPT7IJ5TRCXJMXCWBZTHS43ME42WUFRYUF3LLD6FNKI2BEYIA, seg_pos={0 0}, upload=58052]: Uploading piece... Jul 27 00:07:24.820008849 (60h31m34.542571889s) [upload=58052]: Next returned nil segment Jul 27 00:07:24.820025310 (60h31m34.542588351s) [upload=58052]: Waiting for error group managing segments... Jul 27 00:07:24.887653151 (60h31m34.610216191s) [noise=true , address=79.127.219.47:4048 , piece_id=DS26KSX25RBT3DEEKXDFNXCROHU2AINQKMMS5TN6DJMTNVVUA56Q, seg_pos={0 0}, upload=58052]: Done uploading piece. err: ecclient: upload failed (node:12LWLin5hV2KsxHKkUPMx7p3PDcFGb15CJsvbaGdjSy6wwcgRZL, address:79.127.219.47:4048): protocol: storage node overloaded, request limit: 2000 Jul 27 00:07:25.109410853 (60h31m34.831973893s) [noise=true , address=79.127.205.234:4610 , piece_id=GBFRWZZDAX2SUH55K7SHR4MBTV5ADHFH26JOR2F7D5R5VWVFKVTA, seg_pos={0 0}, upload=58052]: Done uploading piece. err: <nil> Jul 27 00:07:25.138673642 (60h31m34.861236682s) [noise=true , address=79.127.205.235:4687 , piece_id=3J6XO5R7RAKTZZVUSTCQ5S545NDCFCLXO3PBFSHXFWSZANLNDLAA, seg_pos={0 0}, upload=58052]: Done uploading piece. err: <nil> Jul 27 00:07:25.153927788 (60h31m34.876490828s) [noise=true , address=79.127.205.246:4270 , piece_id=AY4DVMJGAVSKY5C2ZP4WAAQZPRA2KNHP6HBSTJHY6OFL3MT5NCNA, seg_pos={0 0}, upload=58052]: Done uploading piece. err: <nil> Jul 27 00:07:25.642669774 (60h31m35.365232814s) [noise=true , address=79.127.220.97:4361 , piece_id=TYJP6IPDMMHMERFCFJNL4NXJZZCG2UYNS2KB3JPGXWXEPINXJ2PA, seg_pos={0 0}, upload=58052]: Done uploading piece. err: <nil> Jul 27 00:07:25.719944903 (60h31m35.442507943s) [noise=true , address=79.127.219.38:4003 , piece_id=N5R7ILGBGWODMYVA72UKNSBLHO2ZCZOHPVGIKSSRILX2TDNAK6QA, seg_pos={0 0}, upload=58052]: Done uploading piece. err: <nil> Jul 27 00:07:25.757257171 (60h31m35.479820201s) [noise=true , address=77.90.31.16:4737 , piece_id=H3GQJUAD3HGZLR4UFSVWNQDWWYFQGABYWJYWQPZULNFETEN2V6MA, seg_pos={0 0}, upload=58052]: Done uploading piece. err: <nil> Jul 27 00:07:35.745696491 (60h31m45.468259532s) [noise=true , address=64.49.13.139:4478 , piece_id=6UJX4IIE6CYNLDMRUJ6BHLD65WT3UXZFVNRPD6MEVIUHPR6GXSAA, seg_pos={0 0}, upload=58052]: Done uploading piece. err: <nil> Jul 27 00:07:40.589949786 (60h31m50.312512837s) [noise=true , address=45.91.116.10:4093 , piece_id=USC6TAFNA35EBKBQLDEJRSIYUESRZWSEI6SMICMWISZY7U7R7RHA, seg_pos={0 0}, upload=58052]: Done uploading piece. err: <nil> Jul 27 00:07:47.462943792 (60h31m57.185506833s) [noise=true , address=79.127.201.215:4355 , piece_id=RV2YPT7IJ5TRCXJMXCWBZTHS43ME42WUFRYUF3LLD6FNKI2BEYIA, seg_pos={0 0}, upload=58052]: Done uploading piece. err: <nil> Jul 27 00:08:22.624850013 (60h32m32.347413073s) [noise=true , address=77.90.31.15:4726 , piece_id=7KKKQMUN7BQHWDMCJTPGNKDKPFGUR33HY4CBCHH343ICI467ROXA, seg_pos={0 0}, upload=58052]: Done uploading piece. err: ecclient: upload failed (node:12kMaukCkCJSc3ukAZu3thLgzXGGzNq44dGyEjCPqoY3PSYVEm3, address:77.90.31.15:4726): protocol: write tcp 188.166.225.170:49330->77.90.31.15:4726: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:08:22.624893377 (60h32m32.347456417s) [noise=true , address=104.36.82.55:4204 , piece_id=VYOPSSZNCYIO2NE5GOSKVWOEDXSNCMXDLIEHMHSJYXF6KQHAE6QA, seg_pos={0 0}, upload=58052]: Done uploading piece. err: ecclient: upload failed (node:1rBZH6aLHsnWh22uNzaMiMwmLJCDMn4bYbNC5jRG7KCEuRfqpH, address:104.36.82.55:4204): protocol: write tcp 188.166.225.170:44342->104.36.82.55:4204: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:08:22.624933333 (60h32m32.347496373s) [noise=true , address=104.36.82.16:4300 , piece_id=SW3L2SBYEEUMIEPT6BNUUBGYLBGZW4CB3KZUWVER2MNHZ3GH6HEA, seg_pos={0 0}, upload=58052]: Done uploading piece. err: ecclient: upload failed (node:12orVauQ7c1EEcy4Kh4ydLcaYo1y8hLGxvsMhd8WDdyojX1Rzvg, address:104.36.82.16:4300): protocol: write tcp 188.166.225.170:38544->104.36.82.16:4300: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:08:22.624963520 (60h32m32.34752656s) [noise=true , address=104.36.82.54:4191 , piece_id=A7MNSY6RGTLUT6V46SLEEAQOCPXDRY5QFXBCQOAZ2KHUBQY73SWA, seg_pos={0 0}, upload=58052]: Done uploading piece. err: ecclient: upload failed (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw, address:104.36.82.54:4191): protocol: write tcp 188.166.225.170:53522->104.36.82.54:4191: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:08:22.624996474 (60h32m32.347559554s) [noise=true , address=104.36.82.46:4493 , piece_id=KTQPHH3XGIMXINOQRHSVA7ZY2VBOHTFI6LLHGUDBO5X5JBKVJE6A, seg_pos={0 0}, upload=58052]: Done uploading piece. err: ecclient: upload failed (node:127drPYZWm7RRXuCLtuzDgSaxxuu3k1tg6eo3KWtrQGjBaUbfRJ, address:104.36.82.46:4493): protocol: write tcp 188.166.225.170:59700->104.36.82.46:4493: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:08:22.625029076 (60h32m32.347592277s) [noise=true , address=104.36.82.47:4526 , piece_id=6DBPGXO5PJCFQ7PKXOZK2IM6AELQQ7IC2MKVDWPPASOYSKX7PMPQ, seg_pos={0 0}, upload=58052]: Done uploading piece. err: ecclient: upload failed (node:1djnDo1bR9DZLEAHGJGoaxg895sQ3tQpYmPgGXpmsAxnpAtQwk, address:104.36.82.47:4526): protocol: write tcp 188.166.225.170:48354->104.36.82.47:4526: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:08:22.625062891 (60h32m32.347626101s) [noise=true , address=104.36.82.53:4142 , piece_id=WMZCU4FYC4KXVD5TO7PQ5S45NSSJL4IN45UNRBZTUZ5N6MBA35WA, seg_pos={0 0}, upload=58052]: Done uploading piece. err: ecclient: upload failed (node:1yLPWJs1RfZgFisHUpW3wwiFsvV98yz7adKtAKAK3unq1KZCSK, address:104.36.82.53:4142): protocol: write tcp 188.166.225.170:54566->104.36.82.53:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:08:22.625098859 (60h32m32.347661899s) [noise=true , address=104.36.82.2:4736 , piece_id=ICMVAPJZ3HNI6AGWXV6NYKA5Y7W7IFOT2IFVGVWSNEQTNGWTDIGQ, seg_pos={0 0}, upload=58052]: Done uploading piece. err: ecclient: upload failed (node:1ejnWsitdrh68vSJvjR2QnRg4kva8DFRzqvvUuDkXM6zKUDWNC, address:104.36.82.2:4736): protocol: write tcp 188.166.225.170:57946->104.36.82.2:4736: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:08:22.625117215 (60h32m32.347680255s) [seg_pos={0 0}, upload=58052]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 27 00:08:22.867247115 (60h32m32.589810155s) [upload=58052]: Done uploading segments... Jul 27 00:08:22.867270961 (60h32m32.589834001s) [upload=58052]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.24443828s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [...XXXX.....XXX..X.....X......X...XX..XX.X.XX...X...........] duration: 50.659346226s FAIL (2025-07-27 00:08:05 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 50.659380051s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [XXX...XXX.X.....X.X..X..X.X....X.X....XXX.X...X......X.XX...] duration: 7.359180024s FAIL (2025-07-27 00:08:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.359199812s edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)