Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [.X...X................X....X...............X..X.X...X...X...] duration: 21.135699781s FAIL (2025-07-09 21:45:28 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 17258]: Got next segment. Inline: false Jul 9 21:42:48.387406701 (396h32m23.158048036s) [upload=317258]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 21:42:48.387413103 (396h32m23.158054439s) [upload=317258]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 21:42:48.834299395 (396h32m23.60494072s) [seg_pos={0 0}, upload=317258]: Begin upload segment... Jul 9 21:42:48.834354481 (396h32m23.604995816s) [seg_pos={0 0}, upload=317258]: Done begin upload segment. Jul 9 21:42:48.834374599 (396h32m23.605015924s) [noise=true , address=77.90.31.10:4523 , piece_id=SKESIP2VBIDJIBXIEGMC2LZESW47O7WJCLSBQBSWLHYWV4PRT7FQ, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.834761038 (396h32m23.605402363s) [noise=true , address=77.90.31.21:4908 , piece_id=HGKISLD6DZXJ6DL4UCZWUJD2V5CTDYRX2BSKDCDJYXCQQBX6Q3RA, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.835061884 (396h32m23.605703219s) [noise=true , address=79.127.201.215:4345 , piece_id=WEQTKXLED3S6W4HMRFM7YUEXBWGUUMQL65E4U4FG27AVCZFRK72A, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.835344354 (396h32m23.605985689s) [noise=true , address=104.36.82.23:4474 , piece_id=JJNF2423XB5JQXQJ3VWRUYGLZ2QR5SFPN7BUJO4GRPSDST6IFBCA, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.835641913 (396h32m23.606283248s) [noise=true , address=45.91.95.2:4022 , piece_id=UJ6PKLMNBID76O3ZR4SXRNLL3CCLNGTYBZSN57E2YBD57TDLKTPQ, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.835925006 (396h32m23.606566341s) [noise=true , address=77.90.31.14:4667 , piece_id=3XNOC4TIZJFEMIJQIMYHYNXA3SD72IYKBVBVWZ5J7NKVG5XOQYMA, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.836153993 (396h32m23.606795328s) [noise=true , address=104.36.82.38:4716 , piece_id=FK7WLNN6677LINQJ26LPGZ5X4JJ7ISGDRUHMVQTGK34OTL44EF6A, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.836372001 (396h32m23.607013336s) [noise=true , address=77.90.31.18:4810 , piece_id=33CXTR5YAVMPV3VGG23FMKLGAKR355O66PAHFZPKVQSFJGQIEYLQ, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.836627199 (396h32m23.607268524s) [noise=true , address=77.90.31.16:4768 , piece_id=LOU4TXWNZRUAIB7BVVZXKEVZSBL44DKOIITBGOCPDPM73L7CYRYQ, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.836860434 (396h32m23.607501759s) [noise=true , address=77.90.31.13:4640 , piece_id=XCMFSHLM372AAMNYL2YVGRUSGWT7RB4RE4TMZ6ONHPVNR26NKUAQ, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.837076287 (396h32m23.607717623s) [noise=true , address=45.91.95.5:4026 , piece_id=QQWGIQENU5TGALRAU4SRUFWJDQYOPW552SN46COWCRFNFBRRPBFA, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.837281300 (396h32m23.607922635s) [noise=true , address=79.127.220.101:4073 , piece_id=NNWGURWIPMP67UIQGVHZPFZJBD2QFOGBQ6KZNNTHLJXCGRIDTXZQ, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.837494658 (396h32m23.608135993s) [noise=true , address=104.36.82.18:4396 , piece_id=2DRRKEK2OLFL54G6BQIIQLEPONIHDUICJUVAEAONC2IOR22POPLA, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.837859957 (396h32m23.608501292s) [noise=true , address=45.91.95.4:4014 , piece_id=V5ZNHUMTQCZ5YXOYDAYCFKBBSAUKHJ2RJ3FSQNME6IXSGFSLIJ4Q, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.838145814 (396h32m23.608787139s) [noise=true , address=77.90.31.18:4838 , piece_id=EGYCA5GBMRI3JPX3SHKQ57GU6ZQV6S33BJNOJVUJHESMEKZU5WTQ, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.838416161 (396h32m23.609057486s) [noise=true , address=45.91.95.3:4019 , piece_id=3XZWJLZH5RXX45ULYQLXAG4SG2RNMOGWXLNFTOVCVKZ2QUQOC6KQ, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.838665709 (396h32m23.609307044s) [noise=true , address=77.90.31.19:4852 , piece_id=MQ7OIWIHD642LXDOEBKZF2DTZRLR5OJT4Q37DCLLDXR7XTJVKXKA, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.838944833 (396h32m23.609586168s) [noise=true , address=77.90.31.12:4621 , piece_id=CL234SZAWOTG4ONGWQ2ZBXEGWUYTQQVR76BVCHAPXETRGF5TQJBQ, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:50.183574564 (396h32m24.954215899s) [upload=317258]: Next returned nil segment Jul 9 21:42:50.183596435 (396h32m24.95423777s) [upload=317258]: Waiting for error group managing segments... Jul 9 21:42:50.548527303 (396h32m25.319168638s) [noise=true , address=79.127.220.101:4073 , piece_id=NNWGURWIPMP67UIQGVHZPFZJBD2QFOGBQ6KZNNTHLJXCGRIDTXZQ, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:50.840602026 (396h32m25.611243361s) [noise=true , address=77.90.31.18:4838 , piece_id=EGYCA5GBMRI3JPX3SHKQ57GU6ZQV6S33BJNOJVUJHESMEKZU5WTQ, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:50.841968832 (396h32m25.612610167s) [noise=true , address=77.90.31.18:4810 , piece_id=33CXTR5YAVMPV3VGG23FMKLGAKR355O66PAHFZPKVQSFJGQIEYLQ, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:50.842708556 (396h32m25.613349951s) [noise=true , address=77.90.31.13:4640 , piece_id=XCMFSHLM372AAMNYL2YVGRUSGWT7RB4RE4TMZ6ONHPVNR26NKUAQ, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:50.843190929 (396h32m25.613832265s) [noise=true , address=77.90.31.14:4667 , piece_id=3XNOC4TIZJFEMIJQIMYHYNXA3SD72IYKBVBVWZ5J7NKVG5XOQYMA, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:50.844590978 (396h32m25.615232313s) [noise=true , address=77.90.31.10:4523 , piece_id=SKESIP2VBIDJIBXIEGMC2LZESW47O7WJCLSBQBSWLHYWV4PRT7FQ, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:50.845022593 (396h32m25.615663928s) [noise=true , address=77.90.31.19:4852 , piece_id=MQ7OIWIHD642LXDOEBKZF2DTZRLR5OJT4Q37DCLLDXR7XTJVKXKA, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:50.846516732 (396h32m25.617158067s) [noise=true , address=77.90.31.12:4621 , piece_id=CL234SZAWOTG4ONGWQ2ZBXEGWUYTQQVR76BVCHAPXETRGF5TQJBQ, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:50.847220148 (396h32m25.617861473s) [noise=true , address=77.90.31.21:4908 , piece_id=HGKISLD6DZXJ6DL4UCZWUJD2V5CTDYRX2BSKDCDJYXCQQBX6Q3RA, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:50.847310801 (396h32m25.617952126s) [noise=true , address=77.90.31.16:4768 , piece_id=LOU4TXWNZRUAIB7BVVZXKEVZSBL44DKOIITBGOCPDPM73L7CYRYQ, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:55.315575878 (396h32m30.086217213s) [noise=true , address=79.127.201.215:4345 , piece_id=WEQTKXLED3S6W4HMRFM7YUEXBWGUUMQL65E4U4FG27AVCZFRK72A, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:43:12.086886240 (396h32m46.857527576s) [noise=true , address=104.36.82.23:4474 , piece_id=JJNF2423XB5JQXQJ3VWRUYGLZ2QR5SFPN7BUJO4GRPSDST6IFBCA, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:43:29.357614308 (396h33m4.128255643s) [noise=true , address=104.36.82.18:4396 , piece_id=2DRRKEK2OLFL54G6BQIIQLEPONIHDUICJUVAEAONC2IOR22POPLA, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:43:38.089997559 (396h33m12.860638884s) [noise=true , address=45.91.95.4:4014 , piece_id=V5ZNHUMTQCZ5YXOYDAYCFKBBSAUKHJ2RJ3FSQNME6IXSGFSLIJ4Q, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:43:38.090021875 (396h33m12.86066321s) [seg_pos={0 0}, upload=317258]: Segment reached optimal threshold of 14 pieces. Jul 9 21:43:38.090108922 (396h33m12.860750257s) [noise=true , address=45.91.95.5:4026 , piece_id=QQWGIQENU5TGALRAU4SRUFWJDQYOPW552SN46COWCRFNFBRRPBFA, seg_pos={0 0}, upload=317258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xxtZ3msZnUkE4kgTFKJmprcNS986Uhi1LjwVyd1DbrTfZS52H): protocol: expected piece hash; context canceled; EOF Jul 9 21:43:38.090133739 (396h33m12.860775074s) [noise=true , address=104.36.82.38:4716 , piece_id=FK7WLNN6677LINQJ26LPGZ5X4JJ7ISGDRUHMVQTGK34OTL44EF6A, seg_pos={0 0}, upload=317258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MAHcnhvjvD74dpLS5S3m751NzfjvJ8a9ZvHwA7mdXUEXm9DAX): protocol: expected piece hash; context canceled; EOF Jul 9 21:43:38.090161603 (396h33m12.860802938s) [noise=true , address=45.91.95.3:4019 , piece_id=3XZWJLZH5RXX45ULYQLXAG4SG2RNMOGWXLNFTOVCVKZ2QUQOC6KQ, seg_pos={0 0}, upload=317258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SGjoGjbfugbxFgDctHygi7qo8upC463TRw8sEceUNnti1iGV1): protocol: expected piece hash; context canceled; EOF Jul 9 21:43:38.090225525 (396h33m12.86086685s) [noise=true , address=45.91.95.2:4022 , piece_id=UJ6PKLMNBID76O3ZR4SXRNLL3CCLNGTYBZSN57E2YBD57TDLKTPQ, seg_pos={0 0}, upload=317258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XGBvsxi6S7dz9d93CWmnQhomJVg1x5evu7DxsGsujkejBkZVN): protocol: write tcp 188.166.225.170:50926->45.91.95.2:4022: use of closed network connection; write tcp 188.166.225.170:50926->45.91.95.2:4022: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50926->45.91.95.2:4022: use of closed network connection Jul 9 21:43:38.090246715 (396h33m12.860888051s) [seg_pos={0 0}, upload=317258]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 21:43:38.090334674 (396h33m12.860976009s) [upload=317258]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 21:43:38.090338631 (396h33m12.860979956s) [upload=317258]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 21:43:38.379746682 (396h33m13.150388017s) [upload=317258]: Done uploading segments... Jul 9 21:43:38.379765919 (396h33m13.150407254s) [upload=317258]: Upload finished. err: <nil> Duration: 21.135715813s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [..X...X................X....X...............X..X.X...X...X..] duration: 40.453250874s FAIL (2025-07-09 21:45:07 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 17258]: Got next segment. Inline: false Jul 9 21:42:48.387406701 (396h32m23.158048036s) [upload=317258]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 21:42:48.387413103 (396h32m23.158054439s) [upload=317258]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 21:42:48.834299395 (396h32m23.60494072s) [seg_pos={0 0}, upload=317258]: Begin upload segment... Jul 9 21:42:48.834354481 (396h32m23.604995816s) [seg_pos={0 0}, upload=317258]: Done begin upload segment. Jul 9 21:42:48.834374599 (396h32m23.605015924s) [noise=true , address=77.90.31.10:4523 , piece_id=SKESIP2VBIDJIBXIEGMC2LZESW47O7WJCLSBQBSWLHYWV4PRT7FQ, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.834761038 (396h32m23.605402363s) [noise=true , address=77.90.31.21:4908 , piece_id=HGKISLD6DZXJ6DL4UCZWUJD2V5CTDYRX2BSKDCDJYXCQQBX6Q3RA, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.835061884 (396h32m23.605703219s) [noise=true , address=79.127.201.215:4345 , piece_id=WEQTKXLED3S6W4HMRFM7YUEXBWGUUMQL65E4U4FG27AVCZFRK72A, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.835344354 (396h32m23.605985689s) [noise=true , address=104.36.82.23:4474 , piece_id=JJNF2423XB5JQXQJ3VWRUYGLZ2QR5SFPN7BUJO4GRPSDST6IFBCA, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.835641913 (396h32m23.606283248s) [noise=true , address=45.91.95.2:4022 , piece_id=UJ6PKLMNBID76O3ZR4SXRNLL3CCLNGTYBZSN57E2YBD57TDLKTPQ, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.835925006 (396h32m23.606566341s) [noise=true , address=77.90.31.14:4667 , piece_id=3XNOC4TIZJFEMIJQIMYHYNXA3SD72IYKBVBVWZ5J7NKVG5XOQYMA, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.836153993 (396h32m23.606795328s) [noise=true , address=104.36.82.38:4716 , piece_id=FK7WLNN6677LINQJ26LPGZ5X4JJ7ISGDRUHMVQTGK34OTL44EF6A, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.836372001 (396h32m23.607013336s) [noise=true , address=77.90.31.18:4810 , piece_id=33CXTR5YAVMPV3VGG23FMKLGAKR355O66PAHFZPKVQSFJGQIEYLQ, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.836627199 (396h32m23.607268524s) [noise=true , address=77.90.31.16:4768 , piece_id=LOU4TXWNZRUAIB7BVVZXKEVZSBL44DKOIITBGOCPDPM73L7CYRYQ, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.836860434 (396h32m23.607501759s) [noise=true , address=77.90.31.13:4640 , piece_id=XCMFSHLM372AAMNYL2YVGRUSGWT7RB4RE4TMZ6ONHPVNR26NKUAQ, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.837076287 (396h32m23.607717623s) [noise=true , address=45.91.95.5:4026 , piece_id=QQWGIQENU5TGALRAU4SRUFWJDQYOPW552SN46COWCRFNFBRRPBFA, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.837281300 (396h32m23.607922635s) [noise=true , address=79.127.220.101:4073 , piece_id=NNWGURWIPMP67UIQGVHZPFZJBD2QFOGBQ6KZNNTHLJXCGRIDTXZQ, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.837494658 (396h32m23.608135993s) [noise=true , address=104.36.82.18:4396 , piece_id=2DRRKEK2OLFL54G6BQIIQLEPONIHDUICJUVAEAONC2IOR22POPLA, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.837859957 (396h32m23.608501292s) [noise=true , address=45.91.95.4:4014 , piece_id=V5ZNHUMTQCZ5YXOYDAYCFKBBSAUKHJ2RJ3FSQNME6IXSGFSLIJ4Q, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.838145814 (396h32m23.608787139s) [noise=true , address=77.90.31.18:4838 , piece_id=EGYCA5GBMRI3JPX3SHKQ57GU6ZQV6S33BJNOJVUJHESMEKZU5WTQ, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.838416161 (396h32m23.609057486s) [noise=true , address=45.91.95.3:4019 , piece_id=3XZWJLZH5RXX45ULYQLXAG4SG2RNMOGWXLNFTOVCVKZ2QUQOC6KQ, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.838665709 (396h32m23.609307044s) [noise=true , address=77.90.31.19:4852 , piece_id=MQ7OIWIHD642LXDOEBKZF2DTZRLR5OJT4Q37DCLLDXR7XTJVKXKA, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.838944833 (396h32m23.609586168s) [noise=true , address=77.90.31.12:4621 , piece_id=CL234SZAWOTG4ONGWQ2ZBXEGWUYTQQVR76BVCHAPXETRGF5TQJBQ, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:50.183574564 (396h32m24.954215899s) [upload=317258]: Next returned nil segment Jul 9 21:42:50.183596435 (396h32m24.95423777s) [upload=317258]: Waiting for error group managing segments... Jul 9 21:42:50.548527303 (396h32m25.319168638s) [noise=true , address=79.127.220.101:4073 , piece_id=NNWGURWIPMP67UIQGVHZPFZJBD2QFOGBQ6KZNNTHLJXCGRIDTXZQ, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:50.840602026 (396h32m25.611243361s) [noise=true , address=77.90.31.18:4838 , piece_id=EGYCA5GBMRI3JPX3SHKQ57GU6ZQV6S33BJNOJVUJHESMEKZU5WTQ, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:50.841968832 (396h32m25.612610167s) [noise=true , address=77.90.31.18:4810 , piece_id=33CXTR5YAVMPV3VGG23FMKLGAKR355O66PAHFZPKVQSFJGQIEYLQ, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:50.842708556 (396h32m25.613349951s) [noise=true , address=77.90.31.13:4640 , piece_id=XCMFSHLM372AAMNYL2YVGRUSGWT7RB4RE4TMZ6ONHPVNR26NKUAQ, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:50.843190929 (396h32m25.613832265s) [noise=true , address=77.90.31.14:4667 , piece_id=3XNOC4TIZJFEMIJQIMYHYNXA3SD72IYKBVBVWZ5J7NKVG5XOQYMA, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:50.844590978 (396h32m25.615232313s) [noise=true , address=77.90.31.10:4523 , piece_id=SKESIP2VBIDJIBXIEGMC2LZESW47O7WJCLSBQBSWLHYWV4PRT7FQ, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:50.845022593 (396h32m25.615663928s) [noise=true , address=77.90.31.19:4852 , piece_id=MQ7OIWIHD642LXDOEBKZF2DTZRLR5OJT4Q37DCLLDXR7XTJVKXKA, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:50.846516732 (396h32m25.617158067s) [noise=true , address=77.90.31.12:4621 , piece_id=CL234SZAWOTG4ONGWQ2ZBXEGWUYTQQVR76BVCHAPXETRGF5TQJBQ, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:50.847220148 (396h32m25.617861473s) [noise=true , address=77.90.31.21:4908 , piece_id=HGKISLD6DZXJ6DL4UCZWUJD2V5CTDYRX2BSKDCDJYXCQQBX6Q3RA, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:50.847310801 (396h32m25.617952126s) [noise=true , address=77.90.31.16:4768 , piece_id=LOU4TXWNZRUAIB7BVVZXKEVZSBL44DKOIITBGOCPDPM73L7CYRYQ, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:55.315575878 (396h32m30.086217213s) [noise=true , address=79.127.201.215:4345 , piece_id=WEQTKXLED3S6W4HMRFM7YUEXBWGUUMQL65E4U4FG27AVCZFRK72A, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:43:12.086886240 (396h32m46.857527576s) [noise=true , address=104.36.82.23:4474 , piece_id=JJNF2423XB5JQXQJ3VWRUYGLZ2QR5SFPN7BUJO4GRPSDST6IFBCA, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:43:29.357614308 (396h33m4.128255643s) [noise=true , address=104.36.82.18:4396 , piece_id=2DRRKEK2OLFL54G6BQIIQLEPONIHDUICJUVAEAONC2IOR22POPLA, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:43:38.089997559 (396h33m12.860638884s) [noise=true , address=45.91.95.4:4014 , piece_id=V5ZNHUMTQCZ5YXOYDAYCFKBBSAUKHJ2RJ3FSQNME6IXSGFSLIJ4Q, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:43:38.090021875 (396h33m12.86066321s) [seg_pos={0 0}, upload=317258]: Segment reached optimal threshold of 14 pieces. Jul 9 21:43:38.090108922 (396h33m12.860750257s) [noise=true , address=45.91.95.5:4026 , piece_id=QQWGIQENU5TGALRAU4SRUFWJDQYOPW552SN46COWCRFNFBRRPBFA, seg_pos={0 0}, upload=317258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xxtZ3msZnUkE4kgTFKJmprcNS986Uhi1LjwVyd1DbrTfZS52H): protocol: expected piece hash; context canceled; EOF Jul 9 21:43:38.090133739 (396h33m12.860775074s) [noise=true , address=104.36.82.38:4716 , piece_id=FK7WLNN6677LINQJ26LPGZ5X4JJ7ISGDRUHMVQTGK34OTL44EF6A, seg_pos={0 0}, upload=317258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MAHcnhvjvD74dpLS5S3m751NzfjvJ8a9ZvHwA7mdXUEXm9DAX): protocol: expected piece hash; context canceled; EOF Jul 9 21:43:38.090161603 (396h33m12.860802938s) [noise=true , address=45.91.95.3:4019 , piece_id=3XZWJLZH5RXX45ULYQLXAG4SG2RNMOGWXLNFTOVCVKZ2QUQOC6KQ, seg_pos={0 0}, upload=317258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SGjoGjbfugbxFgDctHygi7qo8upC463TRw8sEceUNnti1iGV1): protocol: expected piece hash; context canceled; EOF Jul 9 21:43:38.090225525 (396h33m12.86086685s) [noise=true , address=45.91.95.2:4022 , piece_id=UJ6PKLMNBID76O3ZR4SXRNLL3CCLNGTYBZSN57E2YBD57TDLKTPQ, seg_pos={0 0}, upload=317258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XGBvsxi6S7dz9d93CWmnQhomJVg1x5evu7DxsGsujkejBkZVN): protocol: write tcp 188.166.225.170:50926->45.91.95.2:4022: use of closed network connection; write tcp 188.166.225.170:50926->45.91.95.2:4022: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50926->45.91.95.2:4022: use of closed network connection Jul 9 21:43:38.090246715 (396h33m12.860888051s) [seg_pos={0 0}, upload=317258]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 21:43:38.090334674 (396h33m12.860976009s) [upload=317258]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 21:43:38.090338631 (396h33m12.860979956s) [upload=317258]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 21:43:38.379746682 (396h33m13.150388017s) [upload=317258]: Done uploading segments... Jul 9 21:43:38.379765919 (396h33m13.150407254s) [upload=317258]: Upload finished. err: <nil> Duration: 40.45326996s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [.....X......X....X..X.........X...............X..X....X.....] duration: 53.010221732s FAIL (2025-07-09 21:45:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 53.010232592s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [...X...X................X....X...............X..X.X...X...X.] duration: 38.662139123s FAIL (2025-07-09 21:44:26 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 17258]: Got next segment. Inline: false Jul 9 21:42:48.387406701 (396h32m23.158048036s) [upload=317258]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 21:42:48.387413103 (396h32m23.158054439s) [upload=317258]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 21:42:48.834299395 (396h32m23.60494072s) [seg_pos={0 0}, upload=317258]: Begin upload segment... Jul 9 21:42:48.834354481 (396h32m23.604995816s) [seg_pos={0 0}, upload=317258]: Done begin upload segment. Jul 9 21:42:48.834374599 (396h32m23.605015924s) [noise=true , address=77.90.31.10:4523 , piece_id=SKESIP2VBIDJIBXIEGMC2LZESW47O7WJCLSBQBSWLHYWV4PRT7FQ, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.834761038 (396h32m23.605402363s) [noise=true , address=77.90.31.21:4908 , piece_id=HGKISLD6DZXJ6DL4UCZWUJD2V5CTDYRX2BSKDCDJYXCQQBX6Q3RA, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.835061884 (396h32m23.605703219s) [noise=true , address=79.127.201.215:4345 , piece_id=WEQTKXLED3S6W4HMRFM7YUEXBWGUUMQL65E4U4FG27AVCZFRK72A, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.835344354 (396h32m23.605985689s) [noise=true , address=104.36.82.23:4474 , piece_id=JJNF2423XB5JQXQJ3VWRUYGLZ2QR5SFPN7BUJO4GRPSDST6IFBCA, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.835641913 (396h32m23.606283248s) [noise=true , address=45.91.95.2:4022 , piece_id=UJ6PKLMNBID76O3ZR4SXRNLL3CCLNGTYBZSN57E2YBD57TDLKTPQ, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.835925006 (396h32m23.606566341s) [noise=true , address=77.90.31.14:4667 , piece_id=3XNOC4TIZJFEMIJQIMYHYNXA3SD72IYKBVBVWZ5J7NKVG5XOQYMA, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.836153993 (396h32m23.606795328s) [noise=true , address=104.36.82.38:4716 , piece_id=FK7WLNN6677LINQJ26LPGZ5X4JJ7ISGDRUHMVQTGK34OTL44EF6A, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.836372001 (396h32m23.607013336s) [noise=true , address=77.90.31.18:4810 , piece_id=33CXTR5YAVMPV3VGG23FMKLGAKR355O66PAHFZPKVQSFJGQIEYLQ, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.836627199 (396h32m23.607268524s) [noise=true , address=77.90.31.16:4768 , piece_id=LOU4TXWNZRUAIB7BVVZXKEVZSBL44DKOIITBGOCPDPM73L7CYRYQ, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.836860434 (396h32m23.607501759s) [noise=true , address=77.90.31.13:4640 , piece_id=XCMFSHLM372AAMNYL2YVGRUSGWT7RB4RE4TMZ6ONHPVNR26NKUAQ, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.837076287 (396h32m23.607717623s) [noise=true , address=45.91.95.5:4026 , piece_id=QQWGIQENU5TGALRAU4SRUFWJDQYOPW552SN46COWCRFNFBRRPBFA, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.837281300 (396h32m23.607922635s) [noise=true , address=79.127.220.101:4073 , piece_id=NNWGURWIPMP67UIQGVHZPFZJBD2QFOGBQ6KZNNTHLJXCGRIDTXZQ, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.837494658 (396h32m23.608135993s) [noise=true , address=104.36.82.18:4396 , piece_id=2DRRKEK2OLFL54G6BQIIQLEPONIHDUICJUVAEAONC2IOR22POPLA, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.837859957 (396h32m23.608501292s) [noise=true , address=45.91.95.4:4014 , piece_id=V5ZNHUMTQCZ5YXOYDAYCFKBBSAUKHJ2RJ3FSQNME6IXSGFSLIJ4Q, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.838145814 (396h32m23.608787139s) [noise=true , address=77.90.31.18:4838 , piece_id=EGYCA5GBMRI3JPX3SHKQ57GU6ZQV6S33BJNOJVUJHESMEKZU5WTQ, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.838416161 (396h32m23.609057486s) [noise=true , address=45.91.95.3:4019 , piece_id=3XZWJLZH5RXX45ULYQLXAG4SG2RNMOGWXLNFTOVCVKZ2QUQOC6KQ, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.838665709 (396h32m23.609307044s) [noise=true , address=77.90.31.19:4852 , piece_id=MQ7OIWIHD642LXDOEBKZF2DTZRLR5OJT4Q37DCLLDXR7XTJVKXKA, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.838944833 (396h32m23.609586168s) [noise=true , address=77.90.31.12:4621 , piece_id=CL234SZAWOTG4ONGWQ2ZBXEGWUYTQQVR76BVCHAPXETRGF5TQJBQ, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:50.183574564 (396h32m24.954215899s) [upload=317258]: Next returned nil segment Jul 9 21:42:50.183596435 (396h32m24.95423777s) [upload=317258]: Waiting for error group managing segments... Jul 9 21:42:50.548527303 (396h32m25.319168638s) [noise=true , address=79.127.220.101:4073 , piece_id=NNWGURWIPMP67UIQGVHZPFZJBD2QFOGBQ6KZNNTHLJXCGRIDTXZQ, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:50.840602026 (396h32m25.611243361s) [noise=true , address=77.90.31.18:4838 , piece_id=EGYCA5GBMRI3JPX3SHKQ57GU6ZQV6S33BJNOJVUJHESMEKZU5WTQ, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:50.841968832 (396h32m25.612610167s) [noise=true , address=77.90.31.18:4810 , piece_id=33CXTR5YAVMPV3VGG23FMKLGAKR355O66PAHFZPKVQSFJGQIEYLQ, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:50.842708556 (396h32m25.613349951s) [noise=true , address=77.90.31.13:4640 , piece_id=XCMFSHLM372AAMNYL2YVGRUSGWT7RB4RE4TMZ6ONHPVNR26NKUAQ, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:50.843190929 (396h32m25.613832265s) [noise=true , address=77.90.31.14:4667 , piece_id=3XNOC4TIZJFEMIJQIMYHYNXA3SD72IYKBVBVWZ5J7NKVG5XOQYMA, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:50.844590978 (396h32m25.615232313s) [noise=true , address=77.90.31.10:4523 , piece_id=SKESIP2VBIDJIBXIEGMC2LZESW47O7WJCLSBQBSWLHYWV4PRT7FQ, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:50.845022593 (396h32m25.615663928s) [noise=true , address=77.90.31.19:4852 , piece_id=MQ7OIWIHD642LXDOEBKZF2DTZRLR5OJT4Q37DCLLDXR7XTJVKXKA, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:50.846516732 (396h32m25.617158067s) [noise=true , address=77.90.31.12:4621 , piece_id=CL234SZAWOTG4ONGWQ2ZBXEGWUYTQQVR76BVCHAPXETRGF5TQJBQ, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:50.847220148 (396h32m25.617861473s) [noise=true , address=77.90.31.21:4908 , piece_id=HGKISLD6DZXJ6DL4UCZWUJD2V5CTDYRX2BSKDCDJYXCQQBX6Q3RA, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:50.847310801 (396h32m25.617952126s) [noise=true , address=77.90.31.16:4768 , piece_id=LOU4TXWNZRUAIB7BVVZXKEVZSBL44DKOIITBGOCPDPM73L7CYRYQ, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:55.315575878 (396h32m30.086217213s) [noise=true , address=79.127.201.215:4345 , piece_id=WEQTKXLED3S6W4HMRFM7YUEXBWGUUMQL65E4U4FG27AVCZFRK72A, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:43:12.086886240 (396h32m46.857527576s) [noise=true , address=104.36.82.23:4474 , piece_id=JJNF2423XB5JQXQJ3VWRUYGLZ2QR5SFPN7BUJO4GRPSDST6IFBCA, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:43:29.357614308 (396h33m4.128255643s) [noise=true , address=104.36.82.18:4396 , piece_id=2DRRKEK2OLFL54G6BQIIQLEPONIHDUICJUVAEAONC2IOR22POPLA, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:43:38.089997559 (396h33m12.860638884s) [noise=true , address=45.91.95.4:4014 , piece_id=V5ZNHUMTQCZ5YXOYDAYCFKBBSAUKHJ2RJ3FSQNME6IXSGFSLIJ4Q, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:43:38.090021875 (396h33m12.86066321s) [seg_pos={0 0}, upload=317258]: Segment reached optimal threshold of 14 pieces. Jul 9 21:43:38.090108922 (396h33m12.860750257s) [noise=true , address=45.91.95.5:4026 , piece_id=QQWGIQENU5TGALRAU4SRUFWJDQYOPW552SN46COWCRFNFBRRPBFA, seg_pos={0 0}, upload=317258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xxtZ3msZnUkE4kgTFKJmprcNS986Uhi1LjwVyd1DbrTfZS52H): protocol: expected piece hash; context canceled; EOF Jul 9 21:43:38.090133739 (396h33m12.860775074s) [noise=true , address=104.36.82.38:4716 , piece_id=FK7WLNN6677LINQJ26LPGZ5X4JJ7ISGDRUHMVQTGK34OTL44EF6A, seg_pos={0 0}, upload=317258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MAHcnhvjvD74dpLS5S3m751NzfjvJ8a9ZvHwA7mdXUEXm9DAX): protocol: expected piece hash; context canceled; EOF Jul 9 21:43:38.090161603 (396h33m12.860802938s) [noise=true , address=45.91.95.3:4019 , piece_id=3XZWJLZH5RXX45ULYQLXAG4SG2RNMOGWXLNFTOVCVKZ2QUQOC6KQ, seg_pos={0 0}, upload=317258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SGjoGjbfugbxFgDctHygi7qo8upC463TRw8sEceUNnti1iGV1): protocol: expected piece hash; context canceled; EOF Jul 9 21:43:38.090225525 (396h33m12.86086685s) [noise=true , address=45.91.95.2:4022 , piece_id=UJ6PKLMNBID76O3ZR4SXRNLL3CCLNGTYBZSN57E2YBD57TDLKTPQ, seg_pos={0 0}, upload=317258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XGBvsxi6S7dz9d93CWmnQhomJVg1x5evu7DxsGsujkejBkZVN): protocol: write tcp 188.166.225.170:50926->45.91.95.2:4022: use of closed network connection; write tcp 188.166.225.170:50926->45.91.95.2:4022: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50926->45.91.95.2:4022: use of closed network connection Jul 9 21:43:38.090246715 (396h33m12.860888051s) [seg_pos={0 0}, upload=317258]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 21:43:38.090334674 (396h33m12.860976009s) [upload=317258]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 21:43:38.090338631 (396h33m12.860979956s) [upload=317258]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 21:43:38.379746682 (396h33m13.150388017s) [upload=317258]: Done uploading segments... Jul 9 21:43:38.379765919 (396h33m13.150407254s) [upload=317258]: Upload finished. err: <nil> Duration: 38.662156465s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [......X......X....X..X.........X...............X..X....X....] duration: 54.550651669s FAIL (2025-07-09 21:44:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 54.550663162s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [....X...X................X....X...............X..X.X...X...X] duration: 1m0.002472839s FAIL (2025-07-09 21:43:47 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== 17258]: Got next segment. Inline: false Jul 9 21:42:48.387406701 (396h32m23.158048036s) [upload=317258]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 21:42:48.387413103 (396h32m23.158054439s) [upload=317258]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 21:42:48.834299395 (396h32m23.60494072s) [seg_pos={0 0}, upload=317258]: Begin upload segment... Jul 9 21:42:48.834354481 (396h32m23.604995816s) [seg_pos={0 0}, upload=317258]: Done begin upload segment. Jul 9 21:42:48.834374599 (396h32m23.605015924s) [noise=true , address=77.90.31.10:4523 , piece_id=SKESIP2VBIDJIBXIEGMC2LZESW47O7WJCLSBQBSWLHYWV4PRT7FQ, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.834761038 (396h32m23.605402363s) [noise=true , address=77.90.31.21:4908 , piece_id=HGKISLD6DZXJ6DL4UCZWUJD2V5CTDYRX2BSKDCDJYXCQQBX6Q3RA, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.835061884 (396h32m23.605703219s) [noise=true , address=79.127.201.215:4345 , piece_id=WEQTKXLED3S6W4HMRFM7YUEXBWGUUMQL65E4U4FG27AVCZFRK72A, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.835344354 (396h32m23.605985689s) [noise=true , address=104.36.82.23:4474 , piece_id=JJNF2423XB5JQXQJ3VWRUYGLZ2QR5SFPN7BUJO4GRPSDST6IFBCA, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.835641913 (396h32m23.606283248s) [noise=true , address=45.91.95.2:4022 , piece_id=UJ6PKLMNBID76O3ZR4SXRNLL3CCLNGTYBZSN57E2YBD57TDLKTPQ, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.835925006 (396h32m23.606566341s) [noise=true , address=77.90.31.14:4667 , piece_id=3XNOC4TIZJFEMIJQIMYHYNXA3SD72IYKBVBVWZ5J7NKVG5XOQYMA, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.836153993 (396h32m23.606795328s) [noise=true , address=104.36.82.38:4716 , piece_id=FK7WLNN6677LINQJ26LPGZ5X4JJ7ISGDRUHMVQTGK34OTL44EF6A, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.836372001 (396h32m23.607013336s) [noise=true , address=77.90.31.18:4810 , piece_id=33CXTR5YAVMPV3VGG23FMKLGAKR355O66PAHFZPKVQSFJGQIEYLQ, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.836627199 (396h32m23.607268524s) [noise=true , address=77.90.31.16:4768 , piece_id=LOU4TXWNZRUAIB7BVVZXKEVZSBL44DKOIITBGOCPDPM73L7CYRYQ, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.836860434 (396h32m23.607501759s) [noise=true , address=77.90.31.13:4640 , piece_id=XCMFSHLM372AAMNYL2YVGRUSGWT7RB4RE4TMZ6ONHPVNR26NKUAQ, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.837076287 (396h32m23.607717623s) [noise=true , address=45.91.95.5:4026 , piece_id=QQWGIQENU5TGALRAU4SRUFWJDQYOPW552SN46COWCRFNFBRRPBFA, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.837281300 (396h32m23.607922635s) [noise=true , address=79.127.220.101:4073 , piece_id=NNWGURWIPMP67UIQGVHZPFZJBD2QFOGBQ6KZNNTHLJXCGRIDTXZQ, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.837494658 (396h32m23.608135993s) [noise=true , address=104.36.82.18:4396 , piece_id=2DRRKEK2OLFL54G6BQIIQLEPONIHDUICJUVAEAONC2IOR22POPLA, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.837859957 (396h32m23.608501292s) [noise=true , address=45.91.95.4:4014 , piece_id=V5ZNHUMTQCZ5YXOYDAYCFKBBSAUKHJ2RJ3FSQNME6IXSGFSLIJ4Q, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.838145814 (396h32m23.608787139s) [noise=true , address=77.90.31.18:4838 , piece_id=EGYCA5GBMRI3JPX3SHKQ57GU6ZQV6S33BJNOJVUJHESMEKZU5WTQ, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.838416161 (396h32m23.609057486s) [noise=true , address=45.91.95.3:4019 , piece_id=3XZWJLZH5RXX45ULYQLXAG4SG2RNMOGWXLNFTOVCVKZ2QUQOC6KQ, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.838665709 (396h32m23.609307044s) [noise=true , address=77.90.31.19:4852 , piece_id=MQ7OIWIHD642LXDOEBKZF2DTZRLR5OJT4Q37DCLLDXR7XTJVKXKA, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:48.838944833 (396h32m23.609586168s) [noise=true , address=77.90.31.12:4621 , piece_id=CL234SZAWOTG4ONGWQ2ZBXEGWUYTQQVR76BVCHAPXETRGF5TQJBQ, seg_pos={0 0}, upload=317258]: Uploading piece... Jul 9 21:42:50.183574564 (396h32m24.954215899s) [upload=317258]: Next returned nil segment Jul 9 21:42:50.183596435 (396h32m24.95423777s) [upload=317258]: Waiting for error group managing segments... Jul 9 21:42:50.548527303 (396h32m25.319168638s) [noise=true , address=79.127.220.101:4073 , piece_id=NNWGURWIPMP67UIQGVHZPFZJBD2QFOGBQ6KZNNTHLJXCGRIDTXZQ, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:50.840602026 (396h32m25.611243361s) [noise=true , address=77.90.31.18:4838 , piece_id=EGYCA5GBMRI3JPX3SHKQ57GU6ZQV6S33BJNOJVUJHESMEKZU5WTQ, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:50.841968832 (396h32m25.612610167s) [noise=true , address=77.90.31.18:4810 , piece_id=33CXTR5YAVMPV3VGG23FMKLGAKR355O66PAHFZPKVQSFJGQIEYLQ, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:50.842708556 (396h32m25.613349951s) [noise=true , address=77.90.31.13:4640 , piece_id=XCMFSHLM372AAMNYL2YVGRUSGWT7RB4RE4TMZ6ONHPVNR26NKUAQ, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:50.843190929 (396h32m25.613832265s) [noise=true , address=77.90.31.14:4667 , piece_id=3XNOC4TIZJFEMIJQIMYHYNXA3SD72IYKBVBVWZ5J7NKVG5XOQYMA, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:50.844590978 (396h32m25.615232313s) [noise=true , address=77.90.31.10:4523 , piece_id=SKESIP2VBIDJIBXIEGMC2LZESW47O7WJCLSBQBSWLHYWV4PRT7FQ, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:50.845022593 (396h32m25.615663928s) [noise=true , address=77.90.31.19:4852 , piece_id=MQ7OIWIHD642LXDOEBKZF2DTZRLR5OJT4Q37DCLLDXR7XTJVKXKA, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:50.846516732 (396h32m25.617158067s) [noise=true , address=77.90.31.12:4621 , piece_id=CL234SZAWOTG4ONGWQ2ZBXEGWUYTQQVR76BVCHAPXETRGF5TQJBQ, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:50.847220148 (396h32m25.617861473s) [noise=true , address=77.90.31.21:4908 , piece_id=HGKISLD6DZXJ6DL4UCZWUJD2V5CTDYRX2BSKDCDJYXCQQBX6Q3RA, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:50.847310801 (396h32m25.617952126s) [noise=true , address=77.90.31.16:4768 , piece_id=LOU4TXWNZRUAIB7BVVZXKEVZSBL44DKOIITBGOCPDPM73L7CYRYQ, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:42:55.315575878 (396h32m30.086217213s) [noise=true , address=79.127.201.215:4345 , piece_id=WEQTKXLED3S6W4HMRFM7YUEXBWGUUMQL65E4U4FG27AVCZFRK72A, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:43:12.086886240 (396h32m46.857527576s) [noise=true , address=104.36.82.23:4474 , piece_id=JJNF2423XB5JQXQJ3VWRUYGLZ2QR5SFPN7BUJO4GRPSDST6IFBCA, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:43:29.357614308 (396h33m4.128255643s) [noise=true , address=104.36.82.18:4396 , piece_id=2DRRKEK2OLFL54G6BQIIQLEPONIHDUICJUVAEAONC2IOR22POPLA, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:43:38.089997559 (396h33m12.860638884s) [noise=true , address=45.91.95.4:4014 , piece_id=V5ZNHUMTQCZ5YXOYDAYCFKBBSAUKHJ2RJ3FSQNME6IXSGFSLIJ4Q, seg_pos={0 0}, upload=317258]: Done uploading piece. err: <nil> Jul 9 21:43:38.090021875 (396h33m12.86066321s) [seg_pos={0 0}, upload=317258]: Segment reached optimal threshold of 14 pieces. Jul 9 21:43:38.090108922 (396h33m12.860750257s) [noise=true , address=45.91.95.5:4026 , piece_id=QQWGIQENU5TGALRAU4SRUFWJDQYOPW552SN46COWCRFNFBRRPBFA, seg_pos={0 0}, upload=317258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xxtZ3msZnUkE4kgTFKJmprcNS986Uhi1LjwVyd1DbrTfZS52H): protocol: expected piece hash; context canceled; EOF Jul 9 21:43:38.090133739 (396h33m12.860775074s) [noise=true , address=104.36.82.38:4716 , piece_id=FK7WLNN6677LINQJ26LPGZ5X4JJ7ISGDRUHMVQTGK34OTL44EF6A, seg_pos={0 0}, upload=317258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MAHcnhvjvD74dpLS5S3m751NzfjvJ8a9ZvHwA7mdXUEXm9DAX): protocol: expected piece hash; context canceled; EOF Jul 9 21:43:38.090161603 (396h33m12.860802938s) [noise=true , address=45.91.95.3:4019 , piece_id=3XZWJLZH5RXX45ULYQLXAG4SG2RNMOGWXLNFTOVCVKZ2QUQOC6KQ, seg_pos={0 0}, upload=317258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SGjoGjbfugbxFgDctHygi7qo8upC463TRw8sEceUNnti1iGV1): protocol: expected piece hash; context canceled; EOF Jul 9 21:43:38.090225525 (396h33m12.86086685s) [noise=true , address=45.91.95.2:4022 , piece_id=UJ6PKLMNBID76O3ZR4SXRNLL3CCLNGTYBZSN57E2YBD57TDLKTPQ, seg_pos={0 0}, upload=317258]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XGBvsxi6S7dz9d93CWmnQhomJVg1x5evu7DxsGsujkejBkZVN): protocol: write tcp 188.166.225.170:50926->45.91.95.2:4022: use of closed network connection; write tcp 188.166.225.170:50926->45.91.95.2:4022: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50926->45.91.95.2:4022: use of closed network connection Jul 9 21:43:38.090246715 (396h33m12.860888051s) [seg_pos={0 0}, upload=317258]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 21:43:38.090334674 (396h33m12.860976009s) [upload=317258]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 21:43:38.090338631 (396h33m12.860979956s) [upload=317258]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 21:43:38.379746682 (396h33m13.150388017s) [upload=317258]: Done uploading segments... Jul 9 21:43:38.379765919 (396h33m13.150407254s) [upload=317258]: Upload finished. err: <nil> Duration: 1m0.002492859s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [.......X......X....X..X.........X...............X..X....X...] duration: 54.018072407s FAIL (2025-07-09 21:43:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 54.018098147s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [.....X...X................X....X...............X..X.X...X...] duration: 31.217200502s FAIL (2025-07-09 21:42:47 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 727635s) [upload=317212]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 21:39:32.743915214 (396h29m7.514556539s) [seg_pos={0 0}, upload=317212]: Begin upload segment... Jul 9 21:39:32.743955091 (396h29m7.514596416s) [seg_pos={0 0}, upload=317212]: Done begin upload segment. Jul 9 21:39:32.743976182 (396h29m7.514617517s) [noise=true , address=77.90.31.10:4535 , piece_id=KJER6ZFIVPUXCK6WEI5TX5UX3LU3BAO25MYTQW3A7TIHXHLK4ECA, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.744312375 (396h29m7.51495371s) [noise=true , address=104.36.82.19:4425 , piece_id=GMMOB7UIYDEB37LG5WCBGSGIH6IT652DHXBTH66MLWHBFF3IQTBA, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.744344205 (396h29m7.51498554s) [noise=true , address=77.90.31.12:4616 , piece_id=WYOLXV2SNY3PF3VWQS76KBGNRTOPKDR2DDSMZGMAQCVWKM3JRG7A, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.744776322 (396h29m7.515417667s) [noise=true , address=104.36.82.21:4473 , piece_id=65CNRWVKLCWLW6IVVIMV2O3U3ONRWJWTPZ5WA6AB7W7CSKHJJGHQ, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.744879529 (396h29m7.515520864s) [noise=true , address=77.90.31.18:4831 , piece_id=4CSJQKEAD6XPOQMIIOG6SCTBEBNUPCCED3DL6R367YFKHTKXFAVA, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.745527850 (396h29m7.516169185s) [noise=true , address=64.49.13.137:4502 , piece_id=T4BAEHJ2GEW7YVSNSXTCFGX2ANUGGUYMAU7A2AIILIUBHS2AQMJQ, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.745755746 (396h29m7.516397081s) [noise=true , address=45.91.95.3:4004 , piece_id=V3MVFLFQMXW3KO4PSEL3BQUEOWL4OR5HPPWRONMEVFPHOORC5CYQ, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.746225264 (396h29m7.516866599s) [noise=true , address=45.91.95.2:4012 , piece_id=ZNJPEWCNHTPOFCWXHCCTTOCFNDJSZE2NRXJK2SWE5DX2YSSIT2UQ, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.746255782 (396h29m7.516897117s) [noise=true , address=77.90.31.17:4787 , piece_id=SIKBA3Z4FKNVGMLHVE3ODZP2IJU42LJDA26UQQZRGF45GYOICLSA, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.746836593 (396h29m7.517477928s) [noise=true , address=45.91.95.4:4036 , piece_id=GSSBZKTAVOFGTBOD2SQ3C5Y6P3KYWZD7UJFJ4L2ZA2EYZ7DQAZKA, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.747161365 (396h29m7.5178027s) [noise=true , address=77.90.31.16:4767 , piece_id=B3XFM5FUZQHU3CJ5OM3DJBACYYCZNIVON74C4UUSSK5K6GE24JVA, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.747581919 (396h29m7.518223254s) [noise=true , address=104.36.82.23:4452 , piece_id=G4XFNFBBTJBOXSPS7YOTCSGFC7NY3VH5WUQ5BRFWWJ3IWXGPT3SA, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.748234087 (396h29m7.518875412s) [noise=true , address=77.90.31.21:4881 , piece_id=DQRBEHZFEHIMGIKCQIJQQXR6LVG67YOF24ESXW3JLWHBVLWPHEHA, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.748773640 (396h29m7.519414995s) [noise=true , address=77.90.31.11:4575 , piece_id=3EPIKI2CXDS4JWHP5YOT4HWZBBZBTWSJZ2BCYULSZTQPFULAVKBA, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.749532161 (396h29m7.520173496s) [noise=true , address=64.49.13.169:4706 , piece_id=4CU34X6K2PKGF6YFOLFVGF62G4RPZ7BYBOVMVPMLTPYVZ2OAJ4LQ, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.750049140 (396h29m7.520690475s) [noise=true , address=79.127.219.38:4006 , piece_id=6S2FX7PM4HMCULTOQ4WW6TDN52W43CRPCZC5A3DYFI5ON3QJ3WIA, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.750308366 (396h29m7.520949701s) [noise=true , address=79.127.201.215:4352 , piece_id=K4L367RZGAISJISAJZRPNAPGRSNAO2QL6YRLVCTRQGDXMT23DKBA, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.764751689 (396h29m7.535393024s) [noise=true , address=77.90.31.14:4686 , piece_id=4ARR5GY74KXEVSKHBQRAS2WADAACDNCGO2LPNSWO5LRUZQ7GEQIA, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:33.880679709 (396h29m8.651321044s) [upload=317212]: Next returned nil segment Jul 9 21:39:33.880697924 (396h29m8.65133926s) [upload=317212]: Waiting for error group managing segments... Jul 9 21:39:34.264937553 (396h29m9.035578888s) [noise=true , address=64.49.13.137:4502 , piece_id=T4BAEHJ2GEW7YVSNSXTCFGX2ANUGGUYMAU7A2AIILIUBHS2AQMJQ, seg_pos={0 0}, upload=317212]: Done uploading piece. err: <nil> Jul 9 21:39:34.740039134 (396h29m9.510680469s) [noise=true , address=77.90.31.18:4831 , piece_id=4CSJQKEAD6XPOQMIIOG6SCTBEBNUPCCED3DL6R367YFKHTKXFAVA, seg_pos={0 0}, upload=317212]: Done uploading piece. err: <nil> Jul 9 21:39:34.742667242 (396h29m9.513308567s) [noise=true , address=77.90.31.16:4767 , piece_id=B3XFM5FUZQHU3CJ5OM3DJBACYYCZNIVON74C4UUSSK5K6GE24JVA, seg_pos={0 0}, upload=317212]: Done uploading piece. err: <nil> Jul 9 21:39:34.745658134 (396h29m9.516299469s) [noise=true , address=77.90.31.12:4616 , piece_id=WYOLXV2SNY3PF3VWQS76KBGNRTOPKDR2DDSMZGMAQCVWKM3JRG7A, seg_pos={0 0}, upload=317212]: Done uploading piece. err: <nil> Jul 9 21:39:34.746157450 (396h29m9.516798785s) [noise=true , address=79.127.219.38:4006 , piece_id=6S2FX7PM4HMCULTOQ4WW6TDN52W43CRPCZC5A3DYFI5ON3QJ3WIA, seg_pos={0 0}, upload=317212]: Done uploading piece. err: <nil> Jul 9 21:39:34.751901208 (396h29m9.522542533s) [noise=true , address=77.90.31.11:4575 , piece_id=3EPIKI2CXDS4JWHP5YOT4HWZBBZBTWSJZ2BCYULSZTQPFULAVKBA, seg_pos={0 0}, upload=317212]: Done uploading piece. err: <nil> Jul 9 21:39:34.757250002 (396h29m9.527891337s) [noise=true , address=77.90.31.10:4535 , piece_id=KJER6ZFIVPUXCK6WEI5TX5UX3LU3BAO25MYTQW3A7TIHXHLK4ECA, seg_pos={0 0}, upload=317212]: Done uploading piece. err: <nil> Jul 9 21:39:34.757587377 (396h29m9.528228712s) [noise=true , address=77.90.31.17:4787 , piece_id=SIKBA3Z4FKNVGMLHVE3ODZP2IJU42LJDA26UQQZRGF45GYOICLSA, seg_pos={0 0}, upload=317212]: Done uploading piece. err: <nil> Jul 9 21:39:34.761661611 (396h29m9.532302946s) [noise=true , address=77.90.31.21:4881 , piece_id=DQRBEHZFEHIMGIKCQIJQQXR6LVG67YOF24ESXW3JLWHBVLWPHEHA, seg_pos={0 0}, upload=317212]: Done uploading piece. err: <nil> Jul 9 21:39:34.765694217 (396h29m9.536335552s) [noise=true , address=77.90.31.14:4686 , piece_id=4ARR5GY74KXEVSKHBQRAS2WADAACDNCGO2LPNSWO5LRUZQ7GEQIA, seg_pos={0 0}, upload=317212]: Done uploading piece. err: <nil> Jul 9 21:39:40.548306308 (396h29m15.318947683s) [noise=true , address=79.127.201.215:4352 , piece_id=K4L367RZGAISJISAJZRPNAPGRSNAO2QL6YRLVCTRQGDXMT23DKBA, seg_pos={0 0}, upload=317212]: Done uploading piece. err: <nil> Jul 9 21:40:16.301179498 (396h29m51.071820823s) [noise=true , address=64.49.13.169:4706 , piece_id=4CU34X6K2PKGF6YFOLFVGF62G4RPZ7BYBOVMVPMLTPYVZ2OAJ4LQ, seg_pos={0 0}, upload=317212]: Done uploading piece. err: <nil> Jul 9 21:40:31.659193440 (396h30m6.429834776s) [noise=true , address=104.36.82.23:4452 , piece_id=G4XFNFBBTJBOXSPS7YOTCSGFC7NY3VH5WUQ5BRFWWJ3IWXGPT3SA, seg_pos={0 0}, upload=317212]: Done uploading piece. err: ecclient: upload failed (node:1sAh1mLmzbtmc4o2FAZssW8Y7iDDNmNTCXdCS8xrH8PBVQqxG5, address:104.36.82.23:4452): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:40:31.659249508 (396h30m6.429890833s) [noise=true , address=104.36.82.21:4473 , piece_id=65CNRWVKLCWLW6IVVIMV2O3U3ONRWJWTPZ5WA6AB7W7CSKHJJGHQ, seg_pos={0 0}, upload=317212]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:104.36.82.21:4473): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:40:31.659428720 (396h30m6.430070055s) [noise=true , address=45.91.95.4:4036 , piece_id=GSSBZKTAVOFGTBOD2SQ3C5Y6P3KYWZD7UJFJ4L2ZA2EYZ7DQAZKA, seg_pos={0 0}, upload=317212]: Done uploading piece. err: ecclient: upload failed (node:12iVWBkBV5xpsCNemxmJasZtAyWa7L2QxMDyvKK85DcHvnsZ4YS, address:45.91.95.4:4036): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:40:31.659540674 (396h30m6.430182009s) [noise=true , address=45.91.95.3:4004 , piece_id=V3MVFLFQMXW3KO4PSEL3BQUEOWL4OR5HPPWRONMEVFPHOORC5CYQ, seg_pos={0 0}, upload=317212]: Done uploading piece. err: ecclient: upload failed (node:12vyhWVEevgJh7gAqrpfHeHyXeboegjmnKZAZpGKC8nvXdu6fnr, address:45.91.95.3:4004): protocol: write tcp 188.166.225.170:54188->45.91.95.3:4004: use of closed network connection; write tcp 188.166.225.170:54188->45.91.95.3:4004: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54188->45.91.95.3:4004: use of closed network connection Jul 9 21:40:31.659695461 (396h30m6.430336796s) [noise=true , address=104.36.82.19:4425 , piece_id=GMMOB7UIYDEB37LG5WCBGSGIH6IT652DHXBTH66MLWHBFF3IQTBA, seg_pos={0 0}, upload=317212]: Done uploading piece. err: ecclient: upload failed (node:13sAcGFjAXuesAkUn7NuLaaV8TAah3Mr1kqd33wwh9MTELqmpD, address:104.36.82.19:4425): protocol: write tcp 188.166.225.170:47080->104.36.82.19:4425: use of closed network connection; write tcp 188.166.225.170:47080->104.36.82.19:4425: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47080->104.36.82.19:4425: use of closed network connection Jul 9 21:40:31.659717543 (396h30m6.430358868s) [noise=true , address=45.91.95.2:4012 , piece_id=ZNJPEWCNHTPOFCWXHCCTTOCFNDJSZE2NRXJK2SWE5DX2YSSIT2UQ, seg_pos={0 0}, upload=317212]: Done uploading piece. err: ecclient: upload failed (node:12T59HTbN21RvB3a7g4eniRrCRvFiqe9Jpsncf6yUPzWTmJG9Vu, address:45.91.95.2:4012): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:40:31.659748442 (396h30m6.430389777s) [seg_pos={0 0}, upload=317212]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 9 21:40:31.902160282 (396h30m6.672801617s) [upload=317212]: Done uploading segments... Jul 9 21:40:31.902184678 (396h30m6.672826013s) [upload=317212]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 31.217222613s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [......X...X................X....X...............X..X.X...X..] duration: 49.118419028s FAIL (2025-07-09 21:42:16 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 727635s) [upload=317212]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 21:39:32.743915214 (396h29m7.514556539s) [seg_pos={0 0}, upload=317212]: Begin upload segment... Jul 9 21:39:32.743955091 (396h29m7.514596416s) [seg_pos={0 0}, upload=317212]: Done begin upload segment. Jul 9 21:39:32.743976182 (396h29m7.514617517s) [noise=true , address=77.90.31.10:4535 , piece_id=KJER6ZFIVPUXCK6WEI5TX5UX3LU3BAO25MYTQW3A7TIHXHLK4ECA, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.744312375 (396h29m7.51495371s) [noise=true , address=104.36.82.19:4425 , piece_id=GMMOB7UIYDEB37LG5WCBGSGIH6IT652DHXBTH66MLWHBFF3IQTBA, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.744344205 (396h29m7.51498554s) [noise=true , address=77.90.31.12:4616 , piece_id=WYOLXV2SNY3PF3VWQS76KBGNRTOPKDR2DDSMZGMAQCVWKM3JRG7A, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.744776322 (396h29m7.515417667s) [noise=true , address=104.36.82.21:4473 , piece_id=65CNRWVKLCWLW6IVVIMV2O3U3ONRWJWTPZ5WA6AB7W7CSKHJJGHQ, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.744879529 (396h29m7.515520864s) [noise=true , address=77.90.31.18:4831 , piece_id=4CSJQKEAD6XPOQMIIOG6SCTBEBNUPCCED3DL6R367YFKHTKXFAVA, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.745527850 (396h29m7.516169185s) [noise=true , address=64.49.13.137:4502 , piece_id=T4BAEHJ2GEW7YVSNSXTCFGX2ANUGGUYMAU7A2AIILIUBHS2AQMJQ, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.745755746 (396h29m7.516397081s) [noise=true , address=45.91.95.3:4004 , piece_id=V3MVFLFQMXW3KO4PSEL3BQUEOWL4OR5HPPWRONMEVFPHOORC5CYQ, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.746225264 (396h29m7.516866599s) [noise=true , address=45.91.95.2:4012 , piece_id=ZNJPEWCNHTPOFCWXHCCTTOCFNDJSZE2NRXJK2SWE5DX2YSSIT2UQ, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.746255782 (396h29m7.516897117s) [noise=true , address=77.90.31.17:4787 , piece_id=SIKBA3Z4FKNVGMLHVE3ODZP2IJU42LJDA26UQQZRGF45GYOICLSA, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.746836593 (396h29m7.517477928s) [noise=true , address=45.91.95.4:4036 , piece_id=GSSBZKTAVOFGTBOD2SQ3C5Y6P3KYWZD7UJFJ4L2ZA2EYZ7DQAZKA, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.747161365 (396h29m7.5178027s) [noise=true , address=77.90.31.16:4767 , piece_id=B3XFM5FUZQHU3CJ5OM3DJBACYYCZNIVON74C4UUSSK5K6GE24JVA, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.747581919 (396h29m7.518223254s) [noise=true , address=104.36.82.23:4452 , piece_id=G4XFNFBBTJBOXSPS7YOTCSGFC7NY3VH5WUQ5BRFWWJ3IWXGPT3SA, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.748234087 (396h29m7.518875412s) [noise=true , address=77.90.31.21:4881 , piece_id=DQRBEHZFEHIMGIKCQIJQQXR6LVG67YOF24ESXW3JLWHBVLWPHEHA, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.748773640 (396h29m7.519414995s) [noise=true , address=77.90.31.11:4575 , piece_id=3EPIKI2CXDS4JWHP5YOT4HWZBBZBTWSJZ2BCYULSZTQPFULAVKBA, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.749532161 (396h29m7.520173496s) [noise=true , address=64.49.13.169:4706 , piece_id=4CU34X6K2PKGF6YFOLFVGF62G4RPZ7BYBOVMVPMLTPYVZ2OAJ4LQ, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.750049140 (396h29m7.520690475s) [noise=true , address=79.127.219.38:4006 , piece_id=6S2FX7PM4HMCULTOQ4WW6TDN52W43CRPCZC5A3DYFI5ON3QJ3WIA, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.750308366 (396h29m7.520949701s) [noise=true , address=79.127.201.215:4352 , piece_id=K4L367RZGAISJISAJZRPNAPGRSNAO2QL6YRLVCTRQGDXMT23DKBA, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.764751689 (396h29m7.535393024s) [noise=true , address=77.90.31.14:4686 , piece_id=4ARR5GY74KXEVSKHBQRAS2WADAACDNCGO2LPNSWO5LRUZQ7GEQIA, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:33.880679709 (396h29m8.651321044s) [upload=317212]: Next returned nil segment Jul 9 21:39:33.880697924 (396h29m8.65133926s) [upload=317212]: Waiting for error group managing segments... Jul 9 21:39:34.264937553 (396h29m9.035578888s) [noise=true , address=64.49.13.137:4502 , piece_id=T4BAEHJ2GEW7YVSNSXTCFGX2ANUGGUYMAU7A2AIILIUBHS2AQMJQ, seg_pos={0 0}, upload=317212]: Done uploading piece. err: <nil> Jul 9 21:39:34.740039134 (396h29m9.510680469s) [noise=true , address=77.90.31.18:4831 , piece_id=4CSJQKEAD6XPOQMIIOG6SCTBEBNUPCCED3DL6R367YFKHTKXFAVA, seg_pos={0 0}, upload=317212]: Done uploading piece. err: <nil> Jul 9 21:39:34.742667242 (396h29m9.513308567s) [noise=true , address=77.90.31.16:4767 , piece_id=B3XFM5FUZQHU3CJ5OM3DJBACYYCZNIVON74C4UUSSK5K6GE24JVA, seg_pos={0 0}, upload=317212]: Done uploading piece. err: <nil> Jul 9 21:39:34.745658134 (396h29m9.516299469s) [noise=true , address=77.90.31.12:4616 , piece_id=WYOLXV2SNY3PF3VWQS76KBGNRTOPKDR2DDSMZGMAQCVWKM3JRG7A, seg_pos={0 0}, upload=317212]: Done uploading piece. err: <nil> Jul 9 21:39:34.746157450 (396h29m9.516798785s) [noise=true , address=79.127.219.38:4006 , piece_id=6S2FX7PM4HMCULTOQ4WW6TDN52W43CRPCZC5A3DYFI5ON3QJ3WIA, seg_pos={0 0}, upload=317212]: Done uploading piece. err: <nil> Jul 9 21:39:34.751901208 (396h29m9.522542533s) [noise=true , address=77.90.31.11:4575 , piece_id=3EPIKI2CXDS4JWHP5YOT4HWZBBZBTWSJZ2BCYULSZTQPFULAVKBA, seg_pos={0 0}, upload=317212]: Done uploading piece. err: <nil> Jul 9 21:39:34.757250002 (396h29m9.527891337s) [noise=true , address=77.90.31.10:4535 , piece_id=KJER6ZFIVPUXCK6WEI5TX5UX3LU3BAO25MYTQW3A7TIHXHLK4ECA, seg_pos={0 0}, upload=317212]: Done uploading piece. err: <nil> Jul 9 21:39:34.757587377 (396h29m9.528228712s) [noise=true , address=77.90.31.17:4787 , piece_id=SIKBA3Z4FKNVGMLHVE3ODZP2IJU42LJDA26UQQZRGF45GYOICLSA, seg_pos={0 0}, upload=317212]: Done uploading piece. err: <nil> Jul 9 21:39:34.761661611 (396h29m9.532302946s) [noise=true , address=77.90.31.21:4881 , piece_id=DQRBEHZFEHIMGIKCQIJQQXR6LVG67YOF24ESXW3JLWHBVLWPHEHA, seg_pos={0 0}, upload=317212]: Done uploading piece. err: <nil> Jul 9 21:39:34.765694217 (396h29m9.536335552s) [noise=true , address=77.90.31.14:4686 , piece_id=4ARR5GY74KXEVSKHBQRAS2WADAACDNCGO2LPNSWO5LRUZQ7GEQIA, seg_pos={0 0}, upload=317212]: Done uploading piece. err: <nil> Jul 9 21:39:40.548306308 (396h29m15.318947683s) [noise=true , address=79.127.201.215:4352 , piece_id=K4L367RZGAISJISAJZRPNAPGRSNAO2QL6YRLVCTRQGDXMT23DKBA, seg_pos={0 0}, upload=317212]: Done uploading piece. err: <nil> Jul 9 21:40:16.301179498 (396h29m51.071820823s) [noise=true , address=64.49.13.169:4706 , piece_id=4CU34X6K2PKGF6YFOLFVGF62G4RPZ7BYBOVMVPMLTPYVZ2OAJ4LQ, seg_pos={0 0}, upload=317212]: Done uploading piece. err: <nil> Jul 9 21:40:31.659193440 (396h30m6.429834776s) [noise=true , address=104.36.82.23:4452 , piece_id=G4XFNFBBTJBOXSPS7YOTCSGFC7NY3VH5WUQ5BRFWWJ3IWXGPT3SA, seg_pos={0 0}, upload=317212]: Done uploading piece. err: ecclient: upload failed (node:1sAh1mLmzbtmc4o2FAZssW8Y7iDDNmNTCXdCS8xrH8PBVQqxG5, address:104.36.82.23:4452): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:40:31.659249508 (396h30m6.429890833s) [noise=true , address=104.36.82.21:4473 , piece_id=65CNRWVKLCWLW6IVVIMV2O3U3ONRWJWTPZ5WA6AB7W7CSKHJJGHQ, seg_pos={0 0}, upload=317212]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:104.36.82.21:4473): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:40:31.659428720 (396h30m6.430070055s) [noise=true , address=45.91.95.4:4036 , piece_id=GSSBZKTAVOFGTBOD2SQ3C5Y6P3KYWZD7UJFJ4L2ZA2EYZ7DQAZKA, seg_pos={0 0}, upload=317212]: Done uploading piece. err: ecclient: upload failed (node:12iVWBkBV5xpsCNemxmJasZtAyWa7L2QxMDyvKK85DcHvnsZ4YS, address:45.91.95.4:4036): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:40:31.659540674 (396h30m6.430182009s) [noise=true , address=45.91.95.3:4004 , piece_id=V3MVFLFQMXW3KO4PSEL3BQUEOWL4OR5HPPWRONMEVFPHOORC5CYQ, seg_pos={0 0}, upload=317212]: Done uploading piece. err: ecclient: upload failed (node:12vyhWVEevgJh7gAqrpfHeHyXeboegjmnKZAZpGKC8nvXdu6fnr, address:45.91.95.3:4004): protocol: write tcp 188.166.225.170:54188->45.91.95.3:4004: use of closed network connection; write tcp 188.166.225.170:54188->45.91.95.3:4004: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54188->45.91.95.3:4004: use of closed network connection Jul 9 21:40:31.659695461 (396h30m6.430336796s) [noise=true , address=104.36.82.19:4425 , piece_id=GMMOB7UIYDEB37LG5WCBGSGIH6IT652DHXBTH66MLWHBFF3IQTBA, seg_pos={0 0}, upload=317212]: Done uploading piece. err: ecclient: upload failed (node:13sAcGFjAXuesAkUn7NuLaaV8TAah3Mr1kqd33wwh9MTELqmpD, address:104.36.82.19:4425): protocol: write tcp 188.166.225.170:47080->104.36.82.19:4425: use of closed network connection; write tcp 188.166.225.170:47080->104.36.82.19:4425: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47080->104.36.82.19:4425: use of closed network connection Jul 9 21:40:31.659717543 (396h30m6.430358868s) [noise=true , address=45.91.95.2:4012 , piece_id=ZNJPEWCNHTPOFCWXHCCTTOCFNDJSZE2NRXJK2SWE5DX2YSSIT2UQ, seg_pos={0 0}, upload=317212]: Done uploading piece. err: ecclient: upload failed (node:12T59HTbN21RvB3a7g4eniRrCRvFiqe9Jpsncf6yUPzWTmJG9Vu, address:45.91.95.2:4012): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:40:31.659748442 (396h30m6.430389777s) [seg_pos={0 0}, upload=317212]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 9 21:40:31.902160282 (396h30m6.672801617s) [upload=317212]: Done uploading segments... Jul 9 21:40:31.902184678 (396h30m6.672826013s) [upload=317212]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 49.118447514s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [........X......X....X..X.........X...............X..X....X..] duration: 43.514492373s FAIL (2025-07-09 21:41:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 43.514510197s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [.........X......X....X..X.........X...............X..X....X.] duration: 29.773434933s FAIL (2025-07-09 21:41:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 29.773445243s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 6.183115717s FAIL (2025-07-09 21:40:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.183159271s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [.......X...X................X....X...............X..X.X...X.] duration: 14.688662298s FAIL (2025-07-09 21:40:46 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 727635s) [upload=317212]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 21:39:32.743915214 (396h29m7.514556539s) [seg_pos={0 0}, upload=317212]: Begin upload segment... Jul 9 21:39:32.743955091 (396h29m7.514596416s) [seg_pos={0 0}, upload=317212]: Done begin upload segment. Jul 9 21:39:32.743976182 (396h29m7.514617517s) [noise=true , address=77.90.31.10:4535 , piece_id=KJER6ZFIVPUXCK6WEI5TX5UX3LU3BAO25MYTQW3A7TIHXHLK4ECA, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.744312375 (396h29m7.51495371s) [noise=true , address=104.36.82.19:4425 , piece_id=GMMOB7UIYDEB37LG5WCBGSGIH6IT652DHXBTH66MLWHBFF3IQTBA, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.744344205 (396h29m7.51498554s) [noise=true , address=77.90.31.12:4616 , piece_id=WYOLXV2SNY3PF3VWQS76KBGNRTOPKDR2DDSMZGMAQCVWKM3JRG7A, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.744776322 (396h29m7.515417667s) [noise=true , address=104.36.82.21:4473 , piece_id=65CNRWVKLCWLW6IVVIMV2O3U3ONRWJWTPZ5WA6AB7W7CSKHJJGHQ, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.744879529 (396h29m7.515520864s) [noise=true , address=77.90.31.18:4831 , piece_id=4CSJQKEAD6XPOQMIIOG6SCTBEBNUPCCED3DL6R367YFKHTKXFAVA, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.745527850 (396h29m7.516169185s) [noise=true , address=64.49.13.137:4502 , piece_id=T4BAEHJ2GEW7YVSNSXTCFGX2ANUGGUYMAU7A2AIILIUBHS2AQMJQ, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.745755746 (396h29m7.516397081s) [noise=true , address=45.91.95.3:4004 , piece_id=V3MVFLFQMXW3KO4PSEL3BQUEOWL4OR5HPPWRONMEVFPHOORC5CYQ, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.746225264 (396h29m7.516866599s) [noise=true , address=45.91.95.2:4012 , piece_id=ZNJPEWCNHTPOFCWXHCCTTOCFNDJSZE2NRXJK2SWE5DX2YSSIT2UQ, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.746255782 (396h29m7.516897117s) [noise=true , address=77.90.31.17:4787 , piece_id=SIKBA3Z4FKNVGMLHVE3ODZP2IJU42LJDA26UQQZRGF45GYOICLSA, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.746836593 (396h29m7.517477928s) [noise=true , address=45.91.95.4:4036 , piece_id=GSSBZKTAVOFGTBOD2SQ3C5Y6P3KYWZD7UJFJ4L2ZA2EYZ7DQAZKA, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.747161365 (396h29m7.5178027s) [noise=true , address=77.90.31.16:4767 , piece_id=B3XFM5FUZQHU3CJ5OM3DJBACYYCZNIVON74C4UUSSK5K6GE24JVA, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.747581919 (396h29m7.518223254s) [noise=true , address=104.36.82.23:4452 , piece_id=G4XFNFBBTJBOXSPS7YOTCSGFC7NY3VH5WUQ5BRFWWJ3IWXGPT3SA, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.748234087 (396h29m7.518875412s) [noise=true , address=77.90.31.21:4881 , piece_id=DQRBEHZFEHIMGIKCQIJQQXR6LVG67YOF24ESXW3JLWHBVLWPHEHA, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.748773640 (396h29m7.519414995s) [noise=true , address=77.90.31.11:4575 , piece_id=3EPIKI2CXDS4JWHP5YOT4HWZBBZBTWSJZ2BCYULSZTQPFULAVKBA, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.749532161 (396h29m7.520173496s) [noise=true , address=64.49.13.169:4706 , piece_id=4CU34X6K2PKGF6YFOLFVGF62G4RPZ7BYBOVMVPMLTPYVZ2OAJ4LQ, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.750049140 (396h29m7.520690475s) [noise=true , address=79.127.219.38:4006 , piece_id=6S2FX7PM4HMCULTOQ4WW6TDN52W43CRPCZC5A3DYFI5ON3QJ3WIA, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.750308366 (396h29m7.520949701s) [noise=true , address=79.127.201.215:4352 , piece_id=K4L367RZGAISJISAJZRPNAPGRSNAO2QL6YRLVCTRQGDXMT23DKBA, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.764751689 (396h29m7.535393024s) [noise=true , address=77.90.31.14:4686 , piece_id=4ARR5GY74KXEVSKHBQRAS2WADAACDNCGO2LPNSWO5LRUZQ7GEQIA, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:33.880679709 (396h29m8.651321044s) [upload=317212]: Next returned nil segment Jul 9 21:39:33.880697924 (396h29m8.65133926s) [upload=317212]: Waiting for error group managing segments... Jul 9 21:39:34.264937553 (396h29m9.035578888s) [noise=true , address=64.49.13.137:4502 , piece_id=T4BAEHJ2GEW7YVSNSXTCFGX2ANUGGUYMAU7A2AIILIUBHS2AQMJQ, seg_pos={0 0}, upload=317212]: Done uploading piece. err: <nil> Jul 9 21:39:34.740039134 (396h29m9.510680469s) [noise=true , address=77.90.31.18:4831 , piece_id=4CSJQKEAD6XPOQMIIOG6SCTBEBNUPCCED3DL6R367YFKHTKXFAVA, seg_pos={0 0}, upload=317212]: Done uploading piece. err: <nil> Jul 9 21:39:34.742667242 (396h29m9.513308567s) [noise=true , address=77.90.31.16:4767 , piece_id=B3XFM5FUZQHU3CJ5OM3DJBACYYCZNIVON74C4UUSSK5K6GE24JVA, seg_pos={0 0}, upload=317212]: Done uploading piece. err: <nil> Jul 9 21:39:34.745658134 (396h29m9.516299469s) [noise=true , address=77.90.31.12:4616 , piece_id=WYOLXV2SNY3PF3VWQS76KBGNRTOPKDR2DDSMZGMAQCVWKM3JRG7A, seg_pos={0 0}, upload=317212]: Done uploading piece. err: <nil> Jul 9 21:39:34.746157450 (396h29m9.516798785s) [noise=true , address=79.127.219.38:4006 , piece_id=6S2FX7PM4HMCULTOQ4WW6TDN52W43CRPCZC5A3DYFI5ON3QJ3WIA, seg_pos={0 0}, upload=317212]: Done uploading piece. err: <nil> Jul 9 21:39:34.751901208 (396h29m9.522542533s) [noise=true , address=77.90.31.11:4575 , piece_id=3EPIKI2CXDS4JWHP5YOT4HWZBBZBTWSJZ2BCYULSZTQPFULAVKBA, seg_pos={0 0}, upload=317212]: Done uploading piece. err: <nil> Jul 9 21:39:34.757250002 (396h29m9.527891337s) [noise=true , address=77.90.31.10:4535 , piece_id=KJER6ZFIVPUXCK6WEI5TX5UX3LU3BAO25MYTQW3A7TIHXHLK4ECA, seg_pos={0 0}, upload=317212]: Done uploading piece. err: <nil> Jul 9 21:39:34.757587377 (396h29m9.528228712s) [noise=true , address=77.90.31.17:4787 , piece_id=SIKBA3Z4FKNVGMLHVE3ODZP2IJU42LJDA26UQQZRGF45GYOICLSA, seg_pos={0 0}, upload=317212]: Done uploading piece. err: <nil> Jul 9 21:39:34.761661611 (396h29m9.532302946s) [noise=true , address=77.90.31.21:4881 , piece_id=DQRBEHZFEHIMGIKCQIJQQXR6LVG67YOF24ESXW3JLWHBVLWPHEHA, seg_pos={0 0}, upload=317212]: Done uploading piece. err: <nil> Jul 9 21:39:34.765694217 (396h29m9.536335552s) [noise=true , address=77.90.31.14:4686 , piece_id=4ARR5GY74KXEVSKHBQRAS2WADAACDNCGO2LPNSWO5LRUZQ7GEQIA, seg_pos={0 0}, upload=317212]: Done uploading piece. err: <nil> Jul 9 21:39:40.548306308 (396h29m15.318947683s) [noise=true , address=79.127.201.215:4352 , piece_id=K4L367RZGAISJISAJZRPNAPGRSNAO2QL6YRLVCTRQGDXMT23DKBA, seg_pos={0 0}, upload=317212]: Done uploading piece. err: <nil> Jul 9 21:40:16.301179498 (396h29m51.071820823s) [noise=true , address=64.49.13.169:4706 , piece_id=4CU34X6K2PKGF6YFOLFVGF62G4RPZ7BYBOVMVPMLTPYVZ2OAJ4LQ, seg_pos={0 0}, upload=317212]: Done uploading piece. err: <nil> Jul 9 21:40:31.659193440 (396h30m6.429834776s) [noise=true , address=104.36.82.23:4452 , piece_id=G4XFNFBBTJBOXSPS7YOTCSGFC7NY3VH5WUQ5BRFWWJ3IWXGPT3SA, seg_pos={0 0}, upload=317212]: Done uploading piece. err: ecclient: upload failed (node:1sAh1mLmzbtmc4o2FAZssW8Y7iDDNmNTCXdCS8xrH8PBVQqxG5, address:104.36.82.23:4452): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:40:31.659249508 (396h30m6.429890833s) [noise=true , address=104.36.82.21:4473 , piece_id=65CNRWVKLCWLW6IVVIMV2O3U3ONRWJWTPZ5WA6AB7W7CSKHJJGHQ, seg_pos={0 0}, upload=317212]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:104.36.82.21:4473): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:40:31.659428720 (396h30m6.430070055s) [noise=true , address=45.91.95.4:4036 , piece_id=GSSBZKTAVOFGTBOD2SQ3C5Y6P3KYWZD7UJFJ4L2ZA2EYZ7DQAZKA, seg_pos={0 0}, upload=317212]: Done uploading piece. err: ecclient: upload failed (node:12iVWBkBV5xpsCNemxmJasZtAyWa7L2QxMDyvKK85DcHvnsZ4YS, address:45.91.95.4:4036): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:40:31.659540674 (396h30m6.430182009s) [noise=true , address=45.91.95.3:4004 , piece_id=V3MVFLFQMXW3KO4PSEL3BQUEOWL4OR5HPPWRONMEVFPHOORC5CYQ, seg_pos={0 0}, upload=317212]: Done uploading piece. err: ecclient: upload failed (node:12vyhWVEevgJh7gAqrpfHeHyXeboegjmnKZAZpGKC8nvXdu6fnr, address:45.91.95.3:4004): protocol: write tcp 188.166.225.170:54188->45.91.95.3:4004: use of closed network connection; write tcp 188.166.225.170:54188->45.91.95.3:4004: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54188->45.91.95.3:4004: use of closed network connection Jul 9 21:40:31.659695461 (396h30m6.430336796s) [noise=true , address=104.36.82.19:4425 , piece_id=GMMOB7UIYDEB37LG5WCBGSGIH6IT652DHXBTH66MLWHBFF3IQTBA, seg_pos={0 0}, upload=317212]: Done uploading piece. err: ecclient: upload failed (node:13sAcGFjAXuesAkUn7NuLaaV8TAah3Mr1kqd33wwh9MTELqmpD, address:104.36.82.19:4425): protocol: write tcp 188.166.225.170:47080->104.36.82.19:4425: use of closed network connection; write tcp 188.166.225.170:47080->104.36.82.19:4425: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47080->104.36.82.19:4425: use of closed network connection Jul 9 21:40:31.659717543 (396h30m6.430358868s) [noise=true , address=45.91.95.2:4012 , piece_id=ZNJPEWCNHTPOFCWXHCCTTOCFNDJSZE2NRXJK2SWE5DX2YSSIT2UQ, seg_pos={0 0}, upload=317212]: Done uploading piece. err: ecclient: upload failed (node:12T59HTbN21RvB3a7g4eniRrCRvFiqe9Jpsncf6yUPzWTmJG9Vu, address:45.91.95.2:4012): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:40:31.659748442 (396h30m6.430389777s) [seg_pos={0 0}, upload=317212]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 9 21:40:31.902160282 (396h30m6.672801617s) [upload=317212]: Done uploading segments... Jul 9 21:40:31.902184678 (396h30m6.672826013s) [upload=317212]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 14.688682164s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [X.........X......X....X..X.........X...............X..X....X] duration: 1m0.000713526s FAIL (2025-07-09 21:40:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000727322s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [........X...X................X....X...............X..X.X...X] duration: 1m0.245045127s FAIL (2025-07-09 21:40:31 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 727635s) [upload=317212]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 21:39:32.743915214 (396h29m7.514556539s) [seg_pos={0 0}, upload=317212]: Begin upload segment... Jul 9 21:39:32.743955091 (396h29m7.514596416s) [seg_pos={0 0}, upload=317212]: Done begin upload segment. Jul 9 21:39:32.743976182 (396h29m7.514617517s) [noise=true , address=77.90.31.10:4535 , piece_id=KJER6ZFIVPUXCK6WEI5TX5UX3LU3BAO25MYTQW3A7TIHXHLK4ECA, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.744312375 (396h29m7.51495371s) [noise=true , address=104.36.82.19:4425 , piece_id=GMMOB7UIYDEB37LG5WCBGSGIH6IT652DHXBTH66MLWHBFF3IQTBA, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.744344205 (396h29m7.51498554s) [noise=true , address=77.90.31.12:4616 , piece_id=WYOLXV2SNY3PF3VWQS76KBGNRTOPKDR2DDSMZGMAQCVWKM3JRG7A, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.744776322 (396h29m7.515417667s) [noise=true , address=104.36.82.21:4473 , piece_id=65CNRWVKLCWLW6IVVIMV2O3U3ONRWJWTPZ5WA6AB7W7CSKHJJGHQ, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.744879529 (396h29m7.515520864s) [noise=true , address=77.90.31.18:4831 , piece_id=4CSJQKEAD6XPOQMIIOG6SCTBEBNUPCCED3DL6R367YFKHTKXFAVA, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.745527850 (396h29m7.516169185s) [noise=true , address=64.49.13.137:4502 , piece_id=T4BAEHJ2GEW7YVSNSXTCFGX2ANUGGUYMAU7A2AIILIUBHS2AQMJQ, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.745755746 (396h29m7.516397081s) [noise=true , address=45.91.95.3:4004 , piece_id=V3MVFLFQMXW3KO4PSEL3BQUEOWL4OR5HPPWRONMEVFPHOORC5CYQ, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.746225264 (396h29m7.516866599s) [noise=true , address=45.91.95.2:4012 , piece_id=ZNJPEWCNHTPOFCWXHCCTTOCFNDJSZE2NRXJK2SWE5DX2YSSIT2UQ, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.746255782 (396h29m7.516897117s) [noise=true , address=77.90.31.17:4787 , piece_id=SIKBA3Z4FKNVGMLHVE3ODZP2IJU42LJDA26UQQZRGF45GYOICLSA, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.746836593 (396h29m7.517477928s) [noise=true , address=45.91.95.4:4036 , piece_id=GSSBZKTAVOFGTBOD2SQ3C5Y6P3KYWZD7UJFJ4L2ZA2EYZ7DQAZKA, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.747161365 (396h29m7.5178027s) [noise=true , address=77.90.31.16:4767 , piece_id=B3XFM5FUZQHU3CJ5OM3DJBACYYCZNIVON74C4UUSSK5K6GE24JVA, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.747581919 (396h29m7.518223254s) [noise=true , address=104.36.82.23:4452 , piece_id=G4XFNFBBTJBOXSPS7YOTCSGFC7NY3VH5WUQ5BRFWWJ3IWXGPT3SA, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.748234087 (396h29m7.518875412s) [noise=true , address=77.90.31.21:4881 , piece_id=DQRBEHZFEHIMGIKCQIJQQXR6LVG67YOF24ESXW3JLWHBVLWPHEHA, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.748773640 (396h29m7.519414995s) [noise=true , address=77.90.31.11:4575 , piece_id=3EPIKI2CXDS4JWHP5YOT4HWZBBZBTWSJZ2BCYULSZTQPFULAVKBA, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.749532161 (396h29m7.520173496s) [noise=true , address=64.49.13.169:4706 , piece_id=4CU34X6K2PKGF6YFOLFVGF62G4RPZ7BYBOVMVPMLTPYVZ2OAJ4LQ, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.750049140 (396h29m7.520690475s) [noise=true , address=79.127.219.38:4006 , piece_id=6S2FX7PM4HMCULTOQ4WW6TDN52W43CRPCZC5A3DYFI5ON3QJ3WIA, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.750308366 (396h29m7.520949701s) [noise=true , address=79.127.201.215:4352 , piece_id=K4L367RZGAISJISAJZRPNAPGRSNAO2QL6YRLVCTRQGDXMT23DKBA, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:32.764751689 (396h29m7.535393024s) [noise=true , address=77.90.31.14:4686 , piece_id=4ARR5GY74KXEVSKHBQRAS2WADAACDNCGO2LPNSWO5LRUZQ7GEQIA, seg_pos={0 0}, upload=317212]: Uploading piece... Jul 9 21:39:33.880679709 (396h29m8.651321044s) [upload=317212]: Next returned nil segment Jul 9 21:39:33.880697924 (396h29m8.65133926s) [upload=317212]: Waiting for error group managing segments... Jul 9 21:39:34.264937553 (396h29m9.035578888s) [noise=true , address=64.49.13.137:4502 , piece_id=T4BAEHJ2GEW7YVSNSXTCFGX2ANUGGUYMAU7A2AIILIUBHS2AQMJQ, seg_pos={0 0}, upload=317212]: Done uploading piece. err: <nil> Jul 9 21:39:34.740039134 (396h29m9.510680469s) [noise=true , address=77.90.31.18:4831 , piece_id=4CSJQKEAD6XPOQMIIOG6SCTBEBNUPCCED3DL6R367YFKHTKXFAVA, seg_pos={0 0}, upload=317212]: Done uploading piece. err: <nil> Jul 9 21:39:34.742667242 (396h29m9.513308567s) [noise=true , address=77.90.31.16:4767 , piece_id=B3XFM5FUZQHU3CJ5OM3DJBACYYCZNIVON74C4UUSSK5K6GE24JVA, seg_pos={0 0}, upload=317212]: Done uploading piece. err: <nil> Jul 9 21:39:34.745658134 (396h29m9.516299469s) [noise=true , address=77.90.31.12:4616 , piece_id=WYOLXV2SNY3PF3VWQS76KBGNRTOPKDR2DDSMZGMAQCVWKM3JRG7A, seg_pos={0 0}, upload=317212]: Done uploading piece. err: <nil> Jul 9 21:39:34.746157450 (396h29m9.516798785s) [noise=true , address=79.127.219.38:4006 , piece_id=6S2FX7PM4HMCULTOQ4WW6TDN52W43CRPCZC5A3DYFI5ON3QJ3WIA, seg_pos={0 0}, upload=317212]: Done uploading piece. err: <nil> Jul 9 21:39:34.751901208 (396h29m9.522542533s) [noise=true , address=77.90.31.11:4575 , piece_id=3EPIKI2CXDS4JWHP5YOT4HWZBBZBTWSJZ2BCYULSZTQPFULAVKBA, seg_pos={0 0}, upload=317212]: Done uploading piece. err: <nil> Jul 9 21:39:34.757250002 (396h29m9.527891337s) [noise=true , address=77.90.31.10:4535 , piece_id=KJER6ZFIVPUXCK6WEI5TX5UX3LU3BAO25MYTQW3A7TIHXHLK4ECA, seg_pos={0 0}, upload=317212]: Done uploading piece. err: <nil> Jul 9 21:39:34.757587377 (396h29m9.528228712s) [noise=true , address=77.90.31.17:4787 , piece_id=SIKBA3Z4FKNVGMLHVE3ODZP2IJU42LJDA26UQQZRGF45GYOICLSA, seg_pos={0 0}, upload=317212]: Done uploading piece. err: <nil> Jul 9 21:39:34.761661611 (396h29m9.532302946s) [noise=true , address=77.90.31.21:4881 , piece_id=DQRBEHZFEHIMGIKCQIJQQXR6LVG67YOF24ESXW3JLWHBVLWPHEHA, seg_pos={0 0}, upload=317212]: Done uploading piece. err: <nil> Jul 9 21:39:34.765694217 (396h29m9.536335552s) [noise=true , address=77.90.31.14:4686 , piece_id=4ARR5GY74KXEVSKHBQRAS2WADAACDNCGO2LPNSWO5LRUZQ7GEQIA, seg_pos={0 0}, upload=317212]: Done uploading piece. err: <nil> Jul 9 21:39:40.548306308 (396h29m15.318947683s) [noise=true , address=79.127.201.215:4352 , piece_id=K4L367RZGAISJISAJZRPNAPGRSNAO2QL6YRLVCTRQGDXMT23DKBA, seg_pos={0 0}, upload=317212]: Done uploading piece. err: <nil> Jul 9 21:40:16.301179498 (396h29m51.071820823s) [noise=true , address=64.49.13.169:4706 , piece_id=4CU34X6K2PKGF6YFOLFVGF62G4RPZ7BYBOVMVPMLTPYVZ2OAJ4LQ, seg_pos={0 0}, upload=317212]: Done uploading piece. err: <nil> Jul 9 21:40:31.659193440 (396h30m6.429834776s) [noise=true , address=104.36.82.23:4452 , piece_id=G4XFNFBBTJBOXSPS7YOTCSGFC7NY3VH5WUQ5BRFWWJ3IWXGPT3SA, seg_pos={0 0}, upload=317212]: Done uploading piece. err: ecclient: upload failed (node:1sAh1mLmzbtmc4o2FAZssW8Y7iDDNmNTCXdCS8xrH8PBVQqxG5, address:104.36.82.23:4452): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:40:31.659249508 (396h30m6.429890833s) [noise=true , address=104.36.82.21:4473 , piece_id=65CNRWVKLCWLW6IVVIMV2O3U3ONRWJWTPZ5WA6AB7W7CSKHJJGHQ, seg_pos={0 0}, upload=317212]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:104.36.82.21:4473): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:40:31.659428720 (396h30m6.430070055s) [noise=true , address=45.91.95.4:4036 , piece_id=GSSBZKTAVOFGTBOD2SQ3C5Y6P3KYWZD7UJFJ4L2ZA2EYZ7DQAZKA, seg_pos={0 0}, upload=317212]: Done uploading piece. err: ecclient: upload failed (node:12iVWBkBV5xpsCNemxmJasZtAyWa7L2QxMDyvKK85DcHvnsZ4YS, address:45.91.95.4:4036): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:40:31.659540674 (396h30m6.430182009s) [noise=true , address=45.91.95.3:4004 , piece_id=V3MVFLFQMXW3KO4PSEL3BQUEOWL4OR5HPPWRONMEVFPHOORC5CYQ, seg_pos={0 0}, upload=317212]: Done uploading piece. err: ecclient: upload failed (node:12vyhWVEevgJh7gAqrpfHeHyXeboegjmnKZAZpGKC8nvXdu6fnr, address:45.91.95.3:4004): protocol: write tcp 188.166.225.170:54188->45.91.95.3:4004: use of closed network connection; write tcp 188.166.225.170:54188->45.91.95.3:4004: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54188->45.91.95.3:4004: use of closed network connection Jul 9 21:40:31.659695461 (396h30m6.430336796s) [noise=true , address=104.36.82.19:4425 , piece_id=GMMOB7UIYDEB37LG5WCBGSGIH6IT652DHXBTH66MLWHBFF3IQTBA, seg_pos={0 0}, upload=317212]: Done uploading piece. err: ecclient: upload failed (node:13sAcGFjAXuesAkUn7NuLaaV8TAah3Mr1kqd33wwh9MTELqmpD, address:104.36.82.19:4425): protocol: write tcp 188.166.225.170:47080->104.36.82.19:4425: use of closed network connection; write tcp 188.166.225.170:47080->104.36.82.19:4425: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47080->104.36.82.19:4425: use of closed network connection Jul 9 21:40:31.659717543 (396h30m6.430358868s) [noise=true , address=45.91.95.2:4012 , piece_id=ZNJPEWCNHTPOFCWXHCCTTOCFNDJSZE2NRXJK2SWE5DX2YSSIT2UQ, seg_pos={0 0}, upload=317212]: Done uploading piece. err: ecclient: upload failed (node:12T59HTbN21RvB3a7g4eniRrCRvFiqe9Jpsncf6yUPzWTmJG9Vu, address:45.91.95.2:4012): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:40:31.659748442 (396h30m6.430389777s) [seg_pos={0 0}, upload=317212]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 9 21:40:31.902160282 (396h30m6.672801617s) [upload=317212]: Done uploading segments... Jul 9 21:40:31.902184678 (396h30m6.672826013s) [upload=317212]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 1m0.245073821s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 5.946339036s FAIL (2025-07-09 21:39:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.946354226s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [XX.........X......X....X..X.........X...............X..X....] duration: 43.822420082s FAIL (2025-07-09 21:39:18 UTC) orbiter warning: context deadline exceeded Duration: 43.822431954s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 6.14407276s FAIL (2025-07-09 21:39:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.144093048s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [.........X...X................X....X...............X..X.X...] duration: 20.013343214s FAIL (2025-07-09 21:39:11 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== one starting upload Jul 9 21:36:20.318624655 (396h25m55.08926599s) [upload=317168]: Uploading segments... Jul 9 21:36:20.318743822 (396h25m55.089385158s) [upload=317168]: Got next segment. Inline: false Jul 9 21:36:20.318780774 (396h25m55.089422099s) [upload=317168]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 21:36:20.318785232 (396h25m55.089426567s) [upload=317168]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 21:36:20.763719228 (396h25m55.534360563s) [seg_pos={0 0}, upload=317168]: Begin upload segment... Jul 9 21:36:20.763769214 (396h25m55.534410549s) [seg_pos={0 0}, upload=317168]: Done begin upload segment. Jul 9 21:36:20.763790274 (396h25m55.534431599s) [noise=true , address=104.36.82.44:4888 , piece_id=2F3ZGCXCJNMM7X3ITAHDUY5FOV6XBMY7O57POLYSMJGFL7XANRJQ, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.764193555 (396h25m55.53483489s) [noise=true , address=45.91.95.2:4032 , piece_id=NYJR7Y2CBC3IXJT6QBVOZQOUEUT4XS573ROG27LQQWBDVDOMHBUA, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.764504661 (396h25m55.535145996s) [noise=true , address=104.36.82.5:4858 , piece_id=OC55PCL57AGUANZHVBSSQ4GCMZ7PKZ3AAZN5SSKWASDCH2OQ6PVA, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.764789636 (396h25m55.535430971s) [noise=true , address=77.90.31.17:4785 , piece_id=RDX7V2ES2RK3QUKI7OBE6F3T7DNK5TIIS5JGJG2TENONKS26MOFQ, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.764857276 (396h25m55.535498611s) [noise=true , address=104.36.82.49:4996 , piece_id=DSDQOSI7R7BEK4ZS72OLCC2HLJ6WZSI652YOHY4X66FOCDKX2O4Q, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.765360789 (396h25m55.536002124s) [noise=true , address=77.90.31.19:4865 , piece_id=2KYAW2GTL7CF5BQ4AOJRWFOT7ZO6SC3AZOGG45JGIZW2LPGGCNDQ, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.765905511 (396h25m55.536546836s) [noise=true , address=77.90.31.14:4679 , piece_id=ESHQHAOWFYIVNRLXZ64INNNK6UYLBPAIFH4CCX6A6X3VCUZVZUEA, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.766140190 (396h25m55.536781515s) [noise=true , address=104.36.82.38:4692 , piece_id=UBJ3M3HEK65QOWK7YRDJC4W4ZUVMIYDRRBV6TA2IF3ZP747L2N7Q, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.766384127 (396h25m55.537025462s) [noise=true , address=104.36.82.41:4770 , piece_id=SYPUVS5NMG4JKWTFLN5V22POSYMZOCBZAU52W6TSKR24U3PLHTLQ, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.766633814 (396h25m55.537275149s) [noise=true , address=77.90.31.15:4719 , piece_id=6UMEYHDUVSO2OH4K4FMUJCS75EGNCAJZEUKLX2SJLUFX56GNCYHQ, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.766850239 (396h25m55.537491574s) [noise=true , address=77.90.31.11:4564 , piece_id=HP7Z4KKX7O47BKY6NUNWAZF6WGD7EPQKF2DRTJ7FBGKGMPWMQE3A, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.767051463 (396h25m55.537692798s) [noise=true , address=104.36.82.22:4543 , piece_id=KJ5YMX5Z7H3XEGKXSAVXBUTDEGK2YN2BXNJHD4CBWHD7UDFMEBLA, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.767262577 (396h25m55.537903913s) [noise=true , address=77.90.31.18:4820 , piece_id=RJVQ47F34SPY5X5LLGI425LKGRNNAU72VBOYZCAJCRB67OHWCQHA, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.767485283 (396h25m55.538126609s) [noise=true , address=45.91.95.3:4002 , piece_id=NI64XOPSDEKRWTTRRAJI2LZZEFLXG5HYWGH7BTSYFF272BDG3TFQ, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.767685717 (396h25m55.538327052s) [noise=true , address=45.91.95.5:4022 , piece_id=GBTZSDCXLEURLOXZB7K2XTZPIS7QLDCU5YN2ZBF5TPEG2R6MSBTA, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.767898654 (396h25m55.538539989s) [noise=true , address=77.90.31.13:4645 , piece_id=S37SRDGED55EZJUCLN26VCHJT2HFQE4PWZ4EMAIAGNE7WB4VFRTA, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.768113456 (396h25m55.538754791s) [noise=true , address=77.90.31.10:4551 , piece_id=6G46ASL3W34QTB2IJMSL3HGB22YOVBTKD777IRJXTS4XEAEJIKEA, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.768331041 (396h25m55.538972376s) [noise=true , address=45.91.95.4:4003 , piece_id=KP7IPIU663SIWR7UMAZWZNIGWNJSAE4BZOKD3WKLWEBGVBIPY2RA, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:22.356647592 (396h25m57.127288927s) [upload=317168]: Next returned nil segment Jul 9 21:36:22.356668483 (396h25m57.127309828s) [upload=317168]: Waiting for error group managing segments... Jul 9 21:36:22.764717222 (396h25m57.535358557s) [noise=true , address=77.90.31.11:4564 , piece_id=HP7Z4KKX7O47BKY6NUNWAZF6WGD7EPQKF2DRTJ7FBGKGMPWMQE3A, seg_pos={0 0}, upload=317168]: Done uploading piece. err: <nil> Jul 9 21:36:22.765409827 (396h25m57.536051162s) [noise=true , address=77.90.31.14:4679 , piece_id=ESHQHAOWFYIVNRLXZ64INNNK6UYLBPAIFH4CCX6A6X3VCUZVZUEA, seg_pos={0 0}, upload=317168]: Done uploading piece. err: <nil> Jul 9 21:36:22.766268049 (396h25m57.536909384s) [noise=true , address=77.90.31.18:4820 , piece_id=RJVQ47F34SPY5X5LLGI425LKGRNNAU72VBOYZCAJCRB67OHWCQHA, seg_pos={0 0}, upload=317168]: Done uploading piece. err: <nil> Jul 9 21:36:22.767995543 (396h25m57.538636878s) [noise=true , address=77.90.31.17:4785 , piece_id=RDX7V2ES2RK3QUKI7OBE6F3T7DNK5TIIS5JGJG2TENONKS26MOFQ, seg_pos={0 0}, upload=317168]: Done uploading piece. err: <nil> Jul 9 21:36:22.772020253 (396h25m57.542661589s) [noise=true , address=77.90.31.19:4865 , piece_id=2KYAW2GTL7CF5BQ4AOJRWFOT7ZO6SC3AZOGG45JGIZW2LPGGCNDQ, seg_pos={0 0}, upload=317168]: Done uploading piece. err: <nil> Jul 9 21:36:22.780841681 (396h25m57.551483016s) [noise=true , address=77.90.31.15:4719 , piece_id=6UMEYHDUVSO2OH4K4FMUJCS75EGNCAJZEUKLX2SJLUFX56GNCYHQ, seg_pos={0 0}, upload=317168]: Done uploading piece. err: <nil> Jul 9 21:36:22.789850365 (396h25m57.5604917s) [noise=true , address=77.90.31.13:4645 , piece_id=S37SRDGED55EZJUCLN26VCHJT2HFQE4PWZ4EMAIAGNE7WB4VFRTA, seg_pos={0 0}, upload=317168]: Done uploading piece. err: <nil> Jul 9 21:36:22.792072737 (396h25m57.562714072s) [noise=true , address=77.90.31.10:4551 , piece_id=6G46ASL3W34QTB2IJMSL3HGB22YOVBTKD777IRJXTS4XEAEJIKEA, seg_pos={0 0}, upload=317168]: Done uploading piece. err: <nil> Jul 9 21:36:41.743417799 (396h26m16.514059134s) [noise=true , address=104.36.82.38:4692 , piece_id=UBJ3M3HEK65QOWK7YRDJC4W4ZUVMIYDRRBV6TA2IF3ZP747L2N7Q, seg_pos={0 0}, upload=317168]: Done uploading piece. err: <nil> Jul 9 21:36:46.945197674 (396h26m21.715839009s) [noise=true , address=104.36.82.5:4858 , piece_id=OC55PCL57AGUANZHVBSSQ4GCMZ7PKZ3AAZN5SSKWASDCH2OQ6PVA, seg_pos={0 0}, upload=317168]: Done uploading piece. err: <nil> Jul 9 21:36:47.392071762 (396h26m22.162713087s) [noise=true , address=104.36.82.49:4996 , piece_id=DSDQOSI7R7BEK4ZS72OLCC2HLJ6WZSI652YOHY4X66FOCDKX2O4Q, seg_pos={0 0}, upload=317168]: Done uploading piece. err: <nil> Jul 9 21:37:04.301861761 (396h26m39.072503086s) [noise=true , address=104.36.82.41:4770 , piece_id=SYPUVS5NMG4JKWTFLN5V22POSYMZOCBZAU52W6TSKR24U3PLHTLQ, seg_pos={0 0}, upload=317168]: Done uploading piece. err: <nil> Jul 9 21:37:04.463654747 (396h26m39.234296082s) [noise=true , address=104.36.82.22:4543 , piece_id=KJ5YMX5Z7H3XEGKXSAVXBUTDEGK2YN2BXNJHD4CBWHD7UDFMEBLA, seg_pos={0 0}, upload=317168]: Done uploading piece. err: <nil> Jul 9 21:37:19.671277798 (396h26m54.441919133s) [noise=true , address=45.91.95.2:4032 , piece_id=NYJR7Y2CBC3IXJT6QBVOZQOUEUT4XS573ROG27LQQWBDVDOMHBUA, seg_pos={0 0}, upload=317168]: Done uploading piece. err: ecclient: upload failed (node:12BtBTji4bazMXHJXUPMZAmdFxouybziyrzrP8U6oFR8P3obi6f, address:45.91.95.2:4032): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:37:19.671365025 (396h26m54.44200636s) [noise=true , address=104.36.82.44:4888 , piece_id=2F3ZGCXCJNMM7X3ITAHDUY5FOV6XBMY7O57POLYSMJGFL7XANRJQ, seg_pos={0 0}, upload=317168]: Done uploading piece. err: ecclient: upload failed (node:129FkuTqtaF3DmzHFvcCuJRvaubQV6HbaV1CnFtoY95QHH6NcME, address:104.36.82.44:4888): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:37:19.671428787 (396h26m54.442070122s) [noise=true , address=45.91.95.5:4022 , piece_id=GBTZSDCXLEURLOXZB7K2XTZPIS7QLDCU5YN2ZBF5TPEG2R6MSBTA, seg_pos={0 0}, upload=317168]: Done uploading piece. err: ecclient: upload failed (node:12X8Et5FNyAsE7ifEckfQemZkfnvYUuZ2Rn2xdxNxhmCmuDST2M, address:45.91.95.5:4022): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:37:19.671580928 (396h26m54.442222263s) [noise=true , address=45.91.95.4:4003 , piece_id=KP7IPIU663SIWR7UMAZWZNIGWNJSAE4BZOKD3WKLWEBGVBIPY2RA, seg_pos={0 0}, upload=317168]: Done uploading piece. err: ecclient: upload failed (node:12iLBF47YcG1hvb6vSVNe41MSF6JXgEPYFwLfQcrU8Nt31GDmCg, address:45.91.95.4:4003): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:37:19.671662775 (396h26m54.44230411s) [noise=true , address=45.91.95.3:4002 , piece_id=NI64XOPSDEKRWTTRRAJI2LZZEFLXG5HYWGH7BTSYFF272BDG3TFQ, seg_pos={0 0}, upload=317168]: Done uploading piece. err: ecclient: upload failed (node:1dpBPqn6zMpCUkCr8fEZZ17aV6Ag1ytp8Q3pvawZiHQMxGzAUw, address:45.91.95.3:4002): protocol: write tcp 188.166.225.170:42226->45.91.95.3:4002: use of closed network connection; write tcp 188.166.225.170:42226->45.91.95.3:4002: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42226->45.91.95.3:4002: use of closed network connection Jul 9 21:37:19.671678956 (396h26m54.442320291s) [seg_pos={0 0}, upload=317168]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 21:37:19.903470848 (396h26m54.674112173s) [upload=317168]: Done uploading segments... Jul 9 21:37:19.903494724 (396h26m54.674136049s) [upload=317168]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 20.01336193s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [X..X........................................................] duration: 6.126803755s FAIL (2025-07-09 21:39:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.126829355s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [..........X...X................X....X...............X..X.X..] duration: 31.212175479s FAIL (2025-07-09 21:38:51 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== one starting upload Jul 9 21:36:20.318624655 (396h25m55.08926599s) [upload=317168]: Uploading segments... Jul 9 21:36:20.318743822 (396h25m55.089385158s) [upload=317168]: Got next segment. Inline: false Jul 9 21:36:20.318780774 (396h25m55.089422099s) [upload=317168]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 21:36:20.318785232 (396h25m55.089426567s) [upload=317168]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 21:36:20.763719228 (396h25m55.534360563s) [seg_pos={0 0}, upload=317168]: Begin upload segment... Jul 9 21:36:20.763769214 (396h25m55.534410549s) [seg_pos={0 0}, upload=317168]: Done begin upload segment. Jul 9 21:36:20.763790274 (396h25m55.534431599s) [noise=true , address=104.36.82.44:4888 , piece_id=2F3ZGCXCJNMM7X3ITAHDUY5FOV6XBMY7O57POLYSMJGFL7XANRJQ, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.764193555 (396h25m55.53483489s) [noise=true , address=45.91.95.2:4032 , piece_id=NYJR7Y2CBC3IXJT6QBVOZQOUEUT4XS573ROG27LQQWBDVDOMHBUA, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.764504661 (396h25m55.535145996s) [noise=true , address=104.36.82.5:4858 , piece_id=OC55PCL57AGUANZHVBSSQ4GCMZ7PKZ3AAZN5SSKWASDCH2OQ6PVA, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.764789636 (396h25m55.535430971s) [noise=true , address=77.90.31.17:4785 , piece_id=RDX7V2ES2RK3QUKI7OBE6F3T7DNK5TIIS5JGJG2TENONKS26MOFQ, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.764857276 (396h25m55.535498611s) [noise=true , address=104.36.82.49:4996 , piece_id=DSDQOSI7R7BEK4ZS72OLCC2HLJ6WZSI652YOHY4X66FOCDKX2O4Q, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.765360789 (396h25m55.536002124s) [noise=true , address=77.90.31.19:4865 , piece_id=2KYAW2GTL7CF5BQ4AOJRWFOT7ZO6SC3AZOGG45JGIZW2LPGGCNDQ, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.765905511 (396h25m55.536546836s) [noise=true , address=77.90.31.14:4679 , piece_id=ESHQHAOWFYIVNRLXZ64INNNK6UYLBPAIFH4CCX6A6X3VCUZVZUEA, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.766140190 (396h25m55.536781515s) [noise=true , address=104.36.82.38:4692 , piece_id=UBJ3M3HEK65QOWK7YRDJC4W4ZUVMIYDRRBV6TA2IF3ZP747L2N7Q, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.766384127 (396h25m55.537025462s) [noise=true , address=104.36.82.41:4770 , piece_id=SYPUVS5NMG4JKWTFLN5V22POSYMZOCBZAU52W6TSKR24U3PLHTLQ, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.766633814 (396h25m55.537275149s) [noise=true , address=77.90.31.15:4719 , piece_id=6UMEYHDUVSO2OH4K4FMUJCS75EGNCAJZEUKLX2SJLUFX56GNCYHQ, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.766850239 (396h25m55.537491574s) [noise=true , address=77.90.31.11:4564 , piece_id=HP7Z4KKX7O47BKY6NUNWAZF6WGD7EPQKF2DRTJ7FBGKGMPWMQE3A, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.767051463 (396h25m55.537692798s) [noise=true , address=104.36.82.22:4543 , piece_id=KJ5YMX5Z7H3XEGKXSAVXBUTDEGK2YN2BXNJHD4CBWHD7UDFMEBLA, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.767262577 (396h25m55.537903913s) [noise=true , address=77.90.31.18:4820 , piece_id=RJVQ47F34SPY5X5LLGI425LKGRNNAU72VBOYZCAJCRB67OHWCQHA, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.767485283 (396h25m55.538126609s) [noise=true , address=45.91.95.3:4002 , piece_id=NI64XOPSDEKRWTTRRAJI2LZZEFLXG5HYWGH7BTSYFF272BDG3TFQ, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.767685717 (396h25m55.538327052s) [noise=true , address=45.91.95.5:4022 , piece_id=GBTZSDCXLEURLOXZB7K2XTZPIS7QLDCU5YN2ZBF5TPEG2R6MSBTA, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.767898654 (396h25m55.538539989s) [noise=true , address=77.90.31.13:4645 , piece_id=S37SRDGED55EZJUCLN26VCHJT2HFQE4PWZ4EMAIAGNE7WB4VFRTA, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.768113456 (396h25m55.538754791s) [noise=true , address=77.90.31.10:4551 , piece_id=6G46ASL3W34QTB2IJMSL3HGB22YOVBTKD777IRJXTS4XEAEJIKEA, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.768331041 (396h25m55.538972376s) [noise=true , address=45.91.95.4:4003 , piece_id=KP7IPIU663SIWR7UMAZWZNIGWNJSAE4BZOKD3WKLWEBGVBIPY2RA, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:22.356647592 (396h25m57.127288927s) [upload=317168]: Next returned nil segment Jul 9 21:36:22.356668483 (396h25m57.127309828s) [upload=317168]: Waiting for error group managing segments... Jul 9 21:36:22.764717222 (396h25m57.535358557s) [noise=true , address=77.90.31.11:4564 , piece_id=HP7Z4KKX7O47BKY6NUNWAZF6WGD7EPQKF2DRTJ7FBGKGMPWMQE3A, seg_pos={0 0}, upload=317168]: Done uploading piece. err: <nil> Jul 9 21:36:22.765409827 (396h25m57.536051162s) [noise=true , address=77.90.31.14:4679 , piece_id=ESHQHAOWFYIVNRLXZ64INNNK6UYLBPAIFH4CCX6A6X3VCUZVZUEA, seg_pos={0 0}, upload=317168]: Done uploading piece. err: <nil> Jul 9 21:36:22.766268049 (396h25m57.536909384s) [noise=true , address=77.90.31.18:4820 , piece_id=RJVQ47F34SPY5X5LLGI425LKGRNNAU72VBOYZCAJCRB67OHWCQHA, seg_pos={0 0}, upload=317168]: Done uploading piece. err: <nil> Jul 9 21:36:22.767995543 (396h25m57.538636878s) [noise=true , address=77.90.31.17:4785 , piece_id=RDX7V2ES2RK3QUKI7OBE6F3T7DNK5TIIS5JGJG2TENONKS26MOFQ, seg_pos={0 0}, upload=317168]: Done uploading piece. err: <nil> Jul 9 21:36:22.772020253 (396h25m57.542661589s) [noise=true , address=77.90.31.19:4865 , piece_id=2KYAW2GTL7CF5BQ4AOJRWFOT7ZO6SC3AZOGG45JGIZW2LPGGCNDQ, seg_pos={0 0}, upload=317168]: Done uploading piece. err: <nil> Jul 9 21:36:22.780841681 (396h25m57.551483016s) [noise=true , address=77.90.31.15:4719 , piece_id=6UMEYHDUVSO2OH4K4FMUJCS75EGNCAJZEUKLX2SJLUFX56GNCYHQ, seg_pos={0 0}, upload=317168]: Done uploading piece. err: <nil> Jul 9 21:36:22.789850365 (396h25m57.5604917s) [noise=true , address=77.90.31.13:4645 , piece_id=S37SRDGED55EZJUCLN26VCHJT2HFQE4PWZ4EMAIAGNE7WB4VFRTA, seg_pos={0 0}, upload=317168]: Done uploading piece. err: <nil> Jul 9 21:36:22.792072737 (396h25m57.562714072s) [noise=true , address=77.90.31.10:4551 , piece_id=6G46ASL3W34QTB2IJMSL3HGB22YOVBTKD777IRJXTS4XEAEJIKEA, seg_pos={0 0}, upload=317168]: Done uploading piece. err: <nil> Jul 9 21:36:41.743417799 (396h26m16.514059134s) [noise=true , address=104.36.82.38:4692 , piece_id=UBJ3M3HEK65QOWK7YRDJC4W4ZUVMIYDRRBV6TA2IF3ZP747L2N7Q, seg_pos={0 0}, upload=317168]: Done uploading piece. err: <nil> Jul 9 21:36:46.945197674 (396h26m21.715839009s) [noise=true , address=104.36.82.5:4858 , piece_id=OC55PCL57AGUANZHVBSSQ4GCMZ7PKZ3AAZN5SSKWASDCH2OQ6PVA, seg_pos={0 0}, upload=317168]: Done uploading piece. err: <nil> Jul 9 21:36:47.392071762 (396h26m22.162713087s) [noise=true , address=104.36.82.49:4996 , piece_id=DSDQOSI7R7BEK4ZS72OLCC2HLJ6WZSI652YOHY4X66FOCDKX2O4Q, seg_pos={0 0}, upload=317168]: Done uploading piece. err: <nil> Jul 9 21:37:04.301861761 (396h26m39.072503086s) [noise=true , address=104.36.82.41:4770 , piece_id=SYPUVS5NMG4JKWTFLN5V22POSYMZOCBZAU52W6TSKR24U3PLHTLQ, seg_pos={0 0}, upload=317168]: Done uploading piece. err: <nil> Jul 9 21:37:04.463654747 (396h26m39.234296082s) [noise=true , address=104.36.82.22:4543 , piece_id=KJ5YMX5Z7H3XEGKXSAVXBUTDEGK2YN2BXNJHD4CBWHD7UDFMEBLA, seg_pos={0 0}, upload=317168]: Done uploading piece. err: <nil> Jul 9 21:37:19.671277798 (396h26m54.441919133s) [noise=true , address=45.91.95.2:4032 , piece_id=NYJR7Y2CBC3IXJT6QBVOZQOUEUT4XS573ROG27LQQWBDVDOMHBUA, seg_pos={0 0}, upload=317168]: Done uploading piece. err: ecclient: upload failed (node:12BtBTji4bazMXHJXUPMZAmdFxouybziyrzrP8U6oFR8P3obi6f, address:45.91.95.2:4032): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:37:19.671365025 (396h26m54.44200636s) [noise=true , address=104.36.82.44:4888 , piece_id=2F3ZGCXCJNMM7X3ITAHDUY5FOV6XBMY7O57POLYSMJGFL7XANRJQ, seg_pos={0 0}, upload=317168]: Done uploading piece. err: ecclient: upload failed (node:129FkuTqtaF3DmzHFvcCuJRvaubQV6HbaV1CnFtoY95QHH6NcME, address:104.36.82.44:4888): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:37:19.671428787 (396h26m54.442070122s) [noise=true , address=45.91.95.5:4022 , piece_id=GBTZSDCXLEURLOXZB7K2XTZPIS7QLDCU5YN2ZBF5TPEG2R6MSBTA, seg_pos={0 0}, upload=317168]: Done uploading piece. err: ecclient: upload failed (node:12X8Et5FNyAsE7ifEckfQemZkfnvYUuZ2Rn2xdxNxhmCmuDST2M, address:45.91.95.5:4022): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:37:19.671580928 (396h26m54.442222263s) [noise=true , address=45.91.95.4:4003 , piece_id=KP7IPIU663SIWR7UMAZWZNIGWNJSAE4BZOKD3WKLWEBGVBIPY2RA, seg_pos={0 0}, upload=317168]: Done uploading piece. err: ecclient: upload failed (node:12iLBF47YcG1hvb6vSVNe41MSF6JXgEPYFwLfQcrU8Nt31GDmCg, address:45.91.95.4:4003): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:37:19.671662775 (396h26m54.44230411s) [noise=true , address=45.91.95.3:4002 , piece_id=NI64XOPSDEKRWTTRRAJI2LZZEFLXG5HYWGH7BTSYFF272BDG3TFQ, seg_pos={0 0}, upload=317168]: Done uploading piece. err: ecclient: upload failed (node:1dpBPqn6zMpCUkCr8fEZZ17aV6Ag1ytp8Q3pvawZiHQMxGzAUw, address:45.91.95.3:4002): protocol: write tcp 188.166.225.170:42226->45.91.95.3:4002: use of closed network connection; write tcp 188.166.225.170:42226->45.91.95.3:4002: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42226->45.91.95.3:4002: use of closed network connection Jul 9 21:37:19.671678956 (396h26m54.442320291s) [seg_pos={0 0}, upload=317168]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 21:37:19.903470848 (396h26m54.674112173s) [upload=317168]: Done uploading segments... Jul 9 21:37:19.903494724 (396h26m54.674136049s) [upload=317168]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 31.212197592s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 10 out of 60 is failed (threshold=3) [XXX.........X......X....X..X.........X...............X..X...] duration: 33.580156118s FAIL (2025-07-09 21:38:34 UTC) orbiter warning: context deadline exceeded Duration: 33.580179711s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X..X.......................................................] duration: 6.205240934s FAIL (2025-07-09 21:37:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.205264579s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 10 out of 60 is failed (threshold=3) [.XXX.........X......X....X..X.........X...............X..X..] duration: 39.304850086s FAIL (2025-07-09 21:37:45 UTC) orbiter warning: context deadline exceeded Duration: 39.304869842s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [X..........X...X................X....X...............X..X.X.] duration: 19.880698959s FAIL (2025-07-09 21:37:40 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== one starting upload Jul 9 21:36:20.318624655 (396h25m55.08926599s) [upload=317168]: Uploading segments... Jul 9 21:36:20.318743822 (396h25m55.089385158s) [upload=317168]: Got next segment. Inline: false Jul 9 21:36:20.318780774 (396h25m55.089422099s) [upload=317168]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 21:36:20.318785232 (396h25m55.089426567s) [upload=317168]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 21:36:20.763719228 (396h25m55.534360563s) [seg_pos={0 0}, upload=317168]: Begin upload segment... Jul 9 21:36:20.763769214 (396h25m55.534410549s) [seg_pos={0 0}, upload=317168]: Done begin upload segment. Jul 9 21:36:20.763790274 (396h25m55.534431599s) [noise=true , address=104.36.82.44:4888 , piece_id=2F3ZGCXCJNMM7X3ITAHDUY5FOV6XBMY7O57POLYSMJGFL7XANRJQ, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.764193555 (396h25m55.53483489s) [noise=true , address=45.91.95.2:4032 , piece_id=NYJR7Y2CBC3IXJT6QBVOZQOUEUT4XS573ROG27LQQWBDVDOMHBUA, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.764504661 (396h25m55.535145996s) [noise=true , address=104.36.82.5:4858 , piece_id=OC55PCL57AGUANZHVBSSQ4GCMZ7PKZ3AAZN5SSKWASDCH2OQ6PVA, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.764789636 (396h25m55.535430971s) [noise=true , address=77.90.31.17:4785 , piece_id=RDX7V2ES2RK3QUKI7OBE6F3T7DNK5TIIS5JGJG2TENONKS26MOFQ, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.764857276 (396h25m55.535498611s) [noise=true , address=104.36.82.49:4996 , piece_id=DSDQOSI7R7BEK4ZS72OLCC2HLJ6WZSI652YOHY4X66FOCDKX2O4Q, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.765360789 (396h25m55.536002124s) [noise=true , address=77.90.31.19:4865 , piece_id=2KYAW2GTL7CF5BQ4AOJRWFOT7ZO6SC3AZOGG45JGIZW2LPGGCNDQ, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.765905511 (396h25m55.536546836s) [noise=true , address=77.90.31.14:4679 , piece_id=ESHQHAOWFYIVNRLXZ64INNNK6UYLBPAIFH4CCX6A6X3VCUZVZUEA, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.766140190 (396h25m55.536781515s) [noise=true , address=104.36.82.38:4692 , piece_id=UBJ3M3HEK65QOWK7YRDJC4W4ZUVMIYDRRBV6TA2IF3ZP747L2N7Q, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.766384127 (396h25m55.537025462s) [noise=true , address=104.36.82.41:4770 , piece_id=SYPUVS5NMG4JKWTFLN5V22POSYMZOCBZAU52W6TSKR24U3PLHTLQ, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.766633814 (396h25m55.537275149s) [noise=true , address=77.90.31.15:4719 , piece_id=6UMEYHDUVSO2OH4K4FMUJCS75EGNCAJZEUKLX2SJLUFX56GNCYHQ, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.766850239 (396h25m55.537491574s) [noise=true , address=77.90.31.11:4564 , piece_id=HP7Z4KKX7O47BKY6NUNWAZF6WGD7EPQKF2DRTJ7FBGKGMPWMQE3A, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.767051463 (396h25m55.537692798s) [noise=true , address=104.36.82.22:4543 , piece_id=KJ5YMX5Z7H3XEGKXSAVXBUTDEGK2YN2BXNJHD4CBWHD7UDFMEBLA, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.767262577 (396h25m55.537903913s) [noise=true , address=77.90.31.18:4820 , piece_id=RJVQ47F34SPY5X5LLGI425LKGRNNAU72VBOYZCAJCRB67OHWCQHA, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.767485283 (396h25m55.538126609s) [noise=true , address=45.91.95.3:4002 , piece_id=NI64XOPSDEKRWTTRRAJI2LZZEFLXG5HYWGH7BTSYFF272BDG3TFQ, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.767685717 (396h25m55.538327052s) [noise=true , address=45.91.95.5:4022 , piece_id=GBTZSDCXLEURLOXZB7K2XTZPIS7QLDCU5YN2ZBF5TPEG2R6MSBTA, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.767898654 (396h25m55.538539989s) [noise=true , address=77.90.31.13:4645 , piece_id=S37SRDGED55EZJUCLN26VCHJT2HFQE4PWZ4EMAIAGNE7WB4VFRTA, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.768113456 (396h25m55.538754791s) [noise=true , address=77.90.31.10:4551 , piece_id=6G46ASL3W34QTB2IJMSL3HGB22YOVBTKD777IRJXTS4XEAEJIKEA, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.768331041 (396h25m55.538972376s) [noise=true , address=45.91.95.4:4003 , piece_id=KP7IPIU663SIWR7UMAZWZNIGWNJSAE4BZOKD3WKLWEBGVBIPY2RA, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:22.356647592 (396h25m57.127288927s) [upload=317168]: Next returned nil segment Jul 9 21:36:22.356668483 (396h25m57.127309828s) [upload=317168]: Waiting for error group managing segments... Jul 9 21:36:22.764717222 (396h25m57.535358557s) [noise=true , address=77.90.31.11:4564 , piece_id=HP7Z4KKX7O47BKY6NUNWAZF6WGD7EPQKF2DRTJ7FBGKGMPWMQE3A, seg_pos={0 0}, upload=317168]: Done uploading piece. err: <nil> Jul 9 21:36:22.765409827 (396h25m57.536051162s) [noise=true , address=77.90.31.14:4679 , piece_id=ESHQHAOWFYIVNRLXZ64INNNK6UYLBPAIFH4CCX6A6X3VCUZVZUEA, seg_pos={0 0}, upload=317168]: Done uploading piece. err: <nil> Jul 9 21:36:22.766268049 (396h25m57.536909384s) [noise=true , address=77.90.31.18:4820 , piece_id=RJVQ47F34SPY5X5LLGI425LKGRNNAU72VBOYZCAJCRB67OHWCQHA, seg_pos={0 0}, upload=317168]: Done uploading piece. err: <nil> Jul 9 21:36:22.767995543 (396h25m57.538636878s) [noise=true , address=77.90.31.17:4785 , piece_id=RDX7V2ES2RK3QUKI7OBE6F3T7DNK5TIIS5JGJG2TENONKS26MOFQ, seg_pos={0 0}, upload=317168]: Done uploading piece. err: <nil> Jul 9 21:36:22.772020253 (396h25m57.542661589s) [noise=true , address=77.90.31.19:4865 , piece_id=2KYAW2GTL7CF5BQ4AOJRWFOT7ZO6SC3AZOGG45JGIZW2LPGGCNDQ, seg_pos={0 0}, upload=317168]: Done uploading piece. err: <nil> Jul 9 21:36:22.780841681 (396h25m57.551483016s) [noise=true , address=77.90.31.15:4719 , piece_id=6UMEYHDUVSO2OH4K4FMUJCS75EGNCAJZEUKLX2SJLUFX56GNCYHQ, seg_pos={0 0}, upload=317168]: Done uploading piece. err: <nil> Jul 9 21:36:22.789850365 (396h25m57.5604917s) [noise=true , address=77.90.31.13:4645 , piece_id=S37SRDGED55EZJUCLN26VCHJT2HFQE4PWZ4EMAIAGNE7WB4VFRTA, seg_pos={0 0}, upload=317168]: Done uploading piece. err: <nil> Jul 9 21:36:22.792072737 (396h25m57.562714072s) [noise=true , address=77.90.31.10:4551 , piece_id=6G46ASL3W34QTB2IJMSL3HGB22YOVBTKD777IRJXTS4XEAEJIKEA, seg_pos={0 0}, upload=317168]: Done uploading piece. err: <nil> Jul 9 21:36:41.743417799 (396h26m16.514059134s) [noise=true , address=104.36.82.38:4692 , piece_id=UBJ3M3HEK65QOWK7YRDJC4W4ZUVMIYDRRBV6TA2IF3ZP747L2N7Q, seg_pos={0 0}, upload=317168]: Done uploading piece. err: <nil> Jul 9 21:36:46.945197674 (396h26m21.715839009s) [noise=true , address=104.36.82.5:4858 , piece_id=OC55PCL57AGUANZHVBSSQ4GCMZ7PKZ3AAZN5SSKWASDCH2OQ6PVA, seg_pos={0 0}, upload=317168]: Done uploading piece. err: <nil> Jul 9 21:36:47.392071762 (396h26m22.162713087s) [noise=true , address=104.36.82.49:4996 , piece_id=DSDQOSI7R7BEK4ZS72OLCC2HLJ6WZSI652YOHY4X66FOCDKX2O4Q, seg_pos={0 0}, upload=317168]: Done uploading piece. err: <nil> Jul 9 21:37:04.301861761 (396h26m39.072503086s) [noise=true , address=104.36.82.41:4770 , piece_id=SYPUVS5NMG4JKWTFLN5V22POSYMZOCBZAU52W6TSKR24U3PLHTLQ, seg_pos={0 0}, upload=317168]: Done uploading piece. err: <nil> Jul 9 21:37:04.463654747 (396h26m39.234296082s) [noise=true , address=104.36.82.22:4543 , piece_id=KJ5YMX5Z7H3XEGKXSAVXBUTDEGK2YN2BXNJHD4CBWHD7UDFMEBLA, seg_pos={0 0}, upload=317168]: Done uploading piece. err: <nil> Jul 9 21:37:19.671277798 (396h26m54.441919133s) [noise=true , address=45.91.95.2:4032 , piece_id=NYJR7Y2CBC3IXJT6QBVOZQOUEUT4XS573ROG27LQQWBDVDOMHBUA, seg_pos={0 0}, upload=317168]: Done uploading piece. err: ecclient: upload failed (node:12BtBTji4bazMXHJXUPMZAmdFxouybziyrzrP8U6oFR8P3obi6f, address:45.91.95.2:4032): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:37:19.671365025 (396h26m54.44200636s) [noise=true , address=104.36.82.44:4888 , piece_id=2F3ZGCXCJNMM7X3ITAHDUY5FOV6XBMY7O57POLYSMJGFL7XANRJQ, seg_pos={0 0}, upload=317168]: Done uploading piece. err: ecclient: upload failed (node:129FkuTqtaF3DmzHFvcCuJRvaubQV6HbaV1CnFtoY95QHH6NcME, address:104.36.82.44:4888): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:37:19.671428787 (396h26m54.442070122s) [noise=true , address=45.91.95.5:4022 , piece_id=GBTZSDCXLEURLOXZB7K2XTZPIS7QLDCU5YN2ZBF5TPEG2R6MSBTA, seg_pos={0 0}, upload=317168]: Done uploading piece. err: ecclient: upload failed (node:12X8Et5FNyAsE7ifEckfQemZkfnvYUuZ2Rn2xdxNxhmCmuDST2M, address:45.91.95.5:4022): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:37:19.671580928 (396h26m54.442222263s) [noise=true , address=45.91.95.4:4003 , piece_id=KP7IPIU663SIWR7UMAZWZNIGWNJSAE4BZOKD3WKLWEBGVBIPY2RA, seg_pos={0 0}, upload=317168]: Done uploading piece. err: ecclient: upload failed (node:12iLBF47YcG1hvb6vSVNe41MSF6JXgEPYFwLfQcrU8Nt31GDmCg, address:45.91.95.4:4003): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:37:19.671662775 (396h26m54.44230411s) [noise=true , address=45.91.95.3:4002 , piece_id=NI64XOPSDEKRWTTRRAJI2LZZEFLXG5HYWGH7BTSYFF272BDG3TFQ, seg_pos={0 0}, upload=317168]: Done uploading piece. err: ecclient: upload failed (node:1dpBPqn6zMpCUkCr8fEZZ17aV6Ag1ytp8Q3pvawZiHQMxGzAUw, address:45.91.95.3:4002): protocol: write tcp 188.166.225.170:42226->45.91.95.3:4002: use of closed network connection; write tcp 188.166.225.170:42226->45.91.95.3:4002: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42226->45.91.95.3:4002: use of closed network connection Jul 9 21:37:19.671678956 (396h26m54.442320291s) [seg_pos={0 0}, upload=317168]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 21:37:19.903470848 (396h26m54.674112173s) [upload=317168]: Done uploading segments... Jul 9 21:37:19.903494724 (396h26m54.674136049s) [upload=317168]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 19.880714168s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [.X..........X...X................X....X...............X..X.X] duration: 1m0.233461344s FAIL (2025-07-09 21:37:19 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== one starting upload Jul 9 21:36:20.318624655 (396h25m55.08926599s) [upload=317168]: Uploading segments... Jul 9 21:36:20.318743822 (396h25m55.089385158s) [upload=317168]: Got next segment. Inline: false Jul 9 21:36:20.318780774 (396h25m55.089422099s) [upload=317168]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 21:36:20.318785232 (396h25m55.089426567s) [upload=317168]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 21:36:20.763719228 (396h25m55.534360563s) [seg_pos={0 0}, upload=317168]: Begin upload segment... Jul 9 21:36:20.763769214 (396h25m55.534410549s) [seg_pos={0 0}, upload=317168]: Done begin upload segment. Jul 9 21:36:20.763790274 (396h25m55.534431599s) [noise=true , address=104.36.82.44:4888 , piece_id=2F3ZGCXCJNMM7X3ITAHDUY5FOV6XBMY7O57POLYSMJGFL7XANRJQ, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.764193555 (396h25m55.53483489s) [noise=true , address=45.91.95.2:4032 , piece_id=NYJR7Y2CBC3IXJT6QBVOZQOUEUT4XS573ROG27LQQWBDVDOMHBUA, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.764504661 (396h25m55.535145996s) [noise=true , address=104.36.82.5:4858 , piece_id=OC55PCL57AGUANZHVBSSQ4GCMZ7PKZ3AAZN5SSKWASDCH2OQ6PVA, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.764789636 (396h25m55.535430971s) [noise=true , address=77.90.31.17:4785 , piece_id=RDX7V2ES2RK3QUKI7OBE6F3T7DNK5TIIS5JGJG2TENONKS26MOFQ, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.764857276 (396h25m55.535498611s) [noise=true , address=104.36.82.49:4996 , piece_id=DSDQOSI7R7BEK4ZS72OLCC2HLJ6WZSI652YOHY4X66FOCDKX2O4Q, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.765360789 (396h25m55.536002124s) [noise=true , address=77.90.31.19:4865 , piece_id=2KYAW2GTL7CF5BQ4AOJRWFOT7ZO6SC3AZOGG45JGIZW2LPGGCNDQ, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.765905511 (396h25m55.536546836s) [noise=true , address=77.90.31.14:4679 , piece_id=ESHQHAOWFYIVNRLXZ64INNNK6UYLBPAIFH4CCX6A6X3VCUZVZUEA, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.766140190 (396h25m55.536781515s) [noise=true , address=104.36.82.38:4692 , piece_id=UBJ3M3HEK65QOWK7YRDJC4W4ZUVMIYDRRBV6TA2IF3ZP747L2N7Q, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.766384127 (396h25m55.537025462s) [noise=true , address=104.36.82.41:4770 , piece_id=SYPUVS5NMG4JKWTFLN5V22POSYMZOCBZAU52W6TSKR24U3PLHTLQ, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.766633814 (396h25m55.537275149s) [noise=true , address=77.90.31.15:4719 , piece_id=6UMEYHDUVSO2OH4K4FMUJCS75EGNCAJZEUKLX2SJLUFX56GNCYHQ, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.766850239 (396h25m55.537491574s) [noise=true , address=77.90.31.11:4564 , piece_id=HP7Z4KKX7O47BKY6NUNWAZF6WGD7EPQKF2DRTJ7FBGKGMPWMQE3A, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.767051463 (396h25m55.537692798s) [noise=true , address=104.36.82.22:4543 , piece_id=KJ5YMX5Z7H3XEGKXSAVXBUTDEGK2YN2BXNJHD4CBWHD7UDFMEBLA, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.767262577 (396h25m55.537903913s) [noise=true , address=77.90.31.18:4820 , piece_id=RJVQ47F34SPY5X5LLGI425LKGRNNAU72VBOYZCAJCRB67OHWCQHA, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.767485283 (396h25m55.538126609s) [noise=true , address=45.91.95.3:4002 , piece_id=NI64XOPSDEKRWTTRRAJI2LZZEFLXG5HYWGH7BTSYFF272BDG3TFQ, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.767685717 (396h25m55.538327052s) [noise=true , address=45.91.95.5:4022 , piece_id=GBTZSDCXLEURLOXZB7K2XTZPIS7QLDCU5YN2ZBF5TPEG2R6MSBTA, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.767898654 (396h25m55.538539989s) [noise=true , address=77.90.31.13:4645 , piece_id=S37SRDGED55EZJUCLN26VCHJT2HFQE4PWZ4EMAIAGNE7WB4VFRTA, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.768113456 (396h25m55.538754791s) [noise=true , address=77.90.31.10:4551 , piece_id=6G46ASL3W34QTB2IJMSL3HGB22YOVBTKD777IRJXTS4XEAEJIKEA, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:20.768331041 (396h25m55.538972376s) [noise=true , address=45.91.95.4:4003 , piece_id=KP7IPIU663SIWR7UMAZWZNIGWNJSAE4BZOKD3WKLWEBGVBIPY2RA, seg_pos={0 0}, upload=317168]: Uploading piece... Jul 9 21:36:22.356647592 (396h25m57.127288927s) [upload=317168]: Next returned nil segment Jul 9 21:36:22.356668483 (396h25m57.127309828s) [upload=317168]: Waiting for error group managing segments... Jul 9 21:36:22.764717222 (396h25m57.535358557s) [noise=true , address=77.90.31.11:4564 , piece_id=HP7Z4KKX7O47BKY6NUNWAZF6WGD7EPQKF2DRTJ7FBGKGMPWMQE3A, seg_pos={0 0}, upload=317168]: Done uploading piece. err: <nil> Jul 9 21:36:22.765409827 (396h25m57.536051162s) [noise=true , address=77.90.31.14:4679 , piece_id=ESHQHAOWFYIVNRLXZ64INNNK6UYLBPAIFH4CCX6A6X3VCUZVZUEA, seg_pos={0 0}, upload=317168]: Done uploading piece. err: <nil> Jul 9 21:36:22.766268049 (396h25m57.536909384s) [noise=true , address=77.90.31.18:4820 , piece_id=RJVQ47F34SPY5X5LLGI425LKGRNNAU72VBOYZCAJCRB67OHWCQHA, seg_pos={0 0}, upload=317168]: Done uploading piece. err: <nil> Jul 9 21:36:22.767995543 (396h25m57.538636878s) [noise=true , address=77.90.31.17:4785 , piece_id=RDX7V2ES2RK3QUKI7OBE6F3T7DNK5TIIS5JGJG2TENONKS26MOFQ, seg_pos={0 0}, upload=317168]: Done uploading piece. err: <nil> Jul 9 21:36:22.772020253 (396h25m57.542661589s) [noise=true , address=77.90.31.19:4865 , piece_id=2KYAW2GTL7CF5BQ4AOJRWFOT7ZO6SC3AZOGG45JGIZW2LPGGCNDQ, seg_pos={0 0}, upload=317168]: Done uploading piece. err: <nil> Jul 9 21:36:22.780841681 (396h25m57.551483016s) [noise=true , address=77.90.31.15:4719 , piece_id=6UMEYHDUVSO2OH4K4FMUJCS75EGNCAJZEUKLX2SJLUFX56GNCYHQ, seg_pos={0 0}, upload=317168]: Done uploading piece. err: <nil> Jul 9 21:36:22.789850365 (396h25m57.5604917s) [noise=true , address=77.90.31.13:4645 , piece_id=S37SRDGED55EZJUCLN26VCHJT2HFQE4PWZ4EMAIAGNE7WB4VFRTA, seg_pos={0 0}, upload=317168]: Done uploading piece. err: <nil> Jul 9 21:36:22.792072737 (396h25m57.562714072s) [noise=true , address=77.90.31.10:4551 , piece_id=6G46ASL3W34QTB2IJMSL3HGB22YOVBTKD777IRJXTS4XEAEJIKEA, seg_pos={0 0}, upload=317168]: Done uploading piece. err: <nil> Jul 9 21:36:41.743417799 (396h26m16.514059134s) [noise=true , address=104.36.82.38:4692 , piece_id=UBJ3M3HEK65QOWK7YRDJC4W4ZUVMIYDRRBV6TA2IF3ZP747L2N7Q, seg_pos={0 0}, upload=317168]: Done uploading piece. err: <nil> Jul 9 21:36:46.945197674 (396h26m21.715839009s) [noise=true , address=104.36.82.5:4858 , piece_id=OC55PCL57AGUANZHVBSSQ4GCMZ7PKZ3AAZN5SSKWASDCH2OQ6PVA, seg_pos={0 0}, upload=317168]: Done uploading piece. err: <nil> Jul 9 21:36:47.392071762 (396h26m22.162713087s) [noise=true , address=104.36.82.49:4996 , piece_id=DSDQOSI7R7BEK4ZS72OLCC2HLJ6WZSI652YOHY4X66FOCDKX2O4Q, seg_pos={0 0}, upload=317168]: Done uploading piece. err: <nil> Jul 9 21:37:04.301861761 (396h26m39.072503086s) [noise=true , address=104.36.82.41:4770 , piece_id=SYPUVS5NMG4JKWTFLN5V22POSYMZOCBZAU52W6TSKR24U3PLHTLQ, seg_pos={0 0}, upload=317168]: Done uploading piece. err: <nil> Jul 9 21:37:04.463654747 (396h26m39.234296082s) [noise=true , address=104.36.82.22:4543 , piece_id=KJ5YMX5Z7H3XEGKXSAVXBUTDEGK2YN2BXNJHD4CBWHD7UDFMEBLA, seg_pos={0 0}, upload=317168]: Done uploading piece. err: <nil> Jul 9 21:37:19.671277798 (396h26m54.441919133s) [noise=true , address=45.91.95.2:4032 , piece_id=NYJR7Y2CBC3IXJT6QBVOZQOUEUT4XS573ROG27LQQWBDVDOMHBUA, seg_pos={0 0}, upload=317168]: Done uploading piece. err: ecclient: upload failed (node:12BtBTji4bazMXHJXUPMZAmdFxouybziyrzrP8U6oFR8P3obi6f, address:45.91.95.2:4032): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:37:19.671365025 (396h26m54.44200636s) [noise=true , address=104.36.82.44:4888 , piece_id=2F3ZGCXCJNMM7X3ITAHDUY5FOV6XBMY7O57POLYSMJGFL7XANRJQ, seg_pos={0 0}, upload=317168]: Done uploading piece. err: ecclient: upload failed (node:129FkuTqtaF3DmzHFvcCuJRvaubQV6HbaV1CnFtoY95QHH6NcME, address:104.36.82.44:4888): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:37:19.671428787 (396h26m54.442070122s) [noise=true , address=45.91.95.5:4022 , piece_id=GBTZSDCXLEURLOXZB7K2XTZPIS7QLDCU5YN2ZBF5TPEG2R6MSBTA, seg_pos={0 0}, upload=317168]: Done uploading piece. err: ecclient: upload failed (node:12X8Et5FNyAsE7ifEckfQemZkfnvYUuZ2Rn2xdxNxhmCmuDST2M, address:45.91.95.5:4022): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:37:19.671580928 (396h26m54.442222263s) [noise=true , address=45.91.95.4:4003 , piece_id=KP7IPIU663SIWR7UMAZWZNIGWNJSAE4BZOKD3WKLWEBGVBIPY2RA, seg_pos={0 0}, upload=317168]: Done uploading piece. err: ecclient: upload failed (node:12iLBF47YcG1hvb6vSVNe41MSF6JXgEPYFwLfQcrU8Nt31GDmCg, address:45.91.95.4:4003): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:37:19.671662775 (396h26m54.44230411s) [noise=true , address=45.91.95.3:4002 , piece_id=NI64XOPSDEKRWTTRRAJI2LZZEFLXG5HYWGH7BTSYFF272BDG3TFQ, seg_pos={0 0}, upload=317168]: Done uploading piece. err: ecclient: upload failed (node:1dpBPqn6zMpCUkCr8fEZZ17aV6Ag1ytp8Q3pvawZiHQMxGzAUw, address:45.91.95.3:4002): protocol: write tcp 188.166.225.170:42226->45.91.95.3:4002: use of closed network connection; write tcp 188.166.225.170:42226->45.91.95.3:4002: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42226->45.91.95.3:4002: use of closed network connection Jul 9 21:37:19.671678956 (396h26m54.442320291s) [seg_pos={0 0}, upload=317168]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 21:37:19.903470848 (396h26m54.674112173s) [upload=317168]: Done uploading segments... Jul 9 21:37:19.903494724 (396h26m54.674136049s) [upload=317168]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.233481853s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 10 out of 60 is failed (threshold=3) [..XXX.........X......X....X..X.........X...............X..X.] duration: 50.370490149s FAIL (2025-07-09 21:37:01 UTC) orbiter warning: context deadline exceeded Duration: 50.370502583s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X..X......................................................] duration: 6.091796338s FAIL (2025-07-09 21:36:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.091817218s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [..X..........X...X................X....X...............X..X.] duration: 27.223003327s FAIL (2025-07-09 21:36:13 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 96h24m21.205048355s) [upload=317147]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 21:34:46.434424183 (396h24m21.205065518s) [upload=317147]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 21:34:46.879960451 (396h24m21.650601786s) [seg_pos={0 0}, upload=317147]: Begin upload segment... Jul 9 21:34:46.880016248 (396h24m21.650657583s) [seg_pos={0 0}, upload=317147]: Done begin upload segment. Jul 9 21:34:46.880037259 (396h24m21.650678594s) [noise=true , address=45.91.95.4:4023 , piece_id=M62EJFMGKCWPF4VXF5EDS263ZTU6NGNJ3FD4G6GIMU4ADWHED7VQ, seg_pos={0 0}, upload=317147]: Uploading piece... Jul 9 21:34:46.880104728 (396h24m21.650746063s) [noise=true , address=77.90.31.21:4899 , piece_id=VBNJ3F4UOYGMP4UAGXXZRPN5TPL7M7UI4RL6X4QBW4A4PZAWRE2A, seg_pos={0 0}, upload=317147]: Uploading piece... Jul 9 21:34:46.880436733 (396h24m21.651078058s) [noise=true , address=79.127.219.38:4028 , piece_id=NHN7DPEQNHBJ2KW4CSTBZRR532YS4CDBSV3CZ2XIEJG6SAH53H3A, seg_pos={0 0}, upload=317147]: Uploading piece... Jul 9 21:34:46.880595977 (396h24m21.651237312s) [noise=true , address=45.91.95.3:4001 , piece_id=FQUAYGGU5U2FPEKULU3NUEIM3DMSE2C7THKNWQM2M6PPKNW5X7DQ, seg_pos={0 0}, upload=317147]: Uploading piece... Jul 9 21:34:46.880705867 (396h24m21.651347202s) [noise=true , address=79.127.201.215:4325 , piece_id=FDMBJMIULICBLVZJGICJ6MNBA7ZDVFIKVMEMZ5DJYLAKZJW5XDIQ, seg_pos={0 0}, upload=317147]: Uploading piece... Jul 9 21:34:46.880920659 (396h24m21.651561994s) [noise=true , address=45.91.116.8:4072 , piece_id=NEZDKAR2K33S7OUQC5DO3655KIRENC57EJCM7GD24FAM26E2CXVA, seg_pos={0 0}, upload=317147]: Uploading piece... Jul 9 21:34:46.881567216 (396h24m21.652208551s) [noise=true , address=104.36.82.60:4345 , piece_id=UTOQODITF6P25BQHESX2LZ2UB7GEF5XR57ISEAT6KKUMSHC2WB4Q, seg_pos={0 0}, upload=317147]: Uploading piece... Jul 9 21:34:46.881631028 (396h24m21.652272353s) [noise=true , address=45.91.95.2:4024 , piece_id=KPVKM7NRIZVMLT5OGGVYM5LZF4EAU7TCRFGDWUDRRVRDAEZKZNTQ, seg_pos={0 0}, upload=317147]: Uploading piece... Jul 9 21:34:46.882098382 (396h24m21.652739717s) [noise=true , address=77.90.31.16:4767 , piece_id=5RADFBKOMBDSS4XGMOA5KRVKEIGL6P5PCOPV4LYMODBNBUK5BQXA, seg_pos={0 0}, upload=317147]: Uploading piece... Jul 9 21:34:46.882386944 (396h24m21.653028279s) [noise=true , address=77.90.31.12:4606 , piece_id=B5PX27N3ESF665PZ7VQGOLJG4SSFEAAXKX72FFIO734H7XEPAQIA, seg_pos={0 0}, upload=317147]: Uploading piece... Jul 9 21:34:46.882987102 (396h24m21.653628437s) [noise=true , address=77.90.31.18:4818 , piece_id=JPFRUNH2OX4P5TYRSHXDXNH4LLQO2ERXT3RAOIARA2KYJMIQINKQ, seg_pos={0 0}, upload=317147]: Uploading piece... Jul 9 21:34:46.883223445 (396h24m21.65386477s) [noise=true , address=77.90.31.19:4858 , piece_id=AZWZNIRP66BWSPC7YOVWML5IRSK2NPP6ELPGUN6RGOQFPZVXRS4Q, seg_pos={0 0}, upload=317147]: Uploading piece... Jul 9 21:34:46.883807061 (396h24m21.654448396s) [noise=true , address=77.90.31.15:4715 , piece_id=6MF5JAAFOPUWDDZ6ZNWZP6I2K3JJ447KPJ2SVREXZD3NGH6PRZ7A, seg_pos={0 0}, upload=317147]: Uploading piece... Jul 9 21:34:46.883955415 (396h24m21.65459675s) [noise=true , address=77.90.31.13:4641 , piece_id=G7IIH7UASLFHSUIWEJJ5VKV4JVJQFC32UA2G6FVYVEMNUO25TTLQ, seg_pos={0 0}, upload=317147]: Uploading piece... Jul 9 21:34:46.884316034 (396h24m21.65495737s) [noise=true , address=104.36.82.2:4747 , piece_id=RASK6SX5ZPMYSSET6T3UWXNWUNKOTD4TTHB4GLYGQHQQN5WEPUTQ, seg_pos={0 0}, upload=317147]: Uploading piece... Jul 9 21:34:46.884471512 (396h24m21.655112837s) [noise=true , address=104.36.82.35:4598 , piece_id=CNBTKOZNCZM3LJTJIPWTY4UGUCA4TVCXMM4L337N6B5RR5F57GGQ, seg_pos={0 0}, upload=317147]: Uploading piece... Jul 9 21:34:46.884786304 (396h24m21.655427639s) [noise=true , address=104.36.82.8:4983 , piece_id=LT2M37IUHI6BNLR3VDQGT3SSV6SQC64CGSW3I253IAOIGUSYI7WA, seg_pos={0 0}, upload=317147]: Uploading piece... Jul 9 21:34:46.885151844 (396h24m21.655793179s) [noise=true , address=45.91.95.5:4002 , piece_id=H75BYOM2MPHAXNPPGC3NPJM5FQKLMLG2D6FXIVPYKHLOT74S272Q, seg_pos={0 0}, upload=317147]: Uploading piece... Jul 9 21:34:48.378880203 (396h24m23.149521528s) [upload=317147]: Next returned nil segment Jul 9 21:34:48.378896283 (396h24m23.149537618s) [upload=317147]: Waiting for error group managing segments... Jul 9 21:34:48.762202266 (396h24m23.532843602s) [noise=true , address=79.127.201.215:4325 , piece_id=FDMBJMIULICBLVZJGICJ6MNBA7ZDVFIKVMEMZ5DJYLAKZJW5XDIQ, seg_pos={0 0}, upload=317147]: Done uploading piece. err: <nil> Jul 9 21:34:48.881632293 (396h24m23.652273629s) [noise=true , address=77.90.31.21:4899 , piece_id=VBNJ3F4UOYGMP4UAGXXZRPN5TPL7M7UI4RL6X4QBW4A4PZAWRE2A, seg_pos={0 0}, upload=317147]: Done uploading piece. err: <nil> Jul 9 21:34:48.884941755 (396h24m23.65558309s) [noise=true , address=77.90.31.16:4767 , piece_id=5RADFBKOMBDSS4XGMOA5KRVKEIGL6P5PCOPV4LYMODBNBUK5BQXA, seg_pos={0 0}, upload=317147]: Done uploading piece. err: <nil> Jul 9 21:34:48.886518732 (396h24m23.657160097s) [noise=true , address=77.90.31.19:4858 , piece_id=AZWZNIRP66BWSPC7YOVWML5IRSK2NPP6ELPGUN6RGOQFPZVXRS4Q, seg_pos={0 0}, upload=317147]: Done uploading piece. err: <nil> Jul 9 21:34:48.897433825 (396h24m23.66807515s) [noise=true , address=77.90.31.13:4641 , piece_id=G7IIH7UASLFHSUIWEJJ5VKV4JVJQFC32UA2G6FVYVEMNUO25TTLQ, seg_pos={0 0}, upload=317147]: Done uploading piece. err: <nil> Jul 9 21:34:48.899635207 (396h24m23.670276542s) [noise=true , address=45.91.116.8:4072 , piece_id=NEZDKAR2K33S7OUQC5DO3655KIRENC57EJCM7GD24FAM26E2CXVA, seg_pos={0 0}, upload=317147]: Done uploading piece. err: <nil> Jul 9 21:34:48.918446064 (396h24m23.689087399s) [noise=true , address=77.90.31.12:4606 , piece_id=B5PX27N3ESF665PZ7VQGOLJG4SSFEAAXKX72FFIO734H7XEPAQIA, seg_pos={0 0}, upload=317147]: Done uploading piece. err: <nil> Jul 9 21:34:48.924574229 (396h24m23.695215564s) [noise=true , address=77.90.31.15:4715 , piece_id=6MF5JAAFOPUWDDZ6ZNWZP6I2K3JJ447KPJ2SVREXZD3NGH6PRZ7A, seg_pos={0 0}, upload=317147]: Done uploading piece. err: <nil> Jul 9 21:34:49.100685937 (396h24m23.871327272s) [noise=true , address=77.90.31.18:4818 , piece_id=JPFRUNH2OX4P5TYRSHXDXNH4LLQO2ERXT3RAOIARA2KYJMIQINKQ, seg_pos={0 0}, upload=317147]: Done uploading piece. err: <nil> Jul 9 21:34:54.784903656 (396h24m29.555544981s) [noise=true , address=79.127.219.38:4028 , piece_id=NHN7DPEQNHBJ2KW4CSTBZRR532YS4CDBSV3CZ2XIEJG6SAH53H3A, seg_pos={0 0}, upload=317147]: Done uploading piece. err: <nil> Jul 9 21:35:07.154890253 (396h24m41.925531588s) [noise=true , address=104.36.82.60:4345 , piece_id=UTOQODITF6P25BQHESX2LZ2UB7GEF5XR57ISEAT6KKUMSHC2WB4Q, seg_pos={0 0}, upload=317147]: Done uploading piece. err: <nil> Jul 9 21:35:18.264162596 (396h24m53.034803931s) [noise=true , address=104.36.82.8:4983 , piece_id=LT2M37IUHI6BNLR3VDQGT3SSV6SQC64CGSW3I253IAOIGUSYI7WA, seg_pos={0 0}, upload=317147]: Done uploading piece. err: <nil> Jul 9 21:35:42.938004716 (396h25m17.708646041s) [noise=true , address=45.91.95.4:4023 , piece_id=M62EJFMGKCWPF4VXF5EDS263ZTU6NGNJ3FD4G6GIMU4ADWHED7VQ, seg_pos={0 0}, upload=317147]: Done uploading piece. err: <nil> Jul 9 21:35:45.795933630 (396h25m20.566574965s) [noise=true , address=45.91.95.2:4024 , piece_id=KPVKM7NRIZVMLT5OGGVYM5LZF4EAU7TCRFGDWUDRRVRDAEZKZNTQ, seg_pos={0 0}, upload=317147]: Done uploading piece. err: ecclient: upload failed (node:19eaYeTz1sE3KG63t5zvcd1n4wUDSx97a36UFkPMaNdGzDr92y, address:45.91.95.2:4024): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:35:45.795983344 (396h25m20.566624669s) [noise=true , address=104.36.82.2:4747 , piece_id=RASK6SX5ZPMYSSET6T3UWXNWUNKOTD4TTHB4GLYGQHQQN5WEPUTQ, seg_pos={0 0}, upload=317147]: Done uploading piece. err: ecclient: upload failed (node:12mkHFDS6YCheghJdaJFybhcv9yhV4Acfs1RhnCW8yzA23neFRC, address:104.36.82.2:4747): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:35:45.796049642 (396h25m20.566691017s) [noise=true , address=45.91.95.3:4001 , piece_id=FQUAYGGU5U2FPEKULU3NUEIM3DMSE2C7THKNWQM2M6PPKNW5X7DQ, seg_pos={0 0}, upload=317147]: Done uploading piece. err: ecclient: upload failed (node:1L6dYFtTZvGGF9XsUFDHBBaSNPTnYjUVPT7VU5hfVpJpmpn3jL, address:45.91.95.3:4001): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:35:45.796110879 (396h25m20.566752204s) [noise=true , address=104.36.82.35:4598 , piece_id=CNBTKOZNCZM3LJTJIPWTY4UGUCA4TVCXMM4L337N6B5RR5F57GGQ, seg_pos={0 0}, upload=317147]: Done uploading piece. err: ecclient: upload failed (node:1hJMMH7ft1vWUYnQeugCHmuYGHd2wnMRm44KyMg997BZajJb3M, address:104.36.82.35:4598): protocol: write tcp 188.166.225.170:56542->104.36.82.35:4598: use of closed network connection; write tcp 188.166.225.170:56542->104.36.82.35:4598: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56542->104.36.82.35:4598: use of closed network connection Jul 9 21:35:45.796171414 (396h25m20.566812749s) [noise=true , address=45.91.95.5:4002 , piece_id=H75BYOM2MPHAXNPPGC3NPJM5FQKLMLG2D6FXIVPYKHLOT74S272Q, seg_pos={0 0}, upload=317147]: Done uploading piece. err: ecclient: upload failed (node:16MgCJPC1CuwnutbjFMaHYn7fmDzDoP7wyFYChw6m1WDKMLz4x, address:45.91.95.5:4002): protocol: write tcp 188.166.225.170:47794->45.91.95.5:4002: use of closed network connection; write tcp 188.166.225.170:47794->45.91.95.5:4002: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47794->45.91.95.5:4002: use of closed network connection Jul 9 21:35:45.796200740 (396h25m20.566842076s) [seg_pos={0 0}, upload=317147]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 21:35:46.024879517 (396h25m20.795520852s) [upload=317147]: Done uploading segments... Jul 9 21:35:46.024911598 (396h25m20.795552933s) [upload=317147]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 27.223022034s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 10 out of 60 is failed (threshold=3) [...XXX.........X......X....X..X.........X...............X..X] duration: 1m0.000492516s FAIL (2025-07-09 21:36:11 UTC) orbiter warning: context deadline exceeded Duration: 1m0.000514991s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X..X.....................................................] duration: 5.874173453s FAIL (2025-07-09 21:36:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.874195275s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [X..X..........X...X................X....X...............X..X] duration: 1m0.23047057s FAIL (2025-07-09 21:35:46 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 96h24m21.205048355s) [upload=317147]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 21:34:46.434424183 (396h24m21.205065518s) [upload=317147]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 21:34:46.879960451 (396h24m21.650601786s) [seg_pos={0 0}, upload=317147]: Begin upload segment... Jul 9 21:34:46.880016248 (396h24m21.650657583s) [seg_pos={0 0}, upload=317147]: Done begin upload segment. Jul 9 21:34:46.880037259 (396h24m21.650678594s) [noise=true , address=45.91.95.4:4023 , piece_id=M62EJFMGKCWPF4VXF5EDS263ZTU6NGNJ3FD4G6GIMU4ADWHED7VQ, seg_pos={0 0}, upload=317147]: Uploading piece... Jul 9 21:34:46.880104728 (396h24m21.650746063s) [noise=true , address=77.90.31.21:4899 , piece_id=VBNJ3F4UOYGMP4UAGXXZRPN5TPL7M7UI4RL6X4QBW4A4PZAWRE2A, seg_pos={0 0}, upload=317147]: Uploading piece... Jul 9 21:34:46.880436733 (396h24m21.651078058s) [noise=true , address=79.127.219.38:4028 , piece_id=NHN7DPEQNHBJ2KW4CSTBZRR532YS4CDBSV3CZ2XIEJG6SAH53H3A, seg_pos={0 0}, upload=317147]: Uploading piece... Jul 9 21:34:46.880595977 (396h24m21.651237312s) [noise=true , address=45.91.95.3:4001 , piece_id=FQUAYGGU5U2FPEKULU3NUEIM3DMSE2C7THKNWQM2M6PPKNW5X7DQ, seg_pos={0 0}, upload=317147]: Uploading piece... Jul 9 21:34:46.880705867 (396h24m21.651347202s) [noise=true , address=79.127.201.215:4325 , piece_id=FDMBJMIULICBLVZJGICJ6MNBA7ZDVFIKVMEMZ5DJYLAKZJW5XDIQ, seg_pos={0 0}, upload=317147]: Uploading piece... Jul 9 21:34:46.880920659 (396h24m21.651561994s) [noise=true , address=45.91.116.8:4072 , piece_id=NEZDKAR2K33S7OUQC5DO3655KIRENC57EJCM7GD24FAM26E2CXVA, seg_pos={0 0}, upload=317147]: Uploading piece... Jul 9 21:34:46.881567216 (396h24m21.652208551s) [noise=true , address=104.36.82.60:4345 , piece_id=UTOQODITF6P25BQHESX2LZ2UB7GEF5XR57ISEAT6KKUMSHC2WB4Q, seg_pos={0 0}, upload=317147]: Uploading piece... Jul 9 21:34:46.881631028 (396h24m21.652272353s) [noise=true , address=45.91.95.2:4024 , piece_id=KPVKM7NRIZVMLT5OGGVYM5LZF4EAU7TCRFGDWUDRRVRDAEZKZNTQ, seg_pos={0 0}, upload=317147]: Uploading piece... Jul 9 21:34:46.882098382 (396h24m21.652739717s) [noise=true , address=77.90.31.16:4767 , piece_id=5RADFBKOMBDSS4XGMOA5KRVKEIGL6P5PCOPV4LYMODBNBUK5BQXA, seg_pos={0 0}, upload=317147]: Uploading piece... Jul 9 21:34:46.882386944 (396h24m21.653028279s) [noise=true , address=77.90.31.12:4606 , piece_id=B5PX27N3ESF665PZ7VQGOLJG4SSFEAAXKX72FFIO734H7XEPAQIA, seg_pos={0 0}, upload=317147]: Uploading piece... Jul 9 21:34:46.882987102 (396h24m21.653628437s) [noise=true , address=77.90.31.18:4818 , piece_id=JPFRUNH2OX4P5TYRSHXDXNH4LLQO2ERXT3RAOIARA2KYJMIQINKQ, seg_pos={0 0}, upload=317147]: Uploading piece... Jul 9 21:34:46.883223445 (396h24m21.65386477s) [noise=true , address=77.90.31.19:4858 , piece_id=AZWZNIRP66BWSPC7YOVWML5IRSK2NPP6ELPGUN6RGOQFPZVXRS4Q, seg_pos={0 0}, upload=317147]: Uploading piece... Jul 9 21:34:46.883807061 (396h24m21.654448396s) [noise=true , address=77.90.31.15:4715 , piece_id=6MF5JAAFOPUWDDZ6ZNWZP6I2K3JJ447KPJ2SVREXZD3NGH6PRZ7A, seg_pos={0 0}, upload=317147]: Uploading piece... Jul 9 21:34:46.883955415 (396h24m21.65459675s) [noise=true , address=77.90.31.13:4641 , piece_id=G7IIH7UASLFHSUIWEJJ5VKV4JVJQFC32UA2G6FVYVEMNUO25TTLQ, seg_pos={0 0}, upload=317147]: Uploading piece... Jul 9 21:34:46.884316034 (396h24m21.65495737s) [noise=true , address=104.36.82.2:4747 , piece_id=RASK6SX5ZPMYSSET6T3UWXNWUNKOTD4TTHB4GLYGQHQQN5WEPUTQ, seg_pos={0 0}, upload=317147]: Uploading piece... Jul 9 21:34:46.884471512 (396h24m21.655112837s) [noise=true , address=104.36.82.35:4598 , piece_id=CNBTKOZNCZM3LJTJIPWTY4UGUCA4TVCXMM4L337N6B5RR5F57GGQ, seg_pos={0 0}, upload=317147]: Uploading piece... Jul 9 21:34:46.884786304 (396h24m21.655427639s) [noise=true , address=104.36.82.8:4983 , piece_id=LT2M37IUHI6BNLR3VDQGT3SSV6SQC64CGSW3I253IAOIGUSYI7WA, seg_pos={0 0}, upload=317147]: Uploading piece... Jul 9 21:34:46.885151844 (396h24m21.655793179s) [noise=true , address=45.91.95.5:4002 , piece_id=H75BYOM2MPHAXNPPGC3NPJM5FQKLMLG2D6FXIVPYKHLOT74S272Q, seg_pos={0 0}, upload=317147]: Uploading piece... Jul 9 21:34:48.378880203 (396h24m23.149521528s) [upload=317147]: Next returned nil segment Jul 9 21:34:48.378896283 (396h24m23.149537618s) [upload=317147]: Waiting for error group managing segments... Jul 9 21:34:48.762202266 (396h24m23.532843602s) [noise=true , address=79.127.201.215:4325 , piece_id=FDMBJMIULICBLVZJGICJ6MNBA7ZDVFIKVMEMZ5DJYLAKZJW5XDIQ, seg_pos={0 0}, upload=317147]: Done uploading piece. err: <nil> Jul 9 21:34:48.881632293 (396h24m23.652273629s) [noise=true , address=77.90.31.21:4899 , piece_id=VBNJ3F4UOYGMP4UAGXXZRPN5TPL7M7UI4RL6X4QBW4A4PZAWRE2A, seg_pos={0 0}, upload=317147]: Done uploading piece. err: <nil> Jul 9 21:34:48.884941755 (396h24m23.65558309s) [noise=true , address=77.90.31.16:4767 , piece_id=5RADFBKOMBDSS4XGMOA5KRVKEIGL6P5PCOPV4LYMODBNBUK5BQXA, seg_pos={0 0}, upload=317147]: Done uploading piece. err: <nil> Jul 9 21:34:48.886518732 (396h24m23.657160097s) [noise=true , address=77.90.31.19:4858 , piece_id=AZWZNIRP66BWSPC7YOVWML5IRSK2NPP6ELPGUN6RGOQFPZVXRS4Q, seg_pos={0 0}, upload=317147]: Done uploading piece. err: <nil> Jul 9 21:34:48.897433825 (396h24m23.66807515s) [noise=true , address=77.90.31.13:4641 , piece_id=G7IIH7UASLFHSUIWEJJ5VKV4JVJQFC32UA2G6FVYVEMNUO25TTLQ, seg_pos={0 0}, upload=317147]: Done uploading piece. err: <nil> Jul 9 21:34:48.899635207 (396h24m23.670276542s) [noise=true , address=45.91.116.8:4072 , piece_id=NEZDKAR2K33S7OUQC5DO3655KIRENC57EJCM7GD24FAM26E2CXVA, seg_pos={0 0}, upload=317147]: Done uploading piece. err: <nil> Jul 9 21:34:48.918446064 (396h24m23.689087399s) [noise=true , address=77.90.31.12:4606 , piece_id=B5PX27N3ESF665PZ7VQGOLJG4SSFEAAXKX72FFIO734H7XEPAQIA, seg_pos={0 0}, upload=317147]: Done uploading piece. err: <nil> Jul 9 21:34:48.924574229 (396h24m23.695215564s) [noise=true , address=77.90.31.15:4715 , piece_id=6MF5JAAFOPUWDDZ6ZNWZP6I2K3JJ447KPJ2SVREXZD3NGH6PRZ7A, seg_pos={0 0}, upload=317147]: Done uploading piece. err: <nil> Jul 9 21:34:49.100685937 (396h24m23.871327272s) [noise=true , address=77.90.31.18:4818 , piece_id=JPFRUNH2OX4P5TYRSHXDXNH4LLQO2ERXT3RAOIARA2KYJMIQINKQ, seg_pos={0 0}, upload=317147]: Done uploading piece. err: <nil> Jul 9 21:34:54.784903656 (396h24m29.555544981s) [noise=true , address=79.127.219.38:4028 , piece_id=NHN7DPEQNHBJ2KW4CSTBZRR532YS4CDBSV3CZ2XIEJG6SAH53H3A, seg_pos={0 0}, upload=317147]: Done uploading piece. err: <nil> Jul 9 21:35:07.154890253 (396h24m41.925531588s) [noise=true , address=104.36.82.60:4345 , piece_id=UTOQODITF6P25BQHESX2LZ2UB7GEF5XR57ISEAT6KKUMSHC2WB4Q, seg_pos={0 0}, upload=317147]: Done uploading piece. err: <nil> Jul 9 21:35:18.264162596 (396h24m53.034803931s) [noise=true , address=104.36.82.8:4983 , piece_id=LT2M37IUHI6BNLR3VDQGT3SSV6SQC64CGSW3I253IAOIGUSYI7WA, seg_pos={0 0}, upload=317147]: Done uploading piece. err: <nil> Jul 9 21:35:42.938004716 (396h25m17.708646041s) [noise=true , address=45.91.95.4:4023 , piece_id=M62EJFMGKCWPF4VXF5EDS263ZTU6NGNJ3FD4G6GIMU4ADWHED7VQ, seg_pos={0 0}, upload=317147]: Done uploading piece. err: <nil> Jul 9 21:35:45.795933630 (396h25m20.566574965s) [noise=true , address=45.91.95.2:4024 , piece_id=KPVKM7NRIZVMLT5OGGVYM5LZF4EAU7TCRFGDWUDRRVRDAEZKZNTQ, seg_pos={0 0}, upload=317147]: Done uploading piece. err: ecclient: upload failed (node:19eaYeTz1sE3KG63t5zvcd1n4wUDSx97a36UFkPMaNdGzDr92y, address:45.91.95.2:4024): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:35:45.795983344 (396h25m20.566624669s) [noise=true , address=104.36.82.2:4747 , piece_id=RASK6SX5ZPMYSSET6T3UWXNWUNKOTD4TTHB4GLYGQHQQN5WEPUTQ, seg_pos={0 0}, upload=317147]: Done uploading piece. err: ecclient: upload failed (node:12mkHFDS6YCheghJdaJFybhcv9yhV4Acfs1RhnCW8yzA23neFRC, address:104.36.82.2:4747): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:35:45.796049642 (396h25m20.566691017s) [noise=true , address=45.91.95.3:4001 , piece_id=FQUAYGGU5U2FPEKULU3NUEIM3DMSE2C7THKNWQM2M6PPKNW5X7DQ, seg_pos={0 0}, upload=317147]: Done uploading piece. err: ecclient: upload failed (node:1L6dYFtTZvGGF9XsUFDHBBaSNPTnYjUVPT7VU5hfVpJpmpn3jL, address:45.91.95.3:4001): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:35:45.796110879 (396h25m20.566752204s) [noise=true , address=104.36.82.35:4598 , piece_id=CNBTKOZNCZM3LJTJIPWTY4UGUCA4TVCXMM4L337N6B5RR5F57GGQ, seg_pos={0 0}, upload=317147]: Done uploading piece. err: ecclient: upload failed (node:1hJMMH7ft1vWUYnQeugCHmuYGHd2wnMRm44KyMg997BZajJb3M, address:104.36.82.35:4598): protocol: write tcp 188.166.225.170:56542->104.36.82.35:4598: use of closed network connection; write tcp 188.166.225.170:56542->104.36.82.35:4598: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56542->104.36.82.35:4598: use of closed network connection Jul 9 21:35:45.796171414 (396h25m20.566812749s) [noise=true , address=45.91.95.5:4002 , piece_id=H75BYOM2MPHAXNPPGC3NPJM5FQKLMLG2D6FXIVPYKHLOT74S272Q, seg_pos={0 0}, upload=317147]: Done uploading piece. err: ecclient: upload failed (node:16MgCJPC1CuwnutbjFMaHYn7fmDzDoP7wyFYChw6m1WDKMLz4x, address:45.91.95.5:4002): protocol: write tcp 188.166.225.170:47794->45.91.95.5:4002: use of closed network connection; write tcp 188.166.225.170:47794->45.91.95.5:4002: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47794->45.91.95.5:4002: use of closed network connection Jul 9 21:35:45.796200740 (396h25m20.566842076s) [seg_pos={0 0}, upload=317147]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 21:35:46.024879517 (396h25m20.795520852s) [upload=317147]: Done uploading segments... Jul 9 21:35:46.024911598 (396h25m20.795552933s) [upload=317147]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.230482693s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X..X....................................................] duration: 5.874419233s FAIL (2025-07-09 21:35:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.874440093s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....X..X...................................................] duration: 6.343709318s FAIL (2025-07-09 21:34:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.343730237s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [.X..X..........X...X................X....X...............X..] duration: 31.95993617s FAIL (2025-07-09 21:34:45 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== one starting upload Jul 9 21:32:02.509558713 (396h21m37.280200068s) [upload=317114]: Uploading segments... Jul 9 21:32:02.509643425 (396h21m37.28028476s) [upload=317114]: Got next segment. Inline: false Jul 9 21:32:02.509672832 (396h21m37.280314167s) [upload=317114]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 21:32:02.509677090 (396h21m37.280318425s) [upload=317114]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 21:32:02.954858160 (396h21m37.725499495s) [seg_pos={0 0}, upload=317114]: Begin upload segment... Jul 9 21:32:02.954898557 (396h21m37.725539892s) [seg_pos={0 0}, upload=317114]: Done begin upload segment. Jul 9 21:32:02.954924857 (396h21m37.725566192s) [noise=true , address=77.90.31.12:4600 , piece_id=5RA324WSM6Y37EOS3ODKXDOQIWVHCC5BYQPKQJB7ZDAHAQTHHLWQ, seg_pos={0 0}, upload=317114]: Uploading piece... Jul 9 21:32:02.954945667 (396h21m37.725587002s) [noise=true , address=77.90.31.14:4691 , piece_id=UC4TVYEX4W2OXUQNFKAY4IGQM6TVPUY47K5RYQFMLS7ZORXTBBUA, seg_pos={0 0}, upload=317114]: Uploading piece... Jul 9 21:32:02.955440825 (396h21m37.72608216s) [noise=true , address=45.91.95.2:4031 , piece_id=ATCD4TYZKYAPUHYPZIJW5A3LKGSAJCP2X6D7DLB34SIMGTPNYSYQ, seg_pos={0 0}, upload=317114]: Uploading piece... Jul 9 21:32:02.955447547 (396h21m37.726088882s) [noise=true , address=45.91.95.5:4022 , piece_id=U7WC2HGN7JYFDXSLQ6QFWXPSHAVENXGPRPVRUS55IUXNT44GY7DA, seg_pos={0 0}, upload=317114]: Uploading piece... Jul 9 21:32:02.955714017 (396h21m37.726355352s) [noise=true , address=104.36.82.50:4042 , piece_id=IZIYWURR3H3MGCKH5CM2F5WRE6BVMJL23HDAAXGF3MYLJ2QO3XXA, seg_pos={0 0}, upload=317114]: Uploading piece... Jul 9 21:32:02.955807856 (396h21m37.726449192s) [noise=true , address=77.90.31.18:4830 , piece_id=RF6VQI7G3KVYBC55GFVMOB6THWBJDPMESEJMPFPWV2BJYHVUFXNQ, seg_pos={0 0}, upload=317114]: Uploading piece... Jul 9 21:32:02.955960779 (396h21m37.726602115s) [noise=true , address=45.91.95.4:4004 , piece_id=HPHOLVAJBNTZG3VZ6EJUYIXYRFKUDXEQ5SQCONIEHB5DGRL4X7WQ, seg_pos={0 0}, upload=317114]: Uploading piece... Jul 9 21:32:02.956041003 (396h21m37.726682338s) [noise=true , address=77.90.31.17:4788 , piece_id=QAOJLNZOVYLX3ZTIA3O6ADDFFSJAVOR5FG4WEAKBFXAU72ZQGJ5A, seg_pos={0 0}, upload=317114]: Uploading piece... Jul 9 21:32:02.956186792 (396h21m37.726828127s) [noise=true , address=104.36.82.5:4862 , piece_id=UZEXRCCCWUCP2FW2KOSGSCB6NPNLK4HGLTX5HNNE7VR7TBYJ2ATA, seg_pos={0 0}, upload=317114]: Uploading piece... Jul 9 21:32:02.956254751 (396h21m37.726896086s) [noise=true , address=104.36.82.51:4068 , piece_id=6FDENH2R2DDCEYP4HE64YQSCDMMX2PPYI25AJDGQEOK5GQMIHX3A, seg_pos={0 0}, upload=317114]: Uploading piece... Jul 9 21:32:02.956459654 (396h21m37.727100989s) [noise=true , address=77.90.31.16:4751 , piece_id=SOV5AWXE2NMSJURQTAQA4SORO6EDQRXME6RTHLLN6UEEZEX6ZBVQ, seg_pos={0 0}, upload=317114]: Uploading piece... Jul 9 21:32:02.956534187 (396h21m37.727175522s) [noise=true , address=64.49.13.169:4706 , piece_id=HVKNHMYI2DCSJKOYSHOFDYBIUCQ3CEAEG3IUAE4OSJXMKY7ZOKFA, seg_pos={0 0}, upload=317114]: Uploading piece... Jul 9 21:32:02.956686177 (396h21m37.727327512s) [noise=true , address=77.90.31.19:4874 , piece_id=3PZ7OYOGSBZOJSAZ2SH3REVPYQHREQILPUYA2NLCTIYPGQCCTDHQ, seg_pos={0 0}, upload=317114]: Uploading piece... Jul 9 21:32:02.956812068 (396h21m37.727453403s) [noise=true , address=45.91.95.3:4031 , piece_id=MBJUWA5UQ7GH4ZIGU5ZPBCY3HHXXKFPMRXTKL63EJM3AHXWMRDQA, seg_pos={0 0}, upload=317114]: Uploading piece... Jul 9 21:32:02.957105619 (396h21m37.727746954s) [noise=true , address=77.90.31.11:4584 , piece_id=TAA3NQL5IUKVQBBRL352KVYTXO44EZEKX3INIB2DBM5CQPTFBJSA, seg_pos={0 0}, upload=317114]: Uploading piece... Jul 9 21:32:02.957385915 (396h21m37.72802725s) [noise=true , address=77.90.31.18:4832 , piece_id=7WGWSIBCKZAXO4I6FKGXSZJJ6L5UZEYXYY6R24P5F42XR4XUBQNQ, seg_pos={0 0}, upload=317114]: Uploading piece... Jul 9 21:32:02.957657074 (396h21m37.728298409s) [noise=true , address=77.90.31.13:4632 , piece_id=FB76GO7TALNZHNECN3UOKQHQPA4H3VCX4TI2X6CG4VZHQ7DULYOQ, seg_pos={0 0}, upload=317114]: Uploading piece... Jul 9 21:32:02.958236543 (396h21m37.728877878s) [noise=true , address=79.127.219.38:4024 , piece_id=4HDQ6PB2G7NKABGKQFUUQQ2XMDATAS3MEQYE27UT4ABNU3O6X7DQ, seg_pos={0 0}, upload=317114]: Uploading piece... Jul 9 21:32:04.069772770 (396h21m38.840414105s) [upload=317114]: Next returned nil segment Jul 9 21:32:04.069791056 (396h21m38.840432391s) [upload=317114]: Waiting for error group managing segments... Jul 9 21:32:04.947577444 (396h21m39.718218779s) [noise=true , address=77.90.31.13:4632 , piece_id=FB76GO7TALNZHNECN3UOKQHQPA4H3VCX4TI2X6CG4VZHQ7DULYOQ, seg_pos={0 0}, upload=317114]: Done uploading piece. err: <nil> Jul 9 21:32:04.951394947 (396h21m39.722036282s) [noise=true , address=77.90.31.18:4832 , piece_id=7WGWSIBCKZAXO4I6FKGXSZJJ6L5UZEYXYY6R24P5F42XR4XUBQNQ, seg_pos={0 0}, upload=317114]: Done uploading piece. err: <nil> Jul 9 21:32:04.952763405 (396h21m39.72340474s) [noise=true , address=77.90.31.12:4600 , piece_id=5RA324WSM6Y37EOS3ODKXDOQIWVHCC5BYQPKQJB7ZDAHAQTHHLWQ, seg_pos={0 0}, upload=317114]: Done uploading piece. err: <nil> Jul 9 21:32:04.954245150 (396h21m39.724886485s) [noise=true , address=77.90.31.18:4830 , piece_id=RF6VQI7G3KVYBC55GFVMOB6THWBJDPMESEJMPFPWV2BJYHVUFXNQ, seg_pos={0 0}, upload=317114]: Done uploading piece. err: <nil> Jul 9 21:32:04.956787544 (396h21m39.727428869s) [noise=true , address=77.90.31.19:4874 , piece_id=3PZ7OYOGSBZOJSAZ2SH3REVPYQHREQILPUYA2NLCTIYPGQCCTDHQ, seg_pos={0 0}, upload=317114]: Done uploading piece. err: <nil> Jul 9 21:32:04.960355681 (396h21m39.730997016s) [noise=true , address=77.90.31.16:4751 , piece_id=SOV5AWXE2NMSJURQTAQA4SORO6EDQRXME6RTHLLN6UEEZEX6ZBVQ, seg_pos={0 0}, upload=317114]: Done uploading piece. err: <nil> Jul 9 21:32:04.963274956 (396h21m39.733916291s) [noise=true , address=77.90.31.11:4584 , piece_id=TAA3NQL5IUKVQBBRL352KVYTXO44EZEKX3INIB2DBM5CQPTFBJSA, seg_pos={0 0}, upload=317114]: Done uploading piece. err: <nil> Jul 9 21:32:04.966230130 (396h21m39.736871465s) [noise=true , address=77.90.31.14:4691 , piece_id=UC4TVYEX4W2OXUQNFKAY4IGQM6TVPUY47K5RYQFMLS7ZORXTBBUA, seg_pos={0 0}, upload=317114]: Done uploading piece. err: <nil> Jul 9 21:32:04.967618355 (396h21m39.73825969s) [noise=true , address=77.90.31.17:4788 , piece_id=QAOJLNZOVYLX3ZTIA3O6ADDFFSJAVOR5FG4WEAKBFXAU72ZQGJ5A, seg_pos={0 0}, upload=317114]: Done uploading piece. err: <nil> Jul 9 21:32:08.952512712 (396h21m43.723154047s) [noise=true , address=79.127.219.38:4024 , piece_id=4HDQ6PB2G7NKABGKQFUUQQ2XMDATAS3MEQYE27UT4ABNU3O6X7DQ, seg_pos={0 0}, upload=317114]: Done uploading piece. err: <nil> Jul 9 21:32:26.893189191 (396h22m1.663830526s) [noise=true , address=104.36.82.5:4862 , piece_id=UZEXRCCCWUCP2FW2KOSGSCB6NPNLK4HGLTX5HNNE7VR7TBYJ2ATA, seg_pos={0 0}, upload=317114]: Done uploading piece. err: <nil> Jul 9 21:32:38.593603255 (396h22m13.3642446s) [noise=true , address=104.36.82.50:4042 , piece_id=IZIYWURR3H3MGCKH5CM2F5WRE6BVMJL23HDAAXGF3MYLJ2QO3XXA, seg_pos={0 0}, upload=317114]: Done uploading piece. err: <nil> Jul 9 21:32:47.819908973 (396h22m22.590550308s) [noise=true , address=45.91.95.3:4031 , piece_id=MBJUWA5UQ7GH4ZIGU5ZPBCY3HHXXKFPMRXTKL63EJM3AHXWMRDQA, seg_pos={0 0}, upload=317114]: Done uploading piece. err: <nil> Jul 9 21:33:01.863729242 (396h22m36.634370577s) [noise=true , address=45.91.95.5:4022 , piece_id=U7WC2HGN7JYFDXSLQ6QFWXPSHAVENXGPRPVRUS55IUXNT44GY7DA, seg_pos={0 0}, upload=317114]: Done uploading piece. err: ecclient: upload failed (node:12X8Et5FNyAsE7ifEckfQemZkfnvYUuZ2Rn2xdxNxhmCmuDST2M, address:45.91.95.5:4022): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:33:01.863808243 (396h22m36.634449578s) [noise=true , address=104.36.82.51:4068 , piece_id=6FDENH2R2DDCEYP4HE64YQSCDMMX2PPYI25AJDGQEOK5GQMIHX3A, seg_pos={0 0}, upload=317114]: Done uploading piece. err: ecclient: upload failed (node:12Lh9dYpHCxbs7gpZ4Xt7os5Gm5Nco6jrNW1V1FHddVUEMkFfPY, address:104.36.82.51:4068): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:33:01.863879039 (396h22m36.634520384s) [noise=true , address=64.49.13.169:4706 , piece_id=HVKNHMYI2DCSJKOYSHOFDYBIUCQ3CEAEG3IUAE4OSJXMKY7ZOKFA, seg_pos={0 0}, upload=317114]: Done uploading piece. err: ecclient: upload failed (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7, address:64.49.13.169:4706): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:33:01.863924856 (396h22m36.634566191s) [noise=true , address=45.91.95.4:4004 , piece_id=HPHOLVAJBNTZG3VZ6EJUYIXYRFKUDXEQ5SQCONIEHB5DGRL4X7WQ, seg_pos={0 0}, upload=317114]: Done uploading piece. err: ecclient: upload failed (node:12ZeqpAQ3qRdFvheAHDMKU3znEAiv8V3jJbABGBjWsNk3ZGMNM6, address:45.91.95.4:4004): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:33:01.863932541 (396h22m36.634573876s) [noise=true , address=45.91.95.2:4031 , piece_id=ATCD4TYZKYAPUHYPZIJW5A3LKGSAJCP2X6D7DLB34SIMGTPNYSYQ, seg_pos={0 0}, upload=317114]: Done uploading piece. err: ecclient: upload failed (node:12Mwd9RRYqm3GpbmT4B2KnAd2bXKq7HopJCk3E3W4A79Hh2eypL, address:45.91.95.2:4031): protocol: write tcp 188.166.225.170:34462->45.91.95.2:4031: use of closed network connection; write tcp 188.166.225.170:34462->45.91.95.2:4031: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34462->45.91.95.2:4031: use of closed network connection Jul 9 21:33:01.863977877 (396h22m36.634619222s) [seg_pos={0 0}, upload=317114]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 21:33:02.095730796 (396h22m36.866372141s) [upload=317114]: Done uploading segments... Jul 9 21:33:02.095763348 (396h22m36.866404693s) [upload=317114]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 31.959958584s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 10 out of 60 is failed (threshold=3) [X...XXX.........X......X....X..X.........X...............X..] duration: 34.941805148s FAIL (2025-07-09 21:34:45 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 34.941823192s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......X..X..................................................] duration: 6.159290494s FAIL (2025-07-09 21:33:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.15931422s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [..X..X..........X...X................X....X...............X.] duration: 51.254122022s FAIL (2025-07-09 21:33:53 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== one starting upload Jul 9 21:32:02.509558713 (396h21m37.280200068s) [upload=317114]: Uploading segments... Jul 9 21:32:02.509643425 (396h21m37.28028476s) [upload=317114]: Got next segment. Inline: false Jul 9 21:32:02.509672832 (396h21m37.280314167s) [upload=317114]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 21:32:02.509677090 (396h21m37.280318425s) [upload=317114]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 21:32:02.954858160 (396h21m37.725499495s) [seg_pos={0 0}, upload=317114]: Begin upload segment... Jul 9 21:32:02.954898557 (396h21m37.725539892s) [seg_pos={0 0}, upload=317114]: Done begin upload segment. Jul 9 21:32:02.954924857 (396h21m37.725566192s) [noise=true , address=77.90.31.12:4600 , piece_id=5RA324WSM6Y37EOS3ODKXDOQIWVHCC5BYQPKQJB7ZDAHAQTHHLWQ, seg_pos={0 0}, upload=317114]: Uploading piece... Jul 9 21:32:02.954945667 (396h21m37.725587002s) [noise=true , address=77.90.31.14:4691 , piece_id=UC4TVYEX4W2OXUQNFKAY4IGQM6TVPUY47K5RYQFMLS7ZORXTBBUA, seg_pos={0 0}, upload=317114]: Uploading piece... Jul 9 21:32:02.955440825 (396h21m37.72608216s) [noise=true , address=45.91.95.2:4031 , piece_id=ATCD4TYZKYAPUHYPZIJW5A3LKGSAJCP2X6D7DLB34SIMGTPNYSYQ, seg_pos={0 0}, upload=317114]: Uploading piece... Jul 9 21:32:02.955447547 (396h21m37.726088882s) [noise=true , address=45.91.95.5:4022 , piece_id=U7WC2HGN7JYFDXSLQ6QFWXPSHAVENXGPRPVRUS55IUXNT44GY7DA, seg_pos={0 0}, upload=317114]: Uploading piece... Jul 9 21:32:02.955714017 (396h21m37.726355352s) [noise=true , address=104.36.82.50:4042 , piece_id=IZIYWURR3H3MGCKH5CM2F5WRE6BVMJL23HDAAXGF3MYLJ2QO3XXA, seg_pos={0 0}, upload=317114]: Uploading piece... Jul 9 21:32:02.955807856 (396h21m37.726449192s) [noise=true , address=77.90.31.18:4830 , piece_id=RF6VQI7G3KVYBC55GFVMOB6THWBJDPMESEJMPFPWV2BJYHVUFXNQ, seg_pos={0 0}, upload=317114]: Uploading piece... Jul 9 21:32:02.955960779 (396h21m37.726602115s) [noise=true , address=45.91.95.4:4004 , piece_id=HPHOLVAJBNTZG3VZ6EJUYIXYRFKUDXEQ5SQCONIEHB5DGRL4X7WQ, seg_pos={0 0}, upload=317114]: Uploading piece... Jul 9 21:32:02.956041003 (396h21m37.726682338s) [noise=true , address=77.90.31.17:4788 , piece_id=QAOJLNZOVYLX3ZTIA3O6ADDFFSJAVOR5FG4WEAKBFXAU72ZQGJ5A, seg_pos={0 0}, upload=317114]: Uploading piece... Jul 9 21:32:02.956186792 (396h21m37.726828127s) [noise=true , address=104.36.82.5:4862 , piece_id=UZEXRCCCWUCP2FW2KOSGSCB6NPNLK4HGLTX5HNNE7VR7TBYJ2ATA, seg_pos={0 0}, upload=317114]: Uploading piece... Jul 9 21:32:02.956254751 (396h21m37.726896086s) [noise=true , address=104.36.82.51:4068 , piece_id=6FDENH2R2DDCEYP4HE64YQSCDMMX2PPYI25AJDGQEOK5GQMIHX3A, seg_pos={0 0}, upload=317114]: Uploading piece... Jul 9 21:32:02.956459654 (396h21m37.727100989s) [noise=true , address=77.90.31.16:4751 , piece_id=SOV5AWXE2NMSJURQTAQA4SORO6EDQRXME6RTHLLN6UEEZEX6ZBVQ, seg_pos={0 0}, upload=317114]: Uploading piece... Jul 9 21:32:02.956534187 (396h21m37.727175522s) [noise=true , address=64.49.13.169:4706 , piece_id=HVKNHMYI2DCSJKOYSHOFDYBIUCQ3CEAEG3IUAE4OSJXMKY7ZOKFA, seg_pos={0 0}, upload=317114]: Uploading piece... Jul 9 21:32:02.956686177 (396h21m37.727327512s) [noise=true , address=77.90.31.19:4874 , piece_id=3PZ7OYOGSBZOJSAZ2SH3REVPYQHREQILPUYA2NLCTIYPGQCCTDHQ, seg_pos={0 0}, upload=317114]: Uploading piece... Jul 9 21:32:02.956812068 (396h21m37.727453403s) [noise=true , address=45.91.95.3:4031 , piece_id=MBJUWA5UQ7GH4ZIGU5ZPBCY3HHXXKFPMRXTKL63EJM3AHXWMRDQA, seg_pos={0 0}, upload=317114]: Uploading piece... Jul 9 21:32:02.957105619 (396h21m37.727746954s) [noise=true , address=77.90.31.11:4584 , piece_id=TAA3NQL5IUKVQBBRL352KVYTXO44EZEKX3INIB2DBM5CQPTFBJSA, seg_pos={0 0}, upload=317114]: Uploading piece... Jul 9 21:32:02.957385915 (396h21m37.72802725s) [noise=true , address=77.90.31.18:4832 , piece_id=7WGWSIBCKZAXO4I6FKGXSZJJ6L5UZEYXYY6R24P5F42XR4XUBQNQ, seg_pos={0 0}, upload=317114]: Uploading piece... Jul 9 21:32:02.957657074 (396h21m37.728298409s) [noise=true , address=77.90.31.13:4632 , piece_id=FB76GO7TALNZHNECN3UOKQHQPA4H3VCX4TI2X6CG4VZHQ7DULYOQ, seg_pos={0 0}, upload=317114]: Uploading piece... Jul 9 21:32:02.958236543 (396h21m37.728877878s) [noise=true , address=79.127.219.38:4024 , piece_id=4HDQ6PB2G7NKABGKQFUUQQ2XMDATAS3MEQYE27UT4ABNU3O6X7DQ, seg_pos={0 0}, upload=317114]: Uploading piece... Jul 9 21:32:04.069772770 (396h21m38.840414105s) [upload=317114]: Next returned nil segment Jul 9 21:32:04.069791056 (396h21m38.840432391s) [upload=317114]: Waiting for error group managing segments... Jul 9 21:32:04.947577444 (396h21m39.718218779s) [noise=true , address=77.90.31.13:4632 , piece_id=FB76GO7TALNZHNECN3UOKQHQPA4H3VCX4TI2X6CG4VZHQ7DULYOQ, seg_pos={0 0}, upload=317114]: Done uploading piece. err: <nil> Jul 9 21:32:04.951394947 (396h21m39.722036282s) [noise=true , address=77.90.31.18:4832 , piece_id=7WGWSIBCKZAXO4I6FKGXSZJJ6L5UZEYXYY6R24P5F42XR4XUBQNQ, seg_pos={0 0}, upload=317114]: Done uploading piece. err: <nil> Jul 9 21:32:04.952763405 (396h21m39.72340474s) [noise=true , address=77.90.31.12:4600 , piece_id=5RA324WSM6Y37EOS3ODKXDOQIWVHCC5BYQPKQJB7ZDAHAQTHHLWQ, seg_pos={0 0}, upload=317114]: Done uploading piece. err: <nil> Jul 9 21:32:04.954245150 (396h21m39.724886485s) [noise=true , address=77.90.31.18:4830 , piece_id=RF6VQI7G3KVYBC55GFVMOB6THWBJDPMESEJMPFPWV2BJYHVUFXNQ, seg_pos={0 0}, upload=317114]: Done uploading piece. err: <nil> Jul 9 21:32:04.956787544 (396h21m39.727428869s) [noise=true , address=77.90.31.19:4874 , piece_id=3PZ7OYOGSBZOJSAZ2SH3REVPYQHREQILPUYA2NLCTIYPGQCCTDHQ, seg_pos={0 0}, upload=317114]: Done uploading piece. err: <nil> Jul 9 21:32:04.960355681 (396h21m39.730997016s) [noise=true , address=77.90.31.16:4751 , piece_id=SOV5AWXE2NMSJURQTAQA4SORO6EDQRXME6RTHLLN6UEEZEX6ZBVQ, seg_pos={0 0}, upload=317114]: Done uploading piece. err: <nil> Jul 9 21:32:04.963274956 (396h21m39.733916291s) [noise=true , address=77.90.31.11:4584 , piece_id=TAA3NQL5IUKVQBBRL352KVYTXO44EZEKX3INIB2DBM5CQPTFBJSA, seg_pos={0 0}, upload=317114]: Done uploading piece. err: <nil> Jul 9 21:32:04.966230130 (396h21m39.736871465s) [noise=true , address=77.90.31.14:4691 , piece_id=UC4TVYEX4W2OXUQNFKAY4IGQM6TVPUY47K5RYQFMLS7ZORXTBBUA, seg_pos={0 0}, upload=317114]: Done uploading piece. err: <nil> Jul 9 21:32:04.967618355 (396h21m39.73825969s) [noise=true , address=77.90.31.17:4788 , piece_id=QAOJLNZOVYLX3ZTIA3O6ADDFFSJAVOR5FG4WEAKBFXAU72ZQGJ5A, seg_pos={0 0}, upload=317114]: Done uploading piece. err: <nil> Jul 9 21:32:08.952512712 (396h21m43.723154047s) [noise=true , address=79.127.219.38:4024 , piece_id=4HDQ6PB2G7NKABGKQFUUQQ2XMDATAS3MEQYE27UT4ABNU3O6X7DQ, seg_pos={0 0}, upload=317114]: Done uploading piece. err: <nil> Jul 9 21:32:26.893189191 (396h22m1.663830526s) [noise=true , address=104.36.82.5:4862 , piece_id=UZEXRCCCWUCP2FW2KOSGSCB6NPNLK4HGLTX5HNNE7VR7TBYJ2ATA, seg_pos={0 0}, upload=317114]: Done uploading piece. err: <nil> Jul 9 21:32:38.593603255 (396h22m13.3642446s) [noise=true , address=104.36.82.50:4042 , piece_id=IZIYWURR3H3MGCKH5CM2F5WRE6BVMJL23HDAAXGF3MYLJ2QO3XXA, seg_pos={0 0}, upload=317114]: Done uploading piece. err: <nil> Jul 9 21:32:47.819908973 (396h22m22.590550308s) [noise=true , address=45.91.95.3:4031 , piece_id=MBJUWA5UQ7GH4ZIGU5ZPBCY3HHXXKFPMRXTKL63EJM3AHXWMRDQA, seg_pos={0 0}, upload=317114]: Done uploading piece. err: <nil> Jul 9 21:33:01.863729242 (396h22m36.634370577s) [noise=true , address=45.91.95.5:4022 , piece_id=U7WC2HGN7JYFDXSLQ6QFWXPSHAVENXGPRPVRUS55IUXNT44GY7DA, seg_pos={0 0}, upload=317114]: Done uploading piece. err: ecclient: upload failed (node:12X8Et5FNyAsE7ifEckfQemZkfnvYUuZ2Rn2xdxNxhmCmuDST2M, address:45.91.95.5:4022): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:33:01.863808243 (396h22m36.634449578s) [noise=true , address=104.36.82.51:4068 , piece_id=6FDENH2R2DDCEYP4HE64YQSCDMMX2PPYI25AJDGQEOK5GQMIHX3A, seg_pos={0 0}, upload=317114]: Done uploading piece. err: ecclient: upload failed (node:12Lh9dYpHCxbs7gpZ4Xt7os5Gm5Nco6jrNW1V1FHddVUEMkFfPY, address:104.36.82.51:4068): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:33:01.863879039 (396h22m36.634520384s) [noise=true , address=64.49.13.169:4706 , piece_id=HVKNHMYI2DCSJKOYSHOFDYBIUCQ3CEAEG3IUAE4OSJXMKY7ZOKFA, seg_pos={0 0}, upload=317114]: Done uploading piece. err: ecclient: upload failed (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7, address:64.49.13.169:4706): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:33:01.863924856 (396h22m36.634566191s) [noise=true , address=45.91.95.4:4004 , piece_id=HPHOLVAJBNTZG3VZ6EJUYIXYRFKUDXEQ5SQCONIEHB5DGRL4X7WQ, seg_pos={0 0}, upload=317114]: Done uploading piece. err: ecclient: upload failed (node:12ZeqpAQ3qRdFvheAHDMKU3znEAiv8V3jJbABGBjWsNk3ZGMNM6, address:45.91.95.4:4004): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:33:01.863932541 (396h22m36.634573876s) [noise=true , address=45.91.95.2:4031 , piece_id=ATCD4TYZKYAPUHYPZIJW5A3LKGSAJCP2X6D7DLB34SIMGTPNYSYQ, seg_pos={0 0}, upload=317114]: Done uploading piece. err: ecclient: upload failed (node:12Mwd9RRYqm3GpbmT4B2KnAd2bXKq7HopJCk3E3W4A79Hh2eypL, address:45.91.95.2:4031): protocol: write tcp 188.166.225.170:34462->45.91.95.2:4031: use of closed network connection; write tcp 188.166.225.170:34462->45.91.95.2:4031: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34462->45.91.95.2:4031: use of closed network connection Jul 9 21:33:01.863977877 (396h22m36.634619222s) [seg_pos={0 0}, upload=317114]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 21:33:02.095730796 (396h22m36.866372141s) [upload=317114]: Done uploading segments... Jul 9 21:33:02.095763348 (396h22m36.866404693s) [upload=317114]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 51.254138113s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 10 out of 60 is failed (threshold=3) [.X...XXX.........X......X....X..X.........X...............X.] duration: 41.977098932s FAIL (2025-07-09 21:33:48 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 41.977116106s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [X.X...XXX.........X......X....X..X.........X...............X] duration: 1m0.000681779s FAIL (2025-07-09 21:33:06 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000699744s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......X..X.................................................] duration: 6.193829956s FAIL (2025-07-09 21:33:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.193853589s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [...X..X..........X...X................X....X...............X] duration: 1m0.234382889s FAIL (2025-07-09 21:33:02 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== one starting upload Jul 9 21:32:02.509558713 (396h21m37.280200068s) [upload=317114]: Uploading segments... Jul 9 21:32:02.509643425 (396h21m37.28028476s) [upload=317114]: Got next segment. Inline: false Jul 9 21:32:02.509672832 (396h21m37.280314167s) [upload=317114]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 21:32:02.509677090 (396h21m37.280318425s) [upload=317114]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 21:32:02.954858160 (396h21m37.725499495s) [seg_pos={0 0}, upload=317114]: Begin upload segment... Jul 9 21:32:02.954898557 (396h21m37.725539892s) [seg_pos={0 0}, upload=317114]: Done begin upload segment. Jul 9 21:32:02.954924857 (396h21m37.725566192s) [noise=true , address=77.90.31.12:4600 , piece_id=5RA324WSM6Y37EOS3ODKXDOQIWVHCC5BYQPKQJB7ZDAHAQTHHLWQ, seg_pos={0 0}, upload=317114]: Uploading piece... Jul 9 21:32:02.954945667 (396h21m37.725587002s) [noise=true , address=77.90.31.14:4691 , piece_id=UC4TVYEX4W2OXUQNFKAY4IGQM6TVPUY47K5RYQFMLS7ZORXTBBUA, seg_pos={0 0}, upload=317114]: Uploading piece... Jul 9 21:32:02.955440825 (396h21m37.72608216s) [noise=true , address=45.91.95.2:4031 , piece_id=ATCD4TYZKYAPUHYPZIJW5A3LKGSAJCP2X6D7DLB34SIMGTPNYSYQ, seg_pos={0 0}, upload=317114]: Uploading piece... Jul 9 21:32:02.955447547 (396h21m37.726088882s) [noise=true , address=45.91.95.5:4022 , piece_id=U7WC2HGN7JYFDXSLQ6QFWXPSHAVENXGPRPVRUS55IUXNT44GY7DA, seg_pos={0 0}, upload=317114]: Uploading piece... Jul 9 21:32:02.955714017 (396h21m37.726355352s) [noise=true , address=104.36.82.50:4042 , piece_id=IZIYWURR3H3MGCKH5CM2F5WRE6BVMJL23HDAAXGF3MYLJ2QO3XXA, seg_pos={0 0}, upload=317114]: Uploading piece... Jul 9 21:32:02.955807856 (396h21m37.726449192s) [noise=true , address=77.90.31.18:4830 , piece_id=RF6VQI7G3KVYBC55GFVMOB6THWBJDPMESEJMPFPWV2BJYHVUFXNQ, seg_pos={0 0}, upload=317114]: Uploading piece... Jul 9 21:32:02.955960779 (396h21m37.726602115s) [noise=true , address=45.91.95.4:4004 , piece_id=HPHOLVAJBNTZG3VZ6EJUYIXYRFKUDXEQ5SQCONIEHB5DGRL4X7WQ, seg_pos={0 0}, upload=317114]: Uploading piece... Jul 9 21:32:02.956041003 (396h21m37.726682338s) [noise=true , address=77.90.31.17:4788 , piece_id=QAOJLNZOVYLX3ZTIA3O6ADDFFSJAVOR5FG4WEAKBFXAU72ZQGJ5A, seg_pos={0 0}, upload=317114]: Uploading piece... Jul 9 21:32:02.956186792 (396h21m37.726828127s) [noise=true , address=104.36.82.5:4862 , piece_id=UZEXRCCCWUCP2FW2KOSGSCB6NPNLK4HGLTX5HNNE7VR7TBYJ2ATA, seg_pos={0 0}, upload=317114]: Uploading piece... Jul 9 21:32:02.956254751 (396h21m37.726896086s) [noise=true , address=104.36.82.51:4068 , piece_id=6FDENH2R2DDCEYP4HE64YQSCDMMX2PPYI25AJDGQEOK5GQMIHX3A, seg_pos={0 0}, upload=317114]: Uploading piece... Jul 9 21:32:02.956459654 (396h21m37.727100989s) [noise=true , address=77.90.31.16:4751 , piece_id=SOV5AWXE2NMSJURQTAQA4SORO6EDQRXME6RTHLLN6UEEZEX6ZBVQ, seg_pos={0 0}, upload=317114]: Uploading piece... Jul 9 21:32:02.956534187 (396h21m37.727175522s) [noise=true , address=64.49.13.169:4706 , piece_id=HVKNHMYI2DCSJKOYSHOFDYBIUCQ3CEAEG3IUAE4OSJXMKY7ZOKFA, seg_pos={0 0}, upload=317114]: Uploading piece... Jul 9 21:32:02.956686177 (396h21m37.727327512s) [noise=true , address=77.90.31.19:4874 , piece_id=3PZ7OYOGSBZOJSAZ2SH3REVPYQHREQILPUYA2NLCTIYPGQCCTDHQ, seg_pos={0 0}, upload=317114]: Uploading piece... Jul 9 21:32:02.956812068 (396h21m37.727453403s) [noise=true , address=45.91.95.3:4031 , piece_id=MBJUWA5UQ7GH4ZIGU5ZPBCY3HHXXKFPMRXTKL63EJM3AHXWMRDQA, seg_pos={0 0}, upload=317114]: Uploading piece... Jul 9 21:32:02.957105619 (396h21m37.727746954s) [noise=true , address=77.90.31.11:4584 , piece_id=TAA3NQL5IUKVQBBRL352KVYTXO44EZEKX3INIB2DBM5CQPTFBJSA, seg_pos={0 0}, upload=317114]: Uploading piece... Jul 9 21:32:02.957385915 (396h21m37.72802725s) [noise=true , address=77.90.31.18:4832 , piece_id=7WGWSIBCKZAXO4I6FKGXSZJJ6L5UZEYXYY6R24P5F42XR4XUBQNQ, seg_pos={0 0}, upload=317114]: Uploading piece... Jul 9 21:32:02.957657074 (396h21m37.728298409s) [noise=true , address=77.90.31.13:4632 , piece_id=FB76GO7TALNZHNECN3UOKQHQPA4H3VCX4TI2X6CG4VZHQ7DULYOQ, seg_pos={0 0}, upload=317114]: Uploading piece... Jul 9 21:32:02.958236543 (396h21m37.728877878s) [noise=true , address=79.127.219.38:4024 , piece_id=4HDQ6PB2G7NKABGKQFUUQQ2XMDATAS3MEQYE27UT4ABNU3O6X7DQ, seg_pos={0 0}, upload=317114]: Uploading piece... Jul 9 21:32:04.069772770 (396h21m38.840414105s) [upload=317114]: Next returned nil segment Jul 9 21:32:04.069791056 (396h21m38.840432391s) [upload=317114]: Waiting for error group managing segments... Jul 9 21:32:04.947577444 (396h21m39.718218779s) [noise=true , address=77.90.31.13:4632 , piece_id=FB76GO7TALNZHNECN3UOKQHQPA4H3VCX4TI2X6CG4VZHQ7DULYOQ, seg_pos={0 0}, upload=317114]: Done uploading piece. err: <nil> Jul 9 21:32:04.951394947 (396h21m39.722036282s) [noise=true , address=77.90.31.18:4832 , piece_id=7WGWSIBCKZAXO4I6FKGXSZJJ6L5UZEYXYY6R24P5F42XR4XUBQNQ, seg_pos={0 0}, upload=317114]: Done uploading piece. err: <nil> Jul 9 21:32:04.952763405 (396h21m39.72340474s) [noise=true , address=77.90.31.12:4600 , piece_id=5RA324WSM6Y37EOS3ODKXDOQIWVHCC5BYQPKQJB7ZDAHAQTHHLWQ, seg_pos={0 0}, upload=317114]: Done uploading piece. err: <nil> Jul 9 21:32:04.954245150 (396h21m39.724886485s) [noise=true , address=77.90.31.18:4830 , piece_id=RF6VQI7G3KVYBC55GFVMOB6THWBJDPMESEJMPFPWV2BJYHVUFXNQ, seg_pos={0 0}, upload=317114]: Done uploading piece. err: <nil> Jul 9 21:32:04.956787544 (396h21m39.727428869s) [noise=true , address=77.90.31.19:4874 , piece_id=3PZ7OYOGSBZOJSAZ2SH3REVPYQHREQILPUYA2NLCTIYPGQCCTDHQ, seg_pos={0 0}, upload=317114]: Done uploading piece. err: <nil> Jul 9 21:32:04.960355681 (396h21m39.730997016s) [noise=true , address=77.90.31.16:4751 , piece_id=SOV5AWXE2NMSJURQTAQA4SORO6EDQRXME6RTHLLN6UEEZEX6ZBVQ, seg_pos={0 0}, upload=317114]: Done uploading piece. err: <nil> Jul 9 21:32:04.963274956 (396h21m39.733916291s) [noise=true , address=77.90.31.11:4584 , piece_id=TAA3NQL5IUKVQBBRL352KVYTXO44EZEKX3INIB2DBM5CQPTFBJSA, seg_pos={0 0}, upload=317114]: Done uploading piece. err: <nil> Jul 9 21:32:04.966230130 (396h21m39.736871465s) [noise=true , address=77.90.31.14:4691 , piece_id=UC4TVYEX4W2OXUQNFKAY4IGQM6TVPUY47K5RYQFMLS7ZORXTBBUA, seg_pos={0 0}, upload=317114]: Done uploading piece. err: <nil> Jul 9 21:32:04.967618355 (396h21m39.73825969s) [noise=true , address=77.90.31.17:4788 , piece_id=QAOJLNZOVYLX3ZTIA3O6ADDFFSJAVOR5FG4WEAKBFXAU72ZQGJ5A, seg_pos={0 0}, upload=317114]: Done uploading piece. err: <nil> Jul 9 21:32:08.952512712 (396h21m43.723154047s) [noise=true , address=79.127.219.38:4024 , piece_id=4HDQ6PB2G7NKABGKQFUUQQ2XMDATAS3MEQYE27UT4ABNU3O6X7DQ, seg_pos={0 0}, upload=317114]: Done uploading piece. err: <nil> Jul 9 21:32:26.893189191 (396h22m1.663830526s) [noise=true , address=104.36.82.5:4862 , piece_id=UZEXRCCCWUCP2FW2KOSGSCB6NPNLK4HGLTX5HNNE7VR7TBYJ2ATA, seg_pos={0 0}, upload=317114]: Done uploading piece. err: <nil> Jul 9 21:32:38.593603255 (396h22m13.3642446s) [noise=true , address=104.36.82.50:4042 , piece_id=IZIYWURR3H3MGCKH5CM2F5WRE6BVMJL23HDAAXGF3MYLJ2QO3XXA, seg_pos={0 0}, upload=317114]: Done uploading piece. err: <nil> Jul 9 21:32:47.819908973 (396h22m22.590550308s) [noise=true , address=45.91.95.3:4031 , piece_id=MBJUWA5UQ7GH4ZIGU5ZPBCY3HHXXKFPMRXTKL63EJM3AHXWMRDQA, seg_pos={0 0}, upload=317114]: Done uploading piece. err: <nil> Jul 9 21:33:01.863729242 (396h22m36.634370577s) [noise=true , address=45.91.95.5:4022 , piece_id=U7WC2HGN7JYFDXSLQ6QFWXPSHAVENXGPRPVRUS55IUXNT44GY7DA, seg_pos={0 0}, upload=317114]: Done uploading piece. err: ecclient: upload failed (node:12X8Et5FNyAsE7ifEckfQemZkfnvYUuZ2Rn2xdxNxhmCmuDST2M, address:45.91.95.5:4022): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:33:01.863808243 (396h22m36.634449578s) [noise=true , address=104.36.82.51:4068 , piece_id=6FDENH2R2DDCEYP4HE64YQSCDMMX2PPYI25AJDGQEOK5GQMIHX3A, seg_pos={0 0}, upload=317114]: Done uploading piece. err: ecclient: upload failed (node:12Lh9dYpHCxbs7gpZ4Xt7os5Gm5Nco6jrNW1V1FHddVUEMkFfPY, address:104.36.82.51:4068): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:33:01.863879039 (396h22m36.634520384s) [noise=true , address=64.49.13.169:4706 , piece_id=HVKNHMYI2DCSJKOYSHOFDYBIUCQ3CEAEG3IUAE4OSJXMKY7ZOKFA, seg_pos={0 0}, upload=317114]: Done uploading piece. err: ecclient: upload failed (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7, address:64.49.13.169:4706): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:33:01.863924856 (396h22m36.634566191s) [noise=true , address=45.91.95.4:4004 , piece_id=HPHOLVAJBNTZG3VZ6EJUYIXYRFKUDXEQ5SQCONIEHB5DGRL4X7WQ, seg_pos={0 0}, upload=317114]: Done uploading piece. err: ecclient: upload failed (node:12ZeqpAQ3qRdFvheAHDMKU3znEAiv8V3jJbABGBjWsNk3ZGMNM6, address:45.91.95.4:4004): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 21:33:01.863932541 (396h22m36.634573876s) [noise=true , address=45.91.95.2:4031 , piece_id=ATCD4TYZKYAPUHYPZIJW5A3LKGSAJCP2X6D7DLB34SIMGTPNYSYQ, seg_pos={0 0}, upload=317114]: Done uploading piece. err: ecclient: upload failed (node:12Mwd9RRYqm3GpbmT4B2KnAd2bXKq7HopJCk3E3W4A79Hh2eypL, address:45.91.95.2:4031): protocol: write tcp 188.166.225.170:34462->45.91.95.2:4031: use of closed network connection; write tcp 188.166.225.170:34462->45.91.95.2:4031: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34462->45.91.95.2:4031: use of closed network connection Jul 9 21:33:01.863977877 (396h22m36.634619222s) [seg_pos={0 0}, upload=317114]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 21:33:02.095730796 (396h22m36.866372141s) [upload=317114]: Done uploading segments... Jul 9 21:33:02.095763348 (396h22m36.866404693s) [upload=317114]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.234406443s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........X..X................................................] duration: 5.885914754s FAIL (2025-07-09 21:32:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.885939842s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........X..X...............................................] duration: 5.965448067s FAIL (2025-07-09 21:31:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.965479397s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 10 out of 60 is failed (threshold=3) [.X.X...XXX.........X......X....X..X.........X...............] duration: 34.587099489s FAIL (2025-07-09 21:31:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 34.587120228s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [....X..X..........X...X................X....X...............] duration: 33.754516802s FAIL (2025-07-09 21:31:34 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== estItem_SegmentBegin Jul 9 21:17:05.831046046 (396h6m40.601687381s) [seg_pos={0 0}, upload=316919]: Begin upload segment... Jul 9 21:17:05.831118736 (396h6m40.601760071s) [seg_pos={0 0}, upload=316919]: Done begin upload segment. Jul 9 21:17:05.831139856 (396h6m40.601781181s) [noise=true , address=77.90.31.13:4642 , piece_id=HOSOIEVY5QEKOFY2SJP3RRDIOHHGBL6HWCYHCGO32MZTM7KNOSKA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.831147400 (396h6m40.601788736s) [noise=true , address=45.91.95.5:4003 , piece_id=ZHZ533HFZTAYM5AAB7PZMKVEXZUEU7IAYE3JVQHWQ3HZBP5NH2JA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.831278071 (396h6m40.601919446s) [noise=true , address=77.90.31.11:4556 , piece_id=G36QODNWILUHAGZYZSFT766PL2UPC3QIN52ECPRWXAFQ2JRAZAZQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.831787615 (396h6m40.60242895s) [noise=true , address=77.90.31.17:4804 , piece_id=ZEKF53R64TNPR7MV4OIYWFRPLNUPBGMZVSZL5NFYYVXR4W7SV7HA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832239780 (396h6m40.602881125s) [noise=true , address=45.91.95.3:4025 , piece_id=R4NB54WF45YCLP22NLBKBTJJAN7RPK72J6R3F53GIN46HXSLCGZQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832369759 (396h6m40.603011104s) [noise=true , address=64.49.13.169:4706 , piece_id=SSND7CE46HS4ZSIJXYBT3DONMW7VRCH2BW3DBMD5RVBF64IDFA5A, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832744927 (396h6m40.603386262s) [noise=true , address=104.36.82.21:4494 , piece_id=BFXEBEKBKSXG2MDDHXVDCVXAS5QEOBWPTDKJBCMYHTQR2USBYX3Q, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832955970 (396h6m40.603597325s) [noise=true , address=45.91.95.2:4036 , piece_id=P2UR5N6YFZJDCJXM66CDPTKRJWM5RWDVG5ABGBCVZVDOLHDFT4EA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.833358851 (396h6m40.604000206s) [noise=true , address=104.36.82.18:4391 , piece_id=UVDDOI3DARETAL4A67SU62YEQYOBH2GIWEZO3TCOT6HJ4JALZ4KQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.833826405 (396h6m40.60446774s) [noise=true , address=77.90.31.21:4897 , piece_id=COZCWBB4PICQ6MER6R3OF5EXB2FLAGWONROZP6KKVM557P3PET6A, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834183488 (396h6m40.604824823s) [noise=true , address=77.90.31.16:4761 , piece_id=D2M5SNI5UUK7TSYTDVSXQG5DP4L7PATEDD2ERWRGLRN67ZIYNJJA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834474084 (396h6m40.605115419s) [noise=true , address=104.36.82.41:4771 , piece_id=CQARFMA6NNQLUK5GT75ZF22JG7FSFH52AYIUCHQYNOTE2XUKLK3Q, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834723552 (396h6m40.605364877s) [noise=true , address=79.127.205.245:4235 , piece_id=5U7O5IP4Z444B2RFLHUM4TY7EDLHO3SMEGCIITWPLD5UO5NYUHAA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834918645 (396h6m40.60555998s) [noise=true , address=79.127.220.101:4061 , piece_id=LBUTU7VGVTTD5O7STRLF64EDIOVL2DCJXZ7HZZJCATKKGPCJ2MAA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835106364 (396h6m40.605747699s) [noise=true , address=45.91.95.4:4035 , piece_id=7ETHU4GYKYSDG2VELAHYVF3MJHJMQOQ4TRVXW6SROR3AW7QKA3JA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835289223 (396h6m40.605930558s) [noise=true , address=77.90.31.13:4649 , piece_id=PIO4TOCVFILEVYZPTGQ6IP5VWPRGXRMIWRKX5ERRP7JKPY2ASCPQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835388353 (396h6m40.606029699s) [noise=true , address=77.90.31.18:4823 , piece_id=ZULY5IBJK7UUCBD3MKSEUQGGSAETNO6UN4PM5CAYYYTJTGEHYM2A, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835827995 (396h6m40.60646933s) [noise=true , address=104.36.82.4:4823 , piece_id=5K6S55655OAGIXA64XJPIKFNUNDZV6E33SVSJCW7T4YVRUWEBNWA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:06.932899634 (396h6m41.703540969s) [upload=316919]: Next returned nil segment Jul 9 21:17:06.932916737 (396h6m41.703558073s) [upload=316919]: Waiting for error group managing segments... Jul 9 21:17:07.211937644 (396h6m41.982578979s) [noise=true , address=79.127.205.245:4235 , piece_id=5U7O5IP4Z444B2RFLHUM4TY7EDLHO3SMEGCIITWPLD5UO5NYUHAA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.602442718 (396h6m42.373084053s) [noise=true , address=79.127.220.101:4061 , piece_id=LBUTU7VGVTTD5O7STRLF64EDIOVL2DCJXZ7HZZJCATKKGPCJ2MAA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.837414789 (396h6m42.608056114s) [noise=true , address=77.90.31.18:4823 , piece_id=ZULY5IBJK7UUCBD3MKSEUQGGSAETNO6UN4PM5CAYYYTJTGEHYM2A, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.838075924 (396h6m42.608717249s) [noise=true , address=77.90.31.13:4642 , piece_id=HOSOIEVY5QEKOFY2SJP3RRDIOHHGBL6HWCYHCGO32MZTM7KNOSKA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.838447595 (396h6m42.60908893s) [noise=true , address=77.90.31.13:4649 , piece_id=PIO4TOCVFILEVYZPTGQ6IP5VWPRGXRMIWRKX5ERRP7JKPY2ASCPQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840181652 (396h6m42.610822987s) [noise=true , address=77.90.31.11:4556 , piece_id=G36QODNWILUHAGZYZSFT766PL2UPC3QIN52ECPRWXAFQ2JRAZAZQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840571077 (396h6m42.611212412s) [noise=true , address=77.90.31.21:4897 , piece_id=COZCWBB4PICQ6MER6R3OF5EXB2FLAGWONROZP6KKVM557P3PET6A, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840706977 (396h6m42.611348312s) [noise=true , address=77.90.31.16:4761 , piece_id=D2M5SNI5UUK7TSYTDVSXQG5DP4L7PATEDD2ERWRGLRN67ZIYNJJA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840868997 (396h6m42.611510332s) [noise=true , address=77.90.31.17:4804 , piece_id=ZEKF53R64TNPR7MV4OIYWFRPLNUPBGMZVSZL5NFYYVXR4W7SV7HA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:19.635937416 (396h6m54.406578741s) [noise=true , address=104.36.82.21:4494 , piece_id=BFXEBEKBKSXG2MDDHXVDCVXAS5QEOBWPTDKJBCMYHTQR2USBYX3Q, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:20.256775845 (396h6m55.02741718s) [noise=true , address=104.36.82.4:4823 , piece_id=5K6S55655OAGIXA64XJPIKFNUNDZV6E33SVSJCW7T4YVRUWEBNWA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:22.497919674 (396h6m57.268561009s) [noise=true , address=104.36.82.18:4391 , piece_id=UVDDOI3DARETAL4A67SU62YEQYOBH2GIWEZO3TCOT6HJ4JALZ4KQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:34.811072455 (396h7m9.58171378s) [noise=true , address=45.91.95.4:4035 , piece_id=7ETHU4GYKYSDG2VELAHYVF3MJHJMQOQ4TRVXW6SROR3AW7QKA3JA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:18:00.756054942 (396h7m35.526696277s) [noise=true , address=64.49.13.169:4706 , piece_id=SSND7CE46HS4ZSIJXYBT3DONMW7VRCH2BW3DBMD5RVBF64IDFA5A, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:18:00.756079820 (396h7m35.526721145s) [seg_pos={0 0}, upload=316919]: Segment reached optimal threshold of 14 pieces. Jul 9 21:18:00.756241590 (396h7m35.526882915s) [noise=true , address=45.91.95.5:4003 , piece_id=ZHZ533HFZTAYM5AAB7PZMKVEXZUEU7IAYE3JVQHWQ3HZBP5NH2JA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JYAn5wJ8xwQoAaA1wSaLtinKxMBXWF7QazynV8y3K9rDrhCv7): protocol: expected piece hash; context canceled; EOF Jul 9 21:18:00.756308267 (396h7m35.526949602s) [noise=true , address=104.36.82.41:4771 , piece_id=CQARFMA6NNQLUK5GT75ZF22JG7FSFH52AYIUCHQYNOTE2XUKLK3Q, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1PggYtrfQ5ozsJL1HeAVJWc4jLvf5jiHspX3CgFY89swMimWto): protocol: expected piece hash; write tcp 188.166.225.170:41120->104.36.82.41:4771: use of closed network connection; write tcp 188.166.225.170:41120->104.36.82.41:4771: use of closed network connection; EOF Jul 9 21:18:00.756324358 (396h7m35.526965693s) [noise=true , address=45.91.95.2:4036 , piece_id=P2UR5N6YFZJDCJXM66CDPTKRJWM5RWDVG5ABGBCVZVDOLHDFT4EA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12tWFPgstVDQwPbnZgtcndFmmxofodEgG8tX7SPfqkQRrfodftA): protocol: expected piece hash; write tcp 188.166.225.170:48736->45.91.95.2:4036: use of closed network connection; write tcp 188.166.225.170:48736->45.91.95.2:4036: use of closed network connection; EOF Jul 9 21:18:00.756427997 (396h7m35.527069332s) [noise=true , address=45.91.95.3:4025 , piece_id=R4NB54WF45YCLP22NLBKBTJJAN7RPK72J6R3F53GIN46HXSLCGZQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1254mHh9cP8gUCddJB5v1mdWF9Ej6BkUiRHudYG4PJSdgKeTB42): protocol: write tcp 188.166.225.170:50010->45.91.95.3:4025: use of closed network connection; write tcp 188.166.225.170:50010->45.91.95.3:4025: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50010->45.91.95.3:4025: use of closed network connection Jul 9 21:18:00.756439568 (396h7m35.527080893s) [seg_pos={0 0}, upload=316919]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 21:18:00.756594485 (396h7m35.52723581s) [upload=316919]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 21:18:00.756600827 (396h7m35.527242162s) [upload=316919]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 21:18:01.143222271 (396h7m35.913863606s) [upload=316919]: Done uploading segments... Jul 9 21:18:01.143240435 (396h7m35.91388177s) [upload=316919]: Upload finished. err: <nil> Duration: 33.754560664s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........X..X..............................................] duration: 6.05469103s FAIL (2025-07-09 21:31:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.054710436s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 10 out of 60 is failed (threshold=3) [..X.X...XXX.........X......X....X..X.........X..............] duration: 57.634197173s FAIL (2025-07-09 21:30:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 57.634220297s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [X....X..X..........X...X................X....X..............] duration: 37.71350779s FAIL (2025-07-09 21:30:28 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== estItem_SegmentBegin Jul 9 21:17:05.831046046 (396h6m40.601687381s) [seg_pos={0 0}, upload=316919]: Begin upload segment... Jul 9 21:17:05.831118736 (396h6m40.601760071s) [seg_pos={0 0}, upload=316919]: Done begin upload segment. Jul 9 21:17:05.831139856 (396h6m40.601781181s) [noise=true , address=77.90.31.13:4642 , piece_id=HOSOIEVY5QEKOFY2SJP3RRDIOHHGBL6HWCYHCGO32MZTM7KNOSKA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.831147400 (396h6m40.601788736s) [noise=true , address=45.91.95.5:4003 , piece_id=ZHZ533HFZTAYM5AAB7PZMKVEXZUEU7IAYE3JVQHWQ3HZBP5NH2JA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.831278071 (396h6m40.601919446s) [noise=true , address=77.90.31.11:4556 , piece_id=G36QODNWILUHAGZYZSFT766PL2UPC3QIN52ECPRWXAFQ2JRAZAZQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.831787615 (396h6m40.60242895s) [noise=true , address=77.90.31.17:4804 , piece_id=ZEKF53R64TNPR7MV4OIYWFRPLNUPBGMZVSZL5NFYYVXR4W7SV7HA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832239780 (396h6m40.602881125s) [noise=true , address=45.91.95.3:4025 , piece_id=R4NB54WF45YCLP22NLBKBTJJAN7RPK72J6R3F53GIN46HXSLCGZQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832369759 (396h6m40.603011104s) [noise=true , address=64.49.13.169:4706 , piece_id=SSND7CE46HS4ZSIJXYBT3DONMW7VRCH2BW3DBMD5RVBF64IDFA5A, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832744927 (396h6m40.603386262s) [noise=true , address=104.36.82.21:4494 , piece_id=BFXEBEKBKSXG2MDDHXVDCVXAS5QEOBWPTDKJBCMYHTQR2USBYX3Q, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.832955970 (396h6m40.603597325s) [noise=true , address=45.91.95.2:4036 , piece_id=P2UR5N6YFZJDCJXM66CDPTKRJWM5RWDVG5ABGBCVZVDOLHDFT4EA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.833358851 (396h6m40.604000206s) [noise=true , address=104.36.82.18:4391 , piece_id=UVDDOI3DARETAL4A67SU62YEQYOBH2GIWEZO3TCOT6HJ4JALZ4KQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.833826405 (396h6m40.60446774s) [noise=true , address=77.90.31.21:4897 , piece_id=COZCWBB4PICQ6MER6R3OF5EXB2FLAGWONROZP6KKVM557P3PET6A, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834183488 (396h6m40.604824823s) [noise=true , address=77.90.31.16:4761 , piece_id=D2M5SNI5UUK7TSYTDVSXQG5DP4L7PATEDD2ERWRGLRN67ZIYNJJA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834474084 (396h6m40.605115419s) [noise=true , address=104.36.82.41:4771 , piece_id=CQARFMA6NNQLUK5GT75ZF22JG7FSFH52AYIUCHQYNOTE2XUKLK3Q, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834723552 (396h6m40.605364877s) [noise=true , address=79.127.205.245:4235 , piece_id=5U7O5IP4Z444B2RFLHUM4TY7EDLHO3SMEGCIITWPLD5UO5NYUHAA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.834918645 (396h6m40.60555998s) [noise=true , address=79.127.220.101:4061 , piece_id=LBUTU7VGVTTD5O7STRLF64EDIOVL2DCJXZ7HZZJCATKKGPCJ2MAA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835106364 (396h6m40.605747699s) [noise=true , address=45.91.95.4:4035 , piece_id=7ETHU4GYKYSDG2VELAHYVF3MJHJMQOQ4TRVXW6SROR3AW7QKA3JA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835289223 (396h6m40.605930558s) [noise=true , address=77.90.31.13:4649 , piece_id=PIO4TOCVFILEVYZPTGQ6IP5VWPRGXRMIWRKX5ERRP7JKPY2ASCPQ, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835388353 (396h6m40.606029699s) [noise=true , address=77.90.31.18:4823 , piece_id=ZULY5IBJK7UUCBD3MKSEUQGGSAETNO6UN4PM5CAYYYTJTGEHYM2A, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:05.835827995 (396h6m40.60646933s) [noise=true , address=104.36.82.4:4823 , piece_id=5K6S55655OAGIXA64XJPIKFNUNDZV6E33SVSJCW7T4YVRUWEBNWA, seg_pos={0 0}, upload=316919]: Uploading piece... Jul 9 21:17:06.932899634 (396h6m41.703540969s) [upload=316919]: Next returned nil segment Jul 9 21:17:06.932916737 (396h6m41.703558073s) [upload=316919]: Waiting for error group managing segments... Jul 9 21:17:07.211937644 (396h6m41.982578979s) [noise=true , address=79.127.205.245:4235 , piece_id=5U7O5IP4Z444B2RFLHUM4TY7EDLHO3SMEGCIITWPLD5UO5NYUHAA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.602442718 (396h6m42.373084053s) [noise=true , address=79.127.220.101:4061 , piece_id=LBUTU7VGVTTD5O7STRLF64EDIOVL2DCJXZ7HZZJCATKKGPCJ2MAA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.837414789 (396h6m42.608056114s) [noise=true , address=77.90.31.18:4823 , piece_id=ZULY5IBJK7UUCBD3MKSEUQGGSAETNO6UN4PM5CAYYYTJTGEHYM2A, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.838075924 (396h6m42.608717249s) [noise=true , address=77.90.31.13:4642 , piece_id=HOSOIEVY5QEKOFY2SJP3RRDIOHHGBL6HWCYHCGO32MZTM7KNOSKA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.838447595 (396h6m42.60908893s) [noise=true , address=77.90.31.13:4649 , piece_id=PIO4TOCVFILEVYZPTGQ6IP5VWPRGXRMIWRKX5ERRP7JKPY2ASCPQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840181652 (396h6m42.610822987s) [noise=true , address=77.90.31.11:4556 , piece_id=G36QODNWILUHAGZYZSFT766PL2UPC3QIN52ECPRWXAFQ2JRAZAZQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840571077 (396h6m42.611212412s) [noise=true , address=77.90.31.21:4897 , piece_id=COZCWBB4PICQ6MER6R3OF5EXB2FLAGWONROZP6KKVM557P3PET6A, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840706977 (396h6m42.611348312s) [noise=true , address=77.90.31.16:4761 , piece_id=D2M5SNI5UUK7TSYTDVSXQG5DP4L7PATEDD2ERWRGLRN67ZIYNJJA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:07.840868997 (396h6m42.611510332s) [noise=true , address=77.90.31.17:4804 , piece_id=ZEKF53R64TNPR7MV4OIYWFRPLNUPBGMZVSZL5NFYYVXR4W7SV7HA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:19.635937416 (396h6m54.406578741s) [noise=true , address=104.36.82.21:4494 , piece_id=BFXEBEKBKSXG2MDDHXVDCVXAS5QEOBWPTDKJBCMYHTQR2USBYX3Q, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:20.256775845 (396h6m55.02741718s) [noise=true , address=104.36.82.4:4823 , piece_id=5K6S55655OAGIXA64XJPIKFNUNDZV6E33SVSJCW7T4YVRUWEBNWA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:22.497919674 (396h6m57.268561009s) [noise=true , address=104.36.82.18:4391 , piece_id=UVDDOI3DARETAL4A67SU62YEQYOBH2GIWEZO3TCOT6HJ4JALZ4KQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:17:34.811072455 (396h7m9.58171378s) [noise=true , address=45.91.95.4:4035 , piece_id=7ETHU4GYKYSDG2VELAHYVF3MJHJMQOQ4TRVXW6SROR3AW7QKA3JA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:18:00.756054942 (396h7m35.526696277s) [noise=true , address=64.49.13.169:4706 , piece_id=SSND7CE46HS4ZSIJXYBT3DONMW7VRCH2BW3DBMD5RVBF64IDFA5A, seg_pos={0 0}, upload=316919]: Done uploading piece. err: <nil> Jul 9 21:18:00.756079820 (396h7m35.526721145s) [seg_pos={0 0}, upload=316919]: Segment reached optimal threshold of 14 pieces. Jul 9 21:18:00.756241590 (396h7m35.526882915s) [noise=true , address=45.91.95.5:4003 , piece_id=ZHZ533HFZTAYM5AAB7PZMKVEXZUEU7IAYE3JVQHWQ3HZBP5NH2JA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JYAn5wJ8xwQoAaA1wSaLtinKxMBXWF7QazynV8y3K9rDrhCv7): protocol: expected piece hash; context canceled; EOF Jul 9 21:18:00.756308267 (396h7m35.526949602s) [noise=true , address=104.36.82.41:4771 , piece_id=CQARFMA6NNQLUK5GT75ZF22JG7FSFH52AYIUCHQYNOTE2XUKLK3Q, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1PggYtrfQ5ozsJL1HeAVJWc4jLvf5jiHspX3CgFY89swMimWto): protocol: expected piece hash; write tcp 188.166.225.170:41120->104.36.82.41:4771: use of closed network connection; write tcp 188.166.225.170:41120->104.36.82.41:4771: use of closed network connection; EOF Jul 9 21:18:00.756324358 (396h7m35.526965693s) [noise=true , address=45.91.95.2:4036 , piece_id=P2UR5N6YFZJDCJXM66CDPTKRJWM5RWDVG5ABGBCVZVDOLHDFT4EA, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12tWFPgstVDQwPbnZgtcndFmmxofodEgG8tX7SPfqkQRrfodftA): protocol: expected piece hash; write tcp 188.166.225.170:48736->45.91.95.2:4036: use of closed network connection; write tcp 188.166.225.170:48736->45.91.95.2:4036: use of closed network connection; EOF Jul 9 21:18:00.756427997 (396h7m35.527069332s) [noise=true , address=45.91.95.3:4025 , piece_id=R4NB54WF45YCLP22NLBKBTJJAN7RPK72J6R3F53GIN46HXSLCGZQ, seg_pos={0 0}, upload=316919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1254mHh9cP8gUCddJB5v1mdWF9Ej6BkUiRHudYG4PJSdgKeTB42): protocol: write tcp 188.166.225.170:50010->45.91.95.3:4025: use of closed network connection; write tcp 188.166.225.170:50010->45.91.95.3:4025: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50010->45.91.95.3:4025: use of closed network connection Jul 9 21:18:00.756439568 (396h7m35.527080893s) [seg_pos={0 0}, upload=316919]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 21:18:00.756594485 (396h7m35.52723581s) [upload=316919]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 21:18:00.756600827 (396h7m35.527242162s) [upload=316919]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 21:18:01.143222271 (396h7m35.913863606s) [upload=316919]: Done uploading segments... Jul 9 21:18:01.143240435 (396h7m35.91388177s) [upload=316919]: Upload finished. err: <nil> Duration: 37.713534321s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........X..X.............................................] duration: 5.545246937s FAIL (2025-07-09 21:30:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.545263309s edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)