Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 5.652573443s FAIL (2025-07-05 16:02:11 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== JYE6FC5T5QIBUQIILGDHNGXAUDBUIAOSKYFMPBCWPHP2SDJ3A, seg_pos={0 0}, upload=235214]: Done uploading piece. err: <nil> Jul 5 15:17:12.446386719 (294h6m47.217028064s) [seg_pos={0 0}, upload=235214]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 5 15:17:12.446476771 (294h6m47.217118106s) [upload=235214]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 5 15:17:12.446483043 (294h6m47.217124378s) [upload=235214]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 5 15:17:12.905279085 (294h6m47.67592042s) [upload=235214]: Done uploading segments... Jul 5 15:17:12.905303211 (294h6m47.675944546s) [upload=235214]: Upload finished. err: <nil> Jul 5 15:18:10.381454266 (294h7m45.152095611s) [upload=235227]: Starting upload Jul 5 15:18:10.381524069 (294h7m45.152165414s) [upload=235227]: Done starting upload Jul 5 15:18:10.381621115 (294h7m45.15226245s) [upload=235227]: Uploading segments... Jul 5 15:18:10.381740133 (294h7m45.152381468s) [upload=235227]: Got next segment. Inline: false Jul 5 15:18:10.381776342 (294h7m45.152417677s) [upload=235227]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 5 15:18:10.381781973 (294h7m45.152423308s) [upload=235227]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 5 15:18:10.613904835 (294h7m45.38454617s) [seg_pos={0 0}, upload=235227]: Begin upload segment... Jul 5 15:18:10.613947397 (294h7m45.384588732s) [seg_pos={0 0}, upload=235227]: Done begin upload segment. Jul 5 15:18:10.613968548 (294h7m45.384609883s) [noise=true , address=104.36.82.15:4259 , piece_id=LQKQ5MGKXHTPITBR2QAOQ5CX76GEVJGHJT3ZD3G4PFNN5IBTCJBQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.613995038 (294h7m45.384636373s) [noise=true , address=104.36.82.16:4295 , piece_id=VEL6CAUOJKMRN7A7M66ZPDLCMJVD555LZPHM53A24MDQJQOW7ECQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614303969 (294h7m45.384945304s) [noise=true , address=104.36.82.9:4006 , piece_id=47FME2B5NS6WMOHC2ZGE2FZVSPEQUXKYT6E7AOSANAE5XZVDH5VQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614367721 (294h7m45.385009056s) [noise=true , address=45.91.116.4:4937 , piece_id=YMKIHJET2EN3QHEK665ACZNQNHJ3UGE4H5T4DIHWJUHJQBYO7B7Q, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614619272 (294h7m45.385260607s) [noise=true , address=77.90.31.17:4771 , piece_id=V7M65ZHCGXPD2GVJL6LTB4NP3QN6QLSIBM7U34QAYPAIKMN6EBPA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614637758 (294h7m45.385279083s) [noise=true , address=45.91.116.14:4191 , piece_id=WH2BCI36GGFF4QUD4XDYBBDMUR2R2FFVYKFTZD3ZWXPJCFJ3D7HQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614890251 (294h7m45.385531576s) [noise=true , address=104.36.82.3:4767 , piece_id=OGHG4G5ZVC7GVY4XI6DO4KINXF7THHL6IHGDB6DAPDNUDOUFSC4A, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615197719 (294h7m45.385839054s) [noise=true , address=104.36.82.12:4149 , piece_id=YBTLQEWDMFK6JIFTYR3VJDNMXTS3NMWWHX2MPP53BM2UBDI3UUEA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615413552 (294h7m45.386054887s) [noise=true , address=77.90.31.18:4805 , piece_id=I23F6KYUQ7TJI4ZLEANACVNYJDMPKKMMACIZAIA6PC5S3CKQPV2Q, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615608124 (294h7m45.386249459s) [noise=true , address=77.90.31.21:4877 , piece_id=OJBIFN2TJPQ45LQLYCMPKV5UZM6Y2MXNOBT3TSGX26U4SZAL5Q7A, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615803147 (294h7m45.386444472s) [noise=true , address=104.36.82.8:4970 , piece_id=2YJN7VKTTTCMAWQW2MNR65DAJPFMHNIBIW5NWYNG4R6HNPRFH6NQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615998511 (294h7m45.386639836s) [noise=true , address=77.90.31.16:4733 , piece_id=3GQ7NEWE6WZJMQUA3GG3GSZYXEFNOAUEVWHOEH4TSNONYXUT43GQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616193414 (294h7m45.386834749s) [noise=true , address=77.90.31.19:4842 , piece_id=R4FTRBZ3PETCA6QGU6RPUEDZZBWXF7YVH2COEW4WH3YWDLU3VJMA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616449093 (294h7m45.387090429s) [noise=true , address=77.90.31.11:4554 , piece_id=4QRZ5MPQ7HTPRIVRL3O3H4OV7CQA7ESYT5RCPI3GTHXJOHOSAOVA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616635840 (294h7m45.387277175s) [noise=true , address=45.91.116.5:4298 , piece_id=TVTUGWHFTXB32GBY4LYXT3YDB5LIFXEGA7VO5BHFA24Y6EMDTPHQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616818790 (294h7m45.387460125s) [noise=true , address=104.36.82.13:4185 , piece_id=LTQS2CUZ5NRJCBOPEXDKUQ2IJGOIW3JJAPHWTCW7WUMQZPIWRXRA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:12.265123988 (294h7m47.035765313s) [upload=235227]: Next returned nil segment Jul 5 15:18:12.265138164 (294h7m47.035779499s) [upload=235227]: Waiting for error group managing segments... Jul 5 15:18:12.550465694 (294h7m47.321107019s) [noise=true , address=104.36.82.8:4970 , piece_id=2YJN7VKTTTCMAWQW2MNR65DAJPFMHNIBIW5NWYNG4R6HNPRFH6NQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.554315629 (294h7m47.324956954s) [noise=true , address=104.36.82.12:4149 , piece_id=YBTLQEWDMFK6JIFTYR3VJDNMXTS3NMWWHX2MPP53BM2UBDI3UUEA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.603452512 (294h7m47.374093847s) [noise=true , address=104.36.82.3:4767 , piece_id=OGHG4G5ZVC7GVY4XI6DO4KINXF7THHL6IHGDB6DAPDNUDOUFSC4A, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.639796643 (294h7m47.410437978s) [noise=true , address=104.36.82.9:4006 , piece_id=47FME2B5NS6WMOHC2ZGE2FZVSPEQUXKYT6E7AOSANAE5XZVDH5VQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.639894960 (294h7m47.410536285s) [noise=true , address=104.36.82.15:4259 , piece_id=LQKQ5MGKXHTPITBR2QAOQ5CX76GEVJGHJT3ZD3G4PFNN5IBTCJBQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.647222659 (294h7m47.417863994s) [noise=true , address=104.36.82.16:4295 , piece_id=VEL6CAUOJKMRN7A7M66ZPDLCMJVD555LZPHM53A24MDQJQOW7ECQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.665424010 (294h7m47.436065346s) [noise=true , address=104.36.82.13:4185 , piece_id=LTQS2CUZ5NRJCBOPEXDKUQ2IJGOIW3JJAPHWTCW7WUMQZPIWRXRA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.702209574 (294h7m47.47285091s) [noise=true , address=45.91.116.14:4191 , piece_id=WH2BCI36GGFF4QUD4XDYBBDMUR2R2FFVYKFTZD3ZWXPJCFJ3D7HQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.722073145 (294h7m47.49271448s) [noise=true , address=45.91.116.5:4298 , piece_id=TVTUGWHFTXB32GBY4LYXT3YDB5LIFXEGA7VO5BHFA24Y6EMDTPHQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.788621529 (294h7m47.559262864s) [noise=true , address=45.91.116.4:4937 , piece_id=YMKIHJET2EN3QHEK665ACZNQNHJ3UGE4H5T4DIHWJUHJQBYO7B7Q, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:36.752254721 (294h8m11.522896046s) [noise=true , address=77.90.31.19:4842 , piece_id=R4FTRBZ3PETCA6QGU6RPUEDZZBWXF7YVH2COEW4WH3YWDLU3VJMA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:37.783437003 (294h8m12.554078338s) [noise=true , address=77.90.31.17:4771 , piece_id=V7M65ZHCGXPD2GVJL6LTB4NP3QN6QLSIBM7U34QAYPAIKMN6EBPA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:39.733398290 (294h8m14.504039626s) [noise=true , address=77.90.31.18:4805 , piece_id=I23F6KYUQ7TJI4ZLEANACVNYJDMPKKMMACIZAIA6PC5S3CKQPV2Q, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Jul 5 15:18:39.733464387 (294h8m14.504105722s) [noise=true , address=77.90.31.21:4877 , piece_id=OJBIFN2TJPQ45LQLYCMPKV5UZM6Y2MXNOBT3TSGX26U4SZAL5Q7A, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: expected piece hash; context deadline exceeded; EOF Jul 5 15:18:39.733542065 (294h8m14.50418339s) [noise=true , address=77.90.31.11:4554 , piece_id=4QRZ5MPQ7HTPRIVRL3O3H4OV7CQA7ESYT5RCPI3GTHXJOHOSAOVA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection Jul 5 15:18:39.733547496 (294h8m14.504188831s) [noise=true , address=77.90.31.16:4733 , piece_id=3GQ7NEWE6WZJMQUA3GG3GSZYXEFNOAUEVWHOEH4TSNONYXUT43GQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection; write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection Jul 5 15:18:39.733583284 (294h8m14.504224619s) [seg_pos={0 0}, upload=235227]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 5 15:18:39.974321220 (294h8m14.744962555s) [upload=235227]: Done uploading segments... Jul 5 15:18:39.974353041 (294h8m14.744994376s) [upload=235227]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.652601856s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 5.674687366s FAIL (2025-07-05 16:01:20 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== JYE6FC5T5QIBUQIILGDHNGXAUDBUIAOSKYFMPBCWPHP2SDJ3A, seg_pos={0 0}, upload=235214]: Done uploading piece. err: <nil> Jul 5 15:17:12.446386719 (294h6m47.217028064s) [seg_pos={0 0}, upload=235214]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 5 15:17:12.446476771 (294h6m47.217118106s) [upload=235214]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 5 15:17:12.446483043 (294h6m47.217124378s) [upload=235214]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 5 15:17:12.905279085 (294h6m47.67592042s) [upload=235214]: Done uploading segments... Jul 5 15:17:12.905303211 (294h6m47.675944546s) [upload=235214]: Upload finished. err: <nil> Jul 5 15:18:10.381454266 (294h7m45.152095611s) [upload=235227]: Starting upload Jul 5 15:18:10.381524069 (294h7m45.152165414s) [upload=235227]: Done starting upload Jul 5 15:18:10.381621115 (294h7m45.15226245s) [upload=235227]: Uploading segments... Jul 5 15:18:10.381740133 (294h7m45.152381468s) [upload=235227]: Got next segment. Inline: false Jul 5 15:18:10.381776342 (294h7m45.152417677s) [upload=235227]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 5 15:18:10.381781973 (294h7m45.152423308s) [upload=235227]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 5 15:18:10.613904835 (294h7m45.38454617s) [seg_pos={0 0}, upload=235227]: Begin upload segment... Jul 5 15:18:10.613947397 (294h7m45.384588732s) [seg_pos={0 0}, upload=235227]: Done begin upload segment. Jul 5 15:18:10.613968548 (294h7m45.384609883s) [noise=true , address=104.36.82.15:4259 , piece_id=LQKQ5MGKXHTPITBR2QAOQ5CX76GEVJGHJT3ZD3G4PFNN5IBTCJBQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.613995038 (294h7m45.384636373s) [noise=true , address=104.36.82.16:4295 , piece_id=VEL6CAUOJKMRN7A7M66ZPDLCMJVD555LZPHM53A24MDQJQOW7ECQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614303969 (294h7m45.384945304s) [noise=true , address=104.36.82.9:4006 , piece_id=47FME2B5NS6WMOHC2ZGE2FZVSPEQUXKYT6E7AOSANAE5XZVDH5VQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614367721 (294h7m45.385009056s) [noise=true , address=45.91.116.4:4937 , piece_id=YMKIHJET2EN3QHEK665ACZNQNHJ3UGE4H5T4DIHWJUHJQBYO7B7Q, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614619272 (294h7m45.385260607s) [noise=true , address=77.90.31.17:4771 , piece_id=V7M65ZHCGXPD2GVJL6LTB4NP3QN6QLSIBM7U34QAYPAIKMN6EBPA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614637758 (294h7m45.385279083s) [noise=true , address=45.91.116.14:4191 , piece_id=WH2BCI36GGFF4QUD4XDYBBDMUR2R2FFVYKFTZD3ZWXPJCFJ3D7HQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614890251 (294h7m45.385531576s) [noise=true , address=104.36.82.3:4767 , piece_id=OGHG4G5ZVC7GVY4XI6DO4KINXF7THHL6IHGDB6DAPDNUDOUFSC4A, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615197719 (294h7m45.385839054s) [noise=true , address=104.36.82.12:4149 , piece_id=YBTLQEWDMFK6JIFTYR3VJDNMXTS3NMWWHX2MPP53BM2UBDI3UUEA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615413552 (294h7m45.386054887s) [noise=true , address=77.90.31.18:4805 , piece_id=I23F6KYUQ7TJI4ZLEANACVNYJDMPKKMMACIZAIA6PC5S3CKQPV2Q, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615608124 (294h7m45.386249459s) [noise=true , address=77.90.31.21:4877 , piece_id=OJBIFN2TJPQ45LQLYCMPKV5UZM6Y2MXNOBT3TSGX26U4SZAL5Q7A, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615803147 (294h7m45.386444472s) [noise=true , address=104.36.82.8:4970 , piece_id=2YJN7VKTTTCMAWQW2MNR65DAJPFMHNIBIW5NWYNG4R6HNPRFH6NQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615998511 (294h7m45.386639836s) [noise=true , address=77.90.31.16:4733 , piece_id=3GQ7NEWE6WZJMQUA3GG3GSZYXEFNOAUEVWHOEH4TSNONYXUT43GQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616193414 (294h7m45.386834749s) [noise=true , address=77.90.31.19:4842 , piece_id=R4FTRBZ3PETCA6QGU6RPUEDZZBWXF7YVH2COEW4WH3YWDLU3VJMA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616449093 (294h7m45.387090429s) [noise=true , address=77.90.31.11:4554 , piece_id=4QRZ5MPQ7HTPRIVRL3O3H4OV7CQA7ESYT5RCPI3GTHXJOHOSAOVA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616635840 (294h7m45.387277175s) [noise=true , address=45.91.116.5:4298 , piece_id=TVTUGWHFTXB32GBY4LYXT3YDB5LIFXEGA7VO5BHFA24Y6EMDTPHQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616818790 (294h7m45.387460125s) [noise=true , address=104.36.82.13:4185 , piece_id=LTQS2CUZ5NRJCBOPEXDKUQ2IJGOIW3JJAPHWTCW7WUMQZPIWRXRA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:12.265123988 (294h7m47.035765313s) [upload=235227]: Next returned nil segment Jul 5 15:18:12.265138164 (294h7m47.035779499s) [upload=235227]: Waiting for error group managing segments... Jul 5 15:18:12.550465694 (294h7m47.321107019s) [noise=true , address=104.36.82.8:4970 , piece_id=2YJN7VKTTTCMAWQW2MNR65DAJPFMHNIBIW5NWYNG4R6HNPRFH6NQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.554315629 (294h7m47.324956954s) [noise=true , address=104.36.82.12:4149 , piece_id=YBTLQEWDMFK6JIFTYR3VJDNMXTS3NMWWHX2MPP53BM2UBDI3UUEA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.603452512 (294h7m47.374093847s) [noise=true , address=104.36.82.3:4767 , piece_id=OGHG4G5ZVC7GVY4XI6DO4KINXF7THHL6IHGDB6DAPDNUDOUFSC4A, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.639796643 (294h7m47.410437978s) [noise=true , address=104.36.82.9:4006 , piece_id=47FME2B5NS6WMOHC2ZGE2FZVSPEQUXKYT6E7AOSANAE5XZVDH5VQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.639894960 (294h7m47.410536285s) [noise=true , address=104.36.82.15:4259 , piece_id=LQKQ5MGKXHTPITBR2QAOQ5CX76GEVJGHJT3ZD3G4PFNN5IBTCJBQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.647222659 (294h7m47.417863994s) [noise=true , address=104.36.82.16:4295 , piece_id=VEL6CAUOJKMRN7A7M66ZPDLCMJVD555LZPHM53A24MDQJQOW7ECQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.665424010 (294h7m47.436065346s) [noise=true , address=104.36.82.13:4185 , piece_id=LTQS2CUZ5NRJCBOPEXDKUQ2IJGOIW3JJAPHWTCW7WUMQZPIWRXRA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.702209574 (294h7m47.47285091s) [noise=true , address=45.91.116.14:4191 , piece_id=WH2BCI36GGFF4QUD4XDYBBDMUR2R2FFVYKFTZD3ZWXPJCFJ3D7HQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.722073145 (294h7m47.49271448s) [noise=true , address=45.91.116.5:4298 , piece_id=TVTUGWHFTXB32GBY4LYXT3YDB5LIFXEGA7VO5BHFA24Y6EMDTPHQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.788621529 (294h7m47.559262864s) [noise=true , address=45.91.116.4:4937 , piece_id=YMKIHJET2EN3QHEK665ACZNQNHJ3UGE4H5T4DIHWJUHJQBYO7B7Q, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:36.752254721 (294h8m11.522896046s) [noise=true , address=77.90.31.19:4842 , piece_id=R4FTRBZ3PETCA6QGU6RPUEDZZBWXF7YVH2COEW4WH3YWDLU3VJMA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:37.783437003 (294h8m12.554078338s) [noise=true , address=77.90.31.17:4771 , piece_id=V7M65ZHCGXPD2GVJL6LTB4NP3QN6QLSIBM7U34QAYPAIKMN6EBPA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:39.733398290 (294h8m14.504039626s) [noise=true , address=77.90.31.18:4805 , piece_id=I23F6KYUQ7TJI4ZLEANACVNYJDMPKKMMACIZAIA6PC5S3CKQPV2Q, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Jul 5 15:18:39.733464387 (294h8m14.504105722s) [noise=true , address=77.90.31.21:4877 , piece_id=OJBIFN2TJPQ45LQLYCMPKV5UZM6Y2MXNOBT3TSGX26U4SZAL5Q7A, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: expected piece hash; context deadline exceeded; EOF Jul 5 15:18:39.733542065 (294h8m14.50418339s) [noise=true , address=77.90.31.11:4554 , piece_id=4QRZ5MPQ7HTPRIVRL3O3H4OV7CQA7ESYT5RCPI3GTHXJOHOSAOVA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection Jul 5 15:18:39.733547496 (294h8m14.504188831s) [noise=true , address=77.90.31.16:4733 , piece_id=3GQ7NEWE6WZJMQUA3GG3GSZYXEFNOAUEVWHOEH4TSNONYXUT43GQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection; write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection Jul 5 15:18:39.733583284 (294h8m14.504224619s) [seg_pos={0 0}, upload=235227]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 5 15:18:39.974321220 (294h8m14.744962555s) [upload=235227]: Done uploading segments... Jul 5 15:18:39.974353041 (294h8m14.744994376s) [upload=235227]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.674718795s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 5.619308984s FAIL (2025-07-05 16:00:29 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== JYE6FC5T5QIBUQIILGDHNGXAUDBUIAOSKYFMPBCWPHP2SDJ3A, seg_pos={0 0}, upload=235214]: Done uploading piece. err: <nil> Jul 5 15:17:12.446386719 (294h6m47.217028064s) [seg_pos={0 0}, upload=235214]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 5 15:17:12.446476771 (294h6m47.217118106s) [upload=235214]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 5 15:17:12.446483043 (294h6m47.217124378s) [upload=235214]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 5 15:17:12.905279085 (294h6m47.67592042s) [upload=235214]: Done uploading segments... Jul 5 15:17:12.905303211 (294h6m47.675944546s) [upload=235214]: Upload finished. err: <nil> Jul 5 15:18:10.381454266 (294h7m45.152095611s) [upload=235227]: Starting upload Jul 5 15:18:10.381524069 (294h7m45.152165414s) [upload=235227]: Done starting upload Jul 5 15:18:10.381621115 (294h7m45.15226245s) [upload=235227]: Uploading segments... Jul 5 15:18:10.381740133 (294h7m45.152381468s) [upload=235227]: Got next segment. Inline: false Jul 5 15:18:10.381776342 (294h7m45.152417677s) [upload=235227]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 5 15:18:10.381781973 (294h7m45.152423308s) [upload=235227]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 5 15:18:10.613904835 (294h7m45.38454617s) [seg_pos={0 0}, upload=235227]: Begin upload segment... Jul 5 15:18:10.613947397 (294h7m45.384588732s) [seg_pos={0 0}, upload=235227]: Done begin upload segment. Jul 5 15:18:10.613968548 (294h7m45.384609883s) [noise=true , address=104.36.82.15:4259 , piece_id=LQKQ5MGKXHTPITBR2QAOQ5CX76GEVJGHJT3ZD3G4PFNN5IBTCJBQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.613995038 (294h7m45.384636373s) [noise=true , address=104.36.82.16:4295 , piece_id=VEL6CAUOJKMRN7A7M66ZPDLCMJVD555LZPHM53A24MDQJQOW7ECQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614303969 (294h7m45.384945304s) [noise=true , address=104.36.82.9:4006 , piece_id=47FME2B5NS6WMOHC2ZGE2FZVSPEQUXKYT6E7AOSANAE5XZVDH5VQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614367721 (294h7m45.385009056s) [noise=true , address=45.91.116.4:4937 , piece_id=YMKIHJET2EN3QHEK665ACZNQNHJ3UGE4H5T4DIHWJUHJQBYO7B7Q, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614619272 (294h7m45.385260607s) [noise=true , address=77.90.31.17:4771 , piece_id=V7M65ZHCGXPD2GVJL6LTB4NP3QN6QLSIBM7U34QAYPAIKMN6EBPA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614637758 (294h7m45.385279083s) [noise=true , address=45.91.116.14:4191 , piece_id=WH2BCI36GGFF4QUD4XDYBBDMUR2R2FFVYKFTZD3ZWXPJCFJ3D7HQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614890251 (294h7m45.385531576s) [noise=true , address=104.36.82.3:4767 , piece_id=OGHG4G5ZVC7GVY4XI6DO4KINXF7THHL6IHGDB6DAPDNUDOUFSC4A, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615197719 (294h7m45.385839054s) [noise=true , address=104.36.82.12:4149 , piece_id=YBTLQEWDMFK6JIFTYR3VJDNMXTS3NMWWHX2MPP53BM2UBDI3UUEA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615413552 (294h7m45.386054887s) [noise=true , address=77.90.31.18:4805 , piece_id=I23F6KYUQ7TJI4ZLEANACVNYJDMPKKMMACIZAIA6PC5S3CKQPV2Q, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615608124 (294h7m45.386249459s) [noise=true , address=77.90.31.21:4877 , piece_id=OJBIFN2TJPQ45LQLYCMPKV5UZM6Y2MXNOBT3TSGX26U4SZAL5Q7A, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615803147 (294h7m45.386444472s) [noise=true , address=104.36.82.8:4970 , piece_id=2YJN7VKTTTCMAWQW2MNR65DAJPFMHNIBIW5NWYNG4R6HNPRFH6NQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615998511 (294h7m45.386639836s) [noise=true , address=77.90.31.16:4733 , piece_id=3GQ7NEWE6WZJMQUA3GG3GSZYXEFNOAUEVWHOEH4TSNONYXUT43GQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616193414 (294h7m45.386834749s) [noise=true , address=77.90.31.19:4842 , piece_id=R4FTRBZ3PETCA6QGU6RPUEDZZBWXF7YVH2COEW4WH3YWDLU3VJMA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616449093 (294h7m45.387090429s) [noise=true , address=77.90.31.11:4554 , piece_id=4QRZ5MPQ7HTPRIVRL3O3H4OV7CQA7ESYT5RCPI3GTHXJOHOSAOVA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616635840 (294h7m45.387277175s) [noise=true , address=45.91.116.5:4298 , piece_id=TVTUGWHFTXB32GBY4LYXT3YDB5LIFXEGA7VO5BHFA24Y6EMDTPHQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616818790 (294h7m45.387460125s) [noise=true , address=104.36.82.13:4185 , piece_id=LTQS2CUZ5NRJCBOPEXDKUQ2IJGOIW3JJAPHWTCW7WUMQZPIWRXRA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:12.265123988 (294h7m47.035765313s) [upload=235227]: Next returned nil segment Jul 5 15:18:12.265138164 (294h7m47.035779499s) [upload=235227]: Waiting for error group managing segments... Jul 5 15:18:12.550465694 (294h7m47.321107019s) [noise=true , address=104.36.82.8:4970 , piece_id=2YJN7VKTTTCMAWQW2MNR65DAJPFMHNIBIW5NWYNG4R6HNPRFH6NQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.554315629 (294h7m47.324956954s) [noise=true , address=104.36.82.12:4149 , piece_id=YBTLQEWDMFK6JIFTYR3VJDNMXTS3NMWWHX2MPP53BM2UBDI3UUEA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.603452512 (294h7m47.374093847s) [noise=true , address=104.36.82.3:4767 , piece_id=OGHG4G5ZVC7GVY4XI6DO4KINXF7THHL6IHGDB6DAPDNUDOUFSC4A, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.639796643 (294h7m47.410437978s) [noise=true , address=104.36.82.9:4006 , piece_id=47FME2B5NS6WMOHC2ZGE2FZVSPEQUXKYT6E7AOSANAE5XZVDH5VQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.639894960 (294h7m47.410536285s) [noise=true , address=104.36.82.15:4259 , piece_id=LQKQ5MGKXHTPITBR2QAOQ5CX76GEVJGHJT3ZD3G4PFNN5IBTCJBQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.647222659 (294h7m47.417863994s) [noise=true , address=104.36.82.16:4295 , piece_id=VEL6CAUOJKMRN7A7M66ZPDLCMJVD555LZPHM53A24MDQJQOW7ECQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.665424010 (294h7m47.436065346s) [noise=true , address=104.36.82.13:4185 , piece_id=LTQS2CUZ5NRJCBOPEXDKUQ2IJGOIW3JJAPHWTCW7WUMQZPIWRXRA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.702209574 (294h7m47.47285091s) [noise=true , address=45.91.116.14:4191 , piece_id=WH2BCI36GGFF4QUD4XDYBBDMUR2R2FFVYKFTZD3ZWXPJCFJ3D7HQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.722073145 (294h7m47.49271448s) [noise=true , address=45.91.116.5:4298 , piece_id=TVTUGWHFTXB32GBY4LYXT3YDB5LIFXEGA7VO5BHFA24Y6EMDTPHQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.788621529 (294h7m47.559262864s) [noise=true , address=45.91.116.4:4937 , piece_id=YMKIHJET2EN3QHEK665ACZNQNHJ3UGE4H5T4DIHWJUHJQBYO7B7Q, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:36.752254721 (294h8m11.522896046s) [noise=true , address=77.90.31.19:4842 , piece_id=R4FTRBZ3PETCA6QGU6RPUEDZZBWXF7YVH2COEW4WH3YWDLU3VJMA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:37.783437003 (294h8m12.554078338s) [noise=true , address=77.90.31.17:4771 , piece_id=V7M65ZHCGXPD2GVJL6LTB4NP3QN6QLSIBM7U34QAYPAIKMN6EBPA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:39.733398290 (294h8m14.504039626s) [noise=true , address=77.90.31.18:4805 , piece_id=I23F6KYUQ7TJI4ZLEANACVNYJDMPKKMMACIZAIA6PC5S3CKQPV2Q, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Jul 5 15:18:39.733464387 (294h8m14.504105722s) [noise=true , address=77.90.31.21:4877 , piece_id=OJBIFN2TJPQ45LQLYCMPKV5UZM6Y2MXNOBT3TSGX26U4SZAL5Q7A, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: expected piece hash; context deadline exceeded; EOF Jul 5 15:18:39.733542065 (294h8m14.50418339s) [noise=true , address=77.90.31.11:4554 , piece_id=4QRZ5MPQ7HTPRIVRL3O3H4OV7CQA7ESYT5RCPI3GTHXJOHOSAOVA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection Jul 5 15:18:39.733547496 (294h8m14.504188831s) [noise=true , address=77.90.31.16:4733 , piece_id=3GQ7NEWE6WZJMQUA3GG3GSZYXEFNOAUEVWHOEH4TSNONYXUT43GQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection; write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection Jul 5 15:18:39.733583284 (294h8m14.504224619s) [seg_pos={0 0}, upload=235227]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 5 15:18:39.974321220 (294h8m14.744962555s) [upload=235227]: Done uploading segments... Jul 5 15:18:39.974353041 (294h8m14.744994376s) [upload=235227]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.619342058s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 5.826228742s FAIL (2025-07-05 15:59:25 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== JYE6FC5T5QIBUQIILGDHNGXAUDBUIAOSKYFMPBCWPHP2SDJ3A, seg_pos={0 0}, upload=235214]: Done uploading piece. err: <nil> Jul 5 15:17:12.446386719 (294h6m47.217028064s) [seg_pos={0 0}, upload=235214]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 5 15:17:12.446476771 (294h6m47.217118106s) [upload=235214]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 5 15:17:12.446483043 (294h6m47.217124378s) [upload=235214]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 5 15:17:12.905279085 (294h6m47.67592042s) [upload=235214]: Done uploading segments... Jul 5 15:17:12.905303211 (294h6m47.675944546s) [upload=235214]: Upload finished. err: <nil> Jul 5 15:18:10.381454266 (294h7m45.152095611s) [upload=235227]: Starting upload Jul 5 15:18:10.381524069 (294h7m45.152165414s) [upload=235227]: Done starting upload Jul 5 15:18:10.381621115 (294h7m45.15226245s) [upload=235227]: Uploading segments... Jul 5 15:18:10.381740133 (294h7m45.152381468s) [upload=235227]: Got next segment. Inline: false Jul 5 15:18:10.381776342 (294h7m45.152417677s) [upload=235227]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 5 15:18:10.381781973 (294h7m45.152423308s) [upload=235227]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 5 15:18:10.613904835 (294h7m45.38454617s) [seg_pos={0 0}, upload=235227]: Begin upload segment... Jul 5 15:18:10.613947397 (294h7m45.384588732s) [seg_pos={0 0}, upload=235227]: Done begin upload segment. Jul 5 15:18:10.613968548 (294h7m45.384609883s) [noise=true , address=104.36.82.15:4259 , piece_id=LQKQ5MGKXHTPITBR2QAOQ5CX76GEVJGHJT3ZD3G4PFNN5IBTCJBQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.613995038 (294h7m45.384636373s) [noise=true , address=104.36.82.16:4295 , piece_id=VEL6CAUOJKMRN7A7M66ZPDLCMJVD555LZPHM53A24MDQJQOW7ECQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614303969 (294h7m45.384945304s) [noise=true , address=104.36.82.9:4006 , piece_id=47FME2B5NS6WMOHC2ZGE2FZVSPEQUXKYT6E7AOSANAE5XZVDH5VQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614367721 (294h7m45.385009056s) [noise=true , address=45.91.116.4:4937 , piece_id=YMKIHJET2EN3QHEK665ACZNQNHJ3UGE4H5T4DIHWJUHJQBYO7B7Q, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614619272 (294h7m45.385260607s) [noise=true , address=77.90.31.17:4771 , piece_id=V7M65ZHCGXPD2GVJL6LTB4NP3QN6QLSIBM7U34QAYPAIKMN6EBPA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614637758 (294h7m45.385279083s) [noise=true , address=45.91.116.14:4191 , piece_id=WH2BCI36GGFF4QUD4XDYBBDMUR2R2FFVYKFTZD3ZWXPJCFJ3D7HQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614890251 (294h7m45.385531576s) [noise=true , address=104.36.82.3:4767 , piece_id=OGHG4G5ZVC7GVY4XI6DO4KINXF7THHL6IHGDB6DAPDNUDOUFSC4A, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615197719 (294h7m45.385839054s) [noise=true , address=104.36.82.12:4149 , piece_id=YBTLQEWDMFK6JIFTYR3VJDNMXTS3NMWWHX2MPP53BM2UBDI3UUEA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615413552 (294h7m45.386054887s) [noise=true , address=77.90.31.18:4805 , piece_id=I23F6KYUQ7TJI4ZLEANACVNYJDMPKKMMACIZAIA6PC5S3CKQPV2Q, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615608124 (294h7m45.386249459s) [noise=true , address=77.90.31.21:4877 , piece_id=OJBIFN2TJPQ45LQLYCMPKV5UZM6Y2MXNOBT3TSGX26U4SZAL5Q7A, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615803147 (294h7m45.386444472s) [noise=true , address=104.36.82.8:4970 , piece_id=2YJN7VKTTTCMAWQW2MNR65DAJPFMHNIBIW5NWYNG4R6HNPRFH6NQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615998511 (294h7m45.386639836s) [noise=true , address=77.90.31.16:4733 , piece_id=3GQ7NEWE6WZJMQUA3GG3GSZYXEFNOAUEVWHOEH4TSNONYXUT43GQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616193414 (294h7m45.386834749s) [noise=true , address=77.90.31.19:4842 , piece_id=R4FTRBZ3PETCA6QGU6RPUEDZZBWXF7YVH2COEW4WH3YWDLU3VJMA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616449093 (294h7m45.387090429s) [noise=true , address=77.90.31.11:4554 , piece_id=4QRZ5MPQ7HTPRIVRL3O3H4OV7CQA7ESYT5RCPI3GTHXJOHOSAOVA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616635840 (294h7m45.387277175s) [noise=true , address=45.91.116.5:4298 , piece_id=TVTUGWHFTXB32GBY4LYXT3YDB5LIFXEGA7VO5BHFA24Y6EMDTPHQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616818790 (294h7m45.387460125s) [noise=true , address=104.36.82.13:4185 , piece_id=LTQS2CUZ5NRJCBOPEXDKUQ2IJGOIW3JJAPHWTCW7WUMQZPIWRXRA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:12.265123988 (294h7m47.035765313s) [upload=235227]: Next returned nil segment Jul 5 15:18:12.265138164 (294h7m47.035779499s) [upload=235227]: Waiting for error group managing segments... Jul 5 15:18:12.550465694 (294h7m47.321107019s) [noise=true , address=104.36.82.8:4970 , piece_id=2YJN7VKTTTCMAWQW2MNR65DAJPFMHNIBIW5NWYNG4R6HNPRFH6NQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.554315629 (294h7m47.324956954s) [noise=true , address=104.36.82.12:4149 , piece_id=YBTLQEWDMFK6JIFTYR3VJDNMXTS3NMWWHX2MPP53BM2UBDI3UUEA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.603452512 (294h7m47.374093847s) [noise=true , address=104.36.82.3:4767 , piece_id=OGHG4G5ZVC7GVY4XI6DO4KINXF7THHL6IHGDB6DAPDNUDOUFSC4A, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.639796643 (294h7m47.410437978s) [noise=true , address=104.36.82.9:4006 , piece_id=47FME2B5NS6WMOHC2ZGE2FZVSPEQUXKYT6E7AOSANAE5XZVDH5VQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.639894960 (294h7m47.410536285s) [noise=true , address=104.36.82.15:4259 , piece_id=LQKQ5MGKXHTPITBR2QAOQ5CX76GEVJGHJT3ZD3G4PFNN5IBTCJBQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.647222659 (294h7m47.417863994s) [noise=true , address=104.36.82.16:4295 , piece_id=VEL6CAUOJKMRN7A7M66ZPDLCMJVD555LZPHM53A24MDQJQOW7ECQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.665424010 (294h7m47.436065346s) [noise=true , address=104.36.82.13:4185 , piece_id=LTQS2CUZ5NRJCBOPEXDKUQ2IJGOIW3JJAPHWTCW7WUMQZPIWRXRA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.702209574 (294h7m47.47285091s) [noise=true , address=45.91.116.14:4191 , piece_id=WH2BCI36GGFF4QUD4XDYBBDMUR2R2FFVYKFTZD3ZWXPJCFJ3D7HQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.722073145 (294h7m47.49271448s) [noise=true , address=45.91.116.5:4298 , piece_id=TVTUGWHFTXB32GBY4LYXT3YDB5LIFXEGA7VO5BHFA24Y6EMDTPHQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.788621529 (294h7m47.559262864s) [noise=true , address=45.91.116.4:4937 , piece_id=YMKIHJET2EN3QHEK665ACZNQNHJ3UGE4H5T4DIHWJUHJQBYO7B7Q, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:36.752254721 (294h8m11.522896046s) [noise=true , address=77.90.31.19:4842 , piece_id=R4FTRBZ3PETCA6QGU6RPUEDZZBWXF7YVH2COEW4WH3YWDLU3VJMA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:37.783437003 (294h8m12.554078338s) [noise=true , address=77.90.31.17:4771 , piece_id=V7M65ZHCGXPD2GVJL6LTB4NP3QN6QLSIBM7U34QAYPAIKMN6EBPA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:39.733398290 (294h8m14.504039626s) [noise=true , address=77.90.31.18:4805 , piece_id=I23F6KYUQ7TJI4ZLEANACVNYJDMPKKMMACIZAIA6PC5S3CKQPV2Q, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Jul 5 15:18:39.733464387 (294h8m14.504105722s) [noise=true , address=77.90.31.21:4877 , piece_id=OJBIFN2TJPQ45LQLYCMPKV5UZM6Y2MXNOBT3TSGX26U4SZAL5Q7A, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: expected piece hash; context deadline exceeded; EOF Jul 5 15:18:39.733542065 (294h8m14.50418339s) [noise=true , address=77.90.31.11:4554 , piece_id=4QRZ5MPQ7HTPRIVRL3O3H4OV7CQA7ESYT5RCPI3GTHXJOHOSAOVA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection Jul 5 15:18:39.733547496 (294h8m14.504188831s) [noise=true , address=77.90.31.16:4733 , piece_id=3GQ7NEWE6WZJMQUA3GG3GSZYXEFNOAUEVWHOEH4TSNONYXUT43GQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection; write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection Jul 5 15:18:39.733583284 (294h8m14.504224619s) [seg_pos={0 0}, upload=235227]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 5 15:18:39.974321220 (294h8m14.744962555s) [upload=235227]: Done uploading segments... Jul 5 15:18:39.974353041 (294h8m14.744994376s) [upload=235227]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.826254844s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 5.66965111s FAIL (2025-07-05 15:59:01 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== JYE6FC5T5QIBUQIILGDHNGXAUDBUIAOSKYFMPBCWPHP2SDJ3A, seg_pos={0 0}, upload=235214]: Done uploading piece. err: <nil> Jul 5 15:17:12.446386719 (294h6m47.217028064s) [seg_pos={0 0}, upload=235214]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 5 15:17:12.446476771 (294h6m47.217118106s) [upload=235214]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 5 15:17:12.446483043 (294h6m47.217124378s) [upload=235214]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 5 15:17:12.905279085 (294h6m47.67592042s) [upload=235214]: Done uploading segments... Jul 5 15:17:12.905303211 (294h6m47.675944546s) [upload=235214]: Upload finished. err: <nil> Jul 5 15:18:10.381454266 (294h7m45.152095611s) [upload=235227]: Starting upload Jul 5 15:18:10.381524069 (294h7m45.152165414s) [upload=235227]: Done starting upload Jul 5 15:18:10.381621115 (294h7m45.15226245s) [upload=235227]: Uploading segments... Jul 5 15:18:10.381740133 (294h7m45.152381468s) [upload=235227]: Got next segment. Inline: false Jul 5 15:18:10.381776342 (294h7m45.152417677s) [upload=235227]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 5 15:18:10.381781973 (294h7m45.152423308s) [upload=235227]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 5 15:18:10.613904835 (294h7m45.38454617s) [seg_pos={0 0}, upload=235227]: Begin upload segment... Jul 5 15:18:10.613947397 (294h7m45.384588732s) [seg_pos={0 0}, upload=235227]: Done begin upload segment. Jul 5 15:18:10.613968548 (294h7m45.384609883s) [noise=true , address=104.36.82.15:4259 , piece_id=LQKQ5MGKXHTPITBR2QAOQ5CX76GEVJGHJT3ZD3G4PFNN5IBTCJBQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.613995038 (294h7m45.384636373s) [noise=true , address=104.36.82.16:4295 , piece_id=VEL6CAUOJKMRN7A7M66ZPDLCMJVD555LZPHM53A24MDQJQOW7ECQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614303969 (294h7m45.384945304s) [noise=true , address=104.36.82.9:4006 , piece_id=47FME2B5NS6WMOHC2ZGE2FZVSPEQUXKYT6E7AOSANAE5XZVDH5VQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614367721 (294h7m45.385009056s) [noise=true , address=45.91.116.4:4937 , piece_id=YMKIHJET2EN3QHEK665ACZNQNHJ3UGE4H5T4DIHWJUHJQBYO7B7Q, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614619272 (294h7m45.385260607s) [noise=true , address=77.90.31.17:4771 , piece_id=V7M65ZHCGXPD2GVJL6LTB4NP3QN6QLSIBM7U34QAYPAIKMN6EBPA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614637758 (294h7m45.385279083s) [noise=true , address=45.91.116.14:4191 , piece_id=WH2BCI36GGFF4QUD4XDYBBDMUR2R2FFVYKFTZD3ZWXPJCFJ3D7HQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614890251 (294h7m45.385531576s) [noise=true , address=104.36.82.3:4767 , piece_id=OGHG4G5ZVC7GVY4XI6DO4KINXF7THHL6IHGDB6DAPDNUDOUFSC4A, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615197719 (294h7m45.385839054s) [noise=true , address=104.36.82.12:4149 , piece_id=YBTLQEWDMFK6JIFTYR3VJDNMXTS3NMWWHX2MPP53BM2UBDI3UUEA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615413552 (294h7m45.386054887s) [noise=true , address=77.90.31.18:4805 , piece_id=I23F6KYUQ7TJI4ZLEANACVNYJDMPKKMMACIZAIA6PC5S3CKQPV2Q, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615608124 (294h7m45.386249459s) [noise=true , address=77.90.31.21:4877 , piece_id=OJBIFN2TJPQ45LQLYCMPKV5UZM6Y2MXNOBT3TSGX26U4SZAL5Q7A, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615803147 (294h7m45.386444472s) [noise=true , address=104.36.82.8:4970 , piece_id=2YJN7VKTTTCMAWQW2MNR65DAJPFMHNIBIW5NWYNG4R6HNPRFH6NQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615998511 (294h7m45.386639836s) [noise=true , address=77.90.31.16:4733 , piece_id=3GQ7NEWE6WZJMQUA3GG3GSZYXEFNOAUEVWHOEH4TSNONYXUT43GQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616193414 (294h7m45.386834749s) [noise=true , address=77.90.31.19:4842 , piece_id=R4FTRBZ3PETCA6QGU6RPUEDZZBWXF7YVH2COEW4WH3YWDLU3VJMA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616449093 (294h7m45.387090429s) [noise=true , address=77.90.31.11:4554 , piece_id=4QRZ5MPQ7HTPRIVRL3O3H4OV7CQA7ESYT5RCPI3GTHXJOHOSAOVA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616635840 (294h7m45.387277175s) [noise=true , address=45.91.116.5:4298 , piece_id=TVTUGWHFTXB32GBY4LYXT3YDB5LIFXEGA7VO5BHFA24Y6EMDTPHQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616818790 (294h7m45.387460125s) [noise=true , address=104.36.82.13:4185 , piece_id=LTQS2CUZ5NRJCBOPEXDKUQ2IJGOIW3JJAPHWTCW7WUMQZPIWRXRA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:12.265123988 (294h7m47.035765313s) [upload=235227]: Next returned nil segment Jul 5 15:18:12.265138164 (294h7m47.035779499s) [upload=235227]: Waiting for error group managing segments... Jul 5 15:18:12.550465694 (294h7m47.321107019s) [noise=true , address=104.36.82.8:4970 , piece_id=2YJN7VKTTTCMAWQW2MNR65DAJPFMHNIBIW5NWYNG4R6HNPRFH6NQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.554315629 (294h7m47.324956954s) [noise=true , address=104.36.82.12:4149 , piece_id=YBTLQEWDMFK6JIFTYR3VJDNMXTS3NMWWHX2MPP53BM2UBDI3UUEA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.603452512 (294h7m47.374093847s) [noise=true , address=104.36.82.3:4767 , piece_id=OGHG4G5ZVC7GVY4XI6DO4KINXF7THHL6IHGDB6DAPDNUDOUFSC4A, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.639796643 (294h7m47.410437978s) [noise=true , address=104.36.82.9:4006 , piece_id=47FME2B5NS6WMOHC2ZGE2FZVSPEQUXKYT6E7AOSANAE5XZVDH5VQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.639894960 (294h7m47.410536285s) [noise=true , address=104.36.82.15:4259 , piece_id=LQKQ5MGKXHTPITBR2QAOQ5CX76GEVJGHJT3ZD3G4PFNN5IBTCJBQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.647222659 (294h7m47.417863994s) [noise=true , address=104.36.82.16:4295 , piece_id=VEL6CAUOJKMRN7A7M66ZPDLCMJVD555LZPHM53A24MDQJQOW7ECQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.665424010 (294h7m47.436065346s) [noise=true , address=104.36.82.13:4185 , piece_id=LTQS2CUZ5NRJCBOPEXDKUQ2IJGOIW3JJAPHWTCW7WUMQZPIWRXRA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.702209574 (294h7m47.47285091s) [noise=true , address=45.91.116.14:4191 , piece_id=WH2BCI36GGFF4QUD4XDYBBDMUR2R2FFVYKFTZD3ZWXPJCFJ3D7HQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.722073145 (294h7m47.49271448s) [noise=true , address=45.91.116.5:4298 , piece_id=TVTUGWHFTXB32GBY4LYXT3YDB5LIFXEGA7VO5BHFA24Y6EMDTPHQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.788621529 (294h7m47.559262864s) [noise=true , address=45.91.116.4:4937 , piece_id=YMKIHJET2EN3QHEK665ACZNQNHJ3UGE4H5T4DIHWJUHJQBYO7B7Q, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:36.752254721 (294h8m11.522896046s) [noise=true , address=77.90.31.19:4842 , piece_id=R4FTRBZ3PETCA6QGU6RPUEDZZBWXF7YVH2COEW4WH3YWDLU3VJMA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:37.783437003 (294h8m12.554078338s) [noise=true , address=77.90.31.17:4771 , piece_id=V7M65ZHCGXPD2GVJL6LTB4NP3QN6QLSIBM7U34QAYPAIKMN6EBPA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:39.733398290 (294h8m14.504039626s) [noise=true , address=77.90.31.18:4805 , piece_id=I23F6KYUQ7TJI4ZLEANACVNYJDMPKKMMACIZAIA6PC5S3CKQPV2Q, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Jul 5 15:18:39.733464387 (294h8m14.504105722s) [noise=true , address=77.90.31.21:4877 , piece_id=OJBIFN2TJPQ45LQLYCMPKV5UZM6Y2MXNOBT3TSGX26U4SZAL5Q7A, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: expected piece hash; context deadline exceeded; EOF Jul 5 15:18:39.733542065 (294h8m14.50418339s) [noise=true , address=77.90.31.11:4554 , piece_id=4QRZ5MPQ7HTPRIVRL3O3H4OV7CQA7ESYT5RCPI3GTHXJOHOSAOVA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection Jul 5 15:18:39.733547496 (294h8m14.504188831s) [noise=true , address=77.90.31.16:4733 , piece_id=3GQ7NEWE6WZJMQUA3GG3GSZYXEFNOAUEVWHOEH4TSNONYXUT43GQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection; write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection Jul 5 15:18:39.733583284 (294h8m14.504224619s) [seg_pos={0 0}, upload=235227]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 5 15:18:39.974321220 (294h8m14.744962555s) [upload=235227]: Done uploading segments... Jul 5 15:18:39.974353041 (294h8m14.744994376s) [upload=235227]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.669675919s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 5.658738896s FAIL (2025-07-05 15:58:41 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== JYE6FC5T5QIBUQIILGDHNGXAUDBUIAOSKYFMPBCWPHP2SDJ3A, seg_pos={0 0}, upload=235214]: Done uploading piece. err: <nil> Jul 5 15:17:12.446386719 (294h6m47.217028064s) [seg_pos={0 0}, upload=235214]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 5 15:17:12.446476771 (294h6m47.217118106s) [upload=235214]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 5 15:17:12.446483043 (294h6m47.217124378s) [upload=235214]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 5 15:17:12.905279085 (294h6m47.67592042s) [upload=235214]: Done uploading segments... Jul 5 15:17:12.905303211 (294h6m47.675944546s) [upload=235214]: Upload finished. err: <nil> Jul 5 15:18:10.381454266 (294h7m45.152095611s) [upload=235227]: Starting upload Jul 5 15:18:10.381524069 (294h7m45.152165414s) [upload=235227]: Done starting upload Jul 5 15:18:10.381621115 (294h7m45.15226245s) [upload=235227]: Uploading segments... Jul 5 15:18:10.381740133 (294h7m45.152381468s) [upload=235227]: Got next segment. Inline: false Jul 5 15:18:10.381776342 (294h7m45.152417677s) [upload=235227]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 5 15:18:10.381781973 (294h7m45.152423308s) [upload=235227]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 5 15:18:10.613904835 (294h7m45.38454617s) [seg_pos={0 0}, upload=235227]: Begin upload segment... Jul 5 15:18:10.613947397 (294h7m45.384588732s) [seg_pos={0 0}, upload=235227]: Done begin upload segment. Jul 5 15:18:10.613968548 (294h7m45.384609883s) [noise=true , address=104.36.82.15:4259 , piece_id=LQKQ5MGKXHTPITBR2QAOQ5CX76GEVJGHJT3ZD3G4PFNN5IBTCJBQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.613995038 (294h7m45.384636373s) [noise=true , address=104.36.82.16:4295 , piece_id=VEL6CAUOJKMRN7A7M66ZPDLCMJVD555LZPHM53A24MDQJQOW7ECQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614303969 (294h7m45.384945304s) [noise=true , address=104.36.82.9:4006 , piece_id=47FME2B5NS6WMOHC2ZGE2FZVSPEQUXKYT6E7AOSANAE5XZVDH5VQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614367721 (294h7m45.385009056s) [noise=true , address=45.91.116.4:4937 , piece_id=YMKIHJET2EN3QHEK665ACZNQNHJ3UGE4H5T4DIHWJUHJQBYO7B7Q, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614619272 (294h7m45.385260607s) [noise=true , address=77.90.31.17:4771 , piece_id=V7M65ZHCGXPD2GVJL6LTB4NP3QN6QLSIBM7U34QAYPAIKMN6EBPA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614637758 (294h7m45.385279083s) [noise=true , address=45.91.116.14:4191 , piece_id=WH2BCI36GGFF4QUD4XDYBBDMUR2R2FFVYKFTZD3ZWXPJCFJ3D7HQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614890251 (294h7m45.385531576s) [noise=true , address=104.36.82.3:4767 , piece_id=OGHG4G5ZVC7GVY4XI6DO4KINXF7THHL6IHGDB6DAPDNUDOUFSC4A, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615197719 (294h7m45.385839054s) [noise=true , address=104.36.82.12:4149 , piece_id=YBTLQEWDMFK6JIFTYR3VJDNMXTS3NMWWHX2MPP53BM2UBDI3UUEA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615413552 (294h7m45.386054887s) [noise=true , address=77.90.31.18:4805 , piece_id=I23F6KYUQ7TJI4ZLEANACVNYJDMPKKMMACIZAIA6PC5S3CKQPV2Q, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615608124 (294h7m45.386249459s) [noise=true , address=77.90.31.21:4877 , piece_id=OJBIFN2TJPQ45LQLYCMPKV5UZM6Y2MXNOBT3TSGX26U4SZAL5Q7A, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615803147 (294h7m45.386444472s) [noise=true , address=104.36.82.8:4970 , piece_id=2YJN7VKTTTCMAWQW2MNR65DAJPFMHNIBIW5NWYNG4R6HNPRFH6NQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615998511 (294h7m45.386639836s) [noise=true , address=77.90.31.16:4733 , piece_id=3GQ7NEWE6WZJMQUA3GG3GSZYXEFNOAUEVWHOEH4TSNONYXUT43GQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616193414 (294h7m45.386834749s) [noise=true , address=77.90.31.19:4842 , piece_id=R4FTRBZ3PETCA6QGU6RPUEDZZBWXF7YVH2COEW4WH3YWDLU3VJMA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616449093 (294h7m45.387090429s) [noise=true , address=77.90.31.11:4554 , piece_id=4QRZ5MPQ7HTPRIVRL3O3H4OV7CQA7ESYT5RCPI3GTHXJOHOSAOVA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616635840 (294h7m45.387277175s) [noise=true , address=45.91.116.5:4298 , piece_id=TVTUGWHFTXB32GBY4LYXT3YDB5LIFXEGA7VO5BHFA24Y6EMDTPHQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616818790 (294h7m45.387460125s) [noise=true , address=104.36.82.13:4185 , piece_id=LTQS2CUZ5NRJCBOPEXDKUQ2IJGOIW3JJAPHWTCW7WUMQZPIWRXRA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:12.265123988 (294h7m47.035765313s) [upload=235227]: Next returned nil segment Jul 5 15:18:12.265138164 (294h7m47.035779499s) [upload=235227]: Waiting for error group managing segments... Jul 5 15:18:12.550465694 (294h7m47.321107019s) [noise=true , address=104.36.82.8:4970 , piece_id=2YJN7VKTTTCMAWQW2MNR65DAJPFMHNIBIW5NWYNG4R6HNPRFH6NQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.554315629 (294h7m47.324956954s) [noise=true , address=104.36.82.12:4149 , piece_id=YBTLQEWDMFK6JIFTYR3VJDNMXTS3NMWWHX2MPP53BM2UBDI3UUEA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.603452512 (294h7m47.374093847s) [noise=true , address=104.36.82.3:4767 , piece_id=OGHG4G5ZVC7GVY4XI6DO4KINXF7THHL6IHGDB6DAPDNUDOUFSC4A, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.639796643 (294h7m47.410437978s) [noise=true , address=104.36.82.9:4006 , piece_id=47FME2B5NS6WMOHC2ZGE2FZVSPEQUXKYT6E7AOSANAE5XZVDH5VQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.639894960 (294h7m47.410536285s) [noise=true , address=104.36.82.15:4259 , piece_id=LQKQ5MGKXHTPITBR2QAOQ5CX76GEVJGHJT3ZD3G4PFNN5IBTCJBQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.647222659 (294h7m47.417863994s) [noise=true , address=104.36.82.16:4295 , piece_id=VEL6CAUOJKMRN7A7M66ZPDLCMJVD555LZPHM53A24MDQJQOW7ECQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.665424010 (294h7m47.436065346s) [noise=true , address=104.36.82.13:4185 , piece_id=LTQS2CUZ5NRJCBOPEXDKUQ2IJGOIW3JJAPHWTCW7WUMQZPIWRXRA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.702209574 (294h7m47.47285091s) [noise=true , address=45.91.116.14:4191 , piece_id=WH2BCI36GGFF4QUD4XDYBBDMUR2R2FFVYKFTZD3ZWXPJCFJ3D7HQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.722073145 (294h7m47.49271448s) [noise=true , address=45.91.116.5:4298 , piece_id=TVTUGWHFTXB32GBY4LYXT3YDB5LIFXEGA7VO5BHFA24Y6EMDTPHQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.788621529 (294h7m47.559262864s) [noise=true , address=45.91.116.4:4937 , piece_id=YMKIHJET2EN3QHEK665ACZNQNHJ3UGE4H5T4DIHWJUHJQBYO7B7Q, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:36.752254721 (294h8m11.522896046s) [noise=true , address=77.90.31.19:4842 , piece_id=R4FTRBZ3PETCA6QGU6RPUEDZZBWXF7YVH2COEW4WH3YWDLU3VJMA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:37.783437003 (294h8m12.554078338s) [noise=true , address=77.90.31.17:4771 , piece_id=V7M65ZHCGXPD2GVJL6LTB4NP3QN6QLSIBM7U34QAYPAIKMN6EBPA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:39.733398290 (294h8m14.504039626s) [noise=true , address=77.90.31.18:4805 , piece_id=I23F6KYUQ7TJI4ZLEANACVNYJDMPKKMMACIZAIA6PC5S3CKQPV2Q, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Jul 5 15:18:39.733464387 (294h8m14.504105722s) [noise=true , address=77.90.31.21:4877 , piece_id=OJBIFN2TJPQ45LQLYCMPKV5UZM6Y2MXNOBT3TSGX26U4SZAL5Q7A, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: expected piece hash; context deadline exceeded; EOF Jul 5 15:18:39.733542065 (294h8m14.50418339s) [noise=true , address=77.90.31.11:4554 , piece_id=4QRZ5MPQ7HTPRIVRL3O3H4OV7CQA7ESYT5RCPI3GTHXJOHOSAOVA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection Jul 5 15:18:39.733547496 (294h8m14.504188831s) [noise=true , address=77.90.31.16:4733 , piece_id=3GQ7NEWE6WZJMQUA3GG3GSZYXEFNOAUEVWHOEH4TSNONYXUT43GQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection; write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection Jul 5 15:18:39.733583284 (294h8m14.504224619s) [seg_pos={0 0}, upload=235227]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 5 15:18:39.974321220 (294h8m14.744962555s) [upload=235227]: Done uploading segments... Jul 5 15:18:39.974353041 (294h8m14.744994376s) [upload=235227]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.658770787s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 6.007556315s FAIL (2025-07-05 15:58:13 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== JYE6FC5T5QIBUQIILGDHNGXAUDBUIAOSKYFMPBCWPHP2SDJ3A, seg_pos={0 0}, upload=235214]: Done uploading piece. err: <nil> Jul 5 15:17:12.446386719 (294h6m47.217028064s) [seg_pos={0 0}, upload=235214]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 5 15:17:12.446476771 (294h6m47.217118106s) [upload=235214]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 5 15:17:12.446483043 (294h6m47.217124378s) [upload=235214]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 5 15:17:12.905279085 (294h6m47.67592042s) [upload=235214]: Done uploading segments... Jul 5 15:17:12.905303211 (294h6m47.675944546s) [upload=235214]: Upload finished. err: <nil> Jul 5 15:18:10.381454266 (294h7m45.152095611s) [upload=235227]: Starting upload Jul 5 15:18:10.381524069 (294h7m45.152165414s) [upload=235227]: Done starting upload Jul 5 15:18:10.381621115 (294h7m45.15226245s) [upload=235227]: Uploading segments... Jul 5 15:18:10.381740133 (294h7m45.152381468s) [upload=235227]: Got next segment. Inline: false Jul 5 15:18:10.381776342 (294h7m45.152417677s) [upload=235227]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 5 15:18:10.381781973 (294h7m45.152423308s) [upload=235227]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 5 15:18:10.613904835 (294h7m45.38454617s) [seg_pos={0 0}, upload=235227]: Begin upload segment... Jul 5 15:18:10.613947397 (294h7m45.384588732s) [seg_pos={0 0}, upload=235227]: Done begin upload segment. Jul 5 15:18:10.613968548 (294h7m45.384609883s) [noise=true , address=104.36.82.15:4259 , piece_id=LQKQ5MGKXHTPITBR2QAOQ5CX76GEVJGHJT3ZD3G4PFNN5IBTCJBQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.613995038 (294h7m45.384636373s) [noise=true , address=104.36.82.16:4295 , piece_id=VEL6CAUOJKMRN7A7M66ZPDLCMJVD555LZPHM53A24MDQJQOW7ECQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614303969 (294h7m45.384945304s) [noise=true , address=104.36.82.9:4006 , piece_id=47FME2B5NS6WMOHC2ZGE2FZVSPEQUXKYT6E7AOSANAE5XZVDH5VQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614367721 (294h7m45.385009056s) [noise=true , address=45.91.116.4:4937 , piece_id=YMKIHJET2EN3QHEK665ACZNQNHJ3UGE4H5T4DIHWJUHJQBYO7B7Q, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614619272 (294h7m45.385260607s) [noise=true , address=77.90.31.17:4771 , piece_id=V7M65ZHCGXPD2GVJL6LTB4NP3QN6QLSIBM7U34QAYPAIKMN6EBPA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614637758 (294h7m45.385279083s) [noise=true , address=45.91.116.14:4191 , piece_id=WH2BCI36GGFF4QUD4XDYBBDMUR2R2FFVYKFTZD3ZWXPJCFJ3D7HQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614890251 (294h7m45.385531576s) [noise=true , address=104.36.82.3:4767 , piece_id=OGHG4G5ZVC7GVY4XI6DO4KINXF7THHL6IHGDB6DAPDNUDOUFSC4A, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615197719 (294h7m45.385839054s) [noise=true , address=104.36.82.12:4149 , piece_id=YBTLQEWDMFK6JIFTYR3VJDNMXTS3NMWWHX2MPP53BM2UBDI3UUEA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615413552 (294h7m45.386054887s) [noise=true , address=77.90.31.18:4805 , piece_id=I23F6KYUQ7TJI4ZLEANACVNYJDMPKKMMACIZAIA6PC5S3CKQPV2Q, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615608124 (294h7m45.386249459s) [noise=true , address=77.90.31.21:4877 , piece_id=OJBIFN2TJPQ45LQLYCMPKV5UZM6Y2MXNOBT3TSGX26U4SZAL5Q7A, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615803147 (294h7m45.386444472s) [noise=true , address=104.36.82.8:4970 , piece_id=2YJN7VKTTTCMAWQW2MNR65DAJPFMHNIBIW5NWYNG4R6HNPRFH6NQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615998511 (294h7m45.386639836s) [noise=true , address=77.90.31.16:4733 , piece_id=3GQ7NEWE6WZJMQUA3GG3GSZYXEFNOAUEVWHOEH4TSNONYXUT43GQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616193414 (294h7m45.386834749s) [noise=true , address=77.90.31.19:4842 , piece_id=R4FTRBZ3PETCA6QGU6RPUEDZZBWXF7YVH2COEW4WH3YWDLU3VJMA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616449093 (294h7m45.387090429s) [noise=true , address=77.90.31.11:4554 , piece_id=4QRZ5MPQ7HTPRIVRL3O3H4OV7CQA7ESYT5RCPI3GTHXJOHOSAOVA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616635840 (294h7m45.387277175s) [noise=true , address=45.91.116.5:4298 , piece_id=TVTUGWHFTXB32GBY4LYXT3YDB5LIFXEGA7VO5BHFA24Y6EMDTPHQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616818790 (294h7m45.387460125s) [noise=true , address=104.36.82.13:4185 , piece_id=LTQS2CUZ5NRJCBOPEXDKUQ2IJGOIW3JJAPHWTCW7WUMQZPIWRXRA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:12.265123988 (294h7m47.035765313s) [upload=235227]: Next returned nil segment Jul 5 15:18:12.265138164 (294h7m47.035779499s) [upload=235227]: Waiting for error group managing segments... Jul 5 15:18:12.550465694 (294h7m47.321107019s) [noise=true , address=104.36.82.8:4970 , piece_id=2YJN7VKTTTCMAWQW2MNR65DAJPFMHNIBIW5NWYNG4R6HNPRFH6NQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.554315629 (294h7m47.324956954s) [noise=true , address=104.36.82.12:4149 , piece_id=YBTLQEWDMFK6JIFTYR3VJDNMXTS3NMWWHX2MPP53BM2UBDI3UUEA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.603452512 (294h7m47.374093847s) [noise=true , address=104.36.82.3:4767 , piece_id=OGHG4G5ZVC7GVY4XI6DO4KINXF7THHL6IHGDB6DAPDNUDOUFSC4A, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.639796643 (294h7m47.410437978s) [noise=true , address=104.36.82.9:4006 , piece_id=47FME2B5NS6WMOHC2ZGE2FZVSPEQUXKYT6E7AOSANAE5XZVDH5VQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.639894960 (294h7m47.410536285s) [noise=true , address=104.36.82.15:4259 , piece_id=LQKQ5MGKXHTPITBR2QAOQ5CX76GEVJGHJT3ZD3G4PFNN5IBTCJBQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.647222659 (294h7m47.417863994s) [noise=true , address=104.36.82.16:4295 , piece_id=VEL6CAUOJKMRN7A7M66ZPDLCMJVD555LZPHM53A24MDQJQOW7ECQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.665424010 (294h7m47.436065346s) [noise=true , address=104.36.82.13:4185 , piece_id=LTQS2CUZ5NRJCBOPEXDKUQ2IJGOIW3JJAPHWTCW7WUMQZPIWRXRA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.702209574 (294h7m47.47285091s) [noise=true , address=45.91.116.14:4191 , piece_id=WH2BCI36GGFF4QUD4XDYBBDMUR2R2FFVYKFTZD3ZWXPJCFJ3D7HQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.722073145 (294h7m47.49271448s) [noise=true , address=45.91.116.5:4298 , piece_id=TVTUGWHFTXB32GBY4LYXT3YDB5LIFXEGA7VO5BHFA24Y6EMDTPHQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.788621529 (294h7m47.559262864s) [noise=true , address=45.91.116.4:4937 , piece_id=YMKIHJET2EN3QHEK665ACZNQNHJ3UGE4H5T4DIHWJUHJQBYO7B7Q, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:36.752254721 (294h8m11.522896046s) [noise=true , address=77.90.31.19:4842 , piece_id=R4FTRBZ3PETCA6QGU6RPUEDZZBWXF7YVH2COEW4WH3YWDLU3VJMA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:37.783437003 (294h8m12.554078338s) [noise=true , address=77.90.31.17:4771 , piece_id=V7M65ZHCGXPD2GVJL6LTB4NP3QN6QLSIBM7U34QAYPAIKMN6EBPA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:39.733398290 (294h8m14.504039626s) [noise=true , address=77.90.31.18:4805 , piece_id=I23F6KYUQ7TJI4ZLEANACVNYJDMPKKMMACIZAIA6PC5S3CKQPV2Q, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Jul 5 15:18:39.733464387 (294h8m14.504105722s) [noise=true , address=77.90.31.21:4877 , piece_id=OJBIFN2TJPQ45LQLYCMPKV5UZM6Y2MXNOBT3TSGX26U4SZAL5Q7A, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: expected piece hash; context deadline exceeded; EOF Jul 5 15:18:39.733542065 (294h8m14.50418339s) [noise=true , address=77.90.31.11:4554 , piece_id=4QRZ5MPQ7HTPRIVRL3O3H4OV7CQA7ESYT5RCPI3GTHXJOHOSAOVA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection Jul 5 15:18:39.733547496 (294h8m14.504188831s) [noise=true , address=77.90.31.16:4733 , piece_id=3GQ7NEWE6WZJMQUA3GG3GSZYXEFNOAUEVWHOEH4TSNONYXUT43GQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection; write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection Jul 5 15:18:39.733583284 (294h8m14.504224619s) [seg_pos={0 0}, upload=235227]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 5 15:18:39.974321220 (294h8m14.744962555s) [upload=235227]: Done uploading segments... Jul 5 15:18:39.974353041 (294h8m14.744994376s) [upload=235227]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.007584872s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 5.649043921s FAIL (2025-07-05 15:57:34 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== JYE6FC5T5QIBUQIILGDHNGXAUDBUIAOSKYFMPBCWPHP2SDJ3A, seg_pos={0 0}, upload=235214]: Done uploading piece. err: <nil> Jul 5 15:17:12.446386719 (294h6m47.217028064s) [seg_pos={0 0}, upload=235214]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 5 15:17:12.446476771 (294h6m47.217118106s) [upload=235214]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 5 15:17:12.446483043 (294h6m47.217124378s) [upload=235214]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 5 15:17:12.905279085 (294h6m47.67592042s) [upload=235214]: Done uploading segments... Jul 5 15:17:12.905303211 (294h6m47.675944546s) [upload=235214]: Upload finished. err: <nil> Jul 5 15:18:10.381454266 (294h7m45.152095611s) [upload=235227]: Starting upload Jul 5 15:18:10.381524069 (294h7m45.152165414s) [upload=235227]: Done starting upload Jul 5 15:18:10.381621115 (294h7m45.15226245s) [upload=235227]: Uploading segments... Jul 5 15:18:10.381740133 (294h7m45.152381468s) [upload=235227]: Got next segment. Inline: false Jul 5 15:18:10.381776342 (294h7m45.152417677s) [upload=235227]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 5 15:18:10.381781973 (294h7m45.152423308s) [upload=235227]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 5 15:18:10.613904835 (294h7m45.38454617s) [seg_pos={0 0}, upload=235227]: Begin upload segment... Jul 5 15:18:10.613947397 (294h7m45.384588732s) [seg_pos={0 0}, upload=235227]: Done begin upload segment. Jul 5 15:18:10.613968548 (294h7m45.384609883s) [noise=true , address=104.36.82.15:4259 , piece_id=LQKQ5MGKXHTPITBR2QAOQ5CX76GEVJGHJT3ZD3G4PFNN5IBTCJBQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.613995038 (294h7m45.384636373s) [noise=true , address=104.36.82.16:4295 , piece_id=VEL6CAUOJKMRN7A7M66ZPDLCMJVD555LZPHM53A24MDQJQOW7ECQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614303969 (294h7m45.384945304s) [noise=true , address=104.36.82.9:4006 , piece_id=47FME2B5NS6WMOHC2ZGE2FZVSPEQUXKYT6E7AOSANAE5XZVDH5VQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614367721 (294h7m45.385009056s) [noise=true , address=45.91.116.4:4937 , piece_id=YMKIHJET2EN3QHEK665ACZNQNHJ3UGE4H5T4DIHWJUHJQBYO7B7Q, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614619272 (294h7m45.385260607s) [noise=true , address=77.90.31.17:4771 , piece_id=V7M65ZHCGXPD2GVJL6LTB4NP3QN6QLSIBM7U34QAYPAIKMN6EBPA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614637758 (294h7m45.385279083s) [noise=true , address=45.91.116.14:4191 , piece_id=WH2BCI36GGFF4QUD4XDYBBDMUR2R2FFVYKFTZD3ZWXPJCFJ3D7HQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614890251 (294h7m45.385531576s) [noise=true , address=104.36.82.3:4767 , piece_id=OGHG4G5ZVC7GVY4XI6DO4KINXF7THHL6IHGDB6DAPDNUDOUFSC4A, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615197719 (294h7m45.385839054s) [noise=true , address=104.36.82.12:4149 , piece_id=YBTLQEWDMFK6JIFTYR3VJDNMXTS3NMWWHX2MPP53BM2UBDI3UUEA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615413552 (294h7m45.386054887s) [noise=true , address=77.90.31.18:4805 , piece_id=I23F6KYUQ7TJI4ZLEANACVNYJDMPKKMMACIZAIA6PC5S3CKQPV2Q, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615608124 (294h7m45.386249459s) [noise=true , address=77.90.31.21:4877 , piece_id=OJBIFN2TJPQ45LQLYCMPKV5UZM6Y2MXNOBT3TSGX26U4SZAL5Q7A, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615803147 (294h7m45.386444472s) [noise=true , address=104.36.82.8:4970 , piece_id=2YJN7VKTTTCMAWQW2MNR65DAJPFMHNIBIW5NWYNG4R6HNPRFH6NQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615998511 (294h7m45.386639836s) [noise=true , address=77.90.31.16:4733 , piece_id=3GQ7NEWE6WZJMQUA3GG3GSZYXEFNOAUEVWHOEH4TSNONYXUT43GQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616193414 (294h7m45.386834749s) [noise=true , address=77.90.31.19:4842 , piece_id=R4FTRBZ3PETCA6QGU6RPUEDZZBWXF7YVH2COEW4WH3YWDLU3VJMA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616449093 (294h7m45.387090429s) [noise=true , address=77.90.31.11:4554 , piece_id=4QRZ5MPQ7HTPRIVRL3O3H4OV7CQA7ESYT5RCPI3GTHXJOHOSAOVA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616635840 (294h7m45.387277175s) [noise=true , address=45.91.116.5:4298 , piece_id=TVTUGWHFTXB32GBY4LYXT3YDB5LIFXEGA7VO5BHFA24Y6EMDTPHQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616818790 (294h7m45.387460125s) [noise=true , address=104.36.82.13:4185 , piece_id=LTQS2CUZ5NRJCBOPEXDKUQ2IJGOIW3JJAPHWTCW7WUMQZPIWRXRA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:12.265123988 (294h7m47.035765313s) [upload=235227]: Next returned nil segment Jul 5 15:18:12.265138164 (294h7m47.035779499s) [upload=235227]: Waiting for error group managing segments... Jul 5 15:18:12.550465694 (294h7m47.321107019s) [noise=true , address=104.36.82.8:4970 , piece_id=2YJN7VKTTTCMAWQW2MNR65DAJPFMHNIBIW5NWYNG4R6HNPRFH6NQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.554315629 (294h7m47.324956954s) [noise=true , address=104.36.82.12:4149 , piece_id=YBTLQEWDMFK6JIFTYR3VJDNMXTS3NMWWHX2MPP53BM2UBDI3UUEA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.603452512 (294h7m47.374093847s) [noise=true , address=104.36.82.3:4767 , piece_id=OGHG4G5ZVC7GVY4XI6DO4KINXF7THHL6IHGDB6DAPDNUDOUFSC4A, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.639796643 (294h7m47.410437978s) [noise=true , address=104.36.82.9:4006 , piece_id=47FME2B5NS6WMOHC2ZGE2FZVSPEQUXKYT6E7AOSANAE5XZVDH5VQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.639894960 (294h7m47.410536285s) [noise=true , address=104.36.82.15:4259 , piece_id=LQKQ5MGKXHTPITBR2QAOQ5CX76GEVJGHJT3ZD3G4PFNN5IBTCJBQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.647222659 (294h7m47.417863994s) [noise=true , address=104.36.82.16:4295 , piece_id=VEL6CAUOJKMRN7A7M66ZPDLCMJVD555LZPHM53A24MDQJQOW7ECQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.665424010 (294h7m47.436065346s) [noise=true , address=104.36.82.13:4185 , piece_id=LTQS2CUZ5NRJCBOPEXDKUQ2IJGOIW3JJAPHWTCW7WUMQZPIWRXRA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.702209574 (294h7m47.47285091s) [noise=true , address=45.91.116.14:4191 , piece_id=WH2BCI36GGFF4QUD4XDYBBDMUR2R2FFVYKFTZD3ZWXPJCFJ3D7HQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.722073145 (294h7m47.49271448s) [noise=true , address=45.91.116.5:4298 , piece_id=TVTUGWHFTXB32GBY4LYXT3YDB5LIFXEGA7VO5BHFA24Y6EMDTPHQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.788621529 (294h7m47.559262864s) [noise=true , address=45.91.116.4:4937 , piece_id=YMKIHJET2EN3QHEK665ACZNQNHJ3UGE4H5T4DIHWJUHJQBYO7B7Q, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:36.752254721 (294h8m11.522896046s) [noise=true , address=77.90.31.19:4842 , piece_id=R4FTRBZ3PETCA6QGU6RPUEDZZBWXF7YVH2COEW4WH3YWDLU3VJMA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:37.783437003 (294h8m12.554078338s) [noise=true , address=77.90.31.17:4771 , piece_id=V7M65ZHCGXPD2GVJL6LTB4NP3QN6QLSIBM7U34QAYPAIKMN6EBPA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:39.733398290 (294h8m14.504039626s) [noise=true , address=77.90.31.18:4805 , piece_id=I23F6KYUQ7TJI4ZLEANACVNYJDMPKKMMACIZAIA6PC5S3CKQPV2Q, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Jul 5 15:18:39.733464387 (294h8m14.504105722s) [noise=true , address=77.90.31.21:4877 , piece_id=OJBIFN2TJPQ45LQLYCMPKV5UZM6Y2MXNOBT3TSGX26U4SZAL5Q7A, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: expected piece hash; context deadline exceeded; EOF Jul 5 15:18:39.733542065 (294h8m14.50418339s) [noise=true , address=77.90.31.11:4554 , piece_id=4QRZ5MPQ7HTPRIVRL3O3H4OV7CQA7ESYT5RCPI3GTHXJOHOSAOVA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection Jul 5 15:18:39.733547496 (294h8m14.504188831s) [noise=true , address=77.90.31.16:4733 , piece_id=3GQ7NEWE6WZJMQUA3GG3GSZYXEFNOAUEVWHOEH4TSNONYXUT43GQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection; write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection Jul 5 15:18:39.733583284 (294h8m14.504224619s) [seg_pos={0 0}, upload=235227]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 5 15:18:39.974321220 (294h8m14.744962555s) [upload=235227]: Done uploading segments... Jul 5 15:18:39.974353041 (294h8m14.744994376s) [upload=235227]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.649078488s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 5.641849845s FAIL (2025-07-05 15:56:24 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== JYE6FC5T5QIBUQIILGDHNGXAUDBUIAOSKYFMPBCWPHP2SDJ3A, seg_pos={0 0}, upload=235214]: Done uploading piece. err: <nil> Jul 5 15:17:12.446386719 (294h6m47.217028064s) [seg_pos={0 0}, upload=235214]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 5 15:17:12.446476771 (294h6m47.217118106s) [upload=235214]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 5 15:17:12.446483043 (294h6m47.217124378s) [upload=235214]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 5 15:17:12.905279085 (294h6m47.67592042s) [upload=235214]: Done uploading segments... Jul 5 15:17:12.905303211 (294h6m47.675944546s) [upload=235214]: Upload finished. err: <nil> Jul 5 15:18:10.381454266 (294h7m45.152095611s) [upload=235227]: Starting upload Jul 5 15:18:10.381524069 (294h7m45.152165414s) [upload=235227]: Done starting upload Jul 5 15:18:10.381621115 (294h7m45.15226245s) [upload=235227]: Uploading segments... Jul 5 15:18:10.381740133 (294h7m45.152381468s) [upload=235227]: Got next segment. Inline: false Jul 5 15:18:10.381776342 (294h7m45.152417677s) [upload=235227]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 5 15:18:10.381781973 (294h7m45.152423308s) [upload=235227]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 5 15:18:10.613904835 (294h7m45.38454617s) [seg_pos={0 0}, upload=235227]: Begin upload segment... Jul 5 15:18:10.613947397 (294h7m45.384588732s) [seg_pos={0 0}, upload=235227]: Done begin upload segment. Jul 5 15:18:10.613968548 (294h7m45.384609883s) [noise=true , address=104.36.82.15:4259 , piece_id=LQKQ5MGKXHTPITBR2QAOQ5CX76GEVJGHJT3ZD3G4PFNN5IBTCJBQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.613995038 (294h7m45.384636373s) [noise=true , address=104.36.82.16:4295 , piece_id=VEL6CAUOJKMRN7A7M66ZPDLCMJVD555LZPHM53A24MDQJQOW7ECQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614303969 (294h7m45.384945304s) [noise=true , address=104.36.82.9:4006 , piece_id=47FME2B5NS6WMOHC2ZGE2FZVSPEQUXKYT6E7AOSANAE5XZVDH5VQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614367721 (294h7m45.385009056s) [noise=true , address=45.91.116.4:4937 , piece_id=YMKIHJET2EN3QHEK665ACZNQNHJ3UGE4H5T4DIHWJUHJQBYO7B7Q, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614619272 (294h7m45.385260607s) [noise=true , address=77.90.31.17:4771 , piece_id=V7M65ZHCGXPD2GVJL6LTB4NP3QN6QLSIBM7U34QAYPAIKMN6EBPA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614637758 (294h7m45.385279083s) [noise=true , address=45.91.116.14:4191 , piece_id=WH2BCI36GGFF4QUD4XDYBBDMUR2R2FFVYKFTZD3ZWXPJCFJ3D7HQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614890251 (294h7m45.385531576s) [noise=true , address=104.36.82.3:4767 , piece_id=OGHG4G5ZVC7GVY4XI6DO4KINXF7THHL6IHGDB6DAPDNUDOUFSC4A, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615197719 (294h7m45.385839054s) [noise=true , address=104.36.82.12:4149 , piece_id=YBTLQEWDMFK6JIFTYR3VJDNMXTS3NMWWHX2MPP53BM2UBDI3UUEA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615413552 (294h7m45.386054887s) [noise=true , address=77.90.31.18:4805 , piece_id=I23F6KYUQ7TJI4ZLEANACVNYJDMPKKMMACIZAIA6PC5S3CKQPV2Q, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615608124 (294h7m45.386249459s) [noise=true , address=77.90.31.21:4877 , piece_id=OJBIFN2TJPQ45LQLYCMPKV5UZM6Y2MXNOBT3TSGX26U4SZAL5Q7A, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615803147 (294h7m45.386444472s) [noise=true , address=104.36.82.8:4970 , piece_id=2YJN7VKTTTCMAWQW2MNR65DAJPFMHNIBIW5NWYNG4R6HNPRFH6NQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615998511 (294h7m45.386639836s) [noise=true , address=77.90.31.16:4733 , piece_id=3GQ7NEWE6WZJMQUA3GG3GSZYXEFNOAUEVWHOEH4TSNONYXUT43GQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616193414 (294h7m45.386834749s) [noise=true , address=77.90.31.19:4842 , piece_id=R4FTRBZ3PETCA6QGU6RPUEDZZBWXF7YVH2COEW4WH3YWDLU3VJMA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616449093 (294h7m45.387090429s) [noise=true , address=77.90.31.11:4554 , piece_id=4QRZ5MPQ7HTPRIVRL3O3H4OV7CQA7ESYT5RCPI3GTHXJOHOSAOVA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616635840 (294h7m45.387277175s) [noise=true , address=45.91.116.5:4298 , piece_id=TVTUGWHFTXB32GBY4LYXT3YDB5LIFXEGA7VO5BHFA24Y6EMDTPHQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616818790 (294h7m45.387460125s) [noise=true , address=104.36.82.13:4185 , piece_id=LTQS2CUZ5NRJCBOPEXDKUQ2IJGOIW3JJAPHWTCW7WUMQZPIWRXRA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:12.265123988 (294h7m47.035765313s) [upload=235227]: Next returned nil segment Jul 5 15:18:12.265138164 (294h7m47.035779499s) [upload=235227]: Waiting for error group managing segments... Jul 5 15:18:12.550465694 (294h7m47.321107019s) [noise=true , address=104.36.82.8:4970 , piece_id=2YJN7VKTTTCMAWQW2MNR65DAJPFMHNIBIW5NWYNG4R6HNPRFH6NQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.554315629 (294h7m47.324956954s) [noise=true , address=104.36.82.12:4149 , piece_id=YBTLQEWDMFK6JIFTYR3VJDNMXTS3NMWWHX2MPP53BM2UBDI3UUEA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.603452512 (294h7m47.374093847s) [noise=true , address=104.36.82.3:4767 , piece_id=OGHG4G5ZVC7GVY4XI6DO4KINXF7THHL6IHGDB6DAPDNUDOUFSC4A, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.639796643 (294h7m47.410437978s) [noise=true , address=104.36.82.9:4006 , piece_id=47FME2B5NS6WMOHC2ZGE2FZVSPEQUXKYT6E7AOSANAE5XZVDH5VQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.639894960 (294h7m47.410536285s) [noise=true , address=104.36.82.15:4259 , piece_id=LQKQ5MGKXHTPITBR2QAOQ5CX76GEVJGHJT3ZD3G4PFNN5IBTCJBQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.647222659 (294h7m47.417863994s) [noise=true , address=104.36.82.16:4295 , piece_id=VEL6CAUOJKMRN7A7M66ZPDLCMJVD555LZPHM53A24MDQJQOW7ECQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.665424010 (294h7m47.436065346s) [noise=true , address=104.36.82.13:4185 , piece_id=LTQS2CUZ5NRJCBOPEXDKUQ2IJGOIW3JJAPHWTCW7WUMQZPIWRXRA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.702209574 (294h7m47.47285091s) [noise=true , address=45.91.116.14:4191 , piece_id=WH2BCI36GGFF4QUD4XDYBBDMUR2R2FFVYKFTZD3ZWXPJCFJ3D7HQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.722073145 (294h7m47.49271448s) [noise=true , address=45.91.116.5:4298 , piece_id=TVTUGWHFTXB32GBY4LYXT3YDB5LIFXEGA7VO5BHFA24Y6EMDTPHQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.788621529 (294h7m47.559262864s) [noise=true , address=45.91.116.4:4937 , piece_id=YMKIHJET2EN3QHEK665ACZNQNHJ3UGE4H5T4DIHWJUHJQBYO7B7Q, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:36.752254721 (294h8m11.522896046s) [noise=true , address=77.90.31.19:4842 , piece_id=R4FTRBZ3PETCA6QGU6RPUEDZZBWXF7YVH2COEW4WH3YWDLU3VJMA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:37.783437003 (294h8m12.554078338s) [noise=true , address=77.90.31.17:4771 , piece_id=V7M65ZHCGXPD2GVJL6LTB4NP3QN6QLSIBM7U34QAYPAIKMN6EBPA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:39.733398290 (294h8m14.504039626s) [noise=true , address=77.90.31.18:4805 , piece_id=I23F6KYUQ7TJI4ZLEANACVNYJDMPKKMMACIZAIA6PC5S3CKQPV2Q, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Jul 5 15:18:39.733464387 (294h8m14.504105722s) [noise=true , address=77.90.31.21:4877 , piece_id=OJBIFN2TJPQ45LQLYCMPKV5UZM6Y2MXNOBT3TSGX26U4SZAL5Q7A, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: expected piece hash; context deadline exceeded; EOF Jul 5 15:18:39.733542065 (294h8m14.50418339s) [noise=true , address=77.90.31.11:4554 , piece_id=4QRZ5MPQ7HTPRIVRL3O3H4OV7CQA7ESYT5RCPI3GTHXJOHOSAOVA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection Jul 5 15:18:39.733547496 (294h8m14.504188831s) [noise=true , address=77.90.31.16:4733 , piece_id=3GQ7NEWE6WZJMQUA3GG3GSZYXEFNOAUEVWHOEH4TSNONYXUT43GQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection; write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection Jul 5 15:18:39.733583284 (294h8m14.504224619s) [seg_pos={0 0}, upload=235227]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 5 15:18:39.974321220 (294h8m14.744962555s) [upload=235227]: Done uploading segments... Jul 5 15:18:39.974353041 (294h8m14.744994376s) [upload=235227]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.641877419s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 5.759514635s FAIL (2025-07-05 15:55:34 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== JYE6FC5T5QIBUQIILGDHNGXAUDBUIAOSKYFMPBCWPHP2SDJ3A, seg_pos={0 0}, upload=235214]: Done uploading piece. err: <nil> Jul 5 15:17:12.446386719 (294h6m47.217028064s) [seg_pos={0 0}, upload=235214]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 5 15:17:12.446476771 (294h6m47.217118106s) [upload=235214]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 5 15:17:12.446483043 (294h6m47.217124378s) [upload=235214]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 5 15:17:12.905279085 (294h6m47.67592042s) [upload=235214]: Done uploading segments... Jul 5 15:17:12.905303211 (294h6m47.675944546s) [upload=235214]: Upload finished. err: <nil> Jul 5 15:18:10.381454266 (294h7m45.152095611s) [upload=235227]: Starting upload Jul 5 15:18:10.381524069 (294h7m45.152165414s) [upload=235227]: Done starting upload Jul 5 15:18:10.381621115 (294h7m45.15226245s) [upload=235227]: Uploading segments... Jul 5 15:18:10.381740133 (294h7m45.152381468s) [upload=235227]: Got next segment. Inline: false Jul 5 15:18:10.381776342 (294h7m45.152417677s) [upload=235227]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 5 15:18:10.381781973 (294h7m45.152423308s) [upload=235227]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 5 15:18:10.613904835 (294h7m45.38454617s) [seg_pos={0 0}, upload=235227]: Begin upload segment... Jul 5 15:18:10.613947397 (294h7m45.384588732s) [seg_pos={0 0}, upload=235227]: Done begin upload segment. Jul 5 15:18:10.613968548 (294h7m45.384609883s) [noise=true , address=104.36.82.15:4259 , piece_id=LQKQ5MGKXHTPITBR2QAOQ5CX76GEVJGHJT3ZD3G4PFNN5IBTCJBQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.613995038 (294h7m45.384636373s) [noise=true , address=104.36.82.16:4295 , piece_id=VEL6CAUOJKMRN7A7M66ZPDLCMJVD555LZPHM53A24MDQJQOW7ECQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614303969 (294h7m45.384945304s) [noise=true , address=104.36.82.9:4006 , piece_id=47FME2B5NS6WMOHC2ZGE2FZVSPEQUXKYT6E7AOSANAE5XZVDH5VQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614367721 (294h7m45.385009056s) [noise=true , address=45.91.116.4:4937 , piece_id=YMKIHJET2EN3QHEK665ACZNQNHJ3UGE4H5T4DIHWJUHJQBYO7B7Q, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614619272 (294h7m45.385260607s) [noise=true , address=77.90.31.17:4771 , piece_id=V7M65ZHCGXPD2GVJL6LTB4NP3QN6QLSIBM7U34QAYPAIKMN6EBPA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614637758 (294h7m45.385279083s) [noise=true , address=45.91.116.14:4191 , piece_id=WH2BCI36GGFF4QUD4XDYBBDMUR2R2FFVYKFTZD3ZWXPJCFJ3D7HQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614890251 (294h7m45.385531576s) [noise=true , address=104.36.82.3:4767 , piece_id=OGHG4G5ZVC7GVY4XI6DO4KINXF7THHL6IHGDB6DAPDNUDOUFSC4A, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615197719 (294h7m45.385839054s) [noise=true , address=104.36.82.12:4149 , piece_id=YBTLQEWDMFK6JIFTYR3VJDNMXTS3NMWWHX2MPP53BM2UBDI3UUEA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615413552 (294h7m45.386054887s) [noise=true , address=77.90.31.18:4805 , piece_id=I23F6KYUQ7TJI4ZLEANACVNYJDMPKKMMACIZAIA6PC5S3CKQPV2Q, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615608124 (294h7m45.386249459s) [noise=true , address=77.90.31.21:4877 , piece_id=OJBIFN2TJPQ45LQLYCMPKV5UZM6Y2MXNOBT3TSGX26U4SZAL5Q7A, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615803147 (294h7m45.386444472s) [noise=true , address=104.36.82.8:4970 , piece_id=2YJN7VKTTTCMAWQW2MNR65DAJPFMHNIBIW5NWYNG4R6HNPRFH6NQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615998511 (294h7m45.386639836s) [noise=true , address=77.90.31.16:4733 , piece_id=3GQ7NEWE6WZJMQUA3GG3GSZYXEFNOAUEVWHOEH4TSNONYXUT43GQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616193414 (294h7m45.386834749s) [noise=true , address=77.90.31.19:4842 , piece_id=R4FTRBZ3PETCA6QGU6RPUEDZZBWXF7YVH2COEW4WH3YWDLU3VJMA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616449093 (294h7m45.387090429s) [noise=true , address=77.90.31.11:4554 , piece_id=4QRZ5MPQ7HTPRIVRL3O3H4OV7CQA7ESYT5RCPI3GTHXJOHOSAOVA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616635840 (294h7m45.387277175s) [noise=true , address=45.91.116.5:4298 , piece_id=TVTUGWHFTXB32GBY4LYXT3YDB5LIFXEGA7VO5BHFA24Y6EMDTPHQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616818790 (294h7m45.387460125s) [noise=true , address=104.36.82.13:4185 , piece_id=LTQS2CUZ5NRJCBOPEXDKUQ2IJGOIW3JJAPHWTCW7WUMQZPIWRXRA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:12.265123988 (294h7m47.035765313s) [upload=235227]: Next returned nil segment Jul 5 15:18:12.265138164 (294h7m47.035779499s) [upload=235227]: Waiting for error group managing segments... Jul 5 15:18:12.550465694 (294h7m47.321107019s) [noise=true , address=104.36.82.8:4970 , piece_id=2YJN7VKTTTCMAWQW2MNR65DAJPFMHNIBIW5NWYNG4R6HNPRFH6NQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.554315629 (294h7m47.324956954s) [noise=true , address=104.36.82.12:4149 , piece_id=YBTLQEWDMFK6JIFTYR3VJDNMXTS3NMWWHX2MPP53BM2UBDI3UUEA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.603452512 (294h7m47.374093847s) [noise=true , address=104.36.82.3:4767 , piece_id=OGHG4G5ZVC7GVY4XI6DO4KINXF7THHL6IHGDB6DAPDNUDOUFSC4A, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.639796643 (294h7m47.410437978s) [noise=true , address=104.36.82.9:4006 , piece_id=47FME2B5NS6WMOHC2ZGE2FZVSPEQUXKYT6E7AOSANAE5XZVDH5VQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.639894960 (294h7m47.410536285s) [noise=true , address=104.36.82.15:4259 , piece_id=LQKQ5MGKXHTPITBR2QAOQ5CX76GEVJGHJT3ZD3G4PFNN5IBTCJBQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.647222659 (294h7m47.417863994s) [noise=true , address=104.36.82.16:4295 , piece_id=VEL6CAUOJKMRN7A7M66ZPDLCMJVD555LZPHM53A24MDQJQOW7ECQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.665424010 (294h7m47.436065346s) [noise=true , address=104.36.82.13:4185 , piece_id=LTQS2CUZ5NRJCBOPEXDKUQ2IJGOIW3JJAPHWTCW7WUMQZPIWRXRA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.702209574 (294h7m47.47285091s) [noise=true , address=45.91.116.14:4191 , piece_id=WH2BCI36GGFF4QUD4XDYBBDMUR2R2FFVYKFTZD3ZWXPJCFJ3D7HQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.722073145 (294h7m47.49271448s) [noise=true , address=45.91.116.5:4298 , piece_id=TVTUGWHFTXB32GBY4LYXT3YDB5LIFXEGA7VO5BHFA24Y6EMDTPHQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.788621529 (294h7m47.559262864s) [noise=true , address=45.91.116.4:4937 , piece_id=YMKIHJET2EN3QHEK665ACZNQNHJ3UGE4H5T4DIHWJUHJQBYO7B7Q, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:36.752254721 (294h8m11.522896046s) [noise=true , address=77.90.31.19:4842 , piece_id=R4FTRBZ3PETCA6QGU6RPUEDZZBWXF7YVH2COEW4WH3YWDLU3VJMA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:37.783437003 (294h8m12.554078338s) [noise=true , address=77.90.31.17:4771 , piece_id=V7M65ZHCGXPD2GVJL6LTB4NP3QN6QLSIBM7U34QAYPAIKMN6EBPA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:39.733398290 (294h8m14.504039626s) [noise=true , address=77.90.31.18:4805 , piece_id=I23F6KYUQ7TJI4ZLEANACVNYJDMPKKMMACIZAIA6PC5S3CKQPV2Q, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Jul 5 15:18:39.733464387 (294h8m14.504105722s) [noise=true , address=77.90.31.21:4877 , piece_id=OJBIFN2TJPQ45LQLYCMPKV5UZM6Y2MXNOBT3TSGX26U4SZAL5Q7A, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: expected piece hash; context deadline exceeded; EOF Jul 5 15:18:39.733542065 (294h8m14.50418339s) [noise=true , address=77.90.31.11:4554 , piece_id=4QRZ5MPQ7HTPRIVRL3O3H4OV7CQA7ESYT5RCPI3GTHXJOHOSAOVA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection Jul 5 15:18:39.733547496 (294h8m14.504188831s) [noise=true , address=77.90.31.16:4733 , piece_id=3GQ7NEWE6WZJMQUA3GG3GSZYXEFNOAUEVWHOEH4TSNONYXUT43GQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection; write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection Jul 5 15:18:39.733583284 (294h8m14.504224619s) [seg_pos={0 0}, upload=235227]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 5 15:18:39.974321220 (294h8m14.744962555s) [upload=235227]: Done uploading segments... Jul 5 15:18:39.974353041 (294h8m14.744994376s) [upload=235227]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.759540404s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 5.832824317s FAIL (2025-07-05 15:54:34 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== JYE6FC5T5QIBUQIILGDHNGXAUDBUIAOSKYFMPBCWPHP2SDJ3A, seg_pos={0 0}, upload=235214]: Done uploading piece. err: <nil> Jul 5 15:17:12.446386719 (294h6m47.217028064s) [seg_pos={0 0}, upload=235214]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 5 15:17:12.446476771 (294h6m47.217118106s) [upload=235214]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 5 15:17:12.446483043 (294h6m47.217124378s) [upload=235214]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 5 15:17:12.905279085 (294h6m47.67592042s) [upload=235214]: Done uploading segments... Jul 5 15:17:12.905303211 (294h6m47.675944546s) [upload=235214]: Upload finished. err: <nil> Jul 5 15:18:10.381454266 (294h7m45.152095611s) [upload=235227]: Starting upload Jul 5 15:18:10.381524069 (294h7m45.152165414s) [upload=235227]: Done starting upload Jul 5 15:18:10.381621115 (294h7m45.15226245s) [upload=235227]: Uploading segments... Jul 5 15:18:10.381740133 (294h7m45.152381468s) [upload=235227]: Got next segment. Inline: false Jul 5 15:18:10.381776342 (294h7m45.152417677s) [upload=235227]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 5 15:18:10.381781973 (294h7m45.152423308s) [upload=235227]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 5 15:18:10.613904835 (294h7m45.38454617s) [seg_pos={0 0}, upload=235227]: Begin upload segment... Jul 5 15:18:10.613947397 (294h7m45.384588732s) [seg_pos={0 0}, upload=235227]: Done begin upload segment. Jul 5 15:18:10.613968548 (294h7m45.384609883s) [noise=true , address=104.36.82.15:4259 , piece_id=LQKQ5MGKXHTPITBR2QAOQ5CX76GEVJGHJT3ZD3G4PFNN5IBTCJBQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.613995038 (294h7m45.384636373s) [noise=true , address=104.36.82.16:4295 , piece_id=VEL6CAUOJKMRN7A7M66ZPDLCMJVD555LZPHM53A24MDQJQOW7ECQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614303969 (294h7m45.384945304s) [noise=true , address=104.36.82.9:4006 , piece_id=47FME2B5NS6WMOHC2ZGE2FZVSPEQUXKYT6E7AOSANAE5XZVDH5VQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614367721 (294h7m45.385009056s) [noise=true , address=45.91.116.4:4937 , piece_id=YMKIHJET2EN3QHEK665ACZNQNHJ3UGE4H5T4DIHWJUHJQBYO7B7Q, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614619272 (294h7m45.385260607s) [noise=true , address=77.90.31.17:4771 , piece_id=V7M65ZHCGXPD2GVJL6LTB4NP3QN6QLSIBM7U34QAYPAIKMN6EBPA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614637758 (294h7m45.385279083s) [noise=true , address=45.91.116.14:4191 , piece_id=WH2BCI36GGFF4QUD4XDYBBDMUR2R2FFVYKFTZD3ZWXPJCFJ3D7HQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614890251 (294h7m45.385531576s) [noise=true , address=104.36.82.3:4767 , piece_id=OGHG4G5ZVC7GVY4XI6DO4KINXF7THHL6IHGDB6DAPDNUDOUFSC4A, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615197719 (294h7m45.385839054s) [noise=true , address=104.36.82.12:4149 , piece_id=YBTLQEWDMFK6JIFTYR3VJDNMXTS3NMWWHX2MPP53BM2UBDI3UUEA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615413552 (294h7m45.386054887s) [noise=true , address=77.90.31.18:4805 , piece_id=I23F6KYUQ7TJI4ZLEANACVNYJDMPKKMMACIZAIA6PC5S3CKQPV2Q, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615608124 (294h7m45.386249459s) [noise=true , address=77.90.31.21:4877 , piece_id=OJBIFN2TJPQ45LQLYCMPKV5UZM6Y2MXNOBT3TSGX26U4SZAL5Q7A, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615803147 (294h7m45.386444472s) [noise=true , address=104.36.82.8:4970 , piece_id=2YJN7VKTTTCMAWQW2MNR65DAJPFMHNIBIW5NWYNG4R6HNPRFH6NQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615998511 (294h7m45.386639836s) [noise=true , address=77.90.31.16:4733 , piece_id=3GQ7NEWE6WZJMQUA3GG3GSZYXEFNOAUEVWHOEH4TSNONYXUT43GQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616193414 (294h7m45.386834749s) [noise=true , address=77.90.31.19:4842 , piece_id=R4FTRBZ3PETCA6QGU6RPUEDZZBWXF7YVH2COEW4WH3YWDLU3VJMA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616449093 (294h7m45.387090429s) [noise=true , address=77.90.31.11:4554 , piece_id=4QRZ5MPQ7HTPRIVRL3O3H4OV7CQA7ESYT5RCPI3GTHXJOHOSAOVA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616635840 (294h7m45.387277175s) [noise=true , address=45.91.116.5:4298 , piece_id=TVTUGWHFTXB32GBY4LYXT3YDB5LIFXEGA7VO5BHFA24Y6EMDTPHQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616818790 (294h7m45.387460125s) [noise=true , address=104.36.82.13:4185 , piece_id=LTQS2CUZ5NRJCBOPEXDKUQ2IJGOIW3JJAPHWTCW7WUMQZPIWRXRA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:12.265123988 (294h7m47.035765313s) [upload=235227]: Next returned nil segment Jul 5 15:18:12.265138164 (294h7m47.035779499s) [upload=235227]: Waiting for error group managing segments... Jul 5 15:18:12.550465694 (294h7m47.321107019s) [noise=true , address=104.36.82.8:4970 , piece_id=2YJN7VKTTTCMAWQW2MNR65DAJPFMHNIBIW5NWYNG4R6HNPRFH6NQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.554315629 (294h7m47.324956954s) [noise=true , address=104.36.82.12:4149 , piece_id=YBTLQEWDMFK6JIFTYR3VJDNMXTS3NMWWHX2MPP53BM2UBDI3UUEA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.603452512 (294h7m47.374093847s) [noise=true , address=104.36.82.3:4767 , piece_id=OGHG4G5ZVC7GVY4XI6DO4KINXF7THHL6IHGDB6DAPDNUDOUFSC4A, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.639796643 (294h7m47.410437978s) [noise=true , address=104.36.82.9:4006 , piece_id=47FME2B5NS6WMOHC2ZGE2FZVSPEQUXKYT6E7AOSANAE5XZVDH5VQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.639894960 (294h7m47.410536285s) [noise=true , address=104.36.82.15:4259 , piece_id=LQKQ5MGKXHTPITBR2QAOQ5CX76GEVJGHJT3ZD3G4PFNN5IBTCJBQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.647222659 (294h7m47.417863994s) [noise=true , address=104.36.82.16:4295 , piece_id=VEL6CAUOJKMRN7A7M66ZPDLCMJVD555LZPHM53A24MDQJQOW7ECQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.665424010 (294h7m47.436065346s) [noise=true , address=104.36.82.13:4185 , piece_id=LTQS2CUZ5NRJCBOPEXDKUQ2IJGOIW3JJAPHWTCW7WUMQZPIWRXRA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.702209574 (294h7m47.47285091s) [noise=true , address=45.91.116.14:4191 , piece_id=WH2BCI36GGFF4QUD4XDYBBDMUR2R2FFVYKFTZD3ZWXPJCFJ3D7HQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.722073145 (294h7m47.49271448s) [noise=true , address=45.91.116.5:4298 , piece_id=TVTUGWHFTXB32GBY4LYXT3YDB5LIFXEGA7VO5BHFA24Y6EMDTPHQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.788621529 (294h7m47.559262864s) [noise=true , address=45.91.116.4:4937 , piece_id=YMKIHJET2EN3QHEK665ACZNQNHJ3UGE4H5T4DIHWJUHJQBYO7B7Q, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:36.752254721 (294h8m11.522896046s) [noise=true , address=77.90.31.19:4842 , piece_id=R4FTRBZ3PETCA6QGU6RPUEDZZBWXF7YVH2COEW4WH3YWDLU3VJMA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:37.783437003 (294h8m12.554078338s) [noise=true , address=77.90.31.17:4771 , piece_id=V7M65ZHCGXPD2GVJL6LTB4NP3QN6QLSIBM7U34QAYPAIKMN6EBPA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:39.733398290 (294h8m14.504039626s) [noise=true , address=77.90.31.18:4805 , piece_id=I23F6KYUQ7TJI4ZLEANACVNYJDMPKKMMACIZAIA6PC5S3CKQPV2Q, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Jul 5 15:18:39.733464387 (294h8m14.504105722s) [noise=true , address=77.90.31.21:4877 , piece_id=OJBIFN2TJPQ45LQLYCMPKV5UZM6Y2MXNOBT3TSGX26U4SZAL5Q7A, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: expected piece hash; context deadline exceeded; EOF Jul 5 15:18:39.733542065 (294h8m14.50418339s) [noise=true , address=77.90.31.11:4554 , piece_id=4QRZ5MPQ7HTPRIVRL3O3H4OV7CQA7ESYT5RCPI3GTHXJOHOSAOVA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection Jul 5 15:18:39.733547496 (294h8m14.504188831s) [noise=true , address=77.90.31.16:4733 , piece_id=3GQ7NEWE6WZJMQUA3GG3GSZYXEFNOAUEVWHOEH4TSNONYXUT43GQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection; write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection Jul 5 15:18:39.733583284 (294h8m14.504224619s) [seg_pos={0 0}, upload=235227]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 5 15:18:39.974321220 (294h8m14.744962555s) [upload=235227]: Done uploading segments... Jul 5 15:18:39.974353041 (294h8m14.744994376s) [upload=235227]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.832855075s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 5.690291337s FAIL (2025-07-05 15:53:31 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== JYE6FC5T5QIBUQIILGDHNGXAUDBUIAOSKYFMPBCWPHP2SDJ3A, seg_pos={0 0}, upload=235214]: Done uploading piece. err: <nil> Jul 5 15:17:12.446386719 (294h6m47.217028064s) [seg_pos={0 0}, upload=235214]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 5 15:17:12.446476771 (294h6m47.217118106s) [upload=235214]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 5 15:17:12.446483043 (294h6m47.217124378s) [upload=235214]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 5 15:17:12.905279085 (294h6m47.67592042s) [upload=235214]: Done uploading segments... Jul 5 15:17:12.905303211 (294h6m47.675944546s) [upload=235214]: Upload finished. err: <nil> Jul 5 15:18:10.381454266 (294h7m45.152095611s) [upload=235227]: Starting upload Jul 5 15:18:10.381524069 (294h7m45.152165414s) [upload=235227]: Done starting upload Jul 5 15:18:10.381621115 (294h7m45.15226245s) [upload=235227]: Uploading segments... Jul 5 15:18:10.381740133 (294h7m45.152381468s) [upload=235227]: Got next segment. Inline: false Jul 5 15:18:10.381776342 (294h7m45.152417677s) [upload=235227]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 5 15:18:10.381781973 (294h7m45.152423308s) [upload=235227]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 5 15:18:10.613904835 (294h7m45.38454617s) [seg_pos={0 0}, upload=235227]: Begin upload segment... Jul 5 15:18:10.613947397 (294h7m45.384588732s) [seg_pos={0 0}, upload=235227]: Done begin upload segment. Jul 5 15:18:10.613968548 (294h7m45.384609883s) [noise=true , address=104.36.82.15:4259 , piece_id=LQKQ5MGKXHTPITBR2QAOQ5CX76GEVJGHJT3ZD3G4PFNN5IBTCJBQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.613995038 (294h7m45.384636373s) [noise=true , address=104.36.82.16:4295 , piece_id=VEL6CAUOJKMRN7A7M66ZPDLCMJVD555LZPHM53A24MDQJQOW7ECQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614303969 (294h7m45.384945304s) [noise=true , address=104.36.82.9:4006 , piece_id=47FME2B5NS6WMOHC2ZGE2FZVSPEQUXKYT6E7AOSANAE5XZVDH5VQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614367721 (294h7m45.385009056s) [noise=true , address=45.91.116.4:4937 , piece_id=YMKIHJET2EN3QHEK665ACZNQNHJ3UGE4H5T4DIHWJUHJQBYO7B7Q, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614619272 (294h7m45.385260607s) [noise=true , address=77.90.31.17:4771 , piece_id=V7M65ZHCGXPD2GVJL6LTB4NP3QN6QLSIBM7U34QAYPAIKMN6EBPA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614637758 (294h7m45.385279083s) [noise=true , address=45.91.116.14:4191 , piece_id=WH2BCI36GGFF4QUD4XDYBBDMUR2R2FFVYKFTZD3ZWXPJCFJ3D7HQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614890251 (294h7m45.385531576s) [noise=true , address=104.36.82.3:4767 , piece_id=OGHG4G5ZVC7GVY4XI6DO4KINXF7THHL6IHGDB6DAPDNUDOUFSC4A, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615197719 (294h7m45.385839054s) [noise=true , address=104.36.82.12:4149 , piece_id=YBTLQEWDMFK6JIFTYR3VJDNMXTS3NMWWHX2MPP53BM2UBDI3UUEA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615413552 (294h7m45.386054887s) [noise=true , address=77.90.31.18:4805 , piece_id=I23F6KYUQ7TJI4ZLEANACVNYJDMPKKMMACIZAIA6PC5S3CKQPV2Q, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615608124 (294h7m45.386249459s) [noise=true , address=77.90.31.21:4877 , piece_id=OJBIFN2TJPQ45LQLYCMPKV5UZM6Y2MXNOBT3TSGX26U4SZAL5Q7A, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615803147 (294h7m45.386444472s) [noise=true , address=104.36.82.8:4970 , piece_id=2YJN7VKTTTCMAWQW2MNR65DAJPFMHNIBIW5NWYNG4R6HNPRFH6NQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615998511 (294h7m45.386639836s) [noise=true , address=77.90.31.16:4733 , piece_id=3GQ7NEWE6WZJMQUA3GG3GSZYXEFNOAUEVWHOEH4TSNONYXUT43GQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616193414 (294h7m45.386834749s) [noise=true , address=77.90.31.19:4842 , piece_id=R4FTRBZ3PETCA6QGU6RPUEDZZBWXF7YVH2COEW4WH3YWDLU3VJMA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616449093 (294h7m45.387090429s) [noise=true , address=77.90.31.11:4554 , piece_id=4QRZ5MPQ7HTPRIVRL3O3H4OV7CQA7ESYT5RCPI3GTHXJOHOSAOVA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616635840 (294h7m45.387277175s) [noise=true , address=45.91.116.5:4298 , piece_id=TVTUGWHFTXB32GBY4LYXT3YDB5LIFXEGA7VO5BHFA24Y6EMDTPHQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616818790 (294h7m45.387460125s) [noise=true , address=104.36.82.13:4185 , piece_id=LTQS2CUZ5NRJCBOPEXDKUQ2IJGOIW3JJAPHWTCW7WUMQZPIWRXRA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:12.265123988 (294h7m47.035765313s) [upload=235227]: Next returned nil segment Jul 5 15:18:12.265138164 (294h7m47.035779499s) [upload=235227]: Waiting for error group managing segments... Jul 5 15:18:12.550465694 (294h7m47.321107019s) [noise=true , address=104.36.82.8:4970 , piece_id=2YJN7VKTTTCMAWQW2MNR65DAJPFMHNIBIW5NWYNG4R6HNPRFH6NQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.554315629 (294h7m47.324956954s) [noise=true , address=104.36.82.12:4149 , piece_id=YBTLQEWDMFK6JIFTYR3VJDNMXTS3NMWWHX2MPP53BM2UBDI3UUEA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.603452512 (294h7m47.374093847s) [noise=true , address=104.36.82.3:4767 , piece_id=OGHG4G5ZVC7GVY4XI6DO4KINXF7THHL6IHGDB6DAPDNUDOUFSC4A, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.639796643 (294h7m47.410437978s) [noise=true , address=104.36.82.9:4006 , piece_id=47FME2B5NS6WMOHC2ZGE2FZVSPEQUXKYT6E7AOSANAE5XZVDH5VQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.639894960 (294h7m47.410536285s) [noise=true , address=104.36.82.15:4259 , piece_id=LQKQ5MGKXHTPITBR2QAOQ5CX76GEVJGHJT3ZD3G4PFNN5IBTCJBQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.647222659 (294h7m47.417863994s) [noise=true , address=104.36.82.16:4295 , piece_id=VEL6CAUOJKMRN7A7M66ZPDLCMJVD555LZPHM53A24MDQJQOW7ECQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.665424010 (294h7m47.436065346s) [noise=true , address=104.36.82.13:4185 , piece_id=LTQS2CUZ5NRJCBOPEXDKUQ2IJGOIW3JJAPHWTCW7WUMQZPIWRXRA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.702209574 (294h7m47.47285091s) [noise=true , address=45.91.116.14:4191 , piece_id=WH2BCI36GGFF4QUD4XDYBBDMUR2R2FFVYKFTZD3ZWXPJCFJ3D7HQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.722073145 (294h7m47.49271448s) [noise=true , address=45.91.116.5:4298 , piece_id=TVTUGWHFTXB32GBY4LYXT3YDB5LIFXEGA7VO5BHFA24Y6EMDTPHQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.788621529 (294h7m47.559262864s) [noise=true , address=45.91.116.4:4937 , piece_id=YMKIHJET2EN3QHEK665ACZNQNHJ3UGE4H5T4DIHWJUHJQBYO7B7Q, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:36.752254721 (294h8m11.522896046s) [noise=true , address=77.90.31.19:4842 , piece_id=R4FTRBZ3PETCA6QGU6RPUEDZZBWXF7YVH2COEW4WH3YWDLU3VJMA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:37.783437003 (294h8m12.554078338s) [noise=true , address=77.90.31.17:4771 , piece_id=V7M65ZHCGXPD2GVJL6LTB4NP3QN6QLSIBM7U34QAYPAIKMN6EBPA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:39.733398290 (294h8m14.504039626s) [noise=true , address=77.90.31.18:4805 , piece_id=I23F6KYUQ7TJI4ZLEANACVNYJDMPKKMMACIZAIA6PC5S3CKQPV2Q, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Jul 5 15:18:39.733464387 (294h8m14.504105722s) [noise=true , address=77.90.31.21:4877 , piece_id=OJBIFN2TJPQ45LQLYCMPKV5UZM6Y2MXNOBT3TSGX26U4SZAL5Q7A, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: expected piece hash; context deadline exceeded; EOF Jul 5 15:18:39.733542065 (294h8m14.50418339s) [noise=true , address=77.90.31.11:4554 , piece_id=4QRZ5MPQ7HTPRIVRL3O3H4OV7CQA7ESYT5RCPI3GTHXJOHOSAOVA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection Jul 5 15:18:39.733547496 (294h8m14.504188831s) [noise=true , address=77.90.31.16:4733 , piece_id=3GQ7NEWE6WZJMQUA3GG3GSZYXEFNOAUEVWHOEH4TSNONYXUT43GQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection; write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection Jul 5 15:18:39.733583284 (294h8m14.504224619s) [seg_pos={0 0}, upload=235227]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 5 15:18:39.974321220 (294h8m14.744962555s) [upload=235227]: Done uploading segments... Jul 5 15:18:39.974353041 (294h8m14.744994376s) [upload=235227]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.690319702s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 5.836736884s FAIL (2025-07-05 15:52:23 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== JYE6FC5T5QIBUQIILGDHNGXAUDBUIAOSKYFMPBCWPHP2SDJ3A, seg_pos={0 0}, upload=235214]: Done uploading piece. err: <nil> Jul 5 15:17:12.446386719 (294h6m47.217028064s) [seg_pos={0 0}, upload=235214]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 5 15:17:12.446476771 (294h6m47.217118106s) [upload=235214]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 5 15:17:12.446483043 (294h6m47.217124378s) [upload=235214]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 5 15:17:12.905279085 (294h6m47.67592042s) [upload=235214]: Done uploading segments... Jul 5 15:17:12.905303211 (294h6m47.675944546s) [upload=235214]: Upload finished. err: <nil> Jul 5 15:18:10.381454266 (294h7m45.152095611s) [upload=235227]: Starting upload Jul 5 15:18:10.381524069 (294h7m45.152165414s) [upload=235227]: Done starting upload Jul 5 15:18:10.381621115 (294h7m45.15226245s) [upload=235227]: Uploading segments... Jul 5 15:18:10.381740133 (294h7m45.152381468s) [upload=235227]: Got next segment. Inline: false Jul 5 15:18:10.381776342 (294h7m45.152417677s) [upload=235227]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 5 15:18:10.381781973 (294h7m45.152423308s) [upload=235227]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 5 15:18:10.613904835 (294h7m45.38454617s) [seg_pos={0 0}, upload=235227]: Begin upload segment... Jul 5 15:18:10.613947397 (294h7m45.384588732s) [seg_pos={0 0}, upload=235227]: Done begin upload segment. Jul 5 15:18:10.613968548 (294h7m45.384609883s) [noise=true , address=104.36.82.15:4259 , piece_id=LQKQ5MGKXHTPITBR2QAOQ5CX76GEVJGHJT3ZD3G4PFNN5IBTCJBQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.613995038 (294h7m45.384636373s) [noise=true , address=104.36.82.16:4295 , piece_id=VEL6CAUOJKMRN7A7M66ZPDLCMJVD555LZPHM53A24MDQJQOW7ECQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614303969 (294h7m45.384945304s) [noise=true , address=104.36.82.9:4006 , piece_id=47FME2B5NS6WMOHC2ZGE2FZVSPEQUXKYT6E7AOSANAE5XZVDH5VQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614367721 (294h7m45.385009056s) [noise=true , address=45.91.116.4:4937 , piece_id=YMKIHJET2EN3QHEK665ACZNQNHJ3UGE4H5T4DIHWJUHJQBYO7B7Q, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614619272 (294h7m45.385260607s) [noise=true , address=77.90.31.17:4771 , piece_id=V7M65ZHCGXPD2GVJL6LTB4NP3QN6QLSIBM7U34QAYPAIKMN6EBPA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614637758 (294h7m45.385279083s) [noise=true , address=45.91.116.14:4191 , piece_id=WH2BCI36GGFF4QUD4XDYBBDMUR2R2FFVYKFTZD3ZWXPJCFJ3D7HQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614890251 (294h7m45.385531576s) [noise=true , address=104.36.82.3:4767 , piece_id=OGHG4G5ZVC7GVY4XI6DO4KINXF7THHL6IHGDB6DAPDNUDOUFSC4A, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615197719 (294h7m45.385839054s) [noise=true , address=104.36.82.12:4149 , piece_id=YBTLQEWDMFK6JIFTYR3VJDNMXTS3NMWWHX2MPP53BM2UBDI3UUEA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615413552 (294h7m45.386054887s) [noise=true , address=77.90.31.18:4805 , piece_id=I23F6KYUQ7TJI4ZLEANACVNYJDMPKKMMACIZAIA6PC5S3CKQPV2Q, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615608124 (294h7m45.386249459s) [noise=true , address=77.90.31.21:4877 , piece_id=OJBIFN2TJPQ45LQLYCMPKV5UZM6Y2MXNOBT3TSGX26U4SZAL5Q7A, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615803147 (294h7m45.386444472s) [noise=true , address=104.36.82.8:4970 , piece_id=2YJN7VKTTTCMAWQW2MNR65DAJPFMHNIBIW5NWYNG4R6HNPRFH6NQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615998511 (294h7m45.386639836s) [noise=true , address=77.90.31.16:4733 , piece_id=3GQ7NEWE6WZJMQUA3GG3GSZYXEFNOAUEVWHOEH4TSNONYXUT43GQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616193414 (294h7m45.386834749s) [noise=true , address=77.90.31.19:4842 , piece_id=R4FTRBZ3PETCA6QGU6RPUEDZZBWXF7YVH2COEW4WH3YWDLU3VJMA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616449093 (294h7m45.387090429s) [noise=true , address=77.90.31.11:4554 , piece_id=4QRZ5MPQ7HTPRIVRL3O3H4OV7CQA7ESYT5RCPI3GTHXJOHOSAOVA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616635840 (294h7m45.387277175s) [noise=true , address=45.91.116.5:4298 , piece_id=TVTUGWHFTXB32GBY4LYXT3YDB5LIFXEGA7VO5BHFA24Y6EMDTPHQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616818790 (294h7m45.387460125s) [noise=true , address=104.36.82.13:4185 , piece_id=LTQS2CUZ5NRJCBOPEXDKUQ2IJGOIW3JJAPHWTCW7WUMQZPIWRXRA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:12.265123988 (294h7m47.035765313s) [upload=235227]: Next returned nil segment Jul 5 15:18:12.265138164 (294h7m47.035779499s) [upload=235227]: Waiting for error group managing segments... Jul 5 15:18:12.550465694 (294h7m47.321107019s) [noise=true , address=104.36.82.8:4970 , piece_id=2YJN7VKTTTCMAWQW2MNR65DAJPFMHNIBIW5NWYNG4R6HNPRFH6NQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.554315629 (294h7m47.324956954s) [noise=true , address=104.36.82.12:4149 , piece_id=YBTLQEWDMFK6JIFTYR3VJDNMXTS3NMWWHX2MPP53BM2UBDI3UUEA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.603452512 (294h7m47.374093847s) [noise=true , address=104.36.82.3:4767 , piece_id=OGHG4G5ZVC7GVY4XI6DO4KINXF7THHL6IHGDB6DAPDNUDOUFSC4A, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.639796643 (294h7m47.410437978s) [noise=true , address=104.36.82.9:4006 , piece_id=47FME2B5NS6WMOHC2ZGE2FZVSPEQUXKYT6E7AOSANAE5XZVDH5VQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.639894960 (294h7m47.410536285s) [noise=true , address=104.36.82.15:4259 , piece_id=LQKQ5MGKXHTPITBR2QAOQ5CX76GEVJGHJT3ZD3G4PFNN5IBTCJBQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.647222659 (294h7m47.417863994s) [noise=true , address=104.36.82.16:4295 , piece_id=VEL6CAUOJKMRN7A7M66ZPDLCMJVD555LZPHM53A24MDQJQOW7ECQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.665424010 (294h7m47.436065346s) [noise=true , address=104.36.82.13:4185 , piece_id=LTQS2CUZ5NRJCBOPEXDKUQ2IJGOIW3JJAPHWTCW7WUMQZPIWRXRA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.702209574 (294h7m47.47285091s) [noise=true , address=45.91.116.14:4191 , piece_id=WH2BCI36GGFF4QUD4XDYBBDMUR2R2FFVYKFTZD3ZWXPJCFJ3D7HQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.722073145 (294h7m47.49271448s) [noise=true , address=45.91.116.5:4298 , piece_id=TVTUGWHFTXB32GBY4LYXT3YDB5LIFXEGA7VO5BHFA24Y6EMDTPHQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.788621529 (294h7m47.559262864s) [noise=true , address=45.91.116.4:4937 , piece_id=YMKIHJET2EN3QHEK665ACZNQNHJ3UGE4H5T4DIHWJUHJQBYO7B7Q, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:36.752254721 (294h8m11.522896046s) [noise=true , address=77.90.31.19:4842 , piece_id=R4FTRBZ3PETCA6QGU6RPUEDZZBWXF7YVH2COEW4WH3YWDLU3VJMA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:37.783437003 (294h8m12.554078338s) [noise=true , address=77.90.31.17:4771 , piece_id=V7M65ZHCGXPD2GVJL6LTB4NP3QN6QLSIBM7U34QAYPAIKMN6EBPA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:39.733398290 (294h8m14.504039626s) [noise=true , address=77.90.31.18:4805 , piece_id=I23F6KYUQ7TJI4ZLEANACVNYJDMPKKMMACIZAIA6PC5S3CKQPV2Q, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Jul 5 15:18:39.733464387 (294h8m14.504105722s) [noise=true , address=77.90.31.21:4877 , piece_id=OJBIFN2TJPQ45LQLYCMPKV5UZM6Y2MXNOBT3TSGX26U4SZAL5Q7A, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: expected piece hash; context deadline exceeded; EOF Jul 5 15:18:39.733542065 (294h8m14.50418339s) [noise=true , address=77.90.31.11:4554 , piece_id=4QRZ5MPQ7HTPRIVRL3O3H4OV7CQA7ESYT5RCPI3GTHXJOHOSAOVA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection Jul 5 15:18:39.733547496 (294h8m14.504188831s) [noise=true , address=77.90.31.16:4733 , piece_id=3GQ7NEWE6WZJMQUA3GG3GSZYXEFNOAUEVWHOEH4TSNONYXUT43GQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection; write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection Jul 5 15:18:39.733583284 (294h8m14.504224619s) [seg_pos={0 0}, upload=235227]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 5 15:18:39.974321220 (294h8m14.744962555s) [upload=235227]: Done uploading segments... Jul 5 15:18:39.974353041 (294h8m14.744994376s) [upload=235227]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.83676515s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 5.62160611s FAIL (2025-07-05 15:51:11 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== JYE6FC5T5QIBUQIILGDHNGXAUDBUIAOSKYFMPBCWPHP2SDJ3A, seg_pos={0 0}, upload=235214]: Done uploading piece. err: <nil> Jul 5 15:17:12.446386719 (294h6m47.217028064s) [seg_pos={0 0}, upload=235214]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 5 15:17:12.446476771 (294h6m47.217118106s) [upload=235214]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 5 15:17:12.446483043 (294h6m47.217124378s) [upload=235214]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 5 15:17:12.905279085 (294h6m47.67592042s) [upload=235214]: Done uploading segments... Jul 5 15:17:12.905303211 (294h6m47.675944546s) [upload=235214]: Upload finished. err: <nil> Jul 5 15:18:10.381454266 (294h7m45.152095611s) [upload=235227]: Starting upload Jul 5 15:18:10.381524069 (294h7m45.152165414s) [upload=235227]: Done starting upload Jul 5 15:18:10.381621115 (294h7m45.15226245s) [upload=235227]: Uploading segments... Jul 5 15:18:10.381740133 (294h7m45.152381468s) [upload=235227]: Got next segment. Inline: false Jul 5 15:18:10.381776342 (294h7m45.152417677s) [upload=235227]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 5 15:18:10.381781973 (294h7m45.152423308s) [upload=235227]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 5 15:18:10.613904835 (294h7m45.38454617s) [seg_pos={0 0}, upload=235227]: Begin upload segment... Jul 5 15:18:10.613947397 (294h7m45.384588732s) [seg_pos={0 0}, upload=235227]: Done begin upload segment. Jul 5 15:18:10.613968548 (294h7m45.384609883s) [noise=true , address=104.36.82.15:4259 , piece_id=LQKQ5MGKXHTPITBR2QAOQ5CX76GEVJGHJT3ZD3G4PFNN5IBTCJBQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.613995038 (294h7m45.384636373s) [noise=true , address=104.36.82.16:4295 , piece_id=VEL6CAUOJKMRN7A7M66ZPDLCMJVD555LZPHM53A24MDQJQOW7ECQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614303969 (294h7m45.384945304s) [noise=true , address=104.36.82.9:4006 , piece_id=47FME2B5NS6WMOHC2ZGE2FZVSPEQUXKYT6E7AOSANAE5XZVDH5VQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614367721 (294h7m45.385009056s) [noise=true , address=45.91.116.4:4937 , piece_id=YMKIHJET2EN3QHEK665ACZNQNHJ3UGE4H5T4DIHWJUHJQBYO7B7Q, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614619272 (294h7m45.385260607s) [noise=true , address=77.90.31.17:4771 , piece_id=V7M65ZHCGXPD2GVJL6LTB4NP3QN6QLSIBM7U34QAYPAIKMN6EBPA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614637758 (294h7m45.385279083s) [noise=true , address=45.91.116.14:4191 , piece_id=WH2BCI36GGFF4QUD4XDYBBDMUR2R2FFVYKFTZD3ZWXPJCFJ3D7HQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614890251 (294h7m45.385531576s) [noise=true , address=104.36.82.3:4767 , piece_id=OGHG4G5ZVC7GVY4XI6DO4KINXF7THHL6IHGDB6DAPDNUDOUFSC4A, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615197719 (294h7m45.385839054s) [noise=true , address=104.36.82.12:4149 , piece_id=YBTLQEWDMFK6JIFTYR3VJDNMXTS3NMWWHX2MPP53BM2UBDI3UUEA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615413552 (294h7m45.386054887s) [noise=true , address=77.90.31.18:4805 , piece_id=I23F6KYUQ7TJI4ZLEANACVNYJDMPKKMMACIZAIA6PC5S3CKQPV2Q, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615608124 (294h7m45.386249459s) [noise=true , address=77.90.31.21:4877 , piece_id=OJBIFN2TJPQ45LQLYCMPKV5UZM6Y2MXNOBT3TSGX26U4SZAL5Q7A, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615803147 (294h7m45.386444472s) [noise=true , address=104.36.82.8:4970 , piece_id=2YJN7VKTTTCMAWQW2MNR65DAJPFMHNIBIW5NWYNG4R6HNPRFH6NQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615998511 (294h7m45.386639836s) [noise=true , address=77.90.31.16:4733 , piece_id=3GQ7NEWE6WZJMQUA3GG3GSZYXEFNOAUEVWHOEH4TSNONYXUT43GQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616193414 (294h7m45.386834749s) [noise=true , address=77.90.31.19:4842 , piece_id=R4FTRBZ3PETCA6QGU6RPUEDZZBWXF7YVH2COEW4WH3YWDLU3VJMA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616449093 (294h7m45.387090429s) [noise=true , address=77.90.31.11:4554 , piece_id=4QRZ5MPQ7HTPRIVRL3O3H4OV7CQA7ESYT5RCPI3GTHXJOHOSAOVA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616635840 (294h7m45.387277175s) [noise=true , address=45.91.116.5:4298 , piece_id=TVTUGWHFTXB32GBY4LYXT3YDB5LIFXEGA7VO5BHFA24Y6EMDTPHQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616818790 (294h7m45.387460125s) [noise=true , address=104.36.82.13:4185 , piece_id=LTQS2CUZ5NRJCBOPEXDKUQ2IJGOIW3JJAPHWTCW7WUMQZPIWRXRA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:12.265123988 (294h7m47.035765313s) [upload=235227]: Next returned nil segment Jul 5 15:18:12.265138164 (294h7m47.035779499s) [upload=235227]: Waiting for error group managing segments... Jul 5 15:18:12.550465694 (294h7m47.321107019s) [noise=true , address=104.36.82.8:4970 , piece_id=2YJN7VKTTTCMAWQW2MNR65DAJPFMHNIBIW5NWYNG4R6HNPRFH6NQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.554315629 (294h7m47.324956954s) [noise=true , address=104.36.82.12:4149 , piece_id=YBTLQEWDMFK6JIFTYR3VJDNMXTS3NMWWHX2MPP53BM2UBDI3UUEA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.603452512 (294h7m47.374093847s) [noise=true , address=104.36.82.3:4767 , piece_id=OGHG4G5ZVC7GVY4XI6DO4KINXF7THHL6IHGDB6DAPDNUDOUFSC4A, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.639796643 (294h7m47.410437978s) [noise=true , address=104.36.82.9:4006 , piece_id=47FME2B5NS6WMOHC2ZGE2FZVSPEQUXKYT6E7AOSANAE5XZVDH5VQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.639894960 (294h7m47.410536285s) [noise=true , address=104.36.82.15:4259 , piece_id=LQKQ5MGKXHTPITBR2QAOQ5CX76GEVJGHJT3ZD3G4PFNN5IBTCJBQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.647222659 (294h7m47.417863994s) [noise=true , address=104.36.82.16:4295 , piece_id=VEL6CAUOJKMRN7A7M66ZPDLCMJVD555LZPHM53A24MDQJQOW7ECQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.665424010 (294h7m47.436065346s) [noise=true , address=104.36.82.13:4185 , piece_id=LTQS2CUZ5NRJCBOPEXDKUQ2IJGOIW3JJAPHWTCW7WUMQZPIWRXRA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.702209574 (294h7m47.47285091s) [noise=true , address=45.91.116.14:4191 , piece_id=WH2BCI36GGFF4QUD4XDYBBDMUR2R2FFVYKFTZD3ZWXPJCFJ3D7HQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.722073145 (294h7m47.49271448s) [noise=true , address=45.91.116.5:4298 , piece_id=TVTUGWHFTXB32GBY4LYXT3YDB5LIFXEGA7VO5BHFA24Y6EMDTPHQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.788621529 (294h7m47.559262864s) [noise=true , address=45.91.116.4:4937 , piece_id=YMKIHJET2EN3QHEK665ACZNQNHJ3UGE4H5T4DIHWJUHJQBYO7B7Q, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:36.752254721 (294h8m11.522896046s) [noise=true , address=77.90.31.19:4842 , piece_id=R4FTRBZ3PETCA6QGU6RPUEDZZBWXF7YVH2COEW4WH3YWDLU3VJMA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:37.783437003 (294h8m12.554078338s) [noise=true , address=77.90.31.17:4771 , piece_id=V7M65ZHCGXPD2GVJL6LTB4NP3QN6QLSIBM7U34QAYPAIKMN6EBPA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:39.733398290 (294h8m14.504039626s) [noise=true , address=77.90.31.18:4805 , piece_id=I23F6KYUQ7TJI4ZLEANACVNYJDMPKKMMACIZAIA6PC5S3CKQPV2Q, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Jul 5 15:18:39.733464387 (294h8m14.504105722s) [noise=true , address=77.90.31.21:4877 , piece_id=OJBIFN2TJPQ45LQLYCMPKV5UZM6Y2MXNOBT3TSGX26U4SZAL5Q7A, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: expected piece hash; context deadline exceeded; EOF Jul 5 15:18:39.733542065 (294h8m14.50418339s) [noise=true , address=77.90.31.11:4554 , piece_id=4QRZ5MPQ7HTPRIVRL3O3H4OV7CQA7ESYT5RCPI3GTHXJOHOSAOVA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection Jul 5 15:18:39.733547496 (294h8m14.504188831s) [noise=true , address=77.90.31.16:4733 , piece_id=3GQ7NEWE6WZJMQUA3GG3GSZYXEFNOAUEVWHOEH4TSNONYXUT43GQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection; write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection Jul 5 15:18:39.733583284 (294h8m14.504224619s) [seg_pos={0 0}, upload=235227]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 5 15:18:39.974321220 (294h8m14.744962555s) [upload=235227]: Done uploading segments... Jul 5 15:18:39.974353041 (294h8m14.744994376s) [upload=235227]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.621631859s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 5.77727649s FAIL (2025-07-05 15:50:40 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== JYE6FC5T5QIBUQIILGDHNGXAUDBUIAOSKYFMPBCWPHP2SDJ3A, seg_pos={0 0}, upload=235214]: Done uploading piece. err: <nil> Jul 5 15:17:12.446386719 (294h6m47.217028064s) [seg_pos={0 0}, upload=235214]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 5 15:17:12.446476771 (294h6m47.217118106s) [upload=235214]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 5 15:17:12.446483043 (294h6m47.217124378s) [upload=235214]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 5 15:17:12.905279085 (294h6m47.67592042s) [upload=235214]: Done uploading segments... Jul 5 15:17:12.905303211 (294h6m47.675944546s) [upload=235214]: Upload finished. err: <nil> Jul 5 15:18:10.381454266 (294h7m45.152095611s) [upload=235227]: Starting upload Jul 5 15:18:10.381524069 (294h7m45.152165414s) [upload=235227]: Done starting upload Jul 5 15:18:10.381621115 (294h7m45.15226245s) [upload=235227]: Uploading segments... Jul 5 15:18:10.381740133 (294h7m45.152381468s) [upload=235227]: Got next segment. Inline: false Jul 5 15:18:10.381776342 (294h7m45.152417677s) [upload=235227]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 5 15:18:10.381781973 (294h7m45.152423308s) [upload=235227]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 5 15:18:10.613904835 (294h7m45.38454617s) [seg_pos={0 0}, upload=235227]: Begin upload segment... Jul 5 15:18:10.613947397 (294h7m45.384588732s) [seg_pos={0 0}, upload=235227]: Done begin upload segment. Jul 5 15:18:10.613968548 (294h7m45.384609883s) [noise=true , address=104.36.82.15:4259 , piece_id=LQKQ5MGKXHTPITBR2QAOQ5CX76GEVJGHJT3ZD3G4PFNN5IBTCJBQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.613995038 (294h7m45.384636373s) [noise=true , address=104.36.82.16:4295 , piece_id=VEL6CAUOJKMRN7A7M66ZPDLCMJVD555LZPHM53A24MDQJQOW7ECQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614303969 (294h7m45.384945304s) [noise=true , address=104.36.82.9:4006 , piece_id=47FME2B5NS6WMOHC2ZGE2FZVSPEQUXKYT6E7AOSANAE5XZVDH5VQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614367721 (294h7m45.385009056s) [noise=true , address=45.91.116.4:4937 , piece_id=YMKIHJET2EN3QHEK665ACZNQNHJ3UGE4H5T4DIHWJUHJQBYO7B7Q, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614619272 (294h7m45.385260607s) [noise=true , address=77.90.31.17:4771 , piece_id=V7M65ZHCGXPD2GVJL6LTB4NP3QN6QLSIBM7U34QAYPAIKMN6EBPA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614637758 (294h7m45.385279083s) [noise=true , address=45.91.116.14:4191 , piece_id=WH2BCI36GGFF4QUD4XDYBBDMUR2R2FFVYKFTZD3ZWXPJCFJ3D7HQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614890251 (294h7m45.385531576s) [noise=true , address=104.36.82.3:4767 , piece_id=OGHG4G5ZVC7GVY4XI6DO4KINXF7THHL6IHGDB6DAPDNUDOUFSC4A, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615197719 (294h7m45.385839054s) [noise=true , address=104.36.82.12:4149 , piece_id=YBTLQEWDMFK6JIFTYR3VJDNMXTS3NMWWHX2MPP53BM2UBDI3UUEA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615413552 (294h7m45.386054887s) [noise=true , address=77.90.31.18:4805 , piece_id=I23F6KYUQ7TJI4ZLEANACVNYJDMPKKMMACIZAIA6PC5S3CKQPV2Q, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615608124 (294h7m45.386249459s) [noise=true , address=77.90.31.21:4877 , piece_id=OJBIFN2TJPQ45LQLYCMPKV5UZM6Y2MXNOBT3TSGX26U4SZAL5Q7A, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615803147 (294h7m45.386444472s) [noise=true , address=104.36.82.8:4970 , piece_id=2YJN7VKTTTCMAWQW2MNR65DAJPFMHNIBIW5NWYNG4R6HNPRFH6NQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615998511 (294h7m45.386639836s) [noise=true , address=77.90.31.16:4733 , piece_id=3GQ7NEWE6WZJMQUA3GG3GSZYXEFNOAUEVWHOEH4TSNONYXUT43GQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616193414 (294h7m45.386834749s) [noise=true , address=77.90.31.19:4842 , piece_id=R4FTRBZ3PETCA6QGU6RPUEDZZBWXF7YVH2COEW4WH3YWDLU3VJMA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616449093 (294h7m45.387090429s) [noise=true , address=77.90.31.11:4554 , piece_id=4QRZ5MPQ7HTPRIVRL3O3H4OV7CQA7ESYT5RCPI3GTHXJOHOSAOVA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616635840 (294h7m45.387277175s) [noise=true , address=45.91.116.5:4298 , piece_id=TVTUGWHFTXB32GBY4LYXT3YDB5LIFXEGA7VO5BHFA24Y6EMDTPHQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616818790 (294h7m45.387460125s) [noise=true , address=104.36.82.13:4185 , piece_id=LTQS2CUZ5NRJCBOPEXDKUQ2IJGOIW3JJAPHWTCW7WUMQZPIWRXRA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:12.265123988 (294h7m47.035765313s) [upload=235227]: Next returned nil segment Jul 5 15:18:12.265138164 (294h7m47.035779499s) [upload=235227]: Waiting for error group managing segments... Jul 5 15:18:12.550465694 (294h7m47.321107019s) [noise=true , address=104.36.82.8:4970 , piece_id=2YJN7VKTTTCMAWQW2MNR65DAJPFMHNIBIW5NWYNG4R6HNPRFH6NQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.554315629 (294h7m47.324956954s) [noise=true , address=104.36.82.12:4149 , piece_id=YBTLQEWDMFK6JIFTYR3VJDNMXTS3NMWWHX2MPP53BM2UBDI3UUEA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.603452512 (294h7m47.374093847s) [noise=true , address=104.36.82.3:4767 , piece_id=OGHG4G5ZVC7GVY4XI6DO4KINXF7THHL6IHGDB6DAPDNUDOUFSC4A, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.639796643 (294h7m47.410437978s) [noise=true , address=104.36.82.9:4006 , piece_id=47FME2B5NS6WMOHC2ZGE2FZVSPEQUXKYT6E7AOSANAE5XZVDH5VQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.639894960 (294h7m47.410536285s) [noise=true , address=104.36.82.15:4259 , piece_id=LQKQ5MGKXHTPITBR2QAOQ5CX76GEVJGHJT3ZD3G4PFNN5IBTCJBQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.647222659 (294h7m47.417863994s) [noise=true , address=104.36.82.16:4295 , piece_id=VEL6CAUOJKMRN7A7M66ZPDLCMJVD555LZPHM53A24MDQJQOW7ECQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.665424010 (294h7m47.436065346s) [noise=true , address=104.36.82.13:4185 , piece_id=LTQS2CUZ5NRJCBOPEXDKUQ2IJGOIW3JJAPHWTCW7WUMQZPIWRXRA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.702209574 (294h7m47.47285091s) [noise=true , address=45.91.116.14:4191 , piece_id=WH2BCI36GGFF4QUD4XDYBBDMUR2R2FFVYKFTZD3ZWXPJCFJ3D7HQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.722073145 (294h7m47.49271448s) [noise=true , address=45.91.116.5:4298 , piece_id=TVTUGWHFTXB32GBY4LYXT3YDB5LIFXEGA7VO5BHFA24Y6EMDTPHQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.788621529 (294h7m47.559262864s) [noise=true , address=45.91.116.4:4937 , piece_id=YMKIHJET2EN3QHEK665ACZNQNHJ3UGE4H5T4DIHWJUHJQBYO7B7Q, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:36.752254721 (294h8m11.522896046s) [noise=true , address=77.90.31.19:4842 , piece_id=R4FTRBZ3PETCA6QGU6RPUEDZZBWXF7YVH2COEW4WH3YWDLU3VJMA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:37.783437003 (294h8m12.554078338s) [noise=true , address=77.90.31.17:4771 , piece_id=V7M65ZHCGXPD2GVJL6LTB4NP3QN6QLSIBM7U34QAYPAIKMN6EBPA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:39.733398290 (294h8m14.504039626s) [noise=true , address=77.90.31.18:4805 , piece_id=I23F6KYUQ7TJI4ZLEANACVNYJDMPKKMMACIZAIA6PC5S3CKQPV2Q, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Jul 5 15:18:39.733464387 (294h8m14.504105722s) [noise=true , address=77.90.31.21:4877 , piece_id=OJBIFN2TJPQ45LQLYCMPKV5UZM6Y2MXNOBT3TSGX26U4SZAL5Q7A, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: expected piece hash; context deadline exceeded; EOF Jul 5 15:18:39.733542065 (294h8m14.50418339s) [noise=true , address=77.90.31.11:4554 , piece_id=4QRZ5MPQ7HTPRIVRL3O3H4OV7CQA7ESYT5RCPI3GTHXJOHOSAOVA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection Jul 5 15:18:39.733547496 (294h8m14.504188831s) [noise=true , address=77.90.31.16:4733 , piece_id=3GQ7NEWE6WZJMQUA3GG3GSZYXEFNOAUEVWHOEH4TSNONYXUT43GQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection; write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection Jul 5 15:18:39.733583284 (294h8m14.504224619s) [seg_pos={0 0}, upload=235227]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 5 15:18:39.974321220 (294h8m14.744962555s) [upload=235227]: Done uploading segments... Jul 5 15:18:39.974353041 (294h8m14.744994376s) [upload=235227]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.777299884s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 5.79520004s FAIL (2025-07-05 15:49:55 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== JYE6FC5T5QIBUQIILGDHNGXAUDBUIAOSKYFMPBCWPHP2SDJ3A, seg_pos={0 0}, upload=235214]: Done uploading piece. err: <nil> Jul 5 15:17:12.446386719 (294h6m47.217028064s) [seg_pos={0 0}, upload=235214]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 5 15:17:12.446476771 (294h6m47.217118106s) [upload=235214]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 5 15:17:12.446483043 (294h6m47.217124378s) [upload=235214]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 5 15:17:12.905279085 (294h6m47.67592042s) [upload=235214]: Done uploading segments... Jul 5 15:17:12.905303211 (294h6m47.675944546s) [upload=235214]: Upload finished. err: <nil> Jul 5 15:18:10.381454266 (294h7m45.152095611s) [upload=235227]: Starting upload Jul 5 15:18:10.381524069 (294h7m45.152165414s) [upload=235227]: Done starting upload Jul 5 15:18:10.381621115 (294h7m45.15226245s) [upload=235227]: Uploading segments... Jul 5 15:18:10.381740133 (294h7m45.152381468s) [upload=235227]: Got next segment. Inline: false Jul 5 15:18:10.381776342 (294h7m45.152417677s) [upload=235227]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 5 15:18:10.381781973 (294h7m45.152423308s) [upload=235227]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 5 15:18:10.613904835 (294h7m45.38454617s) [seg_pos={0 0}, upload=235227]: Begin upload segment... Jul 5 15:18:10.613947397 (294h7m45.384588732s) [seg_pos={0 0}, upload=235227]: Done begin upload segment. Jul 5 15:18:10.613968548 (294h7m45.384609883s) [noise=true , address=104.36.82.15:4259 , piece_id=LQKQ5MGKXHTPITBR2QAOQ5CX76GEVJGHJT3ZD3G4PFNN5IBTCJBQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.613995038 (294h7m45.384636373s) [noise=true , address=104.36.82.16:4295 , piece_id=VEL6CAUOJKMRN7A7M66ZPDLCMJVD555LZPHM53A24MDQJQOW7ECQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614303969 (294h7m45.384945304s) [noise=true , address=104.36.82.9:4006 , piece_id=47FME2B5NS6WMOHC2ZGE2FZVSPEQUXKYT6E7AOSANAE5XZVDH5VQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614367721 (294h7m45.385009056s) [noise=true , address=45.91.116.4:4937 , piece_id=YMKIHJET2EN3QHEK665ACZNQNHJ3UGE4H5T4DIHWJUHJQBYO7B7Q, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614619272 (294h7m45.385260607s) [noise=true , address=77.90.31.17:4771 , piece_id=V7M65ZHCGXPD2GVJL6LTB4NP3QN6QLSIBM7U34QAYPAIKMN6EBPA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614637758 (294h7m45.385279083s) [noise=true , address=45.91.116.14:4191 , piece_id=WH2BCI36GGFF4QUD4XDYBBDMUR2R2FFVYKFTZD3ZWXPJCFJ3D7HQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614890251 (294h7m45.385531576s) [noise=true , address=104.36.82.3:4767 , piece_id=OGHG4G5ZVC7GVY4XI6DO4KINXF7THHL6IHGDB6DAPDNUDOUFSC4A, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615197719 (294h7m45.385839054s) [noise=true , address=104.36.82.12:4149 , piece_id=YBTLQEWDMFK6JIFTYR3VJDNMXTS3NMWWHX2MPP53BM2UBDI3UUEA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615413552 (294h7m45.386054887s) [noise=true , address=77.90.31.18:4805 , piece_id=I23F6KYUQ7TJI4ZLEANACVNYJDMPKKMMACIZAIA6PC5S3CKQPV2Q, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615608124 (294h7m45.386249459s) [noise=true , address=77.90.31.21:4877 , piece_id=OJBIFN2TJPQ45LQLYCMPKV5UZM6Y2MXNOBT3TSGX26U4SZAL5Q7A, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615803147 (294h7m45.386444472s) [noise=true , address=104.36.82.8:4970 , piece_id=2YJN7VKTTTCMAWQW2MNR65DAJPFMHNIBIW5NWYNG4R6HNPRFH6NQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615998511 (294h7m45.386639836s) [noise=true , address=77.90.31.16:4733 , piece_id=3GQ7NEWE6WZJMQUA3GG3GSZYXEFNOAUEVWHOEH4TSNONYXUT43GQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616193414 (294h7m45.386834749s) [noise=true , address=77.90.31.19:4842 , piece_id=R4FTRBZ3PETCA6QGU6RPUEDZZBWXF7YVH2COEW4WH3YWDLU3VJMA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616449093 (294h7m45.387090429s) [noise=true , address=77.90.31.11:4554 , piece_id=4QRZ5MPQ7HTPRIVRL3O3H4OV7CQA7ESYT5RCPI3GTHXJOHOSAOVA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616635840 (294h7m45.387277175s) [noise=true , address=45.91.116.5:4298 , piece_id=TVTUGWHFTXB32GBY4LYXT3YDB5LIFXEGA7VO5BHFA24Y6EMDTPHQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616818790 (294h7m45.387460125s) [noise=true , address=104.36.82.13:4185 , piece_id=LTQS2CUZ5NRJCBOPEXDKUQ2IJGOIW3JJAPHWTCW7WUMQZPIWRXRA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:12.265123988 (294h7m47.035765313s) [upload=235227]: Next returned nil segment Jul 5 15:18:12.265138164 (294h7m47.035779499s) [upload=235227]: Waiting for error group managing segments... Jul 5 15:18:12.550465694 (294h7m47.321107019s) [noise=true , address=104.36.82.8:4970 , piece_id=2YJN7VKTTTCMAWQW2MNR65DAJPFMHNIBIW5NWYNG4R6HNPRFH6NQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.554315629 (294h7m47.324956954s) [noise=true , address=104.36.82.12:4149 , piece_id=YBTLQEWDMFK6JIFTYR3VJDNMXTS3NMWWHX2MPP53BM2UBDI3UUEA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.603452512 (294h7m47.374093847s) [noise=true , address=104.36.82.3:4767 , piece_id=OGHG4G5ZVC7GVY4XI6DO4KINXF7THHL6IHGDB6DAPDNUDOUFSC4A, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.639796643 (294h7m47.410437978s) [noise=true , address=104.36.82.9:4006 , piece_id=47FME2B5NS6WMOHC2ZGE2FZVSPEQUXKYT6E7AOSANAE5XZVDH5VQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.639894960 (294h7m47.410536285s) [noise=true , address=104.36.82.15:4259 , piece_id=LQKQ5MGKXHTPITBR2QAOQ5CX76GEVJGHJT3ZD3G4PFNN5IBTCJBQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.647222659 (294h7m47.417863994s) [noise=true , address=104.36.82.16:4295 , piece_id=VEL6CAUOJKMRN7A7M66ZPDLCMJVD555LZPHM53A24MDQJQOW7ECQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.665424010 (294h7m47.436065346s) [noise=true , address=104.36.82.13:4185 , piece_id=LTQS2CUZ5NRJCBOPEXDKUQ2IJGOIW3JJAPHWTCW7WUMQZPIWRXRA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.702209574 (294h7m47.47285091s) [noise=true , address=45.91.116.14:4191 , piece_id=WH2BCI36GGFF4QUD4XDYBBDMUR2R2FFVYKFTZD3ZWXPJCFJ3D7HQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.722073145 (294h7m47.49271448s) [noise=true , address=45.91.116.5:4298 , piece_id=TVTUGWHFTXB32GBY4LYXT3YDB5LIFXEGA7VO5BHFA24Y6EMDTPHQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.788621529 (294h7m47.559262864s) [noise=true , address=45.91.116.4:4937 , piece_id=YMKIHJET2EN3QHEK665ACZNQNHJ3UGE4H5T4DIHWJUHJQBYO7B7Q, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:36.752254721 (294h8m11.522896046s) [noise=true , address=77.90.31.19:4842 , piece_id=R4FTRBZ3PETCA6QGU6RPUEDZZBWXF7YVH2COEW4WH3YWDLU3VJMA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:37.783437003 (294h8m12.554078338s) [noise=true , address=77.90.31.17:4771 , piece_id=V7M65ZHCGXPD2GVJL6LTB4NP3QN6QLSIBM7U34QAYPAIKMN6EBPA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:39.733398290 (294h8m14.504039626s) [noise=true , address=77.90.31.18:4805 , piece_id=I23F6KYUQ7TJI4ZLEANACVNYJDMPKKMMACIZAIA6PC5S3CKQPV2Q, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Jul 5 15:18:39.733464387 (294h8m14.504105722s) [noise=true , address=77.90.31.21:4877 , piece_id=OJBIFN2TJPQ45LQLYCMPKV5UZM6Y2MXNOBT3TSGX26U4SZAL5Q7A, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: expected piece hash; context deadline exceeded; EOF Jul 5 15:18:39.733542065 (294h8m14.50418339s) [noise=true , address=77.90.31.11:4554 , piece_id=4QRZ5MPQ7HTPRIVRL3O3H4OV7CQA7ESYT5RCPI3GTHXJOHOSAOVA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection Jul 5 15:18:39.733547496 (294h8m14.504188831s) [noise=true , address=77.90.31.16:4733 , piece_id=3GQ7NEWE6WZJMQUA3GG3GSZYXEFNOAUEVWHOEH4TSNONYXUT43GQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection; write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection Jul 5 15:18:39.733583284 (294h8m14.504224619s) [seg_pos={0 0}, upload=235227]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 5 15:18:39.974321220 (294h8m14.744962555s) [upload=235227]: Done uploading segments... Jul 5 15:18:39.974353041 (294h8m14.744994376s) [upload=235227]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.795224969s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 5.781324267s FAIL (2025-07-05 15:49:28 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== JYE6FC5T5QIBUQIILGDHNGXAUDBUIAOSKYFMPBCWPHP2SDJ3A, seg_pos={0 0}, upload=235214]: Done uploading piece. err: <nil> Jul 5 15:17:12.446386719 (294h6m47.217028064s) [seg_pos={0 0}, upload=235214]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 5 15:17:12.446476771 (294h6m47.217118106s) [upload=235214]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 5 15:17:12.446483043 (294h6m47.217124378s) [upload=235214]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 5 15:17:12.905279085 (294h6m47.67592042s) [upload=235214]: Done uploading segments... Jul 5 15:17:12.905303211 (294h6m47.675944546s) [upload=235214]: Upload finished. err: <nil> Jul 5 15:18:10.381454266 (294h7m45.152095611s) [upload=235227]: Starting upload Jul 5 15:18:10.381524069 (294h7m45.152165414s) [upload=235227]: Done starting upload Jul 5 15:18:10.381621115 (294h7m45.15226245s) [upload=235227]: Uploading segments... Jul 5 15:18:10.381740133 (294h7m45.152381468s) [upload=235227]: Got next segment. Inline: false Jul 5 15:18:10.381776342 (294h7m45.152417677s) [upload=235227]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 5 15:18:10.381781973 (294h7m45.152423308s) [upload=235227]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 5 15:18:10.613904835 (294h7m45.38454617s) [seg_pos={0 0}, upload=235227]: Begin upload segment... Jul 5 15:18:10.613947397 (294h7m45.384588732s) [seg_pos={0 0}, upload=235227]: Done begin upload segment. Jul 5 15:18:10.613968548 (294h7m45.384609883s) [noise=true , address=104.36.82.15:4259 , piece_id=LQKQ5MGKXHTPITBR2QAOQ5CX76GEVJGHJT3ZD3G4PFNN5IBTCJBQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.613995038 (294h7m45.384636373s) [noise=true , address=104.36.82.16:4295 , piece_id=VEL6CAUOJKMRN7A7M66ZPDLCMJVD555LZPHM53A24MDQJQOW7ECQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614303969 (294h7m45.384945304s) [noise=true , address=104.36.82.9:4006 , piece_id=47FME2B5NS6WMOHC2ZGE2FZVSPEQUXKYT6E7AOSANAE5XZVDH5VQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614367721 (294h7m45.385009056s) [noise=true , address=45.91.116.4:4937 , piece_id=YMKIHJET2EN3QHEK665ACZNQNHJ3UGE4H5T4DIHWJUHJQBYO7B7Q, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614619272 (294h7m45.385260607s) [noise=true , address=77.90.31.17:4771 , piece_id=V7M65ZHCGXPD2GVJL6LTB4NP3QN6QLSIBM7U34QAYPAIKMN6EBPA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614637758 (294h7m45.385279083s) [noise=true , address=45.91.116.14:4191 , piece_id=WH2BCI36GGFF4QUD4XDYBBDMUR2R2FFVYKFTZD3ZWXPJCFJ3D7HQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614890251 (294h7m45.385531576s) [noise=true , address=104.36.82.3:4767 , piece_id=OGHG4G5ZVC7GVY4XI6DO4KINXF7THHL6IHGDB6DAPDNUDOUFSC4A, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615197719 (294h7m45.385839054s) [noise=true , address=104.36.82.12:4149 , piece_id=YBTLQEWDMFK6JIFTYR3VJDNMXTS3NMWWHX2MPP53BM2UBDI3UUEA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615413552 (294h7m45.386054887s) [noise=true , address=77.90.31.18:4805 , piece_id=I23F6KYUQ7TJI4ZLEANACVNYJDMPKKMMACIZAIA6PC5S3CKQPV2Q, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615608124 (294h7m45.386249459s) [noise=true , address=77.90.31.21:4877 , piece_id=OJBIFN2TJPQ45LQLYCMPKV5UZM6Y2MXNOBT3TSGX26U4SZAL5Q7A, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615803147 (294h7m45.386444472s) [noise=true , address=104.36.82.8:4970 , piece_id=2YJN7VKTTTCMAWQW2MNR65DAJPFMHNIBIW5NWYNG4R6HNPRFH6NQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615998511 (294h7m45.386639836s) [noise=true , address=77.90.31.16:4733 , piece_id=3GQ7NEWE6WZJMQUA3GG3GSZYXEFNOAUEVWHOEH4TSNONYXUT43GQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616193414 (294h7m45.386834749s) [noise=true , address=77.90.31.19:4842 , piece_id=R4FTRBZ3PETCA6QGU6RPUEDZZBWXF7YVH2COEW4WH3YWDLU3VJMA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616449093 (294h7m45.387090429s) [noise=true , address=77.90.31.11:4554 , piece_id=4QRZ5MPQ7HTPRIVRL3O3H4OV7CQA7ESYT5RCPI3GTHXJOHOSAOVA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616635840 (294h7m45.387277175s) [noise=true , address=45.91.116.5:4298 , piece_id=TVTUGWHFTXB32GBY4LYXT3YDB5LIFXEGA7VO5BHFA24Y6EMDTPHQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616818790 (294h7m45.387460125s) [noise=true , address=104.36.82.13:4185 , piece_id=LTQS2CUZ5NRJCBOPEXDKUQ2IJGOIW3JJAPHWTCW7WUMQZPIWRXRA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:12.265123988 (294h7m47.035765313s) [upload=235227]: Next returned nil segment Jul 5 15:18:12.265138164 (294h7m47.035779499s) [upload=235227]: Waiting for error group managing segments... Jul 5 15:18:12.550465694 (294h7m47.321107019s) [noise=true , address=104.36.82.8:4970 , piece_id=2YJN7VKTTTCMAWQW2MNR65DAJPFMHNIBIW5NWYNG4R6HNPRFH6NQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.554315629 (294h7m47.324956954s) [noise=true , address=104.36.82.12:4149 , piece_id=YBTLQEWDMFK6JIFTYR3VJDNMXTS3NMWWHX2MPP53BM2UBDI3UUEA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.603452512 (294h7m47.374093847s) [noise=true , address=104.36.82.3:4767 , piece_id=OGHG4G5ZVC7GVY4XI6DO4KINXF7THHL6IHGDB6DAPDNUDOUFSC4A, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.639796643 (294h7m47.410437978s) [noise=true , address=104.36.82.9:4006 , piece_id=47FME2B5NS6WMOHC2ZGE2FZVSPEQUXKYT6E7AOSANAE5XZVDH5VQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.639894960 (294h7m47.410536285s) [noise=true , address=104.36.82.15:4259 , piece_id=LQKQ5MGKXHTPITBR2QAOQ5CX76GEVJGHJT3ZD3G4PFNN5IBTCJBQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.647222659 (294h7m47.417863994s) [noise=true , address=104.36.82.16:4295 , piece_id=VEL6CAUOJKMRN7A7M66ZPDLCMJVD555LZPHM53A24MDQJQOW7ECQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.665424010 (294h7m47.436065346s) [noise=true , address=104.36.82.13:4185 , piece_id=LTQS2CUZ5NRJCBOPEXDKUQ2IJGOIW3JJAPHWTCW7WUMQZPIWRXRA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.702209574 (294h7m47.47285091s) [noise=true , address=45.91.116.14:4191 , piece_id=WH2BCI36GGFF4QUD4XDYBBDMUR2R2FFVYKFTZD3ZWXPJCFJ3D7HQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.722073145 (294h7m47.49271448s) [noise=true , address=45.91.116.5:4298 , piece_id=TVTUGWHFTXB32GBY4LYXT3YDB5LIFXEGA7VO5BHFA24Y6EMDTPHQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.788621529 (294h7m47.559262864s) [noise=true , address=45.91.116.4:4937 , piece_id=YMKIHJET2EN3QHEK665ACZNQNHJ3UGE4H5T4DIHWJUHJQBYO7B7Q, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:36.752254721 (294h8m11.522896046s) [noise=true , address=77.90.31.19:4842 , piece_id=R4FTRBZ3PETCA6QGU6RPUEDZZBWXF7YVH2COEW4WH3YWDLU3VJMA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:37.783437003 (294h8m12.554078338s) [noise=true , address=77.90.31.17:4771 , piece_id=V7M65ZHCGXPD2GVJL6LTB4NP3QN6QLSIBM7U34QAYPAIKMN6EBPA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:39.733398290 (294h8m14.504039626s) [noise=true , address=77.90.31.18:4805 , piece_id=I23F6KYUQ7TJI4ZLEANACVNYJDMPKKMMACIZAIA6PC5S3CKQPV2Q, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Jul 5 15:18:39.733464387 (294h8m14.504105722s) [noise=true , address=77.90.31.21:4877 , piece_id=OJBIFN2TJPQ45LQLYCMPKV5UZM6Y2MXNOBT3TSGX26U4SZAL5Q7A, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: expected piece hash; context deadline exceeded; EOF Jul 5 15:18:39.733542065 (294h8m14.50418339s) [noise=true , address=77.90.31.11:4554 , piece_id=4QRZ5MPQ7HTPRIVRL3O3H4OV7CQA7ESYT5RCPI3GTHXJOHOSAOVA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection Jul 5 15:18:39.733547496 (294h8m14.504188831s) [noise=true , address=77.90.31.16:4733 , piece_id=3GQ7NEWE6WZJMQUA3GG3GSZYXEFNOAUEVWHOEH4TSNONYXUT43GQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection; write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection Jul 5 15:18:39.733583284 (294h8m14.504224619s) [seg_pos={0 0}, upload=235227]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 5 15:18:39.974321220 (294h8m14.744962555s) [upload=235227]: Done uploading segments... Jul 5 15:18:39.974353041 (294h8m14.744994376s) [upload=235227]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.781351768s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 5.725059767s FAIL (2025-07-05 15:48:49 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== JYE6FC5T5QIBUQIILGDHNGXAUDBUIAOSKYFMPBCWPHP2SDJ3A, seg_pos={0 0}, upload=235214]: Done uploading piece. err: <nil> Jul 5 15:17:12.446386719 (294h6m47.217028064s) [seg_pos={0 0}, upload=235214]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 5 15:17:12.446476771 (294h6m47.217118106s) [upload=235214]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 5 15:17:12.446483043 (294h6m47.217124378s) [upload=235214]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 5 15:17:12.905279085 (294h6m47.67592042s) [upload=235214]: Done uploading segments... Jul 5 15:17:12.905303211 (294h6m47.675944546s) [upload=235214]: Upload finished. err: <nil> Jul 5 15:18:10.381454266 (294h7m45.152095611s) [upload=235227]: Starting upload Jul 5 15:18:10.381524069 (294h7m45.152165414s) [upload=235227]: Done starting upload Jul 5 15:18:10.381621115 (294h7m45.15226245s) [upload=235227]: Uploading segments... Jul 5 15:18:10.381740133 (294h7m45.152381468s) [upload=235227]: Got next segment. Inline: false Jul 5 15:18:10.381776342 (294h7m45.152417677s) [upload=235227]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 5 15:18:10.381781973 (294h7m45.152423308s) [upload=235227]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 5 15:18:10.613904835 (294h7m45.38454617s) [seg_pos={0 0}, upload=235227]: Begin upload segment... Jul 5 15:18:10.613947397 (294h7m45.384588732s) [seg_pos={0 0}, upload=235227]: Done begin upload segment. Jul 5 15:18:10.613968548 (294h7m45.384609883s) [noise=true , address=104.36.82.15:4259 , piece_id=LQKQ5MGKXHTPITBR2QAOQ5CX76GEVJGHJT3ZD3G4PFNN5IBTCJBQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.613995038 (294h7m45.384636373s) [noise=true , address=104.36.82.16:4295 , piece_id=VEL6CAUOJKMRN7A7M66ZPDLCMJVD555LZPHM53A24MDQJQOW7ECQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614303969 (294h7m45.384945304s) [noise=true , address=104.36.82.9:4006 , piece_id=47FME2B5NS6WMOHC2ZGE2FZVSPEQUXKYT6E7AOSANAE5XZVDH5VQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614367721 (294h7m45.385009056s) [noise=true , address=45.91.116.4:4937 , piece_id=YMKIHJET2EN3QHEK665ACZNQNHJ3UGE4H5T4DIHWJUHJQBYO7B7Q, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614619272 (294h7m45.385260607s) [noise=true , address=77.90.31.17:4771 , piece_id=V7M65ZHCGXPD2GVJL6LTB4NP3QN6QLSIBM7U34QAYPAIKMN6EBPA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614637758 (294h7m45.385279083s) [noise=true , address=45.91.116.14:4191 , piece_id=WH2BCI36GGFF4QUD4XDYBBDMUR2R2FFVYKFTZD3ZWXPJCFJ3D7HQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614890251 (294h7m45.385531576s) [noise=true , address=104.36.82.3:4767 , piece_id=OGHG4G5ZVC7GVY4XI6DO4KINXF7THHL6IHGDB6DAPDNUDOUFSC4A, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615197719 (294h7m45.385839054s) [noise=true , address=104.36.82.12:4149 , piece_id=YBTLQEWDMFK6JIFTYR3VJDNMXTS3NMWWHX2MPP53BM2UBDI3UUEA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615413552 (294h7m45.386054887s) [noise=true , address=77.90.31.18:4805 , piece_id=I23F6KYUQ7TJI4ZLEANACVNYJDMPKKMMACIZAIA6PC5S3CKQPV2Q, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615608124 (294h7m45.386249459s) [noise=true , address=77.90.31.21:4877 , piece_id=OJBIFN2TJPQ45LQLYCMPKV5UZM6Y2MXNOBT3TSGX26U4SZAL5Q7A, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615803147 (294h7m45.386444472s) [noise=true , address=104.36.82.8:4970 , piece_id=2YJN7VKTTTCMAWQW2MNR65DAJPFMHNIBIW5NWYNG4R6HNPRFH6NQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615998511 (294h7m45.386639836s) [noise=true , address=77.90.31.16:4733 , piece_id=3GQ7NEWE6WZJMQUA3GG3GSZYXEFNOAUEVWHOEH4TSNONYXUT43GQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616193414 (294h7m45.386834749s) [noise=true , address=77.90.31.19:4842 , piece_id=R4FTRBZ3PETCA6QGU6RPUEDZZBWXF7YVH2COEW4WH3YWDLU3VJMA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616449093 (294h7m45.387090429s) [noise=true , address=77.90.31.11:4554 , piece_id=4QRZ5MPQ7HTPRIVRL3O3H4OV7CQA7ESYT5RCPI3GTHXJOHOSAOVA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616635840 (294h7m45.387277175s) [noise=true , address=45.91.116.5:4298 , piece_id=TVTUGWHFTXB32GBY4LYXT3YDB5LIFXEGA7VO5BHFA24Y6EMDTPHQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616818790 (294h7m45.387460125s) [noise=true , address=104.36.82.13:4185 , piece_id=LTQS2CUZ5NRJCBOPEXDKUQ2IJGOIW3JJAPHWTCW7WUMQZPIWRXRA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:12.265123988 (294h7m47.035765313s) [upload=235227]: Next returned nil segment Jul 5 15:18:12.265138164 (294h7m47.035779499s) [upload=235227]: Waiting for error group managing segments... Jul 5 15:18:12.550465694 (294h7m47.321107019s) [noise=true , address=104.36.82.8:4970 , piece_id=2YJN7VKTTTCMAWQW2MNR65DAJPFMHNIBIW5NWYNG4R6HNPRFH6NQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.554315629 (294h7m47.324956954s) [noise=true , address=104.36.82.12:4149 , piece_id=YBTLQEWDMFK6JIFTYR3VJDNMXTS3NMWWHX2MPP53BM2UBDI3UUEA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.603452512 (294h7m47.374093847s) [noise=true , address=104.36.82.3:4767 , piece_id=OGHG4G5ZVC7GVY4XI6DO4KINXF7THHL6IHGDB6DAPDNUDOUFSC4A, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.639796643 (294h7m47.410437978s) [noise=true , address=104.36.82.9:4006 , piece_id=47FME2B5NS6WMOHC2ZGE2FZVSPEQUXKYT6E7AOSANAE5XZVDH5VQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.639894960 (294h7m47.410536285s) [noise=true , address=104.36.82.15:4259 , piece_id=LQKQ5MGKXHTPITBR2QAOQ5CX76GEVJGHJT3ZD3G4PFNN5IBTCJBQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.647222659 (294h7m47.417863994s) [noise=true , address=104.36.82.16:4295 , piece_id=VEL6CAUOJKMRN7A7M66ZPDLCMJVD555LZPHM53A24MDQJQOW7ECQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.665424010 (294h7m47.436065346s) [noise=true , address=104.36.82.13:4185 , piece_id=LTQS2CUZ5NRJCBOPEXDKUQ2IJGOIW3JJAPHWTCW7WUMQZPIWRXRA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.702209574 (294h7m47.47285091s) [noise=true , address=45.91.116.14:4191 , piece_id=WH2BCI36GGFF4QUD4XDYBBDMUR2R2FFVYKFTZD3ZWXPJCFJ3D7HQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.722073145 (294h7m47.49271448s) [noise=true , address=45.91.116.5:4298 , piece_id=TVTUGWHFTXB32GBY4LYXT3YDB5LIFXEGA7VO5BHFA24Y6EMDTPHQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.788621529 (294h7m47.559262864s) [noise=true , address=45.91.116.4:4937 , piece_id=YMKIHJET2EN3QHEK665ACZNQNHJ3UGE4H5T4DIHWJUHJQBYO7B7Q, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:36.752254721 (294h8m11.522896046s) [noise=true , address=77.90.31.19:4842 , piece_id=R4FTRBZ3PETCA6QGU6RPUEDZZBWXF7YVH2COEW4WH3YWDLU3VJMA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:37.783437003 (294h8m12.554078338s) [noise=true , address=77.90.31.17:4771 , piece_id=V7M65ZHCGXPD2GVJL6LTB4NP3QN6QLSIBM7U34QAYPAIKMN6EBPA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:39.733398290 (294h8m14.504039626s) [noise=true , address=77.90.31.18:4805 , piece_id=I23F6KYUQ7TJI4ZLEANACVNYJDMPKKMMACIZAIA6PC5S3CKQPV2Q, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Jul 5 15:18:39.733464387 (294h8m14.504105722s) [noise=true , address=77.90.31.21:4877 , piece_id=OJBIFN2TJPQ45LQLYCMPKV5UZM6Y2MXNOBT3TSGX26U4SZAL5Q7A, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: expected piece hash; context deadline exceeded; EOF Jul 5 15:18:39.733542065 (294h8m14.50418339s) [noise=true , address=77.90.31.11:4554 , piece_id=4QRZ5MPQ7HTPRIVRL3O3H4OV7CQA7ESYT5RCPI3GTHXJOHOSAOVA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection Jul 5 15:18:39.733547496 (294h8m14.504188831s) [noise=true , address=77.90.31.16:4733 , piece_id=3GQ7NEWE6WZJMQUA3GG3GSZYXEFNOAUEVWHOEH4TSNONYXUT43GQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection; write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection Jul 5 15:18:39.733583284 (294h8m14.504224619s) [seg_pos={0 0}, upload=235227]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 5 15:18:39.974321220 (294h8m14.744962555s) [upload=235227]: Done uploading segments... Jul 5 15:18:39.974353041 (294h8m14.744994376s) [upload=235227]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.725084033s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 5.600264577s FAIL (2025-07-05 15:47:41 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== JYE6FC5T5QIBUQIILGDHNGXAUDBUIAOSKYFMPBCWPHP2SDJ3A, seg_pos={0 0}, upload=235214]: Done uploading piece. err: <nil> Jul 5 15:17:12.446386719 (294h6m47.217028064s) [seg_pos={0 0}, upload=235214]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 5 15:17:12.446476771 (294h6m47.217118106s) [upload=235214]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 5 15:17:12.446483043 (294h6m47.217124378s) [upload=235214]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 5 15:17:12.905279085 (294h6m47.67592042s) [upload=235214]: Done uploading segments... Jul 5 15:17:12.905303211 (294h6m47.675944546s) [upload=235214]: Upload finished. err: <nil> Jul 5 15:18:10.381454266 (294h7m45.152095611s) [upload=235227]: Starting upload Jul 5 15:18:10.381524069 (294h7m45.152165414s) [upload=235227]: Done starting upload Jul 5 15:18:10.381621115 (294h7m45.15226245s) [upload=235227]: Uploading segments... Jul 5 15:18:10.381740133 (294h7m45.152381468s) [upload=235227]: Got next segment. Inline: false Jul 5 15:18:10.381776342 (294h7m45.152417677s) [upload=235227]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 5 15:18:10.381781973 (294h7m45.152423308s) [upload=235227]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 5 15:18:10.613904835 (294h7m45.38454617s) [seg_pos={0 0}, upload=235227]: Begin upload segment... Jul 5 15:18:10.613947397 (294h7m45.384588732s) [seg_pos={0 0}, upload=235227]: Done begin upload segment. Jul 5 15:18:10.613968548 (294h7m45.384609883s) [noise=true , address=104.36.82.15:4259 , piece_id=LQKQ5MGKXHTPITBR2QAOQ5CX76GEVJGHJT3ZD3G4PFNN5IBTCJBQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.613995038 (294h7m45.384636373s) [noise=true , address=104.36.82.16:4295 , piece_id=VEL6CAUOJKMRN7A7M66ZPDLCMJVD555LZPHM53A24MDQJQOW7ECQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614303969 (294h7m45.384945304s) [noise=true , address=104.36.82.9:4006 , piece_id=47FME2B5NS6WMOHC2ZGE2FZVSPEQUXKYT6E7AOSANAE5XZVDH5VQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614367721 (294h7m45.385009056s) [noise=true , address=45.91.116.4:4937 , piece_id=YMKIHJET2EN3QHEK665ACZNQNHJ3UGE4H5T4DIHWJUHJQBYO7B7Q, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614619272 (294h7m45.385260607s) [noise=true , address=77.90.31.17:4771 , piece_id=V7M65ZHCGXPD2GVJL6LTB4NP3QN6QLSIBM7U34QAYPAIKMN6EBPA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614637758 (294h7m45.385279083s) [noise=true , address=45.91.116.14:4191 , piece_id=WH2BCI36GGFF4QUD4XDYBBDMUR2R2FFVYKFTZD3ZWXPJCFJ3D7HQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614890251 (294h7m45.385531576s) [noise=true , address=104.36.82.3:4767 , piece_id=OGHG4G5ZVC7GVY4XI6DO4KINXF7THHL6IHGDB6DAPDNUDOUFSC4A, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615197719 (294h7m45.385839054s) [noise=true , address=104.36.82.12:4149 , piece_id=YBTLQEWDMFK6JIFTYR3VJDNMXTS3NMWWHX2MPP53BM2UBDI3UUEA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615413552 (294h7m45.386054887s) [noise=true , address=77.90.31.18:4805 , piece_id=I23F6KYUQ7TJI4ZLEANACVNYJDMPKKMMACIZAIA6PC5S3CKQPV2Q, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615608124 (294h7m45.386249459s) [noise=true , address=77.90.31.21:4877 , piece_id=OJBIFN2TJPQ45LQLYCMPKV5UZM6Y2MXNOBT3TSGX26U4SZAL5Q7A, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615803147 (294h7m45.386444472s) [noise=true , address=104.36.82.8:4970 , piece_id=2YJN7VKTTTCMAWQW2MNR65DAJPFMHNIBIW5NWYNG4R6HNPRFH6NQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615998511 (294h7m45.386639836s) [noise=true , address=77.90.31.16:4733 , piece_id=3GQ7NEWE6WZJMQUA3GG3GSZYXEFNOAUEVWHOEH4TSNONYXUT43GQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616193414 (294h7m45.386834749s) [noise=true , address=77.90.31.19:4842 , piece_id=R4FTRBZ3PETCA6QGU6RPUEDZZBWXF7YVH2COEW4WH3YWDLU3VJMA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616449093 (294h7m45.387090429s) [noise=true , address=77.90.31.11:4554 , piece_id=4QRZ5MPQ7HTPRIVRL3O3H4OV7CQA7ESYT5RCPI3GTHXJOHOSAOVA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616635840 (294h7m45.387277175s) [noise=true , address=45.91.116.5:4298 , piece_id=TVTUGWHFTXB32GBY4LYXT3YDB5LIFXEGA7VO5BHFA24Y6EMDTPHQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616818790 (294h7m45.387460125s) [noise=true , address=104.36.82.13:4185 , piece_id=LTQS2CUZ5NRJCBOPEXDKUQ2IJGOIW3JJAPHWTCW7WUMQZPIWRXRA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:12.265123988 (294h7m47.035765313s) [upload=235227]: Next returned nil segment Jul 5 15:18:12.265138164 (294h7m47.035779499s) [upload=235227]: Waiting for error group managing segments... Jul 5 15:18:12.550465694 (294h7m47.321107019s) [noise=true , address=104.36.82.8:4970 , piece_id=2YJN7VKTTTCMAWQW2MNR65DAJPFMHNIBIW5NWYNG4R6HNPRFH6NQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.554315629 (294h7m47.324956954s) [noise=true , address=104.36.82.12:4149 , piece_id=YBTLQEWDMFK6JIFTYR3VJDNMXTS3NMWWHX2MPP53BM2UBDI3UUEA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.603452512 (294h7m47.374093847s) [noise=true , address=104.36.82.3:4767 , piece_id=OGHG4G5ZVC7GVY4XI6DO4KINXF7THHL6IHGDB6DAPDNUDOUFSC4A, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.639796643 (294h7m47.410437978s) [noise=true , address=104.36.82.9:4006 , piece_id=47FME2B5NS6WMOHC2ZGE2FZVSPEQUXKYT6E7AOSANAE5XZVDH5VQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.639894960 (294h7m47.410536285s) [noise=true , address=104.36.82.15:4259 , piece_id=LQKQ5MGKXHTPITBR2QAOQ5CX76GEVJGHJT3ZD3G4PFNN5IBTCJBQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.647222659 (294h7m47.417863994s) [noise=true , address=104.36.82.16:4295 , piece_id=VEL6CAUOJKMRN7A7M66ZPDLCMJVD555LZPHM53A24MDQJQOW7ECQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.665424010 (294h7m47.436065346s) [noise=true , address=104.36.82.13:4185 , piece_id=LTQS2CUZ5NRJCBOPEXDKUQ2IJGOIW3JJAPHWTCW7WUMQZPIWRXRA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.702209574 (294h7m47.47285091s) [noise=true , address=45.91.116.14:4191 , piece_id=WH2BCI36GGFF4QUD4XDYBBDMUR2R2FFVYKFTZD3ZWXPJCFJ3D7HQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.722073145 (294h7m47.49271448s) [noise=true , address=45.91.116.5:4298 , piece_id=TVTUGWHFTXB32GBY4LYXT3YDB5LIFXEGA7VO5BHFA24Y6EMDTPHQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.788621529 (294h7m47.559262864s) [noise=true , address=45.91.116.4:4937 , piece_id=YMKIHJET2EN3QHEK665ACZNQNHJ3UGE4H5T4DIHWJUHJQBYO7B7Q, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:36.752254721 (294h8m11.522896046s) [noise=true , address=77.90.31.19:4842 , piece_id=R4FTRBZ3PETCA6QGU6RPUEDZZBWXF7YVH2COEW4WH3YWDLU3VJMA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:37.783437003 (294h8m12.554078338s) [noise=true , address=77.90.31.17:4771 , piece_id=V7M65ZHCGXPD2GVJL6LTB4NP3QN6QLSIBM7U34QAYPAIKMN6EBPA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:39.733398290 (294h8m14.504039626s) [noise=true , address=77.90.31.18:4805 , piece_id=I23F6KYUQ7TJI4ZLEANACVNYJDMPKKMMACIZAIA6PC5S3CKQPV2Q, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Jul 5 15:18:39.733464387 (294h8m14.504105722s) [noise=true , address=77.90.31.21:4877 , piece_id=OJBIFN2TJPQ45LQLYCMPKV5UZM6Y2MXNOBT3TSGX26U4SZAL5Q7A, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: expected piece hash; context deadline exceeded; EOF Jul 5 15:18:39.733542065 (294h8m14.50418339s) [noise=true , address=77.90.31.11:4554 , piece_id=4QRZ5MPQ7HTPRIVRL3O3H4OV7CQA7ESYT5RCPI3GTHXJOHOSAOVA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection Jul 5 15:18:39.733547496 (294h8m14.504188831s) [noise=true , address=77.90.31.16:4733 , piece_id=3GQ7NEWE6WZJMQUA3GG3GSZYXEFNOAUEVWHOEH4TSNONYXUT43GQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection; write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection Jul 5 15:18:39.733583284 (294h8m14.504224619s) [seg_pos={0 0}, upload=235227]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 5 15:18:39.974321220 (294h8m14.744962555s) [upload=235227]: Done uploading segments... Jul 5 15:18:39.974353041 (294h8m14.744994376s) [upload=235227]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.600292851s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 5.926461417s FAIL (2025-07-05 15:46:43 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== JYE6FC5T5QIBUQIILGDHNGXAUDBUIAOSKYFMPBCWPHP2SDJ3A, seg_pos={0 0}, upload=235214]: Done uploading piece. err: <nil> Jul 5 15:17:12.446386719 (294h6m47.217028064s) [seg_pos={0 0}, upload=235214]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 5 15:17:12.446476771 (294h6m47.217118106s) [upload=235214]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 5 15:17:12.446483043 (294h6m47.217124378s) [upload=235214]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 5 15:17:12.905279085 (294h6m47.67592042s) [upload=235214]: Done uploading segments... Jul 5 15:17:12.905303211 (294h6m47.675944546s) [upload=235214]: Upload finished. err: <nil> Jul 5 15:18:10.381454266 (294h7m45.152095611s) [upload=235227]: Starting upload Jul 5 15:18:10.381524069 (294h7m45.152165414s) [upload=235227]: Done starting upload Jul 5 15:18:10.381621115 (294h7m45.15226245s) [upload=235227]: Uploading segments... Jul 5 15:18:10.381740133 (294h7m45.152381468s) [upload=235227]: Got next segment. Inline: false Jul 5 15:18:10.381776342 (294h7m45.152417677s) [upload=235227]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 5 15:18:10.381781973 (294h7m45.152423308s) [upload=235227]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 5 15:18:10.613904835 (294h7m45.38454617s) [seg_pos={0 0}, upload=235227]: Begin upload segment... Jul 5 15:18:10.613947397 (294h7m45.384588732s) [seg_pos={0 0}, upload=235227]: Done begin upload segment. Jul 5 15:18:10.613968548 (294h7m45.384609883s) [noise=true , address=104.36.82.15:4259 , piece_id=LQKQ5MGKXHTPITBR2QAOQ5CX76GEVJGHJT3ZD3G4PFNN5IBTCJBQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.613995038 (294h7m45.384636373s) [noise=true , address=104.36.82.16:4295 , piece_id=VEL6CAUOJKMRN7A7M66ZPDLCMJVD555LZPHM53A24MDQJQOW7ECQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614303969 (294h7m45.384945304s) [noise=true , address=104.36.82.9:4006 , piece_id=47FME2B5NS6WMOHC2ZGE2FZVSPEQUXKYT6E7AOSANAE5XZVDH5VQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614367721 (294h7m45.385009056s) [noise=true , address=45.91.116.4:4937 , piece_id=YMKIHJET2EN3QHEK665ACZNQNHJ3UGE4H5T4DIHWJUHJQBYO7B7Q, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614619272 (294h7m45.385260607s) [noise=true , address=77.90.31.17:4771 , piece_id=V7M65ZHCGXPD2GVJL6LTB4NP3QN6QLSIBM7U34QAYPAIKMN6EBPA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614637758 (294h7m45.385279083s) [noise=true , address=45.91.116.14:4191 , piece_id=WH2BCI36GGFF4QUD4XDYBBDMUR2R2FFVYKFTZD3ZWXPJCFJ3D7HQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614890251 (294h7m45.385531576s) [noise=true , address=104.36.82.3:4767 , piece_id=OGHG4G5ZVC7GVY4XI6DO4KINXF7THHL6IHGDB6DAPDNUDOUFSC4A, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615197719 (294h7m45.385839054s) [noise=true , address=104.36.82.12:4149 , piece_id=YBTLQEWDMFK6JIFTYR3VJDNMXTS3NMWWHX2MPP53BM2UBDI3UUEA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615413552 (294h7m45.386054887s) [noise=true , address=77.90.31.18:4805 , piece_id=I23F6KYUQ7TJI4ZLEANACVNYJDMPKKMMACIZAIA6PC5S3CKQPV2Q, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615608124 (294h7m45.386249459s) [noise=true , address=77.90.31.21:4877 , piece_id=OJBIFN2TJPQ45LQLYCMPKV5UZM6Y2MXNOBT3TSGX26U4SZAL5Q7A, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615803147 (294h7m45.386444472s) [noise=true , address=104.36.82.8:4970 , piece_id=2YJN7VKTTTCMAWQW2MNR65DAJPFMHNIBIW5NWYNG4R6HNPRFH6NQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615998511 (294h7m45.386639836s) [noise=true , address=77.90.31.16:4733 , piece_id=3GQ7NEWE6WZJMQUA3GG3GSZYXEFNOAUEVWHOEH4TSNONYXUT43GQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616193414 (294h7m45.386834749s) [noise=true , address=77.90.31.19:4842 , piece_id=R4FTRBZ3PETCA6QGU6RPUEDZZBWXF7YVH2COEW4WH3YWDLU3VJMA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616449093 (294h7m45.387090429s) [noise=true , address=77.90.31.11:4554 , piece_id=4QRZ5MPQ7HTPRIVRL3O3H4OV7CQA7ESYT5RCPI3GTHXJOHOSAOVA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616635840 (294h7m45.387277175s) [noise=true , address=45.91.116.5:4298 , piece_id=TVTUGWHFTXB32GBY4LYXT3YDB5LIFXEGA7VO5BHFA24Y6EMDTPHQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616818790 (294h7m45.387460125s) [noise=true , address=104.36.82.13:4185 , piece_id=LTQS2CUZ5NRJCBOPEXDKUQ2IJGOIW3JJAPHWTCW7WUMQZPIWRXRA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:12.265123988 (294h7m47.035765313s) [upload=235227]: Next returned nil segment Jul 5 15:18:12.265138164 (294h7m47.035779499s) [upload=235227]: Waiting for error group managing segments... Jul 5 15:18:12.550465694 (294h7m47.321107019s) [noise=true , address=104.36.82.8:4970 , piece_id=2YJN7VKTTTCMAWQW2MNR65DAJPFMHNIBIW5NWYNG4R6HNPRFH6NQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.554315629 (294h7m47.324956954s) [noise=true , address=104.36.82.12:4149 , piece_id=YBTLQEWDMFK6JIFTYR3VJDNMXTS3NMWWHX2MPP53BM2UBDI3UUEA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.603452512 (294h7m47.374093847s) [noise=true , address=104.36.82.3:4767 , piece_id=OGHG4G5ZVC7GVY4XI6DO4KINXF7THHL6IHGDB6DAPDNUDOUFSC4A, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.639796643 (294h7m47.410437978s) [noise=true , address=104.36.82.9:4006 , piece_id=47FME2B5NS6WMOHC2ZGE2FZVSPEQUXKYT6E7AOSANAE5XZVDH5VQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.639894960 (294h7m47.410536285s) [noise=true , address=104.36.82.15:4259 , piece_id=LQKQ5MGKXHTPITBR2QAOQ5CX76GEVJGHJT3ZD3G4PFNN5IBTCJBQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.647222659 (294h7m47.417863994s) [noise=true , address=104.36.82.16:4295 , piece_id=VEL6CAUOJKMRN7A7M66ZPDLCMJVD555LZPHM53A24MDQJQOW7ECQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.665424010 (294h7m47.436065346s) [noise=true , address=104.36.82.13:4185 , piece_id=LTQS2CUZ5NRJCBOPEXDKUQ2IJGOIW3JJAPHWTCW7WUMQZPIWRXRA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.702209574 (294h7m47.47285091s) [noise=true , address=45.91.116.14:4191 , piece_id=WH2BCI36GGFF4QUD4XDYBBDMUR2R2FFVYKFTZD3ZWXPJCFJ3D7HQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.722073145 (294h7m47.49271448s) [noise=true , address=45.91.116.5:4298 , piece_id=TVTUGWHFTXB32GBY4LYXT3YDB5LIFXEGA7VO5BHFA24Y6EMDTPHQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.788621529 (294h7m47.559262864s) [noise=true , address=45.91.116.4:4937 , piece_id=YMKIHJET2EN3QHEK665ACZNQNHJ3UGE4H5T4DIHWJUHJQBYO7B7Q, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:36.752254721 (294h8m11.522896046s) [noise=true , address=77.90.31.19:4842 , piece_id=R4FTRBZ3PETCA6QGU6RPUEDZZBWXF7YVH2COEW4WH3YWDLU3VJMA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:37.783437003 (294h8m12.554078338s) [noise=true , address=77.90.31.17:4771 , piece_id=V7M65ZHCGXPD2GVJL6LTB4NP3QN6QLSIBM7U34QAYPAIKMN6EBPA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:39.733398290 (294h8m14.504039626s) [noise=true , address=77.90.31.18:4805 , piece_id=I23F6KYUQ7TJI4ZLEANACVNYJDMPKKMMACIZAIA6PC5S3CKQPV2Q, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Jul 5 15:18:39.733464387 (294h8m14.504105722s) [noise=true , address=77.90.31.21:4877 , piece_id=OJBIFN2TJPQ45LQLYCMPKV5UZM6Y2MXNOBT3TSGX26U4SZAL5Q7A, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: expected piece hash; context deadline exceeded; EOF Jul 5 15:18:39.733542065 (294h8m14.50418339s) [noise=true , address=77.90.31.11:4554 , piece_id=4QRZ5MPQ7HTPRIVRL3O3H4OV7CQA7ESYT5RCPI3GTHXJOHOSAOVA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection Jul 5 15:18:39.733547496 (294h8m14.504188831s) [noise=true , address=77.90.31.16:4733 , piece_id=3GQ7NEWE6WZJMQUA3GG3GSZYXEFNOAUEVWHOEH4TSNONYXUT43GQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection; write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection Jul 5 15:18:39.733583284 (294h8m14.504224619s) [seg_pos={0 0}, upload=235227]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 5 15:18:39.974321220 (294h8m14.744962555s) [upload=235227]: Done uploading segments... Jul 5 15:18:39.974353041 (294h8m14.744994376s) [upload=235227]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.926487066s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 5.624486369s FAIL (2025-07-05 15:45:27 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== JYE6FC5T5QIBUQIILGDHNGXAUDBUIAOSKYFMPBCWPHP2SDJ3A, seg_pos={0 0}, upload=235214]: Done uploading piece. err: <nil> Jul 5 15:17:12.446386719 (294h6m47.217028064s) [seg_pos={0 0}, upload=235214]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 5 15:17:12.446476771 (294h6m47.217118106s) [upload=235214]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 5 15:17:12.446483043 (294h6m47.217124378s) [upload=235214]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 5 15:17:12.905279085 (294h6m47.67592042s) [upload=235214]: Done uploading segments... Jul 5 15:17:12.905303211 (294h6m47.675944546s) [upload=235214]: Upload finished. err: <nil> Jul 5 15:18:10.381454266 (294h7m45.152095611s) [upload=235227]: Starting upload Jul 5 15:18:10.381524069 (294h7m45.152165414s) [upload=235227]: Done starting upload Jul 5 15:18:10.381621115 (294h7m45.15226245s) [upload=235227]: Uploading segments... Jul 5 15:18:10.381740133 (294h7m45.152381468s) [upload=235227]: Got next segment. Inline: false Jul 5 15:18:10.381776342 (294h7m45.152417677s) [upload=235227]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 5 15:18:10.381781973 (294h7m45.152423308s) [upload=235227]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 5 15:18:10.613904835 (294h7m45.38454617s) [seg_pos={0 0}, upload=235227]: Begin upload segment... Jul 5 15:18:10.613947397 (294h7m45.384588732s) [seg_pos={0 0}, upload=235227]: Done begin upload segment. Jul 5 15:18:10.613968548 (294h7m45.384609883s) [noise=true , address=104.36.82.15:4259 , piece_id=LQKQ5MGKXHTPITBR2QAOQ5CX76GEVJGHJT3ZD3G4PFNN5IBTCJBQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.613995038 (294h7m45.384636373s) [noise=true , address=104.36.82.16:4295 , piece_id=VEL6CAUOJKMRN7A7M66ZPDLCMJVD555LZPHM53A24MDQJQOW7ECQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614303969 (294h7m45.384945304s) [noise=true , address=104.36.82.9:4006 , piece_id=47FME2B5NS6WMOHC2ZGE2FZVSPEQUXKYT6E7AOSANAE5XZVDH5VQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614367721 (294h7m45.385009056s) [noise=true , address=45.91.116.4:4937 , piece_id=YMKIHJET2EN3QHEK665ACZNQNHJ3UGE4H5T4DIHWJUHJQBYO7B7Q, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614619272 (294h7m45.385260607s) [noise=true , address=77.90.31.17:4771 , piece_id=V7M65ZHCGXPD2GVJL6LTB4NP3QN6QLSIBM7U34QAYPAIKMN6EBPA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614637758 (294h7m45.385279083s) [noise=true , address=45.91.116.14:4191 , piece_id=WH2BCI36GGFF4QUD4XDYBBDMUR2R2FFVYKFTZD3ZWXPJCFJ3D7HQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614890251 (294h7m45.385531576s) [noise=true , address=104.36.82.3:4767 , piece_id=OGHG4G5ZVC7GVY4XI6DO4KINXF7THHL6IHGDB6DAPDNUDOUFSC4A, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615197719 (294h7m45.385839054s) [noise=true , address=104.36.82.12:4149 , piece_id=YBTLQEWDMFK6JIFTYR3VJDNMXTS3NMWWHX2MPP53BM2UBDI3UUEA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615413552 (294h7m45.386054887s) [noise=true , address=77.90.31.18:4805 , piece_id=I23F6KYUQ7TJI4ZLEANACVNYJDMPKKMMACIZAIA6PC5S3CKQPV2Q, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615608124 (294h7m45.386249459s) [noise=true , address=77.90.31.21:4877 , piece_id=OJBIFN2TJPQ45LQLYCMPKV5UZM6Y2MXNOBT3TSGX26U4SZAL5Q7A, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615803147 (294h7m45.386444472s) [noise=true , address=104.36.82.8:4970 , piece_id=2YJN7VKTTTCMAWQW2MNR65DAJPFMHNIBIW5NWYNG4R6HNPRFH6NQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615998511 (294h7m45.386639836s) [noise=true , address=77.90.31.16:4733 , piece_id=3GQ7NEWE6WZJMQUA3GG3GSZYXEFNOAUEVWHOEH4TSNONYXUT43GQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616193414 (294h7m45.386834749s) [noise=true , address=77.90.31.19:4842 , piece_id=R4FTRBZ3PETCA6QGU6RPUEDZZBWXF7YVH2COEW4WH3YWDLU3VJMA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616449093 (294h7m45.387090429s) [noise=true , address=77.90.31.11:4554 , piece_id=4QRZ5MPQ7HTPRIVRL3O3H4OV7CQA7ESYT5RCPI3GTHXJOHOSAOVA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616635840 (294h7m45.387277175s) [noise=true , address=45.91.116.5:4298 , piece_id=TVTUGWHFTXB32GBY4LYXT3YDB5LIFXEGA7VO5BHFA24Y6EMDTPHQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616818790 (294h7m45.387460125s) [noise=true , address=104.36.82.13:4185 , piece_id=LTQS2CUZ5NRJCBOPEXDKUQ2IJGOIW3JJAPHWTCW7WUMQZPIWRXRA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:12.265123988 (294h7m47.035765313s) [upload=235227]: Next returned nil segment Jul 5 15:18:12.265138164 (294h7m47.035779499s) [upload=235227]: Waiting for error group managing segments... Jul 5 15:18:12.550465694 (294h7m47.321107019s) [noise=true , address=104.36.82.8:4970 , piece_id=2YJN7VKTTTCMAWQW2MNR65DAJPFMHNIBIW5NWYNG4R6HNPRFH6NQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.554315629 (294h7m47.324956954s) [noise=true , address=104.36.82.12:4149 , piece_id=YBTLQEWDMFK6JIFTYR3VJDNMXTS3NMWWHX2MPP53BM2UBDI3UUEA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.603452512 (294h7m47.374093847s) [noise=true , address=104.36.82.3:4767 , piece_id=OGHG4G5ZVC7GVY4XI6DO4KINXF7THHL6IHGDB6DAPDNUDOUFSC4A, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.639796643 (294h7m47.410437978s) [noise=true , address=104.36.82.9:4006 , piece_id=47FME2B5NS6WMOHC2ZGE2FZVSPEQUXKYT6E7AOSANAE5XZVDH5VQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.639894960 (294h7m47.410536285s) [noise=true , address=104.36.82.15:4259 , piece_id=LQKQ5MGKXHTPITBR2QAOQ5CX76GEVJGHJT3ZD3G4PFNN5IBTCJBQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.647222659 (294h7m47.417863994s) [noise=true , address=104.36.82.16:4295 , piece_id=VEL6CAUOJKMRN7A7M66ZPDLCMJVD555LZPHM53A24MDQJQOW7ECQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.665424010 (294h7m47.436065346s) [noise=true , address=104.36.82.13:4185 , piece_id=LTQS2CUZ5NRJCBOPEXDKUQ2IJGOIW3JJAPHWTCW7WUMQZPIWRXRA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.702209574 (294h7m47.47285091s) [noise=true , address=45.91.116.14:4191 , piece_id=WH2BCI36GGFF4QUD4XDYBBDMUR2R2FFVYKFTZD3ZWXPJCFJ3D7HQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.722073145 (294h7m47.49271448s) [noise=true , address=45.91.116.5:4298 , piece_id=TVTUGWHFTXB32GBY4LYXT3YDB5LIFXEGA7VO5BHFA24Y6EMDTPHQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.788621529 (294h7m47.559262864s) [noise=true , address=45.91.116.4:4937 , piece_id=YMKIHJET2EN3QHEK665ACZNQNHJ3UGE4H5T4DIHWJUHJQBYO7B7Q, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:36.752254721 (294h8m11.522896046s) [noise=true , address=77.90.31.19:4842 , piece_id=R4FTRBZ3PETCA6QGU6RPUEDZZBWXF7YVH2COEW4WH3YWDLU3VJMA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:37.783437003 (294h8m12.554078338s) [noise=true , address=77.90.31.17:4771 , piece_id=V7M65ZHCGXPD2GVJL6LTB4NP3QN6QLSIBM7U34QAYPAIKMN6EBPA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:39.733398290 (294h8m14.504039626s) [noise=true , address=77.90.31.18:4805 , piece_id=I23F6KYUQ7TJI4ZLEANACVNYJDMPKKMMACIZAIA6PC5S3CKQPV2Q, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Jul 5 15:18:39.733464387 (294h8m14.504105722s) [noise=true , address=77.90.31.21:4877 , piece_id=OJBIFN2TJPQ45LQLYCMPKV5UZM6Y2MXNOBT3TSGX26U4SZAL5Q7A, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: expected piece hash; context deadline exceeded; EOF Jul 5 15:18:39.733542065 (294h8m14.50418339s) [noise=true , address=77.90.31.11:4554 , piece_id=4QRZ5MPQ7HTPRIVRL3O3H4OV7CQA7ESYT5RCPI3GTHXJOHOSAOVA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection Jul 5 15:18:39.733547496 (294h8m14.504188831s) [noise=true , address=77.90.31.16:4733 , piece_id=3GQ7NEWE6WZJMQUA3GG3GSZYXEFNOAUEVWHOEH4TSNONYXUT43GQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection; write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection Jul 5 15:18:39.733583284 (294h8m14.504224619s) [seg_pos={0 0}, upload=235227]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 5 15:18:39.974321220 (294h8m14.744962555s) [upload=235227]: Done uploading segments... Jul 5 15:18:39.974353041 (294h8m14.744994376s) [upload=235227]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.624521886s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 5.70554898s FAIL (2025-07-05 15:44:34 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== JYE6FC5T5QIBUQIILGDHNGXAUDBUIAOSKYFMPBCWPHP2SDJ3A, seg_pos={0 0}, upload=235214]: Done uploading piece. err: <nil> Jul 5 15:17:12.446386719 (294h6m47.217028064s) [seg_pos={0 0}, upload=235214]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 5 15:17:12.446476771 (294h6m47.217118106s) [upload=235214]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 5 15:17:12.446483043 (294h6m47.217124378s) [upload=235214]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 5 15:17:12.905279085 (294h6m47.67592042s) [upload=235214]: Done uploading segments... Jul 5 15:17:12.905303211 (294h6m47.675944546s) [upload=235214]: Upload finished. err: <nil> Jul 5 15:18:10.381454266 (294h7m45.152095611s) [upload=235227]: Starting upload Jul 5 15:18:10.381524069 (294h7m45.152165414s) [upload=235227]: Done starting upload Jul 5 15:18:10.381621115 (294h7m45.15226245s) [upload=235227]: Uploading segments... Jul 5 15:18:10.381740133 (294h7m45.152381468s) [upload=235227]: Got next segment. Inline: false Jul 5 15:18:10.381776342 (294h7m45.152417677s) [upload=235227]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 5 15:18:10.381781973 (294h7m45.152423308s) [upload=235227]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 5 15:18:10.613904835 (294h7m45.38454617s) [seg_pos={0 0}, upload=235227]: Begin upload segment... Jul 5 15:18:10.613947397 (294h7m45.384588732s) [seg_pos={0 0}, upload=235227]: Done begin upload segment. Jul 5 15:18:10.613968548 (294h7m45.384609883s) [noise=true , address=104.36.82.15:4259 , piece_id=LQKQ5MGKXHTPITBR2QAOQ5CX76GEVJGHJT3ZD3G4PFNN5IBTCJBQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.613995038 (294h7m45.384636373s) [noise=true , address=104.36.82.16:4295 , piece_id=VEL6CAUOJKMRN7A7M66ZPDLCMJVD555LZPHM53A24MDQJQOW7ECQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614303969 (294h7m45.384945304s) [noise=true , address=104.36.82.9:4006 , piece_id=47FME2B5NS6WMOHC2ZGE2FZVSPEQUXKYT6E7AOSANAE5XZVDH5VQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614367721 (294h7m45.385009056s) [noise=true , address=45.91.116.4:4937 , piece_id=YMKIHJET2EN3QHEK665ACZNQNHJ3UGE4H5T4DIHWJUHJQBYO7B7Q, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614619272 (294h7m45.385260607s) [noise=true , address=77.90.31.17:4771 , piece_id=V7M65ZHCGXPD2GVJL6LTB4NP3QN6QLSIBM7U34QAYPAIKMN6EBPA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614637758 (294h7m45.385279083s) [noise=true , address=45.91.116.14:4191 , piece_id=WH2BCI36GGFF4QUD4XDYBBDMUR2R2FFVYKFTZD3ZWXPJCFJ3D7HQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614890251 (294h7m45.385531576s) [noise=true , address=104.36.82.3:4767 , piece_id=OGHG4G5ZVC7GVY4XI6DO4KINXF7THHL6IHGDB6DAPDNUDOUFSC4A, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615197719 (294h7m45.385839054s) [noise=true , address=104.36.82.12:4149 , piece_id=YBTLQEWDMFK6JIFTYR3VJDNMXTS3NMWWHX2MPP53BM2UBDI3UUEA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615413552 (294h7m45.386054887s) [noise=true , address=77.90.31.18:4805 , piece_id=I23F6KYUQ7TJI4ZLEANACVNYJDMPKKMMACIZAIA6PC5S3CKQPV2Q, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615608124 (294h7m45.386249459s) [noise=true , address=77.90.31.21:4877 , piece_id=OJBIFN2TJPQ45LQLYCMPKV5UZM6Y2MXNOBT3TSGX26U4SZAL5Q7A, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615803147 (294h7m45.386444472s) [noise=true , address=104.36.82.8:4970 , piece_id=2YJN7VKTTTCMAWQW2MNR65DAJPFMHNIBIW5NWYNG4R6HNPRFH6NQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615998511 (294h7m45.386639836s) [noise=true , address=77.90.31.16:4733 , piece_id=3GQ7NEWE6WZJMQUA3GG3GSZYXEFNOAUEVWHOEH4TSNONYXUT43GQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616193414 (294h7m45.386834749s) [noise=true , address=77.90.31.19:4842 , piece_id=R4FTRBZ3PETCA6QGU6RPUEDZZBWXF7YVH2COEW4WH3YWDLU3VJMA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616449093 (294h7m45.387090429s) [noise=true , address=77.90.31.11:4554 , piece_id=4QRZ5MPQ7HTPRIVRL3O3H4OV7CQA7ESYT5RCPI3GTHXJOHOSAOVA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616635840 (294h7m45.387277175s) [noise=true , address=45.91.116.5:4298 , piece_id=TVTUGWHFTXB32GBY4LYXT3YDB5LIFXEGA7VO5BHFA24Y6EMDTPHQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616818790 (294h7m45.387460125s) [noise=true , address=104.36.82.13:4185 , piece_id=LTQS2CUZ5NRJCBOPEXDKUQ2IJGOIW3JJAPHWTCW7WUMQZPIWRXRA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:12.265123988 (294h7m47.035765313s) [upload=235227]: Next returned nil segment Jul 5 15:18:12.265138164 (294h7m47.035779499s) [upload=235227]: Waiting for error group managing segments... Jul 5 15:18:12.550465694 (294h7m47.321107019s) [noise=true , address=104.36.82.8:4970 , piece_id=2YJN7VKTTTCMAWQW2MNR65DAJPFMHNIBIW5NWYNG4R6HNPRFH6NQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.554315629 (294h7m47.324956954s) [noise=true , address=104.36.82.12:4149 , piece_id=YBTLQEWDMFK6JIFTYR3VJDNMXTS3NMWWHX2MPP53BM2UBDI3UUEA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.603452512 (294h7m47.374093847s) [noise=true , address=104.36.82.3:4767 , piece_id=OGHG4G5ZVC7GVY4XI6DO4KINXF7THHL6IHGDB6DAPDNUDOUFSC4A, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.639796643 (294h7m47.410437978s) [noise=true , address=104.36.82.9:4006 , piece_id=47FME2B5NS6WMOHC2ZGE2FZVSPEQUXKYT6E7AOSANAE5XZVDH5VQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.639894960 (294h7m47.410536285s) [noise=true , address=104.36.82.15:4259 , piece_id=LQKQ5MGKXHTPITBR2QAOQ5CX76GEVJGHJT3ZD3G4PFNN5IBTCJBQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.647222659 (294h7m47.417863994s) [noise=true , address=104.36.82.16:4295 , piece_id=VEL6CAUOJKMRN7A7M66ZPDLCMJVD555LZPHM53A24MDQJQOW7ECQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.665424010 (294h7m47.436065346s) [noise=true , address=104.36.82.13:4185 , piece_id=LTQS2CUZ5NRJCBOPEXDKUQ2IJGOIW3JJAPHWTCW7WUMQZPIWRXRA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.702209574 (294h7m47.47285091s) [noise=true , address=45.91.116.14:4191 , piece_id=WH2BCI36GGFF4QUD4XDYBBDMUR2R2FFVYKFTZD3ZWXPJCFJ3D7HQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.722073145 (294h7m47.49271448s) [noise=true , address=45.91.116.5:4298 , piece_id=TVTUGWHFTXB32GBY4LYXT3YDB5LIFXEGA7VO5BHFA24Y6EMDTPHQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.788621529 (294h7m47.559262864s) [noise=true , address=45.91.116.4:4937 , piece_id=YMKIHJET2EN3QHEK665ACZNQNHJ3UGE4H5T4DIHWJUHJQBYO7B7Q, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:36.752254721 (294h8m11.522896046s) [noise=true , address=77.90.31.19:4842 , piece_id=R4FTRBZ3PETCA6QGU6RPUEDZZBWXF7YVH2COEW4WH3YWDLU3VJMA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:37.783437003 (294h8m12.554078338s) [noise=true , address=77.90.31.17:4771 , piece_id=V7M65ZHCGXPD2GVJL6LTB4NP3QN6QLSIBM7U34QAYPAIKMN6EBPA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:39.733398290 (294h8m14.504039626s) [noise=true , address=77.90.31.18:4805 , piece_id=I23F6KYUQ7TJI4ZLEANACVNYJDMPKKMMACIZAIA6PC5S3CKQPV2Q, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Jul 5 15:18:39.733464387 (294h8m14.504105722s) [noise=true , address=77.90.31.21:4877 , piece_id=OJBIFN2TJPQ45LQLYCMPKV5UZM6Y2MXNOBT3TSGX26U4SZAL5Q7A, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: expected piece hash; context deadline exceeded; EOF Jul 5 15:18:39.733542065 (294h8m14.50418339s) [noise=true , address=77.90.31.11:4554 , piece_id=4QRZ5MPQ7HTPRIVRL3O3H4OV7CQA7ESYT5RCPI3GTHXJOHOSAOVA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection Jul 5 15:18:39.733547496 (294h8m14.504188831s) [noise=true , address=77.90.31.16:4733 , piece_id=3GQ7NEWE6WZJMQUA3GG3GSZYXEFNOAUEVWHOEH4TSNONYXUT43GQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection; write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection Jul 5 15:18:39.733583284 (294h8m14.504224619s) [seg_pos={0 0}, upload=235227]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 5 15:18:39.974321220 (294h8m14.744962555s) [upload=235227]: Done uploading segments... Jul 5 15:18:39.974353041 (294h8m14.744994376s) [upload=235227]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.705575371s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 5.884479712s FAIL (2025-07-05 15:43:39 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== JYE6FC5T5QIBUQIILGDHNGXAUDBUIAOSKYFMPBCWPHP2SDJ3A, seg_pos={0 0}, upload=235214]: Done uploading piece. err: <nil> Jul 5 15:17:12.446386719 (294h6m47.217028064s) [seg_pos={0 0}, upload=235214]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 5 15:17:12.446476771 (294h6m47.217118106s) [upload=235214]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 5 15:17:12.446483043 (294h6m47.217124378s) [upload=235214]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 5 15:17:12.905279085 (294h6m47.67592042s) [upload=235214]: Done uploading segments... Jul 5 15:17:12.905303211 (294h6m47.675944546s) [upload=235214]: Upload finished. err: <nil> Jul 5 15:18:10.381454266 (294h7m45.152095611s) [upload=235227]: Starting upload Jul 5 15:18:10.381524069 (294h7m45.152165414s) [upload=235227]: Done starting upload Jul 5 15:18:10.381621115 (294h7m45.15226245s) [upload=235227]: Uploading segments... Jul 5 15:18:10.381740133 (294h7m45.152381468s) [upload=235227]: Got next segment. Inline: false Jul 5 15:18:10.381776342 (294h7m45.152417677s) [upload=235227]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 5 15:18:10.381781973 (294h7m45.152423308s) [upload=235227]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 5 15:18:10.613904835 (294h7m45.38454617s) [seg_pos={0 0}, upload=235227]: Begin upload segment... Jul 5 15:18:10.613947397 (294h7m45.384588732s) [seg_pos={0 0}, upload=235227]: Done begin upload segment. Jul 5 15:18:10.613968548 (294h7m45.384609883s) [noise=true , address=104.36.82.15:4259 , piece_id=LQKQ5MGKXHTPITBR2QAOQ5CX76GEVJGHJT3ZD3G4PFNN5IBTCJBQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.613995038 (294h7m45.384636373s) [noise=true , address=104.36.82.16:4295 , piece_id=VEL6CAUOJKMRN7A7M66ZPDLCMJVD555LZPHM53A24MDQJQOW7ECQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614303969 (294h7m45.384945304s) [noise=true , address=104.36.82.9:4006 , piece_id=47FME2B5NS6WMOHC2ZGE2FZVSPEQUXKYT6E7AOSANAE5XZVDH5VQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614367721 (294h7m45.385009056s) [noise=true , address=45.91.116.4:4937 , piece_id=YMKIHJET2EN3QHEK665ACZNQNHJ3UGE4H5T4DIHWJUHJQBYO7B7Q, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614619272 (294h7m45.385260607s) [noise=true , address=77.90.31.17:4771 , piece_id=V7M65ZHCGXPD2GVJL6LTB4NP3QN6QLSIBM7U34QAYPAIKMN6EBPA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614637758 (294h7m45.385279083s) [noise=true , address=45.91.116.14:4191 , piece_id=WH2BCI36GGFF4QUD4XDYBBDMUR2R2FFVYKFTZD3ZWXPJCFJ3D7HQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614890251 (294h7m45.385531576s) [noise=true , address=104.36.82.3:4767 , piece_id=OGHG4G5ZVC7GVY4XI6DO4KINXF7THHL6IHGDB6DAPDNUDOUFSC4A, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615197719 (294h7m45.385839054s) [noise=true , address=104.36.82.12:4149 , piece_id=YBTLQEWDMFK6JIFTYR3VJDNMXTS3NMWWHX2MPP53BM2UBDI3UUEA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615413552 (294h7m45.386054887s) [noise=true , address=77.90.31.18:4805 , piece_id=I23F6KYUQ7TJI4ZLEANACVNYJDMPKKMMACIZAIA6PC5S3CKQPV2Q, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615608124 (294h7m45.386249459s) [noise=true , address=77.90.31.21:4877 , piece_id=OJBIFN2TJPQ45LQLYCMPKV5UZM6Y2MXNOBT3TSGX26U4SZAL5Q7A, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615803147 (294h7m45.386444472s) [noise=true , address=104.36.82.8:4970 , piece_id=2YJN7VKTTTCMAWQW2MNR65DAJPFMHNIBIW5NWYNG4R6HNPRFH6NQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615998511 (294h7m45.386639836s) [noise=true , address=77.90.31.16:4733 , piece_id=3GQ7NEWE6WZJMQUA3GG3GSZYXEFNOAUEVWHOEH4TSNONYXUT43GQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616193414 (294h7m45.386834749s) [noise=true , address=77.90.31.19:4842 , piece_id=R4FTRBZ3PETCA6QGU6RPUEDZZBWXF7YVH2COEW4WH3YWDLU3VJMA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616449093 (294h7m45.387090429s) [noise=true , address=77.90.31.11:4554 , piece_id=4QRZ5MPQ7HTPRIVRL3O3H4OV7CQA7ESYT5RCPI3GTHXJOHOSAOVA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616635840 (294h7m45.387277175s) [noise=true , address=45.91.116.5:4298 , piece_id=TVTUGWHFTXB32GBY4LYXT3YDB5LIFXEGA7VO5BHFA24Y6EMDTPHQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616818790 (294h7m45.387460125s) [noise=true , address=104.36.82.13:4185 , piece_id=LTQS2CUZ5NRJCBOPEXDKUQ2IJGOIW3JJAPHWTCW7WUMQZPIWRXRA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:12.265123988 (294h7m47.035765313s) [upload=235227]: Next returned nil segment Jul 5 15:18:12.265138164 (294h7m47.035779499s) [upload=235227]: Waiting for error group managing segments... Jul 5 15:18:12.550465694 (294h7m47.321107019s) [noise=true , address=104.36.82.8:4970 , piece_id=2YJN7VKTTTCMAWQW2MNR65DAJPFMHNIBIW5NWYNG4R6HNPRFH6NQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.554315629 (294h7m47.324956954s) [noise=true , address=104.36.82.12:4149 , piece_id=YBTLQEWDMFK6JIFTYR3VJDNMXTS3NMWWHX2MPP53BM2UBDI3UUEA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.603452512 (294h7m47.374093847s) [noise=true , address=104.36.82.3:4767 , piece_id=OGHG4G5ZVC7GVY4XI6DO4KINXF7THHL6IHGDB6DAPDNUDOUFSC4A, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.639796643 (294h7m47.410437978s) [noise=true , address=104.36.82.9:4006 , piece_id=47FME2B5NS6WMOHC2ZGE2FZVSPEQUXKYT6E7AOSANAE5XZVDH5VQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.639894960 (294h7m47.410536285s) [noise=true , address=104.36.82.15:4259 , piece_id=LQKQ5MGKXHTPITBR2QAOQ5CX76GEVJGHJT3ZD3G4PFNN5IBTCJBQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.647222659 (294h7m47.417863994s) [noise=true , address=104.36.82.16:4295 , piece_id=VEL6CAUOJKMRN7A7M66ZPDLCMJVD555LZPHM53A24MDQJQOW7ECQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.665424010 (294h7m47.436065346s) [noise=true , address=104.36.82.13:4185 , piece_id=LTQS2CUZ5NRJCBOPEXDKUQ2IJGOIW3JJAPHWTCW7WUMQZPIWRXRA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.702209574 (294h7m47.47285091s) [noise=true , address=45.91.116.14:4191 , piece_id=WH2BCI36GGFF4QUD4XDYBBDMUR2R2FFVYKFTZD3ZWXPJCFJ3D7HQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.722073145 (294h7m47.49271448s) [noise=true , address=45.91.116.5:4298 , piece_id=TVTUGWHFTXB32GBY4LYXT3YDB5LIFXEGA7VO5BHFA24Y6EMDTPHQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.788621529 (294h7m47.559262864s) [noise=true , address=45.91.116.4:4937 , piece_id=YMKIHJET2EN3QHEK665ACZNQNHJ3UGE4H5T4DIHWJUHJQBYO7B7Q, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:36.752254721 (294h8m11.522896046s) [noise=true , address=77.90.31.19:4842 , piece_id=R4FTRBZ3PETCA6QGU6RPUEDZZBWXF7YVH2COEW4WH3YWDLU3VJMA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:37.783437003 (294h8m12.554078338s) [noise=true , address=77.90.31.17:4771 , piece_id=V7M65ZHCGXPD2GVJL6LTB4NP3QN6QLSIBM7U34QAYPAIKMN6EBPA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:39.733398290 (294h8m14.504039626s) [noise=true , address=77.90.31.18:4805 , piece_id=I23F6KYUQ7TJI4ZLEANACVNYJDMPKKMMACIZAIA6PC5S3CKQPV2Q, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Jul 5 15:18:39.733464387 (294h8m14.504105722s) [noise=true , address=77.90.31.21:4877 , piece_id=OJBIFN2TJPQ45LQLYCMPKV5UZM6Y2MXNOBT3TSGX26U4SZAL5Q7A, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: expected piece hash; context deadline exceeded; EOF Jul 5 15:18:39.733542065 (294h8m14.50418339s) [noise=true , address=77.90.31.11:4554 , piece_id=4QRZ5MPQ7HTPRIVRL3O3H4OV7CQA7ESYT5RCPI3GTHXJOHOSAOVA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection Jul 5 15:18:39.733547496 (294h8m14.504188831s) [noise=true , address=77.90.31.16:4733 , piece_id=3GQ7NEWE6WZJMQUA3GG3GSZYXEFNOAUEVWHOEH4TSNONYXUT43GQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection; write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection Jul 5 15:18:39.733583284 (294h8m14.504224619s) [seg_pos={0 0}, upload=235227]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 5 15:18:39.974321220 (294h8m14.744962555s) [upload=235227]: Done uploading segments... Jul 5 15:18:39.974353041 (294h8m14.744994376s) [upload=235227]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.884506764s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 5.68693034s FAIL (2025-07-05 15:43:13 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== JYE6FC5T5QIBUQIILGDHNGXAUDBUIAOSKYFMPBCWPHP2SDJ3A, seg_pos={0 0}, upload=235214]: Done uploading piece. err: <nil> Jul 5 15:17:12.446386719 (294h6m47.217028064s) [seg_pos={0 0}, upload=235214]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 5 15:17:12.446476771 (294h6m47.217118106s) [upload=235214]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 5 15:17:12.446483043 (294h6m47.217124378s) [upload=235214]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 5 15:17:12.905279085 (294h6m47.67592042s) [upload=235214]: Done uploading segments... Jul 5 15:17:12.905303211 (294h6m47.675944546s) [upload=235214]: Upload finished. err: <nil> Jul 5 15:18:10.381454266 (294h7m45.152095611s) [upload=235227]: Starting upload Jul 5 15:18:10.381524069 (294h7m45.152165414s) [upload=235227]: Done starting upload Jul 5 15:18:10.381621115 (294h7m45.15226245s) [upload=235227]: Uploading segments... Jul 5 15:18:10.381740133 (294h7m45.152381468s) [upload=235227]: Got next segment. Inline: false Jul 5 15:18:10.381776342 (294h7m45.152417677s) [upload=235227]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 5 15:18:10.381781973 (294h7m45.152423308s) [upload=235227]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 5 15:18:10.613904835 (294h7m45.38454617s) [seg_pos={0 0}, upload=235227]: Begin upload segment... Jul 5 15:18:10.613947397 (294h7m45.384588732s) [seg_pos={0 0}, upload=235227]: Done begin upload segment. Jul 5 15:18:10.613968548 (294h7m45.384609883s) [noise=true , address=104.36.82.15:4259 , piece_id=LQKQ5MGKXHTPITBR2QAOQ5CX76GEVJGHJT3ZD3G4PFNN5IBTCJBQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.613995038 (294h7m45.384636373s) [noise=true , address=104.36.82.16:4295 , piece_id=VEL6CAUOJKMRN7A7M66ZPDLCMJVD555LZPHM53A24MDQJQOW7ECQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614303969 (294h7m45.384945304s) [noise=true , address=104.36.82.9:4006 , piece_id=47FME2B5NS6WMOHC2ZGE2FZVSPEQUXKYT6E7AOSANAE5XZVDH5VQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614367721 (294h7m45.385009056s) [noise=true , address=45.91.116.4:4937 , piece_id=YMKIHJET2EN3QHEK665ACZNQNHJ3UGE4H5T4DIHWJUHJQBYO7B7Q, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614619272 (294h7m45.385260607s) [noise=true , address=77.90.31.17:4771 , piece_id=V7M65ZHCGXPD2GVJL6LTB4NP3QN6QLSIBM7U34QAYPAIKMN6EBPA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614637758 (294h7m45.385279083s) [noise=true , address=45.91.116.14:4191 , piece_id=WH2BCI36GGFF4QUD4XDYBBDMUR2R2FFVYKFTZD3ZWXPJCFJ3D7HQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614890251 (294h7m45.385531576s) [noise=true , address=104.36.82.3:4767 , piece_id=OGHG4G5ZVC7GVY4XI6DO4KINXF7THHL6IHGDB6DAPDNUDOUFSC4A, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615197719 (294h7m45.385839054s) [noise=true , address=104.36.82.12:4149 , piece_id=YBTLQEWDMFK6JIFTYR3VJDNMXTS3NMWWHX2MPP53BM2UBDI3UUEA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615413552 (294h7m45.386054887s) [noise=true , address=77.90.31.18:4805 , piece_id=I23F6KYUQ7TJI4ZLEANACVNYJDMPKKMMACIZAIA6PC5S3CKQPV2Q, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615608124 (294h7m45.386249459s) [noise=true , address=77.90.31.21:4877 , piece_id=OJBIFN2TJPQ45LQLYCMPKV5UZM6Y2MXNOBT3TSGX26U4SZAL5Q7A, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615803147 (294h7m45.386444472s) [noise=true , address=104.36.82.8:4970 , piece_id=2YJN7VKTTTCMAWQW2MNR65DAJPFMHNIBIW5NWYNG4R6HNPRFH6NQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615998511 (294h7m45.386639836s) [noise=true , address=77.90.31.16:4733 , piece_id=3GQ7NEWE6WZJMQUA3GG3GSZYXEFNOAUEVWHOEH4TSNONYXUT43GQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616193414 (294h7m45.386834749s) [noise=true , address=77.90.31.19:4842 , piece_id=R4FTRBZ3PETCA6QGU6RPUEDZZBWXF7YVH2COEW4WH3YWDLU3VJMA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616449093 (294h7m45.387090429s) [noise=true , address=77.90.31.11:4554 , piece_id=4QRZ5MPQ7HTPRIVRL3O3H4OV7CQA7ESYT5RCPI3GTHXJOHOSAOVA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616635840 (294h7m45.387277175s) [noise=true , address=45.91.116.5:4298 , piece_id=TVTUGWHFTXB32GBY4LYXT3YDB5LIFXEGA7VO5BHFA24Y6EMDTPHQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616818790 (294h7m45.387460125s) [noise=true , address=104.36.82.13:4185 , piece_id=LTQS2CUZ5NRJCBOPEXDKUQ2IJGOIW3JJAPHWTCW7WUMQZPIWRXRA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:12.265123988 (294h7m47.035765313s) [upload=235227]: Next returned nil segment Jul 5 15:18:12.265138164 (294h7m47.035779499s) [upload=235227]: Waiting for error group managing segments... Jul 5 15:18:12.550465694 (294h7m47.321107019s) [noise=true , address=104.36.82.8:4970 , piece_id=2YJN7VKTTTCMAWQW2MNR65DAJPFMHNIBIW5NWYNG4R6HNPRFH6NQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.554315629 (294h7m47.324956954s) [noise=true , address=104.36.82.12:4149 , piece_id=YBTLQEWDMFK6JIFTYR3VJDNMXTS3NMWWHX2MPP53BM2UBDI3UUEA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.603452512 (294h7m47.374093847s) [noise=true , address=104.36.82.3:4767 , piece_id=OGHG4G5ZVC7GVY4XI6DO4KINXF7THHL6IHGDB6DAPDNUDOUFSC4A, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.639796643 (294h7m47.410437978s) [noise=true , address=104.36.82.9:4006 , piece_id=47FME2B5NS6WMOHC2ZGE2FZVSPEQUXKYT6E7AOSANAE5XZVDH5VQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.639894960 (294h7m47.410536285s) [noise=true , address=104.36.82.15:4259 , piece_id=LQKQ5MGKXHTPITBR2QAOQ5CX76GEVJGHJT3ZD3G4PFNN5IBTCJBQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.647222659 (294h7m47.417863994s) [noise=true , address=104.36.82.16:4295 , piece_id=VEL6CAUOJKMRN7A7M66ZPDLCMJVD555LZPHM53A24MDQJQOW7ECQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.665424010 (294h7m47.436065346s) [noise=true , address=104.36.82.13:4185 , piece_id=LTQS2CUZ5NRJCBOPEXDKUQ2IJGOIW3JJAPHWTCW7WUMQZPIWRXRA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.702209574 (294h7m47.47285091s) [noise=true , address=45.91.116.14:4191 , piece_id=WH2BCI36GGFF4QUD4XDYBBDMUR2R2FFVYKFTZD3ZWXPJCFJ3D7HQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.722073145 (294h7m47.49271448s) [noise=true , address=45.91.116.5:4298 , piece_id=TVTUGWHFTXB32GBY4LYXT3YDB5LIFXEGA7VO5BHFA24Y6EMDTPHQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.788621529 (294h7m47.559262864s) [noise=true , address=45.91.116.4:4937 , piece_id=YMKIHJET2EN3QHEK665ACZNQNHJ3UGE4H5T4DIHWJUHJQBYO7B7Q, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:36.752254721 (294h8m11.522896046s) [noise=true , address=77.90.31.19:4842 , piece_id=R4FTRBZ3PETCA6QGU6RPUEDZZBWXF7YVH2COEW4WH3YWDLU3VJMA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:37.783437003 (294h8m12.554078338s) [noise=true , address=77.90.31.17:4771 , piece_id=V7M65ZHCGXPD2GVJL6LTB4NP3QN6QLSIBM7U34QAYPAIKMN6EBPA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:39.733398290 (294h8m14.504039626s) [noise=true , address=77.90.31.18:4805 , piece_id=I23F6KYUQ7TJI4ZLEANACVNYJDMPKKMMACIZAIA6PC5S3CKQPV2Q, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Jul 5 15:18:39.733464387 (294h8m14.504105722s) [noise=true , address=77.90.31.21:4877 , piece_id=OJBIFN2TJPQ45LQLYCMPKV5UZM6Y2MXNOBT3TSGX26U4SZAL5Q7A, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: expected piece hash; context deadline exceeded; EOF Jul 5 15:18:39.733542065 (294h8m14.50418339s) [noise=true , address=77.90.31.11:4554 , piece_id=4QRZ5MPQ7HTPRIVRL3O3H4OV7CQA7ESYT5RCPI3GTHXJOHOSAOVA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection Jul 5 15:18:39.733547496 (294h8m14.504188831s) [noise=true , address=77.90.31.16:4733 , piece_id=3GQ7NEWE6WZJMQUA3GG3GSZYXEFNOAUEVWHOEH4TSNONYXUT43GQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection; write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection Jul 5 15:18:39.733583284 (294h8m14.504224619s) [seg_pos={0 0}, upload=235227]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 5 15:18:39.974321220 (294h8m14.744962555s) [upload=235227]: Done uploading segments... Jul 5 15:18:39.974353041 (294h8m14.744994376s) [upload=235227]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.686953465s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 5.669857531s FAIL (2025-07-05 15:42:57 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== JYE6FC5T5QIBUQIILGDHNGXAUDBUIAOSKYFMPBCWPHP2SDJ3A, seg_pos={0 0}, upload=235214]: Done uploading piece. err: <nil> Jul 5 15:17:12.446386719 (294h6m47.217028064s) [seg_pos={0 0}, upload=235214]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 5 15:17:12.446476771 (294h6m47.217118106s) [upload=235214]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 5 15:17:12.446483043 (294h6m47.217124378s) [upload=235214]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 5 15:17:12.905279085 (294h6m47.67592042s) [upload=235214]: Done uploading segments... Jul 5 15:17:12.905303211 (294h6m47.675944546s) [upload=235214]: Upload finished. err: <nil> Jul 5 15:18:10.381454266 (294h7m45.152095611s) [upload=235227]: Starting upload Jul 5 15:18:10.381524069 (294h7m45.152165414s) [upload=235227]: Done starting upload Jul 5 15:18:10.381621115 (294h7m45.15226245s) [upload=235227]: Uploading segments... Jul 5 15:18:10.381740133 (294h7m45.152381468s) [upload=235227]: Got next segment. Inline: false Jul 5 15:18:10.381776342 (294h7m45.152417677s) [upload=235227]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 5 15:18:10.381781973 (294h7m45.152423308s) [upload=235227]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 5 15:18:10.613904835 (294h7m45.38454617s) [seg_pos={0 0}, upload=235227]: Begin upload segment... Jul 5 15:18:10.613947397 (294h7m45.384588732s) [seg_pos={0 0}, upload=235227]: Done begin upload segment. Jul 5 15:18:10.613968548 (294h7m45.384609883s) [noise=true , address=104.36.82.15:4259 , piece_id=LQKQ5MGKXHTPITBR2QAOQ5CX76GEVJGHJT3ZD3G4PFNN5IBTCJBQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.613995038 (294h7m45.384636373s) [noise=true , address=104.36.82.16:4295 , piece_id=VEL6CAUOJKMRN7A7M66ZPDLCMJVD555LZPHM53A24MDQJQOW7ECQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614303969 (294h7m45.384945304s) [noise=true , address=104.36.82.9:4006 , piece_id=47FME2B5NS6WMOHC2ZGE2FZVSPEQUXKYT6E7AOSANAE5XZVDH5VQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614367721 (294h7m45.385009056s) [noise=true , address=45.91.116.4:4937 , piece_id=YMKIHJET2EN3QHEK665ACZNQNHJ3UGE4H5T4DIHWJUHJQBYO7B7Q, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614619272 (294h7m45.385260607s) [noise=true , address=77.90.31.17:4771 , piece_id=V7M65ZHCGXPD2GVJL6LTB4NP3QN6QLSIBM7U34QAYPAIKMN6EBPA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614637758 (294h7m45.385279083s) [noise=true , address=45.91.116.14:4191 , piece_id=WH2BCI36GGFF4QUD4XDYBBDMUR2R2FFVYKFTZD3ZWXPJCFJ3D7HQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614890251 (294h7m45.385531576s) [noise=true , address=104.36.82.3:4767 , piece_id=OGHG4G5ZVC7GVY4XI6DO4KINXF7THHL6IHGDB6DAPDNUDOUFSC4A, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615197719 (294h7m45.385839054s) [noise=true , address=104.36.82.12:4149 , piece_id=YBTLQEWDMFK6JIFTYR3VJDNMXTS3NMWWHX2MPP53BM2UBDI3UUEA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615413552 (294h7m45.386054887s) [noise=true , address=77.90.31.18:4805 , piece_id=I23F6KYUQ7TJI4ZLEANACVNYJDMPKKMMACIZAIA6PC5S3CKQPV2Q, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615608124 (294h7m45.386249459s) [noise=true , address=77.90.31.21:4877 , piece_id=OJBIFN2TJPQ45LQLYCMPKV5UZM6Y2MXNOBT3TSGX26U4SZAL5Q7A, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615803147 (294h7m45.386444472s) [noise=true , address=104.36.82.8:4970 , piece_id=2YJN7VKTTTCMAWQW2MNR65DAJPFMHNIBIW5NWYNG4R6HNPRFH6NQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615998511 (294h7m45.386639836s) [noise=true , address=77.90.31.16:4733 , piece_id=3GQ7NEWE6WZJMQUA3GG3GSZYXEFNOAUEVWHOEH4TSNONYXUT43GQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616193414 (294h7m45.386834749s) [noise=true , address=77.90.31.19:4842 , piece_id=R4FTRBZ3PETCA6QGU6RPUEDZZBWXF7YVH2COEW4WH3YWDLU3VJMA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616449093 (294h7m45.387090429s) [noise=true , address=77.90.31.11:4554 , piece_id=4QRZ5MPQ7HTPRIVRL3O3H4OV7CQA7ESYT5RCPI3GTHXJOHOSAOVA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616635840 (294h7m45.387277175s) [noise=true , address=45.91.116.5:4298 , piece_id=TVTUGWHFTXB32GBY4LYXT3YDB5LIFXEGA7VO5BHFA24Y6EMDTPHQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616818790 (294h7m45.387460125s) [noise=true , address=104.36.82.13:4185 , piece_id=LTQS2CUZ5NRJCBOPEXDKUQ2IJGOIW3JJAPHWTCW7WUMQZPIWRXRA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:12.265123988 (294h7m47.035765313s) [upload=235227]: Next returned nil segment Jul 5 15:18:12.265138164 (294h7m47.035779499s) [upload=235227]: Waiting for error group managing segments... Jul 5 15:18:12.550465694 (294h7m47.321107019s) [noise=true , address=104.36.82.8:4970 , piece_id=2YJN7VKTTTCMAWQW2MNR65DAJPFMHNIBIW5NWYNG4R6HNPRFH6NQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.554315629 (294h7m47.324956954s) [noise=true , address=104.36.82.12:4149 , piece_id=YBTLQEWDMFK6JIFTYR3VJDNMXTS3NMWWHX2MPP53BM2UBDI3UUEA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.603452512 (294h7m47.374093847s) [noise=true , address=104.36.82.3:4767 , piece_id=OGHG4G5ZVC7GVY4XI6DO4KINXF7THHL6IHGDB6DAPDNUDOUFSC4A, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.639796643 (294h7m47.410437978s) [noise=true , address=104.36.82.9:4006 , piece_id=47FME2B5NS6WMOHC2ZGE2FZVSPEQUXKYT6E7AOSANAE5XZVDH5VQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.639894960 (294h7m47.410536285s) [noise=true , address=104.36.82.15:4259 , piece_id=LQKQ5MGKXHTPITBR2QAOQ5CX76GEVJGHJT3ZD3G4PFNN5IBTCJBQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.647222659 (294h7m47.417863994s) [noise=true , address=104.36.82.16:4295 , piece_id=VEL6CAUOJKMRN7A7M66ZPDLCMJVD555LZPHM53A24MDQJQOW7ECQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.665424010 (294h7m47.436065346s) [noise=true , address=104.36.82.13:4185 , piece_id=LTQS2CUZ5NRJCBOPEXDKUQ2IJGOIW3JJAPHWTCW7WUMQZPIWRXRA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.702209574 (294h7m47.47285091s) [noise=true , address=45.91.116.14:4191 , piece_id=WH2BCI36GGFF4QUD4XDYBBDMUR2R2FFVYKFTZD3ZWXPJCFJ3D7HQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.722073145 (294h7m47.49271448s) [noise=true , address=45.91.116.5:4298 , piece_id=TVTUGWHFTXB32GBY4LYXT3YDB5LIFXEGA7VO5BHFA24Y6EMDTPHQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.788621529 (294h7m47.559262864s) [noise=true , address=45.91.116.4:4937 , piece_id=YMKIHJET2EN3QHEK665ACZNQNHJ3UGE4H5T4DIHWJUHJQBYO7B7Q, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:36.752254721 (294h8m11.522896046s) [noise=true , address=77.90.31.19:4842 , piece_id=R4FTRBZ3PETCA6QGU6RPUEDZZBWXF7YVH2COEW4WH3YWDLU3VJMA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:37.783437003 (294h8m12.554078338s) [noise=true , address=77.90.31.17:4771 , piece_id=V7M65ZHCGXPD2GVJL6LTB4NP3QN6QLSIBM7U34QAYPAIKMN6EBPA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:39.733398290 (294h8m14.504039626s) [noise=true , address=77.90.31.18:4805 , piece_id=I23F6KYUQ7TJI4ZLEANACVNYJDMPKKMMACIZAIA6PC5S3CKQPV2Q, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Jul 5 15:18:39.733464387 (294h8m14.504105722s) [noise=true , address=77.90.31.21:4877 , piece_id=OJBIFN2TJPQ45LQLYCMPKV5UZM6Y2MXNOBT3TSGX26U4SZAL5Q7A, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: expected piece hash; context deadline exceeded; EOF Jul 5 15:18:39.733542065 (294h8m14.50418339s) [noise=true , address=77.90.31.11:4554 , piece_id=4QRZ5MPQ7HTPRIVRL3O3H4OV7CQA7ESYT5RCPI3GTHXJOHOSAOVA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection Jul 5 15:18:39.733547496 (294h8m14.504188831s) [noise=true , address=77.90.31.16:4733 , piece_id=3GQ7NEWE6WZJMQUA3GG3GSZYXEFNOAUEVWHOEH4TSNONYXUT43GQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection; write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection Jul 5 15:18:39.733583284 (294h8m14.504224619s) [seg_pos={0 0}, upload=235227]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 5 15:18:39.974321220 (294h8m14.744962555s) [upload=235227]: Done uploading segments... Jul 5 15:18:39.974353041 (294h8m14.744994376s) [upload=235227]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.669928747s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 5.635347621s FAIL (2025-07-05 15:42:41 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== JYE6FC5T5QIBUQIILGDHNGXAUDBUIAOSKYFMPBCWPHP2SDJ3A, seg_pos={0 0}, upload=235214]: Done uploading piece. err: <nil> Jul 5 15:17:12.446386719 (294h6m47.217028064s) [seg_pos={0 0}, upload=235214]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 5 15:17:12.446476771 (294h6m47.217118106s) [upload=235214]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 5 15:17:12.446483043 (294h6m47.217124378s) [upload=235214]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 5 15:17:12.905279085 (294h6m47.67592042s) [upload=235214]: Done uploading segments... Jul 5 15:17:12.905303211 (294h6m47.675944546s) [upload=235214]: Upload finished. err: <nil> Jul 5 15:18:10.381454266 (294h7m45.152095611s) [upload=235227]: Starting upload Jul 5 15:18:10.381524069 (294h7m45.152165414s) [upload=235227]: Done starting upload Jul 5 15:18:10.381621115 (294h7m45.15226245s) [upload=235227]: Uploading segments... Jul 5 15:18:10.381740133 (294h7m45.152381468s) [upload=235227]: Got next segment. Inline: false Jul 5 15:18:10.381776342 (294h7m45.152417677s) [upload=235227]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 5 15:18:10.381781973 (294h7m45.152423308s) [upload=235227]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 5 15:18:10.613904835 (294h7m45.38454617s) [seg_pos={0 0}, upload=235227]: Begin upload segment... Jul 5 15:18:10.613947397 (294h7m45.384588732s) [seg_pos={0 0}, upload=235227]: Done begin upload segment. Jul 5 15:18:10.613968548 (294h7m45.384609883s) [noise=true , address=104.36.82.15:4259 , piece_id=LQKQ5MGKXHTPITBR2QAOQ5CX76GEVJGHJT3ZD3G4PFNN5IBTCJBQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.613995038 (294h7m45.384636373s) [noise=true , address=104.36.82.16:4295 , piece_id=VEL6CAUOJKMRN7A7M66ZPDLCMJVD555LZPHM53A24MDQJQOW7ECQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614303969 (294h7m45.384945304s) [noise=true , address=104.36.82.9:4006 , piece_id=47FME2B5NS6WMOHC2ZGE2FZVSPEQUXKYT6E7AOSANAE5XZVDH5VQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614367721 (294h7m45.385009056s) [noise=true , address=45.91.116.4:4937 , piece_id=YMKIHJET2EN3QHEK665ACZNQNHJ3UGE4H5T4DIHWJUHJQBYO7B7Q, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614619272 (294h7m45.385260607s) [noise=true , address=77.90.31.17:4771 , piece_id=V7M65ZHCGXPD2GVJL6LTB4NP3QN6QLSIBM7U34QAYPAIKMN6EBPA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614637758 (294h7m45.385279083s) [noise=true , address=45.91.116.14:4191 , piece_id=WH2BCI36GGFF4QUD4XDYBBDMUR2R2FFVYKFTZD3ZWXPJCFJ3D7HQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614890251 (294h7m45.385531576s) [noise=true , address=104.36.82.3:4767 , piece_id=OGHG4G5ZVC7GVY4XI6DO4KINXF7THHL6IHGDB6DAPDNUDOUFSC4A, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615197719 (294h7m45.385839054s) [noise=true , address=104.36.82.12:4149 , piece_id=YBTLQEWDMFK6JIFTYR3VJDNMXTS3NMWWHX2MPP53BM2UBDI3UUEA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615413552 (294h7m45.386054887s) [noise=true , address=77.90.31.18:4805 , piece_id=I23F6KYUQ7TJI4ZLEANACVNYJDMPKKMMACIZAIA6PC5S3CKQPV2Q, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615608124 (294h7m45.386249459s) [noise=true , address=77.90.31.21:4877 , piece_id=OJBIFN2TJPQ45LQLYCMPKV5UZM6Y2MXNOBT3TSGX26U4SZAL5Q7A, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615803147 (294h7m45.386444472s) [noise=true , address=104.36.82.8:4970 , piece_id=2YJN7VKTTTCMAWQW2MNR65DAJPFMHNIBIW5NWYNG4R6HNPRFH6NQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615998511 (294h7m45.386639836s) [noise=true , address=77.90.31.16:4733 , piece_id=3GQ7NEWE6WZJMQUA3GG3GSZYXEFNOAUEVWHOEH4TSNONYXUT43GQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616193414 (294h7m45.386834749s) [noise=true , address=77.90.31.19:4842 , piece_id=R4FTRBZ3PETCA6QGU6RPUEDZZBWXF7YVH2COEW4WH3YWDLU3VJMA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616449093 (294h7m45.387090429s) [noise=true , address=77.90.31.11:4554 , piece_id=4QRZ5MPQ7HTPRIVRL3O3H4OV7CQA7ESYT5RCPI3GTHXJOHOSAOVA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616635840 (294h7m45.387277175s) [noise=true , address=45.91.116.5:4298 , piece_id=TVTUGWHFTXB32GBY4LYXT3YDB5LIFXEGA7VO5BHFA24Y6EMDTPHQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616818790 (294h7m45.387460125s) [noise=true , address=104.36.82.13:4185 , piece_id=LTQS2CUZ5NRJCBOPEXDKUQ2IJGOIW3JJAPHWTCW7WUMQZPIWRXRA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:12.265123988 (294h7m47.035765313s) [upload=235227]: Next returned nil segment Jul 5 15:18:12.265138164 (294h7m47.035779499s) [upload=235227]: Waiting for error group managing segments... Jul 5 15:18:12.550465694 (294h7m47.321107019s) [noise=true , address=104.36.82.8:4970 , piece_id=2YJN7VKTTTCMAWQW2MNR65DAJPFMHNIBIW5NWYNG4R6HNPRFH6NQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.554315629 (294h7m47.324956954s) [noise=true , address=104.36.82.12:4149 , piece_id=YBTLQEWDMFK6JIFTYR3VJDNMXTS3NMWWHX2MPP53BM2UBDI3UUEA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.603452512 (294h7m47.374093847s) [noise=true , address=104.36.82.3:4767 , piece_id=OGHG4G5ZVC7GVY4XI6DO4KINXF7THHL6IHGDB6DAPDNUDOUFSC4A, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.639796643 (294h7m47.410437978s) [noise=true , address=104.36.82.9:4006 , piece_id=47FME2B5NS6WMOHC2ZGE2FZVSPEQUXKYT6E7AOSANAE5XZVDH5VQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.639894960 (294h7m47.410536285s) [noise=true , address=104.36.82.15:4259 , piece_id=LQKQ5MGKXHTPITBR2QAOQ5CX76GEVJGHJT3ZD3G4PFNN5IBTCJBQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.647222659 (294h7m47.417863994s) [noise=true , address=104.36.82.16:4295 , piece_id=VEL6CAUOJKMRN7A7M66ZPDLCMJVD555LZPHM53A24MDQJQOW7ECQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.665424010 (294h7m47.436065346s) [noise=true , address=104.36.82.13:4185 , piece_id=LTQS2CUZ5NRJCBOPEXDKUQ2IJGOIW3JJAPHWTCW7WUMQZPIWRXRA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.702209574 (294h7m47.47285091s) [noise=true , address=45.91.116.14:4191 , piece_id=WH2BCI36GGFF4QUD4XDYBBDMUR2R2FFVYKFTZD3ZWXPJCFJ3D7HQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.722073145 (294h7m47.49271448s) [noise=true , address=45.91.116.5:4298 , piece_id=TVTUGWHFTXB32GBY4LYXT3YDB5LIFXEGA7VO5BHFA24Y6EMDTPHQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.788621529 (294h7m47.559262864s) [noise=true , address=45.91.116.4:4937 , piece_id=YMKIHJET2EN3QHEK665ACZNQNHJ3UGE4H5T4DIHWJUHJQBYO7B7Q, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:36.752254721 (294h8m11.522896046s) [noise=true , address=77.90.31.19:4842 , piece_id=R4FTRBZ3PETCA6QGU6RPUEDZZBWXF7YVH2COEW4WH3YWDLU3VJMA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:37.783437003 (294h8m12.554078338s) [noise=true , address=77.90.31.17:4771 , piece_id=V7M65ZHCGXPD2GVJL6LTB4NP3QN6QLSIBM7U34QAYPAIKMN6EBPA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:39.733398290 (294h8m14.504039626s) [noise=true , address=77.90.31.18:4805 , piece_id=I23F6KYUQ7TJI4ZLEANACVNYJDMPKKMMACIZAIA6PC5S3CKQPV2Q, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Jul 5 15:18:39.733464387 (294h8m14.504105722s) [noise=true , address=77.90.31.21:4877 , piece_id=OJBIFN2TJPQ45LQLYCMPKV5UZM6Y2MXNOBT3TSGX26U4SZAL5Q7A, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: expected piece hash; context deadline exceeded; EOF Jul 5 15:18:39.733542065 (294h8m14.50418339s) [noise=true , address=77.90.31.11:4554 , piece_id=4QRZ5MPQ7HTPRIVRL3O3H4OV7CQA7ESYT5RCPI3GTHXJOHOSAOVA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection Jul 5 15:18:39.733547496 (294h8m14.504188831s) [noise=true , address=77.90.31.16:4733 , piece_id=3GQ7NEWE6WZJMQUA3GG3GSZYXEFNOAUEVWHOEH4TSNONYXUT43GQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection; write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection Jul 5 15:18:39.733583284 (294h8m14.504224619s) [seg_pos={0 0}, upload=235227]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 5 15:18:39.974321220 (294h8m14.744962555s) [upload=235227]: Done uploading segments... Jul 5 15:18:39.974353041 (294h8m14.744994376s) [upload=235227]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.635388588s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 5.655894045s FAIL (2025-07-05 15:42:21 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== JYE6FC5T5QIBUQIILGDHNGXAUDBUIAOSKYFMPBCWPHP2SDJ3A, seg_pos={0 0}, upload=235214]: Done uploading piece. err: <nil> Jul 5 15:17:12.446386719 (294h6m47.217028064s) [seg_pos={0 0}, upload=235214]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 5 15:17:12.446476771 (294h6m47.217118106s) [upload=235214]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 5 15:17:12.446483043 (294h6m47.217124378s) [upload=235214]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 5 15:17:12.905279085 (294h6m47.67592042s) [upload=235214]: Done uploading segments... Jul 5 15:17:12.905303211 (294h6m47.675944546s) [upload=235214]: Upload finished. err: <nil> Jul 5 15:18:10.381454266 (294h7m45.152095611s) [upload=235227]: Starting upload Jul 5 15:18:10.381524069 (294h7m45.152165414s) [upload=235227]: Done starting upload Jul 5 15:18:10.381621115 (294h7m45.15226245s) [upload=235227]: Uploading segments... Jul 5 15:18:10.381740133 (294h7m45.152381468s) [upload=235227]: Got next segment. Inline: false Jul 5 15:18:10.381776342 (294h7m45.152417677s) [upload=235227]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 5 15:18:10.381781973 (294h7m45.152423308s) [upload=235227]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 5 15:18:10.613904835 (294h7m45.38454617s) [seg_pos={0 0}, upload=235227]: Begin upload segment... Jul 5 15:18:10.613947397 (294h7m45.384588732s) [seg_pos={0 0}, upload=235227]: Done begin upload segment. Jul 5 15:18:10.613968548 (294h7m45.384609883s) [noise=true , address=104.36.82.15:4259 , piece_id=LQKQ5MGKXHTPITBR2QAOQ5CX76GEVJGHJT3ZD3G4PFNN5IBTCJBQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.613995038 (294h7m45.384636373s) [noise=true , address=104.36.82.16:4295 , piece_id=VEL6CAUOJKMRN7A7M66ZPDLCMJVD555LZPHM53A24MDQJQOW7ECQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614303969 (294h7m45.384945304s) [noise=true , address=104.36.82.9:4006 , piece_id=47FME2B5NS6WMOHC2ZGE2FZVSPEQUXKYT6E7AOSANAE5XZVDH5VQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614367721 (294h7m45.385009056s) [noise=true , address=45.91.116.4:4937 , piece_id=YMKIHJET2EN3QHEK665ACZNQNHJ3UGE4H5T4DIHWJUHJQBYO7B7Q, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614619272 (294h7m45.385260607s) [noise=true , address=77.90.31.17:4771 , piece_id=V7M65ZHCGXPD2GVJL6LTB4NP3QN6QLSIBM7U34QAYPAIKMN6EBPA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614637758 (294h7m45.385279083s) [noise=true , address=45.91.116.14:4191 , piece_id=WH2BCI36GGFF4QUD4XDYBBDMUR2R2FFVYKFTZD3ZWXPJCFJ3D7HQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614890251 (294h7m45.385531576s) [noise=true , address=104.36.82.3:4767 , piece_id=OGHG4G5ZVC7GVY4XI6DO4KINXF7THHL6IHGDB6DAPDNUDOUFSC4A, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615197719 (294h7m45.385839054s) [noise=true , address=104.36.82.12:4149 , piece_id=YBTLQEWDMFK6JIFTYR3VJDNMXTS3NMWWHX2MPP53BM2UBDI3UUEA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615413552 (294h7m45.386054887s) [noise=true , address=77.90.31.18:4805 , piece_id=I23F6KYUQ7TJI4ZLEANACVNYJDMPKKMMACIZAIA6PC5S3CKQPV2Q, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615608124 (294h7m45.386249459s) [noise=true , address=77.90.31.21:4877 , piece_id=OJBIFN2TJPQ45LQLYCMPKV5UZM6Y2MXNOBT3TSGX26U4SZAL5Q7A, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615803147 (294h7m45.386444472s) [noise=true , address=104.36.82.8:4970 , piece_id=2YJN7VKTTTCMAWQW2MNR65DAJPFMHNIBIW5NWYNG4R6HNPRFH6NQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615998511 (294h7m45.386639836s) [noise=true , address=77.90.31.16:4733 , piece_id=3GQ7NEWE6WZJMQUA3GG3GSZYXEFNOAUEVWHOEH4TSNONYXUT43GQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616193414 (294h7m45.386834749s) [noise=true , address=77.90.31.19:4842 , piece_id=R4FTRBZ3PETCA6QGU6RPUEDZZBWXF7YVH2COEW4WH3YWDLU3VJMA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616449093 (294h7m45.387090429s) [noise=true , address=77.90.31.11:4554 , piece_id=4QRZ5MPQ7HTPRIVRL3O3H4OV7CQA7ESYT5RCPI3GTHXJOHOSAOVA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616635840 (294h7m45.387277175s) [noise=true , address=45.91.116.5:4298 , piece_id=TVTUGWHFTXB32GBY4LYXT3YDB5LIFXEGA7VO5BHFA24Y6EMDTPHQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616818790 (294h7m45.387460125s) [noise=true , address=104.36.82.13:4185 , piece_id=LTQS2CUZ5NRJCBOPEXDKUQ2IJGOIW3JJAPHWTCW7WUMQZPIWRXRA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:12.265123988 (294h7m47.035765313s) [upload=235227]: Next returned nil segment Jul 5 15:18:12.265138164 (294h7m47.035779499s) [upload=235227]: Waiting for error group managing segments... Jul 5 15:18:12.550465694 (294h7m47.321107019s) [noise=true , address=104.36.82.8:4970 , piece_id=2YJN7VKTTTCMAWQW2MNR65DAJPFMHNIBIW5NWYNG4R6HNPRFH6NQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.554315629 (294h7m47.324956954s) [noise=true , address=104.36.82.12:4149 , piece_id=YBTLQEWDMFK6JIFTYR3VJDNMXTS3NMWWHX2MPP53BM2UBDI3UUEA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.603452512 (294h7m47.374093847s) [noise=true , address=104.36.82.3:4767 , piece_id=OGHG4G5ZVC7GVY4XI6DO4KINXF7THHL6IHGDB6DAPDNUDOUFSC4A, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.639796643 (294h7m47.410437978s) [noise=true , address=104.36.82.9:4006 , piece_id=47FME2B5NS6WMOHC2ZGE2FZVSPEQUXKYT6E7AOSANAE5XZVDH5VQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.639894960 (294h7m47.410536285s) [noise=true , address=104.36.82.15:4259 , piece_id=LQKQ5MGKXHTPITBR2QAOQ5CX76GEVJGHJT3ZD3G4PFNN5IBTCJBQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.647222659 (294h7m47.417863994s) [noise=true , address=104.36.82.16:4295 , piece_id=VEL6CAUOJKMRN7A7M66ZPDLCMJVD555LZPHM53A24MDQJQOW7ECQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.665424010 (294h7m47.436065346s) [noise=true , address=104.36.82.13:4185 , piece_id=LTQS2CUZ5NRJCBOPEXDKUQ2IJGOIW3JJAPHWTCW7WUMQZPIWRXRA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.702209574 (294h7m47.47285091s) [noise=true , address=45.91.116.14:4191 , piece_id=WH2BCI36GGFF4QUD4XDYBBDMUR2R2FFVYKFTZD3ZWXPJCFJ3D7HQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.722073145 (294h7m47.49271448s) [noise=true , address=45.91.116.5:4298 , piece_id=TVTUGWHFTXB32GBY4LYXT3YDB5LIFXEGA7VO5BHFA24Y6EMDTPHQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.788621529 (294h7m47.559262864s) [noise=true , address=45.91.116.4:4937 , piece_id=YMKIHJET2EN3QHEK665ACZNQNHJ3UGE4H5T4DIHWJUHJQBYO7B7Q, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:36.752254721 (294h8m11.522896046s) [noise=true , address=77.90.31.19:4842 , piece_id=R4FTRBZ3PETCA6QGU6RPUEDZZBWXF7YVH2COEW4WH3YWDLU3VJMA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:37.783437003 (294h8m12.554078338s) [noise=true , address=77.90.31.17:4771 , piece_id=V7M65ZHCGXPD2GVJL6LTB4NP3QN6QLSIBM7U34QAYPAIKMN6EBPA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:39.733398290 (294h8m14.504039626s) [noise=true , address=77.90.31.18:4805 , piece_id=I23F6KYUQ7TJI4ZLEANACVNYJDMPKKMMACIZAIA6PC5S3CKQPV2Q, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Jul 5 15:18:39.733464387 (294h8m14.504105722s) [noise=true , address=77.90.31.21:4877 , piece_id=OJBIFN2TJPQ45LQLYCMPKV5UZM6Y2MXNOBT3TSGX26U4SZAL5Q7A, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: expected piece hash; context deadline exceeded; EOF Jul 5 15:18:39.733542065 (294h8m14.50418339s) [noise=true , address=77.90.31.11:4554 , piece_id=4QRZ5MPQ7HTPRIVRL3O3H4OV7CQA7ESYT5RCPI3GTHXJOHOSAOVA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection Jul 5 15:18:39.733547496 (294h8m14.504188831s) [noise=true , address=77.90.31.16:4733 , piece_id=3GQ7NEWE6WZJMQUA3GG3GSZYXEFNOAUEVWHOEH4TSNONYXUT43GQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection; write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection Jul 5 15:18:39.733583284 (294h8m14.504224619s) [seg_pos={0 0}, upload=235227]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 5 15:18:39.974321220 (294h8m14.744962555s) [upload=235227]: Done uploading segments... Jul 5 15:18:39.974353041 (294h8m14.744994376s) [upload=235227]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.655923051s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 5.625001583s FAIL (2025-07-05 15:41:09 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== JYE6FC5T5QIBUQIILGDHNGXAUDBUIAOSKYFMPBCWPHP2SDJ3A, seg_pos={0 0}, upload=235214]: Done uploading piece. err: <nil> Jul 5 15:17:12.446386719 (294h6m47.217028064s) [seg_pos={0 0}, upload=235214]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 5 15:17:12.446476771 (294h6m47.217118106s) [upload=235214]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 5 15:17:12.446483043 (294h6m47.217124378s) [upload=235214]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 5 15:17:12.905279085 (294h6m47.67592042s) [upload=235214]: Done uploading segments... Jul 5 15:17:12.905303211 (294h6m47.675944546s) [upload=235214]: Upload finished. err: <nil> Jul 5 15:18:10.381454266 (294h7m45.152095611s) [upload=235227]: Starting upload Jul 5 15:18:10.381524069 (294h7m45.152165414s) [upload=235227]: Done starting upload Jul 5 15:18:10.381621115 (294h7m45.15226245s) [upload=235227]: Uploading segments... Jul 5 15:18:10.381740133 (294h7m45.152381468s) [upload=235227]: Got next segment. Inline: false Jul 5 15:18:10.381776342 (294h7m45.152417677s) [upload=235227]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 5 15:18:10.381781973 (294h7m45.152423308s) [upload=235227]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 5 15:18:10.613904835 (294h7m45.38454617s) [seg_pos={0 0}, upload=235227]: Begin upload segment... Jul 5 15:18:10.613947397 (294h7m45.384588732s) [seg_pos={0 0}, upload=235227]: Done begin upload segment. Jul 5 15:18:10.613968548 (294h7m45.384609883s) [noise=true , address=104.36.82.15:4259 , piece_id=LQKQ5MGKXHTPITBR2QAOQ5CX76GEVJGHJT3ZD3G4PFNN5IBTCJBQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.613995038 (294h7m45.384636373s) [noise=true , address=104.36.82.16:4295 , piece_id=VEL6CAUOJKMRN7A7M66ZPDLCMJVD555LZPHM53A24MDQJQOW7ECQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614303969 (294h7m45.384945304s) [noise=true , address=104.36.82.9:4006 , piece_id=47FME2B5NS6WMOHC2ZGE2FZVSPEQUXKYT6E7AOSANAE5XZVDH5VQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614367721 (294h7m45.385009056s) [noise=true , address=45.91.116.4:4937 , piece_id=YMKIHJET2EN3QHEK665ACZNQNHJ3UGE4H5T4DIHWJUHJQBYO7B7Q, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614619272 (294h7m45.385260607s) [noise=true , address=77.90.31.17:4771 , piece_id=V7M65ZHCGXPD2GVJL6LTB4NP3QN6QLSIBM7U34QAYPAIKMN6EBPA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614637758 (294h7m45.385279083s) [noise=true , address=45.91.116.14:4191 , piece_id=WH2BCI36GGFF4QUD4XDYBBDMUR2R2FFVYKFTZD3ZWXPJCFJ3D7HQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614890251 (294h7m45.385531576s) [noise=true , address=104.36.82.3:4767 , piece_id=OGHG4G5ZVC7GVY4XI6DO4KINXF7THHL6IHGDB6DAPDNUDOUFSC4A, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615197719 (294h7m45.385839054s) [noise=true , address=104.36.82.12:4149 , piece_id=YBTLQEWDMFK6JIFTYR3VJDNMXTS3NMWWHX2MPP53BM2UBDI3UUEA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615413552 (294h7m45.386054887s) [noise=true , address=77.90.31.18:4805 , piece_id=I23F6KYUQ7TJI4ZLEANACVNYJDMPKKMMACIZAIA6PC5S3CKQPV2Q, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615608124 (294h7m45.386249459s) [noise=true , address=77.90.31.21:4877 , piece_id=OJBIFN2TJPQ45LQLYCMPKV5UZM6Y2MXNOBT3TSGX26U4SZAL5Q7A, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615803147 (294h7m45.386444472s) [noise=true , address=104.36.82.8:4970 , piece_id=2YJN7VKTTTCMAWQW2MNR65DAJPFMHNIBIW5NWYNG4R6HNPRFH6NQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615998511 (294h7m45.386639836s) [noise=true , address=77.90.31.16:4733 , piece_id=3GQ7NEWE6WZJMQUA3GG3GSZYXEFNOAUEVWHOEH4TSNONYXUT43GQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616193414 (294h7m45.386834749s) [noise=true , address=77.90.31.19:4842 , piece_id=R4FTRBZ3PETCA6QGU6RPUEDZZBWXF7YVH2COEW4WH3YWDLU3VJMA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616449093 (294h7m45.387090429s) [noise=true , address=77.90.31.11:4554 , piece_id=4QRZ5MPQ7HTPRIVRL3O3H4OV7CQA7ESYT5RCPI3GTHXJOHOSAOVA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616635840 (294h7m45.387277175s) [noise=true , address=45.91.116.5:4298 , piece_id=TVTUGWHFTXB32GBY4LYXT3YDB5LIFXEGA7VO5BHFA24Y6EMDTPHQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616818790 (294h7m45.387460125s) [noise=true , address=104.36.82.13:4185 , piece_id=LTQS2CUZ5NRJCBOPEXDKUQ2IJGOIW3JJAPHWTCW7WUMQZPIWRXRA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:12.265123988 (294h7m47.035765313s) [upload=235227]: Next returned nil segment Jul 5 15:18:12.265138164 (294h7m47.035779499s) [upload=235227]: Waiting for error group managing segments... Jul 5 15:18:12.550465694 (294h7m47.321107019s) [noise=true , address=104.36.82.8:4970 , piece_id=2YJN7VKTTTCMAWQW2MNR65DAJPFMHNIBIW5NWYNG4R6HNPRFH6NQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.554315629 (294h7m47.324956954s) [noise=true , address=104.36.82.12:4149 , piece_id=YBTLQEWDMFK6JIFTYR3VJDNMXTS3NMWWHX2MPP53BM2UBDI3UUEA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.603452512 (294h7m47.374093847s) [noise=true , address=104.36.82.3:4767 , piece_id=OGHG4G5ZVC7GVY4XI6DO4KINXF7THHL6IHGDB6DAPDNUDOUFSC4A, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.639796643 (294h7m47.410437978s) [noise=true , address=104.36.82.9:4006 , piece_id=47FME2B5NS6WMOHC2ZGE2FZVSPEQUXKYT6E7AOSANAE5XZVDH5VQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.639894960 (294h7m47.410536285s) [noise=true , address=104.36.82.15:4259 , piece_id=LQKQ5MGKXHTPITBR2QAOQ5CX76GEVJGHJT3ZD3G4PFNN5IBTCJBQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.647222659 (294h7m47.417863994s) [noise=true , address=104.36.82.16:4295 , piece_id=VEL6CAUOJKMRN7A7M66ZPDLCMJVD555LZPHM53A24MDQJQOW7ECQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.665424010 (294h7m47.436065346s) [noise=true , address=104.36.82.13:4185 , piece_id=LTQS2CUZ5NRJCBOPEXDKUQ2IJGOIW3JJAPHWTCW7WUMQZPIWRXRA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.702209574 (294h7m47.47285091s) [noise=true , address=45.91.116.14:4191 , piece_id=WH2BCI36GGFF4QUD4XDYBBDMUR2R2FFVYKFTZD3ZWXPJCFJ3D7HQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.722073145 (294h7m47.49271448s) [noise=true , address=45.91.116.5:4298 , piece_id=TVTUGWHFTXB32GBY4LYXT3YDB5LIFXEGA7VO5BHFA24Y6EMDTPHQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.788621529 (294h7m47.559262864s) [noise=true , address=45.91.116.4:4937 , piece_id=YMKIHJET2EN3QHEK665ACZNQNHJ3UGE4H5T4DIHWJUHJQBYO7B7Q, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:36.752254721 (294h8m11.522896046s) [noise=true , address=77.90.31.19:4842 , piece_id=R4FTRBZ3PETCA6QGU6RPUEDZZBWXF7YVH2COEW4WH3YWDLU3VJMA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:37.783437003 (294h8m12.554078338s) [noise=true , address=77.90.31.17:4771 , piece_id=V7M65ZHCGXPD2GVJL6LTB4NP3QN6QLSIBM7U34QAYPAIKMN6EBPA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:39.733398290 (294h8m14.504039626s) [noise=true , address=77.90.31.18:4805 , piece_id=I23F6KYUQ7TJI4ZLEANACVNYJDMPKKMMACIZAIA6PC5S3CKQPV2Q, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Jul 5 15:18:39.733464387 (294h8m14.504105722s) [noise=true , address=77.90.31.21:4877 , piece_id=OJBIFN2TJPQ45LQLYCMPKV5UZM6Y2MXNOBT3TSGX26U4SZAL5Q7A, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: expected piece hash; context deadline exceeded; EOF Jul 5 15:18:39.733542065 (294h8m14.50418339s) [noise=true , address=77.90.31.11:4554 , piece_id=4QRZ5MPQ7HTPRIVRL3O3H4OV7CQA7ESYT5RCPI3GTHXJOHOSAOVA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection Jul 5 15:18:39.733547496 (294h8m14.504188831s) [noise=true , address=77.90.31.16:4733 , piece_id=3GQ7NEWE6WZJMQUA3GG3GSZYXEFNOAUEVWHOEH4TSNONYXUT43GQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection; write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection Jul 5 15:18:39.733583284 (294h8m14.504224619s) [seg_pos={0 0}, upload=235227]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 5 15:18:39.974321220 (294h8m14.744962555s) [upload=235227]: Done uploading segments... Jul 5 15:18:39.974353041 (294h8m14.744994376s) [upload=235227]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.625030117s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 5.677546247s FAIL (2025-07-05 15:40:02 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== JYE6FC5T5QIBUQIILGDHNGXAUDBUIAOSKYFMPBCWPHP2SDJ3A, seg_pos={0 0}, upload=235214]: Done uploading piece. err: <nil> Jul 5 15:17:12.446386719 (294h6m47.217028064s) [seg_pos={0 0}, upload=235214]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 5 15:17:12.446476771 (294h6m47.217118106s) [upload=235214]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 5 15:17:12.446483043 (294h6m47.217124378s) [upload=235214]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 5 15:17:12.905279085 (294h6m47.67592042s) [upload=235214]: Done uploading segments... Jul 5 15:17:12.905303211 (294h6m47.675944546s) [upload=235214]: Upload finished. err: <nil> Jul 5 15:18:10.381454266 (294h7m45.152095611s) [upload=235227]: Starting upload Jul 5 15:18:10.381524069 (294h7m45.152165414s) [upload=235227]: Done starting upload Jul 5 15:18:10.381621115 (294h7m45.15226245s) [upload=235227]: Uploading segments... Jul 5 15:18:10.381740133 (294h7m45.152381468s) [upload=235227]: Got next segment. Inline: false Jul 5 15:18:10.381776342 (294h7m45.152417677s) [upload=235227]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 5 15:18:10.381781973 (294h7m45.152423308s) [upload=235227]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 5 15:18:10.613904835 (294h7m45.38454617s) [seg_pos={0 0}, upload=235227]: Begin upload segment... Jul 5 15:18:10.613947397 (294h7m45.384588732s) [seg_pos={0 0}, upload=235227]: Done begin upload segment. Jul 5 15:18:10.613968548 (294h7m45.384609883s) [noise=true , address=104.36.82.15:4259 , piece_id=LQKQ5MGKXHTPITBR2QAOQ5CX76GEVJGHJT3ZD3G4PFNN5IBTCJBQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.613995038 (294h7m45.384636373s) [noise=true , address=104.36.82.16:4295 , piece_id=VEL6CAUOJKMRN7A7M66ZPDLCMJVD555LZPHM53A24MDQJQOW7ECQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614303969 (294h7m45.384945304s) [noise=true , address=104.36.82.9:4006 , piece_id=47FME2B5NS6WMOHC2ZGE2FZVSPEQUXKYT6E7AOSANAE5XZVDH5VQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614367721 (294h7m45.385009056s) [noise=true , address=45.91.116.4:4937 , piece_id=YMKIHJET2EN3QHEK665ACZNQNHJ3UGE4H5T4DIHWJUHJQBYO7B7Q, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614619272 (294h7m45.385260607s) [noise=true , address=77.90.31.17:4771 , piece_id=V7M65ZHCGXPD2GVJL6LTB4NP3QN6QLSIBM7U34QAYPAIKMN6EBPA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614637758 (294h7m45.385279083s) [noise=true , address=45.91.116.14:4191 , piece_id=WH2BCI36GGFF4QUD4XDYBBDMUR2R2FFVYKFTZD3ZWXPJCFJ3D7HQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614890251 (294h7m45.385531576s) [noise=true , address=104.36.82.3:4767 , piece_id=OGHG4G5ZVC7GVY4XI6DO4KINXF7THHL6IHGDB6DAPDNUDOUFSC4A, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615197719 (294h7m45.385839054s) [noise=true , address=104.36.82.12:4149 , piece_id=YBTLQEWDMFK6JIFTYR3VJDNMXTS3NMWWHX2MPP53BM2UBDI3UUEA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615413552 (294h7m45.386054887s) [noise=true , address=77.90.31.18:4805 , piece_id=I23F6KYUQ7TJI4ZLEANACVNYJDMPKKMMACIZAIA6PC5S3CKQPV2Q, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615608124 (294h7m45.386249459s) [noise=true , address=77.90.31.21:4877 , piece_id=OJBIFN2TJPQ45LQLYCMPKV5UZM6Y2MXNOBT3TSGX26U4SZAL5Q7A, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615803147 (294h7m45.386444472s) [noise=true , address=104.36.82.8:4970 , piece_id=2YJN7VKTTTCMAWQW2MNR65DAJPFMHNIBIW5NWYNG4R6HNPRFH6NQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615998511 (294h7m45.386639836s) [noise=true , address=77.90.31.16:4733 , piece_id=3GQ7NEWE6WZJMQUA3GG3GSZYXEFNOAUEVWHOEH4TSNONYXUT43GQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616193414 (294h7m45.386834749s) [noise=true , address=77.90.31.19:4842 , piece_id=R4FTRBZ3PETCA6QGU6RPUEDZZBWXF7YVH2COEW4WH3YWDLU3VJMA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616449093 (294h7m45.387090429s) [noise=true , address=77.90.31.11:4554 , piece_id=4QRZ5MPQ7HTPRIVRL3O3H4OV7CQA7ESYT5RCPI3GTHXJOHOSAOVA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616635840 (294h7m45.387277175s) [noise=true , address=45.91.116.5:4298 , piece_id=TVTUGWHFTXB32GBY4LYXT3YDB5LIFXEGA7VO5BHFA24Y6EMDTPHQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616818790 (294h7m45.387460125s) [noise=true , address=104.36.82.13:4185 , piece_id=LTQS2CUZ5NRJCBOPEXDKUQ2IJGOIW3JJAPHWTCW7WUMQZPIWRXRA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:12.265123988 (294h7m47.035765313s) [upload=235227]: Next returned nil segment Jul 5 15:18:12.265138164 (294h7m47.035779499s) [upload=235227]: Waiting for error group managing segments... Jul 5 15:18:12.550465694 (294h7m47.321107019s) [noise=true , address=104.36.82.8:4970 , piece_id=2YJN7VKTTTCMAWQW2MNR65DAJPFMHNIBIW5NWYNG4R6HNPRFH6NQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.554315629 (294h7m47.324956954s) [noise=true , address=104.36.82.12:4149 , piece_id=YBTLQEWDMFK6JIFTYR3VJDNMXTS3NMWWHX2MPP53BM2UBDI3UUEA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.603452512 (294h7m47.374093847s) [noise=true , address=104.36.82.3:4767 , piece_id=OGHG4G5ZVC7GVY4XI6DO4KINXF7THHL6IHGDB6DAPDNUDOUFSC4A, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.639796643 (294h7m47.410437978s) [noise=true , address=104.36.82.9:4006 , piece_id=47FME2B5NS6WMOHC2ZGE2FZVSPEQUXKYT6E7AOSANAE5XZVDH5VQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.639894960 (294h7m47.410536285s) [noise=true , address=104.36.82.15:4259 , piece_id=LQKQ5MGKXHTPITBR2QAOQ5CX76GEVJGHJT3ZD3G4PFNN5IBTCJBQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.647222659 (294h7m47.417863994s) [noise=true , address=104.36.82.16:4295 , piece_id=VEL6CAUOJKMRN7A7M66ZPDLCMJVD555LZPHM53A24MDQJQOW7ECQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.665424010 (294h7m47.436065346s) [noise=true , address=104.36.82.13:4185 , piece_id=LTQS2CUZ5NRJCBOPEXDKUQ2IJGOIW3JJAPHWTCW7WUMQZPIWRXRA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.702209574 (294h7m47.47285091s) [noise=true , address=45.91.116.14:4191 , piece_id=WH2BCI36GGFF4QUD4XDYBBDMUR2R2FFVYKFTZD3ZWXPJCFJ3D7HQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.722073145 (294h7m47.49271448s) [noise=true , address=45.91.116.5:4298 , piece_id=TVTUGWHFTXB32GBY4LYXT3YDB5LIFXEGA7VO5BHFA24Y6EMDTPHQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.788621529 (294h7m47.559262864s) [noise=true , address=45.91.116.4:4937 , piece_id=YMKIHJET2EN3QHEK665ACZNQNHJ3UGE4H5T4DIHWJUHJQBYO7B7Q, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:36.752254721 (294h8m11.522896046s) [noise=true , address=77.90.31.19:4842 , piece_id=R4FTRBZ3PETCA6QGU6RPUEDZZBWXF7YVH2COEW4WH3YWDLU3VJMA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:37.783437003 (294h8m12.554078338s) [noise=true , address=77.90.31.17:4771 , piece_id=V7M65ZHCGXPD2GVJL6LTB4NP3QN6QLSIBM7U34QAYPAIKMN6EBPA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:39.733398290 (294h8m14.504039626s) [noise=true , address=77.90.31.18:4805 , piece_id=I23F6KYUQ7TJI4ZLEANACVNYJDMPKKMMACIZAIA6PC5S3CKQPV2Q, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Jul 5 15:18:39.733464387 (294h8m14.504105722s) [noise=true , address=77.90.31.21:4877 , piece_id=OJBIFN2TJPQ45LQLYCMPKV5UZM6Y2MXNOBT3TSGX26U4SZAL5Q7A, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: expected piece hash; context deadline exceeded; EOF Jul 5 15:18:39.733542065 (294h8m14.50418339s) [noise=true , address=77.90.31.11:4554 , piece_id=4QRZ5MPQ7HTPRIVRL3O3H4OV7CQA7ESYT5RCPI3GTHXJOHOSAOVA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection Jul 5 15:18:39.733547496 (294h8m14.504188831s) [noise=true , address=77.90.31.16:4733 , piece_id=3GQ7NEWE6WZJMQUA3GG3GSZYXEFNOAUEVWHOEH4TSNONYXUT43GQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection; write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection Jul 5 15:18:39.733583284 (294h8m14.504224619s) [seg_pos={0 0}, upload=235227]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 5 15:18:39.974321220 (294h8m14.744962555s) [upload=235227]: Done uploading segments... Jul 5 15:18:39.974353041 (294h8m14.744994376s) [upload=235227]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.677576945s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 5.668518247s FAIL (2025-07-05 15:38:53 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== JYE6FC5T5QIBUQIILGDHNGXAUDBUIAOSKYFMPBCWPHP2SDJ3A, seg_pos={0 0}, upload=235214]: Done uploading piece. err: <nil> Jul 5 15:17:12.446386719 (294h6m47.217028064s) [seg_pos={0 0}, upload=235214]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 5 15:17:12.446476771 (294h6m47.217118106s) [upload=235214]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 5 15:17:12.446483043 (294h6m47.217124378s) [upload=235214]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 5 15:17:12.905279085 (294h6m47.67592042s) [upload=235214]: Done uploading segments... Jul 5 15:17:12.905303211 (294h6m47.675944546s) [upload=235214]: Upload finished. err: <nil> Jul 5 15:18:10.381454266 (294h7m45.152095611s) [upload=235227]: Starting upload Jul 5 15:18:10.381524069 (294h7m45.152165414s) [upload=235227]: Done starting upload Jul 5 15:18:10.381621115 (294h7m45.15226245s) [upload=235227]: Uploading segments... Jul 5 15:18:10.381740133 (294h7m45.152381468s) [upload=235227]: Got next segment. Inline: false Jul 5 15:18:10.381776342 (294h7m45.152417677s) [upload=235227]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 5 15:18:10.381781973 (294h7m45.152423308s) [upload=235227]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 5 15:18:10.613904835 (294h7m45.38454617s) [seg_pos={0 0}, upload=235227]: Begin upload segment... Jul 5 15:18:10.613947397 (294h7m45.384588732s) [seg_pos={0 0}, upload=235227]: Done begin upload segment. Jul 5 15:18:10.613968548 (294h7m45.384609883s) [noise=true , address=104.36.82.15:4259 , piece_id=LQKQ5MGKXHTPITBR2QAOQ5CX76GEVJGHJT3ZD3G4PFNN5IBTCJBQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.613995038 (294h7m45.384636373s) [noise=true , address=104.36.82.16:4295 , piece_id=VEL6CAUOJKMRN7A7M66ZPDLCMJVD555LZPHM53A24MDQJQOW7ECQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614303969 (294h7m45.384945304s) [noise=true , address=104.36.82.9:4006 , piece_id=47FME2B5NS6WMOHC2ZGE2FZVSPEQUXKYT6E7AOSANAE5XZVDH5VQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614367721 (294h7m45.385009056s) [noise=true , address=45.91.116.4:4937 , piece_id=YMKIHJET2EN3QHEK665ACZNQNHJ3UGE4H5T4DIHWJUHJQBYO7B7Q, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614619272 (294h7m45.385260607s) [noise=true , address=77.90.31.17:4771 , piece_id=V7M65ZHCGXPD2GVJL6LTB4NP3QN6QLSIBM7U34QAYPAIKMN6EBPA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614637758 (294h7m45.385279083s) [noise=true , address=45.91.116.14:4191 , piece_id=WH2BCI36GGFF4QUD4XDYBBDMUR2R2FFVYKFTZD3ZWXPJCFJ3D7HQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614890251 (294h7m45.385531576s) [noise=true , address=104.36.82.3:4767 , piece_id=OGHG4G5ZVC7GVY4XI6DO4KINXF7THHL6IHGDB6DAPDNUDOUFSC4A, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615197719 (294h7m45.385839054s) [noise=true , address=104.36.82.12:4149 , piece_id=YBTLQEWDMFK6JIFTYR3VJDNMXTS3NMWWHX2MPP53BM2UBDI3UUEA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615413552 (294h7m45.386054887s) [noise=true , address=77.90.31.18:4805 , piece_id=I23F6KYUQ7TJI4ZLEANACVNYJDMPKKMMACIZAIA6PC5S3CKQPV2Q, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615608124 (294h7m45.386249459s) [noise=true , address=77.90.31.21:4877 , piece_id=OJBIFN2TJPQ45LQLYCMPKV5UZM6Y2MXNOBT3TSGX26U4SZAL5Q7A, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615803147 (294h7m45.386444472s) [noise=true , address=104.36.82.8:4970 , piece_id=2YJN7VKTTTCMAWQW2MNR65DAJPFMHNIBIW5NWYNG4R6HNPRFH6NQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615998511 (294h7m45.386639836s) [noise=true , address=77.90.31.16:4733 , piece_id=3GQ7NEWE6WZJMQUA3GG3GSZYXEFNOAUEVWHOEH4TSNONYXUT43GQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616193414 (294h7m45.386834749s) [noise=true , address=77.90.31.19:4842 , piece_id=R4FTRBZ3PETCA6QGU6RPUEDZZBWXF7YVH2COEW4WH3YWDLU3VJMA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616449093 (294h7m45.387090429s) [noise=true , address=77.90.31.11:4554 , piece_id=4QRZ5MPQ7HTPRIVRL3O3H4OV7CQA7ESYT5RCPI3GTHXJOHOSAOVA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616635840 (294h7m45.387277175s) [noise=true , address=45.91.116.5:4298 , piece_id=TVTUGWHFTXB32GBY4LYXT3YDB5LIFXEGA7VO5BHFA24Y6EMDTPHQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616818790 (294h7m45.387460125s) [noise=true , address=104.36.82.13:4185 , piece_id=LTQS2CUZ5NRJCBOPEXDKUQ2IJGOIW3JJAPHWTCW7WUMQZPIWRXRA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:12.265123988 (294h7m47.035765313s) [upload=235227]: Next returned nil segment Jul 5 15:18:12.265138164 (294h7m47.035779499s) [upload=235227]: Waiting for error group managing segments... Jul 5 15:18:12.550465694 (294h7m47.321107019s) [noise=true , address=104.36.82.8:4970 , piece_id=2YJN7VKTTTCMAWQW2MNR65DAJPFMHNIBIW5NWYNG4R6HNPRFH6NQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.554315629 (294h7m47.324956954s) [noise=true , address=104.36.82.12:4149 , piece_id=YBTLQEWDMFK6JIFTYR3VJDNMXTS3NMWWHX2MPP53BM2UBDI3UUEA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.603452512 (294h7m47.374093847s) [noise=true , address=104.36.82.3:4767 , piece_id=OGHG4G5ZVC7GVY4XI6DO4KINXF7THHL6IHGDB6DAPDNUDOUFSC4A, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.639796643 (294h7m47.410437978s) [noise=true , address=104.36.82.9:4006 , piece_id=47FME2B5NS6WMOHC2ZGE2FZVSPEQUXKYT6E7AOSANAE5XZVDH5VQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.639894960 (294h7m47.410536285s) [noise=true , address=104.36.82.15:4259 , piece_id=LQKQ5MGKXHTPITBR2QAOQ5CX76GEVJGHJT3ZD3G4PFNN5IBTCJBQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.647222659 (294h7m47.417863994s) [noise=true , address=104.36.82.16:4295 , piece_id=VEL6CAUOJKMRN7A7M66ZPDLCMJVD555LZPHM53A24MDQJQOW7ECQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.665424010 (294h7m47.436065346s) [noise=true , address=104.36.82.13:4185 , piece_id=LTQS2CUZ5NRJCBOPEXDKUQ2IJGOIW3JJAPHWTCW7WUMQZPIWRXRA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.702209574 (294h7m47.47285091s) [noise=true , address=45.91.116.14:4191 , piece_id=WH2BCI36GGFF4QUD4XDYBBDMUR2R2FFVYKFTZD3ZWXPJCFJ3D7HQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.722073145 (294h7m47.49271448s) [noise=true , address=45.91.116.5:4298 , piece_id=TVTUGWHFTXB32GBY4LYXT3YDB5LIFXEGA7VO5BHFA24Y6EMDTPHQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.788621529 (294h7m47.559262864s) [noise=true , address=45.91.116.4:4937 , piece_id=YMKIHJET2EN3QHEK665ACZNQNHJ3UGE4H5T4DIHWJUHJQBYO7B7Q, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:36.752254721 (294h8m11.522896046s) [noise=true , address=77.90.31.19:4842 , piece_id=R4FTRBZ3PETCA6QGU6RPUEDZZBWXF7YVH2COEW4WH3YWDLU3VJMA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:37.783437003 (294h8m12.554078338s) [noise=true , address=77.90.31.17:4771 , piece_id=V7M65ZHCGXPD2GVJL6LTB4NP3QN6QLSIBM7U34QAYPAIKMN6EBPA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:39.733398290 (294h8m14.504039626s) [noise=true , address=77.90.31.18:4805 , piece_id=I23F6KYUQ7TJI4ZLEANACVNYJDMPKKMMACIZAIA6PC5S3CKQPV2Q, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Jul 5 15:18:39.733464387 (294h8m14.504105722s) [noise=true , address=77.90.31.21:4877 , piece_id=OJBIFN2TJPQ45LQLYCMPKV5UZM6Y2MXNOBT3TSGX26U4SZAL5Q7A, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: expected piece hash; context deadline exceeded; EOF Jul 5 15:18:39.733542065 (294h8m14.50418339s) [noise=true , address=77.90.31.11:4554 , piece_id=4QRZ5MPQ7HTPRIVRL3O3H4OV7CQA7ESYT5RCPI3GTHXJOHOSAOVA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection Jul 5 15:18:39.733547496 (294h8m14.504188831s) [noise=true , address=77.90.31.16:4733 , piece_id=3GQ7NEWE6WZJMQUA3GG3GSZYXEFNOAUEVWHOEH4TSNONYXUT43GQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection; write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection Jul 5 15:18:39.733583284 (294h8m14.504224619s) [seg_pos={0 0}, upload=235227]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 5 15:18:39.974321220 (294h8m14.744962555s) [upload=235227]: Done uploading segments... Jul 5 15:18:39.974353041 (294h8m14.744994376s) [upload=235227]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.668553393s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 5.604979387s FAIL (2025-07-05 15:38:15 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== JYE6FC5T5QIBUQIILGDHNGXAUDBUIAOSKYFMPBCWPHP2SDJ3A, seg_pos={0 0}, upload=235214]: Done uploading piece. err: <nil> Jul 5 15:17:12.446386719 (294h6m47.217028064s) [seg_pos={0 0}, upload=235214]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 5 15:17:12.446476771 (294h6m47.217118106s) [upload=235214]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 5 15:17:12.446483043 (294h6m47.217124378s) [upload=235214]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 5 15:17:12.905279085 (294h6m47.67592042s) [upload=235214]: Done uploading segments... Jul 5 15:17:12.905303211 (294h6m47.675944546s) [upload=235214]: Upload finished. err: <nil> Jul 5 15:18:10.381454266 (294h7m45.152095611s) [upload=235227]: Starting upload Jul 5 15:18:10.381524069 (294h7m45.152165414s) [upload=235227]: Done starting upload Jul 5 15:18:10.381621115 (294h7m45.15226245s) [upload=235227]: Uploading segments... Jul 5 15:18:10.381740133 (294h7m45.152381468s) [upload=235227]: Got next segment. Inline: false Jul 5 15:18:10.381776342 (294h7m45.152417677s) [upload=235227]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 5 15:18:10.381781973 (294h7m45.152423308s) [upload=235227]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 5 15:18:10.613904835 (294h7m45.38454617s) [seg_pos={0 0}, upload=235227]: Begin upload segment... Jul 5 15:18:10.613947397 (294h7m45.384588732s) [seg_pos={0 0}, upload=235227]: Done begin upload segment. Jul 5 15:18:10.613968548 (294h7m45.384609883s) [noise=true , address=104.36.82.15:4259 , piece_id=LQKQ5MGKXHTPITBR2QAOQ5CX76GEVJGHJT3ZD3G4PFNN5IBTCJBQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.613995038 (294h7m45.384636373s) [noise=true , address=104.36.82.16:4295 , piece_id=VEL6CAUOJKMRN7A7M66ZPDLCMJVD555LZPHM53A24MDQJQOW7ECQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614303969 (294h7m45.384945304s) [noise=true , address=104.36.82.9:4006 , piece_id=47FME2B5NS6WMOHC2ZGE2FZVSPEQUXKYT6E7AOSANAE5XZVDH5VQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614367721 (294h7m45.385009056s) [noise=true , address=45.91.116.4:4937 , piece_id=YMKIHJET2EN3QHEK665ACZNQNHJ3UGE4H5T4DIHWJUHJQBYO7B7Q, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614619272 (294h7m45.385260607s) [noise=true , address=77.90.31.17:4771 , piece_id=V7M65ZHCGXPD2GVJL6LTB4NP3QN6QLSIBM7U34QAYPAIKMN6EBPA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614637758 (294h7m45.385279083s) [noise=true , address=45.91.116.14:4191 , piece_id=WH2BCI36GGFF4QUD4XDYBBDMUR2R2FFVYKFTZD3ZWXPJCFJ3D7HQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614890251 (294h7m45.385531576s) [noise=true , address=104.36.82.3:4767 , piece_id=OGHG4G5ZVC7GVY4XI6DO4KINXF7THHL6IHGDB6DAPDNUDOUFSC4A, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615197719 (294h7m45.385839054s) [noise=true , address=104.36.82.12:4149 , piece_id=YBTLQEWDMFK6JIFTYR3VJDNMXTS3NMWWHX2MPP53BM2UBDI3UUEA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615413552 (294h7m45.386054887s) [noise=true , address=77.90.31.18:4805 , piece_id=I23F6KYUQ7TJI4ZLEANACVNYJDMPKKMMACIZAIA6PC5S3CKQPV2Q, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615608124 (294h7m45.386249459s) [noise=true , address=77.90.31.21:4877 , piece_id=OJBIFN2TJPQ45LQLYCMPKV5UZM6Y2MXNOBT3TSGX26U4SZAL5Q7A, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615803147 (294h7m45.386444472s) [noise=true , address=104.36.82.8:4970 , piece_id=2YJN7VKTTTCMAWQW2MNR65DAJPFMHNIBIW5NWYNG4R6HNPRFH6NQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615998511 (294h7m45.386639836s) [noise=true , address=77.90.31.16:4733 , piece_id=3GQ7NEWE6WZJMQUA3GG3GSZYXEFNOAUEVWHOEH4TSNONYXUT43GQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616193414 (294h7m45.386834749s) [noise=true , address=77.90.31.19:4842 , piece_id=R4FTRBZ3PETCA6QGU6RPUEDZZBWXF7YVH2COEW4WH3YWDLU3VJMA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616449093 (294h7m45.387090429s) [noise=true , address=77.90.31.11:4554 , piece_id=4QRZ5MPQ7HTPRIVRL3O3H4OV7CQA7ESYT5RCPI3GTHXJOHOSAOVA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616635840 (294h7m45.387277175s) [noise=true , address=45.91.116.5:4298 , piece_id=TVTUGWHFTXB32GBY4LYXT3YDB5LIFXEGA7VO5BHFA24Y6EMDTPHQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616818790 (294h7m45.387460125s) [noise=true , address=104.36.82.13:4185 , piece_id=LTQS2CUZ5NRJCBOPEXDKUQ2IJGOIW3JJAPHWTCW7WUMQZPIWRXRA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:12.265123988 (294h7m47.035765313s) [upload=235227]: Next returned nil segment Jul 5 15:18:12.265138164 (294h7m47.035779499s) [upload=235227]: Waiting for error group managing segments... Jul 5 15:18:12.550465694 (294h7m47.321107019s) [noise=true , address=104.36.82.8:4970 , piece_id=2YJN7VKTTTCMAWQW2MNR65DAJPFMHNIBIW5NWYNG4R6HNPRFH6NQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.554315629 (294h7m47.324956954s) [noise=true , address=104.36.82.12:4149 , piece_id=YBTLQEWDMFK6JIFTYR3VJDNMXTS3NMWWHX2MPP53BM2UBDI3UUEA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.603452512 (294h7m47.374093847s) [noise=true , address=104.36.82.3:4767 , piece_id=OGHG4G5ZVC7GVY4XI6DO4KINXF7THHL6IHGDB6DAPDNUDOUFSC4A, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.639796643 (294h7m47.410437978s) [noise=true , address=104.36.82.9:4006 , piece_id=47FME2B5NS6WMOHC2ZGE2FZVSPEQUXKYT6E7AOSANAE5XZVDH5VQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.639894960 (294h7m47.410536285s) [noise=true , address=104.36.82.15:4259 , piece_id=LQKQ5MGKXHTPITBR2QAOQ5CX76GEVJGHJT3ZD3G4PFNN5IBTCJBQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.647222659 (294h7m47.417863994s) [noise=true , address=104.36.82.16:4295 , piece_id=VEL6CAUOJKMRN7A7M66ZPDLCMJVD555LZPHM53A24MDQJQOW7ECQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.665424010 (294h7m47.436065346s) [noise=true , address=104.36.82.13:4185 , piece_id=LTQS2CUZ5NRJCBOPEXDKUQ2IJGOIW3JJAPHWTCW7WUMQZPIWRXRA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.702209574 (294h7m47.47285091s) [noise=true , address=45.91.116.14:4191 , piece_id=WH2BCI36GGFF4QUD4XDYBBDMUR2R2FFVYKFTZD3ZWXPJCFJ3D7HQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.722073145 (294h7m47.49271448s) [noise=true , address=45.91.116.5:4298 , piece_id=TVTUGWHFTXB32GBY4LYXT3YDB5LIFXEGA7VO5BHFA24Y6EMDTPHQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.788621529 (294h7m47.559262864s) [noise=true , address=45.91.116.4:4937 , piece_id=YMKIHJET2EN3QHEK665ACZNQNHJ3UGE4H5T4DIHWJUHJQBYO7B7Q, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:36.752254721 (294h8m11.522896046s) [noise=true , address=77.90.31.19:4842 , piece_id=R4FTRBZ3PETCA6QGU6RPUEDZZBWXF7YVH2COEW4WH3YWDLU3VJMA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:37.783437003 (294h8m12.554078338s) [noise=true , address=77.90.31.17:4771 , piece_id=V7M65ZHCGXPD2GVJL6LTB4NP3QN6QLSIBM7U34QAYPAIKMN6EBPA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:39.733398290 (294h8m14.504039626s) [noise=true , address=77.90.31.18:4805 , piece_id=I23F6KYUQ7TJI4ZLEANACVNYJDMPKKMMACIZAIA6PC5S3CKQPV2Q, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Jul 5 15:18:39.733464387 (294h8m14.504105722s) [noise=true , address=77.90.31.21:4877 , piece_id=OJBIFN2TJPQ45LQLYCMPKV5UZM6Y2MXNOBT3TSGX26U4SZAL5Q7A, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: expected piece hash; context deadline exceeded; EOF Jul 5 15:18:39.733542065 (294h8m14.50418339s) [noise=true , address=77.90.31.11:4554 , piece_id=4QRZ5MPQ7HTPRIVRL3O3H4OV7CQA7ESYT5RCPI3GTHXJOHOSAOVA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection Jul 5 15:18:39.733547496 (294h8m14.504188831s) [noise=true , address=77.90.31.16:4733 , piece_id=3GQ7NEWE6WZJMQUA3GG3GSZYXEFNOAUEVWHOEH4TSNONYXUT43GQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection; write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection Jul 5 15:18:39.733583284 (294h8m14.504224619s) [seg_pos={0 0}, upload=235227]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 5 15:18:39.974321220 (294h8m14.744962555s) [upload=235227]: Done uploading segments... Jul 5 15:18:39.974353041 (294h8m14.744994376s) [upload=235227]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.60500215s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 5.966458194s FAIL (2025-07-05 15:37:45 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== JYE6FC5T5QIBUQIILGDHNGXAUDBUIAOSKYFMPBCWPHP2SDJ3A, seg_pos={0 0}, upload=235214]: Done uploading piece. err: <nil> Jul 5 15:17:12.446386719 (294h6m47.217028064s) [seg_pos={0 0}, upload=235214]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 5 15:17:12.446476771 (294h6m47.217118106s) [upload=235214]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 5 15:17:12.446483043 (294h6m47.217124378s) [upload=235214]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 5 15:17:12.905279085 (294h6m47.67592042s) [upload=235214]: Done uploading segments... Jul 5 15:17:12.905303211 (294h6m47.675944546s) [upload=235214]: Upload finished. err: <nil> Jul 5 15:18:10.381454266 (294h7m45.152095611s) [upload=235227]: Starting upload Jul 5 15:18:10.381524069 (294h7m45.152165414s) [upload=235227]: Done starting upload Jul 5 15:18:10.381621115 (294h7m45.15226245s) [upload=235227]: Uploading segments... Jul 5 15:18:10.381740133 (294h7m45.152381468s) [upload=235227]: Got next segment. Inline: false Jul 5 15:18:10.381776342 (294h7m45.152417677s) [upload=235227]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 5 15:18:10.381781973 (294h7m45.152423308s) [upload=235227]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 5 15:18:10.613904835 (294h7m45.38454617s) [seg_pos={0 0}, upload=235227]: Begin upload segment... Jul 5 15:18:10.613947397 (294h7m45.384588732s) [seg_pos={0 0}, upload=235227]: Done begin upload segment. Jul 5 15:18:10.613968548 (294h7m45.384609883s) [noise=true , address=104.36.82.15:4259 , piece_id=LQKQ5MGKXHTPITBR2QAOQ5CX76GEVJGHJT3ZD3G4PFNN5IBTCJBQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.613995038 (294h7m45.384636373s) [noise=true , address=104.36.82.16:4295 , piece_id=VEL6CAUOJKMRN7A7M66ZPDLCMJVD555LZPHM53A24MDQJQOW7ECQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614303969 (294h7m45.384945304s) [noise=true , address=104.36.82.9:4006 , piece_id=47FME2B5NS6WMOHC2ZGE2FZVSPEQUXKYT6E7AOSANAE5XZVDH5VQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614367721 (294h7m45.385009056s) [noise=true , address=45.91.116.4:4937 , piece_id=YMKIHJET2EN3QHEK665ACZNQNHJ3UGE4H5T4DIHWJUHJQBYO7B7Q, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614619272 (294h7m45.385260607s) [noise=true , address=77.90.31.17:4771 , piece_id=V7M65ZHCGXPD2GVJL6LTB4NP3QN6QLSIBM7U34QAYPAIKMN6EBPA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614637758 (294h7m45.385279083s) [noise=true , address=45.91.116.14:4191 , piece_id=WH2BCI36GGFF4QUD4XDYBBDMUR2R2FFVYKFTZD3ZWXPJCFJ3D7HQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614890251 (294h7m45.385531576s) [noise=true , address=104.36.82.3:4767 , piece_id=OGHG4G5ZVC7GVY4XI6DO4KINXF7THHL6IHGDB6DAPDNUDOUFSC4A, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615197719 (294h7m45.385839054s) [noise=true , address=104.36.82.12:4149 , piece_id=YBTLQEWDMFK6JIFTYR3VJDNMXTS3NMWWHX2MPP53BM2UBDI3UUEA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615413552 (294h7m45.386054887s) [noise=true , address=77.90.31.18:4805 , piece_id=I23F6KYUQ7TJI4ZLEANACVNYJDMPKKMMACIZAIA6PC5S3CKQPV2Q, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615608124 (294h7m45.386249459s) [noise=true , address=77.90.31.21:4877 , piece_id=OJBIFN2TJPQ45LQLYCMPKV5UZM6Y2MXNOBT3TSGX26U4SZAL5Q7A, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615803147 (294h7m45.386444472s) [noise=true , address=104.36.82.8:4970 , piece_id=2YJN7VKTTTCMAWQW2MNR65DAJPFMHNIBIW5NWYNG4R6HNPRFH6NQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615998511 (294h7m45.386639836s) [noise=true , address=77.90.31.16:4733 , piece_id=3GQ7NEWE6WZJMQUA3GG3GSZYXEFNOAUEVWHOEH4TSNONYXUT43GQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616193414 (294h7m45.386834749s) [noise=true , address=77.90.31.19:4842 , piece_id=R4FTRBZ3PETCA6QGU6RPUEDZZBWXF7YVH2COEW4WH3YWDLU3VJMA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616449093 (294h7m45.387090429s) [noise=true , address=77.90.31.11:4554 , piece_id=4QRZ5MPQ7HTPRIVRL3O3H4OV7CQA7ESYT5RCPI3GTHXJOHOSAOVA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616635840 (294h7m45.387277175s) [noise=true , address=45.91.116.5:4298 , piece_id=TVTUGWHFTXB32GBY4LYXT3YDB5LIFXEGA7VO5BHFA24Y6EMDTPHQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616818790 (294h7m45.387460125s) [noise=true , address=104.36.82.13:4185 , piece_id=LTQS2CUZ5NRJCBOPEXDKUQ2IJGOIW3JJAPHWTCW7WUMQZPIWRXRA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:12.265123988 (294h7m47.035765313s) [upload=235227]: Next returned nil segment Jul 5 15:18:12.265138164 (294h7m47.035779499s) [upload=235227]: Waiting for error group managing segments... Jul 5 15:18:12.550465694 (294h7m47.321107019s) [noise=true , address=104.36.82.8:4970 , piece_id=2YJN7VKTTTCMAWQW2MNR65DAJPFMHNIBIW5NWYNG4R6HNPRFH6NQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.554315629 (294h7m47.324956954s) [noise=true , address=104.36.82.12:4149 , piece_id=YBTLQEWDMFK6JIFTYR3VJDNMXTS3NMWWHX2MPP53BM2UBDI3UUEA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.603452512 (294h7m47.374093847s) [noise=true , address=104.36.82.3:4767 , piece_id=OGHG4G5ZVC7GVY4XI6DO4KINXF7THHL6IHGDB6DAPDNUDOUFSC4A, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.639796643 (294h7m47.410437978s) [noise=true , address=104.36.82.9:4006 , piece_id=47FME2B5NS6WMOHC2ZGE2FZVSPEQUXKYT6E7AOSANAE5XZVDH5VQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.639894960 (294h7m47.410536285s) [noise=true , address=104.36.82.15:4259 , piece_id=LQKQ5MGKXHTPITBR2QAOQ5CX76GEVJGHJT3ZD3G4PFNN5IBTCJBQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.647222659 (294h7m47.417863994s) [noise=true , address=104.36.82.16:4295 , piece_id=VEL6CAUOJKMRN7A7M66ZPDLCMJVD555LZPHM53A24MDQJQOW7ECQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.665424010 (294h7m47.436065346s) [noise=true , address=104.36.82.13:4185 , piece_id=LTQS2CUZ5NRJCBOPEXDKUQ2IJGOIW3JJAPHWTCW7WUMQZPIWRXRA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.702209574 (294h7m47.47285091s) [noise=true , address=45.91.116.14:4191 , piece_id=WH2BCI36GGFF4QUD4XDYBBDMUR2R2FFVYKFTZD3ZWXPJCFJ3D7HQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.722073145 (294h7m47.49271448s) [noise=true , address=45.91.116.5:4298 , piece_id=TVTUGWHFTXB32GBY4LYXT3YDB5LIFXEGA7VO5BHFA24Y6EMDTPHQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.788621529 (294h7m47.559262864s) [noise=true , address=45.91.116.4:4937 , piece_id=YMKIHJET2EN3QHEK665ACZNQNHJ3UGE4H5T4DIHWJUHJQBYO7B7Q, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:36.752254721 (294h8m11.522896046s) [noise=true , address=77.90.31.19:4842 , piece_id=R4FTRBZ3PETCA6QGU6RPUEDZZBWXF7YVH2COEW4WH3YWDLU3VJMA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:37.783437003 (294h8m12.554078338s) [noise=true , address=77.90.31.17:4771 , piece_id=V7M65ZHCGXPD2GVJL6LTB4NP3QN6QLSIBM7U34QAYPAIKMN6EBPA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:39.733398290 (294h8m14.504039626s) [noise=true , address=77.90.31.18:4805 , piece_id=I23F6KYUQ7TJI4ZLEANACVNYJDMPKKMMACIZAIA6PC5S3CKQPV2Q, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Jul 5 15:18:39.733464387 (294h8m14.504105722s) [noise=true , address=77.90.31.21:4877 , piece_id=OJBIFN2TJPQ45LQLYCMPKV5UZM6Y2MXNOBT3TSGX26U4SZAL5Q7A, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: expected piece hash; context deadline exceeded; EOF Jul 5 15:18:39.733542065 (294h8m14.50418339s) [noise=true , address=77.90.31.11:4554 , piece_id=4QRZ5MPQ7HTPRIVRL3O3H4OV7CQA7ESYT5RCPI3GTHXJOHOSAOVA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection Jul 5 15:18:39.733547496 (294h8m14.504188831s) [noise=true , address=77.90.31.16:4733 , piece_id=3GQ7NEWE6WZJMQUA3GG3GSZYXEFNOAUEVWHOEH4TSNONYXUT43GQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection; write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection Jul 5 15:18:39.733583284 (294h8m14.504224619s) [seg_pos={0 0}, upload=235227]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 5 15:18:39.974321220 (294h8m14.744962555s) [upload=235227]: Done uploading segments... Jul 5 15:18:39.974353041 (294h8m14.744994376s) [upload=235227]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.966475566s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 5.604930666s FAIL (2025-07-05 15:36:56 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== JYE6FC5T5QIBUQIILGDHNGXAUDBUIAOSKYFMPBCWPHP2SDJ3A, seg_pos={0 0}, upload=235214]: Done uploading piece. err: <nil> Jul 5 15:17:12.446386719 (294h6m47.217028064s) [seg_pos={0 0}, upload=235214]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 5 15:17:12.446476771 (294h6m47.217118106s) [upload=235214]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 5 15:17:12.446483043 (294h6m47.217124378s) [upload=235214]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 5 15:17:12.905279085 (294h6m47.67592042s) [upload=235214]: Done uploading segments... Jul 5 15:17:12.905303211 (294h6m47.675944546s) [upload=235214]: Upload finished. err: <nil> Jul 5 15:18:10.381454266 (294h7m45.152095611s) [upload=235227]: Starting upload Jul 5 15:18:10.381524069 (294h7m45.152165414s) [upload=235227]: Done starting upload Jul 5 15:18:10.381621115 (294h7m45.15226245s) [upload=235227]: Uploading segments... Jul 5 15:18:10.381740133 (294h7m45.152381468s) [upload=235227]: Got next segment. Inline: false Jul 5 15:18:10.381776342 (294h7m45.152417677s) [upload=235227]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 5 15:18:10.381781973 (294h7m45.152423308s) [upload=235227]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 5 15:18:10.613904835 (294h7m45.38454617s) [seg_pos={0 0}, upload=235227]: Begin upload segment... Jul 5 15:18:10.613947397 (294h7m45.384588732s) [seg_pos={0 0}, upload=235227]: Done begin upload segment. Jul 5 15:18:10.613968548 (294h7m45.384609883s) [noise=true , address=104.36.82.15:4259 , piece_id=LQKQ5MGKXHTPITBR2QAOQ5CX76GEVJGHJT3ZD3G4PFNN5IBTCJBQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.613995038 (294h7m45.384636373s) [noise=true , address=104.36.82.16:4295 , piece_id=VEL6CAUOJKMRN7A7M66ZPDLCMJVD555LZPHM53A24MDQJQOW7ECQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614303969 (294h7m45.384945304s) [noise=true , address=104.36.82.9:4006 , piece_id=47FME2B5NS6WMOHC2ZGE2FZVSPEQUXKYT6E7AOSANAE5XZVDH5VQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614367721 (294h7m45.385009056s) [noise=true , address=45.91.116.4:4937 , piece_id=YMKIHJET2EN3QHEK665ACZNQNHJ3UGE4H5T4DIHWJUHJQBYO7B7Q, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614619272 (294h7m45.385260607s) [noise=true , address=77.90.31.17:4771 , piece_id=V7M65ZHCGXPD2GVJL6LTB4NP3QN6QLSIBM7U34QAYPAIKMN6EBPA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614637758 (294h7m45.385279083s) [noise=true , address=45.91.116.14:4191 , piece_id=WH2BCI36GGFF4QUD4XDYBBDMUR2R2FFVYKFTZD3ZWXPJCFJ3D7HQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614890251 (294h7m45.385531576s) [noise=true , address=104.36.82.3:4767 , piece_id=OGHG4G5ZVC7GVY4XI6DO4KINXF7THHL6IHGDB6DAPDNUDOUFSC4A, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615197719 (294h7m45.385839054s) [noise=true , address=104.36.82.12:4149 , piece_id=YBTLQEWDMFK6JIFTYR3VJDNMXTS3NMWWHX2MPP53BM2UBDI3UUEA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615413552 (294h7m45.386054887s) [noise=true , address=77.90.31.18:4805 , piece_id=I23F6KYUQ7TJI4ZLEANACVNYJDMPKKMMACIZAIA6PC5S3CKQPV2Q, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615608124 (294h7m45.386249459s) [noise=true , address=77.90.31.21:4877 , piece_id=OJBIFN2TJPQ45LQLYCMPKV5UZM6Y2MXNOBT3TSGX26U4SZAL5Q7A, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615803147 (294h7m45.386444472s) [noise=true , address=104.36.82.8:4970 , piece_id=2YJN7VKTTTCMAWQW2MNR65DAJPFMHNIBIW5NWYNG4R6HNPRFH6NQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615998511 (294h7m45.386639836s) [noise=true , address=77.90.31.16:4733 , piece_id=3GQ7NEWE6WZJMQUA3GG3GSZYXEFNOAUEVWHOEH4TSNONYXUT43GQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616193414 (294h7m45.386834749s) [noise=true , address=77.90.31.19:4842 , piece_id=R4FTRBZ3PETCA6QGU6RPUEDZZBWXF7YVH2COEW4WH3YWDLU3VJMA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616449093 (294h7m45.387090429s) [noise=true , address=77.90.31.11:4554 , piece_id=4QRZ5MPQ7HTPRIVRL3O3H4OV7CQA7ESYT5RCPI3GTHXJOHOSAOVA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616635840 (294h7m45.387277175s) [noise=true , address=45.91.116.5:4298 , piece_id=TVTUGWHFTXB32GBY4LYXT3YDB5LIFXEGA7VO5BHFA24Y6EMDTPHQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616818790 (294h7m45.387460125s) [noise=true , address=104.36.82.13:4185 , piece_id=LTQS2CUZ5NRJCBOPEXDKUQ2IJGOIW3JJAPHWTCW7WUMQZPIWRXRA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:12.265123988 (294h7m47.035765313s) [upload=235227]: Next returned nil segment Jul 5 15:18:12.265138164 (294h7m47.035779499s) [upload=235227]: Waiting for error group managing segments... Jul 5 15:18:12.550465694 (294h7m47.321107019s) [noise=true , address=104.36.82.8:4970 , piece_id=2YJN7VKTTTCMAWQW2MNR65DAJPFMHNIBIW5NWYNG4R6HNPRFH6NQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.554315629 (294h7m47.324956954s) [noise=true , address=104.36.82.12:4149 , piece_id=YBTLQEWDMFK6JIFTYR3VJDNMXTS3NMWWHX2MPP53BM2UBDI3UUEA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.603452512 (294h7m47.374093847s) [noise=true , address=104.36.82.3:4767 , piece_id=OGHG4G5ZVC7GVY4XI6DO4KINXF7THHL6IHGDB6DAPDNUDOUFSC4A, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.639796643 (294h7m47.410437978s) [noise=true , address=104.36.82.9:4006 , piece_id=47FME2B5NS6WMOHC2ZGE2FZVSPEQUXKYT6E7AOSANAE5XZVDH5VQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.639894960 (294h7m47.410536285s) [noise=true , address=104.36.82.15:4259 , piece_id=LQKQ5MGKXHTPITBR2QAOQ5CX76GEVJGHJT3ZD3G4PFNN5IBTCJBQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.647222659 (294h7m47.417863994s) [noise=true , address=104.36.82.16:4295 , piece_id=VEL6CAUOJKMRN7A7M66ZPDLCMJVD555LZPHM53A24MDQJQOW7ECQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.665424010 (294h7m47.436065346s) [noise=true , address=104.36.82.13:4185 , piece_id=LTQS2CUZ5NRJCBOPEXDKUQ2IJGOIW3JJAPHWTCW7WUMQZPIWRXRA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.702209574 (294h7m47.47285091s) [noise=true , address=45.91.116.14:4191 , piece_id=WH2BCI36GGFF4QUD4XDYBBDMUR2R2FFVYKFTZD3ZWXPJCFJ3D7HQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.722073145 (294h7m47.49271448s) [noise=true , address=45.91.116.5:4298 , piece_id=TVTUGWHFTXB32GBY4LYXT3YDB5LIFXEGA7VO5BHFA24Y6EMDTPHQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.788621529 (294h7m47.559262864s) [noise=true , address=45.91.116.4:4937 , piece_id=YMKIHJET2EN3QHEK665ACZNQNHJ3UGE4H5T4DIHWJUHJQBYO7B7Q, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:36.752254721 (294h8m11.522896046s) [noise=true , address=77.90.31.19:4842 , piece_id=R4FTRBZ3PETCA6QGU6RPUEDZZBWXF7YVH2COEW4WH3YWDLU3VJMA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:37.783437003 (294h8m12.554078338s) [noise=true , address=77.90.31.17:4771 , piece_id=V7M65ZHCGXPD2GVJL6LTB4NP3QN6QLSIBM7U34QAYPAIKMN6EBPA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:39.733398290 (294h8m14.504039626s) [noise=true , address=77.90.31.18:4805 , piece_id=I23F6KYUQ7TJI4ZLEANACVNYJDMPKKMMACIZAIA6PC5S3CKQPV2Q, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Jul 5 15:18:39.733464387 (294h8m14.504105722s) [noise=true , address=77.90.31.21:4877 , piece_id=OJBIFN2TJPQ45LQLYCMPKV5UZM6Y2MXNOBT3TSGX26U4SZAL5Q7A, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: expected piece hash; context deadline exceeded; EOF Jul 5 15:18:39.733542065 (294h8m14.50418339s) [noise=true , address=77.90.31.11:4554 , piece_id=4QRZ5MPQ7HTPRIVRL3O3H4OV7CQA7ESYT5RCPI3GTHXJOHOSAOVA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection Jul 5 15:18:39.733547496 (294h8m14.504188831s) [noise=true , address=77.90.31.16:4733 , piece_id=3GQ7NEWE6WZJMQUA3GG3GSZYXEFNOAUEVWHOEH4TSNONYXUT43GQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection; write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection Jul 5 15:18:39.733583284 (294h8m14.504224619s) [seg_pos={0 0}, upload=235227]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 5 15:18:39.974321220 (294h8m14.744962555s) [upload=235227]: Done uploading segments... Jul 5 15:18:39.974353041 (294h8m14.744994376s) [upload=235227]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.604956296s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 5.71043026s FAIL (2025-07-05 15:36:02 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== JYE6FC5T5QIBUQIILGDHNGXAUDBUIAOSKYFMPBCWPHP2SDJ3A, seg_pos={0 0}, upload=235214]: Done uploading piece. err: <nil> Jul 5 15:17:12.446386719 (294h6m47.217028064s) [seg_pos={0 0}, upload=235214]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 5 15:17:12.446476771 (294h6m47.217118106s) [upload=235214]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 5 15:17:12.446483043 (294h6m47.217124378s) [upload=235214]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 5 15:17:12.905279085 (294h6m47.67592042s) [upload=235214]: Done uploading segments... Jul 5 15:17:12.905303211 (294h6m47.675944546s) [upload=235214]: Upload finished. err: <nil> Jul 5 15:18:10.381454266 (294h7m45.152095611s) [upload=235227]: Starting upload Jul 5 15:18:10.381524069 (294h7m45.152165414s) [upload=235227]: Done starting upload Jul 5 15:18:10.381621115 (294h7m45.15226245s) [upload=235227]: Uploading segments... Jul 5 15:18:10.381740133 (294h7m45.152381468s) [upload=235227]: Got next segment. Inline: false Jul 5 15:18:10.381776342 (294h7m45.152417677s) [upload=235227]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 5 15:18:10.381781973 (294h7m45.152423308s) [upload=235227]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 5 15:18:10.613904835 (294h7m45.38454617s) [seg_pos={0 0}, upload=235227]: Begin upload segment... Jul 5 15:18:10.613947397 (294h7m45.384588732s) [seg_pos={0 0}, upload=235227]: Done begin upload segment. Jul 5 15:18:10.613968548 (294h7m45.384609883s) [noise=true , address=104.36.82.15:4259 , piece_id=LQKQ5MGKXHTPITBR2QAOQ5CX76GEVJGHJT3ZD3G4PFNN5IBTCJBQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.613995038 (294h7m45.384636373s) [noise=true , address=104.36.82.16:4295 , piece_id=VEL6CAUOJKMRN7A7M66ZPDLCMJVD555LZPHM53A24MDQJQOW7ECQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614303969 (294h7m45.384945304s) [noise=true , address=104.36.82.9:4006 , piece_id=47FME2B5NS6WMOHC2ZGE2FZVSPEQUXKYT6E7AOSANAE5XZVDH5VQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614367721 (294h7m45.385009056s) [noise=true , address=45.91.116.4:4937 , piece_id=YMKIHJET2EN3QHEK665ACZNQNHJ3UGE4H5T4DIHWJUHJQBYO7B7Q, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614619272 (294h7m45.385260607s) [noise=true , address=77.90.31.17:4771 , piece_id=V7M65ZHCGXPD2GVJL6LTB4NP3QN6QLSIBM7U34QAYPAIKMN6EBPA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614637758 (294h7m45.385279083s) [noise=true , address=45.91.116.14:4191 , piece_id=WH2BCI36GGFF4QUD4XDYBBDMUR2R2FFVYKFTZD3ZWXPJCFJ3D7HQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614890251 (294h7m45.385531576s) [noise=true , address=104.36.82.3:4767 , piece_id=OGHG4G5ZVC7GVY4XI6DO4KINXF7THHL6IHGDB6DAPDNUDOUFSC4A, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615197719 (294h7m45.385839054s) [noise=true , address=104.36.82.12:4149 , piece_id=YBTLQEWDMFK6JIFTYR3VJDNMXTS3NMWWHX2MPP53BM2UBDI3UUEA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615413552 (294h7m45.386054887s) [noise=true , address=77.90.31.18:4805 , piece_id=I23F6KYUQ7TJI4ZLEANACVNYJDMPKKMMACIZAIA6PC5S3CKQPV2Q, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615608124 (294h7m45.386249459s) [noise=true , address=77.90.31.21:4877 , piece_id=OJBIFN2TJPQ45LQLYCMPKV5UZM6Y2MXNOBT3TSGX26U4SZAL5Q7A, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615803147 (294h7m45.386444472s) [noise=true , address=104.36.82.8:4970 , piece_id=2YJN7VKTTTCMAWQW2MNR65DAJPFMHNIBIW5NWYNG4R6HNPRFH6NQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615998511 (294h7m45.386639836s) [noise=true , address=77.90.31.16:4733 , piece_id=3GQ7NEWE6WZJMQUA3GG3GSZYXEFNOAUEVWHOEH4TSNONYXUT43GQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616193414 (294h7m45.386834749s) [noise=true , address=77.90.31.19:4842 , piece_id=R4FTRBZ3PETCA6QGU6RPUEDZZBWXF7YVH2COEW4WH3YWDLU3VJMA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616449093 (294h7m45.387090429s) [noise=true , address=77.90.31.11:4554 , piece_id=4QRZ5MPQ7HTPRIVRL3O3H4OV7CQA7ESYT5RCPI3GTHXJOHOSAOVA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616635840 (294h7m45.387277175s) [noise=true , address=45.91.116.5:4298 , piece_id=TVTUGWHFTXB32GBY4LYXT3YDB5LIFXEGA7VO5BHFA24Y6EMDTPHQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616818790 (294h7m45.387460125s) [noise=true , address=104.36.82.13:4185 , piece_id=LTQS2CUZ5NRJCBOPEXDKUQ2IJGOIW3JJAPHWTCW7WUMQZPIWRXRA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:12.265123988 (294h7m47.035765313s) [upload=235227]: Next returned nil segment Jul 5 15:18:12.265138164 (294h7m47.035779499s) [upload=235227]: Waiting for error group managing segments... Jul 5 15:18:12.550465694 (294h7m47.321107019s) [noise=true , address=104.36.82.8:4970 , piece_id=2YJN7VKTTTCMAWQW2MNR65DAJPFMHNIBIW5NWYNG4R6HNPRFH6NQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.554315629 (294h7m47.324956954s) [noise=true , address=104.36.82.12:4149 , piece_id=YBTLQEWDMFK6JIFTYR3VJDNMXTS3NMWWHX2MPP53BM2UBDI3UUEA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.603452512 (294h7m47.374093847s) [noise=true , address=104.36.82.3:4767 , piece_id=OGHG4G5ZVC7GVY4XI6DO4KINXF7THHL6IHGDB6DAPDNUDOUFSC4A, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.639796643 (294h7m47.410437978s) [noise=true , address=104.36.82.9:4006 , piece_id=47FME2B5NS6WMOHC2ZGE2FZVSPEQUXKYT6E7AOSANAE5XZVDH5VQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.639894960 (294h7m47.410536285s) [noise=true , address=104.36.82.15:4259 , piece_id=LQKQ5MGKXHTPITBR2QAOQ5CX76GEVJGHJT3ZD3G4PFNN5IBTCJBQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.647222659 (294h7m47.417863994s) [noise=true , address=104.36.82.16:4295 , piece_id=VEL6CAUOJKMRN7A7M66ZPDLCMJVD555LZPHM53A24MDQJQOW7ECQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.665424010 (294h7m47.436065346s) [noise=true , address=104.36.82.13:4185 , piece_id=LTQS2CUZ5NRJCBOPEXDKUQ2IJGOIW3JJAPHWTCW7WUMQZPIWRXRA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.702209574 (294h7m47.47285091s) [noise=true , address=45.91.116.14:4191 , piece_id=WH2BCI36GGFF4QUD4XDYBBDMUR2R2FFVYKFTZD3ZWXPJCFJ3D7HQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.722073145 (294h7m47.49271448s) [noise=true , address=45.91.116.5:4298 , piece_id=TVTUGWHFTXB32GBY4LYXT3YDB5LIFXEGA7VO5BHFA24Y6EMDTPHQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.788621529 (294h7m47.559262864s) [noise=true , address=45.91.116.4:4937 , piece_id=YMKIHJET2EN3QHEK665ACZNQNHJ3UGE4H5T4DIHWJUHJQBYO7B7Q, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:36.752254721 (294h8m11.522896046s) [noise=true , address=77.90.31.19:4842 , piece_id=R4FTRBZ3PETCA6QGU6RPUEDZZBWXF7YVH2COEW4WH3YWDLU3VJMA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:37.783437003 (294h8m12.554078338s) [noise=true , address=77.90.31.17:4771 , piece_id=V7M65ZHCGXPD2GVJL6LTB4NP3QN6QLSIBM7U34QAYPAIKMN6EBPA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:39.733398290 (294h8m14.504039626s) [noise=true , address=77.90.31.18:4805 , piece_id=I23F6KYUQ7TJI4ZLEANACVNYJDMPKKMMACIZAIA6PC5S3CKQPV2Q, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Jul 5 15:18:39.733464387 (294h8m14.504105722s) [noise=true , address=77.90.31.21:4877 , piece_id=OJBIFN2TJPQ45LQLYCMPKV5UZM6Y2MXNOBT3TSGX26U4SZAL5Q7A, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: expected piece hash; context deadline exceeded; EOF Jul 5 15:18:39.733542065 (294h8m14.50418339s) [noise=true , address=77.90.31.11:4554 , piece_id=4QRZ5MPQ7HTPRIVRL3O3H4OV7CQA7ESYT5RCPI3GTHXJOHOSAOVA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection Jul 5 15:18:39.733547496 (294h8m14.504188831s) [noise=true , address=77.90.31.16:4733 , piece_id=3GQ7NEWE6WZJMQUA3GG3GSZYXEFNOAUEVWHOEH4TSNONYXUT43GQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection; write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection Jul 5 15:18:39.733583284 (294h8m14.504224619s) [seg_pos={0 0}, upload=235227]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 5 15:18:39.974321220 (294h8m14.744962555s) [upload=235227]: Done uploading segments... Jul 5 15:18:39.974353041 (294h8m14.744994376s) [upload=235227]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.710460366s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 5.62493116s FAIL (2025-07-05 15:35:34 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== JYE6FC5T5QIBUQIILGDHNGXAUDBUIAOSKYFMPBCWPHP2SDJ3A, seg_pos={0 0}, upload=235214]: Done uploading piece. err: <nil> Jul 5 15:17:12.446386719 (294h6m47.217028064s) [seg_pos={0 0}, upload=235214]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 5 15:17:12.446476771 (294h6m47.217118106s) [upload=235214]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 5 15:17:12.446483043 (294h6m47.217124378s) [upload=235214]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 5 15:17:12.905279085 (294h6m47.67592042s) [upload=235214]: Done uploading segments... Jul 5 15:17:12.905303211 (294h6m47.675944546s) [upload=235214]: Upload finished. err: <nil> Jul 5 15:18:10.381454266 (294h7m45.152095611s) [upload=235227]: Starting upload Jul 5 15:18:10.381524069 (294h7m45.152165414s) [upload=235227]: Done starting upload Jul 5 15:18:10.381621115 (294h7m45.15226245s) [upload=235227]: Uploading segments... Jul 5 15:18:10.381740133 (294h7m45.152381468s) [upload=235227]: Got next segment. Inline: false Jul 5 15:18:10.381776342 (294h7m45.152417677s) [upload=235227]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 5 15:18:10.381781973 (294h7m45.152423308s) [upload=235227]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 5 15:18:10.613904835 (294h7m45.38454617s) [seg_pos={0 0}, upload=235227]: Begin upload segment... Jul 5 15:18:10.613947397 (294h7m45.384588732s) [seg_pos={0 0}, upload=235227]: Done begin upload segment. Jul 5 15:18:10.613968548 (294h7m45.384609883s) [noise=true , address=104.36.82.15:4259 , piece_id=LQKQ5MGKXHTPITBR2QAOQ5CX76GEVJGHJT3ZD3G4PFNN5IBTCJBQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.613995038 (294h7m45.384636373s) [noise=true , address=104.36.82.16:4295 , piece_id=VEL6CAUOJKMRN7A7M66ZPDLCMJVD555LZPHM53A24MDQJQOW7ECQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614303969 (294h7m45.384945304s) [noise=true , address=104.36.82.9:4006 , piece_id=47FME2B5NS6WMOHC2ZGE2FZVSPEQUXKYT6E7AOSANAE5XZVDH5VQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614367721 (294h7m45.385009056s) [noise=true , address=45.91.116.4:4937 , piece_id=YMKIHJET2EN3QHEK665ACZNQNHJ3UGE4H5T4DIHWJUHJQBYO7B7Q, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614619272 (294h7m45.385260607s) [noise=true , address=77.90.31.17:4771 , piece_id=V7M65ZHCGXPD2GVJL6LTB4NP3QN6QLSIBM7U34QAYPAIKMN6EBPA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614637758 (294h7m45.385279083s) [noise=true , address=45.91.116.14:4191 , piece_id=WH2BCI36GGFF4QUD4XDYBBDMUR2R2FFVYKFTZD3ZWXPJCFJ3D7HQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614890251 (294h7m45.385531576s) [noise=true , address=104.36.82.3:4767 , piece_id=OGHG4G5ZVC7GVY4XI6DO4KINXF7THHL6IHGDB6DAPDNUDOUFSC4A, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615197719 (294h7m45.385839054s) [noise=true , address=104.36.82.12:4149 , piece_id=YBTLQEWDMFK6JIFTYR3VJDNMXTS3NMWWHX2MPP53BM2UBDI3UUEA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615413552 (294h7m45.386054887s) [noise=true , address=77.90.31.18:4805 , piece_id=I23F6KYUQ7TJI4ZLEANACVNYJDMPKKMMACIZAIA6PC5S3CKQPV2Q, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615608124 (294h7m45.386249459s) [noise=true , address=77.90.31.21:4877 , piece_id=OJBIFN2TJPQ45LQLYCMPKV5UZM6Y2MXNOBT3TSGX26U4SZAL5Q7A, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615803147 (294h7m45.386444472s) [noise=true , address=104.36.82.8:4970 , piece_id=2YJN7VKTTTCMAWQW2MNR65DAJPFMHNIBIW5NWYNG4R6HNPRFH6NQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615998511 (294h7m45.386639836s) [noise=true , address=77.90.31.16:4733 , piece_id=3GQ7NEWE6WZJMQUA3GG3GSZYXEFNOAUEVWHOEH4TSNONYXUT43GQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616193414 (294h7m45.386834749s) [noise=true , address=77.90.31.19:4842 , piece_id=R4FTRBZ3PETCA6QGU6RPUEDZZBWXF7YVH2COEW4WH3YWDLU3VJMA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616449093 (294h7m45.387090429s) [noise=true , address=77.90.31.11:4554 , piece_id=4QRZ5MPQ7HTPRIVRL3O3H4OV7CQA7ESYT5RCPI3GTHXJOHOSAOVA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616635840 (294h7m45.387277175s) [noise=true , address=45.91.116.5:4298 , piece_id=TVTUGWHFTXB32GBY4LYXT3YDB5LIFXEGA7VO5BHFA24Y6EMDTPHQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616818790 (294h7m45.387460125s) [noise=true , address=104.36.82.13:4185 , piece_id=LTQS2CUZ5NRJCBOPEXDKUQ2IJGOIW3JJAPHWTCW7WUMQZPIWRXRA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:12.265123988 (294h7m47.035765313s) [upload=235227]: Next returned nil segment Jul 5 15:18:12.265138164 (294h7m47.035779499s) [upload=235227]: Waiting for error group managing segments... Jul 5 15:18:12.550465694 (294h7m47.321107019s) [noise=true , address=104.36.82.8:4970 , piece_id=2YJN7VKTTTCMAWQW2MNR65DAJPFMHNIBIW5NWYNG4R6HNPRFH6NQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.554315629 (294h7m47.324956954s) [noise=true , address=104.36.82.12:4149 , piece_id=YBTLQEWDMFK6JIFTYR3VJDNMXTS3NMWWHX2MPP53BM2UBDI3UUEA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.603452512 (294h7m47.374093847s) [noise=true , address=104.36.82.3:4767 , piece_id=OGHG4G5ZVC7GVY4XI6DO4KINXF7THHL6IHGDB6DAPDNUDOUFSC4A, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.639796643 (294h7m47.410437978s) [noise=true , address=104.36.82.9:4006 , piece_id=47FME2B5NS6WMOHC2ZGE2FZVSPEQUXKYT6E7AOSANAE5XZVDH5VQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.639894960 (294h7m47.410536285s) [noise=true , address=104.36.82.15:4259 , piece_id=LQKQ5MGKXHTPITBR2QAOQ5CX76GEVJGHJT3ZD3G4PFNN5IBTCJBQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.647222659 (294h7m47.417863994s) [noise=true , address=104.36.82.16:4295 , piece_id=VEL6CAUOJKMRN7A7M66ZPDLCMJVD555LZPHM53A24MDQJQOW7ECQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.665424010 (294h7m47.436065346s) [noise=true , address=104.36.82.13:4185 , piece_id=LTQS2CUZ5NRJCBOPEXDKUQ2IJGOIW3JJAPHWTCW7WUMQZPIWRXRA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.702209574 (294h7m47.47285091s) [noise=true , address=45.91.116.14:4191 , piece_id=WH2BCI36GGFF4QUD4XDYBBDMUR2R2FFVYKFTZD3ZWXPJCFJ3D7HQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.722073145 (294h7m47.49271448s) [noise=true , address=45.91.116.5:4298 , piece_id=TVTUGWHFTXB32GBY4LYXT3YDB5LIFXEGA7VO5BHFA24Y6EMDTPHQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.788621529 (294h7m47.559262864s) [noise=true , address=45.91.116.4:4937 , piece_id=YMKIHJET2EN3QHEK665ACZNQNHJ3UGE4H5T4DIHWJUHJQBYO7B7Q, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:36.752254721 (294h8m11.522896046s) [noise=true , address=77.90.31.19:4842 , piece_id=R4FTRBZ3PETCA6QGU6RPUEDZZBWXF7YVH2COEW4WH3YWDLU3VJMA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:37.783437003 (294h8m12.554078338s) [noise=true , address=77.90.31.17:4771 , piece_id=V7M65ZHCGXPD2GVJL6LTB4NP3QN6QLSIBM7U34QAYPAIKMN6EBPA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:39.733398290 (294h8m14.504039626s) [noise=true , address=77.90.31.18:4805 , piece_id=I23F6KYUQ7TJI4ZLEANACVNYJDMPKKMMACIZAIA6PC5S3CKQPV2Q, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Jul 5 15:18:39.733464387 (294h8m14.504105722s) [noise=true , address=77.90.31.21:4877 , piece_id=OJBIFN2TJPQ45LQLYCMPKV5UZM6Y2MXNOBT3TSGX26U4SZAL5Q7A, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: expected piece hash; context deadline exceeded; EOF Jul 5 15:18:39.733542065 (294h8m14.50418339s) [noise=true , address=77.90.31.11:4554 , piece_id=4QRZ5MPQ7HTPRIVRL3O3H4OV7CQA7ESYT5RCPI3GTHXJOHOSAOVA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection Jul 5 15:18:39.733547496 (294h8m14.504188831s) [noise=true , address=77.90.31.16:4733 , piece_id=3GQ7NEWE6WZJMQUA3GG3GSZYXEFNOAUEVWHOEH4TSNONYXUT43GQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection; write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection Jul 5 15:18:39.733583284 (294h8m14.504224619s) [seg_pos={0 0}, upload=235227]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 5 15:18:39.974321220 (294h8m14.744962555s) [upload=235227]: Done uploading segments... Jul 5 15:18:39.974353041 (294h8m14.744994376s) [upload=235227]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.624952661s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 5.58589501s FAIL (2025-07-05 15:34:20 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== JYE6FC5T5QIBUQIILGDHNGXAUDBUIAOSKYFMPBCWPHP2SDJ3A, seg_pos={0 0}, upload=235214]: Done uploading piece. err: <nil> Jul 5 15:17:12.446386719 (294h6m47.217028064s) [seg_pos={0 0}, upload=235214]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 5 15:17:12.446476771 (294h6m47.217118106s) [upload=235214]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 5 15:17:12.446483043 (294h6m47.217124378s) [upload=235214]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 5 15:17:12.905279085 (294h6m47.67592042s) [upload=235214]: Done uploading segments... Jul 5 15:17:12.905303211 (294h6m47.675944546s) [upload=235214]: Upload finished. err: <nil> Jul 5 15:18:10.381454266 (294h7m45.152095611s) [upload=235227]: Starting upload Jul 5 15:18:10.381524069 (294h7m45.152165414s) [upload=235227]: Done starting upload Jul 5 15:18:10.381621115 (294h7m45.15226245s) [upload=235227]: Uploading segments... Jul 5 15:18:10.381740133 (294h7m45.152381468s) [upload=235227]: Got next segment. Inline: false Jul 5 15:18:10.381776342 (294h7m45.152417677s) [upload=235227]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 5 15:18:10.381781973 (294h7m45.152423308s) [upload=235227]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 5 15:18:10.613904835 (294h7m45.38454617s) [seg_pos={0 0}, upload=235227]: Begin upload segment... Jul 5 15:18:10.613947397 (294h7m45.384588732s) [seg_pos={0 0}, upload=235227]: Done begin upload segment. Jul 5 15:18:10.613968548 (294h7m45.384609883s) [noise=true , address=104.36.82.15:4259 , piece_id=LQKQ5MGKXHTPITBR2QAOQ5CX76GEVJGHJT3ZD3G4PFNN5IBTCJBQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.613995038 (294h7m45.384636373s) [noise=true , address=104.36.82.16:4295 , piece_id=VEL6CAUOJKMRN7A7M66ZPDLCMJVD555LZPHM53A24MDQJQOW7ECQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614303969 (294h7m45.384945304s) [noise=true , address=104.36.82.9:4006 , piece_id=47FME2B5NS6WMOHC2ZGE2FZVSPEQUXKYT6E7AOSANAE5XZVDH5VQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614367721 (294h7m45.385009056s) [noise=true , address=45.91.116.4:4937 , piece_id=YMKIHJET2EN3QHEK665ACZNQNHJ3UGE4H5T4DIHWJUHJQBYO7B7Q, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614619272 (294h7m45.385260607s) [noise=true , address=77.90.31.17:4771 , piece_id=V7M65ZHCGXPD2GVJL6LTB4NP3QN6QLSIBM7U34QAYPAIKMN6EBPA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614637758 (294h7m45.385279083s) [noise=true , address=45.91.116.14:4191 , piece_id=WH2BCI36GGFF4QUD4XDYBBDMUR2R2FFVYKFTZD3ZWXPJCFJ3D7HQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614890251 (294h7m45.385531576s) [noise=true , address=104.36.82.3:4767 , piece_id=OGHG4G5ZVC7GVY4XI6DO4KINXF7THHL6IHGDB6DAPDNUDOUFSC4A, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615197719 (294h7m45.385839054s) [noise=true , address=104.36.82.12:4149 , piece_id=YBTLQEWDMFK6JIFTYR3VJDNMXTS3NMWWHX2MPP53BM2UBDI3UUEA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615413552 (294h7m45.386054887s) [noise=true , address=77.90.31.18:4805 , piece_id=I23F6KYUQ7TJI4ZLEANACVNYJDMPKKMMACIZAIA6PC5S3CKQPV2Q, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615608124 (294h7m45.386249459s) [noise=true , address=77.90.31.21:4877 , piece_id=OJBIFN2TJPQ45LQLYCMPKV5UZM6Y2MXNOBT3TSGX26U4SZAL5Q7A, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615803147 (294h7m45.386444472s) [noise=true , address=104.36.82.8:4970 , piece_id=2YJN7VKTTTCMAWQW2MNR65DAJPFMHNIBIW5NWYNG4R6HNPRFH6NQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615998511 (294h7m45.386639836s) [noise=true , address=77.90.31.16:4733 , piece_id=3GQ7NEWE6WZJMQUA3GG3GSZYXEFNOAUEVWHOEH4TSNONYXUT43GQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616193414 (294h7m45.386834749s) [noise=true , address=77.90.31.19:4842 , piece_id=R4FTRBZ3PETCA6QGU6RPUEDZZBWXF7YVH2COEW4WH3YWDLU3VJMA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616449093 (294h7m45.387090429s) [noise=true , address=77.90.31.11:4554 , piece_id=4QRZ5MPQ7HTPRIVRL3O3H4OV7CQA7ESYT5RCPI3GTHXJOHOSAOVA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616635840 (294h7m45.387277175s) [noise=true , address=45.91.116.5:4298 , piece_id=TVTUGWHFTXB32GBY4LYXT3YDB5LIFXEGA7VO5BHFA24Y6EMDTPHQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616818790 (294h7m45.387460125s) [noise=true , address=104.36.82.13:4185 , piece_id=LTQS2CUZ5NRJCBOPEXDKUQ2IJGOIW3JJAPHWTCW7WUMQZPIWRXRA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:12.265123988 (294h7m47.035765313s) [upload=235227]: Next returned nil segment Jul 5 15:18:12.265138164 (294h7m47.035779499s) [upload=235227]: Waiting for error group managing segments... Jul 5 15:18:12.550465694 (294h7m47.321107019s) [noise=true , address=104.36.82.8:4970 , piece_id=2YJN7VKTTTCMAWQW2MNR65DAJPFMHNIBIW5NWYNG4R6HNPRFH6NQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.554315629 (294h7m47.324956954s) [noise=true , address=104.36.82.12:4149 , piece_id=YBTLQEWDMFK6JIFTYR3VJDNMXTS3NMWWHX2MPP53BM2UBDI3UUEA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.603452512 (294h7m47.374093847s) [noise=true , address=104.36.82.3:4767 , piece_id=OGHG4G5ZVC7GVY4XI6DO4KINXF7THHL6IHGDB6DAPDNUDOUFSC4A, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.639796643 (294h7m47.410437978s) [noise=true , address=104.36.82.9:4006 , piece_id=47FME2B5NS6WMOHC2ZGE2FZVSPEQUXKYT6E7AOSANAE5XZVDH5VQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.639894960 (294h7m47.410536285s) [noise=true , address=104.36.82.15:4259 , piece_id=LQKQ5MGKXHTPITBR2QAOQ5CX76GEVJGHJT3ZD3G4PFNN5IBTCJBQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.647222659 (294h7m47.417863994s) [noise=true , address=104.36.82.16:4295 , piece_id=VEL6CAUOJKMRN7A7M66ZPDLCMJVD555LZPHM53A24MDQJQOW7ECQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.665424010 (294h7m47.436065346s) [noise=true , address=104.36.82.13:4185 , piece_id=LTQS2CUZ5NRJCBOPEXDKUQ2IJGOIW3JJAPHWTCW7WUMQZPIWRXRA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.702209574 (294h7m47.47285091s) [noise=true , address=45.91.116.14:4191 , piece_id=WH2BCI36GGFF4QUD4XDYBBDMUR2R2FFVYKFTZD3ZWXPJCFJ3D7HQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.722073145 (294h7m47.49271448s) [noise=true , address=45.91.116.5:4298 , piece_id=TVTUGWHFTXB32GBY4LYXT3YDB5LIFXEGA7VO5BHFA24Y6EMDTPHQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.788621529 (294h7m47.559262864s) [noise=true , address=45.91.116.4:4937 , piece_id=YMKIHJET2EN3QHEK665ACZNQNHJ3UGE4H5T4DIHWJUHJQBYO7B7Q, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:36.752254721 (294h8m11.522896046s) [noise=true , address=77.90.31.19:4842 , piece_id=R4FTRBZ3PETCA6QGU6RPUEDZZBWXF7YVH2COEW4WH3YWDLU3VJMA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:37.783437003 (294h8m12.554078338s) [noise=true , address=77.90.31.17:4771 , piece_id=V7M65ZHCGXPD2GVJL6LTB4NP3QN6QLSIBM7U34QAYPAIKMN6EBPA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:39.733398290 (294h8m14.504039626s) [noise=true , address=77.90.31.18:4805 , piece_id=I23F6KYUQ7TJI4ZLEANACVNYJDMPKKMMACIZAIA6PC5S3CKQPV2Q, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Jul 5 15:18:39.733464387 (294h8m14.504105722s) [noise=true , address=77.90.31.21:4877 , piece_id=OJBIFN2TJPQ45LQLYCMPKV5UZM6Y2MXNOBT3TSGX26U4SZAL5Q7A, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: expected piece hash; context deadline exceeded; EOF Jul 5 15:18:39.733542065 (294h8m14.50418339s) [noise=true , address=77.90.31.11:4554 , piece_id=4QRZ5MPQ7HTPRIVRL3O3H4OV7CQA7ESYT5RCPI3GTHXJOHOSAOVA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection Jul 5 15:18:39.733547496 (294h8m14.504188831s) [noise=true , address=77.90.31.16:4733 , piece_id=3GQ7NEWE6WZJMQUA3GG3GSZYXEFNOAUEVWHOEH4TSNONYXUT43GQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection; write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection Jul 5 15:18:39.733583284 (294h8m14.504224619s) [seg_pos={0 0}, upload=235227]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 5 15:18:39.974321220 (294h8m14.744962555s) [upload=235227]: Done uploading segments... Jul 5 15:18:39.974353041 (294h8m14.744994376s) [upload=235227]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.585926921s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 5.70303964s FAIL (2025-07-05 15:33:46 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== JYE6FC5T5QIBUQIILGDHNGXAUDBUIAOSKYFMPBCWPHP2SDJ3A, seg_pos={0 0}, upload=235214]: Done uploading piece. err: <nil> Jul 5 15:17:12.446386719 (294h6m47.217028064s) [seg_pos={0 0}, upload=235214]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 5 15:17:12.446476771 (294h6m47.217118106s) [upload=235214]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 5 15:17:12.446483043 (294h6m47.217124378s) [upload=235214]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 5 15:17:12.905279085 (294h6m47.67592042s) [upload=235214]: Done uploading segments... Jul 5 15:17:12.905303211 (294h6m47.675944546s) [upload=235214]: Upload finished. err: <nil> Jul 5 15:18:10.381454266 (294h7m45.152095611s) [upload=235227]: Starting upload Jul 5 15:18:10.381524069 (294h7m45.152165414s) [upload=235227]: Done starting upload Jul 5 15:18:10.381621115 (294h7m45.15226245s) [upload=235227]: Uploading segments... Jul 5 15:18:10.381740133 (294h7m45.152381468s) [upload=235227]: Got next segment. Inline: false Jul 5 15:18:10.381776342 (294h7m45.152417677s) [upload=235227]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 5 15:18:10.381781973 (294h7m45.152423308s) [upload=235227]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 5 15:18:10.613904835 (294h7m45.38454617s) [seg_pos={0 0}, upload=235227]: Begin upload segment... Jul 5 15:18:10.613947397 (294h7m45.384588732s) [seg_pos={0 0}, upload=235227]: Done begin upload segment. Jul 5 15:18:10.613968548 (294h7m45.384609883s) [noise=true , address=104.36.82.15:4259 , piece_id=LQKQ5MGKXHTPITBR2QAOQ5CX76GEVJGHJT3ZD3G4PFNN5IBTCJBQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.613995038 (294h7m45.384636373s) [noise=true , address=104.36.82.16:4295 , piece_id=VEL6CAUOJKMRN7A7M66ZPDLCMJVD555LZPHM53A24MDQJQOW7ECQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614303969 (294h7m45.384945304s) [noise=true , address=104.36.82.9:4006 , piece_id=47FME2B5NS6WMOHC2ZGE2FZVSPEQUXKYT6E7AOSANAE5XZVDH5VQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614367721 (294h7m45.385009056s) [noise=true , address=45.91.116.4:4937 , piece_id=YMKIHJET2EN3QHEK665ACZNQNHJ3UGE4H5T4DIHWJUHJQBYO7B7Q, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614619272 (294h7m45.385260607s) [noise=true , address=77.90.31.17:4771 , piece_id=V7M65ZHCGXPD2GVJL6LTB4NP3QN6QLSIBM7U34QAYPAIKMN6EBPA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614637758 (294h7m45.385279083s) [noise=true , address=45.91.116.14:4191 , piece_id=WH2BCI36GGFF4QUD4XDYBBDMUR2R2FFVYKFTZD3ZWXPJCFJ3D7HQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614890251 (294h7m45.385531576s) [noise=true , address=104.36.82.3:4767 , piece_id=OGHG4G5ZVC7GVY4XI6DO4KINXF7THHL6IHGDB6DAPDNUDOUFSC4A, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615197719 (294h7m45.385839054s) [noise=true , address=104.36.82.12:4149 , piece_id=YBTLQEWDMFK6JIFTYR3VJDNMXTS3NMWWHX2MPP53BM2UBDI3UUEA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615413552 (294h7m45.386054887s) [noise=true , address=77.90.31.18:4805 , piece_id=I23F6KYUQ7TJI4ZLEANACVNYJDMPKKMMACIZAIA6PC5S3CKQPV2Q, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615608124 (294h7m45.386249459s) [noise=true , address=77.90.31.21:4877 , piece_id=OJBIFN2TJPQ45LQLYCMPKV5UZM6Y2MXNOBT3TSGX26U4SZAL5Q7A, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615803147 (294h7m45.386444472s) [noise=true , address=104.36.82.8:4970 , piece_id=2YJN7VKTTTCMAWQW2MNR65DAJPFMHNIBIW5NWYNG4R6HNPRFH6NQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615998511 (294h7m45.386639836s) [noise=true , address=77.90.31.16:4733 , piece_id=3GQ7NEWE6WZJMQUA3GG3GSZYXEFNOAUEVWHOEH4TSNONYXUT43GQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616193414 (294h7m45.386834749s) [noise=true , address=77.90.31.19:4842 , piece_id=R4FTRBZ3PETCA6QGU6RPUEDZZBWXF7YVH2COEW4WH3YWDLU3VJMA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616449093 (294h7m45.387090429s) [noise=true , address=77.90.31.11:4554 , piece_id=4QRZ5MPQ7HTPRIVRL3O3H4OV7CQA7ESYT5RCPI3GTHXJOHOSAOVA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616635840 (294h7m45.387277175s) [noise=true , address=45.91.116.5:4298 , piece_id=TVTUGWHFTXB32GBY4LYXT3YDB5LIFXEGA7VO5BHFA24Y6EMDTPHQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616818790 (294h7m45.387460125s) [noise=true , address=104.36.82.13:4185 , piece_id=LTQS2CUZ5NRJCBOPEXDKUQ2IJGOIW3JJAPHWTCW7WUMQZPIWRXRA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:12.265123988 (294h7m47.035765313s) [upload=235227]: Next returned nil segment Jul 5 15:18:12.265138164 (294h7m47.035779499s) [upload=235227]: Waiting for error group managing segments... Jul 5 15:18:12.550465694 (294h7m47.321107019s) [noise=true , address=104.36.82.8:4970 , piece_id=2YJN7VKTTTCMAWQW2MNR65DAJPFMHNIBIW5NWYNG4R6HNPRFH6NQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.554315629 (294h7m47.324956954s) [noise=true , address=104.36.82.12:4149 , piece_id=YBTLQEWDMFK6JIFTYR3VJDNMXTS3NMWWHX2MPP53BM2UBDI3UUEA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.603452512 (294h7m47.374093847s) [noise=true , address=104.36.82.3:4767 , piece_id=OGHG4G5ZVC7GVY4XI6DO4KINXF7THHL6IHGDB6DAPDNUDOUFSC4A, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.639796643 (294h7m47.410437978s) [noise=true , address=104.36.82.9:4006 , piece_id=47FME2B5NS6WMOHC2ZGE2FZVSPEQUXKYT6E7AOSANAE5XZVDH5VQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.639894960 (294h7m47.410536285s) [noise=true , address=104.36.82.15:4259 , piece_id=LQKQ5MGKXHTPITBR2QAOQ5CX76GEVJGHJT3ZD3G4PFNN5IBTCJBQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.647222659 (294h7m47.417863994s) [noise=true , address=104.36.82.16:4295 , piece_id=VEL6CAUOJKMRN7A7M66ZPDLCMJVD555LZPHM53A24MDQJQOW7ECQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.665424010 (294h7m47.436065346s) [noise=true , address=104.36.82.13:4185 , piece_id=LTQS2CUZ5NRJCBOPEXDKUQ2IJGOIW3JJAPHWTCW7WUMQZPIWRXRA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.702209574 (294h7m47.47285091s) [noise=true , address=45.91.116.14:4191 , piece_id=WH2BCI36GGFF4QUD4XDYBBDMUR2R2FFVYKFTZD3ZWXPJCFJ3D7HQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.722073145 (294h7m47.49271448s) [noise=true , address=45.91.116.5:4298 , piece_id=TVTUGWHFTXB32GBY4LYXT3YDB5LIFXEGA7VO5BHFA24Y6EMDTPHQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.788621529 (294h7m47.559262864s) [noise=true , address=45.91.116.4:4937 , piece_id=YMKIHJET2EN3QHEK665ACZNQNHJ3UGE4H5T4DIHWJUHJQBYO7B7Q, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:36.752254721 (294h8m11.522896046s) [noise=true , address=77.90.31.19:4842 , piece_id=R4FTRBZ3PETCA6QGU6RPUEDZZBWXF7YVH2COEW4WH3YWDLU3VJMA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:37.783437003 (294h8m12.554078338s) [noise=true , address=77.90.31.17:4771 , piece_id=V7M65ZHCGXPD2GVJL6LTB4NP3QN6QLSIBM7U34QAYPAIKMN6EBPA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:39.733398290 (294h8m14.504039626s) [noise=true , address=77.90.31.18:4805 , piece_id=I23F6KYUQ7TJI4ZLEANACVNYJDMPKKMMACIZAIA6PC5S3CKQPV2Q, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Jul 5 15:18:39.733464387 (294h8m14.504105722s) [noise=true , address=77.90.31.21:4877 , piece_id=OJBIFN2TJPQ45LQLYCMPKV5UZM6Y2MXNOBT3TSGX26U4SZAL5Q7A, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: expected piece hash; context deadline exceeded; EOF Jul 5 15:18:39.733542065 (294h8m14.50418339s) [noise=true , address=77.90.31.11:4554 , piece_id=4QRZ5MPQ7HTPRIVRL3O3H4OV7CQA7ESYT5RCPI3GTHXJOHOSAOVA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection Jul 5 15:18:39.733547496 (294h8m14.504188831s) [noise=true , address=77.90.31.16:4733 , piece_id=3GQ7NEWE6WZJMQUA3GG3GSZYXEFNOAUEVWHOEH4TSNONYXUT43GQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection; write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection Jul 5 15:18:39.733583284 (294h8m14.504224619s) [seg_pos={0 0}, upload=235227]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 5 15:18:39.974321220 (294h8m14.744962555s) [upload=235227]: Done uploading segments... Jul 5 15:18:39.974353041 (294h8m14.744994376s) [upload=235227]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.703069366s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 5.664789474s FAIL (2025-07-05 15:32:39 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== JYE6FC5T5QIBUQIILGDHNGXAUDBUIAOSKYFMPBCWPHP2SDJ3A, seg_pos={0 0}, upload=235214]: Done uploading piece. err: <nil> Jul 5 15:17:12.446386719 (294h6m47.217028064s) [seg_pos={0 0}, upload=235214]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 5 15:17:12.446476771 (294h6m47.217118106s) [upload=235214]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 5 15:17:12.446483043 (294h6m47.217124378s) [upload=235214]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 5 15:17:12.905279085 (294h6m47.67592042s) [upload=235214]: Done uploading segments... Jul 5 15:17:12.905303211 (294h6m47.675944546s) [upload=235214]: Upload finished. err: <nil> Jul 5 15:18:10.381454266 (294h7m45.152095611s) [upload=235227]: Starting upload Jul 5 15:18:10.381524069 (294h7m45.152165414s) [upload=235227]: Done starting upload Jul 5 15:18:10.381621115 (294h7m45.15226245s) [upload=235227]: Uploading segments... Jul 5 15:18:10.381740133 (294h7m45.152381468s) [upload=235227]: Got next segment. Inline: false Jul 5 15:18:10.381776342 (294h7m45.152417677s) [upload=235227]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 5 15:18:10.381781973 (294h7m45.152423308s) [upload=235227]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 5 15:18:10.613904835 (294h7m45.38454617s) [seg_pos={0 0}, upload=235227]: Begin upload segment... Jul 5 15:18:10.613947397 (294h7m45.384588732s) [seg_pos={0 0}, upload=235227]: Done begin upload segment. Jul 5 15:18:10.613968548 (294h7m45.384609883s) [noise=true , address=104.36.82.15:4259 , piece_id=LQKQ5MGKXHTPITBR2QAOQ5CX76GEVJGHJT3ZD3G4PFNN5IBTCJBQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.613995038 (294h7m45.384636373s) [noise=true , address=104.36.82.16:4295 , piece_id=VEL6CAUOJKMRN7A7M66ZPDLCMJVD555LZPHM53A24MDQJQOW7ECQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614303969 (294h7m45.384945304s) [noise=true , address=104.36.82.9:4006 , piece_id=47FME2B5NS6WMOHC2ZGE2FZVSPEQUXKYT6E7AOSANAE5XZVDH5VQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614367721 (294h7m45.385009056s) [noise=true , address=45.91.116.4:4937 , piece_id=YMKIHJET2EN3QHEK665ACZNQNHJ3UGE4H5T4DIHWJUHJQBYO7B7Q, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614619272 (294h7m45.385260607s) [noise=true , address=77.90.31.17:4771 , piece_id=V7M65ZHCGXPD2GVJL6LTB4NP3QN6QLSIBM7U34QAYPAIKMN6EBPA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614637758 (294h7m45.385279083s) [noise=true , address=45.91.116.14:4191 , piece_id=WH2BCI36GGFF4QUD4XDYBBDMUR2R2FFVYKFTZD3ZWXPJCFJ3D7HQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614890251 (294h7m45.385531576s) [noise=true , address=104.36.82.3:4767 , piece_id=OGHG4G5ZVC7GVY4XI6DO4KINXF7THHL6IHGDB6DAPDNUDOUFSC4A, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615197719 (294h7m45.385839054s) [noise=true , address=104.36.82.12:4149 , piece_id=YBTLQEWDMFK6JIFTYR3VJDNMXTS3NMWWHX2MPP53BM2UBDI3UUEA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615413552 (294h7m45.386054887s) [noise=true , address=77.90.31.18:4805 , piece_id=I23F6KYUQ7TJI4ZLEANACVNYJDMPKKMMACIZAIA6PC5S3CKQPV2Q, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615608124 (294h7m45.386249459s) [noise=true , address=77.90.31.21:4877 , piece_id=OJBIFN2TJPQ45LQLYCMPKV5UZM6Y2MXNOBT3TSGX26U4SZAL5Q7A, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615803147 (294h7m45.386444472s) [noise=true , address=104.36.82.8:4970 , piece_id=2YJN7VKTTTCMAWQW2MNR65DAJPFMHNIBIW5NWYNG4R6HNPRFH6NQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615998511 (294h7m45.386639836s) [noise=true , address=77.90.31.16:4733 , piece_id=3GQ7NEWE6WZJMQUA3GG3GSZYXEFNOAUEVWHOEH4TSNONYXUT43GQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616193414 (294h7m45.386834749s) [noise=true , address=77.90.31.19:4842 , piece_id=R4FTRBZ3PETCA6QGU6RPUEDZZBWXF7YVH2COEW4WH3YWDLU3VJMA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616449093 (294h7m45.387090429s) [noise=true , address=77.90.31.11:4554 , piece_id=4QRZ5MPQ7HTPRIVRL3O3H4OV7CQA7ESYT5RCPI3GTHXJOHOSAOVA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616635840 (294h7m45.387277175s) [noise=true , address=45.91.116.5:4298 , piece_id=TVTUGWHFTXB32GBY4LYXT3YDB5LIFXEGA7VO5BHFA24Y6EMDTPHQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616818790 (294h7m45.387460125s) [noise=true , address=104.36.82.13:4185 , piece_id=LTQS2CUZ5NRJCBOPEXDKUQ2IJGOIW3JJAPHWTCW7WUMQZPIWRXRA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:12.265123988 (294h7m47.035765313s) [upload=235227]: Next returned nil segment Jul 5 15:18:12.265138164 (294h7m47.035779499s) [upload=235227]: Waiting for error group managing segments... Jul 5 15:18:12.550465694 (294h7m47.321107019s) [noise=true , address=104.36.82.8:4970 , piece_id=2YJN7VKTTTCMAWQW2MNR65DAJPFMHNIBIW5NWYNG4R6HNPRFH6NQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.554315629 (294h7m47.324956954s) [noise=true , address=104.36.82.12:4149 , piece_id=YBTLQEWDMFK6JIFTYR3VJDNMXTS3NMWWHX2MPP53BM2UBDI3UUEA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.603452512 (294h7m47.374093847s) [noise=true , address=104.36.82.3:4767 , piece_id=OGHG4G5ZVC7GVY4XI6DO4KINXF7THHL6IHGDB6DAPDNUDOUFSC4A, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.639796643 (294h7m47.410437978s) [noise=true , address=104.36.82.9:4006 , piece_id=47FME2B5NS6WMOHC2ZGE2FZVSPEQUXKYT6E7AOSANAE5XZVDH5VQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.639894960 (294h7m47.410536285s) [noise=true , address=104.36.82.15:4259 , piece_id=LQKQ5MGKXHTPITBR2QAOQ5CX76GEVJGHJT3ZD3G4PFNN5IBTCJBQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.647222659 (294h7m47.417863994s) [noise=true , address=104.36.82.16:4295 , piece_id=VEL6CAUOJKMRN7A7M66ZPDLCMJVD555LZPHM53A24MDQJQOW7ECQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.665424010 (294h7m47.436065346s) [noise=true , address=104.36.82.13:4185 , piece_id=LTQS2CUZ5NRJCBOPEXDKUQ2IJGOIW3JJAPHWTCW7WUMQZPIWRXRA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.702209574 (294h7m47.47285091s) [noise=true , address=45.91.116.14:4191 , piece_id=WH2BCI36GGFF4QUD4XDYBBDMUR2R2FFVYKFTZD3ZWXPJCFJ3D7HQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.722073145 (294h7m47.49271448s) [noise=true , address=45.91.116.5:4298 , piece_id=TVTUGWHFTXB32GBY4LYXT3YDB5LIFXEGA7VO5BHFA24Y6EMDTPHQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.788621529 (294h7m47.559262864s) [noise=true , address=45.91.116.4:4937 , piece_id=YMKIHJET2EN3QHEK665ACZNQNHJ3UGE4H5T4DIHWJUHJQBYO7B7Q, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:36.752254721 (294h8m11.522896046s) [noise=true , address=77.90.31.19:4842 , piece_id=R4FTRBZ3PETCA6QGU6RPUEDZZBWXF7YVH2COEW4WH3YWDLU3VJMA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:37.783437003 (294h8m12.554078338s) [noise=true , address=77.90.31.17:4771 , piece_id=V7M65ZHCGXPD2GVJL6LTB4NP3QN6QLSIBM7U34QAYPAIKMN6EBPA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:39.733398290 (294h8m14.504039626s) [noise=true , address=77.90.31.18:4805 , piece_id=I23F6KYUQ7TJI4ZLEANACVNYJDMPKKMMACIZAIA6PC5S3CKQPV2Q, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Jul 5 15:18:39.733464387 (294h8m14.504105722s) [noise=true , address=77.90.31.21:4877 , piece_id=OJBIFN2TJPQ45LQLYCMPKV5UZM6Y2MXNOBT3TSGX26U4SZAL5Q7A, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: expected piece hash; context deadline exceeded; EOF Jul 5 15:18:39.733542065 (294h8m14.50418339s) [noise=true , address=77.90.31.11:4554 , piece_id=4QRZ5MPQ7HTPRIVRL3O3H4OV7CQA7ESYT5RCPI3GTHXJOHOSAOVA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection Jul 5 15:18:39.733547496 (294h8m14.504188831s) [noise=true , address=77.90.31.16:4733 , piece_id=3GQ7NEWE6WZJMQUA3GG3GSZYXEFNOAUEVWHOEH4TSNONYXUT43GQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection; write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection Jul 5 15:18:39.733583284 (294h8m14.504224619s) [seg_pos={0 0}, upload=235227]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 5 15:18:39.974321220 (294h8m14.744962555s) [upload=235227]: Done uploading segments... Jul 5 15:18:39.974353041 (294h8m14.744994376s) [upload=235227]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.664822246s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 5.773752479s FAIL (2025-07-05 15:32:00 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== JYE6FC5T5QIBUQIILGDHNGXAUDBUIAOSKYFMPBCWPHP2SDJ3A, seg_pos={0 0}, upload=235214]: Done uploading piece. err: <nil> Jul 5 15:17:12.446386719 (294h6m47.217028064s) [seg_pos={0 0}, upload=235214]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 5 15:17:12.446476771 (294h6m47.217118106s) [upload=235214]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 5 15:17:12.446483043 (294h6m47.217124378s) [upload=235214]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 5 15:17:12.905279085 (294h6m47.67592042s) [upload=235214]: Done uploading segments... Jul 5 15:17:12.905303211 (294h6m47.675944546s) [upload=235214]: Upload finished. err: <nil> Jul 5 15:18:10.381454266 (294h7m45.152095611s) [upload=235227]: Starting upload Jul 5 15:18:10.381524069 (294h7m45.152165414s) [upload=235227]: Done starting upload Jul 5 15:18:10.381621115 (294h7m45.15226245s) [upload=235227]: Uploading segments... Jul 5 15:18:10.381740133 (294h7m45.152381468s) [upload=235227]: Got next segment. Inline: false Jul 5 15:18:10.381776342 (294h7m45.152417677s) [upload=235227]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 5 15:18:10.381781973 (294h7m45.152423308s) [upload=235227]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 5 15:18:10.613904835 (294h7m45.38454617s) [seg_pos={0 0}, upload=235227]: Begin upload segment... Jul 5 15:18:10.613947397 (294h7m45.384588732s) [seg_pos={0 0}, upload=235227]: Done begin upload segment. Jul 5 15:18:10.613968548 (294h7m45.384609883s) [noise=true , address=104.36.82.15:4259 , piece_id=LQKQ5MGKXHTPITBR2QAOQ5CX76GEVJGHJT3ZD3G4PFNN5IBTCJBQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.613995038 (294h7m45.384636373s) [noise=true , address=104.36.82.16:4295 , piece_id=VEL6CAUOJKMRN7A7M66ZPDLCMJVD555LZPHM53A24MDQJQOW7ECQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614303969 (294h7m45.384945304s) [noise=true , address=104.36.82.9:4006 , piece_id=47FME2B5NS6WMOHC2ZGE2FZVSPEQUXKYT6E7AOSANAE5XZVDH5VQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614367721 (294h7m45.385009056s) [noise=true , address=45.91.116.4:4937 , piece_id=YMKIHJET2EN3QHEK665ACZNQNHJ3UGE4H5T4DIHWJUHJQBYO7B7Q, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614619272 (294h7m45.385260607s) [noise=true , address=77.90.31.17:4771 , piece_id=V7M65ZHCGXPD2GVJL6LTB4NP3QN6QLSIBM7U34QAYPAIKMN6EBPA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614637758 (294h7m45.385279083s) [noise=true , address=45.91.116.14:4191 , piece_id=WH2BCI36GGFF4QUD4XDYBBDMUR2R2FFVYKFTZD3ZWXPJCFJ3D7HQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614890251 (294h7m45.385531576s) [noise=true , address=104.36.82.3:4767 , piece_id=OGHG4G5ZVC7GVY4XI6DO4KINXF7THHL6IHGDB6DAPDNUDOUFSC4A, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615197719 (294h7m45.385839054s) [noise=true , address=104.36.82.12:4149 , piece_id=YBTLQEWDMFK6JIFTYR3VJDNMXTS3NMWWHX2MPP53BM2UBDI3UUEA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615413552 (294h7m45.386054887s) [noise=true , address=77.90.31.18:4805 , piece_id=I23F6KYUQ7TJI4ZLEANACVNYJDMPKKMMACIZAIA6PC5S3CKQPV2Q, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615608124 (294h7m45.386249459s) [noise=true , address=77.90.31.21:4877 , piece_id=OJBIFN2TJPQ45LQLYCMPKV5UZM6Y2MXNOBT3TSGX26U4SZAL5Q7A, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615803147 (294h7m45.386444472s) [noise=true , address=104.36.82.8:4970 , piece_id=2YJN7VKTTTCMAWQW2MNR65DAJPFMHNIBIW5NWYNG4R6HNPRFH6NQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615998511 (294h7m45.386639836s) [noise=true , address=77.90.31.16:4733 , piece_id=3GQ7NEWE6WZJMQUA3GG3GSZYXEFNOAUEVWHOEH4TSNONYXUT43GQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616193414 (294h7m45.386834749s) [noise=true , address=77.90.31.19:4842 , piece_id=R4FTRBZ3PETCA6QGU6RPUEDZZBWXF7YVH2COEW4WH3YWDLU3VJMA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616449093 (294h7m45.387090429s) [noise=true , address=77.90.31.11:4554 , piece_id=4QRZ5MPQ7HTPRIVRL3O3H4OV7CQA7ESYT5RCPI3GTHXJOHOSAOVA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616635840 (294h7m45.387277175s) [noise=true , address=45.91.116.5:4298 , piece_id=TVTUGWHFTXB32GBY4LYXT3YDB5LIFXEGA7VO5BHFA24Y6EMDTPHQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616818790 (294h7m45.387460125s) [noise=true , address=104.36.82.13:4185 , piece_id=LTQS2CUZ5NRJCBOPEXDKUQ2IJGOIW3JJAPHWTCW7WUMQZPIWRXRA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:12.265123988 (294h7m47.035765313s) [upload=235227]: Next returned nil segment Jul 5 15:18:12.265138164 (294h7m47.035779499s) [upload=235227]: Waiting for error group managing segments... Jul 5 15:18:12.550465694 (294h7m47.321107019s) [noise=true , address=104.36.82.8:4970 , piece_id=2YJN7VKTTTCMAWQW2MNR65DAJPFMHNIBIW5NWYNG4R6HNPRFH6NQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.554315629 (294h7m47.324956954s) [noise=true , address=104.36.82.12:4149 , piece_id=YBTLQEWDMFK6JIFTYR3VJDNMXTS3NMWWHX2MPP53BM2UBDI3UUEA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.603452512 (294h7m47.374093847s) [noise=true , address=104.36.82.3:4767 , piece_id=OGHG4G5ZVC7GVY4XI6DO4KINXF7THHL6IHGDB6DAPDNUDOUFSC4A, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.639796643 (294h7m47.410437978s) [noise=true , address=104.36.82.9:4006 , piece_id=47FME2B5NS6WMOHC2ZGE2FZVSPEQUXKYT6E7AOSANAE5XZVDH5VQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.639894960 (294h7m47.410536285s) [noise=true , address=104.36.82.15:4259 , piece_id=LQKQ5MGKXHTPITBR2QAOQ5CX76GEVJGHJT3ZD3G4PFNN5IBTCJBQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.647222659 (294h7m47.417863994s) [noise=true , address=104.36.82.16:4295 , piece_id=VEL6CAUOJKMRN7A7M66ZPDLCMJVD555LZPHM53A24MDQJQOW7ECQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.665424010 (294h7m47.436065346s) [noise=true , address=104.36.82.13:4185 , piece_id=LTQS2CUZ5NRJCBOPEXDKUQ2IJGOIW3JJAPHWTCW7WUMQZPIWRXRA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.702209574 (294h7m47.47285091s) [noise=true , address=45.91.116.14:4191 , piece_id=WH2BCI36GGFF4QUD4XDYBBDMUR2R2FFVYKFTZD3ZWXPJCFJ3D7HQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.722073145 (294h7m47.49271448s) [noise=true , address=45.91.116.5:4298 , piece_id=TVTUGWHFTXB32GBY4LYXT3YDB5LIFXEGA7VO5BHFA24Y6EMDTPHQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.788621529 (294h7m47.559262864s) [noise=true , address=45.91.116.4:4937 , piece_id=YMKIHJET2EN3QHEK665ACZNQNHJ3UGE4H5T4DIHWJUHJQBYO7B7Q, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:36.752254721 (294h8m11.522896046s) [noise=true , address=77.90.31.19:4842 , piece_id=R4FTRBZ3PETCA6QGU6RPUEDZZBWXF7YVH2COEW4WH3YWDLU3VJMA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:37.783437003 (294h8m12.554078338s) [noise=true , address=77.90.31.17:4771 , piece_id=V7M65ZHCGXPD2GVJL6LTB4NP3QN6QLSIBM7U34QAYPAIKMN6EBPA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:39.733398290 (294h8m14.504039626s) [noise=true , address=77.90.31.18:4805 , piece_id=I23F6KYUQ7TJI4ZLEANACVNYJDMPKKMMACIZAIA6PC5S3CKQPV2Q, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Jul 5 15:18:39.733464387 (294h8m14.504105722s) [noise=true , address=77.90.31.21:4877 , piece_id=OJBIFN2TJPQ45LQLYCMPKV5UZM6Y2MXNOBT3TSGX26U4SZAL5Q7A, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: expected piece hash; context deadline exceeded; EOF Jul 5 15:18:39.733542065 (294h8m14.50418339s) [noise=true , address=77.90.31.11:4554 , piece_id=4QRZ5MPQ7HTPRIVRL3O3H4OV7CQA7ESYT5RCPI3GTHXJOHOSAOVA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection Jul 5 15:18:39.733547496 (294h8m14.504188831s) [noise=true , address=77.90.31.16:4733 , piece_id=3GQ7NEWE6WZJMQUA3GG3GSZYXEFNOAUEVWHOEH4TSNONYXUT43GQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection; write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection Jul 5 15:18:39.733583284 (294h8m14.504224619s) [seg_pos={0 0}, upload=235227]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 5 15:18:39.974321220 (294h8m14.744962555s) [upload=235227]: Done uploading segments... Jul 5 15:18:39.974353041 (294h8m14.744994376s) [upload=235227]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.773781204s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 5.671639495s FAIL (2025-07-05 15:31:26 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== JYE6FC5T5QIBUQIILGDHNGXAUDBUIAOSKYFMPBCWPHP2SDJ3A, seg_pos={0 0}, upload=235214]: Done uploading piece. err: <nil> Jul 5 15:17:12.446386719 (294h6m47.217028064s) [seg_pos={0 0}, upload=235214]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 5 15:17:12.446476771 (294h6m47.217118106s) [upload=235214]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 5 15:17:12.446483043 (294h6m47.217124378s) [upload=235214]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 5 15:17:12.905279085 (294h6m47.67592042s) [upload=235214]: Done uploading segments... Jul 5 15:17:12.905303211 (294h6m47.675944546s) [upload=235214]: Upload finished. err: <nil> Jul 5 15:18:10.381454266 (294h7m45.152095611s) [upload=235227]: Starting upload Jul 5 15:18:10.381524069 (294h7m45.152165414s) [upload=235227]: Done starting upload Jul 5 15:18:10.381621115 (294h7m45.15226245s) [upload=235227]: Uploading segments... Jul 5 15:18:10.381740133 (294h7m45.152381468s) [upload=235227]: Got next segment. Inline: false Jul 5 15:18:10.381776342 (294h7m45.152417677s) [upload=235227]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 5 15:18:10.381781973 (294h7m45.152423308s) [upload=235227]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 5 15:18:10.613904835 (294h7m45.38454617s) [seg_pos={0 0}, upload=235227]: Begin upload segment... Jul 5 15:18:10.613947397 (294h7m45.384588732s) [seg_pos={0 0}, upload=235227]: Done begin upload segment. Jul 5 15:18:10.613968548 (294h7m45.384609883s) [noise=true , address=104.36.82.15:4259 , piece_id=LQKQ5MGKXHTPITBR2QAOQ5CX76GEVJGHJT3ZD3G4PFNN5IBTCJBQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.613995038 (294h7m45.384636373s) [noise=true , address=104.36.82.16:4295 , piece_id=VEL6CAUOJKMRN7A7M66ZPDLCMJVD555LZPHM53A24MDQJQOW7ECQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614303969 (294h7m45.384945304s) [noise=true , address=104.36.82.9:4006 , piece_id=47FME2B5NS6WMOHC2ZGE2FZVSPEQUXKYT6E7AOSANAE5XZVDH5VQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614367721 (294h7m45.385009056s) [noise=true , address=45.91.116.4:4937 , piece_id=YMKIHJET2EN3QHEK665ACZNQNHJ3UGE4H5T4DIHWJUHJQBYO7B7Q, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614619272 (294h7m45.385260607s) [noise=true , address=77.90.31.17:4771 , piece_id=V7M65ZHCGXPD2GVJL6LTB4NP3QN6QLSIBM7U34QAYPAIKMN6EBPA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614637758 (294h7m45.385279083s) [noise=true , address=45.91.116.14:4191 , piece_id=WH2BCI36GGFF4QUD4XDYBBDMUR2R2FFVYKFTZD3ZWXPJCFJ3D7HQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614890251 (294h7m45.385531576s) [noise=true , address=104.36.82.3:4767 , piece_id=OGHG4G5ZVC7GVY4XI6DO4KINXF7THHL6IHGDB6DAPDNUDOUFSC4A, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615197719 (294h7m45.385839054s) [noise=true , address=104.36.82.12:4149 , piece_id=YBTLQEWDMFK6JIFTYR3VJDNMXTS3NMWWHX2MPP53BM2UBDI3UUEA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615413552 (294h7m45.386054887s) [noise=true , address=77.90.31.18:4805 , piece_id=I23F6KYUQ7TJI4ZLEANACVNYJDMPKKMMACIZAIA6PC5S3CKQPV2Q, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615608124 (294h7m45.386249459s) [noise=true , address=77.90.31.21:4877 , piece_id=OJBIFN2TJPQ45LQLYCMPKV5UZM6Y2MXNOBT3TSGX26U4SZAL5Q7A, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615803147 (294h7m45.386444472s) [noise=true , address=104.36.82.8:4970 , piece_id=2YJN7VKTTTCMAWQW2MNR65DAJPFMHNIBIW5NWYNG4R6HNPRFH6NQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615998511 (294h7m45.386639836s) [noise=true , address=77.90.31.16:4733 , piece_id=3GQ7NEWE6WZJMQUA3GG3GSZYXEFNOAUEVWHOEH4TSNONYXUT43GQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616193414 (294h7m45.386834749s) [noise=true , address=77.90.31.19:4842 , piece_id=R4FTRBZ3PETCA6QGU6RPUEDZZBWXF7YVH2COEW4WH3YWDLU3VJMA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616449093 (294h7m45.387090429s) [noise=true , address=77.90.31.11:4554 , piece_id=4QRZ5MPQ7HTPRIVRL3O3H4OV7CQA7ESYT5RCPI3GTHXJOHOSAOVA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616635840 (294h7m45.387277175s) [noise=true , address=45.91.116.5:4298 , piece_id=TVTUGWHFTXB32GBY4LYXT3YDB5LIFXEGA7VO5BHFA24Y6EMDTPHQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616818790 (294h7m45.387460125s) [noise=true , address=104.36.82.13:4185 , piece_id=LTQS2CUZ5NRJCBOPEXDKUQ2IJGOIW3JJAPHWTCW7WUMQZPIWRXRA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:12.265123988 (294h7m47.035765313s) [upload=235227]: Next returned nil segment Jul 5 15:18:12.265138164 (294h7m47.035779499s) [upload=235227]: Waiting for error group managing segments... Jul 5 15:18:12.550465694 (294h7m47.321107019s) [noise=true , address=104.36.82.8:4970 , piece_id=2YJN7VKTTTCMAWQW2MNR65DAJPFMHNIBIW5NWYNG4R6HNPRFH6NQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.554315629 (294h7m47.324956954s) [noise=true , address=104.36.82.12:4149 , piece_id=YBTLQEWDMFK6JIFTYR3VJDNMXTS3NMWWHX2MPP53BM2UBDI3UUEA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.603452512 (294h7m47.374093847s) [noise=true , address=104.36.82.3:4767 , piece_id=OGHG4G5ZVC7GVY4XI6DO4KINXF7THHL6IHGDB6DAPDNUDOUFSC4A, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.639796643 (294h7m47.410437978s) [noise=true , address=104.36.82.9:4006 , piece_id=47FME2B5NS6WMOHC2ZGE2FZVSPEQUXKYT6E7AOSANAE5XZVDH5VQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.639894960 (294h7m47.410536285s) [noise=true , address=104.36.82.15:4259 , piece_id=LQKQ5MGKXHTPITBR2QAOQ5CX76GEVJGHJT3ZD3G4PFNN5IBTCJBQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.647222659 (294h7m47.417863994s) [noise=true , address=104.36.82.16:4295 , piece_id=VEL6CAUOJKMRN7A7M66ZPDLCMJVD555LZPHM53A24MDQJQOW7ECQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.665424010 (294h7m47.436065346s) [noise=true , address=104.36.82.13:4185 , piece_id=LTQS2CUZ5NRJCBOPEXDKUQ2IJGOIW3JJAPHWTCW7WUMQZPIWRXRA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.702209574 (294h7m47.47285091s) [noise=true , address=45.91.116.14:4191 , piece_id=WH2BCI36GGFF4QUD4XDYBBDMUR2R2FFVYKFTZD3ZWXPJCFJ3D7HQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.722073145 (294h7m47.49271448s) [noise=true , address=45.91.116.5:4298 , piece_id=TVTUGWHFTXB32GBY4LYXT3YDB5LIFXEGA7VO5BHFA24Y6EMDTPHQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.788621529 (294h7m47.559262864s) [noise=true , address=45.91.116.4:4937 , piece_id=YMKIHJET2EN3QHEK665ACZNQNHJ3UGE4H5T4DIHWJUHJQBYO7B7Q, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:36.752254721 (294h8m11.522896046s) [noise=true , address=77.90.31.19:4842 , piece_id=R4FTRBZ3PETCA6QGU6RPUEDZZBWXF7YVH2COEW4WH3YWDLU3VJMA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:37.783437003 (294h8m12.554078338s) [noise=true , address=77.90.31.17:4771 , piece_id=V7M65ZHCGXPD2GVJL6LTB4NP3QN6QLSIBM7U34QAYPAIKMN6EBPA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:39.733398290 (294h8m14.504039626s) [noise=true , address=77.90.31.18:4805 , piece_id=I23F6KYUQ7TJI4ZLEANACVNYJDMPKKMMACIZAIA6PC5S3CKQPV2Q, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Jul 5 15:18:39.733464387 (294h8m14.504105722s) [noise=true , address=77.90.31.21:4877 , piece_id=OJBIFN2TJPQ45LQLYCMPKV5UZM6Y2MXNOBT3TSGX26U4SZAL5Q7A, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: expected piece hash; context deadline exceeded; EOF Jul 5 15:18:39.733542065 (294h8m14.50418339s) [noise=true , address=77.90.31.11:4554 , piece_id=4QRZ5MPQ7HTPRIVRL3O3H4OV7CQA7ESYT5RCPI3GTHXJOHOSAOVA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection Jul 5 15:18:39.733547496 (294h8m14.504188831s) [noise=true , address=77.90.31.16:4733 , piece_id=3GQ7NEWE6WZJMQUA3GG3GSZYXEFNOAUEVWHOEH4TSNONYXUT43GQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection; write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection Jul 5 15:18:39.733583284 (294h8m14.504224619s) [seg_pos={0 0}, upload=235227]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 5 15:18:39.974321220 (294h8m14.744962555s) [upload=235227]: Done uploading segments... Jul 5 15:18:39.974353041 (294h8m14.744994376s) [upload=235227]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.671680052s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 5.786413682s FAIL (2025-07-05 15:30:16 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== JYE6FC5T5QIBUQIILGDHNGXAUDBUIAOSKYFMPBCWPHP2SDJ3A, seg_pos={0 0}, upload=235214]: Done uploading piece. err: <nil> Jul 5 15:17:12.446386719 (294h6m47.217028064s) [seg_pos={0 0}, upload=235214]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 5 15:17:12.446476771 (294h6m47.217118106s) [upload=235214]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 5 15:17:12.446483043 (294h6m47.217124378s) [upload=235214]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 5 15:17:12.905279085 (294h6m47.67592042s) [upload=235214]: Done uploading segments... Jul 5 15:17:12.905303211 (294h6m47.675944546s) [upload=235214]: Upload finished. err: <nil> Jul 5 15:18:10.381454266 (294h7m45.152095611s) [upload=235227]: Starting upload Jul 5 15:18:10.381524069 (294h7m45.152165414s) [upload=235227]: Done starting upload Jul 5 15:18:10.381621115 (294h7m45.15226245s) [upload=235227]: Uploading segments... Jul 5 15:18:10.381740133 (294h7m45.152381468s) [upload=235227]: Got next segment. Inline: false Jul 5 15:18:10.381776342 (294h7m45.152417677s) [upload=235227]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 5 15:18:10.381781973 (294h7m45.152423308s) [upload=235227]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 5 15:18:10.613904835 (294h7m45.38454617s) [seg_pos={0 0}, upload=235227]: Begin upload segment... Jul 5 15:18:10.613947397 (294h7m45.384588732s) [seg_pos={0 0}, upload=235227]: Done begin upload segment. Jul 5 15:18:10.613968548 (294h7m45.384609883s) [noise=true , address=104.36.82.15:4259 , piece_id=LQKQ5MGKXHTPITBR2QAOQ5CX76GEVJGHJT3ZD3G4PFNN5IBTCJBQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.613995038 (294h7m45.384636373s) [noise=true , address=104.36.82.16:4295 , piece_id=VEL6CAUOJKMRN7A7M66ZPDLCMJVD555LZPHM53A24MDQJQOW7ECQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614303969 (294h7m45.384945304s) [noise=true , address=104.36.82.9:4006 , piece_id=47FME2B5NS6WMOHC2ZGE2FZVSPEQUXKYT6E7AOSANAE5XZVDH5VQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614367721 (294h7m45.385009056s) [noise=true , address=45.91.116.4:4937 , piece_id=YMKIHJET2EN3QHEK665ACZNQNHJ3UGE4H5T4DIHWJUHJQBYO7B7Q, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614619272 (294h7m45.385260607s) [noise=true , address=77.90.31.17:4771 , piece_id=V7M65ZHCGXPD2GVJL6LTB4NP3QN6QLSIBM7U34QAYPAIKMN6EBPA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614637758 (294h7m45.385279083s) [noise=true , address=45.91.116.14:4191 , piece_id=WH2BCI36GGFF4QUD4XDYBBDMUR2R2FFVYKFTZD3ZWXPJCFJ3D7HQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614890251 (294h7m45.385531576s) [noise=true , address=104.36.82.3:4767 , piece_id=OGHG4G5ZVC7GVY4XI6DO4KINXF7THHL6IHGDB6DAPDNUDOUFSC4A, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615197719 (294h7m45.385839054s) [noise=true , address=104.36.82.12:4149 , piece_id=YBTLQEWDMFK6JIFTYR3VJDNMXTS3NMWWHX2MPP53BM2UBDI3UUEA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615413552 (294h7m45.386054887s) [noise=true , address=77.90.31.18:4805 , piece_id=I23F6KYUQ7TJI4ZLEANACVNYJDMPKKMMACIZAIA6PC5S3CKQPV2Q, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615608124 (294h7m45.386249459s) [noise=true , address=77.90.31.21:4877 , piece_id=OJBIFN2TJPQ45LQLYCMPKV5UZM6Y2MXNOBT3TSGX26U4SZAL5Q7A, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615803147 (294h7m45.386444472s) [noise=true , address=104.36.82.8:4970 , piece_id=2YJN7VKTTTCMAWQW2MNR65DAJPFMHNIBIW5NWYNG4R6HNPRFH6NQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615998511 (294h7m45.386639836s) [noise=true , address=77.90.31.16:4733 , piece_id=3GQ7NEWE6WZJMQUA3GG3GSZYXEFNOAUEVWHOEH4TSNONYXUT43GQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616193414 (294h7m45.386834749s) [noise=true , address=77.90.31.19:4842 , piece_id=R4FTRBZ3PETCA6QGU6RPUEDZZBWXF7YVH2COEW4WH3YWDLU3VJMA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616449093 (294h7m45.387090429s) [noise=true , address=77.90.31.11:4554 , piece_id=4QRZ5MPQ7HTPRIVRL3O3H4OV7CQA7ESYT5RCPI3GTHXJOHOSAOVA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616635840 (294h7m45.387277175s) [noise=true , address=45.91.116.5:4298 , piece_id=TVTUGWHFTXB32GBY4LYXT3YDB5LIFXEGA7VO5BHFA24Y6EMDTPHQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616818790 (294h7m45.387460125s) [noise=true , address=104.36.82.13:4185 , piece_id=LTQS2CUZ5NRJCBOPEXDKUQ2IJGOIW3JJAPHWTCW7WUMQZPIWRXRA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:12.265123988 (294h7m47.035765313s) [upload=235227]: Next returned nil segment Jul 5 15:18:12.265138164 (294h7m47.035779499s) [upload=235227]: Waiting for error group managing segments... Jul 5 15:18:12.550465694 (294h7m47.321107019s) [noise=true , address=104.36.82.8:4970 , piece_id=2YJN7VKTTTCMAWQW2MNR65DAJPFMHNIBIW5NWYNG4R6HNPRFH6NQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.554315629 (294h7m47.324956954s) [noise=true , address=104.36.82.12:4149 , piece_id=YBTLQEWDMFK6JIFTYR3VJDNMXTS3NMWWHX2MPP53BM2UBDI3UUEA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.603452512 (294h7m47.374093847s) [noise=true , address=104.36.82.3:4767 , piece_id=OGHG4G5ZVC7GVY4XI6DO4KINXF7THHL6IHGDB6DAPDNUDOUFSC4A, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.639796643 (294h7m47.410437978s) [noise=true , address=104.36.82.9:4006 , piece_id=47FME2B5NS6WMOHC2ZGE2FZVSPEQUXKYT6E7AOSANAE5XZVDH5VQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.639894960 (294h7m47.410536285s) [noise=true , address=104.36.82.15:4259 , piece_id=LQKQ5MGKXHTPITBR2QAOQ5CX76GEVJGHJT3ZD3G4PFNN5IBTCJBQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.647222659 (294h7m47.417863994s) [noise=true , address=104.36.82.16:4295 , piece_id=VEL6CAUOJKMRN7A7M66ZPDLCMJVD555LZPHM53A24MDQJQOW7ECQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.665424010 (294h7m47.436065346s) [noise=true , address=104.36.82.13:4185 , piece_id=LTQS2CUZ5NRJCBOPEXDKUQ2IJGOIW3JJAPHWTCW7WUMQZPIWRXRA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.702209574 (294h7m47.47285091s) [noise=true , address=45.91.116.14:4191 , piece_id=WH2BCI36GGFF4QUD4XDYBBDMUR2R2FFVYKFTZD3ZWXPJCFJ3D7HQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.722073145 (294h7m47.49271448s) [noise=true , address=45.91.116.5:4298 , piece_id=TVTUGWHFTXB32GBY4LYXT3YDB5LIFXEGA7VO5BHFA24Y6EMDTPHQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.788621529 (294h7m47.559262864s) [noise=true , address=45.91.116.4:4937 , piece_id=YMKIHJET2EN3QHEK665ACZNQNHJ3UGE4H5T4DIHWJUHJQBYO7B7Q, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:36.752254721 (294h8m11.522896046s) [noise=true , address=77.90.31.19:4842 , piece_id=R4FTRBZ3PETCA6QGU6RPUEDZZBWXF7YVH2COEW4WH3YWDLU3VJMA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:37.783437003 (294h8m12.554078338s) [noise=true , address=77.90.31.17:4771 , piece_id=V7M65ZHCGXPD2GVJL6LTB4NP3QN6QLSIBM7U34QAYPAIKMN6EBPA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:39.733398290 (294h8m14.504039626s) [noise=true , address=77.90.31.18:4805 , piece_id=I23F6KYUQ7TJI4ZLEANACVNYJDMPKKMMACIZAIA6PC5S3CKQPV2Q, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Jul 5 15:18:39.733464387 (294h8m14.504105722s) [noise=true , address=77.90.31.21:4877 , piece_id=OJBIFN2TJPQ45LQLYCMPKV5UZM6Y2MXNOBT3TSGX26U4SZAL5Q7A, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: expected piece hash; context deadline exceeded; EOF Jul 5 15:18:39.733542065 (294h8m14.50418339s) [noise=true , address=77.90.31.11:4554 , piece_id=4QRZ5MPQ7HTPRIVRL3O3H4OV7CQA7ESYT5RCPI3GTHXJOHOSAOVA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection Jul 5 15:18:39.733547496 (294h8m14.504188831s) [noise=true , address=77.90.31.16:4733 , piece_id=3GQ7NEWE6WZJMQUA3GG3GSZYXEFNOAUEVWHOEH4TSNONYXUT43GQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection; write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection Jul 5 15:18:39.733583284 (294h8m14.504224619s) [seg_pos={0 0}, upload=235227]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 5 15:18:39.974321220 (294h8m14.744962555s) [upload=235227]: Done uploading segments... Jul 5 15:18:39.974353041 (294h8m14.744994376s) [upload=235227]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.786436666s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 5.656734009s FAIL (2025-07-05 15:29:30 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== JYE6FC5T5QIBUQIILGDHNGXAUDBUIAOSKYFMPBCWPHP2SDJ3A, seg_pos={0 0}, upload=235214]: Done uploading piece. err: <nil> Jul 5 15:17:12.446386719 (294h6m47.217028064s) [seg_pos={0 0}, upload=235214]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 5 15:17:12.446476771 (294h6m47.217118106s) [upload=235214]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 5 15:17:12.446483043 (294h6m47.217124378s) [upload=235214]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 5 15:17:12.905279085 (294h6m47.67592042s) [upload=235214]: Done uploading segments... Jul 5 15:17:12.905303211 (294h6m47.675944546s) [upload=235214]: Upload finished. err: <nil> Jul 5 15:18:10.381454266 (294h7m45.152095611s) [upload=235227]: Starting upload Jul 5 15:18:10.381524069 (294h7m45.152165414s) [upload=235227]: Done starting upload Jul 5 15:18:10.381621115 (294h7m45.15226245s) [upload=235227]: Uploading segments... Jul 5 15:18:10.381740133 (294h7m45.152381468s) [upload=235227]: Got next segment. Inline: false Jul 5 15:18:10.381776342 (294h7m45.152417677s) [upload=235227]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 5 15:18:10.381781973 (294h7m45.152423308s) [upload=235227]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 5 15:18:10.613904835 (294h7m45.38454617s) [seg_pos={0 0}, upload=235227]: Begin upload segment... Jul 5 15:18:10.613947397 (294h7m45.384588732s) [seg_pos={0 0}, upload=235227]: Done begin upload segment. Jul 5 15:18:10.613968548 (294h7m45.384609883s) [noise=true , address=104.36.82.15:4259 , piece_id=LQKQ5MGKXHTPITBR2QAOQ5CX76GEVJGHJT3ZD3G4PFNN5IBTCJBQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.613995038 (294h7m45.384636373s) [noise=true , address=104.36.82.16:4295 , piece_id=VEL6CAUOJKMRN7A7M66ZPDLCMJVD555LZPHM53A24MDQJQOW7ECQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614303969 (294h7m45.384945304s) [noise=true , address=104.36.82.9:4006 , piece_id=47FME2B5NS6WMOHC2ZGE2FZVSPEQUXKYT6E7AOSANAE5XZVDH5VQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614367721 (294h7m45.385009056s) [noise=true , address=45.91.116.4:4937 , piece_id=YMKIHJET2EN3QHEK665ACZNQNHJ3UGE4H5T4DIHWJUHJQBYO7B7Q, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614619272 (294h7m45.385260607s) [noise=true , address=77.90.31.17:4771 , piece_id=V7M65ZHCGXPD2GVJL6LTB4NP3QN6QLSIBM7U34QAYPAIKMN6EBPA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614637758 (294h7m45.385279083s) [noise=true , address=45.91.116.14:4191 , piece_id=WH2BCI36GGFF4QUD4XDYBBDMUR2R2FFVYKFTZD3ZWXPJCFJ3D7HQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614890251 (294h7m45.385531576s) [noise=true , address=104.36.82.3:4767 , piece_id=OGHG4G5ZVC7GVY4XI6DO4KINXF7THHL6IHGDB6DAPDNUDOUFSC4A, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615197719 (294h7m45.385839054s) [noise=true , address=104.36.82.12:4149 , piece_id=YBTLQEWDMFK6JIFTYR3VJDNMXTS3NMWWHX2MPP53BM2UBDI3UUEA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615413552 (294h7m45.386054887s) [noise=true , address=77.90.31.18:4805 , piece_id=I23F6KYUQ7TJI4ZLEANACVNYJDMPKKMMACIZAIA6PC5S3CKQPV2Q, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615608124 (294h7m45.386249459s) [noise=true , address=77.90.31.21:4877 , piece_id=OJBIFN2TJPQ45LQLYCMPKV5UZM6Y2MXNOBT3TSGX26U4SZAL5Q7A, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615803147 (294h7m45.386444472s) [noise=true , address=104.36.82.8:4970 , piece_id=2YJN7VKTTTCMAWQW2MNR65DAJPFMHNIBIW5NWYNG4R6HNPRFH6NQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615998511 (294h7m45.386639836s) [noise=true , address=77.90.31.16:4733 , piece_id=3GQ7NEWE6WZJMQUA3GG3GSZYXEFNOAUEVWHOEH4TSNONYXUT43GQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616193414 (294h7m45.386834749s) [noise=true , address=77.90.31.19:4842 , piece_id=R4FTRBZ3PETCA6QGU6RPUEDZZBWXF7YVH2COEW4WH3YWDLU3VJMA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616449093 (294h7m45.387090429s) [noise=true , address=77.90.31.11:4554 , piece_id=4QRZ5MPQ7HTPRIVRL3O3H4OV7CQA7ESYT5RCPI3GTHXJOHOSAOVA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616635840 (294h7m45.387277175s) [noise=true , address=45.91.116.5:4298 , piece_id=TVTUGWHFTXB32GBY4LYXT3YDB5LIFXEGA7VO5BHFA24Y6EMDTPHQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616818790 (294h7m45.387460125s) [noise=true , address=104.36.82.13:4185 , piece_id=LTQS2CUZ5NRJCBOPEXDKUQ2IJGOIW3JJAPHWTCW7WUMQZPIWRXRA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:12.265123988 (294h7m47.035765313s) [upload=235227]: Next returned nil segment Jul 5 15:18:12.265138164 (294h7m47.035779499s) [upload=235227]: Waiting for error group managing segments... Jul 5 15:18:12.550465694 (294h7m47.321107019s) [noise=true , address=104.36.82.8:4970 , piece_id=2YJN7VKTTTCMAWQW2MNR65DAJPFMHNIBIW5NWYNG4R6HNPRFH6NQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.554315629 (294h7m47.324956954s) [noise=true , address=104.36.82.12:4149 , piece_id=YBTLQEWDMFK6JIFTYR3VJDNMXTS3NMWWHX2MPP53BM2UBDI3UUEA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.603452512 (294h7m47.374093847s) [noise=true , address=104.36.82.3:4767 , piece_id=OGHG4G5ZVC7GVY4XI6DO4KINXF7THHL6IHGDB6DAPDNUDOUFSC4A, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.639796643 (294h7m47.410437978s) [noise=true , address=104.36.82.9:4006 , piece_id=47FME2B5NS6WMOHC2ZGE2FZVSPEQUXKYT6E7AOSANAE5XZVDH5VQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.639894960 (294h7m47.410536285s) [noise=true , address=104.36.82.15:4259 , piece_id=LQKQ5MGKXHTPITBR2QAOQ5CX76GEVJGHJT3ZD3G4PFNN5IBTCJBQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.647222659 (294h7m47.417863994s) [noise=true , address=104.36.82.16:4295 , piece_id=VEL6CAUOJKMRN7A7M66ZPDLCMJVD555LZPHM53A24MDQJQOW7ECQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.665424010 (294h7m47.436065346s) [noise=true , address=104.36.82.13:4185 , piece_id=LTQS2CUZ5NRJCBOPEXDKUQ2IJGOIW3JJAPHWTCW7WUMQZPIWRXRA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.702209574 (294h7m47.47285091s) [noise=true , address=45.91.116.14:4191 , piece_id=WH2BCI36GGFF4QUD4XDYBBDMUR2R2FFVYKFTZD3ZWXPJCFJ3D7HQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.722073145 (294h7m47.49271448s) [noise=true , address=45.91.116.5:4298 , piece_id=TVTUGWHFTXB32GBY4LYXT3YDB5LIFXEGA7VO5BHFA24Y6EMDTPHQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.788621529 (294h7m47.559262864s) [noise=true , address=45.91.116.4:4937 , piece_id=YMKIHJET2EN3QHEK665ACZNQNHJ3UGE4H5T4DIHWJUHJQBYO7B7Q, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:36.752254721 (294h8m11.522896046s) [noise=true , address=77.90.31.19:4842 , piece_id=R4FTRBZ3PETCA6QGU6RPUEDZZBWXF7YVH2COEW4WH3YWDLU3VJMA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:37.783437003 (294h8m12.554078338s) [noise=true , address=77.90.31.17:4771 , piece_id=V7M65ZHCGXPD2GVJL6LTB4NP3QN6QLSIBM7U34QAYPAIKMN6EBPA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:39.733398290 (294h8m14.504039626s) [noise=true , address=77.90.31.18:4805 , piece_id=I23F6KYUQ7TJI4ZLEANACVNYJDMPKKMMACIZAIA6PC5S3CKQPV2Q, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Jul 5 15:18:39.733464387 (294h8m14.504105722s) [noise=true , address=77.90.31.21:4877 , piece_id=OJBIFN2TJPQ45LQLYCMPKV5UZM6Y2MXNOBT3TSGX26U4SZAL5Q7A, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: expected piece hash; context deadline exceeded; EOF Jul 5 15:18:39.733542065 (294h8m14.50418339s) [noise=true , address=77.90.31.11:4554 , piece_id=4QRZ5MPQ7HTPRIVRL3O3H4OV7CQA7ESYT5RCPI3GTHXJOHOSAOVA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection Jul 5 15:18:39.733547496 (294h8m14.504188831s) [noise=true , address=77.90.31.16:4733 , piece_id=3GQ7NEWE6WZJMQUA3GG3GSZYXEFNOAUEVWHOEH4TSNONYXUT43GQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection; write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection Jul 5 15:18:39.733583284 (294h8m14.504224619s) [seg_pos={0 0}, upload=235227]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 5 15:18:39.974321220 (294h8m14.744962555s) [upload=235227]: Done uploading segments... Jul 5 15:18:39.974353041 (294h8m14.744994376s) [upload=235227]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.656758065s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 5.606961592s FAIL (2025-07-05 15:28:31 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== JYE6FC5T5QIBUQIILGDHNGXAUDBUIAOSKYFMPBCWPHP2SDJ3A, seg_pos={0 0}, upload=235214]: Done uploading piece. err: <nil> Jul 5 15:17:12.446386719 (294h6m47.217028064s) [seg_pos={0 0}, upload=235214]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 5 15:17:12.446476771 (294h6m47.217118106s) [upload=235214]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 5 15:17:12.446483043 (294h6m47.217124378s) [upload=235214]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 5 15:17:12.905279085 (294h6m47.67592042s) [upload=235214]: Done uploading segments... Jul 5 15:17:12.905303211 (294h6m47.675944546s) [upload=235214]: Upload finished. err: <nil> Jul 5 15:18:10.381454266 (294h7m45.152095611s) [upload=235227]: Starting upload Jul 5 15:18:10.381524069 (294h7m45.152165414s) [upload=235227]: Done starting upload Jul 5 15:18:10.381621115 (294h7m45.15226245s) [upload=235227]: Uploading segments... Jul 5 15:18:10.381740133 (294h7m45.152381468s) [upload=235227]: Got next segment. Inline: false Jul 5 15:18:10.381776342 (294h7m45.152417677s) [upload=235227]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 5 15:18:10.381781973 (294h7m45.152423308s) [upload=235227]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 5 15:18:10.613904835 (294h7m45.38454617s) [seg_pos={0 0}, upload=235227]: Begin upload segment... Jul 5 15:18:10.613947397 (294h7m45.384588732s) [seg_pos={0 0}, upload=235227]: Done begin upload segment. Jul 5 15:18:10.613968548 (294h7m45.384609883s) [noise=true , address=104.36.82.15:4259 , piece_id=LQKQ5MGKXHTPITBR2QAOQ5CX76GEVJGHJT3ZD3G4PFNN5IBTCJBQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.613995038 (294h7m45.384636373s) [noise=true , address=104.36.82.16:4295 , piece_id=VEL6CAUOJKMRN7A7M66ZPDLCMJVD555LZPHM53A24MDQJQOW7ECQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614303969 (294h7m45.384945304s) [noise=true , address=104.36.82.9:4006 , piece_id=47FME2B5NS6WMOHC2ZGE2FZVSPEQUXKYT6E7AOSANAE5XZVDH5VQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614367721 (294h7m45.385009056s) [noise=true , address=45.91.116.4:4937 , piece_id=YMKIHJET2EN3QHEK665ACZNQNHJ3UGE4H5T4DIHWJUHJQBYO7B7Q, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614619272 (294h7m45.385260607s) [noise=true , address=77.90.31.17:4771 , piece_id=V7M65ZHCGXPD2GVJL6LTB4NP3QN6QLSIBM7U34QAYPAIKMN6EBPA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614637758 (294h7m45.385279083s) [noise=true , address=45.91.116.14:4191 , piece_id=WH2BCI36GGFF4QUD4XDYBBDMUR2R2FFVYKFTZD3ZWXPJCFJ3D7HQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614890251 (294h7m45.385531576s) [noise=true , address=104.36.82.3:4767 , piece_id=OGHG4G5ZVC7GVY4XI6DO4KINXF7THHL6IHGDB6DAPDNUDOUFSC4A, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615197719 (294h7m45.385839054s) [noise=true , address=104.36.82.12:4149 , piece_id=YBTLQEWDMFK6JIFTYR3VJDNMXTS3NMWWHX2MPP53BM2UBDI3UUEA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615413552 (294h7m45.386054887s) [noise=true , address=77.90.31.18:4805 , piece_id=I23F6KYUQ7TJI4ZLEANACVNYJDMPKKMMACIZAIA6PC5S3CKQPV2Q, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615608124 (294h7m45.386249459s) [noise=true , address=77.90.31.21:4877 , piece_id=OJBIFN2TJPQ45LQLYCMPKV5UZM6Y2MXNOBT3TSGX26U4SZAL5Q7A, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615803147 (294h7m45.386444472s) [noise=true , address=104.36.82.8:4970 , piece_id=2YJN7VKTTTCMAWQW2MNR65DAJPFMHNIBIW5NWYNG4R6HNPRFH6NQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615998511 (294h7m45.386639836s) [noise=true , address=77.90.31.16:4733 , piece_id=3GQ7NEWE6WZJMQUA3GG3GSZYXEFNOAUEVWHOEH4TSNONYXUT43GQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616193414 (294h7m45.386834749s) [noise=true , address=77.90.31.19:4842 , piece_id=R4FTRBZ3PETCA6QGU6RPUEDZZBWXF7YVH2COEW4WH3YWDLU3VJMA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616449093 (294h7m45.387090429s) [noise=true , address=77.90.31.11:4554 , piece_id=4QRZ5MPQ7HTPRIVRL3O3H4OV7CQA7ESYT5RCPI3GTHXJOHOSAOVA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616635840 (294h7m45.387277175s) [noise=true , address=45.91.116.5:4298 , piece_id=TVTUGWHFTXB32GBY4LYXT3YDB5LIFXEGA7VO5BHFA24Y6EMDTPHQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616818790 (294h7m45.387460125s) [noise=true , address=104.36.82.13:4185 , piece_id=LTQS2CUZ5NRJCBOPEXDKUQ2IJGOIW3JJAPHWTCW7WUMQZPIWRXRA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:12.265123988 (294h7m47.035765313s) [upload=235227]: Next returned nil segment Jul 5 15:18:12.265138164 (294h7m47.035779499s) [upload=235227]: Waiting for error group managing segments... Jul 5 15:18:12.550465694 (294h7m47.321107019s) [noise=true , address=104.36.82.8:4970 , piece_id=2YJN7VKTTTCMAWQW2MNR65DAJPFMHNIBIW5NWYNG4R6HNPRFH6NQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.554315629 (294h7m47.324956954s) [noise=true , address=104.36.82.12:4149 , piece_id=YBTLQEWDMFK6JIFTYR3VJDNMXTS3NMWWHX2MPP53BM2UBDI3UUEA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.603452512 (294h7m47.374093847s) [noise=true , address=104.36.82.3:4767 , piece_id=OGHG4G5ZVC7GVY4XI6DO4KINXF7THHL6IHGDB6DAPDNUDOUFSC4A, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.639796643 (294h7m47.410437978s) [noise=true , address=104.36.82.9:4006 , piece_id=47FME2B5NS6WMOHC2ZGE2FZVSPEQUXKYT6E7AOSANAE5XZVDH5VQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.639894960 (294h7m47.410536285s) [noise=true , address=104.36.82.15:4259 , piece_id=LQKQ5MGKXHTPITBR2QAOQ5CX76GEVJGHJT3ZD3G4PFNN5IBTCJBQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.647222659 (294h7m47.417863994s) [noise=true , address=104.36.82.16:4295 , piece_id=VEL6CAUOJKMRN7A7M66ZPDLCMJVD555LZPHM53A24MDQJQOW7ECQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.665424010 (294h7m47.436065346s) [noise=true , address=104.36.82.13:4185 , piece_id=LTQS2CUZ5NRJCBOPEXDKUQ2IJGOIW3JJAPHWTCW7WUMQZPIWRXRA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.702209574 (294h7m47.47285091s) [noise=true , address=45.91.116.14:4191 , piece_id=WH2BCI36GGFF4QUD4XDYBBDMUR2R2FFVYKFTZD3ZWXPJCFJ3D7HQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.722073145 (294h7m47.49271448s) [noise=true , address=45.91.116.5:4298 , piece_id=TVTUGWHFTXB32GBY4LYXT3YDB5LIFXEGA7VO5BHFA24Y6EMDTPHQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.788621529 (294h7m47.559262864s) [noise=true , address=45.91.116.4:4937 , piece_id=YMKIHJET2EN3QHEK665ACZNQNHJ3UGE4H5T4DIHWJUHJQBYO7B7Q, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:36.752254721 (294h8m11.522896046s) [noise=true , address=77.90.31.19:4842 , piece_id=R4FTRBZ3PETCA6QGU6RPUEDZZBWXF7YVH2COEW4WH3YWDLU3VJMA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:37.783437003 (294h8m12.554078338s) [noise=true , address=77.90.31.17:4771 , piece_id=V7M65ZHCGXPD2GVJL6LTB4NP3QN6QLSIBM7U34QAYPAIKMN6EBPA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:39.733398290 (294h8m14.504039626s) [noise=true , address=77.90.31.18:4805 , piece_id=I23F6KYUQ7TJI4ZLEANACVNYJDMPKKMMACIZAIA6PC5S3CKQPV2Q, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Jul 5 15:18:39.733464387 (294h8m14.504105722s) [noise=true , address=77.90.31.21:4877 , piece_id=OJBIFN2TJPQ45LQLYCMPKV5UZM6Y2MXNOBT3TSGX26U4SZAL5Q7A, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: expected piece hash; context deadline exceeded; EOF Jul 5 15:18:39.733542065 (294h8m14.50418339s) [noise=true , address=77.90.31.11:4554 , piece_id=4QRZ5MPQ7HTPRIVRL3O3H4OV7CQA7ESYT5RCPI3GTHXJOHOSAOVA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection Jul 5 15:18:39.733547496 (294h8m14.504188831s) [noise=true , address=77.90.31.16:4733 , piece_id=3GQ7NEWE6WZJMQUA3GG3GSZYXEFNOAUEVWHOEH4TSNONYXUT43GQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection; write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection Jul 5 15:18:39.733583284 (294h8m14.504224619s) [seg_pos={0 0}, upload=235227]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 5 15:18:39.974321220 (294h8m14.744962555s) [upload=235227]: Done uploading segments... Jul 5 15:18:39.974353041 (294h8m14.744994376s) [upload=235227]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.606980578s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 5.685936169s FAIL (2025-07-05 15:28:14 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== JYE6FC5T5QIBUQIILGDHNGXAUDBUIAOSKYFMPBCWPHP2SDJ3A, seg_pos={0 0}, upload=235214]: Done uploading piece. err: <nil> Jul 5 15:17:12.446386719 (294h6m47.217028064s) [seg_pos={0 0}, upload=235214]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 5 15:17:12.446476771 (294h6m47.217118106s) [upload=235214]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 5 15:17:12.446483043 (294h6m47.217124378s) [upload=235214]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 5 15:17:12.905279085 (294h6m47.67592042s) [upload=235214]: Done uploading segments... Jul 5 15:17:12.905303211 (294h6m47.675944546s) [upload=235214]: Upload finished. err: <nil> Jul 5 15:18:10.381454266 (294h7m45.152095611s) [upload=235227]: Starting upload Jul 5 15:18:10.381524069 (294h7m45.152165414s) [upload=235227]: Done starting upload Jul 5 15:18:10.381621115 (294h7m45.15226245s) [upload=235227]: Uploading segments... Jul 5 15:18:10.381740133 (294h7m45.152381468s) [upload=235227]: Got next segment. Inline: false Jul 5 15:18:10.381776342 (294h7m45.152417677s) [upload=235227]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 5 15:18:10.381781973 (294h7m45.152423308s) [upload=235227]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 5 15:18:10.613904835 (294h7m45.38454617s) [seg_pos={0 0}, upload=235227]: Begin upload segment... Jul 5 15:18:10.613947397 (294h7m45.384588732s) [seg_pos={0 0}, upload=235227]: Done begin upload segment. Jul 5 15:18:10.613968548 (294h7m45.384609883s) [noise=true , address=104.36.82.15:4259 , piece_id=LQKQ5MGKXHTPITBR2QAOQ5CX76GEVJGHJT3ZD3G4PFNN5IBTCJBQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.613995038 (294h7m45.384636373s) [noise=true , address=104.36.82.16:4295 , piece_id=VEL6CAUOJKMRN7A7M66ZPDLCMJVD555LZPHM53A24MDQJQOW7ECQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614303969 (294h7m45.384945304s) [noise=true , address=104.36.82.9:4006 , piece_id=47FME2B5NS6WMOHC2ZGE2FZVSPEQUXKYT6E7AOSANAE5XZVDH5VQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614367721 (294h7m45.385009056s) [noise=true , address=45.91.116.4:4937 , piece_id=YMKIHJET2EN3QHEK665ACZNQNHJ3UGE4H5T4DIHWJUHJQBYO7B7Q, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614619272 (294h7m45.385260607s) [noise=true , address=77.90.31.17:4771 , piece_id=V7M65ZHCGXPD2GVJL6LTB4NP3QN6QLSIBM7U34QAYPAIKMN6EBPA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614637758 (294h7m45.385279083s) [noise=true , address=45.91.116.14:4191 , piece_id=WH2BCI36GGFF4QUD4XDYBBDMUR2R2FFVYKFTZD3ZWXPJCFJ3D7HQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614890251 (294h7m45.385531576s) [noise=true , address=104.36.82.3:4767 , piece_id=OGHG4G5ZVC7GVY4XI6DO4KINXF7THHL6IHGDB6DAPDNUDOUFSC4A, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615197719 (294h7m45.385839054s) [noise=true , address=104.36.82.12:4149 , piece_id=YBTLQEWDMFK6JIFTYR3VJDNMXTS3NMWWHX2MPP53BM2UBDI3UUEA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615413552 (294h7m45.386054887s) [noise=true , address=77.90.31.18:4805 , piece_id=I23F6KYUQ7TJI4ZLEANACVNYJDMPKKMMACIZAIA6PC5S3CKQPV2Q, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615608124 (294h7m45.386249459s) [noise=true , address=77.90.31.21:4877 , piece_id=OJBIFN2TJPQ45LQLYCMPKV5UZM6Y2MXNOBT3TSGX26U4SZAL5Q7A, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615803147 (294h7m45.386444472s) [noise=true , address=104.36.82.8:4970 , piece_id=2YJN7VKTTTCMAWQW2MNR65DAJPFMHNIBIW5NWYNG4R6HNPRFH6NQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615998511 (294h7m45.386639836s) [noise=true , address=77.90.31.16:4733 , piece_id=3GQ7NEWE6WZJMQUA3GG3GSZYXEFNOAUEVWHOEH4TSNONYXUT43GQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616193414 (294h7m45.386834749s) [noise=true , address=77.90.31.19:4842 , piece_id=R4FTRBZ3PETCA6QGU6RPUEDZZBWXF7YVH2COEW4WH3YWDLU3VJMA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616449093 (294h7m45.387090429s) [noise=true , address=77.90.31.11:4554 , piece_id=4QRZ5MPQ7HTPRIVRL3O3H4OV7CQA7ESYT5RCPI3GTHXJOHOSAOVA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616635840 (294h7m45.387277175s) [noise=true , address=45.91.116.5:4298 , piece_id=TVTUGWHFTXB32GBY4LYXT3YDB5LIFXEGA7VO5BHFA24Y6EMDTPHQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616818790 (294h7m45.387460125s) [noise=true , address=104.36.82.13:4185 , piece_id=LTQS2CUZ5NRJCBOPEXDKUQ2IJGOIW3JJAPHWTCW7WUMQZPIWRXRA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:12.265123988 (294h7m47.035765313s) [upload=235227]: Next returned nil segment Jul 5 15:18:12.265138164 (294h7m47.035779499s) [upload=235227]: Waiting for error group managing segments... Jul 5 15:18:12.550465694 (294h7m47.321107019s) [noise=true , address=104.36.82.8:4970 , piece_id=2YJN7VKTTTCMAWQW2MNR65DAJPFMHNIBIW5NWYNG4R6HNPRFH6NQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.554315629 (294h7m47.324956954s) [noise=true , address=104.36.82.12:4149 , piece_id=YBTLQEWDMFK6JIFTYR3VJDNMXTS3NMWWHX2MPP53BM2UBDI3UUEA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.603452512 (294h7m47.374093847s) [noise=true , address=104.36.82.3:4767 , piece_id=OGHG4G5ZVC7GVY4XI6DO4KINXF7THHL6IHGDB6DAPDNUDOUFSC4A, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.639796643 (294h7m47.410437978s) [noise=true , address=104.36.82.9:4006 , piece_id=47FME2B5NS6WMOHC2ZGE2FZVSPEQUXKYT6E7AOSANAE5XZVDH5VQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.639894960 (294h7m47.410536285s) [noise=true , address=104.36.82.15:4259 , piece_id=LQKQ5MGKXHTPITBR2QAOQ5CX76GEVJGHJT3ZD3G4PFNN5IBTCJBQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.647222659 (294h7m47.417863994s) [noise=true , address=104.36.82.16:4295 , piece_id=VEL6CAUOJKMRN7A7M66ZPDLCMJVD555LZPHM53A24MDQJQOW7ECQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.665424010 (294h7m47.436065346s) [noise=true , address=104.36.82.13:4185 , piece_id=LTQS2CUZ5NRJCBOPEXDKUQ2IJGOIW3JJAPHWTCW7WUMQZPIWRXRA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.702209574 (294h7m47.47285091s) [noise=true , address=45.91.116.14:4191 , piece_id=WH2BCI36GGFF4QUD4XDYBBDMUR2R2FFVYKFTZD3ZWXPJCFJ3D7HQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.722073145 (294h7m47.49271448s) [noise=true , address=45.91.116.5:4298 , piece_id=TVTUGWHFTXB32GBY4LYXT3YDB5LIFXEGA7VO5BHFA24Y6EMDTPHQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.788621529 (294h7m47.559262864s) [noise=true , address=45.91.116.4:4937 , piece_id=YMKIHJET2EN3QHEK665ACZNQNHJ3UGE4H5T4DIHWJUHJQBYO7B7Q, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:36.752254721 (294h8m11.522896046s) [noise=true , address=77.90.31.19:4842 , piece_id=R4FTRBZ3PETCA6QGU6RPUEDZZBWXF7YVH2COEW4WH3YWDLU3VJMA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:37.783437003 (294h8m12.554078338s) [noise=true , address=77.90.31.17:4771 , piece_id=V7M65ZHCGXPD2GVJL6LTB4NP3QN6QLSIBM7U34QAYPAIKMN6EBPA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:39.733398290 (294h8m14.504039626s) [noise=true , address=77.90.31.18:4805 , piece_id=I23F6KYUQ7TJI4ZLEANACVNYJDMPKKMMACIZAIA6PC5S3CKQPV2Q, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Jul 5 15:18:39.733464387 (294h8m14.504105722s) [noise=true , address=77.90.31.21:4877 , piece_id=OJBIFN2TJPQ45LQLYCMPKV5UZM6Y2MXNOBT3TSGX26U4SZAL5Q7A, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: expected piece hash; context deadline exceeded; EOF Jul 5 15:18:39.733542065 (294h8m14.50418339s) [noise=true , address=77.90.31.11:4554 , piece_id=4QRZ5MPQ7HTPRIVRL3O3H4OV7CQA7ESYT5RCPI3GTHXJOHOSAOVA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection Jul 5 15:18:39.733547496 (294h8m14.504188831s) [noise=true , address=77.90.31.16:4733 , piece_id=3GQ7NEWE6WZJMQUA3GG3GSZYXEFNOAUEVWHOEH4TSNONYXUT43GQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection; write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection Jul 5 15:18:39.733583284 (294h8m14.504224619s) [seg_pos={0 0}, upload=235227]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 5 15:18:39.974321220 (294h8m14.744962555s) [upload=235227]: Done uploading segments... Jul 5 15:18:39.974353041 (294h8m14.744994376s) [upload=235227]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.6859629s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 5.654957151s FAIL (2025-07-05 15:27:21 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== JYE6FC5T5QIBUQIILGDHNGXAUDBUIAOSKYFMPBCWPHP2SDJ3A, seg_pos={0 0}, upload=235214]: Done uploading piece. err: <nil> Jul 5 15:17:12.446386719 (294h6m47.217028064s) [seg_pos={0 0}, upload=235214]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 5 15:17:12.446476771 (294h6m47.217118106s) [upload=235214]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 5 15:17:12.446483043 (294h6m47.217124378s) [upload=235214]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 5 15:17:12.905279085 (294h6m47.67592042s) [upload=235214]: Done uploading segments... Jul 5 15:17:12.905303211 (294h6m47.675944546s) [upload=235214]: Upload finished. err: <nil> Jul 5 15:18:10.381454266 (294h7m45.152095611s) [upload=235227]: Starting upload Jul 5 15:18:10.381524069 (294h7m45.152165414s) [upload=235227]: Done starting upload Jul 5 15:18:10.381621115 (294h7m45.15226245s) [upload=235227]: Uploading segments... Jul 5 15:18:10.381740133 (294h7m45.152381468s) [upload=235227]: Got next segment. Inline: false Jul 5 15:18:10.381776342 (294h7m45.152417677s) [upload=235227]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 5 15:18:10.381781973 (294h7m45.152423308s) [upload=235227]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 5 15:18:10.613904835 (294h7m45.38454617s) [seg_pos={0 0}, upload=235227]: Begin upload segment... Jul 5 15:18:10.613947397 (294h7m45.384588732s) [seg_pos={0 0}, upload=235227]: Done begin upload segment. Jul 5 15:18:10.613968548 (294h7m45.384609883s) [noise=true , address=104.36.82.15:4259 , piece_id=LQKQ5MGKXHTPITBR2QAOQ5CX76GEVJGHJT3ZD3G4PFNN5IBTCJBQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.613995038 (294h7m45.384636373s) [noise=true , address=104.36.82.16:4295 , piece_id=VEL6CAUOJKMRN7A7M66ZPDLCMJVD555LZPHM53A24MDQJQOW7ECQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614303969 (294h7m45.384945304s) [noise=true , address=104.36.82.9:4006 , piece_id=47FME2B5NS6WMOHC2ZGE2FZVSPEQUXKYT6E7AOSANAE5XZVDH5VQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614367721 (294h7m45.385009056s) [noise=true , address=45.91.116.4:4937 , piece_id=YMKIHJET2EN3QHEK665ACZNQNHJ3UGE4H5T4DIHWJUHJQBYO7B7Q, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614619272 (294h7m45.385260607s) [noise=true , address=77.90.31.17:4771 , piece_id=V7M65ZHCGXPD2GVJL6LTB4NP3QN6QLSIBM7U34QAYPAIKMN6EBPA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614637758 (294h7m45.385279083s) [noise=true , address=45.91.116.14:4191 , piece_id=WH2BCI36GGFF4QUD4XDYBBDMUR2R2FFVYKFTZD3ZWXPJCFJ3D7HQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614890251 (294h7m45.385531576s) [noise=true , address=104.36.82.3:4767 , piece_id=OGHG4G5ZVC7GVY4XI6DO4KINXF7THHL6IHGDB6DAPDNUDOUFSC4A, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615197719 (294h7m45.385839054s) [noise=true , address=104.36.82.12:4149 , piece_id=YBTLQEWDMFK6JIFTYR3VJDNMXTS3NMWWHX2MPP53BM2UBDI3UUEA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615413552 (294h7m45.386054887s) [noise=true , address=77.90.31.18:4805 , piece_id=I23F6KYUQ7TJI4ZLEANACVNYJDMPKKMMACIZAIA6PC5S3CKQPV2Q, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615608124 (294h7m45.386249459s) [noise=true , address=77.90.31.21:4877 , piece_id=OJBIFN2TJPQ45LQLYCMPKV5UZM6Y2MXNOBT3TSGX26U4SZAL5Q7A, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615803147 (294h7m45.386444472s) [noise=true , address=104.36.82.8:4970 , piece_id=2YJN7VKTTTCMAWQW2MNR65DAJPFMHNIBIW5NWYNG4R6HNPRFH6NQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615998511 (294h7m45.386639836s) [noise=true , address=77.90.31.16:4733 , piece_id=3GQ7NEWE6WZJMQUA3GG3GSZYXEFNOAUEVWHOEH4TSNONYXUT43GQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616193414 (294h7m45.386834749s) [noise=true , address=77.90.31.19:4842 , piece_id=R4FTRBZ3PETCA6QGU6RPUEDZZBWXF7YVH2COEW4WH3YWDLU3VJMA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616449093 (294h7m45.387090429s) [noise=true , address=77.90.31.11:4554 , piece_id=4QRZ5MPQ7HTPRIVRL3O3H4OV7CQA7ESYT5RCPI3GTHXJOHOSAOVA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616635840 (294h7m45.387277175s) [noise=true , address=45.91.116.5:4298 , piece_id=TVTUGWHFTXB32GBY4LYXT3YDB5LIFXEGA7VO5BHFA24Y6EMDTPHQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616818790 (294h7m45.387460125s) [noise=true , address=104.36.82.13:4185 , piece_id=LTQS2CUZ5NRJCBOPEXDKUQ2IJGOIW3JJAPHWTCW7WUMQZPIWRXRA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:12.265123988 (294h7m47.035765313s) [upload=235227]: Next returned nil segment Jul 5 15:18:12.265138164 (294h7m47.035779499s) [upload=235227]: Waiting for error group managing segments... Jul 5 15:18:12.550465694 (294h7m47.321107019s) [noise=true , address=104.36.82.8:4970 , piece_id=2YJN7VKTTTCMAWQW2MNR65DAJPFMHNIBIW5NWYNG4R6HNPRFH6NQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.554315629 (294h7m47.324956954s) [noise=true , address=104.36.82.12:4149 , piece_id=YBTLQEWDMFK6JIFTYR3VJDNMXTS3NMWWHX2MPP53BM2UBDI3UUEA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.603452512 (294h7m47.374093847s) [noise=true , address=104.36.82.3:4767 , piece_id=OGHG4G5ZVC7GVY4XI6DO4KINXF7THHL6IHGDB6DAPDNUDOUFSC4A, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.639796643 (294h7m47.410437978s) [noise=true , address=104.36.82.9:4006 , piece_id=47FME2B5NS6WMOHC2ZGE2FZVSPEQUXKYT6E7AOSANAE5XZVDH5VQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.639894960 (294h7m47.410536285s) [noise=true , address=104.36.82.15:4259 , piece_id=LQKQ5MGKXHTPITBR2QAOQ5CX76GEVJGHJT3ZD3G4PFNN5IBTCJBQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.647222659 (294h7m47.417863994s) [noise=true , address=104.36.82.16:4295 , piece_id=VEL6CAUOJKMRN7A7M66ZPDLCMJVD555LZPHM53A24MDQJQOW7ECQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.665424010 (294h7m47.436065346s) [noise=true , address=104.36.82.13:4185 , piece_id=LTQS2CUZ5NRJCBOPEXDKUQ2IJGOIW3JJAPHWTCW7WUMQZPIWRXRA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.702209574 (294h7m47.47285091s) [noise=true , address=45.91.116.14:4191 , piece_id=WH2BCI36GGFF4QUD4XDYBBDMUR2R2FFVYKFTZD3ZWXPJCFJ3D7HQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.722073145 (294h7m47.49271448s) [noise=true , address=45.91.116.5:4298 , piece_id=TVTUGWHFTXB32GBY4LYXT3YDB5LIFXEGA7VO5BHFA24Y6EMDTPHQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.788621529 (294h7m47.559262864s) [noise=true , address=45.91.116.4:4937 , piece_id=YMKIHJET2EN3QHEK665ACZNQNHJ3UGE4H5T4DIHWJUHJQBYO7B7Q, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:36.752254721 (294h8m11.522896046s) [noise=true , address=77.90.31.19:4842 , piece_id=R4FTRBZ3PETCA6QGU6RPUEDZZBWXF7YVH2COEW4WH3YWDLU3VJMA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:37.783437003 (294h8m12.554078338s) [noise=true , address=77.90.31.17:4771 , piece_id=V7M65ZHCGXPD2GVJL6LTB4NP3QN6QLSIBM7U34QAYPAIKMN6EBPA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:39.733398290 (294h8m14.504039626s) [noise=true , address=77.90.31.18:4805 , piece_id=I23F6KYUQ7TJI4ZLEANACVNYJDMPKKMMACIZAIA6PC5S3CKQPV2Q, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Jul 5 15:18:39.733464387 (294h8m14.504105722s) [noise=true , address=77.90.31.21:4877 , piece_id=OJBIFN2TJPQ45LQLYCMPKV5UZM6Y2MXNOBT3TSGX26U4SZAL5Q7A, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: expected piece hash; context deadline exceeded; EOF Jul 5 15:18:39.733542065 (294h8m14.50418339s) [noise=true , address=77.90.31.11:4554 , piece_id=4QRZ5MPQ7HTPRIVRL3O3H4OV7CQA7ESYT5RCPI3GTHXJOHOSAOVA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection Jul 5 15:18:39.733547496 (294h8m14.504188831s) [noise=true , address=77.90.31.16:4733 , piece_id=3GQ7NEWE6WZJMQUA3GG3GSZYXEFNOAUEVWHOEH4TSNONYXUT43GQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection; write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection Jul 5 15:18:39.733583284 (294h8m14.504224619s) [seg_pos={0 0}, upload=235227]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 5 15:18:39.974321220 (294h8m14.744962555s) [upload=235227]: Done uploading segments... Jul 5 15:18:39.974353041 (294h8m14.744994376s) [upload=235227]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.654982802s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 5.706703709s FAIL (2025-07-05 15:26:57 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== JYE6FC5T5QIBUQIILGDHNGXAUDBUIAOSKYFMPBCWPHP2SDJ3A, seg_pos={0 0}, upload=235214]: Done uploading piece. err: <nil> Jul 5 15:17:12.446386719 (294h6m47.217028064s) [seg_pos={0 0}, upload=235214]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 5 15:17:12.446476771 (294h6m47.217118106s) [upload=235214]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 5 15:17:12.446483043 (294h6m47.217124378s) [upload=235214]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 5 15:17:12.905279085 (294h6m47.67592042s) [upload=235214]: Done uploading segments... Jul 5 15:17:12.905303211 (294h6m47.675944546s) [upload=235214]: Upload finished. err: <nil> Jul 5 15:18:10.381454266 (294h7m45.152095611s) [upload=235227]: Starting upload Jul 5 15:18:10.381524069 (294h7m45.152165414s) [upload=235227]: Done starting upload Jul 5 15:18:10.381621115 (294h7m45.15226245s) [upload=235227]: Uploading segments... Jul 5 15:18:10.381740133 (294h7m45.152381468s) [upload=235227]: Got next segment. Inline: false Jul 5 15:18:10.381776342 (294h7m45.152417677s) [upload=235227]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 5 15:18:10.381781973 (294h7m45.152423308s) [upload=235227]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 5 15:18:10.613904835 (294h7m45.38454617s) [seg_pos={0 0}, upload=235227]: Begin upload segment... Jul 5 15:18:10.613947397 (294h7m45.384588732s) [seg_pos={0 0}, upload=235227]: Done begin upload segment. Jul 5 15:18:10.613968548 (294h7m45.384609883s) [noise=true , address=104.36.82.15:4259 , piece_id=LQKQ5MGKXHTPITBR2QAOQ5CX76GEVJGHJT3ZD3G4PFNN5IBTCJBQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.613995038 (294h7m45.384636373s) [noise=true , address=104.36.82.16:4295 , piece_id=VEL6CAUOJKMRN7A7M66ZPDLCMJVD555LZPHM53A24MDQJQOW7ECQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614303969 (294h7m45.384945304s) [noise=true , address=104.36.82.9:4006 , piece_id=47FME2B5NS6WMOHC2ZGE2FZVSPEQUXKYT6E7AOSANAE5XZVDH5VQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614367721 (294h7m45.385009056s) [noise=true , address=45.91.116.4:4937 , piece_id=YMKIHJET2EN3QHEK665ACZNQNHJ3UGE4H5T4DIHWJUHJQBYO7B7Q, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614619272 (294h7m45.385260607s) [noise=true , address=77.90.31.17:4771 , piece_id=V7M65ZHCGXPD2GVJL6LTB4NP3QN6QLSIBM7U34QAYPAIKMN6EBPA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614637758 (294h7m45.385279083s) [noise=true , address=45.91.116.14:4191 , piece_id=WH2BCI36GGFF4QUD4XDYBBDMUR2R2FFVYKFTZD3ZWXPJCFJ3D7HQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614890251 (294h7m45.385531576s) [noise=true , address=104.36.82.3:4767 , piece_id=OGHG4G5ZVC7GVY4XI6DO4KINXF7THHL6IHGDB6DAPDNUDOUFSC4A, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615197719 (294h7m45.385839054s) [noise=true , address=104.36.82.12:4149 , piece_id=YBTLQEWDMFK6JIFTYR3VJDNMXTS3NMWWHX2MPP53BM2UBDI3UUEA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615413552 (294h7m45.386054887s) [noise=true , address=77.90.31.18:4805 , piece_id=I23F6KYUQ7TJI4ZLEANACVNYJDMPKKMMACIZAIA6PC5S3CKQPV2Q, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615608124 (294h7m45.386249459s) [noise=true , address=77.90.31.21:4877 , piece_id=OJBIFN2TJPQ45LQLYCMPKV5UZM6Y2MXNOBT3TSGX26U4SZAL5Q7A, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615803147 (294h7m45.386444472s) [noise=true , address=104.36.82.8:4970 , piece_id=2YJN7VKTTTCMAWQW2MNR65DAJPFMHNIBIW5NWYNG4R6HNPRFH6NQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615998511 (294h7m45.386639836s) [noise=true , address=77.90.31.16:4733 , piece_id=3GQ7NEWE6WZJMQUA3GG3GSZYXEFNOAUEVWHOEH4TSNONYXUT43GQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616193414 (294h7m45.386834749s) [noise=true , address=77.90.31.19:4842 , piece_id=R4FTRBZ3PETCA6QGU6RPUEDZZBWXF7YVH2COEW4WH3YWDLU3VJMA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616449093 (294h7m45.387090429s) [noise=true , address=77.90.31.11:4554 , piece_id=4QRZ5MPQ7HTPRIVRL3O3H4OV7CQA7ESYT5RCPI3GTHXJOHOSAOVA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616635840 (294h7m45.387277175s) [noise=true , address=45.91.116.5:4298 , piece_id=TVTUGWHFTXB32GBY4LYXT3YDB5LIFXEGA7VO5BHFA24Y6EMDTPHQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616818790 (294h7m45.387460125s) [noise=true , address=104.36.82.13:4185 , piece_id=LTQS2CUZ5NRJCBOPEXDKUQ2IJGOIW3JJAPHWTCW7WUMQZPIWRXRA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:12.265123988 (294h7m47.035765313s) [upload=235227]: Next returned nil segment Jul 5 15:18:12.265138164 (294h7m47.035779499s) [upload=235227]: Waiting for error group managing segments... Jul 5 15:18:12.550465694 (294h7m47.321107019s) [noise=true , address=104.36.82.8:4970 , piece_id=2YJN7VKTTTCMAWQW2MNR65DAJPFMHNIBIW5NWYNG4R6HNPRFH6NQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.554315629 (294h7m47.324956954s) [noise=true , address=104.36.82.12:4149 , piece_id=YBTLQEWDMFK6JIFTYR3VJDNMXTS3NMWWHX2MPP53BM2UBDI3UUEA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.603452512 (294h7m47.374093847s) [noise=true , address=104.36.82.3:4767 , piece_id=OGHG4G5ZVC7GVY4XI6DO4KINXF7THHL6IHGDB6DAPDNUDOUFSC4A, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.639796643 (294h7m47.410437978s) [noise=true , address=104.36.82.9:4006 , piece_id=47FME2B5NS6WMOHC2ZGE2FZVSPEQUXKYT6E7AOSANAE5XZVDH5VQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.639894960 (294h7m47.410536285s) [noise=true , address=104.36.82.15:4259 , piece_id=LQKQ5MGKXHTPITBR2QAOQ5CX76GEVJGHJT3ZD3G4PFNN5IBTCJBQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.647222659 (294h7m47.417863994s) [noise=true , address=104.36.82.16:4295 , piece_id=VEL6CAUOJKMRN7A7M66ZPDLCMJVD555LZPHM53A24MDQJQOW7ECQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.665424010 (294h7m47.436065346s) [noise=true , address=104.36.82.13:4185 , piece_id=LTQS2CUZ5NRJCBOPEXDKUQ2IJGOIW3JJAPHWTCW7WUMQZPIWRXRA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.702209574 (294h7m47.47285091s) [noise=true , address=45.91.116.14:4191 , piece_id=WH2BCI36GGFF4QUD4XDYBBDMUR2R2FFVYKFTZD3ZWXPJCFJ3D7HQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.722073145 (294h7m47.49271448s) [noise=true , address=45.91.116.5:4298 , piece_id=TVTUGWHFTXB32GBY4LYXT3YDB5LIFXEGA7VO5BHFA24Y6EMDTPHQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.788621529 (294h7m47.559262864s) [noise=true , address=45.91.116.4:4937 , piece_id=YMKIHJET2EN3QHEK665ACZNQNHJ3UGE4H5T4DIHWJUHJQBYO7B7Q, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:36.752254721 (294h8m11.522896046s) [noise=true , address=77.90.31.19:4842 , piece_id=R4FTRBZ3PETCA6QGU6RPUEDZZBWXF7YVH2COEW4WH3YWDLU3VJMA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:37.783437003 (294h8m12.554078338s) [noise=true , address=77.90.31.17:4771 , piece_id=V7M65ZHCGXPD2GVJL6LTB4NP3QN6QLSIBM7U34QAYPAIKMN6EBPA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:39.733398290 (294h8m14.504039626s) [noise=true , address=77.90.31.18:4805 , piece_id=I23F6KYUQ7TJI4ZLEANACVNYJDMPKKMMACIZAIA6PC5S3CKQPV2Q, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Jul 5 15:18:39.733464387 (294h8m14.504105722s) [noise=true , address=77.90.31.21:4877 , piece_id=OJBIFN2TJPQ45LQLYCMPKV5UZM6Y2MXNOBT3TSGX26U4SZAL5Q7A, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: expected piece hash; context deadline exceeded; EOF Jul 5 15:18:39.733542065 (294h8m14.50418339s) [noise=true , address=77.90.31.11:4554 , piece_id=4QRZ5MPQ7HTPRIVRL3O3H4OV7CQA7ESYT5RCPI3GTHXJOHOSAOVA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection Jul 5 15:18:39.733547496 (294h8m14.504188831s) [noise=true , address=77.90.31.16:4733 , piece_id=3GQ7NEWE6WZJMQUA3GG3GSZYXEFNOAUEVWHOEH4TSNONYXUT43GQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection; write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection Jul 5 15:18:39.733583284 (294h8m14.504224619s) [seg_pos={0 0}, upload=235227]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 5 15:18:39.974321220 (294h8m14.744962555s) [upload=235227]: Done uploading segments... Jul 5 15:18:39.974353041 (294h8m14.744994376s) [upload=235227]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.706727555s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 5.720192924s FAIL (2025-07-05 15:25:49 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== JYE6FC5T5QIBUQIILGDHNGXAUDBUIAOSKYFMPBCWPHP2SDJ3A, seg_pos={0 0}, upload=235214]: Done uploading piece. err: <nil> Jul 5 15:17:12.446386719 (294h6m47.217028064s) [seg_pos={0 0}, upload=235214]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 5 15:17:12.446476771 (294h6m47.217118106s) [upload=235214]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 5 15:17:12.446483043 (294h6m47.217124378s) [upload=235214]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 5 15:17:12.905279085 (294h6m47.67592042s) [upload=235214]: Done uploading segments... Jul 5 15:17:12.905303211 (294h6m47.675944546s) [upload=235214]: Upload finished. err: <nil> Jul 5 15:18:10.381454266 (294h7m45.152095611s) [upload=235227]: Starting upload Jul 5 15:18:10.381524069 (294h7m45.152165414s) [upload=235227]: Done starting upload Jul 5 15:18:10.381621115 (294h7m45.15226245s) [upload=235227]: Uploading segments... Jul 5 15:18:10.381740133 (294h7m45.152381468s) [upload=235227]: Got next segment. Inline: false Jul 5 15:18:10.381776342 (294h7m45.152417677s) [upload=235227]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 5 15:18:10.381781973 (294h7m45.152423308s) [upload=235227]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 5 15:18:10.613904835 (294h7m45.38454617s) [seg_pos={0 0}, upload=235227]: Begin upload segment... Jul 5 15:18:10.613947397 (294h7m45.384588732s) [seg_pos={0 0}, upload=235227]: Done begin upload segment. Jul 5 15:18:10.613968548 (294h7m45.384609883s) [noise=true , address=104.36.82.15:4259 , piece_id=LQKQ5MGKXHTPITBR2QAOQ5CX76GEVJGHJT3ZD3G4PFNN5IBTCJBQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.613995038 (294h7m45.384636373s) [noise=true , address=104.36.82.16:4295 , piece_id=VEL6CAUOJKMRN7A7M66ZPDLCMJVD555LZPHM53A24MDQJQOW7ECQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614303969 (294h7m45.384945304s) [noise=true , address=104.36.82.9:4006 , piece_id=47FME2B5NS6WMOHC2ZGE2FZVSPEQUXKYT6E7AOSANAE5XZVDH5VQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614367721 (294h7m45.385009056s) [noise=true , address=45.91.116.4:4937 , piece_id=YMKIHJET2EN3QHEK665ACZNQNHJ3UGE4H5T4DIHWJUHJQBYO7B7Q, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614619272 (294h7m45.385260607s) [noise=true , address=77.90.31.17:4771 , piece_id=V7M65ZHCGXPD2GVJL6LTB4NP3QN6QLSIBM7U34QAYPAIKMN6EBPA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614637758 (294h7m45.385279083s) [noise=true , address=45.91.116.14:4191 , piece_id=WH2BCI36GGFF4QUD4XDYBBDMUR2R2FFVYKFTZD3ZWXPJCFJ3D7HQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614890251 (294h7m45.385531576s) [noise=true , address=104.36.82.3:4767 , piece_id=OGHG4G5ZVC7GVY4XI6DO4KINXF7THHL6IHGDB6DAPDNUDOUFSC4A, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615197719 (294h7m45.385839054s) [noise=true , address=104.36.82.12:4149 , piece_id=YBTLQEWDMFK6JIFTYR3VJDNMXTS3NMWWHX2MPP53BM2UBDI3UUEA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615413552 (294h7m45.386054887s) [noise=true , address=77.90.31.18:4805 , piece_id=I23F6KYUQ7TJI4ZLEANACVNYJDMPKKMMACIZAIA6PC5S3CKQPV2Q, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615608124 (294h7m45.386249459s) [noise=true , address=77.90.31.21:4877 , piece_id=OJBIFN2TJPQ45LQLYCMPKV5UZM6Y2MXNOBT3TSGX26U4SZAL5Q7A, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615803147 (294h7m45.386444472s) [noise=true , address=104.36.82.8:4970 , piece_id=2YJN7VKTTTCMAWQW2MNR65DAJPFMHNIBIW5NWYNG4R6HNPRFH6NQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615998511 (294h7m45.386639836s) [noise=true , address=77.90.31.16:4733 , piece_id=3GQ7NEWE6WZJMQUA3GG3GSZYXEFNOAUEVWHOEH4TSNONYXUT43GQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616193414 (294h7m45.386834749s) [noise=true , address=77.90.31.19:4842 , piece_id=R4FTRBZ3PETCA6QGU6RPUEDZZBWXF7YVH2COEW4WH3YWDLU3VJMA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616449093 (294h7m45.387090429s) [noise=true , address=77.90.31.11:4554 , piece_id=4QRZ5MPQ7HTPRIVRL3O3H4OV7CQA7ESYT5RCPI3GTHXJOHOSAOVA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616635840 (294h7m45.387277175s) [noise=true , address=45.91.116.5:4298 , piece_id=TVTUGWHFTXB32GBY4LYXT3YDB5LIFXEGA7VO5BHFA24Y6EMDTPHQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616818790 (294h7m45.387460125s) [noise=true , address=104.36.82.13:4185 , piece_id=LTQS2CUZ5NRJCBOPEXDKUQ2IJGOIW3JJAPHWTCW7WUMQZPIWRXRA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:12.265123988 (294h7m47.035765313s) [upload=235227]: Next returned nil segment Jul 5 15:18:12.265138164 (294h7m47.035779499s) [upload=235227]: Waiting for error group managing segments... Jul 5 15:18:12.550465694 (294h7m47.321107019s) [noise=true , address=104.36.82.8:4970 , piece_id=2YJN7VKTTTCMAWQW2MNR65DAJPFMHNIBIW5NWYNG4R6HNPRFH6NQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.554315629 (294h7m47.324956954s) [noise=true , address=104.36.82.12:4149 , piece_id=YBTLQEWDMFK6JIFTYR3VJDNMXTS3NMWWHX2MPP53BM2UBDI3UUEA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.603452512 (294h7m47.374093847s) [noise=true , address=104.36.82.3:4767 , piece_id=OGHG4G5ZVC7GVY4XI6DO4KINXF7THHL6IHGDB6DAPDNUDOUFSC4A, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.639796643 (294h7m47.410437978s) [noise=true , address=104.36.82.9:4006 , piece_id=47FME2B5NS6WMOHC2ZGE2FZVSPEQUXKYT6E7AOSANAE5XZVDH5VQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.639894960 (294h7m47.410536285s) [noise=true , address=104.36.82.15:4259 , piece_id=LQKQ5MGKXHTPITBR2QAOQ5CX76GEVJGHJT3ZD3G4PFNN5IBTCJBQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.647222659 (294h7m47.417863994s) [noise=true , address=104.36.82.16:4295 , piece_id=VEL6CAUOJKMRN7A7M66ZPDLCMJVD555LZPHM53A24MDQJQOW7ECQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.665424010 (294h7m47.436065346s) [noise=true , address=104.36.82.13:4185 , piece_id=LTQS2CUZ5NRJCBOPEXDKUQ2IJGOIW3JJAPHWTCW7WUMQZPIWRXRA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.702209574 (294h7m47.47285091s) [noise=true , address=45.91.116.14:4191 , piece_id=WH2BCI36GGFF4QUD4XDYBBDMUR2R2FFVYKFTZD3ZWXPJCFJ3D7HQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.722073145 (294h7m47.49271448s) [noise=true , address=45.91.116.5:4298 , piece_id=TVTUGWHFTXB32GBY4LYXT3YDB5LIFXEGA7VO5BHFA24Y6EMDTPHQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.788621529 (294h7m47.559262864s) [noise=true , address=45.91.116.4:4937 , piece_id=YMKIHJET2EN3QHEK665ACZNQNHJ3UGE4H5T4DIHWJUHJQBYO7B7Q, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:36.752254721 (294h8m11.522896046s) [noise=true , address=77.90.31.19:4842 , piece_id=R4FTRBZ3PETCA6QGU6RPUEDZZBWXF7YVH2COEW4WH3YWDLU3VJMA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:37.783437003 (294h8m12.554078338s) [noise=true , address=77.90.31.17:4771 , piece_id=V7M65ZHCGXPD2GVJL6LTB4NP3QN6QLSIBM7U34QAYPAIKMN6EBPA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:39.733398290 (294h8m14.504039626s) [noise=true , address=77.90.31.18:4805 , piece_id=I23F6KYUQ7TJI4ZLEANACVNYJDMPKKMMACIZAIA6PC5S3CKQPV2Q, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Jul 5 15:18:39.733464387 (294h8m14.504105722s) [noise=true , address=77.90.31.21:4877 , piece_id=OJBIFN2TJPQ45LQLYCMPKV5UZM6Y2MXNOBT3TSGX26U4SZAL5Q7A, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: expected piece hash; context deadline exceeded; EOF Jul 5 15:18:39.733542065 (294h8m14.50418339s) [noise=true , address=77.90.31.11:4554 , piece_id=4QRZ5MPQ7HTPRIVRL3O3H4OV7CQA7ESYT5RCPI3GTHXJOHOSAOVA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection Jul 5 15:18:39.733547496 (294h8m14.504188831s) [noise=true , address=77.90.31.16:4733 , piece_id=3GQ7NEWE6WZJMQUA3GG3GSZYXEFNOAUEVWHOEH4TSNONYXUT43GQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection; write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection Jul 5 15:18:39.733583284 (294h8m14.504224619s) [seg_pos={0 0}, upload=235227]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 5 15:18:39.974321220 (294h8m14.744962555s) [upload=235227]: Done uploading segments... Jul 5 15:18:39.974353041 (294h8m14.744994376s) [upload=235227]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.720217582s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 5.745733783s FAIL (2025-07-05 15:25:24 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== JYE6FC5T5QIBUQIILGDHNGXAUDBUIAOSKYFMPBCWPHP2SDJ3A, seg_pos={0 0}, upload=235214]: Done uploading piece. err: <nil> Jul 5 15:17:12.446386719 (294h6m47.217028064s) [seg_pos={0 0}, upload=235214]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 5 15:17:12.446476771 (294h6m47.217118106s) [upload=235214]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 5 15:17:12.446483043 (294h6m47.217124378s) [upload=235214]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 5 15:17:12.905279085 (294h6m47.67592042s) [upload=235214]: Done uploading segments... Jul 5 15:17:12.905303211 (294h6m47.675944546s) [upload=235214]: Upload finished. err: <nil> Jul 5 15:18:10.381454266 (294h7m45.152095611s) [upload=235227]: Starting upload Jul 5 15:18:10.381524069 (294h7m45.152165414s) [upload=235227]: Done starting upload Jul 5 15:18:10.381621115 (294h7m45.15226245s) [upload=235227]: Uploading segments... Jul 5 15:18:10.381740133 (294h7m45.152381468s) [upload=235227]: Got next segment. Inline: false Jul 5 15:18:10.381776342 (294h7m45.152417677s) [upload=235227]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 5 15:18:10.381781973 (294h7m45.152423308s) [upload=235227]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 5 15:18:10.613904835 (294h7m45.38454617s) [seg_pos={0 0}, upload=235227]: Begin upload segment... Jul 5 15:18:10.613947397 (294h7m45.384588732s) [seg_pos={0 0}, upload=235227]: Done begin upload segment. Jul 5 15:18:10.613968548 (294h7m45.384609883s) [noise=true , address=104.36.82.15:4259 , piece_id=LQKQ5MGKXHTPITBR2QAOQ5CX76GEVJGHJT3ZD3G4PFNN5IBTCJBQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.613995038 (294h7m45.384636373s) [noise=true , address=104.36.82.16:4295 , piece_id=VEL6CAUOJKMRN7A7M66ZPDLCMJVD555LZPHM53A24MDQJQOW7ECQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614303969 (294h7m45.384945304s) [noise=true , address=104.36.82.9:4006 , piece_id=47FME2B5NS6WMOHC2ZGE2FZVSPEQUXKYT6E7AOSANAE5XZVDH5VQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614367721 (294h7m45.385009056s) [noise=true , address=45.91.116.4:4937 , piece_id=YMKIHJET2EN3QHEK665ACZNQNHJ3UGE4H5T4DIHWJUHJQBYO7B7Q, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614619272 (294h7m45.385260607s) [noise=true , address=77.90.31.17:4771 , piece_id=V7M65ZHCGXPD2GVJL6LTB4NP3QN6QLSIBM7U34QAYPAIKMN6EBPA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614637758 (294h7m45.385279083s) [noise=true , address=45.91.116.14:4191 , piece_id=WH2BCI36GGFF4QUD4XDYBBDMUR2R2FFVYKFTZD3ZWXPJCFJ3D7HQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614890251 (294h7m45.385531576s) [noise=true , address=104.36.82.3:4767 , piece_id=OGHG4G5ZVC7GVY4XI6DO4KINXF7THHL6IHGDB6DAPDNUDOUFSC4A, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615197719 (294h7m45.385839054s) [noise=true , address=104.36.82.12:4149 , piece_id=YBTLQEWDMFK6JIFTYR3VJDNMXTS3NMWWHX2MPP53BM2UBDI3UUEA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615413552 (294h7m45.386054887s) [noise=true , address=77.90.31.18:4805 , piece_id=I23F6KYUQ7TJI4ZLEANACVNYJDMPKKMMACIZAIA6PC5S3CKQPV2Q, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615608124 (294h7m45.386249459s) [noise=true , address=77.90.31.21:4877 , piece_id=OJBIFN2TJPQ45LQLYCMPKV5UZM6Y2MXNOBT3TSGX26U4SZAL5Q7A, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615803147 (294h7m45.386444472s) [noise=true , address=104.36.82.8:4970 , piece_id=2YJN7VKTTTCMAWQW2MNR65DAJPFMHNIBIW5NWYNG4R6HNPRFH6NQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615998511 (294h7m45.386639836s) [noise=true , address=77.90.31.16:4733 , piece_id=3GQ7NEWE6WZJMQUA3GG3GSZYXEFNOAUEVWHOEH4TSNONYXUT43GQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616193414 (294h7m45.386834749s) [noise=true , address=77.90.31.19:4842 , piece_id=R4FTRBZ3PETCA6QGU6RPUEDZZBWXF7YVH2COEW4WH3YWDLU3VJMA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616449093 (294h7m45.387090429s) [noise=true , address=77.90.31.11:4554 , piece_id=4QRZ5MPQ7HTPRIVRL3O3H4OV7CQA7ESYT5RCPI3GTHXJOHOSAOVA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616635840 (294h7m45.387277175s) [noise=true , address=45.91.116.5:4298 , piece_id=TVTUGWHFTXB32GBY4LYXT3YDB5LIFXEGA7VO5BHFA24Y6EMDTPHQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616818790 (294h7m45.387460125s) [noise=true , address=104.36.82.13:4185 , piece_id=LTQS2CUZ5NRJCBOPEXDKUQ2IJGOIW3JJAPHWTCW7WUMQZPIWRXRA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:12.265123988 (294h7m47.035765313s) [upload=235227]: Next returned nil segment Jul 5 15:18:12.265138164 (294h7m47.035779499s) [upload=235227]: Waiting for error group managing segments... Jul 5 15:18:12.550465694 (294h7m47.321107019s) [noise=true , address=104.36.82.8:4970 , piece_id=2YJN7VKTTTCMAWQW2MNR65DAJPFMHNIBIW5NWYNG4R6HNPRFH6NQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.554315629 (294h7m47.324956954s) [noise=true , address=104.36.82.12:4149 , piece_id=YBTLQEWDMFK6JIFTYR3VJDNMXTS3NMWWHX2MPP53BM2UBDI3UUEA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.603452512 (294h7m47.374093847s) [noise=true , address=104.36.82.3:4767 , piece_id=OGHG4G5ZVC7GVY4XI6DO4KINXF7THHL6IHGDB6DAPDNUDOUFSC4A, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.639796643 (294h7m47.410437978s) [noise=true , address=104.36.82.9:4006 , piece_id=47FME2B5NS6WMOHC2ZGE2FZVSPEQUXKYT6E7AOSANAE5XZVDH5VQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.639894960 (294h7m47.410536285s) [noise=true , address=104.36.82.15:4259 , piece_id=LQKQ5MGKXHTPITBR2QAOQ5CX76GEVJGHJT3ZD3G4PFNN5IBTCJBQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.647222659 (294h7m47.417863994s) [noise=true , address=104.36.82.16:4295 , piece_id=VEL6CAUOJKMRN7A7M66ZPDLCMJVD555LZPHM53A24MDQJQOW7ECQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.665424010 (294h7m47.436065346s) [noise=true , address=104.36.82.13:4185 , piece_id=LTQS2CUZ5NRJCBOPEXDKUQ2IJGOIW3JJAPHWTCW7WUMQZPIWRXRA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.702209574 (294h7m47.47285091s) [noise=true , address=45.91.116.14:4191 , piece_id=WH2BCI36GGFF4QUD4XDYBBDMUR2R2FFVYKFTZD3ZWXPJCFJ3D7HQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.722073145 (294h7m47.49271448s) [noise=true , address=45.91.116.5:4298 , piece_id=TVTUGWHFTXB32GBY4LYXT3YDB5LIFXEGA7VO5BHFA24Y6EMDTPHQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.788621529 (294h7m47.559262864s) [noise=true , address=45.91.116.4:4937 , piece_id=YMKIHJET2EN3QHEK665ACZNQNHJ3UGE4H5T4DIHWJUHJQBYO7B7Q, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:36.752254721 (294h8m11.522896046s) [noise=true , address=77.90.31.19:4842 , piece_id=R4FTRBZ3PETCA6QGU6RPUEDZZBWXF7YVH2COEW4WH3YWDLU3VJMA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:37.783437003 (294h8m12.554078338s) [noise=true , address=77.90.31.17:4771 , piece_id=V7M65ZHCGXPD2GVJL6LTB4NP3QN6QLSIBM7U34QAYPAIKMN6EBPA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:39.733398290 (294h8m14.504039626s) [noise=true , address=77.90.31.18:4805 , piece_id=I23F6KYUQ7TJI4ZLEANACVNYJDMPKKMMACIZAIA6PC5S3CKQPV2Q, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Jul 5 15:18:39.733464387 (294h8m14.504105722s) [noise=true , address=77.90.31.21:4877 , piece_id=OJBIFN2TJPQ45LQLYCMPKV5UZM6Y2MXNOBT3TSGX26U4SZAL5Q7A, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: expected piece hash; context deadline exceeded; EOF Jul 5 15:18:39.733542065 (294h8m14.50418339s) [noise=true , address=77.90.31.11:4554 , piece_id=4QRZ5MPQ7HTPRIVRL3O3H4OV7CQA7ESYT5RCPI3GTHXJOHOSAOVA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection Jul 5 15:18:39.733547496 (294h8m14.504188831s) [noise=true , address=77.90.31.16:4733 , piece_id=3GQ7NEWE6WZJMQUA3GG3GSZYXEFNOAUEVWHOEH4TSNONYXUT43GQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection; write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection Jul 5 15:18:39.733583284 (294h8m14.504224619s) [seg_pos={0 0}, upload=235227]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 5 15:18:39.974321220 (294h8m14.744962555s) [upload=235227]: Done uploading segments... Jul 5 15:18:39.974353041 (294h8m14.744994376s) [upload=235227]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.745758449s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 5.843993466s FAIL (2025-07-05 15:24:54 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== JYE6FC5T5QIBUQIILGDHNGXAUDBUIAOSKYFMPBCWPHP2SDJ3A, seg_pos={0 0}, upload=235214]: Done uploading piece. err: <nil> Jul 5 15:17:12.446386719 (294h6m47.217028064s) [seg_pos={0 0}, upload=235214]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 5 15:17:12.446476771 (294h6m47.217118106s) [upload=235214]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 5 15:17:12.446483043 (294h6m47.217124378s) [upload=235214]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 5 15:17:12.905279085 (294h6m47.67592042s) [upload=235214]: Done uploading segments... Jul 5 15:17:12.905303211 (294h6m47.675944546s) [upload=235214]: Upload finished. err: <nil> Jul 5 15:18:10.381454266 (294h7m45.152095611s) [upload=235227]: Starting upload Jul 5 15:18:10.381524069 (294h7m45.152165414s) [upload=235227]: Done starting upload Jul 5 15:18:10.381621115 (294h7m45.15226245s) [upload=235227]: Uploading segments... Jul 5 15:18:10.381740133 (294h7m45.152381468s) [upload=235227]: Got next segment. Inline: false Jul 5 15:18:10.381776342 (294h7m45.152417677s) [upload=235227]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 5 15:18:10.381781973 (294h7m45.152423308s) [upload=235227]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 5 15:18:10.613904835 (294h7m45.38454617s) [seg_pos={0 0}, upload=235227]: Begin upload segment... Jul 5 15:18:10.613947397 (294h7m45.384588732s) [seg_pos={0 0}, upload=235227]: Done begin upload segment. Jul 5 15:18:10.613968548 (294h7m45.384609883s) [noise=true , address=104.36.82.15:4259 , piece_id=LQKQ5MGKXHTPITBR2QAOQ5CX76GEVJGHJT3ZD3G4PFNN5IBTCJBQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.613995038 (294h7m45.384636373s) [noise=true , address=104.36.82.16:4295 , piece_id=VEL6CAUOJKMRN7A7M66ZPDLCMJVD555LZPHM53A24MDQJQOW7ECQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614303969 (294h7m45.384945304s) [noise=true , address=104.36.82.9:4006 , piece_id=47FME2B5NS6WMOHC2ZGE2FZVSPEQUXKYT6E7AOSANAE5XZVDH5VQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614367721 (294h7m45.385009056s) [noise=true , address=45.91.116.4:4937 , piece_id=YMKIHJET2EN3QHEK665ACZNQNHJ3UGE4H5T4DIHWJUHJQBYO7B7Q, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614619272 (294h7m45.385260607s) [noise=true , address=77.90.31.17:4771 , piece_id=V7M65ZHCGXPD2GVJL6LTB4NP3QN6QLSIBM7U34QAYPAIKMN6EBPA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614637758 (294h7m45.385279083s) [noise=true , address=45.91.116.14:4191 , piece_id=WH2BCI36GGFF4QUD4XDYBBDMUR2R2FFVYKFTZD3ZWXPJCFJ3D7HQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614890251 (294h7m45.385531576s) [noise=true , address=104.36.82.3:4767 , piece_id=OGHG4G5ZVC7GVY4XI6DO4KINXF7THHL6IHGDB6DAPDNUDOUFSC4A, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615197719 (294h7m45.385839054s) [noise=true , address=104.36.82.12:4149 , piece_id=YBTLQEWDMFK6JIFTYR3VJDNMXTS3NMWWHX2MPP53BM2UBDI3UUEA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615413552 (294h7m45.386054887s) [noise=true , address=77.90.31.18:4805 , piece_id=I23F6KYUQ7TJI4ZLEANACVNYJDMPKKMMACIZAIA6PC5S3CKQPV2Q, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615608124 (294h7m45.386249459s) [noise=true , address=77.90.31.21:4877 , piece_id=OJBIFN2TJPQ45LQLYCMPKV5UZM6Y2MXNOBT3TSGX26U4SZAL5Q7A, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615803147 (294h7m45.386444472s) [noise=true , address=104.36.82.8:4970 , piece_id=2YJN7VKTTTCMAWQW2MNR65DAJPFMHNIBIW5NWYNG4R6HNPRFH6NQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615998511 (294h7m45.386639836s) [noise=true , address=77.90.31.16:4733 , piece_id=3GQ7NEWE6WZJMQUA3GG3GSZYXEFNOAUEVWHOEH4TSNONYXUT43GQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616193414 (294h7m45.386834749s) [noise=true , address=77.90.31.19:4842 , piece_id=R4FTRBZ3PETCA6QGU6RPUEDZZBWXF7YVH2COEW4WH3YWDLU3VJMA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616449093 (294h7m45.387090429s) [noise=true , address=77.90.31.11:4554 , piece_id=4QRZ5MPQ7HTPRIVRL3O3H4OV7CQA7ESYT5RCPI3GTHXJOHOSAOVA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616635840 (294h7m45.387277175s) [noise=true , address=45.91.116.5:4298 , piece_id=TVTUGWHFTXB32GBY4LYXT3YDB5LIFXEGA7VO5BHFA24Y6EMDTPHQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616818790 (294h7m45.387460125s) [noise=true , address=104.36.82.13:4185 , piece_id=LTQS2CUZ5NRJCBOPEXDKUQ2IJGOIW3JJAPHWTCW7WUMQZPIWRXRA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:12.265123988 (294h7m47.035765313s) [upload=235227]: Next returned nil segment Jul 5 15:18:12.265138164 (294h7m47.035779499s) [upload=235227]: Waiting for error group managing segments... Jul 5 15:18:12.550465694 (294h7m47.321107019s) [noise=true , address=104.36.82.8:4970 , piece_id=2YJN7VKTTTCMAWQW2MNR65DAJPFMHNIBIW5NWYNG4R6HNPRFH6NQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.554315629 (294h7m47.324956954s) [noise=true , address=104.36.82.12:4149 , piece_id=YBTLQEWDMFK6JIFTYR3VJDNMXTS3NMWWHX2MPP53BM2UBDI3UUEA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.603452512 (294h7m47.374093847s) [noise=true , address=104.36.82.3:4767 , piece_id=OGHG4G5ZVC7GVY4XI6DO4KINXF7THHL6IHGDB6DAPDNUDOUFSC4A, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.639796643 (294h7m47.410437978s) [noise=true , address=104.36.82.9:4006 , piece_id=47FME2B5NS6WMOHC2ZGE2FZVSPEQUXKYT6E7AOSANAE5XZVDH5VQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.639894960 (294h7m47.410536285s) [noise=true , address=104.36.82.15:4259 , piece_id=LQKQ5MGKXHTPITBR2QAOQ5CX76GEVJGHJT3ZD3G4PFNN5IBTCJBQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.647222659 (294h7m47.417863994s) [noise=true , address=104.36.82.16:4295 , piece_id=VEL6CAUOJKMRN7A7M66ZPDLCMJVD555LZPHM53A24MDQJQOW7ECQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.665424010 (294h7m47.436065346s) [noise=true , address=104.36.82.13:4185 , piece_id=LTQS2CUZ5NRJCBOPEXDKUQ2IJGOIW3JJAPHWTCW7WUMQZPIWRXRA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.702209574 (294h7m47.47285091s) [noise=true , address=45.91.116.14:4191 , piece_id=WH2BCI36GGFF4QUD4XDYBBDMUR2R2FFVYKFTZD3ZWXPJCFJ3D7HQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.722073145 (294h7m47.49271448s) [noise=true , address=45.91.116.5:4298 , piece_id=TVTUGWHFTXB32GBY4LYXT3YDB5LIFXEGA7VO5BHFA24Y6EMDTPHQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.788621529 (294h7m47.559262864s) [noise=true , address=45.91.116.4:4937 , piece_id=YMKIHJET2EN3QHEK665ACZNQNHJ3UGE4H5T4DIHWJUHJQBYO7B7Q, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:36.752254721 (294h8m11.522896046s) [noise=true , address=77.90.31.19:4842 , piece_id=R4FTRBZ3PETCA6QGU6RPUEDZZBWXF7YVH2COEW4WH3YWDLU3VJMA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:37.783437003 (294h8m12.554078338s) [noise=true , address=77.90.31.17:4771 , piece_id=V7M65ZHCGXPD2GVJL6LTB4NP3QN6QLSIBM7U34QAYPAIKMN6EBPA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:39.733398290 (294h8m14.504039626s) [noise=true , address=77.90.31.18:4805 , piece_id=I23F6KYUQ7TJI4ZLEANACVNYJDMPKKMMACIZAIA6PC5S3CKQPV2Q, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Jul 5 15:18:39.733464387 (294h8m14.504105722s) [noise=true , address=77.90.31.21:4877 , piece_id=OJBIFN2TJPQ45LQLYCMPKV5UZM6Y2MXNOBT3TSGX26U4SZAL5Q7A, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: expected piece hash; context deadline exceeded; EOF Jul 5 15:18:39.733542065 (294h8m14.50418339s) [noise=true , address=77.90.31.11:4554 , piece_id=4QRZ5MPQ7HTPRIVRL3O3H4OV7CQA7ESYT5RCPI3GTHXJOHOSAOVA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection Jul 5 15:18:39.733547496 (294h8m14.504188831s) [noise=true , address=77.90.31.16:4733 , piece_id=3GQ7NEWE6WZJMQUA3GG3GSZYXEFNOAUEVWHOEH4TSNONYXUT43GQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection; write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection Jul 5 15:18:39.733583284 (294h8m14.504224619s) [seg_pos={0 0}, upload=235227]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 5 15:18:39.974321220 (294h8m14.744962555s) [upload=235227]: Done uploading segments... Jul 5 15:18:39.974353041 (294h8m14.744994376s) [upload=235227]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.844022501s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 5.650977849s FAIL (2025-07-05 15:23:45 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== JYE6FC5T5QIBUQIILGDHNGXAUDBUIAOSKYFMPBCWPHP2SDJ3A, seg_pos={0 0}, upload=235214]: Done uploading piece. err: <nil> Jul 5 15:17:12.446386719 (294h6m47.217028064s) [seg_pos={0 0}, upload=235214]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 5 15:17:12.446476771 (294h6m47.217118106s) [upload=235214]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 5 15:17:12.446483043 (294h6m47.217124378s) [upload=235214]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 5 15:17:12.905279085 (294h6m47.67592042s) [upload=235214]: Done uploading segments... Jul 5 15:17:12.905303211 (294h6m47.675944546s) [upload=235214]: Upload finished. err: <nil> Jul 5 15:18:10.381454266 (294h7m45.152095611s) [upload=235227]: Starting upload Jul 5 15:18:10.381524069 (294h7m45.152165414s) [upload=235227]: Done starting upload Jul 5 15:18:10.381621115 (294h7m45.15226245s) [upload=235227]: Uploading segments... Jul 5 15:18:10.381740133 (294h7m45.152381468s) [upload=235227]: Got next segment. Inline: false Jul 5 15:18:10.381776342 (294h7m45.152417677s) [upload=235227]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 5 15:18:10.381781973 (294h7m45.152423308s) [upload=235227]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 5 15:18:10.613904835 (294h7m45.38454617s) [seg_pos={0 0}, upload=235227]: Begin upload segment... Jul 5 15:18:10.613947397 (294h7m45.384588732s) [seg_pos={0 0}, upload=235227]: Done begin upload segment. Jul 5 15:18:10.613968548 (294h7m45.384609883s) [noise=true , address=104.36.82.15:4259 , piece_id=LQKQ5MGKXHTPITBR2QAOQ5CX76GEVJGHJT3ZD3G4PFNN5IBTCJBQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.613995038 (294h7m45.384636373s) [noise=true , address=104.36.82.16:4295 , piece_id=VEL6CAUOJKMRN7A7M66ZPDLCMJVD555LZPHM53A24MDQJQOW7ECQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614303969 (294h7m45.384945304s) [noise=true , address=104.36.82.9:4006 , piece_id=47FME2B5NS6WMOHC2ZGE2FZVSPEQUXKYT6E7AOSANAE5XZVDH5VQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614367721 (294h7m45.385009056s) [noise=true , address=45.91.116.4:4937 , piece_id=YMKIHJET2EN3QHEK665ACZNQNHJ3UGE4H5T4DIHWJUHJQBYO7B7Q, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614619272 (294h7m45.385260607s) [noise=true , address=77.90.31.17:4771 , piece_id=V7M65ZHCGXPD2GVJL6LTB4NP3QN6QLSIBM7U34QAYPAIKMN6EBPA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614637758 (294h7m45.385279083s) [noise=true , address=45.91.116.14:4191 , piece_id=WH2BCI36GGFF4QUD4XDYBBDMUR2R2FFVYKFTZD3ZWXPJCFJ3D7HQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.614890251 (294h7m45.385531576s) [noise=true , address=104.36.82.3:4767 , piece_id=OGHG4G5ZVC7GVY4XI6DO4KINXF7THHL6IHGDB6DAPDNUDOUFSC4A, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615197719 (294h7m45.385839054s) [noise=true , address=104.36.82.12:4149 , piece_id=YBTLQEWDMFK6JIFTYR3VJDNMXTS3NMWWHX2MPP53BM2UBDI3UUEA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615413552 (294h7m45.386054887s) [noise=true , address=77.90.31.18:4805 , piece_id=I23F6KYUQ7TJI4ZLEANACVNYJDMPKKMMACIZAIA6PC5S3CKQPV2Q, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615608124 (294h7m45.386249459s) [noise=true , address=77.90.31.21:4877 , piece_id=OJBIFN2TJPQ45LQLYCMPKV5UZM6Y2MXNOBT3TSGX26U4SZAL5Q7A, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615803147 (294h7m45.386444472s) [noise=true , address=104.36.82.8:4970 , piece_id=2YJN7VKTTTCMAWQW2MNR65DAJPFMHNIBIW5NWYNG4R6HNPRFH6NQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.615998511 (294h7m45.386639836s) [noise=true , address=77.90.31.16:4733 , piece_id=3GQ7NEWE6WZJMQUA3GG3GSZYXEFNOAUEVWHOEH4TSNONYXUT43GQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616193414 (294h7m45.386834749s) [noise=true , address=77.90.31.19:4842 , piece_id=R4FTRBZ3PETCA6QGU6RPUEDZZBWXF7YVH2COEW4WH3YWDLU3VJMA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616449093 (294h7m45.387090429s) [noise=true , address=77.90.31.11:4554 , piece_id=4QRZ5MPQ7HTPRIVRL3O3H4OV7CQA7ESYT5RCPI3GTHXJOHOSAOVA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616635840 (294h7m45.387277175s) [noise=true , address=45.91.116.5:4298 , piece_id=TVTUGWHFTXB32GBY4LYXT3YDB5LIFXEGA7VO5BHFA24Y6EMDTPHQ, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:10.616818790 (294h7m45.387460125s) [noise=true , address=104.36.82.13:4185 , piece_id=LTQS2CUZ5NRJCBOPEXDKUQ2IJGOIW3JJAPHWTCW7WUMQZPIWRXRA, seg_pos={0 0}, upload=235227]: Uploading piece... Jul 5 15:18:12.265123988 (294h7m47.035765313s) [upload=235227]: Next returned nil segment Jul 5 15:18:12.265138164 (294h7m47.035779499s) [upload=235227]: Waiting for error group managing segments... Jul 5 15:18:12.550465694 (294h7m47.321107019s) [noise=true , address=104.36.82.8:4970 , piece_id=2YJN7VKTTTCMAWQW2MNR65DAJPFMHNIBIW5NWYNG4R6HNPRFH6NQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.554315629 (294h7m47.324956954s) [noise=true , address=104.36.82.12:4149 , piece_id=YBTLQEWDMFK6JIFTYR3VJDNMXTS3NMWWHX2MPP53BM2UBDI3UUEA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.603452512 (294h7m47.374093847s) [noise=true , address=104.36.82.3:4767 , piece_id=OGHG4G5ZVC7GVY4XI6DO4KINXF7THHL6IHGDB6DAPDNUDOUFSC4A, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.639796643 (294h7m47.410437978s) [noise=true , address=104.36.82.9:4006 , piece_id=47FME2B5NS6WMOHC2ZGE2FZVSPEQUXKYT6E7AOSANAE5XZVDH5VQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.639894960 (294h7m47.410536285s) [noise=true , address=104.36.82.15:4259 , piece_id=LQKQ5MGKXHTPITBR2QAOQ5CX76GEVJGHJT3ZD3G4PFNN5IBTCJBQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.647222659 (294h7m47.417863994s) [noise=true , address=104.36.82.16:4295 , piece_id=VEL6CAUOJKMRN7A7M66ZPDLCMJVD555LZPHM53A24MDQJQOW7ECQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.665424010 (294h7m47.436065346s) [noise=true , address=104.36.82.13:4185 , piece_id=LTQS2CUZ5NRJCBOPEXDKUQ2IJGOIW3JJAPHWTCW7WUMQZPIWRXRA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.702209574 (294h7m47.47285091s) [noise=true , address=45.91.116.14:4191 , piece_id=WH2BCI36GGFF4QUD4XDYBBDMUR2R2FFVYKFTZD3ZWXPJCFJ3D7HQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.722073145 (294h7m47.49271448s) [noise=true , address=45.91.116.5:4298 , piece_id=TVTUGWHFTXB32GBY4LYXT3YDB5LIFXEGA7VO5BHFA24Y6EMDTPHQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:12.788621529 (294h7m47.559262864s) [noise=true , address=45.91.116.4:4937 , piece_id=YMKIHJET2EN3QHEK665ACZNQNHJ3UGE4H5T4DIHWJUHJQBYO7B7Q, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:36.752254721 (294h8m11.522896046s) [noise=true , address=77.90.31.19:4842 , piece_id=R4FTRBZ3PETCA6QGU6RPUEDZZBWXF7YVH2COEW4WH3YWDLU3VJMA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:37.783437003 (294h8m12.554078338s) [noise=true , address=77.90.31.17:4771 , piece_id=V7M65ZHCGXPD2GVJL6LTB4NP3QN6QLSIBM7U34QAYPAIKMN6EBPA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: <nil> Jul 5 15:18:39.733398290 (294h8m14.504039626s) [noise=true , address=77.90.31.18:4805 , piece_id=I23F6KYUQ7TJI4ZLEANACVNYJDMPKKMMACIZAIA6PC5S3CKQPV2Q, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Jul 5 15:18:39.733464387 (294h8m14.504105722s) [noise=true , address=77.90.31.21:4877 , piece_id=OJBIFN2TJPQ45LQLYCMPKV5UZM6Y2MXNOBT3TSGX26U4SZAL5Q7A, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: expected piece hash; context deadline exceeded; EOF Jul 5 15:18:39.733542065 (294h8m14.50418339s) [noise=true , address=77.90.31.11:4554 , piece_id=4QRZ5MPQ7HTPRIVRL3O3H4OV7CQA7ESYT5RCPI3GTHXJOHOSAOVA, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38250->77.90.31.11:4554: use of closed network connection Jul 5 15:18:39.733547496 (294h8m14.504188831s) [noise=true , address=77.90.31.16:4733 , piece_id=3GQ7NEWE6WZJMQUA3GG3GSZYXEFNOAUEVWHOEH4TSNONYXUT43GQ, seg_pos={0 0}, upload=235227]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection; write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40656->77.90.31.16:4733: use of closed network connection Jul 5 15:18:39.733583284 (294h8m14.504224619s) [seg_pos={0 0}, upload=235227]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 5 15:18:39.974321220 (294h8m14.744962555s) [upload=235227]: Done uploading segments... Jul 5 15:18:39.974353041 (294h8m14.744994376s) [upload=235227]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.651002715s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)