Orbiter History

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [............X..] duration: 337.150296ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/47cd8584e3a0f3e FAIL (2025-11-09 16:04:10 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:56532->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 337.174963ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 9.559419003s trace: https://jaeger.dev.storj.io/trace/3846516a5cecd778 FAIL (2025-11-09 16:03:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 9 15:40:06.410809146 (7h36m4.962279607s) [upload=6714]: Starting upload Nov 9 15:40:06.411046298 (7h36m4.962516769s) [upload=6714]: Done starting upload Nov 9 15:40:06.411314729 (7h36m4.96278519s) [upload=6714]: Uploading segments... Nov 9 15:40:06.411410882 (7h36m4.962881383s) [upload=6714]: Got next segment. Inline: false Nov 9 15:40:06.411464574 (7h36m4.962935035s) [upload=6714]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 9 15:40:06.411475064 (7h36m4.962945555s) [upload=6714]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 9 15:40:06.664789367 (7h36m5.216259828s) [seg_pos={0 0}, upload=6714]: Begin upload segment... Nov 9 15:40:06.664851705 (7h36m5.216322166s) [seg_pos={0 0}, upload=6714]: Done begin upload segment. Nov 9 15:40:06.664891662 (7h36m5.216362123s) [noise=true , address=145.79.227.32:4835 , piece_id=EYIBNSVPQPKLKOBHKZI2VZHVCDKW3C3YJSFEWP4G62UTBXX2YRZA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.664996933 (7h36m5.216467394s) [noise=true , address=45.91.116.14:4190 , piece_id=XSIDZL5HMLZTQE7ZAPURAAUGQLPP3QOZK5BIE5PHRMZPSTN5APSA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.665698789 (7h36m5.21716925s) [noise=true , address=77.90.31.14:4662 , piece_id=HHSJDZPYKGCHURAZV46OQSGBU7D4TFPXREDBFPGTYF7OAUMT2X3A, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.666456111 (7h36m5.217926572s) [noise=true , address=145.79.227.25:4619 , piece_id=23D3MCZRVNPI5KB43ROAHKUALDGXLZBZU5E6EIU6U2FY3YQOVPGA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.666642225 (7h36m5.218112676s) [noise=true , address=145.79.227.17:4971 , piece_id=VTUI377UCF44ICILORHEKUO33ICRXM2FECO5EXS7RUDH4VGV7FJA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667027539 (7h36m5.218498s) [noise=true , address=145.79.227.9:4366 , piece_id=BT3GWQSXIMMSFPUYJC5EQ2LX2DS4BW5DAMW6ILLNVVRUUHZMOQUA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667041675 (7h36m5.218512126s) [noise=true , address=145.79.227.17:4972 , piece_id=AI3KVNUJKJUD33YVU2Z472R72SKQKRYQG65JPKFZ5WBJLKRLW6MA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667502794 (7h36m5.218973255s) [noise=true , address=145.79.227.13:4803 , piece_id=V37DZKBC4CHRGISXEQHTQAGKZFNXQKWXSWBSMHNRQTSJ32WYI2OQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667805441 (7h36m5.219275902s) [noise=true , address=145.79.227.9:4365 , piece_id=7BK6E45EY6Y3SV6PY3A4EVCS42PC4HYLXU3J7MN4HXRLDEFUEFCA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667895652 (7h36m5.219366113s) [noise=true , address=145.79.227.17:4970 , piece_id=5ZP5DDC3XAYQ5UFYJVQV6IZHB5KQQGNG54MTYW3UUYPN2IAK5QVQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.668240909 (7h36m5.219711361s) [noise=true , address=77.90.31.17:4772 , piece_id=CT6BNOYZGDBUEFVGM2RDS4ICMWPYS6N4IOUDA5V7MJQ3IKELNW2Q, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.668671640 (7h36m5.220142101s) [noise=true , address=145.79.227.31:4797 , piece_id=VQEEGJTLJY2IMCDHL2GMVGYEX4VL34TMMXKDYNERFYK3HRV7TC6A, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.669093554 (7h36m5.220564015s) [noise=true , address=145.79.227.21:4474 , piece_id=J5XE2MTCEW4ZHQDDZ4KCVYFK5EX6RQZARKNY4QCH4HUHGJO3SXDQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.669510097 (7h36m5.220980568s) [noise=true , address=77.90.31.18:4807 , piece_id=HDJAJYXNZNZN3LVAQSYVJEOQ3MU5H3BXCCW3NA2GLKPT7ATZHSJQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.671936887 (7h36m5.223407348s) [noise=true , address=145.79.227.20:4439 , piece_id=PPOVWSRAU7Y4CZAKAXV6ZP74EEPHONUC35O6XM57VE47NVJL5WZA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.672494549 (7h36m5.22396502s) [noise=true , address=77.90.31.21:4877 , piece_id=3V3Z6GNF32Q66EOS66WV3AGJNA7IXT5NHVAPKNUGVYU3R46IKKKA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:08.438197970 (7h36m6.989668451s) [upload=6714]: Next returned nil segment Nov 9 15:40:08.438219050 (7h36m6.989689521s) [upload=6714]: Waiting for error group managing segments... Nov 9 15:40:08.829503696 (7h36m7.380974147s) [noise=true , address=77.90.31.18:4807 , piece_id=HDJAJYXNZNZN3LVAQSYVJEOQ3MU5H3BXCCW3NA2GLKPT7ATZHSJQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.846923116 (7h36m7.398393577s) [noise=true , address=45.91.116.14:4190 , piece_id=XSIDZL5HMLZTQE7ZAPURAAUGQLPP3QOZK5BIE5PHRMZPSTN5APSA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.867005608 (7h36m7.418476069s) [noise=true , address=77.90.31.21:4877 , piece_id=3V3Z6GNF32Q66EOS66WV3AGJNA7IXT5NHVAPKNUGVYU3R46IKKKA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.888352807 (7h36m7.439823268s) [noise=true , address=77.90.31.14:4662 , piece_id=HHSJDZPYKGCHURAZV46OQSGBU7D4TFPXREDBFPGTYF7OAUMT2X3A, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.904680649 (7h36m7.45615111s) [noise=true , address=77.90.31.17:4772 , piece_id=CT6BNOYZGDBUEFVGM2RDS4ICMWPYS6N4IOUDA5V7MJQ3IKELNW2Q, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:18.663082162 (7h36m17.214552623s) [noise=true , address=145.79.227.17:4971 , piece_id=VTUI377UCF44ICILORHEKUO33ICRXM2FECO5EXS7RUDH4VGV7FJA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:20.077194714 (7h36m18.628665185s) [noise=true , address=145.79.227.13:4803 , piece_id=V37DZKBC4CHRGISXEQHTQAGKZFNXQKWXSWBSMHNRQTSJ32WYI2OQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:28.517900114 (7h36m27.069370575s) [noise=true , address=145.79.227.32:4835 , piece_id=EYIBNSVPQPKLKOBHKZI2VZHVCDKW3C3YJSFEWP4G62UTBXX2YRZA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:28.567571990 (7h36m27.119042451s) [noise=true , address=145.79.227.31:4797 , piece_id=VQEEGJTLJY2IMCDHL2GMVGYEX4VL34TMMXKDYNERFYK3HRV7TC6A, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:33.651382593 (7h36m32.202853044s) [noise=true , address=145.79.227.20:4439 , piece_id=PPOVWSRAU7Y4CZAKAXV6ZP74EEPHONUC35O6XM57VE47NVJL5WZA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:34.842221635 (7h36m33.393692096s) [noise=true , address=145.79.227.9:4365 , piece_id=7BK6E45EY6Y3SV6PY3A4EVCS42PC4HYLXU3J7MN4HXRLDEFUEFCA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:35.601406517 (7h36m34.152877008s) [noise=true , address=145.79.227.25:4619 , piece_id=23D3MCZRVNPI5KB43ROAHKUALDGXLZBZU5E6EIU6U2FY3YQOVPGA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:35.697082303 (7h36m34.248552774s) [noise=true , address=145.79.227.17:4970 , piece_id=5ZP5DDC3XAYQ5UFYJVQV6IZHB5KQQGNG54MTYW3UUYPN2IAK5QVQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697252518 (7h36m34.248722969s) [noise=true , address=145.79.227.17:4972 , piece_id=AI3KVNUJKJUD33YVU2Z472R72SKQKRYQG65JPKFZ5WBJLKRLW6MA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697302322 (7h36m34.248772773s) [noise=true , address=145.79.227.21:4474 , piece_id=J5XE2MTCEW4ZHQDDZ4KCVYFK5EX6RQZARKNY4QCH4HUHGJO3SXDQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, address:145.79.227.21:4474): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697472546 (7h36m34.248943007s) [noise=true , address=145.79.227.9:4366 , piece_id=BT3GWQSXIMMSFPUYJC5EQ2LX2DS4BW5DAMW6ILLNVVRUUHZMOQUA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697514686 (7h36m34.248985147s) [seg_pos={0 0}, upload=6714]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 9 15:40:35.950211122 (7h36m34.501681583s) [upload=6714]: Done uploading segments... Nov 9 15:40:35.950416643 (7h36m34.501887104s) [upload=6714]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 9.559450752s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [.............X.] duration: 314.448196ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/47cd8584e3a0f3e FAIL (2025-11-09 16:03:28 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:56532->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 314.483261ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 9.774632894s trace: https://jaeger.dev.storj.io/trace/3846516a5cecd778 FAIL (2025-11-09 16:02:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 9 15:40:06.410809146 (7h36m4.962279607s) [upload=6714]: Starting upload Nov 9 15:40:06.411046298 (7h36m4.962516769s) [upload=6714]: Done starting upload Nov 9 15:40:06.411314729 (7h36m4.96278519s) [upload=6714]: Uploading segments... Nov 9 15:40:06.411410882 (7h36m4.962881383s) [upload=6714]: Got next segment. Inline: false Nov 9 15:40:06.411464574 (7h36m4.962935035s) [upload=6714]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 9 15:40:06.411475064 (7h36m4.962945555s) [upload=6714]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 9 15:40:06.664789367 (7h36m5.216259828s) [seg_pos={0 0}, upload=6714]: Begin upload segment... Nov 9 15:40:06.664851705 (7h36m5.216322166s) [seg_pos={0 0}, upload=6714]: Done begin upload segment. Nov 9 15:40:06.664891662 (7h36m5.216362123s) [noise=true , address=145.79.227.32:4835 , piece_id=EYIBNSVPQPKLKOBHKZI2VZHVCDKW3C3YJSFEWP4G62UTBXX2YRZA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.664996933 (7h36m5.216467394s) [noise=true , address=45.91.116.14:4190 , piece_id=XSIDZL5HMLZTQE7ZAPURAAUGQLPP3QOZK5BIE5PHRMZPSTN5APSA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.665698789 (7h36m5.21716925s) [noise=true , address=77.90.31.14:4662 , piece_id=HHSJDZPYKGCHURAZV46OQSGBU7D4TFPXREDBFPGTYF7OAUMT2X3A, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.666456111 (7h36m5.217926572s) [noise=true , address=145.79.227.25:4619 , piece_id=23D3MCZRVNPI5KB43ROAHKUALDGXLZBZU5E6EIU6U2FY3YQOVPGA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.666642225 (7h36m5.218112676s) [noise=true , address=145.79.227.17:4971 , piece_id=VTUI377UCF44ICILORHEKUO33ICRXM2FECO5EXS7RUDH4VGV7FJA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667027539 (7h36m5.218498s) [noise=true , address=145.79.227.9:4366 , piece_id=BT3GWQSXIMMSFPUYJC5EQ2LX2DS4BW5DAMW6ILLNVVRUUHZMOQUA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667041675 (7h36m5.218512126s) [noise=true , address=145.79.227.17:4972 , piece_id=AI3KVNUJKJUD33YVU2Z472R72SKQKRYQG65JPKFZ5WBJLKRLW6MA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667502794 (7h36m5.218973255s) [noise=true , address=145.79.227.13:4803 , piece_id=V37DZKBC4CHRGISXEQHTQAGKZFNXQKWXSWBSMHNRQTSJ32WYI2OQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667805441 (7h36m5.219275902s) [noise=true , address=145.79.227.9:4365 , piece_id=7BK6E45EY6Y3SV6PY3A4EVCS42PC4HYLXU3J7MN4HXRLDEFUEFCA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667895652 (7h36m5.219366113s) [noise=true , address=145.79.227.17:4970 , piece_id=5ZP5DDC3XAYQ5UFYJVQV6IZHB5KQQGNG54MTYW3UUYPN2IAK5QVQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.668240909 (7h36m5.219711361s) [noise=true , address=77.90.31.17:4772 , piece_id=CT6BNOYZGDBUEFVGM2RDS4ICMWPYS6N4IOUDA5V7MJQ3IKELNW2Q, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.668671640 (7h36m5.220142101s) [noise=true , address=145.79.227.31:4797 , piece_id=VQEEGJTLJY2IMCDHL2GMVGYEX4VL34TMMXKDYNERFYK3HRV7TC6A, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.669093554 (7h36m5.220564015s) [noise=true , address=145.79.227.21:4474 , piece_id=J5XE2MTCEW4ZHQDDZ4KCVYFK5EX6RQZARKNY4QCH4HUHGJO3SXDQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.669510097 (7h36m5.220980568s) [noise=true , address=77.90.31.18:4807 , piece_id=HDJAJYXNZNZN3LVAQSYVJEOQ3MU5H3BXCCW3NA2GLKPT7ATZHSJQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.671936887 (7h36m5.223407348s) [noise=true , address=145.79.227.20:4439 , piece_id=PPOVWSRAU7Y4CZAKAXV6ZP74EEPHONUC35O6XM57VE47NVJL5WZA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.672494549 (7h36m5.22396502s) [noise=true , address=77.90.31.21:4877 , piece_id=3V3Z6GNF32Q66EOS66WV3AGJNA7IXT5NHVAPKNUGVYU3R46IKKKA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:08.438197970 (7h36m6.989668451s) [upload=6714]: Next returned nil segment Nov 9 15:40:08.438219050 (7h36m6.989689521s) [upload=6714]: Waiting for error group managing segments... Nov 9 15:40:08.829503696 (7h36m7.380974147s) [noise=true , address=77.90.31.18:4807 , piece_id=HDJAJYXNZNZN3LVAQSYVJEOQ3MU5H3BXCCW3NA2GLKPT7ATZHSJQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.846923116 (7h36m7.398393577s) [noise=true , address=45.91.116.14:4190 , piece_id=XSIDZL5HMLZTQE7ZAPURAAUGQLPP3QOZK5BIE5PHRMZPSTN5APSA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.867005608 (7h36m7.418476069s) [noise=true , address=77.90.31.21:4877 , piece_id=3V3Z6GNF32Q66EOS66WV3AGJNA7IXT5NHVAPKNUGVYU3R46IKKKA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.888352807 (7h36m7.439823268s) [noise=true , address=77.90.31.14:4662 , piece_id=HHSJDZPYKGCHURAZV46OQSGBU7D4TFPXREDBFPGTYF7OAUMT2X3A, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.904680649 (7h36m7.45615111s) [noise=true , address=77.90.31.17:4772 , piece_id=CT6BNOYZGDBUEFVGM2RDS4ICMWPYS6N4IOUDA5V7MJQ3IKELNW2Q, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:18.663082162 (7h36m17.214552623s) [noise=true , address=145.79.227.17:4971 , piece_id=VTUI377UCF44ICILORHEKUO33ICRXM2FECO5EXS7RUDH4VGV7FJA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:20.077194714 (7h36m18.628665185s) [noise=true , address=145.79.227.13:4803 , piece_id=V37DZKBC4CHRGISXEQHTQAGKZFNXQKWXSWBSMHNRQTSJ32WYI2OQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:28.517900114 (7h36m27.069370575s) [noise=true , address=145.79.227.32:4835 , piece_id=EYIBNSVPQPKLKOBHKZI2VZHVCDKW3C3YJSFEWP4G62UTBXX2YRZA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:28.567571990 (7h36m27.119042451s) [noise=true , address=145.79.227.31:4797 , piece_id=VQEEGJTLJY2IMCDHL2GMVGYEX4VL34TMMXKDYNERFYK3HRV7TC6A, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:33.651382593 (7h36m32.202853044s) [noise=true , address=145.79.227.20:4439 , piece_id=PPOVWSRAU7Y4CZAKAXV6ZP74EEPHONUC35O6XM57VE47NVJL5WZA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:34.842221635 (7h36m33.393692096s) [noise=true , address=145.79.227.9:4365 , piece_id=7BK6E45EY6Y3SV6PY3A4EVCS42PC4HYLXU3J7MN4HXRLDEFUEFCA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:35.601406517 (7h36m34.152877008s) [noise=true , address=145.79.227.25:4619 , piece_id=23D3MCZRVNPI5KB43ROAHKUALDGXLZBZU5E6EIU6U2FY3YQOVPGA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:35.697082303 (7h36m34.248552774s) [noise=true , address=145.79.227.17:4970 , piece_id=5ZP5DDC3XAYQ5UFYJVQV6IZHB5KQQGNG54MTYW3UUYPN2IAK5QVQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697252518 (7h36m34.248722969s) [noise=true , address=145.79.227.17:4972 , piece_id=AI3KVNUJKJUD33YVU2Z472R72SKQKRYQG65JPKFZ5WBJLKRLW6MA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697302322 (7h36m34.248772773s) [noise=true , address=145.79.227.21:4474 , piece_id=J5XE2MTCEW4ZHQDDZ4KCVYFK5EX6RQZARKNY4QCH4HUHGJO3SXDQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, address:145.79.227.21:4474): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697472546 (7h36m34.248943007s) [noise=true , address=145.79.227.9:4366 , piece_id=BT3GWQSXIMMSFPUYJC5EQ2LX2DS4BW5DAMW6ILLNVVRUUHZMOQUA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697514686 (7h36m34.248985147s) [seg_pos={0 0}, upload=6714]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 9 15:40:35.950211122 (7h36m34.501681583s) [upload=6714]: Done uploading segments... Nov 9 15:40:35.950416643 (7h36m34.501887104s) [upload=6714]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 9.774665114s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [..............X] duration: 222.175621ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/47cd8584e3a0f3e FAIL (2025-11-09 16:02:14 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:56532->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 222.209144ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 10.397100697s trace: https://jaeger.dev.storj.io/trace/3846516a5cecd778 FAIL (2025-11-09 16:01:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 9 15:40:06.410809146 (7h36m4.962279607s) [upload=6714]: Starting upload Nov 9 15:40:06.411046298 (7h36m4.962516769s) [upload=6714]: Done starting upload Nov 9 15:40:06.411314729 (7h36m4.96278519s) [upload=6714]: Uploading segments... Nov 9 15:40:06.411410882 (7h36m4.962881383s) [upload=6714]: Got next segment. Inline: false Nov 9 15:40:06.411464574 (7h36m4.962935035s) [upload=6714]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 9 15:40:06.411475064 (7h36m4.962945555s) [upload=6714]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 9 15:40:06.664789367 (7h36m5.216259828s) [seg_pos={0 0}, upload=6714]: Begin upload segment... Nov 9 15:40:06.664851705 (7h36m5.216322166s) [seg_pos={0 0}, upload=6714]: Done begin upload segment. Nov 9 15:40:06.664891662 (7h36m5.216362123s) [noise=true , address=145.79.227.32:4835 , piece_id=EYIBNSVPQPKLKOBHKZI2VZHVCDKW3C3YJSFEWP4G62UTBXX2YRZA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.664996933 (7h36m5.216467394s) [noise=true , address=45.91.116.14:4190 , piece_id=XSIDZL5HMLZTQE7ZAPURAAUGQLPP3QOZK5BIE5PHRMZPSTN5APSA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.665698789 (7h36m5.21716925s) [noise=true , address=77.90.31.14:4662 , piece_id=HHSJDZPYKGCHURAZV46OQSGBU7D4TFPXREDBFPGTYF7OAUMT2X3A, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.666456111 (7h36m5.217926572s) [noise=true , address=145.79.227.25:4619 , piece_id=23D3MCZRVNPI5KB43ROAHKUALDGXLZBZU5E6EIU6U2FY3YQOVPGA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.666642225 (7h36m5.218112676s) [noise=true , address=145.79.227.17:4971 , piece_id=VTUI377UCF44ICILORHEKUO33ICRXM2FECO5EXS7RUDH4VGV7FJA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667027539 (7h36m5.218498s) [noise=true , address=145.79.227.9:4366 , piece_id=BT3GWQSXIMMSFPUYJC5EQ2LX2DS4BW5DAMW6ILLNVVRUUHZMOQUA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667041675 (7h36m5.218512126s) [noise=true , address=145.79.227.17:4972 , piece_id=AI3KVNUJKJUD33YVU2Z472R72SKQKRYQG65JPKFZ5WBJLKRLW6MA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667502794 (7h36m5.218973255s) [noise=true , address=145.79.227.13:4803 , piece_id=V37DZKBC4CHRGISXEQHTQAGKZFNXQKWXSWBSMHNRQTSJ32WYI2OQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667805441 (7h36m5.219275902s) [noise=true , address=145.79.227.9:4365 , piece_id=7BK6E45EY6Y3SV6PY3A4EVCS42PC4HYLXU3J7MN4HXRLDEFUEFCA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667895652 (7h36m5.219366113s) [noise=true , address=145.79.227.17:4970 , piece_id=5ZP5DDC3XAYQ5UFYJVQV6IZHB5KQQGNG54MTYW3UUYPN2IAK5QVQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.668240909 (7h36m5.219711361s) [noise=true , address=77.90.31.17:4772 , piece_id=CT6BNOYZGDBUEFVGM2RDS4ICMWPYS6N4IOUDA5V7MJQ3IKELNW2Q, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.668671640 (7h36m5.220142101s) [noise=true , address=145.79.227.31:4797 , piece_id=VQEEGJTLJY2IMCDHL2GMVGYEX4VL34TMMXKDYNERFYK3HRV7TC6A, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.669093554 (7h36m5.220564015s) [noise=true , address=145.79.227.21:4474 , piece_id=J5XE2MTCEW4ZHQDDZ4KCVYFK5EX6RQZARKNY4QCH4HUHGJO3SXDQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.669510097 (7h36m5.220980568s) [noise=true , address=77.90.31.18:4807 , piece_id=HDJAJYXNZNZN3LVAQSYVJEOQ3MU5H3BXCCW3NA2GLKPT7ATZHSJQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.671936887 (7h36m5.223407348s) [noise=true , address=145.79.227.20:4439 , piece_id=PPOVWSRAU7Y4CZAKAXV6ZP74EEPHONUC35O6XM57VE47NVJL5WZA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.672494549 (7h36m5.22396502s) [noise=true , address=77.90.31.21:4877 , piece_id=3V3Z6GNF32Q66EOS66WV3AGJNA7IXT5NHVAPKNUGVYU3R46IKKKA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:08.438197970 (7h36m6.989668451s) [upload=6714]: Next returned nil segment Nov 9 15:40:08.438219050 (7h36m6.989689521s) [upload=6714]: Waiting for error group managing segments... Nov 9 15:40:08.829503696 (7h36m7.380974147s) [noise=true , address=77.90.31.18:4807 , piece_id=HDJAJYXNZNZN3LVAQSYVJEOQ3MU5H3BXCCW3NA2GLKPT7ATZHSJQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.846923116 (7h36m7.398393577s) [noise=true , address=45.91.116.14:4190 , piece_id=XSIDZL5HMLZTQE7ZAPURAAUGQLPP3QOZK5BIE5PHRMZPSTN5APSA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.867005608 (7h36m7.418476069s) [noise=true , address=77.90.31.21:4877 , piece_id=3V3Z6GNF32Q66EOS66WV3AGJNA7IXT5NHVAPKNUGVYU3R46IKKKA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.888352807 (7h36m7.439823268s) [noise=true , address=77.90.31.14:4662 , piece_id=HHSJDZPYKGCHURAZV46OQSGBU7D4TFPXREDBFPGTYF7OAUMT2X3A, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.904680649 (7h36m7.45615111s) [noise=true , address=77.90.31.17:4772 , piece_id=CT6BNOYZGDBUEFVGM2RDS4ICMWPYS6N4IOUDA5V7MJQ3IKELNW2Q, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:18.663082162 (7h36m17.214552623s) [noise=true , address=145.79.227.17:4971 , piece_id=VTUI377UCF44ICILORHEKUO33ICRXM2FECO5EXS7RUDH4VGV7FJA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:20.077194714 (7h36m18.628665185s) [noise=true , address=145.79.227.13:4803 , piece_id=V37DZKBC4CHRGISXEQHTQAGKZFNXQKWXSWBSMHNRQTSJ32WYI2OQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:28.517900114 (7h36m27.069370575s) [noise=true , address=145.79.227.32:4835 , piece_id=EYIBNSVPQPKLKOBHKZI2VZHVCDKW3C3YJSFEWP4G62UTBXX2YRZA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:28.567571990 (7h36m27.119042451s) [noise=true , address=145.79.227.31:4797 , piece_id=VQEEGJTLJY2IMCDHL2GMVGYEX4VL34TMMXKDYNERFYK3HRV7TC6A, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:33.651382593 (7h36m32.202853044s) [noise=true , address=145.79.227.20:4439 , piece_id=PPOVWSRAU7Y4CZAKAXV6ZP74EEPHONUC35O6XM57VE47NVJL5WZA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:34.842221635 (7h36m33.393692096s) [noise=true , address=145.79.227.9:4365 , piece_id=7BK6E45EY6Y3SV6PY3A4EVCS42PC4HYLXU3J7MN4HXRLDEFUEFCA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:35.601406517 (7h36m34.152877008s) [noise=true , address=145.79.227.25:4619 , piece_id=23D3MCZRVNPI5KB43ROAHKUALDGXLZBZU5E6EIU6U2FY3YQOVPGA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:35.697082303 (7h36m34.248552774s) [noise=true , address=145.79.227.17:4970 , piece_id=5ZP5DDC3XAYQ5UFYJVQV6IZHB5KQQGNG54MTYW3UUYPN2IAK5QVQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697252518 (7h36m34.248722969s) [noise=true , address=145.79.227.17:4972 , piece_id=AI3KVNUJKJUD33YVU2Z472R72SKQKRYQG65JPKFZ5WBJLKRLW6MA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697302322 (7h36m34.248772773s) [noise=true , address=145.79.227.21:4474 , piece_id=J5XE2MTCEW4ZHQDDZ4KCVYFK5EX6RQZARKNY4QCH4HUHGJO3SXDQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, address:145.79.227.21:4474): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697472546 (7h36m34.248943007s) [noise=true , address=145.79.227.9:4366 , piece_id=BT3GWQSXIMMSFPUYJC5EQ2LX2DS4BW5DAMW6ILLNVVRUUHZMOQUA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697514686 (7h36m34.248985147s) [seg_pos={0 0}, upload=6714]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 9 15:40:35.950211122 (7h36m34.501681583s) [upload=6714]: Done uploading segments... Nov 9 15:40:35.950416643 (7h36m34.501887104s) [upload=6714]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 10.397143128s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 11.256316294s trace: https://jaeger.dev.storj.io/trace/3846516a5cecd778 FAIL (2025-11-09 16:00:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 9 15:40:06.410809146 (7h36m4.962279607s) [upload=6714]: Starting upload Nov 9 15:40:06.411046298 (7h36m4.962516769s) [upload=6714]: Done starting upload Nov 9 15:40:06.411314729 (7h36m4.96278519s) [upload=6714]: Uploading segments... Nov 9 15:40:06.411410882 (7h36m4.962881383s) [upload=6714]: Got next segment. Inline: false Nov 9 15:40:06.411464574 (7h36m4.962935035s) [upload=6714]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 9 15:40:06.411475064 (7h36m4.962945555s) [upload=6714]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 9 15:40:06.664789367 (7h36m5.216259828s) [seg_pos={0 0}, upload=6714]: Begin upload segment... Nov 9 15:40:06.664851705 (7h36m5.216322166s) [seg_pos={0 0}, upload=6714]: Done begin upload segment. Nov 9 15:40:06.664891662 (7h36m5.216362123s) [noise=true , address=145.79.227.32:4835 , piece_id=EYIBNSVPQPKLKOBHKZI2VZHVCDKW3C3YJSFEWP4G62UTBXX2YRZA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.664996933 (7h36m5.216467394s) [noise=true , address=45.91.116.14:4190 , piece_id=XSIDZL5HMLZTQE7ZAPURAAUGQLPP3QOZK5BIE5PHRMZPSTN5APSA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.665698789 (7h36m5.21716925s) [noise=true , address=77.90.31.14:4662 , piece_id=HHSJDZPYKGCHURAZV46OQSGBU7D4TFPXREDBFPGTYF7OAUMT2X3A, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.666456111 (7h36m5.217926572s) [noise=true , address=145.79.227.25:4619 , piece_id=23D3MCZRVNPI5KB43ROAHKUALDGXLZBZU5E6EIU6U2FY3YQOVPGA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.666642225 (7h36m5.218112676s) [noise=true , address=145.79.227.17:4971 , piece_id=VTUI377UCF44ICILORHEKUO33ICRXM2FECO5EXS7RUDH4VGV7FJA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667027539 (7h36m5.218498s) [noise=true , address=145.79.227.9:4366 , piece_id=BT3GWQSXIMMSFPUYJC5EQ2LX2DS4BW5DAMW6ILLNVVRUUHZMOQUA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667041675 (7h36m5.218512126s) [noise=true , address=145.79.227.17:4972 , piece_id=AI3KVNUJKJUD33YVU2Z472R72SKQKRYQG65JPKFZ5WBJLKRLW6MA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667502794 (7h36m5.218973255s) [noise=true , address=145.79.227.13:4803 , piece_id=V37DZKBC4CHRGISXEQHTQAGKZFNXQKWXSWBSMHNRQTSJ32WYI2OQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667805441 (7h36m5.219275902s) [noise=true , address=145.79.227.9:4365 , piece_id=7BK6E45EY6Y3SV6PY3A4EVCS42PC4HYLXU3J7MN4HXRLDEFUEFCA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667895652 (7h36m5.219366113s) [noise=true , address=145.79.227.17:4970 , piece_id=5ZP5DDC3XAYQ5UFYJVQV6IZHB5KQQGNG54MTYW3UUYPN2IAK5QVQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.668240909 (7h36m5.219711361s) [noise=true , address=77.90.31.17:4772 , piece_id=CT6BNOYZGDBUEFVGM2RDS4ICMWPYS6N4IOUDA5V7MJQ3IKELNW2Q, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.668671640 (7h36m5.220142101s) [noise=true , address=145.79.227.31:4797 , piece_id=VQEEGJTLJY2IMCDHL2GMVGYEX4VL34TMMXKDYNERFYK3HRV7TC6A, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.669093554 (7h36m5.220564015s) [noise=true , address=145.79.227.21:4474 , piece_id=J5XE2MTCEW4ZHQDDZ4KCVYFK5EX6RQZARKNY4QCH4HUHGJO3SXDQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.669510097 (7h36m5.220980568s) [noise=true , address=77.90.31.18:4807 , piece_id=HDJAJYXNZNZN3LVAQSYVJEOQ3MU5H3BXCCW3NA2GLKPT7ATZHSJQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.671936887 (7h36m5.223407348s) [noise=true , address=145.79.227.20:4439 , piece_id=PPOVWSRAU7Y4CZAKAXV6ZP74EEPHONUC35O6XM57VE47NVJL5WZA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.672494549 (7h36m5.22396502s) [noise=true , address=77.90.31.21:4877 , piece_id=3V3Z6GNF32Q66EOS66WV3AGJNA7IXT5NHVAPKNUGVYU3R46IKKKA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:08.438197970 (7h36m6.989668451s) [upload=6714]: Next returned nil segment Nov 9 15:40:08.438219050 (7h36m6.989689521s) [upload=6714]: Waiting for error group managing segments... Nov 9 15:40:08.829503696 (7h36m7.380974147s) [noise=true , address=77.90.31.18:4807 , piece_id=HDJAJYXNZNZN3LVAQSYVJEOQ3MU5H3BXCCW3NA2GLKPT7ATZHSJQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.846923116 (7h36m7.398393577s) [noise=true , address=45.91.116.14:4190 , piece_id=XSIDZL5HMLZTQE7ZAPURAAUGQLPP3QOZK5BIE5PHRMZPSTN5APSA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.867005608 (7h36m7.418476069s) [noise=true , address=77.90.31.21:4877 , piece_id=3V3Z6GNF32Q66EOS66WV3AGJNA7IXT5NHVAPKNUGVYU3R46IKKKA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.888352807 (7h36m7.439823268s) [noise=true , address=77.90.31.14:4662 , piece_id=HHSJDZPYKGCHURAZV46OQSGBU7D4TFPXREDBFPGTYF7OAUMT2X3A, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.904680649 (7h36m7.45615111s) [noise=true , address=77.90.31.17:4772 , piece_id=CT6BNOYZGDBUEFVGM2RDS4ICMWPYS6N4IOUDA5V7MJQ3IKELNW2Q, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:18.663082162 (7h36m17.214552623s) [noise=true , address=145.79.227.17:4971 , piece_id=VTUI377UCF44ICILORHEKUO33ICRXM2FECO5EXS7RUDH4VGV7FJA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:20.077194714 (7h36m18.628665185s) [noise=true , address=145.79.227.13:4803 , piece_id=V37DZKBC4CHRGISXEQHTQAGKZFNXQKWXSWBSMHNRQTSJ32WYI2OQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:28.517900114 (7h36m27.069370575s) [noise=true , address=145.79.227.32:4835 , piece_id=EYIBNSVPQPKLKOBHKZI2VZHVCDKW3C3YJSFEWP4G62UTBXX2YRZA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:28.567571990 (7h36m27.119042451s) [noise=true , address=145.79.227.31:4797 , piece_id=VQEEGJTLJY2IMCDHL2GMVGYEX4VL34TMMXKDYNERFYK3HRV7TC6A, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:33.651382593 (7h36m32.202853044s) [noise=true , address=145.79.227.20:4439 , piece_id=PPOVWSRAU7Y4CZAKAXV6ZP74EEPHONUC35O6XM57VE47NVJL5WZA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:34.842221635 (7h36m33.393692096s) [noise=true , address=145.79.227.9:4365 , piece_id=7BK6E45EY6Y3SV6PY3A4EVCS42PC4HYLXU3J7MN4HXRLDEFUEFCA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:35.601406517 (7h36m34.152877008s) [noise=true , address=145.79.227.25:4619 , piece_id=23D3MCZRVNPI5KB43ROAHKUALDGXLZBZU5E6EIU6U2FY3YQOVPGA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:35.697082303 (7h36m34.248552774s) [noise=true , address=145.79.227.17:4970 , piece_id=5ZP5DDC3XAYQ5UFYJVQV6IZHB5KQQGNG54MTYW3UUYPN2IAK5QVQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697252518 (7h36m34.248722969s) [noise=true , address=145.79.227.17:4972 , piece_id=AI3KVNUJKJUD33YVU2Z472R72SKQKRYQG65JPKFZ5WBJLKRLW6MA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697302322 (7h36m34.248772773s) [noise=true , address=145.79.227.21:4474 , piece_id=J5XE2MTCEW4ZHQDDZ4KCVYFK5EX6RQZARKNY4QCH4HUHGJO3SXDQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, address:145.79.227.21:4474): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697472546 (7h36m34.248943007s) [noise=true , address=145.79.227.9:4366 , piece_id=BT3GWQSXIMMSFPUYJC5EQ2LX2DS4BW5DAMW6ILLNVVRUUHZMOQUA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697514686 (7h36m34.248985147s) [seg_pos={0 0}, upload=6714]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 9 15:40:35.950211122 (7h36m34.501681583s) [upload=6714]: Done uploading segments... Nov 9 15:40:35.950416643 (7h36m34.501887104s) [upload=6714]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 11.256352753s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 8.600107249s trace: https://jaeger.dev.storj.io/trace/3846516a5cecd778 FAIL (2025-11-09 15:59:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 9 15:40:06.410809146 (7h36m4.962279607s) [upload=6714]: Starting upload Nov 9 15:40:06.411046298 (7h36m4.962516769s) [upload=6714]: Done starting upload Nov 9 15:40:06.411314729 (7h36m4.96278519s) [upload=6714]: Uploading segments... Nov 9 15:40:06.411410882 (7h36m4.962881383s) [upload=6714]: Got next segment. Inline: false Nov 9 15:40:06.411464574 (7h36m4.962935035s) [upload=6714]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 9 15:40:06.411475064 (7h36m4.962945555s) [upload=6714]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 9 15:40:06.664789367 (7h36m5.216259828s) [seg_pos={0 0}, upload=6714]: Begin upload segment... Nov 9 15:40:06.664851705 (7h36m5.216322166s) [seg_pos={0 0}, upload=6714]: Done begin upload segment. Nov 9 15:40:06.664891662 (7h36m5.216362123s) [noise=true , address=145.79.227.32:4835 , piece_id=EYIBNSVPQPKLKOBHKZI2VZHVCDKW3C3YJSFEWP4G62UTBXX2YRZA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.664996933 (7h36m5.216467394s) [noise=true , address=45.91.116.14:4190 , piece_id=XSIDZL5HMLZTQE7ZAPURAAUGQLPP3QOZK5BIE5PHRMZPSTN5APSA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.665698789 (7h36m5.21716925s) [noise=true , address=77.90.31.14:4662 , piece_id=HHSJDZPYKGCHURAZV46OQSGBU7D4TFPXREDBFPGTYF7OAUMT2X3A, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.666456111 (7h36m5.217926572s) [noise=true , address=145.79.227.25:4619 , piece_id=23D3MCZRVNPI5KB43ROAHKUALDGXLZBZU5E6EIU6U2FY3YQOVPGA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.666642225 (7h36m5.218112676s) [noise=true , address=145.79.227.17:4971 , piece_id=VTUI377UCF44ICILORHEKUO33ICRXM2FECO5EXS7RUDH4VGV7FJA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667027539 (7h36m5.218498s) [noise=true , address=145.79.227.9:4366 , piece_id=BT3GWQSXIMMSFPUYJC5EQ2LX2DS4BW5DAMW6ILLNVVRUUHZMOQUA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667041675 (7h36m5.218512126s) [noise=true , address=145.79.227.17:4972 , piece_id=AI3KVNUJKJUD33YVU2Z472R72SKQKRYQG65JPKFZ5WBJLKRLW6MA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667502794 (7h36m5.218973255s) [noise=true , address=145.79.227.13:4803 , piece_id=V37DZKBC4CHRGISXEQHTQAGKZFNXQKWXSWBSMHNRQTSJ32WYI2OQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667805441 (7h36m5.219275902s) [noise=true , address=145.79.227.9:4365 , piece_id=7BK6E45EY6Y3SV6PY3A4EVCS42PC4HYLXU3J7MN4HXRLDEFUEFCA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667895652 (7h36m5.219366113s) [noise=true , address=145.79.227.17:4970 , piece_id=5ZP5DDC3XAYQ5UFYJVQV6IZHB5KQQGNG54MTYW3UUYPN2IAK5QVQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.668240909 (7h36m5.219711361s) [noise=true , address=77.90.31.17:4772 , piece_id=CT6BNOYZGDBUEFVGM2RDS4ICMWPYS6N4IOUDA5V7MJQ3IKELNW2Q, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.668671640 (7h36m5.220142101s) [noise=true , address=145.79.227.31:4797 , piece_id=VQEEGJTLJY2IMCDHL2GMVGYEX4VL34TMMXKDYNERFYK3HRV7TC6A, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.669093554 (7h36m5.220564015s) [noise=true , address=145.79.227.21:4474 , piece_id=J5XE2MTCEW4ZHQDDZ4KCVYFK5EX6RQZARKNY4QCH4HUHGJO3SXDQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.669510097 (7h36m5.220980568s) [noise=true , address=77.90.31.18:4807 , piece_id=HDJAJYXNZNZN3LVAQSYVJEOQ3MU5H3BXCCW3NA2GLKPT7ATZHSJQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.671936887 (7h36m5.223407348s) [noise=true , address=145.79.227.20:4439 , piece_id=PPOVWSRAU7Y4CZAKAXV6ZP74EEPHONUC35O6XM57VE47NVJL5WZA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.672494549 (7h36m5.22396502s) [noise=true , address=77.90.31.21:4877 , piece_id=3V3Z6GNF32Q66EOS66WV3AGJNA7IXT5NHVAPKNUGVYU3R46IKKKA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:08.438197970 (7h36m6.989668451s) [upload=6714]: Next returned nil segment Nov 9 15:40:08.438219050 (7h36m6.989689521s) [upload=6714]: Waiting for error group managing segments... Nov 9 15:40:08.829503696 (7h36m7.380974147s) [noise=true , address=77.90.31.18:4807 , piece_id=HDJAJYXNZNZN3LVAQSYVJEOQ3MU5H3BXCCW3NA2GLKPT7ATZHSJQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.846923116 (7h36m7.398393577s) [noise=true , address=45.91.116.14:4190 , piece_id=XSIDZL5HMLZTQE7ZAPURAAUGQLPP3QOZK5BIE5PHRMZPSTN5APSA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.867005608 (7h36m7.418476069s) [noise=true , address=77.90.31.21:4877 , piece_id=3V3Z6GNF32Q66EOS66WV3AGJNA7IXT5NHVAPKNUGVYU3R46IKKKA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.888352807 (7h36m7.439823268s) [noise=true , address=77.90.31.14:4662 , piece_id=HHSJDZPYKGCHURAZV46OQSGBU7D4TFPXREDBFPGTYF7OAUMT2X3A, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.904680649 (7h36m7.45615111s) [noise=true , address=77.90.31.17:4772 , piece_id=CT6BNOYZGDBUEFVGM2RDS4ICMWPYS6N4IOUDA5V7MJQ3IKELNW2Q, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:18.663082162 (7h36m17.214552623s) [noise=true , address=145.79.227.17:4971 , piece_id=VTUI377UCF44ICILORHEKUO33ICRXM2FECO5EXS7RUDH4VGV7FJA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:20.077194714 (7h36m18.628665185s) [noise=true , address=145.79.227.13:4803 , piece_id=V37DZKBC4CHRGISXEQHTQAGKZFNXQKWXSWBSMHNRQTSJ32WYI2OQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:28.517900114 (7h36m27.069370575s) [noise=true , address=145.79.227.32:4835 , piece_id=EYIBNSVPQPKLKOBHKZI2VZHVCDKW3C3YJSFEWP4G62UTBXX2YRZA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:28.567571990 (7h36m27.119042451s) [noise=true , address=145.79.227.31:4797 , piece_id=VQEEGJTLJY2IMCDHL2GMVGYEX4VL34TMMXKDYNERFYK3HRV7TC6A, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:33.651382593 (7h36m32.202853044s) [noise=true , address=145.79.227.20:4439 , piece_id=PPOVWSRAU7Y4CZAKAXV6ZP74EEPHONUC35O6XM57VE47NVJL5WZA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:34.842221635 (7h36m33.393692096s) [noise=true , address=145.79.227.9:4365 , piece_id=7BK6E45EY6Y3SV6PY3A4EVCS42PC4HYLXU3J7MN4HXRLDEFUEFCA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:35.601406517 (7h36m34.152877008s) [noise=true , address=145.79.227.25:4619 , piece_id=23D3MCZRVNPI5KB43ROAHKUALDGXLZBZU5E6EIU6U2FY3YQOVPGA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:35.697082303 (7h36m34.248552774s) [noise=true , address=145.79.227.17:4970 , piece_id=5ZP5DDC3XAYQ5UFYJVQV6IZHB5KQQGNG54MTYW3UUYPN2IAK5QVQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697252518 (7h36m34.248722969s) [noise=true , address=145.79.227.17:4972 , piece_id=AI3KVNUJKJUD33YVU2Z472R72SKQKRYQG65JPKFZ5WBJLKRLW6MA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697302322 (7h36m34.248772773s) [noise=true , address=145.79.227.21:4474 , piece_id=J5XE2MTCEW4ZHQDDZ4KCVYFK5EX6RQZARKNY4QCH4HUHGJO3SXDQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, address:145.79.227.21:4474): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697472546 (7h36m34.248943007s) [noise=true , address=145.79.227.9:4366 , piece_id=BT3GWQSXIMMSFPUYJC5EQ2LX2DS4BW5DAMW6ILLNVVRUUHZMOQUA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697514686 (7h36m34.248985147s) [seg_pos={0 0}, upload=6714]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 9 15:40:35.950211122 (7h36m34.501681583s) [upload=6714]: Done uploading segments... Nov 9 15:40:35.950416643 (7h36m34.501887104s) [upload=6714]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 8.600134279s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 9.092992389s trace: https://jaeger.dev.storj.io/trace/3846516a5cecd778 FAIL (2025-11-09 15:58:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 9 15:40:06.410809146 (7h36m4.962279607s) [upload=6714]: Starting upload Nov 9 15:40:06.411046298 (7h36m4.962516769s) [upload=6714]: Done starting upload Nov 9 15:40:06.411314729 (7h36m4.96278519s) [upload=6714]: Uploading segments... Nov 9 15:40:06.411410882 (7h36m4.962881383s) [upload=6714]: Got next segment. Inline: false Nov 9 15:40:06.411464574 (7h36m4.962935035s) [upload=6714]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 9 15:40:06.411475064 (7h36m4.962945555s) [upload=6714]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 9 15:40:06.664789367 (7h36m5.216259828s) [seg_pos={0 0}, upload=6714]: Begin upload segment... Nov 9 15:40:06.664851705 (7h36m5.216322166s) [seg_pos={0 0}, upload=6714]: Done begin upload segment. Nov 9 15:40:06.664891662 (7h36m5.216362123s) [noise=true , address=145.79.227.32:4835 , piece_id=EYIBNSVPQPKLKOBHKZI2VZHVCDKW3C3YJSFEWP4G62UTBXX2YRZA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.664996933 (7h36m5.216467394s) [noise=true , address=45.91.116.14:4190 , piece_id=XSIDZL5HMLZTQE7ZAPURAAUGQLPP3QOZK5BIE5PHRMZPSTN5APSA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.665698789 (7h36m5.21716925s) [noise=true , address=77.90.31.14:4662 , piece_id=HHSJDZPYKGCHURAZV46OQSGBU7D4TFPXREDBFPGTYF7OAUMT2X3A, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.666456111 (7h36m5.217926572s) [noise=true , address=145.79.227.25:4619 , piece_id=23D3MCZRVNPI5KB43ROAHKUALDGXLZBZU5E6EIU6U2FY3YQOVPGA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.666642225 (7h36m5.218112676s) [noise=true , address=145.79.227.17:4971 , piece_id=VTUI377UCF44ICILORHEKUO33ICRXM2FECO5EXS7RUDH4VGV7FJA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667027539 (7h36m5.218498s) [noise=true , address=145.79.227.9:4366 , piece_id=BT3GWQSXIMMSFPUYJC5EQ2LX2DS4BW5DAMW6ILLNVVRUUHZMOQUA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667041675 (7h36m5.218512126s) [noise=true , address=145.79.227.17:4972 , piece_id=AI3KVNUJKJUD33YVU2Z472R72SKQKRYQG65JPKFZ5WBJLKRLW6MA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667502794 (7h36m5.218973255s) [noise=true , address=145.79.227.13:4803 , piece_id=V37DZKBC4CHRGISXEQHTQAGKZFNXQKWXSWBSMHNRQTSJ32WYI2OQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667805441 (7h36m5.219275902s) [noise=true , address=145.79.227.9:4365 , piece_id=7BK6E45EY6Y3SV6PY3A4EVCS42PC4HYLXU3J7MN4HXRLDEFUEFCA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667895652 (7h36m5.219366113s) [noise=true , address=145.79.227.17:4970 , piece_id=5ZP5DDC3XAYQ5UFYJVQV6IZHB5KQQGNG54MTYW3UUYPN2IAK5QVQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.668240909 (7h36m5.219711361s) [noise=true , address=77.90.31.17:4772 , piece_id=CT6BNOYZGDBUEFVGM2RDS4ICMWPYS6N4IOUDA5V7MJQ3IKELNW2Q, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.668671640 (7h36m5.220142101s) [noise=true , address=145.79.227.31:4797 , piece_id=VQEEGJTLJY2IMCDHL2GMVGYEX4VL34TMMXKDYNERFYK3HRV7TC6A, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.669093554 (7h36m5.220564015s) [noise=true , address=145.79.227.21:4474 , piece_id=J5XE2MTCEW4ZHQDDZ4KCVYFK5EX6RQZARKNY4QCH4HUHGJO3SXDQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.669510097 (7h36m5.220980568s) [noise=true , address=77.90.31.18:4807 , piece_id=HDJAJYXNZNZN3LVAQSYVJEOQ3MU5H3BXCCW3NA2GLKPT7ATZHSJQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.671936887 (7h36m5.223407348s) [noise=true , address=145.79.227.20:4439 , piece_id=PPOVWSRAU7Y4CZAKAXV6ZP74EEPHONUC35O6XM57VE47NVJL5WZA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.672494549 (7h36m5.22396502s) [noise=true , address=77.90.31.21:4877 , piece_id=3V3Z6GNF32Q66EOS66WV3AGJNA7IXT5NHVAPKNUGVYU3R46IKKKA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:08.438197970 (7h36m6.989668451s) [upload=6714]: Next returned nil segment Nov 9 15:40:08.438219050 (7h36m6.989689521s) [upload=6714]: Waiting for error group managing segments... Nov 9 15:40:08.829503696 (7h36m7.380974147s) [noise=true , address=77.90.31.18:4807 , piece_id=HDJAJYXNZNZN3LVAQSYVJEOQ3MU5H3BXCCW3NA2GLKPT7ATZHSJQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.846923116 (7h36m7.398393577s) [noise=true , address=45.91.116.14:4190 , piece_id=XSIDZL5HMLZTQE7ZAPURAAUGQLPP3QOZK5BIE5PHRMZPSTN5APSA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.867005608 (7h36m7.418476069s) [noise=true , address=77.90.31.21:4877 , piece_id=3V3Z6GNF32Q66EOS66WV3AGJNA7IXT5NHVAPKNUGVYU3R46IKKKA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.888352807 (7h36m7.439823268s) [noise=true , address=77.90.31.14:4662 , piece_id=HHSJDZPYKGCHURAZV46OQSGBU7D4TFPXREDBFPGTYF7OAUMT2X3A, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.904680649 (7h36m7.45615111s) [noise=true , address=77.90.31.17:4772 , piece_id=CT6BNOYZGDBUEFVGM2RDS4ICMWPYS6N4IOUDA5V7MJQ3IKELNW2Q, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:18.663082162 (7h36m17.214552623s) [noise=true , address=145.79.227.17:4971 , piece_id=VTUI377UCF44ICILORHEKUO33ICRXM2FECO5EXS7RUDH4VGV7FJA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:20.077194714 (7h36m18.628665185s) [noise=true , address=145.79.227.13:4803 , piece_id=V37DZKBC4CHRGISXEQHTQAGKZFNXQKWXSWBSMHNRQTSJ32WYI2OQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:28.517900114 (7h36m27.069370575s) [noise=true , address=145.79.227.32:4835 , piece_id=EYIBNSVPQPKLKOBHKZI2VZHVCDKW3C3YJSFEWP4G62UTBXX2YRZA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:28.567571990 (7h36m27.119042451s) [noise=true , address=145.79.227.31:4797 , piece_id=VQEEGJTLJY2IMCDHL2GMVGYEX4VL34TMMXKDYNERFYK3HRV7TC6A, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:33.651382593 (7h36m32.202853044s) [noise=true , address=145.79.227.20:4439 , piece_id=PPOVWSRAU7Y4CZAKAXV6ZP74EEPHONUC35O6XM57VE47NVJL5WZA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:34.842221635 (7h36m33.393692096s) [noise=true , address=145.79.227.9:4365 , piece_id=7BK6E45EY6Y3SV6PY3A4EVCS42PC4HYLXU3J7MN4HXRLDEFUEFCA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:35.601406517 (7h36m34.152877008s) [noise=true , address=145.79.227.25:4619 , piece_id=23D3MCZRVNPI5KB43ROAHKUALDGXLZBZU5E6EIU6U2FY3YQOVPGA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:35.697082303 (7h36m34.248552774s) [noise=true , address=145.79.227.17:4970 , piece_id=5ZP5DDC3XAYQ5UFYJVQV6IZHB5KQQGNG54MTYW3UUYPN2IAK5QVQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697252518 (7h36m34.248722969s) [noise=true , address=145.79.227.17:4972 , piece_id=AI3KVNUJKJUD33YVU2Z472R72SKQKRYQG65JPKFZ5WBJLKRLW6MA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697302322 (7h36m34.248772773s) [noise=true , address=145.79.227.21:4474 , piece_id=J5XE2MTCEW4ZHQDDZ4KCVYFK5EX6RQZARKNY4QCH4HUHGJO3SXDQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, address:145.79.227.21:4474): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697472546 (7h36m34.248943007s) [noise=true , address=145.79.227.9:4366 , piece_id=BT3GWQSXIMMSFPUYJC5EQ2LX2DS4BW5DAMW6ILLNVVRUUHZMOQUA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697514686 (7h36m34.248985147s) [seg_pos={0 0}, upload=6714]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 9 15:40:35.950211122 (7h36m34.501681583s) [upload=6714]: Done uploading segments... Nov 9 15:40:35.950416643 (7h36m34.501887104s) [upload=6714]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 9.093024s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 8.387304528s trace: https://jaeger.dev.storj.io/trace/3846516a5cecd778 FAIL (2025-11-09 15:57:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 9 15:40:06.410809146 (7h36m4.962279607s) [upload=6714]: Starting upload Nov 9 15:40:06.411046298 (7h36m4.962516769s) [upload=6714]: Done starting upload Nov 9 15:40:06.411314729 (7h36m4.96278519s) [upload=6714]: Uploading segments... Nov 9 15:40:06.411410882 (7h36m4.962881383s) [upload=6714]: Got next segment. Inline: false Nov 9 15:40:06.411464574 (7h36m4.962935035s) [upload=6714]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 9 15:40:06.411475064 (7h36m4.962945555s) [upload=6714]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 9 15:40:06.664789367 (7h36m5.216259828s) [seg_pos={0 0}, upload=6714]: Begin upload segment... Nov 9 15:40:06.664851705 (7h36m5.216322166s) [seg_pos={0 0}, upload=6714]: Done begin upload segment. Nov 9 15:40:06.664891662 (7h36m5.216362123s) [noise=true , address=145.79.227.32:4835 , piece_id=EYIBNSVPQPKLKOBHKZI2VZHVCDKW3C3YJSFEWP4G62UTBXX2YRZA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.664996933 (7h36m5.216467394s) [noise=true , address=45.91.116.14:4190 , piece_id=XSIDZL5HMLZTQE7ZAPURAAUGQLPP3QOZK5BIE5PHRMZPSTN5APSA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.665698789 (7h36m5.21716925s) [noise=true , address=77.90.31.14:4662 , piece_id=HHSJDZPYKGCHURAZV46OQSGBU7D4TFPXREDBFPGTYF7OAUMT2X3A, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.666456111 (7h36m5.217926572s) [noise=true , address=145.79.227.25:4619 , piece_id=23D3MCZRVNPI5KB43ROAHKUALDGXLZBZU5E6EIU6U2FY3YQOVPGA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.666642225 (7h36m5.218112676s) [noise=true , address=145.79.227.17:4971 , piece_id=VTUI377UCF44ICILORHEKUO33ICRXM2FECO5EXS7RUDH4VGV7FJA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667027539 (7h36m5.218498s) [noise=true , address=145.79.227.9:4366 , piece_id=BT3GWQSXIMMSFPUYJC5EQ2LX2DS4BW5DAMW6ILLNVVRUUHZMOQUA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667041675 (7h36m5.218512126s) [noise=true , address=145.79.227.17:4972 , piece_id=AI3KVNUJKJUD33YVU2Z472R72SKQKRYQG65JPKFZ5WBJLKRLW6MA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667502794 (7h36m5.218973255s) [noise=true , address=145.79.227.13:4803 , piece_id=V37DZKBC4CHRGISXEQHTQAGKZFNXQKWXSWBSMHNRQTSJ32WYI2OQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667805441 (7h36m5.219275902s) [noise=true , address=145.79.227.9:4365 , piece_id=7BK6E45EY6Y3SV6PY3A4EVCS42PC4HYLXU3J7MN4HXRLDEFUEFCA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667895652 (7h36m5.219366113s) [noise=true , address=145.79.227.17:4970 , piece_id=5ZP5DDC3XAYQ5UFYJVQV6IZHB5KQQGNG54MTYW3UUYPN2IAK5QVQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.668240909 (7h36m5.219711361s) [noise=true , address=77.90.31.17:4772 , piece_id=CT6BNOYZGDBUEFVGM2RDS4ICMWPYS6N4IOUDA5V7MJQ3IKELNW2Q, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.668671640 (7h36m5.220142101s) [noise=true , address=145.79.227.31:4797 , piece_id=VQEEGJTLJY2IMCDHL2GMVGYEX4VL34TMMXKDYNERFYK3HRV7TC6A, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.669093554 (7h36m5.220564015s) [noise=true , address=145.79.227.21:4474 , piece_id=J5XE2MTCEW4ZHQDDZ4KCVYFK5EX6RQZARKNY4QCH4HUHGJO3SXDQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.669510097 (7h36m5.220980568s) [noise=true , address=77.90.31.18:4807 , piece_id=HDJAJYXNZNZN3LVAQSYVJEOQ3MU5H3BXCCW3NA2GLKPT7ATZHSJQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.671936887 (7h36m5.223407348s) [noise=true , address=145.79.227.20:4439 , piece_id=PPOVWSRAU7Y4CZAKAXV6ZP74EEPHONUC35O6XM57VE47NVJL5WZA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.672494549 (7h36m5.22396502s) [noise=true , address=77.90.31.21:4877 , piece_id=3V3Z6GNF32Q66EOS66WV3AGJNA7IXT5NHVAPKNUGVYU3R46IKKKA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:08.438197970 (7h36m6.989668451s) [upload=6714]: Next returned nil segment Nov 9 15:40:08.438219050 (7h36m6.989689521s) [upload=6714]: Waiting for error group managing segments... Nov 9 15:40:08.829503696 (7h36m7.380974147s) [noise=true , address=77.90.31.18:4807 , piece_id=HDJAJYXNZNZN3LVAQSYVJEOQ3MU5H3BXCCW3NA2GLKPT7ATZHSJQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.846923116 (7h36m7.398393577s) [noise=true , address=45.91.116.14:4190 , piece_id=XSIDZL5HMLZTQE7ZAPURAAUGQLPP3QOZK5BIE5PHRMZPSTN5APSA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.867005608 (7h36m7.418476069s) [noise=true , address=77.90.31.21:4877 , piece_id=3V3Z6GNF32Q66EOS66WV3AGJNA7IXT5NHVAPKNUGVYU3R46IKKKA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.888352807 (7h36m7.439823268s) [noise=true , address=77.90.31.14:4662 , piece_id=HHSJDZPYKGCHURAZV46OQSGBU7D4TFPXREDBFPGTYF7OAUMT2X3A, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.904680649 (7h36m7.45615111s) [noise=true , address=77.90.31.17:4772 , piece_id=CT6BNOYZGDBUEFVGM2RDS4ICMWPYS6N4IOUDA5V7MJQ3IKELNW2Q, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:18.663082162 (7h36m17.214552623s) [noise=true , address=145.79.227.17:4971 , piece_id=VTUI377UCF44ICILORHEKUO33ICRXM2FECO5EXS7RUDH4VGV7FJA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:20.077194714 (7h36m18.628665185s) [noise=true , address=145.79.227.13:4803 , piece_id=V37DZKBC4CHRGISXEQHTQAGKZFNXQKWXSWBSMHNRQTSJ32WYI2OQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:28.517900114 (7h36m27.069370575s) [noise=true , address=145.79.227.32:4835 , piece_id=EYIBNSVPQPKLKOBHKZI2VZHVCDKW3C3YJSFEWP4G62UTBXX2YRZA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:28.567571990 (7h36m27.119042451s) [noise=true , address=145.79.227.31:4797 , piece_id=VQEEGJTLJY2IMCDHL2GMVGYEX4VL34TMMXKDYNERFYK3HRV7TC6A, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:33.651382593 (7h36m32.202853044s) [noise=true , address=145.79.227.20:4439 , piece_id=PPOVWSRAU7Y4CZAKAXV6ZP74EEPHONUC35O6XM57VE47NVJL5WZA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:34.842221635 (7h36m33.393692096s) [noise=true , address=145.79.227.9:4365 , piece_id=7BK6E45EY6Y3SV6PY3A4EVCS42PC4HYLXU3J7MN4HXRLDEFUEFCA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:35.601406517 (7h36m34.152877008s) [noise=true , address=145.79.227.25:4619 , piece_id=23D3MCZRVNPI5KB43ROAHKUALDGXLZBZU5E6EIU6U2FY3YQOVPGA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:35.697082303 (7h36m34.248552774s) [noise=true , address=145.79.227.17:4970 , piece_id=5ZP5DDC3XAYQ5UFYJVQV6IZHB5KQQGNG54MTYW3UUYPN2IAK5QVQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697252518 (7h36m34.248722969s) [noise=true , address=145.79.227.17:4972 , piece_id=AI3KVNUJKJUD33YVU2Z472R72SKQKRYQG65JPKFZ5WBJLKRLW6MA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697302322 (7h36m34.248772773s) [noise=true , address=145.79.227.21:4474 , piece_id=J5XE2MTCEW4ZHQDDZ4KCVYFK5EX6RQZARKNY4QCH4HUHGJO3SXDQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, address:145.79.227.21:4474): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697472546 (7h36m34.248943007s) [noise=true , address=145.79.227.9:4366 , piece_id=BT3GWQSXIMMSFPUYJC5EQ2LX2DS4BW5DAMW6ILLNVVRUUHZMOQUA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697514686 (7h36m34.248985147s) [seg_pos={0 0}, upload=6714]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 9 15:40:35.950211122 (7h36m34.501681583s) [upload=6714]: Done uploading segments... Nov 9 15:40:35.950416643 (7h36m34.501887104s) [upload=6714]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 8.387330887s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 8.421910419s trace: https://jaeger.dev.storj.io/trace/3846516a5cecd778 FAIL (2025-11-09 15:56:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 9 15:40:06.410809146 (7h36m4.962279607s) [upload=6714]: Starting upload Nov 9 15:40:06.411046298 (7h36m4.962516769s) [upload=6714]: Done starting upload Nov 9 15:40:06.411314729 (7h36m4.96278519s) [upload=6714]: Uploading segments... Nov 9 15:40:06.411410882 (7h36m4.962881383s) [upload=6714]: Got next segment. Inline: false Nov 9 15:40:06.411464574 (7h36m4.962935035s) [upload=6714]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 9 15:40:06.411475064 (7h36m4.962945555s) [upload=6714]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 9 15:40:06.664789367 (7h36m5.216259828s) [seg_pos={0 0}, upload=6714]: Begin upload segment... Nov 9 15:40:06.664851705 (7h36m5.216322166s) [seg_pos={0 0}, upload=6714]: Done begin upload segment. Nov 9 15:40:06.664891662 (7h36m5.216362123s) [noise=true , address=145.79.227.32:4835 , piece_id=EYIBNSVPQPKLKOBHKZI2VZHVCDKW3C3YJSFEWP4G62UTBXX2YRZA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.664996933 (7h36m5.216467394s) [noise=true , address=45.91.116.14:4190 , piece_id=XSIDZL5HMLZTQE7ZAPURAAUGQLPP3QOZK5BIE5PHRMZPSTN5APSA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.665698789 (7h36m5.21716925s) [noise=true , address=77.90.31.14:4662 , piece_id=HHSJDZPYKGCHURAZV46OQSGBU7D4TFPXREDBFPGTYF7OAUMT2X3A, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.666456111 (7h36m5.217926572s) [noise=true , address=145.79.227.25:4619 , piece_id=23D3MCZRVNPI5KB43ROAHKUALDGXLZBZU5E6EIU6U2FY3YQOVPGA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.666642225 (7h36m5.218112676s) [noise=true , address=145.79.227.17:4971 , piece_id=VTUI377UCF44ICILORHEKUO33ICRXM2FECO5EXS7RUDH4VGV7FJA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667027539 (7h36m5.218498s) [noise=true , address=145.79.227.9:4366 , piece_id=BT3GWQSXIMMSFPUYJC5EQ2LX2DS4BW5DAMW6ILLNVVRUUHZMOQUA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667041675 (7h36m5.218512126s) [noise=true , address=145.79.227.17:4972 , piece_id=AI3KVNUJKJUD33YVU2Z472R72SKQKRYQG65JPKFZ5WBJLKRLW6MA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667502794 (7h36m5.218973255s) [noise=true , address=145.79.227.13:4803 , piece_id=V37DZKBC4CHRGISXEQHTQAGKZFNXQKWXSWBSMHNRQTSJ32WYI2OQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667805441 (7h36m5.219275902s) [noise=true , address=145.79.227.9:4365 , piece_id=7BK6E45EY6Y3SV6PY3A4EVCS42PC4HYLXU3J7MN4HXRLDEFUEFCA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667895652 (7h36m5.219366113s) [noise=true , address=145.79.227.17:4970 , piece_id=5ZP5DDC3XAYQ5UFYJVQV6IZHB5KQQGNG54MTYW3UUYPN2IAK5QVQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.668240909 (7h36m5.219711361s) [noise=true , address=77.90.31.17:4772 , piece_id=CT6BNOYZGDBUEFVGM2RDS4ICMWPYS6N4IOUDA5V7MJQ3IKELNW2Q, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.668671640 (7h36m5.220142101s) [noise=true , address=145.79.227.31:4797 , piece_id=VQEEGJTLJY2IMCDHL2GMVGYEX4VL34TMMXKDYNERFYK3HRV7TC6A, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.669093554 (7h36m5.220564015s) [noise=true , address=145.79.227.21:4474 , piece_id=J5XE2MTCEW4ZHQDDZ4KCVYFK5EX6RQZARKNY4QCH4HUHGJO3SXDQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.669510097 (7h36m5.220980568s) [noise=true , address=77.90.31.18:4807 , piece_id=HDJAJYXNZNZN3LVAQSYVJEOQ3MU5H3BXCCW3NA2GLKPT7ATZHSJQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.671936887 (7h36m5.223407348s) [noise=true , address=145.79.227.20:4439 , piece_id=PPOVWSRAU7Y4CZAKAXV6ZP74EEPHONUC35O6XM57VE47NVJL5WZA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.672494549 (7h36m5.22396502s) [noise=true , address=77.90.31.21:4877 , piece_id=3V3Z6GNF32Q66EOS66WV3AGJNA7IXT5NHVAPKNUGVYU3R46IKKKA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:08.438197970 (7h36m6.989668451s) [upload=6714]: Next returned nil segment Nov 9 15:40:08.438219050 (7h36m6.989689521s) [upload=6714]: Waiting for error group managing segments... Nov 9 15:40:08.829503696 (7h36m7.380974147s) [noise=true , address=77.90.31.18:4807 , piece_id=HDJAJYXNZNZN3LVAQSYVJEOQ3MU5H3BXCCW3NA2GLKPT7ATZHSJQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.846923116 (7h36m7.398393577s) [noise=true , address=45.91.116.14:4190 , piece_id=XSIDZL5HMLZTQE7ZAPURAAUGQLPP3QOZK5BIE5PHRMZPSTN5APSA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.867005608 (7h36m7.418476069s) [noise=true , address=77.90.31.21:4877 , piece_id=3V3Z6GNF32Q66EOS66WV3AGJNA7IXT5NHVAPKNUGVYU3R46IKKKA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.888352807 (7h36m7.439823268s) [noise=true , address=77.90.31.14:4662 , piece_id=HHSJDZPYKGCHURAZV46OQSGBU7D4TFPXREDBFPGTYF7OAUMT2X3A, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.904680649 (7h36m7.45615111s) [noise=true , address=77.90.31.17:4772 , piece_id=CT6BNOYZGDBUEFVGM2RDS4ICMWPYS6N4IOUDA5V7MJQ3IKELNW2Q, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:18.663082162 (7h36m17.214552623s) [noise=true , address=145.79.227.17:4971 , piece_id=VTUI377UCF44ICILORHEKUO33ICRXM2FECO5EXS7RUDH4VGV7FJA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:20.077194714 (7h36m18.628665185s) [noise=true , address=145.79.227.13:4803 , piece_id=V37DZKBC4CHRGISXEQHTQAGKZFNXQKWXSWBSMHNRQTSJ32WYI2OQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:28.517900114 (7h36m27.069370575s) [noise=true , address=145.79.227.32:4835 , piece_id=EYIBNSVPQPKLKOBHKZI2VZHVCDKW3C3YJSFEWP4G62UTBXX2YRZA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:28.567571990 (7h36m27.119042451s) [noise=true , address=145.79.227.31:4797 , piece_id=VQEEGJTLJY2IMCDHL2GMVGYEX4VL34TMMXKDYNERFYK3HRV7TC6A, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:33.651382593 (7h36m32.202853044s) [noise=true , address=145.79.227.20:4439 , piece_id=PPOVWSRAU7Y4CZAKAXV6ZP74EEPHONUC35O6XM57VE47NVJL5WZA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:34.842221635 (7h36m33.393692096s) [noise=true , address=145.79.227.9:4365 , piece_id=7BK6E45EY6Y3SV6PY3A4EVCS42PC4HYLXU3J7MN4HXRLDEFUEFCA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:35.601406517 (7h36m34.152877008s) [noise=true , address=145.79.227.25:4619 , piece_id=23D3MCZRVNPI5KB43ROAHKUALDGXLZBZU5E6EIU6U2FY3YQOVPGA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:35.697082303 (7h36m34.248552774s) [noise=true , address=145.79.227.17:4970 , piece_id=5ZP5DDC3XAYQ5UFYJVQV6IZHB5KQQGNG54MTYW3UUYPN2IAK5QVQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697252518 (7h36m34.248722969s) [noise=true , address=145.79.227.17:4972 , piece_id=AI3KVNUJKJUD33YVU2Z472R72SKQKRYQG65JPKFZ5WBJLKRLW6MA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697302322 (7h36m34.248772773s) [noise=true , address=145.79.227.21:4474 , piece_id=J5XE2MTCEW4ZHQDDZ4KCVYFK5EX6RQZARKNY4QCH4HUHGJO3SXDQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, address:145.79.227.21:4474): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697472546 (7h36m34.248943007s) [noise=true , address=145.79.227.9:4366 , piece_id=BT3GWQSXIMMSFPUYJC5EQ2LX2DS4BW5DAMW6ILLNVVRUUHZMOQUA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697514686 (7h36m34.248985147s) [seg_pos={0 0}, upload=6714]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 9 15:40:35.950211122 (7h36m34.501681583s) [upload=6714]: Done uploading segments... Nov 9 15:40:35.950416643 (7h36m34.501887104s) [upload=6714]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 8.421946306s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 8.348702242s trace: https://jaeger.dev.storj.io/trace/3846516a5cecd778 FAIL (2025-11-09 15:55:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 9 15:40:06.410809146 (7h36m4.962279607s) [upload=6714]: Starting upload Nov 9 15:40:06.411046298 (7h36m4.962516769s) [upload=6714]: Done starting upload Nov 9 15:40:06.411314729 (7h36m4.96278519s) [upload=6714]: Uploading segments... Nov 9 15:40:06.411410882 (7h36m4.962881383s) [upload=6714]: Got next segment. Inline: false Nov 9 15:40:06.411464574 (7h36m4.962935035s) [upload=6714]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 9 15:40:06.411475064 (7h36m4.962945555s) [upload=6714]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 9 15:40:06.664789367 (7h36m5.216259828s) [seg_pos={0 0}, upload=6714]: Begin upload segment... Nov 9 15:40:06.664851705 (7h36m5.216322166s) [seg_pos={0 0}, upload=6714]: Done begin upload segment. Nov 9 15:40:06.664891662 (7h36m5.216362123s) [noise=true , address=145.79.227.32:4835 , piece_id=EYIBNSVPQPKLKOBHKZI2VZHVCDKW3C3YJSFEWP4G62UTBXX2YRZA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.664996933 (7h36m5.216467394s) [noise=true , address=45.91.116.14:4190 , piece_id=XSIDZL5HMLZTQE7ZAPURAAUGQLPP3QOZK5BIE5PHRMZPSTN5APSA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.665698789 (7h36m5.21716925s) [noise=true , address=77.90.31.14:4662 , piece_id=HHSJDZPYKGCHURAZV46OQSGBU7D4TFPXREDBFPGTYF7OAUMT2X3A, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.666456111 (7h36m5.217926572s) [noise=true , address=145.79.227.25:4619 , piece_id=23D3MCZRVNPI5KB43ROAHKUALDGXLZBZU5E6EIU6U2FY3YQOVPGA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.666642225 (7h36m5.218112676s) [noise=true , address=145.79.227.17:4971 , piece_id=VTUI377UCF44ICILORHEKUO33ICRXM2FECO5EXS7RUDH4VGV7FJA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667027539 (7h36m5.218498s) [noise=true , address=145.79.227.9:4366 , piece_id=BT3GWQSXIMMSFPUYJC5EQ2LX2DS4BW5DAMW6ILLNVVRUUHZMOQUA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667041675 (7h36m5.218512126s) [noise=true , address=145.79.227.17:4972 , piece_id=AI3KVNUJKJUD33YVU2Z472R72SKQKRYQG65JPKFZ5WBJLKRLW6MA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667502794 (7h36m5.218973255s) [noise=true , address=145.79.227.13:4803 , piece_id=V37DZKBC4CHRGISXEQHTQAGKZFNXQKWXSWBSMHNRQTSJ32WYI2OQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667805441 (7h36m5.219275902s) [noise=true , address=145.79.227.9:4365 , piece_id=7BK6E45EY6Y3SV6PY3A4EVCS42PC4HYLXU3J7MN4HXRLDEFUEFCA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667895652 (7h36m5.219366113s) [noise=true , address=145.79.227.17:4970 , piece_id=5ZP5DDC3XAYQ5UFYJVQV6IZHB5KQQGNG54MTYW3UUYPN2IAK5QVQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.668240909 (7h36m5.219711361s) [noise=true , address=77.90.31.17:4772 , piece_id=CT6BNOYZGDBUEFVGM2RDS4ICMWPYS6N4IOUDA5V7MJQ3IKELNW2Q, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.668671640 (7h36m5.220142101s) [noise=true , address=145.79.227.31:4797 , piece_id=VQEEGJTLJY2IMCDHL2GMVGYEX4VL34TMMXKDYNERFYK3HRV7TC6A, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.669093554 (7h36m5.220564015s) [noise=true , address=145.79.227.21:4474 , piece_id=J5XE2MTCEW4ZHQDDZ4KCVYFK5EX6RQZARKNY4QCH4HUHGJO3SXDQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.669510097 (7h36m5.220980568s) [noise=true , address=77.90.31.18:4807 , piece_id=HDJAJYXNZNZN3LVAQSYVJEOQ3MU5H3BXCCW3NA2GLKPT7ATZHSJQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.671936887 (7h36m5.223407348s) [noise=true , address=145.79.227.20:4439 , piece_id=PPOVWSRAU7Y4CZAKAXV6ZP74EEPHONUC35O6XM57VE47NVJL5WZA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.672494549 (7h36m5.22396502s) [noise=true , address=77.90.31.21:4877 , piece_id=3V3Z6GNF32Q66EOS66WV3AGJNA7IXT5NHVAPKNUGVYU3R46IKKKA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:08.438197970 (7h36m6.989668451s) [upload=6714]: Next returned nil segment Nov 9 15:40:08.438219050 (7h36m6.989689521s) [upload=6714]: Waiting for error group managing segments... Nov 9 15:40:08.829503696 (7h36m7.380974147s) [noise=true , address=77.90.31.18:4807 , piece_id=HDJAJYXNZNZN3LVAQSYVJEOQ3MU5H3BXCCW3NA2GLKPT7ATZHSJQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.846923116 (7h36m7.398393577s) [noise=true , address=45.91.116.14:4190 , piece_id=XSIDZL5HMLZTQE7ZAPURAAUGQLPP3QOZK5BIE5PHRMZPSTN5APSA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.867005608 (7h36m7.418476069s) [noise=true , address=77.90.31.21:4877 , piece_id=3V3Z6GNF32Q66EOS66WV3AGJNA7IXT5NHVAPKNUGVYU3R46IKKKA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.888352807 (7h36m7.439823268s) [noise=true , address=77.90.31.14:4662 , piece_id=HHSJDZPYKGCHURAZV46OQSGBU7D4TFPXREDBFPGTYF7OAUMT2X3A, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.904680649 (7h36m7.45615111s) [noise=true , address=77.90.31.17:4772 , piece_id=CT6BNOYZGDBUEFVGM2RDS4ICMWPYS6N4IOUDA5V7MJQ3IKELNW2Q, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:18.663082162 (7h36m17.214552623s) [noise=true , address=145.79.227.17:4971 , piece_id=VTUI377UCF44ICILORHEKUO33ICRXM2FECO5EXS7RUDH4VGV7FJA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:20.077194714 (7h36m18.628665185s) [noise=true , address=145.79.227.13:4803 , piece_id=V37DZKBC4CHRGISXEQHTQAGKZFNXQKWXSWBSMHNRQTSJ32WYI2OQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:28.517900114 (7h36m27.069370575s) [noise=true , address=145.79.227.32:4835 , piece_id=EYIBNSVPQPKLKOBHKZI2VZHVCDKW3C3YJSFEWP4G62UTBXX2YRZA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:28.567571990 (7h36m27.119042451s) [noise=true , address=145.79.227.31:4797 , piece_id=VQEEGJTLJY2IMCDHL2GMVGYEX4VL34TMMXKDYNERFYK3HRV7TC6A, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:33.651382593 (7h36m32.202853044s) [noise=true , address=145.79.227.20:4439 , piece_id=PPOVWSRAU7Y4CZAKAXV6ZP74EEPHONUC35O6XM57VE47NVJL5WZA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:34.842221635 (7h36m33.393692096s) [noise=true , address=145.79.227.9:4365 , piece_id=7BK6E45EY6Y3SV6PY3A4EVCS42PC4HYLXU3J7MN4HXRLDEFUEFCA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:35.601406517 (7h36m34.152877008s) [noise=true , address=145.79.227.25:4619 , piece_id=23D3MCZRVNPI5KB43ROAHKUALDGXLZBZU5E6EIU6U2FY3YQOVPGA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:35.697082303 (7h36m34.248552774s) [noise=true , address=145.79.227.17:4970 , piece_id=5ZP5DDC3XAYQ5UFYJVQV6IZHB5KQQGNG54MTYW3UUYPN2IAK5QVQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697252518 (7h36m34.248722969s) [noise=true , address=145.79.227.17:4972 , piece_id=AI3KVNUJKJUD33YVU2Z472R72SKQKRYQG65JPKFZ5WBJLKRLW6MA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697302322 (7h36m34.248772773s) [noise=true , address=145.79.227.21:4474 , piece_id=J5XE2MTCEW4ZHQDDZ4KCVYFK5EX6RQZARKNY4QCH4HUHGJO3SXDQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, address:145.79.227.21:4474): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697472546 (7h36m34.248943007s) [noise=true , address=145.79.227.9:4366 , piece_id=BT3GWQSXIMMSFPUYJC5EQ2LX2DS4BW5DAMW6ILLNVVRUUHZMOQUA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697514686 (7h36m34.248985147s) [seg_pos={0 0}, upload=6714]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 9 15:40:35.950211122 (7h36m34.501681583s) [upload=6714]: Done uploading segments... Nov 9 15:40:35.950416643 (7h36m34.501887104s) [upload=6714]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 8.348755553s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 10.014855237s trace: https://jaeger.dev.storj.io/trace/3846516a5cecd778 FAIL (2025-11-09 15:54:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 9 15:40:06.410809146 (7h36m4.962279607s) [upload=6714]: Starting upload Nov 9 15:40:06.411046298 (7h36m4.962516769s) [upload=6714]: Done starting upload Nov 9 15:40:06.411314729 (7h36m4.96278519s) [upload=6714]: Uploading segments... Nov 9 15:40:06.411410882 (7h36m4.962881383s) [upload=6714]: Got next segment. Inline: false Nov 9 15:40:06.411464574 (7h36m4.962935035s) [upload=6714]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 9 15:40:06.411475064 (7h36m4.962945555s) [upload=6714]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 9 15:40:06.664789367 (7h36m5.216259828s) [seg_pos={0 0}, upload=6714]: Begin upload segment... Nov 9 15:40:06.664851705 (7h36m5.216322166s) [seg_pos={0 0}, upload=6714]: Done begin upload segment. Nov 9 15:40:06.664891662 (7h36m5.216362123s) [noise=true , address=145.79.227.32:4835 , piece_id=EYIBNSVPQPKLKOBHKZI2VZHVCDKW3C3YJSFEWP4G62UTBXX2YRZA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.664996933 (7h36m5.216467394s) [noise=true , address=45.91.116.14:4190 , piece_id=XSIDZL5HMLZTQE7ZAPURAAUGQLPP3QOZK5BIE5PHRMZPSTN5APSA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.665698789 (7h36m5.21716925s) [noise=true , address=77.90.31.14:4662 , piece_id=HHSJDZPYKGCHURAZV46OQSGBU7D4TFPXREDBFPGTYF7OAUMT2X3A, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.666456111 (7h36m5.217926572s) [noise=true , address=145.79.227.25:4619 , piece_id=23D3MCZRVNPI5KB43ROAHKUALDGXLZBZU5E6EIU6U2FY3YQOVPGA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.666642225 (7h36m5.218112676s) [noise=true , address=145.79.227.17:4971 , piece_id=VTUI377UCF44ICILORHEKUO33ICRXM2FECO5EXS7RUDH4VGV7FJA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667027539 (7h36m5.218498s) [noise=true , address=145.79.227.9:4366 , piece_id=BT3GWQSXIMMSFPUYJC5EQ2LX2DS4BW5DAMW6ILLNVVRUUHZMOQUA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667041675 (7h36m5.218512126s) [noise=true , address=145.79.227.17:4972 , piece_id=AI3KVNUJKJUD33YVU2Z472R72SKQKRYQG65JPKFZ5WBJLKRLW6MA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667502794 (7h36m5.218973255s) [noise=true , address=145.79.227.13:4803 , piece_id=V37DZKBC4CHRGISXEQHTQAGKZFNXQKWXSWBSMHNRQTSJ32WYI2OQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667805441 (7h36m5.219275902s) [noise=true , address=145.79.227.9:4365 , piece_id=7BK6E45EY6Y3SV6PY3A4EVCS42PC4HYLXU3J7MN4HXRLDEFUEFCA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667895652 (7h36m5.219366113s) [noise=true , address=145.79.227.17:4970 , piece_id=5ZP5DDC3XAYQ5UFYJVQV6IZHB5KQQGNG54MTYW3UUYPN2IAK5QVQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.668240909 (7h36m5.219711361s) [noise=true , address=77.90.31.17:4772 , piece_id=CT6BNOYZGDBUEFVGM2RDS4ICMWPYS6N4IOUDA5V7MJQ3IKELNW2Q, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.668671640 (7h36m5.220142101s) [noise=true , address=145.79.227.31:4797 , piece_id=VQEEGJTLJY2IMCDHL2GMVGYEX4VL34TMMXKDYNERFYK3HRV7TC6A, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.669093554 (7h36m5.220564015s) [noise=true , address=145.79.227.21:4474 , piece_id=J5XE2MTCEW4ZHQDDZ4KCVYFK5EX6RQZARKNY4QCH4HUHGJO3SXDQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.669510097 (7h36m5.220980568s) [noise=true , address=77.90.31.18:4807 , piece_id=HDJAJYXNZNZN3LVAQSYVJEOQ3MU5H3BXCCW3NA2GLKPT7ATZHSJQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.671936887 (7h36m5.223407348s) [noise=true , address=145.79.227.20:4439 , piece_id=PPOVWSRAU7Y4CZAKAXV6ZP74EEPHONUC35O6XM57VE47NVJL5WZA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.672494549 (7h36m5.22396502s) [noise=true , address=77.90.31.21:4877 , piece_id=3V3Z6GNF32Q66EOS66WV3AGJNA7IXT5NHVAPKNUGVYU3R46IKKKA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:08.438197970 (7h36m6.989668451s) [upload=6714]: Next returned nil segment Nov 9 15:40:08.438219050 (7h36m6.989689521s) [upload=6714]: Waiting for error group managing segments... Nov 9 15:40:08.829503696 (7h36m7.380974147s) [noise=true , address=77.90.31.18:4807 , piece_id=HDJAJYXNZNZN3LVAQSYVJEOQ3MU5H3BXCCW3NA2GLKPT7ATZHSJQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.846923116 (7h36m7.398393577s) [noise=true , address=45.91.116.14:4190 , piece_id=XSIDZL5HMLZTQE7ZAPURAAUGQLPP3QOZK5BIE5PHRMZPSTN5APSA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.867005608 (7h36m7.418476069s) [noise=true , address=77.90.31.21:4877 , piece_id=3V3Z6GNF32Q66EOS66WV3AGJNA7IXT5NHVAPKNUGVYU3R46IKKKA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.888352807 (7h36m7.439823268s) [noise=true , address=77.90.31.14:4662 , piece_id=HHSJDZPYKGCHURAZV46OQSGBU7D4TFPXREDBFPGTYF7OAUMT2X3A, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.904680649 (7h36m7.45615111s) [noise=true , address=77.90.31.17:4772 , piece_id=CT6BNOYZGDBUEFVGM2RDS4ICMWPYS6N4IOUDA5V7MJQ3IKELNW2Q, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:18.663082162 (7h36m17.214552623s) [noise=true , address=145.79.227.17:4971 , piece_id=VTUI377UCF44ICILORHEKUO33ICRXM2FECO5EXS7RUDH4VGV7FJA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:20.077194714 (7h36m18.628665185s) [noise=true , address=145.79.227.13:4803 , piece_id=V37DZKBC4CHRGISXEQHTQAGKZFNXQKWXSWBSMHNRQTSJ32WYI2OQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:28.517900114 (7h36m27.069370575s) [noise=true , address=145.79.227.32:4835 , piece_id=EYIBNSVPQPKLKOBHKZI2VZHVCDKW3C3YJSFEWP4G62UTBXX2YRZA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:28.567571990 (7h36m27.119042451s) [noise=true , address=145.79.227.31:4797 , piece_id=VQEEGJTLJY2IMCDHL2GMVGYEX4VL34TMMXKDYNERFYK3HRV7TC6A, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:33.651382593 (7h36m32.202853044s) [noise=true , address=145.79.227.20:4439 , piece_id=PPOVWSRAU7Y4CZAKAXV6ZP74EEPHONUC35O6XM57VE47NVJL5WZA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:34.842221635 (7h36m33.393692096s) [noise=true , address=145.79.227.9:4365 , piece_id=7BK6E45EY6Y3SV6PY3A4EVCS42PC4HYLXU3J7MN4HXRLDEFUEFCA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:35.601406517 (7h36m34.152877008s) [noise=true , address=145.79.227.25:4619 , piece_id=23D3MCZRVNPI5KB43ROAHKUALDGXLZBZU5E6EIU6U2FY3YQOVPGA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:35.697082303 (7h36m34.248552774s) [noise=true , address=145.79.227.17:4970 , piece_id=5ZP5DDC3XAYQ5UFYJVQV6IZHB5KQQGNG54MTYW3UUYPN2IAK5QVQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697252518 (7h36m34.248722969s) [noise=true , address=145.79.227.17:4972 , piece_id=AI3KVNUJKJUD33YVU2Z472R72SKQKRYQG65JPKFZ5WBJLKRLW6MA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697302322 (7h36m34.248772773s) [noise=true , address=145.79.227.21:4474 , piece_id=J5XE2MTCEW4ZHQDDZ4KCVYFK5EX6RQZARKNY4QCH4HUHGJO3SXDQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, address:145.79.227.21:4474): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697472546 (7h36m34.248943007s) [noise=true , address=145.79.227.9:4366 , piece_id=BT3GWQSXIMMSFPUYJC5EQ2LX2DS4BW5DAMW6ILLNVVRUUHZMOQUA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697514686 (7h36m34.248985147s) [seg_pos={0 0}, upload=6714]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 9 15:40:35.950211122 (7h36m34.501681583s) [upload=6714]: Done uploading segments... Nov 9 15:40:35.950416643 (7h36m34.501887104s) [upload=6714]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 10.01489284s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [X..............] duration: 363.819198ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/3ea6c69616edd41 FAIL (2025-11-09 15:53:34 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:59888->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 363.85096ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 10.101655957s trace: https://jaeger.dev.storj.io/trace/3846516a5cecd778 FAIL (2025-11-09 15:52:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 9 15:40:06.410809146 (7h36m4.962279607s) [upload=6714]: Starting upload Nov 9 15:40:06.411046298 (7h36m4.962516769s) [upload=6714]: Done starting upload Nov 9 15:40:06.411314729 (7h36m4.96278519s) [upload=6714]: Uploading segments... Nov 9 15:40:06.411410882 (7h36m4.962881383s) [upload=6714]: Got next segment. Inline: false Nov 9 15:40:06.411464574 (7h36m4.962935035s) [upload=6714]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 9 15:40:06.411475064 (7h36m4.962945555s) [upload=6714]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 9 15:40:06.664789367 (7h36m5.216259828s) [seg_pos={0 0}, upload=6714]: Begin upload segment... Nov 9 15:40:06.664851705 (7h36m5.216322166s) [seg_pos={0 0}, upload=6714]: Done begin upload segment. Nov 9 15:40:06.664891662 (7h36m5.216362123s) [noise=true , address=145.79.227.32:4835 , piece_id=EYIBNSVPQPKLKOBHKZI2VZHVCDKW3C3YJSFEWP4G62UTBXX2YRZA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.664996933 (7h36m5.216467394s) [noise=true , address=45.91.116.14:4190 , piece_id=XSIDZL5HMLZTQE7ZAPURAAUGQLPP3QOZK5BIE5PHRMZPSTN5APSA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.665698789 (7h36m5.21716925s) [noise=true , address=77.90.31.14:4662 , piece_id=HHSJDZPYKGCHURAZV46OQSGBU7D4TFPXREDBFPGTYF7OAUMT2X3A, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.666456111 (7h36m5.217926572s) [noise=true , address=145.79.227.25:4619 , piece_id=23D3MCZRVNPI5KB43ROAHKUALDGXLZBZU5E6EIU6U2FY3YQOVPGA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.666642225 (7h36m5.218112676s) [noise=true , address=145.79.227.17:4971 , piece_id=VTUI377UCF44ICILORHEKUO33ICRXM2FECO5EXS7RUDH4VGV7FJA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667027539 (7h36m5.218498s) [noise=true , address=145.79.227.9:4366 , piece_id=BT3GWQSXIMMSFPUYJC5EQ2LX2DS4BW5DAMW6ILLNVVRUUHZMOQUA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667041675 (7h36m5.218512126s) [noise=true , address=145.79.227.17:4972 , piece_id=AI3KVNUJKJUD33YVU2Z472R72SKQKRYQG65JPKFZ5WBJLKRLW6MA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667502794 (7h36m5.218973255s) [noise=true , address=145.79.227.13:4803 , piece_id=V37DZKBC4CHRGISXEQHTQAGKZFNXQKWXSWBSMHNRQTSJ32WYI2OQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667805441 (7h36m5.219275902s) [noise=true , address=145.79.227.9:4365 , piece_id=7BK6E45EY6Y3SV6PY3A4EVCS42PC4HYLXU3J7MN4HXRLDEFUEFCA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667895652 (7h36m5.219366113s) [noise=true , address=145.79.227.17:4970 , piece_id=5ZP5DDC3XAYQ5UFYJVQV6IZHB5KQQGNG54MTYW3UUYPN2IAK5QVQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.668240909 (7h36m5.219711361s) [noise=true , address=77.90.31.17:4772 , piece_id=CT6BNOYZGDBUEFVGM2RDS4ICMWPYS6N4IOUDA5V7MJQ3IKELNW2Q, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.668671640 (7h36m5.220142101s) [noise=true , address=145.79.227.31:4797 , piece_id=VQEEGJTLJY2IMCDHL2GMVGYEX4VL34TMMXKDYNERFYK3HRV7TC6A, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.669093554 (7h36m5.220564015s) [noise=true , address=145.79.227.21:4474 , piece_id=J5XE2MTCEW4ZHQDDZ4KCVYFK5EX6RQZARKNY4QCH4HUHGJO3SXDQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.669510097 (7h36m5.220980568s) [noise=true , address=77.90.31.18:4807 , piece_id=HDJAJYXNZNZN3LVAQSYVJEOQ3MU5H3BXCCW3NA2GLKPT7ATZHSJQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.671936887 (7h36m5.223407348s) [noise=true , address=145.79.227.20:4439 , piece_id=PPOVWSRAU7Y4CZAKAXV6ZP74EEPHONUC35O6XM57VE47NVJL5WZA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.672494549 (7h36m5.22396502s) [noise=true , address=77.90.31.21:4877 , piece_id=3V3Z6GNF32Q66EOS66WV3AGJNA7IXT5NHVAPKNUGVYU3R46IKKKA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:08.438197970 (7h36m6.989668451s) [upload=6714]: Next returned nil segment Nov 9 15:40:08.438219050 (7h36m6.989689521s) [upload=6714]: Waiting for error group managing segments... Nov 9 15:40:08.829503696 (7h36m7.380974147s) [noise=true , address=77.90.31.18:4807 , piece_id=HDJAJYXNZNZN3LVAQSYVJEOQ3MU5H3BXCCW3NA2GLKPT7ATZHSJQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.846923116 (7h36m7.398393577s) [noise=true , address=45.91.116.14:4190 , piece_id=XSIDZL5HMLZTQE7ZAPURAAUGQLPP3QOZK5BIE5PHRMZPSTN5APSA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.867005608 (7h36m7.418476069s) [noise=true , address=77.90.31.21:4877 , piece_id=3V3Z6GNF32Q66EOS66WV3AGJNA7IXT5NHVAPKNUGVYU3R46IKKKA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.888352807 (7h36m7.439823268s) [noise=true , address=77.90.31.14:4662 , piece_id=HHSJDZPYKGCHURAZV46OQSGBU7D4TFPXREDBFPGTYF7OAUMT2X3A, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.904680649 (7h36m7.45615111s) [noise=true , address=77.90.31.17:4772 , piece_id=CT6BNOYZGDBUEFVGM2RDS4ICMWPYS6N4IOUDA5V7MJQ3IKELNW2Q, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:18.663082162 (7h36m17.214552623s) [noise=true , address=145.79.227.17:4971 , piece_id=VTUI377UCF44ICILORHEKUO33ICRXM2FECO5EXS7RUDH4VGV7FJA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:20.077194714 (7h36m18.628665185s) [noise=true , address=145.79.227.13:4803 , piece_id=V37DZKBC4CHRGISXEQHTQAGKZFNXQKWXSWBSMHNRQTSJ32WYI2OQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:28.517900114 (7h36m27.069370575s) [noise=true , address=145.79.227.32:4835 , piece_id=EYIBNSVPQPKLKOBHKZI2VZHVCDKW3C3YJSFEWP4G62UTBXX2YRZA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:28.567571990 (7h36m27.119042451s) [noise=true , address=145.79.227.31:4797 , piece_id=VQEEGJTLJY2IMCDHL2GMVGYEX4VL34TMMXKDYNERFYK3HRV7TC6A, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:33.651382593 (7h36m32.202853044s) [noise=true , address=145.79.227.20:4439 , piece_id=PPOVWSRAU7Y4CZAKAXV6ZP74EEPHONUC35O6XM57VE47NVJL5WZA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:34.842221635 (7h36m33.393692096s) [noise=true , address=145.79.227.9:4365 , piece_id=7BK6E45EY6Y3SV6PY3A4EVCS42PC4HYLXU3J7MN4HXRLDEFUEFCA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:35.601406517 (7h36m34.152877008s) [noise=true , address=145.79.227.25:4619 , piece_id=23D3MCZRVNPI5KB43ROAHKUALDGXLZBZU5E6EIU6U2FY3YQOVPGA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:35.697082303 (7h36m34.248552774s) [noise=true , address=145.79.227.17:4970 , piece_id=5ZP5DDC3XAYQ5UFYJVQV6IZHB5KQQGNG54MTYW3UUYPN2IAK5QVQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697252518 (7h36m34.248722969s) [noise=true , address=145.79.227.17:4972 , piece_id=AI3KVNUJKJUD33YVU2Z472R72SKQKRYQG65JPKFZ5WBJLKRLW6MA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697302322 (7h36m34.248772773s) [noise=true , address=145.79.227.21:4474 , piece_id=J5XE2MTCEW4ZHQDDZ4KCVYFK5EX6RQZARKNY4QCH4HUHGJO3SXDQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, address:145.79.227.21:4474): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697472546 (7h36m34.248943007s) [noise=true , address=145.79.227.9:4366 , piece_id=BT3GWQSXIMMSFPUYJC5EQ2LX2DS4BW5DAMW6ILLNVVRUUHZMOQUA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697514686 (7h36m34.248985147s) [seg_pos={0 0}, upload=6714]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 9 15:40:35.950211122 (7h36m34.501681583s) [upload=6714]: Done uploading segments... Nov 9 15:40:35.950416643 (7h36m34.501887104s) [upload=6714]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 10.10168944s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [.X.............] duration: 95.392828ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/3ea6c69616edd41 FAIL (2025-11-09 15:52:49 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:59888->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 95.49395ms core website
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [..X............] duration: 62.348394ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/3ea6c69616edd41 FAIL (2025-11-09 15:51:50 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:59888->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 62.378132ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 8.794465806s trace: https://jaeger.dev.storj.io/trace/3846516a5cecd778 FAIL (2025-11-09 15:51:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 9 15:40:06.410809146 (7h36m4.962279607s) [upload=6714]: Starting upload Nov 9 15:40:06.411046298 (7h36m4.962516769s) [upload=6714]: Done starting upload Nov 9 15:40:06.411314729 (7h36m4.96278519s) [upload=6714]: Uploading segments... Nov 9 15:40:06.411410882 (7h36m4.962881383s) [upload=6714]: Got next segment. Inline: false Nov 9 15:40:06.411464574 (7h36m4.962935035s) [upload=6714]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 9 15:40:06.411475064 (7h36m4.962945555s) [upload=6714]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 9 15:40:06.664789367 (7h36m5.216259828s) [seg_pos={0 0}, upload=6714]: Begin upload segment... Nov 9 15:40:06.664851705 (7h36m5.216322166s) [seg_pos={0 0}, upload=6714]: Done begin upload segment. Nov 9 15:40:06.664891662 (7h36m5.216362123s) [noise=true , address=145.79.227.32:4835 , piece_id=EYIBNSVPQPKLKOBHKZI2VZHVCDKW3C3YJSFEWP4G62UTBXX2YRZA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.664996933 (7h36m5.216467394s) [noise=true , address=45.91.116.14:4190 , piece_id=XSIDZL5HMLZTQE7ZAPURAAUGQLPP3QOZK5BIE5PHRMZPSTN5APSA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.665698789 (7h36m5.21716925s) [noise=true , address=77.90.31.14:4662 , piece_id=HHSJDZPYKGCHURAZV46OQSGBU7D4TFPXREDBFPGTYF7OAUMT2X3A, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.666456111 (7h36m5.217926572s) [noise=true , address=145.79.227.25:4619 , piece_id=23D3MCZRVNPI5KB43ROAHKUALDGXLZBZU5E6EIU6U2FY3YQOVPGA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.666642225 (7h36m5.218112676s) [noise=true , address=145.79.227.17:4971 , piece_id=VTUI377UCF44ICILORHEKUO33ICRXM2FECO5EXS7RUDH4VGV7FJA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667027539 (7h36m5.218498s) [noise=true , address=145.79.227.9:4366 , piece_id=BT3GWQSXIMMSFPUYJC5EQ2LX2DS4BW5DAMW6ILLNVVRUUHZMOQUA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667041675 (7h36m5.218512126s) [noise=true , address=145.79.227.17:4972 , piece_id=AI3KVNUJKJUD33YVU2Z472R72SKQKRYQG65JPKFZ5WBJLKRLW6MA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667502794 (7h36m5.218973255s) [noise=true , address=145.79.227.13:4803 , piece_id=V37DZKBC4CHRGISXEQHTQAGKZFNXQKWXSWBSMHNRQTSJ32WYI2OQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667805441 (7h36m5.219275902s) [noise=true , address=145.79.227.9:4365 , piece_id=7BK6E45EY6Y3SV6PY3A4EVCS42PC4HYLXU3J7MN4HXRLDEFUEFCA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667895652 (7h36m5.219366113s) [noise=true , address=145.79.227.17:4970 , piece_id=5ZP5DDC3XAYQ5UFYJVQV6IZHB5KQQGNG54MTYW3UUYPN2IAK5QVQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.668240909 (7h36m5.219711361s) [noise=true , address=77.90.31.17:4772 , piece_id=CT6BNOYZGDBUEFVGM2RDS4ICMWPYS6N4IOUDA5V7MJQ3IKELNW2Q, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.668671640 (7h36m5.220142101s) [noise=true , address=145.79.227.31:4797 , piece_id=VQEEGJTLJY2IMCDHL2GMVGYEX4VL34TMMXKDYNERFYK3HRV7TC6A, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.669093554 (7h36m5.220564015s) [noise=true , address=145.79.227.21:4474 , piece_id=J5XE2MTCEW4ZHQDDZ4KCVYFK5EX6RQZARKNY4QCH4HUHGJO3SXDQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.669510097 (7h36m5.220980568s) [noise=true , address=77.90.31.18:4807 , piece_id=HDJAJYXNZNZN3LVAQSYVJEOQ3MU5H3BXCCW3NA2GLKPT7ATZHSJQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.671936887 (7h36m5.223407348s) [noise=true , address=145.79.227.20:4439 , piece_id=PPOVWSRAU7Y4CZAKAXV6ZP74EEPHONUC35O6XM57VE47NVJL5WZA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.672494549 (7h36m5.22396502s) [noise=true , address=77.90.31.21:4877 , piece_id=3V3Z6GNF32Q66EOS66WV3AGJNA7IXT5NHVAPKNUGVYU3R46IKKKA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:08.438197970 (7h36m6.989668451s) [upload=6714]: Next returned nil segment Nov 9 15:40:08.438219050 (7h36m6.989689521s) [upload=6714]: Waiting for error group managing segments... Nov 9 15:40:08.829503696 (7h36m7.380974147s) [noise=true , address=77.90.31.18:4807 , piece_id=HDJAJYXNZNZN3LVAQSYVJEOQ3MU5H3BXCCW3NA2GLKPT7ATZHSJQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.846923116 (7h36m7.398393577s) [noise=true , address=45.91.116.14:4190 , piece_id=XSIDZL5HMLZTQE7ZAPURAAUGQLPP3QOZK5BIE5PHRMZPSTN5APSA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.867005608 (7h36m7.418476069s) [noise=true , address=77.90.31.21:4877 , piece_id=3V3Z6GNF32Q66EOS66WV3AGJNA7IXT5NHVAPKNUGVYU3R46IKKKA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.888352807 (7h36m7.439823268s) [noise=true , address=77.90.31.14:4662 , piece_id=HHSJDZPYKGCHURAZV46OQSGBU7D4TFPXREDBFPGTYF7OAUMT2X3A, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.904680649 (7h36m7.45615111s) [noise=true , address=77.90.31.17:4772 , piece_id=CT6BNOYZGDBUEFVGM2RDS4ICMWPYS6N4IOUDA5V7MJQ3IKELNW2Q, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:18.663082162 (7h36m17.214552623s) [noise=true , address=145.79.227.17:4971 , piece_id=VTUI377UCF44ICILORHEKUO33ICRXM2FECO5EXS7RUDH4VGV7FJA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:20.077194714 (7h36m18.628665185s) [noise=true , address=145.79.227.13:4803 , piece_id=V37DZKBC4CHRGISXEQHTQAGKZFNXQKWXSWBSMHNRQTSJ32WYI2OQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:28.517900114 (7h36m27.069370575s) [noise=true , address=145.79.227.32:4835 , piece_id=EYIBNSVPQPKLKOBHKZI2VZHVCDKW3C3YJSFEWP4G62UTBXX2YRZA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:28.567571990 (7h36m27.119042451s) [noise=true , address=145.79.227.31:4797 , piece_id=VQEEGJTLJY2IMCDHL2GMVGYEX4VL34TMMXKDYNERFYK3HRV7TC6A, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:33.651382593 (7h36m32.202853044s) [noise=true , address=145.79.227.20:4439 , piece_id=PPOVWSRAU7Y4CZAKAXV6ZP74EEPHONUC35O6XM57VE47NVJL5WZA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:34.842221635 (7h36m33.393692096s) [noise=true , address=145.79.227.9:4365 , piece_id=7BK6E45EY6Y3SV6PY3A4EVCS42PC4HYLXU3J7MN4HXRLDEFUEFCA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:35.601406517 (7h36m34.152877008s) [noise=true , address=145.79.227.25:4619 , piece_id=23D3MCZRVNPI5KB43ROAHKUALDGXLZBZU5E6EIU6U2FY3YQOVPGA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:35.697082303 (7h36m34.248552774s) [noise=true , address=145.79.227.17:4970 , piece_id=5ZP5DDC3XAYQ5UFYJVQV6IZHB5KQQGNG54MTYW3UUYPN2IAK5QVQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697252518 (7h36m34.248722969s) [noise=true , address=145.79.227.17:4972 , piece_id=AI3KVNUJKJUD33YVU2Z472R72SKQKRYQG65JPKFZ5WBJLKRLW6MA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697302322 (7h36m34.248772773s) [noise=true , address=145.79.227.21:4474 , piece_id=J5XE2MTCEW4ZHQDDZ4KCVYFK5EX6RQZARKNY4QCH4HUHGJO3SXDQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, address:145.79.227.21:4474): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697472546 (7h36m34.248943007s) [noise=true , address=145.79.227.9:4366 , piece_id=BT3GWQSXIMMSFPUYJC5EQ2LX2DS4BW5DAMW6ILLNVVRUUHZMOQUA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697514686 (7h36m34.248985147s) [seg_pos={0 0}, upload=6714]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 9 15:40:35.950211122 (7h36m34.501681583s) [upload=6714]: Done uploading segments... Nov 9 15:40:35.950416643 (7h36m34.501887104s) [upload=6714]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 8.794492215s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 9.39485646s trace: https://jaeger.dev.storj.io/trace/3846516a5cecd778 FAIL (2025-11-09 15:50:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 9 15:40:06.410809146 (7h36m4.962279607s) [upload=6714]: Starting upload Nov 9 15:40:06.411046298 (7h36m4.962516769s) [upload=6714]: Done starting upload Nov 9 15:40:06.411314729 (7h36m4.96278519s) [upload=6714]: Uploading segments... Nov 9 15:40:06.411410882 (7h36m4.962881383s) [upload=6714]: Got next segment. Inline: false Nov 9 15:40:06.411464574 (7h36m4.962935035s) [upload=6714]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 9 15:40:06.411475064 (7h36m4.962945555s) [upload=6714]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 9 15:40:06.664789367 (7h36m5.216259828s) [seg_pos={0 0}, upload=6714]: Begin upload segment... Nov 9 15:40:06.664851705 (7h36m5.216322166s) [seg_pos={0 0}, upload=6714]: Done begin upload segment. Nov 9 15:40:06.664891662 (7h36m5.216362123s) [noise=true , address=145.79.227.32:4835 , piece_id=EYIBNSVPQPKLKOBHKZI2VZHVCDKW3C3YJSFEWP4G62UTBXX2YRZA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.664996933 (7h36m5.216467394s) [noise=true , address=45.91.116.14:4190 , piece_id=XSIDZL5HMLZTQE7ZAPURAAUGQLPP3QOZK5BIE5PHRMZPSTN5APSA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.665698789 (7h36m5.21716925s) [noise=true , address=77.90.31.14:4662 , piece_id=HHSJDZPYKGCHURAZV46OQSGBU7D4TFPXREDBFPGTYF7OAUMT2X3A, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.666456111 (7h36m5.217926572s) [noise=true , address=145.79.227.25:4619 , piece_id=23D3MCZRVNPI5KB43ROAHKUALDGXLZBZU5E6EIU6U2FY3YQOVPGA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.666642225 (7h36m5.218112676s) [noise=true , address=145.79.227.17:4971 , piece_id=VTUI377UCF44ICILORHEKUO33ICRXM2FECO5EXS7RUDH4VGV7FJA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667027539 (7h36m5.218498s) [noise=true , address=145.79.227.9:4366 , piece_id=BT3GWQSXIMMSFPUYJC5EQ2LX2DS4BW5DAMW6ILLNVVRUUHZMOQUA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667041675 (7h36m5.218512126s) [noise=true , address=145.79.227.17:4972 , piece_id=AI3KVNUJKJUD33YVU2Z472R72SKQKRYQG65JPKFZ5WBJLKRLW6MA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667502794 (7h36m5.218973255s) [noise=true , address=145.79.227.13:4803 , piece_id=V37DZKBC4CHRGISXEQHTQAGKZFNXQKWXSWBSMHNRQTSJ32WYI2OQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667805441 (7h36m5.219275902s) [noise=true , address=145.79.227.9:4365 , piece_id=7BK6E45EY6Y3SV6PY3A4EVCS42PC4HYLXU3J7MN4HXRLDEFUEFCA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667895652 (7h36m5.219366113s) [noise=true , address=145.79.227.17:4970 , piece_id=5ZP5DDC3XAYQ5UFYJVQV6IZHB5KQQGNG54MTYW3UUYPN2IAK5QVQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.668240909 (7h36m5.219711361s) [noise=true , address=77.90.31.17:4772 , piece_id=CT6BNOYZGDBUEFVGM2RDS4ICMWPYS6N4IOUDA5V7MJQ3IKELNW2Q, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.668671640 (7h36m5.220142101s) [noise=true , address=145.79.227.31:4797 , piece_id=VQEEGJTLJY2IMCDHL2GMVGYEX4VL34TMMXKDYNERFYK3HRV7TC6A, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.669093554 (7h36m5.220564015s) [noise=true , address=145.79.227.21:4474 , piece_id=J5XE2MTCEW4ZHQDDZ4KCVYFK5EX6RQZARKNY4QCH4HUHGJO3SXDQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.669510097 (7h36m5.220980568s) [noise=true , address=77.90.31.18:4807 , piece_id=HDJAJYXNZNZN3LVAQSYVJEOQ3MU5H3BXCCW3NA2GLKPT7ATZHSJQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.671936887 (7h36m5.223407348s) [noise=true , address=145.79.227.20:4439 , piece_id=PPOVWSRAU7Y4CZAKAXV6ZP74EEPHONUC35O6XM57VE47NVJL5WZA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.672494549 (7h36m5.22396502s) [noise=true , address=77.90.31.21:4877 , piece_id=3V3Z6GNF32Q66EOS66WV3AGJNA7IXT5NHVAPKNUGVYU3R46IKKKA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:08.438197970 (7h36m6.989668451s) [upload=6714]: Next returned nil segment Nov 9 15:40:08.438219050 (7h36m6.989689521s) [upload=6714]: Waiting for error group managing segments... Nov 9 15:40:08.829503696 (7h36m7.380974147s) [noise=true , address=77.90.31.18:4807 , piece_id=HDJAJYXNZNZN3LVAQSYVJEOQ3MU5H3BXCCW3NA2GLKPT7ATZHSJQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.846923116 (7h36m7.398393577s) [noise=true , address=45.91.116.14:4190 , piece_id=XSIDZL5HMLZTQE7ZAPURAAUGQLPP3QOZK5BIE5PHRMZPSTN5APSA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.867005608 (7h36m7.418476069s) [noise=true , address=77.90.31.21:4877 , piece_id=3V3Z6GNF32Q66EOS66WV3AGJNA7IXT5NHVAPKNUGVYU3R46IKKKA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.888352807 (7h36m7.439823268s) [noise=true , address=77.90.31.14:4662 , piece_id=HHSJDZPYKGCHURAZV46OQSGBU7D4TFPXREDBFPGTYF7OAUMT2X3A, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.904680649 (7h36m7.45615111s) [noise=true , address=77.90.31.17:4772 , piece_id=CT6BNOYZGDBUEFVGM2RDS4ICMWPYS6N4IOUDA5V7MJQ3IKELNW2Q, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:18.663082162 (7h36m17.214552623s) [noise=true , address=145.79.227.17:4971 , piece_id=VTUI377UCF44ICILORHEKUO33ICRXM2FECO5EXS7RUDH4VGV7FJA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:20.077194714 (7h36m18.628665185s) [noise=true , address=145.79.227.13:4803 , piece_id=V37DZKBC4CHRGISXEQHTQAGKZFNXQKWXSWBSMHNRQTSJ32WYI2OQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:28.517900114 (7h36m27.069370575s) [noise=true , address=145.79.227.32:4835 , piece_id=EYIBNSVPQPKLKOBHKZI2VZHVCDKW3C3YJSFEWP4G62UTBXX2YRZA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:28.567571990 (7h36m27.119042451s) [noise=true , address=145.79.227.31:4797 , piece_id=VQEEGJTLJY2IMCDHL2GMVGYEX4VL34TMMXKDYNERFYK3HRV7TC6A, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:33.651382593 (7h36m32.202853044s) [noise=true , address=145.79.227.20:4439 , piece_id=PPOVWSRAU7Y4CZAKAXV6ZP74EEPHONUC35O6XM57VE47NVJL5WZA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:34.842221635 (7h36m33.393692096s) [noise=true , address=145.79.227.9:4365 , piece_id=7BK6E45EY6Y3SV6PY3A4EVCS42PC4HYLXU3J7MN4HXRLDEFUEFCA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:35.601406517 (7h36m34.152877008s) [noise=true , address=145.79.227.25:4619 , piece_id=23D3MCZRVNPI5KB43ROAHKUALDGXLZBZU5E6EIU6U2FY3YQOVPGA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:35.697082303 (7h36m34.248552774s) [noise=true , address=145.79.227.17:4970 , piece_id=5ZP5DDC3XAYQ5UFYJVQV6IZHB5KQQGNG54MTYW3UUYPN2IAK5QVQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697252518 (7h36m34.248722969s) [noise=true , address=145.79.227.17:4972 , piece_id=AI3KVNUJKJUD33YVU2Z472R72SKQKRYQG65JPKFZ5WBJLKRLW6MA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697302322 (7h36m34.248772773s) [noise=true , address=145.79.227.21:4474 , piece_id=J5XE2MTCEW4ZHQDDZ4KCVYFK5EX6RQZARKNY4QCH4HUHGJO3SXDQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, address:145.79.227.21:4474): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697472546 (7h36m34.248943007s) [noise=true , address=145.79.227.9:4366 , piece_id=BT3GWQSXIMMSFPUYJC5EQ2LX2DS4BW5DAMW6ILLNVVRUUHZMOQUA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697514686 (7h36m34.248985147s) [seg_pos={0 0}, upload=6714]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 9 15:40:35.950211122 (7h36m34.501681583s) [upload=6714]: Done uploading segments... Nov 9 15:40:35.950416643 (7h36m34.501887104s) [upload=6714]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 9.394879994s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [...X...........] duration: 67.522407ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/3ea6c69616edd41 FAIL (2025-11-09 15:50:39 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:59888->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 67.540201ms core website
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [....X..........] duration: 73.712995ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/3ea6c69616edd41 FAIL (2025-11-09 15:49:54 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:59888->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 73.74178ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 9.798383753s trace: https://jaeger.dev.storj.io/trace/3846516a5cecd778 FAIL (2025-11-09 15:49:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 9 15:40:06.410809146 (7h36m4.962279607s) [upload=6714]: Starting upload Nov 9 15:40:06.411046298 (7h36m4.962516769s) [upload=6714]: Done starting upload Nov 9 15:40:06.411314729 (7h36m4.96278519s) [upload=6714]: Uploading segments... Nov 9 15:40:06.411410882 (7h36m4.962881383s) [upload=6714]: Got next segment. Inline: false Nov 9 15:40:06.411464574 (7h36m4.962935035s) [upload=6714]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 9 15:40:06.411475064 (7h36m4.962945555s) [upload=6714]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 9 15:40:06.664789367 (7h36m5.216259828s) [seg_pos={0 0}, upload=6714]: Begin upload segment... Nov 9 15:40:06.664851705 (7h36m5.216322166s) [seg_pos={0 0}, upload=6714]: Done begin upload segment. Nov 9 15:40:06.664891662 (7h36m5.216362123s) [noise=true , address=145.79.227.32:4835 , piece_id=EYIBNSVPQPKLKOBHKZI2VZHVCDKW3C3YJSFEWP4G62UTBXX2YRZA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.664996933 (7h36m5.216467394s) [noise=true , address=45.91.116.14:4190 , piece_id=XSIDZL5HMLZTQE7ZAPURAAUGQLPP3QOZK5BIE5PHRMZPSTN5APSA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.665698789 (7h36m5.21716925s) [noise=true , address=77.90.31.14:4662 , piece_id=HHSJDZPYKGCHURAZV46OQSGBU7D4TFPXREDBFPGTYF7OAUMT2X3A, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.666456111 (7h36m5.217926572s) [noise=true , address=145.79.227.25:4619 , piece_id=23D3MCZRVNPI5KB43ROAHKUALDGXLZBZU5E6EIU6U2FY3YQOVPGA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.666642225 (7h36m5.218112676s) [noise=true , address=145.79.227.17:4971 , piece_id=VTUI377UCF44ICILORHEKUO33ICRXM2FECO5EXS7RUDH4VGV7FJA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667027539 (7h36m5.218498s) [noise=true , address=145.79.227.9:4366 , piece_id=BT3GWQSXIMMSFPUYJC5EQ2LX2DS4BW5DAMW6ILLNVVRUUHZMOQUA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667041675 (7h36m5.218512126s) [noise=true , address=145.79.227.17:4972 , piece_id=AI3KVNUJKJUD33YVU2Z472R72SKQKRYQG65JPKFZ5WBJLKRLW6MA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667502794 (7h36m5.218973255s) [noise=true , address=145.79.227.13:4803 , piece_id=V37DZKBC4CHRGISXEQHTQAGKZFNXQKWXSWBSMHNRQTSJ32WYI2OQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667805441 (7h36m5.219275902s) [noise=true , address=145.79.227.9:4365 , piece_id=7BK6E45EY6Y3SV6PY3A4EVCS42PC4HYLXU3J7MN4HXRLDEFUEFCA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667895652 (7h36m5.219366113s) [noise=true , address=145.79.227.17:4970 , piece_id=5ZP5DDC3XAYQ5UFYJVQV6IZHB5KQQGNG54MTYW3UUYPN2IAK5QVQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.668240909 (7h36m5.219711361s) [noise=true , address=77.90.31.17:4772 , piece_id=CT6BNOYZGDBUEFVGM2RDS4ICMWPYS6N4IOUDA5V7MJQ3IKELNW2Q, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.668671640 (7h36m5.220142101s) [noise=true , address=145.79.227.31:4797 , piece_id=VQEEGJTLJY2IMCDHL2GMVGYEX4VL34TMMXKDYNERFYK3HRV7TC6A, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.669093554 (7h36m5.220564015s) [noise=true , address=145.79.227.21:4474 , piece_id=J5XE2MTCEW4ZHQDDZ4KCVYFK5EX6RQZARKNY4QCH4HUHGJO3SXDQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.669510097 (7h36m5.220980568s) [noise=true , address=77.90.31.18:4807 , piece_id=HDJAJYXNZNZN3LVAQSYVJEOQ3MU5H3BXCCW3NA2GLKPT7ATZHSJQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.671936887 (7h36m5.223407348s) [noise=true , address=145.79.227.20:4439 , piece_id=PPOVWSRAU7Y4CZAKAXV6ZP74EEPHONUC35O6XM57VE47NVJL5WZA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.672494549 (7h36m5.22396502s) [noise=true , address=77.90.31.21:4877 , piece_id=3V3Z6GNF32Q66EOS66WV3AGJNA7IXT5NHVAPKNUGVYU3R46IKKKA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:08.438197970 (7h36m6.989668451s) [upload=6714]: Next returned nil segment Nov 9 15:40:08.438219050 (7h36m6.989689521s) [upload=6714]: Waiting for error group managing segments... Nov 9 15:40:08.829503696 (7h36m7.380974147s) [noise=true , address=77.90.31.18:4807 , piece_id=HDJAJYXNZNZN3LVAQSYVJEOQ3MU5H3BXCCW3NA2GLKPT7ATZHSJQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.846923116 (7h36m7.398393577s) [noise=true , address=45.91.116.14:4190 , piece_id=XSIDZL5HMLZTQE7ZAPURAAUGQLPP3QOZK5BIE5PHRMZPSTN5APSA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.867005608 (7h36m7.418476069s) [noise=true , address=77.90.31.21:4877 , piece_id=3V3Z6GNF32Q66EOS66WV3AGJNA7IXT5NHVAPKNUGVYU3R46IKKKA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.888352807 (7h36m7.439823268s) [noise=true , address=77.90.31.14:4662 , piece_id=HHSJDZPYKGCHURAZV46OQSGBU7D4TFPXREDBFPGTYF7OAUMT2X3A, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.904680649 (7h36m7.45615111s) [noise=true , address=77.90.31.17:4772 , piece_id=CT6BNOYZGDBUEFVGM2RDS4ICMWPYS6N4IOUDA5V7MJQ3IKELNW2Q, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:18.663082162 (7h36m17.214552623s) [noise=true , address=145.79.227.17:4971 , piece_id=VTUI377UCF44ICILORHEKUO33ICRXM2FECO5EXS7RUDH4VGV7FJA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:20.077194714 (7h36m18.628665185s) [noise=true , address=145.79.227.13:4803 , piece_id=V37DZKBC4CHRGISXEQHTQAGKZFNXQKWXSWBSMHNRQTSJ32WYI2OQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:28.517900114 (7h36m27.069370575s) [noise=true , address=145.79.227.32:4835 , piece_id=EYIBNSVPQPKLKOBHKZI2VZHVCDKW3C3YJSFEWP4G62UTBXX2YRZA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:28.567571990 (7h36m27.119042451s) [noise=true , address=145.79.227.31:4797 , piece_id=VQEEGJTLJY2IMCDHL2GMVGYEX4VL34TMMXKDYNERFYK3HRV7TC6A, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:33.651382593 (7h36m32.202853044s) [noise=true , address=145.79.227.20:4439 , piece_id=PPOVWSRAU7Y4CZAKAXV6ZP74EEPHONUC35O6XM57VE47NVJL5WZA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:34.842221635 (7h36m33.393692096s) [noise=true , address=145.79.227.9:4365 , piece_id=7BK6E45EY6Y3SV6PY3A4EVCS42PC4HYLXU3J7MN4HXRLDEFUEFCA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:35.601406517 (7h36m34.152877008s) [noise=true , address=145.79.227.25:4619 , piece_id=23D3MCZRVNPI5KB43ROAHKUALDGXLZBZU5E6EIU6U2FY3YQOVPGA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:35.697082303 (7h36m34.248552774s) [noise=true , address=145.79.227.17:4970 , piece_id=5ZP5DDC3XAYQ5UFYJVQV6IZHB5KQQGNG54MTYW3UUYPN2IAK5QVQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697252518 (7h36m34.248722969s) [noise=true , address=145.79.227.17:4972 , piece_id=AI3KVNUJKJUD33YVU2Z472R72SKQKRYQG65JPKFZ5WBJLKRLW6MA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697302322 (7h36m34.248772773s) [noise=true , address=145.79.227.21:4474 , piece_id=J5XE2MTCEW4ZHQDDZ4KCVYFK5EX6RQZARKNY4QCH4HUHGJO3SXDQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, address:145.79.227.21:4474): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697472546 (7h36m34.248943007s) [noise=true , address=145.79.227.9:4366 , piece_id=BT3GWQSXIMMSFPUYJC5EQ2LX2DS4BW5DAMW6ILLNVVRUUHZMOQUA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697514686 (7h36m34.248985147s) [seg_pos={0 0}, upload=6714]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 9 15:40:35.950211122 (7h36m34.501681583s) [upload=6714]: Done uploading segments... Nov 9 15:40:35.950416643 (7h36m34.501887104s) [upload=6714]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 9.798413749s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 9.149214198s trace: https://jaeger.dev.storj.io/trace/3846516a5cecd778 FAIL (2025-11-09 15:49:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 9 15:40:06.410809146 (7h36m4.962279607s) [upload=6714]: Starting upload Nov 9 15:40:06.411046298 (7h36m4.962516769s) [upload=6714]: Done starting upload Nov 9 15:40:06.411314729 (7h36m4.96278519s) [upload=6714]: Uploading segments... Nov 9 15:40:06.411410882 (7h36m4.962881383s) [upload=6714]: Got next segment. Inline: false Nov 9 15:40:06.411464574 (7h36m4.962935035s) [upload=6714]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 9 15:40:06.411475064 (7h36m4.962945555s) [upload=6714]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 9 15:40:06.664789367 (7h36m5.216259828s) [seg_pos={0 0}, upload=6714]: Begin upload segment... Nov 9 15:40:06.664851705 (7h36m5.216322166s) [seg_pos={0 0}, upload=6714]: Done begin upload segment. Nov 9 15:40:06.664891662 (7h36m5.216362123s) [noise=true , address=145.79.227.32:4835 , piece_id=EYIBNSVPQPKLKOBHKZI2VZHVCDKW3C3YJSFEWP4G62UTBXX2YRZA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.664996933 (7h36m5.216467394s) [noise=true , address=45.91.116.14:4190 , piece_id=XSIDZL5HMLZTQE7ZAPURAAUGQLPP3QOZK5BIE5PHRMZPSTN5APSA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.665698789 (7h36m5.21716925s) [noise=true , address=77.90.31.14:4662 , piece_id=HHSJDZPYKGCHURAZV46OQSGBU7D4TFPXREDBFPGTYF7OAUMT2X3A, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.666456111 (7h36m5.217926572s) [noise=true , address=145.79.227.25:4619 , piece_id=23D3MCZRVNPI5KB43ROAHKUALDGXLZBZU5E6EIU6U2FY3YQOVPGA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.666642225 (7h36m5.218112676s) [noise=true , address=145.79.227.17:4971 , piece_id=VTUI377UCF44ICILORHEKUO33ICRXM2FECO5EXS7RUDH4VGV7FJA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667027539 (7h36m5.218498s) [noise=true , address=145.79.227.9:4366 , piece_id=BT3GWQSXIMMSFPUYJC5EQ2LX2DS4BW5DAMW6ILLNVVRUUHZMOQUA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667041675 (7h36m5.218512126s) [noise=true , address=145.79.227.17:4972 , piece_id=AI3KVNUJKJUD33YVU2Z472R72SKQKRYQG65JPKFZ5WBJLKRLW6MA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667502794 (7h36m5.218973255s) [noise=true , address=145.79.227.13:4803 , piece_id=V37DZKBC4CHRGISXEQHTQAGKZFNXQKWXSWBSMHNRQTSJ32WYI2OQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667805441 (7h36m5.219275902s) [noise=true , address=145.79.227.9:4365 , piece_id=7BK6E45EY6Y3SV6PY3A4EVCS42PC4HYLXU3J7MN4HXRLDEFUEFCA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667895652 (7h36m5.219366113s) [noise=true , address=145.79.227.17:4970 , piece_id=5ZP5DDC3XAYQ5UFYJVQV6IZHB5KQQGNG54MTYW3UUYPN2IAK5QVQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.668240909 (7h36m5.219711361s) [noise=true , address=77.90.31.17:4772 , piece_id=CT6BNOYZGDBUEFVGM2RDS4ICMWPYS6N4IOUDA5V7MJQ3IKELNW2Q, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.668671640 (7h36m5.220142101s) [noise=true , address=145.79.227.31:4797 , piece_id=VQEEGJTLJY2IMCDHL2GMVGYEX4VL34TMMXKDYNERFYK3HRV7TC6A, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.669093554 (7h36m5.220564015s) [noise=true , address=145.79.227.21:4474 , piece_id=J5XE2MTCEW4ZHQDDZ4KCVYFK5EX6RQZARKNY4QCH4HUHGJO3SXDQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.669510097 (7h36m5.220980568s) [noise=true , address=77.90.31.18:4807 , piece_id=HDJAJYXNZNZN3LVAQSYVJEOQ3MU5H3BXCCW3NA2GLKPT7ATZHSJQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.671936887 (7h36m5.223407348s) [noise=true , address=145.79.227.20:4439 , piece_id=PPOVWSRAU7Y4CZAKAXV6ZP74EEPHONUC35O6XM57VE47NVJL5WZA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.672494549 (7h36m5.22396502s) [noise=true , address=77.90.31.21:4877 , piece_id=3V3Z6GNF32Q66EOS66WV3AGJNA7IXT5NHVAPKNUGVYU3R46IKKKA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:08.438197970 (7h36m6.989668451s) [upload=6714]: Next returned nil segment Nov 9 15:40:08.438219050 (7h36m6.989689521s) [upload=6714]: Waiting for error group managing segments... Nov 9 15:40:08.829503696 (7h36m7.380974147s) [noise=true , address=77.90.31.18:4807 , piece_id=HDJAJYXNZNZN3LVAQSYVJEOQ3MU5H3BXCCW3NA2GLKPT7ATZHSJQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.846923116 (7h36m7.398393577s) [noise=true , address=45.91.116.14:4190 , piece_id=XSIDZL5HMLZTQE7ZAPURAAUGQLPP3QOZK5BIE5PHRMZPSTN5APSA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.867005608 (7h36m7.418476069s) [noise=true , address=77.90.31.21:4877 , piece_id=3V3Z6GNF32Q66EOS66WV3AGJNA7IXT5NHVAPKNUGVYU3R46IKKKA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.888352807 (7h36m7.439823268s) [noise=true , address=77.90.31.14:4662 , piece_id=HHSJDZPYKGCHURAZV46OQSGBU7D4TFPXREDBFPGTYF7OAUMT2X3A, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.904680649 (7h36m7.45615111s) [noise=true , address=77.90.31.17:4772 , piece_id=CT6BNOYZGDBUEFVGM2RDS4ICMWPYS6N4IOUDA5V7MJQ3IKELNW2Q, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:18.663082162 (7h36m17.214552623s) [noise=true , address=145.79.227.17:4971 , piece_id=VTUI377UCF44ICILORHEKUO33ICRXM2FECO5EXS7RUDH4VGV7FJA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:20.077194714 (7h36m18.628665185s) [noise=true , address=145.79.227.13:4803 , piece_id=V37DZKBC4CHRGISXEQHTQAGKZFNXQKWXSWBSMHNRQTSJ32WYI2OQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:28.517900114 (7h36m27.069370575s) [noise=true , address=145.79.227.32:4835 , piece_id=EYIBNSVPQPKLKOBHKZI2VZHVCDKW3C3YJSFEWP4G62UTBXX2YRZA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:28.567571990 (7h36m27.119042451s) [noise=true , address=145.79.227.31:4797 , piece_id=VQEEGJTLJY2IMCDHL2GMVGYEX4VL34TMMXKDYNERFYK3HRV7TC6A, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:33.651382593 (7h36m32.202853044s) [noise=true , address=145.79.227.20:4439 , piece_id=PPOVWSRAU7Y4CZAKAXV6ZP74EEPHONUC35O6XM57VE47NVJL5WZA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:34.842221635 (7h36m33.393692096s) [noise=true , address=145.79.227.9:4365 , piece_id=7BK6E45EY6Y3SV6PY3A4EVCS42PC4HYLXU3J7MN4HXRLDEFUEFCA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:35.601406517 (7h36m34.152877008s) [noise=true , address=145.79.227.25:4619 , piece_id=23D3MCZRVNPI5KB43ROAHKUALDGXLZBZU5E6EIU6U2FY3YQOVPGA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:35.697082303 (7h36m34.248552774s) [noise=true , address=145.79.227.17:4970 , piece_id=5ZP5DDC3XAYQ5UFYJVQV6IZHB5KQQGNG54MTYW3UUYPN2IAK5QVQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697252518 (7h36m34.248722969s) [noise=true , address=145.79.227.17:4972 , piece_id=AI3KVNUJKJUD33YVU2Z472R72SKQKRYQG65JPKFZ5WBJLKRLW6MA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697302322 (7h36m34.248772773s) [noise=true , address=145.79.227.21:4474 , piece_id=J5XE2MTCEW4ZHQDDZ4KCVYFK5EX6RQZARKNY4QCH4HUHGJO3SXDQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, address:145.79.227.21:4474): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697472546 (7h36m34.248943007s) [noise=true , address=145.79.227.9:4366 , piece_id=BT3GWQSXIMMSFPUYJC5EQ2LX2DS4BW5DAMW6ILLNVVRUUHZMOQUA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697514686 (7h36m34.248985147s) [seg_pos={0 0}, upload=6714]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 9 15:40:35.950211122 (7h36m34.501681583s) [upload=6714]: Done uploading segments... Nov 9 15:40:35.950416643 (7h36m34.501887104s) [upload=6714]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 9.149261827s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [.....X.........] duration: 77.920176ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/3ea6c69616edd41 FAIL (2025-11-09 15:48:53 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:59888->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 77.954532ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 9.483103219s trace: https://jaeger.dev.storj.io/trace/3846516a5cecd778 FAIL (2025-11-09 15:48:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 9 15:40:06.410809146 (7h36m4.962279607s) [upload=6714]: Starting upload Nov 9 15:40:06.411046298 (7h36m4.962516769s) [upload=6714]: Done starting upload Nov 9 15:40:06.411314729 (7h36m4.96278519s) [upload=6714]: Uploading segments... Nov 9 15:40:06.411410882 (7h36m4.962881383s) [upload=6714]: Got next segment. Inline: false Nov 9 15:40:06.411464574 (7h36m4.962935035s) [upload=6714]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 9 15:40:06.411475064 (7h36m4.962945555s) [upload=6714]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 9 15:40:06.664789367 (7h36m5.216259828s) [seg_pos={0 0}, upload=6714]: Begin upload segment... Nov 9 15:40:06.664851705 (7h36m5.216322166s) [seg_pos={0 0}, upload=6714]: Done begin upload segment. Nov 9 15:40:06.664891662 (7h36m5.216362123s) [noise=true , address=145.79.227.32:4835 , piece_id=EYIBNSVPQPKLKOBHKZI2VZHVCDKW3C3YJSFEWP4G62UTBXX2YRZA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.664996933 (7h36m5.216467394s) [noise=true , address=45.91.116.14:4190 , piece_id=XSIDZL5HMLZTQE7ZAPURAAUGQLPP3QOZK5BIE5PHRMZPSTN5APSA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.665698789 (7h36m5.21716925s) [noise=true , address=77.90.31.14:4662 , piece_id=HHSJDZPYKGCHURAZV46OQSGBU7D4TFPXREDBFPGTYF7OAUMT2X3A, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.666456111 (7h36m5.217926572s) [noise=true , address=145.79.227.25:4619 , piece_id=23D3MCZRVNPI5KB43ROAHKUALDGXLZBZU5E6EIU6U2FY3YQOVPGA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.666642225 (7h36m5.218112676s) [noise=true , address=145.79.227.17:4971 , piece_id=VTUI377UCF44ICILORHEKUO33ICRXM2FECO5EXS7RUDH4VGV7FJA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667027539 (7h36m5.218498s) [noise=true , address=145.79.227.9:4366 , piece_id=BT3GWQSXIMMSFPUYJC5EQ2LX2DS4BW5DAMW6ILLNVVRUUHZMOQUA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667041675 (7h36m5.218512126s) [noise=true , address=145.79.227.17:4972 , piece_id=AI3KVNUJKJUD33YVU2Z472R72SKQKRYQG65JPKFZ5WBJLKRLW6MA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667502794 (7h36m5.218973255s) [noise=true , address=145.79.227.13:4803 , piece_id=V37DZKBC4CHRGISXEQHTQAGKZFNXQKWXSWBSMHNRQTSJ32WYI2OQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667805441 (7h36m5.219275902s) [noise=true , address=145.79.227.9:4365 , piece_id=7BK6E45EY6Y3SV6PY3A4EVCS42PC4HYLXU3J7MN4HXRLDEFUEFCA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667895652 (7h36m5.219366113s) [noise=true , address=145.79.227.17:4970 , piece_id=5ZP5DDC3XAYQ5UFYJVQV6IZHB5KQQGNG54MTYW3UUYPN2IAK5QVQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.668240909 (7h36m5.219711361s) [noise=true , address=77.90.31.17:4772 , piece_id=CT6BNOYZGDBUEFVGM2RDS4ICMWPYS6N4IOUDA5V7MJQ3IKELNW2Q, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.668671640 (7h36m5.220142101s) [noise=true , address=145.79.227.31:4797 , piece_id=VQEEGJTLJY2IMCDHL2GMVGYEX4VL34TMMXKDYNERFYK3HRV7TC6A, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.669093554 (7h36m5.220564015s) [noise=true , address=145.79.227.21:4474 , piece_id=J5XE2MTCEW4ZHQDDZ4KCVYFK5EX6RQZARKNY4QCH4HUHGJO3SXDQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.669510097 (7h36m5.220980568s) [noise=true , address=77.90.31.18:4807 , piece_id=HDJAJYXNZNZN3LVAQSYVJEOQ3MU5H3BXCCW3NA2GLKPT7ATZHSJQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.671936887 (7h36m5.223407348s) [noise=true , address=145.79.227.20:4439 , piece_id=PPOVWSRAU7Y4CZAKAXV6ZP74EEPHONUC35O6XM57VE47NVJL5WZA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.672494549 (7h36m5.22396502s) [noise=true , address=77.90.31.21:4877 , piece_id=3V3Z6GNF32Q66EOS66WV3AGJNA7IXT5NHVAPKNUGVYU3R46IKKKA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:08.438197970 (7h36m6.989668451s) [upload=6714]: Next returned nil segment Nov 9 15:40:08.438219050 (7h36m6.989689521s) [upload=6714]: Waiting for error group managing segments... Nov 9 15:40:08.829503696 (7h36m7.380974147s) [noise=true , address=77.90.31.18:4807 , piece_id=HDJAJYXNZNZN3LVAQSYVJEOQ3MU5H3BXCCW3NA2GLKPT7ATZHSJQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.846923116 (7h36m7.398393577s) [noise=true , address=45.91.116.14:4190 , piece_id=XSIDZL5HMLZTQE7ZAPURAAUGQLPP3QOZK5BIE5PHRMZPSTN5APSA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.867005608 (7h36m7.418476069s) [noise=true , address=77.90.31.21:4877 , piece_id=3V3Z6GNF32Q66EOS66WV3AGJNA7IXT5NHVAPKNUGVYU3R46IKKKA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.888352807 (7h36m7.439823268s) [noise=true , address=77.90.31.14:4662 , piece_id=HHSJDZPYKGCHURAZV46OQSGBU7D4TFPXREDBFPGTYF7OAUMT2X3A, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.904680649 (7h36m7.45615111s) [noise=true , address=77.90.31.17:4772 , piece_id=CT6BNOYZGDBUEFVGM2RDS4ICMWPYS6N4IOUDA5V7MJQ3IKELNW2Q, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:18.663082162 (7h36m17.214552623s) [noise=true , address=145.79.227.17:4971 , piece_id=VTUI377UCF44ICILORHEKUO33ICRXM2FECO5EXS7RUDH4VGV7FJA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:20.077194714 (7h36m18.628665185s) [noise=true , address=145.79.227.13:4803 , piece_id=V37DZKBC4CHRGISXEQHTQAGKZFNXQKWXSWBSMHNRQTSJ32WYI2OQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:28.517900114 (7h36m27.069370575s) [noise=true , address=145.79.227.32:4835 , piece_id=EYIBNSVPQPKLKOBHKZI2VZHVCDKW3C3YJSFEWP4G62UTBXX2YRZA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:28.567571990 (7h36m27.119042451s) [noise=true , address=145.79.227.31:4797 , piece_id=VQEEGJTLJY2IMCDHL2GMVGYEX4VL34TMMXKDYNERFYK3HRV7TC6A, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:33.651382593 (7h36m32.202853044s) [noise=true , address=145.79.227.20:4439 , piece_id=PPOVWSRAU7Y4CZAKAXV6ZP74EEPHONUC35O6XM57VE47NVJL5WZA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:34.842221635 (7h36m33.393692096s) [noise=true , address=145.79.227.9:4365 , piece_id=7BK6E45EY6Y3SV6PY3A4EVCS42PC4HYLXU3J7MN4HXRLDEFUEFCA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:35.601406517 (7h36m34.152877008s) [noise=true , address=145.79.227.25:4619 , piece_id=23D3MCZRVNPI5KB43ROAHKUALDGXLZBZU5E6EIU6U2FY3YQOVPGA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:35.697082303 (7h36m34.248552774s) [noise=true , address=145.79.227.17:4970 , piece_id=5ZP5DDC3XAYQ5UFYJVQV6IZHB5KQQGNG54MTYW3UUYPN2IAK5QVQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697252518 (7h36m34.248722969s) [noise=true , address=145.79.227.17:4972 , piece_id=AI3KVNUJKJUD33YVU2Z472R72SKQKRYQG65JPKFZ5WBJLKRLW6MA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697302322 (7h36m34.248772773s) [noise=true , address=145.79.227.21:4474 , piece_id=J5XE2MTCEW4ZHQDDZ4KCVYFK5EX6RQZARKNY4QCH4HUHGJO3SXDQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, address:145.79.227.21:4474): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697472546 (7h36m34.248943007s) [noise=true , address=145.79.227.9:4366 , piece_id=BT3GWQSXIMMSFPUYJC5EQ2LX2DS4BW5DAMW6ILLNVVRUUHZMOQUA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697514686 (7h36m34.248985147s) [seg_pos={0 0}, upload=6714]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 9 15:40:35.950211122 (7h36m34.501681583s) [upload=6714]: Done uploading segments... Nov 9 15:40:35.950416643 (7h36m34.501887104s) [upload=6714]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 9.483146463s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [X..............] duration: 310.813983ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/47f4a3db63391ab8 FAIL (2025-11-09 15:48:17 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:54114->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 310.851696ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 9.069689221s trace: https://jaeger.dev.storj.io/trace/3846516a5cecd778 FAIL (2025-11-09 15:47:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 9 15:40:06.410809146 (7h36m4.962279607s) [upload=6714]: Starting upload Nov 9 15:40:06.411046298 (7h36m4.962516769s) [upload=6714]: Done starting upload Nov 9 15:40:06.411314729 (7h36m4.96278519s) [upload=6714]: Uploading segments... Nov 9 15:40:06.411410882 (7h36m4.962881383s) [upload=6714]: Got next segment. Inline: false Nov 9 15:40:06.411464574 (7h36m4.962935035s) [upload=6714]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 9 15:40:06.411475064 (7h36m4.962945555s) [upload=6714]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 9 15:40:06.664789367 (7h36m5.216259828s) [seg_pos={0 0}, upload=6714]: Begin upload segment... Nov 9 15:40:06.664851705 (7h36m5.216322166s) [seg_pos={0 0}, upload=6714]: Done begin upload segment. Nov 9 15:40:06.664891662 (7h36m5.216362123s) [noise=true , address=145.79.227.32:4835 , piece_id=EYIBNSVPQPKLKOBHKZI2VZHVCDKW3C3YJSFEWP4G62UTBXX2YRZA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.664996933 (7h36m5.216467394s) [noise=true , address=45.91.116.14:4190 , piece_id=XSIDZL5HMLZTQE7ZAPURAAUGQLPP3QOZK5BIE5PHRMZPSTN5APSA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.665698789 (7h36m5.21716925s) [noise=true , address=77.90.31.14:4662 , piece_id=HHSJDZPYKGCHURAZV46OQSGBU7D4TFPXREDBFPGTYF7OAUMT2X3A, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.666456111 (7h36m5.217926572s) [noise=true , address=145.79.227.25:4619 , piece_id=23D3MCZRVNPI5KB43ROAHKUALDGXLZBZU5E6EIU6U2FY3YQOVPGA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.666642225 (7h36m5.218112676s) [noise=true , address=145.79.227.17:4971 , piece_id=VTUI377UCF44ICILORHEKUO33ICRXM2FECO5EXS7RUDH4VGV7FJA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667027539 (7h36m5.218498s) [noise=true , address=145.79.227.9:4366 , piece_id=BT3GWQSXIMMSFPUYJC5EQ2LX2DS4BW5DAMW6ILLNVVRUUHZMOQUA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667041675 (7h36m5.218512126s) [noise=true , address=145.79.227.17:4972 , piece_id=AI3KVNUJKJUD33YVU2Z472R72SKQKRYQG65JPKFZ5WBJLKRLW6MA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667502794 (7h36m5.218973255s) [noise=true , address=145.79.227.13:4803 , piece_id=V37DZKBC4CHRGISXEQHTQAGKZFNXQKWXSWBSMHNRQTSJ32WYI2OQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667805441 (7h36m5.219275902s) [noise=true , address=145.79.227.9:4365 , piece_id=7BK6E45EY6Y3SV6PY3A4EVCS42PC4HYLXU3J7MN4HXRLDEFUEFCA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667895652 (7h36m5.219366113s) [noise=true , address=145.79.227.17:4970 , piece_id=5ZP5DDC3XAYQ5UFYJVQV6IZHB5KQQGNG54MTYW3UUYPN2IAK5QVQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.668240909 (7h36m5.219711361s) [noise=true , address=77.90.31.17:4772 , piece_id=CT6BNOYZGDBUEFVGM2RDS4ICMWPYS6N4IOUDA5V7MJQ3IKELNW2Q, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.668671640 (7h36m5.220142101s) [noise=true , address=145.79.227.31:4797 , piece_id=VQEEGJTLJY2IMCDHL2GMVGYEX4VL34TMMXKDYNERFYK3HRV7TC6A, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.669093554 (7h36m5.220564015s) [noise=true , address=145.79.227.21:4474 , piece_id=J5XE2MTCEW4ZHQDDZ4KCVYFK5EX6RQZARKNY4QCH4HUHGJO3SXDQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.669510097 (7h36m5.220980568s) [noise=true , address=77.90.31.18:4807 , piece_id=HDJAJYXNZNZN3LVAQSYVJEOQ3MU5H3BXCCW3NA2GLKPT7ATZHSJQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.671936887 (7h36m5.223407348s) [noise=true , address=145.79.227.20:4439 , piece_id=PPOVWSRAU7Y4CZAKAXV6ZP74EEPHONUC35O6XM57VE47NVJL5WZA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.672494549 (7h36m5.22396502s) [noise=true , address=77.90.31.21:4877 , piece_id=3V3Z6GNF32Q66EOS66WV3AGJNA7IXT5NHVAPKNUGVYU3R46IKKKA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:08.438197970 (7h36m6.989668451s) [upload=6714]: Next returned nil segment Nov 9 15:40:08.438219050 (7h36m6.989689521s) [upload=6714]: Waiting for error group managing segments... Nov 9 15:40:08.829503696 (7h36m7.380974147s) [noise=true , address=77.90.31.18:4807 , piece_id=HDJAJYXNZNZN3LVAQSYVJEOQ3MU5H3BXCCW3NA2GLKPT7ATZHSJQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.846923116 (7h36m7.398393577s) [noise=true , address=45.91.116.14:4190 , piece_id=XSIDZL5HMLZTQE7ZAPURAAUGQLPP3QOZK5BIE5PHRMZPSTN5APSA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.867005608 (7h36m7.418476069s) [noise=true , address=77.90.31.21:4877 , piece_id=3V3Z6GNF32Q66EOS66WV3AGJNA7IXT5NHVAPKNUGVYU3R46IKKKA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.888352807 (7h36m7.439823268s) [noise=true , address=77.90.31.14:4662 , piece_id=HHSJDZPYKGCHURAZV46OQSGBU7D4TFPXREDBFPGTYF7OAUMT2X3A, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.904680649 (7h36m7.45615111s) [noise=true , address=77.90.31.17:4772 , piece_id=CT6BNOYZGDBUEFVGM2RDS4ICMWPYS6N4IOUDA5V7MJQ3IKELNW2Q, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:18.663082162 (7h36m17.214552623s) [noise=true , address=145.79.227.17:4971 , piece_id=VTUI377UCF44ICILORHEKUO33ICRXM2FECO5EXS7RUDH4VGV7FJA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:20.077194714 (7h36m18.628665185s) [noise=true , address=145.79.227.13:4803 , piece_id=V37DZKBC4CHRGISXEQHTQAGKZFNXQKWXSWBSMHNRQTSJ32WYI2OQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:28.517900114 (7h36m27.069370575s) [noise=true , address=145.79.227.32:4835 , piece_id=EYIBNSVPQPKLKOBHKZI2VZHVCDKW3C3YJSFEWP4G62UTBXX2YRZA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:28.567571990 (7h36m27.119042451s) [noise=true , address=145.79.227.31:4797 , piece_id=VQEEGJTLJY2IMCDHL2GMVGYEX4VL34TMMXKDYNERFYK3HRV7TC6A, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:33.651382593 (7h36m32.202853044s) [noise=true , address=145.79.227.20:4439 , piece_id=PPOVWSRAU7Y4CZAKAXV6ZP74EEPHONUC35O6XM57VE47NVJL5WZA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:34.842221635 (7h36m33.393692096s) [noise=true , address=145.79.227.9:4365 , piece_id=7BK6E45EY6Y3SV6PY3A4EVCS42PC4HYLXU3J7MN4HXRLDEFUEFCA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:35.601406517 (7h36m34.152877008s) [noise=true , address=145.79.227.25:4619 , piece_id=23D3MCZRVNPI5KB43ROAHKUALDGXLZBZU5E6EIU6U2FY3YQOVPGA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:35.697082303 (7h36m34.248552774s) [noise=true , address=145.79.227.17:4970 , piece_id=5ZP5DDC3XAYQ5UFYJVQV6IZHB5KQQGNG54MTYW3UUYPN2IAK5QVQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697252518 (7h36m34.248722969s) [noise=true , address=145.79.227.17:4972 , piece_id=AI3KVNUJKJUD33YVU2Z472R72SKQKRYQG65JPKFZ5WBJLKRLW6MA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697302322 (7h36m34.248772773s) [noise=true , address=145.79.227.21:4474 , piece_id=J5XE2MTCEW4ZHQDDZ4KCVYFK5EX6RQZARKNY4QCH4HUHGJO3SXDQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, address:145.79.227.21:4474): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697472546 (7h36m34.248943007s) [noise=true , address=145.79.227.9:4366 , piece_id=BT3GWQSXIMMSFPUYJC5EQ2LX2DS4BW5DAMW6ILLNVVRUUHZMOQUA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697514686 (7h36m34.248985147s) [seg_pos={0 0}, upload=6714]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 9 15:40:35.950211122 (7h36m34.501681583s) [upload=6714]: Done uploading segments... Nov 9 15:40:35.950416643 (7h36m34.501887104s) [upload=6714]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 9.069723637s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [......X........] duration: 65.162413ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/3ea6c69616edd41 FAIL (2025-11-09 15:47:41 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:59888->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 65.187229ms core website
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [.X.............] duration: 327.355039ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/47f4a3db63391ab8 FAIL (2025-11-09 15:47:32 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:54114->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 327.385919ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 9.629021124s trace: https://jaeger.dev.storj.io/trace/3846516a5cecd778 FAIL (2025-11-09 15:46:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 9 15:40:06.410809146 (7h36m4.962279607s) [upload=6714]: Starting upload Nov 9 15:40:06.411046298 (7h36m4.962516769s) [upload=6714]: Done starting upload Nov 9 15:40:06.411314729 (7h36m4.96278519s) [upload=6714]: Uploading segments... Nov 9 15:40:06.411410882 (7h36m4.962881383s) [upload=6714]: Got next segment. Inline: false Nov 9 15:40:06.411464574 (7h36m4.962935035s) [upload=6714]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 9 15:40:06.411475064 (7h36m4.962945555s) [upload=6714]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 9 15:40:06.664789367 (7h36m5.216259828s) [seg_pos={0 0}, upload=6714]: Begin upload segment... Nov 9 15:40:06.664851705 (7h36m5.216322166s) [seg_pos={0 0}, upload=6714]: Done begin upload segment. Nov 9 15:40:06.664891662 (7h36m5.216362123s) [noise=true , address=145.79.227.32:4835 , piece_id=EYIBNSVPQPKLKOBHKZI2VZHVCDKW3C3YJSFEWP4G62UTBXX2YRZA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.664996933 (7h36m5.216467394s) [noise=true , address=45.91.116.14:4190 , piece_id=XSIDZL5HMLZTQE7ZAPURAAUGQLPP3QOZK5BIE5PHRMZPSTN5APSA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.665698789 (7h36m5.21716925s) [noise=true , address=77.90.31.14:4662 , piece_id=HHSJDZPYKGCHURAZV46OQSGBU7D4TFPXREDBFPGTYF7OAUMT2X3A, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.666456111 (7h36m5.217926572s) [noise=true , address=145.79.227.25:4619 , piece_id=23D3MCZRVNPI5KB43ROAHKUALDGXLZBZU5E6EIU6U2FY3YQOVPGA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.666642225 (7h36m5.218112676s) [noise=true , address=145.79.227.17:4971 , piece_id=VTUI377UCF44ICILORHEKUO33ICRXM2FECO5EXS7RUDH4VGV7FJA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667027539 (7h36m5.218498s) [noise=true , address=145.79.227.9:4366 , piece_id=BT3GWQSXIMMSFPUYJC5EQ2LX2DS4BW5DAMW6ILLNVVRUUHZMOQUA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667041675 (7h36m5.218512126s) [noise=true , address=145.79.227.17:4972 , piece_id=AI3KVNUJKJUD33YVU2Z472R72SKQKRYQG65JPKFZ5WBJLKRLW6MA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667502794 (7h36m5.218973255s) [noise=true , address=145.79.227.13:4803 , piece_id=V37DZKBC4CHRGISXEQHTQAGKZFNXQKWXSWBSMHNRQTSJ32WYI2OQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667805441 (7h36m5.219275902s) [noise=true , address=145.79.227.9:4365 , piece_id=7BK6E45EY6Y3SV6PY3A4EVCS42PC4HYLXU3J7MN4HXRLDEFUEFCA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667895652 (7h36m5.219366113s) [noise=true , address=145.79.227.17:4970 , piece_id=5ZP5DDC3XAYQ5UFYJVQV6IZHB5KQQGNG54MTYW3UUYPN2IAK5QVQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.668240909 (7h36m5.219711361s) [noise=true , address=77.90.31.17:4772 , piece_id=CT6BNOYZGDBUEFVGM2RDS4ICMWPYS6N4IOUDA5V7MJQ3IKELNW2Q, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.668671640 (7h36m5.220142101s) [noise=true , address=145.79.227.31:4797 , piece_id=VQEEGJTLJY2IMCDHL2GMVGYEX4VL34TMMXKDYNERFYK3HRV7TC6A, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.669093554 (7h36m5.220564015s) [noise=true , address=145.79.227.21:4474 , piece_id=J5XE2MTCEW4ZHQDDZ4KCVYFK5EX6RQZARKNY4QCH4HUHGJO3SXDQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.669510097 (7h36m5.220980568s) [noise=true , address=77.90.31.18:4807 , piece_id=HDJAJYXNZNZN3LVAQSYVJEOQ3MU5H3BXCCW3NA2GLKPT7ATZHSJQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.671936887 (7h36m5.223407348s) [noise=true , address=145.79.227.20:4439 , piece_id=PPOVWSRAU7Y4CZAKAXV6ZP74EEPHONUC35O6XM57VE47NVJL5WZA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.672494549 (7h36m5.22396502s) [noise=true , address=77.90.31.21:4877 , piece_id=3V3Z6GNF32Q66EOS66WV3AGJNA7IXT5NHVAPKNUGVYU3R46IKKKA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:08.438197970 (7h36m6.989668451s) [upload=6714]: Next returned nil segment Nov 9 15:40:08.438219050 (7h36m6.989689521s) [upload=6714]: Waiting for error group managing segments... Nov 9 15:40:08.829503696 (7h36m7.380974147s) [noise=true , address=77.90.31.18:4807 , piece_id=HDJAJYXNZNZN3LVAQSYVJEOQ3MU5H3BXCCW3NA2GLKPT7ATZHSJQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.846923116 (7h36m7.398393577s) [noise=true , address=45.91.116.14:4190 , piece_id=XSIDZL5HMLZTQE7ZAPURAAUGQLPP3QOZK5BIE5PHRMZPSTN5APSA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.867005608 (7h36m7.418476069s) [noise=true , address=77.90.31.21:4877 , piece_id=3V3Z6GNF32Q66EOS66WV3AGJNA7IXT5NHVAPKNUGVYU3R46IKKKA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.888352807 (7h36m7.439823268s) [noise=true , address=77.90.31.14:4662 , piece_id=HHSJDZPYKGCHURAZV46OQSGBU7D4TFPXREDBFPGTYF7OAUMT2X3A, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.904680649 (7h36m7.45615111s) [noise=true , address=77.90.31.17:4772 , piece_id=CT6BNOYZGDBUEFVGM2RDS4ICMWPYS6N4IOUDA5V7MJQ3IKELNW2Q, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:18.663082162 (7h36m17.214552623s) [noise=true , address=145.79.227.17:4971 , piece_id=VTUI377UCF44ICILORHEKUO33ICRXM2FECO5EXS7RUDH4VGV7FJA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:20.077194714 (7h36m18.628665185s) [noise=true , address=145.79.227.13:4803 , piece_id=V37DZKBC4CHRGISXEQHTQAGKZFNXQKWXSWBSMHNRQTSJ32WYI2OQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:28.517900114 (7h36m27.069370575s) [noise=true , address=145.79.227.32:4835 , piece_id=EYIBNSVPQPKLKOBHKZI2VZHVCDKW3C3YJSFEWP4G62UTBXX2YRZA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:28.567571990 (7h36m27.119042451s) [noise=true , address=145.79.227.31:4797 , piece_id=VQEEGJTLJY2IMCDHL2GMVGYEX4VL34TMMXKDYNERFYK3HRV7TC6A, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:33.651382593 (7h36m32.202853044s) [noise=true , address=145.79.227.20:4439 , piece_id=PPOVWSRAU7Y4CZAKAXV6ZP74EEPHONUC35O6XM57VE47NVJL5WZA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:34.842221635 (7h36m33.393692096s) [noise=true , address=145.79.227.9:4365 , piece_id=7BK6E45EY6Y3SV6PY3A4EVCS42PC4HYLXU3J7MN4HXRLDEFUEFCA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:35.601406517 (7h36m34.152877008s) [noise=true , address=145.79.227.25:4619 , piece_id=23D3MCZRVNPI5KB43ROAHKUALDGXLZBZU5E6EIU6U2FY3YQOVPGA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:35.697082303 (7h36m34.248552774s) [noise=true , address=145.79.227.17:4970 , piece_id=5ZP5DDC3XAYQ5UFYJVQV6IZHB5KQQGNG54MTYW3UUYPN2IAK5QVQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697252518 (7h36m34.248722969s) [noise=true , address=145.79.227.17:4972 , piece_id=AI3KVNUJKJUD33YVU2Z472R72SKQKRYQG65JPKFZ5WBJLKRLW6MA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697302322 (7h36m34.248772773s) [noise=true , address=145.79.227.21:4474 , piece_id=J5XE2MTCEW4ZHQDDZ4KCVYFK5EX6RQZARKNY4QCH4HUHGJO3SXDQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, address:145.79.227.21:4474): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697472546 (7h36m34.248943007s) [noise=true , address=145.79.227.9:4366 , piece_id=BT3GWQSXIMMSFPUYJC5EQ2LX2DS4BW5DAMW6ILLNVVRUUHZMOQUA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697514686 (7h36m34.248985147s) [seg_pos={0 0}, upload=6714]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 9 15:40:35.950211122 (7h36m34.501681583s) [upload=6714]: Done uploading segments... Nov 9 15:40:35.950416643 (7h36m34.501887104s) [upload=6714]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 9.629048708s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [.......X.......] duration: 61.2697ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/3ea6c69616edd41 FAIL (2025-11-09 15:46:41 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:59888->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 61.298666ms core website
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [..X............] duration: 338.343144ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/47f4a3db63391ab8 FAIL (2025-11-09 15:46:32 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:54114->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 338.375224ms core website
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [...X...........] duration: 311.951659ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/47f4a3db63391ab8 FAIL (2025-11-09 15:46:06 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:54114->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 312.429189ms core website
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [........X......] duration: 63.498284ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/3ea6c69616edd41 FAIL (2025-11-09 15:45:53 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:59888->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 63.530665ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 9.046042829s trace: https://jaeger.dev.storj.io/trace/3846516a5cecd778 FAIL (2025-11-09 15:45:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 9 15:40:06.410809146 (7h36m4.962279607s) [upload=6714]: Starting upload Nov 9 15:40:06.411046298 (7h36m4.962516769s) [upload=6714]: Done starting upload Nov 9 15:40:06.411314729 (7h36m4.96278519s) [upload=6714]: Uploading segments... Nov 9 15:40:06.411410882 (7h36m4.962881383s) [upload=6714]: Got next segment. Inline: false Nov 9 15:40:06.411464574 (7h36m4.962935035s) [upload=6714]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 9 15:40:06.411475064 (7h36m4.962945555s) [upload=6714]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 9 15:40:06.664789367 (7h36m5.216259828s) [seg_pos={0 0}, upload=6714]: Begin upload segment... Nov 9 15:40:06.664851705 (7h36m5.216322166s) [seg_pos={0 0}, upload=6714]: Done begin upload segment. Nov 9 15:40:06.664891662 (7h36m5.216362123s) [noise=true , address=145.79.227.32:4835 , piece_id=EYIBNSVPQPKLKOBHKZI2VZHVCDKW3C3YJSFEWP4G62UTBXX2YRZA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.664996933 (7h36m5.216467394s) [noise=true , address=45.91.116.14:4190 , piece_id=XSIDZL5HMLZTQE7ZAPURAAUGQLPP3QOZK5BIE5PHRMZPSTN5APSA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.665698789 (7h36m5.21716925s) [noise=true , address=77.90.31.14:4662 , piece_id=HHSJDZPYKGCHURAZV46OQSGBU7D4TFPXREDBFPGTYF7OAUMT2X3A, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.666456111 (7h36m5.217926572s) [noise=true , address=145.79.227.25:4619 , piece_id=23D3MCZRVNPI5KB43ROAHKUALDGXLZBZU5E6EIU6U2FY3YQOVPGA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.666642225 (7h36m5.218112676s) [noise=true , address=145.79.227.17:4971 , piece_id=VTUI377UCF44ICILORHEKUO33ICRXM2FECO5EXS7RUDH4VGV7FJA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667027539 (7h36m5.218498s) [noise=true , address=145.79.227.9:4366 , piece_id=BT3GWQSXIMMSFPUYJC5EQ2LX2DS4BW5DAMW6ILLNVVRUUHZMOQUA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667041675 (7h36m5.218512126s) [noise=true , address=145.79.227.17:4972 , piece_id=AI3KVNUJKJUD33YVU2Z472R72SKQKRYQG65JPKFZ5WBJLKRLW6MA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667502794 (7h36m5.218973255s) [noise=true , address=145.79.227.13:4803 , piece_id=V37DZKBC4CHRGISXEQHTQAGKZFNXQKWXSWBSMHNRQTSJ32WYI2OQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667805441 (7h36m5.219275902s) [noise=true , address=145.79.227.9:4365 , piece_id=7BK6E45EY6Y3SV6PY3A4EVCS42PC4HYLXU3J7MN4HXRLDEFUEFCA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667895652 (7h36m5.219366113s) [noise=true , address=145.79.227.17:4970 , piece_id=5ZP5DDC3XAYQ5UFYJVQV6IZHB5KQQGNG54MTYW3UUYPN2IAK5QVQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.668240909 (7h36m5.219711361s) [noise=true , address=77.90.31.17:4772 , piece_id=CT6BNOYZGDBUEFVGM2RDS4ICMWPYS6N4IOUDA5V7MJQ3IKELNW2Q, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.668671640 (7h36m5.220142101s) [noise=true , address=145.79.227.31:4797 , piece_id=VQEEGJTLJY2IMCDHL2GMVGYEX4VL34TMMXKDYNERFYK3HRV7TC6A, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.669093554 (7h36m5.220564015s) [noise=true , address=145.79.227.21:4474 , piece_id=J5XE2MTCEW4ZHQDDZ4KCVYFK5EX6RQZARKNY4QCH4HUHGJO3SXDQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.669510097 (7h36m5.220980568s) [noise=true , address=77.90.31.18:4807 , piece_id=HDJAJYXNZNZN3LVAQSYVJEOQ3MU5H3BXCCW3NA2GLKPT7ATZHSJQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.671936887 (7h36m5.223407348s) [noise=true , address=145.79.227.20:4439 , piece_id=PPOVWSRAU7Y4CZAKAXV6ZP74EEPHONUC35O6XM57VE47NVJL5WZA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.672494549 (7h36m5.22396502s) [noise=true , address=77.90.31.21:4877 , piece_id=3V3Z6GNF32Q66EOS66WV3AGJNA7IXT5NHVAPKNUGVYU3R46IKKKA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:08.438197970 (7h36m6.989668451s) [upload=6714]: Next returned nil segment Nov 9 15:40:08.438219050 (7h36m6.989689521s) [upload=6714]: Waiting for error group managing segments... Nov 9 15:40:08.829503696 (7h36m7.380974147s) [noise=true , address=77.90.31.18:4807 , piece_id=HDJAJYXNZNZN3LVAQSYVJEOQ3MU5H3BXCCW3NA2GLKPT7ATZHSJQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.846923116 (7h36m7.398393577s) [noise=true , address=45.91.116.14:4190 , piece_id=XSIDZL5HMLZTQE7ZAPURAAUGQLPP3QOZK5BIE5PHRMZPSTN5APSA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.867005608 (7h36m7.418476069s) [noise=true , address=77.90.31.21:4877 , piece_id=3V3Z6GNF32Q66EOS66WV3AGJNA7IXT5NHVAPKNUGVYU3R46IKKKA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.888352807 (7h36m7.439823268s) [noise=true , address=77.90.31.14:4662 , piece_id=HHSJDZPYKGCHURAZV46OQSGBU7D4TFPXREDBFPGTYF7OAUMT2X3A, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.904680649 (7h36m7.45615111s) [noise=true , address=77.90.31.17:4772 , piece_id=CT6BNOYZGDBUEFVGM2RDS4ICMWPYS6N4IOUDA5V7MJQ3IKELNW2Q, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:18.663082162 (7h36m17.214552623s) [noise=true , address=145.79.227.17:4971 , piece_id=VTUI377UCF44ICILORHEKUO33ICRXM2FECO5EXS7RUDH4VGV7FJA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:20.077194714 (7h36m18.628665185s) [noise=true , address=145.79.227.13:4803 , piece_id=V37DZKBC4CHRGISXEQHTQAGKZFNXQKWXSWBSMHNRQTSJ32WYI2OQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:28.517900114 (7h36m27.069370575s) [noise=true , address=145.79.227.32:4835 , piece_id=EYIBNSVPQPKLKOBHKZI2VZHVCDKW3C3YJSFEWP4G62UTBXX2YRZA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:28.567571990 (7h36m27.119042451s) [noise=true , address=145.79.227.31:4797 , piece_id=VQEEGJTLJY2IMCDHL2GMVGYEX4VL34TMMXKDYNERFYK3HRV7TC6A, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:33.651382593 (7h36m32.202853044s) [noise=true , address=145.79.227.20:4439 , piece_id=PPOVWSRAU7Y4CZAKAXV6ZP74EEPHONUC35O6XM57VE47NVJL5WZA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:34.842221635 (7h36m33.393692096s) [noise=true , address=145.79.227.9:4365 , piece_id=7BK6E45EY6Y3SV6PY3A4EVCS42PC4HYLXU3J7MN4HXRLDEFUEFCA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:35.601406517 (7h36m34.152877008s) [noise=true , address=145.79.227.25:4619 , piece_id=23D3MCZRVNPI5KB43ROAHKUALDGXLZBZU5E6EIU6U2FY3YQOVPGA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:35.697082303 (7h36m34.248552774s) [noise=true , address=145.79.227.17:4970 , piece_id=5ZP5DDC3XAYQ5UFYJVQV6IZHB5KQQGNG54MTYW3UUYPN2IAK5QVQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697252518 (7h36m34.248722969s) [noise=true , address=145.79.227.17:4972 , piece_id=AI3KVNUJKJUD33YVU2Z472R72SKQKRYQG65JPKFZ5WBJLKRLW6MA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697302322 (7h36m34.248772773s) [noise=true , address=145.79.227.21:4474 , piece_id=J5XE2MTCEW4ZHQDDZ4KCVYFK5EX6RQZARKNY4QCH4HUHGJO3SXDQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, address:145.79.227.21:4474): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697472546 (7h36m34.248943007s) [noise=true , address=145.79.227.9:4366 , piece_id=BT3GWQSXIMMSFPUYJC5EQ2LX2DS4BW5DAMW6ILLNVVRUUHZMOQUA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697514686 (7h36m34.248985147s) [seg_pos={0 0}, upload=6714]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 9 15:40:35.950211122 (7h36m34.501681583s) [upload=6714]: Done uploading segments... Nov 9 15:40:35.950416643 (7h36m34.501887104s) [upload=6714]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 9.046073006s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [.........X.....] duration: 76.712907ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/3ea6c69616edd41 FAIL (2025-11-09 15:45:07 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:59888->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 76.748274ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 10.303491663s trace: https://jaeger.dev.storj.io/trace/3846516a5cecd778 FAIL (2025-11-09 15:44:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 9 15:40:06.410809146 (7h36m4.962279607s) [upload=6714]: Starting upload Nov 9 15:40:06.411046298 (7h36m4.962516769s) [upload=6714]: Done starting upload Nov 9 15:40:06.411314729 (7h36m4.96278519s) [upload=6714]: Uploading segments... Nov 9 15:40:06.411410882 (7h36m4.962881383s) [upload=6714]: Got next segment. Inline: false Nov 9 15:40:06.411464574 (7h36m4.962935035s) [upload=6714]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 9 15:40:06.411475064 (7h36m4.962945555s) [upload=6714]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 9 15:40:06.664789367 (7h36m5.216259828s) [seg_pos={0 0}, upload=6714]: Begin upload segment... Nov 9 15:40:06.664851705 (7h36m5.216322166s) [seg_pos={0 0}, upload=6714]: Done begin upload segment. Nov 9 15:40:06.664891662 (7h36m5.216362123s) [noise=true , address=145.79.227.32:4835 , piece_id=EYIBNSVPQPKLKOBHKZI2VZHVCDKW3C3YJSFEWP4G62UTBXX2YRZA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.664996933 (7h36m5.216467394s) [noise=true , address=45.91.116.14:4190 , piece_id=XSIDZL5HMLZTQE7ZAPURAAUGQLPP3QOZK5BIE5PHRMZPSTN5APSA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.665698789 (7h36m5.21716925s) [noise=true , address=77.90.31.14:4662 , piece_id=HHSJDZPYKGCHURAZV46OQSGBU7D4TFPXREDBFPGTYF7OAUMT2X3A, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.666456111 (7h36m5.217926572s) [noise=true , address=145.79.227.25:4619 , piece_id=23D3MCZRVNPI5KB43ROAHKUALDGXLZBZU5E6EIU6U2FY3YQOVPGA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.666642225 (7h36m5.218112676s) [noise=true , address=145.79.227.17:4971 , piece_id=VTUI377UCF44ICILORHEKUO33ICRXM2FECO5EXS7RUDH4VGV7FJA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667027539 (7h36m5.218498s) [noise=true , address=145.79.227.9:4366 , piece_id=BT3GWQSXIMMSFPUYJC5EQ2LX2DS4BW5DAMW6ILLNVVRUUHZMOQUA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667041675 (7h36m5.218512126s) [noise=true , address=145.79.227.17:4972 , piece_id=AI3KVNUJKJUD33YVU2Z472R72SKQKRYQG65JPKFZ5WBJLKRLW6MA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667502794 (7h36m5.218973255s) [noise=true , address=145.79.227.13:4803 , piece_id=V37DZKBC4CHRGISXEQHTQAGKZFNXQKWXSWBSMHNRQTSJ32WYI2OQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667805441 (7h36m5.219275902s) [noise=true , address=145.79.227.9:4365 , piece_id=7BK6E45EY6Y3SV6PY3A4EVCS42PC4HYLXU3J7MN4HXRLDEFUEFCA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667895652 (7h36m5.219366113s) [noise=true , address=145.79.227.17:4970 , piece_id=5ZP5DDC3XAYQ5UFYJVQV6IZHB5KQQGNG54MTYW3UUYPN2IAK5QVQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.668240909 (7h36m5.219711361s) [noise=true , address=77.90.31.17:4772 , piece_id=CT6BNOYZGDBUEFVGM2RDS4ICMWPYS6N4IOUDA5V7MJQ3IKELNW2Q, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.668671640 (7h36m5.220142101s) [noise=true , address=145.79.227.31:4797 , piece_id=VQEEGJTLJY2IMCDHL2GMVGYEX4VL34TMMXKDYNERFYK3HRV7TC6A, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.669093554 (7h36m5.220564015s) [noise=true , address=145.79.227.21:4474 , piece_id=J5XE2MTCEW4ZHQDDZ4KCVYFK5EX6RQZARKNY4QCH4HUHGJO3SXDQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.669510097 (7h36m5.220980568s) [noise=true , address=77.90.31.18:4807 , piece_id=HDJAJYXNZNZN3LVAQSYVJEOQ3MU5H3BXCCW3NA2GLKPT7ATZHSJQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.671936887 (7h36m5.223407348s) [noise=true , address=145.79.227.20:4439 , piece_id=PPOVWSRAU7Y4CZAKAXV6ZP74EEPHONUC35O6XM57VE47NVJL5WZA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.672494549 (7h36m5.22396502s) [noise=true , address=77.90.31.21:4877 , piece_id=3V3Z6GNF32Q66EOS66WV3AGJNA7IXT5NHVAPKNUGVYU3R46IKKKA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:08.438197970 (7h36m6.989668451s) [upload=6714]: Next returned nil segment Nov 9 15:40:08.438219050 (7h36m6.989689521s) [upload=6714]: Waiting for error group managing segments... Nov 9 15:40:08.829503696 (7h36m7.380974147s) [noise=true , address=77.90.31.18:4807 , piece_id=HDJAJYXNZNZN3LVAQSYVJEOQ3MU5H3BXCCW3NA2GLKPT7ATZHSJQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.846923116 (7h36m7.398393577s) [noise=true , address=45.91.116.14:4190 , piece_id=XSIDZL5HMLZTQE7ZAPURAAUGQLPP3QOZK5BIE5PHRMZPSTN5APSA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.867005608 (7h36m7.418476069s) [noise=true , address=77.90.31.21:4877 , piece_id=3V3Z6GNF32Q66EOS66WV3AGJNA7IXT5NHVAPKNUGVYU3R46IKKKA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.888352807 (7h36m7.439823268s) [noise=true , address=77.90.31.14:4662 , piece_id=HHSJDZPYKGCHURAZV46OQSGBU7D4TFPXREDBFPGTYF7OAUMT2X3A, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.904680649 (7h36m7.45615111s) [noise=true , address=77.90.31.17:4772 , piece_id=CT6BNOYZGDBUEFVGM2RDS4ICMWPYS6N4IOUDA5V7MJQ3IKELNW2Q, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:18.663082162 (7h36m17.214552623s) [noise=true , address=145.79.227.17:4971 , piece_id=VTUI377UCF44ICILORHEKUO33ICRXM2FECO5EXS7RUDH4VGV7FJA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:20.077194714 (7h36m18.628665185s) [noise=true , address=145.79.227.13:4803 , piece_id=V37DZKBC4CHRGISXEQHTQAGKZFNXQKWXSWBSMHNRQTSJ32WYI2OQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:28.517900114 (7h36m27.069370575s) [noise=true , address=145.79.227.32:4835 , piece_id=EYIBNSVPQPKLKOBHKZI2VZHVCDKW3C3YJSFEWP4G62UTBXX2YRZA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:28.567571990 (7h36m27.119042451s) [noise=true , address=145.79.227.31:4797 , piece_id=VQEEGJTLJY2IMCDHL2GMVGYEX4VL34TMMXKDYNERFYK3HRV7TC6A, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:33.651382593 (7h36m32.202853044s) [noise=true , address=145.79.227.20:4439 , piece_id=PPOVWSRAU7Y4CZAKAXV6ZP74EEPHONUC35O6XM57VE47NVJL5WZA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:34.842221635 (7h36m33.393692096s) [noise=true , address=145.79.227.9:4365 , piece_id=7BK6E45EY6Y3SV6PY3A4EVCS42PC4HYLXU3J7MN4HXRLDEFUEFCA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:35.601406517 (7h36m34.152877008s) [noise=true , address=145.79.227.25:4619 , piece_id=23D3MCZRVNPI5KB43ROAHKUALDGXLZBZU5E6EIU6U2FY3YQOVPGA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:35.697082303 (7h36m34.248552774s) [noise=true , address=145.79.227.17:4970 , piece_id=5ZP5DDC3XAYQ5UFYJVQV6IZHB5KQQGNG54MTYW3UUYPN2IAK5QVQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697252518 (7h36m34.248722969s) [noise=true , address=145.79.227.17:4972 , piece_id=AI3KVNUJKJUD33YVU2Z472R72SKQKRYQG65JPKFZ5WBJLKRLW6MA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697302322 (7h36m34.248772773s) [noise=true , address=145.79.227.21:4474 , piece_id=J5XE2MTCEW4ZHQDDZ4KCVYFK5EX6RQZARKNY4QCH4HUHGJO3SXDQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, address:145.79.227.21:4474): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697472546 (7h36m34.248943007s) [noise=true , address=145.79.227.9:4366 , piece_id=BT3GWQSXIMMSFPUYJC5EQ2LX2DS4BW5DAMW6ILLNVVRUUHZMOQUA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697514686 (7h36m34.248985147s) [seg_pos={0 0}, upload=6714]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 9 15:40:35.950211122 (7h36m34.501681583s) [upload=6714]: Done uploading segments... Nov 9 15:40:35.950416643 (7h36m34.501887104s) [upload=6714]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 10.303549122s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [....X..........] duration: 311.399827ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/47f4a3db63391ab8 FAIL (2025-11-09 15:44:53 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:54114->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 311.436589ms core website
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [.....X.........] duration: 333.221232ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/47f4a3db63391ab8 FAIL (2025-11-09 15:44:35 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:54114->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 333.258603ms core website
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [..........X....] duration: 56.082753ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/3ea6c69616edd41 FAIL (2025-11-09 15:44:21 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:59888->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 56.113653ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 12.364741805s trace: https://jaeger.dev.storj.io/trace/3846516a5cecd778 FAIL (2025-11-09 15:44:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 9 15:40:06.410809146 (7h36m4.962279607s) [upload=6714]: Starting upload Nov 9 15:40:06.411046298 (7h36m4.962516769s) [upload=6714]: Done starting upload Nov 9 15:40:06.411314729 (7h36m4.96278519s) [upload=6714]: Uploading segments... Nov 9 15:40:06.411410882 (7h36m4.962881383s) [upload=6714]: Got next segment. Inline: false Nov 9 15:40:06.411464574 (7h36m4.962935035s) [upload=6714]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 9 15:40:06.411475064 (7h36m4.962945555s) [upload=6714]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 9 15:40:06.664789367 (7h36m5.216259828s) [seg_pos={0 0}, upload=6714]: Begin upload segment... Nov 9 15:40:06.664851705 (7h36m5.216322166s) [seg_pos={0 0}, upload=6714]: Done begin upload segment. Nov 9 15:40:06.664891662 (7h36m5.216362123s) [noise=true , address=145.79.227.32:4835 , piece_id=EYIBNSVPQPKLKOBHKZI2VZHVCDKW3C3YJSFEWP4G62UTBXX2YRZA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.664996933 (7h36m5.216467394s) [noise=true , address=45.91.116.14:4190 , piece_id=XSIDZL5HMLZTQE7ZAPURAAUGQLPP3QOZK5BIE5PHRMZPSTN5APSA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.665698789 (7h36m5.21716925s) [noise=true , address=77.90.31.14:4662 , piece_id=HHSJDZPYKGCHURAZV46OQSGBU7D4TFPXREDBFPGTYF7OAUMT2X3A, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.666456111 (7h36m5.217926572s) [noise=true , address=145.79.227.25:4619 , piece_id=23D3MCZRVNPI5KB43ROAHKUALDGXLZBZU5E6EIU6U2FY3YQOVPGA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.666642225 (7h36m5.218112676s) [noise=true , address=145.79.227.17:4971 , piece_id=VTUI377UCF44ICILORHEKUO33ICRXM2FECO5EXS7RUDH4VGV7FJA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667027539 (7h36m5.218498s) [noise=true , address=145.79.227.9:4366 , piece_id=BT3GWQSXIMMSFPUYJC5EQ2LX2DS4BW5DAMW6ILLNVVRUUHZMOQUA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667041675 (7h36m5.218512126s) [noise=true , address=145.79.227.17:4972 , piece_id=AI3KVNUJKJUD33YVU2Z472R72SKQKRYQG65JPKFZ5WBJLKRLW6MA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667502794 (7h36m5.218973255s) [noise=true , address=145.79.227.13:4803 , piece_id=V37DZKBC4CHRGISXEQHTQAGKZFNXQKWXSWBSMHNRQTSJ32WYI2OQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667805441 (7h36m5.219275902s) [noise=true , address=145.79.227.9:4365 , piece_id=7BK6E45EY6Y3SV6PY3A4EVCS42PC4HYLXU3J7MN4HXRLDEFUEFCA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667895652 (7h36m5.219366113s) [noise=true , address=145.79.227.17:4970 , piece_id=5ZP5DDC3XAYQ5UFYJVQV6IZHB5KQQGNG54MTYW3UUYPN2IAK5QVQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.668240909 (7h36m5.219711361s) [noise=true , address=77.90.31.17:4772 , piece_id=CT6BNOYZGDBUEFVGM2RDS4ICMWPYS6N4IOUDA5V7MJQ3IKELNW2Q, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.668671640 (7h36m5.220142101s) [noise=true , address=145.79.227.31:4797 , piece_id=VQEEGJTLJY2IMCDHL2GMVGYEX4VL34TMMXKDYNERFYK3HRV7TC6A, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.669093554 (7h36m5.220564015s) [noise=true , address=145.79.227.21:4474 , piece_id=J5XE2MTCEW4ZHQDDZ4KCVYFK5EX6RQZARKNY4QCH4HUHGJO3SXDQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.669510097 (7h36m5.220980568s) [noise=true , address=77.90.31.18:4807 , piece_id=HDJAJYXNZNZN3LVAQSYVJEOQ3MU5H3BXCCW3NA2GLKPT7ATZHSJQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.671936887 (7h36m5.223407348s) [noise=true , address=145.79.227.20:4439 , piece_id=PPOVWSRAU7Y4CZAKAXV6ZP74EEPHONUC35O6XM57VE47NVJL5WZA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.672494549 (7h36m5.22396502s) [noise=true , address=77.90.31.21:4877 , piece_id=3V3Z6GNF32Q66EOS66WV3AGJNA7IXT5NHVAPKNUGVYU3R46IKKKA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:08.438197970 (7h36m6.989668451s) [upload=6714]: Next returned nil segment Nov 9 15:40:08.438219050 (7h36m6.989689521s) [upload=6714]: Waiting for error group managing segments... Nov 9 15:40:08.829503696 (7h36m7.380974147s) [noise=true , address=77.90.31.18:4807 , piece_id=HDJAJYXNZNZN3LVAQSYVJEOQ3MU5H3BXCCW3NA2GLKPT7ATZHSJQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.846923116 (7h36m7.398393577s) [noise=true , address=45.91.116.14:4190 , piece_id=XSIDZL5HMLZTQE7ZAPURAAUGQLPP3QOZK5BIE5PHRMZPSTN5APSA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.867005608 (7h36m7.418476069s) [noise=true , address=77.90.31.21:4877 , piece_id=3V3Z6GNF32Q66EOS66WV3AGJNA7IXT5NHVAPKNUGVYU3R46IKKKA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.888352807 (7h36m7.439823268s) [noise=true , address=77.90.31.14:4662 , piece_id=HHSJDZPYKGCHURAZV46OQSGBU7D4TFPXREDBFPGTYF7OAUMT2X3A, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.904680649 (7h36m7.45615111s) [noise=true , address=77.90.31.17:4772 , piece_id=CT6BNOYZGDBUEFVGM2RDS4ICMWPYS6N4IOUDA5V7MJQ3IKELNW2Q, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:18.663082162 (7h36m17.214552623s) [noise=true , address=145.79.227.17:4971 , piece_id=VTUI377UCF44ICILORHEKUO33ICRXM2FECO5EXS7RUDH4VGV7FJA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:20.077194714 (7h36m18.628665185s) [noise=true , address=145.79.227.13:4803 , piece_id=V37DZKBC4CHRGISXEQHTQAGKZFNXQKWXSWBSMHNRQTSJ32WYI2OQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:28.517900114 (7h36m27.069370575s) [noise=true , address=145.79.227.32:4835 , piece_id=EYIBNSVPQPKLKOBHKZI2VZHVCDKW3C3YJSFEWP4G62UTBXX2YRZA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:28.567571990 (7h36m27.119042451s) [noise=true , address=145.79.227.31:4797 , piece_id=VQEEGJTLJY2IMCDHL2GMVGYEX4VL34TMMXKDYNERFYK3HRV7TC6A, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:33.651382593 (7h36m32.202853044s) [noise=true , address=145.79.227.20:4439 , piece_id=PPOVWSRAU7Y4CZAKAXV6ZP74EEPHONUC35O6XM57VE47NVJL5WZA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:34.842221635 (7h36m33.393692096s) [noise=true , address=145.79.227.9:4365 , piece_id=7BK6E45EY6Y3SV6PY3A4EVCS42PC4HYLXU3J7MN4HXRLDEFUEFCA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:35.601406517 (7h36m34.152877008s) [noise=true , address=145.79.227.25:4619 , piece_id=23D3MCZRVNPI5KB43ROAHKUALDGXLZBZU5E6EIU6U2FY3YQOVPGA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:35.697082303 (7h36m34.248552774s) [noise=true , address=145.79.227.17:4970 , piece_id=5ZP5DDC3XAYQ5UFYJVQV6IZHB5KQQGNG54MTYW3UUYPN2IAK5QVQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697252518 (7h36m34.248722969s) [noise=true , address=145.79.227.17:4972 , piece_id=AI3KVNUJKJUD33YVU2Z472R72SKQKRYQG65JPKFZ5WBJLKRLW6MA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697302322 (7h36m34.248772773s) [noise=true , address=145.79.227.21:4474 , piece_id=J5XE2MTCEW4ZHQDDZ4KCVYFK5EX6RQZARKNY4QCH4HUHGJO3SXDQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, address:145.79.227.21:4474): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697472546 (7h36m34.248943007s) [noise=true , address=145.79.227.9:4366 , piece_id=BT3GWQSXIMMSFPUYJC5EQ2LX2DS4BW5DAMW6ILLNVVRUUHZMOQUA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697514686 (7h36m34.248985147s) [seg_pos={0 0}, upload=6714]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 9 15:40:35.950211122 (7h36m34.501681583s) [upload=6714]: Done uploading segments... Nov 9 15:40:35.950416643 (7h36m34.501887104s) [upload=6714]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 12.364914925s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [......X........] duration: 312.185306ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/47f4a3db63391ab8 FAIL (2025-11-09 15:43:56 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:54114->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 312.2295ms core website
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [...........X...] duration: 60.022194ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/3ea6c69616edd41 FAIL (2025-11-09 15:43:45 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:59888->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 60.043985ms core website
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [.......X.......] duration: 332.083424ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/47f4a3db63391ab8 FAIL (2025-11-09 15:43:34 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:54114->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 332.124943ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 10.120406122s trace: https://jaeger.dev.storj.io/trace/3846516a5cecd778 FAIL (2025-11-09 15:43:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 9 15:40:06.410809146 (7h36m4.962279607s) [upload=6714]: Starting upload Nov 9 15:40:06.411046298 (7h36m4.962516769s) [upload=6714]: Done starting upload Nov 9 15:40:06.411314729 (7h36m4.96278519s) [upload=6714]: Uploading segments... Nov 9 15:40:06.411410882 (7h36m4.962881383s) [upload=6714]: Got next segment. Inline: false Nov 9 15:40:06.411464574 (7h36m4.962935035s) [upload=6714]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 9 15:40:06.411475064 (7h36m4.962945555s) [upload=6714]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 9 15:40:06.664789367 (7h36m5.216259828s) [seg_pos={0 0}, upload=6714]: Begin upload segment... Nov 9 15:40:06.664851705 (7h36m5.216322166s) [seg_pos={0 0}, upload=6714]: Done begin upload segment. Nov 9 15:40:06.664891662 (7h36m5.216362123s) [noise=true , address=145.79.227.32:4835 , piece_id=EYIBNSVPQPKLKOBHKZI2VZHVCDKW3C3YJSFEWP4G62UTBXX2YRZA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.664996933 (7h36m5.216467394s) [noise=true , address=45.91.116.14:4190 , piece_id=XSIDZL5HMLZTQE7ZAPURAAUGQLPP3QOZK5BIE5PHRMZPSTN5APSA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.665698789 (7h36m5.21716925s) [noise=true , address=77.90.31.14:4662 , piece_id=HHSJDZPYKGCHURAZV46OQSGBU7D4TFPXREDBFPGTYF7OAUMT2X3A, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.666456111 (7h36m5.217926572s) [noise=true , address=145.79.227.25:4619 , piece_id=23D3MCZRVNPI5KB43ROAHKUALDGXLZBZU5E6EIU6U2FY3YQOVPGA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.666642225 (7h36m5.218112676s) [noise=true , address=145.79.227.17:4971 , piece_id=VTUI377UCF44ICILORHEKUO33ICRXM2FECO5EXS7RUDH4VGV7FJA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667027539 (7h36m5.218498s) [noise=true , address=145.79.227.9:4366 , piece_id=BT3GWQSXIMMSFPUYJC5EQ2LX2DS4BW5DAMW6ILLNVVRUUHZMOQUA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667041675 (7h36m5.218512126s) [noise=true , address=145.79.227.17:4972 , piece_id=AI3KVNUJKJUD33YVU2Z472R72SKQKRYQG65JPKFZ5WBJLKRLW6MA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667502794 (7h36m5.218973255s) [noise=true , address=145.79.227.13:4803 , piece_id=V37DZKBC4CHRGISXEQHTQAGKZFNXQKWXSWBSMHNRQTSJ32WYI2OQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667805441 (7h36m5.219275902s) [noise=true , address=145.79.227.9:4365 , piece_id=7BK6E45EY6Y3SV6PY3A4EVCS42PC4HYLXU3J7MN4HXRLDEFUEFCA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667895652 (7h36m5.219366113s) [noise=true , address=145.79.227.17:4970 , piece_id=5ZP5DDC3XAYQ5UFYJVQV6IZHB5KQQGNG54MTYW3UUYPN2IAK5QVQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.668240909 (7h36m5.219711361s) [noise=true , address=77.90.31.17:4772 , piece_id=CT6BNOYZGDBUEFVGM2RDS4ICMWPYS6N4IOUDA5V7MJQ3IKELNW2Q, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.668671640 (7h36m5.220142101s) [noise=true , address=145.79.227.31:4797 , piece_id=VQEEGJTLJY2IMCDHL2GMVGYEX4VL34TMMXKDYNERFYK3HRV7TC6A, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.669093554 (7h36m5.220564015s) [noise=true , address=145.79.227.21:4474 , piece_id=J5XE2MTCEW4ZHQDDZ4KCVYFK5EX6RQZARKNY4QCH4HUHGJO3SXDQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.669510097 (7h36m5.220980568s) [noise=true , address=77.90.31.18:4807 , piece_id=HDJAJYXNZNZN3LVAQSYVJEOQ3MU5H3BXCCW3NA2GLKPT7ATZHSJQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.671936887 (7h36m5.223407348s) [noise=true , address=145.79.227.20:4439 , piece_id=PPOVWSRAU7Y4CZAKAXV6ZP74EEPHONUC35O6XM57VE47NVJL5WZA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.672494549 (7h36m5.22396502s) [noise=true , address=77.90.31.21:4877 , piece_id=3V3Z6GNF32Q66EOS66WV3AGJNA7IXT5NHVAPKNUGVYU3R46IKKKA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:08.438197970 (7h36m6.989668451s) [upload=6714]: Next returned nil segment Nov 9 15:40:08.438219050 (7h36m6.989689521s) [upload=6714]: Waiting for error group managing segments... Nov 9 15:40:08.829503696 (7h36m7.380974147s) [noise=true , address=77.90.31.18:4807 , piece_id=HDJAJYXNZNZN3LVAQSYVJEOQ3MU5H3BXCCW3NA2GLKPT7ATZHSJQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.846923116 (7h36m7.398393577s) [noise=true , address=45.91.116.14:4190 , piece_id=XSIDZL5HMLZTQE7ZAPURAAUGQLPP3QOZK5BIE5PHRMZPSTN5APSA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.867005608 (7h36m7.418476069s) [noise=true , address=77.90.31.21:4877 , piece_id=3V3Z6GNF32Q66EOS66WV3AGJNA7IXT5NHVAPKNUGVYU3R46IKKKA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.888352807 (7h36m7.439823268s) [noise=true , address=77.90.31.14:4662 , piece_id=HHSJDZPYKGCHURAZV46OQSGBU7D4TFPXREDBFPGTYF7OAUMT2X3A, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.904680649 (7h36m7.45615111s) [noise=true , address=77.90.31.17:4772 , piece_id=CT6BNOYZGDBUEFVGM2RDS4ICMWPYS6N4IOUDA5V7MJQ3IKELNW2Q, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:18.663082162 (7h36m17.214552623s) [noise=true , address=145.79.227.17:4971 , piece_id=VTUI377UCF44ICILORHEKUO33ICRXM2FECO5EXS7RUDH4VGV7FJA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:20.077194714 (7h36m18.628665185s) [noise=true , address=145.79.227.13:4803 , piece_id=V37DZKBC4CHRGISXEQHTQAGKZFNXQKWXSWBSMHNRQTSJ32WYI2OQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:28.517900114 (7h36m27.069370575s) [noise=true , address=145.79.227.32:4835 , piece_id=EYIBNSVPQPKLKOBHKZI2VZHVCDKW3C3YJSFEWP4G62UTBXX2YRZA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:28.567571990 (7h36m27.119042451s) [noise=true , address=145.79.227.31:4797 , piece_id=VQEEGJTLJY2IMCDHL2GMVGYEX4VL34TMMXKDYNERFYK3HRV7TC6A, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:33.651382593 (7h36m32.202853044s) [noise=true , address=145.79.227.20:4439 , piece_id=PPOVWSRAU7Y4CZAKAXV6ZP74EEPHONUC35O6XM57VE47NVJL5WZA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:34.842221635 (7h36m33.393692096s) [noise=true , address=145.79.227.9:4365 , piece_id=7BK6E45EY6Y3SV6PY3A4EVCS42PC4HYLXU3J7MN4HXRLDEFUEFCA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:35.601406517 (7h36m34.152877008s) [noise=true , address=145.79.227.25:4619 , piece_id=23D3MCZRVNPI5KB43ROAHKUALDGXLZBZU5E6EIU6U2FY3YQOVPGA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:35.697082303 (7h36m34.248552774s) [noise=true , address=145.79.227.17:4970 , piece_id=5ZP5DDC3XAYQ5UFYJVQV6IZHB5KQQGNG54MTYW3UUYPN2IAK5QVQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697252518 (7h36m34.248722969s) [noise=true , address=145.79.227.17:4972 , piece_id=AI3KVNUJKJUD33YVU2Z472R72SKQKRYQG65JPKFZ5WBJLKRLW6MA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697302322 (7h36m34.248772773s) [noise=true , address=145.79.227.21:4474 , piece_id=J5XE2MTCEW4ZHQDDZ4KCVYFK5EX6RQZARKNY4QCH4HUHGJO3SXDQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, address:145.79.227.21:4474): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697472546 (7h36m34.248943007s) [noise=true , address=145.79.227.9:4366 , piece_id=BT3GWQSXIMMSFPUYJC5EQ2LX2DS4BW5DAMW6ILLNVVRUUHZMOQUA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697514686 (7h36m34.248985147s) [seg_pos={0 0}, upload=6714]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 9 15:40:35.950211122 (7h36m34.501681583s) [upload=6714]: Done uploading segments... Nov 9 15:40:35.950416643 (7h36m34.501887104s) [upload=6714]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 10.120440546s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 10.805595055s trace: https://jaeger.dev.storj.io/trace/3846516a5cecd778 FAIL (2025-11-09 15:42:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 9 15:40:06.410809146 (7h36m4.962279607s) [upload=6714]: Starting upload Nov 9 15:40:06.411046298 (7h36m4.962516769s) [upload=6714]: Done starting upload Nov 9 15:40:06.411314729 (7h36m4.96278519s) [upload=6714]: Uploading segments... Nov 9 15:40:06.411410882 (7h36m4.962881383s) [upload=6714]: Got next segment. Inline: false Nov 9 15:40:06.411464574 (7h36m4.962935035s) [upload=6714]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 9 15:40:06.411475064 (7h36m4.962945555s) [upload=6714]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 9 15:40:06.664789367 (7h36m5.216259828s) [seg_pos={0 0}, upload=6714]: Begin upload segment... Nov 9 15:40:06.664851705 (7h36m5.216322166s) [seg_pos={0 0}, upload=6714]: Done begin upload segment. Nov 9 15:40:06.664891662 (7h36m5.216362123s) [noise=true , address=145.79.227.32:4835 , piece_id=EYIBNSVPQPKLKOBHKZI2VZHVCDKW3C3YJSFEWP4G62UTBXX2YRZA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.664996933 (7h36m5.216467394s) [noise=true , address=45.91.116.14:4190 , piece_id=XSIDZL5HMLZTQE7ZAPURAAUGQLPP3QOZK5BIE5PHRMZPSTN5APSA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.665698789 (7h36m5.21716925s) [noise=true , address=77.90.31.14:4662 , piece_id=HHSJDZPYKGCHURAZV46OQSGBU7D4TFPXREDBFPGTYF7OAUMT2X3A, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.666456111 (7h36m5.217926572s) [noise=true , address=145.79.227.25:4619 , piece_id=23D3MCZRVNPI5KB43ROAHKUALDGXLZBZU5E6EIU6U2FY3YQOVPGA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.666642225 (7h36m5.218112676s) [noise=true , address=145.79.227.17:4971 , piece_id=VTUI377UCF44ICILORHEKUO33ICRXM2FECO5EXS7RUDH4VGV7FJA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667027539 (7h36m5.218498s) [noise=true , address=145.79.227.9:4366 , piece_id=BT3GWQSXIMMSFPUYJC5EQ2LX2DS4BW5DAMW6ILLNVVRUUHZMOQUA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667041675 (7h36m5.218512126s) [noise=true , address=145.79.227.17:4972 , piece_id=AI3KVNUJKJUD33YVU2Z472R72SKQKRYQG65JPKFZ5WBJLKRLW6MA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667502794 (7h36m5.218973255s) [noise=true , address=145.79.227.13:4803 , piece_id=V37DZKBC4CHRGISXEQHTQAGKZFNXQKWXSWBSMHNRQTSJ32WYI2OQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667805441 (7h36m5.219275902s) [noise=true , address=145.79.227.9:4365 , piece_id=7BK6E45EY6Y3SV6PY3A4EVCS42PC4HYLXU3J7MN4HXRLDEFUEFCA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667895652 (7h36m5.219366113s) [noise=true , address=145.79.227.17:4970 , piece_id=5ZP5DDC3XAYQ5UFYJVQV6IZHB5KQQGNG54MTYW3UUYPN2IAK5QVQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.668240909 (7h36m5.219711361s) [noise=true , address=77.90.31.17:4772 , piece_id=CT6BNOYZGDBUEFVGM2RDS4ICMWPYS6N4IOUDA5V7MJQ3IKELNW2Q, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.668671640 (7h36m5.220142101s) [noise=true , address=145.79.227.31:4797 , piece_id=VQEEGJTLJY2IMCDHL2GMVGYEX4VL34TMMXKDYNERFYK3HRV7TC6A, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.669093554 (7h36m5.220564015s) [noise=true , address=145.79.227.21:4474 , piece_id=J5XE2MTCEW4ZHQDDZ4KCVYFK5EX6RQZARKNY4QCH4HUHGJO3SXDQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.669510097 (7h36m5.220980568s) [noise=true , address=77.90.31.18:4807 , piece_id=HDJAJYXNZNZN3LVAQSYVJEOQ3MU5H3BXCCW3NA2GLKPT7ATZHSJQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.671936887 (7h36m5.223407348s) [noise=true , address=145.79.227.20:4439 , piece_id=PPOVWSRAU7Y4CZAKAXV6ZP74EEPHONUC35O6XM57VE47NVJL5WZA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.672494549 (7h36m5.22396502s) [noise=true , address=77.90.31.21:4877 , piece_id=3V3Z6GNF32Q66EOS66WV3AGJNA7IXT5NHVAPKNUGVYU3R46IKKKA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:08.438197970 (7h36m6.989668451s) [upload=6714]: Next returned nil segment Nov 9 15:40:08.438219050 (7h36m6.989689521s) [upload=6714]: Waiting for error group managing segments... Nov 9 15:40:08.829503696 (7h36m7.380974147s) [noise=true , address=77.90.31.18:4807 , piece_id=HDJAJYXNZNZN3LVAQSYVJEOQ3MU5H3BXCCW3NA2GLKPT7ATZHSJQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.846923116 (7h36m7.398393577s) [noise=true , address=45.91.116.14:4190 , piece_id=XSIDZL5HMLZTQE7ZAPURAAUGQLPP3QOZK5BIE5PHRMZPSTN5APSA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.867005608 (7h36m7.418476069s) [noise=true , address=77.90.31.21:4877 , piece_id=3V3Z6GNF32Q66EOS66WV3AGJNA7IXT5NHVAPKNUGVYU3R46IKKKA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.888352807 (7h36m7.439823268s) [noise=true , address=77.90.31.14:4662 , piece_id=HHSJDZPYKGCHURAZV46OQSGBU7D4TFPXREDBFPGTYF7OAUMT2X3A, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.904680649 (7h36m7.45615111s) [noise=true , address=77.90.31.17:4772 , piece_id=CT6BNOYZGDBUEFVGM2RDS4ICMWPYS6N4IOUDA5V7MJQ3IKELNW2Q, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:18.663082162 (7h36m17.214552623s) [noise=true , address=145.79.227.17:4971 , piece_id=VTUI377UCF44ICILORHEKUO33ICRXM2FECO5EXS7RUDH4VGV7FJA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:20.077194714 (7h36m18.628665185s) [noise=true , address=145.79.227.13:4803 , piece_id=V37DZKBC4CHRGISXEQHTQAGKZFNXQKWXSWBSMHNRQTSJ32WYI2OQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:28.517900114 (7h36m27.069370575s) [noise=true , address=145.79.227.32:4835 , piece_id=EYIBNSVPQPKLKOBHKZI2VZHVCDKW3C3YJSFEWP4G62UTBXX2YRZA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:28.567571990 (7h36m27.119042451s) [noise=true , address=145.79.227.31:4797 , piece_id=VQEEGJTLJY2IMCDHL2GMVGYEX4VL34TMMXKDYNERFYK3HRV7TC6A, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:33.651382593 (7h36m32.202853044s) [noise=true , address=145.79.227.20:4439 , piece_id=PPOVWSRAU7Y4CZAKAXV6ZP74EEPHONUC35O6XM57VE47NVJL5WZA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:34.842221635 (7h36m33.393692096s) [noise=true , address=145.79.227.9:4365 , piece_id=7BK6E45EY6Y3SV6PY3A4EVCS42PC4HYLXU3J7MN4HXRLDEFUEFCA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:35.601406517 (7h36m34.152877008s) [noise=true , address=145.79.227.25:4619 , piece_id=23D3MCZRVNPI5KB43ROAHKUALDGXLZBZU5E6EIU6U2FY3YQOVPGA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:35.697082303 (7h36m34.248552774s) [noise=true , address=145.79.227.17:4970 , piece_id=5ZP5DDC3XAYQ5UFYJVQV6IZHB5KQQGNG54MTYW3UUYPN2IAK5QVQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697252518 (7h36m34.248722969s) [noise=true , address=145.79.227.17:4972 , piece_id=AI3KVNUJKJUD33YVU2Z472R72SKQKRYQG65JPKFZ5WBJLKRLW6MA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697302322 (7h36m34.248772773s) [noise=true , address=145.79.227.21:4474 , piece_id=J5XE2MTCEW4ZHQDDZ4KCVYFK5EX6RQZARKNY4QCH4HUHGJO3SXDQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, address:145.79.227.21:4474): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697472546 (7h36m34.248943007s) [noise=true , address=145.79.227.9:4366 , piece_id=BT3GWQSXIMMSFPUYJC5EQ2LX2DS4BW5DAMW6ILLNVVRUUHZMOQUA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697514686 (7h36m34.248985147s) [seg_pos={0 0}, upload=6714]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 9 15:40:35.950211122 (7h36m34.501681583s) [upload=6714]: Done uploading segments... Nov 9 15:40:35.950416643 (7h36m34.501887104s) [upload=6714]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 10.805624871s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [............X..] duration: 63.176571ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/3ea6c69616edd41 FAIL (2025-11-09 15:42:48 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:59888->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 63.198554ms core website
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [........X......] duration: 309.942964ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/47f4a3db63391ab8 FAIL (2025-11-09 15:42:34 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:54114->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 309.973984ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 9.463300687s trace: https://jaeger.dev.storj.io/trace/3846516a5cecd778 FAIL (2025-11-09 15:42:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 9 15:40:06.410809146 (7h36m4.962279607s) [upload=6714]: Starting upload Nov 9 15:40:06.411046298 (7h36m4.962516769s) [upload=6714]: Done starting upload Nov 9 15:40:06.411314729 (7h36m4.96278519s) [upload=6714]: Uploading segments... Nov 9 15:40:06.411410882 (7h36m4.962881383s) [upload=6714]: Got next segment. Inline: false Nov 9 15:40:06.411464574 (7h36m4.962935035s) [upload=6714]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 9 15:40:06.411475064 (7h36m4.962945555s) [upload=6714]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 9 15:40:06.664789367 (7h36m5.216259828s) [seg_pos={0 0}, upload=6714]: Begin upload segment... Nov 9 15:40:06.664851705 (7h36m5.216322166s) [seg_pos={0 0}, upload=6714]: Done begin upload segment. Nov 9 15:40:06.664891662 (7h36m5.216362123s) [noise=true , address=145.79.227.32:4835 , piece_id=EYIBNSVPQPKLKOBHKZI2VZHVCDKW3C3YJSFEWP4G62UTBXX2YRZA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.664996933 (7h36m5.216467394s) [noise=true , address=45.91.116.14:4190 , piece_id=XSIDZL5HMLZTQE7ZAPURAAUGQLPP3QOZK5BIE5PHRMZPSTN5APSA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.665698789 (7h36m5.21716925s) [noise=true , address=77.90.31.14:4662 , piece_id=HHSJDZPYKGCHURAZV46OQSGBU7D4TFPXREDBFPGTYF7OAUMT2X3A, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.666456111 (7h36m5.217926572s) [noise=true , address=145.79.227.25:4619 , piece_id=23D3MCZRVNPI5KB43ROAHKUALDGXLZBZU5E6EIU6U2FY3YQOVPGA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.666642225 (7h36m5.218112676s) [noise=true , address=145.79.227.17:4971 , piece_id=VTUI377UCF44ICILORHEKUO33ICRXM2FECO5EXS7RUDH4VGV7FJA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667027539 (7h36m5.218498s) [noise=true , address=145.79.227.9:4366 , piece_id=BT3GWQSXIMMSFPUYJC5EQ2LX2DS4BW5DAMW6ILLNVVRUUHZMOQUA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667041675 (7h36m5.218512126s) [noise=true , address=145.79.227.17:4972 , piece_id=AI3KVNUJKJUD33YVU2Z472R72SKQKRYQG65JPKFZ5WBJLKRLW6MA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667502794 (7h36m5.218973255s) [noise=true , address=145.79.227.13:4803 , piece_id=V37DZKBC4CHRGISXEQHTQAGKZFNXQKWXSWBSMHNRQTSJ32WYI2OQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667805441 (7h36m5.219275902s) [noise=true , address=145.79.227.9:4365 , piece_id=7BK6E45EY6Y3SV6PY3A4EVCS42PC4HYLXU3J7MN4HXRLDEFUEFCA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667895652 (7h36m5.219366113s) [noise=true , address=145.79.227.17:4970 , piece_id=5ZP5DDC3XAYQ5UFYJVQV6IZHB5KQQGNG54MTYW3UUYPN2IAK5QVQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.668240909 (7h36m5.219711361s) [noise=true , address=77.90.31.17:4772 , piece_id=CT6BNOYZGDBUEFVGM2RDS4ICMWPYS6N4IOUDA5V7MJQ3IKELNW2Q, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.668671640 (7h36m5.220142101s) [noise=true , address=145.79.227.31:4797 , piece_id=VQEEGJTLJY2IMCDHL2GMVGYEX4VL34TMMXKDYNERFYK3HRV7TC6A, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.669093554 (7h36m5.220564015s) [noise=true , address=145.79.227.21:4474 , piece_id=J5XE2MTCEW4ZHQDDZ4KCVYFK5EX6RQZARKNY4QCH4HUHGJO3SXDQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.669510097 (7h36m5.220980568s) [noise=true , address=77.90.31.18:4807 , piece_id=HDJAJYXNZNZN3LVAQSYVJEOQ3MU5H3BXCCW3NA2GLKPT7ATZHSJQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.671936887 (7h36m5.223407348s) [noise=true , address=145.79.227.20:4439 , piece_id=PPOVWSRAU7Y4CZAKAXV6ZP74EEPHONUC35O6XM57VE47NVJL5WZA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.672494549 (7h36m5.22396502s) [noise=true , address=77.90.31.21:4877 , piece_id=3V3Z6GNF32Q66EOS66WV3AGJNA7IXT5NHVAPKNUGVYU3R46IKKKA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:08.438197970 (7h36m6.989668451s) [upload=6714]: Next returned nil segment Nov 9 15:40:08.438219050 (7h36m6.989689521s) [upload=6714]: Waiting for error group managing segments... Nov 9 15:40:08.829503696 (7h36m7.380974147s) [noise=true , address=77.90.31.18:4807 , piece_id=HDJAJYXNZNZN3LVAQSYVJEOQ3MU5H3BXCCW3NA2GLKPT7ATZHSJQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.846923116 (7h36m7.398393577s) [noise=true , address=45.91.116.14:4190 , piece_id=XSIDZL5HMLZTQE7ZAPURAAUGQLPP3QOZK5BIE5PHRMZPSTN5APSA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.867005608 (7h36m7.418476069s) [noise=true , address=77.90.31.21:4877 , piece_id=3V3Z6GNF32Q66EOS66WV3AGJNA7IXT5NHVAPKNUGVYU3R46IKKKA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.888352807 (7h36m7.439823268s) [noise=true , address=77.90.31.14:4662 , piece_id=HHSJDZPYKGCHURAZV46OQSGBU7D4TFPXREDBFPGTYF7OAUMT2X3A, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.904680649 (7h36m7.45615111s) [noise=true , address=77.90.31.17:4772 , piece_id=CT6BNOYZGDBUEFVGM2RDS4ICMWPYS6N4IOUDA5V7MJQ3IKELNW2Q, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:18.663082162 (7h36m17.214552623s) [noise=true , address=145.79.227.17:4971 , piece_id=VTUI377UCF44ICILORHEKUO33ICRXM2FECO5EXS7RUDH4VGV7FJA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:20.077194714 (7h36m18.628665185s) [noise=true , address=145.79.227.13:4803 , piece_id=V37DZKBC4CHRGISXEQHTQAGKZFNXQKWXSWBSMHNRQTSJ32WYI2OQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:28.517900114 (7h36m27.069370575s) [noise=true , address=145.79.227.32:4835 , piece_id=EYIBNSVPQPKLKOBHKZI2VZHVCDKW3C3YJSFEWP4G62UTBXX2YRZA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:28.567571990 (7h36m27.119042451s) [noise=true , address=145.79.227.31:4797 , piece_id=VQEEGJTLJY2IMCDHL2GMVGYEX4VL34TMMXKDYNERFYK3HRV7TC6A, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:33.651382593 (7h36m32.202853044s) [noise=true , address=145.79.227.20:4439 , piece_id=PPOVWSRAU7Y4CZAKAXV6ZP74EEPHONUC35O6XM57VE47NVJL5WZA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:34.842221635 (7h36m33.393692096s) [noise=true , address=145.79.227.9:4365 , piece_id=7BK6E45EY6Y3SV6PY3A4EVCS42PC4HYLXU3J7MN4HXRLDEFUEFCA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:35.601406517 (7h36m34.152877008s) [noise=true , address=145.79.227.25:4619 , piece_id=23D3MCZRVNPI5KB43ROAHKUALDGXLZBZU5E6EIU6U2FY3YQOVPGA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:35.697082303 (7h36m34.248552774s) [noise=true , address=145.79.227.17:4970 , piece_id=5ZP5DDC3XAYQ5UFYJVQV6IZHB5KQQGNG54MTYW3UUYPN2IAK5QVQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697252518 (7h36m34.248722969s) [noise=true , address=145.79.227.17:4972 , piece_id=AI3KVNUJKJUD33YVU2Z472R72SKQKRYQG65JPKFZ5WBJLKRLW6MA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697302322 (7h36m34.248772773s) [noise=true , address=145.79.227.21:4474 , piece_id=J5XE2MTCEW4ZHQDDZ4KCVYFK5EX6RQZARKNY4QCH4HUHGJO3SXDQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, address:145.79.227.21:4474): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697472546 (7h36m34.248943007s) [noise=true , address=145.79.227.9:4366 , piece_id=BT3GWQSXIMMSFPUYJC5EQ2LX2DS4BW5DAMW6ILLNVVRUUHZMOQUA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697514686 (7h36m34.248985147s) [seg_pos={0 0}, upload=6714]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 9 15:40:35.950211122 (7h36m34.501681583s) [upload=6714]: Done uploading segments... Nov 9 15:40:35.950416643 (7h36m34.501887104s) [upload=6714]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 9.463331937s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 9.347891155s trace: https://jaeger.dev.storj.io/trace/3846516a5cecd778 FAIL (2025-11-09 15:42:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 9 15:40:06.410809146 (7h36m4.962279607s) [upload=6714]: Starting upload Nov 9 15:40:06.411046298 (7h36m4.962516769s) [upload=6714]: Done starting upload Nov 9 15:40:06.411314729 (7h36m4.96278519s) [upload=6714]: Uploading segments... Nov 9 15:40:06.411410882 (7h36m4.962881383s) [upload=6714]: Got next segment. Inline: false Nov 9 15:40:06.411464574 (7h36m4.962935035s) [upload=6714]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 9 15:40:06.411475064 (7h36m4.962945555s) [upload=6714]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 9 15:40:06.664789367 (7h36m5.216259828s) [seg_pos={0 0}, upload=6714]: Begin upload segment... Nov 9 15:40:06.664851705 (7h36m5.216322166s) [seg_pos={0 0}, upload=6714]: Done begin upload segment. Nov 9 15:40:06.664891662 (7h36m5.216362123s) [noise=true , address=145.79.227.32:4835 , piece_id=EYIBNSVPQPKLKOBHKZI2VZHVCDKW3C3YJSFEWP4G62UTBXX2YRZA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.664996933 (7h36m5.216467394s) [noise=true , address=45.91.116.14:4190 , piece_id=XSIDZL5HMLZTQE7ZAPURAAUGQLPP3QOZK5BIE5PHRMZPSTN5APSA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.665698789 (7h36m5.21716925s) [noise=true , address=77.90.31.14:4662 , piece_id=HHSJDZPYKGCHURAZV46OQSGBU7D4TFPXREDBFPGTYF7OAUMT2X3A, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.666456111 (7h36m5.217926572s) [noise=true , address=145.79.227.25:4619 , piece_id=23D3MCZRVNPI5KB43ROAHKUALDGXLZBZU5E6EIU6U2FY3YQOVPGA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.666642225 (7h36m5.218112676s) [noise=true , address=145.79.227.17:4971 , piece_id=VTUI377UCF44ICILORHEKUO33ICRXM2FECO5EXS7RUDH4VGV7FJA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667027539 (7h36m5.218498s) [noise=true , address=145.79.227.9:4366 , piece_id=BT3GWQSXIMMSFPUYJC5EQ2LX2DS4BW5DAMW6ILLNVVRUUHZMOQUA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667041675 (7h36m5.218512126s) [noise=true , address=145.79.227.17:4972 , piece_id=AI3KVNUJKJUD33YVU2Z472R72SKQKRYQG65JPKFZ5WBJLKRLW6MA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667502794 (7h36m5.218973255s) [noise=true , address=145.79.227.13:4803 , piece_id=V37DZKBC4CHRGISXEQHTQAGKZFNXQKWXSWBSMHNRQTSJ32WYI2OQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667805441 (7h36m5.219275902s) [noise=true , address=145.79.227.9:4365 , piece_id=7BK6E45EY6Y3SV6PY3A4EVCS42PC4HYLXU3J7MN4HXRLDEFUEFCA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.667895652 (7h36m5.219366113s) [noise=true , address=145.79.227.17:4970 , piece_id=5ZP5DDC3XAYQ5UFYJVQV6IZHB5KQQGNG54MTYW3UUYPN2IAK5QVQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.668240909 (7h36m5.219711361s) [noise=true , address=77.90.31.17:4772 , piece_id=CT6BNOYZGDBUEFVGM2RDS4ICMWPYS6N4IOUDA5V7MJQ3IKELNW2Q, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.668671640 (7h36m5.220142101s) [noise=true , address=145.79.227.31:4797 , piece_id=VQEEGJTLJY2IMCDHL2GMVGYEX4VL34TMMXKDYNERFYK3HRV7TC6A, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.669093554 (7h36m5.220564015s) [noise=true , address=145.79.227.21:4474 , piece_id=J5XE2MTCEW4ZHQDDZ4KCVYFK5EX6RQZARKNY4QCH4HUHGJO3SXDQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.669510097 (7h36m5.220980568s) [noise=true , address=77.90.31.18:4807 , piece_id=HDJAJYXNZNZN3LVAQSYVJEOQ3MU5H3BXCCW3NA2GLKPT7ATZHSJQ, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.671936887 (7h36m5.223407348s) [noise=true , address=145.79.227.20:4439 , piece_id=PPOVWSRAU7Y4CZAKAXV6ZP74EEPHONUC35O6XM57VE47NVJL5WZA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:06.672494549 (7h36m5.22396502s) [noise=true , address=77.90.31.21:4877 , piece_id=3V3Z6GNF32Q66EOS66WV3AGJNA7IXT5NHVAPKNUGVYU3R46IKKKA, seg_pos={0 0}, upload=6714]: Uploading piece... Nov 9 15:40:08.438197970 (7h36m6.989668451s) [upload=6714]: Next returned nil segment Nov 9 15:40:08.438219050 (7h36m6.989689521s) [upload=6714]: Waiting for error group managing segments... Nov 9 15:40:08.829503696 (7h36m7.380974147s) [noise=true , address=77.90.31.18:4807 , piece_id=HDJAJYXNZNZN3LVAQSYVJEOQ3MU5H3BXCCW3NA2GLKPT7ATZHSJQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.846923116 (7h36m7.398393577s) [noise=true , address=45.91.116.14:4190 , piece_id=XSIDZL5HMLZTQE7ZAPURAAUGQLPP3QOZK5BIE5PHRMZPSTN5APSA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.867005608 (7h36m7.418476069s) [noise=true , address=77.90.31.21:4877 , piece_id=3V3Z6GNF32Q66EOS66WV3AGJNA7IXT5NHVAPKNUGVYU3R46IKKKA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.888352807 (7h36m7.439823268s) [noise=true , address=77.90.31.14:4662 , piece_id=HHSJDZPYKGCHURAZV46OQSGBU7D4TFPXREDBFPGTYF7OAUMT2X3A, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:08.904680649 (7h36m7.45615111s) [noise=true , address=77.90.31.17:4772 , piece_id=CT6BNOYZGDBUEFVGM2RDS4ICMWPYS6N4IOUDA5V7MJQ3IKELNW2Q, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:18.663082162 (7h36m17.214552623s) [noise=true , address=145.79.227.17:4971 , piece_id=VTUI377UCF44ICILORHEKUO33ICRXM2FECO5EXS7RUDH4VGV7FJA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:20.077194714 (7h36m18.628665185s) [noise=true , address=145.79.227.13:4803 , piece_id=V37DZKBC4CHRGISXEQHTQAGKZFNXQKWXSWBSMHNRQTSJ32WYI2OQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:28.517900114 (7h36m27.069370575s) [noise=true , address=145.79.227.32:4835 , piece_id=EYIBNSVPQPKLKOBHKZI2VZHVCDKW3C3YJSFEWP4G62UTBXX2YRZA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:28.567571990 (7h36m27.119042451s) [noise=true , address=145.79.227.31:4797 , piece_id=VQEEGJTLJY2IMCDHL2GMVGYEX4VL34TMMXKDYNERFYK3HRV7TC6A, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:33.651382593 (7h36m32.202853044s) [noise=true , address=145.79.227.20:4439 , piece_id=PPOVWSRAU7Y4CZAKAXV6ZP74EEPHONUC35O6XM57VE47NVJL5WZA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:34.842221635 (7h36m33.393692096s) [noise=true , address=145.79.227.9:4365 , piece_id=7BK6E45EY6Y3SV6PY3A4EVCS42PC4HYLXU3J7MN4HXRLDEFUEFCA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:35.601406517 (7h36m34.152877008s) [noise=true , address=145.79.227.25:4619 , piece_id=23D3MCZRVNPI5KB43ROAHKUALDGXLZBZU5E6EIU6U2FY3YQOVPGA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: <nil> Nov 9 15:40:35.697082303 (7h36m34.248552774s) [noise=true , address=145.79.227.17:4970 , piece_id=5ZP5DDC3XAYQ5UFYJVQV6IZHB5KQQGNG54MTYW3UUYPN2IAK5QVQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697252518 (7h36m34.248722969s) [noise=true , address=145.79.227.17:4972 , piece_id=AI3KVNUJKJUD33YVU2Z472R72SKQKRYQG65JPKFZ5WBJLKRLW6MA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697302322 (7h36m34.248772773s) [noise=true , address=145.79.227.21:4474 , piece_id=J5XE2MTCEW4ZHQDDZ4KCVYFK5EX6RQZARKNY4QCH4HUHGJO3SXDQ, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, address:145.79.227.21:4474): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697472546 (7h36m34.248943007s) [noise=true , address=145.79.227.9:4366 , piece_id=BT3GWQSXIMMSFPUYJC5EQ2LX2DS4BW5DAMW6ILLNVVRUUHZMOQUA, seg_pos={0 0}, upload=6714]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: expected piece hash; context deadline exceeded; EOF Nov 9 15:40:35.697514686 (7h36m34.248985147s) [seg_pos={0 0}, upload=6714]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 9 15:40:35.950211122 (7h36m34.501681583s) [upload=6714]: Done uploading segments... Nov 9 15:40:35.950416643 (7h36m34.501887104s) [upload=6714]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 9.347925872s core uplink us-select-1

Orbiter Check History

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)