Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [...X.X.....X.......X......X............X.................X..] duration: 7.60680057s trace: https://jaeger.dev.storj.io/trace/2af0ef576bba07f6 FAIL (2025-11-22 15:07:32 UTC) orbiter warning: context deadline exceeded Duration: 7.60683231s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [....X.X.....X.......X......X............X.................X.] duration: 22.319117497s trace: https://jaeger.dev.storj.io/trace/2af0ef576bba07f6 FAIL (2025-11-22 15:07:24 UTC) orbiter warning: context deadline exceeded Duration: 22.319137835s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [.......XX..X..XX..X.X.........XX.XX.................XX..X...] duration: 3.664866654s trace: https://jaeger.dev.storj.io/trace/cc91f835756ebe2 FAIL (2025-11-22 15:07:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.664920195s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......................XX.X.........X........X..............] duration: 15.804770221s trace: https://jaeger.dev.storj.io/trace/1efcdf5c2a183d8d FAIL (2025-11-22 15:06:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 934958555s) [noise=true , address=145.79.227.30:4762 , piece_id=4Z7CCNLZEYQAGJNUWGJ2USQSVW6QZUTCNFKC3JWRIOKV6VBVIYKQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.384231169 (318h54m50.93570163s) [noise=true , address=145.79.227.18:4007 , piece_id=KM5TO27LZXJZZAT2DB5JTP7R5LOPJHUU6T34RZI7HH7PRWPW4ZNQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.384687758 (318h54m50.93615822s) [noise=true , address=145.79.227.26:4654 , piece_id=VCRM5O6NHOJOKNQUG5TTZUT3JIDQOMNF2DD6H7KWUJFOQ3P75QGQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.385168865 (318h54m50.936639326s) [noise=true , address=77.90.31.18:4805 , piece_id=3EHB5IFBQRLXFBOVCT25RS4CIXVMSVOE7EKDV26DXOEY2NDJ2YGA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.385605395 (318h54m50.937075867s) [noise=true , address=145.79.227.32:4835 , piece_id=7GD4QUTGLIPOEU25FIDE7VDO3AUJ2RZDU4XKI476KNU6QTETC52A, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.386294879 (318h54m50.93776534s) [noise=true , address=145.79.227.21:4473 , piece_id=D5FC7PVECZTQR64BFVTV3L2HOXCITABN57LLNRSMVCSGQICRYKUQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.386958642 (318h54m50.938429103s) [noise=true , address=145.79.227.14:4837 , piece_id=NR427EUHKVSQ6HMMZCHL4LCBV6VRVTANZDXMX6JHQI2TTQN5R5VQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387097236 (318h54m50.938567687s) [noise=true , address=145.79.227.9:4365 , piece_id=DBW2L6GHAWIHOMOZOTQFQ57Y2MA6KP6SGUO6AJDDF65ERQ324VZQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387456781 (318h54m50.938927242s) [noise=true , address=145.79.227.19:4078 , piece_id=ER3KKYJ2MJQC3HRRBVULC3ING5GLGOHFVD2QBWF2MVAXF6ODMUUA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387755300 (318h54m50.939225761s) [noise=true , address=145.79.227.28:4725 , piece_id=HVMAX2UWJIF2UJQIJS5UBTO5TOIJI5DCDGGURZN4SOMRY3Z4OF4A, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.388354220 (318h54m50.939824681s) [noise=true , address=145.79.227.10:4402 , piece_id=ZU6A2O2R73PIVIDF5NOKMJZH4QX4LOCFSYOQLFOFAA5BEVQH52KA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.388862268 (318h54m50.940332729s) [noise=true , address=145.79.227.22:4509 , piece_id=WRHXVGJZQRTY6ZKMRU6IFYZSMLR6UATXFZOTCPRYKHERA2WKMGNA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.389411814 (318h54m50.940882275s) [noise=true , address=145.79.227.8:4330 , piece_id=LFTZWYQKW56VLMJPOGGLB6IJILPGJNEHIRIYD2F4J34ZH24PCSMQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.389753224 (318h54m50.941223675s) [noise=true , address=145.79.227.16:4909 , piece_id=3DAVIY3ETZVG5CPP5BQB7QL473J5HZG6EQ6BYZBIAN6TWZ5FGNHQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.391102172 (318h54m50.942572633s) [noise=true , address=45.91.116.7:4010 , piece_id=VZCYHOAR73DVCRDT5GBT65JI53NP27OEGA3VYLM66LPQR23T4MTA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:54.404583235 (318h54m52.956053696s) [upload=280863]: Next returned nil segment Nov 22 14:58:54.404601019 (318h54m52.956071481s) [upload=280863]: Waiting for error group managing segments... Nov 22 14:58:54.928222688 (318h54m53.479693149s) [noise=true , address=77.90.31.18:4805 , piece_id=3EHB5IFBQRLXFBOVCT25RS4CIXVMSVOE7EKDV26DXOEY2NDJ2YGA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:58:55.175979631 (318h54m53.727450092s) [noise=true , address=45.91.116.7:4010 , piece_id=VZCYHOAR73DVCRDT5GBT65JI53NP27OEGA3VYLM66LPQR23T4MTA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:03.279622498 (318h55m1.831092959s) [noise=true , address=145.79.227.14:4837 , piece_id=NR427EUHKVSQ6HMMZCHL4LCBV6VRVTANZDXMX6JHQI2TTQN5R5VQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:04.274652248 (318h55m2.826122709s) [noise=true , address=145.79.227.28:4725 , piece_id=HVMAX2UWJIF2UJQIJS5UBTO5TOIJI5DCDGGURZN4SOMRY3Z4OF4A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:05.089136230 (318h55m3.640606701s) [noise=true , address=145.79.227.22:4509 , piece_id=WRHXVGJZQRTY6ZKMRU6IFYZSMLR6UATXFZOTCPRYKHERA2WKMGNA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:05.587545699 (318h55m4.13901616s) [noise=true , address=145.79.227.32:4835 , piece_id=7GD4QUTGLIPOEU25FIDE7VDO3AUJ2RZDU4XKI476KNU6QTETC52A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:06.956814829 (318h55m5.50828528s) [noise=true , address=145.79.227.8:4330 , piece_id=LFTZWYQKW56VLMJPOGGLB6IJILPGJNEHIRIYD2F4J34ZH24PCSMQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:07.254777763 (318h55m5.806248224s) [noise=true , address=145.79.227.30:4762 , piece_id=4Z7CCNLZEYQAGJNUWGJ2USQSVW6QZUTCNFKC3JWRIOKV6VBVIYKQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:19.983388333 (318h55m18.534858794s) [noise=true , address=145.79.227.10:4402 , piece_id=ZU6A2O2R73PIVIDF5NOKMJZH4QX4LOCFSYOQLFOFAA5BEVQH52KA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection Nov 22 14:59:19.983536355 (318h55m18.535006816s) [noise=true , address=145.79.227.21:4473 , piece_id=D5FC7PVECZTQR64BFVTV3L2HOXCITABN57LLNRSMVCSGQICRYKUQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection Nov 22 14:59:19.983802211 (318h55m18.535272703s) [noise=true , address=145.79.227.18:4007 , piece_id=KM5TO27LZXJZZAT2DB5JTP7R5LOPJHUU6T34RZI7HH7PRWPW4ZNQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection Nov 22 14:59:19.983812000 (318h55m18.535282461s) [noise=true , address=145.79.227.9:4365 , piece_id=DBW2L6GHAWIHOMOZOTQFQ57Y2MA6KP6SGUO6AJDDF65ERQ324VZQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection Nov 22 14:59:19.983896611 (318h55m18.535367073s) [noise=true , address=145.79.227.19:4078 , piece_id=ER3KKYJ2MJQC3HRRBVULC3ING5GLGOHFVD2QBWF2MVAXF6ODMUUA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection; write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection Nov 22 14:59:19.984068268 (318h55m18.535538719s) [noise=true , address=145.79.227.26:4654 , piece_id=VCRM5O6NHOJOKNQUG5TTZUT3JIDQOMNF2DD6H7KWUJFOQ3P75QGQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: read tcp 188.166.225.170:36440->145.79.227.26:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984229324 (318h55m18.535699776s) [noise=true , address=145.79.227.8:4329 , piece_id=YJG3LYBDMPS4QQNLG24MS7ZWUM6EOI6EPDGVBGLO33CVEQQKGX6A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: read tcp 188.166.225.170:44902->145.79.227.8:4329: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984374972 (318h55m18.535845423s) [noise=true , address=145.79.227.16:4909 , piece_id=3DAVIY3ETZVG5CPP5BQB7QL473J5HZG6EQ6BYZBIAN6TWZ5FGNHQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: read tcp 188.166.225.170:59064->145.79.227.16:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984400911 (318h55m18.535871362s) [seg_pos={0 0}, upload=280863]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 22 14:59:20.539921660 (318h55m19.091392121s) [upload=280863]: Done uploading segments... Nov 22 14:59:20.539961818 (318h55m19.091432269s) [upload=280863]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 15.804798744s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.....X.X.....X.......X......X............X.................X] duration: 30.00109215s trace: https://jaeger.dev.storj.io/trace/2af0ef576bba07f6 FAIL (2025-11-22 15:06:49 UTC) orbiter warning: context deadline exceeded Duration: 30.001115246s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [........................XX.X.........X........X.............] duration: 14.332931689s trace: https://jaeger.dev.storj.io/trace/1efcdf5c2a183d8d FAIL (2025-11-22 15:06:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 934958555s) [noise=true , address=145.79.227.30:4762 , piece_id=4Z7CCNLZEYQAGJNUWGJ2USQSVW6QZUTCNFKC3JWRIOKV6VBVIYKQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.384231169 (318h54m50.93570163s) [noise=true , address=145.79.227.18:4007 , piece_id=KM5TO27LZXJZZAT2DB5JTP7R5LOPJHUU6T34RZI7HH7PRWPW4ZNQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.384687758 (318h54m50.93615822s) [noise=true , address=145.79.227.26:4654 , piece_id=VCRM5O6NHOJOKNQUG5TTZUT3JIDQOMNF2DD6H7KWUJFOQ3P75QGQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.385168865 (318h54m50.936639326s) [noise=true , address=77.90.31.18:4805 , piece_id=3EHB5IFBQRLXFBOVCT25RS4CIXVMSVOE7EKDV26DXOEY2NDJ2YGA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.385605395 (318h54m50.937075867s) [noise=true , address=145.79.227.32:4835 , piece_id=7GD4QUTGLIPOEU25FIDE7VDO3AUJ2RZDU4XKI476KNU6QTETC52A, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.386294879 (318h54m50.93776534s) [noise=true , address=145.79.227.21:4473 , piece_id=D5FC7PVECZTQR64BFVTV3L2HOXCITABN57LLNRSMVCSGQICRYKUQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.386958642 (318h54m50.938429103s) [noise=true , address=145.79.227.14:4837 , piece_id=NR427EUHKVSQ6HMMZCHL4LCBV6VRVTANZDXMX6JHQI2TTQN5R5VQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387097236 (318h54m50.938567687s) [noise=true , address=145.79.227.9:4365 , piece_id=DBW2L6GHAWIHOMOZOTQFQ57Y2MA6KP6SGUO6AJDDF65ERQ324VZQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387456781 (318h54m50.938927242s) [noise=true , address=145.79.227.19:4078 , piece_id=ER3KKYJ2MJQC3HRRBVULC3ING5GLGOHFVD2QBWF2MVAXF6ODMUUA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387755300 (318h54m50.939225761s) [noise=true , address=145.79.227.28:4725 , piece_id=HVMAX2UWJIF2UJQIJS5UBTO5TOIJI5DCDGGURZN4SOMRY3Z4OF4A, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.388354220 (318h54m50.939824681s) [noise=true , address=145.79.227.10:4402 , piece_id=ZU6A2O2R73PIVIDF5NOKMJZH4QX4LOCFSYOQLFOFAA5BEVQH52KA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.388862268 (318h54m50.940332729s) [noise=true , address=145.79.227.22:4509 , piece_id=WRHXVGJZQRTY6ZKMRU6IFYZSMLR6UATXFZOTCPRYKHERA2WKMGNA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.389411814 (318h54m50.940882275s) [noise=true , address=145.79.227.8:4330 , piece_id=LFTZWYQKW56VLMJPOGGLB6IJILPGJNEHIRIYD2F4J34ZH24PCSMQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.389753224 (318h54m50.941223675s) [noise=true , address=145.79.227.16:4909 , piece_id=3DAVIY3ETZVG5CPP5BQB7QL473J5HZG6EQ6BYZBIAN6TWZ5FGNHQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.391102172 (318h54m50.942572633s) [noise=true , address=45.91.116.7:4010 , piece_id=VZCYHOAR73DVCRDT5GBT65JI53NP27OEGA3VYLM66LPQR23T4MTA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:54.404583235 (318h54m52.956053696s) [upload=280863]: Next returned nil segment Nov 22 14:58:54.404601019 (318h54m52.956071481s) [upload=280863]: Waiting for error group managing segments... Nov 22 14:58:54.928222688 (318h54m53.479693149s) [noise=true , address=77.90.31.18:4805 , piece_id=3EHB5IFBQRLXFBOVCT25RS4CIXVMSVOE7EKDV26DXOEY2NDJ2YGA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:58:55.175979631 (318h54m53.727450092s) [noise=true , address=45.91.116.7:4010 , piece_id=VZCYHOAR73DVCRDT5GBT65JI53NP27OEGA3VYLM66LPQR23T4MTA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:03.279622498 (318h55m1.831092959s) [noise=true , address=145.79.227.14:4837 , piece_id=NR427EUHKVSQ6HMMZCHL4LCBV6VRVTANZDXMX6JHQI2TTQN5R5VQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:04.274652248 (318h55m2.826122709s) [noise=true , address=145.79.227.28:4725 , piece_id=HVMAX2UWJIF2UJQIJS5UBTO5TOIJI5DCDGGURZN4SOMRY3Z4OF4A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:05.089136230 (318h55m3.640606701s) [noise=true , address=145.79.227.22:4509 , piece_id=WRHXVGJZQRTY6ZKMRU6IFYZSMLR6UATXFZOTCPRYKHERA2WKMGNA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:05.587545699 (318h55m4.13901616s) [noise=true , address=145.79.227.32:4835 , piece_id=7GD4QUTGLIPOEU25FIDE7VDO3AUJ2RZDU4XKI476KNU6QTETC52A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:06.956814829 (318h55m5.50828528s) [noise=true , address=145.79.227.8:4330 , piece_id=LFTZWYQKW56VLMJPOGGLB6IJILPGJNEHIRIYD2F4J34ZH24PCSMQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:07.254777763 (318h55m5.806248224s) [noise=true , address=145.79.227.30:4762 , piece_id=4Z7CCNLZEYQAGJNUWGJ2USQSVW6QZUTCNFKC3JWRIOKV6VBVIYKQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:19.983388333 (318h55m18.534858794s) [noise=true , address=145.79.227.10:4402 , piece_id=ZU6A2O2R73PIVIDF5NOKMJZH4QX4LOCFSYOQLFOFAA5BEVQH52KA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection Nov 22 14:59:19.983536355 (318h55m18.535006816s) [noise=true , address=145.79.227.21:4473 , piece_id=D5FC7PVECZTQR64BFVTV3L2HOXCITABN57LLNRSMVCSGQICRYKUQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection Nov 22 14:59:19.983802211 (318h55m18.535272703s) [noise=true , address=145.79.227.18:4007 , piece_id=KM5TO27LZXJZZAT2DB5JTP7R5LOPJHUU6T34RZI7HH7PRWPW4ZNQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection Nov 22 14:59:19.983812000 (318h55m18.535282461s) [noise=true , address=145.79.227.9:4365 , piece_id=DBW2L6GHAWIHOMOZOTQFQ57Y2MA6KP6SGUO6AJDDF65ERQ324VZQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection Nov 22 14:59:19.983896611 (318h55m18.535367073s) [noise=true , address=145.79.227.19:4078 , piece_id=ER3KKYJ2MJQC3HRRBVULC3ING5GLGOHFVD2QBWF2MVAXF6ODMUUA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection; write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection Nov 22 14:59:19.984068268 (318h55m18.535538719s) [noise=true , address=145.79.227.26:4654 , piece_id=VCRM5O6NHOJOKNQUG5TTZUT3JIDQOMNF2DD6H7KWUJFOQ3P75QGQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: read tcp 188.166.225.170:36440->145.79.227.26:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984229324 (318h55m18.535699776s) [noise=true , address=145.79.227.8:4329 , piece_id=YJG3LYBDMPS4QQNLG24MS7ZWUM6EOI6EPDGVBGLO33CVEQQKGX6A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: read tcp 188.166.225.170:44902->145.79.227.8:4329: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984374972 (318h55m18.535845423s) [noise=true , address=145.79.227.16:4909 , piece_id=3DAVIY3ETZVG5CPP5BQB7QL473J5HZG6EQ6BYZBIAN6TWZ5FGNHQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: read tcp 188.166.225.170:59064->145.79.227.16:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984400911 (318h55m18.535871362s) [seg_pos={0 0}, upload=280863]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 22 14:59:20.539921660 (318h55m19.091392121s) [upload=280863]: Done uploading segments... Nov 22 14:59:20.539961818 (318h55m19.091432269s) [upload=280863]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 14.332973439s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.........................XX.X.........X........X............] duration: 14.51940066s trace: https://jaeger.dev.storj.io/trace/1efcdf5c2a183d8d FAIL (2025-11-22 15:06:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 934958555s) [noise=true , address=145.79.227.30:4762 , piece_id=4Z7CCNLZEYQAGJNUWGJ2USQSVW6QZUTCNFKC3JWRIOKV6VBVIYKQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.384231169 (318h54m50.93570163s) [noise=true , address=145.79.227.18:4007 , piece_id=KM5TO27LZXJZZAT2DB5JTP7R5LOPJHUU6T34RZI7HH7PRWPW4ZNQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.384687758 (318h54m50.93615822s) [noise=true , address=145.79.227.26:4654 , piece_id=VCRM5O6NHOJOKNQUG5TTZUT3JIDQOMNF2DD6H7KWUJFOQ3P75QGQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.385168865 (318h54m50.936639326s) [noise=true , address=77.90.31.18:4805 , piece_id=3EHB5IFBQRLXFBOVCT25RS4CIXVMSVOE7EKDV26DXOEY2NDJ2YGA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.385605395 (318h54m50.937075867s) [noise=true , address=145.79.227.32:4835 , piece_id=7GD4QUTGLIPOEU25FIDE7VDO3AUJ2RZDU4XKI476KNU6QTETC52A, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.386294879 (318h54m50.93776534s) [noise=true , address=145.79.227.21:4473 , piece_id=D5FC7PVECZTQR64BFVTV3L2HOXCITABN57LLNRSMVCSGQICRYKUQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.386958642 (318h54m50.938429103s) [noise=true , address=145.79.227.14:4837 , piece_id=NR427EUHKVSQ6HMMZCHL4LCBV6VRVTANZDXMX6JHQI2TTQN5R5VQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387097236 (318h54m50.938567687s) [noise=true , address=145.79.227.9:4365 , piece_id=DBW2L6GHAWIHOMOZOTQFQ57Y2MA6KP6SGUO6AJDDF65ERQ324VZQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387456781 (318h54m50.938927242s) [noise=true , address=145.79.227.19:4078 , piece_id=ER3KKYJ2MJQC3HRRBVULC3ING5GLGOHFVD2QBWF2MVAXF6ODMUUA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387755300 (318h54m50.939225761s) [noise=true , address=145.79.227.28:4725 , piece_id=HVMAX2UWJIF2UJQIJS5UBTO5TOIJI5DCDGGURZN4SOMRY3Z4OF4A, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.388354220 (318h54m50.939824681s) [noise=true , address=145.79.227.10:4402 , piece_id=ZU6A2O2R73PIVIDF5NOKMJZH4QX4LOCFSYOQLFOFAA5BEVQH52KA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.388862268 (318h54m50.940332729s) [noise=true , address=145.79.227.22:4509 , piece_id=WRHXVGJZQRTY6ZKMRU6IFYZSMLR6UATXFZOTCPRYKHERA2WKMGNA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.389411814 (318h54m50.940882275s) [noise=true , address=145.79.227.8:4330 , piece_id=LFTZWYQKW56VLMJPOGGLB6IJILPGJNEHIRIYD2F4J34ZH24PCSMQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.389753224 (318h54m50.941223675s) [noise=true , address=145.79.227.16:4909 , piece_id=3DAVIY3ETZVG5CPP5BQB7QL473J5HZG6EQ6BYZBIAN6TWZ5FGNHQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.391102172 (318h54m50.942572633s) [noise=true , address=45.91.116.7:4010 , piece_id=VZCYHOAR73DVCRDT5GBT65JI53NP27OEGA3VYLM66LPQR23T4MTA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:54.404583235 (318h54m52.956053696s) [upload=280863]: Next returned nil segment Nov 22 14:58:54.404601019 (318h54m52.956071481s) [upload=280863]: Waiting for error group managing segments... Nov 22 14:58:54.928222688 (318h54m53.479693149s) [noise=true , address=77.90.31.18:4805 , piece_id=3EHB5IFBQRLXFBOVCT25RS4CIXVMSVOE7EKDV26DXOEY2NDJ2YGA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:58:55.175979631 (318h54m53.727450092s) [noise=true , address=45.91.116.7:4010 , piece_id=VZCYHOAR73DVCRDT5GBT65JI53NP27OEGA3VYLM66LPQR23T4MTA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:03.279622498 (318h55m1.831092959s) [noise=true , address=145.79.227.14:4837 , piece_id=NR427EUHKVSQ6HMMZCHL4LCBV6VRVTANZDXMX6JHQI2TTQN5R5VQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:04.274652248 (318h55m2.826122709s) [noise=true , address=145.79.227.28:4725 , piece_id=HVMAX2UWJIF2UJQIJS5UBTO5TOIJI5DCDGGURZN4SOMRY3Z4OF4A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:05.089136230 (318h55m3.640606701s) [noise=true , address=145.79.227.22:4509 , piece_id=WRHXVGJZQRTY6ZKMRU6IFYZSMLR6UATXFZOTCPRYKHERA2WKMGNA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:05.587545699 (318h55m4.13901616s) [noise=true , address=145.79.227.32:4835 , piece_id=7GD4QUTGLIPOEU25FIDE7VDO3AUJ2RZDU4XKI476KNU6QTETC52A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:06.956814829 (318h55m5.50828528s) [noise=true , address=145.79.227.8:4330 , piece_id=LFTZWYQKW56VLMJPOGGLB6IJILPGJNEHIRIYD2F4J34ZH24PCSMQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:07.254777763 (318h55m5.806248224s) [noise=true , address=145.79.227.30:4762 , piece_id=4Z7CCNLZEYQAGJNUWGJ2USQSVW6QZUTCNFKC3JWRIOKV6VBVIYKQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:19.983388333 (318h55m18.534858794s) [noise=true , address=145.79.227.10:4402 , piece_id=ZU6A2O2R73PIVIDF5NOKMJZH4QX4LOCFSYOQLFOFAA5BEVQH52KA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection Nov 22 14:59:19.983536355 (318h55m18.535006816s) [noise=true , address=145.79.227.21:4473 , piece_id=D5FC7PVECZTQR64BFVTV3L2HOXCITABN57LLNRSMVCSGQICRYKUQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection Nov 22 14:59:19.983802211 (318h55m18.535272703s) [noise=true , address=145.79.227.18:4007 , piece_id=KM5TO27LZXJZZAT2DB5JTP7R5LOPJHUU6T34RZI7HH7PRWPW4ZNQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection Nov 22 14:59:19.983812000 (318h55m18.535282461s) [noise=true , address=145.79.227.9:4365 , piece_id=DBW2L6GHAWIHOMOZOTQFQ57Y2MA6KP6SGUO6AJDDF65ERQ324VZQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection Nov 22 14:59:19.983896611 (318h55m18.535367073s) [noise=true , address=145.79.227.19:4078 , piece_id=ER3KKYJ2MJQC3HRRBVULC3ING5GLGOHFVD2QBWF2MVAXF6ODMUUA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection; write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection Nov 22 14:59:19.984068268 (318h55m18.535538719s) [noise=true , address=145.79.227.26:4654 , piece_id=VCRM5O6NHOJOKNQUG5TTZUT3JIDQOMNF2DD6H7KWUJFOQ3P75QGQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: read tcp 188.166.225.170:36440->145.79.227.26:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984229324 (318h55m18.535699776s) [noise=true , address=145.79.227.8:4329 , piece_id=YJG3LYBDMPS4QQNLG24MS7ZWUM6EOI6EPDGVBGLO33CVEQQKGX6A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: read tcp 188.166.225.170:44902->145.79.227.8:4329: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984374972 (318h55m18.535845423s) [noise=true , address=145.79.227.16:4909 , piece_id=3DAVIY3ETZVG5CPP5BQB7QL473J5HZG6EQ6BYZBIAN6TWZ5FGNHQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: read tcp 188.166.225.170:59064->145.79.227.16:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984400911 (318h55m18.535871362s) [seg_pos={0 0}, upload=280863]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 22 14:59:20.539921660 (318h55m19.091392121s) [upload=280863]: Done uploading segments... Nov 22 14:59:20.539961818 (318h55m19.091432269s) [upload=280863]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 14.519468369s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 15 out of 60 is failed (threshold=6) [X.......XX..X..XX..X.X.........XX.XX.................XX..X..] duration: 4.580023264s trace: https://jaeger.dev.storj.io/trace/cc91f835756ebe2 FAIL (2025-11-22 15:06:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.58004779s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [......X.X.....X.......X......X............X.................] duration: 7.558750965s trace: https://jaeger.dev.storj.io/trace/1ee0c6bff9434222 FAIL (2025-11-22 15:05:48 UTC) orbiter warning: context deadline exceeded Duration: 7.558785501s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..........................XX.X.........X........X...........] duration: 18.769138842s trace: https://jaeger.dev.storj.io/trace/1efcdf5c2a183d8d FAIL (2025-11-22 15:05:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 934958555s) [noise=true , address=145.79.227.30:4762 , piece_id=4Z7CCNLZEYQAGJNUWGJ2USQSVW6QZUTCNFKC3JWRIOKV6VBVIYKQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.384231169 (318h54m50.93570163s) [noise=true , address=145.79.227.18:4007 , piece_id=KM5TO27LZXJZZAT2DB5JTP7R5LOPJHUU6T34RZI7HH7PRWPW4ZNQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.384687758 (318h54m50.93615822s) [noise=true , address=145.79.227.26:4654 , piece_id=VCRM5O6NHOJOKNQUG5TTZUT3JIDQOMNF2DD6H7KWUJFOQ3P75QGQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.385168865 (318h54m50.936639326s) [noise=true , address=77.90.31.18:4805 , piece_id=3EHB5IFBQRLXFBOVCT25RS4CIXVMSVOE7EKDV26DXOEY2NDJ2YGA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.385605395 (318h54m50.937075867s) [noise=true , address=145.79.227.32:4835 , piece_id=7GD4QUTGLIPOEU25FIDE7VDO3AUJ2RZDU4XKI476KNU6QTETC52A, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.386294879 (318h54m50.93776534s) [noise=true , address=145.79.227.21:4473 , piece_id=D5FC7PVECZTQR64BFVTV3L2HOXCITABN57LLNRSMVCSGQICRYKUQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.386958642 (318h54m50.938429103s) [noise=true , address=145.79.227.14:4837 , piece_id=NR427EUHKVSQ6HMMZCHL4LCBV6VRVTANZDXMX6JHQI2TTQN5R5VQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387097236 (318h54m50.938567687s) [noise=true , address=145.79.227.9:4365 , piece_id=DBW2L6GHAWIHOMOZOTQFQ57Y2MA6KP6SGUO6AJDDF65ERQ324VZQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387456781 (318h54m50.938927242s) [noise=true , address=145.79.227.19:4078 , piece_id=ER3KKYJ2MJQC3HRRBVULC3ING5GLGOHFVD2QBWF2MVAXF6ODMUUA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387755300 (318h54m50.939225761s) [noise=true , address=145.79.227.28:4725 , piece_id=HVMAX2UWJIF2UJQIJS5UBTO5TOIJI5DCDGGURZN4SOMRY3Z4OF4A, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.388354220 (318h54m50.939824681s) [noise=true , address=145.79.227.10:4402 , piece_id=ZU6A2O2R73PIVIDF5NOKMJZH4QX4LOCFSYOQLFOFAA5BEVQH52KA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.388862268 (318h54m50.940332729s) [noise=true , address=145.79.227.22:4509 , piece_id=WRHXVGJZQRTY6ZKMRU6IFYZSMLR6UATXFZOTCPRYKHERA2WKMGNA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.389411814 (318h54m50.940882275s) [noise=true , address=145.79.227.8:4330 , piece_id=LFTZWYQKW56VLMJPOGGLB6IJILPGJNEHIRIYD2F4J34ZH24PCSMQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.389753224 (318h54m50.941223675s) [noise=true , address=145.79.227.16:4909 , piece_id=3DAVIY3ETZVG5CPP5BQB7QL473J5HZG6EQ6BYZBIAN6TWZ5FGNHQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.391102172 (318h54m50.942572633s) [noise=true , address=45.91.116.7:4010 , piece_id=VZCYHOAR73DVCRDT5GBT65JI53NP27OEGA3VYLM66LPQR23T4MTA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:54.404583235 (318h54m52.956053696s) [upload=280863]: Next returned nil segment Nov 22 14:58:54.404601019 (318h54m52.956071481s) [upload=280863]: Waiting for error group managing segments... Nov 22 14:58:54.928222688 (318h54m53.479693149s) [noise=true , address=77.90.31.18:4805 , piece_id=3EHB5IFBQRLXFBOVCT25RS4CIXVMSVOE7EKDV26DXOEY2NDJ2YGA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:58:55.175979631 (318h54m53.727450092s) [noise=true , address=45.91.116.7:4010 , piece_id=VZCYHOAR73DVCRDT5GBT65JI53NP27OEGA3VYLM66LPQR23T4MTA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:03.279622498 (318h55m1.831092959s) [noise=true , address=145.79.227.14:4837 , piece_id=NR427EUHKVSQ6HMMZCHL4LCBV6VRVTANZDXMX6JHQI2TTQN5R5VQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:04.274652248 (318h55m2.826122709s) [noise=true , address=145.79.227.28:4725 , piece_id=HVMAX2UWJIF2UJQIJS5UBTO5TOIJI5DCDGGURZN4SOMRY3Z4OF4A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:05.089136230 (318h55m3.640606701s) [noise=true , address=145.79.227.22:4509 , piece_id=WRHXVGJZQRTY6ZKMRU6IFYZSMLR6UATXFZOTCPRYKHERA2WKMGNA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:05.587545699 (318h55m4.13901616s) [noise=true , address=145.79.227.32:4835 , piece_id=7GD4QUTGLIPOEU25FIDE7VDO3AUJ2RZDU4XKI476KNU6QTETC52A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:06.956814829 (318h55m5.50828528s) [noise=true , address=145.79.227.8:4330 , piece_id=LFTZWYQKW56VLMJPOGGLB6IJILPGJNEHIRIYD2F4J34ZH24PCSMQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:07.254777763 (318h55m5.806248224s) [noise=true , address=145.79.227.30:4762 , piece_id=4Z7CCNLZEYQAGJNUWGJ2USQSVW6QZUTCNFKC3JWRIOKV6VBVIYKQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:19.983388333 (318h55m18.534858794s) [noise=true , address=145.79.227.10:4402 , piece_id=ZU6A2O2R73PIVIDF5NOKMJZH4QX4LOCFSYOQLFOFAA5BEVQH52KA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection Nov 22 14:59:19.983536355 (318h55m18.535006816s) [noise=true , address=145.79.227.21:4473 , piece_id=D5FC7PVECZTQR64BFVTV3L2HOXCITABN57LLNRSMVCSGQICRYKUQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection Nov 22 14:59:19.983802211 (318h55m18.535272703s) [noise=true , address=145.79.227.18:4007 , piece_id=KM5TO27LZXJZZAT2DB5JTP7R5LOPJHUU6T34RZI7HH7PRWPW4ZNQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection Nov 22 14:59:19.983812000 (318h55m18.535282461s) [noise=true , address=145.79.227.9:4365 , piece_id=DBW2L6GHAWIHOMOZOTQFQ57Y2MA6KP6SGUO6AJDDF65ERQ324VZQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection Nov 22 14:59:19.983896611 (318h55m18.535367073s) [noise=true , address=145.79.227.19:4078 , piece_id=ER3KKYJ2MJQC3HRRBVULC3ING5GLGOHFVD2QBWF2MVAXF6ODMUUA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection; write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection Nov 22 14:59:19.984068268 (318h55m18.535538719s) [noise=true , address=145.79.227.26:4654 , piece_id=VCRM5O6NHOJOKNQUG5TTZUT3JIDQOMNF2DD6H7KWUJFOQ3P75QGQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: read tcp 188.166.225.170:36440->145.79.227.26:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984229324 (318h55m18.535699776s) [noise=true , address=145.79.227.8:4329 , piece_id=YJG3LYBDMPS4QQNLG24MS7ZWUM6EOI6EPDGVBGLO33CVEQQKGX6A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: read tcp 188.166.225.170:44902->145.79.227.8:4329: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984374972 (318h55m18.535845423s) [noise=true , address=145.79.227.16:4909 , piece_id=3DAVIY3ETZVG5CPP5BQB7QL473J5HZG6EQ6BYZBIAN6TWZ5FGNHQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: read tcp 188.166.225.170:59064->145.79.227.16:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984400911 (318h55m18.535871362s) [seg_pos={0 0}, upload=280863]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 22 14:59:20.539921660 (318h55m19.091392121s) [upload=280863]: Done uploading segments... Nov 22 14:59:20.539961818 (318h55m19.091432269s) [upload=280863]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 18.769171114s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 15 out of 60 is failed (threshold=6) [.X.......XX..X..XX..X.X.........XX.XX.................XX..X.] duration: 3.618298269s trace: https://jaeger.dev.storj.io/trace/cc91f835756ebe2 FAIL (2025-11-22 15:05:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.618321382s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.......X.X.....X.......X......X............X................] duration: 6.354325738s trace: https://jaeger.dev.storj.io/trace/1ee0c6bff9434222 FAIL (2025-11-22 15:05:10 UTC) orbiter warning: context deadline exceeded Duration: 6.354371635s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...........................XX.X.........X........X..........] duration: 13.202976595s trace: https://jaeger.dev.storj.io/trace/1efcdf5c2a183d8d FAIL (2025-11-22 15:05:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 934958555s) [noise=true , address=145.79.227.30:4762 , piece_id=4Z7CCNLZEYQAGJNUWGJ2USQSVW6QZUTCNFKC3JWRIOKV6VBVIYKQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.384231169 (318h54m50.93570163s) [noise=true , address=145.79.227.18:4007 , piece_id=KM5TO27LZXJZZAT2DB5JTP7R5LOPJHUU6T34RZI7HH7PRWPW4ZNQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.384687758 (318h54m50.93615822s) [noise=true , address=145.79.227.26:4654 , piece_id=VCRM5O6NHOJOKNQUG5TTZUT3JIDQOMNF2DD6H7KWUJFOQ3P75QGQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.385168865 (318h54m50.936639326s) [noise=true , address=77.90.31.18:4805 , piece_id=3EHB5IFBQRLXFBOVCT25RS4CIXVMSVOE7EKDV26DXOEY2NDJ2YGA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.385605395 (318h54m50.937075867s) [noise=true , address=145.79.227.32:4835 , piece_id=7GD4QUTGLIPOEU25FIDE7VDO3AUJ2RZDU4XKI476KNU6QTETC52A, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.386294879 (318h54m50.93776534s) [noise=true , address=145.79.227.21:4473 , piece_id=D5FC7PVECZTQR64BFVTV3L2HOXCITABN57LLNRSMVCSGQICRYKUQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.386958642 (318h54m50.938429103s) [noise=true , address=145.79.227.14:4837 , piece_id=NR427EUHKVSQ6HMMZCHL4LCBV6VRVTANZDXMX6JHQI2TTQN5R5VQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387097236 (318h54m50.938567687s) [noise=true , address=145.79.227.9:4365 , piece_id=DBW2L6GHAWIHOMOZOTQFQ57Y2MA6KP6SGUO6AJDDF65ERQ324VZQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387456781 (318h54m50.938927242s) [noise=true , address=145.79.227.19:4078 , piece_id=ER3KKYJ2MJQC3HRRBVULC3ING5GLGOHFVD2QBWF2MVAXF6ODMUUA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387755300 (318h54m50.939225761s) [noise=true , address=145.79.227.28:4725 , piece_id=HVMAX2UWJIF2UJQIJS5UBTO5TOIJI5DCDGGURZN4SOMRY3Z4OF4A, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.388354220 (318h54m50.939824681s) [noise=true , address=145.79.227.10:4402 , piece_id=ZU6A2O2R73PIVIDF5NOKMJZH4QX4LOCFSYOQLFOFAA5BEVQH52KA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.388862268 (318h54m50.940332729s) [noise=true , address=145.79.227.22:4509 , piece_id=WRHXVGJZQRTY6ZKMRU6IFYZSMLR6UATXFZOTCPRYKHERA2WKMGNA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.389411814 (318h54m50.940882275s) [noise=true , address=145.79.227.8:4330 , piece_id=LFTZWYQKW56VLMJPOGGLB6IJILPGJNEHIRIYD2F4J34ZH24PCSMQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.389753224 (318h54m50.941223675s) [noise=true , address=145.79.227.16:4909 , piece_id=3DAVIY3ETZVG5CPP5BQB7QL473J5HZG6EQ6BYZBIAN6TWZ5FGNHQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.391102172 (318h54m50.942572633s) [noise=true , address=45.91.116.7:4010 , piece_id=VZCYHOAR73DVCRDT5GBT65JI53NP27OEGA3VYLM66LPQR23T4MTA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:54.404583235 (318h54m52.956053696s) [upload=280863]: Next returned nil segment Nov 22 14:58:54.404601019 (318h54m52.956071481s) [upload=280863]: Waiting for error group managing segments... Nov 22 14:58:54.928222688 (318h54m53.479693149s) [noise=true , address=77.90.31.18:4805 , piece_id=3EHB5IFBQRLXFBOVCT25RS4CIXVMSVOE7EKDV26DXOEY2NDJ2YGA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:58:55.175979631 (318h54m53.727450092s) [noise=true , address=45.91.116.7:4010 , piece_id=VZCYHOAR73DVCRDT5GBT65JI53NP27OEGA3VYLM66LPQR23T4MTA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:03.279622498 (318h55m1.831092959s) [noise=true , address=145.79.227.14:4837 , piece_id=NR427EUHKVSQ6HMMZCHL4LCBV6VRVTANZDXMX6JHQI2TTQN5R5VQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:04.274652248 (318h55m2.826122709s) [noise=true , address=145.79.227.28:4725 , piece_id=HVMAX2UWJIF2UJQIJS5UBTO5TOIJI5DCDGGURZN4SOMRY3Z4OF4A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:05.089136230 (318h55m3.640606701s) [noise=true , address=145.79.227.22:4509 , piece_id=WRHXVGJZQRTY6ZKMRU6IFYZSMLR6UATXFZOTCPRYKHERA2WKMGNA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:05.587545699 (318h55m4.13901616s) [noise=true , address=145.79.227.32:4835 , piece_id=7GD4QUTGLIPOEU25FIDE7VDO3AUJ2RZDU4XKI476KNU6QTETC52A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:06.956814829 (318h55m5.50828528s) [noise=true , address=145.79.227.8:4330 , piece_id=LFTZWYQKW56VLMJPOGGLB6IJILPGJNEHIRIYD2F4J34ZH24PCSMQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:07.254777763 (318h55m5.806248224s) [noise=true , address=145.79.227.30:4762 , piece_id=4Z7CCNLZEYQAGJNUWGJ2USQSVW6QZUTCNFKC3JWRIOKV6VBVIYKQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:19.983388333 (318h55m18.534858794s) [noise=true , address=145.79.227.10:4402 , piece_id=ZU6A2O2R73PIVIDF5NOKMJZH4QX4LOCFSYOQLFOFAA5BEVQH52KA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection Nov 22 14:59:19.983536355 (318h55m18.535006816s) [noise=true , address=145.79.227.21:4473 , piece_id=D5FC7PVECZTQR64BFVTV3L2HOXCITABN57LLNRSMVCSGQICRYKUQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection Nov 22 14:59:19.983802211 (318h55m18.535272703s) [noise=true , address=145.79.227.18:4007 , piece_id=KM5TO27LZXJZZAT2DB5JTP7R5LOPJHUU6T34RZI7HH7PRWPW4ZNQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection Nov 22 14:59:19.983812000 (318h55m18.535282461s) [noise=true , address=145.79.227.9:4365 , piece_id=DBW2L6GHAWIHOMOZOTQFQ57Y2MA6KP6SGUO6AJDDF65ERQ324VZQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection Nov 22 14:59:19.983896611 (318h55m18.535367073s) [noise=true , address=145.79.227.19:4078 , piece_id=ER3KKYJ2MJQC3HRRBVULC3ING5GLGOHFVD2QBWF2MVAXF6ODMUUA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection; write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection Nov 22 14:59:19.984068268 (318h55m18.535538719s) [noise=true , address=145.79.227.26:4654 , piece_id=VCRM5O6NHOJOKNQUG5TTZUT3JIDQOMNF2DD6H7KWUJFOQ3P75QGQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: read tcp 188.166.225.170:36440->145.79.227.26:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984229324 (318h55m18.535699776s) [noise=true , address=145.79.227.8:4329 , piece_id=YJG3LYBDMPS4QQNLG24MS7ZWUM6EOI6EPDGVBGLO33CVEQQKGX6A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: read tcp 188.166.225.170:44902->145.79.227.8:4329: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984374972 (318h55m18.535845423s) [noise=true , address=145.79.227.16:4909 , piece_id=3DAVIY3ETZVG5CPP5BQB7QL473J5HZG6EQ6BYZBIAN6TWZ5FGNHQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: read tcp 188.166.225.170:59064->145.79.227.16:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984400911 (318h55m18.535871362s) [seg_pos={0 0}, upload=280863]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 22 14:59:20.539921660 (318h55m19.091392121s) [upload=280863]: Done uploading segments... Nov 22 14:59:20.539961818 (318h55m19.091432269s) [upload=280863]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 13.203013327s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [........X.X.....X.......X......X............X...............] duration: 27.534589817s trace: https://jaeger.dev.storj.io/trace/1ee0c6bff9434222 FAIL (2025-11-22 15:04:30 UTC) orbiter warning: context deadline exceeded Duration: 27.534626157s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [............................XX.X.........X........X.........] duration: 12.117593325s trace: https://jaeger.dev.storj.io/trace/1efcdf5c2a183d8d FAIL (2025-11-22 15:04:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 934958555s) [noise=true , address=145.79.227.30:4762 , piece_id=4Z7CCNLZEYQAGJNUWGJ2USQSVW6QZUTCNFKC3JWRIOKV6VBVIYKQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.384231169 (318h54m50.93570163s) [noise=true , address=145.79.227.18:4007 , piece_id=KM5TO27LZXJZZAT2DB5JTP7R5LOPJHUU6T34RZI7HH7PRWPW4ZNQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.384687758 (318h54m50.93615822s) [noise=true , address=145.79.227.26:4654 , piece_id=VCRM5O6NHOJOKNQUG5TTZUT3JIDQOMNF2DD6H7KWUJFOQ3P75QGQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.385168865 (318h54m50.936639326s) [noise=true , address=77.90.31.18:4805 , piece_id=3EHB5IFBQRLXFBOVCT25RS4CIXVMSVOE7EKDV26DXOEY2NDJ2YGA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.385605395 (318h54m50.937075867s) [noise=true , address=145.79.227.32:4835 , piece_id=7GD4QUTGLIPOEU25FIDE7VDO3AUJ2RZDU4XKI476KNU6QTETC52A, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.386294879 (318h54m50.93776534s) [noise=true , address=145.79.227.21:4473 , piece_id=D5FC7PVECZTQR64BFVTV3L2HOXCITABN57LLNRSMVCSGQICRYKUQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.386958642 (318h54m50.938429103s) [noise=true , address=145.79.227.14:4837 , piece_id=NR427EUHKVSQ6HMMZCHL4LCBV6VRVTANZDXMX6JHQI2TTQN5R5VQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387097236 (318h54m50.938567687s) [noise=true , address=145.79.227.9:4365 , piece_id=DBW2L6GHAWIHOMOZOTQFQ57Y2MA6KP6SGUO6AJDDF65ERQ324VZQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387456781 (318h54m50.938927242s) [noise=true , address=145.79.227.19:4078 , piece_id=ER3KKYJ2MJQC3HRRBVULC3ING5GLGOHFVD2QBWF2MVAXF6ODMUUA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387755300 (318h54m50.939225761s) [noise=true , address=145.79.227.28:4725 , piece_id=HVMAX2UWJIF2UJQIJS5UBTO5TOIJI5DCDGGURZN4SOMRY3Z4OF4A, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.388354220 (318h54m50.939824681s) [noise=true , address=145.79.227.10:4402 , piece_id=ZU6A2O2R73PIVIDF5NOKMJZH4QX4LOCFSYOQLFOFAA5BEVQH52KA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.388862268 (318h54m50.940332729s) [noise=true , address=145.79.227.22:4509 , piece_id=WRHXVGJZQRTY6ZKMRU6IFYZSMLR6UATXFZOTCPRYKHERA2WKMGNA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.389411814 (318h54m50.940882275s) [noise=true , address=145.79.227.8:4330 , piece_id=LFTZWYQKW56VLMJPOGGLB6IJILPGJNEHIRIYD2F4J34ZH24PCSMQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.389753224 (318h54m50.941223675s) [noise=true , address=145.79.227.16:4909 , piece_id=3DAVIY3ETZVG5CPP5BQB7QL473J5HZG6EQ6BYZBIAN6TWZ5FGNHQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.391102172 (318h54m50.942572633s) [noise=true , address=45.91.116.7:4010 , piece_id=VZCYHOAR73DVCRDT5GBT65JI53NP27OEGA3VYLM66LPQR23T4MTA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:54.404583235 (318h54m52.956053696s) [upload=280863]: Next returned nil segment Nov 22 14:58:54.404601019 (318h54m52.956071481s) [upload=280863]: Waiting for error group managing segments... Nov 22 14:58:54.928222688 (318h54m53.479693149s) [noise=true , address=77.90.31.18:4805 , piece_id=3EHB5IFBQRLXFBOVCT25RS4CIXVMSVOE7EKDV26DXOEY2NDJ2YGA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:58:55.175979631 (318h54m53.727450092s) [noise=true , address=45.91.116.7:4010 , piece_id=VZCYHOAR73DVCRDT5GBT65JI53NP27OEGA3VYLM66LPQR23T4MTA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:03.279622498 (318h55m1.831092959s) [noise=true , address=145.79.227.14:4837 , piece_id=NR427EUHKVSQ6HMMZCHL4LCBV6VRVTANZDXMX6JHQI2TTQN5R5VQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:04.274652248 (318h55m2.826122709s) [noise=true , address=145.79.227.28:4725 , piece_id=HVMAX2UWJIF2UJQIJS5UBTO5TOIJI5DCDGGURZN4SOMRY3Z4OF4A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:05.089136230 (318h55m3.640606701s) [noise=true , address=145.79.227.22:4509 , piece_id=WRHXVGJZQRTY6ZKMRU6IFYZSMLR6UATXFZOTCPRYKHERA2WKMGNA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:05.587545699 (318h55m4.13901616s) [noise=true , address=145.79.227.32:4835 , piece_id=7GD4QUTGLIPOEU25FIDE7VDO3AUJ2RZDU4XKI476KNU6QTETC52A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:06.956814829 (318h55m5.50828528s) [noise=true , address=145.79.227.8:4330 , piece_id=LFTZWYQKW56VLMJPOGGLB6IJILPGJNEHIRIYD2F4J34ZH24PCSMQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:07.254777763 (318h55m5.806248224s) [noise=true , address=145.79.227.30:4762 , piece_id=4Z7CCNLZEYQAGJNUWGJ2USQSVW6QZUTCNFKC3JWRIOKV6VBVIYKQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:19.983388333 (318h55m18.534858794s) [noise=true , address=145.79.227.10:4402 , piece_id=ZU6A2O2R73PIVIDF5NOKMJZH4QX4LOCFSYOQLFOFAA5BEVQH52KA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection Nov 22 14:59:19.983536355 (318h55m18.535006816s) [noise=true , address=145.79.227.21:4473 , piece_id=D5FC7PVECZTQR64BFVTV3L2HOXCITABN57LLNRSMVCSGQICRYKUQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection Nov 22 14:59:19.983802211 (318h55m18.535272703s) [noise=true , address=145.79.227.18:4007 , piece_id=KM5TO27LZXJZZAT2DB5JTP7R5LOPJHUU6T34RZI7HH7PRWPW4ZNQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection Nov 22 14:59:19.983812000 (318h55m18.535282461s) [noise=true , address=145.79.227.9:4365 , piece_id=DBW2L6GHAWIHOMOZOTQFQ57Y2MA6KP6SGUO6AJDDF65ERQ324VZQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection Nov 22 14:59:19.983896611 (318h55m18.535367073s) [noise=true , address=145.79.227.19:4078 , piece_id=ER3KKYJ2MJQC3HRRBVULC3ING5GLGOHFVD2QBWF2MVAXF6ODMUUA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection; write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection Nov 22 14:59:19.984068268 (318h55m18.535538719s) [noise=true , address=145.79.227.26:4654 , piece_id=VCRM5O6NHOJOKNQUG5TTZUT3JIDQOMNF2DD6H7KWUJFOQ3P75QGQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: read tcp 188.166.225.170:36440->145.79.227.26:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984229324 (318h55m18.535699776s) [noise=true , address=145.79.227.8:4329 , piece_id=YJG3LYBDMPS4QQNLG24MS7ZWUM6EOI6EPDGVBGLO33CVEQQKGX6A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: read tcp 188.166.225.170:44902->145.79.227.8:4329: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984374972 (318h55m18.535845423s) [noise=true , address=145.79.227.16:4909 , piece_id=3DAVIY3ETZVG5CPP5BQB7QL473J5HZG6EQ6BYZBIAN6TWZ5FGNHQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: read tcp 188.166.225.170:59064->145.79.227.16:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984400911 (318h55m18.535871362s) [seg_pos={0 0}, upload=280863]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 22 14:59:20.539921660 (318h55m19.091392121s) [upload=280863]: Done uploading segments... Nov 22 14:59:20.539961818 (318h55m19.091432269s) [upload=280863]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 12.117650515s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 15 out of 60 is failed (threshold=6) [..X.......XX..X..XX..X.X.........XX.XX.................XX..X] duration: 5.001423292s trace: https://jaeger.dev.storj.io/trace/cc91f835756ebe2 FAIL (2025-11-22 15:04:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001463229s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.............................XX.X.........X........X........] duration: 20.029297466s trace: https://jaeger.dev.storj.io/trace/1efcdf5c2a183d8d FAIL (2025-11-22 15:04:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 934958555s) [noise=true , address=145.79.227.30:4762 , piece_id=4Z7CCNLZEYQAGJNUWGJ2USQSVW6QZUTCNFKC3JWRIOKV6VBVIYKQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.384231169 (318h54m50.93570163s) [noise=true , address=145.79.227.18:4007 , piece_id=KM5TO27LZXJZZAT2DB5JTP7R5LOPJHUU6T34RZI7HH7PRWPW4ZNQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.384687758 (318h54m50.93615822s) [noise=true , address=145.79.227.26:4654 , piece_id=VCRM5O6NHOJOKNQUG5TTZUT3JIDQOMNF2DD6H7KWUJFOQ3P75QGQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.385168865 (318h54m50.936639326s) [noise=true , address=77.90.31.18:4805 , piece_id=3EHB5IFBQRLXFBOVCT25RS4CIXVMSVOE7EKDV26DXOEY2NDJ2YGA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.385605395 (318h54m50.937075867s) [noise=true , address=145.79.227.32:4835 , piece_id=7GD4QUTGLIPOEU25FIDE7VDO3AUJ2RZDU4XKI476KNU6QTETC52A, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.386294879 (318h54m50.93776534s) [noise=true , address=145.79.227.21:4473 , piece_id=D5FC7PVECZTQR64BFVTV3L2HOXCITABN57LLNRSMVCSGQICRYKUQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.386958642 (318h54m50.938429103s) [noise=true , address=145.79.227.14:4837 , piece_id=NR427EUHKVSQ6HMMZCHL4LCBV6VRVTANZDXMX6JHQI2TTQN5R5VQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387097236 (318h54m50.938567687s) [noise=true , address=145.79.227.9:4365 , piece_id=DBW2L6GHAWIHOMOZOTQFQ57Y2MA6KP6SGUO6AJDDF65ERQ324VZQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387456781 (318h54m50.938927242s) [noise=true , address=145.79.227.19:4078 , piece_id=ER3KKYJ2MJQC3HRRBVULC3ING5GLGOHFVD2QBWF2MVAXF6ODMUUA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387755300 (318h54m50.939225761s) [noise=true , address=145.79.227.28:4725 , piece_id=HVMAX2UWJIF2UJQIJS5UBTO5TOIJI5DCDGGURZN4SOMRY3Z4OF4A, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.388354220 (318h54m50.939824681s) [noise=true , address=145.79.227.10:4402 , piece_id=ZU6A2O2R73PIVIDF5NOKMJZH4QX4LOCFSYOQLFOFAA5BEVQH52KA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.388862268 (318h54m50.940332729s) [noise=true , address=145.79.227.22:4509 , piece_id=WRHXVGJZQRTY6ZKMRU6IFYZSMLR6UATXFZOTCPRYKHERA2WKMGNA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.389411814 (318h54m50.940882275s) [noise=true , address=145.79.227.8:4330 , piece_id=LFTZWYQKW56VLMJPOGGLB6IJILPGJNEHIRIYD2F4J34ZH24PCSMQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.389753224 (318h54m50.941223675s) [noise=true , address=145.79.227.16:4909 , piece_id=3DAVIY3ETZVG5CPP5BQB7QL473J5HZG6EQ6BYZBIAN6TWZ5FGNHQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.391102172 (318h54m50.942572633s) [noise=true , address=45.91.116.7:4010 , piece_id=VZCYHOAR73DVCRDT5GBT65JI53NP27OEGA3VYLM66LPQR23T4MTA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:54.404583235 (318h54m52.956053696s) [upload=280863]: Next returned nil segment Nov 22 14:58:54.404601019 (318h54m52.956071481s) [upload=280863]: Waiting for error group managing segments... Nov 22 14:58:54.928222688 (318h54m53.479693149s) [noise=true , address=77.90.31.18:4805 , piece_id=3EHB5IFBQRLXFBOVCT25RS4CIXVMSVOE7EKDV26DXOEY2NDJ2YGA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:58:55.175979631 (318h54m53.727450092s) [noise=true , address=45.91.116.7:4010 , piece_id=VZCYHOAR73DVCRDT5GBT65JI53NP27OEGA3VYLM66LPQR23T4MTA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:03.279622498 (318h55m1.831092959s) [noise=true , address=145.79.227.14:4837 , piece_id=NR427EUHKVSQ6HMMZCHL4LCBV6VRVTANZDXMX6JHQI2TTQN5R5VQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:04.274652248 (318h55m2.826122709s) [noise=true , address=145.79.227.28:4725 , piece_id=HVMAX2UWJIF2UJQIJS5UBTO5TOIJI5DCDGGURZN4SOMRY3Z4OF4A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:05.089136230 (318h55m3.640606701s) [noise=true , address=145.79.227.22:4509 , piece_id=WRHXVGJZQRTY6ZKMRU6IFYZSMLR6UATXFZOTCPRYKHERA2WKMGNA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:05.587545699 (318h55m4.13901616s) [noise=true , address=145.79.227.32:4835 , piece_id=7GD4QUTGLIPOEU25FIDE7VDO3AUJ2RZDU4XKI476KNU6QTETC52A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:06.956814829 (318h55m5.50828528s) [noise=true , address=145.79.227.8:4330 , piece_id=LFTZWYQKW56VLMJPOGGLB6IJILPGJNEHIRIYD2F4J34ZH24PCSMQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:07.254777763 (318h55m5.806248224s) [noise=true , address=145.79.227.30:4762 , piece_id=4Z7CCNLZEYQAGJNUWGJ2USQSVW6QZUTCNFKC3JWRIOKV6VBVIYKQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:19.983388333 (318h55m18.534858794s) [noise=true , address=145.79.227.10:4402 , piece_id=ZU6A2O2R73PIVIDF5NOKMJZH4QX4LOCFSYOQLFOFAA5BEVQH52KA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection Nov 22 14:59:19.983536355 (318h55m18.535006816s) [noise=true , address=145.79.227.21:4473 , piece_id=D5FC7PVECZTQR64BFVTV3L2HOXCITABN57LLNRSMVCSGQICRYKUQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection Nov 22 14:59:19.983802211 (318h55m18.535272703s) [noise=true , address=145.79.227.18:4007 , piece_id=KM5TO27LZXJZZAT2DB5JTP7R5LOPJHUU6T34RZI7HH7PRWPW4ZNQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection Nov 22 14:59:19.983812000 (318h55m18.535282461s) [noise=true , address=145.79.227.9:4365 , piece_id=DBW2L6GHAWIHOMOZOTQFQ57Y2MA6KP6SGUO6AJDDF65ERQ324VZQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection Nov 22 14:59:19.983896611 (318h55m18.535367073s) [noise=true , address=145.79.227.19:4078 , piece_id=ER3KKYJ2MJQC3HRRBVULC3ING5GLGOHFVD2QBWF2MVAXF6ODMUUA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection; write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection Nov 22 14:59:19.984068268 (318h55m18.535538719s) [noise=true , address=145.79.227.26:4654 , piece_id=VCRM5O6NHOJOKNQUG5TTZUT3JIDQOMNF2DD6H7KWUJFOQ3P75QGQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: read tcp 188.166.225.170:36440->145.79.227.26:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984229324 (318h55m18.535699776s) [noise=true , address=145.79.227.8:4329 , piece_id=YJG3LYBDMPS4QQNLG24MS7ZWUM6EOI6EPDGVBGLO33CVEQQKGX6A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: read tcp 188.166.225.170:44902->145.79.227.8:4329: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984374972 (318h55m18.535845423s) [noise=true , address=145.79.227.16:4909 , piece_id=3DAVIY3ETZVG5CPP5BQB7QL473J5HZG6EQ6BYZBIAN6TWZ5FGNHQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: read tcp 188.166.225.170:59064->145.79.227.16:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984400911 (318h55m18.535871362s) [seg_pos={0 0}, upload=280863]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 22 14:59:20.539921660 (318h55m19.091392121s) [upload=280863]: Done uploading segments... Nov 22 14:59:20.539961818 (318h55m19.091432269s) [upload=280863]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 20.02933115s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [...X.......XX..X..XX..X.X.........XX.XX.................XX..] duration: 3.580152402s trace: https://jaeger.dev.storj.io/trace/33e61458f8570628 FAIL (2025-11-22 15:03:27 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.58017761s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..............................XX.X.........X........X.......] duration: 15.235520709s trace: https://jaeger.dev.storj.io/trace/1efcdf5c2a183d8d FAIL (2025-11-22 15:03:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 934958555s) [noise=true , address=145.79.227.30:4762 , piece_id=4Z7CCNLZEYQAGJNUWGJ2USQSVW6QZUTCNFKC3JWRIOKV6VBVIYKQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.384231169 (318h54m50.93570163s) [noise=true , address=145.79.227.18:4007 , piece_id=KM5TO27LZXJZZAT2DB5JTP7R5LOPJHUU6T34RZI7HH7PRWPW4ZNQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.384687758 (318h54m50.93615822s) [noise=true , address=145.79.227.26:4654 , piece_id=VCRM5O6NHOJOKNQUG5TTZUT3JIDQOMNF2DD6H7KWUJFOQ3P75QGQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.385168865 (318h54m50.936639326s) [noise=true , address=77.90.31.18:4805 , piece_id=3EHB5IFBQRLXFBOVCT25RS4CIXVMSVOE7EKDV26DXOEY2NDJ2YGA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.385605395 (318h54m50.937075867s) [noise=true , address=145.79.227.32:4835 , piece_id=7GD4QUTGLIPOEU25FIDE7VDO3AUJ2RZDU4XKI476KNU6QTETC52A, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.386294879 (318h54m50.93776534s) [noise=true , address=145.79.227.21:4473 , piece_id=D5FC7PVECZTQR64BFVTV3L2HOXCITABN57LLNRSMVCSGQICRYKUQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.386958642 (318h54m50.938429103s) [noise=true , address=145.79.227.14:4837 , piece_id=NR427EUHKVSQ6HMMZCHL4LCBV6VRVTANZDXMX6JHQI2TTQN5R5VQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387097236 (318h54m50.938567687s) [noise=true , address=145.79.227.9:4365 , piece_id=DBW2L6GHAWIHOMOZOTQFQ57Y2MA6KP6SGUO6AJDDF65ERQ324VZQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387456781 (318h54m50.938927242s) [noise=true , address=145.79.227.19:4078 , piece_id=ER3KKYJ2MJQC3HRRBVULC3ING5GLGOHFVD2QBWF2MVAXF6ODMUUA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387755300 (318h54m50.939225761s) [noise=true , address=145.79.227.28:4725 , piece_id=HVMAX2UWJIF2UJQIJS5UBTO5TOIJI5DCDGGURZN4SOMRY3Z4OF4A, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.388354220 (318h54m50.939824681s) [noise=true , address=145.79.227.10:4402 , piece_id=ZU6A2O2R73PIVIDF5NOKMJZH4QX4LOCFSYOQLFOFAA5BEVQH52KA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.388862268 (318h54m50.940332729s) [noise=true , address=145.79.227.22:4509 , piece_id=WRHXVGJZQRTY6ZKMRU6IFYZSMLR6UATXFZOTCPRYKHERA2WKMGNA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.389411814 (318h54m50.940882275s) [noise=true , address=145.79.227.8:4330 , piece_id=LFTZWYQKW56VLMJPOGGLB6IJILPGJNEHIRIYD2F4J34ZH24PCSMQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.389753224 (318h54m50.941223675s) [noise=true , address=145.79.227.16:4909 , piece_id=3DAVIY3ETZVG5CPP5BQB7QL473J5HZG6EQ6BYZBIAN6TWZ5FGNHQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.391102172 (318h54m50.942572633s) [noise=true , address=45.91.116.7:4010 , piece_id=VZCYHOAR73DVCRDT5GBT65JI53NP27OEGA3VYLM66LPQR23T4MTA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:54.404583235 (318h54m52.956053696s) [upload=280863]: Next returned nil segment Nov 22 14:58:54.404601019 (318h54m52.956071481s) [upload=280863]: Waiting for error group managing segments... Nov 22 14:58:54.928222688 (318h54m53.479693149s) [noise=true , address=77.90.31.18:4805 , piece_id=3EHB5IFBQRLXFBOVCT25RS4CIXVMSVOE7EKDV26DXOEY2NDJ2YGA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:58:55.175979631 (318h54m53.727450092s) [noise=true , address=45.91.116.7:4010 , piece_id=VZCYHOAR73DVCRDT5GBT65JI53NP27OEGA3VYLM66LPQR23T4MTA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:03.279622498 (318h55m1.831092959s) [noise=true , address=145.79.227.14:4837 , piece_id=NR427EUHKVSQ6HMMZCHL4LCBV6VRVTANZDXMX6JHQI2TTQN5R5VQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:04.274652248 (318h55m2.826122709s) [noise=true , address=145.79.227.28:4725 , piece_id=HVMAX2UWJIF2UJQIJS5UBTO5TOIJI5DCDGGURZN4SOMRY3Z4OF4A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:05.089136230 (318h55m3.640606701s) [noise=true , address=145.79.227.22:4509 , piece_id=WRHXVGJZQRTY6ZKMRU6IFYZSMLR6UATXFZOTCPRYKHERA2WKMGNA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:05.587545699 (318h55m4.13901616s) [noise=true , address=145.79.227.32:4835 , piece_id=7GD4QUTGLIPOEU25FIDE7VDO3AUJ2RZDU4XKI476KNU6QTETC52A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:06.956814829 (318h55m5.50828528s) [noise=true , address=145.79.227.8:4330 , piece_id=LFTZWYQKW56VLMJPOGGLB6IJILPGJNEHIRIYD2F4J34ZH24PCSMQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:07.254777763 (318h55m5.806248224s) [noise=true , address=145.79.227.30:4762 , piece_id=4Z7CCNLZEYQAGJNUWGJ2USQSVW6QZUTCNFKC3JWRIOKV6VBVIYKQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:19.983388333 (318h55m18.534858794s) [noise=true , address=145.79.227.10:4402 , piece_id=ZU6A2O2R73PIVIDF5NOKMJZH4QX4LOCFSYOQLFOFAA5BEVQH52KA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection Nov 22 14:59:19.983536355 (318h55m18.535006816s) [noise=true , address=145.79.227.21:4473 , piece_id=D5FC7PVECZTQR64BFVTV3L2HOXCITABN57LLNRSMVCSGQICRYKUQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection Nov 22 14:59:19.983802211 (318h55m18.535272703s) [noise=true , address=145.79.227.18:4007 , piece_id=KM5TO27LZXJZZAT2DB5JTP7R5LOPJHUU6T34RZI7HH7PRWPW4ZNQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection Nov 22 14:59:19.983812000 (318h55m18.535282461s) [noise=true , address=145.79.227.9:4365 , piece_id=DBW2L6GHAWIHOMOZOTQFQ57Y2MA6KP6SGUO6AJDDF65ERQ324VZQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection Nov 22 14:59:19.983896611 (318h55m18.535367073s) [noise=true , address=145.79.227.19:4078 , piece_id=ER3KKYJ2MJQC3HRRBVULC3ING5GLGOHFVD2QBWF2MVAXF6ODMUUA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection; write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection Nov 22 14:59:19.984068268 (318h55m18.535538719s) [noise=true , address=145.79.227.26:4654 , piece_id=VCRM5O6NHOJOKNQUG5TTZUT3JIDQOMNF2DD6H7KWUJFOQ3P75QGQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: read tcp 188.166.225.170:36440->145.79.227.26:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984229324 (318h55m18.535699776s) [noise=true , address=145.79.227.8:4329 , piece_id=YJG3LYBDMPS4QQNLG24MS7ZWUM6EOI6EPDGVBGLO33CVEQQKGX6A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: read tcp 188.166.225.170:44902->145.79.227.8:4329: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984374972 (318h55m18.535845423s) [noise=true , address=145.79.227.16:4909 , piece_id=3DAVIY3ETZVG5CPP5BQB7QL473J5HZG6EQ6BYZBIAN6TWZ5FGNHQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: read tcp 188.166.225.170:59064->145.79.227.16:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984400911 (318h55m18.535871362s) [seg_pos={0 0}, upload=280863]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 22 14:59:20.539921660 (318h55m19.091392121s) [upload=280863]: Done uploading segments... Nov 22 14:59:20.539961818 (318h55m19.091432269s) [upload=280863]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 15.235544054s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [....X.......XX..X..XX..X.X.........XX.XX.................XX.] duration: 3.58614386s trace: https://jaeger.dev.storj.io/trace/33e61458f8570628 FAIL (2025-11-22 15:02:56 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.586174038s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.........X.X.....X.......X......X............X..............] duration: 6.167414236s trace: https://jaeger.dev.storj.io/trace/1ee0c6bff9434222 FAIL (2025-11-22 15:02:55 UTC) orbiter warning: context deadline exceeded Duration: 6.167449582s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...............................XX.X.........X........X......] duration: 22.938848695s trace: https://jaeger.dev.storj.io/trace/1efcdf5c2a183d8d FAIL (2025-11-22 15:02:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 934958555s) [noise=true , address=145.79.227.30:4762 , piece_id=4Z7CCNLZEYQAGJNUWGJ2USQSVW6QZUTCNFKC3JWRIOKV6VBVIYKQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.384231169 (318h54m50.93570163s) [noise=true , address=145.79.227.18:4007 , piece_id=KM5TO27LZXJZZAT2DB5JTP7R5LOPJHUU6T34RZI7HH7PRWPW4ZNQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.384687758 (318h54m50.93615822s) [noise=true , address=145.79.227.26:4654 , piece_id=VCRM5O6NHOJOKNQUG5TTZUT3JIDQOMNF2DD6H7KWUJFOQ3P75QGQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.385168865 (318h54m50.936639326s) [noise=true , address=77.90.31.18:4805 , piece_id=3EHB5IFBQRLXFBOVCT25RS4CIXVMSVOE7EKDV26DXOEY2NDJ2YGA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.385605395 (318h54m50.937075867s) [noise=true , address=145.79.227.32:4835 , piece_id=7GD4QUTGLIPOEU25FIDE7VDO3AUJ2RZDU4XKI476KNU6QTETC52A, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.386294879 (318h54m50.93776534s) [noise=true , address=145.79.227.21:4473 , piece_id=D5FC7PVECZTQR64BFVTV3L2HOXCITABN57LLNRSMVCSGQICRYKUQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.386958642 (318h54m50.938429103s) [noise=true , address=145.79.227.14:4837 , piece_id=NR427EUHKVSQ6HMMZCHL4LCBV6VRVTANZDXMX6JHQI2TTQN5R5VQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387097236 (318h54m50.938567687s) [noise=true , address=145.79.227.9:4365 , piece_id=DBW2L6GHAWIHOMOZOTQFQ57Y2MA6KP6SGUO6AJDDF65ERQ324VZQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387456781 (318h54m50.938927242s) [noise=true , address=145.79.227.19:4078 , piece_id=ER3KKYJ2MJQC3HRRBVULC3ING5GLGOHFVD2QBWF2MVAXF6ODMUUA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387755300 (318h54m50.939225761s) [noise=true , address=145.79.227.28:4725 , piece_id=HVMAX2UWJIF2UJQIJS5UBTO5TOIJI5DCDGGURZN4SOMRY3Z4OF4A, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.388354220 (318h54m50.939824681s) [noise=true , address=145.79.227.10:4402 , piece_id=ZU6A2O2R73PIVIDF5NOKMJZH4QX4LOCFSYOQLFOFAA5BEVQH52KA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.388862268 (318h54m50.940332729s) [noise=true , address=145.79.227.22:4509 , piece_id=WRHXVGJZQRTY6ZKMRU6IFYZSMLR6UATXFZOTCPRYKHERA2WKMGNA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.389411814 (318h54m50.940882275s) [noise=true , address=145.79.227.8:4330 , piece_id=LFTZWYQKW56VLMJPOGGLB6IJILPGJNEHIRIYD2F4J34ZH24PCSMQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.389753224 (318h54m50.941223675s) [noise=true , address=145.79.227.16:4909 , piece_id=3DAVIY3ETZVG5CPP5BQB7QL473J5HZG6EQ6BYZBIAN6TWZ5FGNHQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.391102172 (318h54m50.942572633s) [noise=true , address=45.91.116.7:4010 , piece_id=VZCYHOAR73DVCRDT5GBT65JI53NP27OEGA3VYLM66LPQR23T4MTA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:54.404583235 (318h54m52.956053696s) [upload=280863]: Next returned nil segment Nov 22 14:58:54.404601019 (318h54m52.956071481s) [upload=280863]: Waiting for error group managing segments... Nov 22 14:58:54.928222688 (318h54m53.479693149s) [noise=true , address=77.90.31.18:4805 , piece_id=3EHB5IFBQRLXFBOVCT25RS4CIXVMSVOE7EKDV26DXOEY2NDJ2YGA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:58:55.175979631 (318h54m53.727450092s) [noise=true , address=45.91.116.7:4010 , piece_id=VZCYHOAR73DVCRDT5GBT65JI53NP27OEGA3VYLM66LPQR23T4MTA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:03.279622498 (318h55m1.831092959s) [noise=true , address=145.79.227.14:4837 , piece_id=NR427EUHKVSQ6HMMZCHL4LCBV6VRVTANZDXMX6JHQI2TTQN5R5VQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:04.274652248 (318h55m2.826122709s) [noise=true , address=145.79.227.28:4725 , piece_id=HVMAX2UWJIF2UJQIJS5UBTO5TOIJI5DCDGGURZN4SOMRY3Z4OF4A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:05.089136230 (318h55m3.640606701s) [noise=true , address=145.79.227.22:4509 , piece_id=WRHXVGJZQRTY6ZKMRU6IFYZSMLR6UATXFZOTCPRYKHERA2WKMGNA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:05.587545699 (318h55m4.13901616s) [noise=true , address=145.79.227.32:4835 , piece_id=7GD4QUTGLIPOEU25FIDE7VDO3AUJ2RZDU4XKI476KNU6QTETC52A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:06.956814829 (318h55m5.50828528s) [noise=true , address=145.79.227.8:4330 , piece_id=LFTZWYQKW56VLMJPOGGLB6IJILPGJNEHIRIYD2F4J34ZH24PCSMQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:07.254777763 (318h55m5.806248224s) [noise=true , address=145.79.227.30:4762 , piece_id=4Z7CCNLZEYQAGJNUWGJ2USQSVW6QZUTCNFKC3JWRIOKV6VBVIYKQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:19.983388333 (318h55m18.534858794s) [noise=true , address=145.79.227.10:4402 , piece_id=ZU6A2O2R73PIVIDF5NOKMJZH4QX4LOCFSYOQLFOFAA5BEVQH52KA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection Nov 22 14:59:19.983536355 (318h55m18.535006816s) [noise=true , address=145.79.227.21:4473 , piece_id=D5FC7PVECZTQR64BFVTV3L2HOXCITABN57LLNRSMVCSGQICRYKUQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection Nov 22 14:59:19.983802211 (318h55m18.535272703s) [noise=true , address=145.79.227.18:4007 , piece_id=KM5TO27LZXJZZAT2DB5JTP7R5LOPJHUU6T34RZI7HH7PRWPW4ZNQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection Nov 22 14:59:19.983812000 (318h55m18.535282461s) [noise=true , address=145.79.227.9:4365 , piece_id=DBW2L6GHAWIHOMOZOTQFQ57Y2MA6KP6SGUO6AJDDF65ERQ324VZQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection Nov 22 14:59:19.983896611 (318h55m18.535367073s) [noise=true , address=145.79.227.19:4078 , piece_id=ER3KKYJ2MJQC3HRRBVULC3ING5GLGOHFVD2QBWF2MVAXF6ODMUUA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection; write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection Nov 22 14:59:19.984068268 (318h55m18.535538719s) [noise=true , address=145.79.227.26:4654 , piece_id=VCRM5O6NHOJOKNQUG5TTZUT3JIDQOMNF2DD6H7KWUJFOQ3P75QGQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: read tcp 188.166.225.170:36440->145.79.227.26:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984229324 (318h55m18.535699776s) [noise=true , address=145.79.227.8:4329 , piece_id=YJG3LYBDMPS4QQNLG24MS7ZWUM6EOI6EPDGVBGLO33CVEQQKGX6A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: read tcp 188.166.225.170:44902->145.79.227.8:4329: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984374972 (318h55m18.535845423s) [noise=true , address=145.79.227.16:4909 , piece_id=3DAVIY3ETZVG5CPP5BQB7QL473J5HZG6EQ6BYZBIAN6TWZ5FGNHQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: read tcp 188.166.225.170:59064->145.79.227.16:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984400911 (318h55m18.535871362s) [seg_pos={0 0}, upload=280863]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 22 14:59:20.539921660 (318h55m19.091392121s) [upload=280863]: Done uploading segments... Nov 22 14:59:20.539961818 (318h55m19.091432269s) [upload=280863]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 22.938878981s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [.....X.......XX..X..XX..X.X.........XX.XX.................XX] duration: 5.001625978s trace: https://jaeger.dev.storj.io/trace/33e61458f8570628 FAIL (2025-11-22 15:02:27 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001649173s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [................................XX.X.........X........X.....] duration: 14.929424748s trace: https://jaeger.dev.storj.io/trace/1efcdf5c2a183d8d FAIL (2025-11-22 15:02:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 934958555s) [noise=true , address=145.79.227.30:4762 , piece_id=4Z7CCNLZEYQAGJNUWGJ2USQSVW6QZUTCNFKC3JWRIOKV6VBVIYKQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.384231169 (318h54m50.93570163s) [noise=true , address=145.79.227.18:4007 , piece_id=KM5TO27LZXJZZAT2DB5JTP7R5LOPJHUU6T34RZI7HH7PRWPW4ZNQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.384687758 (318h54m50.93615822s) [noise=true , address=145.79.227.26:4654 , piece_id=VCRM5O6NHOJOKNQUG5TTZUT3JIDQOMNF2DD6H7KWUJFOQ3P75QGQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.385168865 (318h54m50.936639326s) [noise=true , address=77.90.31.18:4805 , piece_id=3EHB5IFBQRLXFBOVCT25RS4CIXVMSVOE7EKDV26DXOEY2NDJ2YGA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.385605395 (318h54m50.937075867s) [noise=true , address=145.79.227.32:4835 , piece_id=7GD4QUTGLIPOEU25FIDE7VDO3AUJ2RZDU4XKI476KNU6QTETC52A, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.386294879 (318h54m50.93776534s) [noise=true , address=145.79.227.21:4473 , piece_id=D5FC7PVECZTQR64BFVTV3L2HOXCITABN57LLNRSMVCSGQICRYKUQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.386958642 (318h54m50.938429103s) [noise=true , address=145.79.227.14:4837 , piece_id=NR427EUHKVSQ6HMMZCHL4LCBV6VRVTANZDXMX6JHQI2TTQN5R5VQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387097236 (318h54m50.938567687s) [noise=true , address=145.79.227.9:4365 , piece_id=DBW2L6GHAWIHOMOZOTQFQ57Y2MA6KP6SGUO6AJDDF65ERQ324VZQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387456781 (318h54m50.938927242s) [noise=true , address=145.79.227.19:4078 , piece_id=ER3KKYJ2MJQC3HRRBVULC3ING5GLGOHFVD2QBWF2MVAXF6ODMUUA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387755300 (318h54m50.939225761s) [noise=true , address=145.79.227.28:4725 , piece_id=HVMAX2UWJIF2UJQIJS5UBTO5TOIJI5DCDGGURZN4SOMRY3Z4OF4A, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.388354220 (318h54m50.939824681s) [noise=true , address=145.79.227.10:4402 , piece_id=ZU6A2O2R73PIVIDF5NOKMJZH4QX4LOCFSYOQLFOFAA5BEVQH52KA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.388862268 (318h54m50.940332729s) [noise=true , address=145.79.227.22:4509 , piece_id=WRHXVGJZQRTY6ZKMRU6IFYZSMLR6UATXFZOTCPRYKHERA2WKMGNA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.389411814 (318h54m50.940882275s) [noise=true , address=145.79.227.8:4330 , piece_id=LFTZWYQKW56VLMJPOGGLB6IJILPGJNEHIRIYD2F4J34ZH24PCSMQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.389753224 (318h54m50.941223675s) [noise=true , address=145.79.227.16:4909 , piece_id=3DAVIY3ETZVG5CPP5BQB7QL473J5HZG6EQ6BYZBIAN6TWZ5FGNHQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.391102172 (318h54m50.942572633s) [noise=true , address=45.91.116.7:4010 , piece_id=VZCYHOAR73DVCRDT5GBT65JI53NP27OEGA3VYLM66LPQR23T4MTA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:54.404583235 (318h54m52.956053696s) [upload=280863]: Next returned nil segment Nov 22 14:58:54.404601019 (318h54m52.956071481s) [upload=280863]: Waiting for error group managing segments... Nov 22 14:58:54.928222688 (318h54m53.479693149s) [noise=true , address=77.90.31.18:4805 , piece_id=3EHB5IFBQRLXFBOVCT25RS4CIXVMSVOE7EKDV26DXOEY2NDJ2YGA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:58:55.175979631 (318h54m53.727450092s) [noise=true , address=45.91.116.7:4010 , piece_id=VZCYHOAR73DVCRDT5GBT65JI53NP27OEGA3VYLM66LPQR23T4MTA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:03.279622498 (318h55m1.831092959s) [noise=true , address=145.79.227.14:4837 , piece_id=NR427EUHKVSQ6HMMZCHL4LCBV6VRVTANZDXMX6JHQI2TTQN5R5VQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:04.274652248 (318h55m2.826122709s) [noise=true , address=145.79.227.28:4725 , piece_id=HVMAX2UWJIF2UJQIJS5UBTO5TOIJI5DCDGGURZN4SOMRY3Z4OF4A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:05.089136230 (318h55m3.640606701s) [noise=true , address=145.79.227.22:4509 , piece_id=WRHXVGJZQRTY6ZKMRU6IFYZSMLR6UATXFZOTCPRYKHERA2WKMGNA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:05.587545699 (318h55m4.13901616s) [noise=true , address=145.79.227.32:4835 , piece_id=7GD4QUTGLIPOEU25FIDE7VDO3AUJ2RZDU4XKI476KNU6QTETC52A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:06.956814829 (318h55m5.50828528s) [noise=true , address=145.79.227.8:4330 , piece_id=LFTZWYQKW56VLMJPOGGLB6IJILPGJNEHIRIYD2F4J34ZH24PCSMQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:07.254777763 (318h55m5.806248224s) [noise=true , address=145.79.227.30:4762 , piece_id=4Z7CCNLZEYQAGJNUWGJ2USQSVW6QZUTCNFKC3JWRIOKV6VBVIYKQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:19.983388333 (318h55m18.534858794s) [noise=true , address=145.79.227.10:4402 , piece_id=ZU6A2O2R73PIVIDF5NOKMJZH4QX4LOCFSYOQLFOFAA5BEVQH52KA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection Nov 22 14:59:19.983536355 (318h55m18.535006816s) [noise=true , address=145.79.227.21:4473 , piece_id=D5FC7PVECZTQR64BFVTV3L2HOXCITABN57LLNRSMVCSGQICRYKUQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection Nov 22 14:59:19.983802211 (318h55m18.535272703s) [noise=true , address=145.79.227.18:4007 , piece_id=KM5TO27LZXJZZAT2DB5JTP7R5LOPJHUU6T34RZI7HH7PRWPW4ZNQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection Nov 22 14:59:19.983812000 (318h55m18.535282461s) [noise=true , address=145.79.227.9:4365 , piece_id=DBW2L6GHAWIHOMOZOTQFQ57Y2MA6KP6SGUO6AJDDF65ERQ324VZQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection Nov 22 14:59:19.983896611 (318h55m18.535367073s) [noise=true , address=145.79.227.19:4078 , piece_id=ER3KKYJ2MJQC3HRRBVULC3ING5GLGOHFVD2QBWF2MVAXF6ODMUUA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection; write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection Nov 22 14:59:19.984068268 (318h55m18.535538719s) [noise=true , address=145.79.227.26:4654 , piece_id=VCRM5O6NHOJOKNQUG5TTZUT3JIDQOMNF2DD6H7KWUJFOQ3P75QGQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: read tcp 188.166.225.170:36440->145.79.227.26:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984229324 (318h55m18.535699776s) [noise=true , address=145.79.227.8:4329 , piece_id=YJG3LYBDMPS4QQNLG24MS7ZWUM6EOI6EPDGVBGLO33CVEQQKGX6A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: read tcp 188.166.225.170:44902->145.79.227.8:4329: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984374972 (318h55m18.535845423s) [noise=true , address=145.79.227.16:4909 , piece_id=3DAVIY3ETZVG5CPP5BQB7QL473J5HZG6EQ6BYZBIAN6TWZ5FGNHQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: read tcp 188.166.225.170:59064->145.79.227.16:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984400911 (318h55m18.535871362s) [seg_pos={0 0}, upload=280863]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 22 14:59:20.539921660 (318h55m19.091392121s) [upload=280863]: Done uploading segments... Nov 22 14:59:20.539961818 (318h55m19.091432269s) [upload=280863]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 14.929447693s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [X.........X.X.....X.......X......X............X.............] duration: 7.138339628s trace: https://jaeger.dev.storj.io/trace/1ee0c6bff9434222 FAIL (2025-11-22 15:01:53 UTC) orbiter warning: context deadline exceeded Duration: 7.138367612s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [......X.......XX..X..XX..X.X.........XX.XX.................X] duration: 5.003882764s trace: https://jaeger.dev.storj.io/trace/19d4a4fb18f421aa FAIL (2025-11-22 15:01:46 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.003907412s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.X.........X.X.....X.......X......X............X............] duration: 6.804464391s trace: https://jaeger.dev.storj.io/trace/1ee0c6bff9434222 FAIL (2025-11-22 15:01:26 UTC) orbiter warning: context deadline exceeded Duration: 6.804492755s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [.......X.......XX..X..XX..X.X.........XX.XX.................] duration: 3.560063318s trace: https://jaeger.dev.storj.io/trace/58dc5585e5ee8756 FAIL (2025-11-22 15:01:09 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.560096172s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.................................XX.X.........X........X....] duration: 15.672726988s trace: https://jaeger.dev.storj.io/trace/1efcdf5c2a183d8d FAIL (2025-11-22 15:00:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 934958555s) [noise=true , address=145.79.227.30:4762 , piece_id=4Z7CCNLZEYQAGJNUWGJ2USQSVW6QZUTCNFKC3JWRIOKV6VBVIYKQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.384231169 (318h54m50.93570163s) [noise=true , address=145.79.227.18:4007 , piece_id=KM5TO27LZXJZZAT2DB5JTP7R5LOPJHUU6T34RZI7HH7PRWPW4ZNQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.384687758 (318h54m50.93615822s) [noise=true , address=145.79.227.26:4654 , piece_id=VCRM5O6NHOJOKNQUG5TTZUT3JIDQOMNF2DD6H7KWUJFOQ3P75QGQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.385168865 (318h54m50.936639326s) [noise=true , address=77.90.31.18:4805 , piece_id=3EHB5IFBQRLXFBOVCT25RS4CIXVMSVOE7EKDV26DXOEY2NDJ2YGA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.385605395 (318h54m50.937075867s) [noise=true , address=145.79.227.32:4835 , piece_id=7GD4QUTGLIPOEU25FIDE7VDO3AUJ2RZDU4XKI476KNU6QTETC52A, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.386294879 (318h54m50.93776534s) [noise=true , address=145.79.227.21:4473 , piece_id=D5FC7PVECZTQR64BFVTV3L2HOXCITABN57LLNRSMVCSGQICRYKUQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.386958642 (318h54m50.938429103s) [noise=true , address=145.79.227.14:4837 , piece_id=NR427EUHKVSQ6HMMZCHL4LCBV6VRVTANZDXMX6JHQI2TTQN5R5VQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387097236 (318h54m50.938567687s) [noise=true , address=145.79.227.9:4365 , piece_id=DBW2L6GHAWIHOMOZOTQFQ57Y2MA6KP6SGUO6AJDDF65ERQ324VZQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387456781 (318h54m50.938927242s) [noise=true , address=145.79.227.19:4078 , piece_id=ER3KKYJ2MJQC3HRRBVULC3ING5GLGOHFVD2QBWF2MVAXF6ODMUUA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387755300 (318h54m50.939225761s) [noise=true , address=145.79.227.28:4725 , piece_id=HVMAX2UWJIF2UJQIJS5UBTO5TOIJI5DCDGGURZN4SOMRY3Z4OF4A, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.388354220 (318h54m50.939824681s) [noise=true , address=145.79.227.10:4402 , piece_id=ZU6A2O2R73PIVIDF5NOKMJZH4QX4LOCFSYOQLFOFAA5BEVQH52KA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.388862268 (318h54m50.940332729s) [noise=true , address=145.79.227.22:4509 , piece_id=WRHXVGJZQRTY6ZKMRU6IFYZSMLR6UATXFZOTCPRYKHERA2WKMGNA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.389411814 (318h54m50.940882275s) [noise=true , address=145.79.227.8:4330 , piece_id=LFTZWYQKW56VLMJPOGGLB6IJILPGJNEHIRIYD2F4J34ZH24PCSMQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.389753224 (318h54m50.941223675s) [noise=true , address=145.79.227.16:4909 , piece_id=3DAVIY3ETZVG5CPP5BQB7QL473J5HZG6EQ6BYZBIAN6TWZ5FGNHQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.391102172 (318h54m50.942572633s) [noise=true , address=45.91.116.7:4010 , piece_id=VZCYHOAR73DVCRDT5GBT65JI53NP27OEGA3VYLM66LPQR23T4MTA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:54.404583235 (318h54m52.956053696s) [upload=280863]: Next returned nil segment Nov 22 14:58:54.404601019 (318h54m52.956071481s) [upload=280863]: Waiting for error group managing segments... Nov 22 14:58:54.928222688 (318h54m53.479693149s) [noise=true , address=77.90.31.18:4805 , piece_id=3EHB5IFBQRLXFBOVCT25RS4CIXVMSVOE7EKDV26DXOEY2NDJ2YGA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:58:55.175979631 (318h54m53.727450092s) [noise=true , address=45.91.116.7:4010 , piece_id=VZCYHOAR73DVCRDT5GBT65JI53NP27OEGA3VYLM66LPQR23T4MTA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:03.279622498 (318h55m1.831092959s) [noise=true , address=145.79.227.14:4837 , piece_id=NR427EUHKVSQ6HMMZCHL4LCBV6VRVTANZDXMX6JHQI2TTQN5R5VQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:04.274652248 (318h55m2.826122709s) [noise=true , address=145.79.227.28:4725 , piece_id=HVMAX2UWJIF2UJQIJS5UBTO5TOIJI5DCDGGURZN4SOMRY3Z4OF4A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:05.089136230 (318h55m3.640606701s) [noise=true , address=145.79.227.22:4509 , piece_id=WRHXVGJZQRTY6ZKMRU6IFYZSMLR6UATXFZOTCPRYKHERA2WKMGNA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:05.587545699 (318h55m4.13901616s) [noise=true , address=145.79.227.32:4835 , piece_id=7GD4QUTGLIPOEU25FIDE7VDO3AUJ2RZDU4XKI476KNU6QTETC52A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:06.956814829 (318h55m5.50828528s) [noise=true , address=145.79.227.8:4330 , piece_id=LFTZWYQKW56VLMJPOGGLB6IJILPGJNEHIRIYD2F4J34ZH24PCSMQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:07.254777763 (318h55m5.806248224s) [noise=true , address=145.79.227.30:4762 , piece_id=4Z7CCNLZEYQAGJNUWGJ2USQSVW6QZUTCNFKC3JWRIOKV6VBVIYKQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:19.983388333 (318h55m18.534858794s) [noise=true , address=145.79.227.10:4402 , piece_id=ZU6A2O2R73PIVIDF5NOKMJZH4QX4LOCFSYOQLFOFAA5BEVQH52KA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection Nov 22 14:59:19.983536355 (318h55m18.535006816s) [noise=true , address=145.79.227.21:4473 , piece_id=D5FC7PVECZTQR64BFVTV3L2HOXCITABN57LLNRSMVCSGQICRYKUQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection Nov 22 14:59:19.983802211 (318h55m18.535272703s) [noise=true , address=145.79.227.18:4007 , piece_id=KM5TO27LZXJZZAT2DB5JTP7R5LOPJHUU6T34RZI7HH7PRWPW4ZNQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection Nov 22 14:59:19.983812000 (318h55m18.535282461s) [noise=true , address=145.79.227.9:4365 , piece_id=DBW2L6GHAWIHOMOZOTQFQ57Y2MA6KP6SGUO6AJDDF65ERQ324VZQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection Nov 22 14:59:19.983896611 (318h55m18.535367073s) [noise=true , address=145.79.227.19:4078 , piece_id=ER3KKYJ2MJQC3HRRBVULC3ING5GLGOHFVD2QBWF2MVAXF6ODMUUA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection; write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection Nov 22 14:59:19.984068268 (318h55m18.535538719s) [noise=true , address=145.79.227.26:4654 , piece_id=VCRM5O6NHOJOKNQUG5TTZUT3JIDQOMNF2DD6H7KWUJFOQ3P75QGQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: read tcp 188.166.225.170:36440->145.79.227.26:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984229324 (318h55m18.535699776s) [noise=true , address=145.79.227.8:4329 , piece_id=YJG3LYBDMPS4QQNLG24MS7ZWUM6EOI6EPDGVBGLO33CVEQQKGX6A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: read tcp 188.166.225.170:44902->145.79.227.8:4329: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984374972 (318h55m18.535845423s) [noise=true , address=145.79.227.16:4909 , piece_id=3DAVIY3ETZVG5CPP5BQB7QL473J5HZG6EQ6BYZBIAN6TWZ5FGNHQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: read tcp 188.166.225.170:59064->145.79.227.16:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984400911 (318h55m18.535871362s) [seg_pos={0 0}, upload=280863]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 22 14:59:20.539921660 (318h55m19.091392121s) [upload=280863]: Done uploading segments... Nov 22 14:59:20.539961818 (318h55m19.091432269s) [upload=280863]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 15.672805789s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [..X.........X.X.....X.......X......X............X...........] duration: 7.110385077s trace: https://jaeger.dev.storj.io/trace/1ee0c6bff9434222 FAIL (2025-11-22 15:00:46 UTC) orbiter warning: context deadline exceeded Duration: 7.110430154s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..................................XX.X.........X........X...] duration: 14.241439293s trace: https://jaeger.dev.storj.io/trace/1efcdf5c2a183d8d FAIL (2025-11-22 15:00:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 934958555s) [noise=true , address=145.79.227.30:4762 , piece_id=4Z7CCNLZEYQAGJNUWGJ2USQSVW6QZUTCNFKC3JWRIOKV6VBVIYKQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.384231169 (318h54m50.93570163s) [noise=true , address=145.79.227.18:4007 , piece_id=KM5TO27LZXJZZAT2DB5JTP7R5LOPJHUU6T34RZI7HH7PRWPW4ZNQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.384687758 (318h54m50.93615822s) [noise=true , address=145.79.227.26:4654 , piece_id=VCRM5O6NHOJOKNQUG5TTZUT3JIDQOMNF2DD6H7KWUJFOQ3P75QGQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.385168865 (318h54m50.936639326s) [noise=true , address=77.90.31.18:4805 , piece_id=3EHB5IFBQRLXFBOVCT25RS4CIXVMSVOE7EKDV26DXOEY2NDJ2YGA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.385605395 (318h54m50.937075867s) [noise=true , address=145.79.227.32:4835 , piece_id=7GD4QUTGLIPOEU25FIDE7VDO3AUJ2RZDU4XKI476KNU6QTETC52A, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.386294879 (318h54m50.93776534s) [noise=true , address=145.79.227.21:4473 , piece_id=D5FC7PVECZTQR64BFVTV3L2HOXCITABN57LLNRSMVCSGQICRYKUQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.386958642 (318h54m50.938429103s) [noise=true , address=145.79.227.14:4837 , piece_id=NR427EUHKVSQ6HMMZCHL4LCBV6VRVTANZDXMX6JHQI2TTQN5R5VQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387097236 (318h54m50.938567687s) [noise=true , address=145.79.227.9:4365 , piece_id=DBW2L6GHAWIHOMOZOTQFQ57Y2MA6KP6SGUO6AJDDF65ERQ324VZQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387456781 (318h54m50.938927242s) [noise=true , address=145.79.227.19:4078 , piece_id=ER3KKYJ2MJQC3HRRBVULC3ING5GLGOHFVD2QBWF2MVAXF6ODMUUA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387755300 (318h54m50.939225761s) [noise=true , address=145.79.227.28:4725 , piece_id=HVMAX2UWJIF2UJQIJS5UBTO5TOIJI5DCDGGURZN4SOMRY3Z4OF4A, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.388354220 (318h54m50.939824681s) [noise=true , address=145.79.227.10:4402 , piece_id=ZU6A2O2R73PIVIDF5NOKMJZH4QX4LOCFSYOQLFOFAA5BEVQH52KA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.388862268 (318h54m50.940332729s) [noise=true , address=145.79.227.22:4509 , piece_id=WRHXVGJZQRTY6ZKMRU6IFYZSMLR6UATXFZOTCPRYKHERA2WKMGNA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.389411814 (318h54m50.940882275s) [noise=true , address=145.79.227.8:4330 , piece_id=LFTZWYQKW56VLMJPOGGLB6IJILPGJNEHIRIYD2F4J34ZH24PCSMQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.389753224 (318h54m50.941223675s) [noise=true , address=145.79.227.16:4909 , piece_id=3DAVIY3ETZVG5CPP5BQB7QL473J5HZG6EQ6BYZBIAN6TWZ5FGNHQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.391102172 (318h54m50.942572633s) [noise=true , address=45.91.116.7:4010 , piece_id=VZCYHOAR73DVCRDT5GBT65JI53NP27OEGA3VYLM66LPQR23T4MTA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:54.404583235 (318h54m52.956053696s) [upload=280863]: Next returned nil segment Nov 22 14:58:54.404601019 (318h54m52.956071481s) [upload=280863]: Waiting for error group managing segments... Nov 22 14:58:54.928222688 (318h54m53.479693149s) [noise=true , address=77.90.31.18:4805 , piece_id=3EHB5IFBQRLXFBOVCT25RS4CIXVMSVOE7EKDV26DXOEY2NDJ2YGA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:58:55.175979631 (318h54m53.727450092s) [noise=true , address=45.91.116.7:4010 , piece_id=VZCYHOAR73DVCRDT5GBT65JI53NP27OEGA3VYLM66LPQR23T4MTA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:03.279622498 (318h55m1.831092959s) [noise=true , address=145.79.227.14:4837 , piece_id=NR427EUHKVSQ6HMMZCHL4LCBV6VRVTANZDXMX6JHQI2TTQN5R5VQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:04.274652248 (318h55m2.826122709s) [noise=true , address=145.79.227.28:4725 , piece_id=HVMAX2UWJIF2UJQIJS5UBTO5TOIJI5DCDGGURZN4SOMRY3Z4OF4A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:05.089136230 (318h55m3.640606701s) [noise=true , address=145.79.227.22:4509 , piece_id=WRHXVGJZQRTY6ZKMRU6IFYZSMLR6UATXFZOTCPRYKHERA2WKMGNA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:05.587545699 (318h55m4.13901616s) [noise=true , address=145.79.227.32:4835 , piece_id=7GD4QUTGLIPOEU25FIDE7VDO3AUJ2RZDU4XKI476KNU6QTETC52A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:06.956814829 (318h55m5.50828528s) [noise=true , address=145.79.227.8:4330 , piece_id=LFTZWYQKW56VLMJPOGGLB6IJILPGJNEHIRIYD2F4J34ZH24PCSMQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:07.254777763 (318h55m5.806248224s) [noise=true , address=145.79.227.30:4762 , piece_id=4Z7CCNLZEYQAGJNUWGJ2USQSVW6QZUTCNFKC3JWRIOKV6VBVIYKQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:19.983388333 (318h55m18.534858794s) [noise=true , address=145.79.227.10:4402 , piece_id=ZU6A2O2R73PIVIDF5NOKMJZH4QX4LOCFSYOQLFOFAA5BEVQH52KA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection Nov 22 14:59:19.983536355 (318h55m18.535006816s) [noise=true , address=145.79.227.21:4473 , piece_id=D5FC7PVECZTQR64BFVTV3L2HOXCITABN57LLNRSMVCSGQICRYKUQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection Nov 22 14:59:19.983802211 (318h55m18.535272703s) [noise=true , address=145.79.227.18:4007 , piece_id=KM5TO27LZXJZZAT2DB5JTP7R5LOPJHUU6T34RZI7HH7PRWPW4ZNQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection Nov 22 14:59:19.983812000 (318h55m18.535282461s) [noise=true , address=145.79.227.9:4365 , piece_id=DBW2L6GHAWIHOMOZOTQFQ57Y2MA6KP6SGUO6AJDDF65ERQ324VZQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection Nov 22 14:59:19.983896611 (318h55m18.535367073s) [noise=true , address=145.79.227.19:4078 , piece_id=ER3KKYJ2MJQC3HRRBVULC3ING5GLGOHFVD2QBWF2MVAXF6ODMUUA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection; write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection Nov 22 14:59:19.984068268 (318h55m18.535538719s) [noise=true , address=145.79.227.26:4654 , piece_id=VCRM5O6NHOJOKNQUG5TTZUT3JIDQOMNF2DD6H7KWUJFOQ3P75QGQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: read tcp 188.166.225.170:36440->145.79.227.26:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984229324 (318h55m18.535699776s) [noise=true , address=145.79.227.8:4329 , piece_id=YJG3LYBDMPS4QQNLG24MS7ZWUM6EOI6EPDGVBGLO33CVEQQKGX6A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: read tcp 188.166.225.170:44902->145.79.227.8:4329: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984374972 (318h55m18.535845423s) [noise=true , address=145.79.227.16:4909 , piece_id=3DAVIY3ETZVG5CPP5BQB7QL473J5HZG6EQ6BYZBIAN6TWZ5FGNHQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: read tcp 188.166.225.170:59064->145.79.227.16:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984400911 (318h55m18.535871362s) [seg_pos={0 0}, upload=280863]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 22 14:59:20.539921660 (318h55m19.091392121s) [upload=280863]: Done uploading segments... Nov 22 14:59:20.539961818 (318h55m19.091432269s) [upload=280863]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 14.241472527s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [........X.......XX..X..XX..X.X.........XX.XX................] duration: 4.437379926s trace: https://jaeger.dev.storj.io/trace/58dc5585e5ee8756 FAIL (2025-11-22 15:00:11 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.437412658s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [...X.........X.X.....X.......X......X............X..........] duration: 7.00649145s trace: https://jaeger.dev.storj.io/trace/1ee0c6bff9434222 FAIL (2025-11-22 15:00:02 UTC) orbiter warning: context deadline exceeded Duration: 7.006578234s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...................................XX.X.........X........X..] duration: 14.639617784s trace: https://jaeger.dev.storj.io/trace/1efcdf5c2a183d8d FAIL (2025-11-22 14:59:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 934958555s) [noise=true , address=145.79.227.30:4762 , piece_id=4Z7CCNLZEYQAGJNUWGJ2USQSVW6QZUTCNFKC3JWRIOKV6VBVIYKQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.384231169 (318h54m50.93570163s) [noise=true , address=145.79.227.18:4007 , piece_id=KM5TO27LZXJZZAT2DB5JTP7R5LOPJHUU6T34RZI7HH7PRWPW4ZNQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.384687758 (318h54m50.93615822s) [noise=true , address=145.79.227.26:4654 , piece_id=VCRM5O6NHOJOKNQUG5TTZUT3JIDQOMNF2DD6H7KWUJFOQ3P75QGQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.385168865 (318h54m50.936639326s) [noise=true , address=77.90.31.18:4805 , piece_id=3EHB5IFBQRLXFBOVCT25RS4CIXVMSVOE7EKDV26DXOEY2NDJ2YGA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.385605395 (318h54m50.937075867s) [noise=true , address=145.79.227.32:4835 , piece_id=7GD4QUTGLIPOEU25FIDE7VDO3AUJ2RZDU4XKI476KNU6QTETC52A, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.386294879 (318h54m50.93776534s) [noise=true , address=145.79.227.21:4473 , piece_id=D5FC7PVECZTQR64BFVTV3L2HOXCITABN57LLNRSMVCSGQICRYKUQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.386958642 (318h54m50.938429103s) [noise=true , address=145.79.227.14:4837 , piece_id=NR427EUHKVSQ6HMMZCHL4LCBV6VRVTANZDXMX6JHQI2TTQN5R5VQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387097236 (318h54m50.938567687s) [noise=true , address=145.79.227.9:4365 , piece_id=DBW2L6GHAWIHOMOZOTQFQ57Y2MA6KP6SGUO6AJDDF65ERQ324VZQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387456781 (318h54m50.938927242s) [noise=true , address=145.79.227.19:4078 , piece_id=ER3KKYJ2MJQC3HRRBVULC3ING5GLGOHFVD2QBWF2MVAXF6ODMUUA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387755300 (318h54m50.939225761s) [noise=true , address=145.79.227.28:4725 , piece_id=HVMAX2UWJIF2UJQIJS5UBTO5TOIJI5DCDGGURZN4SOMRY3Z4OF4A, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.388354220 (318h54m50.939824681s) [noise=true , address=145.79.227.10:4402 , piece_id=ZU6A2O2R73PIVIDF5NOKMJZH4QX4LOCFSYOQLFOFAA5BEVQH52KA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.388862268 (318h54m50.940332729s) [noise=true , address=145.79.227.22:4509 , piece_id=WRHXVGJZQRTY6ZKMRU6IFYZSMLR6UATXFZOTCPRYKHERA2WKMGNA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.389411814 (318h54m50.940882275s) [noise=true , address=145.79.227.8:4330 , piece_id=LFTZWYQKW56VLMJPOGGLB6IJILPGJNEHIRIYD2F4J34ZH24PCSMQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.389753224 (318h54m50.941223675s) [noise=true , address=145.79.227.16:4909 , piece_id=3DAVIY3ETZVG5CPP5BQB7QL473J5HZG6EQ6BYZBIAN6TWZ5FGNHQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.391102172 (318h54m50.942572633s) [noise=true , address=45.91.116.7:4010 , piece_id=VZCYHOAR73DVCRDT5GBT65JI53NP27OEGA3VYLM66LPQR23T4MTA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:54.404583235 (318h54m52.956053696s) [upload=280863]: Next returned nil segment Nov 22 14:58:54.404601019 (318h54m52.956071481s) [upload=280863]: Waiting for error group managing segments... Nov 22 14:58:54.928222688 (318h54m53.479693149s) [noise=true , address=77.90.31.18:4805 , piece_id=3EHB5IFBQRLXFBOVCT25RS4CIXVMSVOE7EKDV26DXOEY2NDJ2YGA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:58:55.175979631 (318h54m53.727450092s) [noise=true , address=45.91.116.7:4010 , piece_id=VZCYHOAR73DVCRDT5GBT65JI53NP27OEGA3VYLM66LPQR23T4MTA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:03.279622498 (318h55m1.831092959s) [noise=true , address=145.79.227.14:4837 , piece_id=NR427EUHKVSQ6HMMZCHL4LCBV6VRVTANZDXMX6JHQI2TTQN5R5VQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:04.274652248 (318h55m2.826122709s) [noise=true , address=145.79.227.28:4725 , piece_id=HVMAX2UWJIF2UJQIJS5UBTO5TOIJI5DCDGGURZN4SOMRY3Z4OF4A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:05.089136230 (318h55m3.640606701s) [noise=true , address=145.79.227.22:4509 , piece_id=WRHXVGJZQRTY6ZKMRU6IFYZSMLR6UATXFZOTCPRYKHERA2WKMGNA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:05.587545699 (318h55m4.13901616s) [noise=true , address=145.79.227.32:4835 , piece_id=7GD4QUTGLIPOEU25FIDE7VDO3AUJ2RZDU4XKI476KNU6QTETC52A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:06.956814829 (318h55m5.50828528s) [noise=true , address=145.79.227.8:4330 , piece_id=LFTZWYQKW56VLMJPOGGLB6IJILPGJNEHIRIYD2F4J34ZH24PCSMQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:07.254777763 (318h55m5.806248224s) [noise=true , address=145.79.227.30:4762 , piece_id=4Z7CCNLZEYQAGJNUWGJ2USQSVW6QZUTCNFKC3JWRIOKV6VBVIYKQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:19.983388333 (318h55m18.534858794s) [noise=true , address=145.79.227.10:4402 , piece_id=ZU6A2O2R73PIVIDF5NOKMJZH4QX4LOCFSYOQLFOFAA5BEVQH52KA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection Nov 22 14:59:19.983536355 (318h55m18.535006816s) [noise=true , address=145.79.227.21:4473 , piece_id=D5FC7PVECZTQR64BFVTV3L2HOXCITABN57LLNRSMVCSGQICRYKUQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection Nov 22 14:59:19.983802211 (318h55m18.535272703s) [noise=true , address=145.79.227.18:4007 , piece_id=KM5TO27LZXJZZAT2DB5JTP7R5LOPJHUU6T34RZI7HH7PRWPW4ZNQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection Nov 22 14:59:19.983812000 (318h55m18.535282461s) [noise=true , address=145.79.227.9:4365 , piece_id=DBW2L6GHAWIHOMOZOTQFQ57Y2MA6KP6SGUO6AJDDF65ERQ324VZQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection Nov 22 14:59:19.983896611 (318h55m18.535367073s) [noise=true , address=145.79.227.19:4078 , piece_id=ER3KKYJ2MJQC3HRRBVULC3ING5GLGOHFVD2QBWF2MVAXF6ODMUUA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection; write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection Nov 22 14:59:19.984068268 (318h55m18.535538719s) [noise=true , address=145.79.227.26:4654 , piece_id=VCRM5O6NHOJOKNQUG5TTZUT3JIDQOMNF2DD6H7KWUJFOQ3P75QGQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: read tcp 188.166.225.170:36440->145.79.227.26:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984229324 (318h55m18.535699776s) [noise=true , address=145.79.227.8:4329 , piece_id=YJG3LYBDMPS4QQNLG24MS7ZWUM6EOI6EPDGVBGLO33CVEQQKGX6A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: read tcp 188.166.225.170:44902->145.79.227.8:4329: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984374972 (318h55m18.535845423s) [noise=true , address=145.79.227.16:4909 , piece_id=3DAVIY3ETZVG5CPP5BQB7QL473J5HZG6EQ6BYZBIAN6TWZ5FGNHQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: read tcp 188.166.225.170:59064->145.79.227.16:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984400911 (318h55m18.535871362s) [seg_pos={0 0}, upload=280863]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 22 14:59:20.539921660 (318h55m19.091392121s) [upload=280863]: Done uploading segments... Nov 22 14:59:20.539961818 (318h55m19.091432269s) [upload=280863]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 14.6396464s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [....................................XX.X.........X........X.] duration: 14.541514139s trace: https://jaeger.dev.storj.io/trace/1efcdf5c2a183d8d FAIL (2025-11-22 14:59:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 934958555s) [noise=true , address=145.79.227.30:4762 , piece_id=4Z7CCNLZEYQAGJNUWGJ2USQSVW6QZUTCNFKC3JWRIOKV6VBVIYKQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.384231169 (318h54m50.93570163s) [noise=true , address=145.79.227.18:4007 , piece_id=KM5TO27LZXJZZAT2DB5JTP7R5LOPJHUU6T34RZI7HH7PRWPW4ZNQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.384687758 (318h54m50.93615822s) [noise=true , address=145.79.227.26:4654 , piece_id=VCRM5O6NHOJOKNQUG5TTZUT3JIDQOMNF2DD6H7KWUJFOQ3P75QGQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.385168865 (318h54m50.936639326s) [noise=true , address=77.90.31.18:4805 , piece_id=3EHB5IFBQRLXFBOVCT25RS4CIXVMSVOE7EKDV26DXOEY2NDJ2YGA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.385605395 (318h54m50.937075867s) [noise=true , address=145.79.227.32:4835 , piece_id=7GD4QUTGLIPOEU25FIDE7VDO3AUJ2RZDU4XKI476KNU6QTETC52A, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.386294879 (318h54m50.93776534s) [noise=true , address=145.79.227.21:4473 , piece_id=D5FC7PVECZTQR64BFVTV3L2HOXCITABN57LLNRSMVCSGQICRYKUQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.386958642 (318h54m50.938429103s) [noise=true , address=145.79.227.14:4837 , piece_id=NR427EUHKVSQ6HMMZCHL4LCBV6VRVTANZDXMX6JHQI2TTQN5R5VQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387097236 (318h54m50.938567687s) [noise=true , address=145.79.227.9:4365 , piece_id=DBW2L6GHAWIHOMOZOTQFQ57Y2MA6KP6SGUO6AJDDF65ERQ324VZQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387456781 (318h54m50.938927242s) [noise=true , address=145.79.227.19:4078 , piece_id=ER3KKYJ2MJQC3HRRBVULC3ING5GLGOHFVD2QBWF2MVAXF6ODMUUA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387755300 (318h54m50.939225761s) [noise=true , address=145.79.227.28:4725 , piece_id=HVMAX2UWJIF2UJQIJS5UBTO5TOIJI5DCDGGURZN4SOMRY3Z4OF4A, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.388354220 (318h54m50.939824681s) [noise=true , address=145.79.227.10:4402 , piece_id=ZU6A2O2R73PIVIDF5NOKMJZH4QX4LOCFSYOQLFOFAA5BEVQH52KA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.388862268 (318h54m50.940332729s) [noise=true , address=145.79.227.22:4509 , piece_id=WRHXVGJZQRTY6ZKMRU6IFYZSMLR6UATXFZOTCPRYKHERA2WKMGNA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.389411814 (318h54m50.940882275s) [noise=true , address=145.79.227.8:4330 , piece_id=LFTZWYQKW56VLMJPOGGLB6IJILPGJNEHIRIYD2F4J34ZH24PCSMQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.389753224 (318h54m50.941223675s) [noise=true , address=145.79.227.16:4909 , piece_id=3DAVIY3ETZVG5CPP5BQB7QL473J5HZG6EQ6BYZBIAN6TWZ5FGNHQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.391102172 (318h54m50.942572633s) [noise=true , address=45.91.116.7:4010 , piece_id=VZCYHOAR73DVCRDT5GBT65JI53NP27OEGA3VYLM66LPQR23T4MTA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:54.404583235 (318h54m52.956053696s) [upload=280863]: Next returned nil segment Nov 22 14:58:54.404601019 (318h54m52.956071481s) [upload=280863]: Waiting for error group managing segments... Nov 22 14:58:54.928222688 (318h54m53.479693149s) [noise=true , address=77.90.31.18:4805 , piece_id=3EHB5IFBQRLXFBOVCT25RS4CIXVMSVOE7EKDV26DXOEY2NDJ2YGA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:58:55.175979631 (318h54m53.727450092s) [noise=true , address=45.91.116.7:4010 , piece_id=VZCYHOAR73DVCRDT5GBT65JI53NP27OEGA3VYLM66LPQR23T4MTA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:03.279622498 (318h55m1.831092959s) [noise=true , address=145.79.227.14:4837 , piece_id=NR427EUHKVSQ6HMMZCHL4LCBV6VRVTANZDXMX6JHQI2TTQN5R5VQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:04.274652248 (318h55m2.826122709s) [noise=true , address=145.79.227.28:4725 , piece_id=HVMAX2UWJIF2UJQIJS5UBTO5TOIJI5DCDGGURZN4SOMRY3Z4OF4A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:05.089136230 (318h55m3.640606701s) [noise=true , address=145.79.227.22:4509 , piece_id=WRHXVGJZQRTY6ZKMRU6IFYZSMLR6UATXFZOTCPRYKHERA2WKMGNA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:05.587545699 (318h55m4.13901616s) [noise=true , address=145.79.227.32:4835 , piece_id=7GD4QUTGLIPOEU25FIDE7VDO3AUJ2RZDU4XKI476KNU6QTETC52A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:06.956814829 (318h55m5.50828528s) [noise=true , address=145.79.227.8:4330 , piece_id=LFTZWYQKW56VLMJPOGGLB6IJILPGJNEHIRIYD2F4J34ZH24PCSMQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:07.254777763 (318h55m5.806248224s) [noise=true , address=145.79.227.30:4762 , piece_id=4Z7CCNLZEYQAGJNUWGJ2USQSVW6QZUTCNFKC3JWRIOKV6VBVIYKQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:19.983388333 (318h55m18.534858794s) [noise=true , address=145.79.227.10:4402 , piece_id=ZU6A2O2R73PIVIDF5NOKMJZH4QX4LOCFSYOQLFOFAA5BEVQH52KA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection Nov 22 14:59:19.983536355 (318h55m18.535006816s) [noise=true , address=145.79.227.21:4473 , piece_id=D5FC7PVECZTQR64BFVTV3L2HOXCITABN57LLNRSMVCSGQICRYKUQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection Nov 22 14:59:19.983802211 (318h55m18.535272703s) [noise=true , address=145.79.227.18:4007 , piece_id=KM5TO27LZXJZZAT2DB5JTP7R5LOPJHUU6T34RZI7HH7PRWPW4ZNQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection Nov 22 14:59:19.983812000 (318h55m18.535282461s) [noise=true , address=145.79.227.9:4365 , piece_id=DBW2L6GHAWIHOMOZOTQFQ57Y2MA6KP6SGUO6AJDDF65ERQ324VZQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection Nov 22 14:59:19.983896611 (318h55m18.535367073s) [noise=true , address=145.79.227.19:4078 , piece_id=ER3KKYJ2MJQC3HRRBVULC3ING5GLGOHFVD2QBWF2MVAXF6ODMUUA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection; write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection Nov 22 14:59:19.984068268 (318h55m18.535538719s) [noise=true , address=145.79.227.26:4654 , piece_id=VCRM5O6NHOJOKNQUG5TTZUT3JIDQOMNF2DD6H7KWUJFOQ3P75QGQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: read tcp 188.166.225.170:36440->145.79.227.26:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984229324 (318h55m18.535699776s) [noise=true , address=145.79.227.8:4329 , piece_id=YJG3LYBDMPS4QQNLG24MS7ZWUM6EOI6EPDGVBGLO33CVEQQKGX6A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: read tcp 188.166.225.170:44902->145.79.227.8:4329: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984374972 (318h55m18.535845423s) [noise=true , address=145.79.227.16:4909 , piece_id=3DAVIY3ETZVG5CPP5BQB7QL473J5HZG6EQ6BYZBIAN6TWZ5FGNHQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: read tcp 188.166.225.170:59064->145.79.227.16:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984400911 (318h55m18.535871362s) [seg_pos={0 0}, upload=280863]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 22 14:59:20.539921660 (318h55m19.091392121s) [upload=280863]: Done uploading segments... Nov 22 14:59:20.539961818 (318h55m19.091432269s) [upload=280863]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 14.541543715s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....................................XX.X.........X........X] duration: 30.558496078s trace: https://jaeger.dev.storj.io/trace/1efcdf5c2a183d8d FAIL (2025-11-22 14:59:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 934958555s) [noise=true , address=145.79.227.30:4762 , piece_id=4Z7CCNLZEYQAGJNUWGJ2USQSVW6QZUTCNFKC3JWRIOKV6VBVIYKQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.384231169 (318h54m50.93570163s) [noise=true , address=145.79.227.18:4007 , piece_id=KM5TO27LZXJZZAT2DB5JTP7R5LOPJHUU6T34RZI7HH7PRWPW4ZNQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.384687758 (318h54m50.93615822s) [noise=true , address=145.79.227.26:4654 , piece_id=VCRM5O6NHOJOKNQUG5TTZUT3JIDQOMNF2DD6H7KWUJFOQ3P75QGQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.385168865 (318h54m50.936639326s) [noise=true , address=77.90.31.18:4805 , piece_id=3EHB5IFBQRLXFBOVCT25RS4CIXVMSVOE7EKDV26DXOEY2NDJ2YGA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.385605395 (318h54m50.937075867s) [noise=true , address=145.79.227.32:4835 , piece_id=7GD4QUTGLIPOEU25FIDE7VDO3AUJ2RZDU4XKI476KNU6QTETC52A, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.386294879 (318h54m50.93776534s) [noise=true , address=145.79.227.21:4473 , piece_id=D5FC7PVECZTQR64BFVTV3L2HOXCITABN57LLNRSMVCSGQICRYKUQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.386958642 (318h54m50.938429103s) [noise=true , address=145.79.227.14:4837 , piece_id=NR427EUHKVSQ6HMMZCHL4LCBV6VRVTANZDXMX6JHQI2TTQN5R5VQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387097236 (318h54m50.938567687s) [noise=true , address=145.79.227.9:4365 , piece_id=DBW2L6GHAWIHOMOZOTQFQ57Y2MA6KP6SGUO6AJDDF65ERQ324VZQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387456781 (318h54m50.938927242s) [noise=true , address=145.79.227.19:4078 , piece_id=ER3KKYJ2MJQC3HRRBVULC3ING5GLGOHFVD2QBWF2MVAXF6ODMUUA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387755300 (318h54m50.939225761s) [noise=true , address=145.79.227.28:4725 , piece_id=HVMAX2UWJIF2UJQIJS5UBTO5TOIJI5DCDGGURZN4SOMRY3Z4OF4A, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.388354220 (318h54m50.939824681s) [noise=true , address=145.79.227.10:4402 , piece_id=ZU6A2O2R73PIVIDF5NOKMJZH4QX4LOCFSYOQLFOFAA5BEVQH52KA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.388862268 (318h54m50.940332729s) [noise=true , address=145.79.227.22:4509 , piece_id=WRHXVGJZQRTY6ZKMRU6IFYZSMLR6UATXFZOTCPRYKHERA2WKMGNA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.389411814 (318h54m50.940882275s) [noise=true , address=145.79.227.8:4330 , piece_id=LFTZWYQKW56VLMJPOGGLB6IJILPGJNEHIRIYD2F4J34ZH24PCSMQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.389753224 (318h54m50.941223675s) [noise=true , address=145.79.227.16:4909 , piece_id=3DAVIY3ETZVG5CPP5BQB7QL473J5HZG6EQ6BYZBIAN6TWZ5FGNHQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.391102172 (318h54m50.942572633s) [noise=true , address=45.91.116.7:4010 , piece_id=VZCYHOAR73DVCRDT5GBT65JI53NP27OEGA3VYLM66LPQR23T4MTA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:54.404583235 (318h54m52.956053696s) [upload=280863]: Next returned nil segment Nov 22 14:58:54.404601019 (318h54m52.956071481s) [upload=280863]: Waiting for error group managing segments... Nov 22 14:58:54.928222688 (318h54m53.479693149s) [noise=true , address=77.90.31.18:4805 , piece_id=3EHB5IFBQRLXFBOVCT25RS4CIXVMSVOE7EKDV26DXOEY2NDJ2YGA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:58:55.175979631 (318h54m53.727450092s) [noise=true , address=45.91.116.7:4010 , piece_id=VZCYHOAR73DVCRDT5GBT65JI53NP27OEGA3VYLM66LPQR23T4MTA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:03.279622498 (318h55m1.831092959s) [noise=true , address=145.79.227.14:4837 , piece_id=NR427EUHKVSQ6HMMZCHL4LCBV6VRVTANZDXMX6JHQI2TTQN5R5VQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:04.274652248 (318h55m2.826122709s) [noise=true , address=145.79.227.28:4725 , piece_id=HVMAX2UWJIF2UJQIJS5UBTO5TOIJI5DCDGGURZN4SOMRY3Z4OF4A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:05.089136230 (318h55m3.640606701s) [noise=true , address=145.79.227.22:4509 , piece_id=WRHXVGJZQRTY6ZKMRU6IFYZSMLR6UATXFZOTCPRYKHERA2WKMGNA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:05.587545699 (318h55m4.13901616s) [noise=true , address=145.79.227.32:4835 , piece_id=7GD4QUTGLIPOEU25FIDE7VDO3AUJ2RZDU4XKI476KNU6QTETC52A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:06.956814829 (318h55m5.50828528s) [noise=true , address=145.79.227.8:4330 , piece_id=LFTZWYQKW56VLMJPOGGLB6IJILPGJNEHIRIYD2F4J34ZH24PCSMQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:07.254777763 (318h55m5.806248224s) [noise=true , address=145.79.227.30:4762 , piece_id=4Z7CCNLZEYQAGJNUWGJ2USQSVW6QZUTCNFKC3JWRIOKV6VBVIYKQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:19.983388333 (318h55m18.534858794s) [noise=true , address=145.79.227.10:4402 , piece_id=ZU6A2O2R73PIVIDF5NOKMJZH4QX4LOCFSYOQLFOFAA5BEVQH52KA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection Nov 22 14:59:19.983536355 (318h55m18.535006816s) [noise=true , address=145.79.227.21:4473 , piece_id=D5FC7PVECZTQR64BFVTV3L2HOXCITABN57LLNRSMVCSGQICRYKUQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection Nov 22 14:59:19.983802211 (318h55m18.535272703s) [noise=true , address=145.79.227.18:4007 , piece_id=KM5TO27LZXJZZAT2DB5JTP7R5LOPJHUU6T34RZI7HH7PRWPW4ZNQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection Nov 22 14:59:19.983812000 (318h55m18.535282461s) [noise=true , address=145.79.227.9:4365 , piece_id=DBW2L6GHAWIHOMOZOTQFQ57Y2MA6KP6SGUO6AJDDF65ERQ324VZQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection Nov 22 14:59:19.983896611 (318h55m18.535367073s) [noise=true , address=145.79.227.19:4078 , piece_id=ER3KKYJ2MJQC3HRRBVULC3ING5GLGOHFVD2QBWF2MVAXF6ODMUUA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection; write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection Nov 22 14:59:19.984068268 (318h55m18.535538719s) [noise=true , address=145.79.227.26:4654 , piece_id=VCRM5O6NHOJOKNQUG5TTZUT3JIDQOMNF2DD6H7KWUJFOQ3P75QGQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: read tcp 188.166.225.170:36440->145.79.227.26:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984229324 (318h55m18.535699776s) [noise=true , address=145.79.227.8:4329 , piece_id=YJG3LYBDMPS4QQNLG24MS7ZWUM6EOI6EPDGVBGLO33CVEQQKGX6A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: read tcp 188.166.225.170:44902->145.79.227.8:4329: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984374972 (318h55m18.535845423s) [noise=true , address=145.79.227.16:4909 , piece_id=3DAVIY3ETZVG5CPP5BQB7QL473J5HZG6EQ6BYZBIAN6TWZ5FGNHQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: read tcp 188.166.225.170:59064->145.79.227.16:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984400911 (318h55m18.535871362s) [seg_pos={0 0}, upload=280863]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 22 14:59:20.539921660 (318h55m19.091392121s) [upload=280863]: Done uploading segments... Nov 22 14:59:20.539961818 (318h55m19.091432269s) [upload=280863]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.558520925s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [....X.........X.X.....X.......X......X............X.........] duration: 5.858347959s trace: https://jaeger.dev.storj.io/trace/1ee0c6bff9434222 FAIL (2025-11-22 14:59:10 UTC) orbiter warning: context deadline exceeded Duration: 5.858375249s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [X........X.......XX..X..XX..X.X.........XX.XX...............] duration: 3.39390912s trace: https://jaeger.dev.storj.io/trace/58dc5585e5ee8756 FAIL (2025-11-22 14:59:06 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.393942143s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.....X.........X.X.....X.......X......X............X........] duration: 6.146185241s trace: https://jaeger.dev.storj.io/trace/1ee0c6bff9434222 FAIL (2025-11-22 14:58:55 UTC) orbiter warning: context deadline exceeded Duration: 6.146204988s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [.X........X.......XX..X..XX..X.X.........XX.XX..............] duration: 3.565008515s trace: https://jaeger.dev.storj.io/trace/58dc5585e5ee8756 FAIL (2025-11-22 14:58:39 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.565039324s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [..X........X.......XX..X..XX..X.X.........XX.XX.............] duration: 3.796944042s trace: https://jaeger.dev.storj.io/trace/58dc5585e5ee8756 FAIL (2025-11-22 14:58:20 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.796983277s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [X.....X.........X.X.....X.......X......X............X.......] duration: 6.040146866s trace: https://jaeger.dev.storj.io/trace/1ee0c6bff9434222 FAIL (2025-11-22 14:58:10 UTC) orbiter warning: context deadline exceeded Duration: 6.040176602s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [......................................XX.X.........X........] duration: 18.900911581s trace: https://jaeger.dev.storj.io/trace/1f18db6043613191 FAIL (2025-11-22 14:58:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== pload=280745]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 14:50:50.164295669 (318h46m48.71576612s) [seg_pos={0 0}, upload=280745]: Begin upload segment... Nov 22 14:50:50.164353229 (318h46m48.71582369s) [seg_pos={0 0}, upload=280745]: Done begin upload segment. Nov 22 14:50:50.164374059 (318h46m48.71584452s) [noise=true , address=145.79.227.25:4619 , piece_id=OPX6FGOO77ZNHAFYKJQYP34Y6FHXGCL7OVFVUXSCX2PTYJHLJFEQ, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.164385731 (318h46m48.715856182s) [noise=true , address=77.90.31.16:4734 , piece_id=POCRBOZSUZKROP5PPYVWE6ZJBRBIZ5N6NUZBEY37JWXOK7FWCX4A, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.164503194 (318h46m48.715973656s) [noise=true , address=145.79.227.21:4474 , piece_id=HK2OT4RLO7OLQR46P6OU6XE3652OV6ECU4EAF3ABM3MGL3O2XK6A, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.165229057 (318h46m48.716699518s) [noise=true , address=45.91.116.9:4261 , piece_id=YISLO23F3HLAR2HLK3EL6ZDXDJY655XNXRYWAFWM2GSWSM6DOGZA, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.165688792 (318h46m48.717159253s) [noise=true , address=145.79.227.20:4439 , piece_id=HUBBZOWAZ3B4GBMUV4E3SFMUEDA5757BSX7Q4EV5VHZ25B3V4EPA, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.166155391 (318h46m48.717625852s) [noise=true , address=77.90.31.15:4697 , piece_id=FT2H7VSGGQPFMOAUZKWA5YUCARSZBDBJAVDZEQIHQSVNTJTRZS4Q, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.166692824 (318h46m48.718163295s) [noise=true , address=145.79.227.31:4798 , piece_id=GHZGFXHRAPEUBQ5MAVBLLN2TH2TLPOYZU4NHO6GJ7WCJKQXKYX5A, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.167166076 (318h46m48.718636537s) [noise=true , address=77.90.31.18:4807 , piece_id=C347B3KZ7NLDOPNXDOPU3TSVD4B2EMYBGVGISHGSG4GIDZEGPANQ, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.168429381 (318h46m48.719899852s) [noise=true , address=145.79.227.21:4473 , piece_id=TNNL4WDT7WL5QRVJOK2HWZBSAIYV4YVV3APHUMKTOYZAXYHBMEKA, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.169388678 (318h46m48.720859139s) [noise=true , address=145.79.227.26:4653 , piece_id=F3HR6ZS7KLN3KMJPUDP6PDERMF5YMVFP7JV2JM7DBG4PDU2XKUZQ, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.169688388 (318h46m48.721158949s) [noise=true , address=145.79.227.23:4546 , piece_id=YPCRIDSXM3ZMQZKRTEPHO5ANUB6Q5QYLW63WRPOM3U2N4I574DFA, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.170843047 (318h46m48.722313508s) [noise=true , address=77.90.31.14:4661 , piece_id=2AVXMHD667SLHIVB2YS3ZKNQYTXCBZ724FBBYUJWKVRSYFVE6MWA, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.171592384 (318h46m48.723062845s) [noise=true , address=145.79.227.22:4511 , piece_id=FT6YZ2TFUKJS2P6KVKJDQT2GNKACJI2GROSBBU2ATATICOXVJEJQ, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.173336084 (318h46m48.724806545s) [noise=true , address=145.79.227.19:4078 , piece_id=6G7DJTWC54HYI6S3XP5EIHC2VZA4MYTYZCSKQQEGMUHAFMHHIWOA, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.175619112 (318h46m48.727089573s) [noise=true , address=145.79.227.9:4367 , piece_id=66X6OMDN7R74TFNCFSKZHOUGX75JWLYT7HI77MAEPGCJBRBYSENA, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.178391049 (318h46m48.72986151s) [noise=true , address=45.91.116.6:4974 , piece_id=OZOBWBFMONEMFWIVFJAUW7URGRWIR4HBUF7FMPK5F2ASAXVUPLYQ, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:52.218800746 (318h46m50.770271197s) [upload=280745]: Next returned nil segment Nov 22 14:50:52.218818249 (318h46m50.77028871s) [upload=280745]: Waiting for error group managing segments... Nov 22 14:50:52.677314588 (318h46m51.228785049s) [noise=true , address=77.90.31.16:4734 , piece_id=POCRBOZSUZKROP5PPYVWE6ZJBRBIZ5N6NUZBEY37JWXOK7FWCX4A, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:50:53.035891433 (318h46m51.587361914s) [noise=true , address=45.91.116.9:4261 , piece_id=YISLO23F3HLAR2HLK3EL6ZDXDJY655XNXRYWAFWM2GSWSM6DOGZA, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:50:53.142171160 (318h46m51.693641621s) [noise=true , address=77.90.31.18:4807 , piece_id=C347B3KZ7NLDOPNXDOPU3TSVD4B2EMYBGVGISHGSG4GIDZEGPANQ, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:50:53.148762440 (318h46m51.700232901s) [noise=true , address=45.91.116.6:4974 , piece_id=OZOBWBFMONEMFWIVFJAUW7URGRWIR4HBUF7FMPK5F2ASAXVUPLYQ, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:50:53.169042227 (318h46m51.720512688s) [noise=true , address=77.90.31.15:4697 , piece_id=FT2H7VSGGQPFMOAUZKWA5YUCARSZBDBJAVDZEQIHQSVNTJTRZS4Q, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:50:53.525428209 (318h46m52.07689867s) [noise=true , address=77.90.31.14:4661 , piece_id=2AVXMHD667SLHIVB2YS3ZKNQYTXCBZ724FBBYUJWKVRSYFVE6MWA, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:51:02.564730655 (318h47m1.116201116s) [noise=true , address=145.79.227.20:4439 , piece_id=HUBBZOWAZ3B4GBMUV4E3SFMUEDA5757BSX7Q4EV5VHZ25B3V4EPA, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:51:03.499338414 (318h47m2.050808885s) [noise=true , address=145.79.227.19:4078 , piece_id=6G7DJTWC54HYI6S3XP5EIHC2VZA4MYTYZCSKQQEGMUHAFMHHIWOA, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:51:04.992927832 (318h47m3.544398293s) [noise=true , address=145.79.227.22:4511 , piece_id=FT6YZ2TFUKJS2P6KVKJDQT2GNKACJI2GROSBBU2ATATICOXVJEJQ, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:51:14.888732580 (318h47m13.440203041s) [noise=true , address=145.79.227.21:4474 , piece_id=HK2OT4RLO7OLQR46P6OU6XE3652OV6ECU4EAF3ABM3MGL3O2XK6A, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:51:17.793451829 (318h47m16.34492229s) [noise=true , address=145.79.227.31:4798 , piece_id=GHZGFXHRAPEUBQ5MAVBLLN2TH2TLPOYZU4NHO6GJ7WCJKQXKYX5A, seg_pos={0 0}, upload=280745]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:60870->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:60870->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60870->145.79.227.31:4798: use of closed network connection Nov 22 14:51:17.793649536 (318h47m16.345119997s) [noise=true , address=145.79.227.21:4473 , piece_id=TNNL4WDT7WL5QRVJOK2HWZBSAIYV4YVV3APHUMKTOYZAXYHBMEKA, seg_pos={0 0}, upload=280745]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:36642->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:36642->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36642->145.79.227.21:4473: use of closed network connection Nov 22 14:51:17.793741841 (318h47m16.345212302s) [noise=true , address=145.79.227.26:4653 , piece_id=F3HR6ZS7KLN3KMJPUDP6PDERMF5YMVFP7JV2JM7DBG4PDU2XKUZQ, seg_pos={0 0}, upload=280745]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:40892->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:40892->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40892->145.79.227.26:4653: use of closed network connection Nov 22 14:51:17.793873091 (318h47m16.345343552s) [noise=true , address=145.79.227.25:4619 , piece_id=OPX6FGOO77ZNHAFYKJQYP34Y6FHXGCL7OVFVUXSCX2PTYJHLJFEQ, seg_pos={0 0}, upload=280745]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: write tcp 188.166.225.170:51940->145.79.227.25:4619: use of closed network connection; write tcp 188.166.225.170:51940->145.79.227.25:4619: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51940->145.79.227.25:4619: use of closed network connection Nov 22 14:51:17.793963053 (318h47m16.345433514s) [noise=true , address=145.79.227.23:4546 , piece_id=YPCRIDSXM3ZMQZKRTEPHO5ANUB6Q5QYLW63WRPOM3U2N4I574DFA, seg_pos={0 0}, upload=280745]: Done uploading piece. err: ecclient: upload failed (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5, address:145.79.227.23:4546): protocol: write tcp 188.166.225.170:58790->145.79.227.23:4546: use of closed network connection; write tcp 188.166.225.170:58790->145.79.227.23:4546: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58790->145.79.227.23:4546: use of closed network connection Nov 22 14:51:17.794537025 (318h47m16.346007486s) [noise=true , address=145.79.227.9:4367 , piece_id=66X6OMDN7R74TFNCFSKZHOUGX75JWLYT7HI77MAEPGCJBRBYSENA, seg_pos={0 0}, upload=280745]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: read tcp 188.166.225.170:59776->145.79.227.9:4367: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:51:17.794622438 (318h47m16.346092889s) [seg_pos={0 0}, upload=280745]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 22 14:51:18.431733399 (318h47m16.98320385s) [upload=280745]: Done uploading segments... Nov 22 14:51:18.431769258 (318h47m16.983239709s) [upload=280745]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration:* 18.900940636s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.X.....X.........X.X.....X.......X......X............X......] duration: 5.898947988s trace: https://jaeger.dev.storj.io/trace/1ee0c6bff9434222 FAIL (2025-11-22 14:57:47 UTC) orbiter warning: context deadline exceeded Duration: 5.898982925s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......................................XX.X.........X.......] duration: 15.896711161s trace: https://jaeger.dev.storj.io/trace/1f18db6043613191 FAIL (2025-11-22 14:57:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== pload=280745]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 14:50:50.164295669 (318h46m48.71576612s) [seg_pos={0 0}, upload=280745]: Begin upload segment... Nov 22 14:50:50.164353229 (318h46m48.71582369s) [seg_pos={0 0}, upload=280745]: Done begin upload segment. Nov 22 14:50:50.164374059 (318h46m48.71584452s) [noise=true , address=145.79.227.25:4619 , piece_id=OPX6FGOO77ZNHAFYKJQYP34Y6FHXGCL7OVFVUXSCX2PTYJHLJFEQ, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.164385731 (318h46m48.715856182s) [noise=true , address=77.90.31.16:4734 , piece_id=POCRBOZSUZKROP5PPYVWE6ZJBRBIZ5N6NUZBEY37JWXOK7FWCX4A, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.164503194 (318h46m48.715973656s) [noise=true , address=145.79.227.21:4474 , piece_id=HK2OT4RLO7OLQR46P6OU6XE3652OV6ECU4EAF3ABM3MGL3O2XK6A, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.165229057 (318h46m48.716699518s) [noise=true , address=45.91.116.9:4261 , piece_id=YISLO23F3HLAR2HLK3EL6ZDXDJY655XNXRYWAFWM2GSWSM6DOGZA, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.165688792 (318h46m48.717159253s) [noise=true , address=145.79.227.20:4439 , piece_id=HUBBZOWAZ3B4GBMUV4E3SFMUEDA5757BSX7Q4EV5VHZ25B3V4EPA, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.166155391 (318h46m48.717625852s) [noise=true , address=77.90.31.15:4697 , piece_id=FT2H7VSGGQPFMOAUZKWA5YUCARSZBDBJAVDZEQIHQSVNTJTRZS4Q, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.166692824 (318h46m48.718163295s) [noise=true , address=145.79.227.31:4798 , piece_id=GHZGFXHRAPEUBQ5MAVBLLN2TH2TLPOYZU4NHO6GJ7WCJKQXKYX5A, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.167166076 (318h46m48.718636537s) [noise=true , address=77.90.31.18:4807 , piece_id=C347B3KZ7NLDOPNXDOPU3TSVD4B2EMYBGVGISHGSG4GIDZEGPANQ, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.168429381 (318h46m48.719899852s) [noise=true , address=145.79.227.21:4473 , piece_id=TNNL4WDT7WL5QRVJOK2HWZBSAIYV4YVV3APHUMKTOYZAXYHBMEKA, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.169388678 (318h46m48.720859139s) [noise=true , address=145.79.227.26:4653 , piece_id=F3HR6ZS7KLN3KMJPUDP6PDERMF5YMVFP7JV2JM7DBG4PDU2XKUZQ, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.169688388 (318h46m48.721158949s) [noise=true , address=145.79.227.23:4546 , piece_id=YPCRIDSXM3ZMQZKRTEPHO5ANUB6Q5QYLW63WRPOM3U2N4I574DFA, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.170843047 (318h46m48.722313508s) [noise=true , address=77.90.31.14:4661 , piece_id=2AVXMHD667SLHIVB2YS3ZKNQYTXCBZ724FBBYUJWKVRSYFVE6MWA, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.171592384 (318h46m48.723062845s) [noise=true , address=145.79.227.22:4511 , piece_id=FT6YZ2TFUKJS2P6KVKJDQT2GNKACJI2GROSBBU2ATATICOXVJEJQ, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.173336084 (318h46m48.724806545s) [noise=true , address=145.79.227.19:4078 , piece_id=6G7DJTWC54HYI6S3XP5EIHC2VZA4MYTYZCSKQQEGMUHAFMHHIWOA, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.175619112 (318h46m48.727089573s) [noise=true , address=145.79.227.9:4367 , piece_id=66X6OMDN7R74TFNCFSKZHOUGX75JWLYT7HI77MAEPGCJBRBYSENA, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.178391049 (318h46m48.72986151s) [noise=true , address=45.91.116.6:4974 , piece_id=OZOBWBFMONEMFWIVFJAUW7URGRWIR4HBUF7FMPK5F2ASAXVUPLYQ, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:52.218800746 (318h46m50.770271197s) [upload=280745]: Next returned nil segment Nov 22 14:50:52.218818249 (318h46m50.77028871s) [upload=280745]: Waiting for error group managing segments... Nov 22 14:50:52.677314588 (318h46m51.228785049s) [noise=true , address=77.90.31.16:4734 , piece_id=POCRBOZSUZKROP5PPYVWE6ZJBRBIZ5N6NUZBEY37JWXOK7FWCX4A, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:50:53.035891433 (318h46m51.587361914s) [noise=true , address=45.91.116.9:4261 , piece_id=YISLO23F3HLAR2HLK3EL6ZDXDJY655XNXRYWAFWM2GSWSM6DOGZA, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:50:53.142171160 (318h46m51.693641621s) [noise=true , address=77.90.31.18:4807 , piece_id=C347B3KZ7NLDOPNXDOPU3TSVD4B2EMYBGVGISHGSG4GIDZEGPANQ, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:50:53.148762440 (318h46m51.700232901s) [noise=true , address=45.91.116.6:4974 , piece_id=OZOBWBFMONEMFWIVFJAUW7URGRWIR4HBUF7FMPK5F2ASAXVUPLYQ, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:50:53.169042227 (318h46m51.720512688s) [noise=true , address=77.90.31.15:4697 , piece_id=FT2H7VSGGQPFMOAUZKWA5YUCARSZBDBJAVDZEQIHQSVNTJTRZS4Q, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:50:53.525428209 (318h46m52.07689867s) [noise=true , address=77.90.31.14:4661 , piece_id=2AVXMHD667SLHIVB2YS3ZKNQYTXCBZ724FBBYUJWKVRSYFVE6MWA, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:51:02.564730655 (318h47m1.116201116s) [noise=true , address=145.79.227.20:4439 , piece_id=HUBBZOWAZ3B4GBMUV4E3SFMUEDA5757BSX7Q4EV5VHZ25B3V4EPA, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:51:03.499338414 (318h47m2.050808885s) [noise=true , address=145.79.227.19:4078 , piece_id=6G7DJTWC54HYI6S3XP5EIHC2VZA4MYTYZCSKQQEGMUHAFMHHIWOA, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:51:04.992927832 (318h47m3.544398293s) [noise=true , address=145.79.227.22:4511 , piece_id=FT6YZ2TFUKJS2P6KVKJDQT2GNKACJI2GROSBBU2ATATICOXVJEJQ, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:51:14.888732580 (318h47m13.440203041s) [noise=true , address=145.79.227.21:4474 , piece_id=HK2OT4RLO7OLQR46P6OU6XE3652OV6ECU4EAF3ABM3MGL3O2XK6A, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:51:17.793451829 (318h47m16.34492229s) [noise=true , address=145.79.227.31:4798 , piece_id=GHZGFXHRAPEUBQ5MAVBLLN2TH2TLPOYZU4NHO6GJ7WCJKQXKYX5A, seg_pos={0 0}, upload=280745]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:60870->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:60870->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60870->145.79.227.31:4798: use of closed network connection Nov 22 14:51:17.793649536 (318h47m16.345119997s) [noise=true , address=145.79.227.21:4473 , piece_id=TNNL4WDT7WL5QRVJOK2HWZBSAIYV4YVV3APHUMKTOYZAXYHBMEKA, seg_pos={0 0}, upload=280745]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:36642->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:36642->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36642->145.79.227.21:4473: use of closed network connection Nov 22 14:51:17.793741841 (318h47m16.345212302s) [noise=true , address=145.79.227.26:4653 , piece_id=F3HR6ZS7KLN3KMJPUDP6PDERMF5YMVFP7JV2JM7DBG4PDU2XKUZQ, seg_pos={0 0}, upload=280745]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:40892->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:40892->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40892->145.79.227.26:4653: use of closed network connection Nov 22 14:51:17.793873091 (318h47m16.345343552s) [noise=true , address=145.79.227.25:4619 , piece_id=OPX6FGOO77ZNHAFYKJQYP34Y6FHXGCL7OVFVUXSCX2PTYJHLJFEQ, seg_pos={0 0}, upload=280745]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: write tcp 188.166.225.170:51940->145.79.227.25:4619: use of closed network connection; write tcp 188.166.225.170:51940->145.79.227.25:4619: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51940->145.79.227.25:4619: use of closed network connection Nov 22 14:51:17.793963053 (318h47m16.345433514s) [noise=true , address=145.79.227.23:4546 , piece_id=YPCRIDSXM3ZMQZKRTEPHO5ANUB6Q5QYLW63WRPOM3U2N4I574DFA, seg_pos={0 0}, upload=280745]: Done uploading piece. err: ecclient: upload failed (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5, address:145.79.227.23:4546): protocol: write tcp 188.166.225.170:58790->145.79.227.23:4546: use of closed network connection; write tcp 188.166.225.170:58790->145.79.227.23:4546: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58790->145.79.227.23:4546: use of closed network connection Nov 22 14:51:17.794537025 (318h47m16.346007486s) [noise=true , address=145.79.227.9:4367 , piece_id=66X6OMDN7R74TFNCFSKZHOUGX75JWLYT7HI77MAEPGCJBRBYSENA, seg_pos={0 0}, upload=280745]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: read tcp 188.166.225.170:59776->145.79.227.9:4367: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:51:17.794622438 (318h47m16.346092889s) [seg_pos={0 0}, upload=280745]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 22 14:51:18.431733399 (318h47m16.98320385s) [upload=280745]: Done uploading segments... Nov 22 14:51:18.431769258 (318h47m16.983239709s) [upload=280745]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration:* 15.896739986s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [...X........X.......XX..X..XX..X.X.........XX.XX............] duration: 3.484078851s trace: https://jaeger.dev.storj.io/trace/58dc5585e5ee8756 FAIL (2025-11-22 14:57:07 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.484098148s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..X.....X.........X.X.....X.......X......X............X.....] duration: 6.078270298s trace: https://jaeger.dev.storj.io/trace/1ee0c6bff9434222 FAIL (2025-11-22 14:56:55 UTC) orbiter warning: context deadline exceeded Duration: 6.078329852s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [...X.....X.........X.X.....X.......X......X............X....] duration: 12.007242398s trace: https://jaeger.dev.storj.io/trace/1ee0c6bff9434222 FAIL (2025-11-22 14:56:45 UTC) orbiter warning: context deadline exceeded Duration: 12.00732206s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [........................................XX.X.........X......] duration: 16.023835981s trace: https://jaeger.dev.storj.io/trace/1f18db6043613191 FAIL (2025-11-22 14:56:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== pload=280745]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 14:50:50.164295669 (318h46m48.71576612s) [seg_pos={0 0}, upload=280745]: Begin upload segment... Nov 22 14:50:50.164353229 (318h46m48.71582369s) [seg_pos={0 0}, upload=280745]: Done begin upload segment. Nov 22 14:50:50.164374059 (318h46m48.71584452s) [noise=true , address=145.79.227.25:4619 , piece_id=OPX6FGOO77ZNHAFYKJQYP34Y6FHXGCL7OVFVUXSCX2PTYJHLJFEQ, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.164385731 (318h46m48.715856182s) [noise=true , address=77.90.31.16:4734 , piece_id=POCRBOZSUZKROP5PPYVWE6ZJBRBIZ5N6NUZBEY37JWXOK7FWCX4A, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.164503194 (318h46m48.715973656s) [noise=true , address=145.79.227.21:4474 , piece_id=HK2OT4RLO7OLQR46P6OU6XE3652OV6ECU4EAF3ABM3MGL3O2XK6A, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.165229057 (318h46m48.716699518s) [noise=true , address=45.91.116.9:4261 , piece_id=YISLO23F3HLAR2HLK3EL6ZDXDJY655XNXRYWAFWM2GSWSM6DOGZA, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.165688792 (318h46m48.717159253s) [noise=true , address=145.79.227.20:4439 , piece_id=HUBBZOWAZ3B4GBMUV4E3SFMUEDA5757BSX7Q4EV5VHZ25B3V4EPA, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.166155391 (318h46m48.717625852s) [noise=true , address=77.90.31.15:4697 , piece_id=FT2H7VSGGQPFMOAUZKWA5YUCARSZBDBJAVDZEQIHQSVNTJTRZS4Q, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.166692824 (318h46m48.718163295s) [noise=true , address=145.79.227.31:4798 , piece_id=GHZGFXHRAPEUBQ5MAVBLLN2TH2TLPOYZU4NHO6GJ7WCJKQXKYX5A, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.167166076 (318h46m48.718636537s) [noise=true , address=77.90.31.18:4807 , piece_id=C347B3KZ7NLDOPNXDOPU3TSVD4B2EMYBGVGISHGSG4GIDZEGPANQ, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.168429381 (318h46m48.719899852s) [noise=true , address=145.79.227.21:4473 , piece_id=TNNL4WDT7WL5QRVJOK2HWZBSAIYV4YVV3APHUMKTOYZAXYHBMEKA, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.169388678 (318h46m48.720859139s) [noise=true , address=145.79.227.26:4653 , piece_id=F3HR6ZS7KLN3KMJPUDP6PDERMF5YMVFP7JV2JM7DBG4PDU2XKUZQ, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.169688388 (318h46m48.721158949s) [noise=true , address=145.79.227.23:4546 , piece_id=YPCRIDSXM3ZMQZKRTEPHO5ANUB6Q5QYLW63WRPOM3U2N4I574DFA, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.170843047 (318h46m48.722313508s) [noise=true , address=77.90.31.14:4661 , piece_id=2AVXMHD667SLHIVB2YS3ZKNQYTXCBZ724FBBYUJWKVRSYFVE6MWA, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.171592384 (318h46m48.723062845s) [noise=true , address=145.79.227.22:4511 , piece_id=FT6YZ2TFUKJS2P6KVKJDQT2GNKACJI2GROSBBU2ATATICOXVJEJQ, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.173336084 (318h46m48.724806545s) [noise=true , address=145.79.227.19:4078 , piece_id=6G7DJTWC54HYI6S3XP5EIHC2VZA4MYTYZCSKQQEGMUHAFMHHIWOA, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.175619112 (318h46m48.727089573s) [noise=true , address=145.79.227.9:4367 , piece_id=66X6OMDN7R74TFNCFSKZHOUGX75JWLYT7HI77MAEPGCJBRBYSENA, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:50.178391049 (318h46m48.72986151s) [noise=true , address=45.91.116.6:4974 , piece_id=OZOBWBFMONEMFWIVFJAUW7URGRWIR4HBUF7FMPK5F2ASAXVUPLYQ, seg_pos={0 0}, upload=280745]: Uploading piece... Nov 22 14:50:52.218800746 (318h46m50.770271197s) [upload=280745]: Next returned nil segment Nov 22 14:50:52.218818249 (318h46m50.77028871s) [upload=280745]: Waiting for error group managing segments... Nov 22 14:50:52.677314588 (318h46m51.228785049s) [noise=true , address=77.90.31.16:4734 , piece_id=POCRBOZSUZKROP5PPYVWE6ZJBRBIZ5N6NUZBEY37JWXOK7FWCX4A, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:50:53.035891433 (318h46m51.587361914s) [noise=true , address=45.91.116.9:4261 , piece_id=YISLO23F3HLAR2HLK3EL6ZDXDJY655XNXRYWAFWM2GSWSM6DOGZA, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:50:53.142171160 (318h46m51.693641621s) [noise=true , address=77.90.31.18:4807 , piece_id=C347B3KZ7NLDOPNXDOPU3TSVD4B2EMYBGVGISHGSG4GIDZEGPANQ, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:50:53.148762440 (318h46m51.700232901s) [noise=true , address=45.91.116.6:4974 , piece_id=OZOBWBFMONEMFWIVFJAUW7URGRWIR4HBUF7FMPK5F2ASAXVUPLYQ, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:50:53.169042227 (318h46m51.720512688s) [noise=true , address=77.90.31.15:4697 , piece_id=FT2H7VSGGQPFMOAUZKWA5YUCARSZBDBJAVDZEQIHQSVNTJTRZS4Q, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:50:53.525428209 (318h46m52.07689867s) [noise=true , address=77.90.31.14:4661 , piece_id=2AVXMHD667SLHIVB2YS3ZKNQYTXCBZ724FBBYUJWKVRSYFVE6MWA, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:51:02.564730655 (318h47m1.116201116s) [noise=true , address=145.79.227.20:4439 , piece_id=HUBBZOWAZ3B4GBMUV4E3SFMUEDA5757BSX7Q4EV5VHZ25B3V4EPA, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:51:03.499338414 (318h47m2.050808885s) [noise=true , address=145.79.227.19:4078 , piece_id=6G7DJTWC54HYI6S3XP5EIHC2VZA4MYTYZCSKQQEGMUHAFMHHIWOA, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:51:04.992927832 (318h47m3.544398293s) [noise=true , address=145.79.227.22:4511 , piece_id=FT6YZ2TFUKJS2P6KVKJDQT2GNKACJI2GROSBBU2ATATICOXVJEJQ, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:51:14.888732580 (318h47m13.440203041s) [noise=true , address=145.79.227.21:4474 , piece_id=HK2OT4RLO7OLQR46P6OU6XE3652OV6ECU4EAF3ABM3MGL3O2XK6A, seg_pos={0 0}, upload=280745]: Done uploading piece. err: <nil> Nov 22 14:51:17.793451829 (318h47m16.34492229s) [noise=true , address=145.79.227.31:4798 , piece_id=GHZGFXHRAPEUBQ5MAVBLLN2TH2TLPOYZU4NHO6GJ7WCJKQXKYX5A, seg_pos={0 0}, upload=280745]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:60870->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:60870->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60870->145.79.227.31:4798: use of closed network connection Nov 22 14:51:17.793649536 (318h47m16.345119997s) [noise=true , address=145.79.227.21:4473 , piece_id=TNNL4WDT7WL5QRVJOK2HWZBSAIYV4YVV3APHUMKTOYZAXYHBMEKA, seg_pos={0 0}, upload=280745]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:36642->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:36642->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36642->145.79.227.21:4473: use of closed network connection Nov 22 14:51:17.793741841 (318h47m16.345212302s) [noise=true , address=145.79.227.26:4653 , piece_id=F3HR6ZS7KLN3KMJPUDP6PDERMF5YMVFP7JV2JM7DBG4PDU2XKUZQ, seg_pos={0 0}, upload=280745]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: write tcp 188.166.225.170:40892->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:40892->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40892->145.79.227.26:4653: use of closed network connection Nov 22 14:51:17.793873091 (318h47m16.345343552s) [noise=true , address=145.79.227.25:4619 , piece_id=OPX6FGOO77ZNHAFYKJQYP34Y6FHXGCL7OVFVUXSCX2PTYJHLJFEQ, seg_pos={0 0}, upload=280745]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: write tcp 188.166.225.170:51940->145.79.227.25:4619: use of closed network connection; write tcp 188.166.225.170:51940->145.79.227.25:4619: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51940->145.79.227.25:4619: use of closed network connection Nov 22 14:51:17.793963053 (318h47m16.345433514s) [noise=true , address=145.79.227.23:4546 , piece_id=YPCRIDSXM3ZMQZKRTEPHO5ANUB6Q5QYLW63WRPOM3U2N4I574DFA, seg_pos={0 0}, upload=280745]: Done uploading piece. err: ecclient: upload failed (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5, address:145.79.227.23:4546): protocol: write tcp 188.166.225.170:58790->145.79.227.23:4546: use of closed network connection; write tcp 188.166.225.170:58790->145.79.227.23:4546: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58790->145.79.227.23:4546: use of closed network connection Nov 22 14:51:17.794537025 (318h47m16.346007486s) [noise=true , address=145.79.227.9:4367 , piece_id=66X6OMDN7R74TFNCFSKZHOUGX75JWLYT7HI77MAEPGCJBRBYSENA, seg_pos={0 0}, upload=280745]: Done uploading piece. err: ecclient: upload failed (node:12CMjNpf3aMdqEYea6xSrAWGoEMzfy4SbEjXywZN7JPRCbvDpK6, address:145.79.227.9:4367): protocol: read tcp 188.166.225.170:59776->145.79.227.9:4367: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:51:17.794622438 (318h47m16.346092889s) [seg_pos={0 0}, upload=280745]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 22 14:51:18.431733399 (318h47m16.98320385s) [upload=280745]: Done uploading segments... Nov 22 14:51:18.431769258 (318h47m16.983239709s) [upload=280745]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration:* 16.023865747s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [....X........X.......XX..X..XX..X.X.........XX.XX...........] duration: 3.556001685s trace: https://jaeger.dev.storj.io/trace/58dc5585e5ee8756 FAIL (2025-11-22 14:56:38 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.556032926s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)