Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [X.X.....X.......X......X............X.................X.....] duration: 16.820412832s trace: https://jaeger.dev.storj.io/trace/2af0ef576bba07f6 FAIL (2025-11-22 15:10:42 UTC) orbiter warning: context deadline exceeded Duration: 16.820447027s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...................XX.X.........X........X..................] duration: 14.078981084s trace: https://jaeger.dev.storj.io/trace/1efcdf5c2a183d8d FAIL (2025-11-22 15:10:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 934958555s) [noise=true , address=145.79.227.30:4762 , piece_id=4Z7CCNLZEYQAGJNUWGJ2USQSVW6QZUTCNFKC3JWRIOKV6VBVIYKQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.384231169 (318h54m50.93570163s) [noise=true , address=145.79.227.18:4007 , piece_id=KM5TO27LZXJZZAT2DB5JTP7R5LOPJHUU6T34RZI7HH7PRWPW4ZNQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.384687758 (318h54m50.93615822s) [noise=true , address=145.79.227.26:4654 , piece_id=VCRM5O6NHOJOKNQUG5TTZUT3JIDQOMNF2DD6H7KWUJFOQ3P75QGQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.385168865 (318h54m50.936639326s) [noise=true , address=77.90.31.18:4805 , piece_id=3EHB5IFBQRLXFBOVCT25RS4CIXVMSVOE7EKDV26DXOEY2NDJ2YGA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.385605395 (318h54m50.937075867s) [noise=true , address=145.79.227.32:4835 , piece_id=7GD4QUTGLIPOEU25FIDE7VDO3AUJ2RZDU4XKI476KNU6QTETC52A, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.386294879 (318h54m50.93776534s) [noise=true , address=145.79.227.21:4473 , piece_id=D5FC7PVECZTQR64BFVTV3L2HOXCITABN57LLNRSMVCSGQICRYKUQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.386958642 (318h54m50.938429103s) [noise=true , address=145.79.227.14:4837 , piece_id=NR427EUHKVSQ6HMMZCHL4LCBV6VRVTANZDXMX6JHQI2TTQN5R5VQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387097236 (318h54m50.938567687s) [noise=true , address=145.79.227.9:4365 , piece_id=DBW2L6GHAWIHOMOZOTQFQ57Y2MA6KP6SGUO6AJDDF65ERQ324VZQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387456781 (318h54m50.938927242s) [noise=true , address=145.79.227.19:4078 , piece_id=ER3KKYJ2MJQC3HRRBVULC3ING5GLGOHFVD2QBWF2MVAXF6ODMUUA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387755300 (318h54m50.939225761s) [noise=true , address=145.79.227.28:4725 , piece_id=HVMAX2UWJIF2UJQIJS5UBTO5TOIJI5DCDGGURZN4SOMRY3Z4OF4A, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.388354220 (318h54m50.939824681s) [noise=true , address=145.79.227.10:4402 , piece_id=ZU6A2O2R73PIVIDF5NOKMJZH4QX4LOCFSYOQLFOFAA5BEVQH52KA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.388862268 (318h54m50.940332729s) [noise=true , address=145.79.227.22:4509 , piece_id=WRHXVGJZQRTY6ZKMRU6IFYZSMLR6UATXFZOTCPRYKHERA2WKMGNA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.389411814 (318h54m50.940882275s) [noise=true , address=145.79.227.8:4330 , piece_id=LFTZWYQKW56VLMJPOGGLB6IJILPGJNEHIRIYD2F4J34ZH24PCSMQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.389753224 (318h54m50.941223675s) [noise=true , address=145.79.227.16:4909 , piece_id=3DAVIY3ETZVG5CPP5BQB7QL473J5HZG6EQ6BYZBIAN6TWZ5FGNHQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.391102172 (318h54m50.942572633s) [noise=true , address=45.91.116.7:4010 , piece_id=VZCYHOAR73DVCRDT5GBT65JI53NP27OEGA3VYLM66LPQR23T4MTA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:54.404583235 (318h54m52.956053696s) [upload=280863]: Next returned nil segment Nov 22 14:58:54.404601019 (318h54m52.956071481s) [upload=280863]: Waiting for error group managing segments... Nov 22 14:58:54.928222688 (318h54m53.479693149s) [noise=true , address=77.90.31.18:4805 , piece_id=3EHB5IFBQRLXFBOVCT25RS4CIXVMSVOE7EKDV26DXOEY2NDJ2YGA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:58:55.175979631 (318h54m53.727450092s) [noise=true , address=45.91.116.7:4010 , piece_id=VZCYHOAR73DVCRDT5GBT65JI53NP27OEGA3VYLM66LPQR23T4MTA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:03.279622498 (318h55m1.831092959s) [noise=true , address=145.79.227.14:4837 , piece_id=NR427EUHKVSQ6HMMZCHL4LCBV6VRVTANZDXMX6JHQI2TTQN5R5VQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:04.274652248 (318h55m2.826122709s) [noise=true , address=145.79.227.28:4725 , piece_id=HVMAX2UWJIF2UJQIJS5UBTO5TOIJI5DCDGGURZN4SOMRY3Z4OF4A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:05.089136230 (318h55m3.640606701s) [noise=true , address=145.79.227.22:4509 , piece_id=WRHXVGJZQRTY6ZKMRU6IFYZSMLR6UATXFZOTCPRYKHERA2WKMGNA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:05.587545699 (318h55m4.13901616s) [noise=true , address=145.79.227.32:4835 , piece_id=7GD4QUTGLIPOEU25FIDE7VDO3AUJ2RZDU4XKI476KNU6QTETC52A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:06.956814829 (318h55m5.50828528s) [noise=true , address=145.79.227.8:4330 , piece_id=LFTZWYQKW56VLMJPOGGLB6IJILPGJNEHIRIYD2F4J34ZH24PCSMQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:07.254777763 (318h55m5.806248224s) [noise=true , address=145.79.227.30:4762 , piece_id=4Z7CCNLZEYQAGJNUWGJ2USQSVW6QZUTCNFKC3JWRIOKV6VBVIYKQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:19.983388333 (318h55m18.534858794s) [noise=true , address=145.79.227.10:4402 , piece_id=ZU6A2O2R73PIVIDF5NOKMJZH4QX4LOCFSYOQLFOFAA5BEVQH52KA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection Nov 22 14:59:19.983536355 (318h55m18.535006816s) [noise=true , address=145.79.227.21:4473 , piece_id=D5FC7PVECZTQR64BFVTV3L2HOXCITABN57LLNRSMVCSGQICRYKUQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection Nov 22 14:59:19.983802211 (318h55m18.535272703s) [noise=true , address=145.79.227.18:4007 , piece_id=KM5TO27LZXJZZAT2DB5JTP7R5LOPJHUU6T34RZI7HH7PRWPW4ZNQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection Nov 22 14:59:19.983812000 (318h55m18.535282461s) [noise=true , address=145.79.227.9:4365 , piece_id=DBW2L6GHAWIHOMOZOTQFQ57Y2MA6KP6SGUO6AJDDF65ERQ324VZQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection Nov 22 14:59:19.983896611 (318h55m18.535367073s) [noise=true , address=145.79.227.19:4078 , piece_id=ER3KKYJ2MJQC3HRRBVULC3ING5GLGOHFVD2QBWF2MVAXF6ODMUUA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection; write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection Nov 22 14:59:19.984068268 (318h55m18.535538719s) [noise=true , address=145.79.227.26:4654 , piece_id=VCRM5O6NHOJOKNQUG5TTZUT3JIDQOMNF2DD6H7KWUJFOQ3P75QGQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: read tcp 188.166.225.170:36440->145.79.227.26:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984229324 (318h55m18.535699776s) [noise=true , address=145.79.227.8:4329 , piece_id=YJG3LYBDMPS4QQNLG24MS7ZWUM6EOI6EPDGVBGLO33CVEQQKGX6A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: read tcp 188.166.225.170:44902->145.79.227.8:4329: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984374972 (318h55m18.535845423s) [noise=true , address=145.79.227.16:4909 , piece_id=3DAVIY3ETZVG5CPP5BQB7QL473J5HZG6EQ6BYZBIAN6TWZ5FGNHQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: read tcp 188.166.225.170:59064->145.79.227.16:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984400911 (318h55m18.535871362s) [seg_pos={0 0}, upload=280863]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 22 14:59:20.539921660 (318h55m19.091392121s) [upload=280863]: Done uploading segments... Nov 22 14:59:20.539961818 (318h55m19.091432269s) [upload=280863]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 14.079006182s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 15 out of 60 is failed (threshold=6) [....XX..X..XX..X.X.........XX.XX.................XX..X....X.] duration: 3.882667285s trace: https://jaeger.dev.storj.io/trace/21c30b14eb6996aa FAIL (2025-11-22 15:10:11 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.882699646s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.X.X.....X.......X......X............X.................X....] duration: 7.558803397s trace: https://jaeger.dev.storj.io/trace/2af0ef576bba07f6 FAIL (2025-11-22 15:09:39 UTC) orbiter warning: context deadline exceeded Duration: 7.558831471s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [....................XX.X.........X........X.................] duration: 14.607385422s trace: https://jaeger.dev.storj.io/trace/1efcdf5c2a183d8d FAIL (2025-11-22 15:09:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 934958555s) [noise=true , address=145.79.227.30:4762 , piece_id=4Z7CCNLZEYQAGJNUWGJ2USQSVW6QZUTCNFKC3JWRIOKV6VBVIYKQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.384231169 (318h54m50.93570163s) [noise=true , address=145.79.227.18:4007 , piece_id=KM5TO27LZXJZZAT2DB5JTP7R5LOPJHUU6T34RZI7HH7PRWPW4ZNQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.384687758 (318h54m50.93615822s) [noise=true , address=145.79.227.26:4654 , piece_id=VCRM5O6NHOJOKNQUG5TTZUT3JIDQOMNF2DD6H7KWUJFOQ3P75QGQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.385168865 (318h54m50.936639326s) [noise=true , address=77.90.31.18:4805 , piece_id=3EHB5IFBQRLXFBOVCT25RS4CIXVMSVOE7EKDV26DXOEY2NDJ2YGA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.385605395 (318h54m50.937075867s) [noise=true , address=145.79.227.32:4835 , piece_id=7GD4QUTGLIPOEU25FIDE7VDO3AUJ2RZDU4XKI476KNU6QTETC52A, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.386294879 (318h54m50.93776534s) [noise=true , address=145.79.227.21:4473 , piece_id=D5FC7PVECZTQR64BFVTV3L2HOXCITABN57LLNRSMVCSGQICRYKUQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.386958642 (318h54m50.938429103s) [noise=true , address=145.79.227.14:4837 , piece_id=NR427EUHKVSQ6HMMZCHL4LCBV6VRVTANZDXMX6JHQI2TTQN5R5VQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387097236 (318h54m50.938567687s) [noise=true , address=145.79.227.9:4365 , piece_id=DBW2L6GHAWIHOMOZOTQFQ57Y2MA6KP6SGUO6AJDDF65ERQ324VZQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387456781 (318h54m50.938927242s) [noise=true , address=145.79.227.19:4078 , piece_id=ER3KKYJ2MJQC3HRRBVULC3ING5GLGOHFVD2QBWF2MVAXF6ODMUUA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387755300 (318h54m50.939225761s) [noise=true , address=145.79.227.28:4725 , piece_id=HVMAX2UWJIF2UJQIJS5UBTO5TOIJI5DCDGGURZN4SOMRY3Z4OF4A, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.388354220 (318h54m50.939824681s) [noise=true , address=145.79.227.10:4402 , piece_id=ZU6A2O2R73PIVIDF5NOKMJZH4QX4LOCFSYOQLFOFAA5BEVQH52KA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.388862268 (318h54m50.940332729s) [noise=true , address=145.79.227.22:4509 , piece_id=WRHXVGJZQRTY6ZKMRU6IFYZSMLR6UATXFZOTCPRYKHERA2WKMGNA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.389411814 (318h54m50.940882275s) [noise=true , address=145.79.227.8:4330 , piece_id=LFTZWYQKW56VLMJPOGGLB6IJILPGJNEHIRIYD2F4J34ZH24PCSMQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.389753224 (318h54m50.941223675s) [noise=true , address=145.79.227.16:4909 , piece_id=3DAVIY3ETZVG5CPP5BQB7QL473J5HZG6EQ6BYZBIAN6TWZ5FGNHQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.391102172 (318h54m50.942572633s) [noise=true , address=45.91.116.7:4010 , piece_id=VZCYHOAR73DVCRDT5GBT65JI53NP27OEGA3VYLM66LPQR23T4MTA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:54.404583235 (318h54m52.956053696s) [upload=280863]: Next returned nil segment Nov 22 14:58:54.404601019 (318h54m52.956071481s) [upload=280863]: Waiting for error group managing segments... Nov 22 14:58:54.928222688 (318h54m53.479693149s) [noise=true , address=77.90.31.18:4805 , piece_id=3EHB5IFBQRLXFBOVCT25RS4CIXVMSVOE7EKDV26DXOEY2NDJ2YGA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:58:55.175979631 (318h54m53.727450092s) [noise=true , address=45.91.116.7:4010 , piece_id=VZCYHOAR73DVCRDT5GBT65JI53NP27OEGA3VYLM66LPQR23T4MTA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:03.279622498 (318h55m1.831092959s) [noise=true , address=145.79.227.14:4837 , piece_id=NR427EUHKVSQ6HMMZCHL4LCBV6VRVTANZDXMX6JHQI2TTQN5R5VQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:04.274652248 (318h55m2.826122709s) [noise=true , address=145.79.227.28:4725 , piece_id=HVMAX2UWJIF2UJQIJS5UBTO5TOIJI5DCDGGURZN4SOMRY3Z4OF4A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:05.089136230 (318h55m3.640606701s) [noise=true , address=145.79.227.22:4509 , piece_id=WRHXVGJZQRTY6ZKMRU6IFYZSMLR6UATXFZOTCPRYKHERA2WKMGNA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:05.587545699 (318h55m4.13901616s) [noise=true , address=145.79.227.32:4835 , piece_id=7GD4QUTGLIPOEU25FIDE7VDO3AUJ2RZDU4XKI476KNU6QTETC52A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:06.956814829 (318h55m5.50828528s) [noise=true , address=145.79.227.8:4330 , piece_id=LFTZWYQKW56VLMJPOGGLB6IJILPGJNEHIRIYD2F4J34ZH24PCSMQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:07.254777763 (318h55m5.806248224s) [noise=true , address=145.79.227.30:4762 , piece_id=4Z7CCNLZEYQAGJNUWGJ2USQSVW6QZUTCNFKC3JWRIOKV6VBVIYKQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:19.983388333 (318h55m18.534858794s) [noise=true , address=145.79.227.10:4402 , piece_id=ZU6A2O2R73PIVIDF5NOKMJZH4QX4LOCFSYOQLFOFAA5BEVQH52KA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection Nov 22 14:59:19.983536355 (318h55m18.535006816s) [noise=true , address=145.79.227.21:4473 , piece_id=D5FC7PVECZTQR64BFVTV3L2HOXCITABN57LLNRSMVCSGQICRYKUQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection Nov 22 14:59:19.983802211 (318h55m18.535272703s) [noise=true , address=145.79.227.18:4007 , piece_id=KM5TO27LZXJZZAT2DB5JTP7R5LOPJHUU6T34RZI7HH7PRWPW4ZNQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection Nov 22 14:59:19.983812000 (318h55m18.535282461s) [noise=true , address=145.79.227.9:4365 , piece_id=DBW2L6GHAWIHOMOZOTQFQ57Y2MA6KP6SGUO6AJDDF65ERQ324VZQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection Nov 22 14:59:19.983896611 (318h55m18.535367073s) [noise=true , address=145.79.227.19:4078 , piece_id=ER3KKYJ2MJQC3HRRBVULC3ING5GLGOHFVD2QBWF2MVAXF6ODMUUA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection; write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection Nov 22 14:59:19.984068268 (318h55m18.535538719s) [noise=true , address=145.79.227.26:4654 , piece_id=VCRM5O6NHOJOKNQUG5TTZUT3JIDQOMNF2DD6H7KWUJFOQ3P75QGQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: read tcp 188.166.225.170:36440->145.79.227.26:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984229324 (318h55m18.535699776s) [noise=true , address=145.79.227.8:4329 , piece_id=YJG3LYBDMPS4QQNLG24MS7ZWUM6EOI6EPDGVBGLO33CVEQQKGX6A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: read tcp 188.166.225.170:44902->145.79.227.8:4329: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984374972 (318h55m18.535845423s) [noise=true , address=145.79.227.16:4909 , piece_id=3DAVIY3ETZVG5CPP5BQB7QL473J5HZG6EQ6BYZBIAN6TWZ5FGNHQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: read tcp 188.166.225.170:59064->145.79.227.16:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984400911 (318h55m18.535871362s) [seg_pos={0 0}, upload=280863]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 22 14:59:20.539921660 (318h55m19.091392121s) [upload=280863]: Done uploading segments... Nov 22 14:59:20.539961818 (318h55m19.091432269s) [upload=280863]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 14.60740566s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 15 out of 60 is failed (threshold=6) [.....XX..X..XX..X.X.........XX.XX.................XX..X....X] duration: 5.001077619s trace: https://jaeger.dev.storj.io/trace/21c30b14eb6996aa FAIL (2025-11-22 15:08:57 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001091134s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [..X.X.....X.......X......X............X.................X...] duration: 8.404878987s trace: https://jaeger.dev.storj.io/trace/2af0ef576bba07f6 FAIL (2025-11-22 15:08:39 UTC) orbiter warning: context deadline exceeded Duration: 8.404911119s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....................XX.X.........X........X................] duration: 14.691171375s trace: https://jaeger.dev.storj.io/trace/1efcdf5c2a183d8d FAIL (2025-11-22 15:08:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 934958555s) [noise=true , address=145.79.227.30:4762 , piece_id=4Z7CCNLZEYQAGJNUWGJ2USQSVW6QZUTCNFKC3JWRIOKV6VBVIYKQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.384231169 (318h54m50.93570163s) [noise=true , address=145.79.227.18:4007 , piece_id=KM5TO27LZXJZZAT2DB5JTP7R5LOPJHUU6T34RZI7HH7PRWPW4ZNQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.384687758 (318h54m50.93615822s) [noise=true , address=145.79.227.26:4654 , piece_id=VCRM5O6NHOJOKNQUG5TTZUT3JIDQOMNF2DD6H7KWUJFOQ3P75QGQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.385168865 (318h54m50.936639326s) [noise=true , address=77.90.31.18:4805 , piece_id=3EHB5IFBQRLXFBOVCT25RS4CIXVMSVOE7EKDV26DXOEY2NDJ2YGA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.385605395 (318h54m50.937075867s) [noise=true , address=145.79.227.32:4835 , piece_id=7GD4QUTGLIPOEU25FIDE7VDO3AUJ2RZDU4XKI476KNU6QTETC52A, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.386294879 (318h54m50.93776534s) [noise=true , address=145.79.227.21:4473 , piece_id=D5FC7PVECZTQR64BFVTV3L2HOXCITABN57LLNRSMVCSGQICRYKUQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.386958642 (318h54m50.938429103s) [noise=true , address=145.79.227.14:4837 , piece_id=NR427EUHKVSQ6HMMZCHL4LCBV6VRVTANZDXMX6JHQI2TTQN5R5VQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387097236 (318h54m50.938567687s) [noise=true , address=145.79.227.9:4365 , piece_id=DBW2L6GHAWIHOMOZOTQFQ57Y2MA6KP6SGUO6AJDDF65ERQ324VZQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387456781 (318h54m50.938927242s) [noise=true , address=145.79.227.19:4078 , piece_id=ER3KKYJ2MJQC3HRRBVULC3ING5GLGOHFVD2QBWF2MVAXF6ODMUUA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387755300 (318h54m50.939225761s) [noise=true , address=145.79.227.28:4725 , piece_id=HVMAX2UWJIF2UJQIJS5UBTO5TOIJI5DCDGGURZN4SOMRY3Z4OF4A, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.388354220 (318h54m50.939824681s) [noise=true , address=145.79.227.10:4402 , piece_id=ZU6A2O2R73PIVIDF5NOKMJZH4QX4LOCFSYOQLFOFAA5BEVQH52KA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.388862268 (318h54m50.940332729s) [noise=true , address=145.79.227.22:4509 , piece_id=WRHXVGJZQRTY6ZKMRU6IFYZSMLR6UATXFZOTCPRYKHERA2WKMGNA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.389411814 (318h54m50.940882275s) [noise=true , address=145.79.227.8:4330 , piece_id=LFTZWYQKW56VLMJPOGGLB6IJILPGJNEHIRIYD2F4J34ZH24PCSMQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.389753224 (318h54m50.941223675s) [noise=true , address=145.79.227.16:4909 , piece_id=3DAVIY3ETZVG5CPP5BQB7QL473J5HZG6EQ6BYZBIAN6TWZ5FGNHQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.391102172 (318h54m50.942572633s) [noise=true , address=45.91.116.7:4010 , piece_id=VZCYHOAR73DVCRDT5GBT65JI53NP27OEGA3VYLM66LPQR23T4MTA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:54.404583235 (318h54m52.956053696s) [upload=280863]: Next returned nil segment Nov 22 14:58:54.404601019 (318h54m52.956071481s) [upload=280863]: Waiting for error group managing segments... Nov 22 14:58:54.928222688 (318h54m53.479693149s) [noise=true , address=77.90.31.18:4805 , piece_id=3EHB5IFBQRLXFBOVCT25RS4CIXVMSVOE7EKDV26DXOEY2NDJ2YGA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:58:55.175979631 (318h54m53.727450092s) [noise=true , address=45.91.116.7:4010 , piece_id=VZCYHOAR73DVCRDT5GBT65JI53NP27OEGA3VYLM66LPQR23T4MTA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:03.279622498 (318h55m1.831092959s) [noise=true , address=145.79.227.14:4837 , piece_id=NR427EUHKVSQ6HMMZCHL4LCBV6VRVTANZDXMX6JHQI2TTQN5R5VQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:04.274652248 (318h55m2.826122709s) [noise=true , address=145.79.227.28:4725 , piece_id=HVMAX2UWJIF2UJQIJS5UBTO5TOIJI5DCDGGURZN4SOMRY3Z4OF4A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:05.089136230 (318h55m3.640606701s) [noise=true , address=145.79.227.22:4509 , piece_id=WRHXVGJZQRTY6ZKMRU6IFYZSMLR6UATXFZOTCPRYKHERA2WKMGNA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:05.587545699 (318h55m4.13901616s) [noise=true , address=145.79.227.32:4835 , piece_id=7GD4QUTGLIPOEU25FIDE7VDO3AUJ2RZDU4XKI476KNU6QTETC52A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:06.956814829 (318h55m5.50828528s) [noise=true , address=145.79.227.8:4330 , piece_id=LFTZWYQKW56VLMJPOGGLB6IJILPGJNEHIRIYD2F4J34ZH24PCSMQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:07.254777763 (318h55m5.806248224s) [noise=true , address=145.79.227.30:4762 , piece_id=4Z7CCNLZEYQAGJNUWGJ2USQSVW6QZUTCNFKC3JWRIOKV6VBVIYKQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:19.983388333 (318h55m18.534858794s) [noise=true , address=145.79.227.10:4402 , piece_id=ZU6A2O2R73PIVIDF5NOKMJZH4QX4LOCFSYOQLFOFAA5BEVQH52KA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection Nov 22 14:59:19.983536355 (318h55m18.535006816s) [noise=true , address=145.79.227.21:4473 , piece_id=D5FC7PVECZTQR64BFVTV3L2HOXCITABN57LLNRSMVCSGQICRYKUQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection Nov 22 14:59:19.983802211 (318h55m18.535272703s) [noise=true , address=145.79.227.18:4007 , piece_id=KM5TO27LZXJZZAT2DB5JTP7R5LOPJHUU6T34RZI7HH7PRWPW4ZNQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection Nov 22 14:59:19.983812000 (318h55m18.535282461s) [noise=true , address=145.79.227.9:4365 , piece_id=DBW2L6GHAWIHOMOZOTQFQ57Y2MA6KP6SGUO6AJDDF65ERQ324VZQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection Nov 22 14:59:19.983896611 (318h55m18.535367073s) [noise=true , address=145.79.227.19:4078 , piece_id=ER3KKYJ2MJQC3HRRBVULC3ING5GLGOHFVD2QBWF2MVAXF6ODMUUA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection; write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection Nov 22 14:59:19.984068268 (318h55m18.535538719s) [noise=true , address=145.79.227.26:4654 , piece_id=VCRM5O6NHOJOKNQUG5TTZUT3JIDQOMNF2DD6H7KWUJFOQ3P75QGQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: read tcp 188.166.225.170:36440->145.79.227.26:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984229324 (318h55m18.535699776s) [noise=true , address=145.79.227.8:4329 , piece_id=YJG3LYBDMPS4QQNLG24MS7ZWUM6EOI6EPDGVBGLO33CVEQQKGX6A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: read tcp 188.166.225.170:44902->145.79.227.8:4329: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984374972 (318h55m18.535845423s) [noise=true , address=145.79.227.16:4909 , piece_id=3DAVIY3ETZVG5CPP5BQB7QL473J5HZG6EQ6BYZBIAN6TWZ5FGNHQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: read tcp 188.166.225.170:59064->145.79.227.16:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984400911 (318h55m18.535871362s) [seg_pos={0 0}, upload=280863]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 22 14:59:20.539921660 (318h55m19.091392121s) [upload=280863]: Done uploading segments... Nov 22 14:59:20.539961818 (318h55m19.091432269s) [upload=280863]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 14.691203106s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [......XX..X..XX..X.X.........XX.XX.................XX..X....] duration: 3.526292344s trace: https://jaeger.dev.storj.io/trace/cc91f835756ebe2 FAIL (2025-11-22 15:07:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.526360653s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [......................XX.X.........X........X...............] duration: 15.522368531s trace: https://jaeger.dev.storj.io/trace/1efcdf5c2a183d8d FAIL (2025-11-22 15:07:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 934958555s) [noise=true , address=145.79.227.30:4762 , piece_id=4Z7CCNLZEYQAGJNUWGJ2USQSVW6QZUTCNFKC3JWRIOKV6VBVIYKQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.384231169 (318h54m50.93570163s) [noise=true , address=145.79.227.18:4007 , piece_id=KM5TO27LZXJZZAT2DB5JTP7R5LOPJHUU6T34RZI7HH7PRWPW4ZNQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.384687758 (318h54m50.93615822s) [noise=true , address=145.79.227.26:4654 , piece_id=VCRM5O6NHOJOKNQUG5TTZUT3JIDQOMNF2DD6H7KWUJFOQ3P75QGQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.385168865 (318h54m50.936639326s) [noise=true , address=77.90.31.18:4805 , piece_id=3EHB5IFBQRLXFBOVCT25RS4CIXVMSVOE7EKDV26DXOEY2NDJ2YGA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.385605395 (318h54m50.937075867s) [noise=true , address=145.79.227.32:4835 , piece_id=7GD4QUTGLIPOEU25FIDE7VDO3AUJ2RZDU4XKI476KNU6QTETC52A, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.386294879 (318h54m50.93776534s) [noise=true , address=145.79.227.21:4473 , piece_id=D5FC7PVECZTQR64BFVTV3L2HOXCITABN57LLNRSMVCSGQICRYKUQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.386958642 (318h54m50.938429103s) [noise=true , address=145.79.227.14:4837 , piece_id=NR427EUHKVSQ6HMMZCHL4LCBV6VRVTANZDXMX6JHQI2TTQN5R5VQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387097236 (318h54m50.938567687s) [noise=true , address=145.79.227.9:4365 , piece_id=DBW2L6GHAWIHOMOZOTQFQ57Y2MA6KP6SGUO6AJDDF65ERQ324VZQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387456781 (318h54m50.938927242s) [noise=true , address=145.79.227.19:4078 , piece_id=ER3KKYJ2MJQC3HRRBVULC3ING5GLGOHFVD2QBWF2MVAXF6ODMUUA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387755300 (318h54m50.939225761s) [noise=true , address=145.79.227.28:4725 , piece_id=HVMAX2UWJIF2UJQIJS5UBTO5TOIJI5DCDGGURZN4SOMRY3Z4OF4A, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.388354220 (318h54m50.939824681s) [noise=true , address=145.79.227.10:4402 , piece_id=ZU6A2O2R73PIVIDF5NOKMJZH4QX4LOCFSYOQLFOFAA5BEVQH52KA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.388862268 (318h54m50.940332729s) [noise=true , address=145.79.227.22:4509 , piece_id=WRHXVGJZQRTY6ZKMRU6IFYZSMLR6UATXFZOTCPRYKHERA2WKMGNA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.389411814 (318h54m50.940882275s) [noise=true , address=145.79.227.8:4330 , piece_id=LFTZWYQKW56VLMJPOGGLB6IJILPGJNEHIRIYD2F4J34ZH24PCSMQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.389753224 (318h54m50.941223675s) [noise=true , address=145.79.227.16:4909 , piece_id=3DAVIY3ETZVG5CPP5BQB7QL473J5HZG6EQ6BYZBIAN6TWZ5FGNHQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.391102172 (318h54m50.942572633s) [noise=true , address=45.91.116.7:4010 , piece_id=VZCYHOAR73DVCRDT5GBT65JI53NP27OEGA3VYLM66LPQR23T4MTA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:54.404583235 (318h54m52.956053696s) [upload=280863]: Next returned nil segment Nov 22 14:58:54.404601019 (318h54m52.956071481s) [upload=280863]: Waiting for error group managing segments... Nov 22 14:58:54.928222688 (318h54m53.479693149s) [noise=true , address=77.90.31.18:4805 , piece_id=3EHB5IFBQRLXFBOVCT25RS4CIXVMSVOE7EKDV26DXOEY2NDJ2YGA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:58:55.175979631 (318h54m53.727450092s) [noise=true , address=45.91.116.7:4010 , piece_id=VZCYHOAR73DVCRDT5GBT65JI53NP27OEGA3VYLM66LPQR23T4MTA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:03.279622498 (318h55m1.831092959s) [noise=true , address=145.79.227.14:4837 , piece_id=NR427EUHKVSQ6HMMZCHL4LCBV6VRVTANZDXMX6JHQI2TTQN5R5VQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:04.274652248 (318h55m2.826122709s) [noise=true , address=145.79.227.28:4725 , piece_id=HVMAX2UWJIF2UJQIJS5UBTO5TOIJI5DCDGGURZN4SOMRY3Z4OF4A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:05.089136230 (318h55m3.640606701s) [noise=true , address=145.79.227.22:4509 , piece_id=WRHXVGJZQRTY6ZKMRU6IFYZSMLR6UATXFZOTCPRYKHERA2WKMGNA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:05.587545699 (318h55m4.13901616s) [noise=true , address=145.79.227.32:4835 , piece_id=7GD4QUTGLIPOEU25FIDE7VDO3AUJ2RZDU4XKI476KNU6QTETC52A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:06.956814829 (318h55m5.50828528s) [noise=true , address=145.79.227.8:4330 , piece_id=LFTZWYQKW56VLMJPOGGLB6IJILPGJNEHIRIYD2F4J34ZH24PCSMQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:07.254777763 (318h55m5.806248224s) [noise=true , address=145.79.227.30:4762 , piece_id=4Z7CCNLZEYQAGJNUWGJ2USQSVW6QZUTCNFKC3JWRIOKV6VBVIYKQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:19.983388333 (318h55m18.534858794s) [noise=true , address=145.79.227.10:4402 , piece_id=ZU6A2O2R73PIVIDF5NOKMJZH4QX4LOCFSYOQLFOFAA5BEVQH52KA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection Nov 22 14:59:19.983536355 (318h55m18.535006816s) [noise=true , address=145.79.227.21:4473 , piece_id=D5FC7PVECZTQR64BFVTV3L2HOXCITABN57LLNRSMVCSGQICRYKUQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection Nov 22 14:59:19.983802211 (318h55m18.535272703s) [noise=true , address=145.79.227.18:4007 , piece_id=KM5TO27LZXJZZAT2DB5JTP7R5LOPJHUU6T34RZI7HH7PRWPW4ZNQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection Nov 22 14:59:19.983812000 (318h55m18.535282461s) [noise=true , address=145.79.227.9:4365 , piece_id=DBW2L6GHAWIHOMOZOTQFQ57Y2MA6KP6SGUO6AJDDF65ERQ324VZQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection Nov 22 14:59:19.983896611 (318h55m18.535367073s) [noise=true , address=145.79.227.19:4078 , piece_id=ER3KKYJ2MJQC3HRRBVULC3ING5GLGOHFVD2QBWF2MVAXF6ODMUUA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection; write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection Nov 22 14:59:19.984068268 (318h55m18.535538719s) [noise=true , address=145.79.227.26:4654 , piece_id=VCRM5O6NHOJOKNQUG5TTZUT3JIDQOMNF2DD6H7KWUJFOQ3P75QGQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: read tcp 188.166.225.170:36440->145.79.227.26:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984229324 (318h55m18.535699776s) [noise=true , address=145.79.227.8:4329 , piece_id=YJG3LYBDMPS4QQNLG24MS7ZWUM6EOI6EPDGVBGLO33CVEQQKGX6A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: read tcp 188.166.225.170:44902->145.79.227.8:4329: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984374972 (318h55m18.535845423s) [noise=true , address=145.79.227.16:4909 , piece_id=3DAVIY3ETZVG5CPP5BQB7QL473J5HZG6EQ6BYZBIAN6TWZ5FGNHQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: read tcp 188.166.225.170:59064->145.79.227.16:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984400911 (318h55m18.535871362s) [seg_pos={0 0}, upload=280863]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 22 14:59:20.539921660 (318h55m19.091392121s) [upload=280863]: Done uploading segments... Nov 22 14:59:20.539961818 (318h55m19.091432269s) [upload=280863]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 15.522399209s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [...X.X.....X.......X......X............X.................X..] duration: 7.60680057s trace: https://jaeger.dev.storj.io/trace/2af0ef576bba07f6 FAIL (2025-11-22 15:07:32 UTC) orbiter warning: context deadline exceeded Duration: 7.60683231s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [....X.X.....X.......X......X............X.................X.] duration: 22.319117497s trace: https://jaeger.dev.storj.io/trace/2af0ef576bba07f6 FAIL (2025-11-22 15:07:24 UTC) orbiter warning: context deadline exceeded Duration: 22.319137835s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [.......XX..X..XX..X.X.........XX.XX.................XX..X...] duration: 3.664866654s trace: https://jaeger.dev.storj.io/trace/cc91f835756ebe2 FAIL (2025-11-22 15:07:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.664920195s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......................XX.X.........X........X..............] duration: 15.804770221s trace: https://jaeger.dev.storj.io/trace/1efcdf5c2a183d8d FAIL (2025-11-22 15:06:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 934958555s) [noise=true , address=145.79.227.30:4762 , piece_id=4Z7CCNLZEYQAGJNUWGJ2USQSVW6QZUTCNFKC3JWRIOKV6VBVIYKQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.384231169 (318h54m50.93570163s) [noise=true , address=145.79.227.18:4007 , piece_id=KM5TO27LZXJZZAT2DB5JTP7R5LOPJHUU6T34RZI7HH7PRWPW4ZNQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.384687758 (318h54m50.93615822s) [noise=true , address=145.79.227.26:4654 , piece_id=VCRM5O6NHOJOKNQUG5TTZUT3JIDQOMNF2DD6H7KWUJFOQ3P75QGQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.385168865 (318h54m50.936639326s) [noise=true , address=77.90.31.18:4805 , piece_id=3EHB5IFBQRLXFBOVCT25RS4CIXVMSVOE7EKDV26DXOEY2NDJ2YGA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.385605395 (318h54m50.937075867s) [noise=true , address=145.79.227.32:4835 , piece_id=7GD4QUTGLIPOEU25FIDE7VDO3AUJ2RZDU4XKI476KNU6QTETC52A, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.386294879 (318h54m50.93776534s) [noise=true , address=145.79.227.21:4473 , piece_id=D5FC7PVECZTQR64BFVTV3L2HOXCITABN57LLNRSMVCSGQICRYKUQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.386958642 (318h54m50.938429103s) [noise=true , address=145.79.227.14:4837 , piece_id=NR427EUHKVSQ6HMMZCHL4LCBV6VRVTANZDXMX6JHQI2TTQN5R5VQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387097236 (318h54m50.938567687s) [noise=true , address=145.79.227.9:4365 , piece_id=DBW2L6GHAWIHOMOZOTQFQ57Y2MA6KP6SGUO6AJDDF65ERQ324VZQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387456781 (318h54m50.938927242s) [noise=true , address=145.79.227.19:4078 , piece_id=ER3KKYJ2MJQC3HRRBVULC3ING5GLGOHFVD2QBWF2MVAXF6ODMUUA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387755300 (318h54m50.939225761s) [noise=true , address=145.79.227.28:4725 , piece_id=HVMAX2UWJIF2UJQIJS5UBTO5TOIJI5DCDGGURZN4SOMRY3Z4OF4A, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.388354220 (318h54m50.939824681s) [noise=true , address=145.79.227.10:4402 , piece_id=ZU6A2O2R73PIVIDF5NOKMJZH4QX4LOCFSYOQLFOFAA5BEVQH52KA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.388862268 (318h54m50.940332729s) [noise=true , address=145.79.227.22:4509 , piece_id=WRHXVGJZQRTY6ZKMRU6IFYZSMLR6UATXFZOTCPRYKHERA2WKMGNA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.389411814 (318h54m50.940882275s) [noise=true , address=145.79.227.8:4330 , piece_id=LFTZWYQKW56VLMJPOGGLB6IJILPGJNEHIRIYD2F4J34ZH24PCSMQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.389753224 (318h54m50.941223675s) [noise=true , address=145.79.227.16:4909 , piece_id=3DAVIY3ETZVG5CPP5BQB7QL473J5HZG6EQ6BYZBIAN6TWZ5FGNHQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.391102172 (318h54m50.942572633s) [noise=true , address=45.91.116.7:4010 , piece_id=VZCYHOAR73DVCRDT5GBT65JI53NP27OEGA3VYLM66LPQR23T4MTA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:54.404583235 (318h54m52.956053696s) [upload=280863]: Next returned nil segment Nov 22 14:58:54.404601019 (318h54m52.956071481s) [upload=280863]: Waiting for error group managing segments... Nov 22 14:58:54.928222688 (318h54m53.479693149s) [noise=true , address=77.90.31.18:4805 , piece_id=3EHB5IFBQRLXFBOVCT25RS4CIXVMSVOE7EKDV26DXOEY2NDJ2YGA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:58:55.175979631 (318h54m53.727450092s) [noise=true , address=45.91.116.7:4010 , piece_id=VZCYHOAR73DVCRDT5GBT65JI53NP27OEGA3VYLM66LPQR23T4MTA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:03.279622498 (318h55m1.831092959s) [noise=true , address=145.79.227.14:4837 , piece_id=NR427EUHKVSQ6HMMZCHL4LCBV6VRVTANZDXMX6JHQI2TTQN5R5VQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:04.274652248 (318h55m2.826122709s) [noise=true , address=145.79.227.28:4725 , piece_id=HVMAX2UWJIF2UJQIJS5UBTO5TOIJI5DCDGGURZN4SOMRY3Z4OF4A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:05.089136230 (318h55m3.640606701s) [noise=true , address=145.79.227.22:4509 , piece_id=WRHXVGJZQRTY6ZKMRU6IFYZSMLR6UATXFZOTCPRYKHERA2WKMGNA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:05.587545699 (318h55m4.13901616s) [noise=true , address=145.79.227.32:4835 , piece_id=7GD4QUTGLIPOEU25FIDE7VDO3AUJ2RZDU4XKI476KNU6QTETC52A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:06.956814829 (318h55m5.50828528s) [noise=true , address=145.79.227.8:4330 , piece_id=LFTZWYQKW56VLMJPOGGLB6IJILPGJNEHIRIYD2F4J34ZH24PCSMQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:07.254777763 (318h55m5.806248224s) [noise=true , address=145.79.227.30:4762 , piece_id=4Z7CCNLZEYQAGJNUWGJ2USQSVW6QZUTCNFKC3JWRIOKV6VBVIYKQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:19.983388333 (318h55m18.534858794s) [noise=true , address=145.79.227.10:4402 , piece_id=ZU6A2O2R73PIVIDF5NOKMJZH4QX4LOCFSYOQLFOFAA5BEVQH52KA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection Nov 22 14:59:19.983536355 (318h55m18.535006816s) [noise=true , address=145.79.227.21:4473 , piece_id=D5FC7PVECZTQR64BFVTV3L2HOXCITABN57LLNRSMVCSGQICRYKUQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection Nov 22 14:59:19.983802211 (318h55m18.535272703s) [noise=true , address=145.79.227.18:4007 , piece_id=KM5TO27LZXJZZAT2DB5JTP7R5LOPJHUU6T34RZI7HH7PRWPW4ZNQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection Nov 22 14:59:19.983812000 (318h55m18.535282461s) [noise=true , address=145.79.227.9:4365 , piece_id=DBW2L6GHAWIHOMOZOTQFQ57Y2MA6KP6SGUO6AJDDF65ERQ324VZQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection Nov 22 14:59:19.983896611 (318h55m18.535367073s) [noise=true , address=145.79.227.19:4078 , piece_id=ER3KKYJ2MJQC3HRRBVULC3ING5GLGOHFVD2QBWF2MVAXF6ODMUUA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection; write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection Nov 22 14:59:19.984068268 (318h55m18.535538719s) [noise=true , address=145.79.227.26:4654 , piece_id=VCRM5O6NHOJOKNQUG5TTZUT3JIDQOMNF2DD6H7KWUJFOQ3P75QGQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: read tcp 188.166.225.170:36440->145.79.227.26:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984229324 (318h55m18.535699776s) [noise=true , address=145.79.227.8:4329 , piece_id=YJG3LYBDMPS4QQNLG24MS7ZWUM6EOI6EPDGVBGLO33CVEQQKGX6A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: read tcp 188.166.225.170:44902->145.79.227.8:4329: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984374972 (318h55m18.535845423s) [noise=true , address=145.79.227.16:4909 , piece_id=3DAVIY3ETZVG5CPP5BQB7QL473J5HZG6EQ6BYZBIAN6TWZ5FGNHQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: read tcp 188.166.225.170:59064->145.79.227.16:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984400911 (318h55m18.535871362s) [seg_pos={0 0}, upload=280863]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 22 14:59:20.539921660 (318h55m19.091392121s) [upload=280863]: Done uploading segments... Nov 22 14:59:20.539961818 (318h55m19.091432269s) [upload=280863]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 15.804798744s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.....X.X.....X.......X......X............X.................X] duration: 30.00109215s trace: https://jaeger.dev.storj.io/trace/2af0ef576bba07f6 FAIL (2025-11-22 15:06:49 UTC) orbiter warning: context deadline exceeded Duration: 30.001115246s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [........................XX.X.........X........X.............] duration: 14.332931689s trace: https://jaeger.dev.storj.io/trace/1efcdf5c2a183d8d FAIL (2025-11-22 15:06:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 934958555s) [noise=true , address=145.79.227.30:4762 , piece_id=4Z7CCNLZEYQAGJNUWGJ2USQSVW6QZUTCNFKC3JWRIOKV6VBVIYKQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.384231169 (318h54m50.93570163s) [noise=true , address=145.79.227.18:4007 , piece_id=KM5TO27LZXJZZAT2DB5JTP7R5LOPJHUU6T34RZI7HH7PRWPW4ZNQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.384687758 (318h54m50.93615822s) [noise=true , address=145.79.227.26:4654 , piece_id=VCRM5O6NHOJOKNQUG5TTZUT3JIDQOMNF2DD6H7KWUJFOQ3P75QGQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.385168865 (318h54m50.936639326s) [noise=true , address=77.90.31.18:4805 , piece_id=3EHB5IFBQRLXFBOVCT25RS4CIXVMSVOE7EKDV26DXOEY2NDJ2YGA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.385605395 (318h54m50.937075867s) [noise=true , address=145.79.227.32:4835 , piece_id=7GD4QUTGLIPOEU25FIDE7VDO3AUJ2RZDU4XKI476KNU6QTETC52A, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.386294879 (318h54m50.93776534s) [noise=true , address=145.79.227.21:4473 , piece_id=D5FC7PVECZTQR64BFVTV3L2HOXCITABN57LLNRSMVCSGQICRYKUQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.386958642 (318h54m50.938429103s) [noise=true , address=145.79.227.14:4837 , piece_id=NR427EUHKVSQ6HMMZCHL4LCBV6VRVTANZDXMX6JHQI2TTQN5R5VQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387097236 (318h54m50.938567687s) [noise=true , address=145.79.227.9:4365 , piece_id=DBW2L6GHAWIHOMOZOTQFQ57Y2MA6KP6SGUO6AJDDF65ERQ324VZQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387456781 (318h54m50.938927242s) [noise=true , address=145.79.227.19:4078 , piece_id=ER3KKYJ2MJQC3HRRBVULC3ING5GLGOHFVD2QBWF2MVAXF6ODMUUA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387755300 (318h54m50.939225761s) [noise=true , address=145.79.227.28:4725 , piece_id=HVMAX2UWJIF2UJQIJS5UBTO5TOIJI5DCDGGURZN4SOMRY3Z4OF4A, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.388354220 (318h54m50.939824681s) [noise=true , address=145.79.227.10:4402 , piece_id=ZU6A2O2R73PIVIDF5NOKMJZH4QX4LOCFSYOQLFOFAA5BEVQH52KA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.388862268 (318h54m50.940332729s) [noise=true , address=145.79.227.22:4509 , piece_id=WRHXVGJZQRTY6ZKMRU6IFYZSMLR6UATXFZOTCPRYKHERA2WKMGNA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.389411814 (318h54m50.940882275s) [noise=true , address=145.79.227.8:4330 , piece_id=LFTZWYQKW56VLMJPOGGLB6IJILPGJNEHIRIYD2F4J34ZH24PCSMQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.389753224 (318h54m50.941223675s) [noise=true , address=145.79.227.16:4909 , piece_id=3DAVIY3ETZVG5CPP5BQB7QL473J5HZG6EQ6BYZBIAN6TWZ5FGNHQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.391102172 (318h54m50.942572633s) [noise=true , address=45.91.116.7:4010 , piece_id=VZCYHOAR73DVCRDT5GBT65JI53NP27OEGA3VYLM66LPQR23T4MTA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:54.404583235 (318h54m52.956053696s) [upload=280863]: Next returned nil segment Nov 22 14:58:54.404601019 (318h54m52.956071481s) [upload=280863]: Waiting for error group managing segments... Nov 22 14:58:54.928222688 (318h54m53.479693149s) [noise=true , address=77.90.31.18:4805 , piece_id=3EHB5IFBQRLXFBOVCT25RS4CIXVMSVOE7EKDV26DXOEY2NDJ2YGA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:58:55.175979631 (318h54m53.727450092s) [noise=true , address=45.91.116.7:4010 , piece_id=VZCYHOAR73DVCRDT5GBT65JI53NP27OEGA3VYLM66LPQR23T4MTA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:03.279622498 (318h55m1.831092959s) [noise=true , address=145.79.227.14:4837 , piece_id=NR427EUHKVSQ6HMMZCHL4LCBV6VRVTANZDXMX6JHQI2TTQN5R5VQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:04.274652248 (318h55m2.826122709s) [noise=true , address=145.79.227.28:4725 , piece_id=HVMAX2UWJIF2UJQIJS5UBTO5TOIJI5DCDGGURZN4SOMRY3Z4OF4A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:05.089136230 (318h55m3.640606701s) [noise=true , address=145.79.227.22:4509 , piece_id=WRHXVGJZQRTY6ZKMRU6IFYZSMLR6UATXFZOTCPRYKHERA2WKMGNA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:05.587545699 (318h55m4.13901616s) [noise=true , address=145.79.227.32:4835 , piece_id=7GD4QUTGLIPOEU25FIDE7VDO3AUJ2RZDU4XKI476KNU6QTETC52A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:06.956814829 (318h55m5.50828528s) [noise=true , address=145.79.227.8:4330 , piece_id=LFTZWYQKW56VLMJPOGGLB6IJILPGJNEHIRIYD2F4J34ZH24PCSMQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:07.254777763 (318h55m5.806248224s) [noise=true , address=145.79.227.30:4762 , piece_id=4Z7CCNLZEYQAGJNUWGJ2USQSVW6QZUTCNFKC3JWRIOKV6VBVIYKQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:19.983388333 (318h55m18.534858794s) [noise=true , address=145.79.227.10:4402 , piece_id=ZU6A2O2R73PIVIDF5NOKMJZH4QX4LOCFSYOQLFOFAA5BEVQH52KA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection Nov 22 14:59:19.983536355 (318h55m18.535006816s) [noise=true , address=145.79.227.21:4473 , piece_id=D5FC7PVECZTQR64BFVTV3L2HOXCITABN57LLNRSMVCSGQICRYKUQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection Nov 22 14:59:19.983802211 (318h55m18.535272703s) [noise=true , address=145.79.227.18:4007 , piece_id=KM5TO27LZXJZZAT2DB5JTP7R5LOPJHUU6T34RZI7HH7PRWPW4ZNQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection Nov 22 14:59:19.983812000 (318h55m18.535282461s) [noise=true , address=145.79.227.9:4365 , piece_id=DBW2L6GHAWIHOMOZOTQFQ57Y2MA6KP6SGUO6AJDDF65ERQ324VZQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection Nov 22 14:59:19.983896611 (318h55m18.535367073s) [noise=true , address=145.79.227.19:4078 , piece_id=ER3KKYJ2MJQC3HRRBVULC3ING5GLGOHFVD2QBWF2MVAXF6ODMUUA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection; write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection Nov 22 14:59:19.984068268 (318h55m18.535538719s) [noise=true , address=145.79.227.26:4654 , piece_id=VCRM5O6NHOJOKNQUG5TTZUT3JIDQOMNF2DD6H7KWUJFOQ3P75QGQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: read tcp 188.166.225.170:36440->145.79.227.26:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984229324 (318h55m18.535699776s) [noise=true , address=145.79.227.8:4329 , piece_id=YJG3LYBDMPS4QQNLG24MS7ZWUM6EOI6EPDGVBGLO33CVEQQKGX6A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: read tcp 188.166.225.170:44902->145.79.227.8:4329: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984374972 (318h55m18.535845423s) [noise=true , address=145.79.227.16:4909 , piece_id=3DAVIY3ETZVG5CPP5BQB7QL473J5HZG6EQ6BYZBIAN6TWZ5FGNHQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: read tcp 188.166.225.170:59064->145.79.227.16:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984400911 (318h55m18.535871362s) [seg_pos={0 0}, upload=280863]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 22 14:59:20.539921660 (318h55m19.091392121s) [upload=280863]: Done uploading segments... Nov 22 14:59:20.539961818 (318h55m19.091432269s) [upload=280863]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 14.332973439s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.........................XX.X.........X........X............] duration: 14.51940066s trace: https://jaeger.dev.storj.io/trace/1efcdf5c2a183d8d FAIL (2025-11-22 15:06:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 934958555s) [noise=true , address=145.79.227.30:4762 , piece_id=4Z7CCNLZEYQAGJNUWGJ2USQSVW6QZUTCNFKC3JWRIOKV6VBVIYKQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.384231169 (318h54m50.93570163s) [noise=true , address=145.79.227.18:4007 , piece_id=KM5TO27LZXJZZAT2DB5JTP7R5LOPJHUU6T34RZI7HH7PRWPW4ZNQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.384687758 (318h54m50.93615822s) [noise=true , address=145.79.227.26:4654 , piece_id=VCRM5O6NHOJOKNQUG5TTZUT3JIDQOMNF2DD6H7KWUJFOQ3P75QGQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.385168865 (318h54m50.936639326s) [noise=true , address=77.90.31.18:4805 , piece_id=3EHB5IFBQRLXFBOVCT25RS4CIXVMSVOE7EKDV26DXOEY2NDJ2YGA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.385605395 (318h54m50.937075867s) [noise=true , address=145.79.227.32:4835 , piece_id=7GD4QUTGLIPOEU25FIDE7VDO3AUJ2RZDU4XKI476KNU6QTETC52A, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.386294879 (318h54m50.93776534s) [noise=true , address=145.79.227.21:4473 , piece_id=D5FC7PVECZTQR64BFVTV3L2HOXCITABN57LLNRSMVCSGQICRYKUQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.386958642 (318h54m50.938429103s) [noise=true , address=145.79.227.14:4837 , piece_id=NR427EUHKVSQ6HMMZCHL4LCBV6VRVTANZDXMX6JHQI2TTQN5R5VQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387097236 (318h54m50.938567687s) [noise=true , address=145.79.227.9:4365 , piece_id=DBW2L6GHAWIHOMOZOTQFQ57Y2MA6KP6SGUO6AJDDF65ERQ324VZQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387456781 (318h54m50.938927242s) [noise=true , address=145.79.227.19:4078 , piece_id=ER3KKYJ2MJQC3HRRBVULC3ING5GLGOHFVD2QBWF2MVAXF6ODMUUA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387755300 (318h54m50.939225761s) [noise=true , address=145.79.227.28:4725 , piece_id=HVMAX2UWJIF2UJQIJS5UBTO5TOIJI5DCDGGURZN4SOMRY3Z4OF4A, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.388354220 (318h54m50.939824681s) [noise=true , address=145.79.227.10:4402 , piece_id=ZU6A2O2R73PIVIDF5NOKMJZH4QX4LOCFSYOQLFOFAA5BEVQH52KA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.388862268 (318h54m50.940332729s) [noise=true , address=145.79.227.22:4509 , piece_id=WRHXVGJZQRTY6ZKMRU6IFYZSMLR6UATXFZOTCPRYKHERA2WKMGNA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.389411814 (318h54m50.940882275s) [noise=true , address=145.79.227.8:4330 , piece_id=LFTZWYQKW56VLMJPOGGLB6IJILPGJNEHIRIYD2F4J34ZH24PCSMQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.389753224 (318h54m50.941223675s) [noise=true , address=145.79.227.16:4909 , piece_id=3DAVIY3ETZVG5CPP5BQB7QL473J5HZG6EQ6BYZBIAN6TWZ5FGNHQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.391102172 (318h54m50.942572633s) [noise=true , address=45.91.116.7:4010 , piece_id=VZCYHOAR73DVCRDT5GBT65JI53NP27OEGA3VYLM66LPQR23T4MTA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:54.404583235 (318h54m52.956053696s) [upload=280863]: Next returned nil segment Nov 22 14:58:54.404601019 (318h54m52.956071481s) [upload=280863]: Waiting for error group managing segments... Nov 22 14:58:54.928222688 (318h54m53.479693149s) [noise=true , address=77.90.31.18:4805 , piece_id=3EHB5IFBQRLXFBOVCT25RS4CIXVMSVOE7EKDV26DXOEY2NDJ2YGA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:58:55.175979631 (318h54m53.727450092s) [noise=true , address=45.91.116.7:4010 , piece_id=VZCYHOAR73DVCRDT5GBT65JI53NP27OEGA3VYLM66LPQR23T4MTA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:03.279622498 (318h55m1.831092959s) [noise=true , address=145.79.227.14:4837 , piece_id=NR427EUHKVSQ6HMMZCHL4LCBV6VRVTANZDXMX6JHQI2TTQN5R5VQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:04.274652248 (318h55m2.826122709s) [noise=true , address=145.79.227.28:4725 , piece_id=HVMAX2UWJIF2UJQIJS5UBTO5TOIJI5DCDGGURZN4SOMRY3Z4OF4A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:05.089136230 (318h55m3.640606701s) [noise=true , address=145.79.227.22:4509 , piece_id=WRHXVGJZQRTY6ZKMRU6IFYZSMLR6UATXFZOTCPRYKHERA2WKMGNA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:05.587545699 (318h55m4.13901616s) [noise=true , address=145.79.227.32:4835 , piece_id=7GD4QUTGLIPOEU25FIDE7VDO3AUJ2RZDU4XKI476KNU6QTETC52A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:06.956814829 (318h55m5.50828528s) [noise=true , address=145.79.227.8:4330 , piece_id=LFTZWYQKW56VLMJPOGGLB6IJILPGJNEHIRIYD2F4J34ZH24PCSMQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:07.254777763 (318h55m5.806248224s) [noise=true , address=145.79.227.30:4762 , piece_id=4Z7CCNLZEYQAGJNUWGJ2USQSVW6QZUTCNFKC3JWRIOKV6VBVIYKQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:19.983388333 (318h55m18.534858794s) [noise=true , address=145.79.227.10:4402 , piece_id=ZU6A2O2R73PIVIDF5NOKMJZH4QX4LOCFSYOQLFOFAA5BEVQH52KA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection Nov 22 14:59:19.983536355 (318h55m18.535006816s) [noise=true , address=145.79.227.21:4473 , piece_id=D5FC7PVECZTQR64BFVTV3L2HOXCITABN57LLNRSMVCSGQICRYKUQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection Nov 22 14:59:19.983802211 (318h55m18.535272703s) [noise=true , address=145.79.227.18:4007 , piece_id=KM5TO27LZXJZZAT2DB5JTP7R5LOPJHUU6T34RZI7HH7PRWPW4ZNQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection Nov 22 14:59:19.983812000 (318h55m18.535282461s) [noise=true , address=145.79.227.9:4365 , piece_id=DBW2L6GHAWIHOMOZOTQFQ57Y2MA6KP6SGUO6AJDDF65ERQ324VZQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection Nov 22 14:59:19.983896611 (318h55m18.535367073s) [noise=true , address=145.79.227.19:4078 , piece_id=ER3KKYJ2MJQC3HRRBVULC3ING5GLGOHFVD2QBWF2MVAXF6ODMUUA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection; write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection Nov 22 14:59:19.984068268 (318h55m18.535538719s) [noise=true , address=145.79.227.26:4654 , piece_id=VCRM5O6NHOJOKNQUG5TTZUT3JIDQOMNF2DD6H7KWUJFOQ3P75QGQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: read tcp 188.166.225.170:36440->145.79.227.26:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984229324 (318h55m18.535699776s) [noise=true , address=145.79.227.8:4329 , piece_id=YJG3LYBDMPS4QQNLG24MS7ZWUM6EOI6EPDGVBGLO33CVEQQKGX6A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: read tcp 188.166.225.170:44902->145.79.227.8:4329: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984374972 (318h55m18.535845423s) [noise=true , address=145.79.227.16:4909 , piece_id=3DAVIY3ETZVG5CPP5BQB7QL473J5HZG6EQ6BYZBIAN6TWZ5FGNHQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: read tcp 188.166.225.170:59064->145.79.227.16:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984400911 (318h55m18.535871362s) [seg_pos={0 0}, upload=280863]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 22 14:59:20.539921660 (318h55m19.091392121s) [upload=280863]: Done uploading segments... Nov 22 14:59:20.539961818 (318h55m19.091432269s) [upload=280863]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 14.519468369s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 15 out of 60 is failed (threshold=6) [X.......XX..X..XX..X.X.........XX.XX.................XX..X..] duration: 4.580023264s trace: https://jaeger.dev.storj.io/trace/cc91f835756ebe2 FAIL (2025-11-22 15:06:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.58004779s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [......X.X.....X.......X......X............X.................] duration: 7.558750965s trace: https://jaeger.dev.storj.io/trace/1ee0c6bff9434222 FAIL (2025-11-22 15:05:48 UTC) orbiter warning: context deadline exceeded Duration: 7.558785501s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..........................XX.X.........X........X...........] duration: 18.769138842s trace: https://jaeger.dev.storj.io/trace/1efcdf5c2a183d8d FAIL (2025-11-22 15:05:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 934958555s) [noise=true , address=145.79.227.30:4762 , piece_id=4Z7CCNLZEYQAGJNUWGJ2USQSVW6QZUTCNFKC3JWRIOKV6VBVIYKQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.384231169 (318h54m50.93570163s) [noise=true , address=145.79.227.18:4007 , piece_id=KM5TO27LZXJZZAT2DB5JTP7R5LOPJHUU6T34RZI7HH7PRWPW4ZNQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.384687758 (318h54m50.93615822s) [noise=true , address=145.79.227.26:4654 , piece_id=VCRM5O6NHOJOKNQUG5TTZUT3JIDQOMNF2DD6H7KWUJFOQ3P75QGQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.385168865 (318h54m50.936639326s) [noise=true , address=77.90.31.18:4805 , piece_id=3EHB5IFBQRLXFBOVCT25RS4CIXVMSVOE7EKDV26DXOEY2NDJ2YGA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.385605395 (318h54m50.937075867s) [noise=true , address=145.79.227.32:4835 , piece_id=7GD4QUTGLIPOEU25FIDE7VDO3AUJ2RZDU4XKI476KNU6QTETC52A, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.386294879 (318h54m50.93776534s) [noise=true , address=145.79.227.21:4473 , piece_id=D5FC7PVECZTQR64BFVTV3L2HOXCITABN57LLNRSMVCSGQICRYKUQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.386958642 (318h54m50.938429103s) [noise=true , address=145.79.227.14:4837 , piece_id=NR427EUHKVSQ6HMMZCHL4LCBV6VRVTANZDXMX6JHQI2TTQN5R5VQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387097236 (318h54m50.938567687s) [noise=true , address=145.79.227.9:4365 , piece_id=DBW2L6GHAWIHOMOZOTQFQ57Y2MA6KP6SGUO6AJDDF65ERQ324VZQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387456781 (318h54m50.938927242s) [noise=true , address=145.79.227.19:4078 , piece_id=ER3KKYJ2MJQC3HRRBVULC3ING5GLGOHFVD2QBWF2MVAXF6ODMUUA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387755300 (318h54m50.939225761s) [noise=true , address=145.79.227.28:4725 , piece_id=HVMAX2UWJIF2UJQIJS5UBTO5TOIJI5DCDGGURZN4SOMRY3Z4OF4A, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.388354220 (318h54m50.939824681s) [noise=true , address=145.79.227.10:4402 , piece_id=ZU6A2O2R73PIVIDF5NOKMJZH4QX4LOCFSYOQLFOFAA5BEVQH52KA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.388862268 (318h54m50.940332729s) [noise=true , address=145.79.227.22:4509 , piece_id=WRHXVGJZQRTY6ZKMRU6IFYZSMLR6UATXFZOTCPRYKHERA2WKMGNA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.389411814 (318h54m50.940882275s) [noise=true , address=145.79.227.8:4330 , piece_id=LFTZWYQKW56VLMJPOGGLB6IJILPGJNEHIRIYD2F4J34ZH24PCSMQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.389753224 (318h54m50.941223675s) [noise=true , address=145.79.227.16:4909 , piece_id=3DAVIY3ETZVG5CPP5BQB7QL473J5HZG6EQ6BYZBIAN6TWZ5FGNHQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.391102172 (318h54m50.942572633s) [noise=true , address=45.91.116.7:4010 , piece_id=VZCYHOAR73DVCRDT5GBT65JI53NP27OEGA3VYLM66LPQR23T4MTA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:54.404583235 (318h54m52.956053696s) [upload=280863]: Next returned nil segment Nov 22 14:58:54.404601019 (318h54m52.956071481s) [upload=280863]: Waiting for error group managing segments... Nov 22 14:58:54.928222688 (318h54m53.479693149s) [noise=true , address=77.90.31.18:4805 , piece_id=3EHB5IFBQRLXFBOVCT25RS4CIXVMSVOE7EKDV26DXOEY2NDJ2YGA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:58:55.175979631 (318h54m53.727450092s) [noise=true , address=45.91.116.7:4010 , piece_id=VZCYHOAR73DVCRDT5GBT65JI53NP27OEGA3VYLM66LPQR23T4MTA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:03.279622498 (318h55m1.831092959s) [noise=true , address=145.79.227.14:4837 , piece_id=NR427EUHKVSQ6HMMZCHL4LCBV6VRVTANZDXMX6JHQI2TTQN5R5VQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:04.274652248 (318h55m2.826122709s) [noise=true , address=145.79.227.28:4725 , piece_id=HVMAX2UWJIF2UJQIJS5UBTO5TOIJI5DCDGGURZN4SOMRY3Z4OF4A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:05.089136230 (318h55m3.640606701s) [noise=true , address=145.79.227.22:4509 , piece_id=WRHXVGJZQRTY6ZKMRU6IFYZSMLR6UATXFZOTCPRYKHERA2WKMGNA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:05.587545699 (318h55m4.13901616s) [noise=true , address=145.79.227.32:4835 , piece_id=7GD4QUTGLIPOEU25FIDE7VDO3AUJ2RZDU4XKI476KNU6QTETC52A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:06.956814829 (318h55m5.50828528s) [noise=true , address=145.79.227.8:4330 , piece_id=LFTZWYQKW56VLMJPOGGLB6IJILPGJNEHIRIYD2F4J34ZH24PCSMQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:07.254777763 (318h55m5.806248224s) [noise=true , address=145.79.227.30:4762 , piece_id=4Z7CCNLZEYQAGJNUWGJ2USQSVW6QZUTCNFKC3JWRIOKV6VBVIYKQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:19.983388333 (318h55m18.534858794s) [noise=true , address=145.79.227.10:4402 , piece_id=ZU6A2O2R73PIVIDF5NOKMJZH4QX4LOCFSYOQLFOFAA5BEVQH52KA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection Nov 22 14:59:19.983536355 (318h55m18.535006816s) [noise=true , address=145.79.227.21:4473 , piece_id=D5FC7PVECZTQR64BFVTV3L2HOXCITABN57LLNRSMVCSGQICRYKUQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection Nov 22 14:59:19.983802211 (318h55m18.535272703s) [noise=true , address=145.79.227.18:4007 , piece_id=KM5TO27LZXJZZAT2DB5JTP7R5LOPJHUU6T34RZI7HH7PRWPW4ZNQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection Nov 22 14:59:19.983812000 (318h55m18.535282461s) [noise=true , address=145.79.227.9:4365 , piece_id=DBW2L6GHAWIHOMOZOTQFQ57Y2MA6KP6SGUO6AJDDF65ERQ324VZQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection Nov 22 14:59:19.983896611 (318h55m18.535367073s) [noise=true , address=145.79.227.19:4078 , piece_id=ER3KKYJ2MJQC3HRRBVULC3ING5GLGOHFVD2QBWF2MVAXF6ODMUUA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection; write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection Nov 22 14:59:19.984068268 (318h55m18.535538719s) [noise=true , address=145.79.227.26:4654 , piece_id=VCRM5O6NHOJOKNQUG5TTZUT3JIDQOMNF2DD6H7KWUJFOQ3P75QGQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: read tcp 188.166.225.170:36440->145.79.227.26:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984229324 (318h55m18.535699776s) [noise=true , address=145.79.227.8:4329 , piece_id=YJG3LYBDMPS4QQNLG24MS7ZWUM6EOI6EPDGVBGLO33CVEQQKGX6A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: read tcp 188.166.225.170:44902->145.79.227.8:4329: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984374972 (318h55m18.535845423s) [noise=true , address=145.79.227.16:4909 , piece_id=3DAVIY3ETZVG5CPP5BQB7QL473J5HZG6EQ6BYZBIAN6TWZ5FGNHQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: read tcp 188.166.225.170:59064->145.79.227.16:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984400911 (318h55m18.535871362s) [seg_pos={0 0}, upload=280863]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 22 14:59:20.539921660 (318h55m19.091392121s) [upload=280863]: Done uploading segments... Nov 22 14:59:20.539961818 (318h55m19.091432269s) [upload=280863]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 18.769171114s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 15 out of 60 is failed (threshold=6) [.X.......XX..X..XX..X.X.........XX.XX.................XX..X.] duration: 3.618298269s trace: https://jaeger.dev.storj.io/trace/cc91f835756ebe2 FAIL (2025-11-22 15:05:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.618321382s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.......X.X.....X.......X......X............X................] duration: 6.354325738s trace: https://jaeger.dev.storj.io/trace/1ee0c6bff9434222 FAIL (2025-11-22 15:05:10 UTC) orbiter warning: context deadline exceeded Duration: 6.354371635s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...........................XX.X.........X........X..........] duration: 13.202976595s trace: https://jaeger.dev.storj.io/trace/1efcdf5c2a183d8d FAIL (2025-11-22 15:05:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 934958555s) [noise=true , address=145.79.227.30:4762 , piece_id=4Z7CCNLZEYQAGJNUWGJ2USQSVW6QZUTCNFKC3JWRIOKV6VBVIYKQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.384231169 (318h54m50.93570163s) [noise=true , address=145.79.227.18:4007 , piece_id=KM5TO27LZXJZZAT2DB5JTP7R5LOPJHUU6T34RZI7HH7PRWPW4ZNQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.384687758 (318h54m50.93615822s) [noise=true , address=145.79.227.26:4654 , piece_id=VCRM5O6NHOJOKNQUG5TTZUT3JIDQOMNF2DD6H7KWUJFOQ3P75QGQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.385168865 (318h54m50.936639326s) [noise=true , address=77.90.31.18:4805 , piece_id=3EHB5IFBQRLXFBOVCT25RS4CIXVMSVOE7EKDV26DXOEY2NDJ2YGA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.385605395 (318h54m50.937075867s) [noise=true , address=145.79.227.32:4835 , piece_id=7GD4QUTGLIPOEU25FIDE7VDO3AUJ2RZDU4XKI476KNU6QTETC52A, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.386294879 (318h54m50.93776534s) [noise=true , address=145.79.227.21:4473 , piece_id=D5FC7PVECZTQR64BFVTV3L2HOXCITABN57LLNRSMVCSGQICRYKUQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.386958642 (318h54m50.938429103s) [noise=true , address=145.79.227.14:4837 , piece_id=NR427EUHKVSQ6HMMZCHL4LCBV6VRVTANZDXMX6JHQI2TTQN5R5VQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387097236 (318h54m50.938567687s) [noise=true , address=145.79.227.9:4365 , piece_id=DBW2L6GHAWIHOMOZOTQFQ57Y2MA6KP6SGUO6AJDDF65ERQ324VZQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387456781 (318h54m50.938927242s) [noise=true , address=145.79.227.19:4078 , piece_id=ER3KKYJ2MJQC3HRRBVULC3ING5GLGOHFVD2QBWF2MVAXF6ODMUUA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387755300 (318h54m50.939225761s) [noise=true , address=145.79.227.28:4725 , piece_id=HVMAX2UWJIF2UJQIJS5UBTO5TOIJI5DCDGGURZN4SOMRY3Z4OF4A, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.388354220 (318h54m50.939824681s) [noise=true , address=145.79.227.10:4402 , piece_id=ZU6A2O2R73PIVIDF5NOKMJZH4QX4LOCFSYOQLFOFAA5BEVQH52KA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.388862268 (318h54m50.940332729s) [noise=true , address=145.79.227.22:4509 , piece_id=WRHXVGJZQRTY6ZKMRU6IFYZSMLR6UATXFZOTCPRYKHERA2WKMGNA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.389411814 (318h54m50.940882275s) [noise=true , address=145.79.227.8:4330 , piece_id=LFTZWYQKW56VLMJPOGGLB6IJILPGJNEHIRIYD2F4J34ZH24PCSMQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.389753224 (318h54m50.941223675s) [noise=true , address=145.79.227.16:4909 , piece_id=3DAVIY3ETZVG5CPP5BQB7QL473J5HZG6EQ6BYZBIAN6TWZ5FGNHQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.391102172 (318h54m50.942572633s) [noise=true , address=45.91.116.7:4010 , piece_id=VZCYHOAR73DVCRDT5GBT65JI53NP27OEGA3VYLM66LPQR23T4MTA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:54.404583235 (318h54m52.956053696s) [upload=280863]: Next returned nil segment Nov 22 14:58:54.404601019 (318h54m52.956071481s) [upload=280863]: Waiting for error group managing segments... Nov 22 14:58:54.928222688 (318h54m53.479693149s) [noise=true , address=77.90.31.18:4805 , piece_id=3EHB5IFBQRLXFBOVCT25RS4CIXVMSVOE7EKDV26DXOEY2NDJ2YGA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:58:55.175979631 (318h54m53.727450092s) [noise=true , address=45.91.116.7:4010 , piece_id=VZCYHOAR73DVCRDT5GBT65JI53NP27OEGA3VYLM66LPQR23T4MTA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:03.279622498 (318h55m1.831092959s) [noise=true , address=145.79.227.14:4837 , piece_id=NR427EUHKVSQ6HMMZCHL4LCBV6VRVTANZDXMX6JHQI2TTQN5R5VQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:04.274652248 (318h55m2.826122709s) [noise=true , address=145.79.227.28:4725 , piece_id=HVMAX2UWJIF2UJQIJS5UBTO5TOIJI5DCDGGURZN4SOMRY3Z4OF4A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:05.089136230 (318h55m3.640606701s) [noise=true , address=145.79.227.22:4509 , piece_id=WRHXVGJZQRTY6ZKMRU6IFYZSMLR6UATXFZOTCPRYKHERA2WKMGNA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:05.587545699 (318h55m4.13901616s) [noise=true , address=145.79.227.32:4835 , piece_id=7GD4QUTGLIPOEU25FIDE7VDO3AUJ2RZDU4XKI476KNU6QTETC52A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:06.956814829 (318h55m5.50828528s) [noise=true , address=145.79.227.8:4330 , piece_id=LFTZWYQKW56VLMJPOGGLB6IJILPGJNEHIRIYD2F4J34ZH24PCSMQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:07.254777763 (318h55m5.806248224s) [noise=true , address=145.79.227.30:4762 , piece_id=4Z7CCNLZEYQAGJNUWGJ2USQSVW6QZUTCNFKC3JWRIOKV6VBVIYKQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:19.983388333 (318h55m18.534858794s) [noise=true , address=145.79.227.10:4402 , piece_id=ZU6A2O2R73PIVIDF5NOKMJZH4QX4LOCFSYOQLFOFAA5BEVQH52KA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection Nov 22 14:59:19.983536355 (318h55m18.535006816s) [noise=true , address=145.79.227.21:4473 , piece_id=D5FC7PVECZTQR64BFVTV3L2HOXCITABN57LLNRSMVCSGQICRYKUQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection Nov 22 14:59:19.983802211 (318h55m18.535272703s) [noise=true , address=145.79.227.18:4007 , piece_id=KM5TO27LZXJZZAT2DB5JTP7R5LOPJHUU6T34RZI7HH7PRWPW4ZNQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection Nov 22 14:59:19.983812000 (318h55m18.535282461s) [noise=true , address=145.79.227.9:4365 , piece_id=DBW2L6GHAWIHOMOZOTQFQ57Y2MA6KP6SGUO6AJDDF65ERQ324VZQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection Nov 22 14:59:19.983896611 (318h55m18.535367073s) [noise=true , address=145.79.227.19:4078 , piece_id=ER3KKYJ2MJQC3HRRBVULC3ING5GLGOHFVD2QBWF2MVAXF6ODMUUA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection; write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection Nov 22 14:59:19.984068268 (318h55m18.535538719s) [noise=true , address=145.79.227.26:4654 , piece_id=VCRM5O6NHOJOKNQUG5TTZUT3JIDQOMNF2DD6H7KWUJFOQ3P75QGQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: read tcp 188.166.225.170:36440->145.79.227.26:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984229324 (318h55m18.535699776s) [noise=true , address=145.79.227.8:4329 , piece_id=YJG3LYBDMPS4QQNLG24MS7ZWUM6EOI6EPDGVBGLO33CVEQQKGX6A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: read tcp 188.166.225.170:44902->145.79.227.8:4329: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984374972 (318h55m18.535845423s) [noise=true , address=145.79.227.16:4909 , piece_id=3DAVIY3ETZVG5CPP5BQB7QL473J5HZG6EQ6BYZBIAN6TWZ5FGNHQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: read tcp 188.166.225.170:59064->145.79.227.16:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984400911 (318h55m18.535871362s) [seg_pos={0 0}, upload=280863]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 22 14:59:20.539921660 (318h55m19.091392121s) [upload=280863]: Done uploading segments... Nov 22 14:59:20.539961818 (318h55m19.091432269s) [upload=280863]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 13.203013327s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [........X.X.....X.......X......X............X...............] duration: 27.534589817s trace: https://jaeger.dev.storj.io/trace/1ee0c6bff9434222 FAIL (2025-11-22 15:04:30 UTC) orbiter warning: context deadline exceeded Duration: 27.534626157s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [............................XX.X.........X........X.........] duration: 12.117593325s trace: https://jaeger.dev.storj.io/trace/1efcdf5c2a183d8d FAIL (2025-11-22 15:04:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 934958555s) [noise=true , address=145.79.227.30:4762 , piece_id=4Z7CCNLZEYQAGJNUWGJ2USQSVW6QZUTCNFKC3JWRIOKV6VBVIYKQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.384231169 (318h54m50.93570163s) [noise=true , address=145.79.227.18:4007 , piece_id=KM5TO27LZXJZZAT2DB5JTP7R5LOPJHUU6T34RZI7HH7PRWPW4ZNQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.384687758 (318h54m50.93615822s) [noise=true , address=145.79.227.26:4654 , piece_id=VCRM5O6NHOJOKNQUG5TTZUT3JIDQOMNF2DD6H7KWUJFOQ3P75QGQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.385168865 (318h54m50.936639326s) [noise=true , address=77.90.31.18:4805 , piece_id=3EHB5IFBQRLXFBOVCT25RS4CIXVMSVOE7EKDV26DXOEY2NDJ2YGA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.385605395 (318h54m50.937075867s) [noise=true , address=145.79.227.32:4835 , piece_id=7GD4QUTGLIPOEU25FIDE7VDO3AUJ2RZDU4XKI476KNU6QTETC52A, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.386294879 (318h54m50.93776534s) [noise=true , address=145.79.227.21:4473 , piece_id=D5FC7PVECZTQR64BFVTV3L2HOXCITABN57LLNRSMVCSGQICRYKUQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.386958642 (318h54m50.938429103s) [noise=true , address=145.79.227.14:4837 , piece_id=NR427EUHKVSQ6HMMZCHL4LCBV6VRVTANZDXMX6JHQI2TTQN5R5VQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387097236 (318h54m50.938567687s) [noise=true , address=145.79.227.9:4365 , piece_id=DBW2L6GHAWIHOMOZOTQFQ57Y2MA6KP6SGUO6AJDDF65ERQ324VZQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387456781 (318h54m50.938927242s) [noise=true , address=145.79.227.19:4078 , piece_id=ER3KKYJ2MJQC3HRRBVULC3ING5GLGOHFVD2QBWF2MVAXF6ODMUUA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387755300 (318h54m50.939225761s) [noise=true , address=145.79.227.28:4725 , piece_id=HVMAX2UWJIF2UJQIJS5UBTO5TOIJI5DCDGGURZN4SOMRY3Z4OF4A, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.388354220 (318h54m50.939824681s) [noise=true , address=145.79.227.10:4402 , piece_id=ZU6A2O2R73PIVIDF5NOKMJZH4QX4LOCFSYOQLFOFAA5BEVQH52KA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.388862268 (318h54m50.940332729s) [noise=true , address=145.79.227.22:4509 , piece_id=WRHXVGJZQRTY6ZKMRU6IFYZSMLR6UATXFZOTCPRYKHERA2WKMGNA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.389411814 (318h54m50.940882275s) [noise=true , address=145.79.227.8:4330 , piece_id=LFTZWYQKW56VLMJPOGGLB6IJILPGJNEHIRIYD2F4J34ZH24PCSMQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.389753224 (318h54m50.941223675s) [noise=true , address=145.79.227.16:4909 , piece_id=3DAVIY3ETZVG5CPP5BQB7QL473J5HZG6EQ6BYZBIAN6TWZ5FGNHQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.391102172 (318h54m50.942572633s) [noise=true , address=45.91.116.7:4010 , piece_id=VZCYHOAR73DVCRDT5GBT65JI53NP27OEGA3VYLM66LPQR23T4MTA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:54.404583235 (318h54m52.956053696s) [upload=280863]: Next returned nil segment Nov 22 14:58:54.404601019 (318h54m52.956071481s) [upload=280863]: Waiting for error group managing segments... Nov 22 14:58:54.928222688 (318h54m53.479693149s) [noise=true , address=77.90.31.18:4805 , piece_id=3EHB5IFBQRLXFBOVCT25RS4CIXVMSVOE7EKDV26DXOEY2NDJ2YGA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:58:55.175979631 (318h54m53.727450092s) [noise=true , address=45.91.116.7:4010 , piece_id=VZCYHOAR73DVCRDT5GBT65JI53NP27OEGA3VYLM66LPQR23T4MTA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:03.279622498 (318h55m1.831092959s) [noise=true , address=145.79.227.14:4837 , piece_id=NR427EUHKVSQ6HMMZCHL4LCBV6VRVTANZDXMX6JHQI2TTQN5R5VQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:04.274652248 (318h55m2.826122709s) [noise=true , address=145.79.227.28:4725 , piece_id=HVMAX2UWJIF2UJQIJS5UBTO5TOIJI5DCDGGURZN4SOMRY3Z4OF4A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:05.089136230 (318h55m3.640606701s) [noise=true , address=145.79.227.22:4509 , piece_id=WRHXVGJZQRTY6ZKMRU6IFYZSMLR6UATXFZOTCPRYKHERA2WKMGNA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:05.587545699 (318h55m4.13901616s) [noise=true , address=145.79.227.32:4835 , piece_id=7GD4QUTGLIPOEU25FIDE7VDO3AUJ2RZDU4XKI476KNU6QTETC52A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:06.956814829 (318h55m5.50828528s) [noise=true , address=145.79.227.8:4330 , piece_id=LFTZWYQKW56VLMJPOGGLB6IJILPGJNEHIRIYD2F4J34ZH24PCSMQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:07.254777763 (318h55m5.806248224s) [noise=true , address=145.79.227.30:4762 , piece_id=4Z7CCNLZEYQAGJNUWGJ2USQSVW6QZUTCNFKC3JWRIOKV6VBVIYKQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:19.983388333 (318h55m18.534858794s) [noise=true , address=145.79.227.10:4402 , piece_id=ZU6A2O2R73PIVIDF5NOKMJZH4QX4LOCFSYOQLFOFAA5BEVQH52KA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection Nov 22 14:59:19.983536355 (318h55m18.535006816s) [noise=true , address=145.79.227.21:4473 , piece_id=D5FC7PVECZTQR64BFVTV3L2HOXCITABN57LLNRSMVCSGQICRYKUQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection Nov 22 14:59:19.983802211 (318h55m18.535272703s) [noise=true , address=145.79.227.18:4007 , piece_id=KM5TO27LZXJZZAT2DB5JTP7R5LOPJHUU6T34RZI7HH7PRWPW4ZNQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection Nov 22 14:59:19.983812000 (318h55m18.535282461s) [noise=true , address=145.79.227.9:4365 , piece_id=DBW2L6GHAWIHOMOZOTQFQ57Y2MA6KP6SGUO6AJDDF65ERQ324VZQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection Nov 22 14:59:19.983896611 (318h55m18.535367073s) [noise=true , address=145.79.227.19:4078 , piece_id=ER3KKYJ2MJQC3HRRBVULC3ING5GLGOHFVD2QBWF2MVAXF6ODMUUA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection; write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection Nov 22 14:59:19.984068268 (318h55m18.535538719s) [noise=true , address=145.79.227.26:4654 , piece_id=VCRM5O6NHOJOKNQUG5TTZUT3JIDQOMNF2DD6H7KWUJFOQ3P75QGQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: read tcp 188.166.225.170:36440->145.79.227.26:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984229324 (318h55m18.535699776s) [noise=true , address=145.79.227.8:4329 , piece_id=YJG3LYBDMPS4QQNLG24MS7ZWUM6EOI6EPDGVBGLO33CVEQQKGX6A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: read tcp 188.166.225.170:44902->145.79.227.8:4329: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984374972 (318h55m18.535845423s) [noise=true , address=145.79.227.16:4909 , piece_id=3DAVIY3ETZVG5CPP5BQB7QL473J5HZG6EQ6BYZBIAN6TWZ5FGNHQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: read tcp 188.166.225.170:59064->145.79.227.16:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984400911 (318h55m18.535871362s) [seg_pos={0 0}, upload=280863]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 22 14:59:20.539921660 (318h55m19.091392121s) [upload=280863]: Done uploading segments... Nov 22 14:59:20.539961818 (318h55m19.091432269s) [upload=280863]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 12.117650515s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 15 out of 60 is failed (threshold=6) [..X.......XX..X..XX..X.X.........XX.XX.................XX..X] duration: 5.001423292s trace: https://jaeger.dev.storj.io/trace/cc91f835756ebe2 FAIL (2025-11-22 15:04:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001463229s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.............................XX.X.........X........X........] duration: 20.029297466s trace: https://jaeger.dev.storj.io/trace/1efcdf5c2a183d8d FAIL (2025-11-22 15:04:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 934958555s) [noise=true , address=145.79.227.30:4762 , piece_id=4Z7CCNLZEYQAGJNUWGJ2USQSVW6QZUTCNFKC3JWRIOKV6VBVIYKQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.384231169 (318h54m50.93570163s) [noise=true , address=145.79.227.18:4007 , piece_id=KM5TO27LZXJZZAT2DB5JTP7R5LOPJHUU6T34RZI7HH7PRWPW4ZNQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.384687758 (318h54m50.93615822s) [noise=true , address=145.79.227.26:4654 , piece_id=VCRM5O6NHOJOKNQUG5TTZUT3JIDQOMNF2DD6H7KWUJFOQ3P75QGQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.385168865 (318h54m50.936639326s) [noise=true , address=77.90.31.18:4805 , piece_id=3EHB5IFBQRLXFBOVCT25RS4CIXVMSVOE7EKDV26DXOEY2NDJ2YGA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.385605395 (318h54m50.937075867s) [noise=true , address=145.79.227.32:4835 , piece_id=7GD4QUTGLIPOEU25FIDE7VDO3AUJ2RZDU4XKI476KNU6QTETC52A, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.386294879 (318h54m50.93776534s) [noise=true , address=145.79.227.21:4473 , piece_id=D5FC7PVECZTQR64BFVTV3L2HOXCITABN57LLNRSMVCSGQICRYKUQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.386958642 (318h54m50.938429103s) [noise=true , address=145.79.227.14:4837 , piece_id=NR427EUHKVSQ6HMMZCHL4LCBV6VRVTANZDXMX6JHQI2TTQN5R5VQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387097236 (318h54m50.938567687s) [noise=true , address=145.79.227.9:4365 , piece_id=DBW2L6GHAWIHOMOZOTQFQ57Y2MA6KP6SGUO6AJDDF65ERQ324VZQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387456781 (318h54m50.938927242s) [noise=true , address=145.79.227.19:4078 , piece_id=ER3KKYJ2MJQC3HRRBVULC3ING5GLGOHFVD2QBWF2MVAXF6ODMUUA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387755300 (318h54m50.939225761s) [noise=true , address=145.79.227.28:4725 , piece_id=HVMAX2UWJIF2UJQIJS5UBTO5TOIJI5DCDGGURZN4SOMRY3Z4OF4A, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.388354220 (318h54m50.939824681s) [noise=true , address=145.79.227.10:4402 , piece_id=ZU6A2O2R73PIVIDF5NOKMJZH4QX4LOCFSYOQLFOFAA5BEVQH52KA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.388862268 (318h54m50.940332729s) [noise=true , address=145.79.227.22:4509 , piece_id=WRHXVGJZQRTY6ZKMRU6IFYZSMLR6UATXFZOTCPRYKHERA2WKMGNA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.389411814 (318h54m50.940882275s) [noise=true , address=145.79.227.8:4330 , piece_id=LFTZWYQKW56VLMJPOGGLB6IJILPGJNEHIRIYD2F4J34ZH24PCSMQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.389753224 (318h54m50.941223675s) [noise=true , address=145.79.227.16:4909 , piece_id=3DAVIY3ETZVG5CPP5BQB7QL473J5HZG6EQ6BYZBIAN6TWZ5FGNHQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.391102172 (318h54m50.942572633s) [noise=true , address=45.91.116.7:4010 , piece_id=VZCYHOAR73DVCRDT5GBT65JI53NP27OEGA3VYLM66LPQR23T4MTA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:54.404583235 (318h54m52.956053696s) [upload=280863]: Next returned nil segment Nov 22 14:58:54.404601019 (318h54m52.956071481s) [upload=280863]: Waiting for error group managing segments... Nov 22 14:58:54.928222688 (318h54m53.479693149s) [noise=true , address=77.90.31.18:4805 , piece_id=3EHB5IFBQRLXFBOVCT25RS4CIXVMSVOE7EKDV26DXOEY2NDJ2YGA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:58:55.175979631 (318h54m53.727450092s) [noise=true , address=45.91.116.7:4010 , piece_id=VZCYHOAR73DVCRDT5GBT65JI53NP27OEGA3VYLM66LPQR23T4MTA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:03.279622498 (318h55m1.831092959s) [noise=true , address=145.79.227.14:4837 , piece_id=NR427EUHKVSQ6HMMZCHL4LCBV6VRVTANZDXMX6JHQI2TTQN5R5VQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:04.274652248 (318h55m2.826122709s) [noise=true , address=145.79.227.28:4725 , piece_id=HVMAX2UWJIF2UJQIJS5UBTO5TOIJI5DCDGGURZN4SOMRY3Z4OF4A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:05.089136230 (318h55m3.640606701s) [noise=true , address=145.79.227.22:4509 , piece_id=WRHXVGJZQRTY6ZKMRU6IFYZSMLR6UATXFZOTCPRYKHERA2WKMGNA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:05.587545699 (318h55m4.13901616s) [noise=true , address=145.79.227.32:4835 , piece_id=7GD4QUTGLIPOEU25FIDE7VDO3AUJ2RZDU4XKI476KNU6QTETC52A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:06.956814829 (318h55m5.50828528s) [noise=true , address=145.79.227.8:4330 , piece_id=LFTZWYQKW56VLMJPOGGLB6IJILPGJNEHIRIYD2F4J34ZH24PCSMQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:07.254777763 (318h55m5.806248224s) [noise=true , address=145.79.227.30:4762 , piece_id=4Z7CCNLZEYQAGJNUWGJ2USQSVW6QZUTCNFKC3JWRIOKV6VBVIYKQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:19.983388333 (318h55m18.534858794s) [noise=true , address=145.79.227.10:4402 , piece_id=ZU6A2O2R73PIVIDF5NOKMJZH4QX4LOCFSYOQLFOFAA5BEVQH52KA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection Nov 22 14:59:19.983536355 (318h55m18.535006816s) [noise=true , address=145.79.227.21:4473 , piece_id=D5FC7PVECZTQR64BFVTV3L2HOXCITABN57LLNRSMVCSGQICRYKUQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection Nov 22 14:59:19.983802211 (318h55m18.535272703s) [noise=true , address=145.79.227.18:4007 , piece_id=KM5TO27LZXJZZAT2DB5JTP7R5LOPJHUU6T34RZI7HH7PRWPW4ZNQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection Nov 22 14:59:19.983812000 (318h55m18.535282461s) [noise=true , address=145.79.227.9:4365 , piece_id=DBW2L6GHAWIHOMOZOTQFQ57Y2MA6KP6SGUO6AJDDF65ERQ324VZQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection Nov 22 14:59:19.983896611 (318h55m18.535367073s) [noise=true , address=145.79.227.19:4078 , piece_id=ER3KKYJ2MJQC3HRRBVULC3ING5GLGOHFVD2QBWF2MVAXF6ODMUUA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection; write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection Nov 22 14:59:19.984068268 (318h55m18.535538719s) [noise=true , address=145.79.227.26:4654 , piece_id=VCRM5O6NHOJOKNQUG5TTZUT3JIDQOMNF2DD6H7KWUJFOQ3P75QGQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: read tcp 188.166.225.170:36440->145.79.227.26:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984229324 (318h55m18.535699776s) [noise=true , address=145.79.227.8:4329 , piece_id=YJG3LYBDMPS4QQNLG24MS7ZWUM6EOI6EPDGVBGLO33CVEQQKGX6A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: read tcp 188.166.225.170:44902->145.79.227.8:4329: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984374972 (318h55m18.535845423s) [noise=true , address=145.79.227.16:4909 , piece_id=3DAVIY3ETZVG5CPP5BQB7QL473J5HZG6EQ6BYZBIAN6TWZ5FGNHQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: read tcp 188.166.225.170:59064->145.79.227.16:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984400911 (318h55m18.535871362s) [seg_pos={0 0}, upload=280863]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 22 14:59:20.539921660 (318h55m19.091392121s) [upload=280863]: Done uploading segments... Nov 22 14:59:20.539961818 (318h55m19.091432269s) [upload=280863]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 20.02933115s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [...X.......XX..X..XX..X.X.........XX.XX.................XX..] duration: 3.580152402s trace: https://jaeger.dev.storj.io/trace/33e61458f8570628 FAIL (2025-11-22 15:03:27 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.58017761s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..............................XX.X.........X........X.......] duration: 15.235520709s trace: https://jaeger.dev.storj.io/trace/1efcdf5c2a183d8d FAIL (2025-11-22 15:03:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 934958555s) [noise=true , address=145.79.227.30:4762 , piece_id=4Z7CCNLZEYQAGJNUWGJ2USQSVW6QZUTCNFKC3JWRIOKV6VBVIYKQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.384231169 (318h54m50.93570163s) [noise=true , address=145.79.227.18:4007 , piece_id=KM5TO27LZXJZZAT2DB5JTP7R5LOPJHUU6T34RZI7HH7PRWPW4ZNQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.384687758 (318h54m50.93615822s) [noise=true , address=145.79.227.26:4654 , piece_id=VCRM5O6NHOJOKNQUG5TTZUT3JIDQOMNF2DD6H7KWUJFOQ3P75QGQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.385168865 (318h54m50.936639326s) [noise=true , address=77.90.31.18:4805 , piece_id=3EHB5IFBQRLXFBOVCT25RS4CIXVMSVOE7EKDV26DXOEY2NDJ2YGA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.385605395 (318h54m50.937075867s) [noise=true , address=145.79.227.32:4835 , piece_id=7GD4QUTGLIPOEU25FIDE7VDO3AUJ2RZDU4XKI476KNU6QTETC52A, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.386294879 (318h54m50.93776534s) [noise=true , address=145.79.227.21:4473 , piece_id=D5FC7PVECZTQR64BFVTV3L2HOXCITABN57LLNRSMVCSGQICRYKUQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.386958642 (318h54m50.938429103s) [noise=true , address=145.79.227.14:4837 , piece_id=NR427EUHKVSQ6HMMZCHL4LCBV6VRVTANZDXMX6JHQI2TTQN5R5VQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387097236 (318h54m50.938567687s) [noise=true , address=145.79.227.9:4365 , piece_id=DBW2L6GHAWIHOMOZOTQFQ57Y2MA6KP6SGUO6AJDDF65ERQ324VZQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387456781 (318h54m50.938927242s) [noise=true , address=145.79.227.19:4078 , piece_id=ER3KKYJ2MJQC3HRRBVULC3ING5GLGOHFVD2QBWF2MVAXF6ODMUUA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387755300 (318h54m50.939225761s) [noise=true , address=145.79.227.28:4725 , piece_id=HVMAX2UWJIF2UJQIJS5UBTO5TOIJI5DCDGGURZN4SOMRY3Z4OF4A, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.388354220 (318h54m50.939824681s) [noise=true , address=145.79.227.10:4402 , piece_id=ZU6A2O2R73PIVIDF5NOKMJZH4QX4LOCFSYOQLFOFAA5BEVQH52KA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.388862268 (318h54m50.940332729s) [noise=true , address=145.79.227.22:4509 , piece_id=WRHXVGJZQRTY6ZKMRU6IFYZSMLR6UATXFZOTCPRYKHERA2WKMGNA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.389411814 (318h54m50.940882275s) [noise=true , address=145.79.227.8:4330 , piece_id=LFTZWYQKW56VLMJPOGGLB6IJILPGJNEHIRIYD2F4J34ZH24PCSMQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.389753224 (318h54m50.941223675s) [noise=true , address=145.79.227.16:4909 , piece_id=3DAVIY3ETZVG5CPP5BQB7QL473J5HZG6EQ6BYZBIAN6TWZ5FGNHQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.391102172 (318h54m50.942572633s) [noise=true , address=45.91.116.7:4010 , piece_id=VZCYHOAR73DVCRDT5GBT65JI53NP27OEGA3VYLM66LPQR23T4MTA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:54.404583235 (318h54m52.956053696s) [upload=280863]: Next returned nil segment Nov 22 14:58:54.404601019 (318h54m52.956071481s) [upload=280863]: Waiting for error group managing segments... Nov 22 14:58:54.928222688 (318h54m53.479693149s) [noise=true , address=77.90.31.18:4805 , piece_id=3EHB5IFBQRLXFBOVCT25RS4CIXVMSVOE7EKDV26DXOEY2NDJ2YGA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:58:55.175979631 (318h54m53.727450092s) [noise=true , address=45.91.116.7:4010 , piece_id=VZCYHOAR73DVCRDT5GBT65JI53NP27OEGA3VYLM66LPQR23T4MTA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:03.279622498 (318h55m1.831092959s) [noise=true , address=145.79.227.14:4837 , piece_id=NR427EUHKVSQ6HMMZCHL4LCBV6VRVTANZDXMX6JHQI2TTQN5R5VQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:04.274652248 (318h55m2.826122709s) [noise=true , address=145.79.227.28:4725 , piece_id=HVMAX2UWJIF2UJQIJS5UBTO5TOIJI5DCDGGURZN4SOMRY3Z4OF4A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:05.089136230 (318h55m3.640606701s) [noise=true , address=145.79.227.22:4509 , piece_id=WRHXVGJZQRTY6ZKMRU6IFYZSMLR6UATXFZOTCPRYKHERA2WKMGNA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:05.587545699 (318h55m4.13901616s) [noise=true , address=145.79.227.32:4835 , piece_id=7GD4QUTGLIPOEU25FIDE7VDO3AUJ2RZDU4XKI476KNU6QTETC52A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:06.956814829 (318h55m5.50828528s) [noise=true , address=145.79.227.8:4330 , piece_id=LFTZWYQKW56VLMJPOGGLB6IJILPGJNEHIRIYD2F4J34ZH24PCSMQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:07.254777763 (318h55m5.806248224s) [noise=true , address=145.79.227.30:4762 , piece_id=4Z7CCNLZEYQAGJNUWGJ2USQSVW6QZUTCNFKC3JWRIOKV6VBVIYKQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:19.983388333 (318h55m18.534858794s) [noise=true , address=145.79.227.10:4402 , piece_id=ZU6A2O2R73PIVIDF5NOKMJZH4QX4LOCFSYOQLFOFAA5BEVQH52KA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection Nov 22 14:59:19.983536355 (318h55m18.535006816s) [noise=true , address=145.79.227.21:4473 , piece_id=D5FC7PVECZTQR64BFVTV3L2HOXCITABN57LLNRSMVCSGQICRYKUQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection Nov 22 14:59:19.983802211 (318h55m18.535272703s) [noise=true , address=145.79.227.18:4007 , piece_id=KM5TO27LZXJZZAT2DB5JTP7R5LOPJHUU6T34RZI7HH7PRWPW4ZNQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection Nov 22 14:59:19.983812000 (318h55m18.535282461s) [noise=true , address=145.79.227.9:4365 , piece_id=DBW2L6GHAWIHOMOZOTQFQ57Y2MA6KP6SGUO6AJDDF65ERQ324VZQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection Nov 22 14:59:19.983896611 (318h55m18.535367073s) [noise=true , address=145.79.227.19:4078 , piece_id=ER3KKYJ2MJQC3HRRBVULC3ING5GLGOHFVD2QBWF2MVAXF6ODMUUA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection; write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection Nov 22 14:59:19.984068268 (318h55m18.535538719s) [noise=true , address=145.79.227.26:4654 , piece_id=VCRM5O6NHOJOKNQUG5TTZUT3JIDQOMNF2DD6H7KWUJFOQ3P75QGQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: read tcp 188.166.225.170:36440->145.79.227.26:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984229324 (318h55m18.535699776s) [noise=true , address=145.79.227.8:4329 , piece_id=YJG3LYBDMPS4QQNLG24MS7ZWUM6EOI6EPDGVBGLO33CVEQQKGX6A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: read tcp 188.166.225.170:44902->145.79.227.8:4329: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984374972 (318h55m18.535845423s) [noise=true , address=145.79.227.16:4909 , piece_id=3DAVIY3ETZVG5CPP5BQB7QL473J5HZG6EQ6BYZBIAN6TWZ5FGNHQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: read tcp 188.166.225.170:59064->145.79.227.16:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984400911 (318h55m18.535871362s) [seg_pos={0 0}, upload=280863]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 22 14:59:20.539921660 (318h55m19.091392121s) [upload=280863]: Done uploading segments... Nov 22 14:59:20.539961818 (318h55m19.091432269s) [upload=280863]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 15.235544054s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [....X.......XX..X..XX..X.X.........XX.XX.................XX.] duration: 3.58614386s trace: https://jaeger.dev.storj.io/trace/33e61458f8570628 FAIL (2025-11-22 15:02:56 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.586174038s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.........X.X.....X.......X......X............X..............] duration: 6.167414236s trace: https://jaeger.dev.storj.io/trace/1ee0c6bff9434222 FAIL (2025-11-22 15:02:55 UTC) orbiter warning: context deadline exceeded Duration: 6.167449582s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...............................XX.X.........X........X......] duration: 22.938848695s trace: https://jaeger.dev.storj.io/trace/1efcdf5c2a183d8d FAIL (2025-11-22 15:02:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 934958555s) [noise=true , address=145.79.227.30:4762 , piece_id=4Z7CCNLZEYQAGJNUWGJ2USQSVW6QZUTCNFKC3JWRIOKV6VBVIYKQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.384231169 (318h54m50.93570163s) [noise=true , address=145.79.227.18:4007 , piece_id=KM5TO27LZXJZZAT2DB5JTP7R5LOPJHUU6T34RZI7HH7PRWPW4ZNQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.384687758 (318h54m50.93615822s) [noise=true , address=145.79.227.26:4654 , piece_id=VCRM5O6NHOJOKNQUG5TTZUT3JIDQOMNF2DD6H7KWUJFOQ3P75QGQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.385168865 (318h54m50.936639326s) [noise=true , address=77.90.31.18:4805 , piece_id=3EHB5IFBQRLXFBOVCT25RS4CIXVMSVOE7EKDV26DXOEY2NDJ2YGA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.385605395 (318h54m50.937075867s) [noise=true , address=145.79.227.32:4835 , piece_id=7GD4QUTGLIPOEU25FIDE7VDO3AUJ2RZDU4XKI476KNU6QTETC52A, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.386294879 (318h54m50.93776534s) [noise=true , address=145.79.227.21:4473 , piece_id=D5FC7PVECZTQR64BFVTV3L2HOXCITABN57LLNRSMVCSGQICRYKUQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.386958642 (318h54m50.938429103s) [noise=true , address=145.79.227.14:4837 , piece_id=NR427EUHKVSQ6HMMZCHL4LCBV6VRVTANZDXMX6JHQI2TTQN5R5VQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387097236 (318h54m50.938567687s) [noise=true , address=145.79.227.9:4365 , piece_id=DBW2L6GHAWIHOMOZOTQFQ57Y2MA6KP6SGUO6AJDDF65ERQ324VZQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387456781 (318h54m50.938927242s) [noise=true , address=145.79.227.19:4078 , piece_id=ER3KKYJ2MJQC3HRRBVULC3ING5GLGOHFVD2QBWF2MVAXF6ODMUUA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387755300 (318h54m50.939225761s) [noise=true , address=145.79.227.28:4725 , piece_id=HVMAX2UWJIF2UJQIJS5UBTO5TOIJI5DCDGGURZN4SOMRY3Z4OF4A, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.388354220 (318h54m50.939824681s) [noise=true , address=145.79.227.10:4402 , piece_id=ZU6A2O2R73PIVIDF5NOKMJZH4QX4LOCFSYOQLFOFAA5BEVQH52KA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.388862268 (318h54m50.940332729s) [noise=true , address=145.79.227.22:4509 , piece_id=WRHXVGJZQRTY6ZKMRU6IFYZSMLR6UATXFZOTCPRYKHERA2WKMGNA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.389411814 (318h54m50.940882275s) [noise=true , address=145.79.227.8:4330 , piece_id=LFTZWYQKW56VLMJPOGGLB6IJILPGJNEHIRIYD2F4J34ZH24PCSMQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.389753224 (318h54m50.941223675s) [noise=true , address=145.79.227.16:4909 , piece_id=3DAVIY3ETZVG5CPP5BQB7QL473J5HZG6EQ6BYZBIAN6TWZ5FGNHQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.391102172 (318h54m50.942572633s) [noise=true , address=45.91.116.7:4010 , piece_id=VZCYHOAR73DVCRDT5GBT65JI53NP27OEGA3VYLM66LPQR23T4MTA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:54.404583235 (318h54m52.956053696s) [upload=280863]: Next returned nil segment Nov 22 14:58:54.404601019 (318h54m52.956071481s) [upload=280863]: Waiting for error group managing segments... Nov 22 14:58:54.928222688 (318h54m53.479693149s) [noise=true , address=77.90.31.18:4805 , piece_id=3EHB5IFBQRLXFBOVCT25RS4CIXVMSVOE7EKDV26DXOEY2NDJ2YGA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:58:55.175979631 (318h54m53.727450092s) [noise=true , address=45.91.116.7:4010 , piece_id=VZCYHOAR73DVCRDT5GBT65JI53NP27OEGA3VYLM66LPQR23T4MTA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:03.279622498 (318h55m1.831092959s) [noise=true , address=145.79.227.14:4837 , piece_id=NR427EUHKVSQ6HMMZCHL4LCBV6VRVTANZDXMX6JHQI2TTQN5R5VQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:04.274652248 (318h55m2.826122709s) [noise=true , address=145.79.227.28:4725 , piece_id=HVMAX2UWJIF2UJQIJS5UBTO5TOIJI5DCDGGURZN4SOMRY3Z4OF4A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:05.089136230 (318h55m3.640606701s) [noise=true , address=145.79.227.22:4509 , piece_id=WRHXVGJZQRTY6ZKMRU6IFYZSMLR6UATXFZOTCPRYKHERA2WKMGNA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:05.587545699 (318h55m4.13901616s) [noise=true , address=145.79.227.32:4835 , piece_id=7GD4QUTGLIPOEU25FIDE7VDO3AUJ2RZDU4XKI476KNU6QTETC52A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:06.956814829 (318h55m5.50828528s) [noise=true , address=145.79.227.8:4330 , piece_id=LFTZWYQKW56VLMJPOGGLB6IJILPGJNEHIRIYD2F4J34ZH24PCSMQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:07.254777763 (318h55m5.806248224s) [noise=true , address=145.79.227.30:4762 , piece_id=4Z7CCNLZEYQAGJNUWGJ2USQSVW6QZUTCNFKC3JWRIOKV6VBVIYKQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:19.983388333 (318h55m18.534858794s) [noise=true , address=145.79.227.10:4402 , piece_id=ZU6A2O2R73PIVIDF5NOKMJZH4QX4LOCFSYOQLFOFAA5BEVQH52KA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection Nov 22 14:59:19.983536355 (318h55m18.535006816s) [noise=true , address=145.79.227.21:4473 , piece_id=D5FC7PVECZTQR64BFVTV3L2HOXCITABN57LLNRSMVCSGQICRYKUQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection Nov 22 14:59:19.983802211 (318h55m18.535272703s) [noise=true , address=145.79.227.18:4007 , piece_id=KM5TO27LZXJZZAT2DB5JTP7R5LOPJHUU6T34RZI7HH7PRWPW4ZNQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection Nov 22 14:59:19.983812000 (318h55m18.535282461s) [noise=true , address=145.79.227.9:4365 , piece_id=DBW2L6GHAWIHOMOZOTQFQ57Y2MA6KP6SGUO6AJDDF65ERQ324VZQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection Nov 22 14:59:19.983896611 (318h55m18.535367073s) [noise=true , address=145.79.227.19:4078 , piece_id=ER3KKYJ2MJQC3HRRBVULC3ING5GLGOHFVD2QBWF2MVAXF6ODMUUA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection; write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection Nov 22 14:59:19.984068268 (318h55m18.535538719s) [noise=true , address=145.79.227.26:4654 , piece_id=VCRM5O6NHOJOKNQUG5TTZUT3JIDQOMNF2DD6H7KWUJFOQ3P75QGQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: read tcp 188.166.225.170:36440->145.79.227.26:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984229324 (318h55m18.535699776s) [noise=true , address=145.79.227.8:4329 , piece_id=YJG3LYBDMPS4QQNLG24MS7ZWUM6EOI6EPDGVBGLO33CVEQQKGX6A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: read tcp 188.166.225.170:44902->145.79.227.8:4329: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984374972 (318h55m18.535845423s) [noise=true , address=145.79.227.16:4909 , piece_id=3DAVIY3ETZVG5CPP5BQB7QL473J5HZG6EQ6BYZBIAN6TWZ5FGNHQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: read tcp 188.166.225.170:59064->145.79.227.16:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984400911 (318h55m18.535871362s) [seg_pos={0 0}, upload=280863]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 22 14:59:20.539921660 (318h55m19.091392121s) [upload=280863]: Done uploading segments... Nov 22 14:59:20.539961818 (318h55m19.091432269s) [upload=280863]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 22.938878981s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [.....X.......XX..X..XX..X.X.........XX.XX.................XX] duration: 5.001625978s trace: https://jaeger.dev.storj.io/trace/33e61458f8570628 FAIL (2025-11-22 15:02:27 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001649173s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [................................XX.X.........X........X.....] duration: 14.929424748s trace: https://jaeger.dev.storj.io/trace/1efcdf5c2a183d8d FAIL (2025-11-22 15:02:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 934958555s) [noise=true , address=145.79.227.30:4762 , piece_id=4Z7CCNLZEYQAGJNUWGJ2USQSVW6QZUTCNFKC3JWRIOKV6VBVIYKQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.384231169 (318h54m50.93570163s) [noise=true , address=145.79.227.18:4007 , piece_id=KM5TO27LZXJZZAT2DB5JTP7R5LOPJHUU6T34RZI7HH7PRWPW4ZNQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.384687758 (318h54m50.93615822s) [noise=true , address=145.79.227.26:4654 , piece_id=VCRM5O6NHOJOKNQUG5TTZUT3JIDQOMNF2DD6H7KWUJFOQ3P75QGQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.385168865 (318h54m50.936639326s) [noise=true , address=77.90.31.18:4805 , piece_id=3EHB5IFBQRLXFBOVCT25RS4CIXVMSVOE7EKDV26DXOEY2NDJ2YGA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.385605395 (318h54m50.937075867s) [noise=true , address=145.79.227.32:4835 , piece_id=7GD4QUTGLIPOEU25FIDE7VDO3AUJ2RZDU4XKI476KNU6QTETC52A, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.386294879 (318h54m50.93776534s) [noise=true , address=145.79.227.21:4473 , piece_id=D5FC7PVECZTQR64BFVTV3L2HOXCITABN57LLNRSMVCSGQICRYKUQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.386958642 (318h54m50.938429103s) [noise=true , address=145.79.227.14:4837 , piece_id=NR427EUHKVSQ6HMMZCHL4LCBV6VRVTANZDXMX6JHQI2TTQN5R5VQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387097236 (318h54m50.938567687s) [noise=true , address=145.79.227.9:4365 , piece_id=DBW2L6GHAWIHOMOZOTQFQ57Y2MA6KP6SGUO6AJDDF65ERQ324VZQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387456781 (318h54m50.938927242s) [noise=true , address=145.79.227.19:4078 , piece_id=ER3KKYJ2MJQC3HRRBVULC3ING5GLGOHFVD2QBWF2MVAXF6ODMUUA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387755300 (318h54m50.939225761s) [noise=true , address=145.79.227.28:4725 , piece_id=HVMAX2UWJIF2UJQIJS5UBTO5TOIJI5DCDGGURZN4SOMRY3Z4OF4A, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.388354220 (318h54m50.939824681s) [noise=true , address=145.79.227.10:4402 , piece_id=ZU6A2O2R73PIVIDF5NOKMJZH4QX4LOCFSYOQLFOFAA5BEVQH52KA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.388862268 (318h54m50.940332729s) [noise=true , address=145.79.227.22:4509 , piece_id=WRHXVGJZQRTY6ZKMRU6IFYZSMLR6UATXFZOTCPRYKHERA2WKMGNA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.389411814 (318h54m50.940882275s) [noise=true , address=145.79.227.8:4330 , piece_id=LFTZWYQKW56VLMJPOGGLB6IJILPGJNEHIRIYD2F4J34ZH24PCSMQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.389753224 (318h54m50.941223675s) [noise=true , address=145.79.227.16:4909 , piece_id=3DAVIY3ETZVG5CPP5BQB7QL473J5HZG6EQ6BYZBIAN6TWZ5FGNHQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.391102172 (318h54m50.942572633s) [noise=true , address=45.91.116.7:4010 , piece_id=VZCYHOAR73DVCRDT5GBT65JI53NP27OEGA3VYLM66LPQR23T4MTA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:54.404583235 (318h54m52.956053696s) [upload=280863]: Next returned nil segment Nov 22 14:58:54.404601019 (318h54m52.956071481s) [upload=280863]: Waiting for error group managing segments... Nov 22 14:58:54.928222688 (318h54m53.479693149s) [noise=true , address=77.90.31.18:4805 , piece_id=3EHB5IFBQRLXFBOVCT25RS4CIXVMSVOE7EKDV26DXOEY2NDJ2YGA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:58:55.175979631 (318h54m53.727450092s) [noise=true , address=45.91.116.7:4010 , piece_id=VZCYHOAR73DVCRDT5GBT65JI53NP27OEGA3VYLM66LPQR23T4MTA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:03.279622498 (318h55m1.831092959s) [noise=true , address=145.79.227.14:4837 , piece_id=NR427EUHKVSQ6HMMZCHL4LCBV6VRVTANZDXMX6JHQI2TTQN5R5VQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:04.274652248 (318h55m2.826122709s) [noise=true , address=145.79.227.28:4725 , piece_id=HVMAX2UWJIF2UJQIJS5UBTO5TOIJI5DCDGGURZN4SOMRY3Z4OF4A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:05.089136230 (318h55m3.640606701s) [noise=true , address=145.79.227.22:4509 , piece_id=WRHXVGJZQRTY6ZKMRU6IFYZSMLR6UATXFZOTCPRYKHERA2WKMGNA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:05.587545699 (318h55m4.13901616s) [noise=true , address=145.79.227.32:4835 , piece_id=7GD4QUTGLIPOEU25FIDE7VDO3AUJ2RZDU4XKI476KNU6QTETC52A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:06.956814829 (318h55m5.50828528s) [noise=true , address=145.79.227.8:4330 , piece_id=LFTZWYQKW56VLMJPOGGLB6IJILPGJNEHIRIYD2F4J34ZH24PCSMQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:07.254777763 (318h55m5.806248224s) [noise=true , address=145.79.227.30:4762 , piece_id=4Z7CCNLZEYQAGJNUWGJ2USQSVW6QZUTCNFKC3JWRIOKV6VBVIYKQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:19.983388333 (318h55m18.534858794s) [noise=true , address=145.79.227.10:4402 , piece_id=ZU6A2O2R73PIVIDF5NOKMJZH4QX4LOCFSYOQLFOFAA5BEVQH52KA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection Nov 22 14:59:19.983536355 (318h55m18.535006816s) [noise=true , address=145.79.227.21:4473 , piece_id=D5FC7PVECZTQR64BFVTV3L2HOXCITABN57LLNRSMVCSGQICRYKUQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection Nov 22 14:59:19.983802211 (318h55m18.535272703s) [noise=true , address=145.79.227.18:4007 , piece_id=KM5TO27LZXJZZAT2DB5JTP7R5LOPJHUU6T34RZI7HH7PRWPW4ZNQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection Nov 22 14:59:19.983812000 (318h55m18.535282461s) [noise=true , address=145.79.227.9:4365 , piece_id=DBW2L6GHAWIHOMOZOTQFQ57Y2MA6KP6SGUO6AJDDF65ERQ324VZQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection Nov 22 14:59:19.983896611 (318h55m18.535367073s) [noise=true , address=145.79.227.19:4078 , piece_id=ER3KKYJ2MJQC3HRRBVULC3ING5GLGOHFVD2QBWF2MVAXF6ODMUUA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection; write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection Nov 22 14:59:19.984068268 (318h55m18.535538719s) [noise=true , address=145.79.227.26:4654 , piece_id=VCRM5O6NHOJOKNQUG5TTZUT3JIDQOMNF2DD6H7KWUJFOQ3P75QGQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: read tcp 188.166.225.170:36440->145.79.227.26:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984229324 (318h55m18.535699776s) [noise=true , address=145.79.227.8:4329 , piece_id=YJG3LYBDMPS4QQNLG24MS7ZWUM6EOI6EPDGVBGLO33CVEQQKGX6A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: read tcp 188.166.225.170:44902->145.79.227.8:4329: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984374972 (318h55m18.535845423s) [noise=true , address=145.79.227.16:4909 , piece_id=3DAVIY3ETZVG5CPP5BQB7QL473J5HZG6EQ6BYZBIAN6TWZ5FGNHQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: read tcp 188.166.225.170:59064->145.79.227.16:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984400911 (318h55m18.535871362s) [seg_pos={0 0}, upload=280863]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 22 14:59:20.539921660 (318h55m19.091392121s) [upload=280863]: Done uploading segments... Nov 22 14:59:20.539961818 (318h55m19.091432269s) [upload=280863]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 14.929447693s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [X.........X.X.....X.......X......X............X.............] duration: 7.138339628s trace: https://jaeger.dev.storj.io/trace/1ee0c6bff9434222 FAIL (2025-11-22 15:01:53 UTC) orbiter warning: context deadline exceeded Duration: 7.138367612s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [......X.......XX..X..XX..X.X.........XX.XX.................X] duration: 5.003882764s trace: https://jaeger.dev.storj.io/trace/19d4a4fb18f421aa FAIL (2025-11-22 15:01:46 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.003907412s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.X.........X.X.....X.......X......X............X............] duration: 6.804464391s trace: https://jaeger.dev.storj.io/trace/1ee0c6bff9434222 FAIL (2025-11-22 15:01:26 UTC) orbiter warning: context deadline exceeded Duration: 6.804492755s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [.......X.......XX..X..XX..X.X.........XX.XX.................] duration: 3.560063318s trace: https://jaeger.dev.storj.io/trace/58dc5585e5ee8756 FAIL (2025-11-22 15:01:09 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.560096172s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.................................XX.X.........X........X....] duration: 15.672726988s trace: https://jaeger.dev.storj.io/trace/1efcdf5c2a183d8d FAIL (2025-11-22 15:00:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 934958555s) [noise=true , address=145.79.227.30:4762 , piece_id=4Z7CCNLZEYQAGJNUWGJ2USQSVW6QZUTCNFKC3JWRIOKV6VBVIYKQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.384231169 (318h54m50.93570163s) [noise=true , address=145.79.227.18:4007 , piece_id=KM5TO27LZXJZZAT2DB5JTP7R5LOPJHUU6T34RZI7HH7PRWPW4ZNQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.384687758 (318h54m50.93615822s) [noise=true , address=145.79.227.26:4654 , piece_id=VCRM5O6NHOJOKNQUG5TTZUT3JIDQOMNF2DD6H7KWUJFOQ3P75QGQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.385168865 (318h54m50.936639326s) [noise=true , address=77.90.31.18:4805 , piece_id=3EHB5IFBQRLXFBOVCT25RS4CIXVMSVOE7EKDV26DXOEY2NDJ2YGA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.385605395 (318h54m50.937075867s) [noise=true , address=145.79.227.32:4835 , piece_id=7GD4QUTGLIPOEU25FIDE7VDO3AUJ2RZDU4XKI476KNU6QTETC52A, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.386294879 (318h54m50.93776534s) [noise=true , address=145.79.227.21:4473 , piece_id=D5FC7PVECZTQR64BFVTV3L2HOXCITABN57LLNRSMVCSGQICRYKUQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.386958642 (318h54m50.938429103s) [noise=true , address=145.79.227.14:4837 , piece_id=NR427EUHKVSQ6HMMZCHL4LCBV6VRVTANZDXMX6JHQI2TTQN5R5VQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387097236 (318h54m50.938567687s) [noise=true , address=145.79.227.9:4365 , piece_id=DBW2L6GHAWIHOMOZOTQFQ57Y2MA6KP6SGUO6AJDDF65ERQ324VZQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387456781 (318h54m50.938927242s) [noise=true , address=145.79.227.19:4078 , piece_id=ER3KKYJ2MJQC3HRRBVULC3ING5GLGOHFVD2QBWF2MVAXF6ODMUUA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387755300 (318h54m50.939225761s) [noise=true , address=145.79.227.28:4725 , piece_id=HVMAX2UWJIF2UJQIJS5UBTO5TOIJI5DCDGGURZN4SOMRY3Z4OF4A, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.388354220 (318h54m50.939824681s) [noise=true , address=145.79.227.10:4402 , piece_id=ZU6A2O2R73PIVIDF5NOKMJZH4QX4LOCFSYOQLFOFAA5BEVQH52KA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.388862268 (318h54m50.940332729s) [noise=true , address=145.79.227.22:4509 , piece_id=WRHXVGJZQRTY6ZKMRU6IFYZSMLR6UATXFZOTCPRYKHERA2WKMGNA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.389411814 (318h54m50.940882275s) [noise=true , address=145.79.227.8:4330 , piece_id=LFTZWYQKW56VLMJPOGGLB6IJILPGJNEHIRIYD2F4J34ZH24PCSMQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.389753224 (318h54m50.941223675s) [noise=true , address=145.79.227.16:4909 , piece_id=3DAVIY3ETZVG5CPP5BQB7QL473J5HZG6EQ6BYZBIAN6TWZ5FGNHQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.391102172 (318h54m50.942572633s) [noise=true , address=45.91.116.7:4010 , piece_id=VZCYHOAR73DVCRDT5GBT65JI53NP27OEGA3VYLM66LPQR23T4MTA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:54.404583235 (318h54m52.956053696s) [upload=280863]: Next returned nil segment Nov 22 14:58:54.404601019 (318h54m52.956071481s) [upload=280863]: Waiting for error group managing segments... Nov 22 14:58:54.928222688 (318h54m53.479693149s) [noise=true , address=77.90.31.18:4805 , piece_id=3EHB5IFBQRLXFBOVCT25RS4CIXVMSVOE7EKDV26DXOEY2NDJ2YGA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:58:55.175979631 (318h54m53.727450092s) [noise=true , address=45.91.116.7:4010 , piece_id=VZCYHOAR73DVCRDT5GBT65JI53NP27OEGA3VYLM66LPQR23T4MTA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:03.279622498 (318h55m1.831092959s) [noise=true , address=145.79.227.14:4837 , piece_id=NR427EUHKVSQ6HMMZCHL4LCBV6VRVTANZDXMX6JHQI2TTQN5R5VQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:04.274652248 (318h55m2.826122709s) [noise=true , address=145.79.227.28:4725 , piece_id=HVMAX2UWJIF2UJQIJS5UBTO5TOIJI5DCDGGURZN4SOMRY3Z4OF4A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:05.089136230 (318h55m3.640606701s) [noise=true , address=145.79.227.22:4509 , piece_id=WRHXVGJZQRTY6ZKMRU6IFYZSMLR6UATXFZOTCPRYKHERA2WKMGNA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:05.587545699 (318h55m4.13901616s) [noise=true , address=145.79.227.32:4835 , piece_id=7GD4QUTGLIPOEU25FIDE7VDO3AUJ2RZDU4XKI476KNU6QTETC52A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:06.956814829 (318h55m5.50828528s) [noise=true , address=145.79.227.8:4330 , piece_id=LFTZWYQKW56VLMJPOGGLB6IJILPGJNEHIRIYD2F4J34ZH24PCSMQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:07.254777763 (318h55m5.806248224s) [noise=true , address=145.79.227.30:4762 , piece_id=4Z7CCNLZEYQAGJNUWGJ2USQSVW6QZUTCNFKC3JWRIOKV6VBVIYKQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:19.983388333 (318h55m18.534858794s) [noise=true , address=145.79.227.10:4402 , piece_id=ZU6A2O2R73PIVIDF5NOKMJZH4QX4LOCFSYOQLFOFAA5BEVQH52KA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection Nov 22 14:59:19.983536355 (318h55m18.535006816s) [noise=true , address=145.79.227.21:4473 , piece_id=D5FC7PVECZTQR64BFVTV3L2HOXCITABN57LLNRSMVCSGQICRYKUQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection Nov 22 14:59:19.983802211 (318h55m18.535272703s) [noise=true , address=145.79.227.18:4007 , piece_id=KM5TO27LZXJZZAT2DB5JTP7R5LOPJHUU6T34RZI7HH7PRWPW4ZNQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection Nov 22 14:59:19.983812000 (318h55m18.535282461s) [noise=true , address=145.79.227.9:4365 , piece_id=DBW2L6GHAWIHOMOZOTQFQ57Y2MA6KP6SGUO6AJDDF65ERQ324VZQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection Nov 22 14:59:19.983896611 (318h55m18.535367073s) [noise=true , address=145.79.227.19:4078 , piece_id=ER3KKYJ2MJQC3HRRBVULC3ING5GLGOHFVD2QBWF2MVAXF6ODMUUA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection; write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection Nov 22 14:59:19.984068268 (318h55m18.535538719s) [noise=true , address=145.79.227.26:4654 , piece_id=VCRM5O6NHOJOKNQUG5TTZUT3JIDQOMNF2DD6H7KWUJFOQ3P75QGQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: read tcp 188.166.225.170:36440->145.79.227.26:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984229324 (318h55m18.535699776s) [noise=true , address=145.79.227.8:4329 , piece_id=YJG3LYBDMPS4QQNLG24MS7ZWUM6EOI6EPDGVBGLO33CVEQQKGX6A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: read tcp 188.166.225.170:44902->145.79.227.8:4329: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984374972 (318h55m18.535845423s) [noise=true , address=145.79.227.16:4909 , piece_id=3DAVIY3ETZVG5CPP5BQB7QL473J5HZG6EQ6BYZBIAN6TWZ5FGNHQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: read tcp 188.166.225.170:59064->145.79.227.16:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984400911 (318h55m18.535871362s) [seg_pos={0 0}, upload=280863]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 22 14:59:20.539921660 (318h55m19.091392121s) [upload=280863]: Done uploading segments... Nov 22 14:59:20.539961818 (318h55m19.091432269s) [upload=280863]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 15.672805789s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [..X.........X.X.....X.......X......X............X...........] duration: 7.110385077s trace: https://jaeger.dev.storj.io/trace/1ee0c6bff9434222 FAIL (2025-11-22 15:00:46 UTC) orbiter warning: context deadline exceeded Duration: 7.110430154s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..................................XX.X.........X........X...] duration: 14.241439293s trace: https://jaeger.dev.storj.io/trace/1efcdf5c2a183d8d FAIL (2025-11-22 15:00:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 934958555s) [noise=true , address=145.79.227.30:4762 , piece_id=4Z7CCNLZEYQAGJNUWGJ2USQSVW6QZUTCNFKC3JWRIOKV6VBVIYKQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.384231169 (318h54m50.93570163s) [noise=true , address=145.79.227.18:4007 , piece_id=KM5TO27LZXJZZAT2DB5JTP7R5LOPJHUU6T34RZI7HH7PRWPW4ZNQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.384687758 (318h54m50.93615822s) [noise=true , address=145.79.227.26:4654 , piece_id=VCRM5O6NHOJOKNQUG5TTZUT3JIDQOMNF2DD6H7KWUJFOQ3P75QGQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.385168865 (318h54m50.936639326s) [noise=true , address=77.90.31.18:4805 , piece_id=3EHB5IFBQRLXFBOVCT25RS4CIXVMSVOE7EKDV26DXOEY2NDJ2YGA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.385605395 (318h54m50.937075867s) [noise=true , address=145.79.227.32:4835 , piece_id=7GD4QUTGLIPOEU25FIDE7VDO3AUJ2RZDU4XKI476KNU6QTETC52A, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.386294879 (318h54m50.93776534s) [noise=true , address=145.79.227.21:4473 , piece_id=D5FC7PVECZTQR64BFVTV3L2HOXCITABN57LLNRSMVCSGQICRYKUQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.386958642 (318h54m50.938429103s) [noise=true , address=145.79.227.14:4837 , piece_id=NR427EUHKVSQ6HMMZCHL4LCBV6VRVTANZDXMX6JHQI2TTQN5R5VQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387097236 (318h54m50.938567687s) [noise=true , address=145.79.227.9:4365 , piece_id=DBW2L6GHAWIHOMOZOTQFQ57Y2MA6KP6SGUO6AJDDF65ERQ324VZQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387456781 (318h54m50.938927242s) [noise=true , address=145.79.227.19:4078 , piece_id=ER3KKYJ2MJQC3HRRBVULC3ING5GLGOHFVD2QBWF2MVAXF6ODMUUA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387755300 (318h54m50.939225761s) [noise=true , address=145.79.227.28:4725 , piece_id=HVMAX2UWJIF2UJQIJS5UBTO5TOIJI5DCDGGURZN4SOMRY3Z4OF4A, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.388354220 (318h54m50.939824681s) [noise=true , address=145.79.227.10:4402 , piece_id=ZU6A2O2R73PIVIDF5NOKMJZH4QX4LOCFSYOQLFOFAA5BEVQH52KA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.388862268 (318h54m50.940332729s) [noise=true , address=145.79.227.22:4509 , piece_id=WRHXVGJZQRTY6ZKMRU6IFYZSMLR6UATXFZOTCPRYKHERA2WKMGNA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.389411814 (318h54m50.940882275s) [noise=true , address=145.79.227.8:4330 , piece_id=LFTZWYQKW56VLMJPOGGLB6IJILPGJNEHIRIYD2F4J34ZH24PCSMQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.389753224 (318h54m50.941223675s) [noise=true , address=145.79.227.16:4909 , piece_id=3DAVIY3ETZVG5CPP5BQB7QL473J5HZG6EQ6BYZBIAN6TWZ5FGNHQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.391102172 (318h54m50.942572633s) [noise=true , address=45.91.116.7:4010 , piece_id=VZCYHOAR73DVCRDT5GBT65JI53NP27OEGA3VYLM66LPQR23T4MTA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:54.404583235 (318h54m52.956053696s) [upload=280863]: Next returned nil segment Nov 22 14:58:54.404601019 (318h54m52.956071481s) [upload=280863]: Waiting for error group managing segments... Nov 22 14:58:54.928222688 (318h54m53.479693149s) [noise=true , address=77.90.31.18:4805 , piece_id=3EHB5IFBQRLXFBOVCT25RS4CIXVMSVOE7EKDV26DXOEY2NDJ2YGA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:58:55.175979631 (318h54m53.727450092s) [noise=true , address=45.91.116.7:4010 , piece_id=VZCYHOAR73DVCRDT5GBT65JI53NP27OEGA3VYLM66LPQR23T4MTA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:03.279622498 (318h55m1.831092959s) [noise=true , address=145.79.227.14:4837 , piece_id=NR427EUHKVSQ6HMMZCHL4LCBV6VRVTANZDXMX6JHQI2TTQN5R5VQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:04.274652248 (318h55m2.826122709s) [noise=true , address=145.79.227.28:4725 , piece_id=HVMAX2UWJIF2UJQIJS5UBTO5TOIJI5DCDGGURZN4SOMRY3Z4OF4A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:05.089136230 (318h55m3.640606701s) [noise=true , address=145.79.227.22:4509 , piece_id=WRHXVGJZQRTY6ZKMRU6IFYZSMLR6UATXFZOTCPRYKHERA2WKMGNA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:05.587545699 (318h55m4.13901616s) [noise=true , address=145.79.227.32:4835 , piece_id=7GD4QUTGLIPOEU25FIDE7VDO3AUJ2RZDU4XKI476KNU6QTETC52A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:06.956814829 (318h55m5.50828528s) [noise=true , address=145.79.227.8:4330 , piece_id=LFTZWYQKW56VLMJPOGGLB6IJILPGJNEHIRIYD2F4J34ZH24PCSMQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:07.254777763 (318h55m5.806248224s) [noise=true , address=145.79.227.30:4762 , piece_id=4Z7CCNLZEYQAGJNUWGJ2USQSVW6QZUTCNFKC3JWRIOKV6VBVIYKQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:19.983388333 (318h55m18.534858794s) [noise=true , address=145.79.227.10:4402 , piece_id=ZU6A2O2R73PIVIDF5NOKMJZH4QX4LOCFSYOQLFOFAA5BEVQH52KA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection Nov 22 14:59:19.983536355 (318h55m18.535006816s) [noise=true , address=145.79.227.21:4473 , piece_id=D5FC7PVECZTQR64BFVTV3L2HOXCITABN57LLNRSMVCSGQICRYKUQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection Nov 22 14:59:19.983802211 (318h55m18.535272703s) [noise=true , address=145.79.227.18:4007 , piece_id=KM5TO27LZXJZZAT2DB5JTP7R5LOPJHUU6T34RZI7HH7PRWPW4ZNQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection Nov 22 14:59:19.983812000 (318h55m18.535282461s) [noise=true , address=145.79.227.9:4365 , piece_id=DBW2L6GHAWIHOMOZOTQFQ57Y2MA6KP6SGUO6AJDDF65ERQ324VZQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection Nov 22 14:59:19.983896611 (318h55m18.535367073s) [noise=true , address=145.79.227.19:4078 , piece_id=ER3KKYJ2MJQC3HRRBVULC3ING5GLGOHFVD2QBWF2MVAXF6ODMUUA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection; write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection Nov 22 14:59:19.984068268 (318h55m18.535538719s) [noise=true , address=145.79.227.26:4654 , piece_id=VCRM5O6NHOJOKNQUG5TTZUT3JIDQOMNF2DD6H7KWUJFOQ3P75QGQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: read tcp 188.166.225.170:36440->145.79.227.26:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984229324 (318h55m18.535699776s) [noise=true , address=145.79.227.8:4329 , piece_id=YJG3LYBDMPS4QQNLG24MS7ZWUM6EOI6EPDGVBGLO33CVEQQKGX6A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: read tcp 188.166.225.170:44902->145.79.227.8:4329: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984374972 (318h55m18.535845423s) [noise=true , address=145.79.227.16:4909 , piece_id=3DAVIY3ETZVG5CPP5BQB7QL473J5HZG6EQ6BYZBIAN6TWZ5FGNHQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: read tcp 188.166.225.170:59064->145.79.227.16:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984400911 (318h55m18.535871362s) [seg_pos={0 0}, upload=280863]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 22 14:59:20.539921660 (318h55m19.091392121s) [upload=280863]: Done uploading segments... Nov 22 14:59:20.539961818 (318h55m19.091432269s) [upload=280863]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 14.241472527s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [........X.......XX..X..XX..X.X.........XX.XX................] duration: 4.437379926s trace: https://jaeger.dev.storj.io/trace/58dc5585e5ee8756 FAIL (2025-11-22 15:00:11 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.437412658s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [...X.........X.X.....X.......X......X............X..........] duration: 7.00649145s trace: https://jaeger.dev.storj.io/trace/1ee0c6bff9434222 FAIL (2025-11-22 15:00:02 UTC) orbiter warning: context deadline exceeded Duration: 7.006578234s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...................................XX.X.........X........X..] duration: 14.639617784s trace: https://jaeger.dev.storj.io/trace/1efcdf5c2a183d8d FAIL (2025-11-22 14:59:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 934958555s) [noise=true , address=145.79.227.30:4762 , piece_id=4Z7CCNLZEYQAGJNUWGJ2USQSVW6QZUTCNFKC3JWRIOKV6VBVIYKQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.384231169 (318h54m50.93570163s) [noise=true , address=145.79.227.18:4007 , piece_id=KM5TO27LZXJZZAT2DB5JTP7R5LOPJHUU6T34RZI7HH7PRWPW4ZNQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.384687758 (318h54m50.93615822s) [noise=true , address=145.79.227.26:4654 , piece_id=VCRM5O6NHOJOKNQUG5TTZUT3JIDQOMNF2DD6H7KWUJFOQ3P75QGQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.385168865 (318h54m50.936639326s) [noise=true , address=77.90.31.18:4805 , piece_id=3EHB5IFBQRLXFBOVCT25RS4CIXVMSVOE7EKDV26DXOEY2NDJ2YGA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.385605395 (318h54m50.937075867s) [noise=true , address=145.79.227.32:4835 , piece_id=7GD4QUTGLIPOEU25FIDE7VDO3AUJ2RZDU4XKI476KNU6QTETC52A, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.386294879 (318h54m50.93776534s) [noise=true , address=145.79.227.21:4473 , piece_id=D5FC7PVECZTQR64BFVTV3L2HOXCITABN57LLNRSMVCSGQICRYKUQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.386958642 (318h54m50.938429103s) [noise=true , address=145.79.227.14:4837 , piece_id=NR427EUHKVSQ6HMMZCHL4LCBV6VRVTANZDXMX6JHQI2TTQN5R5VQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387097236 (318h54m50.938567687s) [noise=true , address=145.79.227.9:4365 , piece_id=DBW2L6GHAWIHOMOZOTQFQ57Y2MA6KP6SGUO6AJDDF65ERQ324VZQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387456781 (318h54m50.938927242s) [noise=true , address=145.79.227.19:4078 , piece_id=ER3KKYJ2MJQC3HRRBVULC3ING5GLGOHFVD2QBWF2MVAXF6ODMUUA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387755300 (318h54m50.939225761s) [noise=true , address=145.79.227.28:4725 , piece_id=HVMAX2UWJIF2UJQIJS5UBTO5TOIJI5DCDGGURZN4SOMRY3Z4OF4A, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.388354220 (318h54m50.939824681s) [noise=true , address=145.79.227.10:4402 , piece_id=ZU6A2O2R73PIVIDF5NOKMJZH4QX4LOCFSYOQLFOFAA5BEVQH52KA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.388862268 (318h54m50.940332729s) [noise=true , address=145.79.227.22:4509 , piece_id=WRHXVGJZQRTY6ZKMRU6IFYZSMLR6UATXFZOTCPRYKHERA2WKMGNA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.389411814 (318h54m50.940882275s) [noise=true , address=145.79.227.8:4330 , piece_id=LFTZWYQKW56VLMJPOGGLB6IJILPGJNEHIRIYD2F4J34ZH24PCSMQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.389753224 (318h54m50.941223675s) [noise=true , address=145.79.227.16:4909 , piece_id=3DAVIY3ETZVG5CPP5BQB7QL473J5HZG6EQ6BYZBIAN6TWZ5FGNHQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.391102172 (318h54m50.942572633s) [noise=true , address=45.91.116.7:4010 , piece_id=VZCYHOAR73DVCRDT5GBT65JI53NP27OEGA3VYLM66LPQR23T4MTA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:54.404583235 (318h54m52.956053696s) [upload=280863]: Next returned nil segment Nov 22 14:58:54.404601019 (318h54m52.956071481s) [upload=280863]: Waiting for error group managing segments... Nov 22 14:58:54.928222688 (318h54m53.479693149s) [noise=true , address=77.90.31.18:4805 , piece_id=3EHB5IFBQRLXFBOVCT25RS4CIXVMSVOE7EKDV26DXOEY2NDJ2YGA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:58:55.175979631 (318h54m53.727450092s) [noise=true , address=45.91.116.7:4010 , piece_id=VZCYHOAR73DVCRDT5GBT65JI53NP27OEGA3VYLM66LPQR23T4MTA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:03.279622498 (318h55m1.831092959s) [noise=true , address=145.79.227.14:4837 , piece_id=NR427EUHKVSQ6HMMZCHL4LCBV6VRVTANZDXMX6JHQI2TTQN5R5VQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:04.274652248 (318h55m2.826122709s) [noise=true , address=145.79.227.28:4725 , piece_id=HVMAX2UWJIF2UJQIJS5UBTO5TOIJI5DCDGGURZN4SOMRY3Z4OF4A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:05.089136230 (318h55m3.640606701s) [noise=true , address=145.79.227.22:4509 , piece_id=WRHXVGJZQRTY6ZKMRU6IFYZSMLR6UATXFZOTCPRYKHERA2WKMGNA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:05.587545699 (318h55m4.13901616s) [noise=true , address=145.79.227.32:4835 , piece_id=7GD4QUTGLIPOEU25FIDE7VDO3AUJ2RZDU4XKI476KNU6QTETC52A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:06.956814829 (318h55m5.50828528s) [noise=true , address=145.79.227.8:4330 , piece_id=LFTZWYQKW56VLMJPOGGLB6IJILPGJNEHIRIYD2F4J34ZH24PCSMQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:07.254777763 (318h55m5.806248224s) [noise=true , address=145.79.227.30:4762 , piece_id=4Z7CCNLZEYQAGJNUWGJ2USQSVW6QZUTCNFKC3JWRIOKV6VBVIYKQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:19.983388333 (318h55m18.534858794s) [noise=true , address=145.79.227.10:4402 , piece_id=ZU6A2O2R73PIVIDF5NOKMJZH4QX4LOCFSYOQLFOFAA5BEVQH52KA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection Nov 22 14:59:19.983536355 (318h55m18.535006816s) [noise=true , address=145.79.227.21:4473 , piece_id=D5FC7PVECZTQR64BFVTV3L2HOXCITABN57LLNRSMVCSGQICRYKUQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection Nov 22 14:59:19.983802211 (318h55m18.535272703s) [noise=true , address=145.79.227.18:4007 , piece_id=KM5TO27LZXJZZAT2DB5JTP7R5LOPJHUU6T34RZI7HH7PRWPW4ZNQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection Nov 22 14:59:19.983812000 (318h55m18.535282461s) [noise=true , address=145.79.227.9:4365 , piece_id=DBW2L6GHAWIHOMOZOTQFQ57Y2MA6KP6SGUO6AJDDF65ERQ324VZQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection Nov 22 14:59:19.983896611 (318h55m18.535367073s) [noise=true , address=145.79.227.19:4078 , piece_id=ER3KKYJ2MJQC3HRRBVULC3ING5GLGOHFVD2QBWF2MVAXF6ODMUUA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection; write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection Nov 22 14:59:19.984068268 (318h55m18.535538719s) [noise=true , address=145.79.227.26:4654 , piece_id=VCRM5O6NHOJOKNQUG5TTZUT3JIDQOMNF2DD6H7KWUJFOQ3P75QGQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: read tcp 188.166.225.170:36440->145.79.227.26:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984229324 (318h55m18.535699776s) [noise=true , address=145.79.227.8:4329 , piece_id=YJG3LYBDMPS4QQNLG24MS7ZWUM6EOI6EPDGVBGLO33CVEQQKGX6A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: read tcp 188.166.225.170:44902->145.79.227.8:4329: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984374972 (318h55m18.535845423s) [noise=true , address=145.79.227.16:4909 , piece_id=3DAVIY3ETZVG5CPP5BQB7QL473J5HZG6EQ6BYZBIAN6TWZ5FGNHQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: read tcp 188.166.225.170:59064->145.79.227.16:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984400911 (318h55m18.535871362s) [seg_pos={0 0}, upload=280863]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 22 14:59:20.539921660 (318h55m19.091392121s) [upload=280863]: Done uploading segments... Nov 22 14:59:20.539961818 (318h55m19.091432269s) [upload=280863]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 14.6396464s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [....................................XX.X.........X........X.] duration: 14.541514139s trace: https://jaeger.dev.storj.io/trace/1efcdf5c2a183d8d FAIL (2025-11-22 14:59:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 934958555s) [noise=true , address=145.79.227.30:4762 , piece_id=4Z7CCNLZEYQAGJNUWGJ2USQSVW6QZUTCNFKC3JWRIOKV6VBVIYKQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.384231169 (318h54m50.93570163s) [noise=true , address=145.79.227.18:4007 , piece_id=KM5TO27LZXJZZAT2DB5JTP7R5LOPJHUU6T34RZI7HH7PRWPW4ZNQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.384687758 (318h54m50.93615822s) [noise=true , address=145.79.227.26:4654 , piece_id=VCRM5O6NHOJOKNQUG5TTZUT3JIDQOMNF2DD6H7KWUJFOQ3P75QGQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.385168865 (318h54m50.936639326s) [noise=true , address=77.90.31.18:4805 , piece_id=3EHB5IFBQRLXFBOVCT25RS4CIXVMSVOE7EKDV26DXOEY2NDJ2YGA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.385605395 (318h54m50.937075867s) [noise=true , address=145.79.227.32:4835 , piece_id=7GD4QUTGLIPOEU25FIDE7VDO3AUJ2RZDU4XKI476KNU6QTETC52A, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.386294879 (318h54m50.93776534s) [noise=true , address=145.79.227.21:4473 , piece_id=D5FC7PVECZTQR64BFVTV3L2HOXCITABN57LLNRSMVCSGQICRYKUQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.386958642 (318h54m50.938429103s) [noise=true , address=145.79.227.14:4837 , piece_id=NR427EUHKVSQ6HMMZCHL4LCBV6VRVTANZDXMX6JHQI2TTQN5R5VQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387097236 (318h54m50.938567687s) [noise=true , address=145.79.227.9:4365 , piece_id=DBW2L6GHAWIHOMOZOTQFQ57Y2MA6KP6SGUO6AJDDF65ERQ324VZQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387456781 (318h54m50.938927242s) [noise=true , address=145.79.227.19:4078 , piece_id=ER3KKYJ2MJQC3HRRBVULC3ING5GLGOHFVD2QBWF2MVAXF6ODMUUA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387755300 (318h54m50.939225761s) [noise=true , address=145.79.227.28:4725 , piece_id=HVMAX2UWJIF2UJQIJS5UBTO5TOIJI5DCDGGURZN4SOMRY3Z4OF4A, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.388354220 (318h54m50.939824681s) [noise=true , address=145.79.227.10:4402 , piece_id=ZU6A2O2R73PIVIDF5NOKMJZH4QX4LOCFSYOQLFOFAA5BEVQH52KA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.388862268 (318h54m50.940332729s) [noise=true , address=145.79.227.22:4509 , piece_id=WRHXVGJZQRTY6ZKMRU6IFYZSMLR6UATXFZOTCPRYKHERA2WKMGNA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.389411814 (318h54m50.940882275s) [noise=true , address=145.79.227.8:4330 , piece_id=LFTZWYQKW56VLMJPOGGLB6IJILPGJNEHIRIYD2F4J34ZH24PCSMQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.389753224 (318h54m50.941223675s) [noise=true , address=145.79.227.16:4909 , piece_id=3DAVIY3ETZVG5CPP5BQB7QL473J5HZG6EQ6BYZBIAN6TWZ5FGNHQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.391102172 (318h54m50.942572633s) [noise=true , address=45.91.116.7:4010 , piece_id=VZCYHOAR73DVCRDT5GBT65JI53NP27OEGA3VYLM66LPQR23T4MTA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:54.404583235 (318h54m52.956053696s) [upload=280863]: Next returned nil segment Nov 22 14:58:54.404601019 (318h54m52.956071481s) [upload=280863]: Waiting for error group managing segments... Nov 22 14:58:54.928222688 (318h54m53.479693149s) [noise=true , address=77.90.31.18:4805 , piece_id=3EHB5IFBQRLXFBOVCT25RS4CIXVMSVOE7EKDV26DXOEY2NDJ2YGA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:58:55.175979631 (318h54m53.727450092s) [noise=true , address=45.91.116.7:4010 , piece_id=VZCYHOAR73DVCRDT5GBT65JI53NP27OEGA3VYLM66LPQR23T4MTA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:03.279622498 (318h55m1.831092959s) [noise=true , address=145.79.227.14:4837 , piece_id=NR427EUHKVSQ6HMMZCHL4LCBV6VRVTANZDXMX6JHQI2TTQN5R5VQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:04.274652248 (318h55m2.826122709s) [noise=true , address=145.79.227.28:4725 , piece_id=HVMAX2UWJIF2UJQIJS5UBTO5TOIJI5DCDGGURZN4SOMRY3Z4OF4A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:05.089136230 (318h55m3.640606701s) [noise=true , address=145.79.227.22:4509 , piece_id=WRHXVGJZQRTY6ZKMRU6IFYZSMLR6UATXFZOTCPRYKHERA2WKMGNA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:05.587545699 (318h55m4.13901616s) [noise=true , address=145.79.227.32:4835 , piece_id=7GD4QUTGLIPOEU25FIDE7VDO3AUJ2RZDU4XKI476KNU6QTETC52A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:06.956814829 (318h55m5.50828528s) [noise=true , address=145.79.227.8:4330 , piece_id=LFTZWYQKW56VLMJPOGGLB6IJILPGJNEHIRIYD2F4J34ZH24PCSMQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:07.254777763 (318h55m5.806248224s) [noise=true , address=145.79.227.30:4762 , piece_id=4Z7CCNLZEYQAGJNUWGJ2USQSVW6QZUTCNFKC3JWRIOKV6VBVIYKQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:19.983388333 (318h55m18.534858794s) [noise=true , address=145.79.227.10:4402 , piece_id=ZU6A2O2R73PIVIDF5NOKMJZH4QX4LOCFSYOQLFOFAA5BEVQH52KA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection Nov 22 14:59:19.983536355 (318h55m18.535006816s) [noise=true , address=145.79.227.21:4473 , piece_id=D5FC7PVECZTQR64BFVTV3L2HOXCITABN57LLNRSMVCSGQICRYKUQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection Nov 22 14:59:19.983802211 (318h55m18.535272703s) [noise=true , address=145.79.227.18:4007 , piece_id=KM5TO27LZXJZZAT2DB5JTP7R5LOPJHUU6T34RZI7HH7PRWPW4ZNQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection Nov 22 14:59:19.983812000 (318h55m18.535282461s) [noise=true , address=145.79.227.9:4365 , piece_id=DBW2L6GHAWIHOMOZOTQFQ57Y2MA6KP6SGUO6AJDDF65ERQ324VZQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection Nov 22 14:59:19.983896611 (318h55m18.535367073s) [noise=true , address=145.79.227.19:4078 , piece_id=ER3KKYJ2MJQC3HRRBVULC3ING5GLGOHFVD2QBWF2MVAXF6ODMUUA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection; write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection Nov 22 14:59:19.984068268 (318h55m18.535538719s) [noise=true , address=145.79.227.26:4654 , piece_id=VCRM5O6NHOJOKNQUG5TTZUT3JIDQOMNF2DD6H7KWUJFOQ3P75QGQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: read tcp 188.166.225.170:36440->145.79.227.26:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984229324 (318h55m18.535699776s) [noise=true , address=145.79.227.8:4329 , piece_id=YJG3LYBDMPS4QQNLG24MS7ZWUM6EOI6EPDGVBGLO33CVEQQKGX6A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: read tcp 188.166.225.170:44902->145.79.227.8:4329: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984374972 (318h55m18.535845423s) [noise=true , address=145.79.227.16:4909 , piece_id=3DAVIY3ETZVG5CPP5BQB7QL473J5HZG6EQ6BYZBIAN6TWZ5FGNHQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: read tcp 188.166.225.170:59064->145.79.227.16:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984400911 (318h55m18.535871362s) [seg_pos={0 0}, upload=280863]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 22 14:59:20.539921660 (318h55m19.091392121s) [upload=280863]: Done uploading segments... Nov 22 14:59:20.539961818 (318h55m19.091432269s) [upload=280863]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 14.541543715s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....................................XX.X.........X........X] duration: 30.558496078s trace: https://jaeger.dev.storj.io/trace/1efcdf5c2a183d8d FAIL (2025-11-22 14:59:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 934958555s) [noise=true , address=145.79.227.30:4762 , piece_id=4Z7CCNLZEYQAGJNUWGJ2USQSVW6QZUTCNFKC3JWRIOKV6VBVIYKQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.384231169 (318h54m50.93570163s) [noise=true , address=145.79.227.18:4007 , piece_id=KM5TO27LZXJZZAT2DB5JTP7R5LOPJHUU6T34RZI7HH7PRWPW4ZNQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.384687758 (318h54m50.93615822s) [noise=true , address=145.79.227.26:4654 , piece_id=VCRM5O6NHOJOKNQUG5TTZUT3JIDQOMNF2DD6H7KWUJFOQ3P75QGQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.385168865 (318h54m50.936639326s) [noise=true , address=77.90.31.18:4805 , piece_id=3EHB5IFBQRLXFBOVCT25RS4CIXVMSVOE7EKDV26DXOEY2NDJ2YGA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.385605395 (318h54m50.937075867s) [noise=true , address=145.79.227.32:4835 , piece_id=7GD4QUTGLIPOEU25FIDE7VDO3AUJ2RZDU4XKI476KNU6QTETC52A, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.386294879 (318h54m50.93776534s) [noise=true , address=145.79.227.21:4473 , piece_id=D5FC7PVECZTQR64BFVTV3L2HOXCITABN57LLNRSMVCSGQICRYKUQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.386958642 (318h54m50.938429103s) [noise=true , address=145.79.227.14:4837 , piece_id=NR427EUHKVSQ6HMMZCHL4LCBV6VRVTANZDXMX6JHQI2TTQN5R5VQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387097236 (318h54m50.938567687s) [noise=true , address=145.79.227.9:4365 , piece_id=DBW2L6GHAWIHOMOZOTQFQ57Y2MA6KP6SGUO6AJDDF65ERQ324VZQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387456781 (318h54m50.938927242s) [noise=true , address=145.79.227.19:4078 , piece_id=ER3KKYJ2MJQC3HRRBVULC3ING5GLGOHFVD2QBWF2MVAXF6ODMUUA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.387755300 (318h54m50.939225761s) [noise=true , address=145.79.227.28:4725 , piece_id=HVMAX2UWJIF2UJQIJS5UBTO5TOIJI5DCDGGURZN4SOMRY3Z4OF4A, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.388354220 (318h54m50.939824681s) [noise=true , address=145.79.227.10:4402 , piece_id=ZU6A2O2R73PIVIDF5NOKMJZH4QX4LOCFSYOQLFOFAA5BEVQH52KA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.388862268 (318h54m50.940332729s) [noise=true , address=145.79.227.22:4509 , piece_id=WRHXVGJZQRTY6ZKMRU6IFYZSMLR6UATXFZOTCPRYKHERA2WKMGNA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.389411814 (318h54m50.940882275s) [noise=true , address=145.79.227.8:4330 , piece_id=LFTZWYQKW56VLMJPOGGLB6IJILPGJNEHIRIYD2F4J34ZH24PCSMQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.389753224 (318h54m50.941223675s) [noise=true , address=145.79.227.16:4909 , piece_id=3DAVIY3ETZVG5CPP5BQB7QL473J5HZG6EQ6BYZBIAN6TWZ5FGNHQ, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:52.391102172 (318h54m50.942572633s) [noise=true , address=45.91.116.7:4010 , piece_id=VZCYHOAR73DVCRDT5GBT65JI53NP27OEGA3VYLM66LPQR23T4MTA, seg_pos={0 0}, upload=280863]: Uploading piece... Nov 22 14:58:54.404583235 (318h54m52.956053696s) [upload=280863]: Next returned nil segment Nov 22 14:58:54.404601019 (318h54m52.956071481s) [upload=280863]: Waiting for error group managing segments... Nov 22 14:58:54.928222688 (318h54m53.479693149s) [noise=true , address=77.90.31.18:4805 , piece_id=3EHB5IFBQRLXFBOVCT25RS4CIXVMSVOE7EKDV26DXOEY2NDJ2YGA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:58:55.175979631 (318h54m53.727450092s) [noise=true , address=45.91.116.7:4010 , piece_id=VZCYHOAR73DVCRDT5GBT65JI53NP27OEGA3VYLM66LPQR23T4MTA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:03.279622498 (318h55m1.831092959s) [noise=true , address=145.79.227.14:4837 , piece_id=NR427EUHKVSQ6HMMZCHL4LCBV6VRVTANZDXMX6JHQI2TTQN5R5VQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:04.274652248 (318h55m2.826122709s) [noise=true , address=145.79.227.28:4725 , piece_id=HVMAX2UWJIF2UJQIJS5UBTO5TOIJI5DCDGGURZN4SOMRY3Z4OF4A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:05.089136230 (318h55m3.640606701s) [noise=true , address=145.79.227.22:4509 , piece_id=WRHXVGJZQRTY6ZKMRU6IFYZSMLR6UATXFZOTCPRYKHERA2WKMGNA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:05.587545699 (318h55m4.13901616s) [noise=true , address=145.79.227.32:4835 , piece_id=7GD4QUTGLIPOEU25FIDE7VDO3AUJ2RZDU4XKI476KNU6QTETC52A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:06.956814829 (318h55m5.50828528s) [noise=true , address=145.79.227.8:4330 , piece_id=LFTZWYQKW56VLMJPOGGLB6IJILPGJNEHIRIYD2F4J34ZH24PCSMQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:07.254777763 (318h55m5.806248224s) [noise=true , address=145.79.227.30:4762 , piece_id=4Z7CCNLZEYQAGJNUWGJ2USQSVW6QZUTCNFKC3JWRIOKV6VBVIYKQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: <nil> Nov 22 14:59:19.983388333 (318h55m18.534858794s) [noise=true , address=145.79.227.10:4402 , piece_id=ZU6A2O2R73PIVIDF5NOKMJZH4QX4LOCFSYOQLFOFAA5BEVQH52KA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39688->145.79.227.10:4402: use of closed network connection Nov 22 14:59:19.983536355 (318h55m18.535006816s) [noise=true , address=145.79.227.21:4473 , piece_id=D5FC7PVECZTQR64BFVTV3L2HOXCITABN57LLNRSMVCSGQICRYKUQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50672->145.79.227.21:4473: use of closed network connection Nov 22 14:59:19.983802211 (318h55m18.535272703s) [noise=true , address=145.79.227.18:4007 , piece_id=KM5TO27LZXJZZAT2DB5JTP7R5LOPJHUU6T34RZI7HH7PRWPW4ZNQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52564->145.79.227.18:4007: use of closed network connection Nov 22 14:59:19.983812000 (318h55m18.535282461s) [noise=true , address=145.79.227.9:4365 , piece_id=DBW2L6GHAWIHOMOZOTQFQ57Y2MA6KP6SGUO6AJDDF65ERQ324VZQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection; write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59546->145.79.227.9:4365: use of closed network connection Nov 22 14:59:19.983896611 (318h55m18.535367073s) [noise=true , address=145.79.227.19:4078 , piece_id=ER3KKYJ2MJQC3HRRBVULC3ING5GLGOHFVD2QBWF2MVAXF6ODMUUA, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:145.79.227.19:4078): protocol: write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection; write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40688->145.79.227.19:4078: use of closed network connection Nov 22 14:59:19.984068268 (318h55m18.535538719s) [noise=true , address=145.79.227.26:4654 , piece_id=VCRM5O6NHOJOKNQUG5TTZUT3JIDQOMNF2DD6H7KWUJFOQ3P75QGQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: read tcp 188.166.225.170:36440->145.79.227.26:4654: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984229324 (318h55m18.535699776s) [noise=true , address=145.79.227.8:4329 , piece_id=YJG3LYBDMPS4QQNLG24MS7ZWUM6EOI6EPDGVBGLO33CVEQQKGX6A, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW, address:145.79.227.8:4329): protocol: read tcp 188.166.225.170:44902->145.79.227.8:4329: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984374972 (318h55m18.535845423s) [noise=true , address=145.79.227.16:4909 , piece_id=3DAVIY3ETZVG5CPP5BQB7QL473J5HZG6EQ6BYZBIAN6TWZ5FGNHQ, seg_pos={0 0}, upload=280863]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: read tcp 188.166.225.170:59064->145.79.227.16:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 14:59:19.984400911 (318h55m18.535871362s) [seg_pos={0 0}, upload=280863]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 22 14:59:20.539921660 (318h55m19.091392121s) [upload=280863]: Done uploading segments... Nov 22 14:59:20.539961818 (318h55m19.091432269s) [upload=280863]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.558520925s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [....X.........X.X.....X.......X......X............X.........] duration: 5.858347959s trace: https://jaeger.dev.storj.io/trace/1ee0c6bff9434222 FAIL (2025-11-22 14:59:10 UTC) orbiter warning: context deadline exceeded Duration: 5.858375249s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [X........X.......XX..X..XX..X.X.........XX.XX...............] duration: 3.39390912s trace: https://jaeger.dev.storj.io/trace/58dc5585e5ee8756 FAIL (2025-11-22 14:59:06 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.393942143s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.....X.........X.X.....X.......X......X............X........] duration: 6.146185241s trace: https://jaeger.dev.storj.io/trace/1ee0c6bff9434222 FAIL (2025-11-22 14:58:55 UTC) orbiter warning: context deadline exceeded Duration: 6.146204988s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [.X........X.......XX..X..XX..X.X.........XX.XX..............] duration: 3.565008515s trace: https://jaeger.dev.storj.io/trace/58dc5585e5ee8756 FAIL (2025-11-22 14:58:39 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.565039324s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)