Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........X................................................] duration: 3.436038338s trace: https://jaeger.dev.storj.io/trace/39fa53ba9384d9d1 FAIL (2025-11-13 21:05:20 UTC) orbiter warning: context deadline exceeded Duration: 3.436063145s core uplink us-select-1 us1-placement-20
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 794.236816ms trace: https://jaeger.dev.storj.io/trace/2f5ab0f8fc51be4b FAIL (2025-11-13 21:05:12 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:40894->45.38.24.2:443: read: connection reset by peer Duration: 794.258447ms edge authservice http us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............X...............................................] duration: 3.459234152s trace: https://jaeger.dev.storj.io/trace/39fa53ba9384d9d1 FAIL (2025-11-13 21:04:36 UTC) orbiter warning: context deadline exceeded Duration: 3.459261045s core uplink us-select-1 us1-placement-20
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 849.397052ms trace: https://jaeger.dev.storj.io/trace/2f5ab0f8fc51be4b FAIL (2025-11-13 21:04:33 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:40894->45.38.24.2:443: read: connection reset by peer Duration: 849.413593ms edge authservice http us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 6.002277793s trace: https://jaeger.dev.storj.io/trace/10b439e9c7de2697 FAIL (2025-11-13 21:04:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ent. Nov 13 20:29:56.447258646 (108h25m54.998729097s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.448610229 (108h25m55.00008068s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449193800 (108h25m55.000664251s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449745831 (108h25m55.001216292s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449971391 (108h25m55.001441842s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450522891 (108h25m55.001993352s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450954152 (108h25m55.002424613s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.451475856 (108h25m55.002946307s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.452296839 (108h25m55.00376729s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453022320 (108h25m55.004492781s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453944366 (108h25m55.005414827s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.454766252 (108h25m55.006236703s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.455465523 (108h25m55.006935984s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456308909 (108h25m55.00777937s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456875248 (108h25m55.008345709s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.457343329 (108h25m55.00881379s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:58.537001464 (108h25m57.088471915s) [upload=95398]: Next returned nil segment Nov 13 20:29:58.537023396 (108h25m57.088493847s) [upload=95398]: Waiting for error group managing segments... Nov 13 20:29:58.876118614 (108h25m57.427589065s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.165170590 (108h25m57.716641051s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.186011275 (108h25m57.737481726s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.269377922 (108h25m57.820848383s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.283448366 (108h25m57.834918817s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.432453102 (108h25m57.983923553s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:21.904252287 (108h26m20.455722748s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.383852597 (108h26m20.935323058s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.528899226 (108h26m21.080369687s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:25.250258635 (108h26m23.801729096s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection Nov 13 20:30:25.250371059 (108h26m23.80184153s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 20:30:25.250582591 (108h26m23.802053052s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection Nov 13 20:30:25.250622577 (108h26m23.802093038s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection Nov 13 20:30:25.250729441 (108h26m23.802199902s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection Nov 13 20:30:25.250859819 (108h26m23.80233028s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection Nov 13 20:30:25.250973465 (108h26m23.802443927s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection Nov 13 20:30:25.251012680 (108h26m23.802483141s) [seg_pos={0 0}, upload=95398]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 20:30:25.508769584 (108h26m24.060240055s) [upload=95398]: Done uploading segments... Nov 13 20:30:25.508818367 (108h26m24.060288828s) [upload=95398]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 6.002306738s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............X..............................................] duration: 3.504187354s trace: https://jaeger.dev.storj.io/trace/39fa53ba9384d9d1 FAIL (2025-11-13 21:03:48 UTC) orbiter warning: context deadline exceeded Duration: 3.504215658s core uplink us-select-1 us1-placement-20
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 771.118553ms trace: https://jaeger.dev.storj.io/trace/2f5ab0f8fc51be4b FAIL (2025-11-13 21:03:40 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:40894->45.38.24.2:443: read: connection reset by peer Duration: 771.145815ms edge authservice http us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 5.884057914s trace: https://jaeger.dev.storj.io/trace/10b439e9c7de2697 FAIL (2025-11-13 21:03:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ent. Nov 13 20:29:56.447258646 (108h25m54.998729097s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.448610229 (108h25m55.00008068s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449193800 (108h25m55.000664251s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449745831 (108h25m55.001216292s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449971391 (108h25m55.001441842s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450522891 (108h25m55.001993352s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450954152 (108h25m55.002424613s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.451475856 (108h25m55.002946307s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.452296839 (108h25m55.00376729s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453022320 (108h25m55.004492781s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453944366 (108h25m55.005414827s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.454766252 (108h25m55.006236703s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.455465523 (108h25m55.006935984s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456308909 (108h25m55.00777937s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456875248 (108h25m55.008345709s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.457343329 (108h25m55.00881379s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:58.537001464 (108h25m57.088471915s) [upload=95398]: Next returned nil segment Nov 13 20:29:58.537023396 (108h25m57.088493847s) [upload=95398]: Waiting for error group managing segments... Nov 13 20:29:58.876118614 (108h25m57.427589065s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.165170590 (108h25m57.716641051s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.186011275 (108h25m57.737481726s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.269377922 (108h25m57.820848383s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.283448366 (108h25m57.834918817s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.432453102 (108h25m57.983923553s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:21.904252287 (108h26m20.455722748s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.383852597 (108h26m20.935323058s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.528899226 (108h26m21.080369687s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:25.250258635 (108h26m23.801729096s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection Nov 13 20:30:25.250371059 (108h26m23.80184153s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 20:30:25.250582591 (108h26m23.802053052s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection Nov 13 20:30:25.250622577 (108h26m23.802093038s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection Nov 13 20:30:25.250729441 (108h26m23.802199902s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection Nov 13 20:30:25.250859819 (108h26m23.80233028s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection Nov 13 20:30:25.250973465 (108h26m23.802443927s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection Nov 13 20:30:25.251012680 (108h26m23.802483141s) [seg_pos={0 0}, upload=95398]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 20:30:25.508769584 (108h26m24.060240055s) [upload=95398]: Done uploading segments... Nov 13 20:30:25.508818367 (108h26m24.060288828s) [upload=95398]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.88411402s core uplink us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 819.148433ms trace: https://jaeger.dev.storj.io/trace/2f5ab0f8fc51be4b FAIL (2025-11-13 21:03:07 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:40894->45.38.24.2:443: read: connection reset by peer Duration: 819.172417ms edge authservice http us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............X.............................................] duration: 3.467883242s trace: https://jaeger.dev.storj.io/trace/39fa53ba9384d9d1 FAIL (2025-11-13 21:03:03 UTC) orbiter warning: context deadline exceeded Duration: 3.4679183s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 5.931873239s trace: https://jaeger.dev.storj.io/trace/10b439e9c7de2697 FAIL (2025-11-13 21:03:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ent. Nov 13 20:29:56.447258646 (108h25m54.998729097s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.448610229 (108h25m55.00008068s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449193800 (108h25m55.000664251s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449745831 (108h25m55.001216292s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449971391 (108h25m55.001441842s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450522891 (108h25m55.001993352s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450954152 (108h25m55.002424613s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.451475856 (108h25m55.002946307s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.452296839 (108h25m55.00376729s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453022320 (108h25m55.004492781s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453944366 (108h25m55.005414827s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.454766252 (108h25m55.006236703s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.455465523 (108h25m55.006935984s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456308909 (108h25m55.00777937s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456875248 (108h25m55.008345709s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.457343329 (108h25m55.00881379s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:58.537001464 (108h25m57.088471915s) [upload=95398]: Next returned nil segment Nov 13 20:29:58.537023396 (108h25m57.088493847s) [upload=95398]: Waiting for error group managing segments... Nov 13 20:29:58.876118614 (108h25m57.427589065s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.165170590 (108h25m57.716641051s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.186011275 (108h25m57.737481726s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.269377922 (108h25m57.820848383s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.283448366 (108h25m57.834918817s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.432453102 (108h25m57.983923553s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:21.904252287 (108h26m20.455722748s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.383852597 (108h26m20.935323058s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.528899226 (108h26m21.080369687s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:25.250258635 (108h26m23.801729096s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection Nov 13 20:30:25.250371059 (108h26m23.80184153s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 20:30:25.250582591 (108h26m23.802053052s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection Nov 13 20:30:25.250622577 (108h26m23.802093038s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection Nov 13 20:30:25.250729441 (108h26m23.802199902s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection Nov 13 20:30:25.250859819 (108h26m23.80233028s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection Nov 13 20:30:25.250973465 (108h26m23.802443927s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection Nov 13 20:30:25.251012680 (108h26m23.802483141s) [seg_pos={0 0}, upload=95398]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 20:30:25.508769584 (108h26m24.060240055s) [upload=95398]: Done uploading segments... Nov 13 20:30:25.508818367 (108h26m24.060288828s) [upload=95398]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.931904448s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 5.827732802s trace: https://jaeger.dev.storj.io/trace/10b439e9c7de2697 FAIL (2025-11-13 21:02:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ent. Nov 13 20:29:56.447258646 (108h25m54.998729097s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.448610229 (108h25m55.00008068s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449193800 (108h25m55.000664251s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449745831 (108h25m55.001216292s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449971391 (108h25m55.001441842s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450522891 (108h25m55.001993352s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450954152 (108h25m55.002424613s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.451475856 (108h25m55.002946307s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.452296839 (108h25m55.00376729s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453022320 (108h25m55.004492781s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453944366 (108h25m55.005414827s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.454766252 (108h25m55.006236703s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.455465523 (108h25m55.006935984s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456308909 (108h25m55.00777937s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456875248 (108h25m55.008345709s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.457343329 (108h25m55.00881379s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:58.537001464 (108h25m57.088471915s) [upload=95398]: Next returned nil segment Nov 13 20:29:58.537023396 (108h25m57.088493847s) [upload=95398]: Waiting for error group managing segments... Nov 13 20:29:58.876118614 (108h25m57.427589065s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.165170590 (108h25m57.716641051s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.186011275 (108h25m57.737481726s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.269377922 (108h25m57.820848383s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.283448366 (108h25m57.834918817s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.432453102 (108h25m57.983923553s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:21.904252287 (108h26m20.455722748s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.383852597 (108h26m20.935323058s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.528899226 (108h26m21.080369687s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:25.250258635 (108h26m23.801729096s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection Nov 13 20:30:25.250371059 (108h26m23.80184153s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 20:30:25.250582591 (108h26m23.802053052s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection Nov 13 20:30:25.250622577 (108h26m23.802093038s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection Nov 13 20:30:25.250729441 (108h26m23.802199902s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection Nov 13 20:30:25.250859819 (108h26m23.80233028s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection Nov 13 20:30:25.250973465 (108h26m23.802443927s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection Nov 13 20:30:25.251012680 (108h26m23.802483141s) [seg_pos={0 0}, upload=95398]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 20:30:25.508769584 (108h26m24.060240055s) [upload=95398]: Done uploading segments... Nov 13 20:30:25.508818367 (108h26m24.060288828s) [upload=95398]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.827776886s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............X............................................] duration: 3.447072515s trace: https://jaeger.dev.storj.io/trace/39fa53ba9384d9d1 FAIL (2025-11-13 21:02:19 UTC) orbiter warning: context deadline exceeded Duration: 3.447094556s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 5.949578195s trace: https://jaeger.dev.storj.io/trace/10b439e9c7de2697 FAIL (2025-11-13 21:02:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ent. Nov 13 20:29:56.447258646 (108h25m54.998729097s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.448610229 (108h25m55.00008068s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449193800 (108h25m55.000664251s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449745831 (108h25m55.001216292s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449971391 (108h25m55.001441842s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450522891 (108h25m55.001993352s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450954152 (108h25m55.002424613s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.451475856 (108h25m55.002946307s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.452296839 (108h25m55.00376729s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453022320 (108h25m55.004492781s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453944366 (108h25m55.005414827s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.454766252 (108h25m55.006236703s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.455465523 (108h25m55.006935984s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456308909 (108h25m55.00777937s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456875248 (108h25m55.008345709s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.457343329 (108h25m55.00881379s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:58.537001464 (108h25m57.088471915s) [upload=95398]: Next returned nil segment Nov 13 20:29:58.537023396 (108h25m57.088493847s) [upload=95398]: Waiting for error group managing segments... Nov 13 20:29:58.876118614 (108h25m57.427589065s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.165170590 (108h25m57.716641051s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.186011275 (108h25m57.737481726s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.269377922 (108h25m57.820848383s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.283448366 (108h25m57.834918817s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.432453102 (108h25m57.983923553s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:21.904252287 (108h26m20.455722748s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.383852597 (108h26m20.935323058s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.528899226 (108h26m21.080369687s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:25.250258635 (108h26m23.801729096s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection Nov 13 20:30:25.250371059 (108h26m23.80184153s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 20:30:25.250582591 (108h26m23.802053052s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection Nov 13 20:30:25.250622577 (108h26m23.802093038s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection Nov 13 20:30:25.250729441 (108h26m23.802199902s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection Nov 13 20:30:25.250859819 (108h26m23.80233028s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection Nov 13 20:30:25.250973465 (108h26m23.802443927s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection Nov 13 20:30:25.251012680 (108h26m23.802483141s) [seg_pos={0 0}, upload=95398]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 20:30:25.508769584 (108h26m24.060240055s) [upload=95398]: Done uploading segments... Nov 13 20:30:25.508818367 (108h26m24.060288828s) [upload=95398]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.949611438s core uplink us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 853.21387ms trace: https://jaeger.dev.storj.io/trace/2f5ab0f8fc51be4b FAIL (2025-11-13 21:02:00 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:40894->45.38.24.2:443: read: connection reset by peer Duration: 853.233327ms edge authservice http us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 830.640736ms trace: https://jaeger.dev.storj.io/trace/2f5ab0f8fc51be4b FAIL (2025-11-13 21:01:36 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:40894->45.38.24.2:443: read: connection reset by peer Duration: 830.662277ms edge authservice http us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................X...........................................] duration: 3.439105868s trace: https://jaeger.dev.storj.io/trace/39fa53ba9384d9d1 FAIL (2025-11-13 21:01:31 UTC) orbiter warning: context deadline exceeded Duration: 3.439297573s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 5.853389512s trace: https://jaeger.dev.storj.io/trace/10b439e9c7de2697 FAIL (2025-11-13 21:01:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ent. Nov 13 20:29:56.447258646 (108h25m54.998729097s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.448610229 (108h25m55.00008068s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449193800 (108h25m55.000664251s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449745831 (108h25m55.001216292s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449971391 (108h25m55.001441842s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450522891 (108h25m55.001993352s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450954152 (108h25m55.002424613s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.451475856 (108h25m55.002946307s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.452296839 (108h25m55.00376729s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453022320 (108h25m55.004492781s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453944366 (108h25m55.005414827s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.454766252 (108h25m55.006236703s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.455465523 (108h25m55.006935984s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456308909 (108h25m55.00777937s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456875248 (108h25m55.008345709s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.457343329 (108h25m55.00881379s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:58.537001464 (108h25m57.088471915s) [upload=95398]: Next returned nil segment Nov 13 20:29:58.537023396 (108h25m57.088493847s) [upload=95398]: Waiting for error group managing segments... Nov 13 20:29:58.876118614 (108h25m57.427589065s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.165170590 (108h25m57.716641051s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.186011275 (108h25m57.737481726s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.269377922 (108h25m57.820848383s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.283448366 (108h25m57.834918817s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.432453102 (108h25m57.983923553s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:21.904252287 (108h26m20.455722748s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.383852597 (108h26m20.935323058s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.528899226 (108h26m21.080369687s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:25.250258635 (108h26m23.801729096s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection Nov 13 20:30:25.250371059 (108h26m23.80184153s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 20:30:25.250582591 (108h26m23.802053052s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection Nov 13 20:30:25.250622577 (108h26m23.802093038s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection Nov 13 20:30:25.250729441 (108h26m23.802199902s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection Nov 13 20:30:25.250859819 (108h26m23.80233028s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection Nov 13 20:30:25.250973465 (108h26m23.802443927s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection Nov 13 20:30:25.251012680 (108h26m23.802483141s) [seg_pos={0 0}, upload=95398]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 20:30:25.508769584 (108h26m24.060240055s) [upload=95398]: Done uploading segments... Nov 13 20:30:25.508818367 (108h26m24.060288828s) [upload=95398]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.853422395s core uplink us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 821.780293ms trace: https://jaeger.dev.storj.io/trace/2f5ab0f8fc51be4b FAIL (2025-11-13 21:01:15 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:40894->45.38.24.2:443: read: connection reset by peer Duration: 821.819558ms edge authservice http us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................X..........................................] duration: 3.448207176s trace: https://jaeger.dev.storj.io/trace/39fa53ba9384d9d1 FAIL (2025-11-13 21:00:50 UTC) orbiter warning: context deadline exceeded Duration: 3.448306374s core uplink us-select-1 us1-placement-20
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 823.363046ms trace: https://jaeger.dev.storj.io/trace/2f5ab0f8fc51be4b FAIL (2025-11-13 21:00:30 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:40894->45.38.24.2:443: read: connection reset by peer Duration: 823.394987ms edge authservice http us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................X.........................................] duration: 3.454602763s trace: https://jaeger.dev.storj.io/trace/39fa53ba9384d9d1 FAIL (2025-11-13 21:00:19 UTC) orbiter warning: context deadline exceeded Duration: 3.454623593s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 6.082933781s trace: https://jaeger.dev.storj.io/trace/10b439e9c7de2697 FAIL (2025-11-13 21:00:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ent. Nov 13 20:29:56.447258646 (108h25m54.998729097s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.448610229 (108h25m55.00008068s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449193800 (108h25m55.000664251s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449745831 (108h25m55.001216292s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449971391 (108h25m55.001441842s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450522891 (108h25m55.001993352s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450954152 (108h25m55.002424613s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.451475856 (108h25m55.002946307s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.452296839 (108h25m55.00376729s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453022320 (108h25m55.004492781s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453944366 (108h25m55.005414827s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.454766252 (108h25m55.006236703s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.455465523 (108h25m55.006935984s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456308909 (108h25m55.00777937s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456875248 (108h25m55.008345709s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.457343329 (108h25m55.00881379s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:58.537001464 (108h25m57.088471915s) [upload=95398]: Next returned nil segment Nov 13 20:29:58.537023396 (108h25m57.088493847s) [upload=95398]: Waiting for error group managing segments... Nov 13 20:29:58.876118614 (108h25m57.427589065s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.165170590 (108h25m57.716641051s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.186011275 (108h25m57.737481726s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.269377922 (108h25m57.820848383s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.283448366 (108h25m57.834918817s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.432453102 (108h25m57.983923553s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:21.904252287 (108h26m20.455722748s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.383852597 (108h26m20.935323058s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.528899226 (108h26m21.080369687s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:25.250258635 (108h26m23.801729096s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection Nov 13 20:30:25.250371059 (108h26m23.80184153s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 20:30:25.250582591 (108h26m23.802053052s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection Nov 13 20:30:25.250622577 (108h26m23.802093038s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection Nov 13 20:30:25.250729441 (108h26m23.802199902s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection Nov 13 20:30:25.250859819 (108h26m23.80233028s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection Nov 13 20:30:25.250973465 (108h26m23.802443927s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection Nov 13 20:30:25.251012680 (108h26m23.802483141s) [seg_pos={0 0}, upload=95398]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 20:30:25.508769584 (108h26m24.060240055s) [upload=95398]: Done uploading segments... Nov 13 20:30:25.508818367 (108h26m24.060288828s) [upload=95398]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 6.082966243s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 5.910154704s trace: https://jaeger.dev.storj.io/trace/10b439e9c7de2697 FAIL (2025-11-13 20:59:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ent. Nov 13 20:29:56.447258646 (108h25m54.998729097s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.448610229 (108h25m55.00008068s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449193800 (108h25m55.000664251s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449745831 (108h25m55.001216292s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449971391 (108h25m55.001441842s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450522891 (108h25m55.001993352s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450954152 (108h25m55.002424613s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.451475856 (108h25m55.002946307s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.452296839 (108h25m55.00376729s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453022320 (108h25m55.004492781s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453944366 (108h25m55.005414827s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.454766252 (108h25m55.006236703s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.455465523 (108h25m55.006935984s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456308909 (108h25m55.00777937s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456875248 (108h25m55.008345709s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.457343329 (108h25m55.00881379s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:58.537001464 (108h25m57.088471915s) [upload=95398]: Next returned nil segment Nov 13 20:29:58.537023396 (108h25m57.088493847s) [upload=95398]: Waiting for error group managing segments... Nov 13 20:29:58.876118614 (108h25m57.427589065s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.165170590 (108h25m57.716641051s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.186011275 (108h25m57.737481726s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.269377922 (108h25m57.820848383s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.283448366 (108h25m57.834918817s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.432453102 (108h25m57.983923553s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:21.904252287 (108h26m20.455722748s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.383852597 (108h26m20.935323058s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.528899226 (108h26m21.080369687s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:25.250258635 (108h26m23.801729096s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection Nov 13 20:30:25.250371059 (108h26m23.80184153s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 20:30:25.250582591 (108h26m23.802053052s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection Nov 13 20:30:25.250622577 (108h26m23.802093038s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection Nov 13 20:30:25.250729441 (108h26m23.802199902s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection Nov 13 20:30:25.250859819 (108h26m23.80233028s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection Nov 13 20:30:25.250973465 (108h26m23.802443927s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection Nov 13 20:30:25.251012680 (108h26m23.802483141s) [seg_pos={0 0}, upload=95398]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 20:30:25.508769584 (108h26m24.060240055s) [upload=95398]: Done uploading segments... Nov 13 20:30:25.508818367 (108h26m24.060288828s) [upload=95398]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.910183629s core uplink us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 854.050562ms trace: https://jaeger.dev.storj.io/trace/2f5ab0f8fc51be4b FAIL (2025-11-13 20:59:54 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:40894->45.38.24.2:443: read: connection reset by peer Duration: 854.084818ms edge authservice http us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................X........................................] duration: 3.415069946s trace: https://jaeger.dev.storj.io/trace/39fa53ba9384d9d1 FAIL (2025-11-13 20:59:27 UTC) orbiter warning: context deadline exceeded Duration: 3.415096967s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 6.020123221s trace: https://jaeger.dev.storj.io/trace/10b439e9c7de2697 FAIL (2025-11-13 20:59:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ent. Nov 13 20:29:56.447258646 (108h25m54.998729097s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.448610229 (108h25m55.00008068s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449193800 (108h25m55.000664251s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449745831 (108h25m55.001216292s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449971391 (108h25m55.001441842s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450522891 (108h25m55.001993352s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450954152 (108h25m55.002424613s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.451475856 (108h25m55.002946307s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.452296839 (108h25m55.00376729s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453022320 (108h25m55.004492781s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453944366 (108h25m55.005414827s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.454766252 (108h25m55.006236703s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.455465523 (108h25m55.006935984s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456308909 (108h25m55.00777937s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456875248 (108h25m55.008345709s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.457343329 (108h25m55.00881379s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:58.537001464 (108h25m57.088471915s) [upload=95398]: Next returned nil segment Nov 13 20:29:58.537023396 (108h25m57.088493847s) [upload=95398]: Waiting for error group managing segments... Nov 13 20:29:58.876118614 (108h25m57.427589065s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.165170590 (108h25m57.716641051s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.186011275 (108h25m57.737481726s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.269377922 (108h25m57.820848383s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.283448366 (108h25m57.834918817s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.432453102 (108h25m57.983923553s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:21.904252287 (108h26m20.455722748s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.383852597 (108h26m20.935323058s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.528899226 (108h26m21.080369687s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:25.250258635 (108h26m23.801729096s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection Nov 13 20:30:25.250371059 (108h26m23.80184153s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 20:30:25.250582591 (108h26m23.802053052s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection Nov 13 20:30:25.250622577 (108h26m23.802093038s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection Nov 13 20:30:25.250729441 (108h26m23.802199902s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection Nov 13 20:30:25.250859819 (108h26m23.80233028s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection Nov 13 20:30:25.250973465 (108h26m23.802443927s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection Nov 13 20:30:25.251012680 (108h26m23.802483141s) [seg_pos={0 0}, upload=95398]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 20:30:25.508769584 (108h26m24.060240055s) [upload=95398]: Done uploading segments... Nov 13 20:30:25.508818367 (108h26m24.060288828s) [upload=95398]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 6.020152967s core uplink us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 811.078114ms trace: https://jaeger.dev.storj.io/trace/2f5ab0f8fc51be4b FAIL (2025-11-13 20:58:59 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:40894->45.38.24.2:443: read: connection reset by peer Duration: 811.107019ms edge authservice http us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 5.899809028s trace: https://jaeger.dev.storj.io/trace/10b439e9c7de2697 FAIL (2025-11-13 20:58:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ent. Nov 13 20:29:56.447258646 (108h25m54.998729097s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.448610229 (108h25m55.00008068s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449193800 (108h25m55.000664251s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449745831 (108h25m55.001216292s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449971391 (108h25m55.001441842s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450522891 (108h25m55.001993352s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450954152 (108h25m55.002424613s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.451475856 (108h25m55.002946307s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.452296839 (108h25m55.00376729s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453022320 (108h25m55.004492781s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453944366 (108h25m55.005414827s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.454766252 (108h25m55.006236703s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.455465523 (108h25m55.006935984s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456308909 (108h25m55.00777937s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456875248 (108h25m55.008345709s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.457343329 (108h25m55.00881379s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:58.537001464 (108h25m57.088471915s) [upload=95398]: Next returned nil segment Nov 13 20:29:58.537023396 (108h25m57.088493847s) [upload=95398]: Waiting for error group managing segments... Nov 13 20:29:58.876118614 (108h25m57.427589065s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.165170590 (108h25m57.716641051s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.186011275 (108h25m57.737481726s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.269377922 (108h25m57.820848383s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.283448366 (108h25m57.834918817s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.432453102 (108h25m57.983923553s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:21.904252287 (108h26m20.455722748s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.383852597 (108h26m20.935323058s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.528899226 (108h26m21.080369687s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:25.250258635 (108h26m23.801729096s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection Nov 13 20:30:25.250371059 (108h26m23.80184153s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 20:30:25.250582591 (108h26m23.802053052s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection Nov 13 20:30:25.250622577 (108h26m23.802093038s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection Nov 13 20:30:25.250729441 (108h26m23.802199902s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection Nov 13 20:30:25.250859819 (108h26m23.80233028s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection Nov 13 20:30:25.250973465 (108h26m23.802443927s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection Nov 13 20:30:25.251012680 (108h26m23.802483141s) [seg_pos={0 0}, upload=95398]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 20:30:25.508769584 (108h26m24.060240055s) [upload=95398]: Done uploading segments... Nov 13 20:30:25.508818367 (108h26m24.060288828s) [upload=95398]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.899840999s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................X.......................................] duration: 3.410842417s trace: https://jaeger.dev.storj.io/trace/39fa53ba9384d9d1 FAIL (2025-11-13 20:58:28 UTC) orbiter warning: context deadline exceeded Duration: 3.410870301s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 5.902442153s trace: https://jaeger.dev.storj.io/trace/10b439e9c7de2697 FAIL (2025-11-13 20:58:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ent. Nov 13 20:29:56.447258646 (108h25m54.998729097s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.448610229 (108h25m55.00008068s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449193800 (108h25m55.000664251s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449745831 (108h25m55.001216292s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449971391 (108h25m55.001441842s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450522891 (108h25m55.001993352s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450954152 (108h25m55.002424613s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.451475856 (108h25m55.002946307s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.452296839 (108h25m55.00376729s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453022320 (108h25m55.004492781s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453944366 (108h25m55.005414827s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.454766252 (108h25m55.006236703s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.455465523 (108h25m55.006935984s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456308909 (108h25m55.00777937s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456875248 (108h25m55.008345709s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.457343329 (108h25m55.00881379s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:58.537001464 (108h25m57.088471915s) [upload=95398]: Next returned nil segment Nov 13 20:29:58.537023396 (108h25m57.088493847s) [upload=95398]: Waiting for error group managing segments... Nov 13 20:29:58.876118614 (108h25m57.427589065s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.165170590 (108h25m57.716641051s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.186011275 (108h25m57.737481726s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.269377922 (108h25m57.820848383s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.283448366 (108h25m57.834918817s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.432453102 (108h25m57.983923553s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:21.904252287 (108h26m20.455722748s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.383852597 (108h26m20.935323058s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.528899226 (108h26m21.080369687s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:25.250258635 (108h26m23.801729096s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection Nov 13 20:30:25.250371059 (108h26m23.80184153s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 20:30:25.250582591 (108h26m23.802053052s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection Nov 13 20:30:25.250622577 (108h26m23.802093038s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection Nov 13 20:30:25.250729441 (108h26m23.802199902s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection Nov 13 20:30:25.250859819 (108h26m23.80233028s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection Nov 13 20:30:25.250973465 (108h26m23.802443927s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection Nov 13 20:30:25.251012680 (108h26m23.802483141s) [seg_pos={0 0}, upload=95398]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 20:30:25.508769584 (108h26m24.060240055s) [upload=95398]: Done uploading segments... Nov 13 20:30:25.508818367 (108h26m24.060288828s) [upload=95398]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.902486648s core uplink us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 808.424081ms trace: https://jaeger.dev.storj.io/trace/2f5ab0f8fc51be4b FAIL (2025-11-13 20:58:15 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:40894->45.38.24.2:443: read: connection reset by peer Duration: 808.458917ms edge authservice http us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................X......................................] duration: 3.448010263s trace: https://jaeger.dev.storj.io/trace/39fa53ba9384d9d1 FAIL (2025-11-13 20:57:45 UTC) orbiter warning: context deadline exceeded Duration: 3.448103309s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 5.807490567s trace: https://jaeger.dev.storj.io/trace/10b439e9c7de2697 FAIL (2025-11-13 20:57:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ent. Nov 13 20:29:56.447258646 (108h25m54.998729097s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.448610229 (108h25m55.00008068s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449193800 (108h25m55.000664251s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449745831 (108h25m55.001216292s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449971391 (108h25m55.001441842s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450522891 (108h25m55.001993352s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450954152 (108h25m55.002424613s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.451475856 (108h25m55.002946307s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.452296839 (108h25m55.00376729s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453022320 (108h25m55.004492781s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453944366 (108h25m55.005414827s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.454766252 (108h25m55.006236703s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.455465523 (108h25m55.006935984s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456308909 (108h25m55.00777937s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456875248 (108h25m55.008345709s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.457343329 (108h25m55.00881379s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:58.537001464 (108h25m57.088471915s) [upload=95398]: Next returned nil segment Nov 13 20:29:58.537023396 (108h25m57.088493847s) [upload=95398]: Waiting for error group managing segments... Nov 13 20:29:58.876118614 (108h25m57.427589065s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.165170590 (108h25m57.716641051s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.186011275 (108h25m57.737481726s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.269377922 (108h25m57.820848383s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.283448366 (108h25m57.834918817s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.432453102 (108h25m57.983923553s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:21.904252287 (108h26m20.455722748s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.383852597 (108h26m20.935323058s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.528899226 (108h26m21.080369687s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:25.250258635 (108h26m23.801729096s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection Nov 13 20:30:25.250371059 (108h26m23.80184153s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 20:30:25.250582591 (108h26m23.802053052s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection Nov 13 20:30:25.250622577 (108h26m23.802093038s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection Nov 13 20:30:25.250729441 (108h26m23.802199902s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection Nov 13 20:30:25.250859819 (108h26m23.80233028s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection Nov 13 20:30:25.250973465 (108h26m23.802443927s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection Nov 13 20:30:25.251012680 (108h26m23.802483141s) [seg_pos={0 0}, upload=95398]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 20:30:25.508769584 (108h26m24.060240055s) [upload=95398]: Done uploading segments... Nov 13 20:30:25.508818367 (108h26m24.060288828s) [upload=95398]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.807517208s core uplink us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 854.153228ms trace: https://jaeger.dev.storj.io/trace/2f5ab0f8fc51be4b FAIL (2025-11-13 20:57:19 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:40894->45.38.24.2:443: read: connection reset by peer Duration: 854.182704ms edge authservice http us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................X.....................................] duration: 3.543021552s trace: https://jaeger.dev.storj.io/trace/39fa53ba9384d9d1 FAIL (2025-11-13 20:57:14 UTC) orbiter warning: context deadline exceeded Duration: 3.54304099s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 5.952648562s trace: https://jaeger.dev.storj.io/trace/10b439e9c7de2697 FAIL (2025-11-13 20:56:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ent. Nov 13 20:29:56.447258646 (108h25m54.998729097s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.448610229 (108h25m55.00008068s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449193800 (108h25m55.000664251s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449745831 (108h25m55.001216292s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449971391 (108h25m55.001441842s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450522891 (108h25m55.001993352s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450954152 (108h25m55.002424613s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.451475856 (108h25m55.002946307s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.452296839 (108h25m55.00376729s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453022320 (108h25m55.004492781s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453944366 (108h25m55.005414827s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.454766252 (108h25m55.006236703s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.455465523 (108h25m55.006935984s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456308909 (108h25m55.00777937s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456875248 (108h25m55.008345709s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.457343329 (108h25m55.00881379s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:58.537001464 (108h25m57.088471915s) [upload=95398]: Next returned nil segment Nov 13 20:29:58.537023396 (108h25m57.088493847s) [upload=95398]: Waiting for error group managing segments... Nov 13 20:29:58.876118614 (108h25m57.427589065s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.165170590 (108h25m57.716641051s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.186011275 (108h25m57.737481726s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.269377922 (108h25m57.820848383s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.283448366 (108h25m57.834918817s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.432453102 (108h25m57.983923553s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:21.904252287 (108h26m20.455722748s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.383852597 (108h26m20.935323058s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.528899226 (108h26m21.080369687s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:25.250258635 (108h26m23.801729096s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection Nov 13 20:30:25.250371059 (108h26m23.80184153s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 20:30:25.250582591 (108h26m23.802053052s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection Nov 13 20:30:25.250622577 (108h26m23.802093038s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection Nov 13 20:30:25.250729441 (108h26m23.802199902s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection Nov 13 20:30:25.250859819 (108h26m23.80233028s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection Nov 13 20:30:25.250973465 (108h26m23.802443927s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection Nov 13 20:30:25.251012680 (108h26m23.802483141s) [seg_pos={0 0}, upload=95398]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 20:30:25.508769584 (108h26m24.060240055s) [upload=95398]: Done uploading segments... Nov 13 20:30:25.508818367 (108h26m24.060288828s) [upload=95398]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.952684229s core uplink us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 799.378295ms trace: https://jaeger.dev.storj.io/trace/2f5ab0f8fc51be4b FAIL (2025-11-13 20:56:45 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:40894->45.38.24.2:443: read: connection reset by peer Duration: 799.40189ms edge authservice http us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 5.980382123s trace: https://jaeger.dev.storj.io/trace/10b439e9c7de2697 FAIL (2025-11-13 20:56:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ent. Nov 13 20:29:56.447258646 (108h25m54.998729097s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.448610229 (108h25m55.00008068s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449193800 (108h25m55.000664251s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449745831 (108h25m55.001216292s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449971391 (108h25m55.001441842s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450522891 (108h25m55.001993352s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450954152 (108h25m55.002424613s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.451475856 (108h25m55.002946307s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.452296839 (108h25m55.00376729s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453022320 (108h25m55.004492781s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453944366 (108h25m55.005414827s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.454766252 (108h25m55.006236703s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.455465523 (108h25m55.006935984s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456308909 (108h25m55.00777937s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456875248 (108h25m55.008345709s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.457343329 (108h25m55.00881379s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:58.537001464 (108h25m57.088471915s) [upload=95398]: Next returned nil segment Nov 13 20:29:58.537023396 (108h25m57.088493847s) [upload=95398]: Waiting for error group managing segments... Nov 13 20:29:58.876118614 (108h25m57.427589065s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.165170590 (108h25m57.716641051s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.186011275 (108h25m57.737481726s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.269377922 (108h25m57.820848383s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.283448366 (108h25m57.834918817s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.432453102 (108h25m57.983923553s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:21.904252287 (108h26m20.455722748s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.383852597 (108h26m20.935323058s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.528899226 (108h26m21.080369687s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:25.250258635 (108h26m23.801729096s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection Nov 13 20:30:25.250371059 (108h26m23.80184153s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 20:30:25.250582591 (108h26m23.802053052s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection Nov 13 20:30:25.250622577 (108h26m23.802093038s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection Nov 13 20:30:25.250729441 (108h26m23.802199902s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection Nov 13 20:30:25.250859819 (108h26m23.80233028s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection Nov 13 20:30:25.250973465 (108h26m23.802443927s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection Nov 13 20:30:25.251012680 (108h26m23.802483141s) [seg_pos={0 0}, upload=95398]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 20:30:25.508769584 (108h26m24.060240055s) [upload=95398]: Done uploading segments... Nov 13 20:30:25.508818367 (108h26m24.060288828s) [upload=95398]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.980412911s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................X....................................] duration: 3.44441869s trace: https://jaeger.dev.storj.io/trace/39fa53ba9384d9d1 FAIL (2025-11-13 20:56:20 UTC) orbiter warning: context deadline exceeded Duration: 3.444446714s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 5.861715582s trace: https://jaeger.dev.storj.io/trace/10b439e9c7de2697 FAIL (2025-11-13 20:55:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ent. Nov 13 20:29:56.447258646 (108h25m54.998729097s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.448610229 (108h25m55.00008068s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449193800 (108h25m55.000664251s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449745831 (108h25m55.001216292s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449971391 (108h25m55.001441842s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450522891 (108h25m55.001993352s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450954152 (108h25m55.002424613s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.451475856 (108h25m55.002946307s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.452296839 (108h25m55.00376729s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453022320 (108h25m55.004492781s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453944366 (108h25m55.005414827s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.454766252 (108h25m55.006236703s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.455465523 (108h25m55.006935984s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456308909 (108h25m55.00777937s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456875248 (108h25m55.008345709s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.457343329 (108h25m55.00881379s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:58.537001464 (108h25m57.088471915s) [upload=95398]: Next returned nil segment Nov 13 20:29:58.537023396 (108h25m57.088493847s) [upload=95398]: Waiting for error group managing segments... Nov 13 20:29:58.876118614 (108h25m57.427589065s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.165170590 (108h25m57.716641051s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.186011275 (108h25m57.737481726s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.269377922 (108h25m57.820848383s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.283448366 (108h25m57.834918817s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.432453102 (108h25m57.983923553s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:21.904252287 (108h26m20.455722748s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.383852597 (108h26m20.935323058s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.528899226 (108h26m21.080369687s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:25.250258635 (108h26m23.801729096s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection Nov 13 20:30:25.250371059 (108h26m23.80184153s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 20:30:25.250582591 (108h26m23.802053052s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection Nov 13 20:30:25.250622577 (108h26m23.802093038s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection Nov 13 20:30:25.250729441 (108h26m23.802199902s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection Nov 13 20:30:25.250859819 (108h26m23.80233028s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection Nov 13 20:30:25.250973465 (108h26m23.802443927s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection Nov 13 20:30:25.251012680 (108h26m23.802483141s) [seg_pos={0 0}, upload=95398]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 20:30:25.508769584 (108h26m24.060240055s) [upload=95398]: Done uploading segments... Nov 13 20:30:25.508818367 (108h26m24.060288828s) [upload=95398]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.861735059s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................X...................................] duration: 3.485707717s trace: https://jaeger.dev.storj.io/trace/39fa53ba9384d9d1 FAIL (2025-11-13 20:55:44 UTC) orbiter warning: context deadline exceeded Duration: 3.485738886s core uplink us-select-1 us1-placement-20
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 817.94592ms trace: https://jaeger.dev.storj.io/trace/2f5ab0f8fc51be4b FAIL (2025-11-13 20:55:30 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:40894->45.38.24.2:443: read: connection reset by peer Duration: 817.983522ms edge authservice http us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 5.76608348s trace: https://jaeger.dev.storj.io/trace/10b439e9c7de2697 FAIL (2025-11-13 20:55:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ent. Nov 13 20:29:56.447258646 (108h25m54.998729097s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.448610229 (108h25m55.00008068s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449193800 (108h25m55.000664251s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449745831 (108h25m55.001216292s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449971391 (108h25m55.001441842s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450522891 (108h25m55.001993352s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450954152 (108h25m55.002424613s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.451475856 (108h25m55.002946307s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.452296839 (108h25m55.00376729s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453022320 (108h25m55.004492781s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453944366 (108h25m55.005414827s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.454766252 (108h25m55.006236703s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.455465523 (108h25m55.006935984s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456308909 (108h25m55.00777937s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456875248 (108h25m55.008345709s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.457343329 (108h25m55.00881379s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:58.537001464 (108h25m57.088471915s) [upload=95398]: Next returned nil segment Nov 13 20:29:58.537023396 (108h25m57.088493847s) [upload=95398]: Waiting for error group managing segments... Nov 13 20:29:58.876118614 (108h25m57.427589065s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.165170590 (108h25m57.716641051s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.186011275 (108h25m57.737481726s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.269377922 (108h25m57.820848383s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.283448366 (108h25m57.834918817s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.432453102 (108h25m57.983923553s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:21.904252287 (108h26m20.455722748s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.383852597 (108h26m20.935323058s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.528899226 (108h26m21.080369687s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:25.250258635 (108h26m23.801729096s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection Nov 13 20:30:25.250371059 (108h26m23.80184153s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 20:30:25.250582591 (108h26m23.802053052s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection Nov 13 20:30:25.250622577 (108h26m23.802093038s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection Nov 13 20:30:25.250729441 (108h26m23.802199902s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection Nov 13 20:30:25.250859819 (108h26m23.80233028s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection Nov 13 20:30:25.250973465 (108h26m23.802443927s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection Nov 13 20:30:25.251012680 (108h26m23.802483141s) [seg_pos={0 0}, upload=95398]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 20:30:25.508769584 (108h26m24.060240055s) [upload=95398]: Done uploading segments... Nov 13 20:30:25.508818367 (108h26m24.060288828s) [upload=95398]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.766108819s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................X..................................] duration: 3.426077107s trace: https://jaeger.dev.storj.io/trace/39fa53ba9384d9d1 FAIL (2025-11-13 20:54:51 UTC) orbiter warning: context deadline exceeded Duration: 3.426105782s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................X.................................] duration: 3.35632754s trace: https://jaeger.dev.storj.io/trace/39fa53ba9384d9d1 FAIL (2025-11-13 20:54:26 UTC) orbiter warning: context deadline exceeded Duration: 3.356343931s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 5.917969892s trace: https://jaeger.dev.storj.io/trace/10b439e9c7de2697 FAIL (2025-11-13 20:54:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ent. Nov 13 20:29:56.447258646 (108h25m54.998729097s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.448610229 (108h25m55.00008068s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449193800 (108h25m55.000664251s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449745831 (108h25m55.001216292s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449971391 (108h25m55.001441842s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450522891 (108h25m55.001993352s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450954152 (108h25m55.002424613s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.451475856 (108h25m55.002946307s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.452296839 (108h25m55.00376729s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453022320 (108h25m55.004492781s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453944366 (108h25m55.005414827s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.454766252 (108h25m55.006236703s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.455465523 (108h25m55.006935984s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456308909 (108h25m55.00777937s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456875248 (108h25m55.008345709s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.457343329 (108h25m55.00881379s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:58.537001464 (108h25m57.088471915s) [upload=95398]: Next returned nil segment Nov 13 20:29:58.537023396 (108h25m57.088493847s) [upload=95398]: Waiting for error group managing segments... Nov 13 20:29:58.876118614 (108h25m57.427589065s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.165170590 (108h25m57.716641051s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.186011275 (108h25m57.737481726s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.269377922 (108h25m57.820848383s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.283448366 (108h25m57.834918817s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.432453102 (108h25m57.983923553s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:21.904252287 (108h26m20.455722748s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.383852597 (108h26m20.935323058s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.528899226 (108h26m21.080369687s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:25.250258635 (108h26m23.801729096s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection Nov 13 20:30:25.250371059 (108h26m23.80184153s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 20:30:25.250582591 (108h26m23.802053052s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection Nov 13 20:30:25.250622577 (108h26m23.802093038s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection Nov 13 20:30:25.250729441 (108h26m23.802199902s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection Nov 13 20:30:25.250859819 (108h26m23.80233028s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection Nov 13 20:30:25.250973465 (108h26m23.802443927s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection Nov 13 20:30:25.251012680 (108h26m23.802483141s) [seg_pos={0 0}, upload=95398]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 20:30:25.508769584 (108h26m24.060240055s) [upload=95398]: Done uploading segments... Nov 13 20:30:25.508818367 (108h26m24.060288828s) [upload=95398]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.918006082s core uplink us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 845.824287ms trace: https://jaeger.dev.storj.io/trace/2f5ab0f8fc51be4b FAIL (2025-11-13 20:54:16 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:40894->45.38.24.2:443: read: connection reset by peer Duration: 845.865496ms edge authservice http us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 6.014032908s trace: https://jaeger.dev.storj.io/trace/10b439e9c7de2697 FAIL (2025-11-13 20:53:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ent. Nov 13 20:29:56.447258646 (108h25m54.998729097s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.448610229 (108h25m55.00008068s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449193800 (108h25m55.000664251s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449745831 (108h25m55.001216292s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.449971391 (108h25m55.001441842s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450522891 (108h25m55.001993352s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.450954152 (108h25m55.002424613s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.451475856 (108h25m55.002946307s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.452296839 (108h25m55.00376729s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453022320 (108h25m55.004492781s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.453944366 (108h25m55.005414827s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.454766252 (108h25m55.006236703s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.455465523 (108h25m55.006935984s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456308909 (108h25m55.00777937s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.456875248 (108h25m55.008345709s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:56.457343329 (108h25m55.00881379s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Uploading piece... Nov 13 20:29:58.537001464 (108h25m57.088471915s) [upload=95398]: Next returned nil segment Nov 13 20:29:58.537023396 (108h25m57.088493847s) [upload=95398]: Waiting for error group managing segments... Nov 13 20:29:58.876118614 (108h25m57.427589065s) [noise=true , address=77.90.31.21:4878 , piece_id=BIKNYPH7M3YK74WTRZUYMWBBWFEMK5L5CYPP4PYD73NZOR4LUPZA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.165170590 (108h25m57.716641051s) [noise=true , address=45.91.116.7:4009 , piece_id=X4KVFFECCPQ7ADH2CMLV3TSTX5BAXOA2QMDOY3VNABH4X3DUS44A, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.186011275 (108h25m57.737481726s) [noise=true , address=45.91.116.14:4190 , piece_id=7FZ2LAJIQRGUR3OTAAK3TJHL4Q652VLIPBP2ME2ZDEREU6TCLUHA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.269377922 (108h25m57.820848383s) [noise=true , address=45.91.116.12:4226 , piece_id=TDZZQO34YBY7FAHTVIV6DG26P3I5UADWATWOBRYCTTPTASULUWCA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.283448366 (108h25m57.834918817s) [noise=true , address=45.91.116.4:4937 , piece_id=RVYNGPSUBHWSUB545UNHDNF2NGKZBIOTAAK3A4UTARIGKJO4GLIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:29:59.432453102 (108h25m57.983923553s) [noise=true , address=77.90.31.12:4592 , piece_id=FBNPBRYEKSCNICAKDT3CLEAUM5VYFAXDW3CBNJJKLWU536CEKFBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:21.904252287 (108h26m20.455722748s) [noise=true , address=145.79.227.10:4401 , piece_id=T52SWTQ4IB22QLLRT3XS3PO6JXBP4A5FFKM4YSMZCWIUFWCZHG7Q, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.383852597 (108h26m20.935323058s) [noise=true , address=145.79.227.24:4582 , piece_id=PB5KZFLEAL5MBGI32ZOOZLDLIBF7UXFELSGQH2TPOF4IDTZZYAYA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:22.528899226 (108h26m21.080369687s) [noise=true , address=145.79.227.4:4187 , piece_id=FNW3V6NN7JZS2OJSYN66SL73LESZ22OH3RO4GVCSPO2BXXS3Z5JQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: <nil> Nov 13 20:30:25.250258635 (108h26m23.801729096s) [noise=true , address=145.79.227.17:4970 , piece_id=FIR4ESUBT5BVU5SMBYHYZ3KHDGI2CHAIOJKQUG7UWMLJGQGHVXKA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49772->145.79.227.17:4970: use of closed network connection Nov 13 20:30:25.250371059 (108h26m23.80184153s) [noise=true , address=145.79.227.30:4762 , piece_id=Y5EINUVB22VIDLNOLFQJBSTNGCCIMKH3WZNQUY5RN66AOTS32LSQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:145.79.227.30:4762): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 20:30:25.250582591 (108h26m23.802053052s) [noise=true , address=145.79.227.16:4912 , piece_id=CDJX7UKAK6AWROXT4OV7CC2LLXL7COGVPRMIOTAW4EYAFUF7USBQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49004->145.79.227.16:4912: use of closed network connection Nov 13 20:30:25.250622577 (108h26m23.802093038s) [noise=true , address=145.79.227.21:4475 , piece_id=2T2MRQUIL7PGTZVURXICPR7QGB2E6Q6LUKVOKCTSHU2RJBV3XWQQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:128SC59hAFBUVutPFym3iZv9qNYtgiqrk6q9RpfWL2ftyTt5Tzk, address:145.79.227.21:4475): protocol: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50372->145.79.227.21:4475: use of closed network connection Nov 13 20:30:25.250729441 (108h26m23.802199902s) [noise=true , address=145.79.227.23:4547 , piece_id=K6KROL5BYTJAXWNVBN4FAYJ2FAD673F2WS7FCORXR3NIVI7VQWIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35934->145.79.227.23:4547: use of closed network connection Nov 13 20:30:25.250859819 (108h26m23.80233028s) [noise=true , address=145.79.227.31:4799 , piece_id=Z2XQXKTS7WHH3WWBCJPT74QEIO2AFC6TVAAARQYEU4HCB7GYZDFA, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38738->145.79.227.31:4799: use of closed network connection Nov 13 20:30:25.250973465 (108h26m23.802443927s) [noise=true , address=145.79.227.28:4725 , piece_id=FZ7AOYWHNV3SSLXWU6CTICKN523UEEE3KV6EO4FJXOMK3BIOLCIQ, seg_pos={0 0}, upload=95398]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44538->145.79.227.28:4725: use of closed network connection Nov 13 20:30:25.251012680 (108h26m23.802483141s) [seg_pos={0 0}, upload=95398]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 20:30:25.508769584 (108h26m24.060240055s) [upload=95398]: Done uploading segments... Nov 13 20:30:25.508818367 (108h26m24.060288828s) [upload=95398]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 6.014061341s core uplink us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 818.708803ms trace: https://jaeger.dev.storj.io/trace/2f5ab0f8fc51be4b FAIL (2025-11-13 20:53:22 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:40894->45.38.24.2:443: read: connection reset by peer Duration: 818.731585ms edge authservice http us-select-1

Orbiter Check History

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)